Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
# Workflow: release.yml Environment: pypi
# (A *pending* publisher lets the very first release create the project on PyPI.)
# 2. Cut a release whose tag matches the version in pyproject.toml, e.g.:
# gh release create v0.1.1 --generate-notes
# gh release create v0.2.0 --generate-notes
# This workflow then builds the sdist + wheel and uploads them automatically.
name: Release

Expand Down
32 changes: 23 additions & 9 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@
Notable changes to raw2features, newest first. This project follows
[Semantic Versioning](https://semver.org).

## [0.1.1] - 2026-07-16
## [0.2.0] - Unreleased

This is a focused correctness, reproducibility, and interoperability release. It keeps
the v0.1 store layout and geometry identity while making resume decisions stricter and
multi-grid workflows fully additive.
This release extends named-channel multiplex support with a `channelwise` strategy, adds
four patch encoders, and includes a set of correctness and release-safety fixes. The v0.1
store layout and existing brightfield workflows remain unchanged.

### Correctness and reproducibility

Expand All @@ -33,11 +33,21 @@ multi-grid workflows fully additive.

### Models and interoperability

- Add a pluggable multiplex-strategy layer and a built-in `channelwise` strategy for
applying registered RGB patch encoders to named-channel multiplex tissue images. It
records positional channel identity (including a strict metadata-file fallback), marker
selection and order, normalization, RGB conversion, base-model identity, mean/concat
aggregation, and model-agnostic slide pooling without changing brightfield outputs.
- Bind native multiplex outputs to the complete effective positional panel. Nuclear
masking accepts one unambiguous nuclear channel or a recognized same-stain group, such
as DNA1/DNA2 or repeated DAPI channels. It averages each multi-channel group in float32
and binds every physical index into grid identity; `--no-seg` remains supported.
- Add pinned, forward-validated H0-mini, KEEP, OpenMidnight, and OpenPath patch encoders,
with their licenses, access requirements, preprocessing, and weight hashes recorded.
- Correct cached OME-Zarr reads for noncanonical spatial axis order. The reader accepts
x/y in either order (with an optional channel axis) and warns whenever another
non-singleton axis is reduced to index zero.
non-singleton axis is reduced to index zero. Unnamed `omero.channels` entries now
retain their physical positions instead of shifting later marker identities.
- Refresh the development lock to current `ngff-zarr`/`ome-zarr` releases while retaining
the existing declared compatibility floor.
- Choose the float-pixel `[0,1]` versus `[0,255]` convention once per opened slide instead
Expand All @@ -56,11 +66,15 @@ multi-grid workflows fully additive.

### Compatibility and migration

- Existing v0.1 stores remain readable; the embeddings schema and geometry-only
`grid_hash` are unchanged.
- Existing v0.1 stores remain readable; the store layout/schema and model-independent
grid-identity contract are unchanged. Multiplex nuclear masking now binds the resolved
physical nuclear-channel indices. Averaging a recognized same-stain group is an
intentional value change from v0.1 and therefore creates a new grid. Compatible
historical native-multiplex grids remain appendable only when the original OME panel
proves that the v0.1 selector used the same single physical channel.
- The first request for a model output created before fingerprints were introduced will
recompute that model in place. The old header is deliberately not treated as proof that
pre-v0.1.1 revision pins were enforced.
pre-v0.2.0 revision pins were enforced.
- Plain local output IDs remain basename-based for v0.1 compatibility. Source comparison
prevents accidental reuse, and manifest preflight rejects duplicate derived IDs.
- Credentials already persisted by v0.1.0 are not rewritten by this release and should be
Expand Down Expand Up @@ -102,4 +116,4 @@ What 0.1.0 provides:
harness, and six runnable tutorial notebooks.

[0.1.0]: https://github.com/CraigMyles/raw2features/releases/tag/v0.1.0
[0.1.1]: https://github.com/CraigMyles/raw2features/compare/v0.1.0...v0.1.1
[0.2.0]: https://github.com/CraigMyles/raw2features/compare/v0.1.0...v0.2.0
2 changes: 1 addition & 1 deletion CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ keywords:
- foundation models
- feature extraction
license: MIT
version: "0.1.1"
version: "0.2.0"
12 changes: 7 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,15 @@ store too.
level such that embeddings are comparable across slides and
datasets.
- **Modular implementation.** Reader, segmenter, patcher, patch-embedder,
slide-embedder, and sink implementations are plugin seams exposed through Python
entry points. Entry points add implementations/families; the CLI's patch-model names
come from the bundled provenance registry, while Python callers can inject external
embedder instances through `embed_slide(..., embedders=[...])`.
multiplex-strategy, slide-embedder, and sink implementations are plugin seams exposed
through Python entry points. Entry points add implementations/families; the CLI's
patch-model names come from the bundled provenance registry, while Python callers can
inject external embedder instances through `embed_slide(..., embedders=[...])`.
- **FAIR & provenance-first.** Stable models' weights are pinned to an **immutable HuggingFace
revision** (or a sha256-pinned URL), with preprocessing sourced from each
model's card. Every output records that provenance plus a 1:1
coords↔features mapping, so an embedding is reproducible and traceable to the exact
weights that made it. SEAL is explicitly experimental in v0.1.1: its adapter is
weights that made it. SEAL is explicitly experimental in v0.2.0: its adapter is
pinned and verified, but its upstream factory still fetches the frozen base from a
mutable revision; see [`MODELS.md`](docs/MODELS.md).

Expand Down Expand Up @@ -114,6 +114,8 @@ on CPU with no model-access-token. Its figures are pre-rendered on GitHub.
**Full guide: [`docs/usage.md`](docs/usage.md)** - every command, what actually
happens under the hood (exact MPP, decode-once fan-out, output schema), the
rerun-safe / skip-if-complete model, thumbnails, and example SLURM cohort runs.
Named-channel inputs, native multiplex encoders, and the `channelwise` RGB-encoder
strategy are covered in [`docs/MODALITIES.md`](docs/MODALITIES.md).

```bash
raw2features info slide.ome.zarr
Expand Down
165 changes: 86 additions & 79 deletions docs/MODALITIES.md
Original file line number Diff line number Diff line change
@@ -1,87 +1,94 @@
# Modalities - brightfield (H&E) + multiplex (spatial proteomics)
# Brightfield and named-channel multiplex tissue images

raw2features is **H&E-first**, but the same pipeline also handles a second modality:
**multiplex** imaging (CODEX / Phenocycler / spatial proteomics), via marker-aware
encoders like KRONOS. H&E runs are completely unchanged.
raw2features supports RGB brightfield images and named-channel multiplex tissue images.
Multiplex sources use native `(H, W, C)` reads and preserve positional channel identity.
Registered RGB encoders can be applied through a multiplex strategy, while native
multiplex encoders consume the channel stack directly. Ordinary brightfield execution is
unchanged.

## How a model declares its modality
Each registry entry carries `modality: brightfield` (default) or `modality: multiplex`.
The runner reads the modality off the requested embedders and routes the per-slide path:
## Execution paths

| stage | brightfield (H&E) | multiplex |
|---|---|---|
| reader | `read_region` → RGB `[H,W,3]` uint8 | `read_region_channels` → native `[H,W,C]` + `channel_names` |
| segmenter | `otsu` (saturation), `canny`, … | `nuclear` - Otsu on the DAPI/Hoechst channel |
| embedder | RGB transform → `[B,3,H,W]` | per-marker norm → `[B,M,H,W]` + marker ids |
Each registry model declares `modality: brightfield` (the default) or
`modality: multiplex`. A strategy wraps a brightfield model without changing its registry
entry.

## Multiplex data → OME-Zarr
Multiplex slides are usually OME-TIFF / TIFF stacks. Convert to a channel-named OME-Zarr:
| stage | brightfield | `channelwise` RGB strategy | native multiplex encoder |
|---|---|---|---|
| reader | `read_region` → RGB `[H,W,3]` uint8 | `read_region_channels` → native `[H,W,C]` | `read_region_channels` → native `[H,W,C]` |
| channel identity | none | selected positional names | positional names resolved by the model |
| segmentation | `otsu`, `canny`, … | one nuclear channel, a recognized same-stain group, or `--no-seg` | one nuclear channel, a recognized same-stain group, or `--no-seg` |
| embedding | one RGB input per patch | one RGB input per selected marker, then mean/concat | model-specific marker stack |

Panel binding happens before receipt or store completion checks. Native multiplex
fingerprints bind the complete effective positional panel; `channelwise` fingerprints
bind the selected physical channel identities and order (all channels when default
selection is used). The full effective panel remains in source/panel provenance. When
nuclear masking is enabled, the resolved physical nuclear-channel index or same-stain
index group is also part of grid identity.

## Channel metadata

The preferred source of channel identity is `omero.channels` in the OME-Zarr metadata.
The reader preserves unnamed entries as empty positional slots rather than shifting later
labels onto the wrong pixels.

For a source whose labels are absent or incomplete, pass `--channel-names-file` with a
complete ordered panel. UTF-8 `.txt`, `.csv`, and `.tsv` files are accepted, with one
unique name per physical C-axis position. Existing non-empty OME labels are treated as
assertions and must agree with the supplied name at the same index. The override is
in-memory only and never rewrites the source. Identity uses only the resolved names.
Provenance also records whether effective names came from OME metadata or the supplied
file and preserves any differing original OME labels. See
[usage.md](usage.md#rgb-encoders-on-named-channel-multiplex-tissue-images) for examples
and marker-selection rules.

## Converting a multiplex TIFF

Multiplex images are often supplied as OME-TIFF or TIFF channel stacks. The included
conversion helper accepts a `(C,Y,X)` TIFF and writes a multiscale OME-Zarr with one
`omero.channels[].label` per channel:

```bash
pip install "raw2features[kronos]"
python scripts/codex_to_omezarr.py SLIDE.tif SLIDE.ome.zarr --markers markers.json --mpp 0.5
pip install "raw2features[zarr]" tifffile imagecodecs
python scripts/codex_to_omezarr.py SLIDE.tif SLIDE.ome.zarr \
--markers markers.json --mpp 0.5
```

`markers.json` is a list (or `{"raw_markers": [...]}`) of one marker name per channel. The
names are written to `omero.channels[].label`; the reader surfaces them as `channel_names`.

## KRONOS - the first multiplex model
`kronos` (MahmoodLab **KRONOSv1**) is a panel-agnostic spatial-proteomics encoder. Needs
the `[kronos]` extra (the public `kronos` package). ViT-S/16 → **384-d** patch embedding.

- **Marker matching - your panel → KRONOS's vocabulary.** KRONOS doesn't ship a
name-resolver, only a vocabulary: the **public** `marker_metadata.csv` (175 markers,
each with a marker id + per-marker mean/std). Mapping *your* channels to it is the
pipeline's job (`Embedder.set_panel`), and KRONOS is panel-agnostic - it embeds whatever
subset of its vocabulary your slide carries. Each `channel_name` is matched: exact name →
common **synonyms** (`pancytokeratin`→`CYTOKERATIN`, `granzymeb`→`GZMB`) → a **CD-number**
in a compound name (`cla_cd162`→`CD162`); `hoechst*`/`dapi` → DAPI; blank/empty cycles
drop silently.
- **The mapping is recorded in the store.** The header's
`panel.kronos` block records the full per-channel resolution under `mapping` - each kept
channel as `{channel, channel_index, kronos_marker, marker_id}` (e.g.
`pancytokeratin → CYTOKERATIN (id 322)`) - plus `kept` / `dropped` / `unmatched` and a
`vocabulary` pointer pinned to the exact `marker_metadata.csv` revision. The SpatialData
export (`INTEROP.md`) surfaces this same map as a tidy, queryable `uns["raw2features_panel"]`
table.
- **Unmatched named channels are surfaced.** A *named* marker that
doesn't resolve (genuinely outside KRONOS's vocabulary, or a synonym we don't yet know)
is dropped, **warned** about (with the `matched/total` coverage), and listed in
`panel.kronos.unmatched`. (Zero matches is a hard error; partial panels are fine.) If an
unmatched marker *is* a KRONOS marker under another name, add it to `_resolve_marker`'s
synonym map.
- Run it like any model:

```bash
raw2features embed SLIDE.ome.zarr OUT -m kronos --mpp 0.5
```

The runner detects the multiplex modality and routes the `nuclear` segmenter + native
N-channel reads automatically.

**Performance.** A multiplex patch is an M-marker stack (~M× the tokens of a single-channel
ViT). The `kronos` family uses **torch SDPA (flash attention) by default** - no xFormers,
fully portable - which on an A100 (M=41) runs at **~41 patches/s in ~1.3 GB at batch 8**,
vs ~20 p/s / 26 GB with KRONOS's upstream naive-attention fallback (it materialises the full
8k×8k attention matrix). Memory is low and predictable, so the default `--batch-size`
fits comfortably; lower it only on small GPUs or very large panels. (Opt out via the
registry `sdpa: false`.) Validated full-slide on GPU against real CODEX multiplex data - a
56-channel panel (44 markers matched, the rest empty/blank cycles) → 28 tissue patches →
`(28, 384)`, all finite.

> KRONOS weights are **CC-BY-NC-ND** (non-commercial), like UNI / Virchow2 / CONCH - so its
> embeddings are non-commercial too. See `MODEL_LICENSES.md`.

### Marker intensity scaling
The transform brings marker intensities to `[0,1]` **by the source dtype** (uint16 → `/65535`,
uint8 → `/255`, float assumed already normalised), then standardises each marker with its
`marker_mean`/`marker_std`. For the uint16 CODEX/Phenocycler case this matches KRONOS's own
feature-extraction pipeline exactly - its loader divides by `max_value` (default `65535`) then
applies the same per-marker `(x − mean) / std`, so the published marker statistics are defined
on the same `[0,1]` scale. Scaling by the dtype range (rather than a hard-coded `65535`) keeps
uint8 / pre-normalised sources correct instead of silently squashing them.

### Adding more multiplex models
Future marker-conditioned models (e.g. KRONOS successors) are added the same way as any
encoder: a registry entry with `modality: multiplex` + a small family class (or reuse the
`kronos` family). The reader / segmenter / runner routing is shared.
`markers.json` is a list, or an object containing `raw_markers`, `channels`, or `markers`.
The converter requires one name per channel. Prefer a domain converter that preserves all
available OME metadata when one is available; this helper is intentionally small.

## RGB encoders through `channelwise`

`channelwise` normalizes selected channels independently, repeats each single-channel
patch across RGB, runs the registered RGB encoder and pooling, then combines the marker
vectors with mean or ordered concatenation. It records the effective panel, normalization
level and values, RGB conversion, base-model output fingerprint, pooling, aggregation,
and output dimension. Model-agnostic slide poolers can consume the result.

This makes it possible to compare models such as UNI or Virchow2 on named-channel data
without representing them as native multiplex models. The strategy's assumptions and
full CLI contract are documented in [usage.md](usage.md#rgb-encoders-on-named-channel-multiplex-tissue-images).

## Native multiplex encoders

Native multiplex models use the same positional-panel plumbing but consume the marker
stack directly. The current built-in example, `kronos`, maps effective source names to
its pinned marker vocabulary, records the kept/dropped physical mapping, and binds the
complete effective panel into its output fingerprint before resume. Installation and
model-specific license/access details are in [MODELS.md](MODELS.md) and
[MODEL_LICENSES.md](MODEL_LICENSES.md).

```bash
raw2features embed SLIDE.ome.zarr OUT -m kronos --mpp 0.5
```

Use `--channel-names-file` when the source lacks a complete panel, or `--no-seg` when the
entire image should be tiled without a nuclear mask.

## Extending multiplex support

The `raw2features.multiplex_strategies` entry point separates panel/config preparation
from slide-specific binding. Future marker-to-RGB mappings or learned channel adapters
can use this contract if there is demand. A new native multiplex model instead declares
`modality: multiplex` and implements the standard embedder panel-binding seam.
2 changes: 1 addition & 1 deletion docs/MODELS.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ read and decide on. raw2features makes no commercial-use determination.
| `seal_univ2` ⚠ | seal | 1536 | 224 | 0.5 | conv. | CC-BY-NC-ND-4.0 | yes | [HF](https://huggingface.co/MahmoodLab/SEAL) · [GH](https://github.com/mahmoodlab/SEAL) · [paper](https://arxiv.org/abs/2602.14177) |
| `kronos` | kronos | 384 | 224 | -¹² | n/a | CC-BY-NC-ND-4.0 | yes | [HF](https://huggingface.co/MahmoodLab/KRONOS) · [GH](https://github.com/mahmoodlab/KRONOS) · [paper](https://arxiv.org/abs/2506.03373) |

⚠ **SEAL is experimental in v0.1.1 and is outside the exact-weight pinning
⚠ **SEAL is experimental in v0.2.0 and is outside the exact-weight pinning
guarantee.** raw2features pins and SHA-256 verifies the SEAL LoRA adapter, freezes
the adapter constructor, and records its composite identity. The pinned upstream
factory still downloads the frozen CONCH/UNI2-h base from mutable HEAD, however, so
Expand Down
Loading
Loading