__`. The corresponding p-value is saved under the same column name pattern with suffix `p`
-
-```{code-cell} ipython3
-ds.RNA.feats.head()
-```
-
----
-### 4) Visualize pseudotime correlated features
-
-In this section will do deeper on how to use the pseudotime correlation values for further exploratory analysis.
-
-The first step is to export the values in a convenient dataframe format. we can use the `to_pandas_dataframe` methods of the feature attribute table to export the dataframe containing only the columns of choice
-
-```{code-cell} ipython3
-corr_genes_df = ds.RNA.feats.to_pandas_dataframe(
- columns=[
- 'names',
- 'I__RNA_pseudotime__p',
- 'I__RNA_pseudotime__r'
- ],
- key='I')
-
-# Rename the columns to be shorter
-corr_genes_df.columns = ['names', 'p_value', 'r_value']
-```
-
-Let's checkout the genes that are negatively correlated with the pseudotime. These genes increase in expression as the pseudotime progresses., i.e. as cells divide
-
-```{code-cell} ipython3
-corr_genes_df.sort_values('r_value')[:15]
-```
-
-Let's visualize the expression of some of these genes on the UMAP plot
-
-```{code-cell} ipython3
-ds.plot_layout(
- layout_key='RNA_UMAP',
- color_by=['Spp1', 'Dbi', 'Sparc'],
- width=3.5,
- height=3.5,
- point_size=5,
-)
-```
-
-Now let's checkout the genes that are most positively correlated with the pseudotime. These genes **decrease** in expression as the pseudotime progresses
-
-```{code-cell} ipython3
-corr_genes_df.sort_values('r_value', ascending=False)[:10]
-```
-
-```{code-cell} ipython3
-ds.plot_layout(
- layout_key='RNA_UMAP',
- color_by=['Aplp1', 'Gnas', 'Cpe'],
- width=3.5,
- height=3.5,
- point_size=5,
-)
-```
-
----
-### 5) Identify feature modules based on pseudotime
-
-`run_pseudotime_marker_search` is excellent to find the genes are linearly correlated with the pseudotime. This function provides us informative statistical metrics to identify genes that are most strongly correlated with the pseudotime. However, with these methods we do not recover all the dynamic patterns of expression along the pseudotime. For example, there might be certain genes that express only in the middle of the trajectory or in one branch of the trajectory.
-
-`run_pseudotime_aggregation` performs two task: 1) It arranges cells along the pseudotime and creates a smoothened, scaled and binned matrix of data 2) Clustering (KNN+Paris) is performed on this matrix to identify the groups of features/genes that have similar expression patterns along the pseudotime.
-
-```{code-cell} ipython3
-ds.run_pseudotime_aggregation(
- pseudotime_key='RNA_pseudotime',
- cluster_label='pseudotime_clusters',
- n_clusters = 15,
- window_size=200,
- chunk_size=100,
-)
-```
-
-There are two primary results of `run_pseudotime_aggregation`:
-1) The binned matrix is saved under `aggregate___`
-2) Feature clusters are saved under feature attributes table
-
-```{code-cell} ipython3
-# The binned data matrix. Here we print the shape of the matrix indicating the number of features and numner of bins respectively
-ds.z.RNA.aggregated_I_I_RNA_pseudotime.data.shape
-```
-
-```{code-cell} ipython3
-# Fetching pseudotime based cluster identity of features
-ds.RNA.feats.fetch('pseudotime_clusters')
-```
-
-`plot_pseudotime_heatmap` allows visualizing the binned matrix along with the feature clusters
-
-```{code-cell} ipython3
-# Highlighting some marker genes
-genes_to_label = ['S100b', 'Nrarp', 'Atoh8', 'Grin2c', 'Slc35d3',
- 'Sst', 'Mnx1', 'Ins2', 'Gm11837', 'Irx1']
-
-ds.plot_pseudotime_heatmap(
- cell_key='I',
- feat_key='I',
- feature_cluster_key='pseudotime_clusters',
- pseudotime_key='RNA_pseudotime',
- show_features=genes_to_label
-)
-```
-
-The heatmap above shows the gene expression dynamics as the cells progress throught the pseudotime. Here, cluster 1 captures the genes that have highest expression in early pseudotime while cluster 15 captures genes whose expression peak in the late pseudotime.
-
-We can visualize the expression of the above selected genes on UMAP to check whether their cluster identity corroborates their expression pattern.
-
-```{code-cell} ipython3
-ds.plot_layout(
- layout_key='RNA_UMAP',
- color_by=genes_to_label,
- width=3,
- height=3,
- point_size=5,
- n_columns=5,
-)
-```
-
----
-### 6) Merging pseudotime-based feature modules into a new assay
-
-The pseudotime based clusters of features can be used create a new assay. `add_grouped_assay` will take each cluster and take the mean expression of genes from that cluster and add it to a new assay. The motivation behind this approach is that we do not have to add many columns to our cell metadata table and have the mean cluster values readily available for analysis.
-
-Taking mean cluster values is a powerful approach that allows use to explore cumulative pattern of highly correlated genes. Here we create a new assay under title `PTIME_MODULES`
-
-```{code-cell} ipython3
-ds.add_grouped_assay(
- group_key='pseudotime_clusters',
- assay_label='PTIME_MODULES'
-)
-```
-
-```{code-cell} ipython3
-# DataStore summary showing `PTIME_MODULES` assay with 15 features (number of pseudotime based feature clusters)
-ds
-```
-
-The mean values from each cluster are saved within the assay and tagged with names like `group_1`, `group_2`, etc
-
-```{code-cell} ipython3
-ds.PTIME_MODULES.feats.head()
-```
-
-We can visualize these cluster mean values directly on the UMAP like this:
-
-```{code-cell} ipython3
-n_clusters = 15
-ds.plot_layout(
- from_assay='PTIME_MODULES',
- layout_key='RNA_UMAP',
- color_by=[f"group_{i}" for i in range(1,n_clusters+1)],
- width=3,
- height=3,
- point_size=5,
- n_columns=5,
- cmap='coolwarm',
-)
-```
-
-This figure complements the heatmap we generated earlier very nicely. Using this approach we have clearly found **gene modules** that are restricted in expression to certain portion of the pseudotime and differentiation trajectory
-
-+++
-
----
-### 7) Comparing pseudotime based feature modules with cluster markers
-
-Here we will compare the pseudotime based feature module extraction approach with classical cluster marker approach.
-
-```{code-cell} ipython3
-# Running marker search
-ds.run_marker_search(group_key='RNA_cluster')
-```
-
-Here we extract features from pseudotime-based cluster/group 13. These genes are the ones that show high expressio in Beta cells.
-
-```{code-cell} ipython3
-ptime_feat_clusts = ds.RNA.feats.to_pandas_dataframe(
- columns=['names', 'pseudotime_clusters']
-)
-ptime_based_markers = ptime_feat_clusts.names[ptime_feat_clusts.pseudotime_clusters == 13]
-ptime_based_markers.head()
-```
-
-Now we extract all the marker genes for cell cluster 8, this cluster predominantly contains the Beta cells.
-
-```{code-cell} ipython3
-cell_cluster_markers = ds.get_markers(
- group_key='RNA_cluster',
- group_id='8'
-).feature_name
-
-cell_cluster_markers.head()
-```
-
-```{code-cell} ipython3
-# let's checkout the number of Beta cell associated genes from both methods
-ptime_based_markers.shape, cell_cluster_markers.shape
-```
-
-```{code-cell} ipython3
-# let's checkout the number of common Beta cell associated genes from both methods
-len(set(cell_cluster_markers.index).intersection(ptime_based_markers.index))
-```
-
-Let's visualize the cumulative expression of genes that are present only in cluster marker based approach
-
-```{code-cell} ipython3
-temp = list(set(cell_cluster_markers.index).difference(ptime_based_markers.index))
-ds.cells.insert(
- column_name='Cell cluster based markers',
- values=ds.RNA.normed(feat_idx=sorted(temp)).mean(axis=1).compute(),
- overwrite=True)
-
-ds.plot_layout(
- layout_key='RNA_UMAP',
- color_by='Cell cluster based markers',
- width=4, height=4, point_size=5, n_columns=5, cmap='coolwarm',
-)
-```
-
-Let's now do this the other way and visualize the cumulative expression of genes that are present only in pseudotime-based approach
-
-```{code-cell} ipython3
-temp = list(set(ptime_based_markers.index).difference(cell_cluster_markers.index))
-ds.cells.insert(
- column_name='Pseudotime based markers',
- values=ds.RNA.normed(feat_idx=sorted(temp)).mean(axis=1).compute(),
- overwrite=True)
-
-ds.plot_layout(
- layout_key='RNA_UMAP',
- color_by='Pseudotime based markers',
- width=4, height=4, point_size=5, n_columns=5, cmap='coolwarm',
-)
-```
-
-The pseudotime-based approach clearly captures a lot of signal that would be otherwise missed by simply taking a cell cluster marker based approach.
-
----
-That is all for this vignette.
diff --git a/docs/source/vignettes/zarr_explanation.md b/docs/source/vignettes/zarr_explanation.md
deleted file mode 100644
index 62240877..00000000
--- a/docs/source/vignettes/zarr_explanation.md
+++ /dev/null
@@ -1,297 +0,0 @@
----
-jupytext:
- text_representation:
- extension: .md
- format_name: myst
- format_version: 0.13
- jupytext_version: 1.14.1
-kernelspec:
- display_name: Python 3 (ipykernel)
- language: python
- name: python3
----
-
-(data_organization)=
-
-+++
-
-## Understanding how data is organized in Scarf
-
-In this notebook, we provide a more detailed exploration of how the data is organized in Scarf. This can be useful for users who want to customize certain aspects of Scarf or want to extend its functionality.
-
-The following figure shows how different Class structures are connected to each other and the methods they contain. The `DataStore` class is the primary class that allows users to access all the data and functions. In almost all scenarios the user has to directly interact only with the object of `DataStore` class's methods. `DataStore` object is linked to the underlying `Zarr` file hierarchy on the disk. We provide more details in the later sections.
-
-The individual assays (in non mulit-omics datasets there will be only one assay) are represented as objects of the `Assay` class. A subclass of `Assay` class is automatically chosen based on the type of data. For example, `RNAassay` is chosen for single-cell RNA-Seq while `ATACassay`subclass is chosen for scATAC-Seq. Each `Assay` subclass contains its own normalization and feature selection methods.
-
-The `Metadata` class acts like a DataFrame (it is not actually a Dataframe but collections of columns/arrays that are loaded from disk on demand) that contains information on either individual cells or features. The cell metadata can directly be accessed from `DataStore` but feature metdata has to be accessed by the `Assay` object (we show examples below).
-
-The `DataStore` class contains mutiple methods that allow generation of the KNN-graph that can be loaded to perform downsampling (sketching), clustering (Paris/Leiden) and UMAP/tSNE embedding (please see the basic workflows for usage of these methods).
-
-+++
-
-.r2.cloudflarestorage.com
+R2_ACCESS_KEY_ID=key
+R2_SECRET_ACCESS_KEY=secret
diff --git a/profiling/AGENTS.md b/profiling/AGENTS.md
new file mode 100644
index 00000000..6932c001
--- /dev/null
+++ b/profiling/AGENTS.md
@@ -0,0 +1,75 @@
+# Profiling instructions
+
+These instructions apply to local profiling diagnostics and the Modal end-to-end profiler under
+`profiling/`.
+
+## Before cloud work
+
+- Read `profiling/BENCHMARKS.md` for the current reference measurements and their interpretation
+ limits.
+- Run local profiling tests before using cloud resources:
+
+```bash
+uv run pytest -n 0 tests/test_profiling_*.py
+```
+
+- Copy `profiling/config.example.toml` to the ignored `profiling/config.toml`. Do not commit
+ machine-specific configs, endpoints, bucket names, secrets, or result identifiers.
+- Set a fresh, non-empty `runTag` for every end-to-end measurement.
+- Ask before starting a paid or long-running Modal job.
+
+## Deployment boundary
+
+Never run `modal deploy`. Deployment is a user action. Ask the user to run:
+
+```bash
+uv run --group profiling modal deploy --env scarf_profiling \
+ -m profiling.modal_app
+```
+
+Do not create Modal environments, secrets, or credentials.
+
+## Standard workflow
+
+After the user confirms that the current app is deployed, prepare deterministic CELLxGENE samples
+and confirm the requested result before starting the funnel:
+
+```bash
+uv run --group profiling modal run --env scarf_profiling \
+ -m profiling.modal_app -- prepare \
+ --config profiling/config.toml
+
+uv run --group profiling modal run --env scarf_profiling \
+ -m profiling.modal_app -- run-e2e \
+ --config profiling/config.toml --size 1000000
+```
+
+`prepare` spawns work and returns immediately. Confirm that its requested H5AD exists before
+`run-e2e`.
+
+Use `run --stage ...` for a targeted stage, including repair of an incomplete `countsT`; use
+`run-local` for the Modal ephemeral-disk comparison and `io-baseline` for read-pattern diagnostics.
+
+## Durable execution
+
+- Long work must use `.spawn(...)`, never `.remote()`. A blocking call can be cancelled when the
+ local gRPC session disconnects.
+- Wait through short `FunctionCall.get(timeout=...)` polls or durable result JSON instead of one
+ long blocking request.
+- Treat `FAILURE`, `INIT_FAILURE`, `TERMINATED`, and `TIMEOUT` as terminal.
+ Modal can raise an empty `TimeoutError` for a failed input.
+- Give coordinators about 1 CPU and 2 to 4 GiB with `retries=0`. Do not assign stage-worker
+ resources to coordinators.
+- Prefer the broad `eu` region and leave the Modal cloud option unset unless the experiment
+ explicitly measures another placement.
+- Log start, plan, periodic progress, and completion lines.
+- Persist each stage result and `funnel.json` before treating a run as complete.
+
+## Measurement discipline
+
+- Never run two jobs with the same `runTag`.
+- Change one measured variable at a time and keep workflow seeds fixed.
+- Compare runs only when dataset, code revision, settings, storage conditions, and resource
+ envelope are stated.
+- Do not present measurements from different machine sizes as one scaling curve.
+- Do not generalize one run into a hardware guarantee or a biological correctness claim.
diff --git a/profiling/BENCHMARKS.md b/profiling/BENCHMARKS.md
new file mode 100644
index 00000000..5c3c84d9
--- /dev/null
+++ b/profiling/BENCHMARKS.md
@@ -0,0 +1,134 @@
+# Scarf profiling references
+
+This file records the latest completed full-scale Scarf measurements. It does
+not preserve superseded experiments or operational call identifiers.
+
+## Scope
+
+The measurements were completed on 2026-07-29 and 2026-07-30 from commit
+`ce50bbced52e3f8177ebfb492a04ef474dff6918`. Both runs downloaded an H5AD,
+used a fresh object-store-backed Zarr v3 store, and completed the same
+sixteen-stage CPU workflow:
+
+1. cell-major count conversion
+2. `countsT` construction
+3. datastore initialization and quality-control metrics
+4. datastore reopen
+5. cell filtering
+6. highly variable feature selection
+7. normalization
+8. PCA
+9. embedding initialization
+10. approximate-neighbour index construction
+11. neighbour queries
+12. connectivity-map construction
+13. UMAP
+14. Leiden clustering
+15. Paris clustering
+16. marker search using the Leiden groups
+
+The source was the public CELLxGENE H5AD with dataset ID
+`dcfd4feb-18a3-4b30-81d7-1b0c544a8ab3` and version ID
+`1bc30289-9565-4099-abf9-3326328c11ac`. The profiler selected deterministic,
+nested samples with seed 0. The prepared inputs contained exactly 1,000,000 and
+10,000,000 cells.
+
+## Settings
+
+These are the recorded measurement settings. `config.example.toml` is an
+evolving operational template and is not an exact reproduction of these runs.
+
+- 2,000 highly variable features and 50 PCA dimensions
+- 11 neighbours and 1,000 embedding centroids
+- graph seed 4466
+- UMAP and Leiden seed 4444
+- 300 UMAP epochs and Leiden resolution 1.0
+- parallel ANN and UMAP enabled
+- embedding-initialization sample fraction 0.1 and minibatch size 10,000
+- 1st and 99th percentile cell filtering
+- minimum 10 features per cell and 20 cells per feature
+- 1,000-cell H5AD conversion batches
+- S3-compatible object storage in the Modal EU region
+
+The Scarf memory budget controls planned block sizes and concurrency. It is not
+a hard process-memory limit.
+
+## End-to-end results
+
+| Input cells | Cells after QC | CPU | Container memory | Scarf budget | Wall time | Peak cgroup memory |
+| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 1,000,000 | 889,974 | 8 | 32 GiB | 24 GiB | 2,458.5 s (41.0 min) | 28.5 GiB |
+| 10,000,000 | 8,902,268 | 16 | 128 GiB | 96 GiB | 37,118.9 s (10.31 h) | 105.0 GiB |
+
+The source H5AD files were 3.748 GiB and 37.434 GiB. The ten-million-cell run
+used twice the CPUs and four times the container and software memory, so these
+rows are not a same-machine scaling curve.
+
+Stage values below use `wholeFunctionSeconds`, which includes recorded stage
+setup and validation. The funnel total also includes the initial download and
+about six seconds of orchestration.
+
+| Stage | 1M seconds | 10M seconds | 10M / 1M |
+| --- | ---: | ---: | ---: |
+| Dataset download | 90.1 | 2,992.2 | 33.2 |
+| Create count store | 404.6 | 7,316.0 | 18.1 |
+| Write `countsT` | 180.1 | 2,768.7 | 15.4 |
+| Initialize datastore | 101.1 | 1,842.5 | 18.2 |
+| Reopen datastore | 8.1 | 10.4 | 1.3 |
+| Filter cells | 77.5 | 124.4 | 1.6 |
+| Mark HVGs | 172.2 | 3,444.5 | 20.0 |
+| Normalize | 168.3 | 2,201.2 | 13.1 |
+| PCA | 86.1 | 1,135.9 | 13.2 |
+| Build embedding initialization | 33.8 | 341.8 | 10.1 |
+| Build ANN index | 95.7 | 1,060.5 | 11.1 |
+| Query neighbours | 51.9 | 311.7 | 6.0 |
+| Build connectivity map | 57.1 | 87.7 | 1.5 |
+| UMAP | 314.6 | 2,051.6 | 6.5 |
+| Leiden | 269.3 | 3,446.5 | 12.8 |
+| Paris | 114.0 | 793.3 | 7.0 |
+| Marker search | 228.2 | 7,183.1 | 31.5 |
+| **Funnel total** | **2,458.5** | **37,118.9** | **15.1** |
+
+## Interpretation
+
+Each row is one completed run, not a median of repeated runs. Peak cgroup values
+were sampled from `memory.current`, so very short spikes may be missed. The
+profiler did not force garbage collection between stages.
+
+Marker search set the peak-memory value in both runs. At ten million cells, its
+block reads took 6,481.4 seconds and its rank computation took 613.0 seconds.
+The planned feature blocks consumed enough of the 96 GiB software budget that
+inner read concurrency fell to one. The measured 105.0 GiB peak is therefore a
+throughput choice, not a fixed memory requirement for ten million cells.
+
+Storage throughput also varied. The input size grew by 9.99 times, while download wall time grew by
+33.2 times because average throughput fell from 42.6 MiB/s to 12.8 MiB/s. Excluding download, total
+wall time scaled by 14.4 times.
+
+These results establish execution and resource use for the recorded dataset,
+code revision, settings, and cloud conditions. They do not establish biological
+correctness, a hardware guarantee, or superiority over another package.
+
+## Running the profiler
+
+Copy `profiling/config.example.toml` to the ignored
+`profiling/config.toml`, configure your own object store and Modal secret, and
+set a fresh `runTag`. Deployment is a user action. After the app is deployed,
+prepare the deterministic samples, then spawn the current funnel:
+
+```bash
+uv run --group profiling modal run --env scarf_profiling \
+ -m profiling.modal_app -- prepare \
+ --config profiling/config.toml
+
+uv run --group profiling modal run --env scarf_profiling \
+ -m profiling.modal_app -- run-e2e \
+ --config profiling/config.toml --size 1000000
+```
+
+The prepare command spawns work and returns immediately. Confirm that the
+requested H5AD exists before starting `run-e2e`.
+
+Each stage writes a result JSON, and the complete invocation writes
+`funnel.json`. Compare runs only when their dataset, code revision, workflow
+settings, storage conditions, and resource envelope are stated.
diff --git a/profiling/__init__.py b/profiling/__init__.py
new file mode 100644
index 00000000..ad95557b
--- /dev/null
+++ b/profiling/__init__.py
@@ -0,0 +1 @@
+"""Scarf Modal/R2 profiling harness."""
diff --git a/profiling/config.example.toml b/profiling/config.example.toml
new file mode 100644
index 00000000..a7eba17b
--- /dev/null
+++ b/profiling/config.example.toml
@@ -0,0 +1,232 @@
+# Copy to profiling/config.toml and replace secret, region, endpoint, bucket, and prefixes.
+
+modalEnvironmentName = "scarf_profiling"
+modalAppName = "scarf-profiling"
+modalSecretName = "replace-with-modal-secret"
+modalRegion = "eu"
+r2EndpointUrl = "https://replace-with-account-id.r2.cloudflarestorage.com"
+datasetPrefixUri = "s3://example-bucket/scarf-profiling/datasets"
+resultsUri = "s3://example-bucket/scarf-profiling"
+# Optional: isolate stores/results under stores// and results//.
+# runTag = "chunk256m"
+# [storageLayout]
+# targetChunkBytes = 268435456
+# targetShardBytes = 805306368
+targetSizes = [
+ 10000,
+ 25000,
+ 50000,
+ 100000,
+ 250000,
+ 500000,
+ 1000000,
+ 2500000,
+ 5000000,
+ 10000000,
+]
+samplingSeed = 0
+
+# Full Cellxgene prepare loads CSR into RAM (indices downcast to int32).
+[prepareResources]
+modalMemoryRequestMb = 196608
+modalMemoryLimitMb = 212992
+modalCpuRequest = 8.0
+modalCpuLimit = 16.0
+timeoutSeconds = 86400
+ephemeralDiskMb = 524288
+
+[workflow]
+assayName = "RNA"
+cellKey = "I"
+filterAttrs = [
+ "RNA_nCounts",
+ "RNA_nFeatures",
+ "RNA_percentMito",
+ "RNA_percentRibo",
+]
+filterMinQuantile = 0.01
+filterMaxQuantile = 0.99
+minFeaturesPerCell = 10
+minCellsPerFeature = 20
+h5adBatchSize = 1000
+topN = 2000
+hvgMinCells = 20
+hvgKey = "hvgs"
+k = 17
+dims = 50
+nCentroids = 1000
+graphSeed = 4466
+annParallel = false
+umapEpochs = 300
+umapSeed = 4444
+umapParallel = false
+umapLabel = "UMAP"
+leidenResolution = 1.0
+leidenSeed = 4444
+leidenLabel = "leiden_cluster"
+markerFeatureKey = "I"
+# When omitted or null, Scarf chooses a budget-safe batch from on-disk feature chunks.
+# markerGeneBatchSize = 50
+graphLocalCache = "auto"
+
+# Tune memory before large runs. Values are examples.
+
+[stageResources.createStore]
+modalMemoryRequestMb = 65536
+modalMemoryLimitMb = 73728
+modalCpuRequest = 8.0
+modalCpuLimit = 8.0
+scarfMemoryBudget = 51539607552
+workers = 8
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.writeCountsT]
+modalMemoryRequestMb = 65536
+modalMemoryLimitMb = 73728
+modalCpuRequest = 8.0
+modalCpuLimit = 8.0
+scarfMemoryBudget = 51539607552
+workers = 8
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.initializeStore]
+modalMemoryRequestMb = 65536
+modalMemoryLimitMb = 73728
+modalCpuRequest = 8.0
+modalCpuLimit = 8.0
+scarfMemoryBudget = 51539607552
+workers = 8
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.reopenStore]
+modalMemoryRequestMb = 16384
+modalMemoryLimitMb = 20480
+modalCpuRequest = 4.0
+modalCpuLimit = 4.0
+scarfMemoryBudget = 12884901888
+workers = 4
+timeoutSeconds = 3600
+ephemeralDiskMb = 524288
+
+[stageResources.filterCells]
+modalMemoryRequestMb = 32768
+modalMemoryLimitMb = 40960
+modalCpuRequest = 8.0
+modalCpuLimit = 8.0
+scarfMemoryBudget = 25769803776
+workers = 8
+timeoutSeconds = 14400
+ephemeralDiskMb = 524288
+
+[stageResources.markHvgs]
+modalMemoryRequestMb = 65536
+modalMemoryLimitMb = 73728
+modalCpuRequest = 8.0
+modalCpuLimit = 8.0
+scarfMemoryBudget = 51539607552
+workers = 8
+timeoutSeconds = 43200
+ephemeralDiskMb = 524288
+
+[stageResources.runNormalization]
+modalMemoryRequestMb = 131072
+modalMemoryLimitMb = 147456
+modalCpuRequest = 16.0
+modalCpuLimit = 16.0
+scarfMemoryBudget = 120259084288
+workers = 16
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.runPca]
+modalMemoryRequestMb = 131072
+modalMemoryLimitMb = 147456
+modalCpuRequest = 16.0
+modalCpuLimit = 16.0
+scarfMemoryBudget = 120259084288
+workers = 16
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.buildEmbeddingInitialization]
+modalMemoryRequestMb = 131072
+modalMemoryLimitMb = 147456
+modalCpuRequest = 16.0
+modalCpuLimit = 16.0
+scarfMemoryBudget = 120259084288
+workers = 16
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.buildAnnIndex]
+modalMemoryRequestMb = 131072
+modalMemoryLimitMb = 147456
+modalCpuRequest = 16.0
+modalCpuLimit = 16.0
+scarfMemoryBudget = 120259084288
+workers = 16
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.queryNeighbors]
+modalMemoryRequestMb = 131072
+modalMemoryLimitMb = 147456
+modalCpuRequest = 16.0
+modalCpuLimit = 16.0
+scarfMemoryBudget = 120259084288
+workers = 16
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.buildConnectivityMap]
+modalMemoryRequestMb = 131072
+modalMemoryLimitMb = 147456
+modalCpuRequest = 16.0
+modalCpuLimit = 16.0
+scarfMemoryBudget = 120259084288
+workers = 16
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.runUmap]
+modalMemoryRequestMb = 65536
+modalMemoryLimitMb = 73728
+modalCpuRequest = 8.0
+modalCpuLimit = 8.0
+scarfMemoryBudget = 51539607552
+workers = 8
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.runLeiden]
+modalMemoryRequestMb = 32768
+modalMemoryLimitMb = 32768
+modalCpuRequest = 2.0
+modalCpuLimit = 2.0
+scarfMemoryBudget = 25769803776
+workers = 2
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.runClustering]
+modalMemoryRequestMb = 16384
+modalMemoryLimitMb = 16384
+modalCpuRequest = 2.0
+modalCpuLimit = 2.0
+scarfMemoryBudget = 12884901888
+workers = 2
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
+
+[stageResources.findMarkers]
+modalMemoryRequestMb = 131072
+modalMemoryLimitMb = 147456
+modalCpuRequest = 16.0
+modalCpuLimit = 16.0
+scarfMemoryBudget = 120259084288
+workers = 16
+timeoutSeconds = 82800
+ephemeralDiskMb = 524288
diff --git a/profiling/config.py b/profiling/config.py
new file mode 100644
index 00000000..615bb0d7
--- /dev/null
+++ b/profiling/config.py
@@ -0,0 +1,292 @@
+import math
+import tomllib
+from pathlib import Path
+from typing import Any, Literal, Self
+
+from pydantic import BaseModel, ConfigDict, Field, model_validator
+
+from profiling.datasets import DEFAULT_TARGET_SIZES
+
+StageName = Literal[
+ "createStore",
+ "writeCountsT",
+ "initializeStore",
+ "reopenStore",
+ "filterCells",
+ "markHvgs",
+ "runNormalization",
+ "runPca",
+ "buildEmbeddingInitialization",
+ "buildAnnIndex",
+ "queryNeighbors",
+ "buildConnectivityMap",
+ "runUmap",
+ "runLeiden",
+ "runClustering",
+ "findMarkers",
+]
+
+GRAPH_CONSTRUCTION_STAGE_ORDER: tuple[StageName, ...] = (
+ "runNormalization",
+ "runPca",
+ "buildEmbeddingInitialization",
+ "buildAnnIndex",
+ "queryNeighbors",
+ "buildConnectivityMap",
+)
+
+CORE_STAGE_ORDER: tuple[StageName, ...] = (
+ "createStore",
+ "writeCountsT",
+ "initializeStore",
+ "reopenStore",
+ "filterCells",
+ "markHvgs",
+ *GRAPH_CONSTRUCTION_STAGE_ORDER,
+ "runUmap",
+ "runLeiden",
+ "runClustering",
+ "findMarkers",
+)
+
+MAX_TIMEOUT_SECONDS = 86_400
+
+
+class WorkflowParameters(BaseModel):
+ model_config = ConfigDict(extra="forbid", frozen=True)
+
+ assayName: str = "RNA"
+ cellKey: str = "I"
+ filterAttrs: tuple[str, ...] = (
+ "RNA_nCounts",
+ "RNA_nFeatures",
+ "RNA_percentMito",
+ "RNA_percentRibo",
+ )
+ filterMinQuantile: float = 0.01
+ filterMaxQuantile: float = 0.99
+ minFeaturesPerCell: int = 10
+ minCellsPerFeature: int = 20
+ h5adBatchSize: int = 1000
+ topN: int = 2000
+ hvgMinCells: int = 20
+ hvgKey: str = "hvgs"
+ k: int = 17
+ dims: int = 50
+ nCentroids: int = 1000
+ graphSeed: int = 4466
+ kmeansSampling: float = 0.1
+ kmeansBatchSize: int = 10_000
+ annParallel: bool = False
+ umapEpochs: int = 300
+ umapSeed: int = 4444
+ umapParallel: bool = False
+ umapLabel: str = "UMAP"
+ leidenResolution: float = 1.0
+ leidenSeed: int = 4444
+ leidenLabel: str = "leiden_cluster"
+ markerFeatureKey: str = "I"
+ markerGeneBatchSize: int | None = None
+ countsTLayout: Literal["source", "featureMajor"] = "source"
+ graphLocalCache: bool | str = "auto"
+ parisNClusters: int | Literal["auto"] = "auto"
+ parisLabel: str = "paris_cluster"
+ parisMinClusterSize: int | None = None
+
+ @property
+ def resolvedHvgKey(self) -> str:
+ return f"{self.cellKey}__{self.hvgKey}"
+
+ @property
+ def resolvedMarkerGroupKey(self) -> str:
+ if self.cellKey == "I":
+ return f"{self.assayName}_{self.leidenLabel}"
+ return f"{self.assayName}_{self.cellKey}_{self.leidenLabel}"
+
+ @model_validator(mode="after")
+ def _check_workflow(self) -> Self:
+ if not math.isfinite(self.kmeansSampling) or not 0 < self.kmeansSampling <= 1:
+ raise ValueError("kmeansSampling must be greater than 0 and at most 1")
+ if self.kmeansBatchSize <= 0:
+ raise ValueError("kmeansBatchSize must be positive")
+ if self.parisNClusters != "auto" and self.parisNClusters <= 1:
+ raise ValueError("parisNClusters must be > 1")
+ if self.parisMinClusterSize is not None:
+ if self.parisMinClusterSize < 2:
+ raise ValueError("parisMinClusterSize must be >= 2")
+ if self.parisNClusters != "auto":
+ raise ValueError("parisMinClusterSize requires parisNClusters='auto'")
+ return self
+
+
+class StageResources(BaseModel):
+ model_config = ConfigDict(extra="forbid", frozen=True)
+
+ modalMemoryRequestMb: int
+ modalMemoryLimitMb: int
+ modalCpuRequest: float
+ modalCpuLimit: float
+ scarfMemoryBudget: int
+ workers: int
+ timeoutSeconds: int
+ ephemeralDiskMb: int
+
+ @model_validator(mode="after")
+ def _check_bounds(self) -> Self:
+ if self.modalMemoryRequestMb <= 0 or self.modalMemoryLimitMb <= 0:
+ raise ValueError("Modal memory values must be positive")
+ if self.modalMemoryLimitMb < self.modalMemoryRequestMb:
+ raise ValueError("modalMemoryLimitMb must be >= modalMemoryRequestMb")
+ if self.modalCpuRequest <= 0 or self.modalCpuLimit <= 0:
+ raise ValueError("Modal CPU values must be positive")
+ if self.modalCpuLimit < self.modalCpuRequest:
+ raise ValueError("modalCpuLimit must be >= modalCpuRequest")
+ if self.scarfMemoryBudget <= 0:
+ raise ValueError("scarfMemoryBudget must be positive")
+ if self.workers <= 0:
+ raise ValueError("workers must be positive")
+ if not (0 < self.timeoutSeconds <= MAX_TIMEOUT_SECONDS):
+ raise ValueError(f"timeoutSeconds must be in 1..{MAX_TIMEOUT_SECONDS}")
+ if self.ephemeralDiskMb <= 0:
+ raise ValueError("ephemeralDiskMb must be positive")
+ return self
+
+
+class PrepareResources(BaseModel):
+ """Modal sizing for dataset prepare (full CSR resident in memory)."""
+
+ model_config = ConfigDict(extra="forbid", frozen=True)
+
+ modalMemoryRequestMb: int = 196_608
+ modalMemoryLimitMb: int = 212_992
+ modalCpuRequest: float = 8.0
+ modalCpuLimit: float = 16.0
+ timeoutSeconds: int = 86_400
+ ephemeralDiskMb: int = 524_288
+
+ @model_validator(mode="after")
+ def _check_bounds(self) -> Self:
+ if self.modalMemoryRequestMb <= 0 or self.modalMemoryLimitMb <= 0:
+ raise ValueError("Modal memory values must be positive")
+ if self.modalMemoryLimitMb < self.modalMemoryRequestMb:
+ raise ValueError("modalMemoryLimitMb must be >= modalMemoryRequestMb")
+ if self.modalCpuRequest <= 0 or self.modalCpuLimit <= 0:
+ raise ValueError("Modal CPU values must be positive")
+ if self.modalCpuLimit < self.modalCpuRequest:
+ raise ValueError("modalCpuLimit must be >= modalCpuRequest")
+ if not (0 < self.timeoutSeconds <= MAX_TIMEOUT_SECONDS):
+ raise ValueError(f"timeoutSeconds must be in 1..{MAX_TIMEOUT_SECONDS}")
+ if self.ephemeralDiskMb <= 0:
+ raise ValueError("ephemeralDiskMb must be positive")
+ return self
+
+
+class StorageLayout(BaseModel):
+ model_config = ConfigDict(extra="forbid", frozen=True)
+
+ targetChunkBytes: int | None = None
+ targetShardBytes: int | None = None
+
+ @model_validator(mode="after")
+ def _check_bounds(self) -> Self:
+ if self.targetChunkBytes is not None and self.targetChunkBytes <= 0:
+ raise ValueError("targetChunkBytes must be positive when set")
+ if self.targetShardBytes is not None and self.targetShardBytes <= 0:
+ raise ValueError("targetShardBytes must be positive when set")
+ return self
+
+
+class ProfilingConfig(BaseModel):
+ model_config = ConfigDict(extra="forbid", frozen=True)
+
+ modalEnvironmentName: str = "scarf_profiling"
+ modalAppName: str = "scarf-profiling"
+ modalSecretName: str
+ modalRegion: str
+ r2EndpointUrl: str
+ datasetPrefixUri: str
+ resultsUri: str
+ runTag: str = ""
+ storageLayout: StorageLayout = Field(default_factory=StorageLayout)
+ targetSizes: tuple[int, ...] = Field(default_factory=lambda: DEFAULT_TARGET_SIZES)
+ samplingSeed: int = 0
+ workflow: WorkflowParameters = Field(default_factory=WorkflowParameters)
+ prepareResources: PrepareResources = Field(default_factory=PrepareResources)
+ stageResources: dict[StageName, StageResources]
+ # If unset, the complete current funnel runs.
+ stages: tuple[StageName, ...] | None = None
+
+ @property
+ def effectiveStages(self) -> tuple[StageName, ...]:
+ return self.stages if self.stages is not None else CORE_STAGE_ORDER
+
+ @model_validator(mode="after")
+ def _check_config(self) -> Self:
+ if self.modalEnvironmentName != "scarf_profiling":
+ raise ValueError("modalEnvironmentName must be scarf_profiling")
+ if not self.datasetPrefixUri.startswith("s3://"):
+ raise ValueError("datasetPrefixUri must be an s3:// URI")
+ if not self.resultsUri.startswith("s3://"):
+ raise ValueError("resultsUri must be an s3:// URI")
+ if "/" in self.runTag or "\\" in self.runTag or self.runTag in {".", ".."}:
+ raise ValueError("runTag must be a single path segment")
+ if not self.targetSizes:
+ raise ValueError("targetSizes must not be empty")
+ if any(size <= 0 for size in self.targetSizes):
+ raise ValueError("targetSizes must be positive")
+ if tuple(sorted(self.targetSizes)) != self.targetSizes:
+ raise ValueError("targetSizes must be strictly increasing")
+ if len(set(self.targetSizes)) != len(self.targetSizes):
+ raise ValueError("targetSizes must be unique")
+ selected = self.effectiveStages
+ if not selected:
+ raise ValueError("stages must not be empty")
+ if len(set(selected)) != len(selected):
+ raise ValueError("stages must be unique")
+ missing = [stage for stage in selected if stage not in self.stageResources]
+ if missing:
+ raise ValueError(f"Missing stageResources for: {', '.join(missing)}")
+ return self
+
+ def datasetUri(self, nRows: int) -> str:
+ return f"{self.datasetPrefixUri.rstrip('/')}/{nRows}.h5ad"
+
+ def sourceUri(self) -> str:
+ return f"{self.datasetPrefixUri.rstrip('/')}/source.h5ad"
+
+ def _tagged_prefix(self, kind: str) -> str:
+ base = f"{self.resultsUri.rstrip('/')}/{kind}"
+ if self.runTag:
+ return f"{base}/{self.runTag}"
+ return base
+
+ def storeUri(self, nRows: int) -> str:
+ return f"{self._tagged_prefix('stores')}/{nRows}.zarr"
+
+ def resultUri(self, nRows: int, stage: StageName) -> str:
+ return f"{self._tagged_prefix('results')}/{nRows}/{stage}.json"
+
+ def funnelResultUri(self, nRows: int) -> str:
+ return f"{self._tagged_prefix('results')}/{nRows}/funnel.json"
+
+ def e2eClaimUri(self) -> str:
+ return f"{self._tagged_prefix('results')}/e2e-claim.json"
+
+ def resourcesFor(self, stage: StageName) -> StageResources:
+ return self.stageResources[stage]
+
+
+def load_profiling_config(path: str | Path) -> ProfilingConfig:
+ config_path = Path(path)
+ raw = tomllib.loads(config_path.read_text())
+ return ProfilingConfig.model_validate(_normalize_raw_config(raw))
+
+
+def _normalize_raw_config(raw: dict[str, Any]) -> dict[str, Any]:
+ payload = dict(raw)
+ fixed = payload.pop("fixedResources", None)
+ if fixed is not None and "stageResources" not in payload:
+ stages = payload.get("stages")
+ selected = tuple(stages) if stages is not None else CORE_STAGE_ORDER
+ payload["stageResources"] = {stage: fixed for stage in selected}
+ return payload
diff --git a/profiling/datasets.py b/profiling/datasets.py
new file mode 100644
index 00000000..9ef45d5a
--- /dev/null
+++ b/profiling/datasets.py
@@ -0,0 +1,1660 @@
+import hashlib
+import os
+import uuid
+from collections.abc import Callable, Sequence
+from contextlib import closing
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+from urllib.request import urlopen
+
+import h5py
+import numpy as np
+
+DEFAULT_SAMPLING_SEED = 0
+DEFAULT_TARGET_SIZES = (
+ 10_000,
+ 25_000,
+ 50_000,
+ 100_000,
+ 250_000,
+ 500_000,
+ 1_000_000,
+ 2_500_000,
+ 5_000_000,
+ 10_000_000,
+)
+
+_MASK_64 = (1 << 64) - 1
+_SPLITMIX_INCREMENT = 0x9E3779B97F4A7C15
+_SPLITMIX_MULTIPLIER_1 = 0xBF58476D1CE4E5B9
+_SPLITMIX_MULTIPLIER_2 = 0x94D049BB133111EB
+_SAMPLING_DOMAIN = b"scarf-cellxgene-row-sampling-v1"
+_SOURCE_ROWS_DIGEST_DOMAIN = b"scarf-ordered-source-rows-v1\0"
+_DEFAULT_IO_CHUNK_BYTES = 16 * 1024 * 1024
+_DEFAULT_ROW_BATCH_SIZE = 1024
+_DEFAULT_COPY_BUFFER_BYTES = 64 * 1024 * 1024
+_DEFAULT_INDPTR_CHUNK_ROWS = 1_000_000
+_DEFAULT_LOAD_CHUNK_ELEMENTS = 8_388_608
+_H5_DATA_CHUNK_BYTES = 4 * 1024 * 1024
+
+
+def sha256_file(path: str | Path, *, chunkBytes: int = 16 * 1024 * 1024) -> str:
+ if chunkBytes <= 0:
+ raise ValueError("chunkBytes must be positive")
+ digest = hashlib.sha256()
+ with Path(path).open("rb") as handle:
+ while True:
+ chunk = handle.read(chunkBytes)
+ if not chunk:
+ break
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+@dataclass(frozen=True, slots=True)
+class SourceSpec:
+ datasetId: str
+ versionId: str
+ url: str
+ nRows: int
+ nColumns: int
+ nnz: int
+ sourceBytes: int
+ matrixKey: str = "X"
+ matrixEncoding: str = "csr_matrix"
+ rawCounts: bool = True
+ cellIdsKey: str = "obs/_index"
+ featureIdsKey: str = "var/_index"
+ featureNameKey: str = "var/feature_name"
+ dataDtype: str = "float32"
+ indicesDtype: str = "int64"
+ indptrDtype: str = "int64"
+
+
+@dataclass(frozen=True, slots=True)
+class DownloadResult:
+ filePath: Path
+ fileBytes: int
+ sha256: str
+
+
+@dataclass(frozen=True, slots=True)
+class SourceValidation:
+ nRows: int
+ nColumns: int
+ nnz: int
+ finalRowNnz: int
+ dataDtype: str
+ indicesDtype: str
+ indptrDtype: str
+
+
+@dataclass(frozen=True, slots=True)
+class H5adWriteResult:
+ filePath: Path
+ nRows: int
+ nColumns: int
+ nnz: int
+ sourceRowsSha256: str
+ finalSourceRow: int
+ dataDtype: str
+ indicesDtype: str
+ indptrDtype: str
+
+
+@dataclass(frozen=True, slots=True)
+class InMemoryCsrSource:
+ """CSR matrix plus labels held in process memory for fast row gathers."""
+
+ data: np.ndarray
+ indices: np.ndarray
+ indptr: np.ndarray
+ nRows: int
+ nColumns: int
+ dataDtype: np.dtype
+ indicesDtype: np.dtype
+ indptrDtype: np.dtype
+ cellIds: np.ndarray
+ featureIds: np.ndarray
+ featureNames: np.ndarray
+
+ @property
+ def residentBytes(self) -> int:
+ return int(
+ self.data.nbytes
+ + self.indices.nbytes
+ + self.indptr.nbytes
+ + self.cellIds.nbytes
+ + self.featureIds.nbytes
+ + self.featureNames.nbytes
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class PreparedValidation:
+ fileBytes: int
+ sha256: str
+ sourceRowsSha256: str
+ nRows: int
+ nColumns: int
+ nnz: int
+ finalSourceRow: int
+
+
+@dataclass(frozen=True, slots=True)
+class PreparedArtifact:
+ localPath: Path
+ targetRows: int
+ nColumns: int
+ nnz: int
+ fileBytes: int
+ sha256: str
+ sourceRowsSha256: str
+ finalSourceRow: int
+ dataDtype: str
+ indicesDtype: str
+ indptrDtype: str
+
+
+@dataclass(frozen=True, slots=True)
+class PreparationResult:
+ sourcePath: Path
+ sourceSha256: str
+ sourceSpec: SourceSpec
+ seed: int
+ artifacts: tuple[PreparedArtifact, ...]
+
+
+SOURCE_SPEC = SourceSpec(
+ datasetId="dcfd4feb-18a3-4b30-81d7-1b0c544a8ab3",
+ versionId="1bc30289-9565-4099-abf9-3326328c11ac",
+ url=(
+ "https://datasets.cellxgene.cziscience.com/"
+ "1bc30289-9565-4099-abf9-3326328c11ac.h5ad"
+ ),
+ nRows=11_441_407,
+ nColumns=45_525,
+ nnz=19_516_755_155,
+ sourceBytes=46_292_192_475,
+)
+
+
+def splitmix64(value: int) -> int:
+ """Return SplitMix64 for one unsigned 64-bit input."""
+ mixed = (value + _SPLITMIX_INCREMENT) & _MASK_64
+ mixed = ((mixed ^ (mixed >> 30)) * _SPLITMIX_MULTIPLIER_1) & _MASK_64
+ mixed = ((mixed ^ (mixed >> 27)) * _SPLITMIX_MULTIPLIER_2) & _MASK_64
+ return mixed ^ (mixed >> 31)
+
+
+def sampling_salt(
+ seed: int,
+ sourceVersion: str = SOURCE_SPEC.versionId,
+) -> int:
+ material = (
+ _SAMPLING_DOMAIN + b"\0" + sourceVersion.encode() + b"\0" + str(seed).encode()
+ )
+ return int.from_bytes(hashlib.sha256(material).digest()[:8], "little")
+
+
+def row_priorities(
+ nRows: int,
+ *,
+ seed: int = DEFAULT_SAMPLING_SEED,
+ sourceVersion: str = SOURCE_SPEC.versionId,
+) -> np.ndarray:
+ if nRows < 0:
+ raise ValueError("nRows must be nonnegative")
+
+ # For source row i:
+ # priority(i) = SplitMix64((i + salt(sourceVersion, seed)) mod 2**64).
+ # SplitMix64 is a fixed integer permutation, so this is reproducible across
+ # Python and NumPy versions. Selection ranks (priority, source row index),
+ # which makes the row index the explicit collision tie-break.
+ values = np.arange(nRows, dtype=np.uint64)
+ values += np.uint64(sampling_salt(seed, sourceVersion))
+ values += np.uint64(_SPLITMIX_INCREMENT)
+ values = (values ^ (values >> np.uint64(30))) * np.uint64(_SPLITMIX_MULTIPLIER_1)
+ values = (values ^ (values >> np.uint64(27))) * np.uint64(_SPLITMIX_MULTIPLIER_2)
+ return values ^ (values >> np.uint64(31))
+
+
+def _normalize_targets(targetRows: Sequence[int], nRows: int) -> tuple[int, ...]:
+ targets = tuple(int(value) for value in targetRows)
+ if not targets:
+ raise ValueError("At least one target row count is required")
+ if any(value <= 0 for value in targets):
+ raise ValueError("Target row counts must be positive")
+ if tuple(sorted(set(targets))) != targets:
+ raise ValueError("Target row counts must be unique and increasing")
+ if targets[-1] > nRows:
+ raise ValueError(f"Largest target has {targets[-1]} rows, source has {nRows}")
+ return targets
+
+
+def select_rows_from_priorities(
+ priorities: np.ndarray,
+ targetRows: Sequence[int],
+) -> dict[int, np.ndarray]:
+ priorities = np.asarray(priorities)
+ if priorities.ndim != 1:
+ raise ValueError("priorities must be one-dimensional")
+ if not np.issubdtype(priorities.dtype, np.integer):
+ raise TypeError("priorities must use an integer dtype")
+ targets = _normalize_targets(targetRows, len(priorities))
+ source_rows = np.arange(len(priorities), dtype=np.int64)
+ ranked = np.lexsort((source_rows, priorities.astype(np.uint64, copy=False)))
+ return {
+ target: np.sort(ranked[:target].astype(np.int64, copy=False), kind="stable")
+ for target in targets
+ }
+
+
+def select_nested_rows(
+ nRows: int,
+ targetRows: Sequence[int] = DEFAULT_TARGET_SIZES,
+ *,
+ seed: int = DEFAULT_SAMPLING_SEED,
+ sourceVersion: str = SOURCE_SPEC.versionId,
+) -> dict[int, np.ndarray]:
+ priorities = row_priorities(
+ nRows,
+ seed=seed,
+ sourceVersion=sourceVersion,
+ )
+ return select_rows_from_priorities(priorities, targetRows)
+
+
+def ordered_source_row_digest(
+ sourceRows: np.ndarray | Sequence[int],
+ *,
+ chunkRows: int = 1_000_000,
+) -> str:
+ rows = np.asarray(sourceRows)
+ if rows.ndim != 1:
+ raise ValueError("sourceRows must be one-dimensional")
+ if not np.issubdtype(rows.dtype, np.integer):
+ raise TypeError("sourceRows must use an integer dtype")
+ if chunkRows <= 0:
+ raise ValueError("chunkRows must be positive")
+ if len(rows) and int(rows.min()) < 0:
+ raise ValueError("sourceRows cannot contain negative values")
+ if len(rows) and int(rows.max()) > np.iinfo(np.uint64).max:
+ raise ValueError("sourceRows exceed uint64")
+
+ digest = hashlib.sha256()
+ digest.update(_SOURCE_ROWS_DIGEST_DOMAIN)
+ digest.update(len(rows).to_bytes(8, "little"))
+ for start in range(0, len(rows), chunkRows):
+ normalized = np.asarray(rows[start : start + chunkRows], dtype=" DownloadResult:
+ if expectedBytes < 0:
+ raise ValueError("expectedBytes must be nonnegative")
+ if chunkBytes <= 0:
+ raise ValueError("chunkBytes must be positive")
+
+ destination = Path(destination)
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ temporary = destination.with_name(f".{destination.name}.{uuid.uuid4().hex}.part")
+ digest = hashlib.sha256()
+ file_bytes = 0
+ try:
+ with closing(opener(url)) as response:
+ headers = getattr(response, "headers", {})
+ declared = headers.get("Content-Length") or headers.get("content-length")
+ if declared is not None and int(declared) != expectedBytes:
+ raise ValueError(
+ f"Source declares {declared} bytes, expected {expectedBytes}"
+ )
+ with temporary.open("xb") as handle:
+ while chunk := response.read(chunkBytes):
+ handle.write(chunk)
+ digest.update(chunk)
+ file_bytes += len(chunk)
+ if file_bytes != expectedBytes:
+ raise ValueError(f"Downloaded {file_bytes} bytes, expected {expectedBytes}")
+ os.replace(temporary, destination)
+ except BaseException:
+ temporary.unlink(missing_ok=True)
+ raise
+
+ return DownloadResult(
+ filePath=destination,
+ fileBytes=file_bytes,
+ sha256=digest.hexdigest(),
+ )
+
+
+def _decode_text(value: Any) -> str:
+ if isinstance(value, bytes):
+ return value.decode()
+ return str(value)
+
+
+def _require_encoding(
+ value: h5py.File | h5py.Group | h5py.Dataset,
+ path: str,
+ encodingType: str,
+ encodingVersion: str,
+) -> None:
+ actual_type = value.attrs.get("encoding-type")
+ actual_version = value.attrs.get("encoding-version")
+ if actual_type is None or _decode_text(actual_type) != encodingType:
+ raise ValueError(
+ f"{path} encoding-type is {actual_type!r}, expected {encodingType!r}"
+ )
+ if actual_version is None or _decode_text(actual_version) != encodingVersion:
+ raise ValueError(
+ f"{path} encoding-version is {actual_version!r}, "
+ f"expected {encodingVersion!r}"
+ )
+
+
+def _require_group(h5: h5py.File, path: str) -> h5py.Group:
+ if path not in h5 or not isinstance(h5[path], h5py.Group):
+ raise ValueError(f"Required HDF5 group is missing: {path}")
+ return h5[path]
+
+
+def _require_dataset(h5: h5py.File, path: str) -> h5py.Dataset:
+ if path not in h5 or not isinstance(h5[path], h5py.Dataset):
+ raise ValueError(f"Required HDF5 dataset is missing: {path}")
+ return h5[path]
+
+
+def _validate_string_dataset(
+ dataset: h5py.Dataset,
+ path: str,
+ expectedLength: int,
+ *,
+ requireEncoding: bool,
+) -> None:
+ if dataset.shape != (expectedLength,):
+ raise ValueError(
+ f"{path} shape is {dataset.shape}, expected {(expectedLength,)}"
+ )
+ if h5py.check_string_dtype(dataset.dtype) is None:
+ raise ValueError(f"{path} must contain strings, got {dataset.dtype}")
+ if requireEncoding:
+ _require_encoding(dataset, path, "string-array", "0.2.0")
+
+
+def _decode_string_array(values: np.ndarray) -> np.ndarray:
+ flat = np.asarray(values).reshape(-1)
+ if flat.dtype.kind in {"U", "S", "O"}:
+ decoded = [
+ item.decode() if isinstance(item, bytes | np.bytes_) else str(item)
+ for item in flat.tolist()
+ ]
+ return np.asarray(decoded, dtype=object)
+ raise ValueError(f"Unsupported string array dtype: {flat.dtype}")
+
+
+def _load_string_column(
+ h5: h5py.File,
+ path: str,
+ *,
+ expectedLength: int,
+) -> np.ndarray:
+ """Load a string AnnData column from a dataset or categorical group."""
+ if path not in h5:
+ raise ValueError(f"Required HDF5 path is missing: {path}")
+ value = h5[path]
+ if isinstance(value, h5py.Dataset):
+ if value.shape != (expectedLength,):
+ raise ValueError(
+ f"{path} shape is {value.shape}, expected {(expectedLength,)}"
+ )
+ return _decode_string_array(np.asarray(value[:]))
+ if isinstance(value, h5py.Group):
+ encoding = value.attrs.get("encoding-type")
+ if encoding is None or _decode_text(encoding) != "categorical":
+ raise ValueError(f"{path} is a group but not a categorical string column")
+ categories = _decode_string_array(
+ np.asarray(_require_dataset(h5, f"{path}/categories")[:])
+ )
+ codes = np.asarray(_require_dataset(h5, f"{path}/codes")[:])
+ if codes.shape != (expectedLength,):
+ raise ValueError(
+ f"{path}/codes shape is {codes.shape}, expected {(expectedLength,)}"
+ )
+ if codes.min(initial=0) < -1 or codes.max(initial=-1) >= len(categories):
+ raise ValueError(f"{path}/codes contains out-of-range category ids")
+ # AnnData uses -1 for missing; map those to empty strings.
+ output = np.empty(expectedLength, dtype=object)
+ missing = codes < 0
+ output[missing] = ""
+ output[~missing] = categories[codes[~missing]]
+ return output
+ raise ValueError(f"{path} must be a string dataset or categorical group")
+
+
+def _validate_indptr(
+ indptr: h5py.Dataset,
+ *,
+ nRows: int,
+ nnz: int,
+ expectedDtype: str,
+ chunkRows: int,
+) -> int:
+ if chunkRows <= 0:
+ raise ValueError("chunkRows must be positive")
+ if indptr.shape != (nRows + 1,):
+ raise ValueError(f"CSR indptr shape is {indptr.shape}, expected {(nRows + 1,)}")
+ if indptr.dtype != np.dtype(expectedDtype):
+ raise ValueError(
+ f"CSR indptr dtype is {indptr.dtype}, expected {expectedDtype}"
+ )
+ if int(indptr[0]) != 0:
+ raise ValueError("CSR indptr must start at zero")
+ if int(indptr[-1]) != nnz:
+ raise ValueError(f"CSR indptr terminates at {int(indptr[-1])}, expected {nnz}")
+
+ previous: int | None = None
+ for start in range(0, nRows + 1, chunkRows):
+ values = np.asarray(indptr[start : min(start + chunkRows, nRows + 1)])
+ if previous is not None and len(values) and int(values[0]) < previous:
+ raise ValueError(f"CSR indptr decreases at position {start}")
+ if len(values) > 1 and np.any(values[1:] < values[:-1]):
+ offset = int(np.flatnonzero(values[1:] < values[:-1])[0]) + start + 1
+ raise ValueError(f"CSR indptr decreases at position {offset}")
+ if len(values):
+ previous = int(values[-1])
+
+ if nRows == 0:
+ return 0
+ return int(indptr[-1]) - int(indptr[-2])
+
+
+def validate_source_h5ad(
+ path: str | Path,
+ *,
+ spec: SourceSpec = SOURCE_SPEC,
+ indptrChunkRows: int = _DEFAULT_INDPTR_CHUNK_ROWS,
+) -> SourceValidation:
+ path = Path(path)
+ file_bytes = path.stat().st_size
+ if file_bytes != spec.sourceBytes:
+ raise ValueError(f"Source has {file_bytes} bytes, expected {spec.sourceBytes}")
+
+ with h5py.File(path, mode="r") as h5:
+ _require_encoding(h5, "/", "anndata", "0.1.0")
+ matrix = _require_group(h5, spec.matrixKey)
+ _require_encoding(
+ matrix,
+ spec.matrixKey,
+ spec.matrixEncoding,
+ "0.1.0",
+ )
+ shape = tuple(int(value) for value in matrix.attrs.get("shape", ()))
+ expected_shape = (spec.nRows, spec.nColumns)
+ if shape != expected_shape:
+ raise ValueError(
+ f"{spec.matrixKey} shape is {shape}, expected {expected_shape}"
+ )
+
+ data = _require_dataset(h5, f"{spec.matrixKey}/data")
+ indices = _require_dataset(h5, f"{spec.matrixKey}/indices")
+ indptr = _require_dataset(h5, f"{spec.matrixKey}/indptr")
+ if data.shape != (spec.nnz,):
+ raise ValueError(f"CSR data shape is {data.shape}, expected {(spec.nnz,)}")
+ if indices.shape != (spec.nnz,):
+ raise ValueError(
+ f"CSR indices shape is {indices.shape}, expected {(spec.nnz,)}"
+ )
+ if data.dtype != np.dtype(spec.dataDtype):
+ raise ValueError(
+ f"CSR data dtype is {data.dtype}, expected {spec.dataDtype}"
+ )
+ if indices.dtype != np.dtype(spec.indicesDtype):
+ raise ValueError(
+ f"CSR indices dtype is {indices.dtype}, expected {spec.indicesDtype}"
+ )
+ final_row_nnz = _validate_indptr(
+ indptr,
+ nRows=spec.nRows,
+ nnz=spec.nnz,
+ expectedDtype=spec.indptrDtype,
+ chunkRows=indptrChunkRows,
+ )
+
+ cell_ids = _require_dataset(h5, spec.cellIdsKey)
+ feature_ids = _require_dataset(h5, spec.featureIdsKey)
+ _validate_string_dataset(
+ cell_ids,
+ spec.cellIdsKey,
+ spec.nRows,
+ requireEncoding=True,
+ )
+ _validate_string_dataset(
+ feature_ids,
+ spec.featureIdsKey,
+ spec.nColumns,
+ requireEncoding=True,
+ )
+ feature_names = _load_string_column(
+ h5,
+ spec.featureNameKey,
+ expectedLength=spec.nColumns,
+ )
+ if feature_names.shape != (spec.nColumns,):
+ raise ValueError(
+ f"{spec.featureNameKey} resolved length is {feature_names.shape}, "
+ f"expected {(spec.nColumns,)}"
+ )
+
+ return SourceValidation(
+ nRows=spec.nRows,
+ nColumns=spec.nColumns,
+ nnz=spec.nnz,
+ finalRowNnz=final_row_nnz,
+ dataDtype=spec.dataDtype,
+ indicesDtype=spec.indicesDtype,
+ indptrDtype=spec.indptrDtype,
+ )
+
+
+def _validate_selected_rows(
+ sourceRows: np.ndarray | Sequence[int],
+ nRows: int,
+) -> np.ndarray:
+ rows = np.asarray(sourceRows)
+ if rows.ndim != 1:
+ raise ValueError("sourceRows must be one-dimensional")
+ if not np.issubdtype(rows.dtype, np.integer):
+ raise TypeError("sourceRows must use an integer dtype")
+ if len(rows) == 0:
+ raise ValueError("sourceRows cannot be empty")
+ if int(rows.min()) < 0 or int(rows.max()) >= nRows:
+ raise ValueError(f"sourceRows must be between 0 and {nRows - 1}")
+ rows = rows.astype(np.int64, copy=False)
+ if np.any(rows[1:] <= rows[:-1]):
+ raise ValueError("sourceRows must be unique and in source order")
+ return rows
+
+
+def _row_boundaries(
+ indptr: h5py.Dataset,
+ rows: np.ndarray,
+) -> tuple[np.ndarray, np.ndarray]:
+ span_rows = int(rows[-1] - rows[0] + 1)
+ if span_rows <= len(rows) * 4:
+ pointers = np.asarray(indptr[int(rows[0]) : int(rows[-1]) + 2])
+ local_rows = rows - rows[0]
+ return pointers[local_rows], pointers[local_rows + 1]
+ return np.asarray(indptr[rows]), np.asarray(indptr[rows + 1])
+
+
+def _selected_nnz(
+ indptr: h5py.Dataset,
+ rows: np.ndarray,
+ rowBatchSize: int,
+) -> int:
+ total = 0
+ for start in range(0, len(rows), rowBatchSize):
+ batch = rows[start : start + rowBatchSize]
+ row_starts, row_ends = _row_boundaries(indptr, batch)
+ counts = row_ends - row_starts
+ if np.any(counts < 0):
+ raise ValueError("Source CSR indptr is not monotonic")
+ total += int(counts.sum(dtype=np.int64))
+ return total
+
+
+def _chunk_elements(length: int, dtype: np.dtype[Any]) -> int:
+ desired = max(1, _H5_DATA_CHUNK_BYTES // max(1, dtype.itemsize))
+ return max(1, min(max(1, length), desired))
+
+
+def _create_numeric_dataset(
+ group: h5py.Group,
+ name: str,
+ length: int,
+ dtype: np.dtype[Any],
+) -> h5py.Dataset:
+ return group.create_dataset(
+ name,
+ shape=(length,),
+ maxshape=(None,),
+ dtype=dtype,
+ chunks=(_chunk_elements(length, dtype),),
+ compression="gzip",
+ compression_opts=4,
+ )
+
+
+def _create_string_dataset(
+ group: h5py.Group,
+ name: str,
+ length: int,
+) -> h5py.Dataset:
+ dataset = group.create_dataset(
+ name,
+ shape=(length,),
+ maxshape=(None,),
+ dtype=h5py.string_dtype(encoding="utf-8"),
+ chunks=(max(1, min(max(1, length), 65_536)),),
+ compression="gzip",
+ compression_opts=4,
+ )
+ dataset.attrs["encoding-type"] = "string-array"
+ dataset.attrs["encoding-version"] = "0.2.0"
+ return dataset
+
+
+def _create_dataframe_group(
+ h5: h5py.File,
+ name: str,
+ columns: Sequence[str],
+) -> h5py.Group:
+ group = h5.create_group(name)
+ group.attrs["encoding-type"] = "dataframe"
+ group.attrs["encoding-version"] = "0.2.0"
+ group.attrs["_index"] = "_index"
+ group.attrs["column-order"] = np.asarray(
+ columns,
+ dtype=h5py.string_dtype(encoding="utf-8"),
+ )
+ return group
+
+
+def _copy_pair_slice(
+ sourceData: h5py.Dataset,
+ sourceIndices: h5py.Dataset,
+ destinationData: h5py.Dataset,
+ destinationIndices: h5py.Dataset,
+ sourceStart: int,
+ sourceEnd: int,
+ destinationStart: int,
+ copyBufferBytes: int,
+) -> int:
+ item_bytes = sourceData.dtype.itemsize + sourceIndices.dtype.itemsize
+ chunk_elements = max(1, copyBufferBytes // max(1, item_bytes))
+ output_position = destinationStart
+ for start in range(sourceStart, sourceEnd, chunk_elements):
+ end = min(start + chunk_elements, sourceEnd)
+ data_values = np.asarray(sourceData[start:end])
+ index_values = np.asarray(sourceIndices[start:end])
+ output_end = output_position + len(data_values)
+ destinationData[output_position:output_end] = data_values
+ destinationIndices[output_position:output_end] = index_values
+ output_position = output_end
+ return output_position
+
+
+def _copy_masked_csr_window(
+ sourceData: h5py.Dataset,
+ sourceIndices: h5py.Dataset,
+ destinationData: h5py.Dataset,
+ destinationIndices: h5py.Dataset,
+ rowStarts: np.ndarray,
+ rowEnds: np.ndarray,
+ destinationStart: int,
+) -> int:
+ span_start = int(rowStarts[0])
+ span_end = int(rowEnds[-1])
+ span_nnz = span_end - span_start
+ selected_nnz = int((rowEnds - rowStarts).sum(dtype=np.int64))
+ if selected_nnz == 0:
+ return destinationStart
+
+ data_values = np.asarray(sourceData[span_start:span_end])
+ index_values = np.asarray(sourceIndices[span_start:span_end])
+ local_starts = rowStarts.astype(np.int64, copy=False) - span_start
+ local_ends = rowEnds.astype(np.int64, copy=False) - span_start
+ deltas = np.zeros(span_nnz + 1, dtype=np.int16)
+ np.add.at(deltas, local_starts, 1)
+ np.add.at(deltas, local_ends, -1)
+ mask = np.cumsum(deltas[:-1], dtype=np.int32) > 0
+ if int(mask.sum()) != selected_nnz:
+ raise ValueError("Selected CSR row mask has an unexpected size")
+
+ destination_end = destinationStart + selected_nnz
+ destinationData[destinationStart:destination_end] = data_values[mask]
+ destinationIndices[destinationStart:destination_end] = index_values[mask]
+ return destination_end
+
+
+def _copy_csr_batch(
+ sourceData: h5py.Dataset,
+ sourceIndices: h5py.Dataset,
+ destinationData: h5py.Dataset,
+ destinationIndices: h5py.Dataset,
+ rows: np.ndarray,
+ rowStarts: np.ndarray,
+ rowEnds: np.ndarray,
+ destinationStart: int,
+ copyBufferBytes: int,
+) -> int:
+ counts = rowEnds - rowStarts
+ selected_nnz = int(counts.sum(dtype=np.int64))
+ if selected_nnz == 0:
+ return destinationStart
+
+ span_start = int(rowStarts[0])
+ span_end = int(rowEnds[-1])
+ span_nnz = span_end - span_start
+ item_bytes = sourceData.dtype.itemsize + sourceIndices.dtype.itemsize
+ estimated_bytes = span_nnz * (2 * item_bytes + 7)
+ if span_nnz <= selected_nnz * 2 and estimated_bytes <= copyBufferBytes:
+ return _copy_masked_csr_window(
+ sourceData,
+ sourceIndices,
+ destinationData,
+ destinationIndices,
+ rowStarts,
+ rowEnds,
+ destinationStart,
+ )
+
+ output_position = destinationStart
+ max_span_nnz = max(1, copyBufferBytes // max(1, 2 * item_bytes + 7))
+ window_start = 0
+ while window_start < len(rows):
+ window_end = window_start + 1
+ while (
+ window_end < len(rows)
+ and int(rowEnds[window_end] - rowStarts[window_start]) <= max_span_nnz
+ ):
+ window_end += 1
+
+ window_starts = rowStarts[window_start:window_end]
+ window_ends = rowEnds[window_start:window_end]
+ if int(window_ends[-1] - window_starts[0]) > max_span_nnz:
+ output_position = _copy_pair_slice(
+ sourceData,
+ sourceIndices,
+ destinationData,
+ destinationIndices,
+ int(window_starts[0]),
+ int(window_ends[0]),
+ output_position,
+ copyBufferBytes,
+ )
+ else:
+ output_position = _copy_masked_csr_window(
+ sourceData,
+ sourceIndices,
+ destinationData,
+ destinationIndices,
+ window_starts,
+ window_ends,
+ output_position,
+ )
+ window_start = window_end
+ if output_position != destinationStart + selected_nnz:
+ raise ValueError("Copied CSR data length does not match selected rows")
+ return output_position
+
+
+def _copy_selected_strings(
+ source: h5py.Dataset,
+ destination: h5py.Dataset,
+ rows: np.ndarray,
+ rowBatchSize: int,
+) -> None:
+ output_start = 0
+ for start in range(0, len(rows), rowBatchSize):
+ batch = rows[start : start + rowBatchSize]
+ span_rows = int(batch[-1] - batch[0] + 1)
+ if span_rows <= len(batch) * 2:
+ values = np.asarray(source[int(batch[0]) : int(batch[-1]) + 1])
+ values = values[batch - batch[0]]
+ else:
+ values = np.asarray(source[batch])
+ output_end = output_start + len(batch)
+ destination[output_start:output_end] = values
+ output_start = output_end
+
+
+def _check_integer_capacity(dtype: np.dtype[Any], maximum: int) -> None:
+ if not np.issubdtype(dtype, np.integer):
+ raise ValueError(f"CSR index dtype must be integer, got {dtype}")
+ if maximum > int(np.iinfo(dtype).max):
+ raise OverflowError(f"{maximum} does not fit in {dtype}")
+
+
+def _load_numeric_dataset(
+ dataset: h5py.Dataset,
+ *,
+ dtype: np.dtype[Any] | None = None,
+ chunkElements: int = _DEFAULT_LOAD_CHUNK_ELEMENTS,
+) -> np.ndarray:
+ if dataset.ndim != 1:
+ raise ValueError("Expected a 1-D HDF5 dataset")
+ if chunkElements <= 0:
+ raise ValueError("chunkElements must be positive")
+ length = int(dataset.shape[0])
+ out_dtype = np.dtype(dataset.dtype if dtype is None else dtype)
+ output = np.empty(length, dtype=out_dtype)
+ for start in range(0, length, chunkElements):
+ end = min(start + chunkElements, length)
+ output[start:end] = np.asarray(dataset[start:end], dtype=out_dtype)
+ return output
+
+
+def load_csr_source_into_memory(
+ path: str | Path,
+ *,
+ spec: SourceSpec = SOURCE_SPEC,
+ chunkElements: int = _DEFAULT_LOAD_CHUNK_ELEMENTS,
+) -> InMemoryCsrSource:
+ """Load the source CSR into RAM for repeated nested subset writes.
+
+ Column indices are stored as int32 when they fit, which roughly halves
+ resident memory versus the on-disk int64 index array.
+ """
+ path = Path(path)
+ with h5py.File(path, mode="r") as h5:
+ data_dataset = _require_dataset(h5, f"{spec.matrixKey}/data")
+ indices_dataset = _require_dataset(h5, f"{spec.matrixKey}/indices")
+ indptr_dataset = _require_dataset(h5, f"{spec.matrixKey}/indptr")
+ if data_dataset.shape != (spec.nnz,):
+ raise ValueError(
+ f"CSR data shape is {data_dataset.shape}, expected {(spec.nnz,)}"
+ )
+ if indices_dataset.shape != (spec.nnz,):
+ raise ValueError(
+ f"CSR indices shape is {indices_dataset.shape}, expected {(spec.nnz,)}"
+ )
+ if indptr_dataset.shape != (spec.nRows + 1,):
+ raise ValueError(
+ f"CSR indptr shape is {indptr_dataset.shape}, "
+ f"expected {(spec.nRows + 1,)}"
+ )
+
+ data = _load_numeric_dataset(data_dataset, chunkElements=chunkElements)
+ max_column = spec.nColumns - 1
+ if max_column <= int(np.iinfo(np.int32).max):
+ indices = _load_numeric_dataset(
+ indices_dataset,
+ dtype=np.dtype(np.int32),
+ chunkElements=chunkElements,
+ )
+ else:
+ indices = _load_numeric_dataset(
+ indices_dataset,
+ chunkElements=chunkElements,
+ )
+ indptr = _load_numeric_dataset(indptr_dataset, chunkElements=chunkElements)
+ if int(indptr[0]) != 0 or int(indptr[-1]) != spec.nnz:
+ raise ValueError("Loaded CSR indptr does not match source nnz")
+
+ cell_ids = _decode_string_array(_require_dataset(h5, spec.cellIdsKey)[:])
+ feature_ids = _decode_string_array(_require_dataset(h5, spec.featureIdsKey)[:])
+ feature_names = _load_string_column(
+ h5,
+ spec.featureNameKey,
+ expectedLength=spec.nColumns,
+ )
+
+ if cell_ids.shape != (spec.nRows,):
+ raise ValueError(
+ f"Loaded cell ids shape is {cell_ids.shape}, expected {(spec.nRows,)}"
+ )
+ if feature_ids.shape != (spec.nColumns,):
+ raise ValueError(
+ f"Loaded feature ids shape is {feature_ids.shape}, "
+ f"expected {(spec.nColumns,)}"
+ )
+ if feature_names.shape != (spec.nColumns,):
+ raise ValueError(
+ f"Loaded feature names shape is {feature_names.shape}, "
+ f"expected {(spec.nColumns,)}"
+ )
+
+ return InMemoryCsrSource(
+ data=data,
+ indices=indices,
+ indptr=indptr,
+ nRows=spec.nRows,
+ nColumns=spec.nColumns,
+ dataDtype=np.dtype(spec.dataDtype),
+ indicesDtype=np.dtype(spec.indicesDtype),
+ indptrDtype=np.dtype(spec.indptrDtype),
+ cellIds=cell_ids,
+ featureIds=feature_ids,
+ featureNames=feature_names,
+ )
+
+
+def _copy_memory_csr_batch(
+ source: InMemoryCsrSource,
+ destinationData: h5py.Dataset,
+ destinationIndices: h5py.Dataset,
+ rows: np.ndarray,
+ destinationStart: int,
+) -> int:
+ starts = source.indptr[rows]
+ ends = source.indptr[rows + 1]
+ counts = ends - starts
+ if np.any(counts < 0):
+ raise ValueError("Source CSR indptr is not monotonic")
+ selected_nnz = int(counts.sum(dtype=np.int64))
+ if selected_nnz == 0:
+ return destinationStart
+
+ buffer_data = np.empty(selected_nnz, dtype=source.data.dtype)
+ buffer_indices = np.empty(selected_nnz, dtype=source.indices.dtype)
+ position = 0
+ for start, end in zip(starts.tolist(), ends.tolist(), strict=True):
+ width = end - start
+ buffer_data[position : position + width] = source.data[start:end]
+ buffer_indices[position : position + width] = source.indices[start:end]
+ position += width
+ if position != selected_nnz:
+ raise ValueError("Copied CSR batch length does not match selected nnz")
+
+ destination_end = destinationStart + selected_nnz
+ destinationData[destinationStart:destination_end] = buffer_data
+ if buffer_indices.dtype == source.indicesDtype:
+ destinationIndices[destinationStart:destination_end] = buffer_indices
+ else:
+ destinationIndices[destinationStart:destination_end] = buffer_indices.astype(
+ source.indicesDtype,
+ copy=False,
+ )
+ return destination_end
+
+
+def _copy_selected_strings_array(
+ source: np.ndarray,
+ destination: h5py.Dataset,
+ rows: np.ndarray,
+ rowBatchSize: int,
+) -> None:
+ output_start = 0
+ for start in range(0, len(rows), rowBatchSize):
+ batch = rows[start : start + rowBatchSize]
+ output_end = output_start + len(batch)
+ destination[output_start:output_end] = source[batch]
+ output_start = output_end
+
+
+def write_h5ad_sample_from_memory(
+ source: InMemoryCsrSource,
+ destinationPath: str | Path,
+ sourceRows: np.ndarray | Sequence[int],
+ *,
+ rowBatchSize: int = _DEFAULT_ROW_BATCH_SIZE,
+) -> H5adWriteResult:
+ if rowBatchSize <= 0:
+ raise ValueError("rowBatchSize must be positive")
+
+ destination_path = Path(destinationPath)
+ destination_path.parent.mkdir(parents=True, exist_ok=True)
+ if destination_path.exists():
+ raise FileExistsError(f"Destination already exists: {destination_path}")
+ temporary_path = destination_path.with_name(
+ f".{destination_path.name}.{uuid.uuid4().hex}.part"
+ )
+ rows = _validate_selected_rows(sourceRows, source.nRows)
+ nnz = int((source.indptr[rows + 1] - source.indptr[rows]).sum(dtype=np.int64))
+ _check_integer_capacity(source.indptrDtype, nnz)
+ _check_integer_capacity(source.indicesDtype, source.nColumns - 1)
+
+ try:
+ with h5py.File(temporary_path, mode="w") as destination:
+ destination.attrs["encoding-type"] = "anndata"
+ destination.attrs["encoding-version"] = "0.1.0"
+ matrix = destination.create_group("X")
+ matrix.attrs["encoding-type"] = "csr_matrix"
+ matrix.attrs["encoding-version"] = "0.1.0"
+ matrix.attrs["shape"] = np.asarray(
+ [len(rows), source.nColumns],
+ dtype=np.int64,
+ )
+ destination_data = _create_numeric_dataset(
+ matrix,
+ "data",
+ nnz,
+ source.dataDtype,
+ )
+ destination_indices = _create_numeric_dataset(
+ matrix,
+ "indices",
+ nnz,
+ source.indicesDtype,
+ )
+ destination_indptr = _create_numeric_dataset(
+ matrix,
+ "indptr",
+ len(rows) + 1,
+ source.indptrDtype,
+ )
+ destination_indptr[0] = 0
+
+ output_data_position = 0
+ output_row_position = 0
+ for start in range(0, len(rows), rowBatchSize):
+ batch = rows[start : start + rowBatchSize]
+ batch_starts = source.indptr[batch]
+ batch_ends = source.indptr[batch + 1]
+ counts = batch_ends - batch_starts
+ cumulative = np.cumsum(counts, dtype=np.int64) + output_data_position
+ output_row_end = output_row_position + len(batch)
+ destination_indptr[output_row_position + 1 : output_row_end + 1] = (
+ cumulative.astype(source.indptrDtype, copy=False)
+ )
+ output_data_position = _copy_memory_csr_batch(
+ source,
+ destination_data,
+ destination_indices,
+ batch,
+ output_data_position,
+ )
+ output_row_position = output_row_end
+ if output_data_position != nnz:
+ raise ValueError(
+ f"Copied {output_data_position} values, expected {nnz}"
+ )
+
+ obs = _create_dataframe_group(destination, "obs", ())
+ output_cell_ids = _create_string_dataset(obs, "_index", len(rows))
+ _copy_selected_strings_array(
+ source.cellIds,
+ output_cell_ids,
+ rows,
+ rowBatchSize,
+ )
+
+ var = _create_dataframe_group(
+ destination,
+ "var",
+ ("feature_name",),
+ )
+ output_feature_ids = _create_string_dataset(
+ var,
+ "_index",
+ source.nColumns,
+ )
+ output_feature_names = _create_string_dataset(
+ var,
+ "feature_name",
+ source.nColumns,
+ )
+ output_feature_ids[:] = source.featureIds
+ output_feature_names[:] = source.featureNames
+
+ os.replace(temporary_path, destination_path)
+ except BaseException:
+ temporary_path.unlink(missing_ok=True)
+ raise
+
+ return H5adWriteResult(
+ filePath=destination_path,
+ nRows=len(rows),
+ nColumns=source.nColumns,
+ nnz=nnz,
+ sourceRowsSha256=ordered_source_row_digest(rows),
+ finalSourceRow=int(rows[-1]),
+ dataDtype=str(source.dataDtype),
+ indicesDtype=str(source.indicesDtype),
+ indptrDtype=str(source.indptrDtype),
+ )
+
+
+def write_h5ad_sample(
+ sourcePath: str | Path,
+ destinationPath: str | Path,
+ sourceRows: np.ndarray | Sequence[int],
+ *,
+ spec: SourceSpec = SOURCE_SPEC,
+ rowBatchSize: int = _DEFAULT_ROW_BATCH_SIZE,
+ copyBufferBytes: int = _DEFAULT_COPY_BUFFER_BYTES,
+) -> H5adWriteResult:
+ if rowBatchSize <= 0:
+ raise ValueError("rowBatchSize must be positive")
+ if copyBufferBytes <= 0:
+ raise ValueError("copyBufferBytes must be positive")
+
+ source_path = Path(sourcePath)
+ destination_path = Path(destinationPath)
+ destination_path.parent.mkdir(parents=True, exist_ok=True)
+ if destination_path.exists():
+ raise FileExistsError(f"Destination already exists: {destination_path}")
+ temporary_path = destination_path.with_name(
+ f".{destination_path.name}.{uuid.uuid4().hex}.part"
+ )
+
+ try:
+ with (
+ h5py.File(source_path, mode="r") as source,
+ h5py.File(temporary_path, mode="w") as destination,
+ ):
+ source_matrix = _require_group(source, spec.matrixKey)
+ shape = tuple(int(value) for value in source_matrix.attrs.get("shape", ()))
+ if len(shape) != 2:
+ raise ValueError(f"{spec.matrixKey} has invalid shape metadata")
+ n_source_rows, n_columns = shape
+ rows = _validate_selected_rows(sourceRows, n_source_rows)
+
+ source_data = _require_dataset(source, f"{spec.matrixKey}/data")
+ source_indices = _require_dataset(
+ source,
+ f"{spec.matrixKey}/indices",
+ )
+ source_indptr = _require_dataset(
+ source,
+ f"{spec.matrixKey}/indptr",
+ )
+ if source_indptr.shape != (n_source_rows + 1,):
+ raise ValueError("Source CSR indptr length does not match shape")
+ if not np.issubdtype(source_indices.dtype, np.integer):
+ raise ValueError("Source CSR indices must use an integer dtype")
+ if not np.issubdtype(source_indptr.dtype, np.integer):
+ raise ValueError("Source CSR indptr must use an integer dtype")
+ data_dtype = str(source_data.dtype)
+ indices_dtype = str(source_indices.dtype)
+ indptr_dtype = str(source_indptr.dtype)
+
+ nnz = _selected_nnz(source_indptr, rows, rowBatchSize)
+ _check_integer_capacity(source_indptr.dtype, nnz)
+ _check_integer_capacity(source_indices.dtype, n_columns - 1)
+
+ destination.attrs["encoding-type"] = "anndata"
+ destination.attrs["encoding-version"] = "0.1.0"
+ matrix = destination.create_group("X")
+ matrix.attrs["encoding-type"] = "csr_matrix"
+ matrix.attrs["encoding-version"] = "0.1.0"
+ matrix.attrs["shape"] = np.asarray(
+ [len(rows), n_columns],
+ dtype=np.int64,
+ )
+ destination_data = _create_numeric_dataset(
+ matrix,
+ "data",
+ nnz,
+ source_data.dtype,
+ )
+ destination_indices = _create_numeric_dataset(
+ matrix,
+ "indices",
+ nnz,
+ source_indices.dtype,
+ )
+ destination_indptr = _create_numeric_dataset(
+ matrix,
+ "indptr",
+ len(rows) + 1,
+ source_indptr.dtype,
+ )
+ destination_indptr[0] = 0
+
+ output_data_position = 0
+ output_row_position = 0
+ for start in range(0, len(rows), rowBatchSize):
+ batch = rows[start : start + rowBatchSize]
+ row_starts, row_ends = _row_boundaries(source_indptr, batch)
+ counts = row_ends - row_starts
+ cumulative = np.cumsum(counts, dtype=np.int64) + output_data_position
+ output_row_end = output_row_position + len(batch)
+ destination_indptr[output_row_position + 1 : output_row_end + 1] = (
+ cumulative.astype(source_indptr.dtype, copy=False)
+ )
+ output_data_position = _copy_csr_batch(
+ source_data,
+ source_indices,
+ destination_data,
+ destination_indices,
+ batch,
+ row_starts,
+ row_ends,
+ output_data_position,
+ copyBufferBytes,
+ )
+ output_row_position = output_row_end
+ if output_data_position != nnz:
+ raise ValueError(
+ f"Copied {output_data_position} values, expected {nnz}"
+ )
+
+ obs = _create_dataframe_group(destination, "obs", ())
+ output_cell_ids = _create_string_dataset(obs, "_index", len(rows))
+ source_cell_ids = _require_dataset(source, spec.cellIdsKey)
+ _copy_selected_strings(
+ source_cell_ids,
+ output_cell_ids,
+ rows,
+ rowBatchSize,
+ )
+
+ var = _create_dataframe_group(
+ destination,
+ "var",
+ ("feature_name",),
+ )
+ output_feature_ids = _create_string_dataset(
+ var,
+ "_index",
+ n_columns,
+ )
+ output_feature_names = _create_string_dataset(
+ var,
+ "feature_name",
+ n_columns,
+ )
+ source_feature_ids = _require_dataset(source, spec.featureIdsKey)
+ source_feature_names = _load_string_column(
+ source,
+ spec.featureNameKey,
+ expectedLength=n_columns,
+ )
+ output_feature_ids[:] = source_feature_ids[:]
+ output_feature_names[:] = source_feature_names[:]
+
+ os.replace(temporary_path, destination_path)
+ except BaseException:
+ temporary_path.unlink(missing_ok=True)
+ raise
+
+ return H5adWriteResult(
+ filePath=destination_path,
+ nRows=len(rows),
+ nColumns=n_columns,
+ nnz=nnz,
+ sourceRowsSha256=ordered_source_row_digest(rows),
+ finalSourceRow=int(rows[-1]),
+ dataDtype=data_dtype,
+ indicesDtype=indices_dtype,
+ indptrDtype=indptr_dtype,
+ )
+
+
+def read_csr_row(
+ path: str | Path,
+ row: int,
+ *,
+ matrixKey: str = SOURCE_SPEC.matrixKey,
+) -> tuple[np.ndarray, np.ndarray]:
+ with h5py.File(path, mode="r") as h5:
+ matrix = _require_group(h5, matrixKey)
+ shape = tuple(int(value) for value in matrix.attrs.get("shape", ()))
+ if len(shape) != 2 or row < 0 or row >= shape[0]:
+ raise IndexError(f"CSR row {row} is outside matrix shape {shape}")
+ indptr = _require_dataset(h5, f"{matrixKey}/indptr")
+ start, end = (int(value) for value in indptr[row : row + 2])
+ indices = np.asarray(_require_dataset(h5, f"{matrixKey}/indices")[start:end])
+ data = np.asarray(_require_dataset(h5, f"{matrixKey}/data")[start:end])
+ return indices, data
+
+
+def _attribute_strings(value: Any) -> list[str]:
+ return [_decode_text(item) for item in np.asarray(value).reshape(-1)]
+
+
+def _validate_output_dataframe(
+ h5: h5py.File,
+ name: str,
+ expectedLength: int,
+ columns: Sequence[str],
+) -> None:
+ group = _require_group(h5, name)
+ _require_encoding(group, name, "dataframe", "0.2.0")
+ if _decode_text(group.attrs.get("_index")) != "_index":
+ raise ValueError(f"{name} dataframe index must be _index")
+ if _attribute_strings(group.attrs.get("column-order", ())) != list(columns):
+ raise ValueError(f"{name} dataframe columns do not match {list(columns)}")
+ expected_keys = {"_index", *columns}
+ if set(group.keys()) != expected_keys:
+ raise ValueError(
+ f"{name} contains {sorted(group.keys())}, expected {sorted(expected_keys)}"
+ )
+ _validate_string_dataset(
+ _require_dataset(h5, f"{name}/_index"),
+ f"{name}/_index",
+ expectedLength,
+ requireEncoding=True,
+ )
+ for column in columns:
+ _validate_string_dataset(
+ _require_dataset(h5, f"{name}/{column}"),
+ f"{name}/{column}",
+ expectedLength,
+ requireEncoding=True,
+ )
+
+
+def validate_prepared_h5ad(
+ path: str | Path,
+ *,
+ expectedRows: int,
+ expectedColumns: int,
+ expectedNnz: int,
+ expectedSourceRows: np.ndarray | Sequence[int],
+ expectedFinalIndices: np.ndarray,
+ expectedFinalData: np.ndarray,
+ expectedDataDtype: str,
+ expectedIndicesDtype: str,
+ expectedIndptrDtype: str,
+ expectedSha256: str | None = None,
+ indptrChunkRows: int = _DEFAULT_INDPTR_CHUNK_ROWS,
+) -> PreparedValidation:
+ path = Path(path)
+ source_rows = _validate_selected_rows(expectedSourceRows, max(1, 1 << 63))
+ if len(source_rows) != expectedRows:
+ raise ValueError(
+ f"Expected source row list has {len(source_rows)} rows, "
+ f"expected {expectedRows}"
+ )
+
+ with h5py.File(path, mode="r") as h5:
+ if set(h5.keys()) != {"X", "obs", "var"}:
+ raise ValueError(
+ f"Prepared H5AD root contains unexpected keys: {sorted(h5.keys())}"
+ )
+ _require_encoding(h5, "/", "anndata", "0.1.0")
+ matrix = _require_group(h5, "X")
+ _require_encoding(matrix, "X", "csr_matrix", "0.1.0")
+ shape = tuple(int(value) for value in matrix.attrs.get("shape", ()))
+ if shape != (expectedRows, expectedColumns):
+ raise ValueError(
+ f"Prepared matrix shape is {shape}, expected "
+ f"{(expectedRows, expectedColumns)}"
+ )
+
+ data = _require_dataset(h5, "X/data")
+ indices = _require_dataset(h5, "X/indices")
+ indptr = _require_dataset(h5, "X/indptr")
+ if data.shape != (expectedNnz,) or indices.shape != (expectedNnz,):
+ raise ValueError("Prepared CSR data lengths do not match expected nnz")
+ if data.dtype != np.dtype(expectedDataDtype):
+ raise ValueError(
+ f"Prepared data dtype is {data.dtype}, expected {expectedDataDtype}"
+ )
+ if indices.dtype != np.dtype(expectedIndicesDtype):
+ raise ValueError(
+ f"Prepared indices dtype is {indices.dtype}, "
+ f"expected {expectedIndicesDtype}"
+ )
+ _validate_indptr(
+ indptr,
+ nRows=expectedRows,
+ nnz=expectedNnz,
+ expectedDtype=expectedIndptrDtype,
+ chunkRows=indptrChunkRows,
+ )
+
+ final_start, final_end = (
+ int(value) for value in indptr[expectedRows - 1 : expectedRows + 1]
+ )
+ final_indices = np.asarray(indices[final_start:final_end])
+ final_data = np.asarray(data[final_start:final_end])
+ if not np.array_equal(final_indices, expectedFinalIndices):
+ raise ValueError("Prepared final-row indices do not match source")
+ if not np.array_equal(final_data, expectedFinalData):
+ raise ValueError("Prepared final-row data do not match source")
+
+ _validate_output_dataframe(h5, "obs", expectedRows, ())
+ _validate_output_dataframe(
+ h5,
+ "var",
+ expectedColumns,
+ ("feature_name",),
+ )
+
+ import anndata
+
+ backed = anndata.read_h5ad(path, backed="r")
+ try:
+ if backed.shape != (expectedRows, expectedColumns):
+ raise ValueError(
+ f"AnnData sees shape {backed.shape}, expected "
+ f"{(expectedRows, expectedColumns)}"
+ )
+ finally:
+ backed.file.close()
+
+ from scarf.readers import H5adReader
+
+ reader = H5adReader(
+ str(path),
+ matrix_key="X",
+ cell_attrs_key="obs",
+ cell_ids_key="_index",
+ feature_attrs_key="var",
+ feature_ids_key="_index",
+ feature_name_key="feature_name",
+ )
+ try:
+ if (reader.nCells, reader.nFeatures) != (
+ expectedRows,
+ expectedColumns,
+ ):
+ raise ValueError(
+ "Scarf H5adReader dimensions do not match prepared artifact"
+ )
+ if np.dtype(reader.matrixDtype) != np.dtype(expectedDataDtype):
+ raise ValueError("Scarf H5adReader sees an unexpected data dtype")
+ finally:
+ reader.h5.close()
+
+ actual_sha256 = sha256_file(path)
+ if expectedSha256 is not None and actual_sha256 != expectedSha256.lower():
+ raise ValueError(
+ f"Prepared SHA256 is {actual_sha256}, expected {expectedSha256}"
+ )
+ return PreparedValidation(
+ fileBytes=path.stat().st_size,
+ sha256=actual_sha256,
+ sourceRowsSha256=ordered_source_row_digest(source_rows),
+ nRows=expectedRows,
+ nColumns=expectedColumns,
+ nnz=expectedNnz,
+ finalSourceRow=int(source_rows[-1]),
+ )
+
+
+def prepare_local_datasets(
+ sourcePath: str | Path,
+ outputDirectory: str | Path,
+ *,
+ targetRows: Sequence[int] = DEFAULT_TARGET_SIZES,
+ seed: int = DEFAULT_SAMPLING_SEED,
+ spec: SourceSpec = SOURCE_SPEC,
+ rowBatchSize: int = _DEFAULT_ROW_BATCH_SIZE,
+ copyBufferBytes: int = _DEFAULT_COPY_BUFFER_BYTES,
+ onArtifact: Callable[[PreparedArtifact], None] | None = None,
+) -> PreparationResult:
+ del copyBufferBytes # retained for call-site compatibility; unused in-memory path
+ source_path = Path(sourcePath)
+ output_directory = Path(outputDirectory)
+ output_directory.mkdir(parents=True, exist_ok=True)
+ source_validation = validate_source_h5ad(source_path, spec=spec)
+ source_sha256 = sha256_file(source_path)
+ selections = select_nested_rows(
+ spec.nRows,
+ targetRows,
+ seed=seed,
+ sourceVersion=spec.versionId,
+ )
+ memory_source = load_csr_source_into_memory(source_path, spec=spec)
+
+ artifacts: list[PreparedArtifact] = []
+ for target_rows, source_rows in selections.items():
+ output_path = output_directory / f"{target_rows}.h5ad"
+ if output_path.exists():
+ output_path.unlink()
+ written = write_h5ad_sample_from_memory(
+ memory_source,
+ output_path,
+ source_rows,
+ rowBatchSize=rowBatchSize,
+ )
+ final_start = int(memory_source.indptr[written.finalSourceRow])
+ final_end = int(memory_source.indptr[written.finalSourceRow + 1])
+ final_indices = np.asarray(
+ memory_source.indices[final_start:final_end],
+ dtype=memory_source.indicesDtype,
+ )
+ final_data = np.asarray(memory_source.data[final_start:final_end])
+ validated = validate_prepared_h5ad(
+ output_path,
+ expectedRows=target_rows,
+ expectedColumns=spec.nColumns,
+ expectedNnz=written.nnz,
+ expectedSourceRows=source_rows,
+ expectedFinalIndices=final_indices,
+ expectedFinalData=final_data,
+ expectedDataDtype=source_validation.dataDtype,
+ expectedIndicesDtype=source_validation.indicesDtype,
+ expectedIndptrDtype=source_validation.indptrDtype,
+ )
+ if validated.sourceRowsSha256 != written.sourceRowsSha256:
+ raise ValueError("Source-row digest changed during validation")
+ artifact = PreparedArtifact(
+ localPath=output_path,
+ targetRows=target_rows,
+ nColumns=spec.nColumns,
+ nnz=written.nnz,
+ fileBytes=validated.fileBytes,
+ sha256=validated.sha256,
+ sourceRowsSha256=validated.sourceRowsSha256,
+ finalSourceRow=written.finalSourceRow,
+ dataDtype=written.dataDtype,
+ indicesDtype=written.indicesDtype,
+ indptrDtype=written.indptrDtype,
+ )
+ artifacts.append(artifact)
+ if onArtifact is not None:
+ onArtifact(artifact)
+
+ return PreparationResult(
+ sourcePath=source_path,
+ sourceSha256=source_sha256,
+ sourceSpec=spec,
+ seed=seed,
+ artifacts=tuple(artifacts),
+ )
+
+
+def write_fixture_h5ad(
+ destinationPath: str | Path,
+ *,
+ nRows: int,
+ nColumns: int = 500,
+ seed: int = 0,
+ avgNnzPerRow: int = 40,
+) -> PreparedArtifact:
+ """Write a small synthetic CSR H5AD for downstream stage smoke tests."""
+ if nRows <= 0 or nColumns <= 0:
+ raise ValueError("nRows and nColumns must be positive")
+ if avgNnzPerRow <= 0:
+ raise ValueError("avgNnzPerRow must be positive")
+
+ destination_path = Path(destinationPath)
+ destination_path.parent.mkdir(parents=True, exist_ok=True)
+ if destination_path.exists():
+ raise FileExistsError(destination_path)
+
+ rng = np.random.default_rng(seed)
+ nnz_per_row = np.clip(
+ rng.poisson(avgNnzPerRow, size=nRows),
+ 1,
+ nColumns,
+ ).astype(np.int64)
+ indptr = np.empty(nRows + 1, dtype=np.int64)
+ indptr[0] = 0
+ np.cumsum(nnz_per_row, out=indptr[1:])
+ nnz = int(indptr[-1])
+ indices = np.empty(nnz, dtype=np.int64)
+ data = rng.integers(1, 20, size=nnz, dtype=np.int32).astype(np.float32)
+ for row, start, end in zip(
+ range(nRows),
+ indptr[:-1],
+ indptr[1:],
+ strict=True,
+ ):
+ chosen = rng.choice(nColumns, size=int(end - start), replace=False)
+ chosen.sort()
+ indices[start:end] = chosen
+
+ feature_ids = np.asarray(
+ [f"ENSG{i:011d}" for i in range(nColumns)],
+ dtype=object,
+ )
+ feature_names = np.asarray(
+ ["MT-ND1" if i % 50 == 0 else f"GENE{i}" for i in range(nColumns)],
+ dtype=object,
+ )
+ cell_ids = np.asarray([f"cell-{i}" for i in range(nRows)], dtype=object)
+
+ temporary_path = destination_path.with_name(
+ f".{destination_path.name}.{uuid.uuid4().hex}.part"
+ )
+ try:
+ with h5py.File(temporary_path, mode="w") as h5:
+ h5.attrs["encoding-type"] = "anndata"
+ h5.attrs["encoding-version"] = "0.1.0"
+ matrix = h5.create_group("X")
+ matrix.attrs["encoding-type"] = "csr_matrix"
+ matrix.attrs["encoding-version"] = "0.1.0"
+ matrix.attrs["shape"] = np.asarray([nRows, nColumns], dtype=np.int64)
+ _create_numeric_dataset(matrix, "data", nnz, data.dtype)[:] = data
+ _create_numeric_dataset(matrix, "indices", nnz, indices.dtype)[:] = indices
+ _create_numeric_dataset(
+ matrix,
+ "indptr",
+ nRows + 1,
+ indptr.dtype,
+ )[:] = indptr
+
+ obs = _create_dataframe_group(h5, "obs", ())
+ _create_string_dataset(obs, "_index", nRows)[:] = cell_ids
+ var = _create_dataframe_group(h5, "var", ("feature_name",))
+ _create_string_dataset(var, "_index", nColumns)[:] = feature_ids
+ _create_string_dataset(var, "feature_name", nColumns)[:] = feature_names
+ os.replace(temporary_path, destination_path)
+ except BaseException:
+ temporary_path.unlink(missing_ok=True)
+ raise
+
+ return PreparedArtifact(
+ localPath=destination_path,
+ targetRows=nRows,
+ nColumns=nColumns,
+ nnz=nnz,
+ fileBytes=destination_path.stat().st_size,
+ sha256=sha256_file(destination_path),
+ sourceRowsSha256=ordered_source_row_digest(np.arange(nRows, dtype=np.int64)),
+ finalSourceRow=nRows - 1,
+ dataDtype=str(data.dtype),
+ indicesDtype=str(indices.dtype),
+ indptrDtype=str(indptr.dtype),
+ )
+
+
+def prepare_fixture_datasets(
+ outputDirectory: str | Path,
+ *,
+ targetRows: Sequence[int],
+ nColumns: int = 500,
+ seed: int = 0,
+ onArtifact: Callable[[PreparedArtifact], None] | None = None,
+) -> tuple[PreparedArtifact, ...]:
+ output_directory = Path(outputDirectory)
+ output_directory.mkdir(parents=True, exist_ok=True)
+ artifacts: list[PreparedArtifact] = []
+ for n_rows in targetRows:
+ artifact = write_fixture_h5ad(
+ output_directory / f"{n_rows}.h5ad",
+ nRows=int(n_rows),
+ nColumns=nColumns,
+ seed=seed + int(n_rows),
+ )
+ artifacts.append(artifact)
+ if onArtifact is not None:
+ onArtifact(artifact)
+ return tuple(artifacts)
diff --git a/profiling/io_baseline.py b/profiling/io_baseline.py
new file mode 100644
index 00000000..18430c7d
--- /dev/null
+++ b/profiling/io_baseline.py
@@ -0,0 +1,461 @@
+"""R2 I/O baseline for a completed profiling store.
+
+Streams the same access patterns as HVG, markers, and graph construction without
+compute.
+
+Spawn via the deployed app (survives local network drop):
+
+ uv run --group profiling modal run --env scarf_profiling -m profiling.modal_app -- \\
+ io-baseline --config profiling/config.toml --size 1000000
+"""
+
+import os
+import sys
+import time
+from typing import Any
+
+import numpy as np
+
+from profiling.config import ProfilingConfig, StageResources
+from profiling.metrics import ResourceSampler
+from profiling.r2 import put_json, storage_options
+from scarf import DataStore
+from scarf.assay import _read_block
+from scarf.storage.budget import ResourceBudget
+from scarf.storage.feature_stream import FeatureStreamPlan, plan_feature_stream
+from scarf.storage.types import as_zarr_array
+from scarf.utils import iter_column_blocks
+
+_LOG_EVERY_BLOCKS = 25
+
+
+def _log(msg: str) -> None:
+ print(msg, flush=True)
+ sys.stdout.flush()
+
+
+def _fmt_bytes(n: int) -> str:
+ return f"{n / (1024**3):.2f} GiB"
+
+
+def _fmt_rss(peakRssBytes: int | None) -> str:
+ if peakRssBytes is None:
+ return "?"
+ return f"{peakRssBytes / (1024**3):.2f} GiB"
+
+
+def _open_store(storeUri: str, resources: StageResources) -> DataStore:
+ return DataStore(
+ storeUri,
+ nthreads=resources.workers,
+ zarr_mode="r",
+ zarrProfile="cloud" if storeUri.startswith("s3://") else None,
+ storage_options=storage_options(storeUri),
+ mem_budget=resources.scarfMemoryBudget,
+ )
+
+
+def _measure(label: str, fn: Any) -> dict[str, Any]:
+ _log(f"[start] {label}")
+ sampler = ResourceSampler(sampleIntervalSeconds=0.25)
+ sampler.start()
+ t0 = time.perf_counter()
+ detail: dict[str, Any]
+ try:
+ detail = fn()
+ status = "ok"
+ error = None
+ except Exception as exc:
+ detail = {}
+ status = "error"
+ error = f"{type(exc).__name__}: {exc}"
+ seconds = time.perf_counter() - t0
+ measurement = sampler.stop()
+ result = {
+ "pattern": label,
+ "status": status,
+ "seconds": round(seconds, 3),
+ "peakRssBytes": measurement.processTreeRssPeakBytes,
+ "peakCgroupBytes": measurement.cgroupMemoryCurrentPeakBytes
+ or measurement.operationPeakBytes,
+ "error": error,
+ **detail,
+ }
+ bytes_read = detail.get("bytesRead")
+ bytes_part = (
+ f" bytes={_fmt_bytes(bytes_read)}" if isinstance(bytes_read, int) else ""
+ )
+ blocks = detail.get("nBlocks")
+ blocks_part = f" blocks={blocks}" if blocks is not None else ""
+ _log(
+ f"[done] {label} status={status} wall={seconds:.1f}s "
+ f"peakRss={_fmt_rss(result['peakRssBytes'])}{blocks_part}{bytes_part}"
+ + (f" error={error}" if error else "")
+ )
+ return result
+
+
+def _progress(
+ label: str, blockIdx: int, nBlocks: int, bytesRead: int, t0: float
+) -> None:
+ if (
+ blockIdx == 0
+ or (blockIdx + 1) % _LOG_EVERY_BLOCKS != 0
+ and (blockIdx + 1) != nBlocks
+ ):
+ return
+ elapsed = time.perf_counter() - t0
+ rate = bytesRead / elapsed if elapsed > 0 else 0.0
+ _log(
+ f"[progress] {label} {blockIdx + 1}/{nBlocks} "
+ f"bytes={_fmt_bytes(bytesRead)} wall={elapsed:.1f}s "
+ f"rate={rate / (1024**2):.1f} MiB/s"
+ )
+
+
+def _stream_hvg_tiles(
+ assay: Any,
+ cellIdx: np.ndarray,
+ featIdx: np.ndarray,
+) -> dict[str, Any]:
+ """Physical chunk tiles, same layout walk as HVG feature stats."""
+ use_counts_t = getattr(assay, "rawDataT", None) is not None
+ if use_counts_t:
+ zarr_arr = assay.rawDataT
+ chunks = getattr(zarr_arr, "chunks", None)
+ feat_chunk = int(chunks[0]) if chunks and len(chunks) > 0 else len(featIdx)
+ cell_chunk = int(chunks[1]) if chunks and len(chunks) > 1 else len(cellIdx)
+ else:
+ zarr_arr = assay.rawData._backing
+ chunks = getattr(zarr_arr, "chunks", None)
+ cell_chunk = int(chunks[0]) if chunks and len(chunks) > 0 else len(cellIdx)
+ feat_chunk = int(chunks[1]) if chunks and len(chunks) > 1 else len(featIdx)
+ cell_chunk = max(1, cell_chunk)
+ feat_chunk = max(1, feat_chunk)
+
+ cell_bins = np.asarray(cellIdx // cell_chunk, dtype=np.intp)
+ feat_bins = np.asarray(featIdx // feat_chunk, dtype=np.intp)
+ tiles: list[tuple[np.ndarray, np.ndarray]] = []
+ if use_counts_t:
+ for feat_bin in np.unique(feat_bins):
+ cols = featIdx[feat_bins == feat_bin]
+ for cell_bin in np.unique(cell_bins):
+ rows = cellIdx[cell_bins == cell_bin]
+ tiles.append((rows, cols))
+ else:
+ for cell_bin in np.unique(cell_bins):
+ rows = cellIdx[cell_bins == cell_bin]
+ for feat_bin in np.unique(feat_bins):
+ cols = featIdx[feat_bins == feat_bin]
+ tiles.append((rows, cols))
+
+ n_blocks = len(tiles)
+ _log(
+ f"[plan] hvgTiles cells={len(cellIdx)} feats={len(featIdx)} "
+ f"chunks=({cell_chunk},{feat_chunk}) tiles={n_blocks} "
+ f"source={'countsT' if use_counts_t else 'counts'}"
+ )
+ bytes_read = 0
+ chunks_read = 0
+ t0 = time.perf_counter()
+
+ def read_block(block_idx: int) -> np.ndarray:
+ rows, cols = tiles[block_idx]
+ if use_counts_t:
+ return _read_block(zarr_arr, cols, rows).T
+ return _read_block(zarr_arr, rows, cols)
+
+ for block_idx, raw, read_sec, source in iter_column_blocks(
+ n_blocks,
+ read_block,
+ workers=assay.resources.workers,
+ ):
+ bytes_read += int(raw.nbytes)
+ chunks_read += 1
+ if (block_idx + 1) % _LOG_EVERY_BLOCKS == 0 or (block_idx + 1) == n_blocks:
+ _log(
+ f"[progress] hvgTiles {block_idx + 1}/{n_blocks} "
+ f"read={read_sec:.1f}s src={source} "
+ f"bytes={_fmt_bytes(bytes_read)} "
+ f"wall={time.perf_counter() - t0:.1f}s"
+ )
+ del raw
+
+ return {
+ "nCells": int(len(cellIdx)),
+ "nFeatures": int(len(featIdx)),
+ "rowChunk": cell_chunk,
+ "colChunk": feat_chunk,
+ "nBlocks": chunks_read,
+ "bytesRead": bytes_read,
+ "arraySource": "countsT" if use_counts_t else "counts",
+ }
+
+
+def _stream_marker_batches(
+ assay: Any,
+ cellIdx: np.ndarray,
+ plan: FeatureStreamPlan,
+) -> dict[str, Any]:
+ """All cells × gene batches, same path as marker search."""
+ array_source = (
+ "countsT" if getattr(assay, "rawDataT", None) is not None else "counts"
+ )
+ n_blocks = len(plan.blocks)
+ _log(
+ f"[plan] markerBatches cells={len(cellIdx)} "
+ f"feats={sum(len(block.indices) for block in plan.blocks)} "
+ f"batches={n_blocks} source={array_source}"
+ )
+ batches = [block.indices for block in plan.blocks]
+ zarr_arr = (
+ assay.rawDataT
+ if getattr(assay, "rawDataT", None) is not None
+ else assay.rawData._backing
+ )
+
+ def read_block(block_idx: int) -> np.ndarray:
+ columns = batches[block_idx]
+ if array_source == "countsT":
+ return _read_block(zarr_arr, columns, cellIdx).T
+ return _read_block(zarr_arr, cellIdx, columns)
+
+ bytes_read = 0
+ done = 0
+ t0 = time.perf_counter()
+ for block_idx, raw, read_sec, source in iter_column_blocks(
+ n_blocks,
+ read_block,
+ workers=plan.readWorkers,
+ io_concurrency=plan.ioConcurrency,
+ ):
+ feat_cols = batches[block_idx]
+ bytes_read += int(raw.nbytes)
+ done += 1
+ if done % _LOG_EVERY_BLOCKS == 0 or done == n_blocks:
+ _log(
+ f"[progress] markerBatches {done}/{n_blocks} "
+ f"width={len(feat_cols)} read={read_sec:.1f}s src={source} "
+ f"bytes={_fmt_bytes(bytes_read)} "
+ f"wall={time.perf_counter() - t0:.1f}s"
+ )
+ del raw
+ del feat_cols
+ return {
+ "nCells": int(len(cellIdx)),
+ "nFeatures": int(sum(len(batch) for batch in batches)),
+ "geneBatchSize": int(max((len(batch) for batch in batches), default=0)),
+ "nBlocks": done,
+ "bytesRead": bytes_read,
+ "arraySource": array_source,
+ }
+
+
+def _stream_graph_raw_cell_bands(
+ assay: Any,
+ cellIdx: np.ndarray,
+ hvgIdx: np.ndarray,
+) -> dict[str, Any]:
+ """Selected cells by HVG columns in normalization row bands."""
+ zarr_arr = assay.rawData._backing
+ chunks = getattr(zarr_arr, "chunks", None)
+ row_chunk = int(chunks[0]) if chunks and len(chunks) > 0 else len(cellIdx)
+ row_chunk = max(1, row_chunk)
+ n_blocks = int(np.ceil(len(cellIdx) / row_chunk))
+ _log(
+ f"[plan] graphRawCellBands cells={len(cellIdx)} hvgs={len(hvgIdx)} "
+ f"rowChunk={row_chunk} bands={n_blocks}"
+ )
+ bytes_read = 0
+ n_done = 0
+ t0 = time.perf_counter()
+ for start in range(0, len(cellIdx), row_chunk):
+ end = min(start + row_chunk, len(cellIdx))
+ raw = _read_block(zarr_arr, cellIdx[start:end], hvgIdx)
+ bytes_read += int(raw.nbytes)
+ n_done += 1
+ _progress("graphRawCellBands", n_done - 1, n_blocks, bytes_read, t0)
+ del raw
+ return {
+ "nCells": int(len(cellIdx)),
+ "nFeatures": int(len(hvgIdx)),
+ "rowChunk": row_chunk,
+ "nBlocks": n_done,
+ "bytesRead": bytes_read,
+ }
+
+
+def _stream_graph_normed_cell_bands(store: DataStore, assayName: str) -> dict[str, Any]:
+ """Row chunks of the dense normalized-HVG matrix."""
+ assay = store.get_assay(assayName)
+ cell_key = assay.attrs.get("latest_cell_key", "I")
+ feat_key = assay.attrs["latest_feat_key"]
+ loc = f"{store.get_normalized_group_path(assayName, cell_key, feat_key)}/data"
+ arr = as_zarr_array(store.zw[loc], name="data")
+ n_rows, n_cols = int(arr.shape[0]), int(arr.shape[1])
+ chunks = getattr(arr, "chunks", None)
+ row_chunk = int(chunks[0]) if chunks and len(chunks) > 0 else n_rows
+ row_chunk = max(1, row_chunk)
+ n_blocks = int(np.ceil(n_rows / row_chunk))
+ _log(
+ f"[plan] graphNormedCellBands path={loc} shape=({n_rows},{n_cols}) "
+ f"rowChunk={row_chunk} bands={n_blocks} dtype={arr.dtype}"
+ )
+ bytes_read = 0
+ n_done = 0
+ t0 = time.perf_counter()
+ for start in range(0, n_rows, row_chunk):
+ end = min(start + row_chunk, n_rows)
+ block = np.asarray(arr[start:end, :])
+ bytes_read += int(block.nbytes)
+ n_done += 1
+ _progress("graphNormedCellBands", n_done - 1, n_blocks, bytes_read, t0)
+ del block
+ return {
+ "arrayPath": loc,
+ "nCells": n_rows,
+ "nFeatures": n_cols,
+ "rowChunk": row_chunk,
+ "nBlocks": n_done,
+ "bytesRead": bytes_read,
+ "dtype": str(arr.dtype),
+ }
+
+
+def run_io_baseline_body(
+ config: ProfilingConfig,
+ *,
+ nRows: int = 1_000_000,
+ resultLabel: str | None = None,
+ columnOnly: bool = False,
+) -> dict[str, Any]:
+ resources = config.resourcesFor("markHvgs")
+ os.environ.setdefault("R2_ENDPOINT", config.r2EndpointUrl)
+ store_uri = config.storeUri(nRows)
+ _log(
+ f"[job] runTag={config.runTag} store={store_uri} "
+ f"machine={resources.modalCpuRequest}c/{resources.modalMemoryLimitMb}MiB "
+ f"region={config.modalRegion}"
+ )
+
+ store_holder: dict[str, DataStore] = {}
+
+ def _open() -> dict[str, Any]:
+ store_holder["store"] = _open_store(store_uri, resources)
+ return {"storeUri": store_uri}
+
+ open_info = _measure("openStore", _open)
+ if open_info["status"] != "ok":
+ return {
+ "runTag": config.runTag,
+ "nRows": nRows,
+ "storeUri": store_uri,
+ "status": "error",
+ "patterns": [open_info],
+ }
+ store = store_holder["store"]
+ assay = store.get_assay(config.workflow.assayName)
+ cell_idx = np.asarray(
+ store.cells.active_index(config.workflow.cellKey), dtype=np.intp
+ )
+ feat_idx = np.asarray(
+ assay.feats.active_index(config.workflow.markerFeatureKey), dtype=np.intp
+ )
+ # mark_hvgs stores the column as ``{cell_key}__{hvg_key}``.
+ hvg_col = f"{config.workflow.cellKey}__{config.workflow.hvgKey}"
+ hvg_idx = np.asarray(assay.feats.active_index(hvg_col), dtype=np.intp)
+ if getattr(assay, "rawDataT", None) is not None:
+ marker_source = assay.rawDataT
+ feature_axis, cell_axis = 0, 1
+ else:
+ marker_source = assay.rawData._backing
+ feature_axis, cell_axis = 1, 0
+ marker_resources = ResourceBudget(
+ memoryBytes=resources.scarfMemoryBudget,
+ workers=resources.workers,
+ )
+ marker_plan = plan_feature_stream(
+ marker_source,
+ featureAxis=feature_axis,
+ cellAxis=cell_axis,
+ featureIndices=feat_idx,
+ cellIndices=cell_idx,
+ resources=marker_resources,
+ blockBytes=lambda width: max(
+ 1,
+ len(cell_idx) * width * max(1, int(np.dtype(marker_source.dtype).itemsize)),
+ ),
+ requestedBatchSize=config.workflow.markerGeneBatchSize,
+ )
+ marker_batch = max(
+ (len(block.indices) for block in marker_plan.blocks),
+ default=0,
+ )
+ raw_chunks = list(getattr(assay.rawData._backing, "chunks", ()) or ())
+ _log(
+ f"[setup] activeCells={len(cell_idx)} activeFeats={len(feat_idx)} "
+ f"hvgs={len(hvg_idx)} markerBatch={marker_batch} rawChunks={raw_chunks}"
+ )
+
+ results = [open_info]
+ results.append(
+ _measure(
+ "hvgTiles",
+ lambda: _stream_hvg_tiles(assay, cell_idx, feat_idx),
+ )
+ )
+ results.append(
+ _measure(
+ "markerBatches",
+ lambda: _stream_marker_batches(
+ assay,
+ cell_idx,
+ marker_plan,
+ ),
+ )
+ )
+ if not columnOnly:
+ results.append(
+ _measure(
+ "graphRawCellBands",
+ lambda: _stream_graph_raw_cell_bands(assay, cell_idx, hvg_idx),
+ )
+ )
+ results.append(
+ _measure(
+ "graphNormedCellBands",
+ lambda: _stream_graph_normed_cell_bands(
+ store, config.workflow.assayName
+ ),
+ )
+ )
+
+ total_seconds = sum(r["seconds"] for r in results if r["status"] == "ok")
+ summary = {
+ "runTag": config.runTag,
+ "nRows": nRows,
+ "storeUri": store_uri,
+ "modalCpu": resources.modalCpuRequest,
+ "modalMemoryMb": resources.modalMemoryLimitMb,
+ "modalRegion": config.modalRegion,
+ "columnOnly": columnOnly,
+ "markerGeneBatchSize": marker_batch,
+ "nActiveCells": int(len(cell_idx)),
+ "nActiveFeatures": int(len(feat_idx)),
+ "nHvgs": int(len(hvg_idx)),
+ "rawChunks": raw_chunks,
+ "totalSeconds": round(total_seconds, 3),
+ "patterns": results,
+ }
+ _log(f"[job] finished totalSeconds={total_seconds:.1f}")
+ suffix = f"-{resultLabel}" if resultLabel else ""
+ result_uri = (
+ f"{config.resultsUri.rstrip('/')}/io-baseline/{config.runTag}{suffix}.json"
+ )
+ try:
+ put_json(result_uri, summary)
+ _log(f"[job] wrote {result_uri}")
+ summary["resultUri"] = result_uri
+ except Exception as exc:
+ _log(f"[job] failed to write result JSON: {type(exc).__name__}: {exc}")
+ return summary
diff --git a/profiling/leiden_worker.py b/profiling/leiden_worker.py
new file mode 100644
index 00000000..6e1d1d33
--- /dev/null
+++ b/profiling/leiden_worker.py
@@ -0,0 +1,113 @@
+"""Run Leiden clustering in a child process so the parent container stays live.
+
+Modal's runner heartbeat interval is about 900 seconds and is not configurable.
+``leidenalg`` holds the GIL for the whole of a large partition, so a parent that
+called it inline would emit no heartbeat or progress until it returned.
+``profiling.stages`` polls this child every 30 seconds and warns at 1800 seconds
+without killing it. Keep this indirection instead of trying to extend the
+heartbeat threshold.
+"""
+
+import argparse
+import json
+import time
+from collections.abc import Sequence
+from pathlib import Path
+from typing import Any
+
+from scarf import configure_output
+
+from profiling.config import StageResources, WorkflowParameters
+from profiling.stages import _open_datastore
+
+configure_output(progress=False, timestamps=True)
+
+
+def _write_status(statusPath: Path, payload: dict[str, Any]) -> None:
+ statusPath.write_text(json.dumps(payload), encoding="utf-8")
+
+
+def run_leiden_worker(requestPath: Path) -> None:
+ request = json.loads(requestPath.read_text(encoding="utf-8"))
+ if not isinstance(request, dict):
+ raise ValueError("Leiden worker request must be a JSON object")
+
+ store_uri = str(request["storeUri"])
+ workflow = WorkflowParameters.model_validate(request["workflow"])
+ resources = StageResources.model_validate(request["resources"])
+ status_path = Path(str(request["statusPath"]))
+
+ print(
+ f"[leiden_worker] START backend=leidenalg store={store_uri}",
+ flush=True,
+ )
+ worker_started = time.perf_counter()
+ setup_started = worker_started
+ input_setup_seconds: float | None = None
+ operation_started: float | None = None
+ operation_seconds: float | None = None
+ try:
+ store = _open_datastore(
+ store_uri,
+ workflow,
+ resources,
+ initialize=False,
+ )
+ input_setup_seconds = time.perf_counter() - setup_started
+ print(
+ "[leiden_worker] datastore open; ENTER run_leiden_clustering",
+ flush=True,
+ )
+ operation_started = time.perf_counter()
+ store.run_leiden_clustering(
+ from_assay=workflow.assayName,
+ cell_key=workflow.cellKey,
+ feat_key=workflow.hvgKey,
+ resolution=workflow.leidenResolution,
+ label=workflow.leidenLabel,
+ random_seed=workflow.leidenSeed,
+ )
+ operation_seconds = time.perf_counter() - operation_started
+ del store
+ except BaseException as exc:
+ now = time.perf_counter()
+ if input_setup_seconds is None:
+ input_setup_seconds = now - setup_started
+ if operation_started is not None and operation_seconds is None:
+ operation_seconds = now - operation_started
+ error = f"{type(exc).__name__}: {exc}"
+ _write_status(
+ status_path,
+ {
+ "status": "error",
+ "error": error,
+ "inputSetupSeconds": input_setup_seconds,
+ "operationSeconds": operation_seconds,
+ "wholeWorkerSeconds": now - worker_started,
+ },
+ )
+ print(f"[leiden_worker] ERROR {error}", flush=True)
+ raise
+
+ _write_status(
+ status_path,
+ {
+ "status": "ok",
+ "error": None,
+ "inputSetupSeconds": input_setup_seconds,
+ "operationSeconds": operation_seconds,
+ "wholeWorkerSeconds": time.perf_counter() - worker_started,
+ },
+ )
+ print("[leiden_worker] DONE run_leiden_clustering", flush=True)
+
+
+def main(argv: Sequence[str] | None = None) -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--request", required=True, type=Path)
+ args = parser.parse_args(argv)
+ run_leiden_worker(args.request)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/profiling/metrics.py b/profiling/metrics.py
new file mode 100644
index 00000000..7ad9d310
--- /dev/null
+++ b/profiling/metrics.py
@@ -0,0 +1,1113 @@
+import math
+import os
+import shutil
+import threading
+import time
+from collections.abc import Callable, Iterable, Iterator
+from contextlib import contextmanager
+from dataclasses import dataclass
+from io import TextIOBase
+from pathlib import Path
+from types import TracebackType
+from typing import Literal, Self
+
+type PeakScope = Literal["operation", "containerLifetime", "unavailable"]
+type LimitValue = int | Literal["max"] | None
+type _ReadText = Callable[[Path], str]
+type _WriteText = Callable[[Path, str], None]
+type _ListPids = Callable[[Path], Iterable[int]]
+type _AffinityReader = Callable[[int], Iterable[int]]
+type _DiskUsageReader = Callable[[Path], object]
+
+
+@dataclass(frozen=True, slots=True)
+class DiskUsage:
+ totalBytes: int
+ freeBytes: int
+ usedBytes: int
+
+
+@dataclass(frozen=True, slots=True)
+class ResourceMeasurement:
+ sampleCount: int
+ sampleIntervalSeconds: float
+ operationBaselineBytes: int | None = None
+ operationPeakBytes: int | None = None
+ operationIncrementalPeakBytes: int | None = None
+ operationPeakSource: str | None = None
+ processTreeRssBaselineBytes: int | None = None
+ processTreeRssPeakBytes: int | None = None
+ processTreeRssIncrementalPeakBytes: int | None = None
+ processTreeRssAfterBytes: int | None = None
+ cgroupPath: str | None = None
+ cgroupMemoryCurrentBaselineBytes: int | None = None
+ cgroupMemoryCurrentPeakBytes: int | None = None
+ cgroupMemoryCurrentAfterBytes: int | None = None
+ cgroupMemoryPeakBytes: int | None = None
+ cgroupMemoryPeakScope: PeakScope = "unavailable"
+ memoryMaxBytes: LimitValue = None
+ memorySwapCurrentBeforeBytes: int | None = None
+ memorySwapCurrentAfterBytes: int | None = None
+ memorySwapCurrentPeakBytes: int | None = None
+ memorySwapMaxBytes: LimitValue = None
+ memoryEventsBefore: dict[str, int] | None = None
+ memoryEventsAfter: dict[str, int] | None = None
+ memoryEventsDelta: dict[str, int] | None = None
+ cpuQuotaMicros: LimitValue = None
+ cpuPeriodMicros: int | None = None
+ cpuQuotaCores: float | None = None
+ cpuAffinityCpus: tuple[int, ...] | None = None
+ cpuAffinityCount: int | None = None
+ cpuAffinitySource: str | None = None
+ ephemeralDiskPath: str | None = None
+ ephemeralDiskBefore: DiskUsage | None = None
+ ephemeralDiskAfter: DiskUsage | None = None
+ ephemeralDiskPeak: DiskUsage | None = None
+ samplingErrorCount: int = 0
+
+
+@dataclass(frozen=True, slots=True)
+class StageTimings:
+ inputSetupSeconds: float | None = None
+ measuredOperationSeconds: float | None = None
+ validationPersistenceSeconds: float | None = None
+ wholeFunctionSeconds: float | None = None
+
+
+@dataclass(frozen=True, slots=True)
+class _Sample:
+ processTreeRssBytes: int | None
+ cgroupMemoryCurrentBytes: int | None
+ memorySwapCurrentBytes: int | None
+ ephemeralDisk: DiskUsage | None
+
+
+@dataclass(frozen=True, slots=True)
+class _CpuSnapshot:
+ quotaMicros: LimitValue = None
+ periodMicros: int | None = None
+ quotaCores: float | None = None
+ affinityCpus: tuple[int, ...] | None = None
+ affinitySource: str | None = None
+
+
+def _default_read_text(path: Path) -> str:
+ return path.read_text(encoding="utf-8")
+
+
+def _default_write_text(path: Path, value: str) -> None:
+ path.write_text(value, encoding="utf-8")
+
+
+def _default_list_pids(procRoot: Path) -> list[int]:
+ return [
+ int(entry.name)
+ for entry in procRoot.iterdir()
+ if entry.name.isdigit() and entry.is_dir()
+ ]
+
+
+def _default_affinity_reader(pid: int) -> Iterable[int]:
+ reader = getattr(os, "sched_getaffinity", None)
+ if reader is None:
+ raise OSError("CPU affinity is unavailable")
+ return reader(pid)
+
+
+def _default_disk_usage_reader(path: Path) -> object:
+ return shutil.disk_usage(path)
+
+
+def _optional_read(path: Path, readText: _ReadText) -> str | None:
+ try:
+ value = readText(path)
+ except Exception:
+ return None
+ return value if isinstance(value, str) else None
+
+
+def _optional_write(path: Path, value: str, writeText: _WriteText) -> bool:
+ try:
+ writeText(path, value)
+ except Exception:
+ return False
+ return True
+
+
+def _nonnegative_int(value: str | None) -> int | None:
+ if value is None:
+ return None
+ try:
+ parsed = int(value.strip())
+ except (TypeError, ValueError):
+ return None
+ return parsed if parsed >= 0 else None
+
+
+def _limit_value(value: str | None) -> LimitValue:
+ if value is None:
+ return None
+ value = value.strip()
+ if value == "max":
+ return "max"
+ return _nonnegative_int(value)
+
+
+def _camel_case(value: str) -> str:
+ first, *remaining = value.split("_")
+ return first + "".join(part[:1].upper() + part[1:] for part in remaining)
+
+
+def parse_memory_events(value: str) -> dict[str, int]:
+ events: dict[str, int] = {}
+ for line in value.splitlines():
+ parts = line.split()
+ if len(parts) != 2:
+ continue
+ count = _nonnegative_int(parts[1])
+ if count is not None:
+ events[_camel_case(parts[0])] = count
+ return events
+
+
+def _memory_event_delta(
+ before: dict[str, int] | None,
+ after: dict[str, int] | None,
+) -> dict[str, int] | None:
+ if before is None or after is None:
+ return None
+ return {
+ key: after.get(key, 0) - before.get(key, 0)
+ for key in sorted(before.keys() | after.keys())
+ }
+
+
+def _parse_proc_status(value: str) -> tuple[int | None, int | None]:
+ parent_pid: int | None = None
+ rss_bytes: int | None = None
+ for line in value.splitlines():
+ key, separator, raw_value = line.partition(":")
+ if not separator:
+ continue
+ parts = raw_value.split()
+ if not parts:
+ continue
+ if key == "PPid":
+ parent_pid = _nonnegative_int(parts[0])
+ elif key == "VmRSS":
+ amount = _nonnegative_int(parts[0])
+ if amount is None:
+ continue
+ unit = parts[1].lower() if len(parts) > 1 else "b"
+ scale = {"b": 1, "kb": 1024, "mb": 1024**2}.get(unit)
+ if scale is not None:
+ rss_bytes = amount * scale
+ return parent_pid, rss_bytes
+
+
+def read_process_tree_rss_bytes(
+ rootPid: int,
+ *,
+ procRoot: str | Path = "/proc",
+ readText: _ReadText | None = None,
+ listPids: _ListPids | None = None,
+) -> int | None:
+ if isinstance(rootPid, bool) or rootPid <= 0:
+ return None
+ proc_root = Path(procRoot)
+ text_reader = _default_read_text if readText is None else readText
+ pid_reader = _default_list_pids if listPids is None else listPids
+ try:
+ pids = {int(pid) for pid in pid_reader(proc_root)}
+ except Exception:
+ return None
+ pids.add(rootPid)
+
+ records: dict[int, tuple[int | None, int | None]] = {}
+ children: dict[int, set[int]] = {}
+ for pid in pids:
+ status = _optional_read(proc_root / str(pid) / "status", text_reader)
+ if status is None:
+ continue
+ parent_pid, rss_bytes = _parse_proc_status(status)
+ records[pid] = (parent_pid, rss_bytes)
+ if parent_pid is not None:
+ children.setdefault(parent_pid, set()).add(pid)
+
+ tree_pids: set[int] = set()
+ pending = [rootPid]
+ while pending:
+ pid = pending.pop()
+ if pid in tree_pids:
+ continue
+ tree_pids.add(pid)
+ pending.extend(children.get(pid, ()))
+
+ rss_values = [
+ records[pid][1]
+ for pid in tree_pids
+ if pid in records and records[pid][1] is not None
+ ]
+ if not rss_values:
+ return None
+ return sum(rss_values)
+
+
+def _coerce_disk_usage(value: object) -> DiskUsage | None:
+ if isinstance(value, DiskUsage):
+ return value
+ try:
+ if all(hasattr(value, name) for name in ("total", "used", "free")):
+ total = getattr(value, "total")
+ used = getattr(value, "used")
+ free = getattr(value, "free")
+ else:
+ total, used, free = value
+ except (AttributeError, TypeError, ValueError):
+ return None
+ values = (total, free, used)
+ if any(isinstance(item, bool) or not isinstance(item, int) for item in values):
+ return None
+ if any(item < 0 for item in values):
+ return None
+ return DiskUsage(totalBytes=total, freeBytes=free, usedBytes=used)
+
+
+def _parse_cpuset(value: str | None) -> tuple[int, ...] | None:
+ if value is None:
+ return None
+ cpus: set[int] = set()
+ try:
+ for item in value.strip().split(","):
+ if not item:
+ continue
+ if "-" not in item:
+ cpus.add(int(item))
+ continue
+ start, end = (int(part) for part in item.split("-", 1))
+ if start > end:
+ return None
+ cpus.update(range(start, end + 1))
+ except ValueError:
+ return None
+ return tuple(sorted(cpus)) if cpus else None
+
+
+def _relative_cgroup_path(value: str) -> tuple[str, ...] | None:
+ for line in value.splitlines():
+ parts = line.split(":", 2)
+ if len(parts) != 3 or parts[0] != "0" or parts[1]:
+ continue
+ path_parts = tuple(part for part in parts[2].split("/") if part)
+ if any(part in {".", ".."} for part in path_parts):
+ return None
+ return path_parts
+ return None
+
+
+def _relative_cgroup_v1_memory_path(value: str) -> tuple[str, ...] | None:
+ for line in value.splitlines():
+ parts = line.split(":", 2)
+ if len(parts) != 3:
+ continue
+ controllers = {item.strip() for item in parts[1].split(",") if item.strip()}
+ if "memory" not in controllers:
+ continue
+ path_parts = tuple(part for part in parts[2].split("/") if part)
+ if any(part in {".", ".."} for part in path_parts):
+ return None
+ return ("memory",) + path_parts
+ return None
+
+
+_V1_MEMORY_FILES = {
+ "memory.current": "memory.usage_in_bytes",
+ "memory.peak": "memory.max_usage_in_bytes",
+ "memory.max": "memory.limit_in_bytes",
+ "memory.swap.current": "memory.memsw.usage_in_bytes",
+ "memory.swap.max": "memory.memsw.limit_in_bytes",
+}
+
+
+class ResourceSampler:
+ def __init__(
+ self,
+ *,
+ sampleIntervalSeconds: float = 0.1,
+ rootPid: int | None = None,
+ procRoot: str | Path = "/proc",
+ cgroupRoot: str | Path = "/sys/fs/cgroup",
+ cgroupPath: str | Path | None = None,
+ ephemeralDiskPath: str | Path | None = "/tmp",
+ resetCgroupPeak: bool = True,
+ readText: _ReadText | None = None,
+ writeText: _WriteText | None = None,
+ listPids: _ListPids | None = None,
+ affinityReader: _AffinityReader | None = None,
+ diskUsageReader: _DiskUsageReader | None = None,
+ ) -> None:
+ if isinstance(sampleIntervalSeconds, bool):
+ raise ValueError("sampleIntervalSeconds must be a positive finite number")
+ interval = float(sampleIntervalSeconds)
+ if not math.isfinite(interval) or interval <= 0:
+ raise ValueError("sampleIntervalSeconds must be a positive finite number")
+ selected_pid = os.getpid() if rootPid is None else rootPid
+ if isinstance(selected_pid, bool) or selected_pid <= 0:
+ raise ValueError("rootPid must be a positive integer")
+
+ self.sampleIntervalSeconds = interval
+ self.rootPid = selected_pid
+ self.procRoot = Path(procRoot)
+ self.cgroupRoot = Path(cgroupRoot)
+ self._explicitCgroupPath = None if cgroupPath is None else Path(cgroupPath)
+ self.ephemeralDiskPath = (
+ None if ephemeralDiskPath is None else Path(ephemeralDiskPath)
+ )
+ self.resetCgroupPeak = resetCgroupPeak
+ self._readText = _default_read_text if readText is None else readText
+ self._writeText = _default_write_text if writeText is None else writeText
+ self._listPids = _default_list_pids if listPids is None else listPids
+ self._affinityReader = (
+ _default_affinity_reader if affinityReader is None else affinityReader
+ )
+ self._diskUsageReader = (
+ _default_disk_usage_reader if diskUsageReader is None else diskUsageReader
+ )
+ self._usesDefaultTextIo = readText is None and writeText is None
+
+ self._stateLock = threading.Lock()
+ self._lifecycleLock = threading.Lock()
+ self._running = False
+ self._generation = 0
+ self._stopEvent: threading.Event | None = None
+ self._thread: threading.Thread | None = None
+ self._peakHandle: TextIOBase | None = None
+ self._reset_values()
+
+ def _reset_values(self) -> None:
+ self._result: ResourceMeasurement | None = None
+ self._sampleCount = 0
+ self._samplingErrorCount = 0
+ self._cgroupPath: Path | None = None
+ self._cgroupVersion: Literal[1, 2] | None = None
+ self._processTreeBaselineRssBytes: int | None = None
+ self._processTreePeakRssBytes: int | None = None
+ self._cgroupMemoryCurrentBaselineBytes: int | None = None
+ self._cgroupMemoryCurrentPeakBytes: int | None = None
+ self._memorySwapCurrentBeforeBytes: int | None = None
+ self._memorySwapCurrentPeakBytes: int | None = None
+ self._memoryEventsBefore: dict[str, int] | None = None
+ self._memoryMaxBytes: LimitValue = None
+ self._memorySwapMaxBytes: LimitValue = None
+ self._cgroupMemoryPeakInitialBytes: int | None = None
+ self._cgroupMemoryPeakScope: PeakScope = "unavailable"
+ self._cpuSnapshot = _CpuSnapshot()
+ self._ephemeralDiskBefore: DiskUsage | None = None
+ self._ephemeralDiskPeak: DiskUsage | None = None
+
+ @property
+ def isRunning(self) -> bool:
+ with self._stateLock:
+ return self._running
+
+ @property
+ def sampleCount(self) -> int:
+ with self._stateLock:
+ return self._sampleCount
+
+ @property
+ def result(self) -> ResourceMeasurement | None:
+ with self._stateLock:
+ return self._result
+
+ def _note_error(self) -> None:
+ with self._stateLock:
+ self._samplingErrorCount += 1
+
+ def _cgroup_file_name(self, name: str) -> str:
+ if self._cgroupVersion == 1:
+ return _V1_MEMORY_FILES.get(name, name)
+ return name
+
+ def _resolve_cgroup_path(self) -> Path | None:
+ if self._explicitCgroupPath is not None:
+ # Explicit paths are treated as cgroup v2 unless v1 memory files exist.
+ if (
+ _optional_read(
+ self._explicitCgroupPath / "memory.usage_in_bytes",
+ self._readText,
+ )
+ is not None
+ ):
+ self._cgroupVersion = 1
+ else:
+ self._cgroupVersion = 2
+ return self._explicitCgroupPath
+ candidates = (
+ self.procRoot / str(self.rootPid) / "cgroup",
+ self.procRoot / "self" / "cgroup",
+ )
+ for candidate in candidates:
+ value = _optional_read(candidate, self._readText)
+ if value is None:
+ continue
+ relative = _relative_cgroup_path(value)
+ if relative is not None:
+ path = self.cgroupRoot.joinpath(*relative)
+ if (
+ _optional_read(path / "memory.current", self._readText) is not None
+ or _optional_read(path / "cgroup.controllers", self._readText)
+ is not None
+ ):
+ self._cgroupVersion = 2
+ return path
+ relative_v1 = _relative_cgroup_v1_memory_path(value)
+ if relative_v1 is not None:
+ path = self.cgroupRoot.joinpath(*relative_v1)
+ if (
+ _optional_read(path / "memory.usage_in_bytes", self._readText)
+ is not None
+ ):
+ self._cgroupVersion = 1
+ return path
+ flat = self.cgroupRoot / "memory"
+ if (
+ _optional_read(flat / "memory.usage_in_bytes", self._readText)
+ is not None
+ ):
+ self._cgroupVersion = 1
+ return flat
+ probes_v2 = ("cgroup.controllers", "memory.current", "cpu.max")
+ if any(
+ _optional_read(self.cgroupRoot / name, self._readText) is not None
+ for name in probes_v2
+ ):
+ self._cgroupVersion = 2
+ return self.cgroupRoot
+ if (
+ _optional_read(
+ self.cgroupRoot / "memory" / "memory.usage_in_bytes",
+ self._readText,
+ )
+ is not None
+ ):
+ self._cgroupVersion = 1
+ return self.cgroupRoot / "memory"
+ return None
+
+ def _read_cgroup_int(self, name: str) -> int | None:
+ if self._cgroupPath is None:
+ return None
+ return _nonnegative_int(
+ _optional_read(
+ self._cgroupPath / self._cgroup_file_name(name),
+ self._readText,
+ )
+ )
+
+ def _read_cgroup_limit(self, name: str) -> LimitValue:
+ if self._cgroupPath is None:
+ return None
+ return _limit_value(
+ _optional_read(
+ self._cgroupPath / self._cgroup_file_name(name),
+ self._readText,
+ )
+ )
+
+ def _read_memory_events(self) -> dict[str, int] | None:
+ if self._cgroupPath is None or self._cgroupVersion == 1:
+ return None
+ value = _optional_read(self._cgroupPath / "memory.events", self._readText)
+ return None if value is None else parse_memory_events(value)
+
+ def _read_disk_usage(self) -> DiskUsage | None:
+ if self.ephemeralDiskPath is None:
+ return None
+ try:
+ value = self._diskUsageReader(self.ephemeralDiskPath)
+ except Exception:
+ return None
+ return _coerce_disk_usage(value)
+
+ def _read_cpu_snapshot(self) -> _CpuSnapshot:
+ quota: LimitValue = None
+ period: int | None = None
+ if self._cgroupPath is not None:
+ value = _optional_read(self._cgroupPath / "cpu.max", self._readText)
+ if value is not None:
+ parts = value.split()
+ if len(parts) >= 2:
+ quota = _limit_value(parts[0])
+ period = _nonnegative_int(parts[1])
+ if period == 0:
+ period = None
+ quota_cores = (
+ quota / period if isinstance(quota, int) and period is not None else None
+ )
+
+ affinity: tuple[int, ...] | None = None
+ source: str | None = None
+ try:
+ values = self._affinityReader(self.rootPid)
+ affinity = tuple(sorted({int(value) for value in values}))
+ source = "schedGetaffinity"
+ except Exception:
+ pass
+ if affinity is None and self._cgroupPath is not None:
+ for name in ("cpuset.cpus.effective", "cpuset.cpus"):
+ affinity = _parse_cpuset(
+ _optional_read(self._cgroupPath / name, self._readText)
+ )
+ if affinity is not None:
+ source = "cgroupCpuset"
+ break
+ return _CpuSnapshot(
+ quotaMicros=quota,
+ periodMicros=period,
+ quotaCores=quota_cores,
+ affinityCpus=affinity,
+ affinitySource=source,
+ )
+
+ @staticmethod
+ def _verified_peak_reset(
+ initialPeak: int,
+ resetPeak: int | None,
+ currentBefore: int | None,
+ currentAfter: int | None,
+ ) -> bool:
+ if resetPeak is None or resetPeak >= initialPeak:
+ return False
+ if resetPeak == 0:
+ return True
+ current = currentAfter if currentAfter is not None else currentBefore
+ return current is None or resetPeak >= current
+
+ @staticmethod
+ def _read_peak_handle(handle: TextIOBase) -> int | None:
+ try:
+ handle.seek(0)
+ return _nonnegative_int(handle.read())
+ except Exception:
+ return None
+
+ @staticmethod
+ def _write_peak_handle(handle: TextIOBase, value: str) -> bool:
+ try:
+ handle.seek(0)
+ handle.write(value)
+ handle.flush()
+ try:
+ handle.truncate()
+ except OSError:
+ pass
+ except Exception:
+ return False
+ return True
+
+ def _peak_reset_values(self, current: int | None) -> tuple[str, ...]:
+ if current in (None, 0):
+ return ("0",)
+ return ("0", str(current))
+
+ def _prepare_cgroup_peak(self) -> tuple[PeakScope, int | None]:
+ if self._cgroupPath is None:
+ return "unavailable", None
+ path = self._cgroupPath / self._cgroup_file_name("memory.peak")
+ initial_peak = _nonnegative_int(_optional_read(path, self._readText))
+ if initial_peak is None:
+ return "unavailable", None
+ if not self.resetCgroupPeak:
+ return "containerLifetime", initial_peak
+ # cgroup v1 max_usage_in_bytes is usually not resettable mid-run.
+ if self._cgroupVersion == 1:
+ return "containerLifetime", initial_peak
+ current_before = self._read_cgroup_int("memory.current")
+
+ if self._usesDefaultTextIo:
+ try:
+ handle = path.open("r+", encoding="utf-8")
+ except Exception:
+ return "containerLifetime", initial_peak
+ handle_initial = self._read_peak_handle(handle)
+ if handle_initial is not None:
+ initial_peak = handle_initial
+ for reset_value in self._peak_reset_values(current_before):
+ if not self._write_peak_handle(handle, reset_value):
+ continue
+ current_after = self._read_cgroup_int("memory.current")
+ reset_peak = self._read_peak_handle(handle)
+ if self._verified_peak_reset(
+ initial_peak,
+ reset_peak,
+ current_before,
+ current_after,
+ ):
+ self._peakHandle = handle
+ return "operation", initial_peak
+ try:
+ handle.close()
+ except Exception:
+ pass
+ return "containerLifetime", initial_peak
+
+ for reset_value in self._peak_reset_values(current_before):
+ if not _optional_write(path, reset_value, self._writeText):
+ continue
+ current_after = self._read_cgroup_int("memory.current")
+ reset_peak = _nonnegative_int(_optional_read(path, self._readText))
+ if self._verified_peak_reset(
+ initial_peak,
+ reset_peak,
+ current_before,
+ current_after,
+ ):
+ return "operation", initial_peak
+ return "containerLifetime", initial_peak
+
+ def _read_final_cgroup_peak(self) -> int | None:
+ if self._cgroupMemoryPeakScope == "unavailable":
+ return None
+ peak_name = self._cgroup_file_name("memory.peak")
+ if self._cgroupMemoryPeakScope == "operation":
+ if self._peakHandle is not None:
+ return self._read_peak_handle(self._peakHandle)
+ if self._cgroupPath is None:
+ return None
+ return _nonnegative_int(
+ _optional_read(
+ self._cgroupPath / peak_name,
+ self._readText,
+ )
+ )
+ if self._cgroupPath is None:
+ return self._cgroupMemoryPeakInitialBytes
+ final_peak = _nonnegative_int(
+ _optional_read(self._cgroupPath / peak_name, self._readText)
+ )
+ if final_peak is None:
+ return self._cgroupMemoryPeakInitialBytes
+ if self._cgroupMemoryPeakInitialBytes is None:
+ return final_peak
+ return max(final_peak, self._cgroupMemoryPeakInitialBytes)
+
+ def _close_peak_handle(self) -> None:
+ handle, self._peakHandle = self._peakHandle, None
+ if handle is None:
+ return
+ try:
+ handle.close()
+ except Exception:
+ pass
+
+ def _collect_sample(self) -> _Sample:
+ process_rss = read_process_tree_rss_bytes(
+ self.rootPid,
+ procRoot=self.procRoot,
+ readText=self._readText,
+ listPids=self._listPids,
+ )
+ return _Sample(
+ processTreeRssBytes=process_rss,
+ cgroupMemoryCurrentBytes=self._read_cgroup_int("memory.current"),
+ memorySwapCurrentBytes=self._read_cgroup_int("memory.swap.current"),
+ ephemeralDisk=self._read_disk_usage(),
+ )
+
+ def _sample_and_record(
+ self,
+ *,
+ isBaseline: bool,
+ force: bool,
+ generation: int,
+ ) -> _Sample:
+ try:
+ sample = self._collect_sample()
+ except Exception:
+ self._note_error()
+ sample = _Sample(None, None, None, None)
+ with self._stateLock:
+ if generation != self._generation:
+ return sample
+ if not force and not self._running:
+ return sample
+ self._sampleCount += 1
+ if isBaseline:
+ self._processTreeBaselineRssBytes = sample.processTreeRssBytes
+ self._cgroupMemoryCurrentBaselineBytes = sample.cgroupMemoryCurrentBytes
+ self._memorySwapCurrentBeforeBytes = sample.memorySwapCurrentBytes
+ self._ephemeralDiskBefore = sample.ephemeralDisk
+ if sample.processTreeRssBytes is not None:
+ current = self._processTreePeakRssBytes
+ self._processTreePeakRssBytes = max(
+ sample.processTreeRssBytes,
+ current if current is not None else sample.processTreeRssBytes,
+ )
+ if sample.cgroupMemoryCurrentBytes is not None:
+ current = self._cgroupMemoryCurrentPeakBytes
+ self._cgroupMemoryCurrentPeakBytes = max(
+ sample.cgroupMemoryCurrentBytes,
+ (
+ current
+ if current is not None
+ else sample.cgroupMemoryCurrentBytes
+ ),
+ )
+ if sample.memorySwapCurrentBytes is not None:
+ current = self._memorySwapCurrentPeakBytes
+ self._memorySwapCurrentPeakBytes = max(
+ sample.memorySwapCurrentBytes,
+ (current if current is not None else sample.memorySwapCurrentBytes),
+ )
+ if sample.ephemeralDisk is not None:
+ if (
+ self._ephemeralDiskPeak is None
+ or sample.ephemeralDisk.usedBytes
+ > self._ephemeralDiskPeak.usedBytes
+ ):
+ self._ephemeralDiskPeak = sample.ephemeralDisk
+ return sample
+
+ def _thread_main(
+ self,
+ stopEvent: threading.Event,
+ readyEvent: threading.Event,
+ generation: int,
+ ) -> None:
+ readyEvent.wait()
+ if stopEvent.is_set():
+ return
+ while not stopEvent.wait(self.sampleIntervalSeconds):
+ self._sample_and_record(
+ isBaseline=False,
+ force=False,
+ generation=generation,
+ )
+
+ def start(self) -> Self:
+ with self._lifecycleLock:
+ with self._stateLock:
+ if self._running:
+ return self
+ self._close_peak_handle()
+ with self._stateLock:
+ self._reset_values()
+ self._generation += 1
+ generation = self._generation
+ self._running = True
+ stop_event = threading.Event()
+ self._stopEvent = stop_event
+ self._thread = None
+
+ self._cgroupPath = self._resolve_cgroup_path()
+ self._memoryEventsBefore = self._read_memory_events()
+ self._memoryMaxBytes = self._read_cgroup_limit("memory.max")
+ self._memorySwapMaxBytes = self._read_cgroup_limit("memory.swap.max")
+ self._cpuSnapshot = self._read_cpu_snapshot()
+
+ ready_event = threading.Event()
+ thread = threading.Thread(
+ target=self._thread_main,
+ args=(stop_event, ready_event, generation),
+ name=f"resource-sampler-{self.rootPid}",
+ daemon=True,
+ )
+ try:
+ thread.start()
+ except Exception:
+ self._note_error()
+ else:
+ with self._stateLock:
+ self._thread = thread
+
+ peak_scope, initial_peak = self._prepare_cgroup_peak()
+ self._cgroupMemoryPeakScope = peak_scope
+ self._cgroupMemoryPeakInitialBytes = initial_peak
+ self._sample_and_record(
+ isBaseline=True,
+ force=True,
+ generation=generation,
+ )
+ ready_event.set()
+ return self
+
+ def sample(self) -> None:
+ with self._stateLock:
+ generation = self._generation
+ running = self._running
+ if not running:
+ return
+ self._sample_and_record(
+ isBaseline=False,
+ force=False,
+ generation=generation,
+ )
+
+ def _build_result(
+ self,
+ finalSample: _Sample,
+ memoryEventsAfter: dict[str, int] | None,
+ cgroupMemoryPeakBytes: int | None,
+ ) -> ResourceMeasurement:
+ with self._stateLock:
+ sample_count = self._sampleCount
+ error_count = self._samplingErrorCount
+ process_baseline = self._processTreeBaselineRssBytes
+ process_peak = self._processTreePeakRssBytes
+ cgroup_baseline = self._cgroupMemoryCurrentBaselineBytes
+ cgroup_current_peak = self._cgroupMemoryCurrentPeakBytes
+ swap_before = self._memorySwapCurrentBeforeBytes
+ swap_peak = self._memorySwapCurrentPeakBytes
+ events_before = (
+ None
+ if self._memoryEventsBefore is None
+ else dict(self._memoryEventsBefore)
+ )
+ disk_before = self._ephemeralDiskBefore
+ disk_peak = self._ephemeralDiskPeak
+ cpu = self._cpuSnapshot
+
+ operation_baseline = cgroup_baseline
+ operation_peak = cgroup_current_peak
+ operation_source: str | None = (
+ "cgroupMemoryCurrent" if cgroup_baseline is not None else None
+ )
+ if (
+ cgroup_baseline is not None
+ and self._cgroupMemoryPeakScope == "operation"
+ and cgroupMemoryPeakBytes is not None
+ and (operation_peak is None or cgroupMemoryPeakBytes >= operation_peak)
+ ):
+ operation_peak = cgroupMemoryPeakBytes
+ operation_source = "cgroupMemoryPeak"
+ if cgroup_baseline is None:
+ operation_baseline = process_baseline
+ operation_peak = process_peak
+ operation_source = (
+ "processTreeRss" if process_baseline is not None else None
+ )
+ incremental_peak = (
+ max(0, operation_peak - operation_baseline)
+ if operation_peak is not None and operation_baseline is not None
+ else None
+ )
+ process_incremental_peak = (
+ max(0, process_peak - process_baseline)
+ if process_peak is not None and process_baseline is not None
+ else None
+ )
+
+ events_after = None if memoryEventsAfter is None else dict(memoryEventsAfter)
+ affinity_count = None if cpu.affinityCpus is None else len(cpu.affinityCpus)
+ return ResourceMeasurement(
+ sampleCount=sample_count,
+ sampleIntervalSeconds=self.sampleIntervalSeconds,
+ operationBaselineBytes=operation_baseline,
+ operationPeakBytes=operation_peak,
+ operationIncrementalPeakBytes=incremental_peak,
+ operationPeakSource=operation_source,
+ processTreeRssBaselineBytes=process_baseline,
+ processTreeRssPeakBytes=process_peak,
+ processTreeRssIncrementalPeakBytes=process_incremental_peak,
+ processTreeRssAfterBytes=finalSample.processTreeRssBytes,
+ cgroupPath=(None if self._cgroupPath is None else str(self._cgroupPath)),
+ cgroupMemoryCurrentBaselineBytes=cgroup_baseline,
+ cgroupMemoryCurrentPeakBytes=cgroup_current_peak,
+ cgroupMemoryCurrentAfterBytes=(finalSample.cgroupMemoryCurrentBytes),
+ cgroupMemoryPeakBytes=cgroupMemoryPeakBytes,
+ cgroupMemoryPeakScope=self._cgroupMemoryPeakScope,
+ memoryMaxBytes=self._memoryMaxBytes,
+ memorySwapCurrentBeforeBytes=swap_before,
+ memorySwapCurrentAfterBytes=finalSample.memorySwapCurrentBytes,
+ memorySwapCurrentPeakBytes=swap_peak,
+ memorySwapMaxBytes=self._memorySwapMaxBytes,
+ memoryEventsBefore=events_before,
+ memoryEventsAfter=events_after,
+ memoryEventsDelta=_memory_event_delta(
+ events_before,
+ events_after,
+ ),
+ cpuQuotaMicros=cpu.quotaMicros,
+ cpuPeriodMicros=cpu.periodMicros,
+ cpuQuotaCores=cpu.quotaCores,
+ cpuAffinityCpus=cpu.affinityCpus,
+ cpuAffinityCount=affinity_count,
+ cpuAffinitySource=cpu.affinitySource,
+ ephemeralDiskPath=(
+ None if self.ephemeralDiskPath is None else str(self.ephemeralDiskPath)
+ ),
+ ephemeralDiskBefore=disk_before,
+ ephemeralDiskAfter=finalSample.ephemeralDisk,
+ ephemeralDiskPeak=disk_peak,
+ samplingErrorCount=error_count,
+ )
+
+ def stop(self) -> ResourceMeasurement:
+ with self._lifecycleLock:
+ with self._stateLock:
+ if not self._running:
+ if self._result is not None:
+ return self._result
+ result = ResourceMeasurement(
+ sampleCount=0,
+ sampleIntervalSeconds=self.sampleIntervalSeconds,
+ ephemeralDiskPath=(
+ None
+ if self.ephemeralDiskPath is None
+ else str(self.ephemeralDiskPath)
+ ),
+ )
+ self._result = result
+ return result
+ self._running = False
+ generation = self._generation
+ stop_event = self._stopEvent
+ thread = self._thread
+
+ if stop_event is not None:
+ stop_event.set()
+ if thread is not None and thread is not threading.current_thread():
+ try:
+ thread.join(
+ timeout=max(
+ 0.25,
+ min(2.0, self.sampleIntervalSeconds * 2),
+ )
+ )
+ except RuntimeError:
+ self._note_error()
+
+ final_sample = self._sample_and_record(
+ isBaseline=False,
+ force=True,
+ generation=generation,
+ )
+ memory_events_after = self._read_memory_events()
+ cgroup_peak = self._read_final_cgroup_peak()
+ self._close_peak_handle()
+ result = self._build_result(
+ final_sample,
+ memory_events_after,
+ cgroup_peak,
+ )
+ with self._stateLock:
+ self._result = result
+ self._stopEvent = None
+ self._thread = None
+ return result
+
+ def __enter__(self) -> Self:
+ return self.start()
+
+ def __exit__(
+ self,
+ excType: type[BaseException] | None,
+ exc: BaseException | None,
+ traceback: TracebackType | None,
+ ) -> bool:
+ try:
+ self.stop()
+ except Exception:
+ pass
+ return False
+
+
+class StageTimer:
+ def __init__(self, *, clock: Callable[[], float] = time.perf_counter) -> None:
+ self._clock = clock
+ self._active = False
+ self._wholeStarted: float | None = None
+ self._wholeElapsed: float | None = None
+ self._inputSetupElapsed = 0.0
+ self._operationElapsed = 0.0
+ self._validationPersistenceElapsed = 0.0
+ self._hasInputSetup = False
+ self._hasOperation = False
+ self._hasValidationPersistence = False
+
+ def __enter__(self) -> Self:
+ if self._active:
+ raise RuntimeError("StageTimer is already active")
+ self._active = True
+ self._wholeStarted = self._clock()
+ self._wholeElapsed = None
+ self._inputSetupElapsed = 0.0
+ self._operationElapsed = 0.0
+ self._validationPersistenceElapsed = 0.0
+ self._hasInputSetup = False
+ self._hasOperation = False
+ self._hasValidationPersistence = False
+ return self
+
+ def __exit__(
+ self,
+ excType: type[BaseException] | None,
+ exc: BaseException | None,
+ traceback: TracebackType | None,
+ ) -> bool:
+ if self._wholeStarted is not None:
+ self._wholeElapsed = max(0.0, self._clock() - self._wholeStarted)
+ self._wholeStarted = None
+ self._active = False
+ return False
+
+ @contextmanager
+ def inputSetup(self) -> Iterator[None]:
+ started = self._clock()
+ try:
+ yield
+ finally:
+ self._inputSetupElapsed += max(0.0, self._clock() - started)
+ self._hasInputSetup = True
+
+ @contextmanager
+ def operation(self) -> Iterator[None]:
+ started = self._clock()
+ try:
+ yield
+ finally:
+ self._operationElapsed += max(0.0, self._clock() - started)
+ self._hasOperation = True
+
+ @contextmanager
+ def validationPersistence(self) -> Iterator[None]:
+ started = self._clock()
+ try:
+ yield
+ finally:
+ self._validationPersistenceElapsed += max(
+ 0.0,
+ self._clock() - started,
+ )
+ self._hasValidationPersistence = True
+
+ @property
+ def result(self) -> StageTimings:
+ return StageTimings(
+ inputSetupSeconds=(
+ self._inputSetupElapsed if self._hasInputSetup else None
+ ),
+ measuredOperationSeconds=(
+ self._operationElapsed if self._hasOperation else None
+ ),
+ validationPersistenceSeconds=(
+ self._validationPersistenceElapsed
+ if self._hasValidationPersistence
+ else None
+ ),
+ wholeFunctionSeconds=self._wholeElapsed,
+ )
+
+
+__all__ = [
+ "DiskUsage",
+ "LimitValue",
+ "PeakScope",
+ "ResourceMeasurement",
+ "ResourceSampler",
+ "StageTimer",
+ "StageTimings",
+ "parse_memory_events",
+ "read_process_tree_rss_bytes",
+]
diff --git a/profiling/modal_app.py b/profiling/modal_app.py
new file mode 100644
index 00000000..1b692b90
--- /dev/null
+++ b/profiling/modal_app.py
@@ -0,0 +1,1063 @@
+"""Modal entrypoints for one-shot Scarf profiling.
+
+Deploy once (you run this), then trigger jobs that keep running if your laptop
+disconnects:
+
+ uv run --group profiling modal deploy --env scarf_profiling -m profiling.modal_app
+
+ uv run --group profiling modal run --env scarf_profiling \\
+ -m profiling.modal_app -- prepare --config profiling/config.toml
+ uv run --group profiling modal run --env scarf_profiling \\
+ -m profiling.modal_app -- run-e2e --config profiling/config.toml --size 1000000
+
+prepare / run / run-all / run-local / run-e2e spawn and return immediately.
+run-all fans out one size pipeline per container (stages stay sequential on R2).
+run-local runs the full funnel in one container on ephemeral-disk Zarr (fast_local).
+run-e2e runs the current core funnel in one container while keeping Zarr on R2.
+Watch progress with:
+ uv run --group profiling modal app logs scarf-profiling --env scarf_profiling
+"""
+
+import argparse
+import os
+import time
+from pathlib import Path
+from typing import Any
+
+import modal
+
+from profiling.config import (
+ CORE_STAGE_ORDER,
+ MAX_TIMEOUT_SECONDS,
+ ProfilingConfig,
+ StageName,
+ StageResources,
+ load_profiling_config,
+)
+from profiling.datasets import (
+ SOURCE_SPEC,
+ download_source,
+ prepare_fixture_datasets,
+ prepare_local_datasets,
+ sha256_file,
+)
+from profiling.modal_image import COMMON_FUNCTION_OPTIONS, app
+from profiling.modal_resources import (
+ BASE_EPHEMERAL_DISK_MB,
+ modal_function_options,
+ orchestrator_function_options,
+ validate_modal_environment,
+)
+from profiling.io_baseline import run_io_baseline_body
+from profiling.r2 import (
+ download_file,
+ object_exists,
+ object_size,
+ put_json_if_absent,
+ upload_file,
+)
+from profiling.results import (
+ result_exists,
+ write_funnel_result,
+ write_result,
+)
+from profiling.spawn_wait import (
+ DEFAULT_GRACE_SECONDS,
+ DEFAULT_STAGE_SPAWN_ATTEMPTS,
+ await_function_call,
+ await_many_function_calls,
+ await_stage_result,
+)
+from profiling.metrics import ResourceSampler
+from profiling.stages import (
+ run_stage,
+ summarize_resource_measurement,
+)
+
+_WORK = Path("/tmp/scarf-profiling")
+
+
+def _e2e_conflicting_uris(
+ config: ProfilingConfig,
+ nRows: int,
+) -> list[str]:
+ store_uri = config.storeUri(nRows).rstrip("/")
+ candidates = [
+ f"{store_uri}/zarr.json",
+ f"{store_uri}/.zgroup",
+ config.e2eClaimUri(),
+ config.funnelResultUri(nRows),
+ *(config.resultUri(nRows, stage) for stage in CORE_STAGE_ORDER),
+ ]
+ return [uri for uri in candidates if object_exists(uri)]
+
+
+def _e2e_function_options(config: ProfilingConfig) -> dict[str, Any]:
+ envelope = _e2e_resource_envelope(config)
+ resources = _e2e_resources(config)
+ peak = max(
+ resources,
+ key=lambda item: (
+ item.modalMemoryLimitMb,
+ item.modalCpuLimit,
+ item.timeoutSeconds,
+ ),
+ )
+ options = modal_function_options(
+ config,
+ peak,
+ maxContainers=1,
+ retries=0,
+ )
+ options["memory"] = (
+ envelope["modalMemoryRequestMb"],
+ envelope["modalMemoryLimitMb"],
+ )
+ options["cpu"] = (
+ envelope["modalCpuRequest"],
+ envelope["modalCpuLimit"],
+ )
+ options["timeout"] = MAX_TIMEOUT_SECONDS
+ return options
+
+
+def _e2e_resources(config: ProfilingConfig) -> list[StageResources]:
+ missing_resources = [
+ stage for stage in CORE_STAGE_ORDER if stage not in config.stageResources
+ ]
+ if missing_resources:
+ raise ValueError(
+ "run-e2e is missing stageResources for: " + ", ".join(missing_resources)
+ )
+ return [config.resourcesFor(stage) for stage in CORE_STAGE_ORDER]
+
+
+def _e2e_resource_envelope(config: ProfilingConfig) -> dict[str, int | float]:
+ resources = _e2e_resources(config)
+ requested_ephemeral_disk = max(item.ephemeralDiskMb for item in resources)
+ if requested_ephemeral_disk > BASE_EPHEMERAL_DISK_MB:
+ raise ValueError(
+ "run-e2e cannot apply ephemeralDiskMb above "
+ f"{BASE_EPHEMERAL_DISK_MB}; Modal does not allow a dynamic "
+ "ephemeral_disk override"
+ )
+ return {
+ "modalMemoryRequestMb": max(item.modalMemoryRequestMb for item in resources),
+ "modalMemoryLimitMb": max(item.modalMemoryLimitMb for item in resources),
+ "modalCpuRequest": max(item.modalCpuRequest for item in resources),
+ "modalCpuLimit": max(item.modalCpuLimit for item in resources),
+ "ephemeralDiskMb": BASE_EPHEMERAL_DISK_MB,
+ "timeoutSeconds": MAX_TIMEOUT_SECONDS,
+ }
+
+
+@app.function(
+ **COMMON_FUNCTION_OPTIONS,
+ timeout=86_400,
+ memory=196_608,
+ cpu=8.0,
+ ephemeral_disk=BASE_EPHEMERAL_DISK_MB,
+)
+def prepare_datasets(configDict: dict[str, Any]) -> dict[str, Any]:
+ config = ProfilingConfig.model_validate(configDict)
+ os.environ.setdefault("R2_ENDPOINT", config.r2EndpointUrl)
+ work = _WORK / "prepare"
+ work.mkdir(parents=True, exist_ok=True)
+ source_path = work / "source.h5ad"
+ source_uri = config.sourceUri()
+ source_origin = "local-cache"
+ if not source_path.is_file():
+ if object_exists(source_uri):
+ download_file(source_uri, source_path)
+ source_origin = "r2-cache"
+ else:
+ download_source(
+ source_path,
+ url=SOURCE_SPEC.url,
+ expectedBytes=SOURCE_SPEC.sourceBytes,
+ )
+ upload_file(source_path, source_uri)
+ source_origin = "cellxgene+r2-upload"
+
+ uploaded: list[dict[str, Any]] = []
+ skipped: list[dict[str, Any]] = []
+ # Fixture uploads are tiny; real nested samples are much larger.
+ minimum_real_bytes = 5_000_000
+
+ pending_sizes: list[int] = []
+ for n_rows in config.targetSizes:
+ uri = config.datasetUri(n_rows)
+ existing = object_size(uri)
+ if existing is not None and existing >= minimum_real_bytes:
+ skipped.append(
+ {
+ "nRows": n_rows,
+ "uri": uri,
+ "fileBytes": existing,
+ "status": "skipped-existing",
+ }
+ )
+ continue
+ pending_sizes.append(n_rows)
+
+ def _upload_artifact(artifact: Any) -> None:
+ uri = config.datasetUri(artifact.targetRows)
+ upload_file(artifact.localPath, uri)
+ uploaded.append(
+ {
+ "nRows": artifact.targetRows,
+ "uri": uri,
+ "fileBytes": artifact.fileBytes,
+ "nnz": artifact.nnz,
+ "status": "uploaded",
+ }
+ )
+
+ source_sha256 = None
+ if pending_sizes:
+ prepared = prepare_local_datasets(
+ source_path,
+ work / "subsets",
+ targetRows=tuple(pending_sizes),
+ seed=config.samplingSeed,
+ spec=SOURCE_SPEC,
+ onArtifact=_upload_artifact,
+ )
+ source_sha256 = prepared.sourceSha256
+ elif source_path.is_file():
+ source_sha256 = sha256_file(source_path)
+
+ return {
+ "uploaded": uploaded,
+ "skipped": skipped,
+ "sourceSha256": source_sha256,
+ "sourceOrigin": source_origin,
+ "sourceUri": source_uri,
+ }
+
+
+@app.function(
+ **COMMON_FUNCTION_OPTIONS,
+ timeout=3_600,
+ memory=8_192,
+ cpu=2.0,
+ ephemeral_disk=BASE_EPHEMERAL_DISK_MB,
+)
+def prepare_fixture_datasets_job(
+ configDict: dict[str, Any],
+ sizes: list[int] | None = None,
+ nColumns: int = 500,
+) -> dict[str, Any]:
+ """Upload tiny synthetic H5ADs so stage jobs can be tested without Cellxgene."""
+ config = ProfilingConfig.model_validate(configDict)
+ os.environ.setdefault("R2_ENDPOINT", config.r2EndpointUrl)
+ selected = tuple(sizes) if sizes else (10_000,)
+ for size in selected:
+ if size not in config.targetSizes:
+ raise ValueError(
+ f"fixture size {size} is not in config.targetSizes; "
+ "add it to config or choose an existing size"
+ )
+ work = _WORK / "fixture"
+ if work.exists():
+ for path in sorted(work.rglob("*"), reverse=True):
+ if path.is_file():
+ path.unlink()
+ elif path.is_dir():
+ path.rmdir()
+ work.mkdir(parents=True, exist_ok=True)
+ uploaded: list[dict[str, Any]] = []
+
+ def _upload_artifact(artifact: Any) -> None:
+ uri = config.datasetUri(artifact.targetRows)
+ upload_file(artifact.localPath, uri)
+ uploaded.append(
+ {
+ "nRows": artifact.targetRows,
+ "uri": uri,
+ "fileBytes": artifact.fileBytes,
+ "nnz": artifact.nnz,
+ "nColumns": artifact.nColumns,
+ }
+ )
+
+ prepare_fixture_datasets(
+ work,
+ targetRows=selected,
+ nColumns=nColumns,
+ seed=config.samplingSeed,
+ onArtifact=_upload_artifact,
+ )
+ return {"uploaded": uploaded, "kind": "fixture"}
+
+
+@app.function(
+ **COMMON_FUNCTION_OPTIONS,
+ timeout=86_400,
+ memory=(65_536, 65_536),
+ cpu=(8.0, 8.0),
+ ephemeral_disk=BASE_EPHEMERAL_DISK_MB,
+)
+def io_baseline_job(
+ configDict: dict[str, Any],
+ nRows: int = 1_000_000,
+ resultLabel: str | None = None,
+ columnOnly: bool = False,
+) -> dict[str, Any]:
+ """No-compute R2 stream of HVG, marker, and graph read patterns."""
+ config = ProfilingConfig.model_validate(configDict)
+ os.environ.setdefault("R2_ENDPOINT", config.r2EndpointUrl)
+ return run_io_baseline_body(
+ config,
+ nRows=nRows,
+ resultLabel=resultLabel,
+ columnOnly=columnOnly,
+ )
+
+
+@app.function(
+ **COMMON_FUNCTION_OPTIONS,
+ timeout=86_400,
+ memory=65_536,
+ cpu=8.0,
+ ephemeral_disk=BASE_EPHEMERAL_DISK_MB,
+)
+def run_stage_job(
+ configDict: dict[str, Any],
+ nRows: int,
+ stage: StageName,
+ force: bool = False,
+) -> dict[str, Any]:
+ config = ProfilingConfig.model_validate(configDict)
+ resources = config.resourcesFor(stage)
+ os.environ.setdefault("R2_ENDPOINT", config.r2EndpointUrl)
+ if result_exists(config, nRows, stage) and not force:
+ return {
+ "nRows": nRows,
+ "stage": stage,
+ "status": "skipped",
+ "resultUri": config.resultUri(nRows, stage),
+ }
+
+ work = _WORK / f"{nRows}-{stage}"
+ if work.exists():
+ for path in sorted(work.rglob("*"), reverse=True):
+ if path.is_file():
+ path.unlink()
+ elif path.is_dir():
+ path.rmdir()
+ work.mkdir(parents=True, exist_ok=True)
+
+ local_h5ad: Path | None = None
+ if stage == "createStore":
+ local_h5ad = work / f"{nRows}.h5ad"
+ download_file(config.datasetUri(nRows), local_h5ad)
+
+ result = run_stage(
+ stage,
+ nRows=nRows,
+ storeUri=config.storeUri(nRows),
+ workflow=config.workflow,
+ resources=resources,
+ localH5adPath=local_h5ad,
+ storageLayout=config.storageLayout,
+ workDir=work,
+ invalidateCache=force,
+ )
+ write_result(config, result)
+ return result.to_json()
+
+
+@app.function(
+ **COMMON_FUNCTION_OPTIONS,
+ timeout=86_400,
+ memory=32_768,
+ cpu=8.0,
+ ephemeral_disk=BASE_EPHEMERAL_DISK_MB,
+)
+def run_local_funnel_job(
+ configDict: dict[str, Any],
+ nRows: int,
+ stages: list[StageName] | None = None,
+) -> dict[str, Any]:
+ """Full funnel in one container: H5AD + Zarr on ephemeral disk (fast_local).
+
+ Downloads the prepared H5AD from R2 once, writes the store under /tmp, runs
+ stages in-process, and still persists each stage result JSON to R2.
+ """
+ config = ProfilingConfig.model_validate(configDict)
+ os.environ.setdefault("R2_ENDPOINT", config.r2EndpointUrl)
+ os.environ["SCARF_ZARR_PROFILE"] = "fast_local"
+ if nRows not in config.targetSizes:
+ raise ValueError(f"size {nRows} is not in config.targetSizes")
+ selected_stages = tuple(stages) if stages else config.effectiveStages
+
+ work = _WORK / f"local-{config.runTag or 'untagged'}-{nRows}"
+ if work.exists():
+ for path in sorted(work.rglob("*"), reverse=True):
+ if path.is_file():
+ path.unlink()
+ elif path.is_dir():
+ path.rmdir()
+ work.mkdir(parents=True, exist_ok=True)
+
+ local_h5ad = work / f"{nRows}.h5ad"
+ local_store = work / f"{nRows}.zarr"
+ store_uri = str(local_store)
+
+ if "createStore" in selected_stages and not result_exists(
+ config, nRows, "createStore"
+ ):
+ print(f"downloading dataset to {local_h5ad}", flush=True)
+ download_file(config.datasetUri(nRows), local_h5ad)
+
+ outcomes: list[dict[str, Any]] = []
+ for stage in selected_stages:
+ if result_exists(config, nRows, stage):
+ outcomes.append(
+ {
+ "nRows": nRows,
+ "stage": stage,
+ "status": "skipped",
+ "resultUri": config.resultUri(nRows, stage),
+ "storeBackend": "local",
+ }
+ )
+ continue
+ if stage == "createStore" and not local_h5ad.is_file():
+ raise FileNotFoundError(
+ f"createStore needs {local_h5ad}; download failed or was skipped"
+ )
+ if stage != "createStore" and not local_store.exists():
+ raise FileNotFoundError(
+ f"stage {stage} needs local store at {local_store}; "
+ "include createStore or pre-seed the ephemeral workdir"
+ )
+ resources = config.resourcesFor(stage)
+ print(f"local funnel stage start: {stage}", flush=True)
+ result = run_stage(
+ stage,
+ nRows=nRows,
+ storeUri=store_uri,
+ workflow=config.workflow,
+ resources=resources,
+ localH5adPath=local_h5ad if stage == "createStore" else None,
+ storageLayout=config.storageLayout,
+ workDir=work / stage,
+ )
+ write_result(config, result)
+ payload = result.to_json()
+ payload["storeBackend"] = "local"
+ outcomes.append(payload)
+ print(
+ f"local funnel stage done: {stage} status={result.status} "
+ f"seconds={result.seconds}",
+ flush=True,
+ )
+ if result.status == "error":
+ return {
+ "nRows": nRows,
+ "stopped": True,
+ "storeBackend": "local",
+ "storeUri": store_uri,
+ "failed": payload,
+ "outcomes": outcomes,
+ }
+
+ return {
+ "nRows": nRows,
+ "stopped": False,
+ "storeBackend": "local",
+ "storeUri": store_uri,
+ "outcomes": outcomes,
+ }
+
+
+def run_e2e_funnel_body(
+ configDict: dict[str, Any],
+ nRows: int,
+) -> dict[str, Any]:
+ """Run the graph-construction core once in one container against a fresh R2 store."""
+ config = ProfilingConfig.model_validate(configDict)
+ os.environ.setdefault("R2_ENDPOINT", config.r2EndpointUrl)
+ if nRows not in config.targetSizes:
+ raise ValueError(f"size {nRows} is not in config.targetSizes")
+ if not config.runTag.strip():
+ raise ValueError("run-e2e requires a non-empty runTag")
+ resource_envelope = _e2e_resource_envelope(config)
+ conflicts = _e2e_conflicting_uris(config, nRows)
+ if conflicts:
+ raise FileExistsError(
+ "run-e2e requires a fresh runTag; existing R2 objects: "
+ + ", ".join(conflicts)
+ )
+ claimed = put_json_if_absent(
+ config.e2eClaimUri(),
+ {
+ "runTag": config.runTag,
+ "nRows": nRows,
+ "status": "claimed",
+ "storeUri": config.storeUri(nRows),
+ },
+ )
+ if not claimed:
+ raise FileExistsError(
+ f"run-e2e runTag was claimed concurrently: {config.runTag}"
+ )
+
+ work = _WORK / f"e2e-{config.runTag}-{nRows}"
+ work.mkdir(parents=True, exist_ok=False)
+ local_h5ad = work / f"{nRows}.h5ad"
+ store_uri = config.storeUri(nRows)
+
+ sampler = ResourceSampler()
+ sampler.start()
+ started = time.perf_counter()
+ download_seconds: float | None = None
+ outcomes: list[dict[str, Any]] = []
+ completed_stages: list[StageName] = []
+ status = "ok"
+ error: str | None = None
+ failed_stage: StageName | None = None
+ try:
+ download_started = time.perf_counter()
+ print(f"e2e dataset download start: {config.datasetUri(nRows)}", flush=True)
+ download_file(config.datasetUri(nRows), local_h5ad)
+ download_seconds = time.perf_counter() - download_started
+ print(
+ f"e2e dataset download done: seconds={download_seconds:.1f}",
+ flush=True,
+ )
+
+ for stage in CORE_STAGE_ORDER:
+ failed_stage = stage
+ resources = config.resourcesFor(stage)
+ stage_work = work / stage
+ stage_work.mkdir(parents=True, exist_ok=True)
+ print(f"e2e stage start: {stage}", flush=True)
+ result = run_stage(
+ stage,
+ nRows=nRows,
+ storeUri=store_uri,
+ workflow=config.workflow,
+ resources=resources,
+ localH5adPath=local_h5ad if stage == "createStore" else None,
+ storageLayout=config.storageLayout,
+ workDir=stage_work,
+ containerMemoryMb=int(resource_envelope["modalMemoryLimitMb"]),
+ containerCpuRequest=float(resource_envelope["modalCpuRequest"]),
+ containerCpuLimit=float(resource_envelope["modalCpuLimit"]),
+ resetCgroupPeak=False,
+ )
+ result_uri = write_result(config, result)
+ payload = result.to_json()
+ payload["resultUri"] = result_uri
+ payload["storeBackend"] = "r2"
+ outcomes.append(payload)
+ print(
+ f"e2e stage done: {stage} status={result.status} "
+ f"seconds={result.seconds}",
+ flush=True,
+ )
+ if result.status != "ok":
+ status = "error"
+ error = result.error or f"{stage} failed"
+ break
+ completed_stages.append(stage)
+ else:
+ failed_stage = None
+ except Exception as exc: # noqa: BLE001 - persist a durable failure summary
+ status = "error"
+ error = f"{type(exc).__name__}: {exc}"
+ finally:
+ measurement = sampler.stop()
+
+ summary: dict[str, Any] = {
+ "runTag": config.runTag,
+ "nRows": nRows,
+ "status": status,
+ "stopped": status != "ok",
+ "error": error,
+ "failedStage": failed_stage,
+ "storeBackend": "r2",
+ "storeUri": store_uri,
+ "datasetUri": config.datasetUri(nRows),
+ "datasetDownloadSeconds": download_seconds,
+ "wholeFunctionSeconds": time.perf_counter() - started,
+ "modalResources": resource_envelope,
+ "stageOrder": list(CORE_STAGE_ORDER),
+ "completedStages": completed_stages,
+ "outcomes": outcomes,
+ "claimUri": config.e2eClaimUri(),
+ "funnelResultUri": config.funnelResultUri(nRows),
+ **summarize_resource_measurement(measurement),
+ }
+ write_funnel_result(config, nRows, summary)
+ return summary
+
+
+@app.function(
+ **COMMON_FUNCTION_OPTIONS,
+ timeout=MAX_TIMEOUT_SECONDS,
+ memory=32_768,
+ cpu=8.0,
+ ephemeral_disk=BASE_EPHEMERAL_DISK_MB,
+)
+def run_e2e_funnel_job(
+ configDict: dict[str, Any],
+ nRows: int,
+) -> dict[str, Any]:
+ return run_e2e_funnel_body(configDict, nRows)
+
+
+@app.function(
+ **COMMON_FUNCTION_OPTIONS,
+ timeout=86_400,
+ memory=2048,
+ cpu=1.0,
+ ephemeral_disk=BASE_EPHEMERAL_DISK_MB,
+)
+def run_size_jobs(
+ configDict: dict[str, Any],
+ nRows: int,
+ stages: list[StageName] | None = None,
+) -> dict[str, Any]:
+ """Run stages sequentially for one size (skip if result JSON exists)."""
+ config = ProfilingConfig.model_validate(configDict)
+ os.environ.setdefault("R2_ENDPOINT", config.r2EndpointUrl)
+ if nRows not in config.targetSizes:
+ raise ValueError(f"size {nRows} is not in config.targetSizes")
+ selected_stages = tuple(stages) if stages else config.effectiveStages
+ parallel_sizes = max(1, len(config.targetSizes))
+ outcomes: list[dict[str, Any]] = []
+
+ for stage in selected_stages:
+ if result_exists(config, nRows, stage):
+ outcomes.append(
+ {
+ "nRows": nRows,
+ "stage": stage,
+ "status": "skipped",
+ "resultUri": config.resultUri(nRows, stage),
+ }
+ )
+ continue
+ resources = config.resourcesFor(stage)
+ options = (
+ modal_function_options(
+ config,
+ resources,
+ maxContainers=parallel_sizes,
+ retries=0,
+ )
+ if stage == "writeCountsT"
+ else modal_function_options(
+ config,
+ resources,
+ maxContainers=parallel_sizes,
+ )
+ )
+ deadline_seconds = float(resources.timeoutSeconds) + DEFAULT_GRACE_SECONDS
+ result: dict[str, Any] | None = None
+ last_error: BaseException | None = None
+ spawn_attempts = 1 if stage == "writeCountsT" else DEFAULT_STAGE_SPAWN_ATTEMPTS
+ for attempt in range(1, spawn_attempts + 1):
+ if result_exists(config, nRows, stage):
+ result = {
+ "nRows": nRows,
+ "stage": stage,
+ "status": "ok",
+ "resultUri": config.resultUri(nRows, stage),
+ "recoveredFromR2": True,
+ "spawnAttempt": attempt,
+ }
+ break
+ call = run_stage_job.with_options(**options).spawn(
+ configDict,
+ nRows,
+ stage,
+ )
+ try:
+ result = await_stage_result(
+ config,
+ nRows,
+ stage,
+ call,
+ deadlineSeconds=deadline_seconds,
+ )
+ break
+ except Exception as exc: # noqa: BLE001 - Modal surfaces many failure types
+ last_error = exc
+ if result_exists(config, nRows, stage):
+ result = {
+ "nRows": nRows,
+ "stage": stage,
+ "status": "ok",
+ "resultUri": config.resultUri(nRows, stage),
+ "recoveredFromR2": True,
+ "spawnAttempt": attempt,
+ "callError": str(exc),
+ }
+ break
+ if attempt >= spawn_attempts:
+ raise
+ print(
+ f"stage {stage} spawn attempt {attempt}/"
+ f"{spawn_attempts} failed ({exc}); retrying",
+ flush=True,
+ )
+ if result is None:
+ raise RuntimeError(
+ f"stage {stage} produced no result"
+ + (f" after error: {last_error}" if last_error else "")
+ )
+ outcomes.append(result)
+ if result.get("status") == "error":
+ return {
+ "nRows": nRows,
+ "stopped": True,
+ "failed": result,
+ "outcomes": outcomes,
+ }
+
+ return {"nRows": nRows, "stopped": False, "outcomes": outcomes}
+
+
+@app.function(
+ **COMMON_FUNCTION_OPTIONS,
+ timeout=86_400,
+ memory=2048,
+ cpu=1.0,
+ ephemeral_disk=BASE_EPHEMERAL_DISK_MB,
+)
+def run_all_jobs(
+ configDict: dict[str, Any],
+ sizes: list[int] | None = None,
+ stages: list[StageName] | None = None,
+) -> dict[str, Any]:
+ """Run sizes in parallel; stages within each size stay sequential."""
+ config = ProfilingConfig.model_validate(configDict)
+ os.environ.setdefault("R2_ENDPOINT", config.r2EndpointUrl)
+ selected_sizes = tuple(sizes) if sizes else config.targetSizes
+ selected_stages = tuple(stages) if stages else config.effectiveStages
+ for n_rows in selected_sizes:
+ if n_rows not in config.targetSizes:
+ raise ValueError(f"size {n_rows} is not in config.targetSizes")
+
+ parallel_sizes = max(1, len(selected_sizes))
+ orchestrator_options = orchestrator_function_options(
+ config,
+ maxContainers=parallel_sizes,
+ )
+
+ stage_list = list(selected_stages)
+ handles = [
+ run_size_jobs.with_options(**orchestrator_options).spawn(
+ configDict,
+ n_rows,
+ stage_list,
+ )
+ for n_rows in selected_sizes
+ ]
+ size_results = await_many_function_calls(
+ handles,
+ deadlineSeconds=86_400.0,
+ )
+ failed = [item for item in size_results if item.get("stopped")]
+ return {
+ "stopped": bool(failed),
+ "failed": failed[0] if failed else None,
+ "sizes": size_results,
+ }
+
+
+@app.function(
+ **COMMON_FUNCTION_OPTIONS,
+ timeout=300,
+ memory=1024,
+ cpu=1.0,
+ ephemeral_disk=BASE_EPHEMERAL_DISK_MB,
+)
+def smoke_check(configDict: dict[str, Any]) -> dict[str, Any]:
+ config = ProfilingConfig.model_validate(configDict)
+ os.environ.setdefault("R2_ENDPOINT", config.r2EndpointUrl)
+ probe = f"{config.resultsUri.rstrip('/')}/smoke/ok.json"
+ from profiling.r2 import put_json
+
+ put_json(probe, {"ok": True})
+ return {"probeUri": probe, "exists": object_exists(probe)}
+
+
+def _load_config(path: str) -> ProfilingConfig:
+ config = load_profiling_config(path)
+ validate_modal_environment(config)
+ return config
+
+
+def _deployed_function(config: ProfilingConfig, name: str) -> modal.Function:
+ try:
+ return modal.Function.from_name(
+ config.modalAppName,
+ name,
+ environment_name=config.modalEnvironmentName,
+ )
+ except Exception as exc:
+ raise SystemExit(
+ f"Could not find deployed function {config.modalAppName}/{name}. "
+ "Deploy first with:\n"
+ " uv run --group profiling modal deploy "
+ f"--env {config.modalEnvironmentName} -m profiling.modal_app\n"
+ f"Original error: {exc}"
+ ) from exc
+
+
+def _print_spawned(label: str, call: Any) -> None:
+ call_id = (
+ getattr(call, "object_id", None) or getattr(call, "call_id", None) or str(call)
+ )
+ print(f"spawned {label}: {call_id}")
+ print("disconnect is safe; watch with:")
+ print(
+ " uv run --group profiling modal app logs "
+ "scarf-profiling --env scarf_profiling"
+ )
+
+
+@app.local_entrypoint()
+def main(*arg_list: str) -> None:
+ parser = argparse.ArgumentParser(prog="profiling.modal_app")
+ sub = parser.add_subparsers(dest="command", required=True)
+
+ smoke_parser = sub.add_parser("smoke")
+ smoke_parser.add_argument("--config", required=True)
+
+ prepare_parser = sub.add_parser("prepare")
+ prepare_parser.add_argument("--config", required=True)
+
+ fixture_parser = sub.add_parser("prepare-fixture")
+ fixture_parser.add_argument("--config", required=True)
+ fixture_parser.add_argument("--sizes", nargs="*", type=int, default=[10_000])
+ fixture_parser.add_argument("--n-columns", type=int, default=500)
+
+ run_parser = sub.add_parser("run")
+ run_parser.add_argument("--config", required=True)
+ run_parser.add_argument("--size", type=int, required=True)
+ run_parser.add_argument("--stage", choices=CORE_STAGE_ORDER, required=True)
+ run_parser.add_argument(
+ "--force",
+ action="store_true",
+ help=(
+ "Recompute an existing targeted stage, invalidate reusable artifacts, "
+ "and overwrite its stage result JSON."
+ ),
+ )
+ run_parser.add_argument(
+ "--ephemeral",
+ action="store_true",
+ help="Spawn from this modal run app (no deploy). Prefer --detach.",
+ )
+
+ all_parser = sub.add_parser("run-all")
+ all_parser.add_argument("--config", required=True)
+ all_parser.add_argument("--sizes", nargs="*", type=int, default=None)
+ all_parser.add_argument(
+ "--stages", nargs="*", choices=CORE_STAGE_ORDER, default=None
+ )
+ all_parser.add_argument(
+ "--ephemeral",
+ action="store_true",
+ help="Spawn from this modal run app (no deploy). Prefer --detach.",
+ )
+
+ local_parser = sub.add_parser(
+ "run-local",
+ help=(
+ "One-container funnel on ephemeral-disk Zarr (fast_local); "
+ "H5AD downloaded once from R2; stage results still written to R2"
+ ),
+ )
+ local_parser.add_argument("--config", required=True)
+ local_parser.add_argument("--size", type=int, required=True)
+ local_parser.add_argument(
+ "--stages", nargs="*", choices=CORE_STAGE_ORDER, default=None
+ )
+
+ e2e_parser = sub.add_parser(
+ "run-e2e",
+ help="One-container graph-construction funnel with a fresh R2 Zarr store",
+ )
+ e2e_parser.add_argument("--config", required=True)
+ e2e_parser.add_argument("--size", type=int, required=True)
+
+ io_parser = sub.add_parser("io-baseline")
+ io_parser.add_argument("--config", required=True)
+ io_parser.add_argument("--size", type=int, default=1_000_000)
+ io_parser.add_argument("--result-label")
+ io_parser.add_argument("--column-only", action="store_true")
+ io_parser.add_argument("--wait", action="store_true")
+ io_parser.add_argument(
+ "--ephemeral",
+ action="store_true",
+ help="Spawn from this modal run app without a deploy.",
+ )
+
+ args = parser.parse_args(list(arg_list))
+ config = _load_config(args.config)
+ payload = config.model_dump(mode="python")
+
+ if args.command == "smoke":
+ smoke_options = orchestrator_function_options(config)
+ call = (
+ _deployed_function(config, "smoke_check")
+ .with_options(**smoke_options)
+ .spawn(payload)
+ )
+ print(await_function_call(call, deadlineSeconds=300.0))
+ return
+
+ if args.command == "prepare":
+ prepare_options = modal_function_options(
+ config,
+ config.prepareResources,
+ )
+ call = (
+ _deployed_function(config, "prepare_datasets")
+ .with_options(**prepare_options)
+ .spawn(payload)
+ )
+ _print_spawned("prepare_datasets", call)
+ return
+
+ if args.command == "prepare-fixture":
+ sizes = list(args.sizes) if args.sizes else [10_000]
+ for size in sizes:
+ if size not in config.targetSizes:
+ raise SystemExit(f"size {size} is not in config.targetSizes")
+ fixture_options = modal_function_options(
+ config,
+ config.resourcesFor("reopenStore"),
+ )
+ call = (
+ _deployed_function(config, "prepare_fixture_datasets_job")
+ .with_options(**fixture_options)
+ .spawn(payload, sizes, args.n_columns)
+ )
+ _print_spawned("prepare_fixture_datasets_job", call)
+ return
+
+ if args.command == "run":
+ if args.size not in config.targetSizes:
+ raise SystemExit(f"size {args.size} is not in config.targetSizes")
+ if result_exists(config, args.size, args.stage) and not args.force:
+ print(
+ {
+ "nRows": args.size,
+ "stage": args.stage,
+ "status": "skipped",
+ "resultUri": config.resultUri(args.size, args.stage),
+ }
+ )
+ return
+ resources = config.resourcesFor(args.stage)
+ # retries=0 for writeCountsT: a retry that overlaps the original write
+ # leaves countsT marked incomplete again.
+ options = (
+ modal_function_options(config, resources, retries=0)
+ if args.stage == "writeCountsT"
+ else modal_function_options(config, resources)
+ )
+ target = (
+ run_stage_job
+ if args.ephemeral
+ else _deployed_function(config, "run_stage_job")
+ )
+ spawn_args: tuple[Any, ...] = (payload, args.size, args.stage)
+ if args.force:
+ spawn_args += (True,)
+ call = target.with_options(**options).spawn(*spawn_args)
+ _print_spawned(f"run_stage_job {args.size}/{args.stage}", call)
+ return
+
+ if args.command == "run-all":
+ sizes = list(args.sizes) if args.sizes else None
+ stages = list(args.stages) if args.stages else None
+ if sizes:
+ for size in sizes:
+ if size not in config.targetSizes:
+ raise SystemExit(f"size {size} is not in config.targetSizes")
+ coordinator_options = orchestrator_function_options(config)
+ target = (
+ run_all_jobs
+ if args.ephemeral
+ else _deployed_function(config, "run_all_jobs")
+ )
+ call = target.with_options(**coordinator_options).spawn(payload, sizes, stages)
+ _print_spawned("run_all_jobs", call)
+ return
+
+ if args.command == "run-e2e":
+ if args.size not in config.targetSizes:
+ raise SystemExit(f"size {args.size} is not in config.targetSizes")
+ if not config.runTag.strip():
+ raise SystemExit("run-e2e requires a non-empty runTag")
+ options = _e2e_function_options(config)
+ call = (
+ _deployed_function(config, "run_e2e_funnel_job")
+ .with_options(**options)
+ .spawn(payload, args.size)
+ )
+ _print_spawned(f"run_e2e_funnel_job {args.size}", call)
+ print(f"result URI (when done): {config.funnelResultUri(args.size)}")
+ return
+
+ if args.command == "run-local":
+ if args.size not in config.targetSizes:
+ raise SystemExit(f"size {args.size} is not in config.targetSizes")
+ stages = list(args.stages) if args.stages else None
+ selected = tuple(stages) if stages else config.effectiveStages
+ # Size the single container to the hungriest stage in the funnel.
+ peak = max(
+ (config.resourcesFor(stage) for stage in selected),
+ key=lambda item: (
+ item.modalMemoryLimitMb,
+ item.modalCpuLimit,
+ item.timeoutSeconds,
+ ),
+ )
+ options = modal_function_options(config, peak, maxContainers=1)
+ call = (
+ _deployed_function(config, "run_local_funnel_job")
+ .with_options(**options)
+ .spawn(payload, args.size, stages)
+ )
+ _print_spawned(f"run_local_funnel_job {args.size}", call)
+ return
+
+ if args.command == "io-baseline":
+ resources = config.resourcesFor("markHvgs")
+ options = modal_function_options(config, resources, maxContainers=1)
+ target = (
+ io_baseline_job
+ if args.ephemeral
+ else _deployed_function(config, "io_baseline_job")
+ )
+ call = target.with_options(**options).spawn(
+ payload,
+ args.size,
+ args.result_label,
+ args.column_only,
+ )
+ _print_spawned(f"io_baseline_job size={args.size}", call)
+ if args.wait:
+ print(
+ await_function_call(
+ call,
+ deadlineSeconds=float(resources.timeoutSeconds),
+ )
+ )
+ print(
+ "result URI (when done): "
+ f"{config.resultsUri.rstrip('/')}/io-baseline/{config.runTag}"
+ f"{'-' + args.result_label if args.result_label else ''}.json"
+ )
+ return
diff --git a/profiling/modal_image.py b/profiling/modal_image.py
new file mode 100644
index 00000000..d43f7cec
--- /dev/null
+++ b/profiling/modal_image.py
@@ -0,0 +1,32 @@
+import modal
+
+
+MODAL_APP_NAME = "scarf-profiling"
+
+app = modal.App(MODAL_APP_NAME)
+
+image = (
+ modal.Image.debian_slim(python_version="3.12")
+ .apt_install(
+ "build-essential",
+ "git",
+ "libfftw3-dev",
+ "libmetis-dev",
+ "libtbb-dev",
+ )
+ .uv_sync(
+ groups=["profiling"],
+ frozen=True,
+ extra_options="--no-default-groups",
+ )
+ .add_local_python_source("scarf", "profiling", copy=True)
+ .add_local_file("uv.lock", "/root/uv.lock", copy=True)
+ .add_local_file("VERSION", "/root/VERSION", copy=True)
+)
+
+
+COMMON_FUNCTION_OPTIONS = {
+ "image": image,
+ "retries": 0,
+ "single_use_containers": True,
+}
diff --git a/profiling/modal_resources.py b/profiling/modal_resources.py
new file mode 100644
index 00000000..58eaea99
--- /dev/null
+++ b/profiling/modal_resources.py
@@ -0,0 +1,92 @@
+from typing import Any
+
+import modal
+
+from profiling.config import PrepareResources, ProfilingConfig, StageResources
+
+# Modal currently requires ephemeral disk in this range (MiB).
+MIN_EPHEMERAL_DISK_MB = 524_288
+MAX_EPHEMERAL_DISK_MB = 3_145_728
+BASE_EPHEMERAL_DISK_MB = MIN_EPHEMERAL_DISK_MB
+
+# Stage workers: retry after runner loss / InternalFailure. Orchestrators stay at 0.
+STAGE_JOB_RETRIES = modal.Retries(
+ max_retries=3,
+ backoff_coefficient=2.0,
+ initial_delay=5.0,
+)
+
+
+def resolve_ephemeral_disk_mb(requestedMb: int) -> int:
+ if requestedMb > MAX_EPHEMERAL_DISK_MB:
+ raise ValueError(
+ "Requested ephemeral disk is "
+ f"{requestedMb} MiB, but Modal allows at most "
+ f"{MAX_EPHEMERAL_DISK_MB} MiB"
+ )
+ if requestedMb < MIN_EPHEMERAL_DISK_MB:
+ return MIN_EPHEMERAL_DISK_MB
+ return requestedMb
+
+
+def validate_modal_environment(config: ProfilingConfig) -> None:
+ if config.modalEnvironmentName != "scarf_profiling":
+ raise ValueError("Modal environment must be scarf_profiling")
+ environment = modal.Environment.from_name(
+ config.modalEnvironmentName,
+ create_if_missing=False,
+ )
+ environment.hydrate()
+
+
+def modal_function_options(
+ config: ProfilingConfig,
+ resources: StageResources | PrepareResources,
+ *,
+ maxContainers: int = 1,
+ retries: int | modal.Retries = STAGE_JOB_RETRIES,
+) -> dict[str, Any]:
+ if maxContainers <= 0:
+ raise ValueError("maxContainers must be positive")
+ secret = modal.Secret.from_name(
+ config.modalSecretName,
+ environment_name=config.modalEnvironmentName,
+ required_keys=["R2_ACCESS_KEY_ID", "R2_SECRET_ACCESS_KEY"],
+ )
+ # ephemeral_disk is fixed on @app.function; with_options does not accept it.
+ _ = resolve_ephemeral_disk_mb(resources.ephemeralDiskMb)
+ # Do not pass cloud=; pinning aws (or any provider) shrinks Modal capacity.
+ return {
+ "cpu": (resources.modalCpuRequest, resources.modalCpuLimit),
+ "memory": (resources.modalMemoryRequestMb, resources.modalMemoryLimitMb),
+ "env": {"R2_ENDPOINT": config.r2EndpointUrl},
+ "secrets": [secret],
+ "retries": retries,
+ "max_containers": maxContainers,
+ "buffer_containers": 0,
+ "timeout": resources.timeoutSeconds,
+ "region": config.modalRegion,
+ }
+
+
+def orchestrator_function_options(
+ config: ProfilingConfig,
+ *,
+ maxContainers: int = 1,
+) -> dict[str, Any]:
+ """Tiny options for run_all_jobs / run_size_jobs coordinators.
+
+ These only spawn/wait; they must not request stage RAM (32–64 GiB) or they
+ compete with the real stage workers for scarce high-memory capacity.
+ """
+ # Borrow secrets/region/env from a stage resource block, then shrink compute.
+ options = modal_function_options(
+ config,
+ config.resourcesFor("reopenStore"),
+ maxContainers=maxContainers,
+ retries=0,
+ )
+ options["memory"] = (2048, 4096)
+ options["cpu"] = (1.0, 1.0)
+ options["timeout"] = 86_400
+ return options
diff --git a/profiling/paris_worker.py b/profiling/paris_worker.py
new file mode 100644
index 00000000..590a4921
--- /dev/null
+++ b/profiling/paris_worker.py
@@ -0,0 +1,115 @@
+"""Run Paris clustering in a child process, matching the Leiden stage.
+
+Modal's runner heartbeat interval is about 900 seconds and is not configurable,
+and the earlier scikit-network implementation could block the parent for longer
+than that. Scarf's Numba kernels release the GIL, so this isolation is no longer
+strictly required, but profiling keeps it so both clustering stages report
+progress and failures the same way. ``profiling.stages`` polls this child every
+30 seconds and warns at 1800 seconds without killing it.
+"""
+
+import argparse
+import json
+import time
+from collections.abc import Sequence
+from pathlib import Path
+from typing import Any
+
+from scarf import configure_output
+
+from profiling.config import StageResources, WorkflowParameters
+from profiling.stages import _open_datastore
+
+configure_output(progress=False, timestamps=True)
+
+
+def _write_status(statusPath: Path, payload: dict[str, Any]) -> None:
+ statusPath.write_text(json.dumps(payload), encoding="utf-8")
+
+
+def run_paris_worker(requestPath: Path) -> None:
+ request = json.loads(requestPath.read_text(encoding="utf-8"))
+ if not isinstance(request, dict):
+ raise ValueError("Paris worker request must be a JSON object")
+
+ store_uri = str(request["storeUri"])
+ workflow = WorkflowParameters.model_validate(request["workflow"])
+ resources = StageResources.model_validate(request["resources"])
+ status_path = Path(str(request["statusPath"]))
+
+ print(f"[paris_worker] START store={store_uri}", flush=True)
+ worker_started = time.perf_counter()
+ setup_started = worker_started
+ input_setup_seconds: float | None = None
+ operation_started: float | None = None
+ operation_seconds: float | None = None
+ try:
+ store = _open_datastore(
+ store_uri,
+ workflow,
+ resources,
+ initialize=False,
+ )
+ input_setup_seconds = time.perf_counter() - setup_started
+ arguments: dict[str, Any] = {
+ "from_assay": workflow.assayName,
+ "cell_key": workflow.cellKey,
+ "feat_key": workflow.hvgKey,
+ "label": workflow.parisLabel,
+ "n_clusters": workflow.parisNClusters,
+ }
+ if workflow.parisMinClusterSize is not None:
+ arguments["min_cluster_size"] = workflow.parisMinClusterSize
+ print(
+ f"[paris_worker] cut nClusters={workflow.parisNClusters} "
+ f"minClusterSize={workflow.parisMinClusterSize}",
+ flush=True,
+ )
+
+ print("[paris_worker] datastore open; ENTER run_paris_clustering", flush=True)
+ operation_started = time.perf_counter()
+ store.run_paris_clustering(**arguments)
+ operation_seconds = time.perf_counter() - operation_started
+ del store
+ except BaseException as exc:
+ now = time.perf_counter()
+ if input_setup_seconds is None:
+ input_setup_seconds = now - setup_started
+ if operation_started is not None and operation_seconds is None:
+ operation_seconds = now - operation_started
+ error = f"{type(exc).__name__}: {exc}"
+ _write_status(
+ status_path,
+ {
+ "status": "error",
+ "error": error,
+ "inputSetupSeconds": input_setup_seconds,
+ "operationSeconds": operation_seconds,
+ "wholeWorkerSeconds": now - worker_started,
+ },
+ )
+ print(f"[paris_worker] ERROR {error}", flush=True)
+ raise
+
+ _write_status(
+ status_path,
+ {
+ "status": "ok",
+ "error": None,
+ "inputSetupSeconds": input_setup_seconds,
+ "operationSeconds": operation_seconds,
+ "wholeWorkerSeconds": time.perf_counter() - worker_started,
+ },
+ )
+ print("[paris_worker] DONE run_paris_clustering", flush=True)
+
+
+def main(argv: Sequence[str] | None = None) -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--request", required=True, type=Path)
+ args = parser.parse_args(argv)
+ run_paris_worker(args.request)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/profiling/r2.py b/profiling/r2.py
new file mode 100644
index 00000000..fd3c4529
--- /dev/null
+++ b/profiling/r2.py
@@ -0,0 +1,212 @@
+import json
+import os
+import time
+from dataclasses import dataclass
+from datetime import timedelta
+from pathlib import Path
+from typing import Any
+from urllib.parse import urlsplit
+
+from obstore.exceptions import AlreadyExistsError
+from obstore.store import from_url
+
+_ENV_PATH = Path(__file__).resolve().parent / ".env"
+_DEFAULT_TRANSFER_CHUNK_BYTES = 16 * 1024 * 1024
+_CREDENTIAL_KEYS = (
+ "R2_ENDPOINT",
+ "R2_ACCESS_KEY_ID",
+ "R2_SECRET_ACCESS_KEY",
+)
+# Large objects (Cellxgene source ~46 GiB) exceed obstore's default 30s request timeout.
+_CLIENT_OPTIONS = {
+ "timeout": "12h",
+ "connect_timeout": "120s",
+ "read_timeout": "30m",
+}
+_RETRY_CONFIG = {
+ "max_retries": 20,
+ "retry_timeout": timedelta(minutes=30),
+}
+
+
+@dataclass(frozen=True, slots=True)
+class ObjectDownload:
+ fileBytes: int
+ eTag: str | None
+
+
+@dataclass(frozen=True, slots=True)
+class ObjectUpload:
+ fileBytes: int
+ eTag: str | None
+
+
+def _load_local_env(path: Path = _ENV_PATH) -> None:
+ if not path.is_file():
+ return
+ for line in path.read_text().splitlines():
+ line = line.strip()
+ if not line or line.startswith("#") or "=" not in line:
+ continue
+ key, value = line.split("=", 1)
+ key = key.strip()
+ value = value.strip().strip('"').strip("'")
+ if key and key not in os.environ:
+ os.environ[key] = value
+
+
+def storage_options(uri: str) -> dict[str, str] | None:
+ if not uri.startswith("s3://"):
+ return None
+ _load_local_env()
+ missing = [name for name in _CREDENTIAL_KEYS if not os.environ.get(name)]
+ if missing:
+ raise RuntimeError(f"Missing R2 environment settings: {', '.join(missing)}")
+ endpoint = os.environ["R2_ENDPOINT"]
+ access_key = os.environ["R2_ACCESS_KEY_ID"]
+ secret_key = os.environ["R2_SECRET_ACCESS_KEY"]
+ assert endpoint and access_key and secret_key
+ return {
+ "access_key_id": access_key,
+ "secret_access_key": secret_key,
+ "endpoint": endpoint.rstrip("/"),
+ }
+
+
+def open_r2_object(uri: str) -> tuple[Any, str]:
+ parsed = urlsplit(uri)
+ if parsed.scheme != "s3" or not parsed.netloc:
+ raise ValueError(f"Expected an s3:// object URI, got: {uri}")
+ key = parsed.path.lstrip("/")
+ if not key:
+ raise ValueError("R2 object URI must include an object key")
+ options = storage_options(uri)
+ assert options is not None
+ store = from_url(
+ f"s3://{parsed.netloc}",
+ client_options=_CLIENT_OPTIONS,
+ retry_config=_RETRY_CONFIG,
+ **options,
+ )
+ return store, key
+
+
+def join_uri(prefix: str, *parts: str) -> str:
+ base = prefix.rstrip("/")
+ suffix = "/".join(part.strip("/") for part in parts if part)
+ return f"{base}/{suffix}" if suffix else base
+
+
+def object_exists(uri: str) -> bool:
+ store, key = open_r2_object(uri)
+ try:
+ store.head(key)
+ except FileNotFoundError:
+ return False
+ return True
+
+
+def object_size(uri: str) -> int | None:
+ store, key = open_r2_object(uri)
+ try:
+ meta = store.head(key)
+ except FileNotFoundError:
+ return None
+ return int(meta["size"])
+
+
+def get_json(uri: str) -> dict[str, Any]:
+ store, key = open_r2_object(uri)
+ result = store.get(key)
+ body = bytes(result.bytes())
+ payload = json.loads(body.decode("utf-8"))
+ if not isinstance(payload, dict):
+ raise ValueError(f"Expected JSON object at {uri}")
+ return payload
+
+
+def _encode_json(value: dict[str, Any]) -> bytes:
+ return (json.dumps(value, sort_keys=True, separators=(",", ":")) + "\n").encode(
+ "utf-8"
+ )
+
+
+def put_json(uri: str, value: dict[str, Any]) -> None:
+ store, key = open_r2_object(uri)
+ store.put(key, _encode_json(value))
+
+
+def put_json_if_absent(uri: str, value: dict[str, Any]) -> bool:
+ store, key = open_r2_object(uri)
+ try:
+ store.put(
+ key,
+ _encode_json(value),
+ mode="create",
+ use_multipart=False,
+ )
+ except AlreadyExistsError:
+ return False
+ return True
+
+
+def download_file(
+ uri: str,
+ destination: str | Path,
+ *,
+ chunkBytes: int = _DEFAULT_TRANSFER_CHUNK_BYTES,
+ maxAttempts: int = 8,
+) -> ObjectDownload:
+ """Download with ranged GETs so large objects can resume after timeouts."""
+ if chunkBytes <= 0:
+ raise ValueError("chunkBytes must be positive")
+ if maxAttempts <= 0:
+ raise ValueError("maxAttempts must be positive")
+
+ store, key = open_r2_object(uri)
+ destination_path = Path(destination)
+ destination_path.parent.mkdir(parents=True, exist_ok=True)
+ meta = store.head(key)
+ total = int(meta["size"])
+ part_path = destination_path.with_name(f".{destination_path.name}.part")
+ offset = part_path.stat().st_size if part_path.is_file() else 0
+ if offset > total:
+ part_path.unlink()
+ offset = 0
+
+ attempts = 0
+ while offset < total:
+ end = min(offset + chunkBytes, total)
+ try:
+ chunk = bytes(store.get_range(key, start=offset, end=end))
+ except Exception:
+ attempts += 1
+ if attempts >= maxAttempts:
+ raise
+ time.sleep(min(60.0, 2.0**attempts))
+ store, key = open_r2_object(uri)
+ continue
+ if not chunk:
+ raise RuntimeError(f"Empty range response for {uri} at offset {offset}")
+ with part_path.open("ab" if offset else "wb") as handle:
+ handle.write(chunk)
+ offset += len(chunk)
+ attempts = 0
+
+ if offset != total:
+ raise RuntimeError(f"Downloaded {offset} bytes from {uri}, expected {total}")
+ os.replace(part_path, destination_path)
+ e_tag = meta.get("e_tag")
+ return ObjectDownload(fileBytes=total, eTag=str(e_tag) if e_tag else None)
+
+
+def upload_file(source: str | Path, uri: str) -> ObjectUpload:
+ source_path = Path(source)
+ if not source_path.is_file():
+ raise FileNotFoundError(source_path)
+ store, key = open_r2_object(uri)
+ file_bytes = source_path.stat().st_size
+ store.put(key, source_path, use_multipart=True)
+ meta = store.head(key)
+ e_tag = meta.get("e_tag")
+ return ObjectUpload(fileBytes=file_bytes, eTag=str(e_tag) if e_tag else None)
diff --git a/profiling/results.py b/profiling/results.py
new file mode 100644
index 00000000..c6d1631d
--- /dev/null
+++ b/profiling/results.py
@@ -0,0 +1,23 @@
+from profiling.config import ProfilingConfig, StageName
+from profiling.r2 import object_exists, put_json
+from profiling.stages import StageRunResult
+
+
+def result_exists(config: ProfilingConfig, nRows: int, stage: StageName) -> bool:
+ return object_exists(config.resultUri(nRows, stage))
+
+
+def write_result(config: ProfilingConfig, result: StageRunResult) -> str:
+ uri = config.resultUri(result.nRows, result.stage)
+ put_json(uri, result.to_json())
+ return uri
+
+
+def write_funnel_result(
+ config: ProfilingConfig,
+ nRows: int,
+ payload: dict[str, object],
+) -> str:
+ uri = config.funnelResultUri(nRows)
+ put_json(uri, payload)
+ return uri
diff --git a/profiling/spawn_wait.py b/profiling/spawn_wait.py
new file mode 100644
index 00000000..82b20c68
--- /dev/null
+++ b/profiling/spawn_wait.py
@@ -0,0 +1,187 @@
+"""Spawn-safe waits: never use Function.remote() for long profiling work."""
+
+import time
+from collections.abc import Callable
+from typing import Any
+
+from profiling.config import ProfilingConfig, StageName
+from profiling.r2 import get_json
+from profiling.results import result_exists
+
+# How often orchestrators poll R2 / call status. Short polls keep heartbeats alive.
+DEFAULT_POLL_SECONDS = 20.0
+# Extra grace after stage timeout for scheduling + result upload.
+DEFAULT_GRACE_SECONDS = 600.0
+# Re-spawn a stage this many times if the Modal call dies without an R2 result.
+DEFAULT_STAGE_SPAWN_ATTEMPTS = 3
+
+# Modal sometimes surfaces failed calls as empty TimeoutError from get(timeout=...).
+# Detect terminal statuses via the call graph instead of spinning until deadline.
+_TERMINAL_FAILURE_STATUSES = frozenset(
+ {"FAILURE", "INIT_FAILURE", "TERMINATED", "TIMEOUT"}
+)
+
+
+def _call_id(call: Any) -> str | None:
+ # object_id is a property that raises until hydrated; getattr default masks that.
+ if hasattr(call, "hydrate"):
+ try:
+ call.hydrate()
+ except Exception: # noqa: BLE001 - best-effort
+ pass
+ try:
+ object_id = call.object_id
+ except Exception: # noqa: BLE001 - unhydrated / closed client
+ object_id = None
+ if object_id:
+ return object_id
+ return getattr(call, "call_id", None)
+
+
+def _input_status_name(call: Any) -> str | None:
+ """Return InputStatus name for this call, if the call graph exposes it."""
+ call_id = _call_id(call)
+ if not call_id or not hasattr(call, "get_call_graph"):
+ return None
+ try:
+ graph = call.get_call_graph()
+ except Exception: # noqa: BLE001 - best-effort probe
+ return None
+ stack = list(graph or [])
+ while stack:
+ node = stack.pop()
+ if getattr(node, "function_call_id", None) == call_id:
+ status = getattr(node, "status", None)
+ if status is None:
+ return None
+ return getattr(status, "name", str(status))
+ stack.extend(getattr(node, "children", None) or [])
+ return None
+
+
+def _raise_if_terminal_failure(call: Any) -> None:
+ status = _input_status_name(call)
+ if status in _TERMINAL_FAILURE_STATUSES:
+ raise RuntimeError(
+ f"Spawned call {_call_id(call) or ''} ended with status={status}"
+ )
+
+
+def await_function_call(
+ call: Any,
+ *,
+ pollSeconds: float = DEFAULT_POLL_SECONDS,
+ deadlineSeconds: float,
+) -> Any:
+ """Wait on a spawned FunctionCall with short get() polls (not .remote())."""
+ if pollSeconds <= 0:
+ raise ValueError("pollSeconds must be positive")
+ if deadlineSeconds <= 0:
+ raise ValueError("deadlineSeconds must be positive")
+ deadline = time.monotonic() + deadlineSeconds
+ while time.monotonic() < deadline:
+ remaining = max(0.1, deadline - time.monotonic())
+ timeout = min(pollSeconds, remaining)
+ try:
+ return call.get(timeout=timeout)
+ except TimeoutError:
+ _raise_if_terminal_failure(call)
+ continue
+ raise TimeoutError(f"Spawned call did not finish within {deadlineSeconds:.0f}s")
+
+
+def await_many_function_calls(
+ calls: list[Any],
+ *,
+ pollSeconds: float = DEFAULT_POLL_SECONDS,
+ deadlineSeconds: float,
+) -> list[Any]:
+ """Wait on many spawned calls with short interleaved get() polls."""
+ if not calls:
+ return []
+ if pollSeconds <= 0:
+ raise ValueError("pollSeconds must be positive")
+ if deadlineSeconds <= 0:
+ raise ValueError("deadlineSeconds must be positive")
+ deadline = time.monotonic() + deadlineSeconds
+ pending = list(enumerate(calls))
+ results: list[Any | None] = [None] * len(calls)
+ while pending:
+ if time.monotonic() >= deadline:
+ raise TimeoutError(
+ f"Spawned calls did not finish within {deadlineSeconds:.0f}s "
+ f"({len(pending)} still pending)"
+ )
+ still_pending: list[tuple[int, Any]] = []
+ for index, call in pending:
+ remaining = max(0.1, deadline - time.monotonic())
+ timeout = min(pollSeconds, remaining)
+ try:
+ results[index] = call.get(timeout=timeout)
+ except TimeoutError:
+ _raise_if_terminal_failure(call)
+ still_pending.append((index, call))
+ pending = still_pending
+ return [item for item in results]
+
+
+def await_stage_result(
+ config: ProfilingConfig,
+ nRows: int,
+ stage: StageName,
+ call: Any,
+ *,
+ pollSeconds: float = DEFAULT_POLL_SECONDS,
+ deadlineSeconds: float,
+ onPoll: Callable[[], None] | None = None,
+) -> dict[str, Any]:
+ """Prefer durable R2 result JSON; fall back to the call return value."""
+ if pollSeconds <= 0:
+ raise ValueError("pollSeconds must be positive")
+ if deadlineSeconds <= 0:
+ raise ValueError("deadlineSeconds must be positive")
+ deadline = time.monotonic() + deadlineSeconds
+ last_error: BaseException | None = None
+
+ while time.monotonic() < deadline:
+ if result_exists(config, nRows, stage):
+ return get_json(config.resultUri(nRows, stage))
+ if onPoll is not None:
+ onPoll()
+ remaining = max(0.1, deadline - time.monotonic())
+ timeout = min(pollSeconds, remaining)
+ try:
+ payload = call.get(timeout=timeout)
+ except TimeoutError:
+ try:
+ _raise_if_terminal_failure(call)
+ except RuntimeError as exc:
+ last_error = exc
+ if result_exists(config, nRows, stage):
+ return get_json(config.resultUri(nRows, stage))
+ raise
+ continue
+ except Exception as exc: # noqa: BLE001 - Modal surfaces many failure types
+ last_error = exc
+ # Call may have died after writing R2; check once more.
+ if result_exists(config, nRows, stage):
+ return get_json(config.resultUri(nRows, stage))
+ raise
+
+ if result_exists(config, nRows, stage):
+ return get_json(config.resultUri(nRows, stage))
+ if isinstance(payload, dict):
+ return payload
+ raise TypeError(
+ f"Stage {stage} returned non-dict payload: {type(payload).__name__}"
+ )
+
+ if result_exists(config, nRows, stage):
+ return get_json(config.resultUri(nRows, stage))
+ if last_error is not None:
+ raise TimeoutError(
+ f"Timed out waiting for {stage} after call error: {last_error}"
+ ) from last_error
+ raise TimeoutError(
+ f"Timed out waiting for {stage} result at {config.resultUri(nRows, stage)}"
+ )
diff --git a/profiling/stages.py b/profiling/stages.py
new file mode 100644
index 00000000..4f537fef
--- /dev/null
+++ b/profiling/stages.py
@@ -0,0 +1,794 @@
+import json
+import subprocess
+import sys
+import tempfile
+import time
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+from scarf import DataStore, H5adReader, H5adToZarr, configure_output
+from scarf.storage.budget import resolve_budget
+from scarf.storage.sharding import write_counts_t
+from scarf.storage.stores import open_store
+from scarf.storage.types import as_zarr_array, as_zarr_group
+
+from profiling.config import (
+ StageName,
+ StageResources,
+ StorageLayout,
+ WorkflowParameters,
+)
+from profiling.metrics import ResourceMeasurement, ResourceSampler, StageTimer
+from profiling.r2 import storage_options
+
+configure_output(progress=False, timestamps=True)
+
+CHILD_MONITOR_INTERVAL_SECONDS = 30.0
+CHILD_WARNING_SECONDS = 1_800.0
+CHILD_STOP_GRACE_SECONDS = 30.0
+
+
+@dataclass(frozen=True, slots=True)
+class StageRunResult:
+ stage: StageName
+ nRows: int
+ status: str
+ seconds: float | None
+ peakRssBytes: int | None
+ peakCgroupBytes: int | None
+ modalMemoryMb: int
+ scarfMemoryBudget: int
+ storeUri: str
+ error: str | None = None
+ inputSetupSeconds: float | None = None
+ validationPersistenceSeconds: float | None = None
+ wholeFunctionSeconds: float | None = None
+ modalCpuRequest: float | None = None
+ modalCpuLimit: float | None = None
+ rssBaselineBytes: int | None = None
+ rssIncrementalPeakBytes: int | None = None
+ rssAfterBytes: int | None = None
+ cgroupCurrentBaselineBytes: int | None = None
+ cgroupCurrentPeakBytes: int | None = None
+ cgroupCurrentAfterBytes: int | None = None
+ operationBaselineBytes: int | None = None
+ operationIncrementalPeakBytes: int | None = None
+ operationPeakSource: str | None = None
+ cgroupPeakScope: str | None = None
+ details: dict[str, Any] | None = None
+
+ def to_json(self) -> dict[str, Any]:
+ return asdict(self)
+
+
+def _open_datastore(
+ storeUri: str,
+ workflow: WorkflowParameters,
+ resources: StageResources,
+ *,
+ initialize: bool,
+) -> DataStore:
+ options = storage_options(storeUri)
+ arguments: dict[str, Any] = {
+ "nthreads": resources.workers,
+ "zarr_mode": "r+",
+ "zarrProfile": ("cloud" if storeUri.startswith("s3://") else "fast_local"),
+ "storage_options": options,
+ "mem_budget": resources.scarfMemoryBudget,
+ }
+ if initialize:
+ arguments.update(
+ {
+ "assay_types": {workflow.assayName: "RNA"},
+ "default_assay": workflow.assayName,
+ "min_features_per_cell": workflow.minFeaturesPerCell,
+ "min_cells_per_feature": workflow.minCellsPerFeature,
+ }
+ )
+ return DataStore(storeUri, **arguments)
+
+
+def _close_h5ad_reader(reader: H5adReader) -> None:
+ if hasattr(reader, "h5") and hasattr(reader.h5, "close"):
+ reader.h5.close()
+
+
+def _prepare_create_store(
+ *,
+ localH5adPath: Path,
+ storeUri: str,
+ workflow: WorkflowParameters,
+ resources: StageResources,
+ storageLayout: StorageLayout | None = None,
+) -> tuple[H5adReader, H5adToZarr]:
+ options = storage_options(storeUri)
+ reader = H5adReader(
+ str(localH5adPath),
+ matrix_key="X",
+ cell_attrs_key="obs",
+ cell_ids_key="_index",
+ feature_attrs_key="var",
+ feature_ids_key="_index",
+ feature_name_key="feature_name",
+ )
+ layout_kwargs: dict[str, Any] = {}
+ if storageLayout is not None:
+ if storageLayout.targetChunkBytes is not None:
+ layout_kwargs["targetChunkBytes"] = storageLayout.targetChunkBytes
+ if storageLayout.targetShardBytes is not None:
+ layout_kwargs["targetShardBytes"] = storageLayout.targetShardBytes
+ try:
+ writer = H5adToZarr(
+ reader,
+ storeUri,
+ assay_name=workflow.assayName,
+ storage_options=options,
+ mem_budget=resources.scarfMemoryBudget,
+ nthreads=resources.workers,
+ **layout_kwargs,
+ )
+ except BaseException:
+ _close_h5ad_reader(reader)
+ raise
+ return reader, writer
+
+
+def _monitor_child_process(
+ process: subprocess.Popen[bytes],
+ *,
+ stageLabel: str,
+ warningSeconds: float = CHILD_WARNING_SECONDS,
+ pollSeconds: float = CHILD_MONITOR_INTERVAL_SECONDS,
+) -> int:
+ if warningSeconds <= 0:
+ raise ValueError("warningSeconds must be positive")
+ if pollSeconds <= 0:
+ raise ValueError("pollSeconds must be positive")
+
+ started = time.monotonic()
+ warned = False
+ while True:
+ try:
+ return process.wait(timeout=pollSeconds)
+ except subprocess.TimeoutExpired:
+ elapsed = time.monotonic() - started
+ print(
+ f"[{stageLabel}] child still running pid={process.pid} "
+ f"elapsedSeconds={elapsed:.0f}",
+ flush=True,
+ )
+ if not warned and elapsed >= warningSeconds:
+ print(
+ f"[{stageLabel}] WARNING child exceeded "
+ f"{warningSeconds:.0f}s; continuing",
+ flush=True,
+ )
+ warned = True
+
+
+def _stop_child_process(process: subprocess.Popen[bytes]) -> None:
+ if process.poll() is not None:
+ return
+ process.terminate()
+ try:
+ process.wait(timeout=CHILD_STOP_GRACE_SECONDS)
+ except subprocess.TimeoutExpired:
+ process.kill()
+ process.wait()
+
+
+def _read_worker_status(statusPath: Path, *, workerName: str) -> dict[str, Any] | None:
+ if not statusPath.is_file():
+ return None
+ payload = json.loads(statusPath.read_text(encoding="utf-8"))
+ if not isinstance(payload, dict):
+ raise RuntimeError(f"{workerName} status must be a JSON object")
+ return payload
+
+
+def _run_worker_in_subprocess(
+ *,
+ stageLabel: str,
+ workerModule: str,
+ storeUri: str,
+ workflow: WorkflowParameters,
+ resources: StageResources,
+ workDir: Path | None,
+ workDirPrefix: str,
+) -> dict[str, Any]:
+ worker_dir = (
+ workDir if workDir is not None else Path(tempfile.mkdtemp(prefix=workDirPrefix))
+ )
+ worker_dir.mkdir(parents=True, exist_ok=True)
+ request_path = worker_dir / "request.json"
+ status_path = worker_dir / "status.json"
+ status_path.unlink(missing_ok=True)
+ request_path.write_text(
+ json.dumps(
+ {
+ "storeUri": storeUri,
+ "workflow": workflow.model_dump(mode="json"),
+ "resources": resources.model_dump(mode="json"),
+ "statusPath": str(status_path),
+ }
+ ),
+ encoding="utf-8",
+ )
+
+ command = [
+ sys.executable,
+ "-m",
+ workerModule,
+ "--request",
+ str(request_path),
+ ]
+ process = subprocess.Popen(command)
+ print(
+ f"[{stageLabel}] child started pid={process.pid} module={workerModule} "
+ f"warningSeconds={CHILD_WARNING_SECONDS:.0f}",
+ flush=True,
+ )
+ try:
+ return_code = _monitor_child_process(process, stageLabel=stageLabel)
+ except BaseException:
+ _stop_child_process(process)
+ raise
+
+ status = _read_worker_status(status_path, workerName=f"{stageLabel} worker")
+ if return_code != 0:
+ detail = status.get("error") if status is not None else None
+ suffix = f": {detail}" if isinstance(detail, str) else ""
+ raise RuntimeError(
+ f"{stageLabel} worker exited with code {return_code}{suffix}"
+ )
+ if status is None or status.get("status") != "ok":
+ raise RuntimeError(f"{stageLabel} worker exited without a successful status")
+ print(f"[{stageLabel}] child completed pid={process.pid}", flush=True)
+ return status
+
+
+def _run_leiden_in_subprocess(
+ *,
+ storeUri: str,
+ workflow: WorkflowParameters,
+ resources: StageResources,
+ workDir: Path | None,
+) -> dict[str, Any]:
+ return _run_worker_in_subprocess(
+ stageLabel="runLeiden",
+ workerModule="profiling.leiden_worker",
+ storeUri=storeUri,
+ workflow=workflow,
+ resources=resources,
+ workDir=workDir,
+ workDirPrefix="scarf-leiden-",
+ )
+
+
+def _run_paris_in_subprocess(
+ *,
+ storeUri: str,
+ workflow: WorkflowParameters,
+ resources: StageResources,
+ workDir: Path | None,
+) -> dict[str, Any]:
+ return _run_worker_in_subprocess(
+ stageLabel="runClustering",
+ workerModule="profiling.paris_worker",
+ storeUri=storeUri,
+ workflow=workflow,
+ resources=resources,
+ workDir=workDir,
+ workDirPrefix="scarf-paris-",
+ )
+
+
+def _peak_cgroup_bytes(measurement: ResourceMeasurement | None) -> int | None:
+ if measurement is None:
+ return None
+ if measurement.operationPeakSource in {"cgroupMemoryCurrent", "cgroupMemoryPeak"}:
+ return measurement.operationPeakBytes
+ return measurement.cgroupMemoryCurrentPeakBytes
+
+
+def summarize_resource_measurement(
+ measurement: ResourceMeasurement | None,
+) -> dict[str, Any]:
+ return {
+ "peakRssBytes": (
+ measurement.processTreeRssPeakBytes if measurement is not None else None
+ ),
+ "peakCgroupBytes": _peak_cgroup_bytes(measurement),
+ "rssBaselineBytes": (
+ measurement.processTreeRssBaselineBytes if measurement is not None else None
+ ),
+ "rssIncrementalPeakBytes": (
+ measurement.processTreeRssIncrementalPeakBytes
+ if measurement is not None
+ else None
+ ),
+ "rssAfterBytes": (
+ measurement.processTreeRssAfterBytes if measurement is not None else None
+ ),
+ "cgroupCurrentBaselineBytes": (
+ measurement.cgroupMemoryCurrentBaselineBytes
+ if measurement is not None
+ else None
+ ),
+ "cgroupCurrentPeakBytes": (
+ measurement.cgroupMemoryCurrentPeakBytes
+ if measurement is not None
+ else None
+ ),
+ "cgroupCurrentAfterBytes": (
+ measurement.cgroupMemoryCurrentAfterBytes
+ if measurement is not None
+ else None
+ ),
+ "operationBaselineBytes": (
+ measurement.operationBaselineBytes if measurement is not None else None
+ ),
+ "operationIncrementalPeakBytes": (
+ measurement.operationIncrementalPeakBytes
+ if measurement is not None
+ else None
+ ),
+ "operationPeakSource": (
+ measurement.operationPeakSource if measurement is not None else None
+ ),
+ "cgroupPeakScope": (
+ measurement.cgroupMemoryPeakScope if measurement is not None else None
+ ),
+ }
+
+
+@dataclass(frozen=True, slots=True)
+class _CountsTWriteContext:
+ counts: Any
+ group: Any
+ budget: Any
+ beforeComplete: Any
+
+
+def _prepare_counts_t_write(
+ *,
+ storeUri: str,
+ assayName: str,
+ resources: StageResources,
+) -> _CountsTWriteContext:
+ budget = resolve_budget(
+ memory=resources.scarfMemoryBudget,
+ workers=resources.workers,
+ )
+ root = open_store(
+ storeUri,
+ mode="r+",
+ storage_options=storage_options(storeUri),
+ )
+ group = as_zarr_group(root[assayName], name=assayName)
+ counts = as_zarr_array(group["counts"], name=f"{assayName}/counts")
+ before_complete = None
+ if "countsT" in group:
+ before_complete = group["countsT"].attrs.get("complete")
+ return _CountsTWriteContext(
+ counts=counts,
+ group=group,
+ budget=budget,
+ beforeComplete=before_complete,
+ )
+
+
+def _write_counts_t(
+ context: _CountsTWriteContext,
+ *,
+ storeUri: str,
+ assayName: str,
+ featureMajorLayout: bool = False,
+) -> Any:
+ counts_t = write_counts_t(
+ context.counts,
+ context.group,
+ profile="cloud" if storeUri.startswith("s3://") else "fast_local",
+ resources=context.budget,
+ feature_major_layout=featureMajorLayout,
+ )
+ if counts_t is None:
+ raise RuntimeError(
+ f"write_counts_t skipped for {assayName} (Zarr format < 3); "
+ "cannot create countsT"
+ )
+ return counts_t
+
+
+def _validate_counts_t(
+ context: _CountsTWriteContext,
+ countsT: Any,
+ *,
+ storeUri: str,
+ assayName: str,
+ resources: StageResources,
+ nCheckTiles: int,
+ seed: int,
+) -> dict[str, Any]:
+ if countsT.attrs.get("complete") is not True:
+ raise RuntimeError(
+ f"countsT rewrite finished without complete=True at {storeUri}"
+ )
+ countsT.attrs["complete"] = False
+
+ expected_shape = (int(context.counts.shape[1]), int(context.counts.shape[0]))
+ if tuple(countsT.shape) != expected_shape:
+ raise RuntimeError(
+ f"countsT shape {tuple(countsT.shape)} != expected {expected_shape}"
+ )
+ if np.dtype(countsT.dtype) != np.dtype(context.counts.dtype):
+ raise RuntimeError(
+ f"countsT dtype {countsT.dtype} != counts dtype {context.counts.dtype}"
+ )
+
+ rng = np.random.default_rng(seed)
+ feat_chunk = max(1, int(countsT.chunks[0]))
+ cell_chunk = max(1, int(countsT.chunks[1]))
+ n_feats, n_cells = countsT.shape
+ checks: list[dict[str, Any]] = []
+ for _ in range(max(0, nCheckTiles)):
+ feat_start = int(rng.integers(0, n_feats))
+ feat_start = (feat_start // feat_chunk) * feat_chunk
+ cell_start = int(rng.integers(0, n_cells))
+ cell_start = (cell_start // cell_chunk) * cell_chunk
+ feat_end = min(feat_start + feat_chunk, n_feats)
+ cell_end = min(cell_start + cell_chunk, n_cells)
+ got = np.asarray(countsT[feat_start:feat_end, cell_start:cell_end])
+ expect = np.asarray(context.counts[cell_start:cell_end, feat_start:feat_end]).T
+ if got.shape != expect.shape or not np.array_equal(got, expect):
+ raise RuntimeError(
+ "countsT tile mismatch after rewrite "
+ f"feat=[{feat_start}:{feat_end}] cell=[{cell_start}:{cell_end}]"
+ )
+ checks.append(
+ {
+ "featStart": feat_start,
+ "featEnd": feat_end,
+ "cellStart": cell_start,
+ "cellEnd": cell_end,
+ }
+ )
+
+ countsT.attrs["complete"] = True
+ return {
+ "assayName": assayName,
+ "beforeComplete": context.beforeComplete,
+ "complete": True,
+ "shape": list(countsT.shape),
+ "chunks": list(countsT.chunks),
+ "dtype": str(countsT.dtype),
+ "workers": resources.workers,
+ "checkedTiles": checks,
+ }
+
+
+def run_stage(
+ stage: StageName,
+ *,
+ nRows: int,
+ storeUri: str,
+ workflow: WorkflowParameters,
+ resources: StageResources,
+ localH5adPath: Path | None = None,
+ storageLayout: StorageLayout | None = None,
+ workDir: Path | None = None,
+ sampleIntervalSeconds: float = 0.25,
+ containerMemoryMb: int | None = None,
+ containerCpuRequest: float | None = None,
+ containerCpuLimit: float | None = None,
+ resetCgroupPeak: bool = True,
+ invalidateCache: bool = False,
+) -> StageRunResult:
+ timer = StageTimer()
+ sampler = ResourceSampler(
+ sampleIntervalSeconds=sampleIntervalSeconds,
+ resetCgroupPeak=resetCgroupPeak,
+ )
+ error: str | None = None
+ status = "ok"
+ measurement: ResourceMeasurement | None = None
+ details: dict[str, Any] | None = None
+ worker_timings: dict[str, Any] | None = None
+
+ with timer:
+ sampler.start()
+ try:
+ if stage == "createStore":
+ if localH5adPath is None:
+ raise ValueError("createStore requires localH5adPath")
+ reader: H5adReader | None = None
+ writer: H5adToZarr | None = None
+ try:
+ with timer.inputSetup():
+ reader, writer = _prepare_create_store(
+ localH5adPath=localH5adPath,
+ storeUri=storeUri,
+ workflow=workflow,
+ resources=resources,
+ storageLayout=storageLayout,
+ )
+ with timer.operation():
+ assert writer is not None
+ writer.dump(batch_size=workflow.h5adBatchSize)
+ finally:
+ writer = None
+ if reader is not None:
+ _close_h5ad_reader(reader)
+ reader = None
+ elif stage == "writeCountsT":
+ counts_context: _CountsTWriteContext | None = None
+ counts_t: Any = None
+ try:
+ with timer.inputSetup():
+ counts_context = _prepare_counts_t_write(
+ storeUri=storeUri,
+ assayName=workflow.assayName,
+ resources=resources,
+ )
+ with timer.operation():
+ assert counts_context is not None
+ counts_t = _write_counts_t(
+ counts_context,
+ storeUri=storeUri,
+ assayName=workflow.assayName,
+ featureMajorLayout=(
+ workflow.countsTLayout == "featureMajor"
+ ),
+ )
+ with timer.validationPersistence():
+ assert counts_context is not None
+ details = _validate_counts_t(
+ counts_context,
+ counts_t,
+ storeUri=storeUri,
+ assayName=workflow.assayName,
+ resources=resources,
+ nCheckTiles=3,
+ seed=0,
+ )
+ finally:
+ counts_t = None
+ counts_context = None
+ elif stage == "initializeStore":
+ store: DataStore | None = None
+ try:
+ with timer.operation():
+ store = _open_datastore(
+ storeUri, workflow, resources, initialize=True
+ )
+ finally:
+ store = None
+ elif stage == "reopenStore":
+ store = None
+ try:
+ with timer.operation():
+ store = _open_datastore(
+ storeUri, workflow, resources, initialize=False
+ )
+ finally:
+ store = None
+ elif stage == "runLeiden":
+ with timer.operation():
+ worker_timings = _run_leiden_in_subprocess(
+ storeUri=storeUri,
+ workflow=workflow,
+ resources=resources,
+ workDir=workDir,
+ )
+ elif stage == "runClustering":
+ with timer.operation():
+ worker_timings = _run_paris_in_subprocess(
+ storeUri=storeUri,
+ workflow=workflow,
+ resources=resources,
+ workDir=workDir,
+ )
+ else:
+ store = None
+ try:
+ with timer.inputSetup():
+ print(
+ f"[run_stage] ENTER open_datastore stage={stage} "
+ f"store={storeUri}",
+ flush=True,
+ )
+ store = _open_datastore(
+ storeUri, workflow, resources, initialize=False
+ )
+ with timer.operation():
+ assert store is not None
+ print(
+ f"[run_stage] datastore open; ENTER analysis stage={stage}",
+ flush=True,
+ )
+ _run_analysis(
+ stage,
+ store,
+ workflow,
+ resources,
+ invalidateCache=invalidateCache,
+ )
+ print(
+ f"[run_stage] analysis DONE stage={stage}",
+ flush=True,
+ )
+ finally:
+ store = None
+ except Exception as exc:
+ status = "error"
+ error = f"{type(exc).__name__}: {exc}"
+ finally:
+ measurement = sampler.stop()
+
+ timings = timer.result
+ seconds = timings.measuredOperationSeconds
+ input_setup_seconds = timings.inputSetupSeconds
+ if worker_timings is not None:
+ worker_setup = worker_timings.get("inputSetupSeconds")
+ worker_operation = worker_timings.get("operationSeconds")
+ worker_whole = worker_timings.get("wholeWorkerSeconds")
+ if isinstance(worker_setup, int | float) and not isinstance(worker_setup, bool):
+ input_setup_seconds = float(worker_setup)
+ if isinstance(worker_operation, int | float) and not isinstance(
+ worker_operation, bool
+ ):
+ seconds = float(worker_operation)
+ if isinstance(worker_whole, int | float) and not isinstance(worker_whole, bool):
+ details = {
+ "subprocessSeconds": timings.measuredOperationSeconds,
+ "workerWholeSeconds": float(worker_whole),
+ }
+ if stage == "createStore" and details is None:
+ details = {"countsWriteSeconds": seconds}
+ resource_summary = summarize_resource_measurement(measurement)
+ return StageRunResult(
+ stage=stage,
+ nRows=nRows,
+ status=status,
+ seconds=seconds,
+ modalMemoryMb=(
+ resources.modalMemoryLimitMb
+ if containerMemoryMb is None
+ else containerMemoryMb
+ ),
+ modalCpuRequest=(
+ resources.modalCpuRequest
+ if containerCpuRequest is None
+ else containerCpuRequest
+ ),
+ modalCpuLimit=(
+ resources.modalCpuLimit if containerCpuLimit is None else containerCpuLimit
+ ),
+ scarfMemoryBudget=resources.scarfMemoryBudget,
+ storeUri=storeUri,
+ error=error,
+ inputSetupSeconds=input_setup_seconds,
+ validationPersistenceSeconds=timings.validationPersistenceSeconds,
+ wholeFunctionSeconds=timings.wholeFunctionSeconds,
+ details=details,
+ **resource_summary,
+ )
+
+
+def _run_analysis(
+ stage: StageName,
+ store: DataStore,
+ workflow: WorkflowParameters,
+ resources: StageResources,
+ *,
+ invalidateCache: bool = False,
+) -> None:
+ if stage == "filterCells":
+ store.auto_filter_cells(
+ attrs=workflow.filterAttrs,
+ min_p=workflow.filterMinQuantile,
+ max_p=workflow.filterMaxQuantile,
+ show_qc_plots=False,
+ invalidate_cache=invalidateCache,
+ )
+ return
+ if stage == "markHvgs":
+ store.mark_hvgs(
+ from_assay=workflow.assayName,
+ cell_key=workflow.cellKey,
+ min_cells=workflow.hvgMinCells,
+ top_n=workflow.topN,
+ show_plot=False,
+ hvg_key_name=workflow.hvgKey,
+ invalidate_cache=invalidateCache,
+ )
+ return
+ if stage == "runNormalization":
+ store.run_normalization(
+ from_assay=workflow.assayName,
+ cell_key=workflow.cellKey,
+ feat_key=workflow.hvgKey,
+ update_state=True,
+ invalidate_cache=invalidateCache,
+ )
+ return
+ if stage == "runPca":
+ store.run_pca(
+ from_assay=workflow.assayName,
+ dims=workflow.dims,
+ local_cache=workflow.graphLocalCache,
+ show_elbow_plot=False,
+ update_state=True,
+ invalidate_cache=invalidateCache,
+ )
+ return
+ if stage == "buildEmbeddingInitialization":
+ store.build_embedding_initialization(
+ from_assay=workflow.assayName,
+ n_centroids=workflow.nCentroids,
+ rand_state=workflow.graphSeed,
+ kmeans_sampling=workflow.kmeansSampling,
+ kmeans_batch_size=workflow.kmeansBatchSize,
+ update_state=True,
+ invalidate_cache=invalidateCache,
+ )
+ return
+ if stage == "buildAnnIndex":
+ store.build_ann_index(
+ from_assay=workflow.assayName,
+ ann_efc=min(100, max(workflow.k * 3, 50)),
+ ann_ef=min(100, max(workflow.k * 3, 50)),
+ ann_m=min(max(48, int(workflow.dims * 1.5)), 64),
+ ann_parallel=workflow.annParallel,
+ rand_state=workflow.graphSeed,
+ update_state=True,
+ invalidate_cache=invalidateCache,
+ )
+ return
+ if stage == "queryNeighbors":
+ store.query_neighbors(
+ from_assay=workflow.assayName,
+ k=workflow.k,
+ update_state=True,
+ invalidate_cache=invalidateCache,
+ )
+ return
+ if stage == "buildConnectivityMap":
+ store.build_connectivity_map(
+ from_assay=workflow.assayName,
+ update_state=True,
+ invalidate_cache=invalidateCache,
+ )
+ return
+ if stage == "runUmap":
+ store.run_umap(
+ from_assay=workflow.assayName,
+ cell_key=workflow.cellKey,
+ feat_key=workflow.hvgKey,
+ n_epochs=workflow.umapEpochs,
+ random_seed=workflow.umapSeed,
+ label=workflow.umapLabel,
+ parallel=workflow.umapParallel,
+ nthreads=resources.workers,
+ invalidate_cache=invalidateCache,
+ )
+ return
+ if stage == "runLeiden":
+ raise AssertionError("runLeiden must execute in its child process")
+ if stage == "findMarkers":
+ store.run_marker_search(
+ from_assay=workflow.assayName,
+ group_key=workflow.resolvedMarkerGroupKey,
+ cell_key=workflow.cellKey,
+ feat_key=workflow.markerFeatureKey,
+ gene_batch_size=workflow.markerGeneBatchSize,
+ n_threads=resources.workers,
+ skip_save=False,
+ invalidate_cache=invalidateCache,
+ )
+ return
+ if stage == "runClustering":
+ raise AssertionError("runClustering must execute in its child process")
+ raise ValueError(f"No analysis operation for {stage}")
diff --git a/pypi_README.rst b/pypi_README.rst
deleted file mode 100644
index ae51fb2f..00000000
--- a/pypi_README.rst
+++ /dev/null
@@ -1,30 +0,0 @@
-|PyPI| |Docs| |Tests| |Coverage| |Downloads| |Black| |Slack|
-
-`Installation`_ | `Documentation`_ | `Tutorials`_ | `Research-Article`_
-
-|IMG1|
-
-.. |PyPI| image:: https://img.shields.io/pypi/v/scarf.svg
- :target: https://pypi.org/project/scarf
-.. |Docs| image:: https://readthedocs.org/projects/scarf/badge/?version=latest
- :target: https://scarf.readthedocs.io
-.. |Tests| image:: https://github.com/parashardhapola/scarf/actions/workflows/pytest.yml/badge.svg
- :target: https://github.com/parashardhapola/scarf/actions/workflows/pytest.yml
-.. |Coverage| image:: https://codecov.io/gh/parashardhapola/scarf/branch/master/graph/badge.svg?token=ZvJXuYq3pd
- :target: https://codecov.io/gh/parashardhapola/scarf
-.. |Downloads| image:: https://pepy.tech/badge/scarf
- :target: https://pepy.tech/project/scarf
-.. |Black| image:: https://img.shields.io/badge/code%20style-black-000000.svg
- :target: https://github.com/psf/black
-.. |Slack| image:: https://img.shields.io/badge/Slack-4A154B?style=for-the-badge&logo=slack&logoColor=white
- :target: https://scarf-group.slack.com/archives/C0418C7RXU4
-
-.. |IMG1| image:: https://raw.githubusercontent.com/parashardhapola/scarf/master/docs/source/logo_wide.png
- :target: https://github.com/parashardhapola/scarf
-
-Analyze large-scale (aka atlas-scale) single-cell genomics datasets with very low memory requirement.
-
-.. _Installation: https://scarf.readthedocs.io/en/latest/install.html
-.. _Documentation: http://scarf.rtfd.io
-.. _Tutorials: https://scarf.readthedocs.io/en/latest/vignettes/basic_tutorial_scRNAseq.html
-.. _Research-Article: https://www.nature.com/articles/s41467-022-32097-3
diff --git a/pyproject.toml b/pyproject.toml
index 65c8f21f..fb2fa3f5 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,20 +1,132 @@
-[build.system]
-requires = [
- "setuptools",
- "wheel"
-]
+[build-system]
+requires = ["setuptools>=61", "wheel"]
build-backend = "setuptools.build_meta"
+[project]
+name = "scarf"
+dynamic = ["version"]
+description = "Out-of-core, graph-first workflows for million-cell RNA-seq and CITE-seq"
+readme = {file = "README.md", content-type = "text/markdown"}
+requires-python = ">=3.12"
+license = "BSD-3-Clause"
+authors = [
+ {name = "Parashar Dhapola", email = "parashar.dhapola@gmail.com"},
+]
+keywords = [
+ "single-cell",
+ "scRNA-seq",
+ "scATAC-seq",
+ "CITE-seq",
+ "Harmony",
+ "WNN",
+ "batch-correction",
+ "Zarr",
+ "memory-efficient",
+]
+classifiers = [
+ "Natural Language :: English",
+ "Programming Language :: Python :: 3",
+ "Programming Language :: Python :: 3.12",
+ "Programming Language :: Python :: 3.13",
+ "Programming Language :: Python :: 3.14",
+]
+dependencies = [
+ "h5py>=3.11",
+ "hnswlib>=0.8",
+ "leidenalg>=0.12",
+ "loguru>=0.7",
+ "networkx>=3",
+ "numba>=0.60",
+ "numcodecs>=0.16",
+ "numpy>=2",
+ "packaging",
+ "pandas>=2.2.2",
+ "scikit-learn>=1.5",
+ "scipy>=1.13,!=1.17.0",
+ "setuptools>=61",
+ "statsmodels>=0.14.2",
+ "threadpoolctl",
+ "tqdm>=4.60",
+ "umap-learn>=0.5.12",
+ "zarr>=3.2.0",
+ "obstore>=0.5.1",
+ "sgtsnepi",
+ "huggingface-hub>=1.24.0",
+ "zstandard>=0.25.0",
+]
+
+[project.optional-dependencies]
+extra = [
+ "anndata>=0.12",
+ "kneed>=0.8",
+ "matplotlib>=3.9",
+ "seaborn>=0.13",
+ "ipython-autotime>=0.3",
+ "ipywidgets>=8",
+]
+test = [
+ "pytest>=7.0",
+ "pytest-cov>=4",
+ "pytest-xdist>=3",
+ "topacedo>=0.2.10",
+ "anndata>=0.12",
+ "scikit-network>=0.33.1",
+ "rdata>=1.1.0",
+]
+docs = [
+ "jinja2",
+ "jedi",
+ "myst-parser",
+ "myst-nb",
+ "sphinx",
+ "sphinx_autodoc_typehints",
+ "sphinx-book-theme",
+ "sphinx-copybutton",
+ "sphinx-external-toc",
+ "sphinx-tabs",
+ "topacedo",
+ "nbclient",
+ "jupyter-cache>=1.0.1",
+ "sphinx-reredirects>=1.1.0",
+ "sphinxcontrib-mermaid>=2.1.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/parashardhapola/scarf"
+Documentation = "https://scarf.readthedocs.io/en/latest/"
+Installation = "https://scarf.readthedocs.io/en/latest/installation.html"
+Issues = "https://github.com/parashardhapola/scarf/issues"
+
+[tool.setuptools]
+include-package-data = false
+
+[tool.setuptools.dynamic]
+version = {file = "VERSION"}
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["scarf*"]
+exclude = ["profiling*", "scripts*", "tests*", "docs*", "bin*"]
+
[tool.pytest.ini_options]
minversion = "6.0"
-addopts = "-ra -v"
+addopts = "-ra -v -n auto --dist loadfile"
+markers = [
+ "integration: live network tests",
+ "slow: tests that build graphs or run iterative numerical workflows",
+ "visual: deterministic image comparison tests",
+]
testpaths = [
- "scarf/tests",
+ "tests",
]
norecursedirs = [
- "scarf/tests"
+ "datasets",
]
+[tool.coverage.run]
+source = ["scarf"]
+parallel = true
+
[tool.interrogate]
ignore-init-method = true
ignore-init-module = false
@@ -30,7 +142,7 @@ fail-under = 50
exclude = [
"setup.py",
"docs",
- "scarf/tests"
+ "tests"
]
ignore-regex = [
"^get$",
@@ -43,3 +155,75 @@ whitelist-regex = []
color = true
generate-badge = '.'
badge-format = "svg"
+
+[dependency-groups]
+profiling = [
+ "anndata>=0.12.18",
+ "modal>=1.5.2",
+ "pydantic>=2.13.4",
+]
+dev = [
+ "mypy>=2.1.0",
+ "ruff>=0.15.20",
+]
+docs-modal = [
+ "modal>=1.5.2",
+]
+
+[tool.uv]
+no-binary-package = ["pcst-fast", "hnswlib"]
+extra-build-variables = { hnswlib = { HNSWLIB_NO_NATIVE = "1" } }
+
+[tool.uv.pip]
+no-binary = ["hnswlib"]
+extra-build-variables = { hnswlib = { HNSWLIB_NO_NATIVE = "1" } }
+
+[tool.uv.extra-build-dependencies]
+pcst-fast = ["pybind11>=2.11"]
+
+[tool.ruff]
+exclude = [
+ ".git",
+ ".venv",
+ "docs/.jupyter_cache",
+ "docs/build",
+ "docs/source/tutorials/scarf_datasets",
+]
+
+[tool.mypy]
+strict = false
+
+python_version = "3.12"
+exclude = ["^docs/", "^setup\\.py$"]
+
+disable_error_code = ["import-untyped"]
+
+disallow_untyped_defs = true
+disallow_incomplete_defs = true
+
+check_untyped_defs = true
+disallow_untyped_calls = true
+no_implicit_optional = true
+
+warn_return_any = true
+warn_unused_configs = true
+warn_redundant_casts = true
+warn_unused_ignores = false
+
+ignore_missing_imports = false
+
+[[tool.mypy.overrides]]
+module = [
+ "scarf.plotting",
+ "scarf.plotting.*",
+]
+ignore_missing_imports = true
+
+[[tool.mypy.overrides]]
+module = ["profiling.*"]
+disallow_untyped_defs = false
+disallow_incomplete_defs = false
+disallow_untyped_calls = false
+check_untyped_defs = false
+warn_return_any = false
+ignore_errors = true
diff --git a/requirements.txt b/requirements.txt
deleted file mode 100644
index d5866a5d..00000000
--- a/requirements.txt
+++ /dev/null
@@ -1,24 +0,0 @@
-dask[array]<=2024.7.1
-h5py>=3.3.0
-hnswlib==0.8.0
-importlib_metadata
-joblib
-leidenalg
-loguru
-networkx==3.4.2
-numba<=0.61.2
-numcodecs<0.16
-numpy==1.26.4
-packaging
-pandas
-polars<=1.25
-pyarrow
-scikit-learn==1.6.1
-scikit-network<=0.33.2
-scipy==1.15.2
-setuptools
-statsmodels
-threadpoolctl
-tqdm
-umap-learn
-zarr<=2.16.0
diff --git a/requirements_extra.txt b/requirements_extra.txt
deleted file mode 100644
index 1bffaace..00000000
--- a/requirements_extra.txt
+++ /dev/null
@@ -1,9 +0,0 @@
-requests
-gensim
-kneed
-matplotlib
-seaborn
-cmocean
-datashader
-ipython-autotime
-ipywidgets
diff --git a/scarf/__init__.py b/scarf/__init__.py
index 54525001..37e4cd2a 100644
--- a/scarf/__init__.py
+++ b/scarf/__init__.py
@@ -16,12 +16,17 @@
--------
- Modules
+ - cytebase: Browse and download public datasets from Cytebase.
- datastore: Contains the primary interface to interact with data (i.e. DataStore) and its superclasses.
- - readers: A collection of classes for reading in different data formats.
+ - embeddings: Dimension reduction, Harmony correction, and cell embeddings.
+ - matrix: Lazy blockwise matrix operations over NumPy and Zarr arrays.
+ - readers: Classes for reading supported data formats.
- writers: Methods and classes for writing data to disk.
- - meld_assay:
+ - features: Variability, marker, scoring, and genomic feature algorithms.
+ - mapping: Reference mapping, alignment, and confidence algorithms.
+ - metrics: Integration and neighborhood quality metrics.
+ - quality_control: Quality-control algorithms and cell-cycle references.
- utils: Utility methods.
- - downloader: Used to download datasets included in Scarf.
GitHub: https://github.com/parashardhapola/scarf
@@ -32,23 +37,225 @@
PyPI: https://pypi.org/project/scarf/
"""
-import warnings
+import warnings as _warnings
+from importlib import import_module as _import_module
+from importlib.metadata import PackageNotFoundError as _PackageNotFoundError
+from importlib.metadata import version as _distribution_version
+from pathlib import Path as _Path
+from typing import TYPE_CHECKING, Any
-from dask.array import PerformanceWarning
-from importlib_metadata import version
+if TYPE_CHECKING:
+ from . import cytebase as cytebase
+ from .datastore.datastore import DataStore as DataStore
+ from .datastore.summary import DataStoreSummary as DataStoreSummary
+ from .graph.state import (
+ ArtifactSelectionError as ArtifactSelectionError,
+ AssayState as AssayState,
+ )
+ from .lineage import ArtifactLineage as ArtifactLineage
+ from .features.enrichment import (
+ EnrichmentResult as EnrichmentResult,
+ read_gmt as read_gmt,
+ )
+ from .mapping.models import MappingResult as MappingResult
+ from .mapping.reference import MappingReference as MappingReference
+ from .storage.artifacts import ArtifactStatus as ArtifactStatus
+ from .storage.refs import ArtifactRef as ArtifactRef
+ from .features.genomic.gff import GffReader as GffReader
+ from .features.genomic.melding import coordinate_melding as coordinate_melding
+ from .merge import (
+ AssayMerge as AssayMerge,
+ DatasetMerge as DatasetMerge,
+ )
+ from .readers import (
+ CSVReader as CSVReader,
+ CrDirReader as CrDirReader,
+ CrH5Reader as CrH5Reader,
+ CrReader as CrReader,
+ H5adInspectResult as H5adInspectResult,
+ H5adReader as H5adReader,
+ LoomReader as LoomReader,
+ MtxReader as MtxReader,
+ SeuratInspectResult as SeuratInspectResult,
+ SeuratReader as SeuratReader,
+ inspect_h5ad as inspect_h5ad,
+ inspect_mtx as inspect_mtx,
+ inspect_seurat as inspect_seurat,
+ )
+ from .trajectory.results import (
+ FateMappingResult as FateMappingResult,
+ PseudotimeAggregationResult as PseudotimeAggregationResult,
+ PseudotimeMarkerResult as PseudotimeMarkerResult,
+ PseudotimeScoreResult as PseudotimeScoreResult,
+ )
+ from .utils import (
+ clean_array as clean_array,
+ configure_output as configure_output,
+ controlled_compute as controlled_compute,
+ get_log_level as get_log_level,
+ load_zarr as load_zarr,
+ logger as logger,
+ permute_into_chunks as permute_into_chunks,
+ rescale_array as rescale_array,
+ rolling_window as rolling_window,
+ set_verbosity as set_verbosity,
+ show_dask_progress as show_dask_progress,
+ system_call as system_call,
+ tqdmbar as tqdmbar,
+ tqdm_params as tqdm_params,
+ )
+ from .writers import (
+ CSVtoZarr as CSVtoZarr,
+ CrToZarr as CrToZarr,
+ H5adToZarr as H5adToZarr,
+ LoomToZarr as LoomToZarr,
+ MtxToZarr as MtxToZarr,
+ SeuratImportResult as SeuratImportResult,
+ SeuratToZarr as SeuratToZarr,
+ SparseToZarr as SparseToZarr,
+ SubsetZarr as SubsetZarr,
+ create_zarr_count_assay as create_zarr_count_assay,
+ create_zarr_dataset as create_zarr_dataset,
+ create_zarr_obj_array as create_zarr_obj_array,
+ dask_to_zarr as dask_to_zarr,
+ subset_assay_zarr as subset_assay_zarr,
+ to_h5ad as to_h5ad,
+ to_mtx as to_mtx,
+ write_renorm_subset_to_zarr as write_renorm_subset_to_zarr,
+ )
-warnings.filterwarnings("ignore", category=DeprecationWarning)
-warnings.filterwarnings("ignore", category=PerformanceWarning)
+_warnings.filterwarnings(
+ "ignore",
+ message=r"The data type .* does not have a Zarr V3 specification\.",
+ module=r"zarr\.core\.dtype\..*",
+)
try:
- __version__ = version("scarf")
-except ImportError:
- print("Scarf is not installed", flush=True)
-
-from .datastore.datastore import DataStore
-from .readers import *
-from .writers import *
-from .meld_assay import *
-from .utils import *
-from .downloader import *
-from .merge import *
\ No newline at end of file
+ __version__ = _distribution_version("scarf")
+except _PackageNotFoundError:
+ _version_path = _Path(__file__).resolve().parents[1] / "VERSION"
+ __version__ = (
+ _version_path.read_text().strip() if _version_path.is_file() else "unavailable"
+ )
+
+_LAZY_EXPORTS: dict[str, tuple[str, str]] = {
+ "ArtifactLineage": (".lineage", "ArtifactLineage"),
+ "ArtifactRef": (".storage.refs", "ArtifactRef"),
+ "ArtifactSelectionError": (".graph.state", "ArtifactSelectionError"),
+ "ArtifactStatus": (".storage.artifacts", "ArtifactStatus"),
+ "AssayMerge": (".merge", "AssayMerge"),
+ "AssayState": (".graph.state", "AssayState"),
+ "CSVReader": (".readers", "CSVReader"),
+ "CSVtoZarr": (".writers", "CSVtoZarr"),
+ "CrDirReader": (".readers", "CrDirReader"),
+ "CrH5Reader": (".readers", "CrH5Reader"),
+ "CrReader": (".readers", "CrReader"),
+ "CrToZarr": (".writers", "CrToZarr"),
+ "DataStore": (".datastore.datastore", "DataStore"),
+ "DataStoreSummary": (".datastore.summary", "DataStoreSummary"),
+ "DatasetMerge": (".merge", "DatasetMerge"),
+ "EnrichmentResult": (".features.enrichment", "EnrichmentResult"),
+ "FateMappingResult": (".trajectory.results", "FateMappingResult"),
+ "GffReader": (".features.genomic.gff", "GffReader"),
+ "H5adInspectResult": (".readers", "H5adInspectResult"),
+ "H5adReader": (".readers", "H5adReader"),
+ "H5adToZarr": (".writers", "H5adToZarr"),
+ "LoomReader": (".readers", "LoomReader"),
+ "LoomToZarr": (".writers", "LoomToZarr"),
+ "MtxReader": (".readers", "MtxReader"),
+ "MtxToZarr": (".writers", "MtxToZarr"),
+ "SeuratImportResult": (".writers", "SeuratImportResult"),
+ "SeuratInspectResult": (".readers", "SeuratInspectResult"),
+ "SeuratReader": (".readers", "SeuratReader"),
+ "SeuratToZarr": (".writers", "SeuratToZarr"),
+ "MappingReference": (".mapping.reference", "MappingReference"),
+ "MappingResult": (".mapping.models", "MappingResult"),
+ "mount_datastore": (".datastore.datastore", "mount_datastore"),
+ "PseudotimeAggregationResult": (
+ ".trajectory.results",
+ "PseudotimeAggregationResult",
+ ),
+ "PseudotimeMarkerResult": (
+ ".trajectory.results",
+ "PseudotimeMarkerResult",
+ ),
+ "PseudotimeScoreResult": (
+ ".trajectory.results",
+ "PseudotimeScoreResult",
+ ),
+ "SparseToZarr": (".writers", "SparseToZarr"),
+ "SubsetZarr": (".writers", "SubsetZarr"),
+ "clean_array": (".utils", "clean_array"),
+ "configure_output": (".utils", "configure_output"),
+ "controlled_compute": (".utils", "controlled_compute"),
+ "coordinate_melding": (".features.genomic.melding", "coordinate_melding"),
+ "create_zarr_count_assay": (".writers", "create_zarr_count_assay"),
+ "create_zarr_dataset": (".writers", "create_zarr_dataset"),
+ "create_zarr_obj_array": (".writers", "create_zarr_obj_array"),
+ "dask_to_zarr": (".writers", "dask_to_zarr"),
+ "get_log_level": (".utils", "get_log_level"),
+ "inspect_h5ad": (".readers", "inspect_h5ad"),
+ "inspect_mtx": (".readers", "inspect_mtx"),
+ "inspect_seurat": (".readers", "inspect_seurat"),
+ "load_zarr": (".utils", "load_zarr"),
+ "logger": (".utils", "logger"),
+ "permute_into_chunks": (".utils", "permute_into_chunks"),
+ "read_gmt": (".features.enrichment", "read_gmt"),
+ "rescale_array": (".utils", "rescale_array"),
+ "rolling_window": (".utils", "rolling_window"),
+ "set_verbosity": (".utils", "set_verbosity"),
+ "show_dask_progress": (".utils", "show_dask_progress"),
+ "subset_assay_zarr": (".writers", "subset_assay_zarr"),
+ "system_call": (".utils", "system_call"),
+ "to_h5ad": (".writers", "to_h5ad"),
+ "to_mtx": (".writers", "to_mtx"),
+ "tqdmbar": (".utils", "tqdmbar"),
+ "tqdm_params": (".utils", "tqdm_params"),
+ "write_renorm_subset_to_zarr": (
+ ".writers",
+ "write_renorm_subset_to_zarr",
+ ),
+}
+
+_LAZY_MODULES = frozenset(
+ {
+ "assay",
+ "cytebase",
+ "datastore",
+ "embeddings",
+ "features",
+ "mapping",
+ "matrix",
+ "merge",
+ "metadata",
+ "metrics",
+ "quality_control",
+ "readers",
+ "storage",
+ "utils",
+ "writers",
+ }
+)
+
+for _lazy_name in (*_LAZY_EXPORTS, *_LAZY_MODULES):
+ globals().pop(_lazy_name, None)
+del _lazy_name
+
+__all__ = list(_LAZY_EXPORTS)
+
+
+def __getattr__(name: str) -> Any:
+ if name in _LAZY_MODULES:
+ value = _import_module(f".{name}", __name__)
+ elif export := _LAZY_EXPORTS.get(name):
+ module_name, attribute_name = export
+ value = getattr(_import_module(module_name, __name__), attribute_name)
+ else:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()).union(_LAZY_EXPORTS, _LAZY_MODULES))
diff --git a/scarf/ann.py b/scarf/ann.py
deleted file mode 100644
index 845435e7..00000000
--- a/scarf/ann.py
+++ /dev/null
@@ -1,346 +0,0 @@
-from typing import Optional
-
-import dask.array as da
-import numpy as np
-import pandas as pd
-from threadpoolctl import threadpool_limits
-
-from .harmony import run_harmony
-from .utils import controlled_compute, logger, tqdmbar
-
-__all__ = ["AnnStream", "instantiate_knn_index", "fix_knn_query"]
-
-
-def instantiate_knn_index(
- space, dim, max_elements, ef_construction, M, random_seed, ef, num_threads
-):
- import hnswlib
-
- ann_idx = hnswlib.Index(space=space, dim=dim)
- ann_idx.init_index(
- max_elements=max_elements,
- ef_construction=ef_construction,
- M=M,
- random_seed=random_seed,
- )
- ann_idx.set_ef(ef)
- ann_idx.set_num_threads(num_threads)
- return ann_idx
-
-
-def fix_knn_query(indices: np.ndarray, distances: np.ndarray, ref_idx: np.ndarray):
- fixed_ind, fixed_dist = indices.copy()[:, 1:], distances.copy()[:, 1:]
- # Identify positions where first index is not a self loop
- mis_idx = indices[:, 0].reshape(1, -1)[0] != ref_idx
- n_mis = mis_idx.sum()
- if n_mis > 0:
- for n, i, j, k in zip(
- np.where(mis_idx)[0], ref_idx[mis_idx], indices[mis_idx], distances[mis_idx]
- ):
- p = np.where(j == i)[0]
- if len(p) > 0:
- # p is the position of self loop. We exclude this position
- p = p[0]
- j = np.array(list(j[:p]) + list(j[p + 1 :]))
- k = np.array(list(k[:p]) + list(k[p + 1 :]))
- else:
- # No self found at all. Poor recall? simply remove the last k neighbour
- j = j[:-1]
- k = k[:-1]
- fixed_ind[n] = j
- fixed_dist[n] = k
- return fixed_ind, fixed_dist, n_mis
-
-
-class AnnStream:
- def __init__(
- self,
- data,
- k: int,
- n_cluster: int,
- reduction_method: str,
- dims: int,
- loadings: np.ndarray,
- use_for_pca: np.ndarray,
- mu: np.ndarray,
- sigma: np.ndarray,
- ann_metric: str,
- ann_efc: int,
- ann_ef: int,
- ann_m: int,
- nthreads: int,
- ann_parallel: bool,
- rand_state: int,
- do_kmeans_fit: bool,
- disable_scaling: bool,
- ann_idx,
- lsi_skip_first: bool,
- lsi_params: dict,
- harmonize: bool,
- harmonized_data: Optional[da.Array] = None,
- batches: Optional[pd.DataFrame] = None,
- ):
- self.data = data
- self.k = k
- if self.k >= self.data.shape[0]:
- self.k = self.data.shape[0] - 1
- self.nClusters = max(n_cluster, 2)
- self.dims = dims
- self.loadings = loadings
- if self.dims is None and self.loadings is None:
- raise ValueError(
- "ERROR: Provide either value for atleast one: 'dims' or 'loadings'"
- )
- self.annMetric = ann_metric
- self.annEfc = ann_efc
- self.annEf = ann_ef
- self.annM = ann_m
- self.nthreads = nthreads
- if ann_parallel:
- self.annThreads = self.nthreads
- else:
- self.annThreads = 1
- self.randState = rand_state
- self.batchSize = self._handle_batch_size()
- self.method = reduction_method
- self.nCells, self.nFeats = self.data.shape
- self.clusterLabels: np.ndarray = np.repeat(-1, self.nCells)
- self.harmonize = harmonize
- self.harmonizedData = harmonized_data
- self.batches = batches
- disable_reduction = False
- if self.dims < 1:
- disable_reduction = True
- with threadpool_limits(limits=self.nthreads):
- if self.method == "pca":
- self.mu, self.sigma = mu, sigma
- if self.loadings is None or len(self.loadings) == 0:
- if len(use_for_pca) != self.nCells:
- raise ValueError(
- "ERROR: `use_for_pca` does not have sample length as nCells"
- )
- if disable_reduction is False:
- self._fit_pca(disable_scaling, use_for_pca)
- else:
- # Even though the dims might have been already adjusted according to loadings before calling
- # AnnStream, it could still be overwritten by _handle_batch_size. Hence need to hard set it here.
- self.dims = self.loadings.shape[1]
- # it is okay for dimensions to be larger than batch size here because we will not fit the PCA
- if disable_scaling:
- if disable_reduction:
- self.reducer = lambda x: x
- else:
- self.reducer = lambda x: x.dot(self.loadings)
- else:
- if disable_reduction:
- self.reducer = lambda x: self.transform_z(x)
- else:
- self.reducer = lambda x: self.transform_z(x).dot(self.loadings)
- elif self.method == "lsi":
- if self.loadings is None or len(self.loadings) == 0:
- if disable_reduction is False:
- self._fit_lsi(lsi_skip_first, lsi_params)
- else:
- # First dimension of LSI captures depth
- if lsi_skip_first:
- self.loadings = self.loadings[:, 1:]
- self.dims = self.loadings.shape[1]
- if disable_reduction:
- self.reducer = lambda x: x
- else:
- self.reducer = lambda x: x.dot(self.loadings)
- elif self.method == "custom":
- if self.loadings is None or len(self.loadings) == 0:
- logger.warning(
- "No loadings provided for manual dimension reduction"
- )
- else:
- self.dims = self.loadings.shape[1]
- if disable_reduction:
- self.reducer = lambda x: x
- else:
- self.reducer = lambda x: x.dot(self.loadings)
- else:
- raise ValueError(f"ERROR: Unknown reduction method: {self.method}")
- if ann_idx is None:
- self.annIdx = self._fit_ann()
- else:
- self.annIdx = ann_idx
- self.annIdx.set_ef(self.annEf)
- self.annIdx.set_num_threads(self.annThreads)
- self.kmeans = self._fit_kmeans(do_kmeans_fit)
-
- def _handle_batch_size(self):
- if self.dims > self.data.shape[0]:
- self.dims = self.data.shape[0]
- batch_size = self.data.chunksize[0] # Assuming all chunks are same size
- if self.dims >= batch_size:
- self.dims = batch_size - 1 # -1 because we will do PCA +1
- logger.info(
- f"Number of PCA/LSI components reduced to batch size of {batch_size}"
- )
- if self.nClusters > batch_size:
- self.nClusters = batch_size
- logger.info(f"Cluster number reduced to batch size of {batch_size}")
- return batch_size
-
- def iter_blocks(self, msg: str = "") -> np.ndarray:
- for i in tqdmbar(self.data.blocks, desc=msg, total=self.data.numblocks[0]):
- yield controlled_compute(i, self.nthreads)
-
- def transform_z(self, a: np.ndarray) -> np.ndarray:
- return (a - self.mu) / self.sigma
-
- def transform_ann(
- self, a: np.ndarray, k: int = None, self_indices: np.ndarray = None
- ) -> tuple:
- if k is None:
- k = self.k
- # Adding +1 to k because first neighbour will be the query itself
- if self_indices is None:
- i, d = self.annIdx.knn_query(a, k=k)
- return i, d
- else:
- i, d = self.annIdx.knn_query(a, k=k + 1)
- return fix_knn_query(i, d, self_indices)
-
- def _fit_pca(self, disable_scaling, use_for_pca) -> None:
- from sklearn.decomposition import IncrementalPCA
- from numpy.linalg import LinAlgError
-
- # We fit 1 extra PC dim than specified and then ignore the last PC.
- self._pca = IncrementalPCA(
- n_components=self.dims + 1, batch_size=self.batchSize
- )
- do_sample_subset = False if use_for_pca.sum() == self.nCells else True
- s, e = 0, 0
- # We store the first block of values here. if such a case arises that we are left with less dims+1 cells to fit
- # then those cells can be added to end_reservoir for fitting. if there are no such cells then end reservoir is
- # just by itself after fitting rest of the cells. If may be the case that the first batch itself has less than
- # dims+1 cells. in that we keep adding cells to carry_over pile until it is big enough.
- end_reservoir = []
- # carry_over store cells that can yet not be added to end_reservoir ot be used for fitting pca directly.
- carry_over = []
- for i in self.iter_blocks(msg="Fitting PCA"):
- if do_sample_subset:
- e = s + i.shape[0]
- i = i[use_for_pca[s:e]]
- s = e
- if disable_scaling is False:
- i = self.transform_z(i)
- if len(carry_over) > 0:
- i = np.vstack((carry_over, i))
- carry_over = []
- if len(i) < (self.dims + 1):
- carry_over = i
- continue
- if len(end_reservoir) == 0:
- end_reservoir = i
- continue
- try:
- self._pca.partial_fit(i, check_input=False)
- except LinAlgError:
- # Add retry counter to make memory consumption doesn't escalate
- carry_over = i
- if len(carry_over) > 0:
- i = np.vstack((end_reservoir, carry_over))
- else:
- i = end_reservoir
- try:
- self._pca.partial_fit(i, check_input=False)
- except LinAlgError:
- logger.warning(
- "{i.shape[0]} samples were not used in PCA fitting due to LinAlgError",
- flush=True,
- )
- self.loadings = self._pca.components_[:-1, :].T
-
- def _fit_lsi(self, lsi_skip_first, lsi_params) -> None:
- import warnings
-
- with warnings.catch_warnings():
- warnings.simplefilter("ignore", UserWarning)
- from gensim.models import LsiModel
- from gensim.matutils import Dense2Corpus
-
- for i in ["corpus", "num_topics", "id2word", "chunksize", "dtype"]:
- if i in lsi_params:
- del lsi_params[i]
- logger.warning(
- f"Provided parameter, {i}, for LSI model will not be used"
- )
- self._lsiModel = LsiModel(
- corpus=Dense2Corpus(
- controlled_compute(self.data.blocks[0], self.nthreads).T
- ),
- num_topics=self.dims + 1, # +1 because first dim will be discarded
- chunksize=self.data.chunksize[0],
- id2word={x: x for x in range(self.data.shape[1])},
- **lsi_params,
- )
- for n, i in enumerate(self.iter_blocks(msg="Fitting LSI model")):
- if n == 0:
- continue
- self._lsiModel.add_documents(Dense2Corpus(i.T))
- if lsi_skip_first:
- self.loadings = self._lsiModel.get_topics().T[:, 1:]
- else:
- self.loadings = self._lsiModel.get_topics().T
-
- def _fit_ann(self):
- def _transform_values():
- pca_array = []
- for _i in self.iter_blocks(msg="Calculating uncorrected latent dimensions"):
- pca_array.append(self.reducer(_i))
- return np.vstack(pca_array).T
-
- dims = self.dims
- if dims < 1:
- dims = self.data.shape[1]
-
- ann_idx = instantiate_knn_index(
- self.annMetric,
- dims,
- self.nCells,
- self.annEfc,
- self.annM,
- self.randState,
- self.annEf,
- self.annThreads,
- )
- if self.harmonize:
- if self.harmonizedData is None:
- self.harmonizedData = da.from_array(
- run_harmony(_transform_values(), self.batches).T,
- chunks=self.data.chunksize,
- )
- for i in tqdmbar(
- self.harmonizedData.blocks,
- desc="Fitting ANN",
- total=self.harmonizedData.numblocks[0],
- ):
- ann_idx.add_items(controlled_compute(i, self.nthreads))
- else:
- for i in self.iter_blocks(msg="Fitting ANN"):
- ann_idx.add_items(self.reducer(i))
- return ann_idx
-
- def _fit_kmeans(self, do_ann_fit):
- from sklearn.cluster import MiniBatchKMeans
-
- if do_ann_fit is False:
- return None
- kmeans = MiniBatchKMeans(
- n_clusters=self.nClusters,
- random_state=self.randState,
- batch_size=self.batchSize,
- n_init=3,
- )
- temp = []
- with threadpool_limits(limits=self.nthreads):
- for i in self.iter_blocks(msg="Fitting kmeans"):
- kmeans.partial_fit(self.reducer(i))
- for i in self.iter_blocks(msg="Estimating seed partitions"):
- temp.extend(kmeans.predict(self.reducer(i)))
- self.clusterLabels = np.array(temp)
- return kmeans
diff --git a/scarf/assay.py b/scarf/assay.py
deleted file mode 100644
index 6d5da626..00000000
--- a/scarf/assay.py
+++ /dev/null
@@ -1,1256 +0,0 @@
-"""
-- Classes:
- - Assay: A generic Assay class that contains methods to calculate feature level statistics.
- - RNAassay: This assay is designed for feature selection and normalization of scRNA-Seq data.
- - ATACassay: This assay is designed for ATAC-Seq data. It uses TF-IDF normalization and
- performs feature selection by marking most prevalent peaks.
- - ADTassay: This assay is designed for ADT data (surface antibodies) obtained from CITE-Seq
- experiments. It performs CLR normalization of the data but does not have any
- method for feature selection.
-"""
-
-from typing import Generator, List, Optional, Tuple, Union
-
-import numpy as np
-import pandas as pd
-import zarr
-from dask.array.core import Array as daskArrayType
-from dask.array.core import from_zarr
-from scipy.sparse import csr_matrix, vstack
-from zarr import hierarchy as z_hierarchy
-
-from .metadata import MetaData
-from .utils import controlled_compute, logger, show_dask_progress
-
-__all__ = ["Assay", "RNAassay", "ATACassay", "ADTassay"]
-
-
-def norm_dummy(_, counts: daskArrayType) -> daskArrayType:
- """A dummy normalizer. Doesn't perform any normalization. This is useful
- when the 'raw data' is already normalized.
-
- Args:
- _:
- counts: A dask array with 'raw' counts data
-
- Returns: Dask array
- """
- return counts
-
-
-def norm_lib_size(assay, counts: daskArrayType) -> daskArrayType:
- """Performs library size normalization on the data. This is the default
- method for RNA assays.
-
- Args:
- assay: An instance of the assay object
- counts: A dask array with raw counts data
-
- Returns: A dask array (delayed matrix) containing normalized data.
- """
- return assay.sf * counts / assay.scalar.reshape(-1, 1)
-
-
-def norm_lib_size_log(assay, counts: daskArrayType) -> daskArrayType:
- """Performs library size normalization and then transforms the values into
- log scale.
-
- Args:
- assay: An instance of the assay object
- counts: A dask array with raw counts data
-
- Returns: A dask array (delayed matrix) containing normalized data.
- """
- return np.log1p(assay.sf * counts / assay.scalar.reshape(-1, 1))
-
-
-def norm_clr(_, counts: daskArrayType) -> daskArrayType:
- """Performs centered log-ratio normalization (ADT). This is the default
- method for ADT assays.
-
- Args:
- _:
- counts: A dask array with raw counts data
-
- Returns: A dask array (delayed matrix) containing normalized data.
- """
- f = np.exp(np.log1p(counts).sum(axis=0) / len(counts))
- return np.log1p(counts / f)
-
-
-def norm_tf_idf(assay, counts: daskArrayType) -> daskArrayType:
- """Performs TF-IDF normalization This is the default method for ATAC
- assays.
-
- Args:
- assay: An instance of the assay object
- counts: A dask array with raw counts data
-
- Returns: A dask array (delayed matrix) containing normalized data.
- """
- t_f = counts / assay.n_term_per_doc.reshape(-1, 1)
- # TODO: Split TF and IDF functionality to make it similar to norml_lib and zscaling
- idf = np.log2(1 + (assay.n_docs / (assay.n_docs_per_term + 1)))
- return t_f * idf
-
-
-class Assay:
- """A generic Assay class that contains methods to calculate feature level
- statistics. It also provides a method for saving normalized subset of data
- for later KNN graph construction.
-
- Args:
- z (z_hierarchy.Group): Zarr hierarchy where raw data is located
- name (str): A label/name for assay.
- cell_data: Metadata class object for the cell attributes.
- nthreads: number for threads to use for dask parallel computations
- min_cells_per_feature:
-
- Attributes:
- name: A label for the assay instance
- z: Zarr group that contains the assay
- cells: A Metadata class object for cell attributes
- nthreads: number of threads to use for computations
- rawData: dask array containing the raw data
- feats: a MetaData class object for feature attributes
- attrs: Zarr attributes for the zarr group of the assay
- normMethod: normalization method to use.
- sf: scaling factor for doing library-size normalization
- """
-
- def __init__(
- self,
- z: z_hierarchy.Group,
- workspace: Union[str, None],
- name: str, # FIXME change to assay_name
- cell_data: MetaData,
- nthreads: int,
- min_cells_per_feature: int = 10,
- ):
- self.name = name
- self.cells = cell_data
- self.nthreads = nthreads
- if workspace is None:
- self.rawData = from_zarr(z[f"{name}/counts"], inline_array=True)
- self.feats = MetaData(z[f"{name}/featureData"]) # type: ignore
- self.z: zarr.Group = z[self.name] # type: ignore
- else:
- self.rawData = from_zarr(z[f"matrices/{name}/counts"], inline_array=True)
- self.feats = MetaData(z[f"{workspace}/{name}/featureData"]) # type: ignore
- self.z = z[f"{workspace}/{name}"]
- self.attrs = self.z.attrs
- if "percentFeatures" not in self.attrs:
- self.attrs["percentFeatures"] = {}
- self.normMethod = norm_dummy
- self.sf = None
- self._ini_feature_props(min_cells_per_feature)
-
- def normed(
- self,
- cell_idx: Optional[np.ndarray] = None,
- feat_idx: Optional[np.ndarray] = None,
- **kwargs,
- ) -> daskArrayType:
- """This function normalizes the raw and returns a delayed dask array of
- the normalized data.
-
- Args:
- cell_idx: Indices of cells to be included in the normalized matrix
- (Default value: All those marked True in 'I' column of cell
- attribute table)
- feat_idx: Indices of features to be included in the normalized matrix
- (Default value: All those marked True in 'I' column of
- feature attribute table)
- **kwargs:
-
- Returns: A dask array (delayed matrix) containing normalized data.
- """
- if cell_idx is None:
- cell_idx = self.cells.active_index("I")
- if feat_idx is None:
- feat_idx = self.feats.active_index("I")
- counts = self.rawData[:, feat_idx][cell_idx, :]
- return self.normMethod(self, counts)
-
- def to_raw_sparse(self, cell_key) -> csr_matrix:
- """
-
- Args:
- cell_key: A column from cell attribute table. This column must be a boolean
- type. The data will be exported for only those that have a True value
- in this column.
-
- Returns: A sparse matrix containing raw data.
-
- """
- from .utils import tqdmbar
-
- sm = None
- for i in tqdmbar(
- self.rawData[self.cells.active_index(cell_key), :].blocks,
- total=self.rawData.numblocks[0],
- desc=f"INFO: Converting raw data from {self.name} assay into CSR format",
- ):
- s = csr_matrix(controlled_compute(i, self.nthreads))
- if sm is None:
- sm = s
- else:
- sm = vstack([sm, s])
- return sm # type: ignore
-
- def _ini_feature_props(self, min_cells: int) -> None:
- """
-
- Args:
- min_cells: Minimum number of cells per feature. Features below this
- number are marked invalid.
-
- Returns:
-
- """
- if "nCells" in self.feats.columns and "dropOuts" in self.feats.columns:
- pass
- else:
- ncells = show_dask_progress(
- (self.rawData > 0).sum(axis=0),
- f"({self.name}) Computing nCells and dropOuts",
- self.nthreads,
- )
- self.feats.insert("nCells", ncells, overwrite=True)
- self.feats.insert(
- "dropOuts",
- abs(self.cells.N - self.feats.fetch("nCells")),
- overwrite=True,
- )
- self.feats.update_key(ncells > min_cells, "I")
-
- def add_percent_feature(self, feat_pattern: str, name: str) -> None:
- """
-
- Args:
- feat_pattern: A regular expression pattern to identify the features of interest
- name: This will be used as the name of column under which the percentages will
- be saved
-
- Returns:
-
- """
- if name in self.attrs["percentFeatures"]:
- if self.attrs["percentFeatures"][name] == feat_pattern:
- return None
- else:
- logger.info(f"Pattern for percentage feature {name} updated.")
- self.attrs["percentFeatures"] = {
- **{k: v for k, v in self.attrs["percentFeatures"].items()},
- **{name: feat_pattern},
- }
- feat_idx = sorted(
- self.feats.get_index_by(self.feats.grep(feat_pattern), "names")
- )
- if len(feat_idx) == 0:
- logger.warning(
- f"No matches found for pattern {feat_pattern}."
- f" Will not add/update percentage feature"
- )
- return None
- total = show_dask_progress(
- self.rawData[:, feat_idx].sum(axis=1),
- f"({self.name}) Computing {name}",
- self.nthreads,
- )
- if total.sum() == 0:
- logger.warning(
- f"Percentage feature {name} not added because not detected in any cell"
- )
- return None
- self.cells.insert(
- name,
- 100 * total / self.cells.fetch_all(self.name + "_nCounts"),
- overwrite=True,
- )
-
- def _verify_keys(self, cell_key: str, feat_key: str) -> None:
- """Checks if provided key names are present in cells and feature
- attribute tables and that they are of boolean types.
-
- Args:
- cell_key: Name of the key (column) from cell attribute table
- feat_key: Name of the key (column) from feature attribute table
-
- Returns: None
-
- Note on type checking /GA:
- 1. ds.cells.get_dtype(cell_key) == bool returns True because dtype('bool') (from numpy) is conceptually equivalent to Python's bool.
- 2. isinstance(ds.cells.get_dtype(cell_key), bool) returns False because dtype('bool') is a numpy.dtype object, not the native Python bool type.
- 3. Reason: dtype('bool') is a numpy object, and isinstance checks for the exact class, which is numpy.dtype, not bool.
-
- """
- if cell_key not in self.cells.columns or self.cells.get_dtype(cell_key) != bool: # noqa: E721
- raise ValueError(
- f"ERROR: Either {cell_key} does not exist or is not bool type"
- )
- if feat_key not in self.feats.columns or self.feats.get_dtype(feat_key) != bool: # noqa: E721
- raise ValueError(
- f"ERROR: Either {feat_key} does not exist or is not bool type"
- )
-
- def _get_cell_feat_idx(
- self, cell_key: str, feat_key: str
- ) -> Tuple[np.ndarray, np.ndarray]:
- """Verifies the provided key by calling _verify_keys and fetches the
- indices of rows that have True value in respective column.
-
- Args:
- cell_key: Name of the key (column) from cell attribute table
- feat_key: Name of the key (column) from feature attribute table
-
- Returns: A tuple of two numpy arrays corresponding to cell and feature indices
- respectively.
- """
-
- self._verify_keys(cell_key, feat_key)
- cell_idx = self.cells.active_index(cell_key)
- feat_idx = self.feats.active_index(feat_key)
- return cell_idx, feat_idx
-
- @staticmethod
- def _create_subset_hash(cell_idx: np.ndarray, feat_idx: np.ndarray) -> int:
- """Takes two index list and hashes them individually and then computes
- hash of the resulting tuple of two hashes. The objective of this
- function is to generate a unique state identifier for the cell and
- feature indices.
-
- Args:
- cell_idx: Cell row indices
- feat_idx: Feature row indices
-
- Returns: Returns the final hash
- """
- return hash(tuple([hash(tuple(cell_idx)), hash(tuple(feat_idx))]))
-
- @staticmethod
- def _get_summary_stats_loc(cell_key: str) -> Tuple[str, str]:
- """A convenience method that returns the location of feature-wise
- summary statistics Currently summaries are stored under pattern:
- summary_stats_{cell_key}
-
- Args:
- cell_key: Name of the key (column) from cell attribute table
-
- Returns: A tuple of two strings. First is the text that will be prepended to column
- names when summary statistics are loaded onto the feature attributes table. The
- second is the location of the summary statistics group in the zarr hierarchy of
- the assay.
- """
- return f"stats_{cell_key}", f"summary_stats_{cell_key}"
-
- def _validate_stats_loc(
- self,
- stats_loc: str,
- cell_idx: np.ndarray,
- feat_idx: np.ndarray,
- delete_on_fail: bool = True,
- ) -> bool:
- """Check whether the feature-wise summary statistics was previously
- calculated on the same set of features and cells as preset in the
- cell_idx and feat_idx parameters.
-
- Args:
- stats_loc: Location where the feature summary statistics are saved
- cell_idx: The indices of the cell attribute table
- feat_idx: The indices of the feature attribute table
- delete_on_fail: Whether to delete the summary statistics group if the validity check fails (Default: True).
-
- Returns: True is the validity test passes otherwise False
- """
- subset_hash = self._create_subset_hash(cell_idx, feat_idx)
- if stats_loc in self.z:
- attrs = self.z[stats_loc].attrs
- if "subset_hash" in attrs and attrs["subset_hash"] == subset_hash:
- return True
- else:
- # Reset stats loc
- if delete_on_fail:
- del self.z[stats_loc]
- return False
- else:
- return False
-
- def _load_stats_loc(self, cell_key: str) -> str:
- """Loads the feature-wise summary statistics calculated on the cells
- that are True in the 'cell_key' column.
-
- Args:
- cell_key: Name of the key (column) from cell attribute table
-
- Returns: Location of the group group that contains feature-wise summary statistics
- """
- cell_idx, feat_idx = self._get_cell_feat_idx(cell_key, "I")
- identifier, stats_loc = self._get_summary_stats_loc(cell_key)
- if self._validate_stats_loc(stats_loc, cell_idx, feat_idx) is False:
- raise KeyError(
- f"Summary statistics have not been calculated for cell key: {cell_key}"
- )
- if identifier not in self.feats.locations:
- self.feats.mount_location(self.z[stats_loc], identifier)
- else:
- logger.debug(f"Location ({stats_loc}) already mounted")
- return identifier
-
- def save_normalized_data(
- self,
- cell_key: str,
- feat_key: str,
- batch_size: int,
- location: str,
- log_transform: bool,
- renormalize_subset: bool,
- update_keys: bool,
- ) -> daskArrayType:
- """Create a new zarr group and saves the normalized data in the group
- for the selected features only.
-
- Args:
- cell_key: Name of the key (column) from cell attribute table. The data will be saved
- for only those cells that have a True value in this column.
- feat_key: Name of the key (column) from feature attribute table. The data will be saved
- for only those features that have a True value in this column
- batch_size: Number of cells to store in a single chunk. Higher values lead to larger
- memory consumption
- location: Zarr group wherein to save the normalized values
- log_transform: Whether to log transform the values. Is only used if the 'normed' method
- takes this parameter, ex. RNAassay
- renormalize_subset: Only used if the 'normed' method takes this parameter. Please refer
- to the documentation of the 'normed' method of the RNAassay for
- further description of this parameter.
- update_keys: Whether to update the keys. If True then the 'latest_feat_key' and
- 'latest_feat_key' attributes of the assay will be updated. It can be useful
- to set False in case where you only need to save the normalized data but
- don't intend to use it directly. For example, when mapping onto a different
- dataset and aligning features to that dataset.
-
- Returns: Dask array containing the normalized data
- """
-
- from .writers import dask_to_zarr
-
- # FIXME: Extensive documentation needed to justify the naming strategy of slots here
- # Because HVGs and other feature selections have cell key appended in their metadata
- if feat_key != "I":
- feat_key = cell_key + "__" + feat_key
- cell_idx, feat_idx = self._get_cell_feat_idx(cell_key, feat_key)
- subset_hash = self._create_subset_hash(cell_idx, feat_idx)
- subset_params = {
- "log_transform": log_transform,
- "renormalize_subset": renormalize_subset,
- }
- if location in self.z:
- if (
- subset_hash == self.z[location].attrs["subset_hash"]
- and subset_params == self.z[location].attrs["subset_params"]
- ):
- logger.info(
- f"Using existing normalized data with cell key {cell_key} and feat key {feat_key}"
- )
- if update_keys:
- self.attrs["latest_feat_key"] = (
- feat_key.split("__", 1)[1] if feat_key != "I" else "I"
- )
- self.attrs["latest_cell_key"] = cell_key
- return from_zarr(self.z[location + "/data"], inline_array=True)
- else:
- # Creating group here to overwrite all children
- self.z.create_group(location, overwrite=True)
- vals = self.normed(
- cell_idx,
- feat_idx,
- log_transform=log_transform,
- renormalize_subset=renormalize_subset,
- )
- dask_to_zarr(vals, self.z, location + "/data", batch_size, self.nthreads)
- self.z[location].attrs["subset_hash"] = subset_hash
- self.z[location].attrs["subset_params"] = subset_params
- if update_keys:
- self.attrs["latest_feat_key"] = (
- feat_key.split("__", 1)[1] if feat_key != "I" else "I"
- )
- self.attrs["latest_cell_key"] = cell_key
- return from_zarr(self.z[location + "/data"], inline_array=True)
-
- def iter_normed_feature_wise(
- self,
- cell_key: Optional[str],
- feat_key: Optional[str],
- batch_size: int,
- msg: Optional[str],
- as_dataframe: bool = True,
- **norm_params,
- ) -> Generator[Union[pd.DataFrame, Tuple[np.ndarray, np.ndarray]], None, None]:
- """This generator iterates over all the features marked by `feat_key`
- in batches.
-
- Args:
- cell_key: Name of the key (column) from cell attribute table. The data will be fetched
- for only those cells that have a True value in this column. If None then all the cells are used
- feat_key: Name of the key (column) from feature attribute table. The data will be fetched
- for only those features that have a True value in this column. If None then all the features are
- used
- batch_size: Number of genes to be loaded in the memory at a time.
- msg: Message to be displayed in the progress bar
- as_dataframe: If true (default) then the yielded matrices are pandas dataframe
-
- Returns:
- """
- from .utils import tqdmbar
-
- if cell_key is None:
- cell_idx = np.array(list(range(self.cells.N)))
- else:
- cell_idx = self.cells.active_index(cell_key)
-
- if feat_key is None:
- feat_idx = np.array(list(range(self.feats.N)))
- else:
- feat_idx = self.feats.active_index(feat_key)
- if msg is None:
- msg = ""
-
- data: daskArrayType = self.normed(
- cell_idx=cell_idx,
- feat_idx=feat_idx,
- **norm_params,
- )
- logger.debug("Will iterate over data of shape: ", data.shape)
- chunks = np.array_split(
- np.arange(0, data.shape[1]), max(1, int(data.shape[1] / batch_size))
- )
- for chunk in tqdmbar(chunks, desc=msg, total=len(chunks)):
- if as_dataframe:
- yield pd.DataFrame(
- controlled_compute(data[:, chunk], self.nthreads),
- columns=feat_idx[chunk],
- )
- else:
- yield (
- controlled_compute(data[:, chunk], self.nthreads).T,
- feat_idx[chunk],
- )
-
- def save_normed_for_query(
- self, feat_key: Optional[str], batch_size: int, overwrite: bool = True
- ) -> None:
- """This methods dumps normalized values for features (as marked by
- `feat_key`) onto disk in the 'prenormed' slot under the assay's own
- slot.
-
- Args:
- feat_key: Name of the key (column) from feature attribute table. The data will be fetched
- for only those features that have a True value in this column. If None then all the features are
- used
- batch_size: Number of genes to be loaded in the memory at a time.
- overwrite: If True (default value), then will overwrite the existing 'prenormed' slot in the
- assay hierarchy
-
- Returns:
- None
- """
- from joblib import Parallel, delayed
-
- from .writers import create_zarr_obj_array
-
- def write_wrapper(idx: str, v: np.ndarray) -> None:
- create_zarr_obj_array(g, idx, v, np.float64, True, False)
- return None
-
- if "prenormed" in self.z and overwrite is False:
- return None
-
- g = self.z.create_group("prenormed", overwrite=True)
- for mat, inds in self.iter_normed_feature_wise(
- None, feat_key, batch_size, "Saving features", False
- ):
- Parallel(n_jobs=self.nthreads)(
- delayed(write_wrapper)(inds[i], mat[i])
- for i in range(len(inds)) # type: ignore
- )
-
- def save_aggregated_ordering(
- self,
- cell_key: str,
- feat_key: str,
- ordering_key: str,
- min_exp: float = 1e-3,
- window_size: int = 200,
- chunk_size: int = 50,
- smoothen: bool = True,
- z_scale: bool = True,
- batch_size: int = 100,
- **norm_params,
- ):
- """
-
- Args:
- cell_key: Name of the key (column) from cell attribute table. The data will be fetched for only those cells.
- feat_key: Name of the key (column) from feature attribute table.
- ordering_key:
- min_exp:
- window_size:
- chunk_size:
- smoothen:
- z_scale:
- batch_size:
- **norm_params:
-
- Returns:
-
- """
-
- from .utils import rolling_window
- from .writers import create_zarr_dataset
-
- cell_ordering = self.cells.fetch(ordering_key, key=cell_key)
- cell_idx, feat_idx = self._get_cell_feat_idx(cell_key, feat_key)
- hashes = [hash(tuple(x)) for x in (cell_idx, feat_idx, cell_ordering)]
- params = {
- "min_exp": min_exp,
- "window_size": window_size,
- "chunk_size": chunk_size,
- "smoothen": smoothen,
- "z_scale": z_scale,
- "norm_params": norm_params,
- }
- location = f"aggregated_{cell_key}_{feat_key}_{ordering_key}"
- if (
- location in self.z
- and "hashes" in self.z[location].attrs
- and hashes == self.z[location].attrs["hashes"]
- and "params" in self.z[location].attrs
- and params == self.z[location].attrs["params"]
- ):
- logger.info(f"Using existing aggregated data from {location}")
- else:
- if location in self.z:
- del self.z[location]
-
- # The actual size might be smaller due to dynamic filtering of features
- g = create_zarr_dataset(
- self.z,
- location + "/data",
- (batch_size,),
- "float64",
- (feat_idx.shape[0], chunk_size),
- )
- ordering_idx = np.argsort(cell_ordering)
- feat_idx = []
- valid_feats = []
- s = 0
- for df in self.iter_normed_feature_wise(
- cell_key,
- feat_key,
- batch_size,
- "Binning over cell-ordering",
- True,
- **norm_params,
- ):
- valid_feats.extend(list((df.mean() > min_exp).values))
- feat_idx.extend(list(df.columns))
- if smoothen:
- df = rolling_window(df.reindex(ordering_idx).values, window_size)
- if z_scale:
- df = (df - df.mean(axis=0)) / df.std(axis=0)
- df_mean = np.array(
- [x.mean(axis=0) for x in np.array_split(df, chunk_size)]
- ).T
- g[s : s + df_mean.shape[0]] = df_mean
- s += df_mean.shape[0]
-
- g = create_zarr_dataset(
- self.z,
- location + "/feature_indices",
- (len(feat_idx),),
- "uint64",
- (len(feat_idx),),
- )
- g[:] = np.array(feat_idx).astype(int)
-
- g = create_zarr_dataset(
- self.z,
- location + "/valid_features",
- (len(feat_idx),),
- "bool",
- (len(feat_idx),),
- )
- g[:] = np.array(valid_feats).astype(int)
-
- self.z[location].attrs["hashes"] = hashes
- self.z[location].attrs["params"] = params
-
- ret_val1 = from_zarr(self.z[location + "/data"], inline_array=True)
- ret_val2 = self.z[location + "/feature_indices"][:]
-
- if location + "/valid_features" in self.z:
- valid_feats = self.z[location + "/valid_features"][:]
- ret_val1 = ret_val1[valid_feats]
- ret_val2 = ret_val2[valid_feats]
-
- return ret_val1, ret_val2
-
- def score_features(
- self,
- feature_names: List[str],
- cell_key: str,
- ctrl_size: int,
- n_bins: int,
- rand_seed: int,
- ) -> np.ndarray:
- """Calculates the scores (mean values) of selection of features over a
- randomly sampled selected feature set in given cells (as marked by
- cell_key)
-
- Args:
- feature_names: Names (as in 'names' column of the feature attribute table) of features to
- be used for scoring
- cell_key: Name of the key (column) from cell attribute table.
- ctrl_size: Number of reference features to be sampled from each bin.
- n_bins: Number of bins for sampling.
- rand_seed: The seed to use for the random number generation.
-
- Returns: Numpy array of the calculated scores
- """
-
- from .feat_utils import binned_sampling
-
- def _names_to_idx(i):
- return self.feats.get_index_by(i, "names", None)
-
- def _calc_mean(i):
- return (
- self.normed(cell_idx=cell_idx, feat_idx=np.array(sorted(i)))
- .mean(axis=1)
- .compute()
- )
-
- feature_idx = _names_to_idx(feature_names)
- if len(feature_idx) == 0:
- raise ValueError(
- f"ERROR: No feature ids found for any of the provided {len(feature_names)} features"
- )
-
- identifier = self._load_stats_loc(cell_key)
- obs_avg = pd.Series(self.feats.fetch_all(f"{identifier}_avg"))
- control_idx = binned_sampling(
- obs_avg, list(feature_idx), ctrl_size, n_bins, rand_seed
- )
- cell_idx, _ = self._get_cell_feat_idx(cell_key, "I")
- return _calc_mean(feature_idx) - _calc_mean(control_idx)
-
- def __repr__(self):
- f = self.feats.fetch_all("I")
- assay_name = str(self.__class__).split(".")[-1][:-2]
- return f"{assay_name} {self.name} with {f.sum()}({len(f)}) features"
-
-
-class RNAassay(Assay):
- """This subclass of Assay is designed for feature selection and
- normalization of scRNA-Seq data.
-
- Args:
- z (z_hierarchy.Group): Zarr hierarchy where raw data is located
- name (str): A label/name for assay.
- cell_data: Metadata class object for the cell attributes.
- **kwargs: kwargs to be passed to the Assay class
-
- Attributes:
- normMethod: A pointer to the function to be used for normalization of the raw data
- sf: scaling factor for doing library-size normalization
- scalar: This is used to cache the library size of the cells.
- It is set to None until normed method is called.
- """
-
- def __init__(self, z: z_hierarchy.Group, name: str, cell_data: MetaData, **kwargs):
- super().__init__(z=z, name=name, cell_data=cell_data, **kwargs)
- self.normMethod = norm_lib_size
- if "size_factor" in self.attrs:
- self.sf = int(self.attrs["size_factor"])
- else:
- self.sf = 1000
- self.attrs["size_factor"] = self.sf
- self.scalar = None
-
- def normed(
- self,
- cell_idx: Optional[np.ndarray] = None,
- feat_idx: Optional[np.ndarray] = None,
- renormalize_subset: bool = False,
- log_transform: bool = False,
- **kwargs,
- ) -> daskArrayType:
- """This function normalizes the raw and returns a delayed dask array of
- the normalized data. Unlike the `normed` method in the generic Assay
- class this method is optimized for scRNA-Seq data and takes additional
- parameters that will be used by `norm_lib_size` (default normalization
- method for this class).
-
- Args:
- cell_idx: Indices of cells to be included in the normalized matrix
- (Default value: All those marked True in 'I' column of cell
- attribute table)
- feat_idx: Indices of features to be included in the normalized matrix
- (Default value: All those marked True in 'I' column of
- feature attribute table)
- renormalize_subset: If True, then the data is normalized using only those features that are True in
- `feat_key` column rather using total expression of all features in a cell
- (Default value: False)
- log_transform: If True, then the normalized data is log-transformed (Default value: False).
- **kwargs: kwargs have no effect here.
-
- Returns:
- A dask array (delayed matrix) containing normalized data.
- """
- if cell_idx is None:
- cell_idx = self.cells.active_index("I")
- if feat_idx is None:
- feat_idx = self.feats.active_index("I")
- counts = self.rawData[:, feat_idx][cell_idx, :]
- norm_method_cache = self.normMethod
- if log_transform:
- self.normMethod = norm_lib_size_log
- if renormalize_subset:
- a = show_dask_progress(
- counts.sum(axis=1), "Normalizing with feature subset", self.nthreads
- )
- a[a == 0] = 1
- self.scalar = a
- else:
- self.scalar = self.cells.fetch_all(self.name + "_nCounts")[cell_idx]
- val = self.normMethod(self, counts)
- self.normMethod = norm_method_cache
- return val
-
- def set_feature_stats(self, cell_key: str) -> None:
- """Calculates summary statistics for the features of the assay using
- only cells that are marked True by the 'cell_key' parameter.
-
- Args:
- cell_key: Name of the key (column) from cell attribute table.
-
- Returns: None
- """
- feat_key = "I" # Here we choose to calculate stats for all the features
- cell_idx, feat_idx = self._get_cell_feat_idx(cell_key, feat_key)
- identifier, stats_loc = self._get_summary_stats_loc(cell_key)
- if self._validate_stats_loc(stats_loc, cell_idx, feat_idx) is True:
- logger.info(f"Using cached feature stats for cell_key {cell_key}")
- return None
- else:
- if identifier in self.feats.locations:
- del self.feats.locations[identifier]
- n_cells = show_dask_progress(
- (self.normed(cell_idx, feat_idx) > 0).sum(axis=0),
- f"({self.name}) Computing nCells",
- self.nthreads,
- )
- tot = show_dask_progress(
- self.normed(cell_idx, feat_idx).sum(axis=0),
- f"({self.name}) Computing normed_tot",
- self.nthreads,
- )
- sigmas = show_dask_progress(
- self.normed(cell_idx, feat_idx).var(axis=0),
- f"({self.name}) Computing sigmas",
- self.nthreads,
- )
- # idx = n_cells > min_cells
- # self.feats.update_key(idx, key=feat_key)
- # n_cells, tot, sigmas = n_cells[idx], tot[idx], sigmas[idx]
-
- self.z.create_group(stats_loc, overwrite=True)
- self.feats.mount_location(self.z[stats_loc], identifier)
- self.feats.insert(
- "normed_tot", tot.astype(float), overwrite=True, location=identifier
- )
- self.feats.insert(
- "avg",
- (tot / self.cells.N).astype(float),
- overwrite=True,
- location=identifier,
- )
- nz_mean = np.divide(
- tot, n_cells, out=np.zeros_like(tot).astype(float), where=n_cells != 0
- )
- self.feats.insert(
- "nz_mean",
- nz_mean.astype(float),
- overwrite=True,
- location=identifier,
- )
- self.feats.insert(
- "sigmas", sigmas.astype(float), overwrite=True, location=identifier
- )
- self.feats.insert(
- "normed_n", n_cells.astype(float), overwrite=True, location=identifier
- )
- self.z[stats_loc].attrs["subset_hash"] = self._create_subset_hash(
- cell_idx, self.feats.active_index(feat_key)
- )
- self.feats.unmount_location(identifier)
- return None
-
- def set_summary_stats(
- self, cell_key: str = None, n_bins: int = 200, lowess_frac: float = 0.1
- ) -> Tuple[str, str]:
- """Calculates summary statistics for the features of the assay using only cells that are marked True by the 'cell_key' parameter.
-
- Args:
- cell_key: Name of the key (column) from cell attribute table.
- n_bins: Number of bins to divide the data into.
- lowess_frac: Between 0 and 1. The fraction of the data used when estimating the fit between mean and
- variance. This is same as `frac` in statsmodels.nonparametric.smoothers_lowess.lowess
-
- Returns:
- A tuple of two strings.
- identifier: The text that will be prepended to column names when summary statistics are loaded onto the feature attributes table.
- c_var_col: The name of the column in the feature attribute table that contains the corrected variance values.
- """
-
- def col_renamer(x):
- return f"{identifier}_{x}"
-
- if cell_key is None:
- cell_key = "I"
-
- # check lowess_frac is between 0 and 1
- if not 0 <= lowess_frac <= 1:
- raise ValueError("lowess_frac must be between 0 and 1")
-
- self.set_feature_stats(cell_key)
- identifier = self._load_stats_loc(cell_key)
- c_var_col = f"c_var__{n_bins}__{lowess_frac}"
- if col_renamer(c_var_col) in self.feats.columns:
- logger.info("Using existing corrected dispersion values")
- else:
- slots = ["normed_tot", "avg", "nz_mean", "sigmas", "normed_n"]
- for i in slots:
- i = col_renamer(i)
- if i not in self.feats.columns:
- raise KeyError(f"ERROR: {i} not found in feature metadata")
- c_var = self.feats.remove_trend(
- col_renamer("avg"), col_renamer("sigmas"), n_bins, lowess_frac
- )
- self.feats.insert(c_var_col, c_var, overwrite=True, location=identifier)
-
- return identifier, c_var_col
-
- # maybe we should return plot here? If one wants to modify it. /raz
- def mark_hvgs(
- self,
- cell_key: str,
- min_cells: int,
- top_n: int,
- min_var: float,
- max_var: float,
- min_mean: float,
- max_mean: float,
- n_bins: int,
- lowess_frac: float,
- blacklist: str,
- hvg_key_name: str,
- keep_bounds: bool,
- show_plot: bool,
- max_cells: int,
- **plot_kwargs,
- ) -> None:
- """Identifies highly variable genes in the dataset.
-
- The parameters govern the min/max variance (corrected) and mean expression threshold for calling genes highly
- variable. The variance is corrected by first dividing genes into bins based on their mean expression values.
- Genes with minimum variance is selected from each bin and a Lowess curve is fitted to
- the mean-variance trend of these genes. mark_hvgs will by default run on the default assay.
- See `utils.fit_lowess` for further details.
-
- *Modifies the feats table*: adds a column named `__hvgs` to the feature table,
- which contains a True value for genes marked HVGs. The prefix comes from the `cell_key` parameter,
- the naming rule in Scarf dictates that cells used to identify HVGs are prepended to the column name
- (with a double underscore delimiter).
-
- Args:
- cell_key: Specify which cells to use to identify the HVGs. (Default value 'I' use all non-filtered out
- cells).
- min_cells: Minimum number of cells where a gene should have non-zero expression values for it to be
- considered a candidate for HVG selection. Large values for this parameter might make it difficult
- to identify rare populations of cells. Very small values might lead to higher signal to noise
- ratio in the selected features.
- max_cells: Maximum number of cells where a gene should have non-zero expression values for it to be
- considered a candidate for HVG selection. This can be useful to filter out genes that are
- expressed in too many cells. Default value is infinity, meaning no upper limit.
- top_n: Number of top most variable genes to be set as HVGs. This value is ignored if a value is provided
- for `min_var` parameter.
- min_var: Minimum variance threshold for HVG selection.
- max_var: Maximum variance threshold for HVG selection.
- min_mean: Minimum mean value of expression threshold for HVG selection.
- max_mean: Maximum mean value of expression threshold for HVG selection.
- n_bins: Number of bins into which the mean expression is binned.
- lowess_frac: Between 0 and 1. The fraction of the data used when estimating the fit between mean and
- variance. This is same as `frac` in statsmodels.nonparametric.smoothers_lowess.lowess
- blacklist: A regular expression string pattern. Gene names matching to this pattern will be excluded from
- the final highly variable genes list
- hvg_key_name: The label for highly variable genes. This label will be used to mark the HVGs in the
- feature attribute table. The value for 'cell_key' parameter is prepended to this value.
- keep_bounds: If True, then the boundary values are retained and not filtered out.
- show_plot: If True, a plot is produced, that for each gene shows the corrected variance on the y-axis and
- the non-zero mean (means from cells where the gene had a non-zero value) on the x-axis. The
- genes are colored in two gradients which indicate the number of cells where the gene was
- expressed. The colors are yellow to dark red for HVGs, and blue to green for non-HVGs.
- **plot_kwargs: Keyword arguments for matplotlib.pyplot.scatter function
- """
-
- def col_renamer(x):
- return f"{identifier}_{x}"
-
- logger.info("Calculating summary statistics")
- identifier, c_var_col = self.set_summary_stats(cell_key, n_bins, lowess_frac)
- logger.info("Calculating HVGs")
-
- if max_mean != np.inf:
- max_mean = 2**max_mean
- if max_var != np.inf:
- max_var = 2**max_var
- if min_mean != -np.inf:
- min_mean = 2**min_mean
- if min_var != -np.inf:
- min_var = 2**min_var
-
- if blacklist != "":
- bl = self.feats.index_to_bool(
- self.feats.get_index_by(self.feats.grep(blacklist), "names"),
- invert=True,
- )
- else:
- bl = np.ones(self.feats.N).astype(bool)
- if min_var == -np.inf:
- if top_n < 1:
- raise ValueError(
- "ERROR: Please provide a value greater than 0 for `top_n` parameter"
- )
- idx = self.feats.multi_sift(
- [col_renamer("normed_n"), col_renamer("nz_mean")],
- [min_cells, min_mean],
- [max_cells, max_mean],
- keep_bounds=keep_bounds,
- )
- idx = idx & self.feats.fetch_all("I") & bl
- n_valid_feats = idx.sum()
- if top_n > n_valid_feats:
- logger.warning(
- f"WARNING: Number of valid features are less then value "
- f"of parameter `top_n`: {top_n}. Resetting `top_n` to {n_valid_feats}"
- )
- top_n = n_valid_feats - 1
- min_var = (
- pd.Series(self.feats.fetch_all(col_renamer(c_var_col)))[idx]
- .sort_values(ascending=False)
- .values[top_n]
- ) # type: ignore
- hvgs = self.feats.multi_sift(
- [col_renamer(x) for x in ["normed_n", "nz_mean", c_var_col]],
- [min_cells, min_mean, min_var],
- [max_cells, max_mean, max_var],
- keep_bounds=keep_bounds,
- )
- hvgs = hvgs & self.feats.fetch_all("I") & bl
- hvg_key_name = cell_key + "__" + hvg_key_name
- logger.info(f"{sum(hvgs)} genes marked as HVGs")
- self.feats.insert(hvg_key_name, hvgs, fill_value=False, overwrite=True)
-
- if show_plot:
- from .plots import plot_mean_var
-
- nzm, vf, nc = [
- self.feats.fetch(x)
- for x in [col_renamer("nz_mean"), col_renamer(c_var_col), "nCells"]
- ]
- plot_mean_var(nzm, vf, nc, self.feats.fetch(hvg_key_name), **plot_kwargs)
-
- return None
-
-
-class ATACassay(Assay):
- """This subclass of Assay is designed for feature selection and
- normalization of scATAC-Seq data."""
-
- def __init__(self, z: z_hierarchy.Group, name: str, cell_data: MetaData, **kwargs):
- """This Assay subclass is designed for feature selection and
- normalization of scATAC-Seq data.
-
- Args:
- z (z_hierarchy.Group): Zarr hierarchy where raw data is located
- name (str): A label/name for assay.
- cell_data: Metadata class object for the cell attributes.
- **kwargs:
-
- Attributes:
- normMethod: Pointer to the function to be used for normalization of the raw data
- n_term_per_doc: Number of features per cell. Used for TF-IDF normalization
- n_docs: Number of cells. Used for TF-IDF normalization
- n_docs_per_term: Number of cells per feature. Used for TF-IDF normalization
- """
- super().__init__(z=z, name=name, cell_data=cell_data, **kwargs)
- self.normMethod = norm_tf_idf
- self.n_term_per_doc = None
- self.n_docs = None
- self.n_docs_per_term = None
-
- def normed(
- self,
- cell_idx: Optional[np.ndarray] = None,
- feat_idx: Optional[np.ndarray] = None,
- **kwargs,
- ) -> daskArrayType:
- """This function normalizes the raw and returns a delayed dask array of
- the normalized data. Unlike the `normed` method in the generic Assay
- class this method is optimized for scATAC-Seq data. This method uses
- the the normalization indicated by attribute self.normMethod which by
- default is set to `norm_tf_idf`. The TF-IDF normalization is performed
- using only the cells and features indicated by the 'cell_idx' and
- 'feat_idx' parameters.
-
- Args:
- cell_idx: Indices of cells to be included in the normalized matrix
- (Default value: All those marked True in 'I' column of cell
- attribute table)
- feat_idx: Indices of features to be included in the normalized matrix
- (Default value: All those marked True in 'I' column of
- feature attribute table)
- **kwargs:
-
- Returns: A dask array (delayed matrix) containing normalized data.
- """
- if cell_idx is None:
- cell_idx = self.cells.active_index("I")
- if feat_idx is None:
- feat_idx = self.feats.active_index("I")
- counts: daskArrayType = self.rawData[:, feat_idx][cell_idx, :]
- self.n_term_per_doc = self.cells.fetch_all(self.name + "_nFeatures")[cell_idx]
- self.n_docs = len(cell_idx)
- self.n_docs_per_term = self.feats.fetch_all("nCells")[feat_idx]
- return self.normMethod(self, counts)
-
- def set_feature_stats(self, cell_key: str) -> None:
- """Calculates prevalence of each valid feature of the assay using only
- cells that are marked True by the 'cell_key' parameter. Prevalence of a
- feature is the sum of all its TF-IDF normalized values across cells.
-
- Args:
- cell_key: Name of the key (column) from cell attribute table.
-
- Returns: None
- """
- feat_key = "I" # Here we choose to calculate stats for all the features
- cell_idx, feat_idx = self._get_cell_feat_idx(cell_key, feat_key)
- identifier, stats_loc = self._get_summary_stats_loc(cell_key)
- if self._validate_stats_loc(stats_loc, cell_idx, feat_idx) is True:
- logger.info(f"Using cached feature stats for cell_key {cell_key}")
- return None
- prevalence = show_dask_progress(
- self.normed(cell_idx, feat_idx).sum(axis=0),
- f"({self.name}) Calculating peak prevalence across cells",
- self.nthreads,
- )
- self.z.create_group(stats_loc, overwrite=True)
- self.feats.mount_location(self.z[stats_loc], identifier)
- self.feats.insert(
- "prevalence", prevalence.astype(float), overwrite=True, location=identifier
- )
- self.z[stats_loc].attrs["subset_hash"] = self._create_subset_hash(
- cell_idx, feat_idx
- )
- self.feats.unmount_location(identifier)
- return None
-
- def mark_prevalent_peaks(
- self, cell_key: str, top_n: int, prevalence_key_name: str
- ) -> None:
- """Marks `top_n` peaks with highest prevalence as prevalent peaks.
-
- Args:
- cell_key: Cells to use for selection of most prevalent peaks. The provided value for `cell_key` should be a
- column in cell attributes table with boolean values.
- top_n: Number of top prevalent peaks to be selected. This value is ignored if a value is provided
- for `min_var` parameter.
- prevalence_key_name: Base label for marking prevalent peaks in the features attributes column. The value for
- 'cell_key' parameter is prepended to this value.
-
- Returns: None
- """
- if top_n >= self.feats.N:
- raise ValueError(
- f"ERROR: n_top should be less than total number of features ({self.feats.N})]"
- )
- if isinstance(top_n, int) is False or top_n < 1:
- raise TypeError("ERROR: n_top must a positive integer value")
- self.set_feature_stats(cell_key)
- identifier = self._load_stats_loc(cell_key)
- idx = (
- pd.Series(self.feats.fetch_all(f"{identifier}_prevalence"))
- .sort_values(ascending=False)
- .index.values[:top_n]
- )
- prevalence_key_name = cell_key + "__" + prevalence_key_name
- self.feats.insert(
- prevalence_key_name,
- self.feats.index_to_bool(idx),
- fill_value=False,
- overwrite=True,
- )
- return None
-
-
-class ADTassay(Assay):
- """This subclass of Assay is designed for normalization of ADT/HTO
- (feature-barcodes library) data from CITE-Seq experiments.
-
- Args:
- z (z_hierarchy.Group): Zarr hierarchy where raw data is located
- name (str): A label/name for assay.
- cell_data: Metadata class object for the cell attributes.
- **kwargs:
-
- Attributes:
- normMethod: Pointer to the function to be used for normalization of the raw data
- """
-
- def __init__(self, z: z_hierarchy.Group, name: str, cell_data: MetaData, **kwargs):
- """This subclass of Assay is designed for normalization of ADT/HTO
- (feature-barcodes library) data from CITE-Seq experiments."""
- super().__init__(z=z, name=name, cell_data=cell_data, **kwargs)
- self.normMethod = norm_clr
-
- def normed(
- self,
- cell_idx: Optional[np.ndarray] = None,
- feat_idx: Optional[np.ndarray] = None,
- **kwargs,
- ) -> daskArrayType:
- """This function normalizes the raw and returns a delayed dask array of
- the normalized data. This method uses the the normalization indicated
- by attribute self.normMethod which by default is set to `norm_clr`. The
- centered log-ratio normalization is performed using only the cells and
- features indicated by the 'cell_idx' and 'feat_idx' parameters.
-
- Args:
- cell_idx: Indices of cells to be included in the normalized matrix
- (Default value: All those marked True in 'I' column of cell
- attribute table)
- feat_idx: Indices of features to be included in the normalized matrix
- (Default value: All those marked True in 'I' column of
- feature attribute table)
- **kwargs:
-
- Returns: A dask array (delayed matrix) containing normalized data.
- """
- if cell_idx is None:
- cell_idx = self.cells.active_index("I")
- if feat_idx is None:
- feat_idx = self.feats.active_index("I")
- counts = self.rawData[:, feat_idx][cell_idx, :]
- return self.normMethod(self, counts)
diff --git a/scarf/assay/__init__.py b/scarf/assay/__init__.py
new file mode 100644
index 00000000..b8007960
--- /dev/null
+++ b/scarf/assay/__init__.py
@@ -0,0 +1,55 @@
+"""
+- Classes:
+ - Assay: A generic Assay class that contains methods to calculate feature level statistics.
+ - RNAassay: This assay is designed for feature selection and normalization of scRNA-Seq data.
+ - ATACassay: This assay is designed for ATAC-Seq data. It uses TF-IDF normalization and
+ performs feature selection by marking most prevalent peaks.
+ - ADTassay: This assay is designed for ADT data (surface antibodies) obtained from CITE-Seq
+ experiments. It performs CLR normalization of the data but does not have any
+ method for feature selection.
+"""
+
+from .adt import ADTassay as ADTassay
+from .atac import ATACassay as ATACassay
+from .base import Assay as Assay
+from .base import PercentFeatures as PercentFeatures
+from .normalization import NormMethod as NormMethod
+from .normalization import (
+ lib_size_feature_stream_eligible as lib_size_feature_stream_eligible,
+)
+from .normalization import norm_clr as norm_clr
+from .normalization import norm_dummy as norm_dummy
+from .normalization import norm_lib_size as norm_lib_size
+from .normalization import norm_lib_size_log as norm_lib_size_log
+from .normalization import norm_tf_idf as norm_tf_idf
+from .persistence import _read_block as _read_block
+from .rna import RNAassay as RNAassay
+
+__all__ = ["Assay", "RNAassay", "ATACassay", "ADTassay"]
+
+
+def _normalize_public_metadata() -> None:
+ norm_lib_size.__globals__["Assay"] = Assay
+
+ for function in (
+ _read_block,
+ lib_size_feature_stream_eligible,
+ norm_clr,
+ norm_dummy,
+ norm_lib_size,
+ norm_lib_size_log,
+ norm_tf_idf,
+ ):
+ function.__module__ = __name__
+
+ for assay_class in (Assay, RNAassay, ATACassay, ADTassay):
+ assay_class.__module__ = __name__
+ for descriptor in assay_class.__dict__.values():
+ if isinstance(descriptor, staticmethod):
+ descriptor.__func__.__module__ = __name__
+ elif callable(descriptor) and hasattr(descriptor, "__module__"):
+ descriptor.__module__ = __name__
+
+
+_normalize_public_metadata()
+del _normalize_public_metadata
diff --git a/scarf/assay/adt.py b/scarf/assay/adt.py
new file mode 100644
index 00000000..9c7f6a45
--- /dev/null
+++ b/scarf/assay/adt.py
@@ -0,0 +1,84 @@
+from typing import Any
+
+import numpy as np
+import zarr
+
+from ..matrix import ChunkedArray
+from ..metadata import MetaData
+from .base import Assay
+from .normalization import norm_clr
+
+
+class ADTassay(Assay):
+ """This subclass of Assay is designed for normalization of ADT/HTO
+ (feature-barcodes library) data from CITE-Seq experiments.
+
+ Args:
+ z (zarr.Group): Zarr hierarchy where raw data is located
+ name (str): A label/name for assay.
+ cell_data: Metadata class object for the cell attributes.
+ **kwargs:
+
+ Attributes:
+ normMethod: Pointer to the function to be used for normalization of the raw data
+ """
+
+ def __init__(
+ self,
+ z: zarr.Group,
+ name: str,
+ cell_data: MetaData,
+ *,
+ workspace: str | None = None,
+ nthreads: int = 1,
+ min_cells_per_feature: int = 10,
+ **kwargs: Any,
+ ) -> None:
+ """Initialize ADTassay with CLR normalization.
+
+ Args:
+ z: Zarr hierarchy where raw data is located.
+ name: Assay label.
+ cell_data: Cell metadata object.
+ **kwargs: Forwarded to ``Assay.__init__`` (workspace, nthreads, etc.).
+ """
+ super().__init__(
+ z=z,
+ workspace=workspace,
+ name=name,
+ cell_data=cell_data,
+ nthreads=nthreads,
+ min_cells_per_feature=min_cells_per_feature,
+ **kwargs,
+ )
+ self.normMethod = norm_clr
+
+ def normed(
+ self,
+ cell_idx: np.ndarray | None = None,
+ feat_idx: np.ndarray | None = None,
+ **kwargs: Any,
+ ) -> ChunkedArray:
+ """This function normalizes the raw and returns a delayed chunked array of
+ the normalized data. This method uses the normalization indicated
+ by attribute self.normMethod which by default is set to `norm_clr`. The
+ centered log-ratio normalization is performed using only the cells and
+ features indicated by the 'cell_idx' and 'feat_idx' parameters.
+
+ Args:
+ cell_idx: Indices of cells to be included in the normalized matrix
+ (Default value: All those marked True in 'I' column of cell
+ attribute table)
+ feat_idx: Indices of features to be included in the normalized matrix
+ (Default value: All those marked True in 'I' column of
+ feature attribute table)
+ **kwargs:
+
+ Returns: A chunked array (delayed matrix) containing normalized data.
+ """
+ if cell_idx is None:
+ cell_idx = self.cells.active_index("I")
+ if feat_idx is None:
+ feat_idx = self.feats.active_index("I")
+ counts = self.rawData[:, feat_idx][cell_idx, :]
+ return self.normMethod(self, counts)
diff --git a/scarf/assay/atac.py b/scarf/assay/atac.py
new file mode 100644
index 00000000..9c2c38a9
--- /dev/null
+++ b/scarf/assay/atac.py
@@ -0,0 +1,450 @@
+from typing import Any
+
+import numpy as np
+import pandas as pd
+import zarr
+
+from ..matrix import ChunkedArray
+from ..metadata import MetaData
+from ..storage.types import as_zarr_array, as_zarr_group
+from ..utils.arrays import array_digest
+from ..utils.compute import show_dask_progress
+from ..utils.logging import logger
+from .base import Assay
+from .normalization import norm_tf_idf
+
+_CELL_INDEX_DIGEST_ATTR = "cell_index_digest"
+_NORMALIZATION_IDENTITY_ATTR = "normalization_identity"
+_DOCUMENT_FREQUENCY_COLUMN = "document_frequency"
+
+
+class ATACassay(Assay):
+ """This subclass of Assay is designed for feature selection and
+ normalization of scATAC-Seq data."""
+
+ def __init__(
+ self,
+ z: zarr.Group,
+ name: str,
+ cell_data: MetaData,
+ *,
+ workspace: str | None = None,
+ nthreads: int = 1,
+ min_cells_per_feature: int = 10,
+ **kwargs: Any,
+ ) -> None:
+ """This Assay subclass is designed for feature selection and
+ normalization of scATAC-Seq data.
+
+ Args:
+ z (zarr.Group): Zarr hierarchy where raw data is located
+ name (str): A label/name for assay.
+ cell_data: Metadata class object for the cell attributes.
+ **kwargs:
+
+ Attributes:
+ normMethod: Pointer to the function to be used for normalization of the raw data
+ n_term_per_doc: Number of features per cell. Used for TF-IDF normalization
+ n_docs: Number of cells. Used for TF-IDF normalization
+ n_docs_per_term: Number of cells per feature. Used for TF-IDF normalization
+ """
+ super().__init__(
+ z=z,
+ workspace=workspace,
+ name=name,
+ cell_data=cell_data,
+ nthreads=nthreads,
+ min_cells_per_feature=min_cells_per_feature,
+ **kwargs,
+ )
+ self.normMethod = norm_tf_idf
+ self.n_term_per_doc: np.ndarray | None = None
+ self.n_docs: int | None = None
+ self.n_docs_per_term: np.ndarray | None = None
+
+ def normed(
+ self,
+ cell_idx: np.ndarray | None = None,
+ feat_idx: np.ndarray | None = None,
+ **kwargs: Any,
+ ) -> ChunkedArray:
+ """This function normalizes the raw and returns a delayed chunked array of
+ the normalized data. Unlike the `normed` method in the generic Assay
+ class this method is optimized for scATAC-Seq data. This method uses
+ the normalization indicated by attribute self.normMethod which by
+ default is set to `norm_tf_idf`. Document frequency is learned from
+ `cell_idx`. The returned matrix contains `feat_idx`, while term frequency
+ uses total ATAC counts unless subset renormalization is requested.
+
+ Args:
+ cell_idx: Indices of cells to be included in the normalized matrix
+ (Default value: All those marked True in 'I' column of cell
+ attribute table)
+ feat_idx: Indices of features to be included in the normalized matrix
+ (Default value: All those marked True in 'I' column of
+ feature attribute table)
+ **kwargs: `log_transform` must be false. `renormalize_subset` uses
+ counts among `feat_idx` as the term-frequency denominator.
+
+ Returns: A chunked array (delayed matrix) containing normalized data.
+ """
+ if cell_idx is None:
+ cell_idx = self.cells.active_index("I")
+ if feat_idx is None:
+ feat_idx = self.feats.active_index("I")
+ log_transform = kwargs.get("log_transform", False)
+ renormalize_subset = kwargs.get("renormalize_subset", False)
+ if not isinstance(log_transform, (bool, np.bool_)):
+ raise TypeError("log_transform must be a boolean")
+ if not isinstance(renormalize_subset, (bool, np.bool_)):
+ raise TypeError("renormalize_subset must be a boolean")
+ if log_transform:
+ raise ValueError("ATAC TF-IDF does not support log_transform; use False")
+ cell_idx = np.asarray(cell_idx, dtype=np.int64)
+ feat_idx = np.asarray(feat_idx, dtype=np.int64)
+ counts: ChunkedArray = self.rawData[:, feat_idx][cell_idx, :]
+ self.n_term_per_doc = self._terms_per_document(
+ cell_idx,
+ counts=counts,
+ renormalize_subset=bool(renormalize_subset),
+ )
+ self.n_docs = len(cell_idx)
+ if self.normMethod is not norm_tf_idf:
+ self.n_docs_per_term = self.feats.fetch_all("nCells")[feat_idx]
+ elif self.n_docs == 0:
+ self.n_docs_per_term = np.zeros(len(feat_idx), dtype=np.int64)
+ else:
+ document_frequency = self._cached_document_frequency(cell_idx, feat_idx)
+ if document_frequency is None:
+ document_frequency = show_dask_progress(
+ counts.count_nonzero(axis=0),
+ f"({self.name}) Computing document frequency across selected cells",
+ self.nthreads,
+ )
+ self.n_docs_per_term = np.asarray(document_frequency)
+ return self.normMethod(self, counts)
+
+ def _terms_per_document(
+ self,
+ cell_idx: np.ndarray,
+ *,
+ counts: ChunkedArray | None = None,
+ renormalize_subset: bool = False,
+ ) -> np.ndarray:
+ """Return total ATAC counts used as each cell's TF denominator."""
+ if renormalize_subset:
+ if counts is None:
+ raise ValueError(
+ "Selected counts are required for subset renormalization"
+ )
+ if len(cell_idx) == 0:
+ terms = np.zeros(0, dtype=np.float64)
+ else:
+ terms = np.asarray(
+ show_dask_progress(
+ counts.sum(axis=1),
+ f"({self.name}) Recomputing counts across selected peaks",
+ self.nthreads,
+ ),
+ dtype=np.float64,
+ )
+ else:
+ terms = np.asarray(
+ self.cells.fetch_all(self.name + "_nCounts")[cell_idx],
+ dtype=np.float64,
+ )
+ terms[terms == 0] = 1
+ return terms
+
+ @staticmethod
+ def _normalization_identity() -> str:
+ identity = getattr(norm_tf_idf, "artifact_identity", None)
+ if identity is None:
+ raise RuntimeError("norm_tf_idf must define artifact_identity")
+ return str(identity)
+
+ @staticmethod
+ def _cell_index_digest(cell_idx: np.ndarray) -> str:
+ return array_digest(np.asarray(cell_idx, dtype=np.int64))
+
+ def _cached_document_frequency(
+ self,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ ) -> np.ndarray | None:
+ """Return cached document frequency for an identical ordered cell corpus."""
+ if self.normMethod is not norm_tf_idf:
+ return None
+ expected_digest = self._cell_index_digest(cell_idx)
+ expected_identity = self._normalization_identity()
+ for stats_loc in sorted(
+ str(key) for key in self.z.keys() if str(key).startswith("summary_stats_")
+ ):
+ try:
+ stats_group = as_zarr_group(self.z[stats_loc], name=stats_loc)
+ except TypeError:
+ continue
+ if (
+ stats_group.attrs.get(_CELL_INDEX_DIGEST_ATTR) != expected_digest
+ or stats_group.attrs.get(_NORMALIZATION_IDENTITY_ATTR)
+ != expected_identity
+ or _DOCUMENT_FREQUENCY_COLUMN not in stats_group
+ ):
+ continue
+ try:
+ stored = as_zarr_array(
+ stats_group[_DOCUMENT_FREQUENCY_COLUMN],
+ name=f"{stats_loc}/{_DOCUMENT_FREQUENCY_COLUMN}",
+ )
+ except TypeError:
+ continue
+ if stored.shape != (self.feats.N,):
+ continue
+ try:
+ values = np.asarray(stored[:], dtype=np.float64)[feat_idx]
+ except (IndexError, TypeError, ValueError):
+ continue
+ if (
+ np.isfinite(values).all()
+ and np.all(values >= 0)
+ and np.all(values <= len(cell_idx))
+ ):
+ return np.asarray(values)
+ return None
+
+ def _valid_tfidf_stats(
+ self,
+ stats_loc: str,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ ) -> bool:
+ if not self._validate_stats_loc(
+ stats_loc,
+ cell_idx,
+ feat_idx,
+ delete_on_fail=False,
+ ):
+ return False
+ try:
+ stats_group = as_zarr_group(self.z[stats_loc], name=stats_loc)
+ except (KeyError, TypeError):
+ return False
+ if (
+ stats_group.attrs.get(_CELL_INDEX_DIGEST_ATTR)
+ != self._cell_index_digest(cell_idx)
+ or stats_group.attrs.get(_NORMALIZATION_IDENTITY_ATTR)
+ != self._normalization_identity()
+ ):
+ return False
+ for column in ("prevalence", _DOCUMENT_FREQUENCY_COLUMN):
+ if column not in stats_group:
+ return False
+ try:
+ stored = as_zarr_array(
+ stats_group[column],
+ name=f"{stats_loc}/{column}",
+ )
+ except TypeError:
+ return False
+ if stored.shape != (self.feats.N,):
+ return False
+ try:
+ values = np.asarray(stored[:], dtype=np.float64)[feat_idx]
+ except (IndexError, TypeError, ValueError):
+ return False
+ if not np.isfinite(values).all():
+ return False
+ if column == _DOCUMENT_FREQUENCY_COLUMN and (
+ np.any(values < 0) or np.any(values > len(cell_idx))
+ ):
+ return False
+ return True
+
+ def _streaming_tfidf_feature_stats(
+ self,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ ) -> tuple[np.ndarray, np.ndarray]:
+ """Compute document frequency and TF-IDF prevalence in one raw-data pass."""
+ n_docs = len(cell_idx)
+ n_features = len(feat_idx)
+ if n_docs == 0 or n_features == 0:
+ return (
+ np.zeros(n_features, dtype=np.int64),
+ np.zeros(n_features, dtype=np.float64),
+ )
+
+ counts = self.rawData[:, feat_idx][cell_idx, :]
+ terms_per_document = self._terms_per_document(cell_idx)
+ document_frequency = np.zeros(n_features, dtype=np.int64)
+ term_frequency_sum = np.zeros(n_features, dtype=np.float64)
+ float_itemsize = np.dtype(np.float64).itemsize
+ feature_temporaries = 2 * n_features * float_itemsize
+ static_bytes = (
+ terms_per_document.nbytes
+ + document_frequency.nbytes
+ + term_frequency_sum.nbytes
+ + feature_temporaries
+ )
+ decode_bytes = counts._max_decode_bytes()
+ current_rows = min(int(counts.chunksize[0]), n_docs)
+ owned_bytes_per_row = counts._block_owned_bytes() // current_rows
+ working_bytes_per_row = owned_bytes_per_row + n_features * float_itemsize
+ available_bytes = int(self.resources.memoryBytes) - static_bytes - decode_bytes
+ if available_bytes < working_bytes_per_row:
+ required_bytes = static_bytes + decode_bytes + working_bytes_per_row
+ raise MemoryError(
+ "ATAC peak prevalence needs about "
+ f"{required_bytes} bytes for one row, but the operation limit is "
+ f"{self.resources.memoryBytes} bytes"
+ )
+ block_rows = min(current_rows, available_bytes // working_bytes_per_row)
+ counts = counts._with_block_size(block_rows)
+ resident_bytes = static_bytes + block_rows * n_features * float_itemsize
+ row_offset = 0
+ for raw in counts._stream_blocks(
+ nthreads=self.nthreads,
+ msg=f"({self.name}) Calculating peak prevalence across cells",
+ prefetch=1,
+ row_mask=None,
+ resident_bytes=resident_bytes,
+ ):
+ row_stop = row_offset + raw.shape[0]
+ document_frequency += np.count_nonzero(raw, axis=0)
+ scaled = np.asarray(raw, dtype=np.float64)
+ scaled /= terms_per_document[row_offset:row_stop].reshape(-1, 1)
+ term_frequency_sum += scaled.sum(axis=0)
+ row_offset = row_stop
+ if row_offset != n_docs:
+ raise RuntimeError(
+ f"({self.name}) Feature-stat stream produced {row_offset} rows; "
+ f"expected {n_docs}"
+ )
+ idf = np.log2(1 + (n_docs / (document_frequency + 1)))
+ return document_frequency, term_frequency_sum * idf
+
+ def set_feature_stats(self, cell_key: str) -> None:
+ """Calculates prevalence of each valid feature of the assay using only
+ cells that are marked True by the 'cell_key' parameter. Prevalence of a
+ feature is the sum of all its TF-IDF normalized values across cells.
+
+ Args:
+ cell_key: Name of the key (column) from cell attribute table.
+
+ Returns: None
+ """
+ feat_key = "I" # Here we choose to calculate stats for all the features
+ cell_idx, feat_idx = self._get_cell_feat_idx(cell_key, feat_key)
+ identifier, stats_loc = self._get_summary_stats_loc(cell_key)
+ if self.normMethod is norm_tf_idf:
+ cache_is_valid = self._valid_tfidf_stats(
+ stats_loc,
+ cell_idx,
+ feat_idx,
+ )
+ else:
+ cache_is_valid = self._validate_stats_loc(
+ stats_loc,
+ cell_idx,
+ feat_idx,
+ delete_on_fail=False,
+ )
+ if cache_is_valid:
+ logger.debug(f"Using cached feature stats for cell_key {cell_key}")
+ return None
+ if identifier in self.feats.locations:
+ self.feats.unmount_location(identifier)
+ if stats_loc in self.z:
+ del self.z[stats_loc]
+ if len(cell_idx) == 0 or len(feat_idx) == 0:
+ raise ValueError("Peak prevalence requires selected cells and features")
+ document_frequency: np.ndarray | None = None
+ if self.normMethod is norm_tf_idf:
+ document_frequency, prevalence = self._streaming_tfidf_feature_stats(
+ cell_idx,
+ feat_idx,
+ )
+ else:
+ prevalence = show_dask_progress(
+ self.normed(cell_idx, feat_idx).sum(axis=0),
+ f"({self.name}) Calculating peak prevalence across cells",
+ self.nthreads,
+ )
+ self.z.create_group(stats_loc, overwrite=True)
+ self.feats.mount_location(
+ as_zarr_group(self.z[stats_loc], name=stats_loc), identifier
+ )
+ self.feats.insert(
+ "prevalence", prevalence.astype(float), overwrite=True, location=identifier
+ )
+ if document_frequency is not None:
+ self.feats.insert(
+ _DOCUMENT_FREQUENCY_COLUMN,
+ document_frequency.astype(float),
+ overwrite=True,
+ location=identifier,
+ )
+ self.z[stats_loc].attrs["subset_hash"] = self._create_subset_hash(
+ cell_idx, feat_idx
+ )
+ if document_frequency is not None:
+ self.z[stats_loc].attrs[_CELL_INDEX_DIGEST_ATTR] = self._cell_index_digest(
+ cell_idx
+ )
+ self.z[stats_loc].attrs[_NORMALIZATION_IDENTITY_ATTR] = (
+ self._normalization_identity()
+ )
+ self.feats.unmount_location(identifier)
+ return None
+
+ def mark_prevalent_peaks(
+ self, cell_key: str, top_n: int, prevalence_key_name: str
+ ) -> None:
+ """Marks `top_n` peaks with highest prevalence as prevalent peaks.
+
+ Args:
+ cell_key: Cells to use for selection of most prevalent peaks. The provided value for `cell_key` should be a
+ column in cell attributes table with boolean values.
+ top_n: Number of top prevalent peaks to be selected. (Default: 500)
+ prevalence_key_name: Base label for marking prevalent peaks in the features attributes column. The value for
+ 'cell_key' parameter is prepended to this value.
+
+ Returns: None
+ """
+ import warnings
+
+ warnings.warn(
+ "ATACassay.mark_prevalent_peaks writes legacy metadata directly; "
+ "use DataStore.mark_prevalent_peaks for artifact-backed persistence.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ values = self._prevalent_peak_mask(cell_key, top_n)
+ prevalence_key_name = cell_key + "__" + prevalence_key_name
+ self.feats.insert(
+ prevalence_key_name,
+ values,
+ fill_value=False,
+ overwrite=True,
+ )
+ return None
+
+ def _prevalent_peak_mask(
+ self,
+ cell_key: str,
+ top_n: int,
+ ) -> np.ndarray:
+ if top_n >= self.feats.N:
+ raise ValueError(
+ f"ERROR: n_top should be less than total number of features ({self.feats.N})]"
+ )
+ if isinstance(top_n, int) is False or top_n < 1:
+ raise TypeError("ERROR: n_top must a positive integer value")
+ self.set_feature_stats(cell_key)
+ identifier = self._load_stats_loc(cell_key)
+ idx = (
+ pd.Series(self.feats.fetch_all(f"{identifier}_prevalence"))
+ .sort_values(ascending=False)
+ .index.values[:top_n]
+ )
+ return np.asarray(self.feats.index_to_bool(idx), dtype=bool)
diff --git a/scarf/assay/base.py b/scarf/assay/base.py
new file mode 100644
index 00000000..9c1e90d9
--- /dev/null
+++ b/scarf/assay/base.py
@@ -0,0 +1,1130 @@
+from collections.abc import Generator, Sequence
+from contextlib import contextmanager
+from contextvars import ContextVar
+from typing import Any, Literal, cast
+
+import numpy as np
+import pandas as pd
+import zarr
+from numpy.typing import NDArray
+from scipy.sparse import csr_matrix, vstack
+
+from ..matrix import ChunkedArray
+from ..metadata import MetaData
+from ..storage.budget import ResourceBudget, resolve_budget
+from ..storage.types import as_zarr_array, as_zarr_group
+from ..utils.arrays import array_digest
+from ..utils.compute import controlled_compute, show_dask_progress
+from ..utils.logging import logger
+from .normalization import NormMethod, norm_dummy, norm_lib_size
+
+type PercentFeatures = dict[str, str]
+
+_DEFER_FEATURE_PROPS: ContextVar[bool] = ContextVar(
+ "scarf_defer_feature_props",
+ default=False,
+)
+
+
+@contextmanager
+def _defer_feature_props() -> Generator[None, None, None]:
+ token = _DEFER_FEATURE_PROPS.set(True)
+ try:
+ yield
+ finally:
+ _DEFER_FEATURE_PROPS.reset(token)
+
+
+class Assay:
+ """A generic Assay class that contains methods to calculate feature level
+ statistics. It also provides a method for saving normalized subset of data
+ for later KNN graph construction.
+
+ Args:
+ z (zarr.Group): Zarr hierarchy where raw data is located
+ workspace: Workspace name when assays live under ``matrices/`` (None for legacy layout)
+ name (str): A label/name for assay.
+ cell_data: Metadata class object for the cell attributes.
+ nthreads: number of threads to use for parallel computations
+ min_cells_per_feature: Minimum cells expressing a feature for it to be kept
+
+ Attributes:
+ name: A label for the assay instance
+ z: Zarr group that contains the assay
+ cells: A Metadata class object for cell attributes
+ nthreads: number of threads to use for computations
+ rawData: chunked array containing the raw data
+ feats: a MetaData class object for feature attributes
+ attrs: Zarr attributes for the zarr group of the assay
+ normMethod: normalization method to use.
+ sf: scaling factor for doing library-size normalization
+ """
+
+ def __init__(
+ self,
+ z: zarr.Group,
+ workspace: str | None,
+ name: str, # FIXME change to assay_name
+ cell_data: MetaData,
+ nthreads: int,
+ min_cells_per_feature: int = 10,
+ matrix_root: zarr.Group | None = None,
+ resources: ResourceBudget | None = None,
+ ) -> None:
+ self.name = name
+ self.cells = cell_data
+ self.resources = resources or resolve_budget(workers=nthreads)
+ self.nthreads = self.resources.workers
+ matrix_root = z if matrix_root is None else matrix_root
+ if workspace is None:
+ counts_path = f"{name}/counts"
+ counts_t_path = f"{name}/countsT"
+ matrix_group = as_zarr_group(matrix_root[name], name=name)
+ self.rawData = ChunkedArray(
+ as_zarr_array(matrix_root[counts_path], name=counts_path),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ self.feats = MetaData(z[f"{name}/featureData"]) # type: ignore
+ self.z = as_zarr_group(z[name], name=name)
+ else:
+ counts_path = f"matrices/{name}/counts"
+ counts_t_path = f"matrices/{name}/countsT"
+ matrix_group = as_zarr_group(
+ matrix_root[f"matrices/{name}"],
+ name=f"matrices/{name}",
+ )
+ self.rawData = ChunkedArray(
+ as_zarr_array(matrix_root[counts_path], name=counts_path),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ self.feats = MetaData(z[f"{workspace}/{name}/featureData"]) # type: ignore
+ self.z = as_zarr_group(z[f"{workspace}/{name}"], name=f"{workspace}/{name}")
+ self.rawDataT: zarr.Array | None = None
+ if "countsT" in matrix_group:
+ try:
+ counts_t = as_zarr_array(matrix_group["countsT"], name=counts_t_path)
+ except TypeError:
+ logger.warning(
+ f"({self.name}) Ignoring countsT at {counts_t_path}: "
+ "expected a Zarr array"
+ )
+ else:
+ expected_shape = (self.rawData.shape[1], self.rawData.shape[0])
+ if (
+ counts_t.attrs.get("complete") is True
+ and tuple(counts_t.shape) == expected_shape
+ and np.dtype(counts_t.dtype) == np.dtype(self.rawData.dtype)
+ ):
+ self.rawDataT = counts_t
+ else:
+ logger.warning(
+ f"({self.name}) Ignoring countsT at {counts_t_path}: "
+ "incomplete or mismatched with counts"
+ )
+ self.attrs = self.z.attrs
+ if "percentFeatures" not in self.attrs:
+ self.attrs["percentFeatures"] = {}
+ self.normMethod: NormMethod = norm_dummy
+ self.sf: int | None = None
+ self.scalar: np.ndarray | None = None
+ self.n_term_per_doc: np.ndarray | None = None
+ self.n_docs: int | None = None
+ self.n_docs_per_term: np.ndarray | None = None
+ self._deferred_min_cells_per_feature: int | None = None
+ self._ini_feature_props(min_cells_per_feature)
+
+ def _percent_features(self) -> PercentFeatures:
+ raw = self.attrs.get("percentFeatures", {})
+ if not isinstance(raw, dict):
+ return {}
+ return {str(k): str(v) for k, v in raw.items()}
+
+ def normed(
+ self,
+ cell_idx: np.ndarray | None = None,
+ feat_idx: np.ndarray | None = None,
+ **kwargs: Any,
+ ) -> ChunkedArray:
+ """This function normalizes the raw and returns a delayed chunked array of
+ the normalized data.
+
+ Args:
+ cell_idx: Indices of cells to be included in the normalized matrix
+ (Default value: All those marked True in 'I' column of cell
+ attribute table)
+ feat_idx: Indices of features to be included in the normalized matrix
+ (Default value: All those marked True in 'I' column of
+ feature attribute table)
+ **kwargs:
+
+ Returns: A chunked array (delayed matrix) containing normalized data.
+ """
+ if cell_idx is None:
+ cell_idx = self.cells.active_index("I")
+ if feat_idx is None:
+ feat_idx = self.feats.active_index("I")
+ counts = self.rawData[:, feat_idx][cell_idx, :]
+ return self.normMethod(self, counts)
+
+ def to_raw_sparse(self, cell_key: str) -> csr_matrix:
+ """
+
+ Args:
+ cell_key: A column from cell attribute table. This column must be a boolean
+ type. The data will be exported for only those that have a True value
+ in this column.
+
+ Returns: A sparse matrix containing raw data.
+
+ """
+ sm = None
+ selected = self.rawData[self.cells.active_index(cell_key), :]
+ for values in selected.stream_blocks(
+ nthreads=self.nthreads,
+ msg=f"Converting {self.name} raw data to CSR",
+ ):
+ s = csr_matrix(values)
+ if sm is None:
+ sm = s
+ else:
+ sm = vstack([sm, s])
+ return sm # type: ignore
+
+ def _ini_feature_props(self, min_cells: int) -> None:
+ """
+
+ Args:
+ min_cells: Minimum number of cells per feature. Features below this
+ number are marked invalid.
+
+ Returns:
+
+ """
+ if "nCells" in self.feats.columns and "dropOuts" in self.feats.columns:
+ return
+ if _DEFER_FEATURE_PROPS.get():
+ self._deferred_min_cells_per_feature = min_cells
+ return
+ ncells = show_dask_progress(
+ (self.rawData > 0).sum(axis=0),
+ f"({self.name}) Computing nCells and dropOuts",
+ self.nthreads,
+ )
+ self._store_feature_props(ncells, min_cells)
+
+ def _store_feature_props(self, ncells: np.ndarray, min_cells: int) -> None:
+ self.feats.insert("nCells", ncells, overwrite=True)
+ self.feats.insert(
+ "dropOuts",
+ abs(self.cells.N - self.feats.fetch("nCells")),
+ overwrite=True,
+ )
+ self.feats.update_key(ncells > min_cells, "I")
+ self._deferred_min_cells_per_feature = None
+
+ def _stream_initialization_stats(
+ self,
+ *,
+ compute_n_counts: bool,
+ compute_n_features: bool,
+ compute_n_cells: bool,
+ percent_feature_indices: dict[str, np.ndarray],
+ ) -> dict[str, np.ndarray]:
+ n_cells, n_features = self.rawData.shape
+ sum_dtype = np.asarray(np.empty(0, dtype=self.rawData.dtype).sum()).dtype
+ stats: dict[str, np.ndarray] = {}
+ if compute_n_counts:
+ stats["nCounts"] = np.empty(n_cells, dtype=sum_dtype)
+ if compute_n_features:
+ stats["nFeatures"] = np.empty(n_cells, dtype=np.int64)
+ if compute_n_cells:
+ stats["nCells"] = np.zeros(n_features, dtype=np.int64)
+ for name in percent_feature_indices:
+ stats[name] = np.empty(n_cells, dtype=sum_dtype)
+
+ row_start = 0
+ for raw in self.rawData._stream_blocks(
+ nthreads=self.nthreads,
+ msg=f"Computing {self.name} initialization statistics",
+ prefetch=None,
+ row_mask=None,
+ resident_bytes=sum(value.nbytes for value in stats.values()),
+ ):
+ row_stop = row_start + raw.shape[0]
+ if compute_n_counts:
+ stats["nCounts"][row_start:row_stop] = raw.sum(axis=1)
+
+ positive = None
+ if compute_n_features or compute_n_cells:
+ positive = raw > 0
+ if compute_n_features:
+ assert positive is not None
+ stats["nFeatures"][row_start:row_stop] = positive.sum(axis=1)
+ if compute_n_cells:
+ assert positive is not None
+ stats["nCells"] += positive.sum(axis=0)
+
+ for name, feat_idx in percent_feature_indices.items():
+ stats[name][row_start:row_stop] = raw[:, feat_idx].sum(axis=1)
+ row_start = row_stop
+
+ if row_start != n_cells:
+ raise RuntimeError(
+ f"({self.name}) Initialization stream produced {row_start} rows; "
+ f"expected {n_cells}"
+ )
+ return stats
+
+ def _plan_percent_feature(
+ self,
+ feat_pattern: str,
+ name: str,
+ ) -> np.ndarray | None:
+ percent_features = self._percent_features()
+ if name in percent_features:
+ if percent_features[name] == feat_pattern:
+ return None
+ logger.debug(f"Pattern for percentage feature {name} updated")
+ self.attrs["percentFeatures"] = {
+ **percent_features,
+ **{name: feat_pattern},
+ }
+ feat_idx = sorted(
+ self.feats.get_index_by(self.feats.grep(feat_pattern), "names")
+ )
+ if len(feat_idx) == 0:
+ logger.warning(
+ f"No matches found for pattern {feat_pattern}."
+ f" Will not add/update percentage feature"
+ )
+ return None
+ return np.asarray(feat_idx, dtype=np.int64)
+
+ def _write_percent_feature(
+ self,
+ name: str,
+ total: np.ndarray,
+ *,
+ n_counts: np.ndarray | None = None,
+ ) -> None:
+ if total.sum() == 0:
+ logger.warning(
+ f"Percentage feature {name} not added because not detected in any cell"
+ )
+ return
+ if n_counts is None:
+ n_counts = self.cells.fetch_all(self.name + "_nCounts")
+ self.cells.insert(
+ name,
+ 100 * total / n_counts,
+ overwrite=True,
+ )
+
+ def add_percent_feature(self, feat_pattern: str, name: str) -> None:
+ """
+
+ Args:
+ feat_pattern: A regular expression pattern to identify the features of interest
+ name: This will be used as the name of column under which the percentages will
+ be saved
+
+ Returns:
+
+ """
+ feat_idx = self._plan_percent_feature(feat_pattern, name)
+ if feat_idx is None:
+ return None
+ total = show_dask_progress(
+ self.rawData[:, feat_idx].sum(axis=1),
+ f"({self.name}) Computing {name}",
+ self.nthreads,
+ )
+ self._write_percent_feature(name, total)
+
+ def _verify_keys(self, cell_key: str, feat_key: str) -> None:
+ """Checks if provided key names are present in cells and feature
+ attribute tables and that they are of boolean types.
+
+ Args:
+ cell_key: Name of the key (column) from cell attribute table
+ feat_key: Name of the key (column) from feature attribute table
+
+ Returns: None
+
+ Note on type checking /GA:
+ 1. ds.cells.get_dtype(cell_key) == bool returns True because dtype('bool') (from numpy) is conceptually equivalent to Python's bool.
+ 2. isinstance(ds.cells.get_dtype(cell_key), bool) returns False because dtype('bool') is a numpy.dtype object, not the native Python bool type.
+ 3. Reason: dtype('bool') is a numpy object, and isinstance checks for the exact class, which is numpy.dtype, not bool.
+
+ """
+ if cell_key not in self.cells.columns or self.cells.get_dtype(cell_key) != bool: # noqa: E721
+ raise ValueError(
+ f"ERROR: Either {cell_key} does not exist or is not bool type"
+ )
+ if feat_key not in self.feats.columns or self.feats.get_dtype(feat_key) != bool: # noqa: E721
+ raise ValueError(
+ f"ERROR: Either {feat_key} does not exist or is not bool type"
+ )
+
+ def _get_cell_feat_idx(
+ self, cell_key: str, feat_key: str
+ ) -> tuple[np.ndarray, np.ndarray]:
+ """Verifies the provided key by calling _verify_keys and fetches the
+ indices of rows that have True value in respective column.
+
+ Args:
+ cell_key: Name of the key (column) from cell attribute table
+ feat_key: Name of the key (column) from feature attribute table
+
+ Returns: A tuple of two numpy arrays corresponding to cell and feature indices
+ respectively.
+ """
+
+ self._verify_keys(cell_key, feat_key)
+ cell_idx = self.cells.active_index(cell_key)
+ feat_idx = self.feats.active_index(feat_key)
+ return cell_idx, feat_idx
+
+ @staticmethod
+ def _create_subset_hash(cell_idx: np.ndarray, feat_idx: np.ndarray) -> str:
+ """Return a stable content digest for ordered cell and feature selections.
+
+ The digest is persisted as a normalized-data cache key, so it must be
+ deterministic across processes and Python runtimes.
+ """
+ cells = np.ascontiguousarray(np.asarray(cell_idx), dtype=np.int64)
+ feats = np.ascontiguousarray(np.asarray(feat_idx), dtype=np.int64)
+ # Prefix the cell count so the cell/feature boundary is encoded. Without
+ # it, concatenation alone lets different splits (e.g. cells=[0,1],
+ # feats=[2,3] versus cells=[0,1,2], feats=[3]) collide to one digest.
+ boundary = np.array([cells.shape[0]], dtype=np.int64)
+ return array_digest(np.concatenate([boundary, cells, feats]))
+
+ @staticmethod
+ def _get_summary_stats_loc(cell_key: str) -> tuple[str, str]:
+ """A convenience method that returns the location of feature-wise
+ summary statistics Currently summaries are stored under pattern:
+ summary_stats_{cell_key}
+
+ Args:
+ cell_key: Name of the key (column) from cell attribute table
+
+ Returns: A tuple of two strings. First is the text that will be prepended to column
+ names when summary statistics are loaded onto the feature attributes table. The
+ second is the location of the summary statistics group in the zarr hierarchy of
+ the assay.
+ """
+ return f"stats_{cell_key}", f"summary_stats_{cell_key}"
+
+ def _validate_stats_loc(
+ self,
+ stats_loc: str,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ delete_on_fail: bool = True,
+ ) -> bool:
+ """Check whether the feature-wise summary statistics was previously
+ calculated on the same set of features and cells as preset in the
+ cell_idx and feat_idx parameters.
+
+ Args:
+ stats_loc: Location where the feature summary statistics are saved
+ cell_idx: The indices of the cell attribute table
+ feat_idx: The indices of the feature attribute table
+ delete_on_fail: Whether to delete the summary statistics group if the validity check fails (Default: True).
+
+ Returns: True is the validity test passes otherwise False
+ """
+ subset_hash = self._create_subset_hash(cell_idx, feat_idx)
+ if stats_loc in self.z:
+ attrs = self.z[stats_loc].attrs
+ if "subset_hash" in attrs and attrs["subset_hash"] == subset_hash:
+ return True
+ else:
+ # Reset stats loc
+ if delete_on_fail:
+ del self.z[stats_loc]
+ return False
+ else:
+ return False
+
+ def _load_stats_loc(self, cell_key: str) -> str:
+ """Loads the feature-wise summary statistics calculated on the cells
+ that are True in the 'cell_key' column.
+
+ Args:
+ cell_key: Name of the key (column) from cell attribute table
+
+ Returns: Location of the group group that contains feature-wise summary statistics
+ """
+ cell_idx, feat_idx = self._get_cell_feat_idx(cell_key, "I")
+ identifier, stats_loc = self._get_summary_stats_loc(cell_key)
+ if self._validate_stats_loc(stats_loc, cell_idx, feat_idx) is False:
+ raise KeyError(
+ f"Summary statistics have not been calculated for cell key: {cell_key}"
+ )
+ if identifier not in self.feats.locations:
+ self.feats.mount_location(
+ as_zarr_group(self.z[stats_loc], name=stats_loc), identifier
+ )
+ else:
+ logger.debug(f"Location ({stats_loc}) already mounted")
+ return identifier
+
+ @staticmethod
+ def _finalize_staged_mirror(
+ mirror: zarr.Array | None,
+ subset_hash: str,
+ subset_params: dict[str, Any],
+ ) -> None:
+ """Mark a mirrored staging array complete so staging reuses it as-is."""
+ if mirror is None:
+ return
+ mirror.attrs["staged_subset_hash"] = subset_hash
+ mirror.attrs["staged_subset_params"] = subset_params
+ mirror.attrs["staged_complete"] = True
+
+ def save_normalized_data(
+ self,
+ cell_key: str,
+ feat_key: str,
+ location: str,
+ log_transform: bool,
+ renormalize_subset: bool,
+ update_keys: bool,
+ mirror: zarr.Array | None = None,
+ artifact_mode: bool = False,
+ ) -> ChunkedArray:
+ """Create a new zarr group and saves the normalized data in the group
+ for the selected features only.
+
+ Args:
+ cell_key: Name of the key (column) from cell attribute table. The data will be saved
+ for only those cells that have a True value in this column.
+ feat_key: Name of the key (column) from feature attribute table. The data will be saved
+ for only those features that have a True value in this column
+ location: Zarr group wherein to save the normalized values
+ log_transform: Whether to log transform the values. Is only used if the 'normed' method
+ takes this parameter, ex. RNAassay
+ renormalize_subset: Only used if the 'normed' method takes this parameter. Please refer
+ to the documentation of the 'normed' method of the RNAassay for
+ further description of this parameter.
+ update_keys: Whether to update the keys. If True then the 'latest_feat_key' and
+ 'latest_cell_key' attributes of the assay will be updated. It can be useful
+ to set False in case where you only need to save the normalized data but
+ don't intend to use it directly. For example, when mapping onto a different
+ dataset and aligning features to that dataset.
+
+ Returns: A chunked array containing the normalized data
+ """
+
+ from ..storage.materialize import dask_to_zarr
+
+ # FIXME: Extensive documentation needed to justify the naming strategy of slots here
+ # Because HVGs and other feature selections have cell key appended in their metadata
+ if feat_key != "I":
+ feat_key = cell_key + "__" + feat_key
+ cell_idx, feat_idx = self._get_cell_feat_idx(cell_key, feat_key)
+ subset_hash = self._create_subset_hash(cell_idx, feat_idx)
+ subset_params: dict[str, Any] = {
+ "log_transform": log_transform,
+ "renormalize_subset": renormalize_subset,
+ }
+ normalization_identity = getattr(self.normMethod, "artifact_identity", None)
+ if normalization_identity is not None:
+ subset_params["normalization_identity"] = str(normalization_identity)
+ if artifact_mode:
+ if location not in self.z:
+ raise KeyError(f"Artifact group does not exist at {location}")
+ if location + "/data" in self.z:
+ return ChunkedArray(
+ as_zarr_array(
+ self.z[location + "/data"],
+ name=location + "/data",
+ ),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ vals = self.normed(
+ cell_idx,
+ feat_idx,
+ log_transform=log_transform,
+ renormalize_subset=renormalize_subset,
+ )
+ dask_to_zarr(
+ vals,
+ self.z,
+ location + "/data",
+ self.nthreads,
+ mirror=mirror,
+ resources=self.resources,
+ stats_group=(
+ as_zarr_group(self.z[location], name=location)
+ if artifact_mode
+ else None
+ ),
+ )
+ return ChunkedArray(
+ as_zarr_array(self.z[location + "/data"], name=location + "/data"),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ if location in self.z:
+ attrs = self.z[location].attrs
+ if (
+ attrs.get("subset_hash") == subset_hash
+ and attrs.get("subset_params") == subset_params
+ ):
+ logger.debug(
+ f"Using existing normalized data with cell key {cell_key} and feat key {feat_key}"
+ )
+ if update_keys:
+ self.attrs["latest_feat_key"] = (
+ feat_key.split("__", 1)[1] if feat_key != "I" else "I"
+ )
+ self.attrs["latest_cell_key"] = cell_key
+ return ChunkedArray(
+ as_zarr_array(self.z[location + "/data"], name=location + "/data"),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ else:
+ # Creating group here to overwrite all children
+ self.z.create_group(location, overwrite=True)
+ vals = self.normed(
+ cell_idx,
+ feat_idx,
+ log_transform=log_transform,
+ renormalize_subset=renormalize_subset,
+ )
+ dask_to_zarr(
+ vals,
+ self.z,
+ location + "/data",
+ self.nthreads,
+ mirror=mirror,
+ resources=self.resources,
+ )
+ self.z[location].attrs["subset_hash"] = subset_hash
+ self.z[location].attrs["subset_params"] = subset_params
+ self._finalize_staged_mirror(mirror, subset_hash, subset_params)
+ if update_keys:
+ self.attrs["latest_feat_key"] = (
+ feat_key.split("__", 1)[1] if feat_key != "I" else "I"
+ )
+ self.attrs["latest_cell_key"] = cell_key
+ return ChunkedArray(
+ as_zarr_array(self.z[location + "/data"], name=location + "/data"),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+
+ def iter_normed_feature_wise(
+ self,
+ cell_key: str | None,
+ feat_key: str | None,
+ batch_size: int | None,
+ msg: str | None,
+ as_dataframe: bool = True,
+ **norm_params: Any,
+ ) -> Generator[pd.DataFrame | tuple[np.ndarray, np.ndarray], None, None]:
+ """This generator iterates over all the features marked by `feat_key`
+ in batches.
+
+ Args:
+ cell_key: Name of the key (column) from cell attribute table. The data will be fetched
+ for only those cells that have a True value in this column. If None then all the cells are used
+ feat_key: Name of the key (column) from feature attribute table. The data will be fetched
+ for only those features that have a True value in this column. If None then all the features are
+ used
+ batch_size: Number of genes loaded at a time. When None, selected
+ features are grouped into chunk-aligned blocks that fit the
+ operation memory budget.
+ msg: Message to be displayed in the progress bar
+ as_dataframe: If true (default) then the yielded matrices are pandas dataframe
+ **norm_params: Extra keyword arguments forwarded to ``normed``.
+
+ Returns:
+ Generator yielding DataFrames or (matrix, feature index) tuples.
+ """
+ from ..storage.feature_stream import plan_feature_stream
+ from ..utils.progress import iter_progress
+
+ if cell_key is None:
+ cell_idx = np.array(list(range(self.cells.N)))
+ else:
+ cell_idx = self.cells.active_index(cell_key)
+
+ if feat_key is None:
+ feat_idx = np.array(list(range(self.feats.N)))
+ else:
+ feat_idx = self.feats.active_index(feat_key)
+ if msg is None:
+ msg = ""
+ data: ChunkedArray = self.normed(
+ cell_idx=cell_idx,
+ feat_idx=feat_idx,
+ **norm_params,
+ )
+ backing = cast(zarr.Array, self.rawData._backing)
+ raw_itemsize = max(1, int(np.dtype(backing.dtype).itemsize))
+ out_itemsize = max(1, int(np.dtype(data.dtype).itemsize))
+ n_cells = len(cell_idx)
+ plan = plan_feature_stream(
+ backing,
+ featureAxis=1,
+ cellAxis=0,
+ featureIndices=feat_idx,
+ cellIndices=cell_idx,
+ resources=self.resources,
+ blockBytes=lambda width: max(
+ 1,
+ n_cells * width * (raw_itemsize + 2 * out_itemsize),
+ ),
+ requestedBatchSize=batch_size,
+ )
+ logger.debug(
+ f"Will iterate over data of shape {data.shape} "
+ f"in {len(plan.blocks)} feature blocks"
+ )
+ for block in iter_progress(plan.blocks, desc=msg, total=len(plan.blocks)):
+ chunk = block.destinations
+ if as_dataframe:
+ yield pd.DataFrame(
+ controlled_compute(data[:, chunk], self.nthreads),
+ columns=block.indices,
+ )
+ else:
+ yield (
+ controlled_compute(data[:, chunk], self.nthreads).T,
+ block.indices,
+ )
+
+ def _prepare_aggregated_ordering(
+ self,
+ cell_key: str,
+ feat_key: str,
+ ordering_key: str,
+ *,
+ min_exp: float,
+ window_size: int,
+ chunk_size: int,
+ smoothen: bool,
+ z_scale: bool,
+ norm_params: dict[str, Any],
+ ) -> tuple[
+ np.ndarray,
+ np.ndarray,
+ np.ndarray,
+ int,
+ int,
+ list[str],
+ dict[str, Any],
+ ]:
+ cell_ordering = np.asarray(
+ self.cells.fetch(ordering_key, key=cell_key),
+ dtype=float,
+ )
+ cell_idx, feat_idx = self._get_cell_feat_idx(cell_key, feat_key)
+ n_cells = cell_ordering.shape[0]
+ if cell_ordering.ndim != 1 or n_cells == 0:
+ raise ValueError("Cell ordering must be a non-empty one-dimensional array")
+ if not np.isfinite(cell_ordering).all():
+ raise ValueError("Cell ordering must contain only finite values")
+ if not isinstance(window_size, int) or isinstance(window_size, bool):
+ raise TypeError("window_size must be an integer")
+ if not isinstance(chunk_size, int) or isinstance(chunk_size, bool):
+ raise TypeError("chunk_size must be an integer")
+ if window_size <= 0:
+ raise ValueError("window_size must be greater than zero")
+ if chunk_size <= 0:
+ raise ValueError("chunk_size must be greater than zero")
+
+ effective_window = min(window_size, n_cells)
+ effective_bins = min(chunk_size, n_cells)
+ if effective_window != window_size:
+ logger.warning(
+ f"Reducing window_size from {window_size} to {effective_window} "
+ "for the selected cell count"
+ )
+ if effective_bins != chunk_size:
+ logger.warning(
+ f"Reducing chunk_size from {chunk_size} to {effective_bins} "
+ "for the selected cell count"
+ )
+ hashes = [array_digest(x) for x in (cell_idx, feat_idx, cell_ordering)]
+ params = {
+ "min_exp": min_exp,
+ "window_size": window_size,
+ "effective_window": effective_window,
+ "chunk_size": chunk_size,
+ "effective_bins": effective_bins,
+ "smoothen": smoothen,
+ "z_scale": z_scale,
+ "norm_params": norm_params,
+ }
+ return (
+ cell_ordering,
+ cell_idx,
+ feat_idx,
+ effective_window,
+ effective_bins,
+ hashes,
+ params,
+ )
+
+ def _write_aggregated_ordering_group(
+ self,
+ group: zarr.Group,
+ *,
+ cell_key: str,
+ feat_key: str,
+ cell_ordering: np.ndarray,
+ feat_idx: np.ndarray,
+ min_exp: float,
+ effective_window: int,
+ effective_bins: int,
+ smoothen: bool,
+ z_scale: bool,
+ batch_size: int | None,
+ norm_params: dict[str, Any],
+ ) -> tuple[ChunkedArray, np.ndarray, np.ndarray]:
+ from ..storage.arrays import create_numeric_array, create_zarr_dataset
+ from ..storage.layout import row_sharded_array_spec
+ from ..storage.profiles import resolve_storage_profile
+ from ..trajectory.feature_dynamics import aggregate_feature_profiles
+
+ aggregated_shape = (int(feat_idx.shape[0]), int(effective_bins))
+ data_array = create_numeric_array(
+ group,
+ "data",
+ row_sharded_array_spec(
+ aggregated_shape,
+ "float64",
+ profile=resolve_storage_profile(group.store),
+ band_rows=max(1, aggregated_shape[0]),
+ ),
+ )
+ ordering_idx = np.argsort(cell_ordering, kind="stable")
+ stored_feat_idx: list[int] = []
+ valid_feat_flags: list[bool] = []
+ offset = 0
+ for item in self.iter_normed_feature_wise(
+ cell_key,
+ feat_key,
+ batch_size,
+ "Binning over cell-ordering",
+ True,
+ **norm_params,
+ ):
+ frame = cast(pd.DataFrame, item)
+ stored_feat_idx.extend(list(frame.columns))
+ aggregated, valid_features = aggregate_feature_profiles(
+ frame.to_numpy(dtype=float),
+ ordering_idx,
+ np.asarray(frame.columns),
+ min_expression=min_exp,
+ window_size=effective_window,
+ n_bins=effective_bins,
+ smooth=smoothen,
+ z_scale=z_scale,
+ )
+ valid_feat_flags.extend(valid_features.tolist())
+ data_array[offset : offset + aggregated.shape[0]] = aggregated
+ offset += aggregated.shape[0]
+
+ feature_indices = np.asarray(stored_feat_idx, dtype=np.uint64)
+ valid = np.asarray(valid_feat_flags, dtype=bool)
+ feature_array = create_zarr_dataset(
+ group,
+ "feature_indices",
+ (max(len(feature_indices), 1),),
+ "uint64",
+ (len(feature_indices),),
+ )
+ feature_array[:] = feature_indices
+ valid_array = create_zarr_dataset(
+ group,
+ "valid_features",
+ (max(len(valid), 1),),
+ "bool",
+ (len(valid),),
+ )
+ valid_array[:] = valid
+ return (
+ ChunkedArray(
+ data_array,
+ nthreads=self.nthreads,
+ resources=self.resources,
+ ),
+ feature_indices,
+ valid,
+ )
+
+ def save_aggregated_ordering(
+ self,
+ cell_key: str,
+ feat_key: str,
+ ordering_key: str,
+ min_exp: float = 1e-3,
+ window_size: int = 200,
+ chunk_size: int = 50,
+ smoothen: bool = True,
+ z_scale: bool = True,
+ batch_size: int | None = None,
+ **norm_params: Any,
+ ) -> tuple[ChunkedArray, NDArray[Any]]:
+ """Bin normalized expression along a cell ordering and cache the result.
+
+ Args:
+ cell_key: Boolean column in cell metadata selecting cells.
+ feat_key: Boolean column in feature metadata selecting features.
+ ordering_key: Cell metadata column with pseudotime or ordering values.
+ min_exp: Minimum mean expression to retain a feature.
+ window_size: Rolling window size for smoothing along ordering.
+ chunk_size: Number of ordering bins stored per feature row.
+ smoothen: Whether to apply rolling-window smoothing.
+ z_scale: Whether to z-scale values within each feature.
+ batch_size: Feature batch size for iteration. When None, selected
+ features are grouped into chunk-aligned blocks that fit the
+ operation memory budget.
+ **norm_params: Extra keyword arguments forwarded to ``normed``.
+
+ Returns:
+ None
+ """
+ import warnings
+
+ warnings.warn(
+ "Assay.save_aggregated_ordering writes the legacy cache layout; "
+ "use DataStore.run_pseudotime_aggregation for artifact-backed persistence.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+
+ (
+ cell_ordering,
+ _cell_idx,
+ feat_idx,
+ effective_window,
+ effective_bins,
+ hashes,
+ params,
+ ) = Assay._prepare_aggregated_ordering(
+ self,
+ cell_key,
+ feat_key,
+ ordering_key,
+ min_exp=min_exp,
+ window_size=window_size,
+ chunk_size=chunk_size,
+ smoothen=smoothen,
+ z_scale=z_scale,
+ norm_params=dict(norm_params),
+ )
+ location = f"aggregated_{cell_key}_{feat_key}_{ordering_key}"
+
+ def _cached_aggregation_valid() -> bool:
+ if location not in self.z:
+ return False
+ group = as_zarr_group(self.z[location], name=location)
+ attrs = group.attrs
+ if attrs.get("hashes") != hashes or attrs.get("params") != params:
+ return False
+ if not all(
+ name in group for name in ("data", "feature_indices", "valid_features")
+ ):
+ return False
+ data_arr = as_zarr_array(group["data"], name="data")
+ feat_arr = as_zarr_array(group["feature_indices"], name="feature_indices")
+ valid_arr = as_zarr_array(group["valid_features"], name="valid_features")
+ if data_arr.ndim != 2 or data_arr.shape[1] != effective_bins:
+ return False
+ if (
+ feat_arr.shape[0] != data_arr.shape[0]
+ or valid_arr.shape[0] != data_arr.shape[0]
+ ):
+ return False
+ return True
+
+ if _cached_aggregation_valid():
+ logger.debug(f"Using existing aggregated data from {location}")
+ else:
+ if location in self.z:
+ del self.z[location]
+ group = self.z.create_group(location)
+ Assay._write_aggregated_ordering_group(
+ self,
+ group,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ cell_ordering=cell_ordering,
+ feat_idx=feat_idx,
+ min_exp=min_exp,
+ effective_window=effective_window,
+ effective_bins=effective_bins,
+ smoothen=smoothen,
+ z_scale=z_scale,
+ batch_size=batch_size,
+ norm_params=dict(norm_params),
+ )
+ group.attrs["hashes"] = hashes
+ group.attrs["params"] = cast(Any, params)
+
+ ret_val1 = ChunkedArray(
+ as_zarr_array(self.z[location + "/data"], name=location + "/data"),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ ret_val2 = np.asarray(
+ as_zarr_array(
+ self.z[location + "/feature_indices"],
+ name=location + "/feature_indices",
+ )[:]
+ )
+
+ if location + "/valid_features" in self.z:
+ valid_feats = np.asarray(
+ as_zarr_array(
+ self.z[location + "/valid_features"],
+ name=location + "/valid_features",
+ )[:],
+ dtype=bool,
+ )
+ ret_val1 = ret_val1[valid_feats]
+ ret_val2 = ret_val2[valid_feats]
+
+ return ret_val1, ret_val2
+
+ def mean_features(
+ self,
+ feature_names: Sequence[str],
+ cell_key: str = "I",
+ *,
+ missing: Literal["error", "skip"] = "error",
+ ) -> np.ndarray:
+ """Per-cell mean normalized expression over named features.
+
+ Returns one value per active cell under ``cell_key``. Does not write
+ cell metadata. Distinct from ``score_features``, which subtracts a
+ control-gene background.
+ """
+ from .rna import RNAassay
+
+ if missing not in ("error", "skip"):
+ raise ValueError("missing must be 'error' or 'skip'")
+ if not feature_names:
+ raise ValueError("feature_names must be non-empty")
+
+ requested = [str(name) for name in feature_names]
+ if len(set(name.upper() for name in requested)) != len(requested):
+ raise ValueError("feature_names contains duplicate names")
+
+ name_to_indices: dict[str, list[int]] = {}
+ for index, name in enumerate(self.feats.fetch_all("names")):
+ key = str(name).upper()
+ name_to_indices.setdefault(key, []).append(index)
+
+ feature_idx: list[int] = []
+ missing_names: list[str] = []
+ for name in requested:
+ matches = name_to_indices.get(name.upper(), [])
+ if not matches:
+ missing_names.append(name)
+ continue
+ if len(matches) > 1:
+ raise ValueError(f"Feature name {name!r} matches multiple features")
+ feature_idx.append(matches[0])
+
+ if missing_names:
+ if missing == "error":
+ raise ValueError("Features not found: " + ", ".join(missing_names))
+ if not feature_idx:
+ raise ValueError("No requested features were found")
+
+ cell_idx, _ = self._get_cell_feat_idx(cell_key, "I")
+ feat_idx = np.asarray(feature_idx, dtype=int)
+ if isinstance(self, RNAassay) and self.normMethod is norm_lib_size:
+ means = self._mean_normed_feature_groups(
+ cell_idx,
+ {"target": feat_idx},
+ )
+ return np.asarray(means["target"])
+ return np.asarray(
+ self.normed(cell_idx=cell_idx, feat_idx=np.sort(feat_idx))
+ .mean(axis=1)
+ .compute()
+ )
+
+ def score_features(
+ self,
+ feature_names: list[str],
+ cell_key: str,
+ ctrl_size: int,
+ n_bins: int,
+ rand_seed: int,
+ ) -> np.ndarray:
+ """Calculates the scores (mean values) of selection of features over a
+ randomly sampled selected feature set in given cells (as marked by
+ cell_key)
+
+ Args:
+ feature_names: Names (as in 'names' column of the feature attribute table) of features to
+ be used for scoring
+ cell_key: Name of the key (column) from cell attribute table.
+ ctrl_size: Number of reference features to be sampled from each bin.
+ n_bins: Number of bins for sampling.
+ rand_seed: The seed to use for the random number generation.
+
+ Returns: Numpy array of the calculated scores
+ """
+
+ from ..features.scoring import binned_sampling
+ from .rna import RNAassay
+
+ def _names_to_idx(i: list[str]) -> np.ndarray:
+ return self.feats.get_index_by(i, "names", None)
+
+ def _calc_mean(i: np.ndarray | list[str] | list[int]) -> np.ndarray:
+ if isinstance(i, list):
+ if not i or isinstance(i[0], str):
+ feat_selection = self.feats.get_index_by(i, "names", None)
+ else:
+ feat_selection = np.asarray(i, dtype=int)
+ else:
+ feat_selection = i
+ return (
+ self.normed(cell_idx=cell_idx, feat_idx=np.sort(feat_selection))
+ .mean(axis=1)
+ .compute()
+ )
+
+ feature_idx = _names_to_idx(feature_names)
+ if len(feature_idx) == 0:
+ raise ValueError(
+ f"ERROR: No feature ids found for any of the provided {len(feature_names)} features"
+ )
+
+ identifier = self._load_stats_loc(cell_key)
+ obs_avg = pd.Series(self.feats.fetch_all(f"{identifier}_avg"))
+ control_idx = binned_sampling(
+ obs_avg, list(feature_idx), ctrl_size, n_bins, rand_seed
+ )
+ cell_idx, _ = self._get_cell_feat_idx(cell_key, "I")
+ if isinstance(self, RNAassay) and self.normMethod is norm_lib_size:
+ means = self._mean_normed_feature_groups(
+ cell_idx,
+ {
+ "target": np.asarray(feature_idx, dtype=int),
+ "control": np.asarray(control_idx, dtype=int),
+ },
+ )
+ return np.asarray(means["target"] - means["control"])
+ return np.asarray(_calc_mean(feature_idx) - _calc_mean(control_idx))
+
+ def __repr__(self) -> str:
+ f = self.feats.fetch_all("I")
+ assay_name = str(self.__class__).split(".")[-1][:-2]
+ return f"{assay_name} {self.name} with {f.sum()}({len(f)}) features"
diff --git a/scarf/assay/normalization.py b/scarf/assay/normalization.py
new file mode 100644
index 00000000..1f67fb8f
--- /dev/null
+++ b/scarf/assay/normalization.py
@@ -0,0 +1,106 @@
+from collections.abc import Callable
+from typing import TYPE_CHECKING, Any, cast
+
+import numpy as np
+from numpy.typing import NDArray
+
+from ..matrix import ChunkedArray
+
+if TYPE_CHECKING:
+ from .base import Assay
+
+type NormMethod = Callable[["Assay", ChunkedArray], ChunkedArray]
+
+
+def norm_dummy(_: "Assay", counts: ChunkedArray) -> ChunkedArray:
+ """A dummy normalizer. Doesn't perform any normalization. This is useful
+ when the 'raw data' is already normalized.
+
+ Args:
+ _:
+ counts: A chunked array with 'raw' counts data
+
+ Returns: A chunked array
+ """
+ return counts
+
+
+def norm_lib_size(assay: "Assay", counts: ChunkedArray) -> ChunkedArray:
+ """Performs library size normalization on the data. This is the default
+ method for RNA assays.
+
+ Args:
+ assay: An instance of the assay object
+ counts: A chunked array with raw counts data
+
+ Returns: A chunked array (delayed matrix) containing normalized data.
+ """
+ assert assay.sf is not None and assay.scalar is not None
+ return assay.sf * counts / assay.scalar.reshape(-1, 1)
+
+
+def lib_size_feature_stream_eligible(
+ assay: "Assay",
+ *,
+ renormalize_subset: bool = False,
+) -> bool:
+ """True when column-wise lib-size streaming matches ``normed`` semantics."""
+ return (
+ assay.normMethod is norm_lib_size
+ and not renormalize_subset
+ and getattr(assay, "sf", None) is not None
+ )
+
+
+def norm_lib_size_log(assay: "Assay", counts: ChunkedArray) -> ChunkedArray:
+ """Performs library size normalization and then transforms the values into
+ log scale.
+
+ Args:
+ assay: An instance of the assay object
+ counts: A chunked array with raw counts data
+
+ Returns: A chunked array (delayed matrix) containing normalized data.
+ """
+ assert assay.sf is not None and assay.scalar is not None
+ return cast(ChunkedArray, np.log1p(assay.sf * counts / assay.scalar.reshape(-1, 1)))
+
+
+def norm_clr(_: "Assay", counts: ChunkedArray) -> ChunkedArray:
+ """Performs centered log-ratio normalization (ADT). This is the default
+ method for ADT assays.
+
+ Args:
+ _:
+ counts: A chunked array with raw counts data
+
+ Returns: A chunked array (delayed matrix) containing normalized data.
+ """
+ f = np.exp(cast(NDArray[Any], np.log1p(counts).sum(axis=0)) / len(counts))
+ return cast(ChunkedArray, np.log1p(counts / f))
+
+
+def norm_tf_idf(assay: "Assay", counts: ChunkedArray) -> ChunkedArray:
+ """Performs TF-IDF normalization This is the default method for ATAC
+ assays.
+
+ Args:
+ assay: An instance of the assay object
+ counts: A chunked array with raw counts data
+
+ Returns: A chunked array (delayed matrix) containing normalized data.
+ """
+ assert (
+ assay.n_term_per_doc is not None
+ and assay.n_docs is not None
+ and assay.n_docs_per_term is not None
+ )
+ t_f = counts / assay.n_term_per_doc.reshape(-1, 1)
+ # TODO: Split TF and IDF functionality to make it similar to norml_lib and zscaling
+ idf = np.log2(1 + (assay.n_docs / (assay.n_docs_per_term + 1)))
+ return t_f * idf.reshape(1, -1)
+
+
+norm_tf_idf.artifact_identity = ( # type: ignore[attr-defined]
+ "scarf.assay.norm_tf_idf:selected-cell-df:total-count-tf"
+)
diff --git a/scarf/assay/persistence.py b/scarf/assay/persistence.py
new file mode 100644
index 00000000..16f483a4
--- /dev/null
+++ b/scarf/assay/persistence.py
@@ -0,0 +1,31 @@
+import numpy as np
+import zarr
+
+
+def _read_block(
+ zarr_arr: zarr.Array,
+ row_idx: np.ndarray,
+ col_idx: np.ndarray,
+) -> np.ndarray:
+ """Read ``zarr_arr[row_idx, col_idx]`` returning rows/cols in index order.
+
+ A basic slice is used only for an index run that is provably contiguous
+ (consecutive ascending integers), so it selects exactly the requested
+ positions and never includes neighbouring rows or columns. Any other
+ selection falls back to orthogonal (fancy) indexing, which preserves the
+ order of the index arrays. This centralizes the read path so callers never
+ hand-roll ``slice(idx[0], idx[-1] + 1)``.
+ """
+ from ..matrix._indexing import is_contiguous
+
+ def axis_sel(idx: np.ndarray) -> slice | np.ndarray:
+ idx = np.asarray(idx)
+ if idx.size > 0 and is_contiguous(idx):
+ return slice(int(idx[0]), int(idx[-1]) + 1)
+ return idx
+
+ row_sel = axis_sel(row_idx)
+ col_sel = axis_sel(col_idx)
+ if isinstance(row_sel, slice) and isinstance(col_sel, slice):
+ return np.asarray(zarr_arr[row_sel, col_sel])
+ return np.asarray(zarr_arr.get_orthogonal_selection((row_sel, col_sel)))
diff --git a/scarf/assay/rna.py b/scarf/assay/rna.py
new file mode 100644
index 00000000..507cfefd
--- /dev/null
+++ b/scarf/assay/rna.py
@@ -0,0 +1,1210 @@
+from collections.abc import Generator, Sequence
+from typing import Any, Literal, cast
+
+import numpy as np
+import pandas as pd
+import zarr
+
+from ..matrix import ChunkedArray
+from ..metadata import MetaData
+from ..storage.budget import admit_stream
+from ..storage.feature_stream import FeatureStreamPlan, plan_feature_stream
+from ..storage.geometry import array_geometry
+from ..storage.partition import IndexBlock, partition_indices, row_band
+from ..storage.types import as_zarr_array, as_zarr_group
+from ..utils.compute import show_dask_progress
+from ..utils.logging import logger
+from .base import Assay
+from .normalization import (
+ lib_size_feature_stream_eligible,
+ norm_lib_size,
+ norm_lib_size_log,
+)
+
+
+def _read_facade_block(
+ zarr_arr: zarr.Array,
+ row_idx: np.ndarray,
+ col_idx: np.ndarray,
+) -> np.ndarray:
+ from . import _read_block
+
+ return _read_block(zarr_arr, row_idx, col_idx)
+
+
+def _as_feature_indexes(
+ values: Sequence[int],
+ *,
+ n_features: int,
+ name: str,
+ require_unique: bool,
+) -> np.ndarray:
+ if isinstance(values, str):
+ raise TypeError(f"{name} must be a sequence of integer feature indexes")
+ indexes = np.asarray(values)
+ if indexes.ndim != 1:
+ raise ValueError(f"{name} must be one-dimensional")
+ if indexes.size == 0:
+ return np.empty(0, dtype=np.int64)
+ if not np.issubdtype(indexes.dtype, np.integer):
+ raise TypeError(f"{name} must contain only integer feature indexes")
+ indexes = indexes.astype(np.int64, copy=False)
+ if np.any(indexes < 0) or np.any(indexes >= n_features):
+ raise IndexError(f"{name} contains an out-of-range feature index")
+ if require_unique and np.unique(indexes).size != indexes.size:
+ raise ValueError(f"{name} contains duplicate feature indexes")
+ return indexes
+
+
+def _corrected_variance_column(
+ n_bins: int,
+ lowess_frac: float,
+ bin_strategy: Literal["fixed", "adaptive"],
+) -> str:
+ if bin_strategy not in ("fixed", "adaptive"):
+ raise ValueError("bin_strategy must be either 'fixed' or 'adaptive'")
+ if bin_strategy == "adaptive":
+ if isinstance(n_bins, (bool, np.bool_)) or not isinstance(
+ n_bins,
+ (int, np.integer),
+ ):
+ raise TypeError("n_bins must be an integer")
+ if n_bins < 1:
+ raise ValueError("n_bins must be greater than 0")
+ if isinstance(lowess_frac, (bool, np.bool_)) or not isinstance(
+ lowess_frac,
+ (int, float, np.integer, np.floating),
+ ):
+ raise TypeError("lowess_frac must be numeric")
+ if not np.isfinite(lowess_frac) or not 0 <= lowess_frac <= 1:
+ raise ValueError("lowess_frac must be between 0 and 1")
+ return f"c_var__adaptive__{n_bins}__{lowess_frac}"
+ if not 0 <= lowess_frac <= 1:
+ raise ValueError("lowess_frac must be between 0 and 1")
+ return f"c_var__{n_bins}__{lowess_frac}"
+
+
+class RNAassay(Assay):
+ """This subclass of Assay is designed for feature selection and
+ normalization of scRNA-Seq data.
+
+ Args:
+ z (zarr.Group): Zarr hierarchy where raw data is located
+ name (str): A label/name for assay.
+ cell_data: Metadata class object for the cell attributes.
+ **kwargs: kwargs to be passed to the Assay class
+
+ Attributes:
+ normMethod: A pointer to the function to be used for normalization of the raw data
+ sf: scaling factor for doing library-size normalization
+ scalar: This is used to cache the library size of the cells.
+ It is set to None until normed method is called.
+ """
+
+ def __init__(
+ self,
+ z: zarr.Group,
+ name: str,
+ cell_data: MetaData,
+ *,
+ workspace: str | None = None,
+ nthreads: int = 1,
+ min_cells_per_feature: int = 10,
+ **kwargs: Any,
+ ) -> None:
+ super().__init__(
+ z=z,
+ workspace=workspace,
+ name=name,
+ cell_data=cell_data,
+ nthreads=nthreads,
+ min_cells_per_feature=min_cells_per_feature,
+ **kwargs,
+ )
+ self.normMethod = norm_lib_size
+ if "size_factor" in self.attrs:
+ self.sf = int(cast(int, self.attrs["size_factor"]))
+ else:
+ self.sf = 1000
+ self.attrs["size_factor"] = self.sf
+ self.scalar: np.ndarray | None = None
+
+ def iter_normed_feature_wise(
+ self,
+ cell_key: str | None,
+ feat_key: str | None,
+ batch_size: int | None,
+ msg: str | None,
+ as_dataframe: bool = True,
+ **norm_params: Any,
+ ) -> Generator[pd.DataFrame | tuple[np.ndarray, np.ndarray], None, None]:
+ renormalize_subset = bool(norm_params.get("renormalize_subset", False))
+ if not lib_size_feature_stream_eligible(
+ self, renormalize_subset=renormalize_subset
+ ):
+ yield from Assay.iter_normed_feature_wise(
+ self,
+ cell_key,
+ feat_key,
+ batch_size,
+ msg,
+ as_dataframe=as_dataframe,
+ **norm_params,
+ )
+ return
+
+ if cell_key is None:
+ cell_idx = np.array(list(range(self.cells.N)))
+ else:
+ cell_idx = self.cells.active_index(cell_key)
+
+ if feat_key is None:
+ feat_idx = np.array(list(range(self.feats.N)))
+ else:
+ feat_idx = self.feats.active_index(feat_key)
+
+ if msg is None:
+ msg = ""
+
+ sf = self.sf
+ if sf is None:
+ raise ValueError("RNA library-size normalization requires a size factor")
+ scalar = self.cells.fetch_all(self.name + "_nCounts")[cell_idx]
+ log_transform = bool(norm_params.get("log_transform", False))
+ zarr_arr, feature_axis, cell_axis = self._raw_feature_stream_source()
+ raw_itemsize = max(1, int(np.dtype(zarr_arr.dtype).itemsize))
+ n_cells = len(cell_idx)
+ plan = plan_feature_stream(
+ zarr_arr,
+ featureAxis=feature_axis,
+ cellAxis=cell_axis,
+ featureIndices=feat_idx,
+ cellIndices=cell_idx,
+ resources=self.resources,
+ blockBytes=lambda width: max(
+ 1,
+ n_cells
+ * width
+ * (
+ raw_itemsize
+ + np.dtype(np.float32).itemsize
+ + np.dtype(np.float64).itemsize
+ ),
+ ),
+ requestedBatchSize=batch_size,
+ )
+ for mat, cols in self._iter_raw_feature_columns(
+ cell_idx=cell_idx,
+ feat_idx=feat_idx,
+ batch_size=batch_size,
+ scalar=scalar,
+ sf=float(sf),
+ log_transform=log_transform,
+ msg=msg,
+ plan=plan,
+ ):
+ mat64 = np.asarray(mat, dtype=np.float64)
+ if as_dataframe:
+ yield pd.DataFrame(mat64, columns=cols)
+ else:
+ yield mat64.T, cols
+
+ def save_normalized_data(
+ self,
+ cell_key: str,
+ feat_key: str,
+ location: str,
+ log_transform: bool,
+ renormalize_subset: bool,
+ update_keys: bool,
+ mirror: zarr.Array | None = None,
+ artifact_mode: bool = False,
+ ) -> ChunkedArray:
+ if not renormalize_subset:
+ return super().save_normalized_data(
+ cell_key,
+ feat_key,
+ location,
+ log_transform,
+ renormalize_subset,
+ update_keys,
+ mirror=mirror,
+ artifact_mode=artifact_mode,
+ )
+
+ from ..storage.materialize import write_renorm_subset_to_zarr
+
+ if feat_key != "I":
+ feat_key = cell_key + "__" + feat_key
+ cell_idx, feat_idx = self._get_cell_feat_idx(cell_key, feat_key)
+ if artifact_mode:
+ if location not in self.z:
+ raise KeyError(f"Artifact group does not exist at {location}")
+ if location + "/data" in self.z:
+ return ChunkedArray(
+ as_zarr_array(
+ self.z[location + "/data"],
+ name=location + "/data",
+ ),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ write_renorm_subset_to_zarr(
+ self,
+ cell_idx,
+ feat_idx,
+ self.z,
+ location + "/data",
+ self.nthreads,
+ log_transform=log_transform,
+ mirror=mirror,
+ stats_group=(
+ as_zarr_group(self.z[location], name=location)
+ if artifact_mode
+ else None
+ ),
+ )
+ return ChunkedArray(
+ as_zarr_array(self.z[location + "/data"], name=location + "/data"),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ subset_hash = self._create_subset_hash(cell_idx, feat_idx)
+ subset_params = {
+ "log_transform": log_transform,
+ "renormalize_subset": renormalize_subset,
+ }
+ if location in self.z:
+ attrs = self.z[location].attrs
+ if (
+ attrs.get("subset_hash") == subset_hash
+ and attrs.get("subset_params") == subset_params
+ ):
+ logger.debug(
+ f"Using existing normalized data with cell key {cell_key} and feat key {feat_key}"
+ )
+ if update_keys:
+ self.attrs["latest_feat_key"] = (
+ feat_key.split("__", 1)[1] if feat_key != "I" else "I"
+ )
+ self.attrs["latest_cell_key"] = cell_key
+ return ChunkedArray(
+ as_zarr_array(self.z[location + "/data"], name=location + "/data"),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ self.z.create_group(location, overwrite=True)
+
+ write_renorm_subset_to_zarr(
+ self,
+ cell_idx,
+ feat_idx,
+ self.z,
+ location + "/data",
+ self.nthreads,
+ log_transform=log_transform,
+ mirror=mirror,
+ )
+ self.z[location].attrs["subset_hash"] = subset_hash
+ self.z[location].attrs["subset_params"] = subset_params
+ self._finalize_staged_mirror(mirror, subset_hash, subset_params)
+ if update_keys:
+ self.attrs["latest_feat_key"] = (
+ feat_key.split("__", 1)[1] if feat_key != "I" else "I"
+ )
+ self.attrs["latest_cell_key"] = cell_key
+ return ChunkedArray(
+ as_zarr_array(self.z[location + "/data"], name=location + "/data"),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+
+ def normed(
+ self,
+ cell_idx: np.ndarray | None = None,
+ feat_idx: np.ndarray | None = None,
+ renormalize_subset: bool = False,
+ log_transform: bool = False,
+ **kwargs: Any,
+ ) -> ChunkedArray:
+ """This function normalizes the raw and returns a delayed chunked array of
+ the normalized data. Unlike the `normed` method in the generic Assay
+ class this method is optimized for scRNA-Seq data and takes additional
+ parameters that will be used by `norm_lib_size` (default normalization
+ method for this class).
+
+ Args:
+ cell_idx: Indices of cells to be included in the normalized matrix
+ (Default value: All those marked True in 'I' column of cell
+ attribute table)
+ feat_idx: Indices of features to be included in the normalized matrix
+ (Default value: All those marked True in 'I' column of
+ feature attribute table)
+ renormalize_subset: If True, then the data is normalized using only those features that are True in
+ `feat_key` column rather using total expression of all features in a cell
+ (Default value: False)
+ log_transform: If True, then the normalized data is log-transformed (Default value: False).
+ **kwargs: kwargs have no effect here.
+
+ Returns:
+ A chunked array (delayed matrix) containing normalized data.
+ """
+ if cell_idx is None:
+ cell_idx = self.cells.active_index("I")
+ if feat_idx is None:
+ feat_idx = self.feats.active_index("I")
+ counts = self.rawData[:, feat_idx][cell_idx, :]
+ norm_method_cache = self.normMethod
+ scalar_cache = self.scalar
+ try:
+ if log_transform:
+ self.normMethod = norm_lib_size_log
+ if renormalize_subset:
+ scalar = show_dask_progress(
+ counts.sum(axis=1),
+ "Normalizing with feature subset",
+ self.nthreads,
+ )
+ scalar[scalar == 0] = 1
+ self.scalar = scalar
+ else:
+ self.scalar = self.cells.fetch_all(self.name + "_nCounts")[cell_idx]
+ return self.normMethod(self, counts)
+ finally:
+ self.normMethod = norm_method_cache
+ self.scalar = scalar_cache
+
+ def _raw_feature_stream_source(self) -> tuple[zarr.Array, int, int]:
+ """Return the preferred raw array and its feature and cell axes."""
+ if self.rawDataT is not None:
+ return self.rawDataT, 0, 1
+ return cast(zarr.Array, self.rawData._backing), 1, 0
+
+ def iter_raw_column_blocks(
+ self,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ batch_size: int,
+ msg: str | None = None,
+ ) -> Generator[tuple[int, np.ndarray, np.ndarray, float, str], None, None]:
+ """Read raw count column batches with shallow read-ahead."""
+ yield from self._iter_raw_column_blocks(
+ cell_idx=cell_idx,
+ feat_idx=feat_idx,
+ batch_size=batch_size,
+ msg=msg,
+ )
+
+ def _iter_raw_column_blocks(
+ self,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ batch_size: int | None,
+ msg: str | None = None,
+ *,
+ plan: FeatureStreamPlan | None = None,
+ ) -> Generator[tuple[int, np.ndarray, np.ndarray, float, str], None, None]:
+ """Read raw count column batches with shallow read-ahead.
+
+ Yields ``(block_idx, raw, feat_cols, read_sec, source)`` where ``raw`` has
+ shape ``(len(cell_idx), len(feat_cols))``.
+ """
+ from ..utils.prefetch import iter_column_blocks
+
+ cell_idx = np.asarray(cell_idx)
+ feat_idx = np.asarray(feat_idx)
+ zarr_arr, feature_axis, cell_axis = self._raw_feature_stream_source()
+ if plan is None:
+ raw_itemsize = max(1, int(np.dtype(zarr_arr.dtype).itemsize))
+ plan = plan_feature_stream(
+ zarr_arr,
+ featureAxis=feature_axis,
+ cellAxis=cell_axis,
+ featureIndices=feat_idx,
+ cellIndices=cell_idx,
+ resources=self.resources,
+ blockBytes=lambda width: max(
+ 1,
+ len(cell_idx) * width * raw_itemsize,
+ ),
+ requestedBatchSize=batch_size,
+ )
+ batches = [block.indices for block in plan.blocks]
+ n_blocks = len(plan.blocks)
+ if msg:
+ logger.debug(
+ f"({self.name}) {msg}: {len(feat_idx)} features in "
+ f"{n_blocks} geometry-planned blocks "
+ f"(repeated chunk decodes={plan.repeatedDecodeCount})"
+ )
+
+ if feature_axis == 0:
+
+ def read_block(block_idx: int) -> np.ndarray:
+ return _read_facade_block(zarr_arr, batches[block_idx], cell_idx).T
+
+ else:
+
+ def read_block(block_idx: int) -> np.ndarray:
+ return _read_facade_block(zarr_arr, cell_idx, batches[block_idx])
+
+ for block_idx, raw, read_sec, source in iter_column_blocks(
+ n_blocks,
+ read_block,
+ workers=plan.readWorkers,
+ io_concurrency=plan.ioConcurrency,
+ msg=msg,
+ ):
+ yield block_idx, raw, batches[block_idx], read_sec, source
+
+ def iter_raw_feature_major_blocks(
+ self,
+ cell_idx: np.ndarray,
+ plan: FeatureStreamPlan,
+ msg: str | None = None,
+ ) -> Generator[
+ tuple[IndexBlock, np.ndarray, float, str],
+ None,
+ None,
+ ]:
+ """Yield C-contiguous ``(features, cells)`` raw count blocks."""
+ from ..utils.prefetch import iter_column_blocks
+
+ cell_idx = np.asarray(cell_idx)
+ zarr_arr, feature_axis, _ = self._raw_feature_stream_source()
+ if feature_axis != plan.featureAxis:
+ raise ValueError("Feature stream plan does not match the raw source")
+ blocks = plan.blocks
+
+ if feature_axis == 0:
+
+ def read_block(block_idx: int) -> np.ndarray:
+ block = blocks[block_idx]
+ return np.ascontiguousarray(
+ _read_facade_block(
+ zarr_arr,
+ block.indices,
+ cell_idx,
+ )
+ )
+
+ else:
+
+ def read_block(block_idx: int) -> np.ndarray:
+ block = blocks[block_idx]
+ raw = _read_facade_block(
+ zarr_arr,
+ cell_idx,
+ block.indices,
+ )
+ return np.ascontiguousarray(raw.T)
+
+ for block_idx, raw, read_sec, source in iter_column_blocks(
+ len(blocks),
+ read_block,
+ workers=plan.readWorkers,
+ io_concurrency=plan.ioConcurrency,
+ msg=msg,
+ ):
+ yield blocks[block_idx], raw, read_sec, source
+ del raw
+
+ def iter_raw_feature_columns(
+ self,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ batch_size: int,
+ scalar: np.ndarray,
+ sf: float,
+ log_transform: bool = False,
+ msg: str | None = None,
+ ) -> Generator[tuple[np.ndarray, np.ndarray], None, None]:
+ """Iterate library-size normalized feature columns."""
+ yield from self._iter_raw_feature_columns(
+ cell_idx=cell_idx,
+ feat_idx=feat_idx,
+ batch_size=batch_size,
+ scalar=scalar,
+ sf=sf,
+ log_transform=log_transform,
+ msg=msg,
+ )
+
+ def _iter_raw_feature_columns(
+ self,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ batch_size: int | None,
+ scalar: np.ndarray,
+ sf: float,
+ log_transform: bool = False,
+ msg: str | None = None,
+ *,
+ plan: FeatureStreamPlan | None = None,
+ ) -> Generator[tuple[np.ndarray, np.ndarray], None, None]:
+ """Iterate library-size normalized feature columns without streaming
+ the full normalized matrix.
+
+ Raw count columns are read directly from the backing Zarr array in
+ chunk-aligned batches and normalized in memory using a precomputed
+ per-cell scalar (library size). Reads are prefetched in parallel.
+
+ Args:
+ cell_idx: Integer indices of cells to include (in output order).
+ feat_idx: Integer indices of features to iterate over.
+ batch_size: Number of feature columns per batch.
+ scalar: Per-cell normalization factor aligned to ``cell_idx``.
+ sf: Size factor multiplier applied before dividing by ``scalar``.
+ log_transform: If True, apply ``log1p`` after normalization.
+ msg: Progress bar description.
+
+ Yields:
+ Tuples of ``(normed_batch, feat_index_batch)`` where ``normed_batch``
+ has shape ``(len(cell_idx), batch_columns)``.
+ """
+ import time
+
+ from ..utils.process import process_rss_mb
+
+ cell_idx = np.asarray(cell_idx)
+ scalar_col = np.asarray(scalar, dtype=np.float32).reshape(-1, 1)
+ scalar_col[scalar_col == 0] = 1
+ feat_idx = np.asarray(feat_idx)
+ if plan is None:
+ zarr_arr, feature_axis, cell_axis = self._raw_feature_stream_source()
+ raw_itemsize = max(1, int(np.dtype(zarr_arr.dtype).itemsize))
+ plan = plan_feature_stream(
+ zarr_arr,
+ featureAxis=feature_axis,
+ cellAxis=cell_axis,
+ featureIndices=feat_idx,
+ cellIndices=cell_idx,
+ resources=self.resources,
+ blockBytes=lambda width: max(
+ 1,
+ len(cell_idx)
+ * width
+ * (
+ raw_itemsize
+ + np.dtype(np.float32).itemsize
+ + np.dtype(np.float64).itemsize
+ ),
+ ),
+ requestedBatchSize=batch_size,
+ )
+ n_batches = len(plan.blocks)
+
+ for block_idx, raw, cols, read_sec, source in self._iter_raw_column_blocks(
+ cell_idx=cell_idx,
+ feat_idx=feat_idx,
+ batch_size=batch_size,
+ msg=msg,
+ plan=plan,
+ ):
+ t0 = time.perf_counter()
+ normed = (sf * raw.astype(np.float32)) / scalar_col
+ if log_transform:
+ normed = np.log1p(normed)
+ if msg:
+ logger.debug(
+ f"({self.name}) {msg} batch {block_idx + 1}/{n_batches}: "
+ f"cols={len(cols)} read {read_sec:.1f}s ({source}) "
+ f"norm {time.perf_counter() - t0:.1f}s "
+ f"rss {process_rss_mb():.0f} MiB"
+ )
+ yield normed, cols
+
+ def _mean_normed_feature_groups(
+ self,
+ cell_idx: np.ndarray,
+ feature_groups: dict[str, np.ndarray],
+ block_rows: int | None = None,
+ ) -> dict[str, np.ndarray]:
+ """Per-cell mean of library-size normalized counts for each feature group.
+
+ Reads the union of all requested feature columns once and streams over
+ row blocks aligned to the array's on-disk row chunk. This avoids the
+ full ChunkedArray normalization path (and
+ its repeated wide-chunk reads) used by ``normed`` when scoring small,
+ scattered gene sets such as cell cycle markers. Values are computed in
+ float64 to match ``norm_lib_size``. Row blocks are read ahead in
+ parallel and accumulated as they arrive (each writes a disjoint row
+ slice, so order does not matter).
+ """
+ from ..storage.parallel import stream_shards
+
+ zarr_arr = cast(zarr.Array, self.rawData._backing)
+ cell_idx = np.asarray(cell_idx)
+ if self.normMethod is norm_lib_size and self.sf is None:
+ raise ValueError(
+ "RNA library-size normalization requires a size factor (sf), got None"
+ )
+ sf = float(self.sf) if self.sf is not None else 1.0
+ scalar = np.asarray(
+ self.cells.fetch_all(self.name + "_nCounts")[cell_idx], dtype=np.float64
+ )
+ scalar[scalar == 0] = 1
+
+ union = np.unique(
+ np.concatenate([np.asarray(v, dtype=int) for v in feature_groups.values()])
+ )
+ local_pos = {
+ key: np.searchsorted(union, np.asarray(idx, dtype=int))
+ for key, idx in feature_groups.items()
+ }
+
+ n_cells = len(cell_idx)
+ out = {key: np.empty(n_cells, dtype=np.float64) for key in feature_groups}
+ if n_cells == 0:
+ return out
+
+ geometry = array_geometry(zarr_arr)
+ if block_rows is None:
+ block_rows = row_band(geometry, unit="chunk", fallback=n_cells)
+ block_rows = max(1, int(block_rows))
+
+ starts = range(0, n_cells, block_rows)
+
+ def read(start: int) -> tuple[int, np.ndarray]:
+ rows = cell_idx[start : start + block_rows]
+ return start, _read_facade_block(zarr_arr, rows, union)
+
+ block_bytes = (
+ block_rows
+ * max(1, len(union))
+ * (np.dtype(zarr_arr.dtype).itemsize + np.dtype(np.float64).itemsize)
+ )
+ resident_bytes = (
+ scalar.nbytes
+ + union.nbytes
+ + sum(value.nbytes for value in out.values())
+ + sum(value.nbytes for value in local_pos.values())
+ )
+ admission = admit_stream(
+ self.resources,
+ nBlocks=self.resources.workers,
+ blockBytes=block_bytes,
+ decodeBytes=0 if geometry is None else geometry.nominalChunkBytes(),
+ residentBytes=resident_bytes,
+ requested=self.resources.workers,
+ )
+ for start, raw in stream_shards(
+ starts,
+ read,
+ workers=admission.outerWorkers,
+ io_concurrency=admission.ioConcurrency,
+ ):
+ end = start + raw.shape[0]
+ normed = (sf * raw.astype(np.float64)) / scalar[start:end, None]
+ for key, pos in local_pos.items():
+ out[key][start:end] = normed[:, pos].mean(axis=1)
+ return out
+
+ def _streaming_feature_stats(
+ self,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ ) -> dict[str, np.ndarray]:
+ """Per-feature library-size normalized stats in one streaming pass.
+
+ Decodes each selected physical Zarr tile once, normalizes it in float64,
+ and accumulates per-feature nonzero count, sum, and sum of squares.
+ Reads use shallow ordered prefetch. Values match ``norm_lib_size``.
+ Returns ``normed_tot`` (sum), ``normed_n`` (nonzero count), and
+ ``sigmas`` (population variance).
+ """
+ import time
+
+ from ..utils.prefetch import iter_column_blocks
+ from ..utils.process import process_rss_mb
+
+ cell_idx = np.asarray(cell_idx)
+ feat_idx = np.asarray(feat_idx)
+ if self.normMethod is norm_lib_size and self.sf is None:
+ raise ValueError(
+ "RNA library-size normalization requires a size factor (sf), got None"
+ )
+ sf = float(self.sf) if self.sf is not None else 1.0
+ scalar = np.asarray(
+ self.cells.fetch_all(self.name + "_nCounts")[cell_idx], dtype=np.float64
+ )
+ scalar[scalar == 0] = 1
+ inv_scalar = 1.0 / scalar
+
+ n_features = len(feat_idx)
+ n_cells = len(cell_idx)
+ nz = np.zeros(n_features, dtype=np.float64)
+ s1 = np.zeros(n_features, dtype=np.float64)
+ s2 = np.zeros(n_features, dtype=np.float64)
+ if n_cells == 0 or n_features == 0:
+ return {"normed_tot": s1, "normed_n": nz, "sigmas": s2}
+
+ zarr_arr, feature_axis, cell_axis = self._raw_feature_stream_source()
+ use_counts_t = feature_axis == 0
+ geometry = array_geometry(zarr_arr)
+ if geometry is None:
+ raise ValueError("Feature stats require a chunked raw count array")
+ cell_tiles = partition_indices(geometry, cell_axis, cell_idx)
+ feat_tiles = partition_indices(geometry, feature_axis, feat_idx)
+ if use_counts_t:
+ tiles = [(cells, feats) for feats in feat_tiles for cells in cell_tiles]
+ else:
+ tiles = [(cells, feats) for cells in cell_tiles for feats in feat_tiles]
+
+ n_blocks = len(tiles)
+
+ def read_block(block_idx: int) -> np.ndarray:
+ cells, feats = tiles[block_idx]
+ if use_counts_t:
+ return _read_facade_block(zarr_arr, feats.indices, cells.indices).T
+ return _read_facade_block(zarr_arr, cells.indices, feats.indices)
+
+ def accumulate_block(
+ raw: np.ndarray,
+ local_cells: np.ndarray,
+ local_feats: np.ndarray,
+ ) -> None:
+ local_inv = inv_scalar[local_cells]
+ nz[local_feats] += (raw > 0).sum(axis=0)
+ scaled = raw.astype(np.float64, copy=True)
+ scaled *= sf
+ np.multiply(scaled, local_inv[:, None], out=scaled)
+ s1[local_feats] += scaled.sum(axis=0)
+ s2[local_feats] += np.einsum("ij,ij->j", scaled, scaled)
+
+ raw_itemsize = max(1, int(np.dtype(zarr_arr.dtype).itemsize))
+ block_bytes = max(
+ (
+ cells.indices.size
+ * feats.indices.size
+ * (raw_itemsize + np.dtype(np.float64).itemsize)
+ for cells, feats in tiles
+ ),
+ default=1,
+ )
+ resident_bytes = (
+ scalar.nbytes + inv_scalar.nbytes + nz.nbytes + s1.nbytes + s2.nbytes
+ )
+ admission = admit_stream(
+ self.resources,
+ nBlocks=max(1, n_blocks),
+ blockBytes=block_bytes,
+ decodeBytes=geometry.nominalChunkBytes(),
+ residentBytes=resident_bytes,
+ )
+ for block_idx, raw, read_sec, source in iter_column_blocks(
+ n_blocks,
+ read_block,
+ workers=admission.outerWorkers,
+ io_concurrency=admission.ioConcurrency,
+ msg=f"Computing {self.name} feature statistics",
+ ):
+ cells, feats = tiles[block_idx]
+ t_compute = time.perf_counter()
+ accumulate_block(raw, cells.destinations, feats.destinations)
+ compute_sec = time.perf_counter() - t_compute
+ logger.debug(
+ f"({self.name}) feature stats block {block_idx + 1}/{n_blocks}: "
+ f"read {read_sec:.1f}s ({source}) compute {compute_sec:.1f}s "
+ f"rss {process_rss_mb():.0f} MiB"
+ )
+ del raw
+
+ mean = s1 / n_cells
+ sigmas = s2 / n_cells - np.square(mean)
+ return {"normed_tot": s1, "normed_n": nz, "sigmas": sigmas}
+
+ def set_feature_stats(self, cell_key: str) -> None:
+ """Calculates summary statistics for the features of the assay using
+ only cells that are marked True by the 'cell_key' parameter.
+
+ Args:
+ cell_key: Name of the key (column) from cell attribute table.
+
+ Returns: None
+ """
+ feat_key = "I" # Here we choose to calculate stats for all the features
+ cell_idx, feat_idx = self._get_cell_feat_idx(cell_key, feat_key)
+ identifier, stats_loc = self._get_summary_stats_loc(cell_key)
+ if self._validate_stats_loc(stats_loc, cell_idx, feat_idx) is True:
+ logger.debug(f"Using cached feature stats for cell_key {cell_key}")
+ return None
+ else:
+ if identifier in self.feats.locations:
+ del self.feats.locations[identifier]
+ n_used = int(len(cell_idx))
+ # The single-pass streaming path only implements the library-size
+ # normalization formula (sf * raw / nCounts). Any other norm method
+ # (e.g. log-transformed or renormalized variants) falls back to the
+ # generic ChunkedArray reductions, which honour self.normMethod.
+ if self.normMethod is norm_lib_size:
+ stats = self._streaming_feature_stats(cell_idx, feat_idx)
+ n_cells = stats["normed_n"]
+ tot = stats["normed_tot"]
+ sigmas = stats["sigmas"]
+ else:
+ n_cells = show_dask_progress(
+ (self.normed(cell_idx, feat_idx) > 0).sum(axis=0),
+ f"({self.name}) Computing nCells",
+ self.nthreads,
+ )
+ tot = show_dask_progress(
+ self.normed(cell_idx, feat_idx).sum(axis=0),
+ f"({self.name}) Computing normed_tot",
+ self.nthreads,
+ )
+ sigmas = show_dask_progress(
+ self.normed(cell_idx, feat_idx).var(axis=0),
+ f"({self.name}) Computing sigmas",
+ self.nthreads,
+ )
+ # idx = n_cells > min_cells
+ # self.feats.update_key(idx, key=feat_key)
+ # n_cells, tot, sigmas = n_cells[idx], tot[idx], sigmas[idx]
+
+ self.z.create_group(stats_loc, overwrite=True)
+ self.feats.mount_location(
+ as_zarr_group(self.z[stats_loc], name=stats_loc), identifier
+ )
+ self.feats.insert(
+ "normed_tot", tot.astype(float), overwrite=True, location=identifier
+ )
+ # Mean over the cells actually used (cell_key subset), matching the
+ # denominator of the variance computed above. self.cells.N counts all
+ # primary cells, including those filtered out, so it is not used here.
+ self.feats.insert(
+ "avg",
+ (tot / max(1, n_used)).astype(float),
+ overwrite=True,
+ location=identifier,
+ )
+ nz_mean = np.divide(
+ tot, n_cells, out=np.zeros_like(tot).astype(float), where=n_cells != 0
+ )
+ self.feats.insert(
+ "nz_mean",
+ nz_mean.astype(float),
+ overwrite=True,
+ location=identifier,
+ )
+ self.feats.insert(
+ "sigmas", sigmas.astype(float), overwrite=True, location=identifier
+ )
+ self.feats.insert(
+ "normed_n", n_cells.astype(float), overwrite=True, location=identifier
+ )
+ self.z[stats_loc].attrs["subset_hash"] = self._create_subset_hash(
+ cell_idx, self.feats.active_index(feat_key)
+ )
+ self.feats.unmount_location(identifier)
+ return None
+
+ def get_feature_stats(
+ self,
+ cell_key: str,
+ columns: Sequence[str] | None = None,
+ *,
+ feat_key: str = "I",
+ ) -> dict[str, np.ndarray]:
+ """Return cached feature statistics aligned to ``feat_key``.
+
+ This method only reads an existing, valid summary-statistics group. It
+ does not calculate statistics or delete a stale cache. Default reads
+ prefer adaptive corrected variance and fall back to legacy fixed caches.
+ """
+ requested: tuple[str, ...] | None
+ if columns is None:
+ requested = None
+ elif isinstance(columns, str):
+ raise TypeError("columns must be a sequence of column names, not a string")
+ else:
+ requested = tuple(columns)
+ if requested is not None and not all(
+ isinstance(column, str) for column in requested
+ ):
+ raise TypeError("columns must contain only strings")
+
+ cell_idx, all_feat_idx = self._get_cell_feat_idx(cell_key, "I")
+ _, stats_loc = self._get_summary_stats_loc(cell_key)
+ if not self._validate_stats_loc(
+ stats_loc,
+ cell_idx,
+ all_feat_idx,
+ delete_on_fail=False,
+ ):
+ raise KeyError(
+ f"Summary statistics have not been calculated for cell key: {cell_key}"
+ )
+
+ stats_group = as_zarr_group(self.z[stats_loc], name=stats_loc)
+ if requested is None:
+ adaptive = _corrected_variance_column(200, 0.1, "adaptive")
+ fixed = _corrected_variance_column(200, 0.1, "fixed")
+ c_var_col = (
+ adaptive
+ if adaptive in stats_group or fixed not in stats_group
+ else fixed
+ )
+ requested = ("nz_mean", c_var_col, "normed_n")
+ feat_idx = self.feats.active_index(feat_key)
+ values: dict[str, np.ndarray] = {}
+ for column in requested:
+ if column not in stats_group:
+ raise KeyError(
+ f"Feature statistic {column!r} is not available for cell key "
+ f"{cell_key!r}"
+ )
+ array = as_zarr_array(stats_group[column], name=f"{stats_loc}/{column}")
+ values[column] = np.asarray(array[:])[feat_idx]
+ return values
+
+ def set_summary_stats(
+ self,
+ cell_key: str | None = None,
+ n_bins: int = 200,
+ lowess_frac: float = 0.1,
+ *,
+ bin_strategy: Literal["fixed", "adaptive"] = "adaptive",
+ ) -> tuple[str, str]:
+ """Calculates summary statistics for the features of the assay using only cells that are marked True by the 'cell_key' parameter.
+
+ Args:
+ cell_key: Name of the key (column) from cell attribute table.
+ n_bins: Number of bins to divide the data into.
+ lowess_frac: Between 0 and 1. The fraction of the data used when estimating the fit between mean and
+ variance. This is same as `frac` in statsmodels.nonparametric.smoothers_lowess.lowess
+ bin_strategy: Strategy used to construct bins and variance anchors.
+
+ Returns:
+ A tuple of two strings.
+ identifier: The text that will be prepended to column names when summary statistics are loaded onto the feature attributes table.
+ c_var_col: The name of the column in the feature attribute table that contains the corrected variance values.
+ """
+
+ def col_renamer(x: str) -> str:
+ return f"{identifier}_{x}"
+
+ if cell_key is None:
+ cell_key = "I"
+
+ c_var_col = _corrected_variance_column(
+ n_bins,
+ lowess_frac,
+ bin_strategy,
+ )
+ self.set_feature_stats(cell_key)
+ identifier = self._load_stats_loc(cell_key)
+ if col_renamer(c_var_col) in self.feats.columns:
+ logger.debug("Using existing corrected dispersion values")
+ else:
+ slots = ["normed_tot", "avg", "nz_mean", "sigmas", "normed_n"]
+ for i in slots:
+ i = col_renamer(i)
+ if i not in self.feats.columns:
+ raise KeyError(f"ERROR: {i} not found in feature metadata")
+ from ..features.variability import fit_lowess
+
+ mean = self.feats.fetch(col_renamer("avg")).astype(float)
+ variance = self.feats.fetch(col_renamer("sigmas")).astype(float)
+ positive = mean > 0
+ c_var = np.zeros(mean.shape, dtype=float)
+ c_var[positive] = fit_lowess(
+ mean[positive],
+ variance[positive],
+ n_bins,
+ lowess_frac,
+ bin_strategy=bin_strategy,
+ )
+ self.feats.insert(c_var_col, c_var, overwrite=True, location=identifier)
+
+ return identifier, c_var_col
+
+ def set_hvgs(
+ self,
+ cell_key: str,
+ *,
+ mask: np.ndarray | None = None,
+ feature_indexes: Sequence[int] | None = None,
+ hvg_key_name: str = "hvgs",
+ n_bins: int = 200,
+ lowess_frac: float = 0.1,
+ bin_strategy: Literal["fixed", "adaptive"] = "adaptive",
+ blacklist: str | None = None,
+ blacklist_exclusions: str | None = None,
+ blacklist_indexes: Sequence[int] | None = None,
+ ) -> str:
+ """Install a supplied HVG selection and ensure its summary statistics."""
+ if (mask is None) == (feature_indexes is None):
+ raise ValueError("Provide exactly one of mask or feature_indexes")
+
+ if mask is not None:
+ if not isinstance(mask, np.ndarray):
+ raise TypeError("mask must be a NumPy array")
+ if mask.shape != (self.feats.N,):
+ raise ValueError(f"mask must have shape ({self.feats.N},)")
+ if mask.dtype != bool:
+ raise TypeError("mask must have boolean dtype")
+ selected = mask.copy()
+ else:
+ assert feature_indexes is not None
+ indexes = _as_feature_indexes(
+ feature_indexes,
+ n_features=self.feats.N,
+ name="feature_indexes",
+ require_unique=True,
+ )
+ selected = self.feats.index_to_bool(indexes)
+
+ if not selected.any():
+ raise ValueError("HVG selection must contain at least one feature")
+
+ blocked = np.empty(0, dtype=np.int64)
+ if blacklist_indexes is not None:
+ blocked = _as_feature_indexes(
+ blacklist_indexes,
+ n_features=self.feats.N,
+ name="blacklist_indexes",
+ require_unique=False,
+ )
+ elif blacklist is not None:
+ blocked_names = (
+ set() if blacklist == "" else set(self.feats.grep(blacklist))
+ )
+ if blacklist_exclusions is None or blacklist_exclusions == "":
+ excluded_names: set[str] = set()
+ else:
+ excluded_names = set(self.feats.grep(blacklist_exclusions))
+ blocked = self.feats.get_index_by(
+ sorted(blocked_names - excluded_names),
+ "names",
+ ).astype(np.int64, copy=False)
+ elif blacklist_exclusions not in (None, ""):
+ raise ValueError("blacklist_exclusions requires blacklist")
+
+ self.set_summary_stats(
+ cell_key,
+ n_bins,
+ lowess_frac,
+ bin_strategy=bin_strategy,
+ )
+ selected[blocked] = False
+ column_name = f"{cell_key}__{hvg_key_name}"
+ self.feats.insert(column_name, selected, fill_value=False, overwrite=True)
+ return column_name
+
+ # maybe we should return plot here? If one wants to modify it. /raz
+ def mark_hvgs(
+ self,
+ cell_key: str,
+ min_cells: int,
+ top_n: int,
+ min_var: float,
+ max_var: float,
+ min_mean: float,
+ max_mean: float,
+ n_bins: int,
+ lowess_frac: float,
+ blacklist: str,
+ hvg_key_name: str,
+ keep_bounds: bool,
+ show_plot: bool,
+ max_cells: int | float,
+ bin_strategy: Literal["fixed", "adaptive"] = "adaptive",
+ **plot_kwargs: Any,
+ ) -> None:
+ """Identifies highly variable genes in the dataset.
+
+ The parameters govern the min/max variance (corrected) and mean expression threshold for calling genes highly
+ variable. The variance is corrected by first dividing genes into bins based on their mean expression values.
+ The fixed strategy fits a Lowess curve through minimum-variance genes, while the adaptive strategy uses balanced
+ bins and robust variance anchors. mark_hvgs will by default run on the default assay.
+ See `scarf.features.fit_lowess` for further details.
+
+ *Modifies the feats table*: adds a column named `__hvgs` to the feature table,
+ which contains a True value for genes marked HVGs. The prefix comes from the `cell_key` parameter,
+ the naming rule in Scarf dictates that cells used to identify HVGs are prepended to the column name
+ (with a double underscore delimiter).
+
+ Args:
+ cell_key: Specify which cells to use to identify the HVGs. (Default value 'I' use all non-filtered out
+ cells).
+ min_cells: Minimum number of cells where a gene should have non-zero expression values for it to be
+ considered a candidate for HVG selection. Large values for this parameter might make it difficult
+ to identify rare populations of cells. Very small values might lead to higher signal to noise
+ ratio in the selected features.
+ max_cells: Maximum number of cells where a gene should have non-zero expression values for it to be
+ considered a candidate for HVG selection. This can be useful to filter out genes that are
+ expressed in too many cells. Default value is infinity, meaning no upper limit.
+ top_n: Number of top most variable genes to be set as HVGs. This value is ignored if a value is provided
+ for `min_var` parameter.
+ min_var: Minimum variance threshold for HVG selection.
+ max_var: Maximum variance threshold for HVG selection.
+ min_mean: Minimum mean value of expression threshold for HVG selection.
+ max_mean: Maximum mean value of expression threshold for HVG selection.
+ n_bins: Number of bins into which the mean expression is binned.
+ lowess_frac: Between 0 and 1. The fraction of the data used when estimating the fit between mean and
+ variance. This is same as `frac` in statsmodels.nonparametric.smoothers_lowess.lowess
+ bin_strategy: Strategy used to construct bins and variance anchors.
+ blacklist: A regular expression string pattern. Gene names matching to this pattern will be excluded from
+ the final highly variable genes list
+ hvg_key_name: The label for highly variable genes. This label will be used to mark the HVGs in the
+ feature attribute table. The value for 'cell_key' parameter is prepended to this value.
+ keep_bounds: If True, retain upper cell-count and expression-statistic bounds.
+ The ``min_cells`` boundary is always inclusive.
+ show_plot: If True, a plot is produced, that for each gene shows the corrected variance on the y-axis and
+ the non-zero mean (means from cells where the gene had a non-zero value) on the x-axis. The
+ genes are colored in two gradients which indicate the number of cells where the gene was
+ expressed. The colors are yellow to dark red for HVGs, and blue to green for non-HVGs.
+ **plot_kwargs: Keyword arguments for ``scarf.plotting.highly_variable_features``
+ (for example ``figsize``, ``label_size``, ``point_sizes``,
+ ``colormaps``).
+ """
+
+ def col_renamer(x: str) -> str:
+ return f"{identifier}_{x}"
+
+ identifier, c_var_col = self.set_summary_stats(
+ cell_key,
+ n_bins,
+ lowess_frac,
+ bin_strategy=bin_strategy,
+ )
+ from ..features.variability import select_highly_variable_features
+
+ hvgs = select_highly_variable_features(
+ corrected_variance=self.feats.fetch_all(col_renamer(c_var_col)),
+ normalized_cell_counts=self.feats.fetch_all(col_renamer("normed_n")),
+ mean_nonzero=self.feats.fetch_all(col_renamer("nz_mean")),
+ active_features=self.feats.fetch_all("I"),
+ feature_names=self.feats.fetch_all("names"),
+ min_cells=min_cells,
+ max_cells=max_cells,
+ top_n=top_n,
+ min_var=min_var,
+ max_var=max_var,
+ min_mean=min_mean,
+ max_mean=max_mean,
+ blacklist=blacklist,
+ keep_bounds=keep_bounds,
+ )
+ hvg_key_name = cell_key + "__" + hvg_key_name
+ logger.info(f"{sum(hvgs)} genes marked as HVGs")
+ self.feats.insert(hvg_key_name, hvgs, fill_value=False, overwrite=True)
+
+ if show_plot:
+ from ..plotting import highly_variable_features
+
+ nzm, vf, nc = [
+ self.feats.fetch(x)
+ for x in [col_renamer("nz_mean"), col_renamer(c_var_col), "nCells"]
+ ]
+ highly_variable_features(
+ mean_nonzero=nzm,
+ corrected_variance=vf,
+ n_cells=nc,
+ selected=self.feats.fetch(hvg_key_name),
+ show=True,
+ **plot_kwargs,
+ )
+
+ return None
diff --git a/scarf/clustering/__init__.py b/scarf/clustering/__init__.py
new file mode 100644
index 00000000..30de0db3
--- /dev/null
+++ b/scarf/clustering/__init__.py
@@ -0,0 +1,64 @@
+from importlib import import_module as _import_module
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from .balanced_cut import BalancedCut as BalancedCut
+ from .cluster_tree import (
+ CoalesceTree as CoalesceTree,
+ make_digraph as make_digraph,
+ )
+ from .leiden import leiden_membership as leiden_membership
+ from .paris_multiscale import (
+ ParisClusterDiagnostic as ParisClusterDiagnostic,
+ ParisClusteringResult as ParisClusteringResult,
+ adaptive_cut as adaptive_cut,
+ )
+ from .paris import (
+ balanced_cut as balanced_cut,
+ paris_dendrogram as paris_dendrogram,
+ straight_cut as straight_cut,
+ )
+
+__all__ = [
+ "BalancedCut",
+ "CoalesceTree",
+ "ParisClusterDiagnostic",
+ "ParisClusteringResult",
+ "adaptive_cut",
+ "balanced_cut",
+ "leiden_membership",
+ "make_digraph",
+ "paris_dendrogram",
+ "straight_cut",
+]
+
+_LAZY_EXPORTS = {
+ "BalancedCut": (".balanced_cut", "BalancedCut"),
+ "CoalesceTree": (".cluster_tree", "CoalesceTree"),
+ "ParisClusterDiagnostic": (".paris_multiscale", "ParisClusterDiagnostic"),
+ "ParisClusteringResult": (".paris_multiscale", "ParisClusteringResult"),
+ "adaptive_cut": (".paris_multiscale", "adaptive_cut"),
+ "balanced_cut": (".paris", "balanced_cut"),
+ "leiden_membership": (".leiden", "leiden_membership"),
+ "make_digraph": (".cluster_tree", "make_digraph"),
+ "paris_dendrogram": (".paris", "paris_dendrogram"),
+ "straight_cut": (".paris", "straight_cut"),
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+
+def __getattr__(name: str) -> Any:
+ export = _LAZY_EXPORTS.get(name)
+ if export is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ module_name, attribute_name = export
+ value = getattr(_import_module(module_name, __name__), attribute_name)
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()).union(_LAZY_EXPORTS))
diff --git a/scarf/clustering/_paris_core.py b/scarf/clustering/_paris_core.py
new file mode 100644
index 00000000..bf00661a
--- /dev/null
+++ b/scarf/clustering/_paris_core.py
@@ -0,0 +1,626 @@
+import time
+from dataclasses import dataclass
+
+import numpy as np
+from numba import (
+ config,
+ get_num_threads,
+ njit,
+ prange,
+ set_num_threads,
+)
+from scipy.sparse import csr_matrix, spmatrix
+from scipy.sparse.csgraph import connected_components
+
+
+@dataclass(frozen=True)
+class ParisRoundDiagnostics:
+ """Measurements for one reciprocal-neighbor contraction round."""
+
+ component: int
+ round_index: int
+ active_vertices: int
+ active_edges: int
+ merges: int
+ scan_seconds: float
+ sort_seconds: float
+ contraction_seconds: float
+ contraction_remap_seconds: float
+ contraction_filter_seconds: float
+ contraction_build_seconds: float
+ contraction_cleanup_seconds: float
+
+
+@dataclass(frozen=True)
+class ParisFitDiagnostics:
+ """Measurements from a Paris hierarchy fit."""
+
+ preprocessing_seconds: float
+ component_seconds: float
+ fit_seconds: float
+ rounds: tuple[ParisRoundDiagnostics, ...]
+
+
+@dataclass(frozen=True)
+class ParisHierarchy:
+ """Flat-array representation of a Paris hierarchy."""
+
+ children: np.ndarray
+ heights: np.ndarray
+ sizes: np.ndarray
+ component_roots: np.ndarray
+ synthetic_joins: np.ndarray
+ n_leaves: int
+ total_weight: float
+ diagnostics: ParisFitDiagnostics | None = None
+
+ def __post_init__(self) -> None:
+ n_merges = self.n_leaves - 1
+ if self.children.shape != (n_merges, 2):
+ raise ValueError("children must have shape (n_leaves - 1, 2)")
+ for name, values in (
+ ("heights", self.heights),
+ ("sizes", self.sizes),
+ ("synthetic_joins", self.synthetic_joins),
+ ):
+ if values.shape != (n_merges,):
+ raise ValueError(f"{name} must have length n_leaves - 1")
+ for values in (
+ self.children,
+ self.heights,
+ self.sizes,
+ self.component_roots,
+ self.synthetic_joins,
+ ):
+ values.setflags(write=False)
+
+
+@njit(cache=True, nogil=True, parallel=True)
+def _nearest_neighbors(
+ indptr: np.ndarray,
+ indices: np.ndarray,
+ data: np.ndarray,
+ volumes: np.ndarray,
+ logical_ids: np.ndarray,
+ n_workers: int,
+) -> tuple[np.ndarray, np.ndarray]:
+ n_vertices = volumes.shape[0]
+ nearest = np.full(n_vertices, -1, dtype=np.int64)
+ between_weights = np.zeros(n_vertices, dtype=np.float64)
+ for worker in prange(n_workers):
+ for row in range(worker, n_vertices, n_workers):
+ best_neighbor = -1
+ best_logical_id = np.iinfo(np.int64).max
+ best_similarity = -np.inf
+ row_volume = volumes[row]
+ for offset in range(indptr[row], indptr[row + 1]):
+ neighbor = indices[offset]
+ weight = data[offset]
+ if neighbor == row or weight <= 0:
+ continue
+ similarity = weight / (row_volume * volumes[neighbor])
+ logical_id = logical_ids[neighbor]
+ if similarity > best_similarity or (
+ similarity == best_similarity and logical_id < best_logical_id
+ ):
+ best_neighbor = neighbor
+ best_logical_id = logical_id
+ best_similarity = similarity
+ between_weights[row] = weight
+ nearest[row] = best_neighbor
+ return nearest, between_weights
+
+
+@njit(cache=True, nogil=True)
+def _contraction_layout(
+ indptr: np.ndarray,
+ mapping: np.ndarray,
+ n_vertices: int,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ n_old_vertices = mapping.size
+ member_counts = np.zeros(n_vertices, dtype=mapping.dtype)
+ for old_vertex in range(n_old_vertices):
+ member_counts[mapping[old_vertex]] += 1
+
+ member_offsets = np.empty(n_vertices + 1, dtype=mapping.dtype)
+ member_offsets[0] = 0
+ for row in range(n_vertices):
+ member_offsets[row + 1] = member_offsets[row] + member_counts[row]
+
+ cursors = member_offsets[:-1].copy()
+ members = np.empty(n_old_vertices, dtype=mapping.dtype)
+ for old_vertex in range(n_old_vertices):
+ group = mapping[old_vertex]
+ members[cursors[group]] = old_vertex
+ cursors[group] += 1
+
+ workspace_offsets = np.empty(n_vertices + 1, dtype=indptr.dtype)
+ workspace_offsets[0] = 0
+ for row in range(n_vertices):
+ degree = 0
+ for position in range(member_offsets[row], member_offsets[row + 1]):
+ old_vertex = members[position]
+ degree += indptr[old_vertex + 1] - indptr[old_vertex]
+ workspace_offsets[row + 1] = workspace_offsets[row] + degree
+
+ return member_offsets, members, workspace_offsets
+
+
+@njit(cache=True, nogil=True, parallel=True)
+def _aggregate_contracted_rows(
+ indptr: np.ndarray,
+ indices: np.ndarray,
+ data: np.ndarray,
+ mapping: np.ndarray,
+ member_offsets: np.ndarray,
+ members: np.ndarray,
+ workspace_offsets: np.ndarray,
+ n_vertices: int,
+ n_workers: int,
+ values: np.ndarray,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ workspace_size = workspace_offsets[-1]
+ workspace_indices = np.empty(workspace_size, dtype=indices.dtype)
+ workspace_data = np.empty(workspace_size, dtype=data.dtype)
+ unique_counts = np.zeros(n_vertices, dtype=indices.dtype)
+
+ for worker in prange(n_workers):
+ for row in range(worker, n_vertices, n_workers):
+ output_offset = workspace_offsets[row]
+ unique_count = 0
+ for position in range(member_offsets[row], member_offsets[row + 1]):
+ old_vertex = members[position]
+ for edge in range(indptr[old_vertex], indptr[old_vertex + 1]):
+ destination = mapping[indices[edge]]
+ if destination == row:
+ continue
+ value = values[worker, destination]
+ if np.isnan(value):
+ values[worker, destination] = data[edge]
+ workspace_indices[output_offset + unique_count] = destination
+ unique_count += 1
+ else:
+ values[worker, destination] = value + data[edge]
+
+ if unique_count:
+ workspace_indices[output_offset : output_offset + unique_count].sort()
+ for position in range(unique_count):
+ destination = workspace_indices[output_offset + position]
+ workspace_data[output_offset + position] = values[
+ worker,
+ destination,
+ ]
+ values[worker, destination] = np.nan
+ unique_counts[row] = unique_count
+
+ return workspace_indices, workspace_data, unique_counts
+
+
+@njit(cache=True, nogil=True)
+def _compact_contracted_rows(
+ workspace_offsets: np.ndarray,
+ workspace_indices: np.ndarray,
+ workspace_data: np.ndarray,
+ unique_counts: np.ndarray,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ n_vertices = unique_counts.size
+ indptr = np.empty(n_vertices + 1, dtype=workspace_offsets.dtype)
+ indptr[0] = 0
+ for row in range(n_vertices):
+ indptr[row + 1] = indptr[row] + unique_counts[row]
+
+ for row in range(n_vertices):
+ source_offset = workspace_offsets[row]
+ target_offset = indptr[row]
+ count = unique_counts[row]
+ workspace_indices[target_offset : target_offset + count] = workspace_indices[
+ source_offset : source_offset + count
+ ]
+ workspace_data[target_offset : target_offset + count] = workspace_data[
+ source_offset : source_offset + count
+ ]
+ return (
+ indptr,
+ workspace_indices[: indptr[-1]],
+ workspace_data[: indptr[-1]],
+ )
+
+
+def canonicalize_paris_graph(graph: spmatrix) -> csr_matrix:
+ """Build the canonical additive, loop-free Paris graph."""
+ if graph.ndim != 2 or graph.shape[0] != graph.shape[1]:
+ raise ValueError("Paris requires a square adjacency matrix")
+ if graph.shape[0] < 2:
+ raise ValueError("Paris requires at least two vertices")
+
+ directed = csr_matrix(graph, dtype=np.float64, copy=True)
+ directed.sum_duplicates()
+ if not np.isfinite(directed.data).all():
+ raise ValueError("Paris graph weights must be finite")
+ if np.any(directed.data < 0):
+ raise ValueError("Paris graph weights must be non-negative")
+
+ additive = csr_matrix(directed + directed.T, dtype=np.float64)
+ additive.sum_duplicates()
+ additive.setdiag(0)
+ additive.eliminate_zeros()
+ additive.sort_indices()
+ if not np.isfinite(additive.data).all():
+ raise ValueError("Paris graph weights overflowed during symmetrization")
+ return additive
+
+
+def _reciprocal_pairs(
+ nearest: np.ndarray,
+) -> tuple[np.ndarray, np.ndarray]:
+ vertices = np.arange(nearest.size, dtype=np.int64)
+ valid = nearest >= 0
+ reciprocal = np.zeros(nearest.size, dtype=bool)
+ reciprocal[valid] = nearest[nearest[valid]] == vertices[valid]
+ first = np.flatnonzero(reciprocal & (vertices < nearest))
+ second = nearest[first]
+ return first, second
+
+
+@njit(cache=True, nogil=True)
+def _prepare_contraction_groups(
+ logical_ids: np.ndarray,
+ volumes: np.ndarray,
+ sizes: np.ndarray,
+ first: np.ndarray,
+ second: np.ndarray,
+ new_ids: np.ndarray,
+ mapping: np.ndarray,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ n_vertices = logical_ids.size
+ merge_count = first.size
+ unmerged_count = n_vertices - 2 * merge_count
+ new_count = n_vertices - merge_count
+ pair_by_vertex = np.full(n_vertices, -1, dtype=mapping.dtype)
+ for pair in range(merge_count):
+ pair_by_vertex[first[pair]] = pair
+ pair_by_vertex[second[pair]] = pair
+
+ new_logical_ids = np.empty(new_count, dtype=logical_ids.dtype)
+ new_volumes = np.empty(new_count, dtype=volumes.dtype)
+ new_sizes = np.empty(new_count, dtype=sizes.dtype)
+ output = 0
+ for vertex in range(n_vertices):
+ if pair_by_vertex[vertex] < 0:
+ mapping[vertex] = output
+ new_logical_ids[output] = logical_ids[vertex]
+ new_volumes[output] = volumes[vertex]
+ new_sizes[output] = sizes[vertex]
+ output += 1
+
+ if output != unmerged_count:
+ raise RuntimeError("Paris contraction group assembly failed")
+ for pair in range(merge_count):
+ output = unmerged_count + pair
+ left = first[pair]
+ right = second[pair]
+ mapping[left] = output
+ mapping[right] = output
+ new_logical_ids[output] = new_ids[pair]
+ new_volumes[output] = volumes[left] + volumes[right]
+ new_sizes[output] = sizes[left] + sizes[right]
+
+ return new_logical_ids, new_volumes, new_sizes
+
+
+def _contract_graph(
+ graph: csr_matrix,
+ mapping: np.ndarray,
+ n_vertices: int,
+ values: np.ndarray,
+) -> tuple[csr_matrix, float, float, float, float]:
+ remap_start = time.perf_counter()
+ member_offsets, members, workspace_offsets = _contraction_layout(
+ graph.indptr,
+ mapping,
+ n_vertices,
+ )
+ remap_seconds = time.perf_counter() - remap_start
+
+ filter_start = time.perf_counter()
+ n_workers = min(values.shape[0], n_vertices)
+ workspace_indices, workspace_data, unique_counts = _aggregate_contracted_rows(
+ graph.indptr,
+ graph.indices,
+ graph.data,
+ mapping,
+ member_offsets,
+ members,
+ workspace_offsets,
+ n_vertices,
+ n_workers,
+ values,
+ )
+ filter_seconds = time.perf_counter() - filter_start
+
+ build_start = time.perf_counter()
+ indptr, indices, data = _compact_contracted_rows(
+ workspace_offsets,
+ workspace_indices,
+ workspace_data,
+ unique_counts,
+ )
+ build_seconds = time.perf_counter() - build_start
+
+ cleanup_start = time.perf_counter()
+ contracted = csr_matrix(
+ (data, indices, indptr),
+ shape=(n_vertices, n_vertices),
+ copy=False,
+ )
+ cleanup_seconds = time.perf_counter() - cleanup_start
+ return (
+ contracted,
+ remap_seconds,
+ filter_seconds,
+ build_seconds,
+ cleanup_seconds,
+ )
+
+
+def _fit_component(
+ graph: csr_matrix,
+ leaf_ids: np.ndarray,
+ *,
+ component: int,
+ next_node: int,
+ total_weight: float,
+ round_diagnostics: list[ParisRoundDiagnostics],
+) -> tuple[list[np.ndarray], list[np.ndarray], list[np.ndarray], int, int]:
+ logical_ids = np.asarray(leaf_ids, dtype=np.int64)
+ volumes = np.asarray(graph.sum(axis=1)).ravel().astype(np.float64, copy=False)
+ sizes = np.ones(logical_ids.size, dtype=np.int64)
+ contraction_values = np.full(
+ (min(get_num_threads(), logical_ids.size), logical_ids.size),
+ np.nan,
+ dtype=np.float64,
+ )
+ child_blocks: list[np.ndarray] = []
+ height_blocks: list[np.ndarray] = []
+ size_blocks: list[np.ndarray] = []
+ round_index = 0
+
+ while logical_ids.size > 1:
+ active_edges = graph.nnz
+ scan_start = time.perf_counter()
+ n_workers = min(contraction_values.shape[0], logical_ids.size)
+ nearest, between_weights = _nearest_neighbors(
+ graph.indptr,
+ graph.indices,
+ graph.data,
+ volumes,
+ logical_ids,
+ n_workers,
+ )
+ first, second = _reciprocal_pairs(nearest)
+ scan_seconds = time.perf_counter() - scan_start
+ if first.size == 0:
+ raise RuntimeError(
+ "Paris reciprocal-neighbor scan made no progress in a connected component"
+ )
+
+ merge_count = first.size
+ new_ids = np.arange(next_node, next_node + merge_count, dtype=np.int64)
+ first_ids = logical_ids[first]
+ second_ids = logical_ids[second]
+ children = np.column_stack(
+ (np.minimum(first_ids, second_ids), np.maximum(first_ids, second_ids))
+ )
+ heights = (volumes[first] / total_weight) * (
+ volumes[second] / between_weights[first]
+ )
+ if not np.isfinite(heights).all():
+ raise ValueError("Paris merge distances must be finite")
+ merged_sizes = sizes[first] + sizes[second]
+ child_blocks.append(children)
+ height_blocks.append(heights)
+ size_blocks.append(merged_sizes)
+
+ sort_start = time.perf_counter()
+ index_dtype = graph.indices.dtype
+ mapping = np.empty(logical_ids.size, dtype=index_dtype)
+ new_logical_ids, new_volumes, new_sizes = _prepare_contraction_groups(
+ logical_ids,
+ volumes,
+ sizes,
+ first,
+ second,
+ new_ids,
+ mapping,
+ )
+ new_count = new_logical_ids.size
+ sort_seconds = time.perf_counter() - sort_start
+ contract_start = time.perf_counter()
+ (
+ graph,
+ contraction_remap_seconds,
+ contraction_filter_seconds,
+ contraction_build_seconds,
+ contraction_cleanup_seconds,
+ ) = _contract_graph(
+ graph,
+ mapping,
+ new_count,
+ contraction_values,
+ )
+ contract_seconds = time.perf_counter() - contract_start
+
+ round_diagnostics.append(
+ ParisRoundDiagnostics(
+ component=component,
+ round_index=round_index,
+ active_vertices=logical_ids.size,
+ active_edges=active_edges,
+ merges=merge_count,
+ scan_seconds=scan_seconds,
+ sort_seconds=sort_seconds,
+ contraction_seconds=contract_seconds,
+ contraction_remap_seconds=contraction_remap_seconds,
+ contraction_filter_seconds=contraction_filter_seconds,
+ contraction_build_seconds=contraction_build_seconds,
+ contraction_cleanup_seconds=contraction_cleanup_seconds,
+ )
+ )
+ logical_ids = new_logical_ids
+ volumes = new_volumes
+ sizes = new_sizes
+ next_node += merge_count
+ round_index += 1
+
+ return (
+ child_blocks,
+ height_blocks,
+ size_blocks,
+ int(logical_ids[0]),
+ next_node,
+ )
+
+
+def fit_paris_hierarchy(
+ graph: spmatrix,
+ *,
+ n_threads: int | None = None,
+) -> ParisHierarchy:
+ """Fit a deterministic Paris hierarchy with reciprocal-neighbor rounds."""
+ start = time.perf_counter()
+ canonical = canonicalize_paris_graph(graph)
+ preprocessing_seconds = time.perf_counter() - start
+ n_leaves = canonical.shape[0]
+ total_weight = float(canonical.data.sum(dtype=np.float64))
+ if not np.isfinite(total_weight):
+ raise ValueError("Paris graph total weight must be finite")
+
+ component_start = time.perf_counter()
+ n_components, labels = connected_components(
+ canonical,
+ directed=False,
+ return_labels=True,
+ )
+ component_mins = np.full(n_components, n_leaves, dtype=np.int64)
+ np.minimum.at(component_mins, labels, np.arange(n_leaves, dtype=np.int64))
+ component_order = np.argsort(component_mins, kind="stable")
+ vertices_by_component = np.argsort(labels, kind="stable")
+ component_counts = np.bincount(labels, minlength=n_components)
+ component_offsets = np.empty(n_components + 1, dtype=np.int64)
+ component_offsets[0] = 0
+ np.cumsum(component_counts, out=component_offsets[1:])
+ component_seconds = time.perf_counter() - component_start
+
+ requested_threads = (
+ get_num_threads() if n_threads is None else max(1, int(n_threads))
+ )
+ previous_threads = get_num_threads()
+ set_num_threads(min(requested_threads, config.NUMBA_NUM_THREADS))
+ fit_start = time.perf_counter()
+ child_blocks: list[np.ndarray] = []
+ height_blocks: list[np.ndarray] = []
+ size_blocks: list[np.ndarray] = []
+ component_roots: list[int] = []
+ component_sizes: list[int] = []
+ round_diagnostics: list[ParisRoundDiagnostics] = []
+ next_node = n_leaves
+ try:
+ for component_index, component_label in enumerate(component_order):
+ start_offset = component_offsets[component_label]
+ end_offset = component_offsets[component_label + 1]
+ leaf_ids = vertices_by_component[start_offset:end_offset].astype(
+ np.int64,
+ copy=False,
+ )
+ component_sizes.append(leaf_ids.size)
+ if leaf_ids.size == 1:
+ component_roots.append(int(leaf_ids[0]))
+ continue
+ if leaf_ids.size == n_leaves:
+ component_graph = canonical
+ else:
+ component_graph = canonical[leaf_ids][:, leaf_ids].tocsr()
+ (
+ component_children,
+ component_heights,
+ component_merge_sizes,
+ component_root,
+ next_node,
+ ) = _fit_component(
+ component_graph,
+ leaf_ids,
+ component=component_index,
+ next_node=next_node,
+ total_weight=total_weight,
+ round_diagnostics=round_diagnostics,
+ )
+ child_blocks.extend(component_children)
+ height_blocks.extend(component_heights)
+ size_blocks.extend(component_merge_sizes)
+ component_roots.append(component_root)
+ finally:
+ set_num_threads(previous_threads)
+
+ finite_merge_count = sum(block.shape[0] for block in child_blocks)
+ synthetic_count = n_components - 1
+ index_dtype = np.int32 if 2 * n_leaves - 1 <= np.iinfo(np.int32).max else np.int64
+ children = np.empty((n_leaves - 1, 2), dtype=index_dtype)
+ heights = np.empty(n_leaves - 1, dtype=np.float64)
+ size_dtype = np.int32 if n_leaves <= np.iinfo(np.int32).max else np.int64
+ sizes = np.empty(n_leaves - 1, dtype=size_dtype)
+ synthetic_joins = np.zeros(n_leaves - 1, dtype=bool)
+
+ offset = 0
+ for child_block, height_block, size_block in zip(
+ child_blocks,
+ height_blocks,
+ size_blocks,
+ strict=True,
+ ):
+ block_size = child_block.shape[0]
+ children[offset : offset + block_size] = child_block
+ heights[offset : offset + block_size] = height_block
+ sizes[offset : offset + block_size] = size_block
+ offset += block_size
+ if offset != finite_merge_count:
+ raise RuntimeError("Paris hierarchy assembly failed")
+
+ if synthetic_count:
+ root = component_roots[0]
+ root_size = component_sizes[0]
+ for component_root, component_size in zip(
+ component_roots[1:],
+ component_sizes[1:],
+ strict=True,
+ ):
+ children[offset] = (root, component_root)
+ root_size += component_size
+ heights[offset] = np.inf
+ sizes[offset] = root_size
+ synthetic_joins[offset] = True
+ root = next_node
+ next_node += 1
+ offset += 1
+
+ if offset != n_leaves - 1 or next_node != 2 * n_leaves - 1:
+ raise RuntimeError(
+ "Paris hierarchy does not contain exactly n_leaves - 1 merges"
+ )
+
+ fit_seconds = time.perf_counter() - fit_start
+ diagnostics = ParisFitDiagnostics(
+ preprocessing_seconds=preprocessing_seconds,
+ component_seconds=component_seconds,
+ fit_seconds=fit_seconds,
+ rounds=tuple(round_diagnostics),
+ )
+ return ParisHierarchy(
+ children=children,
+ heights=heights,
+ sizes=sizes,
+ component_roots=np.asarray(component_roots, dtype=index_dtype),
+ synthetic_joins=synthetic_joins,
+ n_leaves=n_leaves,
+ total_weight=total_weight,
+ diagnostics=diagnostics,
+ )
diff --git a/scarf/clustering/_paris_modularity.py b/scarf/clustering/_paris_modularity.py
new file mode 100644
index 00000000..9be3ddb3
--- /dev/null
+++ b/scarf/clustering/_paris_modularity.py
@@ -0,0 +1,541 @@
+from dataclasses import dataclass
+
+import numpy as np
+from numba import njit
+from scipy.sparse import csr_matrix
+
+from ._paris_core import ParisHierarchy, canonicalize_paris_graph
+from .paris_multiscale import PlateauForest, _validate_hierarchy
+
+
+def _freeze_arrays(*arrays: np.ndarray) -> None:
+ for values in arrays:
+ values.setflags(write=False)
+
+
+@dataclass(frozen=True, slots=True)
+class TopologyStatistics:
+ """Raw topology statistics aligned to leaves and binary merges."""
+
+ leaf_degrees: np.ndarray
+ lca_edge_counts: np.ndarray
+ component_edge_counts: np.ndarray
+
+ def __post_init__(self) -> None:
+ for name, values in (
+ ("leaf_degrees", self.leaf_degrees),
+ ("lca_edge_counts", self.lca_edge_counts),
+ ("component_edge_counts", self.component_edge_counts),
+ ):
+ if values.ndim != 1 or not np.issubdtype(values.dtype, np.integer):
+ raise TypeError(f"{name} must be a one-dimensional integer array")
+ if np.any(values < 0):
+ raise ValueError(f"{name} must be non-negative")
+ _freeze_arrays(
+ self.leaf_degrees,
+ self.lca_edge_counts,
+ self.component_edge_counts,
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class PlateauModularityStatistics:
+ """Configuration-null statistics aligned to plateau events."""
+
+ cross_edges: np.ndarray
+ volumes: np.ndarray
+
+ def __post_init__(self) -> None:
+ n_events = self.cross_edges.size
+ for name, values in (
+ ("cross_edges", self.cross_edges),
+ ("volumes", self.volumes),
+ ):
+ if values.shape != (n_events,) or not np.issubdtype(
+ values.dtype,
+ np.integer,
+ ):
+ raise TypeError(f"{name} must be an integer array over events")
+ if np.any(values < 0):
+ raise ValueError(f"{name} must be non-negative")
+ _freeze_arrays(
+ self.cross_edges,
+ self.volumes,
+ )
+
+
+@njit(cache=True, nogil=True)
+def _csr_symmetry_error(
+ indptr: np.ndarray,
+ indices: np.ndarray,
+ data: np.ndarray,
+) -> int:
+ n_vertices = indptr.size - 1
+ for source in range(n_vertices):
+ for offset in range(indptr[source], indptr[source + 1]):
+ target = int(indices[offset])
+ if target == source:
+ return 1
+ lower = int(indptr[target])
+ upper = int(indptr[target + 1])
+ while lower < upper:
+ middle = (lower + upper) // 2
+ candidate = int(indices[middle])
+ if candidate < source:
+ lower = middle + 1
+ else:
+ upper = middle
+ if lower >= indptr[target + 1] or int(indices[lower]) != source:
+ return 2
+ if data[lower] != data[offset]:
+ return 3
+ return 0
+
+
+def _validate_canonical_csr(graph: csr_matrix, n_vertices: int | None = None) -> None:
+ if not isinstance(graph, csr_matrix):
+ raise TypeError("graph must be a scipy.sparse.csr_matrix")
+ if graph.ndim != 2 or graph.shape[0] != graph.shape[1]:
+ raise ValueError("canonical graph must be square")
+ if graph.shape[0] < 1:
+ raise ValueError("canonical graph must contain at least one vertex")
+ if n_vertices is not None and graph.shape != (n_vertices, n_vertices):
+ raise ValueError("canonical graph and hierarchy have different leaf counts")
+ if not graph.has_canonical_format or not graph.has_sorted_indices:
+ raise ValueError("graph must be a sorted canonical CSR without duplicates")
+ if not np.isfinite(graph.data).all() or np.any(graph.data <= 0):
+ raise ValueError("canonical graph entries must be finite and positive")
+ symmetry_error = _csr_symmetry_error(
+ graph.indptr,
+ graph.indices,
+ graph.data,
+ )
+ if symmetry_error == 1:
+ raise ValueError("canonical graph must not contain self-loops")
+ if symmetry_error == 2:
+ raise ValueError("canonical graph must be structurally symmetric")
+ if symmetry_error == 3:
+ raise ValueError("canonical graph must be exactly symmetric")
+ if graph.nnz % 2:
+ raise ValueError("canonical graph must contain paired undirected entries")
+
+
+def _leaf_component_ids(hierarchy: ParisHierarchy) -> np.ndarray:
+ _validate_hierarchy(hierarchy)
+ n_leaves = hierarchy.n_leaves
+ if not np.issubdtype(hierarchy.component_roots.dtype, np.integer):
+ raise TypeError("component roots must contain integers")
+
+ n_nodes = 2 * n_leaves - 1
+ owners = np.full(n_nodes, -1, dtype=np.int64)
+ stack = np.empty(n_nodes, dtype=np.int64)
+ for component, raw_root in enumerate(hierarchy.component_roots):
+ root = int(raw_root)
+ if root < 0 or root >= n_nodes:
+ raise ValueError("component root lies outside the hierarchy")
+ stack_size = 1
+ stack[0] = root
+ while stack_size:
+ stack_size -= 1
+ node = int(stack[stack_size])
+ if owners[node] >= 0:
+ raise ValueError("finite hierarchy components overlap")
+ if node >= n_leaves and hierarchy.synthetic_joins[node - n_leaves]:
+ raise ValueError("component roots cannot contain synthetic joins")
+ owners[node] = component
+ if node < n_leaves:
+ continue
+ left, right = hierarchy.children[node - n_leaves]
+ stack[stack_size] = int(left)
+ stack[stack_size + 1] = int(right)
+ stack_size += 2
+
+ finite_nodes = np.ones(n_nodes, dtype=bool)
+ finite_nodes[n_leaves:] = ~hierarchy.synthetic_joins
+ if np.any(owners[finite_nodes] < 0) or np.any(owners[~finite_nodes] >= 0):
+ raise ValueError("component roots do not cover exactly the finite hierarchy")
+ owner_dtype = (
+ np.int32
+ if hierarchy.component_roots.size <= np.iinfo(np.int32).max
+ else np.int64
+ )
+ return owners[:n_leaves].astype(owner_dtype, copy=True)
+
+
+@njit(cache=True, nogil=True, inline="always")
+def _union_find_root(parents: np.ndarray, node: int) -> int:
+ root = node
+ while parents[root] != root:
+ root = parents[root]
+ while parents[node] != node:
+ parent = parents[node]
+ parents[node] = root
+ node = parent
+ return root
+
+
+@njit(cache=True, nogil=True, inline="always")
+def _union_sets(
+ parents: np.ndarray,
+ ranks: np.ndarray,
+ first: int,
+ second: int,
+) -> int:
+ first_root = int(_union_find_root(parents, first))
+ second_root = int(_union_find_root(parents, second))
+ if first_root == second_root:
+ return first_root
+ if ranks[first_root] < ranks[second_root]:
+ parents[first_root] = second_root
+ return second_root
+ parents[second_root] = first_root
+ if ranks[first_root] == ranks[second_root]:
+ ranks[first_root] += 1
+ return first_root
+
+
+@njit(cache=True, nogil=True)
+def _offline_lca_edge_counts(
+ indptr: np.ndarray,
+ indices: np.ndarray,
+ children: np.ndarray,
+ component_roots: np.ndarray,
+ leaf_components: np.ndarray,
+) -> tuple[np.ndarray, np.ndarray, int]:
+ n_leaves = indptr.size - 1
+ n_nodes = 2 * n_leaves - 1
+ lca_edge_counts = np.zeros(n_leaves - 1, dtype=np.int64)
+ component_edge_counts = np.zeros(component_roots.size, dtype=np.int64)
+ parents = np.full(n_nodes, -1, dtype=np.int64)
+ ranks = np.zeros(n_nodes, dtype=np.uint8)
+ ancestors = np.full(n_nodes, -1, dtype=np.int64)
+ black = np.zeros(n_nodes, dtype=np.bool_)
+ stack_nodes = np.empty(n_nodes, dtype=np.int64)
+ stack_states = np.empty(n_nodes, dtype=np.uint8)
+
+ for component in range(component_roots.size):
+ stack_size = 1
+ stack_nodes[0] = component_roots[component]
+ stack_states[0] = 0
+ while stack_size:
+ stack_size -= 1
+ node = int(stack_nodes[stack_size])
+ state = int(stack_states[stack_size])
+ if state == 0:
+ parents[node] = node
+ ancestors[node] = node
+ if node < n_leaves:
+ black[node] = True
+ for offset in range(indptr[node], indptr[node + 1]):
+ neighbor = int(indices[offset])
+ if not black[neighbor]:
+ continue
+ if leaf_components[neighbor] != component:
+ return lca_edge_counts, component_edge_counts, 1
+ root = _union_find_root(parents, neighbor)
+ lca = int(ancestors[root])
+ if lca < n_leaves:
+ return lca_edge_counts, component_edge_counts, 2
+ lca_edge_counts[lca - n_leaves] += 1
+ component_edge_counts[component] += 1
+ continue
+
+ left = int(children[node - n_leaves, 0])
+ stack_nodes[stack_size] = node
+ stack_states[stack_size] = 1
+ stack_nodes[stack_size + 1] = left
+ stack_states[stack_size + 1] = 0
+ stack_size += 2
+ elif state == 1:
+ left = int(children[node - n_leaves, 0])
+ root = _union_sets(parents, ranks, node, left)
+ ancestors[root] = node
+ right = int(children[node - n_leaves, 1])
+ stack_nodes[stack_size] = node
+ stack_states[stack_size] = 2
+ stack_nodes[stack_size + 1] = right
+ stack_states[stack_size + 1] = 0
+ stack_size += 2
+ else:
+ right = int(children[node - n_leaves, 1])
+ root = _union_sets(parents, ranks, node, right)
+ ancestors[root] = node
+ black[node] = True
+
+ return lca_edge_counts, component_edge_counts, 0
+
+
+def collect_topology_statistics(
+ canonical_graph: csr_matrix,
+ hierarchy: ParisHierarchy,
+) -> TopologyStatistics:
+ """Collect deterministic unweighted topology statistics."""
+ _validate_canonical_csr(canonical_graph, hierarchy.n_leaves)
+ leaf_components = _leaf_component_ids(hierarchy)
+ leaf_degrees = np.diff(canonical_graph.indptr.astype(np.int64, copy=False)).astype(
+ np.int64, copy=False
+ )
+ (
+ lca_edge_counts,
+ component_edge_counts,
+ error_code,
+ ) = _offline_lca_edge_counts(
+ canonical_graph.indptr,
+ canonical_graph.indices,
+ hierarchy.children,
+ hierarchy.component_roots,
+ leaf_components,
+ )
+ if error_code == 1:
+ raise ValueError("canonical graph contains an edge across hierarchy components")
+ if error_code == 2:
+ raise RuntimeError("offline LCA resolved an edge to a leaf")
+ expected_edges = canonical_graph.nnz // 2
+ if int(lca_edge_counts.sum()) != expected_edges:
+ raise RuntimeError("offline LCA did not account for every topology edge")
+ if int(leaf_degrees.sum()) != 2 * expected_edges:
+ raise RuntimeError("leaf degrees do not account for every topology edge")
+ return TopologyStatistics(
+ leaf_degrees=np.ascontiguousarray(leaf_degrees),
+ lca_edge_counts=lca_edge_counts,
+ component_edge_counts=component_edge_counts,
+ )
+
+
+def _event_components(
+ hierarchy: ParisHierarchy,
+ forest: PlateauForest,
+) -> np.ndarray:
+ if forest.n_leaves != hierarchy.n_leaves:
+ raise ValueError("plateau forest and hierarchy have different leaf counts")
+ if forest.component_roots.size != hierarchy.component_roots.size:
+ raise ValueError("plateau forest has the wrong number of components")
+
+ n_events = forest.representatives.size
+ n_leaves = hierarchy.n_leaves
+ incoming = np.zeros(n_events, dtype=np.int64)
+ for event in range(n_events):
+ representative = int(forest.representatives[event])
+ if (
+ representative < n_leaves
+ or representative >= 2 * n_leaves - 1
+ or hierarchy.synthetic_joins[representative - n_leaves]
+ ):
+ raise ValueError("plateau representative must be a finite internal node")
+ if int(forest.sizes[event]) != int(hierarchy.sizes[representative - n_leaves]):
+ raise ValueError("plateau event size does not match its representative")
+ for offset in range(
+ int(forest.child_offsets[event]),
+ int(forest.child_offsets[event + 1]),
+ ):
+ child_ref = int(forest.child_refs[offset])
+ if child_ref < 0:
+ leaf = -child_ref - 1
+ if leaf < 0 or leaf >= n_leaves:
+ raise ValueError("plateau child leaf lies outside the hierarchy")
+ continue
+ if child_ref >= event:
+ raise ValueError("plateau child events must precede their parent")
+ incoming[child_ref] += 1
+ if int(forest.parent_events[child_ref]) != event:
+ raise ValueError("plateau parent and child references disagree")
+
+ for event, parent in enumerate(forest.parent_events):
+ if int(parent) < 0:
+ if incoming[event] != 0:
+ raise ValueError("plateau root has an incoming event reference")
+ elif incoming[event] != 1:
+ raise ValueError("non-root plateau event must have one parent")
+
+ component_ids = np.full(n_events, -1, dtype=np.int32)
+ stack = np.empty(max(1, n_events), dtype=np.int64)
+ for component, (forest_root, hierarchy_root) in enumerate(
+ zip(
+ forest.component_roots,
+ hierarchy.component_roots,
+ strict=True,
+ )
+ ):
+ root_ref = int(forest_root)
+ expected_raw_root = int(hierarchy_root)
+ actual_raw_root = (
+ -root_ref - 1 if root_ref < 0 else int(forest.representatives[root_ref])
+ )
+ if actual_raw_root != expected_raw_root:
+ raise ValueError("plateau and hierarchy component roots disagree")
+ if root_ref < 0:
+ continue
+ if root_ref >= n_events or int(forest.parent_events[root_ref]) >= 0:
+ raise ValueError("plateau component root is not a root event")
+ stack_size = 1
+ stack[0] = root_ref
+ while stack_size:
+ stack_size -= 1
+ event = int(stack[stack_size])
+ if component_ids[event] >= 0:
+ raise ValueError("plateau components overlap")
+ component_ids[event] = component
+ for offset in range(
+ int(forest.child_offsets[event]),
+ int(forest.child_offsets[event + 1]),
+ ):
+ child_ref = int(forest.child_refs[offset])
+ if child_ref >= 0:
+ stack[stack_size] = child_ref
+ stack_size += 1
+ if np.any(component_ids < 0):
+ raise ValueError("plateau component roots do not cover every event")
+ return component_ids
+
+
+def aggregate_plateau_statistics(
+ hierarchy: ParisHierarchy,
+ forest: PlateauForest,
+ topology: TopologyStatistics,
+) -> PlateauModularityStatistics:
+ """Aggregate edge counts and degree volumes onto plateau events."""
+ leaf_components = _leaf_component_ids(hierarchy)
+ component_ids = _event_components(hierarchy, forest)
+ n_leaves = hierarchy.n_leaves
+ n_nodes = 2 * n_leaves - 1
+ if topology.leaf_degrees.shape != (n_leaves,):
+ raise ValueError("leaf degree statistics have the wrong length")
+ if topology.lca_edge_counts.shape != (n_leaves - 1,):
+ raise ValueError("LCA edge statistics have the wrong length")
+ if topology.component_edge_counts.shape != hierarchy.component_roots.shape:
+ raise ValueError("component edge statistics have the wrong length")
+ if int(topology.lca_edge_counts.sum()) != int(topology.component_edge_counts.sum()):
+ raise ValueError("raw edge statistics disagree on the total edge count")
+ if np.any(
+ topology.lca_edge_counts[np.asarray(hierarchy.synthetic_joins, dtype=bool)] != 0
+ ):
+ raise ValueError("synthetic joins cannot contain topology edges")
+
+ raw_sizes = np.ones(n_nodes, dtype=np.int64)
+ raw_volumes = np.zeros(n_nodes, dtype=np.int64)
+ raw_volumes[:n_leaves] = topology.leaf_degrees
+ raw_internal_edges = np.zeros(n_nodes, dtype=np.int64)
+ for merge_index in range(n_leaves - 1):
+ node = n_leaves + merge_index
+ left = int(hierarchy.children[merge_index, 0])
+ right = int(hierarchy.children[merge_index, 1])
+ raw_sizes[node] = raw_sizes[left] + raw_sizes[right]
+ raw_volumes[node] = raw_volumes[left] + raw_volumes[right]
+ raw_internal_edges[node] = (
+ raw_internal_edges[left]
+ + raw_internal_edges[right]
+ + topology.lca_edge_counts[merge_index]
+ )
+ if int(raw_sizes[node]) != int(hierarchy.sizes[merge_index]):
+ raise ValueError("hierarchy sizes changed during event aggregation")
+
+ for component, root in enumerate(hierarchy.component_roots):
+ raw_root = int(root)
+ edge_count = int(topology.component_edge_counts[component])
+ if int(raw_internal_edges[raw_root]) != edge_count:
+ raise ValueError("component root does not contain its declared edges")
+ if int(raw_volumes[raw_root]) != 2 * edge_count:
+ raise ValueError("component volume is not twice its edge count")
+ if raw_root < n_leaves and int(leaf_components[raw_root]) != component:
+ raise ValueError("isolated component ownership is inconsistent")
+
+ n_events = forest.representatives.size
+ cross_edges = np.empty(n_events, dtype=np.int64)
+ volumes = np.empty(n_events, dtype=np.int64)
+
+ for event, raw_representative in enumerate(forest.representatives):
+ raw_node = int(raw_representative)
+ size = int(raw_sizes[raw_node])
+ volume = int(raw_volumes[raw_node])
+ internal_edge_count = int(raw_internal_edges[raw_node])
+
+ child_size_sum = 0
+ child_volume_sum = 0
+ child_internal_edge_sum = 0
+ component = int(component_ids[event])
+
+ for offset in range(
+ int(forest.child_offsets[event]),
+ int(forest.child_offsets[event + 1]),
+ ):
+ child_ref = int(forest.child_refs[offset])
+ child_node = (
+ -child_ref - 1
+ if child_ref < 0
+ else int(forest.representatives[child_ref])
+ )
+ child_component = (
+ int(leaf_components[child_node])
+ if child_node < n_leaves
+ else int(component_ids[child_ref])
+ )
+ if child_component != component:
+ raise ValueError("a plateau event crosses hierarchy components")
+ child_size = int(raw_sizes[child_node])
+ child_volume = int(raw_volumes[child_node])
+ child_internal_edge = int(raw_internal_edges[child_node])
+
+ child_size_sum += child_size
+ child_volume_sum += child_volume
+ child_internal_edge_sum += child_internal_edge
+
+ if child_size_sum != size or child_volume_sum != volume:
+ raise ValueError("plateau children do not partition their event")
+
+ cross_edges[event] = internal_edge_count - child_internal_edge_sum
+ volumes[event] = volume
+
+ return PlateauModularityStatistics(
+ cross_edges=cross_edges,
+ volumes=volumes,
+ )
+
+
+def modularity_split_gains(
+ hierarchy: ParisHierarchy,
+ forest: PlateauForest,
+ graph: csr_matrix,
+) -> np.ndarray:
+ """Return the configuration-null modularity gain of splitting each event.
+
+ The gain for event ``p`` is the change in Newman-Girvan modularity when the
+ single block ``p`` is replaced by its immediate plateau children, using
+ unweighted graph topology. A positive value means the split beats the
+ degree-preserving null. A non-positive value means the split only fragments
+ structure the null already explains. Leaf children contribute their degree
+ as volume and no internal edges. The graph is canonicalized to the additive
+ undirected form Paris fits.
+ """
+ topology = collect_topology_statistics(
+ canonicalize_paris_graph(graph),
+ hierarchy,
+ )
+ events = aggregate_plateau_statistics(hierarchy, forest, topology)
+ leaf_degrees = topology.leaf_degrees.astype(np.float64, copy=False)
+ two_m = float(leaf_degrees.sum())
+ n_events = forest.representatives.size
+ gains = np.zeros(n_events, dtype=np.float64)
+ if two_m <= 0:
+ return gains
+
+ volumes = events.volumes.astype(np.float64, copy=False)
+ cross_edges = events.cross_edges.astype(np.float64, copy=False)
+ for event in range(n_events):
+ parent_volume = float(volumes[event])
+ child_volume_square_sum = 0.0
+ for offset in range(
+ int(forest.child_offsets[event]),
+ int(forest.child_offsets[event + 1]),
+ ):
+ child_ref = int(forest.child_refs[offset])
+ if child_ref < 0:
+ child_volume = float(leaf_degrees[-child_ref - 1])
+ else:
+ child_volume = float(volumes[child_ref])
+ child_volume_square_sum += child_volume * child_volume
+ attraction = 2.0 * float(cross_edges[event]) / two_m
+ repulsion = (parent_volume * parent_volume - child_volume_square_sum) / (
+ two_m * two_m
+ )
+ gains[event] = repulsion - attraction
+ return gains
diff --git a/scarf/clustering/balanced_cut.py b/scarf/clustering/balanced_cut.py
new file mode 100644
index 00000000..fdeb1c3f
--- /dev/null
+++ b/scarf/clustering/balanced_cut.py
@@ -0,0 +1,151 @@
+import networkx as nx
+import numpy as np
+
+from ..utils.logging import logger
+from ..utils.progress import tqdmbar
+from .cluster_tree import make_digraph
+
+
+class BalancedCut:
+ """Identify balanced cluster splits from a hierarchy."""
+
+ graph: nx.DiGraph
+ branchpoints: dict[int, list[int]]
+
+ def __init__(
+ self,
+ dendrogram: np.ndarray,
+ max_size: int,
+ min_size: int,
+ max_distance_fc: float,
+ ):
+ self.nCells = dendrogram.shape[0] + 1
+ self.graph = make_digraph(dendrogram)
+ self.maxSize = max_size
+ self.minSize = min_size
+ self.maxDistFc = max_distance_fc
+ self.branchpoints = self._get_branchpoints()
+
+ def _successors(self, start: int, min_leaves: int) -> list[int]:
+ """Get tree downstream of a node."""
+ queue = [start]
+ descendants: list[int] = []
+ while len(queue) > 0:
+ node = queue.pop(0)
+ if self.graph.nodes[node]["nleaves"] > min_leaves:
+ descendants.append(node)
+ queue.extend(list(self.graph.successors(node)))
+ return descendants[1:]
+
+ def _get_mean_dist(self, start_node: int) -> float:
+ """Get mean distances in downstream tree of a node."""
+ nodes = self._successors(start_node, -1)
+ return float(
+ np.array([self.graph.nodes[node]["dist"] for node in nodes]).mean()
+ )
+
+ def _are_subtrees_mergeable(self, s1: int, s2: int) -> bool:
+ n1, n2 = self.graph.nodes[s1]["nleaves"], self.graph.nodes[s2]["nleaves"]
+ if n1 > self.minSize and n2 > self.minSize:
+ d1, d2 = self.graph.nodes[s1]["dist"], self.graph.nodes[s2]["dist"]
+ if d1 / d2 > self.maxDistFc or d2 / d1 > self.maxDistFc:
+ logger.trace(f"Will not merge {s1} and {s2} because of high distance")
+ return False
+ md1, md2 = self._get_mean_dist(s1), self._get_mean_dist(s2)
+ if md1 / md2 > self.maxDistFc or md2 / md1 > self.maxDistFc:
+ logger.trace(
+ f"Will not merge {s1} and {s2} because of high distance of successors"
+ )
+ return False
+ return True
+
+ def _get_branchpoints(self) -> dict[int, list[int]]:
+ """Aggregate leaves bottom up until target size is reached."""
+ n_leaves = int((self.graph.number_of_nodes() + 1) / 2)
+ leaves: dict[int, None] = {node: None for node in range(n_leaves)}
+ branchpoints: dict[int, list[int]] = {}
+ progress = tqdmbar(total=len(leaves), desc="Identifying nodes to split")
+ try:
+ while len(leaves) > 0:
+ leaf, _ = leaves.popitem()
+ progress.update(1)
+ logger.trace(f"FRESH STEP: Leaf {leaf} plucked as base leaf")
+ current = leaf
+ while True:
+ parent = next(self.graph.predecessors(current), None)
+ if parent is None:
+ break
+ if parent in branchpoints:
+ logger.trace(
+ f"Will not climb to {parent} as already a branchpoint"
+ )
+ break
+ if self.graph.nodes[parent]["nleaves"] > self.maxSize:
+ logger.trace(
+ f"Will not climb to {parent} because too many leaves exist"
+ )
+ break
+ successor1, successor2 = list(self.graph.successors(parent))
+ if not self._are_subtrees_mergeable(successor1, successor2):
+ break
+ current = parent
+ logger.trace(f"Aggregating from branch {current} for leaf {leaf}")
+ branchpoints[current] = [leaf]
+ stack = [current]
+ while len(stack) > 0:
+ node = stack.pop()
+ if node in leaves:
+ branchpoints[current].append(node)
+ leaves.pop(node)
+ logger.trace(f"Leaf {node} plucked in aggregation step")
+ progress.update(1)
+ elif node in branchpoints and node != current:
+ logger.trace(
+ f"Skipping branch {node} because its already taken"
+ )
+ elif (
+ self.graph.nodes[node]["nleaves"] >= self.maxSize
+ and node != current
+ ):
+ logger.trace(
+ f"Skipping branch {node} to prevent greedy behaviour"
+ )
+ else:
+ stack.extend(list(self.graph.successors(node)))
+ finally:
+ progress.close()
+ return branchpoints
+
+ def _valid_names_in_branchpoints(self) -> None:
+ leaves: list[int] = []
+ for branchpoint in self.branchpoints:
+ leaves.extend(self.branchpoints[branchpoint])
+ n_leaves = len(leaves)
+ if n_leaves != self.nCells:
+ raise ValueError(
+ "ERROR: Not all leaves present in branchpoints. This bug must be reported"
+ )
+ minimum_leaf = min(leaves)
+ if minimum_leaf != 0:
+ raise ValueError(
+ f"ERROR: minimum leaf label is {minimum_leaf} rather than 0"
+ )
+ maximum_leaf = max(leaves)
+ if n_leaves != maximum_leaf + 1:
+ raise ValueError(
+ f"ERROR: maximum leaf label is {maximum_leaf} "
+ f"while total estimated leaves are {n_leaves}"
+ )
+
+ def get_clusters(self) -> np.ndarray:
+ """Make cluster labels from the identified branchpoints."""
+ self._valid_names_in_branchpoints()
+ clusters = np.zeros(self.nCells).astype(int)
+ for cluster, branchpoint in enumerate(self.branchpoints, start=1):
+ clusters[self.branchpoints[branchpoint]] = cluster
+ if (clusters == 0).sum() > 0:
+ logger.warning(
+ f"{(clusters == 0).sum()} samples were not assigned a cluster"
+ )
+ clusters[clusters == 0] = -1
+ return clusters
diff --git a/scarf/clustering/cluster_tree.py b/scarf/clustering/cluster_tree.py
new file mode 100644
index 00000000..2c9e364d
--- /dev/null
+++ b/scarf/clustering/cluster_tree.py
@@ -0,0 +1,132 @@
+from collections.abc import Iterator
+
+import networkx as nx
+import numpy as np
+import pandas as pd
+
+from ..utils.logging import logger
+from ..utils.progress import iter_progress
+
+
+def make_digraph(
+ d: np.ndarray,
+ clust_info: np.ndarray | None = None,
+) -> nx.DiGraph:
+ """Convert a scipy linkage matrix into a directed tree graph."""
+ graph = nx.DiGraph()
+ node = d.shape[0] + 1
+ if clust_info is not None:
+ if len(clust_info) != d.shape[0] + 1:
+ raise ValueError(
+ "ERROR: cluster information doesn't match number of leaves in dendrogram"
+ )
+ else:
+ clust_info = np.ones(d.shape[0] + 1) * -1
+ for row_values in iter_progress(
+ d,
+ desc="Constructing graph from dendrogram",
+ total=d.shape[0],
+ ):
+ distance = row_values[2]
+ row = row_values.astype(int)
+ graph.add_node(node, nleaves=row[3], dist=distance)
+ if row[0] <= d.shape[0]:
+ graph.add_node(
+ row[0],
+ nleaves=0,
+ dist=distance,
+ cluster=clust_info[row[0]],
+ )
+ if row[1] <= d.shape[0]:
+ graph.add_node(
+ row[1],
+ nleaves=0,
+ dist=distance,
+ cluster=clust_info[row[1]],
+ )
+ graph.add_edge(node, row[0])
+ graph.add_edge(node, row[1])
+ node += 1
+ if graph.number_of_edges() != d.shape[0] * 2:
+ logger.warning(
+ "Number of edges in directed graph not twice the dendrogram shape"
+ )
+ return graph
+
+
+def CoalesceTree(graph: nx.DiGraph, clusters: np.ndarray) -> nx.DiGraph:
+ """Coalesce a hierarchy graph to each cluster's holding node."""
+
+ def calc_steps_to_top(g: nx.DiGraph, c: np.ndarray) -> pd.Series:
+ steps: dict[int, int] = {}
+ for node in range(len(c)):
+ steps[node] = 0
+ queue = [node]
+ while len(queue) > 0:
+ for predecessor in g.predecessors(queue.pop(0)):
+ steps[node] += 1
+ queue.append(predecessor)
+ return pd.Series(steps).sort_values()
+
+ def iter_predecessors(g: nx.DiGraph, node: int) -> Iterator[int]:
+ queue = [node]
+ while len(queue) > 0:
+ for predecessor in g.predecessors(queue.pop(0)):
+ yield predecessor
+ queue.append(predecessor)
+
+ def aggregate_leaves(g: nx.DiGraph, node: int) -> list[int]:
+ queue = [node]
+ leaves: list[int] = []
+ while len(queue) > 0:
+ for successor in g.successors(queue.pop(0)):
+ if g.nodes[successor]["nleaves"] == 0:
+ leaves.append(successor)
+ else:
+ queue.append(successor)
+ return leaves
+
+ def get_holding_nodes(g: nx.DiGraph, c: np.ndarray) -> dict[int, int]:
+ holding_nodes: dict[int, int] = {}
+ steps = calc_steps_to_top(g, c)
+ for cluster in iter_progress(
+ set(c),
+ desc="Identifying the top node for cluster",
+ ):
+ cluster_nodes = set(np.where(c == cluster)[0])
+ n_cluster_nodes = len(cluster_nodes)
+ start_node = steps.reindex(cluster_nodes).idxmin()
+ if n_cluster_nodes == 1:
+ holding_nodes[start_node] = cluster
+ continue
+ for predecessor in iter_predecessors(g, start_node):
+ if g.nodes[predecessor]["nleaves"] >= n_cluster_nodes:
+ leaves = set(aggregate_leaves(g, predecessor))
+ if cluster_nodes.issubset(leaves):
+ if leaves != cluster_nodes or predecessor in holding_nodes:
+ raise ValueError(
+ "Cluster labels are incompatible with the hierarchy: "
+ f"cluster {cluster!r} is not monophyletic"
+ )
+ holding_nodes[predecessor] = cluster
+ break
+ else:
+ raise ValueError(
+ "Cluster labels are incompatible with the hierarchy: "
+ f"cluster {cluster!r} is not monophyletic"
+ )
+ return holding_nodes
+
+ def aggregate_predecessors(g: nx.DiGraph, node: int) -> list[int]:
+ return list(iter_predecessors(g, node))
+
+ def make_subgraph(g: nx.DiGraph, nodes: dict[int, int]) -> nx.DiGraph:
+ subgraph_nodes: list[int] = list(nodes.keys())
+ for node in nodes:
+ subgraph_nodes.extend(aggregate_predecessors(g, node))
+ subgraph = nx.DiGraph(nx.subgraph(g, list(set(subgraph_nodes))))
+ for node in nodes:
+ subgraph.nodes[node]["partition_id"] = nodes[node]
+ return subgraph
+
+ return make_subgraph(graph, get_holding_nodes(graph, clusters))
diff --git a/scarf/clustering/leiden.py b/scarf/clustering/leiden.py
new file mode 100644
index 00000000..0ea2e15c
--- /dev/null
+++ b/scarf/clustering/leiden.py
@@ -0,0 +1,33 @@
+import numpy as np
+from scipy.sparse import spmatrix
+
+
+def leiden_membership(
+ graph: spmatrix,
+ resolution: float,
+ random_seed: int,
+) -> np.ndarray:
+ """Cluster a sparse graph with the Leiden algorithm."""
+ try:
+ import leidenalg
+ except ImportError:
+ raise ImportError(
+ "ERROR: 'leidenalg' package is not installed. Please find the "
+ "installation instructions here: "
+ "https://github.com/vtraag/leidenalg#installation. Also, consider "
+ "running Paris with the `run_paris_clustering` method"
+ )
+ import igraph
+
+ sources, targets = graph.nonzero()
+ igraph_graph = igraph.Graph()
+ igraph_graph.add_vertices(graph.shape[0])
+ igraph_graph.add_edges(list(zip(sources, targets)))
+ igraph_graph.es["weight"] = graph[sources, targets].A1
+ partition = leidenalg.find_partition(
+ igraph_graph,
+ leidenalg.RBConfigurationVertexPartition,
+ resolution_parameter=resolution,
+ seed=random_seed,
+ )
+ return np.array(partition.membership) + 1
diff --git a/scarf/clustering/paris.py b/scarf/clustering/paris.py
new file mode 100644
index 00000000..bffc5383
--- /dev/null
+++ b/scarf/clustering/paris.py
@@ -0,0 +1,160 @@
+import numpy as np
+from scipy.sparse import spmatrix
+
+from ._paris_core import (
+ ParisHierarchy,
+ fit_paris_hierarchy as _fit_paris_hierarchy,
+)
+from .balanced_cut import BalancedCut
+
+
+def fit_paris_hierarchy(
+ graph: spmatrix,
+ *,
+ n_threads: int | None = None,
+) -> ParisHierarchy:
+ """Fit the canonical component-aware Paris hierarchy."""
+ return _fit_paris_hierarchy(graph, n_threads=n_threads)
+
+
+def hierarchy_to_dendrogram(
+ hierarchy: ParisHierarchy,
+ *,
+ compatibility: bool = False,
+) -> np.ndarray:
+ """Convert a typed Paris hierarchy to a SciPy linkage matrix."""
+ dendrogram = np.empty((hierarchy.n_leaves - 1, 4), dtype=np.float64)
+ dendrogram[:, :2] = hierarchy.children
+ dendrogram[:, 2] = hierarchy.heights
+ dendrogram[:, 3] = hierarchy.sizes
+ if compatibility:
+ dendrogram[hierarchy.synthetic_joins, 2] = 0.0
+ return dendrogram
+
+
+def paris_dendrogram(
+ graph: spmatrix,
+ *,
+ n_threads: int | None = None,
+) -> np.ndarray:
+ """Fit Paris and map synthetic component joins to zero."""
+ hierarchy = fit_paris_hierarchy(graph, n_threads=n_threads)
+ return hierarchy_to_dendrogram(hierarchy, compatibility=True)
+
+
+def _validate_linkage_children(dendrogram: np.ndarray) -> np.ndarray:
+ raw_children = np.asarray(dendrogram[:, :2])
+ if np.iscomplexobj(raw_children):
+ raise ValueError("dendrogram child IDs must be finite integers")
+ try:
+ child_values = np.asarray(raw_children, dtype=np.float64)
+ except (TypeError, ValueError, OverflowError):
+ raise ValueError("dendrogram child IDs must be finite integers") from None
+ if not np.all(np.isfinite(child_values)) or not np.all(
+ child_values == np.floor(child_values)
+ ):
+ raise ValueError("dendrogram child IDs must be finite integers")
+ if np.any(child_values < 0):
+ raise ValueError("dendrogram child IDs must be non-negative")
+
+ n_leaves = dendrogram.shape[0] + 1
+ for merge_index, (left, right) in enumerate(child_values):
+ node = n_leaves + merge_index
+ if left == right:
+ raise ValueError(
+ f"dendrogram row {merge_index + 1} must reference two distinct children"
+ )
+ if left >= node or right >= node:
+ raise ValueError(
+ f"dendrogram row {merge_index + 1} must reference only earlier nodes"
+ )
+
+ children = child_values.astype(np.int64, copy=False)
+ n_nodes = 2 * n_leaves - 1
+ consumption_counts = np.bincount(children.ravel(), minlength=n_nodes)
+ repeated = np.flatnonzero(consumption_counts > 1)
+ if repeated.size > 0:
+ raise ValueError(
+ f"dendrogram child node {int(repeated[0])} is consumed more than once"
+ )
+ missing = np.flatnonzero(consumption_counts[:-1] == 0)
+ if missing.size > 0:
+ raise ValueError(
+ "dendrogram must consume every non-root node exactly once; "
+ f"missing node {int(missing[0])}"
+ )
+ return children
+
+
+def straight_cut(dendrogram: np.ndarray, n_clusters: int) -> np.ndarray:
+ """Cut a Paris dendrogram into a fixed number of clusters."""
+ if dendrogram.ndim != 2 or dendrogram.shape[1] != 4:
+ raise ValueError("dendrogram must have shape (n_leaves - 1, 4)")
+ if isinstance(n_clusters, bool) or not isinstance(
+ n_clusters,
+ (int, np.integer),
+ ):
+ raise TypeError("n_clusters must be an integer")
+
+ n_leaves = dendrogram.shape[0] + 1
+ cluster_count = int(n_clusters)
+ if cluster_count < 1 or cluster_count > n_leaves:
+ raise ValueError("n_clusters must be between 1 and the number of leaves")
+ children = _validate_linkage_children(dendrogram)
+ if cluster_count == 1:
+ return np.ones(n_leaves, dtype=np.int64)
+
+ cut_index = n_leaves - cluster_count
+ heights = np.asarray(dendrogram[:, 2])
+ cut_height = np.partition(heights, cut_index)[cut_index]
+ n_nodes = 2 * n_leaves - 1
+ active = np.zeros(n_nodes, dtype=bool)
+ active[:n_leaves] = True
+ sizes = np.zeros(n_nodes, dtype=np.int64)
+ sizes[:n_leaves] = 1
+
+ for merge_index, (left, right) in enumerate(children):
+ node = n_leaves + merge_index
+ if (
+ heights[merge_index] < cut_height
+ and 0 <= left < node
+ and 0 <= right < node
+ and active[left]
+ and active[right]
+ ):
+ active[left] = False
+ active[right] = False
+ active[node] = True
+ sizes[node] = sizes[left] + sizes[right]
+
+ roots = np.flatnonzero(active)
+ roots = roots[np.argsort(-sizes[roots])]
+ labels_by_node = np.full(n_nodes, -1, dtype=np.int64)
+ labels_by_node[roots] = np.arange(roots.size, dtype=np.int64)
+ for merge_index in range(n_leaves - 2, -1, -1):
+ node = n_leaves + merge_index
+ label = labels_by_node[node]
+ if label >= 0:
+ left, right = children[merge_index]
+ labels_by_node[left] = label
+ labels_by_node[right] = label
+
+ labels = labels_by_node[:n_leaves]
+ if np.any(labels < 0):
+ raise ValueError("dendrogram contains invalid child references")
+ return labels + 1
+
+
+def balanced_cut(
+ dendrogram: np.ndarray,
+ max_size: int,
+ min_size: int,
+ max_distance_fc: float,
+) -> np.ndarray:
+ """Cut a hierarchy using balanced size and distance constraints."""
+ return BalancedCut(
+ dendrogram,
+ max_size,
+ min_size,
+ max_distance_fc,
+ ).get_clusters()
diff --git a/scarf/clustering/paris_multiscale.py b/scarf/clustering/paris_multiscale.py
new file mode 100644
index 00000000..2f5b2753
--- /dev/null
+++ b/scarf/clustering/paris_multiscale.py
@@ -0,0 +1,610 @@
+from dataclasses import dataclass
+from typing import Literal
+
+import numpy as np
+
+from ..storage.refs import ArtifactRef
+from ._paris_core import ParisHierarchy
+
+_INFEASIBLE = np.uint8(0)
+_KEEP_SCORE = np.uint8(1)
+_KEEP_CONSTRAINT = np.uint8(2)
+_SPLIT = np.uint8(3)
+_FORCED_ROOT = np.uint8(4)
+
+
+@dataclass(frozen=True)
+class ParisClusterDiagnostic:
+ """Structural diagnostic for one selected Paris cluster."""
+
+ label: int
+ selected_node: int
+ parent_event: int
+ component: int
+ size: int
+ resolution_lower: float | None
+ resolution_upper: float | None
+ persistence: float | None
+ forced: bool
+ blocking_child_count: int
+ folded_cell_count: int
+ decision_margin: float | None
+
+
+@dataclass(frozen=True)
+class ParisClusteringResult:
+ """Labels and diagnostics returned by a Paris cut.
+
+ ``ref`` and ``label_key`` are set when the cut was written to a store, and
+ stay ``None`` for cuts produced directly from a hierarchy.
+ """
+
+ labels: np.ndarray
+ mode: Literal["auto", "fixed"]
+ n_clusters: int
+ diagnostics: tuple[ParisClusterDiagnostic, ...] = ()
+ min_cluster_size: int | None = None
+ label_key: str | None = None
+ hierarchy_generation_id: str | None = None
+ ref: ArtifactRef | None = None
+
+ def __post_init__(self) -> None:
+ if self.labels.ndim != 1 or self.labels.dtype != np.int32:
+ raise ValueError("labels must be a one-dimensional int32 array")
+ if self.n_clusters < 1:
+ raise ValueError("n_clusters must be positive")
+ if self.mode == "fixed" and self.diagnostics:
+ raise ValueError("fixed cuts cannot contain adaptive diagnostics")
+ self.labels.setflags(write=False)
+
+
+@dataclass(frozen=True)
+class PlateauForest:
+ """Compact equal-height event forest."""
+
+ representatives: np.ndarray
+ heights: np.ndarray
+ sizes: np.ndarray
+ parent_events: np.ndarray
+ child_offsets: np.ndarray
+ child_refs: np.ndarray
+ min_leaves: np.ndarray
+ component_roots: np.ndarray
+ n_leaves: int
+
+ def __post_init__(self) -> None:
+ n_events = self.representatives.size
+ for name, values in (
+ ("heights", self.heights),
+ ("sizes", self.sizes),
+ ("parent_events", self.parent_events),
+ ("min_leaves", self.min_leaves),
+ ):
+ if values.shape != (n_events,):
+ raise ValueError(f"{name} must have one value per event")
+ if self.child_offsets.shape != (n_events + 1,):
+ raise ValueError("child_offsets must have one more item than events")
+ if self.child_offsets[0] != 0 or self.child_offsets[-1] != len(self.child_refs):
+ raise ValueError("child offsets do not span child_refs")
+ for values in (
+ self.representatives,
+ self.heights,
+ self.sizes,
+ self.parent_events,
+ self.child_offsets,
+ self.child_refs,
+ self.min_leaves,
+ self.component_roots,
+ ):
+ values.setflags(write=False)
+
+
+def _validate_hierarchy(
+ hierarchy: ParisHierarchy,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ n_leaves = hierarchy.n_leaves
+ if n_leaves < 2:
+ raise ValueError("Paris hierarchy must contain at least two leaves")
+ if not np.issubdtype(hierarchy.children.dtype, np.integer):
+ raise TypeError("hierarchy child references must be integers")
+ if not np.issubdtype(hierarchy.sizes.dtype, np.integer):
+ raise TypeError("hierarchy subtree sizes must be integers")
+
+ n_nodes = 2 * n_leaves - 1
+ index_dtype = hierarchy.children.dtype
+ parents = np.full(n_nodes, -1, dtype=index_dtype)
+ subtree_sizes = np.ones(n_nodes, dtype=hierarchy.sizes.dtype)
+ min_leaves = np.arange(n_nodes, dtype=index_dtype)
+ heights = hierarchy.heights
+
+ if np.isnan(heights).any() or np.any(heights < 0):
+ raise ValueError("Paris merge distances must be non-negative and not NaN")
+ if np.any(~hierarchy.synthetic_joins & ~np.isfinite(heights)):
+ raise ValueError("Only synthetic component joins may have infinite distance")
+ if np.any(hierarchy.synthetic_joins & ~np.isinf(heights)):
+ raise ValueError("Synthetic component joins must have infinite distance")
+
+ for merge_index in range(n_leaves - 1):
+ node = n_leaves + merge_index
+ left = int(hierarchy.children[merge_index, 0])
+ right = int(hierarchy.children[merge_index, 1])
+ if left == right or left < 0 or right < 0 or left >= node or right >= node:
+ raise ValueError(
+ "Hierarchy children must be distinct topological references"
+ )
+ if parents[left] != -1 or parents[right] != -1:
+ raise ValueError("Hierarchy node has more than one parent")
+ parents[left] = node
+ parents[right] = node
+ expected_size = int(subtree_sizes[left]) + int(subtree_sizes[right])
+ if int(hierarchy.sizes[merge_index]) != expected_size:
+ raise ValueError("Hierarchy subtree size does not match its children")
+ subtree_sizes[node] = expected_size
+ min_leaves[node] = min(int(min_leaves[left]), int(min_leaves[right]))
+
+ parent_height = heights[merge_index]
+ if hierarchy.synthetic_joins[merge_index]:
+ continue
+ for child in (left, right):
+ if child < n_leaves:
+ continue
+ child_index = child - n_leaves
+ if hierarchy.synthetic_joins[child_index]:
+ raise ValueError("A finite merge cannot contain a synthetic join")
+ if heights[child_index] > parent_height:
+ raise ValueError("Paris merge distances must be monotone")
+
+ candidate_roots: list[int] = []
+ for node in range(n_nodes):
+ if node >= n_leaves and hierarchy.synthetic_joins[node - n_leaves]:
+ continue
+ parent = int(parents[node])
+ if parent == -1 or (
+ parent >= n_leaves and hierarchy.synthetic_joins[parent - n_leaves]
+ ):
+ candidate_roots.append(node)
+ expected_roots = np.asarray(candidate_roots, dtype=index_dtype)
+ if not np.array_equal(
+ np.sort(expected_roots),
+ np.sort(hierarchy.component_roots),
+ ):
+ raise ValueError("Hierarchy component roots do not match synthetic joins")
+ return parents, subtree_sizes, min_leaves
+
+
+def collapse_equal_height_plateaus(hierarchy: ParisHierarchy) -> PlateauForest:
+ """Collapse exact equal-height parent-child regions into multiway events."""
+ parents, _subtree_sizes, raw_min_leaves = _validate_hierarchy(hierarchy)
+ n_leaves = hierarchy.n_leaves
+ n_nodes = 2 * n_leaves - 1
+ finite = ~hierarchy.synthetic_joins
+ event_tops = np.zeros(n_leaves - 1, dtype=bool)
+
+ for merge_index in range(n_leaves - 1):
+ if not finite[merge_index]:
+ continue
+ node = n_leaves + merge_index
+ parent = int(parents[node])
+ event_tops[merge_index] = (
+ parent == -1
+ or hierarchy.synthetic_joins[parent - n_leaves]
+ or hierarchy.heights[parent - n_leaves] != hierarchy.heights[merge_index]
+ )
+
+ representatives = (
+ np.flatnonzero(event_tops).astype(hierarchy.children.dtype) + n_leaves
+ )
+ n_events = representatives.size
+ event_by_raw = np.full(n_nodes, -1, dtype=hierarchy.children.dtype)
+ event_by_raw[representatives] = np.arange(
+ n_events,
+ dtype=hierarchy.children.dtype,
+ )
+
+ for merge_index in range(n_leaves - 2, -1, -1):
+ if not finite[merge_index]:
+ continue
+ node = n_leaves + merge_index
+ event = int(event_by_raw[node])
+ if event < 0:
+ raise RuntimeError("Plateau event propagation failed")
+ height = hierarchy.heights[merge_index]
+ for child in hierarchy.children[merge_index]:
+ child_node = int(child)
+ if (
+ child_node >= n_leaves
+ and finite[child_node - n_leaves]
+ and hierarchy.heights[child_node - n_leaves] == height
+ ):
+ event_by_raw[child_node] = event
+
+ child_counts = np.zeros(n_events, dtype=hierarchy.children.dtype)
+ for merge_index in range(n_leaves - 1):
+ if not finite[merge_index]:
+ continue
+ event = int(event_by_raw[n_leaves + merge_index])
+ height = hierarchy.heights[merge_index]
+ for child in hierarchy.children[merge_index]:
+ child_node = int(child)
+ same_plateau = (
+ child_node >= n_leaves
+ and finite[child_node - n_leaves]
+ and hierarchy.heights[child_node - n_leaves] == height
+ )
+ if not same_plateau:
+ child_counts[event] += 1
+
+ child_offsets = np.empty(n_events + 1, dtype=hierarchy.children.dtype)
+ child_offsets[0] = 0
+ np.cumsum(child_counts, out=child_offsets[1:])
+ child_refs = np.empty(int(child_offsets[-1]), dtype=hierarchy.children.dtype)
+ write_offsets = child_offsets[:-1].copy()
+ for merge_index in range(n_leaves - 1):
+ if not finite[merge_index]:
+ continue
+ event = int(event_by_raw[n_leaves + merge_index])
+ height = hierarchy.heights[merge_index]
+ for child in hierarchy.children[merge_index]:
+ child_node = int(child)
+ same_plateau = (
+ child_node >= n_leaves
+ and finite[child_node - n_leaves]
+ and hierarchy.heights[child_node - n_leaves] == height
+ )
+ if same_plateau:
+ continue
+ offset = int(write_offsets[event])
+ if child_node < n_leaves:
+ child_refs[offset] = -child_node - 1
+ else:
+ child_event = int(event_by_raw[child_node])
+ if child_event < 0:
+ raise RuntimeError("Plateau boundary does not reference an event")
+ child_refs[offset] = child_event
+ write_offsets[event] += 1
+
+ parent_events = np.full(n_events, -1, dtype=hierarchy.children.dtype)
+ for event, representative in enumerate(representatives):
+ parent = int(parents[int(representative)])
+ if parent != -1 and not hierarchy.synthetic_joins[parent - n_leaves]:
+ parent_event = int(event_by_raw[parent])
+ if parent_event == event or parent_event < 0:
+ raise RuntimeError("Plateau parent mapping failed")
+ parent_events[event] = parent_event
+
+ component_roots = np.empty(
+ len(hierarchy.component_roots),
+ dtype=hierarchy.children.dtype,
+ )
+ for component, root in enumerate(hierarchy.component_roots):
+ raw_root = int(root)
+ component_roots[component] = (
+ -raw_root - 1 if raw_root < n_leaves else event_by_raw[raw_root]
+ )
+
+ representative_indices = representatives - n_leaves
+ return PlateauForest(
+ representatives=representatives,
+ heights=hierarchy.heights[representative_indices].copy(),
+ sizes=hierarchy.sizes[representative_indices].copy(),
+ parent_events=parent_events,
+ child_offsets=child_offsets,
+ child_refs=child_refs,
+ min_leaves=raw_min_leaves[representatives].copy(),
+ component_roots=component_roots,
+ n_leaves=n_leaves,
+ )
+
+
+def _event_keep_score(forest: PlateauForest, event: int) -> float:
+ parent = int(forest.parent_events[event])
+ if parent < 0:
+ return 0.0
+ creation_distance = float(forest.heights[event])
+ parent_distance = float(forest.heights[parent])
+ if parent_distance <= creation_distance:
+ return 0.0
+ # A logistic measure on log resolution is finite and centered at gamma=1.
+ return int(forest.sizes[event]) * (
+ 1.0 / (1.0 + creation_distance) - 1.0 / (1.0 + parent_distance)
+ )
+
+
+def _score_events(
+ forest: PlateauForest,
+ min_cluster_size: int,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+ n_events = forest.representatives.size
+ best_scores = np.full(n_events, -np.inf, dtype=np.float64)
+ states = np.full(n_events, _INFEASIBLE, dtype=np.uint8)
+ blocking_counts = np.zeros(n_events, dtype=np.int32)
+ folded_cells = np.zeros(n_events, dtype=forest.sizes.dtype)
+
+ for event in range(n_events):
+ split_score = 0.0
+ blocking_count = 0
+ folded_count = 0
+ for offset in range(
+ int(forest.child_offsets[event]),
+ int(forest.child_offsets[event + 1]),
+ ):
+ child_ref = int(forest.child_refs[offset])
+ if child_ref < 0:
+ blocking_count += 1
+ folded_count += 1
+ elif states[child_ref] == _INFEASIBLE:
+ blocking_count += 1
+ folded_count += int(forest.sizes[child_ref])
+ else:
+ split_score += best_scores[child_ref]
+ blocking_counts[event] = blocking_count
+ folded_cells[event] = folded_count
+
+ size = int(forest.sizes[event])
+ is_root = forest.parent_events[event] < 0
+ if size < min_cluster_size:
+ if is_root:
+ states[event] = _FORCED_ROOT
+ best_scores[event] = 0.0
+ continue
+
+ split_feasible = blocking_count == 0
+ keep_score = _event_keep_score(forest, event)
+ if is_root:
+ if split_feasible and split_score > 0:
+ states[event] = _SPLIT
+ best_scores[event] = split_score
+ else:
+ states[event] = _FORCED_ROOT
+ best_scores[event] = 0.0
+ elif not split_feasible:
+ states[event] = _KEEP_CONSTRAINT
+ best_scores[event] = keep_score
+ elif keep_score >= split_score:
+ states[event] = _KEEP_SCORE
+ best_scores[event] = keep_score
+ else:
+ states[event] = _SPLIT
+ best_scores[event] = split_score
+
+ return best_scores, states, blocking_counts, folded_cells
+
+
+def _selected_events(
+ forest: PlateauForest,
+ states: np.ndarray,
+ split_gate: np.ndarray | None = None,
+) -> list[tuple[int, int, bool]]:
+ selected: list[tuple[int, int, bool]] = []
+ stack = np.empty(max(1, forest.representatives.size), dtype=np.int64)
+ for component, root_ref in enumerate(forest.component_roots):
+ root = int(root_ref)
+ if root < 0:
+ selected.append((root, component, True))
+ continue
+ stack_size = 1
+ stack[0] = root
+ while stack_size:
+ stack_size -= 1
+ event = int(stack[stack_size])
+ vetoed = split_gate is not None and float(split_gate[event]) <= 0.0
+ if states[event] != _SPLIT or vetoed:
+ selected.append((event, component, bool(states[event] == _FORCED_ROOT)))
+ continue
+ for offset in range(
+ int(forest.child_offsets[event]),
+ int(forest.child_offsets[event + 1]),
+ ):
+ child_ref = int(forest.child_refs[offset])
+ if child_ref < 0:
+ raise RuntimeError("A feasible event split contains a leaf")
+ stack[stack_size] = child_ref
+ stack_size += 1
+ return selected
+
+
+def _fill_labels(
+ hierarchy: ParisHierarchy,
+ forest: PlateauForest,
+ selected: list[tuple[int, int, bool]],
+) -> tuple[np.ndarray, list[tuple[int, int, bool]]]:
+ selected.sort(
+ key=lambda item: (
+ -item[0] - 1 if item[0] < 0 else int(forest.min_leaves[item[0]])
+ )
+ )
+ selected_nodes = np.asarray(
+ [
+ -selected_ref - 1
+ if selected_ref < 0
+ else int(forest.representatives[selected_ref])
+ for selected_ref, _component, _forced in selected
+ ],
+ dtype=hierarchy.children.dtype,
+ )
+ return labels_from_selected_nodes(hierarchy, selected_nodes), selected
+
+
+def labels_from_selected_nodes(
+ hierarchy: ParisHierarchy,
+ selected_nodes: np.ndarray,
+) -> np.ndarray:
+ """Regenerate labels from disjoint selected hierarchy nodes."""
+ selected = np.asarray(selected_nodes)
+ if selected.ndim != 1 or selected.size == 0:
+ raise ValueError("selected_nodes must be a non-empty one-dimensional array")
+ if not np.issubdtype(selected.dtype, np.integer):
+ raise TypeError("selected_nodes must contain integers")
+ if len(np.unique(selected)) != len(selected):
+ raise ValueError("selected_nodes must not contain duplicates")
+
+ n_leaves = hierarchy.n_leaves
+ labels = np.zeros(n_leaves, dtype=np.int32)
+ stack = np.empty(n_leaves, dtype=hierarchy.children.dtype)
+ for label, selected_node in enumerate(selected, start=1):
+ raw_node = int(selected_node)
+ if raw_node < 0 or raw_node >= 2 * n_leaves - 1:
+ raise ValueError("selected node is outside the hierarchy")
+ if raw_node >= n_leaves and hierarchy.synthetic_joins[raw_node - n_leaves]:
+ raise ValueError("synthetic component joins cannot be selected")
+ stack_size = 1
+ stack[0] = raw_node
+ while stack_size:
+ stack_size -= 1
+ node = int(stack[stack_size])
+ if node < n_leaves:
+ if labels[node] != 0:
+ raise RuntimeError("Adaptive Paris clusters overlap")
+ labels[node] = label
+ continue
+ children = hierarchy.children[node - n_leaves]
+ stack[stack_size] = children[0]
+ stack[stack_size + 1] = children[1]
+ stack_size += 2
+ if np.any(labels == 0):
+ raise RuntimeError("Adaptive Paris cut did not cover every leaf")
+ return labels
+
+
+def _diagnostics(
+ forest: PlateauForest,
+ selected: list[tuple[int, int, bool]],
+ best_scores: np.ndarray,
+ states: np.ndarray,
+ blocking_counts: np.ndarray,
+ folded_cells: np.ndarray,
+) -> tuple[ParisClusterDiagnostic, ...]:
+ diagnostics: list[ParisClusterDiagnostic] = []
+ for label, (selected_ref, component, forced) in enumerate(selected, start=1):
+ if selected_ref < 0:
+ leaf = -selected_ref - 1
+ diagnostics.append(
+ ParisClusterDiagnostic(
+ label=label,
+ selected_node=leaf,
+ parent_event=-1,
+ component=component,
+ size=1,
+ resolution_lower=None,
+ resolution_upper=None,
+ persistence=None,
+ forced=True,
+ blocking_child_count=0,
+ folded_cell_count=0,
+ decision_margin=None,
+ )
+ )
+ continue
+
+ event = selected_ref
+ parent = int(forest.parent_events[event])
+ parent_representative = (
+ -1 if parent < 0 else int(forest.representatives[parent])
+ )
+ size = int(forest.sizes[event])
+ if forced or parent < 0:
+ lower = None
+ upper = None
+ persistence = None
+ margin = None
+ else:
+ lower = float(forest.heights[event])
+ upper = float(forest.heights[parent])
+ persistence = _event_keep_score(forest, event)
+ if states[event] != _KEEP_SCORE:
+ margin = None
+ else:
+ split_score = 0.0
+ for offset in range(
+ int(forest.child_offsets[event]),
+ int(forest.child_offsets[event + 1]),
+ ):
+ child_ref = int(forest.child_refs[offset])
+ if child_ref < 0:
+ raise RuntimeError(
+ "Score-selected event has an infeasible split"
+ )
+ split_score += best_scores[child_ref]
+ margin = (persistence - split_score) / size
+
+ diagnostics.append(
+ ParisClusterDiagnostic(
+ label=label,
+ selected_node=int(forest.representatives[event]),
+ parent_event=parent_representative,
+ component=component,
+ size=size,
+ resolution_lower=lower,
+ resolution_upper=upper,
+ persistence=persistence,
+ forced=forced,
+ blocking_child_count=int(blocking_counts[event]),
+ folded_cell_count=int(folded_cells[event]),
+ decision_margin=margin,
+ )
+ )
+ return tuple(diagnostics)
+
+
+def adaptive_cut(
+ hierarchy: ParisHierarchy,
+ min_cluster_size: int = 2,
+ *,
+ plateau_forest: PlateauForest | None = None,
+ split_gate: np.ndarray | None = None,
+) -> ParisClusteringResult:
+ """Select a deterministic branch-adaptive antichain from a Paris hierarchy.
+
+ When ``split_gate`` is given it must hold one value per plateau event. A
+ split the persistence score would otherwise accept is vetoed wherever the
+ gate value is not strictly positive, coarsening the cut. This layers a
+ configuration-null modularity guard on top of persistence without changing
+ the underlying scoring; passing ``None`` reproduces the unguarded cut.
+ """
+ if isinstance(min_cluster_size, (bool, np.bool_)) or not isinstance(
+ min_cluster_size,
+ (int, np.integer),
+ ):
+ raise TypeError("min_cluster_size must be an integer")
+ if min_cluster_size < 2:
+ raise ValueError("min_cluster_size must be at least 2")
+ min_cluster_size = int(min_cluster_size)
+ forest = (
+ collapse_equal_height_plateaus(hierarchy)
+ if plateau_forest is None
+ else plateau_forest
+ )
+ if forest.n_leaves != hierarchy.n_leaves:
+ raise ValueError("Plateau forest and hierarchy have different leaf counts")
+ if split_gate is not None:
+ if split_gate.shape != forest.representatives.shape:
+ raise ValueError("split_gate must have one value per plateau event")
+ if not (
+ np.issubdtype(split_gate.dtype, np.integer)
+ or np.issubdtype(split_gate.dtype, np.floating)
+ ):
+ raise TypeError("split_gate must contain real numbers")
+ if not np.isfinite(split_gate).all():
+ raise ValueError("split_gate must contain only finite values")
+
+ best_scores, states, blocking_counts, folded_cells = _score_events(
+ forest,
+ min_cluster_size,
+ )
+ selected = _selected_events(forest, states, split_gate)
+ labels, selected = _fill_labels(hierarchy, forest, selected)
+ diagnostics = _diagnostics(
+ forest,
+ selected,
+ best_scores,
+ states,
+ blocking_counts,
+ folded_cells,
+ )
+ return ParisClusteringResult(
+ labels=labels,
+ mode="auto",
+ n_clusters=len(diagnostics),
+ diagnostics=diagnostics,
+ min_cluster_size=min_cluster_size,
+ )
diff --git a/scarf/cytebase.py b/scarf/cytebase.py
new file mode 100644
index 00000000..e4008251
--- /dev/null
+++ b/scarf/cytebase.py
@@ -0,0 +1,375 @@
+"""Browse and download public data from Cytebase."""
+
+import os
+from dataclasses import dataclass
+from pathlib import Path, PurePosixPath, PureWindowsPath
+import shutil
+import tarfile
+import tempfile
+from typing import TYPE_CHECKING
+
+from huggingface_hub import (
+ BucketFile,
+ BucketFolder,
+ download_bucket_files,
+ list_bucket_tree,
+)
+
+if TYPE_CHECKING:
+ import zarr
+
+__all__ = ["Repository", "connect", "list_repositories"]
+
+_BUCKET_ID = "Nygen/cytebase"
+_ZARR_ARCHIVE_SUFFIX = ".zarr.tar.gz"
+
+
+def _safe_name(name: str, *, kind: str) -> str:
+ posix_name = PurePosixPath(name)
+ windows_name = PureWindowsPath(name)
+ if (
+ not name
+ or name in {".", ".."}
+ or posix_name.name != name
+ or windows_name.name != name
+ ):
+ raise ValueError(f"Invalid {kind}: {name!r}")
+ return name
+
+
+def _safe_relative_path(path: str, *, kind: str) -> PurePosixPath:
+ if not path or path.startswith("/") or "\\" in path:
+ raise ValueError(f"Invalid {kind}: {path!r}")
+ parts = path.split("/")
+ if any(part in {"", ".", ".."} for part in parts):
+ raise ValueError(f"Invalid {kind}: {path!r}")
+ return PurePosixPath(*parts)
+
+
+def _remote_path(repository: str, path: str | None = None) -> str:
+ if path is None:
+ return repository
+ relative_path = _safe_relative_path(path, kind="remote path")
+ return f"{repository}/{relative_path.as_posix()}"
+
+
+def _relative_file_path(repository: str, file: BucketFile) -> PurePosixPath:
+ repository_prefix = f"{repository}/"
+ if not file.path.startswith(repository_prefix):
+ raise ValueError(f"File is outside repository {repository!r}: {file.path!r}")
+ return _safe_relative_path(
+ file.path.removeprefix(repository_prefix),
+ kind="bucket file path",
+ )
+
+
+def _bucket_files(
+ repository: str,
+ path: str | None = None,
+ *,
+ recursive: bool,
+) -> list[BucketFile]:
+ prefix = _remote_path(repository, path)
+ files = []
+ for item in list_bucket_tree(
+ _BUCKET_ID,
+ prefix=prefix,
+ recursive=recursive,
+ token=False,
+ ):
+ if not isinstance(item, BucketFile):
+ continue
+ if item.path != prefix and not item.path.startswith(f"{prefix}/"):
+ continue
+ _relative_file_path(repository, item)
+ files.append(item)
+ return sorted(files, key=lambda file: file.path)
+
+
+def list_repositories() -> list[str]:
+ """Return the top-level repositories in Cytebase."""
+ repositories = []
+ for item in list_bucket_tree(_BUCKET_ID, recursive=False, token=False):
+ if isinstance(item, BucketFolder) and "/" not in item.path:
+ repositories.append(_safe_name(item.path, kind="repository name"))
+ return sorted(repositories)
+
+
+def connect(repository: str) -> "Repository":
+ """Connect to a public Cytebase repository."""
+ repository = _safe_name(repository, kind="repository name")
+ available = list_repositories()
+ if repository not in available:
+ choices = "\n".join(available)
+ raise KeyError(
+ f"{repository!r} is not a Cytebase repository. "
+ f"Available repositories:\n{choices}"
+ )
+ return Repository(repository)
+
+
+@dataclass(frozen=True, slots=True)
+class Repository:
+ """Read files from one top-level Cytebase repository."""
+
+ name: str
+
+ def __post_init__(self) -> None:
+ _safe_name(self.name, kind="repository name")
+
+ def list_datasets(self) -> list[str]:
+ """Return immediate dataset directories in this repository."""
+ prefix = f"{self.name}/"
+ datasets = []
+ for item in list_bucket_tree(
+ _BUCKET_ID,
+ prefix=self.name,
+ recursive=False,
+ token=False,
+ ):
+ if not isinstance(item, BucketFolder):
+ continue
+ if not item.path.startswith(prefix):
+ continue
+ relative_path = item.path.removeprefix(prefix)
+ if "/" not in relative_path:
+ datasets.append(_safe_name(relative_path, kind="dataset name"))
+ return sorted(datasets)
+
+ def list_files(
+ self,
+ path: str | None = None,
+ *,
+ recursive: bool = True,
+ ) -> list[str]:
+ """Return file paths relative to this repository."""
+ return [
+ _relative_file_path(self.name, file).as_posix()
+ for file in _bucket_files(self.name, path, recursive=recursive)
+ ]
+
+ def download(
+ self,
+ path: str,
+ destination: str | Path = ".",
+ ) -> list[Path]:
+ """Download a file or directory while preserving its repository path."""
+ _safe_relative_path(path, kind="download path")
+ files = _bucket_files(self.name, path, recursive=True)
+ if not files:
+ raise FileNotFoundError(f"No Cytebase files found at {self.name}/{path}")
+ return _download_files(self.name, files, Path(destination))
+
+ def download_dataset(
+ self,
+ name: str,
+ destination: str | Path = ".",
+ *,
+ zarr: bool = False,
+ ) -> Path:
+ """Download one dataset, optionally selecting its Zarr archive."""
+ name = _safe_name(name, kind="dataset name")
+ files = _bucket_files(self.name, name, recursive=True)
+ if not files:
+ choices = "\n".join(self.list_datasets())
+ raise KeyError(
+ f"{name!r} is not in repository {self.name!r}. "
+ f"Available datasets:\n{choices}"
+ )
+
+ if zarr:
+ selected = [
+ file for file in files if file.path.endswith(_ZARR_ARCHIVE_SUFFIX)
+ ]
+ if not selected:
+ raise FileNotFoundError(f"No Zarr archive is available for {name!r}")
+ else:
+ selected = [
+ file for file in files if not file.path.endswith(_ZARR_ARCHIVE_SUFFIX)
+ ]
+
+ destination = Path(destination).absolute()
+ _download_files(self.name, selected, destination)
+ return destination / name
+
+ def open_zarr(self, path: str) -> "zarr.Group":
+ """Open an unpacked Zarr group for anonymous read-only access."""
+ relative_path = _safe_relative_path(path, kind="Zarr path")
+ uri = f"hf://buckets/{_BUCKET_ID}/{self.name}/{relative_path.as_posix()}"
+
+ import zarr
+ from zarr.storage import FsspecStore
+
+ store = FsspecStore.from_url(
+ uri,
+ storage_options={"token": False},
+ read_only=True,
+ )
+ return zarr.open_group(store=store, mode="r")
+
+
+def _download_files(
+ repository: str,
+ files: list[BucketFile],
+ destination: Path,
+) -> list[Path]:
+ destination = destination.absolute()
+ destination.mkdir(parents=True, exist_ok=True)
+ staging_path = Path(
+ tempfile.mkdtemp(
+ dir=destination,
+ prefix=".cytebase-download-",
+ )
+ )
+ downloads = []
+
+ try:
+ for file in files:
+ relative_path = _relative_file_path(repository, file)
+ staged_file = staging_path.joinpath(*relative_path.parts)
+ staged_file.parent.mkdir(parents=True, exist_ok=True)
+ final_file = destination.joinpath(*relative_path.parts)
+ downloads.append((file, staged_file, final_file))
+
+ download_bucket_files(
+ _BUCKET_ID,
+ files=[(file, staged_file) for file, staged_file, _ in downloads],
+ token=False,
+ )
+
+ for file, staged_file, _ in downloads:
+ if not staged_file.is_file():
+ raise FileNotFoundError(f"Cytebase did not download {file.path!r}")
+ if staged_file.stat().st_size != file.size:
+ raise OSError(
+ f"Downloaded {staged_file.stat().st_size} bytes for {file.path!r}, "
+ f"expected {file.size}"
+ )
+
+ for _, staged_file, final_file in downloads:
+ final_file.parent.mkdir(parents=True, exist_ok=True)
+ if final_file.name.endswith(".tar.gz"):
+ _extract_and_replace(staged_file, final_file)
+ else:
+ os.replace(staged_file, final_file)
+ finally:
+ shutil.rmtree(staging_path, ignore_errors=True)
+
+ return [final_file for _, _, final_file in downloads]
+
+
+def _safe_tar_members(
+ archive: tarfile.TarFile,
+ staging_path: Path,
+) -> list[tarfile.TarInfo]:
+ staging_root = staging_path.resolve()
+ members = archive.getmembers()
+
+ for member in members:
+ member_path = PurePosixPath(member.name)
+ if member_path.is_absolute() or ".." in member_path.parts:
+ raise ValueError(f"Unsafe tar archive member: {member.name}")
+
+ target = (staging_root / Path(*member_path.parts)).resolve()
+ if not target.is_relative_to(staging_root):
+ raise ValueError(f"Unsafe tar archive member: {member.name}")
+
+ if member.issym() or member.islnk():
+ link_path = PurePosixPath(member.linkname)
+ if link_path.is_absolute() or ".." in link_path.parts:
+ raise ValueError(f"Unsafe tar archive link: {member.name}")
+
+ link_root = target.parent if member.issym() else staging_root
+ link_target = (link_root / Path(*link_path.parts)).resolve()
+ if not link_target.is_relative_to(staging_root):
+ raise ValueError(f"Unsafe tar archive link: {member.name}")
+ elif not member.isfile() and not member.isdir():
+ raise ValueError(f"Unsupported tar archive member: {member.name}")
+
+ return members
+
+
+def _path_exists(path: Path) -> bool:
+ return path.exists() or path.is_symlink()
+
+
+def _remove_path(path: Path) -> None:
+ if path.is_dir() and not path.is_symlink():
+ shutil.rmtree(path)
+ else:
+ path.unlink(missing_ok=True)
+
+
+def _replace_staged_paths(
+ staged_paths: list[tuple[Path, Path]],
+ backup_path: Path,
+) -> None:
+ replacements: list[tuple[Path, Path | None]] = []
+
+ try:
+ for index, (source, destination) in enumerate(staged_paths):
+ backup = backup_path / str(index)
+ previous = backup if _path_exists(destination) else None
+ if previous is not None:
+ os.replace(destination, previous)
+
+ try:
+ os.replace(source, destination)
+ except BaseException:
+ if previous is not None:
+ os.replace(previous, destination)
+ raise
+ replacements.append((destination, previous))
+ except BaseException:
+ for destination, previous in reversed(replacements):
+ _remove_path(destination)
+ if previous is not None:
+ os.replace(previous, destination)
+ raise
+
+
+def _extract_and_replace(
+ archive_path: Path,
+ destination: Path,
+) -> None:
+ staging_path = Path(
+ tempfile.mkdtemp(
+ dir=destination.parent,
+ prefix=f".{destination.name}.extract-",
+ )
+ )
+ backup_path = Path(
+ tempfile.mkdtemp(
+ dir=destination.parent,
+ prefix=f".{destination.name}.backup-",
+ )
+ )
+
+ try:
+ with tarfile.open(archive_path, "r:gz") as archive:
+ members = _safe_tar_members(archive, staging_path)
+ archive.extractall(staging_path, members=members, filter="data")
+
+ extracted_paths = sorted(staging_path.iterdir(), key=lambda path: path.name)
+ extracted_replacements = [
+ (path, destination.parent / path.name) for path in extracted_paths
+ ]
+ reserved_paths = {destination, archive_path, staging_path, backup_path}
+ if any(
+ extracted_destination in reserved_paths
+ for _, extracted_destination in extracted_replacements
+ ):
+ raise ValueError(
+ "Tar archive member conflicts with an internal download path"
+ )
+
+ _replace_staged_paths(
+ [
+ *extracted_replacements,
+ (archive_path, destination),
+ ],
+ backup_path,
+ )
+ finally:
+ shutil.rmtree(staging_path, ignore_errors=True)
+ shutil.rmtree(backup_path, ignore_errors=True)
diff --git a/scarf/tests/test_plotting.py b/scarf/datastore/_operations/__init__.py
similarity index 100%
rename from scarf/tests/test_plotting.py
rename to scarf/datastore/_operations/__init__.py
diff --git a/scarf/datastore/_operations/clustering.py b/scarf/datastore/_operations/clustering.py
new file mode 100644
index 00000000..4d6ec529
--- /dev/null
+++ b/scarf/datastore/_operations/clustering.py
@@ -0,0 +1,1137 @@
+from dataclasses import asdict, dataclass, replace
+from typing import TYPE_CHECKING, Any, Literal, cast
+
+import numpy as np
+import pandas as pd
+
+from ...graph.paths import StoredAssayGraph
+from ...graph.state import resolve_graph_selection, validate_legacy_graph_selection
+from ...metadata.artifacts import (
+ artifact_values,
+ categorical_display,
+ column_display,
+ continuous_display,
+ link_cell_data_column,
+ plan_cell_data_artifact,
+ write_cell_data_artifact,
+)
+from ...metadata.arguments import LeidenArguments, TopacedoArguments
+from ...storage.artifacts import (
+ ArtifactRef,
+ artifact_path,
+ fingerprint_array,
+ inspect_artifact,
+)
+from ...storage.artifact_writer import (
+ ArrayRequirement,
+ PlannedArtifact,
+ finish_artifact,
+ plan_artifact,
+ reused_artifact_group,
+ start_artifact,
+)
+from ...storage.arrays import create_zarr_dataset
+from ...storage.types import as_zarr_array, as_zarr_group
+from ...utils.logging import logger
+
+if TYPE_CHECKING:
+ from ...clustering.paris_multiscale import ParisClusteringResult
+ from .graph import _GraphOperationsMixin as _ClusteringOperationsBase
+else:
+ _ClusteringOperationsBase = object
+
+
+@dataclass(frozen=True, slots=True)
+class _PreparedLeidenClustering:
+ planned: PlannedArtifact
+ graph_loc: str
+ from_assay: str
+ label_assay: str
+ cell_key: str
+ feat_key: str
+ resolution: float
+ symmetric_graph: bool
+ graph_upper_only: bool
+ random_seed: int
+ label: str
+ n_cells: int
+
+ @property
+ def graph_key(self) -> tuple[str, bool, bool]:
+ return (
+ self.graph_loc,
+ self.symmetric_graph,
+ self.graph_upper_only,
+ )
+
+
+class _ClusteringOperationsMixin(_ClusteringOperationsBase):
+ def _run_paris_from_artifacts(
+ self,
+ *,
+ graph_ref: Any,
+ graph_loc: str,
+ from_assay: str,
+ label_assay: str,
+ cell_key: str,
+ fixed_cluster_count: int | None,
+ effective_min_cluster_size: int | None,
+ label: str,
+ force_recalc: bool,
+ ) -> "ParisClusteringResult":
+ from ...clustering._paris_modularity import modularity_split_gains
+ from ...clustering.paris import (
+ fit_paris_hierarchy,
+ hierarchy_to_dendrogram,
+ straight_cut,
+ )
+ from ...clustering.paris_multiscale import (
+ ParisClusterDiagnostic,
+ ParisClusteringResult,
+ adaptive_cut,
+ collapse_equal_height_plateaus,
+ )
+ from .paris_persistence import (
+ load_hierarchy_group,
+ preflight_hierarchy_artifact_cut,
+ preflight_paris_adaptive_cut,
+ preflight_paris_fit,
+ write_hierarchy_group,
+ )
+
+ artifact_scope = (
+ graph_ref.scope
+ if isinstance(graph_ref, ArtifactRef)
+ else "datastore"
+ if label_assay != from_assay
+ else "assay"
+ )
+ artifact_assay = (
+ graph_ref.assay
+ if isinstance(graph_ref, ArtifactRef)
+ else None
+ if artifact_scope == "datastore"
+ else from_assay
+ )
+ n_cells, _effective_k = self._get_graph_ncells_k(graph_loc)
+ cut_mode: Literal["adaptive", "fixed"] = (
+ "fixed" if fixed_cluster_count is not None else "adaptive"
+ )
+ graph_group = as_zarr_group(self.zw[graph_loc], name=graph_loc)
+ budget = self.resources
+ hierarchy_plan = plan_artifact(
+ self.zw,
+ scope=artifact_scope,
+ assay=artifact_assay,
+ kind="cluster_hierarchy",
+ operation="fit_paris_hierarchy",
+ parameters={},
+ inputs={"connectivity_map": graph_ref},
+ execution_options={"invalidate_cache": force_recalc},
+ invalidate_cache=force_recalc,
+ required_arrays=(
+ ArrayRequirement("children"),
+ ArrayRequirement("heights"),
+ ArrayRequirement("sizes"),
+ ArrayRequirement("component_roots"),
+ ArrayRequirement("synthetic_joins"),
+ ),
+ )
+ hierarchy = plateau_forest = None
+ fitted_graph = None
+ if hierarchy_plan.reused:
+ try:
+ hierarchy_group = reused_artifact_group(
+ self.zw,
+ hierarchy_plan,
+ )
+ preflight_hierarchy_artifact_cut(
+ hierarchy_group,
+ cut_mode,
+ budget,
+ )
+ hierarchy, plateau_forest = load_hierarchy_group(
+ hierarchy_group,
+ hierarchy_plan.ref.artifact_id,
+ )
+ except (KeyError, TypeError, ValueError):
+ hierarchy_plan = plan_artifact(
+ self.zw,
+ scope=artifact_scope,
+ assay=artifact_assay,
+ kind="cluster_hierarchy",
+ operation="fit_paris_hierarchy",
+ parameters={},
+ inputs={"connectivity_map": graph_ref},
+ execution_options={"invalidate_cache": force_recalc},
+ invalidate_cache=True,
+ )
+ if hierarchy is None or plateau_forest is None:
+ estimated_peak_bytes = preflight_paris_fit(
+ graph_group,
+ n_cells,
+ budget,
+ )
+ fitted_graph = self.load_graph(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ symmetric=False,
+ upper_only=False,
+ graph_loc=graph_loc,
+ )
+ hierarchy = fit_paris_hierarchy(
+ fitted_graph,
+ n_threads=budget.workers,
+ )
+ plateau_forest = collapse_equal_height_plateaus(hierarchy)
+ hierarchy_group = start_artifact(self.zw, hierarchy_plan)
+ write_hierarchy_group(hierarchy_group, hierarchy, plateau_forest)
+ hierarchy_group.attrs["estimated_peak_bytes"] = estimated_peak_bytes
+ finish_artifact(hierarchy_group, hierarchy_plan)
+ if hierarchy.n_leaves != n_cells:
+ raise ValueError("Paris hierarchy size does not match graph")
+
+ mode = "fixed" if fixed_cluster_count is not None else "auto"
+ cut_parameters = (
+ {"mode": mode, "n_clusters": fixed_cluster_count}
+ if fixed_cluster_count is not None
+ else {
+ "mode": mode,
+ "min_cluster_size": effective_min_cluster_size,
+ }
+ )
+ current_selection = self._ensure_cell_selection(cell_key)
+ if isinstance(graph_ref, ArtifactRef):
+ cell_selection = self._graph_cell_selection(graph_ref)
+ if not self._selection_artifacts_match(
+ cell_selection,
+ current_selection,
+ ):
+ raise ValueError("cell_key does not match the graph cell selection")
+ else:
+ cell_selection = current_selection
+ cut_inputs = {
+ "cluster_hierarchy": hierarchy_plan.ref,
+ "connectivity_map": graph_ref,
+ "cell_selection": cell_selection,
+ }
+ cut_plan = plan_artifact(
+ self.zw,
+ scope=artifact_scope,
+ assay=artifact_assay,
+ kind="cluster_cut",
+ operation="cut_paris_hierarchy",
+ parameters=cut_parameters,
+ inputs=cut_inputs,
+ execution_options={
+ "label": label,
+ "invalidate_cache": force_recalc,
+ },
+ invalidate_cache=force_recalc,
+ required_arrays=(
+ ArrayRequirement("labels", shape=(n_cells,), dtype_kind="i"),
+ ),
+ )
+ result = None
+ if cut_plan.reused:
+ cut_group = reused_artifact_group(self.zw, cut_plan)
+ try:
+ raw_diagnostics = cut_group.attrs.get("diagnostics", [])
+ if not isinstance(raw_diagnostics, list) or any(
+ not isinstance(diagnostic, dict) for diagnostic in raw_diagnostics
+ ):
+ raise TypeError("Paris diagnostics must be mappings")
+ diagnostics = tuple(
+ ParisClusterDiagnostic(**diagnostic)
+ for diagnostic in raw_diagnostics
+ )
+ labels = np.asarray(
+ as_zarr_array(cut_group["labels"], name="labels")[:],
+ dtype=np.int32,
+ )
+ result = ParisClusteringResult(
+ labels=labels,
+ mode=cast(Literal["auto", "fixed"], mode),
+ n_clusters=int(
+ cast(
+ int | float | str,
+ cut_group.attrs["n_clusters"],
+ )
+ ),
+ diagnostics=diagnostics,
+ min_cluster_size=effective_min_cluster_size,
+ )
+ except (KeyError, TypeError, ValueError):
+ cut_plan = plan_artifact(
+ self.zw,
+ scope=artifact_scope,
+ assay=artifact_assay,
+ kind="cluster_cut",
+ operation="cut_paris_hierarchy",
+ parameters=cut_parameters,
+ inputs=cut_inputs,
+ execution_options={
+ "label": label,
+ "invalidate_cache": True,
+ },
+ invalidate_cache=True,
+ )
+ if result is None:
+ if fixed_cluster_count is None:
+ assert effective_min_cluster_size is not None
+ if fitted_graph is None:
+ preflight_paris_adaptive_cut(
+ graph_group,
+ n_cells,
+ budget,
+ )
+ fitted_graph = self.load_graph(graph_loc=graph_loc)
+ split_gate = modularity_split_gains(
+ hierarchy,
+ plateau_forest,
+ fitted_graph,
+ )
+ result = adaptive_cut(
+ hierarchy,
+ effective_min_cluster_size,
+ plateau_forest=plateau_forest,
+ split_gate=split_gate,
+ )
+ else:
+ dendrogram = hierarchy_to_dendrogram(hierarchy)
+ labels = straight_cut(dendrogram, fixed_cluster_count).astype(
+ np.int32,
+ copy=False,
+ )
+ result = ParisClusteringResult(
+ labels=labels,
+ mode="fixed",
+ n_clusters=int(np.unique(labels).size),
+ )
+ cut_group = start_artifact(self.zw, cut_plan)
+ labels_array = create_zarr_dataset(
+ cut_group,
+ "labels",
+ (min(max(n_cells, 1), 100_000),),
+ "i4",
+ result.labels.shape,
+ )
+ labels_array[:] = result.labels
+ cut_group.attrs["n_clusters"] = int(result.n_clusters)
+ cut_group.attrs["diagnostics"] = [
+ asdict(diagnostic) for diagnostic in result.diagnostics
+ ]
+ finish_artifact(cut_group, cut_plan)
+
+ if fixed_cluster_count is not None:
+ dendrogram_plan = plan_artifact(
+ self.zw,
+ scope=artifact_scope,
+ assay=artifact_assay,
+ kind="dendrogram",
+ operation="materialize_paris_dendrogram",
+ parameters={"compatibility": True},
+ inputs={"cluster_hierarchy": hierarchy_plan.ref},
+ execution_options={},
+ required_arrays=(ArrayRequirement("data", dtype_kind="f"),),
+ )
+ if not dendrogram_plan.reused:
+ dendrogram = hierarchy_to_dendrogram(
+ hierarchy,
+ compatibility=True,
+ )
+ dendrogram_group = start_artifact(self.zw, dendrogram_plan)
+ dendrogram_array = create_zarr_dataset(
+ dendrogram_group,
+ "data",
+ (min(max(dendrogram.shape[0], 1), 5000), 4),
+ "f8",
+ dendrogram.shape,
+ )
+ dendrogram_array[:] = dendrogram
+ finish_artifact(dendrogram_group, dendrogram_plan)
+
+ final_label_key = self._col_renamer(label_assay, cell_key, label)
+ preserved_display = column_display(self.zw, final_label_key)
+ self.cells.insert(
+ final_label_key,
+ result.labels,
+ fill_value=-1,
+ key=cell_key,
+ overwrite=True,
+ )
+ link_cell_data_column(
+ self.zw,
+ final_label_key,
+ cut_plan.ref,
+ value_name="labels",
+ default_display=categorical_display(result.labels),
+ preserved_display=preserved_display,
+ )
+ action = "Reused" if cut_plan.reused else "Stored"
+ logger.info(f"{action} Paris clustering with {result.n_clusters} clusters")
+ return replace(
+ result,
+ label_key=final_label_key,
+ hierarchy_generation_id=hierarchy_plan.ref.artifact_id,
+ ref=cut_plan.ref,
+ )
+
+ def _prepare_leiden_clustering(
+ self,
+ graph: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ resolution: float = 1.0,
+ integrated_graph: str | None = None,
+ symmetric_graph: bool = False,
+ graph_upper_only: bool = False,
+ label: str = "leiden_cluster",
+ random_seed: int = 4444,
+ invalidate_cache: bool = False,
+ ) -> _PreparedLeidenClustering:
+ graph_selection = resolve_graph_selection(
+ self,
+ graph,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ integrated_graph=integrated_graph,
+ )
+ from_assay = graph_selection.from_assay
+ cell_key = graph_selection.cell_key
+ feat_key = graph_selection.feat_key
+ integrated_graph = graph_selection.integrated_label
+ n_cells, _effective_k = self._get_graph_ncells_k(graph_selection.graph_loc)
+ graph_input: object = graph_selection.graph_input
+ artifact_scope = (
+ graph_input.scope
+ if isinstance(graph_input, ArtifactRef)
+ else "datastore"
+ if integrated_graph is not None
+ else "assay"
+ )
+ selection = self._ensure_cell_selection(cell_key)
+ if isinstance(graph_input, ArtifactRef):
+ graph_cell_selection = self._graph_cell_selection(graph_input)
+ if not self._selection_artifacts_match(graph_cell_selection, selection):
+ raise ValueError("cell_key does not match the graph cell selection")
+ selection = graph_cell_selection
+ arguments = LeidenArguments(
+ graph=graph_input,
+ resolution=resolution,
+ symmetric_graph=symmetric_graph,
+ graph_upper_only=graph_upper_only,
+ random_seed=random_seed,
+ label=label,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ integrated_graph=integrated_graph,
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope=artifact_scope,
+ assay=(
+ graph_input.assay
+ if isinstance(graph_input, ArtifactRef) and graph_input.scope == "assay"
+ else from_assay
+ if artifact_scope == "assay"
+ else None
+ ),
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ cell_selection=selection,
+ arrays={"values": ((n_cells,), "i")},
+ invalidate_cache=invalidate_cache,
+ )
+ return _PreparedLeidenClustering(
+ planned=planned,
+ graph_loc=graph_selection.graph_loc,
+ from_assay=from_assay,
+ label_assay=graph_selection.output_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ resolution=resolution,
+ symmetric_graph=symmetric_graph,
+ graph_upper_only=graph_upper_only,
+ random_seed=random_seed,
+ label=label,
+ n_cells=n_cells,
+ )
+
+ def _load_prepared_leiden_graph(
+ self,
+ prepared: _PreparedLeidenClustering,
+ ) -> Any:
+ graph = self.load_graph(
+ from_assay=prepared.from_assay,
+ cell_key=prepared.cell_key,
+ feat_key=prepared.feat_key,
+ symmetric=prepared.symmetric_graph,
+ upper_only=prepared.graph_upper_only,
+ graph_loc=prepared.graph_loc,
+ )
+ return graph.tocsr()
+
+ @staticmethod
+ def _compute_prepared_leiden(
+ prepared: _PreparedLeidenClustering,
+ graph: Any,
+ ) -> np.ndarray:
+ from ...clustering.leiden import leiden_membership
+
+ if prepared.planned.reused:
+ raise ValueError("Cannot recompute a reusable Leiden artifact")
+ return leiden_membership(
+ graph,
+ prepared.resolution,
+ prepared.random_seed,
+ )
+
+ def _publish_prepared_leiden(
+ self,
+ prepared: _PreparedLeidenClustering,
+ membership: np.ndarray | None,
+ ) -> tuple[str, ArtifactRef]:
+ if prepared.planned.reused:
+ artifact_group = as_zarr_group(
+ self.zw[artifact_path(prepared.planned.ref)],
+ name=prepared.planned.ref.artifact_id,
+ )
+ membership = artifact_values(artifact_group, "values")
+ else:
+ if membership is None:
+ raise ValueError("Leiden membership is required for a new artifact")
+ membership = np.asarray(membership)
+ if membership.shape != (prepared.n_cells,):
+ raise ValueError(
+ "Leiden membership must contain one label per graph cell"
+ )
+ if membership.dtype.kind not in {"i", "u"}:
+ raise TypeError("Leiden membership must contain integer labels")
+ write_cell_data_artifact(
+ self.zw,
+ prepared.planned,
+ {"values": membership},
+ )
+ column = self._col_renamer(
+ prepared.label_assay,
+ prepared.cell_key,
+ prepared.label,
+ )
+ preserved_display = column_display(self.zw, column)
+ self.cells.insert(
+ column,
+ membership,
+ fill_value=-1,
+ key=prepared.cell_key,
+ overwrite=True,
+ )
+ link_cell_data_column(
+ self.zw,
+ column,
+ prepared.planned.ref,
+ value_name="values",
+ default_display=categorical_display(membership),
+ preserved_display=preserved_display,
+ )
+ action = "Reused" if prepared.planned.reused else "Stored"
+ logger.info(
+ f"{action} Leiden clustering with {np.unique(membership).size} clusters"
+ )
+ return column, prepared.planned.ref
+
+ def run_leiden_clustering(
+ self,
+ graph: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ resolution: float = 1.0,
+ integrated_graph: str | None = None,
+ symmetric_graph: bool = False,
+ graph_upper_only: bool = False,
+ label: str = "leiden_cluster",
+ random_seed: int = 4444,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Execute Leiden clustering and save identities in cell metadata.
+
+ Args:
+ graph: Connectivity map or integrated graph to partition. The
+ current analysis chain of the assay is used when omitted.
+ from_assay: Assay whose current graph should be used.
+ cell_key: Cell key of the graph.
+ feat_key: Feature key of the graph.
+ resolution: Leiden resolution parameter.
+ integrated_graph: Label of an integrated graph to partition.
+ symmetric_graph: Forwarded to `load_graph`.
+ graph_upper_only: Forwarded to `load_graph`.
+ label: Base name of the cell-metadata column that receives labels.
+ random_seed: Seed for the Leiden optimizer.
+ invalidate_cache: Force a new cluster-labels artifact.
+
+ Returns:
+ Reference to the cluster-labels artifact backing the label column.
+ """
+ prepared = self._prepare_leiden_clustering(
+ graph,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ resolution=resolution,
+ integrated_graph=integrated_graph,
+ symmetric_graph=symmetric_graph,
+ graph_upper_only=graph_upper_only,
+ label=label,
+ random_seed=random_seed,
+ invalidate_cache=invalidate_cache,
+ )
+ membership = None
+ if not prepared.planned.reused:
+ graph_matrix = self._load_prepared_leiden_graph(prepared)
+ membership = self._compute_prepared_leiden(prepared, graph_matrix)
+ _column, ref = self._publish_prepared_leiden(prepared, membership)
+ return ref
+
+ def run_paris_clustering(
+ self,
+ graph: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ n_clusters: int | Literal["auto"] = "auto",
+ integrated_graph: str | None = None,
+ min_cluster_size: int | None = None,
+ force_recalc: bool = False,
+ invalidate_cache: bool = False,
+ label: str = "paris_cluster",
+ ) -> "ParisClusteringResult":
+ """Fit the canonical Paris hierarchy and write a fixed or adaptive cut.
+
+ Pass ``graph`` to partition an explicit connectivity map or integrated
+ graph. The returned result carries the cut artifact in its ``ref``
+ field alongside the labels and diagnostics.
+ """
+ if isinstance(n_clusters, (bool, np.bool_)):
+ raise TypeError("n_clusters must be an integer or 'auto'")
+ if isinstance(n_clusters, str):
+ if n_clusters != "auto":
+ raise ValueError("n_clusters must be an integer or 'auto'")
+ fixed_cluster_count = None
+ elif isinstance(n_clusters, (int, np.integer)):
+ if n_clusters < 1:
+ raise ValueError("n_clusters must be positive")
+ fixed_cluster_count = int(n_clusters)
+ else:
+ raise TypeError("n_clusters must be an integer or 'auto'")
+ if fixed_cluster_count is not None and min_cluster_size is not None:
+ raise ValueError("min_cluster_size is only valid when n_clusters='auto'")
+ invalidate_artifacts = force_recalc or invalidate_cache
+
+ graph_selection = resolve_graph_selection(
+ self,
+ graph,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ integrated_graph=integrated_graph,
+ )
+ from_assay = graph_selection.from_assay
+ cell_key = graph_selection.cell_key
+ graph_loc = graph_selection.graph_loc
+ if graph_selection.integrated_label is None and not isinstance(
+ self._lookup_stored_graph(graph_loc=graph_loc),
+ StoredAssayGraph,
+ ):
+ raise TypeError("Paris clustering requires an assay graph")
+
+ n_cells, effective_k = self._get_graph_ncells_k(graph_loc)
+ active_cell_count = int(np.count_nonzero(self.cells.fetch_all(cell_key)))
+ if active_cell_count != n_cells:
+ raise ValueError(
+ f"cell_key {cell_key!r} selects {active_cell_count} cells, "
+ f"but the graph contains {n_cells}"
+ )
+ if fixed_cluster_count is not None and fixed_cluster_count > n_cells:
+ raise ValueError(f"n_clusters must not exceed the graph size ({n_cells})")
+
+ if fixed_cluster_count is None:
+ if min_cluster_size is None:
+ effective_min_cluster_size = effective_k + 1
+ else:
+ if isinstance(min_cluster_size, (bool, np.bool_)) or not isinstance(
+ min_cluster_size,
+ (int, np.integer),
+ ):
+ raise TypeError("min_cluster_size must be an integer")
+ if min_cluster_size < 2:
+ raise ValueError("min_cluster_size must be at least 2")
+ effective_min_cluster_size = int(min_cluster_size)
+ else:
+ effective_min_cluster_size = None
+
+ return self._run_paris_from_artifacts(
+ graph_ref=graph_selection.graph_input,
+ graph_loc=graph_loc,
+ from_assay=from_assay,
+ label_assay=graph_selection.output_assay,
+ cell_key=cell_key,
+ fixed_cluster_count=fixed_cluster_count,
+ effective_min_cluster_size=effective_min_cluster_size,
+ label=label,
+ force_recalc=invalidate_artifacts,
+ )
+
+ def run_topacedo_sampler(
+ self,
+ graph: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ integrated_graph: str | None = None,
+ cluster_key: str | None = None,
+ use_k: int | None = None,
+ density_depth: int = 2,
+ density_bandwidth: float = 5.0,
+ max_sampling_rate: float = 0.05,
+ min_sampling_rate: float = 0.01,
+ min_cells_per_group: int = 3,
+ snn_bandwidth: float = 5.0,
+ seed_reward: float = 3.0,
+ non_seed_reward: float = 0,
+ edge_cost_multiplier: float = 1.0,
+ edge_cost_bandwidth: float = 10.0,
+ save_sampling_key: str = "sketched",
+ save_density_key: str = "cell_density",
+ save_mean_snn_key: str = "snn_value",
+ save_seeds_key: str = "sketch_seeds",
+ rand_state: int = 4466,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Perform sub-sampling (aka sketching) of cells using TopACeDo
+ algorithm. Sub-sampling required that cells are partitioned in cluster
+ already. Since, sub-sampling is dependent on cluster information,
+ having, large number of homogeneous and even sized cluster improves
+ sub-sampling results.
+
+ Args:
+ graph: Connectivity map or integrated graph to sample. The current
+ analysis chain of the assay is used when omitted.
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
+ feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
+ used feature for the given assay will be used.
+ integrated_graph: Integrated graph label. By default, use the latest assay graph.
+ cluster_key: Name of the column in cell metadata table where cluster information is stored.
+ use_k: Number of top k-nearest neighbours to retain in the graph over which downsampling is performed.
+ BY default all neighbours are used. (Default value: None)
+ density_depth: Same as 'search_depth' parameter in `calc_neighbourhood_density`. (Default value: 2)
+ density_bandwidth: This value is used to scale the penalty affected by neighbourhood density. Higher values
+ will lead to a larger penalty. (Default value: 5.0)
+ max_sampling_rate: Maximum fraction of cells to sample from each group. The effective sampling rate is lower
+ than this value depending on the neighbourhood degree and SNN density of cells.
+ Should be greater than 0 and less than 1. (Default value: 0.05)
+ min_sampling_rate: Minimum sampling rate. Effective sampling rate is not allowed to be lower than this
+ value. Should be greater than 0 and less than 1. (Default value: 0.01)
+ min_cells_per_group: Minimum number of cells to sample from each group. (Default value: 3)
+ snn_bandwidth: Bandwidth for the shared nearest neighbour award. Clusters with higher mean SNN values get
+ lower sampling penalty. This value, is raised to mean SNN value of the cluster to obtain
+ sampling reward of the cluster. (Default value: 5.0)
+ seed_reward: Reward/prize value for seed nodes. (Default value: 3.0)
+ non_seed_reward: Reward/prize for non-seed nodes. (Default value: 0)
+ edge_cost_multiplier: This value is multiplier to each edge's cost. Higher values will make graph traversal
+ costly and might lead to removal of poorly connected nodes (Default value: 1.0)
+ edge_cost_bandwidth: This value is raised to edge cost to get an adjusted edge cost (Default value: 10.0)
+ save_sampling_key: base label for marking the cells that were sampled into a cell metadata column
+ (Default value: 'sketched')
+ save_density_key: base label for saving the cell neighbourhood densities into a cell metadata column
+ (Default value: 'cell_density')
+ save_mean_snn_key: base label for saving the SNN value for each cell (identified by topacedo sampler) into
+ a cell metadata column (Default value: 'snn_value')
+ save_seeds_key: base label for saving the seed cells (identified by topacedo sampler) into a cell
+ metadata column (Default value: 'sketch_seeds')
+ rand_state: A random values to set seed while sampling cells from a cluster randomly. (Default value: 4466)
+
+ Returns:
+ Reference to the sampling artifact. Open it with ``load_artifact``
+ to read the ``edges`` array of Steiner tree edges over graph row
+ indices.
+ """
+
+ graph_selection = resolve_graph_selection(
+ self,
+ graph,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ integrated_graph=integrated_graph,
+ )
+ from_assay = graph_selection.from_assay
+ cell_key = graph_selection.cell_key
+ feat_key = graph_selection.feat_key
+ integrated_graph = graph_selection.integrated_label
+ graph_loc = graph_selection.graph_loc
+ output_assay = graph_selection.output_assay
+ if cluster_key is None:
+ raise ValueError("ERROR: Please provide a value for cluster key")
+ clusters = pd.Series(self.cells.fetch(cluster_key, key=cell_key))
+ if integrated_graph is None and not isinstance(
+ self._lookup_stored_graph(graph_loc=graph_loc),
+ StoredAssayGraph,
+ ):
+ raise TypeError("TopACeDo sampling requires an assay graph")
+ graph_input: object = graph_selection.graph_input
+ if not isinstance(graph_input, ArtifactRef) and integrated_graph is None:
+ validate_legacy_graph_selection(
+ self,
+ graph_loc,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ graph_matrix = self.load_graph(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ symmetric=False,
+ upper_only=False,
+ use_k=use_k,
+ graph_loc=graph_loc,
+ )
+ resolver = getattr(self, "_resolve_cell_data_provenance_input", None)
+ cluster_input_is_descriptor = callable(resolver)
+ if callable(resolver):
+ cluster_input: object = resolver(cluster_key, cell_key=cell_key)
+ else:
+ cluster_input = {"value_fingerprint": fingerprint_array(clusters.values)}
+ cluster_column = as_zarr_array(
+ as_zarr_group(self.zw["cellData"], name="cellData")[cluster_key],
+ name=cluster_key,
+ )
+ raw_cut_ref = cluster_column.attrs.get("source_artifact")
+ if (
+ isinstance(raw_cut_ref, dict)
+ and (cut_ref := ArtifactRef.from_dict(raw_cut_ref)).kind == "cluster_cut"
+ ):
+ if not cluster_input_is_descriptor:
+ cluster_input = cut_ref
+ cut_inputs = inspect_artifact(self.zw, cut_ref).inputs or {}
+ raw_hierarchy_ref = cut_inputs.get("cluster_hierarchy")
+ current_graph_input = (
+ graph_input.to_dict()
+ if isinstance(graph_input, ArtifactRef)
+ else graph_input
+ )
+ if cut_inputs.get("connectivity_map") != current_graph_input:
+ raise ValueError("Cluster cut does not belong to the requested graph")
+ cut_group = as_zarr_group(
+ self.zw[inspect_artifact(self.zw, cut_ref).path],
+ name=cut_ref.artifact_id,
+ )
+ clusters = pd.Series(
+ np.asarray(as_zarr_array(cut_group["labels"], name="labels")[:])
+ )
+ else:
+ raw_hierarchy_ref = None
+ if isinstance(raw_hierarchy_ref, dict):
+ from ...clustering.paris import hierarchy_to_dendrogram
+ from .paris_persistence import load_hierarchy_group
+
+ hierarchy_ref = ArtifactRef.from_dict(raw_hierarchy_ref)
+ dendrogram_plan = plan_artifact(
+ self.zw,
+ scope=hierarchy_ref.scope,
+ assay=hierarchy_ref.assay,
+ kind="dendrogram",
+ operation="materialize_paris_dendrogram",
+ parameters={"compatibility": True},
+ inputs={"cluster_hierarchy": hierarchy_ref},
+ execution_options={},
+ required_arrays=(ArrayRequirement("data", dtype_kind="f"),),
+ )
+ if not dendrogram_plan.reused:
+ hierarchy_group = as_zarr_group(
+ self.zw[inspect_artifact(self.zw, hierarchy_ref).path],
+ name=hierarchy_ref.artifact_id,
+ )
+ hierarchy, _plateau = load_hierarchy_group(
+ hierarchy_group,
+ hierarchy_ref.artifact_id,
+ )
+ dendrogram = hierarchy_to_dendrogram(
+ hierarchy,
+ compatibility=True,
+ )
+ dendrogram_group = start_artifact(self.zw, dendrogram_plan)
+ output = create_zarr_dataset(
+ dendrogram_group,
+ "data",
+ (min(max(dendrogram.shape[0], 1), 5000), 4),
+ "f8",
+ dendrogram.shape,
+ )
+ output[:] = dendrogram
+ finish_artifact(dendrogram_group, dendrogram_plan)
+ else:
+ dendrogram_group = reused_artifact_group(self.zw, dendrogram_plan)
+ dendrogram = np.asarray(
+ as_zarr_array(dendrogram_group["data"], name="data")[:]
+ )
+ dendrogram_input: object = dendrogram_plan.ref
+ else:
+ from .paris_persistence import resolve_compatibility_dendrogram
+
+ try:
+ dendrogram_loc, _generation_id = resolve_compatibility_dendrogram(
+ self.zw,
+ graph_loc,
+ self.resources,
+ )
+ dendrogram = np.asarray(
+ as_zarr_array(
+ self.zw[dendrogram_loc],
+ name=dendrogram_loc,
+ )[:]
+ )
+ except KeyError:
+ raise KeyError(
+ "ERROR: Couldn't find the dendrogram for clustering. Please note "
+ "that TopACeDo requires a dendrogram from Paris clustering."
+ )
+ dendrogram_input = {
+ "value_fingerprint": fingerprint_array(dendrogram),
+ }
+
+ if len(clusters) != graph_matrix.shape[0]:
+ raise ValueError(
+ f"ERROR: cluster information exists for {len(clusters)} cells while graph has "
+ f"{graph_matrix.shape[0]} cells."
+ )
+ selection = self._ensure_cell_selection(cell_key)
+ if isinstance(graph_input, ArtifactRef):
+ graph_cell_selection = self._graph_cell_selection(graph_input)
+ if not self._selection_artifacts_match(graph_cell_selection, selection):
+ raise ValueError("cell_key does not match the graph cell selection")
+ selection = graph_cell_selection
+ artifact_scope = (
+ graph_input.scope
+ if isinstance(graph_input, ArtifactRef)
+ else "datastore"
+ if integrated_graph is not None
+ else "assay"
+ )
+ arguments = TopacedoArguments(
+ graph=graph_input,
+ clusters=cluster_input,
+ dendrogram=dendrogram_input,
+ cell_selection=selection,
+ use_k=use_k,
+ density_depth=density_depth,
+ density_bandwidth=density_bandwidth,
+ max_sampling_rate=max_sampling_rate,
+ min_sampling_rate=min_sampling_rate,
+ min_cells_per_group=min_cells_per_group,
+ snn_bandwidth=snn_bandwidth,
+ seed_reward=seed_reward,
+ non_seed_reward=non_seed_reward,
+ edge_cost_multiplier=edge_cost_multiplier,
+ edge_cost_bandwidth=edge_cost_bandwidth,
+ rand_state=rand_state,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ integrated_graph=integrated_graph,
+ cluster_key=cluster_key,
+ save_sampling_key=save_sampling_key,
+ save_density_key=save_density_key,
+ save_mean_snn_key=save_mean_snn_key,
+ save_seeds_key=save_seeds_key,
+ invalidate_cache=invalidate_cache,
+ )
+ planned = arguments.plan(
+ self.zw,
+ scope=artifact_scope,
+ assay=(
+ graph_input.assay
+ if isinstance(graph_input, ArtifactRef) and graph_input.scope == "assay"
+ else from_assay
+ if artifact_scope == "assay"
+ else None
+ ),
+ invalidate_cache=invalidate_cache,
+ required_arrays=(
+ ArrayRequirement(
+ "sampled",
+ shape=(graph_matrix.shape[0],),
+ dtype_kind="b",
+ ),
+ ArrayRequirement(
+ "density",
+ shape=(graph_matrix.shape[0],),
+ dtype_kind="f",
+ ),
+ ArrayRequirement(
+ "mean_snn",
+ shape=(graph_matrix.shape[0],),
+ dtype_kind="f",
+ ),
+ ArrayRequirement(
+ "seeds",
+ shape=(graph_matrix.shape[0],),
+ dtype_kind="b",
+ ),
+ ArrayRequirement(
+ "edges",
+ shape=(None, 2),
+ dtype_kind="i",
+ ),
+ ),
+ )
+ columns = {
+ "sampled": self._col_renamer(
+ output_assay,
+ cell_key,
+ save_sampling_key,
+ ),
+ "density": self._col_renamer(
+ output_assay,
+ cell_key,
+ save_density_key,
+ ),
+ "mean_snn": self._col_renamer(
+ output_assay,
+ cell_key,
+ save_mean_snn_key,
+ ),
+ "seeds": self._col_renamer(
+ output_assay,
+ cell_key,
+ save_seeds_key,
+ ),
+ }
+ preserved_displays = {
+ name: column_display(self.zw, column) for name, column in columns.items()
+ }
+ if planned.reused:
+ artifact_group = reused_artifact_group(self.zw, planned)
+ sampled = artifact_values(artifact_group, "sampled").astype(bool)
+ density = artifact_values(artifact_group, "density")
+ mean_snn = artifact_values(artifact_group, "mean_snn")
+ seeds = artifact_values(artifact_group, "seeds").astype(bool)
+ else:
+ try:
+ from topacedo import TopacedoSampler
+ except ImportError as error:
+ raise ImportError("Could not find topacedo package") from error
+ sampler = TopacedoSampler(
+ graph_matrix,
+ clusters.values,
+ dendrogram,
+ density_depth,
+ density_bandwidth,
+ max_sampling_rate,
+ min_sampling_rate,
+ min_cells_per_group,
+ snn_bandwidth,
+ seed_reward,
+ non_seed_reward,
+ edge_cost_multiplier,
+ edge_cost_bandwidth,
+ rand_state,
+ )
+ nodes, edges = sampler.run()
+ raw_node_indices = np.asarray(nodes)
+ if raw_node_indices.dtype.kind not in {"i", "u"}:
+ raise ValueError("TopACeDo returned non-integer sampled-cell indices")
+ node_indices = raw_node_indices.astype(np.int64, copy=False)
+ if node_indices.ndim != 1 or np.any(
+ (node_indices < 0) | (node_indices >= graph_matrix.shape[0])
+ ):
+ raise ValueError("TopACeDo returned invalid sampled-cell indices")
+ sampled = np.zeros(graph_matrix.shape[0], dtype=bool)
+ sampled[node_indices] = True
+ density = np.asarray(sampler.densities, dtype=np.float64)
+ mean_snn = np.asarray(sampler.meanSnn, dtype=np.float64)
+ if density.shape != (graph_matrix.shape[0],):
+ raise ValueError("TopACeDo returned invalid cell-density values")
+ if mean_snn.shape != (graph_matrix.shape[0],):
+ raise ValueError("TopACeDo returned invalid mean-SNN values")
+ raw_seed_indices = np.asarray(sampler.seeds)
+ if raw_seed_indices.dtype.kind not in {"i", "u"}:
+ raise ValueError("TopACeDo returned non-integer seed-cell indices")
+ seed_indices = raw_seed_indices.astype(np.int64, copy=False)
+ if seed_indices.ndim != 1 or np.any(
+ (seed_indices < 0) | (seed_indices >= graph_matrix.shape[0])
+ ):
+ raise ValueError("TopACeDo returned invalid seed-cell indices")
+ seeds = np.zeros(graph_matrix.shape[0], dtype=bool)
+ seeds[seed_indices] = True
+ raw_edge_values = np.asarray(edges)
+ if raw_edge_values.size and raw_edge_values.dtype.kind not in {"i", "u"}:
+ raise ValueError("TopACeDo returned non-integer edge pairs")
+ edge_values = raw_edge_values.astype(np.int64, copy=False)
+ if edge_values.size == 0:
+ edge_values = edge_values.reshape(0, 2)
+ elif edge_values.ndim != 2 or edge_values.shape[1] != 2:
+ raise ValueError("TopACeDo returned invalid edge pairs")
+ if np.any((edge_values < 0) | (edge_values >= graph_matrix.shape[0])):
+ raise ValueError("TopACeDo returned out-of-range edge endpoints")
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {
+ "sampled": sampled,
+ "density": density,
+ "mean_snn": mean_snn,
+ "seeds": seeds,
+ "edges": edge_values,
+ },
+ )
+ values = {
+ "sampled": sampled,
+ "density": density,
+ "mean_snn": mean_snn,
+ "seeds": seeds,
+ }
+ defaults = {
+ "sampled": categorical_display(sampled),
+ "density": continuous_display(density),
+ "mean_snn": continuous_display(mean_snn),
+ "seeds": categorical_display(seeds),
+ }
+ for name, column in columns.items():
+ self.cells.insert(
+ column,
+ values[name],
+ fill_value=False if name in {"sampled", "seeds"} else np.nan,
+ key=cell_key,
+ overwrite=True,
+ )
+ for name, column in columns.items():
+ link_cell_data_column(
+ self.zw,
+ column,
+ planned.ref,
+ value_name=name,
+ default_display=defaults[name],
+ preserved_display=preserved_displays[name],
+ )
+ logger.debug(f"Sketched cells saved under column '{columns['sampled']}'")
+ logger.debug(
+ f"Cell neighbourhood densities saved under column: '{columns['density']}'"
+ )
+ logger.debug(f"Mean SNN values saved under column: '{columns['mean_snn']}'")
+ logger.debug(f"Seed cells saved under column: '{columns['seeds']}'")
+ return planned.ref
diff --git a/scarf/datastore/_operations/embeddings.py b/scarf/datastore/_operations/embeddings.py
new file mode 100644
index 00000000..e447c241
--- /dev/null
+++ b/scarf/datastore/_operations/embeddings.py
@@ -0,0 +1,619 @@
+from typing import TYPE_CHECKING
+
+import numpy as np
+from scipy.sparse import csr_matrix
+
+from ...graph.encoded_paths import (
+ lookup_latest_kmeans_path,
+)
+from ...graph.distances import validate_distance_provenance
+from ...graph.paths import StoredAssayGraph
+from ...graph.state import (
+ embedding_initialization_path_from_state,
+ read_assay_state,
+ resolve_graph_selection,
+ validate_legacy_graph_selection,
+)
+from ...metadata.artifacts import (
+ artifact_values,
+ column_display,
+ continuous_display,
+ link_cell_data_column,
+ plan_cell_data_artifact,
+ write_cell_data_artifact,
+)
+from ...metadata.arguments import TsneArguments, UmapArguments
+from ...storage.artifacts import (
+ ArtifactRef,
+ artifact_path,
+ fingerprint_array,
+)
+from ...storage.types import as_zarr_array, as_zarr_group
+from ...utils.logging import logger, progress_enabled
+
+if TYPE_CHECKING:
+ from .graph import _GraphOperationsMixin as _EmbeddingOperationsBase
+else:
+ _EmbeddingOperationsBase = object
+
+
+class _EmbeddingOperationsMixin(_EmbeddingOperationsBase):
+ def _get_ini_embed(
+ self, from_assay: str, cell_key: str, feat_key: str, n_comps: int
+ ) -> np.ndarray:
+ """Runs PCA on kmeans cluster centers and ascribes the PC values to
+ individual cells based on their cluster labels. This is used in
+ `run_umap` and `run_tsne` for initial embedding of cells. Uses
+ `rescale_array` to reduce the magnitude of extreme values.
+
+ Args:
+ from_assay: Name fo the assay for which Kmeans was fit.
+ cell_key: Cell key used.
+ feat_key: Feature key used.
+ n_comps: Number of PC components to use
+
+ Returns:
+ Matrix with n_comps dimensions representing initial embedding of cells.
+ """
+ from ...embeddings.initialization import initial_embedding
+
+ # Initial embedding only needs the reduction and k-means groups, not an
+ # ANN index, KNN graph, or built cell graph.
+ kmeans_loc = embedding_initialization_path_from_state(
+ self.zw,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ if kmeans_loc is None:
+ kmeans_loc = lookup_latest_kmeans_path(
+ self.zw,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ if kmeans_loc is None:
+ raise KeyError("No k-means initialization group found for assay graph")
+ kmeans_grp = as_zarr_group(self.zw[kmeans_loc], name=kmeans_loc)
+ cluster_centers = np.asarray(
+ as_zarr_array(
+ kmeans_grp["cluster_centers"],
+ name="cluster_centers",
+ )[:]
+ )
+ clusters = np.asarray(
+ as_zarr_array(kmeans_grp["cluster_labels"], name="cluster_labels")[:]
+ )
+ return initial_embedding(cluster_centers, clusters, n_comps)
+
+ def run_tsne(
+ self,
+ graph: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ symmetric_graph: bool = False,
+ graph_upper_only: bool = False,
+ ini_embed: np.ndarray | None = None,
+ tsne_dims: int = 2,
+ lambda_scale: float = 1.0,
+ max_iter: int = 500,
+ early_iter: int = 200,
+ alpha: int = 10,
+ box_h: float = 0.7,
+ temp_file_loc: str = ".",
+ label: str = "tSNE",
+ verbose: bool = True,
+ parallel: bool = False,
+ nthreads: int | None = None,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Run SGtSNE-pi (Read more here:
+ https://github.com/fcdimitr/sgtsnepi/tree/v1.0.1). This is an
+ implementation of tSNE that runs directly on graph structures. We use
+ connectivity graphs generated by graph-construction methods to create a
+ layout of cells using tSNE algorithm. This function makes a system call to sgtSNE
+ binary. To get a better understanding of how the parameters affect the
+ embedding, check this out: http://t-sne-pi.cs.duke.edu/
+
+ Args:
+ graph: Connectivity map or integrated graph to embed. The current
+ analysis chain of the assay is used when omitted.
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
+ feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
+ used feature for the given assay will be used.
+ symmetric_graph: This parameter is forwarded to `load_graph` and is same as there. (Default value: False)
+ graph_upper_only: This parameter is forwarded to `load_graph` and is same as there. (Default value: False)
+ ini_embed: Initial embedding coordinates for the cells in cell_key. Should have the same number of columns
+ as tsne_dims. If not value is provided then the initial embedding is obtained using
+ `get_ini_embed`.
+ tsne_dims: Number of tSNE dimensions to compute (Default value: 2)
+ lambda_scale: λ rescaling parameter (Default value: 1.0)
+ max_iter: Maximum number of iterations (Default value: 500)
+ early_iter: Number of early exaggeration iterations (Default value: 200)
+ alpha: Early exaggeration multiplier (Default value: 10)
+ box_h: Grid side length (accuracy control). Lower values might drastically slow down
+ the algorithm (Default value: 0.7)
+ temp_file_loc: Location of temporary file. By default, these files will be created in the current working
+ directory. These files are deleted before the method returns.
+ label: base label for tSNE dimensions in the cell metadata column (Default value: 'tSNE')
+ verbose: If True (default) then the full log from SGtSNEpi algorithm is shown.
+ parallel: Whether to run tSNE in parallel mode. Setting value to True will use `nthreads` threads.
+ The results are not reproducible in parallel mode. (Default value: False)
+ nthreads: If parallel=True then this number of threads will be used to run tSNE. By default the `nthreads`
+ attribute of the class is used. (Default value: None)
+
+ Returns:
+ Reference to the embedding artifact backing the layout columns.
+ """
+ selection = resolve_graph_selection(
+ self,
+ graph,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ )
+ from_assay = selection.from_assay
+ cell_key = selection.cell_key
+ feat_key = selection.feat_key
+ graph_input: object = selection.graph_input
+ if selection.integrated_label is None and not isinstance(
+ graph_input,
+ ArtifactRef,
+ ):
+ validate_legacy_graph_selection(
+ self,
+ selection.graph_loc,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ graph_matrix = self.load_graph(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ symmetric=symmetric_graph,
+ upper_only=graph_upper_only,
+ graph_loc=selection.graph_loc,
+ )
+ user_initialization = ini_embed is not None
+ if ini_embed is None:
+ ini_embed = self._get_ini_embed(from_assay, cell_key, feat_key, tsne_dims)
+ else:
+ if ini_embed.shape != (graph_matrix.shape[0], tsne_dims):
+ raise ValueError(
+ "ERROR: Provided initial embedding does not shape required shape: "
+ f"{(graph_matrix.shape[0], tsne_dims)}"
+ )
+ state = read_assay_state(self.zw, from_assay)
+ initialization_input: object
+ if (
+ not user_initialization
+ and state is not None
+ and state.matches(cell_key, feat_key)
+ and state.embedding_initialization is not None
+ ):
+ initialization_input = state.embedding_initialization
+ else:
+ initialization_input = {"value_fingerprint": fingerprint_array(ini_embed)}
+ if parallel:
+ if nthreads is None:
+ nthreads = self.nthreads
+ else:
+ assert isinstance(nthreads, int)
+ else:
+ nthreads = 1
+ cell_selection = self._ensure_cell_selection(cell_key)
+ if isinstance(graph_input, ArtifactRef):
+ graph_cell_selection = self._graph_cell_selection(graph_input)
+ if not self._selection_artifacts_match(
+ graph_cell_selection,
+ cell_selection,
+ ):
+ raise ValueError("cell_key does not match the graph cell selection")
+ cell_selection = graph_cell_selection
+ artifact_scope = (
+ graph_input.scope if isinstance(graph_input, ArtifactRef) else "assay"
+ )
+ arguments = TsneArguments(
+ graph=graph_input,
+ initialization=initialization_input,
+ symmetric_graph=symmetric_graph,
+ graph_upper_only=graph_upper_only,
+ tsne_dims=tsne_dims,
+ lambda_scale=lambda_scale,
+ max_iter=max_iter,
+ early_iter=early_iter,
+ alpha=alpha,
+ box_h=box_h,
+ parallel=parallel,
+ parallel_threads=nthreads,
+ label=label,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ temp_file_loc=temp_file_loc,
+ verbose=verbose,
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope=artifact_scope,
+ assay=(
+ graph_input.assay
+ if isinstance(graph_input, ArtifactRef) and graph_input.scope == "assay"
+ else from_assay
+ if artifact_scope == "assay"
+ else None
+ ),
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ cell_selection=cell_selection,
+ arrays={"values": ((graph_matrix.shape[0], tsne_dims), "f")},
+ invalidate_cache=invalidate_cache,
+ )
+ columns = [
+ self._col_renamer(
+ selection.output_assay,
+ cell_key,
+ f"{label}{i + 1}",
+ )
+ for i in range(tsne_dims)
+ ]
+ preserved_displays = [column_display(self.zw, column) for column in columns]
+ if planned.reused:
+ artifact_group = as_zarr_group(
+ self.zw[artifact_path(planned.ref)],
+ name=planned.ref.artifact_id,
+ )
+ values = artifact_values(artifact_group, "values")
+ else:
+ import sys
+
+ if sys.platform not in ["posix", "linux"]:
+ raise RuntimeError(
+ f"{sys.platform} operating system is currently not supported."
+ )
+ from ...embeddings.sgtsne import run_sgtsne
+
+ try:
+ raw_embedding = np.asarray(
+ run_sgtsne(
+ graph_matrix,
+ ini_embed,
+ tsne_dims=tsne_dims,
+ max_iter=max_iter,
+ early_iter=early_iter,
+ alpha=alpha,
+ lambda_scale=lambda_scale,
+ box_h=box_h,
+ temp_file_loc=temp_file_loc,
+ verbose=verbose,
+ parallel=parallel,
+ nthreads=nthreads,
+ )
+ )
+ except (FileNotFoundError, ImportError) as exc:
+ raise RuntimeError(
+ "SG-tSNE failed, possibly due to missing sgtsne executable or "
+ f"sgtsnepi package: {exc}"
+ ) from exc
+ if raw_embedding.shape != (tsne_dims, graph_matrix.shape[0]):
+ raise ValueError(
+ "SG-tSNE returned an embedding with shape "
+ f"{raw_embedding.shape}; "
+ f"expected {(tsne_dims, graph_matrix.shape[0])}"
+ )
+ values = raw_embedding.T
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {"values": values},
+ )
+ for i, column in enumerate(columns):
+ self.cells.insert(
+ column,
+ values[:, i],
+ key=cell_key,
+ overwrite=True,
+ )
+ for i, column in enumerate(columns):
+ link_cell_data_column(
+ self.zw,
+ column,
+ planned.ref,
+ value_name="values",
+ value_index=i,
+ default_display=continuous_display(values[:, i]),
+ preserved_display=preserved_displays[i],
+ )
+ action = "Reused" if planned.reused else "Stored"
+ logger.info(
+ f"{action} {tsne_dims}-dimensional t-SNE embedding for {len(values)} cells"
+ )
+ return planned.ref
+
+ def run_umap(
+ self,
+ graph: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ symmetric_graph: bool | None = False,
+ graph_upper_only: bool | None = False,
+ ini_embed: np.ndarray | None = None,
+ umap_dims: int = 2,
+ spread: float = 2.0,
+ min_dist: float = 1,
+ n_epochs: int = 300,
+ repulsion_strength: float = 1.0,
+ initial_alpha: float = 1.0,
+ negative_sample_rate: float = 5,
+ use_density_map: bool = False,
+ dens_lambda: float = 2.0,
+ dens_frac: float = 0.3,
+ dens_var_shift: float = 0.1,
+ random_seed: int = 4444,
+ label: str = "UMAP",
+ integrated_graph: str | None = None,
+ parallel: bool = False,
+ nthreads: int | None = None,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Runs UMAP algorithm using the precomputed cell-neighbourhood graph.
+ The calculated UMAP coordinates are saved in the cell metadata table.
+
+ Args:
+ graph: Connectivity map or integrated graph to embed. The current
+ analysis chain of the assay is used when omitted.
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
+ feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
+ used feature for the given assay will be used.
+ symmetric_graph: This parameter is forwarded to `load_graph` and is same as there. (Default value: False)
+ graph_upper_only: This parameter is forwarded to `load_graph` and is same as there. (Default value: False)
+ ini_embed: Initial embedding coordinates for the cells in cell_key. Should have the same number of columns
+ as umap_dims. If not value is provided then the initial embedding is obtained using
+ `get_ini_embed`.
+ umap_dims: Number of dimensions of UMAP embedding (Default value: 2)
+ spread: Same as spread in UMAP package. The effective scale of embedded points. In combination with
+ ``min_dist`` this determines how clustered/clumped the embedded points are.
+ min_dist: Same as min_dist in UMAP package. The effective minimum distance between embedded points.
+ Smaller values will result in a more clustered/clumped embedding where nearby points on the
+ manifold are drawn closer together, while larger values will result on a more even dispersal of
+ points. The value should be set relative to the ``spread`` value, which determines the scale at
+ which embedded points will be spread out. (Default value: 1)
+ n_epochs: Same as n_epochs in UMAP package. The number of epochs to be used in optimizing the
+ low dimensional embedding. Larger values may result in more accurate embeddings.
+ (Default value: 300)
+ repulsion_strength: Same as repulsion_strength in UMAP package. Weighting applied to negative samples in
+ low dimensional embedding optimization. Values higher than one will result in greater
+ weight being given to negative samples. (Default value: 1.0)
+ initial_alpha: Same as learning_rate in UMAP package. The initial learning rate for the embedding
+ optimization. (Default value: 1.0)
+ negative_sample_rate: Same as negative_sample_rate in UMAP package. The number of negative samples to
+ select per positive sample in the optimization process. Increasing this value will
+ result in greater repulsive force being applied, greater optimization cost, but
+ slightly more accuracy. (Default value: 5)
+ use_density_map: If True, run densMAP (density-preserving UMAP) instead of standard UMAP.
+ dens_lambda: densMAP density preservation strength (Default value: 2.0).
+ dens_frac: Fraction of nearest neighbors used for local density estimation (Default value: 0.3).
+ dens_var_shift: Variance shift for density correction (Default value: 0.1).
+ random_seed: (Default value: 4444)
+ label: base label for UMAP dimensions in the cell metadata column (Default value: 'UMAP')
+ integrated_graph: Label of an integrated graph to embed instead of the assay graph.
+ parallel: Whether to run UMAP in parallel mode. Setting value to True will use `nthreads` threads.
+ The results are not reproducible in parallel mode. (Default value: False)
+ nthreads: If parallel=True then this number of threads will be used to run UMAP. By default, the `nthreads`
+ attribute of the class is used. (Default value: None)
+
+ Returns:
+ Reference to the embedding artifact backing the layout columns.
+ """
+ from ...embeddings.umap import fit_transform
+
+ selection = resolve_graph_selection(
+ self,
+ graph,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ integrated_graph=integrated_graph,
+ )
+ from_assay = selection.from_assay
+ cell_key = selection.cell_key
+ feat_key = selection.feat_key
+ integrated_graph = selection.integrated_label
+ graph_matrix = self.load_graph(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ symmetric=symmetric_graph,
+ upper_only=graph_upper_only,
+ graph_loc=selection.graph_loc,
+ )
+ graph_input: object = selection.graph_input
+ user_initialization = ini_embed is not None
+ if ini_embed is None:
+ ini_embed = self._get_ini_embed(from_assay, cell_key, feat_key, umap_dims)
+ state = read_assay_state(self.zw, from_assay)
+ initialization_input: object
+ if (
+ not user_initialization
+ and state is not None
+ and state.matches(cell_key, feat_key)
+ and state.embedding_initialization is not None
+ ):
+ initialization_input = state.embedding_initialization
+ else:
+ initialization_input = {"value_fingerprint": fingerprint_array(ini_embed)}
+ if nthreads is None:
+ nthreads = self.nthreads
+ cell_selection = self._ensure_cell_selection(cell_key)
+ if isinstance(graph_input, ArtifactRef):
+ graph_cell_selection = self._graph_cell_selection(graph_input)
+ if not self._selection_artifacts_match(
+ graph_cell_selection,
+ cell_selection,
+ ):
+ raise ValueError("cell_key does not match the graph cell selection")
+ cell_selection = graph_cell_selection
+ effective_density_map = use_density_map and integrated_graph is None
+ artifact_scope = (
+ graph_input.scope
+ if isinstance(graph_input, ArtifactRef)
+ else "datastore"
+ if integrated_graph is not None
+ else "assay"
+ )
+ arguments = UmapArguments(
+ graph=graph_input,
+ initialization=initialization_input,
+ symmetric_graph=symmetric_graph,
+ graph_upper_only=graph_upper_only,
+ umap_dims=umap_dims,
+ spread=spread,
+ min_dist=min_dist,
+ n_epochs=n_epochs,
+ repulsion_strength=repulsion_strength,
+ initial_alpha=initial_alpha,
+ negative_sample_rate=negative_sample_rate,
+ use_density_map=effective_density_map,
+ dens_lambda=dens_lambda,
+ dens_frac=dens_frac,
+ dens_var_shift=dens_var_shift,
+ random_seed=random_seed,
+ parallel=parallel,
+ parallel_threads=nthreads if parallel else None,
+ label=label,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ integrated_graph=integrated_graph,
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope=artifact_scope,
+ assay=(
+ graph_input.assay
+ if isinstance(graph_input, ArtifactRef) and graph_input.scope == "assay"
+ else from_assay
+ if artifact_scope == "assay"
+ else None
+ ),
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ cell_selection=cell_selection,
+ arrays={"values": ((graph_matrix.shape[0], umap_dims), "f")},
+ invalidate_cache=invalidate_cache,
+ )
+ verbose = progress_enabled()
+
+ if use_density_map and integrated_graph is not None:
+ logger.warning(
+ "DensMap is not available for integrated graphs. Running standard UMAP."
+ )
+ if effective_density_map:
+ stored = self._lookup_stored_graph(graph_loc=selection.graph_loc)
+ if not isinstance(stored, StoredAssayGraph):
+ raise TypeError("DensMAP requires an assay graph")
+ knn_loc = stored.paths.nearest_neighbors_group_path
+ logger.trace(f"Loading KNN dists and indices from {knn_loc}")
+ validate_distance_provenance(self.zw, knn_loc)
+ knn_group = as_zarr_group(self.zw[knn_loc], name=knn_loc)
+ dists = np.asarray(
+ as_zarr_array(knn_group["distances"], name="distances")[:]
+ )
+ indices = np.asarray(as_zarr_array(knn_group["indices"], name="indices")[:])
+ dmat = csr_matrix(
+ (
+ dists.flatten(),
+ (
+ np.repeat(range(indices.shape[0]), indices.shape[1]),
+ indices.flatten(),
+ ),
+ ),
+ shape=(indices.shape[0], indices.shape[0]),
+ )
+ # dmat = dmat.maximum(dmat.transpose()).todok()
+ logger.trace("Created sparse KNN dists and indices")
+ densmap_kwds = {
+ "lambda": dens_lambda,
+ "frac": dens_frac,
+ "var_shift": dens_var_shift,
+ "n_neighbors": dists.shape[1],
+ "knn_dists": dmat,
+ }
+ else:
+ densmap_kwds = {}
+
+ if planned.reused:
+ artifact_group = as_zarr_group(
+ self.zw[artifact_path(planned.ref)],
+ name=planned.ref.artifact_id,
+ )
+ t = artifact_values(artifact_group, "values")
+ else:
+ t, _a, _b = fit_transform(
+ graph=graph_matrix.tocoo(),
+ ini_embed=ini_embed,
+ spread=spread,
+ min_dist=min_dist,
+ n_epochs=n_epochs,
+ random_seed=random_seed,
+ repulsion_strength=repulsion_strength,
+ initial_alpha=initial_alpha,
+ negative_sample_rate=negative_sample_rate,
+ densmap_kwds=densmap_kwds,
+ parallel=parallel,
+ nthreads=nthreads,
+ verbose=verbose,
+ )
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {"values": t},
+ )
+
+ columns = [
+ self._col_renamer(
+ selection.output_assay,
+ cell_key,
+ f"{label}{i + 1}",
+ )
+ for i in range(umap_dims)
+ ]
+ preserved_displays = [column_display(self.zw, column) for column in columns]
+ for i, column in enumerate(columns):
+ self.cells.insert(
+ column,
+ t[:, i],
+ key=cell_key,
+ overwrite=True,
+ )
+ link_cell_data_column(
+ self.zw,
+ column,
+ planned.ref,
+ value_name="values",
+ value_index=i,
+ default_display=continuous_display(t[:, i]),
+ preserved_display=preserved_displays[i],
+ )
+ action = "Reused" if planned.reused else "Stored"
+ logger.info(
+ f"{action} {umap_dims}-dimensional UMAP embedding for {len(t)} cells"
+ )
+ return planned.ref
diff --git a/scarf/datastore/_operations/enrichment_store.py b/scarf/datastore/_operations/enrichment_store.py
new file mode 100644
index 00000000..1e205032
--- /dev/null
+++ b/scarf/datastore/_operations/enrichment_store.py
@@ -0,0 +1,775 @@
+import hashlib
+import json
+from collections.abc import Callable, Iterator, Sequence
+from contextlib import AbstractContextManager, nullcontext
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+import zarr
+
+from ...assay import Assay
+from ...features.enrichment.results import EnrichmentResult
+from ...graph.arguments import OperationArguments
+from ...storage.artifact_writer import ArrayRequirement
+from ...storage.artifacts import (
+ ArtifactRef,
+ ArtifactStatus,
+ callable_identity,
+ inspect_artifact,
+)
+from ...storage.types import as_zarr_array, as_zarr_group
+from ...utils.arrays import array_digest
+
+
+_ENRICHMENT_LAYOUT = "cells_by_sources"
+_ENRICHMENT_ACTIVE_SLOT = "_active_slot"
+_ENRICHMENT_RUN_PREFIX = "_run_"
+_ENRICHMENT_ARTIFACT_RESULTS = "artifact_results"
+_ENRICHMENT_LEGACY_ARTIFACTS = "artifacts"
+
+
+@dataclass(frozen=True, slots=True)
+class _EnrichmentScorer:
+ """Method-specific enrichment scoring inputs for the shared driver."""
+
+ method: str
+ algorithm_version: int
+ method_payload: dict[str, Any]
+ arguments: OperationArguments
+ cell_index: np.ndarray
+ feature_index: np.ndarray
+ matched_feature_index: np.ndarray
+ source_names: np.ndarray
+ source_sizes: np.ndarray
+ rank_feature_index: np.ndarray | None
+ extra_required_arrays: tuple[ArrayRequirement, ...]
+ score_batches: Callable[[], Iterator[np.ndarray]]
+ write_context: Callable[[], AbstractContextManager[None]] = nullcontext
+
+
+def _validate_enrichment_label(label: str) -> str:
+ if not isinstance(label, str):
+ raise TypeError("Enrichment label must be a string")
+ if not label or label in {".", ".."}:
+ raise ValueError("Enrichment label must be a non-empty path component")
+ if "/" in label or "\\" in label or any(ord(char) < 32 for char in label):
+ raise ValueError(
+ "Enrichment label must not contain path separators or control characters"
+ )
+ return label
+
+
+def _execution_digest(payload: dict[str, Any]) -> str:
+ try:
+ encoded = json.dumps(
+ payload,
+ sort_keys=True,
+ separators=(",", ":"),
+ allow_nan=False,
+ ).encode("utf-8")
+ except (TypeError, ValueError) as exc:
+ raise ValueError("Enrichment execution metadata must be JSON-safe") from exc
+ return hashlib.blake2b(encoded, digest_size=16).hexdigest()
+
+
+def _resolve_enrichment_slot(
+ label_group: zarr.Group,
+ *,
+ label: str,
+) -> zarr.Group:
+ active_slot = label_group.attrs.get(_ENRICHMENT_ACTIVE_SLOT)
+ if active_slot is None:
+ return label_group
+ if (
+ not isinstance(active_slot, str)
+ or not active_slot.startswith(_ENRICHMENT_RUN_PREFIX)
+ or "/" in active_slot
+ or active_slot not in label_group
+ ):
+ raise ValueError(f"Enrichment slot {label!r} has an invalid active result")
+ return as_zarr_group(
+ label_group[active_slot],
+ name=f"{label}/{active_slot}",
+ )
+
+
+def _enrichment_artifact_entry(
+ assay: Assay,
+ label: str,
+) -> tuple[ArtifactRef, str | None, str | None] | None:
+ if "enrichment" not in assay.z:
+ return None
+ enrichment_group = as_zarr_group(
+ assay.z["enrichment"],
+ name=f"{assay.name}/enrichment",
+ )
+ raw_results = enrichment_group.attrs.get(_ENRICHMENT_ARTIFACT_RESULTS)
+ if raw_results is not None:
+ if not isinstance(raw_results, dict):
+ raise ValueError("Enrichment artifact index is invalid")
+ raw_entry = raw_results.get(label)
+ if raw_entry is not None:
+ if not isinstance(raw_entry, dict):
+ raise ValueError(
+ f"Enrichment label {label!r} has an invalid artifact entry"
+ )
+ raw_ref = raw_entry.get("artifact")
+ cell_key = raw_entry.get("cell_key")
+ feat_key = raw_entry.get("feat_key")
+ if (
+ not isinstance(raw_ref, dict)
+ or not isinstance(cell_key, str)
+ or not cell_key
+ or not isinstance(feat_key, str)
+ or not feat_key
+ ):
+ raise ValueError(
+ f"Enrichment label {label!r} has invalid execution metadata"
+ )
+ return ArtifactRef.from_dict(raw_ref), cell_key, feat_key
+ raw_artifacts = enrichment_group.attrs.get(_ENRICHMENT_LEGACY_ARTIFACTS)
+ if raw_artifacts is None:
+ return None
+ if not isinstance(raw_artifacts, dict):
+ raise ValueError("Enrichment artifact index is invalid")
+ raw_ref = raw_artifacts.get(label)
+ if raw_ref is None:
+ return None
+ if not isinstance(raw_ref, dict):
+ raise ValueError(f"Enrichment label {label!r} has an invalid artifact ref")
+ return ArtifactRef.from_dict(raw_ref), None, None
+
+
+def _enrichment_artifact_ref(
+ assay: Assay,
+ label: str,
+) -> ArtifactRef | None:
+ entry = _enrichment_artifact_entry(assay, label)
+ return None if entry is None else entry[0]
+
+
+def _legacy_enrichment_slot(
+ assay: Assay,
+ label: str,
+) -> zarr.Group | None:
+ if "enrichment" not in assay.z:
+ return None
+ enrichment_group = as_zarr_group(
+ assay.z["enrichment"],
+ name=f"{assay.name}/enrichment",
+ )
+ if label not in enrichment_group:
+ return None
+ label_group = as_zarr_group(
+ enrichment_group[label],
+ name=f"{assay.name}/enrichment/{label}",
+ )
+ return _resolve_enrichment_slot(label_group, label=label)
+
+
+def _publish_enrichment_artifact(
+ assay: Assay,
+ label: str,
+ ref: ArtifactRef,
+ *,
+ cell_key: str,
+ feat_key: str,
+) -> None:
+ if "enrichment" not in assay.z:
+ assay.z.create_group("enrichment")
+ enrichment_group = as_zarr_group(
+ assay.z["enrichment"],
+ name=f"{assay.name}/enrichment",
+ )
+ raw_results = enrichment_group.attrs.get(_ENRICHMENT_ARTIFACT_RESULTS, {})
+ if not isinstance(raw_results, dict):
+ raise ValueError("Enrichment artifact index is invalid")
+ results = dict(raw_results)
+ results[label] = {
+ "artifact": ref.to_dict(),
+ "cell_key": cell_key,
+ "feat_key": feat_key,
+ }
+ enrichment_group.attrs[_ENRICHMENT_ARTIFACT_RESULTS] = results
+
+
+def _write_enrichment_slot(
+ slot: zarr.Group,
+ *,
+ attrs: dict[str, Any],
+ score_batches: Iterator[np.ndarray],
+ n_cells: int,
+ source_names: np.ndarray,
+ source_sizes: np.ndarray,
+ cell_index: np.ndarray,
+ matched_feature_index: np.ndarray,
+ rank_feature_index: np.ndarray | None,
+) -> None:
+ from ...storage.arrays import create_metadata_column, create_numeric_array
+ from ...storage.layout import normed_array_spec
+ from ...storage.profiles import resolve_storage_profile
+ from ...storage.sharding import write_dense_from_row_batches
+
+ names = np.asarray(source_names)
+ sizes = np.asarray(source_sizes, dtype=np.int64)
+ cells = np.asarray(cell_index, dtype=np.int64)
+ matched = np.asarray(matched_feature_index, dtype=np.int64)
+ rank = (
+ None
+ if rank_feature_index is None
+ else np.asarray(rank_feature_index, dtype=np.int64)
+ )
+ n_sources = len(names)
+ if n_cells < 1 or len(cells) != n_cells:
+ raise ValueError("Enrichment cell index is empty or misaligned")
+ if n_sources < 1 or len(sizes) != n_sources:
+ raise ValueError("Enrichment source metadata is empty or misaligned")
+ if len(matched) < 1:
+ raise ValueError("Enrichment has no matched features")
+ json.dumps(attrs, sort_keys=True, allow_nan=False)
+
+ slot.attrs["complete"] = False
+ for key, value in attrs.items():
+ slot.attrs[key] = value
+ try:
+ create_metadata_column(
+ slot,
+ "cell_index",
+ data=cells,
+ dtype=np.int64,
+ chunkSize=100_000,
+ )
+ create_metadata_column(
+ slot,
+ "matched_feature_index",
+ data=matched,
+ dtype=np.int64,
+ chunkSize=100_000,
+ )
+ if rank is not None:
+ create_metadata_column(
+ slot,
+ "rank_feature_index",
+ data=rank,
+ dtype=np.int64,
+ chunkSize=100_000,
+ )
+ create_metadata_column(
+ slot,
+ "source_names",
+ data=names,
+ chunkSize=100_000,
+ )
+ create_metadata_column(
+ slot,
+ "source_sizes",
+ data=sizes,
+ dtype=np.int64,
+ chunkSize=100_000,
+ )
+ scores = create_numeric_array(
+ slot,
+ "scores",
+ normed_array_spec(
+ n_cells,
+ n_sources,
+ profile=resolve_storage_profile(slot.store),
+ ),
+ )
+
+ def checked_batches() -> Iterator[np.ndarray]:
+ for batch in score_batches:
+ values = np.asarray(batch, dtype=np.float64)
+ if values.ndim != 2 or values.shape[1] != n_sources:
+ raise ValueError("Enrichment score batch has an invalid shape")
+ if not np.isfinite(values).all():
+ raise ValueError(
+ "Enrichment score batch contains non-finite values"
+ )
+ yield values
+
+ written = write_dense_from_row_batches(
+ scores,
+ checked_batches(),
+ dtype=np.float32,
+ msg=f"Writing {attrs['method']} enrichment",
+ )
+ if written != n_cells:
+ raise ValueError(
+ f"Enrichment writer produced {written} rows, expected {n_cells}"
+ )
+ slot.attrs["complete"] = True
+ except Exception:
+ slot.attrs["complete"] = False
+ raise
+
+
+def _enrichment_artifact_matches(
+ group: zarr.Group,
+ *,
+ attrs: dict[str, Any],
+ cell_index: np.ndarray,
+ matched_feature_index: np.ndarray,
+ source_names: np.ndarray,
+ source_sizes: np.ndarray,
+ rank_feature_index: np.ndarray | None,
+) -> bool:
+ for key, expected in attrs.items():
+ if key in {"cell_key", "complete", "feat_key"}:
+ continue
+ if group.attrs.get(key) != expected:
+ return False
+ try:
+ scores = as_zarr_array(group["scores"], name="scores")
+ except (KeyError, TypeError):
+ return False
+ if (
+ scores.ndim != 2
+ or scores.shape != (len(cell_index), len(source_names))
+ or np.dtype(scores.dtype) != np.dtype(np.float32)
+ ):
+ return False
+ expected_arrays = {
+ "cell_index": np.asarray(cell_index, dtype=np.int64),
+ "matched_feature_index": np.asarray(
+ matched_feature_index,
+ dtype=np.int64,
+ ),
+ "source_names": np.asarray(source_names).astype(str),
+ "source_sizes": np.asarray(source_sizes, dtype=np.int64),
+ }
+ if rank_feature_index is not None:
+ expected_arrays["rank_feature_index"] = np.asarray(
+ rank_feature_index,
+ dtype=np.int64,
+ )
+ try:
+ for name, expected in expected_arrays.items():
+ stored = np.asarray(as_zarr_array(group[name], name=name)[:])
+ if name == "source_names":
+ stored = stored.astype(str)
+ if not np.array_equal(stored, expected):
+ return False
+ except (KeyError, TypeError, ValueError):
+ return False
+ if rank_feature_index is None and "rank_feature_index" in group:
+ return False
+ return True
+
+
+def _validate_enrichment_artifact_provenance(
+ root: zarr.Group,
+ assay: Assay,
+ status: ArtifactStatus,
+ group: zarr.Group,
+ method: str,
+ cell_key: str,
+ feat_key: str,
+) -> tuple[str, str]:
+ parameters = status.parameters or {}
+ inputs = status.inputs or {}
+ execution = status.execution_options or {}
+ expected_parameters: dict[str, Any] = {
+ "algorithm_version": group.attrs["algorithm_version"],
+ "tmin": group.attrs["tmin"],
+ }
+ if method == "waggr":
+ normalization_method = parameters.get("normalization_method")
+ if normalization_method != callable_identity(assay.normMethod):
+ raise ValueError("Enrichment artifact normalization provenance is invalid")
+ expected_parameters.update(
+ {
+ "log_transform": group.attrs["log_transform"],
+ "mode": group.attrs["waggr_mode"],
+ "size_factor": group.attrs["size_factor"],
+ }
+ )
+ else:
+ expected_parameters.update(
+ {
+ "n_up": group.attrs["n_up"],
+ "tie_seed": group.attrs["tie_seed"],
+ }
+ )
+ if any(parameters.get(key) != value for key, value in expected_parameters.items()):
+ raise ValueError("Enrichment artifact parameters do not match its metadata")
+ if inputs.get("network_digest") != group.attrs["network_digest"]:
+ raise ValueError(
+ "Enrichment artifact network input does not match its metadata"
+ )
+ if (
+ execution.get("cell_key") != group.attrs["cell_key"]
+ or execution.get("feat_key") != group.attrs["feat_key"]
+ ):
+ raise ValueError("Enrichment artifact selection keys do not match its metadata")
+
+ resolved_columns: dict[str, str] = {}
+ for (
+ input_name,
+ expected_kind,
+ expected_scope,
+ digest_name,
+ source_column,
+ metadata,
+ ) in (
+ (
+ "cell_selection",
+ "cell_selection",
+ "datastore",
+ "cell_digest",
+ cell_key,
+ assay.cells,
+ ),
+ (
+ "feature_selection",
+ "feature_selection",
+ "assay",
+ "feature_digest",
+ feat_key,
+ assay.feats,
+ ),
+ ):
+ raw_ref = inputs.get(input_name)
+ if not isinstance(raw_ref, dict):
+ raise ValueError(
+ f"Enrichment artifact is missing {input_name!r} provenance"
+ )
+ selection_ref = ArtifactRef.from_dict(raw_ref)
+ selection_status = inspect_artifact(root, selection_ref)
+ if (
+ selection_ref.kind != expected_kind
+ or selection_ref.scope != expected_scope
+ or (expected_scope == "assay" and selection_ref.assay != status.ref.assay)
+ or not selection_status.exists
+ or not selection_status.complete
+ ):
+ raise ValueError(f"Enrichment artifact has an invalid {input_name!r} input")
+ selection_group = as_zarr_group(
+ root[selection_status.path],
+ name=selection_status.path,
+ )
+ selection_values = np.asarray(
+ as_zarr_array(selection_group["values"], name="values")[:]
+ )
+ if selection_values.ndim != 1 or selection_values.dtype != np.dtype(bool):
+ raise ValueError(f"Enrichment artifact has malformed {input_name!r} values")
+ candidates = [source_column]
+ candidates.extend(
+ sorted(column for column in metadata.columns if column != source_column)
+ )
+ resolved_column = None
+ for candidate in candidates:
+ try:
+ current_values = np.asarray(metadata.fetch_all(candidate))
+ except (KeyError, TypeError, ValueError):
+ continue
+ if (
+ current_values.ndim == 1
+ and current_values.dtype == np.dtype(bool)
+ and current_values.shape == selection_values.shape
+ and np.array_equal(current_values, selection_values)
+ ):
+ resolved_column = candidate
+ break
+ if resolved_column is None:
+ raise ValueError(
+ f"Enrichment selection column {source_column!r} "
+ "is unavailable and no equivalent column exists"
+ )
+ resolved_columns[input_name] = resolved_column
+ selected_index = np.flatnonzero(selection_values).astype(np.int64)
+ if array_digest(selected_index) != group.attrs[digest_name]:
+ raise ValueError(
+ f"Enrichment artifact {input_name!r} does not match its metadata"
+ )
+ return (
+ resolved_columns["cell_selection"],
+ resolved_columns["feature_selection"],
+ )
+
+
+def _load_enrichment_result(
+ assay: Assay,
+ *,
+ label: str,
+ sources: Sequence[str] | None,
+ artifact_root: zarr.Group | None = None,
+) -> EnrichmentResult:
+ from ...matrix import ChunkedArray
+
+ artifact_entry = _enrichment_artifact_entry(assay, label)
+ ref = None if artifact_entry is None else artifact_entry[0]
+ indexed_cell_key = None if artifact_entry is None else artifact_entry[1]
+ indexed_feat_key = None if artifact_entry is None else artifact_entry[2]
+ artifact_backed = ref is not None
+ artifact_operation: str | None = None
+ artifact_status: ArtifactStatus | None = None
+ resolved_cell_key: str | None = indexed_cell_key
+ resolved_feat_key: str | None = indexed_feat_key
+ if ref is not None:
+ if artifact_root is None:
+ raise ValueError("Artifact root is required for indexed enrichment")
+ status = inspect_artifact(artifact_root, ref)
+ artifact_status = status
+ if (
+ ref.kind != "enrichment_scores"
+ or ref.scope != "assay"
+ or ref.assay != assay.name
+ or not status.complete
+ ):
+ raise ValueError(f"Enrichment label {label!r} has an invalid artifact")
+ artifact_operation = status.operation
+ slot = as_zarr_group(
+ artifact_root[status.path],
+ name=status.path,
+ )
+ root_path = str(getattr(artifact_root, "path", "")).strip("/")
+ storage_path = f"{root_path}/{status.path}" if root_path else status.path
+ else:
+ storage_path = f"{getattr(assay.z, 'path', assay.name)}/enrichment/{label}"
+ if "enrichment" not in assay.z:
+ raise KeyError(f"Enrichment label {label!r} was not found for {assay.name}")
+ enrichment_group = as_zarr_group(
+ assay.z["enrichment"],
+ name=f"{assay.name}/enrichment",
+ )
+ if label not in enrichment_group:
+ raise KeyError(f"Enrichment label {label!r} was not found for {assay.name}")
+ label_group = as_zarr_group(enrichment_group[label], name=storage_path)
+ slot = _resolve_enrichment_slot(label_group, label=label)
+ if slot.attrs.get("complete") is not True:
+ raise ValueError(f"Enrichment slot {label!r} is incomplete")
+ method = str(slot.attrs.get("method", ""))
+ if method not in {"waggr", "aucell"}:
+ raise ValueError(f"Enrichment slot {label!r} has an unknown method")
+ if artifact_backed and artifact_operation != f"run_{method}":
+ raise ValueError(
+ f"Enrichment label {label!r} has a mismatched artifact operation"
+ )
+ required_attrs = {
+ "algorithm_version",
+ "cell_digest",
+ "cell_key",
+ "feature_digest",
+ "feat_key",
+ "network_digest",
+ "tmin",
+ }
+ if not artifact_backed:
+ required_attrs.add("execution_digest")
+ if not required_attrs.issubset(slot.attrs):
+ raise ValueError(f"Enrichment slot {label!r} is missing required metadata")
+ method_attrs = (
+ {"log_transform", "normalization", "size_factor", "waggr_mode"}
+ if method == "waggr"
+ else {"n_up", "tie_seed"}
+ )
+ if not method_attrs.issubset(slot.attrs):
+ raise ValueError(f"Enrichment slot {label!r} is missing method metadata")
+ algorithm_version = slot.attrs["algorithm_version"]
+ if (
+ isinstance(algorithm_version, bool)
+ or not isinstance(algorithm_version, (int, np.integer))
+ or int(algorithm_version) != 1
+ ):
+ raise ValueError(f"Enrichment slot {label!r} has an unsupported algorithm")
+ tmin = slot.attrs["tmin"]
+ if (
+ isinstance(tmin, bool)
+ or not isinstance(tmin, (int, np.integer))
+ or int(tmin) < 1
+ ):
+ raise ValueError(f"Enrichment slot {label!r} has invalid tmin metadata")
+ for digest_name in (
+ "cell_digest",
+ "feature_digest",
+ "network_digest",
+ ):
+ digest_value = slot.attrs[digest_name]
+ if not isinstance(digest_value, str) or not digest_value:
+ raise ValueError(
+ f"Enrichment slot {label!r} has invalid {digest_name} metadata"
+ )
+ if not artifact_backed:
+ execution_digest = slot.attrs["execution_digest"]
+ if not isinstance(execution_digest, str) or not execution_digest:
+ raise ValueError(
+ f"Enrichment slot {label!r} has invalid execution_digest metadata"
+ )
+ for key_name in ("cell_key", "feat_key"):
+ key_value = slot.attrs[key_name]
+ if not isinstance(key_value, str) or not key_value:
+ raise ValueError(
+ f"Enrichment slot {label!r} has invalid {key_name} metadata"
+ )
+ stored_n_up: int | None = None
+ if method == "waggr":
+ size_factor = slot.attrs["size_factor"]
+ if (
+ isinstance(size_factor, bool)
+ or not isinstance(size_factor, (int, float, np.integer, np.floating))
+ or not np.isfinite(float(size_factor))
+ or float(size_factor) <= 0
+ or slot.attrs["normalization"] != "norm_lib_size"
+ or slot.attrs["waggr_mode"] not in {"wmean", "wsum"}
+ or not isinstance(slot.attrs["log_transform"], bool)
+ ):
+ raise ValueError(f"WAGGR slot {label!r} has invalid method metadata")
+ else:
+ n_up = slot.attrs["n_up"]
+ tie_seed = slot.attrs["tie_seed"]
+ if (
+ isinstance(n_up, bool)
+ or not isinstance(n_up, (int, np.integer))
+ or int(n_up) < 2
+ or isinstance(tie_seed, bool)
+ or not isinstance(tie_seed, (int, np.integer))
+ or int(tie_seed) < 0
+ ):
+ raise ValueError(f"AUCell slot {label!r} has invalid method metadata")
+ stored_n_up = int(n_up)
+ if artifact_backed:
+ assert artifact_root is not None and artifact_status is not None
+ resolved_cell_key, resolved_feat_key = _validate_enrichment_artifact_provenance(
+ artifact_root,
+ assay,
+ artifact_status,
+ slot,
+ method,
+ (
+ indexed_cell_key
+ if indexed_cell_key is not None
+ else str(slot.attrs["cell_key"])
+ ),
+ (
+ indexed_feat_key
+ if indexed_feat_key is not None
+ else str(slot.attrs["feat_key"])
+ ),
+ )
+
+ required_arrays = {
+ "cell_index",
+ "matched_feature_index",
+ "scores",
+ "source_names",
+ "source_sizes",
+ }
+ if not required_arrays.issubset(slot):
+ raise ValueError(f"Enrichment slot {label!r} is missing required arrays")
+ if method == "aucell" and "rank_feature_index" not in slot:
+ raise ValueError(f"AUCell slot {label!r} is missing its ranking universe")
+ if method == "waggr" and "rank_feature_index" in slot:
+ raise ValueError(f"WAGGR slot {label!r} contains unexpected rank metadata")
+
+ scores = as_zarr_array(slot["scores"], name=f"{storage_path}/scores")
+ cell_node = as_zarr_array(slot["cell_index"], name="cell_index")
+ matched_node = as_zarr_array(
+ slot["matched_feature_index"],
+ name="matched_feature_index",
+ )
+ names_node = as_zarr_array(slot["source_names"], name="source_names")
+ sizes_node = as_zarr_array(slot["source_sizes"], name="source_sizes")
+ sidecars = (cell_node, matched_node, names_node, sizes_node)
+ if any(node.ndim != 1 for node in sidecars) or scores.ndim != 2:
+ raise ValueError(f"Enrichment slot {label!r} contains invalid array dimensions")
+ if np.dtype(scores.dtype) != np.dtype(np.float32):
+ raise ValueError(f"Enrichment slot {label!r} has an invalid score dtype")
+ if not np.issubdtype(cell_node.dtype, np.integer) or not np.issubdtype(
+ matched_node.dtype, np.integer
+ ):
+ raise ValueError(f"Enrichment slot {label!r} has invalid index dtypes")
+ if not np.issubdtype(sizes_node.dtype, np.integer):
+ raise ValueError(f"Enrichment slot {label!r} has invalid source sizes")
+
+ cell_index = np.asarray(cell_node[:], dtype=np.int64)
+ matched_feature_index = np.asarray(matched_node[:], dtype=np.int64)
+ source_names = np.asarray(names_node[:]).astype(str)
+ source_sizes = np.asarray(sizes_node[:], dtype=np.int64)
+ if scores.shape != (len(cell_index), len(source_names)):
+ raise ValueError(f"Enrichment slot {label!r} score shape is misaligned")
+ if len(source_names) == 0 or len(source_names) != len(source_sizes):
+ raise ValueError(f"Enrichment slot {label!r} source metadata is misaligned")
+ if np.unique(source_names).size != len(source_names):
+ raise ValueError(f"Enrichment slot {label!r} contains duplicate sources")
+ if np.any(source_names == ""):
+ raise ValueError(f"Enrichment slot {label!r} contains empty source names")
+ if np.any(source_sizes <= 0):
+ raise ValueError(f"Enrichment slot {label!r} contains invalid source sizes")
+ if np.any(cell_index < 0) or np.unique(cell_index).size != len(cell_index):
+ raise ValueError(f"Enrichment slot {label!r} contains duplicate cell indices")
+ if array_digest(cell_index) != slot.attrs["cell_digest"]:
+ raise ValueError(f"Enrichment slot {label!r} has a mismatched cell digest")
+ if (
+ len(matched_feature_index) == 0
+ or np.any(matched_feature_index < 0)
+ or not np.array_equal(matched_feature_index, np.unique(matched_feature_index))
+ ):
+ raise ValueError(f"Enrichment slot {label!r} has invalid matched features")
+ if method == "aucell":
+ rank_node = as_zarr_array(
+ slot["rank_feature_index"],
+ name="rank_feature_index",
+ )
+ if rank_node.ndim != 1 or not np.issubdtype(rank_node.dtype, np.integer):
+ raise ValueError(f"AUCell slot {label!r} has invalid rank features")
+ rank_feature_index = np.asarray(rank_node[:], dtype=np.int64)
+ if (
+ len(rank_feature_index) < 2
+ or np.any(rank_feature_index < 0)
+ or np.unique(rank_feature_index).size != len(rank_feature_index)
+ or stored_n_up is None
+ or stored_n_up > len(rank_feature_index)
+ ):
+ raise ValueError(f"AUCell slot {label!r} has invalid rank features")
+ if array_digest(np.sort(rank_feature_index)) != slot.attrs["feature_digest"]:
+ raise ValueError(f"AUCell slot {label!r} has a mismatched feature digest")
+ if not np.isin(matched_feature_index, rank_feature_index).all():
+ raise ValueError(f"AUCell slot {label!r} has unmatched network features")
+
+ data = ChunkedArray(scores, nthreads=assay.nthreads)
+ if sources is not None:
+ if isinstance(sources, str):
+ raise TypeError("sources must be a sequence of source names, not a string")
+ requested = list(sources)
+ if not requested:
+ raise ValueError("sources must be non-empty when provided")
+ if not all(isinstance(source, str) for source in requested):
+ raise TypeError("sources must contain only strings")
+ if len(set(requested)) != len(requested):
+ raise ValueError("sources contains duplicate names")
+ source_positions = {
+ source: index for index, source in enumerate(source_names.tolist())
+ }
+ missing = [source for source in requested if source not in source_positions]
+ if missing:
+ raise KeyError("Enrichment sources not found: " + ", ".join(missing))
+ positions = np.asarray(
+ [source_positions[source] for source in requested],
+ dtype=np.int64,
+ )
+ data = data[:, positions]
+ source_names = source_names[positions]
+ source_sizes = source_sizes[positions]
+
+ return EnrichmentResult(
+ data=data,
+ source_names=source_names,
+ source_sizes=source_sizes,
+ cell_index=cell_index,
+ label=label,
+ storage_path=storage_path,
+ assay=assay.name,
+ cell_key=(
+ resolved_cell_key
+ if resolved_cell_key is not None
+ else str(slot.attrs["cell_key"])
+ ),
+ feature_key=(
+ resolved_feat_key
+ if resolved_feat_key is not None
+ else str(slot.attrs["feat_key"])
+ ),
+ method=method,
+ )
diff --git a/scarf/datastore/_operations/features.py b/scarf/datastore/_operations/features.py
new file mode 100644
index 00000000..2898b015
--- /dev/null
+++ b/scarf/datastore/_operations/features.py
@@ -0,0 +1,1853 @@
+import time
+from collections.abc import Iterator, Sequence
+from contextlib import contextmanager
+from typing import TYPE_CHECKING, Any, Literal
+
+import numpy as np
+import pandas as pd
+import zarr
+from numpy.typing import NDArray
+
+from ...features.variability import DEFAULT_HVG_BLACKLIST, HVG_UBIQUITOUS_SLACK
+from ...storage.artifact_writer import (
+ ArrayRequirement,
+ AttributeRequirement,
+ finish_artifact,
+ start_artifact,
+)
+from ...storage.artifacts import (
+ ArtifactRef,
+ artifact_path,
+ callable_identity,
+ inspect_artifact,
+)
+from ...storage.selections import resolve_selection_artifact
+from ...storage.types import as_zarr_array, as_zarr_group
+from ...assay import RNAassay, lib_size_feature_stream_eligible
+from ...features.enrichment.results import EnrichmentResult
+from ...features.markers.table import (
+ MARKER_ADJUSTMENT_METHOD,
+ MARKER_ADJUSTMENT_SCOPE,
+ MARKER_ALTERNATIVE,
+ MARKER_CONTINUITY_CORRECTION,
+ MARKER_METHOD,
+ MARKER_STAT_COLUMNS,
+ MARKER_TIE_CORRECTION,
+ _validate_marker_slot,
+ load_marker_table,
+)
+from ...metadata.arguments import AucellArguments, MarkerTableArguments, WaggrArguments
+from ...metadata.artifacts import (
+ categorical_display,
+ feature_column_display,
+ link_feature_data_column,
+)
+from ...utils.arrays import array_digest
+from ...utils.compute import controlled_compute
+from ...utils.logging import logger
+from ...utils.progress import iter_progress
+from .enrichment_store import (
+ _ENRICHMENT_LAYOUT,
+ _EnrichmentScorer,
+ _enrichment_artifact_matches,
+ _enrichment_artifact_ref,
+ _execution_digest,
+ _legacy_enrichment_slot,
+ _load_enrichment_result,
+ _publish_enrichment_artifact,
+ _validate_enrichment_label,
+ _write_enrichment_slot,
+)
+
+if TYPE_CHECKING:
+ from ..mapping_datastore import MappingDatastore as _FeatureOperationsBase
+else:
+ _FeatureOperationsBase = object
+
+_MARKER_STAT_COLUMNS = MARKER_STAT_COLUMNS
+_MARKER_OUT_COLUMNS = ("feature_index", *_MARKER_STAT_COLUMNS)
+
+
+def _shared_marker_feature_index(markers: dict[Any, pd.DataFrame]) -> np.ndarray:
+ shared: np.ndarray | None = None
+ populated_names: set[str] = set()
+ for cluster_id, vals in markers.items():
+ if len(vals) == 0:
+ continue
+ group_name = str(cluster_id)
+ if group_name in populated_names:
+ raise ValueError("Marker group labels must remain unique as strings")
+ populated_names.add(group_name)
+ raw_index = np.asarray(vals.index.values)
+ if raw_index.ndim != 1 or raw_index.dtype.kind not in {"i", "u"}:
+ raise ValueError(
+ "Marker feature indices must use a one-dimensional integer index"
+ )
+ if not vals.index.is_unique:
+ raise ValueError("Marker feature indices must be unique within each group")
+ index = raw_index.astype(np.int64, copy=False)
+ if (index < 0).any() or (index > np.iinfo(np.int32).max).any():
+ raise ValueError("Marker feature indices must fit non-negative int32")
+ ordered = np.sort(index)
+ if shared is None:
+ shared = ordered
+ elif not np.array_equal(ordered, shared):
+ raise ValueError("Marker groups must contain identical feature index sets")
+ if shared is None:
+ raise ValueError("Cannot save empty marker results")
+ return shared.astype(np.int32)
+
+
+def _marker_stats_matrix(vals: pd.DataFrame, feature_index: np.ndarray) -> np.ndarray:
+ aligned = vals.reindex(feature_index)
+ stats = np.asarray(
+ aligned.loc[:, list(_MARKER_STAT_COLUMNS)].to_numpy(dtype=np.float64)
+ )
+ if not np.isfinite(stats).all():
+ raise ValueError("Marker statistics must all be finite")
+ return stats
+
+
+def _write_compact_marker_stats(
+ cluster_group: zarr.Group,
+ stats: np.ndarray,
+) -> None:
+ from ...storage.arrays import create_zarr_dataset
+
+ n_features = int(stats.shape[0])
+ n_stats = int(stats.shape[1])
+ arr = create_zarr_dataset(
+ cluster_group,
+ "stats",
+ (n_features, n_stats),
+ "float64",
+ (n_features, n_stats),
+ )
+ arr[:] = stats
+
+
+def _load_marker_cluster_frame(
+ slot_group: zarr.Group,
+ cluster_group: zarr.Group,
+ feature_names: np.ndarray,
+ *,
+ group_id: Any,
+ feature_ids: np.ndarray | None = None,
+) -> pd.DataFrame:
+ """Thin wrapper around the shared version-aware marker reader."""
+ return load_marker_table(
+ slot_group,
+ cluster_group,
+ feature_names,
+ group_id=group_id,
+ feature_ids=feature_ids,
+ )
+
+
+def _group_assignment_digest(values: np.ndarray) -> str:
+ return array_digest(np.asarray(values).astype(str))
+
+
+class _FeatureOperationsMixin(_FeatureOperationsBase):
+ def set_hvgs(
+ self,
+ *,
+ from_assay: str | None = None,
+ cell_key: str,
+ mask: np.ndarray | None = None,
+ feature_indexes: Sequence[int] | None = None,
+ hvg_key_name: str = "hvgs",
+ n_bins: int = 200,
+ lowess_frac: float = 0.1,
+ bin_strategy: Literal["fixed", "adaptive"] = "adaptive",
+ blacklist: str | None = None,
+ blacklist_exclusions: str | None = None,
+ blacklist_indexes: Sequence[int] | None = None,
+ invalidate_cache: bool = False,
+ ) -> str:
+ """Install a supplied HVG selection on an RNA assay."""
+ assay = self._get_assay(from_assay)
+ if not isinstance(assay, RNAassay):
+ raise TypeError(
+ "set_hvgs can only be applied to an RNAassay; "
+ f"received {type(assay).__name__}"
+ )
+ expected_key = f"{cell_key}__{hvg_key_name}"
+ storage_backed = (
+ hasattr(self, "z") and hasattr(self, "cells") and hasattr(assay, "z")
+ )
+ preserved_display = (
+ feature_column_display(assay.z, expected_key) if storage_backed else None
+ )
+ stored_key = assay.set_hvgs(
+ cell_key,
+ mask=mask,
+ feature_indexes=feature_indexes,
+ hvg_key_name=hvg_key_name,
+ n_bins=n_bins,
+ lowess_frac=lowess_frac,
+ bin_strategy=bin_strategy,
+ blacklist=blacklist,
+ blacklist_exclusions=blacklist_exclusions,
+ blacklist_indexes=blacklist_indexes,
+ )
+ if not storage_backed:
+ return stored_key
+ cell_selection = self._linked_cell_selection(cell_key)
+ if cell_selection is None:
+ cell_selection = self._record_cell_selection(
+ column=cell_key,
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ )
+ feature_selection = resolve_selection_artifact(
+ self.zw,
+ scope="assay",
+ assay=assay.name,
+ kind="feature_selection",
+ values=np.asarray(assay.feats.fetch_all(stored_key)),
+ row_ids=np.asarray(assay.feats.fetch_all("ids")),
+ operation="set_hvgs",
+ parameters={
+ "n_bins": n_bins,
+ "lowess_frac": lowess_frac,
+ "bin_strategy": bin_strategy,
+ "blacklist": blacklist,
+ "blacklist_exclusions": blacklist_exclusions,
+ "blacklist_indexes": list(blacklist_indexes)
+ if blacklist_indexes is not None
+ else None,
+ },
+ inputs={"cell_selection": cell_selection},
+ source_column=stored_key,
+ invalidate_cache=invalidate_cache,
+ )
+ link_feature_data_column(
+ assay.z,
+ stored_key,
+ feature_selection,
+ value_name="values",
+ default_display=categorical_display(
+ np.asarray(assay.feats.fetch_all(stored_key))
+ ),
+ preserved_display=preserved_display,
+ )
+ return stored_key
+
+ def mark_hvgs(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ min_cells: int = 20,
+ top_n: int = 2000,
+ min_var: float = -np.inf,
+ max_var: float = np.inf,
+ min_mean: float = -np.inf,
+ max_mean: float = np.inf,
+ n_bins: int = 200,
+ lowess_frac: float = 0.1,
+ blacklist: str = DEFAULT_HVG_BLACKLIST,
+ keep_bounds: bool = False,
+ show_plot: bool = True,
+ hvg_key_name: str = "hvgs",
+ max_cells: float | None = None,
+ bin_strategy: Literal["fixed", "adaptive"] = "adaptive",
+ invalidate_cache: bool = False,
+ **plot_kwargs: Any,
+ ) -> None:
+ """Identify and mark genes as highly variable genes (HVGs). This is a
+ critical and required feature selection step and is only applicable to
+ RNAassay type of assays.
+
+ Args:
+ from_assay: Assay to use for graph creation. If no value is provided then `defaultAssay` will be used
+ cell_key: Cells to use for HVG selection. By default, all cells with True value in 'I' will be used.
+ The provided value for `cell_key` should be a column in cell metadata table with boolean values.
+ min_cells: Minimum number of cells where a gene should have non-zero expression values for it to be
+ considered a candidate for HVG selection. Large values for this parameter might make it difficult
+ to identify rare populations of cells. Very small values might lead to a higher signal-to-noise
+ ratio in the selected features. (Default: 20)
+ max_cells: Maximum number of cells where a gene should have non-zero expression values for it to be
+ considered a candidate for HVG selection. When omitted, genes missing in at most
+ ``HVG_UBIQUITOUS_SLACK`` selected cells are excluded (``n_selected - 20``). Pass
+ ``inf`` to disable the ubiquitous-gene filter.
+ top_n: Number of top most variable genes to be set as HVGs. This value is ignored if a value is provided
+ for `min_var` parameter. (Default: 2000)
+ min_var: Minimum variance threshold for HVG selection. (Default: -Infinity)
+ max_var: Maximum variance threshold for HVG selection. (Default: Infinity)
+ min_mean: Minimum mean value of expression threshold for HVG selection. (Default: -Infinity)
+ max_mean: Maximum mean value of expression threshold for HVG selection. (Default: Infinity)
+ n_bins: Number of bins into which the mean expression is binned. (Default: 200)
+ lowess_frac: Between 0 and 1. The fraction of the data used when estimating the fit between mean and
+ variance. This is same as `frac` in statsmodels.nonparametric.smoothers_lowess.lowess
+ (Default: 0.1)
+ bin_strategy: Strategy used to construct bins and variance anchors. (Default: 'adaptive')
+ blacklist: Regex of gene names to exclude from HVGs. Matching is case-insensitive.
+ Default excludes mitochondrial, ribosomal, cell-cycle (``CCN*``), HLA/H2, histone,
+ and common human sex-linked genes (``XIST``, Y-chromosome markers).
+ keep_bounds: If True, retain upper cell-count and expression-statistic bounds.
+ The ``min_cells`` boundary is always inclusive. (Default: False)
+ show_plot: If True then a diagnostic scatter plot is shown with HVGs highlighted. (Default: True)
+ hvg_key_name: Base label for HVGs in the features metadata column. The value for
+ 'cell_key' parameter is prepended to this value. (Default value: 'hvgs')
+ plot_kwargs: Named parameters forwarded to ``plotting.highly_variable_features``
+ (``figsize``, ``label_size``, ``point_sizes``, ``colormaps``).
+
+ Returns:
+ None
+ """
+
+ if cell_key is None:
+ cell_key = "I"
+ assay = self._get_assay(from_assay)
+ if type(assay) != RNAassay: # noqa: E721
+ raise TypeError(
+ f"ERROR: This method of feature selection can only be applied to RNAassay type of assay. "
+ f"The provided assay is {type(assay)} type"
+ )
+ n_selected = int(np.asarray(self.cells.fetch_all(cell_key), dtype=bool).sum())
+ if max_cells is None:
+ candidate_max = n_selected - HVG_UBIQUITOUS_SLACK
+ if candidate_max <= min_cells:
+ max_cells_int: int | float = np.inf
+ logger.info(
+ "Skipping ubiquitous-gene HVG filter because too few cells are "
+ f"selected ({n_selected}); need more than "
+ f"{min_cells + HVG_UBIQUITOUS_SLACK} cells to apply "
+ f"max_cells = n_selected - {HVG_UBIQUITOUS_SLACK}."
+ )
+ else:
+ max_cells_int = int(candidate_max)
+ logger.debug(
+ f"Setting `max_cells` to {max_cells_int} "
+ f"(n_selected - {HVG_UBIQUITOUS_SLACK}). Genes detected in at "
+ "least this many cells are excluded as ubiquitous."
+ )
+ elif max_cells == np.inf:
+ max_cells_int = np.inf
+ else:
+ max_cells_int = int(max_cells)
+ stored_key = f"{cell_key}__{hvg_key_name}"
+ preserved_display = feature_column_display(
+ assay.z,
+ stored_key,
+ )
+ assay.mark_hvgs(
+ cell_key=cell_key,
+ min_cells=min_cells,
+ max_cells=max_cells_int,
+ top_n=top_n,
+ min_var=min_var,
+ max_var=max_var,
+ min_mean=min_mean,
+ max_mean=max_mean,
+ n_bins=n_bins,
+ lowess_frac=lowess_frac,
+ bin_strategy=bin_strategy,
+ blacklist=blacklist,
+ hvg_key_name=hvg_key_name,
+ keep_bounds=keep_bounds,
+ show_plot=show_plot,
+ **plot_kwargs,
+ )
+ cell_selection = self._linked_cell_selection(cell_key)
+ if cell_selection is None:
+ cell_selection = self._record_cell_selection(
+ column=cell_key,
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ )
+ feature_selection = resolve_selection_artifact(
+ self.zw,
+ scope="assay",
+ assay=assay.name,
+ kind="feature_selection",
+ values=np.asarray(assay.feats.fetch_all(stored_key)),
+ row_ids=np.asarray(assay.feats.fetch_all("ids")),
+ operation="mark_hvgs",
+ parameters={
+ "min_cells": min_cells,
+ "max_cells": max_cells_int,
+ "top_n": top_n,
+ "min_var": min_var,
+ "max_var": max_var,
+ "min_mean": min_mean,
+ "max_mean": max_mean,
+ "n_bins": n_bins,
+ "lowess_frac": lowess_frac,
+ "blacklist": blacklist,
+ "keep_bounds": keep_bounds,
+ "bin_strategy": bin_strategy,
+ },
+ inputs={"cell_selection": cell_selection},
+ source_column=stored_key,
+ invalidate_cache=invalidate_cache,
+ )
+ link_feature_data_column(
+ assay.z,
+ stored_key,
+ feature_selection,
+ value_name="values",
+ default_display=categorical_display(
+ np.asarray(assay.feats.fetch_all(stored_key))
+ ),
+ preserved_display=preserved_display,
+ )
+
+ def _run_enrichment(
+ self,
+ *,
+ assay: RNAassay,
+ label: str,
+ cell_key: str,
+ feat_key: str,
+ overwrite: bool,
+ invalidate_cache: bool,
+ scorer: _EnrichmentScorer,
+ ) -> EnrichmentResult:
+ """Shared artifact plan, reuse, write, and selection path for enrichment."""
+ cell_index = scorer.cell_index
+ cell_digest = array_digest(cell_index)
+ feature_digest = array_digest(scorer.feature_index)
+ attrs: dict[str, Any] = {
+ "algorithm_version": scorer.algorithm_version,
+ "cell_digest": cell_digest,
+ "cell_key": cell_key,
+ "complete": False,
+ "feature_digest": feature_digest,
+ "feat_key": feat_key,
+ "layout": _ENRICHMENT_LAYOUT,
+ "method": scorer.method,
+ **scorer.method_payload,
+ }
+ execution = _execution_digest(
+ {
+ "algorithm_version": scorer.algorithm_version,
+ "cell_digest": cell_digest,
+ "cell_key": cell_key,
+ "feature_digest": feature_digest,
+ "feat_key": feat_key,
+ "method": scorer.method,
+ **scorer.method_payload,
+ }
+ )
+ required_arrays = (
+ ArrayRequirement(
+ "scores",
+ shape=(len(cell_index), len(scorer.source_names)),
+ dtype_kind="f",
+ ),
+ ArrayRequirement(
+ "cell_index",
+ shape=(len(cell_index),),
+ dtype_kind="i",
+ ),
+ ArrayRequirement("matched_feature_index", dtype_kind="i"),
+ *scorer.extra_required_arrays,
+ ArrayRequirement(
+ "source_names",
+ shape=(len(scorer.source_names),),
+ ),
+ ArrayRequirement(
+ "source_sizes",
+ shape=(len(scorer.source_names),),
+ dtype_kind="i",
+ ),
+ )
+ planned = scorer.arguments.plan(
+ self.zw,
+ scope="assay",
+ assay=assay.name,
+ invalidate_cache=invalidate_cache,
+ required_arrays=required_arrays,
+ required_attributes=(
+ "algorithm_version",
+ "method",
+ "network_digest",
+ ),
+ reuse_validator=lambda _ref, group: _enrichment_artifact_matches(
+ group,
+ attrs=attrs,
+ cell_index=cell_index,
+ matched_feature_index=scorer.matched_feature_index,
+ source_names=scorer.source_names,
+ source_sizes=scorer.source_sizes,
+ rank_feature_index=scorer.rank_feature_index,
+ ),
+ )
+ existing_ref = _enrichment_artifact_ref(assay, label)
+ legacy_slot = (
+ _legacy_enrichment_slot(assay, label) if existing_ref is None else None
+ )
+ if existing_ref is not None:
+ existing_status = inspect_artifact(self.zw, existing_ref)
+ if (
+ existing_ref.kind != "enrichment_scores"
+ or existing_ref.scope != "assay"
+ or existing_ref.assay != assay.name
+ or not existing_status.complete
+ ):
+ raise ValueError(f"Enrichment label {label!r} has an invalid artifact")
+ if existing_status.provenance != planned.provenance and not overwrite:
+ raise ValueError(
+ f"Enrichment label {label!r} already contains a different "
+ f"{existing_status.operation!r} execution; pass overwrite=True "
+ "to replace it"
+ )
+ elif legacy_slot is not None:
+ same_legacy_execution = (
+ str(legacy_slot.attrs.get("execution_digest", "")) == execution
+ )
+ if (
+ legacy_slot.attrs.get("complete") is True
+ and same_legacy_execution
+ and not invalidate_cache
+ ):
+ return _load_enrichment_result(
+ assay,
+ label=label,
+ sources=None,
+ artifact_root=self.zw,
+ )
+ if (
+ legacy_slot.attrs.get("complete") is True
+ and not same_legacy_execution
+ and not overwrite
+ ):
+ method = legacy_slot.attrs.get("method", "unknown")
+ raise ValueError(
+ f"Enrichment label {label!r} already contains a different "
+ f"{method!r} execution; pass overwrite=True to replace it"
+ )
+ if planned.reused:
+ _publish_enrichment_artifact(
+ assay,
+ label,
+ planned.ref,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ )
+ return _load_enrichment_result(
+ assay,
+ label=label,
+ sources=None,
+ artifact_root=self.zw,
+ )
+ slot = start_artifact(self.zw, planned)
+ with scorer.write_context():
+ _write_enrichment_slot(
+ slot,
+ attrs=attrs,
+ score_batches=scorer.score_batches(),
+ n_cells=len(cell_index),
+ source_names=scorer.source_names,
+ source_sizes=scorer.source_sizes,
+ cell_index=cell_index,
+ matched_feature_index=scorer.matched_feature_index,
+ rank_feature_index=scorer.rank_feature_index,
+ )
+ finish_artifact(slot, planned)
+ _publish_enrichment_artifact(
+ assay,
+ label,
+ planned.ref,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ )
+ return _load_enrichment_result(
+ assay,
+ label=label,
+ sources=None,
+ artifact_root=self.zw,
+ )
+
+ def _prepare_enrichment_assay(
+ self,
+ *,
+ display_name: str,
+ from_assay: str | None,
+ cell_key: str,
+ feat_key: str,
+ overwrite: bool,
+ ) -> tuple[RNAassay, np.ndarray, np.ndarray]:
+ if self.zarr_mode != "r+":
+ raise ValueError(
+ f"{display_name} requires a DataStore opened with zarr_mode='r+'"
+ )
+ if not isinstance(overwrite, bool):
+ raise TypeError("overwrite must be a boolean")
+ assay = self._get_assay(from_assay)
+ if not isinstance(assay, RNAassay):
+ raise TypeError(f"{display_name} can only be run on an RNAassay")
+ cell_index, feature_index = assay._get_cell_feat_idx(cell_key, feat_key)
+ cell_index = np.asarray(cell_index, dtype=np.int64)
+ feature_index = np.asarray(feature_index, dtype=np.int64)
+ if len(cell_index) == 0:
+ raise ValueError(f"Cell key {cell_key!r} selects no active cells")
+ return assay, cell_index, feature_index
+
+ def _enrichment_feature_selection(
+ self, assay: RNAassay, feat_key: str
+ ) -> ArtifactRef:
+ feature_values = np.asarray(assay.feats.fetch_all(feat_key), dtype=bool)
+ return self._resolve_selection_input(
+ metadata_group=as_zarr_group(
+ assay.z["featureData"],
+ name="featureData",
+ ),
+ column=feat_key,
+ values=feature_values,
+ row_ids=np.asarray(assay.feats.fetch_all("ids")),
+ scope="assay",
+ kind="feature_selection",
+ assay=assay.name,
+ invalidate_cache=False,
+ )
+
+ def run_waggr(
+ self,
+ net: pd.DataFrame,
+ label: str,
+ *,
+ from_assay: str | None = None,
+ cell_key: str = "I",
+ feat_key: str = "I",
+ mode: Literal["wmean", "wsum"] = "wmean",
+ tmin: int = 5,
+ log_transform: bool = False,
+ overwrite: bool = False,
+ invalidate_cache: bool = False,
+ ) -> EnrichmentResult:
+ """Score weighted gene sets from streamed normalized RNA counts.
+
+ Targets are matched to active feature names without case sensitivity. Sources
+ with fewer than ``tmin`` matched non-zero edges are removed. Results are
+ written to the assay's enrichment group and returned lazily.
+
+ Args:
+ net: Network with ``source`` and ``target`` columns. An optional
+ ``weight`` column supplies signed numeric edge weights. Missing
+ weights default to one.
+ label: Name used to persist and retrieve the result.
+ from_assay: RNA assay to score. The default assay is used when omitted.
+ cell_key: Cell metadata key that selects score rows.
+ feat_key: Feature metadata key that defines the matching universe.
+ mode: ``"wmean"`` divides each weighted sum by the sum of absolute
+ source weights. ``"wsum"`` returns the weighted sum.
+ tmin: Minimum number of matched targets required per source.
+ log_transform: Apply ``log1p`` after library-size normalization.
+ overwrite: Replace a complete result with different execution metadata.
+ The previous result remains active until the replacement is complete.
+
+ Returns:
+ A persisted result with a lazy cells-by-sources score matrix.
+
+ Note:
+ Cache identity covers selections, method parameters, normalization, and
+ the prepared network. It assumes the stored count matrix is immutable.
+ """
+ from ...features.enrichment.net import prepare_network
+ from ...features.enrichment.waggr import (
+ WAGGR_ALGORITHM_VERSION,
+ build_waggr_model,
+ score_waggr_block,
+ )
+
+ _validate_enrichment_label(label)
+ if mode not in {"wmean", "wsum"}:
+ raise ValueError("mode must be 'wmean' or 'wsum'")
+ if not isinstance(log_transform, bool):
+ raise TypeError("log_transform must be a boolean")
+ assay, cell_index, feature_index = self._prepare_enrichment_assay(
+ display_name="WAGGR",
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ overwrite=overwrite,
+ )
+ feature_names = np.asarray(assay.feats.fetch_all("names"))[feature_index]
+ network = prepare_network(
+ net,
+ active_feature_names=feature_names,
+ active_feature_index=feature_index,
+ tmin=tmin,
+ weighted=True,
+ )
+ if not lib_size_feature_stream_eligible(assay):
+ raise ValueError(
+ "WAGGR requires the default norm_lib_size RNA normalization"
+ )
+ if assay.sf is None:
+ raise ValueError("WAGGR requires a finite positive size factor")
+ try:
+ size_factor = float(assay.sf)
+ except (TypeError, ValueError) as exc:
+ raise ValueError("WAGGR requires a finite positive size factor") from exc
+ if not np.isfinite(size_factor) or size_factor <= 0:
+ raise ValueError("WAGGR requires a finite positive size factor")
+
+ arguments = WaggrArguments(
+ cell_selection=self._ensure_cell_selection(cell_key),
+ feature_selection=self._enrichment_feature_selection(assay, feat_key),
+ network_digest=network.network_digest,
+ algorithm_version=WAGGR_ALGORITHM_VERSION,
+ mode=mode,
+ tmin=tmin,
+ log_transform=log_transform,
+ normalization_method=callable_identity(assay.normMethod),
+ size_factor=size_factor,
+ from_assay=assay.name,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ label=label,
+ overwrite=overwrite,
+ invalidate_cache=invalidate_cache,
+ )
+
+ def score_batches() -> Iterator[np.ndarray]:
+ cell_scalars = np.asarray(
+ assay.cells.fetch_all(f"{assay.name}_nCounts")[cell_index],
+ dtype=np.float64,
+ )
+ if not np.isfinite(cell_scalars).all() or np.any(cell_scalars < 0):
+ raise ValueError("WAGGR cell normalization scalars must be finite")
+ cell_scalars[cell_scalars == 0] = 1.0
+ model = build_waggr_model(network)
+ raw = assay.rawData[:, network.matched_feature_index][cell_index, :]
+ offset = 0
+ for raw_block in raw.stream_blocks(
+ nthreads=self.nthreads,
+ msg="Scoring WAGGR",
+ prefetch=1,
+ ):
+ block = np.asarray(raw_block, dtype=np.float64)
+ end = offset + block.shape[0]
+ if end > len(cell_scalars):
+ raise ValueError(
+ "WAGGR raw blocks exceed the active cell selection"
+ )
+ values = size_factor * block / cell_scalars[offset:end].reshape(-1, 1)
+ if log_transform:
+ values = np.log1p(values)
+ yield score_waggr_block(values, model, mode=mode)
+ offset = end
+ if offset != len(cell_scalars):
+ raise ValueError(
+ f"WAGGR streamed {offset} cells, expected {len(cell_scalars)}"
+ )
+
+ return self._run_enrichment(
+ assay=assay,
+ label=label,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ overwrite=overwrite,
+ invalidate_cache=invalidate_cache,
+ scorer=_EnrichmentScorer(
+ method="waggr",
+ algorithm_version=WAGGR_ALGORITHM_VERSION,
+ method_payload={
+ "log_transform": log_transform,
+ "network_digest": network.network_digest,
+ "normalization": "norm_lib_size",
+ "size_factor": size_factor,
+ "tmin": tmin,
+ "waggr_mode": mode,
+ },
+ arguments=arguments,
+ cell_index=cell_index,
+ feature_index=feature_index,
+ matched_feature_index=network.matched_feature_index,
+ source_names=network.source_names,
+ source_sizes=network.source_sizes,
+ rank_feature_index=None,
+ extra_required_arrays=(),
+ score_batches=score_batches,
+ ),
+ )
+
+ def run_aucell(
+ self,
+ net: pd.DataFrame,
+ label: str,
+ *,
+ from_assay: str | None = None,
+ cell_key: str = "I",
+ feat_key: str = "I",
+ tmin: int = 5,
+ n_up: int | None = None,
+ tie_seed: int = 0,
+ overwrite: bool = False,
+ invalidate_cache: bool = False,
+ ) -> EnrichmentResult:
+ """Score gene sets by recovery among each cell's top-ranked RNA features.
+
+ AUCell ranks every feature selected by ``feat_key`` from raw counts. Network
+ weights are ignored. Targets are matched without case sensitivity, then
+ sources with fewer than ``tmin`` matched targets are removed.
+
+ Args:
+ net: Network with ``source`` and ``target`` columns.
+ label: Name used to persist and retrieve the result.
+ from_assay: RNA assay to score. The default assay is used when omitted.
+ cell_key: Cell metadata key that selects score rows.
+ feat_key: Feature metadata key that defines the ranking universe.
+ tmin: Minimum number of matched targets required per source.
+ n_up: Number of top-ranked features used for recovery. When omitted,
+ five percent of the ranking universe is used, clipped to its valid
+ range.
+ tie_seed: Seed for the global feature permutation used to resolve ties.
+ overwrite: Replace a complete result with different execution metadata.
+ The previous result remains active until the replacement is complete.
+
+ Returns:
+ A persisted result with lazy scores in the interval from zero to one.
+
+ Note:
+ Cache identity covers selections, method parameters, and the prepared
+ network. It assumes the stored count matrix is immutable.
+ """
+ from ...features.enrichment.aucell import (
+ AUCELL_ALGORITHM_VERSION,
+ build_gene_set_index,
+ make_rank_permutation,
+ resolve_n_up,
+ score_aucell_block,
+ )
+ from ...features.enrichment.net import prepare_network
+
+ _validate_enrichment_label(label)
+ assay, cell_index, feature_index = self._prepare_enrichment_assay(
+ display_name="AUCell",
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ overwrite=overwrite,
+ )
+ resolved_n_up = resolve_n_up(len(feature_index), n_up)
+ feature_names = np.asarray(assay.feats.fetch_all("names"))[feature_index]
+ network = prepare_network(
+ net,
+ active_feature_names=feature_names,
+ active_feature_index=feature_index,
+ tmin=tmin,
+ weighted=False,
+ )
+ permutation = make_rank_permutation(len(feature_index), tie_seed)
+ rank_feature_index = feature_index[permutation]
+ sets = build_gene_set_index(network, rank_feature_index)
+
+ arguments = AucellArguments(
+ cell_selection=self._ensure_cell_selection(cell_key),
+ feature_selection=self._enrichment_feature_selection(assay, feat_key),
+ network_digest=network.network_digest,
+ algorithm_version=AUCELL_ALGORITHM_VERSION,
+ tmin=tmin,
+ n_up=resolved_n_up,
+ tie_seed=tie_seed,
+ from_assay=assay.name,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ label=label,
+ overwrite=overwrite,
+ invalidate_cache=invalidate_cache,
+ )
+
+ def score_batches() -> Iterator[np.ndarray]:
+ raw = assay.rawData[:, feature_index][cell_index, :]
+ offset = 0
+ for raw_block in raw.stream_blocks(
+ nthreads=self.nthreads,
+ msg="Scoring AUCell",
+ prefetch=1,
+ ):
+ scores = score_aucell_block(
+ np.asarray(raw_block),
+ permutation,
+ sets,
+ n_up=resolved_n_up,
+ )
+ offset += scores.shape[0]
+ if offset > len(cell_index):
+ raise ValueError(
+ "AUCell raw blocks exceed the active cell selection"
+ )
+ yield scores
+ if offset != len(cell_index):
+ raise ValueError(
+ f"AUCell streamed {offset} cells, expected {len(cell_index)}"
+ )
+
+ @contextmanager
+ def aucell_write_context() -> Iterator[None]:
+ import numba
+
+ previous_threads = numba.get_num_threads()
+ numba.set_num_threads(
+ min(max(1, int(self.nthreads)), numba.config.NUMBA_NUM_THREADS)
+ )
+ try:
+ yield
+ finally:
+ numba.set_num_threads(previous_threads)
+
+ return self._run_enrichment(
+ assay=assay,
+ label=label,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ overwrite=overwrite,
+ invalidate_cache=invalidate_cache,
+ scorer=_EnrichmentScorer(
+ method="aucell",
+ algorithm_version=AUCELL_ALGORITHM_VERSION,
+ method_payload={
+ "n_up": resolved_n_up,
+ "network_digest": network.network_digest,
+ "tie_seed": tie_seed,
+ "tmin": tmin,
+ },
+ arguments=arguments,
+ cell_index=cell_index,
+ feature_index=feature_index,
+ matched_feature_index=network.matched_feature_index,
+ source_names=network.source_names,
+ source_sizes=network.source_sizes,
+ rank_feature_index=rank_feature_index,
+ extra_required_arrays=(
+ ArrayRequirement(
+ "rank_feature_index",
+ shape=(len(rank_feature_index),),
+ dtype_kind="i",
+ ),
+ ),
+ score_batches=score_batches,
+ write_context=aucell_write_context,
+ ),
+ )
+
+ def get_enrichment(
+ self,
+ label: str,
+ *,
+ from_assay: str | None = None,
+ sources: Sequence[str] | None = None,
+ ) -> EnrichmentResult:
+ """Load a persisted enrichment result without materializing its scores.
+
+ Args:
+ label: Label passed to ``run_waggr`` or ``run_aucell``.
+ from_assay: RNA assay that owns the result. The default assay is used
+ when omitted.
+ sources: Optional source names to select and order.
+
+ Returns:
+ The stored metadata and a lazy cells-by-sources score matrix.
+ """
+ _validate_enrichment_label(label)
+ assay = self._get_assay(from_assay)
+ if not isinstance(assay, RNAassay):
+ raise TypeError("Enrichment results are only available for an RNAassay")
+ return _load_enrichment_result(
+ assay,
+ label=label,
+ sources=sources,
+ artifact_root=self.zw,
+ )
+
+ def run_marker_search(
+ self,
+ from_assay: str | None = None,
+ group_key: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ gene_batch_size: int | None = None,
+ n_threads: int | None = None,
+ skip_save: bool = False,
+ invalidate_cache: bool = False,
+ **norm_params: Any,
+ ) -> dict[str, Any] | None:
+ """Identifies group specific features for a given assay.
+
+ Please check out the ``find_markers_by_rank`` function for further details of how marker features for groups
+ are identified. The results are saved into the Zarr hierarchy under `markers` group.
+
+ Args:
+ from_assay: Name of the assay to be used. If no value is provided then the default assay will be used.
+ group_key: Required parameter. This has to be a column name from cell metadata table. This column dictates
+ how the cells will be grouped. Usually this would be a column denoting cell clusters.
+ cell_key: To run the test on specific subset of cells, provide the name of a boolean column in
+ the cell metadata table. (Default value: 'I')
+ feat_key: Boolean feature metadata column selecting features (default: ``'I'``).
+ gene_batch_size: Number of genes loaded per batch. When None,
+ selected genes are grouped into chunk-aligned blocks that fit
+ the operation memory budget.
+ n_threads: Threads for marker search.
+ skip_save: If True, return results without writing to Zarr.
+ **norm_params: Extra keyword arguments forwarded to ``normed``.
+
+ Returns:
+ Marker dict if ``skip_save`` is True, else None.
+ """
+ from ...features.markers import find_markers_by_rank
+
+ if group_key is None:
+ raise ValueError(
+ "ERROR: Please provide a value for `group_key`. This should be the name of a column from "
+ "cell metadata object that has information on how cells should be grouped."
+ )
+ from_assay, cell_key, _ = self._get_latest_keys(
+ from_assay,
+ cell_key,
+ feat_key if feat_key is not None else "I",
+ )
+ if feat_key is None:
+ feat_key = "I"
+ if n_threads is None:
+ n_threads = self.nthreads
+ assay = self._get_assay(from_assay)
+ resolved_norm_params = {
+ **norm_params,
+ "log_transform": norm_params.get("log_transform", False),
+ "renormalize_subset": norm_params.get(
+ "renormalize_subset",
+ False,
+ ),
+ }
+
+ slot_name = f"{cell_key}__{group_key}"
+ logger.debug(
+ f"Running marker search for {from_assay}/{slot_name} "
+ f"(feat_key={feat_key}, "
+ f"batch_size={gene_batch_size if gene_batch_size is not None else 'auto'})"
+ )
+ planned = None
+ group_cell_counts: dict[Any, tuple[int, int]] = {}
+ if not skip_save:
+ assay_grp = as_zarr_group(self.zw[assay.name], name=assay.name)
+ if "markers" not in assay_grp:
+ assay_grp.create_group("markers")
+ markers_grp = as_zarr_group(assay_grp["markers"], name="markers")
+ cell_selection = self._ensure_cell_selection(cell_key)
+ cluster_input = self._resolve_cell_data_provenance_input(
+ group_key,
+ cell_key=cell_key,
+ )
+ feature_values = np.asarray(
+ assay.feats.fetch_all(feat_key),
+ dtype=bool,
+ )
+ preserved_feature_display = feature_column_display(
+ assay.z,
+ feat_key,
+ )
+ feature_selection = resolve_selection_artifact(
+ self.zw,
+ scope="assay",
+ assay=from_assay,
+ kind="feature_selection",
+ values=feature_values,
+ row_ids=np.asarray(assay.feats.fetch_all("ids")),
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column=feat_key,
+ invalidate_cache=False,
+ )
+ link_feature_data_column(
+ assay.z,
+ feat_key,
+ feature_selection,
+ value_name="values",
+ default_display=categorical_display(feature_values),
+ preserved_display=preserved_feature_display,
+ )
+ group_labels = assay.cells.fetch(group_key, cell_key)
+ group_sizes = pd.Series(group_labels).value_counts()
+ n_selected = int(len(group_labels))
+ group_cell_counts = {
+ group_id: (
+ int(group_size),
+ int(n_selected - group_size),
+ )
+ for group_id, group_size in group_sizes.items()
+ }
+ expected_group_cell_counts: dict[str, tuple[int, int]] = {}
+ for group_id, counts in group_cell_counts.items():
+ group_name = str(group_id)
+ if group_name in expected_group_cell_counts:
+ raise ValueError(
+ "Marker group labels must remain unique after string conversion"
+ )
+ expected_group_cell_counts[group_name] = counts
+ feature_names_for_validation = np.asarray(assay.feats.fetch_all("names"))
+ expected_feature_index = np.flatnonzero(feature_values)
+
+ def marker_reuse_is_valid(
+ _ref: ArtifactRef,
+ candidate: zarr.Group,
+ ) -> bool:
+ try:
+ stored_feature_index = np.asarray(
+ as_zarr_array(
+ candidate["feature_index"],
+ name="feature_index",
+ )[:]
+ )
+ if stored_feature_index.dtype.kind not in {
+ "i",
+ "u",
+ } or not np.array_equal(
+ stored_feature_index.astype(np.int64, copy=False),
+ expected_feature_index,
+ ):
+ return False
+ _validate_marker_slot(
+ candidate,
+ feature_names_for_validation,
+ expected_group_cell_counts=expected_group_cell_counts,
+ )
+ except (IndexError, KeyError, TypeError, ValueError):
+ return False
+ return True
+
+ arguments = MarkerTableArguments(
+ cell_selection=cell_selection,
+ feature_selection=feature_selection,
+ clusters=cluster_input,
+ normalization=resolved_norm_params,
+ normalization_method=callable_identity(assay.normMethod),
+ size_factor=getattr(assay, "sf", None),
+ method=MARKER_METHOD,
+ alternative=MARKER_ALTERNATIVE,
+ tie_correction=MARKER_TIE_CORRECTION,
+ continuity_correction=MARKER_CONTINUITY_CORRECTION,
+ adjustment_method=MARKER_ADJUSTMENT_METHOD,
+ adjustment_scope=MARKER_ADJUSTMENT_SCOPE,
+ group_key=group_key,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ gene_batch_size=gene_batch_size,
+ n_threads=n_threads,
+ invalidate_cache=invalidate_cache,
+ )
+ planned = arguments.plan(
+ self.zw,
+ scope="assay",
+ assay=from_assay,
+ invalidate_cache=invalidate_cache,
+ required_arrays=(
+ ArrayRequirement(
+ "feature_index",
+ shape=(int(feature_values.sum()),),
+ dtype_kind="i",
+ ),
+ ),
+ required_attributes=(
+ AttributeRequirement(
+ "stat_columns",
+ expected_types=(list, tuple),
+ ),
+ ),
+ reuse_validator=marker_reuse_is_valid,
+ )
+
+ def select_marker_artifact(ref: ArtifactRef) -> None:
+ raw_artifacts = markers_grp.attrs.get(
+ "artifacts",
+ {},
+ )
+ if "artifacts" in markers_grp.attrs and not isinstance(
+ raw_artifacts,
+ dict,
+ ):
+ raise RuntimeError("Marker artifact index is invalid")
+ artifacts = (
+ dict(raw_artifacts) if isinstance(raw_artifacts, dict) else {}
+ )
+ artifacts[slot_name] = ref.to_dict()
+ markers_grp.attrs["artifacts"] = artifacts
+
+ if planned.reused:
+ select_marker_artifact(planned.ref)
+ return None
+
+ markers = find_markers_by_rank(
+ assay=assay,
+ group_key=group_key,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ batch_size=gene_batch_size,
+ n_threads=n_threads,
+ **resolved_norm_params,
+ )
+
+ if skip_save:
+ return markers
+
+ from ...storage.stores import is_remote_datastore
+
+ remote = is_remote_datastore(self.zarr_loc, self.z)
+ t_save = time.perf_counter()
+ assert planned is not None
+ remote_slot = start_artifact(self.zw, planned)
+ workers = max(1, int(n_threads or self.nthreads))
+ self._write_marker_slot(
+ remote_slot,
+ markers,
+ workers=workers if remote else 1,
+ group_cell_counts=group_cell_counts,
+ )
+ finish_artifact(remote_slot, planned)
+ select_marker_artifact(planned.ref)
+ logger.info(f"Stored marker results for {len(markers)} clusters")
+ logger.debug(
+ f"Saved marker results to {artifact_path(planned.ref)} "
+ f"in {time.perf_counter() - t_save:.1f}s"
+ )
+ return None
+
+ @staticmethod
+ def _write_marker_slot(
+ group: zarr.Group,
+ markers: dict[Any, pd.DataFrame],
+ *,
+ workers: int = 1,
+ group_cell_counts: dict[Any, tuple[int, int]],
+ ) -> None:
+ from ...storage.arrays import create_metadata_column
+
+ populated_groups = {
+ cluster_id for cluster_id, values in markers.items() if len(values)
+ }
+ missing_counts = populated_groups.difference(group_cell_counts)
+ if missing_counts:
+ raise ValueError(
+ "Marker writes require target and reference counts "
+ "for every populated group"
+ )
+ if any(
+ isinstance(count, bool) or not isinstance(count, int) or count < 2
+ for cluster_id in populated_groups
+ for count in group_cell_counts[cluster_id]
+ ):
+ raise ValueError("Marker target and reference counts must be integers >= 2")
+ feature_index = _shared_marker_feature_index(markers)
+ stats_by_group = {
+ cluster_id: _marker_stats_matrix(values, feature_index)
+ for cluster_id, values in markers.items()
+ if len(values)
+ }
+ group.attrs["stat_columns"] = list(_MARKER_STAT_COLUMNS)
+ group.attrs["method"] = MARKER_METHOD
+ group.attrs["alternative"] = MARKER_ALTERNATIVE
+ group.attrs["tie_correction"] = MARKER_TIE_CORRECTION
+ group.attrs["continuity_correction"] = MARKER_CONTINUITY_CORRECTION
+ group.attrs["adjustment_method"] = MARKER_ADJUSTMENT_METHOD
+ group.attrs["adjustment_scope"] = MARKER_ADJUSTMENT_SCOPE
+ create_metadata_column(
+ group,
+ "feature_index",
+ data=feature_index,
+ dtype=np.int32,
+ overwrite=True,
+ )
+
+ def write_cluster(item: tuple[Any, pd.DataFrame]) -> None:
+ cluster_id, vals = item
+ if len(vals) == 0:
+ return
+ cluster_group = group.create_group(str(cluster_id))
+ _write_compact_marker_stats(
+ cluster_group,
+ stats_by_group[cluster_id],
+ )
+ n_group, n_reference = group_cell_counts[cluster_id]
+ cluster_group.attrs["n_group"] = n_group
+ cluster_group.attrs["n_reference"] = n_reference
+
+ items = list(markers.items())
+ if workers <= 1:
+ for item in items:
+ write_cluster(item)
+ else:
+ from concurrent.futures import ThreadPoolExecutor
+
+ with ThreadPoolExecutor(max_workers=workers) as ex:
+ list(ex.map(write_cluster, items))
+
+ def _resolve_marker_group(
+ self,
+ from_assay: str | None,
+ cell_key: str,
+ group_key: str,
+ ) -> zarr.Group:
+ assay = self._get_assay(from_assay)
+ markers_group = as_zarr_group(
+ assay.z["markers"],
+ name="markers",
+ )
+ slot_name = f"{cell_key}__{group_key}"
+ raw_artifacts = markers_group.attrs.get("artifacts", {})
+ if "artifacts" in markers_group.attrs and not isinstance(
+ raw_artifacts,
+ dict,
+ ):
+ raise ValueError("Marker artifact index is invalid")
+ artifacts = dict(raw_artifacts) if isinstance(raw_artifacts, dict) else {}
+ raw_ref = artifacts.get(slot_name)
+ if slot_name in artifacts and not isinstance(raw_ref, dict):
+ raise ValueError("Marker artifact index is invalid")
+ if isinstance(raw_ref, dict):
+ ref = ArtifactRef.from_dict(raw_ref)
+ if (
+ ref.kind != "marker_table"
+ or ref.scope != "assay"
+ or ref.assay != assay.name
+ ):
+ raise ValueError("Marker artifact ref is invalid")
+ status = self.inspect_artifact(ref)
+ if not status.complete:
+ raise ValueError("Marker artifact is incomplete")
+ inputs = status.inputs or {}
+ current_selection = self._ensure_cell_selection(cell_key)
+ stored_selection = inputs.get("cell_selection")
+ if not isinstance(stored_selection, dict) or not (
+ self._selection_artifacts_match(
+ ArtifactRef.from_dict(stored_selection),
+ current_selection,
+ )
+ ):
+ raise ValueError("Marker artifact cell selection is stale")
+ current_clusters = self._resolve_cell_data_provenance_input(
+ group_key,
+ cell_key=cell_key,
+ )
+ stored_clusters = inputs.get("clusters")
+ if (
+ stored_clusters != current_clusters
+ and stored_clusters != current_clusters["artifact"]
+ ):
+ raise ValueError("Marker artifact cluster labels are stale")
+ return as_zarr_group(
+ self.zw[artifact_path(ref)],
+ name=artifact_path(ref),
+ )
+ return as_zarr_group(
+ markers_group[slot_name],
+ name=slot_name,
+ )
+
+ def get_markers(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ group_key: str | None = None,
+ group_id: str | int | None = None,
+ min_score: float = 0.25,
+ min_frac_exp: float = 0.2,
+ ) -> pd.DataFrame:
+ """Return marker features from `run_marker_search`.
+
+ When ``group_id`` is ``None`` (default), markers for every group under
+ ``group_key`` are returned in one long table with a ``group_id`` column.
+ Pass a specific ``group_id`` to return markers for that group only.
+ For a wide export of marker names only, use ``export_markers_to_csv``.
+
+ Args:
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ cell_key: To run the test on specific subset of cells, provide the name of a boolean column in
+ the cell metadata table.
+ group_key: Required parameter. This has to be a column name from cell metadata table.
+ Usually this would be a column denoting cell clusters. Please use the same value as used
+ when ran `run_marker_search`
+ group_id: One value from the ``group_key`` column, or ``None`` for all groups.
+ min_score: This value dictates how specific the feature value has to be in a group before it is
+ considered a marker for that group. The value has to be greater than 0 but less than or equal to
+ 1 (Default value: 0.25)
+ min_frac_exp: Minimum fraction of cells in a group that must have a non-zero value for a gene to be
+ considered a marker for that group.
+
+ Returns:
+ Pandas dataframe with marker statistics. All-group results include a ``group_id`` column.
+ """
+
+ if cell_key is None:
+ from_assay, cell_key, _ = self._get_latest_keys(from_assay, cell_key, None)
+ if group_key is None:
+ raise ValueError(
+ "ERROR: Please provide a value for group_key. "
+ "This should be same as used for `run_marker_search`"
+ )
+ assay = self._get_assay(from_assay)
+ try:
+ g = self._resolve_marker_group(
+ from_assay,
+ cell_key,
+ group_key,
+ )
+ except KeyError:
+ raise KeyError(
+ "ERROR: Couldn't find the location of markers. Please make sure that you have already called "
+ "`run_marker_search` method with same value of `cell_key` and `group_key`"
+ )
+ out_cols = list(_MARKER_OUT_COLUMNS)
+ gids = sorted(set(assay.cells.fetch(group_key, key=cell_key)))
+ if group_id is not None:
+ gids = [group_id]
+
+ feature_names = assay.feats.fetch_all("names")
+ feature_ids = assay.feats.fetch_all("ids")
+ dfs = []
+ for gid in gids:
+ group_name = str(gid)
+ if group_name in g:
+ marker_grp = as_zarr_group(g[group_name], name=group_name)
+ df = _load_marker_cluster_frame(
+ g,
+ marker_grp,
+ feature_names,
+ group_id=gid,
+ feature_ids=feature_ids,
+ )
+ else:
+ logger.debug(f"No markers found for {gid} returning empty dataframe")
+ empty_cols = [
+ "group_id",
+ "feature_name",
+ "feature_index",
+ *out_cols[1:],
+ ]
+ df = pd.DataFrame(
+ {name: pd.Series(dtype=object) for name in empty_cols}
+ )
+ dfs.append(df)
+ dfs = pd.concat(dfs, ignore_index=True)
+ keep = np.ones(len(dfs), dtype=bool)
+ if "score" in dfs and dfs["score"].notna().any():
+ keep &= dfs["score"].fillna(-np.inf).to_numpy() >= min_score
+ if "frac_exp" in dfs and dfs["frac_exp"].notna().any():
+ keep &= dfs["frac_exp"].fillna(-np.inf).to_numpy() >= min_frac_exp
+ return dfs.loc[keep].reset_index(drop=True)
+
+ def export_markers_to_csv(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ group_key: str | None = None,
+ csv_filename: str | None = None,
+ min_score: float = 0.25,
+ min_frac_exp: float = 0.2,
+ ) -> None:
+ """Export markers of each cluster/group to a CSV file where each column
+ contains the marker names sorted by score (descending order, highest
+ first). This function does not export the scores of markers as they can
+ be obtained using `get_markers` function.
+
+ Args:
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ cell_key: To run the test on specific subset of cells, provide the name of a boolean column in
+ the cell metadata table.
+ group_key: Required parameter. This has to be a column name from cell metadata table.
+ Usually this would be a column denoting cell clusters. Please use the same value as used
+ when ran `run_marker_search`
+ csv_filename: Required parameter. Name, with path, of CSV file where the marker table is to be saved.
+ min_score: This value dictates how specific the feature value has to be in a group before it is
+ considered a marker for that group. The value has to be greater than 0 but less than or equal to
+ 1 (Default value: 0.25)
+ min_frac_exp: Minimum fraction of cells in a group that must have a non-zero value for a gene to be
+ considered a marker for that group.
+
+ Returns:
+ """
+ # Not testing the values of from_assay and cell_key because they will be tested in `get_markers`
+ if group_key is None:
+ raise ValueError(
+ "ERROR: Please provide a value for group_key. "
+ "This should be same as used for `run_marker_search`"
+ )
+ if csv_filename is None:
+ raise ValueError(
+ "ERROR: Please provide a value for parameter `csv_filename`"
+ )
+ from_assay, cell_key, _ = self._get_latest_keys(from_assay, cell_key, None)
+ clusters = self.cells.fetch(group_key, key=cell_key)
+ markers_table = {}
+ for group_id in sorted(set(clusters)):
+ m = self.get_markers(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ group_key=group_key,
+ group_id=group_id,
+ min_score=min_score,
+ min_frac_exp=min_frac_exp,
+ )
+ if len(m) > 0:
+ markers_table[group_id] = m["feature_name"].reset_index(drop=True)
+ else:
+ markers_table[group_id] = pd.Series([])
+ pd.DataFrame(markers_table).fillna("").to_csv(csv_filename, index=False)
+ return None
+
+ def add_grouped_assay(
+ self,
+ from_assay: str | None = None,
+ group_key: str | None = None,
+ assay_label: str | None = None,
+ exclude_values: list | None = None,
+ ) -> None:
+ """Add a new assay to the DataStore by grouping together multiple
+ features and taking their means. This method requires that the features
+ are already assigned a group/cluster identity. The new assay will have
+ all the cells but only features that marked by 'feat_key' and contain a
+ group identity not present in `exclude_values`.
+
+ Args:
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ group_key: This is mandatory parameter. Name of the column in feature metadata table to be used for
+ grouping features.
+ assay_label: This is mandatory parameter. A name for the new assay.
+ exclude_values: These groups/clusters will be ignored and not added to new assay. By default, it is set to
+ [-1], this means that all the features that have the group identity of -1 are not used.
+
+ Returns: None
+ """
+
+ from ...storage.sharding import write_dense_in_shard_rows
+
+ from ...storage.schema import create_zarr_count_assay
+
+ if assay_label is None:
+ raise ValueError(
+ "ERROR: Please provide a value for `assay_label`. "
+ "It will be used to create a new assay"
+ )
+ if group_key is None:
+ raise ValueError(
+ "ERROR: Please provide a value for `group_key`. "
+ "This should be name of the column in the feature attribute table that contains the group/cluster "
+ "identity of each feature."
+ )
+
+ assay = self._get_assay(from_assay)
+ groups = assay.feats.fetch_all(group_key)
+ if exclude_values is None:
+ exclude_values = [-1]
+ group_set = sorted(set(groups).difference(exclude_values))
+
+ module_ids = [f"group_{x}" for x in group_set]
+ g = create_zarr_count_assay(
+ z=self.zw,
+ assay_name=assay_label,
+ workspace=self.workspace,
+ n_cells=assay.cells.N,
+ feat_ids=module_ids,
+ feat_names=module_ids,
+ dtype="float",
+ profile=self.storageProfile,
+ )
+
+ cell_idx = np.array(list(range(assay.cells.N)))
+ n_groups = len(group_set)
+ matrix = np.zeros((assay.cells.N, n_groups), dtype=np.float64)
+ for n, i in iter_progress(
+ enumerate(group_set), desc="Computing grouped means", total=len(group_set)
+ ):
+ feat_idx = np.where(groups == i)[0]
+ matrix[:, n] = (
+ assay.normed(cell_idx=cell_idx, feat_idx=feat_idx)
+ .mean(axis=1)
+ .compute()
+ )
+ write_dense_in_shard_rows(
+ g,
+ lambda start, end: matrix[start:end, :],
+ msg="Writing grouped assay",
+ resources=self.resources,
+ )
+
+ self._load_assays(min_cells=0, custom_assay_types={assay_label: "Assay"})
+ self._ini_cell_props(min_features=0, mito_pattern="", ribo_pattern="")
+ grouped_assay = self._get_assay(assay_label)
+ grouped_assay.attrs["grouped_from_assay"] = assay.name
+ grouped_assay.attrs["grouped_group_key"] = group_key
+ group_column = as_zarr_array(
+ as_zarr_group(assay.z["featureData"], name="featureData")[group_key],
+ name=group_key,
+ )
+ raw_source = group_column.attrs.get("source_artifact")
+ if isinstance(raw_source, dict):
+ try:
+ source_ref = ArtifactRef.from_dict(raw_source)
+ source_status = inspect_artifact(self.zw, source_ref)
+ except (KeyError, TypeError, ValueError):
+ source_ref = None
+ if source_ref is not None and source_status.complete:
+ grouped_assay.attrs["grouped_group_artifact"] = source_ref.to_dict()
+ if "grouped_group_digest" in grouped_assay.attrs:
+ del grouped_assay.attrs["grouped_group_digest"]
+ else:
+ grouped_assay.attrs["grouped_group_digest"] = _group_assignment_digest(
+ groups
+ )
+ else:
+ grouped_assay.attrs["grouped_group_digest"] = _group_assignment_digest(
+ groups
+ )
+
+ def add_melded_assay(
+ self,
+ from_assay: str | None = None,
+ external_bed_fn: str | None = None,
+ assay_label: str | None = None,
+ peaks_col: str = "ids",
+ scalar_coeff: float = 1e5,
+ renormalization: bool = True,
+ assay_type: str = "Assay",
+ cell_key: str = "I",
+ ) -> None:
+ """This method performs "assay melding" and can be only be used for
+ assay's wherein features have genomic coordinates. In the process of
+ melding the input genomic coordinates from `external_bed_fn` are
+ intersected with the assay's features. Based on this intersection a
+ mapping is created wherein each coordinate interval maps to one or more
+ feature coordinates from the assay.
+
+ This method has been designed for snATAC-Seq data and can be used to quantify accessibility of specific
+ genomic loci such as gene bodies, promoters, enhancers, motifs, etc.
+ Features from the BED file are retained even when they do not overlap any peak; those zero-count features
+ are marked invalid during assay initialization.
+
+ Args:
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ external_bed_fn: This is mandatory parameter. This file should be a BED format file with at least five
+ columns containing: chromosome, start position, end position, feature id and feature name.
+ Coordinates should be in half open format. That means that actual end position is -1
+ assay_label: This is mandatory parameter. A name for the new assay.
+ peaks_col: The column in feature metadata table that contains the genomic coordinate information of each
+ feature. The genomic coordinates are represented as strings in this format: chr:start-end
+ (Default value: 'ids')
+ scalar_coeff: An arbitrary scalar multiplier. Only used when renormalization is True (Default value: 1e5)
+ renormalization: Whether to rescale the sum of feature values for each cell to `scalar_coeff`
+ (Default value: True)
+ assay_type: The new assay (melded assay) is saved as this type. This can be any type of Assay class from
+ `assay` module. Please provide string representation of class. By default, the assay is assigned
+ a generic class and has a dummy normalization function (Default value: 'Assay')
+ cell_key: Cells used to learn peak document frequency. Every cell is
+ still scored so the new assay remains row-aligned.
+
+ Returns:
+ None
+ """
+
+ from ...features.genomic.melding import coordinate_melding
+
+ if assay_label is None:
+ raise ValueError(
+ "ERROR: Please provide a value for `assay_label`. "
+ "It will be used to create a new assay"
+ )
+ if external_bed_fn is None:
+ raise ValueError(
+ "ERROR: Please provide a value for `feature_bed_fn`. "
+ "This should be a BED format file with atleast 5 columns."
+ )
+
+ assay = self._get_assay(from_assay)
+ idf_cell_idx = assay.cells.active_index(cell_key)
+ if len(idf_cell_idx) == 0:
+ raise ValueError("Gene-score IDF requires at least one selected cell")
+ feature_bed = pd.read_csv(external_bed_fn, header=None, sep="\t").sort_values(
+ by=[0, 1] # type: ignore
+ )
+
+ peaks_coords = assay.feats.fetch_all(peaks_col)
+ coords_ser = pd.Series(peaks_coords, dtype="object")
+ string_mask = coords_ser.map(lambda x: isinstance(x, str))
+ colon_counts = coords_ser.str.count(":")
+ hyphen_counts = coords_ser.str.split(":").str[-1].str.count("-")
+ invalid_mask = (
+ ~string_mask
+ | colon_counts.ne(1).fillna(True)
+ | hyphen_counts.ne(1).fillna(True)
+ )
+ invalid_coords = invalid_mask.to_numpy(dtype=bool)
+ if invalid_coords.any():
+ n = int(np.flatnonzero(invalid_coords)[0])
+ raise ValueError(
+ f"ERROR: Coordinate format check failed for element: {peaks_coords[n]} (position {n}). "
+ f"The format should be chr:start-end. Please note the colon and hyphen position"
+ )
+
+ coordinate_melding(
+ assay,
+ workspace=self.workspace,
+ feature_bed=feature_bed,
+ new_assay_name=assay_label,
+ peaks_col=peaks_col,
+ scalar_coeff=scalar_coeff,
+ renormalization=renormalization,
+ peaks_coords=peaks_coords,
+ idf_cell_idx=idf_cell_idx,
+ )
+
+ self._load_assays(min_cells=10, custom_assay_types={assay_label: assay_type})
+ self._ini_cell_props(min_features=0, mito_pattern=None, ribo_pattern=None)
+
+ def make_bulk(
+ self,
+ from_assay: str | None = None,
+ cell_key: str = "I",
+ group_key: str | None = None,
+ secondary_group_key: str | None = None,
+ aggr_type: Literal["mean", "sum"] = "mean",
+ return_fraction: bool = False,
+ feature_label: Literal["index", "id", "name"] = "index",
+ remove_empty_features: bool = True,
+ pseudo_reps: int = 1,
+ null_vals: list[Any] | None = None,
+ secondary_null_vals: list[Any] | None = None,
+ random_seed: int = 4466,
+ ) -> pd.DataFrame | tuple[pd.DataFrame, pd.DataFrame]:
+ """Merge data from cells to create a bulk profile.
+
+ Args:
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ cell_key: Name of the column in cell metadata table to be used for selecting cells.
+ group_key: Required cell metadata column used to group cells.
+ Passing ``None`` raises ``ValueError``.
+ secondary_group_key: Name of the column in cell metadata table to be used for sub-grouping cells.
+ aggr_type: Type of aggregation to be used. Can be either 'mean' or 'sum'. (Default value: 'mean')
+ return_fraction: Return the fraction of cells expressing a gene in each group. (Default value: False)
+ feature_label: The column in feature metadata table to use as row labels. (Default value: 'index')
+ pseudo_reps: Within each group, randomly split cells into this many
+ pseudo-replicates. Values greater than 1 produce descriptive
+ resamples of the same cells, not independent biological
+ replicates. (Default value: 1)
+ remove_empty_features: Remove features that are not expressed in any cell. (Default value: True)
+ null_vals: Values to be considered as missing values in the `group_key` column. These values will be skipped.
+ secondary_null_vals: Values to be considered as missing values in the `secondary_group_key` column.
+ These values will be skipped.
+ random_seed: Seed used when assigning cells to pseudo-replicates.
+
+ Returns:
+ A pandas dataframe containing the bulk profile. If `return_fraction` is True, then a tuple of two dataframes
+ is returned. The second dataframe contains the fraction of cells expressing each feature in each group.
+ """
+
+ def make_reps(v: NDArray[Any], n_reps: int, seed: int) -> list[NDArray[Any]]:
+ v_list = list(v)
+ random_state = np.random.RandomState(seed)
+ shuffled_idx = random_state.choice(v_list, len(v_list), replace=False)
+ rep_idx = np.array_split(shuffled_idx, n_reps)
+ return [np.array(sorted(x)) for x in rep_idx]
+
+ if pseudo_reps < 1:
+ pseudo_reps = 1
+ if pseudo_reps > 1:
+ logger.warning(
+ "make_bulk with pseudo_reps > 1 randomly splits cells within each "
+ "group into descriptive resamples. These are not independent "
+ "biological replicates and must not be used as such for "
+ "differential expression."
+ )
+ if null_vals is None:
+ null_vals = []
+ if secondary_null_vals is None:
+ secondary_null_vals = []
+ if group_key is None:
+ raise ValueError("ERROR: Please provide a value for `group_key` parameter")
+ else:
+ groups = self.cells.fetch_all(group_key)
+ active_idx = self.cells.active_index(cell_key)
+ groups_set = sorted(set(groups[active_idx]))
+ if secondary_group_key is None:
+ sec_groups: NDArray[Any] = np.array([None], dtype=object)
+ sec_groups_set: list[Any] = [None]
+ else:
+ sec_groups = self.cells.fetch_all(secondary_group_key)
+ sec_groups_set = sorted(set(sec_groups[active_idx]))
+
+ assay = self._get_assay(from_assay)
+
+ vals: dict[str, NDArray[Any]] = {}
+ fracs: dict[str, NDArray[Any]] = {}
+ all_feat_idx = np.arange(assay.feats.N)
+ active_mask = np.zeros(self.cells.N, dtype=bool)
+ active_mask[active_idx] = True
+ for g in iter_progress(
+ groups_set,
+ desc="Aggregating pseudo-replicates",
+ total=len(groups_set),
+ ):
+ if g in null_vals:
+ continue
+ for sg in sec_groups_set: # type: ignore
+ if sg in secondary_null_vals:
+ continue
+ if sg is None and len(sec_groups) == 1:
+ g_idx = np.where((groups == g) & active_mask)[0]
+ else:
+ g_idx = np.where((groups == g) & (sec_groups == sg) & active_mask)[
+ 0
+ ]
+ rep_indices = make_reps(g_idx, pseudo_reps, random_seed)
+ for n, idx in enumerate(rep_indices):
+ if sg is None and len(sec_groups) == 1:
+ col_name = f"{g}"
+ else:
+ col_name = f"{g}_{sg}"
+ if pseudo_reps > 1:
+ col_name += f"_Rep{n + 1}"
+ if len(idx) == 0:
+ vals[col_name] = np.zeros(assay.feats.N)
+ continue
+ if aggr_type == "sum":
+ vals[col_name] = controlled_compute(
+ assay.rawData[idx].sum(axis=0), self.nthreads
+ )
+ elif aggr_type == "mean":
+ vals[col_name] = controlled_compute(
+ assay.normed(cell_idx=idx, feat_idx=all_feat_idx).mean(
+ axis=0
+ ),
+ self.nthreads,
+ )
+ else:
+ raise ValueError(
+ "ERROR: `aggr_type` can only be either 'sum' or 'mean'"
+ )
+ if return_fraction:
+ fracs[col_name] = (
+ (assay.rawData[idx] > 0).mean(axis=0).compute()
+ )
+
+ vals_df = pd.DataFrame(vals).fillna(0)
+
+ empty_idx = None
+ if remove_empty_features:
+ empty_idx = vals_df.sum(axis=1) != 0
+ vals_df = vals_df.loc[empty_idx]
+
+ if feature_label == "id":
+ vals_df.set_index(
+ pd.Series(assay.feats.fetch_all("ids")).reindex(vals_df.index).values,
+ inplace=True,
+ drop=True,
+ )
+ elif feature_label == "name":
+ vals_df.set_index(
+ pd.Series(assay.feats.fetch_all("names")).reindex(vals_df.index).values,
+ inplace=True,
+ drop=True,
+ )
+
+ if return_fraction:
+ fracs_df = pd.DataFrame(fracs).fillna(0)
+ if empty_idx is not None:
+ fracs_df = fracs_df[empty_idx]
+ fracs_df.set_index(vals_df.index, inplace=True, drop=True)
+ return vals_df, fracs_df
+ return vals_df
diff --git a/scarf/datastore/_operations/graph.py b/scarf/datastore/_operations/graph.py
new file mode 100644
index 00000000..cd9fe0bd
--- /dev/null
+++ b/scarf/datastore/_operations/graph.py
@@ -0,0 +1,3893 @@
+import math
+import os
+import tempfile
+from collections.abc import Callable, Iterator
+from contextlib import contextmanager
+from dataclasses import replace
+from typing import TYPE_CHECKING, Any, Literal, cast
+from weakref import WeakKeyDictionary
+
+import numpy as np
+import pandas as pd
+import zarr
+from numpy.typing import NDArray
+from scipy.sparse import coo_matrix, csr_matrix
+
+from ...embeddings.reduction import _streaming_lsi_accumulator_bytes
+from ...storage.types import as_zarr_array, as_zarr_group
+from ...graph.arguments import (
+ AnnIndexArguments,
+ ConnectivityMapArguments,
+ CustomReductionArguments,
+ EmbeddingInitializationArguments,
+ FeatureScalingArguments,
+ HarmonyArguments,
+ LsiArguments,
+ NeighborQueryArguments,
+ NormalizationArguments,
+ PcaArguments,
+ _positive_integer,
+)
+from ...graph.encoded_paths import (
+ is_integrated_graph_path,
+ lookup_latest_assay_graph,
+ lookup_latest_nearest_neighbor_paths,
+ lookup_stored_integrated_graph,
+ make_integrated_graph_path,
+ make_normalized_group_path,
+ nearest_neighbor_paths_from_loc,
+ nearest_neighbors_group_path_from_cell_graph,
+ parse_assay_graph_paths,
+ parse_nearest_neighbors_group_path,
+ parse_neighbor_index_group_path,
+ parse_reduction_group_path,
+)
+from ...graph.distances import validate_distance_provenance
+from ...graph.paths import StoredAssayGraph, StoredGraph
+from ...graph.state import (
+ AssayState,
+ named_result_mismatch,
+ normalized_path_from_state,
+ read_assay_state,
+ stored_assay_graph_from_ref,
+ stored_assay_graph_from_state,
+ validate_artifact_graph_selection,
+ validate_cell_selection_artifact,
+ validate_imported_coordinates_artifact,
+ validate_legacy_graph_selection,
+ validate_neighbors_artifact_selection,
+ validate_normalized_artifact_selection,
+ write_assay_state,
+)
+from ...matrix import ChunkedArray
+from ...metadata.artifacts import (
+ categorical_display,
+ column_display,
+ continuous_display,
+ link_cell_data_column,
+ link_feature_data_column,
+)
+from ...neighbors.stages import (
+ AnnIndexStage,
+ BatchCorrectionStage,
+ ChunkedCoordinateStream,
+ CoordinateSource,
+ KMeansInitializationStage,
+ LazyTransformStream,
+ NeighborQueryStage,
+ ReductionTransform,
+)
+from ...neighbors.stream import AnnStream
+from ...storage.ann_index import (
+ has_ann_index,
+ legacy_ann_index_path,
+ load_ann_index,
+ load_ann_index_from_path,
+ save_ann_index,
+)
+from ...storage.arrays import create_numeric_array, create_zarr_dataset
+from ...storage.artifact_writer import (
+ ArrayRequirement,
+ PlannedArtifact,
+ finish_artifact,
+ plan_artifact,
+ reused_artifact_group,
+ start_artifact,
+)
+from ...storage.artifacts import (
+ ArtifactRef,
+ ArtifactScope,
+ artifact_group,
+ artifact_path,
+ fingerprint_array,
+ fingerprint_stored_arrays,
+ fingerprint_strings,
+ group_at,
+ inspect_artifact,
+ parse_artifact_path,
+ require_complete_artifact,
+ serialize_artifact_value,
+)
+from ...storage.copy import (
+ copy_zarr_array,
+ create_or_open_staged_normed_array,
+)
+from ...storage.budget import ResourceBudget
+from ...storage.geometry import array_geometry
+from ...storage.layout import (
+ _group_zarr_format,
+ array_shard_rows,
+ iter_shard_row_slices,
+ row_sharded_array_spec,
+)
+from ...storage.profiles import resolve_storage_profile
+from ...storage.sharding import write_dense_from_row_batches
+from ...storage.stores import is_remote_datastore, zarr_root_path
+from ...storage.selections import (
+ resolve_metadata_snapshot,
+ resolve_selection_artifact,
+)
+from ...utils.arrays import clean_array
+from ...utils.compute import show_dask_progress
+from ...utils.logging import logger
+
+if TYPE_CHECKING:
+ from ..base_datastore import BaseDataStore as _GraphOperationsBase
+else:
+ _GraphOperationsBase = object
+
+
+def _row_block(
+ array: zarr.Array,
+ requested: int | None,
+ *,
+ minimum: int | None = None,
+) -> int:
+ """Resolve a row block, defaulting to the array's own on-disk row band.
+
+ An explicit value that is not a whole number of row bands makes every
+ block straddle a band boundary, so each band is fetched and decoded more
+ than once.
+ """
+ band = max(1, array_shard_rows(array))
+ n_rows = max(1, int(array.shape[0]))
+ if requested is None:
+ resolved = min(band, n_rows)
+ else:
+ resolved = min(_positive_integer(requested, "batch_size"), n_rows)
+ if minimum is not None and resolved < minimum:
+ aligned = min(n_rows, ((minimum + band - 1) // band) * band)
+ if requested is not None:
+ logger.warning(
+ f"batch_size {resolved} is below the required minimum of "
+ f"{minimum}; using the aligned batch_size {aligned}."
+ )
+ resolved = aligned
+ if resolved % band and resolved < n_rows:
+ logger.warning(
+ f"batch_size {resolved} is not a multiple of the {band}-row band "
+ f"of {array.name}; blocks will straddle band boundaries and reread "
+ "them. Leave batch_size unset to follow the stored layout."
+ )
+ return resolved
+
+
+def _streaming_lsi_block_rows(
+ array: zarr.Array,
+ resources: ResourceBudget,
+ *,
+ n_components: int,
+ n_oversamples: int,
+) -> int:
+ n_rows, n_features = map(int, array.shape)
+ width = min(n_rows, n_features, n_components + n_oversamples)
+ accumulator_bytes = _streaming_lsi_accumulator_bytes(n_features, width)
+ geometry = array_geometry(array)
+ decode_bytes = 0 if geometry is None else geometry.nominalChunkBytes()
+ available = resources.memoryBytes - accumulator_bytes - decode_bytes
+ input_itemsize = max(int(np.dtype(array.dtype).itemsize), 1)
+ row_bytes = (
+ n_features * (input_itemsize + np.dtype(np.float64).itemsize)
+ + width * np.dtype(np.float64).itemsize
+ )
+ if available < row_bytes:
+ required = accumulator_bytes + decode_bytes + row_bytes
+ raise MemoryError(
+ f"Streaming LSI needs about {required} bytes for one row block, "
+ f"but the operation limit is {resources.memoryBytes} bytes"
+ )
+ return max(1, min(n_rows, available // row_bytes))
+
+
+def _sampling_fraction(value: Any, name: str) -> float:
+ if isinstance(value, bool):
+ raise TypeError(f"{name} must be a number")
+ try:
+ resolved = float(value)
+ except (TypeError, ValueError):
+ raise TypeError(f"{name} must be a number") from None
+ if not math.isfinite(resolved) or not 0 < resolved <= 1:
+ raise ValueError(f"{name} must be greater than 0 and at most 1")
+ return resolved
+
+
+class _GraphOperationsMixin(_GraphOperationsBase):
+ _annStreamPaths: WeakKeyDictionary[AnnStream, str]
+ _annStreamNeighborPaths: WeakKeyDictionary[AnnStream, str]
+ _normalizedArtifactCache: dict[ArtifactRef, ChunkedArray]
+ _artifactExecutionContext: dict[str, Any]
+
+ if TYPE_CHECKING:
+
+ def _build_mapping_reference_artifact(
+ self,
+ *,
+ reduction: ArtifactRef,
+ batch_correction: ArtifactRef,
+ ann_index: ArtifactRef,
+ neighbors: ArtifactRef,
+ invalidate_cache: bool,
+ ) -> ArtifactRef: ...
+
+ def _remember_ann_stream_path(self, ann_obj: AnnStream, path: str) -> None:
+ try:
+ paths = self._annStreamPaths
+ except AttributeError:
+ paths = WeakKeyDictionary()
+ self._annStreamPaths = paths
+ paths[ann_obj] = path
+
+ def _remember_ann_stream_neighbors(
+ self,
+ ann_obj: AnnStream,
+ path: str,
+ ) -> None:
+ try:
+ paths = self._annStreamNeighborPaths
+ except AttributeError:
+ paths = WeakKeyDictionary()
+ self._annStreamNeighborPaths = paths
+ paths[ann_obj] = path
+
+ def _get_latest_keys(
+ self,
+ from_assay: str | None,
+ cell_key: str | None,
+ feat_key: str | None,
+ ) -> tuple[str, str, str]:
+ if from_assay is None:
+ from_assay = self._defaultAssay
+ if from_assay is None:
+ raise ValueError("No default assay is configured")
+ if cell_key is None:
+ cell_key = self._get_latest_cell_key(from_assay)
+ if feat_key is None:
+ feat_key = self._get_latest_feat_key(from_assay)
+ return from_assay, cell_key, feat_key
+
+ def get_normalized_group_path(
+ self, from_assay: str, cell_key: str, feat_key: str
+ ) -> str:
+ """Return the selected normalized artifact or released-layout path.
+
+ Args:
+ from_assay: Name of the assay.
+ cell_key: Cell key used (or to be used) for the graph.
+ feat_key: Feature key used (or to be used) for the graph.
+
+ Explicit keys use matching assay state when present, then the historical
+ ``normed__`` convention.
+ """
+ state_path = normalized_path_from_state(
+ self.zw,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ if state_path is not None:
+ return state_path
+ return make_normalized_group_path(from_assay, cell_key, feat_key)
+
+ def get_latest_graph_loc(
+ self, from_assay: str, cell_key: str, feat_key: str
+ ) -> str:
+ """Return the location of the latest graph in the Zarr hierarchy.
+
+ Args:
+ from_assay: Name of the assay.
+ cell_key: Cell key used to create the graph.
+ feat_key: Feature key used to create the graph.
+
+ Returns:
+ Path of graph in the Zarr hierarchy
+ """
+ stored = self._lookup_stored_graph(from_assay, cell_key, feat_key)
+ if not isinstance(stored, StoredAssayGraph):
+ raise TypeError("Latest assay graph lookup returned a non-assay graph")
+ return stored.paths.cell_graph_group_path
+
+ def _resolve_integrated_graph_path(self, label: str) -> str:
+ index_path = self._integratedGraphsLoc
+ if index_path in self.zw:
+ index_group = as_zarr_group(self.zw[index_path], name=index_path)
+ raw_artifacts = index_group.attrs.get("artifacts", {})
+ if "artifacts" in index_group.attrs and not isinstance(
+ raw_artifacts,
+ dict,
+ ):
+ raise RuntimeError("Integrated graph artifact index is invalid")
+ if isinstance(raw_artifacts, dict):
+ raw_ref = raw_artifacts.get(label)
+ if label in raw_artifacts and not isinstance(raw_ref, dict):
+ raise RuntimeError(
+ f"Integrated graph index for {label!r} is invalid"
+ )
+ if isinstance(raw_ref, dict):
+ try:
+ ref = ArtifactRef.from_dict(raw_ref)
+ if ref.scope != "datastore" or ref.kind != "integrated_graph":
+ raise ValueError(
+ "Integrated graph index has an invalid ref"
+ )
+ status = inspect_artifact(self.zw, ref)
+ except (KeyError, TypeError, ValueError) as exc:
+ raise RuntimeError(
+ f"Integrated graph index for {label!r} is invalid"
+ ) from exc
+ if not status.exists or not status.complete:
+ raise RuntimeError(
+ f"Integrated graph index for {label!r} is incomplete"
+ )
+ return status.path
+ return make_integrated_graph_path(index_path, label)
+
+ def _lookup_stored_graph(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ graph_loc: str | None = None,
+ ) -> StoredGraph:
+ """Return a stored assay or integrated graph without mutating the store.
+
+ When ``graph_loc`` is omitted, resolves assay state before released
+ ``latest_*`` pointers. Explicit logical and encoded locations remain
+ readable without changing state.
+ """
+ if graph_loc is not None:
+ if is_integrated_graph_path(graph_loc, self._integratedGraphsLoc):
+ return lookup_stored_integrated_graph(self.zw, graph_loc)
+ try:
+ ref = parse_artifact_path(graph_loc)
+ except ValueError:
+ explicit_stored = parse_assay_graph_paths(graph_loc)
+ validate_legacy_graph_selection(
+ self,
+ graph_loc,
+ explicit_stored.from_assay,
+ explicit_stored.cell_key,
+ explicit_stored.feat_key,
+ )
+ return explicit_stored
+ if ref.scope == "datastore" and ref.kind == "integrated_graph":
+ status = inspect_artifact(self.zw, ref)
+ if not status.exists or not status.complete:
+ raise RuntimeError(
+ f"Integrated graph artifact is incomplete: {graph_loc}"
+ )
+ return lookup_stored_integrated_graph(self.zw, graph_loc)
+ if ref.scope != "assay" or ref.kind != "connectivity_map":
+ raise ValueError(f"Not an assay connectivity-map artifact: {graph_loc}")
+ return stored_assay_graph_from_ref(self.zw, ref)
+
+ state_assay = from_assay or self._defaultAssay
+ if state_assay is not None:
+ state = read_assay_state(self.zw, state_assay)
+ if (
+ state is not None
+ and (cell_key is None or cell_key == state.cell_key)
+ and (feat_key is None or feat_key == state.feat_key)
+ ):
+ selected_cell_key = cell_key or state.cell_key
+ selected_feat_key = feat_key or state.feat_key
+ state_stored = stored_assay_graph_from_state(
+ self.zw,
+ state_assay,
+ selected_cell_key,
+ selected_feat_key,
+ )
+ if state_stored is not None:
+ return state_stored
+ from_assay, cell_key, feat_key = self._get_latest_keys(
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ legacy_stored = lookup_latest_assay_graph(
+ self.zw,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ validate_legacy_graph_selection(
+ self,
+ legacy_stored.paths.cell_graph_group_path,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ return legacy_stored
+
+ def _get_latest_knn_loc(self, from_assay: str | None = None) -> str:
+ """Convenience function to identify location of the latest KNN graph in
+ the Zarr hierarchy.
+
+ Args:
+ from_assay: Name of the assay.
+
+ Returns:
+ Path of KNN graph in the Zarr hierarchy
+ """
+ if from_assay is None:
+ logger.debug("Using the default assay for the KNN graph")
+ from_assay = self._load_default_assay()
+
+ if from_assay not in self.assay_names:
+ raise ValueError(f"ERROR: Assay {from_assay} does not exist")
+
+ state = read_assay_state(self.zw, from_assay)
+ if state is not None:
+ if state.neighbors is None:
+ raise RuntimeError("AssayState has no neighbors artifact")
+ status = inspect_artifact(self.zw, state.neighbors)
+ if not status.exists or not status.complete:
+ raise RuntimeError("AssayState selects incomplete neighbors")
+ return status.path
+
+ latest_cell_key = cast(
+ str,
+ as_zarr_group(self.zw[from_assay], name=from_assay).attrs[
+ "latest_cell_key"
+ ],
+ )
+ latest_feat_key = cast(
+ str,
+ as_zarr_group(self.zw[from_assay], name=from_assay).attrs[
+ "latest_feat_key"
+ ],
+ )
+ paths = lookup_latest_nearest_neighbor_paths(
+ self.zw,
+ from_assay,
+ latest_cell_key,
+ latest_feat_key,
+ )
+ reduction_loc = paths.reduction_group_path
+ reduction_grp = as_zarr_group(self.zw[reduction_loc], name=reduction_loc)
+ if "reduction" not in reduction_grp:
+ raise ValueError(f"ERROR: PCA Reduction not found in {reduction_loc}")
+ return paths.nearest_neighbors_group_path
+
+ def _resolve_ann_index(
+ self,
+ ann_loc: str,
+ ann_metric: str,
+ dim: int,
+ expected_count: int | None = None,
+ ) -> Any:
+ """Load an ANN index from Zarr or a legacy file."""
+ ann_group: zarr.Group | None = (
+ as_zarr_group(self.zw[ann_loc], name=ann_loc)
+ if ann_loc in self.zw
+ else None
+ )
+
+ if ann_group is not None and has_ann_index(ann_group):
+ return load_ann_index(
+ ann_group,
+ ann_metric,
+ dim,
+ expected_count=expected_count,
+ )
+
+ legacy = legacy_ann_index_path(zarr_root_path(self.zw), ann_loc)
+ if legacy is not None and os.path.exists(legacy):
+ return load_ann_index_from_path(
+ legacy,
+ ann_metric,
+ dim,
+ expected_count=expected_count,
+ )
+
+ logger.debug(
+ "ANN index not found in store; will rebuild from normalized data and loadings"
+ )
+ return None
+
+ def _persist_ann_index(
+ self,
+ ann_loc: str,
+ ann_idx: Any,
+ *,
+ ann_metric: str,
+ dimensions: int,
+ element_count: int,
+ ) -> None:
+ """Save an hnswlib index into the Zarr hierarchy."""
+ if self.zarr_mode != "r+":
+ logger.debug("Skipping ANN index persistence on read-only store")
+ return
+ if ann_loc not in self.zw:
+ self.zw.create_group(ann_loc, overwrite=True)
+ save_ann_index(
+ as_zarr_group(self.zw[ann_loc], name=ann_loc),
+ ann_idx,
+ profile=self.storageProfile,
+ metric=ann_metric,
+ dimensions=dimensions,
+ element_count=element_count,
+ )
+
+ def _load_or_compute_norm_stats(
+ self,
+ normed_loc: str,
+ data: ChunkedArray,
+ reduction_method: str,
+ ) -> tuple[np.ndarray, np.ndarray]:
+ mu, sigma = np.ndarray([]), np.ndarray([])
+ if reduction_method not in ["pca", "manual"]:
+ return mu, sigma
+ normed_grp = as_zarr_group(self.zw[normed_loc], name=normed_loc)
+ need_mu = "mu" not in normed_grp
+ need_sigma = "sigma" not in normed_grp
+ if not need_mu:
+ mu = np.asarray(as_zarr_array(normed_grp["mu"], name="mu")[:])
+ if not need_sigma:
+ sigma = np.asarray(as_zarr_array(normed_grp["sigma"], name="sigma")[:])
+ if need_mu and need_sigma:
+ mu_raw, sigma_raw = data.mean_and_std(
+ nthreads=self.nthreads,
+ msg="Calculating normalization statistics",
+ )
+ mu = clean_array(mu_raw)
+ sigma = clean_array(sigma_raw, 1)
+ if self.zarr_mode == "r+":
+ g = create_zarr_dataset(normed_grp, "mu", (100000,), "f8", mu.shape)
+ g[:] = mu
+ g = create_zarr_dataset(
+ normed_grp, "sigma", (100000,), "f8", sigma.shape
+ )
+ g[:] = sigma
+ else:
+ logger.debug("Skipping mu/sigma persistence on read-only store")
+ elif need_mu:
+ mu = clean_array(
+ show_dask_progress(
+ data.mean(axis=0),
+ "Calculating mean of norm. data",
+ self.nthreads,
+ )
+ )
+ if self.zarr_mode == "r+":
+ g = create_zarr_dataset(normed_grp, "mu", (100000,), "f8", mu.shape)
+ g[:] = mu
+ else:
+ logger.debug("Skipping mu persistence on read-only store")
+ elif need_sigma:
+ sigma = clean_array(
+ show_dask_progress(
+ data.std(axis=0),
+ "Calculating std. dev. of norm. data",
+ self.nthreads,
+ ),
+ 1,
+ )
+ if self.zarr_mode == "r+":
+ g = create_zarr_dataset(
+ normed_grp, "sigma", (100000,), "f8", sigma.shape
+ )
+ g[:] = sigma
+ else:
+ logger.debug("Skipping sigma persistence on read-only store")
+ return mu, sigma
+
+ def _load_artifact_ann_stream(
+ self,
+ from_assay: str,
+ cell_key: str,
+ feat_key: str,
+ feat_scaling: bool,
+ neighbors_ref: ArtifactRef | None = None,
+ ) -> AnnStream | None:
+ def input_ref(owner: ArtifactRef, name: str) -> ArtifactRef:
+ value = (inspect_artifact(self.zw, owner).inputs or {}).get(name)
+ if not isinstance(value, dict):
+ raise ValueError(f"{owner.kind} has no {name!r} artifact input")
+ return ArtifactRef.from_dict(value)
+
+ correction_ref = None
+ if neighbors_ref is None:
+ state = read_assay_state(self.zw, from_assay)
+ if state is None or not state.matches(cell_key, feat_key):
+ return None
+ if (
+ state.normalized is None
+ or state.feature_scaling is None
+ or state.reduction is None
+ or state.ann_index is None
+ or state.neighbors is None
+ ):
+ raise KeyError("AssayState has no complete ANN stream")
+ normalized_ref = state.normalized
+ scaling_ref = state.feature_scaling
+ reduction_ref = state.reduction
+ ann_ref = state.ann_index
+ neighbors_ref = state.neighbors
+ correction_ref = state.batch_correction
+ else:
+ ann_ref = input_ref(neighbors_ref, "ann_index")
+ coordinates_ref = input_ref(neighbors_ref, "coordinates")
+ if coordinates_ref.kind == "batch_correction":
+ correction_ref = coordinates_ref
+ reduction_ref = input_ref(correction_ref, "reduction")
+ elif coordinates_ref.kind == "reduction":
+ reduction_ref = coordinates_ref
+ else:
+ raise ValueError("Unsupported neighbor coordinate artifact")
+ normalized_ref = input_ref(reduction_ref, "normalized")
+ scaling_ref = input_ref(reduction_ref, "feature_scaling")
+ normalized_group = artifact_group(self.zw, normalized_ref)
+ scaling_group = artifact_group(self.zw, scaling_ref)
+ reduction_group = artifact_group(self.zw, reduction_ref)
+ neighbors_group = artifact_group(self.zw, neighbors_ref)
+ data = ChunkedArray(
+ as_zarr_array(normalized_group["data"], name="data"),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ mu = np.asarray(as_zarr_array(scaling_group["mean"], name="mean")[:])
+ sigma = np.asarray(as_zarr_array(scaling_group["scale"], name="scale")[:])
+ loadings = (
+ np.asarray(as_zarr_array(reduction_group["loadings"], name="loadings")[:])
+ if "loadings" in reduction_group
+ else None
+ )
+ reduction_status = inspect_artifact(self.zw, reduction_ref)
+ reduction_params = reduction_status.parameters or {}
+ reduction_execution = reduction_status.execution_options or {}
+ operation = reduction_status.operation
+ reduction_method = (
+ {
+ "run_pca": "pca",
+ "run_lsi": "lsi",
+ "run_custom_reduction": "custom",
+ }.get(operation)
+ if operation is not None
+ else None
+ )
+ if reduction_method is None:
+ reduction_method = str(reduction_params.get("reduction_method", "pca"))
+ ann_params = inspect_artifact(self.zw, ann_ref).parameters or {}
+ neighbor_params = inspect_artifact(self.zw, neighbors_ref).parameters or {}
+ cached_scaling = bool(
+ reduction_params.get(
+ "feat_scaling",
+ reduction_method == "pca",
+ )
+ )
+ if cached_scaling != feat_scaling:
+ raise ValueError(
+ f"Reduction was built with feat_scaling={cached_scaling}, "
+ f"not {feat_scaling}"
+ )
+ corrected = None
+ if correction_ref is not None:
+ correction_group = artifact_group(self.zw, correction_ref)
+ corrected = ChunkedArray(
+ as_zarr_array(correction_group["data"], name="data"),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ ann_metric = str(ann_params.get("ann_metric", "l2"))
+ raw_dims = reduction_params.get("dims")
+ dims = int(
+ raw_dims
+ if raw_dims is not None
+ else loadings.shape[1]
+ if loadings is not None
+ else data.shape[1]
+ )
+ ann_idx = self._resolve_ann_index(
+ artifact_path(ann_ref),
+ ann_metric,
+ dims if dims > 0 else data.shape[1],
+ expected_count=int(data.shape[0]),
+ )
+ rebuilt_ann = ann_idx is None
+ neighbor_indices = as_zarr_array(
+ neighbors_group["indices"],
+ name="indices",
+ )
+ k_value = neighbor_params.get("k")
+ ann_obj = AnnStream(
+ data=data,
+ k=int(neighbor_indices.shape[1] if k_value is None else k_value),
+ n_cluster=2,
+ reduction_method=reduction_method,
+ dims=dims,
+ loadings=loadings,
+ use_for_pca=(
+ self.cells.fetch(
+ str(reduction_execution.get("pca_cell_key", cell_key)),
+ key=cell_key,
+ )
+ if reduction_method == "pca"
+ else np.ones(data.shape[0], dtype=bool)
+ ),
+ mu=mu,
+ sigma=sigma,
+ ann_metric=ann_metric,
+ ann_efc=int(ann_params.get("ann_efc", 50)),
+ ann_ef=int(ann_params.get("ann_ef", 50)),
+ ann_m=int(ann_params.get("ann_m", 48)),
+ nthreads=self.nthreads,
+ ann_parallel=bool(ann_params.get("ann_parallel", False)),
+ rand_state=int(ann_params.get("rand_state", 4466)),
+ do_kmeans_fit=False,
+ disable_scaling=not cached_scaling,
+ ann_idx=ann_idx,
+ lsi_skip_first=bool(
+ reduction_params.get(
+ "skip_first",
+ reduction_params.get("lsi_skip_first", True),
+ )
+ ),
+ lsi_params={},
+ harmonize=correction_ref is not None,
+ harmonized_data=corrected,
+ batches=None,
+ )
+ persisted_ann_threads = int(ann_params.get("parallel_threads") or 1)
+ AnnIndexStage.configure(
+ ann_obj.annIdx,
+ ef=int(ann_params.get("ann_ef", 50)),
+ threads=persisted_ann_threads,
+ )
+ ann_obj.annThreads = persisted_ann_threads
+ self._remember_ann_stream_path(
+ ann_obj,
+ artifact_path(ann_ref),
+ )
+ assert neighbors_ref is not None
+ self._remember_ann_stream_neighbors(
+ ann_obj,
+ artifact_path(neighbors_ref),
+ )
+ if rebuilt_ann and self.zarr_mode == "r+":
+ self._persist_ann_index(
+ artifact_path(ann_ref),
+ ann_obj.annIdx,
+ ann_metric=ann_metric,
+ dimensions=dims if dims > 0 else data.shape[1],
+ element_count=int(data.shape[0]),
+ )
+ return ann_obj
+
+ def _load_ann_stream(
+ self,
+ from_assay: str,
+ cell_key: str,
+ feat_key: str,
+ feat_scaling: bool = True,
+ knn_loc: str | None = None,
+ ) -> AnnStream:
+ """Load an AnnStream from an existing graph without recomputing KNN."""
+
+ artifact_neighbors = None
+ if knn_loc is not None:
+ try:
+ candidate = parse_artifact_path(knn_loc)
+ except ValueError:
+ pass
+ else:
+ if candidate.kind == "neighbors":
+ artifact_neighbors = candidate
+ self._artifact_chain_state(
+ candidate,
+ cell_key_override=cell_key,
+ feat_key_override=feat_key,
+ )
+ if knn_loc is None:
+ state = read_assay_state(self.zw, from_assay)
+ if (
+ state is not None
+ and state.matches(cell_key, feat_key)
+ and state.normalized is not None
+ ):
+ validate_normalized_artifact_selection(
+ self.zw,
+ state.normalized,
+ cell_key,
+ feat_key,
+ )
+ if knn_loc is None or artifact_neighbors is not None:
+ artifact_stream = self._load_artifact_ann_stream(
+ from_assay,
+ cell_key,
+ feat_key,
+ feat_scaling,
+ neighbors_ref=artifact_neighbors,
+ )
+ if artifact_stream is not None:
+ return artifact_stream
+ if knn_loc is None:
+ chain = lookup_latest_nearest_neighbor_paths(
+ self.zw,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ normed_loc = chain.normalized_group_path
+ if normed_loc not in self.zw:
+ raise KeyError(f"No normalized data at {normed_loc}")
+ reduction_loc = chain.reduction_group_path
+ ann_loc = chain.neighbor_index_group_path
+ knn_loc = chain.nearest_neighbors_group_path
+ else:
+ chain = nearest_neighbor_paths_from_loc(knn_loc)
+ validate_legacy_graph_selection(
+ self,
+ knn_loc,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ ann_loc = chain.neighbor_index_group_path
+ reduction_loc = chain.reduction_group_path
+ normed_loc = chain.normalized_group_path
+
+ if knn_loc not in self.zw:
+ raise KeyError(f"KNN graph not found at {knn_loc}")
+
+ (
+ ann_metric,
+ ann_efc,
+ ann_ef,
+ ann_m,
+ rand_state,
+ _,
+ _,
+ ) = parse_neighbor_index_group_path(ann_loc)
+ reduction_method, dims, pca_cell_key = parse_reduction_group_path(reduction_loc)
+ k = parse_nearest_neighbors_group_path(knn_loc)
+
+ data = ChunkedArray(
+ as_zarr_array(
+ as_zarr_group(self.zw[normed_loc], name=normed_loc)["data"],
+ name="data",
+ ),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ mu, sigma = self._load_or_compute_norm_stats(normed_loc, data, reduction_method)
+
+ loadings: NDArray[Any] | None = None
+ reduction_grp = as_zarr_group(self.zw[reduction_loc], name=reduction_loc)
+ if "reduction" in reduction_grp:
+ loadings = np.asarray(
+ as_zarr_array(reduction_grp["reduction"], name="reduction")[:]
+ )
+
+ ann_grp = as_zarr_group(self.zw[ann_loc], name=ann_loc)
+ cached_scaling = bool(ann_grp.attrs.get("featureScaling", True))
+ if cached_scaling != feat_scaling:
+ raise ValueError(
+ f"ANN index at {ann_loc} was built with featureScaling="
+ f"{cached_scaling}, not {feat_scaling}. Rebuild the graph."
+ )
+ harmonize = cast(bool, ann_grp.attrs.get("isHarmonized", False))
+ harmonized_data = None
+ batches = None
+ if harmonize and "harmonizedData" in reduction_grp:
+ harmonized_arr = as_zarr_array(
+ reduction_grp["harmonizedData"], name="harmonizedData"
+ )
+ harmonized_data = ChunkedArray(
+ harmonized_arr,
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ batch_columns = cast(list[str] | None, harmonized_arr.attrs.get("batches"))
+ if batch_columns:
+ batches = pd.DataFrame(
+ {
+ x: self.cells.fetch(x, key=cell_key).astype(object)
+ for x in batch_columns
+ }
+ )
+
+ temp_dim = dims if dims > 0 else data.shape[1]
+ ann_idx = self._resolve_ann_index(
+ ann_loc,
+ ann_metric,
+ temp_dim,
+ expected_count=int(data.shape[0]),
+ )
+ rebuilt_ann = ann_idx is None
+
+ use_for_pca = self.cells.fetch(pca_cell_key, key=cell_key)
+ ann_obj = AnnStream(
+ data=data,
+ k=k,
+ n_cluster=2,
+ reduction_method=reduction_method,
+ dims=dims,
+ loadings=loadings,
+ use_for_pca=use_for_pca,
+ mu=mu,
+ sigma=sigma,
+ ann_metric=ann_metric,
+ ann_efc=ann_efc,
+ ann_ef=ann_ef,
+ ann_m=ann_m,
+ nthreads=self.nthreads,
+ ann_parallel=False,
+ rand_state=rand_state,
+ do_kmeans_fit=False,
+ disable_scaling=not feat_scaling,
+ ann_idx=ann_idx,
+ lsi_skip_first=True,
+ lsi_params={},
+ harmonize=harmonize,
+ harmonized_data=harmonized_data,
+ batches=batches,
+ )
+ self._remember_ann_stream_path(ann_obj, ann_loc)
+ self._remember_ann_stream_neighbors(ann_obj, knn_loc)
+ if rebuilt_ann and self.zarr_mode == "r+":
+ self._persist_ann_index(
+ ann_loc,
+ ann_obj.annIdx,
+ ann_metric=ann_metric,
+ dimensions=int(temp_dim),
+ element_count=int(data.shape[0]),
+ )
+ if rebuilt_ann:
+ logger.info(f"Built ANN index for {data.shape[0]} cells")
+ else:
+ logger.info(f"Reused stored ANN index for {data.shape[0]} cells")
+ return ann_obj
+
+ def _get_graph_ncells_k(self, graph_loc: str) -> tuple[int, int]:
+ """
+
+ Args:
+ graph_loc:
+
+ Returns:
+
+ """
+ if is_integrated_graph_path(graph_loc, self._integratedGraphsLoc):
+ stored = lookup_stored_integrated_graph(self.zw, graph_loc)
+ if stored.n_cells is None or stored.n_neighbors is None:
+ raise KeyError(
+ f"Integrated graph at {graph_loc} is missing n_cells/n_neighbors"
+ )
+ return stored.n_cells, stored.n_neighbors
+ graph_group = as_zarr_group(self.zw[graph_loc], name=graph_loc)
+ if "n_cells" in graph_group.attrs and "n_neighbors" in graph_group.attrs:
+ return (
+ int(cast(int | float | str, graph_group.attrs["n_cells"])),
+ int(cast(int | float | str, graph_group.attrs["n_neighbors"])),
+ )
+ knn_loc = nearest_neighbors_group_path_from_cell_graph(graph_loc)
+ knn_grp = as_zarr_group(self.zw[knn_loc], name=knn_loc)
+ indices = as_zarr_array(knn_grp["indices"], name="indices")
+ return indices.shape[0], indices.shape[1]
+
+ def _store_to_sparse(
+ self, graph_loc: str, sparse_format: str = "csr", use_k: int | None = None
+ ) -> tuple[int, csr_matrix | coo_matrix]:
+ """
+
+ Args:
+ graph_loc:
+ sparse_format:
+ use_k:
+
+ Returns:
+
+ """
+ logger.debug(f"Loading graph from location: {graph_loc}")
+ store = as_zarr_group(self.zw[graph_loc], name=graph_loc)
+ n_cells, k = self._get_graph_ncells_k(graph_loc)
+ if use_k is None:
+ use_k = k
+ if use_k > k:
+ use_k = k
+ if use_k < 1:
+ use_k = 1
+ w = np.asarray(as_zarr_array(store["weights"], name="weights")[:])
+ e = np.asarray(as_zarr_array(store["edges"], name="edges")[:])
+ if use_k != k:
+ from ...neighbors.graph import take_nearest_per_row
+
+ w, e = take_nearest_per_row(w, e, n_cells, use_k)
+ if sparse_format == "csr":
+ return n_cells, csr_matrix(
+ (w, (e[:, 0], e[:, 1])), shape=(n_cells, n_cells)
+ )
+ else:
+ return n_cells, coo_matrix(
+ (w, (e[:, 0], e[:, 1])), shape=(n_cells, n_cells)
+ )
+
+ @staticmethod
+ def _resolve_local_cache_plan(
+ zarr_loc: Any,
+ group: zarr.Group,
+ local_cache: bool | str,
+ ) -> tuple[bool, str | None, bool]:
+ """Return whether to stage, cache base directory, and remove-on-success flag."""
+ if local_cache is False or not is_remote_datastore(zarr_loc, group):
+ return False, None, False
+ if local_cache is True or local_cache == "auto":
+ return True, tempfile.mkdtemp(prefix="scarf_local_cache_"), True
+ if isinstance(local_cache, str):
+ os.makedirs(local_cache, exist_ok=True)
+ return True, local_cache, False
+ raise TypeError(
+ f"local_cache must be 'auto', True, False, or a path string, got {local_cache!r}"
+ )
+
+ def _require_complete_artifact(
+ self,
+ ref: ArtifactRef,
+ kind: str,
+ *,
+ assay: str | None = None,
+ ) -> Any:
+ if ref.kind != kind:
+ raise ValueError(f"Expected {kind!r} artifact, got {ref.kind!r}")
+ if assay is not None and (ref.scope != "assay" or ref.assay != assay):
+ raise ValueError(f"Artifact must belong to assay {assay!r}")
+ return require_complete_artifact(self.zw, ref)
+
+ def _artifact_input_ref(
+ self,
+ ref: ArtifactRef,
+ name: str,
+ kind: str,
+ *,
+ require_input_complete: bool = True,
+ ) -> ArtifactRef:
+ status = self._require_complete_artifact(ref, ref.kind)
+ raw_ref = (status.inputs or {}).get(name)
+ if not isinstance(raw_ref, dict):
+ raise ValueError(f"{ref.kind} artifact has no {name!r} input")
+ input_ref = ArtifactRef.from_dict(raw_ref)
+ if require_input_complete:
+ self._require_complete_artifact(input_ref, kind)
+ return input_ref
+
+ def _resolve_selection_input(
+ self,
+ *,
+ metadata_group: zarr.Group,
+ column: str,
+ values: np.ndarray,
+ row_ids: np.ndarray,
+ scope: ArtifactScope,
+ kind: str,
+ assay: str | None,
+ invalidate_cache: bool,
+ ) -> ArtifactRef:
+ ref = None
+ if not invalidate_cache:
+ column_array = as_zarr_array(
+ metadata_group[column],
+ name=column,
+ )
+ raw_ref = column_array.attrs.get("source_artifact")
+ if isinstance(raw_ref, dict):
+ try:
+ candidate = ArtifactRef.from_dict(raw_ref)
+ status = inspect_artifact(self.zw, candidate)
+ except (KeyError, TypeError, ValueError):
+ pass
+ else:
+ if (
+ candidate.scope == scope
+ and candidate.kind == kind
+ and candidate.assay == assay
+ and status.complete
+ and (status.inputs or {}).get("ordered_row_ids_fingerprint")
+ == fingerprint_strings(row_ids)
+ ):
+ group = group_at(self.zw, status.path)
+ if "values" in group:
+ stored = np.asarray(
+ as_zarr_array(
+ group["values"],
+ name="values",
+ )[:]
+ )
+ if (
+ stored.ndim == 1
+ and stored.dtype == np.dtype(bool)
+ and values.ndim == 1
+ and values.dtype == np.dtype(bool)
+ and stored.shape == values.shape
+ and np.array_equal(stored, values)
+ ):
+ ref = candidate
+ if ref is None:
+ if getattr(self, "zarr_mode", "r+") != "r+":
+ raise PermissionError(
+ f"Selection provenance for {column!r} is unavailable "
+ "in the read-only store"
+ )
+ ref = resolve_selection_artifact(
+ self.zw,
+ scope=scope,
+ assay=assay,
+ kind=kind,
+ values=values,
+ row_ids=row_ids,
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column=column,
+ invalidate_cache=invalidate_cache,
+ )
+ if getattr(self, "zarr_mode", "r+") != "r+":
+ return ref
+ if scope == "assay" and assay is not None:
+ link_feature_data_column(
+ self._get_assay(assay).z,
+ column,
+ ref,
+ value_name="values",
+ default_display=categorical_display(values),
+ )
+ else:
+ target = as_zarr_array(
+ metadata_group[column],
+ name=column,
+ )
+ target.attrs["source_artifact"] = ref.to_dict()
+ target.attrs["source_value"] = "values"
+ return ref
+
+ def _selected_artifact(
+ self,
+ from_assay: str | None,
+ field_name: str,
+ kind: str,
+ ) -> ArtifactRef:
+ assay = from_assay or self._defaultAssay
+ if assay is None:
+ raise ValueError("No assay was provided and no default is configured")
+ state = read_assay_state(self.zw, assay)
+ if state is None:
+ raise KeyError(f"Assay {assay!r} has no selected artifact state")
+ ref = getattr(state, field_name)
+ if not isinstance(ref, ArtifactRef):
+ raise KeyError(f"AssayState has no selected {field_name} artifact")
+ self._require_complete_artifact(ref, kind, assay=assay)
+ return ref
+
+ def _artifact_chain_state(
+ self,
+ terminal: ArtifactRef,
+ *,
+ embedding_initialization: ArtifactRef | None = None,
+ named_results: dict[str, ArtifactRef] | None = None,
+ cell_key_override: str | None = None,
+ feat_key_override: str | None = None,
+ ) -> AssayState:
+ normalized = feature_scaling = reduction = None
+ batch_correction = ann_index = neighbors = connectivity_map = None
+ current = terminal
+ if current.kind == "connectivity_map":
+ connectivity_map = current
+ current = self._artifact_input_ref(
+ current,
+ "neighbors",
+ "neighbors",
+ )
+ if current.kind == "neighbors":
+ neighbors = current
+ ann_index = self._artifact_input_ref(
+ current,
+ "ann_index",
+ "ann_index",
+ )
+ neighbor_status = self._require_complete_artifact(
+ current,
+ "neighbors",
+ )
+ raw_coordinates = (neighbor_status.inputs or {}).get("coordinates")
+ if not isinstance(raw_coordinates, dict):
+ raise ValueError("Neighbors artifact has no coordinates input")
+ current = ArtifactRef.from_dict(raw_coordinates)
+ if current.kind not in {
+ "reduction",
+ "batch_correction",
+ "imported_coordinates",
+ }:
+ raise ValueError(
+ "Neighbor coordinates must be reduction, batch_correction, "
+ "or imported_coordinates"
+ )
+ self._require_complete_artifact(current, current.kind)
+ ann_coordinates = self._artifact_input_ref(
+ ann_index,
+ "coordinates",
+ current.kind,
+ )
+ if ann_coordinates != current:
+ raise ValueError("Neighbors and ANN index use different coordinates")
+ elif current.kind == "ann_index":
+ ann_index = current
+ status = self._require_complete_artifact(current, "ann_index")
+ raw_coordinates = (status.inputs or {}).get("coordinates")
+ if not isinstance(raw_coordinates, dict):
+ raise ValueError("ANN artifact has no coordinates input")
+ current = ArtifactRef.from_dict(raw_coordinates)
+ if current.kind == "imported_coordinates":
+ raise ValueError(
+ "Imported coordinates are not part of the normalized AssayState "
+ "graph chain; pass update_state=False"
+ )
+ if current.kind == "batch_correction":
+ batch_correction = current
+ current = self._artifact_input_ref(
+ current,
+ "reduction",
+ "reduction",
+ )
+ if current.kind == "reduction":
+ reduction = current
+ normalized = self._artifact_input_ref(
+ current,
+ "normalized",
+ "normalized",
+ )
+ feature_scaling = self._artifact_input_ref(
+ current,
+ "feature_scaling",
+ "feature_scaling",
+ )
+ elif current.kind == "normalized":
+ normalized = current
+ else:
+ raise ValueError(f"Cannot select graph state from {terminal.kind!r}")
+ assert normalized is not None
+ normalized_status = self._require_complete_artifact(
+ normalized,
+ "normalized",
+ )
+ execution = normalized_status.execution_options or {}
+ cell_key = execution.get("cell_key")
+ feat_key = execution.get("feat_key")
+ if not isinstance(cell_key, str) or not isinstance(feat_key, str):
+ raise ValueError("Normalized artifact is missing cell_key or feat_key")
+ if normalized.assay is None:
+ raise ValueError("Normalized artifact has no assay")
+ previous = read_assay_state(self.zw, normalized.assay)
+ if (cell_key_override is None) != (feat_key_override is None):
+ raise ValueError(
+ "cell_key and feat_key overrides must be provided together"
+ )
+ if cell_key_override is not None and feat_key_override is not None:
+ cell_key = cell_key_override
+ feat_key = feat_key_override
+ elif previous is not None and previous.normalized == normalized:
+ cell_key = previous.cell_key
+ feat_key = previous.feat_key
+ validate_normalized_artifact_selection(
+ self.zw,
+ normalized,
+ cell_key,
+ feat_key,
+ )
+ if embedding_initialization is None and reduction is not None:
+ if previous is not None and previous.embedding_initialization is not None:
+ previous_reduction = self._artifact_input_ref(
+ previous.embedding_initialization,
+ "reduction",
+ "reduction",
+ )
+ if previous_reduction == reduction:
+ embedding_initialization = previous.embedding_initialization
+ if connectivity_map is None and neighbors is not None:
+ if previous is not None and previous.connectivity_map is not None:
+ try:
+ previous_neighbors = self._artifact_input_ref(
+ previous.connectivity_map,
+ "neighbors",
+ "neighbors",
+ )
+ except (KeyError, RuntimeError, ValueError):
+ # Publishing a new chain is how a store recovers from a graph
+ # artifact that was removed, so a stale ref cannot block it.
+ previous_neighbors = None
+ if previous_neighbors == neighbors:
+ connectivity_map = previous.connectivity_map
+ state = AssayState(
+ assay=normalized.assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ normalized=normalized,
+ feature_scaling=feature_scaling,
+ reduction=reduction,
+ batch_correction=batch_correction,
+ ann_index=ann_index,
+ embedding_initialization=embedding_initialization,
+ neighbors=neighbors,
+ connectivity_map=connectivity_map,
+ named_results=named_results or {},
+ )
+ if named_results is None and previous is not None and previous.named_results:
+ carried = {}
+ for name, ref in previous.named_results.items():
+ try:
+ fits = named_result_mismatch(self.zw, name, ref, state) is None
+ except (KeyError, RuntimeError, TypeError, ValueError):
+ continue
+ if fits:
+ carried[name] = ref
+ if carried:
+ state = replace(state, named_results=carried)
+ return state
+
+ def _publish_current_artifact(
+ self,
+ ref: ArtifactRef,
+ *,
+ update_state: bool,
+ embedding_initialization: ArtifactRef | None = None,
+ named_results: dict[str, ArtifactRef] | None = None,
+ cell_key_override: str | None = None,
+ feat_key_override: str | None = None,
+ ) -> None:
+ if not update_state:
+ return
+ candidate = self._artifact_chain_state(
+ ref,
+ embedding_initialization=embedding_initialization,
+ named_results=named_results,
+ cell_key_override=cell_key_override,
+ feat_key_override=feat_key_override,
+ )
+ previous = read_assay_state(self.zw, candidate.assay)
+ field_name = {
+ "normalized": "normalized",
+ "reduction": "reduction",
+ "batch_correction": "batch_correction",
+ "ann_index": "ann_index",
+ "neighbors": "neighbors",
+ "connectivity_map": "connectivity_map",
+ }.get(ref.kind)
+ if (
+ previous is not None
+ and previous.matches(candidate.cell_key, candidate.feat_key)
+ and field_name is not None
+ and getattr(previous, field_name) == ref
+ and embedding_initialization is None
+ and named_results is None
+ ):
+ candidate = previous
+ write_assay_state(
+ self.zw,
+ candidate,
+ )
+
+ def _graph_cell_selection(
+ self,
+ graph_ref: ArtifactRef,
+ ) -> ArtifactRef:
+ if graph_ref.kind == "connectivity_map":
+ state = self._artifact_chain_state(graph_ref)
+ if state.normalized is None:
+ raise ValueError("Graph has no normalized input")
+ selection = self._artifact_input_ref(
+ state.normalized,
+ "cell_selection",
+ "cell_selection",
+ require_input_complete=False,
+ )
+ validate_cell_selection_artifact(self.zw, selection, state.cell_key)
+ return selection
+ if graph_ref.kind == "integrated_graph":
+ status = self._require_complete_artifact(
+ graph_ref,
+ "integrated_graph",
+ )
+ raw_selection = (status.inputs or {}).get("cell_selection")
+ if not isinstance(raw_selection, dict):
+ raise ValueError("Integrated graph has no shared cell selection")
+ selection = ArtifactRef.from_dict(raw_selection)
+ selection_status = inspect_artifact(self.zw, selection)
+ source_column = (selection_status.execution_options or {}).get(
+ "source_column"
+ )
+ if not isinstance(source_column, str):
+ if not selection_status.exists or not selection_status.complete:
+ validate_cell_selection_artifact(self.zw, selection, "")
+ raise ValueError("Integrated graph cell selection key is unavailable")
+ validate_cell_selection_artifact(
+ self.zw,
+ selection,
+ source_column,
+ )
+ return selection
+ raise ValueError("Graph ref must be connectivity_map or integrated_graph")
+
+ def _load_normalized_artifact(
+ self,
+ ref: ArtifactRef,
+ *,
+ batch_size: int | None,
+ ) -> ChunkedArray:
+ try:
+ cached = self._normalizedArtifactCache.get(ref)
+ except AttributeError:
+ cached = None
+ if cached is not None:
+ return cached
+ status = self._require_complete_artifact(ref, "normalized")
+ group = group_at(self.zw, status.path)
+ backing = as_zarr_array(group["data"], name="data")
+ return ChunkedArray(
+ backing,
+ block_size=_row_block(backing, batch_size),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+
+ @contextmanager
+ def _cache_normalized_artifact(
+ self,
+ ref: ArtifactRef,
+ local_cache: bool | str,
+ batch_size: int | None,
+ ) -> Iterator[None]:
+ try:
+ already_cached = ref in self._normalizedArtifactCache
+ except AttributeError:
+ already_cached = False
+ if already_cached:
+ yield
+ return
+ enabled, cache_base, remove_on_success = self._resolve_local_cache_plan(
+ self.zarr_loc,
+ self.z,
+ local_cache,
+ )
+ if not enabled:
+ yield
+ return
+ if cache_base is None:
+ raise RuntimeError("Local cache path is missing")
+ status = self._require_complete_artifact(ref, "normalized")
+ source_group = group_at(self.zw, status.path)
+ source = as_zarr_array(source_group["data"], name="data")
+ cache_path = os.path.join(
+ cache_base,
+ ref.artifact_id,
+ "normed.zarr",
+ )
+ os.makedirs(os.path.dirname(cache_path), exist_ok=True)
+ staged = create_or_open_staged_normed_array(
+ cache_path,
+ (int(source.shape[0]), int(source.shape[1])),
+ )
+ if not bool(staged.attrs.get("complete", False)):
+ copy_zarr_array(
+ source,
+ staged,
+ msg="Staging normalized data locally",
+ resources=self.resources,
+ )
+ staged.attrs["complete"] = True
+ try:
+ cache = self._normalizedArtifactCache
+ except AttributeError:
+ cache = {}
+ self._normalizedArtifactCache = cache
+ cache[ref] = ChunkedArray(
+ staged,
+ block_size=_row_block(staged, batch_size),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ try:
+ yield
+ finally:
+ cache.pop(ref, None)
+ if remove_on_success:
+ import shutil
+
+ shutil.rmtree(cache_base, ignore_errors=True)
+
+ def _load_reduction_stream(
+ self,
+ reduction_ref: ArtifactRef,
+ *,
+ batch_size: int | None,
+ ) -> tuple[ReductionTransform, LazyTransformStream]:
+ status = self._require_complete_artifact(
+ reduction_ref,
+ "reduction",
+ )
+ parameters = status.parameters or {}
+ operation = status.operation
+ method = (
+ {
+ "run_pca": "pca",
+ "run_lsi": "lsi",
+ "run_custom_reduction": "custom",
+ }.get(operation)
+ if operation is not None
+ else None
+ )
+ if method is None:
+ method = str(parameters.get("reduction_method", "pca"))
+ normalized_ref = self._artifact_input_ref(
+ reduction_ref,
+ "normalized",
+ "normalized",
+ )
+ scaling_ref = self._artifact_input_ref(
+ reduction_ref,
+ "feature_scaling",
+ "feature_scaling",
+ )
+ normalized = self._load_normalized_artifact(
+ normalized_ref,
+ batch_size=batch_size,
+ )
+ block_rows = int(normalized.chunksize[0])
+ scaling_group = artifact_group(self.zw, scaling_ref)
+ mu = np.asarray(as_zarr_array(scaling_group["mean"], name="mean")[:])
+ sigma = np.asarray(as_zarr_array(scaling_group["scale"], name="scale")[:])
+ reduction_group = group_at(self.zw, status.path)
+ loadings = (
+ np.asarray(
+ as_zarr_array(
+ reduction_group["loadings"],
+ name="loadings",
+ )[:]
+ )
+ if "loadings" in reduction_group
+ else None
+ )
+ raw_dims = parameters.get("dims")
+ dims = int(
+ raw_dims
+ if raw_dims is not None
+ else loadings.shape[1]
+ if loadings is not None
+ else 0
+ )
+ transform = ReductionTransform(
+ data=normalized,
+ method=method,
+ dims=dims,
+ loadings=loadings,
+ use_for_pca=np.ones(normalized.shape[0], dtype=bool),
+ mu=mu,
+ sigma=sigma,
+ batch_size=block_rows,
+ nthreads=self.nthreads,
+ rand_state=int(parameters.get("rand_state", 4466)),
+ disable_scaling=not bool(parameters.get("feat_scaling", method == "pca")),
+ lsi_skip_first=bool(
+ parameters.get(
+ "skip_first",
+ parameters.get("lsi_skip_first", True),
+ )
+ ),
+ lsi_params={},
+ )
+ stream = LazyTransformStream(
+ data=normalized,
+ transform=transform.transform,
+ nthreads=self.nthreads,
+ batch_size=block_rows,
+ )
+ return transform, stream
+
+ def _coordinate_source(
+ self,
+ coordinates: ArtifactRef,
+ *,
+ batch_size: int | None,
+ ) -> tuple[CoordinateSource, int, int]:
+ if coordinates.kind == "imported_coordinates":
+ status = self._require_complete_artifact(
+ coordinates,
+ "imported_coordinates",
+ )
+ execution = status.execution_options or {}
+ cell_key = execution.get("cell_key")
+ if not isinstance(cell_key, str) or not cell_key:
+ raise ValueError(
+ "Imported-coordinate artifact has no cell selection key"
+ )
+ validate_imported_coordinates_artifact(
+ self.zw,
+ coordinates,
+ cell_key=cell_key,
+ )
+ group = group_at(self.zw, status.path)
+ backing = as_zarr_array(group["data"], name="data")
+ data = ChunkedArray(
+ backing,
+ block_size=_row_block(backing, batch_size),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ logger.debug(
+ f"Using imported coordinates from {status.path} "
+ f"(shape={data.shape[0]} x {data.shape[1]})"
+ )
+ return (
+ ChunkedCoordinateStream(data, self.nthreads),
+ int(data.shape[0]),
+ int(data.shape[1]),
+ )
+ if coordinates.kind == "batch_correction":
+ status = self._require_complete_artifact(
+ coordinates,
+ "batch_correction",
+ )
+ group = group_at(self.zw, status.path)
+ backing = as_zarr_array(group["data"], name="data")
+ data = ChunkedArray(
+ backing,
+ block_size=_row_block(backing, batch_size),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ logger.debug(
+ f"Using materialized batch-correction coordinates from "
+ f"{status.path} (shape={data.shape[0]} x {data.shape[1]})"
+ )
+ return (
+ ChunkedCoordinateStream(data, self.nthreads),
+ int(data.shape[0]),
+ int(data.shape[1]),
+ )
+ if coordinates.kind == "reduction":
+ status = self._require_complete_artifact(
+ coordinates,
+ "reduction",
+ )
+ group = group_at(self.zw, status.path)
+ method_label = {
+ "run_pca": "PCA",
+ "run_lsi": "LSI",
+ "run_custom_reduction": "custom reduction",
+ }.get(
+ status.operation or "",
+ str((status.parameters or {}).get("reduction_method", "reduction")),
+ )
+ if "data" in group:
+ backing = as_zarr_array(group["data"], name="data")
+ data = ChunkedArray(
+ backing,
+ block_size=_row_block(backing, batch_size),
+ nthreads=self.nthreads,
+ resources=self.resources,
+ )
+ logger.debug(
+ f"Using materialized {method_label} coordinates from "
+ f"{status.path} (shape={data.shape[0]} x {data.shape[1]})"
+ )
+ return (
+ ChunkedCoordinateStream(data, self.nthreads),
+ int(data.shape[0]),
+ int(data.shape[1]),
+ )
+ logger.warning(
+ f"Materialized {method_label} coordinates missing at "
+ f"{status.path} (no 'data' array); falling back to on-the-fly "
+ f"projection from normalized data. Downstream ANN, neighbors, "
+ f"and embedding init will re-read normalized features instead "
+ f"of stored reduced coordinates."
+ )
+ transform, stream = self._load_reduction_stream(
+ coordinates,
+ batch_size=batch_size,
+ )
+ dims = (
+ int(transform.dims)
+ if transform.dims is not None and transform.dims > 0
+ else int(stream.data.shape[1])
+ )
+ return stream, int(stream.data.shape[0]), dims
+ raise ValueError(
+ "Coordinates must reference reduction, batch_correction, "
+ "or imported_coordinates"
+ )
+
+ def _plan_assay_artifact(
+ self,
+ assay: str,
+ arguments: Any,
+ *,
+ required_arrays: tuple[str | ArrayRequirement, ...] = (),
+ invalidate_cache: bool,
+ reuse_validator: Callable[[ArtifactRef, zarr.Group], bool] | None = None,
+ ) -> PlannedArtifact:
+ record = arguments.to_record()
+ execution_options = dict(record.execution_options)
+ try:
+ execution_options.update(self._artifactExecutionContext)
+ except AttributeError:
+ pass
+ return plan_artifact(
+ self.zw,
+ scope="assay",
+ assay=assay,
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=execution_options,
+ invalidate_cache=invalidate_cache,
+ required_arrays=required_arrays,
+ reuse_validator=reuse_validator,
+ )
+
+ @contextmanager
+ def _artifact_execution_context(
+ self,
+ options: dict[str, Any],
+ ) -> Iterator[None]:
+ try:
+ previous = self._artifactExecutionContext
+ except AttributeError:
+ previous = {}
+ self._artifactExecutionContext = {**previous, **options}
+ try:
+ yield
+ finally:
+ self._artifactExecutionContext = previous
+
+ def run_normalization(
+ self,
+ from_assay: str | None = None,
+ cell_key: str = "I",
+ feat_key: str | None = None,
+ *,
+ log_transform: bool | None = None,
+ renormalize_subset: bool | None = None,
+ update_state: bool = True,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Normalize one cell and feature selection into an artifact.
+
+ An identical operation reuses a complete artifact unless
+ ``invalidate_cache`` is true. The returned reference can be passed
+ directly to a reduction method.
+
+ Args:
+ from_assay: Assay to normalize. Uses the default assay when omitted.
+ cell_key: Boolean cell metadata column selecting rows.
+ feat_key: Boolean feature metadata key selecting columns. For a
+ non-``I`` key, the stored feature column is
+ ``{cell_key}__{feat_key}``.
+ log_transform: Whether to apply the assay log transform. When
+ omitted, reuse the selected artifact setting or default to
+ true. ATAC defaults to false and rejects true.
+ renormalize_subset: Whether to recompute the normalization
+ denominator from selected features. When omitted, reuse the
+ selected setting. ATAC defaults to false; other assays default
+ to true.
+ update_state: Select the result as the assay's current normalized
+ artifact.
+ invalidate_cache: Force a new artifact instead of reusing an
+ identical complete result.
+
+ Returns:
+ Reference to the normalized artifact.
+
+ Raises:
+ KeyError: If the feature selection does not exist.
+ TypeError: If a selection is not boolean.
+ ValueError: If the assay, feature key, or selected data is invalid.
+ """
+ assay_name = from_assay or self._defaultAssay
+ if assay_name is None:
+ raise ValueError("No assay was provided and no default is configured")
+ if feat_key is None:
+ raise ValueError("feat_key is required for normalization")
+ assay = self._get_assay(assay_name)
+ stored_feat_key = feat_key if feat_key == "I" else f"{cell_key}__{feat_key}"
+ if stored_feat_key not in assay.feats.columns:
+ raise KeyError(
+ f"Feature selection column {stored_feat_key!r} does not exist"
+ )
+ cell_values = np.asarray(self.cells.fetch_all(cell_key))
+ feature_values = np.asarray(assay.feats.fetch_all(stored_feat_key))
+ if cell_values.dtype != bool or feature_values.dtype != bool:
+ raise TypeError("Cell and feature selections must be boolean")
+ n_cells = int(cell_values.sum())
+ n_features = int(feature_values.sum())
+ if n_cells < 1 or n_features < 1:
+ raise ValueError("Normalization requires selected cells and features")
+ state = read_assay_state(self.zw, assay_name)
+ stored_parameters: dict[str, Any] = {}
+ if (
+ state is not None
+ and state.matches(cell_key, feat_key)
+ and state.normalized is not None
+ ):
+ stored_parameters = (
+ inspect_artifact(self.zw, state.normalized).parameters or {}
+ )
+ from ...assay import ATACassay
+
+ if isinstance(assay, ATACassay):
+ stored_method_is_current = stored_parameters.get(
+ "normalization_method"
+ ) == serialize_artifact_value(assay.normMethod)
+ if log_transform is None:
+ log_transform = (
+ bool(stored_parameters.get("log_transform", False))
+ if stored_method_is_current
+ else False
+ )
+ elif not isinstance(log_transform, (bool, np.bool_)):
+ raise TypeError("log_transform must be a boolean")
+ if log_transform:
+ raise ValueError(
+ "ATAC TF-IDF does not support log_transform; use False"
+ )
+ else:
+ log_transform = False
+ if renormalize_subset is None:
+ renormalize_subset = (
+ bool(stored_parameters.get("renormalize_subset", False))
+ if stored_method_is_current
+ else False
+ )
+ elif not isinstance(renormalize_subset, (bool, np.bool_)):
+ raise TypeError("renormalize_subset must be a boolean")
+ else:
+ renormalize_subset = bool(renormalize_subset)
+ else:
+ if log_transform is None:
+ log_transform = bool(stored_parameters.get("log_transform", True))
+ if renormalize_subset is None:
+ renormalize_subset = bool(
+ stored_parameters.get("renormalize_subset", True)
+ )
+ cell_data = as_zarr_group(self.zw["cellData"], name="cellData")
+ feature_data = as_zarr_group(
+ assay.z["featureData"],
+ name="featureData",
+ )
+ cell_selection = self._resolve_selection_input(
+ metadata_group=cell_data,
+ column=cell_key,
+ values=cell_values,
+ row_ids=np.asarray(self.cells.fetch_all("ids")),
+ scope="datastore",
+ kind="cell_selection",
+ assay=None,
+ invalidate_cache=invalidate_cache,
+ )
+ feature_selection = self._resolve_selection_input(
+ metadata_group=feature_data,
+ column=stored_feat_key,
+ values=feature_values,
+ row_ids=np.asarray(assay.feats.fetch_all("ids")),
+ scope="assay",
+ kind="feature_selection",
+ assay=assay_name,
+ invalidate_cache=invalidate_cache,
+ )
+ normalization_method = assay.normMethod
+ if callable(normalization_method):
+ method_qualname = str(getattr(normalization_method, "__qualname__", ""))
+ if (
+ "" in method_qualname or "" in method_qualname
+ ) and getattr(normalization_method, "artifact_identity", None) is None:
+ raise ValueError(
+ "Dynamic normalization callables must define "
+ "artifact_identity for provenance"
+ )
+ raw_size_factor = getattr(assay, "sf", None)
+ arguments = NormalizationArguments(
+ from_assay=assay_name,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ cell_selection=cell_selection,
+ feature_selection=feature_selection,
+ normalization_method=normalization_method,
+ size_factor=(
+ float(cast(int | float, raw_size_factor))
+ if raw_size_factor is not None
+ else None
+ ),
+ log_transform=log_transform,
+ renormalize_subset=renormalize_subset,
+ update_state=update_state,
+ invalidate_cache=invalidate_cache,
+ )
+ planned = self._plan_assay_artifact(
+ assay_name,
+ arguments,
+ required_arrays=(
+ ArrayRequirement(
+ "data",
+ shape=(n_cells, n_features),
+ dtype=np.float32,
+ ),
+ ArrayRequirement(
+ "feature_sum",
+ shape=(n_features,),
+ dtype=np.float64,
+ ),
+ ArrayRequirement(
+ "feature_squared_sum",
+ shape=(n_features,),
+ dtype=np.float64,
+ ),
+ ),
+ invalidate_cache=invalidate_cache,
+ )
+ self._ensure_dataset_fingerprint(assay_name)
+ if not planned.reused:
+ group = start_artifact(self.zw, planned)
+ relative_path = artifact_path(planned.ref).removeprefix(f"{assay_name}/")
+ assay.save_normalized_data(
+ cell_key,
+ feat_key,
+ relative_path,
+ log_transform,
+ renormalize_subset,
+ False,
+ artifact_mode=True,
+ )
+ finish_artifact(group, planned)
+ self._publish_current_artifact(
+ planned.ref,
+ update_state=update_state,
+ cell_key_override=cell_key,
+ feat_key_override=feat_key,
+ )
+ action = "Reused" if planned.reused else "Stored"
+ logger.info(
+ f"{action} normalized data for {n_cells} cells and {n_features} features"
+ )
+ return planned.ref
+
+ def _run_reduction_artifact(
+ self,
+ *,
+ method: str,
+ normalized: ArtifactRef | None,
+ from_assay: str | None,
+ dims: int,
+ pca_cell_key: str | None,
+ feat_scaling: bool,
+ lsi_skip_first: bool,
+ custom_loadings: np.ndarray | None,
+ rand_state: int,
+ batch_size: int | None,
+ local_cache: bool | str,
+ show_elbow_plot: bool,
+ update_state: bool,
+ invalidate_cache: bool,
+ lsi_solver: Literal["streaming", "materialized"] = "streaming",
+ lsi_n_iter: int = 5,
+ lsi_n_oversamples: int = 10,
+ ) -> ArtifactRef:
+ requested_dims = _positive_integer(dims, "dims")
+ if batch_size is not None:
+ _positive_integer(batch_size, "batch_size")
+ normalized_ref = normalized or self._selected_artifact(
+ from_assay,
+ "normalized",
+ "normalized",
+ )
+ status = self._require_complete_artifact(normalized_ref, "normalized")
+ group = group_at(self.zw, status.path)
+ data = as_zarr_array(group["data"], name="data")
+ effective_batch_size = _row_block(
+ data,
+ batch_size,
+ minimum=(requested_dims + 1 if method == "pca" else None),
+ )
+ if method == "lsi" and lsi_solver == "streaming":
+ memory_limited_rows = _streaming_lsi_block_rows(
+ data,
+ self.resources,
+ n_components=requested_dims + int(lsi_skip_first),
+ n_oversamples=lsi_n_oversamples,
+ )
+ if effective_batch_size > memory_limited_rows:
+ logger.warning(
+ f"Reducing LSI batch_size from {effective_batch_size} to "
+ f"{memory_limited_rows} rows to honor the memory budget"
+ )
+ effective_batch_size = memory_limited_rows
+ with self._artifact_execution_context({"local_cache": local_cache}):
+ with self._cache_normalized_artifact(
+ normalized_ref,
+ local_cache,
+ effective_batch_size,
+ ):
+ return self._run_reduction_artifact_impl(
+ method=method,
+ normalized=normalized_ref,
+ from_assay=from_assay,
+ dims=requested_dims,
+ pca_cell_key=pca_cell_key,
+ feat_scaling=feat_scaling,
+ lsi_skip_first=lsi_skip_first,
+ custom_loadings=custom_loadings,
+ rand_state=rand_state,
+ batch_size=effective_batch_size,
+ show_elbow_plot=show_elbow_plot,
+ update_state=update_state,
+ invalidate_cache=invalidate_cache,
+ lsi_solver=lsi_solver,
+ lsi_n_iter=lsi_n_iter,
+ lsi_n_oversamples=lsi_n_oversamples,
+ )
+
+ def _run_reduction_artifact_impl(
+ self,
+ *,
+ method: str,
+ normalized: ArtifactRef | None,
+ from_assay: str | None,
+ dims: int,
+ pca_cell_key: str | None,
+ feat_scaling: bool,
+ lsi_skip_first: bool,
+ custom_loadings: np.ndarray | None,
+ rand_state: int,
+ batch_size: int | None,
+ show_elbow_plot: bool,
+ update_state: bool,
+ invalidate_cache: bool,
+ lsi_solver: Literal["streaming", "materialized"] = "streaming",
+ lsi_n_iter: int = 5,
+ lsi_n_oversamples: int = 10,
+ ) -> ArtifactRef:
+ normalized_ref = normalized or self._selected_artifact(
+ from_assay,
+ "normalized",
+ "normalized",
+ )
+ normalized_status = self._require_complete_artifact(
+ normalized_ref,
+ "normalized",
+ )
+ if normalized_ref.assay is None:
+ raise ValueError("Normalized artifact has no assay")
+ assay_name = normalized_ref.assay
+ execution = normalized_status.execution_options or {}
+ cell_key = execution.get("cell_key")
+ feat_key = execution.get("feat_key")
+ if not isinstance(cell_key, str) or not isinstance(feat_key, str):
+ raise ValueError("Normalized artifact has no cell_key or feat_key")
+ state = read_assay_state(self.zw, assay_name)
+ if state is not None and state.normalized == normalized_ref:
+ cell_key = state.cell_key
+ feat_key = state.feat_key
+ validate_normalized_artifact_selection(
+ self.zw,
+ normalized_ref,
+ cell_key,
+ feat_key,
+ )
+ data_group = as_zarr_group(
+ self.zw[normalized_status.path],
+ name=normalized_status.path,
+ )
+ data_array = as_zarr_array(data_group["data"], name="data")
+ n_cells, n_features = map(int, data_array.shape)
+ effective_batch_size = min(
+ _positive_integer(batch_size, "batch_size"),
+ n_cells,
+ )
+ effective_dims = _positive_integer(dims, "dims")
+ if custom_loadings is not None:
+ if custom_loadings.ndim != 2:
+ raise ValueError("Custom loadings must be a two-dimensional matrix")
+ if custom_loadings.shape[0] != n_features:
+ raise ValueError("Custom loadings rows must match normalized features")
+ effective_dims = int(custom_loadings.shape[1])
+ if effective_dims < 1:
+ raise ValueError("Custom loadings must contain at least one dimension")
+ pca_key = pca_cell_key or cell_key
+ pca_selection = None
+ pca_use_values: np.ndarray | None = None
+ if method == "pca":
+ pca_use_values = np.asarray(self.cells.fetch(pca_key, key=cell_key))
+ if pca_use_values.dtype != bool or pca_use_values.shape != (n_cells,):
+ raise TypeError(
+ "pca_cell_key must select one boolean value per normalized cell"
+ )
+ selected_pca_cells = int(pca_use_values.sum())
+ if selected_pca_cells < effective_dims + 1:
+ raise ValueError("PCA requires at least dims + 1 selected cells")
+ if n_features < effective_dims + 1:
+ raise ValueError("PCA requires at least dims + 1 selected features")
+ if effective_batch_size < effective_dims + 1:
+ raise ValueError("PCA batch_size must be at least dims + 1")
+ pca_values = np.asarray(self.cells.fetch_all(pca_key))
+ if pca_values.dtype != bool:
+ raise TypeError("pca_cell_key must reference a boolean column")
+ cell_data = as_zarr_group(
+ self.zw["cellData"],
+ name="cellData",
+ )
+ pca_selection = self._resolve_selection_input(
+ metadata_group=cell_data,
+ column=pca_key,
+ values=pca_values,
+ row_ids=np.asarray(self.cells.fetch_all("ids")),
+ scope="datastore",
+ kind="cell_selection",
+ assay=None,
+ invalidate_cache=invalidate_cache,
+ )
+ elif method == "lsi":
+ required_rank = effective_dims + int(lsi_skip_first)
+ if required_rank > min(n_cells, n_features):
+ raise ValueError(
+ "LSI dimensions, including the skipped component, exceed "
+ "the normalized matrix rank"
+ )
+ enabled_scaling = method == "pca" and feat_scaling
+ scaling_arguments = FeatureScalingArguments(
+ normalized=normalized_ref,
+ enabled=enabled_scaling,
+ batch_size=effective_batch_size,
+ invalidate_cache=invalidate_cache,
+ )
+ scaling_shape = n_features if enabled_scaling else 0
+ scaling_plan = self._plan_assay_artifact(
+ assay_name,
+ scaling_arguments,
+ required_arrays=(
+ ArrayRequirement(
+ "mean",
+ shape=(scaling_shape,),
+ dtype=np.float64,
+ ),
+ ArrayRequirement(
+ "scale",
+ shape=(scaling_shape,),
+ dtype=np.float64,
+ ),
+ ),
+ invalidate_cache=invalidate_cache,
+ )
+ normalized_data = self._load_normalized_artifact(
+ normalized_ref,
+ batch_size=effective_batch_size,
+ )
+ if scaling_plan.reused:
+ scaling_group = reused_artifact_group(
+ self.zw,
+ scaling_plan,
+ )
+ mu = np.asarray(as_zarr_array(scaling_group["mean"], name="mean")[:])
+ sigma = np.asarray(as_zarr_array(scaling_group["scale"], name="scale")[:])
+ else:
+ if enabled_scaling:
+ if "feature_sum" in data_group and "feature_squared_sum" in data_group:
+ total = np.asarray(
+ as_zarr_array(
+ data_group["feature_sum"],
+ name="feature_sum",
+ )[:],
+ dtype=np.float64,
+ )
+ squared_total = np.asarray(
+ as_zarr_array(
+ data_group["feature_squared_sum"],
+ name="feature_squared_sum",
+ )[:],
+ dtype=np.float64,
+ )
+ mu_raw = total / n_cells
+ variance = squared_total / n_cells - np.square(mu_raw)
+ sigma_raw = np.sqrt(np.clip(variance, 0, None))
+ else:
+ mu_raw, sigma_raw = normalized_data.mean_and_std(
+ nthreads=self.nthreads,
+ msg="Calculating normalization statistics",
+ )
+ mu = clean_array(mu_raw)
+ sigma = clean_array(sigma_raw, 1)
+ else:
+ mu = np.array([], dtype=np.float64)
+ sigma = np.array([], dtype=np.float64)
+ scaling_group = start_artifact(self.zw, scaling_plan)
+ mean_array = create_zarr_dataset(
+ scaling_group,
+ "mean",
+ (100000,),
+ "f8",
+ mu.shape,
+ )
+ mean_array[:] = mu
+ scale_array = create_zarr_dataset(
+ scaling_group,
+ "scale",
+ (100000,),
+ "f8",
+ sigma.shape,
+ )
+ scale_array[:] = sigma
+ finish_artifact(scaling_group, scaling_plan)
+ if method == "pca":
+ assert pca_selection is not None
+ arguments: Any = PcaArguments(
+ normalized=normalized_ref,
+ feature_scaling=scaling_plan.ref,
+ pca_cell_selection=pca_selection,
+ pca_cell_key=pca_key,
+ dims=effective_dims,
+ feat_scaling=feat_scaling,
+ batch_size=effective_batch_size,
+ show_elbow_plot=show_elbow_plot,
+ update_state=update_state,
+ invalidate_cache=invalidate_cache,
+ )
+ elif method == "lsi":
+ arguments = LsiArguments(
+ normalized=normalized_ref,
+ feature_scaling=scaling_plan.ref,
+ dims=effective_dims,
+ skip_first=lsi_skip_first,
+ rand_state=rand_state,
+ solver=lsi_solver,
+ n_iter=lsi_n_iter,
+ n_oversamples=lsi_n_oversamples,
+ batch_size=effective_batch_size,
+ update_state=update_state,
+ invalidate_cache=invalidate_cache,
+ )
+ else:
+ if custom_loadings is None:
+ raise ValueError("Custom reduction requires loadings")
+ arguments = CustomReductionArguments(
+ normalized=normalized_ref,
+ feature_scaling=scaling_plan.ref,
+ loadings=custom_loadings,
+ update_state=update_state,
+ invalidate_cache=invalidate_cache,
+ )
+ required_arrays: tuple[str | ArrayRequirement, ...] = (
+ ArrayRequirement(
+ "loadings",
+ shape=(n_features, effective_dims),
+ dtype=np.float64,
+ ),
+ ArrayRequirement(
+ "data",
+ shape=(n_cells, effective_dims),
+ dtype=np.float32,
+ ),
+ )
+ planned = self._plan_assay_artifact(
+ assay_name,
+ arguments,
+ required_arrays=required_arrays,
+ invalidate_cache=invalidate_cache,
+ )
+ transform = None
+ if not planned.reused:
+ use_for_pca = (
+ pca_use_values if method == "pca" else np.ones(n_cells, dtype=bool)
+ )
+ assert use_for_pca is not None
+ transform = ReductionTransform(
+ data=normalized_data,
+ method=method,
+ dims=effective_dims,
+ loadings=custom_loadings,
+ use_for_pca=use_for_pca,
+ mu=mu,
+ sigma=sigma,
+ batch_size=effective_batch_size,
+ nthreads=self.nthreads,
+ rand_state=rand_state,
+ disable_scaling=not feat_scaling,
+ lsi_skip_first=lsi_skip_first,
+ lsi_params={
+ "solver": lsi_solver,
+ "n_iter": lsi_n_iter,
+ "n_oversamples": lsi_n_oversamples,
+ },
+ )
+ reduction_group = start_artifact(self.zw, planned)
+ if transform.loadings is not None:
+ output = create_zarr_dataset(
+ reduction_group,
+ "loadings",
+ normalized_data.chunksize,
+ "f8",
+ transform.loadings.shape,
+ )
+ output[:, :] = transform.loadings
+ score_spec = row_sharded_array_spec(
+ (n_cells, effective_dims),
+ np.float32,
+ profile=resolve_storage_profile(reduction_group.store),
+ band_rows=min(n_cells, 1_000_000),
+ zarr_format=_group_zarr_format(reduction_group),
+ fill_value=0.0,
+ )
+ scores = create_numeric_array(
+ reduction_group,
+ "data",
+ score_spec,
+ )
+
+ def score_blocks() -> Iterator[np.ndarray]:
+ for block in normalized_data.stream_blocks(
+ nthreads=self.nthreads,
+ msg="Calculating reduced coordinates",
+ ):
+ yield np.asarray(
+ transform.transform(block),
+ dtype=np.float32,
+ )
+
+ write_dense_from_row_batches(
+ scores,
+ score_blocks(),
+ dtype=np.float32,
+ msg="Writing reduced coordinates",
+ resources=self.resources,
+ )
+ finish_artifact(reduction_group, planned)
+ if show_elbow_plot and method == "pca":
+ from ...plotting import elbow
+
+ if transform is None or transform.pca is None:
+ logger.warning("PCA was not fitted so no elbow plot is available")
+ else:
+ elbow(
+ variance_explained=(100 * transform.pca.explained_variance_ratio_),
+ show=True,
+ )
+ self._publish_current_artifact(
+ planned.ref,
+ update_state=update_state,
+ )
+ action = "Reused" if planned.reused else "Stored"
+ logger.info(
+ f"{action} {method.upper()} reduction for {n_cells} cells "
+ f"with {effective_dims} dimensions"
+ )
+ return planned.ref
+
+ def run_pca(
+ self,
+ normalized: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ dims: int = 25,
+ pca_cell_key: str | None = None,
+ feat_scaling: bool = True,
+ batch_size: int | None = None,
+ local_cache: bool | str = "auto",
+ show_elbow_plot: bool = False,
+ update_state: bool = True,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Fit or reuse PCA for a normalized artifact.
+
+ Args:
+ normalized: Normalized artifact to reduce. Uses the selected assay
+ state when omitted.
+ from_assay: Assay used to resolve the selected normalized artifact.
+ dims: Requested number of principal components.
+ pca_cell_key: Optional boolean cell column used to fit PCA while
+ projecting every selected cell.
+ feat_scaling: Whether to standardize features before fitting PCA.
+ batch_size: Number of selected cells processed per block. When
+ omitted, whole stored row bands are combined as needed to fit
+ at least ``dims + 1`` rows. An explicit smaller value is
+ expanded to that aligned minimum with a warning.
+ local_cache: Local staging policy for normalized data on remote
+ stores.
+ show_elbow_plot: Whether to display explained variance after a new
+ PCA fit.
+ update_state: Select the result as the current reduction.
+ invalidate_cache: Force a new reduction artifact.
+
+ Returns:
+ Reference to the PCA reduction artifact.
+ """
+ return self._run_reduction_artifact(
+ method="pca",
+ normalized=normalized,
+ from_assay=from_assay,
+ dims=dims,
+ pca_cell_key=pca_cell_key,
+ feat_scaling=feat_scaling,
+ lsi_skip_first=False,
+ custom_loadings=None,
+ rand_state=4466,
+ batch_size=batch_size,
+ local_cache=local_cache,
+ show_elbow_plot=show_elbow_plot,
+ update_state=update_state,
+ invalidate_cache=invalidate_cache,
+ )
+
+ def run_lsi(
+ self,
+ normalized: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ dims: int = 11,
+ skip_first: bool = True,
+ rand_state: int = 4466,
+ solver: Literal["streaming", "materialized"] = "streaming",
+ n_iter: int = 5,
+ n_oversamples: int = 10,
+ batch_size: int | None = None,
+ local_cache: bool | str = "auto",
+ update_state: bool = True,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Fit or reuse latent semantic indexing for normalized data.
+
+ Args:
+ normalized: Normalized artifact to reduce. Uses selected assay
+ state when omitted.
+ from_assay: Assay used to resolve selected normalized data.
+ dims: Requested number of retained LSI dimensions.
+ skip_first: Whether to omit the first singular component.
+ rand_state: Seed used by the randomized decomposition.
+ solver: Memory-bounded streaming solver or materialized compatibility
+ solver.
+ n_iter: Power iterations used by randomized LSI.
+ n_oversamples: Extra random vectors used to stabilize the fitted
+ singular subspace.
+ batch_size: Number of selected cells processed per block.
+ local_cache: Local staging policy for normalized data on remote
+ stores.
+ update_state: Select the result as the current reduction.
+ invalidate_cache: Force a new reduction artifact.
+
+ Returns:
+ Reference to the LSI reduction artifact.
+ """
+ if solver not in {"streaming", "materialized"}:
+ raise ValueError("solver must be 'streaming' or 'materialized'")
+ if isinstance(n_iter, bool) or not isinstance(n_iter, (int, np.integer)):
+ raise TypeError("n_iter must be an integer")
+ if isinstance(n_oversamples, bool) or not isinstance(
+ n_oversamples,
+ (int, np.integer),
+ ):
+ raise TypeError("n_oversamples must be an integer")
+ if n_iter < 0:
+ raise ValueError("n_iter must be nonnegative")
+ if n_oversamples < 0:
+ raise ValueError("n_oversamples must be nonnegative")
+ return self._run_reduction_artifact(
+ method="lsi",
+ normalized=normalized,
+ from_assay=from_assay,
+ dims=dims,
+ pca_cell_key=None,
+ feat_scaling=False,
+ lsi_skip_first=skip_first,
+ custom_loadings=None,
+ rand_state=rand_state,
+ batch_size=batch_size,
+ local_cache=local_cache,
+ show_elbow_plot=False,
+ update_state=update_state,
+ invalidate_cache=invalidate_cache,
+ lsi_solver=solver,
+ lsi_n_iter=int(n_iter),
+ lsi_n_oversamples=int(n_oversamples),
+ )
+
+ def run_custom_reduction(
+ self,
+ loadings: np.ndarray,
+ normalized: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ batch_size: int | None = None,
+ local_cache: bool | str = "auto",
+ update_state: bool = True,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Register custom feature loadings as a reusable reduction.
+
+ Args:
+ loadings: Two-dimensional feature-by-dimension loading matrix. Its
+ row count must match the normalized feature selection.
+ normalized: Normalized artifact associated with the loadings. Uses
+ selected assay state when omitted.
+ from_assay: Assay used to resolve selected normalized data.
+ batch_size: Number of selected cells processed per block.
+ local_cache: Local staging policy for normalized data on remote
+ stores.
+ update_state: Select the result as the current reduction.
+ invalidate_cache: Force a new reduction artifact.
+
+ Returns:
+ Reference to the custom reduction artifact.
+ """
+ loading_values = np.asarray(loadings)
+ if loading_values.ndim != 2 or loading_values.shape[1] < 1:
+ raise ValueError(
+ "Custom loadings must be a two-dimensional matrix with columns"
+ )
+ return self._run_reduction_artifact(
+ method="custom",
+ normalized=normalized,
+ from_assay=from_assay,
+ dims=int(loading_values.shape[1]),
+ pca_cell_key=None,
+ feat_scaling=False,
+ lsi_skip_first=False,
+ custom_loadings=loading_values,
+ rand_state=4466,
+ batch_size=batch_size,
+ local_cache=local_cache,
+ show_elbow_plot=False,
+ update_state=update_state,
+ invalidate_cache=invalidate_cache,
+ )
+
+ def run_harmony(
+ self,
+ batch_columns: list[str],
+ reduction: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ harmony_params: dict[str, Any] | None = None,
+ batch_size: int | None = None,
+ update_state: bool = True,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Fit or reuse Harmony correction for a reduction artifact."""
+ reduction_ref = reduction or self._selected_artifact(
+ from_assay,
+ "reduction",
+ "reduction",
+ )
+ self._require_complete_artifact(
+ reduction_ref,
+ "reduction",
+ )
+ if reduction_ref.assay is None:
+ raise ValueError("Reduction artifact has no assay")
+ normalized_ref = self._artifact_input_ref(
+ reduction_ref,
+ "normalized",
+ "normalized",
+ )
+ normalized_status = self._require_complete_artifact(
+ normalized_ref,
+ "normalized",
+ )
+ execution = normalized_status.execution_options or {}
+ cell_key = execution.get("cell_key")
+ feat_key = execution.get("feat_key")
+ if not isinstance(cell_key, str) or not isinstance(feat_key, str):
+ raise ValueError("Normalized artifact has no cell_key or feat_key")
+ state = read_assay_state(self.zw, reduction_ref.assay)
+ if state is not None and state.normalized == normalized_ref:
+ cell_key = state.cell_key
+ feat_key = state.feat_key
+ validate_normalized_artifact_selection(
+ self.zw,
+ normalized_ref,
+ cell_key,
+ feat_key,
+ )
+ if not isinstance(batch_columns, list) or not batch_columns:
+ raise ValueError("batch_columns must be a non-empty list")
+ if len(set(batch_columns)) != len(batch_columns):
+ raise ValueError("batch_columns must be unique")
+ requested_batch_size = (
+ None if batch_size is None else _positive_integer(batch_size, "batch_size")
+ )
+ batches = pd.DataFrame(
+ {
+ column: self.cells.fetch(column, key=cell_key).astype(object)
+ for column in batch_columns
+ }
+ )
+ source, n_cells, dims = self._coordinate_source(
+ reduction_ref,
+ batch_size=requested_batch_size,
+ )
+ source_data = getattr(source, "data", None)
+ source_batch_size = (
+ int(source_data.chunksize[0]) if source_data is not None else n_cells
+ )
+ effective_batch_size = min(
+ source_batch_size if requested_batch_size is None else requested_batch_size,
+ n_cells,
+ )
+ cell_selection = self._artifact_input_ref(
+ normalized_ref,
+ "cell_selection",
+ "cell_selection",
+ )
+ batch_values = resolve_metadata_snapshot(
+ self.zw,
+ values=batches.to_numpy(),
+ row_ids=np.asarray(self.cells.fetch("ids", key=cell_key)),
+ operation="snapshot_metadata",
+ parameters={"columns": batch_columns},
+ inputs={"cell_selection": cell_selection},
+ source_columns=batch_columns,
+ invalidate_cache=invalidate_cache,
+ )
+ arguments = HarmonyArguments(
+ reduction=reduction_ref,
+ batch_values=batch_values,
+ batch_columns=tuple(batch_columns),
+ harmony_parameters=harmony_params or {},
+ algorithm_version="centroid_snapshot_v2",
+ batch_size=effective_batch_size,
+ invalidate_cache=invalidate_cache,
+ )
+ planned = self._plan_assay_artifact(
+ reduction_ref.assay,
+ arguments,
+ required_arrays=(
+ ArrayRequirement(
+ "data",
+ shape=(n_cells, dims),
+ dtype=np.float32,
+ ),
+ ArrayRequirement("cluster_mass", dtype=np.float64),
+ ArrayRequirement("raw_centroids", dtype=np.float64),
+ ArrayRequirement("corrected_centroids", dtype=np.float64),
+ ArrayRequirement("centroids", dtype=np.float64),
+ ArrayRequirement("sigma", dtype=np.float64),
+ ArrayRequirement("ridge", dtype=np.float64),
+ ),
+ invalidate_cache=invalidate_cache,
+ )
+ if not planned.reused:
+ correction = BatchCorrectionStage(
+ stream=source,
+ n_cells=n_cells,
+ dims=dims,
+ batch_size=effective_batch_size,
+ batches=batches,
+ parameters=harmony_params or {},
+ corrected_data=None,
+ nthreads=self.nthreads,
+ )
+ corrected = correction.ensure_corrected()
+ result = correction.result
+ if result is None:
+ raise RuntimeError("Harmony did not return fit metadata")
+ from ...mapping.symphony import weighted_centroids
+
+ cluster_mass, raw_centroids = weighted_centroids(
+ result.original.T,
+ result.assignments,
+ )
+ _, corrected_centroids = weighted_centroids(
+ result.corrected.T,
+ result.assignments,
+ )
+ group = start_artifact(self.zw, planned)
+ output = create_numeric_array(
+ group,
+ "data",
+ row_sharded_array_spec(
+ corrected.shape,
+ np.float32,
+ profile=resolve_storage_profile(group.store),
+ band_rows=min(n_cells, 1_000_000),
+ zarr_format=_group_zarr_format(group),
+ fill_value=0.0,
+ ),
+ )
+ for start, stop in iter_shard_row_slices(
+ n_cells,
+ array_shard_rows(output),
+ ):
+ output[start:stop, :] = np.asarray(
+ result.corrected[:, start:stop].T,
+ dtype=np.float32,
+ )
+ for name, values in (
+ ("cluster_mass", cluster_mass),
+ ("raw_centroids", raw_centroids),
+ ("corrected_centroids", corrected_centroids),
+ ("centroids", result.centroids),
+ ("sigma", result.sigma),
+ ("ridge", result.ridge),
+ ):
+ result_array = create_zarr_dataset(
+ group,
+ name,
+ tuple(max(int(size), 1) for size in values.shape),
+ "f8",
+ values.shape,
+ )
+ result_array[...] = values
+ group.attrs["batch_levels"] = [
+ list(levels) for levels in result.batch_levels
+ ]
+ finish_artifact(group, planned)
+ self._publish_current_artifact(
+ planned.ref,
+ update_state=update_state,
+ )
+ action = "Reused" if planned.reused else "Stored"
+ logger.info(
+ f"{action} Harmony coordinates for {n_cells} cells with {dims} dimensions"
+ )
+ return planned.ref
+
+ def _build_embedding_initialization(
+ self,
+ reduction: ArtifactRef,
+ *,
+ n_centroids: int,
+ rand_state: int,
+ batch_size: int | None,
+ invalidate_cache: bool,
+ kmeans_sampling: float = 0.1,
+ kmeans_batch_size: int = 10_000,
+ algorithm_version: str = "minibatch_kmeans_v2",
+ ) -> ArtifactRef:
+ if reduction.assay is None:
+ raise ValueError("Reduction artifact has no assay")
+ resolved_batch_size = (
+ None if batch_size is None else _positive_integer(batch_size, "batch_size")
+ )
+ requested_clusters = _positive_integer(n_centroids, "n_centroids")
+ resolved_rand_state = _positive_integer(rand_state, "rand_state")
+ resolved_kmeans_sampling = _sampling_fraction(
+ kmeans_sampling,
+ "kmeans_sampling",
+ )
+ requested_kmeans_batch_size = _positive_integer(
+ kmeans_batch_size,
+ "kmeans_batch_size",
+ )
+ stream, n_cells, coordinate_dims = self._coordinate_source(
+ reduction,
+ batch_size=resolved_batch_size,
+ )
+ source_data = getattr(stream, "data", None)
+ source_batch_size = (
+ int(source_data.chunksize[0]) if source_data is not None else n_cells
+ )
+ requested_batch_size = (
+ source_batch_size if resolved_batch_size is None else resolved_batch_size
+ )
+ effective_batch_size = min(int(requested_batch_size), n_cells)
+ if requested_clusters < 2 or n_cells < 2:
+ raise ValueError(
+ "Embedding initialization requires at least two cells and centroids"
+ )
+ effective_clusters = min(
+ requested_clusters,
+ n_cells,
+ )
+ effective_kmeans_batch_size = min(
+ n_cells,
+ max(requested_kmeans_batch_size, effective_clusters),
+ )
+ arguments = EmbeddingInitializationArguments(
+ reduction=reduction,
+ n_centroids=effective_clusters,
+ rand_state=resolved_rand_state,
+ batch_size=effective_batch_size,
+ kmeans_sampling=resolved_kmeans_sampling,
+ kmeans_batch_size=effective_kmeans_batch_size,
+ algorithm_version=algorithm_version,
+ invalidate_cache=invalidate_cache,
+ )
+ planned = self._plan_assay_artifact(
+ reduction.assay,
+ arguments,
+ required_arrays=(
+ ArrayRequirement(
+ "cluster_centers",
+ shape=(effective_clusters, coordinate_dims),
+ dtype_kind="f",
+ ),
+ ArrayRequirement(
+ "cluster_labels",
+ shape=(n_cells,),
+ dtype=np.uint32,
+ ),
+ ),
+ invalidate_cache=invalidate_cache,
+ )
+ if not planned.reused:
+ initialization = KMeansInitializationStage.fit(
+ stream=stream,
+ n_rows=n_cells,
+ batch_size=effective_batch_size,
+ n_clusters=effective_clusters,
+ rand_state=resolved_rand_state,
+ nthreads=self.nthreads,
+ enabled=True,
+ kmeans_sampling=resolved_kmeans_sampling,
+ kmeans_batch_size=effective_kmeans_batch_size,
+ )
+ if initialization.model is None:
+ raise RuntimeError("K-means initialization did not fit")
+ group = start_artifact(self.zw, planned)
+ centers = create_zarr_dataset(
+ group,
+ "cluster_centers",
+ (1000, 1000),
+ "f8",
+ initialization.model.cluster_centers_.shape,
+ )
+ centers[:, :] = initialization.model.cluster_centers_
+ labels = create_zarr_dataset(
+ group,
+ "cluster_labels",
+ (100000,),
+ np.uint32,
+ initialization.labels.shape,
+ )
+ labels[:] = initialization.labels
+ finish_artifact(group, planned)
+ action = "Reused" if planned.reused else "Stored"
+ logger.info(
+ f"{action} embedding initialization with {effective_clusters} centroids"
+ )
+ return planned.ref
+
+ def build_embedding_initialization(
+ self,
+ reduction: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ n_centroids: int = 1000,
+ rand_state: int = 4466,
+ batch_size: int | None = None,
+ kmeans_sampling: float = 0.1,
+ kmeans_batch_size: int = 10_000,
+ update_state: bool = True,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Build or reuse K-means embedding initialization for a reduction.
+
+ Downstream UMAP reads this artifact from ``AssayState`` unless you pass
+ ``ini_embed`` explicitly.
+
+ Args:
+ reduction: Reduction artifact to cluster. Uses the selected assay
+ reduction when omitted.
+ from_assay: Assay used to resolve the selected reduction.
+ n_centroids: Requested number of K-means centroids.
+ rand_state: K-means random seed.
+ batch_size: Number of cells processed per block.
+ kmeans_sampling: Fraction of cells considered during centroid seeding.
+ kmeans_batch_size: Number of cells per internal K-means update.
+ update_state: Select the result as the current embedding
+ initialization.
+ invalidate_cache: Force a new initialization artifact.
+
+ Returns:
+ Reference to the embedding-initialization artifact.
+ """
+ if reduction is None:
+ assay = from_assay or self._defaultAssay
+ if assay is None:
+ raise ValueError("No assay was provided and no default is configured")
+ state = read_assay_state(self.zw, assay)
+ if state is None or state.reduction is None:
+ raise KeyError(f"AssayState for {assay!r} has no selected reduction")
+ reduction = state.reduction
+ initialization = self._build_embedding_initialization(
+ reduction,
+ n_centroids=n_centroids,
+ rand_state=rand_state,
+ batch_size=batch_size,
+ invalidate_cache=invalidate_cache,
+ kmeans_sampling=kmeans_sampling,
+ kmeans_batch_size=kmeans_batch_size,
+ )
+ if update_state:
+ if reduction.assay is None:
+ raise ValueError("Reduction artifact has no assay")
+ state = read_assay_state(self.zw, reduction.assay)
+ if state is not None and state.reduction == reduction:
+ write_assay_state(
+ self.zw,
+ replace(
+ state,
+ embedding_initialization=initialization,
+ ),
+ )
+ else:
+ self._publish_current_artifact(
+ reduction,
+ update_state=True,
+ embedding_initialization=initialization,
+ )
+ return initialization
+
+ def build_ann_index(
+ self,
+ coordinates: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ ann_metric: str = "l2",
+ ann_efc: int = 50,
+ ann_ef: int = 50,
+ ann_m: int = 48,
+ ann_parallel: bool = False,
+ rand_state: int = 4466,
+ batch_size: int | None = None,
+ update_state: bool = True,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Build or reuse an approximate nearest-neighbor index."""
+ if coordinates is None:
+ assay = from_assay or self._defaultAssay
+ if assay is None:
+ raise ValueError("No assay was provided and no default is configured")
+ state = read_assay_state(self.zw, assay)
+ if state is None or state.reduction is None:
+ raise KeyError("AssayState has no selected reduction")
+ coordinates = (
+ state.batch_correction
+ if state.batch_correction is not None
+ else state.reduction
+ )
+ if coordinates.assay is None:
+ raise ValueError("Coordinate artifact has no assay")
+ if coordinates.kind == "imported_coordinates" and update_state:
+ raise ValueError(
+ "Imported coordinates cannot activate AssayState; "
+ "pass update_state=False"
+ )
+ if ann_metric not in {"l2", "cosine"}:
+ raise ValueError("ann_metric must be one of: l2, cosine")
+ resolved_ann_efc = _positive_integer(ann_efc, "ann_efc")
+ resolved_ann_ef = _positive_integer(ann_ef, "ann_ef")
+ resolved_ann_m = _positive_integer(ann_m, "ann_m")
+ resolved_rand_state = _positive_integer(rand_state, "rand_state")
+ if resolved_ann_m < 2:
+ raise ValueError("ann_m must be at least two")
+ if not isinstance(ann_parallel, bool):
+ raise TypeError("ann_parallel must be a boolean")
+ resolved_batch_size = (
+ None if batch_size is None else _positive_integer(batch_size, "batch_size")
+ )
+ coordinate_source, n_cells, dims = self._coordinate_source(
+ coordinates,
+ batch_size=resolved_batch_size,
+ )
+ source_data = getattr(coordinate_source, "data", None)
+ source_batch_size = (
+ int(source_data.chunksize[0]) if source_data is not None else n_cells
+ )
+ requested_batch_size = (
+ source_batch_size if resolved_batch_size is None else resolved_batch_size
+ )
+ effective_batch_size = min(int(requested_batch_size), n_cells)
+ parallel_threads = self.nthreads if ann_parallel else None
+ arguments = AnnIndexArguments(
+ coordinates=coordinates,
+ ann_metric=ann_metric,
+ ann_efc=resolved_ann_efc,
+ ann_ef=resolved_ann_ef,
+ ann_m=resolved_ann_m,
+ rand_state=resolved_rand_state,
+ ann_parallel=ann_parallel,
+ parallel_threads=parallel_threads,
+ batch_size=effective_batch_size,
+ invalidate_cache=invalidate_cache,
+ )
+
+ def valid_ann_artifact(
+ _ref: ArtifactRef,
+ group: zarr.Group,
+ ) -> bool:
+ try:
+ load_ann_index(
+ group,
+ ann_metric,
+ dims,
+ expected_count=n_cells,
+ )
+ except (FileNotFoundError, RuntimeError, ValueError):
+ return False
+ return True
+
+ planned = self._plan_assay_artifact(
+ coordinates.assay,
+ arguments,
+ required_arrays=(ArrayRequirement("ann_idx_bytes", dtype=np.uint8),),
+ invalidate_cache=invalidate_cache,
+ reuse_validator=valid_ann_artifact,
+ )
+ if not planned.reused:
+ ann_idx = AnnIndexStage.fit(
+ coordinates=coordinate_source,
+ metric=ann_metric,
+ dims=dims,
+ n_cells=n_cells,
+ ef_construction=resolved_ann_efc,
+ ef=resolved_ann_ef,
+ m=resolved_ann_m,
+ rand_state=resolved_rand_state,
+ ann_threads=(self.nthreads if ann_parallel else 1),
+ )
+ group = start_artifact(self.zw, planned)
+ self._persist_ann_index(
+ artifact_path(planned.ref),
+ ann_idx,
+ ann_metric=ann_metric,
+ dimensions=dims,
+ element_count=n_cells,
+ )
+ finish_artifact(group, planned)
+ self._publish_current_artifact(
+ planned.ref,
+ update_state=update_state,
+ )
+ action = "Reused" if planned.reused else "Stored"
+ logger.info(f"{action} ANN index for {n_cells} cells")
+ return planned.ref
+
+ def query_neighbors(
+ self,
+ ann_index: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ coordinates: ArtifactRef | None = None,
+ k: int = 17,
+ batch_size: int | None = None,
+ update_state: bool = True,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Query an ANN artifact and persist compact neighbor matrices."""
+ ann_ref = ann_index or self._selected_artifact(
+ from_assay,
+ "ann_index",
+ "ann_index",
+ )
+ ann_status = self._require_complete_artifact(
+ ann_ref,
+ "ann_index",
+ )
+ if ann_ref.assay is None:
+ raise ValueError("ANN artifact has no assay")
+ raw_coordinates = (ann_status.inputs or {}).get("coordinates")
+ if not isinstance(raw_coordinates, dict):
+ raise ValueError("ANN artifact has no coordinates input")
+ stored_coordinates = ArtifactRef.from_dict(raw_coordinates)
+ if stored_coordinates.kind not in {
+ "reduction",
+ "batch_correction",
+ "imported_coordinates",
+ }:
+ raise ValueError(
+ "ANN coordinates must be reduction, batch_correction, "
+ "or imported_coordinates"
+ )
+ self._require_complete_artifact(
+ stored_coordinates,
+ stored_coordinates.kind,
+ )
+ if coordinates is not None and coordinates != stored_coordinates:
+ raise ValueError("coordinates do not match the ANN artifact input")
+ if stored_coordinates.kind == "imported_coordinates" and update_state:
+ raise ValueError(
+ "Neighbors from imported coordinates cannot activate AssayState; "
+ "pass update_state=False"
+ )
+ requested_k = _positive_integer(k, "k")
+ resolved_batch_size = (
+ None if batch_size is None else _positive_integer(batch_size, "batch_size")
+ )
+ coordinate_source, n_cells, dims = self._coordinate_source(
+ stored_coordinates,
+ batch_size=resolved_batch_size,
+ )
+ if n_cells < 2:
+ raise ValueError("Neighbor queries require at least two cells")
+ effective_k = min(requested_k, n_cells - 1)
+ if n_cells - 1 > np.iinfo(np.uint32).max:
+ raise ValueError("Neighbor indices require fewer than 2**32 cells")
+ source_data = getattr(coordinate_source, "data", None)
+ source_batch_size = (
+ int(source_data.chunksize[0]) if source_data is not None else n_cells
+ )
+ requested_batch_size = (
+ source_batch_size if resolved_batch_size is None else resolved_batch_size
+ )
+ effective_batch_size = min(int(requested_batch_size), n_cells)
+ ann_parameters = ann_status.parameters or {}
+ ann_metric = ann_parameters.get("ann_metric")
+ if ann_metric not in {"l2", "cosine"}:
+ raise ValueError("ANN artifact has no supported distance metric")
+ arguments = NeighborQueryArguments(
+ ann_index=ann_ref,
+ coordinates=stored_coordinates,
+ k=effective_k,
+ distance_metric=str(ann_metric),
+ batch_size=effective_batch_size,
+ invalidate_cache=invalidate_cache,
+ )
+ planned = self._plan_assay_artifact(
+ ann_ref.assay,
+ arguments,
+ required_arrays=(
+ ArrayRequirement(
+ "indices",
+ shape=(n_cells, effective_k),
+ dtype=np.uint32,
+ ),
+ ArrayRequirement(
+ "distances",
+ shape=(n_cells, effective_k),
+ dtype=np.float32,
+ ),
+ ),
+ invalidate_cache=invalidate_cache,
+ )
+ if not planned.reused:
+ ann_idx = self._resolve_ann_index(
+ ann_status.path,
+ str(ann_metric),
+ dims,
+ expected_count=n_cells,
+ )
+ if ann_idx is None:
+ raise RuntimeError("ANN artifact has no readable index")
+ ann_idx = AnnIndexStage.configure(
+ ann_idx,
+ ef=int(ann_parameters.get("ann_ef", 50)),
+ threads=(int(ann_parameters.get("parallel_threads") or 1)),
+ )
+ query = NeighborQueryStage(
+ ann_idx,
+ effective_k,
+ str(ann_metric),
+ )
+ indices = np.empty((n_cells, effective_k), dtype=np.uint32)
+ distances = np.empty((n_cells, effective_k), dtype=np.float32)
+ start = 0
+ missed_self_hits = 0
+ for block in coordinate_source.iter_coordinate_blocks(
+ "Identifying neighbors"
+ ):
+ stop = start + len(block)
+ result = query.query(
+ block,
+ self_indices=np.arange(start, stop),
+ )
+ block_indices, block_distances, missed = cast(
+ tuple[np.ndarray, np.ndarray, int],
+ result,
+ )
+ if np.any(block_indices < 0) or np.any(block_indices >= n_cells):
+ raise ValueError("ANN query returned an invalid cell index")
+ indices[start:stop, :] = block_indices
+ distances[start:stop, :] = block_distances
+ missed_self_hits += missed
+ start = stop
+ if start != n_cells:
+ raise ValueError(
+ f"Coordinate source contains {start} rows, expected {n_cells}"
+ )
+ group = start_artifact(self.zw, planned)
+ array_profile = resolve_storage_profile(group.store)
+ zarr_format = _group_zarr_format(group)
+ indices_array = create_numeric_array(
+ group,
+ "indices",
+ row_sharded_array_spec(
+ indices.shape,
+ np.uint32,
+ profile=array_profile,
+ band_rows=min(n_cells, 1_000_000),
+ zarr_format=zarr_format,
+ ),
+ )
+ distances_array = create_numeric_array(
+ group,
+ "distances",
+ row_sharded_array_spec(
+ distances.shape,
+ np.float32,
+ profile=array_profile,
+ band_rows=min(n_cells, 1_000_000),
+ zarr_format=zarr_format,
+ fill_value=0.0,
+ ),
+ )
+ indices_array[:, :] = indices
+ distances_array[:, :] = distances
+ group.attrs["n_cells"] = n_cells
+ group.attrs["n_neighbors"] = effective_k
+ group.attrs["self_hit_rate"] = (
+ 100.0 * (n_cells - missed_self_hits) / n_cells
+ )
+ finish_artifact(group, planned)
+ self._publish_current_artifact(
+ planned.ref,
+ update_state=update_state,
+ )
+ action = "Reused" if planned.reused else "Stored"
+ logger.info(f"{action} {effective_k} neighbors for each of {n_cells} cells")
+ return planned.ref
+
+ def build_connectivity_map(
+ self,
+ neighbors: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ local_connectivity: float = 1.0,
+ bandwidth: float = 1.5,
+ update_state: bool = True,
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Convert persisted neighbors into a weighted connectivity graph.
+
+ Args:
+ neighbors: Neighbors artifact. Uses current assay state when
+ omitted.
+ from_assay: Assay used to resolve current neighbors.
+ local_connectivity: UMAP-style local-connectivity adjustment.
+ bandwidth: Distance-kernel bandwidth multiplier.
+ update_state: Select the result as the current connectivity map.
+ invalidate_cache: Force a new connectivity artifact.
+
+ Returns:
+ Reference to the connectivity-map artifact.
+ """
+ neighbors_ref = neighbors or self._selected_artifact(
+ from_assay,
+ "neighbors",
+ "neighbors",
+ )
+ status = self._require_complete_artifact(
+ neighbors_ref,
+ "neighbors",
+ )
+ if neighbors_ref.assay is None:
+ raise ValueError("Neighbors artifact has no assay")
+ group = group_at(self.zw, status.path)
+ indices = as_zarr_array(group["indices"], name="indices")
+ n_cells, n_neighbors = map(int, indices.shape)
+ validate_distance_provenance(self.zw, status.path)
+ arguments = ConnectivityMapArguments(
+ neighbors=neighbors_ref,
+ local_connectivity=local_connectivity,
+ bandwidth=bandwidth,
+ invalidate_cache=invalidate_cache,
+ )
+ planned = self._plan_assay_artifact(
+ neighbors_ref.assay,
+ arguments,
+ required_arrays=(
+ ArrayRequirement(
+ "edges",
+ shape=(None, 2),
+ dtype=np.uint32,
+ ),
+ ArrayRequirement(
+ "weights",
+ shape=(None,),
+ dtype=np.float32,
+ ),
+ ),
+ invalidate_cache=invalidate_cache,
+ )
+ if not planned.reused:
+ from ...neighbors.graph import build_connectivity_arrays
+
+ distance_values = np.asarray(
+ as_zarr_array(
+ group["distances"],
+ name="distances",
+ )[:]
+ )
+ edge_values, weight_values = build_connectivity_arrays(
+ np.asarray(indices[:]),
+ distance_values,
+ local_connectivity=local_connectivity,
+ bandwidth=bandwidth,
+ )
+ output = start_artifact(self.zw, planned)
+ profile = resolve_storage_profile(output.store)
+ zarr_format = _group_zarr_format(output)
+ edge_band_rows = min(n_cells, 1_000_000) * n_neighbors
+ edges = create_numeric_array(
+ output,
+ "edges",
+ row_sharded_array_spec(
+ edge_values.shape,
+ np.uint32,
+ profile=profile,
+ band_rows=edge_band_rows,
+ zarr_format=zarr_format,
+ ),
+ )
+ weights = create_numeric_array(
+ output,
+ "weights",
+ row_sharded_array_spec(
+ weight_values.shape,
+ np.float32,
+ profile=profile,
+ band_rows=edge_band_rows,
+ zarr_format=zarr_format,
+ fill_value=0.0,
+ ),
+ )
+ edges[:, :] = edge_values
+ weights[:] = weight_values
+ output.attrs["n_cells"] = n_cells
+ output.attrs["n_neighbors"] = n_neighbors
+ finish_artifact(output, planned)
+ self._publish_current_artifact(
+ planned.ref,
+ update_state=update_state,
+ )
+ action = "Reused" if planned.reused else "Stored"
+ logger.info(f"{action} connectivity map for {n_cells} cells")
+ return planned.ref
+
+ def load_graph(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ symmetric: bool | None = None,
+ upper_only: bool | None = None,
+ use_k: int | None = None,
+ graph_loc: str | None = None,
+ ) -> csr_matrix:
+ """Load the cell neighbourhood as a scipy sparse matrix.
+
+ Args:
+ from_assay: Name of the assay. If None then the default assay is used.
+ cell_key: Cell key used to create the graph. If None then the latest feature key used for creating a
+ KNN graph is used.
+ feat_key: Feature key used to create the graph. If None then the latest feature key used for creating a
+ KNN graph is used.
+ symmetric: If True, makes the graph symmetric by adding it to its transpose.
+ upper_only: If True, then only the values from upper triangular of the matrix are returned. This is only
+ used when symmetric is True.
+ use_k: Number of top k-nearest neighbours to keep in the graph. This value must be greater than 0 and less
+ the parameter k used. By default, all neighbours are used. (Default value: None)
+ graph_loc: Zarr hierarchy where the graph is stored. If no value is provided then graph location is
+ obtained from `get_latest_graph_loc` method.
+
+ Returns:
+ A scipy sparse matrix representing cell neighbourhood graph.
+ """
+
+ def symmetrize(g: csr_matrix) -> csr_matrix:
+ t = g + g.T
+ t = t - g.multiply(g.T)
+ return t
+
+ from scipy.sparse import triu
+
+ if graph_loc is None:
+ stored = self._lookup_stored_graph(from_assay, cell_key, feat_key)
+ if not isinstance(stored, StoredAssayGraph):
+ raise TypeError("Expected an assay graph for load_graph lookup")
+ graph_loc = stored.paths.cell_graph_group_path
+ try:
+ explicit_ref = parse_artifact_path(graph_loc)
+ except ValueError:
+ if not is_integrated_graph_path(
+ graph_loc,
+ self._integratedGraphsLoc,
+ ):
+ stored = self._lookup_stored_graph(graph_loc=graph_loc)
+ if not isinstance(stored, StoredAssayGraph):
+ raise ValueError("Expected an assay graph")
+ if from_assay is not None and from_assay != stored.from_assay:
+ raise ValueError("from_assay does not match the graph location")
+ if cell_key is not None and cell_key != stored.cell_key:
+ raise ValueError("cell_key does not match the graph location")
+ if feat_key is not None and feat_key != stored.feat_key:
+ raise ValueError("feat_key does not match the graph location")
+ else:
+ status = inspect_artifact(self.zw, explicit_ref)
+ if not status.exists or not status.complete:
+ raise RuntimeError(f"Graph artifact is incomplete: {graph_loc}")
+ if (
+ explicit_ref.scope == "assay"
+ and explicit_ref.kind == "connectivity_map"
+ ):
+ stored = stored_assay_graph_from_ref(self.zw, explicit_ref)
+ if from_assay is not None and from_assay != stored.from_assay:
+ raise ValueError("from_assay does not match the graph artifact")
+ if cell_key is not None and cell_key != stored.cell_key:
+ raise ValueError("cell_key does not match the graph artifact")
+ if feat_key is not None and feat_key != stored.feat_key:
+ raise ValueError("feat_key does not match the graph artifact")
+ elif (
+ explicit_ref.scope == "datastore"
+ and explicit_ref.kind == "integrated_graph"
+ ):
+ selection = self._graph_cell_selection(explicit_ref)
+ source_column = (
+ inspect_artifact(self.zw, selection).execution_options or {}
+ ).get("source_column")
+ selected_cell_key = cell_key or source_column
+ if not isinstance(selected_cell_key, str):
+ raise ValueError(
+ "Integrated graph cell selection key is unavailable"
+ )
+ validate_cell_selection_artifact(
+ self.zw,
+ selection,
+ selected_cell_key,
+ )
+ else:
+ raise ValueError(f"Not a graph artifact: {graph_loc}")
+ if graph_loc not in self.zw:
+ raise ValueError(
+ f"{graph_loc} not found in zarr location. "
+ f"Build graph artifacts for assay {from_assay}"
+ )
+ n_cells, graph = self._store_to_sparse(graph_loc, "csr", use_k)
+ if symmetric is True:
+ graph = symmetrize(graph)
+ if upper_only is True:
+ graph = triu(graph)
+ return graph
+
+ def integrate_assays(
+ self,
+ assays: list[str],
+ label: str,
+ method: str = "snn",
+ chunk_size: int = 10000,
+ invalidate_cache: bool = False,
+ l2_normalize: bool = True,
+ ) -> ArtifactRef:
+ """Integrate the latest neighbourhood graphs for selected assays.
+
+ SNN combines shared edge support across two or more assays. WNN accepts
+ exactly two assays and uses Hao-inspired per-cell modality weights.
+ Scarf WNN scores only the union of the existing self-free KNN rows and
+ uses a simple k-th-neighbour bandwidth, so it is not bit-identical to
+ Seurat's default wider search and SNN-far bandwidth.
+
+ Args:
+ assays: Name of the input assays. The latest constructed graph from each assay is used.
+ label: Label for integrated graph
+ method: Choose a method for modality integration. Available options: 'snn': Shared nearest neighbour
+ approach and 'wnn': Hao-inspired weighted nearest neighbor integration.
+ chunk_size: number of cells to be loaded at a time while reading and writing the graph
+ invalidate_cache: Force a new integrated-graph artifact.
+ l2_normalize: L2-normalize modality coordinates during WNN scoring.
+ This algorithmic setting is stored in artifact provenance.
+
+ Returns:
+ Reference to the integrated-graph artifact. Pass it to `run_umap`,
+ `run_tsne`, or the clustering methods as their ``graph`` argument,
+ or keep using ``integrated_graph=label``.
+
+ WNN stores two modality-weight columns named
+ ``{label}_{assay}_weight`` in cell metadata.
+ """
+ from ...neighbors.graph import merge_graphs
+ from ...neighbors.integration import wnn_integration
+
+ if method not in {"snn", "wnn"}:
+ raise ValueError(
+ f"Method {method} not supported, choose one of these: 'snn', 'wnn'"
+ )
+ if method == "wnn" and len(assays) != 2:
+ raise ValueError(
+ "WNN integration in Scarf can currently be performed using only two assays"
+ )
+ if method == "wnn" and not isinstance(l2_normalize, bool | np.bool_):
+ raise TypeError("l2_normalize must be a boolean")
+
+ def materialize_coordinate_blocks(
+ blocks: Iterator[np.ndarray],
+ n_cells: int,
+ *,
+ expected_dims: int | None = None,
+ ) -> np.ndarray:
+ coordinates: np.ndarray | None = None
+ start = 0
+ for values in blocks:
+ block = np.asarray(values)
+ if block.ndim != 2:
+ raise ValueError("WNN coordinate blocks must be matrices")
+ if expected_dims is not None and block.shape[1] != expected_dims:
+ raise ValueError("WNN coordinate dimensions changed between blocks")
+ if coordinates is None:
+ coordinates = np.empty(
+ (n_cells, block.shape[1]),
+ dtype=block.dtype,
+ )
+ stop = start + len(block)
+ if stop > n_cells:
+ raise ValueError("WNN coordinate stream exceeded its cell count")
+ coordinates[start:stop] = block
+ start = stop
+ if coordinates is None or start != n_cells:
+ raise ValueError("WNN coordinate stream did not cover every cell")
+ return coordinates
+
+ def neighbor_coordinates_ref(neighbors: ArtifactRef) -> ArtifactRef:
+ raw_coordinates = (inspect_artifact(self.zw, neighbors).inputs or {}).get(
+ "coordinates"
+ )
+ if not isinstance(raw_coordinates, dict):
+ raise ValueError("Neighbors artifact has no coordinates input")
+ coordinates = ArtifactRef.from_dict(raw_coordinates)
+ if coordinates.kind not in {"reduction", "batch_correction"}:
+ raise ValueError(
+ "Neighbor coordinates must be reduction or batch_correction"
+ )
+ self._require_complete_artifact(coordinates, coordinates.kind)
+ return coordinates
+
+ source_inputs: dict[str, Any] = {}
+ legacy_wnn_coordinates: dict[str, np.ndarray] = {}
+ legacy_wnn_neighbor_paths: dict[str, str] = {}
+ shared_selection: ArtifactRef | None = None
+ shared_cell_key: str | None = None
+ for assay_name in assays:
+ if assay_name not in self.assay_names:
+ raise ValueError(f"ERROR: Assay {assay_name} was not found.")
+ state = read_assay_state(self.zw, assay_name)
+ artifact_source = state is not None and (
+ (method == "wnn" and state.neighbors is not None)
+ or (method == "snn" and state.connectivity_map is not None)
+ )
+ if artifact_source:
+ assert state is not None
+ if state.normalized is None:
+ raise ValueError(
+ f"Assay {assay_name!r} has no normalized graph input"
+ )
+ if method == "wnn":
+ assert state.neighbors is not None
+ self._require_complete_artifact(state.neighbors, "neighbors")
+ validate_neighbors_artifact_selection(
+ self.zw,
+ state.neighbors,
+ state.cell_key,
+ state.feat_key,
+ )
+ else:
+ assert state.connectivity_map is not None
+ validate_artifact_graph_selection(
+ self.zw,
+ state.connectivity_map,
+ state.cell_key,
+ state.feat_key,
+ )
+ selection = self._artifact_input_ref(
+ state.normalized,
+ "cell_selection",
+ "cell_selection",
+ )
+ if method == "wnn":
+ assert state.neighbors is not None
+ coordinates_ref = neighbor_coordinates_ref(state.neighbors)
+ source_inputs[assay_name] = {
+ "neighbors": state.neighbors,
+ "coordinates": coordinates_ref,
+ }
+ else:
+ assert state.connectivity_map is not None
+ source_inputs[assay_name] = state.connectivity_map
+ cell_key = state.cell_key
+ else:
+ legacy_graph_path = self.get_latest_graph_loc(
+ assay_name,
+ self._get_latest_cell_key(assay_name),
+ self._get_latest_feat_key(assay_name),
+ )
+ legacy_graph = as_zarr_group(
+ self.zw[legacy_graph_path],
+ name=legacy_graph_path,
+ )
+ cell_key = self._get_latest_cell_key(assay_name)
+ selection = self._ensure_cell_selection(cell_key)
+ if method == "wnn":
+ neighbors_path = nearest_neighbors_group_path_from_cell_graph(
+ legacy_graph_path
+ )
+ neighbors_group = as_zarr_group(
+ self.zw[neighbors_path],
+ name=neighbors_path,
+ )
+ legacy_wnn_neighbor_paths[assay_name] = neighbors_path
+ legacy_input: dict[str, Any] = {
+ "legacy_graph_fingerprint": fingerprint_stored_arrays(
+ neighbors_group,
+ ("indices",),
+ ),
+ }
+ feat_key = self._get_latest_feat_key(assay_name)
+ ann = self._load_ann_stream(
+ assay_name,
+ cell_key,
+ feat_key,
+ )
+ if ann.harmonizedData is not None:
+ coordinates = materialize_coordinate_blocks(
+ (
+ np.asarray(block.compute())
+ for block in ann.harmonizedData.blocks
+ ),
+ ann.nCells,
+ )
+ else:
+ coordinates = materialize_coordinate_blocks(
+ (
+ ann.reducer(block)
+ for block in ann.iter_blocks(
+ f"Loading {assay_name} coordinates"
+ )
+ ),
+ ann.nCells,
+ )
+ legacy_wnn_coordinates[assay_name] = coordinates
+ legacy_input["legacy_coordinates_fingerprint"] = fingerprint_array(
+ coordinates
+ )
+ else:
+ legacy_input = {
+ "legacy_graph_fingerprint": fingerprint_stored_arrays(
+ legacy_graph,
+ ("edges", "weights"),
+ ),
+ }
+ source_inputs[assay_name] = legacy_input
+ if shared_selection is None:
+ shared_selection = selection
+ shared_cell_key = cell_key
+ elif not self._selection_artifacts_match(
+ shared_selection,
+ selection,
+ ):
+ raise ValueError("Integrated graphs require one shared cell selection")
+ if shared_selection is None:
+ raise ValueError("No assay cell selection was resolved")
+ if shared_cell_key is None:
+ raise RuntimeError("No cell key was resolved for assay integration")
+ source_inputs["cell_selection"] = shared_selection
+ parameters: dict[str, Any] = {"method": method, "assays": assays}
+ required_arrays: list[ArrayRequirement] = [
+ ArrayRequirement("edges"),
+ ArrayRequirement("weights", dtype_kind="f"),
+ ]
+ if method == "wnn":
+ parameters["l2_normalize"] = bool(l2_normalize)
+ required_arrays.append(
+ ArrayRequirement(
+ "modality_weights",
+ shape=(None, 2),
+ dtype=np.float32,
+ )
+ )
+ integrated_plan = plan_artifact(
+ self.zw,
+ scope="datastore",
+ kind="integrated_graph",
+ operation="integrate_assays",
+ parameters=parameters,
+ inputs=source_inputs,
+ execution_options={"label": label, "chunk_size": chunk_size},
+ invalidate_cache=invalidate_cache,
+ required_arrays=tuple(required_arrays),
+ )
+
+ def select_integrated_artifact() -> None:
+ index_path = self._integratedGraphsLoc
+ index_group = (
+ as_zarr_group(self.zw[index_path], name=index_path)
+ if index_path in self.zw
+ else self.zw.create_group(index_path)
+ )
+ raw_artifacts = index_group.attrs.get("artifacts", {})
+ if "artifacts" in index_group.attrs and not isinstance(
+ raw_artifacts,
+ dict,
+ ):
+ raise RuntimeError("Integrated graph artifact index is invalid")
+ artifacts = dict(raw_artifacts) if isinstance(raw_artifacts, dict) else {}
+ artifacts[label] = integrated_plan.ref.to_dict()
+ index_group.attrs["artifacts"] = artifacts
+
+ weight_columns = (
+ [f"{label}_{assay_name}_weight" for assay_name in assays]
+ if method == "wnn"
+ else []
+ )
+ preserved_weight_displays = {
+ column: column_display(self.zw, column) for column in weight_columns
+ }
+
+ def publish_modality_weights() -> None:
+ if method != "wnn":
+ return
+ group = artifact_group(self.zw, integrated_plan.ref)
+ try:
+ stored_n_cells = _positive_integer(
+ group.attrs.get("n_cells"),
+ "stored n_cells",
+ )
+ except (TypeError, ValueError) as error:
+ raise RuntimeError(
+ "Stored WNN modality weights have invalid cell metadata"
+ ) from error
+ values = np.asarray(
+ as_zarr_array(
+ group["modality_weights"],
+ name="modality_weights",
+ )[:],
+ dtype=np.float32,
+ )
+ if values.shape != (stored_n_cells, 2):
+ raise RuntimeError("Stored WNN modality weights have an invalid shape")
+ for index, column in enumerate(weight_columns):
+ column_values = values[:, index]
+ self.cells.insert(
+ column,
+ column_values,
+ overwrite=True,
+ key=shared_cell_key,
+ )
+ link_cell_data_column(
+ self.zw,
+ column,
+ integrated_plan.ref,
+ value_name="modality_weights",
+ value_index=index,
+ default_display=continuous_display(column_values),
+ preserved_display=preserved_weight_displays[column],
+ )
+
+ if integrated_plan.reused:
+ select_integrated_artifact()
+ publish_modality_weights()
+ return integrated_plan.ref
+
+ def load_wnn_inputs(assay_name: str) -> tuple[np.ndarray, NDArray[Any]]:
+ state = read_assay_state(self.zw, assay_name)
+ if state is not None and state.neighbors is not None:
+ neighbors_group = artifact_group(self.zw, state.neighbors)
+ indices = np.asarray(
+ as_zarr_array(
+ neighbors_group["indices"],
+ name="indices",
+ )[:]
+ )
+ coordinates_ref = neighbor_coordinates_ref(state.neighbors)
+ coordinate_source, _n_cells, _dims = self._coordinate_source(
+ coordinates_ref,
+ batch_size=None,
+ )
+ coordinates = materialize_coordinate_blocks(
+ (
+ np.asarray(block)
+ for block in coordinate_source.iter_coordinate_blocks(
+ f"Loading {assay_name} coordinates",
+ )
+ ),
+ _n_cells,
+ expected_dims=_dims,
+ )
+ if indices.shape[0] != _n_cells:
+ raise ValueError(
+ f"WNN neighbors and coordinates for {assay_name} "
+ "contain different cell counts"
+ )
+ return indices, coordinates
+ neighbors_path = legacy_wnn_neighbor_paths[assay_name]
+ neighbors_group = as_zarr_group(
+ self.zw[neighbors_path],
+ name=neighbors_path,
+ )
+ indices = np.asarray(
+ as_zarr_array(
+ neighbors_group["indices"],
+ name="indices",
+ )[:]
+ )
+ return indices, legacy_wnn_coordinates[assay_name]
+
+ modality_weights: np.ndarray | None = None
+ if method == "snn":
+ graphs: list[csr_matrix] = []
+ for assay in assays:
+ if assay not in self.assay_names:
+ raise ValueError(f"ERROR: Assay {assay} was not found.")
+ graphs.append(
+ self.load_graph(
+ from_assay=assay,
+ cell_key=None,
+ feat_key=None,
+ symmetric=False,
+ upper_only=False,
+ ).tocsr()
+ )
+ merged_graph = merge_graphs(graphs)
+ elif method == "wnn":
+ indices1, ld1 = load_wnn_inputs(assays[0])
+ indices2, ld2 = load_wnn_inputs(assays[1])
+ merged_graph, modality_weights = wnn_integration(
+ assays[0],
+ indices1,
+ ld1,
+ assays[1],
+ indices2,
+ ld2,
+ self.nthreads,
+ l2_normalize=l2_normalize,
+ )
+ n_cells = merged_graph.shape[0]
+ n_neighbors = int(merged_graph.size / n_cells)
+
+ store = start_artifact(self.zw, integrated_plan)
+ store.attrs["n_cells"] = n_cells
+ store.attrs["n_neighbors"] = n_neighbors
+ store.attrs["assays"] = list(assays)
+
+ edge_chunk = chunk_size * n_neighbors
+ zge = create_zarr_dataset(
+ store,
+ "edges",
+ (edge_chunk,),
+ np.uint32,
+ (n_cells * n_neighbors, 2),
+ )
+ zgw = create_zarr_dataset(
+ store,
+ "weights",
+ (edge_chunk,),
+ np.float32,
+ (n_cells * n_neighbors,),
+ )
+
+ zge[:, 0] = merged_graph.row
+ zge[:, 1] = merged_graph.col
+ zgw[:] = merged_graph.data
+ if modality_weights is not None:
+ stored_modality_weights = create_zarr_dataset(
+ store,
+ "modality_weights",
+ (min(chunk_size, n_cells), 2),
+ np.float32,
+ modality_weights.shape,
+ )
+ stored_modality_weights[:, :] = modality_weights
+ finish_artifact(store, integrated_plan)
+ select_integrated_artifact()
+ publish_modality_weights()
+ return integrated_plan.ref
diff --git a/scarf/datastore/_operations/integration_metrics.py b/scarf/datastore/_operations/integration_metrics.py
new file mode 100644
index 00000000..7a08a3fe
--- /dev/null
+++ b/scarf/datastore/_operations/integration_metrics.py
@@ -0,0 +1,684 @@
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any, Literal, cast
+
+import numpy as np
+import zarr
+
+from ...graph.encoded_paths import (
+ is_integrated_graph_path,
+ parse_assay_keys_from_nearest_neighbors_path,
+)
+from ...graph.distances import validate_distance_provenance
+from ...graph.paths import StoredAssayGraph
+from ...graph.state import (
+ read_assay_state,
+ validate_legacy_graph_selection,
+ validate_normalized_artifact_selection,
+)
+from ...storage.artifacts import (
+ ArtifactRef,
+ group_at,
+ inspect_artifact,
+ parse_artifact_path,
+)
+from ...storage.types import as_zarr_array, as_zarr_group
+from ...utils.logging import logger
+
+if TYPE_CHECKING:
+ from ...metrics import ClusterSeparabilityResult
+ from ..mapping_datastore import MappingDatastore as _IntegrationMetricsBase
+else:
+ _IntegrationMetricsBase = object
+
+
+class _IntegrationMetricsOperationsMixin(_IntegrationMetricsBase):
+ def _keys_from_knn_path(
+ self,
+ from_assay: str,
+ knn_loc: str,
+ ) -> tuple[str, str]:
+ try:
+ ref = parse_artifact_path(knn_loc)
+ except ValueError:
+ (
+ path_assay,
+ cell_key,
+ feat_key,
+ ) = parse_assay_keys_from_nearest_neighbors_path(knn_loc)
+ if path_assay != from_assay:
+ raise ValueError(
+ f"KNN path belongs to assay {path_assay!r}, not {from_assay!r}"
+ )
+ validate_legacy_graph_selection(
+ self,
+ knn_loc,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ return cell_key, feat_key
+ if ref.kind != "neighbors":
+ raise ValueError(f"Not a neighbors artifact: {knn_loc}")
+ if ref.scope != "assay" or ref.assay != from_assay:
+ raise ValueError(
+ f"KNN artifact belongs to assay {ref.assay!r}, not {from_assay!r}"
+ )
+ state = read_assay_state(self.zw, from_assay)
+
+ def require_artifact(
+ candidate: ArtifactRef,
+ expected_kind: str,
+ ) -> Any:
+ status = inspect_artifact(self.zw, candidate)
+ if (
+ candidate.kind != expected_kind
+ or candidate.scope != "assay"
+ or candidate.assay != from_assay
+ or not status.exists
+ or not status.complete
+ ):
+ raise ValueError(f"{expected_kind} artifact is incomplete or invalid")
+ return status
+
+ def input_ref(status: Any, owner_kind: str, name: str) -> ArtifactRef:
+ value = (status.inputs or {}).get(name)
+ if not isinstance(value, dict):
+ raise ValueError(f"{owner_kind} has no {name!r} input")
+ return ArtifactRef.from_dict(value)
+
+ neighbors_status = require_artifact(ref, "neighbors")
+ coordinates = input_ref(neighbors_status, "neighbors", "coordinates")
+ ann_index = input_ref(neighbors_status, "neighbors", "ann_index")
+ ann_status = require_artifact(ann_index, "ann_index")
+ ann_coordinates = input_ref(ann_status, "ann_index", "coordinates")
+ if ann_coordinates != coordinates:
+ raise ValueError(
+ "Neighbors and ANN artifacts reference different coordinates"
+ )
+ if coordinates.kind not in {"reduction", "batch_correction"}:
+ raise ValueError("Neighbors coordinates have an invalid artifact kind")
+ coordinates_status = require_artifact(coordinates, coordinates.kind)
+ reduction = (
+ input_ref(coordinates_status, "batch_correction", "reduction")
+ if coordinates.kind == "batch_correction"
+ else coordinates
+ )
+ reduction_status = require_artifact(reduction, "reduction")
+ normalized = input_ref(reduction_status, "reduction", "normalized")
+ normalized_status = require_artifact(normalized, "normalized")
+ selected_cell_key: object
+ selected_feature_key: object
+ if state is not None and state.neighbors == ref:
+ selected_cell_key = state.cell_key
+ selected_feature_key = state.feat_key
+ else:
+ normalized_execution = normalized_status.execution_options or {}
+ selected_cell_key = normalized_execution.get("cell_key")
+ selected_feature_key = normalized_execution.get("feat_key")
+ if not isinstance(selected_cell_key, str) or not isinstance(
+ selected_feature_key,
+ str,
+ ):
+ raise ValueError("Selection source columns are missing")
+ validate_normalized_artifact_selection(
+ self.zw,
+ normalized,
+ selected_cell_key,
+ selected_feature_key,
+ )
+ return selected_cell_key, selected_feature_key
+
+ def _load_metric_knn(
+ self,
+ use_latest_knn: bool,
+ from_assay: str | None,
+ knn_loc: str | None,
+ ) -> tuple[str, str, str, zarr.Array, zarr.Array]:
+ if from_assay is None:
+ from_assay = self._load_default_assay()
+
+ if use_latest_knn and knn_loc is None:
+ resolved_knn_loc = self._get_latest_knn_loc(from_assay)
+ logger.debug(f"Using the latest KNN graph at {resolved_knn_loc}")
+ else:
+ if knn_loc is None:
+ raise ValueError("Please provide values for the KNN graph location.")
+ if knn_loc not in self.zw:
+ raise ValueError(f"Could not find the knn graph at location: {knn_loc}")
+ resolved_knn_loc = knn_loc
+ logger.debug(f"Using the KNN graph at {resolved_knn_loc}")
+
+ cell_key, _ = self._keys_from_knn_path(from_assay, resolved_knn_loc)
+ validate_distance_provenance(self.zw, resolved_knn_loc)
+ knn_grp = as_zarr_group(
+ self.zw[resolved_knn_loc],
+ name=resolved_knn_loc,
+ )
+ distances = as_zarr_array(knn_grp["distances"], name="distances")
+ indices = as_zarr_array(knn_grp["indices"], name="indices")
+ return from_assay, resolved_knn_loc, cell_key, distances, indices
+
+ def metric_lisi(
+ self,
+ label_columns: Sequence[str],
+ use_latest_knn: bool = True,
+ from_assay: str | None = None,
+ knn_loc: str | None = None,
+ perplexity: float = 30,
+ ) -> dict[str, np.ndarray]:
+ """Calculate Local Inverse Simpson Index (LISI) scores for cell populations.
+
+ LISI measures how well mixed different cell populations are in the local neighborhood
+ of each cell. Higher scores indicate better mixing of different populations.
+
+ Args:
+ label_columns: Column names from cell metadata containing population labels
+ use_latest_knn: Whether to use the most recent KNN graph (default: True)
+ from_assay: Name of assay to use if not using latest KNN
+ knn_loc: Location of KNN graph if not using latest (default: None)
+ perplexity: Effective neighborhood size used by LISI. It is reduced
+ with a warning when the graph has fewer than three times this
+ many neighbors.
+
+ Returns:
+ A mapping from each label column to its per-cell LISI scores.
+
+ Raises:
+ ValueError: If KNN inputs, perplexity, or labels are invalid
+ KeyError: If label columns not found in cell metadata
+
+ Notes:
+ LISI scores are computed for each label column separately.
+ Scores near 1 indicate cells grouped with similar labels.
+ Higher scores indicate more mixing between different labels.
+ """
+
+ if isinstance(label_columns, str):
+ raise TypeError("label_columns must be a sequence of column names")
+ label_cols = list(label_columns)
+ if not label_cols:
+ raise ValueError("label_columns must be non-empty")
+ if not all(isinstance(column, str) for column in label_cols):
+ raise TypeError("label_columns must contain only strings")
+ if len(set(label_cols)) != len(label_cols):
+ raise ValueError("label_columns contains duplicate names")
+
+ _, _, cell_key, distances, indices = self._load_metric_knn(
+ use_latest_knn,
+ from_assay,
+ knn_loc,
+ )
+ try:
+ metadata = self.cells.to_pandas_dataframe(columns=label_cols + [cell_key])
+ metadata = metadata[metadata[cell_key]]
+ except KeyError:
+ raise KeyError(
+ f"Could not find the column(s) {label_cols} in the cell metadata table."
+ )
+
+ from ...metrics import compute_lisi
+
+ lisi_scores = compute_lisi(
+ distances,
+ indices,
+ metadata,
+ label_cols,
+ perplexity=perplexity,
+ )
+ return {
+ column: scores
+ for column, scores in zip(label_cols, lisi_scores.T, strict=True)
+ }
+
+ def metric_ilisi(
+ self,
+ batch_colname: str,
+ use_latest_knn: bool = True,
+ from_assay: str | None = None,
+ knn_loc: str | None = None,
+ perplexity: float | None = None,
+ scale: bool = True,
+ ) -> float:
+ """Compute scIB integration LISI on a persisted KNN graph.
+
+ Args:
+ batch_colname: Cell metadata column containing batch labels.
+ use_latest_knn: Use the latest KNN graph when ``knn_loc`` is not
+ provided.
+ from_assay: Assay used to resolve the latest KNN graph.
+ knn_loc: Explicit persisted KNN location.
+ perplexity: Effective neighborhood size. ``None`` uses
+ ``floor(k / 3)``.
+ scale: Scale the median LISI by the number of observed batches.
+
+ Returns:
+ Median iLISI, scaled so higher values indicate better batch mixing
+ when ``scale`` is true.
+
+ Notes:
+ Scarf persisted KNN graphs exclude self-neighbors, as required by
+ this metric.
+ """
+ from ...metrics import ilisi_knn
+
+ _, _, cell_key, distances, indices = self._load_metric_knn(
+ use_latest_knn,
+ from_assay,
+ knn_loc,
+ )
+ batch_labels = self.cells.fetch(batch_colname, key=cell_key)
+ return ilisi_knn(
+ distances,
+ indices,
+ batch_labels,
+ perplexity=perplexity,
+ scale=scale,
+ )
+
+ def metric_clisi(
+ self,
+ label_colname: str,
+ use_latest_knn: bool = True,
+ from_assay: str | None = None,
+ knn_loc: str | None = None,
+ perplexity: float | None = None,
+ scale: bool = True,
+ ) -> float:
+ """Compute scIB cell-type LISI on a persisted KNN graph.
+
+ Args:
+ label_colname: Cell metadata column containing biological labels.
+ use_latest_knn: Use the latest KNN graph when ``knn_loc`` is not
+ provided.
+ from_assay: Assay used to resolve the latest KNN graph.
+ knn_loc: Explicit persisted KNN location.
+ perplexity: Effective neighborhood size. ``None`` uses
+ ``floor(k / 3)``.
+ scale: Invert and scale the median LISI by the number of observed
+ labels.
+
+ Returns:
+ Median cLISI, scaled so higher values indicate better label
+ conservation when ``scale`` is true.
+
+ Notes:
+ Scarf persisted KNN graphs exclude self-neighbors, as required by
+ this metric.
+ """
+ from ...metrics import clisi_knn
+
+ _, _, cell_key, distances, indices = self._load_metric_knn(
+ use_latest_knn,
+ from_assay,
+ knn_loc,
+ )
+ cell_labels = self.cells.fetch(label_colname, key=cell_key)
+ return clisi_knn(
+ distances,
+ indices,
+ cell_labels,
+ perplexity=perplexity,
+ scale=scale,
+ )
+
+ def metric_graph_connectivity(
+ self,
+ label_colname: str,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ graph_loc: str | None = None,
+ ) -> float:
+ """Score label connectivity on a persisted, symmetrized assay graph.
+
+ Args:
+ label_colname: Cell metadata column containing biological labels.
+ from_assay: Assay used to resolve the latest graph.
+ cell_key: Cell-selection key used to resolve the latest graph.
+ feat_key: Feature-selection key used to resolve the latest graph.
+ graph_loc: Explicit persisted standard-assay graph location.
+
+ Returns:
+ Mean fraction of cells retained in the largest connected component
+ for each label.
+
+ Notes:
+ Persisted directed edges are treated as undirected. This follows
+ the original scIB symmetrized-graph definition and intentionally
+ differs from the directed strong-component calculation currently
+ used by YosefLab ``scib-metrics``. Integrated graph locations are
+ rejected because they do not preserve safe cell-key provenance.
+ """
+ from ...metrics import graph_connectivity
+
+ if graph_loc is None:
+ from_assay, cell_key, feat_key = self._get_latest_keys(
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ graph_loc = self.get_latest_graph_loc(
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ else:
+ try:
+ explicit_ref = parse_artifact_path(graph_loc)
+ except ValueError:
+ explicit_ref = None
+ if is_integrated_graph_path(
+ graph_loc,
+ self._integratedGraphsLoc,
+ ) or (explicit_ref is not None and explicit_ref.kind == "integrated_graph"):
+ raise ValueError(
+ "Integrated graph connectivity is unavailable because the "
+ "graph does not record its cell-key provenance"
+ )
+ if graph_loc not in self.zw:
+ raise ValueError(f"Could not find the graph at location: {graph_loc}")
+
+ stored_graph = self._lookup_stored_graph(graph_loc=graph_loc)
+ if not isinstance(stored_graph, StoredAssayGraph):
+ raise ValueError("Graph connectivity requires an assay graph")
+
+ path_assay = stored_graph.from_assay
+ path_cell_key = stored_graph.cell_key
+ path_feat_key = stored_graph.feat_key
+ if from_assay is not None and from_assay != path_assay:
+ raise ValueError("from_assay does not match the graph location")
+ if cell_key is not None and cell_key != path_cell_key:
+ raise ValueError("cell_key does not match the graph location")
+ if feat_key is not None and feat_key != path_feat_key:
+ raise ValueError("feat_key does not match the graph location")
+ from_assay = path_assay
+ cell_key = path_cell_key
+ feat_key = path_feat_key
+
+ n_cells, _ = self._get_graph_ncells_k(graph_loc)
+ labels = self.cells.fetch(label_colname, key=cell_key)
+ if len(labels) != n_cells:
+ raise ValueError("Graph labels must match the number of cells in the graph")
+
+ graph_grp = as_zarr_group(self.zw[graph_loc], name=graph_loc)
+ edges = as_zarr_array(graph_grp["edges"], name="edges")
+ return graph_connectivity(edges, labels)
+
+ def metric_graph_silhouette(
+ self,
+ use_latest_knn: bool = True,
+ res_label: str = "leiden_cluster",
+ from_assay: str | None = None,
+ knn_loc: str | None = None,
+ random_seed: int = 4444,
+ sample_size: int = 11,
+ ) -> np.ndarray | None:
+ """Calculate modified silhouette scores for evaluating cluster separation.
+
+ This implements a graph-based silhouette score that measures how similar cells
+ are to their own cluster compared to the nearest neighboring cluster.
+
+ Args:
+ use_latest_knn: Whether to use most recent KNN graph (default: True)
+ res_label: Base or full column name containing cluster labels
+ (default: "leiden_cluster")
+ from_assay: Name of assay to use if not using latest KNN (default: None)
+ knn_loc: Location of KNN graph if not using latest (default: None)
+ random_seed: Seed used for cluster sampling.
+ sample_size: Maximum size of each sampled cluster group.
+
+ Returns:
+ numpy array of silhouette scores for each cluster, or None if computation fails
+
+ Raises:
+ ValueError: If graph, labels, sampling, or embedding data are invalid
+
+ Notes:
+ Scores range from -1 to 1:
+ - Near 1: Cluster is well-separated from neighboring clusters
+ - Near 0: Cluster overlaps with neighboring clusters
+ - Near -1: Cluster may be incorrectly assigned
+
+ Implementation uses sampling for efficiency with large datasets.
+ NaN values indicate clusters that couldn't be scored due to size constraints.
+ """
+
+ if from_assay is None:
+ from_assay = self._load_default_assay()
+
+ if use_latest_knn and knn_loc is None:
+ knn_loc = self._get_latest_knn_loc(from_assay)
+ logger.debug(
+ f"Using the latest knn graph at location: {knn_loc} for assay: {from_assay}"
+ )
+
+ else:
+ if knn_loc is None:
+ raise ValueError("Please provide values for the KNN graph location.")
+ if knn_loc not in self.zw:
+ raise ValueError(f"Could not find the knn graph at location: {knn_loc}")
+ logger.debug(f"Using the KNN graph at {knn_loc}")
+
+ from ...metrics import silhouette_scoring
+
+ cell_key, feat_key_parsed = self._keys_from_knn_path(from_assay, knn_loc)
+ ann_obj = self._load_ann_stream(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key_parsed,
+ knn_loc=knn_loc,
+ )
+
+ validate_distance_provenance(self.zw, knn_loc)
+ knn_grp = as_zarr_group(self.zw[knn_loc], name=knn_loc)
+ neighbor_indices = as_zarr_array(knn_grp["indices"], name="indices")
+ neighbor_distances = as_zarr_array(knn_grp["distances"], name="distances")
+ if ann_obj.harmonizedData is not None:
+ metric_data = ann_obj.harmonizedData
+ data_is_reduced = True
+ else:
+ if ann_obj.harmonize:
+ raise ValueError("Harmony coordinates are missing for this KNN graph")
+ metric_data = ann_obj.data
+ data_is_reduced = False
+ scores = silhouette_scoring(
+ self, # type: ignore[arg-type]
+ ann_obj,
+ None,
+ metric_data,
+ from_assay,
+ res_label,
+ cell_key=cell_key,
+ random_seed=random_seed,
+ sample_size=sample_size,
+ data_is_reduced=data_is_reduced,
+ distance_metric=cast(Any, ann_obj.annMetric),
+ neighbor_indices=neighbor_indices,
+ neighbor_distances=neighbor_distances,
+ )
+ return scores
+
+ def _validate_reduction_cell_selection(
+ self,
+ reduction: ArtifactRef,
+ cell_key: str,
+ ) -> None:
+ normalized = self._artifact_input_ref(reduction, "normalized", "normalized")
+ execution = (
+ self._require_complete_artifact(normalized, "normalized").execution_options
+ or {}
+ )
+ selected_cell_key = execution.get("cell_key")
+ selected_feat_key = execution.get("feat_key")
+ state = (
+ read_assay_state(self.zw, reduction.assay)
+ if reduction.assay is not None
+ else None
+ )
+ if state is not None and state.normalized == normalized:
+ selected_cell_key = state.cell_key
+ selected_feat_key = state.feat_key
+ if not isinstance(selected_cell_key, str) or not isinstance(
+ selected_feat_key,
+ str,
+ ):
+ raise ValueError("Reduction selection source columns are missing")
+ if cell_key != selected_cell_key:
+ raise ValueError(
+ f"cell_key {cell_key!r} does not match the cell selection "
+ f"{selected_cell_key!r} used to build the reduction"
+ )
+ validate_normalized_artifact_selection(
+ self.zw,
+ normalized,
+ selected_cell_key,
+ selected_feat_key,
+ )
+
+ def metric_cluster_separability(
+ self,
+ pca: ArtifactRef,
+ cluster_columns: Sequence[str],
+ *,
+ cell_key: str = "I",
+ n_folds: int = 5,
+ max_sample_cells: int = 50_000,
+ max_silhouette_cells: int = 10_000,
+ random_seed: int = 4444,
+ svm_c: float = 1.0,
+ svm_max_iter: int = 10_000,
+ ) -> "ClusterSeparabilityResult":
+ """Evaluate cluster-label separability in PCA coordinates."""
+ from ...metrics import evaluate_cluster_separability
+
+ if isinstance(cluster_columns, str):
+ raise TypeError("cluster_columns must be a sequence of column names")
+ columns = list(cluster_columns)
+ if not columns:
+ raise ValueError("cluster_columns must be non-empty")
+ if not all(isinstance(column, str) and column for column in columns):
+ raise TypeError("cluster_columns must contain non-empty strings")
+ if len(set(columns)) != len(columns):
+ raise ValueError("cluster_columns contains duplicate names")
+
+ status = self._require_complete_artifact(pca, "reduction")
+ if status.operation != "run_pca":
+ raise ValueError("pca must reference a PCA reduction artifact")
+ self._validate_reduction_cell_selection(pca, cell_key)
+ group = group_at(self.zw, status.path)
+ if "data" not in group:
+ raise ValueError("PCA reduction coordinates are missing")
+ coordinates = as_zarr_array(group["data"], name="PCA coordinates")
+ clusterings = {
+ column: np.asarray(self.cells.fetch(column, key=cell_key))
+ for column in columns
+ }
+ for column, labels in clusterings.items():
+ if len(labels) != coordinates.shape[0]:
+ raise ValueError(
+ f"Cluster column {column!r} does not align with PCA rows"
+ )
+
+ return evaluate_cluster_separability(
+ coordinates,
+ clusterings,
+ n_folds=n_folds,
+ max_sample_cells=max_sample_cells,
+ max_silhouette_cells=max_silhouette_cells,
+ random_seed=random_seed,
+ svm_c=svm_c,
+ svm_max_iter=svm_max_iter,
+ )
+
+ def metric_label_concordance(
+ self,
+ label_columns: Sequence[str],
+ metric: Literal["ari", "nmi"] = "ari",
+ *,
+ cell_key: str = "I",
+ ) -> float:
+ """Compare two metadata label partitions using ARI or NMI.
+
+ This measures whether two labelings of the same cells agree, for
+ example predicted clusters against imported reference annotations. It
+ does not measure batch mixing; use :meth:`metric_ilisi`,
+ :meth:`metric_proportional_batch_mixing`, or :meth:`metric_lisi`
+ for that.
+
+ Args:
+ label_columns: Exactly two cell metadata column names to compare.
+ metric: ``"ari"`` for the adjusted Rand index or ``"nmi"`` for
+ normalized mutual information.
+ cell_key: Boolean cell metadata column selecting rows to compare.
+
+ Returns:
+ Agreement between the two partitions. ARI ranges from -1 to 1 and
+ NMI from 0 to 1, with higher values meaning stronger agreement.
+
+ Raises:
+ ValueError: If the number of columns or the metric name is invalid.
+ """
+ from ...metrics import label_concordance_score
+
+ label_values = [
+ np.asarray(self.cells.fetch(column, key=cell_key))
+ for column in label_columns
+ ]
+ return label_concordance_score(label_values, metric)
+
+ def metric_proportional_batch_mixing(
+ self,
+ label_colname: str,
+ use_latest_knn: bool = True,
+ from_assay: str | None = None,
+ knn_loc: str | None = None,
+ perplexity: float = 30,
+ ) -> float:
+ """Summarize batch LISI as a normalized neighborhood-mixing score.
+
+ This computes batch LISI on the current KNN graph and rescales its mean
+ against the mixing that perfectly integrated data would reach given the
+ dataset's batch sizes. Unlike raw LISI, the result is bounded in
+ ``[0, 1]``, which makes it easier to compare across graphs and datasets.
+
+ Args:
+ label_colname: Cell metadata column holding the batch assignment.
+ use_latest_knn: Whether to use the most recent KNN graph
+ (default: True).
+ from_assay: Name of assay to use if not using the latest KNN.
+ knn_loc: Location of the KNN graph if not using the latest.
+ perplexity: Effective neighborhood size passed to LISI.
+
+ Returns:
+ A value in ``[0, 1]``. Scores near 1 indicate that neighborhoods mix
+ batches as well as the global composition allows, and scores near 0
+ indicate poorly mixed batches.
+
+ Raises:
+ ValueError: If KNN inputs are invalid or the column has fewer than
+ two batches.
+ """
+ from ...metrics import lisi_batch_mixing_score
+
+ if from_assay is None:
+ from_assay = self._load_default_assay()
+ resolved_knn_loc = knn_loc
+ if use_latest_knn and resolved_knn_loc is None:
+ resolved_knn_loc = self._get_latest_knn_loc(from_assay)
+ if resolved_knn_loc is None:
+ raise ValueError("Please provide values for the KNN graph location.")
+
+ lisi_result = self.metric_lisi(
+ label_columns=[label_colname],
+ use_latest_knn=use_latest_knn,
+ from_assay=from_assay,
+ knn_loc=resolved_knn_loc,
+ perplexity=perplexity,
+ )
+
+ cell_key, _ = self._keys_from_knn_path(
+ from_assay,
+ resolved_knn_loc,
+ )
+ batch_labels = self.cells.fetch(label_colname, key=cell_key)
+ return lisi_batch_mixing_score(lisi_result[label_colname], batch_labels)
diff --git a/scarf/datastore/_operations/mapping.py b/scarf/datastore/_operations/mapping.py
new file mode 100644
index 00000000..b19e5f46
--- /dev/null
+++ b/scarf/datastore/_operations/mapping.py
@@ -0,0 +1,1354 @@
+import os
+from collections.abc import Generator
+from typing import TYPE_CHECKING, Any, cast
+
+import numpy as np
+import pandas as pd
+import zarr
+
+from ...graph.state import validate_cell_selection_artifact
+from ...metadata.artifacts import (
+ artifact_values,
+ column_display,
+ continuous_display,
+ link_cell_data_column,
+ plan_cell_data_artifact,
+ write_cell_data_artifact,
+)
+from ...storage.artifacts import (
+ ArtifactRef,
+ ValueFingerprintBuilder,
+ artifact_group,
+ canonical_bytes,
+ fingerprint_array,
+ inspect_artifact,
+)
+from ...storage.types import as_zarr_array
+from ...assay import RNAassay
+from ...mapping.features import AlignedFeatureStream
+from ...mapping.models import MappingResult
+from ...mapping.projection import (
+ NO_QUERY_BATCH_FINGERPRINT,
+ PROJECTION_RERUN_MESSAGE,
+ ProjectionWriter,
+ load_projection,
+ plan_projection,
+ resolve_projection,
+)
+from ...mapping.reference import MappingReference
+from ...mapping.symphony import (
+ SYMPHONY_ALGORITHM,
+ accumulate_sufficient_statistics,
+ apply_query_correction,
+ initialize_sufficient_statistics,
+ project_pca,
+ scaled_dispersion_sum,
+ soft_cluster_assignments,
+ solve_query_correction,
+ zero_norm_rows,
+)
+from ...neighbors.stages import (
+ AnnIndexStage,
+ NeighborQueryStage,
+)
+from ...storage.ann_index import has_ann_index, load_ann_index
+from ...storage.geometry import array_geometry
+from ...storage.partition import row_band
+from ...storage.selections import resolve_selection_artifact
+from ...storage.stores import zarr_root_path
+from ...utils.logging import logger
+
+if TYPE_CHECKING:
+ from ..graph_datastore import GraphDataStore as _MappingOperationsBase
+else:
+ _MappingOperationsBase = object
+
+
+def _finite_in_range(
+ value: Any,
+ message: str,
+ *,
+ low: float,
+ high: float | None = None,
+ low_open: bool = False,
+) -> float:
+ """Coerce one numeric argument, rejecting bools and out-of-range values."""
+ if isinstance(value, bool | np.bool_) or not isinstance(
+ value,
+ int | float | np.integer | np.floating,
+ ):
+ raise ValueError(message)
+ resolved = float(value)
+ if not np.isfinite(resolved) or resolved < low:
+ raise ValueError(message)
+ if low_open and resolved == low:
+ raise ValueError(message)
+ if high is not None and resolved > high:
+ raise ValueError(message)
+ return resolved
+
+
+def _normalized_store_location(value: object) -> str | None:
+ if not isinstance(value, str) or not value:
+ return None
+ location = value.rstrip("/")
+ if location.startswith("file://"):
+ location = location[7:]
+ if "://" in location:
+ return location
+ return os.path.realpath(os.path.abspath(os.path.expanduser(location)))
+
+
+def _physical_store_tokens(datastore: Any) -> set[tuple[str, str | int]]:
+ root = datastore.z
+ store = root.store
+ tokens: set[tuple[str, str | int]] = {("object", id(store))}
+ root_path = _normalized_store_location(zarr_root_path(root))
+ if root_path is not None:
+ tokens.add(("root", root_path))
+ location = _normalized_store_location(getattr(datastore, "zarr_loc", None))
+ if location is not None:
+ tokens.add(("location", location))
+ store_root = _normalized_store_location(str(getattr(store, "root", "")))
+ if store_root is not None:
+ tokens.add(("root", store_root))
+ return tokens
+
+
+def _same_physical_store(query: Any, reference: MappingReference) -> bool:
+ reference_datastore = reference.datastore
+ if not hasattr(reference_datastore, "z"):
+ raise TypeError("reference.datastore must be an open DataStore")
+ return bool(
+ _physical_store_tokens(query) & _physical_store_tokens(reference_datastore)
+ )
+
+
+def _query_batch_fingerprint(query_batches: pd.DataFrame) -> str:
+ columns = [
+ {
+ "type": f"{type(column).__module__}.{type(column).__qualname__}",
+ "value": repr(column),
+ }
+ for column in query_batches.columns
+ ]
+ row_hashes = pd.util.hash_pandas_object(
+ query_batches,
+ index=False,
+ categorize=True,
+ ).to_numpy(dtype=np.uint64, copy=True)
+ builder = ValueFingerprintBuilder()
+ builder.update_bytes(
+ "query_batch_schema",
+ canonical_bytes(
+ {
+ "columns": columns,
+ "dtypes": [str(dtype) for dtype in query_batches.dtypes],
+ }
+ ),
+ )
+ builder.update_array("query_batch_rows", row_hashes)
+ return builder.hexdigest()
+
+
+def _mapping_memory_reservations(
+ reference: MappingReference,
+ *,
+ n_batches: int,
+ save_k: int,
+ batch_codes: np.ndarray | None,
+) -> tuple[int, int]:
+ float_bytes = np.dtype(np.float64).itemsize
+ model_arrays = (
+ reference.model.feature_means,
+ reference.model.feature_scales,
+ reference.model.loadings,
+ reference.feature_ids,
+ )
+ resident = sum(np.asarray(values).nbytes for values in model_arrays)
+ if batch_codes is not None:
+ resident += batch_codes.nbytes
+
+ n_features = reference.model.n_features
+ n_dims = reference.model.n_dims
+ per_row = (
+ 2 * n_features * float_bytes
+ + 2 * n_dims * float_bytes
+ + save_k * (np.dtype(np.uint64).itemsize + float_bytes)
+ + np.dtype(bool).itemsize
+ )
+
+ symphony = reference.symphony_state
+ if symphony is None:
+ return resident, per_row
+
+ correction_arrays = (
+ symphony.centroids,
+ symphony.raw_centroids,
+ symphony.corrected_centroids,
+ symphony.cluster_mass,
+ symphony.sigma,
+ )
+ resident += sum(np.asarray(values).nbytes for values in correction_arrays)
+ count_bytes = n_batches * symphony.n_clusters * float_bytes
+ sum_bytes = count_bytes * symphony.n_dims
+ solve_bytes = (n_batches + 1) ** 2 * float_bytes + 2 * (
+ n_batches + 1
+ ) * symphony.n_dims * float_bytes
+ resident += 2 * count_bytes + 2 * sum_bytes + solve_bytes
+ per_row += (
+ 3 * symphony.n_clusters + 3 * symphony.n_dims
+ ) * float_bytes + symphony.n_clusters * symphony.n_dims * float_bytes
+ return resident, per_row
+
+
+def _reference_available_k(reference: MappingReference) -> int:
+ root = reference.datastore.zw
+ rebuild = "Rebuild it with build_mapping_reference(neighbors)."
+ if reference.ref.assay != reference.assay_name:
+ raise ValueError(f"Mapping reference assay identity is inconsistent. {rebuild}")
+ if reference.model.n_features != len(reference.feature_ids):
+ raise ValueError(
+ f"Mapping reference feature dimensions are inconsistent. {rebuild}"
+ )
+ if reference.method == "pca":
+ if reference.symphony_state is not None:
+ raise ValueError(f"Plain mapping reference has Symphony state. {rebuild}")
+ elif reference.method == "symphony":
+ if reference.symphony_state is None:
+ raise ValueError(
+ f"Symphony mapping reference has no correction state. {rebuild}"
+ )
+ else:
+ raise ValueError(f"Mapping reference method is unsupported. {rebuild}")
+
+ expected = (
+ (reference.ref, "build_mapping_reference"),
+ (reference.reduction, "run_pca"),
+ (reference.ann_index, "build_ann_index"),
+ (reference.neighbors, "query_neighbors"),
+ )
+ statuses = {}
+ for ref, operation in expected:
+ status = inspect_artifact(root, ref)
+ if not status.exists or not status.complete or status.operation != operation:
+ raise ValueError(f"Mapping reference graph chain is incomplete. {rebuild}")
+ statuses[ref] = status
+
+ ann_status = statuses[reference.ann_index]
+ ann_parameters = ann_status.parameters or {}
+ if ann_parameters.get("ann_metric") != reference.ann_metric:
+ raise ValueError(f"Mapping reference ANN metric is inconsistent. {rebuild}")
+ ann_ef = ann_parameters.get("ann_ef", 50)
+ if isinstance(ann_ef, bool) or not isinstance(ann_ef, int) or ann_ef < 1:
+ raise ValueError(f"Mapping reference ANN search depth is invalid. {rebuild}")
+
+ neighbors_status = statuses[reference.neighbors]
+ raw_ann = (neighbors_status.inputs or {}).get("ann_index")
+ if (
+ not isinstance(raw_ann, dict)
+ or ArtifactRef.from_dict(raw_ann) != reference.ann_index
+ ):
+ raise ValueError(
+ f"Mapping reference neighbors use another ANN index. {rebuild}"
+ )
+ if (neighbors_status.parameters or {}).get(
+ "distance_metric"
+ ) != reference.ann_metric:
+ raise ValueError(
+ f"Mapping reference neighbor metric is inconsistent. {rebuild}"
+ )
+
+ neighbors_group = artifact_group(root, reference.neighbors)
+ indices = as_zarr_array(neighbors_group["indices"], name="indices")
+ distances = as_zarr_array(neighbors_group["distances"], name="distances")
+ if (
+ indices.ndim != 2
+ or distances.shape != indices.shape
+ or int(indices.shape[0]) != reference.selected_cell_count
+ or int(indices.shape[1]) < 1
+ ):
+ raise ValueError(f"Mapping reference neighbor payload is invalid. {rebuild}")
+ ann_group = artifact_group(root, reference.ann_index)
+ if not has_ann_index(ann_group):
+ raise ValueError(f"Mapping reference ANN index is missing. {rebuild}")
+ return int(indices.shape[1])
+
+
+def _load_reference_neighbor_query(
+ reference: MappingReference,
+ *,
+ save_k: int,
+ workers: int,
+) -> NeighborQueryStage:
+ root = reference.datastore.zw
+ ann_status = inspect_artifact(root, reference.ann_index)
+ parameters = ann_status.parameters or {}
+ index = load_ann_index(
+ artifact_group(root, reference.ann_index),
+ reference.ann_metric,
+ reference.model.n_dims,
+ expected_count=reference.selected_cell_count,
+ )
+ configured = AnnIndexStage.configure(
+ index,
+ ef=int(parameters.get("ann_ef", 50)),
+ threads=workers,
+ )
+ return NeighborQueryStage(configured, save_k, reference.ann_metric)
+
+
+class _MappingOperationsMixin(_MappingOperationsBase):
+ @staticmethod
+ def _projection_block_size(indices: Any) -> int:
+ return row_band(
+ array_geometry(indices),
+ unit="chunk",
+ fallback=min(max(int(indices.shape[0]), 1), 10_000),
+ )
+
+ def _projection_arrays(
+ self,
+ ref: ArtifactRef,
+ ) -> tuple[zarr.Array, zarr.Array, zarr.Array]:
+ projection = artifact_group(self.zw, ref)
+ return (
+ as_zarr_array(projection["indices"], name="indices"),
+ as_zarr_array(projection["distances"], name="distances"),
+ as_zarr_array(projection["uninformative"], name="uninformative"),
+ )
+
+ def _projection_cell_selection(
+ self,
+ ref: ArtifactRef,
+ ) -> tuple[ArtifactRef, str]:
+ status = inspect_artifact(self.zw, ref)
+ raw_selection = (status.inputs or {}).get("cell_selection")
+ try:
+ if not isinstance(raw_selection, dict):
+ raise TypeError("cell_selection input is missing")
+ cell_selection = ArtifactRef.from_dict(raw_selection)
+ selection_status = inspect_artifact(self.zw, cell_selection)
+ cell_key = (selection_status.execution_options or {}).get("source_column")
+ if not isinstance(cell_key, str) or not cell_key:
+ raise ValueError("cell selection has no source column")
+ validate_cell_selection_artifact(
+ self.zw,
+ cell_selection,
+ cell_key,
+ )
+ except (KeyError, TypeError, ValueError) as exc:
+ raise ValueError(
+ f"Projection cell selection is invalid: {exc}. "
+ f"{PROJECTION_RERUN_MESSAGE}"
+ ) from None
+ return cell_selection, cell_key
+
+ def run_mapping(
+ self,
+ reference: MappingReference,
+ mapping_name: str,
+ *,
+ query_assay: str | None = None,
+ cell_key: str = "I",
+ save_k: int = 3,
+ missing_feature_policy: str = "reference_mean",
+ query_batches: pd.DataFrame | None = None,
+ invalidate_cache: bool = False,
+ ) -> MappingResult:
+ """Map selected query cells into an immutable prepared reference."""
+ if not isinstance(reference, MappingReference):
+ raise TypeError("reference must be a MappingReference")
+ if not isinstance(mapping_name, str) or not mapping_name.strip():
+ raise TypeError("mapping_name must be a non-empty string")
+ if query_assay is not None and (
+ not isinstance(query_assay, str) or not query_assay.strip()
+ ):
+ raise TypeError("query_assay must be a non-empty string or None")
+ if not isinstance(cell_key, str) or not cell_key.strip():
+ raise TypeError("cell_key must be a non-empty string")
+ if isinstance(save_k, bool) or not isinstance(save_k, int | np.integer):
+ raise TypeError("save_k must be an integer")
+ save_k = int(save_k)
+ if save_k < 1:
+ raise ValueError("save_k must be positive")
+ if not isinstance(missing_feature_policy, str):
+ raise TypeError("missing_feature_policy must be a string")
+ if missing_feature_policy not in {"reference_mean", "zero", "error"}:
+ raise ValueError(
+ "missing_feature_policy must be 'reference_mean', 'zero', or 'error'"
+ )
+ if query_batches is not None and not isinstance(query_batches, pd.DataFrame):
+ raise TypeError("query_batches must be a pandas DataFrame or None")
+ if not isinstance(invalidate_cache, bool):
+ raise TypeError("invalidate_cache must be a boolean")
+ if self.zarr_mode != "r+":
+ raise ValueError("Mapping requires a read-write query datastore")
+ if _same_physical_store(self, reference):
+ raise ValueError(
+ "Query and reference cannot use the same physical Zarr store. "
+ "Mount the query into a separate writable datastore."
+ )
+
+ reference.validate_dataset_fingerprint()
+ assay_name = query_assay or self._defaultAssay
+ if assay_name is None:
+ raise ValueError("No query assay was provided and no default is configured")
+ if assay_name not in self.assay_names:
+ raise ValueError(f"Query assay {assay_name!r} was not found")
+ assay = self._get_assay(assay_name)
+ if not isinstance(assay, RNAassay):
+ raise TypeError("Mapping currently supports RNA query assays only")
+
+ cell_mask = np.asarray(self.cells.fetch_all(cell_key))
+ if cell_mask.ndim != 1 or cell_mask.dtype != np.dtype(bool):
+ raise TypeError("cell_key must identify a one-dimensional boolean column")
+ if len(cell_mask) != self.cells.N:
+ raise ValueError("cell_key does not align with query cell metadata")
+ query_cell_indices = np.flatnonzero(cell_mask).astype(np.int64, copy=False)
+ n_cells = len(query_cell_indices)
+ if n_cells < 1:
+ raise ValueError("cell_key must select at least one query cell")
+
+ symphony_state = reference.symphony_state
+ if symphony_state is None:
+ if query_batches is not None:
+ raise ValueError(
+ "query_batches are only supported by Symphony references"
+ )
+ batch_codes = None
+ n_batches = 1
+ query_batch_fingerprint = NO_QUERY_BATCH_FINGERPRINT
+ correction_method = "none"
+ algorithm_variant = "scaled_pca"
+ else:
+ if query_batches is None:
+ batch_codes = np.zeros(n_cells, dtype=np.int64)
+ n_batches = 1
+ query_batch_fingerprint = NO_QUERY_BATCH_FINGERPRINT
+ else:
+ batch_codes, n_batches = self._query_batch_codes(
+ query_batches,
+ n_cells,
+ )
+ query_batch_fingerprint = _query_batch_fingerprint(query_batches)
+ correction_method = "symphony"
+ algorithm_variant = SYMPHONY_ALGORITHM
+
+ available_k = _reference_available_k(reference)
+ if save_k > available_k:
+ logger.warning(f"`save_k` was decreased to {available_k}")
+ save_k = available_k
+
+ reserved_resident, reserved_per_row = _mapping_memory_reservations(
+ reference,
+ n_batches=n_batches,
+ save_k=save_k,
+ batch_codes=batch_codes,
+ )
+ stream = AlignedFeatureStream(
+ query_assay=assay,
+ query_cell_indices=query_cell_indices,
+ reference_feature_ids=reference.feature_ids,
+ reference_normalized_means=reference.model.feature_means,
+ reference_normalization_parameters=reference.normalization_parameters,
+ missing_feature_policy=missing_feature_policy,
+ resources=self.resources,
+ reserved_resident_bytes=reserved_resident,
+ reserved_per_row_bytes=reserved_per_row,
+ )
+ selected_expression_fingerprint = stream.raw_expression_fingerprint
+
+ cell_selection = resolve_selection_artifact(
+ self.zw,
+ scope="datastore",
+ kind="cell_selection",
+ values=cell_mask,
+ row_ids=np.asarray(self.cells.fetch_all("ids")),
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column=cell_key,
+ )
+ feature_mask = np.zeros(assay.feats.N, dtype=bool)
+ feature_mask[stream.query_feature_indices] = True
+ feature_selection = resolve_selection_artifact(
+ self.zw,
+ scope="assay",
+ assay=assay_name,
+ kind="feature_selection",
+ values=feature_mask,
+ row_ids=np.asarray(assay.feats.fetch_all("ids")),
+ operation="map_query_feature_selection",
+ parameters={},
+ inputs={
+ "alignment_map_hash": stream.alignment_map_hash,
+ "mapping_reference": reference.external_ref,
+ },
+ source_column="mapping_reference_overlap",
+ )
+ projection_plan = plan_projection(
+ self.zw,
+ query_assay=assay_name,
+ mapping_name=mapping_name,
+ n_cells=n_cells,
+ save_k=save_k,
+ missing_feature_policy=missing_feature_policy,
+ correction_method=correction_method,
+ cell_selection=cell_selection,
+ feature_selection=feature_selection,
+ selected_expression_fingerprint=selected_expression_fingerprint,
+ query_batch_fingerprint=query_batch_fingerprint,
+ mapping_reference=reference.external_ref,
+ reference_cell_count=reference.selected_cell_count,
+ invalidate_cache=invalidate_cache,
+ )
+ if projection_plan.reused:
+ return load_projection(
+ self.zw,
+ projection_plan.ref,
+ reference=reference,
+ )
+
+ writer = ProjectionWriter(
+ self.zw,
+ projection_plan,
+ chunk_rows=stream.row_geometry.block_rows,
+ profile=self.storageProfile,
+ )
+ try:
+ neighbor_query = _load_reference_neighbor_query(
+ reference,
+ save_k=save_k,
+ workers=self.resources.workers,
+ )
+ if symphony_state is not None:
+ assert batch_codes is not None
+ counts, sums = initialize_sufficient_statistics(
+ n_batches,
+ symphony_state,
+ )
+ expected_start = 0
+ for block in stream:
+ if block.row_offset != expected_start:
+ raise RuntimeError("Aligned query blocks are not contiguous")
+ coordinates = project_pca(block.values, reference.model)
+ assignments = soft_cluster_assignments(
+ coordinates,
+ symphony_state,
+ )
+ stop = block.row_offset + len(coordinates)
+ informative = ~zero_norm_rows(coordinates)
+ if informative.any():
+ accumulate_sufficient_statistics(
+ counts,
+ sums,
+ coordinates[informative],
+ assignments[informative],
+ batch_codes[block.row_offset : stop][informative],
+ )
+ expected_start = stop
+ if expected_start != n_cells:
+ raise RuntimeError(
+ "Symphony statistics did not cover all query cells"
+ )
+ correction = solve_query_correction(
+ counts,
+ sums,
+ symphony_state,
+ )
+
+ zero_norm_count = 0
+ expected_start = 0
+ dispersion_total = 0.0
+ informative_total = 0
+ for block in stream:
+ if block.row_offset != expected_start:
+ raise RuntimeError("Aligned query blocks are not contiguous")
+ coordinates = project_pca(block.values, reference.model)
+ uninformative = zero_norm_rows(coordinates)
+ zero_norm_count += int(np.count_nonzero(uninformative))
+ informative = ~uninformative
+ if informative.any():
+ dispersion_total += scaled_dispersion_sum(
+ block.values[informative],
+ reference.model,
+ )
+ informative_total += int(np.count_nonzero(informative))
+ query_coordinates = coordinates
+ if symphony_state is not None:
+ assert batch_codes is not None
+ assignments = soft_cluster_assignments(
+ coordinates,
+ symphony_state,
+ )
+ stop = block.row_offset + len(coordinates)
+ query_coordinates = apply_query_correction(
+ coordinates,
+ assignments,
+ batch_codes[block.row_offset : stop],
+ symphony_state,
+ correction,
+ )
+ query_coordinates[uninformative] = coordinates[uninformative]
+ queried = cast(
+ tuple[np.ndarray, np.ndarray],
+ neighbor_query.query(query_coordinates),
+ )
+ indices, distances = queried
+ writer.write_block(
+ block.row_offset,
+ np.asarray(indices, dtype=np.uint64),
+ np.asarray(distances, dtype=np.float64),
+ np.asarray(uninformative, dtype=bool),
+ )
+ expected_start = block.row_offset + len(coordinates)
+ if expected_start != n_cells:
+ raise RuntimeError("Mapping did not cover all selected query cells")
+ denominator = informative_total * reference.model.n_features
+ writer.finish(
+ {
+ "featureCoverage": stream.feature_coverage,
+ "queryBatchCount": n_batches,
+ "algorithmVariant": algorithm_variant,
+ "zeroNormCellCount": zero_norm_count,
+ "queryScaledDispersion": (
+ dispersion_total / denominator if denominator else 0.0
+ ),
+ }
+ )
+ return load_projection(
+ self.zw,
+ projection_plan.ref,
+ reference=reference,
+ )
+ except BaseException:
+ if writer.finished:
+ artifact_group(self.zw, writer.ref).attrs["complete"] = False
+ else:
+ writer.abort()
+ raise
+
+ @staticmethod
+ def _query_batch_codes(
+ query_batches: pd.DataFrame, n_cells: int
+ ) -> tuple[np.ndarray, int]:
+ if len(query_batches) != n_cells:
+ raise ValueError("query_batches must have one row per selected query cell")
+ if query_batches.shape[1] == 0:
+ raise ValueError("query_batches must include at least one column")
+ if query_batches.columns.duplicated().any():
+ raise ValueError("query_batches column names must be unique")
+ if query_batches.isna().any().any():
+ raise ValueError("query_batches cannot contain missing values")
+ rows = pd.MultiIndex.from_frame(query_batches)
+ codes, levels = pd.factorize(rows, sort=False)
+ resolved = np.asarray(codes, dtype=np.int64)
+ if np.any(resolved < 0):
+ raise ValueError("query_batches contain an unencodable row")
+ return resolved, len(levels)
+
+ @staticmethod
+ def _label_vote_decision(
+ reference_labels: np.ndarray,
+ neighbors: np.ndarray,
+ weights: np.ndarray,
+ threshold_fraction: float,
+ na_val: str,
+ force_unknown: bool = False,
+ ) -> tuple[Any, float, float, float, bool, dict[Any, float]]:
+ votes: dict[Any, float] = {}
+ for neighbor, weight in zip(neighbors, weights):
+ label = reference_labels[neighbor]
+ votes[label] = votes.get(label, 0.0) + float(weight)
+ total = float(sum(votes.values()))
+ if total <= 0 or not votes:
+ return na_val, 0.0, 0.0, 0.0, True, {}
+ votes = {label: value / total for label, value in votes.items()}
+ ordered = sorted(votes.items(), key=lambda item: item[1], reverse=True)
+ top_vote = ordered[0][1]
+ second_vote = ordered[1][1] if len(ordered) > 1 else 0.0
+ winners = [label for label, vote in ordered if np.isclose(vote, top_vote)]
+ is_unknown = force_unknown or top_vote < threshold_fraction or len(winners) != 1
+ entropy = -sum(value * np.log(value) for _, value in ordered if value > 0)
+ prediction = na_val if is_unknown else winners[0]
+ return (
+ prediction,
+ top_vote,
+ float(entropy),
+ top_vote - second_vote,
+ is_unknown,
+ votes,
+ )
+
+ def get_mapping_result(
+ self,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None = None,
+ query_assay: str | None = None,
+ load_arrays: bool = False,
+ ) -> MappingResult:
+ """Load one complete query-owned mapping projection."""
+ if reference is not None and not isinstance(reference, MappingReference):
+ raise TypeError("reference must be a MappingReference or None")
+ if query_assay is not None and not isinstance(result, str):
+ raise ValueError("query_assay is only valid when result is a string")
+
+ projection_ref: ArtifactRef
+ session_reference: MappingReference | None = None
+ if isinstance(result, MappingResult):
+ projection_ref = result.ref
+ session_reference = result.reference
+ elif isinstance(result, ArtifactRef):
+ projection_ref = result
+ elif isinstance(result, str):
+ if not result.strip():
+ raise TypeError("result must be a non-empty mapping name")
+ else:
+ raise TypeError(
+ "result must be a MappingResult, ArtifactRef, or mapping name"
+ )
+
+ if reference is not None and session_reference is not None:
+ if reference.external_ref != session_reference.external_ref:
+ raise ValueError(
+ "Explicit reference does not match the MappingResult "
+ "reference handle"
+ )
+ resolved_reference = reference or session_reference
+ if resolved_reference is None:
+ raise ValueError(
+ "A MappingReference is required unless result is an in-session "
+ "MappingResult carrying its reference"
+ )
+
+ if isinstance(result, str):
+ assay_name = query_assay or self._defaultAssay
+ if assay_name is None:
+ raise ValueError(
+ "query_assay is required when the query store has no default assay"
+ )
+ projection_ref = resolve_projection(
+ self.zw,
+ query_assay=assay_name,
+ mapping_name=result,
+ mapping_reference=resolved_reference.external_ref,
+ )
+
+ return load_projection(
+ self.zw,
+ projection_ref,
+ load_arrays=load_arrays,
+ reference=resolved_reference,
+ )
+
+ def get_mapping_score(
+ self,
+ result: MappingResult | ArtifactRef | str,
+ target_groups: np.ndarray | None = None,
+ *,
+ reference: MappingReference | None = None,
+ query_assay: str | None = None,
+ log_transform: bool = True,
+ multiplier: float = 1000,
+ weighted: bool = True,
+ fixed_weight: float = 0.1,
+ ) -> Generator[tuple[Any, np.ndarray], None, None]:
+ """Yield reference-sized mapping scores for each requested query group."""
+ if not isinstance(log_transform, bool):
+ raise TypeError("log_transform must be a boolean")
+ if not isinstance(weighted, bool):
+ raise TypeError("weighted must be a boolean")
+ scale = _finite_in_range(
+ multiplier,
+ "multiplier must be finite and non-negative",
+ low=0.0,
+ )
+ weight = _finite_in_range(
+ fixed_weight,
+ "fixed_weight must be finite and positive",
+ low=0.0,
+ low_open=True,
+ )
+
+ loaded = self.get_mapping_result(
+ result,
+ reference=reference,
+ query_assay=query_assay,
+ load_arrays=False,
+ )
+ assert loaded.reference is not None
+ indices, distances, uninformative = self._projection_arrays(loaded.ref)
+ n_cells = loaded.n_cells
+ n_k = int(indices.shape[1])
+
+ if target_groups is None:
+ groups = np.zeros(n_cells, dtype=np.uint8)
+ else:
+ groups = np.asarray(target_groups)
+ if groups.shape != (n_cells,):
+ raise ValueError(
+ "target_groups must contain one value per projected query cell"
+ )
+ requested_groups = pd.unique(groups)
+
+ from ...mapping.confidence import mapping_score_weights
+
+ for group in requested_groups:
+ score = np.zeros(
+ loaded.reference.selected_cell_count,
+ dtype=np.float64,
+ )
+ informative_count = 0
+ block_size = self._projection_block_size(indices)
+ for start in range(0, n_cells, block_size):
+ stop = min(start + block_size, n_cells)
+ block_groups = groups[start:stop]
+ block_uninformative = np.asarray(
+ uninformative[start:stop],
+ dtype=bool,
+ )
+ if bool(pd.isna(group)):
+ group_mask = np.asarray(pd.isna(block_groups), dtype=bool)
+ else:
+ group_mask = np.asarray(block_groups == group, dtype=bool)
+ informative_mask = group_mask & ~block_uninformative
+ if not informative_mask.any():
+ continue
+ block_indices = np.asarray(indices[start:stop])[informative_mask]
+ if weighted:
+ block_weights = mapping_score_weights(
+ np.asarray(distances[start:stop])[informative_mask]
+ )
+ else:
+ block_weights = np.full(
+ block_indices.shape,
+ weight,
+ dtype=np.float64,
+ )
+ np.add.at(
+ score,
+ block_indices.reshape(-1),
+ block_weights.reshape(-1),
+ )
+ informative_count += int(np.count_nonzero(informative_mask))
+ if informative_count:
+ score *= scale / (informative_count * n_k)
+ if log_transform:
+ score = np.log1p(score)
+ yield group, score
+
+ def get_target_classes(
+ self,
+ result: MappingResult | ArtifactRef | str,
+ reference_class_group: str,
+ *,
+ reference: MappingReference | None = None,
+ query_assay: str | None = None,
+ threshold_fraction: float = 0.5,
+ target_subset: list[int] | None = None,
+ na_val: str = "NA",
+ ) -> pd.Series:
+ """Transfer one reference label column to projected query cells."""
+ if not isinstance(reference_class_group, str) or not reference_class_group:
+ raise TypeError("reference_class_group must be a non-empty string")
+ threshold = _finite_in_range(
+ threshold_fraction,
+ "threshold_fraction must be between zero and one",
+ low=0.0,
+ high=1.0,
+ )
+ if not isinstance(na_val, str):
+ raise TypeError("na_val must be a string")
+
+ loaded = self.get_mapping_result(
+ result,
+ reference=reference,
+ query_assay=query_assay,
+ load_arrays=False,
+ )
+ assert loaded.reference is not None
+ indices, distances, uninformative = self._projection_arrays(loaded.ref)
+ reference_labels = loaded.reference.fetch_cell_column(reference_class_group)
+
+ target_subset_set: dict[int, None] | None = None
+ if target_subset is not None:
+ if not isinstance(target_subset, list):
+ raise TypeError("target_subset must be a list or None")
+ target_subset_set = {}
+ for index in target_subset:
+ if isinstance(index, bool | np.bool_) or not isinstance(
+ index,
+ int | np.integer,
+ ):
+ raise TypeError("target_subset entries must be integers")
+ resolved_index = int(index)
+ if not 0 <= resolved_index < loaded.n_cells:
+ raise ValueError("target_subset contains an out-of-range index")
+ target_subset_set[resolved_index] = None
+
+ from ...mapping.confidence import distance_weights
+
+ preds: list[Any] = []
+ prediction_indices: list[int] = []
+ block_size = self._projection_block_size(indices)
+ for start in range(0, loaded.n_cells, block_size):
+ stop = min(start + block_size, loaded.n_cells)
+ if target_subset_set is None:
+ selected_offsets = np.arange(stop - start, dtype=np.intp)
+ else:
+ selected_offsets = np.asarray(
+ [
+ offset
+ for offset in range(stop - start)
+ if start + offset in target_subset_set
+ ],
+ dtype=np.intp,
+ )
+ if not selected_offsets.size:
+ continue
+ block_uninformative = np.asarray(
+ uninformative[start:stop],
+ dtype=bool,
+ )
+ selected_uninformative = block_uninformative[selected_offsets]
+ informative_offsets = selected_offsets[~selected_uninformative]
+ informative_neighbors: np.ndarray | None = None
+ informative_weights: np.ndarray | None = None
+ if informative_offsets.size:
+ informative_neighbors = np.asarray(indices[start:stop])[
+ informative_offsets
+ ]
+ informative_weights = distance_weights(
+ np.asarray(distances[start:stop])[informative_offsets]
+ )
+ informative_position = 0
+ for selected_position, offset in enumerate(selected_offsets):
+ row_index = start + int(offset)
+ if selected_uninformative[selected_position]:
+ prediction = na_val
+ else:
+ assert informative_neighbors is not None
+ assert informative_weights is not None
+ prediction, _, _, _, _, _ = self._label_vote_decision(
+ reference_labels,
+ informative_neighbors[informative_position],
+ informative_weights[informative_position],
+ threshold,
+ na_val,
+ )
+ informative_position += 1
+ preds.append(prediction)
+ prediction_indices.append(row_index)
+ return pd.Series(preds, index=prediction_indices)
+
+ def get_target_label_evidence(
+ self,
+ result: MappingResult | ArtifactRef | str,
+ reference_class_group: str,
+ *,
+ reference: MappingReference | None = None,
+ query_assay: str | None = None,
+ threshold_fraction: float = 0.5,
+ na_val: str = "NA",
+ max_distance: float | None = None,
+ calibration_nonconformity: np.ndarray | None = None,
+ conformal_alpha: float = 0.1,
+ ) -> pd.DataFrame:
+ """Return neighbor-vote evidence, novelty context, and unknown assignments.
+
+ ``calibration_nonconformity`` optionally adds split-conformal prediction
+ sets. Its calibration rows must be exchangeable with future queries.
+ """
+ if not isinstance(reference_class_group, str) or not reference_class_group:
+ raise TypeError("reference_class_group must be a non-empty string")
+ threshold = _finite_in_range(
+ threshold_fraction,
+ "threshold_fraction must be between zero and one",
+ low=0.0,
+ high=1.0,
+ )
+ if not isinstance(na_val, str):
+ raise TypeError("na_val must be a string")
+ distance_limit = (
+ None
+ if max_distance is None
+ else _finite_in_range(
+ max_distance,
+ "max_distance must be finite and non-negative",
+ low=0.0,
+ )
+ )
+
+ loaded = self.get_mapping_result(
+ result,
+ reference=reference,
+ query_assay=query_assay,
+ load_arrays=False,
+ )
+ assert loaded.reference is not None
+ indices, distances, uninformative = self._projection_arrays(loaded.ref)
+ reference_labels = loaded.reference.fetch_cell_column(reference_class_group)
+ class_labels = np.asarray(pd.unique(reference_labels), dtype=object)
+ class_positions = {
+ label: position for position, label in enumerate(class_labels)
+ }
+
+ from ...mapping.confidence import distance_weights
+
+ predictions = np.full(loaded.n_cells, na_val, dtype=object)
+ vote_fraction = np.full(loaded.n_cells, np.nan, dtype=np.float64)
+ vote_entropy = np.full(loaded.n_cells, np.nan, dtype=np.float64)
+ top_two_margin = np.full(loaded.n_cells, np.nan, dtype=np.float64)
+ best_distances = np.full(loaded.n_cells, np.nan, dtype=np.float64)
+ label_scores = (
+ np.zeros(
+ (loaded.n_cells, len(class_labels)),
+ dtype=np.float64,
+ )
+ if calibration_nonconformity is not None
+ else None
+ )
+ is_unknown = np.ones(loaded.n_cells, dtype=bool)
+ block_size = self._projection_block_size(indices)
+ for start in range(0, loaded.n_cells, block_size):
+ stop = min(start + block_size, loaded.n_cells)
+ block_uninformative = np.asarray(
+ uninformative[start:stop],
+ dtype=bool,
+ )
+ informative_offsets = np.flatnonzero(~block_uninformative)
+ if not informative_offsets.size:
+ continue
+ block_indices = np.asarray(indices[start:stop])[informative_offsets]
+ block_distances = np.asarray(distances[start:stop])[informative_offsets]
+ block_weights = distance_weights(block_distances)
+ for position, offset in enumerate(informative_offsets):
+ row_index = start + int(offset)
+ (
+ prediction,
+ top_vote,
+ entropy,
+ margin,
+ row_unknown,
+ votes,
+ ) = self._label_vote_decision(
+ reference_labels,
+ block_indices[position],
+ block_weights[position],
+ threshold,
+ na_val,
+ )
+ best_distance = float(block_distances[position, 0])
+ if distance_limit is not None and best_distance > distance_limit:
+ prediction = na_val
+ row_unknown = True
+ predictions[row_index] = prediction
+ vote_fraction[row_index] = top_vote
+ vote_entropy[row_index] = entropy
+ top_two_margin[row_index] = margin
+ best_distances[row_index] = best_distance
+ is_unknown[row_index] = row_unknown
+ if label_scores is not None:
+ for label, score in votes.items():
+ label_scores[row_index, class_positions[label]] = score
+
+ distance_quantiles = loaded.reference.reference_distance_quantiles
+ distance_values = loaded.reference.reference_distance_values
+ unique_distance_values = np.unique(distance_values)
+ right_indices = (
+ np.searchsorted(distance_values, unique_distance_values, side="right") - 1
+ )
+ unique_distance_quantiles = distance_quantiles[right_indices]
+ distance_percentile = np.full(
+ loaded.n_cells,
+ np.nan,
+ dtype=np.float64,
+ )
+ informative = np.isfinite(best_distances)
+ if informative.any():
+ distance_percentile[informative] = np.interp(
+ best_distances[informative],
+ unique_distance_values,
+ unique_distance_quantiles,
+ left=0.0,
+ right=1.0,
+ )
+ feature_coverage = float(loaded.diagnostics["featureCoverage"])
+ evidence = pd.DataFrame(
+ {
+ "label": predictions,
+ "voteFraction": vote_fraction,
+ "voteEntropy": vote_entropy,
+ "topTwoMargin": top_two_margin,
+ "featureCoverage": feature_coverage,
+ "queryScaledDispersion": float(
+ loaded.diagnostics["queryScaledDispersion"]
+ ),
+ "referenceDistancePercentile": distance_percentile,
+ "isUnknown": is_unknown,
+ }
+ )
+ if calibration_nonconformity is not None:
+ from ...mapping.confidence import conformal_prediction_sets
+
+ assert label_scores is not None
+ prediction_masks = conformal_prediction_sets(
+ label_scores,
+ calibration_nonconformity,
+ alpha=conformal_alpha,
+ )
+ evidence["predictionSet"] = [
+ tuple(class_labels[mask].tolist()) if informative[row_index] else ()
+ for row_index, mask in enumerate(prediction_masks)
+ ]
+ return evidence
+
+ @staticmethod
+ def calibrate_label_transfer_threshold(
+ vote_fractions: np.ndarray,
+ correct: np.ndarray,
+ target_coverage: float = 0.9,
+ ) -> dict[str, float]:
+ """Choose a vote threshold on held-out, donor-level validation data."""
+ fractions = np.asarray(vote_fractions, dtype=np.float64)
+ correct = np.asarray(correct, dtype=bool)
+ if fractions.ndim != 1 or correct.shape != fractions.shape:
+ raise ValueError("vote_fractions and correct must be matching vectors")
+ if not 0 < target_coverage <= 1:
+ raise ValueError("target_coverage must be in (0, 1]")
+ valid = fractions[correct]
+ if valid.size == 0:
+ raise ValueError("At least one correct held-out prediction is required")
+ threshold = float(np.quantile(valid, 1 - target_coverage))
+ selected = fractions >= threshold
+ accuracy = float(correct[selected].mean()) if selected.any() else 0.0
+ return {
+ "voteThreshold": threshold,
+ "validationCoverage": float(selected.mean()),
+ "validationAccuracy": accuracy,
+ }
+
+ def _load_mapping_reference_embedding(
+ self,
+ result: MappingResult | ArtifactRef | str,
+ reference_layout_key: str,
+ *,
+ reference: MappingReference | None = None,
+ query_assay: str | None = None,
+ ) -> np.ndarray:
+ """Load persisted query coordinates for one reference layout."""
+ if not isinstance(reference_layout_key, str) or not reference_layout_key:
+ raise TypeError("reference_layout_key must be a non-empty string")
+ loaded = self.get_mapping_result(
+ result,
+ reference=reference,
+ query_assay=query_assay,
+ load_arrays=False,
+ )
+ assert loaded.reference is not None
+ query_assay_name = loaded.ref.assay
+ if query_assay_name is None:
+ raise ValueError(
+ f"Projection has no query assay. {PROJECTION_RERUN_MESSAGE}"
+ )
+ cell_selection, _ = self._projection_cell_selection(loaded.ref)
+ reference_layout = loaded.reference.fetch_layout(reference_layout_key)
+ layout_source = loaded.reference.layout_source(reference_layout_key)
+ layout_input = (
+ layout_source.to_dict()
+ if layout_source is not None
+ else {"value_fingerprint": fingerprint_array(reference_layout)}
+ )
+ expected_inputs = {
+ "projection": loaded.ref.to_dict(),
+ "reference_layout": layout_input,
+ "cell_selection": cell_selection.to_dict(),
+ }
+ candidates: list[tuple[int, ArtifactRef]] = []
+ for ref in self.list_artifacts(
+ kind="embedding",
+ from_assay=query_assay_name,
+ scope="assay",
+ complete_only=True,
+ ):
+ status = inspect_artifact(self.zw, ref)
+ if (
+ status.operation != "project_reference_embedding"
+ or status.parameters != {"reference_layout_key": reference_layout_key}
+ or status.inputs != expected_inputs
+ ):
+ continue
+ group = artifact_group(self.zw, ref)
+ created_at_ns = group.attrs.get("created_at_ns")
+ if (
+ isinstance(created_at_ns, bool | np.bool_)
+ or not isinstance(created_at_ns, int | np.integer)
+ or int(created_at_ns) < 1
+ ):
+ raise ValueError(
+ "Persisted reference embedding has an invalid creation time"
+ )
+ candidates.append((int(created_at_ns), ref))
+
+ if not candidates:
+ raise ValueError(
+ "No persisted query reference embedding matches mapping "
+ f"{loaded.mapping_name!r} and layout {reference_layout_key!r}. "
+ "Call query_store.project_reference_embedding("
+ "result, reference_layout_key=..., reference=reference) before plotting."
+ )
+ candidates.sort(
+ key=lambda item: (item[0], item[1].artifact_id),
+ reverse=True,
+ )
+ embedding_ref = candidates[0][1]
+ values = artifact_values(
+ artifact_group(self.zw, embedding_ref),
+ "values",
+ )
+ if values.shape != (loaded.n_cells, 2):
+ raise ValueError(
+ "Persisted query reference embedding must have shape "
+ f"({loaded.n_cells}, 2)"
+ )
+ if values.dtype.kind != "f":
+ raise TypeError(
+ "Persisted query reference embedding must use a floating dtype"
+ )
+ coordinates = np.asarray(values, dtype=np.float64)
+ if not np.all(np.isfinite(coordinates) | np.isnan(coordinates)):
+ raise ValueError(
+ "Persisted query reference embedding contains infinite coordinates"
+ )
+ return np.array(coordinates, copy=True)
+
+ def project_reference_embedding(
+ self,
+ result: MappingResult | ArtifactRef | str,
+ reference_layout_key: str,
+ *,
+ reference: MappingReference | None = None,
+ query_assay: str | None = None,
+ label: str = "ref_UMAP",
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ """Persist query cells projected into an unchanged reference layout."""
+ if not isinstance(reference_layout_key, str) or not reference_layout_key:
+ raise TypeError("reference_layout_key must be a non-empty string")
+ if not isinstance(label, str) or not label:
+ raise TypeError("label must be a non-empty string")
+ if not isinstance(invalidate_cache, bool):
+ raise TypeError("invalidate_cache must be a boolean")
+ if self.zarr_mode != "r+":
+ raise ValueError(
+ "Projecting a reference embedding requires a read-write query store"
+ )
+
+ loaded = self.get_mapping_result(
+ result,
+ reference=reference,
+ query_assay=query_assay,
+ load_arrays=False,
+ )
+ assert loaded.reference is not None
+ query_assay_name = loaded.ref.assay
+ if query_assay_name is None:
+ raise ValueError(
+ f"Projection has no query assay. {PROJECTION_RERUN_MESSAGE}"
+ )
+
+ cell_selection, cell_key = self._projection_cell_selection(loaded.ref)
+
+ reference_layout = loaded.reference.fetch_layout(reference_layout_key)
+ layout_source = loaded.reference.layout_source(reference_layout_key)
+ layout_input: object = (
+ layout_source
+ if layout_source is not None
+ else {"value_fingerprint": fingerprint_array(reference_layout)}
+ )
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope="assay",
+ assay=query_assay_name,
+ kind="embedding",
+ operation="project_reference_embedding",
+ parameters={"reference_layout_key": reference_layout_key},
+ inputs={
+ "projection": loaded.ref,
+ "reference_layout": layout_input,
+ },
+ execution_options={
+ "cell_key": cell_key,
+ "label": label,
+ },
+ cell_selection=cell_selection,
+ arrays={"values": ((loaded.n_cells, 2), "f")},
+ invalidate_cache=invalidate_cache,
+ )
+ columns = [
+ self._col_renamer(
+ query_assay_name,
+ cell_key,
+ f"{label}{dimension}",
+ )
+ for dimension in (1, 2)
+ ]
+ preserved_displays = [column_display(self.zw, column) for column in columns]
+ if planned.reused:
+ values = artifact_values(
+ artifact_group(self.zw, planned.ref),
+ "values",
+ )
+ else:
+ from ...mapping.confidence import distance_weights
+
+ indices, distances, uninformative = self._projection_arrays(loaded.ref)
+ values = np.full(
+ (loaded.n_cells, 2),
+ np.nan,
+ dtype=np.float64,
+ )
+ block_size = self._projection_block_size(indices)
+ for start in range(0, loaded.n_cells, block_size):
+ stop = min(start + block_size, loaded.n_cells)
+ informative = ~np.asarray(
+ uninformative[start:stop],
+ dtype=bool,
+ )
+ if not informative.any():
+ continue
+ block_indices = np.asarray(indices[start:stop])[informative]
+ block_weights = distance_weights(
+ np.asarray(distances[start:stop])[informative]
+ )
+ block_values = values[start:stop]
+ block_values[informative] = np.einsum(
+ "nk,nkd->nd",
+ block_weights,
+ reference_layout[block_indices],
+ )
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {"values": values},
+ )
+
+ for dimension, column in enumerate(columns):
+ self.cells.insert(
+ column,
+ values[:, dimension],
+ key=cell_key,
+ overwrite=True,
+ )
+ link_cell_data_column(
+ self.zw,
+ column,
+ planned.ref,
+ value_name="values",
+ value_index=dimension,
+ default_display=continuous_display(values[:, dimension]),
+ preserved_display=preserved_displays[dimension],
+ )
+ return planned.ref
diff --git a/scarf/datastore/_operations/mapping_reference.py b/scarf/datastore/_operations/mapping_reference.py
new file mode 100644
index 00000000..0332615a
--- /dev/null
+++ b/scarf/datastore/_operations/mapping_reference.py
@@ -0,0 +1,452 @@
+from collections.abc import Mapping
+from typing import TYPE_CHECKING, Any
+
+import numpy as np
+
+from ...assay import RNAassay
+from ...graph.distances import validate_distance_provenance
+from ...graph.state import read_assay_state, validate_normalized_artifact_selection
+from ...mapping.artifact import (
+ MAPPING_REFERENCE_REBUILD_MESSAGE,
+ load_artifact_mapping_reference,
+ write_artifact_mapping_reference,
+)
+from ...mapping.confidence import _distance_quantile_summary
+from ...mapping.models import (
+ ScaledPCAProjectionModel,
+ SymphonyCorrectionModel,
+)
+from ...mapping.reference import MappingReference
+from ...storage.artifact_writer import (
+ AttributeRequirement,
+ finish_artifact,
+ plan_artifact,
+ start_artifact,
+)
+from ...storage.artifacts import (
+ ArtifactRef,
+ artifact_group,
+ artifact_path,
+)
+from ...storage.types import as_zarr_array
+
+if TYPE_CHECKING:
+ from .graph import _GraphOperationsMixin as _MappingReferenceOperationsBase
+else:
+ _MappingReferenceOperationsBase = object
+
+
+class _MappingReferenceOperationsMixin(_MappingReferenceOperationsBase):
+ def get_mapping_reference(
+ self,
+ reference: ArtifactRef | None = None,
+ *,
+ from_assay: str | None = None,
+ ) -> MappingReference:
+ """Load one mapping reference from an explicit or named artifact."""
+ resolved: ArtifactRef | None
+ if reference is not None:
+ if not isinstance(reference, ArtifactRef):
+ raise TypeError("reference must be an ArtifactRef or None")
+ if (
+ reference.scope != "assay"
+ or reference.assay is None
+ or reference.kind != "mapping_reference"
+ ):
+ raise ValueError(
+ "reference must identify an assay-scoped mapping_reference artifact"
+ )
+ if from_assay is not None and reference.assay != from_assay:
+ raise ValueError(
+ f"reference belongs to assay {reference.assay!r}, not "
+ f"{from_assay!r}"
+ )
+ resolved = reference
+ else:
+ assay_name = from_assay or self._defaultAssay
+ if assay_name is None:
+ raise ValueError("No assay was provided and no default is configured")
+ state = read_assay_state(self.zw, assay_name)
+ resolved = (
+ state.named_results.get("mapping_reference")
+ if state is not None
+ else None
+ )
+ if resolved is None:
+ raise ValueError(
+ "The selected assay has no mapping reference under "
+ "AssayState.named_results. " + MAPPING_REFERENCE_REBUILD_MESSAGE
+ )
+ assert resolved is not None
+ return load_artifact_mapping_reference(self, resolved)
+
+ def build_mapping_reference(
+ self,
+ neighbors: ArtifactRef,
+ *,
+ invalidate_cache: bool = False,
+ ) -> MappingReference:
+ """Package an existing scaled-PCA neighbor chain for mapping."""
+ if self.zarr_mode != "r+":
+ raise ValueError("Building a mapping reference requires a read-write store")
+ if not isinstance(neighbors, ArtifactRef):
+ raise TypeError("neighbors must be an ArtifactRef")
+ if not isinstance(invalidate_cache, bool):
+ raise TypeError("invalidate_cache must be a boolean")
+ if (
+ neighbors.scope != "assay"
+ or neighbors.assay is None
+ or neighbors.kind != "neighbors"
+ ):
+ raise ValueError(
+ "neighbors must identify an assay-scoped neighbors artifact"
+ )
+ assay_name = neighbors.assay
+ assay = self._get_assay(assay_name)
+ if not isinstance(assay, RNAassay):
+ raise TypeError("Mapping references currently support RNA assays only")
+
+ neighbors_status = self._require_complete_artifact(
+ neighbors,
+ "neighbors",
+ assay=assay_name,
+ )
+ if neighbors_status.operation != "query_neighbors":
+ raise ValueError("Mapping references require query_neighbors artifacts")
+ ann_index = self._artifact_input_ref(
+ neighbors,
+ "ann_index",
+ "ann_index",
+ )
+ ann_status = self._require_complete_artifact(
+ ann_index,
+ "ann_index",
+ assay=assay_name,
+ )
+ if ann_status.operation != "build_ann_index":
+ raise ValueError("Mapping references require build_ann_index artifacts")
+ coordinates = _artifact_input(neighbors_status.inputs, "coordinates")
+ if (
+ coordinates.scope != "assay"
+ or coordinates.assay != assay_name
+ or coordinates.kind not in {"reduction", "batch_correction"}
+ ):
+ raise ValueError(
+ "Neighbor coordinates must be a PCA reduction or batch correction"
+ )
+ self._require_complete_artifact(
+ coordinates,
+ coordinates.kind,
+ assay=assay_name,
+ )
+ if (
+ self._artifact_input_ref(
+ ann_index,
+ "coordinates",
+ coordinates.kind,
+ )
+ != coordinates
+ ):
+ raise ValueError("Neighbors and ANN index use different coordinates")
+
+ batch_correction = None
+ if coordinates.kind == "batch_correction":
+ batch_correction = coordinates
+ correction_status = self._require_complete_artifact(
+ batch_correction,
+ "batch_correction",
+ assay=assay_name,
+ )
+ if correction_status.operation != "run_harmony":
+ raise ValueError(
+ "Symphony mapping references require run_harmony correction"
+ )
+ reduction = self._artifact_input_ref(
+ batch_correction,
+ "reduction",
+ "reduction",
+ )
+ method = "symphony"
+ else:
+ correction_status = None
+ reduction = coordinates
+ method = "pca"
+
+ reduction_status = self._require_complete_artifact(
+ reduction,
+ "reduction",
+ assay=assay_name,
+ )
+ if reduction_status.operation != "run_pca":
+ raise ValueError("Mapping references require a run_pca reduction")
+ if (reduction_status.parameters or {}).get("feat_scaling") is not True:
+ raise ValueError("Mapping references require PCA with feature scaling")
+ normalized = self._artifact_input_ref(
+ reduction,
+ "normalized",
+ "normalized",
+ )
+ feature_scaling = self._artifact_input_ref(
+ reduction,
+ "feature_scaling",
+ "feature_scaling",
+ )
+ normalized_status = self._require_complete_artifact(
+ normalized,
+ "normalized",
+ assay=assay_name,
+ )
+ scaling_status = self._require_complete_artifact(
+ feature_scaling,
+ "feature_scaling",
+ assay=assay_name,
+ )
+ if normalized_status.operation != "run_normalization":
+ raise ValueError("Mapping references require a run_normalization artifact")
+ if (
+ scaling_status.operation != "calculate_feature_scaling"
+ or (scaling_status.parameters or {}).get("enabled") is not True
+ or self._artifact_input_ref(
+ feature_scaling,
+ "normalized",
+ "normalized",
+ )
+ != normalized
+ ):
+ raise ValueError(
+ "Mapping references require enabled scaling for the same normalized data"
+ )
+ cell_selection = self._artifact_input_ref(
+ normalized,
+ "cell_selection",
+ "cell_selection",
+ )
+ feature_selection = self._artifact_input_ref(
+ normalized,
+ "feature_selection",
+ "feature_selection",
+ )
+
+ normalized_execution = normalized_status.execution_options or {}
+ cell_key = normalized_execution.get("cell_key")
+ feature_key = normalized_execution.get("feat_key")
+ if (
+ not isinstance(cell_key, str)
+ or not cell_key
+ or not isinstance(feature_key, str)
+ or not feature_key
+ ):
+ raise ValueError(
+ "Normalized artifact is missing its cell or feature selection key"
+ )
+ validate_normalized_artifact_selection(
+ self.zw,
+ normalized,
+ cell_key,
+ feature_key,
+ )
+
+ ann_metric = (ann_status.parameters or {}).get("ann_metric")
+ if ann_metric not in {"l2", "cosine"}:
+ raise ValueError(
+ "Mapping references support only l2 and cosine ANN metrics"
+ )
+ if (neighbors_status.parameters or {}).get("distance_metric") != ann_metric:
+ raise ValueError("Neighbor and ANN distance metrics do not match")
+ normalization_parameters = normalized_status.parameters or {}
+ size_factor = normalization_parameters.get("size_factor")
+ if (
+ isinstance(size_factor, bool)
+ or not isinstance(size_factor, int | float)
+ or not np.isfinite(size_factor)
+ or float(size_factor) <= 0
+ ):
+ raise ValueError(
+ "Mapping reference normalization size_factor must be finite and positive"
+ )
+
+ reduction_group = artifact_group(self.zw, reduction)
+ scaling_group = artifact_group(self.zw, feature_scaling)
+ neighbors_group = artifact_group(self.zw, neighbors)
+ loadings = np.asarray(
+ as_zarr_array(reduction_group["loadings"], name="loadings")[:]
+ )
+ model = ScaledPCAProjectionModel(
+ feature_means=np.asarray(
+ as_zarr_array(scaling_group["mean"], name="mean")[:]
+ ),
+ feature_scales=np.asarray(
+ as_zarr_array(scaling_group["scale"], name="scale")[:]
+ ),
+ loadings=loadings,
+ )
+ feature_column = (
+ feature_key if feature_key == "I" else f"{cell_key}__{feature_key}"
+ )
+ feature_ids = np.asarray(assay.feats.fetch("ids", key=feature_column))
+ if len(feature_ids) != model.n_features:
+ raise ValueError("Selected reference features do not match PCA loadings")
+ selected_cell_count = len(self.cells.fetch("ids", key=cell_key))
+ if selected_cell_count < 1:
+ raise ValueError("Mapping references require at least one selected cell")
+ reduction_data = as_zarr_array(reduction_group["data"], name="data")
+ distances = as_zarr_array(neighbors_group["distances"], name="distances")
+ if (
+ int(reduction_data.shape[0]) != selected_cell_count
+ or int(distances.shape[0]) != selected_cell_count
+ ):
+ raise ValueError(
+ "PCA and neighbor rows must match the selected reference cells"
+ )
+
+ symphony_state = None
+ if correction_status is not None and batch_correction is not None:
+ correction_group = artifact_group(self.zw, batch_correction)
+ symphony_state = SymphonyCorrectionModel(
+ centroids=np.asarray(
+ as_zarr_array(
+ correction_group["centroids"],
+ name="centroids",
+ )[:]
+ ).T,
+ raw_centroids=np.asarray(
+ as_zarr_array(
+ correction_group["raw_centroids"],
+ name="raw_centroids",
+ )[:]
+ ),
+ corrected_centroids=np.asarray(
+ as_zarr_array(
+ correction_group["corrected_centroids"],
+ name="corrected_centroids",
+ )[:]
+ ),
+ cluster_mass=np.asarray(
+ as_zarr_array(
+ correction_group["cluster_mass"],
+ name="cluster_mass",
+ )[:]
+ ),
+ sigma=np.asarray(
+ as_zarr_array(correction_group["sigma"], name="sigma")[:]
+ ),
+ )
+ if symphony_state.n_dims != model.n_dims:
+ raise ValueError(
+ "Harmony correction dimensions do not match PCA loadings"
+ )
+
+ validate_distance_provenance(self.zw, artifact_path(neighbors))
+ distance_quantiles, distance_values = _distance_quantile_summary(distances)
+ dataset_fingerprint = self._ensure_dataset_fingerprint(assay_name)
+ metadata: dict[str, Any] = {
+ "method": method,
+ "assay": assay_name,
+ "cell_key": cell_key,
+ "feature_key": feature_key,
+ "selected_cell_count": selected_cell_count,
+ "ann_metric": ann_metric,
+ "normalization_parameters": dict(normalization_parameters),
+ "dataset_fingerprint": dataset_fingerprint,
+ }
+ if correction_status is not None and batch_correction is not None:
+ correction_parameters = correction_status.parameters or {}
+ correction_group = artifact_group(self.zw, batch_correction)
+ batch_levels = correction_group.attrs.get("batch_levels", [])
+ if not isinstance(batch_levels, list):
+ raise ValueError("Harmony batch levels must be a list")
+ metadata.update(
+ {
+ "batch_columns": list(
+ correction_parameters.get("batch_columns", [])
+ ),
+ "harmony_parameters": dict(
+ correction_parameters.get("harmony_parameters", {})
+ ),
+ "batch_levels": batch_levels,
+ }
+ )
+
+ inputs: dict[str, ArtifactRef] = {
+ "reduction": reduction,
+ "ann_index": ann_index,
+ "neighbors": neighbors,
+ "cell_selection": cell_selection,
+ "feature_selection": feature_selection,
+ }
+ if batch_correction is not None:
+ inputs["batch_correction"] = batch_correction
+ required_arrays: tuple[str, ...] = (
+ "feature_ids",
+ "feature_means",
+ "feature_scales",
+ "loadings",
+ "reference_distance_quantiles",
+ "reference_distance_values",
+ )
+ required_attributes: tuple[str | AttributeRequirement, ...] = (
+ AttributeRequirement(
+ "reference_metadata",
+ expected_types=(dict,),
+ ),
+ )
+ if symphony_state is not None:
+ required_arrays += (
+ "centroids",
+ "raw_centroids",
+ "corrected_centroids",
+ "cluster_mass",
+ "sigma",
+ )
+
+ def valid_reference(candidate: ArtifactRef, _group: Any) -> bool:
+ if not bool(_group.attrs.get("complete", False)):
+ return True
+ try:
+ load_artifact_mapping_reference(self, candidate)
+ except (KeyError, TypeError, ValueError):
+ return False
+ return True
+
+ planned = plan_artifact(
+ self.zw,
+ scope="assay",
+ assay=assay_name,
+ kind="mapping_reference",
+ operation="build_mapping_reference",
+ parameters={"method": method},
+ inputs=inputs,
+ execution_options=dict(getattr(self, "_artifactExecutionContext", {})),
+ invalidate_cache=invalidate_cache,
+ required_arrays=required_arrays,
+ required_attributes=required_attributes,
+ reuse_validator=valid_reference,
+ )
+ if not planned.reused:
+ group = start_artifact(self.zw, planned)
+ write_artifact_mapping_reference(
+ group,
+ model,
+ symphony_state,
+ feature_ids,
+ metadata,
+ distance_quantiles,
+ distance_values,
+ )
+ finish_artifact(group, planned)
+ self._publish_current_artifact(
+ neighbors,
+ update_state=True,
+ named_results={"mapping_reference": planned.ref},
+ )
+ return load_artifact_mapping_reference(self, planned.ref)
+
+
+def _artifact_input(
+ inputs: Mapping[str, Any] | None,
+ name: str,
+) -> ArtifactRef:
+ raw_ref = (inputs or {}).get(name)
+ if not isinstance(raw_ref, Mapping):
+ raise ValueError(f"Artifact has no {name!r} input")
+ return ArtifactRef.from_dict(raw_ref)
diff --git a/scarf/datastore/_operations/paris_persistence.py b/scarf/datastore/_operations/paris_persistence.py
new file mode 100644
index 00000000..56b9fa11
--- /dev/null
+++ b/scarf/datastore/_operations/paris_persistence.py
@@ -0,0 +1,496 @@
+from math import prod
+from typing import Literal, cast
+
+import numpy as np
+import zarr
+
+from ...clustering._paris_core import ParisHierarchy
+from ...clustering.paris_multiscale import PlateauForest
+from ...storage.arrays import create_zarr_dataset
+from ...storage.budget import ResourceBudget
+from ...storage.types import as_zarr_array, as_zarr_group
+
+PARIS_HIERARCHY_ROOT = "paris_hierarchy"
+LATEST_PARIS_GENERATION = "latest_paris_generation"
+_PARIS_HIERARCHY_ARRAYS = (
+ "children",
+ "heights",
+ "sizes",
+ "component_roots",
+ "synthetic_joins",
+)
+_PARIS_PLATEAU_ARRAYS = (
+ "representatives",
+ "heights",
+ "sizes",
+ "parent_events",
+ "child_offsets",
+ "child_refs",
+ "min_leaves",
+ "component_roots",
+)
+_MEMORY_HEADROOM = 1.35
+_CACHED_FIXED_TRANSIENT_BYTES_PER_CELL = 128
+_CACHED_ADAPTIVE_TRANSIENT_BYTES_PER_CELL = 96
+
+
+def generation_location(graph_loc: str, generation_id: str) -> str:
+ return f"{graph_loc}/{PARIS_HIERARCHY_ROOT}/{generation_id}"
+
+
+def _paris_memory_components(
+ n_cells: int,
+ edge_count: int,
+ edge_itemsize: int,
+ weight_itemsize: int,
+) -> tuple[int, int, int, int, int, int, int, int]:
+ largest_index = max(2 * n_cells - 1, 2 * edge_count)
+ index_bytes = 4 if largest_index <= np.iinfo(np.int32).max else 8
+ pointer_bytes = (n_cells + 1) * index_bytes
+ stored_edges = edge_count * (2 * edge_itemsize + weight_itemsize)
+ directed_csr = edge_count * (8 + index_bytes) + pointer_bytes
+ canonical_edges = 2 * edge_count
+ canonical_csr = canonical_edges * (8 + index_bytes) + pointer_bytes
+ load_peak = stored_edges + directed_csr
+ symmetrize_peak = 2 * directed_csr + 2 * canonical_csr
+ hierarchy_bytes = (n_cells - 1) * (2 * index_bytes + 8 + 4 + 1)
+ plateau_bytes = n_cells * (6 * index_bytes + 8 + 4)
+ modularity_guard_bytes = n_cells * 128
+ return (
+ index_bytes,
+ load_peak,
+ symmetrize_peak,
+ directed_csr,
+ canonical_csr,
+ hierarchy_bytes,
+ plateau_bytes,
+ modularity_guard_bytes,
+ )
+
+
+def estimate_paris_peak_bytes(
+ n_cells: int,
+ edge_count: int,
+ edge_itemsize: int,
+ weight_itemsize: int,
+ *,
+ n_threads: int = 1,
+) -> int:
+ """Conservatively estimate peak bytes for canonical Paris fitting."""
+ (
+ index_bytes,
+ load_peak,
+ symmetrize_peak,
+ directed_csr,
+ canonical_csr,
+ hierarchy_bytes,
+ plateau_bytes,
+ modularity_guard_bytes,
+ ) = _paris_memory_components(
+ n_cells,
+ edge_count,
+ edge_itemsize,
+ weight_itemsize,
+ )
+ contraction_workspaces = canonical_csr
+ contraction_layout = n_cells * (6 * index_bytes)
+ contraction_thread_tables = max(1, n_threads) * n_cells * 8
+ contraction_peak = (
+ directed_csr
+ + canonical_csr
+ + contraction_layout
+ + contraction_thread_tables
+ + contraction_workspaces
+ )
+
+ fit_node_buffers = n_cells * (8 + 8 + 8 + 8 + 8 + index_bytes)
+ fit_peak = contraction_peak + hierarchy_bytes + fit_node_buffers
+ cut_peak = (
+ directed_csr
+ + canonical_csr
+ + hierarchy_bytes
+ + plateau_bytes
+ + modularity_guard_bytes
+ )
+ return int(_MEMORY_HEADROOM * max(load_peak + fit_peak, symmetrize_peak, cut_peak))
+
+
+def estimate_paris_adaptive_cut_peak_bytes(
+ n_cells: int,
+ edge_count: int,
+ edge_itemsize: int,
+ weight_itemsize: int,
+) -> int:
+ """Estimate peak bytes for a guarded cut over a cached hierarchy."""
+ (
+ _index_bytes,
+ load_peak,
+ symmetrize_peak,
+ directed_csr,
+ canonical_csr,
+ hierarchy_bytes,
+ plateau_bytes,
+ modularity_guard_bytes,
+ ) = _paris_memory_components(
+ n_cells,
+ edge_count,
+ edge_itemsize,
+ weight_itemsize,
+ )
+ cut_peak = (
+ directed_csr
+ + canonical_csr
+ + hierarchy_bytes
+ + plateau_bytes
+ + modularity_guard_bytes
+ )
+ return int(_MEMORY_HEADROOM * max(load_peak, symmetrize_peak, cut_peak))
+
+
+def _zarr_array_nbytes(group: zarr.Group, name: str) -> int:
+ values = as_zarr_array(group[name], name=name)
+ return prod(values.shape) * np.dtype(values.dtype).itemsize
+
+
+def estimate_cached_paris_peak_bytes(
+ root: zarr.Group,
+ graph_loc: str,
+ generation_id: str,
+ cut_mode: Literal["adaptive", "fixed"],
+) -> int:
+ """Estimate hierarchy loading plus fixed or cached-adaptive cut buffers."""
+ location = generation_location(graph_loc, generation_id)
+ generation = as_zarr_group(root[location], name=location)
+ return estimate_hierarchy_group_peak_bytes(generation, cut_mode)
+
+
+def estimate_hierarchy_group_peak_bytes(
+ generation: zarr.Group,
+ cut_mode: Literal["adaptive", "fixed"],
+) -> int:
+ """Estimate loading and cutting an independent hierarchy artifact."""
+ n_cells = cast(int, generation.attrs["n_leaves"])
+ plateau = as_zarr_group(generation["plateau"], name="plateau")
+ hierarchy_bytes = sum(
+ _zarr_array_nbytes(generation, name)
+ for name in (
+ "children",
+ "heights",
+ "sizes",
+ "component_roots",
+ "synthetic_joins",
+ )
+ )
+ plateau_bytes = sum(
+ _zarr_array_nbytes(plateau, name)
+ for name in (
+ "representatives",
+ "heights",
+ "sizes",
+ "parent_events",
+ "child_offsets",
+ "child_refs",
+ "min_leaves",
+ "component_roots",
+ )
+ )
+ if cut_mode == "fixed":
+ transient_bytes = n_cells * _CACHED_FIXED_TRANSIENT_BYTES_PER_CELL
+ elif cut_mode == "adaptive":
+ transient_bytes = n_cells * _CACHED_ADAPTIVE_TRANSIENT_BYTES_PER_CELL
+ else:
+ raise ValueError("cut_mode must be 'adaptive' or 'fixed'")
+ return int(_MEMORY_HEADROOM * (hierarchy_bytes + plateau_bytes + transient_bytes))
+
+
+def _raise_if_over_budget(
+ estimate: int,
+ budget: ResourceBudget,
+ operation: str,
+) -> None:
+ if estimate <= budget.memoryBytes:
+ return
+ required_gib = estimate / 1024**3
+ budget_gib = budget.memoryBytes / 1024**3
+ raise MemoryError(
+ f"{operation} is estimated to require "
+ f"{required_gib:.2f} GiB including headroom, but the active "
+ f"resource budget is {budget_gib:.2f} GiB"
+ )
+
+
+def preflight_paris_fit(
+ graph_group: zarr.Group,
+ n_cells: int,
+ budget: ResourceBudget,
+) -> int:
+ """Fail before loading graph arrays when the Paris estimate exceeds budget."""
+ edges = as_zarr_array(graph_group["edges"], name="edges")
+ weights = as_zarr_array(graph_group["weights"], name="weights")
+ edge_count = int(edges.shape[0])
+ estimate = estimate_paris_peak_bytes(
+ n_cells,
+ edge_count,
+ np.dtype(edges.dtype).itemsize,
+ np.dtype(weights.dtype).itemsize,
+ n_threads=budget.workers,
+ )
+ _raise_if_over_budget(estimate, budget, "Paris hierarchy fit")
+ return estimate
+
+
+def preflight_cached_paris_cut(
+ root: zarr.Group,
+ graph_loc: str,
+ generation_id: str,
+ cut_mode: Literal["adaptive", "fixed"],
+ budget: ResourceBudget,
+) -> int:
+ """Fail before loading a cached hierarchy and cut workspaces."""
+ estimate = estimate_cached_paris_peak_bytes(
+ root,
+ graph_loc,
+ generation_id,
+ cut_mode,
+ )
+ _raise_if_over_budget(estimate, budget, f"Cached Paris {cut_mode} cut")
+ return estimate
+
+
+def preflight_hierarchy_artifact_cut(
+ hierarchy_group: zarr.Group,
+ cut_mode: Literal["adaptive", "fixed"],
+ budget: ResourceBudget,
+) -> int:
+ estimate = estimate_hierarchy_group_peak_bytes(
+ hierarchy_group,
+ cut_mode,
+ )
+ _raise_if_over_budget(
+ estimate,
+ budget,
+ f"Cached Paris {cut_mode} cut",
+ )
+ return estimate
+
+
+def preflight_paris_adaptive_cut(
+ graph_group: zarr.Group,
+ n_cells: int,
+ budget: ResourceBudget,
+) -> int:
+ """Fail before loading graph arrays for a cached-hierarchy adaptive cut."""
+ edges = as_zarr_array(graph_group["edges"], name="edges")
+ weights = as_zarr_array(graph_group["weights"], name="weights")
+ estimate = estimate_paris_adaptive_cut_peak_bytes(
+ n_cells,
+ int(edges.shape[0]),
+ np.dtype(edges.dtype).itemsize,
+ np.dtype(weights.dtype).itemsize,
+ )
+ _raise_if_over_budget(estimate, budget, "Paris adaptive cut")
+ return estimate
+
+
+def _array_chunks(values: np.ndarray) -> tuple[int, ...]:
+ first = max(1, min(100_000, values.shape[0]))
+ return (first, *values.shape[1:])
+
+
+def _write_array(group: zarr.Group, name: str, values: np.ndarray) -> None:
+ target = create_zarr_dataset(
+ group,
+ name,
+ _array_chunks(values),
+ values.dtype,
+ values.shape,
+ )
+ target[:] = values
+
+
+def write_hierarchy_group(
+ generation: zarr.Group,
+ hierarchy: ParisHierarchy,
+ plateau_forest: PlateauForest,
+) -> None:
+ generation.attrs.update(
+ {
+ "n_leaves": hierarchy.n_leaves,
+ "total_weight": hierarchy.total_weight,
+ }
+ )
+ _write_array(generation, "children", hierarchy.children)
+ _write_array(generation, "heights", hierarchy.heights)
+ _write_array(generation, "sizes", hierarchy.sizes)
+ _write_array(generation, "component_roots", hierarchy.component_roots)
+ _write_array(generation, "synthetic_joins", hierarchy.synthetic_joins)
+
+ plateau = generation.create_group("plateau", overwrite=True)
+ _write_array(plateau, "representatives", plateau_forest.representatives)
+ _write_array(plateau, "heights", plateau_forest.heights)
+ _write_array(plateau, "sizes", plateau_forest.sizes)
+ _write_array(plateau, "parent_events", plateau_forest.parent_events)
+ _write_array(plateau, "child_offsets", plateau_forest.child_offsets)
+ _write_array(plateau, "child_refs", plateau_forest.child_refs)
+ _write_array(plateau, "min_leaves", plateau_forest.min_leaves)
+ _write_array(plateau, "component_roots", plateau_forest.component_roots)
+
+ if hierarchy.diagnostics is not None:
+ generation.attrs.update(
+ {
+ "preprocessing_seconds": hierarchy.diagnostics.preprocessing_seconds,
+ "component_seconds": hierarchy.diagnostics.component_seconds,
+ "fit_seconds": hierarchy.diagnostics.fit_seconds,
+ "reciprocal_rounds": len(hierarchy.diagnostics.rounds),
+ }
+ )
+
+
+def _read_array(group: zarr.Group, name: str) -> np.ndarray:
+ return np.asarray(as_zarr_array(group[name], name=name)[:])
+
+
+def load_hierarchy_generation(
+ root: zarr.Group,
+ graph_loc: str,
+ generation_id: str,
+) -> tuple[ParisHierarchy, PlateauForest]:
+ """Load and validate a completed hierarchy generation."""
+ location = generation_location(graph_loc, generation_id)
+ generation = as_zarr_group(root[location], name=location)
+ if generation.attrs.get("complete") is not True:
+ raise ValueError(f"Paris hierarchy generation {generation_id!r} is incomplete")
+ return load_hierarchy_group(generation, location)
+
+
+def load_hierarchy_group(
+ generation: zarr.Group,
+ label: str,
+) -> tuple[ParisHierarchy, PlateauForest]:
+ missing = [name for name in _PARIS_HIERARCHY_ARRAYS if name not in generation]
+ if missing or "plateau" not in generation:
+ raise ValueError(f"Paris hierarchy {label!r} is missing required arrays")
+ plateau_group = as_zarr_group(generation["plateau"], name=f"{label}/plateau")
+ missing_plateau = [
+ name for name in _PARIS_PLATEAU_ARRAYS if name not in plateau_group
+ ]
+ if missing_plateau:
+ raise ValueError(f"Paris hierarchy {label!r} is missing plateau arrays")
+ n_leaves = cast(int, generation.attrs["n_leaves"])
+ hierarchy = ParisHierarchy(
+ children=_read_array(generation, "children"),
+ heights=_read_array(generation, "heights"),
+ sizes=_read_array(generation, "sizes"),
+ component_roots=_read_array(generation, "component_roots"),
+ synthetic_joins=_read_array(generation, "synthetic_joins").astype(
+ bool,
+ copy=False,
+ ),
+ n_leaves=n_leaves,
+ total_weight=cast(float, generation.attrs["total_weight"]),
+ )
+ plateau_forest = PlateauForest(
+ representatives=_read_array(plateau_group, "representatives"),
+ heights=_read_array(plateau_group, "heights"),
+ sizes=_read_array(plateau_group, "sizes"),
+ parent_events=_read_array(plateau_group, "parent_events"),
+ child_offsets=_read_array(plateau_group, "child_offsets"),
+ child_refs=_read_array(plateau_group, "child_refs"),
+ min_leaves=_read_array(plateau_group, "min_leaves"),
+ component_roots=_read_array(plateau_group, "component_roots"),
+ n_leaves=n_leaves,
+ )
+ return hierarchy, plateau_forest
+
+
+def ensure_compatibility_dendrogram(
+ root: zarr.Group,
+ graph_loc: str,
+ generation_id: str,
+ hierarchy: ParisHierarchy,
+ *,
+ update_alias: bool = True,
+) -> str:
+ """Materialize the compatibility linkage and update its alias."""
+ from ...clustering.paris import hierarchy_to_dendrogram
+
+ location = generation_location(graph_loc, generation_id)
+ dendrogram_loc = f"{location}/dendrogram"
+ generation = as_zarr_group(root[location], name=location)
+ if (
+ dendrogram_loc not in root
+ or generation.attrs.get("dendrogram_complete") is not True
+ ):
+ dendrogram = hierarchy_to_dendrogram(hierarchy, compatibility=True)
+ target = create_zarr_dataset(
+ generation,
+ "dendrogram",
+ (min(5000, dendrogram.shape[0]), 4),
+ "f8",
+ dendrogram.shape,
+ )
+ target[:] = dendrogram
+ generation.attrs["dendrogram_complete"] = True
+ if update_alias:
+ graph_group = as_zarr_group(root[graph_loc], name=graph_loc)
+ graph_group.attrs["latest_dendrogram"] = dendrogram_loc
+ return dendrogram_loc
+
+
+def resolve_compatibility_dendrogram(
+ root: zarr.Group,
+ graph_loc: str,
+ budget: ResourceBudget,
+) -> tuple[str, str | None]:
+ """Resolve a linkage, falling back only for legacy stores."""
+ graph_group = as_zarr_group(root[graph_loc], name=graph_loc)
+ latest_value = graph_group.attrs.get(LATEST_PARIS_GENERATION)
+ generation_id = None if latest_value is None else str(latest_value)
+ # A stale generation pointer (for example an old paris_hierarchy/v2 layout)
+ # must fall through to the legacy alias or dendrogram rather than crash.
+ if (
+ generation_id is not None
+ and generation_location(graph_loc, generation_id) in root
+ ):
+ location = generation_location(graph_loc, generation_id)
+ dendrogram_loc = f"{location}/dendrogram"
+ generation = as_zarr_group(root[location], name=location)
+ if (
+ dendrogram_loc in root
+ and generation.attrs.get("dendrogram_complete") is True
+ ):
+ graph_group.attrs["latest_dendrogram"] = dendrogram_loc
+ return dendrogram_loc, generation_id
+ preflight_cached_paris_cut(
+ root,
+ graph_loc,
+ generation_id,
+ "fixed",
+ budget,
+ )
+ hierarchy, _plateau_forest = load_hierarchy_generation(
+ root,
+ graph_loc,
+ generation_id,
+ )
+ return (
+ ensure_compatibility_dendrogram(
+ root,
+ graph_loc,
+ generation_id,
+ hierarchy,
+ update_alias=True,
+ ),
+ generation_id,
+ )
+
+ alias = graph_group.attrs.get("latest_dendrogram")
+ if alias is not None and str(alias) in root:
+ return str(alias), None
+ legacy_location = f"{graph_loc}/dendrogram"
+ if legacy_location in root:
+ return legacy_location, None
+ raise KeyError(
+ "No Paris hierarchy is available for this graph. "
+ "Run run_paris_clustering first."
+ )
diff --git a/scarf/datastore/_operations/presentation.py b/scarf/datastore/_operations/presentation.py
new file mode 100644
index 00000000..7311756b
--- /dev/null
+++ b/scarf/datastore/_operations/presentation.py
@@ -0,0 +1,909 @@
+from collections.abc import Sequence
+from typing import TYPE_CHECKING, Any, Literal
+
+import numpy as np
+import pandas as pd
+import zarr
+from scipy.sparse import csr_matrix, vstack
+
+from ...storage.types import as_zarr_array, as_zarr_group
+from ...storage.arrays import create_zarr_dataset
+from ...storage.artifacts import (
+ ArtifactRef,
+ artifact_path,
+ fingerprint_stored_arrays,
+ inspect_artifact,
+ parse_artifact_path,
+)
+from ...storage.artifact_writer import (
+ ArrayRequirement,
+ finish_artifact,
+ plan_artifact,
+ reused_artifact_group,
+ start_artifact,
+)
+from ...graph.state import (
+ resolve_stored_graph_input,
+ validate_legacy_graph_selection,
+)
+from ...metadata.arguments import (
+ MembershipStrengthArguments,
+ SmartLabelArguments,
+)
+from ...metadata.artifacts import (
+ artifact_values,
+ categorical_display,
+ column_display,
+ continuous_display,
+ link_cell_data_column,
+ plan_cell_data_artifact,
+ write_cell_data_artifact,
+)
+from ...utils.logging import logger
+
+if TYPE_CHECKING:
+ from ..mapping_datastore import MappingDatastore as _PresentationOperationsBase
+else:
+ _PresentationOperationsBase = object
+
+
+class _PresentationOperationsMixin(_PresentationOperationsBase):
+ def to_anndata(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ layers: dict[str, str] | None = None,
+ *,
+ matrix: Literal["raw", "normed"] = "raw",
+ feature_indexes: Sequence[int] | None = None,
+ feature_names: Sequence[str] | None = None,
+ ) -> Any:
+ """Return an assay as an in-memory AnnData object.
+
+ Cell and feature metadata are copied to ``obs`` and ``var``. Layout
+ coordinates remain ordinary ``obs`` columns; this method does not
+ populate ``obsm``.
+
+ Args:
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ cell_key: Name of column from cell metadata that has boolean values. This is used to subset cells
+ layers: A mapping of layer names to assay names. Ex. {'spliced': 'RNA', 'unspliced': 'URNA'}. The raw data
+ from the assays will be stored as sparse arrays in the corresponding layer in anndata.
+ matrix: Whether ``X`` contains raw counts or normalized values.
+ feature_indexes: Global feature rows to export, in the requested order.
+ feature_names: Feature names to export, in the requested order.
+
+ Returns:
+ An AnnData object, or ``None`` when ``anndata`` is unavailable.
+ """
+ try:
+ # noinspection PyPackageRequirements
+ from anndata import AnnData # type: ignore
+ except ImportError:
+ logger.error(
+ "Package anndata is not installed because its an optional dependency. "
+ "Install via `pip install anndata` or `conda install anndata -c conda-forge`"
+ )
+ return None
+
+ if cell_key is None:
+ cell_key = "I"
+ assay = self._get_assay(from_assay)
+ if matrix not in ("raw", "normed"):
+ raise ValueError("matrix must be either 'raw' or 'normed'")
+ if feature_indexes is not None and feature_names is not None:
+ raise ValueError("feature_indexes and feature_names are mutually exclusive")
+
+ if feature_indexes is not None:
+ if isinstance(feature_indexes, str):
+ raise TypeError(
+ "feature_indexes must be a sequence of integer feature indexes"
+ )
+ feat_idx = np.asarray(feature_indexes)
+ if feat_idx.ndim != 1:
+ raise ValueError("feature_indexes must be one-dimensional")
+ if feat_idx.size == 0:
+ feat_idx = np.empty(0, dtype=np.int64)
+ elif not np.issubdtype(feat_idx.dtype, np.integer):
+ raise TypeError("feature_indexes must contain only integers")
+ else:
+ feat_idx = feat_idx.astype(np.int64, copy=False)
+ if np.unique(feat_idx).size != feat_idx.size:
+ raise ValueError("feature_indexes must contain unique indexes")
+ if np.any(feat_idx < 0) or np.any(feat_idx >= assay.feats.N):
+ raise IndexError("feature_indexes contains an out-of-range index")
+ elif feature_names is not None:
+ if isinstance(feature_names, str):
+ raise TypeError(
+ "feature_names must be a sequence of feature names, not a string"
+ )
+ requested_names = list(feature_names)
+ if not all(isinstance(name, str) for name in requested_names):
+ raise TypeError("feature_names must contain only strings")
+ if len(set(requested_names)) != len(requested_names):
+ raise ValueError("feature_names must contain unique names")
+ name_positions: dict[str, list[int]] = {}
+ for index, name in enumerate(assay.feats.fetch_all("names").astype(str)):
+ name_positions.setdefault(name, []).append(index)
+ missing = [name for name in requested_names if name not in name_positions]
+ if missing:
+ raise KeyError("Feature names not found: " + ", ".join(missing))
+ ambiguous = [
+ name for name in requested_names if len(name_positions[name]) != 1
+ ]
+ if ambiguous:
+ raise ValueError(
+ "Feature names are not unique in the assay: " + ", ".join(ambiguous)
+ )
+ feat_idx = np.asarray(
+ [name_positions[name][0] for name in requested_names],
+ dtype=np.int64,
+ )
+ else:
+ feat_idx = np.arange(assay.feats.N, dtype=np.int64)
+
+ cell_idx = self.cells.active_index(cell_key)
+ df = self.cells.to_pandas_dataframe(self.cells.columns, key=cell_key)
+ obs = df.reset_index(drop=True).set_index("ids")
+ df = assay.feats.to_pandas_dataframe(assay.feats.columns).iloc[feat_idx]
+ var = df.rename(columns={"ids": "gene_ids"}).set_index("gene_ids")
+ if matrix == "raw":
+ x = assay.to_raw_sparse(cell_key)[:, feat_idx].tocsr()
+ else:
+ normed = assay.normed(cell_idx=cell_idx, feat_idx=feat_idx)
+ blocks = [csr_matrix(block) for block in normed.stream_blocks()]
+ x = (
+ vstack(blocks, format="csr")
+ if blocks
+ else csr_matrix((len(cell_idx), len(feat_idx)))
+ )
+ adata = AnnData(x, obs=obs, var=var)
+ if layers is not None:
+ selected_ids = assay.feats.fetch_all("ids").astype(str)[feat_idx]
+ if np.unique(selected_ids).size != selected_ids.size:
+ raise ValueError(
+ "Selected feature IDs must be unique when exporting layers"
+ )
+ for layer, assay_name in layers.items():
+ layer_assay = self._get_assay(assay_name)
+ layer_matrix = layer_assay.to_raw_sparse(cell_key)
+ layer_id_positions: dict[str, list[int]] = {}
+ for index, feature_id in enumerate(
+ layer_assay.feats.fetch_all("ids").astype(str)
+ ):
+ layer_id_positions.setdefault(feature_id, []).append(index)
+ missing_ids = [
+ feature_id
+ for feature_id in selected_ids
+ if feature_id not in layer_id_positions
+ ]
+ ambiguous_ids = [
+ feature_id
+ for feature_id in selected_ids
+ if len(layer_id_positions.get(feature_id, ())) > 1
+ ]
+ if missing_ids or ambiguous_ids:
+ details = []
+ if missing_ids:
+ details.append("missing: " + ", ".join(missing_ids))
+ if ambiguous_ids:
+ details.append("ambiguous: " + ", ".join(ambiguous_ids))
+ raise ValueError(
+ f"Layer {layer!r} cannot align selected feature IDs ("
+ + "; ".join(details)
+ + ")"
+ )
+ layer_feat_idx = np.asarray(
+ [layer_id_positions[feature_id][0] for feature_id in selected_ids],
+ dtype=np.int64,
+ )
+ adata.layers[layer] = layer_matrix[:, layer_feat_idx].tocsr()
+ return adata
+
+ def show_zarr_tree(self, start: str = "/", depth: int = 2) -> None:
+ """Prints the Zarr hierarchy of the DataStore.
+
+ Args:
+ start: Location in Zarr hierarchy to be used as the root for display
+ depth: Depth of Zarr hierarchy to be displayed.
+
+ Returns:
+ None
+ """
+ from ...storage.layout import array_info
+
+ root = start.strip("/")
+ node: zarr.Group = (
+ self.zw if root == "" else as_zarr_group(self.zw[root], name=root)
+ )
+ print(node.tree(level=depth))
+ for key in node.array_keys():
+ print(f" {key}: {array_info(as_zarr_array(node[key], name=key))}")
+
+ def calc_membership_strength(
+ self,
+ from_assay: str,
+ cell_key: str,
+ feat_key: str,
+ clust_key: str,
+ invalidate_cache: bool = False,
+ ) -> None:
+ """Store per-cell cluster membership strength from the latest KNN graph.
+
+ For each cell, computes the fraction of KNN neighbors sharing the most
+ common cluster label and saves it in cell metadata.
+
+ Args:
+ from_assay: Assay used to locate the KNN graph.
+ cell_key: Boolean column selecting cells.
+ feat_key: Feature key used when the graph was built.
+ clust_key: Cell metadata column with cluster assignments.
+
+ Returns:
+ None
+ """
+ loc = self.get_latest_graph_loc(
+ from_assay=from_assay, cell_key=cell_key, feat_key=feat_key
+ )
+ n_cells, k = self._get_graph_ncells_k(graph_loc=loc)
+ selection = self._ensure_cell_selection(cell_key)
+ graph_input: object = resolve_stored_graph_input(self.zw, loc)
+ if isinstance(graph_input, ArtifactRef):
+ graph_selection = self._graph_cell_selection(graph_input)
+ if not self._selection_artifacts_match(graph_selection, selection):
+ raise ValueError("cell_key does not match the graph cell selection")
+ else:
+ validate_legacy_graph_selection(
+ self,
+ loc,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ cluster_input = self._resolve_cell_data_provenance_input(
+ clust_key,
+ cell_key=cell_key,
+ )
+ output_key = f"{from_assay}_{cell_key}_cluster_membership_strength"
+ arguments = MembershipStrengthArguments(
+ connectivity_map=graph_input,
+ clusters=cluster_input,
+ cell_selection=selection,
+ algorithm_version=2,
+ decimals=3,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ clust_key=clust_key,
+ output_key=output_key,
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope="assay",
+ assay=from_assay,
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ cell_selection=selection,
+ arrays={"values": ((n_cells,), "f")},
+ invalidate_cache=invalidate_cache,
+ )
+ preserved_display = column_display(self.zw, output_key)
+ if planned.reused:
+ artifact_group = as_zarr_group(
+ self.zw[artifact_path(planned.ref)],
+ name=planned.ref.artifact_id,
+ )
+ values = artifact_values(artifact_group, "values")
+ self.cells.insert(
+ output_key,
+ values,
+ key=cell_key,
+ overwrite=True,
+ )
+ link_cell_data_column(
+ self.zw,
+ output_key,
+ planned.ref,
+ value_name="values",
+ default_display={
+ **continuous_display(values),
+ "minimum": 0.0,
+ "maximum": 1.0,
+ },
+ preserved_display=preserved_display,
+ )
+ return None
+ clusts = self.cells.fetch(clust_key, key=cell_key)
+ graph_grp = as_zarr_group(self.zw[loc], name=loc)
+ edges = np.asarray(as_zarr_array(graph_grp["edges"], name="edges")[:])
+ if edges.shape != (n_cells * k, 2):
+ raise ValueError(
+ "Graph edges do not match the stored cell and k dimensions"
+ )
+ edge_rows = edges.reshape(n_cells, k, 2)
+ expected_sources = np.broadcast_to(
+ np.arange(n_cells, dtype=edge_rows.dtype)[:, None],
+ (n_cells, k),
+ )
+ if not np.array_equal(edge_rows[:, :, 0], expected_sources):
+ raise ValueError("Graph edges are not stored in cell-major order")
+ neighbor_clusters = np.asarray(clusts)[edge_rows[:, :, 1]]
+ values = np.asarray(
+ [
+ pd.Series(row).value_counts(dropna=False).iloc[0] / k
+ for row in neighbor_clusters
+ ],
+ dtype=np.float64,
+ ).round(3)
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {"values": values},
+ )
+ self.cells.insert(
+ output_key,
+ values,
+ key=cell_key,
+ overwrite=True,
+ )
+ link_cell_data_column(
+ self.zw,
+ output_key,
+ planned.ref,
+ value_name="values",
+ default_display={
+ **continuous_display(values),
+ "minimum": 0.0,
+ "maximum": 1.0,
+ },
+ preserved_display=preserved_display,
+ )
+ return None
+
+ def smart_label(
+ self,
+ to_relabel: str,
+ base_label: str,
+ cell_key: str = "I",
+ new_col_name: str | None = None,
+ invalidate_cache: bool = False,
+ ) -> None | list[str]:
+ """A convenience function to relabel the values in a cell attribute
+ column (A) based on the values in another cell attribute column (B).
+ For each unique value in A, the most frequently occurring value in B is
+ found. If two or more values in A have maximum overlap with the same
+ value in B, then they all get the same label as B along with different
+ suffixes like, 'a', 'b', etc. The suffixes are ordered based on where
+ the largest fraction of the B label lies. If one label from A takes up
+ multiple labels from B then all the labels from B are included, and they
+ are delimited by hyphens.
+
+ Args:
+ to_relabel: Cell attributes column to relabel
+ base_label: Cell attributes column to relabel
+ cell_key: Cell key fetching column values
+ new_col_name: Name of new column where relabeled values will be saved. If None then values
+ are returned and not saved in cell attributes table
+
+ Returns: None or a list of relabelled values
+ """
+ values_to_relabel = np.asarray(self.cells.fetch(to_relabel, key=cell_key))
+ base_values = np.asarray(self.cells.fetch(base_label, key=cell_key))
+ if len(values_to_relabel) == 0:
+ if new_col_name is None:
+ return []
+ raise ValueError(f"cell_key {cell_key!r} selects no cells")
+ planned = None
+ preserved_display = None
+ if new_col_name is not None:
+ selection = self._ensure_cell_selection(cell_key)
+ arguments = SmartLabelArguments(
+ values=self._resolve_cell_data_provenance_input(
+ to_relabel,
+ cell_key=cell_key,
+ ),
+ base_labels=self._resolve_cell_data_provenance_input(
+ base_label,
+ cell_key=cell_key,
+ ),
+ cell_selection=selection,
+ algorithm_version=2,
+ suffix_style="lowercase_letter",
+ to_relabel=to_relabel,
+ base_label=base_label,
+ cell_key=cell_key,
+ new_col_name=new_col_name,
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope="datastore",
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ cell_selection=selection,
+ arrays={"values": ((len(self.cells.active_index(cell_key)),), None)},
+ invalidate_cache=invalidate_cache,
+ )
+ preserved_display = column_display(self.zw, new_col_name)
+ if planned.reused:
+ artifact_group = as_zarr_group(
+ self.zw[artifact_path(planned.ref)],
+ name=planned.ref.artifact_id,
+ )
+ values = artifact_values(artifact_group, "values")
+ self.cells.insert(
+ new_col_name,
+ values,
+ key=cell_key,
+ overwrite=True,
+ )
+ link_cell_data_column(
+ self.zw,
+ new_col_name,
+ planned.ref,
+ value_name="values",
+ default_display=categorical_display(values),
+ preserved_display=preserved_display,
+ )
+ return None
+
+ df = pd.crosstab(
+ base_values,
+ values_to_relabel,
+ )
+ normed_frac = df.divide(df.sum(axis=1), axis="index")
+ idxmax = df.idxmax()
+ new_names = {}
+ for i in sorted(idxmax.unique()):
+ j = normed_frac[idxmax[idxmax == i].index].loc[i]
+ j = j.sort_values(ascending=False).index
+ for n, k in enumerate(j, start=1):
+ a = chr(ord("@") + n)
+ new_names[k] = f"{i}{a.lower()}"
+
+ missing_vals = df.index.difference(
+ pd.Index(idxmax.unique()),
+ sort=False,
+ ).tolist()
+ if len(missing_vals) > 0:
+ miss_idxmax = df.loc[missing_vals].idxmax(axis=1).to_dict()
+ for k, v in miss_idxmax.items():
+ new_names[v] = f"{new_names[v][:-1]}-{k}{new_names[v][-1]}"
+
+ ret_val = [new_names[x] for x in values_to_relabel]
+ if new_col_name is None:
+ return ret_val
+ assert planned is not None
+ values = np.asarray(ret_val)
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {"values": values},
+ )
+ self.cells.insert(
+ new_col_name,
+ values,
+ key=cell_key,
+ overwrite=True,
+ )
+ link_cell_data_column(
+ self.zw,
+ new_col_name,
+ planned.ref,
+ value_name="values",
+ default_display=categorical_display(values),
+ preserved_display=preserved_display,
+ )
+ return None
+
+ def _prepare_artifact_cluster_tree(
+ self,
+ *,
+ graph_ref: ArtifactRef | dict[str, Any],
+ graph_loc: str,
+ from_assay: str,
+ cell_key: str,
+ feat_key: str,
+ integrated_graph: str | None,
+ cluster_key: str,
+ fill_by_value: str | None,
+ invalidate_cache: bool,
+ ) -> dict[str, Any]:
+ from networkx import DiGraph, to_pandas_edgelist
+
+ from ...clustering.cluster_tree import CoalesceTree, make_digraph
+ from ...clustering.paris import hierarchy_to_dendrogram
+ from .paris_persistence import load_hierarchy_group
+
+ cell_data = as_zarr_group(self.zw["cellData"], name="cellData")
+ cluster_column = as_zarr_array(cell_data[cluster_key], name=cluster_key)
+ raw_cut_ref = cluster_column.attrs.get("source_artifact")
+ if not isinstance(raw_cut_ref, dict):
+ raise ValueError("Cluster column has no source artifact")
+ cut_ref = ArtifactRef.from_dict(raw_cut_ref)
+ cut_inputs = inspect_artifact(self.zw, cut_ref).inputs or {}
+ raw_graph_ref = cut_inputs.get("connectivity_map")
+ expected_graph_input = (
+ graph_ref.to_dict() if isinstance(graph_ref, ArtifactRef) else graph_ref
+ )
+ if raw_graph_ref != expected_graph_input:
+ raise ValueError("Cluster cut does not belong to the requested graph")
+ raw_hierarchy_ref = cut_inputs.get("cluster_hierarchy")
+ if not isinstance(raw_hierarchy_ref, dict):
+ raise ValueError("Cluster cut has no hierarchy input")
+ hierarchy_ref = ArtifactRef.from_dict(raw_hierarchy_ref)
+ hierarchy_group = as_zarr_group(
+ self.zw[inspect_artifact(self.zw, hierarchy_ref).path],
+ name=hierarchy_ref.artifact_id,
+ )
+ hierarchy, _plateau = load_hierarchy_group(
+ hierarchy_group,
+ hierarchy_ref.artifact_id,
+ )
+ dendrogram_plan = plan_artifact(
+ self.zw,
+ scope=hierarchy_ref.scope,
+ assay=hierarchy_ref.assay,
+ kind="dendrogram",
+ operation="materialize_paris_dendrogram",
+ parameters={"compatibility": True},
+ inputs={"cluster_hierarchy": hierarchy_ref},
+ execution_options={},
+ invalidate_cache=invalidate_cache,
+ required_arrays=(ArrayRequirement("data", dtype_kind="f"),),
+ )
+ if dendrogram_plan.reused:
+ dendrogram_group = reused_artifact_group(self.zw, dendrogram_plan)
+ else:
+ dendrogram = hierarchy_to_dendrogram(hierarchy, compatibility=True)
+ dendrogram_group = start_artifact(self.zw, dendrogram_plan)
+ output = create_zarr_dataset(
+ dendrogram_group,
+ "data",
+ (min(max(dendrogram.shape[0], 1), 5000), 4),
+ "f8",
+ dendrogram.shape,
+ )
+ output[:] = dendrogram
+ finish_artifact(dendrogram_group, dendrogram_plan)
+ dendrogram = np.asarray(as_zarr_array(dendrogram_group["data"], name="data")[:])
+ cut_group = as_zarr_group(
+ self.zw[artifact_path(cut_ref)],
+ name=artifact_path(cut_ref),
+ )
+ clusters = np.asarray(as_zarr_array(cut_group["labels"], name="labels")[:])
+ coalesced_plan = plan_artifact(
+ self.zw,
+ scope=cut_ref.scope,
+ assay=cut_ref.assay,
+ kind="coalesced_tree",
+ operation="coalesce_cluster_tree",
+ parameters={},
+ inputs={
+ "dendrogram": dendrogram_plan.ref,
+ "cluster_cut": cut_ref,
+ },
+ execution_options={"cluster_key": cluster_key},
+ invalidate_cache=invalidate_cache,
+ required_arrays=(
+ ArrayRequirement("edgelist"),
+ ArrayRequirement("nodelist"),
+ ArrayRequirement("partition_id"),
+ ),
+ )
+ if coalesced_plan.reused:
+ coalesced_group = reused_artifact_group(self.zw, coalesced_plan)
+ subgraph = DiGraph()
+ subgraph.add_edges_from(
+ np.asarray(
+ as_zarr_array(
+ coalesced_group["edgelist"],
+ name="edgelist",
+ )[:]
+ )
+ )
+ nodelist = np.asarray(
+ as_zarr_array(coalesced_group["nodelist"], name="nodelist")[:]
+ )
+ partition_ids = np.asarray(
+ as_zarr_array(
+ coalesced_group["partition_id"],
+ name="partition_id",
+ )[:]
+ )
+ cluster_labels = {str(value): value for value in set(clusters)}
+ for node_data, partition_id in zip(
+ nodelist,
+ partition_ids,
+ strict=True,
+ ):
+ node = int(node_data[0])
+ subgraph.nodes[node]["nleaves"] = int(node_data[1])
+ if str(partition_id) != "-1":
+ subgraph.nodes[node]["partition_id"] = cluster_labels.get(
+ str(partition_id),
+ partition_id,
+ )
+ else:
+ subgraph = CoalesceTree(make_digraph(dendrogram), clusters)
+ edge_list = to_pandas_edgelist(subgraph).values
+ coalesced_group = start_artifact(self.zw, coalesced_plan)
+ edge_array = create_zarr_dataset(
+ coalesced_group,
+ "edgelist",
+ (100000,),
+ "u8",
+ edge_list.shape,
+ )
+ edge_array[:] = edge_list
+ node_list = []
+ partition_id_values = []
+ for node in subgraph.nodes():
+ node_data = subgraph.nodes[node]
+ node_list.append((node, node_data["nleaves"]))
+ partition_id_values.append(str(node_data.get("partition_id", -1)))
+ node_values = np.asarray(node_list)
+ node_array = create_zarr_dataset(
+ coalesced_group,
+ "nodelist",
+ (100000,),
+ node_values.dtype,
+ node_values.shape,
+ )
+ node_array[:] = node_values
+ partition_array = create_zarr_dataset(
+ coalesced_group,
+ "partition_id",
+ (100000,),
+ str,
+ (len(partition_id_values),),
+ )
+ partition_array[:] = partition_id_values
+ finish_artifact(coalesced_group, coalesced_plan)
+ color_values = (
+ self.get_cell_vals(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ k=fill_by_value,
+ )
+ if fill_by_value is not None
+ else None
+ )
+ return {
+ "graph": subgraph,
+ "clusters": clusters,
+ "color_values": color_values,
+ "from_assay": from_assay,
+ "cell_key": cell_key,
+ "feat_key": feat_key,
+ "integrated_graph": integrated_graph,
+ "cluster_key": cluster_key,
+ "coalesced_location": inspect_artifact(
+ self.zw,
+ coalesced_plan.ref,
+ ).path,
+ }
+
+ def _prepare_cluster_tree(
+ self,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ integrated_graph: str | None = None,
+ cluster_key: str | None = None,
+ fill_by_value: str | None = None,
+ invalidate_cache: bool = False,
+ ) -> dict[str, Any]:
+ from networkx import DiGraph, to_pandas_edgelist
+
+ from ...clustering.cluster_tree import CoalesceTree, make_digraph
+ from ...utils.arrays import array_digest
+ from .paris_persistence import resolve_compatibility_dendrogram
+
+ from_assay, cell_key, feat_key = self._get_latest_keys(
+ from_assay, cell_key, feat_key
+ )
+ if cluster_key is None:
+ raise ValueError(
+ "ERROR: Please provide a value for `cluster_key` parameter"
+ )
+
+ if integrated_graph is None:
+ graph_loc = self.get_latest_graph_loc(from_assay, cell_key, feat_key)
+ else:
+ graph_loc = self._resolve_integrated_graph_path(integrated_graph)
+ if graph_loc is None:
+ raise KeyError(
+ f"An integrated graph with label {integrated_graph!r} does not exist"
+ )
+ cell_data = as_zarr_group(self.zw["cellData"], name="cellData")
+ cluster_column = as_zarr_array(cell_data[cluster_key], name=cluster_key)
+ raw_cut_ref = cluster_column.attrs.get("source_artifact")
+ if integrated_graph is None and isinstance(raw_cut_ref, dict):
+ cut_ref = ArtifactRef.from_dict(raw_cut_ref)
+ if cut_ref.kind == "cluster_cut":
+ cut_inputs = inspect_artifact(self.zw, cut_ref).inputs or {}
+ raw_graph_ref = cut_inputs.get("connectivity_map")
+ if isinstance(raw_graph_ref, dict):
+ try:
+ selected_graph_ref = ArtifactRef.from_dict(raw_graph_ref)
+ except (KeyError, TypeError, ValueError):
+ pass
+ else:
+ graph_loc = artifact_path(selected_graph_ref)
+ try:
+ graph_ref: ArtifactRef | dict[str, str] | None = parse_artifact_path(
+ graph_loc
+ )
+ except ValueError:
+ if isinstance(raw_cut_ref, dict):
+ graph_group = as_zarr_group(self.zw[graph_loc], name=graph_loc)
+ graph_ref = {
+ "legacy_graph_fingerprint": fingerprint_stored_arrays(
+ graph_group,
+ ("edges", "weights"),
+ )
+ }
+ else:
+ graph_ref = None
+ if isinstance(raw_cut_ref, dict):
+ assert graph_ref is not None
+ return self._prepare_artifact_cluster_tree(
+ graph_ref=graph_ref,
+ graph_loc=graph_loc,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ integrated_graph=integrated_graph,
+ cluster_key=cluster_key,
+ fill_by_value=fill_by_value,
+ invalidate_cache=invalidate_cache,
+ )
+ if isinstance(graph_ref, ArtifactRef):
+ raise ValueError("Cluster column has no source artifact for this graph")
+ clusters = np.asarray(self.cells.fetch(cluster_key, key=cell_key))
+ dendrogram_loc, generation_id = resolve_compatibility_dendrogram(
+ self.zw,
+ graph_loc,
+ self.resources,
+ )
+ if clusters.dtype.hasobject:
+ hashed_clusters = pd.util.hash_pandas_object(
+ pd.Series(clusters),
+ index=False,
+ categorize=True,
+ ).to_numpy(dtype=np.uint64)
+ cluster_digest = array_digest(hashed_clusters)
+ else:
+ cluster_digest = array_digest(clusters)
+ coalesced_loc = f"{dendrogram_loc}_coalesced_{cluster_digest}"
+ cache_hit = False
+ if coalesced_loc in self.zw:
+ coalesced_group = as_zarr_group(
+ self.zw[coalesced_loc],
+ name=coalesced_loc,
+ )
+ cache_hit = (
+ coalesced_group.attrs.get("complete") is True
+ and coalesced_group.attrs.get("cluster_digest") == cluster_digest
+ and coalesced_group.attrs.get("hierarchy_generation_id")
+ == (generation_id or "legacy")
+ )
+
+ if cache_hit:
+ subgraph = DiGraph()
+ subgraph.add_edges_from(
+ np.asarray(
+ as_zarr_array(
+ self.zw[coalesced_loc + "/edgelist"],
+ name=f"{coalesced_loc}/edgelist",
+ )[:]
+ )
+ )
+ nodelist = np.asarray(
+ as_zarr_array(
+ self.zw[coalesced_loc + "/nodelist"],
+ name=f"{coalesced_loc}/nodelist",
+ )[:]
+ )
+ partition_ids = np.asarray(
+ as_zarr_array(
+ self.zw[coalesced_loc + "/partition_id"],
+ name=f"{coalesced_loc}/partition_id",
+ )[:]
+ )
+ cluster_labels = {str(value): value for value in set(clusters)}
+ for node_data, partition_id in zip(
+ nodelist,
+ partition_ids,
+ strict=True,
+ ):
+ node = int(node_data[0])
+ subgraph.nodes[node]["nleaves"] = int(node_data[1])
+ partition_text = str(partition_id)
+ if partition_text != "-1":
+ subgraph.nodes[node]["partition_id"] = cluster_labels.get(
+ partition_text, partition_id
+ )
+ else:
+ dendrogram = np.asarray(
+ as_zarr_array(self.zw[dendrogram_loc], name=dendrogram_loc)[:]
+ )
+ subgraph = CoalesceTree(make_digraph(dendrogram), clusters)
+ edge_list = to_pandas_edgelist(subgraph).values
+ coalesced_group = self.zw.create_group(coalesced_loc, overwrite=True)
+ coalesced_group.attrs.update(
+ {
+ "complete": False,
+ "cluster_digest": cluster_digest,
+ "hierarchy_generation_id": generation_id or "legacy",
+ "cluster_key": cluster_key,
+ }
+ )
+ store = create_zarr_dataset(
+ coalesced_group,
+ "edgelist",
+ (100000,),
+ "u8",
+ edge_list.shape,
+ )
+ store[:] = edge_list
+
+ node_list = []
+ partition_id_list = []
+ for node in subgraph.nodes():
+ node_data = subgraph.nodes[node]
+ partition_id = node_data.get("partition_id", -1)
+ node_list.append((node, node_data["nleaves"]))
+ partition_id_list.append(str(partition_id))
+
+ node_list_arr = np.asarray(node_list)
+ store = create_zarr_dataset(
+ coalesced_group,
+ "nodelist",
+ (100000,),
+ node_list_arr.dtype,
+ node_list_arr.shape,
+ )
+ store[:] = node_list_arr
+
+ store = create_zarr_dataset(
+ coalesced_group,
+ "partition_id",
+ (100000,),
+ str,
+ (len(partition_id_list),),
+ )
+ store[:] = partition_id_list
+ coalesced_group.attrs["complete"] = True
+
+ color_values = (
+ self.get_cell_vals(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ k=fill_by_value,
+ )
+ if fill_by_value is not None
+ else None
+ )
+ return {
+ "graph": subgraph,
+ "clusters": clusters,
+ "color_values": color_values,
+ "from_assay": from_assay,
+ "cell_key": cell_key,
+ "feat_key": feat_key,
+ "integrated_graph": integrated_graph,
+ "cluster_key": cluster_key,
+ "coalesced_location": coalesced_loc,
+ }
diff --git a/scarf/datastore/_operations/quality_control.py b/scarf/datastore/_operations/quality_control.py
new file mode 100644
index 00000000..406a8f13
--- /dev/null
+++ b/scarf/datastore/_operations/quality_control.py
@@ -0,0 +1,1159 @@
+from collections.abc import Iterable
+from numbers import Real
+from typing import TYPE_CHECKING, Any, cast
+
+import numpy as np
+import pandas as pd
+
+from ...assay import ATACassay, RNAassay
+from ...graph.state import read_assay_state, validate_artifact_graph_selection
+from ...quality_control.cell_cycle import assign_cell_cycle_phase
+from ...quality_control.filtering import (
+ _metric_policy,
+ _sample_aware_mad_mask,
+ _validated_sample_labels,
+ _validated_work_scale,
+ gaussian_quantile_bounds,
+)
+from ...quality_control.hto import _hto_demux_method, hto_demux
+from ...metadata.artifacts import (
+ artifact_values,
+ categorical_display,
+ column_display,
+ continuous_display,
+ feature_column_display,
+ link_cell_data_column,
+ link_feature_data_column,
+ plan_cell_data_artifact,
+ write_cell_data_artifact,
+)
+from ...metadata.arguments import (
+ CellCycleArguments,
+ DoubletScoreArguments,
+ HtoIdentityArguments,
+ PrevalentPeakArguments,
+)
+from ...storage.artifacts import (
+ artifact_path,
+ canonical_bytes,
+ callable_identity,
+ fingerprint_array,
+ fingerprint_strings,
+)
+from ...storage.refs import ArtifactRef
+from ...storage.selections import resolve_generated_selection_artifact
+from ...storage.types import as_zarr_group
+from ...utils.compute import controlled_compute
+from ...utils.logging import logger
+
+if TYPE_CHECKING:
+ from ...storage.profiles import ZarrLocation
+ from ..mapping_datastore import MappingDatastore as _QualityControlOperationsBase
+else:
+ _QualityControlOperationsBase = object
+
+
+class _QualityControlOperationsMixin(_QualityControlOperationsBase):
+ if TYPE_CHECKING:
+
+ def _create_temporary_datastore(
+ self,
+ zarr_loc: ZarrLocation,
+ *,
+ default_assay: str,
+ assay_types: dict[str, str],
+ nthreads: int,
+ ) -> _QualityControlOperationsBase: ...
+
+ def filter_cells(
+ self,
+ attrs: Iterable[str],
+ lows: Iterable[int],
+ highs: Iterable[int],
+ reset_previous: bool = False,
+ keep_bounds: bool = False,
+ invalidate_cache: bool = False,
+ ) -> None:
+ """Filter cells based on the cell metadata column values. Filtering
+ triggers `update` method on 'I' column of cell metadata which uses
+ 'and' operation. This means that cells that are not within the
+ filtering thresholds will have value set as False in 'I' column of cell
+ metadata table. When performing filtering repeatedly, the cells that
+ were previously filtered out remain filtered out and 'I' column is
+ updated only for those cells that are filtered out due to the latest
+ filtering attempt.
+
+ Args:
+ attrs: Names of columns to be used for filtering
+ lows: Lower bounds of thresholds for filtering. Should be in same order as the names in `attrs` parameter
+ highs: Upper bounds of thresholds for filtering. Should be in same order as the names in `attrs` parameter
+ reset_previous: If True, then results of previous filtering will be undone completely.
+ (Default value: False)
+ keep_bounds: If True, then the boundary values are retained and not filtered out (Default value: False)
+
+ Returns:
+ """
+ attrs = list(attrs)
+ lows = list(lows)
+ highs = list(highs)
+ input_fingerprints = {}
+ new_bool = np.ones(self.cells.N).astype(bool)
+ for i, j, k in zip(attrs, lows, highs, strict=False):
+ # Checking here to avoid hard error from metadata class
+ if i not in self.cells.columns:
+ logger.warning(
+ f"{i} not found in cell metadata. Will ignore {i} for filtering"
+ )
+ continue
+ if j is None:
+ j = -np.inf
+ if k is None:
+ k = np.inf
+ x = self.cells.sift(i, j, k, keep_bounds=keep_bounds)
+ values = np.asarray(self.cells.fetch_all(i))
+ input_fingerprints[i] = (
+ fingerprint_strings(values)
+ if values.dtype.kind in {"O", "S", "U"}
+ else fingerprint_array(values)
+ )
+ new_bool = new_bool & x
+ if reset_previous:
+ self.cells.reset_key(key="I")
+ self.cells.update_key(new_bool, key="I")
+ self._record_cell_selection(
+ column="I",
+ operation="filter_cells",
+ parameters={
+ "attrs": attrs,
+ "lows": lows,
+ "highs": highs,
+ "reset_previous": reset_previous,
+ "keep_bounds": keep_bounds,
+ },
+ inputs={"metadata_fingerprints": input_fingerprints},
+ invalidate_cache=invalidate_cache,
+ )
+ remaining = int(np.asarray(self.cells.fetch_all("I"), dtype=bool).sum())
+ logger.info(f"Cell filtering retained {remaining}/{self.cells.N} cells")
+
+ def auto_filter_cells(
+ self,
+ attrs: Iterable[str] | None = None,
+ min_p: float = 0.01,
+ max_p: float = 0.99,
+ show_qc_plots: bool = True,
+ invalidate_cache: bool = False,
+ sample_column: str | None = None,
+ n_mads: float = 3.0,
+ min_cells_per_sample: int = 20,
+ ) -> None:
+ """Automatically filter cells based on columns of the cell metadata
+ table.
+
+ By default this is a wrapper around ``filter_cells`` that determines the
+ thresholds for each column. It models a normal distribution centered on
+ the column median and using the column standard deviation, then
+ evaluates its quantiles at ``min_p`` and ``max_p``.
+
+ When ``sample_column`` is supplied, thresholds are instead calculated
+ independently within each sample using median absolute deviation (MAD).
+ ``n_mads`` controls that path. ``min_p`` and ``max_p`` remain
+ global-Gaussian parameters and must stay at their defaults when
+ ``sample_column`` is set.
+
+ Args:
+ attrs: Column names to be used for filtering.
+ min_p: Quantile used for the lower threshold (Gaussian path only).
+ max_p: Quantile used for the upper threshold (Gaussian path only).
+ show_qc_plots: Show pre-filtering and post-filtering distributions
+ for the columns used.
+ sample_column: Optional cell-metadata column with sample labels.
+ When set, MAD bounds are calculated within each sample.
+ n_mads: Number of scaled MADs used for per-sample bounds.
+ min_cells_per_sample: Samples with fewer active cells than this are
+ retained without MAD filtering and emit a warning.
+
+ Returns:
+ None
+ """
+ from ...plotting import distribution
+
+ if attrs is None:
+ attrs = []
+ for i in ["nCounts", "nFeatures", "percentMito", "percentRibo"]:
+ i = f"{self._defaultAssay}_{i}"
+ if i in self.cells.columns:
+ attrs.append(i)
+
+ attrs_list = list(attrs)
+ if sample_column is not None:
+ self._auto_filter_cells_sample_mad(
+ attrs=attrs_list,
+ min_p=min_p,
+ max_p=max_p,
+ show_qc_plots=show_qc_plots,
+ invalidate_cache=invalidate_cache,
+ sample_column=sample_column,
+ n_mads=n_mads,
+ min_cells_per_sample=min_cells_per_sample,
+ )
+ return
+
+ attrs_used: list[str] = []
+ lower_bounds: list[int] = []
+ upper_bounds: list[int] = []
+ resolved_bounds: dict[str, dict[str, float]] = {}
+ for i in attrs_list:
+ if i not in self.cells.columns:
+ logger.warning(
+ f"{i} not found in cell metadata. Will ignore {i} for filtering"
+ )
+ continue
+ a = self.cells.fetch_all(i)
+ low, high = gaussian_quantile_bounds(a, min_p, max_p)
+ resolved_bounds[i] = {"low": float(low), "high": float(high)}
+ attrs_used.append(i)
+ lower_bounds.append(cast(int, low))
+ upper_bounds.append(cast(int, high))
+
+ if attrs_used:
+ self.filter_cells(
+ attrs=attrs_used,
+ lows=lower_bounds,
+ highs=upper_bounds,
+ invalidate_cache=False,
+ )
+
+ self._record_cell_selection(
+ column="I",
+ operation="auto_filter_cells",
+ parameters={
+ "attrs": attrs_used,
+ "min_p": min_p,
+ "max_p": max_p,
+ "resolved_bounds": resolved_bounds,
+ },
+ inputs={},
+ invalidate_cache=invalidate_cache,
+ )
+
+ if show_qc_plots and attrs_used:
+ # Match the previous plot_cells_dists contract: pre uses every cell,
+ # post uses the filtered active set under cell key I.
+ distribution(
+ self,
+ keys=attrs_used,
+ cell_key=None,
+ color="steelblue",
+ title="Pre-filtering distribution",
+ show=True,
+ )
+ distribution(
+ self,
+ keys=attrs_used,
+ cell_key="I",
+ color="coral",
+ title="Post-filtering distribution",
+ show=True,
+ )
+
+ def _auto_filter_cells_sample_mad(
+ self,
+ *,
+ attrs: list[str],
+ min_p: float,
+ max_p: float,
+ show_qc_plots: bool,
+ invalidate_cache: bool,
+ sample_column: str,
+ n_mads: float,
+ min_cells_per_sample: int,
+ ) -> None:
+ from ...plotting import distribution
+
+ if min_p != 0.01 or max_p != 0.99:
+ raise ValueError(
+ "min_p and max_p apply only to the global Gaussian path. "
+ "Leave them at their defaults (0.01 and 0.99) when "
+ "sample_column is set, and use n_mads to control MAD bounds"
+ )
+ if isinstance(n_mads, bool) or not isinstance(n_mads, Real):
+ raise TypeError("n_mads must be a positive number")
+ resolved_n_mads = float(n_mads)
+ if not np.isfinite(resolved_n_mads) or resolved_n_mads <= 0:
+ raise ValueError("n_mads must be finite and greater than 0")
+ if (
+ not isinstance(min_cells_per_sample, int)
+ or isinstance(min_cells_per_sample, bool)
+ or min_cells_per_sample < 2
+ ):
+ raise ValueError("min_cells_per_sample must be an integer >= 2")
+ if sample_column not in self.cells.columns:
+ raise ValueError(
+ f"sample_column '{sample_column}' not found in cell metadata"
+ )
+
+ active = np.asarray(self.cells.fetch_all("I"), dtype=bool)
+ sample_labels = np.asarray(self.cells.fetch_all(sample_column))
+ sample_labels = _validated_sample_labels(
+ sample_labels,
+ active,
+ label_name=f"sample_column '{sample_column}'",
+ )
+ active_labels = sample_labels[active]
+ if active_labels.size == 0:
+ raise ValueError("No active cells are available for sample-aware filtering")
+
+ attrs_used: list[str] = []
+ values_by_attr: dict[str, np.ndarray] = {}
+ for attr in attrs:
+ if attr not in self.cells.columns:
+ logger.warning(
+ f"{attr} not found in cell metadata. Will ignore {attr} for filtering"
+ )
+ continue
+ values = np.asarray(self.cells.fetch_all(attr), dtype=float)
+ _validated_work_scale(
+ values[active],
+ attr=attr,
+ transform=_metric_policy(attr)["transform"],
+ )
+ attrs_used.append(attr)
+ values_by_attr[attr] = values
+
+ parameters: dict[str, Any] = {
+ "attrs": attrs_used,
+ "sample_column": sample_column,
+ "n_mads": resolved_n_mads,
+ "min_cells_per_sample": int(min_cells_per_sample),
+ "resolved_bounds": {},
+ }
+
+ mad_provenance = None
+ if attrs_used:
+ keep, mad_provenance = _sample_aware_mad_mask(
+ values_by_attr=values_by_attr,
+ sample_labels=sample_labels,
+ active=active,
+ n_mads=resolved_n_mads,
+ min_cells_per_sample=int(min_cells_per_sample),
+ attrs=attrs_used,
+ )
+ parameters.update(
+ {
+ "mad_scale": mad_provenance["mad_scale"],
+ "metric_policies": mad_provenance["metric_policies"],
+ "sample_sizes": mad_provenance["sample_sizes"],
+ "skip_reasons": mad_provenance["skip_reasons"],
+ "resolved_bounds": mad_provenance["resolved_bounds"],
+ }
+ )
+ fingerprint_inputs: dict[str, Any] = {
+ "sample_assignments_fingerprint": fingerprint_strings(active_labels),
+ "qc_metric_fingerprints": {
+ attr: fingerprint_array(values_by_attr[attr][active])
+ for attr in attrs_used
+ },
+ }
+ canonical_bytes(
+ {
+ "operation": "auto_filter_cells",
+ "parameters": parameters,
+ "inputs": fingerprint_inputs,
+ }
+ )
+ prior_selection = self._ensure_cell_selection("I")
+ inputs: dict[str, Any] = {
+ "prior_cell_selection": prior_selection,
+ **fingerprint_inputs,
+ }
+ canonical_bytes(
+ {
+ "operation": "auto_filter_cells",
+ "parameters": parameters,
+ "inputs": inputs,
+ }
+ )
+
+ if attrs_used:
+ assert mad_provenance is not None
+ for message in mad_provenance["warnings"]:
+ logger.warning(message)
+ self.cells.update_key(keep, key="I")
+ remaining = int(np.asarray(self.cells.fetch_all("I"), dtype=bool).sum())
+ logger.info(f"Cell filtering retained {remaining}/{self.cells.N} cells")
+
+ self._record_cell_selection(
+ column="I",
+ operation="auto_filter_cells",
+ parameters=parameters,
+ inputs=inputs,
+ invalidate_cache=invalidate_cache,
+ )
+
+ if show_qc_plots and attrs_used:
+ distribution(
+ self,
+ keys=attrs_used,
+ cell_key=None,
+ color="steelblue",
+ title="Pre-filtering distribution",
+ show=True,
+ )
+ distribution(
+ self,
+ keys=attrs_used,
+ cell_key="I",
+ color="coral",
+ title="Post-filtering distribution",
+ show=True,
+ )
+
+ def mark_hto_identities(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ label: str = "Hashtag_identity",
+ random_seed: int = 0,
+ invalidate_cache: bool = False,
+ ) -> str:
+ """Assign HTO hashtag identities to cells using demultiplexing.
+
+ Args:
+ from_assay: HTO assay name (default: ``'HTO'``).
+ cell_key: Boolean cell metadata column selecting cells (default: latest for assay).
+ label: Column name to store identities in cell metadata.
+ random_seed: Seed used for HTO demultiplexing.
+ invalidate_cache: Recompute even when matching provenance exists.
+
+ Returns:
+ Name of the cell metadata column containing HTO identities.
+ """
+ if from_assay is None:
+ from_assay = "HTO"
+ if cell_key is None:
+ cell_key = self._get_latest_cell_key(from_assay)
+ assay = self._get_assay(from_assay)
+ if isinstance(random_seed, bool) or not isinstance(random_seed, int):
+ raise TypeError("random_seed must be an integer")
+ n_cells = len(self.cells.active_index(cell_key))
+ required_cells = assay.feats.N + 1
+ if n_cells < required_cells:
+ raise ValueError(
+ f"HTO demultiplexing requires at least {required_cells} selected cells"
+ )
+ selection = self._ensure_cell_selection(cell_key)
+ arguments = HtoIdentityArguments(
+ cell_selection=selection,
+ feature_ids_fingerprint=fingerprint_strings(
+ np.asarray(assay.feats.fetch_all("ids"))
+ ),
+ method=_hto_demux_method(),
+ random_seed=random_seed,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ label=label,
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope="assay",
+ assay=from_assay,
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ cell_selection=selection,
+ arrays={"values": ((n_cells,), None)},
+ invalidate_cache=invalidate_cache,
+ )
+ preserved_display = column_display(self.zw, label)
+ if planned.reused:
+ artifact_group = as_zarr_group(
+ self.zw[artifact_path(planned.ref)],
+ name=planned.ref.artifact_id,
+ )
+ values = artifact_values(artifact_group, "values")
+ self.cells.insert(
+ column_name=label,
+ values=values,
+ overwrite=True,
+ key=cell_key,
+ )
+ link_cell_data_column(
+ self.zw,
+ label,
+ planned.ref,
+ value_name="values",
+ default_display=categorical_display(values),
+ preserved_display=preserved_display,
+ )
+ return label
+ counts = controlled_compute(
+ assay.rawData[self.cells.fetch_all(cell_key)], self.nthreads
+ )
+ hto_idents = hto_demux(
+ pd.DataFrame(counts, columns=assay.feats.fetch_all("ids")),
+ random_seed=random_seed,
+ )
+ values = np.asarray(hto_idents.values)
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {"values": values},
+ )
+ self.cells.insert(
+ column_name=label,
+ values=values,
+ overwrite=True,
+ key=cell_key,
+ )
+ link_cell_data_column(
+ self.zw,
+ label,
+ planned.ref,
+ value_name="values",
+ default_display=categorical_display(values),
+ preserved_display=preserved_display,
+ )
+ return label
+
+ def run_doublet_detection(
+ self,
+ cluster_key: str,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ cluster_sample_fraction: float = 0.05,
+ max_cells_per_cluster: int = 100,
+ simulation_ratio: float = 1.0,
+ heterotypic_fraction: float = 0.8,
+ save_k: int = 5,
+ smoothing_t: int = 2,
+ normalize_scores: bool = True,
+ label: str = "doublet_score",
+ random_seed: int = 4444,
+ invalidate_cache: bool = False,
+ ) -> str:
+ """Flag potential doublets by simulating and mapping synthetic doublets.
+
+ Synthetic doublets are simulated by summing the raw counts of pairs of
+ observed cells drawn from a per-cluster subsample, with a tunable bias
+ toward cross-cluster (heterotypic) pairs. The simulated profiles are
+ projected onto the existing reference graph with `run_mapping`, and each
+ reference cell is scored by how frequently it appears among the nearest
+ neighbours of the simulated doublets (`get_mapping_score`). The score is
+ then diffused over the KNN graph using the same operator as
+ `get_imputed`. The final per-cell score is written to cell metadata so
+ that users can threshold and filter doublets themselves. This is a
+ graph-native adaptation of the Scrublet and DoubletFinder approach.
+
+ Args:
+ cluster_key: Cell metadata column with cluster or group labels used
+ to stratify the candidate pool (for example ``'RNA_cluster'``).
+ from_assay: Assay to use. Defaults to the latest used assay. Only
+ RNAassay type assays are supported.
+ cell_key: Cell key matching the desired graph (default: ``'I'``).
+ feat_key: Feature key matching the desired graph. Defaults to the
+ latest used feature key for the assay.
+ cluster_sample_fraction: Fraction of cells sampled from each cluster
+ to build the candidate pool. (Default value: 0.05)
+ max_cells_per_cluster: Cap on the number of cells sampled per cluster.
+ (Default value: 100)
+ simulation_ratio: Number of simulated doublets expressed as a
+ multiple of the number of reference cells. (Default value: 1.0)
+ heterotypic_fraction: Fraction of simulated doublets forced to be
+ cross-cluster. Set to 0 to disable the bias. (Default value: 0.8)
+ save_k: Number of reference neighbours stored per simulated doublet.
+ (Default value: 5)
+ smoothing_t: Diffusion power used to smoothen scores over the graph,
+ same as the ``t`` parameter of `get_imputed`. (Default value: 2)
+ normalize_scores: If True, the final score is min-max scaled to the
+ 0-1 range for interpretability. (Default value: True)
+ label: Base name for the score column in cell metadata. The assay
+ name (and cell key when not ``'I'``) is prepended.
+ (Default value: 'doublet_score')
+ random_seed: Seed for reproducible sampling. (Default value: 4444)
+
+ Returns:
+ Name of the cell-metadata column containing the final scores.
+ """
+ import shutil
+ import tempfile
+
+ from scipy.sparse import csr_matrix
+
+ from ...quality_control.doublets import (
+ sample_cluster_pool,
+ simulate_doublet_pairs,
+ write_doublet_target_zarr,
+ )
+
+ from_assay, cell_key, feat_key = self._get_latest_keys(
+ from_assay, cell_key, feat_key
+ )
+ source_assay = self._get_assay(from_assay)
+ if type(source_assay) != RNAassay: # noqa: E721
+ raise TypeError(
+ "ERROR: Doublet detection is only supported for RNAassay type assays. "
+ f"The provided assay is {type(source_assay)} type"
+ )
+ if cluster_key not in self.cells.columns:
+ raise ValueError(
+ f"ERROR: `cluster_key` {cluster_key} not found in cell metadata. Provide a column "
+ f"with cluster or group labels, for example '{from_assay}_cluster'"
+ )
+ selection = self._ensure_cell_selection(cell_key)
+ cluster_input = self._resolve_cell_data_provenance_input(
+ cluster_key,
+ cell_key=cell_key,
+ )
+ state = read_assay_state(self.zw, from_assay)
+ if (
+ state is None
+ or not state.matches(cell_key, feat_key)
+ or state.connectivity_map is None
+ ):
+ raise ValueError(
+ "Doublet detection requires the selected RNA graph to be an "
+ "artifact-backed connectivity chain. Rebuild and select the graph "
+ "with query_neighbors(...) and build_connectivity_map(...)."
+ )
+ connectivity = state.connectivity_map
+ connectivity_status = self._require_complete_artifact(
+ connectivity,
+ "connectivity_map",
+ assay=from_assay,
+ )
+ if connectivity_status.operation != "build_connectivity_map":
+ raise ValueError(
+ "Doublet detection requires a build_connectivity_map artifact. "
+ "Rebuild and select the RNA graph before running doublet detection."
+ )
+ validate_artifact_graph_selection(
+ self.zw,
+ connectivity,
+ cell_key,
+ feat_key,
+ )
+ neighbors = self._artifact_input_ref(
+ connectivity,
+ "neighbors",
+ "neighbors",
+ )
+ if (
+ neighbors.scope != "assay"
+ or neighbors.assay != from_assay
+ or state.neighbors != neighbors
+ ):
+ raise ValueError(
+ "The selected connectivity artifact does not match the selected "
+ "RNA neighbors chain. Rebuild and select the graph before running "
+ "doublet detection."
+ )
+ graph_selection = self._graph_cell_selection(connectivity)
+ if not self._selection_artifacts_match(graph_selection, selection):
+ raise ValueError("cell_key does not match the graph cell selection")
+ neighbors_status = self._require_complete_artifact(
+ neighbors,
+ "neighbors",
+ assay=from_assay,
+ )
+ raw_coordinates = (neighbors_status.inputs or {}).get("coordinates")
+ if not isinstance(raw_coordinates, dict):
+ raise ValueError("The selected neighbors have no coordinate provenance")
+ coordinates = ArtifactRef.from_dict(raw_coordinates)
+ if coordinates.kind != "reduction":
+ raise ValueError(
+ "Doublet detection requires a plain scaled-PCA mapping reference. "
+ "The selected neighbors use batch-corrected or Symphony coordinates. "
+ "Rebuild and select an uncorrected scaled-PCA connectivity graph first."
+ )
+ n_active = len(self.cells.active_index(cell_key))
+ arguments = DoubletScoreArguments(
+ clusters=cluster_input,
+ connectivity_map=connectivity,
+ cluster_sample_fraction=cluster_sample_fraction,
+ max_cells_per_cluster=max_cells_per_cluster,
+ simulation_ratio=simulation_ratio,
+ heterotypic_fraction=heterotypic_fraction,
+ save_k=save_k,
+ smoothing_t=smoothing_t,
+ normalize_scores=normalize_scores,
+ random_seed=random_seed,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ label=label,
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope="assay",
+ assay=from_assay,
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ cell_selection=selection,
+ arrays={"values": ((n_active,), "f")},
+ invalidate_cache=invalidate_cache,
+ )
+ final_col = self._col_renamer(from_assay, cell_key, label)
+ preserved_display = column_display(self.zw, final_col)
+ if planned.reused:
+ artifact_group = as_zarr_group(
+ self.zw[artifact_path(planned.ref)],
+ name=planned.ref.artifact_id,
+ )
+ scores = artifact_values(artifact_group, "values")
+ self.cells.insert(
+ final_col,
+ scores,
+ key=cell_key,
+ overwrite=True,
+ )
+ link_cell_data_column(
+ self.zw,
+ final_col,
+ planned.ref,
+ value_name="values",
+ default_display=continuous_display(scores),
+ preserved_display=preserved_display,
+ )
+ return final_col
+
+ reference = None
+ named_reference = state.named_results.get("mapping_reference")
+ if named_reference is not None:
+ try:
+ candidate = self.get_mapping_reference(
+ named_reference,
+ from_assay=from_assay,
+ )
+ except (KeyError, RuntimeError, TypeError, ValueError):
+ pass
+ else:
+ if (
+ candidate.neighbors == neighbors
+ and candidate.assay_name == from_assay
+ and candidate.cell_key == cell_key
+ and candidate.feature_key == feat_key
+ and candidate.method == "pca"
+ and candidate.batch_correction is None
+ and candidate.symphony_state is None
+ ):
+ reference = candidate
+ if reference is None:
+ reference = self.build_mapping_reference(neighbors)
+ named_results = dict(state.named_results)
+ named_results["mapping_reference"] = reference.ref
+ self._publish_current_artifact(
+ connectivity,
+ update_state=True,
+ named_results=named_results,
+ )
+ if (
+ reference.neighbors != neighbors
+ or reference.assay_name != from_assay
+ or reference.cell_key != cell_key
+ or reference.feature_key != feat_key
+ or reference.method != "pca"
+ or reference.batch_correction is not None
+ or reference.symphony_state is not None
+ ):
+ raise RuntimeError(
+ "The prepared plain mapping reference does not match the selected "
+ "RNA graph chain"
+ )
+
+ rng = np.random.default_rng(random_seed)
+ active_idx = self.cells.active_index(cell_key)
+ clusters = self.cells.fetch(cluster_key, key=cell_key)
+
+ pool_positions = sample_cluster_pool(
+ clusters, cluster_sample_fraction, max_cells_per_cluster, rng
+ )
+ pool_clusters = np.asarray(clusters)[pool_positions]
+ pool_raw_rows = np.asarray(active_idx)[pool_positions]
+ logger.debug(
+ f"Sampled {len(pool_positions)} cells across "
+ f"{len(np.unique(pool_clusters))} clusters to seed doublet simulation"
+ )
+
+ pool_counts = controlled_compute(
+ source_assay.rawData[pool_raw_rows, :], self.nthreads
+ )
+ pool_csr = csr_matrix(pool_counts)
+
+ n_sim = max(1, int(round(simulation_ratio * n_active)))
+ left, right = simulate_doublet_pairs(
+ pool_clusters, n_sim, heterotypic_fraction, rng
+ )
+ sim_counts = (pool_csr[left] + pool_csr[right]).tocsr()
+ logger.debug(f"Simulated {n_sim} synthetic doublets")
+
+ temp_dir = tempfile.mkdtemp(prefix="scarf_doublet_")
+ target_name = f"_doublet_sim_{from_assay}"
+ try:
+ write_doublet_target_zarr(
+ zarr_loc=temp_dir,
+ assay_name=from_assay,
+ sim_counts=sim_counts,
+ feat_ids=source_assay.feats.fetch_all("ids"),
+ feat_names=source_assay.feats.fetch_all("names"),
+ dtype=str(source_assay.rawData.dtype),
+ mem_budget=self.memoryBytes,
+ nthreads=self.nthreads,
+ profile="fast_local",
+ )
+ target_ds = self._create_temporary_datastore(
+ temp_dir,
+ default_assay=from_assay,
+ assay_types={from_assay: "RNA"},
+ nthreads=self.nthreads,
+ )
+ result = target_ds.run_mapping(
+ reference,
+ mapping_name=target_name,
+ query_assay=from_assay,
+ save_k=save_k,
+ )
+
+ try:
+ _, raw_scores = next(
+ target_ds.get_mapping_score(
+ result,
+ reference=reference,
+ log_transform=True,
+ )
+ )
+ except StopIteration:
+ raise RuntimeError(
+ "ERROR: Mapping scores could not be computed for simulated doublets"
+ ) from None
+ raw_scores = np.asarray(raw_scores)
+ if raw_scores.shape != (n_active,):
+ raise RuntimeError(
+ "Doublet mapping scores do not match the selected reference cells"
+ )
+
+ temp_col = self._col_renamer(from_assay, cell_key, f"{label}__raw")
+ self.cells.insert(temp_col, raw_scores, key=cell_key, overwrite=True)
+ try:
+ smoothed = self.get_imputed(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feature_name=temp_col,
+ feat_key=feat_key,
+ t=smoothing_t,
+ )
+ finally:
+ self.cells.drop(temp_col)
+
+ scores = np.asarray(smoothed, dtype=float)
+ if normalize_scores:
+ lo, hi = scores.min(), scores.max()
+ scores = (scores - lo) / (hi - lo) if hi > lo else np.zeros_like(scores)
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {"values": scores},
+ )
+ self.cells.insert(final_col, scores, key=cell_key, overwrite=True)
+ link_cell_data_column(
+ self.zw,
+ final_col,
+ planned.ref,
+ value_name="values",
+ default_display=continuous_display(scores),
+ preserved_display=preserved_display,
+ )
+ logger.info(
+ f"Stored doublet scores in '{final_col}' using "
+ f"{n_sim} synthetic doublets"
+ )
+ finally:
+ shutil.rmtree(temp_dir, ignore_errors=True)
+ return final_col
+
+ def mark_prevalent_peaks(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ top_n: int = 10000,
+ prevalence_key_name: str = "prevalent_peaks",
+ invalidate_cache: bool = False,
+ ) -> None:
+ """Feature selection method for ATACassay type assays.
+
+ This method first calculates prevalence of each peak by computing sum of TF-IDF normalized values for each peak
+ and then marks `top_n` peaks with the highest prevalence as prevalent peaks.
+
+ Args:
+ from_assay: Assay to use for graph creation. If no value is provided then `defaultAssay` will be used
+ cell_key: Cells to use for selection of most prevalent peaks. By default, all cells with True value in
+ 'I' will be used. The provided value for `cell_key` should be a column in cell metadata table
+ with boolean values.
+ top_n: Number of top prevalent peaks to be selected. (Default: 10000)
+ prevalence_key_name: Base label for marking prevalent peaks in the features metadata column. The value for
+ 'cell_key' parameter is prepended to this value. (Default value: 'prevalent_peaks')
+
+ Returns:
+ None
+ """
+ if cell_key is None:
+ cell_key = "I"
+ assay = self._get_assay(from_assay)
+ if type(assay) != ATACassay: # noqa: E721
+ raise TypeError(
+ f"ERROR: This method of feature selection can only be applied to ATACassay type of assay. "
+ f"The provided assay is {type(assay)} type"
+ )
+ output_key = f"{cell_key}__{prevalence_key_name}"
+ preserved_display = feature_column_display(assay.z, output_key)
+ cell_selection = self._ensure_cell_selection(cell_key)
+ feature_values = np.asarray(assay.feats.fetch_all("I"), dtype=bool)
+ feature_selection = self._resolve_selection_input(
+ metadata_group=as_zarr_group(
+ assay.z["featureData"],
+ name="featureData",
+ ),
+ column="I",
+ values=feature_values,
+ row_ids=np.asarray(assay.feats.fetch_all("ids")),
+ scope="assay",
+ kind="feature_selection",
+ assay=assay.name,
+ invalidate_cache=False,
+ )
+ arguments = PrevalentPeakArguments(
+ cell_selection=cell_selection,
+ feature_selection=feature_selection,
+ normalization_method=callable_identity(assay.normMethod),
+ algorithm_version=1,
+ top_n=top_n,
+ from_assay=assay.name,
+ cell_key=cell_key,
+ prevalence_key_name=prevalence_key_name,
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ values = assay._prevalent_peak_mask(cell_key, top_n)
+ selection, values = resolve_generated_selection_artifact(
+ self.zw,
+ scope="assay",
+ assay=assay.name,
+ kind=arguments.artifact_kind,
+ values=values,
+ row_ids=np.asarray(assay.feats.fetch_all("ids")),
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ source_column=output_key,
+ invalidate_cache=invalidate_cache,
+ )
+ assay.feats.insert(
+ output_key,
+ values,
+ fill_value=False,
+ overwrite=True,
+ )
+ link_feature_data_column(
+ assay.z,
+ output_key,
+ selection,
+ value_name="values",
+ default_display=categorical_display(values),
+ preserved_display=preserved_display,
+ )
+
+ def run_cell_cycle_scoring(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ s_genes: list[str] | None = None,
+ g2m_genes: list[str] | None = None,
+ n_bins: int = 50,
+ rand_seed: int = 4466,
+ s_score_label: str = "S_score",
+ g2m_score_label: str = "G2M_score",
+ phase_label: str = "cell_cycle_phase",
+ invalidate_cache: bool = False,
+ ) -> None:
+ """Computes S and G2M phase scores by taking into account the average
+ expression of S and G2M phase genes respectively. Following steps are
+ taken for each phase:
+
+ - Average expression of all the genes in across `cell_key` cells is calculated
+ - The log average expression is divided in `n_bins` bins
+ - A control set of genes is identified by sampling genes from same expression bins where phase's genes are present.
+ - The average expression of phase genes (Ep) and control genes (Ec) is calculated per cell.
+ - A phase score is calculated as ``Ep - Ec``.
+ - G1 is assigned when both scores are negative.
+ - G2M is assigned when the G2M score exceeds the S score.
+ - S is assigned otherwise, including tied non-negative scores.
+
+ Args:
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
+ s_genes: A list of S phase genes. If not provided then Scarf loads pre-saved genes accessible at
+ `scarf.quality_control.s_phase_genes`
+ g2m_genes: A list of G2M phase genes. If not provided then Scarf loads pre-saved genes accessible at
+ `scarf.quality_control.g2m_phase_genes`
+ n_bins: Number of bins into which average expression of genes is divided.
+ rand_seed: A random values to set seed while sampling cells from a cluster randomly. (Default value: 4466)
+ s_score_label: A base label for saving the S phase scores into a cell metadata column
+ (Default value: 'S_score')
+ g2m_score_label: A base label for saving the G2M phase scores into a cell metadata column
+ (Default value: 'G2M_score')
+ phase_label: A base label for saving the inferred cell cycle phase into a cell metadata column
+ (Default value: 'cell_cycle_phase')
+
+ Returns: None
+ """
+ if from_assay is None:
+ from_assay = self._defaultAssay
+ assay = self._get_assay(from_assay)
+ if cell_key is None:
+ cell_key = "I"
+ if s_genes is None:
+ from ...quality_control.cell_cycle_genes import s_phase_genes
+
+ s_genes = list(s_phase_genes)
+ if g2m_genes is None:
+ from ...quality_control.cell_cycle_genes import g2m_phase_genes
+
+ g2m_genes = list(g2m_phase_genes)
+ control_size = min(len(s_genes), len(g2m_genes))
+ s_gene_indices = assay.feats.get_index_by(
+ s_genes,
+ "names",
+ None,
+ ).tolist()
+ g2m_gene_indices = assay.feats.get_index_by(
+ g2m_genes,
+ "names",
+ None,
+ ).tolist()
+ selection = self._ensure_cell_selection(cell_key)
+ n_cells = len(self.cells.active_index(cell_key))
+ arguments = CellCycleArguments(
+ s_gene_indices=tuple(s_gene_indices),
+ g2m_gene_indices=tuple(g2m_gene_indices),
+ normalization_method=callable_identity(assay.normMethod),
+ size_factor=getattr(assay, "sf", None),
+ control_size=control_size,
+ n_bins=n_bins,
+ rand_seed=rand_seed,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ s_score_label=s_score_label,
+ g2m_score_label=g2m_score_label,
+ phase_label=phase_label,
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope="assay",
+ assay=from_assay,
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ cell_selection=selection,
+ arrays={
+ "s_score": ((n_cells,), "f"),
+ "g2m_score": ((n_cells,), "f"),
+ "phase": ((n_cells,), None),
+ },
+ invalidate_cache=invalidate_cache,
+ )
+ if planned.reused:
+ artifact_group = as_zarr_group(
+ self.zw[artifact_path(planned.ref)],
+ name=planned.ref.artifact_id,
+ )
+ s_score = artifact_values(artifact_group, "s_score")
+ g2m_score = artifact_values(artifact_group, "g2m_score")
+ phase = artifact_values(artifact_group, "phase")
+ else:
+ try:
+ assay._load_stats_loc(cell_key)
+ except KeyError:
+ cast(Any, assay).set_feature_stats(cell_key)
+ s_score = assay.score_features(
+ s_genes,
+ cell_key,
+ control_size,
+ n_bins,
+ rand_seed,
+ )
+ g2m_score = assay.score_features(
+ g2m_genes,
+ cell_key,
+ control_size,
+ n_bins,
+ rand_seed,
+ )
+ phase = np.asarray(assign_cell_cycle_phase(s_score, g2m_score))
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {
+ "s_score": np.asarray(s_score),
+ "g2m_score": np.asarray(g2m_score),
+ "phase": phase,
+ },
+ )
+ s_score_label = self._col_renamer(
+ from_assay,
+ cell_key,
+ s_score_label,
+ )
+ g2m_score_label = self._col_renamer(
+ from_assay,
+ cell_key,
+ g2m_score_label,
+ )
+ phase_label = self._col_renamer(
+ from_assay,
+ cell_key,
+ phase_label,
+ )
+ s_display = column_display(self.zw, s_score_label)
+ g2m_display = column_display(self.zw, g2m_score_label)
+ phase_display = column_display(self.zw, phase_label)
+ self.cells.insert(s_score_label, s_score, key=cell_key, overwrite=True)
+ link_cell_data_column(
+ self.zw,
+ s_score_label,
+ planned.ref,
+ value_name="s_score",
+ default_display=continuous_display(s_score),
+ preserved_display=s_display,
+ )
+ self.cells.insert(g2m_score_label, g2m_score, key=cell_key, overwrite=True)
+ link_cell_data_column(
+ self.zw,
+ g2m_score_label,
+ planned.ref,
+ value_name="g2m_score",
+ default_display=continuous_display(g2m_score),
+ preserved_display=g2m_display,
+ )
+ self.cells.insert(phase_label, np.asarray(phase), key=cell_key, overwrite=True)
+ link_cell_data_column(
+ self.zw,
+ phase_label,
+ planned.ref,
+ value_name="phase",
+ default_display=categorical_display(np.asarray(phase)),
+ preserved_display=phase_display,
+ )
diff --git a/scarf/datastore/_operations/trajectory.py b/scarf/datastore/_operations/trajectory.py
new file mode 100644
index 00000000..e92fedd6
--- /dev/null
+++ b/scarf/datastore/_operations/trajectory.py
@@ -0,0 +1,1595 @@
+from typing import TYPE_CHECKING, Any, Literal, cast
+
+import numpy as np
+import pandas as pd
+from scipy.sparse import coo_matrix
+
+from ...assay import Assay
+from ...graph.state import resolve_stored_graph_input, validate_legacy_graph_selection
+from ...matrix import ChunkedArray
+from ...metadata.arguments import (
+ FateMappingArguments,
+ PseudotimeAggregationArguments,
+ PseudotimeMarkerArguments,
+ PseudotimeScoringArguments,
+)
+from ...metadata.artifacts import (
+ artifact_values,
+ categorical_display,
+ column_display,
+ continuous_display,
+ feature_column_display,
+ link_cell_data_column,
+ link_feature_data_column,
+ plan_cell_data_artifact,
+ write_cell_data_artifact,
+)
+from ...storage.artifact_writer import (
+ ArrayRequirement,
+ AttributeRequirement,
+ finish_artifact,
+ plan_artifact,
+ reused_artifact_group,
+ start_artifact,
+)
+from ...storage.artifacts import (
+ ArtifactRef,
+ artifact_path,
+ callable_identity,
+ provenance_hash,
+)
+from ...storage.types import as_zarr_array, as_zarr_group
+from ...storage.arrays import create_zarr_dataset
+from ...trajectory.feature_dynamics import (
+ scatter_feature_clusters as _scatter_feature_clusters_impl,
+ validate_pseudotime_regressor,
+)
+from ...trajectory.fate import (
+ compute_fate_probabilities as _compute_fate_probabilities_impl,
+ make_sink_tokens as _make_sink_tokens_impl,
+)
+from ...trajectory.pseudotime import (
+ make_source_sink_vector as _make_source_sink_vector_impl,
+ random_walk_laplacian_transpose as _random_walk_laplacian_transpose_impl,
+ select_pseudotime_component as _select_pseudotime_component_impl,
+ truncated_pba_potential as _truncated_pba_potential_impl,
+ validate_source_sink_labels as _validate_source_sink_labels_impl,
+ validate_source_sink_vector as _validate_source_sink_vector_impl,
+)
+from ...trajectory.results import (
+ FateMappingResult,
+ PseudotimeAggregationResult,
+ PseudotimeMarkerResult,
+ PseudotimeScoreResult,
+)
+from ...utils.arrays import array_digest
+from ...utils.logging import logger
+
+if TYPE_CHECKING:
+ from ..mapping_datastore import (
+ MappingDatastore as _TrajectoryFeatureOperationsBase,
+ )
+ from .graph import _GraphOperationsMixin as _TrajectoryOperationsBase
+else:
+ _TrajectoryFeatureOperationsBase = object
+ _TrajectoryOperationsBase = object
+
+
+def _group_assignment_digest(values: np.ndarray) -> str:
+ return array_digest(np.asarray(values).astype(str))
+
+
+def _validate_assay_pseudotime(
+ assay: Assay,
+ cell_key: str,
+ pseudotime_key: str,
+) -> np.ndarray:
+ try:
+ values = assay.cells.fetch(pseudotime_key, key=cell_key)
+ except (TypeError, ValueError) as exc:
+ raise TypeError(
+ f"Pseudotime column '{pseudotime_key}' must be numeric"
+ ) from exc
+ expected_size = assay.cells.active_index(cell_key).shape[0]
+ return validate_pseudotime_regressor(
+ values,
+ expected_size,
+ pseudotime_key,
+ cell_key,
+ has_validity_column=f"{pseudotime_key}__valid" in assay.cells.columns,
+ )
+
+
+def _stored_graph_input(
+ store: Any,
+ from_assay: str,
+ cell_key: str,
+ feat_key: str,
+) -> tuple[str, object]:
+ graph_loc = store.get_latest_graph_loc(from_assay, cell_key, feat_key)
+ return graph_loc, resolve_stored_graph_input(store.zw, graph_loc)
+
+
+class _TrajectoryOperationsMixin(_TrajectoryOperationsBase):
+ def get_diffusion_operator(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ t: int = 2,
+ *,
+ cache_operator: bool = True,
+ invalidate_cache: bool = False,
+ ) -> coo_matrix:
+ """Load or calculate the sparse MAGIC diffusion operator."""
+ from ...neighbors.diffusion import diffusion_operator
+
+ from_assay, cell_key, feat_key = self._get_latest_keys(
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ graph_loc = self.get_latest_graph_loc(from_assay, cell_key, feat_key)
+ graph_input: object = resolve_stored_graph_input(self.zw, graph_loc)
+ planned = plan_artifact(
+ self.zw,
+ scope="assay",
+ assay=from_assay,
+ kind="diffusion_operator",
+ operation="get_diffusion_operator",
+ parameters={"t": t},
+ inputs={"connectivity_map": graph_input},
+ execution_options={
+ "cache_operator": cache_operator,
+ "invalidate_cache": invalidate_cache,
+ },
+ invalidate_cache=invalidate_cache,
+ required_arrays=(
+ ArrayRequirement("row", dtype_kind="u"),
+ ArrayRequirement("col", dtype_kind="u"),
+ ArrayRequirement("data", dtype_kind="f"),
+ ),
+ )
+ magic_loc = planned.ref.artifact_id
+ read_only_magic_loc = f"read_only:{provenance_hash(planned.provenance)}"
+ if not cache_operator:
+ self._cachedMagicOperator = None
+ self._cachedMagicOperatorLoc = None
+ cache_lookup_loc = (
+ read_only_magic_loc
+ if self.zarr_mode != "r+" and not planned.reused
+ else magic_loc
+ )
+ cached = (
+ self._cachedMagicOperator
+ if self._cachedMagicOperatorLoc == cache_lookup_loc
+ else None
+ )
+ if self.zarr_mode != "r+" and not planned.reused:
+ if cache_operator and not invalidate_cache and cached is not None:
+ return cast(coo_matrix, cached)
+ legacy_path = f"{graph_loc}/magic_{t}"
+ if not invalidate_cache and legacy_path in self.zw:
+ legacy = as_zarr_group(self.zw[legacy_path], name=legacy_path)
+ n_cells, _ = self._get_graph_ncells_k(graph_loc)
+ diff_op = coo_matrix(
+ (
+ np.asarray(as_zarr_array(legacy["data"], name="data")[:]),
+ (
+ np.asarray(as_zarr_array(legacy["row"], name="row")[:]),
+ np.asarray(as_zarr_array(legacy["col"], name="col")[:]),
+ ),
+ ),
+ shape=(n_cells, n_cells),
+ )
+ else:
+ graph = self.load_graph(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ symmetric=True,
+ upper_only=False,
+ )
+ diff_op = diffusion_operator(graph, t)
+ if cache_operator:
+ self._cachedMagicOperator = diff_op
+ self._cachedMagicOperatorLoc = read_only_magic_loc # type: ignore[assignment]
+ return diff_op
+ if planned.reused:
+ store = reused_artifact_group(self.zw, planned)
+ if cached is not None:
+ diff_op = cast(coo_matrix, cached)
+ else:
+ logger.debug("Using existing MAGIC diffusion operator")
+ n_cells, _ = self._get_graph_ncells_k(graph_loc)
+ diff_op = coo_matrix(
+ (
+ np.asarray(as_zarr_array(store["data"], name="data")[:]),
+ (
+ np.asarray(as_zarr_array(store["row"], name="row")[:]),
+ np.asarray(as_zarr_array(store["col"], name="col")[:]),
+ ),
+ ),
+ shape=(n_cells, n_cells),
+ )
+ else:
+ graph = self.load_graph(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ symmetric=True,
+ upper_only=False,
+ )
+ diff_op = diffusion_operator(graph, t)
+ shape = diff_op.data.shape
+ store = start_artifact(self.zw, planned)
+ for name, dtype in zip(
+ ("row", "col", "data"),
+ ("uint32", "uint32", "float64"),
+ strict=True,
+ ):
+ array = create_zarr_dataset(store, name, (1000000,), dtype, shape)
+ array[:] = getattr(diff_op, name)
+ store.attrs["n_cells"] = int(graph.shape[0])
+ finish_artifact(store, planned)
+
+ if cache_operator:
+ self._cachedMagicOperator = diff_op
+ self._cachedMagicOperatorLoc = magic_loc # type: ignore[assignment]
+ else:
+ self._cachedMagicOperator = None
+ self._cachedMagicOperatorLoc = None
+ return diff_op
+
+ def get_imputed(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feature_name: str | None = None,
+ feat_key: str | None = None,
+ t: int = 2,
+ cache_operator: bool = True,
+ invalidate_cache: bool = False,
+ ) -> np.ndarray:
+ """Impute feature values by diffusing along the KNN graph (MAGIC-style).
+
+ Args:
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
+ feature_name: Name of the feature to be imputed
+ feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
+ used feature for the given assay will be used.
+ t: Same as the t parameter in MAGIC. Higher values lead to larger diffusion of values. Too large values
+ can slow down the algorithm and cause over-smoothening. (Default value: 2)
+ cache_operator: Whether to keep the diffusion operator in memory after the method returns. Can be useful
+ to set to True if many features are to imputed in a batch but can lead to increased memory
+ usage. (Default value: True)
+
+ Returns:
+ An array of imputed values for the given feature
+
+ """
+
+ from_assay, cell_key, feat_key = self._get_latest_keys(
+ from_assay, cell_key, feat_key
+ )
+ if feature_name is None:
+ raise ValueError(
+ "ERROR: Please provide name for the feature to be imputed. It can, for example, "
+ "be a gene name."
+ )
+ data = self.get_cell_vals(
+ from_assay=from_assay, cell_key=cell_key, k=feature_name
+ )
+ diff_op = self.get_diffusion_operator(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ t=t,
+ cache_operator=cache_operator,
+ invalidate_cache=invalidate_cache,
+ )
+ return cast(np.ndarray, diff_op.dot(data))
+
+ def run_pseudotime_scoring(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ subset_cell_key: str | None = None,
+ feat_key: str | None = None,
+ n_singular_vals: int = 30,
+ source_sink_key: str | None = None,
+ sources: list[Any] | None = None,
+ sinks: list[Any] | None = None,
+ ss_vec: np.ndarray | None = None,
+ min_max_norm_ptime: bool = True,
+ random_seed: int = 4444,
+ label: str = "pseudotime",
+ component_policy: Literal["largest", "error"] = "largest",
+ invalidate_cache: bool = False,
+ ) -> PseudotimeScoreResult:
+ """
+ Calculate differentiation potential of cells. This function is a reimplementation of population balance
+ analysis (PBA) approach published in Weinreb et al. 2017, PNAS. This function computes the random walk
+ normalized Laplacian matrix of the reference graph, L_rw = I-A/D and then calculates a Moore-Penrose
+ pseudoinverse of L_rw.
+
+ Args:
+ from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
+ cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
+ subset_cell_key: Cell key for the subset of cells for which pseudotime scoring is to be performed.
+ feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
+ used feature for the given assay will be used.
+ n_singular_vals: Number of the smallest singular values to save.
+ source_sink_key: Cell metadata column containing source and sink
+ group labels. Required with ``sources`` or ``sinks`` and omitted
+ when ``ss_vec`` is supplied.
+ sources: A list of group/clusters ids from `source_sink_key` column to be treated as sources. Sources are
+ usually progenitor/precursor or other actively dividing cell states.
+ sinks: A list of group/clusters ids from `source_sink_key` column to be treated as sinks. Sinks are usually
+ more differentiated (or terminally differentiated) cell states.
+ ss_vec: Custom source-sink value for each selected cell. It must sum
+ to zero, with negative values for sources and positive values for
+ sinks. This is mutually exclusive with label-based arguments.
+ min_max_norm_ptime: Whether to perform min-max normalization on the final pseudotime values so that values
+ are in 0 to 1 range. (Default: True)
+ random_seed: A random seed for svds (Default: 4444)
+ label: Base label for the pseudotime cell metadata column.
+ component_policy: How to handle a disconnected selected graph. ``'largest'`` scores the largest connected
+ component and marks other selected cells as unscored. ``'error'`` raises instead.
+
+ Returns:
+ Pseudotime values, validity mask, and their saved metadata keys.
+ """
+
+ from_assay, cell_key, feat_key = self._get_latest_keys(
+ from_assay, cell_key, feat_key
+ )
+ if subset_cell_key is None:
+ subset_cell_key = cell_key
+ cell_idx = self.cells.fetch(subset_cell_key, key=cell_key)
+ else:
+ cell_idx = self.cells.fetch(subset_cell_key, key=cell_key)
+ if cell_idx.sum() != self.cells.fetch_all(subset_cell_key).sum():
+ raise ValueError("subset_cell_key is not a complete subset of cell_key")
+
+ logger.info(
+ f"Pseudotime scoring: loading graph "
+ f"({from_assay}, cell_key={cell_key}, feat_key={feat_key})"
+ )
+ graph_loc, graph_input = _stored_graph_input(
+ self,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ if not isinstance(graph_input, ArtifactRef):
+ validate_legacy_graph_selection(
+ self,
+ graph_loc,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ graph = self.load_graph(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ symmetric=True,
+ upper_only=False,
+ )
+ if cell_idx.shape[0] != cell_idx.sum():
+ graph = graph[cell_idx][:, cell_idx]
+
+ if graph.shape[0] == 0:
+ raise ValueError("No cells were selected for pseudotime scoring")
+ parent_cell_indices = self.cells.active_index(cell_key)
+ selected_cell_indices = parent_cell_indices[np.asarray(cell_idx, dtype=bool)]
+ retained_mask, component_sizes = _select_pseudotime_component_impl(
+ graph,
+ selected_cell_indices,
+ component_policy,
+ )
+ retained_graph = graph[retained_mask][:, retained_mask].tocsr()
+ if len(component_sizes) > 1:
+ logger.warning(
+ f"Selected graph components have sizes {component_sizes}. "
+ f"Scoring the largest component with {retained_graph.shape[0]} cells"
+ )
+
+ retained_n_cells = retained_graph.shape[0]
+ if not isinstance(n_singular_vals, int) or isinstance(n_singular_vals, bool):
+ raise TypeError("n_singular_vals must be an integer")
+ if n_singular_vals < 2:
+ raise ValueError("n_singular_vals must be at least 2")
+ if retained_n_cells < 4:
+ raise ValueError(
+ "The retained graph must contain at least 4 cells for pseudotime scoring"
+ )
+ effective_k = min(n_singular_vals, retained_n_cells - 2)
+ if effective_k != n_singular_vals:
+ logger.warning(
+ f"Reducing n_singular_vals from {n_singular_vals} to {effective_k} "
+ "for the retained graph size"
+ )
+
+ label_arguments_supplied = (
+ source_sink_key is not None or sources is not None or sinks is not None
+ )
+ if ss_vec is not None and label_arguments_supplied:
+ raise ValueError(
+ "Provide either ss_vec or source_sink_key with source/sink labels, not both"
+ )
+ if ss_vec is None and source_sink_key is None:
+ if sources is not None or sinks is not None:
+ raise ValueError(
+ "source_sink_key is required when sources or sinks are provided"
+ )
+ raise ValueError("Provide source/sink labels or a custom zero-sum ss_vec")
+
+ if ss_vec is not None:
+ full_source_sink = _validate_source_sink_vector_impl(
+ ss_vec,
+ graph.shape[0],
+ "ss_vec",
+ )
+ source_sink_input: object = full_source_sink
+ retained_source_sink = _validate_source_sink_vector_impl(
+ full_source_sink[retained_mask],
+ retained_n_cells,
+ "ss_vec restricted to the retained component",
+ )
+ else:
+ if sources is not None and not isinstance(sources, list):
+ raise TypeError("sources must be a list")
+ if sinks is not None and not isinstance(sinks, list):
+ raise TypeError("sinks must be a list")
+ source_labels = [] if sources is None else sources
+ sink_labels = [] if sinks is None else sinks
+ if not source_labels and not sink_labels:
+ raise ValueError("At least one source or sink label must be provided")
+
+ selected_labels = pd.Series(
+ self.cells.fetch(cast(str, source_sink_key), key=cell_key)[cell_idx]
+ )
+ _validate_source_sink_labels_impl(
+ selected_labels,
+ source_labels,
+ sink_labels,
+ "the selected cells",
+ )
+ retained_labels = selected_labels.iloc[
+ np.flatnonzero(retained_mask)
+ ].reset_index(drop=True)
+ _validate_source_sink_labels_impl(
+ retained_labels,
+ source_labels,
+ sink_labels,
+ "the retained connected component",
+ )
+ retained_source_sink = _validate_source_sink_vector_impl(
+ _make_source_sink_vector_impl(
+ retained_labels,
+ source_labels,
+ sink_labels,
+ ),
+ retained_n_cells,
+ "generated source/sink vector",
+ )
+ source_sink_input = self._resolve_cell_data_provenance_input(
+ cast(str, source_sink_key),
+ cell_key=cell_key,
+ )
+
+ graph_selection = self._ensure_cell_selection(cell_key)
+ if isinstance(graph_input, ArtifactRef):
+ stored_selection = self._graph_cell_selection(graph_input)
+ if not self._selection_artifacts_match(
+ stored_selection,
+ graph_selection,
+ ):
+ raise ValueError("cell_key does not match the graph cell selection")
+ result_selection = self._ensure_cell_selection(subset_cell_key)
+ arguments = PseudotimeScoringArguments(
+ connectivity_map=graph_input,
+ source_sink=source_sink_input,
+ cell_selection=result_selection,
+ n_singular_vals=n_singular_vals,
+ sources=tuple([] if sources is None else sources),
+ sinks=tuple([] if sinks is None else sinks),
+ min_max_norm_ptime=min_max_norm_ptime,
+ random_seed=random_seed,
+ component_policy=component_policy,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ subset_cell_key=subset_cell_key,
+ feat_key=feat_key,
+ label=label,
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ artifact_scope = (
+ graph_input.scope if isinstance(graph_input, ArtifactRef) else "assay"
+ )
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope=artifact_scope,
+ assay=(
+ graph_input.assay
+ if isinstance(graph_input, ArtifactRef) and graph_input.scope == "assay"
+ else from_assay
+ if artifact_scope == "assay"
+ else None
+ ),
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ cell_selection=result_selection,
+ arrays={
+ "pseudotime": ((graph.shape[0],), "f"),
+ "valid": ((graph.shape[0],), "b"),
+ },
+ invalidate_cache=invalidate_cache,
+ )
+ output_column = self._col_renamer(from_assay, subset_cell_key, label)
+ validity_column = f"{output_column}__valid"
+ preserved_displays = {
+ output_column: column_display(self.zw, output_column),
+ validity_column: column_display(self.zw, validity_column),
+ }
+
+ def publish_result(
+ values: np.ndarray,
+ valid: np.ndarray,
+ ) -> PseudotimeScoreResult:
+ self.cells.insert(
+ output_column,
+ values,
+ key=subset_cell_key,
+ overwrite=True,
+ )
+ self.cells.insert(
+ validity_column,
+ valid,
+ fill_value=False,
+ key=subset_cell_key,
+ overwrite=True,
+ )
+ link_cell_data_column(
+ self.zw,
+ output_column,
+ planned.ref,
+ value_name="pseudotime",
+ default_display=continuous_display(values),
+ preserved_display=preserved_displays[output_column],
+ )
+ link_cell_data_column(
+ self.zw,
+ validity_column,
+ planned.ref,
+ value_name="valid",
+ default_display=categorical_display(valid),
+ preserved_display=preserved_displays[validity_column],
+ )
+ if not valid.all():
+ logger.warning(
+ f"Unscored cells contain NaN pseudotime. Use cell key "
+ f"'{validity_column}' for downstream analysis"
+ )
+ logger.info(
+ f"Stored pseudotime scores for {int(valid.sum())}/{len(valid)} cells"
+ )
+ return PseudotimeScoreResult(
+ pseudotime_key=output_column,
+ validity_key=validity_column,
+ assay=from_assay,
+ graph_cell_key=cell_key,
+ result_cell_key=subset_cell_key,
+ feature_key=feat_key,
+ values=values,
+ valid=valid,
+ )
+
+ if planned.reused:
+ artifact_group = reused_artifact_group(self.zw, planned)
+ return publish_result(
+ artifact_values(artifact_group, "pseudotime"),
+ artifact_values(artifact_group, "valid").astype(bool),
+ )
+
+ logger.debug("Pseudotime scoring: constructing Laplacian")
+ laplacian_transpose = _random_walk_laplacian_transpose_impl(retained_graph)
+ retained_ptime = _truncated_pba_potential_impl(
+ laplacian_transpose,
+ effective_k,
+ random_seed,
+ retained_source_sink,
+ )
+ if not np.isfinite(retained_ptime).all():
+ raise ValueError("Pseudotime calculation produced non-finite values")
+ value_range = float(np.ptp(retained_ptime))
+ potential_scale = max(1.0, float(np.abs(retained_ptime).max()))
+ if value_range <= np.finfo(float).eps * potential_scale:
+ raise ValueError("Pseudotime calculation produced a constant potential")
+ if min_max_norm_ptime:
+ retained_ptime = (retained_ptime - retained_ptime.min()) / value_range
+ retained_ptime = np.clip(retained_ptime, 0.0, 1.0)
+ if not np.isfinite(retained_ptime).all():
+ raise ValueError("Pseudotime normalization produced non-finite values")
+
+ ptime = np.full(graph.shape[0], np.nan, dtype=float)
+ ptime[retained_mask] = retained_ptime
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {
+ "pseudotime": ptime,
+ "valid": retained_mask,
+ },
+ )
+ logger.debug("Pseudotime scoring: saving pseudotime")
+ return publish_result(ptime, retained_mask)
+
+ def run_fate_mapping(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ subset_cell_key: str | None = None,
+ feat_key: str | None = None,
+ pseudotime_key: str | None = None,
+ sink_key: str | None = None,
+ sinks: list[Any] | None = None,
+ beta: float = 10.0,
+ solver_tol: float = 1e-6,
+ max_iterations: int = 1000,
+ label: str = "fate",
+ invalidate_cache: bool = False,
+ ) -> FateMappingResult:
+ """Compute cell fate probabilities toward user-provided sink groups.
+
+ Args:
+ from_assay: Assay whose neighborhood graph should be used.
+ cell_key: Cell key used to create the neighborhood graph.
+ subset_cell_key: Cell key restricting the graph for this calculation.
+ Use the pseudotime validity key when unscored cells
+ contain non-finite pseudotime.
+ feat_key: Feature key used to create the neighborhood graph.
+ pseudotime_key: Numeric cell metadata column containing pseudotime.
+ sink_key: Cell metadata column containing the sink labels.
+ sinks: Ordered sink labels. Every matching selected cell becomes a
+ fate boundary.
+ beta: Strength of the penalty applied to backward graph edges.
+ solver_tol: Relative tolerance used by the GMRES solver.
+ max_iterations: Maximum number of GMRES inner iterations per sink.
+ GMRES restart is fixed at 20.
+ label: Base label used for saved fate probability columns.
+
+ Returns:
+ Fate probabilities, validity mask, sink labels, and saved metadata keys.
+ """
+ if pseudotime_key is None:
+ raise ValueError("pseudotime_key must be provided")
+ if sink_key is None:
+ raise ValueError("sink_key must be provided")
+ if sinks is None:
+ raise ValueError("sinks must be provided")
+ if not isinstance(sinks, list):
+ raise TypeError("sinks must be a list")
+ if not sinks:
+ raise ValueError("At least one sink label must be provided")
+ if not isinstance(label, str):
+ raise TypeError("label must be a string")
+ if not label:
+ raise ValueError("label must not be empty")
+
+ from_assay, cell_key, feat_key = self._get_latest_keys(
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ if subset_cell_key is None:
+ subset_cell_key = cell_key
+ cell_idx = np.asarray(self.cells.fetch(subset_cell_key, key=cell_key))
+ if cell_idx.ndim != 1 or cell_idx.dtype.kind != "b":
+ raise TypeError("subset_cell_key must select cells with boolean values")
+ if int(cell_idx.sum()) != int(self.cells.fetch_all(subset_cell_key).sum()):
+ raise ValueError("subset_cell_key is not a complete subset of cell_key")
+ if not cell_idx.any():
+ raise ValueError("No cells were selected for fate mapping")
+
+ assay = self._get_assay(from_assay)
+ pseudotime = _validate_assay_pseudotime(
+ assay,
+ subset_cell_key,
+ pseudotime_key,
+ )
+ sink_values = np.asarray(assay.cells.fetch(sink_key, key=subset_cell_key))
+ storage_backed = all(
+ hasattr(self, name)
+ for name in (
+ "zw",
+ "get_latest_graph_loc",
+ "_ensure_cell_selection",
+ "_resolve_cell_data_provenance_input",
+ )
+ )
+ if storage_backed:
+ graph_loc, graph_input = _stored_graph_input(
+ self,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ if not isinstance(graph_input, ArtifactRef):
+ validate_legacy_graph_selection(
+ self,
+ graph_loc,
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ pseudotime_input = self._resolve_cell_data_provenance_input(
+ pseudotime_key,
+ cell_key=subset_cell_key,
+ )
+ sink_input = self._resolve_cell_data_provenance_input(
+ sink_key,
+ cell_key=subset_cell_key,
+ )
+
+ logger.info(
+ f"Fate mapping: loading graph "
+ f"({from_assay}, cell_key={cell_key}, feat_key={feat_key})"
+ )
+ graph = self.load_graph(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ symmetric=True,
+ upper_only=False,
+ )
+ if cell_idx.shape[0] != int(cell_idx.sum()):
+ graph = graph[cell_idx][:, cell_idx].tocsr()
+
+ sink_labels = tuple(sinks)
+ output_base = self._col_renamer(from_assay, subset_cell_key, label)
+ fate_keys = tuple(
+ f"{output_base}_{token}" for token in _make_sink_tokens_impl(sink_labels)
+ )
+ validity_key = f"{output_base}__valid"
+ planned = None
+ preserved_displays: dict[str, dict[str, Any] | None] = {}
+ if storage_backed:
+ graph_selection = self._ensure_cell_selection(cell_key)
+ if isinstance(graph_input, ArtifactRef):
+ stored_selection = self._graph_cell_selection(graph_input)
+ if not self._selection_artifacts_match(
+ stored_selection,
+ graph_selection,
+ ):
+ raise ValueError("cell_key does not match the graph cell selection")
+ result_selection = self._ensure_cell_selection(subset_cell_key)
+ arguments = FateMappingArguments(
+ connectivity_map=graph_input,
+ pseudotime=pseudotime_input,
+ sink_labels=sink_input,
+ cell_selection=result_selection,
+ sinks=sink_labels,
+ beta=beta,
+ solver_tol=solver_tol,
+ max_iterations=max_iterations,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ subset_cell_key=subset_cell_key,
+ feat_key=feat_key,
+ pseudotime_key=pseudotime_key,
+ sink_key=sink_key,
+ label=label,
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ artifact_scope = (
+ graph_input.scope if isinstance(graph_input, ArtifactRef) else "assay"
+ )
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope=artifact_scope,
+ assay=(
+ graph_input.assay
+ if isinstance(graph_input, ArtifactRef)
+ and graph_input.scope == "assay"
+ else from_assay
+ if artifact_scope == "assay"
+ else None
+ ),
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ cell_selection=result_selection,
+ arrays={
+ "probabilities": (
+ (graph.shape[0], len(sink_labels)),
+ "f",
+ ),
+ "valid": ((graph.shape[0],), "b"),
+ },
+ invalidate_cache=invalidate_cache,
+ )
+ preserved_displays = {
+ column: column_display(self.zw, column)
+ for column in (*fate_keys, validity_key)
+ }
+ if planned is not None and planned.reused:
+ artifact_group = reused_artifact_group(self.zw, planned)
+ probabilities = artifact_values(artifact_group, "probabilities")
+ valid = artifact_values(artifact_group, "valid").astype(bool)
+ else:
+ probabilities, valid, computed_sink_labels = (
+ _compute_fate_probabilities_impl(
+ graph,
+ pseudotime,
+ sink_values,
+ sinks,
+ beta=beta,
+ solver_tol=solver_tol,
+ max_iterations=max_iterations,
+ _copy_graph=False,
+ )
+ )
+ if computed_sink_labels != sink_labels:
+ raise ValueError("Computed fate labels do not match requested sinks")
+ if planned is not None:
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {
+ "probabilities": probabilities,
+ "valid": valid,
+ },
+ )
+
+ logger.debug("Fate mapping: saving probabilities")
+ for index, fate_key in enumerate(fate_keys):
+ self.cells.insert(
+ fate_key,
+ probabilities[:, index],
+ key=subset_cell_key,
+ overwrite=True,
+ )
+ self.cells.insert(
+ validity_key,
+ valid,
+ fill_value=False,
+ key=subset_cell_key,
+ overwrite=True,
+ )
+ if planned is not None:
+ for index, fate_key in enumerate(fate_keys):
+ link_cell_data_column(
+ self.zw,
+ fate_key,
+ planned.ref,
+ value_name="probabilities",
+ value_index=index,
+ default_display=continuous_display(probabilities[:, index]),
+ preserved_display=preserved_displays[fate_key],
+ )
+ link_cell_data_column(
+ self.zw,
+ validity_key,
+ planned.ref,
+ value_name="valid",
+ default_display=categorical_display(valid),
+ preserved_display=preserved_displays[validity_key],
+ )
+ logger.info(
+ f"Stored fate probabilities for {len(fate_keys)} sinks "
+ f"across {int(valid.sum())}/{len(valid)} cells"
+ )
+ return FateMappingResult(
+ fate_keys=fate_keys,
+ validity_key=validity_key,
+ sink_labels=sink_labels,
+ assay=from_assay,
+ graph_cell_key=cell_key,
+ result_cell_key=subset_cell_key,
+ feature_key=feat_key,
+ pseudotime_key=pseudotime_key,
+ sink_key=sink_key,
+ values=probabilities,
+ valid=valid,
+ )
+
+
+class _TrajectoryFeatureOperationsMixin(_TrajectoryFeatureOperationsBase):
+ def run_pseudotime_marker_search(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ pseudotime_key: str | None = None,
+ min_cells: int = 10,
+ gene_batch_size: int | None = None,
+ invalidate_cache: bool = False,
+ **norm_params: Any,
+ ) -> PseudotimeMarkerResult:
+ """Identify genes correlated with a pseudotime ordering of cells.
+
+ Args:
+ from_assay: Name of the assay to use. The default assay is used when omitted.
+ cell_key: Boolean cell metadata column selecting cells.
+ feat_key: Boolean feature metadata column selecting features.
+ pseudotime_key: Numeric cell metadata column containing pseudotime values.
+ min_cells: Minimum number of expressing cells required for a feature.
+ gene_batch_size: Number of features loaded per batch. When None,
+ selected features are grouped into chunk-aligned blocks that
+ fit the operation memory budget.
+ **norm_params: Extra keyword arguments forwarded to normalized expression.
+
+ Returns:
+ Correlation table and the feature metadata keys where it was saved.
+ """
+ from ...features.markers import find_markers_by_regression
+
+ if pseudotime_key is None:
+ raise ValueError(
+ "ERROR: Please provide a value for `pseudotime_key`. This should be the name of a column from "
+ "cell metadata object where pseudotime values are stored. If you ran `run_pseudotime_scoring` then "
+ "the values are stored under `RNA_pseudotime` by default."
+ )
+ if cell_key is None:
+ cell_key = "I"
+ if feat_key is None:
+ feat_key = "I"
+ assay = self._get_assay(from_assay)
+ ptime = _validate_assay_pseudotime(
+ assay,
+ cell_key,
+ pseudotime_key,
+ )
+ resolved_norm_params = {
+ **norm_params,
+ "log_transform": norm_params.get("log_transform", False),
+ "renormalize_subset": norm_params.get(
+ "renormalize_subset",
+ False,
+ ),
+ }
+ n_cells = len(assay.cells.active_index(cell_key))
+ feature_index = assay.feats.active_index(feat_key)
+ n_feats = len(feature_index)
+ logger.info(
+ f"Pseudotime markers: correlating features "
+ f"(cells={n_cells}, features={n_feats}, "
+ f"batch_size={gene_batch_size if gene_batch_size is not None else 'auto'})"
+ )
+ correlation_key = f"{cell_key}__{pseudotime_key}__r"
+ p_value_key = f"{cell_key}__{pseudotime_key}__p"
+ p_value_adjusted_key = f"{cell_key}__{pseudotime_key}__padj"
+ storage_backed = all(
+ hasattr(self, name)
+ for name in (
+ "zw",
+ "_ensure_cell_selection",
+ "_resolve_cell_data_provenance_input",
+ )
+ ) and hasattr(assay, "z")
+ planned = None
+ legacy_planned = None
+ preserved_displays: dict[str, dict[str, Any] | None] = {}
+ if storage_backed:
+ cell_selection = self._ensure_cell_selection(cell_key)
+ pseudotime_input = self._resolve_cell_data_provenance_input(
+ pseudotime_key,
+ cell_key=cell_key,
+ )
+ feature_values = np.asarray(
+ assay.feats.fetch_all(feat_key),
+ dtype=bool,
+ )
+ feature_selection = self._resolve_selection_input(
+ metadata_group=as_zarr_group(
+ assay.z["featureData"],
+ name="featureData",
+ ),
+ column=feat_key,
+ values=feature_values,
+ row_ids=np.asarray(assay.feats.fetch_all("ids")),
+ scope="assay",
+ kind="feature_selection",
+ assay=assay.name,
+ invalidate_cache=False,
+ )
+ arguments = PseudotimeMarkerArguments(
+ cell_selection=cell_selection,
+ feature_selection=feature_selection,
+ pseudotime=pseudotime_input,
+ normalization=resolved_norm_params,
+ normalization_method=callable_identity(assay.normMethod),
+ size_factor=getattr(assay, "sf", None),
+ association_method="pearson",
+ p_value_method="student_t",
+ adjustment_method="fdr_bh",
+ adjustment_scope="tested_features",
+ min_cells=min_cells,
+ from_assay=assay.name,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ pseudotime_key=pseudotime_key,
+ gene_batch_size=gene_batch_size,
+ n_threads=int(
+ getattr(
+ assay,
+ "nthreads",
+ getattr(self, "nthreads", 1),
+ )
+ ),
+ invalidate_cache=invalidate_cache,
+ )
+ record = arguments.to_record()
+ planned = arguments.plan(
+ self.zw,
+ scope="assay",
+ assay=assay.name,
+ invalidate_cache=invalidate_cache,
+ required_arrays=(
+ ArrayRequirement(
+ "r_value",
+ shape=(n_feats,),
+ dtype_kind="f",
+ ),
+ ArrayRequirement(
+ "p_value",
+ shape=(n_feats,),
+ dtype_kind="f",
+ ),
+ ArrayRequirement(
+ "p_value_adjusted",
+ shape=(n_feats,),
+ dtype_kind="f",
+ ),
+ ),
+ )
+ if not planned.reused and not invalidate_cache:
+ legacy_parameters = dict(record.parameters)
+ for field_name in (
+ "association_method",
+ "p_value_method",
+ "adjustment_method",
+ "adjustment_scope",
+ ):
+ legacy_parameters.pop(field_name)
+ legacy_planned = plan_artifact(
+ self.zw,
+ scope="assay",
+ assay=assay.name,
+ kind=arguments.artifact_kind,
+ operation=arguments.operation,
+ parameters=legacy_parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ required_arrays=(
+ ArrayRequirement(
+ "r_value",
+ shape=(n_feats,),
+ dtype_kind="f",
+ ),
+ ArrayRequirement(
+ "p_value",
+ shape=(n_feats,),
+ dtype_kind="f",
+ ),
+ ),
+ )
+ if not legacy_planned.reused:
+ legacy_planned = None
+ else:
+ legacy_planned = None
+ preserved_displays = {
+ correlation_key: feature_column_display(
+ assay.z,
+ correlation_key,
+ ),
+ p_value_key: feature_column_display(
+ assay.z,
+ p_value_key,
+ ),
+ p_value_adjusted_key: feature_column_display(
+ assay.z,
+ p_value_adjusted_key,
+ ),
+ }
+ if planned is not None and planned.reused:
+ artifact_group = reused_artifact_group(self.zw, planned)
+ r_values = artifact_values(artifact_group, "r_value")
+ p_values = artifact_values(artifact_group, "p_value")
+ p_values_adjusted = artifact_values(
+ artifact_group,
+ "p_value_adjusted",
+ )
+ markers = pd.DataFrame(
+ {
+ "r_value": r_values,
+ "p_value": p_values,
+ "p_value_adjusted": p_values_adjusted,
+ },
+ index=feature_index,
+ )
+ elif legacy_planned is not None and legacy_planned.reused:
+ from ...features.markers.correction import _bh_adjusted_pvalues
+
+ artifact_group = reused_artifact_group(self.zw, legacy_planned)
+ r_values = artifact_values(artifact_group, "r_value")
+ p_values = artifact_values(artifact_group, "p_value")
+ p_values_adjusted = _bh_adjusted_pvalues(p_values)
+ markers = pd.DataFrame(
+ {
+ "r_value": r_values,
+ "p_value": p_values,
+ "p_value_adjusted": p_values_adjusted,
+ },
+ index=feature_index,
+ )
+ if planned is not None and getattr(self, "zarr_mode", "r+") == "r+":
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {
+ "r_value": r_values,
+ "p_value": p_values,
+ "p_value_adjusted": p_values_adjusted,
+ },
+ )
+ else:
+ markers = find_markers_by_regression(
+ assay=assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ regressor=ptime,
+ min_cells=min_cells,
+ batch_size=gene_batch_size,
+ **resolved_norm_params,
+ )
+ markers = markers.reindex(feature_index)
+ if markers["r_value"].isna().any():
+ raise ValueError(
+ "Pseudotime marker results are not aligned to feat_key"
+ )
+ r_values = np.asarray(markers["r_value"].values)
+ p_values = np.asarray(markers["p_value"].values)
+ p_values_adjusted = np.asarray(markers["p_value_adjusted"].values)
+ if planned is not None and getattr(self, "zarr_mode", "r+") == "r+":
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {
+ "r_value": r_values,
+ "p_value": p_values,
+ "p_value_adjusted": p_values_adjusted,
+ },
+ )
+ publish_metadata = (
+ not storage_backed
+ or getattr(
+ self,
+ "zarr_mode",
+ "r+",
+ )
+ == "r+"
+ )
+ if publish_metadata:
+ logger.debug("Pseudotime markers: saving marker scores")
+ assay.feats.insert(
+ correlation_key,
+ r_values,
+ key=feat_key,
+ overwrite=True,
+ )
+ assay.feats.insert(
+ p_value_key,
+ p_values,
+ key=feat_key,
+ overwrite=True,
+ )
+ assay.feats.insert(
+ p_value_adjusted_key,
+ p_values_adjusted,
+ key=feat_key,
+ overwrite=True,
+ )
+ if planned is not None:
+ link_feature_data_column(
+ assay.z,
+ correlation_key,
+ planned.ref,
+ value_name="r_value",
+ default_display=continuous_display(r_values),
+ preserved_display=preserved_displays[correlation_key],
+ )
+ link_feature_data_column(
+ assay.z,
+ p_value_key,
+ planned.ref,
+ value_name="p_value",
+ default_display=continuous_display(p_values),
+ preserved_display=preserved_displays[p_value_key],
+ )
+ link_feature_data_column(
+ assay.z,
+ p_value_adjusted_key,
+ planned.ref,
+ value_name="p_value_adjusted",
+ default_display=continuous_display(p_values_adjusted),
+ preserved_display=preserved_displays[p_value_adjusted_key],
+ )
+ table = markers.rename_axis("feature_index").reset_index()
+ feature_names = np.asarray(assay.feats.fetch_all("names"), dtype=object)
+ table.insert(
+ 1,
+ "feature_name",
+ feature_names[table["feature_index"].to_numpy(dtype=np.int64)],
+ )
+ if publish_metadata:
+ logger.info(f"Stored pseudotime marker scores for {len(table)} features")
+ else:
+ logger.info(
+ f"Loaded pseudotime marker scores for {len(table)} features "
+ "without modifying the read-only store"
+ )
+ return PseudotimeMarkerResult(
+ table=table,
+ correlation_key=correlation_key,
+ p_value_key=p_value_key,
+ assay=assay.name,
+ cell_key=cell_key,
+ feature_key=feat_key,
+ pseudotime_key=pseudotime_key,
+ )
+
+ def run_pseudotime_aggregation(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ pseudotime_key: str | None = None,
+ cluster_label: str | None = None,
+ min_exp: float = 1e-3,
+ window_size: int = 200,
+ chunk_size: int = 50,
+ smoothen: bool = True,
+ z_scale: bool = True,
+ n_neighbours: int = 11,
+ n_clusters: int = 10,
+ batch_size: int | None = None,
+ ann_params: dict | None = None,
+ nan_cluster_value: int = -1,
+ invalidate_cache: bool = False,
+ **norm_params: Any,
+ ) -> PseudotimeAggregationResult:
+ """Cluster features by their pseudotime-ordered expression profiles.
+
+ Args:
+ from_assay: Name of the assay to use. The default assay is used when omitted.
+ cell_key: Boolean cell metadata column selecting cells.
+ feat_key: Boolean feature metadata column selecting features.
+ pseudotime_key: Required numeric cell metadata column containing
+ pseudotime values.
+ cluster_label: Required new or existing feature metadata column where
+ module identities are saved. Existing values are overwritten.
+ min_exp: Minimum mean normalized expression required for clustering.
+ window_size: Rolling window size used to smooth feature values.
+ chunk_size: Number of pseudotime bins to create.
+ smoothen: Whether to smooth expression along pseudotime.
+ z_scale: Whether to standardize each retained feature.
+ n_neighbours: Number of neighbors in the feature graph.
+ n_clusters: Number of feature modules to create.
+ batch_size: Number of features processed per batch. When None,
+ selected features are grouped into chunk-aligned blocks that
+ fit the operation memory budget.
+ ann_params: Parameters forwarded to the HNSW index.
+ nan_cluster_value: Value assigned to features excluded from clustering.
+ **norm_params: Extra keyword arguments forwarded to normalized expression.
+
+ Returns:
+ Lazy aggregated matrix with aligned feature indices and clusters.
+ """
+ from ...trajectory.feature_dynamics import knn_clustering
+
+ feat_key = feat_key or "I"
+ from_assay, cell_key, _ = self._get_latest_keys(
+ from_assay,
+ cell_key,
+ feat_key,
+ )
+ assay = self._get_assay(from_assay)
+
+ if pseudotime_key is None:
+ raise ValueError(
+ "ERROR: Please provide a value for `pseudotime_key` parameter. This is the column in "
+ "the cell attribute table that contains the pseudotime values."
+ )
+ if cluster_label is None:
+ raise ValueError(
+ "ERROR: Please provide a value for cluster_label. "
+ "It will be used to create new column in feature attribute table. The module identity "
+ "of each feature will be saved under this column name. If this column already exists "
+ "then it will be overwritten."
+ )
+ if not isinstance(nan_cluster_value, (int, np.integer)) or isinstance(
+ nan_cluster_value,
+ (bool, np.bool_),
+ ):
+ raise TypeError("nan_cluster_value must be an integer")
+ nan_cluster_value = int(nan_cluster_value)
+ _validate_assay_pseudotime(assay, cell_key, pseudotime_key)
+ resolved_norm_params = {
+ **norm_params,
+ "log_transform": norm_params.get("log_transform", False),
+ "renormalize_subset": norm_params.get(
+ "renormalize_subset",
+ False,
+ ),
+ }
+ resolved_ann_params = {} if ann_params is None else dict(ann_params)
+ (
+ cell_ordering,
+ _cell_indices,
+ feature_indices,
+ effective_window,
+ effective_bins,
+ input_fingerprints,
+ _legacy_parameters,
+ ) = assay._prepare_aggregated_ordering(
+ cell_key,
+ feat_key,
+ pseudotime_key,
+ min_exp=min_exp,
+ window_size=window_size,
+ chunk_size=chunk_size,
+ smoothen=smoothen,
+ z_scale=z_scale,
+ norm_params=resolved_norm_params,
+ )
+ cell_selection = self._ensure_cell_selection(cell_key)
+ pseudotime_input = self._resolve_cell_data_provenance_input(
+ pseudotime_key,
+ cell_key=cell_key,
+ )
+ feature_values = np.asarray(
+ assay.feats.fetch_all(feat_key),
+ dtype=bool,
+ )
+ feature_selection = self._resolve_selection_input(
+ metadata_group=as_zarr_group(
+ assay.z["featureData"],
+ name="featureData",
+ ),
+ column=feat_key,
+ values=feature_values,
+ row_ids=np.asarray(assay.feats.fetch_all("ids")),
+ scope="assay",
+ kind="feature_selection",
+ assay=assay.name,
+ invalidate_cache=False,
+ )
+ arguments = PseudotimeAggregationArguments(
+ cell_selection=cell_selection,
+ feature_selection=feature_selection,
+ pseudotime=pseudotime_input,
+ normalization=resolved_norm_params,
+ normalization_method=callable_identity(assay.normMethod),
+ size_factor=getattr(assay, "sf", None),
+ min_exp=min_exp,
+ window_size=window_size,
+ chunk_size=chunk_size,
+ smoothen=smoothen,
+ z_scale=z_scale,
+ n_neighbours=n_neighbours,
+ n_clusters=n_clusters,
+ ann_params=resolved_ann_params,
+ nan_cluster_value=nan_cluster_value,
+ from_assay=assay.name,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ pseudotime_key=pseudotime_key,
+ cluster_label=cluster_label,
+ batch_size=batch_size,
+ n_threads=self.nthreads,
+ invalidate_cache=invalidate_cache,
+ )
+ planned = arguments.plan(
+ self.zw,
+ scope="assay",
+ assay=assay.name,
+ invalidate_cache=invalidate_cache,
+ required_arrays=(
+ ArrayRequirement(
+ "data",
+ shape=(len(feature_indices), effective_bins),
+ dtype_kind="f",
+ ),
+ ArrayRequirement(
+ "feature_indices",
+ shape=(len(feature_indices),),
+ dtype_kind="u",
+ ),
+ ArrayRequirement(
+ "valid_features",
+ shape=(len(feature_indices),),
+ dtype_kind="b",
+ ),
+ ArrayRequirement(
+ "feature_clusters",
+ shape=(len(feature_indices),),
+ dtype_kind="i",
+ ),
+ ArrayRequirement(
+ "cluster_values",
+ shape=(assay.feats.N,),
+ dtype_kind="i",
+ ),
+ ),
+ required_attributes=(
+ AttributeRequirement(
+ "input_fingerprints",
+ expected_types=(list, tuple),
+ ),
+ AttributeRequirement(
+ "nan_cluster_value",
+ expected_types=(int,),
+ ),
+ ),
+ )
+ preserved_display = feature_column_display(assay.z, cluster_label)
+ if planned.reused:
+ aggregation_group = reused_artifact_group(self.zw, planned)
+ full_data = ChunkedArray(
+ as_zarr_array(aggregation_group["data"], name="data"),
+ nthreads=self.nthreads,
+ )
+ stored_feature_indices = np.asarray(
+ as_zarr_array(
+ aggregation_group["feature_indices"],
+ name="feature_indices",
+ )[:]
+ )
+ valid_features = np.asarray(
+ as_zarr_array(
+ aggregation_group["valid_features"],
+ name="valid_features",
+ )[:],
+ dtype=bool,
+ )
+ stored_feature_clusters = np.asarray(
+ as_zarr_array(
+ aggregation_group["feature_clusters"],
+ name="feature_clusters",
+ )[:],
+ dtype=np.int64,
+ )
+ cluster_values = np.asarray(
+ as_zarr_array(
+ aggregation_group["cluster_values"],
+ name="cluster_values",
+ )[:],
+ dtype=np.int64,
+ )
+ else:
+ logger.info("Pseudotime modules: aggregating feature profiles")
+ aggregation_group = start_artifact(self.zw, planned)
+ full_data, stored_feature_indices, valid_features = (
+ assay._write_aggregated_ordering_group(
+ aggregation_group,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ cell_ordering=cell_ordering,
+ feat_idx=feature_indices,
+ min_exp=min_exp,
+ effective_window=effective_window,
+ effective_bins=effective_bins,
+ smoothen=smoothen,
+ z_scale=z_scale,
+ batch_size=batch_size,
+ norm_params=resolved_norm_params,
+ )
+ )
+ valid_data = full_data[valid_features]
+ valid_feature_indices = stored_feature_indices[valid_features]
+ clusts = knn_clustering(
+ d_array=valid_data,
+ n_neighbours=n_neighbours,
+ n_clusters=n_clusters,
+ n_threads=self.nthreads,
+ ann_params=resolved_ann_params,
+ )
+ stored_feature_clusters = np.full(
+ len(stored_feature_indices),
+ nan_cluster_value,
+ dtype=np.int64,
+ )
+ stored_feature_clusters[valid_features] = clusts
+ cluster_values = _scatter_feature_clusters_impl(
+ assay.feats.N,
+ valid_feature_indices,
+ clusts,
+ nan_cluster_value,
+ )
+ for name, values in (
+ ("feature_clusters", stored_feature_clusters),
+ ("cluster_values", cluster_values),
+ ):
+ output = create_zarr_dataset(
+ aggregation_group,
+ name,
+ (max(len(values), 1),),
+ values.dtype,
+ values.shape,
+ )
+ output[:] = values
+ aggregation_group.attrs["input_fingerprints"] = input_fingerprints
+ aggregation_group.attrs["nan_cluster_value"] = nan_cluster_value
+ aggregation_group.attrs["effective_window"] = effective_window
+ aggregation_group.attrs["effective_bins"] = effective_bins
+ finish_artifact(aggregation_group, planned)
+ df = full_data[valid_features]
+ feat_ids = stored_feature_indices[valid_features]
+ clusts = stored_feature_clusters[valid_features]
+ logger.debug("Pseudotime modules: saving module labels")
+ assay.feats.insert(
+ cluster_label,
+ cluster_values,
+ fill_value=nan_cluster_value,
+ overwrite=True,
+ )
+ link_feature_data_column(
+ assay.z,
+ cluster_label,
+ planned.ref,
+ value_name="cluster_values",
+ default_display=categorical_display(cluster_values),
+ preserved_display=preserved_display,
+ )
+
+ legacy_cluster_digest: str | None = None
+ for assay_name in self.assay_names:
+ grouped_assay = self._get_assay(assay_name)
+ if (
+ grouped_assay.attrs.get("grouped_from_assay") != assay.name
+ or grouped_assay.attrs.get("grouped_group_key") != cluster_label
+ ):
+ continue
+ raw_group_artifact = grouped_assay.attrs.get("grouped_group_artifact")
+ if isinstance(raw_group_artifact, dict):
+ stale = raw_group_artifact != planned.ref.to_dict()
+ else:
+ if legacy_cluster_digest is None:
+ legacy_cluster_digest = _group_assignment_digest(cluster_values)
+ stale = (
+ grouped_assay.attrs.get("grouped_group_digest")
+ != legacy_cluster_digest
+ )
+ if stale:
+ logger.warning(
+ f"Grouped assay '{assay_name}' is stale after updating "
+ f"feature groups in '{cluster_label}'. Rerun add_grouped_assay"
+ )
+ logger.info(f"Stored {np.unique(clusts).size} pseudotime modules")
+ return PseudotimeAggregationResult(
+ data=df,
+ feature_indices=np.asarray(feat_ids),
+ feature_clusters=np.asarray(clusts),
+ cluster_key=cluster_label,
+ storage_path=(
+ f"{str(self.zw.path).strip('/')}/{artifact_path(planned.ref)}"
+ if str(self.zw.path).strip("/")
+ else artifact_path(planned.ref)
+ ),
+ assay=assay.name,
+ cell_key=cell_key,
+ feature_key=feat_key,
+ pseudotime_key=pseudotime_key,
+ )
diff --git a/scarf/datastore/_plot_accessor.py b/scarf/datastore/_plot_accessor.py
new file mode 100644
index 00000000..6088074c
--- /dev/null
+++ b/scarf/datastore/_plot_accessor.py
@@ -0,0 +1,960 @@
+"""Plotting functions bound to a datastore instance."""
+
+from collections.abc import Hashable, Mapping, Sequence
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, Literal
+
+import numpy as np
+
+from ..mapping.models import MappingResult
+from ..mapping.reference import MappingReference
+from ..plotting._contracts import (
+ CategoricalScale,
+ CellField,
+ ColorScale,
+ DensityOverlay,
+ DistKind,
+ FeatureRef,
+ FrameStyle,
+ Highlight,
+ LegendLoc,
+ NormalizationSpec,
+ SizeScale,
+ StudyDesign,
+)
+from ..plotting._figure import PlotResult
+from ..plotting.recipes import PlotRecipe, PlotRecipeResult
+from ..storage.refs import ArtifactRef
+
+if TYPE_CHECKING:
+ from .datastore import DataStore
+
+
+class DataStorePlotAccessor:
+ """Datastore-bound facade for store-first plotting functions."""
+
+ __slots__ = ("_store",)
+
+ def __init__(self, store: "DataStore") -> None:
+ self._store = store
+
+ def embedding(
+ self,
+ *,
+ layout_key: str | Sequence[str],
+ color_by: (
+ "str"
+ " | FeatureRef"
+ " | CellField"
+ " | Sequence[str | FeatureRef | CellField]"
+ " | None"
+ ) = None,
+ facet_by: str | None = None,
+ facet_order: Sequence[Any] | None = None,
+ cell_key: str = "I",
+ from_assay: str | None = None,
+ normalization: "NormalizationSpec | None" = None,
+ point_size: float | None = None,
+ point_sizes: "np.ndarray | Sequence[float] | None" = None,
+ point_size_range: tuple[float, float] = (1.0, 28.0),
+ point_edgecolor: str | None = None,
+ point_edgewidth: float | None = None,
+ point_alpha: float = 1.0,
+ sort_values: bool = False,
+ color_scale: "ColorScale | None" = None,
+ categorical_scale: "CategoricalScale | None" = None,
+ default_color: str = "steelblue",
+ missing_color: str | None = None,
+ clip_fraction: float = 0.0,
+ subset_by: str | None = None,
+ groups: Sequence[Any] | None = None,
+ n_columns: int | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ legend_loc: "LegendLoc" = "auto",
+ max_on_data_labels: int = 40,
+ show_legend: bool = True,
+ show_titles: bool = True,
+ frame: "FrameStyle" = "minimal",
+ density_overlay: "DensityOverlay | None" = None,
+ highlight: "Highlight | None" = None,
+ seed: int | None = None,
+ rasterize_threshold: int = 50_000,
+ show: bool = True,
+ ) -> "PlotResult":
+ """Plot cells in a stored two-dimensional embedding."""
+ from ..plotting import embedding
+
+ return embedding(
+ self._store,
+ layout_key=layout_key,
+ color_by=color_by,
+ facet_by=facet_by,
+ facet_order=facet_order,
+ cell_key=cell_key,
+ from_assay=from_assay,
+ normalization=normalization,
+ point_size=point_size,
+ point_sizes=point_sizes,
+ point_size_range=point_size_range,
+ point_edgecolor=point_edgecolor,
+ point_edgewidth=point_edgewidth,
+ point_alpha=point_alpha,
+ sort_values=sort_values,
+ color_scale=color_scale,
+ categorical_scale=categorical_scale,
+ default_color=default_color,
+ missing_color=missing_color,
+ clip_fraction=clip_fraction,
+ subset_by=subset_by,
+ groups=groups,
+ n_columns=n_columns,
+ target=target,
+ figsize=figsize,
+ theme=theme,
+ legend_loc=legend_loc,
+ max_on_data_labels=max_on_data_labels,
+ show_legend=show_legend,
+ show_titles=show_titles,
+ frame=frame,
+ density_overlay=density_overlay,
+ highlight=highlight,
+ seed=seed,
+ rasterize_threshold=rasterize_threshold,
+ show=show,
+ )
+
+ def embedding_raster(
+ self,
+ *,
+ layout_key: str,
+ color_by: "str | CellField | None" = None,
+ cell_key: str = "I",
+ pixels: int = 400,
+ block_rows: int | None = None,
+ color_scale: "ColorScale | None" = None,
+ missing_color: str = "white",
+ subset_by: str | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ seed: int = 0,
+ show: bool = True,
+ ) -> "PlotResult":
+ """Rasterize continuous cell metadata over a stored embedding."""
+ from ..plotting import embedding_raster
+
+ return embedding_raster(
+ self._store,
+ layout_key=layout_key,
+ color_by=color_by,
+ cell_key=cell_key,
+ pixels=pixels,
+ block_rows=block_rows,
+ color_scale=color_scale,
+ missing_color=missing_color,
+ subset_by=subset_by,
+ target=target,
+ figsize=figsize,
+ theme=theme,
+ seed=seed,
+ show=show,
+ )
+
+ def mapping_score(
+ self,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None = None,
+ target_groups: Sequence[Any] | np.ndarray | None = None,
+ layout_key: str | None = None,
+ kind: Literal["embedding", "histogram"] = "embedding",
+ query_assay: str | None = None,
+ log_transform: bool = True,
+ multiplier: float = 1000,
+ weighted: bool = True,
+ fixed_weight: float = 0.1,
+ bins: int = 40,
+ point_size: float | None = None,
+ color_scale: ColorScale | None = None,
+ categorical_scale: CategoricalScale | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+ ) -> PlotResult:
+ """Plot reference-cell mapping scores for one or more query groups."""
+ from ..plotting import mapping_score
+
+ return mapping_score(
+ self._store,
+ result,
+ reference=reference,
+ target_groups=target_groups,
+ layout_key=layout_key,
+ kind=kind,
+ query_assay=query_assay,
+ log_transform=log_transform,
+ multiplier=multiplier,
+ weighted=weighted,
+ fixed_weight=fixed_weight,
+ bins=bins,
+ point_size=point_size,
+ color_scale=color_scale,
+ categorical_scale=categorical_scale,
+ target=target,
+ figsize=figsize,
+ theme=theme,
+ show_legend=show_legend,
+ show=show,
+ )
+
+ def mapping_evidence(
+ self,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None = None,
+ reference_class_group: str,
+ target_groups: Sequence[Any] | np.ndarray | None = None,
+ metrics: Sequence[str] = (
+ "voteFraction",
+ "topTwoMargin",
+ "voteEntropy",
+ "referenceDistancePercentile",
+ ),
+ kind: Literal["histogram", "box", "embedding"] = "histogram",
+ reference_layout_key: str | None = None,
+ bins: int = 30,
+ query_assay: str | None = None,
+ threshold_fraction: float = 0.5,
+ na_val: str = "NA",
+ max_distance: float | None = None,
+ categorical_scale: CategoricalScale | None = None,
+ color_scale: ColorScale | None = None,
+ point_size: float | None = None,
+ show_unknown: bool = True,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+ ) -> PlotResult:
+ """Plot query-level label-transfer evidence."""
+ from ..plotting import mapping_evidence
+
+ return mapping_evidence(
+ self._store,
+ result,
+ reference=reference,
+ reference_class_group=reference_class_group,
+ target_groups=target_groups,
+ metrics=metrics,
+ kind=kind,
+ reference_layout_key=reference_layout_key,
+ bins=bins,
+ query_assay=query_assay,
+ threshold_fraction=threshold_fraction,
+ na_val=na_val,
+ max_distance=max_distance,
+ categorical_scale=categorical_scale,
+ color_scale=color_scale,
+ point_size=point_size,
+ show_unknown=show_unknown,
+ target=target,
+ figsize=figsize,
+ theme=theme,
+ show_legend=show_legend,
+ show=show,
+ )
+
+ def mapping_confusion(
+ self,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None = None,
+ reference_class_group: str,
+ known_labels: Sequence[Any] | np.ndarray,
+ normalize: Literal["none", "true", "predicted", "all"] = "true",
+ known_order: Sequence[Any] | None = None,
+ predicted_order: Sequence[Any] | None = None,
+ query_assay: str | None = None,
+ threshold_fraction: float = 0.5,
+ na_val: str = "NA",
+ max_distance: float | None = None,
+ color_scale: ColorScale | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+ ) -> PlotResult:
+ """Plot known query labels against transferred labels."""
+ from ..plotting import mapping_confusion
+
+ return mapping_confusion(
+ self._store,
+ result,
+ reference=reference,
+ reference_class_group=reference_class_group,
+ known_labels=known_labels,
+ normalize=normalize,
+ known_order=known_order,
+ predicted_order=predicted_order,
+ query_assay=query_assay,
+ threshold_fraction=threshold_fraction,
+ na_val=na_val,
+ max_distance=max_distance,
+ color_scale=color_scale,
+ target=target,
+ figsize=figsize,
+ theme=theme,
+ show_legend=show_legend,
+ show=show,
+ )
+
+ def mapping_calibration(
+ self,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None = None,
+ reference_class_group: str,
+ known_labels: Sequence[Any] | np.ndarray,
+ metric: str = "voteFraction",
+ direction: Literal["auto", "higher", "lower"] = "auto",
+ thresholds: Sequence[float] | np.ndarray | None = None,
+ n_thresholds: int = 50,
+ chosen_threshold: float | None = None,
+ query_assay: str | None = None,
+ na_val: str = "NA",
+ max_distance: float | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show: bool = True,
+ ) -> PlotResult:
+ """Plot held-out label accuracy against retained mapping coverage."""
+ from ..plotting import mapping_calibration
+
+ return mapping_calibration(
+ self._store,
+ result,
+ reference=reference,
+ reference_class_group=reference_class_group,
+ known_labels=known_labels,
+ metric=metric,
+ direction=direction,
+ thresholds=thresholds,
+ n_thresholds=n_thresholds,
+ chosen_threshold=chosen_threshold,
+ query_assay=query_assay,
+ na_val=na_val,
+ max_distance=max_distance,
+ target=target,
+ figsize=figsize,
+ theme=theme,
+ show=show,
+ )
+
+ def mapping_projection(
+ self,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None = None,
+ reference_layout_key: str,
+ reference_groups: str | Sequence[Any] | np.ndarray | None = None,
+ target_groups: Sequence[Any] | np.ndarray | None = None,
+ reference_class_group: str | None = None,
+ threshold_fraction: float = 0.5,
+ na_val: str = "NA",
+ query_assay: str | None = None,
+ ref_name: str = "reference",
+ reference_mode: Literal["background", "groups"] = "background",
+ categorical_scale: CategoricalScale | None = None,
+ point_size: float | None = None,
+ reference_alpha: float = 0.22,
+ target_alpha: float = 0.95,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+ ) -> PlotResult:
+ """Plot query cells projected into an unchanged reference layout."""
+ from ..plotting import mapping_projection
+
+ return mapping_projection(
+ self._store,
+ result,
+ reference=reference,
+ reference_layout_key=reference_layout_key,
+ reference_groups=reference_groups,
+ target_groups=target_groups,
+ reference_class_group=reference_class_group,
+ threshold_fraction=threshold_fraction,
+ na_val=na_val,
+ query_assay=query_assay,
+ ref_name=ref_name,
+ reference_mode=reference_mode,
+ categorical_scale=categorical_scale,
+ point_size=point_size,
+ reference_alpha=reference_alpha,
+ target_alpha=target_alpha,
+ target=target,
+ figsize=figsize,
+ theme=theme,
+ show_legend=show_legend,
+ show=show,
+ )
+
+ def dotplot(
+ self,
+ *,
+ features: (
+ "Sequence[str | FeatureRef] | Mapping[str, Sequence[str | FeatureRef]]"
+ ),
+ group_by: str | tuple[str, ...],
+ cell_key: str = "I",
+ from_assay: str | None = None,
+ sample_by: str | None = None,
+ study_design: "StudyDesign | None" = None,
+ normalization: "NormalizationSpec | None" = None,
+ expression_cutoff: float = 0.0,
+ standardize: str = "none",
+ color_scale: "ColorScale | None" = None,
+ size_scale: "SizeScale | None" = None,
+ categorical_scale: "CategoricalScale | None" = None,
+ group_order: Sequence[Any] | None = None,
+ feature_order: Sequence[str] | None = None,
+ swap_axes: bool = False,
+ marker_edgecolor: str | None = None,
+ marker_linewidth: float = 0.3,
+ label_wrap: int | None = None,
+ italicize_features: bool = False,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ max_figure_width: float | None = 7.5,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+ ) -> "PlotResult":
+ """Summarize feature expression as a dot plot."""
+ from ..plotting import dotplot
+
+ return dotplot(
+ self._store,
+ features=features,
+ group_by=group_by,
+ cell_key=cell_key,
+ from_assay=from_assay,
+ sample_by=sample_by,
+ study_design=study_design,
+ normalization=normalization,
+ expression_cutoff=expression_cutoff,
+ standardize=standardize,
+ color_scale=color_scale,
+ size_scale=size_scale,
+ categorical_scale=categorical_scale,
+ group_order=group_order,
+ feature_order=feature_order,
+ swap_axes=swap_axes,
+ marker_edgecolor=marker_edgecolor,
+ marker_linewidth=marker_linewidth,
+ label_wrap=label_wrap,
+ italicize_features=italicize_features,
+ target=target,
+ figsize=figsize,
+ max_figure_width=max_figure_width,
+ theme=theme,
+ show_legend=show_legend,
+ show=show,
+ )
+
+ def matrixplot(
+ self,
+ *,
+ features: (
+ "Sequence[str | FeatureRef] | Mapping[str, Sequence[str | FeatureRef]]"
+ ),
+ group_by: str | tuple[str, ...],
+ cell_key: str = "I",
+ from_assay: str | None = None,
+ sample_by: str | None = None,
+ study_design: "StudyDesign | None" = None,
+ normalization: "NormalizationSpec | None" = None,
+ expression_cutoff: float = 0.0,
+ value: str = "mean",
+ standardize: str = "none",
+ color_scale: "ColorScale | None" = None,
+ feature_order: Sequence[Any] | None = None,
+ group_order: Sequence[Any] | None = None,
+ cluster_features: bool = False,
+ cluster_groups: bool = False,
+ cluster_method: str = "average",
+ cluster_metric: str = "euclidean",
+ row_annotations: (
+ Mapping[str, Mapping[Any, Any] | Sequence[Any]] | None
+ ) = None,
+ column_annotations: (
+ Mapping[str, Mapping[Any, Any] | Sequence[Any]] | None
+ ) = None,
+ annotation_scales: Mapping[str, CategoricalScale] | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+ ) -> "PlotResult":
+ """Summarize feature expression as a matrix plot."""
+ from ..plotting import matrixplot
+
+ return matrixplot(
+ self._store,
+ features=features,
+ group_by=group_by,
+ cell_key=cell_key,
+ from_assay=from_assay,
+ sample_by=sample_by,
+ study_design=study_design,
+ normalization=normalization,
+ expression_cutoff=expression_cutoff,
+ value=value,
+ standardize=standardize,
+ color_scale=color_scale,
+ feature_order=feature_order,
+ group_order=group_order,
+ cluster_features=cluster_features,
+ cluster_groups=cluster_groups,
+ cluster_method=cluster_method,
+ cluster_metric=cluster_metric,
+ row_annotations=row_annotations,
+ column_annotations=column_annotations,
+ annotation_scales=annotation_scales,
+ target=target,
+ figsize=figsize,
+ theme=theme,
+ show_legend=show_legend,
+ show=show,
+ )
+
+ def composition(
+ self,
+ *,
+ category_by: str,
+ cell_key: str = "I",
+ sample_by: str | None = None,
+ subject_by: str | None = None,
+ pair_by: str | None = None,
+ condition_by: str | None = None,
+ study_design: "StudyDesign | None" = None,
+ kind: Literal["stacked", "per_sample"] = "stacked",
+ show_summary: bool = True,
+ uncertainty: Literal["none", "sd", "se", "ci95"] | None = None,
+ categorical_scale: "CategoricalScale | None" = None,
+ bar_width: float = 0.82,
+ bar_gap: float = 0.12,
+ segment_edgecolor: str | None = None,
+ segment_linewidth: float = 0.5,
+ show_percent_labels: bool = False,
+ label_min_fraction: float = 0.08,
+ percent_format: str = "{:.0%}",
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ max_figure_width: float | None = 7.5,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+ ) -> "PlotResult":
+ """Plot category composition for the selected cells."""
+ from ..plotting import composition
+
+ return composition(
+ self._store,
+ category_by=category_by,
+ cell_key=cell_key,
+ sample_by=sample_by,
+ subject_by=subject_by,
+ pair_by=pair_by,
+ condition_by=condition_by,
+ study_design=study_design,
+ kind=kind,
+ show_summary=show_summary,
+ uncertainty=uncertainty,
+ categorical_scale=categorical_scale,
+ bar_width=bar_width,
+ bar_gap=bar_gap,
+ segment_edgecolor=segment_edgecolor,
+ segment_linewidth=segment_linewidth,
+ show_percent_labels=show_percent_labels,
+ label_min_fraction=label_min_fraction,
+ percent_format=percent_format,
+ target=target,
+ figsize=figsize,
+ max_figure_width=max_figure_width,
+ theme=theme,
+ show_legend=show_legend,
+ show=show,
+ )
+
+ def distribution(
+ self,
+ keys: ("str | CellField | FeatureRef | Sequence[str | CellField | FeatureRef]"),
+ *,
+ group_by: str | None = None,
+ groups: Sequence[Any] | None = None,
+ split_by: str | None = None,
+ sample_by: str | None = None,
+ study_design: "StudyDesign | None" = None,
+ sample_stat: Literal["mean", "median", "fraction"] = "mean",
+ expression_cutoff: float = 0.0,
+ subset_by: str | None = None,
+ cell_key: str | None = "I",
+ from_assay: str | None = None,
+ normalization: "NormalizationSpec | None" = None,
+ categorical_scale: "CategoricalScale | None" = None,
+ split_scale: "CategoricalScale | None" = None,
+ kind: "DistKind" = "violin",
+ bins: int = 40,
+ max_points: int = 10000,
+ point_size: float = 0.8,
+ point_alpha: float = 0.28,
+ seed: int = 0,
+ color: str = "steelblue",
+ orientation: Literal["vertical", "horizontal"] = "vertical",
+ row_standardize: bool = False,
+ share_y: bool | None = None,
+ violin_inner: str | None = "quartile",
+ violin_linewidth: float = 0.8,
+ violin_alpha: float = 0.9,
+ italicize_features: bool = False,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ max_figure_width: float | None = 7.5,
+ title: str | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+ ) -> "PlotResult":
+ """Plot distributions of cell metadata or feature values."""
+ from ..plotting import distribution
+
+ return distribution(
+ self._store,
+ keys,
+ group_by=group_by,
+ groups=groups,
+ split_by=split_by,
+ sample_by=sample_by,
+ study_design=study_design,
+ sample_stat=sample_stat,
+ expression_cutoff=expression_cutoff,
+ subset_by=subset_by,
+ cell_key=cell_key,
+ from_assay=from_assay,
+ normalization=normalization,
+ categorical_scale=categorical_scale,
+ split_scale=split_scale,
+ kind=kind,
+ bins=bins,
+ max_points=max_points,
+ point_size=point_size,
+ point_alpha=point_alpha,
+ seed=seed,
+ color=color,
+ orientation=orientation,
+ row_standardize=row_standardize,
+ share_y=share_y,
+ violin_inner=violin_inner,
+ violin_linewidth=violin_linewidth,
+ violin_alpha=violin_alpha,
+ italicize_features=italicize_features,
+ target=target,
+ figsize=figsize,
+ max_figure_width=max_figure_width,
+ title=title,
+ theme=theme,
+ show_legend=show_legend,
+ show=show,
+ )
+
+ def marker_heatmap(
+ self,
+ *,
+ from_assay: str | None = None,
+ group_key: str | None = None,
+ cell_key: str | None = None,
+ topn: int = 5,
+ log_transform: bool | None = None,
+ vmin: float = -1,
+ vmax: float = 2,
+ figsize: tuple[float, float] | None = None,
+ fontsize: float = 10,
+ width_factor: float = 0.03,
+ height_factor: float = 0.02,
+ cmap: Any = "magma_r",
+ color_scale: "ColorScale | None" = None,
+ row_order: Sequence[Any] | None = None,
+ column_order: Sequence[Any] | None = None,
+ cluster_rows: bool = True,
+ cluster_columns: bool = True,
+ cluster_method: str = "ward",
+ cluster_metric: str = "euclidean",
+ row_annotations: (
+ Mapping[str, Mapping[Any, Any] | Sequence[Any]] | None
+ ) = None,
+ column_annotations: (
+ Mapping[str, Mapping[Any, Any] | Sequence[Any]] | None
+ ) = None,
+ annotation_scales: Mapping[str, CategoricalScale] | None = None,
+ target: Any | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+ **heatmap_kwargs: Any,
+ ) -> "PlotResult":
+ """Plot the stored marker table as a heatmap."""
+ from ..plotting import marker_heatmap
+
+ return marker_heatmap(
+ self._store,
+ from_assay=from_assay,
+ group_key=group_key,
+ cell_key=cell_key,
+ topn=topn,
+ log_transform=log_transform,
+ vmin=vmin,
+ vmax=vmax,
+ figsize=figsize,
+ fontsize=fontsize,
+ width_factor=width_factor,
+ height_factor=height_factor,
+ cmap=cmap,
+ color_scale=color_scale,
+ row_order=row_order,
+ column_order=column_order,
+ cluster_rows=cluster_rows,
+ cluster_columns=cluster_columns,
+ cluster_method=cluster_method,
+ cluster_metric=cluster_metric,
+ row_annotations=row_annotations,
+ column_annotations=column_annotations,
+ annotation_scales=annotation_scales,
+ target=target,
+ theme=theme,
+ show_legend=show_legend,
+ show=show,
+ **heatmap_kwargs,
+ )
+
+ def run_recipe(
+ self,
+ recipe: "PlotRecipe | str | Path",
+ *,
+ artifacts: Mapping[str, Any] | None = None,
+ targets: Mapping[str, Any] | None = None,
+ output_dir: str | Path | None = None,
+ show: bool = False,
+ continue_on_error: bool = False,
+ ) -> "PlotRecipeResult":
+ """Run a declarative plotting recipe against this datastore."""
+ from ..plotting import run_recipe
+
+ return run_recipe(
+ self._store,
+ recipe,
+ artifacts=artifacts,
+ targets=targets,
+ output_dir=output_dir,
+ show=show,
+ continue_on_error=continue_on_error,
+ )
+
+ def cluster_connectivity(
+ self,
+ *,
+ group_by: str,
+ layout_key: str,
+ cell_key: str = "I",
+ from_assay: str | None = None,
+ feat_key: str | None = None,
+ graph_loc: str | None = None,
+ position: Literal["median", "mean"] = "median",
+ positions: Mapping[Any, tuple[float, float]] | None = None,
+ categorical_scale: "CategoricalScale | None" = None,
+ size_scale: "SizeScale | None" = None,
+ minimum_edge_weight: float = 0.02,
+ max_edges_per_node: int | None = None,
+ show_cells: bool = False,
+ cell_size: float | None = None,
+ cell_alpha: float = 0.3,
+ cell_color: str | None = None,
+ node_edgecolor: str | None = None,
+ node_linewidth: float = 0.8,
+ edge_color: str | None = None,
+ edge_alpha: float = 0.45,
+ edge_width_range: tuple[float, float] = (0.4, 5.0),
+ labels: bool = True,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show: bool = True,
+ ) -> "PlotResult":
+ """Summarize cell graph connectivity between embedding clusters."""
+ from ..plotting import cluster_connectivity
+
+ return cluster_connectivity(
+ self._store,
+ group_by=group_by,
+ layout_key=layout_key,
+ cell_key=cell_key,
+ from_assay=from_assay,
+ feat_key=feat_key,
+ graph_loc=graph_loc,
+ position=position,
+ positions=positions,
+ categorical_scale=categorical_scale,
+ size_scale=size_scale,
+ minimum_edge_weight=minimum_edge_weight,
+ max_edges_per_node=max_edges_per_node,
+ show_cells=show_cells,
+ cell_size=cell_size,
+ cell_alpha=cell_alpha,
+ cell_color=cell_color,
+ node_edgecolor=node_edgecolor,
+ node_linewidth=node_linewidth,
+ edge_color=edge_color,
+ edge_alpha=edge_alpha,
+ edge_width_range=edge_width_range,
+ labels=labels,
+ target=target,
+ figsize=figsize,
+ theme=theme,
+ show=show,
+ )
+
+ def cluster_tree(
+ self,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ integrated_graph: str | None = None,
+ cluster_key: str | None = None,
+ fill_by_value: str | None = None,
+ force_ints_as_cats: bool = True,
+ width: float = 1,
+ lvr_factor: float = 0.5,
+ vert_gap: float = 0.2,
+ min_node_size: float = 10,
+ node_size_multiplier: float = 10_000.0,
+ node_power: float = 1.2,
+ root_size: float = 100,
+ non_leaf_size: float = 10,
+ show_labels: bool = True,
+ fontsize: float = 10,
+ root_color: str = "#C0C0C0",
+ non_leaf_color: str = "k",
+ cmap: str = "tab20",
+ color_key: dict[Any, str] | None = None,
+ edgecolors: str = "k",
+ edgewidth: float = 1,
+ alpha: float = 0.7,
+ figsize: tuple[float, float] = (5, 5),
+ ax: Any = None,
+ theme: str = "notebook",
+ show: bool = True,
+ ) -> "PlotResult":
+ """Plot a stored hierarchical clustering tree."""
+ from ..plotting import cluster_tree
+
+ return cluster_tree(
+ self._store,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ integrated_graph=integrated_graph,
+ cluster_key=cluster_key,
+ fill_by_value=fill_by_value,
+ force_ints_as_cats=force_ints_as_cats,
+ width=width,
+ lvr_factor=lvr_factor,
+ vert_gap=vert_gap,
+ min_node_size=min_node_size,
+ node_size_multiplier=node_size_multiplier,
+ node_power=node_power,
+ root_size=root_size,
+ non_leaf_size=non_leaf_size,
+ show_labels=show_labels,
+ fontsize=fontsize,
+ root_color=root_color,
+ non_leaf_color=non_leaf_color,
+ cmap=cmap,
+ color_key=color_key,
+ edgecolors=edgecolors,
+ edgewidth=edgewidth,
+ alpha=alpha,
+ figsize=figsize,
+ ax=ax,
+ theme=theme,
+ show=show,
+ )
+
+ def pseudotime_heatmap(
+ self,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ feature_cluster_key: str | None = None,
+ pseudotime_key: str | None = None,
+ show_features: list[str] | None = None,
+ feature_order: Sequence[str] | None = None,
+ feature_cluster_order: Sequence[Any] | None = None,
+ figsize: tuple[float, float] = (5, 10),
+ vmin: float = -2.0,
+ vmax: float = 2.0,
+ heatmap_cmap: str | None = None,
+ pseudotime_cmap: str | None = None,
+ clusterbar_cmap: str | None = None,
+ color_scale: "ColorScale | None" = None,
+ feature_cluster_scale: "CategoricalScale | None" = None,
+ pseudotime_scale: "ColorScale | None" = None,
+ tick_fontsize: int = 10,
+ axis_fontsize: int = 12,
+ feature_label_fontsize: int = 12,
+ target: Mapping[Hashable, Any] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+ ) -> "PlotResult":
+ """Plot feature profiles ordered by stored pseudotime."""
+ from ..plotting import pseudotime_heatmap
+
+ return pseudotime_heatmap(
+ self._store,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ feature_cluster_key=feature_cluster_key,
+ pseudotime_key=pseudotime_key,
+ show_features=show_features,
+ feature_order=feature_order,
+ feature_cluster_order=feature_cluster_order,
+ figsize=figsize,
+ vmin=vmin,
+ vmax=vmax,
+ heatmap_cmap=heatmap_cmap,
+ pseudotime_cmap=pseudotime_cmap,
+ clusterbar_cmap=clusterbar_cmap,
+ color_scale=color_scale,
+ feature_cluster_scale=feature_cluster_scale,
+ pseudotime_scale=pseudotime_scale,
+ tick_fontsize=tick_fontsize,
+ axis_fontsize=axis_fontsize,
+ feature_label_fontsize=feature_label_fontsize,
+ target=target,
+ theme=theme,
+ show_legend=show_legend,
+ show=show,
+ )
diff --git a/scarf/datastore/base_datastore.py b/scarf/datastore/base_datastore.py
index a1088b6f..4b258dd2 100644
--- a/scarf/datastore/base_datastore.py
+++ b/scarf/datastore/base_datastore.py
@@ -1,44 +1,84 @@
-from typing import List, Union, Optional
-
import numpy as np
import zarr
-from loguru import logger
+from collections.abc import Iterable, Mapping, Sequence
+from typing import TYPE_CHECKING, Any, cast
+from ..storage.artifacts import (
+ ArtifactRef,
+ ArtifactScope,
+ ArtifactStatus,
+ ValueFingerprintBuilder,
+ artifact_path,
+ canonical_bytes,
+ fingerprint_array,
+ fingerprint_strings,
+ inspect_artifact,
+ list_artifacts as list_artifact_refs,
+)
+from ..storage.types import ZarrMode, as_zarr_array, as_zarr_group
+from ..storage.budget import ResourceBudget
from ..assay import RNAassay, ATACassay, ADTassay, Assay
+from ..assay.base import _defer_feature_props
from ..metadata import MetaData
-from ..utils import show_dask_progress, controlled_compute, load_zarr, ZARRLOC
+from ..metadata.artifacts import (
+ artifact_values,
+ link_cell_data_column,
+ plan_cell_data_artifact,
+ write_cell_data_artifact,
+)
+from ..storage.schema import validate_assay_name
+from ..storage.profiles import StorageProfile, ZarrLocation
+from ..storage.stores import load_zarr, resolve_matrix_source
+from ..storage.selections import resolve_selection_artifact
+from ..utils.compute import controlled_compute
+from ..utils.logging import logger
+
+if TYPE_CHECKING:
+ from ..graph.state import AssayState
+ from ..lineage import ArtifactLineage
+ from ..mapping.reference import MappingReference
+ from .summary import DataStoreSummary
def sanitize_hierarchy(
- z: zarr.Group, assay_name: str, workspace: Union[str, None]
+ z: zarr.Group,
+ assay_name: str,
+ workspace: str | None,
+ matrix_root: zarr.Group | None = None,
) -> bool:
"""Test if an assay node in zarr object was created properly.
Args:
z: Zarr hierarchy object
assay_name: String value with name of assay.
+ workspace: Workspace name (None for legacy layout without ``matrices/``).
+ matrix_root: Optional root that owns count matrices. Defaults to ``z``.
Returns:
True if assay_name is present in z and contains `counts` and `featureData` child nodes else raises error
"""
+ matrix_root = z if matrix_root is None else matrix_root
if workspace is None:
zw = z
else:
- zw = z[workspace]
- if assay_name in zw:
- if "featureData" not in zw[assay_name]:
- raise KeyError(f"ERROR: 'featureData' not found in {assay_name}")
- else:
+ zw = as_zarr_group(z[workspace], name=workspace)
+ if assay_name not in zw:
raise KeyError(f"ERROR: {assay_name} not found in zarr file")
+ assay_zw = as_zarr_group(zw[assay_name], name=assay_name)
+ if "featureData" not in assay_zw:
+ raise KeyError(f"ERROR: 'featureData' not found in {assay_name}")
if workspace is None:
- if "counts" not in z[assay_name]:
+ matrix_assay = as_zarr_group(matrix_root[assay_name], name=assay_name)
+ if "counts" not in matrix_assay:
raise KeyError(f"ERROR: 'counts' not found in {assay_name}")
else:
- if "matrices" not in z:
- raise KeyError(f"ERROR: Workspace defined but no 'matrices' slot found")
- if assay_name not in z["matrices"]:
+ if "matrices" not in matrix_root:
+ raise KeyError("ERROR: Workspace defined but no 'matrices' slot found")
+ matrices = as_zarr_group(matrix_root["matrices"], name="matrices")
+ if assay_name not in matrices:
raise KeyError(f"ERROR: {assay_name} not found in workspace matrices slot")
- if "counts" not in z["matrices"][assay_name]:
+ matrix_assay = as_zarr_group(matrices[assay_name], name=assay_name)
+ if "counts" not in matrix_assay:
raise KeyError(
f"ERROR: 'counts' not found in {assay_name} in workspace matrices slot"
)
@@ -60,13 +100,12 @@ class BaseDataStore:
will be filtered out.
min_cells_per_feature: Minimum number of cells where a feature has a non-zero value. Genes with values
less than this will be filtered out
- mito_pattern: Regex pattern to capture mitochondrial genes (default: 'MT-')
- ribo_pattern: Regex pattern to capture ribosomal genes (default: 'RPS|RPL|MRPS|MRPL')
+ mito_pattern: Regex pattern to capture mitochondrial genes. When None, uses ``MT-|mt``.
+ ribo_pattern: Regex pattern to capture ribosomal genes. When None, uses
+ ``RPS|RPL|MRPS|MRPL``.
nthreads: Number of maximum threads to use in all multi-threaded functions
zarr_mode: For read-write mode use r+' or for read-only use 'r' (Default value: 'r+')
- synchronizer: Used as `synchronizer` parameter when opening the Zarr file. Please refer to this page for
- more details: https://zarr.readthedocs.io/en/stable/api/sync.html. By default
- ThreadSynchronizer will be used.
+ workspace: Workspace name within the Zarr store (None for legacy single-workspace layout).
Attributes:
cells: MetaData object with cells and info about each cell (e. g. RNA_nCounts ids).
@@ -76,21 +115,53 @@ class BaseDataStore:
def __init__(
self,
- zarr_loc: ZARRLOC,
- assay_types: dict,
+ zarr_loc: ZarrLocation,
+ assay_types: dict[str, str],
default_assay: str,
min_features_per_cell: int,
min_cells_per_feature: int,
mito_pattern: str,
ribo_pattern: str,
- nthreads: int,
- zarr_mode: str,
- workspace: Union[str, None],
- synchronizer,
+ zarr_mode: ZarrMode,
+ workspace: str | None,
+ resources: ResourceBudget,
+ storage_profile: StorageProfile,
+ storage_options: dict[str, Any] | None = None,
):
- self.z = load_zarr(zarr_loc=zarr_loc, mode=zarr_mode, synchronizer=synchronizer)
- self.workspace = workspace
- self.nthreads = nthreads
+ self.zarr_mode = zarr_mode
+ self.zarr_loc = zarr_loc
+ self.z = load_zarr(
+ zarr_loc=zarr_loc,
+ mode=zarr_mode,
+ storage_options=storage_options,
+ )
+ resolved = resolve_matrix_source(
+ self.z,
+ storage_options=storage_options,
+ )
+ if resolved is None:
+ self._matrix_z = None
+ self.workspace = workspace
+ else:
+ self._matrix_z, source_workspace = resolved
+ if workspace is None:
+ self.workspace = source_workspace
+ elif workspace != source_workspace:
+ raise ValueError(
+ "workspace does not match the mounted matrixSource workspace"
+ )
+ else:
+ self.workspace = workspace
+ import_source = self.zw.attrs.get("scarf:import_source")
+ if import_source is not None and not bool(
+ self.zw.attrs.get("scarf:import_complete", False)
+ ):
+ raise RuntimeError(f"{import_source} import is incomplete")
+ self.resources = resources
+ self.nthreads = self.resources.workers
+ self.memoryBytes = self.resources.memoryBytes
+ self.storageProfile = storage_profile
+ _ = self.assay_names
# The order is critical here:
self.cells = self._load_cells()
self._defaultAssay = self._load_default_assay(default_assay)
@@ -105,12 +176,116 @@ def __init__(
@property
def zw(self) -> zarr.Group:
+ """Return the active root or workspace Zarr group."""
if self.workspace is None:
ret_val: zarr.Group = self.z
else:
ret_val: zarr.Group = self.z[self.workspace] # type: ignore
return ret_val
+ def inspect_artifact(self, ref: ArtifactRef) -> ArtifactStatus:
+ """Inspect a logical artifact without mutating the store."""
+ return inspect_artifact(self.zw, ref)
+
+ def lineage(
+ self,
+ target: ArtifactRef | Mapping[str, ArtifactRef],
+ *,
+ references: "MappingReference | Sequence[MappingReference] | None" = None,
+ ) -> "ArtifactLineage":
+ """Build a read-only upstream lineage report for artifact outputs."""
+ from ..lineage import ArtifactLineage
+ from ..mapping.reference import MappingReference
+
+ if references is None:
+ resolved_references: Sequence[MappingReference] = ()
+ elif isinstance(references, MappingReference):
+ resolved_references = (references,)
+ elif isinstance(references, Sequence) and not isinstance(
+ references, str | bytes
+ ):
+ resolved_references = references
+ else:
+ raise TypeError(
+ "references must be a MappingReference, a sequence of "
+ "MappingReference values, or None"
+ )
+
+ external_roots: dict[str, zarr.Group] = {}
+ for index, reference in enumerate(resolved_references):
+ if not isinstance(reference, MappingReference):
+ raise TypeError(f"references[{index}] must be a MappingReference")
+ reference.validate_dataset_fingerprint()
+ fingerprint = reference.external_ref.dataset_fingerprint
+ root = reference.datastore.zw
+ existing = external_roots.get(fingerprint)
+ if existing is not None and str(existing.store_path) != str(
+ root.store_path
+ ):
+ raise ValueError(
+ "References contain duplicate dataset fingerprint "
+ f"{fingerprint!r} for conflicting roots"
+ )
+ external_roots[fingerprint] = root
+
+ return ArtifactLineage.from_store(
+ self.zw,
+ target,
+ external_roots=external_roots,
+ )
+
+ def load_artifact(self, ref: ArtifactRef) -> zarr.Group:
+ """Open a complete artifact through a read-only Zarr group."""
+ status = self.inspect_artifact(ref)
+ if not status.exists:
+ raise KeyError(f"Artifact does not exist: {status.path}")
+ if not status.complete:
+ raise RuntimeError(f"Artifact is incomplete: {status.path}")
+ workspace_path = str(getattr(self.zw, "path", "")).strip("/")
+ store_path = (
+ f"{workspace_path}/{status.path}" if workspace_path else status.path
+ )
+ return zarr.open_group(
+ store=self.zw.store,
+ path=store_path,
+ mode="r",
+ )
+
+ def get_assay_state(self, from_assay: str | None = None) -> "AssayState | None":
+ """Return the selected artifact state for one assay."""
+ from ..graph.state import read_assay_state
+
+ assay = from_assay or self._defaultAssay
+ if assay is None:
+ raise ValueError("No assay was provided and no default is configured")
+ return read_assay_state(self.zw, assay)
+
+ def list_artifacts(
+ self,
+ *,
+ kind: str | None = None,
+ from_assay: str | None = None,
+ scope: ArtifactScope = "assay",
+ complete_only: bool = False,
+ ) -> list[ArtifactRef]:
+ """List logical artifact references in one scope."""
+ if scope == "assay" and from_assay is None:
+ from_assay = self._defaultAssay
+ return list_artifact_refs(
+ self.zw,
+ scope=scope,
+ assay=from_assay,
+ kind=kind,
+ complete_only=complete_only,
+ )
+
+ def summary(self) -> "DataStoreSummary":
+ """Return a read-only, metadata-only summary of this datastore."""
+ from .. import __version__
+ from .summary import build_datastore_summary
+
+ return build_datastore_summary(self, scarf_version=__version__)
+
def _load_cells(self) -> MetaData:
"""This convenience function loads cellData level from the Zarr
hierarchy.
@@ -119,13 +294,13 @@ def _load_cells(self) -> MetaData:
Metadata object
"""
try:
- cell_data: zarr.Group = self.zw["cellData"] # type: ignore
+ cell_data = as_zarr_group(self.zw["cellData"], name="cellData")
except KeyError as e:
raise KeyError(f"cellData not found in zarr file at {self.z.path}") from e
return MetaData(cell_data)
@property
- def assay_names(self) -> List[str]:
+ def assay_names(self) -> list[str]:
"""Load all assay names present in the Zarr file. Zarr writers create
an 'is_assay' attribute in the assay level and this function looks for
presence of those attributes to load assay names.
@@ -134,13 +309,21 @@ def assay_names(self) -> List[str]:
Names of assays present in a Zarr file
"""
assays = []
- for i in self.zw.group_keys():
+ # Object-store listings can repeat a group and may not preserve order
+ # across calls, so keep unique names in sorted order.
+ for i in sorted(dict.fromkeys(self.zw.group_keys())):
if "is_assay" in self.zw[i].attrs.keys():
- sanitize_hierarchy(self.z, i, self.workspace)
+ validate_assay_name(i)
+ sanitize_hierarchy(
+ self.z,
+ i,
+ self.workspace,
+ matrix_root=self._matrix_z,
+ )
assays.append(i)
return assays
- def _load_default_assay(self, assay_name: Optional[str] = None) -> str:
+ def _load_default_assay(self, assay_name: str | None = None) -> str:
"""This function sets a given assay name as defaultAssay attribute. If
`assay_name` value is None then the top-level directory attributes in
the Zarr file are looked up for presence of previously used default
@@ -154,11 +337,12 @@ def _load_default_assay(self, assay_name: Optional[str] = None) -> str:
"""
if assay_name is None:
if "defaultAssay" in self.zw.attrs:
- assay_name = self.zw.attrs["defaultAssay"]
+ assay_name = cast(str, self.zw.attrs["defaultAssay"])
else:
if len(self.assay_names) == 1:
assay_name = self.assay_names[0]
- self.zw.attrs["defaultAssay"] = assay_name
+ if self.zarr_mode == "r+":
+ self.zw.attrs["defaultAssay"] = assay_name
else:
raise ValueError(
"ERROR: You have more than one assay data. "
@@ -172,17 +356,19 @@ def _load_default_assay(self, assay_name: Optional[str] = None) -> str:
logger.info(
f"Default assay changed from {self.zw.attrs['defaultAssay']} to {assay_name}"
)
- self.zw.attrs["defaultAssay"] = assay_name
+ if self.zarr_mode == "r+":
+ self.zw.attrs["defaultAssay"] = assay_name
else:
raise ValueError(
f"ERROR: The provided default assay name: {assay_name} was not found. "
f"Please Choose one from: {' '.join(self.assay_names)}\n"
"Please note that the names are case-sensitive."
)
+ assert assay_name is not None
return assay_name
def _load_assays(
- self, min_cells: int, custom_assay_types: Optional[dict] = None
+ self, min_cells: int, custom_assay_types: dict | None = None
) -> None:
"""This function loads all the assay names present in attribute
`assayNames` as Assay objects. An attempt is made to automatically
@@ -208,6 +394,9 @@ def _load_assays(
"ATAC": ATACassay,
"ADT": ADTassay,
"HTO": ADTassay,
+ "CRISPR": Assay,
+ "ANTIGEN": Assay,
+ "CUSTOM": Assay,
"GeneActivity": RNAassay,
"GeneScores": RNAassay,
"URNA": RNAassay,
@@ -226,7 +415,12 @@ def _load_assays(
)
if "assayTypes" not in self.zw.attrs:
self.zw.attrs["assayTypes"] = {}
- z_attrs = dict(self.zw.attrs["assayTypes"])
+ raw_types = self.zw.attrs["assayTypes"]
+ z_attrs: dict[str, str] = (
+ {str(k): str(v) for k, v in raw_types.items()}
+ if isinstance(raw_types, dict)
+ else {}
+ )
if custom_assay_types is None:
custom_assay_types = {}
for i in self.assay_names:
@@ -263,26 +457,26 @@ def _load_assays(
else:
z_attrs[i] = assay_name
logger.debug(f"Setting assay {i} to assay type: {assay.__name__}")
- setattr(
- self,
- i,
- assay(
+ with _defer_feature_props():
+ loaded_assay = assay(
z=self.z,
workspace=self.workspace,
name=i,
cell_data=self.cells,
min_cells_per_feature=min_cells,
nthreads=self.nthreads,
- ),
- )
+ matrix_root=self._matrix_z,
+ resources=self.resources,
+ )
+ setattr(self, i, loaded_assay)
if self.zw.attrs["assayTypes"] != z_attrs:
self.zw.attrs["assayTypes"] = z_attrs
return None
def _get_assay(
self,
- from_assay: Union[str, None],
- ) -> Union[Assay, RNAassay, ADTassay, ATACassay]:
+ from_assay: str | None,
+ ) -> Assay | RNAassay | ADTassay | ATACassay:
"""This is a convenience function used internally to quickly obtain the
assay object that is linked to an assay name.
@@ -293,7 +487,9 @@ def _get_assay(
"""
if from_assay is None or from_assay == "":
from_assay = self._defaultAssay
- return self.__getattribute__(from_assay)
+ return cast(
+ Assay | RNAassay | ADTassay | ATACassay, self.__getattribute__(from_assay)
+ )
def _get_latest_feat_key(self, from_assay: str) -> str:
"""Looks up the value in assay level attributes for key
@@ -305,8 +501,13 @@ def _get_latest_feat_key(self, from_assay: str) -> str:
Returns:
Name of the latest feature that was used to run `save_normalized_data`
"""
+ from ..graph.state import read_assay_state
+
+ state = read_assay_state(self.zw, from_assay)
+ if state is not None:
+ return state.feat_key
assay = self._get_assay(from_assay)
- return assay.attrs["latest_feat_key"]
+ return cast(str, assay.attrs["latest_feat_key"])
def _get_latest_cell_key(self, from_assay: str) -> str:
"""Looks up the value in assay level attributes for key
@@ -318,14 +519,307 @@ def _get_latest_cell_key(self, from_assay: str) -> str:
Returns:
Name of the latest feature that was used to run `save_normalized_data`
"""
+ from ..graph.state import read_assay_state
+
+ state = read_assay_state(self.zw, from_assay)
+ if state is not None:
+ return state.cell_key
assay = self._get_assay(from_assay)
- return assay.attrs["latest_cell_key"]
+ return cast(str, assay.attrs.get("latest_cell_key", "I"))
+
+ def _ensure_dataset_fingerprint(self, from_assay: str) -> str:
+ assay = self._get_assay(from_assay)
+ existing = assay.attrs.get("dataset_fingerprint")
+ if existing is not None:
+ return str(existing)
+ if self.zarr_mode != "r+":
+ raise PermissionError(
+ "dataset_fingerprint is missing and cannot be stored read-only"
+ )
+ fingerprint = self._calculate_dataset_fingerprint(from_assay)
+ assay.attrs["dataset_fingerprint"] = fingerprint
+ return fingerprint
+
+ def _calculate_dataset_fingerprint(self, from_assay: str) -> str:
+ assay = self._get_assay(from_assay)
+ builder = ValueFingerprintBuilder()
+ builder.update_bytes(
+ "dataset",
+ canonical_bytes(
+ {
+ "assay": from_assay,
+ "shape": list(assay.rawData.shape),
+ "dtype": np.dtype(assay.rawData.dtype).str,
+ }
+ ),
+ )
+ builder.update_array(
+ "cell_ids",
+ np.asarray(self.cells.fetch_all("ids")).astype(str),
+ )
+ builder.update_array(
+ "feature_ids",
+ np.asarray(assay.feats.fetch_all("ids")).astype(str),
+ )
+ builder.update_array(
+ "cell_n_counts",
+ np.asarray(self.cells.fetch_all(f"{from_assay}_nCounts")),
+ )
+ builder.update_array(
+ "cell_n_features",
+ np.asarray(self.cells.fetch_all(f"{from_assay}_nFeatures")),
+ )
+ builder.update_array(
+ "feature_n_cells",
+ np.asarray(assay.feats.fetch_all("nCells")),
+ )
+ return builder.hexdigest()
+
+ def _record_cell_selection(
+ self,
+ *,
+ column: str,
+ operation: str,
+ parameters: dict[str, Any],
+ inputs: dict[str, Any],
+ invalidate_cache: bool = False,
+ ) -> ArtifactRef:
+ ref = resolve_selection_artifact(
+ self.zw,
+ scope="datastore",
+ kind="cell_selection",
+ values=np.asarray(self.cells.fetch_all(column)),
+ row_ids=np.asarray(self.cells.fetch_all("ids")),
+ operation=operation,
+ parameters=parameters,
+ inputs=inputs,
+ source_column=column,
+ invalidate_cache=invalidate_cache,
+ )
+ column_array = as_zarr_array(
+ as_zarr_group(self.zw["cellData"], name="cellData")[column],
+ name=column,
+ )
+ column_array.attrs["source_artifact"] = ref.to_dict()
+ column_array.attrs["source_value"] = "values"
+ return ref
+
+ def _linked_cell_selection(self, column: str) -> ArtifactRef | None:
+ cell_data = as_zarr_group(self.zw["cellData"], name="cellData")
+ column_array = as_zarr_array(cell_data[column], name=column)
+ raw_ref = column_array.attrs.get("source_artifact")
+ if not isinstance(raw_ref, dict):
+ return None
+ try:
+ ref = ArtifactRef.from_dict(raw_ref)
+ status = self.inspect_artifact(ref)
+ except (KeyError, TypeError, ValueError):
+ return None
+ if (
+ ref.kind != "cell_selection"
+ or ref.scope != "datastore"
+ or not status.complete
+ ):
+ return None
+ inputs = status.inputs or {}
+ if inputs.get("ordered_row_ids_fingerprint") != fingerprint_strings(
+ np.asarray(self.cells.fetch_all("ids"))
+ ):
+ return None
+ group = as_zarr_group(self.zw[status.path], name=status.path)
+ if "values" not in group:
+ return None
+ stored = np.asarray(as_zarr_array(group["values"], name="values")[:])
+ current = np.asarray(self.cells.fetch_all(column))
+ return (
+ ref
+ if stored.ndim == 1
+ and stored.dtype == np.dtype(bool)
+ and current.ndim == 1
+ and current.dtype == np.dtype(bool)
+ and stored.shape == current.shape
+ and np.array_equal(stored, current)
+ else None
+ )
+
+ def _ensure_cell_selection(self, column: str) -> ArtifactRef:
+ ref = self._linked_cell_selection(column)
+ if ref is not None:
+ return ref
+ return self._record_cell_selection(
+ column=column,
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ )
+
+ def _selection_artifacts_match(
+ self,
+ first: ArtifactRef,
+ second: ArtifactRef,
+ ) -> bool:
+ if first == second:
+ return True
+ if (
+ first.kind != second.kind
+ or first.scope != second.scope
+ or first.assay != second.assay
+ ):
+ return False
+ try:
+ first_status = inspect_artifact(self.zw, first)
+ second_status = inspect_artifact(self.zw, second)
+ if (
+ not first_status.complete
+ or not second_status.complete
+ or (first_status.inputs or {}).get("ordered_row_ids_fingerprint")
+ != (second_status.inputs or {}).get("ordered_row_ids_fingerprint")
+ ):
+ return False
+ first_group = as_zarr_group(
+ self.zw[artifact_path(first)],
+ name=artifact_path(first),
+ )
+ second_group = as_zarr_group(
+ self.zw[artifact_path(second)],
+ name=artifact_path(second),
+ )
+ first_values = np.asarray(
+ as_zarr_array(
+ first_group["values"],
+ name="values",
+ )[:]
+ )
+ second_values = np.asarray(
+ as_zarr_array(
+ second_group["values"],
+ name="values",
+ )[:]
+ )
+ except (KeyError, TypeError, ValueError):
+ return False
+ return (
+ first_values.ndim == 1
+ and second_values.ndim == 1
+ and first_values.dtype == np.dtype(bool)
+ and second_values.dtype == np.dtype(bool)
+ and first_values.shape == second_values.shape
+ and np.array_equal(first_values, second_values)
+ )
+
+ def _resolve_cell_data_input(
+ self,
+ column: str,
+ *,
+ cell_key: str,
+ ) -> ArtifactRef:
+ selection = self._ensure_cell_selection(cell_key)
+ current = np.asarray(self.cells.fetch(column, key=cell_key))
+ cell_data = as_zarr_group(self.zw["cellData"], name="cellData")
+ column_array = as_zarr_array(cell_data[column], name=column)
+ raw_ref = column_array.attrs.get("source_artifact")
+ if isinstance(raw_ref, dict):
+ try:
+ ref = ArtifactRef.from_dict(raw_ref)
+ status = self.inspect_artifact(ref)
+ except (KeyError, TypeError, ValueError):
+ pass
+ else:
+ inputs = status.inputs or {}
+ source_value = column_array.attrs.get(
+ "source_value",
+ "values",
+ )
+ value_index = column_array.attrs.get("value_index")
+ if (
+ status.complete
+ and inputs.get("cell_selection") == selection.to_dict()
+ and isinstance(source_value, str)
+ ):
+ group = as_zarr_group(
+ self.zw[status.path],
+ name=status.path,
+ )
+ if source_value in group:
+ stored = artifact_values(
+ group,
+ source_value,
+ (
+ int(value_index)
+ if isinstance(value_index, int)
+ else None
+ ),
+ )
+ if np.array_equal(stored, current):
+ return ref
+ values_fingerprint = (
+ fingerprint_strings(current)
+ if current.dtype.kind in {"O", "S", "U"}
+ else fingerprint_array(current)
+ )
+ planned = plan_cell_data_artifact(
+ self.zw,
+ scope="datastore",
+ kind="metadata_snapshot",
+ operation="manual_cell_data",
+ parameters={"dtype": str(current.dtype)},
+ inputs={"values_fingerprint": values_fingerprint},
+ execution_options={"source_column": column},
+ cell_selection=selection,
+ arrays={
+ "values": (
+ current.shape,
+ (
+ None
+ if current.dtype.kind in {"O", "S", "U"}
+ else current.dtype.kind
+ ),
+ )
+ },
+ )
+ write_cell_data_artifact(
+ self.zw,
+ planned,
+ {"values": current},
+ )
+ link_cell_data_column(
+ self.zw,
+ column,
+ planned.ref,
+ value_name="values",
+ )
+ return planned.ref
+
+ def _resolve_cell_data_provenance_input(
+ self,
+ column: str,
+ *,
+ cell_key: str,
+ ) -> dict[str, Any]:
+ ref = self._resolve_cell_data_input(column, cell_key=cell_key)
+ column_array = as_zarr_array(
+ as_zarr_group(self.zw["cellData"], name="cellData")[column],
+ name=column,
+ )
+ source_value = column_array.attrs.get("source_value", "values")
+ if not isinstance(source_value, str):
+ raise TypeError("Cell-data source_value must be a string")
+ raw_index = column_array.attrs.get("value_index")
+ if raw_index is not None and (
+ isinstance(raw_index, bool) or not isinstance(raw_index, int | np.integer)
+ ):
+ raise TypeError("Cell-data value_index must be an integer")
+ return {
+ "artifact": ref.to_dict(),
+ "source_value": source_value,
+ "value_index": int(raw_index) if raw_index is not None else None,
+ }
def _ini_cell_props(
self,
min_features: int,
- mito_pattern: Optional[str],
- ribo_pattern: Optional[str],
+ mito_pattern: str | None,
+ ribo_pattern: str | None,
) -> None:
"""This function is called on class initialization. For each assay, it
calculates per-cell statistics i.e. nCounts, nFeatures, percentMito and
@@ -342,53 +836,98 @@ def _ini_cell_props(
for from_assay in self.assay_names:
assay = self._get_assay(from_assay)
- var_name = from_assay + "_nCounts"
- if var_name not in self.cells.columns:
- n_c = show_dask_progress(
- assay.rawData.sum(axis=1),
- f"({from_assay}) Computing nCounts",
- self.nthreads,
+ n_counts_name = from_assay + "_nCounts"
+ compute_n_counts = n_counts_name not in self.cells.columns
+ n_features_name = from_assay + "_nFeatures"
+ compute_n_features = n_features_name not in self.cells.columns
+ pending_min_cells = assay._deferred_min_cells_per_feature
+ compute_n_cells = pending_min_cells is not None
+
+ percent_feature_indices: dict[str, np.ndarray] = {}
+ if isinstance(assay, RNAassay):
+ if mito_pattern != "":
+ resolved_mito_pattern = (
+ "MT-|mt" if mito_pattern is None else mito_pattern
+ )
+ percent_mito_name = from_assay + "_percentMito"
+ mito_idx = assay._plan_percent_feature(
+ resolved_mito_pattern,
+ percent_mito_name,
+ )
+ if mito_idx is not None:
+ percent_feature_indices[percent_mito_name] = mito_idx
+
+ if ribo_pattern != "":
+ resolved_ribo_pattern = (
+ "RPS|RPL|MRPS|MRPL" if ribo_pattern is None else ribo_pattern
+ )
+ percent_ribo_name = from_assay + "_percentRibo"
+ ribo_idx = assay._plan_percent_feature(
+ resolved_ribo_pattern,
+ percent_ribo_name,
+ )
+ if ribo_idx is not None:
+ percent_feature_indices[percent_ribo_name] = ribo_idx
+
+ stats: dict[str, np.ndarray] = {}
+ if (
+ compute_n_counts
+ or compute_n_features
+ or compute_n_cells
+ or percent_feature_indices
+ ):
+ stats = assay._stream_initialization_stats(
+ compute_n_counts=compute_n_counts,
+ compute_n_features=compute_n_features,
+ compute_n_cells=compute_n_cells,
+ percent_feature_indices=percent_feature_indices,
+ )
+
+ if pending_min_cells is not None:
+ assay._store_feature_props(stats["nCells"], pending_min_cells)
+
+ computed_n_counts: np.ndarray | None = None
+ if compute_n_counts:
+ n_c = stats["nCounts"]
+ computed_n_counts = n_c.astype(np.float64)
+ self.cells.insert(
+ n_counts_name,
+ computed_n_counts,
+ overwrite=True,
)
- self.cells.insert(var_name, n_c.astype(np.float64), overwrite=True)
- if type(assay) == RNAassay:
+ if isinstance(assay, RNAassay):
min_nc = min(n_c)
if min(n_c) < assay.sf:
logger.warning(
f"Minimum cell count ({min_nc}) is lower than "
f"size factor multiplier ({assay.sf})"
)
- var_name = from_assay + "_nFeatures"
- if var_name not in self.cells.columns:
- n_f = show_dask_progress(
- (assay.rawData > 0).sum(axis=1),
- f"({from_assay}) Computing nFeatures",
- self.nthreads,
+
+ if compute_n_features:
+ self.cells.insert(
+ n_features_name,
+ stats["nFeatures"].astype(np.float64),
+ overwrite=True,
)
- self.cells.insert(var_name, n_f.astype(np.float64), overwrite=True)
- if type(assay) == RNAassay:
- if mito_pattern == "":
- pass
- else:
- if mito_pattern is None:
- mito_pattern = "MT-|mt"
- var_name = from_assay + "_percentMito"
- assay.add_percent_feature(mito_pattern, var_name)
+ for name in percent_feature_indices:
+ assay._write_percent_feature(
+ name,
+ stats[name],
+ n_counts=computed_n_counts,
+ )
- if ribo_pattern == "":
- pass
- else:
- if ribo_pattern is None:
- ribo_pattern = "RPS|RPL|MRPS|MRPL"
- var_name = from_assay + "_percentRibo"
- assay.add_percent_feature(ribo_pattern, var_name)
+ if assay._deferred_min_cells_per_feature is not None:
+ raise RuntimeError(
+ f"({from_assay}) Deferred feature initialization was not completed"
+ )
if from_assay == self._defaultAssay:
v = self.cells.fetch(from_assay + "_nFeatures", key="I")
if min_features > np.median(v):
logger.warning(
- f"More than of half of the less have less than {min_features} features for assay: "
- f"{from_assay}. Will not remove low quality cells automatically."
+ f"More than half of the cells have fewer than {min_features} features "
+ f"for assay: {from_assay}. Will not remove low quality cells automatically."
)
else:
bv = self.cells.sift(
@@ -432,7 +971,9 @@ def set_default_assay(self, assay_name: str) -> None:
"""
if assay_name not in self.assay_names:
available = ", ".join(self.assay_names)
- raise ValueError(f"Assay '{assay_name}' not found. Available assays: {available}")
+ raise ValueError(
+ f"Assay '{assay_name}' not found. Available assays: {available}"
+ )
self._defaultAssay = assay_name
self.zw.attrs["defaultAssay"] = assay_name
@@ -442,8 +983,6 @@ def get_cell_vals(
cell_key: str,
k: str,
clip_fraction: float = 0,
- use_precached: bool = True,
- cache_key: str = "prenormed",
) -> np.ndarray:
"""Fetches data from the Zarr file.
@@ -451,14 +990,11 @@ def get_cell_vals(
given feature from normalized matrix.
Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: One of the columns from cell metadata table that indicates the cells to be used. The values in
- the chosen column should be boolean (Default value: 'I')
- k: A cell metadata column or name of a feature.
- clip_fraction: This value is multiplied by 100 and the percentiles are soft-clipped from either end.
- (Default value: 0)
- use_precached: Whether to use pre-calculated values from 'prenormed' slot. Used only if 'prenormed' is
- present (Default value: True)
+ from_assay: Name of assay to be used.
+ cell_key: Boolean column in cell metadata selecting cells. Required; pass ``'I'``
+ for the default active-cell key.
+ k: Cell metadata column name or feature name whose values are fetched.
+ clip_fraction: Fraction (0-1) for soft percentile clipping of numeric values.
Returns:
The requested values
@@ -474,28 +1010,9 @@ def get_cell_vals(
logger.warning(
f"Plotting mean of {len(feat_idx)} features because {k} is not unique."
)
- vals = None
- if use_precached and cache_key in assay.z:
- g = assay.z[cache_key]
- vals = np.zeros(assay.cells.N)
- n_feats = 0
- for i in feat_idx:
- if i in g:
- vals += assay.z[cache_key][i][:]
- n_feats += 1
- if n_feats == 0:
- logger.debug(f"Could not find prenormed values for feat: {k}")
- vals = None
- elif n_feats > 1:
- vals = vals / n_feats
- else:
- pass
- if vals is not None:
- vals = vals[cell_idx]
- if vals is None:
- vals = controlled_compute(
- assay.normed(cell_idx, feat_idx).mean(axis=1), self.nthreads
- ).astype(np.float64)
+ vals = controlled_compute(
+ assay.normed(cell_idx, feat_idx).mean(axis=1), self.nthreads
+ ).astype(np.float64)
else:
vals = self.cells.fetch(k, key=cell_key)
if clip_fraction < 0 or clip_fraction > 1:
@@ -510,8 +1027,8 @@ def get_cell_vals(
vals[vals > max_v] = max_v
return vals
- def __repr__(self):
- def formatter(label, iter_vals):
+ def __repr__(self) -> str:
+ def formatter(label: str | None, iter_vals: Iterable[str]) -> str:
if label is None:
line = ""
else:
@@ -545,11 +1062,15 @@ def formatter(label, iter_vals):
f"features and following metadata:"
)
res += formatter(None, assay.feats.columns)
- if "projections" in self.zw[i]:
- targets = []
- layouts = []
- for j in self.zw[i]["projections"]:
- if isinstance(self.zw[i]["projections"][j], zarr.Group): # type: ignore
+ assay_group = as_zarr_group(self.zw[i], name=i)
+ if "projections" in assay_group:
+ targets: list[str] = []
+ layouts: list[str] = []
+ projections = as_zarr_group(
+ assay_group["projections"], name="projections"
+ )
+ for j in projections:
+ if isinstance(projections[j], zarr.Group):
targets.append(j)
else:
layouts.append(j)
diff --git a/scarf/datastore/datastore.py b/scarf/datastore/datastore.py
index c0b4b55d..37608316 100644
--- a/scarf/datastore/datastore.py
+++ b/scarf/datastore/datastore.py
@@ -1,27 +1,76 @@
-from typing import Iterable, List, Literal, Optional, Tuple, Union
+from typing import TYPE_CHECKING, Any, cast
+
+from ..storage.types import ZarrMode
+from ..assay import Assay
+from ..storage.profiles import StorageProfile, ZarrLocation
+from ..storage.stores import create_matrix_source
+from ._operations.features import _FeatureOperationsMixin
+from ._operations.integration_metrics import _IntegrationMetricsOperationsMixin
+from ._operations.presentation import _PresentationOperationsMixin
+from ._operations.quality_control import _QualityControlOperationsMixin
+from ._operations.trajectory import _TrajectoryFeatureOperationsMixin
+from .mapping_datastore import MappingDatastore
-import numpy as np
-import pandas as pd
-from dask import array as daskarr
-from loguru import logger
+if TYPE_CHECKING:
+ from .pipeline_accessor import PipelineAccessor
+ from .plot_accessor import DataStorePlotAccessor
-from ..assay import Assay, ATACassay, RNAassay
-from ..feat_utils import hto_demux
-from ..utils import ZARRLOC, controlled_compute, tqdmbar
-from ..writers import create_zarr_dataset, create_zarr_obj_array
-from .mapping_datastore import MappingDatastore
+__all__ = ["DataStore", "mount_datastore"]
-__all__ = ["DataStore"]
+def mount_datastore(
+ source: str,
+ at: ZarrLocation,
+ *,
+ workspace: str | None = None,
+ storage_options: dict[str, Any] | None = None,
+ **datastore_options: Any,
+) -> "DataStore":
+ """Create a writable DataStore whose count matrices come from ``source``.
-class DataStore(MappingDatastore):
+ The target store receives copied cell and feature metadata plus all new
+ analysis artifacts. Count matrices remain in the read-only source and are
+ remounted automatically when the target is reopened with ``DataStore``.
+ """
+ if "zarr_loc" in datastore_options:
+ raise TypeError("mount_datastore takes the target location through 'at'")
+ zarr_mode = datastore_options.pop("zarr_mode", "r+")
+ if zarr_mode != "r+":
+ raise ValueError(
+ f"A mounted datastore is writable and needs zarr_mode 'r+', got "
+ f"{zarr_mode!r}. Reopen the target with DataStore to read it."
+ )
+
+ create_matrix_source(
+ source,
+ at,
+ workspace=workspace,
+ storage_options=storage_options,
+ )
+ return DataStore(
+ at,
+ zarr_mode=zarr_mode,
+ workspace=workspace,
+ storage_options=storage_options,
+ **datastore_options,
+ )
+
+
+class DataStore(
+ _QualityControlOperationsMixin,
+ _FeatureOperationsMixin,
+ _TrajectoryFeatureOperationsMixin,
+ _IntegrationMetricsOperationsMixin,
+ _PresentationOperationsMixin,
+ MappingDatastore,
+):
"""This class extends MappingDatastore and consequently inherits methods of
all the other DataStore classes.
- This class is the main user facing class as it provides most of the plotting functions.
- It also contains methods for cell filtering, feature selection, marker features identification,
- subsetting and aggregating cells. This class also contains methods that perform in-memory data exports.
- In other words, DataStore objects provide the primary interface to interact with the data.
+ DataStore is the primary interface for filtering cells, selecting features,
+ building graphs, mapping datasets, finding markers, aggregating cells, and
+ exporting data. Store-backed plots are available through `DataStore.plots`;
+ the same functions remain available through `scarf.plotting`.
Args:
zarr_loc: Path to Zarr file created using one of writer functions of Scarf.
@@ -33,30 +82,39 @@ class DataStore(MappingDatastore):
will be filtered out.
min_cells_per_feature: Minimum number of cells where a feature has a non-zero value. Genes with values
less than this will be filtered out.
- mito_pattern: Regex pattern to capture mitochondrial genes. (default: 'MT-')
- ribo_pattern: Regex pattern to capture ribosomal genes. (default: 'RPS|RPL|MRPS|MRPL')
+ mito_pattern: Regex pattern to capture mitochondrial genes. When None, uses ``MT-|mt``.
+ ribo_pattern: Regex pattern to capture ribosomal genes. When None, uses
+ ``RPS|RPL|MRPS|MRPL``.
nthreads: Number of maximum threads to use in all multi-threaded functions
zarr_mode: For read-write mode use r+' or for read-only use 'r'. (Default value: 'r+')
workspace: Workspace for the data
- synchronizer: Used as `synchronizer` parameter when opening the Zarr file. Please refer to this page for
- more details: https://zarr.readthedocs.io/en/stable/api/sync.html. By default
- ThreadSynchronizer will be used.
+ mem_budget: Memory budget bounding streaming and concurrency. Accepts bytes, a suffixed size
+ (e.g. '8G'), or a fraction of total system memory (e.g. '0.6'). When None, it is
+ auto-detected (SCARF_MEM_BUDGET env var, else total system memory). Override it to
+ simulate reading on a machine with a different memory size than the writer.
"""
def __init__(
self,
- zarr_loc: ZARRLOC,
- assay_types: Optional[dict] = None,
- default_assay: Optional[str] = None,
+ zarr_loc: ZarrLocation,
+ assay_types: dict[str, str] | None = None,
+ default_assay: str | None = None,
min_features_per_cell: int = 10,
min_cells_per_feature: int = 20,
- mito_pattern: Optional[str] = None,
- ribo_pattern: Optional[str] = None,
+ mito_pattern: str | None = None,
+ ribo_pattern: str | None = None,
nthreads: int = 2,
- zarr_mode: str = "r+",
- workspace: Union[str, None] = None,
- synchronizer=None,
- ):
+ zarr_mode: ZarrMode = "r+",
+ workspace: str | None = None,
+ zarrProfile: StorageProfile | None = None,
+ storage_options: dict[str, Any] | None = None,
+ mem_budget: int | str | None = None,
+ ) -> None:
+ from ..storage.budget import resolve_budget
+ from ..storage.profiles import resolve_storage_profile
+
+ resources = resolve_budget(memory=mem_budget, workers=nthreads)
+ profile = resolve_storage_profile(zarr_loc, zarrProfile)
if zarr_mode not in ["r", "r+"]:
raise ValueError(
"ERROR: Zarr file can only be accessed using either 'r' or 'r+' mode"
@@ -69,12 +127,27 @@ def __init__(
min_cells_per_feature=min_cells_per_feature,
mito_pattern=mito_pattern,
ribo_pattern=ribo_pattern,
- nthreads=nthreads,
zarr_mode=zarr_mode,
workspace=workspace,
- synchronizer=synchronizer,
+ resources=resources,
+ storage_profile=profile,
+ storage_options=storage_options,
)
+ @property
+ def plots(self) -> "DataStorePlotAccessor":
+ """Return store-bound equivalents of store-first plotting functions."""
+ from .plot_accessor import DataStorePlotAccessor
+
+ return DataStorePlotAccessor(self)
+
+ @property
+ def pipeline(self) -> "PipelineAccessor":
+ """Return the store-bound analysis recipe runner."""
+ from .pipeline_accessor import PipelineAccessor
+
+ return PipelineAccessor(self)
+
def get_assay(self, assay_name: str) -> Assay:
"""Returns the assay object for the given assay name.
@@ -87,2174 +160,20 @@ def get_assay(self, assay_name: str) -> Assay:
if assay_name not in self.assay_names:
raise ValueError(f"ERROR: Assay {assay_name} not found in the Zarr file")
else:
- return getattr(self, assay_name)
-
- def filter_cells(
- self,
- attrs: Iterable[str],
- lows: Iterable[int],
- highs: Iterable[int],
- reset_previous: bool = False,
- keep_bounds: bool = False,
- ) -> None:
- """Filter cells based on the cell metadata column values. Filtering
- triggers `update` method on 'I' column of cell metadata which uses
- 'and' operation. This means that cells that are not within the
- filtering thresholds will have value set as False in 'I' column of cell
- metadata table. When performing filtering repeatedly, the cells that
- were previously filtered out remain filtered out and 'I' column is
- updated only for those cells that are filtered out due to the latest
- filtering attempt.
-
- Args:
- attrs: Names of columns to be used for filtering
- lows: Lower bounds of thresholds for filtering. Should be in same order as the names in `attrs` parameter
- highs: Upper bounds of thresholds for filtering. Should be in same order as the names in `attrs` parameter
- reset_previous: If True, then results of previous filtering will be undone completely.
- (Default value: False)
- keep_bounds: If True, then the boundary values are retained and not filtered out (Default value: False)
-
- Returns:
- """
- new_bool = np.ones(self.cells.N).astype(bool)
- for i, j, k in zip(attrs, lows, highs):
- # Checking here to avoid hard error from metadata class
- if i not in self.cells.columns:
- logger.warning(
- f"{i} not found in cell metadata. Will ignore {i} for filtering"
- )
- continue
- if j is None:
- j = -np.inf
- if k is None:
- k = np.inf
- x = self.cells.sift(i, j, k, keep_bounds=keep_bounds)
- logger.info(
- f"{len(x) - x.sum()} cells flagged for filtering out using attribute {i}"
- )
- new_bool = new_bool & x
- if reset_previous:
- self.cells.reset_key(key="I")
- self.cells.update_key(new_bool, key="I")
-
- def auto_filter_cells(
- self,
- attrs: Optional[Iterable[str]] = None,
- min_p: float = 0.01,
- max_p: float = 0.99,
- show_qc_plots: bool = True,
- ) -> None:
- """Automatically filter cells based on columns of the cell metadata
- table.
-
- This is a wrapper function for `filer_cells` and determines the threshold values to be used for each column.
- For each cell metadata column, the function models a normal distribution using the median value and standard
- deviation of the column and then determines the point estimates of values at `min_p` and `max_p`
- fraction of densities.
-
- Args:
- attrs: Column names to be used for filtering.
- min_p: Fractional density point to be used for calculating lower bounds of threshold.
- max_p: Fractional density point to be used for calculating lower bounds of threshold.
- show_qc_plots: If True then violin plots with per cell distribution of features will be shown. This does
- not have an effect if `auto_filter` is False.
-
- Returns:
- None
- """
- from scipy.stats import norm
-
- if attrs is None:
- attrs = []
- for i in ["nCounts", "nFeatures", "percentMito", "percentRibo"]:
- i = f"{self._defaultAssay}_{i}"
- if i in self.cells.columns:
- attrs.append(i)
-
- attrs_used = []
- for i in attrs:
- if i not in self.cells.columns:
- logger.warning(
- f"{i} not found in cell metadata. Will ignore {i} for filtering"
- )
- continue
- a = self.cells.fetch_all(i)
- dist = norm(np.median(a), np.std(a))
- self.filter_cells(
- attrs=[i], lows=[dist.ppf(min_p)], highs=[dist.ppf(max_p)]
- )
- attrs_used.append(i)
-
- if show_qc_plots:
- self.plot_cells_dists(
- cols=attrs_used, sup_title="Pre-filtering distribution"
- )
- self.plot_cells_dists(
- cols=attrs_used,
- cell_key="I",
- color="coral",
- sup_title="Post-filtering distribution",
- )
-
- def mark_hto_identities(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- label: str = "Hashtag_identity",
- ) -> None:
- if from_assay is None:
- from_assay = "HTO"
- if cell_key is None:
- cell_key = self._get_latest_cell_key(from_assay)
- assay = self._get_assay(from_assay)
- counts = controlled_compute(
- assay.rawData[self.cells.fetch_all(cell_key)], self.nthreads
- )
- hto_idents = hto_demux(
- pd.DataFrame(counts, columns=assay.feats.fetch_all("ids"))
- )
- self.cells.insert(
- column_name=label,
- values=np.array(hto_idents.values),
- overwrite=True,
- key=cell_key,
- )
-
- def mark_hvgs(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- min_cells: Optional[int] = None,
- top_n: int = 500,
- min_var: float = -np.inf,
- max_var: float = np.inf,
- min_mean: float = -np.inf,
- max_mean: float = np.inf,
- n_bins: int = 200,
- lowess_frac: float = 0.1,
- blacklist: str = "^MT-|^RPS|^RPL|^MRPS|^MRPL|^CCN|^HLA-|^H2-|^HIST",
- keep_bounds: bool = False,
- show_plot: bool = True,
- hvg_key_name: str = "hvgs",
- max_cells: int | None = np.inf,
- **plot_kwargs,
- ) -> None:
- """Identify and mark genes as highly variable genes (HVGs). This is a
- critical and required feature selection step and is only applicable to
- RNAassay type of assays.
-
- Args:
- from_assay: Assay to use for graph creation. If no value is provided then `defaultAssay` will be used
- cell_key: Cells to use for HVG selection. By default, all cells with True value in 'I' will be used.
- The provided value for `cell_key` should be a column in cell metadata table with boolean values.
- min_cells: Minimum number of cells where a gene should have non-zero expression values for it to be
- considered a candidate for HVG selection. Large values for this parameter might make it difficult
- to identify rare populations of cells. Very small values might lead to a higher signal-to-noise
- ratio in the selected features. By default, a value is set assuming smallest population has no
- less than 1% of all cells. So for example, if you have 1000 cells (as per cell_key parameter)
- then `min-cells` will be set to 10.
- max_cells: Maximum number of cells where a gene should have non-zero expression values for it to be
- considered a candidate for HVG selection. This can be useful to filter out genes that are
- expressed in too many cells. Default value is infinity, meaning no upper limit.
- top_n: Number of top most variable genes to be set as HVGs. This value is ignored if a value is provided
- for `min_var` parameter. (Default: 500)
- min_var: Minimum variance threshold for HVG selection. (Default: -Infinity)
- max_var: Maximum variance threshold for HVG selection. (Default: Infinity)
- min_mean: Minimum mean value of expression threshold for HVG selection. (Default: -Infinity)
- max_mean: Maximum mean value of expression threshold for HVG selection. (Default: Infinity)
- n_bins: Number of bins into which the mean expression is binned. (Default: 200)
- lowess_frac: Between 0 and 1. The fraction of the data used when estimating the fit between mean and
- variance. This is same as `frac` in statsmodels.nonparametric.smoothers_lowess.lowess
- (Default: 0.1)
- blacklist: This is a regular expression (regex) string that can be used to exclude genes from being marked
- as HVGs. By default, we exclude mitochondrial, ribosomal, some cell-cycle related, histone and
- HLA genes. (Default: '^MT- | ^RPS | ^RPL | ^MRPS | ^MRPL | ^CCN | ^HLA- | ^H2- | ^HIST' )
- keep_bounds: If True, then the boundary values are retained and not filtered out (Default value: False)
- show_plot: If True then a diagnostic scatter plot is shown with HVGs highlighted. (Default: True)
- hvg_key_name: Base label for HVGs in the features metadata column. The value for
- 'cell_key' parameter is prepended to this value. (Default value: 'hvgs')
- plot_kwargs: These named parameters are passed to plotting.plot_mean_var
-
- Returns:
- None
- """
-
- if cell_key is None:
- cell_key = "I"
- assay = self._get_assay(from_assay)
- if type(assay) != RNAassay: # noqa: E721
- raise TypeError(
- f"ERROR: This method of feature selection can only be applied to RNAassay type of assay. "
- f"The provided assay is {type(assay)} type"
- )
- if min_cells is None:
- min_cells = int(0.01 * self.cells.N)
- logger.info(
- f"Setting `min_cells` to {min_cells}. Only those genes that are present in atleast this number "
- f"of cells will be considered HVGs."
- )
- if max_cells is None:
- max_cells = np.inf
- assay.mark_hvgs(
- cell_key=cell_key,
- min_cells=min_cells,
- max_cells=max_cells,
- top_n=top_n,
- min_var=min_var,
- max_var=max_var,
- min_mean=min_mean,
- max_mean=max_mean,
- n_bins=n_bins,
- lowess_frac=lowess_frac,
- blacklist=blacklist,
- hvg_key_name=hvg_key_name,
- keep_bounds=keep_bounds,
- show_plot=show_plot,
- **plot_kwargs,
- )
-
- def mark_prevalent_peaks(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- top_n: int = 10000,
- prevalence_key_name: str = "prevalent_peaks",
- ) -> None:
- """Feature selection method for ATACassay type assays.
-
- This method first calculates prevalence of each peak by computing sum of TF-IDF normalized values for each peak
- and then marks `top_n` peaks with the highest prevalence as prevalent peaks.
-
- Args:
- from_assay: Assay to use for graph creation. If no value is provided then `defaultAssay` will be used
- cell_key: Cells to use for selection of most prevalent peaks. By default, all cells with True value in
- 'I' will be used. The provided value for `cell_key` should be a column in cell metadata table
- with boolean values.
- top_n: Number of top prevalent peaks to be selected. This value is ignored if a value is provided
- for `min_var` parameter. (Default: 500)
- prevalence_key_name: Base label for marking prevalent peaks in the features metadata column. The value for
- 'cell_key' parameter is prepended to this value. (Default value: 'prevalent_peaks')
-
- Returns:
- None
- """
- if cell_key is None:
- cell_key = "I"
- assay = self._get_assay(from_assay)
- if type(assay) != ATACassay: # noqa: E721
- raise TypeError(
- f"ERROR: This method of feature selection can only be applied to ATACassay type of assay. "
- f"The provided assay is {type(assay)} type"
- )
- assay.mark_prevalent_peaks(cell_key, top_n, prevalence_key_name)
-
- def run_marker_search(
- self,
- from_assay: Optional[str] = None,
- group_key: Optional[str] = None,
- cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- gene_batch_size: int = 50,
- use_prenormed: bool = False,
- prenormed_store: Optional[str] = None,
- n_threads: Optional[int] = None,
- skip_save: bool = False,
- **norm_params,
- ) -> Optional[dict]:
- """Identifies group specific features for a given assay.
-
- Please check out the ``find_markers_by_rank`` function for further details of how marker features for groups
- are identified. The results are saved into the Zarr hierarchy under `markers` group.
-
- Args:
- from_assay: Name of the assay to be used. If no value is provided then the default assay will be used.
- group_key: Required parameter. This has to be a column name from cell metadata table. This column dictates
- how the cells will be grouped. Usually this would be a column denoting cell clusters.
- cell_key: To run the test on specific subset of cells, provide the name of a boolean column in
- the cell metadata table. (Default value: 'I')
- feat_key:
- gene_batch_size: Number of genes to be loaded in memory at a time. All cells (from ell_key) are loaded for
- these number of cells at a time.
- use_prenormed: If True then prenormalized cache generated using Assay.save_normed_for_query is used.
- This can speed up the results. (Default value: True)
- prenormed_store: If prenormalized values were computed in a custom manner then, the Zarr group's location
- can be provided here. (Default value: None)
- n_threads: Number of threads to use to run the marker search. Only used if use_prenormed is True.
- skip_save: If True then the results are not saved to the Zarr hierarchy.
-
- Returns:
- None
- """
- from ..markers import find_markers_by_rank
-
- if group_key is None:
- raise ValueError(
- "ERROR: Please provide a value for `group_key`. This should be the name of a column from "
- "cell metadata object that has information on how cells should be grouped."
- )
- if cell_key is None:
- cell_key = "I"
- if feat_key is None:
- feat_key = "I"
- if n_threads is None:
- n_threads = self.nthreads
- assay = self._get_assay(from_assay)
-
- slot_name = f"{cell_key}__{group_key}"
- z = self.zw[assay.name]
- if "markers" not in z:
- z.create_group("markers")
- group = z["markers"].create_group(slot_name, overwrite=True)
-
- markers = find_markers_by_rank(
- assay=assay,
- group_key=group_key,
- cell_key=cell_key,
- feat_key=feat_key,
- batch_size=gene_batch_size,
- use_prenormed=use_prenormed,
- prenormed_store=prenormed_store,
- n_threads=n_threads,
- **norm_params,
- )
-
- if skip_save:
- return markers
- else:
- for i in markers:
- g = group.create_group(i)
- vals = markers[i]
- if len(vals) != 0:
- for j in vals.columns:
- create_zarr_obj_array(g, j, vals[j].values, dtype=vals[j].dtype)
- return None
-
- def run_pseudotime_marker_search(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- pseudotime_key: Optional[str] = None,
- min_cells: int = 10,
- gene_batch_size: int = 50,
- **norm_params,
- ) -> None:
- """Identify genes that a correlated with a given pseudotime ordering of
- cells. The results are saved in feature attribute tables. For example,
- the r value can be found under, 'I__RNA_pseudotime__r' and the
- corresponding p values can be found under 'I__RNA_pseudotime__p' The
- values are saved with patten {cell_key}__{regressor_key}__r/p.
-
- Args:
- from_assay: Name of the assay to be used. If no value is provided then the default assay will be used.
- cell_key: To run the test on specific subset of cells, provide the name of a boolean column in
- the cell metadata table. (Default value: 'I')
- feat_key:
- pseudotime_key: Required parameter. This has to be a column name from cell metadata table. This column
- contains values for pseudotime ordering of the cells.
- min_cells: Minimum number of cells where a gene should have non-zero value to be considered for test.
- (Default: 10)
- gene_batch_size: Number of genes to be loaded in memory at a time. (Default value: 50).
-
- Returns: None
- """
-
- from ..markers import find_markers_by_regression
-
- if pseudotime_key is None:
- raise ValueError(
- "ERROR: Please provide a value for `pseudotime_key`. This should be the name of a column from "
- "cell metadata object where pseudotime values are stored. If you ran `run_pseudotime_scoring` then "
- "the values are stored under `RNA_pseudotime` by default."
- )
- if cell_key is None:
- cell_key = "I"
- if feat_key is None:
- feat_key = "I"
- assay = self._get_assay(from_assay)
- ptime = assay.cells.fetch(pseudotime_key, key=cell_key)
- markers = find_markers_by_regression(
- assay=assay,
- cell_key=cell_key,
- feat_key=feat_key,
- regressor=ptime,
- min_cells=min_cells,
- batch_size=gene_batch_size,
- **norm_params,
- )
- assay.feats.insert(
- f"{cell_key}__{pseudotime_key}__r",
- np.array(markers["r_value"].values),
- overwrite=True,
- )
- assay.feats.insert(
- f"{cell_key}__{pseudotime_key}__p",
- np.array(markers["p_value"].values),
- overwrite=True,
- )
-
- def run_pseudotime_aggregation(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- pseudotime_key: Optional[str] = None,
- cluster_label: Optional[str] = None,
- min_exp: float = 1e-3,
- window_size: int = 200,
- chunk_size: int = 50,
- smoothen: bool = True,
- z_scale: bool = True,
- n_neighbours: int = 11,
- n_clusters: int = 10,
- batch_size: int = 100,
- ann_params: Optional[dict] = None,
- nan_cluster_value: Union[int, str] = -1,
- ) -> None:
- """This method performs clustering of features based on pseudotime
- ordered cells. The values from the pseudotime ordered cells are
- smoothened, scaled and binned. The resulting binned matrix is used to
- perform a KNN-Paris clustering of the features. This function can be
- used an alternative to `run_marker_search` and
- `run_pseudotime_marker_search`
-
- Args:
- from_assay: Name of the assay to be used. If no value is provided then the default assay will be used.
- cell_key: To run the test on specific subset of cells, provide the name of a boolean column in
- the cell metadata table. (Default value: The cell key that was used to generate the latest graph)
- feat_key: To use only a subset of features, provide the name of a boolean column in the feature
- metadata/attribute table. Default value: The cell key that was used to generate the latest graph)
- pseudotime_key: Required parameter. This has to be a column name from cell attribute table. This
- column contains values for pseudotime ordering of the cells.
- cluster_label: Required parameter. Name of the column under which the feature cluster identity will be
- saved in the feature attribute table.
- min_exp: Features with mean normalized expression than this value are dropped and hence not assigned
- a cluster identity (Default value: 10)
- window_size: The window for calculating rolling mean of feature values along pseudotime ordering. Larger
- values will slow down processing but produce more smoothened. The choice of value here depends
- on the number of cells in the analysis. Larger value will be useful to produce smooth profiles
- when number of cells are large. (Default value: 200)
- chunk_size: Number of bins of cells to create. Larger values will increase memory consumption but will
- provide improved resolution (Default value: 50)
- smoothen: Whether to perform the rolling window averaging (Default value: True)
- z_scale: Whether to perform standard scaling of each feature. Turning this off may not be a good choice.
- (Default value: True)
- n_neighbours: Number of neighbours to save in the KNN graph of features(Default value: 11)
- n_clusters: Number of feature clusters to create. (Default value: 10)
- batch_size: Number of features to load at a time when processing the data. Larger values will increase
- memory consumption (Default value: 100)
- ann_params: The parameter to forward to HNSWlib index instantiation step. (Default value: {})
- nan_cluster_value: The value to use for features that are not assigned a cluster identity.
- (Default value: -1)
-
- Returns: None
- """
- from ..markers import knn_clustering
-
- from_assay, cell_key, _ = self._get_latest_keys(from_assay, cell_key, feat_key)
- if feat_key is None:
- feat_key = "I"
- assay = self._get_assay(from_assay)
-
- if pseudotime_key is None:
- raise ValueError(
- "ERROR: Please provide a value for `pseudotime_key` parameter. This is the column in "
- "the cell attribute table that contains the pseudotime values."
- )
- if cluster_label is None:
- raise ValueError(
- "ERROR: Please provide a value for cluster_label. "
- "It will be used to create new column in feature attribute table. The module identity "
- "of each feature will be saved under this column name. If this column already exists "
- "then it will be overwritten."
- )
-
- df, feat_ids = assay.save_aggregated_ordering(
- cell_key=cell_key,
- feat_key=feat_key,
- ordering_key=pseudotime_key,
- min_exp=min_exp,
- window_size=window_size,
- chunk_size=chunk_size,
- smoothen=smoothen,
- z_scale=z_scale,
- batch_size=batch_size,
- )
- if ann_params is None:
- ann_params = {}
- clusts = knn_clustering(
- d_array=df,
- n_neighbours=n_neighbours,
- n_clusters=n_clusters,
- n_threads=self.nthreads,
- ann_params=ann_params,
- )
- temp = np.ones(assay.feats.N) * -1
- temp[feat_ids] = clusts
- assay.feats.insert(
- cluster_label, temp.astype(int), fill_value=-1, overwrite=True
- )
- return None
-
- def get_markers(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- group_key: Optional[str] = None,
- group_id: Optional[Union[str, int]] = None,
- min_score: float = 0.25,
- min_frac_exp: float = 0.2,
- ) -> pd.DataFrame:
- """Returns a table of markers features obtained through
- `run_marker_search` for a given group.
-
- The table contains names of marker features and feature ids are used as table index.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: To run the test on specific subset of cells, provide the name of a boolean column in
- the cell metadata table.
- group_key: Required parameter. This has to be a column name from cell metadata table.
- Usually this would be a column denoting cell clusters. Please use the same value as used
- when ran `run_marker_search`
- group_id: This is one of the value in `group_key` column of cell metadata.
- Results are returned for this group
- min_score: This value dictates how specific the feature value has to be in a group before it is
- considered a marker for that group. The value has to be greater than 0 but less than or equal to
- 1 (Default value: 0.25)
- min_frac_exp: Minimum fraction of cells in a group that must have a non-zero value for a gene to be
- considered a marker for that group.
-
- Returns:
- Pandas dataframe
- """
-
- if cell_key is None:
- cell_key = "I"
- if group_key is None:
- raise ValueError(
- "ERROR: Please provide a value for group_key. "
- "This should be same as used for `run_marker_search`"
- )
- assay = self._get_assay(from_assay)
- try:
- g = assay.z["markers"][f"{cell_key}__{group_key}"] # type: ignore
- except KeyError:
- raise KeyError(
- "ERROR: Couldn't find the location of markers. Please make sure that you have already called "
- "`run_marker_search` method with same value of `cell_key` and `group_key`"
- )
- out_cols = [
- "feature_index",
- "score",
- "mean",
- "mean_rest",
- "frac_exp",
- "frac_exp_rest",
- "fold_change",
- "p_value",
- ]
- gids = sorted(set(assay.cells.fetch(group_key, key=cell_key)))
- if group_id is not None:
- gids = [group_id]
-
- dfs = []
- for gid in gids:
- if gid in g:
- cols = [g[gid][x][:] for x in out_cols] # type: ignore
- df = pd.DataFrame(
- cols,
- index=out_cols,
- ).T
- df["group_id"] = gid
- df["feature_name"] = assay.feats.fetch_all("names")[
- df.feature_index.astype("int")
- ]
- else:
- logger.debug(f"No markers found for {gid} returning empty dataframe")
- df = pd.DataFrame([[] for _ in out_cols], index=out_cols).T
- df["group_id"] = []
- df["feature_name"] = []
- df = df[["group_id", "feature_name"] + out_cols]
- dfs.append(df)
- dfs = pd.concat(dfs)
- return dfs[
- (dfs.score >= min_score) & (dfs.frac_exp >= min_frac_exp)
- ].reset_index(drop=True)
-
- def export_markers_to_csv(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- group_key: Optional[str] = None,
- csv_filename: Optional[str] = None,
- min_score: float = 0.25,
- min_frac_exp: float = 0.2,
- ) -> None:
- """Export markers of each cluster/group to a CSV file where each column
- contains the marker names sorted by score (descending order, highest
- first). This function does not export the scores of markers as they can
- be obtained using `get_markers` function.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: To run the test on specific subset of cells, provide the name of a boolean column in
- the cell metadata table.
- group_key: Required parameter. This has to be a column name from cell metadata table.
- Usually this would be a column denoting cell clusters. Please use the same value as used
- when ran `run_marker_search`
- csv_filename: Required parameter. Name, with path, of CSV file where the marker table is to be saved.
- min_score: This value dictates how specific the feature value has to be in a group before it is
- considered a marker for that group. The value has to be greater than 0 but less than or equal to
- 1 (Default value: 0.25)
- min_frac_exp: Minimum fraction of cells in a group that must have a non-zero value for a gene to be
- considered a marker for that group.
-
- Returns:
- """
- # Not testing the values of from_assay and cell_key because they will be tested in `get_markers`
- if group_key is None:
- raise ValueError(
- "ERROR: Please provide a value for group_key. "
- "This should be same as used for `run_marker_search`"
- )
- if csv_filename is None:
- raise ValueError(
- "ERROR: Please provide a value for parameter `csv_filename`"
- )
- from_assay, cell_key, _ = self._get_latest_keys(from_assay, cell_key, None)
- clusters = self.cells.fetch(group_key, key=cell_key)
- markers_table = {}
- for group_id in sorted(set(clusters)):
- m = self.get_markers(
- from_assay=from_assay,
- cell_key=cell_key,
- group_key=group_key,
- group_id=group_id,
- min_score=min_score,
- min_frac_exp=min_frac_exp,
- )
- if len(m) > 0:
- markers_table[group_id] = m["feature_name"].reset_index(drop=True)
- else:
- markers_table[group_id] = pd.Series([])
- pd.DataFrame(markers_table).fillna("").to_csv(csv_filename, index=False)
- return None
-
- def run_cell_cycle_scoring(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- s_genes: Optional[List[str]] = None,
- g2m_genes: Optional[List[str]] = None,
- n_bins: int = 50,
- rand_seed: int = 4466,
- s_score_label: str = "S_score",
- g2m_score_label: str = "G2M_score",
- phase_label: str = "cell_cycle_phase",
- ) -> None:
- """Computes S and G2M phase scores by taking into account the average
- expression of S and G2M phase genes respectively. Following steps are
- taken for each phase:
-
- - Average expression of all the genes in across `cell_key` cells is calculated
- - The log average expression is divided in `n_bins` bins
- - A control set of genes is identified by sampling genes from same expression bins where phase's genes are present.
- - The average expression of phase genes (Ep) and control genes (Ec) is calculated per cell.
- - A phase score is calculated as: Ep-Ec Cell cycle phase is assigned to each cell based on following rule set:
- - G1 phase: S score < -1 > G2M sore
- - S phase: S score > G2M score
- - G2M phase: G2M score > S score
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- s_genes: A list of S phase genes. If not provided then Scarf loads pre-saved genes accessible at
- `scarf.bio_data.s_phase_genes`
- g2m_genes: A list of G2M phase genes. If not provided then Scarf loads pre-saved genes accessible at
- `scarf.bio_data.g2m_phase_genes`
- n_bins: Number of bins into which average expression of genes is divided.
- rand_seed: A random values to set seed while sampling cells from a cluster randomly. (Default value: 4466)
- s_score_label: A base label for saving the S phase scores into a cell metadata column
- (Default value: 'S_score')
- g2m_score_label: A base label for saving the G2M phase scores into a cell metadata column
- (Default value: 'G2M_score')
- phase_label: A base label for saving the inferred cell cycle phase into a cell metadata column
- (Default value: 'cell_cycle_phase')
-
- Returns: None
- """
- if from_assay is None:
- from_assay = self._defaultAssay
- assay = self._get_assay(from_assay)
- if cell_key is None:
- cell_key = "I"
- if s_genes is None:
- from ..bio_data import s_phase_genes
-
- s_genes = list(s_phase_genes)
- if g2m_genes is None:
- from ..bio_data import g2m_phase_genes
-
- g2m_genes = list(g2m_phase_genes)
- control_size = min(len(s_genes), len(g2m_genes))
-
- s_score = assay.score_features(
- s_genes, cell_key, control_size, n_bins, rand_seed
- )
- s_score_label = self._col_renamer(from_assay, cell_key, s_score_label)
- self.cells.insert(s_score_label, s_score, key=cell_key, overwrite=True)
-
- g2m_score = assay.score_features(
- g2m_genes, cell_key, control_size, n_bins, rand_seed
- )
- g2m_score_label = self._col_renamer(from_assay, cell_key, g2m_score_label)
- self.cells.insert(g2m_score_label, g2m_score, key=cell_key, overwrite=True)
-
- phase = pd.Series(
- ["S" for _ in range(self.cells.fetch(cell_key, key=cell_key).sum())]
- )
- phase[g2m_score > s_score] = "G2M"
- phase[(g2m_score < 0) & (s_score < 0)] = "G1"
- phase_label = self._col_renamer(from_assay, cell_key, phase_label)
- self.cells.insert(
- phase_label, np.array(phase.values), key=cell_key, overwrite=True
- )
-
- def add_grouped_assay(
- self,
- from_assay: Optional[str] = None,
- group_key: Optional[str] = None,
- assay_label: Optional[str] = None,
- exclude_values: Optional[list] = None,
- ) -> None:
- """Add a new assay to the DataStore by grouping together multiple
- features and taking their means. This method requires that the features
- are already assigned a group/cluster identity. The new assay will have
- all the cells but only features that marked by 'feat_key' and contain a
- group identity not present in `exclude_values`.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- group_key: This is mandatory parameter. Name of the column in feature metadata table to be used for
- grouping features.
- assay_label: This is mandatory parameter. A name for the new assay.
- exclude_values: These groups/clusters will be ignored and not added to new assay. By default, it is set to
- [-1], this means that all the features that have the group identity of -1 are not used.
-
- Returns: None
- """
-
- from ..writers import create_zarr_count_assay
-
- if assay_label is None:
- raise ValueError(
- "ERROR: Please provide a value for `assay_label`. "
- "It will be used to create a new assay"
- )
- if group_key is None:
- raise ValueError(
- "ERROR: Please provide a value for `group_key`. "
- "This should be name of the column in the feature attribute table that contains the group/cluster "
- "identity of each feature."
- )
-
- assay = self._get_assay(from_assay)
- groups = assay.feats.fetch_all(group_key)
- if exclude_values is None:
- exclude_values = [-1]
- group_set = sorted(set(groups).difference(exclude_values))
-
- module_ids = [f"group_{x}" for x in group_set]
- g = create_zarr_count_assay(
- z=assay.z["/"], # type: ignore
- assay_name=assay_label,
- workspace=self.workspace,
- chunk_size=assay.rawData.chunksize, # type: ignore
- n_cells=assay.cells.N,
- feat_ids=module_ids,
- feat_names=module_ids,
- dtype="float",
- )
-
- cell_idx = np.array(list(range(assay.cells.N)))
- for n, i in tqdmbar(
- enumerate(group_set), desc="Writing to Zarr", total=len(group_set)
- ):
- feat_idx = np.where(groups == i)[0]
- temp = np.zeros(assay.cells.N)
- temp[cell_idx] = (
- assay.normed(cell_idx=cell_idx, feat_idx=feat_idx)
- .mean(axis=1)
- .compute()
- )
- g[:, n] = temp
-
- self._load_assays(min_cells=0, custom_assay_types={assay_label: "Assay"})
- self._ini_cell_props(min_features=0, mito_pattern="", ribo_pattern="")
-
- def add_melded_assay(
- self,
- from_assay: Optional[str] = None,
- external_bed_fn: Optional[str] = None,
- assay_label: Optional[str] = None,
- peaks_col: str = "ids",
- scalar_coeff: float = 1e5,
- renormalization: bool = True,
- assay_type: str = "Assay",
- ) -> None:
- """This method performs "assay melding" and can be only be used for
- assay's wherein features have genomic coordinates. In the process of
- melding the input genomic coordinates from `external_bed_fn` are
- intersected with the assay's features. Based on this intersection a
- mapping is created wherein each coordinate interval maps to one or more
- feature coordinates from the assay.
-
- This method has been designed for snATAC-Seq data and can be used to quantify accessibility of specific
- genomic loci such as gene bodies, promoters, enhancers, motifs, etc.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- external_bed_fn: This is mandatory parameter. This file should be a BED format file with at least five
- columns containing: chromosome, start position, end position, feature id and feature name.
- Coordinates should be in half open format. That means that actual end position is -1
- assay_label: This is mandatory parameter. A name for the new assay.
- peaks_col: The column in feature metadata table that contains the genomic coordinate information of each
- feature. The genomic coordinates are represented as strings in this format: chr:start-end
- (Default value: 'ids')
- scalar_coeff: An arbitrary scalar multiplier. Only used when renormalization is True (Default value: 1e5)
- renormalization: Whether to rescale the sum of feature values for each cell to `scalar_coeff`
- (Default value: True)
- assay_type: The new assay (melded assay) is saved as this type. This can be any type of Assay class from
- `assay` module. Please provide string representation of class. By default, the assay is assigned
- a generic class and has a dummy normalization function (Default value: 'Assay')
-
- Returns:
- None
- """
-
- from ..meld_assay import coordinate_melding
-
- if assay_label is None:
- raise ValueError(
- "ERROR: Please provide a value for `assay_label`. "
- "It will be used to create a new assay"
- )
- if external_bed_fn is None:
- raise ValueError(
- "ERROR: Please provide a value for `feature_bed_fn`. "
- "This should be a BED format file with atleast 5 columns."
- )
-
- assay = self._get_assay(from_assay)
- feature_bed = pd.read_csv(external_bed_fn, header=None, sep="\t").sort_values(
- by=[0, 1] # type: ignore
- )
-
- peaks_coords = assay.feats.fetch_all(peaks_col)
- for n, i in enumerate(peaks_coords):
- error_msg = (
- f"ERROR: Coordinate format check failed for element: {i} (position {n}). The format should "
- f"be chr:start-end. Please note the colon and hyphen position"
- )
- if len(i.split(":")) != 2:
- raise ValueError(error_msg)
- if len(i.split(":")[1].split("-")) != 2:
- raise ValueError(error_msg)
-
- coordinate_melding(
- assay,
- workspace=self.workspace,
- feature_bed=feature_bed,
- new_assay_name=assay_label,
- peaks_col=peaks_col,
- scalar_coeff=scalar_coeff,
- renormalization=renormalization,
- )
-
- self._load_assays(min_cells=10, custom_assay_types={assay_label: assay_type})
- self._ini_cell_props(min_features=0, mito_pattern=None, ribo_pattern=None)
-
- def make_bulk(
- self,
- from_assay: Optional[str] = None,
- cell_key: str = "I",
- group_key: Optional[str] = None,
- secondary_group_key: Optional[str] = None,
- aggr_type: Literal["mean", "sum"] = "mean",
- return_fraction: bool = False,
- feature_label: Literal["index", "id", "name"] = "index",
- remove_empty_features: bool = True,
- pseudo_reps: int = 1,
- null_vals: Optional[list] = None,
- secondary_null_vals: Optional[list] = None,
- random_seed: int = 4466,
- ) -> Union[pd.DataFrame, Tuple[pd.DataFrame, pd.DataFrame]]:
- """Merge data from cells to create a bulk profile.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Name of the column in cell metadata table to be used for selecting cells.
- group_key: Name of the column in cell metadata table to be used for grouping cells.
- secondary_group_key: Name of the column in cell metadata table to be used for sub-grouping cells.
- aggr_type: Type of aggregation to be used. Can be either 'mean' or 'sum'. (Default value: 'mean')
- return_fraction: Return the fraction of cells expressing a gene in each group. (Default value: False)
- feature_label: The column in feature metadata table to use as row labels. (Default value: 'index')
- pseudo_reps: Within each group, cells will randomly be split into `pseudo_reps` partitions. Each partition
- is considered a pseudo-replicate. (Default value: 3)
- remove_empty_features: Remove features that are not expressed in any cell. (Default value: True)
- null_vals: Values to be considered as missing values in the `group_key` column. These values will be skipped.
- secondary_null_vals: Values to be considered as missing values in the `secondary_group_key` column.
- These values will be skipped.
- random_seed: A random values to set seed while creating `pseudo_reps` partitions cells randomly.
-
- Returns:
- A pandas dataframe containing the bulk profile. If `return_fraction` is True, then a tuple of two dataframes
- is returned. The second dataframe contains the fraction of cells expressing each feature in each group.
- """
-
- def make_reps(v, n_reps: int, seed: int) -> List[np.ndarray]:
- v = list(v)
- random_state = np.random.RandomState(seed)
- shuffled_idx = random_state.choice(v, len(v), replace=False)
- rep_idx = np.array_split(shuffled_idx, n_reps)
- return [np.array(sorted(x)) for x in rep_idx]
-
- if pseudo_reps < 1:
- pseudo_reps = 1
- if null_vals is None:
- null_vals = []
- if secondary_null_vals is None:
- secondary_null_vals = []
- if group_key is None:
- raise ValueError("ERROR: Please provide a value for `group_key` parameter")
- else:
- groups = self.cells.fetch_all(group_key)
- groups_set = sorted(set(self.cells.fetch(group_key, key=cell_key)))
- if secondary_group_key is None:
- sec_groups = [None]
- sec_groups_set = [None]
- else:
- sec_groups = self.cells.fetch_all(secondary_group_key)
- sec_groups_set = sorted(
- set(self.cells.fetch(secondary_group_key, key=cell_key))
- )
-
- assay = self._get_assay(from_assay)
-
- vals = {}
- fracs = {}
- all_feat_idx = np.arange(assay.feats.N)
- for g in tqdmbar(groups_set):
- if g in null_vals:
- continue
- for sg in sec_groups_set: # type: ignore
- if sg in secondary_null_vals:
- continue
- if sg is None and len(sec_groups) == 1:
- g_idx = np.where(groups == g)[0]
- else:
- g_idx = np.where((groups == g) & (sec_groups == sg))[0]
- rep_indices = make_reps(g_idx, pseudo_reps, random_seed)
- for n, idx in enumerate(rep_indices):
- if sg is None and len(sec_groups) == 1:
- col_name = f"{g}"
- else:
- col_name = f"{g}_{sg}"
- if pseudo_reps > 1:
- col_name += f"_Rep{n + 1}"
- if len(idx) == 0:
- vals[col_name] = np.zeros(assay.feats.N)
- continue
- if aggr_type == "sum":
- vals[col_name] = controlled_compute(
- assay.rawData[idx].sum(axis=0), self.nthreads
- )
- elif aggr_type == "mean":
- vals[col_name] = controlled_compute(
- assay.normed(cell_idx=idx, feat_idx=all_feat_idx).mean(
- axis=0
- ),
- self.nthreads,
- )
- else:
- raise ValueError(
- "ERROR: `aggr_type` can only be either 'sum' or 'mean'"
- )
- if return_fraction:
- fracs[col_name] = (
- (assay.rawData[idx] > 0).mean(axis=0).compute()
- )
-
- vals = pd.DataFrame(vals).fillna(0)
-
- empty_idx = None
- if remove_empty_features:
- empty_idx = vals.sum(axis=1) != 0
- vals = vals.loc[empty_idx]
-
- if feature_label == "id":
- vals.set_index(
- pd.Series(assay.feats.fetch_all("ids")).reindex(vals.index).values,
- inplace=True,
- drop=True,
- )
- elif feature_label == "name":
- vals.set_index(
- pd.Series(assay.feats.fetch_all("names")).reindex(vals.index).values,
- inplace=True,
- drop=True,
- )
-
- if return_fraction:
- fracs = pd.DataFrame(fracs).fillna(0)
- if empty_idx is not None:
- fracs = fracs[empty_idx]
- fracs.set_index(vals.index, inplace=True, drop=True)
- return vals, fracs
- else:
- return vals
-
- def to_anndata(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- layers: Optional[dict] = None,
- ):
- """Writes an assay of the Zarr hierarchy to AnnData file format.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Name of column from cell metadata that has boolean values. This is used to subset cells
- layers: A mapping of layer names to assay names. Ex. {'spliced': 'RNA', 'unspliced': 'URNA'}. The raw data
- from the assays will be stored as sparse arrays in the corresponding layer in anndata.
-
- Returns: anndata object
- """
- try:
- # noinspection PyPackageRequirements
- from anndata import AnnData # type: ignore
- except ImportError:
- logger.error(
- "Package anndata is not installed because its an optional dependency. "
- "Install via `pip install anndata` or `conda install anndata -c conda-forge`"
- )
- return None
-
- if cell_key is None:
- cell_key = "I"
- assay = self._get_assay(from_assay)
- df = self.cells.to_pandas_dataframe(self.cells.columns, key=cell_key)
- obs = df.reset_index(drop=True).set_index("ids")
- df = assay.feats.to_pandas_dataframe(assay.feats.columns)
- var = df.rename(columns={"ids": "gene_ids"}).set_index("gene_ids")
- adata = AnnData(assay.to_raw_sparse(cell_key), obs=obs, var=var)
- if layers is not None:
- for layer, assay_name in layers.items():
- adata.layers[layer] = self._get_assay(assay_name).to_raw_sparse(
- cell_key
- )
- return adata
-
- def show_zarr_tree(self, start: str = "/", depth: int = 2) -> None:
- """Prints the Zarr hierarchy of the DataStore.
-
- Args:
- start: Location in Zarr hierarchy to be used as the root for display
- depth: Depth of Zarr hierarchy to be displayed.
-
- Returns:
- None
- """
- print(self.zw[start].tree(expand=True, level=depth))
-
- def calc_membership_strength(
- self, from_assay: str, cell_key: str, feat_key: str, clust_key: str
- ) -> None:
- loc = self._get_latest_graph_loc(
- from_assay=from_assay, cell_key=cell_key, feat_key=feat_key
- )
- n_cells, k = self._get_graph_ncells_k(graph_loc=loc)
- clusts = self.cells.fetch(clust_key, key=cell_key)
- v = pd.DataFrame(clusts[self.zw[loc]["edges"][:, 1].reshape(k, n_cells)])
- x = np.array([v[x].value_counts().index[0] for x in v])
- self.cells.insert(
- f"{from_assay}_{cell_key}_cluster_membership_strength",
- (np.array((v == x).sum().values) / k).round(3),
- key=cell_key,
- overwrite=True,
- )
- return None
-
- def smart_label(
- self,
- to_relabel: str,
- base_label: str,
- cell_key: str = "I",
- new_col_name: Optional[str] = None,
- ) -> Union[None, List[str]]:
- """A convenience function to relabel the values in a cell attribute
- column (A) based on the values in another cell attribute column (B).
- For each unique value in A, the most frequently occurring value in B is
- found. If two or more values in A have maximum overlap with the same
- value in B, then they all get the same label as B along with different
- suffixes like, 'a', 'b', etc. The suffixes are ordered based on where
- the largest fraction of the B label lies. If one label from A takes up
- multiple labels from B then all the labels from B are included, and they
- are delimited by hyphens.
-
- Args:
- to_relabel: Cell attributes column to relabel
- base_label: Cell attributes column to relabel
- cell_key: Cell key fetching column values
- new_col_name: Name of new column where relabeled values will be saved. If None then values
- are returned and not saved in cell attributes table
-
- Returns: None or a list of relabelled values
- """
- df = pd.crosstab(
- self.cells.fetch(base_label, key=cell_key),
- self.cells.fetch(to_relabel, key=cell_key),
- )
- normed_frac = df.divide(df.sum(axis=1), axis="index")
- idxmax = df.idxmax()
- new_names = {}
- for i in sorted(idxmax.unique()):
- j = normed_frac[idxmax[idxmax == i].index].loc[i]
- j = j.sort_values(ascending=False).index
- for n, k in enumerate(j, start=1):
- a = chr(ord("@") + n)
- new_names[k] = f"{i}{a.lower()}"
-
- missing_vals = list(set(df.index).difference(idxmax.unique()))
- if len(missing_vals) > 0:
- miss_idxmax = df.loc[missing_vals].idxmax(axis=1).to_dict()
- for k, v in miss_idxmax.items():
- new_names[v] = f"{new_names[v][:-1]}-{k}{new_names[v][-1]}"
-
- ret_val = [new_names[x] for x in self.cells.fetch(to_relabel, key=cell_key)]
- if new_col_name is None:
- return ret_val
- else:
- self.cells.insert(new_col_name, ret_val, overwrite=True)
-
- def plot_cells_dists(
- self,
- from_assay: Optional[str] = None,
- cols: Optional[List[str]] = None,
- cell_key: Optional[str] = None,
- group_key: Optional[str] = None,
- color: str = "steelblue",
- cmap: str = "tab20",
- fig_size: Optional[tuple] = None,
- label_size: float = 10.0,
- title_size: float = 10.0,
- sup_title: Optional[str] = None,
- sup_title_size: float = 12.0,
- scatter_size: float = 1.0,
- max_points: int = 10000,
- show_on_single_row: bool = True,
- show_fig: bool = True,
- ) -> None:
- """Makes violin plots of the distribution of values present in cell
- metadata. This method is designed to distribution of nCounts,
- nFeatures, percentMito and percentRibo cell attributes.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cols: Column names from cell metadata table to be used for plotting. Be default, nCounts, nFeatures,
- percentMito and percentRibo columns are chosen.
- cell_key: One of the columns from cell metadata table that indicates the cells to be used for plotting.
- The values in the chosen column should be boolean (Default value: 'I')
- group_key: A column name from cell metadata table that indicates how cells should be grouped. This can be
- any column that has either boolean or categorical values. By default, no grouping will be
- performed (Default value: None)
- color: Face color of the violin plots. The value can be valid matplotlib named colour. This is used only
- when there is a single group. (Default value: 'steelblue')
- cmap: A matplotlib colormap to be used to color different groups. (Default value: 'tab20')
- fig_size: A tuple of figure width and figure height (Default value: Automatically determined by `plot_qc`)
- label_size: The font size of y-axis labels (Default value: 10.0)
- title_size: The font size of title. Median value is printed as title of each violin plot
- (Default value: 10.0)
- sup_title: The title for complete figure panel (Default value: 12.0 )
- sup_title_size: The font size of title for complete figure panel (Default value: 12.0 )
- scatter_size: Size of each point in the violin plot (Default value: 1.0)
- max_points: Maximum number of points to display over violin plot. Random uniform sampling will be performed
- to bring down the number of datapoints to this value. This does not affect the violin plot.
- (Default value: 10000)
- show_on_single_row: Show all subplots in a single row. It might be useful to set this to False if you have
- too many groups within each subplot (Default value: True)
- show_fig: Whether to render the figure and display it using plt.show() (Default value: True)
-
- Returns:
- None
- """
-
- from ..plots import plot_qc
-
- if from_assay is None:
- from_assay = self._defaultAssay
- if cell_key is None:
- # Show all cells
- pass
-
- if cols is not None:
- if type(cols) != list: # noqa: E721
- raise ValueError("ERROR: 'cols' argument must be of type list")
- plot_cols = []
- for i in cols:
- if i in self.cells.columns:
- if i not in plot_cols:
- plot_cols.append(i)
- else:
- logger.warning(f"{i} not found in cell metadata")
- else:
- cols = ["nCounts", "nFeatures", "percentMito", "percentRibo"]
- cols = [f"{from_assay}_{x}" for x in cols]
- plot_cols = [x for x in cols if x in self.cells.columns]
-
- debug_print_cols = "\n".join(plot_cols)
- logger.debug(
- f"(plot_cells_dists): Will plot following columns: {debug_print_cols}"
- )
-
- df = self.cells.to_pandas_dataframe(plot_cols)
- if group_key is not None:
- df["groups"] = self.cells.to_pandas_dataframe([group_key])
- else:
- df["groups"] = np.zeros(len(df))
- if cell_key is not None:
- idx = self.cells.active_index(cell_key)
- df = df.reindex(idx)
-
- plot_qc(
- df,
- color=color,
- cmap=cmap,
- fig_size=fig_size,
- label_size=label_size,
- title_size=title_size,
- sup_title=sup_title,
- sup_title_size=sup_title_size,
- scatter_size=scatter_size,
- max_points=max_points,
- show_on_single_row=show_on_single_row,
- show_fig=show_fig,
- )
- return None
-
- def plot_layout(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- layout_key: Optional[str] = None,
- color_by: Optional[str] = None,
- subselection_key: Optional[str] = None,
- size_vals: Union[np.ndarray, List[float], None] = None,
- clip_fraction: float = 0.01,
- width: float = 6,
- height: float = 6,
- default_color: str = "steelblue",
- cmap: Optional[str] = None,
- color_key: Optional[dict] = None,
- mask_values: Optional[list] = None,
- mask_name: str = "NA",
- mask_color: str = "k",
- point_size: float = 10,
- do_shading: bool = False,
- shade_npixels: int = 1000,
- shade_min_alpha: int = 10,
- spread_pixels: int = 1,
- spread_threshold: float = 0.2,
- ax_label_size: float = 12,
- frame_offset: float = 0.05,
- spine_width: float = 0.5,
- spine_color: str = "k",
- displayed_sides: tuple = ("bottom", "left"),
- legend_ondata: bool = True,
- legend_onside: bool = True,
- legend_size: float = 12,
- legends_per_col: int = 20,
- title: Union[str, List[str], None] = None,
- title_size: int = 12,
- hide_title: bool = False,
- cbar_shrink: float = 0.6,
- marker_scale: float = 70,
- lspacing: float = 0.1,
- cspacing: float = 1,
- shuffle_df: bool = False,
- sort_values: bool = False,
- savename: Optional[str] = None,
- save_dpi: int = 300,
- ax=None,
- force_ints_as_cats: bool = True,
- n_columns: int = 4,
- w_pad: float = 1,
- h_pad: float = 1,
- show_fig: bool = True,
- scatter_kwargs: Optional[dict] = None,
- ):
- """Create a scatter plot with a chosen layout. The method fetches the
- coordinates based from the cell metadata columns with `layout_key`
- prefix. DataShader library is used to draw fast rasterized image is
- `do_shading` is True. This can be useful when large number of cells are
- present to quickly render the plot and avoid over-plotting. The
- description of shading parameters has mostly been copied from the
- Datashader API that can be found here:
- https://holoviews.org/_modules/holoviews/operation/datashader.html.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: One of the columns from cell metadata table that indicates the cells to be used.
- The values in the chosen column should be boolean (Default value: 'I')
- layout_key: A prefix to cell metadata columns that contains the coordinates for the 2D layout of the cells.
- For example, 'RNA_UMAP' or 'RNA_tSNE'. If a list of prefixes is provided a grid of plots will be
- made.
- color_by: One (or a list) of the columns of the metadata table or a feature name (for example gene, GATA2).
- If a list of names is provided a grid of plots will be made.
- (Default: None)
- subselection_key: A column from cell metadata table to be used to show only a sub-selection of cells. This
- key can be used to hide certain cells from a 2D layout. (Default value: None)
- size_vals: An array of values to be used to set sizes of each cell's datapoint in the layout.
- By default, all cells are of same size determined by `point_size` parameter.
- Has no effect if `do_shading` is True (Default value: None)
- clip_fraction: Same as `clip_fraction` parameter of 'get_cell_vals' method. This value is multiplied by 100
- and the percentiles are soft-clipped from either end. (Default value: 0)
- width: Figure width (Default value: 6)
- height: Figure height (Default value: 6)
- default_color: A default color for the cells. (Default value: steelblue)
- cmap: A matplotlib colourmap to be used to colour categorical or continuous values plotted on the cells.
- (Default value: tab20 for categorical variables and cmocean.deep for continuous variables)
- color_key: A custom colour map for cells. These can be used for categorical variables only. The keys in this
- dictionary should be the category label as present in the `color_by` column and values should be
- valid matplotlib colour names or hex codes of colours. (Default value: None)
- mask_values: These can a subset of categorical variables that are present in `color_by` which you would like
- to mask away. These values would be combined under a same label (`mask_name`) and will be given
- same colour (`mask_color`)
- mask_name: Label to replace the masked value labels. (Default value : None)
- mask_color: Color to be used for masked values. This should be a valid matplotlib named colour or a hexcode
- of a colour. (Default value: 'k')
- point_size: Size of each scatter point. This is overridden if `size_vals` is provided. Has no effect if
- `do_shading` is True. (Default value: 10)
- do_shading: Sets shading mode on/off. If shading mode is off (default) then matplotlib's scatter function is
- is used otherwise a rasterized image is generated using datashader library. Turn this on if you
- have more than 100K cells to improve render time and also to avoid issues with over-plotting.
- (Default value: False)
- shade_npixels: Number of pixels to rasterize (for both height and width). This controls the resolution of
- the figure. Adjust this according to the size of the image you want to generate.
- (Default value: 1000)
- shade_min_alpha: The minimum alpha value to use for non-empty pixels when doing color-mapping, in [0, 255].
- Use a higher value to avoid under-saturation, i.e. poorly visible low-value datapoints, at
- the expense of the overall dynamic range. (Default value: 10)
- spread_pixels: Maximum number of pixels to spread on all sides (Default value: 1)
- spread_threshold: When spreading, determines how far to spread. Spreading starts at 1 pixel, and stops
- when the fraction of adjacent non-empty pixels reaches this threshold. Higher values
- give more spreading, up to the `spread_pixels` allowed. (Default value: 0.2)
- ax_label_size: Font size for the x and y-axis labels. (Default value: 12)
- frame_offset: Extend the x and y-axis limits by this fraction (Default value: 0.05)
- spine_width: Line width of the displayed spines (Default value: 0.5)
- spine_color: Colour of the displayed spines. (Default value: 'k')
- displayed_sides: Determines which figure spines are chosen. The spines to be shown can be supplied as a
- tuple. The options are: top, bottom, left and right. (Default value: ('bottom', 'left) )
- legend_ondata: Whether to show category labels on the data (scatter points). The position of the label is
- the centroid of the corresponding values. Has no effect if `color_by` has continuous values.
- (Default value: True)
- legend_onside: Whether to draw a legend table on the side of the figure. (Default value: True)
- legend_size: Font size of the legend text. (Default value: 12)
- legends_per_col: Number of legends to be used on each legend column. This value determines how many
- legend columns will be drawn (Default value: 20)
- title: Title to be used for plot/plots. If more than one plot are being plotted then the value should be a
- list of strings. By default, the titles are automatically inferred from color_by parameter
- (Default value: None)
- title_size: Size of each axis/subplots title (Default value: 12)
- hide_title: If True, then the title of the sublots is not shown (Default value: False)
- cbar_shrink: Shrinking factor for the width of color bar (Default value: 0.6)
- marker_scale: The relative size of legend markers compared with the originally drawn ones.
- (Default value: 70)
- lspacing: The vertical space between the legend entries. Measured in font-size units. (Default value: 0.1)
- cspacing: The spacing between columns. Measured in font-size units. (Default value: 1)
- savename: Path where the rendered figure is to be saved. The format of the saved image depends on the
- the extension present in the parameter value. (Default value: None)
- save_dpi: DPI when saving figure (Default value: 300)
- shuffle_df: Shuffle the order of cells in the plot (Default value: False)
- sort_values: Sort the values before plotting. Setting True will cause the datapoints with
- (cells) with larger values to be plotted over the ones with lower values.
- (Default value: False)
- ax: An instance of Matplotlib's Axes object. This can be used to plot the figure into an already
- created axes. It is ignored if `do_shading` is set to True. (Default value: None)
- force_ints_as_cats: Force integer labels in `color_by` as categories. If False, then integer will be
- treated as continuous variables otherwise as categories. This effects how colormaps
- are chosen and how legends are rendered. Set this to False if you are large number of
- unique integer entries (Default: True)
- n_columns: If plotting several plots in a grid this argument decides the layout by how many columns in the
- grid. Defaults to 4 but if the total number of plots is less than 4 it will default to that
- number.
- w_pad: When plotting in multiple plots in a grid this decides the width padding between the plots.
- If None is provided the padding will be automatically added to avoid overlap.
- Ignored if only plotting one scatterplot.
- h_pad: When plotting in multiple plots in a grid this decides the height padding between the plots.
- If None is provided the padding will be automatically added to avoid overlap.
- Ignored if only plotting one scatterplot.
- show_fig: Whether to render the figure and display it using plt.show() (Default value: True)
- scatter_kwargs: Keyword argument to be passed to matplotlib's scatter command
-
- Returns:
- None
- """
-
- # TODO: add support for providing a list of subselections, from_assay and cell_keys
- # TODO: add support for different kinds of point markers
-
- from ..plots import plot_scatter, shade_scatter
-
- if from_assay is None:
- from_assay = self._defaultAssay
- if cell_key is None:
- cell_key = "I"
- if layout_key is None:
- raise ValueError("Please provide a value for `layout_key` parameter.")
- if clip_fraction >= 0.5:
- raise ValueError(
- "ERROR: clip_fraction cannot be larger than or equal to 0.5"
- )
- if isinstance(layout_key, str):
- layout_key: List[str] = [layout_key]
- # If a list of layout keys and color_by (e.g. layout_key=['UMAP', 'tSNE'], color_by=['gene1', 'gene2'] the
- # grid layout will be: plot1: UMAP + gene1, plot2: UMAP + gene2, plot3: tSNE + gene1, plot4: tSNE + gene2
- dfs = []
- for lk in layout_key:
- x = self.cells.fetch(f"{lk}1", key=cell_key)
- y = self.cells.fetch(f"{lk}2", key=cell_key)
- if color_by is None:
- color_by = "vc"
- if isinstance(color_by, str):
- color_by: List[str] = [color_by]
- for c in color_by:
- if c == "vc":
- v = np.ones(len(x)).astype(int)
- else:
- v = self.get_cell_vals(
- from_assay=from_assay,
- cell_key=cell_key,
- k=c,
- clip_fraction=clip_fraction,
- )
- df = pd.DataFrame({f"{lk} 1": x, f"{lk} 2": y, c: v})
- if size_vals is not None:
- if len(size_vals) != len(x):
- raise ValueError(
- "ERROR: `size_vals` is not of same size as layout_key"
- )
- df["s"] = size_vals
- if subselection_key is not None:
- idx = self.cells.fetch(subselection_key, cell_key)
- if idx.dtype != bool:
- logger.warning(
- f"`subselection_key` {subselection_key} is not bool type. Will not sub-select"
- )
- else:
- df = df[idx]
- if shuffle_df:
- df = df.sample(frac=1)
- if sort_values:
- df = df.sort_values(by=c)
- dfs.append(df)
-
- if n_columns > len(dfs):
- n_columns = len(dfs)
-
- if do_shading:
- return shade_scatter(
- dfs,
- ax,
- width,
- shade_npixels,
- spread_pixels,
- spread_threshold,
- shade_min_alpha,
- cmap,
- color_key,
- mask_values,
- mask_name,
- mask_color,
- ax_label_size,
- frame_offset,
- spine_width,
- spine_color,
- displayed_sides,
- legend_ondata,
- legend_onside,
- legend_size,
- legends_per_col,
- title,
- title_size,
- hide_title,
- cbar_shrink,
- marker_scale,
- lspacing,
- cspacing,
- savename,
- save_dpi,
- force_ints_as_cats,
- n_columns,
- w_pad,
- h_pad,
- show_fig,
- )
- else:
- return plot_scatter(
- dfs,
- ax,
- width,
- height,
- default_color,
- cmap,
- color_key,
- mask_values,
- mask_name,
- mask_color,
- point_size,
- ax_label_size,
- frame_offset,
- spine_width,
- spine_color,
- displayed_sides,
- legend_ondata,
- legend_onside,
- legend_size,
- legends_per_col,
- title,
- title_size,
- hide_title,
- cbar_shrink,
- marker_scale,
- lspacing,
- cspacing,
- savename,
- save_dpi,
- force_ints_as_cats,
- n_columns,
- w_pad,
- h_pad,
- show_fig,
- scatter_kwargs,
- )
-
- def plot_cluster_tree(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- cluster_key: Optional[str] = None,
- fill_by_value: Optional[str] = None,
- force_ints_as_cats: bool = True,
- width: float = 1,
- lvr_factor: float = 0.5,
- vert_gap: float = 0.2,
- min_node_size: float = 10,
- node_size_multiplier: float = 1e4,
- node_power: float = 1.2,
- root_size: float = 100,
- non_leaf_size: float = 10,
- show_labels: bool = True,
- fontsize: float = 10,
- root_color: str = "#C0C0C0",
- non_leaf_color: str = "k",
- cmap="tab20",
- color_key: Optional[dict] = None,
- edgecolors: str = "k",
- edgewidth: float = 1,
- alpha: float = 0.7,
- figsize=(5, 5),
- ax=None,
- show_fig: bool = True,
- savename: Optional[str] = None,
- save_dpi: int = 300,
- ):
- """Plots a hierarchical layout of the clusters detected using
- `run_clustering` in a binary tree form. This helps evaluate the
- relationships between the clusters. This figure can complement
- embeddings likes tSNE where global distances are not preserved. The
- plot shows clusters as coloured nodes and the nodes are sized
- proportionally to the number of cells within the clusters. Root and
- branching nodes are shown to visually track the branching pattern of
- the tree. This figure is not scaled, i.e. the distances between the
- nodes are meaningless and only the branching pattern of the nodes must
- be evaluated.
-
- https://epidemicsonnetworks.readthedocs.io/en/latest/functions/EoN.hierarchy_pos.html
-
- Args:
- color_key: A custom colour map for cells. These can be used for categorical variables only. The keys in this
- dictionary should be the category label as present in the `color_by` column and values should be
- valid matplotlib colour names or hex codes of colours. (Default value: None)
- force_ints_as_cats: Force integer labels in `color_by` as categories. If False, then integer will be
- treated as continuous variables otherwise as categories. This effects how colourmaps
- are chosen and how legends are rendered. Set this to False if you are large number of
- unique integer entries (Default: True)
- fill_by_value: ..
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: One of the columns from cell metadata table that indicates the cells to be used.
- Should be same as the one that was used in one of the `run_clustering` calls for the given assay.
- The values in the chosen column should be boolean (Default value: 'I')
- feat_key: Feature key. Should be same as the one that was used in `run_clustering` calls for the
- given assay. By default, the latest used feature for the given assay will be used.
- cluster_key: Should be one of the columns from cell metadata table that contains the output of
- `run_clustering` method. For example if chosen assay is `RNA` and default value for `label`
- parameter was used in `run_clustering` then `cluster_key` can be 'RNA_cluster'
- width: Horizontal space allocated for the branches. Larger values may disrupt the hierarchical layout of
- the cells (Default value: 1)
- lvr_factor: Leaf vs root factor. Controls the relative nodes horizontal spacing between as one moves up or
- down the tree. Higher values will cause terminal nodes to be more spread out at cost of nodes
- closer to the root and vice versa. (Default value: 0.5)
- vert_gap: Gap between levels of hierarchy (Default value: 0.2)
- min_node_size: Minimum size of a node (Default value: 10 )
- node_size_multiplier: Size of each leaf node is increased by this factor (Default value: 1e4)
- node_power: The number of cells within each cluster is raised to this value to scale up the node size.
- (Default value: 1.2)
- root_size: Size of the root node (Default value: 100)
- non_leaf_size: Size of the nodes that represent branch points in the tree (Default value: 10)
- show_labels: Whether to show the cluster labels on the cluster nodes (Default value: True)
- fontsize: Font size of cluster labels. Only used when `do_label` is True (Default value: 10)
- root_color: Colour for root node. Acceptable values are Matplotlib named colours or hexcodes for colours.
- (Default value: '#C0C0C0')
- non_leaf_color: Colour for branch-point nodes. Acceptable values are Matplotlib named colours or hexcodes
- for colours. (Default value: 'k')
- cmap: A colormap to be used to colour cluster nodes. Should be one of Matplotlib colormaps.
- (Default value: 'tab20')
- edgecolors: Edge colour of circles representing nodes in the hierarchical tree (Default value: 'k)
- edgewidth: Line width of the edges circles representing nodes in the hierarchical tree (Default value: 1)
- alpha: Alpha level (Opacity) of the displayed nodes in the figure. (Default value: 0.7)
- figsize: A tuple with describing figure width and height (Default value: (5, 5))
- ax: An instance of Matplotlib's Axes object. This can be used to plot the figure into an already
- created axes. (Default value: None)
- show_fig: If, False then axes object is returned rather than rendering the plot (Default value: True)
- savename: Path where the rendered figure is to be saved. The format of the saved image depends on
- the extension present in the parameter value. (Default value: None)
- save_dpi: DPI when saving figure (Default value: 300)
-
- Returns:
- None
- """
-
- from networkx import DiGraph, to_pandas_edgelist
-
- from ..dendrogram import CoalesceTree, make_digraph
- from ..plots import plot_cluster_hierarchy
-
- from_assay, cell_key, feat_key = self._get_latest_keys(
- from_assay, cell_key, feat_key
- )
-
- if cluster_key is None:
- raise ValueError(
- "ERROR: Please provide a value for `cluster_key` parameter"
- )
- clusts = self.cells.fetch(cluster_key, key=cell_key)
- graph_loc = self._get_latest_graph_loc(from_assay, cell_key, feat_key)
- dendrogram_loc = self.zw[graph_loc].attrs["latest_dendrogram"]
- n_clusts = len(set(clusts))
- coalesced_loc = dendrogram_loc + f"_coalesced_{n_clusts}"
- if coalesced_loc in self.zw:
- subgraph = DiGraph()
- subgraph.add_edges_from(self.zw[coalesced_loc + "/edgelist"][:])
- for i, j in zip(
- self.zw[coalesced_loc + "/nodelist"][:],
- self.zw[coalesced_loc + "/partition_id"][:],
- ):
- node = int(i[0])
- subgraph.nodes[node]["nleaves"] = int(i[1])
- if j != "-1":
- subgraph.nodes[node]["partition_id"] = j
- else:
- subgraph = CoalesceTree(make_digraph(self.zw[dendrogram_loc][:]), clusts)
- edge_list = to_pandas_edgelist(subgraph).values
- store = create_zarr_dataset(
- self.zw, coalesced_loc + "/edgelist", (100000,), "u8", edge_list.shape
- )
- store[:] = edge_list
- node_list = []
- partition_id_list = []
- for i in subgraph.nodes():
- d = subgraph.nodes[i]
- p = d["partition_id"] if "partition_id" in d else -1
- node_list.append((i, d["nleaves"]))
- partition_id_list.append(str(p))
-
- node_list = np.array(node_list)
- store = create_zarr_dataset(
- self.zw,
- coalesced_loc + "/nodelist",
- (100000,),
- node_list.dtype,
- node_list.shape,
- )
- store[:] = node_list
-
- store = create_zarr_dataset(
- self.zw,
- coalesced_loc + "/partition_id",
- (100000,),
- str,
- (len(partition_id_list),),
- )
- store[:] = partition_id_list
-
- if fill_by_value is not None:
- color_values = self.get_cell_vals(
- from_assay=from_assay, cell_key=cell_key, k=fill_by_value
- )
- else:
- color_values = None
- plot_cluster_hierarchy(
- subgraph,
- clusts,
- color_values,
- force_ints_as_cats=force_ints_as_cats,
- width=width,
- lvr_factor=lvr_factor,
- vert_gap=vert_gap,
- min_node_size=min_node_size,
- node_size_multiplier=node_size_multiplier,
- node_power=node_power,
- root_size=root_size,
- non_leaf_size=non_leaf_size,
- show_labels=show_labels,
- fontsize=fontsize,
- root_color=root_color,
- non_leaf_color=non_leaf_color,
- cmap=cmap,
- color_key=color_key,
- edgecolors=edgecolors,
- edgewidth=edgewidth,
- alpha=alpha,
- figsize=figsize,
- ax=ax,
- show_fig=show_fig,
- savename=savename,
- save_dpi=save_dpi,
- )
-
- def plot_marker_heatmap(
- self,
- from_assay: Optional[str] = None,
- group_key: Optional[str] = None,
- cell_key: Optional[str] = None,
- topn: int = 5,
- log_transform: bool = True,
- vmin: float = -1,
- vmax: float = 2,
- savename: Optional[str] = None,
- save_dpi: int = 300,
- show_fig: bool = True,
- **heatmap_kwargs,
- ):
- """Displays a heatmap of top marker gene expression for the chosen
- groups (usually cell clusters).
-
- Z-scores are calculated for each marker gene before plotting them. The groups are subjected to hierarchical
- clustering to bring groups with similar expression pattern in proximity.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- group_key: Required parameter. This has to be a column name from cell metadata table. This column dictates
- how the cells will be grouped. This value should be same as used for `run_marker_search`
- cell_key: One of the columns from cell metadata table that indicates the cells to be used.
- Should be same as the one that was used in one of the `run_marker_search` calls for the given
- assay. The values in the chosen column should be boolean (Default value: 'I')
- topn: Number of markers to be displayed for each group in `group_key` column. The markers are sorted based
- on obtained scores by `run_marker_search`. (Default value: 5)
- log_transform: Whether to log-transform the values before displaying them in the heatmap.
- (Default value: True)
- vmin: z-scores lower than this value are ceiled to this value. (Default value: -1)
- vmax: z-scores higher than this value are floored to this value. (Default value: 2)
- savename: Path where the rendered figure is to be saved. The format of the saved image depends on
- the extension present in the parameter value. (Default value: None)
- save_dpi: DPI when saving figure. (Default value: 300)
- show_fig: Whether to render the figure and display it using plt.show() (Default value: True)
- **heatmap_kwargs: Keyword arguments to be forwarded to seaborn.clustermap.
-
- Returns:
- None
- """
- from ..plots import plot_heatmap
-
- assay = self._get_assay(from_assay)
- if group_key is None:
- raise ValueError("ERROR: Please provide a value for `group_key`")
- if cell_key is None:
- cell_key = "I"
- if "markers" not in self.zw[assay.name]:
- raise KeyError("ERROR: Please run `run_marker_search` first")
- slot_name = f"{cell_key}__{group_key}"
- if slot_name not in self.zw[assay.name]["markers"]:
- raise KeyError(
- f"ERROR: Please run `run_marker_search` first with {group_key} as `group_key` and "
- f"{cell_key} as `cell_key`"
- )
- g = self.zw[assay.name]["markers"][slot_name]
- feat_idx = []
- for i in g.keys():
- if "feature_index" in g[i]:
- feat_idx.extend(g[i]["feature_index"][:][:topn])
- if len(feat_idx) == 0:
- raise ValueError("ERROR: Marker list is empty for all the groups")
- feat_idx = np.array(sorted(set(feat_idx))).astype(int)
- cell_idx = np.array(assay.cells.active_index(cell_key))
- normed_data = assay.normed(
- cell_idx=cell_idx,
- feat_idx=feat_idx,
- log_transform=log_transform,
- )
- nc = normed_data.chunks[0]
- # FIXME: avoid conversion to dask dataframe here
- # Unfortunately doing this dask array in a loop is 10x slower
- normed_data = normed_data.to_dask_dataframe()
- groups = daskarr.from_array(
- assay.cells.fetch(group_key, cell_key), chunks=nc
- ).to_dask_dataframe()
- df = controlled_compute(normed_data.groupby(groups).mean(), self.nthreads)
- df = df.apply(lambda x: (x - x.mean()) / x.std(), axis=0)
- df.columns = assay.feats.fetch_all("names")[feat_idx]
- df = df.T
- # noinspection PyTypeChecker
- df[df < vmin] = vmin
- # noinspection PyTypeChecker
- df[df > vmax] = vmax
- plot_heatmap(
- df,
- savename=savename,
- save_dpi=save_dpi,
- show_fig=show_fig,
- **heatmap_kwargs,
- )
-
- def plot_pseudotime_heatmap(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- feature_cluster_key: Optional[str] = None,
- pseudotime_key: Optional[str] = None,
- show_features: Optional[list] = None,
- width: int = 5,
- height: int = 10,
- vmin: float = -2.0,
- vmax: float = 2.0,
- heatmap_cmap: Optional[str] = None,
- pseudotime_cmap: Optional[str] = None,
- clusterbar_cmap: Optional[str] = None,
- tick_fontsize: int = 10,
- axis_fontsize: int = 12,
- feature_label_fontsize: int = 12,
- savename: Optional[str] = None,
- save_dpi: int = 300,
- show_fig: bool = True,
- ) -> None:
- """Plot heatmap for the matrix calculated by running
- `run_pseudotime_aggregation`. The heatmap shows the cell bins ordered
- as per pseudotime values and features ordered by clusters. The clusters
- themselves are ordered in a fashion such that features that have mean
- maximum expression in early pseudotime appear first and the feature
- cluster that has mean maxima in the later pseudotime appears last.
-
- CAUTION: This make take a long time to render and consume large amount of memory if your data has too many
- features or if you create too many bins for cell ordering.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Required parameter. One of the columns from cell attribute table that indicates the cells to be
- used. The values in the chosen column should be boolean. This value should be same as used for
- `run_pseudotime_aggregation`. (Default value: The cell key used for latest graph created)
- feat_key: Required parameter. One of the columns from feature attribute table that indicates the cells to be
- used. The values in the chosen column should be boolean. This value should be same as used for
- `run_pseudotime_aggregation`. (Default value: The cell key used for latest graph created)
- feature_cluster_key: Required parameter. The name of column from feature attribute table that contains
- information about feature clusters.
- pseudotime_key: Required parameter. The name of the column from cell attribute table that contains the
- pseudotime values. This should be same as the one used from the relevant run of
- `run_pseudotime_aggregation`.
- show_features: A list of feature names to be highlighted/labelled on the heatmap.
- width: Width of the heatmap (Default value: 5)
- height: Height of the heatmap (Default value: 10)
- vmin: The minimum value to be displayed on the heatmap. The values lower than this will ceiled to this
- value. (Default value: -2.0)
- vmax: The maximum value to be displayed on the heatmap. The values higher than this will floored to this
- value. (Default value: 2.0)
- heatmap_cmap: Colormap for the heatmap (Default value: coolwarm)
- pseudotime_cmap: Colormap for the pseudotime bar. It should be some kind of continuous colormap.
- (Default value: cmocean.deep)
- clusterbar_cmap: Colormap for the cluster bar showing the span of each feature cluster.
- (Default value: tab20)
- tick_fontsize: Font size for cbar ticks (Default value: 10)
- axis_fontsize: Font size for labels along each axis(Default value: 12)
- feature_label_fontsize: Font size for feature labels on the heatmap (Default value: 12)
- savename: Path where the rendered figure is to be saved. The format of the saved image depends on
- the extension present in the parameter value. (Default value: None)
- save_dpi: DPI when saving figure (Default value: 300)
- show_fig: If, False then axes object is returned rather than rendering the plot (Default value: True)
-
- Returns: None
- """
-
- from ..plots import plot_annotated_heatmap
-
- assay = self._get_assay(from_assay)
- for i in [cell_key, feat_key, feature_cluster_key, feature_cluster_key]:
- if i is None:
- var_name = list(dict(i=i).keys())[0] # Trick to get variables own name
- raise ValueError(
- f"ERROR: Please provide a value for parameter `{var_name}`"
- )
-
- cell_ordering = assay.cells.fetch(pseudotime_key, key=cell_key)
- # noinspection PyProtectedMember
- cell_idx, feat_idx = assay._get_cell_feat_idx(cell_key, feat_key)
- hashes = [hash(tuple(x)) for x in (cell_idx, feat_idx, cell_ordering)]
- location = f"aggregated_{cell_key}_{feat_key}_{pseudotime_key}"
- if location not in assay.z:
- raise KeyError(
- f"ERROR: Could not find aggregated feature values at location '{location}' "
- f"Please make sure that you have run `run_pseudotime_aggregation` with the same values for "
- f"parameters: `cell_key`, `feat_key` and `pseudotime_key`"
- )
- else:
- if hashes != assay.z[location].attrs["hashes"]:
- raise ValueError(
- "ERROR: The values under one or more of these columns: `cell_key`, `feat_key` or/and "
- "`pseudotime_key have been updated after running `run_pseudotime_aggregation`"
- )
-
- da = daskarr.from_zarr(assay.z[location + "/data"], inline_array=True)
- feature_indices = assay.z[location + "/feature_indices"][:]
- da = da[: feature_indices.shape[0]]
-
- feature_clusters = assay.feats.fetch_all(feature_cluster_key)[feature_indices]
- feature_labels = assay.feats.fetch_all("names")[feature_indices]
-
- idx = np.argsort(feature_clusters)
- feature_clusters = feature_clusters[idx]
- feature_labels = feature_labels[idx]
- da = da.compute()[idx]
-
- ordering = assay.cells.fetch(pseudotime_key, key=cell_key)
-
- plot_annotated_heatmap(
- df=da,
- xbar_values=ordering,
- ybar_values=feature_clusters,
- display_row_labels=show_features,
- row_labels=feature_labels,
- width=width,
- height=height,
- vmin=vmin,
- vmax=vmax,
- heatmap_cmap=heatmap_cmap,
- xbar_cmap=pseudotime_cmap,
- ybar_cmap=clusterbar_cmap,
- tick_fontsize=tick_fontsize,
- axis_fontsize=axis_fontsize,
- row_label_fontsize=feature_label_fontsize,
- savename=savename,
- save_dpi=save_dpi,
- show_fig=show_fig,
- )
-
- def metric_lisi(
- self,
- label_colnames: Iterable[str],
- use_latest_knn: bool = True,
- from_assay: Optional[str] = None,
- knn_loc: Optional[str] = None,
- save_result: bool = False,
- return_lisi: bool = True,
- ) -> Optional[List[Tuple[str, np.ndarray]]]:
- """Calculate Local Inverse Simpson Index (LISI) scores for cell populations.
-
- LISI measures how well mixed different cell populations are in the local neighborhood
- of each cell. Higher scores indicate better mixing of different populations.
-
- Args:
- label_colnames: Column names from cell metadata containing population labels
- use_latest_knn: Whether to use the most recent KNN graph (default: True)
- from_assay: Name of assay to use if not using latest KNN
- knn_loc: Location of KNN graph if not using latest (default: None)
- save_result: Whether to save LISI scores to cell metadata (default: True)
- return_lisi: Whether to return LISI scores (default: False)
-
- Returns:
- If return_lisi is True, returns list of tuples containing:
- - Label column name
- - numpy array of LISI scores for that label
- If return_lisi is False, returns None
-
- Raises:
- ValueError: If using custom KNN graph but required parameters are missing
- KeyError: If label columns not found in cell metadata
-
- Notes:
- LISI scores are computed for each label column separately.
- Scores near 1 indicate cells grouped with similar labels.
- Higher scores indicate more mixing between different labels.
- """
-
- if use_latest_knn and knn_loc is None:
- knn_loc = self._get_latest_knn_loc(from_assay)
- cell_key = self.zw[self._load_default_assay()].attrs["latest_cell_key"]
- logger.info(f"Using the latest knn graph at location: {knn_loc}")
-
- else:
- if knn_loc is None:
- raise ValueError("Please provide values for the KNN graph location.")
- if knn_loc not in self.zw:
- raise ValueError(f"Could not find the knn graph at location: {knn_loc}")
-
- logger.info(f"Using the knn graph at location: {knn_loc}")
-
- knn = self.zw[knn_loc]
-
- distances = knn["distances"]
- indices = knn["indices"]
-
- try:
- metadata = self.cells.to_pandas_dataframe(
- columns=label_colnames + [cell_key]
- )
- metadata = metadata[metadata[cell_key]]
- except KeyError:
- raise KeyError(
- f"Could not find the column(s) {label_colnames} in the cell metadata table."
- )
-
- from ..metrics import compute_lisi
-
- lisi_scores = compute_lisi(distances, indices, metadata, label_colnames)
- # lisi_scores Shape -> (n_cells, n_labels)
- if save_result:
- for col, vals in zip(label_colnames, lisi_scores.T):
- col_name = f"lisi__{col}__{knn_loc.split('/')[-1]}"
- self.cells.insert(column_name=col_name, values=vals, overwrite=True)
-
- if return_lisi:
- return list(zip(label_colnames, lisi_scores.T))
- else:
- return None
-
- def metric_silhouette(
- self,
- use_latest_knn: bool = True,
- res_label: str = "leiden_cluster",
- from_assay: Optional[str] = None,
- knn_loc: Optional[str] = None,
- ) -> Optional[np.ndarray]:
- """Calculate modified silhouette scores for evaluating cluster separation.
-
- This implements a graph-based silhouette score that measures how similar cells
- are to their own cluster compared to the nearest neighboring cluster.
-
- Args:
- use_latest_knn: Whether to use most recent KNN graph (default: True)
- res_label: Column name containing cluster labels (default: "leiden_cluster")
- from_assay: Name of assay to use if not using latest KNN (default: None)
- knn_loc: Location of KNN graph if not using latest (default: None)
-
- Returns:
- numpy array of silhouette scores for each cluster, or None if computation fails
-
- Raises:
- ValueError: If using custom KNN graph but required parameters are missing
-
- Notes:
- Scores range from -1 to 1:
- - Near 1: Cluster is well-separated from neighboring clusters
- - Near 0: Cluster overlaps with neighboring clusters
- - Near -1: Cluster may be incorrectly assigned
-
- Implementation uses sampling for efficiency with large datasets.
- NaN values indicate clusters that couldn't be scored due to size constraints.
- """
-
- def compute_graph_feats(knn_loc: str):
- k = knn_loc.rsplit("/", 1)[-1].split("__")[-1]
- dims = knn_loc.rsplit("/", 2)[0].split("__")[-2]
- feat_key = knn_loc.split("/")[1].split("__")[-1]
- return k, dims, feat_key
-
- if from_assay is None:
- from_assay = self._load_default_assay()
-
- if use_latest_knn and knn_loc is None:
- knn_loc = self._get_latest_knn_loc(from_assay)
- k, dims, feat_key = compute_graph_feats(knn_loc)
- logger.info(
- f"Using the latest knn graph at location: {knn_loc} for assay: {from_assay}"
- )
-
- else:
- if knn_loc is None:
- raise ValueError("Please provide values for the KNN graph location.")
- if knn_loc not in self.zw:
- raise ValueError(f"Could not find the knn graph at location: {knn_loc}")
- k, dims, feat_key, from_assay = compute_graph_feats(knn_loc)
- logger.info(f"Using the knn graph at location: {knn_loc}")
-
- from ..metrics import knn_to_csr_matrix, silhouette_scoring
-
- isHarmonized = self.zw[knn_loc.rsplit("/", 1)[0]].attrs["isHarmonized"]
-
- batches = None
- if isHarmonized:
- batches = self.zw[knn_loc.rsplit("/", 2)[0] + "/harmonizedData"].attrs[
- "batches"
- ]
-
- ann_obj = self.make_graph(
- feat_key=feat_key,
- dims=dims,
- k=k,
- return_ann_object=True,
- harmonize=isHarmonized,
- batch_columns=batches,
- )
-
- graph = knn_to_csr_matrix(self.z[knn_loc].indices, self.z[knn_loc].distances)
- hvg_data = self.z[knn_loc.rsplit("/", 3)[0] + "/data"]
- scores = silhouette_scoring(
- self, ann_obj, graph, hvg_data, from_assay, res_label
+ return cast(Assay, getattr(self, assay_name))
+
+ def _create_temporary_datastore(
+ self,
+ zarr_loc: ZarrLocation,
+ *,
+ default_assay: str,
+ assay_types: dict[str, str],
+ nthreads: int,
+ ) -> "DataStore":
+ return DataStore(
+ zarr_loc,
+ default_assay=default_assay,
+ assay_types=assay_types,
+ nthreads=nthreads,
+ mem_budget=self.memoryBytes,
)
- return scores
-
- def metric_integration(
- self, batch_labels: List[str], metric: Literal["ari", "nmi"] = "ari"
- ) -> Optional[float]:
- """Calculate integration score between different batch labels.
-
- Measures how well aligned different batches are after integration by comparing
- their cluster assignments using standard clustering metrics.
-
- Args:
- batch_labels: List of column names containing batch labels to compare
- metric: Metric to use for comparison (default: "ari")
- - "ari": Adjusted Rand Index
- - "nmi": Normalized Mutual Information
-
- Returns:
- Integration score between 0 and 1, or None if metric is not recognized
- Higher scores indicate better alignment between batches
-
- Notes:
- ARI and NMI measure the agreement between different batch labelings:
- - Score near 1: Batches are well integrated
- - Score near 0: Batches show poor integration
-
- ARI is adjusted for chance and generally more stringent than NMI.
- """
- from ..metrics import integration_score
-
- batch_labels_vals = []
- for batch in batch_labels:
- vals = np.array(self.cells.fetch_all(batch))
- batch_labels_vals.append(vals)
-
- scores = integration_score(batch_labels_vals, metric)
- return scores
diff --git a/scarf/datastore/graph_datastore.py b/scarf/datastore/graph_datastore.py
index 7ba0b1df..3b37ba50 100644
--- a/scarf/datastore/graph_datastore.py
+++ b/scarf/datastore/graph_datastore.py
@@ -1,19 +1,23 @@
-import os
-from typing import Callable, List, Optional, Tuple, Union
-
-import numpy as np
-import pandas as pd
-from dask.array import from_zarr # type: ignore
-from loguru import logger
-from scipy.sparse import coo_matrix, csr_matrix
-
-from ..assay import Assay
-from ..utils import clean_array, show_dask_progress, system_call, tqdmbar
-from ..writers import create_zarr_dataset
+from typing import Any
+from weakref import WeakKeyDictionary
+
+from ..neighbors.stream import AnnStream
+from ._operations.clustering import _ClusteringOperationsMixin
+from ._operations.embeddings import _EmbeddingOperationsMixin
+from ._operations.graph import _GraphOperationsMixin
+from ._operations.mapping_reference import _MappingReferenceOperationsMixin
+from ._operations.trajectory import _TrajectoryOperationsMixin
from .base_datastore import BaseDataStore
-class GraphDataStore(BaseDataStore):
+class GraphDataStore(
+ _EmbeddingOperationsMixin,
+ _ClusteringOperationsMixin,
+ _TrajectoryOperationsMixin,
+ _MappingReferenceOperationsMixin,
+ _GraphOperationsMixin,
+ BaseDataStore,
+):
"""This class extends BaseDataStore by providing methods required to
generate a cell-cell neighbourhood graph.
@@ -26,2082 +30,6 @@ class GraphDataStore(BaseDataStore):
z: The Zarr file (directory) used for this datastore instance.
"""
- def __init__(self, **kwargs):
+ def __init__(self, **kwargs: Any) -> None:
super().__init__(**kwargs)
-
- @staticmethod
- def _choose_reduction_method(assay: Assay, reduction_method: str) -> str:
- """This is a convenience function to determine the linear dimension
- reduction method to be used for a given assay. It is uses a
- predetermined rule to make this determination.
-
- Args:
- assay: Assay object.
- reduction_method: Name of reduction method to use. It can be one from either: 'pca', 'lsi', 'auto'.
-
- Returns:
- The name of dimension reduction method to be used. Either 'pca' or 'lsi'
-
- Raises:
- ValueError: If `reduction_method` is not one of either 'pca', 'lsi', 'auto'
- """
- reduction_method = reduction_method.lower()
- if reduction_method not in ["pca", "lsi", "auto", "custom"]:
- raise ValueError(
- "ERROR: Please choose either 'pca' or 'lsi' as reduction method"
- )
- if reduction_method == "auto":
- assay_type = str(assay.__class__).split(".")[-1][:-2]
- if assay_type == "ATACassay":
- logger.debug("Using LSI for dimension reduction")
- reduction_method = "lsi"
- else:
- logger.debug("Using PCA for dimension reduction")
- reduction_method = "pca"
- return reduction_method
-
- def _set_graph_params(
- self,
- from_assay,
- cell_key,
- feat_key,
- log_transform=None,
- renormalize_subset=None,
- reduction_method="auto",
- dims=None,
- pca_cell_key=None,
- ann_metric=None,
- ann_efc=None,
- ann_ef=None,
- ann_m=None,
- rand_state=None,
- k=None,
- n_centroids=None,
- local_connectivity=None,
- bandwidth=None,
- ) -> tuple:
- """This function allows determination of values for the parameters of
- `make_graph` function. This function harbours the default values for
- each parameter. If parameter value is None, then before choosing the
- default, it tries to use the values from the latest iteration of the step
- within the same hierarchy tree. Find details for parameters in the
- `make_graph` method.
-
- Args:
- from_assay: Same as from_assay in make_graph
- cell_key: Same as cell_key in make_graph
- feat_key: Same as feat_key in make_graph
- log_transform: Same as log_transform in make_graph
- renormalize_subset: Same as renormalize_subset in make_graph
- reduction_method: Same as reduction_method in make_graph
- dims: Same as dims in make_graph
- pca_cell_key: Same as pca_cell_key in make_graph
- ann_metric: Same as ann_metric in make_graph
- ann_efc: Same as ann_efc in make_graph
- ann_ef: Same as ann_ef in make_graph
- ann_m: Same as ann_m in make_graph
- rand_state: Same as rand_state in make_graph
- k: Same as k in make_graph
- n_centroids: Same as n_centroids in make_graph
- local_connectivity: Same as local_connectivity in make_graph
- bandwidth: Same as bandwidth in make_graph
-
- Returns:
- Finalized values for the all the optional parameters in the same order
- """
-
- def log_message(category, name, value, custom_msg=None):
- """Convenience function to log variable usage messages for
- make_graph.
-
- Args:
- category:
- name:
- value:
- custom_msg:
-
- Returns:
- """
- msg = f"No value provided for parameter `{name}`. "
- if category == "default":
- msg += f"Will use default value: {value}"
- logger.debug(msg)
- elif category == "cached":
- msg += f"Will use previously used value: {value}"
- logger.debug(msg)
- else:
- if custom_msg is None:
- return False
- else:
- logger.info(custom_msg)
- return True
-
- default_values = {
- "log_transform": True,
- "renormalize_subset": True,
- "dims": 11,
- "ann_metric": "l2",
- "rand_state": 4466,
- "k": 11,
- "n_centroids": 1000,
- "local_connectivity": 1.0,
- "bandwidth": 1.5,
- }
-
- normed_loc = f"{from_assay}/normed__{cell_key}__{feat_key}"
- if log_transform is None or renormalize_subset is None:
- if normed_loc in self.zw and "subset_params" in self.zw[normed_loc].attrs:
- # This works in coordination with save_normalized_data
- subset_params = self.zw[normed_loc].attrs["subset_params"]
- c_log_transform, c_renormalize_subset = (
- subset_params["log_transform"],
- subset_params["renormalize_subset"],
- )
- else:
- c_log_transform, c_renormalize_subset = None, None
- if log_transform is None:
- if c_log_transform is not None:
- log_transform = bool(c_log_transform)
- log_message("cached", "log_transform", log_transform)
- else:
- log_transform = default_values["log_transform"]
- log_message("default", "log_transform", log_transform)
- if renormalize_subset is None:
- if c_renormalize_subset is not None:
- renormalize_subset = bool(c_renormalize_subset)
- log_message("cached", "renormalize_subset", renormalize_subset)
- else:
- renormalize_subset = default_values["renormalize_subset"]
- log_message("default", "renormalize_subset", renormalize_subset)
- log_transform = bool(log_transform)
- renormalize_subset = bool(renormalize_subset)
-
- if dims is None or pca_cell_key is None:
- if (
- normed_loc in self.zw
- and "latest_reduction" in self.zw[normed_loc].attrs
- ):
- reduction_loc = self.zw[normed_loc].attrs["latest_reduction"]
- c_dims, c_pca_cell_key = reduction_loc.rsplit("__", 2)[1:]
- else:
- c_dims, c_pca_cell_key = None, None
- if dims is None:
- if c_dims is not None:
- dims = int(c_dims)
- log_message("cached", "dims", dims)
- else:
- dims = default_values["dims"]
- log_message("default", "dims", dims)
- if pca_cell_key is None:
- if c_pca_cell_key is not None:
- pca_cell_key = c_pca_cell_key
- log_message("cached", "pca_cell_key", pca_cell_key)
- else:
- pca_cell_key = cell_key
- log_message("default", "pca_cell_key", pca_cell_key)
- else:
- if pca_cell_key not in self.cells.columns:
- raise ValueError(
- f"ERROR: `pca_use_cell_key` {pca_cell_key} does not exist in cell metadata"
- )
- if self.cells.get_dtype(pca_cell_key) != bool:
- raise TypeError(
- "ERROR: Type of `pca_use_cell_key` column in cell metadata should be `bool`"
- )
- dims = int(dims)
- reduction_method = self._choose_reduction_method(
- self._get_assay(from_assay), reduction_method
- )
- reduction_loc = (
- f"{normed_loc}/reduction__{reduction_method}__{dims}__{pca_cell_key}"
- )
-
- if (
- ann_metric is None
- or ann_efc is None
- or ann_ef is None
- or ann_m is None
- or rand_state is None
- ):
- if (
- reduction_loc in self.zw
- and "latest_ann" in self.zw[reduction_loc].attrs
- ):
- ann_loc = self.zw[reduction_loc].attrs["latest_ann"]
- (
- c_ann_metric,
- c_ann_efc,
- c_ann_ef,
- c_ann_m,
- c_rand_state,
- ) = ann_loc.rsplit("/", 1)[1].split("__")[1:]
- else:
- c_ann_metric, c_ann_efc, c_ann_ef, c_ann_m, c_rand_state = (
- None,
- None,
- None,
- None,
- None,
- )
- if ann_metric is None:
- if c_ann_metric is not None:
- ann_metric = c_ann_metric
- log_message("cached", "ann_metric", ann_metric)
- else:
- ann_metric = default_values["ann_metric"]
- log_message("default", "ann_metric", ann_metric)
- if ann_efc is None:
- if c_ann_efc is not None:
- ann_efc = int(c_ann_efc)
- log_message("cached", "ann_efc", ann_efc)
- else:
- ann_efc = None # Will be set after value for k is determined
- log_message("default", "ann_efc", f"min(100, max(k * 3, 50))")
- if ann_ef is None:
- if c_ann_ef is not None:
- ann_ef = int(c_ann_ef)
- log_message("cached", "ann_ef", ann_ef)
- else:
- ann_ef = None # Will be set after value for k is determined
- log_message("default", "ann_ef", f"min(100, max(k * 3, 50))")
- if ann_m is None:
- if c_ann_m is not None:
- ann_m = int(c_ann_m)
- log_message("cached", "ann_m", ann_m)
- else:
- ann_m = min(max(48, int(dims * 1.5)), 64)
- log_message("default", "ann_m", ann_m)
- if rand_state is None:
- if c_rand_state is not None:
- rand_state = int(c_rand_state)
- log_message("cached", "rand_state", rand_state)
- else:
- rand_state = default_values["rand_state"]
- log_message("default", "rand_state", rand_state)
- ann_metric = str(ann_metric)
- ann_m = int(ann_m)
- rand_state = int(rand_state)
-
- if k is None:
- if (
- reduction_loc in self.zw
- and "latest_ann" in self.zw[reduction_loc].attrs
- ):
- ann_loc = self.zw[reduction_loc].attrs["latest_ann"]
- knn_loc = self.zw[ann_loc].attrs["latest_knn"]
- k = int(knn_loc.rsplit("__", 1)[1])
- log_message("cached", "k", k)
- else:
- k = default_values["k"]
- log_message("default", "k", k)
- k = int(k)
- if ann_ef is None:
- ann_ef = min(100, max(k * 3, 50))
- ann_ef = int(ann_ef)
- if ann_efc is None:
- ann_efc = min(100, max(k * 3, 50))
- ann_efc = int(ann_efc)
- ann_loc = f"{reduction_loc}/ann__{ann_metric}__{ann_efc}__{ann_ef}__{ann_m}__{rand_state}"
- knn_loc = f"{ann_loc}/knn__{k}"
-
- if n_centroids is None:
- if (
- reduction_loc in self.zw
- and "latest_kmeans" in self.zw[reduction_loc].attrs
- ):
- kmeans_loc = self.zw[reduction_loc].attrs["latest_kmeans"]
- n_centroids = int(
- kmeans_loc.split("/")[-1].split("__")[1]
- ) # depends on param_joiner
- log_message("default", "n_centroids", n_centroids)
- else:
- # n_centroids = min(data.shape[0]/10, max(500, data.shape[0]/100))
- n_centroids = default_values["n_centroids"]
- log_message("default", "n_centroids", n_centroids)
- n_centroids = int(n_centroids)
-
- if local_connectivity is None or bandwidth is None:
- if knn_loc in self.zw and "latest_graph" in self.zw[knn_loc].attrs:
- graph_loc = self.zw[knn_loc].attrs["latest_graph"]
- c_local_connectivity, c_bandwidth = map(
- float, graph_loc.rsplit("/")[-1].split("__")[1:]
- )
- else:
- c_local_connectivity, c_bandwidth = None, None
- if local_connectivity is None:
- if c_local_connectivity is not None:
- local_connectivity = c_local_connectivity
- log_message("cached", "local_connectivity", local_connectivity)
- else:
- local_connectivity = default_values["local_connectivity"]
- log_message("default", "local_connectivity", local_connectivity)
- if bandwidth is None:
- if c_bandwidth is not None:
- bandwidth = c_bandwidth
- log_message("cached", "bandwidth", bandwidth)
- else:
- bandwidth = default_values["bandwidth"]
- log_message("default", "bandwidth", bandwidth)
- local_connectivity = float(local_connectivity)
- bandwidth = float(bandwidth)
-
- return (
- log_transform,
- renormalize_subset,
- reduction_method,
- dims,
- pca_cell_key,
- ann_metric,
- ann_efc,
- ann_ef,
- ann_m,
- rand_state,
- k,
- n_centroids,
- local_connectivity,
- bandwidth,
- )
-
- def _get_latest_keys(
- self,
- from_assay: Optional[str],
- cell_key: Optional[str],
- feat_key: Optional[str],
- ) -> Tuple[str, str, str]:
- if from_assay is None:
- from_assay = self._defaultAssay
- if cell_key is None:
- cell_key = self._get_latest_cell_key(from_assay)
- if feat_key is None:
- feat_key = self._get_latest_feat_key(from_assay)
- return from_assay, cell_key, feat_key
-
- def _get_latest_graph_loc(
- self, from_assay: str, cell_key: str, feat_key: str
- ) -> str:
- """Convenience function to identify location of the latest graph in the
- Zarr hierarchy.
-
- Args:
- from_assay: Name of the assay.
- cell_key: Cell key used to create the graph.
- feat_key: Feature key used to create the graph.
-
- Returns:
- Path of graph in the Zarr hierarchy
- """
- normed_loc = f"{from_assay}/normed__{cell_key}__{feat_key}"
- reduction_loc = self.zw[normed_loc].attrs["latest_reduction"]
- ann_loc = self.zw[reduction_loc].attrs["latest_ann"]
- knn_loc = self.zw[ann_loc].attrs["latest_knn"]
- return self.zw[knn_loc].attrs["latest_graph"]
-
- def _get_latest_knn_loc(self, from_assay: str = None) -> str:
- """Convenience function to identify location of the latest KNN graph in
- the Zarr hierarchy.
-
- Args:
- from_assay: Name of the assay.
-
- Returns:
- Path of KNN graph in the Zarr hierarchy
- """
- if from_assay is None:
- logger.info("Using default assay for KNN graph.")
- from_assay = self._load_default_assay()
-
- if from_assay not in self.assay_names:
- raise ValueError(f"ERROR: Assay {from_assay} does not exist")
-
- latest_cell_key = self.zw[from_assay].attrs["latest_cell_key"]
- latest_feat_key = self.zw[from_assay].attrs["latest_feat_key"]
- normed_loc = f"{from_assay}/normed__{latest_cell_key}__{latest_feat_key}"
- reduction_loc = self.zw[normed_loc].attrs["latest_reduction"]
- if "reduction" not in self.zw[reduction_loc]:
- raise ValueError(f"ERROR: PCA Reduction not found in {reduction_loc}")
- latest_ann = self.zw[reduction_loc].attrs["latest_ann"]
- ann_loc = self.zw[latest_ann]
- latest_knn = ann_loc.attrs["latest_knn"]
- return latest_knn
-
- def _get_ini_embed(
- self, from_assay: str, cell_key: str, feat_key: str, n_comps: int
- ) -> np.ndarray:
- """Runs PCA on kmeans cluster centers and ascribes the PC values to
- individual cells based on their cluster labels. This is used in
- `run_umap` and `run_tsne` for initial embedding of cells. Uses
- `rescale_array` to reduce the magnitude of extreme values.
-
- Args:
- from_assay: Name fo the assay for which Kmeans was fit.
- cell_key: Cell key used.
- feat_key: Feature key used.
- n_comps: Number of PC components to use
-
- Returns:
- Matrix with n_comps dimensions representing initial embedding of cells.
- """
- from sklearn.decomposition import PCA
-
- from ..utils import rescale_array
-
- normed_loc = f"{from_assay}/normed__{cell_key}__{feat_key}"
- reduction_loc = self.zw[normed_loc].attrs["latest_reduction"]
- kmeans_loc = self.zw[reduction_loc].attrs["latest_kmeans"]
- pc = PCA(n_components=n_comps).fit_transform(
- self.zw[kmeans_loc]["cluster_centers"][:]
- )
- for i in range(n_comps):
- pc[:, i] = rescale_array(pc[:, i])
- clusters = self.zw[kmeans_loc]["cluster_labels"][:].astype(np.uint32)
- return np.array([pc[x] for x in clusters]).astype(np.float32, order="C")
-
- def _get_graph_ncells_k(self, graph_loc: str) -> Tuple[int, int]:
- """
-
- Args:
- graph_loc:
-
- Returns:
-
- """
- if graph_loc.startswith(self._integratedGraphsLoc):
- attrs = self.zw[graph_loc].attrs
- return attrs["n_cells"], attrs["n_neighbors"]
- knn_loc = self.zw[graph_loc.rsplit("/", 1)[0]]
- return knn_loc["indices"].shape
-
- def _store_to_sparse(
- self, graph_loc: str, sparse_format: str = "csr", use_k: Optional[int] = None
- ) -> tuple:
- """
-
- Args:
- graph_loc:
- sparse_format:
- use_k:
-
- Returns:
-
- """
- logger.debug(f"Loading graph from location: {graph_loc}")
- store = self.zw[graph_loc]
- n_cells, k = self._get_graph_ncells_k(graph_loc)
- # TODO: can we have a progress bar for graph loading. Append to coo matrix?
- if use_k is None:
- use_k = k
- if use_k > k:
- use_k = k
- if use_k < 1:
- use_k = 1
- if use_k != k:
- indexer = np.tile([True] * use_k + [False] * (k - use_k), n_cells)
- else:
- indexer = None
- w, e = store["weights"][:], store["edges"][:]
- if indexer is not None:
- w, e = w[indexer], e[indexer]
- if sparse_format == "csr":
- return n_cells, csr_matrix(
- (w, (e[:, 0], e[:, 1])), shape=(n_cells, n_cells)
- )
- else:
- return n_cells, coo_matrix(
- (w, (e[:, 0], e[:, 1])), shape=(n_cells, n_cells)
- )
-
- def make_graph(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- pca_cell_key: Optional[str] = None,
- reduction_method: str = "auto",
- dims: Optional[int] = None,
- k: Optional[int] = None,
- ann_metric: Optional[str] = None,
- ann_efc: Optional[int] = None,
- ann_ef: Optional[int] = None,
- ann_m: Optional[int] = None,
- ann_parallel: bool = False,
- rand_state: Optional[int] = None,
- n_centroids: Optional[int] = None,
- batch_size: Optional[int] = None,
- log_transform: Optional[bool] = None,
- renormalize_subset: Optional[bool] = None,
- local_connectivity: Optional[float] = None,
- bandwidth: Optional[float] = None,
- update_keys: bool = True,
- return_ann_object: bool = False,
- custom_loadings: Optional[np.ndarray] = None,
- feat_scaling: bool = True,
- lsi_skip_first: bool = True,
- harmonize: bool = False,
- batch_columns: Optional[List[str]] = None,
- show_elbow_plot: bool = False,
- ann_index_fetcher: Optional[Callable] = None,
- ann_index_saver: Optional[Callable] = None,
- ):
- """Creates a cell neighbourhood graph. Performs following steps in the
- process:
-
- - Normalizes the data calling the `save_normalized_data` for the assay
- - instantiates ANNStream class which perform dimension reduction, feature scaling (optional) and fits ANN index
- - queries ANN index for nearest neighbours and saves the distances and indices of the neighbours
- - recalculates the distances to bound them into 0 and 1 (check out knn_utils module for details)
- - saves the indices and distances in sparse graph friendly form
- - fits a MiniBatch kmeans on the data
-
- The data for all the steps is saved in the Zarr in the following hierarchy which is organized based on data
- dependency. Parameter values for each step are incorporated into group names in the hierarchy::
-
- RNA
- ├── normed__I__hvgs
- │ ├── data (7648, 2000) float64 # Normalized data
- │ ├── mu (2000,) float64 # Means of normalized feature values
- │ ├── sigma (2000,) float64 # Std dev. of normalized feature values
- │ └── reduction__pca__31__I # Dimension reduction group
- │ ├── reduction (2000, 31) float64 # PCA loadings matrix
- │ ├── ann__l2__63__63__48__4466 # ANN group named with ANN parameters
- │ │ └── knn__21 # KNN group with value of k in name
- │ │ ├── distances (7648, 21) float64 # Raw distance matrix for k neighbours
- │ │ ├── indices (7648, 21) uint64 # Indices for k neighbours
- │ │ └── graph__1.0__1.5 # sparse graph with continuous form distance values
- │ │ ├── edges (160608, 2) uint64
- │ │ └── weights (160608,) float64
- │ └── kmeans__100__4466 # Kmeans groups
- │ ├── cluster_centers (100, 31) float64 # Centroid matrix
- │ └── cluster_labels (7648,) float64 # Cluster labels for cells
- ...
-
- The most recent child of each hierarchy node is noted for quick retrieval and in cases where multiple child
- nodes exist. Parameters starting with `ann` are forwarded to HNSWlib. More details about these parameters can
- be found here: https://github.com/nmslib/hnswlib/blob/master/ALGO_PARAMS.md
-
- Args:
- from_assay: Assay to use for graph creation. If no value is provided then `defaultAssay` will be used
- cell_key: Cells to use for graph creation. By default all cells with True value in 'I' will be used.
- The provided value for `cell_key` should be a column in cell metadata table with boolean values.
- feat_key: Features to use for graph creation. It is a required parameter. We have chosen not to set this
- to 'I' by default because this might lead to usage of too many features and may lead to poor
- results. The value for `feat_key` should be a column in feature metadata from the `from_assay`
- assay and should be boolean type.
- pca_cell_key: Name of a column from cell metadata table. This column should be boolean type. If no value is
- provided then the value is set to same as `cell_key` which means all the cells in the
- normalized data will be used for fitting the pca. This parameter, hence, basically provides a
- mechanism to subset the normalized data only for PCA fitting step. This parameter can be
- useful, for example, the data has cells from multiple replicates which wont merge together, in
- which case the `pca_cell_key` can be used to fit PCA on cells from only one of the replicate.
- reduction_method: Method to use for linear dimension reduction. Could be either 'pca', 'lsi' or 'auto'. In
- case of 'auto' `_choose_reduction_method` will be used to determine the best reduction
- type for the assay.
- dims: Number of top reduced dimensions to use (Default value: 11)
- k: Number of nearest neighbours to query for each cell (Default value: 11)
- ann_metric: Refer to HNSWlib link above (Default value: 'l2')
- ann_efc: Refer to HNSWlib link above (Default value: min(100, max(k * 3, 50)))
- ann_ef: Refer to HNSWlib link above (Default value: min(100, max(k * 3, 50)))
- ann_m: Refer to HNSWlib link above (Default value: min(max(48, int(dims * 1.5)), 64) )
- ann_parallel: If True, then ANN graph is created in parallel mode using DataStore.nthreads number of
- threads. Results obtained in parallel mode will not be reproducible. (Default: False)
- rand_state: Random seed number (Default value: 4466)
- n_centroids: Number of centroids for Kmeans clustering. As a general indication, have a value of 1+ for
- every 100 cells. Small (<2000 cells) and very small (<500 cells) use a ballpark number for max
- expected number of clusters (Default value: 500). The results of kmeans clustering are only
- used to provide initial embedding for UMAP and tSNE. (Default value: 500)
- batch_size: Number of cells in a batch. This number is guided by number of features being used and the
- amount of available free memory. Though the full data is already divided into chunks, however,
- if only a fraction of features is being used in the normalized dataset, then the chunk size
- can be increased to speed up the computation (i.e. PCA fitting and ANN index building).
- (Default value: 1000)
- log_transform: If True, then the normalized data is log-transformed (only affects RNAassay type assays).
- (Default value: True)
- renormalize_subset: If True, then the data is normalized using only those features that are True in
- `feat_key` column rather using total expression of all features in a cell (only affects
- RNAassay type assays). (Default value: True)
- local_connectivity: This parameter is forwarded to `smooth_knn_dist` function from UMAP package. Higher
- value will push distribution of edge weights towards terminal values (binary like).
- Lower values will accumulate edge weights around the mean produced by `bandwidth`
- parameter. (Default value: 1.0)
- bandwidth: This parameter is forwarded to `smooth_knn_dist` function from UMAP package. Higher value will
- push the mean of distribution of graph edge weights towards right. (Default value: 1.5). Read
- more about `smooth_knn_dist` function here:
- https://umap-learn.readthedocs.io/en/latest/api.html#umap.umap_.smooth_knn_dist
- update_keys: If True (default) then `latest_feat_key` zarr attribute of the assay will be updated.
- Choose False if you are experimenting with a `feat_key` do not want to override existing
- `latest_feat_key` and by extension `latest_graph`.
- return_ann_object: If True then returns the ANNStream object. This allows one to directly interact with the
- PCA transformer and HNSWlib index. Check out ANNStream documentation to know more.
- (Default: False)
- custom_loadings: Custom loadings/transformer for linear dimension reduction. If provided, should have a form
- (d x p) where d is same the number of active features in feat_key and p is the number of
- reduced dimensions. `dims` parameter is ignored when this is provided.
- (Default value: None)
- feat_scaling: If True (default) then the feature will be z-scaled otherwise not. It is highly recommended
- that this is kept as True unless you know what you are doing. `feat_scaling` is internally
- turned off when during cross sample mapping using CORAL normalized values are being used.
- Read more about this in `run_mapping` method.
- lsi_skip_first: Whether to remove the first LSI dimension when using ATAC-Seq data.
- harmonize: Run Harmony to perform batch integration
- batch_columns: Columns in cell metadata table to use for batch integration
- show_elbow_plot: If True, then an elbow plot is shown when PCA is fitted to the data. Not shown when using
- existing PCA loadings or custom loadings. (Default value: False)
- ann_index_fetcher:
- ann_index_saver:
-
- Returns:
- Either None or `AnnStream` object
- """
- from ..ann import AnnStream
-
- if from_assay is None:
- from_assay = self._defaultAssay
- assay = self._get_assay(from_assay)
- if batch_size is None:
- batch_size = assay.rawData.chunksize[0]
- if cell_key is None:
- cell_key = "I"
- if feat_key is None:
- bool_cols = [
- x.split("__", 1)
- for x in assay.feats.columns
- if assay.feats.get_dtype(x) == bool and x != "I"
- ]
- bool_cols = [f"{x[1]}({x[0]})" for x in bool_cols]
- bool_cols = " ".join(map(str, bool_cols))
- raise ValueError(
- "ERROR: You have to choose which features that should be used for graph construction. "
- "Ideally you should have performed a feature selection step before making this graph. "
- "Feature selection step adds a column to your feature table. \n"
- "You have following boolean columns in the feature "
- f"metadata of assay {from_assay} which you can choose from: {bool_cols}\n The values in "
- f"brackets indicate the cell_key for which the feat_key is available. Choosing 'I' "
- f"as `feat_key` means that you will use all the genes for graph creation."
- )
- if custom_loadings is not None:
- reduction_method = "custom"
- dims = custom_loadings.shape[1]
- logger.info(
- f"`dims` parameter and its default value ignored as using custom loadings "
- f"with {dims} dims"
- )
-
- (
- log_transform,
- renormalize_subset,
- reduction_method,
- dims,
- pca_cell_key,
- ann_metric,
- ann_efc,
- ann_ef,
- ann_m,
- rand_state,
- k,
- n_centroids,
- local_connectivity,
- bandwidth,
- ) = self._set_graph_params(
- from_assay,
- cell_key,
- feat_key,
- log_transform,
- renormalize_subset,
- reduction_method,
- dims,
- pca_cell_key,
- ann_metric,
- ann_efc,
- ann_ef,
- ann_m,
- rand_state,
- k,
- n_centroids,
- local_connectivity,
- bandwidth,
- )
- batches = None
- if harmonize:
- if batch_columns is None:
- raise ValueError(f"Harmonization requested but no batches provided")
- else:
- if isinstance(batch_columns, list) is False:
- raise ValueError(
- f"batches must be a list of columns in cell metadata that represent batches"
- )
- batches = pd.DataFrame(
- {
- x: self.cells.fetch(x, key=cell_key).astype(object)
- for x in batch_columns
- }
- )
-
- normed_loc = f"{from_assay}/normed__{cell_key}__{feat_key}"
- reduction_loc = (
- f"{normed_loc}/reduction__{reduction_method}__{dims}__{pca_cell_key}"
- )
- ann_loc = f"{reduction_loc}/ann__{ann_metric}__{ann_efc}__{ann_ef}__{ann_m}__{rand_state}"
- knn_loc = f"{ann_loc}/knn__{k}"
- kmeans_loc = f"{reduction_loc}/kmeans__{n_centroids}__{rand_state}"
- graph_loc = f"{knn_loc}/graph__{local_connectivity}__{bandwidth}"
-
- data = assay.save_normalized_data(
- cell_key,
- feat_key,
- batch_size,
- normed_loc.split("/")[-1],
- log_transform,
- renormalize_subset,
- update_keys,
- )
- if custom_loadings is not None and data.shape[1] != custom_loadings.shape[0]:
- raise ValueError(
- f"Provided custom loadings has {custom_loadings.shape[0]} features while the data "
- f"has {data.shape[1]} features."
- )
- loadings = None
- fit_kmeans = True
- mu, sigma = np.ndarray([]), np.ndarray([])
- use_for_pca = self.cells.fetch(pca_cell_key, key=cell_key)
- harmonized_data = None
-
- if reduction_method in ["pca", "manual"]:
- if "mu" in self.zw[normed_loc]:
- mu = self.zw[normed_loc]["mu"][:]
- else:
- mu = clean_array(
- show_dask_progress(
- data.mean(axis=0),
- "Calculating mean of norm. data",
- self.nthreads,
- )
- )
- g = create_zarr_dataset(
- self.zw[normed_loc], "mu", (100000,), "f8", mu.shape
- )
- g[:] = mu
- if "sigma" in self.zw[normed_loc]:
- sigma = self.zw[normed_loc]["sigma"][:]
- else:
- sigma = clean_array(
- show_dask_progress(
- data.std(axis=0),
- "Calculating std. dev. of norm. data",
- self.nthreads,
- ),
- 1,
- )
- g = create_zarr_dataset(
- self.zw[normed_loc], "sigma", (100000,), "f8", sigma.shape
- )
- g[:] = sigma
- if reduction_loc in self.zw:
- if "reduction" in self.zw[reduction_loc]:
- loadings = self.zw[reduction_loc]["reduction"][:]
- if data.shape[1] != loadings.shape[0]:
- logger.warning(
- "Consistency breached in loading pre-cached loadings. Will perform fresh reduction."
- )
- loadings = None
- del self.zw[reduction_loc]
- if harmonize and "harmonizedData" in self.zw[reduction_loc]:
- if "batches" in self.zw[reduction_loc]["harmonizedData"].attrs:
- if (
- self.zw[reduction_loc]["harmonizedData"].attrs["batches"]
- == batch_columns
- ):
- harmonized_data = from_zarr(
- self.zw[reduction_loc]["harmonizedData"], inline_array=True
- )
-
- if custom_loadings is None:
- if loadings is not None:
- logger.info(
- f"Using existing loadings for {reduction_method} with {dims} dims"
- )
- else:
- if loadings is not None and np.array_equal(loadings, custom_loadings):
- logger.info("Custom loadings same as used before. Loading from cache")
- else:
- loadings = custom_loadings
- logger.info(
- f"Using custom loadings with {dims} dims. Will overwrite any "
- f"previously used custom loadings"
- )
- if reduction_loc in self.zw:
- del self.zw[reduction_loc]
- if harmonized_data is not None:
- # Moved this statement here to print after the status of dim loadings
- logger.info(f"Using existing harmonized data with {dims} dims")
-
- ann_idx = None
- if ann_loc in self.zw:
- reset_ann = False
- if "isHarmonized" in self.zw[ann_loc].attrs:
- if self.zw[ann_loc].attrs["isHarmonized"]:
- if harmonize is False:
- reset_ann = True
- if harmonized_data is None:
- reset_ann = True
- else:
- if harmonize:
- reset_ann = True
- else:
- if harmonize: # Mostly for backward compatibility
- reset_ann = True
-
- if reset_ann:
- del self.zw[ann_loc]
- else:
- if ann_index_fetcher is None:
- if hasattr(self.zw.chunk_store, "path"):
- ann_index_fn = os.path.join(
- self.zw.chunk_store.path, ann_loc, "ann_idx"
- )
- else:
- ann_index_fn = None
- logger.warning(
- f"No custom `ann_index_fetcher` provided and zarr path is not local"
- )
- else:
- # noinspection PyBroadException
- try:
- ann_index_fn = ann_index_fetcher(ann_loc)
- except:
- ann_index_fn = None
- logger.warning(f"Custom `ann_index_fetcher` failed")
- if ann_index_fn is None or os.path.exists(ann_index_fn) is False:
- logger.warning(f"Ann index file expected but could not be found")
- else:
- import hnswlib
-
- temp = dims if dims > 0 else data.shape[1]
- ann_idx = hnswlib.Index(space=ann_metric, dim=temp)
- ann_idx.load_index(ann_index_fn)
- # TODO: check if ANN is index is trained with expected number of cells.
- logger.info(f"Using existing ANN index")
-
- if kmeans_loc in self.zw:
- fit_kmeans = False
- logger.info(f"using existing kmeans cluster centers")
- disable_scaling = True if feat_scaling is False else False
-
- # TODO: expose LSImodel parameters
- ann_obj = AnnStream(
- data=data,
- k=k,
- n_cluster=n_centroids,
- reduction_method=reduction_method,
- dims=dims,
- loadings=loadings,
- use_for_pca=use_for_pca,
- mu=mu,
- sigma=sigma,
- ann_metric=ann_metric,
- ann_efc=ann_efc,
- ann_ef=ann_ef,
- ann_m=ann_m,
- nthreads=self.nthreads,
- ann_parallel=ann_parallel,
- rand_state=rand_state,
- do_kmeans_fit=fit_kmeans,
- disable_scaling=disable_scaling,
- ann_idx=ann_idx,
- lsi_skip_first=lsi_skip_first,
- lsi_params={},
- harmonize=harmonize,
- harmonized_data=harmonized_data,
- batches=batches,
- )
-
- if reduction_loc not in self.zw:
- logger.debug(f"Saving loadings to {reduction_loc}")
- self.zw.create_group(reduction_loc, overwrite=True)
- if ann_obj.loadings is not None:
- # can be None when no dimred is performed
- g = create_zarr_dataset(
- self.zw[reduction_loc],
- "reduction",
- data.chunksize,
- "f8",
- ann_obj.loadings.shape,
- )
- g[:, :] = ann_obj.loadings
- if harmonize and harmonized_data is None and ann_obj.harmonizedData is not None:
- g = create_zarr_dataset(
- self.zw[reduction_loc],
- "harmonizedData",
- data.chunksize,
- "f8",
- ann_obj.harmonizedData.shape,
- )
- g[:] = ann_obj.harmonizedData
- g.attrs["batches"] = batch_columns
- if ann_loc not in self.zw:
- logger.debug(f"Saving ANN index to {ann_loc}")
- self.zw.create_group(ann_loc, overwrite=True)
- if ann_idx is None:
- if ann_index_saver is None:
- if hasattr(self.zw.chunk_store, "path"):
- ann_obj.annIdx.save_index(
- os.path.join(self.zw.chunk_store.path, ann_loc, "ann_idx")
- )
- else:
- logger.warning(
- "No custom `ann_index_saver` provided and local path is unknown"
- )
- if ann_index_saver is not None:
- try:
- ann_index_saver(ann_obj.annIdx, ann_loc)
- except:
- logger.warning("Custom `ann_index_saver` failed")
-
- if fit_kmeans:
- logger.debug(f"Saving kmeans clusters to {kmeans_loc}")
- self.zw.create_group(kmeans_loc, overwrite=True)
- g = create_zarr_dataset(
- self.zw[kmeans_loc],
- "cluster_centers",
- (1000, 1000),
- "f8",
- ann_obj.kmeans.cluster_centers_.shape,
- )
- g[:, :] = ann_obj.kmeans.cluster_centers_
- g = create_zarr_dataset(
- self.zw[kmeans_loc],
- "cluster_labels",
- (100000,),
- "f8",
- ann_obj.clusterLabels.shape,
- )
- g[:] = ann_obj.clusterLabels
- if knn_loc in self.zw and graph_loc in self.zw:
- logger.info(f"KNN graph already exists will not recompute.")
- else:
- from ..knn_utils import self_query_knn, smoothen_dists
-
- recall = None
- if knn_loc not in self.zw:
- recall = self_query_knn(
- ann_obj=ann_obj,
- store=self.zw.create_group(knn_loc, overwrite=True),
- chunk_size=batch_size,
- nthreads=self.nthreads,
- )
- recall = "%.2f" % recall
-
- smoothen_dists(
- self.zw.create_group(graph_loc, overwrite=True),
- self.zw[knn_loc]["indices"],
- self.zw[knn_loc]["distances"],
- local_connectivity,
- bandwidth,
- batch_size,
- )
- if recall is not None:
- logger.info(f"ANN recall: {recall}%")
-
- self.zw[normed_loc].attrs["latest_reduction"] = reduction_loc
- self.zw[reduction_loc].attrs["latest_ann"] = ann_loc
- self.zw[reduction_loc].attrs["latest_kmeans"] = kmeans_loc
- self.zw[ann_loc].attrs["isHarmonized"] = harmonize
- self.zw[ann_loc].attrs["latest_knn"] = knn_loc
- self.zw[knn_loc].attrs["latest_graph"] = graph_loc
- if return_ann_object:
- return ann_obj
- if show_elbow_plot:
- from ..plots import plot_elbow
-
- try:
- var_exp = 100 * ann_obj._pca.explained_variance_ratio_
- except AttributeError:
- logger.warning("PCA was not fitted so not showing an Elbow plot")
- else:
- plot_elbow(var_exp)
- return None
-
- def load_graph(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- symmetric: Optional[bool] = None,
- upper_only: Optional[bool] = None,
- use_k: Optional[int] = None,
- graph_loc: Optional[str] = None,
- ) -> csr_matrix:
- """Load the cell neighbourhood as a scipy sparse matrix.
-
- Args:
- from_assay: Name of the assay. If None then the default assay is used.
- cell_key: Cell key used to create the graph. If None then the latest feature key used for creating a
- KNN graph is used.
- feat_key: Feature key used to create the graph. If None then the latest feature key used for creating a
- KNN graph is used.
- symmetric: If True, makes the graph symmetric by adding it to its transpose.
- upper_only: If True, then only the values from upper triangular of the matrix are returned. This is only
- used when symmetric is True.
- use_k: Number of top k-nearest neighbours to keep in the graph. This value must be greater than 0 and less
- the parameter k used. By default, all neighbours are used. (Default value: None)
- graph_loc: Zarr hierarchy where the graph is stored. If no value is provided then graph location is
- obtained from `_get_latest_graph_loc` method.
-
- Returns:
- A scipy sparse matrix representing cell neighbourhood graph.
- """
-
- def symmetrize(g):
- t = g + g.T
- t = t - g.multiply(g.T)
- return t
-
- from scipy.sparse import triu
-
- from_assay, cell_key, feat_key = self._get_latest_keys(
- from_assay, cell_key, feat_key
- )
-
- if graph_loc is None:
- graph_loc = self._get_latest_graph_loc(from_assay, cell_key, feat_key)
- if graph_loc not in self.zw:
- raise ValueError(
- f"{graph_loc} not found in zarr location. "
- f"Run `make_graph` for assay {from_assay}"
- )
- n_cells, graph = self._store_to_sparse(graph_loc, "csr", use_k)
- if symmetric is True:
- graph = symmetrize(graph)
- if upper_only is True:
- graph = triu(graph)
- return graph
- # idx = None
- # if min_edge_weight > 0:
- # idx = graph.data > min_edge_weight
- # # Following if-else block is for purpose for improving performance when no filtering is performed.
- # if idx is None:
- # return graph
- # elif idx.sum() == graph.data.shape[0]:
- # return graph
- # else:
- # graph = graph.tocoo()
- # return csr_matrix((graph.data[idx], (graph.row[idx], graph.col[idx])), shape=(n_cells, n_cells))
-
- def run_tsne(
- self,
- from_assay: str = None,
- cell_key: str = None,
- feat_key: str = None,
- symmetric_graph: bool = False,
- graph_upper_only: bool = False,
- ini_embed: np.ndarray = None,
- tsne_dims: int = 2,
- lambda_scale: float = 1.0,
- max_iter: int = 500,
- early_iter: int = 200,
- alpha: int = 10,
- box_h: float = 0.7,
- temp_file_loc: str = ".",
- label: str = "tSNE",
- verbose: bool = True,
- parallel: bool = False,
- nthreads: int = None,
- ) -> None:
- """Run SGtSNE-pi (Read more here:
- https://github.com/fcdimitr/sgtsnepi/tree/v1.0.1). This is an
- implementation of tSNE that runs directly on graph structures. We use
- the graphs generated by `make_graph` method to create a layout of cells
- using tSNE algorithm. This function makes a system call to sgtSNE
- binary. To get a better understanding of how the parameters affect the
- embedding, check this out: http://t-sne-pi.cs.duke.edu/
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
- used feature for the given assay will be used.
- symmetric_graph: This parameter is forwarded to `load_graph` and is same as there. (Default value: False)
- graph_upper_only: This parameter is forwarded to `load_graph` and is same as there. (Default value: False)
- ini_embed: Initial embedding coordinates for the cells in cell_key. Should have the same number of columns
- as tsne_dims. If not value is provided then the initial embedding is obtained using
- `get_ini_embed`.
- tsne_dims: Number of tSNE dimensions to compute (Default value: 2)
- lambda_scale: λ rescaling parameter (Default value: 1.0)
- max_iter: Maximum number of iterations (Default value: 500)
- early_iter: Number of early exaggeration iterations (Default value: 200)
- alpha: Early exaggeration multiplier (Default value: 10)
- box_h: Grid side length (accuracy control). Lower values might drastically slow down
- the algorithm (Default value: 0.7)
- temp_file_loc: Location of temporary file. By default, these files will be created in the current working
- directory. These files are deleted before the method returns.
- label: base label for tSNE dimensions in the cell metadata column (Default value: 'tSNE')
- verbose: If True (default) then the full log from SGtSNEpi algorithm is shown.
- parallel: Whether to run tSNE in parallel mode. Setting value to True will use `nthreads` threads.
- The results are not reproducible in parallel mode. (Default value: False)
- nthreads: If parallel=True then this number of threads will be used to run tSNE. By default the `nthreads`
- attribute of the class is used. (Default value: None)
-
- Returns:
- """
- from uuid import uuid4
- from ..knn_utils import export_knn_to_mtx
- from pathlib import Path
- import sys
-
- if sys.platform not in ["posix", "linux"]:
- logger.error(f"{sys.platform} operating system is currently not supported.")
- return None
-
- from_assay, cell_key, feat_key = self._get_latest_keys(
- from_assay, cell_key, feat_key
- )
-
- uid = str(uuid4())
- knn_mtx_fn = Path(temp_file_loc, f"{uid}.mtx").resolve()
- graph = self.load_graph(
- from_assay=from_assay,
- cell_key=cell_key,
- feat_key=feat_key,
- symmetric=symmetric_graph,
- upper_only=graph_upper_only,
- )
- export_knn_to_mtx(str(knn_mtx_fn), graph)
-
- ini_emb_fn = Path(temp_file_loc, f"{uid}.txt").resolve()
- with open(ini_emb_fn, "w") as h:
- if ini_embed is None:
- ini_embed = self._get_ini_embed(
- from_assay, cell_key, feat_key, tsne_dims
- ).flatten()
- else:
- if ini_embed.shape != (graph.shape[0], tsne_dims):
- raise ValueError(
- "ERROR: Provided initial embedding does not shape required shape: "
- f"{(graph.shape[0], tsne_dims)}"
- )
- h.write("\n".join(map(str, ini_embed)))
- out_fn = Path(temp_file_loc, f"{uid}_output.txt").resolve()
- if parallel:
- if nthreads is None:
- nthreads = self.nthreads
- else:
- assert type(nthreads) == int
- else:
- nthreads = 1
- cmd = (
- f"sgtsne -m {max_iter} -l {lambda_scale} -d {tsne_dims} -e {early_iter} -p {nthreads} -a {alpha}"
- f" -h {box_h} -i {ini_emb_fn} -o {out_fn} {knn_mtx_fn}"
- )
- if verbose:
- system_call(cmd)
- else:
- os.system(cmd)
- try:
- emb = pd.read_csv(out_fn, header=None, sep=" ")[
- list(range(tsne_dims))
- ].values.T
- for i in range(tsne_dims):
- self.cells.insert(
- self._col_renamer(from_assay, cell_key, f"{label}{i + 1}"),
- emb[i],
- key=cell_key,
- overwrite=True,
- )
- for fn in [out_fn, knn_mtx_fn, ini_emb_fn]:
- Path.unlink(fn)
- except FileNotFoundError:
- logger.error(
- "SG-tSNE failed, possibly due to missing libraries or file permissions. SG-tSNE currently "
- "fails on readthedocs"
- )
- for fn in [knn_mtx_fn, ini_emb_fn]:
- Path.unlink(fn)
-
- def run_umap(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- symmetric_graph: Optional[bool] = False,
- graph_upper_only: Optional[bool] = False,
- ini_embed: Optional[np.ndarray] = None,
- umap_dims: int = 2,
- spread: float = 2.0,
- min_dist: float = 1,
- n_epochs: int = 300,
- repulsion_strength: float = 1.0,
- initial_alpha: float = 1.0,
- negative_sample_rate: float = 5,
- use_density_map: bool = False,
- dens_lambda: float = 2.0,
- dens_frac: float = 0.3,
- dens_var_shift: float = 0.1,
- random_seed: int = 4444,
- label: str = "UMAP",
- integrated_graph: Optional[str] = None,
- parallel: bool = False,
- nthreads: Optional[int] = None,
- ) -> None:
- """Runs UMAP algorithm using the precomputed cell-neighbourhood graph.
- The calculated UMAP coordinates are saved in the cell metadata table.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
- used feature for the given assay will be used.
- symmetric_graph: This parameter is forwarded to `load_graph` and is same as there. (Default value: False)
- graph_upper_only: This parameter is forwarded to `load_graph` and is same as there. (Default value: False)
- ini_embed: Initial embedding coordinates for the cells in cell_key. Should have the same number of columns
- as umap_dims. If not value is provided then the initial embedding is obtained using
- `get_ini_embed`.
- umap_dims: Number of dimensions of UMAP embedding (Default value: 2)
- spread: Same as spread in UMAP package. The effective scale of embedded points. In combination with
- ``min_dist`` this determines how clustered/clumped the embedded points are.
- min_dist: Same as min_dist in UMAP package. The effective minimum distance between embedded points.
- Smaller values will result in a more clustered/clumped embedding where nearby points on the
- manifold are drawn closer together, while larger values will result on a more even dispersal of
- points. The value should be set relative to the ``spread`` value, which determines the scale at
- which embedded points will be spread out. (Default value: 1)
- n_epochs: Same as n_epochs in UMAP package. The number of epochs to be used in optimizing the
- low dimensional embedding. Larger values may result in more accurate embeddings.
- (Default value: 300)
- repulsion_strength: Same as repulsion_strength in UMAP package. Weighting applied to negative samples in
- low dimensional embedding optimization. Values higher than one will result in greater
- weight being given to negative samples. (Default value: 1.0)
- initial_alpha: Same as learning_rate in UMAP package. The initial learning rate for the embedding
- optimization. (Default value: 1.0)
- negative_sample_rate: Same as negative_sample_rate in UMAP package. The number of negative samples to
- select per positive sample in the optimization process. Increasing this value will
- result in greater repulsive force being applied, greater optimization cost, but
- slightly more accuracy. (Default value: 5)
- use_density_map:
- dens_lambda:
- dens_frac:
- dens_var_shift:
- random_seed: (Default value: 4444)
- label: base label for UMAP dimensions in the cell metadata column (Default value: 'UMAP')
- integrated_graph:
- parallel: Whether to run UMAP in parallel mode. Setting value to True will use `nthreads` threads.
- The results are not reproducible in parallel mode. (Default value: False)
- nthreads: If parallel=True then this number of threads will be used to run UMAP. By default, the `nthreads`
- attribute of the class is used. (Default value: None)
-
- Returns:
- """
- from ..umap import fit_transform
- from ..utils import get_log_level
-
- from_assay, cell_key, feat_key = self._get_latest_keys(
- from_assay, cell_key, feat_key
- )
- graph_loc = None
- if integrated_graph is not None:
- graph_loc = f"{self._integratedGraphsLoc}/{integrated_graph}"
- if graph_loc not in self.zw:
- raise KeyError(
- f"ERROR: An integrated graph with label: {integrated_graph} does not exist"
- )
- graph = self.load_graph(
- from_assay=from_assay,
- cell_key=cell_key,
- feat_key=feat_key,
- symmetric=symmetric_graph,
- upper_only=graph_upper_only,
- graph_loc=graph_loc,
- )
-
- if ini_embed is None:
- ini_embed = self._get_ini_embed(from_assay, cell_key, feat_key, umap_dims)
- if nthreads is None:
- nthreads = self.nthreads
- verbose = False
- if get_log_level() <= 20:
- verbose = True
-
- if use_density_map:
- if integrated_graph is not None:
- logger.warning(
- "DensMap is not available for integrated graphs. Will run without UMAP without DensMap"
- )
- graph_loc = self._get_latest_graph_loc(from_assay, cell_key, feat_key)
- knn_loc = graph_loc.rsplit("/", 1)[0]
- logger.trace(f"Loading KNN dists and indices from {knn_loc}")
- dists = self.zw[knn_loc].distances[:]
- indices = self.zw[knn_loc].indices[:]
- dmat = csr_matrix(
- (
- dists.flatten(),
- (
- np.repeat(range(indices.shape[0]), indices.shape[1]),
- indices.flatten(),
- ),
- ),
- shape=(indices.shape[0], indices.shape[0]),
- )
- # dmat = dmat.maximum(dmat.transpose()).todok()
- logger.trace(f"Created sparse KNN dists and indices")
- densmap_kwds = {
- "lambda": dens_lambda,
- "frac": dens_frac,
- "var_shift": dens_var_shift,
- "n_neighbors": dists.shape[1],
- "knn_dists": dmat,
- }
- else:
- densmap_kwds = {}
-
- t, a, b = fit_transform(
- graph=graph.tocoo(),
- ini_embed=ini_embed,
- spread=spread,
- min_dist=min_dist,
- n_epochs=n_epochs,
- random_seed=random_seed,
- repulsion_strength=repulsion_strength,
- initial_alpha=initial_alpha,
- negative_sample_rate=negative_sample_rate,
- densmap_kwds=densmap_kwds,
- parallel=parallel,
- nthreads=nthreads,
- verbose=verbose,
- )
-
- if integrated_graph is not None:
- from_assay = integrated_graph
- for i in range(umap_dims):
- self.cells.insert(
- self._col_renamer(from_assay, cell_key, f"{label}{i + 1}"),
- t[:, i],
- key=cell_key,
- overwrite=True,
- )
- return None
-
- def run_leiden_clustering(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- resolution: float = 1.0,
- integrated_graph: Optional[str] = None,
- symmetric_graph: bool = False,
- graph_upper_only: bool = False,
- label: str = "leiden_cluster",
- random_seed: int = 4444,
- ) -> None:
- """Executes Leiden graph clustering algorithm on the cell-neighbourhood
- graph and saves cluster identities in the cell metadata column.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
- used feature for the given assay will be used.
- resolution: Resolution parameter for `RBConfigurationVertexPartition` configuration
- integrated_graph:
- symmetric_graph: This parameter is forwarded to `load_graph` and is same as there. (Default value: True)
- graph_upper_only: This parameter is forwarded to `load_graph` and is same as there. (Default value: True)
- label: base label for cluster identity in the cell metadata column (Default value: 'leiden_cluster')
- random_seed: (Default value: 4444)
-
- Returns:
- """
- try:
- # noinspection PyPackageRequirements
- import leidenalg
- except ImportError:
- raise ImportError(
- "ERROR: 'leidenalg' package is not installed. Please find the installation instructions "
- "here: https://github.com/vtraag/leidenalg#installation. Also, consider running Paris "
- "instead of Leiden clustering using `run_clustering` method"
- )
- # noinspection PyPackageRequirements
- import igraph # python-igraph
-
- from_assay, cell_key, feat_key = self._get_latest_keys(
- from_assay, cell_key, feat_key
- )
- graph_loc = None
- if integrated_graph is not None:
- graph_loc = f"{self._integratedGraphsLoc}/{integrated_graph}"
- if graph_loc not in self.zw:
- raise KeyError(
- f"ERROR: An integrated graph with label: {integrated_graph} does not exist"
- )
- graph = self.load_graph(
- from_assay=from_assay,
- cell_key=cell_key,
- feat_key=feat_key,
- symmetric=symmetric_graph,
- upper_only=graph_upper_only,
- graph_loc=graph_loc,
- )
- sources, targets = graph.nonzero()
- g = igraph.Graph()
- g.add_vertices(graph.shape[0])
- g.add_edges(list(zip(sources, targets)))
- g.es["weight"] = graph[sources, targets].A1
- part = leidenalg.find_partition(
- g,
- leidenalg.RBConfigurationVertexPartition,
- resolution_parameter=resolution,
- seed=random_seed,
- )
-
- if integrated_graph is not None:
- from_assay = integrated_graph
- self.cells.insert(
- self._col_renamer(from_assay, cell_key, label),
- np.array(part.membership) + 1,
- fill_value=-1,
- key=cell_key,
- overwrite=True,
- )
- return None
-
- def run_clustering(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- n_clusters: Optional[int] = None,
- integrated_graph: Optional[str] = None,
- symmetric_graph: bool = False,
- graph_upper_only: bool = False,
- balanced_cut: bool = False,
- max_size: Optional[int] = None,
- min_size: Optional[int] = None,
- max_distance_fc: float = 2,
- force_recalc: bool = False,
- label: str = "cluster",
- ) -> None:
- """Executes Paris clustering algorithm
- (https://arxiv.org/pdf/1806.01664.pdf) on the cell-neighbourhood graph.
- The algorithm captures the multiscale structure of the graph in to an
- ordinary dendrogram structure. The distances in the dendrogram are
- based on probability of sampling node (aka cell) pairs. These methods
- create this dendrogram if it doesn't already exist for the graph and
- induces either a straight cut or balanced cut to obtain clusters of
- cells.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
- used feature for the given assay will be used.
- n_clusters: Number of desired clusters (required if balanced_cut is False)
- integrated_graph:
- symmetric_graph: This parameter is forwarded to `load_graph` and is same as there. (Default value: True)
- graph_upper_only: This parameter is forwarded to `load_graph` and is same as there. (Default value: True)
- balanced_cut: If True, then uses the balanced cut algorithm as implemented in ``BalancedCut`` to obtain
- clusters (Default value: False)
- max_size: Same as `max_size` in ``BalancedCut``. The limit for a maximum number of cells in a cluster.
- This parameter value is required if `balanced_cut` is True.
- min_size: Same as `min_size` in ``BalancedCut``. The limit for a minimum number of cells in a cluster.
- This parameter value is required if `balanced_cut` is True.
- max_distance_fc: Same as `max_distance_fc` in ``BalancedCut``. The threshold of ratio of distance between
- two clusters beyond which they will not be merged. (Default value: 2)
- force_recalc: Forces recalculation of dendrogram even if one already exists for the graph
- label: Base label for cluster identity in the cell metadata column (Default value: 'cluster')
-
- Returns:
- None
- """
- import sknetwork as skn
-
- if balanced_cut is False:
- if n_clusters is None:
- raise ValueError(
- "ERROR: Please provide a value for n_clusters parameter. We are working on making "
- "this parameter free"
- )
- else:
- if n_clusters is not None:
- logger.info(
- "Using balanced cut method for cutting dendrogram. `n_clusters` will be ignored."
- )
- if max_size is None or min_size is None:
- raise ValueError(
- "ERROR: Please provide value for max_size and min_size"
- )
-
- from_assay, cell_key, feat_key = self._get_latest_keys(
- from_assay, cell_key, feat_key
- )
-
- graph_loc = self._get_latest_graph_loc(from_assay, cell_key, feat_key)
- if integrated_graph is not None:
- graph_loc = f"{self._integratedGraphsLoc}/{integrated_graph}"
- if graph_loc not in self.zw:
- raise KeyError(
- f"ERROR: An integrated graph with label: {integrated_graph} does not exist"
- )
-
- dendrogram_loc = f"{graph_loc}/dendrogram"
- # tuple are changed to list when saved as zarr attrs
- if dendrogram_loc in self.zw and force_recalc is False:
- dendrogram = self.zw[dendrogram_loc][:]
- logger.info("Using existing dendrogram")
- else:
- paris = skn.hierarchy.Paris(reorder=False)
- graph = self.load_graph(
- from_assay=from_assay,
- cell_key=cell_key,
- feat_key=feat_key,
- symmetric=symmetric_graph,
- upper_only=graph_upper_only,
- graph_loc=graph_loc,
- )
- dendrogram = paris.fit_transform(graph)
- dendrogram[dendrogram == np.inf] = 0
- g = create_zarr_dataset(
- self.zw[graph_loc],
- dendrogram_loc.rsplit("/", 1)[1],
- (5000,),
- "f8",
- (graph.shape[0] - 1, 4),
- )
- g[:] = dendrogram
- self.zw[graph_loc].attrs["latest_dendrogram"] = dendrogram_loc
-
- if balanced_cut:
- from ..dendrogram import BalancedCut
-
- labels = BalancedCut(
- dendrogram, max_size, min_size, max_distance_fc
- ).get_clusters()
- logger.info(f"{len(set(labels))} clusters found")
- else:
- labels = skn.hierarchy.cut_straight(dendrogram, n_clusters=n_clusters) + 1
-
- if integrated_graph is not None:
- from_assay = integrated_graph
- self.cells.insert(
- self._col_renamer(from_assay, cell_key, label),
- labels,
- fill_value=-1,
- key=cell_key,
- overwrite=True,
- )
-
- def run_topacedo_sampler(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- cluster_key: Optional[str] = None,
- use_k: Optional[int] = None,
- density_depth: int = 2,
- density_bandwidth: float = 5.0,
- max_sampling_rate: float = 0.05,
- min_sampling_rate: float = 0.01,
- min_cells_per_group: int = 3,
- snn_bandwidth: float = 5.0,
- seed_reward: float = 3.0,
- non_seed_reward: float = 0,
- edge_cost_multiplier: float = 1.0,
- edge_cost_bandwidth: float = 10.0,
- save_sampling_key: str = "sketched",
- save_density_key: str = "cell_density",
- save_mean_snn_key: str = "snn_value",
- save_seeds_key: str = "sketch_seeds",
- rand_state: int = 4466,
- return_edges: bool = False,
- ) -> Union[None, List]:
- """Perform sub-sampling (aka sketching) of cells using TopACeDo
- algorithm. Sub-sampling required that cells are partitioned in cluster
- already. Since, sub-sampling is dependent on cluster information,
- having, large number of homogeneous and even sized cluster improves
- sub-sampling results.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
- used feature for the given assay will be used.
- cluster_key: Name of the column in cell metadata table where cluster information is stored.
- use_k: Number of top k-nearest neighbours to retain in the graph over which downsampling is performed.
- BY default all neighbours are used. (Default value: None)
- density_depth: Same as 'search_depth' parameter in `calc_neighbourhood_density`. (Default value: 2)
- density_bandwidth: This value is used to scale the penalty affected by neighbourhood density. Higher values
- will lead to a larger penalty. (Default value: 5.0)
- max_sampling_rate: Maximum fraction of cells to sample from each group. The effective sampling rate is lower
- than this value depending on the neighbourhood degree and SNN density of cells.
- Should be greater than 0 and less than 1. (Default value: 0.1)
- min_sampling_rate: Minimum sampling rate. Effective sampling rate is not allowed to be lower than this
- value. Should be greater than 0 and less than 1. (Default value: 0.01)
- min_cells_per_group: Minimum number of cells to sample from each group. (Default value: 3)
- snn_bandwidth: Bandwidth for the shared nearest neighbour award. Clusters with higher mean SNN values get
- lower sampling penalty. This value, is raised to mean SNN value of the cluster to obtain
- sampling reward of the cluster. (Default value: 5.0)
- seed_reward: Reward/prize value for seed nodes. (Default value: 3.0)
- non_seed_reward: Reward/prize for non-seed nodes. (Default value: 0.1)
- edge_cost_multiplier: This value is multiplier to each edge's cost. Higher values will make graph traversal
- costly and might lead to removal of poorly connected nodes (Default value: 1.0)
- edge_cost_bandwidth: This value is raised to edge cost to get an adjusted edge cost (Default value: 1.0)
- save_sampling_key: base label for marking the cells that were sampled into a cell metadata column
- (Default value: 'sketched')
- save_density_key: base label for saving the cell neighbourhood densities into a cell metadata column
- (Default value: 'cell_density')
- save_mean_snn_key: base label for saving the SNN value for each cell (identified by topacedo sampler) into
- a cell metadata column (Default value: 'snn_value')
- save_seeds_key: base label for saving the seed cells (identified by topacedo sampler) into a cell
- metadata column (Default value: 'sketch_seeds')
- rand_state: A random values to set seed while sampling cells from a cluster randomly. (Default value: 4466)
- return_edges: If True, then steiner nodes and edges are returned. (Default value: False)
-
- Returns:
- """
-
- try:
- from topacedo import TopacedoSampler
- except ImportError:
- logger.error("Could not find topacedo package")
- return None
-
- from_assay, cell_key, feat_key = self._get_latest_keys(
- from_assay, cell_key, feat_key
- )
- if cluster_key is None:
- raise ValueError("ERROR: Please provide a value for cluster key")
- clusters = pd.Series(self.cells.fetch(cluster_key, key=cell_key))
- graph = self.load_graph(
- from_assay=from_assay,
- cell_key=cell_key,
- feat_key=feat_key,
- symmetric=False,
- upper_only=False,
- use_k=use_k,
- )
- graph_loc = self._get_latest_graph_loc(from_assay, cell_key, feat_key)
- try:
- dendrogram = self.zw[f"{graph_loc}/dendrogram"][:]
- except KeyError:
- raise KeyError(
- "ERROR: Couldn't find the dendrogram for clustering. Please note that "
- "TopACeDo requires a dendrogram from Paris clustering."
- )
-
- if len(clusters) != graph.shape[0]:
- raise ValueError(
- f"ERROR: cluster information exists for {len(clusters)} cells while graph has "
- f"{graph.shape[0]} cells."
- )
- sampler = TopacedoSampler(
- graph,
- clusters.values,
- dendrogram,
- density_depth,
- density_bandwidth,
- max_sampling_rate,
- min_sampling_rate,
- min_cells_per_group,
- snn_bandwidth,
- seed_reward,
- non_seed_reward,
- edge_cost_multiplier,
- edge_cost_bandwidth,
- rand_state,
- )
- nodes, edges = sampler.run()
- a = np.zeros(self.cells.fetch_all(cell_key).sum()).astype(bool)
- a[nodes] = True
- key = self._col_renamer(from_assay, cell_key, save_sampling_key)
- self.cells.insert(key, a, fill_value=False, key=cell_key, overwrite=True)
- logger.debug(f"Sketched cells saved under column '{key}'")
-
- key = self._col_renamer(from_assay, cell_key, save_density_key)
- self.cells.insert(key, sampler.densities, key=cell_key, overwrite=True)
- logger.debug(f"Cell neighbourhood densities saved under column: '{key}'")
-
- key = self._col_renamer(from_assay, cell_key, save_mean_snn_key)
- self.cells.insert(key, sampler.meanSnn, key=cell_key, overwrite=True)
- logger.debug(f"Mean SNN values saved under column: '{key}'")
-
- a = np.zeros(self.cells.fetch_all(cell_key).sum()).astype(bool)
- a[sampler.seeds] = True
- key = self._col_renamer(from_assay, cell_key, save_seeds_key)
- self.cells.insert(key, a, fill_value=False, key=cell_key, overwrite=True)
- logger.debug(f"Seed cells saved under column: '{key}'")
-
- if return_edges:
- return edges
-
- def get_imputed(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- feature_name: Optional[str] = None,
- feat_key: Optional[str] = None,
- t: int = 2,
- cache_operator: bool = True,
- ) -> np.ndarray:
- """
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- feature_name: Name of the feature to be imputed
- feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
- used feature for the given assay will be used.
- t: Same as the t parameter in MAGIC. Higher values lead to larger diffusion of values. Too large values
- can slow down the algorithm and cause over-smoothening. (Default value: 2)
- cache_operator: Whether to keep the diffusion operator in memory after the method returns. Can be useful
- to set to True if many features are to imputed in a batch but can lead to increased memory
- usage. (Default value: True)
-
- Returns:
- An array of imputed values for the given feature
-
- """
-
- def calc_diff_operator(g: csr_matrix, to_power: int) -> coo_matrix:
- d = np.ravel(g.sum(axis=1))
- d[d != 0] = 1 / d[d != 0]
- n = g.shape[0]
- d = csr_matrix((d, (range(n), range(n))), shape=[n, n])
- return d.dot(g).__pow__(to_power).tocoo()
-
- from_assay, cell_key, feat_key = self._get_latest_keys(
- from_assay, cell_key, feat_key
- )
- if feature_name is None:
- raise ValueError(
- "ERROR: Please provide name for the feature to be imputed. It can, for example, "
- "be a gene name."
- )
- data = self.get_cell_vals(
- from_assay=from_assay, cell_key=cell_key, k=feature_name
- )
-
- graph_loc = self._get_latest_graph_loc(from_assay, cell_key, feat_key)
- magic_loc = f"{graph_loc}/magic_{t}"
- if magic_loc in self.zw:
- logger.info("Using existing MAGIC diffusion operator")
- if self._cachedMagicOperatorLoc == magic_loc:
- diff_op = self._cachedMagicOperator
- else:
- n_cells, _ = self._get_graph_ncells_k(graph_loc)
- store = self.zw[magic_loc]
- diff_op = coo_matrix(
- (store["data"][:], (store["row"][:], store["col"][:])),
- shape=(n_cells, n_cells),
- )
- if cache_operator:
- self._cachedMagicOperator = diff_op
- self._cachedMagicOperatorLoc = magic_loc
- else:
- self._cachedMagicOperator = None
- self._cachedMagicOperatorLoc = None
- else:
- graph = self.load_graph(
- from_assay=from_assay,
- cell_key=cell_key,
- feat_key=feat_key,
- symmetric=True,
- upper_only=False,
- )
- diff_op = calc_diff_operator(graph, t)
- shape = diff_op.data.shape
- store = self.zw.create_group(magic_loc, overwrite=True)
- for i, j in zip(["row", "col", "data"], ["uint32", "uint32", "float32"]):
- zg = create_zarr_dataset(store, i, (1000000,), j, shape)
- zg[:] = diff_op.__getattribute__(i)
- self.zw[graph_loc].attrs["latest_magic"] = magic_loc
- if cache_operator:
- self._cachedMagicOperator = diff_op
- self._cachedMagicOperatorLoc = magic_loc
- else:
- self._cachedMagicOperator = None
- self._cachedMagicOperatorLoc = None
- return diff_op.dot(data)
-
- def run_pseudotime_scoring(
- self,
- from_assay: Optional[str] = None,
- cell_key: Optional[str] = None,
- subset_cell_key: Optional[str] = None,
- feat_key: Optional[str] = None,
- n_singular_vals: int = 30,
- source_sink_key: Optional[str] = None,
- sources: Optional[List] = None,
- sinks: Optional[List] = None,
- ss_vec: Optional[np.ndarray] = None,
- min_max_norm_ptime: bool = True,
- random_seed: int = 4444,
- label: str = "pseudotime",
- ) -> None:
- """
- Calculate differentiation potential of cells. This function is a reimplementation of population balance
- analysis (PBA) approach published in Weinreb et al. 2017, PNAS. This function computes the random walk
- normalized Laplacian matrix of the reference graph, L_rw = I-A/D and then calculates a Moore-Penrose
- pseudoinverse of L_rw.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- subset_cell_key: Cell key for the subset of cells for which pseudotime scoring is to be performed.
- feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
- used feature for the given assay will be used.
- n_singular_vals: Number of the smallest singular values to save.
- source_sink_key: Name of a column from cell attributes table that shall be used for fetching source and
- sink groups. Usually this will a column containing cell cluster/group identities.
- sources: A list of group/clusters ids from `source_sink_key` column to be treated as sources. Sources are
- usually progenitor/precursor or other actively dividing cell states.
- sinks: A list of group/clusters ids from `source_sink_key` column to be treated as sinks. Sinks are usually
- more differentiated (or terminally differentiated) cell states.
- ss_vec: A vector that contains source sink values for each cell. If not provided then, this vector is
- internally computed using the `sources` and `sinks` parameter. This vector should add up to 0 and
- should have negative values for source cells and positive values for sink cells.
- min_max_norm_ptime: Whether to perform min-max normalization on the final pseudotime values so that values
- are in 0 to 1 range. (Default: True)
- random_seed: A random seed for svds (Default: 4444)
- label: label: Base label for pseudotime in the cell metadata column (Default value: 'pseudotime')
-
- Returns:
-
- """
-
- from scipy.sparse.linalg import svds
-
- def inverse_degree(g):
- d = np.ravel(g.sum(axis=1))
- n = g.shape[0]
- d[d != 0] = 1 / d[d != 0]
- return csr_matrix((d, (range(n), range(n))), shape=[n, n])
-
- def laplacian(g, inv_deg):
- n = g.shape[0]
- identity = csr_matrix((np.ones(n), (range(n), range(n))), shape=[n, n])
- return identity - g.dot(inv_deg)
-
- def make_source_sink_vector(c, source, sink):
- ss = list(source) + list(sink)
-
- r = np.zeros(c.shape[0])
- r[c.isin(sink)] = 1
- r[c.isin(source)] = -1
-
- n = c.isin(ss).sum()
- v = (0 - r.sum()) / (r.shape[0] - n)
- r[~c.isin(ss)] = v
- return r
-
- def pseudo_inverse(lap, k, rseed, r):
- random_state = np.random.RandomState(rseed)
- # noinspection PyArgumentList
- v0 = random_state.rand(lap.shape[0])
- # TODO: add thread management here
- logger.info(
- "Calculating SVD of graph laplacian. This might take a while...",
- )
- u, s, vt = svds(lap, k=k, which="SM", v0=v0)
- # Because the order of singular values is not guaranteed
- idx = np.argsort(s)
- # Extracting the second smallest values
- s = s[idx][1:].T
- s = 1 / s
- u = u[:, idx][:, 1:]
- vt = vt[idx, :][1:, :].T
- # Computing matmul in an iterative way to save memory
- n = u.shape[0]
- # TODO: Use numba for this part
- ilap = np.zeros(n)
- for i in tqdmbar(range(n), desc="Calculating pseudotime"):
- ilap[i] = (vt * u[i, :] * s * r).sum()
- return ilap
-
- from_assay, cell_key, feat_key = self._get_latest_keys(
- from_assay, cell_key, feat_key
- )
- if subset_cell_key is None:
- subset_cell_key = cell_key
- cell_idx = self.cells.fetch(subset_cell_key, key=cell_key)
- else:
- cell_idx = self.cells.fetch(subset_cell_key, key=cell_key)
- if cell_idx.sum() != self.cells.fetch_all(subset_cell_key).sum():
- raise ValueError("subset_cell_key is not a complete subset of cell_key")
-
- graph = self.load_graph(
- from_assay=from_assay,
- cell_key=cell_key,
- feat_key=feat_key,
- symmetric=True,
- upper_only=False,
- )
- if cell_idx.shape[0] != cell_idx.sum():
- graph = graph[cell_idx][:, cell_idx]
-
- if source_sink_key is None:
- if sources is not None or sinks is not None:
- logger.warning(
- "Provide `sources` and `sinks` will not be used because `source_sink_key` has not been "
- "provided"
- )
- if ss_vec is None:
- if source_sink_key is None:
- logger.warning(
- "No source/sink info or custom source sink vector provided. The results might not be "
- "reflect true pseudotime."
- )
- ss_vec = np.ones(graph.shape[0])
- else:
- clusts = pd.Series(
- self.cells.fetch(source_sink_key, key=cell_key)[cell_idx]
- )
- if sources is None:
- sources = []
- else:
- if isinstance(sources, list) is False:
- raise ValueError(
- "ERROR: Parameter `sources` should be of 'list' type"
- )
- if sinks is None:
- sinks = []
- else:
- if isinstance(sinks, list) is False:
- raise ValueError(
- "ERROR: Parameter `sinks` should be of 'list' type"
- )
- ss_vec = make_source_sink_vector(clusts, sources, sinks)
- else:
- if source_sink_key is not None:
- logger.warning(
- "Sources/sinks from `source_sink_key` will not be because custom vector `ss_vec` is "
- "provided"
- )
- ss_vec = np.array(ss_vec)
- if ss_vec.shape[0] != graph.shape[0]:
- raise ValueError(
- f"ERROR: Size mismatch between `ss_vec` ({ss_vec.shape[0]}) and "
- f"graph ({graph.shape[0]:})"
- )
- if ss_vec.sum() > 1e-10:
- raise ValueError(
- f"ERROR: The sum of all the values in `ss_vec` should be zero. Here we test if the sum is less"
- f" 1e-10"
- )
-
- ss_vec = ss_vec.reshape(-1, 1)
-
- ptime = pseudo_inverse(
- laplacian(graph, inverse_degree(graph)),
- n_singular_vals,
- random_seed,
- ss_vec,
- )
- if min_max_norm_ptime:
- # noinspection PyArgumentList
- ptime = ptime - ptime.min()
- ptime = ptime / ptime.max()
-
- self.cells.insert(
- self._col_renamer(from_assay, subset_cell_key, label),
- ptime,
- key=subset_cell_key,
- overwrite=True,
- )
- return None
-
- def integrate_assays(
- self,
- assays: List[str],
- label: str,
- method: str = "snn",
- chunk_size: int = 10000,
- ) -> None:
- """Merges KNN graphs of two or more assays from within the same
- DataStore. The input KNN graphs should have been constructed on the
- same set of cells and should each have been constructed with equal
- number of neighbours (parameter: k) The merged KNN graph has the same
- size and shape as the input graphs.
-
- Args:
- assays: Name of the input assays. The latest constructed graph from each assay is used.
- label: Label for integrated graph
- method: Choose a method for modality integration. Available options: 'snn': Shared nearest neighbour
- approach and 'wnn': Weighted nearest neighbor approach based on Hao et. alm Cell 2022.
- chunk_size: number of cells to be loaded at a time while reading and writing the graph
-
- Returns: None
- """
- from ..knn_utils import merge_graphs, wnn_integration
-
- def load_pca_knn(assay_name):
- g = self.load_graph(
- from_assay=assay_name,
- cell_key=None,
- feat_key=None,
- symmetric=False,
- upper_only=False,
- ).tocsr()
- ao = self.make_graph(
- from_assay=assay_name,
- feat_key=self._get_latest_feat_key(assay_name),
- return_ann_object=True,
- update_keys=False,
- )
- if ao.loadings is None:
- logger.warning(
- f"No dimension reduction was user for {assay_name} data. "
- f"Memory consumption will be high."
- )
- return g, ao.data.compute()
- else:
- return g, ao.data.dot(ao.loadings).compute()
-
- if method == "snn":
- merged_graph = []
- for assay in assays:
- if assay not in self.assay_names:
- raise ValueError(f"ERROR: Assay {assay} was not found.")
- merged_graph.append(
- self.load_graph(
- from_assay=assay,
- cell_key=None,
- feat_key=None,
- symmetric=False,
- upper_only=False,
- ).tocsr()
- )
- merged_graph = merge_graphs(merged_graph)
- elif method == "wnn":
- if len(assays) != 2:
- raise ValueError(
- "WNN integration in Scarf can currently be performed using only two assays"
- )
- g1, ld1 = load_pca_knn(assays[0])
- g2, ld2 = load_pca_knn(assays[1])
- merged_graph = wnn_integration(
- assays[0], g1, ld1, assays[1], g2, ld2, self.nthreads
- )
- else:
- raise ValueError(
- f"Method {method} not supported, choose one of these: 'snn', 'wnn'"
- )
-
- n_cells = merged_graph.shape[0]
- n_neighbors = int(merged_graph.size / n_cells)
-
- ig_loc = self._integratedGraphsLoc
- if ig_loc not in self.zw:
- self.zw.create_group(ig_loc)
- if label in self.zw[ig_loc]:
- del self.zw[f"{ig_loc}/{label}"]
- store = self.zw.create_group(f"{ig_loc}/{label}")
- store.attrs["n_cells"] = n_cells
- store.attrs["n_neighbors"] = n_neighbors
-
- zge = create_zarr_dataset(
- store,
- f"edges",
- (chunk_size,),
- (np.uint32, np.uint32),
- (n_cells * n_neighbors, 2),
- )
- zgw = create_zarr_dataset(
- store, f"weights", (chunk_size,), np.float64, (n_cells * n_neighbors)
- )
-
- zge[:, 0] = merged_graph.row
- zge[:, 1] = merged_graph.col
- zgw[:] = merged_graph.data
+ self._annStreamPaths: WeakKeyDictionary[AnnStream, str] = WeakKeyDictionary()
diff --git a/scarf/datastore/mapping_datastore.py b/scarf/datastore/mapping_datastore.py
index 5aef6d42..b7c4c299 100644
--- a/scarf/datastore/mapping_datastore.py
+++ b/scarf/datastore/mapping_datastore.py
@@ -1,877 +1,9 @@
-import os
-from typing import Generator, Tuple, List, Dict, Union, Callable, Optional
-
-import numpy as np
-import pandas as pd
-from dask import array as daskarr
-from loguru import logger
-from scipy.sparse import csr_matrix
-
+from ._operations.mapping import _MappingOperationsMixin
from .graph_datastore import GraphDataStore
-from ..assay import Assay, RNAassay
-from ..utils import (
- show_dask_progress,
- clean_array,
- tqdmbar,
- controlled_compute,
- system_call,
-)
-from ..writers import create_zarr_dataset
-class MappingDatastore(GraphDataStore):
+class MappingDatastore(_MappingOperationsMixin, GraphDataStore):
"""This class extends GraphDataStore by providing methods for mapping/
projection of cells from one DataStore onto another. It also contains the methods
required for label transfer, mapping score generation and co-embedding.
"""
-
- def __init__(self, **kwargs):
- super().__init__(**kwargs)
-
- def run_mapping(
- self,
- target_assay: Assay,
- target_name: str,
- target_feat_key: str,
- target_cell_key: str = "I",
- from_assay: Optional[str] = None,
- cell_key: str = "I",
- feat_key: Optional[str] = None,
- save_k: int = 3,
- batch_size: int = 1000,
- ref_mu: bool = True,
- ref_sigma: bool = True,
- run_coral: bool = False,
- exclude_missing: bool = False,
- filter_null: bool = False,
- feat_scaling: bool = True,
- ann_index_fetcher: Optional[Callable] = None,
- ann_index_saver: Optional[Callable] = None,
- ) -> None:
- """Projects cells from external assays into the cell-neighbourhood
- graph using existing PCA loadings and ANN index. For each external cell
- (target) nearest neighbours are identified and save within the Zarr
- hierarchy group `projections`.
-
- Args:
- target_assay: Assay object of the target dataset.
- target_name: Name of target data. This used to keep track of projections in the Zarr hierarchy
- target_feat_key: This will be used to name wherein the normalized target data will be saved in its own
- zarr hierarchy.
- target_cell_key: Cell key for the target data. (Default value: 'I')
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
- used feature for the given assay will be used.
- save_k: Number of the nearest neighbours to identify for each target cell (Default value: 3)
- batch_size: Number of cells that will be projected as a batch. This used to decide the chunk size when
- normalized data for the target cells is saved to disk.
- ref_mu: If True (default), Then mean values of features as in the reference are used,
- otherwise mean is calculated using target cells. Turning this to False is not recommended.
- ref_sigma: If True (default), Then standard deviation values of features as present in the reference are
- used, otherwise std. dev. is calculated using target cells. Turning this to False is not
- recommended.
- run_coral: If True then CORAL feature rescaling algorithm is used to correct for domain shift in target
- cells. Read more about CORAL algorithm in function ``coral``. This algorithm creates a m by m
- matrix where m is the number of features being used for mapping; so it is not advised to use this
- in a case where a large number of features are being used (>10k for example).
- (Default value: False)
- exclude_missing: If set to True then only those features that are present in both reference and
- target are used. If not all reference features from `feat_key` are present in target data
- then a new graph will be created for reference and mapping will be done onto that graph.
- (Default value: False)
- filter_null: If True then those features that have a total sum of 0 in the target cells are removed.
- This has an affect only when `exclude_missing` is True. (Default value: False)
- feat_scaling: If False then features from target cells are not scaled. This is automatically set to False
- if `run_coral` is True (Default value: True). Setting this to False is not recommended.
- ann_index_fetcher:
- ann_index_saver:
-
- Returns:
- None
- """
- from ..mapping_utils import align_features, coral
-
- from_assay, cell_key, feat_key = self._get_latest_keys(
- from_assay, cell_key, feat_key
- )
- source_assay = self._get_assay(from_assay)
-
- if type(target_assay) != type(source_assay):
- raise TypeError(
- f"ERROR: Source assay ({type(source_assay)}) and target assay "
- f"({type(target_assay)}) are of different types. "
- f"Mapping can only be performed between same assay types"
- )
- if type(target_assay) == RNAassay:
- if target_assay.sf != source_assay.sf:
- logger.info(
- f"Resetting target assay's size factor from {target_assay.sf} to {source_assay.sf}"
- )
- target_assay.sf = source_assay.sf
-
- if target_feat_key == feat_key:
- raise ValueError(
- f"ERROR: `target_feat_key` cannot be sample as `feat_key`: {feat_key}"
- )
-
- feat_idx = align_features(
- source_assay,
- target_assay,
- cell_key,
- feat_key,
- target_feat_key,
- target_cell_key,
- filter_null,
- exclude_missing,
- self.nthreads,
- )
- logger.debug(f"{len(feat_idx)} features being used for mapping")
- if np.all(
- source_assay.feats.active_index(cell_key + "__" + feat_key) == feat_idx
- ):
- ann_feat_key = feat_key
- else:
- ann_feat_key = f"{feat_key}_common_{target_name}"
- a = np.zeros(source_assay.feats.N).astype(bool)
- a[feat_idx] = True
- source_assay.feats.insert(
- cell_key + "__" + ann_feat_key, a, fill_value=False, overwrite=True
- )
- if run_coral:
- feat_scaling = False
- ann_obj = self.make_graph(
- from_assay=from_assay,
- cell_key=cell_key,
- feat_key=ann_feat_key,
- return_ann_object=True,
- update_keys=False,
- feat_scaling=feat_scaling,
- ann_index_fetcher=ann_index_fetcher,
- ann_index_saver=ann_index_saver,
- )
- if save_k > ann_obj.k:
- logger.warning(f"`save_k` was decreased to {ann_obj.k}")
- save_k = ann_obj.k
- target_data = daskarr.from_zarr(
- target_assay.z[f"normed__{target_cell_key}__{target_feat_key}/data"],
- inline_array=True,
- )
- if run_coral is True:
- # Reversing coral here to correct target data
- coral(
- target_data,
- ann_obj.data,
- target_assay,
- target_feat_key,
- target_cell_key,
- self.nthreads,
- )
- target_data = daskarr.from_zarr(
- target_assay.z[
- f"normed__{target_cell_key}__{target_feat_key}/data_coral"
- ],
- inline_array=True,
- )
- if ann_obj.method == "pca" and run_coral is False:
- if ref_mu is False:
- mu = show_dask_progress(
- target_data.mean(axis=0),
- "Calculating mean of target norm. data",
- self.nthreads,
- )
- ann_obj.mu = clean_array(mu)
- if ref_sigma is False:
- sigma = show_dask_progress(
- target_data.std(axis=0),
- "Calculating std. dev. of target norm. data",
- self.nthreads,
- )
- ann_obj.sigma = clean_array(sigma, 1)
- if "projections" not in source_assay.z:
- source_assay.z.create_group("projections")
- store = source_assay.z["projections"].create_group(target_name, overwrite=True)
- nc, nk = target_assay.cells.fetch_all("I").sum(), save_k
- zi = create_zarr_dataset(store, "indices", (batch_size,), "u8", (nc, nk))
- zd = create_zarr_dataset(store, "distances", (batch_size,), "f8", (nc, nk))
- entry_start = 0
- for i in tqdmbar(
- target_data.blocks,
- desc=f"Mapping cells from {target_name}",
- total=target_data.numblocks[0],
- ):
- a: np.ndarray = controlled_compute(i, self.nthreads)
- ki, kd = ann_obj.transform_ann(ann_obj.reducer(a), k=save_k)
- entry_end = entry_start + len(ki)
- zi[entry_start:entry_end, :] = ki
- zd[entry_start:entry_end, :] = kd
- entry_start = entry_end
- return None
-
- def get_mapping_score(
- self,
- target_name: str,
- target_groups: Optional[np.ndarray] = None,
- from_assay: Optional[str] = None,
- cell_key: str = "I",
- log_transform: bool = True,
- multiplier: float = 1000,
- weighted: bool = True,
- fixed_weight: float = 0.1,
- ) -> Generator[Tuple[str, np.ndarray], None, None]:
- """Yields the mapping scores that were a result of a mapping.
-
- Mapping scores are an indication of degree of similarity of reference cells in the graph to the target cells.
- The more often a reference cell is found in the nearest neighbour list of the target cells, the higher
- the mapping score will be for that cell.
-
- Args:
- target_name: Name of target data. This used to keep track of projections in the Zarr hierarchy
- target_groups: Group/cluster identity of target cells. This will then be used to calculate mapping score
- for each group separately.
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- log_transform: If True (default) then the mapping scores will be log transformed
- multiplier: A scaling factor for mapping scores. All scores al multiplied this value. This mostly intended
- for visualization of mapping scores (Default: 1000)
- weighted: Use distance weights when calculating mapping scores (default: True). If False then the actual
- distances between the reference and target cells are ignored.
- fixed_weight: Used when `weighted` is False. This is the value that is added to mapping score of each
- reference cell for every projected target cell. Can be any value >0.
-
- Yields:
- A tuple of group name and mapping score of reference cells for that target group.
- """
- if from_assay is None:
- from_assay = self._defaultAssay
- store_loc = f"{from_assay}/projections/{target_name}"
- if store_loc not in self.zw:
- raise KeyError(
- f"ERROR: Projections have not been computed for {target_name} in th latest graph. Please"
- f" run `run_mapping` or update latest_graph by running `make_graph` with desired parameters"
- )
- store = self.zw[store_loc]
-
- indices = store["indices"][:]
- dists = store["distances"][:]
- # TODO: add more robust options for distance calculation here
- dists = 1 / (np.log1p(dists) + 1)
- n_cells, n_k = indices.shape
-
- if target_groups is not None:
- if len(target_groups) != n_cells:
- raise ValueError(
- f"ERROR: Length of target_groups {len(target_groups)} not same as number of target "
- f"cells in the projection {n_cells}"
- )
- groups = pd.Series(target_groups)
- else:
- groups = pd.Series(np.zeros(n_cells))
-
- ref_n_cells = self.cells.fetch_all(cell_key).sum()
- for group in sorted(groups.unique()):
- coi = {x: None for x in groups[groups == group].index.values}
- ms = np.zeros(ref_n_cells)
- for n, i, j in zip(range(len(indices)), indices, dists):
- if n in coi:
- for x, y in zip(i, j):
- if weighted:
- ms[x] += y
- else:
- ms[x] += fixed_weight
- ms = multiplier * ms / (len(coi) * n_k)
- if log_transform:
- ms = np.log1p(ms)
- yield group, ms
-
- def get_target_classes(
- self,
- target_name: str,
- from_assay: Optional[str] = None,
- cell_key: str = "I",
- reference_class_group: Optional[str] = None,
- threshold_fraction: float = 0.5,
- target_subset: Optional[List[int]] = None,
- na_val: str = "NA",
- ) -> pd.Series:
- """Perform classification of target cells using a reference group.
-
- Args:
- target_name: Name of target data. This value should be the same as that used for `run_mapping` earlier.
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- reference_class_group: Group/cluster identity of the reference cells. These are the target labels for the
- classifier. The value here should be a column from cell metadata table. For
- example, to use default clustering identity one could use `RNA_cluster`
- threshold_fraction: The threshold for deciding if a cell belongs to a group or not.
- Constrained between 0 and 1. (Default value: 0.5)
- target_subset: Choose only a subset of target cells to be classified. The value should be a list of
- indices of the target cells. (Default: None)
- na_val: Value to be used if a cell is not classified to any of the `reference_class_group`.
- (Default value: 'NA')
-
- Returns: A pandas Series containing predicted class for each cell in the projected sample (`target_name`).
- """
- if from_assay is None:
- from_assay = self._defaultAssay
- store_loc = f"{from_assay}/projections/{target_name}"
- if store_loc not in self.zw:
- raise KeyError(
- f"ERROR: Projections have not been computed for {target_name} in th latest graph. Please"
- f" run `run_mapping` or update latest_graph by running `make_graph` with desired parameters"
- )
- if reference_class_group is None:
- raise ValueError(
- "ERROR: A value is required for the parameter `reference_class_group`. "
- "This can be any cell metadata column. Please choose the value that contains cluster or "
- "group information"
- )
- ref_groups = self.cells.fetch(reference_class_group, key=cell_key)
- if threshold_fraction < 0 or threshold_fraction > 1:
- raise ValueError(
- "ERROR: `threshold_fraction` should have a value between 0 and 1"
- )
- if target_subset is not None:
- if type(target_subset) != list:
- raise TypeError("ERROR: `target_subset` should be type")
- target_subset = {x: None for x in target_subset}
-
- store = self.zw[store_loc]
- indices = store["indices"][:]
- dists = store["distances"][:]
-
- preds = []
- weights = 1 - (dists / dists.max(axis=1).reshape(-1, 1))
- for n in range(indices.shape[0]):
- if target_subset is not None and n not in target_subset:
- continue
- wd = {}
- for i, j in zip(indices[n, :-1], weights[n, :-1]):
- k = ref_groups[i]
- if k not in wd:
- wd[k] = 0
- wd[k] += j
- temp = na_val
- s = weights[n, :-1].sum()
- for i, j in wd.items():
- if j / s > threshold_fraction:
- if temp == na_val:
- temp = i
- else:
- temp = na_val
- break
- preds.append(temp)
- return pd.Series(preds)
-
- def load_unified_graph(
- self,
- from_assay: str,
- cell_key: str,
- feat_key: str,
- target_names: List[str],
- use_k: int,
- target_weight: float,
- ) -> Tuple[List[int], csr_matrix]:
- """This is similar to ``load_graph`` but includes projected cells and
- their edges.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
- used feature for the given assay will be used.
- target_names: Name of target datasets to be included in the unified graph
- use_k: Number of nearest neighbour edges of each projected cell to be included. If this value is larger
- than `save_k` parameter while running mapping for the `target_name` target then `use_k` is reset to
- 'save_k'
- target_weight: A constant uniform weight to be ascribed to each target-reference edge.
-
- Returns:
- """
- # TODO: allow loading multiple targets
-
- if from_assay is None:
- from_assay = self._defaultAssay
- if feat_key is None:
- feat_key = self._get_latest_feat_key(from_assay)
- graph_loc = self._get_latest_graph_loc(from_assay, cell_key, feat_key)
- edges = self.zw[graph_loc].edges[:]
- weights = self.zw[graph_loc].weights[:]
- ref_n_cells = self.cells.fetch_all(cell_key).sum()
- store = self.zw[from_assay].projections
- pidx = np.vstack([store[x].indices[:, :use_k] for x in target_names])
- n_cells = [ref_n_cells] + [store[x].indices.shape[0] for x in target_names]
- ne = []
- nw = []
- for n, i in enumerate(pidx):
- for j in i:
- ne.append([ref_n_cells + n, j])
- # TODO: Better way to weigh the target edges
- nw.append(target_weight)
- me = np.vstack([edges, ne]).astype(int)
- mw = np.hstack([weights, nw])
- tot_cells = ref_n_cells + pidx.shape[0]
- graph = csr_matrix((mw, (me[:, 0], me[:, 1])), shape=(tot_cells, tot_cells))
- return n_cells, graph
-
- def _get_uni_ini_embed(
- self,
- from_assay: str,
- cell_key: str,
- feat_key: str,
- graph: csr_matrix,
- ini_embed_with: str,
- ref_n_cells: int,
- ) -> np.ndarray:
- if ini_embed_with == "kmeans":
- ini_embed = self._get_ini_embed(from_assay, cell_key, feat_key, 2)
- else:
- x = self.cells.fetch(f"{ini_embed_with}1", key=cell_key)
- y = self.cells.fetch(f"{ini_embed_with}2", key=cell_key)
- ini_embed = np.array([x, y]).T.astype(np.float32, order="C")
- targets_best_nn = np.array(np.argmax(graph, axis=1)).reshape(1, -1)[0][
- ref_n_cells:
- ]
- return np.vstack([ini_embed, ini_embed[targets_best_nn]])
-
- def _save_embedding(
- self,
- from_assay: str,
- cell_key: str,
- label: str,
- embedding: np.ndarray,
- n_cells: List[int],
- target_names: List[str],
- ) -> None:
- g = create_zarr_dataset(
- self.zw[from_assay].projections,
- label,
- (1000, 2),
- "float64",
- embedding.shape,
- )
- g[:] = embedding
- g.attrs["n_cells"] = [
- int(x) for x in n_cells
- ] # forcing int type here otherwise json raises TypeError
- g.attrs["target_names"] = target_names
- for i in range(2):
- self.cells.insert(
- self._col_renamer(from_assay, cell_key, f"{label}{i + 1}"),
- embedding[: n_cells[0], i],
- key=cell_key,
- overwrite=True,
- )
- return None
-
- def run_unified_umap(
- self,
- target_names: List[str],
- from_assay: Optional[str] = None,
- cell_key: str = "I",
- feat_key: Optional[str] = None,
- use_k: int = 3,
- target_weight: float = 0.1,
- spread: float = 2.0,
- min_dist: float = 1,
- n_epochs: int = 200,
- repulsion_strength: float = 1.0,
- initial_alpha: float = 1.0,
- negative_sample_rate: float = 5,
- random_seed: int = 4444,
- ini_embed_with: str = "kmeans",
- label: str = "unified_UMAP",
- parallel: bool = False,
- nthreads: Optional[int] = None,
- ) -> None:
- """Calculates the UMAP embedding for graph obtained using
- ``load_unified_graph``.
-
- The loaded graph is processed the same way as the graph as in ``run_umap``.
-
- Args:
- target_names: Names of target datasets to be included in the unified UMAP.
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
- used feature for the given assay will be used.
- use_k: Number of nearest neighbour edges of each projected cell to be included. If this value is larger
- than `save_k` parameter while running mapping for the `target_name` target then `use_k` is reset to
- 'save_k'
- target_weight: A constant uniform weight to be ascribed to each target-reference edge.
- spread: Same as spread in UMAP package. The effective scale of embedded points. In combination with
- ``min_dist`` this determines how clustered/clumped the embedded points are.
- min_dist: Same as min_dist in UMAP package. The effective minimum distance between embedded points.
- Smaller values will result in a more clustered/clumped embedding where nearby points on the
- manifold are drawn closer together, while larger values will result on a more even dispersal of
- points. The value should be set relative to the ``spread`` value, which determines the scale at
- which embedded points will be spread out. (Default value: 1)
- n_epochs: Same as n_epochs in UMAP package. The number of training epochs to be used in optimizing the
- low dimensional embedding. Larger values result in more accurate embeddings.
- (Default value: 200)
- repulsion_strength: Same as repulsion_strength in UMAP package. Weighting applied to negative samples in
- low dimensional embedding optimization. Values higher than one will result in greater
- weight being given to negative samples. (Default value: 1.0)
- initial_alpha: Same as learning_rate in UMAP package. The initial learning rate for the embedding
- optimization. (Default value: 1.0)
- negative_sample_rate: Same as negative_sample_rate in UMAP package. The number of negative samples to
- select per positive sample in the optimization process. Increasing this value will
- result in greater repulsive force being applied, greater optimization cost, but
- slightly more accuracy. (Default value: 5)
- random_seed: (Default value: 4444)
- ini_embed_with: either 'kmeans' or a column from cell metadata to be used as initial embedding coordinates
- label: base label for UMAP dimensions in the cell metadata column (Default value: 'UMAP')
- parallel: Whether to run UMAP in parallel mode. Setting value to True will use `nthreads` threads.
- The results are not reproducible in parallel mode. (Default value: False)
- nthreads: If parallel=True then this number of threads will be used to run UMAP. By default, the `nthreads`
- attribute of the class is used. (Default value: None)
-
- Returns:
- None
- """
- from ..umap import fit_transform
- from ..utils import get_log_level
-
- if from_assay is None:
- from_assay = self._defaultAssay
- if feat_key is None:
- feat_key = self._get_latest_feat_key(from_assay)
- n_cells, graph = self.load_unified_graph(
- from_assay=from_assay,
- cell_key=cell_key,
- feat_key=feat_key,
- target_names=target_names,
- use_k=use_k,
- target_weight=target_weight,
- )
- ini_embed = self._get_uni_ini_embed(
- from_assay, cell_key, feat_key, graph, ini_embed_with, n_cells[0]
- )
- if nthreads is None:
- nthreads = self.nthreads
- verbose = False
- if get_log_level() <= 20:
- verbose = True
- t, a, b = fit_transform(
- graph=graph.tocoo(),
- ini_embed=ini_embed,
- spread=spread,
- min_dist=min_dist,
- n_epochs=n_epochs,
- random_seed=random_seed,
- repulsion_strength=repulsion_strength,
- initial_alpha=initial_alpha,
- negative_sample_rate=negative_sample_rate,
- densmap_kwds={},
- parallel=parallel,
- nthreads=nthreads,
- verbose=verbose,
- )
- self._save_embedding(from_assay, cell_key, label, t, n_cells, target_names)
- return None
-
- def run_unified_tsne(
- self,
- target_names: List[str],
- from_assay: Optional[str] = None,
- cell_key: str = "I",
- feat_key: Optional[str] = None,
- use_k: int = 3,
- target_weight: float = 0.5,
- lambda_scale: float = 1.0,
- max_iter: int = 500,
- early_iter: int = 200,
- alpha: int = 10,
- box_h: float = 0.7,
- temp_file_loc: str = ".",
- verbose: bool = True,
- ini_embed_with: str = "kmeans",
- label: str = "unified_tSNE",
- ) -> None:
- """Calculates the tSNE embedding for graph obtained using
- ``load_unified_graph``. The loaded graph is processed the same way as
- the graph as in ``run_tsne``.
-
- Args:
- target_names: Names of target datasets to be included in the unified tSNE.
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- cell_key: Cell key. Should be same as the one that was used in the desired graph. (Default value: 'I')
- feat_key: Feature key. Should be same as the one that was used in the desired graph. By default, the latest
- used feature for the given assay will be used.
- use_k: Number of nearest neighbour edges of each projected cell to be included. If this value is larger
- than `save_k` parameter while running mapping for the `target_name` target then `use_k` is reset to
- 'save_k'.
- target_weight: A constant uniform weight to be ascribed to each target-reference edge.
- lambda_scale: λ rescaling parameter. (Default value: 1.0)
- max_iter: Maximum number of iterations. (Default value: 500)
- early_iter: Number of early exaggeration iterations. (Default value: 200)
- alpha: Early exaggeration multiplier. (Default value: 10)
- box_h: Grid side length (accuracy control). Lower values might drastically slow down
- the algorithm (Default value: 0.7)
- temp_file_loc: Location of temporary file. By default, these files will be created in the current working
- directory. These files are deleted before the method returns.
- verbose: If True (default) then the full log from SGtSNEpi algorithm is shown.
- ini_embed_with: Initial embedding coordinates for the cells in cell_key. Should have the same number of
- columns as tsne_dims. If not value is provided then the initial embedding is obtained using
- `get_ini_embed`.
- label: Base label for tSNE dimensions in the cell metadata column. (Default value: 'tSNE')
-
- Returns:
- """
- from uuid import uuid4
- from ..knn_utils import export_knn_to_mtx
- from pathlib import Path
-
- if from_assay is None:
- from_assay = self._defaultAssay
- if feat_key is None:
- feat_key = self._get_latest_feat_key(from_assay)
- n_cells, graph = self.load_unified_graph(
- from_assay=from_assay,
- cell_key=cell_key,
- feat_key=feat_key,
- target_names=target_names,
- use_k=use_k,
- target_weight=target_weight,
- )
- ini_embed = self._get_uni_ini_embed(
- from_assay, cell_key, feat_key, graph, ini_embed_with, n_cells[0]
- )
-
- uid = str(uuid4())
- ini_emb_fn = Path(temp_file_loc, f"{uid}.txt").resolve()
- with open(ini_emb_fn, "w") as h:
- h.write("\n".join(map(str, ini_embed.flatten())))
- del ini_embed
- knn_mtx_fn = Path(temp_file_loc, f"{uid}.mtx").resolve()
- export_knn_to_mtx(str(knn_mtx_fn), graph)
- out_fn = Path(temp_file_loc, f"{uid}_output.txt").resolve()
- cmd = (
- f"sgtsne -m {max_iter} -l {lambda_scale} -d {2} -e {early_iter} -p 1 -a {alpha}"
- f" -h {box_h} -i {ini_emb_fn} -o {out_fn} {knn_mtx_fn}"
- )
- if verbose:
- system_call(cmd)
- else:
- os.system(cmd)
- t = pd.read_csv(out_fn, header=None, sep=" ")[[0, 1]].values
- self._save_embedding(from_assay, cell_key, label, t, n_cells, target_names)
- for fn in [out_fn, knn_mtx_fn, ini_emb_fn]:
- Path.unlink(fn)
- return None
-
- def plot_unified_layout(
- self,
- from_assay: Optional[str] = None,
- layout_key: Optional[str] = None,
- show_target_only: bool = False,
- ref_name: str = "reference",
- target_groups: Optional[List[str]] = None,
- width: float = 6,
- height: float = 6,
- cmap: Optional[str] = None,
- color_key: Optional[Dict] = None,
- mask_color: str = "k",
- point_size: float = 10,
- ax_label_size: float = 12,
- frame_offset: float = 0.05,
- spine_width: float = 0.5,
- spine_color: str = "k",
- displayed_sides: tuple = ("bottom", "left"),
- legend_ondata: bool = False,
- legend_onside: bool = True,
- legend_size: float = 12,
- legends_per_col: int = 20,
- title: Union[str, List[str], None] = None,
- title_size: int = 12,
- hide_title: bool = False,
- cbar_shrink: float = 0.6,
- marker_scale: float = 70,
- lspacing: float = 0.1,
- cspacing: float = 1,
- savename: Optional[str] = None,
- save_dpi: int = 300,
- ax=None,
- force_ints_as_cats: bool = True,
- n_columns: int = 1,
- w_pad: float = 1,
- h_pad: float = 1,
- scatter_kwargs: Optional[Dict] = None,
- shuffle_zorder: bool = True,
- show_fig: bool = True,
- ):
- """Plots the reference and target cells in their unified space.
-
- This function helps to plot the reference and target cells, the coordinates for which were obtained from
- either `run_unified_tsne` or `run_unified_umap`. Since the coordinates are not saved in the cell metadata
- but rather in the projections slot of the Zarr hierarchy, this function is needed to correctly fetch the values
- for reference and target cells. Additionally, this function provides a way to colour target cells by bringing in
- external annotations for those cells.
-
- Args:
- from_assay: Name of assay to be used. If no value is provided then the default assay will be used.
- layout_key: Should be same as the parameter value for `label` in `run_unified_umap` or `run_unified_tsne`
- (Default value: 'UMAP')
- show_target_only: If True then the reference cells are not shown (Default value: False)
- ref_name: A label for reference cells to be used in the legend. (Default value: 'reference')
- target_groups: Categorical values to be used to colourmap target cells. (Default value: None)
- width: Figure width (Default value: 6)
- height: Figure height (Default value: 6)
- cmap: A matplotlib colourmap to be used to colour categorical or continuous values plotted on the cells.
- (Default value: tab20 for categorical variables and cmocean.deep for continuous variables)
- color_key: A custom colour map for cells. These can be used for categorical variables only. The keys in this
- dictionary should be the category label as present in the `color_by` column and values should be
- valid matplotlib colour names or hex codes of colours. (Default value: None)
- mask_color: Color to be used for masked values. This should be a valid matplotlib named colour or a hexcode
- of a colour. (Default value: 'k')
- point_size: Size of each scatter point. This is overridden if `size_vals` is provided. Has no effect if
- `do_shading` is True. (Default value: 10)
- ax_label_size: Font size for the x and y-axis labels. (Default value: 12)
- frame_offset: Extend the x and y-axis limits by this fraction (Default value: 0.05)
- spine_width: Line width of the displayed spines (Default value: 0.5)
- spine_color: Colour of the displayed spines. (Default value: 'k')
- displayed_sides: Determines which figure spines are chosen. The spines to be shown can be supplied as a
- tuple. The options are: top, bottom, left and right. (Default value: ('bottom', 'left) )
- legend_ondata: Whether to show category labels on the data (scatter points). The position of the label is
- the centroid of the corresponding values.
- (Default value: True)
- legend_onside: Whether to draw a legend table on the side of the figure. (Default value: True)
- legend_size: Font size of the legend text. (Default value: 12)
- legends_per_col: Number of legends to be used on each legend column. This value determines how many
- legend columns will be drawn (Default value: 20)
- title: Title to be used for plot. (Default value: None)
- title_size: Size of each axis/subplots title (Default value: 12)
- hide_title: If True, then the title of the sublots is not shown (Default value: False)
- cbar_shrink: Shrinking factor for the width of color bar (Default value: 0.6)
- marker_scale: The relative size of legend markers compared with the originally drawn ones.
- (Default value: 70)
- lspacing: The vertical space between the legend entries. Measured in font-size units. (Default value: 0.1)
- cspacing: The spacing between columns. Measured in font-size units. (Default value: 1)
- savename: Path where the rendered figure is to be saved. The format of the saved image depends on
- the extension present in the parameter value. (Default value: None)
- save_dpi: DPI when saving figure (Default value: 300)
- ax: An instance of Matplotlib's Axes object. This can be used to plot the figure into an already
- created axes. (Default value: None)
- force_ints_as_cats: Force integer labels in `color_by` as categories. If False, then integer will be
- treated as continuous variables otherwise as categories. This effects how colormaps
- are chosen and how legends are rendered. Set this to False if you are large number of
- unique integer entries (Default: True)
- n_columns: Number of columns in the grid
- w_pad: When plotting in multiple plots in a grid this decides the width padding between the plots.
- If None is provided the padding will be automatically added to avoid overlap.
- Ignored if only plotting one scatterplot.
- h_pad: When plotting in multiple plots in a grid this decides the height padding between the plots.
- If None is provided the padding will be automatically added to avoid overlap.
- Ignored if only plotting one scatterplot.
- scatter_kwargs: Keyword argument to be passed to matplotlib's scatter command
- shuffle_zorder: Whether to shuffle the plot order of data points in the figure. (Default value: True)
- show_fig: Whether to render the figure and display it using plt.show() (Default value: True)
-
- Returns:
- None
- """
-
- from ..plots import plot_scatter
-
- if from_assay is None:
- from_assay = self._defaultAssay
- if layout_key is None:
- raise ValueError(
- "ERROR: Please provide a value for the `layout_key` parameter. This should be same as "
- "that for either `run_unified_umap` or `run_unified_tsne`. Please see the default values "
- "for `label` parameter in those functions if unsure."
- )
- t = self.zw[from_assay].projections[layout_key][:]
- attrs = dict(self.zw[from_assay].projections[layout_key].attrs)
- t_names = attrs["target_names"]
- ref_n_cells = attrs["n_cells"][0]
- t_n_cells = attrs["n_cells"][1:]
- x = t[:, 0]
- y = t[:, 1]
- df = pd.DataFrame({f"{layout_key}1": x, f"{layout_key}2": y})
- if target_groups is None:
- if color_key is not None:
- temp_raise_error = False
- if ref_name not in color_key:
- temp_raise_error = True
- for i in t_names:
- if i not in color_key:
- temp_raise_error = True
- if temp_raise_error:
- temp = " ".join(t_names)
- raise KeyError(
- f"ERROR: `color_key` must contain these keys: '{ref_name}' and '{temp}'"
- )
- else:
- import seaborn as sns
-
- temp_cmap = sns.color_palette("hls", n_colors=len(t_names) + 1).as_hex()
- color_key = {k: v for k, v in zip(t_names, temp_cmap[1:])}
- color_key[ref_name] = temp_cmap[0]
- target_groups = []
- for i, j in zip(t_names, t_n_cells):
- target_groups.extend([i for _ in range(j)])
- target_groups = np.array(target_groups).astype(object)
- mask_values = None
- mask_name = "NA"
- else:
- if len(target_groups) == len(t_names):
- temp = []
- for i in target_groups:
- temp.extend(list(i))
- target_groups = list(temp)
- color_key = None
- mask_values = [ref_name]
- mask_name = ref_name
- target_groups = np.array(target_groups).astype(object)
- if len(target_groups) != sum(t_n_cells):
- raise ValueError(
- "ERROR: Number of values in `target_groups` should be same as no. of target cells"
- )
- # Turning array to object forces np.NaN to 'nan'
- if any(target_groups == "nan"):
- raise ValueError("ERROR: `target_groups` cannot contain nan values")
- df["vc"] = np.hstack(
- [[ref_name for _ in range(ref_n_cells)], target_groups]
- ).astype(object)
- if show_target_only:
- df = df[ref_n_cells:]
- if shuffle_zorder:
- df = df.sample(frac=1)
- return plot_scatter(
- [df],
- ax,
- width,
- height,
- mask_color,
- cmap,
- color_key,
- mask_values,
- mask_name,
- mask_color,
- point_size,
- ax_label_size,
- frame_offset,
- spine_width,
- spine_color,
- displayed_sides,
- legend_ondata,
- legend_onside,
- legend_size,
- legends_per_col,
- title,
- title_size,
- hide_title,
- cbar_shrink,
- marker_scale,
- lspacing,
- cspacing,
- savename,
- save_dpi,
- force_ints_as_cats,
- n_columns,
- w_pad,
- h_pad,
- show_fig,
- scatter_kwargs,
- )
diff --git a/scarf/datastore/pipeline_accessor.py b/scarf/datastore/pipeline_accessor.py
new file mode 100644
index 00000000..d0709c59
--- /dev/null
+++ b/scarf/datastore/pipeline_accessor.py
@@ -0,0 +1,806 @@
+from collections.abc import Callable, Iterator
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from contextlib import contextmanager
+from dataclasses import dataclass
+from typing import Any, Literal
+
+import numpy as np
+
+from ..metadata.artifacts import column_display, link_cell_data_column
+from ..storage.artifacts import ArtifactRef, group_at
+from ..storage.types import as_zarr_array, as_zarr_group
+from ..utils.logging import logger
+
+
+type StepOptions = dict[str, Any] | Literal[False] | None
+type PipelineEventKind = Literal[
+ "stage_started",
+ "stage_completed",
+ "stage_failed",
+]
+
+
+@dataclass(frozen=True, slots=True)
+class PipelineEvent:
+ kind: PipelineEventKind
+ stage: str
+ error: Exception | None = None
+
+
+type PipelineCallback = Callable[[PipelineEvent], None]
+
+
+class _PipelineEventEmitter:
+ __slots__ = ("_callback",)
+
+ def __init__(self, callback: PipelineCallback | None) -> None:
+ self._callback = callback
+
+ def emit(
+ self,
+ kind: PipelineEventKind,
+ stage: str,
+ error: Exception | None = None,
+ ) -> None:
+ if self._callback is None:
+ return
+ event = PipelineEvent(kind=kind, stage=stage, error=error)
+ try:
+ self._callback(event)
+ except Exception:
+ logger.exception(
+ f"Pipeline callback failed while handling {kind} for {stage}"
+ )
+
+ def start(self, stage: str) -> None:
+ logger.info(f"Running pipeline stage: {stage.replace('_', ' ')}")
+ self.emit("stage_started", stage)
+
+ def complete(self, stage: str) -> None:
+ self.emit("stage_completed", stage)
+ logger.info(f"Completed pipeline stage: {stage.replace('_', ' ')}")
+
+ @contextmanager
+ def stage(self, stage: str) -> Iterator[None]:
+ self.start(stage)
+ try:
+ yield
+ except Exception as error:
+ self.emit("stage_failed", stage, error)
+ raise
+ self.complete(stage)
+
+
+_DEFAULT_LEIDEN: dict[float, dict[str, Any]] = {
+ 0.5: {},
+ 0.75: {},
+ 1.0: {},
+ 1.25: {},
+}
+
+_SELECTED_CLUSTER_LABEL = "clusters"
+
+
+class PipelineAccessor:
+ __slots__ = ("_store",)
+
+ def __init__(self, store: Any) -> None:
+ self._store = store
+
+ def _column_ref(self, column: str) -> ArtifactRef:
+ cell_data = as_zarr_group(
+ self._store.zw["cellData"],
+ name="cellData",
+ )
+ raw_ref = as_zarr_array(
+ cell_data[column],
+ name=column,
+ ).attrs.get("source_artifact")
+ if not isinstance(raw_ref, dict):
+ raise RuntimeError(f"Pipeline output column {column!r} has no artifact ref")
+ return ArtifactRef.from_dict(raw_ref)
+
+ def _column_source_value(self, column: str) -> str:
+ cell_data = as_zarr_group(
+ self._store.zw["cellData"],
+ name="cellData",
+ )
+ value_name = as_zarr_array(
+ cell_data[column],
+ name=column,
+ ).attrs.get("source_value")
+ if not isinstance(value_name, str):
+ raise RuntimeError(
+ f"Cluster column {column!r} is not linked to an artifact value"
+ )
+ return value_name
+
+ def _feature_ref(self, assay_name: str, column: str) -> ArtifactRef:
+ assay = self._store._get_assay(assay_name)
+ feature_data = as_zarr_group(
+ assay.z["featureData"],
+ name="featureData",
+ )
+ raw_ref = as_zarr_array(
+ feature_data[column],
+ name=column,
+ ).attrs.get("source_artifact")
+ if not isinstance(raw_ref, dict):
+ raise RuntimeError(
+ f"Pipeline feature column {column!r} has no artifact ref"
+ )
+ return ArtifactRef.from_dict(raw_ref)
+
+ def _marker_ref(
+ self,
+ assay_name: str,
+ cell_key: str,
+ group_key: str,
+ ) -> ArtifactRef:
+ assay = self._store._get_assay(assay_name)
+ markers = as_zarr_group(assay.z["markers"], name="markers")
+ raw_artifacts = markers.attrs.get("artifacts", {})
+ if not isinstance(raw_artifacts, dict):
+ raise RuntimeError("Marker artifact index is invalid")
+ raw_ref = raw_artifacts.get(f"{cell_key}__{group_key}")
+ if not isinstance(raw_ref, dict):
+ raise RuntimeError("Marker search did not write an artifact")
+ return ArtifactRef.from_dict(raw_ref)
+
+ @staticmethod
+ def _options(value: StepOptions) -> dict[str, Any]:
+ return {} if value is None or value is False else dict(value)
+
+ @staticmethod
+ def _resolution_label(resolution: float) -> str:
+ return f"leiden_{resolution}"
+
+ @staticmethod
+ def _cluster_recipe_key(value: Any) -> str:
+ if value == "paris":
+ return "paris"
+ if isinstance(value, dict) and set(value) == {"leiden"}:
+ return PipelineAccessor._resolution_label(float(value["leiden"]))
+ if isinstance(value, int | float):
+ return PipelineAccessor._resolution_label(float(value))
+ if isinstance(value, str) and value.startswith("leiden_"):
+ return value
+ raise ValueError(
+ "clusters must be 'paris', a Leiden resolution, or {'leiden': resolution}"
+ )
+
+ def _load_pca_coordinates(self, reduction: ArtifactRef) -> np.ndarray:
+ status = self._store.inspect_artifact(reduction)
+ group = group_at(self._store.zw, status.path)
+ if "data" not in group:
+ source, _n_cells, _dims = self._store._coordinate_source(
+ reduction,
+ batch_size=None,
+ )
+ blocks = list(source.iter_coordinate_blocks("Loading PCA for silhouette"))
+ if not blocks:
+ raise RuntimeError("PCA reduction produced no coordinate blocks")
+ return np.vstack(blocks)
+ return np.asarray(as_zarr_array(group["data"], name="data")[:], dtype=float)
+
+ def _select_clusters_by_pca_silhouette(
+ self,
+ *,
+ reduction: ArtifactRef,
+ cell_key: str,
+ cluster_columns: dict[str, str],
+ ) -> str:
+ from sklearn.metrics import silhouette_score
+
+ if not cluster_columns:
+ raise ValueError(
+ "Silhouette cluster selection requires at least one clustering result"
+ )
+ coordinates = self._load_pca_coordinates(reduction)
+ sample_size = min(10_000, coordinates.shape[0])
+ best_key: str | None = None
+ best_score = float("-inf")
+ scores: dict[str, float] = {}
+ for recipe_key, column in cluster_columns.items():
+ labels = np.asarray(self._store.cells.fetch(column, key=cell_key))
+ if labels.shape[0] != coordinates.shape[0]:
+ raise RuntimeError(
+ "PCA coordinates and cluster labels cover different cells"
+ )
+ n_labels = len(np.unique(labels))
+ if n_labels < 2:
+ logger.warning(
+ f"Skipping silhouette for {recipe_key}: fewer than two clusters"
+ )
+ continue
+ score = float(
+ silhouette_score(
+ coordinates,
+ labels,
+ sample_size=sample_size if sample_size < labels.shape[0] else None,
+ random_state=4466,
+ )
+ )
+ scores[recipe_key] = score
+ if score > best_score:
+ best_score = score
+ best_key = recipe_key
+ if best_key is None:
+ raise RuntimeError(
+ "Could not score any clustering partition with silhouette"
+ )
+ logger.info(
+ "Cluster silhouette scores on PCA: "
+ + ", ".join(f"{key}={value:.4f}" for key, value in scores.items())
+ + f"; selected {best_key}"
+ )
+ return best_key
+
+ def _publish_selected_clusters(
+ self,
+ *,
+ assay_name: str,
+ cell_key: str,
+ source_column: str,
+ ref: ArtifactRef,
+ ) -> str:
+ store = self._store
+ column: str = store._col_renamer(assay_name, cell_key, _SELECTED_CLUSTER_LABEL)
+ if column == source_column:
+ return column
+ labels = np.asarray(store.cells.fetch(source_column, key=cell_key))
+ store.cells.insert(
+ column,
+ labels,
+ fill_value=-1,
+ key=cell_key,
+ overwrite=True,
+ )
+ link_cell_data_column(
+ store.zw,
+ column,
+ ref,
+ value_name=self._column_source_value(source_column),
+ default_display=column_display(store.zw, source_column),
+ )
+ logger.info(f"Selected {source_column} as {column}")
+ return column
+
+ def _run_clustering_jobs(
+ self,
+ *,
+ graph: ArtifactRef,
+ assay_name: str,
+ cell_key: str,
+ feat_key: str,
+ leiden_options: dict[float, dict[str, Any]],
+ paris_options: dict[str, Any] | None,
+ clustering_concurrency: int,
+ events: _PipelineEventEmitter,
+ ) -> tuple[dict[str, str], dict[str, ArtifactRef]]:
+ store = self._store
+ cluster_columns: dict[str, str] = {}
+ artifacts: dict[str, ArtifactRef] = {}
+ if not leiden_options and paris_options is None:
+ return cluster_columns, artifacts
+
+ job_order: list[str] = []
+ prepared_leiden: dict[str, Any] = {}
+ started_jobs: set[str] = set()
+ terminal_jobs: set[str] = set()
+
+ def start_job(recipe_key: str) -> None:
+ if recipe_key not in started_jobs:
+ events.start(recipe_key)
+ started_jobs.add(recipe_key)
+
+ def complete_job(recipe_key: str) -> None:
+ events.complete(recipe_key)
+ terminal_jobs.add(recipe_key)
+
+ def fail_job(recipe_key: str, error: Exception) -> None:
+ start_job(recipe_key)
+ events.emit("stage_failed", recipe_key, error)
+ terminal_jobs.add(recipe_key)
+
+ def fail_unpublished_jobs() -> None:
+ for recipe_key in job_order:
+ if recipe_key in started_jobs and recipe_key not in terminal_jobs:
+ fail_job(
+ recipe_key,
+ RuntimeError(
+ f"Clustering job {recipe_key} was not written because "
+ "another clustering job failed"
+ ),
+ )
+
+ for raw_resolution, raw_options in leiden_options.items():
+ resolution = float(raw_resolution)
+ options = dict(raw_options)
+ recipe_key = self._resolution_label(resolution)
+ if recipe_key in prepared_leiden:
+ raise ValueError(f"Duplicate Leiden resolution {resolution}")
+ label = str(options.pop("label", recipe_key))
+ job_order.append(recipe_key)
+ try:
+ prepared_leiden[recipe_key] = store._prepare_leiden_clustering(
+ graph,
+ from_assay=assay_name,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ resolution=resolution,
+ label=label,
+ **options,
+ )
+ except Exception as error:
+ fail_job(recipe_key, error)
+ raise
+
+ paris_job: dict[str, Any] | None = None
+ if paris_options is not None:
+ options = dict(paris_options)
+ paris_label = str(options.pop("label", "paris_cluster"))
+ job_order.append("paris")
+ paris_job = {
+ "label": paris_label,
+ "options": options,
+ }
+
+ graph_cache: dict[tuple[str, bool, bool], Any] = {}
+ for recipe_key, prepared in prepared_leiden.items():
+ if prepared.planned.reused or prepared.graph_key in graph_cache:
+ continue
+ try:
+ graph_cache[prepared.graph_key] = store._load_prepared_leiden_graph(
+ prepared
+ )
+ except Exception as error:
+ fail_job(recipe_key, error)
+ raise
+
+ compute_results: dict[str, np.ndarray] = {}
+ completed: dict[str, str] = {}
+ artifact_refs: dict[str, ArtifactRef] = {}
+ first_error: Exception | None = None
+ runnable_jobs = sum(
+ not prepared.planned.reused for prepared in prepared_leiden.values()
+ ) + int(paris_job is not None)
+ if runnable_jobs:
+ workers = max(1, min(clustering_concurrency, runnable_jobs))
+ with ThreadPoolExecutor(max_workers=workers) as executor:
+ futures: dict[Any, str] = {}
+ for recipe_key in job_order:
+ if recipe_key == "paris":
+ assert paris_job is not None
+ start_job(recipe_key)
+ try:
+ future = executor.submit(
+ store.run_paris_clustering,
+ graph,
+ from_assay=assay_name,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ label=paris_job["label"],
+ **paris_job["options"],
+ )
+ except Exception as error:
+ fail_job(recipe_key, error)
+ if first_error is None:
+ first_error = error
+ continue
+ else:
+ prepared = prepared_leiden[recipe_key]
+ if prepared.planned.reused:
+ continue
+ start_job(recipe_key)
+ try:
+ future = executor.submit(
+ store._compute_prepared_leiden,
+ prepared,
+ graph_cache[prepared.graph_key],
+ )
+ except Exception as error:
+ fail_job(recipe_key, error)
+ if first_error is None:
+ first_error = error
+ continue
+ futures[future] = recipe_key
+ for future in as_completed(futures):
+ recipe_key = futures[future]
+ try:
+ result = future.result()
+ if recipe_key == "paris":
+ if result is None or result.label_key is None:
+ raise RuntimeError(
+ "Paris clustering did not write labels"
+ )
+ if result.ref is None:
+ raise RuntimeError(
+ "Paris clustering did not record an artifact"
+ )
+ completed[recipe_key] = result.label_key
+ artifact_refs[recipe_key] = result.ref
+ complete_job(recipe_key)
+ else:
+ compute_results[recipe_key] = result
+ except Exception as error:
+ fail_job(recipe_key, error)
+ if first_error is None:
+ first_error = error
+
+ if first_error is not None:
+ fail_unpublished_jobs()
+ raise first_error
+
+ for recipe_key, prepared in prepared_leiden.items():
+ start_job(recipe_key)
+ try:
+ column, ref = store._publish_prepared_leiden(
+ prepared,
+ compute_results.get(recipe_key),
+ )
+ completed[recipe_key] = column
+ artifact_refs[recipe_key] = ref
+ except Exception as error:
+ fail_job(recipe_key, error)
+ fail_unpublished_jobs()
+ raise
+ complete_job(recipe_key)
+
+ for recipe_key in job_order:
+ column = completed[recipe_key]
+ cluster_columns[recipe_key] = column
+ artifacts[recipe_key] = artifact_refs[recipe_key]
+ return cluster_columns, artifacts
+
+ def run(
+ self,
+ pipeline_id: str = "basic_rna_analysis",
+ *,
+ from_assay: str | None = None,
+ cell_key: str = "I",
+ filtering: StepOptions = None,
+ cell_cycle_scoring: StepOptions = None,
+ highly_variable_features: StepOptions = None,
+ normalization: StepOptions = None,
+ pca: StepOptions = None,
+ harmony: dict[str, Any] | None = None,
+ ann_index: StepOptions = None,
+ neighbors: StepOptions = None,
+ connectivity: StepOptions = None,
+ umap: StepOptions = None,
+ leiden: dict[float, dict[str, Any]] | None = None,
+ paris: StepOptions = None,
+ clustering_concurrency: int = 2,
+ doublet_scoring: StepOptions = None,
+ markers: StepOptions = None,
+ callback: PipelineCallback | None = None,
+ ) -> dict[str, ArtifactRef]:
+ """Run the standard provenance-backed RNA analysis recipe.
+
+ Most step options accept ``None`` to run with defaults, ``False`` to
+ skip, or a dictionary forwarded to the underlying operation. Harmony
+ is skipped when omitted and requires a dictionary containing
+ ``batch_columns``. Leiden defaults to resolutions 0.5, 0.75, 1.0, and
+ 1.25. Leiden and Paris membership work can overlap under
+ ``clustering_concurrency`` while store writes stay serialized. When
+ more than one partition is available, the one with the highest
+ silhouette score on PCA coordinates is selected. Its labels are copied
+ to ``{assay}_clusters``, linked to the same artifact, and used by
+ doublet scoring and marker search unless those steps name a partition
+ through ``clusters``. Highly variable feature selection is mandatory.
+ When provided, ``callback`` receives
+ serialized stage events on the calling thread. Callback errors are
+ logged without interrupting the pipeline. Stable stage names are
+ ``filtering``, ``cell_cycle_scoring``, ``highly_variable_features``,
+ ``normalization``, ``pca``, ``harmony``, ``ann_index``, ``neighbors``,
+ ``connectivity``, ``embedding_initialization``, ``umap``,
+ ``cluster_selection``, ``doublet_scoring``, and ``markers``. Clustering
+ jobs use ``leiden_`` and ``paris``. Skipped stages emit no
+ events. ``stage_completed`` means the expected output has finished
+ writing and is available. If one clustering job fails, any started
+ sibling that cannot finish writing
+ emits ``stage_failed`` with an abort error; the original job error is
+ re-raised.
+
+ Args:
+ pipeline_id: Recipe identifier. Only ``basic_rna_analysis`` is
+ currently available.
+ from_assay: RNA assay to analyze. Uses the default assay when
+ omitted.
+ cell_key: Boolean cell selection used throughout the recipe.
+ filtering: Filtering options, including ``method="auto"`` or
+ ``method="manual"``.
+ cell_cycle_scoring: Cell-cycle scoring options or ``False``.
+ highly_variable_features: HVG selection options. Cannot be
+ ``False``.
+ normalization: Normalization options.
+ pca: PCA options. ``n_centroids`` is consumed by embedding
+ initialization.
+ harmony: Harmony options with a non-empty ``batch_columns`` list.
+ ann_index: ANN-index construction options.
+ neighbors: Neighbor-query options.
+ connectivity: Connectivity-map options.
+ umap: UMAP options or ``False``.
+ leiden: Mapping from resolution to Leiden options. Use an empty
+ mapping to run no Leiden clustering.
+ paris: Paris clustering options or ``False``.
+ clustering_concurrency: Maximum number of Leiden/Paris jobs that
+ may run at once. Membership compute can overlap; store writes
+ are serialized. Use ``1`` for a fully serial path.
+ doublet_scoring: Doublet-scoring options or ``False``.
+ markers: Marker-search options or ``False``.
+ callback: Optional callable receiving ``PipelineEvent`` values.
+
+ Returns:
+ Artifact references keyed by pipeline result name.
+
+ Raises:
+ ValueError: If the recipe identifier or dependent step options are
+ invalid.
+ RuntimeError: If a step does not write its expected artifact.
+ """
+ if pipeline_id != "basic_rna_analysis":
+ raise ValueError(
+ f"Unknown pipeline_id {pipeline_id!r}; "
+ "available pipelines: basic_rna_analysis"
+ )
+ if (
+ isinstance(clustering_concurrency, bool)
+ or not isinstance(clustering_concurrency, int)
+ or clustering_concurrency < 1
+ ):
+ raise ValueError("clustering_concurrency must be an integer >= 1")
+ if callback is not None and not callable(callback):
+ raise TypeError("callback must be callable")
+ store = self._store
+ assay_name = from_assay or store._defaultAssay
+ if assay_name is None:
+ raise ValueError("No assay was provided and no default is configured")
+ if filtering is not False and cell_key != "I":
+ raise ValueError(
+ "basic_rna_analysis filtering currently requires cell_key='I'"
+ )
+ if highly_variable_features is False:
+ raise ValueError(
+ "basic_rna_analysis requires highly_variable_features; "
+ "pass options or omit the argument to use defaults"
+ )
+ if isinstance(markers, dict) and markers.get("skip_save") is True:
+ raise ValueError("basic_rna_analysis markers cannot use skip_save=True")
+ artifacts: dict[str, ArtifactRef] = {}
+ events = _PipelineEventEmitter(callback)
+
+ if filtering is not False:
+ with events.stage("filtering"):
+ options = self._options(filtering)
+ method = options.pop("method", "auto")
+ if method == "auto":
+ options.setdefault("show_qc_plots", False)
+ if "attrs" not in options:
+ options["attrs"] = [
+ column
+ for suffix in (
+ "nCounts",
+ "nFeatures",
+ "percentMito",
+ "percentRibo",
+ )
+ if (column := f"{assay_name}_{suffix}")
+ in store.cells.columns
+ ]
+ store.auto_filter_cells(**options)
+ elif method == "manual":
+ store.filter_cells(**options)
+ else:
+ raise ValueError("filtering method must be 'auto' or 'manual'")
+ artifacts["cell_selection"] = store._ensure_cell_selection(cell_key)
+
+ if cell_cycle_scoring is not False:
+ with events.stage("cell_cycle_scoring"):
+ options = self._options(cell_cycle_scoring)
+ store.run_cell_cycle_scoring(
+ from_assay=assay_name,
+ cell_key=cell_key,
+ **options,
+ )
+ phase_label = options.get("phase_label", "cell_cycle_phase")
+ phase_column = store._col_renamer(
+ assay_name,
+ cell_key,
+ phase_label,
+ )
+ artifacts["cell_cycle"] = self._column_ref(phase_column)
+
+ with events.stage("highly_variable_features"):
+ hvg_options = self._options(highly_variable_features)
+ hvg_name = str(hvg_options.get("hvg_key_name", "hvgs"))
+ hvg_options.setdefault("show_plot", False)
+ hvg_options.setdefault("top_n", 2000)
+ hvg_options.setdefault("min_cells", 20)
+ store.mark_hvgs(
+ from_assay=assay_name,
+ cell_key=cell_key,
+ **hvg_options,
+ )
+ feature_column = f"{cell_key}__{hvg_name}"
+ artifacts["highly_variable_features"] = self._feature_ref(
+ assay_name,
+ feature_column,
+ )
+
+ with events.stage("normalization"):
+ normalization_options = self._options(normalization)
+ normalization_options.setdefault("log_transform", True)
+ normalization_options.setdefault("renormalize_subset", True)
+ normalized = store.run_normalization(
+ from_assay=assay_name,
+ cell_key=cell_key,
+ feat_key=hvg_name,
+ update_state=False,
+ **normalization_options,
+ )
+ artifacts["normalized"] = normalized
+
+ with events.stage("pca"):
+ pca_options = self._options(pca)
+ n_centroids = int(pca_options.pop("n_centroids", 1000))
+ initialization_rand_state = int(pca_options.pop("rand_state", 4466))
+ pca_options.setdefault("dims", 25)
+ reduction = store.run_pca(
+ normalized,
+ update_state=False,
+ **pca_options,
+ )
+ artifacts["pca"] = reduction
+
+ coordinates = reduction
+ if harmony is not None:
+ with events.stage("harmony"):
+ harmony_options = dict(harmony)
+ batch_columns = harmony_options.pop("batch_columns", None)
+ if not isinstance(batch_columns, list) or not batch_columns:
+ raise ValueError("harmony requires a non-empty batch_columns list")
+ coordinates = store.run_harmony(
+ batch_columns,
+ reduction,
+ update_state=False,
+ **harmony_options,
+ )
+ artifacts["harmony"] = coordinates
+
+ with events.stage("ann_index"):
+ ann = store.build_ann_index(
+ coordinates,
+ update_state=False,
+ **self._options(ann_index),
+ )
+ artifacts["ann_index"] = ann
+
+ with events.stage("neighbors"):
+ neighbor_options = self._options(neighbors)
+ neighbor_options.setdefault("k", 17)
+ neighbor_ref = store.query_neighbors(
+ ann,
+ coordinates=coordinates,
+ update_state=False,
+ **neighbor_options,
+ )
+ artifacts["neighbors"] = neighbor_ref
+
+ with events.stage("connectivity"):
+ graph = store.build_connectivity_map(
+ neighbor_ref,
+ update_state=False,
+ **self._options(connectivity),
+ )
+ artifacts["connectivity_map"] = graph
+
+ with events.stage("embedding_initialization"):
+ initialization = store._build_embedding_initialization(
+ reduction,
+ n_centroids=n_centroids,
+ rand_state=initialization_rand_state,
+ batch_size=None,
+ invalidate_cache=False,
+ )
+ store._publish_current_artifact(
+ graph,
+ update_state=True,
+ embedding_initialization=initialization,
+ )
+ artifacts["embedding_initialization"] = initialization
+
+ if umap is not False:
+ with events.stage("umap"):
+ umap_options = self._options(umap)
+ artifacts["umap"] = store.run_umap(
+ graph,
+ from_assay=assay_name,
+ cell_key=cell_key,
+ feat_key=hvg_name,
+ **umap_options,
+ )
+
+ leiden_options = dict(_DEFAULT_LEIDEN) if leiden is None else dict(leiden)
+ paris_options = None if paris is False else self._options(paris)
+ cluster_columns, cluster_artifacts = self._run_clustering_jobs(
+ graph=graph,
+ assay_name=assay_name,
+ cell_key=cell_key,
+ feat_key=hvg_name,
+ leiden_options=leiden_options,
+ paris_options=paris_options,
+ clustering_concurrency=clustering_concurrency,
+ events=events,
+ )
+ artifacts.update(cluster_artifacts)
+
+ doublet_options = (
+ None if doublet_scoring is False else self._options(doublet_scoring)
+ )
+ marker_options = None if markers is False else self._options(markers)
+ selected_column: str | None = None
+ if cluster_columns:
+ with events.stage("cluster_selection"):
+ if len(cluster_columns) == 1:
+ selected_recipe_key = next(iter(cluster_columns))
+ else:
+ selected_recipe_key = self._select_clusters_by_pca_silhouette(
+ reduction=reduction,
+ cell_key=cell_key,
+ cluster_columns=cluster_columns,
+ )
+ artifacts["selected_clusters"] = artifacts[selected_recipe_key]
+ selected_column = self._publish_selected_clusters(
+ assay_name=assay_name,
+ cell_key=cell_key,
+ source_column=cluster_columns[selected_recipe_key],
+ ref=artifacts[selected_recipe_key],
+ )
+
+ def _cluster_column(options: dict[str, Any], step: str) -> str:
+ if "clusters" in options:
+ recipe_key = self._cluster_recipe_key(options.pop("clusters"))
+ if recipe_key not in cluster_columns:
+ raise ValueError(
+ f"{step} cluster result {recipe_key!r} is unavailable"
+ )
+ return cluster_columns[recipe_key]
+ if selected_column is None:
+ raise ValueError(
+ f"{step} needs a clustering result; run Leiden or Paris, "
+ "or disable this step"
+ )
+ return selected_column
+
+ if doublet_options is not None:
+ with events.stage("doublet_scoring"):
+ options = dict(doublet_options)
+ score_column = store.run_doublet_detection(
+ cluster_key=_cluster_column(options, "Doublet"),
+ from_assay=assay_name,
+ cell_key=cell_key,
+ feat_key=hvg_name,
+ **options,
+ )
+ artifacts["doublets"] = self._column_ref(score_column)
+
+ if marker_options is not None:
+ with events.stage("markers"):
+ options = dict(marker_options)
+ group_key = _cluster_column(options, "Marker")
+ options.setdefault("feat_key", "I")
+ store.run_marker_search(
+ from_assay=assay_name,
+ cell_key=cell_key,
+ group_key=group_key,
+ **options,
+ )
+ artifacts["markers"] = self._marker_ref(
+ assay_name,
+ cell_key,
+ group_key,
+ )
+ logger.info(f"Pipeline completed with {len(artifacts)} artifacts")
+ return artifacts
diff --git a/scarf/datastore/plot_accessor.py b/scarf/datastore/plot_accessor.py
new file mode 100644
index 00000000..5ab70a4b
--- /dev/null
+++ b/scarf/datastore/plot_accessor.py
@@ -0,0 +1,7 @@
+"""Public datastore plotting accessor."""
+
+from ._plot_accessor import DataStorePlotAccessor as DataStorePlotAccessor
+
+__all__ = ["DataStorePlotAccessor"]
+
+DataStorePlotAccessor.__module__ = __name__
diff --git a/scarf/datastore/summary.py b/scarf/datastore/summary.py
new file mode 100644
index 00000000..7029cf45
--- /dev/null
+++ b/scarf/datastore/summary.py
@@ -0,0 +1,213 @@
+from collections.abc import Mapping
+from dataclasses import dataclass
+from typing import Any, Protocol
+
+import zarr
+
+from ..assay import Assay
+from ..graph.state import AssayState
+from ..metadata import MetaData
+from ..storage.artifacts import ArtifactStatus
+from ..storage.budget import ResourceBudget
+from ..storage.profiles import StorageProfile
+from ..storage.refs import ArtifactRef, ArtifactScope
+from ..storage.types import ZarrMode
+
+
+@dataclass(frozen=True, slots=True)
+class ResourceSummary:
+ memory_bytes: int
+ workers: int
+ storage_profile: StorageProfile
+
+ def to_dict(self) -> dict[str, int | str]:
+ return {
+ "memory_bytes": self.memory_bytes,
+ "workers": self.workers,
+ "storage_profile": self.storage_profile,
+ }
+
+
+@dataclass(frozen=True, slots=True)
+class ArtifactSummary:
+ ref: ArtifactRef
+ operation: str | None
+ complete: bool
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "ref": self.ref.to_dict(),
+ "operation": self.operation,
+ "complete": self.complete,
+ }
+
+
+@dataclass(frozen=True, slots=True)
+class AssaySummary:
+ name: str
+ assay_type: str
+ total_features: int
+ active_features: int
+ feature_columns: tuple[str, ...]
+ dataset_fingerprint: str | None
+ state: AssayState | None
+ artifacts: tuple[ArtifactSummary, ...]
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "name": self.name,
+ "assay_type": self.assay_type,
+ "total_features": self.total_features,
+ "active_features": self.active_features,
+ "feature_columns": list(self.feature_columns),
+ "dataset_fingerprint": self.dataset_fingerprint,
+ "state": self.state.to_dict() if self.state is not None else None,
+ "artifacts": [artifact.to_dict() for artifact in self.artifacts],
+ }
+
+
+@dataclass(frozen=True, slots=True)
+class DataStoreSummary:
+ zarr_mode: ZarrMode
+ workspace: str | None
+ default_assay: str
+ scarf_version: str
+ resources: ResourceSummary
+ total_cells: int
+ active_cells: int
+ cell_columns: tuple[str, ...]
+ assays: tuple[AssaySummary, ...]
+ artifacts: tuple[ArtifactSummary, ...]
+
+ def to_dict(self) -> dict[str, Any]:
+ """Return a deterministic representation containing JSON-safe values."""
+ return {
+ "zarr_mode": self.zarr_mode,
+ "workspace": self.workspace,
+ "default_assay": self.default_assay,
+ "scarf_version": self.scarf_version,
+ "resources": self.resources.to_dict(),
+ "total_cells": self.total_cells,
+ "active_cells": self.active_cells,
+ "cell_columns": list(self.cell_columns),
+ "assays": [assay.to_dict() for assay in self.assays],
+ "artifacts": [artifact.to_dict() for artifact in self.artifacts],
+ }
+
+
+class _SummaryStore(Protocol):
+ zarr_mode: ZarrMode
+ workspace: str | None
+ _defaultAssay: str
+ resources: ResourceBudget
+ storageProfile: StorageProfile
+ cells: MetaData
+
+ @property
+ def assay_names(self) -> list[str]: ...
+
+ @property
+ def zw(self) -> zarr.Group: ...
+
+ def _get_assay(self, from_assay: str | None) -> Assay: ...
+
+ def get_assay_state(self, from_assay: str | None = None) -> AssayState | None: ...
+
+ def list_artifacts(
+ self,
+ *,
+ kind: str | None = None,
+ from_assay: str | None = None,
+ scope: ArtifactScope = "assay",
+ complete_only: bool = False,
+ ) -> list[ArtifactRef]: ...
+
+ def inspect_artifact(self, ref: ArtifactRef) -> ArtifactStatus: ...
+
+
+def _count_active(metadata: MetaData) -> int:
+ if metadata.N == 0:
+ return 0
+ return sum(
+ int(block.active_global_indices.size)
+ for block in metadata.iter_row_blocks(cell_key="I")
+ )
+
+
+def _summarize_artifacts(
+ store: _SummaryStore,
+ refs: list[ArtifactRef],
+) -> tuple[ArtifactSummary, ...]:
+ summaries = []
+ for ref in sorted(
+ refs,
+ key=lambda item: (item.kind, item.assay or "", item.artifact_id),
+ ):
+ status = store.inspect_artifact(ref)
+ summaries.append(
+ ArtifactSummary(
+ ref=ref,
+ operation=status.operation,
+ complete=status.complete,
+ )
+ )
+ return tuple(summaries)
+
+
+def _assay_type_mapping(root: zarr.Group) -> dict[str, str]:
+ value = root.attrs.get("assayTypes", {})
+ if not isinstance(value, Mapping):
+ return {}
+ return {str(name): str(assay_type) for name, assay_type in value.items()}
+
+
+def build_datastore_summary(
+ store: _SummaryStore,
+ *,
+ scarf_version: str,
+) -> DataStoreSummary:
+ assay_types = _assay_type_mapping(store.zw)
+ assays = []
+ for assay_name in sorted(store.assay_names):
+ assay = store._get_assay(assay_name)
+ fingerprint = assay.attrs.get("dataset_fingerprint")
+ assays.append(
+ AssaySummary(
+ name=assay_name,
+ assay_type=assay_types.get(assay_name, "Assay"),
+ total_features=int(assay.feats.N),
+ active_features=_count_active(assay.feats),
+ feature_columns=tuple(sorted(assay.feats.columns)),
+ dataset_fingerprint=(
+ str(fingerprint) if fingerprint is not None else None
+ ),
+ state=store.get_assay_state(assay_name),
+ artifacts=_summarize_artifacts(
+ store,
+ store.list_artifacts(
+ from_assay=assay_name,
+ scope="assay",
+ ),
+ ),
+ )
+ )
+
+ return DataStoreSummary(
+ zarr_mode=store.zarr_mode,
+ workspace=store.workspace,
+ default_assay=store._defaultAssay,
+ scarf_version=scarf_version,
+ resources=ResourceSummary(
+ memory_bytes=store.resources.memoryBytes,
+ workers=store.resources.workers,
+ storage_profile=store.storageProfile,
+ ),
+ total_cells=int(store.cells.N),
+ active_cells=_count_active(store.cells),
+ cell_columns=tuple(sorted(store.cells.columns)),
+ assays=tuple(assays),
+ artifacts=_summarize_artifacts(
+ store,
+ store.list_artifacts(scope="datastore"),
+ ),
+ )
diff --git a/scarf/dendrogram.py b/scarf/dendrogram.py
deleted file mode 100644
index 1c70ca6a..00000000
--- a/scarf/dendrogram.py
+++ /dev/null
@@ -1,239 +0,0 @@
-from typing import List, Dict
-
-import networkx as nx
-import numpy as np
-
-from .utils import logger, tqdmbar
-
-__all__ = ["BalancedCut", "CoalesceTree", "make_digraph"]
-
-
-def make_digraph(d: np.ndarray, clust_info=None) -> nx.DiGraph:
- """Convert dendrogram into directed graph."""
-
- g = nx.DiGraph()
- n = d.shape[0] + 1 # Dendrogram contains one less sample
- if clust_info is not None:
- if len(clust_info) != d.shape[0] + 1:
- raise ValueError(
- "ERROR: cluster information doesn't match number of leaves in dendrogram"
- )
- else:
- clust_info = np.ones(d.shape[0] + 1) * -1
- for i in tqdmbar(d, desc="Constructing graph from dendrogram"):
- v = i[2] # Distance between clusters
- i = i.astype(int)
- g.add_node(n, nleaves=i[3], dist=v)
- if i[0] <= d.shape[0]:
- g.add_node(i[0], nleaves=0, dist=v, cluster=clust_info[i[0]])
- if i[1] <= d.shape[0]:
- g.add_node(i[1], nleaves=0, dist=v, cluster=clust_info[i[1]])
- g.add_edge(n, i[0])
- g.add_edge(n, i[1])
- n += 1
- if g.number_of_edges() != d.shape[0] * 2:
- logger.warning(
- "Number of edges in directed graph not twice the dendrogram shape"
- )
- return g
-
-
-def CoalesceTree(graph: nx.DiGraph, clusters: np.ndarray) -> nx.DiGraph:
- def calc_steps_to_top(g: nx.DiGraph, c: np.ndarray):
- import pandas as pd
-
- s = {}
- for i in range(len(c)):
- s[i] = 0
- q = [i]
- while len(q) > 0:
- for j in g.predecessors(q.pop(0)):
- s[i] += 1
- q.append(j)
- return pd.Series(s).sort_values()
-
- def iter_predecessors(g: nx.DiGraph, v):
- q = [v]
- while len(q) > 0:
- for i in g.predecessors(q.pop(0)):
- yield i
- q.append(i)
-
- def aggregate_leaves(g: nx.DiGraph, v):
- q = [v]
- l = []
- while len(q) > 0:
- for i in g.successors(q.pop(0)):
- if g.nodes[i]["nleaves"] == 0:
- l.append(i)
- else:
- q.append(i)
- return l
-
- def get_holding_nodes(g: nx.DiGraph, c):
- hn = {}
- s = calc_steps_to_top(g, c)
- for i in tqdmbar(set(c), desc="Identifying the top node for cluster"):
- l = set(np.where(c == i)[0])
- nl = len(l)
- for j in iter_predecessors(g, s.reindex(l).idxmin()):
- if g.nodes[j]["nleaves"] >= nl:
- l2 = aggregate_leaves(g, j)
- if len(l.intersection(l2)) == nl:
- hn[j] = i
- break
- return hn
-
- def aggregate_predecessors(g: nx.DiGraph, v):
- p = []
- for i in iter_predecessors(g, v):
- p.append(i)
- return p
-
- def make_subgraph(g: nx.DiGraph, vs):
- sn = list(vs.keys())
- for i in vs:
- sn.extend(aggregate_predecessors(g, i))
- sn = list(set(sn))
- sg = nx.DiGraph(nx.subgraph(g, sn))
- for i in vs:
- sg.nodes[i]["partition_id"] = vs[i]
- return sg
-
- return make_subgraph(graph, get_holding_nodes(graph, clusters))
-
-
-class BalancedCut:
- def __init__(
- self,
- dendrogram: np.ndarray,
- max_size: int,
- min_size: int,
- max_distance_fc: float,
- ):
- self.nCells = dendrogram.shape[0] + 1
- self.graph = make_digraph(dendrogram)
- self.maxSize = max_size
- self.minSize = min_size
- self.maxDistFc = max_distance_fc
- self.branchpoints = self._get_branchpoints()
-
- def _successors(self, start: int, min_leaves: int) -> List[int]:
- """Get tree downstream of a node."""
- q = [start]
- d = []
- while len(q) > 0:
- i = q.pop(0)
- if self.graph.nodes[i]["nleaves"] > min_leaves:
- d.append(i)
- q.extend(list(self.graph.successors(i)))
- return d[1:]
-
- def _get_mean_dist(self, start_node: int) -> float:
- """Get mean distances in downstream tree of a node."""
- s_nodes = self._successors(start_node, -1)
- return np.array([self.graph.nodes[x]["dist"] for x in s_nodes]).mean()
-
- def _are_subtrees_mergeable(self, s1: int, s2: int) -> bool:
- n1, n2 = self.graph.nodes[s1]["nleaves"], self.graph.nodes[s2]["nleaves"]
- if n1 > self.minSize and n2 > self.minSize:
- d1, d2 = self.graph.nodes[s1]["dist"], self.graph.nodes[s2]["dist"]
- if d1 / d2 > self.maxDistFc or d2 / d1 > self.maxDistFc:
- logger.trace(f"Will not merge {s1} and {s2} because of high distance")
- return False
- else:
- md1, md2 = self._get_mean_dist(s1), self._get_mean_dist(s2)
- if md1 / md2 > self.maxDistFc or md2 / md1 > self.maxDistFc:
- logger.trace(
- f"Will not merge {s1} and {s2} because of high distance of successors"
- )
- return False
- return True
-
- def _get_branchpoints(self) -> Dict[int, List[int]]:
- """Aggregate leaves bottom up until target size is reached."""
- n_leaves = int((self.graph.number_of_nodes() + 1) / 2)
- leaves = {x: None for x in range(n_leaves)}
- bps = {}
- pbar = tqdmbar(total=len(leaves), desc="Identifying nodes to split")
- while len(leaves) > 0:
- leaf, _ = leaves.popitem()
- pbar.update(1)
- logger.trace(f"FRESH STEP: Leaf {leaf} plucked as base leaf")
- cur = leaf
- while True:
- temp = next(self.graph.predecessors(cur))
- if temp in bps:
- logger.trace(f"Will not climb to {temp} as already a branchpoint")
- break
- if self.graph.nodes[temp]["nleaves"] > self.maxSize:
- logger.trace(
- f"Will not climb to {temp} because too many leaves exist"
- )
- break
- s1, s2 = list(self.graph.successors(temp))
- if self._are_subtrees_mergeable(s1, s2) is False:
- break
- cur = temp
- logger.trace(f"Aggregating from branch {cur} for leaf {leaf}")
- bps[cur] = [leaf]
- s = [cur]
- while len(s) > 0:
- i = s.pop()
- if i in leaves:
- bps[cur].append(i)
- leaves.pop(i)
- logger.trace(f"Leaf {i} plucked in aggregation step")
- pbar.update(1)
- elif i in bps and i != cur:
- logger.trace(f"Skipping branch {i} because its already taken")
- elif self.graph.nodes[i]["nleaves"] >= self.maxSize and i != cur:
- logger.trace(f"Skipping branch {i} to prevent greedy behaviour")
- else:
- s.extend(list(self.graph.successors(i)))
- pbar.close()
- return bps
-
- def _valid_names_in_branchpoints(self) -> None:
- leaves = []
- for i in self.branchpoints:
- leaves.extend(self.branchpoints[i])
- n_leaves = len(leaves)
- if n_leaves != self.nCells:
- raise ValueError(
- "ERROR: Not all leaves present in branchpoints. This bug must be reported"
- )
- minl = min(leaves)
- if minl != 0:
- raise ValueError(f"ERROR: minimum leaf label is {minl} rather than 0")
- maxl = max(leaves)
- if n_leaves != maxl + 1:
- raise ValueError(
- f"ERROR: maximum leaf label is {maxl} while total estimated leaves are {n_leaves}"
- )
- return None
-
- def get_clusters(self) -> np.ndarray:
- """Make cluster label array from `get_branchpoints` output."""
- self._valid_names_in_branchpoints()
- c = np.zeros(self.nCells).astype(int)
- for n, i in enumerate(self.branchpoints, start=1):
- c[self.branchpoints[i]] = n
- if (c == 0).sum() > 0:
- logger.warning(f"{(c == 0).sum()} samples were not assigned a cluster")
- c[c == 0] = -1
- return c
-
- # def test(self):
- # n = 30
- # np.random.seed(154)
- # randz = ward(gamma.rvs(0.3, size=n * 4).reshape((n, 4)))
- #
- # graph = z_to_g(randz)
- # branchpoints = get_branchpoints(graph, max_leaf=10, min_leaves=2, fc=1.5)
- # clusters = bps_to_clusts(branchpoints)
- # dend = dendrogram(randz)
- # clusters[dend['leaves']]
- #
- # res = [9, 5, 5, 1, 1, 1, 1, 7, 7, 7, 3, 3, 3, 3, 3, 4, 4, 4, 2, 2, 2, 8,
- # 8, 8, 6, 6, 6, 6, 6, 6]
diff --git a/scarf/downloader.py b/scarf/downloader.py
deleted file mode 100644
index 14147f61..00000000
--- a/scarf/downloader.py
+++ /dev/null
@@ -1,246 +0,0 @@
-"""
-Used to download datasets included in Scarf.
-
-Classes:
- OSFdownloader: A class for downloading datasets from OSF.
-
-- Methods:
- - handle_download: carry out the download of a specified dataset
- - show_available_datasets: list datasets offered through Scarf
- - fetch_dataset: Downloads datasets from online repositories and saves them in as-is format
-"""
-
-import io
-import os
-import tarfile
-import time
-from json import JSONDecodeError
-
-import pandas as pd
-
-from .utils import logger, tqdmbar
-
-__all__ = ["show_available_datasets", "fetch_dataset"]
-
-
-class OSFdownloader:
- """
- A class for downloading datasets from OSF.
-
- Attributes:
- projectId:
- storages:
- url:
- datasets:
- sourceFile:
- sources:
-
- Methods:
- get_json:
- get_all_pages:
- show_datasets:
- get_dataset_file_ids:
- """
-
- def __init__(self, project_id):
- self.projectId = project_id
- self.storages = ["osfstorage", "figshare"]
- self.url = f"https://api.osf.io/v2/nodes/{self.projectId}/files/"
- self.datasets, self.sourceFile = self._populate_datasets()
- self.sources = self._populate_sources()
-
- def get_json(self, storage, endpoint, url):
- import requests
-
- if endpoint != "":
- endpoint = endpoint + "/"
- if url is None:
- url = self.url + f"{storage}/{endpoint}"
- return requests.get(url).json()
-
- def get_all_pages(self, storage, endpoint=""):
- data = []
- url = None
- n_attempts = 0
- while n_attempts < 5:
- try:
- r = self.get_json(storage, endpoint, url)
- except JSONDecodeError:
- time.sleep(1)
- n_attempts += 1
- continue
- if "data" in r:
- data.extend(r["data"])
- url = r["links"]["next"]
- if url is None:
- break
- else:
- n_attempts += 1
- time.sleep(1)
- return data
-
- @staticmethod
- def _process_path(node):
- return node["attributes"]["path"].rstrip("/").lstrip("/")
-
- def _populate_datasets(self):
- datasets = {}
- source_fn = ""
- for storage in self.storages:
- for i in self.get_all_pages(storage):
- path = self._process_path(i)
- if i["attributes"]["name"] == "sources":
- source_fn = path
- continue
- datasets[i["attributes"]["name"]] = (path, storage)
- return datasets, source_fn
-
- def _populate_sources(self):
- import requests
-
- n_attempts = 0
- while n_attempts < 5:
- try:
- source_fn = self._get_files_for_node("osfstorage", self.sourceFile)[
- "sources.csv"
- ]
- return (
- pd.read_csv(io.StringIO(requests.get(source_fn).text))
- .set_index("id")
- .to_dict()
- )
- except KeyError:
- time.sleep(1)
- n_attempts += 1
-
- def show_datasets(self):
- print("\n".join(sorted(self.datasets.keys())))
-
- def _get_files_for_node(self, storage, file_id):
- base_url = f"https://files.de-1.osf.io/v1/resources/{self.projectId}/providers/"
- ret_val = {}
- for i in self.get_all_pages(storage, file_id):
- path = self._process_path(i)
- ret_val[i["attributes"]["name"]] = base_url + f"{storage}/{path}"
- return ret_val
-
- def get_dataset_file_ids(self, dataset_name):
- if dataset_name not in self.datasets:
- raise KeyError(
- f"ERROR: {dataset_name} was not found. "
- f"Please choose one of the following:\n{self.show_datasets()}"
- )
- file_id, storage = self.datasets[dataset_name]
- return self._get_files_for_node(storage, file_id)
-
-
-osfd = None
-
-
-def handle_download(url: str, out_fn: str, seq_counter: str = "") -> None:
- """
- Carry out the download of a specified dataset.
-
- Args:
- url: URL of file to be downloaded
- out_fn: Absolute path to the file where the downloaded data is to be saved
- seq_counter: show the sequence number of the download
-
- Returns: None
-
- """
-
- import requests
- from pathlib import Path
-
- chunk_size = int(1e7)
- r = requests.head(url, allow_redirects=True)
- size = int(r.headers.get("content-length", -1))
- size = size // chunk_size + 1
-
- r = requests.get(url, stream=True)
- with open(out_fn, "wb") as handle:
- for chunk in tqdmbar(
- r.iter_content(chunk_size=chunk_size),
- total=size,
- desc=f"Downloading {seq_counter}",
- ):
- if chunk:
- handle.write(chunk)
- logger.debug(f"Download finished! File saved here: {out_fn}")
- if out_fn.endswith("tar.gz"):
- tar = tarfile.open(out_fn, "r:gz")
- tar.extractall(str(Path(out_fn).parent.absolute()))
- tar.close()
-
-
-def show_available_datasets() -> None:
- """
- List datasets offered through Scarf.
-
- Prints the list of datasets.
-
- Returns:
- None
- """
- global osfd
- if osfd is None:
- osfd = OSFdownloader("zeupv")
- osfd.show_datasets()
-
-
-def fetch_dataset(
- dataset_name: str, save_path: str = ".", as_zarr: bool = False
-) -> None:
- """
- Downloads datasets from online repositories and saves them in as-is format.
-
- Args:
- dataset_name: Name of the dataset
- save_path: Save location without name of the file
- as_zarr: If True, then a Zarr format file, if available, is downloaded instead
-
- Returns:
- None
- """
-
- zarr_ext = ".zarr.tar.gz"
-
- def has_zarr(entry):
- for e in entry:
- if e.endswith(zarr_ext):
- return True
- return False
-
- def get_zarr_entry(entry):
- for e in entry:
- if e.endswith(zarr_ext):
- return e, entry[e]
- return False, False
-
- global osfd
- if osfd is None:
- osfd = OSFdownloader("zeupv")
-
- files = osfd.get_dataset_file_ids(dataset_name)
-
- if as_zarr:
- if has_zarr(files) is False:
- logger.error(
- f"Zarr file does not exist for {dataset_name}. Nothing downloaded"
- )
- return None
-
- save_dir = os.path.join(save_path, dataset_name)
- if os.path.isdir(save_dir) is False:
- os.makedirs(save_dir)
-
- if as_zarr:
- sp, url = get_zarr_entry(files)
- sp = os.path.abspath(os.path.join(save_dir, sp))
- handle_download(url, sp)
- else:
- valid_files = [x for x in files if not x.endswith(zarr_ext)]
- for n, i in enumerate(valid_files, start=1):
- sp = os.path.abspath(os.path.join(save_dir, i))
- handle_download(files[i], sp, f"{n}/{len(valid_files)}")
diff --git a/scarf/embeddings/__init__.py b/scarf/embeddings/__init__.py
new file mode 100644
index 00000000..38e48feb
--- /dev/null
+++ b/scarf/embeddings/__init__.py
@@ -0,0 +1,81 @@
+from importlib import import_module as _import_module
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from .harmony import (
+ Harmony as Harmony,
+ HarmonyResult as HarmonyResult,
+ fit_harmony as fit_harmony,
+ run_harmony as run_harmony,
+ )
+ from .initialization import initial_embedding as initial_embedding
+ from .imported import (
+ validate_imported_embedding_artifact as validate_imported_embedding_artifact,
+ write_imported_coordinates as write_imported_coordinates,
+ write_imported_embedding as write_imported_embedding,
+ )
+ from .sgtsne import (
+ export_knn_to_mtx as export_knn_to_mtx,
+ run_sgtsne as run_sgtsne,
+ )
+ from .umap import (
+ calc_dens_map_params as calc_dens_map_params,
+ fit_transform as fit_transform,
+ fuzzy_simplicial_set as fuzzy_simplicial_set,
+ simplicial_set_embedding as simplicial_set_embedding,
+ )
+
+__all__ = [
+ "Harmony",
+ "HarmonyResult",
+ "calc_dens_map_params",
+ "export_knn_to_mtx",
+ "fit_harmony",
+ "fit_transform",
+ "fuzzy_simplicial_set",
+ "initial_embedding",
+ "run_harmony",
+ "run_sgtsne",
+ "simplicial_set_embedding",
+ "validate_imported_embedding_artifact",
+ "write_imported_coordinates",
+ "write_imported_embedding",
+]
+
+_LAZY_EXPORTS = {
+ "Harmony": (".harmony", "Harmony"),
+ "HarmonyResult": (".harmony", "HarmonyResult"),
+ "calc_dens_map_params": (".umap", "calc_dens_map_params"),
+ "export_knn_to_mtx": (".sgtsne", "export_knn_to_mtx"),
+ "fit_harmony": (".harmony", "fit_harmony"),
+ "fit_transform": (".umap", "fit_transform"),
+ "fuzzy_simplicial_set": (".umap", "fuzzy_simplicial_set"),
+ "initial_embedding": (".initialization", "initial_embedding"),
+ "validate_imported_embedding_artifact": (
+ ".imported",
+ "validate_imported_embedding_artifact",
+ ),
+ "write_imported_coordinates": (".imported", "write_imported_coordinates"),
+ "write_imported_embedding": (".imported", "write_imported_embedding"),
+ "run_harmony": (".harmony", "run_harmony"),
+ "run_sgtsne": (".sgtsne", "run_sgtsne"),
+ "simplicial_set_embedding": (".umap", "simplicial_set_embedding"),
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+
+def __getattr__(name: str) -> Any:
+ export = _LAZY_EXPORTS.get(name)
+ if export is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ module_name, attribute_name = export
+ value = getattr(_import_module(module_name, __name__), attribute_name)
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()).union(_LAZY_EXPORTS))
diff --git a/scarf/embeddings/harmony/__init__.py b/scarf/embeddings/harmony/__init__.py
new file mode 100644
index 00000000..f671a557
--- /dev/null
+++ b/scarf/embeddings/harmony/__init__.py
@@ -0,0 +1,34 @@
+"""Harmony correction for reduced cell embeddings."""
+
+from .api import fit_harmony, run_harmony
+from .models import ClusterFn, HarmonyResult
+from .optimizer import Harmony, moe_correct_ridge, safe_entropy
+
+__all__ = [
+ "ClusterFn",
+ "Harmony",
+ "HarmonyResult",
+ "fit_harmony",
+ "moe_correct_ridge",
+ "run_harmony",
+ "safe_entropy",
+]
+
+for _public_object in (
+ Harmony,
+ HarmonyResult,
+ fit_harmony,
+ moe_correct_ridge,
+ run_harmony,
+ safe_entropy,
+):
+ _public_object.__module__ = __name__
+
+for _method in Harmony.__dict__.values():
+ if isinstance(_method, staticmethod):
+ _method = _method.__func__
+ if callable(_method) and hasattr(_method, "__module__"):
+ _method.__module__ = __name__
+
+del _method
+del _public_object
diff --git a/scarf/embeddings/harmony/api.py b/scarf/embeddings/harmony/api.py
new file mode 100644
index 00000000..27f355c1
--- /dev/null
+++ b/scarf/embeddings/harmony/api.py
@@ -0,0 +1,192 @@
+import numpy as np
+import pandas as pd
+
+from .models import ClusterFn, HarmonyResult
+
+
+def run_harmony(
+ data_mat: np.ndarray,
+ meta_data: pd.DataFrame,
+ theta: float | int | np.ndarray | list[float] | None = None,
+ lamb: float | int | np.ndarray | list[float] | None = None,
+ sigma: float | np.ndarray = 0.1,
+ nclust: int | None = None,
+ tau: float = 0,
+ block_size: float = 0.05,
+ max_iter_harmony: int = 50,
+ max_iter_kmeans: int = 20,
+ epsilon_cluster: float = 1e-5,
+ epsilon_harmony: float = 1e-4,
+ random_state: int = 0,
+ cluster_fn: ClusterFn = "kmeans",
+) -> np.ndarray:
+ """Run Harmony batch correction on a PCA embedding."""
+ from .. import fit_harmony
+
+ return fit_harmony(
+ data_mat,
+ meta_data,
+ theta=theta,
+ lamb=lamb,
+ sigma=sigma,
+ nclust=nclust,
+ tau=tau,
+ block_size=block_size,
+ max_iter_harmony=max_iter_harmony,
+ max_iter_kmeans=max_iter_kmeans,
+ epsilon_cluster=epsilon_cluster,
+ epsilon_harmony=epsilon_harmony,
+ random_state=random_state,
+ cluster_fn=cluster_fn,
+ ).corrected
+
+
+def fit_harmony(
+ data_mat: np.ndarray,
+ meta_data: pd.DataFrame,
+ theta: float | int | np.ndarray | list[float] | None = None,
+ lamb: float | int | np.ndarray | list[float] | None = None,
+ sigma: float | np.ndarray = 0.1,
+ nclust: int | None = None,
+ tau: float = 0,
+ block_size: float = 0.05,
+ max_iter_harmony: int = 50,
+ max_iter_kmeans: int = 20,
+ epsilon_cluster: float = 1e-5,
+ epsilon_harmony: float = 1e-4,
+ random_state: int = 0,
+ cluster_fn: ClusterFn = "kmeans",
+) -> HarmonyResult:
+ """Fit Harmony and return corrected coordinates with portable state."""
+ from .. import Harmony
+
+ if data_mat.ndim != 2:
+ raise ValueError("Harmony data_mat must be two-dimensional")
+ if data_mat.shape[1] != len(meta_data):
+ raise ValueError(
+ "Harmony metadata rows must match the number of embedding columns"
+ )
+ if data_mat.shape[1] < 2:
+ raise ValueError("Harmony requires at least two cells")
+ if meta_data.empty:
+ raise ValueError("Harmony requires at least one batch metadata column")
+ if meta_data.columns.duplicated().any():
+ raise ValueError("Harmony batch metadata column names must be unique")
+ if meta_data.isna().any().any():
+ raise ValueError("Harmony batch metadata cannot contain missing values")
+ if not np.all(np.isfinite(data_mat)):
+ raise ValueError("Harmony input contains non-finite values")
+
+ n_cells = data_mat.shape[1]
+ if nclust is None:
+ nclust = max(1, int(np.min([np.round(n_cells / 30.0), 100])))
+ elif nclust < 1 or nclust > n_cells:
+ raise ValueError("Harmony nclust must be between one and the cell count")
+
+ sigma_arr = np.asarray(sigma, dtype=np.float64)
+ if sigma_arr.ndim == 0:
+ sigma_arr = np.full(nclust, float(sigma_arr.item()), dtype=np.float64)
+ elif sigma_arr.shape != (nclust,):
+ raise ValueError("Harmony sigma must be scalar or have one value per cluster")
+ if not np.all(np.isfinite(sigma_arr)) or np.any(sigma_arr <= 0):
+ raise ValueError("Harmony sigma values must be finite and positive")
+
+ categorical_metadata = meta_data.astype(
+ {column: "category" for column in meta_data.columns}
+ )
+ phi_frame = pd.get_dummies(categorical_metadata)
+ phi = phi_frame.to_numpy().T
+ phi_n = np.asarray(
+ [meta_data[column].nunique() for column in meta_data.columns],
+ dtype=int,
+ )
+
+ def _expand_parameter(
+ values: float | int | np.ndarray | list[float] | None,
+ name: str,
+ ) -> np.ndarray:
+ if values is None:
+ return np.ones(int(np.sum(phi_n)), dtype=np.float64)
+ array = np.asarray(values, dtype=np.float64)
+ if array.ndim == 0:
+ return np.full(int(np.sum(phi_n)), float(array.item()))
+ if array.shape == (len(phi_n),):
+ return np.repeat(array, phi_n)
+ if array.shape != (int(np.sum(phi_n)),):
+ raise ValueError(f"Each Harmony batch level must have a {name}")
+ return array
+
+ theta_arr = _expand_parameter(theta, "theta")
+ lamb_arr = _expand_parameter(lamb, "lambda")
+ if not np.all(np.isfinite(theta_arr)) or np.any(theta_arr < 0):
+ raise ValueError("Harmony theta values must be finite and non-negative")
+ if not np.all(np.isfinite(lamb_arr)) or np.any(lamb_arr < 0):
+ raise ValueError("Harmony lambda values must be finite and non-negative")
+
+ batch_counts = phi.sum(axis=1)
+ batch_proportions = batch_counts / n_cells
+ if tau > 0:
+ theta_arr = theta_arr * (1 - np.exp(-((batch_counts / (nclust * tau)) ** 2)))
+
+ lamb_mat = np.diag(np.insert(lamb_arr, 0, 0))
+ phi_moe = np.vstack((np.repeat(1, n_cells), phi))
+ if isinstance(cluster_fn, str) and cluster_fn != "kmeans":
+ raise ValueError("Harmony cluster_fn must be 'kmeans' or a callable")
+
+ optimizer = Harmony(
+ data_mat,
+ phi,
+ phi_moe,
+ batch_proportions,
+ sigma_arr,
+ theta_arr,
+ max_iter_harmony,
+ max_iter_kmeans,
+ epsilon_cluster,
+ epsilon_harmony,
+ nclust,
+ block_size,
+ lamb_mat,
+ random_state,
+ cluster_fn,
+ )
+
+ batch_columns = tuple(str(column) for column in meta_data.columns)
+ batch_levels = tuple(
+ tuple(str(value) for value in pd.unique(meta_data[column]))
+ for column in meta_data.columns
+ )
+ cluster_backend = (
+ "sklearn.cluster.KMeans"
+ if isinstance(cluster_fn, str)
+ else (
+ f"{getattr(cluster_fn, '__module__', type(cluster_fn).__module__)}."
+ f"{getattr(cluster_fn, '__qualname__', type(cluster_fn).__qualname__)}"
+ )
+ )
+ parameters: dict[str, object] = {
+ "nclust": int(nclust),
+ "sigma": sigma_arr.tolist(),
+ "theta": theta_arr.tolist(),
+ "lambda": lamb_arr.tolist(),
+ "tau": float(tau),
+ "blockSize": float(block_size),
+ "maxIterHarmony": int(max_iter_harmony),
+ "maxIterKmeans": int(max_iter_kmeans),
+ "epsilonCluster": float(epsilon_cluster),
+ "epsilonHarmony": float(epsilon_harmony),
+ "randomState": int(random_state),
+ "clusterBackend": cluster_backend,
+ "phiColumns": [str(column) for column in phi_frame.columns],
+ }
+ return HarmonyResult(
+ original=optimizer.Z_orig,
+ corrected=optimizer.result(),
+ assignments=optimizer.R,
+ centroids=optimizer.Y,
+ sigma=sigma_arr.copy(),
+ ridge=lamb_mat,
+ batch_columns=batch_columns,
+ batch_levels=batch_levels,
+ parameters=parameters,
+ )
diff --git a/scarf/embeddings/harmony/models.py b/scarf/embeddings/harmony/models.py
new file mode 100644
index 00000000..a273b728
--- /dev/null
+++ b/scarf/embeddings/harmony/models.py
@@ -0,0 +1,22 @@
+from collections.abc import Callable
+from dataclasses import dataclass
+
+import numpy as np
+
+
+type ClusterFn = str | Callable[[np.ndarray, int], np.ndarray]
+
+
+@dataclass(frozen=True)
+class HarmonyResult:
+ """Converged Harmony state required for portable reference mapping."""
+
+ original: np.ndarray
+ corrected: np.ndarray
+ assignments: np.ndarray
+ centroids: np.ndarray
+ sigma: np.ndarray
+ ridge: np.ndarray
+ batch_columns: tuple[str, ...]
+ batch_levels: tuple[tuple[str, ...], ...]
+ parameters: dict[str, object]
diff --git a/scarf/embeddings/harmony/optimizer.py b/scarf/embeddings/harmony/optimizer.py
new file mode 100644
index 00000000..bc96be8e
--- /dev/null
+++ b/scarf/embeddings/harmony/optimizer.py
@@ -0,0 +1,263 @@
+from collections.abc import Callable
+from functools import partial
+
+import numpy as np
+from sklearn.cluster import KMeans
+
+from ...utils.logging import logger
+from ...utils.progress import tqdmbar
+from .models import ClusterFn
+
+
+class Harmony:
+ def __init__(
+ self,
+ Z: np.ndarray,
+ Phi: np.ndarray,
+ Phi_moe: np.ndarray,
+ Pr_b: np.ndarray,
+ sigma: np.ndarray,
+ theta: np.ndarray,
+ max_iter_harmony: int,
+ max_iter_kmeans: int,
+ epsilon_kmeans: float,
+ epsilon_harmony: float,
+ K: int,
+ block_size: float,
+ lamb: np.ndarray,
+ random_state: int | None = None,
+ cluster_fn: ClusterFn = "kmeans",
+ ) -> None:
+ self.Z_orig = np.array(Z, dtype=np.float64, copy=True)
+ self.Z_corr = np.array(self.Z_orig, copy=True)
+ self.Z_cos = _normalize_columns(self.Z_orig)
+ self._rng = np.random.RandomState(random_state)
+
+ self.Phi = Phi
+ self.Phi_moe = Phi_moe
+ self.N = self.Z_corr.shape[1]
+ self.Pr_b = Pr_b
+ self.B = self.Phi.shape[0]
+ self.d = self.Z_corr.shape[0]
+ self.window_size = 3
+ self.epsilon_kmeans = epsilon_kmeans
+ self.epsilon_harmony = epsilon_harmony
+
+ self.lamb = lamb
+ self.sigma = sigma
+ self.sigma_prior = sigma
+ self.block_size = block_size
+ self.K = K
+ self.max_iter_harmony = max_iter_harmony
+ self.max_iter_kmeans = max_iter_kmeans
+ self.theta = theta
+
+ self.objective_harmony: list[float] = []
+ self.objective_kmeans: list[float] = []
+ self.objective_kmeans_dist: list[float] = []
+ self.objective_kmeans_entropy: list[float] = []
+ self.objective_kmeans_cross: list[float] = []
+ self.kmeans_rounds: list[int] = []
+
+ self.allocate_buffers()
+ resolved_cluster_fn: Callable[[np.ndarray, int], np.ndarray]
+ if isinstance(cluster_fn, str):
+ resolved_cluster_fn = partial(
+ Harmony._cluster_kmeans,
+ random_state=random_state,
+ )
+ else:
+ resolved_cluster_fn = cluster_fn
+ self.init_cluster(resolved_cluster_fn)
+ self.harmonize(self.max_iter_harmony)
+
+ def result(self) -> np.ndarray:
+ return self.Z_corr
+
+ def allocate_buffers(self) -> None:
+ self._scale_dist = np.zeros((self.K, self.N))
+ self.dist_mat = np.zeros((self.K, self.N))
+ self.O = np.zeros((self.K, self.B))
+ self.E = np.zeros((self.K, self.B))
+ self.W = np.zeros((self.B + 1, self.d))
+ self.Phi_Rk = np.zeros((self.B + 1, self.N))
+
+ @staticmethod
+ def _cluster_kmeans(
+ data: np.ndarray,
+ K: int,
+ random_state: int | None,
+ ) -> np.ndarray:
+ centers = (
+ KMeans(
+ n_clusters=K,
+ init="k-means++",
+ n_init=10,
+ max_iter=25,
+ random_state=random_state,
+ )
+ .fit(data)
+ .cluster_centers_
+ )
+ return np.asarray(centers)
+
+ def init_cluster(self, cluster_fn: Callable[[np.ndarray, int], np.ndarray]) -> None:
+ self.Y = cluster_fn(self.Z_cos.T, self.K).T
+ self.Y = _normalize_columns(self.Y)
+ self.dist_mat = 2 * (1 - np.dot(self.Y.T, self.Z_cos))
+ self.R = -self.dist_mat
+ self.R = self.R / self.sigma[:, None]
+ self.R -= np.max(self.R, axis=0)
+ self.R = np.exp(self.R)
+ self.R = self.R / np.sum(self.R, axis=0)
+ self.E = np.outer(np.sum(self.R, axis=1), self.Pr_b)
+ self.O = np.inner(self.R, self.Phi)
+ self.compute_objective()
+ self.objective_harmony.append(self.objective_kmeans[-1])
+
+ def compute_objective(self) -> None:
+ kmeans_error = np.sum(np.multiply(self.R, self.dist_mat))
+ entropy = np.sum(safe_entropy(self.R) * self.sigma[:, np.newaxis])
+ x = self.R * self.sigma[:, np.newaxis]
+ y = np.tile(self.theta[:, np.newaxis], self.K).T
+ z = np.log((self.O + 1) / (self.E + 1))
+ w = np.dot(y * z, self.Phi)
+ cross_entropy = np.sum(x * w)
+ self.objective_kmeans.append(kmeans_error + entropy + cross_entropy)
+ self.objective_kmeans_dist.append(kmeans_error)
+ self.objective_kmeans_entropy.append(entropy)
+ self.objective_kmeans_cross.append(cross_entropy)
+
+ def harmonize(self, iter_harmony: int = 10) -> int:
+ converged = False
+ progress = tqdmbar(
+ desc="Harmonizing batches",
+ total=iter_harmony,
+ )
+ try:
+ for _ in range(1, iter_harmony + 1):
+ self.cluster()
+ self.Z_cos, self.Z_corr, self.W, self.Phi_Rk = moe_correct_ridge(
+ self.Z_orig,
+ self.R,
+ self.W,
+ self.K,
+ self.Phi_Rk,
+ self.Phi_moe,
+ self.lamb,
+ )
+ progress.update()
+ converged = self.check_convergence(1)
+ if converged:
+ progress.total = progress.n
+ progress.refresh()
+ break
+ finally:
+ progress.close()
+ self.Y = _normalize_columns(np.dot(self.Z_cos, self.R.T))
+ self.dist_mat = 2 * (1 - np.dot(self.Y.T, self.Z_cos))
+ if not converged:
+ logger.warning("Harmony stopped before convergence")
+ return 0
+
+ def cluster(self) -> int:
+ self.dist_mat = 2 * (1 - np.dot(self.Y.T, self.Z_cos))
+ for iteration in range(self.max_iter_kmeans):
+ self.Y = np.dot(self.Z_cos, self.R.T)
+ self.Y = self.Y / np.linalg.norm(self.Y, ord=2, axis=0)
+ self.dist_mat = 2 * (1 - np.dot(self.Y.T, self.Z_cos))
+ self.update_R()
+ self.compute_objective()
+ if iteration > self.window_size and self.check_convergence(0):
+ break
+ self.kmeans_rounds.append(iteration)
+ self.objective_harmony.append(self.objective_kmeans[-1])
+ return 0
+
+ def update_R(self) -> int:
+ self._scale_dist = -self.dist_mat
+ self._scale_dist = self._scale_dist / self.sigma[:, None]
+ self._scale_dist -= np.max(self._scale_dist, axis=0)
+ self._scale_dist = np.exp(self._scale_dist)
+ update_order = self._rng.permutation(self.N)
+ n_blocks = np.ceil(1 / self.block_size).astype(int)
+ blocks = np.array_split(update_order, n_blocks)
+ for block in blocks:
+ self.E -= np.outer(np.sum(self.R[:, block], axis=1), self.Pr_b)
+ self.O -= np.dot(self.R[:, block], self.Phi[:, block].T)
+ self.R[:, block] = self._scale_dist[:, block]
+ self.R[:, block] = np.multiply(
+ self.R[:, block],
+ np.dot(
+ np.power((self.E + 1) / (self.O + 1), self.theta),
+ self.Phi[:, block],
+ ),
+ )
+ self.R[:, block] = self.R[:, block] / np.linalg.norm(
+ self.R[:, block],
+ ord=1,
+ axis=0,
+ )
+ self.E += np.outer(np.sum(self.R[:, block], axis=1), self.Pr_b)
+ self.O += np.dot(self.R[:, block], self.Phi[:, block].T)
+ return 0
+
+ def check_convergence(self, i_type: int) -> bool:
+ obj_old = 0.0
+ obj_new = 0.0
+ if i_type == 0:
+ objective_count = len(self.objective_kmeans)
+ for offset in range(self.window_size):
+ obj_old += self.objective_kmeans[objective_count - 2 - offset]
+ obj_new += self.objective_kmeans[objective_count - 1 - offset]
+ if abs(obj_old - obj_new) / abs(obj_old) < self.epsilon_kmeans:
+ return True
+ return False
+ if i_type == 1:
+ obj_old = self.objective_harmony[-2]
+ obj_new = self.objective_harmony[-1]
+ error = (obj_old - obj_new) / abs(obj_old)
+ logger.debug(
+ f"Harmony error after {len(self.objective_harmony)} iterations: {error}"
+ )
+ if error < self.epsilon_harmony:
+ return True
+ return False
+ return True
+
+
+def safe_entropy(x: np.ndarray) -> np.ndarray:
+ y = np.multiply(x, np.log(x))
+ y[~np.isfinite(y)] = 0.0
+ return np.asarray(y)
+
+
+def moe_correct_ridge(
+ Z_orig: np.ndarray,
+ R: np.ndarray,
+ W: np.ndarray,
+ K: int,
+ Phi_Rk: np.ndarray,
+ Phi_moe: np.ndarray,
+ lamb: np.ndarray,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+ Z_corr = Z_orig.copy()
+ for cluster_index in range(K):
+ Phi_Rk = np.multiply(Phi_moe, R[cluster_index, :])
+ x = np.dot(Phi_Rk, Phi_moe.T) + lamb
+ try:
+ W = np.linalg.solve(x, np.dot(Phi_Rk, Z_orig.T))
+ except np.linalg.LinAlgError as exc:
+ raise ValueError("Harmony ridge system is singular") from exc
+ W[0, :] = 0
+ Z_corr -= np.dot(W.T, Phi_Rk)
+ Z_cos = _normalize_columns(Z_corr)
+ return Z_cos, Z_corr, W, Phi_Rk
+
+
+def _normalize_columns(values: np.ndarray) -> np.ndarray:
+ norms = np.linalg.norm(values, ord=2, axis=0)
+ normalized = np.zeros_like(values, dtype=np.float64)
+ nonzero = norms > 0
+ normalized[:, nonzero] = values[:, nonzero] / norms[nonzero]
+ return normalized
diff --git a/scarf/embeddings/imported.py b/scarf/embeddings/imported.py
new file mode 100644
index 00000000..468fe9cd
--- /dev/null
+++ b/scarf/embeddings/imported.py
@@ -0,0 +1,983 @@
+from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
+from dataclasses import dataclass, replace
+from typing import Any, cast
+
+import numpy as np
+import zarr
+
+from ..graph.state import (
+ ArtifactRef,
+ ArtifactSelectionError,
+ AssayState,
+ ImportedArtifactStorage,
+ fingerprint_selected_stored_strings,
+ read_assay_state,
+ validate_cell_selection_artifact,
+ validate_imported_coordinates_artifact,
+ write_assay_state,
+)
+
+type ImportedBlockSource = (
+ np.ndarray | zarr.Array | Iterable[np.ndarray] | Callable[[], Iterable[np.ndarray]]
+)
+type ImportedStringSource = Sequence[str | bytes] | np.ndarray | zarr.Array
+type ImportedFeatureIdSource = (
+ ImportedStringSource | Iterable[np.ndarray] | Callable[[], Iterable[np.ndarray]]
+)
+
+
+@dataclass(slots=True)
+class _ArraySource:
+ shape: tuple[int, ...]
+ dtype: np.dtype[Any]
+ produce: Callable[[], Iterator[np.ndarray]]
+ reusable: bool
+
+
+def _positive_block_rows(value: int) -> int:
+ if isinstance(value, bool) or not isinstance(value, int | np.integer):
+ raise TypeError("block_rows must be a positive integer")
+ if int(value) < 1:
+ raise ValueError("block_rows must be greater than zero")
+ return int(value)
+
+
+def _resolve_source(
+ values: ImportedBlockSource,
+ *,
+ name: str,
+ shape: tuple[int, ...] | None,
+ dtype: Any | None,
+ block_rows: int,
+) -> _ArraySource:
+ if isinstance(values, np.ndarray | zarr.Array):
+ resolved_shape = tuple(int(size) for size in values.shape)
+ resolved_dtype = np.dtype(values.dtype)
+
+ def produce() -> Iterator[np.ndarray]:
+ for start in range(0, resolved_shape[0], block_rows):
+ stop = min(start + block_rows, resolved_shape[0])
+ yield np.asarray(values[start:stop])
+
+ return _ArraySource(resolved_shape, resolved_dtype, produce, True)
+
+ if shape is None or dtype is None:
+ raise ValueError(
+ f"{name}_shape and {name}_dtype are required for streamed blocks"
+ )
+ resolved_shape = tuple(int(size) for size in shape)
+ resolved_dtype = np.dtype(dtype)
+ if callable(values):
+
+ def produce() -> Iterator[np.ndarray]:
+ return iter(values())
+
+ reusable = True
+ else:
+ used = False
+
+ def produce() -> Iterator[np.ndarray]:
+ nonlocal used
+ if used:
+ raise RuntimeError(f"{name} block source can only be consumed once")
+ used = True
+ return iter(values)
+
+ reusable = False
+
+ return _ArraySource(resolved_shape, resolved_dtype, produce, reusable)
+
+
+def _validate_numeric_source(source: _ArraySource, name: str, ndim: int) -> None:
+ if len(source.shape) != ndim or any(size < 1 for size in source.shape):
+ raise ValueError(f"{name} must have {ndim} non-empty dimensions")
+ if source.dtype.kind != "f":
+ raise TypeError(f"{name} must use a floating-point dtype")
+
+
+def _validate_fingerprint(value: str, name: str) -> str:
+ if not isinstance(value, str) or len(value) != 64 or value.lower() != value:
+ raise ValueError(f"{name} must be a 64-character lowercase hex fingerprint")
+ try:
+ bytes.fromhex(value)
+ except ValueError as exc:
+ raise ValueError(f"{name} must be a hexadecimal fingerprint") from exc
+ return value
+
+
+def _required_payload_fingerprints(
+ values: Mapping[str, str],
+ required: set[str],
+) -> dict[str, str]:
+ if not isinstance(values, Mapping):
+ raise TypeError("payload_fingerprints must be a mapping")
+ fingerprints = {
+ str(name): _validate_fingerprint(fingerprint, f"{name} fingerprint")
+ for name, fingerprint in values.items()
+ }
+ missing = required - set(fingerprints)
+ if missing:
+ raise ValueError("Missing payload fingerprints: " + ", ".join(sorted(missing)))
+ unexpected = set(fingerprints) - required
+ if unexpected:
+ raise ValueError(
+ "Unexpected payload fingerprints: " + ", ".join(sorted(unexpected))
+ )
+ return fingerprints
+
+
+def _validate_source_digest(value: bytes) -> bytes:
+ if not isinstance(value, bytes) or len(value) != 32:
+ raise TypeError("source_digest must contain exactly 32 bytes")
+ return value
+
+
+def _string_source_length(values: ImportedStringSource) -> int:
+ shape = getattr(values, "shape", None)
+ if shape is not None:
+ if len(shape) != 1:
+ raise ValueError("source_cell_ids must be one-dimensional")
+ return int(shape[0])
+ return len(cast(Sequence[str | bytes], values))
+
+
+def _string_block(
+ values: ImportedStringSource,
+ start: int,
+ stop: int,
+ *,
+ name: str = "source_cell_ids",
+) -> np.ndarray:
+ raw = values[start:stop]
+ array = np.asarray(raw)
+ if array.ndim != 1 or len(array) != stop - start:
+ raise ValueError(f"{name} do not support bounded one-dimensional reads")
+ normalized: list[str] = []
+ for value in array:
+ if isinstance(value, bytes | np.bytes_):
+ try:
+ text = bytes(value).decode("utf-8")
+ except UnicodeDecodeError as exc:
+ raise ValueError("source_cell_ids contain invalid UTF-8") from exc
+ elif isinstance(value, str | np.str_):
+ text = str(value)
+ else:
+ raise TypeError(f"{name} must contain strings")
+ try:
+ text.encode("utf-8")
+ except UnicodeEncodeError as exc:
+ raise ValueError(f"{name} contain invalid Unicode") from exc
+ if not text or "\x00" in text:
+ raise ValueError(f"{name} contain an invalid identifier")
+ normalized.append(text)
+ return np.asarray(normalized)
+
+
+def _resolve_feature_id_source(
+ values: ImportedFeatureIdSource,
+ *,
+ shape: tuple[int] | None,
+ dtype: Any | None,
+ block_rows: int,
+) -> _ArraySource:
+ if (
+ not callable(values)
+ and not isinstance(values, np.ndarray | zarr.Array)
+ and shape is None
+ and dtype is None
+ and hasattr(values, "__len__")
+ and hasattr(values, "__getitem__")
+ ):
+ strings = cast(ImportedStringSource, values)
+ resolved_shape = (_string_source_length(strings),)
+
+ def produce() -> Iterator[np.ndarray]:
+ for start in range(0, resolved_shape[0], block_rows):
+ stop = min(start + block_rows, resolved_shape[0])
+ yield _string_block(
+ strings,
+ start,
+ stop,
+ name="feature_ids",
+ )
+
+ return _ArraySource(resolved_shape, np.dtype(object), produce, True)
+ return _resolve_source(
+ cast(ImportedBlockSource, values),
+ name="feature_ids",
+ shape=shape,
+ dtype=dtype,
+ block_rows=block_rows,
+ )
+
+
+def _selection_alignment(
+ root: zarr.Group,
+ *,
+ cell_selection: ArtifactRef,
+ cell_key: str,
+ source_cell_ids: ImportedStringSource,
+ coordinate_rows: int,
+ block_rows: int,
+) -> tuple[zarr.Array, int, str]:
+ storage = ImportedArtifactStorage(root)
+ validate_cell_selection_artifact(root, cell_selection, cell_key)
+ selection_group = storage.artifact_group(cell_selection)
+ selection = storage.as_array(selection_group["values"], "values")
+ cell_data = storage.as_group(root["cellData"], "cellData")
+ stored_ids = storage.as_array(cell_data["ids"], "ids")
+ source_count = _string_source_length(source_cell_ids)
+ context: dict[str, str | int | float | bool | None] = {
+ "cell_key": cell_key,
+ "coordinate_rows": int(coordinate_rows),
+ "source_cell_count": source_count,
+ "artifact_id": cell_selection.artifact_id,
+ }
+ if source_count != coordinate_rows:
+ raise ArtifactSelectionError(
+ "Imported coordinates do not match the source cell count",
+ code="dimreduc_row_count_mismatch",
+ context=context,
+ )
+ source_start = 0
+ selected_count = 0
+ for start in range(0, int(selection.shape[0]), block_rows):
+ stop = min(start + block_rows, int(selection.shape[0]))
+ mask = np.asarray(selection[start:stop], dtype=bool)
+ selected = int(np.count_nonzero(mask))
+ source_stop = source_start + selected
+ if source_stop > source_count:
+ context["selected_count"] = source_stop
+ raise ArtifactSelectionError(
+ "Imported coordinates do not match the selected cell count",
+ code="dimreduc_row_count_mismatch",
+ context=context,
+ )
+ expected = _string_block(source_cell_ids, source_start, source_stop)
+ actual = _string_block(
+ np.asarray(stored_ids[start:stop])[mask],
+ 0,
+ selected,
+ )
+ if not np.array_equal(expected, actual):
+ context["selected_count"] = selected_count + selected
+ raise ArtifactSelectionError(
+ "Imported cell IDs do not match the selected cell order",
+ code="dimreduc_cell_identity_mismatch",
+ context=context,
+ )
+ source_start = source_stop
+ selected_count += selected
+ context["selected_count"] = selected_count
+ if selected_count != coordinate_rows or source_start != source_count:
+ raise ArtifactSelectionError(
+ "Imported coordinates do not match the selected cell count",
+ code="dimreduc_row_count_mismatch",
+ context=context,
+ )
+ fingerprint, fingerprint_count = fingerprint_selected_stored_strings(
+ stored_ids,
+ selection,
+ )
+ if fingerprint_count != selected_count:
+ raise RuntimeError("Selected cell count changed during alignment")
+ return selection, selected_count, fingerprint
+
+
+def _create_numeric_destination(
+ storage: ImportedArtifactStorage,
+ group: zarr.Group,
+ name: str,
+ source: _ArraySource,
+ block_rows: int,
+) -> zarr.Array:
+ return storage.create_numeric(
+ group,
+ name,
+ shape=source.shape,
+ dtype=source.dtype,
+ block_rows=block_rows,
+ )
+
+
+def _write_numeric_payload(
+ storage: ImportedArtifactStorage,
+ group: zarr.Group,
+ name: str,
+ source: _ArraySource,
+ expected_fingerprint: str,
+ block_rows: int,
+) -> zarr.Array:
+ destination = _create_numeric_destination(
+ storage,
+ group,
+ name,
+ source,
+ block_rows,
+ )
+ builder = storage.fingerprint_builder()
+ builder.begin_array("values", source.shape, source.dtype)
+ start = 0
+ blocks = source.produce()
+ try:
+ for raw_block in blocks:
+ block = np.asarray(raw_block)
+ if (
+ block.ndim != len(source.shape)
+ or block.shape[1:] != source.shape[1:]
+ or np.dtype(block.dtype) != source.dtype
+ ):
+ raise ValueError(f"{name} block has an invalid shape or dtype")
+ if block.shape[0] == 0:
+ continue
+ if not np.all(np.isfinite(block)):
+ raise ValueError(f"{name} contains non-finite values")
+ stop = start + int(block.shape[0])
+ if stop > source.shape[0]:
+ raise ValueError(f"{name} stream exceeds its declared row count")
+ offset = (start,) + (0,) * (block.ndim - 1)
+ builder.update_array_block("values", offset, block)
+ index = (slice(start, stop),) + (slice(None),) * (block.ndim - 1)
+ destination[index] = block
+ start = stop
+ finally:
+ close = getattr(blocks, "close", None)
+ if callable(close):
+ close()
+ if start != source.shape[0]:
+ raise ValueError(
+ f"{name} stream contains {start} rows, expected {source.shape[0]}"
+ )
+ builder.end_array("values")
+ if builder.hexdigest() != expected_fingerprint:
+ raise ValueError(f"{name} payload fingerprint does not match its source")
+ return destination
+
+
+def _write_feature_ids(
+ storage: ImportedArtifactStorage,
+ group: zarr.Group,
+ source: _ArraySource,
+ expected_fingerprint: str,
+ block_rows: int,
+) -> zarr.Array:
+ if len(source.shape) != 1:
+ raise ValueError("feature_ids must be one-dimensional")
+ if source.dtype.hasobject:
+ if not source.reusable:
+ raise ValueError(
+ "Object-typed feature_ids blocks require a callable source "
+ "that can be read twice"
+ )
+ maximum = 1
+ for raw_block in source.produce():
+ raw = np.asarray(raw_block)
+ block = _string_block(raw, 0, len(raw), name="feature_ids")
+ if block.size:
+ maximum = max(maximum, max(len(value) for value in block))
+ string_dtype = np.dtype(f"U{maximum}")
+ else:
+ string_dtype = np.empty(0, dtype=source.dtype).astype(str).dtype
+ destination = storage.create_metadata(
+ group,
+ "feature_ids",
+ dtype=string_dtype,
+ shape=source.shape[0],
+ block_rows=block_rows,
+ )
+ builder = storage.fingerprint_builder()
+ builder.begin_array("values", source.shape, string_dtype)
+ start = 0
+ for raw_block in source.produce():
+ raw = np.asarray(raw_block)
+ block = _string_block(raw, 0, len(raw), name="feature_ids").astype(string_dtype)
+ stop = start + int(block.shape[0])
+ if block.ndim != 1 or stop > source.shape[0]:
+ raise ValueError("feature_ids block has an invalid shape")
+ builder.update_array_block("values", (start,), block)
+ destination[start:stop] = block
+ start = stop
+ if start != source.shape[0]:
+ raise ValueError(
+ f"feature_ids stream contains {start} rows, expected {source.shape[0]}"
+ )
+ builder.end_array("values")
+ if builder.hexdigest() != expected_fingerprint:
+ raise ValueError("feature_ids payload fingerprint does not match its source")
+ return destination
+
+
+def _stored_numeric_fingerprint(
+ storage: ImportedArtifactStorage,
+ array: zarr.Array,
+) -> str:
+ builder = storage.fingerprint_builder()
+ builder.begin_array("values", array.shape, array.dtype)
+ block_rows = storage.block_rows(array)
+ for start in range(0, int(array.shape[0]), block_rows):
+ stop = min(start + block_rows, int(array.shape[0]))
+ block = np.asarray(array[start:stop])
+ builder.update_array_block(
+ "values",
+ (start,) + (0,) * (array.ndim - 1),
+ block,
+ )
+ builder.end_array("values")
+ return builder.hexdigest()
+
+
+def _stored_payload_fingerprint(
+ storage: ImportedArtifactStorage,
+ group: zarr.Group,
+ name: str,
+) -> str:
+ array = storage.as_array(group[name], name)
+ if name == "feature_ids":
+ return storage.fingerprint_stored_strings(array)
+ return _stored_numeric_fingerprint(storage, array)
+
+
+def _payloads_match(
+ storage: ImportedArtifactStorage,
+ group: zarr.Group,
+ *,
+ shapes: Mapping[str, tuple[int, ...]],
+ fingerprints: Mapping[str, str],
+) -> bool:
+ try:
+ for name, shape in shapes.items():
+ if name not in group:
+ return False
+ array = storage.as_array(group[name], name)
+ if tuple(array.shape) != tuple(shape):
+ return False
+ if _stored_payload_fingerprint(storage, group, name) != fingerprints[name]:
+ return False
+ except (KeyError, TypeError, ValueError):
+ return False
+ return True
+
+
+def _register_named_result(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ name: str,
+ *,
+ cell_key: str,
+ feat_key: str | None,
+) -> None:
+ if not isinstance(name, str) or not name:
+ raise ValueError("named_result must be a non-empty string")
+ if ref.assay is None:
+ raise ValueError("Named imported results must be assay-scoped")
+ state = read_assay_state(root, ref.assay)
+ if state is None:
+ if feat_key is None:
+ raise ValueError(
+ "feat_key is required to register a named result without AssayState"
+ )
+ state = AssayState(
+ assay=ref.assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ )
+ named_results = dict(state.named_results)
+ named_results[name] = ref
+ write_assay_state(
+ root,
+ replace(state, named_results=named_results),
+ )
+
+
+def write_imported_coordinates(
+ root: zarr.Group,
+ *,
+ assay: str,
+ dimreduc_key: str,
+ role: str,
+ coordinates: ImportedBlockSource,
+ source_digest: bytes,
+ payload_fingerprints: Mapping[str, str],
+ source_cell_ids: ImportedStringSource,
+ cell_selection: ArtifactRef,
+ cell_key: str,
+ coordinate_shape: tuple[int, int] | None = None,
+ coordinate_dtype: Any | None = None,
+ loadings: ImportedBlockSource | None = None,
+ loadings_shape: tuple[int, int] | None = None,
+ loadings_dtype: Any | None = None,
+ feature_ids: ImportedFeatureIdSource | None = None,
+ feature_id_shape: tuple[int] | None = None,
+ feature_id_dtype: Any | None = None,
+ stdev: ImportedBlockSource | None = None,
+ stdev_shape: tuple[int] | None = None,
+ stdev_dtype: Any | None = None,
+ named_result: str | None = None,
+ feat_key: str | None = None,
+ block_rows: int = 100_000,
+ invalidate_cache: bool = False,
+) -> ArtifactRef:
+ storage = ImportedArtifactStorage(root)
+ source_digest = _validate_source_digest(source_digest)
+ if not isinstance(dimreduc_key, str) or not dimreduc_key:
+ raise ValueError("dimreduc_key must be a non-empty string")
+ if not isinstance(role, str) or not role or role.lower() in {"umap", "tsne"}:
+ raise ValueError("Imported graph coordinates require a non-layout role")
+ block_rows = _positive_block_rows(block_rows)
+ data_source = _resolve_source(
+ coordinates,
+ name="coordinate",
+ shape=coordinate_shape,
+ dtype=coordinate_dtype,
+ block_rows=block_rows,
+ )
+ _validate_numeric_source(data_source, "coordinates", 2)
+ dims = int(data_source.shape[1])
+ _selection, selected_count, ordered_cell_ids_fingerprint = _selection_alignment(
+ root,
+ cell_selection=cell_selection,
+ cell_key=cell_key,
+ source_cell_ids=source_cell_ids,
+ coordinate_rows=data_source.shape[0],
+ block_rows=block_rows,
+ )
+
+ if (loadings is None) != (feature_ids is None):
+ raise ValueError("loadings and feature_ids must be provided together")
+ loading_source = (
+ _resolve_source(
+ loadings,
+ name="loadings",
+ shape=loadings_shape,
+ dtype=loadings_dtype,
+ block_rows=block_rows,
+ )
+ if loadings is not None
+ else None
+ )
+ if loading_source is not None:
+ _validate_numeric_source(loading_source, "loadings", 2)
+ if loading_source.shape[1] != dims:
+ raise ValueError("loadings dimensions must match coordinates")
+ assert feature_ids is not None
+ feature_source = _resolve_feature_id_source(
+ feature_ids,
+ shape=feature_id_shape,
+ dtype=feature_id_dtype,
+ block_rows=block_rows,
+ )
+ if feature_source.shape != (loading_source.shape[0],):
+ raise ValueError("feature_ids must align with loadings rows")
+ else:
+ feature_source = None
+ stdev_source = (
+ _resolve_source(
+ stdev,
+ name="stdev",
+ shape=stdev_shape,
+ dtype=stdev_dtype,
+ block_rows=block_rows,
+ )
+ if stdev is not None
+ else None
+ )
+ if stdev_source is not None:
+ _validate_numeric_source(stdev_source, "stdev", 1)
+ if stdev_source.shape != (dims,):
+ raise ValueError("stdev length must match coordinate dimensions")
+
+ required_payloads = {"data"}
+ if loading_source is not None:
+ required_payloads.update({"loadings", "feature_ids"})
+ if stdev_source is not None:
+ required_payloads.add("stdev")
+ fingerprints = _required_payload_fingerprints(
+ payload_fingerprints,
+ required_payloads,
+ )
+ sources = {"data": data_source}
+ if loading_source is not None:
+ sources["loadings"] = loading_source
+ assert feature_source is not None
+ sources["feature_ids"] = feature_source
+ if stdev_source is not None:
+ sources["stdev"] = stdev_source
+ shapes = {name: source.shape for name, source in sources.items()}
+ requirements = tuple(
+ storage.requirement(
+ name,
+ shape=shape,
+ dtype_kind=None if name == "feature_ids" else "f",
+ )
+ for name, shape in shapes.items()
+ )
+ planned = storage.plan(
+ assay=assay,
+ kind="imported_coordinates",
+ parameters={
+ "dimreduc_key": dimreduc_key,
+ "role": role.lower(),
+ "dims": dims,
+ "loadings_stored": loading_source is not None,
+ "feature_ids_stored": feature_source is not None,
+ "stdev_stored": stdev_source is not None,
+ },
+ inputs={
+ "source_digest": source_digest,
+ "payload_fingerprints": fingerprints,
+ "ordered_cell_ids_fingerprint": ordered_cell_ids_fingerprint,
+ "cell_selection": cell_selection,
+ },
+ execution_options={
+ "cell_key": cell_key,
+ "block_rows": block_rows,
+ },
+ invalidate_cache=invalidate_cache,
+ required_arrays=requirements,
+ reuse_validator=lambda _ref, group: _payloads_match(
+ storage,
+ group,
+ shapes=shapes,
+ fingerprints=fingerprints,
+ ),
+ )
+ if not planned.reused:
+ group = storage.start(planned)
+ _write_numeric_payload(
+ storage,
+ group,
+ "data",
+ data_source,
+ fingerprints["data"],
+ block_rows,
+ )
+ if loading_source is not None:
+ _write_numeric_payload(
+ storage,
+ group,
+ "loadings",
+ loading_source,
+ fingerprints["loadings"],
+ block_rows,
+ )
+ assert feature_source is not None
+ _write_feature_ids(
+ storage,
+ group,
+ feature_source,
+ fingerprints["feature_ids"],
+ block_rows,
+ )
+ if stdev_source is not None:
+ _write_numeric_payload(
+ storage,
+ group,
+ "stdev",
+ stdev_source,
+ fingerprints["stdev"],
+ block_rows,
+ )
+ storage.finish(group, planned)
+ validate_imported_coordinates_artifact(
+ root,
+ planned.ref,
+ cell_key=cell_key,
+ )
+ if named_result is not None:
+ _register_named_result(
+ root,
+ planned.ref,
+ named_result,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ )
+ if selected_count != data_source.shape[0]:
+ raise RuntimeError("Imported coordinate selection changed during writing")
+ return planned.ref
+
+
+def validate_imported_embedding_artifact(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ *,
+ cell_key: str | None = None,
+) -> None:
+ storage = ImportedArtifactStorage(root)
+ if ref.kind != "embedding" or ref.scope != "assay" or ref.assay is None:
+ raise ValueError("Imported embedding must be an assay-scoped embedding")
+ status = storage.require_complete(ref)
+ if status.operation != "import_dimreduc":
+ raise ValueError("Imported embedding operation must be 'import_dimreduc'")
+ execution = status.execution_options or {}
+ stored_cell_key = execution.get("cell_key")
+ if not isinstance(stored_cell_key, str) or not stored_cell_key:
+ raise ValueError("Imported embedding has no cell selection key")
+ if cell_key is not None and cell_key != stored_cell_key:
+ raise ValueError("cell_key does not match the imported embedding")
+ block_rows = execution.get("block_rows")
+ if (
+ isinstance(block_rows, bool)
+ or not isinstance(block_rows, int | np.integer)
+ or int(block_rows) < 1
+ ):
+ raise ValueError("Imported embedding block_rows is invalid")
+ inputs = status.inputs or {}
+ raw_selection = inputs.get("cell_selection")
+ if not isinstance(raw_selection, Mapping):
+ raise ValueError("Imported embedding has no cell selection input")
+ selection = ArtifactRef.from_dict(raw_selection)
+ validate_cell_selection_artifact(root, selection, stored_cell_key)
+ group = storage.artifact_group(ref)
+ if "values" not in group:
+ raise ValueError("Imported embedding has no values array")
+ values = storage.as_array(group["values"], "values")
+ parameters = status.parameters or {}
+ dimreduc_key = parameters.get("dimreduc_key")
+ if not isinstance(dimreduc_key, str) or not dimreduc_key:
+ raise ValueError("Imported embedding source key is missing")
+ dims = parameters.get("dims")
+ role = parameters.get("role")
+ if (
+ values.ndim != 2
+ or int(values.shape[0]) < 1
+ or np.dtype(values.dtype).kind != "f"
+ or isinstance(dims, bool)
+ or not isinstance(dims, int | np.integer)
+ or int(dims) < 1
+ or tuple(values.shape)[1] != int(dims)
+ or role not in {"umap", "tsne"}
+ ):
+ raise ValueError("Imported embedding payload is malformed")
+ selection_values = storage.as_array(
+ storage.artifact_group(selection)["values"],
+ "values",
+ )
+ cell_data = storage.as_group(root["cellData"], "cellData")
+ ids = storage.as_array(cell_data["ids"], "ids")
+ selected_fingerprint, selected_count = fingerprint_selected_stored_strings(
+ ids,
+ selection_values,
+ )
+ if int(values.shape[0]) != selected_count:
+ raise ValueError("Imported embedding rows do not match its cell selection")
+ if inputs.get("ordered_cell_ids_fingerprint") != selected_fingerprint:
+ raise ValueError("Imported embedding cell IDs are out of order")
+ source_digest = inputs.get("source_digest")
+ if (
+ not isinstance(source_digest, Mapping)
+ or set(source_digest) != {"bytes_hex"}
+ or not isinstance(source_digest.get("bytes_hex"), str)
+ or len(source_digest["bytes_hex"]) != 64
+ or source_digest["bytes_hex"].lower() != source_digest["bytes_hex"]
+ ):
+ raise ValueError("Imported embedding source digest is missing")
+ try:
+ bytes.fromhex(source_digest["bytes_hex"])
+ except ValueError as exc:
+ raise ValueError("Imported embedding source digest is not hexadecimal") from exc
+ fingerprints = inputs.get("payload_fingerprints")
+ if (
+ not isinstance(fingerprints, Mapping)
+ or set(fingerprints) != {"values"}
+ or fingerprints.get("values") != _stored_numeric_fingerprint(storage, values)
+ ):
+ raise ValueError("Imported embedding payload fingerprint does not match")
+
+
+def _default_embedding_columns(
+ assay: str,
+ cell_key: str,
+ role: str,
+ dims: int,
+) -> tuple[str, ...]:
+ label = "UMAP" if role == "umap" else "tSNE"
+ prefix = f"{assay}_{label}" if cell_key == "I" else f"{assay}_{cell_key}_{label}"
+ return tuple(f"{prefix}{index + 1}" for index in range(dims))
+
+
+def _publish_embedding_columns(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ *,
+ selection: zarr.Array,
+ columns: Sequence[str],
+ cell_data: zarr.Group,
+ block_rows: int,
+) -> None:
+ storage = ImportedArtifactStorage(root)
+ group = storage.artifact_group(ref)
+ values = storage.as_array(group["values"], "values")
+ if len(columns) != int(values.shape[1]):
+ raise ValueError(
+ "metadata_columns must contain one name per embedding dimension"
+ )
+ if len(set(columns)) != len(columns) or any(
+ not isinstance(column, str) or not column or column in {"I", "ids", "names"}
+ for column in columns
+ ):
+ raise ValueError("metadata_columns contains invalid or duplicate names")
+ n_rows = int(selection.shape[0])
+ outputs = [
+ storage.create_metadata(
+ cell_data,
+ column,
+ dtype=values.dtype,
+ shape=n_rows,
+ block_rows=block_rows,
+ )
+ for column in columns
+ ]
+ for output in outputs:
+ output[:] = np.nan
+ source_row = 0
+ for start in range(0, n_rows, block_rows):
+ stop = min(start + block_rows, n_rows)
+ block_mask = np.asarray(selection[start:stop], dtype=bool)
+ selected = int(block_mask.sum())
+ source_stop = source_row + selected
+ source_values = np.asarray(values[source_row:source_stop])
+ if source_values.shape != (selected, len(columns)):
+ raise ValueError("Imported embedding values ended before cell metadata")
+ if selected:
+ for index, output in enumerate(outputs):
+ block = np.full(stop - start, np.nan, dtype=values.dtype)
+ block[block_mask] = source_values[:, index]
+ output[start:stop] = block
+ source_row = source_stop
+ if source_row != int(values.shape[0]):
+ raise ValueError("Imported embedding values exceed the cell selection")
+ for index, output in enumerate(outputs):
+ output.attrs["source_artifact"] = ref.to_dict()
+ output.attrs["source_value"] = "values"
+ output.attrs["value_index"] = index
+
+
+def write_imported_embedding(
+ root: zarr.Group,
+ *,
+ assay: str,
+ dimreduc_key: str,
+ role: str,
+ coordinates: ImportedBlockSource,
+ source_digest: bytes,
+ payload_fingerprints: Mapping[str, str],
+ source_cell_ids: ImportedStringSource,
+ cell_selection: ArtifactRef,
+ cell_key: str,
+ coordinate_shape: tuple[int, int] | None = None,
+ coordinate_dtype: Any | None = None,
+ metadata_columns: Sequence[str] | None = None,
+ cell_data: zarr.Group | None = None,
+ named_result: str | None = None,
+ feat_key: str | None = None,
+ block_rows: int = 100_000,
+ invalidate_cache: bool = False,
+) -> ArtifactRef:
+ storage = ImportedArtifactStorage(root)
+ if not isinstance(role, str):
+ raise TypeError("role must be a string")
+ normalized_role = role.lower()
+ if normalized_role not in {"umap", "tsne"}:
+ raise ValueError("Imported embeddings require role 'umap' or 'tsne'")
+ source_digest = _validate_source_digest(source_digest)
+ if not isinstance(dimreduc_key, str) or not dimreduc_key:
+ raise ValueError("dimreduc_key must be a non-empty string")
+ block_rows = _positive_block_rows(block_rows)
+ source = _resolve_source(
+ coordinates,
+ name="coordinate",
+ shape=coordinate_shape,
+ dtype=coordinate_dtype,
+ block_rows=block_rows,
+ )
+ _validate_numeric_source(source, "coordinates", 2)
+ selection, _selected_count, ordered_cell_ids_fingerprint = _selection_alignment(
+ root,
+ cell_selection=cell_selection,
+ cell_key=cell_key,
+ source_cell_ids=source_cell_ids,
+ coordinate_rows=source.shape[0],
+ block_rows=block_rows,
+ )
+ fingerprints = _required_payload_fingerprints(
+ payload_fingerprints,
+ {"values"},
+ )
+ planned = storage.plan(
+ assay=assay,
+ kind="embedding",
+ parameters={
+ "dimreduc_key": dimreduc_key,
+ "role": normalized_role,
+ "dims": int(source.shape[1]),
+ },
+ inputs={
+ "source_digest": source_digest,
+ "payload_fingerprints": fingerprints,
+ "ordered_cell_ids_fingerprint": ordered_cell_ids_fingerprint,
+ "cell_selection": cell_selection,
+ },
+ execution_options={
+ "cell_key": cell_key,
+ "block_rows": block_rows,
+ },
+ invalidate_cache=invalidate_cache,
+ required_arrays=(
+ storage.requirement("values", shape=source.shape, dtype_kind="f"),
+ ),
+ reuse_validator=lambda _ref, group: _payloads_match(
+ storage,
+ group,
+ shapes={"values": source.shape},
+ fingerprints=fingerprints,
+ ),
+ )
+ if not planned.reused:
+ group = storage.start(planned)
+ _write_numeric_payload(
+ storage,
+ group,
+ "values",
+ source,
+ fingerprints["values"],
+ block_rows,
+ )
+ storage.finish(group, planned)
+ validate_imported_embedding_artifact(
+ root,
+ planned.ref,
+ cell_key=cell_key,
+ )
+ resolved_cell_data = cell_data
+ if resolved_cell_data is None and "cellData" in root:
+ resolved_cell_data = storage.as_group(root["cellData"], "cellData")
+ if resolved_cell_data is not None:
+ columns = (
+ tuple(metadata_columns)
+ if metadata_columns is not None
+ else _default_embedding_columns(
+ assay,
+ cell_key,
+ normalized_role,
+ int(source.shape[1]),
+ )
+ )
+ _publish_embedding_columns(
+ root,
+ planned.ref,
+ selection=selection,
+ columns=columns,
+ cell_data=resolved_cell_data,
+ block_rows=block_rows,
+ )
+ elif metadata_columns is not None:
+ raise ValueError("cell_data is required when metadata_columns are provided")
+ if named_result is not None:
+ _register_named_result(
+ root,
+ planned.ref,
+ named_result,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ )
+ return planned.ref
diff --git a/scarf/embeddings/initialization.py b/scarf/embeddings/initialization.py
new file mode 100644
index 00000000..01f3c194
--- /dev/null
+++ b/scarf/embeddings/initialization.py
@@ -0,0 +1,40 @@
+import numpy as np
+
+from ..utils.arrays import rescale_array
+
+
+def initial_embedding(
+ cluster_centers: np.ndarray,
+ cluster_labels: np.ndarray,
+ n_components: int,
+) -> np.ndarray:
+ """Create cell coordinates from PCA-projected seed partitions."""
+ from sklearn.decomposition import PCA
+
+ centers = np.asarray(cluster_centers)
+ labels = np.asarray(cluster_labels)
+ if labels.ndim != 1:
+ raise ValueError("Cluster labels must be one-dimensional")
+ if np.issubdtype(labels.dtype, np.floating):
+ if not np.all(np.isfinite(labels)) or not np.array_equal(
+ labels,
+ np.floor(labels),
+ ):
+ raise ValueError("Floating cluster labels must be finite integers")
+ elif not np.issubdtype(labels.dtype, np.integer):
+ raise TypeError("Cluster labels must contain numeric integers")
+ if np.any(labels < 0) or np.any(labels >= len(centers)):
+ raise ValueError("Cluster labels are outside the center range")
+ if n_components < 1 or n_components > min(centers.shape):
+ raise ValueError(
+ "Embedding components cannot exceed the center count or dimensions"
+ )
+ index_labels = labels.astype(np.intp, copy=False)
+ principal_components = PCA(n_components=n_components).fit_transform(centers)
+ for component in range(n_components):
+ principal_components[:, component] = rescale_array(
+ principal_components[:, component]
+ )
+ return np.array([principal_components[label] for label in index_labels]).astype(
+ np.float32, order="C"
+ )
diff --git a/scarf/embeddings/reduction.py b/scarf/embeddings/reduction.py
new file mode 100644
index 00000000..a069cd05
--- /dev/null
+++ b/scarf/embeddings/reduction.py
@@ -0,0 +1,447 @@
+from collections.abc import Callable
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+
+from ..matrix import ChunkedArray
+from ..utils.logging import logger
+
+_GRAM_PCA_MAX_FEATURES = 4096
+
+
+@dataclass(frozen=True, slots=True)
+class _GramPcaModel:
+ components_: np.ndarray
+ explained_variance_: np.ndarray
+ explained_variance_ratio_: np.ndarray
+ singular_values_: np.ndarray
+ mean_: np.ndarray
+ n_components_: int
+ n_features_in_: int
+ n_samples_seen_: int
+
+
+def _mutable_fit_block(block: np.ndarray) -> np.ndarray:
+ values = np.asarray(block)
+ if values.flags.owndata and values.flags.writeable and values.flags.c_contiguous:
+ return values
+ return np.array(values, copy=True, order="C")
+
+
+def _gram_pca_dispatch(
+ n_features: int,
+ block_rows: int,
+ n_blocks: int,
+) -> tuple[bool, str | None]:
+ if n_blocks <= 1:
+ return False, "the input has only one row block"
+ if n_features > block_rows:
+ return (
+ False,
+ f"{n_features} features exceed {block_rows} rows per block",
+ )
+ if n_features > _GRAM_PCA_MAX_FEATURES:
+ return (
+ False,
+ f"{n_features} features exceed the {_GRAM_PCA_MAX_FEATURES}-feature limit",
+ )
+ return True, None
+
+
+def _fit_sklearn_incremental_pca(
+ data: ChunkedArray,
+ *,
+ dims: int,
+ batch_size: int,
+ row_mask: np.ndarray | None,
+ scale: Callable[[np.ndarray], np.ndarray] | None,
+ nthreads: int,
+) -> tuple[np.ndarray, Any]:
+ from numpy.linalg import LinAlgError
+ from sklearn.decomposition import IncrementalPCA
+
+ model = IncrementalPCA(
+ n_components=dims + 1,
+ batch_size=batch_size,
+ )
+ end_reservoir: np.ndarray | None = None
+ carry_over: np.ndarray | None = None
+ for block in data._stream_blocks(
+ nthreads=nthreads,
+ msg="Fitting PCA",
+ prefetch=None,
+ row_mask=row_mask,
+ ):
+ if scale is not None:
+ block = scale(block)
+ if carry_over is not None:
+ block = np.vstack((carry_over, block))
+ carry_over = None
+ if len(block) < (dims + 1):
+ carry_over = block
+ continue
+ if end_reservoir is None:
+ end_reservoir = block
+ continue
+ try:
+ model.partial_fit(_mutable_fit_block(block), check_input=False)
+ except LinAlgError:
+ carry_over = block
+
+ if carry_over is not None:
+ fit_batch = (
+ np.vstack((end_reservoir, carry_over))
+ if end_reservoir is not None
+ else carry_over
+ )
+ else:
+ assert end_reservoir is not None
+ fit_batch = end_reservoir
+ model.partial_fit(_mutable_fit_block(fit_batch), check_input=False)
+ return model.components_[:-1, :].T, model
+
+
+def _fit_gram_pca(
+ data: ChunkedArray,
+ *,
+ dims: int,
+ row_mask: np.ndarray | None,
+ selected_samples: int,
+ scale: Callable[[np.ndarray], np.ndarray] | None,
+ nthreads: int,
+) -> tuple[np.ndarray, _GramPcaModel]:
+ from scipy.linalg import blas, eigh
+ from sklearn.utils.extmath import svd_flip
+ from threadpoolctl import threadpool_limits
+
+ n_features = data.shape[1]
+ n_components = dims + 1
+ gram = np.zeros(
+ (n_features, n_features),
+ dtype=np.float64,
+ order="F",
+ )
+ column_sum = np.zeros(n_features, dtype=np.float64)
+ n_samples_seen = 0
+
+ with threadpool_limits(limits=nthreads):
+ for block in data._stream_blocks(
+ nthreads=nthreads,
+ msg="Fitting PCA",
+ prefetch=None,
+ row_mask=row_mask,
+ ):
+ if scale is not None:
+ block = scale(block)
+ values = np.asfortranarray(block, dtype=np.float64)
+ gram = blas.dsyrk(
+ 1.0,
+ values,
+ beta=1.0,
+ c=gram,
+ trans=1,
+ lower=0,
+ overwrite_c=1,
+ )
+ column_sum += values.sum(axis=0, dtype=np.float64)
+ n_samples_seen += len(values)
+
+ if n_samples_seen != selected_samples:
+ raise RuntimeError(
+ f"PCA streamed {n_samples_seen} rows, expected {selected_samples}"
+ )
+
+ mean = column_sum / n_samples_seen
+ gram = blas.dsyr(
+ -float(n_samples_seen),
+ mean,
+ lower=0,
+ a=gram,
+ overwrite_a=1,
+ )
+ gram /= n_samples_seen - 1
+ total_variance = float(np.trace(gram))
+ if not np.isfinite(total_variance) or total_variance <= 0:
+ raise ValueError("PCA input must have positive finite variance")
+
+ eigenvalues, eigenvectors = eigh(
+ gram,
+ lower=False,
+ subset_by_index=(
+ n_features - n_components,
+ n_features - 1,
+ ),
+ overwrite_a=True,
+ check_finite=False,
+ driver="evr",
+ )
+
+ eigenvalues = np.clip(eigenvalues[::-1], 0.0, None)
+ components = np.array(
+ eigenvectors[:, ::-1].T,
+ dtype=np.float64,
+ copy=True,
+ order="C",
+ )
+ _, components = svd_flip(
+ None,
+ components,
+ u_based_decision=False,
+ )
+ model = _GramPcaModel(
+ components_=components,
+ explained_variance_=eigenvalues,
+ explained_variance_ratio_=eigenvalues / total_variance,
+ singular_values_=np.sqrt(eigenvalues * (n_samples_seen - 1)),
+ mean_=mean,
+ n_components_=n_components,
+ n_features_in_=n_features,
+ n_samples_seen_=n_samples_seen,
+ )
+ return model.components_[:-1, :].T, model
+
+
+def fit_incremental_pca(
+ data: ChunkedArray,
+ *,
+ dims: int,
+ batch_size: int,
+ use_for_pca: np.ndarray,
+ scale: Callable[[np.ndarray], np.ndarray] | None,
+ nthreads: int,
+) -> tuple[np.ndarray, Any]:
+ """Fit streaming PCA and return loadings with the fitted model."""
+ use_for_pca = np.asarray(use_for_pca)
+ if use_for_pca.dtype != bool or use_for_pca.shape != (data.shape[0],):
+ raise ValueError("use_for_pca must be a boolean vector matching data rows")
+
+ n_components = dims + 1
+ selected_samples = int(np.count_nonzero(use_for_pca))
+ if selected_samples < n_components:
+ raise ValueError(f"PCA requires at least {n_components} selected rows")
+ if data.shape[1] < n_components:
+ raise ValueError(f"PCA requires at least {n_components} features")
+
+ subset_samples = selected_samples != data.shape[0]
+ row_mask = use_for_pca if subset_samples else None
+ n_features = data.shape[1]
+ block_rows = data.chunksize[0]
+ use_gram, fallback_reason = _gram_pca_dispatch(
+ n_features,
+ block_rows,
+ data.numblocks[0],
+ )
+ if use_gram:
+ accumulator_mib = n_features * n_features * 8 / (1024**2)
+ logger.debug(
+ "Fitting PCA with the Gram covariance solver "
+ f"({n_features} features, {block_rows} rows per block, "
+ f"{accumulator_mib:.0f} MiB accumulator)"
+ )
+ return _fit_gram_pca(
+ data,
+ dims=dims,
+ row_mask=row_mask,
+ selected_samples=selected_samples,
+ scale=scale,
+ nthreads=nthreads,
+ )
+
+ logger.debug(
+ "Fitting PCA with the IncrementalPCA solver "
+ f"({n_features} features, {block_rows} rows per block; "
+ f"falling back because {fallback_reason})"
+ )
+ return _fit_sklearn_incremental_pca(
+ data,
+ dims=dims,
+ batch_size=batch_size,
+ row_mask=row_mask,
+ scale=scale,
+ nthreads=nthreads,
+ )
+
+
+def fit_lsi(
+ data: ChunkedArray,
+ *,
+ dims: int,
+ skip_first: bool,
+ params: dict[str, Any],
+ random_state: int,
+ nthreads: int,
+) -> np.ndarray:
+ """Fit uncentered LSI loadings with a streamed or materialized solver."""
+ reserved = {"n_components", "random_state"}
+ for key in list(params):
+ if key in reserved:
+ del params[key]
+ logger.warning(f"Provided parameter, {key}, for LSI model will not be used")
+
+ n_components = dims + int(skip_first)
+ if n_components > min(data.shape):
+ raise ValueError("LSI components cannot exceed the input matrix rank")
+ solver_params = dict(params)
+ solver = solver_params.pop("solver", "streaming")
+ if solver == "streaming":
+ allowed = {"n_iter", "n_oversamples"}
+ unsupported = sorted(set(solver_params) - allowed)
+ if unsupported:
+ joined = ", ".join(unsupported)
+ raise ValueError(f"Streaming LSI does not support parameters: {joined}")
+ return _fit_streaming_lsi(
+ data,
+ n_components=n_components,
+ skip_first=skip_first,
+ n_iter=solver_params.get("n_iter", 5),
+ n_oversamples=solver_params.get("n_oversamples", 10),
+ random_state=random_state,
+ nthreads=nthreads,
+ )
+ if solver != "materialized":
+ raise ValueError("LSI solver must be 'streaming' or 'materialized'")
+ return _fit_materialized_lsi(
+ data,
+ n_components=n_components,
+ skip_first=skip_first,
+ params=solver_params,
+ random_state=random_state,
+ nthreads=nthreads,
+ )
+
+
+def _fit_materialized_lsi(
+ data: ChunkedArray,
+ *,
+ n_components: int,
+ skip_first: bool,
+ params: dict[str, Any],
+ random_state: int,
+ nthreads: int,
+) -> np.ndarray:
+ from sklearn.decomposition import TruncatedSVD
+
+ matrix = np.vstack(
+ list(
+ data.stream_blocks(nthreads=nthreads, msg="Fitting materialized LSI model")
+ )
+ )
+ model = TruncatedSVD(
+ n_components=n_components,
+ random_state=random_state,
+ **params,
+ )
+ model.fit(matrix)
+ components = model.components_.T
+ if skip_first:
+ return np.asarray(components[:, 1:])
+ return np.asarray(components)
+
+
+def _nonnegative_integer(value: Any, name: str) -> int:
+ if isinstance(value, bool) or not isinstance(value, (int, np.integer)):
+ raise TypeError(f"{name} must be an integer")
+ resolved = int(value)
+ if resolved < 0:
+ raise ValueError(f"{name} must be nonnegative")
+ return resolved
+
+
+def _streaming_lsi_accumulator_bytes(n_features: int, width: int) -> int:
+ itemsize = np.dtype(np.float64).itemsize
+ return 3 * n_features * width * itemsize + 2 * width * width * itemsize
+
+
+def _streaming_lsi_resident_bytes(data: ChunkedArray, width: int) -> int:
+ block_rows = min(int(data.chunksize[0]), int(data.shape[0]))
+ block_temporaries = (
+ block_rows * (int(data.shape[1]) + width) * np.dtype(np.float64).itemsize
+ )
+ return (
+ _streaming_lsi_accumulator_bytes(int(data.shape[1]), width) + block_temporaries
+ )
+
+
+def _stream_lsi_gram_action(
+ data: ChunkedArray,
+ basis: np.ndarray,
+ *,
+ nthreads: int,
+ message: str,
+) -> np.ndarray:
+ result = np.zeros_like(basis, dtype=np.float64)
+ rows_seen = 0
+ resident_bytes = _streaming_lsi_resident_bytes(data, basis.shape[1])
+ for block in data._stream_blocks(
+ nthreads=nthreads,
+ msg=message,
+ prefetch=1,
+ row_mask=None,
+ resident_bytes=resident_bytes,
+ ):
+ values = np.asarray(block)
+ projected = values @ basis
+ result += values.T @ projected
+ rows_seen += len(values)
+ if rows_seen != data.shape[0]:
+ raise RuntimeError(f"LSI streamed {rows_seen} rows, expected {data.shape[0]}")
+ if not np.isfinite(result).all():
+ raise ValueError("LSI input must contain only finite values")
+ return result
+
+
+def _fit_streaming_lsi(
+ data: ChunkedArray,
+ *,
+ n_components: int,
+ skip_first: bool,
+ n_iter: Any,
+ n_oversamples: Any,
+ random_state: int,
+ nthreads: int,
+) -> np.ndarray:
+ iterations = _nonnegative_integer(n_iter, "n_iter")
+ oversamples = _nonnegative_integer(n_oversamples, "n_oversamples")
+ width = min(min(data.shape), n_components + oversamples)
+ rng = np.random.default_rng(random_state)
+ basis = rng.standard_normal((data.shape[1], width), dtype=np.float64)
+ basis, _ = np.linalg.qr(basis, mode="reduced")
+
+ for iteration in range(iterations + 1):
+ basis = _stream_lsi_gram_action(
+ data,
+ basis,
+ nthreads=nthreads,
+ message=f"Fitting streaming LSI model ({iteration + 1}/{iterations + 2})",
+ )
+ basis, _ = np.linalg.qr(basis, mode="reduced")
+
+ projected_gram = np.zeros((width, width), dtype=np.float64)
+ rows_seen = 0
+ resident_bytes = _streaming_lsi_resident_bytes(data, width)
+ for block in data._stream_blocks(
+ nthreads=nthreads,
+ msg=f"Fitting streaming LSI model ({iterations + 2}/{iterations + 2})",
+ prefetch=1,
+ row_mask=None,
+ resident_bytes=resident_bytes,
+ ):
+ projected = np.asarray(block) @ basis
+ projected_gram += projected.T @ projected
+ rows_seen += len(projected)
+ if rows_seen != data.shape[0]:
+ raise RuntimeError(f"LSI streamed {rows_seen} rows, expected {data.shape[0]}")
+ if not np.isfinite(projected_gram).all():
+ raise ValueError("LSI input must contain only finite values")
+
+ eigenvalues, rotations = np.linalg.eigh(projected_gram)
+ order = np.argsort(eigenvalues)[::-1][:n_components]
+ loadings = np.asarray(basis @ rotations[:, order], dtype=np.float64)
+ largest = np.argmax(np.abs(loadings), axis=0)
+ signs = np.sign(loadings[largest, np.arange(loadings.shape[1])])
+ signs[signs == 0] = 1
+ loadings *= signs
+ if skip_first:
+ return loadings[:, 1:]
+ return loadings
diff --git a/scarf/embeddings/sgtsne.py b/scarf/embeddings/sgtsne.py
new file mode 100644
index 00000000..65fb4e26
--- /dev/null
+++ b/scarf/embeddings/sgtsne.py
@@ -0,0 +1,137 @@
+import os
+import shutil
+from pathlib import Path
+from uuid import uuid4
+
+import numpy as np
+import pandas as pd
+from scipy.sparse import coo_matrix, csr_matrix
+
+from ..utils.logging import logger
+from ..utils.process import system_call
+from ..utils.progress import iter_progress
+
+
+def export_knn_to_mtx(
+ mtx: str,
+ csr_graph: csr_matrix,
+ batch_size: int = 1000,
+) -> None:
+ """Export a KNN matrix in Matrix Market format."""
+ n_cells = csr_graph.shape[0]
+ with open(mtx, "w") as handle:
+ handle.write(
+ "%%MatrixMarket matrix coordinate real general\n% Generated by Scarf\n"
+ )
+ handle.write(f"{n_cells} {n_cells} {csr_graph.nnz}\n")
+ start = 0
+ for end in iter_progress(
+ range(batch_size, n_cells + batch_size, batch_size),
+ desc="Saving KNN matrix in MTX format",
+ ):
+ if end > n_cells:
+ end = n_cells
+ subgraph = csr_graph[start:end].tocoo()
+ frame = pd.DataFrame(
+ {
+ "row": subgraph.row + start + 1,
+ "col": subgraph.col + 1,
+ "d": subgraph.data,
+ }
+ )
+ frame.to_csv(
+ handle,
+ sep=" ",
+ header=False,
+ index=False,
+ mode="a",
+ )
+ start = end
+ if start != n_cells:
+ raise ValueError(
+ "ERROR: Internal loop count error in export_knn_to_mtx. "
+ "Please report this bug"
+ )
+
+
+def run_sgtsne(
+ graph: csr_matrix | coo_matrix,
+ ini_embed: np.ndarray,
+ *,
+ tsne_dims: int = 2,
+ max_iter: int = 500,
+ early_iter: int = 200,
+ alpha: int = 10,
+ lambda_scale: float = 1.0,
+ box_h: float = 0.7,
+ temp_file_loc: str = ".",
+ verbose: bool = True,
+ parallel: bool = False,
+ nthreads: int = 1,
+) -> np.ndarray:
+ """Run SG-t-SNE using the CLI or Python backend."""
+ n_cells = graph.shape[0]
+ ini_embed = np.asarray(ini_embed)
+ if ini_embed.shape == (n_cells * tsne_dims,):
+ ini_embed = ini_embed.reshape(n_cells, tsne_dims)
+ if ini_embed.shape != (n_cells, tsne_dims):
+ raise ValueError(
+ f"ini_embed must have shape ({n_cells}, {tsne_dims}), got {ini_embed.shape}"
+ )
+
+ if shutil.which("sgtsne") is not None:
+ uid = str(uuid4())
+ knn_mtx_fn = Path(temp_file_loc, f"{uid}.mtx").resolve()
+ export_knn_to_mtx(str(knn_mtx_fn), graph)
+ ini_emb_fn = Path(temp_file_loc, f"{uid}.txt").resolve()
+ with open(ini_emb_fn, "w") as handle:
+ handle.write("\n".join(map(str, ini_embed.flatten())))
+ out_fn = Path(temp_file_loc, f"{uid}_output.txt").resolve()
+ threads = nthreads if parallel else 1
+ command = (
+ f"sgtsne -m {max_iter} -l {lambda_scale} -d {tsne_dims} "
+ f"-e {early_iter} -p {threads} -a {alpha} -h {box_h} "
+ f"-i {ini_emb_fn} -o {out_fn} {knn_mtx_fn}"
+ )
+ if verbose:
+ system_call(command)
+ else:
+ os.system(command)
+ try:
+ embedding = np.asarray(
+ pd.read_csv(out_fn, header=None, sep=" ")[
+ list(range(tsne_dims))
+ ].values.T
+ )
+ finally:
+ for path in (out_fn, knn_mtx_fn, ini_emb_fn):
+ if path.exists():
+ path.unlink()
+ return embedding
+
+ try:
+ from sgtsnepi import sgtsnepi
+ except ImportError as exc:
+ raise ImportError(
+ "SG-t-SNE requires the sgtsne executable on PATH or the sgtsnepi package."
+ ) from exc
+
+ if parallel:
+ logger.warning(
+ "parallel=True is not supported by the sgtsnepi Python backend; "
+ "running single-threaded"
+ )
+
+ return np.asarray(
+ sgtsnepi(
+ graph,
+ y0=ini_embed.T,
+ d=tsne_dims,
+ max_iter=max_iter,
+ early_exag=early_iter,
+ lambda_par=lambda_scale,
+ h=box_h,
+ alpha=alpha,
+ silent=not verbose,
+ )
+ )
diff --git a/scarf/umap.py b/scarf/embeddings/umap.py
similarity index 56%
rename from scarf/umap.py
rename to scarf/embeddings/umap.py
index 22f4bc73..242dc22f 100644
--- a/scarf/umap.py
+++ b/scarf/embeddings/umap.py
@@ -1,20 +1,20 @@
-# Author: Leland McInnes
-# Modified file
-#
-# License: BSD 3 clause
-
import locale
+from typing import Any
-from .utils import logger
-
-locale.setlocale(locale.LC_NUMERIC, "C")
+import numpy as np
-__all__ = ["fit_transform"]
+from ..utils.logging import logger
+from ..utils.numba import restore_numba_threads
+from ..utils.progress import tqdm_params as default_tqdm_params
+locale.setlocale(locale.LC_NUMERIC, "C")
-def calc_dens_map_params(graph, dists):
- import numpy as np
+def calc_dens_map_params(
+ graph: Any,
+ dists: np.ndarray,
+) -> tuple[np.ndarray, np.ndarray]:
+ """Compute densMAP correction terms from graph and KNN distances."""
n_vertices = graph.shape[0]
mu_sum = np.zeros(n_vertices, dtype=np.float32)
ro = np.zeros(n_vertices, dtype=np.float32)
@@ -24,45 +24,43 @@ def calc_dens_map_params(graph, dists):
j = head[i]
k = tail[i]
- D = dists[j, k] * dists[j, k] # match sq-Euclidean used for embedding
+ distance = dists[j, k] * dists[j, k]
mu = graph.data[i]
- ro[j] += mu * D
- ro[k] += mu * D
+ ro[j] += mu * distance
+ ro[k] += mu * distance
mu_sum[j] += mu
mu_sum[k] += mu
epsilon = 1e-8
ro = np.log(epsilon + (ro / mu_sum))
- R = (ro - np.mean(ro)) / np.std(ro)
- return mu_sum, R
+ standardized_ro = (ro - np.mean(ro)) / np.std(ro)
+ return mu_sum, standardized_ro
+@restore_numba_threads
def simplicial_set_embedding(
- g,
- embedding,
- n_epochs,
- a,
- b,
- random_seed,
- gamma,
- initial_alpha,
- negative_sample_rate,
- densmap_kwds,
- parallel,
- nthreads,
- verbose,
-):
+ g: Any,
+ embedding: np.ndarray,
+ n_epochs: int,
+ a: float,
+ b: float,
+ random_seed: int,
+ gamma: float,
+ initial_alpha: float,
+ negative_sample_rate: float,
+ densmap_kwds: dict[str, Any],
+ parallel: bool,
+ nthreads: int,
+ verbose: bool,
+) -> np.ndarray:
+ """Run UMAP simplicial-set embedding with optional densMAP."""
import numba
+ from sklearn.utils import check_random_state
from threadpoolctl import threadpool_limits
- from umap.umap_ import make_epochs_per_sample
from umap.layouts import optimize_layout_euclidean
- from sklearn.utils import check_random_state
- import numpy as np
- from .utils import tqdm_params
+ from umap.umap_ import make_epochs_per_sample
- # g.data[g.data < (g.data.max() / float(n_epochs))] = 0.0
- # g.eliminate_zeros()
epochs_per_sample = make_epochs_per_sample(g.data, n_epochs)
logger.trace("calculated epochs_per_sample")
rng_state = (
@@ -70,25 +68,27 @@ def simplicial_set_embedding(
.randint(np.iinfo(np.int32).min + 1, np.iinfo(np.int32).max - 1, 3)
.astype(np.int64)
)
- # Since threadpool_limits doesnt work well with numba. We will use numba's set_num_threads to limit threads
if numba.config.NUMBA_NUM_THREADS > nthreads:
numba.set_num_threads(nthreads)
if densmap_kwds != {}:
with threadpool_limits(limits=nthreads):
- mu_sum, R = calc_dens_map_params(g, densmap_kwds["knn_dists"])
+ mu_sum, standardized_ro = calc_dens_map_params(
+ g,
+ densmap_kwds["knn_dists"],
+ )
densmap_kwds["mu_sum"] = mu_sum
- densmap_kwds["R"] = R
+ densmap_kwds["R"] = standardized_ro
densmap_kwds["mu"] = g.data
densmap = True
logger.trace("calculated densmap params")
else:
densmap = False
- # tqdm will be activated if https://github.com/lmcinnes/umap/pull/739
- # is merged and when it is released
- tqdm_params = dict(tqdm_params)
+ tqdm_params = dict(default_tqdm_params)
tqdm_params["desc"] = "Training UMAP"
+ if "disable" not in tqdm_params:
+ tqdm_params["disable"] = not verbose
with threadpool_limits(limits=nthreads):
embedding = optimize_layout_euclidean(
@@ -106,41 +106,47 @@ def simplicial_set_embedding(
initial_alpha=initial_alpha,
negative_sample_rate=negative_sample_rate,
parallel=parallel,
- verbose=verbose,
+ verbose=False,
densmap=densmap,
densmap_kwds=densmap_kwds,
tqdm_kwds=tqdm_params,
move_other=True,
)
- return embedding
+ return np.asarray(embedding)
-def fuzzy_simplicial_set(g, set_op_mix_ratio):
- tg = g.transpose()
- prod = g.multiply(tg)
- res = set_op_mix_ratio * (g + tg - prod) + (1.0 - set_op_mix_ratio) * prod
- res.eliminate_zeros()
- return res.tocoo()
+def fuzzy_simplicial_set(g: Any, set_op_mix_ratio: float) -> Any:
+ """Combine directed and undirected graph components for UMAP."""
+ transposed_graph = g.transpose()
+ product = g.multiply(transposed_graph)
+ result = (
+ set_op_mix_ratio * (g + transposed_graph - product)
+ + (1.0 - set_op_mix_ratio) * product
+ )
+ result.eliminate_zeros()
+ return result.tocoo()
def fit_transform(
- graph,
- ini_embed,
- spread,
- min_dist,
- n_epochs,
- random_seed,
- repulsion_strength,
- initial_alpha,
- negative_sample_rate,
- densmap_kwds,
- parallel,
- nthreads,
- verbose,
-):
- from umap.umap_ import find_ab_params
+ graph: Any,
+ ini_embed: np.ndarray,
+ spread: float,
+ min_dist: float,
+ n_epochs: int,
+ random_seed: int,
+ repulsion_strength: float,
+ initial_alpha: float,
+ negative_sample_rate: float,
+ densmap_kwds: dict[str, Any],
+ parallel: bool,
+ nthreads: int,
+ verbose: bool,
+) -> tuple[np.ndarray, float, float]:
+ """Fit UMAP embedding from a fuzzy simplicial set graph."""
import warnings
+ from umap.umap_ import find_ab_params
+
with warnings.catch_warnings():
warnings.simplefilter("ignore", RuntimeWarning)
a, b = find_ab_params(spread=spread, min_dist=min_dist)
diff --git a/scarf/feat_utils.py b/scarf/feat_utils.py
deleted file mode 100644
index 0e53287e..00000000
--- a/scarf/feat_utils.py
+++ /dev/null
@@ -1,162 +0,0 @@
-"""Utility functions for features."""
-
-from typing import List, Tuple
-
-import numpy as np
-import pandas as pd
-
-__all__ = ["fit_lowess", "binned_sampling", "hto_demux"]
-
-
-def fit_lowess(a, b, n_bins: int, lowess_frac: float) -> np.ndarray:
- """Fits a LOWESS (Locally Weighted Scatterplot Smoothing) curve.
-
- Args:
- a:
- b:
- n_bins:
- lowess_frac:
-
- Returns:
- """
- from statsmodels.nonparametric.smoothers_lowess import lowess
-
- stats = pd.DataFrame({"a": a, "b": b}).apply(np.log)
- bin_edges = np.histogram(stats.a, bins=n_bins)[1]
- bin_edges[-1] += 0.1 # For including last gene
- bin_idx = []
- for i in range(n_bins):
- idx = pd.Series((stats.a >= bin_edges[i]) & (stats.a < bin_edges[i + 1]))
- if sum(idx) > 0:
- bin_idx.append(list(idx[idx].index))
- bin_vals = []
- for idx in bin_idx:
- temp_stat = stats.reindex(idx)
- temp_gene = temp_stat.idxmin().b
- bin_vals.append([temp_stat.b[temp_gene], temp_stat.a[temp_gene]])
- bin_vals = np.array(bin_vals).T
- bin_cor_fac = lowess(
- bin_vals[0], bin_vals[1], return_sorted=False, frac=lowess_frac, it=100
- ).T
- fixed_var = {}
- for bcf, indices in zip(bin_cor_fac, bin_idx):
- for idx in indices:
- fixed_var[idx] = np.e ** (stats.b[idx] - bcf)
- return np.array([fixed_var[x] for x in range(len(a))])
-
-
-def binned_sampling(
- values: pd.Series,
- feature_list: List[str],
- ctrl_size: int,
- n_bins: int,
- rand_seed: int,
-) -> List[str]:
- """Score a set of genes [Satija15]_. The score is the average expression of
- a set of genes subtracted with the average expression of a reference set of
- genes. The reference set is randomly sampled from the `gene_pool` for each
- binned expression value.
-
- This reproduces the approach in Seurat [Satija15]_ and has been implemented
- for Scanpy by Davide Cittaro.
-
- This function is adapted from Scanpy's `score_genes`.
-
- Args:
- values: The values for the features.
- feature_list: The list of features to use for score calculation.
- ctrl_size: Number of reference features to be sampled from each bin.
- n_bins: Number of bins for sampling.
- rand_seed: The seed to use for the random number generation.
-
- Returns:
- A list of sampled features.
- """
- n_items = int(np.round(len(values) / (n_bins - 1)))
- feature_list = set(feature_list)
- # Made following more linter friendly
- # obs_cut = obs_avg.rank(method='min') // n_items
- obs_cut: pd.Series = values.fillna(0).rank(method="min").divide(n_items).astype(int)
-
- control_genes = set()
- for cut in np.unique(obs_cut[list(feature_list)]):
- sub_obs = obs_cut[obs_cut == cut]
- if len(sub_obs) == 0:
- continue
- if len(sub_obs) < ctrl_size:
- sample_size = len(sub_obs)
- else:
- sample_size = ctrl_size
- r_genes = sub_obs.sample(n=sample_size, random_state=rand_seed).index
- control_genes.update(set(r_genes))
- return list(control_genes - feature_list)
-
-
-def hto_demux(hto_counts: pd.DataFrame) -> pd.Series:
- """Assigns HTO identity to each cell based on the HTO count distribution.
- The algorithm is adapted from the Seurat package's HTOdemux function [Satija15]_.
-
- Args:
- hto_counts: A dataframe containing the raw HTO counts for each cell.
-
- Returns:
- A series containing the HTO identity for each cell.
- """
- from sklearn.cluster import KMeans
- from statsmodels.discrete.discrete_model import NegativeBinomial
- from scipy.stats import nbinom
-
- def clr_normalize(df: pd.DataFrame) -> pd.DataFrame:
- f = np.exp(np.log1p(df).sum(axis=0) / len(df))
- return np.log1p(df / f)
-
- def calc_cluster_labels(
- df: pd.DataFrame, n_centers: int | None = None, n_starts: int = 100
- ):
- if n_centers is None:
- n_centers = df.shape[1] + 1
- kmeans = KMeans(n_clusters=n_centers, init="random", n_init=n_starts)
- kmeans.fit(df)
- return kmeans.labels_
-
- def calc_cluster_avg_exp(df: pd.DataFrame) -> Tuple[pd.Series, pd.DataFrame]:
- df["cluster"] = calc_cluster_labels(df)
- return df["cluster"], df.groupby("cluster").mean()
-
- def get_background_cutoff(vals: np.ndarray, quantile: float = 0.99) -> int:
- fit = NegativeBinomial(vals, np.ones_like(vals)).fit(
- start_params=[1, 1], disp=0
- )
- mu = np.exp(fit.params[0])
- p = 1 / (1 + np.exp(fit.params[0]) * fit.params[1])
- n = np.exp(fit.params[0]) * p / (1 - p)
- dist = nbinom(n=n, p=p, loc=mu)
- return round(dist.ppf(quantile))
-
- def discretize_counts(
- df: pd.DataFrame, clust_labels: pd.Series, clust_exp: pd.DataFrame
- ) -> pd.DataFrame:
- min_clust = clust_exp.idxmin()
- cutoffs = {}
- for hto in df:
- bg_values = df[hto][clust_labels == min_clust[hto]].values
- cutoffs[hto] = get_background_cutoff(bg_values)
- cutoffs = pd.Series(cutoffs)
- return df > cutoffs
-
- def identity_renamer(x: int):
- if x == 0:
- return "Negative"
- elif x == 1:
- return "Singlet"
- else:
- return "Doublet"
-
- cluster_labels, avg_exp = calc_cluster_avg_exp(clr_normalize(hto_counts))
- # Seurat does the following check and hard stops the process if the assertion fails
- assert any(avg_exp.sum(axis=1) == 0) is False
- hto_discrete = discretize_counts(hto_counts, cluster_labels, avg_exp)
- g_class = hto_discrete.sum(axis=1).apply(identity_renamer)
- singlet_ident = hto_counts[g_class == "Singlet"].idxmax(axis=1)
- g_class[singlet_ident.index] = singlet_ident
- return g_class
diff --git a/scarf/features/__init__.py b/scarf/features/__init__.py
new file mode 100644
index 00000000..e992b4bc
--- /dev/null
+++ b/scarf/features/__init__.py
@@ -0,0 +1,90 @@
+from importlib import import_module as _import_module
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from .enrichment import (
+ EnrichmentResult as EnrichmentResult,
+ read_gmt as read_gmt,
+ )
+ from .genomic import (
+ GffReader as GffReader,
+ binary_search as binary_search,
+ coordinate_melding as coordinate_melding,
+ create_bed_from_coord_ids as create_bed_from_coord_ids,
+ create_counts_mat as create_counts_mat,
+ get_feature_mappings as get_feature_mappings,
+ get_ranges as get_ranges,
+ )
+ from .markers import (
+ find_markers_by_rank as find_markers_by_rank,
+ find_markers_by_regression as find_markers_by_regression,
+ mannwhitneyu_from_ranks as mannwhitneyu_from_ranks,
+ resolve_marker_gene_batch_size as resolve_marker_gene_batch_size,
+ sort_marker_results as sort_marker_results,
+ )
+ from .scoring import binned_sampling as binned_sampling
+ from .variability import (
+ fit_lowess as fit_lowess,
+ select_highly_variable_features as select_highly_variable_features,
+ )
+
+__all__ = [
+ "EnrichmentResult",
+ "GffReader",
+ "binary_search",
+ "binned_sampling",
+ "coordinate_melding",
+ "create_bed_from_coord_ids",
+ "create_counts_mat",
+ "find_markers_by_rank",
+ "find_markers_by_regression",
+ "fit_lowess",
+ "get_feature_mappings",
+ "get_ranges",
+ "mannwhitneyu_from_ranks",
+ "read_gmt",
+ "resolve_marker_gene_batch_size",
+ "select_highly_variable_features",
+ "sort_marker_results",
+]
+
+_LAZY_EXPORTS = {
+ "EnrichmentResult": (".enrichment", "EnrichmentResult"),
+ "GffReader": (".genomic", "GffReader"),
+ "binary_search": (".genomic", "binary_search"),
+ "binned_sampling": (".scoring", "binned_sampling"),
+ "coordinate_melding": (".genomic", "coordinate_melding"),
+ "create_bed_from_coord_ids": (".genomic", "create_bed_from_coord_ids"),
+ "create_counts_mat": (".genomic", "create_counts_mat"),
+ "find_markers_by_rank": (".markers", "find_markers_by_rank"),
+ "find_markers_by_regression": (".markers", "find_markers_by_regression"),
+ "fit_lowess": (".variability", "fit_lowess"),
+ "get_feature_mappings": (".genomic", "get_feature_mappings"),
+ "get_ranges": (".genomic", "get_ranges"),
+ "mannwhitneyu_from_ranks": (".markers", "mannwhitneyu_from_ranks"),
+ "read_gmt": (".enrichment", "read_gmt"),
+ "resolve_marker_gene_batch_size": (".markers", "resolve_marker_gene_batch_size"),
+ "select_highly_variable_features": (
+ ".variability",
+ "select_highly_variable_features",
+ ),
+ "sort_marker_results": (".markers", "sort_marker_results"),
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+
+def __getattr__(name: str) -> Any:
+ export = _LAZY_EXPORTS.get(name)
+ if export is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ module_name, attribute_name = export
+ value = getattr(_import_module(module_name, __name__), attribute_name)
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()).union(_LAZY_EXPORTS))
diff --git a/scarf/features/enrichment/__init__.py b/scarf/features/enrichment/__init__.py
new file mode 100644
index 00000000..2f932cd3
--- /dev/null
+++ b/scarf/features/enrichment/__init__.py
@@ -0,0 +1,31 @@
+from importlib import import_module as _import_module
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from .net import read_gmt as read_gmt
+ from .results import EnrichmentResult as EnrichmentResult
+
+__all__ = ["EnrichmentResult", "read_gmt"]
+
+_LAZY_EXPORTS = {
+ "EnrichmentResult": (".results", "EnrichmentResult"),
+ "read_gmt": (".net", "read_gmt"),
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+
+def __getattr__(name: str) -> Any:
+ export = _LAZY_EXPORTS.get(name)
+ if export is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ module_name, attribute_name = export
+ value = getattr(_import_module(module_name, __name__), attribute_name)
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()).union(_LAZY_EXPORTS))
diff --git a/scarf/features/enrichment/aucell.py b/scarf/features/enrichment/aucell.py
new file mode 100644
index 00000000..a45cbe0e
--- /dev/null
+++ b/scarf/features/enrichment/aucell.py
@@ -0,0 +1,234 @@
+from dataclasses import dataclass
+
+import numpy as np
+from numba import njit, prange
+
+from .net import PreparedNetwork
+
+__all__ = [
+ "AUCELL_ALGORITHM_VERSION",
+ "GeneSetIndex",
+ "build_gene_set_index",
+ "make_rank_permutation",
+ "resolve_n_up",
+ "score_aucell_block",
+]
+
+AUCELL_ALGORITHM_VERSION = 1
+
+
+def _owned_readonly(values: np.ndarray) -> np.ndarray:
+ array = np.asarray(values, dtype=np.int64).copy()
+ array.setflags(write=False)
+ return array
+
+
+@dataclass(frozen=True, slots=True)
+class GeneSetIndex:
+ connections: np.ndarray
+ starts: np.ndarray
+ offsets: np.ndarray
+
+ def __post_init__(self) -> None:
+ if (
+ self.connections.ndim != 1
+ or self.starts.ndim != 1
+ or self.offsets.ndim != 1
+ ):
+ raise ValueError("Gene-set index arrays must be one-dimensional")
+ if len(self.starts) == 0 or len(self.starts) != len(self.offsets):
+ raise ValueError(
+ "Gene-set starts and offsets must be non-empty and aligned"
+ )
+ if any(
+ not np.issubdtype(values.dtype, np.integer)
+ for values in (self.connections, self.starts, self.offsets)
+ ):
+ raise ValueError("Gene-set index arrays must have integer dtypes")
+ if np.any(self.offsets <= 0):
+ raise ValueError("Gene-set offsets must be positive")
+ if np.any(self.connections < 0) or np.any(self.starts < 0):
+ raise ValueError("Gene-set indices must be non-negative")
+ if int(self.offsets.sum()) != len(self.connections):
+ raise ValueError("Gene-set connections do not match offsets")
+ expected_starts = np.zeros(len(self.offsets), dtype=np.int64)
+ expected_starts[1:] = np.cumsum(self.offsets[:-1])
+ if not np.array_equal(self.starts, expected_starts):
+ raise ValueError("Gene-set starts are invalid")
+ for start, size in zip(self.starts, self.offsets, strict=True):
+ values = self.connections[start : start + size]
+ if np.unique(values).size != len(values):
+ raise ValueError("Gene sets must not contain duplicate connections")
+
+
+def resolve_n_up(n_features: int, n_up: int | None) -> int:
+ """Resolve the number of top-ranked features used for AUCell."""
+ if isinstance(n_features, bool) or not isinstance(n_features, int):
+ raise TypeError("n_features must be an integer")
+ if n_features < 2:
+ raise ValueError("AUCell requires at least two ranking features")
+ if n_up is None:
+ return int(np.clip(np.ceil(0.05 * n_features), 2, n_features))
+ if isinstance(n_up, bool) or not isinstance(n_up, int):
+ raise TypeError("n_up must be an integer or None")
+ if not 1 < n_up <= n_features:
+ raise ValueError(
+ f"n_up must be greater than 1 and at most {n_features}, got {n_up}"
+ )
+ return n_up
+
+
+def make_rank_permutation(n_features: int, tie_seed: int) -> np.ndarray:
+ """Create the one global feature permutation used to break rank ties."""
+ if isinstance(n_features, bool) or not isinstance(n_features, int):
+ raise TypeError("n_features must be an integer")
+ if n_features < 2:
+ raise ValueError("AUCell requires at least two ranking features")
+ if isinstance(tie_seed, bool) or not isinstance(tie_seed, int):
+ raise TypeError("tie_seed must be an integer")
+ if tie_seed < 0:
+ raise ValueError("tie_seed must be non-negative")
+ permutation = np.random.default_rng(tie_seed).permutation(n_features)
+ return _owned_readonly(permutation)
+
+
+def build_gene_set_index(
+ network: PreparedNetwork,
+ rank_feature_index: np.ndarray,
+) -> GeneSetIndex:
+ """Index source targets by position in the permuted ranking universe."""
+ rank_index = np.asarray(rank_feature_index)
+ if rank_index.ndim != 1 or len(rank_index) < 2:
+ raise ValueError("rank_feature_index must contain at least two features")
+ if not np.issubdtype(rank_index.dtype, np.integer):
+ raise ValueError("rank_feature_index must have an integer dtype")
+ rank_index = np.asarray(rank_index, dtype=np.int64)
+ if np.any(rank_index < 0):
+ raise ValueError("rank_feature_index must be non-negative")
+ if np.unique(rank_index).size != len(rank_index):
+ raise ValueError("rank_feature_index must contain unique feature indices")
+ positions = {int(feature): pos for pos, feature in enumerate(rank_index)}
+ per_source: list[list[int]] = [[] for _ in network.source_names]
+ for source, feature in zip(
+ network.edge_source_index,
+ network.edge_feature_index,
+ strict=True,
+ ):
+ try:
+ position = positions[int(feature)]
+ except KeyError as exc:
+ raise ValueError(
+ "Network target is absent from the AUCell ranking universe"
+ ) from exc
+ per_source[int(source)].append(position)
+
+ offsets = np.asarray([len(values) for values in per_source], dtype=np.int64)
+ if not np.array_equal(offsets, network.source_sizes):
+ raise ValueError("Gene-set sizes do not align with prepared network sources")
+ starts = np.zeros(len(offsets), dtype=np.int64)
+ starts[1:] = np.cumsum(offsets[:-1])
+ connections = np.concatenate(
+ [np.asarray(sorted(values), dtype=np.int64) for values in per_source]
+ )
+ return GeneSetIndex(
+ connections=_owned_readonly(connections),
+ starts=_owned_readonly(starts),
+ offsets=_owned_readonly(offsets),
+ )
+
+
+@njit(cache=True, parallel=True)
+def _score_ranked_row(
+ ranks: np.ndarray,
+ connections: np.ndarray,
+ starts: np.ndarray,
+ offsets: np.ndarray,
+ n_up: int,
+) -> np.ndarray:
+ n_sources = starts.size
+ scores = np.zeros(n_sources, dtype=np.float64)
+ for source in prange(n_sources):
+ start = starts[source]
+ size = offsets[source]
+
+ max_auc = 0.0
+ threshold_count = min(size, n_up - 1)
+ for threshold in range(1, threshold_count + 1):
+ next_threshold = threshold + 1
+ if threshold == threshold_count:
+ next_threshold = n_up
+ max_auc += (next_threshold - threshold) * threshold
+
+ selected = np.empty(size, dtype=np.int64)
+ selected_count = 0
+ for offset in range(size):
+ rank = ranks[connections[start + offset]]
+ if rank <= n_up:
+ selected[selected_count] = rank
+ selected_count += 1
+ if selected_count == 0:
+ continue
+ selected_values = np.sort(selected[:selected_count])
+ area = 0.0
+ for index in range(selected_count):
+ left = selected_values[index]
+ right = n_up
+ if index + 1 < selected_count:
+ right = selected_values[index + 1]
+ area += (right - left) * (index + 1)
+ scores[source] = area / max_auc
+ return scores
+
+
+def score_aucell_block(
+ values: np.ndarray,
+ permutation: np.ndarray,
+ sets: GeneSetIndex,
+ *,
+ n_up: int,
+) -> np.ndarray:
+ """Score one raw count block using AUCell recovery curves."""
+ matrix = np.asarray(values)
+ if matrix.ndim != 2:
+ raise ValueError("AUCell values must be two-dimensional")
+ if (
+ not np.issubdtype(matrix.dtype, np.number)
+ or np.issubdtype(matrix.dtype, np.complexfloating)
+ or not np.isfinite(matrix).all()
+ ):
+ raise ValueError("AUCell values must be finite and numeric")
+ permutation_array = np.asarray(permutation)
+ if permutation_array.ndim != 1 or len(permutation_array) != matrix.shape[1]:
+ raise ValueError("AUCell permutation must align with matrix features")
+ if not np.issubdtype(permutation_array.dtype, np.integer):
+ raise ValueError("AUCell permutation must have an integer dtype")
+ permutation_array = np.asarray(permutation_array, dtype=np.int64)
+ if not np.array_equal(
+ np.sort(permutation_array), np.arange(matrix.shape[1], dtype=np.int64)
+ ):
+ raise ValueError("AUCell permutation must contain every feature position once")
+ resolved_n_up = resolve_n_up(matrix.shape[1], n_up)
+ if np.any(sets.connections < 0) or np.any(sets.connections >= matrix.shape[1]):
+ raise ValueError("Gene-set connections are outside the ranking universe")
+
+ scores = np.zeros((matrix.shape[0], len(sets.starts)), dtype=np.float64)
+ ordinal = np.arange(1, matrix.shape[1] + 1, dtype=np.int64)
+ for row_index, row in enumerate(matrix):
+ if not np.any(row != 0):
+ continue
+ permuted = np.asarray(row[permutation_array], dtype=np.float64)
+ order = np.argsort(-permuted, kind="stable")
+ ranks = np.empty(matrix.shape[1], dtype=np.int64)
+ ranks[order] = ordinal
+ scores[row_index] = _score_ranked_row(
+ ranks,
+ sets.connections,
+ sets.starts,
+ sets.offsets,
+ resolved_n_up,
+ )
+
+ tolerance = 1e-12
+ if np.any(scores < -tolerance) or np.any(scores > 1.0 + tolerance):
+ raise ValueError("AUCell produced scores outside the expected [0, 1] range")
+ return np.clip(scores, 0.0, 1.0)
diff --git a/scarf/features/enrichment/net.py b/scarf/features/enrichment/net.py
new file mode 100644
index 00000000..ca19b4bb
--- /dev/null
+++ b/scarf/features/enrichment/net.py
@@ -0,0 +1,301 @@
+import hashlib
+import os
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+import pandas as pd
+from numpy.typing import DTypeLike
+
+__all__ = ["PreparedNetwork", "prepare_network", "read_gmt"]
+
+
+def _owned_readonly(values: np.ndarray, dtype: DTypeLike) -> np.ndarray:
+ array = np.asarray(values, dtype=dtype).copy()
+ array.setflags(write=False)
+ return array
+
+
+def _update_text(digest: Any, value: str) -> None:
+ encoded = value.encode("utf-8")
+ digest.update(len(encoded).to_bytes(8, "little", signed=False))
+ digest.update(encoded)
+
+
+def _network_digest(
+ source_names: np.ndarray,
+ edge_source_index: np.ndarray,
+ edge_feature_index: np.ndarray,
+ edge_weight: np.ndarray,
+ *,
+ weighted: bool,
+) -> str:
+ digest = hashlib.blake2b(digest_size=16)
+ digest.update(b"weighted\0" if weighted else b"unweighted\0")
+ digest.update(len(source_names).to_bytes(8, "little", signed=False))
+ for source in source_names:
+ _update_text(digest, str(source))
+ for values, dtype in (
+ (edge_source_index, " None:
+ one_dimensional = (
+ self.source_names,
+ self.source_sizes,
+ self.matched_feature_index,
+ self.edge_source_index,
+ self.edge_feature_index,
+ self.edge_weight,
+ )
+ if any(values.ndim != 1 for values in one_dimensional):
+ raise ValueError("Prepared network arrays must be one-dimensional")
+ if len(self.source_names) == 0:
+ raise ValueError("Prepared network must contain at least one source")
+ if len(self.source_names) != len(self.source_sizes):
+ raise ValueError("Source names and sizes must be aligned")
+ n_edges = len(self.edge_source_index)
+ if len(self.edge_feature_index) != n_edges or len(self.edge_weight) != n_edges:
+ raise ValueError("Prepared network edge arrays must be aligned")
+ if n_edges == 0 or len(self.matched_feature_index) == 0:
+ raise ValueError("Prepared network must contain at least one matched edge")
+ if np.any(self.source_sizes <= 0):
+ raise ValueError("Prepared network source sizes must be positive")
+ if np.unique(self.source_names).size != len(self.source_names):
+ raise ValueError("Prepared network source names must be unique")
+ if not all(isinstance(source, str) and source for source in self.source_names):
+ raise ValueError("Prepared network source names must be non-empty strings")
+ if not np.array_equal(self.source_names, np.sort(self.source_names)):
+ raise ValueError("Prepared network source names must be sorted")
+ integer_arrays = (
+ self.source_sizes,
+ self.matched_feature_index,
+ self.edge_source_index,
+ self.edge_feature_index,
+ )
+ if any(
+ not np.issubdtype(values.dtype, np.integer) for values in integer_arrays
+ ):
+ raise ValueError("Prepared network index arrays must have integer dtypes")
+ if np.any(self.matched_feature_index < 0) or not np.array_equal(
+ self.matched_feature_index, np.unique(self.matched_feature_index)
+ ):
+ raise ValueError(
+ "Prepared network matched features must be sorted and unique"
+ )
+ if np.any(self.edge_source_index < 0) or np.any(
+ self.edge_source_index >= len(self.source_names)
+ ):
+ raise ValueError("Prepared network source indices are out of bounds")
+ if (
+ np.any(self.edge_feature_index < 0)
+ or not np.isin(self.edge_feature_index, self.matched_feature_index).all()
+ ):
+ raise ValueError("Prepared network feature indices are out of bounds")
+ expected_sizes = np.bincount(
+ self.edge_source_index,
+ minlength=len(self.source_names),
+ )
+ if not np.array_equal(self.source_sizes, expected_sizes):
+ raise ValueError("Prepared network source sizes do not match its edges")
+ edge_order = np.lexsort((self.edge_feature_index, self.edge_source_index))
+ if not np.array_equal(edge_order, np.arange(n_edges)):
+ raise ValueError("Prepared network edges must be canonically sorted")
+ if not np.isfinite(self.edge_weight).all():
+ raise ValueError("Prepared network weights must be finite")
+ if not isinstance(self.network_digest, str) or not self.network_digest:
+ raise ValueError("Prepared network digest must be a non-empty string")
+
+
+def read_gmt(path: str | os.PathLike[str]) -> pd.DataFrame:
+ """Read gene sets from a tab-separated GMT file.
+
+ The first field is used as the source name, the second description field is
+ ignored, and each remaining non-empty field becomes one target row.
+
+ Args:
+ path: Path to a GMT file with at least three fields per non-empty line.
+
+ Returns:
+ A DataFrame with ``source`` and ``target`` columns.
+ """
+ records: list[tuple[str, str]] = []
+ with open(path, encoding="utf-8") as handle:
+ for line_number, raw_line in enumerate(handle, start=1):
+ line = raw_line.rstrip("\r\n")
+ if not line.strip():
+ continue
+ fields = line.split("\t")
+ if len(fields) < 3:
+ raise ValueError(
+ f"Malformed GMT line {line_number}: expected at least 3 tab-separated fields"
+ )
+ source = fields[0].strip()
+ targets = [target.strip() for target in fields[2:] if target.strip()]
+ if not source:
+ raise ValueError(f"Malformed GMT line {line_number}: source is empty")
+ if not targets:
+ raise ValueError(f"Malformed GMT line {line_number}: no targets found")
+ records.extend((source, target) for target in targets)
+ if not records:
+ raise ValueError("GMT file contains no gene sets")
+ return pd.DataFrame.from_records(records, columns=["source", "target"])
+
+
+def prepare_network(
+ net: pd.DataFrame,
+ *,
+ active_feature_names: np.ndarray,
+ active_feature_index: np.ndarray,
+ tmin: int,
+ weighted: bool,
+) -> PreparedNetwork:
+ """Validate, match, prune, and canonically order a gene-set network."""
+ if not isinstance(net, pd.DataFrame):
+ raise TypeError("net must be a pandas DataFrame")
+ if isinstance(tmin, bool) or not isinstance(tmin, int) or tmin < 1:
+ raise ValueError("tmin must be an integer greater than or equal to 1")
+ if not isinstance(weighted, bool):
+ raise TypeError("weighted must be a boolean")
+ if not {"source", "target"}.issubset(net.columns):
+ raise ValueError("net must contain 'source' and 'target' columns")
+ if any(
+ list(net.columns).count(column) > 1 for column in ("source", "target", "weight")
+ ):
+ raise ValueError("Network columns must have unique names")
+
+ feature_names = np.asarray(active_feature_names)
+ raw_feature_index = np.asarray(active_feature_index)
+ if feature_names.ndim != 1 or raw_feature_index.ndim != 1:
+ raise ValueError("Active feature names and indices must be one-dimensional")
+ if len(feature_names) == 0:
+ raise ValueError("Feature key selects no active features")
+ if len(feature_names) != len(raw_feature_index):
+ raise ValueError("Active feature names and indices must be aligned")
+ if not np.issubdtype(raw_feature_index.dtype, np.integer):
+ raise ValueError("Active feature indices must have an integer dtype")
+ feature_index = np.asarray(raw_feature_index, dtype=np.int64)
+ if np.any(feature_index < 0) or np.unique(feature_index).size != len(feature_index):
+ raise ValueError("Active feature indices must be non-negative and unique")
+
+ columns = ["source", "target"]
+ if "weight" in net.columns:
+ columns.append("weight")
+ frame = net.loc[:, columns].copy(deep=True).reset_index(drop=True)
+ if frame[["source", "target"]].isna().any().any():
+ raise ValueError("Network source and target values must not be missing")
+ frame["source"] = frame["source"].map(lambda value: str(value).strip())
+ frame["target"] = frame["target"].map(lambda value: str(value).strip())
+ if (frame["source"] == "").any() or (frame["target"] == "").any():
+ raise ValueError("Network source and target values must be non-empty")
+ if frame.duplicated(subset=["source", "target"]).any():
+ raise ValueError("Network contains duplicate source-target edges")
+
+ if "weight" not in frame.columns:
+ frame["weight"] = 1.0
+ if frame["weight"].map(lambda value: isinstance(value, (bool, np.bool_))).any():
+ raise ValueError("Network weights must be numeric and not boolean")
+ try:
+ weights = pd.to_numeric(frame["weight"], errors="raise").to_numpy(
+ dtype=np.float64
+ )
+ except (TypeError, ValueError) as exc:
+ raise ValueError("Network weights must be numeric") from exc
+ if not np.isfinite(weights).all():
+ raise ValueError("Network weights must be finite")
+ frame["weight"] = weights
+ if weighted:
+ frame = frame.loc[frame["weight"] != 0.0].copy()
+ if frame.empty:
+ raise ValueError("Weighted network contains no non-zero edges")
+ else:
+ frame["weight"] = 1.0
+
+ name_to_indices: dict[str, list[int]] = {}
+ for name, index in zip(feature_names, feature_index, strict=True):
+ name_to_indices.setdefault(str(name).upper(), []).append(int(index))
+
+ matched_rows: list[int] = []
+ matched_indices: list[int] = []
+ for row_index, target in zip(frame.index, frame["target"], strict=True):
+ matches = name_to_indices.get(str(target).upper(), [])
+ if not matches:
+ continue
+ if len(matches) > 1:
+ raise ValueError(
+ f"Network target {target!r} matches multiple active assay features"
+ )
+ matched_rows.append(int(row_index))
+ matched_indices.append(matches[0])
+
+ if not matched_rows:
+ raise ValueError("Network has no targets overlapping the active assay features")
+ frame = frame.loc[matched_rows].copy()
+ frame["feature_index"] = np.asarray(matched_indices, dtype=np.int64)
+ if frame.duplicated(subset=["source", "feature_index"]).any():
+ raise ValueError(
+ "Network contains duplicate edges after case-insensitive feature matching"
+ )
+
+ counts = frame.groupby("source", sort=False, observed=True).size()
+ kept_sources = sorted(str(source) for source in counts[counts >= tmin].index)
+ if not kept_sources:
+ raise ValueError(
+ "No network sources remain after feature overlap and "
+ f"tmin={tmin} pruning ({len(frame)} matched edges)"
+ )
+ frame = frame.loc[frame["source"].isin(kept_sources)].copy()
+
+ source_names = np.asarray(kept_sources, dtype="U")
+ source_to_index = {
+ source: index for index, source in enumerate(source_names.tolist())
+ }
+ frame["source_index"] = frame["source"].map(source_to_index).astype(np.int64)
+ frame = frame.sort_values(
+ ["source_index", "feature_index"], kind="mergesort"
+ ).reset_index(drop=True)
+
+ edge_source_index = frame["source_index"].to_numpy(dtype=np.int64)
+ edge_feature_index = frame["feature_index"].to_numpy(dtype=np.int64)
+ edge_weight = frame["weight"].to_numpy(dtype=np.float64)
+ matched_feature_index = np.unique(edge_feature_index).astype(np.int64)
+ source_sizes = np.bincount(edge_source_index, minlength=len(source_names)).astype(
+ np.int64
+ )
+ digest = _network_digest(
+ source_names,
+ edge_source_index,
+ edge_feature_index,
+ edge_weight,
+ weighted=weighted,
+ )
+
+ return PreparedNetwork(
+ source_names=_owned_readonly(source_names, source_names.dtype),
+ source_sizes=_owned_readonly(source_sizes, np.int64),
+ matched_feature_index=_owned_readonly(matched_feature_index, np.int64),
+ edge_source_index=_owned_readonly(edge_source_index, np.int64),
+ edge_feature_index=_owned_readonly(edge_feature_index, np.int64),
+ edge_weight=_owned_readonly(edge_weight, np.float64),
+ network_digest=digest,
+ )
diff --git a/scarf/features/enrichment/results.py b/scarf/features/enrichment/results.py
new file mode 100644
index 00000000..7285d838
--- /dev/null
+++ b/scarf/features/enrichment/results.py
@@ -0,0 +1,60 @@
+from dataclasses import dataclass, field
+
+import numpy as np
+
+from ...matrix import ChunkedArray
+
+__all__ = ["EnrichmentResult"]
+
+
+@dataclass(frozen=True, slots=True, eq=False)
+class EnrichmentResult:
+ """A persisted gene-set enrichment result.
+
+ Attributes:
+ data: Lazy score matrix with cells as rows and sources as columns.
+ source_names: Source names aligned to the score columns.
+ source_sizes: Matched target counts aligned to ``source_names``.
+ cell_index: Assay cell indices aligned to the score rows.
+ label: User-supplied persistence label.
+ storage_path: Zarr path that owns the result.
+ assay: Name of the RNA assay used for scoring.
+ cell_key: Cell selection used when the result was created.
+ feature_key: Feature selection used when the result was created.
+ method: Enrichment method, either ``"waggr"`` or ``"aucell"``.
+ """
+
+ data: ChunkedArray = field(repr=False)
+ source_names: np.ndarray = field(repr=False)
+ source_sizes: np.ndarray = field(repr=False)
+ cell_index: np.ndarray = field(repr=False)
+ label: str
+ storage_path: str
+ assay: str
+ cell_key: str
+ feature_key: str
+ method: str
+
+ def __post_init__(self) -> None:
+ if len(self.data.shape) != 2:
+ raise ValueError("Enrichment data must be two-dimensional")
+ if (
+ self.source_names.ndim != 1
+ or self.source_sizes.ndim != 1
+ or self.cell_index.ndim != 1
+ ):
+ raise ValueError("Enrichment metadata arrays must be one-dimensional")
+ if self.data.shape[0] != len(self.cell_index):
+ raise ValueError("Enrichment cell indices do not align with score rows")
+ if self.data.shape[1] != len(self.source_names):
+ raise ValueError("Enrichment source names do not align with score columns")
+ if len(self.source_names) != len(self.source_sizes):
+ raise ValueError("Enrichment source names and sizes must be aligned")
+ if np.unique(self.source_names).size != len(self.source_names):
+ raise ValueError("Enrichment source names must be unique")
+ if np.unique(self.cell_index).size != len(self.cell_index):
+ raise ValueError("Enrichment cell indices must be unique")
+ if np.any(self.source_sizes <= 0):
+ raise ValueError("Enrichment source sizes must be positive")
+ if self.method not in {"waggr", "aucell"}:
+ raise ValueError(f"Unknown enrichment method: {self.method!r}")
diff --git a/scarf/features/enrichment/waggr.py b/scarf/features/enrichment/waggr.py
new file mode 100644
index 00000000..0773d29c
--- /dev/null
+++ b/scarf/features/enrichment/waggr.py
@@ -0,0 +1,89 @@
+from dataclasses import dataclass
+from typing import Literal
+
+import numpy as np
+from scipy.sparse import csc_matrix
+
+from .net import PreparedNetwork
+
+__all__ = [
+ "WAGGR_ALGORITHM_VERSION",
+ "WaggrModel",
+ "build_waggr_model",
+ "score_waggr_block",
+]
+
+WAGGR_ALGORITHM_VERSION = 1
+
+
+@dataclass(frozen=True, slots=True)
+class WaggrModel:
+ adjacency: csc_matrix
+ denominator: np.ndarray
+
+ def __post_init__(self) -> None:
+ if self.adjacency.ndim != 2:
+ raise ValueError("WAGGR adjacency matrix must be two-dimensional")
+ if self.denominator.ndim != 1:
+ raise ValueError("WAGGR denominator must be one-dimensional")
+ if self.adjacency.shape[1] != len(self.denominator):
+ raise ValueError("WAGGR adjacency and denominator must be aligned")
+ if np.any(self.denominator <= 0) or not np.isfinite(self.denominator).all():
+ raise ValueError("WAGGR denominators must be finite and positive")
+
+
+def build_waggr_model(network: PreparedNetwork) -> WaggrModel:
+ """Build a sparse target-by-source matrix for weighted aggregation."""
+ feature_positions = np.searchsorted(
+ network.matched_feature_index, network.edge_feature_index
+ )
+ if np.any(feature_positions >= len(network.matched_feature_index)) or np.any(
+ network.matched_feature_index[feature_positions] != network.edge_feature_index
+ ):
+ raise ValueError("Network edges do not align with matched features")
+ adjacency = csc_matrix(
+ (
+ np.asarray(network.edge_weight, dtype=np.float64),
+ (
+ np.asarray(feature_positions, dtype=np.int64),
+ np.asarray(network.edge_source_index, dtype=np.int64),
+ ),
+ ),
+ shape=(len(network.matched_feature_index), len(network.source_names)),
+ dtype=np.float64,
+ )
+ denominator = np.asarray(np.abs(adjacency).sum(axis=0)).reshape(-1)
+ denominator = np.asarray(denominator, dtype=np.float64)
+ denominator.setflags(write=False)
+ return WaggrModel(adjacency=adjacency, denominator=denominator)
+
+
+def score_waggr_block(
+ values: np.ndarray,
+ model: WaggrModel,
+ *,
+ mode: Literal["wmean", "wsum"],
+) -> np.ndarray:
+ """Score one normalized cell block with weighted sums or means."""
+ if mode not in ("wmean", "wsum"):
+ raise ValueError("mode must be 'wmean' or 'wsum'")
+ raw_values = np.asarray(values)
+ if raw_values.ndim != 2:
+ raise ValueError("WAGGR values must be two-dimensional")
+ if (
+ not np.issubdtype(raw_values.dtype, np.number)
+ or np.issubdtype(raw_values.dtype, np.complexfloating)
+ or not np.isfinite(raw_values).all()
+ ):
+ raise ValueError("WAGGR values must be finite and numeric")
+ matrix = np.asarray(raw_values, dtype=np.float64)
+ if matrix.shape[1] != model.adjacency.shape[0]:
+ raise ValueError("WAGGR values and adjacency features are not aligned")
+ scores = np.asarray(model.adjacency.T.dot(matrix.T).T, dtype=np.float64)
+ if mode == "wmean":
+ scores = scores / model.denominator.reshape(1, -1)
+ if scores.shape != (matrix.shape[0], model.adjacency.shape[1]):
+ raise ValueError("WAGGR output shape is invalid")
+ if not np.isfinite(scores).all():
+ raise ValueError("WAGGR produced non-finite scores")
+ return scores
diff --git a/scarf/features/genomic/__init__.py b/scarf/features/genomic/__init__.py
new file mode 100644
index 00000000..cb6c450e
--- /dev/null
+++ b/scarf/features/genomic/__init__.py
@@ -0,0 +1,53 @@
+from importlib import import_module as _import_module
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from .gff import GffReader as GffReader
+ from .intervals import (
+ binary_search as binary_search,
+ create_bed_from_coord_ids as create_bed_from_coord_ids,
+ get_feature_mappings as get_feature_mappings,
+ get_ranges as get_ranges,
+ )
+ from .melding import (
+ create_counts_mat as create_counts_mat,
+ coordinate_melding as coordinate_melding,
+ )
+
+__all__ = [
+ "GffReader",
+ "binary_search",
+ "coordinate_melding",
+ "create_bed_from_coord_ids",
+ "create_counts_mat",
+ "get_feature_mappings",
+ "get_ranges",
+]
+
+_LAZY_EXPORTS = {
+ "GffReader": (".gff", "GffReader"),
+ "binary_search": (".intervals", "binary_search"),
+ "coordinate_melding": (".melding", "coordinate_melding"),
+ "create_bed_from_coord_ids": (".intervals", "create_bed_from_coord_ids"),
+ "create_counts_mat": (".melding", "create_counts_mat"),
+ "get_feature_mappings": (".intervals", "get_feature_mappings"),
+ "get_ranges": (".intervals", "get_ranges"),
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+
+def __getattr__(name: str) -> Any:
+ export = _LAZY_EXPORTS.get(name)
+ if export is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ module_name, attribute_name = export
+ value = getattr(_import_module(module_name, __name__), attribute_name)
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()).union(_LAZY_EXPORTS))
diff --git a/scarf/features/genomic/gff.py b/scarf/features/genomic/gff.py
new file mode 100644
index 00000000..f41bee91
--- /dev/null
+++ b/scarf/features/genomic/gff.py
@@ -0,0 +1,127 @@
+import gzip
+from collections.abc import Callable
+from typing import Any
+
+import numpy as np
+import pandas as pd
+
+from ...utils.logging import logger
+from ...utils.progress import iter_progress
+
+__all__ = ["GffReader"]
+
+
+class GffReader:
+ """Reader for a GFF3 format file."""
+
+ def __init__(
+ self,
+ gff_fn: str,
+ up_offset: int = 1000,
+ down_offset: int = 500,
+ chunk_size: int = 100000,
+ ):
+ self.gffFn = gff_fn
+ self.header = self.fetch_header_lines()
+ self.nHeaderLines = len(self.header)
+ self.up = up_offset
+ self.down = down_offset
+ self.chunksize = chunk_size
+
+ def fetch_header_lines(self) -> list[str]:
+ """Fetch header lines starting with `#` from a GFF file."""
+ temp = []
+ if self.gffFn.endswith("gz"):
+ h = gzip.open(self.gffFn, "rt")
+ else:
+ h = open(self.gffFn)
+ for line in h:
+ if line[0] != "#":
+ break
+ temp.append(line.rstrip())
+ h.close()
+ return temp
+
+ def stream(self) -> pd.DataFrame:
+ """Stream the GFF file in chunks as pandas dataframes."""
+ stream = pd.read_csv(
+ self.gffFn,
+ skiprows=self.nHeaderLines,
+ chunksize=int(self.chunksize),
+ sep="\t",
+ header=None,
+ )
+ for df in stream:
+ yield df
+
+ def get_promoter(self, v: pd.Series) -> tuple[int, int]:
+ """Create strand-aware promoter coordinates."""
+ if v[6] == "+":
+ return max(0, v[3] - self.up), v[3] + self.down
+ if v[6] == "-":
+ return v[4] - 1 - self.down, v[4] + self.up
+ raise ValueError(f"ERROR: Unknown symbol for strand: {v[6]}")
+
+ def get_body(self, v: pd.Series) -> tuple[int, int]:
+ """Create strand-aware gene body and promoter coordinates."""
+ if v[6] == "+":
+ return max(v[3] - self.up, 0), v[4]
+ if v[6] == "-":
+ return v[3], v[4] + self.up
+ raise ValueError(f"ERROR: Unknown symbol for strand: {v[6]}")
+
+ @staticmethod
+ def get_ids_names(v: pd.Series) -> tuple[str | None, str | None]:
+ """Extract gene ID and gene name values from a GFF record."""
+ gid, name = None, None
+ for i in v[8].split(";"):
+ j, k = i.split("=")
+ if j == "gene_id":
+ gid = k
+ elif j == "gene_name":
+ name = k
+ return gid, name
+
+ @staticmethod
+ def d_apply(d: pd.DataFrame, func: Callable[[pd.Series], Any]) -> np.ndarray:
+ """Apply a function over dataframe rows and return an array."""
+ values = d.apply(func, axis=1)
+ return np.array(list(values.values))
+
+ def to_bed(
+ self,
+ out_bed_fn: str,
+ flavour: str = "body",
+ ) -> None:
+ """Convert gene annotations from GFF to a six-column BED file."""
+ bed = []
+ if flavour not in ["body", "promoter"]:
+ raise ValueError(
+ "ERROR: The value of flavour must be one of either 'body' or 'promoter'"
+ )
+ for df in iter_progress(
+ self.stream(),
+ desc="Reading gene annotations",
+ ):
+ df = df[df[2] == "gene"]
+ if flavour == "promoter":
+ coords = self.d_apply(df, self.get_promoter)
+ else:
+ coords = self.d_apply(df, self.get_body)
+
+ anno = self.d_apply(df, self.get_ids_names)
+ odf = pd.DataFrame(
+ {
+ 0: df[0].values,
+ 1: coords[:, 0],
+ 2: coords[:, 1],
+ 3: anno[:, 0],
+ 4: anno[:, 1],
+ 5: df[6].values,
+ }
+ )
+ bed.append(odf)
+ merged_bed = pd.concat(bed)
+ merged_bed.to_csv(out_bed_fn, sep="\t", header=False, index=False)
+ logger.info(f"{merged_bed.shape[0]} genes saved to BED file")
+ return None
diff --git a/scarf/features/genomic/intervals.py b/scarf/features/genomic/intervals.py
new file mode 100644
index 00000000..56f96329
--- /dev/null
+++ b/scarf/features/genomic/intervals.py
@@ -0,0 +1,171 @@
+from collections.abc import Iterable
+
+import numpy as np
+import pandas as pd
+from numba import jit
+from scipy.sparse import coo_matrix, csc_matrix
+
+from ...utils.logging import logger
+
+__all__ = [
+ "binary_search",
+ "create_bed_from_coord_ids",
+ "get_feature_mappings",
+ "get_ranges",
+]
+
+
+def create_bed_from_coord_ids(ids: Iterable[str]) -> pd.DataFrame:
+ """Create a three-column BED dataframe from `chrom:start-end` strings."""
+ ser = pd.Series(list(ids), dtype="object")
+ chrom_rest = ser.str.split(":", expand=True)
+ start_end = chrom_rest[1].str.split("-", expand=True)
+ df = pd.DataFrame(
+ {
+ 0: chrom_rest[0].to_numpy(),
+ 1: start_end[0].astype(np.int64).to_numpy(),
+ 2: start_end[1].astype(np.int64).to_numpy(),
+ }
+ )
+ return df.sort_values(by=[0, 1])
+
+
+@jit(nopython=True)
+def binary_search(ranges: np.ndarray, queries: np.ndarray) -> np.ndarray:
+ """Find overlapping range positions for sorted query intervals."""
+ max_len = (ranges[:, 1] - ranges[:, 0]).max()
+ n = queries.shape[0]
+ ret_val = np.full((n, 2), 0)
+ for i in range(n):
+ start = max(0, queries[i][0] - max_len)
+ end = queries[i][1]
+
+ lo = 0
+ hi = ranges.shape[0]
+ while lo < hi:
+ mid = (lo + hi) // 2
+ if ranges[mid][0] < start:
+ lo = mid + 1
+ else:
+ hi = mid
+ starts_after = lo
+
+ lo = 0
+ hi = ranges.shape[0]
+ while lo < hi:
+ mid = (lo + hi) // 2
+ if end < ranges[mid][0]:
+ hi = mid
+ else:
+ lo = mid + 1
+ ends_before = lo
+
+ if starts_after == ends_before:
+ ret_val[i][0] = -1
+ ret_val[i][1] = -1
+ else:
+ start = queries[i][0]
+ m_pos_s, m_pos_e = -1, -1
+ for j in range(starts_after, ends_before):
+ if start < ranges[j][1] and end > ranges[j][0]:
+ if m_pos_s == -1:
+ m_pos_s = j
+ m_pos_e = j + 1
+ ret_val[i][0] = m_pos_s
+ ret_val[i][1] = m_pos_e
+
+ return ret_val
+
+
+def get_ranges(df: pd.DataFrame, idx: np.ndarray) -> np.ndarray:
+ """Extract integer start and end positions from a BED dataframe."""
+ return np.asarray(df[[1, 2]][idx].values, dtype=int)
+
+
+def get_feature_mappings(
+ peaks_bed_df: pd.DataFrame,
+ features_bed_df: pd.DataFrame,
+) -> tuple[np.ndarray, np.ndarray, csc_matrix]:
+ """Build a sparse peak-to-feature interval overlap mapping."""
+ feats_ids: list[str] = []
+ feats_names: list[str] = []
+ id_counter: dict[str, int] = {}
+ map_peak_rows: list[int] = []
+ map_feat_cols: list[int] = []
+ n_no_match = 0
+ feat_col = 0
+ peak_chroms = set(peaks_bed_df[0].unique())
+
+ for chrom in features_bed_df[0].unique():
+ feats_chrom_idx = (features_bed_df[0] == chrom).values
+ chrom_names = features_bed_df[4][feats_chrom_idx].values
+ chrom_ids = features_bed_df[3][feats_chrom_idx].values
+
+ feats_names.extend(chrom_names)
+ for feature_id in chrom_ids:
+ if feature_id not in id_counter:
+ id_counter[feature_id] = 0
+ id_counter[feature_id] += 1
+ if id_counter[feature_id] > 1:
+ feature_id = feature_id + f"_{id_counter[feature_id]}"
+ feats_ids.append(feature_id)
+
+ if chrom not in peak_chroms:
+ logger.warning(f"Chromosome {chrom} not in the input peak coordinates")
+ n_no_match += len(chrom_ids)
+ feat_col += len(chrom_ids)
+ continue
+
+ peaks_chrom_idx = (peaks_bed_df[0] == chrom).values
+ match_indices = binary_search(
+ get_ranges(peaks_bed_df, peaks_chrom_idx),
+ get_ranges(features_bed_df, feats_chrom_idx),
+ ).astype(int)
+
+ peak_idx = np.array(peaks_bed_df.index[peaks_chrom_idx])
+ for match in match_indices:
+ if match[0] == -1:
+ assert match[1] == -1
+ n_no_match += 1
+ else:
+ peaks_for_feat = peak_idx[match[0] : match[1]]
+ map_peak_rows.extend(peaks_for_feat.tolist())
+ map_feat_cols.extend([feat_col] * peaks_for_feat.shape[0])
+ feat_col += 1
+
+ if len(feats_ids) == 0:
+ raise ValueError(
+ "ERROR: None of the features were found in the assay. Melding failed"
+ )
+ feats_ids_arr = np.array(feats_ids)
+ feats_names_arr = np.array(feats_names)
+ n_features = feats_ids_arr.shape[0]
+ if n_no_match == n_features:
+ raise ValueError(
+ "None of the provided features overlap with the peak coordinates"
+ )
+ if n_no_match:
+ logger.warning(
+ f"{n_no_match}/{n_features} features did not overlap with any peak"
+ )
+ logger.info(
+ f"Mapped {n_features - n_no_match}/{n_features} features to peak coordinates"
+ )
+ if len(set(feats_ids_arr)) != n_features:
+ raise ValueError(
+ "ERROR: encountered an unexpected error. Somehow the feature ids are not unique "
+ "despite our attempt to make them unique by appending a suffix. Please report this "
+ "bug on Github"
+ )
+ assert feats_ids_arr.shape[0] == feats_names_arr.shape[0]
+ mapping = coo_matrix(
+ (
+ np.ones(len(map_peak_rows), dtype=np.float64),
+ (
+ np.asarray(map_peak_rows, dtype=np.int64),
+ np.asarray(map_feat_cols, dtype=np.int64),
+ ),
+ ),
+ shape=(peaks_bed_df.shape[0], n_features),
+ ).tocsc()
+ return feats_ids_arr, feats_names_arr, mapping
diff --git a/scarf/features/genomic/melding.py b/scarf/features/genomic/melding.py
new file mode 100644
index 00000000..b28f7c08
--- /dev/null
+++ b/scarf/features/genomic/melding.py
@@ -0,0 +1,357 @@
+from collections.abc import Iterator
+
+import numpy as np
+import pandas as pd
+import zarr
+from scipy.sparse import coo_matrix, csc_matrix, csr_matrix, diags
+
+from ...assay import Assay
+from ...storage.layout import array_shard_rows
+from ...storage.schema import create_zarr_count_assay
+from ...storage.sharding import (
+ accumulate_sparse_to_shards,
+ sparse_matrix_bytes,
+ sparse_producer_peak_bytes,
+)
+from ...utils.arrays import array_digest
+from .intervals import create_bed_from_coord_ids, get_feature_mappings
+
+__all__ = ["create_counts_mat", "coordinate_melding"]
+
+
+def _source_working_bytes(
+ source_rows: int,
+ n_source_features: int,
+ source_itemsize: int,
+) -> int:
+ source_elements = max(0, int(source_rows)) * max(0, int(n_source_features))
+ return (
+ source_elements * (source_itemsize + 4 * np.dtype(np.float64).itemsize)
+ + (max(0, int(source_rows)) + 1) * np.dtype(np.int64).itemsize
+ )
+
+
+def _producer_reserve_bytes(
+ *,
+ source_rows: int,
+ shard_rows: int,
+ n_docs: int,
+ n_source_features: int,
+ n_target_features: int,
+ source_itemsize: int,
+ store_itemsize: int,
+ decode_bytes: int,
+) -> int:
+ buffered_rows = min(n_docs, max(0, int(source_rows)) + max(1, int(shard_rows)))
+ return (
+ sparse_producer_peak_bytes(
+ buffered_rows * n_target_features,
+ max(0, int(source_rows)) * n_target_features,
+ store_itemsize,
+ )
+ + _source_working_bytes(
+ source_rows,
+ n_source_features,
+ source_itemsize,
+ )
+ + max(0, int(decode_bytes))
+ )
+
+
+def _max_source_rows_for_budget(
+ *,
+ available_bytes: int,
+ shard_rows: int,
+ n_docs: int,
+ n_source_features: int,
+ n_target_features: int,
+ source_itemsize: int,
+ store_itemsize: int,
+ preferred_rows: int,
+ decode_bytes: int,
+) -> int:
+ preferred = max(1, min(int(preferred_rows), int(n_docs)))
+ if preferred <= 1:
+ return 1
+ if (
+ _producer_reserve_bytes(
+ source_rows=preferred,
+ shard_rows=shard_rows,
+ n_docs=n_docs,
+ n_source_features=n_source_features,
+ n_target_features=n_target_features,
+ source_itemsize=source_itemsize,
+ store_itemsize=store_itemsize,
+ decode_bytes=decode_bytes,
+ )
+ <= available_bytes
+ ):
+ return preferred
+
+ low = 1
+ high = preferred
+ while low < high:
+ mid = (low + high + 1) // 2
+ reserve = _producer_reserve_bytes(
+ source_rows=mid,
+ shard_rows=shard_rows,
+ n_docs=n_docs,
+ n_source_features=n_source_features,
+ n_target_features=n_target_features,
+ source_itemsize=source_itemsize,
+ store_itemsize=store_itemsize,
+ decode_bytes=decode_bytes,
+ )
+ if reserve <= available_bytes:
+ low = mid
+ else:
+ high = mid - 1
+ return max(1, low)
+
+
+def create_counts_mat(
+ assay: Assay,
+ store: zarr.Array,
+ mapping: csc_matrix,
+ scalar_coeff: float,
+ renormalization: bool,
+ *,
+ idf_cell_idx: np.ndarray | None = None,
+) -> None:
+ """Populate a melded count matrix in a Zarr array."""
+ n_docs = int(store.shape[0])
+ n_source_features = int(mapping.shape[0])
+ n_target_features = int(store.shape[1])
+ mapping_bytes = sparse_matrix_bytes(mapping)
+ source_itemsize = np.dtype(assay.rawData.dtype).itemsize
+ decode_bytes = assay.rawData._max_decode_bytes()
+ if idf_cell_idx is None:
+ selected_cells = None
+ selected_cell_count = n_docs
+ else:
+ raw_selected_cells = np.asarray(idf_cell_idx)
+ if raw_selected_cells.dtype == bool or not np.issubdtype(
+ raw_selected_cells.dtype,
+ np.integer,
+ ):
+ raise TypeError("idf_cell_idx must contain integer indices")
+ selected_cells = np.asarray(raw_selected_cells, dtype=np.int64)
+ if selected_cells.ndim != 1:
+ raise ValueError("idf_cell_idx must be one-dimensional")
+ if len(selected_cells) == 0:
+ raise ValueError("Gene-score IDF requires at least one selected cell")
+ if np.any(selected_cells < 0) or np.any(selected_cells >= n_docs):
+ raise IndexError("idf_cell_idx contains an out-of-range cell")
+ selected_cells = np.unique(selected_cells)
+ selected_cell_count = len(selected_cells)
+
+ n_term_per_doc = np.asarray(
+ assay.cells.fetch_all(assay.name + "_nCounts"),
+ dtype=np.float64,
+ )
+ n_term_per_doc[n_term_per_doc == 0] = 1
+ selected_mask: np.ndarray | None = None
+ if selected_cell_count == n_docs:
+ n_docs_per_term = np.asarray(
+ assay.feats.fetch_all("nCells"),
+ dtype=np.float64,
+ )
+ else:
+ assert selected_cells is not None
+ selected_mask = np.zeros(n_docs, dtype=bool)
+ selected_mask[selected_cells] = True
+ n_docs_per_term = np.zeros(n_source_features, dtype=np.int64)
+ feature_temporaries = n_source_features * np.dtype(np.int64).itemsize
+ static_bytes = (
+ mapping_bytes
+ + selected_cells.nbytes
+ + selected_mask.nbytes
+ + n_term_per_doc.nbytes
+ + n_docs_per_term.nbytes
+ + feature_temporaries
+ )
+ current_rows = min(int(assay.rawData.chunksize[0]), n_docs)
+ selected_copy_per_row = n_source_features * source_itemsize
+ stream_owned_per_row = n_source_features * source_itemsize
+ working_bytes_per_row = selected_copy_per_row + stream_owned_per_row
+ available_bytes = int(assay.resources.memoryBytes) - static_bytes - decode_bytes
+ if available_bytes < working_bytes_per_row:
+ required_bytes = static_bytes + decode_bytes + working_bytes_per_row
+ raise MemoryError(
+ "Gene-score document frequency needs about "
+ f"{required_bytes} bytes for one row, but the operation limit is "
+ f"{assay.resources.memoryBytes} bytes"
+ )
+ document_frequency_rows = min(
+ current_rows,
+ available_bytes // working_bytes_per_row,
+ )
+ document_frequency_data = assay.rawData._with_block_size(
+ document_frequency_rows
+ )
+ stream_resident_bytes = (
+ static_bytes + document_frequency_rows * selected_copy_per_row
+ )
+ row_offset = 0
+ for block_values in document_frequency_data._stream_blocks(
+ nthreads=assay.nthreads,
+ msg="Computing gene-score document frequency",
+ prefetch=1,
+ row_mask=None,
+ resident_bytes=stream_resident_bytes,
+ ):
+ row_stop = row_offset + block_values.shape[0]
+ block_mask = selected_mask[row_offset:row_stop]
+ if block_mask.any():
+ n_docs_per_term += np.count_nonzero(
+ block_values[block_mask],
+ axis=0,
+ )
+ row_offset = row_stop
+ if row_offset != n_docs:
+ raise RuntimeError(
+ f"Gene-score document-frequency stream produced {row_offset} rows; "
+ f"expected {n_docs}"
+ )
+ idf = np.log2(1 + (selected_cell_count / (n_docs_per_term + 1)))
+ del n_docs_per_term, selected_cells, selected_mask
+
+ shard_rows = array_shard_rows(store)
+ store_itemsize = np.dtype(store.dtype).itemsize
+ preferred_rows = min(int(assay.rawData.chunksize[0]), n_docs)
+ resident_bytes = mapping_bytes + n_term_per_doc.nbytes + idf.nbytes
+ available_bytes = max(1, int(assay.resources.memoryBytes) - resident_bytes)
+ # Leave headroom above the static producer estimate for sparse band bytes
+ # that accumulate in write_sparse_bands and for densifying one destination
+ # shard when it is flushed (_sparse_task_working_bytes).
+ dense_shard_bytes = max(1, int(shard_rows)) * n_target_features * store_itemsize
+ write_headroom = max(64 * 1024 * 1024, 4 * dense_shard_bytes)
+ producer_budget = max(1, available_bytes - write_headroom)
+ source_rows = _max_source_rows_for_budget(
+ available_bytes=producer_budget,
+ shard_rows=shard_rows,
+ n_docs=n_docs,
+ n_source_features=n_source_features,
+ n_target_features=n_target_features,
+ source_itemsize=source_itemsize,
+ store_itemsize=store_itemsize,
+ preferred_rows=preferred_rows,
+ decode_bytes=decode_bytes,
+ )
+ producer_reserve = _producer_reserve_bytes(
+ source_rows=source_rows,
+ shard_rows=shard_rows,
+ n_docs=n_docs,
+ n_source_features=n_source_features,
+ n_target_features=n_target_features,
+ source_itemsize=source_itemsize,
+ store_itemsize=store_itemsize,
+ decode_bytes=decode_bytes,
+ )
+ if producer_reserve > producer_budget:
+ raise MemoryError(
+ "Gene-score melding needs about "
+ f"{resident_bytes + producer_reserve + write_headroom} bytes, but "
+ f"the operation limit is {assay.resources.memoryBytes} bytes"
+ )
+
+ source_data = assay.rawData._with_block_size(source_rows)
+ stream_charged_bytes = (
+ source_rows * n_source_features * source_itemsize + decode_bytes
+ )
+ source_stream_resident = resident_bytes + max(
+ 0,
+ producer_reserve - stream_charged_bytes,
+ )
+
+ def block_stream() -> Iterator[coo_matrix]:
+ start = 0
+ for block_values in source_data._stream_blocks(
+ nthreads=1,
+ msg="Melding assay",
+ prefetch=1,
+ row_mask=None,
+ resident_bytes=source_stream_resident,
+ ):
+ row = 0
+ while row < block_values.shape[0]:
+ stop = min(row + source_rows, block_values.shape[0])
+ values = block_values[row:stop]
+ tf = values / n_term_per_doc[start : start + values.shape[0]].reshape(
+ -1, 1
+ )
+ tfidf = tf * idf
+ block = (csr_matrix(tfidf) @ mapping).tocsr()
+ if renormalization:
+ row_sums = np.asarray(block.sum(axis=1)).reshape(-1)
+ scale = np.zeros(row_sums.shape[0], dtype=np.float64)
+ nonzero = row_sums != 0
+ scale[nonzero] = scalar_coeff / row_sums[nonzero]
+ block = diags(scale) @ block
+ yield block.tocoo()
+ start += values.shape[0]
+ row = stop
+
+ accumulate_sparse_to_shards(
+ store,
+ block_stream(),
+ resources=assay.resources,
+ residentBytes=resident_bytes,
+ producerReserveBytes=producer_reserve,
+ )
+
+
+def coordinate_melding(
+ assay: Assay,
+ workspace: str | None,
+ feature_bed: pd.DataFrame,
+ new_assay_name: str,
+ peaks_col: str = "ids",
+ scalar_coeff: float = 1e5,
+ renormalization: bool = True,
+ peaks_coords: np.ndarray | None = None,
+ idf_cell_idx: np.ndarray | None = None,
+) -> None:
+ """Transfer coordinate-based assay values to overlapping external features."""
+ if peaks_coords is None:
+ peaks_coords = assay.feats.fetch_all(peaks_col)
+ peaks_bed = create_bed_from_coord_ids(peaks_coords)
+ feat_ids, feat_names, mapping = get_feature_mappings(peaks_bed, feature_bed)
+
+ from ...storage.stores import zarr_group_root
+ from ...storage.profiles import resolve_storage_profile
+
+ store_root = zarr_group_root(assay.z, mode="r+")
+ group = create_zarr_count_assay(
+ z=store_root,
+ assay_name=new_assay_name,
+ workspace=workspace,
+ n_cells=assay.rawData.shape[0],
+ feat_ids=feat_ids,
+ feat_names=feat_names,
+ dtype="float",
+ profile=resolve_storage_profile(store_root.store),
+ )
+
+ create_counts_mat(
+ assay=assay,
+ store=group,
+ mapping=mapping,
+ scalar_coeff=scalar_coeff,
+ renormalization=renormalization,
+ idf_cell_idx=idf_cell_idx,
+ )
+ selected_cells = (
+ np.arange(assay.rawData.shape[0], dtype=np.int64)
+ if idf_cell_idx is None
+ else np.unique(np.asarray(idf_cell_idx, dtype=np.int64))
+ )
+ assay_group_path = (
+ new_assay_name if workspace is None else f"{workspace}/{new_assay_name}"
+ )
+ assay_group = store_root[assay_group_path]
+ assay_group.attrs["idfCellIndexDigest"] = array_digest(selected_cells)
+ assay_group.attrs["idfCellCount"] = int(len(selected_cells))
+ assay_group.attrs["sourceAssay"] = assay.name
+ assay_group.attrs["tfDenominator"] = "total_counts"
+ return None
diff --git a/scarf/features/markers/__init__.py b/scarf/features/markers/__init__.py
new file mode 100644
index 00000000..b96accc2
--- /dev/null
+++ b/scarf/features/markers/__init__.py
@@ -0,0 +1,49 @@
+from importlib import import_module as _import_module
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from .batching import (
+ resolve_marker_gene_batch_size as resolve_marker_gene_batch_size,
+ )
+ from .rank import (
+ mannwhitneyu_from_ranks as mannwhitneyu_from_ranks,
+ sort_marker_results as sort_marker_results,
+ )
+ from .search import (
+ find_markers_by_rank as find_markers_by_rank,
+ find_markers_by_regression as find_markers_by_regression,
+ )
+
+__all__ = [
+ "find_markers_by_rank",
+ "find_markers_by_regression",
+ "mannwhitneyu_from_ranks",
+ "resolve_marker_gene_batch_size",
+ "sort_marker_results",
+]
+
+_LAZY_EXPORTS = {
+ "find_markers_by_rank": (".search", "find_markers_by_rank"),
+ "find_markers_by_regression": (".search", "find_markers_by_regression"),
+ "mannwhitneyu_from_ranks": (".rank", "mannwhitneyu_from_ranks"),
+ "resolve_marker_gene_batch_size": (".batching", "resolve_marker_gene_batch_size"),
+ "sort_marker_results": (".rank", "sort_marker_results"),
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+
+def __getattr__(name: str) -> Any:
+ export = _LAZY_EXPORTS.get(name)
+ if export is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ module_name, attribute_name = export
+ value = getattr(_import_module(module_name, __name__), attribute_name)
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()).union(_LAZY_EXPORTS))
diff --git a/scarf/features/markers/batching.py b/scarf/features/markers/batching.py
new file mode 100644
index 00000000..c2e8c63c
--- /dev/null
+++ b/scarf/features/markers/batching.py
@@ -0,0 +1,59 @@
+"""Compatibility surface for the pre-planner marker batching helpers."""
+
+import warnings
+from typing import Any
+
+from ...storage.budget import DEFAULT_READ_AHEAD_BLOCKS, ResourceBudget
+from ...storage.feature_stream import feature_column_chunk as _feature_column_chunk
+
+_MARKER_BYTES_PER_CELL_FEATURE = 32
+
+__all__ = ["feature_column_chunk", "resolve_marker_gene_batch_size"]
+
+
+def feature_column_chunk(assay: Any, n_features: int) -> int:
+ """Return the on-disk feature width of one stored chunk."""
+ # RNA feature-column streams (markers, HVG, pseudotime) prefer countsT
+ # when present; other assays keep cell-major batch sizing.
+ from ...assay import RNAassay
+
+ if isinstance(assay, RNAassay):
+ counts_t = getattr(assay, "rawDataT", None)
+ if counts_t is not None:
+ return _feature_column_chunk(counts_t, featureAxis=0)
+ backing = getattr(getattr(assay, "rawData", None), "_backing", None)
+ if backing is not None:
+ return _feature_column_chunk(backing, featureAxis=1)
+ return max(1, int(n_features))
+
+
+def resolve_marker_gene_batch_size(
+ *,
+ n_features: int,
+ n_cells: int,
+ column_chunk: int,
+ resources: ResourceBudget,
+) -> int:
+ """Choose a marker batch that fits the supplied memory limit.
+
+ Superseded by ``scarf.storage.feature_stream.plan_feature_stream``, which
+ reads the stored chunk geometry instead of assuming a fixed per-element
+ footprint. Kept so callers written against the earlier releases still work.
+ """
+ warnings.warn(
+ "resolve_marker_gene_batch_size is deprecated; call "
+ "scarf.storage.feature_stream.plan_feature_stream instead.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ n_features = max(1, int(n_features))
+ n_cells = max(1, int(n_cells))
+ column_chunk = max(1, int(column_chunk))
+ bytes_per_feature = n_cells * _MARKER_BYTES_PER_CELL_FEATURE
+ concurrency = min(resources.workers, DEFAULT_READ_AHEAD_BLOCKS)
+ budget_cap = resources.memoryBytes // (concurrency * bytes_per_feature)
+ if budget_cap < 1:
+ raise MemoryError(
+ "One marker feature batch does not fit the operation memory limit"
+ )
+ return min(column_chunk, n_features, budget_cap)
diff --git a/scarf/features/markers/correction.py b/scarf/features/markers/correction.py
new file mode 100644
index 00000000..5ff9a0ff
--- /dev/null
+++ b/scarf/features/markers/correction.py
@@ -0,0 +1,20 @@
+"""Private multiple-testing helpers for marker statistics."""
+
+import numpy as np
+from numpy.typing import NDArray
+
+__all__ = ["_bh_adjusted_pvalues"]
+
+
+def _bh_adjusted_pvalues(p_values: NDArray[np.floating]) -> NDArray[np.float64]:
+ """Return Benjamini-Hochberg adjusted p-values for finite entries."""
+ from statsmodels.stats.multitest import multipletests
+
+ values = np.asarray(p_values, dtype=np.float64)
+ adjusted = np.full(values.shape, np.nan, dtype=np.float64)
+ mask = np.isfinite(values)
+ if not np.any(mask):
+ return adjusted
+ _, corrected, _, _ = multipletests(values[mask], method="fdr_bh")
+ adjusted[mask] = corrected
+ return adjusted
diff --git a/scarf/features/markers/rank.py b/scarf/features/markers/rank.py
new file mode 100644
index 00000000..10cbee36
--- /dev/null
+++ b/scarf/features/markers/rank.py
@@ -0,0 +1,373 @@
+import numpy as np
+import pandas as pd
+from numba import njit, prange
+from scipy.stats import norm
+
+_MARKER_SORT_BY = ("score", "p_value")
+_MARKER_SORT_ASCENDING = (False, True)
+
+__all__ = [
+ "_batch_stats",
+ "_batch_stats_gene_major",
+ "_marker_stats_batch",
+ "_marker_stats_gene_major",
+ "gene_major_rank_scratch_bytes",
+ "mannwhitneyu_from_ranks",
+ "sort_marker_results",
+]
+
+
+def sort_marker_results(df: pd.DataFrame) -> pd.DataFrame:
+ frame = df.copy()
+ if "feature_index" not in frame.columns:
+ frame["feature_index"] = frame.index
+ sort_by = list(_MARKER_SORT_BY)
+ ascending = list(_MARKER_SORT_ASCENDING)
+ if "feature_name" in frame.columns:
+ sort_by.append("feature_name")
+ ascending.append(True)
+ else:
+ sort_by.append("feature_index")
+ ascending.append(True)
+ return frame.sort_values(by=sort_by, ascending=ascending)
+
+
+def mannwhitneyu_from_ranks(
+ ranked_df: pd.DataFrame,
+ groups: np.ndarray,
+ group_set: np.ndarray,
+) -> pd.DataFrame:
+ """Calculate two-sided Mann-Whitney U p-values from precomputed ranks."""
+ n_total = len(groups)
+ rank_sums = ranked_df.groupby(groups).sum().reindex(group_set)
+ group_counts = pd.Series(groups).value_counts().reindex(group_set).values
+ tie_corrections = np.zeros(ranked_df.shape[1])
+
+ for col_idx in range(ranked_df.shape[1]):
+ ranks = ranked_df.iloc[:, col_idx].values
+ _, counts = np.unique(ranks, return_counts=True)
+ tied_counts = counts[counts > 1].astype(np.float64)
+ if len(tied_counts) > 0:
+ tie_sum = np.sum(tied_counts**3 - tied_counts)
+ tie_corrections[col_idx] = tie_sum / (n_total * (n_total - 1))
+
+ pvals = {}
+ for idx, cluster in enumerate(group_set):
+ n1 = group_counts[idx]
+ n2 = n_total - n1
+ r1 = rank_sums.iloc[idx].values
+ u1 = r1 - (n1 * (n1 + 1)) / 2
+ mu_u = (n1 * n2) / 2
+ sigma_u = np.sqrt((n1 * n2 / 12) * ((n_total + 1) - tie_corrections))
+ delta = u1 - mu_u
+ z = np.zeros_like(delta, dtype=float)
+ np.divide(
+ delta - 0.5 * np.sign(delta),
+ sigma_u,
+ out=z,
+ where=sigma_u > 0,
+ )
+ pvals[cluster] = 2 * norm.sf(np.abs(z))
+
+ return pd.DataFrame(pvals, index=ranked_df.columns).T
+
+
+@njit(parallel=True, cache=True)
+def _marker_stats_batch(
+ data: np.ndarray,
+ int_indices: np.ndarray,
+ group_counts: np.ndarray,
+ n_total: float,
+) -> np.ndarray:
+ """Compute per-feature, per-group marker statistics for one batch."""
+ n_cells = data.shape[0]
+ n_genes = data.shape[1]
+ n_groups = group_counts.shape[0]
+ out = np.zeros((n_genes, n_groups, 8))
+ for g in prange(n_genes):
+ v = data[:, g]
+ order = np.argsort(v)
+ ar = np.empty(n_cells)
+ dr = np.empty(n_cells)
+ tie_sum = 0.0
+ i = 0
+ rank = 0.0
+ while i < n_cells:
+ j = i
+ vi = v[order[i]]
+ while j + 1 < n_cells and v[order[j + 1]] == vi:
+ j += 1
+ avg = (i + j + 2) / 2.0
+ rank += 1.0
+ t = j - i + 1
+ t_float = float(t)
+ for k in range(i, j + 1):
+ ar[order[k]] = avg
+ dr[order[k]] = rank
+ if t > 1:
+ tie_sum += t_float * t_float * t_float - t_float
+ i = j + 1
+ sum_g = np.zeros(n_groups)
+ nz_g = np.zeros(n_groups)
+ rank_g = np.zeros(n_groups)
+ drank_g = np.zeros(n_groups)
+ for c in range(n_cells):
+ grp = int_indices[c]
+ val = v[c]
+ sum_g[grp] += val
+ if val > 0:
+ nz_g[grp] += 1.0
+ rank_g[grp] += ar[c]
+ drank_g[grp] += dr[c]
+ total_sum = 0.0
+ total_nz = 0.0
+ for x in range(n_groups):
+ total_sum += sum_g[x]
+ total_nz += nz_g[x]
+ r_sum = 0.0
+ r_vals = np.empty(n_groups)
+ for x in range(n_groups):
+ cnt = group_counts[x]
+ r_vals[x] = drank_g[x] / cnt if cnt > 0 else 0.0
+ r_sum += r_vals[x]
+ if n_total > 1:
+ tie_corr = tie_sum / (n_total * (n_total - 1))
+ else:
+ tie_corr = 0.0
+ for x in range(n_groups):
+ cnt = group_counts[x]
+ rest = n_total - cnt
+ m = sum_g[x] / cnt if cnt > 0 else 0.0
+ m_o = (total_sum - sum_g[x]) / rest if rest > 0 else 0.0
+ e = nz_g[x] / cnt if cnt > 0 else 0.0
+ e_o = (total_nz - nz_g[x]) / rest if rest > 0 else 0.0
+ if m_o == 0.0:
+ fc = 0.0 if m == 0.0 else 100.1
+ else:
+ fc = m / m_o
+ r = r_vals[x] / r_sum if r_sum > 0 else 0.0
+ n1 = cnt
+ n2 = rest
+ r1 = rank_g[x]
+ u1 = r1 - (n1 * (n1 + 1.0)) / 2.0
+ mu = (n1 * n2) / 2.0
+ var = (n1 * n2 / 12.0) * ((n_total + 1.0) - tie_corr)
+ delta = u1 - mu
+ if var > 0.0:
+ z = (delta - 0.5 * np.sign(delta)) / np.sqrt(var)
+ else:
+ z = 0.0
+ if n1 > 0.0 and n2 > 0.0:
+ auc = u1 / (n1 * n2)
+ else:
+ auc = np.nan
+ out[g, x, 0] = r
+ out[g, x, 1] = m
+ out[g, x, 2] = m_o
+ out[g, x, 3] = e
+ out[g, x, 4] = e_o
+ out[g, x, 5] = fc
+ out[g, x, 6] = z
+ out[g, x, 7] = auc
+ return out
+
+
+@njit(parallel=True, cache=True)
+def _marker_stats_gene_major(
+ raw: np.ndarray,
+ scalar: np.ndarray,
+ size_factor: np.float32,
+ log_transform: bool,
+ int_indices: np.ndarray,
+ group_counts: np.ndarray,
+ n_total: np.float32,
+ destination_rows: np.ndarray,
+ out: np.ndarray,
+) -> None:
+ """Compute marker statistics from non-negative feature-major raw counts."""
+ n_genes = raw.shape[0]
+ n_cells = raw.shape[1]
+ n_groups = group_counts.shape[0]
+ for g in prange(n_genes):
+ nz_values = np.empty(n_cells, dtype=np.float32)
+ nz_cells = np.empty(n_cells, dtype=np.int64)
+ zero_g = np.zeros(n_groups)
+ sum_g = np.zeros(n_groups)
+ nz_g = np.zeros(n_groups)
+ rank_g = np.zeros(n_groups)
+ drank_g = np.zeros(n_groups)
+ n_nz = 0
+ for c in range(n_cells):
+ grp = int_indices[c]
+ value = (size_factor * np.float32(raw[g, c])) / scalar[c]
+ if log_transform:
+ value = np.log1p(value)
+ if value > 0.0:
+ nz_values[n_nz] = value
+ nz_cells[n_nz] = c
+ n_nz += 1
+ sum_g[grp] += value
+ nz_g[grp] += 1.0
+ else:
+ zero_g[grp] += 1.0
+
+ n_zero = n_cells - n_nz
+ tie_sum = 0.0
+ if n_zero > 0:
+ zero_rank = (n_zero + 1.0) / 2.0
+ zero_t = float(n_zero)
+ if n_zero > 1:
+ tie_sum += zero_t * zero_t * zero_t - zero_t
+ for x in range(n_groups):
+ rank_g[x] = zero_g[x] * zero_rank
+ drank_g[x] = zero_g[x]
+
+ order = np.argsort(nz_values[:n_nz])
+ i = 0
+ dense_rank = 1.0 if n_zero > 0 else 0.0
+ while i < n_nz:
+ j = i
+ value = nz_values[order[i]]
+ while j + 1 < n_nz and nz_values[order[j + 1]] == value:
+ j += 1
+ dense_rank += 1.0
+ average_rank = n_zero + (i + j + 2.0) / 2.0
+ tied = j - i + 1
+ tied_float = float(tied)
+ if tied > 1:
+ tie_sum += tied_float * tied_float * tied_float - tied_float
+ for k in range(i, j + 1):
+ cell = nz_cells[order[k]]
+ grp = int_indices[cell]
+ rank_g[grp] += average_rank
+ drank_g[grp] += dense_rank
+ i = j + 1
+
+ total_sum = 0.0
+ total_nz = 0.0
+ for x in range(n_groups):
+ total_sum += sum_g[x]
+ total_nz += nz_g[x]
+ rank_total = 0.0
+ rank_values = np.empty(n_groups)
+ for x in range(n_groups):
+ count = group_counts[x]
+ rank_values[x] = drank_g[x] / count if count > 0 else 0.0
+ rank_total += rank_values[x]
+ tie_correction = tie_sum / (n_total * (n_total - 1.0)) if n_total > 1 else 0.0
+
+ row = destination_rows[g]
+ for x in range(n_groups):
+ count = group_counts[x]
+ rest = n_total - count
+ mean = sum_g[x] / count if count > 0 else 0.0
+ mean_rest = (total_sum - sum_g[x]) / rest if rest > 0 else 0.0
+ fraction = nz_g[x] / count if count > 0 else 0.0
+ fraction_rest = (total_nz - nz_g[x]) / rest if rest > 0 else 0.0
+ if mean_rest == 0.0:
+ fold_change = 0.0 if mean == 0.0 else 100.1
+ else:
+ fold_change = mean / mean_rest
+ score = rank_values[x] / rank_total if rank_total > 0 else 0.0
+ n1 = count
+ n2 = rest
+ rank_sum = rank_g[x]
+ u1 = rank_sum - (n1 * (n1 + 1.0)) / 2.0
+ mu = (n1 * n2) / 2.0
+ variance = (n1 * n2 / 12.0) * ((n_total + 1.0) - tie_correction)
+ delta = u1 - mu
+ z = (
+ (delta - 0.5 * np.sign(delta)) / np.sqrt(variance)
+ if variance > 0.0
+ else 0.0
+ )
+ if n1 > 0.0 and n2 > 0.0:
+ auc = u1 / (n1 * n2)
+ else:
+ auc = np.nan
+ out[row, x, 0] = score
+ out[row, x, 1] = mean
+ out[row, x, 2] = mean_rest
+ out[row, x, 3] = fraction
+ out[row, x, 4] = fraction_rest
+ out[row, x, 5] = fold_change
+ out[row, x, 6] = z
+ out[row, x, 7] = auc
+
+
+def gene_major_rank_scratch_bytes(
+ *,
+ n_cells: int,
+ n_groups: int,
+ n_threads: int,
+) -> int:
+ """Return the worst-case scratch owned by active gene workers."""
+ cells = max(0, int(n_cells))
+ groups = max(0, int(n_groups))
+ threads = max(1, int(n_threads))
+ per_thread = (
+ cells * (np.dtype(np.float32).itemsize + 2 * np.dtype(np.int64).itemsize)
+ + groups * 6 * np.dtype(np.float64).itemsize
+ )
+ return threads * per_thread
+
+
+def _batch_stats_gene_major(
+ raw: np.ndarray,
+ scalar: np.ndarray,
+ size_factor: float,
+ log_transform: bool,
+ int_indices: np.ndarray,
+ group_counts: np.ndarray,
+ n_total: int,
+) -> np.ndarray:
+ """Run the feature-major kernel and convert z statistics to p-values."""
+ n_genes = int(raw.shape[0])
+ out = np.zeros((n_genes, len(group_counts), 8), dtype=np.float64)
+ _marker_stats_gene_major(
+ np.ascontiguousarray(raw),
+ np.asarray(scalar, dtype=np.float32),
+ np.float32(size_factor),
+ bool(log_transform),
+ np.asarray(int_indices, dtype=np.int64),
+ np.asarray(group_counts, dtype=np.float32),
+ np.float32(n_total),
+ np.arange(n_genes, dtype=np.int64),
+ out,
+ )
+ out[:, :, 6] = 2.0 * norm.sf(np.abs(out[:, :, 6]))
+ return out
+
+
+def _batch_stats(
+ data: np.ndarray,
+ int_indices: np.ndarray,
+ group_counts: np.ndarray,
+ n_total: int,
+ feature_labels: np.ndarray | None = None,
+) -> np.ndarray:
+ """Run the marker kernel and convert z statistics to p-values."""
+ values = np.asarray(data)
+ if values.ndim != 2:
+ raise ValueError("Marker data must be a two-dimensional array")
+ labels = (
+ np.arange(values.shape[1])
+ if feature_labels is None
+ else np.asarray(feature_labels)
+ )
+ if labels.shape != (values.shape[1],):
+ raise ValueError("Feature labels must match the marker data columns")
+ for column in range(values.shape[1]):
+ if not np.isfinite(values[:, column]).all():
+ raise ValueError(
+ f"Feature {labels[column]!r} contains non-finite normalized values"
+ )
+ kernel_dtype = np.float32 if values.dtype == np.float32 else np.float64
+ out = _marker_stats_batch(
+ np.ascontiguousarray(values, dtype=kernel_dtype),
+ int_indices,
+ group_counts.astype(np.float32),
+ np.float32(n_total),
+ )
+ out[:, :, 6] = 2.0 * norm.sf(np.abs(out[:, :, 6]))
+ return np.asarray(out)
diff --git a/scarf/features/markers/regression.py b/scarf/features/markers/regression.py
new file mode 100644
index 00000000..47e28abb
--- /dev/null
+++ b/scarf/features/markers/regression.py
@@ -0,0 +1,134 @@
+from typing import cast
+
+import numpy as np
+from numba import njit, prange
+from scipy.stats import linregress, t as student_t
+
+_LINREGRESS_TINY = 1.0e-20
+_REG_OK = 0
+_REG_SENTINEL = 1
+_REG_NONFINITE = 2
+
+__all__ = [
+ "_REG_NONFINITE",
+ "_REG_OK",
+ "_REG_SENTINEL",
+ "_regression_batch_results",
+ "_regression_p_values",
+ "_regression_r_batch",
+]
+
+
+@njit(parallel=True, cache=True)
+def _regression_r_batch(
+ data: np.ndarray,
+ x_centered: np.ndarray,
+ ssxm: float,
+ min_cells: int,
+ eps: float,
+) -> tuple[np.ndarray, np.ndarray]:
+ """Calculate Pearson r per feature and return per-feature status codes."""
+ n_cells = data.shape[0]
+ n_genes = data.shape[1]
+ r_out = np.empty(n_genes, dtype=np.float64)
+ status = np.empty(n_genes, dtype=np.int8)
+ inv_n = 1.0 / n_cells
+ for g in prange(n_genes):
+ v = data[:, g]
+ finite = True
+ nz = 0
+ vmin = v[0]
+ vmax = v[0]
+ y_sum = 0.0
+ for c in range(n_cells):
+ val = v[c]
+ if not np.isfinite(val):
+ finite = False
+ break
+ y_sum += val
+ if val > 0.0:
+ nz += 1
+ if val < vmin:
+ vmin = val
+ if val > vmax:
+ vmax = val
+ if not finite:
+ r_out[g] = 0.0
+ status[g] = _REG_NONFINITE
+ continue
+ if nz < min_cells or (vmax - vmin) <= eps:
+ r_out[g] = 0.0
+ status[g] = _REG_SENTINEL
+ continue
+ y_mean = y_sum * inv_n
+ ssym = 0.0
+ ssxym = 0.0
+ for c in range(n_cells):
+ yd = v[c] - y_mean
+ ssym += yd * yd
+ ssxym += x_centered[c] * yd
+ ssym *= inv_n
+ ssxym *= inv_n
+ if ssxm == 0.0 or ssym == 0.0:
+ r_out[g] = 0.0
+ status[g] = _REG_SENTINEL
+ continue
+ r = ssxym / np.sqrt(ssxm * ssym)
+ if r > 1.0:
+ r = 1.0
+ elif r < -1.0:
+ r = -1.0
+ r_out[g] = r
+ status[g] = _REG_OK
+ return r_out, status
+
+
+def _regression_p_values(r: np.ndarray, n_cells: int) -> np.ndarray:
+ """Calculate two-sided Student-t p-values matching `linregress`."""
+ df = float(n_cells - 2)
+ denom = (1.0 - r + _LINREGRESS_TINY) * (1.0 + r + _LINREGRESS_TINY)
+ t_stat = r * np.sqrt(df / denom)
+ return cast(np.ndarray, 2.0 * student_t.sf(np.abs(t_stat), df))
+
+
+def _regression_batch_results(
+ data: np.ndarray,
+ x_centered: np.ndarray,
+ ssxm: float,
+ regressor: np.ndarray,
+ min_cells: int,
+ feature_labels: np.ndarray,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ """Calculate r, p, and status for one feature batch."""
+ n_cells = data.shape[0]
+ eps = float(np.finfo(float).eps)
+ if n_cells == 2:
+ r_vals = np.empty(data.shape[1], dtype=np.float64)
+ p_vals = np.full(data.shape[1], np.nan, dtype=np.float64)
+ status = np.full(data.shape[1], _REG_SENTINEL, dtype=np.int8)
+ for g in range(data.shape[1]):
+ v = data[:, g]
+ if not np.isfinite(v).all():
+ raise ValueError(
+ f"Feature {feature_labels[g]!r} contains non-finite "
+ "normalized values"
+ )
+ if (v > 0).sum() >= min_cells and np.ptp(v) > eps:
+ lin_obj = linregress(regressor, v)
+ r_vals[g] = float(lin_obj.rvalue)
+ else:
+ r_vals[g] = 0.0
+ return r_vals, p_vals, status
+
+ r_vals, status = _regression_r_batch(data, x_centered, ssxm, int(min_cells), eps)
+ bad = np.flatnonzero(status == _REG_NONFINITE)
+ if bad.size:
+ raise ValueError(
+ f"Feature {feature_labels[bad[0]]!r} contains non-finite normalized values"
+ )
+ p_vals = np.full(data.shape[1], np.nan, dtype=np.float64)
+ ok = status == _REG_OK
+ if np.any(ok):
+ p_vals[ok] = _regression_p_values(r_vals[ok], n_cells)
+ r_vals = np.where(ok, r_vals, 0.0)
+ return r_vals, p_vals, status
diff --git a/scarf/features/markers/search.py b/scarf/features/markers/search.py
new file mode 100644
index 00000000..78b919b6
--- /dev/null
+++ b/scarf/features/markers/search.py
@@ -0,0 +1,325 @@
+import time
+from typing import Any
+
+import numba
+import numpy as np
+import pandas as pd
+from numba import set_num_threads
+from scipy.special import ndtr
+
+from ...assay import Assay, RNAassay, lib_size_feature_stream_eligible
+from ...storage.feature_stream import plan_feature_stream
+from ...utils.logging import logger
+from ...utils.numba import restore_numba_threads
+from ...utils.process import process_rss_mb
+from .correction import _bh_adjusted_pvalues
+from .rank import (
+ _batch_stats,
+ _marker_stats_gene_major,
+ gene_major_rank_scratch_bytes,
+ sort_marker_results,
+)
+from .regression import (
+ _REG_NONFINITE,
+ _REG_OK,
+ _REG_SENTINEL,
+ _regression_batch_results,
+)
+from .table import MARKER_STAT_COLUMNS
+
+__all__ = ["find_markers_by_rank", "find_markers_by_regression"]
+
+_KERNEL_STAT_COLUMNS = (
+ "score",
+ "mean",
+ "mean_rest",
+ "frac_exp",
+ "frac_exp_rest",
+ "fold_change",
+ "p_value",
+ "auc",
+)
+
+
+def _validate_rank_marker_groups(group_counts: np.ndarray, n_total: int) -> None:
+ populated = np.asarray(group_counts, dtype=np.float64)
+ if populated.size < 2:
+ raise ValueError("Rank marker search requires at least two populated groups")
+ if np.any(populated < 2):
+ raise ValueError(
+ "Rank marker search requires at least two cells in every group"
+ )
+ reference = n_total - populated
+ if np.any(reference < 2):
+ raise ValueError(
+ "Rank marker search requires at least two cells in every "
+ "one-versus-rest reference complement"
+ )
+
+
+@restore_numba_threads
+def find_markers_by_rank(
+ assay: Assay,
+ group_key: str,
+ cell_key: str,
+ feat_key: str,
+ batch_size: int | None = None,
+ n_threads: int = 1,
+ **norm_params: Any,
+) -> dict[Any, pd.DataFrame]:
+ """Identify marker features for groups with rank-based statistics."""
+ groups = assay.cells.fetch(group_key, cell_key)
+ group_set = np.array(sorted(set(groups)))
+ n_groups = len(group_set)
+ idx_map = dict(zip(group_set, range(n_groups)))
+ int_indices = np.asarray([idx_map[x] for x in groups], dtype=np.int64)
+ out_cols = ["feature_index", *MARKER_STAT_COLUMNS]
+ results: dict[Any, pd.DataFrame] = {}
+ worker_limit = getattr(
+ getattr(assay, "resources", None),
+ "workers",
+ n_threads,
+ )
+ set_num_threads(
+ min(
+ max(1, n_threads),
+ max(1, int(worker_limit)),
+ numba.config.NUMBA_NUM_THREADS,
+ )
+ )
+ group_counts = pd.Series(groups).value_counts().reindex(group_set).values
+ n_total = len(groups)
+ _validate_rank_marker_groups(group_counts, n_total)
+
+ renormalize_subset = bool(norm_params.get("renormalize_subset", False))
+ log_transform = bool(norm_params.get("log_transform", False))
+ use_fast = lib_size_feature_stream_eligible(
+ assay,
+ renormalize_subset=renormalize_subset,
+ )
+ if use_fast and isinstance(assay, RNAassay):
+ raw_source, _, _ = assay._raw_feature_stream_source()
+ use_fast = np.issubdtype(raw_source.dtype, np.unsignedinteger)
+
+ if use_fast:
+ if not isinstance(assay, RNAassay):
+ raise TypeError(
+ "Fast raw-count marker search requires an RNAassay instance"
+ )
+ cell_idx = assay.cells.active_index(cell_key)
+ feat_idx = assay.feats.active_index(feat_key)
+ scalar = assay.cells.fetch_all(assay.name + "_nCounts")[cell_idx]
+ sf = assay.sf
+ if sf is None:
+ raise ValueError("RNA library-size normalization requires a size factor")
+ scalar_values = np.asarray(scalar, dtype=np.float32)
+ scalar_values[scalar_values == 0] = 1
+ group_counts32 = np.asarray(group_counts, dtype=np.float32)
+ stats_matrix = np.zeros(
+ (len(feat_idx), n_groups, 8),
+ dtype=np.float64,
+ )
+ raw_source, feature_axis, cell_axis = assay._raw_feature_stream_source()
+ active_threads = numba.get_num_threads()
+ resident_bytes = (
+ scalar_values.nbytes
+ + int_indices.nbytes
+ + group_counts32.nbytes
+ + stats_matrix.nbytes
+ + gene_major_rank_scratch_bytes(
+ n_cells=len(cell_idx),
+ n_groups=n_groups,
+ n_threads=active_threads,
+ )
+ )
+ orientation_buffers = 1 if feature_axis == 0 else 2
+ raw_itemsize = max(1, int(np.dtype(raw_source.dtype).itemsize))
+ plan = plan_feature_stream(
+ raw_source,
+ featureAxis=feature_axis,
+ cellAxis=cell_axis,
+ featureIndices=feat_idx,
+ cellIndices=cell_idx,
+ resources=assay.resources,
+ residentBytes=resident_bytes,
+ blockBytes=lambda width: max(
+ 1,
+ len(cell_idx) * width * raw_itemsize * orientation_buffers,
+ ),
+ requestedBatchSize=batch_size,
+ )
+ logger.debug(
+ f"Marker search plan: features={len(feat_idx)} groups={n_groups} "
+ f"blocks={len(plan.blocks)} readWorkers={plan.readWorkers} "
+ f"ioConcurrency={plan.ioConcurrency} numbaThreads={active_threads} "
+ f"repeatedDecodes={plan.repeatedDecodeCount}"
+ )
+ block_idx = 0
+ for block, raw, read_sec, source in assay.iter_raw_feature_major_blocks(
+ cell_idx=cell_idx,
+ plan=plan,
+ msg="Finding markers",
+ ):
+ block_idx += 1
+ compute_started = time.perf_counter()
+ cpu_started = time.process_time()
+ _marker_stats_gene_major(
+ raw,
+ scalar_values,
+ np.float32(sf),
+ log_transform,
+ int_indices,
+ group_counts32,
+ np.float32(n_total),
+ block.destinations,
+ stats_matrix,
+ )
+ compute_seconds = time.perf_counter() - compute_started
+ cpu_seconds = time.process_time() - cpu_started
+ effective_cores = (
+ cpu_seconds / compute_seconds if compute_seconds > 0 else 0.0
+ )
+ logger.debug(
+ f"Marker block {block_idx}/{len(plan.blocks)}: "
+ f"width={block.indices.size} read={read_sec:.1f}s ({source}) "
+ f"compute={compute_seconds:.1f}s cpu={cpu_seconds:.1f}s "
+ f"effectiveCores={effective_cores:.2f} "
+ f"rss={process_rss_mb():.0f} MiB"
+ )
+ del raw
+ z_values = np.asarray(stats_matrix[:, :, 6], dtype=np.float64)
+ stats_matrix[:, :, 6] = 2.0 * ndtr(-np.abs(z_values))
+ else:
+ batch_stats = []
+ iterator = iter(
+ assay.iter_normed_feature_wise(
+ cell_key=cell_key,
+ feat_key=feat_key,
+ batch_size=batch_size,
+ msg="Finding markers",
+ **norm_params,
+ )
+ )
+ while True:
+ try:
+ mat = next(iterator)
+ except StopIteration:
+ break
+ if isinstance(mat, pd.DataFrame):
+ values = mat.to_numpy()
+ feature_labels = np.asarray(mat.columns)
+ else:
+ feature_major, feature_labels = mat
+ values = np.asarray(feature_major).T
+ stats = _batch_stats(
+ values,
+ int_indices,
+ group_counts,
+ n_total,
+ feature_labels=np.asarray(feature_labels),
+ )
+ batch_stats.append(stats)
+ stats_matrix = np.vstack(batch_stats)
+ feat_index = assay.feats.active_index(feat_key)
+ pval_col = "p_value"
+ for n, i in enumerate(group_set):
+ kernel = pd.DataFrame(
+ stats_matrix[:, n, :],
+ columns=list(_KERNEL_STAT_COLUMNS),
+ index=feat_index,
+ )
+ adjusted = _bh_adjusted_pvalues(
+ kernel[pval_col].to_numpy(dtype=np.float64, copy=False)
+ )
+ df = kernel.copy()
+ df["p_value_adjusted"] = adjusted
+ df = df.loc[:, list(MARKER_STAT_COLUMNS)]
+ cols_to_round = [
+ col for col in df.columns if col not in {pval_col, "p_value_adjusted"}
+ ]
+ df.loc[:, cols_to_round] = df.loc[:, cols_to_round].round(5)
+ df["feature_index"] = df.index
+ results[i] = sort_marker_results(df)[out_cols]
+ return results
+
+
+@restore_numba_threads
+def find_markers_by_regression(
+ assay: Assay,
+ cell_key: str,
+ feat_key: str,
+ regressor: np.ndarray,
+ min_cells: int,
+ batch_size: int | None = None,
+ **norm_params: Any,
+) -> pd.DataFrame:
+ """Find features correlated with a continuous variable."""
+ regressor = np.asarray(regressor, dtype=np.float64)
+ if regressor.ndim != 1:
+ raise ValueError("regressor must be one-dimensional")
+ if not np.isfinite(regressor).all():
+ raise ValueError("regressor must contain only finite values")
+ if regressor.size < 2 or np.unique(regressor).size < 2:
+ raise ValueError("regressor must contain at least two distinct values")
+ if min_cells < 1:
+ raise ValueError("min_cells must be at least 1")
+
+ n_threads = getattr(assay, "nthreads", 1)
+ set_num_threads(min(max(1, int(n_threads)), numba.config.NUMBA_NUM_THREADS))
+ x_centered = regressor - regressor.mean()
+ ssxm = float(np.dot(x_centered, x_centered) / regressor.shape[0])
+
+ labels: list[Any] = []
+ r_parts: list[np.ndarray] = []
+ p_parts: list[np.ndarray] = []
+ status_parts: list[np.ndarray] = []
+ for feature_batch in assay.iter_normed_feature_wise(
+ cell_key=cell_key,
+ feat_key=feat_key,
+ batch_size=batch_size,
+ msg="Finding correlated features",
+ **norm_params,
+ ):
+ if not isinstance(feature_batch, pd.DataFrame):
+ raise TypeError("Expected normalized feature batches as DataFrames.")
+ if feature_batch.shape[0] != regressor.shape[0]:
+ raise ValueError(
+ "Regressor length does not match the number of selected cells"
+ )
+ data = np.ascontiguousarray(feature_batch.to_numpy(dtype=np.float64))
+ feat_labels = np.asarray(feature_batch.columns)
+ r_vals, p_vals, status = _regression_batch_results(
+ data,
+ x_centered,
+ ssxm,
+ regressor,
+ min_cells,
+ feat_labels,
+ )
+ labels.extend(feat_labels.tolist())
+ r_parts.append(r_vals)
+ p_parts.append(p_vals)
+ status_parts.append(status)
+
+ if not labels:
+ return pd.DataFrame(columns=["r_value", "p_value", "p_value_adjusted"])
+ r_values = np.concatenate(r_parts)
+ p_values = np.concatenate(p_parts)
+ status = np.concatenate(status_parts)
+ if np.any(status == _REG_NONFINITE):
+ raise ValueError("Regression results contain non-finite feature status")
+ adjusted = np.full(p_values.shape, np.nan, dtype=np.float64)
+ tested = status == _REG_OK
+ if np.any(tested):
+ adjusted[tested] = _bh_adjusted_pvalues(p_values[tested])
+ untested = status == _REG_SENTINEL
+ p_values = p_values.astype(np.float64, copy=True)
+ p_values[untested] = np.nan
+ return pd.DataFrame(
+ {
+ "r_value": r_values,
+ "p_value": p_values,
+ "p_value_adjusted": adjusted,
+ },
+ index=labels,
+ )
diff --git a/scarf/features/markers/table.py b/scarf/features/markers/table.py
new file mode 100644
index 00000000..97cd6223
--- /dev/null
+++ b/scarf/features/markers/table.py
@@ -0,0 +1,412 @@
+"""Readers for persisted marker statistics."""
+
+from collections.abc import Sequence
+from typing import Any
+
+import numpy as np
+import pandas as pd
+import zarr
+
+from .rank import sort_marker_results
+
+LEGACY_STAT_COLUMNS = (
+ "score",
+ "mean",
+ "mean_rest",
+ "frac_exp",
+ "frac_exp_rest",
+ "fold_change",
+ "p_value",
+)
+MARKER_STAT_COLUMNS = (
+ *LEGACY_STAT_COLUMNS,
+ "auc",
+ "p_value_adjusted",
+)
+MARKER_METHOD = "mannwhitneyu"
+MARKER_ALTERNATIVE = "two-sided"
+MARKER_TIE_CORRECTION = True
+MARKER_CONTINUITY_CORRECTION = True
+MARKER_ADJUSTMENT_METHOD = "fdr_bh"
+MARKER_ADJUSTMENT_SCOPE = "within_group_all_tested_features"
+_MARKER_INDEX_COLUMN = "feature_index"
+_MARKER_METADATA = {
+ "method": MARKER_METHOD,
+ "alternative": MARKER_ALTERNATIVE,
+ "tie_correction": MARKER_TIE_CORRECTION,
+ "continuity_correction": MARKER_CONTINUITY_CORRECTION,
+ "adjustment_method": MARKER_ADJUSTMENT_METHOD,
+ "adjustment_scope": MARKER_ADJUSTMENT_SCOPE,
+}
+
+__all__ = [
+ "LEGACY_STAT_COLUMNS",
+ "MARKER_STAT_COLUMNS",
+ "load_marker_table",
+ "read_legacy_marker_table",
+]
+
+
+def _empty_marker_frame(columns: tuple[str, ...]) -> pd.DataFrame:
+ return pd.DataFrame({name: pd.Series(dtype=object) for name in columns})
+
+
+def _array_values(group: zarr.Group, name: str) -> np.ndarray:
+ value = group[name]
+ if not isinstance(value, zarr.Array):
+ raise TypeError(f"Marker field {name!r} must be an array")
+ return np.asarray(value[:])
+
+
+def _stored_stat_columns(
+ slot_group: zarr.Group,
+ *,
+ required: bool,
+) -> list[str] | None:
+ stored = slot_group.attrs.get("stat_columns")
+ if stored is None:
+ if required:
+ raise ValueError("Canonical marker tables require stat_columns metadata")
+ return None
+ if isinstance(stored, str) or not isinstance(stored, Sequence):
+ raise ValueError("Marker stat_columns metadata must be a sequence of names")
+ columns: list[str] = []
+ for name in stored:
+ if not isinstance(name, str):
+ raise ValueError("Marker stat_columns metadata must contain only strings")
+ columns.append(name)
+ if len(columns) != len(set(columns)):
+ raise ValueError("Marker stat_columns metadata contains duplicate names")
+ unknown = set(columns).difference(MARKER_STAT_COLUMNS)
+ if unknown:
+ raise ValueError(
+ "Marker stat_columns metadata contains unknown columns: "
+ + ", ".join(sorted(unknown))
+ )
+ return columns
+
+
+def _validated_feature_indices(
+ values: np.ndarray,
+ *,
+ n_features: int,
+ require_integer_dtype: bool,
+ require_unique: bool = False,
+) -> np.ndarray:
+ raw = np.asarray(values)
+ if raw.ndim != 1:
+ raise ValueError("Marker feature_index must be one-dimensional")
+ if require_integer_dtype and raw.dtype.kind not in {"i", "u"}:
+ raise ValueError("Canonical marker feature_index must use an integer dtype")
+ try:
+ numeric = np.asarray(raw, dtype=np.float64)
+ except (TypeError, ValueError) as exc:
+ raise ValueError("Marker feature_index must contain integer values") from exc
+ if not np.isfinite(numeric).all() or not np.equal(numeric, np.floor(numeric)).all():
+ raise ValueError("Marker feature_index must contain finite integer values")
+ indices = numeric.astype(np.int64)
+ invalid = (indices < 0) | (indices >= n_features)
+ if invalid.any():
+ unresolved = ", ".join(str(value) for value in indices[invalid][:5])
+ raise ValueError(
+ "Marker feature_index contains unresolved or out-of-range values: "
+ + unresolved
+ )
+ if require_unique and np.unique(indices).size != indices.size:
+ raise ValueError("Canonical marker feature_index must contain unique values")
+ return indices
+
+
+def _display_frame(
+ df: pd.DataFrame,
+ *,
+ group_id: Any,
+ feature_names: np.ndarray,
+ out_stat_columns: tuple[str, ...],
+ require_integer_indices: bool = False,
+ require_unique_indices: bool = False,
+) -> pd.DataFrame:
+ frame = df.copy()
+ frame["group_id"] = group_id
+ if "feature_index" in frame.columns:
+ names = np.asarray(feature_names, dtype=object)
+ if names.ndim != 1:
+ raise ValueError("Feature names must be one-dimensional")
+ indices = _validated_feature_indices(
+ frame["feature_index"].to_numpy(copy=False),
+ n_features=len(names),
+ require_integer_dtype=require_integer_indices,
+ require_unique=require_unique_indices,
+ )
+ frame["feature_index"] = indices
+ frame["feature_name"] = names[indices]
+ ordered = ["group_id", "feature_name", "feature_index", *out_stat_columns]
+ for column in ordered:
+ if column not in frame.columns:
+ frame[column] = np.nan
+ return frame[ordered]
+
+
+def _resolve_compact_stat_columns(
+ slot_group: zarr.Group,
+ n_columns: int,
+) -> list[str]:
+ stored = _stored_stat_columns(slot_group, required=False)
+ if stored is not None:
+ if len(stored) != n_columns:
+ raise ValueError(
+ "Compact marker stats width does not match stored stat_columns"
+ )
+ return stored
+ if n_columns == len(LEGACY_STAT_COLUMNS):
+ return list(LEGACY_STAT_COLUMNS)
+ raise ValueError(
+ "Compact marker stats lack stat_columns and do not match a known layout"
+ )
+
+
+def _maybe_attach_adjusted_pvalues(frame: pd.DataFrame) -> pd.DataFrame:
+ if "p_value_adjusted" in frame.columns and frame["p_value_adjusted"].notna().any():
+ return frame
+ if "p_value" not in frame.columns or not frame["p_value"].notna().any():
+ if "p_value_adjusted" not in frame.columns:
+ frame = frame.copy()
+ frame["p_value_adjusted"] = np.nan
+ return frame
+ from .correction import _bh_adjusted_pvalues
+
+ frame = frame.copy()
+ frame["p_value_adjusted"] = _bh_adjusted_pvalues(
+ frame["p_value"].to_numpy(dtype=np.float64, copy=False)
+ )
+ return frame
+
+
+def read_legacy_marker_table(
+ slot_group: zarr.Group,
+ cluster_group: zarr.Group,
+ feature_names: np.ndarray,
+ *,
+ group_id: Any,
+ feature_ids: np.ndarray | None = None,
+) -> pd.DataFrame:
+ """Read master and unversioned marker layouts into a named table."""
+ legacy_index_and_stats = (_MARKER_INDEX_COLUMN, *LEGACY_STAT_COLUMNS)
+ out_cols = list(legacy_index_and_stats)
+ if "feature_index" in slot_group and "stats" in cluster_group:
+ feature_index = _array_values(slot_group, "feature_index")
+ stats = _array_values(cluster_group, "stats")
+ if stats.ndim != 2:
+ raise ValueError("Compact marker stats must be two-dimensional")
+ if stats.shape[0] != feature_index.shape[0]:
+ raise ValueError("Compact marker stats do not align with feature_index")
+ columns = _resolve_compact_stat_columns(slot_group, stats.shape[1])
+ df = pd.DataFrame(stats, columns=columns)
+ df["feature_index"] = feature_index
+ for column in LEGACY_STAT_COLUMNS:
+ if column not in df.columns:
+ df[column] = np.nan
+ display_columns = (
+ MARKER_STAT_COLUMNS
+ if {"auc", "p_value_adjusted"}.intersection(columns)
+ else LEGACY_STAT_COLUMNS
+ )
+ displayed = _display_frame(
+ df,
+ group_id=group_id,
+ feature_names=feature_names,
+ out_stat_columns=display_columns,
+ )
+ return sort_marker_results(displayed)
+
+ if "names" in cluster_group and "scores" in cluster_group:
+ stored_names = _array_values(cluster_group, "names").astype(str)
+ scores = np.asarray(_array_values(cluster_group, "scores"), dtype=np.float64)
+ if stored_names.ndim != 1 or scores.ndim != 1:
+ raise ValueError("Legacy marker names and scores must be one-dimensional")
+ if stored_names.shape != scores.shape:
+ raise ValueError("Legacy marker names and scores must align")
+ lookup_values = feature_names if feature_ids is None else feature_ids
+ lookup = {str(value): index for index, value in enumerate(lookup_values)}
+ for index, value in enumerate(feature_names):
+ lookup.setdefault(str(value), index)
+ resolved_indices = [lookup.get(value) for value in stored_names]
+ feature_index = pd.array(resolved_indices, dtype="Int64")
+ names = np.asarray(feature_names, dtype=object)
+ display_names = [
+ names[index] if index is not None else value
+ for value, index in zip(stored_names, resolved_indices, strict=True)
+ ]
+ frame = pd.DataFrame(
+ {
+ "group_id": group_id,
+ "feature_name": display_names,
+ "feature_index": feature_index,
+ "score": scores,
+ }
+ )
+ for column in LEGACY_STAT_COLUMNS:
+ if column not in frame.columns:
+ frame[column] = np.nan
+ ordered = [
+ "group_id",
+ "feature_name",
+ "feature_index",
+ *LEGACY_STAT_COLUMNS,
+ ]
+ return (
+ frame[ordered]
+ .sort_values(
+ ["score", "feature_name"],
+ ascending=[False, True],
+ )
+ .reset_index(drop=True)
+ )
+
+ available_cols = [col for col in out_cols if col in cluster_group]
+ if not available_cols:
+ return _display_frame(
+ _empty_marker_frame(tuple(out_cols)),
+ group_id=group_id,
+ feature_names=feature_names,
+ out_stat_columns=LEGACY_STAT_COLUMNS,
+ )
+ cols = [_array_values(cluster_group, column) for column in available_cols]
+ df = pd.DataFrame(dict(zip(available_cols, cols, strict=True)))
+ for column in LEGACY_STAT_COLUMNS:
+ if column not in df.columns:
+ df[column] = np.nan
+ if "feature_index" not in df.columns:
+ raise ValueError("Legacy per-column marker tables require feature_index")
+ return _display_frame(
+ df,
+ group_id=group_id,
+ feature_names=feature_names,
+ out_stat_columns=LEGACY_STAT_COLUMNS,
+ )
+
+
+def _read_canonical_marker_table(
+ slot_group: zarr.Group,
+ cluster_group: zarr.Group,
+ feature_names: np.ndarray,
+ *,
+ group_id: Any,
+) -> pd.DataFrame:
+ """Read canonical compact marker tables with validated metadata."""
+ if "feature_index" not in slot_group or "stats" not in cluster_group:
+ raise ValueError("Canonical marker tables require feature_index and stats")
+ for name, expected in _MARKER_METADATA.items():
+ if name not in slot_group.attrs:
+ raise ValueError(f"Canonical marker tables require {name} metadata")
+ value = slot_group.attrs[name]
+ if type(value) is not type(expected) or value != expected:
+ raise ValueError(f"Canonical marker metadata {name!r} must be {expected!r}")
+ columns = _stored_stat_columns(slot_group, required=True)
+ assert columns is not None
+ required = set(MARKER_STAT_COLUMNS)
+ if set(columns) != required:
+ raise ValueError(
+ "Canonical marker tables must store the complete named stat_columns"
+ )
+ for name in ("n_group", "n_reference"):
+ value = cluster_group.attrs.get(name)
+ if (
+ isinstance(value, bool)
+ or not isinstance(value, int | np.integer)
+ or int(value) < 2
+ ):
+ raise ValueError(
+ f"Canonical marker groups require integer {name} metadata >= 2"
+ )
+ feature_index = _array_values(slot_group, "feature_index")
+ stats = _array_values(cluster_group, "stats")
+ if stats.ndim != 2 or stats.shape[1] != len(columns):
+ raise ValueError("Canonical marker stats do not match stat_columns")
+ if stats.shape[0] != feature_index.shape[0]:
+ raise ValueError("Canonical marker stats do not align with feature_index")
+ if stats.shape[0] == 0:
+ raise ValueError("Canonical marker groups must contain marker rows")
+ if stats.dtype.kind != "f":
+ raise ValueError("Canonical marker stats must use a floating dtype")
+ adjusted = stats[:, columns.index("p_value_adjusted")]
+ if not np.isfinite(adjusted).all():
+ raise ValueError("Canonical marker p_value_adjusted values must all be finite")
+ if not np.isfinite(stats).all():
+ raise ValueError("Canonical marker statistics must all be finite")
+ df = pd.DataFrame(stats, columns=columns)
+ df["feature_index"] = feature_index
+ displayed = _display_frame(
+ df,
+ group_id=group_id,
+ feature_names=feature_names,
+ out_stat_columns=MARKER_STAT_COLUMNS,
+ require_integer_indices=True,
+ require_unique_indices=True,
+ )
+ return sort_marker_results(displayed)
+
+
+def _validate_marker_slot(
+ slot_group: zarr.Group,
+ feature_names: np.ndarray,
+ *,
+ expected_group_cell_counts: dict[str, tuple[int, int]] | None = None,
+) -> None:
+ group_names = sorted(slot_group.group_keys())
+ if not group_names:
+ raise ValueError("Canonical marker tables must contain populated groups")
+ if expected_group_cell_counts is not None and set(group_names) != set(
+ expected_group_cell_counts
+ ):
+ raise ValueError("Canonical marker groups do not match the requested groups")
+ for group_name in group_names:
+ cluster_group = slot_group[group_name]
+ if not isinstance(cluster_group, zarr.Group):
+ raise TypeError(f"Marker group {group_name!r} must be a group")
+ _read_canonical_marker_table(
+ slot_group,
+ cluster_group,
+ feature_names,
+ group_id=group_name,
+ )
+ if expected_group_cell_counts is None:
+ continue
+ expected_group, expected_reference = expected_group_cell_counts[group_name]
+ if (
+ cluster_group.attrs.get("n_group") != expected_group
+ or cluster_group.attrs.get("n_reference") != expected_reference
+ ):
+ raise ValueError(
+ f"Canonical marker group {group_name!r} has stale cell counts"
+ )
+
+
+def load_marker_table(
+ slot_group: zarr.Group,
+ cluster_group: zarr.Group,
+ feature_names: np.ndarray,
+ *,
+ group_id: Any,
+ feature_ids: np.ndarray | None = None,
+) -> pd.DataFrame:
+ """Dispatch marker reading by structural layout."""
+ canonical_metadata = any(
+ name in slot_group.attrs for name in _MARKER_METADATA
+ ) or any(name in cluster_group.attrs for name in ("n_group", "n_reference"))
+ if canonical_metadata:
+ return _read_canonical_marker_table(
+ slot_group,
+ cluster_group,
+ feature_names,
+ group_id=group_id,
+ )
+
+ frame = read_legacy_marker_table(
+ slot_group,
+ cluster_group,
+ feature_names,
+ group_id=group_id,
+ feature_ids=feature_ids,
+ )
+ return _maybe_attach_adjusted_pvalues(frame)
diff --git a/scarf/features/scoring.py b/scarf/features/scoring.py
new file mode 100644
index 00000000..e10ea0bd
--- /dev/null
+++ b/scarf/features/scoring.py
@@ -0,0 +1,51 @@
+from typing import Any
+
+import numpy as np
+import pandas as pd
+
+__all__ = ["binned_sampling"]
+
+
+def binned_sampling(
+ values: pd.Series,
+ feature_list: list[str],
+ ctrl_size: int,
+ n_bins: int,
+ rand_seed: int,
+) -> list[str]:
+ """Score a set of genes [Satija15]_. The score is the average expression of
+ a set of genes subtracted with the average expression of a reference set of
+ genes. The reference set is randomly sampled from the `gene_pool` for each
+ binned expression value.
+
+ This reproduces the approach in Seurat [Satija15]_ and has been implemented
+ for Scanpy by Davide Cittaro.
+
+ This function is adapted from Scanpy's `score_genes`.
+
+ Args:
+ values: The values for the features.
+ feature_list: The list of features to use for score calculation.
+ ctrl_size: Number of reference features to be sampled from each bin.
+ n_bins: Number of bins for sampling.
+ rand_seed: The seed to use for the random number generation.
+
+ Returns:
+ A list of sampled features.
+ """
+ n_items = int(np.round(len(values) / (n_bins - 1)))
+ feature_set = set(feature_list)
+ obs_cut: pd.Series = values.fillna(0).rank(method="min").divide(n_items).astype(int)
+
+ control_genes: set[Any] = set()
+ for cut in np.unique(obs_cut[list(feature_set)]):
+ sub_obs = obs_cut[obs_cut == cut]
+ if len(sub_obs) == 0:
+ continue
+ if len(sub_obs) < ctrl_size:
+ sample_size = len(sub_obs)
+ else:
+ sample_size = ctrl_size
+ r_genes = sub_obs.sample(n=sample_size, random_state=rand_seed).index
+ control_genes.update(set(r_genes))
+ return list(control_genes - feature_set)
diff --git a/scarf/features/variability.py b/scarf/features/variability.py
new file mode 100644
index 00000000..7ef9ddd0
--- /dev/null
+++ b/scarf/features/variability.py
@@ -0,0 +1,285 @@
+import re
+from typing import Any, Literal
+
+import numpy as np
+import pandas as pd
+
+from ..utils.logging import logger
+
+__all__ = [
+ "DEFAULT_HVG_BLACKLIST",
+ "HVG_UBIQUITOUS_SLACK",
+ "fit_lowess",
+ "select_highly_variable_features",
+]
+
+_ADAPTIVE_MIN_BIN_SIZE = 25
+_ADAPTIVE_ANCHOR_QUANTILE = 0.1
+
+# Case-insensitive via uppercasing in select_highly_variable_features / MetaData.grep.
+DEFAULT_HVG_BLACKLIST = (
+ "^MT-|^RPS|^RPL|^MRPS|^MRPL|^CCN|^HLA-|^H2-|^HIST|"
+ "^XIST$|^DDX3Y$|^USP9Y$|^EIF1AY$|^KDM5D$|^SRY$|^ZFY$|^UTY$|^TMSB4Y$|^NLGN4Y$"
+)
+HVG_UBIQUITOUS_SLACK = 20
+
+
+def _fit_lowess_adaptive(
+ a: np.ndarray,
+ b: np.ndarray,
+ n_bins: int,
+ lowess_frac: float,
+) -> np.ndarray:
+ from statsmodels.nonparametric.smoothers_lowess import lowess
+
+ means = np.asarray(a, dtype=float)
+ variances = np.asarray(b, dtype=float)
+ if means.ndim != 1 or variances.ndim != 1 or means.shape != variances.shape:
+ raise ValueError("LOWESS inputs must be one-dimensional arrays of equal length")
+ if isinstance(n_bins, (bool, np.bool_)) or not isinstance(
+ n_bins,
+ (int, np.integer),
+ ):
+ raise TypeError("n_bins must be an integer")
+ if n_bins < 1:
+ raise ValueError("n_bins must be greater than 0")
+ if isinstance(lowess_frac, (bool, np.bool_)) or not isinstance(
+ lowess_frac,
+ (int, float, np.integer, np.floating),
+ ):
+ raise TypeError("lowess_frac must be numeric")
+ if not np.isfinite(lowess_frac) or not 0 <= lowess_frac <= 1:
+ raise ValueError("lowess_frac must be between 0 and 1")
+
+ corrected = np.zeros(means.shape, dtype=float)
+ valid = np.isfinite(means) & np.isfinite(variances) & (means > 0) & (variances > 0)
+ if not valid.any():
+ return corrected
+
+ log_means = np.log(means[valid])
+ log_variances = np.log(variances[valid])
+ order = np.argsort(log_means, kind="stable")
+ sorted_means = log_means[order]
+ sorted_variances = log_variances[order]
+
+ bin_slices: list[slice] = []
+ bins_left = max(1, min(n_bins, len(order) // _ADAPTIVE_MIN_BIN_SIZE))
+ start = 0
+ while start < len(order):
+ remaining = len(order) - start
+ if bins_left == 1:
+ end = len(order)
+ else:
+ target_size = (remaining + bins_left - 1) // bins_left
+ end = start + target_size
+ while end < len(order) and sorted_means[end] == sorted_means[end - 1]:
+ end += 1
+ bins_after = min(
+ bins_left - 1,
+ (len(order) - end) // _ADAPTIVE_MIN_BIN_SIZE,
+ )
+ if bins_after == 0:
+ end = len(order)
+ bin_slices.append(slice(start, end))
+ start = end
+ bins_left = bins_after
+
+ anchor_means = np.fromiter(
+ (np.median(sorted_means[indices]) for indices in bin_slices),
+ dtype=float,
+ count=len(bin_slices),
+ )
+ anchor_variances = np.fromiter(
+ (
+ np.quantile(
+ sorted_variances[indices],
+ _ADAPTIVE_ANCHOR_QUANTILE,
+ )
+ for indices in bin_slices
+ ),
+ dtype=float,
+ count=len(bin_slices),
+ )
+
+ if len(anchor_means) == 1:
+ correction = np.full(log_means.shape, anchor_variances[0], dtype=float)
+ else:
+ fitted = np.asarray(
+ lowess(
+ anchor_variances,
+ anchor_means,
+ return_sorted=False,
+ frac=lowess_frac,
+ it=100,
+ ),
+ dtype=float,
+ )
+ if fitted.shape != anchor_means.shape or not np.all(np.isfinite(fitted)):
+ raise ValueError("LOWESS returned invalid adaptive trend values")
+ correction = np.interp(log_means, anchor_means, fitted)
+
+ corrected[valid] = np.exp(log_variances - correction)
+ return corrected
+
+
+def fit_lowess(
+ a: np.ndarray,
+ b: np.ndarray,
+ n_bins: int,
+ lowess_frac: float,
+ *,
+ bin_strategy: Literal["fixed", "adaptive"] = "adaptive",
+) -> np.ndarray:
+ """Fit a LOWESS curve and return corrected variance estimates."""
+ if bin_strategy == "adaptive":
+ return _fit_lowess_adaptive(a, b, n_bins, lowess_frac)
+ if bin_strategy != "fixed":
+ raise ValueError("bin_strategy must be either 'fixed' or 'adaptive'")
+
+ from statsmodels.nonparametric.smoothers_lowess import lowess
+
+ stats = pd.DataFrame({"a": a, "b": b}).apply(np.log)
+ bin_edges = np.histogram(stats.a, bins=n_bins)[1]
+ bin_edges[-1] += 0.1
+ bin_idx: list[list[Any]] = []
+ for index in range(n_bins):
+ idx = pd.Series(
+ (stats.a >= bin_edges[index]) & (stats.a < bin_edges[index + 1])
+ )
+ if sum(idx) > 0:
+ bin_idx.append(list(idx[idx].index))
+ bin_vals: list[list[float]] = []
+ for idx in bin_idx:
+ temp_stat = stats.reindex(idx)
+ temp_gene = temp_stat.idxmin().b
+ bin_vals.append([temp_stat.b[temp_gene], temp_stat.a[temp_gene]])
+ bin_array = np.array(bin_vals).T
+ bin_cor_fac = lowess(
+ bin_array[0],
+ bin_array[1],
+ return_sorted=False,
+ frac=lowess_frac,
+ it=100,
+ ).T
+ fixed_var: dict[Any, float] = {}
+ for correction, indices in zip(bin_cor_fac, bin_idx):
+ for idx in indices:
+ fixed_var[idx] = np.e ** (stats.b[idx] - correction)
+ return np.array([fixed_var[index] for index in range(len(a))])
+
+
+def _bounded(
+ values: np.ndarray,
+ lower: float,
+ upper: float,
+ *,
+ keep_bounds: bool,
+) -> np.ndarray:
+ if keep_bounds:
+ return (values >= lower) & (values <= upper)
+ return (values > lower) & (values < upper)
+
+
+def _linear_threshold(value: float, unbounded_value: float) -> float:
+ if value == unbounded_value:
+ return value
+ return float(2**value)
+
+
+def select_highly_variable_features(
+ corrected_variance: np.ndarray,
+ normalized_cell_counts: np.ndarray,
+ mean_nonzero: np.ndarray,
+ active_features: np.ndarray,
+ feature_names: np.ndarray,
+ *,
+ min_cells: int,
+ max_cells: int | float,
+ top_n: int,
+ min_var: float,
+ max_var: float,
+ min_mean: float,
+ max_mean: float,
+ blacklist: str,
+ keep_bounds: bool,
+) -> np.ndarray:
+ """Select highly variable features from precomputed feature statistics."""
+ corrected_variance = np.asarray(corrected_variance)
+ normalized_cell_counts = np.asarray(normalized_cell_counts)
+ mean_nonzero = np.asarray(mean_nonzero)
+ active_features = np.asarray(active_features, dtype=bool)
+ feature_names = np.asarray(feature_names)
+ size = corrected_variance.shape[0]
+ if any(
+ values.shape != (size,)
+ for values in (
+ normalized_cell_counts,
+ mean_nonzero,
+ active_features,
+ feature_names,
+ )
+ ):
+ raise ValueError("HVG inputs must be one-dimensional arrays of equal length")
+
+ min_var = _linear_threshold(min_var, -np.inf)
+ max_var = _linear_threshold(max_var, np.inf)
+ min_mean = _linear_threshold(min_mean, -np.inf)
+ max_mean = _linear_threshold(max_mean, np.inf)
+
+ if blacklist:
+ pattern = re.compile(blacklist.upper())
+ allowed = np.fromiter(
+ (pattern.match(str(name).upper()) is None for name in feature_names),
+ dtype=bool,
+ count=size,
+ )
+ else:
+ allowed = np.ones(size, dtype=bool)
+
+ cell_count_candidates = normalized_cell_counts >= min_cells
+ cell_count_candidates &= (
+ normalized_cell_counts <= max_cells
+ if keep_bounds
+ else normalized_cell_counts < max_cells
+ )
+ candidates = (
+ cell_count_candidates
+ & _bounded(mean_nonzero, min_mean, max_mean, keep_bounds=keep_bounds)
+ & active_features
+ & allowed
+ )
+ if min_var == -np.inf:
+ if top_n < 1:
+ raise ValueError(
+ "ERROR: Please provide a value greater than 0 for `top_n` parameter"
+ )
+ n_valid_features = int(candidates.sum())
+ if n_valid_features == 0:
+ raise ValueError(
+ "No features passed HVG candidate filters "
+ f"(min_cells={min_cells}, max_cells={max_cells}, "
+ f"min_mean={min_mean}, max_mean={max_mean})."
+ )
+ if top_n >= n_valid_features:
+ logger.warning(
+ f"WARNING: Number of valid features are less then value "
+ f"of parameter `top_n`: {top_n}. Resetting `top_n` to "
+ f"{n_valid_features - 1}"
+ )
+ top_n = n_valid_features - 1
+ min_var = (
+ pd.Series(corrected_variance)[candidates]
+ .sort_values(ascending=False)
+ .values[top_n]
+ )
+
+ return np.asarray(
+ candidates
+ & _bounded(
+ corrected_variance,
+ min_var,
+ max_var,
+ keep_bounds=keep_bounds,
+ ),
+ dtype=bool,
+ )
diff --git a/scarf/graph/__init__.py b/scarf/graph/__init__.py
new file mode 100644
index 00000000..8367fb4c
--- /dev/null
+++ b/scarf/graph/__init__.py
@@ -0,0 +1,15 @@
+"""Graph package public surface.
+
+Analyst-facing types live here. Path helpers, encoded-path parsers, and
+operation argument planners stay package-internal.
+"""
+
+from .state import (
+ ArtifactSelectionError as ArtifactSelectionError,
+ AssayState as AssayState,
+)
+
+__all__ = [
+ "ArtifactSelectionError",
+ "AssayState",
+]
diff --git a/scarf/graph/arguments.py b/scarf/graph/arguments.py
new file mode 100644
index 00000000..72dcc3a9
--- /dev/null
+++ b/scarf/graph/arguments.py
@@ -0,0 +1,322 @@
+import operator
+from collections.abc import Callable, Mapping
+from dataclasses import MISSING, dataclass, field, fields
+from types import MappingProxyType
+from typing import Any, ClassVar, Literal
+
+import numpy as np
+import zarr
+
+from ..storage.artifact_writer import (
+ ArrayRequirement,
+ AttributeRequirement,
+ PlannedArtifact,
+ plan_artifact,
+)
+from ..storage.artifacts import (
+ ArtifactRef,
+ ArtifactScope,
+ make_provenance,
+ provenance_hash,
+ serialize_artifact_value,
+)
+
+type ArgumentRole = Literal["input", "parameter", "execution"]
+
+
+def _positive_integer(value: Any, name: str) -> int:
+ if isinstance(value, bool):
+ raise TypeError(f"{name} must be a positive integer")
+ try:
+ resolved = operator.index(value)
+ except TypeError:
+ raise TypeError(f"{name} must be a positive integer") from None
+ if resolved < 1:
+ raise ValueError(f"{name} must be greater than zero")
+ return int(resolved)
+
+
+def parameter(
+ default: Any = MISSING,
+ *,
+ default_factory: Any = MISSING,
+) -> Any:
+ if default is not MISSING and default_factory is not MISSING:
+ raise ValueError("Cannot specify both default and default_factory")
+ if default_factory is not MISSING:
+ return field(
+ default_factory=default_factory,
+ metadata={"argument_role": "parameter"},
+ )
+ if default is not MISSING:
+ return field(default=default, metadata={"argument_role": "parameter"})
+ return field(metadata={"argument_role": "parameter"})
+
+
+def execution(
+ default: Any = MISSING,
+ *,
+ default_factory: Any = MISSING,
+) -> Any:
+ if default is not MISSING and default_factory is not MISSING:
+ raise ValueError("Cannot specify both default and default_factory")
+ if default_factory is not MISSING:
+ return field(
+ default_factory=default_factory,
+ metadata={"argument_role": "execution"},
+ )
+ if default is not MISSING:
+ return field(default=default, metadata={"argument_role": "execution"})
+ return field(metadata={"argument_role": "execution"})
+
+
+def artifact_input(
+ default: Any = MISSING,
+ *,
+ default_factory: Any = MISSING,
+) -> Any:
+ if default is not MISSING and default_factory is not MISSING:
+ raise ValueError("Cannot specify both default and default_factory")
+ if default_factory is not MISSING:
+ return field(
+ default_factory=default_factory,
+ metadata={"argument_role": "input"},
+ )
+ if default is not MISSING:
+ return field(default=default, metadata={"argument_role": "input"})
+ return field(metadata={"argument_role": "input"})
+
+
+@dataclass(frozen=True, slots=True)
+class ArgumentRecord:
+ parameters: dict[str, Any]
+ execution_options: dict[str, Any]
+ inputs: dict[str, Any]
+
+
+@dataclass(frozen=True, slots=True)
+class OperationArguments:
+ operation: ClassVar[str]
+ artifact_kind: ClassVar[str]
+
+ def to_record(self) -> ArgumentRecord:
+ partitions: dict[ArgumentRole, dict[str, Any]] = {
+ "input": {},
+ "parameter": {},
+ "execution": {},
+ }
+ for model_field in fields(self):
+ role = model_field.metadata.get("argument_role")
+ if role not in partitions:
+ raise TypeError(
+ f"{type(self).__name__}.{model_field.name} has no argument role"
+ )
+ partitions[role][model_field.name] = serialize_artifact_value(
+ getattr(self, model_field.name)
+ )
+ return ArgumentRecord(
+ parameters=partitions["parameter"],
+ execution_options=partitions["execution"],
+ inputs=partitions["input"],
+ )
+
+ def provenance(self) -> dict[str, Any]:
+ record = self.to_record()
+ return make_provenance(
+ operation=self.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ )
+
+ def provenance_hash(self) -> str:
+ return provenance_hash(self.provenance())
+
+ def plan(
+ self,
+ root: zarr.Group,
+ *,
+ scope: ArtifactScope,
+ assay: str | None = None,
+ invalidate_cache: bool = False,
+ required_arrays: tuple[str | ArrayRequirement, ...] = (),
+ required_attributes: tuple[str | AttributeRequirement, ...] = (),
+ reuse_validator: Callable[[ArtifactRef, zarr.Group], bool] | None = None,
+ ) -> PlannedArtifact:
+ record = self.to_record()
+ return plan_artifact(
+ root,
+ scope=scope,
+ assay=assay,
+ kind=self.artifact_kind,
+ operation=self.operation,
+ parameters=record.parameters,
+ inputs=record.inputs,
+ execution_options=record.execution_options,
+ invalidate_cache=invalidate_cache,
+ required_arrays=required_arrays,
+ required_attributes=required_attributes,
+ reuse_validator=reuse_validator,
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class NormalizationArguments(OperationArguments):
+ operation: ClassVar[str] = "run_normalization"
+ artifact_kind: ClassVar[str] = "normalized"
+
+ from_assay: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ cell_selection: ArtifactRef = artifact_input()
+ feature_selection: ArtifactRef = artifact_input()
+ normalization_method: Callable[..., Any] | str = parameter()
+ size_factor: float | None = parameter()
+ log_transform: bool = parameter()
+ renormalize_subset: bool = parameter()
+ update_state: bool = execution()
+ invalidate_cache: bool = execution(False)
+
+
+@dataclass(frozen=True, slots=True)
+class FeatureScalingArguments(OperationArguments):
+ operation: ClassVar[str] = "calculate_feature_scaling"
+ artifact_kind: ClassVar[str] = "feature_scaling"
+
+ normalized: ArtifactRef = artifact_input()
+ enabled: bool = parameter()
+ batch_size: int = execution()
+ invalidate_cache: bool = execution(False)
+
+
+@dataclass(frozen=True, slots=True)
+class PcaArguments(OperationArguments):
+ operation: ClassVar[str] = "run_pca"
+ artifact_kind: ClassVar[str] = "reduction"
+
+ normalized: ArtifactRef = artifact_input()
+ feature_scaling: ArtifactRef = artifact_input()
+ pca_cell_selection: ArtifactRef = artifact_input()
+ pca_cell_key: str = execution()
+ dims: int = parameter()
+ feat_scaling: bool = parameter()
+ batch_size: int = execution()
+ show_elbow_plot: bool = execution()
+ update_state: bool = execution()
+ invalidate_cache: bool = execution(False)
+
+
+@dataclass(frozen=True, slots=True)
+class LsiArguments(OperationArguments):
+ operation: ClassVar[str] = "run_lsi"
+ artifact_kind: ClassVar[str] = "reduction"
+
+ normalized: ArtifactRef = artifact_input()
+ feature_scaling: ArtifactRef = artifact_input()
+ dims: int = parameter()
+ skip_first: bool = parameter()
+ rand_state: int = parameter()
+ solver: Literal["streaming", "materialized"] = parameter()
+ n_iter: int = parameter()
+ n_oversamples: int = parameter()
+ batch_size: int = execution()
+ update_state: bool = execution()
+ invalidate_cache: bool = execution(False)
+
+
+@dataclass(frozen=True, slots=True)
+class CustomReductionArguments(OperationArguments):
+ operation: ClassVar[str] = "run_custom_reduction"
+ artifact_kind: ClassVar[str] = "reduction"
+
+ normalized: ArtifactRef = artifact_input()
+ feature_scaling: ArtifactRef = artifact_input()
+ loadings: np.ndarray = artifact_input()
+ update_state: bool = execution()
+ invalidate_cache: bool = execution(False)
+
+
+@dataclass(frozen=True, slots=True)
+class HarmonyArguments(OperationArguments):
+ operation: ClassVar[str] = "run_harmony"
+ artifact_kind: ClassVar[str] = "batch_correction"
+
+ reduction: ArtifactRef = artifact_input()
+ batch_values: ArtifactRef = artifact_input()
+ batch_columns: tuple[str, ...] = parameter()
+ harmony_parameters: Mapping[str, Any] = parameter()
+ algorithm_version: str = parameter()
+ batch_size: int = execution()
+ invalidate_cache: bool = execution(False)
+
+ def __post_init__(self) -> None:
+ object.__setattr__(
+ self,
+ "harmony_parameters",
+ MappingProxyType(dict(self.harmony_parameters)),
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class AnnIndexArguments(OperationArguments):
+ operation: ClassVar[str] = "build_ann_index"
+ artifact_kind: ClassVar[str] = "ann_index"
+
+ coordinates: ArtifactRef = artifact_input()
+ ann_metric: str = parameter()
+ ann_efc: int = parameter()
+ ann_ef: int = parameter()
+ ann_m: int = parameter()
+ rand_state: int = parameter()
+ ann_parallel: bool = parameter()
+ parallel_threads: int | None = parameter()
+ batch_size: int = execution()
+ invalidate_cache: bool = execution(False)
+
+
+@dataclass(frozen=True, slots=True)
+class NeighborQueryArguments(OperationArguments):
+ operation: ClassVar[str] = "query_neighbors"
+ artifact_kind: ClassVar[str] = "neighbors"
+
+ ann_index: ArtifactRef = artifact_input()
+ coordinates: ArtifactRef = artifact_input()
+ k: int = parameter()
+ distance_metric: str = parameter()
+ batch_size: int = execution()
+ invalidate_cache: bool = execution(False)
+
+
+@dataclass(frozen=True, slots=True)
+class ConnectivityMapArguments(OperationArguments):
+ operation: ClassVar[str] = "build_connectivity_map"
+ artifact_kind: ClassVar[str] = "connectivity_map"
+
+ neighbors: ArtifactRef = artifact_input()
+ local_connectivity: float = parameter()
+ bandwidth: float = parameter()
+ invalidate_cache: bool = execution(False)
+
+ def __post_init__(self) -> None:
+ from ..neighbors.graph import validate_connectivity_parameters
+
+ local_connectivity, bandwidth = validate_connectivity_parameters(
+ self.local_connectivity,
+ self.bandwidth,
+ )
+ object.__setattr__(self, "local_connectivity", local_connectivity)
+ object.__setattr__(self, "bandwidth", bandwidth)
+
+
+@dataclass(frozen=True, slots=True)
+class EmbeddingInitializationArguments(OperationArguments):
+ operation: ClassVar[str] = "build_embedding_initialization"
+ artifact_kind: ClassVar[str] = "embedding_initialization"
+
+ reduction: ArtifactRef = artifact_input()
+ n_centroids: int = parameter()
+ rand_state: int = parameter()
+ batch_size: int = parameter()
+ kmeans_sampling: float = parameter(0.1)
+ kmeans_batch_size: int = parameter(10_000)
+ algorithm_version: str = parameter("minibatch_kmeans_v2")
+ invalidate_cache: bool = execution(False)
diff --git a/scarf/graph/distances.py b/scarf/graph/distances.py
new file mode 100644
index 00000000..60e7a342
--- /dev/null
+++ b/scarf/graph/distances.py
@@ -0,0 +1,38 @@
+"""Read-side contract for persisted nearest-neighbor distances."""
+
+from collections.abc import Mapping
+from typing import Any
+
+from ..storage.artifacts import ArtifactRef, inspect_artifact, parse_artifact_path
+
+NEIGHBOR_DISTANCE_METRICS = frozenset({"l2", "cosine"})
+
+
+def validate_distance_provenance(zw: Any, knn_loc: str) -> None:
+ """Check that the distances stored at ``knn_loc`` have a known metric.
+
+ Encoded locations carry no provenance and are read as written. Artifact
+ locations must name the metric of their stored distances and agree with the
+ index those distances were queried from.
+ """
+ try:
+ ref = parse_artifact_path(knn_loc)
+ except ValueError:
+ return
+ if ref.kind != "neighbors":
+ return
+ status = inspect_artifact(zw, ref)
+ metric = (status.parameters or {}).get("distance_metric")
+ if metric not in NEIGHBOR_DISTANCE_METRICS:
+ raise ValueError(
+ "Neighbors artifact does not name the metric of its stored "
+ "distances; recompute neighbors"
+ )
+ source = (status.inputs or {}).get("ann_index")
+ if not isinstance(source, Mapping):
+ raise ValueError("Neighbors artifact has no ANN index input")
+ source_metric = (
+ inspect_artifact(zw, ArtifactRef.from_dict(source)).parameters or {}
+ ).get("ann_metric")
+ if source_metric != metric:
+ raise ValueError("Neighbors distance metric does not match its ANN index input")
diff --git a/scarf/graph/encoded_paths.py b/scarf/graph/encoded_paths.py
new file mode 100644
index 00000000..d8b59256
--- /dev/null
+++ b/scarf/graph/encoded_paths.py
@@ -0,0 +1,496 @@
+"""Construct, parse, and look up parameter-encoded assay graph paths.
+
+This module is the sole parser for the assay graph chain: ``normed__``,
+reduction, ANN (optional ``__unscaled`` / ``__harmony_{hash}``), KNN, graph,
+k-means, and ``latest_*`` traversal. It does not mutate stores.
+"""
+
+from typing import Any
+
+from .paths import (
+ AssayGraphPaths,
+ AssayNearestNeighborPaths,
+ StoredAssayGraph,
+ StoredIntegratedGraph,
+)
+
+
+def make_normalized_leaf_name(cell_key: str, feat_key: str) -> str:
+ """Return ``normed__{cell_key}__{feat_key}``."""
+ return f"normed__{cell_key}__{feat_key}"
+
+
+def make_normalized_group_path(from_assay: str, cell_key: str, feat_key: str) -> str:
+ """Return ``{from_assay}/normed__{cell_key}__{feat_key}``."""
+ return f"{from_assay}/{make_normalized_leaf_name(cell_key, feat_key)}"
+
+
+def make_reduction_group_path(
+ normalized_group_path: str,
+ reduction_method: str,
+ dims: int,
+ pca_cell_key: str,
+) -> str:
+ return (
+ f"{normalized_group_path}/reduction__{reduction_method}__{dims}__{pca_cell_key}"
+ )
+
+
+def make_neighbor_index_group_path(
+ reduction_group_path: str,
+ ann_metric: str,
+ ann_efc: int,
+ ann_ef: int,
+ ann_m: int,
+ rand_state: int,
+ *,
+ feat_scaling: bool = True,
+ harmony_contract_hash: str | None = None,
+) -> str:
+ path = (
+ f"{reduction_group_path}/ann__{ann_metric}__{ann_efc}__{ann_ef}__"
+ f"{ann_m}__{rand_state}"
+ )
+ if not feat_scaling:
+ path = f"{path}__unscaled"
+ if harmony_contract_hash is not None:
+ path = f"{path}__harmony_{harmony_contract_hash}"
+ return path
+
+
+def make_nearest_neighbors_group_path(neighbor_index_group_path: str, k: int) -> str:
+ return f"{neighbor_index_group_path}/knn__{k}"
+
+
+def make_cell_graph_group_path(
+ nearest_neighbors_group_path: str,
+ local_connectivity: float,
+ bandwidth: float,
+) -> str:
+ return f"{nearest_neighbors_group_path}/graph__{local_connectivity}__{bandwidth}"
+
+
+def make_kmeans_initialization_group_path(
+ reduction_group_path: str, n_centroids: int, rand_state: int
+) -> str:
+ return f"{reduction_group_path}/kmeans__{n_centroids}__{rand_state}"
+
+
+def make_integrated_graph_path(integrated_graphs_loc: str, label: str) -> str:
+ return f"{integrated_graphs_loc}/{label}"
+
+
+def is_integrated_graph_path(graph_loc: str, integrated_graphs_loc: str) -> bool:
+ return graph_loc == integrated_graphs_loc or graph_loc.startswith(
+ f"{integrated_graphs_loc}/"
+ )
+
+
+def nearest_neighbors_group_path_from_cell_graph(cell_graph_group_path: str) -> str:
+ """Return the parent KNN group path for a cell-graph leaf."""
+ if "/" not in cell_graph_group_path:
+ raise ValueError(
+ f"Cell graph path has no parent group: {cell_graph_group_path}"
+ )
+ return cell_graph_group_path.rsplit("/", 1)[0]
+
+
+def parse_normalized_leaf_name(name: str) -> tuple[str, str]:
+ if not name.startswith("normed__"):
+ raise ValueError(f"Not a normalized group name: {name}")
+ rest = name.removeprefix("normed__")
+ cell_key, feat_key = rest.split("__", 1)
+ return cell_key, feat_key
+
+
+def parse_reduction_leaf_name(name: str) -> tuple[str, int, str]:
+ if not name.startswith("reduction__"):
+ raise ValueError(f"Not a reduction group name: {name}")
+ body = name.removeprefix("reduction__")
+ reduction_method, dims_s, pca_cell_key = body.split("__", 2)
+ return reduction_method, int(dims_s), pca_cell_key
+
+
+def parse_reduction_group_path(
+ reduction_group_path: str,
+) -> tuple[str, int, str]:
+ """Parse the reduction leaf of a complete reduction group path."""
+ return parse_reduction_leaf_name(_leaf(reduction_group_path))
+
+
+def parse_ann_leaf_name(
+ name: str,
+) -> tuple[str, int, int, int, int, bool, str | None]:
+ """Parse ANN leaf name; ``__unscaled`` and ``__harmony_{hash}`` are optional."""
+ if not name.startswith("ann__"):
+ raise ValueError(f"Not an ANN group name: {name}")
+ parts = name.split("__")
+ if len(parts) < 6:
+ raise ValueError(f"ANN group name missing required fields: {name}")
+ ann_metric = parts[1]
+ ann_efc, ann_ef, ann_m, rand_state = (
+ int(parts[2]),
+ int(parts[3]),
+ int(parts[4]),
+ int(parts[5]),
+ )
+ feat_scaling = True
+ harmony_contract_hash: str | None = None
+ for token in parts[6:]:
+ if token == "unscaled":
+ feat_scaling = False
+ elif token.startswith("harmony_"):
+ harmony_contract_hash = token.removeprefix("harmony_")
+ else:
+ raise ValueError(f"Unrecognized ANN path suffix {token!r} in {name}")
+ return (
+ ann_metric,
+ ann_efc,
+ ann_ef,
+ ann_m,
+ rand_state,
+ feat_scaling,
+ harmony_contract_hash,
+ )
+
+
+def parse_neighbor_index_group_path(
+ neighbor_index_group_path: str,
+) -> tuple[str, int, int, int, int, bool, str | None]:
+ """Parse the ANN leaf of a complete neighbor-index group path."""
+ return parse_ann_leaf_name(_leaf(neighbor_index_group_path))
+
+
+def parse_knn_leaf_name(name: str) -> int:
+ if not name.startswith("knn__"):
+ raise ValueError(f"Not a KNN group name: {name}")
+ return int(name.removeprefix("knn__"))
+
+
+def parse_nearest_neighbors_group_path(
+ nearest_neighbors_group_path: str,
+) -> int:
+ """Parse the KNN leaf of a complete nearest-neighbors group path."""
+ return parse_knn_leaf_name(_leaf(nearest_neighbors_group_path))
+
+
+def parse_graph_leaf_name(name: str) -> tuple[float, float]:
+ if not name.startswith("graph__"):
+ raise ValueError(f"Not a cell-graph group name: {name}")
+ body = name.removeprefix("graph__")
+ local_connectivity_s, bandwidth_s = body.split("__", 1)
+ return float(local_connectivity_s), float(bandwidth_s)
+
+
+def parse_kmeans_leaf_name(name: str) -> tuple[int, int]:
+ if not name.startswith("kmeans__"):
+ raise ValueError(f"Not a k-means group name: {name}")
+ body = name.removeprefix("kmeans__")
+ n_centroids_s, rand_state_s = body.split("__", 1)
+ return int(n_centroids_s), int(rand_state_s)
+
+
+def _leaf(path: str) -> str:
+ return path.rsplit("/", 1)[-1]
+
+
+def parse_assay_keys_from_nearest_neighbors_path(
+ nearest_neighbors_group_path: str,
+) -> tuple[str, str, str]:
+ """Return assay, cell key, and feature key encoded by a KNN path."""
+ paths = nearest_neighbor_paths_from_loc(nearest_neighbors_group_path)
+ cell_key, feat_key = parse_normalized_leaf_name(_leaf(paths.normalized_group_path))
+ if "/" not in paths.normalized_group_path:
+ raise ValueError(
+ f"Normalized path has no assay parent: {paths.normalized_group_path}"
+ )
+ from_assay = paths.normalized_group_path.rsplit("/", 1)[0]
+ return from_assay, cell_key, feat_key
+
+
+def stored_assay_graph_from_paths(
+ paths: AssayGraphPaths,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+) -> StoredAssayGraph:
+ """Fill cheap scientific parameters from encoded leaf names when possible."""
+ resolved_assay = from_assay
+ resolved_cell = cell_key
+ resolved_feat = feat_key
+ try:
+ parsed_cell, parsed_feat = parse_normalized_leaf_name(
+ _leaf(paths.normalized_group_path)
+ )
+ if resolved_cell is None:
+ resolved_cell = parsed_cell
+ if resolved_feat is None:
+ resolved_feat = parsed_feat
+ if resolved_assay is None:
+ parent = paths.normalized_group_path.rsplit("/", 1)[0]
+ resolved_assay = parent
+ except ValueError:
+ pass
+
+ reduction_method = dims = pca_cell_key = None
+ try:
+ reduction_method, dims, pca_cell_key = parse_reduction_leaf_name(
+ _leaf(paths.reduction_group_path)
+ )
+ except ValueError:
+ pass
+
+ ann_metric = ann_efc = ann_ef = ann_m = rand_state = None
+ feat_scaling = harmony_contract_hash = None
+ try:
+ (
+ ann_metric,
+ ann_efc,
+ ann_ef,
+ ann_m,
+ rand_state,
+ feat_scaling,
+ harmony_contract_hash,
+ ) = parse_ann_leaf_name(_leaf(paths.neighbor_index_group_path))
+ except ValueError:
+ pass
+
+ k = None
+ try:
+ k = parse_knn_leaf_name(_leaf(paths.nearest_neighbors_group_path))
+ except ValueError:
+ pass
+
+ local_connectivity = bandwidth = None
+ try:
+ local_connectivity, bandwidth = parse_graph_leaf_name(
+ _leaf(paths.cell_graph_group_path)
+ )
+ except ValueError:
+ pass
+
+ n_centroids = None
+ if paths.kmeans_initialization_group_path is not None:
+ try:
+ n_centroids, kmeans_rand = parse_kmeans_leaf_name(
+ _leaf(paths.kmeans_initialization_group_path)
+ )
+ if rand_state is None:
+ rand_state = kmeans_rand
+ except ValueError:
+ pass
+
+ if resolved_assay is None or resolved_cell is None or resolved_feat is None:
+ raise ValueError(
+ "Could not determine assay, cell_key, and feat_key for stored graph"
+ )
+
+ return StoredAssayGraph(
+ paths=paths,
+ from_assay=resolved_assay,
+ cell_key=resolved_cell,
+ feat_key=resolved_feat,
+ reduction_method=reduction_method,
+ dims=dims,
+ pca_cell_key=pca_cell_key,
+ ann_metric=ann_metric,
+ ann_efc=ann_efc,
+ ann_ef=ann_ef,
+ ann_m=ann_m,
+ rand_state=rand_state,
+ k=k,
+ local_connectivity=local_connectivity,
+ bandwidth=bandwidth,
+ feat_scaling=feat_scaling,
+ harmony_contract_hash=harmony_contract_hash,
+ n_centroids=n_centroids,
+ )
+
+
+def parse_assay_graph_paths(
+ graph_loc: str,
+ *,
+ kmeans_initialization_group_path: str | None = None,
+) -> StoredAssayGraph:
+ """Parse a parameter-encoded cell-graph path into a ``StoredAssayGraph``."""
+ segments = graph_loc.strip("/").split("/")
+ if len(segments) < 6:
+ raise ValueError(f"Not an encoded assay graph path: {graph_loc}")
+
+ graph_name = segments[-1]
+ knn_name = segments[-2]
+ ann_name = segments[-3]
+ reduction_name = segments[-4]
+ normed_name = segments[-5]
+ from_assay = "/".join(segments[:-5])
+ if not from_assay:
+ raise ValueError(f"Assay graph path missing assay prefix: {graph_loc}")
+
+ # Validate leaf encodings early so malformed paths fail here.
+ parse_normalized_leaf_name(normed_name)
+ parse_reduction_leaf_name(reduction_name)
+ parse_ann_leaf_name(ann_name)
+ parse_knn_leaf_name(knn_name)
+ parse_graph_leaf_name(graph_name)
+
+ normalized = "/".join(segments[:-4])
+ reduction = "/".join(segments[:-3])
+ neighbor_index = "/".join(segments[:-2])
+ nearest_neighbors = "/".join(segments[:-1])
+ paths = AssayGraphPaths(
+ normalized_group_path=normalized,
+ reduction_group_path=reduction,
+ neighbor_index_group_path=neighbor_index,
+ nearest_neighbors_group_path=nearest_neighbors,
+ cell_graph_group_path=graph_loc.strip("/"),
+ kmeans_initialization_group_path=kmeans_initialization_group_path,
+ )
+ return stored_assay_graph_from_paths(paths, from_assay=from_assay)
+
+
+def _group_attrs(group: Any) -> Any:
+ return group.attrs
+
+
+def lookup_latest_reduction_group_path(zw: Any, normalized_group_path: str) -> str:
+ """Return the existing reduction selected by a normalized group."""
+ normalized_group = zw[normalized_group_path]
+ reduction_group_path = str(_group_attrs(normalized_group)["latest_reduction"])
+ return reduction_group_path
+
+
+def lookup_latest_neighbor_index_group_path(zw: Any, reduction_group_path: str) -> str:
+ """Return the existing neighbor index selected by a reduction group."""
+ reduction_group = zw[reduction_group_path]
+ neighbor_index_group_path = str(_group_attrs(reduction_group)["latest_ann"])
+ return neighbor_index_group_path
+
+
+def lookup_latest_nearest_neighbors_group_path(
+ zw: Any, neighbor_index_group_path: str
+) -> str:
+ """Return the existing KNN group selected by a neighbor-index group."""
+ neighbor_index_group = zw[neighbor_index_group_path]
+ nearest_neighbors_group_path = str(_group_attrs(neighbor_index_group)["latest_knn"])
+ return nearest_neighbors_group_path
+
+
+def lookup_latest_cell_graph_group_path(
+ zw: Any, nearest_neighbors_group_path: str
+) -> str:
+ """Return the existing cell graph selected by a KNN group."""
+ nearest_neighbors_group = zw[nearest_neighbors_group_path]
+ cell_graph_group_path = str(_group_attrs(nearest_neighbors_group)["latest_graph"])
+ return cell_graph_group_path
+
+
+def lookup_latest_kmeans_group_path(zw: Any, reduction_group_path: str) -> str | None:
+ """Return the k-means group selected by a reduction, when recorded."""
+ reduction_group = zw[reduction_group_path]
+ reduction_attrs = _group_attrs(reduction_group)
+ if "latest_kmeans" not in reduction_attrs:
+ return None
+ return str(reduction_attrs["latest_kmeans"])
+
+
+def lookup_latest_assay_paths(
+ zw: Any,
+ from_assay: str,
+ cell_key: str,
+ feat_key: str,
+) -> AssayGraphPaths:
+ """Follow ``latest_*`` pointers through a complete assay graph chain."""
+ nearest_neighbor_paths = lookup_latest_nearest_neighbor_paths(
+ zw, from_assay, cell_key, feat_key
+ )
+ reduction = nearest_neighbor_paths.reduction_group_path
+ nearest_neighbors = nearest_neighbor_paths.nearest_neighbors_group_path
+ cell_graph = lookup_latest_cell_graph_group_path(zw, nearest_neighbors)
+ kmeans = lookup_latest_kmeans_group_path(zw, reduction)
+ return AssayGraphPaths(
+ normalized_group_path=nearest_neighbor_paths.normalized_group_path,
+ reduction_group_path=reduction,
+ neighbor_index_group_path=nearest_neighbor_paths.neighbor_index_group_path,
+ nearest_neighbors_group_path=nearest_neighbors,
+ cell_graph_group_path=cell_graph,
+ kmeans_initialization_group_path=kmeans,
+ )
+
+
+def lookup_latest_nearest_neighbor_paths(
+ zw: Any,
+ from_assay: str,
+ cell_key: str,
+ feat_key: str,
+) -> AssayNearestNeighborPaths:
+ """Follow ``latest_*`` pointers only through the nearest-neighbors group."""
+ normalized = make_normalized_group_path(from_assay, cell_key, feat_key)
+ reduction = lookup_latest_reduction_group_path(zw, normalized)
+ neighbor_index = lookup_latest_neighbor_index_group_path(zw, reduction)
+ nearest_neighbors = lookup_latest_nearest_neighbors_group_path(zw, neighbor_index)
+ _ = zw[nearest_neighbors]
+ return AssayNearestNeighborPaths(
+ normalized_group_path=normalized,
+ reduction_group_path=reduction,
+ neighbor_index_group_path=neighbor_index,
+ nearest_neighbors_group_path=nearest_neighbors,
+ )
+
+
+def lookup_latest_kmeans_path(
+ zw: Any, from_assay: str, cell_key: str, feat_key: str
+) -> str | None:
+ """Return the latest k-means group path without requiring a graph.
+
+ Follows ``normed -> latest_reduction -> latest_kmeans`` only. Does not
+ require an ANN index, KNN graph, or cell graph to exist, matching the
+ minimal inputs needed for initial embedding. Returns None when the
+ reduction has no k-means initialization recorded.
+ """
+ normalized = make_normalized_group_path(from_assay, cell_key, feat_key)
+ reduction = lookup_latest_reduction_group_path(zw, normalized)
+ return lookup_latest_kmeans_group_path(zw, reduction)
+
+
+def lookup_latest_assay_graph(
+ zw: Any, from_assay: str, cell_key: str, feat_key: str
+) -> StoredAssayGraph:
+ """Follow ``latest_*`` pointers for an assay graph without mutating the store."""
+ paths = lookup_latest_assay_paths(zw, from_assay, cell_key, feat_key)
+ return stored_assay_graph_from_paths(
+ paths,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ )
+
+
+def lookup_stored_integrated_graph(zw: Any, graph_loc: str) -> StoredIntegratedGraph:
+ """Read an integrated graph path and its stored metadata only."""
+ group = zw[graph_loc]
+ attrs = _group_attrs(group)
+ n_cells = int(attrs["n_cells"]) if "n_cells" in attrs else None
+ n_neighbors = int(attrs["n_neighbors"]) if "n_neighbors" in attrs else None
+ return StoredIntegratedGraph(
+ cell_graph_group_path=graph_loc,
+ n_cells=n_cells,
+ n_neighbors=n_neighbors,
+ )
+
+
+def nearest_neighbor_paths_from_loc(knn_loc: str) -> AssayNearestNeighborPaths:
+ """Parse a KNN location into exact parent-chain paths."""
+ neighbor_index = knn_loc.rsplit("/", 1)[0]
+ reduction = neighbor_index.rsplit("/", 1)[0]
+ normalized = reduction.rsplit("/", 1)[0]
+ parse_normalized_leaf_name(_leaf(normalized))
+ parse_reduction_leaf_name(_leaf(reduction))
+ parse_ann_leaf_name(_leaf(neighbor_index))
+ parse_knn_leaf_name(_leaf(knn_loc))
+ return AssayNearestNeighborPaths(
+ normalized_group_path=normalized,
+ reduction_group_path=reduction,
+ neighbor_index_group_path=neighbor_index,
+ nearest_neighbors_group_path=knn_loc,
+ )
diff --git a/scarf/graph/paths.py b/scarf/graph/paths.py
new file mode 100644
index 00000000..db175c84
--- /dev/null
+++ b/scarf/graph/paths.py
@@ -0,0 +1,65 @@
+"""Value types for assay and integrated graph locations.
+
+These dataclasses carry Zarr group paths and cheaply parsed parameters only.
+They do not compute, write, or import mapping code.
+"""
+
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True, slots=True)
+class AssayNearestNeighborPaths:
+ """Exact current Zarr group locations through nearest neighbors."""
+
+ normalized_group_path: str
+ reduction_group_path: str
+ neighbor_index_group_path: str
+ nearest_neighbors_group_path: str
+
+
+@dataclass(frozen=True, slots=True)
+class AssayGraphPaths:
+ """Exact current Zarr group locations for one assay graph chain."""
+
+ normalized_group_path: str
+ reduction_group_path: str
+ neighbor_index_group_path: str
+ nearest_neighbors_group_path: str
+ cell_graph_group_path: str
+ kmeans_initialization_group_path: str | None = None
+
+
+@dataclass(frozen=True, slots=True)
+class StoredAssayGraph:
+ """A graph built from one assay, with paths and parsed parameters."""
+
+ paths: AssayGraphPaths
+ from_assay: str
+ cell_key: str
+ feat_key: str
+ reduction_method: str | None = None
+ dims: int | None = None
+ pca_cell_key: str | None = None
+ ann_metric: str | None = None
+ ann_efc: int | None = None
+ ann_ef: int | None = None
+ ann_m: int | None = None
+ rand_state: int | None = None
+ k: int | None = None
+ local_connectivity: float | None = None
+ bandwidth: float | None = None
+ feat_scaling: bool | None = None
+ harmony_contract_hash: str | None = None
+ n_centroids: int | None = None
+
+
+@dataclass(frozen=True, slots=True)
+class StoredIntegratedGraph:
+ """An integrated graph group and the metadata actually stored on it."""
+
+ cell_graph_group_path: str
+ n_cells: int | None = None
+ n_neighbors: int | None = None
+
+
+StoredGraph = StoredAssayGraph | StoredIntegratedGraph
diff --git a/scarf/graph/state.py b/scarf/graph/state.py
new file mode 100644
index 00000000..ca7bd9e4
--- /dev/null
+++ b/scarf/graph/state.py
@@ -0,0 +1,1522 @@
+import re
+from collections.abc import Callable, Mapping
+from dataclasses import dataclass, field
+from types import MappingProxyType
+from typing import Any
+
+import numpy as np
+import zarr
+
+from ..storage.arrays import create_metadata_column, create_numeric_array
+from ..storage.artifact_writer import (
+ ArrayRequirement,
+ PlannedArtifact,
+ finish_artifact,
+ plan_artifact,
+ start_artifact,
+)
+from ..storage.artifacts import (
+ ArtifactRef,
+ ValueFingerprintBuilder,
+ artifact_group,
+ artifact_path,
+ fingerprint_stored_arrays,
+ fingerprint_stored_strings,
+ fingerprint_strings,
+ group_at,
+ inspect_artifact,
+ parse_artifact_path,
+ require_complete_artifact,
+)
+from ..storage.geometry import array_geometry
+from ..storage.layout import _group_zarr_format, row_sharded_array_spec
+from ..storage.partition import row_band
+from ..storage.profiles import resolve_storage_profile
+from ..storage.selections import fingerprint_selected_stored_strings
+from ..storage.types import as_zarr_array, as_zarr_group
+from .paths import AssayGraphPaths, StoredAssayGraph
+
+_GRAPH_REF_KINDS = {
+ "normalized": "normalized",
+ "feature_scaling": "feature_scaling",
+ "reduction": "reduction",
+ "batch_correction": "batch_correction",
+ "ann_index": "ann_index",
+ "embedding_initialization": "embedding_initialization",
+ "neighbors": "neighbors",
+ "connectivity_map": "connectivity_map",
+}
+_RESULT_NAME_PATTERN = re.compile(r"^[a-z][a-z0-9_]*$")
+_NAMED_RESULT_KINDS = {"mapping_reference": "mapping_reference"}
+
+
+type _SelectionContextValue = str | int | float | bool | None
+
+
+class ArtifactSelectionError(ValueError):
+ """A machine-readable failure to validate an artifact's stored selection."""
+
+ def __init__(
+ self,
+ message: str,
+ *,
+ code: str,
+ context: Mapping[str, _SelectionContextValue],
+ ) -> None:
+ super().__init__(message)
+ self.code = code
+ self.context = dict(context)
+
+ def __reduce__(self) -> tuple[Any, tuple[Any, ...]]:
+ return (
+ _restore_artifact_selection_error,
+ (type(self), str(self), self.code, self.context),
+ )
+
+
+def _restore_artifact_selection_error(
+ error_type: type[ArtifactSelectionError],
+ message: str,
+ code: str,
+ context: Mapping[str, _SelectionContextValue],
+) -> ArtifactSelectionError:
+ return error_type(message, code=code, context=context)
+
+
+@dataclass(frozen=True, slots=True)
+class ImportedArtifactStorage:
+ """Narrow storage adapter for imported embedding-domain artifacts."""
+
+ root: zarr.Group
+
+ @staticmethod
+ def requirement(
+ name: str,
+ *,
+ shape: tuple[int, ...],
+ dtype_kind: str | None,
+ ) -> ArrayRequirement:
+ return ArrayRequirement(name, shape=shape, dtype_kind=dtype_kind)
+
+ def plan(
+ self,
+ *,
+ assay: str,
+ kind: str,
+ parameters: dict[str, Any],
+ inputs: dict[str, Any],
+ execution_options: dict[str, Any],
+ invalidate_cache: bool,
+ required_arrays: tuple[ArrayRequirement, ...],
+ reuse_validator: Callable[[ArtifactRef, zarr.Group], bool],
+ ) -> PlannedArtifact:
+ return plan_artifact(
+ self.root,
+ scope="assay",
+ assay=assay,
+ kind=kind,
+ operation="import_dimreduc",
+ parameters=parameters,
+ inputs=inputs,
+ execution_options=execution_options,
+ invalidate_cache=invalidate_cache,
+ required_arrays=required_arrays,
+ reuse_validator=reuse_validator,
+ )
+
+ def start(self, planned: PlannedArtifact) -> zarr.Group:
+ return start_artifact(self.root, planned)
+
+ @staticmethod
+ def finish(group: zarr.Group, planned: PlannedArtifact) -> None:
+ finish_artifact(group, planned)
+
+ def artifact_group(self, ref: ArtifactRef) -> zarr.Group:
+ return artifact_group(self.root, ref)
+
+ def require_complete(self, ref: ArtifactRef) -> Any:
+ return require_complete_artifact(self.root, ref)
+
+ @staticmethod
+ def as_array(node: zarr.Array | zarr.Group, name: str) -> zarr.Array:
+ return as_zarr_array(node, name=name)
+
+ @staticmethod
+ def as_group(node: zarr.Array | zarr.Group, name: str) -> zarr.Group:
+ return as_zarr_group(node, name=name)
+
+ @staticmethod
+ def fingerprint_builder() -> ValueFingerprintBuilder:
+ return ValueFingerprintBuilder()
+
+ @staticmethod
+ def fingerprint_strings(values: np.ndarray) -> str:
+ return fingerprint_strings(values)
+
+ @staticmethod
+ def fingerprint_stored_strings(array: zarr.Array) -> str:
+ return fingerprint_stored_strings(array)
+
+ @staticmethod
+ def block_rows(array: zarr.Array) -> int:
+ return row_band(array_geometry(array), unit="chunk", fallback=1)
+
+ @staticmethod
+ def create_numeric(
+ group: zarr.Group,
+ name: str,
+ *,
+ shape: tuple[int, ...],
+ dtype: Any,
+ block_rows: int,
+ ) -> zarr.Array:
+ return create_numeric_array(
+ group,
+ name,
+ row_sharded_array_spec(
+ shape,
+ dtype,
+ profile=resolve_storage_profile(group.store),
+ band_rows=min(max(shape[0], 1), block_rows),
+ zarr_format=_group_zarr_format(group),
+ fill_value=0.0,
+ ),
+ )
+
+ @staticmethod
+ def create_metadata(
+ group: zarr.Group,
+ name: str,
+ *,
+ dtype: Any,
+ shape: int,
+ block_rows: int,
+ ) -> zarr.Array:
+ return create_metadata_column(
+ group,
+ name,
+ dtype=dtype,
+ shape=shape,
+ chunkSize=min(max(shape, 1), block_rows),
+ overwrite=True,
+ )
+
+
+def _legacy_subset_hash(
+ cell_indices: np.ndarray,
+ feature_indices: np.ndarray,
+) -> int:
+ cells = tuple(np.asarray(cell_indices))
+ features = tuple(np.asarray(feature_indices))
+ return hash((hash(cells), hash(features)))
+
+
+def validate_legacy_graph_selection(
+ store: Any,
+ graph_loc: str,
+ from_assay: str,
+ cell_key: str,
+ feat_key: str,
+) -> None:
+ from .encoded_paths import (
+ nearest_neighbor_paths_from_loc,
+ parse_assay_graph_paths,
+ )
+
+ try:
+ normalized_path = parse_assay_graph_paths(graph_loc).paths.normalized_group_path
+ except ValueError:
+ try:
+ normalized_path = nearest_neighbor_paths_from_loc(
+ graph_loc
+ ).normalized_group_path
+ except ValueError as exc:
+ raise ValueError(
+ "Legacy graph selection provenance cannot be resolved"
+ ) from exc
+ if normalized_path not in store.zw:
+ raise ValueError("Legacy graph normalized data is missing")
+ normalized_group = as_zarr_group(
+ store.zw[normalized_path],
+ name=normalized_path,
+ )
+ stored_hash = normalized_group.attrs.get("subset_hash")
+ assay = store._get_assay(from_assay)
+ feature_column = "I" if feat_key == "I" else f"{cell_key}__{feat_key}"
+ try:
+ cell_indices = store.cells.active_index(cell_key)
+ feature_indices = assay.feats.active_index(feature_column)
+ except (KeyError, TypeError, ValueError) as exc:
+ raise ValueError("Legacy graph selection columns cannot be validated") from exc
+ if stored_hash is None:
+ raise ValueError(
+ "Legacy graph selection provenance is missing, so it cannot be "
+ "selected safely"
+ )
+ if isinstance(stored_hash, str):
+ current_hash: str | int = assay._create_subset_hash(
+ cell_indices,
+ feature_indices,
+ )
+ elif not isinstance(stored_hash, bool) and isinstance(
+ stored_hash,
+ (int, np.integer),
+ ):
+ current_hash = _legacy_subset_hash(cell_indices, feature_indices)
+ stored_hash = int(stored_hash)
+ else:
+ raise ValueError("Legacy graph selection provenance has an invalid type")
+ if current_hash != stored_hash:
+ raise ValueError(
+ "cell_key or feat_key no longer matches the legacy graph selection"
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class AssayState:
+ assay: str
+ cell_key: str
+ feat_key: str
+ normalized: ArtifactRef | None = None
+ feature_scaling: ArtifactRef | None = None
+ reduction: ArtifactRef | None = None
+ batch_correction: ArtifactRef | None = None
+ ann_index: ArtifactRef | None = None
+ embedding_initialization: ArtifactRef | None = None
+ neighbors: ArtifactRef | None = None
+ connectivity_map: ArtifactRef | None = None
+ named_results: Mapping[str, ArtifactRef] = field(default_factory=dict)
+
+ def __post_init__(self) -> None:
+ if not self.assay or "/" in self.assay:
+ raise ValueError("AssayState requires a valid assay name")
+ if not self.cell_key or not self.feat_key:
+ raise ValueError("AssayState requires cell_key and feat_key")
+ for field_name, expected_kind in _GRAPH_REF_KINDS.items():
+ ref = getattr(self, field_name)
+ if ref is not None:
+ self._validate_ref(ref, expected_kind, field_name)
+ for name, ref in self.named_results.items():
+ if _RESULT_NAME_PATTERN.fullmatch(name) is None:
+ raise ValueError("named_results keys must be snake_case identifiers")
+ self._validate_ref(ref, ref.kind, f"named_results[{name!r}]")
+ expected_named_kind = _NAMED_RESULT_KINDS.get(name)
+ if expected_named_kind is not None and ref.kind != expected_named_kind:
+ raise ValueError(
+ f"named_results[{name!r}] requires kind {expected_named_kind!r}"
+ )
+ object.__setattr__(
+ self,
+ "named_results",
+ MappingProxyType(dict(self.named_results)),
+ )
+
+ def _validate_ref(
+ self,
+ ref: ArtifactRef,
+ expected_kind: str,
+ field_name: str,
+ ) -> None:
+ if ref.scope != "assay" or ref.assay != self.assay:
+ raise ValueError(f"{field_name} must reference assay {self.assay!r}")
+ if ref.kind != expected_kind:
+ raise ValueError(
+ f"{field_name} requires kind {expected_kind!r}, got {ref.kind!r}"
+ )
+
+ def matches(self, cell_key: str, feat_key: str) -> bool:
+ return self.cell_key == cell_key and self.feat_key == feat_key
+
+ def to_dict(self) -> dict[str, Any]:
+ value: dict[str, Any] = {
+ "assay": self.assay,
+ "cell_key": self.cell_key,
+ "feat_key": self.feat_key,
+ "named_results": {
+ name: ref.to_dict() for name, ref in sorted(self.named_results.items())
+ },
+ }
+ for field_name in _GRAPH_REF_KINDS:
+ ref = getattr(self, field_name)
+ value[field_name] = ref.to_dict() if ref is not None else None
+ return value
+
+ @classmethod
+ def from_dict(cls, value: Mapping[str, Any]) -> "AssayState":
+ refs: dict[str, ArtifactRef | None] = {}
+ for field_name in _GRAPH_REF_KINDS:
+ raw_ref = value.get(field_name)
+ if raw_ref is None:
+ refs[field_name] = None
+ elif isinstance(raw_ref, Mapping):
+ refs[field_name] = ArtifactRef.from_dict(raw_ref)
+ else:
+ raise TypeError(f"{field_name} must be an artifact reference or null")
+ raw_named = value.get("named_results", {})
+ if not isinstance(raw_named, Mapping):
+ raise TypeError("named_results must be a mapping")
+ named_results = {}
+ for name, raw_ref in raw_named.items():
+ if not isinstance(name, str) or not isinstance(raw_ref, Mapping):
+ raise TypeError("Every named result must be a named artifact reference")
+ named_results[name] = ArtifactRef.from_dict(raw_ref)
+ assay = value.get("assay")
+ cell_key = value.get("cell_key")
+ feat_key = value.get("feat_key")
+ if (
+ not isinstance(assay, str)
+ or not isinstance(cell_key, str)
+ or not isinstance(feat_key, str)
+ ):
+ raise TypeError("Assay state assay, cell_key, and feat_key must be strings")
+ return cls(
+ assay=assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ named_results=named_results,
+ **refs,
+ )
+
+
+def assay_state_path(assay: str) -> str:
+ if not assay or "/" in assay:
+ raise ValueError("Invalid assay name")
+ return f"{assay}/state"
+
+
+def read_assay_state(root: zarr.Group, assay: str) -> AssayState | None:
+ path = assay_state_path(assay)
+ if path not in root:
+ return None
+ group = as_zarr_group(root[path], name=path)
+ if "state" not in group.attrs:
+ return None
+ value = group.attrs["state"]
+ if not isinstance(value, Mapping):
+ raise TypeError(f"Assay state at {path} must be a mapping")
+ state = AssayState.from_dict(value)
+ if state.assay != assay:
+ raise ValueError(f"Assay state at {path} names assay {state.assay!r}")
+ return state
+
+
+def named_result_mismatch(
+ root: zarr.Group,
+ name: str,
+ ref: ArtifactRef,
+ state: AssayState,
+) -> str | None:
+ """Report why a named result cannot belong to ``state``.
+
+ Returns None when the handle fits the chain. Both the write path and the
+ carry-forward in ``_artifact_chain_state`` consult this, so a handle can
+ never be preserved into a state that the write would then reject.
+ """
+ if name != "mapping_reference":
+ return None
+ status = require_complete_artifact(root, ref)
+ expected = {
+ "reduction": state.reduction,
+ "ann_index": state.ann_index,
+ "neighbors": state.neighbors,
+ }
+ inputs = status.inputs or {}
+ if "batch_correction" in inputs:
+ expected["batch_correction"] = state.batch_correction
+ elif state.batch_correction is not None:
+ return "Plain PCA mapping reference cannot select batch correction"
+ missing = [
+ input_name for input_name, input_ref in expected.items() if input_ref is None
+ ]
+ if missing:
+ return "Mapping reference state is missing " + ", ".join(missing)
+ for input_name, input_ref in expected.items():
+ assert input_ref is not None
+ if inputs.get(input_name) != input_ref.to_dict():
+ return f"Mapping reference input {input_name!r} does not match AssayState"
+ return None
+
+
+def write_assay_state(root: zarr.Group, state: AssayState) -> None:
+ for field_name in _GRAPH_REF_KINDS:
+ ref = getattr(state, field_name)
+ if ref is None:
+ continue
+ require_complete_artifact(root, ref)
+ for name, ref in state.named_results.items():
+ require_complete_artifact(root, ref)
+ if ref.kind == "imported_coordinates":
+ validate_imported_coordinates_artifact(root, ref)
+ reason = named_result_mismatch(root, name, ref, state)
+ if reason is not None:
+ raise ValueError(reason)
+ if state.connectivity_map is not None:
+ _stored_assay_graph(root, state)
+ path = assay_state_path(state.assay)
+ group = group_at(root, path) if path in root else root.create_group(path)
+ group.attrs["state"] = state.to_dict()
+
+
+def _complete_path(root: zarr.Group, ref: ArtifactRef, field_name: str) -> str:
+ try:
+ return require_complete_artifact(root, ref).path
+ except KeyError as exc:
+ raise KeyError(
+ f"Assay state {field_name} artifact does not exist: {artifact_path(ref)}"
+ ) from exc
+ except RuntimeError as exc:
+ raise RuntimeError(
+ f"Assay state {field_name} artifact is incomplete: {artifact_path(ref)}"
+ ) from exc
+
+
+def normalized_path_from_state(
+ root: zarr.Group,
+ assay: str,
+ cell_key: str,
+ feat_key: str,
+) -> str | None:
+ state = read_assay_state(root, assay)
+ if state is None or not state.matches(cell_key, feat_key):
+ return None
+ if state.normalized is None:
+ raise KeyError("AssayState has no selected normalized artifact")
+ return _complete_path(root, state.normalized, "normalized")
+
+
+def embedding_initialization_path_from_state(
+ root: zarr.Group,
+ assay: str,
+ cell_key: str,
+ feat_key: str,
+) -> str | None:
+ state = read_assay_state(root, assay)
+ if state is None or not state.matches(cell_key, feat_key):
+ return None
+ if state.embedding_initialization is None:
+ raise KeyError(
+ "AssayState has no selected embedding initialization; call "
+ "build_embedding_initialization or pass ini_embed"
+ )
+ return _complete_path(
+ root,
+ state.embedding_initialization,
+ "embedding_initialization",
+ )
+
+
+def _parameters(root: zarr.Group, ref: ArtifactRef | None) -> dict[str, Any]:
+ if ref is None:
+ return {}
+ status = inspect_artifact(root, ref)
+ if not status.exists or not status.complete:
+ return {}
+ return status.parameters or {}
+
+
+def _input_ref(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ name: str,
+ *,
+ require_input_complete: bool = True,
+) -> ArtifactRef:
+ status = require_complete_artifact(root, ref)
+ inputs = status.inputs or {}
+ value = inputs.get(name)
+ if not isinstance(value, Mapping):
+ raise ValueError(f"{ref.kind} artifact has no {name!r} artifact input")
+ input_ref = ArtifactRef.from_dict(value)
+ if require_input_complete:
+ require_complete_artifact(root, input_ref)
+ return input_ref
+
+
+def resolve_stored_graph_input(
+ root: zarr.Group,
+ graph_loc: str,
+) -> ArtifactRef | dict[str, str]:
+ try:
+ return parse_artifact_path(graph_loc)
+ except ValueError:
+ graph_group = group_at(root, graph_loc)
+ return {
+ "legacy_graph_fingerprint": fingerprint_stored_arrays(
+ graph_group,
+ ("edges", "weights"),
+ )
+ }
+
+
+@dataclass(frozen=True, slots=True)
+class GraphSelection:
+ """A resolved graph and the keys that name the columns written from it."""
+
+ graph_loc: str
+ graph_input: ArtifactRef | dict[str, str]
+ from_assay: str
+ cell_key: str
+ feat_key: str
+ integrated_label: str | None
+
+ @property
+ def output_assay(self) -> str:
+ """Prefix used for cell-metadata columns written from this graph."""
+ return self.integrated_label or self.from_assay
+
+
+def integrated_graph_label(store: Any, ref: ArtifactRef) -> str:
+ """Return the label an integrated-graph artifact is registered under."""
+ index_path = store._integratedGraphsLoc
+ labels: list[str] = []
+ if index_path in store.zw:
+ index_group = as_zarr_group(store.zw[index_path], name=index_path)
+ raw_artifacts = index_group.attrs.get("artifacts", {})
+ if "artifacts" in index_group.attrs and not isinstance(raw_artifacts, dict):
+ raise RuntimeError("Integrated graph artifact index is invalid")
+ if isinstance(raw_artifacts, dict):
+ for label, raw_ref in raw_artifacts.items():
+ if not isinstance(raw_ref, dict):
+ raise RuntimeError(
+ f"Integrated graph index for {label!r} is invalid"
+ )
+ if ArtifactRef.from_dict(raw_ref) == ref:
+ labels.append(str(label))
+ if not labels:
+ raise KeyError("Integrated graph artifact is not registered under a label")
+ if len(labels) > 1:
+ raise ValueError(
+ "Integrated graph artifact is shared by labels "
+ f"{', '.join(sorted(labels))}; pass integrated_graph to choose one"
+ )
+ return labels[0]
+
+
+def resolve_graph_selection(
+ store: Any,
+ graph: ArtifactRef | None,
+ *,
+ from_assay: str | None,
+ cell_key: str | None,
+ feat_key: str | None,
+ integrated_graph: str | None = None,
+) -> GraphSelection:
+ """Resolve which graph an operation reads and how it names its outputs.
+
+ Omitting ``graph`` keeps the implicit contract: the current analysis chain
+ of the assay, or the integrated graph named by ``integrated_graph``.
+ Passing ``graph`` reads that connectivity map or integrated graph instead
+ and takes the assay, cell key, and feature key from the artifact.
+ """
+ if graph is not None and integrated_graph is not None:
+ raise ValueError("Pass either graph or integrated_graph, not both")
+ integrated_label = integrated_graph
+ if graph is None:
+ resolved_assay, resolved_cell_key, resolved_feat_key = store._get_latest_keys(
+ from_assay, cell_key, feat_key
+ )
+ if integrated_graph is None:
+ graph_loc = store.get_latest_graph_loc(
+ resolved_assay,
+ resolved_cell_key,
+ resolved_feat_key,
+ )
+ else:
+ graph_loc = store._resolve_integrated_graph_path(integrated_graph)
+ if graph_loc not in store.zw:
+ raise KeyError(
+ f"An integrated graph with label {integrated_graph!r} does not exist"
+ )
+ elif not isinstance(graph, ArtifactRef):
+ raise TypeError("graph must be an artifact reference")
+ elif graph.kind == "integrated_graph":
+ graph_loc = str(require_complete_artifact(store.zw, graph).path)
+ integrated_label = integrated_graph_label(store, graph)
+ resolved_assay, resolved_cell_key, resolved_feat_key = store._get_latest_keys(
+ from_assay, cell_key, feat_key
+ )
+ elif graph.kind == "connectivity_map":
+ stored = stored_assay_graph_from_ref(store.zw, graph)
+ for name, requested, resolved in (
+ ("from_assay", from_assay, stored.from_assay),
+ ("cell_key", cell_key, stored.cell_key),
+ ("feat_key", feat_key, stored.feat_key),
+ ):
+ if requested is not None and requested != resolved:
+ raise ValueError(
+ f"{name} {requested!r} does not match the graph, "
+ f"which was built with {resolved!r}"
+ )
+ resolved_assay = stored.from_assay
+ resolved_cell_key = stored.cell_key
+ resolved_feat_key = stored.feat_key
+ graph_loc = stored.paths.cell_graph_group_path
+ else:
+ raise ValueError(
+ "graph must reference a connectivity map or an integrated graph"
+ )
+ return GraphSelection(
+ graph_loc=graph_loc,
+ graph_input=resolve_stored_graph_input(store.zw, graph_loc),
+ from_assay=resolved_assay,
+ cell_key=resolved_cell_key,
+ feat_key=resolved_feat_key,
+ integrated_label=integrated_label,
+ )
+
+
+def _require_input(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ name: str,
+ expected: ArtifactRef,
+) -> None:
+ actual = _input_ref(root, ref, name)
+ if actual != expected:
+ raise ValueError(
+ f"{ref.kind} artifact input {name!r} does not match AssayState"
+ )
+
+
+def _stored_assay_graph(root: zarr.Group, state: AssayState) -> StoredAssayGraph:
+ required = {
+ "normalized": state.normalized,
+ "feature_scaling": state.feature_scaling,
+ "reduction": state.reduction,
+ "ann_index": state.ann_index,
+ "neighbors": state.neighbors,
+ "connectivity_map": state.connectivity_map,
+ }
+ if any(ref is None for ref in required.values()):
+ missing = [name for name, ref in required.items() if ref is None]
+ raise KeyError(
+ f"AssayState has no complete selected graph; missing {', '.join(missing)}"
+ )
+ assert state.normalized is not None
+ assert state.feature_scaling is not None
+ assert state.reduction is not None
+ assert state.ann_index is not None
+ assert state.neighbors is not None
+ assert state.connectivity_map is not None
+ _require_input(
+ root,
+ state.reduction,
+ "normalized",
+ state.normalized,
+ )
+ _require_input(
+ root,
+ state.feature_scaling,
+ "normalized",
+ state.normalized,
+ )
+ _require_input(
+ root,
+ state.reduction,
+ "feature_scaling",
+ state.feature_scaling,
+ )
+ coordinates = state.reduction
+ if state.batch_correction is not None:
+ _require_input(
+ root,
+ state.batch_correction,
+ "reduction",
+ state.reduction,
+ )
+ coordinates = state.batch_correction
+ _require_input(root, state.ann_index, "coordinates", coordinates)
+ _require_input(root, state.neighbors, "ann_index", state.ann_index)
+ _require_input(root, state.neighbors, "coordinates", coordinates)
+ _require_input(
+ root,
+ state.connectivity_map,
+ "neighbors",
+ state.neighbors,
+ )
+ if state.embedding_initialization is not None:
+ _require_input(
+ root,
+ state.embedding_initialization,
+ "reduction",
+ state.reduction,
+ )
+ paths = {
+ name: _complete_path(root, ref, name)
+ for name, optional_ref in required.items()
+ if (ref := optional_ref) is not None
+ }
+ initialization_path = (
+ _complete_path(
+ root,
+ state.embedding_initialization,
+ "embedding_initialization",
+ )
+ if state.embedding_initialization is not None
+ else None
+ )
+ reduction_status = (
+ inspect_artifact(root, state.reduction) if state.reduction is not None else None
+ )
+ reduction = (
+ reduction_status.parameters or {} if reduction_status is not None else {}
+ )
+ reduction_execution = (
+ inspect_artifact(root, state.reduction).execution_options or {}
+ if state.reduction is not None
+ else {}
+ )
+ ann_index = _parameters(root, state.ann_index)
+ neighbors = _parameters(root, state.neighbors)
+ connectivity = _parameters(root, state.connectivity_map)
+ initialization = _parameters(root, state.embedding_initialization)
+ reduction_operation = (
+ reduction_status.operation if reduction_status is not None else None
+ )
+ reduction_method = (
+ {
+ "run_pca": "pca",
+ "run_lsi": "lsi",
+ "run_custom_reduction": "custom",
+ }.get(reduction_operation)
+ if reduction_operation is not None
+ else None
+ )
+ if reduction_method is None:
+ reduction_method = _optional_str(reduction.get("reduction_method"))
+ reduction_dims: int | None
+ if reduction_operation in {"run_pca", "run_lsi"}:
+ reduction_dims = _required_int(
+ reduction,
+ "dims",
+ "reduction",
+ )
+ else:
+ reduction_dims = _optional_int(reduction.get("dims"))
+ if reduction_dims is None and state.reduction is not None:
+ reduction_group = artifact_group(root, state.reduction)
+ if "loadings" in reduction_group:
+ reduction_dims = int(
+ as_zarr_array(
+ reduction_group["loadings"],
+ name="loadings",
+ ).shape[1]
+ )
+ pca_cell_key: str | None
+ feat_scaling: bool | None
+ if reduction_operation == "run_pca":
+ pca_cell_key = _required_str(
+ reduction_execution,
+ "pca_cell_key",
+ "reduction execution",
+ )
+ feat_scaling = _required_bool(
+ reduction,
+ "feat_scaling",
+ "reduction",
+ )
+ elif reduction_operation in {"run_lsi", "run_custom_reduction"}:
+ pca_cell_key = None
+ feat_scaling = False
+ else:
+ pca_cell_key = _optional_str(
+ reduction_execution.get(
+ "pca_cell_key",
+ reduction.get("pca_cell_key"),
+ )
+ )
+ feat_scaling = _optional_bool(reduction.get("feat_scaling"))
+ stored = StoredAssayGraph(
+ paths=AssayGraphPaths(
+ normalized_group_path=paths["normalized"],
+ reduction_group_path=paths["reduction"],
+ neighbor_index_group_path=paths["ann_index"],
+ nearest_neighbors_group_path=paths["neighbors"],
+ cell_graph_group_path=paths["connectivity_map"],
+ kmeans_initialization_group_path=initialization_path,
+ ),
+ from_assay=state.assay,
+ cell_key=state.cell_key,
+ feat_key=state.feat_key,
+ reduction_method=reduction_method,
+ dims=reduction_dims,
+ pca_cell_key=pca_cell_key,
+ ann_metric=_required_str(ann_index, "ann_metric", "ann_index"),
+ ann_efc=_required_int(ann_index, "ann_efc", "ann_index"),
+ ann_ef=_required_int(ann_index, "ann_ef", "ann_index"),
+ ann_m=_required_int(ann_index, "ann_m", "ann_index"),
+ rand_state=_required_int(ann_index, "rand_state", "ann_index"),
+ k=_required_int(neighbors, "k", "neighbors"),
+ local_connectivity=_required_float(
+ connectivity,
+ "local_connectivity",
+ "connectivity_map",
+ ),
+ bandwidth=_required_float(
+ connectivity,
+ "bandwidth",
+ "connectivity_map",
+ ),
+ feat_scaling=feat_scaling,
+ n_centroids=(
+ _required_int(
+ initialization,
+ "n_centroids",
+ "embedding_initialization",
+ )
+ if state.embedding_initialization is not None
+ else None
+ ),
+ )
+ return stored
+
+
+def stored_assay_graph_from_state(
+ root: zarr.Group,
+ assay: str,
+ cell_key: str,
+ feat_key: str,
+) -> StoredAssayGraph | None:
+ state = read_assay_state(root, assay)
+ if state is None or not state.matches(cell_key, feat_key):
+ return None
+ stored = _stored_assay_graph(root, state)
+ assert state.connectivity_map is not None
+ validate_artifact_graph_selection(
+ root,
+ state.connectivity_map,
+ state.cell_key,
+ state.feat_key,
+ )
+ return stored
+
+
+def _require_artifact_ref(
+ ref: ArtifactRef,
+ *,
+ kind: str,
+ scope: str,
+ assay: str | None,
+) -> None:
+ if ref.kind != kind or ref.scope != scope or ref.assay != assay:
+ raise ArtifactSelectionError(
+ f"Expected {scope}-scoped {kind} artifact",
+ code="artifact_reference_mismatch",
+ context={
+ "expected_scope": scope,
+ "expected_assay": assay,
+ "expected_kind": kind,
+ "actual_scope": ref.scope,
+ "actual_assay": ref.assay,
+ "actual_kind": ref.kind,
+ "artifact_id": ref.artifact_id,
+ },
+ )
+
+
+def _selection_error_context(
+ ref: ArtifactRef,
+ *,
+ table_path: str,
+ column: str,
+) -> dict[str, _SelectionContextValue]:
+ return {
+ "scope": ref.scope,
+ "assay": ref.assay,
+ "kind": ref.kind,
+ "artifact_id": ref.artifact_id,
+ "table": table_path,
+ "column": column,
+ }
+
+
+def _validate_selection_artifact(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ *,
+ kind: str,
+ scope: str,
+ assay: str | None,
+ table_path: str,
+ column: str,
+) -> None:
+ _require_artifact_ref(
+ ref,
+ kind=kind,
+ scope=scope,
+ assay=assay,
+ )
+ status = inspect_artifact(root, ref)
+ context = _selection_error_context(
+ ref,
+ table_path=table_path,
+ column=column,
+ )
+ if not status.exists:
+ raise ArtifactSelectionError(
+ f"{kind} artifact does not exist",
+ code="artifact_missing",
+ context=context,
+ )
+ if not status.complete:
+ raise ArtifactSelectionError(
+ f"{kind} artifact is incomplete",
+ code="artifact_incomplete",
+ context=context,
+ )
+ if table_path not in root:
+ raise ArtifactSelectionError(
+ f"Selection table {table_path!r} is unavailable",
+ code="selection_table_missing",
+ context=context,
+ )
+ table = group_at(root, table_path)
+ if column not in table:
+ raise ArtifactSelectionError(
+ f"Selection source column {column!r} is unavailable",
+ code="selection_column_missing",
+ context=context,
+ )
+ if "ids" not in table:
+ raise ArtifactSelectionError(
+ "Selection row identifier column 'ids' is unavailable",
+ code="selection_row_ids_missing",
+ context=context,
+ )
+ selection_group = artifact_group(root, ref)
+ if "values" not in selection_group:
+ raise ArtifactSelectionError(
+ f"{kind} artifact has no values",
+ code="selection_values_missing",
+ context=context,
+ )
+ stored_values = as_zarr_array(selection_group["values"], name="values")
+ current_values = as_zarr_array(table[column], name=column)
+ row_ids = as_zarr_array(table["ids"], name="ids")
+ expected_row_ids = (status.inputs or {}).get("ordered_row_ids_fingerprint")
+ if not isinstance(
+ expected_row_ids, str
+ ) or expected_row_ids != fingerprint_stored_strings(row_ids):
+ raise ArtifactSelectionError(
+ f"{kind} row identity does not match its metadata table",
+ code="row_identity_mismatch",
+ context=context,
+ )
+ if (
+ stored_values.ndim != 1
+ or np.dtype(stored_values.dtype) != np.dtype(bool)
+ or current_values.ndim != 1
+ or np.dtype(current_values.dtype) != np.dtype(bool)
+ or stored_values.shape != current_values.shape
+ ):
+ raise ArtifactSelectionError(
+ f"Selection source column {column!r} no longer matches its artifact",
+ code="selection_values_changed",
+ context=context,
+ )
+ block_rows = min(
+ row_band(array_geometry(stored_values), unit="chunk", fallback=1),
+ row_band(array_geometry(current_values), unit="chunk", fallback=1),
+ )
+ for start in range(0, int(stored_values.shape[0]), block_rows):
+ stop = min(start + block_rows, int(stored_values.shape[0]))
+ if not np.array_equal(
+ np.asarray(stored_values[start:stop], dtype=bool),
+ np.asarray(current_values[start:stop], dtype=bool),
+ ):
+ raise ArtifactSelectionError(
+ f"Selection source column {column!r} no longer matches its artifact",
+ code="selection_values_changed",
+ context=context,
+ )
+
+
+def validate_cell_selection_artifact(
+ root: zarr.Group,
+ selection: ArtifactRef,
+ cell_key: str,
+) -> None:
+ _validate_selection_artifact(
+ root,
+ selection,
+ kind="cell_selection",
+ scope="datastore",
+ assay=None,
+ table_path="cellData",
+ column=cell_key,
+ )
+
+
+def _stored_value_fingerprint(array: zarr.Array) -> str:
+ builder = ValueFingerprintBuilder()
+ builder.begin_array("values", array.shape, array.dtype)
+ block_rows = row_band(array_geometry(array), unit="chunk", fallback=1)
+ for start in range(0, int(array.shape[0]), block_rows):
+ stop = min(start + block_rows, int(array.shape[0]))
+ block = np.asarray(array[start:stop])
+ builder.update_array_block(
+ "values",
+ (start,) + (0,) * (array.ndim - 1),
+ block,
+ )
+ builder.end_array("values")
+ return builder.hexdigest()
+
+
+def _payload_fingerprint(group: zarr.Group, name: str) -> str:
+ array = as_zarr_array(group[name], name=name)
+ if name == "feature_ids":
+ return fingerprint_stored_strings(array)
+ return _stored_value_fingerprint(array)
+
+
+def validate_imported_coordinates_artifact(
+ root: zarr.Group,
+ coordinates: ArtifactRef,
+ *,
+ cell_key: str | None = None,
+) -> None:
+ if coordinates.assay is None:
+ raise ValueError("Imported-coordinate artifact has no assay")
+ _require_artifact_ref(
+ coordinates,
+ kind="imported_coordinates",
+ scope="assay",
+ assay=coordinates.assay,
+ )
+ status = require_complete_artifact(root, coordinates)
+ if status.operation != "import_dimreduc":
+ raise ValueError(
+ "Imported-coordinate artifact operation must be 'import_dimreduc'"
+ )
+ execution = status.execution_options or {}
+ stored_cell_key = execution.get("cell_key")
+ if not isinstance(stored_cell_key, str) or not stored_cell_key:
+ raise ValueError("Imported-coordinate artifact has no cell selection key")
+ if cell_key is not None and cell_key != stored_cell_key:
+ raise ValueError(
+ f"cell_key {cell_key!r} does not match imported coordinates "
+ f"built for {stored_cell_key!r}"
+ )
+ block_rows = execution.get("block_rows")
+ if (
+ isinstance(block_rows, bool)
+ or not isinstance(block_rows, int | np.integer)
+ or int(block_rows) < 1
+ ):
+ raise ValueError("Imported-coordinate block_rows is invalid")
+ selection = _input_ref(
+ root,
+ coordinates,
+ "cell_selection",
+ require_input_complete=False,
+ )
+ validate_cell_selection_artifact(root, selection, stored_cell_key)
+
+ inputs = status.inputs or {}
+ source_digest = inputs.get("source_digest")
+ if (
+ not isinstance(source_digest, Mapping)
+ or set(source_digest) != {"bytes_hex"}
+ or not isinstance(source_digest.get("bytes_hex"), str)
+ or len(source_digest["bytes_hex"]) != 64
+ or source_digest["bytes_hex"].lower() != source_digest["bytes_hex"]
+ ):
+ raise ValueError("Imported-coordinate source digest is missing")
+ try:
+ bytes.fromhex(source_digest["bytes_hex"])
+ except ValueError as exc:
+ raise ValueError(
+ "Imported-coordinate source digest is not hexadecimal"
+ ) from exc
+
+ payload_fingerprints = inputs.get("payload_fingerprints")
+ if not isinstance(payload_fingerprints, Mapping) or not all(
+ isinstance(name, str)
+ and isinstance(fingerprint, str)
+ and len(fingerprint) == 64
+ for name, fingerprint in payload_fingerprints.items()
+ ):
+ raise ValueError("Imported-coordinate payload fingerprints are malformed")
+
+ group = artifact_group(root, coordinates)
+ if "data" not in group:
+ raise ValueError("Imported-coordinate artifact has no data array")
+ data = as_zarr_array(group["data"], name="data")
+ if data.ndim != 2 or int(data.shape[0]) < 1 or np.dtype(data.dtype).kind != "f":
+ raise ValueError("Imported-coordinate data must be a floating-point matrix")
+
+ parameters = status.parameters or {}
+ dimreduc_key = parameters.get("dimreduc_key")
+ if not isinstance(dimreduc_key, str) or not dimreduc_key:
+ raise ValueError("Imported-coordinate source key is missing")
+ dims = parameters.get("dims")
+ if isinstance(dims, bool) or not isinstance(dims, int | np.integer):
+ raise ValueError("Imported-coordinate dimensions are missing")
+ if int(dims) < 1 or int(data.shape[1]) != int(dims):
+ raise ValueError("Imported-coordinate dimensions do not match data")
+ role = parameters.get("role")
+ if (
+ not isinstance(role, str)
+ or not role
+ or role.lower() != role
+ or role in {"umap", "tsne"}
+ ):
+ raise ValueError("Imported-coordinate role is invalid")
+
+ selection_group = artifact_group(root, selection)
+ mask = as_zarr_array(selection_group["values"], name="values")
+ cell_data = group_at(root, "cellData")
+ row_ids = as_zarr_array(cell_data["ids"], name="ids")
+ selected_fingerprint, selected_count = fingerprint_selected_stored_strings(
+ row_ids,
+ mask,
+ )
+ if int(data.shape[0]) != selected_count:
+ raise ValueError(
+ "Imported-coordinate rows do not match the exact cell selection"
+ )
+ ordered_fingerprint = inputs.get("ordered_cell_ids_fingerprint")
+ if (
+ not isinstance(ordered_fingerprint, str)
+ or ordered_fingerprint != selected_fingerprint
+ ):
+ raise ValueError(
+ "Imported-coordinate cell IDs do not match the selected cell order"
+ )
+
+ optional = {
+ "loadings": parameters.get("loadings_stored"),
+ "feature_ids": parameters.get("feature_ids_stored"),
+ "stdev": parameters.get("stdev_stored"),
+ }
+ for name, stored in optional.items():
+ if not isinstance(stored, bool) or stored != (name in group):
+ raise ValueError(
+ f"Imported-coordinate {name!r} storage flag does not match payload"
+ )
+ if optional["loadings"] != optional["feature_ids"]:
+ raise ValueError(
+ "Imported-coordinate loadings and feature IDs must be stored together"
+ )
+ if "loadings" in group:
+ loadings = as_zarr_array(group["loadings"], name="loadings")
+ feature_ids = as_zarr_array(group["feature_ids"], name="feature_ids")
+ if (
+ loadings.ndim != 2
+ or np.dtype(loadings.dtype).kind != "f"
+ or int(loadings.shape[0]) < 1
+ or int(loadings.shape[1]) != int(dims)
+ or feature_ids.ndim != 1
+ or np.dtype(feature_ids.dtype).kind not in {"O", "S", "U"}
+ or int(feature_ids.shape[0]) != int(loadings.shape[0])
+ ):
+ raise ValueError(
+ "Imported-coordinate loadings and feature IDs are misaligned"
+ )
+ if "stdev" in group:
+ stdev = as_zarr_array(group["stdev"], name="stdev")
+ if (
+ stdev.ndim != 1
+ or np.dtype(stdev.dtype).kind != "f"
+ or tuple(stdev.shape) != (int(dims),)
+ ):
+ raise ValueError("Imported-coordinate stdev does not match dimensions")
+
+ payload_names = ("data", "loadings", "feature_ids", "stdev")
+ stored_payloads = {name for name in payload_names if name in group}
+ if set(payload_fingerprints) != stored_payloads:
+ raise ValueError(
+ "Imported-coordinate payload fingerprints do not match stored payloads"
+ )
+ for name in payload_names:
+ if name not in group:
+ continue
+ expected = payload_fingerprints.get(name)
+ if not isinstance(expected, str):
+ raise ValueError(
+ f"Imported-coordinate payload fingerprint for {name!r} is missing"
+ )
+ if _payload_fingerprint(group, name) != expected:
+ raise ValueError(
+ f"Imported-coordinate payload fingerprint for {name!r} does not match"
+ )
+
+
+def validate_artifact_graph_selection(
+ root: zarr.Group,
+ connectivity_map: ArtifactRef,
+ cell_key: str,
+ feat_key: str,
+) -> None:
+ if connectivity_map.assay is None:
+ raise ValueError("Connectivity-map artifact has no assay")
+ assay = connectivity_map.assay
+ _require_artifact_ref(
+ connectivity_map,
+ kind="connectivity_map",
+ scope="assay",
+ assay=assay,
+ )
+ neighbors = _input_ref(root, connectivity_map, "neighbors")
+ _require_artifact_ref(
+ neighbors,
+ kind="neighbors",
+ scope="assay",
+ assay=assay,
+ )
+ validate_neighbors_artifact_selection(root, neighbors, cell_key, feat_key)
+
+
+def validate_neighbors_artifact_selection(
+ root: zarr.Group,
+ neighbors: ArtifactRef,
+ cell_key: str,
+ feat_key: str,
+) -> None:
+ if neighbors.assay is None:
+ raise ValueError("Neighbors artifact has no assay")
+ assay = neighbors.assay
+ _require_artifact_ref(
+ neighbors,
+ kind="neighbors",
+ scope="assay",
+ assay=assay,
+ )
+ ann_index = _input_ref(root, neighbors, "ann_index")
+ _require_artifact_ref(
+ ann_index,
+ kind="ann_index",
+ scope="assay",
+ assay=assay,
+ )
+ coordinates = _input_ref(root, neighbors, "coordinates")
+ if coordinates.kind == "batch_correction":
+ _require_artifact_ref(
+ coordinates,
+ kind="batch_correction",
+ scope="assay",
+ assay=assay,
+ )
+ reduction = _input_ref(root, coordinates, "reduction")
+ elif coordinates.kind == "reduction":
+ reduction = coordinates
+ elif coordinates.kind == "imported_coordinates":
+ _require_artifact_ref(
+ coordinates,
+ kind="imported_coordinates",
+ scope="assay",
+ assay=assay,
+ )
+ validate_imported_coordinates_artifact(
+ root,
+ coordinates,
+ cell_key=cell_key,
+ )
+ reduction = None
+ else:
+ raise ValueError(
+ "Neighbor coordinates must be reduction, batch_correction, "
+ "or imported_coordinates"
+ )
+ if _input_ref(root, ann_index, "coordinates") != coordinates:
+ raise ValueError("ANN index and neighbors use different coordinates")
+ if reduction is None:
+ return
+ _require_artifact_ref(
+ reduction,
+ kind="reduction",
+ scope="assay",
+ assay=assay,
+ )
+ normalized = _input_ref(root, reduction, "normalized")
+ _require_artifact_ref(
+ normalized,
+ kind="normalized",
+ scope="assay",
+ assay=assay,
+ )
+ validate_normalized_artifact_selection(
+ root,
+ normalized,
+ cell_key,
+ feat_key,
+ )
+
+
+def validate_normalized_artifact_selection(
+ root: zarr.Group,
+ normalized: ArtifactRef,
+ cell_key: str,
+ feat_key: str,
+) -> None:
+ if normalized.assay is None:
+ raise ValueError("Normalized artifact has no assay")
+ assay = normalized.assay
+ _require_artifact_ref(
+ normalized,
+ kind="normalized",
+ scope="assay",
+ assay=assay,
+ )
+ cell_selection = _input_ref(
+ root,
+ normalized,
+ "cell_selection",
+ require_input_complete=False,
+ )
+ feature_selection = _input_ref(
+ root,
+ normalized,
+ "feature_selection",
+ require_input_complete=False,
+ )
+ feature_column = "I" if feat_key == "I" else f"{cell_key}__{feat_key}"
+ validate_cell_selection_artifact(root, cell_selection, cell_key)
+ _validate_selection_artifact(
+ root,
+ feature_selection,
+ kind="feature_selection",
+ scope="assay",
+ assay=assay,
+ table_path=f"{assay}/featureData",
+ column=feature_column,
+ )
+
+
+def stored_assay_graph_from_ref(
+ root: zarr.Group,
+ connectivity_map: ArtifactRef,
+) -> StoredAssayGraph:
+ if connectivity_map.scope != "assay" or connectivity_map.assay is None:
+ raise ValueError("Connectivity-map reference must be assay-scoped")
+ if connectivity_map.kind != "connectivity_map":
+ raise ValueError("Expected a connectivity_map artifact reference")
+ neighbors = _input_ref(root, connectivity_map, "neighbors")
+ ann_index = _input_ref(root, neighbors, "ann_index")
+ coordinates = _input_ref(root, neighbors, "coordinates")
+ if coordinates.kind == "batch_correction":
+ batch_correction = coordinates
+ reduction = _input_ref(root, batch_correction, "reduction")
+ elif coordinates.kind == "reduction":
+ batch_correction = None
+ reduction = coordinates
+ else:
+ raise ValueError("Neighbor coordinates must be reduction or batch_correction")
+ if _input_ref(root, ann_index, "coordinates") != coordinates:
+ raise ValueError("ANN index and neighbors use different coordinates")
+ normalized = _input_ref(root, reduction, "normalized")
+ feature_scaling = _input_ref(root, reduction, "feature_scaling")
+ _require_artifact_ref(
+ feature_scaling,
+ kind="feature_scaling",
+ scope="assay",
+ assay=connectivity_map.assay,
+ )
+ normalized_status = inspect_artifact(root, normalized)
+ normalized_execution = normalized_status.execution_options or {}
+ cell_key = normalized_execution.get("cell_key")
+ feat_key = normalized_execution.get("feat_key")
+ if not isinstance(cell_key, str) or not isinstance(feat_key, str):
+ raise ValueError("Normalized artifact selection keys are missing")
+ state = AssayState(
+ assay=connectivity_map.assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ normalized=normalized,
+ feature_scaling=feature_scaling,
+ reduction=reduction,
+ batch_correction=batch_correction,
+ ann_index=ann_index,
+ neighbors=neighbors,
+ connectivity_map=connectivity_map,
+ )
+ stored = _stored_assay_graph(root, state)
+ validate_artifact_graph_selection(
+ root,
+ connectivity_map,
+ cell_key,
+ feat_key,
+ )
+ return stored
+
+
+def _optional_str(value: Any) -> str | None:
+ if value is None:
+ return None
+ if not isinstance(value, str):
+ raise TypeError("Optional provenance value must be a string")
+ return value
+
+
+def _optional_int(value: Any) -> int | None:
+ if value is None:
+ return None
+ if isinstance(value, bool) or not isinstance(value, (int, np.integer)):
+ raise TypeError("Optional provenance value must be an integer")
+ return int(value)
+
+
+def _optional_bool(value: Any) -> bool | None:
+ if value is None:
+ return None
+ if not isinstance(value, (bool, np.bool_)):
+ raise TypeError("Optional provenance value must be boolean")
+ return bool(value)
+
+
+def _required_value(
+ values: Mapping[str, Any],
+ name: str,
+ owner: str,
+) -> Any:
+ if name not in values:
+ raise ValueError(f"{owner} provenance is missing {name!r}")
+ return values[name]
+
+
+def _required_str(
+ values: Mapping[str, Any],
+ name: str,
+ owner: str,
+) -> str:
+ value = _required_value(values, name, owner)
+ if not isinstance(value, str):
+ raise TypeError(f"{owner} provenance {name!r} must be a string")
+ return value
+
+
+def _required_int(
+ values: Mapping[str, Any],
+ name: str,
+ owner: str,
+) -> int:
+ value = _required_value(values, name, owner)
+ if isinstance(value, bool) or not isinstance(value, (int, np.integer)):
+ raise TypeError(f"{owner} provenance {name!r} must be an integer")
+ return int(value)
+
+
+def _required_float(
+ values: Mapping[str, Any],
+ name: str,
+ owner: str,
+) -> float:
+ value = _required_value(values, name, owner)
+ if isinstance(value, bool) or not isinstance(
+ value,
+ (int, float, np.integer, np.floating),
+ ):
+ raise TypeError(f"{owner} provenance {name!r} must be numeric")
+ return float(value)
+
+
+def _required_bool(
+ values: Mapping[str, Any],
+ name: str,
+ owner: str,
+) -> bool:
+ value = _required_value(values, name, owner)
+ if not isinstance(value, (bool, np.bool_)):
+ raise TypeError(f"{owner} provenance {name!r} must be boolean")
+ return bool(value)
diff --git a/scarf/harmony.py b/scarf/harmony.py
deleted file mode 100644
index 291bdf1b..00000000
--- a/scarf/harmony.py
+++ /dev/null
@@ -1,322 +0,0 @@
-from functools import partial
-
-import numpy as np
-import pandas as pd
-from sklearn.cluster import KMeans
-
-from .utils import tqdmbar, logger
-
-
-def run_harmony(
- data_mat: np.ndarray,
- meta_data: pd.DataFrame,
- theta=None,
- lamb=None,
- sigma=0.1,
- nclust=None,
- tau=0,
- block_size=0.05,
- max_iter_harmony=50,
- max_iter_kmeans=20,
- epsilon_cluster=1e-5,
- epsilon_harmony=1e-4,
- random_state=0,
- cluster_fn="kmeans",
-):
- """Run Harmony."""
-
- N = data_mat.shape[1]
- if nclust is None:
- nclust = np.min([np.round(N / 30.0), 100]).astype(int)
-
- if type(sigma) is float and nclust > 1:
- sigma = np.repeat(sigma, nclust)
-
- phi = pd.get_dummies(meta_data).to_numpy().T
- phi_n = meta_data.describe().loc["unique"].to_numpy().astype(int)
-
- if theta is None:
- theta = np.repeat([1] * len(phi_n), phi_n)
- elif isinstance(theta, float) or isinstance(theta, int):
- theta = np.repeat([theta] * len(phi_n), phi_n)
- elif len(theta) == len(phi_n):
- theta = np.repeat([theta], phi_n)
-
- assert len(theta) == np.sum(phi_n), "each batch variable must have a theta"
-
- if lamb is None:
- lamb = np.repeat([1] * len(phi_n), phi_n)
- elif isinstance(lamb, float) or isinstance(lamb, int):
- lamb = np.repeat([lamb] * len(phi_n), phi_n)
- elif len(lamb) == len(phi_n):
- lamb = np.repeat([lamb], phi_n)
-
- assert len(lamb) == np.sum(phi_n), "each batch variable must have a lambda"
-
- # Number of items in each category.
- N_b = phi.sum(axis=1)
- # Proportion of items in each category.
- Pr_b = N_b / N
-
- if tau > 0:
- theta = theta * (1 - np.exp(-((N_b / (nclust * tau)) ** 2)))
-
- lamb_mat = np.diag(np.insert(lamb, 0, 0))
-
- phi_moe = np.vstack((np.repeat(1, N), phi))
-
- np.random.seed(random_state)
-
- ho = Harmony(
- data_mat,
- phi,
- phi_moe,
- Pr_b,
- sigma,
- theta,
- max_iter_harmony,
- max_iter_kmeans,
- epsilon_cluster,
- epsilon_harmony,
- nclust,
- block_size,
- lamb_mat,
- random_state,
- cluster_fn,
- )
-
- return ho.result()
-
-
-class Harmony(object):
- def __init__(
- self,
- Z,
- Phi,
- Phi_moe,
- Pr_b,
- sigma,
- theta,
- max_iter_harmony,
- max_iter_kmeans,
- epsilon_kmeans,
- epsilon_harmony,
- K,
- block_size,
- lamb,
- random_state=None,
- cluster_fn="kmeans",
- ):
- self.Z_corr = np.array(Z)
- self.Z_orig = np.array(Z)
-
- self.Z_cos = self.Z_orig / self.Z_orig.max(axis=0)
- self.Z_cos = self.Z_cos / np.linalg.norm(self.Z_cos, ord=2, axis=0)
-
- self.Phi = Phi
- self.Phi_moe = Phi_moe
- self.N = self.Z_corr.shape[1]
- self.Pr_b = Pr_b
- self.B = self.Phi.shape[0] # number of batch variables
- self.d = self.Z_corr.shape[0]
- self.window_size = 3
- self.epsilon_kmeans = epsilon_kmeans
- self.epsilon_harmony = epsilon_harmony
-
- self.lamb = lamb
- self.sigma = sigma
- self.sigma_prior = sigma
- self.block_size = block_size
- self.K = K # number of clusters
- self.max_iter_harmony = max_iter_harmony
- self.max_iter_kmeans = max_iter_kmeans
- self.theta = theta
-
- self.objective_harmony = []
- self.objective_kmeans = []
- self.objective_kmeans_dist = []
- self.objective_kmeans_entropy = []
- self.objective_kmeans_cross = []
- self.kmeans_rounds = []
-
- self.allocate_buffers()
- if cluster_fn == "kmeans":
- cluster_fn = partial(Harmony._cluster_kmeans, random_state=random_state)
- self.init_cluster(cluster_fn)
- self.harmonize(self.max_iter_harmony)
-
- def result(self):
- return self.Z_corr
-
- def allocate_buffers(self):
- self._scale_dist = np.zeros((self.K, self.N))
- self.dist_mat = np.zeros((self.K, self.N))
- self.O = np.zeros((self.K, self.B))
- self.E = np.zeros((self.K, self.B))
- self.W = np.zeros((self.B + 1, self.d))
- self.Phi_Rk = np.zeros((self.B + 1, self.N))
-
- @staticmethod
- def _cluster_kmeans(data, K, random_state):
- # Start with cluster centroids
- return (
- KMeans(
- n_clusters=K,
- init="k-means++",
- n_init=10,
- max_iter=25,
- random_state=random_state,
- )
- .fit(data)
- .cluster_centers_
- )
-
- def init_cluster(self, cluster_fn):
- self.Y = cluster_fn(self.Z_cos.T, self.K).T
- # (1) Normalize
- self.Y = self.Y / np.linalg.norm(self.Y, ord=2, axis=0)
- # (2) Assign cluster probabilities
- self.dist_mat = 2 * (1 - np.dot(self.Y.T, self.Z_cos))
- self.R = -self.dist_mat
- self.R = self.R / self.sigma[:, None]
- self.R -= np.max(self.R, axis=0)
- self.R = np.exp(self.R)
- self.R = self.R / np.sum(self.R, axis=0)
- # (3) Batch diversity statistics
- self.E = np.outer(np.sum(self.R, axis=1), self.Pr_b)
- self.O = np.inner(self.R, self.Phi)
- self.compute_objective()
- # Save results
- self.objective_harmony.append(self.objective_kmeans[-1])
-
- def compute_objective(self):
- kmeans_error = np.sum(np.multiply(self.R, self.dist_mat))
- # Entropy
- _entropy = np.sum(safe_entropy(self.R) * self.sigma[:, np.newaxis])
- # Cross Entropy
- x = self.R * self.sigma[:, np.newaxis]
- y = np.tile(self.theta[:, np.newaxis], self.K).T
- z = np.log((self.O + 1) / (self.E + 1))
- w = np.dot(y * z, self.Phi)
- _cross_entropy = np.sum(x * w)
- # Save results
- self.objective_kmeans.append(kmeans_error + _entropy + _cross_entropy)
- self.objective_kmeans_dist.append(kmeans_error)
- self.objective_kmeans_entropy.append(_entropy)
- self.objective_kmeans_cross.append(_cross_entropy)
-
- def harmonize(self, iter_harmony=10):
- converged = False
- for i in tqdmbar(range(1, iter_harmony + 1), desc="Harmonizing batches"):
- # STEP 1: Clustering
- self.cluster()
- # STEP 2: Regress out covariates
- self.Z_cos, self.Z_corr, self.W, self.Phi_Rk = moe_correct_ridge(
- self.Z_orig,
- self.R,
- self.W,
- self.K,
- self.Phi_Rk,
- self.Phi_moe,
- self.lamb,
- )
- # STEP 3: Check for convergence
- converged = self.check_convergence(1)
- if converged:
- break
- if not converged:
- logger.warning("Stopped before convergence")
- return 0
-
- def cluster(self):
- # Z_cos has changed
- # R is assumed to not have changed
- # Update Y to match new integrated data
- self.dist_mat = 2 * (1 - np.dot(self.Y.T, self.Z_cos))
- for i in range(self.max_iter_kmeans):
- # print("kmeans {}".format(i))
- # STEP 1: Update Y
- self.Y = np.dot(self.Z_cos, self.R.T)
- self.Y = self.Y / np.linalg.norm(self.Y, ord=2, axis=0)
- # STEP 2: Update dist_mat
- self.dist_mat = 2 * (1 - np.dot(self.Y.T, self.Z_cos))
- # STEP 3: Update R
- self.update_R()
- # STEP 4: Check for convergence
- self.compute_objective()
- if i > self.window_size:
- converged = self.check_convergence(0)
- if converged:
- break
- self.kmeans_rounds.append(i)
- self.objective_harmony.append(self.objective_kmeans[-1])
- return 0
-
- def update_R(self):
- self._scale_dist = -self.dist_mat
- self._scale_dist = self._scale_dist / self.sigma[:, None]
- self._scale_dist -= np.max(self._scale_dist, axis=0)
- self._scale_dist = np.exp(self._scale_dist)
- # Update cells in blocks
- update_order = np.arange(self.N)
- np.random.shuffle(update_order)
- n_blocks = np.ceil(1 / self.block_size).astype(int)
- blocks = np.array_split(update_order, n_blocks)
- for b in blocks:
- # STEP 1: Remove cells
- self.E -= np.outer(np.sum(self.R[:, b], axis=1), self.Pr_b)
- self.O -= np.dot(self.R[:, b], self.Phi[:, b].T)
- # STEP 2: Recompute R for removed cells
- self.R[:, b] = self._scale_dist[:, b]
- self.R[:, b] = np.multiply(
- self.R[:, b],
- np.dot(
- np.power((self.E + 1) / (self.O + 1), self.theta), self.Phi[:, b]
- ),
- )
- self.R[:, b] = self.R[:, b] / np.linalg.norm(self.R[:, b], ord=1, axis=0)
- # STEP 3: Put cells back
- self.E += np.outer(np.sum(self.R[:, b], axis=1), self.Pr_b)
- self.O += np.dot(self.R[:, b], self.Phi[:, b].T)
- return 0
-
- def check_convergence(self, i_type):
- obj_old = 0.0
- obj_new = 0.0
- # Clustering, compute new window mean
- if i_type == 0:
- okl = len(self.objective_kmeans)
- for i in range(self.window_size):
- obj_old += self.objective_kmeans[okl - 2 - i]
- obj_new += self.objective_kmeans[okl - 1 - i]
- if abs(obj_old - obj_new) / abs(obj_old) < self.epsilon_kmeans:
- return True
- return False
- # Harmony
- if i_type == 1:
- obj_old = self.objective_harmony[-2]
- obj_new = self.objective_harmony[-1]
- e = (obj_old - obj_new) / abs(obj_old)
- logger.info(f"Error after {len(self.objective_harmony)} iterations: {e}")
- if e < self.epsilon_harmony:
- return True
- return False
- return True
-
-
-def safe_entropy(x: np.array):
- y = np.multiply(x, np.log(x))
- y[~np.isfinite(y)] = 0.0
- return y
-
-
-def moe_correct_ridge(Z_orig, R, W, K, Phi_Rk, Phi_moe, lamb):
- Z_corr = Z_orig.copy()
- for i in range(K):
- Phi_Rk = np.multiply(Phi_moe, R[i, :])
- x = np.dot(Phi_Rk, Phi_moe.T) + lamb
- W = np.dot(np.dot(np.linalg.inv(x), Phi_Rk), Z_orig.T)
- W[0, :] = 0 # do not remove the intercept
- Z_corr -= np.dot(W.T, Phi_Rk)
- Z_cos = Z_corr / np.linalg.norm(Z_corr, ord=2, axis=0)
- return Z_cos, Z_corr, W, Phi_Rk
diff --git a/scarf/knn_utils.py b/scarf/knn_utils.py
deleted file mode 100644
index 92606541..00000000
--- a/scarf/knn_utils.py
+++ /dev/null
@@ -1,354 +0,0 @@
-"""Utility functions for running the KNN algorithm."""
-
-from typing import List, Tuple
-
-import numpy as np
-import pandas as pd
-from numba import jit
-from scipy.sparse import csr_matrix, coo_matrix
-
-from .ann import AnnStream
-from .utils import tqdmbar, controlled_compute
-from .writers import create_zarr_dataset
-
-__all__ = [
- "self_query_knn",
- "smoothen_dists",
- "export_knn_to_mtx",
- "merge_graphs",
- "wnn_integration",
-]
-
-
-def self_query_knn(ann_obj: AnnStream, store, chunk_size: int, nthreads: int) -> float:
- """Constructs KNN graph.
-
- Args:
- ann_obj ():
- store ():
- chunk_size ():
- nthreads (): Number of threads to use.
-
- Returns:
- None
- """
-
- def get_transformed_data():
- msg = "Identifying neighbors"
- if ann_obj.harmonizedData is None:
- for _i in tqdmbar(
- ann_obj.data.blocks, desc=msg, total=ann_obj.data.numblocks[0]
- ):
- yield ann_obj.reducer(controlled_compute(_i, nthreads))
- else:
- for _i in tqdmbar(
- ann_obj.harmonizedData.blocks,
- desc=msg,
- total=ann_obj.harmonizedData.numblocks[0],
- ):
- yield controlled_compute(_i, nthreads)
-
- from threadpoolctl import threadpool_limits
-
- n_cells, n_neighbors = ann_obj.nCells, ann_obj.k
- z_knn = create_zarr_dataset(
- store, "indices", (chunk_size,), "u8", (n_cells, n_neighbors)
- )
- z_dist = create_zarr_dataset(
- store, "distances", (chunk_size,), "f8", (n_cells, n_neighbors)
- )
- nsample_start = 0
- tnm = 0 # Number of missed recall
- with threadpool_limits(limits=nthreads):
- for i in get_transformed_data():
- nsample_end = nsample_start + i.shape[0]
- ki, kv, nm = ann_obj.transform_ann(
- i,
- k=n_neighbors,
- self_indices=np.arange(nsample_start, nsample_end),
- )
- z_knn[nsample_start:nsample_end, :] = ki
- z_dist[nsample_start:nsample_end, :] = kv
- nsample_start = nsample_end
- tnm += nm
- recall = ann_obj.data.shape[0] - tnm
- recall = 100 * recall / ann_obj.data.shape[0]
- return recall
-
-
-def _is_umap_version_new():
- import umap
- from packaging import version
-
- if version.parse(umap.__version__) >= version.parse("0.5.0"):
- return True
- else:
- return False
-
-
-def smoothen_dists(store, z_idx, z_dist, lc: float, bw: float, chunk_size: int):
- """Smoothens KNN distances.
-
- Args:
- store ():
- z_idx ():
- z_dist ():
- lc ():
- bw ():
- chunk_size ():
-
- Returns:
- None
- """
- from umap.umap_ import smooth_knn_dist, compute_membership_strengths
-
- umap_is_latest = _is_umap_version_new()
-
- n_cells, n_neighbors = z_idx.shape
- zge = create_zarr_dataset(
- store,
- f"edges",
- (chunk_size * n_neighbors,),
- ("u8", "u8"),
- (n_cells * n_neighbors, 2),
- )
- zgw = create_zarr_dataset(
- store, f"weights", (chunk_size * n_neighbors,), "f8", (n_cells * n_neighbors,)
- )
- last_row = 0
- val_counts = 0
- null_idx = []
- global_min = 1
- for i in tqdmbar(range(0, n_cells, chunk_size), desc="Smoothening KNN distances"):
- if i + chunk_size > n_cells:
- ki, kv = z_idx[i:n_cells, :], z_dist[i:n_cells, :]
- else:
- ki, kv = z_idx[i : i + chunk_size, :], z_dist[i : i + chunk_size, :]
- kv = kv.astype(np.float32, order="C")
- sigmas, rhos = smooth_knn_dist(
- kv, k=n_neighbors, local_connectivity=lc, bandwidth=bw
- )
- if umap_is_latest:
- rows, cols, vals, _ = compute_membership_strengths(ki, kv, sigmas, rhos)
- else:
- rows, cols, vals = compute_membership_strengths(ki, kv, sigmas, rhos)
- rows = rows + last_row
- start = val_counts
- end = val_counts + len(rows)
- last_row = rows[-1] + 1
- val_counts += len(rows)
- zge[start:end, 0] = rows
- zge[start:end, 1] = cols
- zgw[start:end] = vals
-
- # Fixing edges with 0 weights
- # We are doing these steps here to have minimum operations outside
- # the scope of a progress bar
- nidx = vals == 0
- if nidx.sum() > 0:
- min_val = vals[~nidx].min()
- if min_val < global_min:
- global_min = min_val
- null_idx.extend(nidx)
-
- # The whole zarr array needs to copied, modified and written back.
- # Or is this assumption wrong?
- w = zgw[:]
- w[null_idx] = global_min
- zgw[:] = w
- return None
-
-
-def export_knn_to_mtx(mtx: str, csr_graph, batch_size: int = 1000) -> None:
- """Exports KNN matrix in Matrix Market format.
-
- Args:
- mtx:
- csr_graph:
- batch_size:
-
- Returns:
- None
- """
- n_cells = csr_graph.shape[0]
- with open(mtx, "w") as h:
- h.write("%%MatrixMarket matrix coordinate real general\n% Generated by Scarf\n")
- h.write(f"{n_cells} {n_cells} {csr_graph.nnz}\n")
- s = 0
- for e in tqdmbar(
- range(batch_size, n_cells + batch_size, batch_size),
- desc="Saving KNN matrix in MTX format",
- ):
- if e > n_cells:
- e = n_cells
- sg = csr_graph[s:e].tocoo()
- df = pd.DataFrame({"row": sg.row + s + 1, "col": sg.col + 1, "d": sg.data})
- df.to_csv(h, sep=" ", header=False, index=False, mode="a")
- s = e
- if s != n_cells:
- raise ValueError(
- "ERROR: Internal loop count error in export_knn_to_mtx. Please report this bug"
- )
- return None
-
-
-@jit(nopython=True)
-def calc_snn(indices: np.ndarray) -> np.ndarray:
- """Calculates shared nearest neighbour between each node and its neighbour.
-
- Args:
- indices: KNN graph indices
-
- Returns: A numpy matrix of shape (n_cells, n neighbours)
- """
- ncells, nk = indices.shape
- snn = np.zeros((ncells, nk))
- for i in range(ncells):
- for j in range(nk):
- k = indices[i][j]
- snn[i][j] = len(set(indices[i]).intersection(set(indices[k])))
- return snn / (nk - 1)
-
-
-def weight_sort_indices(
- i: np.ndarray, w: np.ndarray, wn: np.ndarray, n: int
-) -> Tuple[np.ndarray, np.ndarray]:
- """Sort the array i and w based on values of wn. Only keep the top n
- values.
-
- Args:
- i: A 1D array of indices
- w: A 1D array of weights
- wn: A 1D array of weights. These weights are used for sorting
- n: Number of neighbours to retain.
-
- Returns: A tuple of two 1D arrays representing sorted and filtered
- indices and their corresponding weights
- """
-
- idx = np.argsort(wn)[::-1]
- i = i[idx]
- w = w[idx]
- # Removing duplicate neighbours
- _, idx = np.unique(i, return_index=True)
- idx = sorted(idx)
- return i[idx][:n], w[idx][:n]
-
-
-def merge_graphs(csr_mats: List[csr_matrix]) -> coo_matrix:
- """Merge multiple graphs of same size and shape such that the merged graph
- have the same size and shape. Edge values are sorted based on their weight
- and the shared neighbours.
-
- Args:
- csr_mats: A list of two or more CSR matrices representing the graphs to be merged.
-
- Returns: A merged graph in CSR matrix form.
- The merged graph has the same number of edges as each input graph
- """
- try:
- assert len(set([x.shape for x in csr_mats])) == 1
- except AssertionError:
- raise ValueError("ERROR: All graphs do not have the same shape.")
- try:
- assert len(set([x.size for x in csr_mats])) == 1
- except AssertionError:
- raise ValueError("ERROR: All graphs do not have the same number of edges")
-
- nk = csr_mats[0][0].indices.shape[0]
- snns = []
- for mat in tqdmbar(csr_mats, desc="Identifying SNNs in graphs"):
- snns.append(
- calc_snn(mat.indices.reshape((mat.shape[0], mat[0].indices.shape[0])))
- )
- col, data = [], []
- for i in tqdmbar(range(csr_mats[0].shape[0]), desc="Merging graph edges"):
- mi = np.hstack([mat[i].indices for mat in csr_mats])
- mwn = np.hstack([mat[i].data + snns[n][i] for n, mat in enumerate(csr_mats)])
- mw = np.hstack([mat[i].data for mat in csr_mats])
- mi, mw = weight_sort_indices(mi, mw, mwn, nk)
- col.extend(mi)
- data.extend(mw)
- s = csr_mats[0].shape
- row = np.repeat(range(s[0]), nk)
- return coo_matrix((data, (row, col)), shape=s)
-
-
-def wnn_integration(
- name1: str, g1: csr_matrix, ld1, name2: str, g2: csr_matrix, ld2, n_threads: int
-):
- """
-
- Args:
- name1:
- g1:
- ld1:
- name2:
- g2:
- ld2:
- n_threads:
-
- Returns:
-
- """
-
- def make_estimates(g, ld, msg=""):
- return np.array(
- [
- ld[g[i].indices].mean(axis=0)
- for i in tqdmbar(range(g.shape[0]), desc=msg)
- ]
- )
-
- def get_kth_l(g, ld, k):
- return ld[[g[x].indices[k] for x in range(g.shape[0])]]
-
- def calc_theta(ld, le, b, c):
- a = np.sqrt(((ld - le) ** 2).sum(axis=1))
- d = a - b
- d[d < 0] = 0
- return np.exp(((-1 * d) / (c - d)).astype(np.float128))
-
- def calc_affinity_ratios(
- g_self, g_other, ld, sigma: int = -2, epsilon: float = 10e-4, name=""
- ):
- l_self = make_estimates(
- g_self, ld, msg=f"({name}) Predicting within modality profile"
- )
- l_cross = make_estimates(
- g_other, ld, msg=f"({name}) Predicting cross modality profile"
- )
-
- b = np.sqrt(((ld - get_kth_l(g_self, ld, 0)) ** 2).sum(axis=1))
- c = np.sqrt(((ld - get_kth_l(g_self, ld, sigma)) ** 2).sum(axis=1))
-
- theta_self = calc_theta(ld, l_self, b, c)
- theta_cross = calc_theta(ld, l_cross, b, c)
-
- return (theta_self / (theta_cross + epsilon)).astype(np.float128)
-
- from threadpoolctl import threadpool_limits
-
- with threadpool_limits(limits=n_threads):
- sr = calc_affinity_ratios(g1, g2, ld1, name=name1)
- sp = calc_affinity_ratios(g2, g1, ld2, name=name2)
-
- wr = np.exp(sr) / (np.exp(sr) + np.exp(sp))
- wp = np.exp(sp) / (np.exp(sr) + np.exp(sp))
-
- nk = g1[0].indices.shape[0]
- col, data = [], []
- for n in tqdmbar(range(g1.shape[0]), desc="Building WNN graph"):
- mixed_k = np.array(sorted(set(g1[n].indices).union(g2[n].indices)))
- dr = np.sqrt(((ld1[n] - ld1[mixed_k]) ** 2).sum(axis=1)) * wr[n]
- dp = np.sqrt(((ld2[n] - ld2[mixed_k]) ** 2).sum(axis=1)) * wp[n]
- w_d = dr + dp
- idx = np.argsort(w_d)[:nk]
- col.append(mixed_k[idx])
- v = w_d[idx]
- data.append(np.exp(-((v - v[0]) / (v - v[0]).mean())))
-
- data = np.hstack(data)
- row = np.repeat(range(g1.shape[0]), nk)
- col = np.hstack(col)
- return coo_matrix((data, (row, col)), shape=g1.shape)
diff --git a/scarf/lineage.py b/scarf/lineage.py
new file mode 100644
index 00000000..70919d5a
--- /dev/null
+++ b/scarf/lineage.py
@@ -0,0 +1,474 @@
+import json
+from collections import defaultdict
+from collections.abc import Iterator, Mapping, Sequence
+from dataclasses import dataclass
+from types import MappingProxyType
+from typing import Any, cast
+
+import networkx as nx
+import zarr
+
+from .storage.artifacts import ArtifactStatus, inspect_artifact
+from .storage.refs import ArtifactLocator, ArtifactRef, ExternalArtifactRef
+
+type LineageTarget = ArtifactRef | Mapping[str, ArtifactRef]
+
+
+@dataclass(frozen=True, slots=True)
+class _ExternalLineageRef:
+ dataset_fingerprint: str
+ source_assay: str
+ ref: ArtifactRef
+
+
+type _LineageLocator = ArtifactLocator | _ExternalLineageRef
+
+
+_DETAIL_ITEM_LIMIT = 12
+_DETAIL_VALUE_LIMIT = 160
+_OMITTED = object()
+
+
+def _ref_sort_key(ref: ArtifactRef) -> tuple[str, str, str, str]:
+ return (ref.scope, ref.assay or "", ref.kind, ref.artifact_id)
+
+
+def _locator_sort_key(
+ locator: _LineageLocator,
+) -> tuple[str, str, str, str, str, str]:
+ if isinstance(locator, ExternalArtifactRef | _ExternalLineageRef):
+ return (
+ "external",
+ locator.dataset_fingerprint,
+ *_ref_sort_key(locator.ref),
+ )
+ return ("local", "", *_ref_sort_key(locator))
+
+
+def _input_path(parts: tuple[str, ...]) -> str:
+ value = ""
+ for part in parts:
+ if part.startswith("["):
+ value += part
+ elif value:
+ value += f".{part}"
+ else:
+ value = part
+ return value or "input"
+
+
+def _artifact_inputs(
+ value: Any,
+ path: tuple[str, ...] = (),
+) -> Iterator[tuple[str, ArtifactLocator]]:
+ if isinstance(value, ExternalArtifactRef):
+ yield _input_path(path), value
+ return
+ if isinstance(value, ArtifactRef):
+ yield _input_path(path), value
+ return
+ if isinstance(value, Mapping):
+ if value.get("type") == "external_artifact":
+ try:
+ external_ref = ExternalArtifactRef.from_dict(value)
+ except (TypeError, ValueError) as exc:
+ location = _input_path(path)
+ raise ValueError(
+ f"Invalid external artifact reference at lineage input {location!r}"
+ ) from exc
+ yield _input_path(path), external_ref
+ return
+ if value.get("type") == "artifact":
+ try:
+ artifact_ref = ArtifactRef.from_dict(value)
+ except (TypeError, ValueError) as exc:
+ location = _input_path(path)
+ raise ValueError(
+ f"Invalid artifact reference at lineage input {location!r}"
+ ) from exc
+ yield _input_path(path), artifact_ref
+ return
+ for key in sorted(value):
+ yield from _artifact_inputs(value[key], (*path, str(key)))
+ return
+ if isinstance(value, Sequence) and not isinstance(value, str | bytes):
+ for index, item in enumerate(value):
+ yield from _artifact_inputs(item, (*path, f"[{index}]"))
+
+
+def _external_value(value: Any) -> Any:
+ if isinstance(value, ArtifactRef | ExternalArtifactRef):
+ return _OMITTED
+ if isinstance(value, Mapping):
+ if value.get("type") in ("artifact", "external_artifact"):
+ return _OMITTED
+ result = {}
+ for key in sorted(value):
+ item = _external_value(value[key])
+ if item is not _OMITTED:
+ result[str(key)] = item
+ return result if result else _OMITTED
+ if isinstance(value, Sequence) and not isinstance(value, str | bytes):
+ result = {}
+ for index, raw_item in enumerate(value):
+ item = _external_value(raw_item)
+ if item is not _OMITTED:
+ result[f"[{index}]"] = item
+ return result if result else _OMITTED
+ return value
+
+
+def _external_inputs(inputs: Mapping[str, Any] | None) -> dict[str, Any]:
+ if inputs is None:
+ return {}
+ value = _external_value(inputs)
+ return {} if value is _OMITTED else cast(dict[str, Any], value)
+
+
+def _normalize_outputs(target: LineageTarget) -> dict[str, ArtifactRef]:
+ if isinstance(target, ArtifactRef):
+ return {"output": target}
+ if not isinstance(target, Mapping):
+ raise TypeError("lineage target must be an ArtifactRef or a named mapping")
+ if not target:
+ raise ValueError("lineage output mapping cannot be empty")
+ outputs = {}
+ for name, ref in target.items():
+ if not isinstance(name, str) or not name:
+ raise TypeError("lineage output names must be non-empty strings")
+ if not isinstance(ref, ArtifactRef):
+ raise TypeError(f"lineage output {name!r} must be an ArtifactRef")
+ outputs[name] = ref
+ return dict(sorted(outputs.items()))
+
+
+def _normalize_external_roots(
+ external_roots: Mapping[str, zarr.Group] | None,
+) -> dict[str, zarr.Group]:
+ if external_roots is None:
+ return {}
+ if not isinstance(external_roots, Mapping):
+ raise TypeError("external_roots must be a mapping or None")
+ roots = {}
+ for fingerprint, root in external_roots.items():
+ if not isinstance(fingerprint, str) or not fingerprint:
+ raise ValueError("External root fingerprints must be non-empty strings")
+ if not isinstance(root, zarr.Group):
+ raise TypeError(
+ f"External root for fingerprint {fingerprint!r} must be a Zarr group"
+ )
+ roots[fingerprint] = root
+ return roots
+
+
+def _inspect_external_artifact(
+ root: zarr.Group,
+ locator: ExternalArtifactRef | _ExternalLineageRef,
+) -> ArtifactStatus:
+ assay_name = (
+ locator.ref.assay
+ if isinstance(locator, ExternalArtifactRef)
+ else locator.source_assay
+ )
+ assert assay_name is not None
+ if assay_name not in root or not isinstance(root[assay_name], zarr.Group):
+ raise ValueError(
+ f"External root for dataset fingerprint "
+ f"{locator.dataset_fingerprint!r} is missing assay group "
+ f"{assay_name!r}"
+ )
+ assay = root[assay_name]
+ received = assay.attrs.get("dataset_fingerprint")
+ if received is None:
+ raise ValueError(
+ f"External root assay {assay_name!r} has no stored "
+ f"dataset_fingerprint; expected {locator.dataset_fingerprint!r}"
+ )
+ if received != locator.dataset_fingerprint:
+ raise ValueError(
+ f"External root assay {assay_name!r} dataset fingerprint mismatch. "
+ f"Expected {locator.dataset_fingerprint!r}, received {received!r}"
+ )
+ return inspect_artifact(root, locator.ref)
+
+
+def _build_graph(
+ root: zarr.Group,
+ outputs: Mapping[str, ArtifactRef],
+ external_roots: Mapping[str, zarr.Group],
+) -> nx.DiGraph:
+ graph = nx.DiGraph()
+ visited: set[_LineageLocator] = set()
+
+ def visit(locator: _LineageLocator) -> None:
+ if locator in visited:
+ return
+ visited.add(locator)
+ if isinstance(locator, ExternalArtifactRef | _ExternalLineageRef):
+ external_root = external_roots.get(locator.dataset_fingerprint)
+ status = (
+ None
+ if external_root is None
+ else _inspect_external_artifact(external_root, locator)
+ )
+ else:
+ status = inspect_artifact(root, locator)
+ graph.add_node(locator, status=status, outputs=())
+ if status is None:
+ return
+ dependencies = sorted(
+ _artifact_inputs(status.inputs),
+ key=lambda item: (item[0], _locator_sort_key(item[1])),
+ )
+ for input_name, input_locator in dependencies:
+ dependency: _LineageLocator = input_locator
+ if isinstance(
+ locator,
+ ExternalArtifactRef | _ExternalLineageRef,
+ ) and isinstance(input_locator, ArtifactRef):
+ source_assay = (
+ locator.ref.assay
+ if isinstance(locator, ExternalArtifactRef)
+ else locator.source_assay
+ )
+ assert source_assay is not None
+ if (
+ input_locator.scope == "assay"
+ and input_locator.assay == source_assay
+ ):
+ dependency = ExternalArtifactRef(
+ dataset_fingerprint=locator.dataset_fingerprint,
+ ref=input_locator,
+ )
+ else:
+ dependency = _ExternalLineageRef(
+ dataset_fingerprint=locator.dataset_fingerprint,
+ source_assay=source_assay,
+ ref=input_locator,
+ )
+ if graph.has_edge(dependency, locator):
+ labels = set(graph.edges[dependency, locator]["inputs"])
+ labels.add(input_name)
+ graph.edges[dependency, locator]["inputs"] = tuple(sorted(labels))
+ else:
+ graph.add_edge(dependency, locator, inputs=(input_name,))
+ visit(dependency)
+
+ for ref in sorted(set(outputs.values()), key=_ref_sort_key):
+ visit(ref)
+
+ output_names: dict[ArtifactRef, list[str]] = defaultdict(list)
+ for name, ref in outputs.items():
+ output_names[ref].append(name)
+ for locator in graph:
+ names = output_names[locator] if isinstance(locator, ArtifactRef) else []
+ graph.nodes[locator]["outputs"] = tuple(sorted(names))
+
+ if not nx.is_directed_acyclic_graph(graph):
+ raise ValueError("Artifact provenance contains a dependency cycle")
+ return nx.freeze(graph)
+
+
+def _status_label(status: ArtifactStatus | None) -> str:
+ if status is None:
+ return "unresolved external"
+ if not status.exists:
+ return "missing"
+ if not status.complete:
+ return "incomplete"
+ return "complete"
+
+
+def _located_ref(locator: _LineageLocator) -> ArtifactRef:
+ if isinstance(locator, ExternalArtifactRef | _ExternalLineageRef):
+ return locator.ref
+ return locator
+
+
+def _display_scope(locator: _LineageLocator) -> str:
+ ref = _located_ref(locator)
+ scope = ref.assay if ref.assay is not None else "datastore"
+ if isinstance(locator, ExternalArtifactRef | _ExternalLineageRef):
+ return f"external {locator.dataset_fingerprint[:12]} / {scope}"
+ return ref.assay if ref.assay is not None else "datastore"
+
+
+def _mermaid_text(value: str) -> str:
+ return " ".join(value.replace('"', "'").replace("\\", "/").split())
+
+
+def _detail_value(value: Any) -> str:
+ text = json.dumps(
+ value,
+ ensure_ascii=False,
+ sort_keys=True,
+ separators=(",", ":"),
+ default=str,
+ )
+ text = " ".join(text.replace("`", "'").split())
+ if len(text) > _DETAIL_VALUE_LIMIT:
+ return f"{text[: _DETAIL_VALUE_LIMIT - 3]}..."
+ return text
+
+
+def _detail_mapping(values: Mapping[str, Any]) -> str:
+ items = sorted(values.items())
+ shown = [
+ f"{key}={_detail_value(value)}" for key, value in items[:_DETAIL_ITEM_LIMIT]
+ ]
+ hidden = len(items) - len(shown)
+ if hidden:
+ shown.append(f"... {hidden} more")
+ return "; ".join(shown)
+
+
+class ArtifactLineage:
+ __slots__ = ("_graph", "_outputs")
+
+ def __init__(
+ self,
+ graph: nx.DiGraph,
+ outputs: Mapping[str, ArtifactRef],
+ ) -> None:
+ self._graph = graph
+ self._outputs = MappingProxyType(dict(outputs))
+
+ @classmethod
+ def from_store(
+ cls,
+ root: zarr.Group,
+ target: LineageTarget,
+ *,
+ external_roots: Mapping[str, zarr.Group] | None = None,
+ ) -> "ArtifactLineage":
+ outputs = _normalize_outputs(target)
+ roots = _normalize_external_roots(external_roots)
+ return cls(_build_graph(root, outputs, roots), outputs)
+
+ @property
+ def graph(self) -> nx.DiGraph:
+ return self._graph
+
+ @property
+ def outputs(self) -> Mapping[str, ArtifactRef]:
+ return self._outputs
+
+ def _ordered_refs(self) -> list[_LineageLocator]:
+ return list(
+ nx.lexicographical_topological_sort(
+ self._graph,
+ key=_locator_sort_key,
+ )
+ )
+
+ def to_mermaid(self) -> str:
+ locators = self._ordered_refs()
+ node_ids = {
+ locator: f"artifact{index}" for index, locator in enumerate(locators)
+ }
+ lines = ["flowchart LR"]
+ for locator in locators:
+ status = cast(
+ ArtifactStatus | None,
+ self._graph.nodes[locator]["status"],
+ )
+ ref = _located_ref(locator)
+ label_parts = [
+ f"{_display_scope(locator)} / {ref.kind}",
+ (
+ status.operation or "operation unavailable"
+ if status is not None
+ else "external source unresolved"
+ ),
+ ref.artifact_id[:12],
+ ]
+ output_names = cast(
+ tuple[str, ...],
+ self._graph.nodes[locator]["outputs"],
+ )
+ if output_names:
+ label_parts.append(f"outputs: {', '.join(output_names)}")
+ status_name = _status_label(status)
+ if status_name != "complete":
+ label_parts.append(f"status: {status_name}")
+ label = _mermaid_text(" | ".join(label_parts))
+ lines.append(f' {node_ids[locator]}["{label}"]')
+
+ edges = sorted(
+ self._graph.edges,
+ key=lambda edge: (node_ids[edge[1]], node_ids[edge[0]]),
+ )
+ for source, target in edges:
+ input_names = cast(
+ tuple[str, ...],
+ self._graph.edges[source, target]["inputs"],
+ )
+ label = _mermaid_text(", ".join(input_names))
+ lines.append(f' {node_ids[source]} -->|"{label}"| {node_ids[target]}')
+ return "\n".join(lines)
+
+ def to_markdown(self) -> str:
+ lines = [
+ "```mermaid",
+ self.to_mermaid(),
+ "```",
+ "",
+ "### Artifact details",
+ ]
+ for locator in self._ordered_refs():
+ status = cast(
+ ArtifactStatus | None,
+ self._graph.nodes[locator]["status"],
+ )
+ ref = _located_ref(locator)
+ lines.extend(
+ [
+ "",
+ (
+ f"#### {_display_scope(locator)} / {ref.kind} / "
+ f"{ref.artifact_id[:12]}"
+ ),
+ f"- Status: `{_status_label(status)}`",
+ ]
+ )
+ if isinstance(locator, ExternalArtifactRef | _ExternalLineageRef):
+ lines.append(f"- Dataset fingerprint: `{locator.dataset_fingerprint}`")
+ if status is None:
+ lines.append("- Resolution: `No matching external root was supplied`")
+ continue
+ lines.append(f"- Path: `{status.path}`")
+ if status.operation is not None:
+ lines.append(f"- Operation: `{status.operation}`")
+ output_names = cast(
+ tuple[str, ...],
+ self._graph.nodes[locator]["outputs"],
+ )
+ if output_names:
+ lines.append(
+ "- Outputs: "
+ + ", ".join(
+ f"`{name.replace('`', chr(39))}`" for name in output_names
+ )
+ )
+ parameters = status.parameters or {}
+ if parameters:
+ lines.append(f"- Parameters: `{_detail_mapping(parameters)}`")
+ execution_options = status.execution_options or {}
+ if execution_options:
+ lines.append(
+ f"- Execution options: `{_detail_mapping(execution_options)}`"
+ )
+ external_inputs = _external_inputs(status.inputs)
+ if external_inputs:
+ lines.append(f"- Other inputs: `{_detail_mapping(external_inputs)}`")
+ return "\n".join(lines)
+
+ def _repr_markdown_(self) -> str:
+ return self.to_markdown()
+
+ def __repr__(self) -> str:
+ return (
+ f"ArtifactLineage(outputs={len(self._outputs)}, "
+ f"artifacts={self._graph.number_of_nodes()}, "
+ f"dependencies={self._graph.number_of_edges()})"
+ )
diff --git a/scarf/mapping/__init__.py b/scarf/mapping/__init__.py
new file mode 100644
index 00000000..08f84da4
--- /dev/null
+++ b/scarf/mapping/__init__.py
@@ -0,0 +1,48 @@
+from .confidence import (
+ conformal_prediction_sets,
+ distance_weights,
+ mapping_score_weights,
+)
+from .hashing import array_hash, array_store_hash
+from .models import (
+ MappingResult,
+ QueryCorrection,
+ ScaledPCAProjectionModel,
+ SymphonyCorrectionModel,
+)
+from .reference import MappingReference
+from .symphony import (
+ SYMPHONY_ALGORITHM,
+ accumulate_sufficient_statistics,
+ apply_query_correction,
+ initialize_sufficient_statistics,
+ project_pca,
+ scaled_dispersion_sum,
+ soft_cluster_assignments,
+ solve_query_correction,
+ weighted_centroids,
+ zero_norm_rows,
+)
+
+__all__ = [
+ "MappingReference",
+ "MappingResult",
+ "QueryCorrection",
+ "ScaledPCAProjectionModel",
+ "SYMPHONY_ALGORITHM",
+ "SymphonyCorrectionModel",
+ "accumulate_sufficient_statistics",
+ "apply_query_correction",
+ "array_hash",
+ "array_store_hash",
+ "conformal_prediction_sets",
+ "distance_weights",
+ "initialize_sufficient_statistics",
+ "mapping_score_weights",
+ "project_pca",
+ "scaled_dispersion_sum",
+ "soft_cluster_assignments",
+ "solve_query_correction",
+ "weighted_centroids",
+ "zero_norm_rows",
+]
diff --git a/scarf/mapping/artifact.py b/scarf/mapping/artifact.py
new file mode 100644
index 00000000..5b386b17
--- /dev/null
+++ b/scarf/mapping/artifact.py
@@ -0,0 +1,539 @@
+"""Persistence and validation for mapping reference artifacts."""
+
+from collections.abc import Mapping
+from typing import Any
+
+import numpy as np
+import zarr
+
+from ..assay import RNAassay
+from ..graph.distances import validate_distance_provenance
+from ..graph.state import validate_normalized_artifact_selection
+from ..storage.arrays import create_zarr_dataset, create_zarr_obj_array
+from ..storage.artifacts import (
+ ArtifactRef,
+ ArtifactStatus,
+ artifact_group,
+ artifact_path,
+ inspect_artifact,
+)
+from ..storage.types import as_zarr_array
+from .models import ScaledPCAProjectionModel, SymphonyCorrectionModel
+from .reference import MappingReference
+
+MAPPING_REFERENCE_REBUILD_MESSAGE = (
+ "Rebuild it with build_mapping_reference(neighbors)."
+)
+_COMMON_ARRAYS = frozenset(
+ {
+ "feature_ids",
+ "feature_means",
+ "feature_scales",
+ "loadings",
+ "reference_distance_quantiles",
+ "reference_distance_values",
+ }
+)
+_SYMPHONY_ARRAYS = frozenset(
+ {
+ "centroids",
+ "raw_centroids",
+ "corrected_centroids",
+ "cluster_mass",
+ "sigma",
+ }
+)
+
+
+def write_artifact_mapping_reference(
+ group: zarr.Group,
+ model: ScaledPCAProjectionModel,
+ symphony_state: SymphonyCorrectionModel | None,
+ feature_ids: np.ndarray,
+ metadata: dict[str, Any],
+ reference_distance_quantiles: np.ndarray,
+ reference_distance_values: np.ndarray,
+) -> None:
+ """Write the conditional payload of one planned mapping reference."""
+ create_zarr_obj_array(group, "feature_ids", np.asarray(feature_ids))
+ _write_array(group, "feature_means", model.feature_means)
+ _write_array(group, "feature_scales", model.feature_scales)
+ _write_array(group, "loadings", model.loadings)
+ _write_array(
+ group,
+ "reference_distance_quantiles",
+ reference_distance_quantiles,
+ )
+ _write_array(
+ group,
+ "reference_distance_values",
+ reference_distance_values,
+ )
+ if symphony_state is not None:
+ _write_array(group, "centroids", symphony_state.centroids)
+ _write_array(group, "raw_centroids", symphony_state.raw_centroids)
+ _write_array(
+ group,
+ "corrected_centroids",
+ symphony_state.corrected_centroids,
+ )
+ _write_array(group, "cluster_mass", symphony_state.cluster_mass)
+ _write_array(group, "sigma", symphony_state.sigma)
+ group.attrs["reference_metadata"] = metadata
+
+
+def load_artifact_mapping_reference(
+ datastore: Any,
+ ref: ArtifactRef,
+) -> MappingReference:
+ """Load one mapping reference after validating its immutable graph chain."""
+ if (
+ not isinstance(ref, ArtifactRef)
+ or ref.scope != "assay"
+ or ref.assay is None
+ or ref.kind != "mapping_reference"
+ ):
+ raise _contract_error("Expected an assay-scoped mapping reference artifact")
+ status = inspect_artifact(datastore.zw, ref)
+ if not status.exists or not status.complete:
+ raise _contract_error("Mapping reference artifact is missing or incomplete")
+ if status.operation != "build_mapping_reference":
+ raise _contract_error("Mapping reference artifact has an old operation")
+
+ parameters = status.parameters or {}
+ method = parameters.get("method")
+ if method not in {"pca", "symphony"}:
+ raise _contract_error("Mapping reference method is missing or unsupported")
+
+ inputs = status.inputs or {}
+ expected_inputs = {
+ "reduction",
+ "ann_index",
+ "neighbors",
+ "cell_selection",
+ "feature_selection",
+ }
+ if method == "symphony":
+ expected_inputs.add("batch_correction")
+ if set(inputs) != expected_inputs:
+ raise _contract_error(
+ "Mapping reference artifact inputs do not match the current contract"
+ )
+ reduction = _input_ref(
+ datastore.zw,
+ inputs,
+ "reduction",
+ kind="reduction",
+ scope="assay",
+ assay=ref.assay,
+ )
+ ann_index = _input_ref(
+ datastore.zw,
+ inputs,
+ "ann_index",
+ kind="ann_index",
+ scope="assay",
+ assay=ref.assay,
+ )
+ neighbors = _input_ref(
+ datastore.zw,
+ inputs,
+ "neighbors",
+ kind="neighbors",
+ scope="assay",
+ assay=ref.assay,
+ )
+ cell_selection = _input_ref(
+ datastore.zw,
+ inputs,
+ "cell_selection",
+ kind="cell_selection",
+ scope="datastore",
+ assay=None,
+ )
+ feature_selection = _input_ref(
+ datastore.zw,
+ inputs,
+ "feature_selection",
+ kind="feature_selection",
+ scope="assay",
+ assay=ref.assay,
+ )
+ batch_correction = (
+ _input_ref(
+ datastore.zw,
+ inputs,
+ "batch_correction",
+ kind="batch_correction",
+ scope="assay",
+ assay=ref.assay,
+ )
+ if method == "symphony"
+ else None
+ )
+ if method == "pca" and "batch_correction" in inputs:
+ raise _contract_error("Plain PCA reference includes batch correction")
+
+ reduction_status = inspect_artifact(datastore.zw, reduction)
+ ann_status = inspect_artifact(datastore.zw, ann_index)
+ neighbors_status = inspect_artifact(datastore.zw, neighbors)
+ if reduction_status.operation != "run_pca":
+ raise _contract_error("Mapping reference reduction is not PCA")
+ if (reduction_status.parameters or {}).get("feat_scaling") is not True:
+ raise _contract_error("Mapping reference PCA did not enable feature scaling")
+ if ann_status.operation != "build_ann_index":
+ raise _contract_error("Mapping reference ANN input has an old operation")
+ if neighbors_status.operation != "query_neighbors":
+ raise _contract_error("Mapping reference neighbors input has an old operation")
+
+ coordinates = batch_correction or reduction
+ if _ref_from_input(ann_status, "coordinates") != coordinates:
+ raise _contract_error("ANN index uses different coordinates")
+ if (
+ _ref_from_input(neighbors_status, "ann_index") != ann_index
+ or _ref_from_input(neighbors_status, "coordinates") != coordinates
+ ):
+ raise _contract_error("Neighbors use a different ANN coordinate chain")
+ if batch_correction is not None:
+ correction_status = inspect_artifact(datastore.zw, batch_correction)
+ if correction_status.operation != "run_harmony":
+ raise _contract_error("Symphony reference correction is not Harmony")
+ if _ref_from_input(correction_status, "reduction") != reduction:
+ raise _contract_error("Batch correction uses a different PCA reduction")
+
+ normalized = _ref_from_input(reduction_status, "normalized")
+ feature_scaling = _ref_from_input(reduction_status, "feature_scaling")
+ _require_ref(
+ normalized,
+ kind="normalized",
+ scope="assay",
+ assay=ref.assay,
+ label="normalized",
+ )
+ _require_ref(
+ feature_scaling,
+ kind="feature_scaling",
+ scope="assay",
+ assay=ref.assay,
+ label="feature_scaling",
+ )
+ normalized_status = _complete_status(datastore.zw, normalized, "normalized")
+ scaling_status = _complete_status(
+ datastore.zw,
+ feature_scaling,
+ "feature_scaling",
+ )
+ if normalized_status.operation != "run_normalization":
+ raise _contract_error("Normalized input has an old operation")
+ if (
+ scaling_status.operation != "calculate_feature_scaling"
+ or (scaling_status.parameters or {}).get("enabled") is not True
+ or _ref_from_input(scaling_status, "normalized") != normalized
+ ):
+ raise _contract_error("Feature scaling does not match the PCA input")
+ if (
+ _ref_from_input(normalized_status, "cell_selection") != cell_selection
+ or _ref_from_input(normalized_status, "feature_selection") != feature_selection
+ ):
+ raise _contract_error("Stored selections do not match normalized data")
+
+ group = artifact_group(datastore.zw, ref)
+ _validate_payload_names(group, method)
+ raw_metadata = group.attrs.get("reference_metadata")
+ if not isinstance(raw_metadata, Mapping):
+ raise _contract_error("Mapping reference metadata is missing")
+ metadata = dict(raw_metadata)
+ if any(
+ name in metadata
+ for name in ("schemaVersion", "schema_version", "modelVersion", "model_version")
+ ):
+ raise _contract_error("Mapping reference metadata uses a versioned contract")
+ assay_name = _metadata_string(metadata, "assay")
+ cell_key = _metadata_string(metadata, "cell_key")
+ feature_key = _metadata_string(metadata, "feature_key")
+ if assay_name != ref.assay or metadata.get("method") != method:
+ raise _contract_error("Mapping reference metadata does not match its artifact")
+
+ normalization_parameters = metadata.get("normalization_parameters")
+ if not isinstance(normalization_parameters, Mapping):
+ raise _contract_error("Mapping reference normalization parameters are missing")
+ normalization_parameters = dict(normalization_parameters)
+ if normalization_parameters != (normalized_status.parameters or {}):
+ raise _contract_error("Mapping reference normalization parameters changed")
+ size_factor = normalization_parameters.get("size_factor")
+ if (
+ isinstance(size_factor, bool)
+ or not isinstance(size_factor, int | float)
+ or not np.isfinite(size_factor)
+ or float(size_factor) <= 0
+ ):
+ raise _contract_error(
+ "Mapping reference normalization size_factor must be finite and positive"
+ )
+
+ ann_metric = (ann_status.parameters or {}).get("ann_metric")
+ if ann_metric not in {"l2", "cosine"}:
+ raise _contract_error("Mapping reference ANN metric is unsupported")
+ if (
+ metadata.get("ann_metric") != ann_metric
+ or (neighbors_status.parameters or {}).get("distance_metric") != ann_metric
+ ):
+ raise _contract_error("Mapping reference distance metrics do not agree")
+ dataset_fingerprint = metadata.get("dataset_fingerprint")
+ if not isinstance(dataset_fingerprint, str) or not dataset_fingerprint:
+ raise _contract_error("Mapping reference dataset fingerprint is missing")
+ assay = datastore._get_assay(assay_name)
+ if not isinstance(assay, RNAassay):
+ raise _contract_error("Mapping references currently support RNA assays only")
+ if assay.attrs.get("dataset_fingerprint") != dataset_fingerprint:
+ raise _contract_error(
+ "Live assay dataset fingerprint does not match the mapping reference"
+ )
+
+ selected_cell_count = metadata.get("selected_cell_count")
+ if (
+ isinstance(selected_cell_count, bool)
+ or not isinstance(selected_cell_count, int)
+ or selected_cell_count < 1
+ ):
+ raise _contract_error("Selected reference cell count is missing")
+ selected_count = int(
+ np.count_nonzero(
+ np.asarray(
+ as_zarr_array(
+ artifact_group(datastore.zw, cell_selection)["values"],
+ name="values",
+ )[:],
+ dtype=bool,
+ )
+ )
+ )
+ if selected_count != selected_cell_count:
+ raise _contract_error("Selected reference cell count does not match its input")
+ try:
+ validate_normalized_artifact_selection(
+ datastore.zw,
+ normalized,
+ cell_key,
+ feature_key,
+ )
+ except (KeyError, TypeError, ValueError) as exc:
+ raise _contract_error(
+ "Mapping reference cell or feature selection no longer matches"
+ ) from exc
+
+ feature_ids = _values(group, "feature_ids")
+ try:
+ model = ScaledPCAProjectionModel(
+ feature_means=_values(group, "feature_means"),
+ feature_scales=_values(group, "feature_scales"),
+ loadings=_values(group, "loadings"),
+ )
+ except ValueError as exc:
+ raise _contract_error("Mapping reference PCA model is invalid") from exc
+ if feature_ids.ndim != 1 or len(feature_ids) != model.n_features:
+ raise _contract_error("Mapping reference feature IDs do not match PCA loadings")
+ if np.asarray(feature_ids).dtype.kind not in {"O", "S", "U"}:
+ raise _contract_error("Mapping reference feature IDs must contain strings")
+ feature_column = feature_key if feature_key == "I" else f"{cell_key}__{feature_key}"
+ if not np.array_equal(
+ feature_ids.astype(str),
+ np.asarray(assay.feats.fetch("ids", key=feature_column)).astype(str),
+ ):
+ raise _contract_error(
+ "Mapping reference feature IDs do not match the selected features"
+ )
+
+ symphony_state = None
+ if method == "symphony":
+ try:
+ symphony_state = SymphonyCorrectionModel(
+ centroids=_values(group, "centroids"),
+ raw_centroids=_values(group, "raw_centroids"),
+ corrected_centroids=_values(group, "corrected_centroids"),
+ cluster_mass=_values(group, "cluster_mass"),
+ sigma=_values(group, "sigma"),
+ )
+ except ValueError as exc:
+ raise _contract_error(
+ "Mapping reference Symphony correction model is invalid"
+ ) from exc
+ if symphony_state.n_dims != model.n_dims:
+ raise _contract_error(
+ "Symphony correction dimensions do not match PCA loadings"
+ )
+
+ distance_quantiles = _values(group, "reference_distance_quantiles")
+ distance_values = _values(group, "reference_distance_values")
+ if (
+ distance_quantiles.ndim != 1
+ or distance_values.shape != distance_quantiles.shape
+ or distance_quantiles.size < 1
+ or not np.all(np.isfinite(distance_quantiles))
+ or not np.all(np.isfinite(distance_values))
+ or np.any(distance_quantiles < 0)
+ or np.any(distance_quantiles > 1)
+ or np.any(distance_values < 0)
+ or np.any(np.diff(distance_quantiles) < 0)
+ or np.any(np.diff(distance_values) < 0)
+ ):
+ raise _contract_error("Mapping reference distance summary is invalid")
+ try:
+ validate_distance_provenance(datastore.zw, artifact_path(neighbors))
+ except (KeyError, TypeError, ValueError) as exc:
+ raise _contract_error(
+ "Mapping reference neighbor distances are invalid"
+ ) from exc
+
+ return MappingReference(
+ datastore=datastore,
+ ref=ref,
+ assay_name=assay_name,
+ cell_key=cell_key,
+ feature_key=feature_key,
+ reduction=reduction,
+ ann_index=ann_index,
+ neighbors=neighbors,
+ cell_selection=cell_selection,
+ feature_selection=feature_selection,
+ batch_correction=batch_correction,
+ dataset_fingerprint=dataset_fingerprint,
+ selected_cell_count=selected_cell_count,
+ model=model,
+ symphony_state=symphony_state,
+ feature_ids=feature_ids,
+ metadata=metadata,
+ reference_distance_quantiles=distance_quantiles,
+ reference_distance_values=distance_values,
+ )
+
+
+def _input_ref(
+ root: zarr.Group,
+ inputs: Mapping[str, Any],
+ name: str,
+ *,
+ kind: str,
+ scope: str,
+ assay: str | None,
+) -> ArtifactRef:
+ raw_ref = inputs.get(name)
+ if not isinstance(raw_ref, Mapping):
+ raise _contract_error(f"Mapping reference input {name!r} is missing")
+ try:
+ ref = ArtifactRef.from_dict(raw_ref)
+ except (TypeError, ValueError) as exc:
+ raise _contract_error(f"Mapping reference input {name!r} is malformed") from exc
+ _require_ref(
+ ref,
+ kind=kind,
+ scope=scope,
+ assay=assay,
+ label=name,
+ )
+ _complete_status(root, ref, name)
+ return ref
+
+
+def _require_ref(
+ ref: ArtifactRef,
+ *,
+ kind: str,
+ scope: str,
+ assay: str | None,
+ label: str,
+) -> None:
+ if ref.kind != kind or ref.scope != scope or ref.assay != assay:
+ raise _contract_error(
+ f"Mapping reference input {label!r} has the wrong artifact kind or scope"
+ )
+
+
+def _complete_status(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ label: str,
+) -> ArtifactStatus:
+ status = inspect_artifact(root, ref)
+ if not status.exists or not status.complete:
+ raise _contract_error(
+ f"Mapping reference input {label!r} is missing or incomplete"
+ )
+ return status
+
+
+def _ref_from_input(status: ArtifactStatus, name: str) -> ArtifactRef:
+ raw_ref = (status.inputs or {}).get(name)
+ if not isinstance(raw_ref, Mapping):
+ raise _contract_error(
+ f"{status.ref.kind} input {name!r} is missing from the graph chain"
+ )
+ try:
+ return ArtifactRef.from_dict(raw_ref)
+ except (TypeError, ValueError) as exc:
+ raise _contract_error(f"{status.ref.kind} input {name!r} is malformed") from exc
+
+
+def _validate_payload_names(group: zarr.Group, method: str) -> None:
+ if set(group.group_keys()):
+ raise _contract_error(
+ "Mapping reference contains groups outside the current contract"
+ )
+ arrays = set(group.array_keys())
+ expected = set(_COMMON_ARRAYS)
+ if method == "symphony":
+ expected.update(_SYMPHONY_ARRAYS)
+ missing = expected - arrays
+ unexpected = arrays - expected
+ if missing:
+ raise _contract_error(
+ "Mapping reference is missing required arrays: "
+ + ", ".join(sorted(missing))
+ )
+ if unexpected:
+ raise _contract_error(
+ "Mapping reference contains arrays outside the current contract: "
+ + ", ".join(sorted(unexpected))
+ )
+ if any(
+ name in group.attrs
+ for name in ("schemaVersion", "schema_version", "modelVersion", "model_version")
+ ):
+ raise _contract_error("Mapping reference uses an old versioned contract")
+
+
+def _metadata_string(metadata: Mapping[str, Any], name: str) -> str:
+ value = metadata.get(name)
+ if not isinstance(value, str) or not value:
+ raise _contract_error(f"Mapping reference metadata {name!r} is missing")
+ return value
+
+
+def _values(group: zarr.Group, name: str) -> np.ndarray:
+ return np.asarray(as_zarr_array(group[name], name=name)[:])
+
+
+def _write_array(group: zarr.Group, name: str, values: np.ndarray) -> None:
+ array_values = np.asarray(values, dtype=np.float64)
+ if array_values.ndim not in {1, 2}:
+ raise ValueError(f"Mapping reference array {name!r} must have one or two axes")
+ chunks = (
+ (min(max(array_values.shape[0], 1), 10_000),)
+ if array_values.ndim == 1
+ else (
+ min(max(array_values.shape[0], 1), 1_000),
+ max(array_values.shape[1], 1),
+ )
+ )
+ array = create_zarr_dataset(
+ group,
+ name,
+ chunks,
+ "f8",
+ array_values.shape,
+ )
+ array[...] = array_values
+
+
+def _contract_error(detail: str) -> ValueError:
+ return ValueError(f"{detail}. {MAPPING_REFERENCE_REBUILD_MESSAGE}")
diff --git a/scarf/mapping/confidence.py b/scarf/mapping/confidence.py
new file mode 100644
index 00000000..0c596b4c
--- /dev/null
+++ b/scarf/mapping/confidence.py
@@ -0,0 +1,116 @@
+from typing import Any, cast
+
+import numpy as np
+
+from ..storage.geometry import array_geometry
+from ..storage.partition import row_band
+
+
+def _distance_quantile_summary(
+ distances: Any,
+ max_samples: int = 100_000,
+ n_quantiles: int = 1_001,
+) -> tuple[np.ndarray, np.ndarray]:
+ """Summarize first-neighbor distances with deterministic row sampling."""
+ shape = tuple(int(value) for value in distances.shape)
+ if len(shape) not in {1, 2}:
+ raise ValueError("Neighbor distances must be one- or two-dimensional")
+ n_rows = shape[0]
+ if n_rows < 1:
+ raise ValueError("Neighbor distances are empty")
+ if len(shape) == 2 and shape[1] < 1:
+ raise ValueError("Neighbor distances do not contain any neighbors")
+ if max_samples < 1 or n_quantiles < 1:
+ raise ValueError("Sampling and quantile counts must be positive")
+
+ stride = max(int(np.ceil(n_rows / max_samples)), 1)
+ block_size = row_band(
+ array_geometry(distances),
+ unit="chunk",
+ fallback=min(n_rows, 10_000),
+ )
+ sampled: list[np.ndarray] = []
+ for start in range(0, n_rows, block_size):
+ stop = min(start + block_size, n_rows)
+ block = np.asarray(distances[start:stop])
+ if block.ndim == 2:
+ block = block[:, 0]
+ mask = np.arange(start, stop, dtype=np.int64) % stride == 0
+ sampled.append(np.asarray(block[mask], dtype=np.float64))
+ values = np.concatenate(sampled)
+ quantiles = np.linspace(0.0, 1.0, min(n_quantiles, len(values)))
+ return quantiles, np.quantile(values, quantiles)
+
+
+def _validated_distances(distances: np.ndarray) -> np.ndarray:
+ values = np.asarray(distances, dtype=np.float64)
+ if values.ndim != 2:
+ raise ValueError("Expected a two-dimensional distance array")
+ if not np.all(np.isfinite(values)):
+ raise ValueError("Neighbor distances must be finite")
+ if np.any(values < 0):
+ raise ValueError("Neighbor distances must be non-negative")
+ return values
+
+
+def mapping_score_weights(distances: np.ndarray) -> np.ndarray:
+ """Return absolute neighbor weights for reference-side mapping scores.
+
+ The weight of one reference neighbor is ``1 / (log(distance + 1) + 1)``.
+ Weights are not normalized per query cell, so a query cell that sits far
+ from the reference contributes less total weight than one that lands on the
+ reference manifold. Normalizing per row would erase that contrast and make
+ the score a plain neighbor count.
+ """
+ weights: np.ndarray = 1.0 / (np.log1p(_validated_distances(distances)) + 1.0)
+ return weights
+
+
+def distance_weights(distances: np.ndarray) -> np.ndarray:
+ """Convert metric distances into normalized inverse-distance weights."""
+ values = _validated_distances(distances)
+
+ weights = np.zeros_like(values)
+ zero_mask = values == 0
+ zero_count = zero_mask.sum(axis=1)
+ rows_with_zero = zero_count > 0
+ if rows_with_zero.any():
+ weights[rows_with_zero] = (
+ zero_mask[rows_with_zero] / zero_count[rows_with_zero, np.newaxis]
+ )
+ rows_without_zero = ~rows_with_zero
+ if rows_without_zero.any():
+ positive = values[rows_without_zero]
+ minimum = positive.min(axis=1, keepdims=True)
+ inverse_ratios = minimum / positive
+ weights[rows_without_zero] = inverse_ratios / inverse_ratios.sum(
+ axis=1,
+ keepdims=True,
+ )
+ return weights
+
+
+def conformal_prediction_sets(
+ label_scores: np.ndarray,
+ calibration_nonconformity: np.ndarray,
+ alpha: float = 0.1,
+) -> np.ndarray:
+ """Return class-membership masks from split-conformal p-values."""
+ scores = np.asarray(label_scores, dtype=np.float64)
+ calibration = np.asarray(calibration_nonconformity, dtype=np.float64)
+ if scores.ndim != 2:
+ raise ValueError("label_scores must be a two-dimensional array")
+ if calibration.ndim != 1 or calibration.size == 0:
+ raise ValueError("calibration_nonconformity must be a non-empty vector")
+ if not 0 < alpha < 1:
+ raise ValueError("alpha must be strictly between zero and one")
+ if not np.all(np.isfinite(scores)) or not np.all(np.isfinite(calibration)):
+ raise ValueError("Conformal inputs must be finite")
+ nonconformity = 1.0 - scores
+ p_values = (
+ (calibration[np.newaxis, np.newaxis, :] >= nonconformity[:, :, np.newaxis]).sum(
+ axis=2
+ )
+ + 1
+ ) / (len(calibration) + 1)
+ return cast(np.ndarray, p_values > alpha)
diff --git a/scarf/mapping/features.py b/scarf/mapping/features.py
new file mode 100644
index 00000000..ef047b6c
--- /dev/null
+++ b/scarf/mapping/features.py
@@ -0,0 +1,579 @@
+import copy
+from collections.abc import Iterator, Mapping
+from dataclasses import dataclass
+from typing import TYPE_CHECKING, Any
+
+import numpy as np
+from numpy.typing import DTypeLike
+
+from ..assay import RNAassay, _read_block, norm_lib_size
+from ..storage.artifacts import ValueFingerprintBuilder, callable_identity
+from ..storage.budget import ResourceBudget, admit_stream
+from ..storage.geometry import ArrayGeometry, array_geometry
+from ..storage.parallel import stream_shards
+from ..storage.partition import (
+ affordable_width,
+ checked_indices,
+ contiguous_ranges,
+ row_band,
+)
+
+if TYPE_CHECKING:
+ from ..assay import Assay
+
+
+@dataclass(frozen=True, slots=True)
+class AlignedFeatureBlock:
+ """One aligned row block and its offset in the selected query rows."""
+
+ row_offset: int
+ values: np.ndarray
+
+
+@dataclass(frozen=True, slots=True)
+class AlignedRowGeometry:
+ """Source geometry and deterministic selected-row block boundaries."""
+
+ source: ArrayGeometry | None
+ block_rows: int
+ boundaries: tuple[tuple[int, int], ...]
+
+
+def _read_only_array(
+ values: Any,
+ *,
+ dtype: DTypeLike | None = None,
+) -> np.ndarray:
+ array = np.array(values, dtype=dtype, copy=True)
+ array.setflags(write=False)
+ return array
+
+
+def _feature_ids(values: Any, *, name: str) -> np.ndarray:
+ raw = np.asarray(values)
+ if raw.ndim != 1:
+ raise ValueError(f"{name} must be one-dimensional")
+ if raw.size == 0:
+ raise ValueError(f"{name} cannot be empty")
+ if raw.dtype.kind not in {"O", "S", "U"}:
+ raise TypeError(f"{name} must contain strings")
+ if raw.dtype.kind == "O" and any(
+ not isinstance(value, str | bytes | np.str_ | np.bytes_) for value in raw
+ ):
+ raise TypeError(f"{name} must contain strings")
+ identifiers = np.asarray(raw).astype(str)
+ unique, counts = np.unique(identifiers, return_counts=True)
+ if np.any(counts > 1):
+ duplicates = unique[counts > 1][:5].tolist()
+ raise ValueError(f"{name} must be unique: {duplicates}")
+ return _read_only_array(identifiers)
+
+
+def _reference_means(values: Any, *, n_features: int) -> np.ndarray:
+ raw = np.asarray(values)
+ if raw.dtype.kind not in {"i", "u", "f"}:
+ raise ValueError("Reference normalized means must be real numeric values")
+ try:
+ means = np.asarray(raw, dtype=np.float64)
+ except (TypeError, ValueError) as exc:
+ raise ValueError("Reference normalized means must be numeric") from exc
+ if means.shape != (n_features,):
+ raise ValueError(
+ "Reference normalized means must have one value per reference feature"
+ )
+ if not np.all(np.isfinite(means)):
+ raise ValueError("Reference normalized means must be finite")
+ return _read_only_array(means, dtype=np.float64)
+
+
+def _normalization_parameters(parameters: Mapping[str, Any]) -> dict[str, Any]:
+ values = copy.deepcopy(dict(parameters))
+ required = {
+ "normalization_method",
+ "size_factor",
+ "log_transform",
+ "renormalize_subset",
+ }
+ missing = required - set(values)
+ if missing:
+ raise ValueError(
+ "Reference normalization parameters are missing: "
+ + ", ".join(sorted(missing))
+ )
+ unknown = set(values) - required
+ if unknown:
+ raise ValueError(
+ "Unsupported reference normalization parameters: "
+ + ", ".join(sorted(unknown))
+ )
+
+ method = values["normalization_method"]
+ supported_method = callable_identity(norm_lib_size)
+ if not isinstance(method, Mapping):
+ raise ValueError(
+ "Unsupported reference normalization method. "
+ f"Expected {supported_method!r}, received {method!r}"
+ )
+ method_identity = dict(method)
+ external_hook = method_identity.pop("external_hook", True)
+ if external_hook is not True or method_identity != supported_method:
+ raise ValueError(
+ "Unsupported reference normalization method. "
+ f"Expected {supported_method!r}, received {method!r}"
+ )
+ values["normalization_method"] = dict(method)
+
+ size_factor = values["size_factor"]
+ if (
+ isinstance(size_factor, bool | np.bool_)
+ or not isinstance(size_factor, int | float | np.integer | np.floating)
+ or not np.isfinite(size_factor)
+ or float(size_factor) <= 0
+ ):
+ raise ValueError(
+ "Reference normalization size_factor must be finite and positive"
+ )
+ values["size_factor"] = float(size_factor)
+
+ for name in ("log_transform", "renormalize_subset"):
+ value = values[name]
+ if not isinstance(value, bool | np.bool_):
+ raise TypeError(f"Reference normalization {name} must be a boolean")
+ values[name] = bool(value)
+ return values
+
+
+class AlignedFeatureStream:
+ """Replay normalized query rows in immutable reference feature order."""
+
+ _RAW_FINGERPRINT_NAME = "selected_raw_query_expression"
+
+ def __init__(
+ self,
+ query_assay: "Assay",
+ query_cell_indices: np.ndarray,
+ reference_feature_ids: np.ndarray,
+ reference_normalized_means: np.ndarray,
+ reference_normalization_parameters: Mapping[str, Any],
+ missing_feature_policy: str,
+ resources: ResourceBudget,
+ *,
+ reserved_resident_bytes: int = 0,
+ reserved_per_row_bytes: int = 0,
+ ) -> None:
+ if not isinstance(query_assay, RNAassay):
+ raise TypeError("AlignedFeatureStream supports RNA query assays only")
+ if not isinstance(resources, ResourceBudget):
+ raise TypeError("resources must be a ResourceBudget")
+ if resources.memoryBytes < 1 or resources.workers < 1:
+ raise ValueError("ResourceBudget fields must be positive")
+ for name, value in (
+ ("reserved_resident_bytes", reserved_resident_bytes),
+ ("reserved_per_row_bytes", reserved_per_row_bytes),
+ ):
+ if isinstance(value, bool) or not isinstance(value, int | np.integer):
+ raise TypeError(f"{name} must be an integer")
+ if int(value) < 0:
+ raise ValueError(f"{name} must be non-negative")
+ if missing_feature_policy not in {"reference_mean", "zero", "error"}:
+ raise ValueError(
+ "missing_feature_policy must be one of "
+ "'reference_mean', 'zero', or 'error'"
+ )
+
+ raw_data = query_assay.rawData
+ if len(raw_data.shape) != 2:
+ raise ValueError("Query raw counts must be two-dimensional")
+ raw_dtype = np.dtype(raw_data.dtype)
+ if raw_dtype.hasobject or not np.issubdtype(raw_dtype, np.number):
+ raise TypeError("Query raw counts must have a numeric dtype")
+
+ cell_indices = checked_indices(
+ query_cell_indices,
+ limit=int(raw_data.shape[0]),
+ name="query_cell_indices",
+ )
+ if cell_indices.size == 0:
+ raise ValueError("query_cell_indices cannot be empty")
+ self._query_cell_indices = _read_only_array(cell_indices, dtype=np.int64)
+
+ self._reference_feature_ids = _feature_ids(
+ reference_feature_ids,
+ name="Reference feature identifiers",
+ )
+ self._reference_normalized_means = _reference_means(
+ reference_normalized_means,
+ n_features=len(self._reference_feature_ids),
+ )
+ self._normalization_parameters = _normalization_parameters(
+ reference_normalization_parameters
+ )
+ self._missing_feature_policy = missing_feature_policy
+ self._resources = resources
+ self._query_assay = query_assay
+ self._raw_backing = raw_data._backing
+ self._raw_dtype = raw_dtype
+ self._source_geometry = array_geometry(self._raw_backing)
+ if self._source_geometry is not None and len(self._source_geometry.shape) != 2:
+ raise ValueError("Query raw count geometry must be two-dimensional")
+
+ query_feature_ids = _feature_ids(
+ query_assay.feats.fetch_all("ids"),
+ name="Query feature identifiers",
+ )
+ if len(query_feature_ids) != raw_data.shape[1]:
+ raise ValueError(
+ "Query feature identifiers do not match the raw count columns"
+ )
+ query_lookup = {
+ identifier: index for index, identifier in enumerate(query_feature_ids)
+ }
+ reference_to_query = np.fromiter(
+ (
+ query_lookup.get(identifier, -1)
+ for identifier in self._reference_feature_ids
+ ),
+ dtype=np.int64,
+ count=len(self._reference_feature_ids),
+ )
+ reference_indices = np.flatnonzero(reference_to_query >= 0).astype(
+ np.int64,
+ copy=False,
+ )
+ if reference_indices.size == 0:
+ raise ValueError("No reference features overlap the query feature IDs")
+ if missing_feature_policy == "error" and reference_indices.size != len(
+ self._reference_feature_ids
+ ):
+ missing = len(self._reference_feature_ids) - reference_indices.size
+ raise ValueError(
+ f"Query data is missing {missing} required reference features"
+ )
+ query_indices = reference_to_query[reference_indices]
+ self._reference_to_query_index_map = _read_only_array(
+ reference_to_query,
+ dtype=np.int64,
+ )
+ self._reference_index_map = _read_only_array(
+ reference_indices,
+ dtype=np.int64,
+ )
+ self._query_index_map = _read_only_array(query_indices, dtype=np.int64)
+ self._feature_coverage = float(
+ reference_indices.size / len(self._reference_feature_ids)
+ )
+ self._alignment_map_fingerprint = self._fingerprint_alignment_map(
+ query_feature_ids
+ )
+ self._n_query_features = int(raw_data.shape[1])
+
+ self._cell_scalars: np.ndarray | None = None
+ if not self.renormalize_subset:
+ scalar_name = f"{query_assay.name}_nCounts"
+ try:
+ all_scalars = np.asarray(
+ query_assay.cells.fetch_all(scalar_name),
+ dtype=np.float64,
+ )
+ except (KeyError, TypeError, ValueError) as exc:
+ raise ValueError(
+ f"Query assay requires numeric {scalar_name!r} metadata"
+ ) from exc
+ if all_scalars.shape != (raw_data.shape[0],):
+ raise ValueError(
+ f"Query assay {scalar_name!r} metadata has the wrong shape"
+ )
+ scalars = np.array(
+ all_scalars[self._query_cell_indices],
+ dtype=np.float64,
+ copy=True,
+ )
+ if not np.all(np.isfinite(scalars)) or np.any(scalars < 0):
+ raise ValueError(
+ f"Query assay {scalar_name!r} metadata must be finite "
+ "and non-negative"
+ )
+ scalars[scalars == 0] = 1
+ scalars.setflags(write=False)
+ self._cell_scalars = scalars
+
+ self._decode_bytes = (
+ 0
+ if self._source_geometry is None
+ else self._source_geometry.nominalChunkBytes()
+ )
+ self._resident_bytes = self._calculate_resident_bytes() + int(
+ reserved_resident_bytes
+ )
+ normalized_bytes = np.dtype(np.float64).itemsize
+ self._stream_row_bytes = (
+ len(self._query_index_map) * (raw_dtype.itemsize + normalized_bytes)
+ + len(self._reference_feature_ids) * normalized_bytes
+ + (normalized_bytes if self.renormalize_subset else 0)
+ + int(reserved_per_row_bytes)
+ )
+ block_rows, boundaries, io_concurrency = self._plan_rows(
+ bytes_per_row=self._stream_row_bytes,
+ )
+ self._row_geometry = AlignedRowGeometry(
+ source=self._source_geometry,
+ block_rows=block_rows,
+ boundaries=boundaries,
+ )
+ self._io_concurrency = io_concurrency
+ self._raw_expression_fingerprint: str | None = None
+
+ @property
+ def shape(self) -> tuple[int, int]:
+ return (
+ len(self._query_cell_indices),
+ len(self._reference_feature_ids),
+ )
+
+ @property
+ def dtype(self) -> np.dtype[Any]:
+ return np.dtype(np.float64)
+
+ @property
+ def row_geometry(self) -> AlignedRowGeometry:
+ return self._row_geometry
+
+ @property
+ def block_boundaries(self) -> tuple[tuple[int, int], ...]:
+ return self._row_geometry.boundaries
+
+ @property
+ def feature_coverage(self) -> float:
+ return self._feature_coverage
+
+ @property
+ def reference_feature_ids(self) -> np.ndarray:
+ return self._reference_feature_ids.view()
+
+ @property
+ def reference_normalized_means(self) -> np.ndarray:
+ return self._reference_normalized_means.view()
+
+ @property
+ def query_cell_indices(self) -> np.ndarray:
+ return self._query_cell_indices.view()
+
+ @property
+ def normalization_parameters(self) -> dict[str, Any]:
+ return copy.deepcopy(self._normalization_parameters)
+
+ @property
+ def missing_feature_policy(self) -> str:
+ return self._missing_feature_policy
+
+ @property
+ def size_factor(self) -> float:
+ return float(self._normalization_parameters["size_factor"])
+
+ @property
+ def log_transform(self) -> bool:
+ return bool(self._normalization_parameters["log_transform"])
+
+ @property
+ def renormalize_subset(self) -> bool:
+ return bool(self._normalization_parameters["renormalize_subset"])
+
+ @property
+ def query_index_map(self) -> np.ndarray:
+ return self._query_index_map.view()
+
+ @property
+ def reference_index_map(self) -> np.ndarray:
+ return self._reference_index_map.view()
+
+ @property
+ def reference_to_query_index_map(self) -> np.ndarray:
+ return self._reference_to_query_index_map.view()
+
+ @property
+ def query_feature_indices(self) -> np.ndarray:
+ return self._query_index_map.view()
+
+ @property
+ def reference_feature_indices(self) -> np.ndarray:
+ return self._reference_index_map.view()
+
+ @property
+ def alignment_map_fingerprint(self) -> str:
+ return self._alignment_map_fingerprint
+
+ @property
+ def alignment_map_hash(self) -> str:
+ return self._alignment_map_fingerprint
+
+ @property
+ def resident_bytes(self) -> int:
+ return self._resident_bytes
+
+ @property
+ def decoded_chunk_bytes(self) -> int:
+ return self._decode_bytes
+
+ @property
+ def stream_row_bytes(self) -> int:
+ return self._stream_row_bytes
+
+ @property
+ def raw_expression_fingerprint(self) -> str:
+ if self._raw_expression_fingerprint is None:
+ self._raw_expression_fingerprint = self._fingerprint_raw_expression()
+ return self._raw_expression_fingerprint
+
+ def _calculate_resident_bytes(self) -> int:
+ arrays = [
+ self._query_cell_indices,
+ self._reference_feature_ids,
+ self._reference_normalized_means,
+ self._reference_to_query_index_map,
+ self._reference_index_map,
+ self._query_index_map,
+ ]
+ if self._cell_scalars is not None:
+ arrays.append(self._cell_scalars)
+ return sum(array.nbytes for array in arrays) + np.dtype(np.float64).itemsize
+
+ def _fingerprint_alignment_map(self, query_feature_ids: np.ndarray) -> str:
+ builder = ValueFingerprintBuilder()
+ builder.update_array("reference_feature_ids", self._reference_feature_ids)
+ builder.update_array("query_feature_ids", query_feature_ids)
+ builder.update_array(
+ "reference_to_query_index",
+ self._reference_to_query_index_map,
+ )
+ return builder.hexdigest()
+
+ def _plan_rows(
+ self,
+ *,
+ bytes_per_row: int,
+ resident_extra: int = 0,
+ ) -> tuple[int, tuple[tuple[int, int], ...], int]:
+ n_rows = len(self._query_cell_indices)
+ resident = self._resident_bytes + max(0, int(resident_extra))
+ preferred = min(
+ n_rows,
+ row_band(
+ self._source_geometry,
+ unit="chunk",
+ fallback=n_rows,
+ ),
+ )
+
+ def fits(rows: int) -> bool:
+ return (
+ resident + rows * max(1, int(bytes_per_row)) + self._decode_bytes
+ <= self._resources.memoryBytes
+ )
+
+ block_rows = affordable_width(fits, preferred)
+ if block_rows < 1:
+ minimum = resident + max(1, int(bytes_per_row)) + self._decode_bytes
+ raise MemoryError(
+ f"One aligned row needs about {minimum} bytes, but the operation "
+ f"limit is {self._resources.memoryBytes} bytes"
+ )
+ boundaries = tuple(contiguous_ranges(n_rows, block_rows))
+ admission = admit_stream(
+ self._resources,
+ nBlocks=1,
+ blockBytes=max(1, block_rows * int(bytes_per_row)),
+ decodeBytes=self._decode_bytes,
+ residentBytes=resident,
+ requested=1,
+ )
+ return block_rows, boundaries, admission.ioConcurrency
+
+ def _read_raw(
+ self,
+ start: int,
+ end: int,
+ columns: np.ndarray,
+ ) -> np.ndarray:
+ rows = self._query_cell_indices[start:end]
+ if isinstance(self._raw_backing, np.ndarray):
+ return np.asarray(self._raw_backing[np.ix_(rows, columns)])
+ return _read_block(self._raw_backing, rows, columns)
+
+ def __iter__(self) -> Iterator[AlignedFeatureBlock]:
+ return self.iter_blocks()
+
+ def iter_blocks(self) -> Iterator[AlignedFeatureBlock]:
+ """Return a fresh iterator over normalized aligned row blocks."""
+
+ def read(boundary: tuple[int, int]) -> tuple[int, np.ndarray]:
+ start, end = boundary
+ return start, self._read_raw(
+ start,
+ end,
+ self._query_index_map,
+ )
+
+ raw_blocks = stream_shards(
+ self._row_geometry.boundaries,
+ read,
+ workers=1,
+ io_concurrency=self._io_concurrency,
+ total=len(self._row_geometry.boundaries),
+ )
+ for start, raw in raw_blocks:
+ normalized = raw.astype(np.float64, copy=True)
+ normalized *= self.size_factor
+ if self.renormalize_subset:
+ denominator = raw.sum(axis=1, dtype=np.float64)
+ denominator[denominator == 0] = 1
+ else:
+ assert self._cell_scalars is not None
+ denominator = self._cell_scalars[start : start + len(raw)]
+ normalized /= denominator[:, np.newaxis]
+ if self.log_transform:
+ np.log1p(normalized, out=normalized)
+
+ values = np.empty(
+ (len(raw), len(self._reference_feature_ids)),
+ dtype=self.dtype,
+ )
+ if self._missing_feature_policy == "reference_mean":
+ values[:] = self._reference_normalized_means
+ else:
+ values.fill(0)
+ values[:, self._reference_index_map] = normalized
+ yield AlignedFeatureBlock(row_offset=start, values=values)
+
+ def _fingerprint_raw_expression(self) -> str:
+ all_columns = np.arange(self._n_query_features, dtype=np.int64)
+ raw_row_bytes = self._n_query_features * self._raw_dtype.itemsize
+ _, boundaries, io_concurrency = self._plan_rows(
+ bytes_per_row=max(1, 2 * raw_row_bytes),
+ resident_extra=all_columns.nbytes,
+ )
+ builder = ValueFingerprintBuilder()
+ builder.begin_array(
+ self._RAW_FINGERPRINT_NAME,
+ self.shape[:1] + (self._n_query_features,),
+ self._raw_dtype,
+ )
+
+ def read(boundary: tuple[int, int]) -> tuple[int, np.ndarray]:
+ start, end = boundary
+ return start, self._read_raw(start, end, all_columns)
+
+ raw_blocks = stream_shards(
+ boundaries,
+ read,
+ workers=1,
+ io_concurrency=io_concurrency,
+ total=len(boundaries),
+ )
+ for start, raw in raw_blocks:
+ builder.update_array_block(
+ self._RAW_FINGERPRINT_NAME,
+ (start, 0),
+ raw,
+ )
+ builder.end_array(self._RAW_FINGERPRINT_NAME)
+ return builder.hexdigest()
diff --git a/scarf/mapping/hashing.py b/scarf/mapping/hashing.py
new file mode 100644
index 00000000..f18d023c
--- /dev/null
+++ b/scarf/mapping/hashing.py
@@ -0,0 +1,49 @@
+import hashlib
+from typing import Any
+
+import numpy as np
+
+from ..storage.geometry import array_geometry
+from ..storage.partition import row_band
+
+
+def array_hash(values: np.ndarray | list[Any]) -> str:
+ """Return a stable content hash for numeric or identifier arrays."""
+ arr = np.asarray(values)
+ digest = hashlib.sha256()
+ digest.update(str(arr.shape).encode())
+ digest.update(arr.dtype.str.encode())
+ if arr.dtype.kind in {"O", "S", "U"}:
+ digest.update(
+ "\x1f".join(str(value) for value in arr.reshape(-1)).encode("utf-8")
+ )
+ else:
+ digest.update(np.ascontiguousarray(arr).tobytes())
+ return digest.hexdigest()
+
+
+def array_store_hash(values: Any) -> str:
+ """Hash a row-addressable array without materializing it in memory."""
+ shape = tuple(int(value) for value in values.shape)
+ dtype = np.dtype(values.dtype)
+ digest = hashlib.sha256()
+ digest.update(str(shape).encode())
+ digest.update(dtype.str.encode())
+ if not shape:
+ digest.update(np.asarray(values[...]).tobytes())
+ return digest.hexdigest()
+ row_chunk = row_band(
+ array_geometry(values),
+ unit="chunk",
+ fallback=min(max(shape[0], 1), 10_000),
+ )
+ for start in range(0, shape[0], row_chunk):
+ stop = min(start + row_chunk, shape[0])
+ block = np.asarray(values[start:stop])
+ if dtype.kind in {"O", "S", "U"}:
+ digest.update(
+ "\x1f".join(str(value) for value in block.reshape(-1)).encode("utf-8")
+ )
+ else:
+ digest.update(np.ascontiguousarray(block).tobytes())
+ return digest.hexdigest()
diff --git a/scarf/mapping/models.py b/scarf/mapping/models.py
new file mode 100644
index 00000000..f151f0fe
--- /dev/null
+++ b/scarf/mapping/models.py
@@ -0,0 +1,134 @@
+from dataclasses import dataclass, field
+from typing import TYPE_CHECKING
+
+import numpy as np
+
+from ..storage.refs import ArtifactRef
+
+if TYPE_CHECKING:
+ from .reference import MappingReference
+
+
+@dataclass(frozen=True)
+class ScaledPCAProjectionModel:
+ feature_means: np.ndarray
+ feature_scales: np.ndarray
+ loadings: np.ndarray
+
+ def __post_init__(self) -> None:
+ if self.loadings.ndim != 2:
+ raise ValueError("Reference PCA loadings must be two-dimensional")
+ n_features = self.loadings.shape[0]
+ if self.feature_means.shape != (n_features,):
+ raise ValueError("Reference feature means have incompatible dimensions")
+ if self.feature_scales.shape != (n_features,):
+ raise ValueError("Reference feature scales have incompatible dimensions")
+ if np.any(self.feature_scales <= 0):
+ raise ValueError("Reference feature scales must be positive")
+ for values in (
+ self.feature_means,
+ self.feature_scales,
+ self.loadings,
+ ):
+ if not np.all(np.isfinite(values)):
+ raise ValueError(
+ "Reference projection model contains non-finite values"
+ )
+
+ @property
+ def n_features(self) -> int:
+ return int(self.loadings.shape[0])
+
+ @property
+ def n_dims(self) -> int:
+ return int(self.loadings.shape[1])
+
+
+@dataclass(frozen=True)
+class SymphonyCorrectionModel:
+ centroids: np.ndarray
+ raw_centroids: np.ndarray
+ corrected_centroids: np.ndarray
+ cluster_mass: np.ndarray
+ sigma: np.ndarray
+
+ def __post_init__(self) -> None:
+ if self.centroids.ndim != 2:
+ raise ValueError("Reference centroids must be two-dimensional")
+ n_clusters = self.centroids.shape[0]
+ n_dims = self.centroids.shape[1]
+ if self.raw_centroids.shape != (n_clusters, n_dims):
+ raise ValueError("Reference raw centroids have incompatible dimensions")
+ if self.corrected_centroids.shape != (n_clusters, n_dims):
+ raise ValueError(
+ "Reference corrected centroids have incompatible dimensions"
+ )
+ if self.cluster_mass.shape != (n_clusters,) or np.any(self.cluster_mass <= 0):
+ raise ValueError("Reference cluster masses must be positive")
+ if self.sigma.shape != (n_clusters,) or np.any(self.sigma <= 0):
+ raise ValueError("Reference kernel widths must be positive")
+ for values in (
+ self.centroids,
+ self.raw_centroids,
+ self.corrected_centroids,
+ self.cluster_mass,
+ self.sigma,
+ ):
+ if not np.all(np.isfinite(values)):
+ raise ValueError("Symphony correction model contains non-finite values")
+
+ @property
+ def n_dims(self) -> int:
+ return int(self.centroids.shape[1])
+
+ @property
+ def n_clusters(self) -> int:
+ return int(self.centroids.shape[0])
+
+
+@dataclass(frozen=True)
+class QueryCorrection:
+ batch_offsets: np.ndarray
+ batch_counts: np.ndarray
+
+ def __post_init__(self) -> None:
+ if self.batch_offsets.ndim != 3:
+ raise ValueError(
+ "Batch offsets must have batch, cluster, and dimension axes"
+ )
+ if self.batch_counts.shape != self.batch_offsets.shape[:2]:
+ raise ValueError("Batch counts must match batch offsets")
+ if not np.all(np.isfinite(self.batch_offsets)):
+ raise ValueError("Batch offsets contain non-finite values")
+
+
+@dataclass(frozen=True)
+class MappingResult:
+ ref: ArtifactRef
+ mapping_name: str
+ n_cells: int
+ correction_method: str
+ diagnostics: dict[str, float | int | str]
+ indices: np.ndarray | None = None
+ distances: np.ndarray | None = None
+ uninformative: np.ndarray | None = None
+ reference: "MappingReference | None" = field(
+ default=None,
+ repr=False,
+ compare=False,
+ )
+
+ def __repr__(self) -> str:
+ loaded = [
+ name
+ for name in ("indices", "distances", "uninformative")
+ if getattr(self, name) is not None
+ ]
+ return (
+ f"MappingResult(ref={self.ref!r}, "
+ f"mapping_name={self.mapping_name!r}, "
+ f"n_cells={self.n_cells}, "
+ f"correction_method={self.correction_method!r}, "
+ f"diagnostics={self.diagnostics!r}, "
+ f"arrays={loaded or 'not loaded'})"
+ )
diff --git a/scarf/mapping/projection.py b/scarf/mapping/projection.py
new file mode 100644
index 00000000..071dcaa2
--- /dev/null
+++ b/scarf/mapping/projection.py
@@ -0,0 +1,843 @@
+"""Persistence for query-owned mapping projection artifacts."""
+
+from collections.abc import Mapping
+from dataclasses import dataclass, field
+from typing import Any
+
+import numpy as np
+import zarr
+
+from ..storage.arrays import create_zarr_dataset
+from ..storage.artifact_writer import (
+ ArrayRequirement,
+ AttributeRequirement,
+ PlannedArtifact,
+ finish_artifact,
+ plan_artifact,
+ start_artifact,
+)
+from ..storage.artifacts import (
+ ArtifactRef,
+ ExternalArtifactRef,
+ artifact_group,
+ fingerprint_array,
+ inspect_artifact,
+ list_artifacts,
+)
+from ..storage.geometry import array_geometry
+from ..storage.partition import row_band
+from ..storage.profiles import StorageProfile
+from ..storage.types import as_zarr_array
+from .models import MappingResult
+from .reference import MappingReference
+
+PROJECTION_RERUN_MESSAGE = "Re-run run_mapping to create a new query projection."
+NO_QUERY_BATCH_FINGERPRINT = fingerprint_array(np.empty(0, dtype=np.int64))
+
+_PARAMETER_NAMES = frozenset(
+ {
+ "mapping_name",
+ "save_k",
+ "missing_feature_policy",
+ "correction_method",
+ }
+)
+_INPUT_NAMES = frozenset(
+ {
+ "cell_selection",
+ "feature_selection",
+ "selected_expression_fingerprint",
+ "query_batch_fingerprint",
+ "mapping_reference",
+ }
+)
+_ARRAY_NAMES = frozenset({"indices", "distances", "uninformative"})
+_DIAGNOSTIC_NAMES = frozenset(
+ {
+ "featureCoverage",
+ "queryBatchCount",
+ "algorithmVariant",
+ "zeroNormCellCount",
+ "queryScaledDispersion",
+ }
+)
+_ATTRIBUTE_NAMES = frozenset(
+ {
+ "artifact_id",
+ "kind",
+ "provenance",
+ "execution_options",
+ "created_at_ns",
+ "complete",
+ "diagnostics",
+ }
+)
+
+
+@dataclass(frozen=True, slots=True)
+class ProjectionPlan:
+ """A projection artifact plan that has not opened a writer."""
+
+ artifact: PlannedArtifact = field(repr=False)
+ n_cells: int
+ save_k: int
+ reference_cell_count: int
+
+ @property
+ def ref(self) -> ArtifactRef:
+ return self.artifact.ref
+
+ @property
+ def reused(self) -> bool:
+ return self.artifact.reused
+
+
+class ProjectionWriter:
+ """Write one query projection in contiguous bounded row blocks."""
+
+ def __init__(
+ self,
+ root: zarr.Group,
+ plan: ProjectionPlan,
+ *,
+ chunk_rows: int,
+ profile: StorageProfile | None = None,
+ ) -> None:
+ if not isinstance(plan, ProjectionPlan):
+ raise TypeError("plan must be a ProjectionPlan")
+ if plan.reused:
+ raise ValueError("A reused projection plan must be loaded without a writer")
+ resolved_chunk_rows = _positive_int(chunk_rows, "chunk_rows")
+ self._root = root
+ self._plan = plan
+ self._next_row = 0
+ self._uninformative_count = 0
+ self._finished = False
+ self._aborted = False
+ self._group = start_artifact(root, plan.artifact)
+ row_chunk = min(resolved_chunk_rows, plan.n_cells)
+ try:
+ self._indices = create_zarr_dataset(
+ self._group,
+ "indices",
+ (row_chunk, plan.save_k),
+ np.uint64,
+ (plan.n_cells, plan.save_k),
+ profile=profile,
+ )
+ self._distances = create_zarr_dataset(
+ self._group,
+ "distances",
+ (row_chunk, plan.save_k),
+ np.float64,
+ (plan.n_cells, plan.save_k),
+ profile=profile,
+ )
+ self._uninformative = create_zarr_dataset(
+ self._group,
+ "uninformative",
+ (row_chunk,),
+ bool,
+ (plan.n_cells,),
+ profile=profile,
+ )
+ except BaseException:
+ self._group.attrs["complete"] = False
+ self._aborted = True
+ raise
+
+ @property
+ def ref(self) -> ArtifactRef:
+ return self._plan.ref
+
+ @property
+ def next_row(self) -> int:
+ return self._next_row
+
+ @property
+ def finished(self) -> bool:
+ return self._finished
+
+ @property
+ def aborted(self) -> bool:
+ return self._aborted
+
+ def write_block(
+ self,
+ start: int,
+ indices: np.ndarray,
+ distances: np.ndarray,
+ uninformative: np.ndarray,
+ ) -> None:
+ """Write the next contiguous row block."""
+ self._require_open()
+ try:
+ if isinstance(start, bool) or not isinstance(start, int | np.integer):
+ raise TypeError("Projection block start must be an integer")
+ resolved_start = int(start)
+ if resolved_start != self._next_row:
+ raise ValueError(
+ "Projection blocks must be contiguous; "
+ f"expected {self._next_row}, received {resolved_start}"
+ )
+ index_values = np.asarray(indices)
+ distance_values = np.asarray(distances)
+ uninformative_values = np.asarray(uninformative)
+ if index_values.dtype.kind != "u":
+ raise TypeError("Projection indices must use an unsigned integer dtype")
+ if distance_values.dtype.kind != "f":
+ raise TypeError("Projection distances must use a floating dtype")
+ if uninformative_values.dtype != np.dtype(bool):
+ raise TypeError("Projection uninformative values must be boolean")
+ if index_values.ndim != 2 or index_values.shape[1] != self._plan.save_k:
+ raise ValueError(
+ "Projection index blocks must have shape "
+ f"(rows, {self._plan.save_k})"
+ )
+ if distance_values.shape != index_values.shape:
+ raise ValueError(
+ "Projection distance blocks must match the index block shape"
+ )
+ if uninformative_values.shape != (index_values.shape[0],):
+ raise ValueError(
+ "Projection uninformative blocks must have one value per row"
+ )
+ if index_values.shape[0] < 1:
+ raise ValueError("Projection blocks cannot be empty")
+ if not np.all(np.isfinite(distance_values)):
+ raise ValueError("Projection distances must be finite")
+ if np.any(distance_values < 0):
+ raise ValueError("Projection distances must be non-negative")
+ if np.any(index_values >= self._plan.reference_cell_count):
+ raise ValueError(
+ "Projection indices must identify selected reference cells"
+ )
+ stop = resolved_start + index_values.shape[0]
+ if stop > self._plan.n_cells:
+ raise ValueError("Projection block exceeds the declared cell count")
+ self._indices[resolved_start:stop] = index_values
+ self._distances[resolved_start:stop] = distance_values
+ self._uninformative[resolved_start:stop] = uninformative_values
+ self._next_row = stop
+ self._uninformative_count += int(np.count_nonzero(uninformative_values))
+ except BaseException:
+ self._group.attrs["complete"] = False
+ self._aborted = True
+ raise
+
+ def finish(self, diagnostics: Mapping[str, Any]) -> ArtifactRef:
+ """Validate and complete the projection artifact."""
+ self._require_open()
+ try:
+ if self._next_row != self._plan.n_cells:
+ raise ValueError(
+ "Projection rows are incomplete: "
+ f"wrote {self._next_row} of {self._plan.n_cells}"
+ )
+ validated = _validated_diagnostics(
+ diagnostics,
+ n_cells=self._plan.n_cells,
+ uninformative_count=self._uninformative_count,
+ )
+ self._group.attrs["diagnostics"] = validated
+ finish_artifact(self._group, self._plan.artifact)
+ except BaseException:
+ self._group.attrs["complete"] = False
+ self._aborted = True
+ raise
+ self._finished = True
+ return self._plan.ref
+
+ def abort(self) -> None:
+ """Leave an unfinished projection explicitly incomplete."""
+ if self._finished:
+ raise RuntimeError("A completed projection artifact cannot be aborted")
+ self._group.attrs["complete"] = False
+ self._aborted = True
+
+ def _require_open(self) -> None:
+ if self._finished:
+ raise RuntimeError("Projection writer is already finished")
+ if self._aborted:
+ raise RuntimeError("Projection writer is aborted")
+
+
+def plan_projection(
+ root: zarr.Group,
+ *,
+ query_assay: str,
+ mapping_name: str,
+ n_cells: int,
+ save_k: int,
+ missing_feature_policy: str,
+ correction_method: str,
+ cell_selection: ArtifactRef,
+ feature_selection: ArtifactRef,
+ selected_expression_fingerprint: str,
+ query_batch_fingerprint: str,
+ mapping_reference: ExternalArtifactRef,
+ reference_cell_count: int,
+ invalidate_cache: bool = False,
+) -> ProjectionPlan:
+ """Plan one immutable query-owned projection."""
+ assay = _nonempty_string(query_assay, "query_assay")
+ name = _nonempty_string(mapping_name, "mapping_name")
+ resolved_n_cells = _positive_int(n_cells, "n_cells")
+ resolved_save_k = _positive_int(save_k, "save_k")
+ resolved_reference_cell_count = _positive_int(
+ reference_cell_count,
+ "reference_cell_count",
+ )
+ policy = _nonempty_string(missing_feature_policy, "missing_feature_policy")
+ if policy not in {"reference_mean", "zero", "error"}:
+ raise ValueError(
+ "missing_feature_policy must be 'reference_mean', 'zero', or 'error'"
+ )
+ correction = _nonempty_string(correction_method, "correction_method")
+ expression_fingerprint = _nonempty_string(
+ selected_expression_fingerprint,
+ "selected_expression_fingerprint",
+ )
+ batch_fingerprint = _nonempty_string(
+ query_batch_fingerprint,
+ "query_batch_fingerprint",
+ )
+ _validate_local_selection(
+ root,
+ cell_selection,
+ kind="cell_selection",
+ scope="datastore",
+ assay=None,
+ )
+ selected_cells = _selection_count(root, cell_selection)
+ if selected_cells != resolved_n_cells:
+ raise ValueError("n_cells must equal the selected row count in cell_selection")
+ _validate_local_selection(
+ root,
+ feature_selection,
+ kind="feature_selection",
+ scope="assay",
+ assay=assay,
+ )
+ external = _validate_external_mapping_reference(mapping_reference)
+
+ def valid_projection(_ref: ArtifactRef, group: zarr.Group) -> bool:
+ try:
+ _validate_payload(
+ group,
+ expected_n_cells=resolved_n_cells,
+ expected_save_k=resolved_save_k,
+ reference_cell_count=resolved_reference_cell_count,
+ )
+ except (KeyError, RuntimeError, TypeError, ValueError):
+ return False
+ return True
+
+ planned = plan_artifact(
+ root,
+ scope="assay",
+ assay=assay,
+ kind="projection",
+ operation="map_query",
+ parameters={
+ "mapping_name": name,
+ "save_k": resolved_save_k,
+ "missing_feature_policy": policy,
+ "correction_method": correction,
+ },
+ inputs={
+ "cell_selection": cell_selection,
+ "feature_selection": feature_selection,
+ "selected_expression_fingerprint": expression_fingerprint,
+ "query_batch_fingerprint": batch_fingerprint,
+ "mapping_reference": external,
+ },
+ execution_options={},
+ invalidate_cache=invalidate_cache,
+ required_arrays=(
+ ArrayRequirement(
+ "indices",
+ shape=(resolved_n_cells, resolved_save_k),
+ dtype_kind="u",
+ ),
+ ArrayRequirement(
+ "distances",
+ shape=(resolved_n_cells, resolved_save_k),
+ dtype_kind="f",
+ ),
+ ArrayRequirement(
+ "uninformative",
+ shape=(resolved_n_cells,),
+ dtype=bool,
+ ),
+ ),
+ required_attributes=(
+ AttributeRequirement("diagnostics", expected_types=(dict,)),
+ ),
+ reuse_validator=valid_projection,
+ )
+ return ProjectionPlan(
+ artifact=planned,
+ n_cells=resolved_n_cells,
+ save_k=resolved_save_k,
+ reference_cell_count=resolved_reference_cell_count,
+ )
+
+
+def load_projection(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ *,
+ load_arrays: bool = False,
+ reference: MappingReference | None = None,
+) -> MappingResult:
+ """Load a query projection after validating the complete contract."""
+ if not isinstance(load_arrays, bool):
+ raise TypeError("load_arrays must be a boolean")
+ if reference is not None and not isinstance(reference, MappingReference):
+ raise TypeError("reference must be a MappingReference or None")
+ try:
+ return _load_projection(
+ root,
+ ref,
+ load_arrays=load_arrays,
+ reference=reference,
+ )
+ except (KeyError, RuntimeError, TypeError, ValueError) as exc:
+ raise _contract_error(str(exc)) from None
+
+
+def resolve_projection(
+ root: zarr.Group,
+ *,
+ query_assay: str,
+ mapping_name: str,
+ mapping_reference: ExternalArtifactRef,
+) -> ArtifactRef:
+ """Resolve the newest projection for one query name and reference."""
+ try:
+ assay = _nonempty_string(query_assay, "query_assay")
+ name = _nonempty_string(mapping_name, "mapping_name")
+ external = _validate_external_mapping_reference(mapping_reference)
+ candidates: list[tuple[int, ArtifactRef]] = []
+ for ref in list_artifacts(
+ root,
+ scope="assay",
+ assay=assay,
+ kind="projection",
+ ):
+ status = inspect_artifact(root, ref)
+ if not status.complete or status.operation != "map_query":
+ continue
+ parameters = status.parameters or {}
+ if parameters.get("mapping_name") != name:
+ continue
+ if set(parameters) != _PARAMETER_NAMES:
+ raise ValueError(
+ f"Projection {name!r} has malformed map_query parameters"
+ )
+ inputs = status.inputs or {}
+ if set(inputs) != _INPUT_NAMES:
+ raise ValueError(f"Projection {name!r} has malformed map_query inputs")
+ raw_external = inputs["mapping_reference"]
+ if not isinstance(raw_external, Mapping):
+ raise TypeError("Projection mapping_reference input is malformed")
+ stored_external = ExternalArtifactRef.from_dict(raw_external)
+ _validate_external_mapping_reference(stored_external)
+ if stored_external != external:
+ continue
+ group = artifact_group(root, ref)
+ created_at_ns = _created_at_ns(group)
+ candidates.append((created_at_ns, ref))
+ if not candidates:
+ raise ValueError(
+ f"No complete projection named {name!r} matches the reference"
+ )
+ candidates.sort(
+ key=lambda item: (item[0], item[1].artifact_id),
+ reverse=True,
+ )
+ return candidates[0][1]
+ except (KeyError, RuntimeError, TypeError, ValueError) as exc:
+ raise _contract_error(str(exc)) from None
+
+
+def _load_projection(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ *,
+ load_arrays: bool,
+ reference: MappingReference | None,
+) -> MappingResult:
+ _validate_projection_ref(ref)
+ status = inspect_artifact(root, ref)
+ if not status.exists or not status.complete:
+ raise ValueError("Projection artifact is missing or incomplete")
+ if status.operation != "map_query":
+ raise ValueError("Projection artifact has an old operation")
+
+ parameters = status.parameters or {}
+ if set(parameters) != _PARAMETER_NAMES:
+ raise ValueError("Projection parameters do not match the map_query contract")
+ mapping_name = _nonempty_string(parameters["mapping_name"], "mapping_name")
+ save_k = _positive_int(parameters["save_k"], "save_k")
+ policy = _nonempty_string(
+ parameters["missing_feature_policy"],
+ "missing_feature_policy",
+ )
+ if policy not in {"reference_mean", "zero", "error"}:
+ raise ValueError("Projection missing_feature_policy is unsupported")
+ correction_method = _nonempty_string(
+ parameters["correction_method"],
+ "correction_method",
+ )
+
+ inputs = status.inputs or {}
+ if set(inputs) != _INPUT_NAMES:
+ raise ValueError("Projection inputs do not match the map_query contract")
+ cell_selection = _local_ref_from_input(
+ inputs,
+ "cell_selection",
+ kind="cell_selection",
+ scope="datastore",
+ assay=None,
+ )
+ feature_selection = _local_ref_from_input(
+ inputs,
+ "feature_selection",
+ kind="feature_selection",
+ scope="assay",
+ assay=ref.assay,
+ )
+ _nonempty_string(
+ inputs["selected_expression_fingerprint"],
+ "selected_expression_fingerprint",
+ )
+ _nonempty_string(
+ inputs["query_batch_fingerprint"],
+ "query_batch_fingerprint",
+ )
+ raw_external = inputs["mapping_reference"]
+ if not isinstance(raw_external, Mapping):
+ raise TypeError("Projection mapping_reference input is malformed")
+ external = _validate_external_mapping_reference(
+ ExternalArtifactRef.from_dict(raw_external)
+ )
+ reference_cell_count = None
+ if reference is not None:
+ reference.validate_dataset_fingerprint()
+ provided_external = reference.external_ref
+ if provided_external != external:
+ raise ValueError(
+ "Provided mapping reference does not match the projection input; "
+ f"expected {external!r}, received {provided_external!r}"
+ )
+ reference_cell_count = _positive_int(
+ reference.selected_cell_count,
+ "Mapping reference selected_cell_count",
+ )
+ _validate_local_selection(
+ root,
+ cell_selection,
+ kind="cell_selection",
+ scope="datastore",
+ assay=None,
+ )
+ _validate_local_selection(
+ root,
+ feature_selection,
+ kind="feature_selection",
+ scope="assay",
+ assay=ref.assay,
+ )
+
+ group = artifact_group(root, ref)
+ n_cells, diagnostics = _validate_payload(
+ group,
+ expected_save_k=save_k,
+ reference_cell_count=reference_cell_count,
+ )
+ if _selection_count(root, cell_selection) != n_cells:
+ raise ValueError("Projection rows do not match the stored query cell selection")
+
+ indices = distances = uninformative = None
+ if load_arrays:
+ indices = np.array(
+ as_zarr_array(group["indices"], name="indices")[:],
+ copy=True,
+ )
+ distances = np.array(
+ as_zarr_array(group["distances"], name="distances")[:],
+ copy=True,
+ )
+ uninformative = np.array(
+ as_zarr_array(group["uninformative"], name="uninformative")[:],
+ copy=True,
+ )
+ return MappingResult(
+ ref=ref,
+ mapping_name=mapping_name,
+ n_cells=n_cells,
+ correction_method=correction_method,
+ diagnostics=diagnostics,
+ indices=indices,
+ distances=distances,
+ uninformative=uninformative,
+ reference=reference,
+ )
+
+
+def _validate_payload(
+ group: zarr.Group,
+ *,
+ expected_n_cells: int | None = None,
+ expected_save_k: int | None = None,
+ reference_cell_count: int | None = None,
+) -> tuple[int, dict[str, float | int | str]]:
+ if set(group.group_keys()):
+ raise ValueError("Projection payload contains unexpected groups")
+ arrays = set(group.array_keys())
+ if arrays != _ARRAY_NAMES:
+ missing = _ARRAY_NAMES - arrays
+ extra = arrays - _ARRAY_NAMES
+ details = []
+ if missing:
+ details.append("missing " + ", ".join(sorted(missing)))
+ if extra:
+ details.append("unexpected " + ", ".join(sorted(extra)))
+ raise ValueError("Projection payload arrays are invalid: " + "; ".join(details))
+ if set(group.attrs) != _ATTRIBUTE_NAMES:
+ raise ValueError("Projection attributes do not match the map_query contract")
+ _created_at_ns(group)
+
+ indices = as_zarr_array(group["indices"], name="indices")
+ distances = as_zarr_array(group["distances"], name="distances")
+ uninformative = as_zarr_array(
+ group["uninformative"],
+ name="uninformative",
+ )
+ if indices.ndim != 2 or indices.shape[0] < 1 or indices.shape[1] < 1:
+ raise ValueError("Projection indices must be a non-empty matrix")
+ if np.dtype(indices.dtype).kind != "u":
+ raise TypeError("Projection indices must use an unsigned integer dtype")
+ if distances.shape != indices.shape or np.dtype(distances.dtype).kind != "f":
+ raise TypeError(
+ "Projection distances must be a floating matrix matching indices"
+ )
+ n_cells = int(indices.shape[0])
+ save_k = int(indices.shape[1])
+ if expected_n_cells is not None and n_cells != expected_n_cells:
+ raise ValueError("Projection cell count does not match its plan")
+ if expected_save_k is not None and save_k != expected_save_k:
+ raise ValueError("Projection neighbor count does not match save_k")
+ if uninformative.shape != (n_cells,) or np.dtype(uninformative.dtype) != np.dtype(
+ bool
+ ):
+ raise TypeError("Projection uninformative must be a boolean row vector")
+
+ block_rows = min(
+ row_band(array_geometry(indices), unit="chunk", fallback=1),
+ row_band(array_geometry(distances), unit="chunk", fallback=1),
+ row_band(array_geometry(uninformative), unit="chunk", fallback=1),
+ )
+ uninformative_count = 0
+ for start in range(0, n_cells, block_rows):
+ stop = min(start + block_rows, n_cells)
+ if reference_cell_count is not None:
+ index_block = np.asarray(indices[start:stop])
+ if np.any(index_block >= reference_cell_count):
+ raise ValueError(
+ "Projection indices contain a neighbor outside the selected "
+ "reference cell range"
+ )
+ distance_block = np.asarray(distances[start:stop])
+ if not np.all(np.isfinite(distance_block)):
+ raise ValueError("Projection distances must be finite")
+ if np.any(distance_block < 0):
+ raise ValueError("Projection distances must be non-negative")
+ uninformative_count += int(
+ np.count_nonzero(np.asarray(uninformative[start:stop], dtype=bool))
+ )
+ raw_diagnostics = group.attrs["diagnostics"]
+ if not isinstance(raw_diagnostics, Mapping):
+ raise TypeError("Projection diagnostics must be a mapping")
+ diagnostics = _validated_diagnostics(
+ raw_diagnostics,
+ n_cells=n_cells,
+ uninformative_count=uninformative_count,
+ )
+ return n_cells, diagnostics
+
+
+def _validated_diagnostics(
+ diagnostics: Mapping[str, Any],
+ *,
+ n_cells: int,
+ uninformative_count: int,
+) -> dict[str, float | int | str]:
+ if set(diagnostics) != _DIAGNOSTIC_NAMES:
+ missing = sorted(_DIAGNOSTIC_NAMES - set(diagnostics))
+ detail = "Projection diagnostics must contain exactly " + ", ".join(
+ sorted(_DIAGNOSTIC_NAMES)
+ )
+ if missing:
+ detail += ", and is missing " + ", ".join(missing)
+ raise ValueError(detail)
+ feature_coverage = diagnostics["featureCoverage"]
+ if (
+ isinstance(feature_coverage, bool | np.bool_)
+ or not isinstance(feature_coverage, float | np.floating)
+ or not np.isfinite(feature_coverage)
+ or not 0 < float(feature_coverage) <= 1
+ ):
+ raise ValueError("featureCoverage must be a finite float in (0, 1]")
+ query_batch_count = _positive_int(
+ diagnostics["queryBatchCount"],
+ "queryBatchCount",
+ )
+ if query_batch_count > n_cells:
+ raise ValueError("queryBatchCount cannot exceed the projection cell count")
+ algorithm_variant = _nonempty_string(
+ diagnostics["algorithmVariant"],
+ "algorithmVariant",
+ )
+ zero_norm_count = _nonnegative_int(
+ diagnostics["zeroNormCellCount"],
+ "zeroNormCellCount",
+ )
+ if zero_norm_count > n_cells:
+ raise ValueError("zeroNormCellCount cannot exceed the projection cell count")
+ if zero_norm_count != uninformative_count:
+ raise ValueError(
+ "zeroNormCellCount must equal the number of uninformative rows"
+ )
+ dispersion = diagnostics["queryScaledDispersion"]
+ if (
+ isinstance(dispersion, bool | np.bool_)
+ or not isinstance(dispersion, float | np.floating)
+ or not np.isfinite(dispersion)
+ or float(dispersion) < 0
+ ):
+ raise ValueError("queryScaledDispersion must be a finite non-negative float")
+ return {
+ "featureCoverage": float(feature_coverage),
+ "queryBatchCount": query_batch_count,
+ "algorithmVariant": algorithm_variant,
+ "zeroNormCellCount": zero_norm_count,
+ "queryScaledDispersion": float(dispersion),
+ }
+
+
+def _validate_projection_ref(ref: ArtifactRef) -> None:
+ if (
+ not isinstance(ref, ArtifactRef)
+ or ref.scope != "assay"
+ or ref.assay is None
+ or ref.kind != "projection"
+ ):
+ raise ValueError("Expected an assay-scoped projection ArtifactRef")
+
+
+def _validate_external_mapping_reference(
+ reference: ExternalArtifactRef,
+) -> ExternalArtifactRef:
+ if not isinstance(reference, ExternalArtifactRef):
+ raise TypeError("mapping_reference must be an ExternalArtifactRef")
+ if reference.ref.kind != "mapping_reference":
+ raise ValueError("mapping_reference must identify a mapping_reference artifact")
+ return reference
+
+
+def _local_ref_from_input(
+ inputs: Mapping[str, Any],
+ name: str,
+ *,
+ kind: str,
+ scope: str,
+ assay: str | None,
+) -> ArtifactRef:
+ raw_ref = inputs[name]
+ if not isinstance(raw_ref, Mapping):
+ raise TypeError(f"Projection input {name!r} must be an ArtifactRef")
+ expected_keys = {"type", "scope", "kind", "artifact_id"}
+ if scope == "assay":
+ expected_keys.add("assay")
+ if set(raw_ref) != expected_keys:
+ raise ValueError(f"Projection input {name!r} is malformed")
+ ref = ArtifactRef.from_dict(raw_ref)
+ if ref.kind != kind or ref.scope != scope or ref.assay != assay:
+ raise ValueError(f"Projection input {name!r} has the wrong kind or scope")
+ return ref
+
+
+def _validate_local_selection(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ *,
+ kind: str,
+ scope: str,
+ assay: str | None,
+) -> None:
+ if not isinstance(ref, ArtifactRef):
+ raise TypeError(f"{kind} must be an ArtifactRef")
+ if ref.kind != kind or ref.scope != scope or ref.assay != assay:
+ raise ValueError(f"{kind} has the wrong artifact kind or scope")
+ status = inspect_artifact(root, ref)
+ if not status.exists or not status.complete:
+ raise ValueError(f"{kind} must be a complete local artifact")
+ values = as_zarr_array(
+ artifact_group(root, ref)["values"],
+ name=f"{kind}.values",
+ )
+ if values.ndim != 1 or np.dtype(values.dtype) != np.dtype(bool):
+ raise TypeError(f"{kind} values must be a boolean row vector")
+
+
+def _selection_count(root: zarr.Group, ref: ArtifactRef) -> int:
+ values = as_zarr_array(
+ artifact_group(root, ref)["values"],
+ name=f"{ref.kind}.values",
+ )
+ block_rows = row_band(array_geometry(values), unit="chunk", fallback=1)
+ selected = 0
+ for start in range(0, int(values.shape[0]), block_rows):
+ stop = min(start + block_rows, int(values.shape[0]))
+ selected += int(np.count_nonzero(np.asarray(values[start:stop], dtype=bool)))
+ return selected
+
+
+def _created_at_ns(group: zarr.Group) -> int:
+ value = group.attrs.get("created_at_ns")
+ if (
+ isinstance(value, bool | np.bool_)
+ or not isinstance(value, int | np.integer)
+ or int(value) < 1
+ ):
+ raise ValueError("Projection created_at_ns must be a positive integer")
+ return int(value)
+
+
+def _nonempty_string(value: Any, name: str) -> str:
+ if not isinstance(value, str) or not value.strip():
+ raise TypeError(f"{name} must be a non-empty string")
+ return value
+
+
+def _positive_int(value: Any, name: str) -> int:
+ resolved = _nonnegative_int(value, name)
+ if resolved < 1:
+ raise ValueError(f"{name} must be positive")
+ return resolved
+
+
+def _nonnegative_int(value: Any, name: str) -> int:
+ if isinstance(value, bool | np.bool_) or not isinstance(value, int | np.integer):
+ raise TypeError(f"{name} must be an integer")
+ resolved = int(value)
+ if resolved < 0:
+ raise ValueError(f"{name} must be non-negative")
+ return resolved
+
+
+def _contract_error(detail: str) -> ValueError:
+ return ValueError(f"{detail}. {PROJECTION_RERUN_MESSAGE}")
diff --git a/scarf/mapping/reference.py b/scarf/mapping/reference.py
new file mode 100644
index 00000000..e4e5493d
--- /dev/null
+++ b/scarf/mapping/reference.py
@@ -0,0 +1,201 @@
+"""Persistent handles for immutable mapping references."""
+
+from dataclasses import dataclass
+from collections.abc import Mapping
+from typing import Any
+
+import numpy as np
+
+from ..storage.artifacts import artifact_group, inspect_artifact
+from ..storage.refs import ArtifactRef, ExternalArtifactRef
+from ..storage.types import as_zarr_array, as_zarr_group
+from .models import (
+ ScaledPCAProjectionModel,
+ SymphonyCorrectionModel,
+)
+
+
+@dataclass(frozen=True)
+class MappingReference:
+ """An immutable scaled-PCA reference loaded from a Scarf Zarr store."""
+
+ datastore: Any
+ ref: ArtifactRef
+ assay_name: str
+ cell_key: str
+ feature_key: str
+ reduction: ArtifactRef
+ ann_index: ArtifactRef
+ neighbors: ArtifactRef
+ cell_selection: ArtifactRef
+ feature_selection: ArtifactRef
+ batch_correction: ArtifactRef | None
+ dataset_fingerprint: str
+ selected_cell_count: int
+ model: ScaledPCAProjectionModel
+ symphony_state: SymphonyCorrectionModel | None
+ feature_ids: np.ndarray
+ metadata: dict[str, Any]
+ reference_distance_quantiles: np.ndarray
+ reference_distance_values: np.ndarray
+
+ @property
+ def method(self) -> str:
+ return str(self.metadata["method"])
+
+ @property
+ def ann_metric(self) -> str:
+ return str(self.metadata["ann_metric"])
+
+ @property
+ def normalization_parameters(self) -> dict[str, Any]:
+ values = self.metadata["normalization_parameters"]
+ if not isinstance(values, Mapping):
+ raise TypeError("Mapping reference normalization parameters are invalid")
+ return dict(values)
+
+ @property
+ def size_factor(self) -> float:
+ return float(self.normalization_parameters["size_factor"])
+
+ @property
+ def external_ref(self) -> ExternalArtifactRef:
+ return ExternalArtifactRef(
+ dataset_fingerprint=self.dataset_fingerprint,
+ ref=self.ref,
+ )
+
+ def validate_dataset_fingerprint(self) -> None:
+ assay = self.datastore._get_assay(self.assay_name)
+ live = assay.attrs.get("dataset_fingerprint")
+ if live is None:
+ raise ValueError(
+ f"Reference assay {self.assay_name!r} has no stored dataset "
+ "fingerprint. Rebuild it with build_mapping_reference(neighbors)."
+ )
+ if live != self.dataset_fingerprint:
+ raise ValueError(
+ f"Reference assay {self.assay_name!r} dataset fingerprint mismatch. "
+ f"Expected {self.dataset_fingerprint!r}, received {live!r}. "
+ "Rebuild it with build_mapping_reference(neighbors)."
+ )
+
+ def fetch_cell_column(self, column: str) -> np.ndarray:
+ """Fetch one reference cell column through the stored cell selection."""
+ from ..graph.state import validate_cell_selection_artifact
+
+ self.validate_dataset_fingerprint()
+ validate_cell_selection_artifact(
+ self.datastore.zw,
+ self.cell_selection,
+ self.cell_key,
+ )
+ values = np.asarray(self.datastore.cells.fetch(column, key=self.cell_key))
+ if len(values) != self.selected_cell_count:
+ raise ValueError(
+ "The selected reference cell count has changed. Rebuild the "
+ "mapping reference with build_mapping_reference(neighbors)."
+ )
+ return values
+
+ def fetch_layout(self, layout_key: str) -> np.ndarray:
+ """Fetch a two-column layout for the selected reference cells."""
+ source = self.layout_source(layout_key)
+ if source is None:
+ raw_layout = np.column_stack(
+ (
+ self.fetch_cell_column(f"{layout_key}1"),
+ self.fetch_cell_column(f"{layout_key}2"),
+ )
+ )
+ else:
+ cell_data = as_zarr_group(
+ self.datastore.zw["cellData"],
+ name="cellData",
+ )
+ first_column = as_zarr_array(
+ cell_data[f"{layout_key}1"],
+ name=f"{layout_key}1",
+ )
+ source_value = first_column.attrs.get("source_value")
+ if not isinstance(source_value, str) or not source_value:
+ raise ValueError("Linked reference layout source is invalid")
+ source_values = as_zarr_array(
+ artifact_group(self.datastore.zw, source.ref)[source_value],
+ name=source_value,
+ )
+ raw_layout = np.asarray(source_values[:, :2])
+ try:
+ layout = np.asarray(raw_layout, dtype=np.float64)
+ except (TypeError, ValueError) as exc:
+ raise TypeError("Reference layout coordinates must be numeric") from exc
+ if layout.shape != (self.selected_cell_count, 2):
+ raise ValueError(
+ "Reference layout must have two columns and one row per "
+ "selected reference cell"
+ )
+ if not np.all(np.isfinite(layout) | np.isnan(layout)):
+ raise ValueError("Reference layout contains infinite coordinates")
+ return np.array(layout, copy=True)
+
+ def layout_source(self, layout_key: str) -> ExternalArtifactRef | None:
+ """Return the shared assay artifact linked to two layout columns."""
+ self.validate_dataset_fingerprint()
+ cell_data = as_zarr_group(
+ self.datastore.zw["cellData"],
+ name="cellData",
+ )
+ refs: list[ArtifactRef] = []
+ source_values: list[str] = []
+ for expected_index, column in enumerate((f"{layout_key}1", f"{layout_key}2")):
+ try:
+ values = as_zarr_array(cell_data[column], name=column)
+ except (KeyError, TypeError):
+ return None
+ raw_ref = values.attrs.get("source_artifact")
+ source_value = values.attrs.get("source_value")
+ value_index = values.attrs.get("value_index")
+ if not isinstance(raw_ref, Mapping):
+ return None
+ if not isinstance(source_value, str) or not source_value:
+ return None
+ if (
+ isinstance(value_index, bool | np.bool_)
+ or not isinstance(value_index, int | np.integer)
+ or int(value_index) != expected_index
+ ):
+ return None
+ try:
+ ref = ArtifactRef.from_dict(raw_ref)
+ status = inspect_artifact(self.datastore.zw, ref)
+ except (KeyError, TypeError, ValueError):
+ return None
+ if (
+ ref.scope != "assay"
+ or ref.assay != self.assay_name
+ or not status.complete
+ ):
+ return None
+ refs.append(ref)
+ source_values.append(source_value)
+ if refs[0] != refs[1]:
+ return None
+ if source_values[0] != source_values[1]:
+ return None
+ try:
+ source = as_zarr_array(
+ artifact_group(self.datastore.zw, refs[0])[source_values[0]],
+ name=source_values[0],
+ )
+ except (KeyError, TypeError, ValueError):
+ return None
+ if (
+ source.ndim != 2
+ or int(source.shape[0]) != self.selected_cell_count
+ or int(source.shape[1]) < 2
+ ):
+ return None
+ return ExternalArtifactRef(
+ dataset_fingerprint=self.dataset_fingerprint,
+ ref=refs[0],
+ )
diff --git a/scarf/mapping/symphony.py b/scarf/mapping/symphony.py
new file mode 100644
index 00000000..fefddbc8
--- /dev/null
+++ b/scarf/mapping/symphony.py
@@ -0,0 +1,194 @@
+"""Portable numerical primitives for Symphony-style reference mapping."""
+
+from typing import cast
+
+import numpy as np
+
+from .models import (
+ QueryCorrection,
+ ScaledPCAProjectionModel,
+ SymphonyCorrectionModel,
+)
+
+SYMPHONY_ALGORITHM = "symphony"
+
+
+def project_pca(values: np.ndarray, model: ScaledPCAProjectionModel) -> np.ndarray:
+ """Project normalized expression onto immutable reference PCA loadings."""
+ values = np.asarray(values, dtype=np.float64)
+ if values.ndim != 2 or values.shape[1] != model.n_features:
+ raise ValueError(
+ f"Expected query matrix with {model.n_features} features, got {values.shape}"
+ )
+ projected = ((values - model.feature_means) / model.feature_scales) @ model.loadings
+ if not np.all(np.isfinite(projected)):
+ raise ValueError("PCA projection produced non-finite values")
+ return cast(np.ndarray, projected)
+
+
+def scaled_dispersion_sum(values: np.ndarray, model: ScaledPCAProjectionModel) -> float:
+ """Return the summed squared scaled deviation of one aligned query block.
+
+ The reference PCA is fitted on z-scored features, so dividing this sum by the
+ cell and feature counts returns exactly 1 for the reference itself. A query
+ that returns much less than 1 occupies a narrower region of the same space
+ and its cells collect near the middle of the reference cloud, where the
+ retrieved neighbors stop reflecting the query's own structure.
+ """
+ values = np.asarray(values, dtype=np.float64)
+ if values.ndim != 2 or values.shape[1] != model.n_features:
+ raise ValueError(
+ f"Expected query matrix with {model.n_features} features, got {values.shape}"
+ )
+ scaled = (values - model.feature_means) / model.feature_scales
+ return float(np.einsum("ij,ij->", scaled, scaled))
+
+
+def soft_cluster_assignments(
+ coordinates: np.ndarray, model: SymphonyCorrectionModel
+) -> np.ndarray:
+ """Calculate cosine-kernel soft assignments to reference centroids."""
+ values = np.asarray(coordinates, dtype=np.float64)
+ if values.ndim != 2 or values.shape[1] != model.n_dims:
+ raise ValueError("Query coordinates have incompatible dimensions")
+ query_unit = _normalize_rows(values)
+ centroids_unit = _normalize_rows(model.centroids)
+ distances = 2.0 * (1.0 - query_unit @ centroids_unit.T)
+ logits = -distances / model.sigma[np.newaxis, :]
+ logits -= logits.max(axis=1, keepdims=True)
+ assignments = np.exp(logits)
+ assignments /= assignments.sum(axis=1, keepdims=True)
+ if not np.all(np.isfinite(assignments)):
+ raise ValueError("Soft cluster assignments contain non-finite values")
+ return cast(np.ndarray, assignments)
+
+
+def zero_norm_rows(coordinates: np.ndarray) -> np.ndarray:
+ """Return rows without directional information in reference PC space."""
+ values = np.asarray(coordinates, dtype=np.float64)
+ if values.ndim != 2:
+ raise ValueError("Query coordinates must be two-dimensional")
+ return cast(np.ndarray, np.linalg.norm(values, axis=1) == 0)
+
+
+def initialize_sufficient_statistics(
+ n_batches: int, model: SymphonyCorrectionModel
+) -> tuple[np.ndarray, np.ndarray]:
+ if n_batches < 1:
+ raise ValueError("At least one query batch is required")
+ return (
+ np.zeros((n_batches, model.n_clusters), dtype=np.float64),
+ np.zeros((n_batches, model.n_clusters, model.n_dims), dtype=np.float64),
+ )
+
+
+def accumulate_sufficient_statistics(
+ counts: np.ndarray,
+ sums: np.ndarray,
+ coordinates: np.ndarray,
+ assignments: np.ndarray,
+ batch_codes: np.ndarray,
+) -> None:
+ """Accumulate per-query-batch, per-cluster weighted coordinate sums."""
+ if coordinates.shape[0] != len(batch_codes) or assignments.shape[0] != len(
+ batch_codes
+ ):
+ raise ValueError("Query batch codes must match coordinate rows")
+ if assignments.shape[1] != counts.shape[1]:
+ raise ValueError("Assignment count does not match reference clusters")
+ if np.any(batch_codes < 0) or np.any(batch_codes >= counts.shape[0]):
+ raise ValueError("Query batch codes are out of range")
+
+ for batch_code in np.unique(batch_codes):
+ batch_mask = batch_codes == batch_code
+ batch_assignments = assignments[batch_mask]
+ counts[batch_code] += batch_assignments.sum(axis=0)
+ sums[batch_code] += batch_assignments.T @ coordinates[batch_mask]
+
+
+def solve_query_correction(
+ counts: np.ndarray,
+ sums: np.ndarray,
+ model: SymphonyCorrectionModel,
+) -> QueryCorrection:
+ """Fit Symphony's joint cluster-aware query batch correction."""
+ if counts.ndim != 2 or counts.shape[1] != model.n_clusters:
+ raise ValueError("Query count statistics have incompatible dimensions")
+ if sums.shape != (counts.shape[0], model.n_clusters, model.n_dims):
+ raise ValueError("Query sum statistics have incompatible dimensions")
+ n_batches = counts.shape[0]
+ offsets = np.zeros_like(sums)
+ for cluster in range(model.n_clusters):
+ cluster_counts = counts[:, cluster]
+ design_crossproduct = np.zeros(
+ (n_batches + 1, n_batches + 1),
+ dtype=np.float64,
+ )
+ design_crossproduct[0, 0] = cluster_counts.sum() + model.cluster_mass[cluster]
+ design_crossproduct[0, 1:] = cluster_counts
+ design_crossproduct[1:, 0] = cluster_counts
+ design_crossproduct[1:, 1:] = np.diag(cluster_counts + 1.0)
+
+ coordinate_crossproduct = np.empty(
+ (n_batches + 1, model.n_dims),
+ dtype=np.float64,
+ )
+ coordinate_crossproduct[0] = (
+ sums[:, cluster].sum(axis=0)
+ + model.cluster_mass[cluster] * model.corrected_centroids[cluster]
+ )
+ coordinate_crossproduct[1:] = sums[:, cluster]
+ coefficients = np.linalg.solve(
+ design_crossproduct,
+ coordinate_crossproduct,
+ )
+ offsets[:, cluster] = coefficients[1:]
+ return QueryCorrection(batch_offsets=offsets, batch_counts=counts.copy())
+
+
+def apply_query_correction(
+ coordinates: np.ndarray,
+ assignments: np.ndarray,
+ batch_codes: np.ndarray,
+ model: SymphonyCorrectionModel,
+ correction: QueryCorrection,
+) -> np.ndarray:
+ """Map query PCA coordinates into the fixed corrected reference space."""
+ if coordinates.shape[0] != assignments.shape[0] or coordinates.shape[0] != len(
+ batch_codes
+ ):
+ raise ValueError("Query coordinate, assignment, and batch rows must agree")
+ if assignments.shape[1] != model.n_clusters:
+ raise ValueError("Query assignments have incompatible cluster count")
+ if correction.batch_offsets.shape[1:] != (model.n_clusters, model.n_dims):
+ raise ValueError("Query correction has incompatible reference dimensions")
+ query_offsets = np.einsum(
+ "nk,nkd->nd", assignments, correction.batch_offsets[batch_codes]
+ )
+ corrected = coordinates - query_offsets
+ if not np.all(np.isfinite(corrected)):
+ raise ValueError("Query correction produced non-finite coordinates")
+ return cast(np.ndarray, corrected)
+
+
+def weighted_centroids(
+ coordinates: np.ndarray, assignments: np.ndarray
+) -> tuple[np.ndarray, np.ndarray]:
+ """Return soft cluster masses and centroids for reference compression."""
+ values = np.asarray(coordinates, dtype=np.float64)
+ weights = np.asarray(assignments, dtype=np.float64)
+ if values.ndim != 2 or weights.ndim != 2 or values.shape[0] != weights.shape[1]:
+ raise ValueError("Coordinate and assignment dimensions do not agree")
+ mass = weights.sum(axis=1)
+ if np.any(mass <= 0):
+ raise ValueError("Reference assignments include an empty cluster")
+ centroids = (weights @ values / mass[:, np.newaxis]).astype(np.float64)
+ return mass, centroids
+
+
+def _normalize_rows(values: np.ndarray) -> np.ndarray:
+ norms = np.linalg.norm(values, axis=1, keepdims=True)
+ normalized = np.zeros_like(values, dtype=np.float64)
+ nonzero = norms[:, 0] > 0
+ normalized[nonzero] = values[nonzero] / norms[nonzero]
+ return normalized
diff --git a/scarf/mapping_utils.py b/scarf/mapping_utils.py
deleted file mode 100644
index cd10f5cb..00000000
--- a/scarf/mapping_utils.py
+++ /dev/null
@@ -1,214 +0,0 @@
-"""Utility functions for the mapping."""
-
-from typing import Tuple
-
-import dask.array as daskarr
-import numpy as np
-import pandas as pd
-
-from .assay import Assay
-from .utils import controlled_compute, show_dask_progress, logger, tqdmbar
-
-__all__ = ["align_features", "coral"]
-
-
-def _cov_diaged(da: daskarr) -> daskarr:
- a = daskarr.cov(da, rowvar=0)
- a[a == np.inf] = 0
- a[a == np.nan] = 0
- return a + np.eye(a.shape[0])
-
-
-def _correlation_alignment(s: daskarr, t: daskarr, nthreads: int) -> daskarr:
- from scipy.linalg import fractional_matrix_power as fmp
- from threadpoolctl import threadpool_limits
-
- s_cov = show_dask_progress(
- _cov_diaged(s), f"CORAL: Computing source covariance", nthreads
- )
- t_cov = show_dask_progress(
- _cov_diaged(t), f"CORAL: Computing target covariance", nthreads
- )
- logger.info(
- "Calculating fractional power of covariance matrices. This might take a while... "
- )
- with threadpool_limits(limits=nthreads):
- a_coral = np.dot(fmp(s_cov, -0.5), fmp(t_cov, 0.5))
- logger.info("Fractional power calculation complete")
- return daskarr.dot(s, a_coral)
-
-
-def coral(source_data, target_data, assay, feat_key: str, cell_key: str, nthreads: int):
- """Applies CORAL error correction to the input data.
-
- Args:
- source_data ():
- target_data ():
- assay ():
- feat_key ():
- cell_key ():
- nthreads ():
- """
- from .writers import dask_to_zarr
- from .utils import clean_array
-
- sm = clean_array(
- show_dask_progress(
- source_data.mean(axis=0),
- "CORAL: Computing source feature means",
- nthreads,
- )
- )
- sd = clean_array(
- show_dask_progress(
- source_data.std(axis=0),
- "CORAL: Computing source feature stdev",
- nthreads,
- ),
- 1,
- )
- tm = clean_array(
- show_dask_progress(
- target_data.mean(axis=0),
- "CORAL: Computing target feature means",
- nthreads,
- )
- )
- td = clean_array(
- show_dask_progress(
- target_data.std(axis=0),
- "CORAL: Computing target feature stdev",
- nthreads,
- ),
- 1,
- )
- data = _correlation_alignment(
- (source_data - sm) / sd, (target_data - tm) / td, nthreads
- )
- dask_to_zarr(
- data,
- assay.z["/"],
- f"{assay.z.name}/normed__{cell_key}__{feat_key}/data_coral",
- 1000,
- nthreads,
- msg="Writing out coral corrected data",
- )
-
-
-def _order_features(
- s_assay,
- t_assay,
- s_feat_ids: np.ndarray,
- filter_null: bool,
- exclude_missing: bool,
- nthreads: int,
- target_cell_key: str = "I",
-) -> Tuple[np.ndarray, np.ndarray]:
- s_ids = pd.Series(s_assay.feats.fetch_all("ids"))
- t_ids = pd.Series(t_assay.feats.fetch_all("ids"))
- t_idx = t_ids.isin(s_feat_ids)
- if t_idx.sum() == 0:
- raise ValueError(
- "ERROR: None of the features from reference were found in the target data"
- )
- if filter_null:
- if exclude_missing is False:
- logger.warning(
- "`filter_null` has not effect because `exclude_missing` is False"
- )
- else:
- t_idx[t_idx] = (
- controlled_compute(
- t_assay.rawData[:, list(t_idx[t_idx].index)][
- t_assay.cells.active_index(target_cell_key), :
- ].sum(axis=0),
- nthreads,
- )
- != 0
- )
- t_idx = t_idx[t_idx].index
- if exclude_missing:
- s_idx = s_ids.isin(t_ids.values[t_idx])
- else:
- s_idx = s_ids.isin(s_feat_ids)
- s_idx = s_idx[s_idx].index
- t_idx_map = {v: k for k, v in t_ids.to_dict().items()}
- t_re_idx = np.array(
- [t_idx_map[x] if x in t_idx_map else -1 for x in s_ids.values[s_idx]]
- )
- if len(s_idx) != len(t_re_idx):
- raise AssertionError(
- "ERROR: Feature ordering failed. Please report this issue. "
- f"This is an unexpected scenario. Source has {len(s_idx)} features while target has "
- f"{len(t_re_idx)} features"
- )
- return s_idx.values, t_re_idx
-
-
-def align_features(
- source_assay: Assay,
- target_assay: Assay,
- source_cell_key: str,
- source_feat_key: str,
- target_feat_key: str,
- target_cell_key: str,
- filter_null: bool,
- exclude_missing: bool,
- nthreads: int,
-) -> np.ndarray:
- """Aligns target features to source features.
-
- Args:
- source_assay ():
- target_assay ():
- source_cell_key ():
- source_feat_key ():
- target_feat_key ():
- filter_null ():
- exclude_missing ():
- nthreads ():
-
- Returns:
- """
- from .writers import create_zarr_dataset
-
- source_feat_ids = source_assay.feats.fetch(
- "ids", key=source_cell_key + "__" + source_feat_key
- )
- s_idx, t_idx = _order_features(
- source_assay,
- target_assay,
- source_feat_ids,
- filter_null,
- exclude_missing,
- nthreads,
- target_cell_key,
- )
- logger.info(f"{(t_idx == -1).sum()} features missing in target data")
- normed_loc = f"normed__{source_cell_key}__{source_feat_key}"
- norm_params = source_assay.z[normed_loc].attrs["subset_params"]
- sorted_t_idx = np.array(sorted(t_idx[t_idx != -1]))
-
- normed_data = target_assay.normed(
- target_assay.cells.active_index(target_cell_key), sorted_t_idx, **norm_params
- )
- loc = f"{target_assay.z.name}/normed__{target_cell_key}__{target_feat_key}/data"
-
- og = create_zarr_dataset(
- target_assay.z["/"], loc, (1000,), "float64", (normed_data.shape[0], len(t_idx))
- )
- pos_start, pos_end = 0, 0
- unsorter_idx = np.argsort(np.argsort(t_idx[t_idx != -1]))
- for i in tqdmbar(
- normed_data.blocks,
- total=normed_data.numblocks[0],
- desc=f"({target_assay.name}) Writing aligned data to {loc.split('/')[1]}",
- ):
- pos_end += i.shape[0]
- a = np.ones((i.shape[0], len(t_idx)))
- a[:, np.where(t_idx != -1)[0]] = controlled_compute(i, nthreads)[
- :, unsorter_idx
- ]
- og[pos_start:pos_end, :] = a
- pos_start = pos_end
- return s_idx
diff --git a/scarf/markers.py b/scarf/markers.py
deleted file mode 100644
index e9c5573a..00000000
--- a/scarf/markers.py
+++ /dev/null
@@ -1,439 +0,0 @@
-"""Module to find biomarkers."""
-
-from typing import Optional
-
-import numpy as np
-import pandas as pd
-from numba import jit
-from scipy.stats import linregress, norm
-from scipy.stats import rankdata
-
-from scarf.assay import Assay
-from scarf.utils import logger, tqdmbar
-
-
-def read_prenormed_batches(store, cell_idx: np.ndarray, batch_size: int, desc: str):
- batch = {}
- for i in tqdmbar(store.keys(), desc=desc):
- batch[int(i)] = store[i][:][cell_idx]
- if len(batch) == batch_size:
- yield pd.DataFrame(batch)
- batch = {}
- if len(batch) > 0:
- yield pd.DataFrame(batch)
-
-
-def mannwhitneyu_from_ranks(ranked_df, groups, group_set):
- """
- Vectorized Mann-Whitney U test using pre-computed ranks with tie correction.
-
- This function calculates two-sided p-values by reusing rank data that has
- already been computed, avoiding redundant ranking operations. Includes tie
- correction which is critical for zero-inflated data (e.g., scRNA-seq) where
- many values are identical.
-
- Args:
- ranked_df: DataFrame with ranks (same shape as original data)
- groups: Array of group labels for each sample
- group_set: Sorted array of unique group labels
-
- Returns:
- DataFrame of two-sided p-values (groups x features)
- """
- n_total = len(groups)
-
- # Calculate rank sums for each group and feature (vectorized)
- rank_sums = ranked_df.groupby(groups).sum().reindex(group_set)
-
- # Group sizes
- group_counts = pd.Series(groups).value_counts().reindex(group_set).values
-
- # Calculate tie correction factor for each feature (column)
- # This is critical for zero-inflated data where many cells have the same value
- # T = Σ(t³ - t) / (n * (n - 1)) where t is the size of each tied group
- tie_corrections = np.zeros(ranked_df.shape[1])
-
- for col_idx in range(ranked_df.shape[1]):
- ranks = ranked_df.iloc[:, col_idx].values
- # Count occurrences of each unique rank
- unique_ranks, counts = np.unique(ranks, return_counts=True)
- # Only tied ranks (appearing more than once) contribute to correction
- tied_counts = counts[counts > 1]
- if len(tied_counts) > 0:
- # Σ(t³ - t) for all tied groups
- tie_sum = np.sum(tied_counts ** 3 - tied_counts)
- # Normalize by n*(n-1) to get the correction term
- tie_corrections[col_idx] = tie_sum / (n_total * (n_total - 1))
-
- pvals = {}
-
- for idx, cluster in enumerate(group_set):
- n1 = group_counts[idx] # Size of current cluster
- n2 = n_total - n1 # Size of rest
-
- # Get rank sum for this cluster (vectorized across all features)
- R1 = rank_sums.iloc[idx].values
-
- # Calculate U statistic from rank sum
- # U = R1 - n1*(n1+1)/2
- U1 = R1 - (n1 * (n1 + 1)) / 2
-
- # Mean of U under null hypothesis
- mu_U = (n1 * n2) / 2
-
- # Standard deviation of U with tie correction
- # Without ties: σ = sqrt(n1*n2*(n+1)/12)
- # With ties: σ = sqrt((n1*n2/12) * ((n+1) - T))
- # where T = Σ(t³-t)/(n*(n-1))
- sigma_U = np.sqrt((n1 * n2 / 12) * ((n_total + 1) - tie_corrections))
-
- # Continuity correction for normal approximation
- # Reduces bias when approximating discrete distribution with continuous
- z = (U1 - mu_U - 0.5) / sigma_U
-
- # Two-sided p-value: test if cluster is different (either direction)
- # P(|U - μ| > |observed - μ|) = 2 * P(Z > |z|)
- pvals[cluster] = 2 * norm.sf(np.abs(z))
-
- return pd.DataFrame(pvals, index=ranked_df.columns).T
-
-
-def find_markers_by_rank(
- assay: Assay,
- group_key: str,
- cell_key: str,
- feat_key: str,
- batch_size: int,
- use_prenormed: bool,
- prenormed_store: Optional[str],
- n_threads: int,
- **norm_params,
-) -> dict:
- """Identify marker genes/features for given groups using a rank-based approach.
-
- Uses a two-sided Mann-Whitney U test with tie correction to identify genes
- that are differentially expressed (either up or down) in each group compared
- to all other groups.
-
- Args:
- assay: An Assay object containing the data to analyze (accessed via iter_normed_feature_wise)
- group_key: Column name in cell metadata containing group labels
- cell_key: Column name in cell metadata indicating which cells to use
- feat_key: Column name in feature metadata indicating which features to analyze
- batch_size: Number of features to process at once for memory efficiency
- use_prenormed: Whether to use pre-normalized data if available
- prenormed_store: Name of the store containing pre-normalized data
- n_threads: Number of threads to use for parallel processing
- **norm_params: Additional parameters to pass to normalization functions
-
- Returns:
- dict: Dictionary containing marker analysis results for each group, with statistics
- like fold changes, two-sided p-values, and effect sizes
- """
-
- from joblib import Parallel, delayed
-
- def calc(vdf):
- # Rank data once for all subsequent calculations
- ranked_vdf = vdf.rank(method="dense")
- ranked_vdf_average = vdf.rank(method="average")
-
- # Calculate normalized mean ranks
- r = ranked_vdf.groupby(groups).mean().reindex(group_set)
- r = r / r.sum()
-
- g = np.array([pd.Series(groups).value_counts().reindex(group_set).values]).T
- g_o = len(groups) - g
-
- s = vdf.groupby(groups).sum().reindex(group_set)
- m = s / g
- m_o = (s.sum() - s) / g_o
-
- s = (vdf > 0).groupby(groups).sum().reindex(group_set)
- e = s / g
- e_o = (s.sum() - s) / g_o
-
- fc = (m / m_o).fillna(0)
-
- # Vectorized Mann-Whitney U test using pre-computed ranks
- pvals = mannwhitneyu_from_ranks(ranked_vdf_average, groups, group_set)
-
- return np.array(
- [
- r.values,
- m.values,
- m_o.values,
- e.values,
- e_o.values,
- fc.values,
- pvals.values,
- ]
- ).T
-
- @jit(nopython=True)
- def calc_rank_mean(v):
- """Calculates the mean rank of the data."""
- r = np.ones(n_groups)
- for x in range(n_groups):
- r[x] = v[int_indices == x].mean()
- return r / r.sum()
-
- @jit(nopython=True)
- def calc_frac_fc(v):
- """Calculates the mean rank of the data."""
- m = np.zeros(n_groups)
- m_o = np.zeros(n_groups)
- e = np.zeros(n_groups)
- e_o = np.zeros(n_groups)
- fc = np.zeros(n_groups)
- for x in range(n_groups):
- i = int_indices == x
- m[x] = v[i].mean()
- m_o[x] = v[~i].mean()
- e[x] = v[i].nonzero()[0].shape[0] / i.sum()
- e_o[x] = v[~i].nonzero()[0].shape[0] / (i.shape[0] - i.sum())
- if m_o[x] == 0:
- fc[x] = 100.100
- else:
- fc[x] = m[x] / (m_o[x])
- return m, m_o, e, e_o, fc
-
- def prenormed_mean_rank_wrapper(gene_idx):
- d = prenormed_store[gene_idx][:][cell_idx]
- r = calc_rank_mean(rankdata(d, method="dense"))
- m, m_o, e, e_o, fc = calc_frac_fc(d)
- # Calculate p-values for this single feature
- ranked_d = rankdata(d, method="average").reshape(-1, 1)
- ranked_df = pd.DataFrame(ranked_d)
- pvals_df = mannwhitneyu_from_ranks(ranked_df, groups, group_set)
- p = pvals_df.iloc[:, 0].values # Extract p-values for all groups
- return gene_idx, np.vstack([r, m, m_o, e, e_o, fc, p])
-
- groups = assay.cells.fetch(group_key, cell_key)
- group_set = np.array(sorted(set(groups)))
- n_groups = len(group_set)
- idx_map = dict(zip(group_set, range(n_groups)))
- int_indices = np.array([idx_map[x] for x in groups])
- out_cols = [
- "feature_index",
- "score",
- "mean",
- "mean_rest",
- "frac_exp",
- "frac_exp_rest",
- "fold_change",
- "p_value",
- ]
- if use_prenormed:
- if prenormed_store is None:
- if "prenormed" in assay.z:
- prenormed_store = assay.z["prenormed"]
- else:
- raise ValueError(
- "Could not find prenormed values. Run with use_prenormed=False or create pre-normed values."
- )
-
- results = {x: [] for x in group_set}
- cell_idx = assay.cells.active_index(cell_key)
- batch_iterator = tqdmbar(prenormed_store.keys(), desc="Finding markers")
- temp = Parallel(n_jobs=n_threads)(
- delayed(prenormed_mean_rank_wrapper)(i) for i in batch_iterator
- )
- for i in temp:
- for j, k in zip(group_set, i[1].T):
- results[j].append([i[0]] + list(k))
- for i in results:
- results[i] = (
- pd.DataFrame(results[i], columns=out_cols)
- .sort_values(by="score", ascending=False)
- .round(5)
- )
- return results
- else:
- batch_iterator = assay.iter_normed_feature_wise(
- cell_key=cell_key,
- feat_key=feat_key,
- batch_size=batch_size,
- msg="Finding markers",
- **norm_params,
- )
- temp = np.vstack([calc(x) for x in batch_iterator])
- results = {}
- feat_index = assay.feats.active_index(feat_key)
- pval_col = "p_value"
- for n, i in enumerate(group_set):
- df = pd.DataFrame(
- temp[:, n, :], columns=out_cols[1:], index=feat_index
- ).sort_values(by="score", ascending=False)
-
- cols_to_round = [col for col in df.columns if col != pval_col]
- df.loc[:, cols_to_round] = df.loc[:, cols_to_round].round(5)
- results[i] = df
-
- results[i]["feature_index"] = results[i].index
- results[i] = results[i][out_cols]
- return results
-
-
-def find_markers_by_regression(
- assay: Assay,
- cell_key: str,
- feat_key: str,
- regressor: np.ndarray,
- min_cells: int,
- batch_size: int = 50,
- **norm_params,
-) -> pd.DataFrame:
- """Find features that correlate with a continuous variable using linear regression.
-
- Args:
- assay: An Assay object containing the data to analyze
- cell_key: Column name in cell metadata indicating which cells to use
- feat_key: Column name in feature metadata indicating which features to analyze
- regressor: 1D numpy array containing the continuous variable to correlate against
- min_cells: Minimum number of cells where feature must be expressed to be analyzed
- batch_size: Number of features to process at once for memory efficiency
- **norm_params: Additional parameters to pass to normalization functions
-
- Returns:
- pd.DataFrame: DataFrame containing correlation results with columns:
- - r_value: Pearson correlation coefficient
- - p_value: Statistical significance of correlation
- """
-
- res = {}
- for df in assay.iter_normed_feature_wise(
- cell_key=cell_key,
- feat_key=feat_key,
- batch_size=batch_size,
- msg="Finding correlated features",
- **norm_params,
- ):
- for i in df:
- v = df[i].values
- if (v > 0).sum() > min_cells:
- lin_obj = linregress(regressor, v)
- res[i] = (lin_obj.rvalue, lin_obj.pvalue)
- else:
- res[i] = (0, 1)
- res = pd.DataFrame(res, index=["r_value", "p_value"]).T
- return res
-
-
-def knn_clustering(
- d_array, n_neighbours: int, n_clusters: int, n_threads: int, ann_params: dict = None
-) -> np.ndarray:
- """
-
- Args:
- d_array: 2D numpy array of data to cluster (n_samples x n_features)
- n_neighbours: Number of nearest neighbors to use for building the graph
- n_clusters: Number of clusters to generate
- n_threads: Number of threads to use for parallel processing
- ann_params: Dictionary of parameters for approximate nearest neighbor search.
- See default_ann_params in function for available options.
-
- Returns:
- np.ndarray: 1D array of cluster assignments (integers from 1 to n_clusters)
- """
-
- from .ann import instantiate_knn_index, fix_knn_query
- from .utils import controlled_compute, tqdmbar, show_dask_progress
- from scipy.sparse import csr_matrix
-
- def make_knn_mat(data, k, t):
- """Create a sparse KNN adjacency matrix from the input data.
-
- Args:
- data: Input data array to build KNN graph from
- k: Number of nearest neighbors to find for each point
- t: Number of threads to use for parallel processing
-
- Returns:
- scipy.sparse.csr_matrix: Sparse adjacency matrix representing the KNN graph
- """
-
- for i in tqdmbar(data.blocks, desc="Fitting KNNs", total=data.numblocks[0]):
- i = controlled_compute(i, t)
- ann_idx.add_items(i)
- s, e = 0, 0
- indices = []
- for i in tqdmbar(
- data.blocks, desc="Identifying feature KNNs", total=data.numblocks[0]
- ):
- e += i.shape[0]
- i = controlled_compute(i, t)
- inds, d = ann_idx.knn_query(i, k=k + 1)
- inds, _, _ = fix_knn_query(inds, d, np.arange(s, e))
- indices.append(inds)
- s = e
- indices = np.vstack(indices)
- assert indices.shape[0] == data.shape[0]
-
- return csr_matrix(
- (
- np.ones(indices.shape[0] * indices.shape[1]),
- (
- np.repeat(range(indices.shape[0]), indices.shape[1]),
- indices.flatten(),
- ),
- ),
- shape=(indices.shape[0], indices.shape[0]),
- )
-
- def make_clusters(mat, nc):
- """Generate clusters from a KNN adjacency matrix using hierarchical clustering.
-
- Args:
- mat: Sparse adjacency matrix representing the KNN graph
- nc: Number of clusters to generate
-
- Returns:
- np.ndarray: Cluster assignments for each point
- """
- import sknetwork as skn
-
- paris = skn.hierarchy.Paris(reorder=False)
- logger.info("Performing clustering, this might take a while...")
- dendrogram = paris.fit_transform(mat)
- return skn.hierarchy.cut_straight(dendrogram, n_clusters=nc)
-
- def fix_cluster_order(data, clusters, t):
- """Reorder cluster labels based on feature expression patterns.
-
- Args:
- data: Original data array used for clustering
- clusters: Initial cluster assignments
- t: Number of threads to use for parallel processing
-
- Returns:
- np.ndarray: Reordered cluster assignments (1-based indexing)
- """
-
- idxmax = show_dask_progress(data.argmax(axis=1), "Sorting clusters", t)
- cmm = pd.DataFrame([idxmax, clusters]).T.groupby(1).median()[0].sort_values()
- return (
- pd.Series(clusters)
- .replace(dict(zip(cmm.index, range(1, 1 + len(cmm)))))
- .values
- )
-
- default_ann_params = {
- "space": "l2",
- "dim": d_array.shape[1],
- "max_elements": d_array.shape[0],
- "ef_construction": 80,
- "M": 50,
- "random_seed": 444,
- "ef": 80,
- "num_threads": 1,
- }
- if ann_params is None:
- ann_params = {}
- default_ann_params.update(ann_params)
- ann_idx = instantiate_knn_index(**default_ann_params)
- return fix_cluster_order(
- d_array,
- make_clusters(make_knn_mat(d_array, n_neighbours, n_threads), n_clusters),
- n_threads,
- )
diff --git a/scarf/matrix/__init__.py b/scarf/matrix/__init__.py
new file mode 100644
index 00000000..cd60852a
--- /dev/null
+++ b/scarf/matrix/__init__.py
@@ -0,0 +1,24 @@
+"""Blockwise matrix abstractions."""
+
+from . import blocks as _blocks
+from . import _reductions as _reduction_module
+from .blocks import Block
+from .chunked import ChunkedArray
+
+__all__ = ["Block", "ChunkedArray"]
+
+setattr(_blocks, "ChunkedArray", ChunkedArray)
+setattr(_reduction_module, "ChunkedArray", ChunkedArray)
+
+for _public_class in (Block, ChunkedArray):
+ _public_class.__module__ = __name__
+ for _member in _public_class.__dict__.values():
+ if isinstance(_member, (classmethod, staticmethod)):
+ _member = _member.__func__
+ if isinstance(_member, property):
+ _member = _member.fget
+ if callable(_member) and hasattr(_member, "__module__"):
+ _member.__module__ = __name__
+
+del _member
+del _public_class
diff --git a/scarf/matrix/_indexing.py b/scarf/matrix/_indexing.py
new file mode 100644
index 00000000..a6d5ffdf
--- /dev/null
+++ b/scarf/matrix/_indexing.py
@@ -0,0 +1,26 @@
+import numpy as np
+
+
+def is_contiguous(indices: np.ndarray) -> bool:
+ """Return whether indices form a strictly increasing consecutive run."""
+ if indices.size == 0:
+ return True
+ return bool(
+ indices[0] >= 0
+ and np.array_equal(
+ indices,
+ np.arange(indices[0], indices[0] + indices.size),
+ )
+ )
+
+
+def local_positions(key: object, length: int) -> np.ndarray | None:
+ """Resolve an index key into integer positions."""
+ if isinstance(key, slice):
+ if key == slice(None):
+ return None
+ return np.asarray(np.arange(length)[key])
+ key_array = np.asarray(key)
+ if key_array.dtype == bool:
+ return np.asarray(np.arange(length)[key_array])
+ return np.asarray(key_array.astype(int))
diff --git a/scarf/matrix/_operations.py b/scarf/matrix/_operations.py
new file mode 100644
index 00000000..14634adb
--- /dev/null
+++ b/scarf/matrix/_operations.py
@@ -0,0 +1,140 @@
+from collections.abc import Callable
+from typing import Any, Literal, cast
+
+import numpy as np
+from numpy.typing import NDArray
+
+
+type OpKind = Literal["unary", "binary", "matmul"]
+type BType = Literal["scalar", "col", "row", "full"]
+type UfuncSide = Literal["left", "right"]
+
+
+class _Op:
+ """One deferred element-wise or matrix multiplication operation."""
+
+ __slots__ = ("kind", "func", "operand", "side", "btype")
+
+ def __init__(
+ self,
+ kind: OpKind,
+ func: Callable[..., NDArray[Any]] | None = None,
+ operand: object | None = None,
+ side: UfuncSide = "left",
+ btype: BType | None = None,
+ ) -> None:
+ self.kind = kind
+ self.func = func
+ self.operand = operand
+ self.side = side
+ self.btype = btype
+
+ def apply(self, a: NDArray[Any], start: int, end: int) -> NDArray[Any]:
+ if self.kind == "unary":
+ assert self.func is not None
+ return np.asarray(self.func(a))
+ if self.kind == "matmul":
+ return np.asarray(a @ self.operand)
+ assert self.func is not None
+ operand = self.operand
+ if self.btype == "col":
+ operand = np.asarray(operand)[start:end]
+ if np.asarray(operand).ndim == 1:
+ operand = np.asarray(operand).reshape(-1, 1)
+ elif self.btype == "full":
+ operand = np.asarray(operand)[start:end]
+ return (
+ np.asarray(self.func(a, operand))
+ if self.side == "left"
+ else np.asarray(self.func(operand, a))
+ )
+
+ def subset_cols(self, col_idx: np.ndarray) -> "_Op":
+ if self.kind == "matmul":
+ raise NotImplementedError("Column-indexing after .dot is not supported")
+ if self.kind == "binary" and self.btype == "row":
+ operand = np.asarray(self.operand)
+ return _Op(
+ "binary",
+ self.func,
+ operand[col_idx] if operand.ndim == 1 else operand[:, col_idx],
+ self.side,
+ "row",
+ )
+ if self.kind == "binary" and self.btype == "full":
+ return _Op(
+ "binary",
+ self.func,
+ np.asarray(self.operand)[:, col_idx],
+ self.side,
+ "full",
+ )
+ return self
+
+ def subset_rows(self, row_idx: np.ndarray) -> "_Op":
+ if self.kind == "binary" and self.btype == "col":
+ return _Op(
+ "binary",
+ self.func,
+ np.asarray(self.operand)[row_idx],
+ self.side,
+ "col",
+ )
+ if self.kind == "binary" and self.btype == "full":
+ return _Op(
+ "binary",
+ self.func,
+ np.asarray(self.operand)[row_idx],
+ self.side,
+ "full",
+ )
+ return self
+
+
+def _unary_op(func: Callable[..., NDArray[Any]]) -> _Op:
+ return _Op("unary", func=func)
+
+
+def _matmul_op(operand: np.ndarray) -> _Op:
+ return _Op("matmul", operand=operand)
+
+
+def _classify_operand(
+ other: object,
+ n_rows: int,
+ n_cols: int,
+) -> tuple[BType, object]:
+ if np.isscalar(other):
+ return "scalar", other
+ array = np.asarray(other)
+ if array.ndim == 0:
+ return "scalar", array
+ shape = array.shape
+ if (array.ndim == 1 and shape[0] == n_rows) or (
+ array.ndim == 2 and shape == (n_rows, 1)
+ ):
+ return "col", array
+ if (array.ndim == 1 and shape[0] == n_cols) or (
+ array.ndim == 2 and shape == (1, n_cols)
+ ):
+ return "row", array
+ if array.ndim == 2 and shape[0] == n_rows:
+ return "full", array
+ if array.size == 1:
+ return "scalar", array
+ return "row", array
+
+
+def _binary_op(
+ func: Callable[..., NDArray[Any]],
+ other: object,
+ side: str,
+ kind: BType,
+) -> _Op:
+ return _Op(
+ "binary",
+ func=func,
+ operand=other,
+ side=cast(UfuncSide, side),
+ btype=kind,
+ )
diff --git a/scarf/matrix/_reductions.py b/scarf/matrix/_reductions.py
new file mode 100644
index 00000000..24462334
--- /dev/null
+++ b/scarf/matrix/_reductions.py
@@ -0,0 +1,127 @@
+from collections.abc import Callable, Iterator
+from typing import TYPE_CHECKING, Any, Literal, cast
+
+import numpy as np
+from numpy.typing import NDArray
+
+if TYPE_CHECKING:
+ from .chunked import ChunkedArray
+
+
+type ReductionOp = Literal["sum", "mean", "var", "std", "count_nonzero", "argmax"]
+type UfuncSide = Literal["left", "right"]
+
+
+class _Reduction:
+ """A deferred and cached reduction over a ChunkedArray."""
+
+ __slots__ = ("_parent", "_op", "_axis", "_cached")
+
+ def __init__(
+ self,
+ parent: "ChunkedArray",
+ op: ReductionOp,
+ axis: int | None,
+ ) -> None:
+ self._parent = parent
+ self._op = op
+ self._axis = axis
+ self._cached: np.ndarray | None = None
+
+ def compute(
+ self,
+ nthreads: int | None = None,
+ msg: str | None = None,
+ ) -> np.ndarray:
+ if self._cached is None:
+ self._cached = self._parent._reduce(
+ self._op,
+ self._axis,
+ nthreads,
+ msg,
+ )
+ return self._cached
+
+ @property
+ def _arr(self) -> np.ndarray:
+ return self.compute()
+
+ def __array__(self, dtype: np.dtype[Any] | None = None) -> np.ndarray:
+ array = self._arr
+ return array.astype(dtype) if dtype is not None else array
+
+ def __array_ufunc__(
+ self,
+ ufunc: Any,
+ method: str,
+ *inputs: Any,
+ **kwargs: Any,
+ ) -> Any:
+ if method != "__call__":
+ return NotImplemented
+ resolved = tuple(
+ item._arr if isinstance(item, _Reduction) else item for item in inputs
+ )
+ return ufunc(*resolved, **kwargs)
+
+ def __getattr__(self, item: str) -> Any:
+ return getattr(self._arr, item)
+
+ def __len__(self) -> int:
+ return len(self._arr)
+
+ def __iter__(self) -> Iterator[Any]:
+ return iter(self._arr)
+
+ def __getitem__(self, key: object) -> Any:
+ return self._arr[cast(Any, key)]
+
+ def _bin(
+ self,
+ other: object,
+ func: Callable[[NDArray[Any], NDArray[Any]], NDArray[Any]],
+ side: UfuncSide,
+ ) -> NDArray[Any]:
+ other_array = other._arr if isinstance(other, _Reduction) else np.asarray(other)
+ if side == "left":
+ return func(self._arr, other_array)
+ return func(other_array, self._arr)
+
+ def __mul__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.multiply, "left")
+
+ def __rmul__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.multiply, "right")
+
+ def __truediv__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.true_divide, "left")
+
+ def __rtruediv__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.true_divide, "right")
+
+ def __add__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.add, "left")
+
+ def __radd__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.add, "right")
+
+ def __sub__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.subtract, "left")
+
+ def __rsub__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.subtract, "right")
+
+ def __gt__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.greater, "left")
+
+ def __lt__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.less, "left")
+
+ def __ge__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.greater_equal, "left")
+
+ def __le__(self, other: object) -> NDArray[Any]:
+ return self._bin(other, np.less_equal, "left")
+
+ def __repr__(self) -> str:
+ return f""
diff --git a/scarf/matrix/blocks.py b/scarf/matrix/blocks.py
new file mode 100644
index 00000000..441755af
--- /dev/null
+++ b/scarf/matrix/blocks.py
@@ -0,0 +1,61 @@
+from typing import TYPE_CHECKING, Any
+
+import numpy as np
+
+if TYPE_CHECKING:
+ from .chunked import ChunkedArray
+
+
+class Block:
+ """A single row-chunk view of a ChunkedArray."""
+
+ __slots__ = ("_parent", "_start", "_end", "_row_perm")
+
+ def __init__(
+ self,
+ parent: "ChunkedArray",
+ start: int,
+ end: int,
+ row_perm: np.ndarray | None = None,
+ ) -> None:
+ self._parent = parent
+ self._start = start
+ self._end = end
+ self._row_perm = row_perm
+
+ @property
+ def shape(self) -> tuple[int, int]:
+ n_rows = (
+ self._end - self._start if self._row_perm is None else len(self._row_perm)
+ )
+ return (n_rows, self._parent.out_cols)
+
+ @property
+ def dtype(self) -> np.dtype[Any]:
+ return self._parent.dtype
+
+ def compute(
+ self,
+ nthreads: int | None = None,
+ msg: str | None = None,
+ ) -> np.ndarray:
+ array = self._parent._materialize_range(self._start, self._end)
+ if self._row_perm is not None:
+ array = array[self._row_perm]
+ return array
+
+ def __array__(self, dtype: np.dtype[Any] | None = None) -> np.ndarray:
+ array = self.compute()
+ return array.astype(dtype) if dtype is not None else array
+
+ def __getitem__(self, key: object) -> "Block":
+ if isinstance(key, tuple):
+ row_key = key[0]
+ else:
+ row_key = key
+ return Block(
+ self._parent,
+ self._start,
+ self._end,
+ row_perm=np.asarray(row_key),
+ )
diff --git a/scarf/matrix/chunked.py b/scarf/matrix/chunked.py
new file mode 100644
index 00000000..82a9b312
--- /dev/null
+++ b/scarf/matrix/chunked.py
@@ -0,0 +1,675 @@
+"""Lazy blockwise matrix operations over NumPy and Zarr arrays."""
+
+from collections.abc import Callable, Iterator
+from typing import Any, cast
+
+import numpy as np
+import zarr
+from numpy.typing import NDArray
+
+from ..storage.budget import (
+ DEFAULT_READ_AHEAD_BLOCKS,
+ ResourceBudget,
+ admit_stream,
+ admitted_worker_count,
+)
+from ..storage.geometry import ArrayGeometry, array_geometry
+from ._indexing import is_contiguous, local_positions
+from ._operations import (
+ _Op,
+ _binary_op,
+ _classify_operand,
+ _matmul_op,
+ _unary_op,
+)
+from ._reductions import ReductionOp, _Reduction
+from .blocks import Block
+
+__all__ = ["ChunkedArray"]
+
+type Backing = np.ndarray | zarr.Array
+type BlockFn = Callable[[int, int, int], NDArray[Any]]
+
+
+class ChunkedArray:
+ """A lazy, row-chunked 2D matrix backed by Zarr or NumPy."""
+
+ __array_priority__ = 1000.0
+
+ def __init__(
+ self,
+ backing: Backing,
+ rows: np.ndarray | None = None,
+ cols: np.ndarray | None = None,
+ ops: list[_Op] | None = None,
+ out_cols: int | None = None,
+ block_size: int | None = None,
+ nthreads: int = 1,
+ resources: ResourceBudget | None = None,
+ is_numpy: bool | None = None,
+ ) -> None:
+ self._backing = backing
+ self._rows = None if rows is None else np.asarray(rows)
+ self._cols = None if cols is None else np.asarray(cols)
+ self._ops: list[_Op] = list(ops) if ops else []
+ self._resources = resources
+ self._nthreads = (
+ max(1, min(int(nthreads), resources.workers))
+ if resources is not None
+ else max(1, int(nthreads))
+ )
+ if is_numpy is None:
+ is_numpy = isinstance(backing, np.ndarray)
+ self._is_numpy = is_numpy
+ backing_rows, backing_cols = backing.shape
+ self._n_rows = backing_rows if self._rows is None else int(self._rows.size)
+ base_cols = backing_cols if self._cols is None else int(self._cols.size)
+ self._out_cols = base_cols if out_cols is None else int(out_cols)
+ if block_size is None:
+ if self._is_numpy:
+ block_size = self._n_rows if self._n_rows > 0 else 1
+ else:
+ from ..storage.partition import row_band
+
+ block_size = row_band(
+ self._geometry(),
+ fallback=int(self._backing.shape[0]),
+ )
+ self._block_size = max(int(block_size), 1)
+
+ @classmethod
+ def from_numpy(
+ cls,
+ arr: np.ndarray,
+ block_size: int | None = None,
+ nthreads: int = 1,
+ resources: ResourceBudget | None = None,
+ ) -> "ChunkedArray":
+ arr = np.asarray(arr)
+ return cls(
+ arr,
+ block_size=block_size,
+ nthreads=nthreads,
+ resources=resources,
+ is_numpy=True,
+ )
+
+ @property
+ def shape(self) -> tuple[int, int]:
+ return (self._n_rows, self._out_cols)
+
+ @property
+ def out_cols(self) -> int:
+ return self._out_cols
+
+ @property
+ def dtype(self) -> np.dtype[Any]:
+ if not self._ops:
+ return self._backing.dtype
+ base_columns = (
+ int(self._backing.shape[1]) if self._cols is None else int(self._cols.size)
+ )
+ sample = np.empty((0, base_columns), dtype=self._backing.dtype)
+ for operation in self._ops:
+ sample = operation.apply(sample, 0, 0)
+ return sample.dtype
+
+ @property
+ def chunksize(self) -> tuple[int, int]:
+ return (
+ min(self._block_size, self._n_rows) if self._n_rows else self._block_size,
+ self._out_cols,
+ )
+
+ @property
+ def numblocks(self) -> tuple[int, int]:
+ return (self._n_block_count(), 1)
+
+ @property
+ def chunks(self) -> tuple[tuple[int, ...], tuple[int, ...]]:
+ sizes = tuple(end - start for start, end in self._ranges())
+ return (sizes if sizes else (0,), (self._out_cols,))
+
+ @property
+ def nthreads(self) -> int:
+ return self._nthreads
+
+ def __len__(self) -> int:
+ return self._n_rows
+
+ def _n_block_count(self) -> int:
+ if self._n_rows == 0:
+ return 0
+ return int(np.ceil(self._n_rows / self._block_size))
+
+ def _ranges(self) -> list[tuple[int, int]]:
+ from ..storage.partition import contiguous_ranges
+
+ return contiguous_ranges(self._n_rows, self._block_size)
+
+ def _read(self, start: int, end: int) -> np.ndarray:
+ if self._rows is None:
+ row_selection: slice | np.ndarray = slice(start, end)
+ rows_contiguous = True
+ else:
+ row_selection = self._rows[start:end]
+ rows_contiguous = is_contiguous(row_selection)
+ if self._is_numpy:
+ numpy_backing = cast(np.ndarray, self._backing)
+ if self._cols is None:
+ return np.asarray(numpy_backing[row_selection])
+ return np.asarray(numpy_backing[row_selection][:, self._cols])
+
+ zarr_backing = cast(zarr.Array, self._backing)
+ if self._cols is None:
+ if isinstance(row_selection, slice):
+ return np.asarray(zarr_backing[row_selection, :])
+ if rows_contiguous:
+ return np.asarray(
+ zarr_backing[
+ int(row_selection[0]) : int(row_selection[-1]) + 1,
+ :,
+ ]
+ )
+ return np.asarray(
+ zarr_backing.get_orthogonal_selection(
+ (row_selection, slice(None)),
+ )
+ )
+ if isinstance(row_selection, slice):
+ return np.asarray(
+ zarr_backing.get_orthogonal_selection(
+ (row_selection, self._cols),
+ )
+ )
+ if rows_contiguous:
+ return np.asarray(
+ zarr_backing.get_orthogonal_selection(
+ (
+ slice(
+ int(row_selection[0]),
+ int(row_selection[-1]) + 1,
+ ),
+ self._cols,
+ )
+ )
+ )
+ return np.asarray(
+ zarr_backing.get_orthogonal_selection(
+ (row_selection, self._cols),
+ )
+ )
+
+ def _materialize_range(self, start: int, end: int) -> np.ndarray:
+ array = self._read(start, end)
+ for operation in self._ops:
+ array = operation.apply(array, start, end)
+ return array
+
+ def _geometry(self) -> ArrayGeometry | None:
+ return array_geometry(self._backing)
+
+ def _max_decode_bytes(self) -> int:
+ geometry = self._geometry()
+ return 0 if geometry is None else geometry.nominalChunkBytes()
+
+ def _block_owned_bytes(self) -> int:
+ """Bytes one materialized row block owns, excluding its chunk decode."""
+ rows = min(self._block_size, max(1, self._n_rows))
+ elements = rows * max(1, self._out_cols)
+ input_bytes = elements * max(1, int(self._backing.dtype.itemsize))
+ output_bytes = elements * max(1, int(self.dtype.itemsize))
+ return input_bytes + (output_bytes if self._ops else 0)
+
+ def _block_task_bytes(self) -> int:
+ """Bytes one row block owns where its reader decodes one chunk at a time."""
+ return self._block_owned_bytes() + self._max_decode_bytes()
+
+ def _map_blocks(
+ self,
+ fn: BlockFn,
+ nthreads: int | None,
+ msg: str | None,
+ ) -> list[NDArray[Any]]:
+ from ..storage.parallel import map_shards
+
+ ranges = self._ranges()
+ requested = self._nthreads if nthreads is None else max(1, int(nthreads))
+ workers = requested
+ if self._resources is not None:
+ # map_shards pins Zarr to one decode per worker, so one chunk is exact.
+ workers = admitted_worker_count(
+ self._resources,
+ taskBytes=self._block_task_bytes(),
+ requested=requested,
+ )
+ results = map_shards(ranges, fn, workers=workers, msg=msg)
+ return [np.asarray(result) for result in results]
+
+ def stream_blocks(
+ self,
+ nthreads: int | None = None,
+ msg: str | None = None,
+ prefetch: int | None = None,
+ ) -> Iterator[np.ndarray]:
+ """Yield materialized row blocks with bounded read-ahead."""
+ yield from self._stream_blocks(
+ nthreads=nthreads,
+ msg=msg,
+ prefetch=prefetch,
+ row_mask=None,
+ resident_bytes=0,
+ )
+
+ def _stream_blocks(
+ self,
+ *,
+ nthreads: int | None,
+ msg: str | None,
+ prefetch: int | None,
+ row_mask: np.ndarray | None,
+ resident_bytes: int = 0,
+ ) -> Iterator[np.ndarray]:
+ from ..storage.parallel import stream_shards
+
+ threads = self._nthreads if nthreads is None else max(1, int(nthreads))
+ requested = (
+ DEFAULT_READ_AHEAD_BLOCKS if prefetch is None else max(1, int(prefetch))
+ )
+ depth = min(threads, requested)
+ ranges = self._ranges()
+ mask = None if row_mask is None else np.asarray(row_mask)
+ if mask is not None:
+ if mask.dtype != bool or mask.shape != (self._n_rows,):
+ raise ValueError(
+ "row_mask must be a boolean vector matching array rows"
+ )
+ ranges = [
+ (start, end) for start, end in ranges if bool(mask[start:end].any())
+ ]
+
+ io_concurrency: int | None = None
+ if self._resources is not None:
+ admission = admit_stream(
+ self._resources,
+ nBlocks=min(depth, max(1, len(ranges))),
+ blockBytes=self._block_owned_bytes(),
+ decodeBytes=self._max_decode_bytes(),
+ residentBytes=max(0, int(resident_bytes)),
+ )
+ depth = admission.outerWorkers
+ io_concurrency = admission.ioConcurrency
+ within = max(1, threads // depth)
+
+ def materialize(interval: tuple[int, int]) -> np.ndarray:
+ start, end = interval
+ values = self._materialize_range(start, end)
+ return values if mask is None else values[mask[start:end]]
+
+ yield from stream_shards(
+ ranges,
+ materialize,
+ workers=depth,
+ within_block_threads=within,
+ io_concurrency=io_concurrency,
+ msg=msg,
+ total=len(ranges),
+ )
+
+ def map_blocks(
+ self,
+ fn: BlockFn,
+ nthreads: int | None = None,
+ msg: str | None = None,
+ ) -> list[NDArray[Any]]:
+ """Map a function over row blocks in order."""
+ return self._map_blocks(fn, nthreads, msg)
+
+ def compute(
+ self,
+ nthreads: int | None = None,
+ msg: str | None = None,
+ ) -> np.ndarray:
+ if self._n_rows == 0:
+ return np.empty((0, self._out_cols), dtype=self.dtype)
+
+ def materialize(_: int, start: int, end: int) -> NDArray[Any]:
+ return self._materialize_range(start, end)
+
+ parts = self._map_blocks(materialize, nthreads, msg)
+ return np.vstack(parts) if len(parts) > 1 else parts[0]
+
+ def __array__(self, dtype: np.dtype[Any] | None = None) -> np.ndarray:
+ array = self.compute()
+ return array.astype(dtype) if dtype is not None else array
+
+ @property
+ def blocks(self) -> Iterator[Block]:
+ for start, end in self._ranges():
+ yield Block(self, start, end)
+
+ def _with_op(
+ self,
+ operation: _Op,
+ out_cols: int | None = None,
+ ) -> "ChunkedArray":
+ return ChunkedArray(
+ self._backing,
+ rows=self._rows,
+ cols=self._cols,
+ ops=self._ops + [operation],
+ out_cols=self._out_cols if out_cols is None else out_cols,
+ block_size=self._block_size,
+ nthreads=self._nthreads,
+ resources=self._resources,
+ is_numpy=self._is_numpy,
+ )
+
+ def _with_block_size(self, block_size: int) -> "ChunkedArray":
+ if block_size < 1:
+ raise ValueError("block_size must be greater than zero")
+ return ChunkedArray(
+ self._backing,
+ rows=self._rows,
+ cols=self._cols,
+ ops=self._ops,
+ out_cols=self._out_cols,
+ block_size=block_size,
+ nthreads=self._nthreads,
+ resources=self._resources,
+ is_numpy=self._is_numpy,
+ )
+
+ def _unary(self, func: Callable[..., NDArray[Any]]) -> "ChunkedArray":
+ return self._with_op(_unary_op(func))
+
+ def _binary(
+ self,
+ func: Callable[..., NDArray[Any]],
+ other: object,
+ side: str,
+ ) -> "ChunkedArray":
+ if isinstance(other, _Reduction):
+ other = other._arr
+ kind, operand = _classify_operand(
+ other,
+ self._n_rows,
+ self._out_cols,
+ )
+ return self._with_op(_binary_op(func, operand, side, kind))
+
+ def __array_ufunc__(
+ self,
+ ufunc: Any,
+ method: str,
+ *inputs: Any,
+ **kwargs: Any,
+ ) -> Any:
+ if method != "__call__" or kwargs.get("out") is not None:
+ return NotImplemented
+ if len(inputs) == 1:
+ return self._unary(ufunc)
+ if len(inputs) == 2:
+ left, right = inputs
+ if left is self:
+ return self._binary(ufunc, right, "left")
+ return self._binary(ufunc, left, "right")
+ return NotImplemented
+
+ def __mul__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.multiply, o, "left")
+
+ def __rmul__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.multiply, o, "right")
+
+ def __truediv__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.true_divide, o, "left")
+
+ def __rtruediv__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.true_divide, o, "right")
+
+ def __add__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.add, o, "left")
+
+ def __radd__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.add, o, "right")
+
+ def __sub__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.subtract, o, "left")
+
+ def __rsub__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.subtract, o, "right")
+
+ def __gt__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.greater, o, "left")
+
+ def __lt__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.less, o, "left")
+
+ def __ge__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.greater_equal, o, "left")
+
+ def __le__(self, o: object) -> "ChunkedArray":
+ return self._binary(np.less_equal, o, "left")
+
+ def dot(self, b: np.ndarray | NDArray[Any]) -> "ChunkedArray":
+ operand_array = np.asarray(b)
+ return self._with_op(
+ _matmul_op(operand_array),
+ out_cols=operand_array.shape[1],
+ )
+
+ @staticmethod
+ def _local_positions(key: object, length: int) -> np.ndarray | None:
+ return local_positions(key, length)
+
+ def __getitem__(self, key: object) -> "ChunkedArray":
+ if isinstance(key, tuple):
+ if len(key) != 2:
+ raise IndexError("ChunkedArray supports at most 2D indexing")
+ row_key, col_key = key
+ else:
+ row_key, col_key = key, slice(None)
+
+ rows = self._rows
+ cols = self._cols
+ operations = self._ops
+ out_cols = self._out_cols
+ n_rows = self._n_rows
+
+ col_positions = self._local_positions(col_key, out_cols)
+ if col_positions is not None:
+ operations = [
+ operation.subset_cols(col_positions) for operation in operations
+ ]
+ base_cols = cols if cols is not None else np.arange(self._backing.shape[1])
+ cols = base_cols[col_positions]
+ out_cols = int(col_positions.size)
+
+ row_positions = self._local_positions(row_key, n_rows)
+ if row_positions is not None:
+ operations = [
+ operation.subset_rows(row_positions) for operation in operations
+ ]
+ base_rows = rows if rows is not None else np.arange(self._backing.shape[0])
+ rows = base_rows[row_positions]
+ n_rows = int(row_positions.size)
+
+ block_size = n_rows if self._is_numpy and n_rows > 0 else self._block_size
+ return ChunkedArray(
+ self._backing,
+ rows=rows,
+ cols=cols,
+ ops=operations,
+ out_cols=out_cols,
+ block_size=block_size,
+ nthreads=self._nthreads,
+ resources=self._resources,
+ is_numpy=self._is_numpy,
+ )
+
+ def sum(self, axis: int | None = None) -> _Reduction:
+ return _Reduction(self, "sum", axis)
+
+ def mean(self, axis: int | None = None) -> _Reduction:
+ return _Reduction(self, "mean", axis)
+
+ def var(self, axis: int | None = None) -> _Reduction:
+ return _Reduction(self, "var", axis)
+
+ def std(self, axis: int | None = None) -> _Reduction:
+ return _Reduction(self, "std", axis)
+
+ def mean_and_std(
+ self,
+ axis: int = 0,
+ nthreads: int | None = None,
+ msg: str | None = None,
+ ) -> tuple[NDArray[Any], NDArray[Any]]:
+ """Compute column mean and standard deviation in one pass."""
+ if axis != 0:
+ raise NotImplementedError("mean_and_std only supports axis=0")
+
+ def summarize(_: int, start: int, end: int) -> NDArray[Any]:
+ array = self._materialize_range(start, end).astype(
+ np.float64,
+ copy=False,
+ )
+ return np.array(
+ [
+ array.sum(axis=0),
+ np.square(array).sum(axis=0),
+ ]
+ )
+
+ parts = self._map_blocks(summarize, nthreads, msg)
+ stacked = np.sum(parts, axis=0)
+ total, squared_total = stacked[0], stacked[1]
+ mean = total / self._n_rows
+ variance = squared_total / self._n_rows - np.square(mean)
+ return np.asarray(mean), np.asarray(np.sqrt(np.clip(variance, 0, None)))
+
+ def count_nonzero(self, axis: int | None = None) -> _Reduction:
+ return _Reduction(self, "count_nonzero", axis)
+
+ def argmax(self, axis: int | None = None) -> _Reduction:
+ return _Reduction(self, "argmax", axis)
+
+ def _reduce(
+ self,
+ op: ReductionOp,
+ axis: int | None,
+ nthreads: int | None,
+ msg: str | None,
+ ) -> np.ndarray:
+ if axis == 1 or axis is None:
+ return self._reduce_axis1(op, axis, nthreads, msg)
+ return self._reduce_axis0(op, nthreads, msg)
+
+ def _reduce_axis1(
+ self,
+ op: ReductionOp,
+ axis: int | None,
+ nthreads: int | None,
+ msg: str | None,
+ ) -> np.ndarray:
+ def reduce_block(_: int, start: int, end: int) -> NDArray[Any]:
+ array = self._materialize_range(start, end)
+ if op == "sum":
+ return np.asarray(array.sum(axis=axis))
+ if op == "mean":
+ return np.asarray(
+ array.sum() if axis is None else array.mean(axis=axis)
+ )
+ if op == "var":
+ if axis is None:
+ values = array.astype(np.float64, copy=False)
+ return np.asarray([values.sum(), np.square(values).sum()])
+ return np.asarray(array.var(axis=axis))
+ if op == "std":
+ if axis is None:
+ values = array.astype(np.float64, copy=False)
+ return np.asarray([values.sum(), np.square(values).sum()])
+ return np.asarray(array.std(axis=axis))
+ if op == "count_nonzero":
+ return np.asarray(np.count_nonzero(array, axis=axis))
+ if op == "argmax":
+ return np.asarray(array.argmax(axis=axis))
+ raise ValueError(f"Unknown reduction {op}")
+
+ parts = self._map_blocks(reduce_block, nthreads, msg)
+ if axis is None:
+ array = np.asarray(parts)
+ if op == "sum":
+ return np.asarray(array.sum())
+ if op == "mean":
+ return np.asarray(array.sum() / (self._n_rows * self._out_cols))
+ if op in ("var", "std"):
+ total, squared_total = np.sum(parts, axis=0)
+ count = self._n_rows * self._out_cols
+ mean = total / count
+ variance = squared_total / count - np.square(mean)
+ if op == "std":
+ return np.asarray(np.sqrt(max(float(variance), 0.0)))
+ return np.asarray(variance)
+ if op == "count_nonzero":
+ return np.asarray(array.sum())
+ raise ValueError(f"Reduction {op} with axis=None is not supported")
+ return np.concatenate(parts)
+
+ def _reduce_axis0(
+ self,
+ op: ReductionOp,
+ nthreads: int | None,
+ msg: str | None,
+ ) -> np.ndarray:
+ if op in ("sum", "mean"):
+
+ def sum_block(_: int, start: int, end: int) -> NDArray[Any]:
+ array = self._materialize_range(start, end)
+ return np.asarray(array.sum(axis=0))
+
+ parts = self._map_blocks(sum_block, nthreads, msg)
+ total = np.sum(parts, axis=0)
+ if op == "mean":
+ return np.asarray(total / self._n_rows)
+ return np.asarray(total)
+ if op == "count_nonzero":
+
+ def count_block(_: int, start: int, end: int) -> NDArray[Any]:
+ array = self._materialize_range(start, end)
+ return np.asarray(np.count_nonzero(array, axis=0))
+
+ parts = self._map_blocks(count_block, nthreads, msg)
+ return np.asarray(np.sum(parts, axis=0))
+ if op in ("var", "std"):
+
+ def variance_block(_: int, start: int, end: int) -> NDArray[Any]:
+ array = self._materialize_range(start, end).astype(
+ np.float64,
+ copy=False,
+ )
+ return np.array(
+ [
+ array.sum(axis=0),
+ np.square(array).sum(axis=0),
+ ]
+ )
+
+ parts = self._map_blocks(variance_block, nthreads, msg)
+ stacked = np.sum(parts, axis=0)
+ total, squared_total = stacked[0], stacked[1]
+ mean = total / self._n_rows
+ variance = squared_total / self._n_rows - np.square(mean)
+ if op == "std":
+ return np.asarray(np.sqrt(np.clip(variance, 0, None)))
+ return np.asarray(variance)
+ if op == "argmax":
+ raise NotImplementedError("argmax(axis=0) is not supported")
+ raise ValueError(f"Unknown reduction {op}")
+
+ def __repr__(self) -> str:
+ return (
+ f"ChunkedArray(shape={self.shape}, dtype={self.dtype}, "
+ f"chunksize={self.chunksize}, numblocks={self.numblocks[0]})"
+ )
diff --git a/scarf/meld_assay.py b/scarf/meld_assay.py
deleted file mode 100644
index 16a45d32..00000000
--- a/scarf/meld_assay.py
+++ /dev/null
@@ -1,493 +0,0 @@
-"""
-- Classes:
- - GffReader: A class for reading and GFF3 format files and convert to BED format files
-
-- Methods:
- - create_bed_from_coord_ids:
- - binary_search:
- - get_feature_mappings:
- - create_counts_mat:
- - coordinate_melding:
-"""
-
-import gzip
-import logging
-from typing import Tuple, List, Union
-
-import numpy as np
-import pandas as pd
-from numba import jit
-from scipy.sparse import coo_matrix
-from zarr import hierarchy
-
-from .utils import controlled_compute, logger, tqdmbar
-from .writers import create_zarr_count_assay
-
-__all__ = ["GffReader", "coordinate_melding"]
-
-
-class GffReader:
- """Reader for a GFF3 format file."""
-
- def __init__(
- self,
- gff_fn: str,
- up_offset: int = 1000,
- down_offset: int = 500,
- chunk_size=100000,
- ):
- """
-
- Args:
- gff_fn: Path to GFF file to be read
- up_offset: Upstream shift from TSS. (Default value: 1000 bases)
- down_offset: Only used when flavour='body' in `to_bed`.
- Downstream shift from TSS (Default value: 500 bases)
- chunk_size: Number of lines to read at a time
- """
-
- self.gffFn = gff_fn
- self.header = self.fetch_header_lines()
- self.nHeaderLines = len(self.header)
- self.up = up_offset
- self.down = down_offset
- self.chunksize = chunk_size
-
- def fetch_header_lines(self) -> List[str]:
- """Fetch header lines (starting with '#') from GFF file.
-
- Returns: A list of all the header lines
- """
- temp = []
- if self.gffFn.endswith("gz"):
- h = gzip.open(self.gffFn, "rt")
- else:
- h = open(self.gffFn)
- for line in h:
- if line[0] != "#":
- break
- else:
- temp.append(line.rstrip())
- h.close()
- return temp
-
- def stream(self) -> pd.DataFrame:
- """Stream the GFF file in chunks as Pandas DataFrame.
-
- Returns:
- Pandas DataFrame of the GFF file using \t as separator
- """
- stream = pd.read_csv(
- self.gffFn,
- skiprows=self.nHeaderLines,
- chunksize=int(self.chunksize),
- sep="\t",
- header=None,
- )
- for df in stream:
- yield df
-
- def get_promoter(self, v: pd.Series) -> Tuple[int, int]:
- """Create strand-aware promoter coordinates using gene start and end
- coordinates.
-
- Args:
- v: A row from the GFF file in Pandas Series format
-
- Returns:
- A Tuple of start and end coordinates for the promoter
- """
- if v[6] == "+":
- return max(0, v[3] - self.up), v[3] + self.down
- elif v[6] == "-":
- return v[4] - 1 - self.down, v[4] + self.up
- else:
- raise ValueError(f"ERROR: Unknown symbol for strand: {v[6]}")
-
- def get_body(self, v: pd.Series) -> Tuple[int, int]:
- """Create strand-aware gene body + promoter coordinates using gene
- start and end coordinates.
-
- Args:
- v: A row from the GFF file in Pandas Series format
-
- Returns:
- A Tuple of start and end coordinates
- """
- if v[6] == "+":
- return max(v[3] - self.up, 0), v[4]
- elif v[6] == "-":
- return v[3], v[4] + self.up
- else:
- raise ValueError(f"ERROR: Unknown symbol for strand: {v[6]}")
-
- @staticmethod
- def get_ids_names(v: pd.Series) -> Tuple[str, str]:
- """Extracts gene_id and gene_name values from last (9th) column of GFF
- file record.
-
- Args:
- v: A Pandas Series representing a row from GFF file
-
- Returns:
- Tuple of gene ID and gene name
- """
- gid, name = None, None
- for i in v[8].split(";"):
- j, k = i.split("=")
- if j == "gene_id":
- gid = k
- elif j == "gene_name":
- name = k
- return gid, name
-
- @staticmethod
- def d_apply(d: pd.DataFrame, func) -> np.ndarray:
- """A convenience method to apply arbitrary functions over a dataframe.
-
- Args:
- d: A pandas dataframe over which function is to applied over axis 1
- func: Function to be applied
-
- Returns:
- Numpy array of values returned by func
- """
- v = d.apply(func, axis=1)
- return np.array(list(v.values))
-
- def to_bed(
- self,
- out_bed_fn: str,
- flavour: str = "body",
- ) -> None:
- """Converts the 'gene' annotations from the GFF file to a 6-column BED
- file. The columns '3' and '4' contain te gene names and gene IDs
- respectively.
-
- Args:
- out_bed_fn: Path of output BED file.
- flavour: Should be either 'promoter' or 'body' (Default value: 'body')
-
- Returns:
- None
- """
- bed = []
- if flavour not in ["body", "promoter"]:
- raise ValueError(
- "ERROR: The value of flavour must be one of either 'body' or 'promoter'"
- )
- for df in tqdmbar(self.stream()):
- df = df[df[2] == "gene"]
- if flavour == "promoter":
- coords = self.d_apply(df, self.get_promoter)
- else:
- coords = self.d_apply(df, self.get_body)
-
- anno = self.d_apply(df, self.get_ids_names)
- odf = pd.DataFrame(
- {
- 0: df[0].values,
- 1: coords[:, 0],
- 2: coords[:, 1],
- 3: anno[:, 0],
- 4: anno[:, 1],
- 5: df[6].values,
- }
- )
- bed.append(odf)
- bed = pd.concat(bed)
- bed.to_csv(out_bed_fn, sep="\t", header=False, index=False)
- logger.info(f"{bed.shape[0]} genes saved to BED file")
- return None
-
-
-def create_bed_from_coord_ids(ids: list) -> pd.DataFrame:
- """Creates a 3 column BED file from list of strings in format:
-
- Args:
- ids: List of strings in format:
-
- Returns:
- A 3 column Pandas dataframe sorted by chromosome and start position
- """
-
- out = []
- for i in ids:
- j = i.split(":")
- o = [j[0], int(j[1].split("-")[0]), int(j[1].split("-")[1])]
- out.append(o)
- return pd.DataFrame(out).sort_values(by=[0, 1])
-
-
-@jit(nopython=True)
-def binary_search(ranges: np.ndarray, queries: np.ndarray) -> np.ndarray:
- """Identify the position of intervals in `queries` in the `ranges` interval
- list using binary search algorithm.
-
- Args:
- ranges: A sorted numpy array of shape (n, 2)
- queries: A sorted numpy array of shape (m, 2)
-
- Returns:
- A numpy array of shape (m, 2). The values are indices of ranges which overlapped the query
- intervals.
- """
-
- max_len = (ranges[:, 1] - ranges[:, 0]).max()
- n = queries.shape[0]
- ret_val = np.full((n, 2), 0)
- for i in range(n):
- start = max(0, queries[i][0] - max_len)
- end = queries[i][1]
-
- # left search
- lo = 0
- hi = ranges.shape[0]
- while lo < hi:
- mid = (lo + hi) // 2
- if ranges[mid][0] < start:
- lo = mid + 1
- else:
- hi = mid
- starts_after = lo
-
- # right search
- lo = 0
- hi = ranges.shape[0]
- while lo < hi:
- mid = (lo + hi) // 2
- if end < ranges[mid][0]:
- hi = mid
- else:
- lo = mid + 1
- ends_before = lo
-
- if starts_after == ends_before:
- ret_val[i][0] = -1
- ret_val[i][1] = -1
-
- else:
- start = queries[i][0]
- m_pos_s, m_pos_e = -1, -1
- for j in range(starts_after, ends_before):
- if start < ranges[j][1] and end > ranges[j][0]:
- if m_pos_s == -1:
- m_pos_s = j
- m_pos_e = j + 1
- ret_val[i][0] = m_pos_s
- ret_val[i][1] = m_pos_e
-
- return ret_val
-
-
-def get_ranges(df: pd.DataFrame, idx: np.ndarray) -> np.ndarray:
- """Convenience function to extract column 1 and 2 of the dataframe df and
- return int type values.
-
- Args:
- df: A pandas dataframe with minimum three columns.
- idx: Boolean indexer for the dataframe
-
- Returns:
- A numpy array with start and end positions
- """
-
- return df[[1, 2]][idx].values.astype(int)
-
-
-def get_feature_mappings(
- peaks_bed_df: pd.DataFrame, features_bed_df: pd.DataFrame
-) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:
- """Identify which intervals from `features_bed_df` overlap with those from
- `peaks_bed_df`.
-
- Args:
- peaks_bed_df: DataFrame containing reference intervals. Must have atleast 3 columns:
- 'chrom', 'start', 'end'. It should be sorted by chromosome and start
- features_bed_df: DataFrame containing reference intervals. Must have at least 5 columns:
- 'chrom', 'start', 'end', 'ids', 'names'.
- It should be sorted by chromosome and start
-
- Returns:
- A tuple containing three numpy arrays, the first two are simply the 'ids' and 'names'
- columns from `features_bed_df`. These are returned to ensure that they are in same order
- as the indices in the third array. The third array has shape (features_bed_df.shape[0], 2).
- The values are indices of the overlapping intervals from peaks_bed_df. If no overlap is found
- then that row has value [-1, -1].
- """
-
- cross_indices = []
- feats_ids, feats_names = [], []
- id_counter = {}
- n_no_match = 0
- for chrom in peaks_bed_df[0].unique():
- feats_chrom_idx = np.array(features_bed_df[0] == chrom)
- if feats_chrom_idx.shape[0] == 0:
- logger.warning(f"Chromosome {chrom} not in the input feature BED")
- continue
-
- feats_names.extend(features_bed_df[4][feats_chrom_idx].values)
- # Making IDs unique
- for i in features_bed_df[3][feats_chrom_idx].values:
- if i not in id_counter:
- id_counter[i] = 0
- id_counter[i] += 1
- if id_counter[i] > 1:
- i = i + f"_{id_counter[i]}"
- feats_ids.append(i)
-
- peaks_chrom_idx = np.array(peaks_bed_df[0] == chrom)
-
- match_indices = binary_search(
- get_ranges(peaks_bed_df, peaks_chrom_idx),
- get_ranges(features_bed_df, feats_chrom_idx),
- ).astype(int)
-
- # Now this is the main trick. Since the peak_bed_df is a sorted dataframe.
- # The dataframe index might itself not be in sorted order.
- peak_idx = np.array(peaks_bed_df.index[peaks_chrom_idx])
- for i in match_indices:
- if i[0] == -1:
- assert i[1] == -1
- cross_indices.append(None)
- n_no_match += 1
- else:
- cross_indices.append(list(peak_idx[i[0] : i[1]]))
-
- if len(feats_ids) == 0:
- raise ValueError(
- "ERROR: None of the features were found in the assay. Melding failed"
- )
- feats_ids = np.array(feats_ids)
- feats_names = np.array(feats_names)
- cross_indices = np.array(cross_indices, dtype=object)
- if n_no_match == len(cross_indices):
- logging.critical(
- "None of the provided features overlap with the peak coordinates. "
- "Melding has possibly failed."
- )
- else:
- logger.info(
- f"{n_no_match}/{feats_ids.shape[0]} features did not overlap with any peak"
- )
- if len(set(feats_ids)) != feats_ids.shape[0]:
- raise ValueError(
- "ERROR: encountered an unexpected error. Somehow the feature ids are not unique "
- "despite our attempt to make them unique by appending a suffix. Please report this "
- "bug on Github"
- )
- assert feats_ids.shape[0] == feats_names.shape[0] == cross_indices.shape[0]
- return feats_ids, feats_names, cross_indices
-
-
-def create_counts_mat(
- assay,
- store: hierarchy,
- cross_map: np.ndarray,
- scalar_coeff: float,
- renormalization: bool,
-) -> None:
- """Populate the count matrix in the Zarr store.
-
- Args:
- assay: Scarf Assay object which contains the rawData attribute representing Dask array of count matrix
- store: Output Zarr Dataset
- cross_map: Mapping of indices. as obtained from get_feature_mappings function
- scalar_coeff: An arbitrary scalar multiplier. Only used when renormalization is True.
- renormalization: Whether to rescale the sum of feature values for each cell to `scalar_coeff`
-
- Returns:
- None
- """
-
- idx = np.where(cross_map)[0]
- feat_idx = np.repeat(idx, list(map(len, cross_map[idx])))
- peak_idx = np.array(
- sum(list(cross_map[idx]), [])
- ) # There is no guarantee that these are in sorted order
- assert feat_idx.shape == peak_idx.shape
-
- n_term_per_doc = assay.cells.fetch_all(assay.name + "_nFeatures")
- n_docs = n_term_per_doc.shape[0]
- n_docs_per_term = assay.feats.fetch_all("nCells")
-
- s = 0
- for a in tqdmbar(assay.rawData.blocks, total=assay.rawData.numblocks[0]):
- a = controlled_compute(a, assay.nthreads)
- tf = a / n_term_per_doc[s : s + a.shape[0]].reshape(-1, 1)
- idf = np.log2(1 + (n_docs / (n_docs_per_term + 1)))
- a = tf * idf
-
- df = pd.DataFrame(a[:, peak_idx]).T
- df["fidx"] = feat_idx
- df = df.groupby("fidx").sum().T
- if renormalization:
- df = (scalar_coeff * df) / df.sum(axis=1).values.reshape(-1, 1)
- assert df.shape[1] == idx.shape[0]
-
- coord_renamer = dict(enumerate(df.columns))
- coo = coo_matrix(df.values)
- coo.col = np.array([coord_renamer[x] for x in coo.col])
- store.set_coordinate_selection((s + coo.row, coo.col), coo.data)
- s += a.shape[0]
-
-
-def coordinate_melding(
- assay,
- workspace: Union[str, None],
- feature_bed: pd.DataFrame,
- new_assay_name: str,
- peaks_col: str = "ids",
- scalar_coeff: float = 1e5,
- renormalization: bool = True,
-) -> None:
- """This function the coordinates of the features of the given assay and
- overlaps (genomics intersection) them with a user provided set of external
- features. Based on the overlap the original feature values are transferred
- to the new features and the new feature set is saved as a new assay. If the
- new feature overlaps with multiple original features than we sum the values
- from original feature. Similarly, a single original feature may overlap
- with multiple new features and hence its value will used for multiple new
- features. The values are TF-IDF normalized before they are saved into the
- new assay.
-
- Args:
- assay: Scarf Assay object which contains the rawData attribute representing Dask array of count matrix.
- workspace:
- feature_bed: DataFrame containing reference intervals. Must have at least 5 columns representing
- 'chrom', 'start', 'end', 'ids', 'names'. But the column names should 0,1,2,3,4
- new_assay_name: Name of the new melded assay
- peaks_col: The name of the column in the feature metadata that contains the coordinate information. This
- column should have coordinates in this format . The values from this
- column will be processed into BED format dataframe using `create_bed_from_coord_ids`
- scalar_coeff: An arbitrary scalar multiplier. Only used when renormalization is True (Default value: 1e5)
- renormalization: Whether to rescale the sum of feature values for each cell to `scalar_coeff`
- (Default value: True)
-
- Returns:
- None
- """
-
- peaks_bed = create_bed_from_coord_ids(assay.feats.fetch_all(peaks_col))
- feat_ids, feat_names, mappings = get_feature_mappings(peaks_bed, feature_bed)
-
- g = create_zarr_count_assay(
- z=assay.z["/"],
- assay_name=new_assay_name,
- workspace=workspace,
- chunk_size=assay.rawData.chunksize,
- n_cells=assay.rawData.shape[0],
- feat_ids=feat_ids,
- feat_names=feat_names,
- dtype="float",
- )
-
- create_counts_mat(
- assay=assay,
- store=g,
- cross_map=mappings,
- scalar_coeff=scalar_coeff,
- renormalization=renormalization,
- )
- return None
diff --git a/scarf/merge.py b/scarf/merge.py
deleted file mode 100644
index c5d75fec..00000000
--- a/scarf/merge.py
+++ /dev/null
@@ -1,819 +0,0 @@
-"""
-Methods and classes for merging datasets
-
-"""
-
-import os
-import re
-from collections import Counter
-from typing import Dict, List, Optional, Tuple, Union
-
-import numpy as np
-import pandas as pd
-import polars as pl
-import zarr
-from dask.array import from_array
-from dask.array.core import Array as daskArrayType
-from scipy.sparse import coo_matrix
-
-from .assay import Assay
-from .datastore.datastore import DataStore
-from .metadata import MetaData
-from .utils import (
- ZARRLOC,
- controlled_compute,
- load_zarr,
- logger,
- permute_into_chunks,
- tqdmbar,
-)
-from .writers import create_zarr_count_assay, create_zarr_obj_array
-
-__all__ = [
- "DatasetMerge",
- "AssayMerge",
- "ZarrMerge",
-]
-
-
-# Creating a dummy Assay object
-class DummyAssay:
- """
- A dummy assay object to be used in the AssayMerge class when an assay is missing in a dataset.
- """
-
- def __init__(
- self,
- ds: DataStore,
- counts: daskArrayType,
- feats: MetaData,
- name: str,
- ):
- self.rawData = counts
- self.feats = feats
- self.cells = ds.cells
- self.name = name
-
-
-class AssayMerge:
- # class ZarrMerge is renamed to AssayMerge for better understanding
- """Merge multiple Zarr files into a single Zarr file.
-
- Args:
- zarr_path: Name of the new, merged Zarr file with path.
- assays: List of assay objects to be merged. For example, [ds1.RNA, ds2.RNA].
- names: Names of each of the assay objects in the `assays` parameter. They should be in the same order as in
- `assays` parameter.
- merge_assay_name: Name of assay in the merged Zarr file. For example, for scRNA-Seq it could be simply,
- 'RNA'.
- chunk_size: Tuple of cell and feature chunk size. (Default value: (1000, 1000)).
- dtype: Dtype of the raw values in the assay. Dtype is automatically inferred from the provided assays. If
- assays have different dtypes then a float type is used.
- overwrite: If True, then overwrites previously created assay in the Zarr file. (Default value: False).
- prepend_text: This text is pre-appended to each column name (Default value: 'orig').
- reset_cell_filter: If True, then the cell filtering information is removed, i.e. even the filtered out cells
- are set as True as in the 'I' column. To keep the filtering information set the value for
- this parameter to False. (Default value: True)
- seed: Seed for randomization of rows in the assays.
- feat_name_ids_same: If True, then feature names and feature ids are same in the assays. (Default value: False)
-
- Attributes:
- assays: List of assay objects to be merged. For example, [ds1.RNA, ds2.RNA].
- names: Names of each assay objects in the `assays` parameter.
- mergedCells:
- nCells: Number of cells in dataset.
- featCollection:
- mergedFeats:
- nFeats: Number of features in the dataset.
- featOrder:
- z: The merged Zarr file.
- assayGroup:
- """
-
- def __init__(
- self,
- zarr_path: ZARRLOC,
- assays: List[Assay],
- names: List[str],
- merge_assay_name: str,
- in_workspaces: Union[list[str], None] = None,
- out_workspace: Union[str, None] = None,
- chunk_size=(1000, 1000),
- dtype: Optional[str] = None,
- overwrite: bool = False,
- prepend_text: Optional[str] = "orig",
- reset_cell_filter: bool = True,
- seed: Optional[int] = 42,
- ):
- self.assays = assays
- self.names = names
- self.inWorkspaces = in_workspaces
- self.outWorkspace = out_workspace
- self.merge_assay_name = merge_assay_name
- self.chunk_size = chunk_size
- (
- self.permutations_rows,
- self.permutations_rows_offset,
- self.coordinates_permutations,
- ) = self.perform_randomization_rows(seed)
- self.mergedCells: pl.DataFrame = self._merge_cell_table(
- reset_cell_filter, prepend_text
- )
- self.nCells: int = self.mergedCells.shape[0]
- self.featCollection: List[Dict[str, str]] = self._get_feat_ids(assays)
- self.feat_name_ids_same: bool = self.check_feat_ids(self.featCollection)
-
- if self.feat_name_ids_same is True:
- self.feat_suffix: Dict[int, int] = self.get_feat_suffix()
- self.featCollection = self.update_feat_ids()
- self.featCollection_map: List[Dict[str, str]] = (
- self.update_feat_ids_for_map()
- )
- else:
- self.featCollection_map: List[Dict[str, str]] = self.featCollection.copy()
-
- self.mergedFeats: pl.DataFrame = self._merge_order_feats(self.featCollection)
- self.mergedFeats_map: pl.DataFrame = self._merge_order_feats(
- self.featCollection_map
- )
- self.nFeats: int = self.mergedFeats_map.shape[0]
- self.featOrder: List[np.ndarray] = self._ref_order_feat_idx()
-
- if self.feat_name_ids_same is True:
- self.featOrder_map: List[np.ndarray] = self._ref_order_feat_idx_map()
- else:
- self.featOrder_map: List[np.ndarray] = self.featOrder.copy()
-
- self.cellOrder: Dict[int, Dict[int, np.ndarray]] = self._ref_order_cell_idx()
- self.z: zarr.Group = self._use_existing_zarr(
- zarr_path, merge_assay_name, overwrite
- )
- self._ini_cell_data(overwrite)
- if dtype is None:
- if len(set([str(x.rawData.dtype) for x in self.assays])) == 1:
- dtype = str(self.assays[0].rawData.dtype)
- else:
- dtype = "float"
-
- self.assayGroup = create_zarr_count_assay(
- z=self.z,
- assay_name=merge_assay_name,
- workspace=self.outWorkspace,
- chunk_size=chunk_size,
- n_cells=self.nCells,
- feat_ids=np.array(self.mergedFeats_map["ids"]),
- feat_names=np.array(self.mergedFeats_map["names"]),
- dtype=dtype,
- )
-
- def perform_randomization_rows(
- self, seed: Optional[int] = 42
- ) -> Tuple[
- Dict[int, Dict[int, np.ndarray]], Dict[int, Dict[int, np.ndarray]], np.ndarray
- ]:
- """
- Perform randomization of rows in the assays.
- Args:
- seed: Seed for randomization
- Returns:
- """
- rng = np.random.default_rng(seed=seed)
- chunkSize = np.array([x.rawData.chunksize[0] for x in self.assays])
- nCells = np.array([x.rawData.shape[0] for x in self.assays])
- permutations = {
- i: permute_into_chunks(nCells[i], chunkSize[i])
- for i in range(len(self.assays))
- } # Randomize the rows in chunks
-
- # Create a dictionary of arrays. This is the same data in `permutations` but in a different format. We index the arrays by the chunk number.
- # Example:
- # permutation = {0: [array([2, 0, 1]), array([3, 4, 5]), array([8, 7, 6]), array([9])], 1: [array([2, 0, 1]), array([3, 4, 5]), array([8, 7, 6]), array([9])]}
- # permutations_rows = {0: {0: array([2, 0, 1]), 1: array([3, 4, 5]), 2: array([8, 7, 6]), 3: array([9])}, 1: {0: array([2, 0, 1]), 1: array([3, 4, 5]), 2: array([8, 7, 6]), 3: array([9])}}
- permutations_rows = {}
- for key, arrays in permutations.items():
- in_dict = {i: x for i, x in enumerate(arrays)}
- permutations_rows[key] = in_dict
-
- # Set the offset for each chunk. Offset calculated by adding the number of cells in the previous chunks. This will be helpful when we merge the cells metadata in the end.
- # Example:
- # {0: {0: array([2, 0, 1]), 1: array([3, 4, 5]), 2: array([8, 7, 6]), 3: array([9])}, 1: {0: array([12, 10, 11]), 1: array([13, 14, 15]), 2: array([18, 17, 16]), 3: array([19])}}
- permutations_rows_offset = {}
- offset = 0
- for key, val_dict in permutations_rows.items():
- in__dict: dict[int, np.ndarray] = {}
- for in_key, arrs in val_dict.items():
- in__dict[in_key] = arrs + offset
- permutations_rows_offset[key] = in__dict
- offset += nCells[key]
-
- # Set the random order in which the rows will be merged. The last chunk of each assay is appended at the end of the list to account for potential incomplete chunks.
- # Example:
- # coordinates_permutations = [[0, 0], [0, 1], [1, 2], [0, 2], [1, 1], [1, 0], [0, 3], [1, 3]]
- # Here [0, 0] means the first chunk of the first assay, [0, 1] means the second chunk of the first assay, [1, 2] means the third chunk of the second assay, and so on will be the order in which the rows will be merged.
- coordinates = []
- extra = []
- for i in range(len(self.assays)):
- for j in range(len(permutations[i])):
- if j == len(permutations[i]) - 1: # if j is last, append extra
- extra.append([i, j])
- continue
- coordinates.append([i, j])
- coordinates_permutations = rng.permutation(
- coordinates
- ) # Randomize the order of the coordinates
- if len(coordinates_permutations) > 0:
- coordinates_permutations = np.concatenate(
- [coordinates_permutations, extra], axis=0
- )
- else:
- coordinates_permutations = np.array(extra)
-
- try:
- assert permutations_rows_offset[0][0].min() == 0
- except AssertionError:
- raise AssertionError(
- "ERROR: Randomization of rows failed. The first row should be at 0.",
- "Please report this issue.",
- )
- try:
- assert (
- permutations_rows_offset[list(permutations_rows_offset.keys())[-1]][
- list(
- permutations_rows_offset[
- list(permutations_rows_offset.keys())[-1]
- ].keys()
- )[-1]
- ].max()
- == nCells.sum() - 1
- )
- except AssertionError:
- raise AssertionError(
- "ERROR: Randomization of rows failed. The last row should be at the end of the dataset.",
- "Please report this issue.",
- )
- return permutations_rows, permutations_rows_offset, coordinates_permutations
-
- def _ref_order_cell_idx(self) -> Dict[int, Dict[int, np.ndarray]]:
- """
- Calculate the order of the cells in the merged assay.
- """
- # We calculate the order of the cells in the merged assay by using the permutations_rows and coordinates_permutations. This is essentially the one-to-one mapping of the cells in the assays to the cells in the merged assay.
- # Example:
- # cellOrder = {0: {0: array([0, 1, 2]), 1: array([3, 4, 5]), 2: array([ 9, 10, 11]), 3: array([18])}, 1: {0: array([15, 16, 17]), 1: array([12, 13, 14]), 2: array([6, 7, 8]), 3: array([19])}}
- # Here we see that the cells [2, 0, 1] from the first chunk of the first assay are mapped to [0, 1, 2] in the merged assay. Similarly, the cells [2, 0, 1] from the first chunk of the second assay are mapped to [15, 16, 17] in the merged assay.
- new_cells = {}
- for i in range(len(self.assays)):
- in_dict: dict[int, np.ndarray] = {}
- for j in range(len(self.permutations_rows[i])):
- in_dict[j] = np.array([])
- new_cells[i] = in_dict
- offset = 0
- for i, (x, y) in enumerate(self.coordinates_permutations):
- arr = self.permutations_rows[x][y]
- arr = np.array(range(len(arr)))
- arr = arr + offset
- new_cells[x][y] = arr
- offset = arr.max() + 1
- return new_cells
-
- def _merge_cell_table(
- self, reset: bool, prepend_text: Optional[str] = None
- ) -> pl.DataFrame:
- """Merges the cell metadata table for each sample.
-
- Args:
- reset: whether to remove filtering information
- prepend_text: string to add as prefix for each cell column
-
- Returns:
- """
- if len(self.assays) != len(set(self.names)):
- raise ValueError(
- "ERROR: A unique name should be provided for each of the assay"
- )
- if prepend_text == "":
- prepend_text = None
- ret_val = []
- for assay, name in zip(self.assays, self.names):
- a = assay.cells.to_polars_dataframe(assay.cells.columns)
- a = a.with_columns(
- [pl.Series("ids", np.array([f"{name}__{x}" for x in a["ids"]]))]
- )
- for i in a.columns:
- if i not in ["ids", "I", "names"] and prepend_text is not None:
- a = a.with_columns(
- [pl.Series(f"{prepend_text}_{i}", assay.cells.fetch_all(i))]
- )
- a = a.drop([i])
- if reset:
- a = a.with_columns(
- [pl.Series("I", np.ones(len(a["ids"])).astype(bool))]
- )
- ret_val.append(a.to_pandas())
-
- # Here we merge the cell metadata tables for each sample. We simply concatenate the tables and reset the index.
- ret_val_df = pd.concat(ret_val, axis=0).reset_index(drop=True)
- # Now we use the offsets stored in permutations_rows_offset along with the coordinates_permutations to reorder the cells in the merged assay. The offsets are used to bring the cells in the same order as the rows in the merged assay.
- compiled_idx = [
- self.permutations_rows_offset[i][j]
- for i, j in self.coordinates_permutations
- ]
- compiled_idx = np.concatenate(compiled_idx)
- # Index the merged cell metadata table with the compiled_idx to get the final randomized merged cell metadata table.
- ret_val_df = ret_val_df.iloc[compiled_idx]
- if sum([x.cells.N for x in self.assays]) != ret_val_df.shape[0]:
- raise AssertionError(
- "Unexpected number of cells in the merged table. This is unexpected, "
- " please report this bug"
- )
- return ret_val_df
-
- @staticmethod
- def _get_feat_ids(assays) -> List[Dict[str, str]]:
- """Fetches ID->names mapping of features from each assay.
-
- Args:
- assays: List of Assay objects
-
- Returns:
- A list of dictionaries. Each dictionary is a id to name
- mapping for each feature in the corresponding assay
- """
- ret_val = []
- for i in assays:
- df = i.feats.to_polars_dataframe(["names", "ids"])
- ret_val.append(dict(zip(df["ids"].to_numpy(), df["names"].to_numpy())))
- return ret_val
-
- def check_feat_ids(self, featCollection: List[Dict[str, str]]) -> bool:
- """
- Check if feature names and feature ids are different in the assays.
- """
- isSame = False
- for i, dict_ in enumerate(featCollection):
- keys = np.array(list(dict_.keys()))
- values = np.array(list(dict_.values()))
- if np.equal(keys, values).all():
- logger.info(
- f"Encountered same feature names and ids for feature {self.assays[i].name} in dataset {self.names[i]}. The feature ids will be updated with feature names."
- )
- isSame = True
- break
- return isSame
-
- def get_feat_suffix(self) -> Dict[int, int]:
- """
- Get the suffix of the feature ids.
- """
- feat_suffix = {}
- for i, dict_ in enumerate(self.featCollection):
- keys = np.array(list(dict_.keys()))
- ends_0 = np.array([x.endswith("_0") for x in keys]).sum()
- ends_1 = np.array([x.endswith("_1") for x in keys]).sum()
- ends_2 = np.array([x.endswith("_2") for x in keys]).sum()
- if ends_0 > 0:
- feat_suffix[i] = 0
- elif ends_1 > 0:
- feat_suffix[i] = 1
- elif ends_0 > 0 and ends_1 > 0:
- feat_suffix[i] = 0
- elif ends_2 > 0:
- raise ValueError(
- "Feature Numbering starts with 2, this is erroneous. Kindly check the data"
- )
- else:
- feat_suffix[i] = -1
- return feat_suffix
-
- def update_feat_ids(self) -> List[Dict[str, str]]:
- """
- Update the feature ids in case of same feature names and ids.
-
- Returns:
- `List[Dict[str, str]]`: List of dictionaries containing the updated feature ids for the merged assay.
-
- This function updates the feature ids for the merged assay in case the feature names and ids are the same in the assays.
- This function will generate a new feature id and name for the duplicate feature names and ids.
- We will append a numeric suffix to the feature ids to make them unique. We use this later to map multiple feature ids to a single feature id.
- """
- pattern = re.compile(r"_\d+$")
- # feat_suffix = self.get_feat_suffix()
- vals = np.array(list(self.feat_suffix.values()))
- vals = vals[vals > -1]
- min_val = vals.min() if len(vals) > 0 else 0
- new_featCollection = []
- for i, dict_ in enumerate(self.featCollection):
- in_dict = {}
- counter = Counter(dict_.values())
- if self.feat_suffix[i] == -1:
- sum_counter = {x: 0 for x in np.unique(list(dict_.values()))}
- # Update all values from 'val' to 'val_{min}'
- for _, val in dict_.items():
- if counter[val] == 1: # Unique value
- in_dict[val] = val
- else: # Multiple values -- update
- updated_val = f"{val}_{min_val+sum_counter[val]}"
- in_dict[updated_val] = updated_val
- sum_counter[val] += 1
- else:
- for _, val in dict_.items():
- # check if the value ends with a number
- if pattern.search(val):
- num = int(val.split("_")[-1])
- # replace the number with min_val
- updated_val = pattern.sub(
- f"_{min_val-self.feat_suffix[i]+num}", val
- )
- in_dict[updated_val] = updated_val
- else:
- updated_val = f"{val}" # _{min_val}"
- in_dict[updated_val] = updated_val
- new_featCollection.append(in_dict)
- return new_featCollection
-
- def update_feat_ids_for_map(self) -> List[Dict[str, str]]:
- """
- Get the updated feature ids mapping for the merged assay in case of same feature names and ids.
-
- Returns:
- `List[Dict[str, str]]`: List of dictionaries containing the updated feature ids for the merged assay.
-
- This function updates the feature ids for the merged assay in case the feature names and ids are the same in the assays.
- This function will remove the numeric suffix from the feature ids and update them with the feature names.
- """
- pattern = re.compile(r"_\d+$")
- new_featCollection = []
- for dict_ in self.featCollection:
- in_dict = {}
- for x in dict_.values():
- # check if the value ends with a number
- if pattern.search(x):
- val = x.split("_")[:-1]
- val = "_".join(val)
- if val not in in_dict:
- in_dict[val] = val
- else:
- in_dict[x] = x
- new_featCollection.append(in_dict)
- return new_featCollection
-
- def _merge_order_feats(self, FeatCollection) -> pl.DataFrame:
- """Merge features from all the assays and determine their order.
-
- Returns:
- """
- union_set = {}
- for ids in FeatCollection:
- for i in ids:
- if i not in union_set:
- union_set[i] = ids[i]
- ret_val = pl.DataFrame(
- {
- "idx": range(len(union_set)),
- "names": list(union_set.values()),
- "ids": list(union_set.keys()),
- }
- )
-
- r = ret_val.shape[0] / sum([x.feats.N for x in self.assays])
- if r == 1:
- raise ValueError(
- "No overlapping features found! Will not merge the files. Please check the features ids "
- " are comparable across the assays"
- )
- if r > 0.9:
- logger.warning("The number overlapping features is very low.")
- return ret_val
-
- def _ref_order_feat_idx(self) -> List[np.ndarray]:
- ret_val = []
- for ids in self.featCollection:
- ordered_ids = pl.DataFrame({"ids": list(ids.keys())})
- # vals = self.mergedFeats.filter(pl.col("ids").is_in(list(ids.keys())))["idx"]
- vals = ordered_ids.join(self.mergedFeats, on="ids", how="left")[
- "idx"
- ].to_numpy()
- ret_val.append(np.array(vals))
- return ret_val
-
- def _ref_order_feat_idx_map(self) -> List[np.ndarray]:
- """
- Get the order of the features in the merged assay.
-
- Returns:
- `List[np.ndarray]`: List of numpy arrays containing the order of the features in the merged assay.
-
- This function returns the order of the features in the merged assay. The order is determined by the feature
- """
- featorder = []
- names_to_idx = self.mergedFeats_map.select(["names", "idx"]).to_dict(
- as_series=False
- )
- name_to_idx_dict = dict(zip(names_to_idx["names"], names_to_idx["idx"]))
- pattern = re.compile(r"_\d+$")
- for dict_ in self.featCollection:
- vals = []
- values_list = []
- for val in dict_.values():
- if pattern.search(val):
- val = "_".join(val.split("_")[:-1]) # Remove the numeric suffix.
- values_list.append(val)
- vals = [name_to_idx_dict[name] for name in values_list]
- featorder.append(np.array(vals))
- return featorder
-
- def _use_existing_zarr(
- self, zarr_loc: ZARRLOC, merge_assay_name, overwrite
- ) -> zarr.Group:
- if self.outWorkspace is None:
- cell_slot = "cellData"
- assay_slot = merge_assay_name
- else:
- cell_slot = f"{self.outWorkspace}/cellData"
- assay_slot = f"{self.outWorkspace}/merge_assay_name"
-
- try:
- z = load_zarr(zarr_loc, mode="r")
- if cell_slot not in z:
- raise ValueError(
- f"ERROR: Zarr file exists but seems corrupted. Either delete the " # noqa: F541
- "existing file or choose another path"
- )
- if assay_slot in z:
- if overwrite is False:
- raise ValueError(
- f"ERROR: Zarr file already contains {merge_assay_name} assay. Choose "
- "a different zarr path or a different assay name. Otherwise set overwrite to True"
- )
- try:
- if not all(
- z[cell_slot]["ids"][:]
- == np.array(np.array(self.mergedCells["ids"])) # type: ignore
- ):
- raise ValueError(
- f"ERROR: order of cells does not match the one in existing file" # noqa: F541
- )
- except KeyError:
- raise ValueError(
- f"ERROR: 'cell data seems corrupted. Either delete the " # noqa: F541
- "existing file or choose another path"
- )
- return load_zarr(zarr_loc, mode="r+")
- except ValueError:
- # So no zarr file with same name exists. Check if a non zarr folder with the same name exists
- if isinstance(zarr_loc, str) and os.path.exists(zarr_loc):
- raise ValueError(
- f"ERROR: Directory/file with name `{zarr_loc}`exists. "
- f"Either delete it or use another name"
- )
- # creating a new zarr file
- return load_zarr(zarr_loc, mode="w")
-
- def _ini_cell_data(self, overwrite) -> None:
- """Save cell attributes to Zarr.
-
- Returns:
- None
- """
- if self.outWorkspace is None:
- cell_slot = "cellData"
- else:
- cell_slot = f"{self.outWorkspace}/cellData"
-
- if (cell_slot in self.z and overwrite is True) or cell_slot not in self.z:
- g = self.z.create_group(cell_slot, overwrite=True)
- for i in self.mergedCells.columns:
- vals = np.array(self.mergedCells[i])
- create_zarr_obj_array(g, str(i), vals, vals.dtype, overwrite=True)
- else:
- logger.info(
- f"cellData already exists so skipping _ini_cell_data" # noqa: F541
- )
-
- def _dask_to_coo(
- self, d_arr, order: np.ndarray, order_map: np.ndarray, n_threads: int
- ) -> coo_matrix:
- """
- Convert a Dask array to a sparse COO matrix.
- Args:
- d_arr: Dask array to be converted
- order: Original feature indices
- order_map: Consolidated feature indices
- n_threads: Number of threads to use for computation
- Returns:
- Sparse COO matrix
-
- This function takes a Dask array and converts it to a sparse COO matrix.
- The `order` is the original feature indices and `order_map` is the consolidated feature indices
- i.e. the indices of the features in the merged assay. If the `order` and `order_map` are the same,
- then the function will directly convert the Dask array to a COO matrix. If they are different,
- then the function will consolidate the data from the Dask array to the COO matrix using the `order_map`.
- For multiple indices mapping to the same consolidated index, the data is summed up.
- """
- mat = np.zeros((d_arr.shape[0], self.nFeats))
- computed_data = controlled_compute(d_arr, n_threads)
- # Create a mapping from original feature indices to their consolidated indices
- consolidation_map = {orig: cons for orig, cons in zip(order, order_map)}
- # Iterate through the columns of the computed data
- for i, col_data in enumerate(computed_data.T):
- consolidated_idx = consolidation_map[order[i]]
- mat[:, consolidated_idx] += col_data
-
- return coo_matrix(mat)
-
- def dump(self, nthreads=4):
- """Copy the values from individual assays to the merged assay.
-
- Args:
- nthreads: Number of compute threads to use. (Default value: 2)
-
- Returns:
- """
- counter = 0
- for i, (assay, feat_order, feat_order_map) in enumerate(
- zip(self.assays, self.featOrder, self.featOrder_map)
- ):
- for j, block in tqdmbar(
- enumerate(assay.rawData.blocks),
- total=assay.rawData.numblocks[0],
- desc=f"Writing data from assay {i+1}/{len(self.assays)} to merged file",
- ):
- # Perform the inter-chunk permutation of the rows
- perm_order = self.permutations_rows[i][j]
- perm_order = perm_order - perm_order.min()
- block = block[perm_order, :]
- a = self._dask_to_coo(block, feat_order, feat_order_map, nthreads)
- # Here we use the one-to-one mapping of the chunks in the assays to the chunks in the merged assay to bring the data in the same order.
- row_idx = self.cellOrder[i][j]
- self.assayGroup.set_coordinate_selection(
- (a.row + row_idx.min(), a.col), a.data.astype(self.assayGroup.dtype)
- )
- counter += a.shape[0]
- try:
- assert counter == self.nCells
- except AssertionError:
- raise AssertionError(
- "ERROR: Mismatch in number of cells in the merged assay. Please report this issue."
- )
-
-
-# Alias for ZarrMerge
-class ZarrMerge(AssayMerge):
- """
- Alias for AssayMerge for backward compatibility.
- """
-
- def __init__(self, *args, **kwargs):
- logger.warning(
- "The 'ZarrMerge' class is deprecated and will be removed in a future release. Please use 'AssayMerge' instead."
- )
- super().__init__(*args, **kwargs)
-
-
-class DatasetMerge:
- """
- Merge multiple datastores, handling different assay types and generating missing assays on the fly.
-
- Args:
- datasets: List of DataStore objects to be merged.
- zarr_path: Name of the new, merged Zarr file with path.
- names: Names of each of the dataset objects in the `datasets` parameter. They should be in the same order as in
- `datasets` parameter.
- in_workspaces: List of workspaces to be merged. If None, all workspaces are merged.
- out_workspace: Name of the workspace in the merged Zarr file. If None, the name of the first workspace is used.
- chunk_size: Tuple of cell and feature chunk size. (Default value: (1000, 1000)).
- dtype: Dtype of the raw values in the assay. Dtype is automatically inferred from the provided assays. If
- assays have different dtypes then a float type is used.
- overwrite: If True, then overwrites previously created assay in the Zarr file. (Default value: False).
- prepend_text: This text is pre-appended to each column name (Default value: 'orig').
- reset_cell_filter: If True, then the cell filtering information is removed, i.e. even the filtered out cells
- are set as True as in the 'I' column. To keep the filtering information set the value for
- this parameter to False. (Default value: True)
- seed: Seed for randomization of rows in the assays.
-
- Example:
- >>> # Assuming ds1, ds2 and ds3 are DataStore objects
- >>> # ds1 has RNA and ADT assays. ds2 has RNA assay. ds3 has ADT assay.
- >>> # Merge RNA and ADT assays from all the datastores
- >>> merge = DatasetMerge(
- >>> datasets=[ds1, ds2, ds3],
- >>> zarr_path="merged.zarr",
- >>> names=["ds1", "ds2", "ds3"],
- >>> overwrite = True
- >>> )
- >>> merge.dump()
- >>> # The merged.zarr file will have RNA and ADT assays from all the datastores
- """
-
- def __init__(
- self,
- datasets: List[DataStore],
- zarr_path: ZARRLOC,
- names: List[str],
- in_workspaces: Union[list[str], None] = None,
- out_workspace: Union[str, None] = None,
- chunk_size=(1000, 1000),
- dtype: Optional[str] = None,
- overwrite: bool = False,
- prepend_text: Optional[str] = "orig",
- reset_cell_filter: bool = True,
- seed: Optional[int] = 42,
- ):
- self.datasets = datasets
- self.names = names
- self.zarr_path = zarr_path
- self.in_workspaces = in_workspaces
- self.out_workspace = out_workspace
- self.chunk_size = chunk_size
- self.dtype = dtype
- self.overwrite = overwrite
- self.prepend_text = prepend_text
- self.reset_cell_filter = reset_cell_filter
- self.seed = seed
- self.unique_assays = self.get_unique_assays()
- self.n_unique_assays = len(self.unique_assays)
- self.merge_generators = self.create_merge_generators()
-
- def get_unique_assays(self) -> List[str]:
- """
- Get unique assays from both datasets
- """
- unique_assays = set()
- for ds in self.datasets:
- unique_assays.update(ds.assay_names)
- return list(unique_assays)
-
- def create_merge_generators(self) -> List[AssayMerge]:
- """
- Create AssayMerge objects for each unique assay
- """
- gens = []
- for assay in self.unique_assays:
- assay_list = []
- for ds in self.datasets:
- if assay in ds.assay_names:
- assay_list.append(ds.get_assay(assay))
- else:
- # Generate a dummy assay on the fly
- dummy_assay: DummyAssay = self.generate_dummy_assay(ds, assay)
- assay_list.append(dummy_assay)
- gens.append(
- AssayMerge(
- zarr_path=self.zarr_path,
- assays=assay_list,
- names=self.names,
- merge_assay_name=assay,
- in_workspaces=self.in_workspaces,
- out_workspace=self.out_workspace,
- chunk_size=self.chunk_size,
- dtype=self.dtype,
- overwrite=self.overwrite,
- prepend_text=self.prepend_text,
- reset_cell_filter=self.reset_cell_filter,
- seed=self.seed,
- )
- )
- return gens
-
- def generate_dummy_assay(self, ds: DataStore, assay_name: str) -> DummyAssay:
- """
- Generate a dummy assay for a datastore that doesn't have the specified assay
- """
- # Find a datastore that has this assay to get feature information
- reference_ds = next(
- ds_ for ds_ in self.datasets if assay_name in ds_.assay_names
- )
- reference_assay = reference_ds.get_assay(assay_name)
-
- reference_chunk = [
- ds.get_assay(ref).rawData.chunksize[0] for ref in ds.assay_names
- ]
- # check if entries in reference_chunk are the same
- if not all(x == reference_chunk[0] for x in reference_chunk):
- rowChunkShape = reference_chunk[0]
- else:
- rowChunkShape = max(reference_chunk)
- colChunk = reference_assay.rawData.chunksize[1]
- chunkShape = (rowChunkShape, colChunk)
-
- # Create a dummy assay with zero counts and matching features
- dummy_shape = (ds.cells.N, reference_assay.feats.N)
- dummy_counts = zarr.zeros(
- dummy_shape, chunks=chunkShape, dtype=reference_assay.rawData.dtype
- )
- dummy_counts = from_array(dummy_counts, chunks=chunkShape)
- dummy_assay = DummyAssay(
- ds, dummy_counts, reference_assay.feats, reference_assay.name
- )
- logger.info(f"Generated dummy {assay_name} assay for datastore {ds}")
- return dummy_assay
-
- def dump(self, nthreads=4) -> None:
- """
- Dump the merged data to the zarr file
- """
- for gen in self.merge_generators:
- logger.info(f"Dumping {gen.merge_assay_name}")
- gen.dump(nthreads)
- logger.info("Merging complete")
- return None
diff --git a/scarf/merge/__init__.py b/scarf/merge/__init__.py
new file mode 100644
index 00000000..b7f3cb18
--- /dev/null
+++ b/scarf/merge/__init__.py
@@ -0,0 +1,73 @@
+"""Methods and classes for merging datasets."""
+
+import sys
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from .assays import (
+ AssayMerge as AssayMerge,
+ DummyAssay as DummyAssay,
+ MergeAssay as MergeAssay,
+ _RowPlan as _RowPlan,
+ controlled_compute as controlled_compute,
+ load_zarr as load_zarr,
+ )
+ from .datasets import DatasetMerge as DatasetMerge
+
+__all__ = [
+ "DatasetMerge",
+ "AssayMerge",
+]
+
+_LAZY_EXPORTS = {
+ "DatasetMerge": "datasets",
+ "AssayMerge": "assays",
+ "DummyAssay": "assays",
+ "MergeAssay": "assays",
+ "_RowPlan": "assays",
+ "load_zarr": "assays",
+ "controlled_compute": "assays",
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+_CLASS_EXPORTS = frozenset({"DatasetMerge", "AssayMerge", "DummyAssay"})
+
+
+def _normalize_class_metadata(value: type[Any]) -> None:
+ value.__module__ = __name__
+ for descriptor in value.__dict__.values():
+ if isinstance(descriptor, (classmethod, staticmethod)):
+ method = descriptor.__func__
+ else:
+ method = descriptor
+ if callable(method) and hasattr(method, "__module__"):
+ method.__module__ = __name__
+
+
+def _cache_loaded_exports() -> None:
+ for export_name, implementation_name in _LAZY_EXPORTS.items():
+ module = sys.modules.get(f"{__name__}.{implementation_name}")
+ if module is None or not hasattr(module, export_name):
+ continue
+ value = getattr(module, export_name)
+ if export_name in _CLASS_EXPORTS:
+ _normalize_class_metadata(value)
+ globals().setdefault(export_name, value)
+
+
+def __getattr__(name: str) -> Any:
+ module_name = _LAZY_EXPORTS.get(name)
+ if module_name is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ import_module(f".{module_name}", __name__)
+ _cache_loaded_exports()
+ return globals()[name]
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()) | set(_LAZY_EXPORTS))
diff --git a/scarf/merge/assays.py b/scarf/merge/assays.py
new file mode 100644
index 00000000..aca9f55c
--- /dev/null
+++ b/scarf/merge/assays.py
@@ -0,0 +1,978 @@
+import os
+import re
+import sys
+from collections import Counter
+from collections.abc import Iterator
+from typing import Any
+
+import numpy as np
+import pandas as pd
+import scarf
+import zarr
+from scipy.sparse import coo_matrix
+
+from ..storage.types import as_zarr_array, as_zarr_group
+from ..assay import Assay
+from ..matrix import Block, ChunkedArray
+from ..metadata import MetaData
+from ..storage.arrays import create_zarr_obj_array
+from ..storage.budget import admitted_worker_count, resolve_budget
+from ..storage.layout import array_shard_rows
+from ..storage.profiles import (
+ StorageProfile,
+ ZarrLocation,
+ is_local_zarr_path,
+ resolve_storage_profile,
+)
+from ..storage.schema import create_zarr_count_assay, validate_assay_name
+from ..storage.sharding import (
+ accumulate_sparse_to_shards,
+ sparse_producer_peak_bytes,
+)
+from ..storage.stores import load_zarr as load_zarr
+from ..utils.arrays import canonicalize_sparse, permute_into_chunks
+from ..utils.compute import controlled_compute as controlled_compute
+from ..utils.logging import logger
+from ..utils.progress import iter_progress
+
+
+# Creating a dummy Assay object
+class DummyAssay:
+ """
+ A dummy assay object to be used in the AssayMerge class when an assay is missing in a dataset.
+ """
+
+ def __init__(
+ self,
+ ds: "scarf.DataStore",
+ counts: ChunkedArray,
+ feats: MetaData,
+ name: str,
+ ):
+ self.rawData = counts
+ self.feats = feats
+ self.cells = ds.cells
+ self.name = name
+
+
+type MergeAssay = Assay | DummyAssay
+
+type _RowPlan = tuple[
+ dict[int, dict[int, np.ndarray]],
+ dict[int, dict[int, np.ndarray]],
+ np.ndarray,
+]
+
+
+def _dtype_for_integer_sum(dtype: np.dtype[Any], copies: int) -> np.dtype[Any]:
+ if copies <= 1 or dtype.kind not in "biu":
+ return dtype
+ if dtype.kind in "bu":
+ lower = 0
+ upper = (1 if dtype.kind == "b" else np.iinfo(dtype).max) * copies
+ for candidate in (np.uint8, np.uint16, np.uint32, np.uint64):
+ candidate_info = np.iinfo(candidate)
+ if lower >= candidate_info.min and upper <= candidate_info.max:
+ return np.dtype(candidate)
+ else:
+ info = np.iinfo(dtype)
+ lower = info.min * copies
+ upper = info.max * copies
+ for signed_candidate in (np.int8, np.int16, np.int32, np.int64):
+ candidate_info = np.iinfo(signed_candidate)
+ if lower >= candidate_info.min and upper <= candidate_info.max:
+ return np.dtype(signed_candidate)
+ return np.dtype(np.uint64 if dtype.kind in "bu" else np.int64)
+
+
+class AssayMerge:
+ """Merge multiple Zarr files into a single Zarr file.
+
+ Args:
+ zarr_path: Name of the new, merged Zarr file with path.
+ assays: List of assay objects to be merged. For example, [ds1.RNA, ds2.RNA].
+ names: Names of each of the assay objects in the `assays` parameter. They should be in the same order as in
+ `assays` parameter.
+ merge_assay_name: Name of assay in the merged Zarr file. For example, for scRNA-Seq it could be simply,
+ 'RNA'.
+ in_workspaces: Source workspace per assay (None uses each assay's default layout).
+ out_workspace: Target workspace name in the merged Zarr file.
+ dtype: Dtype of the raw values in the assay. Dtype is automatically inferred from the provided assays. If
+ assays have different dtypes then a float type is used.
+ overwrite: If True, then overwrites previously created assay in the Zarr file. (Default value: False).
+ prepend_text: This text is pre-appended to each column name (Default value: 'orig').
+ reset_cell_filter: If True, then the cell filtering information is removed, i.e. even the filtered out cells
+ are set as True as in the 'I' column. To keep the filtering information set the value for
+ this parameter to False. (Default value: True)
+ seed: Seed for randomization of rows in the assays.
+ source_column: Optional cell-metadata column populated from each entry in ``names``.
+
+ Attributes:
+ assays: List of assay objects to be merged. For example, [ds1.RNA, ds2.RNA].
+ names: Names of each assay objects in the `assays` parameter.
+ mergedCells:
+ nCells: Number of cells in dataset.
+ featCollection:
+ mergedFeats:
+ nFeats: Number of features in the dataset.
+ featOrder:
+ z: The merged Zarr file.
+ assayGroup:
+ """
+
+ def __init__(
+ self,
+ zarr_path: ZarrLocation,
+ assays: list[MergeAssay],
+ names: list[str],
+ merge_assay_name: str,
+ in_workspaces: list[str] | None = None,
+ out_workspace: str | None = None,
+ dtype: str | None = None,
+ overwrite: bool = False,
+ prepend_text: str | None = "orig",
+ reset_cell_filter: bool = True,
+ seed: int | None = 42,
+ storage_options: dict[str, Any] | None = None,
+ source_column: str | None = None,
+ mem_budget: int | str | None = None,
+ nthreads: int | None = None,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+ _row_plan: _RowPlan | None = None,
+ _row_chunk_sizes: list[int] | None = None,
+ ) -> None:
+ validate_assay_name(merge_assay_name)
+ self.assays = assays
+ self.names = names
+ self.inWorkspaces = in_workspaces
+ self.outWorkspace = out_workspace
+ self.merge_assay_name = merge_assay_name
+ assay_resources = [
+ assay.resources for assay in assays if hasattr(assay, "resources")
+ ]
+ resolved_memory = (
+ mem_budget
+ if mem_budget is not None
+ else (
+ min(resource.memoryBytes for resource in assay_resources)
+ if assay_resources
+ else None
+ )
+ )
+ resolved_workers = (
+ nthreads
+ if nthreads is not None
+ else (
+ min(resource.workers for resource in assay_resources)
+ if assay_resources
+ else None
+ )
+ )
+ self.resources = resolve_budget(resolved_memory, resolved_workers)
+ if _row_chunk_sizes is not None and len(_row_chunk_sizes) != len(assays):
+ raise ValueError("Row chunk sizes must match the number of assays")
+ if _row_chunk_sizes is not None and any(rows <= 0 for rows in _row_chunk_sizes):
+ raise ValueError("Row chunk sizes must be positive")
+ plan_rows = sum(int(assay.rawData.shape[0]) for assay in assays)
+ chunk_rows = (
+ [int(assay.rawData.chunksize[0]) for assay in assays]
+ if _row_chunk_sizes is None
+ else [int(rows) for rows in _row_chunk_sizes]
+ )
+ plan_chunks = sum(
+ (int(assay.rawData.shape[0]) + rows - 1) // rows
+ for assay, rows in zip(assays, chunk_rows, strict=True)
+ )
+ plan_features = sum(int(assay.rawData.shape[1]) for assay in assays)
+ index_bytes = np.dtype(np.int64).itemsize
+ row_plan_bytes = (
+ 3 * plan_rows * index_bytes
+ + 2 * plan_chunks * index_bytes
+ + 2 * plan_features * index_bytes
+ )
+ admitted_worker_count(
+ self.resources,
+ taskBytes=max(1, row_plan_bytes),
+ requested=1,
+ )
+ self.profile = resolve_storage_profile(zarr_path, profile)
+ self.storage_options = storage_options
+ self._usesSharedRowPlan = _row_plan is not None or _row_chunk_sizes is not None
+ row_plan = (
+ self.perform_randomization_rows(seed, _row_chunk_sizes)
+ if _row_plan is None
+ else _row_plan
+ )
+ (
+ self.permutations_rows,
+ self.permutations_rows_offset,
+ self.coordinates_permutations,
+ ) = row_plan
+ for assay_idx, assay in enumerate(self.assays):
+ planned_rows = sum(
+ rows.size for rows in self.permutations_rows[assay_idx].values()
+ )
+ if planned_rows != assay.rawData.shape[0]:
+ raise ValueError(
+ "All assays from a datastore must contain the same number of cells"
+ )
+ self.mergedCells: pd.DataFrame = self._merge_cell_table(
+ reset_cell_filter, prepend_text, source_column
+ )
+ self.nCells: int = self.mergedCells.shape[0]
+ self.featCollection: list[dict[str, str]] = self._get_feat_ids(assays)
+ self.feat_name_ids_same: bool = self.check_feat_ids(self.featCollection)
+ self.featCollection_map: list[dict[str, str]]
+
+ if self.feat_name_ids_same is True:
+ self.feat_suffix: dict[int, int] = self.get_feat_suffix()
+ self.featCollection = self.update_feat_ids()
+ self.featCollection_map = self.update_feat_ids_for_map()
+ else:
+ self.featCollection_map = self.featCollection.copy()
+
+ self.mergedFeats: pd.DataFrame = self._merge_order_feats(self.featCollection)
+ self.mergedFeats_map: pd.DataFrame = self._merge_order_feats(
+ self.featCollection_map
+ )
+ self.nFeats: int = self.mergedFeats_map.shape[0]
+ self.featOrder: list[np.ndarray] = self._ref_order_feat_idx()
+ self.featOrder_map: list[np.ndarray]
+
+ if self.feat_name_ids_same is True:
+ self.featOrder_map = self._ref_order_feat_idx_map()
+ else:
+ self.featOrder_map = self.featOrder.copy()
+
+ self.cellOrder: dict[int, dict[int, np.ndarray]] = self._ref_order_cell_idx()
+ admitted_worker_count(
+ self.resources,
+ taskBytes=1,
+ residentBytes=(
+ self._metadata_resident_bytes() + self._row_plan_resident_bytes()
+ ),
+ requested=1,
+ )
+ self.z: zarr.Group = self._use_existing_zarr(
+ zarr_path, merge_assay_name, overwrite
+ )
+ self._ini_cell_data(overwrite)
+ if dtype is None:
+ if len(set([str(x.rawData.dtype) for x in self.assays])) == 1:
+ max_copies = max(
+ (
+ int(np.unique(order_map, return_counts=True)[1].max())
+ for order_map in self.featOrder_map
+ if order_map.size
+ ),
+ default=1,
+ )
+ dtype = str(
+ _dtype_for_integer_sum(
+ np.dtype(self.assays[0].rawData.dtype),
+ max_copies,
+ )
+ )
+ else:
+ dtype = "float"
+
+ self.assayGroup = create_zarr_count_assay(
+ z=self.z,
+ assay_name=merge_assay_name,
+ workspace=self.outWorkspace,
+ n_cells=self.nCells,
+ feat_ids=np.array(self.mergedFeats_map["ids"]),
+ feat_names=np.array(self.mergedFeats_map["names"]),
+ dtype=dtype,
+ profile=self.profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ )
+
+ def _metadata_resident_bytes(self) -> int:
+ frames = (self.mergedCells, self.mergedFeats, self.mergedFeats_map)
+ frame_bytes = sum(
+ int(frame.memory_usage(index=True, deep=True).sum()) for frame in frames
+ )
+ feature_bytes = 0
+ seen: set[int] = set()
+ for collection in (self.featCollection, self.featCollection_map):
+ feature_bytes += sys.getsizeof(collection)
+ for mapping in collection:
+ if id(mapping) in seen:
+ continue
+ seen.add(id(mapping))
+ feature_bytes += sys.getsizeof(mapping)
+ for key, value in mapping.items():
+ for item in (key, value):
+ if id(item) not in seen:
+ seen.add(id(item))
+ feature_bytes += sys.getsizeof(item)
+ return frame_bytes + feature_bytes
+
+ def _row_plan_resident_bytes(self) -> int:
+ arrays = [
+ self.coordinates_permutations,
+ *(
+ rows
+ for chunks in self.permutations_rows.values()
+ for rows in chunks.values()
+ ),
+ *(
+ rows
+ for chunks in self.permutations_rows_offset.values()
+ for rows in chunks.values()
+ ),
+ *(rows for chunks in self.cellOrder.values() for rows in chunks.values()),
+ *self.featOrder,
+ *self.featOrder_map,
+ ]
+ array_bytes = sum(
+ array.nbytes for array in {id(array): array for array in arrays}.values()
+ )
+ mappings = (
+ self.permutations_rows,
+ self.permutations_rows_offset,
+ self.cellOrder,
+ )
+ container_bytes = (
+ sys.getsizeof(self.featOrder)
+ + sys.getsizeof(self.featOrder_map)
+ + sum(
+ sys.getsizeof(mapping)
+ + sum(sys.getsizeof(chunks) for chunks in mapping.values())
+ for mapping in mappings
+ )
+ )
+ return array_bytes + container_bytes
+
+ def perform_randomization_rows(
+ self,
+ seed: int | None = 42,
+ row_chunk_sizes: list[int] | None = None,
+ ) -> tuple[
+ dict[int, dict[int, np.ndarray]], dict[int, dict[int, np.ndarray]], np.ndarray
+ ]:
+ """
+ Perform randomization of rows in the assays.
+ Args:
+ seed: Seed for randomization
+ Returns:
+ """
+ rng = np.random.default_rng(seed=seed)
+ if row_chunk_sizes is None:
+ chunkSize = np.array([x.rawData.chunksize[0] for x in self.assays])
+ else:
+ if len(row_chunk_sizes) != len(self.assays):
+ raise ValueError("Row chunk sizes must match the number of assays")
+ chunkSize = np.asarray(row_chunk_sizes, dtype=int)
+ if np.any(chunkSize <= 0):
+ raise ValueError("Row chunk sizes must be positive")
+ nCells = np.array([x.rawData.shape[0] for x in self.assays])
+ permutations = {
+ i: permute_into_chunks(nCells[i], chunkSize[i])
+ for i in range(len(self.assays))
+ } # Randomize the rows in chunks
+
+ # Create a dictionary of arrays. This is the same data in `permutations` but in a different format. We index the arrays by the chunk number.
+ # Example:
+ # permutation = {0: [array([2, 0, 1]), array([3, 4, 5]), array([8, 7, 6]), array([9])], 1: [array([2, 0, 1]), array([3, 4, 5]), array([8, 7, 6]), array([9])]}
+ # permutations_rows = {0: {0: array([2, 0, 1]), 1: array([3, 4, 5]), 2: array([8, 7, 6]), 3: array([9])}, 1: {0: array([2, 0, 1]), 1: array([3, 4, 5]), 2: array([8, 7, 6]), 3: array([9])}}
+ permutations_rows = {}
+ for key, arrays in permutations.items():
+ in_dict = {i: x for i, x in enumerate(arrays)}
+ permutations_rows[key] = in_dict
+
+ # Set the offset for each chunk. Offset calculated by adding the number of cells in the previous chunks. This will be helpful when we merge the cells metadata in the end.
+ # Example:
+ # {0: {0: array([2, 0, 1]), 1: array([3, 4, 5]), 2: array([8, 7, 6]), 3: array([9])}, 1: {0: array([12, 10, 11]), 1: array([13, 14, 15]), 2: array([18, 17, 16]), 3: array([19])}}
+ permutations_rows_offset = {}
+ offset = 0
+ for key, val_dict in permutations_rows.items():
+ in__dict: dict[int, np.ndarray] = {}
+ for in_key, arrs in val_dict.items():
+ in__dict[in_key] = arrs + offset
+ permutations_rows_offset[key] = in__dict
+ offset += nCells[key]
+
+ # Set the random order in which the rows will be merged. The last chunk of each assay is appended at the end of the list to account for potential incomplete chunks.
+ # Example:
+ # coordinates_permutations = [[0, 0], [0, 1], [1, 2], [0, 2], [1, 1], [1, 0], [0, 3], [1, 3]]
+ # Here [0, 0] means the first chunk of the first assay, [0, 1] means the second chunk of the first assay, [1, 2] means the third chunk of the second assay, and so on will be the order in which the rows will be merged.
+ coordinates = []
+ extra = []
+ for i in range(len(self.assays)):
+ for j in range(len(permutations[i])):
+ if j == len(permutations[i]) - 1: # if j is last, append extra
+ extra.append([i, j])
+ continue
+ coordinates.append([i, j])
+ coordinates_permutations = rng.permutation(
+ coordinates
+ ) # Randomize the order of the coordinates
+ if len(coordinates_permutations) > 0:
+ coordinates_permutations = np.concatenate(
+ [coordinates_permutations, extra], axis=0
+ )
+ else:
+ coordinates_permutations = np.array(extra)
+
+ try:
+ assert permutations_rows_offset[0][0].min() == 0
+ except AssertionError:
+ raise AssertionError(
+ "ERROR: Randomization of rows failed. The first row should be at 0.",
+ "Please report this issue.",
+ )
+ try:
+ assert (
+ permutations_rows_offset[list(permutations_rows_offset.keys())[-1]][
+ list(
+ permutations_rows_offset[
+ list(permutations_rows_offset.keys())[-1]
+ ].keys()
+ )[-1]
+ ].max()
+ == nCells.sum() - 1
+ )
+ except AssertionError:
+ raise AssertionError(
+ "ERROR: Randomization of rows failed. The last row should be at the end of the dataset.",
+ "Please report this issue.",
+ )
+ return permutations_rows, permutations_rows_offset, coordinates_permutations
+
+ def _ref_order_cell_idx(self) -> dict[int, dict[int, np.ndarray]]:
+ """
+ Calculate the order of the cells in the merged assay.
+ """
+ # We calculate the order of the cells in the merged assay by using the permutations_rows and coordinates_permutations. This is essentially the one-to-one mapping of the cells in the assays to the cells in the merged assay.
+ # Example:
+ # cellOrder = {0: {0: array([0, 1, 2]), 1: array([3, 4, 5]), 2: array([ 9, 10, 11]), 3: array([18])}, 1: {0: array([15, 16, 17]), 1: array([12, 13, 14]), 2: array([6, 7, 8]), 3: array([19])}}
+ # Here we see that the cells [2, 0, 1] from the first chunk of the first assay are mapped to [0, 1, 2] in the merged assay. Similarly, the cells [2, 0, 1] from the first chunk of the second assay are mapped to [15, 16, 17] in the merged assay.
+ new_cells: dict[int, dict[int, np.ndarray]] = {}
+ for i in range(len(self.assays)):
+ new_cells[i] = {}
+ offset = 0
+ for x, y in self.coordinates_permutations:
+ size = self.permutations_rows[x][y].size
+ new_cells[x][y] = np.arange(
+ offset,
+ offset + size,
+ dtype=np.int64,
+ )
+ offset += size
+ return new_cells
+
+ def _merge_cell_table(
+ self,
+ reset: bool,
+ prepend_text: str | None = None,
+ source_column: str | None = None,
+ ) -> pd.DataFrame:
+ """Merges the cell metadata table for each sample.
+
+ Args:
+ reset: whether to remove filtering information
+ prepend_text: string to add as prefix for each cell column
+ source_column: optional column populated with each source name
+
+ Returns:
+ """
+ if len(self.assays) != len(set(self.names)):
+ raise ValueError(
+ "ERROR: A unique name should be provided for each of the assay"
+ )
+ if prepend_text == "":
+ prepend_text = None
+ if source_column is not None and (
+ not isinstance(source_column, str)
+ or not source_column.strip()
+ or source_column in {"ids", "I", "names"}
+ ):
+ raise ValueError(
+ "source_column must be a non-empty string that is not ids, I, or names"
+ )
+ ret_val = []
+ for assay, name in zip(self.assays, self.names):
+ a = assay.cells.to_pandas_dataframe(assay.cells.columns)
+ a["ids"] = np.array([f"{name}__{x}" for x in a["ids"]])
+ for i in list(a.columns):
+ if i not in ["ids", "I", "names"] and prepend_text is not None:
+ a[f"{prepend_text}_{i}"] = assay.cells.fetch_all(i)
+ a = a.drop(columns=[i])
+ if source_column is not None:
+ if source_column in a.columns:
+ raise ValueError(
+ f"source_column {source_column!r} conflicts with merged metadata"
+ )
+ a[source_column] = np.repeat(name, len(a))
+ if reset:
+ a["I"] = np.ones(len(a["ids"])).astype(bool)
+ ret_val.append(a)
+
+ # Here we merge the cell metadata tables for each sample. We simply concatenate the tables and reset the index.
+ ret_val_df = pd.concat(ret_val, axis=0).reset_index(drop=True)
+ # Now we use the offsets stored in permutations_rows_offset along with the coordinates_permutations to reorder the cells in the merged assay. The offsets are used to bring the cells in the same order as the rows in the merged assay.
+ compiled_idx = np.concatenate(
+ [
+ self.permutations_rows_offset[i][j]
+ for i, j in self.coordinates_permutations
+ ]
+ )
+ # Index the merged cell metadata table with the compiled_idx to get the final randomized merged cell metadata table.
+ ret_val_df = ret_val_df.iloc[compiled_idx]
+ if sum([x.cells.N for x in self.assays]) != ret_val_df.shape[0]:
+ raise AssertionError(
+ "Unexpected number of cells in the merged table. This is unexpected, "
+ " please report this bug"
+ )
+ return ret_val_df
+
+ @staticmethod
+ def _get_feat_ids(assays: list[MergeAssay]) -> list[dict[str, str]]:
+ """Fetches ID->names mapping of features from each assay.
+
+ Args:
+ assays: List of Assay objects
+
+ Returns:
+ A list of dictionaries. Each dictionary is a id to name
+ mapping for each feature in the corresponding assay
+ """
+ ret_val = []
+ for i in assays:
+ df = i.feats.to_pandas_dataframe(["names", "ids"])
+ ret_val.append(dict(zip(df["ids"].to_numpy(), df["names"].to_numpy())))
+ return ret_val
+
+ def check_feat_ids(self, featCollection: list[dict[str, str]]) -> bool:
+ """
+ Check if feature names and feature ids are different in the assays.
+ """
+ isSame = False
+ for i, dict_ in enumerate(featCollection):
+ keys = np.array(list(dict_.keys()))
+ values = np.array(list(dict_.values()))
+ if np.equal(keys, values).all():
+ logger.warning(
+ f"Feature names and IDs are identical for assay "
+ f"{self.assays[i].name} in dataset {self.names[i]}; "
+ "feature names will be used as IDs"
+ )
+ isSame = True
+ break
+ return isSame
+
+ def get_feat_suffix(self) -> dict[int, int]:
+ """
+ Get the suffix of the feature ids.
+ """
+ feat_suffix = {}
+ for i, dict_ in enumerate(self.featCollection):
+ keys = np.array(list(dict_.keys()))
+ ends_0 = np.array([x.endswith("_0") for x in keys]).sum()
+ ends_1 = np.array([x.endswith("_1") for x in keys]).sum()
+ ends_2 = np.array([x.endswith("_2") for x in keys]).sum()
+ if ends_0 > 0:
+ feat_suffix[i] = 0
+ elif ends_1 > 0:
+ feat_suffix[i] = 1
+ elif ends_0 > 0 and ends_1 > 0:
+ feat_suffix[i] = 0
+ elif ends_2 > 0:
+ raise ValueError(
+ "Feature Numbering starts with 2, this is erroneous. Kindly check the data"
+ )
+ else:
+ feat_suffix[i] = -1
+ return feat_suffix
+
+ def update_feat_ids(self) -> list[dict[str, str]]:
+ """
+ Update the feature ids in case of same feature names and ids.
+
+ Returns:
+ `list[dict[str, str]]`: List of dictionaries containing the updated feature ids for the merged assay.
+
+ This function updates the feature ids for the merged assay in case the feature names and ids are the same in the assays.
+ This function will generate a new feature id and name for the duplicate feature names and ids.
+ We will append a numeric suffix to the feature ids to make them unique. We use this later to map multiple feature ids to a single feature id.
+ """
+ pattern = re.compile(r"_\d+$")
+ # feat_suffix = self.get_feat_suffix()
+ vals = np.array(list(self.feat_suffix.values()))
+ vals = vals[vals > -1]
+ min_val = vals.min() if len(vals) > 0 else 0
+ new_featCollection = []
+ for i, dict_ in enumerate(self.featCollection):
+ in_dict = {}
+ counter = Counter(dict_.values())
+ if self.feat_suffix[i] == -1:
+ sum_counter = {x: 0 for x in np.unique(list(dict_.values()))}
+ # Update all values from 'val' to 'val_{min}'
+ for _, val in dict_.items():
+ if counter[val] == 1: # Unique value
+ in_dict[val] = val
+ else: # Multiple values -- update
+ updated_val = f"{val}_{min_val + sum_counter[val]}"
+ in_dict[updated_val] = updated_val
+ sum_counter[val] += 1
+ else:
+ for _, val in dict_.items():
+ # check if the value ends with a number
+ if pattern.search(val):
+ num = int(val.split("_")[-1])
+ # replace the number with min_val
+ updated_val = pattern.sub(
+ f"_{min_val - self.feat_suffix[i] + num}", val
+ )
+ in_dict[updated_val] = updated_val
+ else:
+ updated_val = f"{val}" # _{min_val}"
+ in_dict[updated_val] = updated_val
+ new_featCollection.append(in_dict)
+ return new_featCollection
+
+ def update_feat_ids_for_map(self) -> list[dict[str, str]]:
+ """
+ Get the updated feature ids mapping for the merged assay in case of same feature names and ids.
+
+ Returns:
+ `list[dict[str, str]]`: List of dictionaries containing the updated feature ids for the merged assay.
+
+ This function updates the feature ids for the merged assay in case the feature names and ids are the same in the assays.
+ This function will remove the numeric suffix from the feature ids and update them with the feature names.
+ """
+ pattern = re.compile(r"_\d+$")
+ new_featCollection = []
+ for dict_ in self.featCollection:
+ in_dict: dict[str, str] = {}
+ for feat_val in dict_.values():
+ if pattern.search(feat_val):
+ base_val = "_".join(feat_val.split("_")[:-1])
+ if base_val not in in_dict:
+ in_dict[base_val] = base_val
+ else:
+ in_dict[feat_val] = feat_val
+ new_featCollection.append(in_dict)
+ return new_featCollection
+
+ def _merge_order_feats(self, feat_collection: list[dict[str, str]]) -> pd.DataFrame:
+ """Merge features from all the assays and determine their order.
+
+ Returns:
+ """
+ union_set: dict[str, str] = {}
+ for ids in feat_collection:
+ for i in ids:
+ if i not in union_set:
+ union_set[i] = ids[i]
+ ret_val = pd.DataFrame(
+ {
+ "idx": list(range(len(union_set))),
+ "names": list(union_set.values()),
+ "ids": list(union_set.keys()),
+ }
+ )
+
+ r = ret_val.shape[0] / sum([x.feats.N for x in self.assays])
+ if r == 1:
+ raise ValueError(
+ "No overlapping features found! Will not merge the files. Please check the features ids "
+ " are comparable across the assays"
+ )
+ if r > 0.9:
+ logger.warning("Fewer than 10% of features overlap across the assays")
+ return ret_val
+
+ def _ref_order_feat_idx(self) -> list[np.ndarray]:
+ ret_val = []
+ for ids in self.featCollection:
+ ordered_ids = pd.DataFrame({"ids": list(ids.keys())})
+ vals = ordered_ids.merge(self.mergedFeats, on="ids", how="left")[
+ "idx"
+ ].to_numpy()
+ ret_val.append(np.array(vals))
+ return ret_val
+
+ def _ref_order_feat_idx_map(self) -> list[np.ndarray]:
+ """
+ Get the order of the features in the merged assay.
+
+ Returns:
+ `list[np.ndarray]`: List of numpy arrays containing the order of the features in the merged assay.
+
+ This function returns the order of the features in the merged assay. The order is determined by the feature
+ """
+ featorder = []
+ name_to_idx_dict = dict(
+ zip(self.mergedFeats_map["names"], self.mergedFeats_map["idx"])
+ )
+ pattern = re.compile(r"_\d+$")
+ for dict_ in self.featCollection:
+ vals = []
+ values_list = []
+ for val in dict_.values():
+ if pattern.search(val):
+ val = "_".join(val.split("_")[:-1]) # Remove the numeric suffix.
+ values_list.append(val)
+ vals = [name_to_idx_dict[name] for name in values_list]
+ featorder.append(np.array(vals))
+ return featorder
+
+ def _use_existing_zarr(
+ self, zarr_loc: ZarrLocation, merge_assay_name: str, overwrite: bool
+ ) -> zarr.Group:
+ if self.outWorkspace is None:
+ cell_slot = "cellData"
+ assay_slot = merge_assay_name
+ else:
+ cell_slot = f"{self.outWorkspace}/cellData"
+ assay_slot = f"{self.outWorkspace}/{merge_assay_name}"
+
+ try:
+ z = load_zarr(zarr_loc, mode="r", storage_options=self.storage_options)
+ if cell_slot not in z:
+ raise ValueError(
+ f"ERROR: Zarr file exists but seems corrupted. Either delete the " # noqa: F541
+ "existing file or choose another path"
+ )
+ if assay_slot in z:
+ if overwrite is False:
+ raise ValueError(
+ f"ERROR: Zarr file already contains {merge_assay_name} assay. Choose "
+ "a different zarr path or a different assay name. Otherwise set overwrite to True"
+ )
+ try:
+ cell_data = as_zarr_group(z[cell_slot], name=cell_slot)
+ cell_ids = as_zarr_array(cell_data["ids"], name="ids")
+ if not all(
+ np.asarray(cell_ids[:]) == np.array(self.mergedCells["ids"])
+ ):
+ raise ValueError(
+ f"ERROR: order of cells does not match the one in existing file" # noqa: F541
+ )
+ except KeyError:
+ raise ValueError(
+ f"ERROR: 'cell data seems corrupted. Either delete the " # noqa: F541
+ "existing file or choose another path"
+ )
+ return load_zarr(zarr_loc, mode="r+", storage_options=self.storage_options)
+ except FileNotFoundError:
+ # So no zarr file with same name exists. Check if a non zarr folder with the same name exists
+ if (
+ is_local_zarr_path(zarr_loc)
+ and isinstance(zarr_loc, str)
+ and os.path.exists(zarr_loc)
+ ):
+ raise ValueError(
+ f"ERROR: Directory/file with name `{zarr_loc}`exists. "
+ f"Either delete it or use another name"
+ )
+ # creating a new zarr file
+ return load_zarr(zarr_loc, mode="w", storage_options=self.storage_options)
+
+ def _ini_cell_data(self, overwrite: bool) -> None:
+ """Save cell attributes to Zarr.
+
+ Returns:
+ None
+ """
+ if self.outWorkspace is None:
+ cell_slot = "cellData"
+ else:
+ cell_slot = f"{self.outWorkspace}/cellData"
+
+ if (cell_slot in self.z and overwrite is True) or cell_slot not in self.z:
+ g = self.z.create_group(cell_slot, overwrite=True)
+ for i in self.mergedCells.columns:
+ vals = np.array(self.mergedCells[i])
+ create_zarr_obj_array(g, str(i), vals, vals.dtype, overwrite=True)
+ else:
+ logger.debug("Cell metadata already exists; skipping initialization")
+
+ def _dask_to_coo(
+ self,
+ d_arr: Any,
+ order: np.ndarray,
+ order_map: np.ndarray,
+ n_threads: int,
+ ) -> coo_matrix:
+ """
+ Convert a chunked array block to a sparse COO matrix.
+ Args:
+ d_arr: Chunked array block to be converted
+ order: Original feature indices
+ order_map: Consolidated feature indices
+ n_threads: Number of threads to use for computation
+ Returns:
+ Sparse COO matrix
+
+ Each source column is remapped through `order_map` to its merged feature
+ index. If multiple source columns map to the same merged feature, their
+ values are summed.
+ """
+ from . import controlled_compute
+
+ computed_data = controlled_compute(d_arr, n_threads)
+ if (
+ order.shape != order_map.shape
+ or order_map.shape[0] != computed_data.shape[1]
+ ):
+ raise ValueError("Feature order does not match the source matrix width")
+
+ source = coo_matrix(computed_data)
+ mapped = coo_matrix(
+ (source.data, (source.row, order_map[source.col])),
+ shape=(computed_data.shape[0], self.nFeats),
+ )
+ if not bool(mapped.has_canonical_format):
+ destination = getattr(self, "assayGroup", None)
+ mapped = canonicalize_sparse(
+ mapped,
+ None if destination is None else destination.dtype,
+ )
+ return mapped
+
+ def dump(self) -> None:
+ """Copy the values from individual assays to the merged assay.
+
+ Returns:
+ """
+ assay_blocks = (
+ []
+ if self._usesSharedRowPlan
+ else [list(assay.rawData.blocks) for assay in self.assays]
+ )
+ coordinates = [
+ (int(assay_idx), int(block_idx))
+ for assay_idx, block_idx in self.coordinates_permutations
+ ]
+
+ expected_start = 0
+ for assay_idx, block_idx in coordinates:
+ row_idx = self.cellOrder[assay_idx][block_idx]
+ if row_idx.size == 0 or int(row_idx[0]) != expected_start:
+ raise AssertionError(
+ "ERROR: Merged block order does not match the cell metadata order."
+ )
+ expected_start += row_idx.size
+
+ def convert_block(coordinate: tuple[int, int]) -> coo_matrix:
+ assay_idx, block_idx = coordinate
+ perm_order = self.permutations_rows[assay_idx][block_idx]
+ if self._usesSharedRowPlan:
+ row_start = int(perm_order.min())
+ row_end = int(perm_order.max()) + 1
+ block = Block(
+ self.assays[assay_idx].rawData,
+ row_start,
+ row_end,
+ row_perm=perm_order - row_start,
+ )
+ else:
+ local_order = perm_order - perm_order.min()
+ block = assay_blocks[assay_idx][block_idx][local_order, :]
+ return self._dask_to_coo(
+ block,
+ self.featOrder[assay_idx],
+ self.featOrder_map[assay_idx],
+ self.resources.workers,
+ )
+
+ def block_stream() -> Iterator[coo_matrix]:
+ blocks = map(convert_block, coordinates)
+ yield from iter_progress(
+ blocks,
+ total=len(coordinates),
+ desc="Writing merged assay",
+ )
+
+ source_rows = max(
+ (
+ int(rows.size)
+ for chunks in self.permutations_rows.values()
+ for rows in chunks.values()
+ ),
+ default=0,
+ )
+ buffered_rows = min(
+ self.nCells,
+ source_rows + array_shard_rows(self.assayGroup),
+ )
+ row_nnz: list[np.ndarray] = []
+ for assay in self.assays:
+ column = f"{assay.name}_nFeatures"
+ if column in assay.cells.columns:
+ counts = np.asarray(
+ assay.cells.fetch_all(column),
+ dtype=np.int64,
+ )
+ else:
+ counts = np.full(
+ int(assay.rawData.shape[0]),
+ int(assay.rawData.shape[1]),
+ dtype=np.int64,
+ )
+ row_nnz.append(counts)
+ ordered_nnz = np.concatenate(
+ [
+ row_nnz[assay_idx][self.permutations_rows[assay_idx][block_idx]]
+ for assay_idx, block_idx in coordinates
+ ]
+ )
+ source_nnz = max(
+ (
+ int(
+ row_nnz[assay_idx][
+ self.permutations_rows[assay_idx][block_idx]
+ ].sum()
+ )
+ for assay_idx, block_idx in coordinates
+ ),
+ default=0,
+ )
+ if ordered_nnz.size:
+ width = min(buffered_rows, ordered_nnz.size)
+ cumulative = np.empty(ordered_nnz.size + 1, dtype=np.int64)
+ cumulative[0] = 0
+ np.cumsum(ordered_nnz, dtype=np.int64, out=cumulative[1:])
+ buffered_nnz = int(np.max(cumulative[width:] - cumulative[:-width]))
+ del cumulative
+ else:
+ buffered_nnz = 0
+ dense_source_elements = max(
+ (
+ rows.size * int(self.assays[assay_idx].rawData.shape[1])
+ for assay_idx, chunks in self.permutations_rows.items()
+ for rows in chunks.values()
+ ),
+ default=0,
+ )
+ del row_nnz, ordered_nnz
+ value_bytes = max(
+ np.dtype(self.assayGroup.dtype).itemsize,
+ *(np.dtype(assay.rawData.dtype).itemsize for assay in self.assays),
+ )
+ resident_bytes = (
+ self._row_plan_resident_bytes() + self._metadata_resident_bytes()
+ )
+ counter = accumulate_sparse_to_shards(
+ self.assayGroup,
+ block_stream(),
+ resources=self.resources,
+ residentBytes=resident_bytes,
+ producerReserveBytes=sparse_producer_peak_bytes(
+ buffered_nnz,
+ source_nnz,
+ value_bytes,
+ )
+ + dense_source_elements * value_bytes,
+ )
+ if counter != self.nCells or expected_start != self.nCells:
+ raise AssertionError(
+ "ERROR: Mismatch in number of cells in the merged assay. Please report this issue."
+ )
diff --git a/scarf/merge/datasets.py b/scarf/merge/datasets.py
new file mode 100644
index 00000000..24c875a1
--- /dev/null
+++ b/scarf/merge/datasets.py
@@ -0,0 +1,221 @@
+from typing import Any
+
+import scarf
+import zarr
+
+from ..matrix import ChunkedArray
+from ..storage.arrays import create_zarr_dataset
+from ..storage.profiles import StorageProfile, ZarrLocation
+from ..utils.logging import logger
+from .assays import AssayMerge, DummyAssay, MergeAssay, _RowPlan
+
+
+class DatasetMerge:
+ """
+ Merge multiple datastores, handling different assay types and generating missing assays on the fly.
+
+ Args:
+ datasets: List of DataStore objects to be merged.
+ zarr_path: Name of the new, merged Zarr file with path.
+ names: Names of each of the dataset objects in the `datasets` parameter. They should be in the same order as in
+ `datasets` parameter.
+ in_workspaces: List of workspaces to be merged. If None, all workspaces are merged.
+ out_workspace: Name of the workspace in the merged Zarr file. If None, the name of the first workspace is used.
+ dtype: Dtype of the raw values in the assay. Dtype is automatically inferred from the provided assays. If
+ assays have different dtypes then a float type is used.
+ overwrite: If True, then overwrites previously created assay in the Zarr file. (Default value: False).
+ prepend_text: This text is pre-appended to each column name (Default value: 'orig').
+ reset_cell_filter: If True, then the cell filtering information is removed, i.e. even the filtered out cells
+ are set as True as in the 'I' column. To keep the filtering information set the value for
+ this parameter to False. (Default value: True)
+ seed: Seed for randomization of rows in the assays.
+ source_column: Optional cell-metadata column populated from each entry in ``names``.
+
+ Example:
+ >>> # Assuming ds1, ds2 and ds3 are DataStore objects
+ >>> # ds1 has RNA and ADT assays. ds2 has RNA assay. ds3 has ADT assay.
+ >>> # Merge RNA and ADT assays from all the datastores
+ >>> merge = DatasetMerge(
+ >>> datasets=[ds1, ds2, ds3],
+ >>> zarr_path="merged.zarr",
+ >>> names=["ds1", "ds2", "ds3"],
+ >>> overwrite = True
+ >>> )
+ >>> merge.dump()
+ >>> # The merged.zarr file will have RNA and ADT assays from all the datastores
+ """
+
+ def __init__(
+ self,
+ datasets: list["scarf.DataStore"],
+ zarr_path: ZarrLocation,
+ names: list[str],
+ in_workspaces: list[str] | None = None,
+ out_workspace: str | None = None,
+ dtype: str | None = None,
+ overwrite: bool = False,
+ prepend_text: str | None = "orig",
+ reset_cell_filter: bool = True,
+ seed: int | None = 42,
+ storage_options: dict[str, Any] | None = None,
+ source_column: str | None = None,
+ mem_budget: int | str | None = None,
+ nthreads: int | None = None,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+ ) -> None:
+ self.datasets = datasets
+ self.names = names
+ self.zarr_path = zarr_path
+ self.in_workspaces = in_workspaces
+ self.out_workspace = out_workspace
+ self.dtype = dtype
+ self.overwrite = overwrite
+ self.prepend_text = prepend_text
+ self.reset_cell_filter = reset_cell_filter
+ self.seed = seed
+ self.storage_options = storage_options
+ self.source_column = source_column
+ self.memBudget = (
+ mem_budget
+ if mem_budget is not None
+ else min(ds.memoryBytes for ds in datasets)
+ )
+ self.nthreads = (
+ nthreads if nthreads is not None else min(ds.nthreads for ds in datasets)
+ )
+ self.profile = profile
+ self.targetChunkBytes = targetChunkBytes
+ self.targetShardBytes = targetShardBytes
+ self.unique_assays = self.get_unique_assays()
+ self.n_unique_assays = len(self.unique_assays)
+ self.merge_generators = self.create_merge_generators()
+
+ def get_unique_assays(self) -> list[str]:
+ """
+ Get unique assays from both datasets
+ """
+ unique_assays = []
+ seen = set()
+ for ds in self.datasets:
+ for assay_name in ds.assay_names:
+ if assay_name not in seen:
+ seen.add(assay_name)
+ unique_assays.append(assay_name)
+ return unique_assays
+
+ def create_merge_generators(self) -> list[AssayMerge]:
+ """
+ Create AssayMerge objects for each unique assay
+ """
+ from . import AssayMerge
+
+ gens: list[AssayMerge] = []
+ shared_row_plan: _RowPlan | None = None
+ row_chunk_sizes = [
+ min(
+ int(ds.get_assay(assay_name).rawData.chunksize[0])
+ for assay_name in ds.assay_names
+ )
+ for ds in self.datasets
+ ]
+ for assay in self.unique_assays:
+ assay_list: list[MergeAssay] = []
+ for ds in self.datasets:
+ if assay in ds.assay_names:
+ assay_list.append(ds.get_assay(assay))
+ else:
+ assay_list.append(self.generate_dummy_assay(ds, assay))
+ generator = AssayMerge(
+ zarr_path=self.zarr_path,
+ assays=assay_list,
+ names=self.names,
+ merge_assay_name=assay,
+ in_workspaces=self.in_workspaces,
+ out_workspace=self.out_workspace,
+ dtype=self.dtype,
+ overwrite=self.overwrite,
+ prepend_text=self.prepend_text,
+ reset_cell_filter=self.reset_cell_filter,
+ seed=self.seed,
+ storage_options=self.storage_options,
+ source_column=self.source_column,
+ mem_budget=self.memBudget,
+ nthreads=self.nthreads,
+ profile=self.profile,
+ targetChunkBytes=self.targetChunkBytes,
+ targetShardBytes=self.targetShardBytes,
+ _row_plan=shared_row_plan,
+ _row_chunk_sizes=row_chunk_sizes,
+ )
+ gens.append(generator)
+ if shared_row_plan is None:
+ shared_row_plan = (
+ generator.permutations_rows,
+ generator.permutations_rows_offset,
+ generator.coordinates_permutations,
+ )
+ return gens
+
+ def generate_dummy_assay(
+ self,
+ ds: "scarf.DataStore",
+ assay_name: str,
+ ) -> DummyAssay:
+ """
+ Generate a dummy assay for a datastore that doesn't have the specified assay
+ """
+ from . import DummyAssay
+
+ # Find a datastore that has this assay to get feature information
+ reference_ds = next(
+ ds_ for ds_ in self.datasets if assay_name in ds_.assay_names
+ )
+ reference_assay = reference_ds.get_assay(assay_name)
+
+ reference_chunk = [
+ ds.get_assay(ref).rawData.chunksize[0] for ref in ds.assay_names
+ ]
+ # check if entries in reference_chunk are the same
+ if not all(x == reference_chunk[0] for x in reference_chunk):
+ rowChunkShape = reference_chunk[0]
+ else:
+ rowChunkShape = max(reference_chunk)
+ colChunk = reference_assay.rawData.chunksize[1]
+ chunkShape = (rowChunkShape, colChunk)
+
+ # Create a dummy assay with zero counts and matching features
+ dummy_shape = (ds.cells.N, reference_assay.feats.N)
+ mem_store = zarr.storage.MemoryStore()
+ mem_group = zarr.open_group(store=mem_store, mode="w")
+ dummy_array = create_zarr_dataset(
+ mem_group,
+ "counts",
+ chunkShape,
+ reference_assay.rawData.dtype,
+ dummy_shape,
+ )
+ dummy_counts = ChunkedArray(
+ dummy_array,
+ nthreads=ds.nthreads,
+ resources=ds.resources,
+ )
+ dummy_assay = DummyAssay(
+ ds, dummy_counts, reference_assay.feats, reference_assay.name
+ )
+ logger.warning(
+ f"Generated an empty {assay_name} assay for a dataset missing that assay"
+ )
+ return dummy_assay
+
+ def dump(self) -> None:
+ """
+ Dump the merged data to the zarr file
+ """
+ logger.info(f"Merging {len(self.merge_generators)} assays")
+ for gen in self.merge_generators:
+ logger.debug(f"Writing merged assay {gen.merge_assay_name}")
+ gen.dump()
+ logger.info(f"Merged {len(self.merge_generators)} assays")
+ return None
diff --git a/scarf/metadata.py b/scarf/metadata.py
deleted file mode 100644
index 62c28f6f..00000000
--- a/scarf/metadata.py
+++ /dev/null
@@ -1,620 +0,0 @@
-"""Contains the MetaData class, which is used for storing metadata about cells
-and features."""
-
-import re
-from typing import List, Iterable, Any, Dict, Tuple, Optional, Union
-
-import numpy as np
-import pandas as pd
-import polars as pl
-from zarr import hierarchy as z_hierarchy
-
-from .feat_utils import fit_lowess
-from .utils import logger
-from .writers import create_zarr_obj_array
-
-zarrGroup = z_hierarchy.Group
-
-__all__ = ["MetaData"]
-
-
-def _all_true(bools: np.ndarray) -> np.ndarray:
- """
-
- Args:
- bools:
-
- Returns:
-
- """
- a = bools.sum(axis=0)
- a[a < bools.shape[0]] = 0
- return a.astype(bool)
-
-
-class MetaData:
- """MetaData class for cells and features.
-
- This class provides an interface to perform CRUD operations on metadata, saved in the Zarr hierarchy.
- All the changes at the metadata are synchronized on disk.
-
- Attributes:
- locations: The locations for where the metadata is stored.
- N: The size of the primary data.
- index: A numpy array with the indices of the cells/features.
- """
-
- def __init__(self, zgrp: zarrGroup):
- """
- Args:
- zgrp: Zarr hierarchy object wherein metadata arrays are saved.
- """
- self.locations: Dict[str, zarrGroup] = {"primary": zgrp}
- self.N = self._get_size(self.locations["primary"], strict_mode=True)
- self.index = np.array(range(self.N))
-
- def _get_size(self, zgrp: zarrGroup, strict_mode: bool = False) -> int:
- """
-
- Args:
- zgrp:
- strict_mode:
-
- Returns:
-
- """
- sizes = []
- for i in zgrp.keys():
- try:
- sizes.append(zgrp[i].shape[0])
- except Exception:
- pass
- if len(sizes) > 0:
- if len(set(sizes)) != 1:
- raise ValueError(
- "ERROR: Metadata table is corrupted. Not all columns are of same length"
- )
- return sizes[0]
- else:
- if strict_mode:
- raise ValueError("Attempted to get size of empty zarr group")
- else:
- return self.N
-
- @staticmethod
- def _col_renamer(loc: str, col: str) -> str:
- """
-
- Args:
- loc:
- col:
-
- Returns:
-
- """
- if loc != "primary":
- return f"{loc}_{col}"
- return col
-
- def _column_map(self) -> Dict[str, Union[str, Tuple[str, str]]]:
- """
-
- Returns:
-
- """
- reserved_cols = ["I", "ids", "names"]
- col_map: Dict[str, Union[str, Tuple[str, str]]] = {
- x: "primary" for x in reserved_cols
- }
- for loc, zgrp in self.locations.items():
- for i in zgrp.keys():
- j = self._col_renamer(loc, i)
- if j in col_map and j not in reserved_cols:
- logger.warning(
- f" {i} is duplicate in metadata loc {loc}. This means something has failed "
- f"upstream. This is quite unexpected. Please report this issue."
- )
- col_map[j] = (loc, i)
- return col_map
-
- def _get_loc(self, column: str) -> Tuple[str, str]:
- """
-
- Args:
- column:
-
- Returns:
-
- """
- col_map = self._column_map()
- if column not in col_map:
- raise KeyError(f"{column} does not exist in the metadata columns.")
- loc, col = col_map[column]
- return loc, col
-
- def _get_array(self, column: str) -> z_hierarchy.Array:
- """
-
- Args:
- column:
-
- Returns:
-
- """
- loc, col = self._get_loc(column)
- return self.locations[loc][col] # type: ignore
-
- def get_dtype(self, column: str) -> type:
- """Returns the dtype for the given column.
-
- Args:
- column: Column name of the table.
- """
- return self._get_array(column).dtype
-
- def _verify_bool(self, key: str) -> bool:
- """Validates if a give table column (parameter 'key') is bool type.
-
- Args:
- key: Name of the column to query
-
- Returns: None
- """
-
- if self.get_dtype(key) != bool:
- raise TypeError(
- "ERROR: `key` should be name of a boolean type column in Metadata table"
- )
- return True
-
- def mount_location(self, zgrp: zarrGroup, identifier: str) -> None:
- """
-
- Args:
- zgrp:
- identifier:
-
- Returns:
-
- """
- if identifier in self.locations:
- raise ValueError(
- f"ERROR: a location with identifier '{identifier}' already mounted"
- )
- size = self._get_size(zgrp)
- if size != self.N:
- raise ValueError(
- f"ERROR: The index size of the mount location ({size}) is not same as primary ({self.N})"
- )
- new_cols = [self._col_renamer(identifier, x) for x in zgrp.keys()]
- cols = self.columns
- conflict_names = [x for x in new_cols if x in cols]
- if len(conflict_names) > 0:
- conflict_names = " ".join(conflict_names)
- raise ValueError(
- f"ERROR: These names in location conflict with existing names: {conflict_names}\n. "
- f"Please try with a different identifier value."
- )
- self.locations[identifier] = zgrp
-
- def unmount_location(self, identifier: str) -> None:
- """
-
- Args:
- identifier:
-
- Returns:
-
- """
- if identifier == "primary":
- raise ValueError("Cannot unmount the primary location")
- if identifier not in self.locations:
- logger.warning(f"{identifier} is not mounted. Nothing to unmount")
- return None
- self.locations.pop(identifier)
-
- @property
- def columns(self) -> List[str]:
- """
-
- Returns:
-
- """
- return list(self._column_map().keys())
-
- def fetch_all(self, column: str) -> np.ndarray:
- """
-
- Args:
- column:
-
- Returns:
-
- """
- return self._get_array(column)[:]
-
- def active_index(self, key: str) -> np.ndarray:
- """
-
- Args:
- key:
-
- Returns:
-
- """
- if self._verify_bool(key):
- return self.index[self.fetch_all(key)]
- else:
- raise ValueError(
- "ERROR: Unexpected error when verifying boolean key. Please report this issue"
- )
-
- def fetch(self, column: str, key: str = "I") -> np.ndarray:
- """Get column values for only valid rows.
-
- Args:
- column:
- key:
-
- Returns:
- """
-
- return self.fetch_all(column)[self.active_index(key)]
-
- def _save(
- self, column_name: str, values: np.ndarray, location: str = "primary"
- ) -> None:
- """
-
- Args:
- column_name:
- values:
-
- Returns:
-
- """
- if location not in self.locations:
- raise KeyError(
- f"ERROR: '{location}' has not been mounted. Save data request failed!"
- )
- if values.shape != (self.N,):
- raise ValueError(
- f"ERROR: Values are of shape: {values.shape}. Expected shape is: ({self.N},)"
- )
- create_zarr_obj_array(
- self.locations[location], column_name, values, values.dtype
- )
- return None
-
- def _fill_to_index(
- self, values: np.ndarray, fill_value, key: str, auto_fill_disable: bool = False
- ) -> np.ndarray:
- """Makes sure that the array being added to the table is of the same
- shape. If the array has same shape as the table then the input array is
- added straightaway. It is assumed that the input array is in same order
- as table rows.
-
- Args:
- values:
- fill_value:
- key:
- auto_fill_disable (bool = False): `fill_value` autocorrection based on dytype is disabled.
-
- Returns:
- """
-
- if isinstance(values, np.ndarray) is False:
- values = np.array(values)
- if auto_fill_disable is False:
- if values.dtype == bool:
- fill_value = False
- elif np.issubdtype(values.dtype, np.integer):
- try:
- if np.isnan(fill_value):
- if min(values) > -1:
- fill_value = 0
- else:
- raise ValueError("`fill_value should be an integer value. ")
- except TypeError:
- raise ValueError("`fill_value should be an interger value. ")
-
- n = values.shape[0]
- if n == self.N:
- return values
- else:
- self._verify_bool(key)
- k = self.fetch_all(key)
- v_n = k.sum()
- if len(values) != v_n:
- raise ValueError(
- f"ERROR: `values` are of incorrect length ({n}). "
- f" Chosen key ({key}) has {v_n} active rows"
- )
- else:
- a = np.empty(self.N).astype(values.dtype)
- a[k] = values
- a[~k] = fill_value
- return a
-
- def get_index_by(
- self, value_targets: List[Any], column: str, key: Optional[str] = None
- ) -> np.ndarray:
- """
-
- Args:
- value_targets:
- column:
- key:
- Returns:
-
- """
- if isinstance(value_targets, Iterable) and not isinstance(value_targets, str):
- if key is None:
- values = self.fetch_all(column)
- else:
- values = self.fetch(column, key)
- value_map = {}
- for n, x in enumerate(values):
- x = x.upper()
- if x not in value_map:
- value_map[x] = []
- value_map[x].append(n)
- ret_val = []
- missing_count = 0
- for i in value_targets:
- i = i.upper()
- if i in value_map:
- ret_val.extend(value_map[i])
- else:
- missing_count += 1
- if missing_count > 0:
- logger.warning(
- f"{missing_count} values were not found in the table column {column}"
- )
- return np.array(ret_val)
- else:
- raise TypeError("ERROR: Please provide the `value_targets` as list")
-
- def index_to_bool(self, idx: np.ndarray, invert: bool = False) -> np.ndarray:
- """
-
- Args:
- idx:
- invert:
-
- Returns:
-
- """
- a = np.zeros(self.N, dtype=bool)
- if len(idx) > 0:
- a[idx] = True
- if invert:
- a = ~a
- return a
-
- def insert(
- self,
- column_name: str,
- values: Union[np.ndarray, List],
- fill_value: Any = np.nan,
- key: str = "I",
- overwrite: bool = False,
- location: str = "primary",
- force: bool = False,
- ) -> None:
- """Insert a column into the table.
-
- Args:
- column_name (str): Name of column to modify.
- values (np.array): Values the column should contain.
- fill_value (Any = np.nan): Value to fill unassigned slots with.
- key (str = 'I'):
- overwrite (bool = False): Should function overwrite column if it already exists?
- location (str = 'primary'):
- force (bool = False): Enforce change to column, even if column is a protected column name ('I' or 'ids').
-
- Returns:
- None
- """
- col = self._col_renamer(location, column_name)
- if col in ["I", "ids"] and force is False:
- raise ValueError(
- f"ERROR: {col} is a protected column name in MetaData class."
- )
- if col in self.columns and overwrite is False:
- raise ValueError(
- f"ERROR: {col} already exists. Please set `overwrite` to True to overwrite."
- )
- if isinstance(values, list):
- logger.debug(
- "'values' parameter is of `list` type and not `np.ndarray` as expected. The correct dtype "
- "may not be assigned to the column"
- )
- v = self._fill_to_index(np.array(values), fill_value, key)
- self._save(column_name, v, location=location)
- return None
-
- def update_key(self, values: np.ndarray, key: str) -> None:
- """Modify a column in the metadata table, specified with `key`.
-
- Args:
- values: The values to update the column with.
- key: Which column in the metadata table to update.
-
- Returns:
- None
- """
- v = self._fill_to_index(values, False, key)
- v = _all_true(np.array([v, self.fetch_all(key)]))
- self._save(key, v)
- return None
-
- def reset_key(self, key: str) -> None:
- """
-
- Args:
- key:
-
- Returns:
-
- """
- values = np.array([True for _ in range(self.N)]).astype(bool)
- self._save(key, values)
- return None
-
- def drop(self, column: str) -> None:
- """
-
- Args:
- column:
-
- Returns:
-
- """
- if column in ["I", "ids", "names"]:
- raise ValueError(
- f"ERROR: {column} is a protected name in MetaData class. Cannot be deleted"
- )
- # noinspection PyUnusedLocal
- loc, col = self._get_loc(column)
- del self.locations[loc][col]
- return None
-
- def sift(
- self,
- column: str,
- min_v: float = -np.inf,
- max_v: float = np.inf,
- keep_bounds: bool = False,
- ) -> np.ndarray:
- """
-
- Args:
- column:
- min_v:
- max_v:
- keep_bounds:
-
- Returns:
-
- """
- values = self.fetch_all(column)
- if keep_bounds:
- return (values >= min_v) & (values <= max_v)
- else:
- return (values > min_v) & (values < max_v)
-
- def multi_sift(
- self,
- columns: List[str],
- lows: Iterable,
- highs: Iterable,
- keep_bounds: bool = False,
- ) -> np.ndarray:
- """
-
- Args:
- columns:
- lows:
- highs:
- keep_bounds:
-
- Returns:
-
- """
- ret_val = _all_true(
- np.array(
- [
- self.sift(i, j, k, keep_bounds=keep_bounds)
- for i, j, k in zip(columns, lows, highs)
- ]
- )
- )
- return ret_val
-
- def head(self, n: int = 5) -> pd.DataFrame:
- """
-
- Args:
- n:
-
- Returns:
-
- """
- df = pd.DataFrame({x: self.fetch_all(x)[:n] for x in self.columns})
- return df
-
- def to_pandas_dataframe(
- self, columns: List[str], key: Optional[str] = None
- ) -> pd.DataFrame:
- """Returns the requested columns as a Pandas dataframe, sorted on
- key."""
- valid_cols = self.columns
- df = pd.DataFrame({x: self.fetch_all(x) for x in columns if x in valid_cols})
- if key is not None:
- df = df.reindex(self.active_index(key))
- return df
-
- def to_polars_dataframe(
- self, columns: List[str], key: Optional[str] = None
- ) -> pl.DataFrame:
- """Returns the requested columns as a Polars DataFrame, sorted on
- key."""
- valid_cols = self.columns
- df = pl.DataFrame({x: self.fetch_all(x) for x in columns if x in valid_cols})
- if key is not None:
- df = df[self.active_index(key)]
- return df
-
- def grep(self, pattern: str, only_valid=False) -> List[str]:
- """
-
- Args:
- pattern:
- only_valid:
-
- Returns:
-
- """
- names = np.array(list(map(str.upper, self.fetch_all("names"))))
- if only_valid:
- names = names[self.active_index("I")]
- return sorted(
- set([x for x in names if re.match(pattern.upper(), x) is not None])
- )
-
- def remove_trend(
- self,
- x: str,
- y: str,
- n_bins: int = 200,
- lowess_frac: float = 0.1,
- fill_value: float = 0,
- ) -> np.ndarray:
- """
-
- Args:
- x:
- y:
- n_bins:
- lowess_frac:
- fill_value:
-
- Returns:
-
- """
- a = self.fetch(x).astype(float)
- b = self.fetch(y).astype(float)
- idx = a > 0
- c = fit_lowess(
- a[idx],
- b[idx],
- n_bins,
- lowess_frac,
- )
- ret_val = np.repeat(fill_value, len(a)).astype(float)
- ret_val[idx] = c
- return ret_val
-
- def __repr__(self):
- return f"MetaData of {self.fetch_all('I').sum()}({self.N}) elements"
diff --git a/scarf/metadata/__init__.py b/scarf/metadata/__init__.py
new file mode 100644
index 00000000..b3ee48d1
--- /dev/null
+++ b/scarf/metadata/__init__.py
@@ -0,0 +1,9 @@
+"""Zarr-backed metadata tables."""
+
+from .rows import MetaDataRowBlock
+from .table import MetaData, zarrGroup as zarrGroup
+
+__all__ = ["MetaData", "MetaDataRowBlock"]
+
+MetaData.__module__ = __name__
+MetaDataRowBlock.__module__ = __name__
diff --git a/scarf/metadata/arguments.py b/scarf/metadata/arguments.py
new file mode 100644
index 00000000..657f4feb
--- /dev/null
+++ b/scarf/metadata/arguments.py
@@ -0,0 +1,393 @@
+from dataclasses import dataclass
+from typing import Any, ClassVar
+
+from ..graph.arguments import (
+ OperationArguments,
+ artifact_input,
+ execution,
+ parameter,
+)
+from ..storage.artifacts import ArtifactRef
+
+
+@dataclass(frozen=True, slots=True)
+class UmapArguments(OperationArguments):
+ operation: ClassVar[str] = "run_umap"
+ artifact_kind: ClassVar[str] = "embedding"
+
+ graph: Any = artifact_input()
+ initialization: Any = artifact_input()
+ symmetric_graph: bool | None = parameter()
+ graph_upper_only: bool | None = parameter()
+ umap_dims: int = parameter()
+ spread: float = parameter()
+ min_dist: float = parameter()
+ n_epochs: int = parameter()
+ repulsion_strength: float = parameter()
+ initial_alpha: float = parameter()
+ negative_sample_rate: float = parameter()
+ use_density_map: bool = parameter()
+ dens_lambda: float = parameter()
+ dens_frac: float = parameter()
+ dens_var_shift: float = parameter()
+ random_seed: int = parameter()
+ parallel: bool = parameter()
+ parallel_threads: int | None = parameter()
+ label: str = execution()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ integrated_graph: str | None = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class TsneArguments(OperationArguments):
+ operation: ClassVar[str] = "run_tsne"
+ artifact_kind: ClassVar[str] = "embedding"
+
+ graph: Any = artifact_input()
+ initialization: Any = artifact_input()
+ symmetric_graph: bool = parameter()
+ graph_upper_only: bool = parameter()
+ tsne_dims: int = parameter()
+ lambda_scale: float = parameter()
+ max_iter: int = parameter()
+ early_iter: int = parameter()
+ alpha: int = parameter()
+ box_h: float = parameter()
+ parallel: bool = parameter()
+ parallel_threads: int = parameter()
+ label: str = execution()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ temp_file_loc: str = execution()
+ verbose: bool = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class LeidenArguments(OperationArguments):
+ operation: ClassVar[str] = "run_leiden_clustering"
+ artifact_kind: ClassVar[str] = "cluster_labels"
+
+ graph: Any = artifact_input()
+ resolution: float = parameter()
+ symmetric_graph: bool = parameter()
+ graph_upper_only: bool = parameter()
+ random_seed: int = parameter()
+ label: str = execution()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ integrated_graph: str | None = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class TopacedoArguments(OperationArguments):
+ operation: ClassVar[str] = "run_topacedo_sampler"
+ artifact_kind: ClassVar[str] = "sampling"
+
+ graph: Any = artifact_input()
+ clusters: Any = artifact_input()
+ dendrogram: Any = artifact_input()
+ cell_selection: ArtifactRef = artifact_input()
+ use_k: int | None = parameter()
+ density_depth: int = parameter()
+ density_bandwidth: float = parameter()
+ max_sampling_rate: float = parameter()
+ min_sampling_rate: float = parameter()
+ min_cells_per_group: int = parameter()
+ snn_bandwidth: float = parameter()
+ seed_reward: float = parameter()
+ non_seed_reward: float = parameter()
+ edge_cost_multiplier: float = parameter()
+ edge_cost_bandwidth: float = parameter()
+ rand_state: int = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ integrated_graph: str | None = execution()
+ cluster_key: str = execution()
+ save_sampling_key: str = execution()
+ save_density_key: str = execution()
+ save_mean_snn_key: str = execution()
+ save_seeds_key: str = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class DoubletScoreArguments(OperationArguments):
+ operation: ClassVar[str] = "run_doublet_detection"
+ artifact_kind: ClassVar[str] = "doublet_score"
+
+ clusters: Any = artifact_input()
+ connectivity_map: Any = artifact_input()
+ cluster_sample_fraction: float = parameter()
+ max_cells_per_cluster: int = parameter()
+ simulation_ratio: float = parameter()
+ heterotypic_fraction: float = parameter()
+ save_k: int = parameter()
+ smoothing_t: int = parameter()
+ normalize_scores: bool = parameter()
+ random_seed: int = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ label: str = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class CellCycleArguments(OperationArguments):
+ operation: ClassVar[str] = "run_cell_cycle_scoring"
+ artifact_kind: ClassVar[str] = "cell_cycle"
+
+ s_gene_indices: tuple[int, ...] = artifact_input()
+ g2m_gene_indices: tuple[int, ...] = artifact_input()
+ normalization_method: dict[str, str] = parameter()
+ size_factor: float | None = parameter()
+ control_size: int = parameter()
+ n_bins: int = parameter()
+ rand_seed: int = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ s_score_label: str = execution()
+ g2m_score_label: str = execution()
+ phase_label: str = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class MarkerTableArguments(OperationArguments):
+ operation: ClassVar[str] = "run_marker_search"
+ artifact_kind: ClassVar[str] = "marker_table"
+
+ cell_selection: ArtifactRef = artifact_input()
+ feature_selection: ArtifactRef = artifact_input()
+ clusters: Any = artifact_input()
+ normalization: dict[str, Any] = parameter()
+ normalization_method: dict[str, str] = parameter()
+ size_factor: float | None = parameter()
+ method: str = parameter()
+ alternative: str = parameter()
+ tie_correction: bool = parameter()
+ continuity_correction: bool = parameter()
+ adjustment_method: str = parameter()
+ adjustment_scope: str = parameter()
+ group_key: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ gene_batch_size: int | None = execution()
+ n_threads: int = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class PseudotimeScoringArguments(OperationArguments):
+ operation: ClassVar[str] = "run_pseudotime_scoring"
+ artifact_kind: ClassVar[str] = "pseudotime"
+
+ connectivity_map: Any = artifact_input()
+ source_sink: Any = artifact_input()
+ cell_selection: ArtifactRef = artifact_input()
+ n_singular_vals: int = parameter()
+ sources: tuple[Any, ...] = parameter()
+ sinks: tuple[Any, ...] = parameter()
+ min_max_norm_ptime: bool = parameter()
+ random_seed: int = parameter()
+ component_policy: str = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ subset_cell_key: str = execution()
+ feat_key: str = execution()
+ label: str = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class FateMappingArguments(OperationArguments):
+ operation: ClassVar[str] = "run_fate_mapping"
+ artifact_kind: ClassVar[str] = "fate_map"
+
+ connectivity_map: Any = artifact_input()
+ pseudotime: Any = artifact_input()
+ sink_labels: Any = artifact_input()
+ cell_selection: ArtifactRef = artifact_input()
+ sinks: tuple[Any, ...] = parameter()
+ beta: float = parameter()
+ solver_tol: float = parameter()
+ max_iterations: int = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ subset_cell_key: str = execution()
+ feat_key: str = execution()
+ pseudotime_key: str = execution()
+ sink_key: str = execution()
+ label: str = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class PseudotimeMarkerArguments(OperationArguments):
+ operation: ClassVar[str] = "run_pseudotime_marker_search"
+ artifact_kind: ClassVar[str] = "pseudotime_markers"
+
+ cell_selection: ArtifactRef = artifact_input()
+ feature_selection: ArtifactRef = artifact_input()
+ pseudotime: Any = artifact_input()
+ normalization: dict[str, Any] = parameter()
+ normalization_method: dict[str, str] = parameter()
+ size_factor: float | None = parameter()
+ association_method: str = parameter()
+ p_value_method: str = parameter()
+ adjustment_method: str = parameter()
+ adjustment_scope: str = parameter()
+ min_cells: int = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ pseudotime_key: str = execution()
+ gene_batch_size: int | None = execution()
+ n_threads: int = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class PseudotimeAggregationArguments(OperationArguments):
+ operation: ClassVar[str] = "run_pseudotime_aggregation"
+ artifact_kind: ClassVar[str] = "pseudotime_aggregation"
+
+ cell_selection: ArtifactRef = artifact_input()
+ feature_selection: ArtifactRef = artifact_input()
+ pseudotime: Any = artifact_input()
+ normalization: dict[str, Any] = parameter()
+ normalization_method: dict[str, str] = parameter()
+ size_factor: float | None = parameter()
+ min_exp: float = parameter()
+ window_size: int = parameter()
+ chunk_size: int = parameter()
+ smoothen: bool = parameter()
+ z_scale: bool = parameter()
+ n_neighbours: int = parameter()
+ n_clusters: int = parameter()
+ ann_params: dict[str, Any] = parameter()
+ nan_cluster_value: int = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ pseudotime_key: str = execution()
+ cluster_label: str = execution()
+ batch_size: int | None = execution()
+ n_threads: int = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class PrevalentPeakArguments(OperationArguments):
+ operation: ClassVar[str] = "mark_prevalent_peaks"
+ artifact_kind: ClassVar[str] = "feature_selection"
+
+ cell_selection: ArtifactRef = artifact_input()
+ feature_selection: ArtifactRef = artifact_input()
+ normalization_method: dict[str, str] = parameter()
+ algorithm_version: int = parameter()
+ top_n: int = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ prevalence_key_name: str = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class WaggrArguments(OperationArguments):
+ operation: ClassVar[str] = "run_waggr"
+ artifact_kind: ClassVar[str] = "enrichment_scores"
+
+ cell_selection: ArtifactRef = artifact_input()
+ feature_selection: ArtifactRef = artifact_input()
+ network_digest: str = artifact_input()
+ algorithm_version: int = parameter()
+ mode: str = parameter()
+ tmin: int = parameter()
+ log_transform: bool = parameter()
+ normalization_method: dict[str, str] = parameter()
+ size_factor: float = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ label: str = execution()
+ overwrite: bool = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class AucellArguments(OperationArguments):
+ operation: ClassVar[str] = "run_aucell"
+ artifact_kind: ClassVar[str] = "enrichment_scores"
+
+ cell_selection: ArtifactRef = artifact_input()
+ feature_selection: ArtifactRef = artifact_input()
+ network_digest: str = artifact_input()
+ algorithm_version: int = parameter()
+ tmin: int = parameter()
+ n_up: int = parameter()
+ tie_seed: int = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ label: str = execution()
+ overwrite: bool = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class HtoIdentityArguments(OperationArguments):
+ operation: ClassVar[str] = "mark_hto_identities"
+ artifact_kind: ClassVar[str] = "hto_identity"
+
+ cell_selection: ArtifactRef = artifact_input()
+ feature_ids_fingerprint: str = artifact_input()
+ method: dict[str, object] = parameter()
+ random_seed: int = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ label: str = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class MembershipStrengthArguments(OperationArguments):
+ operation: ClassVar[str] = "calc_membership_strength"
+ artifact_kind: ClassVar[str] = "membership_strength"
+
+ connectivity_map: Any = artifact_input()
+ clusters: Any = artifact_input()
+ cell_selection: ArtifactRef = artifact_input()
+ algorithm_version: int = parameter()
+ decimals: int = parameter()
+ from_assay: str = execution()
+ cell_key: str = execution()
+ feat_key: str = execution()
+ clust_key: str = execution()
+ output_key: str = execution()
+ invalidate_cache: bool = execution()
+
+
+@dataclass(frozen=True, slots=True)
+class SmartLabelArguments(OperationArguments):
+ operation: ClassVar[str] = "smart_label"
+ artifact_kind: ClassVar[str] = "smart_label"
+
+ values: Any = artifact_input()
+ base_labels: Any = artifact_input()
+ cell_selection: ArtifactRef = artifact_input()
+ algorithm_version: int = parameter()
+ suffix_style: str = parameter()
+ to_relabel: str = execution()
+ base_label: str = execution()
+ cell_key: str = execution()
+ new_col_name: str = execution()
+ invalidate_cache: bool = execution()
diff --git a/scarf/metadata/artifacts.py b/scarf/metadata/artifacts.py
new file mode 100644
index 00000000..b1ee3e06
--- /dev/null
+++ b/scarf/metadata/artifacts.py
@@ -0,0 +1,402 @@
+import colorsys
+import re
+from collections.abc import Mapping
+from typing import Any
+
+import numpy as np
+import zarr
+
+from ..storage.arrays import create_metadata_column, create_zarr_dataset
+from ..storage.artifact_writer import (
+ ArrayRequirement,
+ PlannedArtifact,
+ finish_artifact,
+ plan_artifact,
+ reused_artifact_group,
+ start_artifact,
+)
+from ..storage.artifacts import ArtifactRef, ArtifactScope, artifact_path
+from ..storage.types import as_zarr_array, as_zarr_group
+
+_CATEGORY_COLORS = (
+ "#4e79a7",
+ "#f28e2b",
+ "#e15759",
+ "#76b7b2",
+ "#59a14f",
+ "#edc948",
+ "#b07aa1",
+ "#ff9da7",
+ "#9c755f",
+ "#bab0ab",
+)
+_HEX_COLOR = re.compile(r"^#[0-9a-fA-F]{6}$")
+
+
+def validate_display_metadata(
+ display: Mapping[str, Any],
+) -> dict[str, Any]:
+ value = dict(display)
+ kind = value.get("kind")
+ if kind == "continuous":
+ required = {"kind", "colormap", "minimum", "maximum", "scale"}
+ if not required.issubset(value):
+ raise ValueError("Continuous display metadata is incomplete")
+ if set(value) != required:
+ raise ValueError("Continuous display metadata has unknown fields")
+ if not isinstance(value.get("colormap"), str):
+ raise TypeError("Continuous display colormap must be a string")
+ if value.get("scale") not in {"linear", "log", "symlog"}:
+ raise ValueError("Continuous display scale is invalid")
+ minimum = value.get("minimum")
+ maximum = value.get("maximum")
+ if minimum is not None and (
+ isinstance(minimum, bool)
+ or not isinstance(minimum, int | float)
+ or not np.isfinite(float(minimum))
+ ):
+ raise TypeError("Continuous display minimum must be numeric or null")
+ if maximum is not None and (
+ isinstance(maximum, bool)
+ or not isinstance(maximum, int | float)
+ or not np.isfinite(float(maximum))
+ ):
+ raise TypeError("Continuous display maximum must be numeric or null")
+ if (
+ minimum is not None
+ and maximum is not None
+ and float(minimum) > float(maximum)
+ ):
+ raise ValueError("Continuous display minimum exceeds maximum")
+ return value
+ if kind == "categorical":
+ if not {"kind", "categories"}.issubset(value):
+ raise ValueError("Categorical display metadata is incomplete")
+ allowed = {
+ "kind",
+ "categories",
+ "missing_label",
+ "missing_color",
+ }
+ if not set(value).issubset(allowed):
+ raise ValueError("Categorical display metadata has unknown fields")
+ categories = value.get("categories")
+ if not isinstance(categories, list):
+ raise TypeError("Categorical display categories must be a list")
+ seen_values: set[tuple[str, Any]] = set()
+ mapping_values: list[Any] = []
+ for category in categories:
+ if not isinstance(category, dict):
+ raise TypeError("Each display category must be a mapping")
+ if set(category) != {"value", "label", "color"}:
+ raise ValueError(
+ "Each display category requires value, label, and color"
+ )
+ category_value = category["value"]
+ if category_value is None or (
+ not isinstance(
+ category_value,
+ bool | int | float | str,
+ )
+ or (
+ isinstance(category_value, float)
+ and not np.isfinite(category_value)
+ )
+ ):
+ raise TypeError("Display category value must be a JSON scalar")
+ typed_value = (
+ type(category_value).__name__,
+ category_value,
+ )
+ if typed_value in seen_values:
+ raise ValueError("Display category values must be unique")
+ if any(category_value == existing for existing in mapping_values):
+ raise ValueError("Display category values collide as mapping keys")
+ seen_values.add(typed_value)
+ mapping_values.append(category_value)
+ if not isinstance(category.get("label"), str):
+ raise TypeError("Display category label must be a string")
+ color = category.get("color")
+ if not isinstance(color, str) or _HEX_COLOR.fullmatch(color) is None:
+ raise ValueError("Display category color must be a hex color")
+ missing_label = value.get("missing_label")
+ if missing_label is not None and not isinstance(missing_label, str):
+ raise TypeError("Categorical missing_label must be a string")
+ missing_color = value.get("missing_color")
+ if missing_color is not None and (
+ not isinstance(missing_color, str)
+ or _HEX_COLOR.fullmatch(missing_color) is None
+ ):
+ raise ValueError("Categorical missing_color must be a hex color")
+ return value
+ raise ValueError("Display kind must be continuous or categorical")
+
+
+def plan_cell_data_artifact(
+ root: zarr.Group,
+ *,
+ scope: ArtifactScope,
+ kind: str,
+ operation: str,
+ parameters: dict[str, Any],
+ inputs: dict[str, Any],
+ execution_options: dict[str, Any],
+ cell_selection: ArtifactRef,
+ arrays: Mapping[str, tuple[tuple[int, ...], str | None]],
+ assay: str | None = None,
+ invalidate_cache: bool = False,
+) -> PlannedArtifact:
+ if cell_selection.kind != "cell_selection":
+ raise ValueError("cell_selection must reference a cell-selection artifact")
+ selection_group = as_zarr_group(
+ root[artifact_path(cell_selection)],
+ name=artifact_path(cell_selection),
+ )
+ mask = np.asarray(
+ as_zarr_array(selection_group["values"], name="values")[:],
+ dtype=bool,
+ )
+ selected_count = int(mask.sum())
+ if any(shape[0] != selected_count for shape, _kind in arrays.values()):
+ raise ValueError("Artifact arrays must align with the selected cell count")
+ artifact_inputs = {**inputs, "cell_selection": cell_selection}
+ requirements = tuple(
+ ArrayRequirement(name, shape=shape, dtype_kind=dtype_kind)
+ for name, (shape, dtype_kind) in arrays.items()
+ )
+ return plan_artifact(
+ root,
+ scope=scope,
+ assay=assay,
+ kind=kind,
+ operation=operation,
+ parameters=parameters,
+ inputs=artifact_inputs,
+ execution_options=execution_options,
+ invalidate_cache=invalidate_cache,
+ required_arrays=requirements,
+ )
+
+
+def write_cell_data_artifact(
+ root: zarr.Group,
+ planned: PlannedArtifact,
+ arrays: Mapping[str, np.ndarray],
+) -> zarr.Group:
+ if planned.reused:
+ return reused_artifact_group(root, planned)
+ group = start_artifact(root, planned)
+ for name, raw_values in arrays.items():
+ values = np.asarray(raw_values)
+ if values.ndim < 1:
+ raise ValueError("Artifact arrays must have at least one dimension")
+ if values.dtype.kind in {"O", "S", "U"}:
+ if values.ndim != 1:
+ raise ValueError("String artifact arrays must be one-dimensional")
+ create_metadata_column(
+ group,
+ name,
+ data=values.astype(str),
+ overwrite=True,
+ )
+ else:
+ chunks = (
+ min(max(int(values.shape[0]), 1), 100_000),
+ *values.shape[1:],
+ )
+ output = create_zarr_dataset(
+ group,
+ name,
+ chunks,
+ values.dtype,
+ values.shape,
+ )
+ output[...] = values
+ finish_artifact(group, planned)
+ return group
+
+
+def artifact_values(
+ group: zarr.Group,
+ name: str,
+ value_index: int | None = None,
+) -> np.ndarray:
+ values = np.asarray(as_zarr_array(group[name], name=name)[:])
+ return values if value_index is None else values[:, value_index]
+
+
+def link_cell_data_column(
+ root: zarr.Group,
+ column: str,
+ ref: ArtifactRef,
+ *,
+ value_name: str,
+ value_index: int | None = None,
+ default_display: Mapping[str, Any] | None = None,
+ preserved_display: Mapping[str, Any] | None = None,
+) -> None:
+ cell_data = as_zarr_group(root["cellData"], name="cellData")
+ target = as_zarr_array(cell_data[column], name=column)
+ display_to_write = None
+ if "display" in target.attrs:
+ current_display = target.attrs["display"]
+ if not isinstance(current_display, Mapping):
+ raise TypeError("Existing display metadata must be a mapping")
+ display_to_write = validate_display_metadata(current_display)
+ elif preserved_display is not None:
+ display_to_write = validate_display_metadata(preserved_display)
+ elif default_display is not None:
+ display_to_write = validate_display_metadata(default_display)
+ target.attrs["source_artifact"] = ref.to_dict()
+ target.attrs["source_value"] = value_name
+ if value_index is None:
+ if "value_index" in target.attrs:
+ del target.attrs["value_index"]
+ else:
+ target.attrs["value_index"] = int(value_index)
+ if display_to_write is not None:
+ target.attrs["display"] = display_to_write
+
+
+def column_display(
+ root: zarr.Group,
+ column: str,
+) -> dict[str, Any] | None:
+ cell_data = as_zarr_group(root["cellData"], name="cellData")
+ if column not in cell_data:
+ return None
+ attrs = as_zarr_array(
+ cell_data[column],
+ name=column,
+ ).attrs
+ if "display" not in attrs:
+ return None
+ raw_display = attrs["display"]
+ if not isinstance(raw_display, Mapping):
+ raise TypeError("Existing display metadata must be a mapping")
+ return validate_display_metadata(raw_display)
+
+
+def continuous_display(
+ values: np.ndarray,
+ *,
+ colormap: str = "viridis",
+ scale: str = "linear",
+) -> dict[str, Any]:
+ array = np.asarray(values, dtype=np.float64)
+ finite = array[np.isfinite(array)]
+ minimum = float(finite.min()) if len(finite) else None
+ maximum = float(finite.max()) if len(finite) else None
+ return {
+ "kind": "continuous",
+ "colormap": colormap,
+ "minimum": minimum,
+ "maximum": maximum,
+ "scale": scale,
+ }
+
+
+def _category_color(index: int, count: int) -> str:
+ if count <= len(_CATEGORY_COLORS):
+ return _CATEGORY_COLORS[index]
+ red, green, blue = colorsys.hsv_to_rgb(
+ (index * 0.618033988749895) % 1.0,
+ 0.55,
+ 0.85,
+ )
+ return f"#{round(red * 255):02x}{round(green * 255):02x}{round(blue * 255):02x}"
+
+
+def categorical_display(values: np.ndarray) -> dict[str, Any]:
+ categories: list[Any] = []
+ seen: set[tuple[str, str]] = set()
+ has_missing = False
+ for raw_value in np.asarray(values).reshape(-1):
+ value = raw_value.item() if isinstance(raw_value, np.generic) else raw_value
+ if isinstance(value, float) and np.isnan(value):
+ value = None
+ if value is None:
+ has_missing = True
+ continue
+ key = (type(value).__name__, repr(value))
+ if key not in seen:
+ seen.add(key)
+ categories.append(value)
+ try:
+ categories = sorted(categories)
+ except TypeError:
+ pass
+ count = len(categories)
+ display = {
+ "kind": "categorical",
+ "categories": [
+ {
+ "value": value,
+ "label": "NA" if value is None else str(value),
+ "color": _category_color(index, count),
+ }
+ for index, value in enumerate(categories)
+ ],
+ }
+ if has_missing:
+ display["missing_label"] = "NA"
+ display["missing_color"] = "#bdbdbd"
+ return display
+
+
+def link_feature_data_column(
+ assay_group: zarr.Group,
+ column: str,
+ ref: ArtifactRef,
+ *,
+ value_name: str,
+ value_index: int | None = None,
+ default_display: Mapping[str, Any] | None = None,
+ preserved_display: Mapping[str, Any] | None = None,
+) -> None:
+ feature_data = as_zarr_group(
+ assay_group["featureData"],
+ name="featureData",
+ )
+ target = as_zarr_array(feature_data[column], name=column)
+ display_to_write = None
+ if "display" in target.attrs:
+ current_display = target.attrs["display"]
+ if not isinstance(current_display, Mapping):
+ raise TypeError("Existing display metadata must be a mapping")
+ display_to_write = validate_display_metadata(current_display)
+ elif preserved_display is not None:
+ display_to_write = validate_display_metadata(preserved_display)
+ elif default_display is not None:
+ display_to_write = validate_display_metadata(default_display)
+ target.attrs["source_artifact"] = ref.to_dict()
+ target.attrs["source_value"] = value_name
+ if value_index is None:
+ if "value_index" in target.attrs:
+ del target.attrs["value_index"]
+ else:
+ target.attrs["value_index"] = int(value_index)
+ if display_to_write is not None:
+ target.attrs["display"] = display_to_write
+
+
+def feature_column_display(
+ assay_group: zarr.Group,
+ column: str,
+) -> dict[str, Any] | None:
+ feature_data = as_zarr_group(
+ assay_group["featureData"],
+ name="featureData",
+ )
+ if column not in feature_data:
+ return None
+ attrs = as_zarr_array(
+ feature_data[column],
+ name=column,
+ ).attrs
+ if "display" not in attrs:
+ return None
+ raw_display = attrs["display"]
+ if not isinstance(raw_display, Mapping):
+ raise TypeError("Existing display metadata must be a mapping")
+ return validate_display_metadata(raw_display)
diff --git a/scarf/metadata/queries.py b/scarf/metadata/queries.py
new file mode 100644
index 00000000..c65e737e
--- /dev/null
+++ b/scarf/metadata/queries.py
@@ -0,0 +1,129 @@
+import re
+from collections.abc import Iterable
+from typing import Protocol
+
+import numpy as np
+import pandas as pd
+
+
+class _QueryableMetaData(Protocol):
+ @property
+ def columns(self) -> list[str]: ...
+
+ def active_index(self, key: str) -> np.ndarray: ...
+
+ def fetch(self, column: str, key: str = "I") -> np.ndarray: ...
+
+ def fetch_all(self, column: str) -> np.ndarray: ...
+
+ def sift(
+ self,
+ column: str,
+ min_v: float = -np.inf,
+ max_v: float = np.inf,
+ keep_bounds: bool = False,
+ ) -> np.ndarray: ...
+
+
+def _all_true(bools: np.ndarray) -> np.ndarray:
+ combined = bools.sum(axis=0)
+ combined[combined < bools.shape[0]] = 0
+ return np.asarray(combined, dtype=bool)
+
+
+def sift(
+ metadata: _QueryableMetaData,
+ column: str,
+ min_v: float = -np.inf,
+ max_v: float = np.inf,
+ keep_bounds: bool = False,
+) -> np.ndarray:
+ """Return rows whose values fall within the requested bounds."""
+ values = metadata.fetch_all(column)
+ if keep_bounds:
+ return (values >= min_v) & (values <= max_v)
+ return (values > min_v) & (values < max_v)
+
+
+def multi_sift(
+ metadata: _QueryableMetaData,
+ columns: list[str],
+ lows: Iterable,
+ highs: Iterable,
+ keep_bounds: bool = False,
+) -> np.ndarray:
+ """Return rows that satisfy every requested column filter."""
+ return _all_true(
+ np.array(
+ [
+ metadata.sift(low_column, low, high, keep_bounds=keep_bounds)
+ for low_column, low, high in zip(columns, lows, highs)
+ ]
+ )
+ )
+
+
+def head(metadata: _QueryableMetaData, n: int = 5) -> pd.DataFrame:
+ """Return the first rows of every metadata column."""
+ return pd.DataFrame(
+ {column: metadata.fetch_all(column)[:n] for column in metadata.columns}
+ )
+
+
+def to_pandas_dataframe(
+ metadata: _QueryableMetaData,
+ columns: list[str],
+ key: str | None = None,
+) -> pd.DataFrame:
+ """Return requested metadata columns as a pandas DataFrame."""
+ valid_columns = metadata.columns
+ frame = pd.DataFrame(
+ {
+ column: metadata.fetch_all(column)
+ for column in columns
+ if column in valid_columns
+ }
+ )
+ if key is not None:
+ frame = frame.reindex(metadata.active_index(key))
+ return frame
+
+
+def grep(
+ metadata: _QueryableMetaData,
+ pattern: str,
+ only_valid: bool = False,
+) -> list[str]:
+ """Return feature names that match a case-insensitive regex."""
+ names = np.array(list(map(str.upper, metadata.fetch_all("names"))))
+ if only_valid:
+ names = names[metadata.active_index("I")]
+ return sorted(
+ {name for name in names if re.match(pattern.upper(), name) is not None}
+ )
+
+
+def remove_trend(
+ metadata: _QueryableMetaData,
+ x: str,
+ y: str,
+ n_bins: int = 200,
+ lowess_frac: float = 0.1,
+ fill_value: float = 0,
+) -> np.ndarray:
+ """Remove a LOWESS trend from one metadata column."""
+ from ..features.variability import fit_lowess
+
+ x_values = metadata.fetch(x).astype(float)
+ y_values = metadata.fetch(y).astype(float)
+ positive = x_values > 0
+ trend = fit_lowess(
+ x_values[positive],
+ y_values[positive],
+ n_bins,
+ lowess_frac,
+ bin_strategy="fixed",
+ )
+ residuals = np.repeat(fill_value, len(x_values)).astype(float)
+ residuals[positive] = trend
+ return residuals
diff --git a/scarf/metadata/rows.py b/scarf/metadata/rows.py
new file mode 100644
index 00000000..209b2d0f
--- /dev/null
+++ b/scarf/metadata/rows.py
@@ -0,0 +1,85 @@
+from collections.abc import Iterable, Iterator
+from dataclasses import dataclass
+from typing import Any, Protocol
+
+import numpy as np
+
+from ..storage.geometry import array_geometry
+from ..storage.partition import row_band
+
+
+class _RowReadableMetaData(Protocol):
+ N: int
+
+ @property
+ def columns(self) -> list[str]: ...
+
+ def _get_array(self, column: str) -> Any: ...
+
+ def _verify_bool(self, key: str) -> bool: ...
+
+ def default_block_rows(self, column: str = "I") -> int: ...
+
+
+@dataclass(frozen=True, slots=True)
+class MetaDataRowBlock:
+ """One contiguous slice of a metadata table for blockwise scans."""
+
+ start: int
+ stop: int
+ active_global_indices: np.ndarray
+ values: dict[str, np.ndarray]
+
+
+def default_block_rows(metadata: _RowReadableMetaData, column: str = "I") -> int:
+ """Return a row block size aligned with the backing Zarr chunks."""
+ return row_band(
+ array_geometry(metadata._get_array(column)),
+ unit="chunk",
+ fallback=min(metadata.N, 100_000),
+ )
+
+
+def iter_row_blocks(
+ metadata: _RowReadableMetaData,
+ *,
+ cell_key: str = "I",
+ columns: Iterable[str] | None = None,
+ block_rows: int | None = None,
+) -> Iterator[MetaDataRowBlock]:
+ """Yield contiguous active row blocks from a metadata table."""
+ metadata._verify_bool(cell_key)
+ if block_rows is None:
+ block_rows = metadata.default_block_rows(cell_key)
+ if block_rows < 1:
+ raise ValueError("block_rows must be >= 1")
+
+ if columns is None:
+ column_names: list[str] = []
+ else:
+ column_names = list(columns)
+ for column in column_names:
+ if column not in metadata.columns:
+ raise KeyError(f"{column} does not exist in the metadata columns.")
+
+ key_array = metadata._get_array(cell_key)
+ column_arrays = {column: metadata._get_array(column) for column in column_names}
+
+ for start in range(0, metadata.N, block_rows):
+ stop = min(start + block_rows, metadata.N)
+ key_slice = np.asarray(key_array[start:stop], dtype=bool)
+ local_indices = np.flatnonzero(key_slice)
+ active_global_indices = (local_indices + start).astype(
+ np.int64,
+ copy=False,
+ )
+ values: dict[str, np.ndarray] = {}
+ for column, array in column_arrays.items():
+ block = np.asarray(array[start:stop])
+ values[column] = block[local_indices]
+ yield MetaDataRowBlock(
+ start=start,
+ stop=stop,
+ active_global_indices=active_global_indices,
+ values=values,
+ )
diff --git a/scarf/metadata/table.py b/scarf/metadata/table.py
new file mode 100644
index 00000000..39dc4e8f
--- /dev/null
+++ b/scarf/metadata/table.py
@@ -0,0 +1,396 @@
+from collections.abc import Iterable, Iterator
+from typing import Any
+
+import numpy as np
+import pandas as pd
+import zarr
+
+from ..storage.types import as_zarr_array
+from ..storage.arrays import create_zarr_obj_array
+from ..utils.logging import logger
+from .queries import (
+ _all_true,
+ grep as _grep,
+ head as _head,
+ multi_sift as _multi_sift,
+ remove_trend as _remove_trend,
+ sift as _sift,
+ to_pandas_dataframe as _to_pandas_dataframe,
+)
+from .rows import (
+ MetaDataRowBlock,
+ default_block_rows as _default_block_rows,
+ iter_row_blocks as _iter_row_blocks,
+)
+
+zarrGroup = zarr.Group
+_INTERNAL_METADATA_PREFIX = "__scarf_missing__"
+
+
+class MetaData:
+ """Metadata table for cells and features backed by Zarr arrays.
+
+ Changes made through this class are synchronized with the backing store.
+ """
+
+ def __init__(self, zgrp: zarrGroup):
+ self.locations: dict[str, zarrGroup] = {"primary": zgrp}
+ self.N = self._get_size(self.locations["primary"], strict_mode=True)
+ self.index = np.array(range(self.N))
+
+ def _get_size(self, zgrp: zarrGroup, strict_mode: bool = False) -> int:
+ sizes = []
+ for key in zgrp.keys():
+ try:
+ child = zgrp[key]
+ if isinstance(child, zarr.Array):
+ sizes.append(child.shape[0])
+ except Exception:
+ pass
+ if sizes:
+ if len(set(sizes)) != 1:
+ raise ValueError(
+ "ERROR: Metadata table is corrupted. Not all columns are "
+ "of same length"
+ )
+ return sizes[0]
+ if strict_mode:
+ raise ValueError("Attempted to get size of empty zarr group")
+ return self.N
+
+ @staticmethod
+ def _col_renamer(loc: str, col: str) -> str:
+ if loc != "primary":
+ return f"{loc}_{col}"
+ return col
+
+ def _column_map(self) -> dict[str, str | tuple[str, str]]:
+ reserved_cols = ["I", "ids", "names"]
+ col_map: dict[str, str | tuple[str, str]] = {
+ column: "primary" for column in reserved_cols
+ }
+ for location, group in self.locations.items():
+ for column in group.keys():
+ if column.startswith(_INTERNAL_METADATA_PREFIX):
+ continue
+ public_name = self._col_renamer(location, column)
+ if public_name in col_map and public_name not in reserved_cols:
+ logger.warning(
+ f" {column} is duplicate in metadata loc {location}. "
+ "This means something has failed upstream. This is "
+ "quite unexpected. Please report this issue."
+ )
+ col_map[public_name] = (location, column)
+ return col_map
+
+ def _get_loc(self, column: str) -> tuple[str, str]:
+ col_map = self._column_map()
+ if column not in col_map:
+ raise KeyError(f"{column} does not exist in the metadata columns.")
+ entry = col_map[column]
+ if isinstance(entry, str):
+ return "primary", column
+ location, stored_column = entry
+ return location, stored_column
+
+ def _get_array(self, column: str) -> zarr.Array:
+ location, stored_column = self._get_loc(column)
+ return as_zarr_array(
+ self.locations[location][stored_column],
+ name=stored_column,
+ )
+
+ def get_dtype(self, column: str) -> np.dtype[Any]:
+ """Return the dtype of a metadata column."""
+ return self._get_array(column).dtype
+
+ def _verify_bool(self, key: str) -> bool:
+ if self.get_dtype(key) != bool: # noqa: E721
+ raise TypeError(
+ "ERROR: `key` should be name of a boolean type column in Metadata table"
+ )
+ return True
+
+ def mount_location(self, zgrp: zarrGroup, identifier: str) -> None:
+ if identifier in self.locations:
+ raise ValueError(
+ f"ERROR: a location with identifier '{identifier}' already mounted"
+ )
+ size = self._get_size(zgrp)
+ if size != self.N:
+ raise ValueError(
+ f"ERROR: The index size of the mount location ({size}) is not "
+ f"same as primary ({self.N})"
+ )
+ new_cols = [self._col_renamer(identifier, column) for column in zgrp.keys()]
+ conflict_names = [column for column in new_cols if column in self.columns]
+ if conflict_names:
+ conflict_str = " ".join(conflict_names)
+ raise ValueError(
+ "ERROR: These names in location conflict with existing names: "
+ f"{conflict_str}\n. Please try with a different identifier value."
+ )
+ self.locations[identifier] = zgrp
+
+ def unmount_location(self, identifier: str) -> None:
+ if identifier == "primary":
+ raise ValueError("Cannot unmount the primary location")
+ if identifier not in self.locations:
+ logger.warning(f"{identifier} is not mounted. Nothing to unmount")
+ return None
+ self.locations.pop(identifier)
+
+ @property
+ def columns(self) -> list[str]:
+ """Return all mounted metadata column names."""
+ return list(self._column_map().keys())
+
+ def fetch_all(self, column: str) -> np.ndarray:
+ """Return all values from a metadata column."""
+ return np.asarray(self._get_array(column)[:])
+
+ def active_index(self, key: str) -> np.ndarray:
+ """Return global row indices selected by a boolean column."""
+ if self._verify_bool(key):
+ return np.asarray(self.index[self.fetch_all(key)])
+ raise ValueError(
+ "ERROR: Unexpected error when verifying boolean key. "
+ "Please report this issue"
+ )
+
+ def fetch(self, column: str, key: str = "I") -> np.ndarray:
+ """Return column values for rows selected by ``key``."""
+ return np.asarray(self.fetch_all(column)[self.active_index(key)])
+
+ def default_block_rows(self, column: str = "I") -> int:
+ """Prefer the Zarr chunk length of ``column`` for row iteration."""
+ return _default_block_rows(self, column)
+
+ def iter_row_blocks(
+ self,
+ *,
+ cell_key: str = "I",
+ columns: Iterable[str] | None = None,
+ block_rows: int | None = None,
+ ) -> Iterator[MetaDataRowBlock]:
+ """Yield contiguous row blocks over this table.
+
+ Each block covers a half-open global index range ``[start, stop)``.
+ ``active_global_indices`` lists rows in that range that pass
+ ``cell_key``. Column arrays are aligned to those active indices only.
+ """
+ return _iter_row_blocks(
+ self,
+ cell_key=cell_key,
+ columns=columns,
+ block_rows=block_rows,
+ )
+
+ def _save(
+ self,
+ column_name: str,
+ values: np.ndarray,
+ location: str = "primary",
+ ) -> None:
+ if location not in self.locations:
+ raise KeyError(
+ f"ERROR: '{location}' has not been mounted. Save data request failed!"
+ )
+ if values.shape != (self.N,):
+ raise ValueError(
+ f"ERROR: Values are of shape: {values.shape}. "
+ f"Expected shape is: ({self.N},)"
+ )
+ create_zarr_obj_array(
+ self.locations[location],
+ column_name,
+ values,
+ values.dtype,
+ )
+
+ def _fill_to_index(
+ self,
+ values: np.ndarray,
+ fill_value: Any,
+ key: str,
+ auto_fill_disable: bool = False,
+ ) -> np.ndarray:
+ """Fill values that do not cover every metadata row."""
+ if not isinstance(values, np.ndarray):
+ values = np.array(values)
+ if auto_fill_disable is False:
+ if values.dtype == bool:
+ fill_value = False
+ elif np.issubdtype(values.dtype, np.integer):
+ try:
+ if np.isnan(fill_value):
+ if min(values) > -1:
+ fill_value = 0
+ else:
+ raise ValueError("`fill_value` should be an integer value.")
+ except TypeError:
+ raise ValueError("`fill_value` should be an integer value.")
+
+ n_values = values.shape[0]
+ if n_values == self.N:
+ return values
+
+ self._verify_bool(key)
+ selected = self.fetch_all(key)
+ selected_count = selected.sum()
+ if len(values) != selected_count:
+ raise ValueError(
+ f"ERROR: `values` are of incorrect length ({n_values}). "
+ f" Chosen key ({key}) has {selected_count} active rows"
+ )
+ filled = np.empty(self.N, dtype=values.dtype)
+ filled[selected] = values
+ filled[~selected] = fill_value
+ return filled
+
+ def get_index_by(
+ self,
+ value_targets: list[Any],
+ column: str,
+ key: str | None = None,
+ ) -> np.ndarray:
+ """Return row indices for requested values in a metadata column."""
+ if not isinstance(value_targets, Iterable) or isinstance(value_targets, str):
+ raise TypeError("ERROR: Please provide the `value_targets` as list")
+ if key is None:
+ values = self.fetch_all(column)
+ else:
+ values = self.fetch(column, key)
+ value_map: dict[str, list[int]] = {}
+ for index, value in enumerate(values):
+ normalized = value.upper()
+ value_map.setdefault(normalized, []).append(index)
+ result = []
+ missing_count = 0
+ for target in value_targets:
+ normalized = target.upper()
+ if normalized in value_map:
+ result.extend(value_map[normalized])
+ else:
+ missing_count += 1
+ if missing_count > 0:
+ logger.warning(
+ f"{missing_count} values were not found in the table column {column}"
+ )
+ return np.array(result)
+
+ def index_to_bool(self, idx: np.ndarray, invert: bool = False) -> np.ndarray:
+ """Convert row indices into a table-sized boolean array."""
+ values = np.zeros(self.N, dtype=bool)
+ if len(idx) > 0:
+ values[idx] = True
+ if invert:
+ values = ~values
+ return values
+
+ def insert(
+ self,
+ column_name: str,
+ values: np.ndarray | list,
+ fill_value: Any = np.nan,
+ key: str = "I",
+ overwrite: bool = False,
+ location: str = "primary",
+ force: bool = False,
+ ) -> None:
+ """Insert a column into the table."""
+ column = self._col_renamer(location, column_name)
+ if column in ["I", "ids"] and force is False:
+ raise ValueError(
+ f"ERROR: {column} is a protected column name in MetaData class."
+ )
+ if column in self.columns and overwrite is False:
+ raise ValueError(
+ f"ERROR: {column} already exists. Please set `overwrite` to "
+ "True to overwrite."
+ )
+ if isinstance(values, list):
+ logger.debug(
+ "'values' parameter is of `list` type and not `np.ndarray` as "
+ "expected. The correct dtype may not be assigned to the column"
+ )
+ filled = self._fill_to_index(np.array(values), fill_value, key)
+ self._save(column_name, filled, location=location)
+
+ def update_key(self, values: np.ndarray, key: str) -> None:
+ """Restrict a boolean metadata key using the supplied values."""
+ filled = self._fill_to_index(values, False, key)
+ filled = _all_true(np.array([filled, self.fetch_all(key)]))
+ self._save(key, filled)
+
+ def reset_key(self, key: str) -> None:
+ """Set every value in a boolean metadata key to true."""
+ values = np.array([True for _ in range(self.N)]).astype(bool)
+ self._save(key, values)
+
+ def drop(self, column: str) -> None:
+ """Delete an unprotected metadata column."""
+ if column in ["I", "ids", "names"]:
+ raise ValueError(
+ f"ERROR: {column} is a protected name in MetaData class. "
+ "Cannot be deleted"
+ )
+ location, stored_column = self._get_loc(column)
+ del self.locations[location][stored_column]
+
+ def sift(
+ self,
+ column: str,
+ min_v: float = -np.inf,
+ max_v: float = np.inf,
+ keep_bounds: bool = False,
+ ) -> np.ndarray:
+ """Return rows whose values fall within the requested bounds."""
+ return _sift(self, column, min_v, max_v, keep_bounds)
+
+ def multi_sift(
+ self,
+ columns: list[str],
+ lows: Iterable,
+ highs: Iterable,
+ keep_bounds: bool = False,
+ ) -> np.ndarray:
+ """Return a boolean mask where all column filters are satisfied."""
+ return _multi_sift(self, columns, lows, highs, keep_bounds)
+
+ def head(self, n: int = 5) -> pd.DataFrame:
+ """Return the first ``n`` rows of all metadata columns."""
+ return _head(self, n)
+
+ def to_pandas_dataframe(
+ self,
+ columns: list[str],
+ key: str | None = None,
+ ) -> pd.DataFrame:
+ """Return requested columns as a DataFrame, optionally filtered by key."""
+ return _to_pandas_dataframe(self, columns, key)
+
+ def grep(self, pattern: str, only_valid: bool = False) -> list[str]:
+ """Return feature names matching a case-insensitive regex."""
+ return _grep(self, pattern, only_valid)
+
+ def remove_trend(
+ self,
+ x: str,
+ y: str,
+ n_bins: int = 200,
+ lowess_frac: float = 0.1,
+ fill_value: float = 0,
+ ) -> np.ndarray:
+ """Remove a LOWESS trend of column ``y`` with respect to column ``x``."""
+ return _remove_trend(
+ self,
+ x,
+ y,
+ n_bins,
+ lowess_frac,
+ fill_value,
+ )
+
+ def __repr__(self) -> str:
+ return f"MetaData of {self.fetch_all('I').sum()}({self.N}) elements"
diff --git a/scarf/metrics.py b/scarf/metrics.py
deleted file mode 100644
index fe91a200..00000000
--- a/scarf/metrics.py
+++ /dev/null
@@ -1,465 +0,0 @@
-"""
-Methods and classes for evluation
-"""
-
-from typing import Iterable, Optional, Sequence, Tuple, Union
-
-import numpy as np
-import pandas as pd
-from scipy.sparse import csr_matrix
-from zarr.core import Array as zarrArrayType
-
-from .ann import AnnStream
-from .datastore.datastore import DataStore
-from .utils import (
- logger,
- tqdmbar,
-)
-
-
-# LISI - The Local Inverse Simpson Index
-def compute_lisi(
- distances: zarrArrayType,
- indices: zarrArrayType,
- metadata: pd.DataFrame,
- label_colnames: Iterable[str],
- perplexity: float = 30,
-) -> np.ndarray:
- """Compute the Local Inverse Simpson Index (LISI) for each column in metadata.
-
- LISI measures how well mixed different groups of cells are in the neighborhood of each cell.
- Higher values indicate better mixing of different groups.
-
- Args:
- distances: Pre-computed distances between cells, stored in zarr array format
- indices: Pre-computed nearest neighbor indices, stored in zarr array format
- metadata: DataFrame containing categorical labels for each cell
- label_colnames: Column names in metadata to compute LISI for
- perplexity: Parameter controlling the effective number of neighbors (default: 30)
-
- Returns:
- np.ndarray: Matrix of LISI scores with shape (n_cells, n_labels)
- Each column corresponds to LISI scores for one label column in metadata
-
- Example:
- For metadata with a 'batch' column having 3 categories:
- - LISI ≈ 3: Cell has neighbors from all 3 batches (well mixed)
- - LISI ≈ 1: Cell has neighbors from only 1 batch (poorly mixed)
-
- References:
- Korsunsky et al. 2019 doi: 10.1038/s41592-019-0619-0
- """
-
- n_cells = metadata.shape[0]
- n_labels = len(label_colnames)
- # Don't count yourself
- indices = indices[:, 1:]
- distances = distances[:, 1:]
- lisi_df = np.zeros((n_cells, n_labels))
- for i, label in enumerate(label_colnames):
- logger.info(f"Computing LISI for {label}")
- labels = pd.Categorical(metadata[label])
- n_categories = len(labels.categories)
- simpson = compute_simpson(
- distances.T, indices.T, labels, n_categories, perplexity
- )
- lisi_df[:, i] = 1 / simpson
- return lisi_df
-
-
-def compute_simpson(
- distances: np.ndarray,
- indices: np.ndarray,
- labels: pd.Categorical,
- perplexity: float,
- tol: float = 1e-5,
-) -> np.ndarray:
- """Compute Simpson's diversity index with Gaussian kernel weighting.
-
- This function implements the core calculation for LISI, computing a diversity score
- based on the distribution of categories in each cell's neighborhood.
-
- Args:
- distances: Distance matrix between points, shape (n_neighbors, n_points)
- indices: Index matrix for nearest neighbors, shape (n_neighbors, n_points)
- labels: Categorical labels for each point
- perplexity: Target perplexity for Gaussian kernel
- tol: Convergence tolerance for perplexity calibration (default: 1e-5)
-
- Returns:
- np.ndarray: Array of Simpson's diversity indices, one per point
- """
- n = distances.shape[1]
- P = np.zeros(distances.shape[0])
- simpson = np.zeros(n)
- logU = np.log(perplexity)
- # Loop through each cell.
- for i in tqdmbar(range(n), desc="Computing Simpson's Diversity Index"):
- beta = 1
- betamin = -np.inf
- betamax = np.inf
- # Compute Hdiff
- P = np.exp(-distances[:, i] * beta)
- P_sum = np.sum(P)
- if P_sum == 0:
- H = 0
- P = np.zeros(distances.shape[0])
- else:
- H = np.log(P_sum) + beta * np.sum(distances[:, i] * P) / P_sum
- P = P / P_sum
- Hdiff = H - logU
- n_tries = 50
- for t in range(n_tries):
- # Stop when we reach the tolerance
- if abs(Hdiff) < tol:
- break
- # Update beta
- if Hdiff > 0:
- betamin = beta
- if not np.isfinite(betamax):
- beta *= 2
- else:
- beta = (beta + betamax) / 2
- else:
- betamax = beta
- if not np.isfinite(betamin):
- beta /= 2
- else:
- beta = (beta + betamin) / 2
- # Compute Hdiff
- P = np.exp(-distances[:, i] * beta)
- P_sum = np.sum(P)
- if P_sum == 0:
- H = 0
- P = np.zeros(distances.shape[0])
- else:
- H = np.log(P_sum) + beta * np.sum(distances[:, i] * P) / P_sum
- P = P / P_sum
- Hdiff = H - logU
- # distancesefault value
- if H == 0:
- simpson[i] = -1
- # Simpson's index
- for label_category in labels.categories:
- ix = indices[:, i]
- q = labels[ix] == label_category
- if np.any(q):
- P_sum = np.sum(P[q])
- simpson[i] += P_sum * P_sum
- return simpson
-
-
-# SILHOUETTE SCORE - The Silhouette Score
-def knn_to_csr_matrix(
- neighbor_indices: np.ndarray, neighbor_distances: np.ndarray
-) -> csr_matrix:
- """Convert k-nearest neighbors data to a Compressed Sparse Row (CSR) matrix.
-
- Creates a sparse adjacency matrix representation of the k-nearest neighbors graph
- where edge weights are the distances between points.
-
- Args:
- neighbor_indices: Indices matrix from k-nearest neighbors, shape (n_samples, k)
- neighbor_distances: Distances matrix from k-nearest neighbors, shape (n_samples, k)
-
- Returns:
- scipy.sparse.csr_matrix: Sparse adjacency matrix of shape (n_samples, n_samples)
- where non-zero entries represent distances between neighboring points
- """
- num_samples, num_neighbors = neighbor_indices.shape
- row_indices = np.repeat(np.arange(num_samples), num_neighbors)
- return csr_matrix(
- (neighbor_distances[:].flatten(), (row_indices, neighbor_indices[:].flatten())),
- shape=(num_samples, num_samples),
- )
-
-
-def calculate_weighted_cluster_similarity(
- knn_graph: csr_matrix, cluster_labels: np.ndarray
-) -> np.ndarray:
- """Calculate similarity between clusters based on shared weighted edges.
-
- Uses a weighted Jaccard index to compute similarities between clusters in a KNN graph.
-
- Args:
- knn_graph: CSR matrix representing the KNN graph, shape (n_samples, n_samples)
- cluster_labels: Cluster assignments for each node, must be contiguous integers
- starting from 0
-
- Returns:
- np.ndarray: Symmetric matrix of shape (n_clusters, n_clusters) containing
- pairwise similarities between clusters
-
- Raises:
- AssertionError: If cluster labels are not contiguous integers starting from 0
- """
- unique_cluster_ids = np.unique(cluster_labels)
- expected_cluster_ids = np.arange(0, len(unique_cluster_ids))
- assert np.array_equal(unique_cluster_ids, expected_cluster_ids), (
- "Cluster labels must be contiguous integers starting at 1"
- )
-
- num_clusters = len(unique_cluster_ids)
- inter_cluster_weights = np.zeros((num_clusters, num_clusters))
-
- for cluster_id in unique_cluster_ids:
- nodes_in_cluster = np.where(cluster_labels == cluster_id)[0]
- neighbor_cluster_labels = cluster_labels[knn_graph[nodes_in_cluster].indices]
- neighbor_edge_weights = knn_graph[nodes_in_cluster].data
-
- for neighbor_cluster, edge_weight in zip(
- neighbor_cluster_labels, neighbor_edge_weights
- ):
- inter_cluster_weights[cluster_id, neighbor_cluster] += edge_weight
-
- # assert inter_cluster_weights.sum() == knn_graph.data.sum()
-
- # Ensure symmetry
- inter_cluster_weights = (inter_cluster_weights + inter_cluster_weights.T) / 2
-
- # Calculate total weights for each cluster
- total_cluster_weights = np.array(
- [inter_cluster_weights[i - 1].sum() for i in unique_cluster_ids]
- )
-
- # Calculate similarity using weighted Jaccard index
- similarity_matrix = np.zeros((num_clusters, num_clusters))
-
- for i in range(num_clusters):
- for j in range(i, num_clusters):
- weight_union = (
- total_cluster_weights[i]
- + total_cluster_weights[j]
- - inter_cluster_weights[i, j]
- )
- if weight_union > 0:
- similarity = inter_cluster_weights[i, j] / weight_union
- similarity_matrix[i, j] = similarity_matrix[j, i] = similarity
-
- # Set diagonal to 1 (self-similarity)
- # np.fill_diagonal(similarity_matrix, 1.0)
-
- return similarity_matrix
-
-
-def calculate_top_k_neighbor_distances(
- matrix_a: np.ndarray, matrix_b: np.ndarray, k: int
-) -> np.ndarray:
- """Calculate distances to k nearest neighbors between two sets of points.
-
- For each point in matrix_a, finds the k nearest neighbors in matrix_b
- and returns their distances.
-
- Args:
- matrix_a: First set of points, shape (m, d)
- matrix_b: Second set of points, shape (n, d)
- k: Number of nearest neighbors to find
-
- Returns:
- np.ndarray: Matrix of shape (m, k) containing the distances to the
- k nearest neighbors in matrix_b for each point in matrix_a
-
- Raises:
- AssertionError: If matrices don't have the same number of features
- """
- # Check if the matrices have the same number of features (d)
- assert matrix_a.shape[1] == matrix_b.shape[1], (
- "Matrices must have the same number of features"
- )
-
- # Ensure k is not larger than the number of points in matrix_b
- k = min(k, matrix_b.shape[0])
-
- # Calculate squared Euclidean distances
- a_squared = np.sum(np.square(matrix_a), axis=1, keepdims=True)
- b_squared = np.sum(np.square(matrix_b), axis=1)
-
- # Use broadcasting to compute pairwise distances
- distances = a_squared + b_squared - 2 * np.dot(matrix_a, matrix_b.T)
-
- # Use np.maximum to avoid small negative values due to floating point errors
- distances = np.maximum(distances, 0)
-
- # Find the k smallest distances for each point in matrix_a
- top_k_distances = np.partition(distances, k, axis=1)[:, :k]
-
- # Calculate the square root to get Euclidean distances
- return np.sqrt(top_k_distances)
-
-
-def process_cluster(
- cluster_cells: np.ndarray,
- hvg_data: Union[np.ndarray, zarrArrayType],
- ann_obj: AnnStream,
- k: int,
-) -> Tuple[np.ndarray, np.ndarray]:
- """Process a cluster of cells to prepare data for silhouette scoring.
-
- Randomly splits cluster cells into two groups and applies dimensionality reduction.
-
- Args:
- cluster_cells: Indices of cells belonging to the cluster
- hvg_data: Expression data for highly variable genes
- ann_obj: Object containing dimensionality reduction method
- k: Number of cells to sample from cluster
-
- Returns:
- Tuple[np.ndarray, np.ndarray]: Two arrays containing reduced data for
- different subsets of cells from the cluster
- """
- np.random.shuffle(cluster_cells)
- data_cells = np.array(
- [ann_obj.reducer(hvg_data[i]) for i in sorted(cluster_cells[:k])]
- )
- data_cells_2 = np.array(
- [ann_obj.reducer(hvg_data[i]) for i in sorted(cluster_cells[k : 2 * k])]
- )
- return data_cells, data_cells_2
-
-
-def silhouette_scoring(
- ds: DataStore,
- ann_obj: AnnStream,
- graph: csr_matrix,
- hvg_data: Union[np.ndarray, zarrArrayType],
- assay_type: str,
- res_label: str,
-) -> Optional[np.ndarray]:
- """Compute modified silhouette scores for clusters in single-cell data.
-
- This implementation differs from the standard silhouette score by using
- a graph-based approach and comparing clusters to their nearest neighbors.
-
- Args:
- ds: DataStore object containing cell metadata
- ann_obj: Object containing dimensionality reduction method
- graph: CSR matrix representing the KNN graph
- hvg_data: Expression data for highly variable genes
- assay_type: Type of assay (e.g., 'RNA', 'ATAC')
- res_label: Label for clustering resolution
-
- Returns:
- Optional[np.ndarray]: Array of silhouette scores for each cluster,
- or None if cluster labels are not found
-
- Notes:
- Scores are calculated using a sampling approach for efficiency.
- NaN values indicate clusters that couldn't be scored due to size constraints.
- """
- try:
- clusters = ds.cells.fetch(f"{assay_type}_{res_label}") - 1 # RNA_{res_label}
- except KeyError:
- logger.error(f"Cluster labels not found for {assay_type}_{res_label}")
- return None
-
- cluster_similarity = calculate_weighted_cluster_similarity(graph, clusters)
-
- k = 11
- score = []
-
- for n, i in enumerate(cluster_similarity):
- this_cluster_cells = np.where(clusters == n)[0]
- if len(this_cluster_cells) < 2 * k:
- k = int(len(this_cluster_cells) / 2)
- logger.warning(
- f"Warning: Cluster {n} has fewer than 22 cells. Will adjust k to {k} instead"
- )
-
- for n, i in tqdmbar(
- enumerate(cluster_similarity),
- total=len(cluster_similarity),
- desc="Calculating Silhouette Scores",
- ):
- this_cluster_cells = np.where(clusters == n)[0]
- np.random.shuffle(this_cluster_cells)
- data_this_cells, data_this_cells_2 = process_cluster(
- # n,
- this_cluster_cells,
- hvg_data,
- ann_obj,
- k,
- )
-
- if data_this_cells.size == 0 or data_this_cells_2.size == 0:
- logger.warning(f"Warning: Reduced data for cluster {n} is empty. Skipping.")
- score.append(np.nan)
- continue
-
- k_neighbors = min(k - 1, data_this_cells_2.shape[0] - 1)
-
- if k_neighbors < 1:
- logger.warning(
- f"Warning: Not enough points in cluster {n} for comparison. Skipping."
- )
- score.append(np.nan)
- continue
-
- self_dist = calculate_top_k_neighbor_distances(
- data_this_cells, data_this_cells_2, k - 1
- ).mean()
-
- nearest_cluster = np.argsort(i)[-1]
- nearest_cluster_cells = np.where(clusters == nearest_cluster)[0]
- np.random.shuffle(nearest_cluster_cells)
-
- if len(nearest_cluster_cells) < k:
- logger.warning(
- f"Warning: Nearest cluster {nearest_cluster} has fewer than {k} cells. Skipping."
- )
- score.append(np.nan)
- continue
-
- data_nearest_cells, _ = process_cluster(
- # nearest_cluster,
- nearest_cluster_cells,
- hvg_data,
- ann_obj,
- k,
- )
-
- if data_nearest_cells.size == 0:
- logger.warning(
- f"Warning: Reduced data for nearest cluster {nearest_cluster} is empty. Skipping."
- )
- score.append(np.nan)
- continue
-
- other_dist = calculate_top_k_neighbor_distances(
- data_this_cells, data_nearest_cells, k - 1
- ).mean()
-
- score.append((other_dist - self_dist) / max(self_dist, other_dist))
-
- return np.array(score)
-
-
-def integration_score(
- batch_labels: Sequence[np.ndarray], metric: str = "ari"
-) -> Optional[float]:
- """Calculate integration score between two sets of batch labels.
-
- Args:
- batch_labels: Sequence containing two arrays of batch labels to compare
- metric: Metric to use for comparison, one of:
- - 'ari': Adjusted Rand Index
- - 'nmi': Normalized Mutual Information
-
- Returns:
- Optional[float]: Integration score between 0 and 1, or None if metric
- is not recognized
-
- Notes:
- Higher scores indicate better agreement between batch labels,
- suggesting more effective batch integration.
- """
- from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
-
- if metric == "ari":
- return adjusted_rand_score(batch_labels[0], batch_labels[1])
- elif metric == "nmi":
- return normalized_mutual_info_score(batch_labels[0], batch_labels[1])
- else:
- logger.error(
- f"Metric {metric} not recognized. Please choose from 'ari', 'nmi', 'calinski_harabasz', or 'davies_bouldin'."
- )
- return None
diff --git a/scarf/metrics/__init__.py b/scarf/metrics/__init__.py
new file mode 100644
index 00000000..7e4c84e3
--- /dev/null
+++ b/scarf/metrics/__init__.py
@@ -0,0 +1,46 @@
+"""
+Methods and classes for evaluation
+"""
+
+from ._types import MatrixData, NeighborMetric, ZarrArray
+from .cluster_separability import (
+ ClusterSeparabilityResult,
+ evaluate_cluster_separability,
+)
+from .connectivity import graph_connectivity
+from .concordance import label_concordance_score
+from .graph import (
+ calculate_knn_cluster_similarity,
+ calculate_top_k_neighbor_distances,
+ calculate_weighted_cluster_similarity,
+ knn_to_csr_matrix,
+)
+from .lisi import (
+ clisi_knn,
+ compute_lisi,
+ compute_simpson,
+ ilisi_knn,
+ lisi_batch_mixing_score,
+)
+from .silhouette import process_cluster, silhouette_scoring
+
+__all__ = [
+ "ClusterSeparabilityResult",
+ "MatrixData",
+ "NeighborMetric",
+ "ZarrArray",
+ "calculate_knn_cluster_similarity",
+ "calculate_top_k_neighbor_distances",
+ "calculate_weighted_cluster_similarity",
+ "clisi_knn",
+ "compute_lisi",
+ "compute_simpson",
+ "evaluate_cluster_separability",
+ "graph_connectivity",
+ "ilisi_knn",
+ "knn_to_csr_matrix",
+ "label_concordance_score",
+ "lisi_batch_mixing_score",
+ "process_cluster",
+ "silhouette_scoring",
+]
diff --git a/scarf/metrics/_rows.py b/scarf/metrics/_rows.py
new file mode 100644
index 00000000..ce7ad542
--- /dev/null
+++ b/scarf/metrics/_rows.py
@@ -0,0 +1,14 @@
+import numpy as np
+import zarr
+
+from ..matrix import ChunkedArray
+from ._types import MatrixData
+
+
+def read_matrix_rows(data: MatrixData, row_indices: np.ndarray) -> np.ndarray:
+ row_indices = np.asarray(row_indices, dtype=np.int64)
+ if isinstance(data, ChunkedArray):
+ return data[row_indices].compute()
+ if isinstance(data, zarr.Array):
+ return np.asarray(data.get_orthogonal_selection((row_indices, slice(None))))
+ return np.asarray(data[row_indices])
diff --git a/scarf/metrics/_types.py b/scarf/metrics/_types.py
new file mode 100644
index 00000000..dd2ea973
--- /dev/null
+++ b/scarf/metrics/_types.py
@@ -0,0 +1,10 @@
+from typing import Literal
+
+import numpy as np
+import zarr
+
+from ..matrix import ChunkedArray
+
+type ZarrArray = zarr.Array
+type MatrixData = np.ndarray | ZarrArray | ChunkedArray
+type NeighborMetric = Literal["l2", "cosine", "ip"]
diff --git a/scarf/metrics/cluster_separability.py b/scarf/metrics/cluster_separability.py
new file mode 100644
index 00000000..f450e2f4
--- /dev/null
+++ b/scarf/metrics/cluster_separability.py
@@ -0,0 +1,410 @@
+from collections.abc import Mapping, Sequence
+from dataclasses import dataclass, field
+
+import numpy as np
+import pandas as pd
+
+from ._rows import read_matrix_rows
+from ._types import MatrixData
+
+_CLUSTERING_SCORE_COLUMNS = (
+ "clustering",
+ "n_sampled_cells",
+ "n_clusters",
+ "macro_f1_mean",
+ "macro_f1_standard_error",
+ "weighted_f1_mean",
+ "silhouette_score",
+ "status",
+ "status_reason",
+)
+_CLUSTER_SCORE_COLUMNS = (
+ "clustering",
+ "cluster_label",
+ "n_sampled_cells",
+ "f1_score",
+)
+_CONFUSION_COLUMNS = (
+ "clustering",
+ "true_cluster",
+ "predicted_cluster",
+ "n_cells",
+ "fraction_of_true_cluster",
+)
+
+
+@dataclass(frozen=True, slots=True, eq=False)
+class ClusterSeparabilityResult:
+ clustering_scores: pd.DataFrame = field(repr=False)
+ cluster_scores: pd.DataFrame = field(repr=False)
+ confusion: pd.DataFrame = field(repr=False)
+ sample_indices: np.ndarray = field(repr=False)
+
+ def __post_init__(self) -> None:
+ expected_columns = (
+ (self.clustering_scores, _CLUSTERING_SCORE_COLUMNS),
+ (self.cluster_scores, _CLUSTER_SCORE_COLUMNS),
+ (self.confusion, _CONFUSION_COLUMNS),
+ )
+ for table, columns in expected_columns:
+ if tuple(table.columns) != columns:
+ raise ValueError("Cluster separability result columns are invalid")
+ indices = np.asarray(self.sample_indices)
+ if indices.ndim != 1 or not np.issubdtype(indices.dtype, np.integer):
+ raise ValueError("sample_indices must be a one-dimensional integer array")
+ indices.setflags(write=False)
+ object.__setattr__(self, "sample_indices", indices)
+
+
+def _positive_integer(value: object, name: str, *, minimum: int = 1) -> int:
+ if isinstance(value, bool) or not isinstance(value, int | np.integer):
+ raise TypeError(f"{name} must be an integer")
+ resolved = int(value)
+ if resolved < minimum:
+ raise ValueError(f"{name} must be at least {minimum}")
+ return resolved
+
+
+def _validated_clusterings(
+ clusterings: Mapping[str, Sequence[object] | np.ndarray],
+ n_cells: int,
+) -> dict[str, np.ndarray]:
+ if not isinstance(clusterings, Mapping):
+ raise TypeError("clusterings must be a mapping")
+ if not clusterings:
+ raise ValueError("clusterings must be non-empty")
+
+ validated: dict[str, np.ndarray] = {}
+ for name, labels in clusterings.items():
+ if not isinstance(name, str) or not name:
+ raise TypeError("clustering names must be non-empty strings")
+ values = np.asarray(labels)
+ if values.ndim != 1:
+ raise ValueError(f"Clustering {name!r} labels must be one-dimensional")
+ if len(values) != n_cells:
+ raise ValueError(
+ f"Clustering {name!r} labels must match the coordinate rows"
+ )
+ if pd.isna(values).any():
+ raise ValueError(
+ f"Clustering {name!r} labels must not contain missing values"
+ )
+ validated[name] = values
+ return validated
+
+
+def _stratified_sample_indices(
+ labels: np.ndarray,
+ max_cells: int,
+ rng: np.random.Generator,
+) -> np.ndarray:
+ n_cells = len(labels)
+ if n_cells <= max_cells:
+ return np.arange(n_cells, dtype=np.int64)
+
+ codes, categories = pd.factorize(labels, sort=False)
+ n_clusters = len(categories)
+ if max_cells < n_clusters:
+ raise ValueError(
+ "max_sample_cells must be at least the number of clusters in the "
+ "finest clustering"
+ )
+
+ counts = np.bincount(codes, minlength=n_clusters)
+ targets = counts * (max_cells / n_cells)
+ quotas = np.minimum(counts, np.maximum(1, np.floor(targets).astype(np.int64)))
+
+ while int(quotas.sum()) > max_cells:
+ candidates = np.flatnonzero(quotas > 1)
+ excess = quotas[candidates] - targets[candidates]
+ quotas[candidates[int(np.argmax(excess))]] -= 1
+ while int(quotas.sum()) < max_cells:
+ candidates = np.flatnonzero(quotas < counts)
+ deficits = targets[candidates] - quotas[candidates]
+ quotas[candidates[int(np.argmax(deficits))]] += 1
+
+ sampled = []
+ for cluster_code, quota in enumerate(quotas):
+ cluster_indices = np.flatnonzero(codes == cluster_code)
+ sampled.append(
+ rng.choice(cluster_indices, size=int(quota), replace=False).astype(
+ np.int64,
+ copy=False,
+ )
+ )
+ return np.sort(np.concatenate(sampled))
+
+
+def _silhouette_score(
+ coordinates: np.ndarray,
+ cluster_codes: np.ndarray,
+ *,
+ max_cells: int,
+ random_seed: int,
+) -> float:
+ from sklearn.metrics import silhouette_score
+
+ if len(cluster_codes) > max_cells:
+ if max_cells < int(np.unique(cluster_codes).size):
+ return float("nan")
+ indices = _stratified_sample_indices(
+ cluster_codes,
+ max_cells,
+ np.random.default_rng(random_seed),
+ )
+ coordinates = coordinates[indices]
+ cluster_codes = cluster_codes[indices]
+
+ n_clusters = int(np.unique(cluster_codes).size)
+ if n_clusters < 2 or n_clusters >= len(cluster_codes):
+ return float("nan")
+ return float(silhouette_score(coordinates, cluster_codes))
+
+
+def evaluate_cluster_separability(
+ coordinates: MatrixData,
+ clusterings: Mapping[str, Sequence[object] | np.ndarray],
+ *,
+ n_folds: int = 5,
+ max_sample_cells: int = 50_000,
+ max_silhouette_cells: int = 10_000,
+ random_seed: int = 4444,
+ svm_c: float = 1.0,
+ svm_max_iter: int = 10_000,
+) -> ClusterSeparabilityResult:
+ """Evaluate how well the coordinates recover alternative clusterings."""
+ from sklearn.metrics import (
+ confusion_matrix,
+ f1_score,
+ precision_recall_fscore_support,
+ )
+ from sklearn.model_selection import StratifiedKFold
+ from sklearn.pipeline import make_pipeline
+ from sklearn.preprocessing import StandardScaler
+ from sklearn.svm import LinearSVC
+
+ if len(coordinates.shape) != 2:
+ raise ValueError("coordinates must be two-dimensional")
+ n_cells, n_dimensions = (int(value) for value in coordinates.shape)
+ if n_cells < 1 or n_dimensions < 1:
+ raise ValueError("coordinates must contain cells and dimensions")
+
+ n_folds = _positive_integer(n_folds, "n_folds", minimum=2)
+ max_sample_cells = _positive_integer(max_sample_cells, "max_sample_cells")
+ max_silhouette_cells = _positive_integer(
+ max_silhouette_cells,
+ "max_silhouette_cells",
+ )
+ random_seed = _positive_integer(random_seed, "random_seed", minimum=0)
+ svm_max_iter = _positive_integer(svm_max_iter, "svm_max_iter")
+ if isinstance(svm_c, bool) or not isinstance(
+ svm_c,
+ int | float | np.integer | np.floating,
+ ):
+ raise TypeError("svm_c must be numeric")
+ svm_c = float(svm_c)
+ if not np.isfinite(svm_c) or svm_c <= 0:
+ raise ValueError("svm_c must be finite and greater than zero")
+
+ clustering_values = _validated_clusterings(clusterings, n_cells)
+ finest_name = max(
+ clustering_values,
+ key=lambda name: int(pd.unique(clustering_values[name]).size),
+ )
+ sample_indices = _stratified_sample_indices(
+ clustering_values[finest_name],
+ min(max_sample_cells, n_cells),
+ np.random.default_rng(random_seed),
+ )
+ sampled_coordinates = np.asarray(
+ read_matrix_rows(coordinates, sample_indices),
+ dtype=np.float64,
+ )
+ if sampled_coordinates.shape != (len(sample_indices), n_dimensions):
+ raise ValueError("Sampled coordinate rows have an unexpected shape")
+ if not np.all(np.isfinite(sampled_coordinates)):
+ raise ValueError("coordinates must contain only finite values")
+
+ clustering_rows: list[dict[str, object]] = []
+ cluster_rows: list[dict[str, object]] = []
+ confusion_rows: list[dict[str, object]] = []
+
+ for clustering_name, full_labels in clustering_values.items():
+ sampled_labels = full_labels[sample_indices]
+ cluster_codes, cluster_labels = pd.factorize(sampled_labels, sort=False)
+ cluster_codes = np.asarray(cluster_codes, dtype=np.int64)
+ cluster_labels = np.asarray(cluster_labels)
+ n_clusters = len(cluster_labels)
+ cluster_counts = np.bincount(cluster_codes, minlength=n_clusters)
+ silhouette = _silhouette_score(
+ sampled_coordinates,
+ cluster_codes,
+ max_cells=max_silhouette_cells,
+ random_seed=random_seed,
+ )
+
+ reason: str | None = None
+ if n_clusters < 2:
+ reason = "fewer than two clusters"
+ elif int(cluster_counts.min()) < n_folds:
+ reason = f"a cluster has fewer than {n_folds} sampled cells"
+
+ if reason is not None:
+ clustering_rows.append(
+ {
+ "clustering": clustering_name,
+ "n_sampled_cells": len(sample_indices),
+ "n_clusters": n_clusters,
+ "macro_f1_mean": np.nan,
+ "macro_f1_standard_error": np.nan,
+ "weighted_f1_mean": np.nan,
+ "silhouette_score": silhouette,
+ "status": "unscorable",
+ "status_reason": reason,
+ }
+ )
+ for cluster_label, count in zip(
+ cluster_labels,
+ cluster_counts,
+ strict=True,
+ ):
+ cluster_rows.append(
+ {
+ "clustering": clustering_name,
+ "cluster_label": cluster_label,
+ "n_sampled_cells": int(count),
+ "f1_score": np.nan,
+ }
+ )
+ continue
+
+ cross_validation = StratifiedKFold(
+ n_splits=n_folds,
+ shuffle=True,
+ random_state=random_seed,
+ )
+ out_of_fold = np.full(len(cluster_codes), -1, dtype=np.int64)
+ macro_f1: list[float] = []
+ weighted_f1: list[float] = []
+ for train_indices, test_indices in cross_validation.split(
+ sampled_coordinates,
+ cluster_codes,
+ ):
+ estimator = make_pipeline(
+ StandardScaler(),
+ LinearSVC(
+ C=svm_c,
+ class_weight="balanced",
+ dual="auto",
+ max_iter=svm_max_iter,
+ random_state=random_seed,
+ ),
+ )
+ estimator.fit(
+ sampled_coordinates[train_indices],
+ cluster_codes[train_indices],
+ )
+ predictions = estimator.predict(sampled_coordinates[test_indices])
+ out_of_fold[test_indices] = predictions
+ macro_f1.append(
+ float(
+ f1_score(
+ cluster_codes[test_indices],
+ predictions,
+ average="macro",
+ zero_division=0,
+ )
+ )
+ )
+ weighted_f1.append(
+ float(
+ f1_score(
+ cluster_codes[test_indices],
+ predictions,
+ average="weighted",
+ zero_division=0,
+ )
+ )
+ )
+ if np.any(out_of_fold < 0):
+ raise RuntimeError("Cross-validation did not predict every sampled cell")
+
+ per_cluster = precision_recall_fscore_support(
+ cluster_codes,
+ out_of_fold,
+ labels=np.arange(n_clusters),
+ zero_division=0,
+ )
+ for cluster_label, count, score in zip(
+ cluster_labels,
+ per_cluster[3],
+ per_cluster[2],
+ strict=True,
+ ):
+ cluster_rows.append(
+ {
+ "clustering": clustering_name,
+ "cluster_label": cluster_label,
+ "n_sampled_cells": int(count),
+ "f1_score": float(score),
+ }
+ )
+
+ counts = confusion_matrix(
+ cluster_codes,
+ out_of_fold,
+ labels=np.arange(n_clusters),
+ )
+ row_totals = counts.sum(axis=1)
+ rates = np.divide(
+ counts,
+ row_totals[:, None],
+ out=np.zeros_like(counts, dtype=np.float64),
+ where=row_totals[:, None] > 0,
+ )
+ for true_code, true_label in enumerate(cluster_labels):
+ for predicted_code, predicted_label in enumerate(cluster_labels):
+ confusion_rows.append(
+ {
+ "clustering": clustering_name,
+ "true_cluster": true_label,
+ "predicted_cluster": predicted_label,
+ "n_cells": int(counts[true_code, predicted_code]),
+ "fraction_of_true_cluster": float(
+ rates[true_code, predicted_code]
+ ),
+ }
+ )
+
+ macro_values = np.asarray(macro_f1)
+ clustering_rows.append(
+ {
+ "clustering": clustering_name,
+ "n_sampled_cells": len(sample_indices),
+ "n_clusters": n_clusters,
+ "macro_f1_mean": float(macro_values.mean()),
+ "macro_f1_standard_error": float(
+ macro_values.std(ddof=1) / np.sqrt(n_folds)
+ ),
+ "weighted_f1_mean": float(np.mean(weighted_f1)),
+ "silhouette_score": silhouette,
+ "status": "scored",
+ "status_reason": None,
+ }
+ )
+
+ return ClusterSeparabilityResult(
+ clustering_scores=pd.DataFrame(
+ clustering_rows,
+ columns=_CLUSTERING_SCORE_COLUMNS,
+ ),
+ cluster_scores=pd.DataFrame(
+ cluster_rows,
+ columns=_CLUSTER_SCORE_COLUMNS,
+ ),
+ confusion=pd.DataFrame(
+ confusion_rows,
+ columns=_CONFUSION_COLUMNS,
+ ),
+ sample_indices=sample_indices,
+ )
diff --git a/scarf/metrics/concordance.py b/scarf/metrics/concordance.py
new file mode 100644
index 00000000..0c862f1a
--- /dev/null
+++ b/scarf/metrics/concordance.py
@@ -0,0 +1,39 @@
+from collections.abc import Sequence
+from typing import Literal
+
+import numpy as np
+import pandas as pd
+
+
+def label_concordance_score(
+ label_sets: Sequence[np.ndarray],
+ metric: Literal["ari", "nmi"] = "ari",
+) -> float:
+ """Compare two label partitions using ARI or NMI.
+
+ Args:
+ label_sets: Two arrays of labels to compare.
+ metric: Either ``"ari"`` or ``"nmi"``.
+
+ Returns:
+ Label agreement. ARI ranges from -1 to 1 and NMI from 0 to 1.
+ """
+ from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
+
+ if len(label_sets) != 2:
+ raise ValueError("Exactly two label arrays are required")
+
+ first = np.asarray(label_sets[0])
+ second = np.asarray(label_sets[1])
+ if first.ndim != 1 or second.ndim != 1:
+ raise ValueError("Label arrays must be one-dimensional")
+ if len(first) != len(second):
+ raise ValueError("Label arrays must have matching lengths")
+ if pd.isna(first).any() or pd.isna(second).any():
+ raise ValueError("Label arrays must not contain missing values")
+
+ if metric == "ari":
+ return float(adjusted_rand_score(first, second))
+ if metric == "nmi":
+ return float(normalized_mutual_info_score(first, second))
+ raise ValueError(f"Metric {metric!r} is not one of 'ari' or 'nmi'")
diff --git a/scarf/metrics/connectivity.py b/scarf/metrics/connectivity.py
new file mode 100644
index 00000000..50f0e222
--- /dev/null
+++ b/scarf/metrics/connectivity.py
@@ -0,0 +1,117 @@
+from collections.abc import Sequence
+
+from numba import njit
+import numpy as np
+import pandas as pd
+import zarr
+
+from ..utils.progress import iter_progress
+from ._types import ZarrArray
+
+_CONNECTIVITY_BATCH_ROWS = 100_000
+
+
+@njit(cache=True)
+def _find_root(parent: np.ndarray, node: int) -> int:
+ while parent[node] != node:
+ parent[node] = parent[parent[node]]
+ node = parent[node]
+ return node
+
+
+@njit(cache=True)
+def _union_same_label_edges(
+ edges: np.ndarray,
+ label_codes: np.ndarray,
+ parent: np.ndarray,
+ component_sizes: np.ndarray,
+) -> None:
+ for edge_index in range(edges.shape[0]):
+ source = edges[edge_index, 0]
+ target = edges[edge_index, 1]
+ if label_codes[source] != label_codes[target]:
+ continue
+
+ source_root = _find_root(parent, source)
+ target_root = _find_root(parent, target)
+ if source_root == target_root:
+ continue
+ if component_sizes[source_root] < component_sizes[target_root]:
+ source_root, target_root = target_root, source_root
+
+ parent[target_root] = source_root
+ component_sizes[source_root] += component_sizes[target_root]
+
+
+@njit(cache=True)
+def _compress_paths(parent: np.ndarray) -> None:
+ for node in range(len(parent)):
+ parent[node] = _find_root(parent, node)
+
+
+def graph_connectivity(
+ edges: np.ndarray | ZarrArray,
+ labels: Sequence[object] | np.ndarray,
+ batch_rows: int = _CONNECTIVITY_BATCH_ROWS,
+) -> float:
+ """Score label connectivity on a persisted, implicitly undirected graph.
+
+ Each directed edge is treated as an undirected connection. The result is
+ the mean, across labels, of the fraction of cells in the largest connected
+ component for that label.
+
+ This follows the original scIB symmetrized-graph definition. It does not
+ match the directed strong-component calculation currently implemented by
+ YosefLab ``scib-metrics``.
+
+ References:
+ Luecken et al. 2022 doi: 10.1038/s41592-021-01336-8
+ """
+ if edges.ndim != 2 or edges.shape[1] != 2:
+ raise ValueError("Graph edges must have shape (n_edges, 2)")
+ if not np.issubdtype(edges.dtype, np.integer):
+ raise TypeError("Graph edges must contain integers")
+ if batch_rows < 1:
+ raise ValueError("batch_rows must be greater than zero")
+
+ categorical = pd.Categorical(labels)
+ n_cells = len(categorical)
+ if n_cells == 0:
+ raise ValueError("Graph connectivity requires at least one cell")
+ if np.any(categorical.codes < 0):
+ raise ValueError("Graph connectivity labels must not contain missing values")
+
+ _, label_codes = np.unique(categorical.codes, return_inverse=True)
+ label_codes = np.asarray(label_codes, dtype=np.int64)
+ n_labels = int(label_codes.max()) + 1
+
+ parent = np.arange(n_cells, dtype=np.int64)
+ component_sizes = np.ones(n_cells, dtype=np.int64)
+ chunk_rows = batch_rows
+ if isinstance(edges, zarr.Array):
+ chunk_rows = min(chunk_rows, int(edges.chunks[0]))
+
+ n_edges = edges.shape[0]
+ total = (n_edges + chunk_rows - 1) // chunk_rows
+ for start in iter_progress(
+ range(0, n_edges, chunk_rows),
+ total=total,
+ desc="Computing graph connectivity",
+ ):
+ end = min(start + chunk_rows, n_edges)
+ edge_block = np.asarray(edges[start:end])
+ if np.any(edge_block < 0) or np.any(edge_block >= n_cells):
+ raise IndexError("Graph edge index is outside the label array")
+ _union_same_label_edges(
+ np.asarray(edge_block, dtype=np.int64),
+ label_codes,
+ parent,
+ component_sizes,
+ )
+
+ _compress_paths(parent)
+ roots = np.flatnonzero(parent == np.arange(n_cells))
+ largest = np.zeros(n_labels, dtype=np.int64)
+ np.maximum.at(largest, label_codes[roots], component_sizes[roots])
+ label_sizes = np.bincount(label_codes, minlength=n_labels)
+ return float(np.mean(largest / label_sizes))
diff --git a/scarf/metrics/graph.py b/scarf/metrics/graph.py
new file mode 100644
index 00000000..c04c9cc0
--- /dev/null
+++ b/scarf/metrics/graph.py
@@ -0,0 +1,260 @@
+import numpy as np
+from scipy.sparse import csr_matrix
+
+from ._types import NeighborMetric, ZarrArray
+
+_EDGE_BATCH_ROWS = 100_000
+
+
+def knn_to_csr_matrix(
+ neighbor_indices: np.ndarray,
+ neighbor_distances: np.ndarray,
+ *,
+ use_affinities: bool = False,
+) -> csr_matrix:
+ """Convert k-nearest neighbors data to a Compressed Sparse Row (CSR) matrix.
+
+ Creates a sparse adjacency matrix representation of a KNN graph. Distances
+ can optionally be converted to affinities.
+
+ Args:
+ neighbor_indices: Indices matrix from k-nearest neighbors, shape (n_samples, k)
+ neighbor_distances: Distances matrix from k-nearest neighbors, shape (n_samples, k)
+ use_affinities: Convert distances using ``1 / (log1p(distance) + 1)``.
+
+ Returns:
+ scipy.sparse.csr_matrix: Sparse adjacency matrix of shape (n_samples, n_samples)
+ where non-zero entries represent neighbor weights
+ """
+ neighbor_indices = np.asarray(neighbor_indices)
+ neighbor_distances = np.asarray(neighbor_distances, dtype=np.float64)
+ if neighbor_indices.ndim != 2 or neighbor_distances.ndim != 2:
+ raise ValueError("Neighbor indices and distances must be two-dimensional")
+ if neighbor_indices.shape != neighbor_distances.shape:
+ raise ValueError("Neighbor indices and distances must have matching shapes")
+ if not np.issubdtype(neighbor_indices.dtype, np.integer):
+ raise TypeError("Neighbor indices must contain integers")
+ if not np.all(np.isfinite(neighbor_distances)) or np.any(neighbor_distances < 0):
+ raise ValueError("Neighbor distances must be finite and non-negative")
+
+ num_samples, num_neighbors = neighbor_indices.shape
+ if num_samples == 0 or num_neighbors == 0:
+ raise ValueError("KNN data must contain cells and neighbors")
+ if np.any(neighbor_indices < 0) or np.any(neighbor_indices >= num_samples):
+ raise IndexError("Neighbor index is outside the graph")
+
+ weights = neighbor_distances
+ if use_affinities:
+ weights = 1 / (np.log1p(neighbor_distances) + 1)
+
+ indptr = np.arange(
+ 0,
+ num_samples * num_neighbors + 1,
+ num_neighbors,
+ dtype=np.int64,
+ )
+ return csr_matrix(
+ (weights.reshape(-1), neighbor_indices.reshape(-1), indptr),
+ shape=(num_samples, num_samples),
+ )
+
+
+def _validated_cluster_labels(
+ cluster_labels: np.ndarray, n_nodes: int
+) -> tuple[np.ndarray, int]:
+ cluster_labels = np.asarray(cluster_labels)
+ if len(cluster_labels) != n_nodes:
+ raise ValueError("Cluster labels must have one value per graph node")
+ if not np.issubdtype(cluster_labels.dtype, np.integer):
+ raise TypeError("Cluster labels must contain integers")
+
+ unique_cluster_ids = np.unique(cluster_labels)
+ expected_cluster_ids = np.arange(0, len(unique_cluster_ids))
+ if not np.array_equal(unique_cluster_ids, expected_cluster_ids):
+ raise ValueError("Cluster labels must be contiguous integers starting at 0")
+ return cluster_labels, len(unique_cluster_ids)
+
+
+def _finalize_cluster_similarity(inter_cluster_weights: np.ndarray) -> np.ndarray:
+ inter_cluster_weights = (inter_cluster_weights + inter_cluster_weights.T) / 2
+ total_cluster_weights = inter_cluster_weights.sum(axis=1)
+ weight_union = (
+ total_cluster_weights[:, None]
+ + total_cluster_weights[None, :]
+ - inter_cluster_weights
+ )
+ similarity_matrix = np.divide(
+ inter_cluster_weights,
+ weight_union,
+ out=np.zeros_like(inter_cluster_weights),
+ where=weight_union > 0,
+ )
+ np.fill_diagonal(similarity_matrix, 1.0)
+ return np.asarray(similarity_matrix)
+
+
+def calculate_weighted_cluster_similarity(
+ knn_graph: csr_matrix, cluster_labels: np.ndarray
+) -> np.ndarray:
+ """Calculate similarity between clusters based on shared weighted edges.
+
+ Uses a weighted Jaccard index to compute similarities between clusters in a KNN graph.
+
+ Args:
+ knn_graph: CSR matrix representing the KNN graph, shape (n_samples, n_samples)
+ cluster_labels: Cluster assignments for each node, must be contiguous integers
+ starting from 0
+
+ Returns:
+ np.ndarray: Symmetric matrix of shape (n_clusters, n_clusters) containing
+ pairwise similarities between clusters
+
+ Raises:
+ ValueError: If cluster labels are not contiguous integers starting from 0
+ """
+ knn_graph = knn_graph.tocsr(copy=False)
+ if knn_graph.shape[0] != knn_graph.shape[1]:
+ raise ValueError("KNN graph must be square")
+ if not np.all(np.isfinite(knn_graph.data)) or np.any(knn_graph.data < 0):
+ raise ValueError("KNN graph weights must be finite and non-negative")
+
+ cluster_labels, num_clusters = _validated_cluster_labels(
+ cluster_labels, knn_graph.shape[0]
+ )
+ inter_cluster_weights = np.zeros((num_clusters, num_clusters), dtype=np.float64)
+ for row_start in range(0, knn_graph.shape[0], _EDGE_BATCH_ROWS):
+ row_end = min(row_start + _EDGE_BATCH_ROWS, knn_graph.shape[0])
+ edge_start = knn_graph.indptr[row_start]
+ edge_end = knn_graph.indptr[row_end]
+ source_clusters = np.repeat(
+ cluster_labels[row_start:row_end],
+ np.diff(knn_graph.indptr[row_start : row_end + 1]),
+ )
+ target_clusters = cluster_labels[knn_graph.indices[edge_start:edge_end]]
+ pair_ids = source_clusters * num_clusters + target_clusters
+ weight_counts = np.asarray(
+ np.bincount(
+ pair_ids,
+ weights=knn_graph.data[edge_start:edge_end],
+ minlength=num_clusters * num_clusters,
+ ),
+ dtype=np.float64,
+ )
+ inter_cluster_weights += weight_counts.reshape(num_clusters, num_clusters)
+
+ return _finalize_cluster_similarity(inter_cluster_weights)
+
+
+def calculate_knn_cluster_similarity(
+ neighbor_indices: np.ndarray | ZarrArray,
+ neighbor_distances: np.ndarray | ZarrArray,
+ cluster_labels: np.ndarray,
+ *,
+ batch_rows: int = _EDGE_BATCH_ROWS,
+) -> np.ndarray:
+ """Stream KNN rows and calculate cluster similarity from distance affinities."""
+ if neighbor_indices.ndim != 2 or neighbor_distances.ndim != 2:
+ raise ValueError("Neighbor indices and distances must be two-dimensional")
+ if neighbor_indices.shape != neighbor_distances.shape:
+ raise ValueError("Neighbor indices and distances must have matching shapes")
+ if neighbor_indices.shape[1] == 0:
+ raise ValueError("KNN data must contain neighbors")
+ if batch_rows < 1:
+ raise ValueError("batch_rows must be greater than zero")
+
+ n_nodes, n_neighbors = neighbor_indices.shape
+ cluster_labels, num_clusters = _validated_cluster_labels(cluster_labels, n_nodes)
+ inter_cluster_weights = np.zeros((num_clusters, num_clusters), dtype=np.float64)
+ for row_start in range(0, n_nodes, batch_rows):
+ row_end = min(row_start + batch_rows, n_nodes)
+ index_block = np.asarray(neighbor_indices[row_start:row_end])
+ distance_block = np.asarray(
+ neighbor_distances[row_start:row_end], dtype=np.float64
+ )
+ if not np.issubdtype(index_block.dtype, np.integer):
+ raise TypeError("Neighbor indices must contain integers")
+ if np.any(index_block < 0) or np.any(index_block >= n_nodes):
+ raise IndexError("Neighbor index is outside the graph")
+ if not np.all(np.isfinite(distance_block)) or np.any(distance_block < 0):
+ raise ValueError("Neighbor distances must be finite and non-negative")
+
+ affinities = 1 / (np.log1p(distance_block) + 1)
+ source_clusters = np.repeat(cluster_labels[row_start:row_end], n_neighbors)
+ target_clusters = cluster_labels[index_block.reshape(-1)]
+ pair_ids = source_clusters * num_clusters + target_clusters
+ weight_counts = np.asarray(
+ np.bincount(
+ pair_ids,
+ weights=affinities.reshape(-1),
+ minlength=num_clusters * num_clusters,
+ ),
+ dtype=np.float64,
+ )
+ inter_cluster_weights += weight_counts.reshape(num_clusters, num_clusters)
+
+ return _finalize_cluster_similarity(inter_cluster_weights)
+
+
+def calculate_top_k_neighbor_distances(
+ matrix_a: np.ndarray,
+ matrix_b: np.ndarray,
+ k: int,
+ metric: NeighborMetric = "l2",
+) -> np.ndarray:
+ """Calculate distances to k nearest neighbors between two sets of points.
+
+ For each point in matrix_a, finds the k nearest neighbors in matrix_b
+ and returns their distances.
+
+ Args:
+ matrix_a: First set of points, shape (m, d)
+ matrix_b: Second set of points, shape (n, d)
+ k: Number of nearest neighbors to find
+
+ Returns:
+ np.ndarray: Matrix of shape (m, k) containing the distances to the
+ k nearest neighbors in matrix_b for each point in matrix_a
+
+ Raises:
+ ValueError: If the inputs are empty, incompatible, or k is invalid
+ """
+ matrix_a = np.asarray(matrix_a, dtype=np.float64)
+ matrix_b = np.asarray(matrix_b, dtype=np.float64)
+ if matrix_a.ndim != 2 or matrix_b.ndim != 2:
+ raise ValueError("Input matrices must be two-dimensional")
+ if matrix_a.shape[1] != matrix_b.shape[1]:
+ raise ValueError("Matrices must have the same number of features")
+ if matrix_a.shape[0] == 0 or matrix_b.shape[0] == 0:
+ raise ValueError("Input matrices must contain at least one point")
+ if not np.all(np.isfinite(matrix_a)) or not np.all(np.isfinite(matrix_b)):
+ raise ValueError("Input matrices must contain finite values")
+ if k < 1:
+ raise ValueError("k must be greater than zero")
+
+ # Ensure k is not larger than the number of points in matrix_b
+ k = min(k, matrix_b.shape[0])
+
+ products = np.dot(matrix_a, matrix_b.T)
+ if metric == "l2":
+ a_squared = np.sum(np.square(matrix_a), axis=1, keepdims=True)
+ b_squared = np.sum(np.square(matrix_b), axis=1)
+ distances = np.sqrt(np.maximum(a_squared + b_squared - 2 * products, 0))
+ elif metric == "cosine":
+ norms = np.outer(
+ np.linalg.norm(matrix_a, axis=1),
+ np.linalg.norm(matrix_b, axis=1),
+ )
+ similarities = np.divide(
+ products,
+ norms,
+ out=np.zeros_like(products),
+ where=norms > 0,
+ )
+ distances = np.clip(1 - similarities, 0, 2)
+ elif metric == "ip":
+ distances = np.maximum(1 - products, 0)
+ else:
+ raise ValueError(f"Unsupported neighbor metric: {metric}")
+
+ # Find the k smallest distances for each point in matrix_a
+ return np.partition(distances, k - 1, axis=1)[:, :k]
diff --git a/scarf/metrics/lisi.py b/scarf/metrics/lisi.py
new file mode 100644
index 00000000..6dd8942a
--- /dev/null
+++ b/scarf/metrics/lisi.py
@@ -0,0 +1,388 @@
+from collections.abc import Iterable, Sequence
+
+import numpy as np
+import pandas as pd
+import zarr
+
+from ..utils.logging import logger
+from ..utils.progress import iter_progress
+from ._types import ZarrArray
+
+_LISI_BATCH_SIZE = 10_000
+
+
+def _effective_perplexity(perplexity: float, n_neighbors: int) -> float:
+ if not np.isfinite(perplexity) or perplexity < 1:
+ raise ValueError("Perplexity must be a finite value greater than or equal to 1")
+ if n_neighbors < 3:
+ raise ValueError("LISI requires at least three neighbors per cell")
+
+ max_perplexity = n_neighbors / 3
+ if perplexity > max_perplexity:
+ logger.warning(
+ f"Perplexity {perplexity:g} requires at least "
+ f"{int(np.ceil(3 * perplexity))} neighbors, but the graph has "
+ f"{n_neighbors}. Using perplexity {max_perplexity:g}."
+ )
+ return max_perplexity
+ return perplexity
+
+
+def _neighbor_probabilities(
+ distances: np.ndarray,
+ perplexity: float,
+ tol: float,
+ max_iter: int = 50,
+) -> np.ndarray:
+ distances = np.asarray(distances, dtype=np.float64)
+ if distances.ndim != 2 or distances.shape[1] == 0:
+ raise ValueError("Distances must be a non-empty two-dimensional matrix")
+ if not np.all(np.isfinite(distances)) or np.any(distances < 0):
+ raise ValueError("Distances must contain finite, non-negative values")
+
+ centered = distances - distances.min(axis=1, keepdims=True)
+ n_points = distances.shape[0]
+ beta = np.ones(n_points, dtype=np.float64)
+ beta_min = np.full(n_points, -np.inf, dtype=np.float64)
+ beta_max = np.full(n_points, np.inf, dtype=np.float64)
+ target_entropy = np.log(perplexity)
+
+ for _ in range(max_iter):
+ with np.errstate(over="ignore", under="ignore"):
+ weights = np.exp(-centered * beta[:, None])
+ weight_sums = weights.sum(axis=1)
+ entropy = np.log(weight_sums) + (
+ beta * np.sum(centered * weights, axis=1) / weight_sums
+ )
+ entropy_diff = entropy - target_entropy
+ active = np.abs(entropy_diff) >= tol
+ if not np.any(active):
+ break
+
+ too_diffuse = active & (entropy_diff > 0)
+ too_concentrated = active & ~too_diffuse
+
+ beta_min[too_diffuse] = beta[too_diffuse]
+ bounded_above = too_diffuse & np.isfinite(beta_max)
+ beta[bounded_above] = (beta[bounded_above] + beta_max[bounded_above]) / 2
+ beta[too_diffuse & ~np.isfinite(beta_max)] *= 2
+
+ beta_max[too_concentrated] = beta[too_concentrated]
+ bounded_below = too_concentrated & np.isfinite(beta_min)
+ beta[bounded_below] = (beta[bounded_below] + beta_min[bounded_below]) / 2
+ beta[too_concentrated & ~np.isfinite(beta_min)] /= 2
+
+ with np.errstate(over="ignore", under="ignore"):
+ probabilities = np.exp(-centered * beta[:, None])
+ probabilities /= probabilities.sum(axis=1, keepdims=True)
+ return np.asarray(probabilities)
+
+
+def _simpson_from_probabilities(
+ probabilities: np.ndarray,
+ indices: np.ndarray,
+ label_codes: np.ndarray,
+ n_categories: int,
+) -> np.ndarray:
+ indices = np.asarray(indices)
+ if indices.shape != probabilities.shape:
+ raise ValueError("Neighbor indices and probabilities must have matching shapes")
+ if not np.issubdtype(indices.dtype, np.integer):
+ raise TypeError("Neighbor indices must contain integers")
+ if np.any(indices < 0) or np.any(indices >= len(label_codes)):
+ raise IndexError("Neighbor index is outside the label array")
+
+ neighbor_codes = label_codes[indices]
+ simpson = np.zeros(probabilities.shape[0], dtype=np.float64)
+ if n_categories <= probabilities.shape[1]:
+ for category in range(n_categories):
+ category_mass = np.sum(probabilities * (neighbor_codes == category), axis=1)
+ simpson += np.square(category_mass)
+ else:
+ for neighbor in range(probabilities.shape[1]):
+ same_category = neighbor_codes == neighbor_codes[:, neighbor, None]
+ category_mass = np.sum(probabilities * same_category, axis=1)
+ simpson += probabilities[:, neighbor] * category_mass
+
+ if not np.all(np.isfinite(simpson)) or np.any(simpson <= 0):
+ raise FloatingPointError("Could not compute finite Simpson indices")
+ return np.clip(simpson, np.finfo(np.float64).tiny, 1.0)
+
+
+def compute_lisi(
+ distances: np.ndarray | ZarrArray,
+ indices: np.ndarray | ZarrArray,
+ metadata: pd.DataFrame,
+ label_colnames: Iterable[str],
+ perplexity: float = 30,
+) -> np.ndarray:
+ """Compute the Local Inverse Simpson Index (LISI) for each column in metadata.
+
+ LISI measures how well mixed different groups of cells are in the neighborhood of each cell.
+ Higher values indicate better mixing of different groups.
+
+ Args:
+ distances: Pre-computed distances between cells, stored in zarr array format
+ indices: Pre-computed nearest neighbor indices, stored in zarr array format
+ metadata: DataFrame containing categorical labels for each cell
+ label_colnames: Column names in metadata to compute LISI for
+ perplexity: Parameter controlling the effective number of neighbors (default: 30)
+
+ Returns:
+ np.ndarray: Matrix of LISI scores with shape (n_cells, n_labels)
+ Each column corresponds to LISI scores for one label column in metadata
+
+ Example:
+ For metadata with a 'batch' column having 3 categories:
+ - LISI ≈ 3: Cell has neighbors from all 3 batches (well mixed)
+ - LISI ≈ 1: Cell has neighbors from only 1 batch (poorly mixed)
+
+ References:
+ Korsunsky et al. 2019 doi: 10.1038/s41592-019-0619-0
+ """
+
+ if distances.ndim != 2 or indices.ndim != 2:
+ raise ValueError("KNN distances and indices must be two-dimensional")
+ if distances.shape != indices.shape:
+ raise ValueError("KNN distances and indices must have matching shapes")
+
+ n_cells, n_neighbors = distances.shape
+ if metadata.shape[0] != n_cells:
+ raise ValueError(
+ "Metadata rows must match the number of cells in the KNN graph"
+ )
+
+ label_cols = list(label_colnames)
+ n_labels = len(label_cols)
+ if n_labels == 0:
+ return np.empty((n_cells, 0), dtype=np.float64)
+
+ effective_perplexity = _effective_perplexity(perplexity, n_neighbors)
+ categoricals: list[pd.Categorical] = []
+ for label in label_cols:
+ categorical = pd.Categorical(metadata[label])
+ if np.any(categorical.codes < 0):
+ raise ValueError(f"Label column {label!r} contains missing values")
+ categoricals.append(categorical)
+ logger.debug(f"Preparing LISI labels from {label}")
+
+ chunk_rows = _LISI_BATCH_SIZE
+ if isinstance(distances, zarr.Array):
+ chunk_rows = min(chunk_rows, int(distances.chunks[0]))
+
+ lisi_df = np.empty((n_cells, n_labels), dtype=np.float64)
+ starts = range(0, n_cells, chunk_rows)
+ total = int(np.ceil(n_cells / chunk_rows))
+ for start in iter_progress(starts, total=total, desc="Computing LISI"):
+ end = min(start + chunk_rows, n_cells)
+ distance_block = np.asarray(distances[start:end])
+ index_block = np.asarray(indices[start:end])
+ probabilities = _neighbor_probabilities(
+ distance_block, effective_perplexity, tol=1e-5
+ )
+ for label_index, labels in enumerate(categoricals):
+ simpson = _simpson_from_probabilities(
+ probabilities,
+ index_block,
+ np.asarray(labels.codes),
+ len(labels.categories),
+ )
+ lisi_df[start:end, label_index] = 1 / simpson
+ return lisi_df
+
+
+def _lisi_knn_summary(
+ distances: np.ndarray | ZarrArray,
+ indices: np.ndarray | ZarrArray,
+ labels: Sequence[object] | np.ndarray,
+ perplexity: float | None,
+ scale: bool,
+ *,
+ invert: bool,
+ label_name: str,
+) -> float:
+ if distances.ndim != 2 or indices.ndim != 2:
+ raise ValueError("KNN distances and indices must be two-dimensional")
+ if distances.shape != indices.shape:
+ raise ValueError("KNN distances and indices must have matching shapes")
+
+ n_cells, n_neighbors = distances.shape
+ if n_neighbors < 3:
+ raise ValueError("LISI requires at least three neighbors per cell")
+
+ categorical = pd.Categorical(labels)
+ if len(categorical) != n_cells:
+ raise ValueError(f"{label_name} labels must match the number of cells")
+ if np.any(categorical.codes < 0):
+ raise ValueError(f"{label_name} labels must not contain missing values")
+
+ n_categories = int(np.unique(categorical.codes).size)
+ if n_categories < 2:
+ raise ValueError(f"{label_name} LISI requires at least two categories")
+
+ resolved_perplexity = (
+ float(np.floor(n_neighbors / 3)) if perplexity is None else perplexity
+ )
+ metadata = pd.DataFrame({"labels": categorical})
+ per_cell = compute_lisi(
+ distances,
+ indices,
+ metadata,
+ ["labels"],
+ perplexity=resolved_perplexity,
+ )[:, 0]
+ summary = float(np.nanmedian(per_cell))
+ if not scale:
+ return summary
+ if invert:
+ return float((n_categories - summary) / (n_categories - 1))
+ return float((summary - 1) / (n_categories - 1))
+
+
+def ilisi_knn(
+ distances: np.ndarray | ZarrArray,
+ indices: np.ndarray | ZarrArray,
+ batch_labels: Sequence[object] | np.ndarray,
+ perplexity: float | None = None,
+ scale: bool = True,
+) -> float:
+ """Compute the median integration LISI score from a self-free KNN graph.
+
+ When scaled, higher values indicate better batch mixing. This follows the
+ iLISI aggregation and scaling used by scIB.
+
+ The KNN arrays must exclude each cell from its own neighbor row. Scarf's
+ persisted KNN graphs satisfy this requirement. Self-including KNN arrays
+ produce a different statistic and are not adjusted automatically.
+
+ References:
+ Luecken et al. 2022 doi: 10.1038/s41592-021-01336-8
+ """
+ return _lisi_knn_summary(
+ distances,
+ indices,
+ batch_labels,
+ perplexity,
+ scale,
+ invert=False,
+ label_name="Batch",
+ )
+
+
+def clisi_knn(
+ distances: np.ndarray | ZarrArray,
+ indices: np.ndarray | ZarrArray,
+ cell_labels: Sequence[object] | np.ndarray,
+ perplexity: float | None = None,
+ scale: bool = True,
+) -> float:
+ """Compute the median cell-type LISI score from a self-free KNN graph.
+
+ When scaled, higher values indicate better conservation of cell labels.
+ This follows the cLISI aggregation and scaling used by scIB.
+
+ The KNN arrays must exclude each cell from its own neighbor row. Scarf's
+ persisted KNN graphs satisfy this requirement. Self-including KNN arrays
+ produce a different statistic and are not adjusted automatically.
+
+ References:
+ Luecken et al. 2022 doi: 10.1038/s41592-021-01336-8
+ """
+ return _lisi_knn_summary(
+ distances,
+ indices,
+ cell_labels,
+ perplexity,
+ scale,
+ invert=True,
+ label_name="Cell",
+ )
+
+
+def compute_simpson(
+ distances: np.ndarray,
+ indices: np.ndarray,
+ labels: pd.Categorical,
+ perplexity: float,
+ tol: float = 1e-5,
+) -> np.ndarray:
+ """Compute Simpson's diversity index with Gaussian kernel weighting.
+
+ This function implements the core calculation for LISI, computing a diversity score
+ based on the distribution of categories in each cell's neighborhood.
+
+ Args:
+ distances: Distance matrix between points, shape (n_neighbors, n_points)
+ indices: Index matrix for nearest neighbors, shape (n_neighbors, n_points)
+ labels: Categorical labels for each point
+ perplexity: Target perplexity for Gaussian kernel
+ tol: Convergence tolerance for perplexity calibration (default: 1e-5)
+
+ Returns:
+ np.ndarray: Array of Simpson's diversity indices, one per point
+ """
+ distances = np.asarray(distances)
+ indices = np.asarray(indices)
+ if distances.ndim != 2 or indices.ndim != 2:
+ raise ValueError("Distances and indices must be two-dimensional")
+ if distances.shape != indices.shape:
+ raise ValueError("Distances and indices must have matching shapes")
+ if np.any(labels.codes < 0):
+ raise ValueError("Labels contain missing values")
+
+ n_neighbors = distances.shape[0]
+ effective_perplexity = _effective_perplexity(perplexity, n_neighbors)
+ probabilities = _neighbor_probabilities(distances.T, effective_perplexity, tol=tol)
+ return _simpson_from_probabilities(
+ probabilities,
+ indices.T,
+ np.asarray(labels.codes),
+ len(labels.categories),
+ )
+
+
+def lisi_batch_mixing_score(
+ lisi_scores: np.ndarray,
+ batch_labels: Sequence[object] | np.ndarray,
+) -> float:
+ """Normalize mean batch LISI against the dataset's batch proportions.
+
+ Raw batch LISI depends on how many batches are present and how unevenly
+ cells are split between them, so its values are hard to compare across
+ datasets. This score rescales the mean batch LISI onto a fixed range by
+ dividing the observed neighborhood mixing by the mixing that perfectly
+ integrated data would reach. The reference point is the inverse Simpson
+ index of the global batch proportions, which is the LISI a neighborhood
+ would show if it mirrored the whole dataset.
+
+ Args:
+ lisi_scores: Per-cell batch LISI values, typically the array returned
+ by :func:`compute_lisi` for a batch label.
+ batch_labels: Batch assignment for each cell, aligned with
+ ``lisi_scores``.
+
+ Returns:
+ A value in ``[0, 1]``. Scores near 1 mean neighborhoods mix batches as
+ well as the global composition allows, and scores near 0 mean batches
+ stay separated.
+
+ Raises:
+ ValueError: If the inputs are misaligned, contain non-finite scores or
+ missing labels, or describe fewer than two batches.
+ """
+ scores = np.asarray(lisi_scores, dtype=np.float64)
+ labels = pd.Categorical(batch_labels)
+ if scores.ndim != 1 or len(scores) != len(labels):
+ raise ValueError("LISI scores and batch labels must be aligned vectors")
+ if not np.all(np.isfinite(scores)):
+ raise ValueError("LISI scores must contain finite values")
+ if np.any(labels.codes < 0):
+ raise ValueError("Batch labels must not contain missing values")
+ if len(labels.categories) < 2:
+ raise ValueError("Batch mixing requires at least two batches")
+
+ counts = np.bincount(labels.codes, minlength=len(labels.categories))
+ proportions = counts / counts.sum()
+ ideal_lisi = 1 / np.square(proportions).sum()
+ normalized = (scores.mean() - 1) / (ideal_lisi - 1)
+ return float(np.clip(normalized, 0, 1))
diff --git a/scarf/metrics/silhouette.py b/scarf/metrics/silhouette.py
new file mode 100644
index 00000000..a002482a
--- /dev/null
+++ b/scarf/metrics/silhouette.py
@@ -0,0 +1,241 @@
+from typing import TYPE_CHECKING, cast
+
+import numpy as np
+import pandas as pd
+from scipy.sparse import csr_matrix
+
+from ..neighbors.stream import AnnStream
+from ..utils.logging import logger
+from ..utils.progress import iter_progress
+from ._rows import read_matrix_rows
+from ._types import MatrixData, NeighborMetric, ZarrArray
+from .graph import (
+ calculate_knn_cluster_similarity,
+ calculate_top_k_neighbor_distances,
+ calculate_weighted_cluster_similarity,
+)
+
+if TYPE_CHECKING:
+ from ..datastore.datastore import DataStore
+
+
+def _embed_rows(
+ row_indices: np.ndarray,
+ data: MatrixData,
+ ann_obj: AnnStream,
+ *,
+ data_is_reduced: bool,
+) -> np.ndarray:
+ rows = read_matrix_rows(data, np.sort(row_indices))
+ if data_is_reduced:
+ return np.asarray(rows)
+ return np.asarray(ann_obj.reducer(rows))
+
+
+def _sample_cluster_embeddings(
+ cluster_cells: np.ndarray,
+ data: MatrixData,
+ ann_obj: AnnStream,
+ count: int,
+ rng: np.random.Generator,
+ *,
+ data_is_reduced: bool,
+) -> np.ndarray:
+ if count < 1 or count > len(cluster_cells):
+ raise ValueError("Sample count must fit within the cluster")
+ sampled = rng.choice(cluster_cells, size=count, replace=False)
+ return _embed_rows(sampled, data, ann_obj, data_is_reduced=data_is_reduced)
+
+
+def process_cluster(
+ cluster_cells: np.ndarray,
+ hvg_data: MatrixData,
+ ann_obj: AnnStream,
+ k: int,
+ *,
+ rng: np.random.Generator | None = None,
+ data_is_reduced: bool = False,
+) -> tuple[np.ndarray, np.ndarray]:
+ """Process a cluster of cells to prepare data for silhouette scoring.
+
+ Randomly splits cluster cells into two groups and applies dimensionality reduction.
+
+ Args:
+ cluster_cells: Indices of cells belonging to the cluster
+ hvg_data: Expression data for highly variable genes
+ ann_obj: Object containing dimensionality reduction method
+ k: Number of cells to sample from cluster
+
+ Returns:
+ tuple[np.ndarray, np.ndarray]: Two arrays containing reduced data for
+ different subsets of cells from the cluster
+ """
+ if k < 1 or len(cluster_cells) < 2 * k:
+ raise ValueError("A cluster must contain at least 2 * k cells")
+ if rng is None:
+ rng = np.random.default_rng(4444)
+
+ selected = rng.choice(cluster_cells, size=2 * k, replace=False)
+ data_cells = _embed_rows(
+ selected[:k],
+ hvg_data,
+ ann_obj,
+ data_is_reduced=data_is_reduced,
+ )
+ data_cells_2 = _embed_rows(
+ selected[k:],
+ hvg_data,
+ ann_obj,
+ data_is_reduced=data_is_reduced,
+ )
+ return data_cells, data_cells_2
+
+
+def silhouette_scoring(
+ ds: "DataStore",
+ ann_obj: AnnStream,
+ graph: csr_matrix | None,
+ hvg_data: MatrixData,
+ assay_type: str,
+ res_label: str,
+ *,
+ cell_key: str = "I",
+ random_seed: int = 4444,
+ sample_size: int = 11,
+ data_is_reduced: bool = False,
+ distance_metric: NeighborMetric | None = None,
+ neighbor_indices: np.ndarray | ZarrArray | None = None,
+ neighbor_distances: np.ndarray | ZarrArray | None = None,
+) -> np.ndarray | None:
+ """Compute modified silhouette scores for clusters in single-cell data.
+
+ This implementation differs from the standard silhouette score by using
+ a graph-based approach and comparing clusters to their nearest neighbors.
+
+ Args:
+ ds: DataStore object containing cell metadata
+ ann_obj: Object containing dimensionality reduction method
+ graph: Optional CSR matrix representing the weighted KNN graph
+ hvg_data: Expression data for highly variable genes
+ assay_type: Type of assay (e.g., 'RNA', 'ATAC')
+ res_label: Label for clustering resolution
+
+ Returns:
+ np.ndarray | None: Array of silhouette scores for each cluster,
+ or None if cluster labels are not found
+
+ Notes:
+ Scores are calculated using a sampling approach for efficiency.
+ NaN values indicate clusters that couldn't be scored due to size constraints.
+ """
+ if sample_size < 1:
+ raise ValueError("sample_size must be greater than zero")
+
+ if res_label in ds.cells.columns:
+ cluster_column = res_label
+ else:
+ prefix = assay_type if cell_key == "I" else f"{assay_type}_{cell_key}"
+ cluster_column = f"{prefix}_{res_label}"
+
+ try:
+ raw_clusters = ds.cells.fetch(cluster_column, key=cell_key)
+ except KeyError:
+ logger.error(f"Cluster labels not found for {cluster_column}")
+ return None
+
+ categorical = pd.Categorical(raw_clusters)
+ if np.any(categorical.codes < 0):
+ raise ValueError(f"Cluster column {cluster_column!r} contains missing values")
+ clusters = np.asarray(categorical.codes, dtype=np.int64)
+ if hvg_data.shape[0] != len(clusters):
+ raise ValueError(
+ "Embedding data and cluster labels must contain the same cells"
+ )
+
+ if graph is not None:
+ if graph.shape != (len(clusters), len(clusters)):
+ raise ValueError("KNN graph and cluster labels must contain the same cells")
+ cluster_similarity = calculate_weighted_cluster_similarity(graph, clusters)
+ elif neighbor_indices is not None and neighbor_distances is not None:
+ cluster_similarity = calculate_knn_cluster_similarity(
+ neighbor_indices,
+ neighbor_distances,
+ clusters,
+ )
+ else:
+ raise ValueError("Provide a KNN graph or neighbor indices and distances")
+ num_clusters = len(categorical.categories)
+ if num_clusters < 2:
+ logger.warning("Silhouette scoring requires at least two clusters")
+ return np.full(num_clusters, np.nan)
+
+ order = np.argsort(clusters, kind="stable")
+ counts = np.bincount(clusters, minlength=num_clusters)
+ boundaries = np.cumsum(counts)
+ starts = np.concatenate(([0], boundaries[:-1]))
+ cluster_cells = [order[start:end] for start, end in zip(starts, boundaries)]
+
+ metric = distance_metric or cast(NeighborMetric, ann_obj.annMetric)
+ if metric not in {"l2", "cosine", "ip"}:
+ raise ValueError(f"Unsupported neighbor metric: {metric}")
+
+ rng = np.random.default_rng(random_seed)
+ score: list[float] = []
+ for cluster_id, similarities in iter_progress(
+ enumerate(cluster_similarity),
+ total=len(cluster_similarity),
+ desc="Calculating silhouette scores",
+ ):
+ this_cluster_cells = cluster_cells[cluster_id]
+ k = min(sample_size, len(this_cluster_cells) // 2)
+ if k < 1:
+ logger.warning(
+ f"Cluster {categorical.categories[cluster_id]!r} has fewer than two cells"
+ )
+ score.append(np.nan)
+ continue
+
+ data_this_cells, data_this_cells_2 = process_cluster(
+ this_cluster_cells,
+ hvg_data,
+ ann_obj,
+ k,
+ rng=rng,
+ data_is_reduced=data_is_reduced,
+ )
+
+ self_dist = calculate_top_k_neighbor_distances(
+ data_this_cells,
+ data_this_cells_2,
+ min(k, len(data_this_cells_2)),
+ metric=metric,
+ ).mean()
+
+ other_similarities = similarities.copy()
+ other_similarities[cluster_id] = -np.inf
+ nearest_cluster = int(np.argmax(other_similarities))
+ nearest_cluster_cells = cluster_cells[nearest_cluster]
+ nearest_sample_size = min(k, len(nearest_cluster_cells))
+ data_nearest_cells = _sample_cluster_embeddings(
+ nearest_cluster_cells,
+ hvg_data,
+ ann_obj,
+ nearest_sample_size,
+ rng,
+ data_is_reduced=data_is_reduced,
+ )
+
+ other_dist = calculate_top_k_neighbor_distances(
+ data_this_cells,
+ data_nearest_cells,
+ min(k, len(data_nearest_cells)),
+ metric=metric,
+ ).mean()
+
+ denominator = max(self_dist, other_dist)
+ if denominator <= np.finfo(np.float64).eps:
+ score.append(0.0)
+ else:
+ score.append(float((other_dist - self_dist) / denominator))
+
+ return np.asarray(score)
diff --git a/scarf/neighbors/__init__.py b/scarf/neighbors/__init__.py
new file mode 100644
index 00000000..7149a001
--- /dev/null
+++ b/scarf/neighbors/__init__.py
@@ -0,0 +1,59 @@
+from importlib import import_module as _import_module
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from .diffusion import diffusion_operator as diffusion_operator
+ from .graph import (
+ calc_snn as calc_snn,
+ merge_graphs as merge_graphs,
+ smooth_knn_chunk as smooth_knn_chunk,
+ weight_sort_indices as weight_sort_indices,
+ )
+ from .index import (
+ fix_knn_query as fix_knn_query,
+ instantiate_knn_index as instantiate_knn_index,
+ )
+ from .integration import wnn_integration as wnn_integration
+ from .stream import AnnStream as AnnStream
+
+__all__ = [
+ "AnnStream",
+ "calc_snn",
+ "diffusion_operator",
+ "fix_knn_query",
+ "instantiate_knn_index",
+ "merge_graphs",
+ "smooth_knn_chunk",
+ "weight_sort_indices",
+ "wnn_integration",
+]
+
+_LAZY_EXPORTS = {
+ "AnnStream": (".stream", "AnnStream"),
+ "calc_snn": (".graph", "calc_snn"),
+ "diffusion_operator": (".diffusion", "diffusion_operator"),
+ "fix_knn_query": (".index", "fix_knn_query"),
+ "instantiate_knn_index": (".index", "instantiate_knn_index"),
+ "merge_graphs": (".graph", "merge_graphs"),
+ "smooth_knn_chunk": (".graph", "smooth_knn_chunk"),
+ "weight_sort_indices": (".graph", "weight_sort_indices"),
+ "wnn_integration": (".integration", "wnn_integration"),
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+
+def __getattr__(name: str) -> Any:
+ export = _LAZY_EXPORTS.get(name)
+ if export is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ module_name, attribute_name = export
+ value = getattr(_import_module(module_name, __name__), attribute_name)
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()).union(_LAZY_EXPORTS))
diff --git a/scarf/neighbors/diffusion.py b/scarf/neighbors/diffusion.py
new file mode 100644
index 00000000..7514fc11
--- /dev/null
+++ b/scarf/neighbors/diffusion.py
@@ -0,0 +1,17 @@
+import numpy as np
+from scipy.sparse import coo_matrix, csr_matrix
+
+
+def diffusion_operator(graph: csr_matrix, power: int) -> coo_matrix:
+ """Construct a powered row-normalized graph diffusion operator."""
+ inverse_degree = np.ravel(graph.sum(axis=1))
+ inverse_degree[inverse_degree != 0] = 1 / inverse_degree[inverse_degree != 0]
+ n_cells = graph.shape[0]
+ diagonal = csr_matrix(
+ (
+ inverse_degree,
+ (range(n_cells), range(n_cells)),
+ ),
+ shape=[n_cells, n_cells],
+ )
+ return diagonal.dot(graph).__pow__(power).tocoo()
diff --git a/scarf/neighbors/graph.py b/scarf/neighbors/graph.py
new file mode 100644
index 00000000..6e685302
--- /dev/null
+++ b/scarf/neighbors/graph.py
@@ -0,0 +1,207 @@
+from numbers import Real
+
+import numpy as np
+from numba import jit
+from scipy.sparse import coo_matrix, csr_matrix
+
+from ..utils.progress import iter_progress
+
+
+def smooth_knn_chunk(
+ indices: np.ndarray,
+ distances: np.ndarray,
+ *,
+ local_connectivity: float,
+ bandwidth: float,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ """Convert one KNN distance block into fuzzy graph edges."""
+ from umap.umap_ import compute_membership_strengths, smooth_knn_dist
+
+ indices_array = np.asarray(indices)
+ distance_array = np.asarray(distances, dtype=np.float32)
+ if not distance_array.flags.c_contiguous:
+ distance_array = np.ascontiguousarray(distance_array)
+ sigmas, rhos = smooth_knn_dist(
+ distance_array,
+ k=indices_array.shape[1],
+ local_connectivity=local_connectivity,
+ bandwidth=bandwidth,
+ )
+ rows, columns, values, _ = compute_membership_strengths(
+ indices_array,
+ distance_array,
+ sigmas,
+ rhos,
+ )
+ return np.asarray(rows), np.asarray(columns), np.asarray(values)
+
+
+def validate_connectivity_parameters(
+ local_connectivity: float,
+ bandwidth: float,
+) -> tuple[float, float]:
+ if isinstance(local_connectivity, bool) or not isinstance(
+ local_connectivity,
+ Real,
+ ):
+ raise TypeError("local_connectivity must be a real number")
+ if isinstance(bandwidth, bool) or not isinstance(bandwidth, Real):
+ raise TypeError("bandwidth must be a real number")
+ resolved_local_connectivity = float(local_connectivity)
+ resolved_bandwidth = float(bandwidth)
+ if not np.isfinite(resolved_local_connectivity) or resolved_local_connectivity < 0:
+ raise ValueError("local_connectivity must be finite and non-negative")
+ if not np.isfinite(resolved_bandwidth) or resolved_bandwidth <= 0:
+ raise ValueError("bandwidth must be finite and greater than zero")
+ return resolved_local_connectivity, resolved_bandwidth
+
+
+def build_connectivity_arrays(
+ indices: np.ndarray,
+ distances: np.ndarray,
+ *,
+ local_connectivity: float,
+ bandwidth: float,
+) -> tuple[np.ndarray, np.ndarray]:
+ """Build compact edge and weight arrays for a complete KNN matrix."""
+ local_connectivity, bandwidth = validate_connectivity_parameters(
+ local_connectivity,
+ bandwidth,
+ )
+ neighbor_indices = np.asarray(indices)
+ neighbor_distances = np.asarray(distances)
+ if neighbor_indices.ndim != 2 or neighbor_distances.shape != neighbor_indices.shape:
+ raise ValueError("Neighbor indices and distances must have matching matrices")
+ if not np.issubdtype(neighbor_indices.dtype, np.integer):
+ raise TypeError("Neighbor indices must be integers")
+ if np.any(neighbor_indices < 0) or np.any(
+ neighbor_indices >= neighbor_indices.shape[0]
+ ):
+ raise ValueError("Neighbor indices are outside the cell range")
+ if not np.all(np.isfinite(neighbor_distances)) or np.any(neighbor_distances < 0):
+ raise ValueError("Neighbor distances must be finite and non-negative")
+ rows, columns, values = smooth_knn_chunk(
+ neighbor_indices,
+ neighbor_distances,
+ local_connectivity=local_connectivity,
+ bandwidth=bandwidth,
+ )
+ expected = int(neighbor_indices.size)
+ if (
+ rows.shape != (expected,)
+ or columns.shape != (expected,)
+ or values.shape != (expected,)
+ ):
+ raise ValueError("UMAP membership output does not match the KNN matrix")
+ if (
+ np.any(rows < 0)
+ or np.any(columns < 0)
+ or np.any(rows > np.iinfo(np.uint32).max)
+ or np.any(columns > np.iinfo(np.uint32).max)
+ ):
+ raise ValueError("Connectivity endpoints exceed uint32 bounds")
+ edges = np.empty((expected, 2), dtype=np.uint32)
+ edges[:, 0] = rows
+ edges[:, 1] = columns
+ weights = np.asarray(values, dtype=np.float32)
+ if not np.all(np.isfinite(weights)):
+ raise ValueError("Connectivity weights must be finite")
+ positive = weights > 0
+ return edges[positive], weights[positive]
+
+
+def take_nearest_per_row(
+ weights: np.ndarray,
+ edges: np.ndarray,
+ n_cells: int,
+ use_k: int,
+) -> tuple[np.ndarray, np.ndarray]:
+ """Keep the ``use_k`` nearest stored edges of every cell.
+
+ Edges are stored row major and ordered by increasing distance inside a cell,
+ so a row's leading entries are its nearest neighbors. Row widths are counted
+ rather than assumed, because zero-weight edges are dropped before storage and
+ can leave a cell with fewer than ``k`` edges.
+ """
+ sources = np.asarray(edges[:, 0], dtype=np.intp)
+ if np.any(np.diff(sources) < 0):
+ raise ValueError("Graph edges are not grouped by source cell")
+ counts = np.bincount(sources, minlength=n_cells)
+ row_starts = np.repeat(np.cumsum(counts) - counts, counts)
+ rank_in_row = np.arange(sources.size, dtype=np.intp) - row_starts
+ keep = rank_in_row < use_k
+ return weights[keep], edges[keep]
+
+
+@jit(nopython=True)
+def calc_snn(indices: np.ndarray) -> np.ndarray:
+ """Calculate shared-neighbor fractions for a KNN index matrix."""
+ ncells, nk = indices.shape
+ snn = np.zeros((ncells, nk))
+ for i in range(ncells):
+ for j in range(nk):
+ k = indices[i][j]
+ snn[i][j] = len(set(indices[i]).intersection(set(indices[k])))
+ return np.asarray(snn / (nk - 1))
+
+
+def weight_sort_indices(
+ i: np.ndarray,
+ w: np.ndarray,
+ wn: np.ndarray,
+ n: int,
+) -> tuple[np.ndarray, np.ndarray]:
+ """Sort neighbors by adjusted weight and remove duplicates."""
+ idx = np.argsort(wn)[::-1]
+ i = i[idx]
+ w = w[idx]
+ _, unique_idx = np.unique(i, return_index=True)
+ unique_idx_arr = np.asarray(sorted(unique_idx))
+ return i[unique_idx_arr][:n], w[unique_idx_arr][:n]
+
+
+def merge_graphs(csr_mats: list[csr_matrix]) -> coo_matrix:
+ """Merge regular graphs using edge weights and shared neighbors."""
+ if not csr_mats:
+ raise ValueError("At least one graph is required")
+ if len({matrix.shape for matrix in csr_mats}) != 1:
+ raise ValueError("ERROR: All graphs do not have the same shape.")
+ row_counts = [np.diff(matrix.indptr) for matrix in csr_mats]
+ if any(
+ len(counts) == 0 or not np.all(counts == counts[0]) for counts in row_counts
+ ):
+ raise ValueError("ERROR: All graphs must have a regular neighbor count")
+ neighbor_counts = {int(counts[0]) for counts in row_counts}
+ if len(neighbor_counts) != 1:
+ raise ValueError("ERROR: All graphs do not have the same number of edges")
+ nk = neighbor_counts.pop()
+ if nk < 2:
+ raise ValueError("SNN integration requires at least two neighbors per cell")
+ snns = []
+ for matrix in iter_progress(csr_mats, desc="Identifying SNNs in graphs"):
+ snns.append(calc_snn(matrix.indices.reshape((matrix.shape[0], nk))))
+ columns: list[int] = []
+ data: list[float] = []
+ for row_idx in iter_progress(
+ range(csr_mats[0].shape[0]),
+ desc="Merging graph edges",
+ ):
+ merged_indices = np.hstack([matrix[row_idx].indices for matrix in csr_mats])
+ adjusted_weights = np.hstack(
+ [
+ matrix[row_idx].data + snns[index][row_idx]
+ for index, matrix in enumerate(csr_mats)
+ ]
+ )
+ merged_weights = np.hstack([matrix[row_idx].data for matrix in csr_mats])
+ merged_indices, merged_weights = weight_sort_indices(
+ merged_indices,
+ merged_weights,
+ adjusted_weights,
+ nk,
+ )
+ columns.extend(merged_indices)
+ data.extend(merged_weights)
+ shape = csr_mats[0].shape
+ rows = np.repeat(range(shape[0]), nk)
+ return coo_matrix((data, (rows, columns)), shape=shape)
diff --git a/scarf/neighbors/index.py b/scarf/neighbors/index.py
new file mode 100644
index 00000000..46b3c852
--- /dev/null
+++ b/scarf/neighbors/index.py
@@ -0,0 +1,61 @@
+from typing import Any
+
+import numpy as np
+
+
+def instantiate_knn_index(
+ space: str,
+ dim: int,
+ max_elements: int,
+ ef_construction: int,
+ M: int,
+ random_seed: int,
+ ef: int,
+ num_threads: int,
+) -> Any:
+ """Create and configure an hnswlib KNN index."""
+ import hnswlib
+
+ ann_idx = hnswlib.Index(space=space, dim=dim)
+ ann_idx.init_index(
+ max_elements=max_elements,
+ ef_construction=ef_construction,
+ M=M,
+ random_seed=random_seed,
+ )
+ ann_idx.set_ef(ef)
+ ann_idx.set_num_threads(num_threads)
+ return ann_idx
+
+
+def fix_knn_query(
+ indices: np.ndarray,
+ distances: np.ndarray,
+ ref_idx: np.ndarray,
+) -> tuple[np.ndarray, np.ndarray, int]:
+ """Remove self-neighbor entries from KNN query results."""
+ neighbor_indices = np.asarray(indices)
+ neighbor_distances = np.asarray(distances)
+ references = np.asarray(ref_idx)
+ if (
+ neighbor_indices.ndim != 2
+ or neighbor_distances.shape != neighbor_indices.shape
+ or references.shape != (neighbor_indices.shape[0],)
+ or neighbor_indices.shape[1] < 2
+ ):
+ raise ValueError("KNN query arrays have incompatible shapes")
+ matches = neighbor_indices == references[:, np.newaxis]
+ has_self = matches.any(axis=1)
+ self_positions = matches.argmax(axis=1)
+ drop_positions = np.where(
+ has_self,
+ self_positions,
+ neighbor_indices.shape[1] - 1,
+ )
+ columns = np.arange(neighbor_indices.shape[1])[np.newaxis, :]
+ keep = columns != drop_positions[:, np.newaxis]
+ output_shape = (neighbor_indices.shape[0], neighbor_indices.shape[1] - 1)
+ fixed_indices = neighbor_indices[keep].reshape(output_shape)
+ fixed_distances = neighbor_distances[keep].reshape(output_shape)
+ missed_self_hits = int(np.count_nonzero(~has_self))
+ return fixed_indices, fixed_distances, missed_self_hits
diff --git a/scarf/neighbors/integration.py b/scarf/neighbors/integration.py
new file mode 100644
index 00000000..b375ca58
--- /dev/null
+++ b/scarf/neighbors/integration.py
@@ -0,0 +1,268 @@
+import numpy as np
+from scipy.sparse import coo_matrix
+
+from ..utils.logging import logger
+from ..utils.progress import iter_progress
+
+
+def _validate_neighbor_indices(
+ name: str,
+ values: np.ndarray,
+ expected_cells: int | None = None,
+) -> np.ndarray:
+ indices = np.asarray(values)
+ if indices.ndim != 2 or indices.shape[0] == 0:
+ raise ValueError(f"WNN neighbors for {name} must be a non-empty matrix")
+ if indices.shape[1] < 2:
+ raise ValueError(
+ f"WNN neighbors for {name} must contain at least two neighbors per cell"
+ )
+ if expected_cells is not None and indices.shape[0] != expected_cells:
+ raise ValueError("WNN neighbor matrices must have the same number of cells")
+ if not np.issubdtype(indices.dtype, np.integer):
+ raise TypeError(f"WNN neighbors for {name} must contain integer indices")
+
+ n_cells = indices.shape[0]
+ if int(indices.min()) < 0 or int(indices.max()) >= n_cells:
+ raise ValueError(f"WNN neighbors for {name} contain indices outside cell range")
+ for start in range(0, n_cells, 100_000):
+ stop = min(start + 100_000, n_cells)
+ block = indices[start:stop]
+ rows = np.arange(start, stop)[:, np.newaxis]
+ if np.any(block == rows):
+ raise ValueError(f"WNN neighbors for {name} must exclude self")
+ ordered = np.sort(block, axis=1)
+ if np.any(ordered[:, 1:] == ordered[:, :-1]):
+ raise ValueError(f"WNN neighbors for {name} must be unique within each row")
+ return indices
+
+
+def _validate_embedding(
+ name: str,
+ values: np.ndarray,
+ n_cells: int,
+) -> np.ndarray:
+ embedding = np.asarray(values)
+ if embedding.ndim != 2 or embedding.shape[1] == 0:
+ raise ValueError(f"WNN embedding for {name} must be a non-empty matrix")
+ if embedding.shape[0] != n_cells:
+ raise ValueError(f"WNN embedding for {name} must have one row per graph cell")
+ if not np.issubdtype(embedding.dtype, np.number) or np.issubdtype(
+ embedding.dtype,
+ np.complexfloating,
+ ):
+ raise TypeError(f"WNN embedding for {name} must contain real numeric values")
+ if not np.issubdtype(embedding.dtype, np.floating):
+ embedding = embedding.astype(np.float64)
+ for start in range(0, n_cells, 100_000):
+ if not np.all(np.isfinite(embedding[start : start + 100_000])):
+ raise ValueError(f"WNN embedding for {name} contains non-finite values")
+ return embedding
+
+
+def _inverse_row_norms(values: np.ndarray) -> np.ndarray:
+ norms = np.sqrt(
+ np.einsum(
+ "ij,ij->i",
+ values,
+ values,
+ dtype=np.float64,
+ optimize=True,
+ )
+ )
+ inverse = np.zeros(norms.shape, dtype=np.float64)
+ np.divide(1.0, norms, out=inverse, where=norms > 0)
+ return inverse
+
+
+def _kernel_affinity(
+ distances: np.ndarray,
+ nearest_distance: float,
+ bandwidth: float,
+) -> np.ndarray:
+ adjusted = np.maximum(distances - nearest_distance, 0.0)
+ # A bandwidth this close to zero is indistinguishable from rounding noise in
+ # the distance reduction, so the tolerance tracks the local distance scale
+ # to keep the result invariant to a rescaling of the embedding.
+ tolerance = 8.0 * np.finfo(np.float64).eps * nearest_distance
+ if bandwidth <= tolerance:
+ return (adjusted <= tolerance).astype(np.float64)
+ return np.exp(-(adjusted / bandwidth))
+
+
+def _prediction_affinity(
+ point: np.ndarray,
+ estimate: np.ndarray,
+ nearest_distance: float,
+ bandwidth: float,
+) -> float:
+ estimate_distance = float(np.linalg.norm(point - estimate))
+ return float(
+ _kernel_affinity(
+ np.asarray([estimate_distance]),
+ nearest_distance,
+ bandwidth,
+ )[0]
+ )
+
+
+def wnn_integration(
+ name1: str,
+ indices1: np.ndarray,
+ ld1: np.ndarray,
+ name2: str,
+ indices2: np.ndarray,
+ ld2: np.ndarray,
+ n_threads: int,
+ *,
+ l2_normalize: bool = True,
+) -> tuple[coo_matrix, np.ndarray]:
+ """Build a Hao-inspired WNN graph and per-cell modality weights.
+
+ Candidates are the union of two self-free neighbour-index rows. Each
+ modality uses its own nearest and k-th-neighbour distances to convert
+ distances into affinities. Rows are L2-normalized during scoring by default.
+ The returned COO graph stores blended affinity as float32 edge weights, and
+ the second array stores two float32 modality weights per cell.
+
+ This bounded candidate pool and simple bandwidth differ from Seurat's
+ default wider search and SNN-far bandwidth. At 100,000 cells with 20
+ neighbours per modality the path measures 137 to 146 microseconds per cell
+ single-threaded, and peak memory grows by about 0.9 MB per 1,000 cells.
+ """
+ if not isinstance(l2_normalize, bool | np.bool_):
+ raise TypeError("l2_normalize must be a boolean")
+
+ neighbor_indices1 = _validate_neighbor_indices(name1, indices1)
+ n_cells, k1 = neighbor_indices1.shape
+ neighbor_indices2 = _validate_neighbor_indices(name2, indices2, n_cells)
+ k2 = neighbor_indices2.shape[1]
+ ld1 = _validate_embedding(name1, ld1, n_cells)
+ ld2 = _validate_embedding(name2, ld2, n_cells)
+
+ nk = min(k1, k2)
+ if k1 != k2:
+ logger.warning(
+ f"WNN graphs have different neighbor counts ({name1}: {k1}, "
+ f"{name2}: {k2}). The integrated graph will retain {nk} "
+ "neighbors per cell."
+ )
+
+ from threadpoolctl import threadpool_limits
+
+ inverse_norms1 = _inverse_row_norms(ld1) if l2_normalize else None
+ inverse_norms2 = _inverse_row_norms(ld2) if l2_normalize else None
+ index_dtype = np.uint32 if n_cells < 2**32 else np.uint64
+ output_size = n_cells * nk
+ merged_columns = np.empty(output_size, dtype=index_dtype)
+ merged_data = np.empty(output_size, dtype=np.float32)
+ modality_weights = np.empty((n_cells, 2), dtype=np.float32)
+
+ with threadpool_limits(limits=n_threads):
+ for cell_idx in iter_progress(
+ range(n_cells),
+ desc="Building WNN graph",
+ total=n_cells,
+ ):
+ neighbors1 = neighbor_indices1[cell_idx]
+ neighbors2 = neighbor_indices2[cell_idx]
+ mixed_k = np.union1d(neighbors1, neighbors2)
+
+ candidate_data1 = ld1[mixed_k]
+ candidate_data2 = ld2[mixed_k]
+ point1 = ld1[cell_idx]
+ point2 = ld2[cell_idx]
+ if inverse_norms1 is not None:
+ candidate_data1 = candidate_data1 * inverse_norms1[mixed_k, np.newaxis]
+ point1 = point1 * inverse_norms1[cell_idx]
+ if inverse_norms2 is not None:
+ candidate_data2 = candidate_data2 * inverse_norms2[mixed_k, np.newaxis]
+ point2 = point2 * inverse_norms2[cell_idx]
+
+ distances1 = np.linalg.norm(point1 - candidate_data1, axis=1)
+ distances2 = np.linalg.norm(point2 - candidate_data2, axis=1)
+ positions1 = np.searchsorted(mixed_k, neighbors1)
+ positions2 = np.searchsorted(mixed_k, neighbors2)
+ self_distances1 = distances1[positions1]
+ self_distances2 = distances2[positions2]
+ ranked_distances1 = np.sort(self_distances1)
+ ranked_distances2 = np.sort(self_distances2)
+ nearest_distance1 = float(ranked_distances1[0])
+ nearest_distance2 = float(ranked_distances2[0])
+ bandwidth1 = float(ranked_distances1[-1] - nearest_distance1)
+ bandwidth2 = float(ranked_distances2[-1] - nearest_distance2)
+
+ theta_self1 = _prediction_affinity(
+ point1,
+ candidate_data1[positions1].mean(axis=0),
+ nearest_distance1,
+ bandwidth1,
+ )
+ theta_cross1 = _prediction_affinity(
+ point1,
+ candidate_data1[positions2].mean(axis=0),
+ nearest_distance1,
+ bandwidth1,
+ )
+ theta_self2 = _prediction_affinity(
+ point2,
+ candidate_data2[positions2].mean(axis=0),
+ nearest_distance2,
+ bandwidth2,
+ )
+ theta_cross2 = _prediction_affinity(
+ point2,
+ candidate_data2[positions1].mean(axis=0),
+ nearest_distance2,
+ bandwidth2,
+ )
+ score1 = float(
+ np.clip(
+ theta_self1 / (theta_cross1 + 1e-4),
+ 0,
+ 200,
+ )
+ )
+ score2 = float(
+ np.clip(
+ theta_self2 / (theta_cross2 + 1e-4),
+ 0,
+ 200,
+ )
+ )
+
+ max_score = max(score1, score2)
+ exp1 = np.exp(score1 - max_score)
+ exp2 = np.exp(score2 - max_score)
+ normalizer = exp1 + exp2
+ weight1 = float(exp1 / normalizer)
+ weight2 = float(exp2 / normalizer)
+ modality_weights[cell_idx] = (weight1, weight2)
+
+ combined_affinity = weight1 * _kernel_affinity(
+ distances1,
+ nearest_distance1,
+ bandwidth1,
+ ) + weight2 * _kernel_affinity(
+ distances2,
+ nearest_distance2,
+ bandwidth2,
+ )
+ selected = np.lexsort((mixed_k, -combined_affinity))[:nk]
+ output_slice = slice(cell_idx * nk, (cell_idx + 1) * nk)
+ merged_columns[output_slice] = mixed_k[selected]
+ merged_data[output_slice] = np.maximum(
+ np.clip(combined_affinity[selected], 0.0, 1.0),
+ np.finfo(np.float32).tiny,
+ )
+
+ if not np.all(np.isfinite(merged_data)):
+ raise FloatingPointError("WNN integration produced non-finite graph weights")
+ if not np.all(np.isfinite(modality_weights)):
+ raise FloatingPointError("WNN integration produced non-finite modality weights")
+ rows = np.repeat(np.arange(n_cells, dtype=index_dtype), nk)
+ graph = coo_matrix(
+ (merged_data, (rows, merged_columns)),
+ shape=(n_cells, n_cells),
+ )
+ return graph, modality_weights
diff --git a/scarf/neighbors/stages.py b/scarf/neighbors/stages.py
new file mode 100644
index 00000000..94e66dd0
--- /dev/null
+++ b/scarf/neighbors/stages.py
@@ -0,0 +1,724 @@
+import math
+import operator
+import time
+from collections.abc import Callable, Generator, Iterator, Mapping
+from dataclasses import dataclass
+from typing import Any, Protocol
+
+import numpy as np
+import pandas as pd
+from threadpoolctl import threadpool_info, threadpool_limits
+
+from ..embeddings.harmony import HarmonyResult, fit_harmony
+from ..matrix import ChunkedArray
+from ..utils.logging import logger
+from ..utils.process import process_rss_mb
+from .index import fix_knn_query, instantiate_knn_index
+
+
+class ReductionTransform:
+ def __init__(
+ self,
+ *,
+ data: ChunkedArray,
+ method: str,
+ dims: int | None,
+ loadings: np.ndarray | None,
+ use_for_pca: np.ndarray,
+ mu: np.ndarray,
+ sigma: np.ndarray,
+ batch_size: int,
+ nthreads: int,
+ rand_state: int,
+ disable_scaling: bool,
+ lsi_skip_first: bool,
+ lsi_params: dict[str, Any],
+ ) -> None:
+ self.data = data
+ self.method = method
+ self.dims = dims
+ self.loadings = loadings
+ self.mu = mu
+ self.sigma = sigma
+ self.batch_size = batch_size
+ self.nthreads = nthreads
+ self.rand_state = rand_state
+ self.feature_scaling = not disable_scaling
+ self.pca: Any | None = None
+ disable_reduction = self.dims is not None and self.dims < 1
+
+ if self.method == "pca":
+ if self.loadings is None or len(self.loadings) == 0:
+ if len(use_for_pca) != self.data.shape[0]:
+ raise ValueError(
+ "ERROR: `use_for_pca` does not have sample length as nCells"
+ )
+ if not disable_reduction:
+ with threadpool_limits(limits=self.nthreads):
+ self._fit_pca(disable_scaling, use_for_pca)
+ else:
+ self.dims = self.loadings.shape[1]
+ self._transform = self._pca_transform(
+ disable_scaling,
+ disable_reduction,
+ )
+ elif self.method == "lsi":
+ if self.loadings is None or len(self.loadings) == 0:
+ if not disable_reduction:
+ with threadpool_limits(limits=self.nthreads):
+ self._fit_lsi(lsi_skip_first, lsi_params)
+ else:
+ self.dims = self.loadings.shape[1]
+ self._transform = self._linear_transform(disable_reduction)
+ elif self.method == "custom":
+ if self.loadings is None or len(self.loadings) == 0:
+ logger.warning("No loadings provided for manual dimension reduction")
+ else:
+ self.dims = self.loadings.shape[1]
+ self._transform = self._linear_transform(disable_reduction)
+ else:
+ raise ValueError(f"ERROR: Unknown reduction method: {self.method}")
+
+ def _pca_transform(
+ self,
+ disable_scaling: bool,
+ disable_reduction: bool,
+ ) -> Callable[[np.ndarray], np.ndarray]:
+ if disable_scaling:
+ if disable_reduction:
+ return lambda values: values
+ assert self.loadings is not None
+ loadings = self.loadings
+ return lambda values: np.asarray(values.dot(loadings))
+ if disable_reduction:
+ return self.transform_z
+ assert self.loadings is not None
+ loadings = self.loadings
+ return lambda values: np.asarray(self.transform_z(values).dot(loadings))
+
+ def _linear_transform(
+ self,
+ disable_reduction: bool,
+ ) -> Callable[[np.ndarray], np.ndarray]:
+ if disable_reduction:
+ return lambda values: values
+ assert self.loadings is not None
+ loadings = self.loadings
+ return lambda values: np.asarray(values.dot(loadings))
+
+ def transform(self, values: np.ndarray) -> np.ndarray:
+ return np.asarray(self._transform(values))
+
+ def transform_z(self, values: np.ndarray) -> np.ndarray:
+ return np.asarray((values - self.mu) / self.sigma)
+
+ def _fit_pca(
+ self,
+ disable_scaling: bool,
+ use_for_pca: np.ndarray,
+ ) -> None:
+ from ..embeddings.reduction import fit_incremental_pca
+
+ assert self.dims is not None
+ scale = self.transform_z if not disable_scaling else None
+ self.loadings, self.pca = fit_incremental_pca(
+ self.data,
+ dims=self.dims,
+ batch_size=self.batch_size,
+ use_for_pca=use_for_pca,
+ scale=scale,
+ nthreads=self.nthreads,
+ )
+
+ def _fit_lsi(
+ self,
+ lsi_skip_first: bool,
+ lsi_params: dict[str, Any],
+ ) -> None:
+ from ..embeddings.reduction import fit_lsi
+
+ assert self.dims is not None
+ self.loadings = fit_lsi(
+ self.data,
+ dims=self.dims,
+ skip_first=lsi_skip_first,
+ params=lsi_params,
+ random_state=self.rand_state,
+ nthreads=self.nthreads,
+ )
+
+
+class LazyTransformStream:
+ def __init__(
+ self,
+ *,
+ data: ChunkedArray,
+ transform: Callable[[np.ndarray], np.ndarray],
+ nthreads: int,
+ batch_size: int,
+ ) -> None:
+ self.data = data
+ self.transform = transform
+ self.nthreads = nthreads
+ self.batch_size = batch_size
+
+ def iter_raw(self, message: str = "") -> Iterator[np.ndarray]:
+ yield from self.data.stream_blocks(nthreads=self.nthreads, msg=message)
+
+ def iter_transformed(self, message: str = "") -> Iterator[np.ndarray]:
+ for block in self.iter_raw(message):
+ yield self.transform(block)
+
+ def iter_coordinate_blocks(self, message: str) -> Iterator[np.ndarray]:
+ yield from self.iter_transformed(message)
+
+
+class BatchCorrectionStage:
+ def __init__(
+ self,
+ *,
+ stream: "CoordinateSource",
+ n_cells: int,
+ dims: int,
+ batch_size: int,
+ batches: pd.DataFrame | None,
+ parameters: Mapping[str, Any],
+ corrected_data: ChunkedArray | None,
+ nthreads: int,
+ ) -> None:
+ self.stream = stream
+ self.n_cells = int(n_cells)
+ self.dims = int(dims)
+ self.batch_size = int(batch_size)
+ self.batches = batches
+ self.parameters = dict(parameters)
+ self.corrected_data = corrected_data
+ self.nthreads = nthreads
+ self.result: HarmonyResult | None = None
+
+ def ensure_corrected(self) -> ChunkedArray:
+ if self.corrected_data is not None:
+ return self.corrected_data
+ if self.batches is None:
+ raise ValueError("Harmony requires batch metadata")
+ uncorrected = np.empty(
+ (self.dims, self.n_cells),
+ dtype=np.float64,
+ )
+ start = 0
+ for block in self.stream.iter_coordinate_blocks(
+ "Loading uncorrected latent dimensions",
+ ):
+ values = np.asarray(block)
+ stop = start + int(values.shape[0])
+ if values.shape != (stop - start, self.dims) or stop > self.n_cells:
+ raise ValueError("Coordinate block has an invalid shape")
+ uncorrected[:, start:stop] = values.T
+ start = stop
+ if start != self.n_cells:
+ raise ValueError(
+ f"Coordinate source contains {start} rows, expected {self.n_cells}"
+ )
+ with threadpool_limits(limits=self.nthreads):
+ self.result = fit_harmony(
+ uncorrected,
+ self.batches,
+ **self.parameters,
+ )
+ self.corrected_data = ChunkedArray.from_numpy(
+ self.result.corrected.T,
+ block_size=self.batch_size,
+ nthreads=self.nthreads,
+ )
+ return self.corrected_data
+
+
+class CoordinateSource(Protocol):
+ def iter_coordinate_blocks(self, message: str) -> Iterator[np.ndarray]: ...
+
+
+class ChunkedCoordinateStream:
+ def __init__(self, data: ChunkedArray, nthreads: int) -> None:
+ self.data = data
+ self.nthreads = nthreads
+
+ def iter_coordinate_blocks(self, message: str) -> Iterator[np.ndarray]:
+ yield from self.data.stream_blocks(
+ nthreads=self.nthreads,
+ msg=message,
+ )
+
+
+class AnnIndexStage:
+ @staticmethod
+ def configure(index: Any, *, ef: int, threads: int) -> Any:
+ index.set_ef(ef)
+ index.set_num_threads(threads)
+ return index
+
+ @staticmethod
+ def create(
+ *,
+ metric: str,
+ dims: int,
+ n_cells: int,
+ ef_construction: int,
+ ef: int,
+ m: int,
+ rand_state: int,
+ ann_threads: int,
+ ) -> Any:
+ return instantiate_knn_index(
+ metric,
+ dims,
+ n_cells,
+ ef_construction,
+ m,
+ rand_state,
+ ef,
+ ann_threads,
+ )
+
+ @staticmethod
+ def populate(index: Any, coordinates: CoordinateSource) -> Any:
+ for block in coordinates.iter_coordinate_blocks("Fitting ANN"):
+ index.add_items(block)
+ return index
+
+ @classmethod
+ def fit(
+ cls,
+ *,
+ coordinates: CoordinateSource,
+ metric: str,
+ dims: int,
+ n_cells: int,
+ ef_construction: int,
+ ef: int,
+ m: int,
+ rand_state: int,
+ ann_threads: int,
+ ) -> Any:
+ index = cls.create(
+ metric=metric,
+ dims=dims,
+ n_cells=n_cells,
+ ef_construction=ef_construction,
+ ef=ef,
+ m=m,
+ rand_state=rand_state,
+ ann_threads=ann_threads,
+ )
+ return cls.populate(index, coordinates)
+
+
+class NeighborQueryStage:
+ def __init__(self, index: Any, k: int, metric: str) -> None:
+ self.index = index
+ self.k = k
+ self.metric = metric
+
+ def _metric_distances(self, distances: np.ndarray) -> np.ndarray:
+ values = np.asarray(distances)
+ if not np.all(np.isfinite(values)):
+ raise ValueError("ANN metric produced non-finite neighbor distances")
+ if self.metric in {"l2", "cosine"}:
+ if np.any(values < -1e-6):
+ raise ValueError("ANN metric produced negative neighbor distances")
+ np.maximum(values, 0, out=values)
+ if self.metric == "l2":
+ np.sqrt(values, out=values)
+ if self.metric != "ip" and np.any(values < 0):
+ raise ValueError("ANN metric produced negative neighbor distances")
+ return values
+
+ def query(
+ self,
+ values: np.ndarray,
+ *,
+ k: int | None = None,
+ self_indices: np.ndarray | None = None,
+ ) -> tuple[np.ndarray, np.ndarray] | tuple[np.ndarray, np.ndarray, int]:
+ use_k = self.k if k is None else k
+ if self_indices is None:
+ indices, distances = self.index.knn_query(values, k=use_k)
+ return np.asarray(indices), self._metric_distances(distances)
+ indices, distances = self.index.knn_query(values, k=use_k + 1)
+ fixed_indices, fixed_distances, missed = fix_knn_query(
+ indices,
+ distances,
+ self_indices,
+ )
+ return (
+ fixed_indices,
+ self._metric_distances(fixed_distances),
+ missed,
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class KMeansInitialization:
+ model: Any | None
+ labels: np.ndarray
+
+
+class KMeansInitializationStage:
+ @staticmethod
+ def fit(
+ *,
+ stream: CoordinateSource,
+ n_rows: int | None = None,
+ batch_size: int | None = None,
+ n_clusters: int,
+ rand_state: int,
+ nthreads: int,
+ enabled: bool,
+ kmeans_sampling: float = 0.1,
+ kmeans_batch_size: int = 10_000,
+ ) -> KMeansInitialization:
+ if n_rows is None:
+ data = getattr(stream, "data", None)
+ if data is None:
+ raise ValueError("n_rows is required for this coordinate source")
+ n_rows = int(data.shape[0])
+ if batch_size is None:
+ raw_batch_size = getattr(stream, "batch_size", None)
+ if raw_batch_size is None:
+ raise ValueError("batch_size is required for this coordinate source")
+ batch_size = int(raw_batch_size)
+ if not enabled:
+ return KMeansInitialization(
+ model=None,
+ labels=np.repeat(-1, n_rows),
+ )
+ if n_rows == 0:
+ raise ValueError("K-means initialization requires at least one row")
+ if isinstance(kmeans_sampling, bool):
+ raise TypeError("kmeans_sampling must be a number")
+ try:
+ resolved_kmeans_sampling = float(kmeans_sampling)
+ except (TypeError, ValueError):
+ raise TypeError("kmeans_sampling must be a number") from None
+ if (
+ not math.isfinite(resolved_kmeans_sampling)
+ or not 0 < resolved_kmeans_sampling <= 1
+ ):
+ raise ValueError("kmeans_sampling must be greater than 0 and at most 1")
+ if isinstance(kmeans_batch_size, bool):
+ raise TypeError("kmeans_batch_size must be a positive integer")
+ try:
+ requested_kmeans_batch_size = operator.index(kmeans_batch_size)
+ except TypeError:
+ raise TypeError("kmeans_batch_size must be a positive integer") from None
+ if requested_kmeans_batch_size < 1:
+ raise ValueError("kmeans_batch_size must be a positive integer")
+ from sklearn.cluster import MiniBatchKMeans, kmeans_plusplus
+ from sklearn.utils.random import sample_without_replacement
+
+ effective_clusters = min(
+ max(n_clusters, 2),
+ n_rows,
+ )
+ if effective_clusters < 2:
+ raise ValueError("K-means initialization requires at least two rows")
+ effective_kmeans_batch_size = min(
+ n_rows,
+ max(requested_kmeans_batch_size, effective_clusters),
+ )
+ init_size = min(
+ n_rows,
+ max(effective_clusters, math.ceil(n_rows * resolved_kmeans_sampling)),
+ )
+
+ def make_model(*, init: str | np.ndarray = "k-means++") -> Any:
+ return MiniBatchKMeans(
+ n_clusters=effective_clusters,
+ random_state=rand_state,
+ batch_size=effective_kmeans_batch_size,
+ init_size=init_size,
+ init=init,
+ n_init=1,
+ )
+
+ def timed_blocks(
+ message: str,
+ ) -> Generator[tuple[int, np.ndarray, float, float]]:
+ blocks = iter(stream.iter_coordinate_blocks(message))
+ block_idx = 0
+ while True:
+ wall_started = time.perf_counter()
+ cpu_started = time.process_time()
+ try:
+ block = next(blocks)
+ except StopIteration:
+ return
+ block_idx += 1
+ yield (
+ block_idx,
+ block,
+ time.perf_counter() - wall_started,
+ time.process_time() - cpu_started,
+ )
+
+ with threadpool_limits(limits=nthreads):
+ pools = sorted(
+ (
+ str(pool.get("user_api")),
+ str(pool.get("internal_api")),
+ int(pool.get("num_threads", 0)),
+ )
+ for pool in threadpool_info()
+ )
+ logger.debug(
+ f"KMeans initialization plan: rows={n_rows} "
+ f"readBatchSize={batch_size} clusters={effective_clusters} "
+ f"samplingFraction={resolved_kmeans_sampling:.4f} "
+ f"initSize={init_size} "
+ f"kmeansBatchSize={effective_kmeans_batch_size} "
+ f"requestedThreads={nthreads} threadPools={pools}"
+ )
+ coordinate_blocks = timed_blocks("Loading kmeans coordinates")
+ try:
+ first_block = next(coordinate_blocks)
+ except StopIteration:
+ raise ValueError(
+ "K-means initialization coordinate source is empty"
+ ) from None
+ block_idx, block, read_seconds, read_cpu_seconds = first_block
+ if block.ndim != 2:
+ raise ValueError("K-means coordinate blocks must be two-dimensional")
+ if block.shape[0] == n_rows:
+ try:
+ next(coordinate_blocks)
+ except StopIteration:
+ pass
+ else:
+ coordinate_blocks.close()
+ raise ValueError(
+ "K-means coordinate source yielded rows after a complete block"
+ )
+ model = make_model()
+ compute_started = time.perf_counter()
+ compute_cpu_started = time.process_time()
+ model.fit(block)
+ compute_seconds = time.perf_counter() - compute_started
+ compute_cpu_seconds = time.process_time() - compute_cpu_started
+ logger.debug(
+ f"KMeans minibatch fit block {block_idx}: rows={block.shape[0]} "
+ f"read={read_seconds:.3f}s readCpu={read_cpu_seconds:.3f}s "
+ f"readCores={read_cpu_seconds / max(read_seconds, 1e-12):.2f} "
+ f"compute={compute_seconds:.3f}s "
+ f"computeCpu={compute_cpu_seconds:.3f}s "
+ f"computeCores="
+ f"{compute_cpu_seconds / max(compute_seconds, 1e-12):.2f} "
+ f"steps={model.n_steps_} iterations={model.n_iter_} "
+ f"inertiaPerRow={float(model.inertia_) / n_rows:.6f} "
+ f"rss={process_rss_mb():.0f} MiB"
+ )
+ return KMeansInitialization(
+ model=model,
+ labels=np.asarray(model.labels_, dtype=np.uint32),
+ )
+
+ coordinate_dims = int(block.shape[1])
+ coordinate_dtype = block.dtype
+ sample_indices = np.sort(
+ sample_without_replacement(
+ n_rows,
+ init_size,
+ method="reservoir_sampling",
+ random_state=rand_state,
+ )
+ )
+ sample = np.empty((init_size, coordinate_dims), dtype=block.dtype)
+ rows_seen = 0
+ sample_blocks = 0
+ sample_read_seconds = 0.0
+ sample_read_cpu_seconds = 0.0
+ sample_started = time.perf_counter()
+ sample_cpu_started = time.process_time()
+ while True:
+ if (
+ block.ndim != 2
+ or int(block.shape[1]) != coordinate_dims
+ or block.dtype != coordinate_dtype
+ ):
+ raise ValueError("K-means coordinate block dimensions changed")
+ block_rows = int(block.shape[0])
+ block_stop = rows_seen + block_rows
+ if block_stop > n_rows:
+ raise ValueError("K-means coordinate source has too many rows")
+ sample_start = int(
+ np.searchsorted(sample_indices, rows_seen, side="left")
+ )
+ sample_stop = int(
+ np.searchsorted(sample_indices, block_stop, side="left")
+ )
+ local_indices = sample_indices[sample_start:sample_stop] - rows_seen
+ sample[sample_start:sample_stop] = block[local_indices]
+ rows_seen = block_stop
+ sample_blocks += 1
+ sample_read_seconds += read_seconds
+ sample_read_cpu_seconds += read_cpu_seconds
+ try:
+ block_idx, block, read_seconds, read_cpu_seconds = next(
+ coordinate_blocks
+ )
+ except StopIteration:
+ break
+ if rows_seen != n_rows:
+ raise ValueError(
+ f"K-means coordinate source contains {rows_seen} rows, "
+ f"expected {n_rows}"
+ )
+ sample_seconds = time.perf_counter() - sample_started
+ sample_cpu_seconds = time.process_time() - sample_cpu_started
+ logger.debug(
+ f"KMeans sampling pass: blocks={sample_blocks} rows={rows_seen} "
+ f"sampleRows={init_size} wall={sample_seconds:.3f}s "
+ f"cpu={sample_cpu_seconds:.3f}s "
+ f"effectiveCores="
+ f"{sample_cpu_seconds / max(sample_seconds, 1e-12):.2f} "
+ f"read={sample_read_seconds:.3f}s "
+ f"readCpu={sample_read_cpu_seconds:.3f}s "
+ f"rss={process_rss_mb():.0f} MiB"
+ )
+
+ seed_started = time.perf_counter()
+ seed_cpu_started = time.process_time()
+ initial_centers, _ = kmeans_plusplus(
+ sample,
+ n_clusters=effective_clusters,
+ random_state=rand_state,
+ )
+ seed_seconds = time.perf_counter() - seed_started
+ seed_cpu_seconds = time.process_time() - seed_cpu_started
+ logger.debug(
+ f"KMeans centroid seeding: sampleRows={init_size} "
+ f"compute={seed_seconds:.3f}s cpu={seed_cpu_seconds:.3f}s "
+ f"effectiveCores="
+ f"{seed_cpu_seconds / max(seed_seconds, 1e-12):.2f} "
+ f"rss={process_rss_mb():.0f} MiB"
+ )
+ del sample, sample_indices
+
+ model = make_model(init=np.asarray(initial_centers))
+ update_buffer = np.empty(
+ (effective_kmeans_batch_size, coordinate_dims),
+ dtype=coordinate_dtype,
+ )
+ buffered_rows = 0
+ fitted_rows = 0
+ fit_blocks = 0
+ update_count = 0
+ fit_read_seconds = 0.0
+ fit_read_cpu_seconds = 0.0
+ fit_compute_seconds = 0.0
+ fit_compute_cpu_seconds = 0.0
+ for _, block, read_seconds, read_cpu_seconds in timed_blocks(
+ "Fitting kmeans"
+ ):
+ if (
+ block.ndim != 2
+ or int(block.shape[1]) != coordinate_dims
+ or block.dtype != coordinate_dtype
+ ):
+ raise ValueError("K-means coordinate block dimensions changed")
+ block_rows = int(block.shape[0])
+ fitted_rows += block_rows
+ if fitted_rows > n_rows:
+ raise ValueError("K-means coordinate source has too many rows")
+ fit_blocks += 1
+ fit_read_seconds += read_seconds
+ fit_read_cpu_seconds += read_cpu_seconds
+ compute_started = time.perf_counter()
+ compute_cpu_started = time.process_time()
+ block_offset = 0
+ while block_offset < block_rows:
+ rows_to_copy = min(
+ effective_kmeans_batch_size - buffered_rows,
+ block_rows - block_offset,
+ )
+ update_buffer[buffered_rows : buffered_rows + rows_to_copy] = block[
+ block_offset : block_offset + rows_to_copy
+ ]
+ buffered_rows += rows_to_copy
+ block_offset += rows_to_copy
+ if buffered_rows == effective_kmeans_batch_size:
+ model.partial_fit(update_buffer)
+ update_count += 1
+ buffered_rows = 0
+ fit_compute_seconds += time.perf_counter() - compute_started
+ fit_compute_cpu_seconds += time.process_time() - compute_cpu_started
+ if fitted_rows != n_rows:
+ raise ValueError(
+ f"K-means coordinate source contains {fitted_rows} rows, "
+ f"expected {n_rows}"
+ )
+ if buffered_rows:
+ compute_started = time.perf_counter()
+ compute_cpu_started = time.process_time()
+ model.partial_fit(update_buffer[:buffered_rows])
+ update_count += 1
+ fit_compute_seconds += time.perf_counter() - compute_started
+ fit_compute_cpu_seconds += time.process_time() - compute_cpu_started
+ del update_buffer
+ logger.debug(
+ f"KMeans streaming fit: blocks={fit_blocks} rows={fitted_rows} "
+ f"updates={update_count} read={fit_read_seconds:.3f}s "
+ f"readCpu={fit_read_cpu_seconds:.3f}s "
+ f"compute={fit_compute_seconds:.3f}s "
+ f"computeCpu={fit_compute_cpu_seconds:.3f}s "
+ f"computeCores="
+ f"{fit_compute_cpu_seconds / max(fit_compute_seconds, 1e-12):.2f} "
+ f"rss={process_rss_mb():.0f} MiB"
+ )
+
+ labels = np.empty(n_rows, dtype=np.uint32)
+ predicted_rows = 0
+ predict_blocks = 0
+ predict_read_seconds = 0.0
+ predict_read_cpu_seconds = 0.0
+ predict_compute_seconds = 0.0
+ predict_compute_cpu_seconds = 0.0
+ for _, block, read_seconds, read_cpu_seconds in timed_blocks(
+ "Estimating seed partitions"
+ ):
+ if (
+ block.ndim != 2
+ or int(block.shape[1]) != coordinate_dims
+ or block.dtype != coordinate_dtype
+ ):
+ raise ValueError("K-means coordinate block dimensions changed")
+ block_rows = int(block.shape[0])
+ block_stop = predicted_rows + block_rows
+ if block_stop > n_rows:
+ raise ValueError("K-means coordinate source has too many rows")
+ predict_blocks += 1
+ predict_read_seconds += read_seconds
+ predict_read_cpu_seconds += read_cpu_seconds
+ compute_started = time.perf_counter()
+ compute_cpu_started = time.process_time()
+ labels[predicted_rows:block_stop] = model.predict(block)
+ predict_compute_seconds += time.perf_counter() - compute_started
+ predict_compute_cpu_seconds += time.process_time() - compute_cpu_started
+ predicted_rows = block_stop
+ if predicted_rows != n_rows:
+ raise ValueError(
+ f"K-means coordinate source contains {predicted_rows} rows, "
+ f"expected {n_rows}"
+ )
+ logger.debug(
+ f"KMeans prediction pass: blocks={predict_blocks} "
+ f"rows={predicted_rows} read={predict_read_seconds:.3f}s "
+ f"readCpu={predict_read_cpu_seconds:.3f}s "
+ f"compute={predict_compute_seconds:.3f}s "
+ f"computeCpu={predict_compute_cpu_seconds:.3f}s "
+ f"computeCores="
+ f"{predict_compute_cpu_seconds / max(predict_compute_seconds, 1e-12):.2f} "
+ f"rss={process_rss_mb():.0f} MiB"
+ )
+ return KMeansInitialization(
+ model=model,
+ labels=labels,
+ )
diff --git a/scarf/neighbors/stream.py b/scarf/neighbors/stream.py
new file mode 100644
index 00000000..203fda27
--- /dev/null
+++ b/scarf/neighbors/stream.py
@@ -0,0 +1,236 @@
+from collections.abc import Callable, Generator
+from typing import Any
+
+import numpy as np
+import pandas as pd
+from threadpoolctl import threadpool_limits
+
+from ..embeddings.harmony import HarmonyResult
+from ..matrix import ChunkedArray
+from ..utils.logging import logger
+from .stages import (
+ AnnIndexStage,
+ BatchCorrectionStage,
+ ChunkedCoordinateStream,
+ CoordinateSource,
+ KMeansInitializationStage,
+ LazyTransformStream,
+ NeighborQueryStage,
+ ReductionTransform,
+)
+
+__all__ = ["AnnStream"]
+
+
+class AnnStream:
+ """Compatibility adapter over lazy graph-computation stages."""
+
+ reducer: Callable[[np.ndarray], np.ndarray]
+
+ def __init__(
+ self,
+ data: ChunkedArray,
+ k: int,
+ n_cluster: int,
+ reduction_method: str,
+ dims: int | None,
+ loadings: np.ndarray | None,
+ use_for_pca: np.ndarray,
+ mu: np.ndarray,
+ sigma: np.ndarray,
+ ann_metric: str,
+ ann_efc: int,
+ ann_ef: int,
+ ann_m: int,
+ nthreads: int,
+ ann_parallel: bool,
+ rand_state: int,
+ do_kmeans_fit: bool,
+ disable_scaling: bool,
+ ann_idx: Any | None,
+ lsi_skip_first: bool,
+ lsi_params: dict[str, Any],
+ harmonize: bool,
+ harmonized_data: ChunkedArray | None = None,
+ batches: pd.DataFrame | None = None,
+ harmony_params: dict[str, Any] | None = None,
+ ) -> None:
+ self.data = data
+ self.k = min(k, self.data.shape[0] - 1)
+ self.nClusters = max(n_cluster, 2)
+ self.dims = dims
+ self.loadings = loadings
+ if self.dims is None and self.loadings is None:
+ raise ValueError(
+ "ERROR: Provide either value for atleast one: 'dims' or 'loadings'"
+ )
+ self.annMetric = ann_metric
+ self.annEfc = ann_efc
+ self.annEf = ann_ef
+ self.annM = ann_m
+ self.nthreads = nthreads
+ self.annThreads = self.nthreads if ann_parallel else 1
+ self.randState = rand_state
+ self.batchSize = self._handle_batch_size()
+ self.method = reduction_method
+ self.nCells, self.nFeats = self.data.shape
+ self.clusterLabels: np.ndarray = np.repeat(-1, self.nCells)
+ self.harmonize = harmonize
+ self.harmonizedData = harmonized_data
+ self.harmonyResult: HarmonyResult | None = None
+ self.batches = batches
+ self.harmonyParams = dict(harmony_params or {})
+ self.featureScaling = not disable_scaling
+
+ with threadpool_limits(limits=self.nthreads):
+ self.reduction = ReductionTransform(
+ data=self.data,
+ method=self.method,
+ dims=self.dims,
+ loadings=self.loadings,
+ use_for_pca=use_for_pca,
+ mu=mu,
+ sigma=sigma,
+ batch_size=self.batchSize,
+ nthreads=self.nthreads,
+ rand_state=self.randState,
+ disable_scaling=disable_scaling,
+ lsi_skip_first=lsi_skip_first,
+ lsi_params=lsi_params,
+ )
+ self.dims = self.reduction.dims
+ self.loadings = self.reduction.loadings
+ self.reducer = self.reduction.transform
+ if self.method == "pca":
+ self.mu = self.reduction.mu
+ self.sigma = self.reduction.sigma
+ if self.reduction.pca is not None:
+ self._pca = self.reduction.pca
+
+ self.transform_stream = LazyTransformStream(
+ data=self.data,
+ transform=self.reducer,
+ nthreads=self.nthreads,
+ batch_size=self.batchSize,
+ )
+ self.batch_correction = (
+ BatchCorrectionStage(
+ stream=self.transform_stream,
+ n_cells=self.nCells,
+ dims=int(self.dims or self.nFeats),
+ batch_size=self.batchSize,
+ batches=self.batches,
+ parameters=self.harmonyParams,
+ corrected_data=self.harmonizedData,
+ nthreads=self.nthreads,
+ )
+ if self.harmonize
+ else None
+ )
+ if ann_idx is None:
+ self.annIdx = self._fit_ann()
+ else:
+ self.annIdx = AnnIndexStage.configure(
+ ann_idx,
+ ef=self.annEf,
+ threads=self.annThreads,
+ )
+ self.neighbor_query = NeighborQueryStage(
+ self.annIdx,
+ self.k,
+ self.annMetric,
+ )
+ self._sync_batch_correction()
+ self.kmeans = self._fit_kmeans(do_kmeans_fit)
+
+ def _sync_batch_correction(self) -> None:
+ if self.batch_correction is None:
+ return
+ self.harmonizedData = self.batch_correction.corrected_data
+ self.harmonyResult = self.batch_correction.result
+
+ def iter_blocks(self, msg: str = "") -> Generator[np.ndarray, None, None]:
+ yield from self.transform_stream.iter_raw(msg)
+
+ def transform_query(self, values: np.ndarray) -> np.ndarray:
+ query = np.asarray(values)
+ if query.ndim != 2:
+ raise ValueError("Query data must be a two-dimensional array")
+ if query.shape[1] != self.nFeats:
+ raise ValueError(
+ f"Query has {query.shape[1]} features but reference expects {self.nFeats}"
+ )
+ result = np.asarray(self.reducer(query))
+ expected_dims = (
+ self.dims if self.dims is not None and self.dims > 0 else self.nFeats
+ )
+ if result.ndim != 2 or result.shape[1] != expected_dims:
+ raise ValueError("Query transform did not produce the ANN index dimensions")
+ if not np.all(np.isfinite(result)):
+ raise ValueError("Query transform produced non-finite values")
+ return result
+
+ def transform_ann(
+ self,
+ values: np.ndarray,
+ k: int | None = None,
+ self_indices: np.ndarray | None = None,
+ ) -> tuple[np.ndarray, np.ndarray] | tuple[np.ndarray, np.ndarray, int]:
+ return self.neighbor_query.query(
+ values,
+ k=k,
+ self_indices=self_indices,
+ )
+
+ def _fit_ann(self) -> Any:
+ ann_dims = (
+ self.dims
+ if self.dims is not None and self.dims >= 1
+ else self.data.shape[1]
+ )
+ index = AnnIndexStage.create(
+ metric=self.annMetric,
+ dims=ann_dims,
+ n_cells=self.nCells,
+ ef_construction=self.annEfc,
+ ef=self.annEf,
+ m=self.annM,
+ rand_state=self.randState,
+ ann_threads=self.annThreads,
+ )
+ coordinates: CoordinateSource = self.transform_stream
+ if self.batch_correction is not None:
+ coordinates = ChunkedCoordinateStream(
+ self.batch_correction.ensure_corrected(),
+ self.nthreads,
+ )
+ index = AnnIndexStage.populate(index, coordinates)
+ self._sync_batch_correction()
+ return index
+
+ def _fit_kmeans(self, enabled: bool) -> Any | None:
+ result = KMeansInitializationStage.fit(
+ stream=self.transform_stream,
+ n_rows=self.nCells,
+ batch_size=self.batchSize,
+ n_clusters=self.nClusters,
+ rand_state=self.randState,
+ nthreads=self.nthreads,
+ enabled=enabled,
+ )
+ self.clusterLabels = result.labels
+ return result.model
+
+ def _handle_batch_size(self) -> int:
+ if self.dims is not None and self.dims > self.data.shape[0]:
+ self.dims = self.data.shape[0]
+ batch_size = self.data.chunksize[0]
+ if self.dims is not None and self.dims >= batch_size:
+ self.dims = batch_size - 1
+ logger.warning(
+ f"Number of PCA/LSI components reduced to batch size of {batch_size}"
+ )
+ if self.nClusters > batch_size:
+ self.nClusters = batch_size
+ logger.warning(f"Cluster number reduced to batch size of {batch_size}")
+ return batch_size
diff --git a/scarf/plots.py b/scarf/plots.py
deleted file mode 100644
index 352d7799..00000000
--- a/scarf/plots.py
+++ /dev/null
@@ -1,1335 +0,0 @@
-"""Contains the code for plotting in Scarf."""
-
-from typing import Tuple, Optional, Union, List
-
-import matplotlib as mpl
-import matplotlib.pyplot as plt
-import numpy as np
-import pandas as pd
-import seaborn as sns
-from cmocean import cm
-
-from .utils import logger
-
-plt.rcParams["svg.fonttype"] = "none"
-
-
-# These palettes were lifted from scanpy.plotting.palettes
-custom_palettes = {
- 10: [
- "#1f77b4",
- "#ff7f0e",
- "#279e68",
- "#d62728",
- "#aa40fc",
- "#8c564b",
- "#e377c2",
- "#7f7f7f",
- "#b5bd61",
- "#17becf",
- ],
- 20: [
- "#1f77b4",
- "#aec7e8",
- "#ff7f0e",
- "#ffbb78",
- "#2ca02c",
- "#98df8a",
- "#d62728",
- "#ff9896",
- "#9467bd",
- "#c5b0d5",
- "#8c564b",
- "#c49c94",
- "#e377c2",
- "#f7b6d2",
- "#7f7f7f",
- "#c7c7c7",
- "#bcbd22",
- "#dbdb8d",
- "#17becf",
- "#9edae5",
- ],
- 28: [
- "#023fa5",
- "#7d87b9",
- "#bec1d4",
- "#d6bcc0",
- "#bb7784",
- "#8e063b",
- "#4a6fe3",
- "#8595e1",
- "#b5bbe3",
- "#e6afb9",
- "#e07b91",
- "#d33f6a",
- "#11c638",
- "#8dd593",
- "#c6dec7",
- "#ead3c6",
- "#f0b98d",
- "#ef9708",
- "#0fcfc0",
- "#9cded6",
- "#d5eae7",
- "#f3e1eb",
- "#f6c4e1",
- "#f79cd4",
- "#7f7f7f",
- "#c7c7c7",
- "#1CE6FF",
- "#336600",
- ],
- 102: [
- "#FFFF00",
- "#1CE6FF",
- "#FF34FF",
- "#FF4A46",
- "#008941",
- "#006FA6",
- "#A30059",
- "#FFDBE5",
- "#7A4900",
- "#0000A6",
- "#63FFAC",
- "#B79762",
- "#004D43",
- "#8FB0FF",
- "#997D87",
- "#5A0007",
- "#809693",
- "#6A3A4C",
- "#1B4400",
- "#4FC601",
- "#3B5DFF",
- "#4A3B53",
- "#FF2F80",
- "#61615A",
- "#BA0900",
- "#6B7900",
- "#00C2A0",
- "#FFAA92",
- "#FF90C9",
- "#B903AA",
- "#D16100",
- "#DDEFFF",
- "#000035",
- "#7B4F4B",
- "#A1C299",
- "#300018",
- "#0AA6D8",
- "#013349",
- "#00846F",
- "#372101",
- "#FFB500",
- "#C2FFED",
- "#A079BF",
- "#CC0744",
- "#C0B9B2",
- "#C2FF99",
- "#001E09",
- "#00489C",
- "#6F0062",
- "#0CBD66",
- "#EEC3FF",
- "#456D75",
- "#B77B68",
- "#7A87A1",
- "#788D66",
- "#885578",
- "#FAD09F",
- "#FF8A9A",
- "#D157A0",
- "#BEC459",
- "#456648",
- "#0086ED",
- "#886F4C",
- "#34362D",
- "#B4A8BD",
- "#00A6AA",
- "#452C2C",
- "#636375",
- "#A3C8C9",
- "#FF913F",
- "#938A81",
- "#575329",
- "#00FECF",
- "#B05B6F",
- "#8CD0FF",
- "#3B9700",
- "#04F757",
- "#C8A1A1",
- "#1E6E00",
- "#7900D7",
- "#A77500",
- "#6367A9",
- "#A05837",
- "#6B002C",
- "#772600",
- "#D790FF",
- "#9B9700",
- "#549E79",
- "#FFF69F",
- "#201625",
- "#72418F",
- "#BC23FF",
- "#99ADC0",
- "#3A2465",
- "#922329",
- "#5B4534",
- "#FDE8DC",
- "#404E55",
- "#0089A3",
- "#CB7E98",
- "#A4E804",
- "#324E72",
- ],
-}
-
-
-def clean_axis(ax, ts=11, ga=0.4):
- """Cleans a given matplotlib axis."""
- ax.xaxis.set_tick_params(labelsize=ts)
- ax.yaxis.set_tick_params(labelsize=ts)
- for i in ["top", "bottom", "left", "right"]:
- ax.spines[i].set_visible(False)
- ax.grid(which="major", linestyle="--", alpha=ga)
- ax.figure.patch.set_alpha(0)
- ax.patch.set_alpha(0)
- return True
-
-
-def plot_graph_qc(g):
- # TODO: add docstring description. Is this for qc of a graph, or for plotting a qc plot of a graph?
- _, axis = plt.subplots(1, 2, figsize=(12, 4))
- ax = axis[0]
- x = np.array((g != 0).sum(axis=0))[0]
- y = pd.Series(x).value_counts().sort_index()
- ax.bar(y.index, y.values, width=0.5)
- xlim = np.percentile(x, 99.5) + 5
- ax.set_xlim((0, xlim))
- ax.set_xlabel("Node degree")
- ax.set_ylabel("Frequency")
- ax.text(
- xlim,
- y.values.max(),
- f"plot is clipped (max degree: {y.index.max()})",
- ha="right",
- fontsize=9,
- )
- clean_axis(ax)
- ax = axis[1]
- ax.hist(g.data, bins=30)
- ax.set_xlabel("Edge weight")
- ax.set_ylabel("Frequency")
- clean_axis(ax)
- plt.tight_layout()
- plt.show()
-
-
-def plot_qc(
- data: pd.DataFrame,
- color: str = "steelblue",
- cmap: str = "tab20",
- fig_size: Optional[Tuple] = None,
- label_size: float = 10.0,
- title_size: float = 10,
- sup_title: Optional[str] = None,
- sup_title_size: float = 12,
- scatter_size: float = 1.0,
- max_points: int = 10000,
- show_on_single_row: bool = True,
- show_fig: bool = True,
-):
- # TODO: add docstring description. Is this for qc of a plot, or for plotting a qc plot?
- n_plots = data.shape[1] - 1
- n_groups = data["groups"].nunique()
- if n_groups > 5 and show_on_single_row is True:
- logger.info(
- f"Too many groups in the plot. If you think that plot is too wide then consider turning "
- f"`show_on_single_row` parameter to False"
- )
- if show_on_single_row is True:
- n_rows = 1
- n_cols = n_plots
- else:
- n_rows = n_plots
- n_cols = 1
- if fig_size is None:
- fig_width = min(15, n_groups + (2 * n_cols))
- fig_height = 1 + 2.5 * n_rows
- fig_size = (fig_width, fig_height)
- fig = plt.figure(figsize=fig_size)
- grouped = data.groupby("groups")
- for i in range(n_plots):
- if data.columns[i] == "groups":
- continue
- vals = {"g": [], "v": []}
- for j in sorted(data["groups"].unique()):
- val = grouped.get_group(j)[data.columns[i]].values
- vals["g"].extend([j for _ in range(len(val))])
- vals["v"].extend(list(val))
- vals = pd.DataFrame(vals)
- ax = fig.add_subplot(n_rows, n_cols, i + 1)
- if n_groups == 1:
- sns.violinplot(
- y="v",
- x="g",
- data=vals,
- linewidth=1,
- orient="v",
- alpha=0.6,
- inner=None,
- cut=0,
- color=color,
- )
- else:
- sns.violinplot(
- y="v",
- x="g",
- data=vals,
- linewidth=1,
- orient="v",
- alpha=0.6,
- inner=None,
- cut=0,
- palette=cmap,
- )
- if len(vals) > max_points:
- sns.stripplot(
- x="g",
- y="v",
- data=vals.sample(n=max_points),
- jitter=0.4,
- ax=ax,
- orient="v",
- s=scatter_size,
- color="k",
- alpha=0.4,
- )
- else:
- sns.stripplot(
- x="g",
- y="v",
- data=vals,
- jitter=0.4,
- ax=ax,
- orient="v",
- s=scatter_size,
- color="k",
- alpha=0.4,
- )
- ax.set_ylabel(data.columns[i], fontsize=label_size)
- ax.set_xlabel("")
- if n_groups == 1:
- ax.set_xticks([])
- ax.set_xticklabels([])
- if data["groups"].nunique() == 1:
- ax.set_title(
- "Median: %.1f" % (int(np.median(vals["v"]))), fontsize=title_size
- )
- # clean_axis(ax)
- ax.figure.patch.set_alpha(0)
- ax.patch.set_alpha(0)
- fig.suptitle(sup_title, fontsize=sup_title_size)
- plt.tight_layout()
- if show_fig:
- plt.show()
- else:
- return fig
-
-
-def plot_mean_var(
- nzm: np.ndarray,
- fv: np.ndarray,
- n_cells: np.ndarray,
- hvg: np.ndarray,
- ax_label_fs: float = 12,
- fig_size: Tuple[float, float] = (4.5, 4.0),
- ss: Tuple[float, float] = (3, 30),
- cmaps: Tuple[str, str] = ("winter", "magma_r"),
-):
- """Shows a mean-variance plot."""
- _, ax = plt.subplots(1, 1, figsize=fig_size)
- nzm = np.log2(nzm)
- fv = np.log2(fv)
- ax.scatter(nzm[~hvg], fv[~hvg], alpha=0.6, c=n_cells[~hvg], cmap=cmaps[0], s=ss[0])
- ax.scatter(
- nzm[hvg],
- fv[hvg],
- alpha=0.8,
- c=n_cells[hvg],
- cmap=cmaps[1],
- s=ss[1],
- edgecolor="k",
- lw=0.5,
- )
- ax.set_xlabel("Log mean non-zero expression", fontsize=ax_label_fs)
- ax.set_ylabel("Log corrected variance", fontsize=ax_label_fs)
- clean_axis(ax)
- plt.tight_layout()
- plt.show()
-
-
-def plot_elbow(var_exp, figsize: Tuple[float, float] = (None, 2)):
- from kneed import KneeLocator
-
- x = range(len(var_exp))
- kneedle = KneeLocator(x, var_exp, S=1.0, curve="convex", direction="decreasing")
- if figsize[0] is None:
- figsize = (0.25 * len(var_exp), figsize[1])
- fig, ax = plt.subplots(1, 1, figsize=figsize)
- ax.plot(x, var_exp, lw=1)
- ax.set_xticks(x)
- ax.axvline(kneedle.elbow, lw=1, c="r", label="Elbow")
- ax.set_ylabel("% Variance explained", fontsize=9)
- ax.set_xlabel("Principal components", fontsize=9)
- clean_axis(ax, ts=8)
- ax.legend(frameon=False, fontsize=9)
- plt.tight_layout()
- plt.show()
-
-
-def plot_heatmap(
- cdf,
- fontsize: float = 10,
- width_factor: float = 0.03,
- height_factor: float = 0.02,
- cmap=cm.matter_r,
- savename: str = None,
- save_dpi: int = 300,
- figsize=None,
- show_fig: bool = True,
- **heatmap_kwargs,
-):
- """Shows a heatmap plot."""
- if figsize is None:
- figsize = (
- cdf.shape[1] * fontsize * width_factor,
- fontsize * cdf.shape[0] * height_factor,
- )
- cgx = sns.clustermap(
- cdf,
- yticklabels=cdf.index,
- xticklabels=cdf.columns,
- method="ward",
- figsize=figsize,
- cmap=cmap,
- rasterized=True,
- **heatmap_kwargs,
- )
- cgx.ax_heatmap.set_yticklabels(
- cdf.index[cgx.dendrogram_row.reordered_ind], fontsize=fontsize
- )
- cgx.ax_heatmap.set_xticklabels(
- cdf.columns[cgx.dendrogram_col.reordered_ind], fontsize=fontsize
- )
- cgx.ax_heatmap.figure.patch.set_alpha(0)
- cgx.ax_heatmap.patch.set_alpha(0)
- if savename:
- plt.savefig(savename, dpi=save_dpi)
- if show_fig:
- plt.show()
- else:
- return cgx
-
-
-def _scatter_fix_type(v: pd.Series, ints_as_cats: bool) -> pd.Series:
- vt = v.dtype
- if v.nunique() == 1:
- return pd.Series(np.ones(len(v)), index=v.index).astype(np.float64)
- if vt in [np.bool_]:
- # converting first to int to handle bool
- return v.astype(np.int_).astype("category")
- if vt in [str, object] or vt.name == "category":
- return v.astype("category")
- elif np.issubdtype(vt.type, np.integer) and ints_as_cats:
- if v.nunique() > 100:
- logger.warning("Too many categories. set force_ints_as_cats to false")
- return v.astype(np.int_).astype("category")
- else:
- return v.astype(np.float64)
-
-
-def _scatter_fix_mask(v: pd.Series, mask_vals: list, mask_name: str) -> pd.Series:
- if mask_vals is None:
- mask_vals = []
- mask_vals += [np.nan]
- iscat = False
- if v.dtype.name == "category":
- iscat = True
- v = v.astype(object)
- # There is a bug in pandas which causes failure above 1M rows
- # v[v.isin(mask_vals)] = mask_name
- v[np.isin(v, mask_vals)] = mask_name
- if iscat:
- v = v.astype("category")
- return v
-
-
-def _scatter_make_colors(
- v: pd.Series, cmap, color_key: Optional[dict], mask_color: str, mask_name: str
-):
- from matplotlib.pyplot import get_cmap
-
- na_idx = v == mask_name
- uv = v[~na_idx].unique()
-
- if v.dtype.name != "category":
- if cmap is None:
- return cm.deep, None
- else:
- return get_cmap(cmap), None
- else:
- if cmap is None:
- cmap = "custom"
-
- if color_key is not None:
- for i in uv:
- if i not in color_key:
- raise KeyError(f"ERROR: key {i} missing in `color_key`")
- if na_idx.sum() > 0:
- if mask_name not in color_key:
- color_key[mask_name] = mpl.colors.to_hex(mask_color)
- return None, color_key
- else:
- if cmap == "custom" and len(uv) <= 102:
- if len(uv) <= 10:
- pal = custom_palettes[10]
- elif len(uv) <= 20:
- pal = custom_palettes[20]
- elif len(uv) <= 28:
- pal = custom_palettes[28]
- else:
- pal = custom_palettes[102]
- else:
- if cmap == "custom":
- cmap = None
- pal = sns.color_palette(palette=cmap, n_colors=len(uv)).as_hex()
- color_key = dict(zip(sorted(uv), pal))
- if na_idx.sum() > 0:
- color_key[mask_name] = mpl.colors.to_hex(mask_color)
- return None, color_key
-
-
-def _scatter_cleanup(ax, sw: float, sc: str, ds: tuple) -> None:
- for i in ["bottom", "left", "top", "right"]:
- spine = ax.spines[i]
- if i in ds:
- spine.set_visible(True)
- spine.set_linewidth(sw)
- spine.set_edgecolor(sc)
- else:
- spine.set_visible(False)
- ax.figure.patch.set_alpha(0)
- ax.patch.set_alpha(0)
- ax.set_aspect("auto")
- return None
-
-
-def _scatter_label_axis(df, ax, fs: float, fo: float):
- x, y = df.columns[:2]
- ax.set_xlabel(x, fontsize=fs)
- ax.set_ylabel(y, fontsize=fs)
- vmin, vmax = df[x].min(), df[x].max()
- ax.set_xlim((vmin - abs(vmin * fo), vmax + abs(vmax * fo)))
- vmin, vmax = df[y].min(), df[y].max()
- ax.set_ylim((vmin - abs(vmin * fo), vmax + abs(vmax * fo)))
- ax.set_xticks([])
- ax.set_yticks([])
- return None
-
-
-def _scatter_legends(
- df,
- ax,
- cmap,
- ck,
- ondata: bool,
- onside: bool,
- fontsize: float,
- title: str,
- title_fontsize: float,
- hide_title: bool,
- n_per_col: int,
- scale: float,
- ls: float,
- cs: float,
- cbs: float,
-) -> None:
- """
-
- Args:
- df: dataframe
- ax: axis object
- cmap: color map
- ck: color key
- ondata: display legend over scatter plot?
- onside: display legend on side?
- fontsize: fontsize of legend text
- title: Title of subplot/axes
- hide_title: Whether to hide the title
- n_per_col: number of legends per column
- scale: scale legend marker size
- ls: line spacing
- cs: column spacing
- cbs: Cbar shrink factor
-
- Returns:
-
- """
- from matplotlib.colors import Normalize
- from matplotlib.colorbar import ColorbarBase, make_axes_gridspec
-
- x, y, vc = df.columns[:3]
- v = df[vc]
- cax = make_axes_gridspec(ax, location="top", shrink=cbs, aspect=25, fraction=0.1)[0]
- if v.nunique() <= 1:
- cax.set_axis_off()
- return None
- if v.dtype.name == "category":
- if hide_title is False:
- if title is not None:
- ax.title.set_text(title)
- else:
- ax.title.set_text(vc)
- ax.title.set_fontsize(title_fontsize)
- centers = df[[x, y, vc]].groupby(vc).median().T
- for i in centers:
- if ondata:
- ax.text(
- centers[i][x],
- centers[i][y],
- i,
- fontsize=fontsize,
- ha="center",
- va="center",
- )
- if onside:
- ax.scatter(
- [float(centers[i][x])],
- [float(centers[i][y])],
- c=ck[i],
- label=i,
- alpha=1,
- s=0.01,
- )
- if onside:
- n_cols = v.nunique() // n_per_col
- if v.nunique() % n_per_col > 0:
- n_cols += 1
- ax.legend(
- ncol=n_cols,
- loc=(1, 0),
- frameon=False,
- fontsize=fontsize,
- markerscale=scale,
- labelspacing=ls,
- columnspacing=cs,
- )
- cax.set_axis_off()
- else:
- norm = Normalize(vmin=v.min(), vmax=v.max())
- cb = ColorbarBase(cax, cmap=cmap, norm=norm, orientation="horizontal")
- if hide_title is False:
- if title is not None:
- cb.set_label(title, fontsize=title_fontsize)
- else:
- cb.set_label(vc, fontsize=title_fontsize)
- cb.ax.xaxis.set_label_position("bottom")
- cb.ax.xaxis.set_ticks_position("top")
- cb.outline.set_visible(False)
- return None
-
-
-def _make_grid(width, height, w_pad, h_pad, n_panels, n_columns):
- n_columns = np.minimum(n_panels, n_columns)
- n_rows = np.ceil(n_panels / n_columns).astype(int)
- if w_pad is None and h_pad is None:
- constrained = True
- else:
- constrained = False
- fig, axes = plt.subplots(
- n_rows,
- n_columns,
- figsize=(width * n_columns, height * n_rows),
- squeeze=False,
- constrained_layout=constrained,
- )
- diff = (n_rows * n_columns) - n_panels
- while diff > 0:
- fig.delaxes(axes[n_rows - 1, n_columns - diff])
- diff -= 1
- if not constrained:
- plt.tight_layout(w_pad=w_pad, h_pad=h_pad)
-
- return fig, axes
-
-
-def _create_axes(dfs, in_ax, width, height, w_pad, h_pad, n_columns):
- if len(dfs) > 1:
- if in_ax is not None:
- logger.warning(
- f"'in_ax' will not be used as multiple attributes will be plotted. Using internal grid"
- f"layout"
- )
- _, axs = _make_grid(width, height, w_pad, h_pad, len(dfs), n_columns)
- else:
- if in_ax is None:
- _, axs = plt.subplots(1, 1, figsize=(width, height), squeeze=False)
- else:
- axs = in_ax
- return axs
-
-
-def _iter_dataframes(dfs, mask_values, mask_name, force_ints_as_cats):
- for n, df in enumerate(dfs):
- vc = df.columns[2]
- v = _scatter_fix_mask(df[vc].copy(), mask_values, mask_name)
- df[vc] = _scatter_fix_type(v, force_ints_as_cats)
- yield n, df
-
-
-def _handle_titles_type(titles, n_df):
- if titles is not None:
- if n_df > 1:
- if len(titles) != n_df or type(titles) != list:
- logger.warning(
- "Number of titles is not same as the the number of titles. Provided titles cannot be used"
- )
- titles = None
- else:
- if type(titles) == str:
- titles = [titles]
- return titles
-
-
-def plot_scatter(
- dfs,
- in_ax=None,
- width: float = 6,
- height: float = 6,
- default_color: str = "steelblue",
- color_map=None,
- color_key: dict = None,
- mask_values: list = None,
- mask_name: str = "NA",
- mask_color: str = "k",
- point_size: float = 10,
- ax_label_size: float = 12,
- frame_offset: float = 0.05,
- spine_width: float = 0.5,
- spine_color: str = "k",
- displayed_sides: tuple = ("bottom", "left"),
- legend_ondata: bool = True,
- legend_onside: bool = True,
- legend_size: float = 12,
- legends_per_col: int = 20,
- titles: str = None,
- title_size: int = 12,
- hide_title: bool = False,
- cbar_shrink: float = 0.6,
- marker_scale: float = 70,
- lspacing: float = 0.1,
- cspacing: float = 1,
- savename: str = None,
- dpi: int = 300,
- force_ints_as_cats: bool = True,
- n_columns: int = 4,
- w_pad: float = 1,
- h_pad: float = 1,
- show_fig: bool = True,
- scatter_kwargs: dict = None,
-):
- """Shows scatter plots.
-
- If more then one dataframe is provided it will place the
- scatterplots in a grid.
- """
- from matplotlib.colors import to_hex
-
- def _handle_scatter_kwargs(sk):
- if sk is None:
- sk = {}
- if "c" in sk:
- logger.warning("scatter_kwarg value `c` will be ignored")
- del sk["c"]
- if "s" in sk:
- logger.warning("scatter_kwarg value `s` will be ignored")
- del sk["s"]
- if "lw" not in sk:
- sk["lw"] = 0.1
- if "edgecolors" not in sk:
- sk["edgecolors"] = "k"
- return sk
-
- titles = _handle_titles_type(titles, len(dfs))
-
- axs = _create_axes(dfs, in_ax, width, height, w_pad, h_pad, n_columns)
- for n, df in _iter_dataframes(dfs, mask_values, mask_name, force_ints_as_cats):
- v = df[df.columns[2]]
- col_map, col_key = _scatter_make_colors(
- v, color_map, color_key, mask_color, mask_name
- )
- if v.dtype.name == "category":
- df["c"] = [col_key[x] for x in v]
- else:
- if v.nunique() == 1:
- df["c"] = [default_color for _ in v]
- else:
- v = v.copy().fillna(0)
- mmv = (v - v.min()) / (v.max() - v.min())
- df["c"] = [to_hex(col_map(x)) for x in mmv]
- if "s" not in df:
- df["s"] = [point_size for _ in df.index]
- scatter_kwargs = _handle_scatter_kwargs(sk=scatter_kwargs)
- ax = axs[int(n / n_columns), n % n_columns]
- ax.scatter(
- df.values[:, 0],
- df.values[:, 1],
- c=df["c"].values,
- s=df["s"].values,
- rasterized=True,
- **scatter_kwargs,
- )
- _scatter_label_axis(df, ax, ax_label_size, frame_offset)
- _scatter_cleanup(ax, spine_width, spine_color, displayed_sides)
- if titles is not None:
- title = titles[n]
- else:
- title = None
- _scatter_legends(
- df,
- ax,
- col_map,
- col_key,
- legend_ondata,
- legend_onside,
- legend_size,
- title,
- title_size,
- hide_title,
- legends_per_col,
- marker_scale,
- lspacing,
- cspacing,
- cbar_shrink,
- )
-
- if savename:
- plt.savefig(savename, dpi=dpi, bbox_inches="tight")
- if show_fig:
- plt.show()
- else:
- return axs
-
-
-def shade_scatter(
- dfs,
- in_ax=None,
- figsize: float = 6,
- pixels: int = 1000,
- spread_px: int = 1,
- spread_threshold: float = 0.2,
- min_alpha: int = 10,
- color_map=None,
- color_key: dict = None,
- mask_values: list = None,
- mask_name: str = "NA",
- mask_color: str = "k",
- ax_label_size: float = 12,
- frame_offset: float = 0.05,
- spine_width: float = 0.5,
- spine_color: str = "k",
- displayed_sides: tuple = ("bottom", "left"),
- legend_ondata: bool = True,
- legend_onside: bool = True,
- legend_size: float = 12,
- legends_per_col: int = 20,
- titles: Union[str, List[str]] = None,
- title_size: int = 12,
- hide_title: bool = False,
- cbar_shrink: float = 0.6,
- marker_scale: float = 70,
- lspacing: float = 0.1,
- cspacing: float = 1,
- savename: str = None,
- dpi: int = 300,
- force_ints_as_cats: bool = True,
- n_columns: int = 4,
- w_pad: float = None,
- h_pad: float = None,
- show_fig: bool = True,
-):
- """Shows shaded scatter plots.
-
- If more then one dataframe is provided it will place the
- scatterplots in a grid.
- """
- import datashader as dsh
- from datashader.mpl_ext import dsshow
- import datashader.transfer_functions as tf
- from functools import partial
-
- titles = _handle_titles_type(titles, len(dfs))
- axs = _create_axes(dfs, in_ax, figsize, figsize, w_pad, h_pad, n_columns)
- for n, df in _iter_dataframes(dfs, mask_values, mask_name, force_ints_as_cats):
- dim1, dim2, vc = df.columns[:3]
- v = df[vc]
- col_map, col_key = _scatter_make_colors(
- v, color_map, color_key, mask_color, mask_name
- )
- if v.dtype.name == "category":
- agg = dsh.count_cat(vc)
- else:
- if v.nunique() == 1:
- agg = dsh.count(vc)
- else:
- agg = dsh.mean(vc)
-
- ax = axs[int(n / n_columns), n % n_columns]
- artist = dsshow(
- df,
- dsh.Point(dim1, dim2),
- aggregator=agg,
- norm="eq_hist",
- color_key=col_key,
- cmap=col_map,
- alpha_range=(min_alpha, 255),
- shade_hook=partial(
- tf.dynspread, threshold=spread_threshold, max_px=spread_px
- ),
- plot_height=pixels,
- plot_width=pixels,
- aspect="equal",
- width_scale=1,
- height_scale=1,
- ax=ax,
- )
-
- _scatter_label_axis(df, ax, ax_label_size, frame_offset)
- _scatter_cleanup(ax, spine_width, spine_color, displayed_sides)
- if titles is not None:
- title = titles[n]
- else:
- title = None
- _scatter_legends(
- df,
- ax,
- col_map,
- col_key,
- legend_ondata,
- legend_onside,
- legend_size,
- title,
- title_size,
- hide_title,
- legends_per_col,
- marker_scale,
- lspacing,
- cspacing,
- cbar_shrink,
- )
-
- if savename:
- plt.savefig(savename, dpi=dpi, bbox_inches="tight")
- if show_fig:
- plt.show()
- else:
- return axs
-
-
-def _draw_pie(ax, dist, colors, xpos, ypos, size):
- # https://stackoverflow.com/questions/56337732/how-to-plot-scatter-pie-chart-using-matplotlib
- cumsum = np.cumsum(dist)
- cumsum = cumsum / cumsum[-1]
- pie = [0] + cumsum.tolist()
- for r1, r2, c in zip(pie[:-1], pie[1:], colors):
- angles = np.linspace(2 * np.pi * r1, 2 * np.pi * r2)
- x = [0] + np.cos(angles).tolist()
- y = [0] + np.sin(angles).tolist()
- xy = np.column_stack([x, y])
- ax.scatter([xpos], [ypos], marker=xy, s=size, c=c)
-
-
-def hierarchy_pos(
- g, root=None, width=1.0, vert_gap=0.2, vert_loc=0, leaf_vs_root_factor=0.5
-):
- """This function was lifted from here: https://github.com/springer-
- math/Mathematics-of-Epidemics-on-
- Networks/blob/80c8accbe0c6b7710c0a189df17529696ac31bf9/EoN/auxiliary.py.
-
- If the graph is a tree this will return the positions to plot this in a
- hierarchical layout.
-
- Based on Joel's answer at https://stackoverflow.com/a/29597209/2966723,
- but with some modifications.
- We include this because it may be useful for plotting transmission trees,
- and there is currently no networkx equivalent (though it may be coming soon).
-
- There are two basic approaches we think of to allocate the horizontal
- location of a node.
-
- - Top down: we allocate horizontal space to a node. Then its ``k``
- descendants split up that horizontal space equally. This tends to result
- in overlapping nodes when some have many descendants.
- - Bottom up: we allocate horizontal space to each leaf node. A node at a
- higher level gets the entire space allocated to its descendant leaves.
- Based on this, leaf nodes at higher levels get the same space as leaf
- nodes very deep in the tree.
-
- We use use both of these approaches simultaneously with ``leaf_vs_root_factor``
- determining how much of the horizontal space is based on the bottom up
- or top down approaches. ``0`` gives pure bottom up, while 1 gives pure top
- down.
-
- Args:
- g: the graph (must be a tree)
- root: the root node of the tree
- - if the tree is directed and this is not given, the root will be found and used
- - if the tree is directed and this is given, then the positions will be just for the descendants of
- this node.
- - if the tree is undirected and not given, then a random choice will be used.
- width: horizontal space allocated for this branch - avoids overlap with other branches
- vert_gap: gap between levels of hierarchy
- vert_loc: vertical location of root
- leaf_vs_root_factor: leaf_vs_root_factor
- xcenter: horizontal location of root
- """
-
- import networkx as nx
-
- if not nx.is_tree(g):
- raise TypeError("cannot use hierarchy_pos on a graph that is not a tree")
-
- if root is None:
- if isinstance(g, nx.DiGraph):
- root = next(
- iter(nx.topological_sort(g))
- ) # allows back compatibility with nx version 1.11
- else:
- root = np.random.choice(list(g.nodes))
-
- def _hierarchy_pos(
- g,
- root,
- leftmost,
- width,
- leafdx=0.2,
- vert_gap=0.2,
- vert_loc=0,
- xcenter=0.5,
- rootpos=None,
- leafpos=None,
- parent=None,
- ):
- """
- see hierarchy_pos docstring for most arguments
- pos: a dict saying where all nodes go if they have been assigned
- parent: parent of this branch. - only affects it if non-directed
- """
-
- if rootpos is None:
- rootpos = {root: (xcenter, vert_loc)}
- else:
- rootpos[root] = (xcenter, vert_loc)
- if leafpos is None:
- leafpos = {}
- children = list(g.neighbors(root))
- leaf_count = 0
- if not isinstance(g, nx.DiGraph) and parent is not None:
- children.remove(parent)
- if len(children) != 0:
- rootdx = width / len(children)
- nextx = xcenter - width / 2 - rootdx / 2
- for child in children:
- nextx += rootdx
- rootpos, leafpos, newleaves = _hierarchy_pos(
- g,
- child,
- leftmost + leaf_count * leafdx,
- width=rootdx,
- leafdx=leafdx,
- vert_gap=vert_gap,
- vert_loc=vert_loc - vert_gap,
- xcenter=nextx,
- rootpos=rootpos,
- leafpos=leafpos,
- parent=root,
- )
- leaf_count += newleaves
-
- leftmostchild = min((x for x, y in [leafpos[child] for child in children]))
- rightmostchild = max((x for x, y in [leafpos[child] for child in children]))
- leafpos[root] = ((leftmostchild + rightmostchild) / 2, vert_loc)
- else:
- leaf_count = 1
- leafpos[root] = (leftmost, vert_loc)
- # pos[root] = (leftmost + (leaf_count-1)*dx/2., vert_loc)
- # print(leaf_count)
- return rootpos, leafpos, leaf_count
-
- xcenter = width / 2.0
- if isinstance(g, nx.DiGraph):
- leafcount = len(
- [node for node in nx.descendants(g, root) if g.out_degree(node) == 0]
- )
- elif isinstance(g, nx.Graph):
- leafcount = len(
- [
- node
- for node in nx.node_connected_component(g, root)
- if g.degree(node) == 1 and node != root
- ]
- )
- rootpos, leafpos, leaf_count = _hierarchy_pos(
- g,
- root,
- 0,
- width,
- leafdx=width * 1.0 / leafcount,
- vert_gap=vert_gap,
- vert_loc=vert_loc,
- xcenter=xcenter,
- )
- pos = {}
- for node in rootpos:
- pos[node] = (
- leaf_vs_root_factor * leafpos[node][0]
- + (1 - leaf_vs_root_factor) * rootpos[node][0],
- leafpos[node][1],
- )
- xmax = max(x for x, y in pos.values())
- for node in pos:
- pos[node] = (pos[node][0] * width / xmax, pos[node][1])
- return pos
-
-
-def plot_cluster_hierarchy(
- sg,
- clusts,
- color_values=None,
- force_ints_as_cats: bool = True,
- width: float = 2,
- lvr_factor: float = 0.5,
- vert_gap: float = 0.2,
- min_node_size: float = 10,
- node_size_multiplier: float = 1e4,
- node_power: float = 1,
- root_size: float = 100,
- non_leaf_size: float = 10,
- show_labels: bool = False,
- fontsize=10,
- root_color: str = "#C0C0C0",
- non_leaf_color: str = "k",
- cmap: str = None,
- color_key: bool = None,
- edgecolors: str = "k",
- edgewidth: float = 1,
- alpha: float = 0.7,
- figsize=(5, 5),
- ax=None,
- show_fig: bool = True,
- savename: str = None,
- save_dpi=300,
-):
- """Shows a plot showing cluster hierarchy.
-
- Returns:
- If requested (with parameter `show_fig`) a matplotlib Axes object containing the plot
- (which is the modified `ax` parameter if given).
- """
- import networkx as nx
- import math
- from matplotlib.colors import to_hex
-
- if color_values is None:
- color_values = pd.Series(clusts)
- using_clust_for_colors = True
- else:
- color_values = pd.Series(color_values)
- using_clust_for_colors = False
- color_values = _scatter_fix_type(color_values, force_ints_as_cats)
- cmap, color_key = _scatter_make_colors(
- color_values, cmap, color_key, "k", "longdummyvaluesofh3489hfpiqehdcbla"
- )
- pos = hierarchy_pos(
- sg, width=width * math.pi, leaf_vs_root_factor=lvr_factor, vert_gap=vert_gap
- )
- new_pos = {
- u: (r * math.cos(theta), r * math.sin(theta)) for u, (theta, r) in pos.items()
- }
-
- if color_key is None:
- cluster_values = (
- pd.DataFrame({"clusters": clusts, "v": color_values})
- .groupby("clusters")
- .mean()["v"]
- )
- mmv: pd.Series = (cluster_values - cluster_values.min()) / (
- cluster_values.max() - cluster_values.min()
- )
- color_key = {k: to_hex(cmap(v)) for k, v in mmv.to_dict().items()}
- else:
- cluster_values = None
-
- cs = pd.Series(clusts).value_counts()
- cs = (node_size_multiplier * ((cs / cs.sum()) ** node_power)).to_dict()
- nc, ns = [], []
-
- for i in sg.nodes():
- if "partition_id" in sg.nodes[i]:
- clust_id = sg.nodes[i]["partition_id"]
- if cluster_values is not None or using_clust_for_colors:
- nc.append(color_key[clust_id])
- ns.append(max(cs[clust_id], min_node_size))
- else:
- nc.append("white")
- ns.append(0)
- else:
- if sg.nodes[i]["nleaves"] == len(clusts):
- nc.append(root_color)
- ns.append(root_size)
- else:
- nc.append(non_leaf_color)
- ns.append(non_leaf_size)
- if ax is None:
- fig, ax = plt.subplots(1, 1, figsize=figsize)
- nx.draw(
- sg,
- pos=new_pos,
- node_size=ns,
- node_color=nc,
- ax=ax,
- edgecolors=edgecolors,
- alpha=alpha,
- linewidths=edgewidth,
- )
-
- if cluster_values is None and using_clust_for_colors is False:
- for i in sg.nodes():
- if "partition_id" in sg.nodes[i]:
- clust_id = sg.nodes[i]["partition_id"]
- idx = clusts == clust_id
- counts = color_values[idx].value_counts()
- _draw_pie(
- ax,
- counts.values,
- [color_key[x] for x in counts.index],
- new_pos[i][0],
- new_pos[i][1],
- max(cs[clust_id], min_node_size),
- )
-
- if show_labels:
- for i in sg.nodes():
- if "partition_id" in sg.nodes[i]:
- clust_id = sg.nodes[i]["partition_id"]
- ax.text(
- new_pos[i][0],
- new_pos[i][1],
- clust_id,
- fontsize=fontsize,
- ha="center",
- va="center",
- )
- if savename:
- plt.savefig(savename, dpi=save_dpi)
- if show_fig:
- plt.show()
- else:
- return ax
-
-
-def plot_annotated_heatmap(
- df: np.array,
- xbar_values: np.ndarray,
- ybar_values: np.ndarray,
- display_row_labels: list = None,
- row_labels: list = None,
- width: int = 5,
- height: int = 10,
- vmin: float = -2.0,
- vmax: float = 2.0,
- heatmap_cmap: str = None,
- xbar_cmap: str = None,
- ybar_cmap: str = None,
- tick_fontsize: int = 10,
- axis_fontsize: int = 12,
- row_label_fontsize: int = 12,
- savename: str = None,
- save_dpi: int = 300,
- show_fig: bool = True,
-):
- import matplotlib.ticker as mticker
-
- if display_row_labels is None:
- display_row_labels = []
- if row_labels is None:
- row_labels = list(map(str, range(df.shape[0])))
- else:
- if len(row_labels) != df.shape[0]:
- raise ValueError(
- "ERROR: Number of provided feature labels and size of the data array does not match"
- )
-
- whr = height / width
- fig = plt.figure(constrained_layout=False, figsize=(width, height))
- gs = fig.add_gridspec(nrows=int(20 * whr), ncols=20, wspace=0, hspace=0)
- heatmap_ax = fig.add_subplot(gs[:-2, 1:16])
- clustbar_ax = fig.add_subplot(gs[:-2, 17:18])
- cbar_ax = fig.add_subplot(gs[round(7 * whr) : round(12 * whr), -1:])
- ptime_ax = fig.add_subplot(gs[-1:, 1:16])
-
- if heatmap_cmap is None:
- heatmap_cmap = "coolwarm"
- sns.heatmap(
- df,
- ax=heatmap_ax,
- cbar_ax=cbar_ax,
- xticklabels=[],
- yticklabels=[],
- vmin=vmin,
- vmax=vmax,
- cmap=heatmap_cmap,
- )
-
- if len(display_row_labels) > 0:
- row_labels = {x.lower(): n for n, x in enumerate(row_labels)}
- display_row_labels = [x for x in display_row_labels if x.lower() in row_labels]
- heatmap_ax.set_yticks([row_labels[x.lower()] for x in display_row_labels])
- heatmap_ax.set_yticklabels(display_row_labels, fontsize=row_label_fontsize)
-
- heatmap_ax.set_title(f"{df.shape[0]} features", fontsize=axis_fontsize)
- ticks_loc = cbar_ax.get_yticks().tolist()
- cbar_ax.yaxis.set_major_locator(mticker.FixedLocator(ticks_loc))
- cbar_ax.set_yticklabels([x for x in ticks_loc], fontsize=tick_fontsize)
-
- if ybar_cmap is None:
- ybar_cmap = "tab20"
- clustbar_ax.imshow(ybar_values.reshape(-1, 1), aspect="auto", cmap=ybar_cmap)
- clustbar_ax.set_xticks([])
- clustbar_ax.set_yticks([])
-
- for i in set(ybar_values):
- y = np.where(ybar_values == i)[0].mean()
- clustbar_ax.text(
- 0,
- y,
- i,
- fontsize=axis_fontsize,
- ha="center",
- va="center",
- )
-
- binned_ptime = [x.mean() for x in np.array_split(sorted(xbar_values), df.shape[1])]
- if xbar_cmap is None:
- xbar_cmap = cm.deep
- ptime_ax.imshow([binned_ptime], aspect="auto", cmap=xbar_cmap)
- ptime_ax.set_xticks([])
- ptime_ax.set_yticks([])
- ptime_ax.set_xlabel("------ Pseudotime----->", fontsize=axis_fontsize)
- if savename is not None:
- plt.savefig(savename, dpi=save_dpi)
- if show_fig:
- plt.show()
diff --git a/scarf/plotting/__init__.py b/scarf/plotting/__init__.py
new file mode 100644
index 00000000..0c5e5a9d
--- /dev/null
+++ b/scarf/plotting/__init__.py
@@ -0,0 +1,194 @@
+from importlib import import_module as _import_module
+from types import ModuleType as _ModuleType
+from typing import TYPE_CHECKING, Any
+import sys as _sys
+
+if TYPE_CHECKING:
+ from ._contracts import (
+ CategoricalScale as CategoricalScale,
+ CellField as CellField,
+ ColorScale as ColorScale,
+ DensityOverlay as DensityOverlay,
+ FeatureRef as FeatureRef,
+ FeatureSummary as FeatureSummary,
+ Highlight as Highlight,
+ NormalizationSpec as NormalizationSpec,
+ PlotProvenance as PlotProvenance,
+ SizeScale as SizeScale,
+ StudyDesign as StudyDesign,
+ )
+ from ._figure import (
+ LegendSpec as LegendSpec,
+ PlotResult as PlotResult,
+ collect_legends as collect_legends,
+ compose_results as compose_results,
+ label_panels as label_panels,
+ )
+ from ._style import THEMES as THEMES
+ from ._style import register_theme as register_theme
+ from ._style import theme_context as theme_context
+ from .recipes import (
+ PlotOutput as PlotOutput,
+ PlotOutputSettings as PlotOutputSettings,
+ PlotPanelTarget as PlotPanelTarget,
+ PlotRecipe as PlotRecipe,
+ PlotRecipeResult as PlotRecipeResult,
+ PlotStep as PlotStep,
+ run_recipe as run_recipe,
+ )
+ from .composition import composition as composition
+ from .cluster_connectivity import cluster_connectivity as cluster_connectivity
+ from .diagnostics import (
+ elbow as elbow,
+ graph_qc as graph_qc,
+ highly_variable_features as highly_variable_features,
+ qc as qc,
+ )
+ from .distribution import distribution as distribution
+ from .embedding import embedding as embedding
+ from .embedding_raster import embedding_raster as embedding_raster
+ from .cluster_tree import cluster_tree as cluster_tree
+ from .heatmaps import (
+ marker_heatmap as marker_heatmap,
+ pseudotime_heatmap as pseudotime_heatmap,
+ )
+ from .mapping import (
+ mapping_calibration as mapping_calibration,
+ mapping_confusion as mapping_confusion,
+ mapping_evidence as mapping_evidence,
+ mapping_projection as mapping_projection,
+ mapping_score as mapping_score,
+ )
+ from .summary import dotplot as dotplot
+ from .summary import matrixplot as matrixplot
+
+__all__ = [
+ "CategoricalScale",
+ "CellField",
+ "ColorScale",
+ "DensityOverlay",
+ "FeatureRef",
+ "FeatureSummary",
+ "Highlight",
+ "LegendSpec",
+ "NormalizationSpec",
+ "PlotProvenance",
+ "PlotOutput",
+ "PlotOutputSettings",
+ "PlotPanelTarget",
+ "PlotRecipe",
+ "PlotRecipeResult",
+ "PlotResult",
+ "PlotStep",
+ "SizeScale",
+ "StudyDesign",
+ "THEMES",
+ "cluster_tree",
+ "cluster_connectivity",
+ "collect_legends",
+ "compose_results",
+ "composition",
+ "distribution",
+ "dotplot",
+ "elbow",
+ "embedding",
+ "embedding_raster",
+ "graph_qc",
+ "highly_variable_features",
+ "label_panels",
+ "marker_heatmap",
+ "mapping_calibration",
+ "mapping_confusion",
+ "mapping_evidence",
+ "mapping_projection",
+ "mapping_score",
+ "matrixplot",
+ "pseudotime_heatmap",
+ "qc",
+ "register_theme",
+ "run_recipe",
+ "theme_context",
+]
+
+_LAZY_EXPORTS: dict[str, tuple[str, str]] = {
+ "CategoricalScale": ("._contracts", "CategoricalScale"),
+ "CellField": ("._contracts", "CellField"),
+ "ColorScale": ("._contracts", "ColorScale"),
+ "DensityOverlay": ("._contracts", "DensityOverlay"),
+ "FeatureRef": ("._contracts", "FeatureRef"),
+ "FeatureSummary": ("._contracts", "FeatureSummary"),
+ "Highlight": ("._contracts", "Highlight"),
+ "LegendSpec": ("._figure", "LegendSpec"),
+ "NormalizationSpec": ("._contracts", "NormalizationSpec"),
+ "PlotProvenance": ("._contracts", "PlotProvenance"),
+ "PlotOutput": (".recipes", "PlotOutput"),
+ "PlotOutputSettings": (".recipes", "PlotOutputSettings"),
+ "PlotPanelTarget": (".recipes", "PlotPanelTarget"),
+ "PlotRecipe": (".recipes", "PlotRecipe"),
+ "PlotRecipeResult": (".recipes", "PlotRecipeResult"),
+ "PlotResult": ("._figure", "PlotResult"),
+ "PlotStep": (".recipes", "PlotStep"),
+ "SizeScale": ("._contracts", "SizeScale"),
+ "StudyDesign": ("._contracts", "StudyDesign"),
+ "THEMES": ("._style", "THEMES"),
+ "cluster_tree": (".cluster_tree", "cluster_tree"),
+ "cluster_connectivity": (".cluster_connectivity", "cluster_connectivity"),
+ "collect_legends": ("._figure", "collect_legends"),
+ "compose_results": ("._figure", "compose_results"),
+ "composition": (".composition", "composition"),
+ "distribution": (".distribution", "distribution"),
+ "dotplot": (".summary", "dotplot"),
+ "elbow": (".diagnostics", "elbow"),
+ "embedding": (".embedding", "embedding"),
+ "embedding_raster": (".embedding_raster", "embedding_raster"),
+ "graph_qc": (".diagnostics", "graph_qc"),
+ "highly_variable_features": (".diagnostics", "highly_variable_features"),
+ "label_panels": ("._figure", "label_panels"),
+ "marker_heatmap": (".heatmaps", "marker_heatmap"),
+ "mapping_calibration": (".mapping", "mapping_calibration"),
+ "mapping_confusion": (".mapping", "mapping_confusion"),
+ "mapping_evidence": (".mapping", "mapping_evidence"),
+ "mapping_projection": (".mapping", "mapping_projection"),
+ "mapping_score": (".mapping", "mapping_score"),
+ "matrixplot": (".summary", "matrixplot"),
+ "pseudotime_heatmap": (".heatmaps", "pseudotime_heatmap"),
+ "qc": (".diagnostics", "qc"),
+ "register_theme": ("._style", "register_theme"),
+ "run_recipe": (".recipes", "run_recipe"),
+ "theme_context": ("._style", "theme_context"),
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+
+def _load_export(name: str) -> Any:
+ export = _LAZY_EXPORTS.get(name)
+ if export is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ module_name, attribute_name = export
+ value = getattr(_import_module(module_name, __name__), attribute_name)
+ globals()[name] = value
+ return value
+
+
+def __getattr__(name: str) -> Any:
+ return _load_export(name)
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()).union(_LAZY_EXPORTS))
+
+
+class _PlottingFacadeModule(_ModuleType):
+ def __getattribute__(self, name: str) -> Any:
+ namespace = _ModuleType.__getattribute__(self, "__dict__")
+ if name in namespace.get("_LAZY_EXPORTS", {}):
+ current = namespace.get(name)
+ if isinstance(current, _ModuleType):
+ return namespace["_load_export"](name)
+ return _ModuleType.__getattribute__(self, name)
+
+
+_sys.modules[__name__].__class__ = _PlottingFacadeModule
diff --git a/scarf/plotting/_contracts.py b/scarf/plotting/_contracts.py
new file mode 100644
index 00000000..e3348b7e
--- /dev/null
+++ b/scarf/plotting/_contracts.py
@@ -0,0 +1,327 @@
+"""Public contracts for scarf.plotting."""
+
+from dataclasses import dataclass, field
+from functools import cache
+from importlib.metadata import version
+from typing import Any, Literal
+
+import numpy as np
+import pandas as pd
+
+
+@cache
+def installed_scarf_version() -> str:
+ """Return the installed scarf version, or "unknown" when it cannot be read."""
+ try:
+ return version("scarf")
+ except Exception:
+ return "unknown"
+
+
+LookupBy = Literal["name", "id", "index"]
+FeatureReduction = Literal["mean", "sum"]
+CellFieldKind = Literal["auto", "categorical", "continuous"]
+NormSource = Literal["assay", "raw"]
+NormTransform = Literal["none", "log1p"]
+Standardize = Literal["none", "feature"]
+LegendLoc = Literal["auto", "right", "on_data", "none"]
+FrameStyle = Literal["axes", "minimal", "none"]
+DistKind = Literal["violin", "stacked_violin", "box", "hist", "ecdf"]
+ContourKind = Literal["line", "filled"]
+
+
+@dataclass(frozen=True, slots=True)
+class FeatureRef:
+ """Reference to one assay feature.
+
+ Parameters:
+ value: Feature name, id, or physical index (see ``by``).
+ assay: Assay name. Defaults to the store default assay when omitted.
+ by: How to look up ``value``: ``name``, ``id``, or ``index``.
+ label: Optional display label.
+ reduction: Required when multiple features match; ``mean`` or ``sum``.
+ """
+
+ value: str | int
+ assay: str | None = None
+ by: LookupBy = "name"
+ label: str | None = None
+ reduction: FeatureReduction | None = None
+
+ def __post_init__(self) -> None:
+ if self.by not in ("name", "id", "index"):
+ raise ValueError("by must be 'name', 'id', or 'index'")
+ if self.reduction not in (None, "mean", "sum"):
+ raise ValueError("reduction must be 'mean', 'sum', or None")
+
+
+@dataclass(frozen=True, slots=True)
+class CellField:
+ """Point ``color_by`` or similar arguments at a cell-metadata column.
+
+ Use this when the column name alone is ambiguous. ``kind="categorical"``
+ forces discrete colors (useful for integer cluster ids).
+ ``kind="continuous"`` forces a colorbar. ``kind="auto"`` chooses from the
+ dtype and number of unique values.
+ """
+
+ key: str
+ kind: CellFieldKind = "auto"
+ label: str | None = None
+
+ def __post_init__(self) -> None:
+ if self.kind not in ("auto", "categorical", "continuous"):
+ raise ValueError("kind must be 'auto', 'categorical', or 'continuous'")
+
+
+@dataclass(frozen=True, slots=True)
+class StudyDesign:
+ """Describe samples and conditions for composition and summary plots.
+
+ ``sample_by`` is the column that identifies biological samples.
+ ``condition_by`` is the experimental condition (for example treatment).
+ For paired composition plots, also set ``subject_by`` or ``pair_by`` so the
+ same donor or pair can be connected across conditions.
+ """
+
+ sample_by: str
+ condition_by: str | None = None
+ subject_by: str | None = None
+ pair_by: str | None = None
+ technical_replicate_by: str | None = None
+ technical_replicate_reduction: Literal["sum", "mean"] | None = None
+
+ def __post_init__(self) -> None:
+ unsupported = [
+ name
+ for name, value in (
+ ("technical_replicate_by", self.technical_replicate_by),
+ (
+ "technical_replicate_reduction",
+ self.technical_replicate_reduction,
+ ),
+ )
+ if value is not None
+ ]
+ if unsupported:
+ raise NotImplementedError(
+ "These StudyDesign fields are not supported yet: "
+ + ", ".join(unsupported)
+ + ". Collapse technical replicates into sample_by first, "
+ "or omit them."
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class NormalizationSpec:
+ """How feature values are read for gene-colored plots.
+
+ ``source="assay"`` uses the assay's current normalization settings.
+ ``source="raw"`` reads raw counts. ``transform="log1p"`` applies log1p
+ after that fetch, which is the usual choice for gene UMAPs and dotplots
+ when you want a compressed expression scale.
+ """
+
+ source: NormSource = "assay"
+ transform: NormTransform = "none"
+
+ def __post_init__(self) -> None:
+ if self.source not in ("assay", "raw"):
+ raise ValueError("source must be 'assay' or 'raw'")
+ if self.transform not in ("none", "log1p"):
+ raise ValueError("transform must be 'none' or 'log1p'")
+
+
+@dataclass(frozen=True, slots=True)
+class ColorScale:
+ """Continuous color mapping for embeddings and summary plots.
+
+ Set ``vmin`` / ``vmax`` for fixed limits, or ``quantiles=(low, high)`` to
+ ignore extreme tails (for example ``(0.0, 0.99)`` clips the top 1%).
+ ``vcenter`` is for diverging maps such as scaled fold changes.
+ ``scope`` controls whether limits are computed per feature (default), per
+ panel, or shared across every continuous panel in the figure.
+ """
+
+ cmap: str | None = None
+ vmin: float | None = None
+ vmax: float | None = None
+ vcenter: float | None = None
+ quantiles: tuple[float, float] | None = None
+ missing_color: str = "#bdbdbd"
+ scope: Literal["feature", "panel", "shared"] = "feature"
+ scale: Literal["linear", "log", "symlog"] = "linear"
+
+ def __post_init__(self) -> None:
+ if self.quantiles is not None:
+ low, high = self.quantiles
+ if not (0.0 <= low < high <= 1.0):
+ raise ValueError("quantiles must satisfy 0 <= low < high <= 1")
+ if self.vmin is not None and self.vmax is not None and self.vmax < self.vmin:
+ raise ValueError("vmax must be greater than or equal to vmin")
+ if (
+ self.vcenter is not None
+ and self.vmin is not None
+ and self.vmax is not None
+ and not self.vmin < self.vcenter < self.vmax
+ ):
+ raise ValueError("vcenter must be strictly between vmin and vmax")
+ if self.scope not in ("feature", "panel", "shared"):
+ raise ValueError("scope must be 'feature', 'panel', or 'shared'")
+ if self.scale not in ("linear", "log", "symlog"):
+ raise ValueError("scale must be 'linear', 'log', or 'symlog'")
+
+
+@dataclass(frozen=True, slots=True)
+class CategoricalScale:
+ """Category order and colors for discrete embeddings and compositions.
+
+ ``order`` sets legend and axis order. ``palette`` maps each category to a
+ color string. Categories missing from ``palette`` raise an error, so pass
+ a complete map when you customize colors.
+ """
+
+ order: tuple[Any, ...] | None = None
+ palette: dict[Any, str] | None = None
+ labels: dict[Any, str] | None = None
+ missing_color: str = "#bdbdbd"
+ missing_label: str = "NA"
+ palette_name: Literal["default", "colorblind"] = "default"
+
+ def __post_init__(self) -> None:
+ if self.palette_name not in ("default", "colorblind"):
+ raise ValueError("palette_name must be 'default' or 'colorblind'")
+
+
+@dataclass(frozen=True, slots=True)
+class SizeScale:
+ """Maps numeric values to marker area. Detection fraction uses domain [0, 1]."""
+
+ vmin: float = 0.0
+ vmax: float = 1.0
+ size_min: float = 10.0
+ size_max: float = 200.0
+
+ def __post_init__(self) -> None:
+ if self.size_min < 0 or self.size_max < self.size_min:
+ raise ValueError("size range must satisfy 0 <= size_min <= size_max")
+
+ def areas(self, values: np.ndarray) -> np.ndarray:
+ v = np.asarray(values, dtype=np.float64)
+ span = self.vmax - self.vmin
+ if span <= 0:
+ return np.full(v.shape, self.size_min, dtype=np.float64)
+ t = np.clip((v - self.vmin) / span, 0.0, 1.0)
+ return self.size_min + t * (self.size_max - self.size_min)
+
+
+@dataclass(frozen=True, slots=True)
+class DensityOverlay:
+ """Contours drawn over an embedding.
+
+ ``group_by`` and ``groups`` optionally restrict the cells used to estimate
+ the surface without changing the cells displayed underneath.
+ ``statistic="mean"`` contours a smoothed local mean of the continuous
+ panel values and attenuates pixels with less than ``min_support`` effective
+ support. ``max_hotspots`` can retain only the strongest connected regions
+ at the lowest contour level.
+ """
+
+ kind: ContourKind = "line"
+ statistic: Literal["density", "mean"] = "density"
+ pixels: int = 160
+ sigma: float = 2.0
+ min_support: float = 0.25
+ levels: int | tuple[float, ...] = 5
+ max_hotspots: int | None = None
+ group_by: str | None = None
+ groups: tuple[Any, ...] | None = None
+ color: str | None = None
+ cmap: str | None = None
+ alpha: float = 0.75
+ linewidth: float = 0.8
+ halo_color: str | None = None
+ halo_width: float = 0.0
+ zorder: float = 4.0
+
+ def __post_init__(self) -> None:
+ if self.kind not in ("line", "filled"):
+ raise ValueError("kind must be 'line' or 'filled'")
+ if self.statistic not in ("density", "mean"):
+ raise ValueError("statistic must be 'density' or 'mean'")
+ if self.pixels < 16:
+ raise ValueError("pixels must be at least 16")
+ if self.sigma < 0:
+ raise ValueError("sigma must be non-negative")
+ if self.min_support <= 0:
+ raise ValueError("min_support must be positive")
+ if isinstance(self.levels, int):
+ if self.levels < 1:
+ raise ValueError("levels must be positive")
+ elif not self.levels or any(not np.isfinite(value) for value in self.levels):
+ raise ValueError("levels must contain finite values")
+ if self.max_hotspots is not None and (
+ isinstance(self.max_hotspots, bool)
+ or not isinstance(self.max_hotspots, int)
+ or self.max_hotspots < 1
+ ):
+ raise ValueError("max_hotspots must be a positive integer")
+ if self.groups is not None and self.group_by is None:
+ raise ValueError("groups requires group_by")
+ if not 0 <= self.alpha <= 1:
+ raise ValueError("alpha must be between 0 and 1")
+ if self.linewidth < 0:
+ raise ValueError("linewidth must be non-negative")
+ if self.halo_width < 0:
+ raise ValueError("halo_width must be non-negative")
+
+
+@dataclass(frozen=True, slots=True)
+class Highlight:
+ """Emphasize a metadata selection while retaining surrounding context."""
+
+ by: str | None = None
+ groups: tuple[Any, ...] | None = None
+ indices: tuple[int, ...] | None = None
+ color: str = "#d62728"
+ dim_alpha: float = 0.12
+ alpha: float = 1.0
+ size_multiplier: float = 1.5
+ halo_color: str | None = None
+ halo_width: float = 0.8
+
+ def __post_init__(self) -> None:
+ if (self.by is None) == (self.indices is None):
+ raise ValueError("Set exactly one of by or indices")
+ if self.by == "":
+ raise ValueError("by must be non-empty")
+ if self.indices is not None and any(index < 0 for index in self.indices):
+ raise ValueError("indices must be non-negative")
+ if self.groups is not None and self.by is None:
+ raise ValueError("groups requires by")
+ if not 0 <= self.dim_alpha <= 1 or not 0 <= self.alpha <= 1:
+ raise ValueError("highlight alpha values must be between 0 and 1")
+ if self.size_multiplier <= 0:
+ raise ValueError("size_multiplier must be positive")
+ if self.halo_width < 0:
+ raise ValueError("halo_width must be non-negative")
+
+
+@dataclass(frozen=True, slots=True)
+class PlotProvenance:
+ scarf_version: str = field(default_factory=installed_scarf_version)
+ assay: str | None = None
+ cell_key: str | None = None
+ n_cells: int = 0
+ n_samples: int | None = None
+ renderer: str = "matplotlib"
+ notes: tuple[str, ...] = ()
+ extras: dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass(slots=True)
+class FeatureSummary:
+ """Bounded tables behind summary plots."""
+
+ aggregate: pd.DataFrame
+ per_sample: pd.DataFrame | None = None
diff --git a/scarf/plotting/_data.py b/scarf/plotting/_data.py
new file mode 100644
index 00000000..ce1d073b
--- /dev/null
+++ b/scarf/plotting/_data.py
@@ -0,0 +1,366 @@
+"""Feature resolution and bounded group reducers."""
+
+from collections.abc import Mapping, Sequence
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+import pandas as pd
+
+from ._contracts import (
+ FeatureRef,
+ FeatureReduction,
+ LookupBy,
+ NormalizationSpec,
+ Standardize,
+ StudyDesign,
+)
+from ._style import sort_categories
+
+
+@dataclass(frozen=True, slots=True)
+class ResolvedFeature:
+ assay: str
+ by: LookupBy
+ indices: tuple[int, ...]
+ ids: tuple[str, ...]
+ names: tuple[str, ...]
+ label: str
+ reduction: FeatureReduction | None
+ raw: FeatureRef | str
+
+ def scale_key(
+ self,
+ normalization: NormalizationSpec,
+ standardize: Standardize = "none",
+ ) -> tuple[Any, ...]:
+ return (
+ self.assay,
+ self.by,
+ self.ids,
+ self.reduction,
+ normalization.source,
+ normalization.transform,
+ standardize,
+ )
+
+
+def coerce_feature_list(
+ features: Sequence[str | FeatureRef] | Mapping[str, Sequence[str | FeatureRef]],
+) -> list[tuple[str | None, str | FeatureRef]]:
+ """Return (group_label, feature) pairs preserving order."""
+ if isinstance(features, Mapping):
+ out: list[tuple[str | None, str | FeatureRef]] = []
+ for group, items in features.items():
+ for item in items:
+ out.append((str(group), item))
+ return out
+ return [(None, item) for item in features]
+
+
+def resolve_cell_selection(
+ n: int,
+ *,
+ subset: np.ndarray | None = None,
+ subset_name: str | None = None,
+ category_values: np.ndarray | None = None,
+ groups: Sequence[Any] | None = None,
+) -> tuple[np.ndarray, list[Any] | None]:
+ """Build a boolean mask from ``subset`` and optional category ``groups``.
+
+ ``subset`` must be boolean and length ``n`` when provided. ``groups`` keeps
+ only those categories from ``category_values`` and defines their order.
+ When ``groups`` is omitted, category order is natural via
+ :func:`sort_categories` over observed values (or ``None`` if no categories).
+ """
+ mask = np.ones(n, dtype=bool)
+ if subset is not None:
+ sub = np.asarray(subset)
+ if sub.dtype != bool:
+ label = subset_name or "subset_by"
+ raise TypeError(f"{label!r} must be boolean; got {sub.dtype}")
+ if len(sub) != n:
+ raise ValueError("subset_by length must match selected cells")
+ mask &= sub
+
+ group_order: list[Any] | None = None
+ if category_values is not None:
+ cats = np.asarray(category_values)
+ if len(cats) != n:
+ raise ValueError("category values length must match selected cells")
+ present = set(pd.unique(cats).tolist())
+ if groups is not None:
+ group_order = list(groups)
+ if not group_order:
+ raise ValueError("groups must be non-empty when provided")
+ missing = [g for g in group_order if g not in present]
+ if missing:
+ raise ValueError(
+ "groups contains labels not present in the data: "
+ + ", ".join(map(str, missing[:10]))
+ )
+ mask &= np.isin(cats, group_order)
+ elif mask.any():
+ group_order = sort_categories(list(pd.unique(cats[mask])))
+ else:
+ group_order = []
+
+ if not mask.any():
+ raise ValueError("No cells remain after applying subset/groups filters")
+ return mask, group_order
+
+
+def resolve_feature(
+ store: Any,
+ feature: str | FeatureRef,
+ *,
+ from_assay: str | None = None,
+) -> ResolvedFeature:
+ """Resolve a feature against one assay. Case-sensitive. No silent averaging."""
+ if isinstance(feature, FeatureRef):
+ ref = feature
+ else:
+ ref = FeatureRef(value=feature, assay=from_assay)
+
+ assay_name = ref.assay or from_assay or store._defaultAssay
+ assay = store._get_assay(assay_name)
+
+ if ref.by == "index":
+ idx = int(ref.value)
+ if idx < 0 or idx >= assay.feats.N:
+ raise KeyError(
+ f"Feature index {idx} out of range for assay {assay_name!r} "
+ f"(N={assay.feats.N})"
+ )
+ indices = [idx]
+ elif ref.by == "id":
+ indices = list(assay.feats.get_index_by([str(ref.value)], "ids"))
+ else:
+ indices = list(assay.feats.get_index_by([str(ref.value)], "names"))
+
+ if len(indices) == 0:
+ raise KeyError(
+ f"Feature {ref.value!r} not found in assay {assay_name!r} by {ref.by!r}"
+ )
+ if len(indices) > 1 and ref.reduction is None:
+ raise ValueError(
+ f"Feature {ref.value!r} matches {len(indices)} entries in assay "
+ f"{assay_name!r} at indices {indices}. Pass reduction='mean' or "
+ f"reduction='sum', or look up by id/index."
+ )
+
+ idx_arr = np.asarray(indices)
+ names = tuple(str(x) for x in assay.feats.fetch_all("names")[idx_arr])
+ ids = tuple(str(x) for x in assay.feats.fetch_all("ids")[idx_arr])
+ label = ref.label or (
+ names[0] if len(names) == 1 else f"{ref.value}:{ref.reduction}"
+ )
+ return ResolvedFeature(
+ assay=assay_name,
+ by=ref.by,
+ indices=tuple(int(i) for i in indices),
+ ids=ids,
+ names=names,
+ label=label,
+ reduction=ref.reduction,
+ raw=ref if isinstance(feature, FeatureRef) else feature,
+ )
+
+
+def fetch_normalized_feature_matrix(
+ store: Any,
+ resolved: Sequence[ResolvedFeature],
+ cell_idx: np.ndarray,
+ normalization: NormalizationSpec | None = None,
+) -> np.ndarray:
+ """Return assay-native or raw feature values in requested feature order."""
+ from ..utils.compute import controlled_compute
+
+ normalization = normalization or NormalizationSpec()
+ if not resolved:
+ return np.empty((len(cell_idx), 0), dtype=np.float64)
+
+ output = np.empty((len(cell_idx), len(resolved)), dtype=np.float64)
+ assay_slots: dict[str, list[int]] = {}
+ for slot, feat in enumerate(resolved):
+ assay_slots.setdefault(feat.assay, []).append(slot)
+
+ for assay_name, slots in assay_slots.items():
+ assay = store._get_assay(assay_name)
+ physical_indices = np.unique(
+ np.concatenate(
+ [np.asarray(resolved[slot].indices, dtype=np.int64) for slot in slots]
+ )
+ )
+ if normalization.source == "raw":
+ values = assay.rawData[:, physical_indices][cell_idx, :]
+ else:
+ values = assay.normed(
+ cell_idx=cell_idx,
+ feat_idx=physical_indices,
+ )
+ normalized = controlled_compute(values, store.nthreads).astype(np.float64)
+ if normalized.ndim == 1:
+ normalized = normalized.reshape(-1, 1)
+ if normalization.transform == "log1p":
+ normalized = np.log1p(normalized)
+
+ for slot in slots:
+ feat = resolved[slot]
+ local = np.searchsorted(
+ physical_indices, np.asarray(feat.indices, dtype=np.int64)
+ )
+ values = normalized[:, local]
+ if values.shape[1] == 1:
+ output[:, slot] = values[:, 0]
+ elif feat.reduction == "sum":
+ output[:, slot] = values.sum(axis=1)
+ else:
+ output[:, slot] = values.mean(axis=1)
+ return output
+
+
+def summarize_features_by_group(
+ store: Any,
+ *,
+ features: Sequence[str | FeatureRef] | Mapping[str, Sequence[str | FeatureRef]],
+ group_by: str | tuple[str, ...],
+ cell_key: str = "I",
+ from_assay: str | None = None,
+ sample_by: str | None = None,
+ study_design: StudyDesign | None = None,
+ normalization: NormalizationSpec | None = None,
+ expression_cutoff: float = 0.0,
+ max_groups: int = 500,
+ max_features: int = 2000,
+ max_samples: int = 500,
+) -> tuple[pd.DataFrame, pd.DataFrame | None]:
+ """Aggregate features by group. With sample_by, samples get equal weight.
+
+ Missing group combinations are omitted (not filled with zeros).
+ """
+ condition_by: str | None = None
+ if study_design is not None:
+ sample_by = study_design.sample_by
+ condition_by = study_design.condition_by
+
+ pairs = coerce_feature_list(features)
+ if len(pairs) > max_features:
+ raise ValueError(
+ f"Too many features ({len(pairs)} > {max_features}). "
+ "Raise max_features explicitly if intentional."
+ )
+ resolved = [
+ resolve_feature(store, feat, from_assay=from_assay) for _, feat in pairs
+ ]
+ group_labels = [g for g, _ in pairs]
+ feature_labels = [r.label for r in resolved]
+
+ if isinstance(group_by, str):
+ group_keys: tuple[str, ...] = (group_by,)
+ else:
+ group_keys = tuple(group_by)
+ if len(group_keys) == 0 or len(group_keys) > 2:
+ raise ValueError("group_by must have 1 or 2 keys")
+
+ cells = store.cells
+ cell_idx = cells.active_index(cell_key)
+ group_cols = [cells.fetch(k, key=cell_key) for k in group_keys]
+ n_groups = int(
+ pd.DataFrame({k: c for k, c in zip(group_keys, group_cols)})
+ .drop_duplicates()
+ .shape[0]
+ )
+ if n_groups > max_groups:
+ raise ValueError(
+ f"Too many groups ({n_groups} > {max_groups}). "
+ "Raise max_groups explicitly if intentional."
+ )
+
+ expr = fetch_normalized_feature_matrix(
+ store,
+ resolved,
+ cell_idx,
+ normalization=normalization,
+ )
+ frac_mask = expr > expression_cutoff
+ base = pd.DataFrame({gk: col for gk, col in zip(group_keys, group_cols)})
+
+ if sample_by is not None:
+ samples = cells.fetch(sample_by, key=cell_key)
+ if condition_by is not None:
+ conditions = cells.fetch(condition_by, key=cell_key)
+ check = pd.DataFrame({"sample": samples, "condition": conditions})
+ nunique = check.groupby("sample", observed=False)["condition"].nunique()
+ bad = nunique[nunique > 1]
+ if len(bad):
+ raise ValueError(
+ "condition_by is not constant within sample(s): "
+ + ", ".join(map(str, list(bad.index[:10])))
+ )
+ valid = pd.notna(samples) & (np.asarray(samples, dtype=object) != "")
+ if int(valid.sum()) == 0:
+ raise ValueError("No cells with valid sample_by values")
+ uniq_samples = pd.unique(np.asarray(samples)[valid])
+ if len(uniq_samples) > max_samples:
+ raise ValueError(
+ f"Too many samples ({len(uniq_samples)} > {max_samples}). "
+ "Raise max_samples explicitly if intentional."
+ )
+ parts: list[pd.DataFrame] = []
+ base_v = base.loc[valid].copy()
+ base_v["sample"] = np.asarray(samples)[valid]
+ for j in range(len(resolved)):
+ part = base_v.copy()
+ part["feature"] = feature_labels[j]
+ part["feature_group"] = group_labels[j]
+ part["value"] = expr[valid, j]
+ part["detected"] = frac_mask[valid, j]
+ parts.append(part)
+ long = pd.concat(parts, ignore_index=True)
+ gb_keys = ["sample", *group_keys, "feature", "feature_group"]
+ per_sample = (
+ long.groupby(gb_keys, observed=False, dropna=False)
+ .agg(
+ mean=("value", "mean"),
+ fraction=("detected", "mean"),
+ n_cells=("value", "size"),
+ variance=("value", "var"),
+ )
+ .reset_index()
+ )
+ agg_keys = [*group_keys, "feature", "feature_group"]
+ aggregate = (
+ per_sample.groupby(agg_keys, observed=False, dropna=False)
+ .agg(
+ mean=("mean", "mean"),
+ fraction=("fraction", "mean"),
+ n_cells=("n_cells", "sum"),
+ n_samples=("sample", "nunique"),
+ variance=("variance", "mean"),
+ )
+ .reset_index()
+ )
+ return aggregate, per_sample
+
+ parts = []
+ for j in range(len(resolved)):
+ part = base.copy()
+ part["feature"] = feature_labels[j]
+ part["feature_group"] = group_labels[j]
+ part["value"] = expr[:, j]
+ part["detected"] = frac_mask[:, j]
+ parts.append(part)
+ long = pd.concat(parts, ignore_index=True)
+ agg_keys = [*group_keys, "feature", "feature_group"]
+ aggregate = (
+ long.groupby(agg_keys, observed=False, dropna=False)
+ .agg(
+ mean=("value", "mean"),
+ fraction=("detected", "mean"),
+ n_cells=("value", "size"),
+ variance=("value", "var"),
+ )
+ .reset_index()
+ )
+ return aggregate, None
diff --git a/scarf/plotting/_deps.py b/scarf/plotting/_deps.py
new file mode 100644
index 00000000..a9142817
--- /dev/null
+++ b/scarf/plotting/_deps.py
@@ -0,0 +1,34 @@
+"""Lazy loaders for optional plotting dependencies."""
+
+from typing import Any
+
+
+def require_matplotlib() -> tuple[Any, Any]:
+ try:
+ import matplotlib as mpl
+ import matplotlib.pyplot as plt
+ except ImportError as exc:
+ raise ImportError(
+ "Scarf plotting requires matplotlib. Install with: pip install 'scarf[extra]'"
+ ) from exc
+ return plt, mpl
+
+
+def require_seaborn() -> Any:
+ try:
+ import seaborn as sns
+ except ImportError as exc:
+ raise ImportError(
+ "Scarf plotting requires seaborn. Install with: pip install 'scarf[extra]'"
+ ) from exc
+ return sns
+
+
+def require_kneed() -> Any:
+ try:
+ from kneed import KneeLocator
+ except ImportError as exc:
+ raise ImportError(
+ "Scarf elbow detection requires kneed. Install with: pip install 'scarf[extra]'"
+ ) from exc
+ return KneeLocator
diff --git a/scarf/plotting/_display.py b/scarf/plotting/_display.py
new file mode 100644
index 00000000..648a9e33
--- /dev/null
+++ b/scarf/plotting/_display.py
@@ -0,0 +1,39 @@
+"""Resolve stored display metadata into plotting scale contracts."""
+
+from typing import Any
+
+from ..metadata.artifacts import column_display
+from ._contracts import CategoricalScale
+
+
+def stored_display_metadata(store: Any, column: str) -> dict[str, Any] | None:
+ """Return validated display metadata for a cell column, when available."""
+ try:
+ root = store.zw
+ except AttributeError:
+ return None
+ return column_display(root, column)
+
+
+def stored_categorical_scale(store: Any, column: str) -> CategoricalScale | None:
+ """Resolve a stored categorical display contract for one cell column."""
+ display = stored_display_metadata(store, column)
+ if display is None or display["kind"] != "categorical":
+ return None
+ categories = display["categories"]
+ return CategoricalScale(
+ order=tuple(category["value"] for category in categories),
+ palette={category["value"]: str(category["color"]) for category in categories},
+ labels={category["value"]: str(category["label"]) for category in categories},
+ missing_color=str(display.get("missing_color", "#bdbdbd")),
+ missing_label=str(display.get("missing_label", "NA")),
+ )
+
+
+def resolve_categorical_scale(
+ store: Any,
+ column: str,
+ explicit: CategoricalScale | None,
+) -> CategoricalScale | None:
+ """Prefer an explicit scale, then stored cell display metadata."""
+ return explicit if explicit is not None else stored_categorical_scale(store, column)
diff --git a/scarf/plotting/_figure.py b/scarf/plotting/_figure.py
new file mode 100644
index 00000000..d916836f
--- /dev/null
+++ b/scarf/plotting/_figure.py
@@ -0,0 +1,730 @@
+"""PlotResult and figure ownership helpers."""
+
+import json
+from collections.abc import Mapping, Sequence
+from dataclasses import asdict, dataclass, field, is_dataclass
+from pathlib import Path
+from typing import Any, Hashable, cast
+
+import numpy as np
+import pandas as pd
+
+from ._contracts import CategoricalScale, ColorScale, PlotProvenance, SizeScale
+from ._deps import require_matplotlib
+from ._style import refresh_layout_point_sizes, theme_context
+
+
+def _json_ready(value: Any) -> Any:
+ if not isinstance(value, type) and is_dataclass(value):
+ return _json_ready(asdict(cast(Any, value)))
+ if isinstance(value, Mapping):
+ return {str(key): _json_ready(item) for key, item in value.items()}
+ if isinstance(value, (list, tuple, set)):
+ return [_json_ready(item) for item in value]
+ if isinstance(value, np.ndarray):
+ return _json_ready(value.tolist())
+ if isinstance(value, np.generic):
+ return _json_ready(value.item())
+ if isinstance(value, Path):
+ return str(value)
+ if isinstance(value, float):
+ return value if np.isfinite(value) else None
+ if value is None or isinstance(value, (str, int, bool)):
+ return value
+ raise TypeError(f"Value of type {type(value).__name__} is not JSON serializable")
+
+
+@dataclass(slots=True)
+class LegendSpec:
+ """Description of a legend or colorbar attached to a plot."""
+
+ kind: str
+ label: str | None = None
+ scale_key: Hashable | None = None
+ extras: dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass(slots=True)
+class PlotResult:
+ """Return value from scarf.plotting functions.
+
+ ``figure`` is the matplotlib figure. ``axes`` maps panel keys to axes.
+ ``tables`` holds any summary data used to build the plot. ``owns_figure``
+ is True when Scarf created the figure. ``show()`` and ``close()`` release
+ owned figures without discarding the result metadata.
+ """
+
+ figure: Any
+ axes: dict[Hashable, Any]
+ tables: dict[str, pd.DataFrame]
+ legends: tuple[LegendSpec, ...]
+ scales: tuple[Any, ...]
+ provenance: PlotProvenance
+ owns_figure: bool
+ theme: str = "notebook"
+ _rendered: bool = field(default=False, init=False, repr=False, compare=False)
+
+ def show(self) -> None:
+ plt, _ = require_matplotlib()
+ in_ipython = False
+ try:
+ from IPython import get_ipython
+
+ in_ipython = get_ipython() is not None # type: ignore[no-untyped-call]
+ except ImportError:
+ pass
+ try:
+ if in_ipython:
+ from IPython.display import display
+
+ display(self.figure) # type: ignore[no-untyped-call]
+ self._rendered = True
+ return
+ if (
+ getattr(
+ self.figure.canvas,
+ "required_interactive_framework",
+ None,
+ )
+ is not None
+ ):
+ plt.show()
+ self._rendered = True
+ finally:
+ if self.owns_figure:
+ plt.close(self.figure)
+
+ def __repr__(self) -> str:
+ return (
+ "PlotResult("
+ f"axes={len(self.axes)}, "
+ f"tables={len(self.tables)}, "
+ f"legends={len(self.legends)}, "
+ f"scales={len(self.scales)}, "
+ f"owns_figure={self.owns_figure}, "
+ f"rendered={self._rendered}"
+ ")"
+ )
+
+ def _ipython_display_(self) -> None:
+ # show() already displayed the figure, so echoing the summary repr
+ # underneath it only adds noise. Results that were never displayed keep
+ # their plain-text repr.
+ if self._rendered:
+ return
+ from IPython.display import display
+
+ display({"text/plain": repr(self)}, raw=True) # type: ignore[no-untyped-call]
+
+ def close(self) -> None:
+ if not self.owns_figure:
+ return
+ plt, _ = require_matplotlib()
+ plt.close(self.figure)
+
+ def save_provenance(
+ self,
+ path: str | Path,
+ *,
+ figure_path: str | Path | None = None,
+ dpi: float | None = None,
+ ) -> Path:
+ """Write a JSON file describing scales, legends, and cell counts."""
+ out = Path(path)
+ out.parent.mkdir(parents=True, exist_ok=True)
+ width, height = self.figure.get_size_inches()
+ payload = {
+ "provenance": self.provenance,
+ "figure": {
+ "width_inches": float(width),
+ "height_inches": float(height),
+ "dpi": float(self.figure.dpi),
+ },
+ "legends": self.legends,
+ "scales": [
+ {
+ "type": type(scale).__name__,
+ "values": scale,
+ }
+ for scale in self.scales
+ ],
+ "tables": {
+ name: {
+ "rows": len(table),
+ "columns": [str(column) for column in table.columns],
+ }
+ for name, table in self.tables.items()
+ },
+ }
+ if figure_path is not None:
+ exported = Path(figure_path)
+ payload["export"] = {
+ "filename": exported.name,
+ "format": exported.suffix.lower().lstrip("."),
+ "dpi": float(dpi if dpi is not None else self.figure.dpi),
+ }
+ out.write_text(
+ json.dumps(
+ _json_ready(payload),
+ indent=2,
+ sort_keys=True,
+ allow_nan=False,
+ )
+ + "\n",
+ encoding="utf-8",
+ )
+ return out
+
+ def save(
+ self,
+ path: str | Path,
+ *,
+ dpi: int | None = None,
+ transparent: bool = False,
+ exact_size: bool = True,
+ tiff_compression: str = "tiff_lzw",
+ provenance_sidecar: bool | str | Path = False,
+ ) -> Path:
+ """Write the figure to disk (PNG, PDF, SVG, TIFF, and other backends).
+
+ The default background is opaque white for light themes and charcoal
+ for the dark theme. Pass ``transparent=True`` when the destination
+ supplies its own background. ``exact_size=True`` keeps the inch size you set
+ when the figure was created; set it to False only when you want a tight
+ crop for display. ``provenance_sidecar=True`` writes a sibling JSON
+ file with the plot metadata.
+ """
+ out = Path(path)
+ if not out.suffix:
+ raise ValueError("Export path must include a file extension")
+ if dpi is not None and dpi <= 0:
+ raise ValueError("dpi must be positive")
+ file_type = out.suffix.lower().lstrip(".")
+ supported = self.figure.canvas.get_supported_filetypes()
+ if file_type not in supported:
+ raise ValueError(
+ f"Unsupported export format {out.suffix!r}; choose from "
+ f"{sorted(supported)}"
+ )
+ out.parent.mkdir(parents=True, exist_ok=True)
+ kwargs: dict[str, Any] = {"transparent": transparent}
+ if dpi is not None:
+ kwargs["dpi"] = dpi
+ elif file_type in ("tif", "tiff"):
+ kwargs["dpi"] = 300
+ if file_type in ("tif", "tiff"):
+ kwargs["pil_kwargs"] = {"compression": tiff_compression}
+ # Exact-size export must not crop; tight bbox changes physical size.
+ kwargs["bbox_inches"] = None if exact_size else "tight"
+ _, mpl = require_matplotlib()
+ # Honour explicit transparent=False even when the active theme prefers none.
+ export_rc: dict[str, Any] = {"savefig.bbox": None} if exact_size else {}
+ figure_face = self.figure.patch.get_facecolor()
+ figure_alpha = self.figure.patch.get_alpha()
+ axes_faces = [
+ (ax, ax.patch.get_facecolor(), ax.patch.get_alpha())
+ for ax in self.figure.axes
+ ]
+ if transparent:
+ export_rc["savefig.transparent"] = True
+ export_rc["savefig.facecolor"] = "none"
+ else:
+ background = "#111111" if self.theme == "dark" else "white"
+ export_rc["savefig.transparent"] = False
+ export_rc["savefig.facecolor"] = background
+ self.figure.patch.set_facecolor(background)
+ self.figure.patch.set_alpha(1.0)
+ for ax in self.figure.axes:
+ ax.patch.set_facecolor(background)
+ ax.patch.set_alpha(1.0)
+ try:
+ with theme_context(self.theme), mpl.rc_context(export_rc):
+ self.figure.savefig(out, **kwargs)
+ finally:
+ self.figure.patch.set_facecolor(figure_face)
+ self.figure.patch.set_alpha(figure_alpha)
+ for ax, face, alpha in axes_faces:
+ ax.patch.set_facecolor(face)
+ ax.patch.set_alpha(alpha)
+ if provenance_sidecar:
+ sidecar = (
+ out.with_suffix(out.suffix + ".json")
+ if provenance_sidecar is True
+ else Path(provenance_sidecar)
+ )
+ if sidecar.resolve() == out.resolve():
+ raise ValueError("Provenance sidecar path must differ from figure path")
+ self.save_provenance(
+ sidecar,
+ figure_path=out,
+ dpi=float(kwargs.get("dpi", self.figure.dpi)),
+ )
+ return out
+
+
+def normalize_axes_target(
+ target: Any | None,
+ *,
+ panel_keys: Sequence[Hashable],
+ figsize: tuple[float, float] | None,
+ n_columns: int | None = None,
+) -> tuple[Any, dict[Hashable, Any], bool]:
+ """Return (figure, axes_map, owns_figure)."""
+ plt, _ = require_matplotlib()
+ keys = list(panel_keys)
+ if not keys:
+ raise ValueError("panel_keys must be non-empty")
+
+ if target is None:
+ if figsize is None:
+ from ._style import DEFAULT_PANEL_INCHES
+
+ panel = DEFAULT_PANEL_INCHES
+ figsize = (
+ (panel * len(keys), panel) if len(keys) > 1 else (panel + 0.4, panel)
+ )
+ if len(keys) == 1:
+ fig, ax = plt.subplots(1, 1, figsize=figsize, layout="constrained")
+ return fig, {keys[0]: ax}, True
+ ncols = n_columns if n_columns is not None else min(len(keys), 4)
+ ncols = max(1, min(ncols, len(keys)))
+ nrows = int(np.ceil(len(keys) / ncols))
+ fig, axs = plt.subplots(
+ nrows, ncols, figsize=figsize, squeeze=False, layout="constrained"
+ )
+ mapping: dict[Hashable, Any] = {}
+ for i, key in enumerate(keys):
+ mapping[key] = axs[i // ncols, i % ncols]
+ for j in range(len(keys), nrows * ncols):
+ fig.delaxes(axs[j // ncols, j % ncols])
+ return fig, mapping, True
+
+ if figsize is not None:
+ raise ValueError("figsize is invalid when a caller-owned target is provided")
+
+ if isinstance(target, Mapping):
+ missing = [k for k in keys if k not in target]
+ if missing:
+ raise KeyError(f"target mapping missing panel keys: {missing}")
+ mapping = {k: target[k] for k in keys}
+ figures = {id(ax.figure): ax.figure for ax in mapping.values()}
+ if len(figures) != 1:
+ raise ValueError("All target axes must belong to the same figure")
+ return next(iter(figures.values())), mapping, False
+
+ if isinstance(target, (list, tuple)) or (
+ isinstance(target, np.ndarray) and target.dtype == object
+ ):
+ arr = np.asarray(target, dtype=object).ravel()
+ if len(arr) != len(keys):
+ raise ValueError(f"Expected {len(keys)} axes for panels, got {len(arr)}")
+ figures = {id(ax.figure): ax.figure for ax in arr}
+ if len(figures) != 1:
+ raise ValueError("All target axes must belong to the same figure")
+ fig = next(iter(figures.values()))
+ return fig, {k: ax for k, ax in zip(keys, arr)}, False
+
+ if len(keys) != 1:
+ raise TypeError(
+ "A single Axes target is only valid for one-panel plots; "
+ "pass a sequence or mapping of axes for faceted plots"
+ )
+ return target.figure, {keys[0]: target}, False
+
+
+def label_panels(
+ axes: Mapping[Hashable, Any] | Sequence[Any],
+ *,
+ labels: Sequence[str] | None = None,
+ fontsize: float | str | None = None,
+ fontweight: str = "bold",
+ x: float = -0.06,
+ y: float = 1.04,
+) -> None:
+ """Add A/B/C panel labels to axes in order."""
+ if fontsize is None:
+ _, mpl = require_matplotlib()
+ fontsize = mpl.rcParams["axes.titlesize"]
+ if isinstance(axes, Mapping):
+ ax_list = list(axes.values())
+ else:
+ ax_list = list(axes)
+ if labels is None:
+ labels = [chr(ord("A") + i) for i in range(len(ax_list))]
+ if len(labels) != len(ax_list):
+ raise ValueError("labels length must match number of axes")
+ for ax, lab in zip(ax_list, labels):
+ ax.text(
+ x,
+ y,
+ lab,
+ transform=ax.transAxes,
+ ha="left",
+ va="bottom",
+ fontsize=fontsize,
+ fontweight=fontweight,
+ )
+
+
+def collect_legends(
+ figure: Any,
+ results: Sequence[PlotResult],
+) -> tuple[LegendSpec, ...]:
+ """Combine legend/colorbar descriptors from several plot results."""
+ legends: list[LegendSpec] = []
+ for result in results:
+ legends.extend(result.legends)
+ out = tuple(legends)
+ figure._scarf_legends = out # type: ignore[attr-defined]
+ return out
+
+
+def _deduplicate_legend_specs(
+ legends: Sequence[LegendSpec],
+) -> tuple[LegendSpec, ...]:
+ seen: set[str] = set()
+ out: list[LegendSpec] = []
+ for legend in legends:
+ key = json.dumps(_json_ready(legend), sort_keys=True)
+ if key in seen:
+ continue
+ seen.add(key)
+ out.append(legend)
+ return tuple(out)
+
+
+_OUTSIDE_LEGEND_SLOTS = {
+ 1: ("outside right center",),
+ 2: ("outside right upper", "outside right lower"),
+ 3: (
+ "outside right upper",
+ "outside right center",
+ "outside right lower",
+ ),
+}
+_MAX_OUTSIDE_LEGENDS = max(_OUTSIDE_LEGEND_SLOTS)
+
+
+def _merged_legend_block(
+ blocks: Sequence[tuple[str | None, list[Any], list[str]]],
+) -> tuple[str | None, list[Any], list[str]]:
+ merged_handles: list[Any] = []
+ merged_labels: list[str] = []
+ for title, handles, labels in blocks:
+ merged_handles.extend(handles)
+ merged_labels.extend(
+ f"{title}: {label}" if title else label for label in labels
+ )
+ return None, merged_handles, merged_labels
+
+
+def _legend_blocks_overlap(figure: Any, legends: Sequence[Any]) -> bool:
+ if len(legends) < 2:
+ return False
+ figure.canvas.draw()
+ renderer = figure.canvas.get_renderer()
+ boxes = [legend.get_window_extent(renderer) for legend in legends]
+ return any(
+ boxes[first].overlaps(boxes[second])
+ for first in range(len(boxes))
+ for second in range(first + 1, len(boxes))
+ )
+
+
+def _place_legend_blocks(
+ figure: Any,
+ blocks: Sequence[tuple[str | None, list[Any], list[str]]],
+) -> None:
+ """Draw outside legend blocks, merging them when separate blocks collide."""
+ if not blocks:
+ return
+ placed = list(blocks)
+ if len(placed) > _MAX_OUTSIDE_LEGENDS:
+ keep = list(placed[: _MAX_OUTSIDE_LEGENDS - 1])
+ keep.append(_merged_legend_block(placed[_MAX_OUTSIDE_LEGENDS - 1 :]))
+ placed = keep
+ legends: list[Any] = []
+ for slot, (title, handles, labels) in zip(
+ _OUTSIDE_LEGEND_SLOTS[len(placed)], placed
+ ):
+ legends.append(
+ figure.legend(
+ handles=handles,
+ labels=labels,
+ title=title,
+ frameon=False,
+ loc=slot,
+ ncols=max(1, int(np.ceil(len(handles) / 20))),
+ )
+ )
+ if _legend_blocks_overlap(figure, legends):
+ for legend in legends:
+ legend.remove()
+ title, handles, labels = _merged_legend_block(placed)
+ figure.legend(
+ handles=handles,
+ labels=labels,
+ title=title,
+ frameon=False,
+ loc="outside right center",
+ ncols=max(1, int(np.ceil(len(handles) / 20))),
+ )
+
+
+def _render_shared_legends(
+ figure: Any,
+ results: Sequence[PlotResult],
+) -> None:
+ plt, mpl = require_matplotlib()
+ categorical_seen: set[tuple[Any, ...]] = set()
+ continuous_seen: set[tuple[Any, ...]] = set()
+ size_seen: set[tuple[Any, ...]] = set()
+ marker_seen: set[tuple[Any, ...]] = set()
+ blocks: list[tuple[str | None, list[Any], list[str]]] = []
+ for result in results:
+ categorical = [
+ scale for scale in result.scales if isinstance(scale, CategoricalScale)
+ ]
+ continuous = [scale for scale in result.scales if isinstance(scale, ColorScale)]
+ sizes = [scale for scale in result.scales if isinstance(scale, SizeScale)]
+ categorical_scale_index = 0
+ continuous_scale_index = 0
+ size_scale_index = 0
+ for legend in result.legends:
+ if legend.kind == "categorical" and categorical:
+ categorical_scale = categorical[
+ min(categorical_scale_index, len(categorical) - 1)
+ ]
+ categorical_scale_index += 1
+ order = tuple(categorical_scale.order or ())
+ if not order or categorical_scale.palette is None:
+ continue
+ categorical_key = (
+ legend.label,
+ order,
+ tuple(
+ (str(value), categorical_scale.palette[value])
+ for value in order
+ ),
+ )
+ if categorical_key in categorical_seen:
+ continue
+ categorical_seen.add(categorical_key)
+ handles = [
+ mpl.lines.Line2D(
+ [],
+ [],
+ marker="o",
+ linestyle="",
+ markerfacecolor=categorical_scale.palette[value],
+ markeredgecolor="none",
+ markersize=5,
+ )
+ for value in order
+ ]
+ labels = [
+ (
+ categorical_scale.labels.get(value, str(value))
+ if categorical_scale.labels is not None
+ else str(value)
+ )
+ for value in order
+ ]
+ blocks.append((legend.label, handles, labels))
+ elif legend.kind == "colorbar" and continuous:
+ color_scale = continuous[
+ min(continuous_scale_index, len(continuous) - 1)
+ ]
+ continuous_scale_index += 1
+ vmin = legend.extras.get("vmin", color_scale.vmin)
+ vmax = legend.extras.get("vmax", color_scale.vmax)
+ if vmin is None or vmax is None:
+ continue
+ continuous_key = (
+ legend.label,
+ color_scale.cmap,
+ float(vmin),
+ float(vmax),
+ color_scale.vcenter,
+ )
+ if continuous_key in continuous_seen:
+ continue
+ continuous_seen.add(continuous_key)
+ if color_scale.vcenter is not None:
+ norm = mpl.colors.TwoSlopeNorm(
+ vmin=float(vmin),
+ vcenter=float(color_scale.vcenter),
+ vmax=float(vmax),
+ )
+ else:
+ norm = mpl.colors.Normalize(vmin=float(vmin), vmax=float(vmax))
+ mappable = plt.cm.ScalarMappable(
+ cmap=color_scale.cmap or "viridis",
+ norm=norm,
+ )
+ colorbar = figure.colorbar(
+ mappable,
+ ax=list(dict.fromkeys(result.axes.values())),
+ location="bottom",
+ orientation="horizontal",
+ shrink=0.45,
+ fraction=0.04,
+ pad=0.04,
+ )
+ colorbar.set_label(legend.label or "")
+ elif legend.kind == "size" and sizes:
+ size_scale = sizes[min(size_scale_index, len(sizes) - 1)]
+ size_scale_index += 1
+ domain = legend.extras.get(
+ "domain",
+ [size_scale.vmin, size_scale.vmax],
+ )
+ low, high = float(domain[0]), float(domain[1])
+ size_key = (
+ legend.label,
+ low,
+ high,
+ size_scale.size_min,
+ size_scale.size_max,
+ )
+ if size_key in size_seen:
+ continue
+ size_seen.add(size_key)
+ values = np.linspace(low, high, 4)
+ areas = size_scale.areas(values)
+ area_factor = min(1.0, 180.0 / max(float(areas.max()), 1.0))
+ handles = [
+ mpl.lines.Line2D(
+ [],
+ [],
+ marker="o",
+ linestyle="",
+ markerfacecolor="#bdbdbd",
+ markeredgecolor="#666666",
+ markersize=float(np.sqrt(area * area_factor)),
+ )
+ for area in areas
+ ]
+ labels = (
+ [f"{value:.0%}" for value in values]
+ if low >= 0 and high <= 1
+ else [f"{value:g}" for value in values]
+ )
+ blocks.append((legend.label, handles, labels))
+ elif legend.kind == "marker":
+ marker_values = list(legend.extras.get("values", ()))
+ markers = list(legend.extras.get("markers", ()))
+ marker_key = (
+ legend.label,
+ tuple(map(str, marker_values)),
+ tuple(map(str, markers)),
+ )
+ if not marker_values or len(marker_values) != len(markers):
+ continue
+ if marker_key in marker_seen:
+ continue
+ marker_seen.add(marker_key)
+ handles = [
+ mpl.lines.Line2D(
+ [],
+ [],
+ marker=marker,
+ linestyle="",
+ markerfacecolor="#9e9e9e",
+ markeredgecolor="#4d4d4d",
+ markersize=5,
+ )
+ for marker in markers
+ ]
+ blocks.append(
+ (legend.label, handles, [str(value) for value in marker_values])
+ )
+ _place_legend_blocks(figure, blocks)
+
+
+def _remove_child_legend_artists(figure: Any, axes: Sequence[Any]) -> None:
+ """Remove rendered child legends before drawing shared equivalents."""
+ from matplotlib.legend import Legend
+
+ main_axes = set(axes)
+ for ax in main_axes:
+ for artist in list(ax.get_children()):
+ if isinstance(artist, Legend):
+ artist.remove()
+ for legend in list(figure.legends):
+ legend.remove()
+ for ax in list(figure.axes):
+ if ax in main_axes:
+ continue
+ if getattr(ax, "_colorbar", None) is not None or ax.get_label() == "":
+ figure.delaxes(ax)
+
+
+def compose_results(
+ figure: Any,
+ results: Mapping[Hashable, PlotResult] | Sequence[PlotResult],
+ *,
+ panel_labels: bool | Sequence[str] = True,
+ shared_legends: bool = True,
+ owns_figure: bool = False,
+ theme: str | None = None,
+) -> PlotResult:
+ """Merge caller-targeted plot results into one inspectable figure result."""
+ if isinstance(results, Mapping):
+ named_results = list(results.items())
+ else:
+ named_results = list(enumerate(results))
+ if not named_results:
+ raise ValueError("results must be non-empty")
+ children = [result for _, result in named_results]
+ if any(result.figure is not figure for result in children):
+ raise ValueError("All child results must use the supplied figure")
+
+ axes: dict[Hashable, Any] = {}
+ tables: dict[str, pd.DataFrame] = {}
+ legends: list[LegendSpec] = []
+ scales: list[Any] = []
+ for namespace, result in named_results:
+ for key, ax in result.axes.items():
+ composite_key: Hashable = (namespace, key)
+ axes[composite_key] = ax
+ tables.update(
+ {
+ f"{namespace}:{table_name}": table
+ for table_name, table in result.tables.items()
+ }
+ )
+ legends.extend(result.legends)
+ scales.extend(result.scales)
+
+ resolved_theme = theme or children[0].theme
+ label_axes = list(dict.fromkeys(axes.values()))
+ with theme_context(resolved_theme):
+ if shared_legends:
+ _remove_child_legend_artists(figure, label_axes)
+ if panel_labels:
+ labels = None if panel_labels is True else list(panel_labels)
+ label_panels(label_axes, labels=labels)
+ if shared_legends:
+ _render_shared_legends(figure, children)
+ refresh_layout_point_sizes(figure)
+ provenance = PlotProvenance(
+ scarf_version=children[0].provenance.scarf_version,
+ n_cells=max(child.provenance.n_cells for child in children),
+ renderer="matplotlib",
+ notes=("composite",),
+ extras={
+ "children": {str(name): child.provenance for name, child in named_results},
+ "shared_legends": shared_legends,
+ },
+ )
+ return PlotResult(
+ figure=figure,
+ axes=axes,
+ tables=tables,
+ legends=_deduplicate_legend_specs(legends),
+ scales=tuple(scales),
+ provenance=provenance,
+ owns_figure=owns_figure,
+ theme=resolved_theme,
+ )
diff --git a/scarf/plotting/_heatmap_utils.py b/scarf/plotting/_heatmap_utils.py
new file mode 100644
index 00000000..1ac4cb71
--- /dev/null
+++ b/scarf/plotting/_heatmap_utils.py
@@ -0,0 +1,249 @@
+"""Shared ordering and annotation helpers for heatmaps."""
+
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+import numpy as np
+import pandas as pd
+
+from ._contracts import CategoricalScale
+from ._style import categorical_color_map, sort_categories
+
+
+def _explicit_order(
+ labels: Sequence[Any],
+ requested: Sequence[Any] | None,
+ *,
+ axis_name: str,
+) -> list[Any] | None:
+ if requested is None:
+ return None
+ order = list(requested)
+ if len(order) != len(set(order)):
+ raise ValueError(f"{axis_name}_order cannot contain duplicates")
+ observed = list(labels)
+ missing = [label for label in observed if label not in order]
+ unexpected = [label for label in order if label not in observed]
+ if missing or unexpected:
+ details = []
+ if missing:
+ details.append("missing: " + ", ".join(map(str, missing[:10])))
+ if unexpected:
+ details.append("unexpected: " + ", ".join(map(str, unexpected[:10])))
+ raise ValueError(
+ f"{axis_name}_order must contain every observed label ("
+ + "; ".join(details)
+ + ")"
+ )
+ return order
+
+
+def _finite_linkage_values(values: np.ndarray) -> np.ndarray:
+ data = np.asarray(values, dtype=np.float64).copy()
+ if np.isfinite(data).all():
+ return data
+ column_means = np.nanmean(data, axis=0)
+ column_means = np.nan_to_num(column_means, nan=0.0)
+ missing_row, missing_column = np.where(~np.isfinite(data))
+ data[missing_row, missing_column] = column_means[missing_column]
+ return data
+
+
+def order_heatmap(
+ matrix: pd.DataFrame,
+ *,
+ row_order: Sequence[Any] | None,
+ column_order: Sequence[Any] | None,
+ cluster_rows: bool,
+ cluster_columns: bool,
+ method: str,
+ metric: str,
+) -> tuple[pd.DataFrame, np.ndarray | None, np.ndarray | None]:
+ from scipy.cluster.hierarchy import leaves_list, linkage
+
+ explicit_rows = _explicit_order(
+ list(matrix.index),
+ row_order,
+ axis_name="row",
+ )
+ explicit_columns = _explicit_order(
+ list(matrix.columns),
+ column_order,
+ axis_name="column",
+ )
+ row_linkage = None
+ column_linkage = None
+ resolved_rows = list(matrix.index)
+ resolved_columns = list(matrix.columns)
+ if explicit_rows is not None:
+ resolved_rows = explicit_rows
+ elif cluster_rows and matrix.shape[0] > 1:
+ row_linkage = linkage(
+ _finite_linkage_values(matrix.to_numpy()),
+ method=method,
+ metric=metric,
+ optimal_ordering=True,
+ )
+ resolved_rows = [matrix.index[index] for index in leaves_list(row_linkage)]
+ if explicit_columns is not None:
+ resolved_columns = explicit_columns
+ elif cluster_columns and matrix.shape[1] > 1:
+ column_linkage = linkage(
+ _finite_linkage_values(matrix.to_numpy().T),
+ method=method,
+ metric=metric,
+ optimal_ordering=True,
+ )
+ resolved_columns = [
+ matrix.columns[index] for index in leaves_list(column_linkage)
+ ]
+ return (
+ matrix.reindex(index=resolved_rows, columns=resolved_columns),
+ row_linkage,
+ column_linkage,
+ )
+
+
+def normalize_annotations(
+ labels: Sequence[Any],
+ annotations: Mapping[
+ str,
+ Mapping[Any, Any] | Sequence[Any],
+ ]
+ | None,
+ *,
+ axis_name: str,
+) -> pd.DataFrame:
+ index = pd.Index(labels)
+ if annotations is None:
+ return pd.DataFrame(index=index)
+ columns: dict[str, list[Any]] = {}
+ for name, values in annotations.items():
+ if isinstance(values, Mapping):
+ missing = [label for label in index if label not in values]
+ if missing:
+ raise ValueError(
+ f"{axis_name} annotation {name!r} is missing labels: "
+ + ", ".join(map(str, missing[:10]))
+ )
+ columns[name] = [values[label] for label in index]
+ else:
+ resolved = list(values)
+ if len(resolved) != len(index):
+ raise ValueError(
+ f"{axis_name} annotation {name!r} must have {len(index)} values"
+ )
+ columns[name] = resolved
+ return pd.DataFrame(columns, index=index)
+
+
+def annotation_colors(
+ annotations: pd.DataFrame,
+ scales: Mapping[str, CategoricalScale] | None,
+) -> tuple[pd.DataFrame, list[CategoricalScale]]:
+ colors = pd.DataFrame(index=annotations.index)
+ resolved_scales: list[CategoricalScale] = []
+ for name in annotations:
+ values = annotations[name].to_numpy(dtype=object)
+ observed = [value for value in pd.unique(values) if pd.notna(value)]
+ scale = scales.get(name) if scales is not None else None
+ if scale is not None and scale.order is not None:
+ missing = [value for value in observed if value not in scale.order]
+ if missing:
+ raise ValueError(
+ f"annotation scale {name!r} is missing values: "
+ + ", ".join(map(str, missing[:10]))
+ )
+ order = [value for value in scale.order if value in set(observed)]
+ else:
+ order = sort_categories(observed)
+ palette = categorical_color_map(
+ order,
+ palette=scale.palette if scale is not None else None,
+ palette_name=scale.palette_name if scale is not None else "default",
+ )
+ missing_color = scale.missing_color if scale is not None else "#bdbdbd"
+ colors[name] = [
+ missing_color if pd.isna(value) else palette[value] for value in values
+ ]
+ resolved_scales.append(
+ CategoricalScale(
+ order=tuple(order),
+ palette=palette,
+ labels=scale.labels if scale is not None else None,
+ missing_color=missing_color,
+ missing_label=scale.missing_label if scale is not None else "NA",
+ palette_name=(scale.palette_name if scale is not None else "default"),
+ )
+ )
+ return colors, resolved_scales
+
+
+def draw_annotation_strips(
+ ax: Any,
+ *,
+ row_colors: pd.DataFrame,
+ column_colors: pd.DataFrame,
+ n_rows: int,
+ n_columns: int,
+) -> tuple[tuple[float, float], tuple[float, float]]:
+ """Draw annotation strips and return the expanded data limits they require."""
+ from matplotlib.patches import Rectangle
+
+ tick_labels = ax.get_yticklabels() or ax.get_xticklabels()
+ label_size = tick_labels[0].get_fontsize() if tick_labels else 8.0
+ row_width = max(0.18, n_columns * 0.025)
+ column_height = max(0.18, n_rows * 0.025)
+ row_label_y = -0.5 - len(column_colors.columns) * column_height - 0.08
+ for annotation_index, name in enumerate(row_colors):
+ left = -0.5 - (annotation_index + 1) * row_width
+ for row_index, color in enumerate(row_colors[name]):
+ ax.add_patch(
+ Rectangle(
+ (left, row_index - 0.5),
+ row_width,
+ 1,
+ facecolor=color,
+ edgecolor="none",
+ clip_on=False,
+ )
+ )
+ ax.text(
+ left + row_width / 2,
+ row_label_y,
+ name,
+ rotation=90,
+ ha="center",
+ va="bottom",
+ fontsize=label_size * 0.9,
+ clip_on=False,
+ )
+ for annotation_index, name in enumerate(column_colors):
+ top = -0.5 - (annotation_index + 1) * column_height
+ for column_index, color in enumerate(column_colors[name]):
+ ax.add_patch(
+ Rectangle(
+ (column_index - 0.5, top),
+ 1,
+ column_height,
+ facecolor=color,
+ edgecolor="none",
+ clip_on=False,
+ )
+ )
+ ax.text(
+ n_columns - 0.4,
+ top + column_height / 2,
+ name,
+ ha="left",
+ va="center",
+ fontsize=label_size * 0.9,
+ clip_on=False,
+ )
+ return (
+ (-0.5 - len(row_colors.columns) * row_width, n_columns - 0.5),
+ (
+ n_rows - 0.5,
+ -0.5 - len(column_colors.columns) * column_height,
+ ),
+ )
diff --git a/scarf/plotting/_raster.py b/scarf/plotting/_raster.py
new file mode 100644
index 00000000..5a262717
--- /dev/null
+++ b/scarf/plotting/_raster.py
@@ -0,0 +1,320 @@
+"""Blockwise embedding raster (two-pass; no full-column materialization)."""
+
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+
+from ._deps import require_matplotlib
+from ._style import continuous_norm
+
+
+@dataclass(frozen=True, slots=True)
+class RasterCanvas:
+ """Rasterized embedding panel."""
+
+ image: np.ndarray # float (H, W), NaN where empty
+ counts: np.ndarray # int (H, W)
+ extent: tuple[float, float, float, float] # xmin, xmax, ymin, ymax
+ vmin: float
+ vmax: float
+ n_cells: int
+ n_blocks: int
+
+
+def density_canvas_from_points(
+ x: np.ndarray,
+ y: np.ndarray,
+ *,
+ extent: tuple[float, float, float, float],
+ pixels: int,
+) -> RasterCanvas:
+ """Bin materialized coordinates into the shared raster canvas contract."""
+ if pixels < 8:
+ raise ValueError("pixels must be at least 8")
+ xx = np.asarray(x, dtype=np.float64)
+ yy = np.asarray(y, dtype=np.float64)
+ if len(xx) != len(yy):
+ raise ValueError("x and y lengths must match")
+ finite = np.isfinite(xx) & np.isfinite(yy)
+ xmin, xmax, ymin, ymax = map(float, extent)
+ if xmax <= xmin or ymax <= ymin:
+ raise ValueError("extent must have increasing coordinate limits")
+ counts_xy, _, _ = np.histogram2d(
+ xx[finite],
+ yy[finite],
+ bins=pixels,
+ range=((xmin, xmax), (ymin, ymax)),
+ )
+ counts = np.flipud(counts_xy.T).astype(np.int64, copy=False)
+ image = np.full(counts.shape, np.nan, dtype=np.float64)
+ occupied = counts > 0
+ image[occupied] = np.log1p(counts[occupied])
+ return RasterCanvas(
+ image=image,
+ counts=counts,
+ extent=(xmin, xmax, ymin, ymax),
+ vmin=0.0,
+ vmax=float(np.nanmax(image)) if occupied.any() else 1.0,
+ n_cells=int(finite.sum()),
+ n_blocks=1,
+ )
+
+
+def _finite_minmax(values: np.ndarray) -> tuple[float, float] | None:
+ v = values[np.isfinite(values)]
+ if len(v) == 0:
+ return None
+ return float(v.min()), float(v.max())
+
+
+def _priority_sample_update(
+ sample_values: np.ndarray,
+ sample_priorities: np.ndarray,
+ n_sampled: int,
+ values: np.ndarray,
+ *,
+ capacity: int,
+ rng: np.random.Generator,
+) -> tuple[np.ndarray, np.ndarray, int]:
+ """Update an exact uniform sample using independent random priorities."""
+ vals = values[np.isfinite(values)]
+ if len(vals) == 0:
+ return sample_values, sample_priorities, n_sampled
+
+ new_priorities = rng.random(len(vals))
+ combined_values = np.concatenate((sample_values[:n_sampled], vals))
+ combined_priorities = np.concatenate(
+ (sample_priorities[:n_sampled], new_priorities)
+ )
+ keep = min(capacity, len(combined_values))
+ if len(combined_values) > keep:
+ selected = np.argpartition(combined_priorities, -keep)[-keep:]
+ combined_values = combined_values[selected]
+ combined_priorities = combined_priorities[selected]
+ sample_values[:keep] = combined_values
+ sample_priorities[:keep] = combined_priorities
+ return sample_values, sample_priorities, keep
+
+
+def raster_from_metadata(
+ cells: Any,
+ *,
+ x_key: str,
+ y_key: str,
+ color_key: str | None = None,
+ cell_key: str = "I",
+ subset_by: str | None = None,
+ pixels: int = 400,
+ block_rows: int | None = None,
+ quantiles: tuple[float, float] | None = (0.01, 0.99),
+ seed: int = 0,
+ sample_capacity: int = 50_000,
+) -> RasterCanvas:
+ """Two-pass raster of metadata columns via ``MetaData.iter_row_blocks``.
+
+ Pass 1: bounds and color limits (exact min/max, or approximate quantiles via
+ reservoir sampling). Pass 2: accumulate mean color per pixel.
+ """
+ if pixels < 8:
+ raise ValueError("pixels must be >= 8")
+ if sample_capacity < 1:
+ raise ValueError("sample_capacity must be >= 1")
+ if quantiles is not None:
+ q0, q1 = quantiles
+ if not (0.0 <= q0 < q1 <= 1.0):
+ raise ValueError("quantiles must satisfy 0 <= low < high <= 1")
+ cols = [x_key, y_key]
+ if color_key is not None:
+ cols.append(color_key)
+ if subset_by is not None:
+ if subset_by not in cells.columns:
+ raise KeyError(f"subset_by {subset_by!r} not found in cell metadata")
+ cols.append(subset_by)
+ rng = np.random.default_rng(seed)
+
+ # --- Pass 1: bounds + color scale ---
+ xmin = ymin = np.inf
+ xmax = ymax = -np.inf
+ cmin = np.inf
+ cmax = -np.inf
+ sample_values = np.empty(sample_capacity, dtype=np.float64)
+ sample_priorities = np.empty(sample_capacity, dtype=np.float64)
+ n_sampled = 0
+ n_cells = 0
+ n_blocks = 0
+ for block in cells.iter_row_blocks(
+ cell_key=cell_key, columns=cols, block_rows=block_rows
+ ):
+ n_blocks += 1
+ if len(block.active_global_indices) == 0:
+ continue
+ x = np.asarray(block.values[x_key], dtype=np.float64)
+ y = np.asarray(block.values[y_key], dtype=np.float64)
+ finite = np.isfinite(x) & np.isfinite(y)
+ if subset_by is not None:
+ sub = np.asarray(block.values[subset_by])
+ if sub.dtype != bool:
+ raise TypeError(
+ f"subset_by {subset_by!r} must be boolean; got {sub.dtype}"
+ )
+ finite &= sub
+ if not finite.any():
+ continue
+ n_cells += int(finite.sum())
+ xmin = min(xmin, float(x[finite].min()))
+ xmax = max(xmax, float(x[finite].max()))
+ ymin = min(ymin, float(y[finite].min()))
+ ymax = max(ymax, float(y[finite].max()))
+ if color_key is not None:
+ c = np.asarray(block.values[color_key], dtype=np.float64)[finite]
+ mm = _finite_minmax(c)
+ if mm is not None:
+ cmin = min(cmin, mm[0])
+ cmax = max(cmax, mm[1])
+ if quantiles is not None:
+ sample_values, sample_priorities, n_sampled = _priority_sample_update(
+ sample_values,
+ sample_priorities,
+ n_sampled,
+ c,
+ capacity=sample_capacity,
+ rng=rng,
+ )
+
+ if n_cells == 0 or not np.isfinite(xmin):
+ empty = np.full((pixels, pixels), np.nan, dtype=np.float64)
+ return RasterCanvas(
+ image=empty,
+ counts=np.zeros((pixels, pixels), dtype=np.int64),
+ extent=(0.0, 1.0, 0.0, 1.0),
+ vmin=0.0,
+ vmax=1.0,
+ n_cells=0,
+ n_blocks=n_blocks,
+ )
+
+ if color_key is None:
+ vmin, vmax = 0.0, 1.0
+ elif not np.isfinite(cmin):
+ vmin, vmax = 0.0, 1.0
+ elif quantiles is not None and n_sampled > 0:
+ sample = sample_values[:n_sampled]
+ q0, q1 = quantiles
+ vmin = float(np.quantile(sample, q0))
+ vmax = float(np.quantile(sample, q1))
+ if vmax <= vmin:
+ vmin, vmax = float(cmin), float(cmax if cmax > cmin else cmin + 1.0)
+ else:
+ vmin, vmax = float(cmin), float(cmax if cmax > cmin else cmin + 1.0)
+
+ # Pad extent slightly so edge points land inside bins.
+ dx = xmax - xmin
+ dy = ymax - ymin
+ pad_x = 0.01 * dx if dx > 0 else 0.5
+ pad_y = 0.01 * dy if dy > 0 else 0.5
+ xmin -= pad_x
+ xmax += pad_x
+ ymin -= pad_y
+ ymax += pad_y
+ if xmax == xmin:
+ xmax = xmin + 1.0
+ if ymax == ymin:
+ ymax = ymin + 1.0
+
+ sums = np.zeros((pixels, pixels), dtype=np.float64)
+ counts = np.zeros((pixels, pixels), dtype=np.int64)
+
+ # --- Pass 2: accumulate ---
+ for block in cells.iter_row_blocks(
+ cell_key=cell_key, columns=cols, block_rows=block_rows
+ ):
+ if len(block.active_global_indices) == 0:
+ continue
+ x = np.asarray(block.values[x_key], dtype=np.float64)
+ y = np.asarray(block.values[y_key], dtype=np.float64)
+ finite = np.isfinite(x) & np.isfinite(y)
+ if subset_by is not None:
+ sub = np.asarray(block.values[subset_by])
+ if sub.dtype != bool:
+ raise TypeError(
+ f"subset_by {subset_by!r} must be boolean; got {sub.dtype}"
+ )
+ finite &= sub
+ if not finite.any():
+ continue
+ x = x[finite]
+ y = y[finite]
+ ix = np.clip(
+ ((x - xmin) / (xmax - xmin) * (pixels - 1e-9)).astype(np.int64),
+ 0,
+ pixels - 1,
+ )
+ iy = np.clip(
+ ((y - ymin) / (ymax - ymin) * (pixels - 1e-9)).astype(np.int64),
+ 0,
+ pixels - 1,
+ )
+ # Image row 0 is top; flip y for display extent mapping.
+ iy_img = pixels - 1 - iy
+ if color_key is None:
+ np.add.at(counts, (iy_img, ix), 1)
+ else:
+ c = np.asarray(block.values[color_key], dtype=np.float64)[finite]
+ finite_color = np.isfinite(c)
+ np.add.at(sums, (iy_img[finite_color], ix[finite_color]), c[finite_color])
+ np.add.at(counts, (iy_img[finite_color], ix[finite_color]), 1)
+
+ image = np.full((pixels, pixels), np.nan, dtype=np.float64)
+ nonzero = counts > 0
+ if color_key is None:
+ image[nonzero] = np.log1p(counts[nonzero])
+ vmin = 0.0
+ vmax = float(image[nonzero].max()) if nonzero.any() else 1.0
+ else:
+ image[nonzero] = sums[nonzero] / counts[nonzero]
+ return RasterCanvas(
+ image=image,
+ counts=counts,
+ extent=(xmin, xmax, ymin, ymax),
+ vmin=vmin,
+ vmax=vmax,
+ n_cells=n_cells,
+ n_blocks=n_blocks,
+ )
+
+
+def draw_raster_canvas(
+ ax: Any,
+ canvas: RasterCanvas,
+ *,
+ cmap: str = "viridis",
+ missing_color: str = "white",
+ vcenter: float | None = None,
+) -> Any:
+ """Draw a ``RasterCanvas`` onto a matplotlib axes; return the mappable."""
+ _, mpl = require_matplotlib()
+ data = canvas.image.copy()
+ cmap_obj = mpl.colormaps.get_cmap(cmap).with_extremes(bad=missing_color)
+ norm = continuous_norm(
+ mpl,
+ vmin=canvas.vmin,
+ vmax=canvas.vmax,
+ vcenter=vcenter,
+ )
+ im = ax.imshow(
+ data,
+ origin="upper",
+ extent=[
+ canvas.extent[0],
+ canvas.extent[1],
+ canvas.extent[2],
+ canvas.extent[3],
+ ],
+ cmap=cmap_obj,
+ norm=norm,
+ aspect="equal",
+ interpolation="nearest",
+ )
+ ax.set_facecolor(missing_color)
+ return im
diff --git a/scarf/plotting/_style.py b/scarf/plotting/_style.py
new file mode 100644
index 00000000..f3a2ed7b
--- /dev/null
+++ b/scarf/plotting/_style.py
@@ -0,0 +1,643 @@
+"""Themes and categorical palettes for scarf.plotting."""
+
+import re
+from collections.abc import Iterator, Mapping, Sequence
+from contextlib import contextmanager
+from typing import Any
+from weakref import WeakKeyDictionary
+
+import numpy as np
+
+from ._contracts import FrameStyle, LegendLoc
+
+# Shared Scarf figure defaults used by embedding-like plots.
+DEFAULT_POINT_SIZE = 10.0
+DEFAULT_POINT_EDGEWIDTH = 0.1
+DEFAULT_RASTERIZE_THRESHOLD = 50_000
+DEFAULT_PANEL_INCHES = 3.2
+MAX_FIGURE_WIDTH_INCHES = 7.5
+LEGEND_SIDE_MAX_CATEGORIES = 12
+LEGEND_ON_DATA_MAX_CATEGORIES = 40
+LEGEND_SIDE_ENTRIES_PER_COLUMN = 20
+LEGEND_SIDE_MAX_COLUMNS = 4
+LEGEND_SIDE_MAX_ENTRIES = LEGEND_SIDE_ENTRIES_PER_COLUMN * LEGEND_SIDE_MAX_COLUMNS
+
+_LAYOUT_POINT_SIZE_SPECS: WeakKeyDictionary[
+ Any,
+ tuple[int, float, float, float],
+] = WeakKeyDictionary()
+
+# Okabe-Ito plus four high-contrast extensions for categorical figures.
+COLORBLIND_PALETTE = [
+ "#0072B2",
+ "#E69F00",
+ "#009E73",
+ "#D55E00",
+ "#CC79A7",
+ "#56B4E9",
+ "#F0E442",
+ "#000000",
+ "#6F4E7C",
+ "#2E8B57",
+ "#A05195",
+ "#8C564B",
+]
+
+# Lifted from scanpy.plotting.palettes.
+CUSTOM_PALETTES: dict[int, list[str]] = {
+ 10: [
+ "#1f77b4",
+ "#ff7f0e",
+ "#279e68",
+ "#d62728",
+ "#aa40fc",
+ "#8c564b",
+ "#e377c2",
+ "#7f7f7f",
+ "#b5bd61",
+ "#17becf",
+ ],
+ 20: [
+ "#1f77b4",
+ "#aec7e8",
+ "#ff7f0e",
+ "#ffbb78",
+ "#2ca02c",
+ "#98df8a",
+ "#d62728",
+ "#ff9896",
+ "#9467bd",
+ "#c5b0d5",
+ "#8c564b",
+ "#c49c94",
+ "#e377c2",
+ "#f7b6d2",
+ "#7f7f7f",
+ "#c7c7c7",
+ "#bcbd22",
+ "#dbdb8d",
+ "#17becf",
+ "#9edae5",
+ ],
+ 28: [
+ "#023fa5",
+ "#7d87b9",
+ "#bec1d4",
+ "#d6bcc0",
+ "#bb7784",
+ "#8e063b",
+ "#4a6fe3",
+ "#8595e1",
+ "#b5bbe3",
+ "#e6afb9",
+ "#e07b91",
+ "#d33f6a",
+ "#11c638",
+ "#8dd593",
+ "#c6dec7",
+ "#ead3c6",
+ "#f0b98d",
+ "#ef9708",
+ "#0fcfc0",
+ "#9cded6",
+ "#d5eae7",
+ "#f3e1eb",
+ "#f6c4e1",
+ "#f79cd4",
+ "#7f7f7f",
+ "#c7c7c7",
+ "#1CE6FF",
+ "#336600",
+ ],
+ 102: [
+ "#FFFF00",
+ "#1CE6FF",
+ "#FF34FF",
+ "#FF4A46",
+ "#008941",
+ "#006FA6",
+ "#A30059",
+ "#FFDBE5",
+ "#7A4900",
+ "#0000A6",
+ "#63FFAC",
+ "#B79762",
+ "#004D43",
+ "#8FB0FF",
+ "#997D87",
+ "#5A0007",
+ "#809693",
+ "#6A3A4C",
+ "#1B4400",
+ "#4FC601",
+ "#3B5DFF",
+ "#4A3B53",
+ "#FF2F80",
+ "#61615A",
+ "#BA0900",
+ "#6B7900",
+ "#00C2A0",
+ "#FFAA92",
+ "#FF90C9",
+ "#B903AA",
+ "#D16100",
+ "#DDEFFF",
+ "#000035",
+ "#7B4F4B",
+ "#A1C299",
+ "#300018",
+ "#0AA6D8",
+ "#013349",
+ "#00846F",
+ "#372101",
+ "#FFB500",
+ "#C2FFED",
+ "#A079BF",
+ "#CC0744",
+ "#C0B9B2",
+ "#C2FF99",
+ "#001E09",
+ "#00489C",
+ "#6F0062",
+ "#0CBD66",
+ "#EEC3FF",
+ "#456D75",
+ "#B77B68",
+ "#7A87A1",
+ "#788D66",
+ "#885578",
+ "#FAD09F",
+ "#FF8A9A",
+ "#D157A0",
+ "#BEC459",
+ "#456648",
+ "#0086ED",
+ "#886F4C",
+ "#34362D",
+ "#B4A8BD",
+ "#00A6AA",
+ "#452C2C",
+ "#636375",
+ "#A3C8C9",
+ "#FF913F",
+ "#938A81",
+ "#575329",
+ "#00FECF",
+ "#B05B6F",
+ "#8CD0FF",
+ "#3B9700",
+ "#04F757",
+ "#C8A1A1",
+ "#1E6E00",
+ "#7900D7",
+ "#A77500",
+ "#6367A9",
+ "#A05837",
+ "#6B002C",
+ "#772600",
+ "#D790FF",
+ "#9B9700",
+ "#549E79",
+ "#FFF69F",
+ "#201625",
+ "#72418F",
+ "#BC23FF",
+ "#99ADC0",
+ "#3A2465",
+ "#922329",
+ "#5B4534",
+ "#FDE8DC",
+ "#404E55",
+ "#0089A3",
+ "#CB7E98",
+ "#A4E804",
+ "#324E72",
+ ],
+}
+
+THEMES: dict[str, dict[str, Any]] = {
+ "paper": {
+ "font.size": 8,
+ "axes.titlesize": 9,
+ "axes.labelsize": 8,
+ "xtick.labelsize": 7,
+ "ytick.labelsize": 7,
+ "legend.fontsize": 7,
+ "axes.linewidth": 0.6,
+ "lines.linewidth": 0.8,
+ "axes.spines.top": False,
+ "axes.spines.right": False,
+ "figure.facecolor": "white",
+ "axes.facecolor": "white",
+ "savefig.facecolor": "white",
+ "savefig.transparent": False,
+ "svg.fonttype": "none",
+ "pdf.fonttype": 42,
+ "savefig.dpi": 300,
+ "figure.dpi": 150,
+ },
+ "notebook": {
+ "font.size": 10,
+ "axes.titlesize": 11,
+ "axes.labelsize": 10,
+ "xtick.labelsize": 9,
+ "ytick.labelsize": 9,
+ "legend.fontsize": 9,
+ "axes.linewidth": 0.8,
+ "axes.spines.top": False,
+ "axes.spines.right": False,
+ "figure.facecolor": "white",
+ "axes.facecolor": "white",
+ "savefig.facecolor": "white",
+ "savefig.transparent": False,
+ "svg.fonttype": "none",
+ "pdf.fonttype": 42,
+ "savefig.dpi": 150,
+ "figure.dpi": 100,
+ },
+ "minimal": {
+ "font.size": 9,
+ "axes.spines.top": False,
+ "axes.spines.right": False,
+ "figure.facecolor": "white",
+ "axes.facecolor": "white",
+ "savefig.facecolor": "white",
+ "savefig.transparent": False,
+ "svg.fonttype": "none",
+ "pdf.fonttype": 42,
+ },
+ "dark": {
+ "font.size": 10,
+ "axes.titlesize": 11,
+ "axes.labelsize": 10,
+ "xtick.labelsize": 9,
+ "ytick.labelsize": 9,
+ "legend.fontsize": 9,
+ "axes.linewidth": 0.8,
+ "axes.spines.top": False,
+ "axes.spines.right": False,
+ "figure.facecolor": "none",
+ "axes.facecolor": "none",
+ "savefig.facecolor": "none",
+ "savefig.transparent": True,
+ "text.color": "#e8e8e8",
+ "axes.labelcolor": "#e8e8e8",
+ "axes.edgecolor": "#e8e8e8",
+ "xtick.color": "#e8e8e8",
+ "ytick.color": "#e8e8e8",
+ "axes.titlecolor": "#e8e8e8",
+ "svg.fonttype": "none",
+ "pdf.fonttype": 42,
+ "savefig.dpi": 150,
+ "figure.dpi": 100,
+ },
+}
+
+
+def default_point_size(
+ n_cells: int,
+ *,
+ panel_area: float = DEFAULT_PANEL_INCHES**2,
+ size_min: float = 1.0,
+ size_max: float = 28.0,
+) -> float:
+ """Marker area derived from selected cells and physical panel area."""
+ if panel_area <= 0:
+ raise ValueError("panel_area must be positive")
+ if size_min <= 0 or size_max < size_min:
+ raise ValueError("point-size bounds must satisfy 0 < size_min <= size_max")
+ n = max(1, int(n_cells))
+ reference_area = DEFAULT_PANEL_INCHES**2
+ area_factor = (float(panel_area) / reference_area) ** 0.72
+ population_factor = (1_000.0 / n) ** 0.5
+ return float(min(size_max, max(size_min, 16.0 * area_factor * population_factor)))
+
+
+def default_point_edgewidth(
+ n_cells: int,
+ *,
+ point_size: float | None = None,
+) -> float:
+ """Tune point outlines to marker area and cloud density."""
+ n = max(1, int(n_cells))
+ area = point_size if point_size is not None else default_point_size(n)
+ if n >= 20_000 or area < 2.5:
+ return 0.0
+ if area < 7.0 or n >= 10_000:
+ return 0.05
+ return 0.15
+
+
+def register_layout_point_size(
+ collection: Any,
+ *,
+ n_points: int,
+ size_min: float,
+ size_max: float,
+ multiplier: float = 1.0,
+) -> None:
+ _LAYOUT_POINT_SIZE_SPECS[collection] = (
+ int(n_points),
+ float(size_min),
+ float(size_max),
+ float(multiplier),
+ )
+
+
+def refresh_layout_point_sizes(figure: Any) -> None:
+ """Refresh marked scatter artists after figure layout is resolved."""
+ marked = [
+ (ax, collection, specification)
+ for ax in figure.axes
+ for collection in ax.collections
+ if (specification := _LAYOUT_POINT_SIZE_SPECS.get(collection)) is not None
+ ]
+ if not marked:
+ return
+ figure.canvas.draw()
+ for ax, collection, specification in marked:
+ bbox = ax.get_position()
+ width, height = figure.get_size_inches()
+ panel_area = float(bbox.width * width * bbox.height * height)
+ n_points, size_min, size_max, multiplier = specification
+ point_size = default_point_size(
+ n_points,
+ panel_area=panel_area,
+ size_min=size_min,
+ size_max=size_max,
+ )
+ collection.set_sizes(
+ np.full(
+ len(collection.get_offsets()),
+ point_size * multiplier,
+ dtype=np.float64,
+ )
+ )
+
+
+def resolve_legend_loc(n_categories: int, legend_loc: LegendLoc = "auto") -> LegendLoc:
+ """Choose a legend placement that survives many clusters."""
+ if legend_loc != "auto":
+ if legend_loc not in ("right", "on_data", "none"):
+ raise ValueError(
+ "legend_loc must be one of 'auto', 'right', 'on_data', 'none'"
+ )
+ return legend_loc
+ if n_categories <= LEGEND_SIDE_MAX_CATEGORIES:
+ return "right"
+ if n_categories <= LEGEND_ON_DATA_MAX_CATEGORIES:
+ return "on_data"
+ return "right"
+
+
+def legend_side_columns(n_entries: int) -> int:
+ """Columns for a side legend, bounded so wide category sets stay readable."""
+ columns = int(np.ceil(max(int(n_entries), 1) / LEGEND_SIDE_ENTRIES_PER_COLUMN))
+ return max(1, min(columns, LEGEND_SIDE_MAX_COLUMNS))
+
+
+def capped_figsize(
+ width: float,
+ height: float,
+ *,
+ max_width: float | None = MAX_FIGURE_WIDTH_INCHES,
+) -> tuple[float, float]:
+ """Clamp figure width so atlas-scale category counts stay page-sized."""
+ resolved_width = float(width)
+ if max_width is not None:
+ if max_width <= 0:
+ raise ValueError("max_width must be positive or None")
+ resolved_width = min(resolved_width, float(max_width))
+ return (resolved_width, float(height))
+
+
+def _category_sort_key(value: Any) -> tuple[Any, ...]:
+ """Sort key: numbers in numeric order, then natural string order."""
+ import math
+
+ import numpy as np
+ import pandas as pd
+
+ if value is None or (isinstance(value, float) and math.isnan(value)):
+ return (2, ())
+ try:
+ if pd.isna(value):
+ return (2, ())
+ except (TypeError, ValueError):
+ pass
+
+ if isinstance(value, (bool, np.bool_)):
+ return (1, (str(bool(value)).lower(),))
+ if isinstance(value, (int, np.integer)):
+ return (0, (float(value),))
+ if isinstance(value, (float, np.floating)) and math.isfinite(float(value)):
+ return (0, (float(value),))
+
+ text = str(value)
+ try:
+ return (0, (float(text),))
+ except ValueError:
+ pass
+
+ tokens = tuple(
+ int(part) if part.isdigit() else part.casefold()
+ for part in re.split(r"(\d+)", text)
+ if part != ""
+ )
+ return (1, tokens)
+
+
+def sort_categories(values: Sequence[Any]) -> list[Any]:
+ """Order categories with numeric ids before natural strings (1, 2, 10)."""
+ return sorted(values, key=_category_sort_key)
+
+
+def palette_for_n(
+ n: int,
+ *,
+ palette_name: str = "default",
+) -> list[str]:
+ if palette_name not in ("default", "colorblind"):
+ raise ValueError("palette_name must be 'default' or 'colorblind'")
+ if palette_name == "colorblind":
+ if n <= len(COLORBLIND_PALETTE):
+ return list(COLORBLIND_PALETTE[:n])
+ from ..utils.logging import logger
+ from ._deps import require_seaborn
+
+ logger.warning(
+ f"Requested {n} colorblind-safe colors but only "
+ f"{len(COLORBLIND_PALETTE)} are available; "
+ "falling back to evenly spaced hues that are distinct but not "
+ "guaranteed colorblind safe"
+ )
+ sns = require_seaborn()
+ return list(sns.color_palette("husl", n_colors=n).as_hex())
+ if n <= 10:
+ return list(CUSTOM_PALETTES[10][:n])
+ if n <= 20:
+ return list(CUSTOM_PALETTES[20][:n])
+ if n <= 28:
+ return list(CUSTOM_PALETTES[28][:n])
+ if n <= 102:
+ return list(CUSTOM_PALETTES[102][:n])
+ from ._deps import require_seaborn
+
+ sns = require_seaborn()
+ return list(sns.color_palette("husl", n_colors=n).as_hex())
+
+
+def categorical_color_map(
+ categories: list[Any],
+ *,
+ palette: Mapping[Any, str] | None = None,
+ palette_name: str = "default",
+ missing_label: str | None = None,
+ missing_color: str = "#bdbdbd",
+) -> dict[Any, str]:
+ cats = list(categories)
+ if palette is not None:
+ out = dict(palette)
+ for cat in cats:
+ if cat not in out:
+ raise KeyError(f"Category {cat!r} missing from palette")
+ else:
+ colors = palette_for_n(len(cats), palette_name=palette_name)
+ out = dict(zip(cats, colors))
+ if missing_label is not None:
+ out[missing_label] = missing_color
+ return out
+
+
+def continuous_norm(
+ mpl: Any,
+ *,
+ vmin: float,
+ vmax: float,
+ vcenter: float | None,
+) -> Any:
+ if vmax <= vmin:
+ vmax = vmin + 1.0
+ if vcenter is None:
+ return mpl.colors.Normalize(vmin=vmin, vmax=vmax)
+ if not vmin < vcenter < vmax:
+ raise ValueError("vcenter must be strictly between the color limits")
+ return mpl.colors.TwoSlopeNorm(vmin=vmin, vcenter=vcenter, vmax=vmax)
+
+
+def square_axis_limits(
+ xlim: tuple[float, float],
+ ylim: tuple[float, float],
+) -> tuple[tuple[float, float], tuple[float, float]]:
+ """Pad axis limits so the data window is square.
+
+ Keeps equal-aspect embeddings visually square after legends and colorbars
+ shrink the available axes width.
+ """
+ x0, x1 = float(xlim[0]), float(xlim[1])
+ y0, y1 = float(ylim[0]), float(ylim[1])
+ span = max(x1 - x0, y1 - y0, 1e-12)
+ cx = 0.5 * (x0 + x1)
+ cy = 0.5 * (y0 + y1)
+ half = 0.5 * span
+ return (cx - half, cx + half), (cy - half, cy + half)
+
+
+def scatter_edgecolor(theme: str = "notebook") -> str:
+ """Marker edge color that stays readable on light and dark themes."""
+ if theme == "dark":
+ # Mid grey keeps dark fills legible without turning markers into rings.
+ return "#8f8f8f"
+ return "#333333"
+
+
+def foreground_color(theme: str = "notebook") -> str:
+ """High-contrast foreground for annotations and segment borders."""
+ return "#e8e8e8" if theme == "dark" else "#333333"
+
+
+def finish_embedding_axes(
+ ax: Any,
+ *,
+ xlim: tuple[float, float],
+ ylim: tuple[float, float],
+ xlabel: str = "",
+ ylabel: str = "",
+ title: str | None = None,
+ frame: FrameStyle = "minimal",
+) -> None:
+ """Apply shared Scarf chrome to a 2D embedding axes."""
+ if frame not in ("axes", "minimal", "none"):
+ raise ValueError("frame must be one of 'axes', 'minimal', 'none'")
+ ax.set_xlim(xlim)
+ ax.set_ylim(ylim)
+ ax.set_aspect("equal", adjustable="box")
+ ax.set_box_aspect(1)
+ ax.set_xticks([])
+ ax.set_yticks([])
+ if frame == "axes":
+ ax.set_xlabel(xlabel)
+ ax.set_ylabel(ylabel)
+ else:
+ ax.set_xlabel("")
+ ax.set_ylabel("")
+ if frame == "none" and hasattr(ax, "spines"):
+ for spine in ax.spines.values():
+ spine.set_visible(False)
+ if title:
+ ax.set_title(title)
+
+
+def apply_figure_chrome(figure: Any, theme: str = "notebook") -> None:
+ """Apply Scarf figure background and spine defaults after axes creation."""
+ opaque = theme != "dark"
+ if opaque:
+ figure.patch.set_facecolor("white")
+ figure.patch.set_alpha(1.0)
+ else:
+ figure.patch.set_alpha(0)
+ for ax in figure.axes:
+ if opaque:
+ ax.patch.set_facecolor("white")
+ ax.patch.set_alpha(1.0)
+ else:
+ ax.patch.set_alpha(0)
+ if hasattr(ax, "spines"):
+ ax.spines["top"].set_visible(False)
+ ax.spines["right"].set_visible(False)
+
+
+@contextmanager
+def theme_context(name: str = "notebook") -> Iterator[None]:
+ if name not in THEMES:
+ raise KeyError(f"Unknown theme {name!r}. Choose from: {sorted(THEMES)}")
+ from ._deps import require_matplotlib
+
+ _, mpl = require_matplotlib()
+ with mpl.rc_context(THEMES[name]):
+ yield
+
+
+def register_theme(
+ name: str,
+ rcparams: Mapping[str, Any],
+ *,
+ base: str | None = "notebook",
+ overwrite: bool = False,
+) -> None:
+ """Register a Matplotlib rcParams theme for later plot calls."""
+ if not name:
+ raise ValueError("Theme name must be non-empty")
+ if name in THEMES and not overwrite:
+ raise ValueError(f"Theme {name!r} already exists")
+ if base is not None and base not in THEMES:
+ raise KeyError(f"Unknown base theme {base!r}")
+ from ._deps import require_matplotlib
+
+ _, mpl = require_matplotlib()
+ invalid = sorted(set(rcparams) - set(mpl.rcParams))
+ if invalid:
+ raise KeyError(f"Unknown Matplotlib rcParams: {invalid}")
+ values = dict(THEMES[base]) if base is not None else {}
+ values.update(dict(rcparams))
+ THEMES[name] = values
diff --git a/scarf/plotting/cluster_connectivity.py b/scarf/plotting/cluster_connectivity.py
new file mode 100644
index 00000000..0500a9ff
--- /dev/null
+++ b/scarf/plotting/cluster_connectivity.py
@@ -0,0 +1,662 @@
+"""Cluster connectivity plots derived from a sparse cell graph."""
+
+from collections.abc import Mapping
+from typing import Any, Literal
+
+import numpy as np
+import pandas as pd
+from scipy import sparse
+
+from ._contracts import CategoricalScale, PlotProvenance, SizeScale
+from ._deps import require_matplotlib
+from ._display import resolve_categorical_scale
+from ._figure import LegendSpec, PlotResult, normalize_axes_target
+from ._style import (
+ apply_figure_chrome,
+ categorical_color_map,
+ finish_embedding_axes,
+ refresh_layout_point_sizes,
+ register_layout_point_size,
+ scatter_edgecolor,
+ sort_categories,
+ square_axis_limits,
+ theme_context,
+)
+
+
+def _fetch_inputs(
+ store: Any,
+ *,
+ group_by: str,
+ layout_key: str,
+ cell_key: str,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ try:
+ x = np.asarray(
+ store.cells.fetch(f"{layout_key}1", key=cell_key),
+ dtype=np.float64,
+ )
+ y = np.asarray(
+ store.cells.fetch(f"{layout_key}2", key=cell_key),
+ dtype=np.float64,
+ )
+ except (TypeError, ValueError) as exc:
+ raise TypeError(f"Layout {layout_key!r} coordinates must be numeric") from exc
+ groups = np.asarray(store.cells.fetch(group_by, key=cell_key), dtype=object)
+
+ for name, values in (
+ (f"{layout_key}1", x),
+ (f"{layout_key}2", y),
+ (group_by, groups),
+ ):
+ if values.ndim != 1:
+ raise ValueError(f"Cell data {name!r} must be one-dimensional")
+ if len(x) == 0:
+ raise ValueError(f"No cells selected by cell_key {cell_key!r}")
+ if len(y) != len(x) or len(groups) != len(x):
+ raise ValueError(
+ "Layout coordinates and group values must have matching lengths "
+ f"for cell_key {cell_key!r}"
+ )
+ if not np.isfinite(x).all() or not np.isfinite(y).all():
+ raise ValueError(f"Layout {layout_key!r} contains non-finite coordinates")
+ if pd.Series(groups, copy=False).isna().any():
+ raise ValueError(f"group_by {group_by!r} contains missing values")
+ return x, y, groups
+
+
+def _resolve_categories(
+ groups: np.ndarray,
+ categorical_scale: CategoricalScale | None,
+) -> tuple[list[Any], dict[Any, str], dict[Any, str], CategoricalScale]:
+ try:
+ observed = list(pd.unique(groups))
+ observed_set = set(observed)
+ except TypeError as exc:
+ raise TypeError("group_by values must be hashable categories") from exc
+
+ if categorical_scale is not None and categorical_scale.order is not None:
+ requested_order = list(categorical_scale.order)
+ if len(set(requested_order)) != len(requested_order):
+ raise ValueError("categorical_scale.order cannot contain duplicates")
+ missing = [category for category in observed if category not in requested_order]
+ if missing:
+ raise ValueError(
+ "categorical_scale.order is missing observed values: "
+ + ", ".join(map(str, missing[:10]))
+ )
+ order = [category for category in requested_order if category in observed_set]
+ else:
+ order = sort_categories(observed)
+
+ palette = categorical_color_map(
+ order,
+ palette=(categorical_scale.palette if categorical_scale is not None else None),
+ palette_name=(
+ categorical_scale.palette_name
+ if categorical_scale is not None
+ else "default"
+ ),
+ )
+ explicit_labels = (
+ categorical_scale.labels if categorical_scale is not None else None
+ )
+ display_labels = {
+ category: str(
+ explicit_labels.get(category, category)
+ if explicit_labels is not None
+ else category
+ )
+ for category in order
+ }
+ resolved_labels = dict(display_labels) if explicit_labels is not None else None
+ resolved_scale = CategoricalScale(
+ order=tuple(order),
+ palette=dict(palette),
+ labels=resolved_labels,
+ missing_color=(
+ categorical_scale.missing_color
+ if categorical_scale is not None
+ else "#bdbdbd"
+ ),
+ missing_label=(
+ categorical_scale.missing_label if categorical_scale is not None else "NA"
+ ),
+ palette_name=(
+ categorical_scale.palette_name
+ if categorical_scale is not None
+ else "default"
+ ),
+ )
+ return order, palette, display_labels, resolved_scale
+
+
+def _category_codes(groups: np.ndarray, order: list[Any]) -> np.ndarray:
+ category_index = {category: index for index, category in enumerate(order)}
+ try:
+ return np.fromiter(
+ (category_index[value] for value in groups),
+ dtype=np.intp,
+ count=len(groups),
+ )
+ except (KeyError, TypeError) as exc:
+ raise ValueError("Could not map every group value to a category") from exc
+
+
+def _resolve_node_positions(
+ x: np.ndarray,
+ y: np.ndarray,
+ codes: np.ndarray,
+ order: list[Any],
+ *,
+ position: Literal["median", "mean"],
+ positions: Mapping[Any, tuple[float, float]] | None,
+) -> tuple[np.ndarray, np.ndarray]:
+ if position not in ("median", "mean"):
+ raise ValueError("position must be 'median' or 'mean'")
+
+ if positions is not None:
+ expected = set(order)
+ missing = [category for category in order if category not in positions]
+ unexpected = [category for category in positions if category not in expected]
+ if missing or unexpected:
+ details: list[str] = []
+ if missing:
+ details.append("missing: " + ", ".join(map(str, missing)))
+ if unexpected:
+ details.append("unexpected: " + ", ".join(map(str, unexpected)))
+ raise ValueError(
+ "positions must contain exactly the observed categories ("
+ + "; ".join(details)
+ + ")"
+ )
+ resolved = np.empty((len(order), 2), dtype=np.float64)
+ for index, category in enumerate(order):
+ try:
+ coordinate = np.asarray(positions[category], dtype=np.float64)
+ except (TypeError, ValueError) as exc:
+ raise TypeError(
+ f"Position for category {category!r} must be a numeric pair"
+ ) from exc
+ if coordinate.shape != (2,) or not np.isfinite(coordinate).all():
+ raise ValueError(
+ f"Position for category {category!r} must be a finite pair"
+ )
+ resolved[index] = coordinate
+ return resolved[:, 0], resolved[:, 1]
+
+ node_x = np.empty(len(order), dtype=np.float64)
+ node_y = np.empty(len(order), dtype=np.float64)
+ reducer = np.median if position == "median" else np.mean
+ for index in range(len(order)):
+ mask = codes == index
+ node_x[index] = float(reducer(x[mask]))
+ node_y[index] = float(reducer(y[mask]))
+ return node_x, node_y
+
+
+def _load_graph(
+ store: Any,
+ *,
+ n_cells: int,
+ from_assay: str | None,
+ cell_key: str,
+ feat_key: str | None,
+ graph_loc: str | None,
+) -> sparse.csr_matrix:
+ graph = store.load_graph(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ graph_loc=graph_loc,
+ symmetric=True,
+ )
+ if not sparse.issparse(graph):
+ raise TypeError("load_graph must return a scipy sparse matrix")
+ if graph.shape != (n_cells, n_cells):
+ raise ValueError(
+ "Graph shape must match the selected cells; "
+ f"expected {(n_cells, n_cells)}, got {graph.shape}"
+ )
+
+ graph = graph.tocsr(copy=True).astype(np.float64, copy=False)
+ graph.sum_duplicates()
+ graph.eliminate_zeros()
+ if not np.isfinite(graph.data).all():
+ raise ValueError("Graph contains non-finite edge weights")
+ if (graph.data < 0).any():
+ raise ValueError("Graph edge weights must be non-negative")
+ graph.setdiag(0.0)
+ graph.eliminate_zeros()
+ return graph
+
+
+def _aggregate_intercluster_edges(
+ graph: sparse.csr_matrix,
+ codes: np.ndarray,
+ n_categories: int,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+ """Aggregate each undirected cell edge once and normalize symmetrically."""
+ incident_by_cell = np.asarray(graph.sum(axis=1), dtype=np.float64).ravel()
+ incident_by_category = np.bincount(
+ codes,
+ weights=incident_by_cell,
+ minlength=n_categories,
+ )
+
+ upper = sparse.triu(graph, k=1, format="coo")
+ source_codes = codes[upper.row]
+ target_codes = codes[upper.col]
+ intercluster = source_codes != target_codes
+ if not intercluster.any():
+ empty_int = np.empty(0, dtype=np.intp)
+ empty_float = np.empty(0, dtype=np.float64)
+ return empty_int, empty_int.copy(), empty_float, empty_float.copy()
+
+ first = np.minimum(source_codes[intercluster], target_codes[intercluster])
+ second = np.maximum(source_codes[intercluster], target_codes[intercluster])
+ pair_keys = first.astype(np.int64) * n_categories + second
+ unique_keys, inverse = np.unique(pair_keys, return_inverse=True)
+ raw_weights = np.bincount(
+ inverse,
+ weights=np.asarray(upper.data[intercluster], dtype=np.float64),
+ )
+ source = (unique_keys // n_categories).astype(np.intp, copy=False)
+ target = (unique_keys % n_categories).astype(np.intp, copy=False)
+
+ denominator = np.sqrt(incident_by_category[source] * incident_by_category[target])
+ normalized = np.divide(
+ raw_weights,
+ denominator,
+ out=np.zeros_like(raw_weights),
+ where=denominator > 0,
+ )
+ return source, target, raw_weights, normalized
+
+
+def _filter_edges(
+ source: np.ndarray,
+ target: np.ndarray,
+ raw_weights: np.ndarray,
+ normalized_weights: np.ndarray,
+ *,
+ minimum_edge_weight: float,
+ max_edges_per_node: int | None,
+ n_categories: int,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+ retained = normalized_weights >= minimum_edge_weight
+ source = source[retained]
+ target = target[retained]
+ raw_weights = raw_weights[retained]
+ normalized_weights = normalized_weights[retained]
+
+ if max_edges_per_node is not None and len(source):
+ ranked = np.lexsort((target, source, -raw_weights, -normalized_weights))
+ degree = np.zeros(n_categories, dtype=np.intp)
+ keep = np.zeros(len(source), dtype=bool)
+ for edge_index in ranked:
+ left = source[edge_index]
+ right = target[edge_index]
+ if (
+ degree[left] >= max_edges_per_node
+ or degree[right] >= max_edges_per_node
+ ):
+ continue
+ keep[edge_index] = True
+ degree[left] += 1
+ degree[right] += 1
+ source = source[keep]
+ target = target[keep]
+ raw_weights = raw_weights[keep]
+ normalized_weights = normalized_weights[keep]
+
+ ordered = np.lexsort((target, source))
+ return (
+ source[ordered],
+ target[ordered],
+ raw_weights[ordered],
+ normalized_weights[ordered],
+ )
+
+
+def _edge_widths(
+ weights: np.ndarray,
+ edge_width_range: tuple[float, float],
+) -> np.ndarray:
+ if len(weights) == 0:
+ return np.empty(0, dtype=np.float64)
+ minimum, maximum = edge_width_range
+ weight_min = float(weights.min())
+ weight_max = float(weights.max())
+ if weight_max <= weight_min:
+ return np.full(len(weights), maximum, dtype=np.float64)
+ scaled = (weights - weight_min) / (weight_max - weight_min)
+ return minimum + scaled * (maximum - minimum)
+
+
+def _axis_limits(
+ x: np.ndarray,
+ y: np.ndarray,
+ node_x: np.ndarray,
+ node_y: np.ndarray,
+ *,
+ include_cells: bool,
+) -> tuple[tuple[float, float], tuple[float, float]]:
+ limit_x = np.concatenate((x, node_x)) if include_cells else node_x
+ limit_y = np.concatenate((y, node_y)) if include_cells else node_y
+ x_span = float(limit_x.max() - limit_x.min())
+ y_span = float(limit_y.max() - limit_y.min())
+ x_pad = 0.05 * (x_span if x_span > 0 else 1.0)
+ y_pad = 0.05 * (y_span if y_span > 0 else 1.0)
+ return square_axis_limits(
+ (float(limit_x.min() - x_pad), float(limit_x.max() + x_pad)),
+ (float(limit_y.min() - y_pad), float(limit_y.max() + y_pad)),
+ )
+
+
+def cluster_connectivity(
+ store: Any,
+ *,
+ group_by: str,
+ layout_key: str,
+ cell_key: str = "I",
+ from_assay: str | None = None,
+ feat_key: str | None = None,
+ graph_loc: str | None = None,
+ position: Literal["median", "mean"] = "median",
+ positions: Mapping[Any, tuple[float, float]] | None = None,
+ categorical_scale: CategoricalScale | None = None,
+ size_scale: SizeScale | None = None,
+ minimum_edge_weight: float = 0.02,
+ max_edges_per_node: int | None = None,
+ show_cells: bool = False,
+ cell_size: float | None = None,
+ cell_alpha: float = 0.3,
+ cell_color: str | None = None,
+ node_edgecolor: str | None = None,
+ node_linewidth: float = 0.8,
+ edge_color: str | None = None,
+ edge_alpha: float = 0.45,
+ edge_width_range: tuple[float, float] = (0.4, 5.0),
+ labels: bool = True,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show: bool = True,
+) -> PlotResult:
+ """Plot cluster connectivity on an existing two-dimensional cell layout.
+
+ ``rawWeight`` is the sum of intercluster weights after reciprocal sparse
+ graph entries are counted once. ``normalizedWeight`` divides that value by
+ ``sqrt(incidentWeight[source] * incidentWeight[target])``. Incident weight
+ is the sum of weighted cell degrees in each cluster, including internal
+ cluster edges. Clusters with zero incident weight receive a normalized
+ value of zero.
+ """
+ if not np.isfinite(minimum_edge_weight) or minimum_edge_weight < 0:
+ raise ValueError("minimum_edge_weight must be finite and non-negative")
+ if max_edges_per_node is not None and (
+ isinstance(max_edges_per_node, bool)
+ or not isinstance(max_edges_per_node, int)
+ or max_edges_per_node < 0
+ ):
+ raise ValueError("max_edges_per_node must be a non-negative integer or None")
+ if cell_size is not None and (not np.isfinite(cell_size) or cell_size < 0):
+ raise ValueError("cell_size must be finite and non-negative")
+ if not np.isfinite(cell_alpha) or not 0 <= cell_alpha <= 1:
+ raise ValueError("cell_alpha must be between 0 and 1")
+ if not np.isfinite(node_linewidth) or node_linewidth < 0:
+ raise ValueError("node_linewidth must be finite and non-negative")
+ if not np.isfinite(edge_alpha) or not 0 <= edge_alpha <= 1:
+ raise ValueError("edge_alpha must be between 0 and 1")
+ if len(edge_width_range) != 2:
+ raise ValueError("edge_width_range must contain two values")
+ edge_width_min, edge_width_max = edge_width_range
+ if (
+ not np.isfinite(edge_width_min)
+ or not np.isfinite(edge_width_max)
+ or edge_width_min < 0
+ or edge_width_max < edge_width_min
+ ):
+ raise ValueError(
+ "edge_width_range must be finite and satisfy 0 <= minimum <= maximum"
+ )
+
+ x, y, groups = _fetch_inputs(
+ store,
+ group_by=group_by,
+ layout_key=layout_key,
+ cell_key=cell_key,
+ )
+ categorical_scale = resolve_categorical_scale(
+ store,
+ group_by,
+ categorical_scale,
+ )
+ order, palette, display_labels, resolved_categorical_scale = _resolve_categories(
+ groups, categorical_scale
+ )
+ codes = _category_codes(groups, order)
+ node_x, node_y = _resolve_node_positions(
+ x,
+ y,
+ codes,
+ order,
+ position=position,
+ positions=positions,
+ )
+
+ graph = _load_graph(
+ store,
+ n_cells=len(x),
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ graph_loc=graph_loc,
+ )
+ source_index, target_index, raw_weights, normalized_weights = (
+ _aggregate_intercluster_edges(graph, codes, len(order))
+ )
+ aggregated_edge_count = len(source_index)
+ source_index, target_index, raw_weights, normalized_weights = _filter_edges(
+ source_index,
+ target_index,
+ raw_weights,
+ normalized_weights,
+ minimum_edge_weight=minimum_edge_weight,
+ max_edges_per_node=max_edges_per_node,
+ n_categories=len(order),
+ )
+
+ counts = np.bincount(codes, minlength=len(order))
+ proportions = counts.astype(np.float64) / len(groups)
+ resolved_size_scale = size_scale or SizeScale(size_min=70.0, size_max=650.0)
+ node_sizes = resolved_size_scale.areas(proportions)
+ if not np.isfinite(node_sizes).all() or (node_sizes < 0).any():
+ raise ValueError("size_scale produced invalid marker areas")
+
+ category_values = np.empty(len(order), dtype=object)
+ category_values[:] = order
+ nodes = pd.DataFrame(
+ {
+ "category": category_values,
+ "x": node_x,
+ "y": node_y,
+ "nCells": counts,
+ "proportion": proportions,
+ "size": node_sizes,
+ "displayLabel": [display_labels[category] for category in order],
+ }
+ )
+ edges = pd.DataFrame(
+ {
+ "source": category_values[source_index],
+ "target": category_values[target_index],
+ "rawWeight": raw_weights,
+ "normalizedWeight": normalized_weights,
+ }
+ )
+
+ require_matplotlib()
+ from matplotlib import patheffects
+ from matplotlib.collections import LineCollection
+
+ panel_key = "cluster_connectivity"
+ with theme_context(theme):
+ resolved_figsize = (5.0, 5.0) if target is None and figsize is None else figsize
+ figure, axes, owns_figure = normalize_axes_target(
+ target,
+ panel_keys=[panel_key],
+ figsize=resolved_figsize,
+ )
+ ax = axes[panel_key]
+ resolved_node_edgecolor = node_edgecolor or scatter_edgecolor(theme)
+ resolved_edge_color = edge_color or scatter_edgecolor(theme)
+
+ cell_artist = None
+ if show_cells:
+ resolved_cell_size = 6.0 if cell_size is None else cell_size
+ cell_artist = ax.scatter(
+ x,
+ y,
+ s=resolved_cell_size,
+ c=(
+ [palette[group] for group in groups]
+ if cell_color is None
+ else cell_color
+ ),
+ alpha=cell_alpha,
+ linewidths=0,
+ rasterized=len(x) >= 50_000,
+ zorder=0,
+ )
+ if cell_size is None:
+ register_layout_point_size(
+ cell_artist,
+ n_points=len(x),
+ size_min=4.0,
+ size_max=12.0,
+ )
+
+ segments = np.empty((len(edges), 2, 2), dtype=np.float64)
+ if len(edges):
+ segments[:, 0, 0] = node_x[source_index]
+ segments[:, 0, 1] = node_y[source_index]
+ segments[:, 1, 0] = node_x[target_index]
+ segments[:, 1, 1] = node_y[target_index]
+ line_collection = LineCollection(
+ segments.tolist(),
+ colors=resolved_edge_color,
+ linewidths=_edge_widths(normalized_weights, edge_width_range),
+ alpha=edge_alpha,
+ zorder=1,
+ )
+ ax.add_collection(line_collection)
+ ax.scatter(
+ node_x,
+ node_y,
+ s=node_sizes,
+ c=[palette[category] for category in order],
+ edgecolors=resolved_node_edgecolor,
+ linewidths=node_linewidth,
+ zorder=2,
+ )
+
+ if labels:
+ text_color = "#f5f5f5" if theme == "dark" else "#222222"
+ stroke_color = "#222222" if theme == "dark" else "#ffffff"
+ diameters = np.sqrt(np.asarray(node_sizes, dtype=np.float64))
+ for node_index, node in enumerate(nodes.itertuples(index=False)):
+ text = str(node.displayLabel)
+ # Shrink long labels so they stay inside their marker.
+ fitted = 0.85 * diameters[node_index] / (0.62 * max(len(text), 1))
+ ax.text(
+ node.x,
+ node.y,
+ text,
+ ha="center",
+ va="center",
+ fontsize=float(np.clip(fitted, 5.0, 8.0)),
+ color=text_color,
+ path_effects=[
+ patheffects.withStroke(
+ linewidth=2.0,
+ foreground=stroke_color,
+ )
+ ],
+ zorder=3,
+ )
+
+ xlim, ylim = _axis_limits(
+ x,
+ y,
+ node_x,
+ node_y,
+ include_cells=positions is None or show_cells,
+ )
+ finish_embedding_axes(
+ ax,
+ xlim=xlim,
+ ylim=ylim,
+ frame="none",
+ )
+ apply_figure_chrome(figure, theme)
+ if cell_artist is not None and cell_size is None:
+ refresh_layout_point_sizes(figure)
+
+ result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables={"nodes": nodes, "edges": edges},
+ legends=(
+ LegendSpec(
+ kind="categorical",
+ label=group_by,
+ extras={
+ "categories": list(order),
+ "placement": "nodes" if labels else "none",
+ },
+ ),
+ ),
+ scales=(resolved_categorical_scale, resolved_size_scale),
+ provenance=PlotProvenance(
+ assay=from_assay or getattr(store, "_defaultAssay", None),
+ cell_key=cell_key,
+ n_cells=len(groups),
+ renderer="matplotlib",
+ notes=("cluster_connectivity", "materialized", f"layout={layout_key}"),
+ extras={
+ "group_by": group_by,
+ "layout_key": layout_key,
+ "feat_key": feat_key,
+ "graph_loc": graph_loc,
+ "position": "explicit" if positions is not None else position,
+ "minimum_edge_weight": minimum_edge_weight,
+ "max_edges_per_node": max_edges_per_node,
+ "show_cells": show_cells,
+ "cell_size": (
+ None if cell_artist is None else float(cell_artist.get_sizes()[0])
+ ),
+ "cell_size_source": (
+ "hidden"
+ if not show_cells
+ else "panel"
+ if cell_size is None
+ else "explicit"
+ ),
+ "cell_alpha": cell_alpha,
+ "cell_color": "category" if cell_color is None else cell_color,
+ "n_nodes": len(nodes),
+ "n_aggregated_edges": aggregated_edge_count,
+ "n_edges": len(edges),
+ "normalization": (
+ "rawWeight / sqrt(incidentWeight[source] * incidentWeight[target])"
+ ),
+ },
+ ),
+ owns_figure=owns_figure,
+ theme=theme,
+ )
+ if show:
+ result.show()
+ return result
diff --git a/scarf/plotting/cluster_tree.py b/scarf/plotting/cluster_tree.py
new file mode 100644
index 00000000..1606565e
--- /dev/null
+++ b/scarf/plotting/cluster_tree.py
@@ -0,0 +1,471 @@
+"""Native cluster-tree plotting."""
+
+import math
+from typing import Any, cast
+
+import numpy as np
+import pandas as pd
+
+from ._contracts import CategoricalScale, ColorScale, PlotProvenance, SizeScale
+from ._deps import require_matplotlib
+from ._figure import LegendSpec, PlotResult, normalize_axes_target
+from ._style import (
+ apply_figure_chrome,
+ continuous_norm,
+ sort_categories,
+ theme_context,
+)
+
+
+def _hierarchy_positions(
+ graph: Any,
+ *,
+ root: Any | None = None,
+ width: float = 1.0,
+ vert_gap: float = 0.2,
+ vert_loc: float = 0.0,
+ leaf_vs_root_factor: float = 0.5,
+) -> dict[Any, tuple[float, float]]:
+ import networkx as nx
+
+ if not nx.is_tree(graph):
+ raise TypeError("cannot position a graph that is not a tree")
+ if root is None:
+ root = (
+ next(iter(nx.topological_sort(graph)))
+ if isinstance(graph, nx.DiGraph)
+ else next(iter(graph.nodes))
+ )
+
+ def place(
+ node: Any,
+ *,
+ leftmost: float,
+ branch_width: float,
+ leaf_dx: float,
+ y: float,
+ x_center: float,
+ root_positions: dict[Any, tuple[float, float]],
+ leaf_positions: dict[Any, tuple[float, float]],
+ parent: Any | None,
+ ) -> int:
+ root_positions[node] = (x_center, y)
+ children = list(graph.neighbors(node))
+ if not isinstance(graph, nx.DiGraph) and parent is not None:
+ children.remove(parent)
+ if not children:
+ leaf_positions[node] = (leftmost, y)
+ return 1
+
+ root_dx = branch_width / len(children)
+ next_x = x_center - branch_width / 2 - root_dx / 2
+ leaf_count = 0
+ for child in children:
+ next_x += root_dx
+ leaf_count += place(
+ child,
+ leftmost=leftmost + leaf_count * leaf_dx,
+ branch_width=root_dx,
+ leaf_dx=leaf_dx,
+ y=y - vert_gap,
+ x_center=next_x,
+ root_positions=root_positions,
+ leaf_positions=leaf_positions,
+ parent=node,
+ )
+ child_x = [leaf_positions[child][0] for child in children]
+ leaf_positions[node] = (
+ (min(child_x) + max(child_x)) / 2,
+ y,
+ )
+ return leaf_count
+
+ if isinstance(graph, nx.DiGraph):
+ leaf_count = len(
+ [
+ node
+ for node in nx.descendants(graph, root)
+ if graph.out_degree(node) == 0
+ ]
+ )
+ else:
+ leaf_count = len(
+ [
+ node
+ for node in nx.node_connected_component(graph, root)
+ if graph.degree(node) == 1 and node != root
+ ]
+ )
+ leaf_count = max(leaf_count, 1)
+ root_positions: dict[Any, tuple[float, float]] = {}
+ leaf_positions: dict[Any, tuple[float, float]] = {}
+ place(
+ root,
+ leftmost=0.0,
+ branch_width=width,
+ leaf_dx=width / leaf_count,
+ y=vert_loc,
+ x_center=width / 2,
+ root_positions=root_positions,
+ leaf_positions=leaf_positions,
+ parent=None,
+ )
+ positions = {
+ node: (
+ leaf_vs_root_factor * leaf_positions[node][0]
+ + (1 - leaf_vs_root_factor) * root_positions[node][0],
+ leaf_positions[node][1],
+ )
+ for node in root_positions
+ }
+ x_max = max((x for x, _ in positions.values()), default=0.0)
+ if x_max != 0:
+ positions = {node: (x * width / x_max, y) for node, (x, y) in positions.items()}
+ return positions
+
+
+def _tree_color_series(
+ values: np.ndarray,
+ *,
+ force_ints_as_cats: bool,
+) -> tuple[pd.Series, bool]:
+ series = pd.Series(values)
+ if series.nunique() == 1:
+ return pd.Series(np.ones(len(series)), index=series.index), False
+ categorical = (
+ pd.api.types.is_bool_dtype(series.dtype)
+ or pd.api.types.is_string_dtype(series.dtype)
+ or isinstance(series.dtype, pd.CategoricalDtype)
+ or (pd.api.types.is_integer_dtype(series.dtype) and force_ints_as_cats)
+ )
+ if categorical:
+ return series.astype("category"), True
+ return series.astype(np.float64), False
+
+
+def _tree_palette(
+ mpl: Any,
+ categories: list[Any],
+ *,
+ cmap: str,
+ color_key: dict[Any, str] | None,
+) -> dict[Any, str]:
+ if color_key is not None:
+ palette = dict(color_key)
+ missing = [category for category in categories if category not in palette]
+ if missing:
+ raise KeyError(f"ERROR: key {missing[0]} missing in `color_key`")
+ return palette
+ colormap = mpl.colormaps.get_cmap(cmap).resampled(max(len(categories), 1))
+ return {
+ category: mpl.colors.to_hex(colormap(index))
+ for index, category in enumerate(categories)
+ }
+
+
+def _draw_tree_pie(
+ ax: Any,
+ values: np.ndarray,
+ colors: list[str],
+ *,
+ x: float,
+ y: float,
+ size: float,
+) -> None:
+ cumulative = np.cumsum(values, dtype=np.float64)
+ cumulative /= cumulative[-1]
+ bounds = [0.0, *cumulative.tolist()]
+ for start, stop, color in zip(bounds[:-1], bounds[1:], colors):
+ angles = np.linspace(2 * np.pi * start, 2 * np.pi * stop)
+ marker_x = [0.0, *np.cos(angles).tolist()]
+ marker_y = [0.0, *np.sin(angles).tolist()]
+ marker = np.column_stack([marker_x, marker_y])
+ ax.scatter([x], [y], marker=marker, s=size, c=color, zorder=3)
+
+
+def cluster_tree(
+ store: Any,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ integrated_graph: str | None = None,
+ cluster_key: str | None = None,
+ fill_by_value: str | None = None,
+ force_ints_as_cats: bool = True,
+ width: float = 1,
+ lvr_factor: float = 0.5,
+ vert_gap: float = 0.2,
+ min_node_size: float = 10,
+ node_size_multiplier: float = 10_000.0,
+ node_power: float = 1.2,
+ root_size: float = 100,
+ non_leaf_size: float = 10,
+ show_labels: bool = True,
+ fontsize: float = 10,
+ root_color: str = "#C0C0C0",
+ non_leaf_color: str = "k",
+ cmap: str = "tab20",
+ color_key: dict[Any, str] | None = None,
+ edgecolors: str = "k",
+ edgewidth: float = 1,
+ alpha: float = 0.7,
+ figsize: tuple[float, float] = (5, 5),
+ ax: Any = None,
+ theme: str = "notebook",
+ show: bool = True,
+) -> PlotResult:
+ """Plot the coalesced hierarchy for a clustering result."""
+ import networkx as nx
+
+ prepared = store._prepare_cluster_tree(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ integrated_graph=integrated_graph,
+ cluster_key=cluster_key,
+ fill_by_value=fill_by_value,
+ )
+ graph = prepared["graph"]
+ clusters = np.asarray(prepared["clusters"])
+ raw_color_values = (
+ clusters
+ if prepared["color_values"] is None
+ else np.asarray(prepared["color_values"])
+ )
+ if raw_color_values.shape[0] != clusters.shape[0]:
+ raise ValueError("Cluster colors and cluster assignments are misaligned")
+ using_clusters = prepared["color_values"] is None
+ color_values, categorical = _tree_color_series(
+ raw_color_values,
+ force_ints_as_cats=force_ints_as_cats,
+ )
+
+ angular_positions = _hierarchy_positions(
+ graph,
+ width=width * math.pi,
+ leaf_vs_root_factor=lvr_factor,
+ vert_gap=vert_gap,
+ )
+ positions = {
+ node: (radius * math.cos(theta), radius * math.sin(theta))
+ for node, (theta, radius) in angular_positions.items()
+ }
+ cluster_counts = pd.Series(clusters).value_counts()
+ cluster_sizes = (
+ node_size_multiplier * ((cluster_counts / cluster_counts.sum()) ** node_power)
+ ).to_dict()
+
+ plt, mpl = require_matplotlib()
+ categories: list[Any] = []
+ palette: dict[Any, str] | None = None
+ cluster_means: pd.Series | None = None
+ norm = None
+ colormap = mpl.colormaps.get_cmap(cmap)
+ if categorical:
+ categories = sort_categories(list(color_values.dropna().unique()))
+ palette = _tree_palette(
+ mpl,
+ categories,
+ cmap=cmap,
+ color_key=color_key,
+ )
+ else:
+ cluster_means = (
+ pd.DataFrame({"cluster": clusters, "value": color_values})
+ .groupby("cluster", observed=False)["value"]
+ .mean()
+ )
+ color_min = float(cluster_means.min())
+ color_max = float(cluster_means.max())
+ norm = continuous_norm(
+ mpl,
+ vmin=color_min,
+ vmax=color_max,
+ vcenter=None,
+ )
+
+ node_order = list(graph.nodes())
+ node_colors: list[str] = []
+ node_sizes: list[float] = []
+ for node in node_order:
+ node_data = graph.nodes[node]
+ if "partition_id" in node_data:
+ cluster_id = node_data["partition_id"]
+ if categorical and using_clusters:
+ assert palette is not None
+ node_colors.append(palette[cluster_id])
+ node_sizes.append(max(float(cluster_sizes[cluster_id]), min_node_size))
+ elif not categorical:
+ assert cluster_means is not None and norm is not None
+ node_colors.append(
+ mpl.colors.to_hex(colormap(norm(cluster_means.loc[cluster_id])))
+ )
+ node_sizes.append(max(float(cluster_sizes[cluster_id]), min_node_size))
+ else:
+ node_colors.append("white")
+ node_sizes.append(0.0)
+ elif node_data["nleaves"] == len(clusters):
+ node_colors.append(root_color)
+ node_sizes.append(root_size)
+ else:
+ node_colors.append(non_leaf_color)
+ node_sizes.append(non_leaf_size)
+
+ with theme_context(theme):
+ fig, axes, owns_figure = normalize_axes_target(
+ ax,
+ panel_keys=["tree"],
+ figsize=figsize if ax is None else None,
+ )
+ tree_ax = axes["tree"]
+ nx.draw_networkx_edges(
+ graph,
+ pos=positions,
+ ax=tree_ax,
+ edge_color=edgecolors,
+ width=edgewidth,
+ alpha=alpha,
+ )
+ nx.draw_networkx_nodes(
+ graph,
+ pos=positions,
+ ax=tree_ax,
+ nodelist=node_order,
+ node_size=node_sizes,
+ node_color=node_colors,
+ edgecolors=edgecolors,
+ linewidths=edgewidth,
+ alpha=alpha,
+ )
+
+ if categorical and not using_clusters:
+ assert palette is not None
+ for node in node_order:
+ node_data = graph.nodes[node]
+ if "partition_id" not in node_data:
+ continue
+ cluster_id = node_data["partition_id"]
+ counts = color_values[clusters == cluster_id].value_counts()
+ _draw_tree_pie(
+ tree_ax,
+ counts.to_numpy(),
+ [palette[value] for value in counts.index],
+ x=positions[node][0],
+ y=positions[node][1],
+ size=max(float(cluster_sizes[cluster_id]), min_node_size),
+ )
+
+ if show_labels:
+ for node in node_order:
+ node_data = graph.nodes[node]
+ if "partition_id" not in node_data:
+ continue
+ cluster_id = node_data["partition_id"]
+ tree_ax.text(
+ positions[node][0],
+ positions[node][1],
+ str(cluster_id),
+ fontsize=fontsize,
+ ha="center",
+ va="center",
+ )
+ tree_ax.set_axis_off()
+ tree_ax.set_aspect("equal", adjustable="datalim")
+
+ if not categorical:
+ assert norm is not None
+ scalar_mappable = mpl.cm.ScalarMappable(norm=norm, cmap=colormap)
+ colorbar = fig.colorbar(
+ scalar_mappable,
+ ax=tree_ax,
+ location="right",
+ shrink=0.65,
+ pad=0.02,
+ )
+ colorbar.set_label(fill_by_value or cast(str, prepared["cluster_key"]))
+ axes["colorbar"] = colorbar.ax
+ apply_figure_chrome(fig, theme)
+
+ nodes = pd.DataFrame(
+ [
+ {
+ "node": node,
+ "nleaves": graph.nodes[node]["nleaves"],
+ "partition_id": graph.nodes[node].get("partition_id"),
+ }
+ for node in node_order
+ ]
+ )
+ edges = pd.DataFrame(graph.edges(), columns=["source", "target"])
+ position_table = pd.DataFrame(
+ [{"node": node, "x": xy[0], "y": xy[1]} for node, xy in positions.items()]
+ )
+ cluster_summary = (
+ cluster_counts.rename_axis("cluster").rename("n_cells").reset_index()
+ )
+
+ if categorical:
+ assert palette is not None
+ color_scale: Any = CategoricalScale(
+ order=tuple(categories),
+ palette=palette,
+ )
+ legend = LegendSpec(
+ kind="categorical",
+ label=fill_by_value or cast(str, prepared["cluster_key"]),
+ )
+ else:
+ assert cluster_means is not None
+ color_min = float(cluster_means.min())
+ color_max = float(cluster_means.max())
+ if color_max <= color_min:
+ color_max = color_min + 1.0
+ color_scale = ColorScale(
+ cmap=cmap,
+ vmin=color_min,
+ vmax=color_max,
+ )
+ legend = LegendSpec(
+ kind="colorbar",
+ label=fill_by_value or cast(str, prepared["cluster_key"]),
+ extras={"vmin": color_min, "vmax": color_max},
+ )
+ size_scale = SizeScale(
+ vmin=0.0,
+ vmax=1.0,
+ size_min=min_node_size,
+ size_max=node_size_multiplier,
+ )
+ result = PlotResult(
+ figure=fig,
+ axes=axes,
+ tables={
+ "nodes": nodes,
+ "edges": edges,
+ "positions": position_table,
+ "cluster_summary": cluster_summary,
+ },
+ legends=(legend,),
+ scales=(color_scale, size_scale),
+ provenance=PlotProvenance(
+ assay=cast(str, prepared["from_assay"]),
+ cell_key=cast(str, prepared["cell_key"]),
+ n_cells=len(clusters),
+ renderer="matplotlib",
+ notes=("cluster_tree", "coalesced"),
+ extras={
+ "feat_key": prepared["feat_key"],
+ "cluster_key": prepared["cluster_key"],
+ "fill_by_value": fill_by_value,
+ "coalesced_location": prepared["coalesced_location"],
+ "force_ints_as_cats": force_ints_as_cats,
+ "node_power": node_power,
+ },
+ ),
+ owns_figure=owns_figure,
+ theme=theme,
+ )
+ if show:
+ result.show()
+ return result
diff --git a/scarf/plotting/composition.py b/scarf/plotting/composition.py
new file mode 100644
index 00000000..e6d22b20
--- /dev/null
+++ b/scarf/plotting/composition.py
@@ -0,0 +1,955 @@
+"""Sample-level composition figures, including paired subject lines."""
+
+from typing import Any, Hashable, Literal
+
+import numpy as np
+import pandas as pd
+
+from ._contracts import CategoricalScale, PlotProvenance, StudyDesign
+from ._deps import require_matplotlib
+from ._display import resolve_categorical_scale
+from ._figure import (
+ LegendSpec,
+ PlotResult,
+ _place_legend_blocks,
+ normalize_axes_target,
+)
+from ._style import (
+ apply_figure_chrome,
+ capped_figsize,
+ categorical_color_map,
+ foreground_color,
+ scatter_edgecolor,
+ sort_categories,
+ theme_context,
+)
+
+_MISSING_CATEGORY = "\x00scarf_missing_category\x00"
+
+
+def _constant_within_sample(
+ samples: np.ndarray,
+ values: np.ndarray,
+ *,
+ field_name: str,
+) -> pd.Series:
+ """Return one value per sample; raise if a sample has multiple values."""
+ df = pd.DataFrame({"sample": samples, "value": values})
+ nunique = df.groupby("sample", observed=False)["value"].nunique(dropna=False)
+ bad = nunique[nunique > 1]
+ if len(bad):
+ raise ValueError(
+ f"{field_name} is not constant within sample(s): "
+ + ", ".join(map(str, list(bad.index[:10])))
+ )
+ return df.groupby("sample", observed=False)["value"].first()
+
+
+def _attach_sample_meta(
+ per_sample: pd.DataFrame,
+ *,
+ samples: np.ndarray,
+ subject_vals: np.ndarray | None,
+ pair_vals: np.ndarray | None,
+ condition_vals: np.ndarray | None,
+) -> pd.DataFrame:
+ meta_parts: list[pd.Series] = []
+ if subject_vals is not None:
+ meta_parts.append(
+ _constant_within_sample(
+ samples, subject_vals, field_name="subject_by"
+ ).rename("subject")
+ )
+ if pair_vals is not None:
+ meta_parts.append(
+ _constant_within_sample(samples, pair_vals, field_name="pair_by").rename(
+ "pair"
+ )
+ )
+ if condition_vals is not None:
+ meta_parts.append(
+ _constant_within_sample(
+ samples, condition_vals, field_name="condition_by"
+ ).rename("condition")
+ )
+ if not meta_parts:
+ return per_sample
+ meta = pd.concat(meta_parts, axis=1).reset_index()
+ return per_sample.merge(meta, on="sample", how="left")
+
+
+def _sort_conditions(values: list[Any]) -> list[Any]:
+ """Order conditions for paired plots with a light before/after preference."""
+ preferred = (
+ "before",
+ "pre",
+ "baseline",
+ "control",
+ "ctrl",
+ "untreated",
+ "after",
+ "post",
+ "treated",
+ "stimulated",
+ "stim",
+ )
+
+ def key(value: Any) -> tuple[int, str]:
+ text = str(value).lower()
+ try:
+ return preferred.index(text), text
+ except ValueError:
+ return len(preferred), text
+
+ return sorted(values, key=key)
+
+
+def _draw_pair_lines(
+ ax: Any,
+ per_sample: pd.DataFrame,
+ cat_order: list[Any],
+ condition_order: list[Any],
+ pair_col: str,
+) -> int:
+ """Draw one line per category and pair across conditions."""
+ n_lines = 0
+ positions = {
+ (cat, condition): float(index)
+ for index, (cat, condition) in enumerate(
+ (cat, condition) for cat in cat_order for condition in condition_order
+ )
+ }
+ for cat in cat_order:
+ category_rows = per_sample[per_sample["category"] == cat]
+ valid_pair = pd.notna(category_rows[pair_col]) & (
+ category_rows[pair_col].astype(str) != ""
+ )
+ category_rows = category_rows[valid_pair]
+ for _, group in category_rows.groupby(
+ pair_col,
+ observed=False,
+ dropna=True,
+ ):
+ xs: list[float] = []
+ ys: list[float] = []
+ for condition in condition_order:
+ rows = group[group["condition"] == condition]
+ values = rows["proportion"].to_numpy(dtype=np.float64)
+ finite = values[np.isfinite(values)]
+ if len(finite) == 0:
+ continue
+ xs.append(positions[(cat, condition)])
+ ys.append(float(np.mean(finite)))
+ if len(xs) >= 2:
+ ax.plot(
+ xs,
+ ys,
+ color="#757575",
+ alpha=0.45,
+ linewidth=0.9,
+ zorder=1,
+ solid_capstyle="round",
+ )
+ n_lines += 1
+ return n_lines
+
+
+def _summarize_proportions(
+ per_sample: pd.DataFrame,
+ *,
+ by_condition: bool,
+ uncertainty: Literal["none", "sd", "se", "ci95"],
+) -> pd.DataFrame:
+ columns = ["category"]
+ if by_condition:
+ columns.append("condition")
+ grouped = per_sample.groupby(
+ columns,
+ observed=False,
+ dropna=False,
+ )["proportion"]
+ summary = grouped.agg(["mean", "std", "count"]).reset_index()
+ summary = summary.rename(
+ columns={
+ "mean": "mean_proportion",
+ "count": "n_samples",
+ }
+ )
+ spread = np.zeros(len(summary), dtype=np.float64)
+ standard_deviation = summary["std"].fillna(0).to_numpy(dtype=np.float64)
+ sample_count = summary["n_samples"].to_numpy(dtype=np.float64)
+ if uncertainty == "sd":
+ spread = standard_deviation
+ elif uncertainty in ("se", "ci95"):
+ standard_error = np.divide(
+ standard_deviation,
+ np.sqrt(sample_count),
+ out=np.zeros_like(standard_deviation),
+ where=sample_count > 1,
+ )
+ if uncertainty == "se":
+ spread = standard_error
+ else:
+ from scipy.stats import t
+
+ critical = np.ones_like(sample_count)
+ valid = sample_count > 1
+ critical[valid] = t.ppf(0.975, sample_count[valid] - 1)
+ critical[~valid] = 0
+ spread = standard_error * critical
+ mean = summary["mean_proportion"].to_numpy(dtype=np.float64)
+ summary["lower"] = np.clip(mean - spread, 0, 1)
+ summary["upper"] = np.clip(mean + spread, 0, 1)
+ return summary.drop(columns="std")
+
+
+def _summary_marker_handle(
+ mpl: Any,
+ *,
+ edgecolor: str,
+ uncertainty: str,
+) -> Any:
+ spread_label = {
+ "sd": "mean +/- SD",
+ "se": "mean +/- SE",
+ "ci95": "mean with 95% CI",
+ }.get(uncertainty, "mean")
+ return mpl.lines.Line2D(
+ [],
+ [],
+ marker="D",
+ linestyle="",
+ markerfacecolor="#9e9e9e",
+ markeredgecolor=edgecolor,
+ markersize=5,
+ label=spread_label,
+ )
+
+
+def _place_axis_legend_blocks(
+ ax: Any,
+ blocks: list[tuple[str, list[Any]]],
+) -> None:
+ slots = {
+ 1: (("upper left", (1.02, 1.0)),),
+ 2: (("upper left", (1.02, 1.0)), ("lower left", (1.02, 0.0))),
+ 3: (
+ ("upper left", (1.02, 1.0)),
+ ("center left", (1.02, 0.5)),
+ ("lower left", (1.02, 0.0)),
+ ),
+ }
+ legends: list[Any] = []
+ for (location, anchor), (title, handles) in zip(slots[len(blocks)], blocks):
+ if legends:
+ ax.add_artist(legends[-1])
+ legends.append(
+ ax.legend(
+ handles=handles,
+ frameon=False,
+ loc=location,
+ bbox_to_anchor=anchor,
+ title=title,
+ )
+ )
+
+
+def _draw_summary_markers(
+ ax: Any,
+ summary: pd.DataFrame,
+ *,
+ positions: dict[tuple[Any, Any | None], float],
+ palette: dict[Any, str],
+ edgecolor: str,
+) -> None:
+ for row in summary.itertuples(index=False):
+ condition = getattr(row, "condition", None)
+ position = positions[(row.category, condition)]
+ lower = float(row.lower)
+ upper = float(row.upper)
+ mean = float(row.mean_proportion)
+ ax.errorbar(
+ position,
+ mean,
+ yerr=np.asarray([[mean - lower], [upper - mean]]),
+ fmt="D",
+ markersize=4,
+ markerfacecolor=palette[row.category],
+ markeredgecolor=edgecolor,
+ markeredgewidth=0.5,
+ ecolor=edgecolor,
+ elinewidth=1.0,
+ capsize=2,
+ zorder=4,
+ )
+
+
+def composition(
+ store: Any,
+ *,
+ category_by: str,
+ cell_key: str = "I",
+ sample_by: str | None = None,
+ subject_by: str | None = None,
+ pair_by: str | None = None,
+ condition_by: str | None = None,
+ study_design: StudyDesign | None = None,
+ kind: Literal["stacked", "per_sample"] = "stacked",
+ show_summary: bool = True,
+ uncertainty: Literal["none", "sd", "se", "ci95"] | None = None,
+ categorical_scale: CategoricalScale | None = None,
+ bar_width: float = 0.82,
+ bar_gap: float = 0.12,
+ segment_edgecolor: str | None = None,
+ segment_linewidth: float = 0.5,
+ show_percent_labels: bool = False,
+ label_min_fraction: float = 0.08,
+ percent_format: str = "{:.0%}",
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ max_figure_width: float | None = 7.5,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+) -> PlotResult:
+ """Show how cell categories (clusters, cell types) vary across samples.
+
+ ``kind="stacked"`` draws stacked bars. With ``sample_by``, there is one bar
+ per sample; without it, one bar for the whole dataset.
+ ``kind="per_sample"`` draws one point per sample in each category (requires
+ ``sample_by`` or a :class:`StudyDesign`).
+
+ For paired before/after designs, pass ``condition_by`` together with
+ ``subject_by`` or ``pair_by`` (directly or via ``study_design``). Scarf
+ then connects the same subject across conditions inside each category.
+ When a subject has several samples in one condition, those proportions are
+ averaged for the connecting line.
+ Per-sample plots show sample means with 95% confidence intervals by
+ default. Use ``uncertainty`` to select standard deviation, standard error,
+ or no interval.
+
+ Figure width is capped so large category lists stay page-sized.
+ """
+ _, mpl = require_matplotlib()
+ if kind not in ("stacked", "per_sample"):
+ raise ValueError("kind must be 'stacked' or 'per_sample'")
+ if uncertainty not in (None, "none", "sd", "se", "ci95"):
+ raise ValueError("uncertainty must be 'none', 'sd', 'se', 'ci95', or None")
+ if kind == "stacked" and uncertainty not in (None, "none"):
+ raise ValueError("uncertainty is available only for kind='per_sample'")
+ resolved_uncertainty: Literal["none", "sd", "se", "ci95"] = (
+ "ci95"
+ if kind == "per_sample" and uncertainty is None
+ else uncertainty or "none"
+ )
+ if bar_width <= 0 or bar_gap < 0:
+ raise ValueError("bar_width must be positive and bar_gap non-negative")
+ if segment_linewidth < 0:
+ raise ValueError("segment_linewidth must be non-negative")
+ if not 0 <= label_min_fraction <= 1:
+ raise ValueError("label_min_fraction must be between 0 and 1")
+ if study_design is not None:
+ sample_by = study_design.sample_by
+ subject_by = study_design.subject_by or subject_by
+ pair_by = study_design.pair_by or pair_by
+ condition_by = study_design.condition_by or condition_by
+ if (subject_by is not None or pair_by is not None) and condition_by is None:
+ raise ValueError(
+ "Paired composition requires condition_by together with "
+ "subject_by or pair_by"
+ )
+ categorical_scale = resolve_categorical_scale(
+ store,
+ category_by,
+ categorical_scale,
+ )
+
+ cats = np.asarray(
+ store.cells.fetch(category_by, key=cell_key),
+ dtype=object,
+ ).copy()
+ if len(cats) == 0:
+ raise ValueError(f"No cells selected by cell_key {cell_key!r}")
+ missing_label = (
+ categorical_scale.missing_label if categorical_scale is not None else "NA"
+ )
+ missing_mask = pd.isna(cats)
+ cats[missing_mask] = _MISSING_CATEGORY
+ observed_categories = [
+ value for value in pd.unique(cats) if value != _MISSING_CATEGORY
+ ]
+ if categorical_scale and categorical_scale.order is not None:
+ cat_order = list(categorical_scale.order)
+ unlisted = [
+ category for category in observed_categories if category not in cat_order
+ ]
+ if unlisted:
+ raise ValueError(
+ "categorical_scale.order is missing observed values: "
+ + ", ".join(map(str, unlisted[:10]))
+ )
+ else:
+ cat_order = sort_categories(observed_categories)
+ if missing_mask.any():
+ cat_order.append(_MISSING_CATEGORY)
+
+ pair_col: str | None = None
+ n_pair_lines = 0
+ n_unpaired_samples = 0
+ dropped_sample_cells = 0
+ summary_table: pd.DataFrame | None = None
+ condition_order: list[Any] | None = None
+
+ if sample_by is None:
+ if kind == "per_sample":
+ raise ValueError("kind='per_sample' requires sample_by or StudyDesign")
+ if subject_by is not None or pair_by is not None:
+ raise ValueError("subject_by / pair_by require sample_by or StudyDesign")
+ counts = pd.Series(cats).value_counts()
+ props = counts.reindex(cat_order).fillna(0) / max(counts.sum(), 1)
+ aggregate = pd.DataFrame(
+ {
+ "category": cat_order,
+ "proportion": [props.get(c, 0.0) for c in cat_order],
+ }
+ )
+ per_sample = None
+ props_mat = None
+ else:
+ samples = store.cells.fetch(sample_by, key=cell_key)
+ valid = pd.notna(samples) & (np.asarray(samples, dtype=object) != "")
+ if not valid.any():
+ raise ValueError(f"No cells have valid values for sample_by {sample_by!r}")
+ dropped_sample_cells = int((~valid).sum())
+ sample_vals_valid = np.asarray(samples)[valid]
+ df = pd.DataFrame(
+ {
+ "sample": sample_vals_valid,
+ "category": np.asarray(cats)[valid],
+ }
+ )
+ ct = pd.crosstab(df["sample"], df["category"])
+ ct = ct.reindex(columns=cat_order, fill_value=0)
+ props_mat = ct.div(ct.sum(axis=1).replace(0, np.nan), axis=0)
+ cell_counts = ct.stack()
+ cell_counts.index = cell_counts.index.set_names(["sample", "category"])
+ cell_counts = cell_counts.rename("n_cells").reset_index()
+ per_sample = props_mat.reset_index().melt(
+ id_vars="sample", var_name="category", value_name="proportion"
+ )
+ per_sample = per_sample.merge(
+ cell_counts, on=["sample", "category"], how="left"
+ )
+
+ subject_vals = (
+ np.asarray(store.cells.fetch(subject_by, key=cell_key))[valid]
+ if subject_by is not None
+ else None
+ )
+ pair_vals = (
+ np.asarray(store.cells.fetch(pair_by, key=cell_key))[valid]
+ if pair_by is not None
+ else None
+ )
+ condition_vals = (
+ np.asarray(store.cells.fetch(condition_by, key=cell_key))[valid]
+ if condition_by is not None
+ else None
+ )
+ per_sample = _attach_sample_meta(
+ per_sample,
+ samples=sample_vals_valid,
+ subject_vals=subject_vals,
+ pair_vals=pair_vals,
+ condition_vals=condition_vals,
+ )
+ if pair_by is not None:
+ pair_col = "pair"
+ elif subject_by is not None:
+ pair_col = "subject"
+ if pair_col is not None:
+ sample_pairs = per_sample[["sample", pair_col]].drop_duplicates()
+ missing_pair = pd.isna(sample_pairs[pair_col]) | (
+ sample_pairs[pair_col].astype(str) == ""
+ )
+ n_unpaired_samples = int(missing_pair.sum())
+
+ aggregate = (
+ per_sample.groupby("category", observed=False)["proportion"]
+ .mean()
+ .reindex(cat_order)
+ .reset_index()
+ )
+ if kind == "per_sample" and show_summary:
+ summary_source = per_sample
+ if condition_by is not None:
+ valid_condition = pd.notna(per_sample["condition"]) & (
+ per_sample["condition"].astype(str) != ""
+ )
+ summary_source = per_sample.loc[valid_condition]
+ summary_table = _summarize_proportions(
+ summary_source,
+ by_condition=condition_by is not None,
+ uncertainty=resolved_uncertainty,
+ )
+
+ nonmissing_order = [
+ category for category in cat_order if category != _MISSING_CATEGORY
+ ]
+ palette = categorical_color_map(
+ nonmissing_order,
+ palette=categorical_scale.palette if categorical_scale else None,
+ palette_name=(
+ categorical_scale.palette_name if categorical_scale else "default"
+ ),
+ )
+ resolved_missing_color = (
+ categorical_scale.missing_color if categorical_scale is not None else "#bdbdbd"
+ )
+ if missing_mask.any():
+ palette[_MISSING_CATEGORY] = resolved_missing_color
+ display_labels = (
+ categorical_scale.labels
+ if categorical_scale is not None and categorical_scale.labels is not None
+ else {}
+ )
+
+ def category_label(value: Any) -> str:
+ if value == _MISSING_CATEGORY:
+ return missing_label
+ return display_labels.get(value, str(value))
+
+ panel_key: Hashable = "composition"
+ resolved_figsize = figsize
+ if resolved_figsize is None and target is None:
+ if kind == "per_sample" and pair_col is not None:
+ n_conditions = max(
+ 2,
+ int(per_sample["condition"].nunique()) if per_sample is not None else 2,
+ )
+ resolved_figsize = capped_figsize(
+ max(5.5, 0.42 * len(cat_order) * n_conditions + 2.0),
+ 3.8,
+ max_width=max_figure_width,
+ )
+ elif kind == "stacked" and sample_by is not None and per_sample is not None:
+ n_samples = int(per_sample["sample"].nunique())
+ resolved_figsize = capped_figsize(
+ max(4.5, 0.28 * n_samples + 1.8),
+ 3.6,
+ max_width=max_figure_width,
+ )
+ else:
+ resolved_figsize = capped_figsize(
+ max(4.5, 0.35 * len(cat_order) + 1.8),
+ 3.6,
+ max_width=max_figure_width,
+ )
+ edgecolor = scatter_edgecolor(theme)
+ segment_border = segment_edgecolor or foreground_color(theme)
+
+ with theme_context(theme):
+ fig, axes, owns = normalize_axes_target(
+ target,
+ panel_keys=[panel_key],
+ figsize=resolved_figsize,
+ )
+ ax = axes[panel_key]
+ if kind == "stacked":
+
+ def segment_text_color(face: str) -> str:
+ rgb = mpl.colors.to_rgb(face)
+ luminance = 0.2126 * rgb[0] + 0.7152 * rgb[1] + 0.0722 * rgb[2]
+ return "#111111" if luminance > 0.58 else "#ffffff"
+
+ if sample_by is None:
+ bottom = 0.0
+ for cat in cat_order:
+ val = float(
+ aggregate.loc[aggregate["category"] == cat, "proportion"].iloc[
+ 0
+ ]
+ )
+ ax.bar(
+ 0,
+ val,
+ bottom=bottom,
+ color=palette[cat],
+ label=category_label(cat),
+ width=bar_width,
+ edgecolor=segment_border,
+ linewidth=segment_linewidth,
+ )
+ if show_percent_labels and val >= label_min_fraction:
+ ax.text(
+ 0,
+ bottom + val / 2,
+ percent_format.format(val),
+ ha="center",
+ va="center",
+ color=segment_text_color(palette[cat]),
+ fontsize=7,
+ clip_on=True,
+ )
+ bottom += val
+ ax.set_xticks([])
+ ax.set_ylabel("proportion")
+ else:
+ assert per_sample is not None and props_mat is not None
+ samples_order = list(props_mat.index)
+ bottoms = np.zeros(len(samples_order))
+ x = np.arange(len(samples_order)) * (bar_width + bar_gap)
+ for cat in cat_order:
+ heights = props_mat[cat].to_numpy(dtype=np.float64)
+ heights = np.nan_to_num(heights, nan=0.0)
+ ax.bar(
+ x,
+ heights,
+ bottom=bottoms,
+ color=palette[cat],
+ label=category_label(cat),
+ width=bar_width,
+ edgecolor=segment_border,
+ linewidth=segment_linewidth,
+ )
+ if show_percent_labels:
+ for xpos, bottom, height in zip(x, bottoms, heights):
+ if height < label_min_fraction:
+ continue
+ ax.text(
+ xpos,
+ bottom + height / 2,
+ percent_format.format(height),
+ ha="center",
+ va="center",
+ color=segment_text_color(palette[cat]),
+ fontsize=6.5,
+ clip_on=True,
+ )
+ bottoms += heights
+ ax.set_xticks(x)
+ ax.set_xticklabels(
+ [str(s) for s in samples_order], rotation=45, ha="right"
+ )
+ ax.set_xlabel(sample_by)
+ ax.set_ylabel("proportion")
+ if show_legend:
+ legend_kwargs = {
+ "frameon": False,
+ "title": category_by,
+ "ncols": max(1, int(np.ceil(len(cat_order) / 20))),
+ "columnspacing": 0.8,
+ }
+ if owns:
+ fig.legend(loc="outside right center", **legend_kwargs)
+ else:
+ ax.legend(
+ loc="upper left",
+ bbox_to_anchor=(1.02, 1),
+ borderaxespad=0,
+ **legend_kwargs,
+ )
+ else:
+ assert per_sample is not None
+ if condition_by is not None:
+ valid_conditions = per_sample["condition"].dropna()
+ valid_conditions = valid_conditions[valid_conditions.astype(str) != ""]
+ condition_order = _sort_conditions(list(valid_conditions.unique()))
+ if not condition_order:
+ raise ValueError("condition_by has no valid sample values")
+ if pair_col is not None and len(condition_order) < 2:
+ raise ValueError(
+ "Paired composition requires at least two condition values"
+ )
+ if pair_col is not None:
+ n_pair_lines = _draw_pair_lines(
+ ax,
+ per_sample,
+ cat_order,
+ condition_order,
+ pair_col,
+ )
+ plot_groups = [
+ (category, condition)
+ for category in cat_order
+ for condition in condition_order
+ ]
+ summary_positions = {
+ (category, condition): float(index)
+ for index, (category, condition) in enumerate(plot_groups)
+ }
+ markers = ["o", "s", "^", "D", "v", "P"]
+ for index, (category, condition) in enumerate(plot_groups):
+ rows = per_sample[
+ (per_sample["category"] == category)
+ & (per_sample["condition"] == condition)
+ ]
+ jitter = (np.arange(len(rows)) - (len(rows) - 1) / 2) * 0.02
+ condition_index = list(condition_order).index(condition)
+ ax.scatter(
+ np.full(len(rows), index) + jitter,
+ rows["proportion"],
+ c=palette[category],
+ s=40,
+ marker=markers[condition_index % len(markers)],
+ edgecolors=edgecolor,
+ linewidths=0.3,
+ zorder=2,
+ )
+ if summary_table is not None:
+ _draw_summary_markers(
+ ax,
+ summary_table,
+ positions=summary_positions,
+ palette=palette,
+ edgecolor=edgecolor,
+ )
+ # One tick per category (block center); conditions use marker shape.
+ n_conditions = len(condition_order)
+ centers = [
+ i * n_conditions + (n_conditions - 1) / 2
+ for i in range(len(cat_order))
+ ]
+ ax.set_xticks(centers)
+ ax.set_xticklabels(
+ [category_label(category) for category in cat_order],
+ rotation=45,
+ ha="right",
+ )
+ for boundary in range(
+ n_conditions,
+ len(plot_groups),
+ n_conditions,
+ ):
+ ax.axvline(boundary - 0.5, color="#bdbdbd", linewidth=0.6, zorder=0)
+ handles = [
+ mpl.lines.Line2D(
+ [],
+ [],
+ marker="o",
+ linestyle="",
+ markerfacecolor=palette[cat],
+ markeredgecolor=edgecolor,
+ markersize=6,
+ label=category_label(cat),
+ )
+ for cat in cat_order
+ ]
+ condition_handles = [
+ mpl.lines.Line2D(
+ [],
+ [],
+ marker=markers[index % len(markers)],
+ linestyle="",
+ markerfacecolor="#9e9e9e",
+ markeredgecolor=edgecolor,
+ markersize=6,
+ label=str(condition),
+ )
+ for index, condition in enumerate(condition_order)
+ ]
+ summary_handle = (
+ _summary_marker_handle(
+ mpl,
+ edgecolor=edgecolor,
+ uncertainty=resolved_uncertainty,
+ )
+ if summary_table is not None
+ else None
+ )
+ if show_legend:
+ blocks = [
+ (category_by, handles),
+ ("Condition", condition_handles),
+ ]
+ if summary_handle is not None:
+ blocks.append(("Summary", [summary_handle]))
+ if owns:
+ _place_legend_blocks(
+ fig,
+ [
+ (
+ title,
+ block_handles,
+ [
+ str(handle.get_label())
+ for handle in block_handles
+ ],
+ )
+ for title, block_handles in blocks
+ ],
+ )
+ else:
+ _place_axis_legend_blocks(ax, blocks)
+ else:
+ summary_positions = {
+ (category, None): float(index)
+ for index, category in enumerate(cat_order)
+ }
+ for i, cat in enumerate(cat_order):
+ sub = per_sample[per_sample["category"] == cat]
+ jitter = (np.arange(len(sub)) - (len(sub) - 1) / 2) * 0.02
+ ax.scatter(
+ np.full(len(sub), i) + jitter,
+ sub["proportion"],
+ c=palette[cat],
+ s=40,
+ edgecolors=edgecolor,
+ linewidths=0.3,
+ label=category_label(cat),
+ zorder=2,
+ )
+ if summary_table is not None:
+ _draw_summary_markers(
+ ax,
+ summary_table,
+ positions=summary_positions,
+ palette=palette,
+ edgecolor=edgecolor,
+ )
+ ax.set_xticks(range(len(cat_order)))
+ ax.set_xticklabels(
+ [category_label(c) for c in cat_order],
+ rotation=45,
+ ha="right",
+ )
+ summary_handle = (
+ _summary_marker_handle(
+ mpl,
+ edgecolor=edgecolor,
+ uncertainty=resolved_uncertainty,
+ )
+ if summary_table is not None
+ else None
+ )
+ if show_legend:
+ blocks = [(category_by, list(ax.get_legend_handles_labels()[0]))]
+ if summary_handle is not None:
+ blocks.append(("Summary", [summary_handle]))
+ if owns:
+ _place_legend_blocks(
+ fig,
+ [
+ (
+ title,
+ block_handles,
+ [
+ str(handle.get_label())
+ for handle in block_handles
+ ],
+ )
+ for title, block_handles in blocks
+ ],
+ )
+ else:
+ _place_axis_legend_blocks(ax, blocks)
+ ax.set_xlabel(category_by)
+ ax.set_ylabel("proportion")
+ ymax = float(np.nanmax(per_sample["proportion"].to_numpy(dtype=np.float64)))
+ if summary_table is not None:
+ ymax = max(
+ ymax,
+ float(np.nanmax(summary_table["upper"].to_numpy(dtype=np.float64))),
+ )
+ ax.set_ylim(-0.02, min(1.02, max(0.2, ymax * 1.15)))
+ apply_figure_chrome(fig, theme)
+
+ aggregate_table = aggregate.copy()
+ aggregate_table["category"] = aggregate_table["category"].mask(
+ aggregate_table["category"] == _MISSING_CATEGORY,
+ None,
+ )
+ tables = {"aggregate": aggregate_table}
+ if per_sample is not None:
+ per_sample_table = per_sample.copy()
+ per_sample_table["category"] = per_sample_table["category"].mask(
+ per_sample_table["category"] == _MISSING_CATEGORY,
+ None,
+ )
+ tables["per_sample"] = per_sample_table
+ if summary_table is not None:
+ summary_output = summary_table.copy()
+ summary_output["category"] = summary_output["category"].mask(
+ summary_output["category"] == _MISSING_CATEGORY,
+ None,
+ )
+ tables["summary"] = summary_output
+
+ notes = ["composition", kind]
+ if pair_col is not None:
+ notes.append(f"paired_by={pair_col}")
+
+ legends = [LegendSpec(kind="categorical", label=category_by)]
+ if condition_order is not None:
+ legends.append(
+ LegendSpec(
+ kind="marker",
+ label=condition_by,
+ extras={
+ "values": list(condition_order),
+ "markers": [
+ ["o", "s", "^", "D", "v", "P"][index % 6]
+ for index in range(len(condition_order))
+ ],
+ },
+ )
+ )
+
+ result = PlotResult(
+ figure=fig,
+ axes=axes,
+ tables=tables,
+ legends=tuple(legends),
+ scales=(
+ CategoricalScale(
+ order=tuple(nonmissing_order),
+ palette={category: palette[category] for category in nonmissing_order},
+ labels=(dict(display_labels) if display_labels else None),
+ missing_color=resolved_missing_color,
+ missing_label=(
+ categorical_scale.missing_label
+ if categorical_scale is not None
+ else "NA"
+ ),
+ palette_name=(
+ categorical_scale.palette_name
+ if categorical_scale is not None
+ else "default"
+ ),
+ ),
+ ),
+ provenance=PlotProvenance(
+ cell_key=cell_key,
+ n_cells=int(len(cats)),
+ n_samples=int(per_sample["sample"].nunique())
+ if per_sample is not None
+ else None,
+ renderer="matplotlib",
+ notes=tuple(notes),
+ extras={
+ "subject_by": subject_by,
+ "pair_by": pair_by,
+ "condition_by": condition_by,
+ "show_summary": show_summary if kind == "per_sample" else False,
+ "uncertainty": (
+ resolved_uncertainty
+ if kind == "per_sample" and show_summary
+ else "none"
+ ),
+ "n_pair_lines": n_pair_lines,
+ "n_unpaired_samples": n_unpaired_samples,
+ "dropped_sample_cells": dropped_sample_cells,
+ "bar_width": bar_width,
+ "bar_gap": bar_gap,
+ "segment_edgecolor": segment_border,
+ "segment_linewidth": segment_linewidth,
+ "show_percent_labels": show_percent_labels,
+ "label_min_fraction": label_min_fraction,
+ },
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ result.show()
+ return result
diff --git a/scarf/plotting/diagnostics.py b/scarf/plotting/diagnostics.py
new file mode 100644
index 00000000..a58fa2f2
--- /dev/null
+++ b/scarf/plotting/diagnostics.py
@@ -0,0 +1,526 @@
+"""Native diagnostic plots."""
+
+from typing import Any
+
+import numpy as np
+import pandas as pd
+
+from ._contracts import CategoricalScale, PlotProvenance
+from ._deps import require_kneed, require_matplotlib, require_seaborn
+from ._figure import LegendSpec, PlotResult, normalize_axes_target
+from ._style import apply_figure_chrome, sort_categories, theme_context
+
+
+def _clean_axis(ax: Any, *, tick_size: float = 10.0) -> None:
+ ax.tick_params(axis="both", labelsize=tick_size)
+ for spine in ax.spines.values():
+ spine.set_visible(False)
+ ax.set_axisbelow(True)
+ ax.grid(which="major", linestyle="--", alpha=0.4)
+
+
+def _return_result(result: PlotResult, *, show: bool) -> PlotResult:
+ if show:
+ result.show()
+ return result
+
+
+def qc(
+ data: pd.DataFrame,
+ color: str = "steelblue",
+ cmap: str = "tab20",
+ figsize: tuple[float, float] | None = None,
+ label_size: float = 10.0,
+ title_size: float = 10,
+ sup_title: str | None = None,
+ sup_title_size: float = 12,
+ scatter_size: float = 1.0,
+ max_points: int = 10_000,
+ seed: int = 0,
+ show_on_single_row: bool = True,
+ show: bool = True,
+ theme: str = "notebook",
+) -> PlotResult:
+ """Plot grouped distributions for cell-level QC metrics."""
+ require_matplotlib()
+ sns = require_seaborn()
+ if "groups" not in data.columns:
+ raise KeyError("data must contain a 'groups' column")
+ if data.empty:
+ raise ValueError("data must contain at least one row")
+ if max_points < 0:
+ raise ValueError("max_points must be non-negative")
+
+ metric_columns = [column for column in data.columns if column != "groups"]
+ if not metric_columns:
+ raise ValueError("data must contain at least one metric column")
+ if len(set(metric_columns)) != len(metric_columns):
+ raise ValueError("metric column names must be unique")
+
+ numeric_values: dict[Any, np.ndarray] = {}
+ for metric in metric_columns:
+ try:
+ numeric_values[metric] = pd.to_numeric(
+ data[metric], errors="raise"
+ ).to_numpy(dtype=np.float64)
+ except (TypeError, ValueError) as exc:
+ raise TypeError(f"QC metric {metric!r} must be numeric") from exc
+
+ group_values = data["groups"].astype(object).where(data["groups"].notna(), "NA")
+ group_order = sort_categories(list(pd.unique(group_values)))
+ n_groups = len(group_order)
+ n_panels = len(metric_columns)
+ n_columns = n_panels if show_on_single_row else 1
+ n_rows = 1 if show_on_single_row else n_panels
+ resolved_figsize = figsize
+ if resolved_figsize is None:
+ resolved_figsize = (
+ min(15.0, n_groups + (2.0 * n_columns)),
+ 1.0 + (2.5 * n_rows),
+ )
+
+ palette_colors = sns.color_palette(cmap, n_colors=n_groups).as_hex()
+ palette = dict(zip(group_order, palette_colors, strict=True))
+ if n_groups == 1:
+ palette[group_order[0]] = color
+
+ long_frames: list[pd.DataFrame] = []
+ displayed_points: dict[str, int] = {}
+ with theme_context(theme):
+ figure, axes, owns_figure = normalize_axes_target(
+ None,
+ panel_keys=metric_columns,
+ figsize=resolved_figsize,
+ n_columns=n_columns,
+ )
+ for metric in metric_columns:
+ ax = axes[metric]
+ metric_frame = pd.DataFrame(
+ {
+ "group": group_values.to_numpy(),
+ "metric": str(metric),
+ "value": numeric_values[metric],
+ }
+ )
+ long_frames.append(metric_frame)
+ collections_before = len(ax.collections)
+ if n_groups == 1:
+ sns.violinplot(
+ data=metric_frame,
+ x="group",
+ y="value",
+ order=group_order,
+ ax=ax,
+ color=color,
+ linewidth=1,
+ inner=None,
+ cut=0,
+ saturation=1,
+ )
+ else:
+ sns.violinplot(
+ data=metric_frame,
+ x="group",
+ y="value",
+ order=group_order,
+ hue="group",
+ hue_order=group_order,
+ palette=palette,
+ legend=False,
+ dodge=False,
+ ax=ax,
+ linewidth=1,
+ inner=None,
+ cut=0,
+ saturation=1,
+ )
+ for collection in ax.collections[collections_before:]:
+ collection.set_alpha(0.6)
+
+ if max_points == 0:
+ display_frame = metric_frame.iloc[:0]
+ elif len(metric_frame) > max_points:
+ display_frame = metric_frame.sample(
+ n=max_points,
+ random_state=seed,
+ )
+ else:
+ display_frame = metric_frame
+ displayed_points[str(metric)] = len(display_frame)
+ if not display_frame.empty:
+ sns.stripplot(
+ data=display_frame,
+ x="group",
+ y="value",
+ order=group_order,
+ jitter=0.4,
+ ax=ax,
+ size=scatter_size,
+ color="black",
+ alpha=0.4,
+ )
+
+ ax.set_ylabel(str(metric), fontsize=label_size)
+ ax.set_xlabel("")
+ if n_groups == 1:
+ ax.set_xticks([])
+ finite = numeric_values[metric][np.isfinite(numeric_values[metric])]
+ if len(finite):
+ ax.set_title(
+ f"Median: {np.median(finite):.1f}",
+ fontsize=title_size,
+ )
+ _clean_axis(ax, tick_size=label_size)
+
+ if sup_title is not None:
+ figure.suptitle(sup_title, fontsize=sup_title_size)
+ apply_figure_chrome(figure, theme)
+
+ long_data = pd.concat(long_frames, ignore_index=True)
+ summary = (
+ long_data.groupby(["group", "metric"], observed=False, dropna=False, sort=False)
+ .agg(count=("value", "count"), median=("value", "median"))
+ .reset_index()
+ )
+ result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables={
+ "data": data.copy(),
+ "summary": summary,
+ },
+ legends=(
+ LegendSpec(
+ kind="categorical",
+ label="groups",
+ extras={"categories": list(group_order)},
+ ),
+ ),
+ scales=(
+ CategoricalScale(
+ order=tuple(group_order),
+ palette=palette,
+ ),
+ ),
+ provenance=PlotProvenance(
+ n_cells=len(data),
+ renderer="matplotlib",
+ notes=("qc",),
+ extras={
+ "metrics": [str(metric) for metric in metric_columns],
+ "groups": list(group_order),
+ "max_points": max_points,
+ "seed": seed,
+ "displayed_points": displayed_points,
+ "show_on_single_row": show_on_single_row,
+ },
+ ),
+ owns_figure=owns_figure,
+ theme=theme,
+ )
+ return _return_result(result, show=show)
+
+
+def elbow(
+ variance_explained: np.ndarray | list[float],
+ figsize: tuple[float | None, float] = (None, 2),
+ theme: str = "notebook",
+ show: bool = True,
+) -> PlotResult:
+ """Plot explained variance and mark the detected elbow."""
+ require_matplotlib()
+ values = np.asarray(variance_explained, dtype=np.float64)
+ if values.ndim != 1 or len(values) == 0:
+ raise ValueError("variance_explained must be a non-empty one-dimensional array")
+ if not np.isfinite(values).all():
+ raise ValueError("variance_explained must contain only finite values")
+
+ components = np.arange(len(values), dtype=np.int64)
+ knee_locator = require_kneed()
+ locator = knee_locator(
+ components,
+ values,
+ S=1.0,
+ curve="convex",
+ direction="decreasing",
+ )
+ detected = getattr(locator, "elbow", None)
+ elbow_component = None if detected is None else int(detected)
+ width, height = figsize
+ resolved_figsize = (
+ 0.25 * len(values) if width is None else float(width),
+ float(height),
+ )
+
+ with theme_context(theme):
+ figure, axes, owns_figure = normalize_axes_target(
+ None,
+ panel_keys=["elbow"],
+ figsize=resolved_figsize,
+ )
+ ax = axes["elbow"]
+ ax.plot(components, values, linewidth=1)
+ ax.set_xticks(components)
+ if elbow_component is not None:
+ ax.axvline(
+ elbow_component,
+ linewidth=1,
+ color="red",
+ label="Elbow",
+ )
+ ax.legend(frameon=False, fontsize=9)
+ ax.set_ylabel("% Variance explained", fontsize=9)
+ ax.set_xlabel("Principal components", fontsize=9)
+ _clean_axis(ax, tick_size=8)
+ apply_figure_chrome(figure, theme)
+
+ table = pd.DataFrame(
+ {
+ "component": components,
+ "variance_explained": values,
+ "is_elbow": components == elbow_component,
+ }
+ )
+ result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables={"variance_explained": table},
+ legends=(
+ ()
+ if elbow_component is None
+ else (
+ LegendSpec(
+ kind="line",
+ label="Elbow",
+ scale_key="component",
+ extras={"component": elbow_component},
+ ),
+ )
+ ),
+ scales=(),
+ provenance=PlotProvenance(
+ n_cells=0,
+ renderer="matplotlib",
+ notes=("elbow",),
+ extras={
+ "elbow": elbow_component,
+ "n_components": len(values),
+ "sensitivity": 1.0,
+ },
+ ),
+ owns_figure=owns_figure,
+ theme=theme,
+ )
+ return _return_result(result, show=show)
+
+
+def graph_qc(
+ graph: Any,
+ theme: str = "notebook",
+ show: bool = True,
+) -> PlotResult:
+ """Plot node degree and edge weight distributions for a sparse graph."""
+ require_matplotlib()
+ shape = getattr(graph, "shape", None)
+ if shape is None or len(shape) != 2:
+ raise TypeError("graph must be a two-dimensional sparse matrix")
+ if not hasattr(graph, "data"):
+ raise TypeError("graph must expose sparse edge weights through .data")
+
+ try:
+ degrees = np.asarray((graph != 0).sum(axis=0)).ravel().astype(np.int64)
+ except (AttributeError, TypeError, ValueError) as exc:
+ raise TypeError(
+ "graph must support sparse non-zero degree calculation"
+ ) from exc
+ edge_weights = np.asarray(graph.data, dtype=np.float64).ravel()
+ node_degrees = pd.DataFrame(
+ {
+ "node": np.arange(len(degrees), dtype=np.int64),
+ "degree": degrees,
+ }
+ )
+ degree_frequencies = (
+ pd.Series(degrees, name="degree")
+ .value_counts(sort=False)
+ .sort_index()
+ .rename_axis("degree")
+ .reset_index(name="frequency")
+ )
+ degree_clip_limit = (
+ float(np.percentile(degrees, 99.5) + 5.0) if len(degrees) else 1.0
+ )
+
+ with theme_context(theme):
+ figure, axes, owns_figure = normalize_axes_target(
+ None,
+ panel_keys=["node_degree", "edge_weight"],
+ figsize=(12.0, 4.0),
+ n_columns=2,
+ )
+ degree_ax = axes["node_degree"]
+ degree_ax.bar(
+ degree_frequencies["degree"],
+ degree_frequencies["frequency"],
+ width=0.5,
+ )
+ degree_ax.set_xlim((0.0, max(1.0, degree_clip_limit)))
+ degree_ax.set_xlabel("Node degree")
+ degree_ax.set_ylabel("Frequency")
+ if len(degrees) and degrees.max() > degree_clip_limit:
+ degree_ax.text(
+ max(1.0, degree_clip_limit),
+ float(degree_frequencies["frequency"].max()),
+ f"plot is clipped (max degree: {degrees.max()})",
+ ha="right",
+ fontsize=9,
+ )
+ _clean_axis(degree_ax)
+
+ weight_ax = axes["edge_weight"]
+ weight_ax.hist(edge_weights, bins=30)
+ weight_ax.set_xlabel("Edge weight")
+ weight_ax.set_ylabel("Frequency")
+ _clean_axis(weight_ax)
+ apply_figure_chrome(figure, theme)
+
+ result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables={
+ "node_degrees": node_degrees,
+ "degree_frequencies": degree_frequencies,
+ "edge_weights": pd.DataFrame({"edge_weight": edge_weights}),
+ },
+ legends=(
+ LegendSpec(kind="bar", label="Node degree frequency"),
+ LegendSpec(
+ kind="histogram",
+ label="Edge weight",
+ extras={"bins": 30},
+ ),
+ ),
+ scales=(),
+ provenance=PlotProvenance(
+ n_cells=len(degrees),
+ renderer="matplotlib",
+ notes=("graph_qc",),
+ extras={
+ "graph_shape": [int(shape[0]), int(shape[1])],
+ "n_edges": len(edge_weights),
+ "degree_clip_limit": degree_clip_limit,
+ },
+ ),
+ owns_figure=owns_figure,
+ theme=theme,
+ )
+ return _return_result(result, show=show)
+
+
+def highly_variable_features(
+ mean_nonzero: np.ndarray,
+ corrected_variance: np.ndarray,
+ n_cells: np.ndarray,
+ selected: np.ndarray,
+ *,
+ label_size: float = 12,
+ figsize: tuple[float, float] = (4.5, 4.0),
+ point_sizes: tuple[float, float] = (3, 30),
+ colormaps: tuple[str, str] = ("winter", "magma_r"),
+ theme: str = "notebook",
+ show: bool = True,
+) -> PlotResult:
+ """Plot mean-variance diagnostics for feature selection."""
+ require_matplotlib()
+ means = np.asarray(mean_nonzero, dtype=np.float64)
+ variances = np.asarray(corrected_variance, dtype=np.float64)
+ cell_counts = np.asarray(n_cells, dtype=np.float64)
+ selected_mask = np.asarray(selected)
+ arrays = (means, variances, cell_counts, selected_mask)
+ if any(array.ndim != 1 for array in arrays):
+ raise ValueError("feature diagnostic inputs must be one-dimensional")
+ if len({len(array) for array in arrays}) != 1:
+ raise ValueError("feature diagnostic inputs must have matching lengths")
+ selected_mask = selected_mask.astype(bool, copy=False)
+ if len(point_sizes) != 2:
+ raise ValueError("point_sizes must contain two values")
+ if len(colormaps) != 2:
+ raise ValueError("colormaps must contain two values")
+
+ with np.errstate(divide="ignore", invalid="ignore"):
+ log_means = np.log2(means)
+ log_variances = np.log2(variances)
+
+ with theme_context(theme):
+ figure, axes, owns_figure = normalize_axes_target(
+ None,
+ panel_keys=["highly_variable_features"],
+ figsize=figsize,
+ )
+ ax = axes["highly_variable_features"]
+ ax.scatter(
+ log_means[~selected_mask],
+ log_variances[~selected_mask],
+ alpha=0.6,
+ c=cell_counts[~selected_mask],
+ cmap=colormaps[0],
+ s=point_sizes[0],
+ label="Not selected",
+ )
+ ax.scatter(
+ log_means[selected_mask],
+ log_variances[selected_mask],
+ alpha=0.8,
+ c=cell_counts[selected_mask],
+ cmap=colormaps[1],
+ s=point_sizes[1],
+ edgecolor="black",
+ linewidth=0.5,
+ label="Selected",
+ )
+ ax.set_xlabel("Log mean non-zero expression", fontsize=label_size)
+ ax.set_ylabel("Log corrected variance", fontsize=label_size)
+ ax.legend(frameon=False, fontsize=max(8.0, label_size - 2.0))
+ _clean_axis(ax, tick_size=label_size)
+ apply_figure_chrome(figure, theme)
+
+ feature_table = pd.DataFrame(
+ {
+ "mean_nonzero": means,
+ "corrected_variance": variances,
+ "n_cells": cell_counts,
+ "selected": selected_mask,
+ "log2_mean_nonzero": log_means,
+ "log2_corrected_variance": log_variances,
+ }
+ )
+ result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables={"features": feature_table},
+ legends=(
+ LegendSpec(
+ kind="categorical",
+ label="Highly variable",
+ extras={"categories": ["Not selected", "Selected"]},
+ ),
+ ),
+ scales=(),
+ provenance=PlotProvenance(
+ n_cells=0,
+ renderer="matplotlib",
+ notes=("highly_variable_features",),
+ extras={
+ "n_features": len(means),
+ "n_selected": int(selected_mask.sum()),
+ "max_expressing_cells": (
+ float(np.nanmax(cell_counts)) if len(cell_counts) else 0.0
+ ),
+ "point_sizes": list(point_sizes),
+ "colormaps": list(colormaps),
+ },
+ ),
+ owns_figure=owns_figure,
+ theme=theme,
+ )
+ return _return_result(result, show=show)
diff --git a/scarf/plotting/distribution.py b/scarf/plotting/distribution.py
new file mode 100644
index 00000000..ffd27f3b
--- /dev/null
+++ b/scarf/plotting/distribution.py
@@ -0,0 +1,931 @@
+"""Distribution plots for metadata and feature values."""
+
+from collections.abc import Iterator, Sequence
+from contextlib import contextmanager
+from typing import Any, Hashable, Literal
+
+import numpy as np
+import pandas as pd
+
+from ._contracts import (
+ CategoricalScale,
+ CellField,
+ DistKind,
+ FeatureRef,
+ NormalizationSpec,
+ PlotProvenance,
+ StudyDesign,
+)
+from ._data import (
+ fetch_normalized_feature_matrix,
+ resolve_cell_selection,
+ resolve_feature,
+)
+from ._deps import require_matplotlib, require_seaborn
+from ._display import resolve_categorical_scale
+from ._figure import LegendSpec, PlotResult, normalize_axes_target
+from ._style import (
+ MAX_FIGURE_WIDTH_INCHES,
+ apply_figure_chrome,
+ capped_figsize,
+ categorical_color_map,
+ sort_categories,
+ theme_context,
+)
+
+
+def _cell_index(store: Any, cell_key: str | None) -> np.ndarray:
+ if cell_key is None:
+ return np.arange(store.cells.N, dtype=np.int64)
+ return np.asarray(store.cells.active_index(cell_key))
+
+
+def _fetch_cell_column(store: Any, column: str, cell_key: str | None) -> np.ndarray:
+ if cell_key is None:
+ return np.asarray(store.cells.fetch_all(column))
+ return np.asarray(store.cells.fetch(column, key=cell_key))
+
+
+def _fetch_series(
+ store: Any,
+ key: str | CellField | FeatureRef,
+ *,
+ cell_key: str | None,
+ from_assay: str | None,
+ normalization: NormalizationSpec,
+) -> tuple[np.ndarray, str, bool]:
+ """Return (values, label, is_feature)."""
+ if isinstance(key, CellField):
+ return (
+ _fetch_cell_column(store, key.key, cell_key),
+ key.label or key.key,
+ False,
+ )
+ if isinstance(key, FeatureRef) or (
+ isinstance(key, str) and key not in store.cells.columns
+ ):
+ resolved = resolve_feature(store, key, from_assay=from_assay)
+ cell_idx = _cell_index(store, cell_key)
+ mat = fetch_normalized_feature_matrix(
+ store,
+ [resolved],
+ cell_idx,
+ normalization=normalization,
+ )
+ return mat[:, 0], resolved.label, True
+ return _fetch_cell_column(store, str(key), cell_key), str(key), False
+
+
+def _subsample_frame(
+ df: pd.DataFrame,
+ *,
+ max_points: int,
+ rng: np.random.Generator,
+) -> tuple[pd.DataFrame, bool]:
+ """Return (frame, was_subsampled)."""
+ if max_points <= 0 or len(df) <= max_points:
+ return df, False
+ idx = rng.choice(len(df), size=max_points, replace=False)
+ return df.iloc[np.sort(idx)], True
+
+
+def _group_palette(
+ order: list[Any],
+ categorical_scale: CategoricalScale | None,
+) -> dict[Any, str]:
+ return categorical_color_map(
+ order,
+ palette=categorical_scale.palette if categorical_scale else None,
+ palette_name=(
+ categorical_scale.palette_name if categorical_scale else "default"
+ ),
+ missing_label=None,
+ )
+
+
+@contextmanager
+def _seed_legacy_numpy(rng: np.random.Generator) -> Iterator[None]:
+ state = np.random.get_state()
+ np.random.seed(int(rng.integers(0, np.iinfo(np.uint32).max)))
+ try:
+ yield
+ finally:
+ np.random.set_state(state)
+
+
+def _draw_violin_or_box(
+ ax: Any,
+ sns: Any,
+ df: pd.DataFrame,
+ *,
+ kind: Literal["violin", "box"],
+ color: str,
+ max_points: int,
+ point_size: float,
+ rng: np.random.Generator,
+ order: list[Any] | None = None,
+ palette: dict[Any, str] | None = None,
+ orientation: Literal["vertical", "horizontal"] = "vertical",
+ violin_inner: str | None = "quartile",
+ linewidth: float = 0.8,
+ fill_alpha: float = 0.9,
+ point_alpha: float = 0.28,
+ split_order: list[Any] | None = None,
+ split_palette: dict[Any, str] | None = None,
+ show_legend: bool = True,
+) -> bool:
+ plot_kw: dict[str, Any] = {}
+ if order is not None:
+ plot_kw["order"] = order
+ split = (
+ split_order is not None and split_palette is not None and len(split_order) == 2
+ )
+ grouped = order is not None and palette is not None and len(order) > 0
+ if split:
+ plot_kw["hue"] = "split"
+ plot_kw["hue_order"] = split_order
+ plot_kw["palette"] = split_palette
+ plot_kw["legend"] = show_legend
+ elif grouped:
+ plot_kw["hue"] = "group"
+ plot_kw["hue_order"] = order
+ plot_kw["palette"] = palette
+ plot_kw["legend"] = False
+ plot_kw["dodge"] = False
+
+ x_key = "group" if orientation == "vertical" else "value"
+ y_key = "value" if orientation == "vertical" else "group"
+ collections_before = len(ax.collections)
+ if kind == "violin":
+ sns.violinplot(
+ data=df,
+ x=x_key,
+ y=y_key,
+ ax=ax,
+ color=None if grouped else color,
+ inner=violin_inner,
+ cut=0,
+ linewidth=linewidth,
+ saturation=0.9,
+ split=split,
+ **plot_kw,
+ )
+ else:
+ sns.boxplot(
+ data=df,
+ x=x_key,
+ y=y_key,
+ ax=ax,
+ color=None if grouped else color,
+ showfliers=max_points <= 0,
+ linewidth=linewidth,
+ fliersize=2,
+ **plot_kw,
+ )
+ for collection in ax.collections[collections_before:]:
+ collection.set_alpha(fill_alpha)
+
+ subsampled = False
+ if max_points > 0 and len(df) > 0:
+ pts, subsampled = _subsample_frame(df, max_points=max_points, rng=rng)
+ strip_kw = {k: v for k, v in plot_kw.items() if k != "legend"}
+ if split:
+ strip_kw["legend"] = False
+ strip_kw["dodge"] = True
+ elif "palette" in strip_kw:
+ strip_kw.pop("palette", None)
+ strip_kw.pop("hue", None)
+ with _seed_legacy_numpy(rng):
+ sns.stripplot(
+ data=pts,
+ x=x_key,
+ y=y_key,
+ ax=ax,
+ color=None if split else "0.15",
+ size=point_size,
+ jitter=0.28,
+ alpha=point_alpha,
+ **(
+ strip_kw
+ if split
+ else {k: v for k, v in strip_kw.items() if k == "order"}
+ ),
+ )
+ return subsampled
+
+
+def _sample_aggregate(
+ frame: pd.DataFrame,
+ *,
+ statistic: Literal["mean", "median", "fraction"],
+ expression_cutoff: float,
+ split: bool,
+) -> pd.DataFrame:
+ valid_sample = pd.notna(frame["sample"]) & (frame["sample"].astype(str) != "")
+ frame = frame.loc[valid_sample].copy()
+ if frame.empty:
+ raise ValueError("No selected cells have a valid sample value")
+ columns = ["sample", "group"]
+ if split:
+ columns.append("split")
+ grouped = frame.groupby(columns, observed=False, dropna=False)["raw_value"]
+ if statistic == "mean":
+ values = grouped.mean()
+ elif statistic == "median":
+ values = grouped.median()
+ else:
+ values = grouped.apply(
+ lambda value: float(
+ np.mean(value.to_numpy(dtype=np.float64) > expression_cutoff)
+ )
+ )
+ counts = grouped.size().rename("nCells")
+ return pd.concat(
+ (values.rename("raw_value"), counts),
+ axis=1,
+ ).reset_index()
+
+
+def _draw_hist(
+ ax: Any,
+ df: pd.DataFrame,
+ *,
+ color: str,
+ bins: int,
+ group_by: str | None,
+ order: list[Any] | None,
+ palette: dict[Any, str] | None,
+ show_legend: bool,
+) -> None:
+ if group_by is None:
+ vals = df["value"].to_numpy(dtype=np.float64)
+ vals = vals[np.isfinite(vals)]
+ ax.hist(vals, bins=bins, color=color, edgecolor="white", linewidth=0.4)
+ return
+ all_values = df["value"].to_numpy(dtype=np.float64)
+ all_values = all_values[np.isfinite(all_values)]
+ shared_bins = (
+ np.histogram_bin_edges(all_values, bins=bins) if len(all_values) else bins
+ )
+ levels = order if order is not None else list(pd.unique(df["group"]))
+ for g in levels:
+ sub = df[df["group"] == g]
+ vals = sub["value"].to_numpy(dtype=np.float64)
+ vals = vals[np.isfinite(vals)]
+ if len(vals) == 0:
+ continue
+ face = palette[g] if palette is not None and g in palette else None
+ ax.hist(
+ vals,
+ bins=shared_bins,
+ alpha=0.45,
+ label=str(g),
+ color=face,
+ edgecolor="white",
+ linewidth=0.3,
+ )
+ if show_legend:
+ ax.legend(frameon=False, fontsize=8)
+
+
+def _ecdf_xy(values: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+ v = np.sort(values[np.isfinite(values)])
+ if len(v) == 0:
+ return np.array([]), np.array([])
+ y = np.arange(1, len(v) + 1, dtype=np.float64) / len(v)
+ return v, y
+
+
+def _draw_ecdf(
+ ax: Any,
+ df: pd.DataFrame,
+ *,
+ color: str,
+ max_points: int,
+ rng: np.random.Generator,
+ group_by: str | None,
+ order: list[Any] | None,
+ palette: dict[Any, str] | None,
+ show_legend: bool,
+) -> bool:
+ """Draw ECDF. May subsample values for display; returns whether subsampled."""
+ subsampled = False
+ if group_by is None:
+ vals = df["value"].to_numpy(dtype=np.float64)
+ if max_points > 0 and len(vals) > max_points:
+ vals = vals[rng.choice(len(vals), size=max_points, replace=False)]
+ subsampled = True
+ x, y = _ecdf_xy(vals)
+ if len(x):
+ ax.step(x, y, where="post", color=color, linewidth=1.5)
+ return subsampled
+
+ levels = order if order is not None else list(pd.unique(df["group"]))
+ for g in levels:
+ sub = df[df["group"] == g]
+ vals = sub["value"].to_numpy(dtype=np.float64)
+ if max_points > 0 and len(vals) > max_points:
+ vals = vals[rng.choice(len(vals), size=max_points, replace=False)]
+ subsampled = True
+ x, y = _ecdf_xy(vals)
+ if len(x):
+ line_color = palette[g] if palette is not None and g in palette else None
+ ax.step(
+ x,
+ y,
+ where="post",
+ linewidth=1.2,
+ label=str(g),
+ color=line_color,
+ )
+ if show_legend:
+ ax.legend(frameon=False, fontsize=8)
+ ax.set_ylim(-0.02, 1.02)
+ return subsampled
+
+
+def distribution(
+ store: Any,
+ keys: str | CellField | FeatureRef | Sequence[str | CellField | FeatureRef],
+ *,
+ group_by: str | None = None,
+ groups: Sequence[Any] | None = None,
+ split_by: str | None = None,
+ sample_by: str | None = None,
+ study_design: StudyDesign | None = None,
+ sample_stat: Literal["mean", "median", "fraction"] = "mean",
+ expression_cutoff: float = 0.0,
+ subset_by: str | None = None,
+ cell_key: str | None = "I",
+ from_assay: str | None = None,
+ normalization: NormalizationSpec | None = None,
+ categorical_scale: CategoricalScale | None = None,
+ split_scale: CategoricalScale | None = None,
+ kind: DistKind = "violin",
+ bins: int = 40,
+ max_points: int = 10000,
+ point_size: float = 0.8,
+ point_alpha: float = 0.28,
+ seed: int = 0,
+ color: str = "steelblue",
+ orientation: Literal["vertical", "horizontal"] = "vertical",
+ row_standardize: bool = False,
+ share_y: bool | None = None,
+ violin_inner: str | None = "quartile",
+ violin_linewidth: float = 0.8,
+ violin_alpha: float = 0.9,
+ italicize_features: bool = False,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ max_figure_width: float | None = MAX_FIGURE_WIDTH_INCHES,
+ title: str | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+) -> PlotResult:
+ """Compare value distributions for QC metrics or genes.
+
+ ``keys`` may be cell-metadata columns (for example ``RNA_nCounts``) or gene
+ names. ``kind`` selects the display: ``"violin"``, ``"box"``, ``"hist"``,
+ or ``"ecdf"``. With ``group_by``, each category gets its own distribution
+ along the x-axis and a distinct color.
+
+ Cell selection (same knobs as embeddings):
+
+ - ``cell_key``: boolean metadata column selecting cells (default ``"I"``).
+ Pass ``None`` to include every cell, including those marked inactive.
+ - ``subset_by``: boolean metadata column; keep ``True`` cells
+ - ``groups``: keep / order these ``group_by`` categories
+
+ For violins and boxes, Scarf can overlay a subsample of cells as points.
+ ``max_points`` limits how many points are drawn (``0`` disables points).
+ Histograms always use every selected cell. Regular multi-gene violin and
+ box panels share their value scale by default. Stacked violins keep
+ independent scales unless ``share_y=True``. With horizontal orientation,
+ the same option shares the x-axis value scale.
+
+ Set ``sample_by`` or ``study_design`` to summarize cells within biological
+ samples before plotting. ``split_by`` draws two violin halves for a second
+ categorical variable.
+ """
+ require_matplotlib()
+ normalization = normalization or NormalizationSpec()
+ if kind not in ("violin", "stacked_violin", "box", "hist", "ecdf"):
+ raise ValueError(
+ "kind must be 'violin', 'stacked_violin', 'box', 'hist', or 'ecdf'"
+ )
+ if orientation not in ("vertical", "horizontal"):
+ raise ValueError("orientation must be 'vertical' or 'horizontal'")
+ if kind in ("hist", "ecdf") and orientation != "vertical":
+ raise ValueError("orientation applies only to violin and box plots")
+ if row_standardize and kind != "stacked_violin":
+ raise ValueError("row_standardize is available only for stacked_violin")
+ if share_y is not None and kind not in ("violin", "stacked_violin", "box"):
+ raise ValueError("share_y applies only to violin and box plots")
+ if violin_linewidth < 0:
+ raise ValueError("violin_linewidth must be non-negative")
+ if not 0 <= violin_alpha <= 1 or not 0 <= point_alpha <= 1:
+ raise ValueError("violin and point alpha values must be between 0 and 1")
+ if groups is not None and group_by is None:
+ raise ValueError("groups requires group_by")
+ if split_by is not None and group_by is None:
+ raise ValueError("split_by requires group_by")
+ if split_by is not None and kind not in ("violin", "stacked_violin"):
+ raise ValueError("split_by is available only for violin plots")
+ if split_by == group_by and split_by is not None:
+ raise ValueError("split_by and group_by must refer to different columns")
+ if sample_stat not in ("mean", "median", "fraction"):
+ raise ValueError("sample_stat must be 'mean', 'median', or 'fraction'")
+ if study_design is not None:
+ if sample_by is not None and sample_by != study_design.sample_by:
+ raise ValueError("sample_by conflicts with study_design.sample_by")
+ sample_by = study_design.sample_by
+ if kind in ("violin", "stacked_violin", "box"):
+ sns = require_seaborn()
+ else:
+ sns = None
+ if bins < 1:
+ raise ValueError("bins must be >= 1")
+
+ if isinstance(keys, (str, CellField, FeatureRef)):
+ key_list: list[str | CellField | FeatureRef] = [keys]
+ else:
+ key_list = list(keys)
+ if not key_list:
+ raise ValueError("keys must be non-empty")
+ if group_by is not None:
+ categorical_scale = resolve_categorical_scale(
+ store,
+ group_by,
+ categorical_scale,
+ )
+ if split_by is not None:
+ split_scale = resolve_categorical_scale(
+ store,
+ split_by,
+ split_scale,
+ )
+ feature_assays: set[str] = set()
+ for key in key_list:
+ if not (
+ isinstance(key, FeatureRef)
+ or (isinstance(key, str) and key not in store.cells.columns)
+ ):
+ continue
+ assay_name = (
+ key.assay
+ if isinstance(key, FeatureRef) and key.assay is not None
+ else from_assay or store._defaultAssay
+ )
+ feature_assays.add(assay_name)
+
+ series_list = [
+ _fetch_series(
+ store,
+ k,
+ cell_key=cell_key,
+ from_assay=from_assay,
+ normalization=normalization,
+ )
+ for k in key_list
+ ]
+ n = len(series_list[0][0])
+ if group_by is not None:
+ groups_arr = _fetch_cell_column(store, group_by, cell_key)
+ if len(groups_arr) != n:
+ raise ValueError("group_by length does not match selected cells")
+ else:
+ groups_arr = np.zeros(n, dtype=int)
+ split_arr = (
+ _fetch_cell_column(store, split_by, cell_key) if split_by is not None else None
+ )
+ sample_arr = (
+ _fetch_cell_column(store, sample_by, cell_key)
+ if sample_by is not None
+ else None
+ )
+ if split_arr is not None and len(split_arr) != n:
+ raise ValueError("split_by length does not match selected cells")
+ if sample_arr is not None and len(sample_arr) != n:
+ raise ValueError("sample_by length does not match selected cells")
+
+ subset_vals = (
+ _fetch_cell_column(store, subset_by, cell_key)
+ if subset_by is not None
+ else None
+ )
+ selection_mask, group_order = resolve_cell_selection(
+ n,
+ subset=subset_vals,
+ subset_name=subset_by,
+ category_values=groups_arr if group_by is not None else None,
+ groups=groups,
+ )
+ dropped_sample_cells = 0
+ if sample_arr is not None:
+ valid_sample = pd.notna(sample_arr) & (np.asarray(sample_arr, dtype=str) != "")
+ dropped_sample_cells = int((selection_mask & ~valid_sample).sum())
+ selection_mask &= valid_sample
+ dropped_split_cells = 0
+ if split_arr is not None:
+ valid_split = pd.notna(split_arr) & (np.asarray(split_arr, dtype=str) != "")
+ dropped_split_cells = int((selection_mask & ~valid_split).sum())
+ selection_mask &= valid_split
+ if not selection_mask.any():
+ raise ValueError("No cells remain after distribution selections")
+ if group_by is None:
+ group_order = None
+ elif groups is None:
+ observed_values = list(pd.unique(groups_arr[selection_mask]))
+ if categorical_scale is not None and categorical_scale.order is not None:
+ observed = set(observed_values)
+ missing = [
+ value for value in observed if value not in categorical_scale.order
+ ]
+ if missing:
+ raise ValueError(
+ "categorical_scale.order is missing observed values: "
+ + ", ".join(map(str, missing[:10]))
+ )
+ group_order = [
+ value for value in categorical_scale.order if value in observed
+ ]
+ else:
+ group_order = sort_categories(observed_values)
+
+ series_list = [
+ (np.asarray(vals)[selection_mask], label, is_feature)
+ for vals, label, is_feature in series_list
+ ]
+ groups_arr = groups_arr[selection_mask]
+ if split_arr is not None:
+ split_arr = split_arr[selection_mask]
+ if sample_arr is not None:
+ sample_arr = sample_arr[selection_mask]
+ n = int(selection_mask.sum())
+ any_feature = any(is_feature for _, _, is_feature in series_list)
+
+ panel_keys: list[Hashable] = [label for _, label, _ in series_list]
+ if len(set(panel_keys)) != len(panel_keys):
+ panel_keys = list(range(len(panel_keys)))
+
+ n_groups = 1 if group_by is None else len(group_order or [])
+ n_panels = len(panel_keys)
+ palette = (
+ _group_palette(list(group_order), categorical_scale)
+ if group_by is not None and group_order is not None
+ else None
+ )
+ split_order: list[Any] | None = None
+ split_palette: dict[Any, str] | None = None
+ if split_arr is not None:
+ observed_split = list(pd.unique(split_arr))
+ if split_scale is not None and split_scale.order is not None:
+ missing_split = [
+ value for value in observed_split if value not in split_scale.order
+ ]
+ if missing_split:
+ raise ValueError(
+ "split_scale.order is missing observed values: "
+ + ", ".join(map(str, missing_split[:10]))
+ )
+ split_order = [
+ value for value in split_scale.order if value in set(observed_split)
+ ]
+ else:
+ split_order = sort_categories(observed_split)
+ if len(split_order) != 2:
+ raise ValueError(
+ "split_by must contain exactly two observed categories; "
+ f"found {len(split_order)}"
+ )
+ split_palette = _group_palette(split_order, split_scale)
+ # Width scales with category count so rotated labels stay readable; wrap
+ # to extra rows before exceeding the page width.
+ width_cap = (
+ MAX_FIGURE_WIDTH_INCHES if max_figure_width is None else float(max_figure_width)
+ )
+ if width_cap <= 0:
+ raise ValueError("max_figure_width must be positive or None")
+ panel_width = min(width_cap, max(3.6, 0.55 * max(n_groups, 1) + 1.8))
+ if figsize is None and target is None:
+ n_columns = (
+ 1
+ if kind == "stacked_violin"
+ else max(1, min(n_panels, int(width_cap // panel_width) or 1))
+ )
+ n_rows = int(np.ceil(n_panels / n_columns))
+ row_height = 1.3 if kind == "stacked_violin" else 4.0
+ figsize = capped_figsize(
+ panel_width * n_columns,
+ max(2.4, row_height * n_rows + 0.8),
+ max_width=max_figure_width,
+ )
+ else:
+ n_columns = n_panels
+
+ fig, axes, owns = normalize_axes_target(
+ target,
+ panel_keys=panel_keys,
+ figsize=figsize,
+ n_columns=n_columns,
+ )
+
+ rng = np.random.default_rng(seed)
+ any_subsampled = False
+ y_limits: list[tuple[float, float]] = []
+ panel_tables: list[tuple[str, pd.DataFrame]] = []
+ with theme_context(theme):
+ for panel_index, ((vals, label, is_feature), panel_key) in enumerate(
+ zip(series_list, panel_keys)
+ ):
+ ax = axes[panel_key]
+ cell_frame = pd.DataFrame(
+ {
+ "raw_value": np.asarray(vals, dtype=np.float64),
+ "group": groups_arr,
+ }
+ )
+ if split_arr is not None:
+ cell_frame["split"] = split_arr
+ if sample_arr is not None:
+ cell_frame["sample"] = sample_arr
+ df = _sample_aggregate(
+ cell_frame,
+ statistic=sample_stat,
+ expression_cutoff=expression_cutoff,
+ split=split_arr is not None,
+ )
+ else:
+ df = cell_frame
+ display_values = df["raw_value"].to_numpy(dtype=np.float64)
+ if row_standardize:
+ finite = np.isfinite(display_values)
+ mean = float(np.mean(display_values[finite])) if finite.any() else 0.0
+ std = float(np.std(display_values[finite])) if finite.any() else 0.0
+ display_values = (
+ (display_values - mean) / std
+ if std > 0
+ else np.zeros_like(display_values)
+ )
+ df["value"] = display_values
+ table = df.rename(
+ columns={
+ "raw_value": "value",
+ "value": "display_value",
+ }
+ )
+ panel_tables.append((label, table))
+ if sample_arr is None:
+ value_axis_label = label
+ elif sample_stat == "fraction":
+ value_axis_label = f"Sample fraction > {expression_cutoff:g}"
+ else:
+ value_axis_label = f"Sample {sample_stat} {label}"
+ if kind in ("violin", "stacked_violin", "box"):
+ assert sns is not None
+ subsampled = _draw_violin_or_box(
+ ax,
+ sns,
+ df,
+ kind=("violin" if kind == "stacked_violin" else kind), # type: ignore[arg-type]
+ color=color,
+ max_points=max_points,
+ point_size=point_size,
+ rng=rng,
+ order=None if group_by is None else list(group_order or []),
+ palette=palette,
+ orientation=orientation,
+ violin_inner=violin_inner,
+ linewidth=violin_linewidth,
+ fill_alpha=violin_alpha,
+ point_alpha=point_alpha,
+ split_order=split_order,
+ split_palette=split_palette,
+ show_legend=(
+ show_legend
+ and split_order is not None
+ and panel_index == n_panels - 1
+ ),
+ )
+ any_subsampled = any_subsampled or subsampled
+ if split_order is not None and show_legend:
+ legend = ax.get_legend()
+ if legend is not None:
+ handles, labels = ax.get_legend_handles_labels()
+ legend.remove()
+ ax.legend(
+ handles,
+ labels,
+ title=split_by or "split",
+ frameon=False,
+ loc="upper left",
+ bbox_to_anchor=(1.02, 1),
+ borderaxespad=0,
+ )
+ if group_by is None:
+ if orientation == "vertical":
+ ax.set_xticks([])
+ else:
+ ax.set_yticks([])
+ else:
+ category_axis = "x" if orientation == "vertical" else "y"
+ ax.tick_params(axis=category_axis, labelrotation=45)
+ ticks = (
+ ax.get_xticklabels()
+ if orientation == "vertical"
+ else ax.get_yticklabels()
+ )
+ for tick in ticks:
+ tick.set_ha("right")
+ elif kind == "hist":
+ _draw_hist(
+ ax,
+ df,
+ color=color,
+ bins=bins,
+ group_by=group_by,
+ order=None if group_by is None else list(group_order or []),
+ palette=palette,
+ show_legend=show_legend,
+ )
+ else:
+ subsampled = _draw_ecdf(
+ ax,
+ df,
+ color=color,
+ max_points=max_points,
+ rng=rng,
+ group_by=group_by,
+ order=None if group_by is None else list(group_order or []),
+ palette=palette,
+ show_legend=show_legend,
+ )
+ any_subsampled = any_subsampled or subsampled
+ ax.set_ylabel("ECDF")
+
+ ax.set_title(
+ "" if kind == "stacked_violin" and orientation == "vertical" else label,
+ fontstyle=("italic" if italicize_features and is_feature else "normal"),
+ loc=("left" if kind == "stacked_violin" else "center"),
+ )
+ if kind in ("hist", "ecdf"):
+ ax.set_xlabel(value_axis_label)
+ if kind == "hist":
+ ax.set_ylabel("count")
+ else:
+ # Panel titles already name the key, so the value axis only
+ # repeats it when the title is suppressed or the values were
+ # aggregated per sample.
+ keeps_key_on_axis = kind == "stacked_violin" or sample_arr is not None
+ measurement_label = (
+ "standardized value"
+ if row_standardize
+ else value_axis_label
+ if keeps_key_on_axis
+ else "value"
+ )
+ if orientation == "vertical":
+ ax.set_xlabel(group_by or "")
+ ax.set_ylabel(
+ measurement_label,
+ fontstyle=(
+ "italic"
+ if italicize_features
+ and is_feature
+ and not row_standardize
+ and keeps_key_on_axis
+ else "normal"
+ ),
+ )
+ else:
+ ax.set_xlabel(
+ measurement_label,
+ fontstyle=(
+ "italic"
+ if italicize_features
+ and is_feature
+ and not row_standardize
+ and keeps_key_on_axis
+ else "normal"
+ ),
+ )
+ ax.set_ylabel(group_by or "")
+ if kind == "stacked_violin" and panel_index < n_panels - 1:
+ if orientation == "vertical":
+ ax.tick_params(axis="x", labelbottom=False)
+ ax.set_xlabel("")
+ else:
+ ax.tick_params(axis="y", labelleft=False)
+ ax.set_ylabel("")
+ finite = display_values[np.isfinite(display_values)]
+ if len(finite):
+ y_limits.append((float(finite.min()), float(finite.max())))
+
+ resolved_share_y = (
+ any_feature and kind in ("violin", "box") if share_y is None else share_y
+ )
+ if resolved_share_y and len(y_limits) > 1:
+ ymin = min(lo for lo, _ in y_limits)
+ ymax = max(hi for _, hi in y_limits)
+ if ymax > ymin:
+ pad = 0.05 * (ymax - ymin)
+ for ax in axes.values():
+ if orientation == "vertical":
+ ax.set_ylim(ymin - pad, ymax + pad)
+ else:
+ ax.set_xlim(ymin - pad, ymax + pad)
+
+ if title is not None:
+ fig.suptitle(title)
+ apply_figure_chrome(fig, theme)
+
+ label_counts = pd.Series([label for _, label, _ in series_list]).value_counts()
+ tables = {}
+ for index, (label, table) in enumerate(panel_tables):
+ table_name = label if label_counts[label] == 1 else f"{index}:{label}"
+ tables[table_name] = table
+ notes = ["distribution", kind]
+ if any_subsampled:
+ notes.append("subsampled_display")
+
+ result = PlotResult(
+ figure=fig,
+ axes=axes,
+ tables=tables,
+ legends=(
+ (LegendSpec(kind="categorical", label=split_by or "split"),)
+ if split_order is not None
+ else (LegendSpec(kind="distribution", label=kind),)
+ ),
+ scales=(
+ (
+ CategoricalScale(
+ order=tuple(split_order),
+ palette=split_palette,
+ labels=(split_scale.labels if split_scale is not None else None),
+ missing_color=(
+ split_scale.missing_color
+ if split_scale is not None
+ else "#bdbdbd"
+ ),
+ missing_label=(
+ split_scale.missing_label if split_scale is not None else "NA"
+ ),
+ palette_name=(
+ split_scale.palette_name
+ if split_scale is not None
+ else "default"
+ ),
+ ),
+ )
+ if split_order is not None
+ else (() if categorical_scale is None else (categorical_scale,))
+ ),
+ provenance=PlotProvenance(
+ assay=(next(iter(feature_assays)) if len(feature_assays) == 1 else None),
+ cell_key=cell_key,
+ n_cells=n,
+ n_samples=(
+ int(pd.Series(sample_arr).nunique()) if sample_arr is not None else None
+ ),
+ renderer="matplotlib",
+ notes=tuple(notes),
+ extras={
+ "max_points": max_points,
+ "seed": seed,
+ "group_by": group_by,
+ "groups": None if groups is None else list(groups),
+ "split_by": split_by,
+ "split_order": split_order,
+ "sample_by": sample_by,
+ "sample_stat": sample_stat if sample_by is not None else None,
+ "expression_cutoff": (
+ expression_cutoff
+ if sample_by is not None and sample_stat == "fraction"
+ else None
+ ),
+ "dropped_sample_cells": dropped_sample_cells,
+ "dropped_split_cells": dropped_split_cells,
+ "subset_by": subset_by,
+ "bins": bins if kind == "hist" else None,
+ "title": title,
+ "orientation": orientation,
+ "row_standardize": row_standardize,
+ "share_y": resolved_share_y,
+ "violin_inner": violin_inner,
+ "italicize_features": italicize_features,
+ "approximate": any_subsampled,
+ "normalization": {
+ "source": normalization.source,
+ "transform": normalization.transform,
+ },
+ "assays": sorted(feature_assays),
+ },
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ result.show()
+ return result
diff --git a/scarf/plotting/embedding.py b/scarf/plotting/embedding.py
new file mode 100644
index 00000000..194cf0c0
--- /dev/null
+++ b/scarf/plotting/embedding.py
@@ -0,0 +1,2001 @@
+"""Embedding scatter plots."""
+
+from collections.abc import Sequence
+from typing import Any, Hashable
+
+import numpy as np
+import pandas as pd
+
+from ._contracts import (
+ CategoricalScale,
+ CellField,
+ ColorScale,
+ DensityOverlay,
+ FeatureRef,
+ Highlight,
+ NormalizationSpec,
+ PlotProvenance,
+)
+from ._data import (
+ fetch_normalized_feature_matrix,
+ resolve_cell_selection,
+ resolve_feature,
+)
+from ._deps import require_matplotlib
+from ._display import stored_display_metadata
+from ._figure import LegendSpec, PlotResult, normalize_axes_target
+from ._style import (
+ DEFAULT_PANEL_INCHES,
+ DEFAULT_POINT_EDGEWIDTH,
+ DEFAULT_RASTERIZE_THRESHOLD,
+ LEGEND_SIDE_MAX_ENTRIES,
+ FrameStyle,
+ LegendLoc,
+ apply_figure_chrome,
+ categorical_color_map,
+ continuous_norm,
+ default_point_edgewidth,
+ default_point_size,
+ finish_embedding_axes,
+ legend_side_columns,
+ resolve_legend_loc,
+ scatter_edgecolor,
+ sort_categories,
+ square_axis_limits,
+ theme_context,
+)
+
+
+def _is_categorical(values: pd.Series, kind: str) -> bool:
+ if kind == "categorical":
+ return True
+ if kind == "continuous":
+ return False
+ if values.dtype.name in (
+ "category",
+ "object",
+ "bool",
+ "string",
+ ) or pd.api.types.is_string_dtype(values):
+ return True
+ if pd.api.types.is_integer_dtype(values) and values.nunique(dropna=True) <= 100:
+ return True
+ return False
+
+
+def _coerce_color_items(
+ color_by: str
+ | FeatureRef
+ | CellField
+ | Sequence[str | FeatureRef | CellField]
+ | None,
+) -> list[str | FeatureRef | CellField | None]:
+ if color_by is None:
+ return [None]
+ if isinstance(color_by, (str, FeatureRef, CellField)):
+ return [color_by]
+ return list(color_by)
+
+
+def _coerce_layout_items(layout_key: str | Sequence[str]) -> list[str]:
+ if isinstance(layout_key, str):
+ return [layout_key]
+ layouts = list(layout_key)
+ if not layouts:
+ raise ValueError("layout_key must contain at least one layout")
+ if any(not isinstance(layout, str) for layout in layouts):
+ raise TypeError("Every layout_key entry must be a string")
+ if len(set(layouts)) != len(layouts):
+ raise ValueError("layout_key entries must be unique")
+ return layouts
+
+
+def _color_labels(
+ store: Any,
+ color_items: Sequence[str | FeatureRef | CellField | None],
+ *,
+ from_assay: str | None,
+) -> list[str]:
+ labels: list[str] = []
+ for item in color_items:
+ if item is None:
+ labels.append("cells")
+ elif isinstance(item, CellField):
+ labels.append(item.label or item.key)
+ elif isinstance(item, str) and item in store.cells.columns:
+ labels.append(item)
+ else:
+ labels.append(resolve_feature(store, item, from_assay=from_assay).label)
+ return labels
+
+
+def _embedding_panel_keys(
+ labels: Sequence[str],
+ facets: Sequence[Any],
+) -> list[Hashable]:
+ panel_keys: list[Hashable] = []
+ for label in labels:
+ for facet in facets:
+ panel_keys.append(label if facet is None else (label, facet))
+ if len(set(panel_keys)) != len(panel_keys):
+ return [(index, key) for index, key in enumerate(panel_keys)]
+ return panel_keys
+
+
+def _layout_panel_key(layout: str, panel_key: Hashable) -> Hashable:
+ if isinstance(panel_key, tuple):
+ return (layout, *panel_key)
+ return (layout, panel_key)
+
+
+def _multi_layout_facets(
+ store: Any,
+ *,
+ facet_by: str | None,
+ facet_order: Sequence[Any] | None,
+ groups: Sequence[Any] | None,
+ subset_by: str | None,
+ cell_key: str,
+) -> list[Any]:
+ if facet_by is None:
+ return [None]
+ if groups is not None:
+ return list(groups)
+ if facet_order is not None:
+ return list(facet_order)
+
+ values = np.asarray(store.cells.fetch(facet_by, key=cell_key))
+ subset = (
+ np.asarray(store.cells.fetch(subset_by, key=cell_key))
+ if subset_by is not None
+ else None
+ )
+ selection, _ = resolve_cell_selection(
+ len(values),
+ subset=subset,
+ subset_name=subset_by,
+ )
+ return sort_categories(list(pd.unique(values[selection])))
+
+
+def _embedding_multiple_layouts(
+ store: Any,
+ *,
+ layout_keys: Sequence[str],
+ color_by: str
+ | FeatureRef
+ | CellField
+ | Sequence[str | FeatureRef | CellField]
+ | None,
+ facet_by: str | None,
+ facet_order: Sequence[Any] | None,
+ cell_key: str,
+ from_assay: str | None,
+ normalization: NormalizationSpec | None,
+ groups: Sequence[Any] | None,
+ subset_by: str | None,
+ n_columns: int | None,
+ target: Any | None,
+ figsize: tuple[float, float] | None,
+ theme: str,
+ show: bool,
+ child_kwargs: dict[str, Any],
+) -> PlotResult:
+ color_items = _coerce_color_items(color_by)
+ if not color_items:
+ raise ValueError("color_by must contain at least one item")
+ labels = _color_labels(store, color_items, from_assay=from_assay)
+ facets = _multi_layout_facets(
+ store,
+ facet_by=facet_by,
+ facet_order=facet_order,
+ groups=groups,
+ subset_by=subset_by,
+ cell_key=cell_key,
+ )
+ child_panel_keys = _embedding_panel_keys(labels, facets)
+ panel_keys = [
+ _layout_panel_key(layout, panel_key)
+ for layout in layout_keys
+ for panel_key in child_panel_keys
+ ]
+ resolved_columns = n_columns if n_columns is not None else len(child_panel_keys)
+ resolved_columns = max(1, min(resolved_columns, len(panel_keys)))
+ if figsize is None and target is None:
+ nrows = int(np.ceil(len(panel_keys) / resolved_columns))
+ figsize = (
+ DEFAULT_PANEL_INCHES * resolved_columns + 0.4,
+ DEFAULT_PANEL_INCHES * nrows + 0.2,
+ )
+
+ with theme_context(theme):
+ figure, axes, owns = normalize_axes_target(
+ target,
+ panel_keys=panel_keys,
+ figsize=figsize,
+ n_columns=resolved_columns,
+ )
+
+ children: list[tuple[str, PlotResult]] = []
+ for layout in layout_keys:
+ child_target = {
+ panel_key: axes[_layout_panel_key(layout, panel_key)]
+ for panel_key in child_panel_keys
+ }
+ child = embedding(
+ store,
+ layout_key=layout,
+ color_by=color_by,
+ facet_by=facet_by,
+ facet_order=facets if facet_by is not None else facet_order,
+ cell_key=cell_key,
+ from_assay=from_assay,
+ normalization=normalization,
+ groups=groups,
+ subset_by=subset_by,
+ n_columns=len(child_panel_keys),
+ target=child_target,
+ figsize=None,
+ theme=theme,
+ show=False,
+ **child_kwargs,
+ )
+ children.append((layout, child))
+
+ tables: dict[str, pd.DataFrame] = {}
+ legends: list[LegendSpec] = []
+ scales: list[Any] = []
+ for layout, child in children:
+ tables.update({f"{layout}:{key}": value for key, value in child.tables.items()})
+ for legend in child.legends:
+ if legend not in legends:
+ legends.append(legend)
+ for scale in child.scales:
+ if not any(
+ type(existing) is type(scale) and existing == scale
+ for existing in scales
+ ):
+ scales.append(scale)
+
+ provenances = {layout: child.provenance for layout, child in children}
+ first_provenance = children[0][1].provenance
+ assays = sorted(
+ {
+ assay
+ for provenance in provenances.values()
+ for assay in provenance.extras.get("assays", [])
+ }
+ )
+ assay_values = {
+ provenance.assay
+ for provenance in provenances.values()
+ if provenance.assay is not None
+ }
+ extras = dict(first_provenance.extras)
+ extras.update(
+ {
+ "layouts": list(layout_keys),
+ "n_layouts": len(layout_keys),
+ "panel_keys": [str(key) for key in panel_keys],
+ "layout_provenance": provenances,
+ "color_limits_by_layout": {
+ layout: provenance.extras.get("color_limits")
+ for layout, provenance in provenances.items()
+ },
+ "invalid_coordinate_cells": {
+ layout: provenance.extras.get("invalid_coordinate_cells", 0)
+ for layout, provenance in provenances.items()
+ },
+ "assays": assays,
+ }
+ )
+ notes = tuple(
+ dict.fromkeys(
+ (
+ "embedding",
+ "materialized",
+ "multi_layout",
+ *(
+ note
+ for provenance in provenances.values()
+ for note in provenance.notes
+ if note not in ("embedding", "materialized")
+ ),
+ )
+ )
+ )
+ result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables=tables,
+ legends=tuple(legends),
+ scales=tuple(scales),
+ provenance=PlotProvenance(
+ scarf_version=first_provenance.scarf_version,
+ assay=next(iter(assay_values)) if len(assay_values) == 1 else None,
+ cell_key=cell_key,
+ n_cells=max(provenance.n_cells for provenance in provenances.values()),
+ renderer=first_provenance.renderer,
+ notes=notes,
+ extras=extras,
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ result.show()
+ return result
+
+
+def _prefetch_colors(
+ store: Any,
+ color_items: Sequence[str | FeatureRef | CellField | None],
+ *,
+ from_assay: str | None,
+ cell_key: str,
+ n_cells: int,
+ normalization: NormalizationSpec,
+) -> list[tuple[np.ndarray, str, bool, bool]]:
+ """Return list of (values, label, is_categorical, is_uniform)."""
+ out: list[tuple[np.ndarray, str, bool, bool]] = []
+
+ # Batch RNA-like feature refs / gene strings for one matrix read.
+ feature_slots: list[tuple[int, Any]] = []
+ for i, item in enumerate(color_items):
+ if item is None:
+ out.append((np.ones(n_cells), "cells", False, True))
+ continue
+ if isinstance(item, CellField):
+ vals = store.cells.fetch(item.key, key=cell_key)
+ series = pd.Series(vals)
+ out.append(
+ (
+ np.asarray(vals),
+ item.label or item.key,
+ _is_categorical(series, item.kind),
+ False,
+ )
+ )
+ continue
+ if isinstance(item, str) and item in store.cells.columns:
+ vals = store.cells.fetch(item, key=cell_key)
+ series = pd.Series(vals)
+ out.append((np.asarray(vals), item, _is_categorical(series, "auto"), False))
+ continue
+ # Feature path: placeholder, fill after batch resolve
+ out.append((np.zeros(n_cells), "", False, False))
+ feature_slots.append((i, item))
+
+ if feature_slots:
+ resolved = [
+ resolve_feature(store, item, from_assay=from_assay)
+ for _, item in feature_slots
+ ]
+ cell_idx = store.cells.active_index(cell_key)
+ mat = fetch_normalized_feature_matrix(
+ store,
+ resolved,
+ cell_idx,
+ normalization=normalization,
+ )
+ for col_i, (slot_i, _) in enumerate(feature_slots):
+ feat = resolved[col_i]
+ out[slot_i] = (mat[:, col_i], feat.label, False, False)
+ return out
+
+
+def _continuous_limits(
+ values: np.ndarray,
+ color_scale: ColorScale,
+) -> tuple[float, float]:
+ v = np.asarray(values, dtype=np.float64)
+ finite = np.isfinite(v)
+ if not finite.any():
+ return 0.0, 1.0
+ if color_scale.quantiles is not None:
+ q0, q1 = color_scale.quantiles
+ vmin = float(np.nanquantile(v[finite], q0))
+ vmax = float(np.nanquantile(v[finite], q1))
+ else:
+ vmin = float(np.nanmin(v[finite]))
+ vmax = float(np.nanmax(v[finite]))
+ if color_scale.vmin is not None:
+ vmin = color_scale.vmin
+ if color_scale.vmax is not None:
+ vmax = color_scale.vmax
+ if vmax <= vmin:
+ if color_scale.vmin is not None or color_scale.vmax is not None:
+ if color_scale.scale == "log":
+ if vmin <= 0:
+ raise ValueError("Log color scale requires positive values")
+ vmin *= 0.99
+ vmax *= 1.01
+ else:
+ padding = max(abs(vmin) * 0.01, 0.5)
+ vmin -= padding
+ vmax += padding
+ else:
+ vmax = vmin + 1.0
+ return vmin, vmax
+
+
+def _scatter_edges(edgecolor: str, edgewidth: float) -> tuple[str | float, float]:
+ if edgewidth <= 0:
+ return "none", 0.0
+ return edgecolor, float(edgewidth)
+
+
+def _panel_area_inches(ax: Any) -> float:
+ bounds = ax.get_position()
+ width = max(float(bounds.width * ax.figure.get_figwidth()), 0.1)
+ height = max(float(bounds.height * ax.figure.get_figheight()), 0.1)
+ return width * height
+
+
+def _resolve_highlight_mask(
+ store: Any,
+ highlight: Highlight | None,
+ *,
+ cell_key: str,
+ n_cells: int,
+) -> np.ndarray | None:
+ if highlight is None:
+ return None
+ if highlight.indices is not None:
+ indices = np.asarray(highlight.indices, dtype=np.int64)
+ if len(indices) and int(indices.max()) >= n_cells:
+ raise IndexError("highlight index is outside the selected cell range")
+ mask = np.zeros(n_cells, dtype=bool)
+ mask[indices] = True
+ return mask
+ assert highlight.by is not None
+ values = np.asarray(store.cells.fetch(highlight.by, key=cell_key))
+ if len(values) != n_cells:
+ raise ValueError("highlight metadata length does not match selected cells")
+ if highlight.groups is None:
+ if values.dtype != bool:
+ raise TypeError(
+ "Highlight without groups requires a boolean metadata column"
+ )
+ return values.astype(bool, copy=False)
+ return np.isin(values, np.asarray(highlight.groups, dtype=object))
+
+
+def _density_selection_mask(
+ store: Any,
+ overlay: DensityOverlay | None,
+ *,
+ cell_key: str,
+ n_cells: int,
+) -> np.ndarray | None:
+ if overlay is None or overlay.group_by is None:
+ return None
+ values = np.asarray(store.cells.fetch(overlay.group_by, key=cell_key))
+ if len(values) != n_cells:
+ raise ValueError("density metadata length does not match selected cells")
+ if overlay.groups is None:
+ return np.asarray(pd.notna(values), dtype=bool)
+ return np.isin(values, np.asarray(overlay.groups, dtype=object))
+
+
+def _weighted_quantiles(
+ values: np.ndarray,
+ weights: np.ndarray,
+ quantiles: np.ndarray,
+) -> np.ndarray:
+ order = np.argsort(values)
+ sorted_values = values[order]
+ sorted_weights = weights[order]
+ cumulative = np.cumsum(sorted_weights)
+ if cumulative[-1] <= 0:
+ return np.asarray(
+ np.quantile(sorted_values, quantiles),
+ dtype=np.float64,
+ )
+ return np.asarray(
+ np.interp(quantiles * cumulative[-1], cumulative, sorted_values),
+ dtype=np.float64,
+ )
+
+
+def _smoothed_local_mean(
+ xx: np.ndarray,
+ yy: np.ndarray,
+ values: np.ndarray,
+ *,
+ grid_pixels: int,
+ x_range: tuple[float, float],
+ y_range: tuple[float, float],
+ sigma: float,
+ min_support: float,
+) -> tuple[np.ndarray, np.ndarray]:
+ from scipy.ndimage import gaussian_filter
+
+ numeric_values = np.asarray(values, dtype=np.float64)
+ if len(numeric_values) != len(xx):
+ raise ValueError("contour values must match the selected coordinates")
+ finite = np.isfinite(xx) & np.isfinite(yy) & np.isfinite(numeric_values)
+ sums_xy, _, _ = np.histogram2d(
+ xx[finite],
+ yy[finite],
+ bins=grid_pixels,
+ range=(x_range, y_range),
+ weights=numeric_values[finite],
+ )
+ counts_xy, _, _ = np.histogram2d(
+ xx[finite],
+ yy[finite],
+ bins=grid_pixels,
+ range=(x_range, y_range),
+ )
+ support = gaussian_filter(
+ counts_xy.T,
+ sigma=sigma,
+ mode="constant",
+ )
+ smoothed_sum = gaussian_filter(
+ sums_xy.T,
+ sigma=sigma,
+ mode="constant",
+ )
+ local_mean = np.zeros_like(smoothed_sum)
+ np.divide(
+ smoothed_sum,
+ support,
+ out=local_mean,
+ where=(support > 0) & np.isfinite(smoothed_sum),
+ )
+ support_taper = np.clip(support / min_support, 0.0, 1.0)
+ return local_mean * support_taper**2, support
+
+
+def _retain_strongest_hotspots(
+ surface: np.ndarray,
+ support: np.ndarray,
+ *,
+ level: float,
+ max_hotspots: int | None,
+) -> np.ndarray:
+ if max_hotspots is None:
+ return surface
+ from scipy.ndimage import binary_fill_holes
+ from scipy.ndimage import label as label_components
+
+ above_level = np.isfinite(surface) & (surface >= level)
+ component_ids, n_components = label_components(
+ above_level,
+ structure=np.ones((3, 3), dtype=np.uint8),
+ )
+ if n_components <= max_hotspots:
+ retained = set(range(1, n_components + 1))
+ else:
+ component_scores: list[tuple[float, int]] = []
+ for component_id in range(1, n_components + 1):
+ component_mask = component_ids == component_id
+ excess = np.maximum(surface[component_mask] - level, 0)
+ score = float(
+ np.sum(support[component_mask] * (excess + np.finfo(np.float64).eps))
+ )
+ component_scores.append((score, component_id))
+ component_scores.sort(reverse=True)
+ retained = {component_id for _, component_id in component_scores[:max_hotspots]}
+ remove = above_level & ~np.isin(component_ids, list(retained))
+ filtered = surface.copy()
+ filtered[remove] = np.nextafter(level, -np.inf)
+ filled_retained = np.zeros_like(above_level)
+ for component_id in retained:
+ filled_retained |= binary_fill_holes(component_ids == component_id)
+ holes = filled_retained & ~above_level
+ filtered[holes] = np.nextafter(level, np.inf)
+ return filtered
+
+
+def _draw_density_overlay(
+ ax: Any,
+ xx: np.ndarray,
+ yy: np.ndarray,
+ *,
+ overlay: DensityOverlay,
+ values: np.ndarray | None,
+ xlim: tuple[float, float],
+ ylim: tuple[float, float],
+ theme: str,
+) -> None:
+ if len(xx) < 3:
+ return
+ from scipy.ndimage import gaussian_filter
+
+ padding_pixels = max(2, int(np.ceil(4 * overlay.sigma)))
+ grid_pixels = overlay.pixels + 2 * padding_pixels
+ x_step = (xlim[1] - xlim[0]) / overlay.pixels
+ y_step = (ylim[1] - ylim[0]) / overlay.pixels
+ padded_xlim = (
+ xlim[0] - padding_pixels * x_step,
+ xlim[1] + padding_pixels * x_step,
+ )
+ padded_ylim = (
+ ylim[0] - padding_pixels * y_step,
+ ylim[1] + padding_pixels * y_step,
+ )
+ if overlay.statistic == "mean":
+ if values is None:
+ raise ValueError(
+ "DensityOverlay(statistic='mean') requires a continuous color panel"
+ )
+ surface, support = _smoothed_local_mean(
+ xx,
+ yy,
+ values,
+ grid_pixels=grid_pixels,
+ x_range=padded_xlim,
+ y_range=padded_ylim,
+ sigma=overlay.sigma,
+ min_support=overlay.min_support,
+ )
+ supported = support >= overlay.min_support
+ supported_values = surface[supported & np.isfinite(surface)]
+ if len(supported_values):
+ if np.all(supported_values >= 0):
+ positive_values = supported_values[
+ supported_values
+ > max(float(np.nanmax(supported_values)) * 1e-6, 1e-12)
+ ]
+ level_values = (
+ positive_values if len(positive_values) else supported_values
+ )
+ else:
+ level_values = supported_values
+ else:
+ level_values = supported_values
+ else:
+ from ._raster import density_canvas_from_points
+
+ canvas = density_canvas_from_points(
+ xx,
+ yy,
+ extent=(padded_xlim[0], padded_xlim[1], padded_ylim[0], padded_ylim[1]),
+ pixels=grid_pixels,
+ )
+ support = gaussian_filter(
+ np.flipud(canvas.counts).astype(np.float64),
+ sigma=overlay.sigma,
+ mode="constant",
+ )
+ surface = support
+ level_values = surface[np.isfinite(surface)]
+ finite_surface = level_values
+ if overlay.statistic == "density":
+ finite_surface = finite_surface[finite_surface > 0]
+ if len(finite_surface) == 0:
+ return
+ level_weights: np.ndarray | None = None
+ if overlay.statistic == "mean":
+ level_mask = (support >= overlay.min_support) & np.isfinite(surface)
+ if np.all(finite_surface >= 0):
+ level_mask &= surface > max(
+ float(np.nanmax(finite_surface)) * 1e-6,
+ 1e-12,
+ )
+ level_weights = support[level_mask]
+ if isinstance(overlay.levels, int):
+ quantiles = np.linspace(0.55, 0.97, overlay.levels)
+ levels = np.unique(
+ _weighted_quantiles(finite_surface, level_weights, quantiles)
+ if level_weights is not None
+ else np.quantile(finite_surface, quantiles)
+ )
+ else:
+ requested = np.asarray(overlay.levels, dtype=np.float64)
+ if np.all((0 < requested) & (requested < 1)):
+ levels = np.unique(
+ _weighted_quantiles(finite_surface, level_weights, requested)
+ if level_weights is not None
+ else np.quantile(finite_surface, requested)
+ )
+ else:
+ levels = np.unique(requested)
+ surface_min = float(np.nanmin(surface))
+ surface_max = float(np.nanmax(surface))
+ levels = levels[(levels > surface_min) & (levels < surface_max)]
+ if len(levels) == 0:
+ return
+ surface = _retain_strongest_hotspots(
+ surface,
+ support,
+ level=float(levels[0]),
+ max_hotspots=overlay.max_hotspots,
+ )
+ surface_max = float(np.nanmax(surface))
+ xcentres = np.linspace(
+ padded_xlim[0],
+ padded_xlim[1],
+ grid_pixels,
+ endpoint=False,
+ )
+ ycentres = np.linspace(
+ padded_ylim[0],
+ padded_ylim[1],
+ grid_pixels,
+ endpoint=False,
+ )
+ xcentres += (padded_xlim[1] - padded_xlim[0]) / (2 * grid_pixels)
+ ycentres += (padded_ylim[1] - padded_ylim[0]) / (2 * grid_pixels)
+ color = overlay.color or ("#f0f0f0" if theme == "dark" else "#202020")
+ contour_surface = np.ma.masked_invalid(surface)
+ if overlay.kind == "filled":
+ upper = np.nextafter(surface_max, np.inf)
+ fill_levels = np.unique(np.append(levels, upper))
+ if len(fill_levels) >= 2:
+ ax.contourf(
+ xcentres,
+ ycentres,
+ contour_surface,
+ levels=fill_levels,
+ cmap=overlay.cmap,
+ colors=None if overlay.cmap else [color],
+ alpha=overlay.alpha,
+ antialiased=True,
+ zorder=overlay.zorder,
+ )
+ else:
+ contours = ax.contour(
+ xcentres,
+ ycentres,
+ contour_surface,
+ levels=levels,
+ cmap=overlay.cmap,
+ colors=None if overlay.cmap else color,
+ alpha=overlay.alpha,
+ linewidths=overlay.linewidth,
+ zorder=overlay.zorder,
+ )
+ if overlay.halo_width > 0:
+ from matplotlib import patheffects
+
+ halo_color = overlay.halo_color or (
+ "#202020" if theme == "dark" else "#ffffff"
+ )
+ contours.set_path_effects(
+ [
+ patheffects.withStroke(
+ linewidth=overlay.linewidth + 2 * overlay.halo_width,
+ foreground=halo_color,
+ ),
+ patheffects.Normal(),
+ ]
+ )
+
+
+def _draw_highlight(
+ ax: Any,
+ xx: np.ndarray,
+ yy: np.ndarray,
+ sizes: np.ndarray,
+ *,
+ highlight: Highlight,
+ edgecolor: str,
+ rasterized: bool,
+) -> Any:
+ if len(xx) == 0:
+ return
+ halo = highlight.halo_color or edgecolor
+ return ax.scatter(
+ xx,
+ yy,
+ s=sizes * highlight.size_multiplier,
+ c=highlight.color,
+ alpha=highlight.alpha,
+ edgecolors=halo,
+ linewidths=highlight.halo_width,
+ rasterized=rasterized,
+ zorder=5,
+ )
+
+
+def _draw_categorical(
+ ax: Any,
+ xx: np.ndarray,
+ yy: np.ndarray,
+ vv: np.ndarray,
+ ss: np.ndarray,
+ *,
+ order: list[Any],
+ palette: dict[Any, str],
+ missing_color: str,
+ edgecolor: str,
+ edgewidth: float = DEFAULT_POINT_EDGEWIDTH,
+ alpha: float = 1.0,
+ rasterized: bool,
+) -> Any:
+ colors = [
+ missing_color if pd.isna(val) or val not in palette else palette[val]
+ for val in vv
+ ]
+ edges, lw = _scatter_edges(edgecolor, edgewidth)
+ return ax.scatter(
+ xx,
+ yy,
+ c=colors,
+ s=ss,
+ linewidths=lw,
+ edgecolors=edges,
+ alpha=alpha,
+ rasterized=rasterized,
+ )
+
+
+def _add_categorical_legend(
+ ax: Any,
+ fig: Any,
+ mpl: Any,
+ *,
+ order: list[Any],
+ palette: dict[Any, str],
+ labels: dict[Any, str] | None,
+ label: str,
+ missing: bool,
+ missing_color: str,
+ missing_label: str,
+ edgecolor: str,
+ figure_level: bool,
+ values: np.ndarray | None = None,
+ max_entries: int = LEGEND_SIDE_MAX_ENTRIES,
+) -> list[Any]:
+ if values is not None and len(order) > max_entries:
+ counts = pd.Series(np.asarray(values, dtype=object)).value_counts(dropna=True)
+ ranked = sorted(
+ enumerate(order),
+ key=lambda item: (-int(counts.get(item[1], 0)), item[0]),
+ )
+ selected = {index for index, _ in ranked[:max_entries]}
+ shown = [value for index, value in enumerate(order) if index in selected]
+ omitted = [value for index, value in enumerate(order) if index not in selected]
+ else:
+ shown = list(order[:max_entries])
+ omitted = list(order[max_entries:])
+ handles = [
+ mpl.lines.Line2D(
+ [],
+ [],
+ marker="o",
+ linestyle="",
+ markerfacecolor=palette[value],
+ markeredgecolor=edgecolor,
+ markeredgewidth=0.3,
+ markersize=5,
+ label=(labels.get(value, str(value)) if labels is not None else str(value)),
+ )
+ for value in shown
+ ]
+ if missing:
+ handles.append(
+ mpl.lines.Line2D(
+ [],
+ [],
+ marker="o",
+ linestyle="",
+ markerfacecolor=missing_color,
+ markeredgecolor=edgecolor,
+ markeredgewidth=0.3,
+ markersize=5,
+ label=missing_label,
+ )
+ )
+ title = label or None
+ if omitted:
+ title = f"{label} ({len(shown)} of {len(order)})"
+ legend_kwargs = {
+ "handles": handles,
+ "title": title,
+ "frameon": False,
+ "borderaxespad": 0,
+ "ncols": legend_side_columns(len(handles)),
+ "columnspacing": 0.8,
+ }
+ if figure_level:
+ # Outside legends participate in constrained layout and stay inside
+ # exact-size exports.
+ try:
+ fig.legend(
+ **legend_kwargs,
+ loc="outside right center",
+ )
+ except (TypeError, ValueError):
+ fig.legend(
+ **legend_kwargs,
+ loc="center left",
+ bbox_to_anchor=(1.02, 0.5),
+ )
+ else:
+ ax.legend(
+ **legend_kwargs,
+ loc="upper left",
+ bbox_to_anchor=(1.02, 1.0),
+ )
+ return omitted
+
+
+def _add_on_data_labels(
+ ax: Any,
+ xx: np.ndarray,
+ yy: np.ndarray,
+ vv: np.ndarray,
+ *,
+ order: list[Any],
+ labels: dict[Any, str] | None,
+ theme: str,
+ max_labels: int,
+) -> list[Any]:
+ from matplotlib import patheffects
+
+ text_color = "#f5f5f5" if theme == "dark" else "#222222"
+ stroke = "#222222" if theme == "dark" else "#ffffff"
+ candidates: list[tuple[Any, float, float, int]] = []
+ for value in order:
+ mask = np.asarray([not pd.isna(v) and v == value for v in vv], dtype=bool)
+ if not mask.any():
+ continue
+ candidates.append(
+ (
+ value,
+ float(np.median(xx[mask])),
+ float(np.median(yy[mask])),
+ int(mask.sum()),
+ )
+ )
+ candidates.sort(key=lambda item: (-item[3], str(item[0])))
+ omitted = [value for value, _, _, _ in candidates[max_labels:]]
+ chosen = candidates[:max_labels]
+ if chosen:
+ yrange = max(float(np.ptp(yy)), 1e-12)
+ minimum_gap = 0.035 * yrange
+ ordered = sorted(chosen, key=lambda item: item[2])
+ adjusted: list[tuple[Any, float, float, int]] = []
+ previous_y = -np.inf
+ for value, xpos, ypos, count in ordered:
+ adjusted_y = max(ypos, previous_y + minimum_gap)
+ adjusted.append((value, xpos, adjusted_y, count))
+ previous_y = adjusted_y
+ chosen = adjusted
+ for value, xpos, ypos, _ in chosen:
+ ax.text(
+ xpos,
+ ypos,
+ (labels.get(value, str(value)) if labels is not None else str(value)),
+ ha="center",
+ va="center",
+ fontsize=8,
+ color=text_color,
+ path_effects=[patheffects.withStroke(linewidth=2.0, foreground=stroke)],
+ zorder=5,
+ )
+ return omitted
+
+
+def _draw_continuous(
+ ax: Any,
+ fig: Any,
+ xx: np.ndarray,
+ yy: np.ndarray,
+ vnum: np.ndarray,
+ ss: np.ndarray,
+ *,
+ limits: tuple[float, float],
+ cmap_name: str | None,
+ vcenter: float | None,
+ scale: str,
+ missing_color: str,
+ default_color: str,
+ edgecolor: str,
+ edgewidth: float = DEFAULT_POINT_EDGEWIDTH,
+ alpha: float,
+ label: str,
+ is_uniform: bool,
+ sort_values: bool,
+ rng: np.random.Generator | None,
+ add_colorbar: bool,
+ rasterized: bool,
+ plt: Any,
+ mpl: Any,
+) -> Any:
+ finite = np.isfinite(vnum)
+ order_idx = np.arange(len(vnum))
+ if sort_values and finite.any():
+ order_idx = np.argsort(np.where(finite, vnum, -np.inf))
+ elif rng is not None:
+ order_idx = rng.permutation(len(vnum))
+ xx = xx[order_idx]
+ yy = yy[order_idx]
+ vnum = vnum[order_idx]
+ ss = ss[order_idx]
+ finite = finite[order_idx]
+ edges, lw = _scatter_edges(edgecolor, edgewidth)
+
+ if is_uniform or len(xx) == 0:
+ return ax.scatter(
+ xx,
+ yy,
+ c=[default_color] * len(xx),
+ s=ss if len(ss) else 10,
+ linewidths=lw,
+ edgecolors=edges,
+ alpha=alpha,
+ rasterized=rasterized,
+ )
+
+ vmin, vmax = limits
+ if vmax == vmin:
+ vmax = vmin + 1.0
+ if scale == "log":
+ if vmin <= 0:
+ raise ValueError("Log color scale requires positive values")
+ norm = mpl.colors.LogNorm(vmin=vmin, vmax=vmax)
+ elif scale == "symlog":
+ norm = mpl.colors.SymLogNorm(
+ linthresh=max(abs(vmax - vmin) * 0.001, 1e-12),
+ vmin=vmin,
+ vmax=vmax,
+ )
+ else:
+ norm = continuous_norm(
+ mpl,
+ vmin=vmin,
+ vmax=vmax,
+ vcenter=vcenter,
+ )
+ cmap = plt.get_cmap(cmap_name or "viridis")
+ face = np.empty((len(vnum), 4))
+ face[:] = mpl.colors.to_rgba(missing_color)
+ if finite.any():
+ face[finite] = cmap(norm(vnum[finite]))
+ collection = ax.scatter(
+ xx,
+ yy,
+ c=face,
+ s=ss,
+ linewidths=lw,
+ edgecolors=edges,
+ alpha=alpha,
+ rasterized=rasterized,
+ )
+ if add_colorbar:
+ sm = plt.cm.ScalarMappable(cmap=cmap, norm=norm)
+ sm.set_array([])
+ # Top colorbars keep the embedding panel square and avoid colliding
+ # with neighboring y-axis labels in multi-panel figures.
+ cb = fig.colorbar(
+ sm,
+ ax=ax,
+ location="top",
+ orientation="horizontal",
+ shrink=0.8,
+ fraction=0.06,
+ pad=0.02,
+ )
+ cb.set_label(label)
+ return collection
+
+
+def _soft_clip(values: np.ndarray, clip_fraction: float) -> np.ndarray:
+ if clip_fraction <= 0:
+ return values
+ if clip_fraction >= 0.5:
+ raise ValueError("clip_fraction must be in [0, 0.5)")
+ v = np.asarray(values, dtype=np.float64).copy()
+ finite = np.isfinite(v)
+ if not finite.any():
+ return v
+ lo = float(np.percentile(v[finite], 100 * clip_fraction))
+ hi = float(np.percentile(v[finite], 100 - 100 * clip_fraction))
+ v[finite & (v < lo)] = lo
+ v[finite & (v > hi)] = hi
+ return v
+
+
+def embedding(
+ store: Any,
+ *,
+ layout_key: str | Sequence[str],
+ color_by: str
+ | FeatureRef
+ | CellField
+ | Sequence[str | FeatureRef | CellField]
+ | None = None,
+ facet_by: str | None = None,
+ facet_order: Sequence[Any] | None = None,
+ cell_key: str = "I",
+ from_assay: str | None = None,
+ normalization: NormalizationSpec | None = None,
+ point_size: float | None = None,
+ point_sizes: np.ndarray | Sequence[float] | None = None,
+ point_size_range: tuple[float, float] = (1.0, 28.0),
+ point_edgecolor: str | None = None,
+ point_edgewidth: float | None = None,
+ point_alpha: float = 1.0,
+ sort_values: bool = False,
+ color_scale: ColorScale | None = None,
+ categorical_scale: CategoricalScale | None = None,
+ default_color: str = "steelblue",
+ missing_color: str | None = None,
+ clip_fraction: float = 0.0,
+ subset_by: str | None = None,
+ groups: Sequence[Any] | None = None,
+ n_columns: int | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ legend_loc: LegendLoc = "auto",
+ max_on_data_labels: int = 40,
+ show_legend: bool = True,
+ show_titles: bool = True,
+ frame: FrameStyle = "minimal",
+ density_overlay: DensityOverlay | None = None,
+ highlight: Highlight | None = None,
+ seed: int | None = None,
+ rasterize_threshold: int = DEFAULT_RASTERIZE_THRESHOLD,
+ show: bool = True,
+) -> PlotResult:
+ """Scatter cells on one or more 2D layouts (UMAP, t-SNE, and similar).
+
+ ``color_by`` accepts a cell-metadata column, a gene name, or a list of
+ either. With several genes, Scarf draws one panel per gene. With
+ ``facet_by``, each gene becomes a row and each facet level a column.
+ Color limits for a gene are shared across facets so panels stay comparable.
+ Multiple ``layout_key`` values produce the cartesian product of layouts and
+ colors in one figure.
+
+ Common choices:
+
+ - ``subset_by``: keep cells marked ``True`` in a boolean metadata column.
+ - ``groups``: keep only these categories. Applies to ``facet_by`` when set,
+ otherwise to the first categorical ``color_by`` column. Also sets legend
+ order when ``categorical_scale.order`` is omitted.
+ - ``legend_loc``: ``"auto"`` puts a side legend when there are few
+ categories, labels on the clusters for medium category counts, and a
+ wrapped side legend for larger sets. Use ``"right"``, ``"on_data"``, or
+ ``"none"`` to force a placement.
+ - ``frame``: ``"minimal"`` (default) keeps an L-shaped axes edge without
+ UMAP tick labels. ``"none"`` removes the box. ``"axes"`` keeps UMAP1 /
+ UMAP2 labels on the outer panels.
+ - ``point_size``: leave as ``None`` to size markers from the cell count.
+ Pass a number (matplotlib marker area) or ``point_sizes`` for per-cell
+ sizes such as mapping confidence.
+ - ``sort_values=True``: for continuous colors, draw high values last so
+ expressing cells sit on top of the cloud.
+ - ``theme``: ``"notebook"`` or ``"paper"`` for white figures; ``"dark"``
+ for dark notebook themes.
+
+ Returns a :class:`PlotResult`. Access ``.figure`` in notebooks, or call
+ ``.save(...)`` / ``.close()`` when you own the figure.
+ """
+ layout_keys = _coerce_layout_items(layout_key)
+ if len(layout_keys) > 1:
+ return _embedding_multiple_layouts(
+ store,
+ layout_keys=layout_keys,
+ color_by=color_by,
+ facet_by=facet_by,
+ facet_order=facet_order,
+ cell_key=cell_key,
+ from_assay=from_assay,
+ normalization=normalization,
+ groups=groups,
+ subset_by=subset_by,
+ n_columns=n_columns,
+ target=target,
+ figsize=figsize,
+ theme=theme,
+ show=show,
+ child_kwargs={
+ "point_size": point_size,
+ "point_sizes": point_sizes,
+ "point_size_range": point_size_range,
+ "point_edgecolor": point_edgecolor,
+ "point_edgewidth": point_edgewidth,
+ "point_alpha": point_alpha,
+ "sort_values": sort_values,
+ "color_scale": color_scale,
+ "categorical_scale": categorical_scale,
+ "default_color": default_color,
+ "missing_color": missing_color,
+ "clip_fraction": clip_fraction,
+ "legend_loc": legend_loc,
+ "max_on_data_labels": max_on_data_labels,
+ "show_legend": show_legend,
+ "frame": frame,
+ "density_overlay": density_overlay,
+ "highlight": highlight,
+ "seed": seed,
+ "rasterize_threshold": rasterize_threshold,
+ },
+ )
+ layout_key = layout_keys[0]
+
+ plt, mpl = require_matplotlib()
+ if rasterize_threshold < 0:
+ raise ValueError("rasterize_threshold must be >= 0")
+ if point_size_range[0] <= 0 or point_size_range[1] < point_size_range[0]:
+ raise ValueError("point_size_range must satisfy 0 < minimum <= maximum")
+ if point_edgewidth is not None and point_edgewidth < 0:
+ raise ValueError("point_edgewidth must be non-negative")
+ if not 0 <= point_alpha <= 1:
+ raise ValueError("point_alpha must be between 0 and 1")
+ if max_on_data_labels < 1:
+ raise ValueError("max_on_data_labels must be positive")
+ normalization = normalization or NormalizationSpec()
+ color_scale_was_explicit = color_scale is not None
+ categorical_scale_was_explicit = categorical_scale is not None
+ color_scale = color_scale or ColorScale(cmap="viridis", scope="feature")
+ resolved_missing_color = missing_color or "#bdbdbd"
+
+ x = np.asarray(store.cells.fetch(f"{layout_key}1", key=cell_key), dtype=np.float64)
+ y = np.asarray(store.cells.fetch(f"{layout_key}2", key=cell_key), dtype=np.float64)
+ n = len(x)
+ if n == 0:
+ raise ValueError(f"No cells selected by cell_key {cell_key!r}")
+ finite_coordinates = np.isfinite(x) & np.isfinite(y)
+ if not finite_coordinates.any():
+ raise ValueError(f"Layout {layout_key!r} has no finite coordinates")
+ if point_sizes is not None and len(point_sizes) != n:
+ raise ValueError("point_sizes length must match number of selected cells")
+ if point_size is not None and (not np.isfinite(point_size) or point_size <= 0):
+ raise ValueError("point_size must be finite and positive")
+ auto_point_size = point_size is None and point_sizes is None
+ resolved_point_size = (
+ float(point_size)
+ if point_size is not None
+ else default_point_size(
+ n,
+ size_min=point_size_range[0],
+ size_max=point_size_range[1],
+ )
+ )
+ size_arr = (
+ np.asarray(point_sizes, dtype=np.float64)
+ if point_sizes is not None
+ else np.full(n, resolved_point_size, dtype=np.float64)
+ )
+ if not np.all(np.isfinite(size_arr)) or np.any(size_arr <= 0):
+ raise ValueError("point_sizes must contain finite positive values")
+
+ color_items = _coerce_color_items(color_by)
+ stored_displays: list[dict[str, Any] | None] = []
+ for item in color_items:
+ column = (
+ item
+ if isinstance(item, str)
+ else item.key
+ if isinstance(item, CellField)
+ else None
+ )
+ stored_displays.append(
+ None if column is None else stored_display_metadata(store, column)
+ )
+ color_cache = _prefetch_colors(
+ store,
+ color_items,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ n_cells=n,
+ normalization=normalization,
+ )
+ classified_cache: list[tuple[np.ndarray, str, bool, bool]] = []
+ for index, entry in enumerate(color_cache):
+ values, label, is_categorical, is_uniform = entry
+ item = color_items[index]
+ display = stored_displays[index]
+ cell_kind = item.kind if isinstance(item, CellField) else "auto"
+ if display is not None and cell_kind == "auto":
+ is_categorical = display["kind"] == "categorical"
+ classified_cache.append((values, label, is_categorical, is_uniform))
+ color_cache = classified_cache
+ stored_color_scales: dict[int, ColorScale] = {}
+ stored_categorical_scales: dict[int, CategoricalScale] = {}
+ for index, display in enumerate(stored_displays):
+ if display is None:
+ continue
+ if display["kind"] == "continuous" and not color_scale_was_explicit:
+ stored_color_scales[index] = ColorScale(
+ cmap=str(display["colormap"]),
+ vmin=(
+ float(display["minimum"])
+ if display["minimum"] is not None and clip_fraction == 0
+ else None
+ ),
+ vmax=(
+ float(display["maximum"])
+ if display["maximum"] is not None and clip_fraction == 0
+ else None
+ ),
+ scope="feature",
+ scale=str(display["scale"]), # type: ignore[arg-type]
+ )
+ elif display["kind"] == "categorical" and not categorical_scale_was_explicit:
+ categories = display["categories"]
+ stored_categorical_scales[index] = CategoricalScale(
+ order=tuple(category["value"] for category in categories),
+ palette={
+ category["value"]: str(category["color"]) for category in categories
+ },
+ labels={
+ category["value"]: str(category["label"]) for category in categories
+ },
+ missing_color=(
+ missing_color
+ if missing_color is not None
+ else str(display.get("missing_color", "#bdbdbd"))
+ ),
+ missing_label=str(display.get("missing_label", "NA")),
+ )
+ if clip_fraction > 0:
+ clipped: list[tuple[np.ndarray, str, bool, bool]] = []
+ for vals, label, is_cat, is_uniform in color_cache:
+ if is_cat or is_uniform:
+ clipped.append((vals, label, is_cat, is_uniform))
+ else:
+ clipped.append(
+ (_soft_clip(vals, clip_fraction), label, is_cat, is_uniform)
+ )
+ color_cache = clipped
+
+ subset_vals = (
+ np.asarray(store.cells.fetch(subset_by, key=cell_key))
+ if subset_by is not None
+ else None
+ )
+ highlight_mask = _resolve_highlight_mask(
+ store,
+ highlight,
+ cell_key=cell_key,
+ n_cells=n,
+ )
+ density_filter = _density_selection_mask(
+ store,
+ density_overlay,
+ cell_key=cell_key,
+ n_cells=n,
+ )
+ facet_values: np.ndarray | None = None
+ groups_category: np.ndarray | None = None
+ groups_color_index: int | None = None
+ if facet_by is not None:
+ facet_values = np.asarray(store.cells.fetch(facet_by, key=cell_key))
+ if groups is not None:
+ groups_category = facet_values
+ elif groups is not None:
+ cat_columns = [
+ (index, vals)
+ for index, (vals, _, is_cat, is_uniform) in enumerate(color_cache)
+ if is_cat and not is_uniform
+ ]
+ if not cat_columns:
+ raise ValueError(
+ "groups requires a categorical color_by column, or facet_by"
+ )
+ groups_color_index, group_values = cat_columns[0]
+ groups_category = np.asarray(group_values)
+
+ selection_mask, group_order = resolve_cell_selection(
+ n,
+ subset=subset_vals,
+ subset_name=subset_by,
+ category_values=groups_category,
+ groups=groups,
+ )
+ base_mask = finite_coordinates & selection_mask
+
+ if facet_by is not None:
+ assert facet_values is not None
+ if groups is not None and group_order is not None:
+ facets = list(group_order)
+ elif facet_order is not None:
+ facets = list(facet_order)
+ else:
+ facets = sort_categories(list(pd.unique(facet_values[base_mask])))
+ else:
+ facets = [None]
+
+ labels = [label for _, label, _, _ in color_cache]
+ panel_keys = _embedding_panel_keys(labels, facets)
+
+ n_colors = len(color_cache)
+ n_facets = len(facets)
+ if n_columns is None:
+ n_columns = n_facets if facet_by is not None else min(n_colors, 4)
+ n_columns = max(1, min(n_columns, len(panel_keys)))
+
+ selected_x = x[base_mask]
+ selected_y = y[base_mask]
+ xpad = 0.05 * (float(selected_x.max() - selected_x.min()) or 1.0)
+ ypad = 0.05 * (float(selected_y.max() - selected_y.min()) or 1.0)
+ xlim = (float(selected_x.min() - xpad), float(selected_x.max() + xpad))
+ ylim = (float(selected_y.min() - ypad), float(selected_y.max() + ypad))
+ xlim, ylim = square_axis_limits(xlim, ylim)
+ edgecolor = point_edgecolor or scatter_edgecolor(theme)
+
+ label_counts = pd.Series(labels).value_counts()
+
+ def display_key(index: int, label: str) -> str:
+ return label if label_counts[label] == 1 else f"{index}:{label}"
+
+ limit_map: dict[int, tuple[float, float]] = {}
+ categorical_maps: dict[int, tuple[list[Any], dict[Any, str]]] = {}
+ resolved_categorical_scales: dict[int, CategoricalScale | None] = {}
+ resolved_color_scales: dict[int, ColorScale] = {}
+ shared_limits: tuple[float, float] | None = None
+ if color_scale.scope == "shared":
+ shared_values = [
+ np.asarray(values, dtype=np.float64)[base_mask]
+ for values, _, is_categorical, is_uniform in color_cache
+ if not is_categorical and not is_uniform
+ ]
+ if shared_values:
+ shared_limits = _continuous_limits(
+ np.concatenate(shared_values),
+ color_scale,
+ )
+
+ for color_index, (vals, _, is_cat, is_uniform) in enumerate(color_cache):
+ if is_uniform:
+ continue
+ vals_sel = np.asarray(vals)[base_mask]
+ active_color_scale = color_scale
+ if is_cat:
+ active_categorical_scale = (
+ categorical_scale
+ if categorical_scale_was_explicit
+ else stored_categorical_scales.get(color_index)
+ )
+ observed = list(pd.Series(vals_sel).dropna().unique())
+ if (
+ categorical_scale_was_explicit
+ and active_categorical_scale
+ and active_categorical_scale.order is not None
+ ):
+ order = list(active_categorical_scale.order)
+ if (
+ groups is not None
+ and group_order is not None
+ and facet_by is None
+ and groups_category is not None
+ and color_index == groups_color_index
+ ):
+ selected_groups = set(group_order)
+ order = [value for value in order if value in selected_groups]
+ unlisted = [value for value in observed if value not in order]
+ if unlisted:
+ raise ValueError(
+ "categorical_scale.order is missing observed values: "
+ + ", ".join(map(str, unlisted[:10]))
+ )
+ elif (
+ groups is not None
+ and group_order is not None
+ and facet_by is None
+ and groups_category is not None
+ and color_index == groups_color_index
+ ):
+ order = [g for g in group_order if g in set(observed)]
+ elif (
+ active_categorical_scale and active_categorical_scale.order is not None
+ ):
+ order = list(active_categorical_scale.order)
+ else:
+ order = sort_categories(observed)
+ palette = categorical_color_map(
+ order,
+ palette=(
+ active_categorical_scale.palette
+ if active_categorical_scale
+ else None
+ ),
+ palette_name=(
+ active_categorical_scale.palette_name
+ if active_categorical_scale
+ else "default"
+ ),
+ missing_label=None,
+ )
+ categorical_maps[color_index] = (order, palette)
+ resolved_categorical_scales[color_index] = active_categorical_scale
+ else:
+ active_color_scale = stored_color_scales.get(
+ color_index,
+ color_scale,
+ )
+ resolved_color_scales[color_index] = active_color_scale
+ if not is_cat and active_color_scale.scope != "panel":
+ limit_map[color_index] = (
+ shared_limits
+ if shared_limits is not None and color_scale_was_explicit
+ else _continuous_limits(
+ vals_sel,
+ active_color_scale,
+ )
+ )
+
+ legend_locs: dict[int, LegendLoc] = {
+ color_index: (
+ resolve_legend_loc(len(order), legend_loc) if show_legend else "none"
+ )
+ for color_index, (order, _) in categorical_maps.items()
+ }
+ needs_side_legend = any(loc == "right" for loc in legend_locs.values())
+ if figsize is None and target is None:
+ nrows = int(np.ceil(len(panel_keys) / n_columns))
+ needs_top_cbar = any(
+ (not is_cat) and (not is_uniform)
+ for _, _, is_cat, is_uniform in color_cache
+ )
+ panel = DEFAULT_PANEL_INCHES
+ side_columns = max(
+ (
+ legend_side_columns(len(order))
+ for color_index, (order, _) in categorical_maps.items()
+ if legend_locs[color_index] == "right"
+ ),
+ default=0,
+ )
+ width = panel * n_columns + (
+ 0.95 * side_columns + 0.4 if needs_side_legend else 0.25
+ )
+ height = panel * nrows + (0.55 if needs_top_cbar else 0.2)
+ figsize = (width, height)
+
+ legends: list[LegendSpec] = []
+ scales_out: list[Any] = [
+ resolved_color_scales[index] for index in sorted(resolved_color_scales)
+ ]
+ if not scales_out:
+ scales_out.append(color_scale)
+ for color_index, (order, palette) in categorical_maps.items():
+ label = labels[color_index]
+ active_categorical_scale = resolved_categorical_scales.get(color_index)
+ scales_out.append(
+ CategoricalScale(
+ order=tuple(order),
+ palette=dict(palette),
+ labels=(
+ {
+ value: active_categorical_scale.labels.get(
+ value,
+ str(value),
+ )
+ for value in order
+ }
+ if active_categorical_scale is not None
+ and active_categorical_scale.labels is not None
+ else None
+ ),
+ missing_color=(
+ active_categorical_scale.missing_color
+ if active_categorical_scale is not None
+ else resolved_missing_color
+ ),
+ missing_label=(
+ active_categorical_scale.missing_label
+ if active_categorical_scale is not None
+ else "NA"
+ ),
+ palette_name=(
+ active_categorical_scale.palette_name
+ if active_categorical_scale is not None
+ else "default"
+ ),
+ )
+ )
+ legends.append(
+ LegendSpec(
+ kind="categorical",
+ label=label,
+ extras={"legend_loc": legend_locs[color_index]},
+ )
+ )
+ for color_index, (_, label, is_categorical, is_uniform) in enumerate(color_cache):
+ if is_categorical or is_uniform:
+ continue
+ limits = limit_map.get(color_index)
+ legends.append(
+ LegendSpec(
+ kind="colorbar",
+ label=label,
+ extras=(
+ {"vmin": limits[0], "vmax": limits[1]}
+ if limits is not None
+ else {"scope": "panel"}
+ ),
+ )
+ )
+
+ rng = np.random.default_rng(seed) if seed is not None else None
+ panel_limit_map: dict[str, tuple[float, float]] = {}
+ panel_point_sizes: dict[str, float] = {}
+ panel_edgewidths: dict[str, float] = {}
+ omitted_labels: dict[str, list[Any]] = {}
+ omitted_legend_entries: dict[str, list[str]] = {}
+ auto_size_artists: list[tuple[Any, int, str, Any, Any | None]] = []
+
+ with theme_context(theme):
+ fig, axes, owns = normalize_axes_target(
+ target, panel_keys=panel_keys, figsize=figsize, n_columns=n_columns
+ )
+ panel_i = 0
+ for color_index, (vals, label, is_cat, is_uniform) in enumerate(color_cache):
+ for fac_i, fac in enumerate(facets):
+ panel_key = panel_keys[panel_i]
+ ax = axes[panel_key]
+ if facet_values is None:
+ mask = base_mask.copy()
+ elif pd.isna(fac):
+ mask = base_mask & pd.isna(facet_values)
+ else:
+ mask = base_mask & (facet_values == fac)
+
+ xx = x[mask]
+ yy = y[mask]
+ vv = np.asarray(vals)[mask]
+ if auto_point_size:
+ panel_size = default_point_size(
+ int(mask.sum()),
+ panel_area=_panel_area_inches(ax),
+ size_min=point_size_range[0],
+ size_max=point_size_range[1],
+ )
+ ss = np.full(int(mask.sum()), panel_size, dtype=np.float64)
+ else:
+ ss = size_arr[mask]
+ panel_size = (
+ float(np.nanmedian(ss)) if len(ss) else resolved_point_size
+ )
+ active_edgewidth = (
+ float(point_edgewidth)
+ if point_edgewidth is not None
+ else default_point_edgewidth(
+ int(mask.sum()),
+ point_size=panel_size,
+ )
+ )
+ panel_point_sizes[str(panel_key)] = panel_size
+ panel_edgewidths[str(panel_key)] = active_edgewidth
+ rasterized = len(xx) >= rasterize_threshold
+ base_alpha = (
+ highlight.dim_alpha if highlight is not None else point_alpha
+ )
+
+ if mask.sum() == 0:
+ ax.set_axis_off()
+ ax.set_title(
+ f"{label} | {facet_by}={fac} (empty)"
+ if fac is not None
+ else f"{label} (empty)"
+ )
+ panel_i += 1
+ continue
+
+ if is_cat:
+ order, palette = categorical_maps[color_index]
+ active_categorical_scale = resolved_categorical_scales.get(
+ color_index
+ )
+ category_missing_color = (
+ active_categorical_scale.missing_color
+ if active_categorical_scale is not None
+ else resolved_missing_color
+ )
+ base_artist = _draw_categorical(
+ ax,
+ xx,
+ yy,
+ vv,
+ ss,
+ order=order,
+ palette=palette,
+ missing_color=category_missing_color,
+ edgecolor=edgecolor,
+ edgewidth=active_edgewidth,
+ alpha=base_alpha,
+ rasterized=rasterized,
+ )
+ panel_legend = legend_locs[color_index]
+ if panel_legend == "on_data":
+ omitted = _add_on_data_labels(
+ ax,
+ xx,
+ yy,
+ vv,
+ order=order,
+ labels=(
+ active_categorical_scale.labels
+ if active_categorical_scale is not None
+ else None
+ ),
+ theme=theme,
+ max_labels=max_on_data_labels,
+ )
+ if omitted:
+ omitted_labels[str(panel_key)] = omitted
+ elif panel_legend == "right" and fac_i == n_facets - 1:
+ omitted = _add_categorical_legend(
+ ax,
+ fig,
+ mpl,
+ order=order,
+ palette=palette,
+ labels=(
+ active_categorical_scale.labels
+ if active_categorical_scale is not None
+ else None
+ ),
+ label=label,
+ missing=bool(pd.isna(np.asarray(vals)[base_mask]).any()),
+ missing_color=category_missing_color,
+ missing_label=(
+ active_categorical_scale.missing_label
+ if active_categorical_scale is not None
+ else "NA"
+ ),
+ edgecolor=edgecolor,
+ figure_level=owns,
+ values=np.asarray(vals)[base_mask],
+ )
+ if omitted:
+ omitted_legend_entries[str(panel_key)] = [
+ str(value) for value in omitted
+ ]
+ else:
+ active_color_scale = resolved_color_scales.get(
+ color_index,
+ color_scale,
+ )
+ vnum = pd.to_numeric(pd.Series(vv), errors="coerce").to_numpy(
+ dtype=np.float64
+ )
+ if is_uniform:
+ limits = (0.0, 1.0)
+ elif active_color_scale.scope == "panel":
+ limits = _continuous_limits(
+ vnum,
+ active_color_scale,
+ )
+ panel_limit_map[str(panel_key)] = limits
+ else:
+ limits = limit_map[color_index]
+ add_cb = (
+ show_legend
+ and (not is_uniform)
+ and (
+ active_color_scale.scope == "panel"
+ or facet_by is None
+ or fac_i == n_facets - 1
+ )
+ )
+ base_artist = _draw_continuous(
+ ax,
+ fig,
+ xx,
+ yy,
+ vnum,
+ ss,
+ limits=limits,
+ cmap_name=active_color_scale.cmap,
+ vcenter=active_color_scale.vcenter,
+ scale=active_color_scale.scale,
+ missing_color=active_color_scale.missing_color,
+ default_color=default_color,
+ edgecolor=edgecolor,
+ edgewidth=active_edgewidth,
+ alpha=base_alpha,
+ label=label,
+ is_uniform=is_uniform,
+ sort_values=sort_values,
+ rng=rng,
+ add_colorbar=add_cb,
+ rasterized=rasterized,
+ plt=plt,
+ mpl=mpl,
+ )
+
+ if density_overlay is not None:
+ density_mask = mask.copy()
+ if density_filter is not None:
+ density_mask &= density_filter
+ contour_values = None
+ if density_overlay.statistic == "mean":
+ if is_cat or is_uniform:
+ raise ValueError(
+ "Mean contours require a continuous color_by panel"
+ )
+ contour_values = pd.to_numeric(
+ pd.Series(np.asarray(vals)[density_mask]),
+ errors="coerce",
+ ).to_numpy(dtype=np.float64)
+ _draw_density_overlay(
+ ax,
+ x[density_mask],
+ y[density_mask],
+ overlay=density_overlay,
+ values=contour_values,
+ xlim=xlim,
+ ylim=ylim,
+ theme=theme,
+ )
+ highlight_artist = None
+ if highlight is not None and highlight_mask is not None:
+ panel_highlight = highlight_mask[mask]
+ highlight_artist = _draw_highlight(
+ ax,
+ xx[panel_highlight],
+ yy[panel_highlight],
+ ss[panel_highlight],
+ highlight=highlight,
+ edgecolor=edgecolor,
+ rasterized=rasterized,
+ )
+ if auto_point_size:
+ auto_size_artists.append(
+ (
+ ax,
+ int(mask.sum()),
+ str(panel_key),
+ base_artist,
+ highlight_artist,
+ )
+ )
+
+ if not show_titles:
+ title = None
+ elif fac is None:
+ # A labelled colorbar already names the panel.
+ title = None if (not is_cat and add_cb) else label
+ else:
+ title = (
+ f"{label} | {facet_by}={fac}" if label else f"{facet_by}={fac}"
+ )
+ col = panel_i % n_columns
+ row = panel_i // n_columns
+ nrows = int(np.ceil(len(panel_keys) / n_columns))
+ finish_embedding_axes(
+ ax,
+ xlim=xlim,
+ ylim=ylim,
+ xlabel=f"{layout_key}1" if row == nrows - 1 else "",
+ ylabel=f"{layout_key}2" if col == 0 else "",
+ title=title or None,
+ frame=frame,
+ )
+ panel_i += 1
+ apply_figure_chrome(fig, theme)
+ if auto_size_artists:
+ fig.canvas.draw()
+ fig.canvas.draw()
+ for ax, n_points, key, base_artist, highlight_artist in auto_size_artists:
+ panel_size = default_point_size(
+ n_points,
+ panel_area=_panel_area_inches(ax),
+ size_min=point_size_range[0],
+ size_max=point_size_range[1],
+ )
+ active_edgewidth = (
+ float(point_edgewidth)
+ if point_edgewidth is not None
+ else default_point_edgewidth(
+ n_points,
+ point_size=panel_size,
+ )
+ )
+ base_artist.set_sizes(
+ np.full(
+ len(base_artist.get_offsets()), panel_size, dtype=np.float64
+ )
+ )
+ edges, linewidth = _scatter_edges(edgecolor, active_edgewidth)
+ base_artist.set_edgecolors(edges)
+ base_artist.set_linewidths(linewidth)
+ if highlight_artist is not None and highlight is not None:
+ highlight_artist.set_sizes(
+ np.full(
+ len(highlight_artist.get_offsets()),
+ panel_size * highlight.size_multiplier,
+ dtype=np.float64,
+ )
+ )
+ panel_point_sizes[key] = panel_size
+ panel_edgewidths[key] = active_edgewidth
+
+ if color_scale.scope == "panel":
+ color_limits = panel_limit_map
+ else:
+ color_limits = {
+ display_key(index, labels[index]): limits
+ for index, limits in limit_map.items()
+ }
+ feature_assays: set[str] = set()
+ for item in color_items:
+ if not (
+ isinstance(item, FeatureRef)
+ or (isinstance(item, str) and item not in store.cells.columns)
+ ):
+ continue
+ assay_name = (
+ item.assay
+ if isinstance(item, FeatureRef) and item.assay is not None
+ else from_assay or getattr(store, "_defaultAssay", None)
+ )
+ if assay_name is not None:
+ feature_assays.add(assay_name)
+
+ result = PlotResult(
+ figure=fig,
+ axes=axes,
+ tables={},
+ legends=tuple(legends),
+ scales=tuple(scales_out),
+ provenance=PlotProvenance(
+ assay=next(iter(feature_assays)) if len(feature_assays) == 1 else None,
+ cell_key=cell_key,
+ n_cells=int(base_mask.sum()),
+ renderer="matplotlib",
+ notes=("embedding", "materialized", f"layout={layout_key}"),
+ extras={
+ "sort_values": sort_values,
+ "color_scale_scope": color_scale.scope,
+ "color_limits": color_limits,
+ "facet_by": facet_by,
+ "n_colors": n_colors,
+ "n_facets": n_facets,
+ "panel_keys": [str(k) for k in panel_keys],
+ "clip_fraction": clip_fraction,
+ "subset_by": subset_by,
+ "groups": None if groups is None else list(groups),
+ "input_n_cells": n,
+ "invalid_coordinate_cells": int((~finite_coordinates).sum()),
+ "rasterize_threshold": rasterize_threshold,
+ "point_size_by_panel": panel_point_sizes,
+ "point_edgewidth_by_panel": panel_edgewidths,
+ "point_size_range": list(point_size_range),
+ "show_titles": show_titles,
+ "omitted_labels": omitted_labels,
+ "omitted_legend_entries": omitted_legend_entries,
+ "density_overlay": (
+ None
+ if density_overlay is None
+ else {
+ "kind": density_overlay.kind,
+ "statistic": density_overlay.statistic,
+ "pixels": density_overlay.pixels,
+ "sigma": density_overlay.sigma,
+ "min_support": density_overlay.min_support,
+ "levels": density_overlay.levels,
+ "max_hotspots": density_overlay.max_hotspots,
+ "group_by": density_overlay.group_by,
+ "groups": density_overlay.groups,
+ "halo_width": density_overlay.halo_width,
+ }
+ ),
+ "highlight": (
+ None
+ if highlight is None
+ else {
+ "by": highlight.by,
+ "groups": highlight.groups,
+ "indices": highlight.indices,
+ "n_highlighted": int(
+ (highlight_mask & base_mask).sum()
+ if highlight_mask is not None
+ else 0
+ ),
+ }
+ ),
+ "normalization": {
+ "source": normalization.source,
+ "transform": normalization.transform,
+ },
+ "assays": sorted(feature_assays),
+ },
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ result.show()
+ return result
diff --git a/scarf/plotting/embedding_raster.py b/scarf/plotting/embedding_raster.py
new file mode 100644
index 00000000..a89757e3
--- /dev/null
+++ b/scarf/plotting/embedding_raster.py
@@ -0,0 +1,236 @@
+"""Blockwise raster embedding for large cell counts."""
+
+from dataclasses import replace
+from typing import Any, Hashable
+
+import numpy as np
+
+from ._contracts import CellField, ColorScale, PlotProvenance
+from ._deps import require_matplotlib
+from ._figure import LegendSpec, PlotResult, normalize_axes_target
+from ._raster import draw_raster_canvas, raster_from_metadata
+from ._style import apply_figure_chrome, square_axis_limits, theme_context
+
+
+def _is_categorical_column(
+ cells: Any,
+ *,
+ key: str,
+ cell_key: str,
+ block_rows: int | None,
+ kind: str,
+ max_categories: int = 100,
+) -> bool:
+ if kind == "categorical":
+ return True
+ if kind == "continuous":
+ return False
+ dtype = cells.get_dtype(key)
+ dtype_kind = getattr(dtype, "kind", None)
+ if dtype_kind in ("b", "O", "S", "U", "T"):
+ return True
+ if dtype_kind not in ("i", "u"):
+ return False
+
+ categories: set[Any] = set()
+ for block in cells.iter_row_blocks(
+ cell_key=cell_key,
+ columns=[key],
+ block_rows=block_rows,
+ ):
+ categories.update(np.unique(block.values[key]).tolist())
+ if len(categories) > max_categories:
+ return False
+ return True
+
+
+def embedding_raster(
+ store: Any,
+ *,
+ layout_key: str,
+ color_by: str | CellField | None = None,
+ cell_key: str = "I",
+ pixels: int = 400,
+ block_rows: int | None = None,
+ color_scale: ColorScale | None = None,
+ missing_color: str = "white",
+ subset_by: str | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ seed: int = 0,
+ show: bool = True,
+) -> PlotResult:
+ """Draw a layout as a pixel image for large cell counts.
+
+ Prefer this over :func:`embedding` when you only need a continuous
+ cell-metadata color (for example ``RNA_nCounts``) and the dataset is large
+ enough that a full scatter is slow or memory-heavy. Values are read in row
+ blocks, so full columns are not loaded at once.
+
+ ``color_by`` must be continuous metadata, not a gene. Pass
+ ``CellField(key, kind="continuous")`` when an integer column should be
+ treated as continuous. Categorical colors are not supported here.
+ Empty pixels use a white background by default (override with
+ ``missing_color``). Pass ``subset_by`` to keep cells marked ``True`` in a
+ boolean cell-metadata column.
+ """
+ require_matplotlib()
+ color_scale = color_scale or ColorScale(cmap="viridis", quantiles=(0.01, 0.99))
+ if color_scale.scale != "linear":
+ raise NotImplementedError(
+ "embedding_raster currently supports only linear color scales"
+ )
+ x_key = f"{layout_key}1"
+ y_key = f"{layout_key}2"
+ for key in (x_key, y_key):
+ if key not in store.cells.columns:
+ raise KeyError(f"Layout column {key!r} not found in cell metadata")
+
+ color_key: str | None
+ color_label: str | None
+ if isinstance(color_by, CellField):
+ color_key = color_by.key
+ color_label = color_by.label or color_by.key
+ color_kind = color_by.kind
+ else:
+ color_key = color_by
+ color_label = color_by
+ color_kind = "auto"
+ if color_key is not None and color_key not in store.cells.columns:
+ raise KeyError(
+ f"color_by {color_key!r} must be a cell-metadata column for "
+ "embedding_raster (gene coloring uses embedding() for now)"
+ )
+ if color_key is not None and _is_categorical_column(
+ store.cells,
+ key=color_key,
+ cell_key=cell_key,
+ block_rows=block_rows,
+ kind=color_kind,
+ ):
+ raise NotImplementedError(
+ "embedding_raster supports continuous color values only; use "
+ "embedding() for categorical cell metadata"
+ )
+
+ if subset_by is not None and subset_by not in store.cells.columns:
+ raise KeyError(f"subset_by {subset_by!r} not found in cell metadata")
+
+ quantiles = color_scale.quantiles
+ canvas = raster_from_metadata(
+ store.cells,
+ x_key=x_key,
+ y_key=y_key,
+ color_key=color_key,
+ cell_key=cell_key,
+ subset_by=subset_by,
+ pixels=pixels,
+ block_rows=block_rows,
+ quantiles=quantiles,
+ seed=seed,
+ )
+ if color_scale.vmin is not None or color_scale.vmax is not None:
+ canvas = replace(
+ canvas,
+ vmin=(
+ float(color_scale.vmin) if color_scale.vmin is not None else canvas.vmin
+ ),
+ vmax=(
+ float(color_scale.vmax) if color_scale.vmax is not None else canvas.vmax
+ ),
+ )
+ if canvas.vmax <= canvas.vmin:
+ raise ValueError("Color limits must satisfy vmin < vmax")
+
+ panel_key: Hashable = color_label or layout_key
+ resolved_figsize = figsize
+ if resolved_figsize is None and target is None:
+ resolved_figsize = (5.2, 5.0)
+ fig, axes, owns = normalize_axes_target(
+ target,
+ panel_keys=[panel_key],
+ figsize=resolved_figsize,
+ )
+ ax = axes[panel_key]
+ with theme_context(theme):
+ # Square the data window first, then draw into that extent so the
+ # image fills the axes (no floating grey rectangle in empty margins).
+ xlim, ylim = square_axis_limits(
+ (canvas.extent[0], canvas.extent[1]),
+ (canvas.extent[2], canvas.extent[3]),
+ )
+ squared = replace(
+ canvas,
+ extent=(xlim[0], xlim[1], ylim[0], ylim[1]),
+ )
+ im = draw_raster_canvas(
+ ax,
+ squared,
+ cmap=color_scale.cmap or "viridis",
+ missing_color=missing_color,
+ vcenter=color_scale.vcenter,
+ )
+ ax.set_xlim(xlim)
+ ax.set_ylim(ylim)
+ ax.set_aspect("equal", adjustable="box")
+ ax.set_box_aspect(1)
+ ax.set_xticks([])
+ ax.set_yticks([])
+ ax.set_xlabel(f"{layout_key}1")
+ ax.set_ylabel(f"{layout_key}2")
+ cb = fig.colorbar(
+ im,
+ ax=ax,
+ location="top",
+ orientation="horizontal",
+ shrink=0.8,
+ fraction=0.06,
+ pad=0.08,
+ )
+ cb.set_label(color_label or "log1p cell count")
+ apply_figure_chrome(fig, theme)
+
+ result = PlotResult(
+ figure=fig,
+ axes=axes,
+ tables={},
+ legends=(
+ LegendSpec(
+ kind="colorbar",
+ label=color_label or "log1p cell count",
+ ),
+ ),
+ scales=(color_scale,),
+ provenance=PlotProvenance(
+ cell_key=cell_key,
+ n_cells=canvas.n_cells,
+ renderer="matplotlib-raster",
+ notes=(
+ "embedding_raster",
+ "two_pass",
+ f"layout={layout_key}",
+ *(
+ ("approximate_quantiles",)
+ if color_key is not None and quantiles is not None
+ else ()
+ ),
+ ),
+ extras={
+ "pixels": pixels,
+ "block_rows": block_rows,
+ "n_blocks": canvas.n_blocks,
+ "vmin": canvas.vmin,
+ "vmax": canvas.vmax,
+ "color_by": color_key,
+ "color_mode": "continuous" if color_key is not None else "density",
+ "subset_by": subset_by,
+ "missing_color": missing_color,
+ },
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ result.show()
+ return result
diff --git a/scarf/plotting/heatmaps.py b/scarf/plotting/heatmaps.py
new file mode 100644
index 00000000..163178ef
--- /dev/null
+++ b/scarf/plotting/heatmaps.py
@@ -0,0 +1,1196 @@
+"""Native heatmap and cluster-tree plotting."""
+
+from collections.abc import Hashable, Mapping, Sequence
+from typing import Any, cast
+
+import numpy as np
+import pandas as pd
+
+from ..assay import ATACassay
+from ..features.markers.table import load_marker_table
+from ..storage.artifacts import ArtifactRef, inspect_artifact
+from ..storage.types import as_zarr_array, as_zarr_group
+from ..matrix import ChunkedArray
+from ..utils.arrays import array_digest
+from ..utils.logging import logger
+from ._contracts import CategoricalScale, ColorScale, PlotProvenance
+from ._deps import require_matplotlib, require_seaborn
+from ._figure import LegendSpec, PlotResult, normalize_axes_target
+from ._heatmap_utils import (
+ annotation_colors,
+ draw_annotation_strips,
+ normalize_annotations,
+ order_heatmap,
+)
+from ._style import (
+ apply_figure_chrome,
+ categorical_color_map,
+ continuous_norm,
+ sort_categories,
+ theme_context,
+)
+
+
+def _marker_log_transform(assay: Any, value: bool | None) -> bool:
+ if value is None:
+ return not isinstance(assay, ATACassay)
+ if not isinstance(value, (bool, np.bool_)):
+ raise TypeError("log_transform must be a boolean or None")
+ return bool(value)
+
+
+def _place_clustermap_annotation_legend(
+ figure: Any,
+ axes: Mapping[Hashable, Any],
+ handles: Sequence[Any],
+ *,
+ has_column_dendrogram: bool,
+) -> None:
+ header = axes["column_dendrogram"].get_position()
+ longest_label = max(len(str(handle.get_label())) for handle in handles)
+ estimated_column_width = 0.45 + 0.09 * longest_label
+ available_width = max(0.5, (1.0 - header.x0) * figure.get_figwidth() - 0.15)
+ n_columns = max(
+ 1,
+ min(
+ 4,
+ len(handles),
+ int(available_width // estimated_column_width),
+ ),
+ )
+ if has_column_dendrogram:
+ anchor = (header.x0, 0.99)
+ else:
+ anchor = (header.x0, header.y1)
+ legend = figure.legend(
+ handles=handles,
+ title="Annotations",
+ frameon=False,
+ loc="upper left",
+ bbox_to_anchor=anchor,
+ bbox_transform=figure.transFigure,
+ ncols=n_columns,
+ )
+ if has_column_dendrogram:
+ figure.canvas.draw()
+ renderer = figure.canvas.get_renderer()
+ legend_bottom = (
+ legend.get_window_extent(renderer)
+ .transformed(figure.transFigure.inverted())
+ .y0
+ )
+ top = max(axis.get_position().y1 for axis in dict.fromkeys(axes.values()))
+ scale = min(1.0, (legend_bottom - 0.01) / top)
+ if scale < 1.0:
+ for axis in dict.fromkeys(axes.values()):
+ position = axis.get_position()
+ axis.set_position(
+ [
+ position.x0,
+ position.y0 * scale,
+ position.width,
+ position.height * scale,
+ ],
+ which="both",
+ )
+
+
+def _prepare_marker_heatmap(
+ store: Any,
+ *,
+ from_assay: str | None,
+ group_key: str | None,
+ cell_key: str | None,
+ topn: int,
+ log_transform: bool | None,
+ vmin: float,
+ vmax: float,
+) -> dict[str, Any]:
+ assay = store._get_assay(from_assay)
+ if group_key is None:
+ raise ValueError("ERROR: Please provide a value for `group_key`")
+ if cell_key is None:
+ cell_key = "I"
+
+ assay_group = as_zarr_group(store.zw[assay.name], name=assay.name)
+ if "markers" not in assay_group:
+ raise KeyError("ERROR: Please run `run_marker_search` first")
+ try:
+ marker_slot = store._resolve_marker_group(
+ assay.name,
+ cell_key,
+ group_key,
+ )
+ except KeyError:
+ raise KeyError(
+ "ERROR: Please run `run_marker_search` first with "
+ f"{group_key} as `group_key` and {cell_key} as `cell_key`"
+ ) from None
+
+ feature_indices: list[int] = []
+ marker_rows: list[dict[str, Any]] = []
+ feature_names = np.asarray(assay.feats.fetch_all("names"))
+ feature_ids = np.asarray(assay.feats.fetch_all("ids"))
+ for group_name in marker_slot.group_keys():
+ marker_group = as_zarr_group(marker_slot[group_name], name=group_name)
+ markers = load_marker_table(
+ marker_slot,
+ marker_group,
+ feature_names,
+ group_id=group_name,
+ feature_ids=feature_ids,
+ )
+ if markers.empty:
+ continue
+ unresolved = markers["feature_index"].isna()
+ if bool(unresolved.any()):
+ logger.warning(
+ f"Skipping {int(unresolved.sum())} unresolved legacy marker "
+ f"feature(s) for group '{group_name}'"
+ )
+ markers = markers.loc[~unresolved].copy()
+ if markers.empty:
+ continue
+ if "score" in markers.columns and markers["score"].notna().any():
+ ranked = markers.sort_values(
+ ["score", "feature_name"],
+ ascending=[False, True],
+ kind="mergesort",
+ ).head(topn)
+ else:
+ ranked = markers.head(topn)
+ selected = ranked["feature_index"].to_numpy(dtype=int)
+ feature_indices.extend(selected.tolist())
+ marker_rows.extend(
+ {
+ "group": group_name,
+ "rank": rank,
+ "feature_index": int(feature_index),
+ "score": float(score) if pd.notna(score) else np.nan,
+ }
+ for rank, (feature_index, score) in enumerate(
+ zip(
+ selected,
+ ranked["score"].to_numpy(dtype=float)
+ if "score" in ranked.columns
+ else np.full(len(selected), np.nan),
+ strict=True,
+ ),
+ start=1,
+ )
+ )
+
+ if not feature_indices:
+ raise ValueError("ERROR: Marker list is empty for all the groups")
+ feature_index = np.asarray(sorted(set(feature_indices)), dtype=int)
+ cell_index = np.asarray(assay.cells.active_index(cell_key))
+ resolved_log_transform = _marker_log_transform(assay, log_transform)
+ normalized = assay.normed(
+ cell_idx=cell_index,
+ feat_idx=feature_index,
+ log_transform=resolved_log_transform,
+ )
+ groups = np.asarray(assay.cells.fetch(group_key, cell_key))
+
+ group_sums: dict[Any, np.ndarray] = {}
+ group_counts: dict[Any, int] = {}
+ row_start = 0
+ for values in normalized.stream_blocks(
+ nthreads=store.nthreads,
+ msg="Aggregating marker values per group",
+ ):
+ block_groups = groups[row_start : row_start + values.shape[0]]
+ row_start += values.shape[0]
+ block_frame = pd.DataFrame(values)
+ block_frame["__group__"] = block_groups
+ grouped = block_frame.groupby("__group__")
+ block_sum = grouped.sum()
+ block_count = grouped.size()
+ for label in block_sum.index:
+ summed = block_sum.loc[label].to_numpy(dtype=np.float64)
+ if label not in group_sums:
+ group_sums[label] = summed
+ group_counts[label] = int(block_count.loc[label])
+ else:
+ group_sums[label] += summed
+ group_counts[label] += int(block_count.loc[label])
+
+ labels = sort_categories(list(group_sums))
+ group_means = pd.DataFrame(
+ np.vstack([group_sums[label] / group_counts[label] for label in labels]),
+ index=labels,
+ )
+ group_means = group_means.apply(
+ lambda values: (values - values.mean()) / values.std(),
+ axis=0,
+ )
+ feature_names = np.asarray(assay.feats.fetch_all("names"))
+ group_means.columns = feature_names[feature_index]
+ matrix = group_means.T
+ matrix[matrix < vmin] = vmin
+ matrix[matrix > vmax] = vmax
+
+ marker_table = pd.DataFrame(marker_rows)
+ marker_table["feature"] = feature_names[
+ marker_table["feature_index"].to_numpy(dtype=int)
+ ]
+ return {
+ "matrix": matrix,
+ "markers": marker_table,
+ "assay": assay.name,
+ "cell_key": cell_key,
+ "group_key": group_key,
+ "n_cells": len(cell_index),
+ }
+
+
+def marker_heatmap(
+ store: Any,
+ *,
+ from_assay: str | None = None,
+ group_key: str | None = None,
+ cell_key: str | None = None,
+ topn: int = 5,
+ log_transform: bool | None = None,
+ vmin: float = -1,
+ vmax: float = 2,
+ figsize: tuple[float, float] | None = None,
+ fontsize: float = 10,
+ width_factor: float = 0.03,
+ height_factor: float = 0.02,
+ cmap: Any = "magma_r",
+ color_scale: ColorScale | None = None,
+ row_order: Sequence[Any] | None = None,
+ column_order: Sequence[Any] | None = None,
+ cluster_rows: bool = True,
+ cluster_columns: bool = True,
+ cluster_method: str = "ward",
+ cluster_metric: str = "euclidean",
+ row_annotations: Mapping[
+ str,
+ Mapping[Any, Any] | Sequence[Any],
+ ]
+ | None = None,
+ column_annotations: Mapping[
+ str,
+ Mapping[Any, Any] | Sequence[Any],
+ ]
+ | None = None,
+ annotation_scales: Mapping[str, CategoricalScale] | None = None,
+ target: Any | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+ **heatmap_kwargs: Any,
+) -> PlotResult:
+ """Plot standardized expression of the top marker features."""
+ cluster_rows = bool(heatmap_kwargs.pop("row_cluster", cluster_rows))
+ cluster_columns = bool(heatmap_kwargs.pop("col_cluster", cluster_columns))
+ cluster_method = str(heatmap_kwargs.pop("method", cluster_method))
+ cluster_metric = str(heatmap_kwargs.pop("metric", cluster_metric))
+ if "row_linkage" in heatmap_kwargs or "col_linkage" in heatmap_kwargs:
+ raise ValueError(
+ "Pass clustering controls through cluster_rows, cluster_columns, "
+ "cluster_method, and cluster_metric"
+ )
+ if "z_score" in heatmap_kwargs or "standard_scale" in heatmap_kwargs:
+ raise ValueError("marker_heatmap already standardizes features before plotting")
+ resolved_color_scale = color_scale or ColorScale(
+ cmap=(cmap if isinstance(cmap, str) else getattr(cmap, "name", None)),
+ vmin=vmin,
+ vmax=vmax,
+ )
+ if resolved_color_scale.scale != "linear":
+ raise NotImplementedError("marker_heatmap supports only linear color scales")
+ resolved_vmin = (
+ float(resolved_color_scale.vmin)
+ if resolved_color_scale.vmin is not None
+ else float(vmin)
+ )
+ resolved_vmax = (
+ float(resolved_color_scale.vmax)
+ if resolved_color_scale.vmax is not None
+ else float(vmax)
+ )
+ resolved_cmap = resolved_color_scale.cmap or cmap
+ if resolved_vmax <= resolved_vmin:
+ raise ValueError("vmax must be greater than vmin")
+ prepared = _prepare_marker_heatmap(
+ store,
+ from_assay=from_assay,
+ group_key=group_key,
+ cell_key=cell_key,
+ topn=topn,
+ log_transform=log_transform,
+ vmin=resolved_vmin,
+ vmax=resolved_vmax,
+ )
+ matrix = cast(pd.DataFrame, prepared["matrix"])
+ row_annotation_values = normalize_annotations(
+ list(matrix.index),
+ row_annotations,
+ axis_name="row",
+ )
+ column_annotation_values = normalize_annotations(
+ list(matrix.columns),
+ column_annotations,
+ axis_name="column",
+ )
+ ordered_matrix, row_linkage, column_linkage = order_heatmap(
+ matrix,
+ row_order=row_order,
+ column_order=column_order,
+ cluster_rows=cluster_rows,
+ cluster_columns=cluster_columns,
+ method=cluster_method,
+ metric=cluster_metric,
+ )
+ row_colors, row_annotation_scales = annotation_colors(
+ row_annotation_values,
+ annotation_scales,
+ )
+ column_colors, column_annotation_scales = annotation_colors(
+ column_annotation_values,
+ annotation_scales,
+ )
+ resolved_annotation_scales = row_annotation_scales + column_annotation_scales
+ annotation_names = list(row_annotation_values.columns) + list(
+ column_annotation_values.columns
+ )
+ if figsize is None and target is None:
+ figsize = (
+ ordered_matrix.shape[1] * fontsize * width_factor,
+ fontsize * ordered_matrix.shape[0] * height_factor,
+ )
+
+ _, mpl = require_matplotlib()
+ if target is None:
+ sns = require_seaborn()
+ plot_index = (
+ list(matrix.index)
+ if row_linkage is not None
+ else list(ordered_matrix.index)
+ )
+ plot_columns = (
+ list(matrix.columns)
+ if column_linkage is not None
+ else list(ordered_matrix.columns)
+ )
+ plot_matrix = matrix.reindex(index=plot_index, columns=plot_columns)
+ clustermap_kwargs = {
+ "yticklabels": True,
+ "xticklabels": True,
+ "figsize": figsize,
+ "cmap": resolved_cmap,
+ "vmin": resolved_vmin,
+ "vmax": resolved_vmax,
+ "center": resolved_color_scale.vcenter,
+ "rasterized": True,
+ "row_cluster": row_linkage is not None,
+ "col_cluster": column_linkage is not None,
+ "row_linkage": row_linkage,
+ "col_linkage": column_linkage,
+ "row_colors": None if row_colors.empty else row_colors,
+ "col_colors": None if column_colors.empty else column_colors,
+ "cbar_pos": (0.02, 0.8, 0.03, 0.15) if show_legend else None,
+ }
+ clustermap_kwargs.update(heatmap_kwargs)
+ with theme_context(theme):
+ cluster_grid = sns.clustermap(plot_matrix, **clustermap_kwargs)
+ displayed_matrix = cluster_grid.data2d.copy()
+ if clustermap_kwargs["yticklabels"] is not False:
+ cluster_grid.ax_heatmap.set_yticklabels(
+ displayed_matrix.index,
+ fontsize=fontsize,
+ )
+ if clustermap_kwargs["xticklabels"] is not False:
+ cluster_grid.ax_heatmap.set_xticklabels(
+ displayed_matrix.columns,
+ fontsize=fontsize,
+ )
+ if show_legend and cluster_grid.cax is not None:
+ cluster_grid.cax.set_ylabel("standardized expression")
+ apply_figure_chrome(cluster_grid.fig, theme)
+ figure = cluster_grid.fig
+ axes: dict[Hashable, Any] = {
+ "heatmap": cluster_grid.ax_heatmap,
+ "row_dendrogram": cluster_grid.ax_row_dendrogram,
+ "column_dendrogram": cluster_grid.ax_col_dendrogram,
+ }
+ if cluster_grid.cax is not None:
+ axes["colorbar"] = cluster_grid.cax
+ owns_figure = True
+ else:
+ allowed_native_kwargs = {
+ "aspect",
+ "interpolation",
+ "rasterized",
+ }
+ unknown = set(heatmap_kwargs) - allowed_native_kwargs
+ if unknown:
+ raise TypeError(
+ "Unsupported heatmap keyword(s) with target: "
+ + ", ".join(sorted(unknown))
+ )
+ with theme_context(theme):
+ figure, axes, owns_figure = normalize_axes_target(
+ target,
+ panel_keys=["heatmap"],
+ figsize=figsize,
+ )
+ heatmap_ax = axes["heatmap"]
+ norm = continuous_norm(
+ mpl,
+ vmin=resolved_vmin,
+ vmax=resolved_vmax,
+ vcenter=resolved_color_scale.vcenter,
+ )
+ image = heatmap_ax.imshow(
+ ordered_matrix.to_numpy(dtype=np.float64),
+ cmap=resolved_cmap,
+ norm=norm,
+ aspect=heatmap_kwargs.get("aspect", "auto"),
+ interpolation=heatmap_kwargs.get("interpolation", "nearest"),
+ rasterized=heatmap_kwargs.get("rasterized", True),
+ )
+ heatmap_ax.set_xticks(range(ordered_matrix.shape[1]))
+ heatmap_ax.set_xticklabels(
+ ordered_matrix.columns,
+ fontsize=fontsize,
+ )
+ heatmap_ax.set_yticks(range(ordered_matrix.shape[0]))
+ heatmap_ax.set_yticklabels(
+ ordered_matrix.index,
+ fontsize=fontsize,
+ )
+ annotation_xlim, annotation_ylim = draw_annotation_strips(
+ heatmap_ax,
+ row_colors=row_colors.reindex(ordered_matrix.index),
+ column_colors=column_colors.reindex(ordered_matrix.columns),
+ n_rows=ordered_matrix.shape[0],
+ n_columns=ordered_matrix.shape[1],
+ )
+ heatmap_ax.set_xlim(annotation_xlim)
+ heatmap_ax.set_ylim(annotation_ylim)
+ if show_legend:
+ colorbar = figure.colorbar(
+ image,
+ ax=heatmap_ax,
+ location="top",
+ orientation="horizontal",
+ shrink=0.8,
+ fraction=0.06,
+ pad=0.04,
+ )
+ colorbar.set_label("standardized expression")
+ axes["colorbar"] = colorbar.ax
+ apply_figure_chrome(figure, theme)
+ displayed_matrix = ordered_matrix
+
+ if show_legend and resolved_annotation_scales:
+ annotation_handles: list[Any] = []
+ for name, scale in zip(
+ annotation_names,
+ resolved_annotation_scales,
+ strict=True,
+ ):
+ if scale.order is None or scale.palette is None:
+ continue
+ annotation_handles.extend(
+ mpl.lines.Line2D(
+ [],
+ [],
+ marker="s",
+ linestyle="",
+ markerfacecolor=scale.palette[value],
+ markeredgecolor="none",
+ markersize=5,
+ label=(
+ f"{name}: "
+ + (
+ scale.labels.get(value, str(value))
+ if scale.labels is not None
+ else str(value)
+ )
+ ),
+ )
+ for value in scale.order
+ )
+ if annotation_handles:
+ if owns_figure:
+ _place_clustermap_annotation_legend(
+ figure,
+ axes,
+ annotation_handles,
+ has_column_dendrogram=column_linkage is not None,
+ )
+ else:
+ axes["heatmap"].legend(
+ handles=annotation_handles,
+ title="Annotations",
+ frameon=False,
+ loc="upper left",
+ bbox_to_anchor=(1.02, 1),
+ )
+
+ resolved_scale = ColorScale(
+ cmap=(
+ resolved_cmap
+ if isinstance(resolved_cmap, str)
+ else getattr(resolved_cmap, "name", None)
+ ),
+ vmin=resolved_vmin,
+ vmax=resolved_vmax,
+ vcenter=resolved_color_scale.vcenter,
+ missing_color=resolved_color_scale.missing_color,
+ )
+ tables = {
+ "matrix": displayed_matrix.copy(),
+ "markers": cast(pd.DataFrame, prepared["markers"]).copy(),
+ }
+ if not row_annotation_values.empty:
+ tables["row_annotations"] = row_annotation_values.reindex(
+ displayed_matrix.index
+ )
+ if not column_annotation_values.empty:
+ tables["column_annotations"] = column_annotation_values.reindex(
+ displayed_matrix.columns
+ )
+ result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables=tables,
+ legends=(
+ LegendSpec(
+ kind="colorbar",
+ label="standardized expression",
+ extras={"vmin": resolved_vmin, "vmax": resolved_vmax},
+ ),
+ *(LegendSpec(kind="categorical", label=name) for name in annotation_names),
+ ),
+ scales=(
+ resolved_scale,
+ *resolved_annotation_scales,
+ CategoricalScale(order=tuple(displayed_matrix.columns)),
+ ),
+ provenance=PlotProvenance(
+ assay=cast(str, prepared["assay"]),
+ cell_key=cast(str, prepared["cell_key"]),
+ n_cells=cast(int, prepared["n_cells"]),
+ renderer="matplotlib",
+ notes=(
+ "marker_heatmap",
+ "clustered"
+ if row_linkage is not None or column_linkage is not None
+ else "ordered",
+ ),
+ extras={
+ "group_key": prepared["group_key"],
+ "topn": topn,
+ "log_transform": log_transform,
+ "vmin": resolved_vmin,
+ "vmax": resolved_vmax,
+ "row_order": list(displayed_matrix.index),
+ "column_order": list(displayed_matrix.columns),
+ "cluster_rows": row_linkage is not None,
+ "cluster_columns": column_linkage is not None,
+ "cluster_method": cluster_method,
+ "cluster_metric": cluster_metric,
+ "row_annotations": list(row_annotation_values.columns),
+ "column_annotations": list(column_annotation_values.columns),
+ },
+ ),
+ owns_figure=owns_figure,
+ theme=theme,
+ )
+ if show:
+ result.show()
+ return result
+
+
+def _prepare_pseudotime_heatmap(
+ store: Any,
+ *,
+ from_assay: str | None,
+ cell_key: str | None,
+ feat_key: str | None,
+ feature_cluster_key: str | None,
+ pseudotime_key: str | None,
+) -> dict[str, Any]:
+ assay = store._get_assay(from_assay)
+ if cell_key is None:
+ raise ValueError("ERROR: Please provide a value for parameter `cell_key`")
+ if feat_key is None:
+ raise ValueError("ERROR: Please provide a value for parameter `feat_key`")
+ if feature_cluster_key is None:
+ raise ValueError(
+ "ERROR: Please provide a value for parameter `feature_cluster_key`"
+ )
+ if pseudotime_key is None:
+ raise ValueError("ERROR: Please provide a value for parameter `pseudotime_key`")
+
+ cell_ordering = np.asarray(
+ assay.cells.fetch(pseudotime_key, key=cell_key),
+ dtype=float,
+ )
+ cell_index, feature_index = assay._get_cell_feat_idx(cell_key, feat_key)
+ hashes = [
+ array_digest(np.asarray(values))
+ for values in (cell_index, feature_index, cell_ordering)
+ ]
+ location = f"aggregated_{cell_key}_{feat_key}_{pseudotime_key}"
+ aggregation_group = None
+ artifact_backed = False
+ feature_data = as_zarr_group(assay.z["featureData"], name="featureData")
+ if feature_cluster_key in feature_data:
+ cluster_column = as_zarr_array(
+ feature_data[feature_cluster_key],
+ name=feature_cluster_key,
+ )
+ raw_ref = cluster_column.attrs.get("source_artifact")
+ if isinstance(raw_ref, dict):
+ try:
+ aggregation_ref = ArtifactRef.from_dict(raw_ref)
+ status = inspect_artifact(store.zw, aggregation_ref)
+ except (KeyError, TypeError, ValueError) as exc:
+ raise ValueError(
+ "Feature cluster column has an invalid source artifact. "
+ "Rerun run_pseudotime_aggregation"
+ ) from exc
+ source_value = cluster_column.attrs.get("source_value")
+ if (
+ aggregation_ref.kind != "pseudotime_aggregation"
+ or aggregation_ref.scope != "assay"
+ or aggregation_ref.assay != assay.name
+ or status.operation != "run_pseudotime_aggregation"
+ or not status.complete
+ or source_value != "cluster_values"
+ or "value_index" in cluster_column.attrs
+ ):
+ raise ValueError(
+ "Feature cluster column is not linked to a complete "
+ "pseudotime aggregation artifact for this assay"
+ )
+ aggregation_group = as_zarr_group(
+ store.zw[status.path],
+ name=status.path,
+ )
+ location = status.path
+ artifact_backed = True
+ elif raw_ref is not None:
+ raise ValueError(
+ "Feature cluster column has a malformed source artifact. "
+ "Rerun run_pseudotime_aggregation"
+ )
+ if aggregation_group is None:
+ if location not in assay.z:
+ raise KeyError(
+ "ERROR: Could not find aggregated feature values. "
+ "Please run `run_pseudotime_aggregation` with the same "
+ "`cell_key`, `feat_key`, and `pseudotime_key`"
+ )
+ aggregation_group = as_zarr_group(assay.z[location], name=location)
+ if (
+ ("input_fingerprints" if artifact_backed else "hashes")
+ not in aggregation_group.attrs
+ or "data" not in aggregation_group
+ or "feature_indices" not in aggregation_group
+ or "valid_features" not in aggregation_group
+ ):
+ raise ValueError(
+ f"Aggregated data at '{location}' is incomplete. "
+ "Rerun run_pseudotime_aggregation before plotting"
+ )
+ fingerprint_attr = "input_fingerprints" if artifact_backed else "hashes"
+ if hashes != cast(list[str], aggregation_group.attrs[fingerprint_attr]):
+ raise ValueError(
+ "ERROR: The values under one or more of these columns: `cell_key`, "
+ "`feat_key` or/and `pseudotime_key have been updated after running "
+ "`run_pseudotime_aggregation`"
+ )
+
+ data = ChunkedArray(
+ as_zarr_array(aggregation_group["data"], name="data"),
+ nthreads=store.nthreads,
+ )
+ feature_indices = np.asarray(
+ as_zarr_array(aggregation_group["feature_indices"], name="feature_indices")[:]
+ )
+ if "valid_features" not in aggregation_group:
+ raise ValueError(
+ f"Aggregated data at '{location}' has no valid_features mask. "
+ "Rerun run_pseudotime_aggregation"
+ )
+ valid_features = np.asarray(
+ as_zarr_array(aggregation_group["valid_features"], name="valid_features")[:],
+ dtype=bool,
+ )
+ if valid_features.shape[0] != feature_indices.shape[0]:
+ raise ValueError("Aggregated feature indices and validity mask are misaligned")
+ matrix = np.asarray(data[: feature_indices.shape[0]])
+ if matrix.shape[0] != feature_indices.shape[0]:
+ raise ValueError("Aggregated feature matrix and feature indices are misaligned")
+ matrix = matrix[valid_features]
+ feature_indices = feature_indices[valid_features]
+ if not np.isfinite(matrix).all():
+ raise ValueError("Aggregated feature matrix contains non-finite values")
+
+ all_feature_clusters = assay.feats.fetch_all(feature_cluster_key)
+ if artifact_backed:
+ if "cluster_values" not in aggregation_group:
+ raise ValueError(
+ "Aggregation artifact has no stored feature-cluster values"
+ )
+ stored_cluster_values = np.asarray(
+ as_zarr_array(
+ aggregation_group["cluster_values"],
+ name="cluster_values",
+ )[:]
+ )
+ if not np.array_equal(stored_cluster_values, all_feature_clusters):
+ logger.warning(
+ f"Feature cluster column '{feature_cluster_key}' changed after "
+ "aggregation and may be stale"
+ )
+ else:
+ cached_cluster_label = aggregation_group.attrs.get("cluster_label")
+ cached_cluster_digest = aggregation_group.attrs.get("cluster_digest")
+ current_cluster_digest = array_digest(
+ np.asarray(all_feature_clusters).astype(str)
+ )
+ if cached_cluster_label is None or cached_cluster_digest is None:
+ raise ValueError(
+ "Aggregated data has no completed feature-clustering provenance. "
+ "Rerun run_pseudotime_aggregation"
+ )
+ if cached_cluster_label != feature_cluster_key:
+ logger.warning(
+ f"Heatmap requested feature clusters '{feature_cluster_key}', but "
+ f"the aggregation cache was clustered as '{cached_cluster_label}'"
+ )
+ if cached_cluster_digest != current_cluster_digest:
+ logger.warning(
+ f"Feature cluster column '{feature_cluster_key}' changed after "
+ "aggregation and may be stale"
+ )
+
+ feature_clusters = np.asarray(all_feature_clusters)[feature_indices]
+ feature_labels = np.asarray(assay.feats.fetch_all("names"))[feature_indices]
+ order = np.argsort(feature_clusters)
+ return {
+ "matrix": matrix[order],
+ "feature_indices": feature_indices[order],
+ "feature_clusters": feature_clusters[order],
+ "feature_labels": feature_labels[order],
+ "pseudotime": np.asarray(
+ assay.cells.fetch(pseudotime_key, key=cell_key),
+ dtype=float,
+ ),
+ "assay": assay.name,
+ "cell_key": cell_key,
+ "feat_key": feat_key,
+ "feature_cluster_key": feature_cluster_key,
+ "pseudotime_key": pseudotime_key,
+ "aggregation_location": location,
+ }
+
+
+def pseudotime_heatmap(
+ store: Any,
+ *,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ feature_cluster_key: str | None = None,
+ pseudotime_key: str | None = None,
+ show_features: list[str] | None = None,
+ feature_order: Sequence[str] | None = None,
+ feature_cluster_order: Sequence[Any] | None = None,
+ figsize: tuple[float, float] = (5, 10),
+ vmin: float = -2.0,
+ vmax: float = 2.0,
+ heatmap_cmap: str | None = None,
+ pseudotime_cmap: str | None = None,
+ clusterbar_cmap: str | None = None,
+ color_scale: ColorScale | None = None,
+ feature_cluster_scale: CategoricalScale | None = None,
+ pseudotime_scale: ColorScale | None = None,
+ tick_fontsize: int = 10,
+ axis_fontsize: int = 12,
+ feature_label_fontsize: int = 12,
+ target: Mapping[Hashable, Any] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+) -> PlotResult:
+ """Plot binned feature dynamics ordered by pseudotime."""
+ resolved_color_scale = color_scale or ColorScale(
+ cmap=heatmap_cmap or "coolwarm",
+ vmin=vmin,
+ vmax=vmax,
+ )
+ if resolved_color_scale.scale != "linear":
+ raise NotImplementedError("pseudotime_heatmap supports linear scales")
+ resolved_vmin = (
+ float(resolved_color_scale.vmin)
+ if resolved_color_scale.vmin is not None
+ else float(vmin)
+ )
+ resolved_vmax = (
+ float(resolved_color_scale.vmax)
+ if resolved_color_scale.vmax is not None
+ else float(vmax)
+ )
+ if resolved_vmax <= resolved_vmin:
+ raise ValueError("vmax must be greater than vmin")
+ prepared = _prepare_pseudotime_heatmap(
+ store,
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ feature_cluster_key=feature_cluster_key,
+ pseudotime_key=pseudotime_key,
+ )
+ matrix = np.asarray(prepared["matrix"])
+ feature_clusters = np.asarray(prepared["feature_clusters"])
+ feature_labels = np.asarray(prepared["feature_labels"])
+ pseudotime = np.asarray(prepared["pseudotime"], dtype=float)
+ show_features = [] if show_features is None else list(show_features)
+ resolved_heatmap_cmap = resolved_color_scale.cmap or heatmap_cmap or "coolwarm"
+ resolved_pseudotime_scale = pseudotime_scale or ColorScale(
+ cmap=pseudotime_cmap or "viridis",
+ vmin=float(np.min(pseudotime)),
+ vmax=float(np.max(pseudotime)),
+ )
+ if resolved_pseudotime_scale.scale != "linear":
+ raise NotImplementedError("pseudotime annotations support linear scales")
+ resolved_pseudotime_cmap = (
+ resolved_pseudotime_scale.cmap or pseudotime_cmap or "viridis"
+ )
+ observed_clusters = list(pd.unique(feature_clusters))
+ requested_cluster_order = (
+ list(feature_cluster_order)
+ if feature_cluster_order is not None
+ else list(feature_cluster_scale.order)
+ if feature_cluster_scale is not None and feature_cluster_scale.order is not None
+ else sort_categories(observed_clusters)
+ )
+ if len(requested_cluster_order) != len(set(requested_cluster_order)):
+ raise ValueError("feature_cluster_order cannot contain duplicates")
+ missing_clusters = [
+ value for value in observed_clusters if value not in requested_cluster_order
+ ]
+ unexpected_clusters = [
+ value for value in requested_cluster_order if value not in observed_clusters
+ ]
+ if missing_clusters or unexpected_clusters:
+ raise ValueError(
+ "feature_cluster_order must contain every observed feature cluster"
+ )
+ cluster_order = requested_cluster_order
+ if feature_order is not None:
+ requested_features = list(feature_order)
+ if len(requested_features) != len(set(requested_features)):
+ raise ValueError("feature_order cannot contain duplicates")
+ observed_features = list(feature_labels)
+ if set(requested_features) != set(observed_features):
+ raise ValueError("feature_order must contain every plotted feature")
+ feature_index = {label: index for index, label in enumerate(feature_labels)}
+ row_order = np.asarray(
+ [feature_index[label] for label in requested_features],
+ dtype=np.intp,
+ )
+ else:
+ cluster_index = {cluster: index for index, cluster in enumerate(cluster_order)}
+ row_order = np.argsort(
+ np.asarray([cluster_index[value] for value in feature_clusters]),
+ kind="stable",
+ )
+ matrix = matrix[row_order]
+ feature_clusters = feature_clusters[row_order]
+ feature_labels = feature_labels[row_order]
+ cluster_codes = {cluster: index for index, cluster in enumerate(cluster_order)}
+ encoded_clusters = np.asarray(
+ [cluster_codes[value] for value in feature_clusters],
+ dtype=float,
+ )
+ binned_pseudotime = np.asarray(
+ [
+ values.mean()
+ for values in np.array_split(np.sort(pseudotime), matrix.shape[1])
+ ],
+ dtype=float,
+ )
+
+ plt, mpl = require_matplotlib()
+ if feature_cluster_scale is not None:
+ cluster_palette = categorical_color_map(
+ cluster_order,
+ palette=feature_cluster_scale.palette,
+ palette_name=feature_cluster_scale.palette_name,
+ )
+ else:
+ legacy_cluster_cmap = plt.get_cmap(clusterbar_cmap or "tab20")
+ cluster_palette = {
+ cluster: mpl.colors.to_hex(
+ legacy_cluster_cmap(index / max(len(cluster_order) - 1, 1))
+ )
+ for index, cluster in enumerate(cluster_order)
+ }
+ resolved_feature_cluster_scale = CategoricalScale(
+ order=tuple(cluster_order),
+ palette=cluster_palette,
+ labels=(
+ feature_cluster_scale.labels if feature_cluster_scale is not None else None
+ ),
+ missing_color=(
+ feature_cluster_scale.missing_color
+ if feature_cluster_scale is not None
+ else "#bdbdbd"
+ ),
+ missing_label=(
+ feature_cluster_scale.missing_label
+ if feature_cluster_scale is not None
+ else "NA"
+ ),
+ palette_name=(
+ feature_cluster_scale.palette_name
+ if feature_cluster_scale is not None
+ else "default"
+ ),
+ )
+ cluster_cmap = mpl.colors.ListedColormap(
+ [cluster_palette[cluster] for cluster in cluster_order]
+ )
+ pseudotime_vmin = (
+ float(resolved_pseudotime_scale.vmin)
+ if resolved_pseudotime_scale.vmin is not None
+ else float(np.min(pseudotime))
+ )
+ pseudotime_vmax = (
+ float(resolved_pseudotime_scale.vmax)
+ if resolved_pseudotime_scale.vmax is not None
+ else float(np.max(pseudotime))
+ )
+ with theme_context(theme):
+ if target is None:
+ fig = plt.figure(constrained_layout=False, figsize=figsize)
+ grid = fig.add_gridspec(
+ nrows=20,
+ ncols=20,
+ wspace=0,
+ hspace=0,
+ )
+ heatmap_ax = fig.add_subplot(grid[:-2, 1:16])
+ cluster_ax = fig.add_subplot(grid[:-2, 17:18])
+ colorbar_ax = fig.add_subplot(grid[7:12, -1:])
+ pseudotime_ax = fig.add_subplot(grid[-1:, 1:16])
+ axes: dict[Hashable, Any] = {
+ "heatmap": heatmap_ax,
+ "feature_clusters": cluster_ax,
+ "colorbar": colorbar_ax,
+ "pseudotime": pseudotime_ax,
+ }
+ owns_figure = True
+ else:
+ required_targets = {
+ "heatmap",
+ "feature_clusters",
+ "pseudotime",
+ }
+ missing_targets = required_targets - set(target)
+ if missing_targets:
+ raise ValueError(
+ "pseudotime_heatmap target is missing axes: "
+ + ", ".join(sorted(map(str, missing_targets)))
+ )
+ axes = dict(target)
+ heatmap_ax = axes["heatmap"]
+ cluster_ax = axes["feature_clusters"]
+ pseudotime_ax = axes["pseudotime"]
+ fig = heatmap_ax.figure
+ if any(axis.figure is not fig for axis in axes.values()):
+ raise ValueError(
+ "All pseudotime_heatmap target axes must share a figure"
+ )
+ colorbar_ax = axes.get("colorbar")
+ owns_figure = False
+
+ heatmap_norm = continuous_norm(
+ mpl,
+ vmin=resolved_vmin,
+ vmax=resolved_vmax,
+ vcenter=resolved_color_scale.vcenter,
+ )
+ image = heatmap_ax.imshow(
+ matrix,
+ aspect="auto",
+ interpolation="nearest",
+ cmap=resolved_heatmap_cmap,
+ norm=heatmap_norm,
+ rasterized=True,
+ )
+ heatmap_ax.set_xticks([])
+ heatmap_ax.set_yticks([])
+ if show_features:
+ row_index = {
+ str(label).lower(): index for index, label in enumerate(feature_labels)
+ }
+ visible_features = [
+ feature for feature in show_features if feature.lower() in row_index
+ ]
+ heatmap_ax.set_yticks(
+ [row_index[feature.lower()] for feature in visible_features]
+ )
+ heatmap_ax.set_yticklabels(
+ visible_features,
+ fontsize=feature_label_fontsize,
+ )
+ heatmap_ax.set_title(
+ f"{matrix.shape[0]} features",
+ fontsize=axis_fontsize,
+ )
+ if show_legend:
+ colorbar = (
+ fig.colorbar(image, cax=colorbar_ax)
+ if colorbar_ax is not None
+ else fig.colorbar(
+ image,
+ ax=heatmap_ax,
+ location="right",
+ fraction=0.06,
+ pad=0.04,
+ )
+ )
+ colorbar.ax.tick_params(labelsize=tick_fontsize)
+ colorbar.set_label("standardized expression", fontsize=axis_fontsize)
+ axes["colorbar"] = colorbar.ax
+ elif colorbar_ax is not None:
+ colorbar_ax.set_axis_off()
+
+ cluster_ax.imshow(
+ encoded_clusters.reshape(-1, 1),
+ aspect="auto",
+ interpolation="nearest",
+ cmap=cluster_cmap,
+ vmin=-0.5,
+ vmax=max(len(cluster_order) - 0.5, 0.5),
+ )
+ cluster_ax.set_xticks([])
+ cluster_ax.set_yticks([])
+ for cluster in cluster_order:
+ row = float(np.where(feature_clusters == cluster)[0].mean())
+ cluster_ax.text(
+ 0,
+ row,
+ str(cluster),
+ fontsize=axis_fontsize,
+ ha="center",
+ va="center",
+ )
+
+ pseudotime_ax.imshow(
+ binned_pseudotime.reshape(1, -1),
+ aspect="auto",
+ interpolation="nearest",
+ cmap=resolved_pseudotime_cmap,
+ vmin=pseudotime_vmin,
+ vmax=pseudotime_vmax,
+ )
+ pseudotime_ax.set_xticks([0, len(binned_pseudotime) - 1])
+ pseudotime_ax.set_xticklabels(
+ [f"{binned_pseudotime[0]:.2g}", f"{binned_pseudotime[-1]:.2g}"],
+ fontsize=tick_fontsize,
+ )
+ pseudotime_ax.set_yticks([])
+ pseudotime_ax.set_xlabel(
+ "Pseudotime",
+ fontsize=axis_fontsize,
+ )
+ apply_figure_chrome(fig, theme)
+
+ feature_indices = np.asarray(prepared["feature_indices"])[row_order]
+ feature_table = pd.DataFrame(
+ {
+ "feature_index": feature_indices,
+ "feature": feature_labels,
+ "cluster": feature_clusters,
+ }
+ )
+ matrix_table = pd.DataFrame(
+ matrix,
+ index=pd.Index(feature_labels, name="feature"),
+ )
+ pseudotime_table = pd.DataFrame(
+ {
+ "cell_order": np.arange(len(pseudotime)),
+ "pseudotime": pseudotime,
+ }
+ )
+ pseudotime_bins = pd.DataFrame(
+ {
+ "bin": np.arange(len(binned_pseudotime)),
+ "pseudotime": binned_pseudotime,
+ }
+ )
+ result = PlotResult(
+ figure=fig,
+ axes=axes,
+ tables={
+ "matrix": matrix_table,
+ "features": feature_table,
+ "pseudotime": pseudotime_table,
+ "pseudotime_bins": pseudotime_bins,
+ },
+ legends=(
+ LegendSpec(
+ kind="colorbar",
+ label="standardized expression",
+ extras={"vmin": resolved_vmin, "vmax": resolved_vmax},
+ ),
+ LegendSpec(kind="categorical", label=feature_cluster_key),
+ LegendSpec(
+ kind="colorbar",
+ label=pseudotime_key,
+ extras={"vmin": pseudotime_vmin, "vmax": pseudotime_vmax},
+ ),
+ ),
+ scales=(
+ ColorScale(
+ cmap=resolved_heatmap_cmap,
+ vmin=resolved_vmin,
+ vmax=resolved_vmax,
+ vcenter=resolved_color_scale.vcenter,
+ ),
+ resolved_feature_cluster_scale,
+ ColorScale(
+ cmap=resolved_pseudotime_cmap,
+ vmin=pseudotime_vmin,
+ vmax=pseudotime_vmax,
+ ),
+ ),
+ provenance=PlotProvenance(
+ assay=cast(str, prepared["assay"]),
+ cell_key=cast(str, prepared["cell_key"]),
+ n_cells=len(pseudotime),
+ renderer="matplotlib",
+ notes=("pseudotime_heatmap", "aggregated"),
+ extras={
+ "feat_key": prepared["feat_key"],
+ "feature_cluster_key": prepared["feature_cluster_key"],
+ "pseudotime_key": prepared["pseudotime_key"],
+ "aggregation_location": prepared["aggregation_location"],
+ "n_features": matrix.shape[0],
+ "n_bins": matrix.shape[1],
+ "show_features": show_features,
+ "feature_order": list(feature_labels),
+ "feature_cluster_order": list(cluster_order),
+ "show_legend": show_legend,
+ },
+ ),
+ owns_figure=owns_figure,
+ theme=theme,
+ )
+ if show:
+ result.show()
+ return result
diff --git a/scarf/plotting/mapping.py b/scarf/plotting/mapping.py
new file mode 100644
index 00000000..3cfcd18c
--- /dev/null
+++ b/scarf/plotting/mapping.py
@@ -0,0 +1,1472 @@
+"""Diagnostics for reference mapping and label transfer."""
+
+from collections.abc import Sequence
+from typing import Any, Hashable, Literal
+import warnings
+
+import numpy as np
+import pandas as pd
+
+from ..mapping.models import MappingResult
+from ..mapping.reference import MappingReference
+from ..storage.refs import ArtifactRef
+from ._contracts import (
+ CategoricalScale,
+ ColorScale,
+ PlotProvenance,
+)
+from ._deps import require_matplotlib
+from ._figure import LegendSpec, PlotResult, normalize_axes_target
+from ._style import (
+ apply_figure_chrome,
+ categorical_color_map,
+ continuous_norm,
+ default_point_size,
+ finish_embedding_axes,
+ scatter_edgecolor,
+ sort_categories,
+ square_axis_limits,
+ theme_context,
+)
+
+
+def _mapping_result(
+ store: Any,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None,
+ query_assay: str | None,
+) -> MappingResult:
+ loader = getattr(store, "get_mapping_result", None)
+ if not callable(loader):
+ raise TypeError("store does not provide mapping result data")
+ loaded = loader(
+ result,
+ reference=reference,
+ query_assay=query_assay,
+ load_arrays=False,
+ )
+ if not isinstance(loaded, MappingResult):
+ raise TypeError("store returned an invalid mapping result")
+ if loaded.reference is None:
+ raise ValueError("Resolved mapping result has no MappingReference")
+ return loaded
+
+
+def _mapping_reference_embedding(
+ store: Any,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None,
+ reference_layout_key: str,
+ query_assay: str | None,
+) -> np.ndarray:
+ loader = getattr(store, "_load_mapping_reference_embedding", None)
+ if not callable(loader):
+ raise TypeError("store does not provide persisted mapping embedding data")
+ coordinates = np.asarray(
+ loader(
+ result,
+ reference_layout_key,
+ reference=reference,
+ query_assay=query_assay,
+ ),
+ dtype=np.float64,
+ )
+ if coordinates.ndim != 2 or coordinates.shape[1] != 2:
+ raise ValueError("Persisted mapping embedding must have two coordinate columns")
+ if not np.all(np.isfinite(coordinates) | np.isnan(coordinates)):
+ raise ValueError("Persisted mapping embedding contains infinite coordinates")
+ return coordinates
+
+
+def _reference_layout(
+ reference: MappingReference,
+ layout_key: str,
+) -> np.ndarray:
+ layout = np.asarray(reference.fetch_layout(layout_key), dtype=np.float64)
+ if layout.shape != (reference.selected_cell_count, 2):
+ raise ValueError(
+ "Reference layout must have two columns and one row per selected "
+ "reference cell"
+ )
+ if not np.all(np.isfinite(layout) | np.isnan(layout)):
+ raise ValueError("Reference layout contains infinite coordinates")
+ return layout
+
+
+def _external_groups(
+ values: Sequence[Any] | np.ndarray | None,
+ n_rows: int,
+ *,
+ default: str,
+ argument_name: str,
+) -> np.ndarray:
+ if values is None:
+ return np.full(n_rows, default, dtype=object)
+ groups = np.asarray(values, dtype=object)
+ if groups.ndim != 1 or len(groups) != n_rows:
+ raise ValueError(f"{argument_name} must have one value per mapped cell")
+ if pd.isna(groups).any():
+ raise ValueError(f"{argument_name} cannot contain missing values")
+ return groups
+
+
+def _categorical_contract(
+ values: np.ndarray,
+ scale: CategoricalScale | None,
+) -> tuple[list[Any], dict[Any, str], CategoricalScale]:
+ observed = list(pd.unique(values))
+ if scale is not None and scale.order is not None:
+ missing = [value for value in observed if value not in scale.order]
+ if missing:
+ raise ValueError(
+ "categorical_scale.order is missing values: "
+ + ", ".join(map(str, missing[:10]))
+ )
+ order = [value for value in scale.order if value in set(observed)]
+ else:
+ order = sort_categories(observed)
+ palette = categorical_color_map(
+ order,
+ palette=scale.palette if scale is not None else None,
+ palette_name=scale.palette_name if scale is not None else "default",
+ )
+ return (
+ order,
+ palette,
+ CategoricalScale(
+ order=tuple(order),
+ palette=palette,
+ labels=scale.labels if scale is not None else None,
+ missing_color=scale.missing_color if scale is not None else "#bdbdbd",
+ missing_label=scale.missing_label if scale is not None else "NA",
+ palette_name=scale.palette_name if scale is not None else "default",
+ ),
+ )
+
+
+def _axis_limits(x: np.ndarray, y: np.ndarray) -> tuple[Any, Any]:
+ if x.shape != y.shape:
+ raise ValueError("Coordinate columns must have matching shapes")
+ finite = np.isfinite(x) & np.isfinite(y)
+ if not finite.any():
+ raise ValueError("No finite coordinates are available to plot")
+ finite_x = x[finite]
+ finite_y = y[finite]
+ xpad = 0.05 * (float(np.ptp(finite_x)) or 1.0)
+ ypad = 0.05 * (float(np.ptp(finite_y)) or 1.0)
+ return square_axis_limits(
+ (float(np.min(finite_x) - xpad), float(np.max(finite_x) + xpad)),
+ (float(np.min(finite_y) - ypad), float(np.max(finite_y) + ypad)),
+ )
+
+
+def mapping_score(
+ store: Any,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None = None,
+ target_groups: Sequence[Any] | np.ndarray | None = None,
+ layout_key: str | None = None,
+ kind: Literal["embedding", "histogram"] = "embedding",
+ query_assay: str | None = None,
+ log_transform: bool = True,
+ multiplier: float = 1000,
+ weighted: bool = True,
+ fixed_weight: float = 0.1,
+ bins: int = 40,
+ point_size: float | None = None,
+ color_scale: ColorScale | None = None,
+ categorical_scale: CategoricalScale | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+) -> PlotResult:
+ """Plot reference-cell mapping scores for one or more query groups."""
+ if kind not in ("embedding", "histogram"):
+ raise ValueError("kind must be 'embedding' or 'histogram'")
+ if kind == "embedding" and layout_key is None:
+ raise ValueError("layout_key is required for an embedding mapping score")
+ if bins < 1:
+ raise ValueError("bins must be positive")
+ mapping = _mapping_result(
+ store,
+ result,
+ reference=reference,
+ query_assay=query_assay,
+ )
+ score_loader = getattr(store, "get_mapping_score", None)
+ if not callable(score_loader):
+ raise TypeError("store does not provide mapping score data")
+ score_rows = list(
+ score_loader(
+ mapping,
+ target_groups=(
+ None if target_groups is None else np.asarray(target_groups)
+ ),
+ log_transform=log_transform,
+ multiplier=multiplier,
+ weighted=weighted,
+ fixed_weight=fixed_weight,
+ )
+ )
+ if not score_rows:
+ raise ValueError(f"Mapping {mapping.mapping_name!r} produced no score groups")
+ labels = [row[0] for row in score_rows]
+ display_labels = {label: str(label) for label in labels}
+ if target_groups is None and len(labels) == 1:
+ display_labels[labels[0]] = mapping.mapping_name
+ score_arrays = [np.asarray(row[1], dtype=np.float64) for row in score_rows]
+ n_reference = len(score_arrays[0])
+ if any(values.shape != (n_reference,) for values in score_arrays):
+ raise ValueError("Mapping score groups have incompatible lengths")
+ assert mapping.reference is not None
+ if n_reference != mapping.reference.selected_cell_count:
+ raise ValueError("Mapping scores do not match the selected reference cells")
+ score_table = pd.concat(
+ (
+ pd.DataFrame(
+ {
+ "group": label,
+ "referenceIndex": np.arange(n_reference),
+ "score": values,
+ }
+ )
+ for label, values in zip(labels, score_arrays, strict=True)
+ ),
+ ignore_index=True,
+ )
+ _, mpl = require_matplotlib()
+ # Sparse mapping scores are mostly zero. Clip the color ceiling to the upper
+ # tail so the few reference cells that received query weight remain visible.
+ color_scale = color_scale or ColorScale(
+ cmap="magma",
+ vmin=0.0,
+ quantiles=(0.0, 0.995),
+ )
+ panel_keys: list[Hashable] = (
+ list(labels) if kind == "embedding" else ["mapping_score"]
+ )
+ if figsize is None and target is None:
+ figsize = (3.6 * len(panel_keys), 3.4) if kind == "embedding" else (5.0, 3.6)
+ legends: list[LegendSpec] = []
+ scales: list[Any] = []
+ with theme_context(theme):
+ figure, axes, owns = normalize_axes_target(
+ target,
+ panel_keys=panel_keys,
+ figsize=figsize,
+ )
+ if kind == "embedding":
+ assert layout_key is not None
+ layout = _reference_layout(mapping.reference, layout_key)
+ x = layout[:, 0]
+ y = layout[:, 1]
+ xlim, ylim = _axis_limits(x, y)
+ shared_values = np.concatenate(
+ [values[np.isfinite(values) & (values > 0)] for values in score_arrays]
+ )
+ if shared_values.size == 0:
+ shared_values = np.concatenate(score_arrays)
+ shared_limits = (
+ _continuous_limits(shared_values, color_scale)
+ if color_scale.scope == "shared"
+ else None
+ )
+ resolved_point_size = (
+ float(point_size)
+ if point_size is not None
+ else default_point_size(n_reference)
+ )
+ for label, values in zip(labels, score_arrays, strict=True):
+ ax = axes[label]
+ positive = values[np.isfinite(values) & (values > 0)]
+ limits = shared_limits or _continuous_limits(
+ positive if positive.size else values,
+ color_scale,
+ )
+ norm = continuous_norm(
+ mpl,
+ vmin=limits[0],
+ vmax=limits[1],
+ vcenter=color_scale.vcenter,
+ )
+ visible = np.isfinite(x) & np.isfinite(y) & np.isfinite(values)
+ # Draw the full reference cloud first so zero-score cells stay
+ # readable as context, then overlay the cells that received weight.
+ ax.scatter(
+ x[visible],
+ y[visible],
+ c=color_scale.missing_color,
+ s=resolved_point_size,
+ edgecolors="none",
+ rasterized=n_reference >= 50_000,
+ zorder=0,
+ )
+ highlighted = visible & (values > 0)
+ order_index = np.flatnonzero(highlighted)
+ order_index = order_index[np.argsort(values[order_index])]
+ artist = ax.scatter(
+ x[order_index],
+ y[order_index],
+ c=values[order_index],
+ s=resolved_point_size * 1.35,
+ cmap=color_scale.cmap or "magma",
+ norm=norm,
+ edgecolors="none",
+ rasterized=n_reference >= 50_000,
+ zorder=1,
+ )
+ finish_embedding_axes(
+ ax,
+ xlim=xlim,
+ ylim=ylim,
+ title=display_labels[label],
+ frame="minimal",
+ )
+ if show_legend:
+ colorbar = figure.colorbar(
+ artist,
+ ax=ax,
+ location="top",
+ orientation="horizontal",
+ shrink=0.8,
+ fraction=0.06,
+ pad=0.03,
+ )
+ colorbar.set_label("Mapping score")
+ legends.append(
+ LegendSpec(
+ kind="colorbar",
+ label=f"Mapping score: {display_labels[label]}",
+ extras={"vmin": limits[0], "vmax": limits[1]},
+ )
+ )
+ scales.append(color_scale)
+ else:
+ group_values = np.asarray(labels, dtype=object)
+ order, palette, resolved_categorical = _categorical_contract(
+ group_values,
+ categorical_scale,
+ )
+ ax = axes["mapping_score"]
+ finite = np.concatenate(
+ [values[np.isfinite(values)] for values in score_arrays]
+ )
+ shared_bins: int | np.ndarray = (
+ np.histogram_bin_edges(finite, bins=bins) if len(finite) else bins
+ )
+ for label in order:
+ values = score_arrays[labels.index(label)]
+ ax.hist(
+ values[np.isfinite(values)],
+ bins=shared_bins,
+ histtype="step",
+ linewidth=1.5,
+ color=palette[label],
+ label=display_labels[label],
+ )
+ ax.set_xlabel("Mapping score")
+ ax.set_ylabel("Reference cells")
+ if show_legend:
+ ax.legend(frameon=False, title="Query group")
+ legends.append(LegendSpec(kind="categorical", label="Query group"))
+ scales.append(resolved_categorical)
+ apply_figure_chrome(figure, theme)
+ plot_result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables={"scores": score_table},
+ legends=tuple(legends),
+ scales=tuple(scales),
+ provenance=PlotProvenance(
+ assay=mapping.ref.assay,
+ cell_key=None,
+ n_cells=n_reference,
+ renderer="matplotlib",
+ notes=("mapping_score", kind),
+ extras={
+ "mapping_name": mapping.mapping_name,
+ "layout_key": layout_key,
+ "groups": list(labels),
+ "log_transform": log_transform,
+ "multiplier": multiplier,
+ "weighted": weighted,
+ },
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ plot_result.show()
+ return plot_result
+
+
+def _continuous_limits(
+ values: np.ndarray,
+ scale: ColorScale,
+) -> tuple[float, float]:
+ finite = values[np.isfinite(values)]
+ if len(finite) == 0:
+ return (0.0, 1.0)
+ if scale.quantiles is not None:
+ low, high = np.quantile(finite, scale.quantiles)
+ else:
+ low, high = float(np.min(finite)), float(np.max(finite))
+ if scale.vmin is not None:
+ low = scale.vmin
+ if scale.vmax is not None:
+ high = scale.vmax
+ if high <= low:
+ high = low + 1.0
+ return float(low), float(high)
+
+
+def _label_evidence(
+ store: Any,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None,
+ reference_class_group: str,
+ query_assay: str | None,
+ threshold_fraction: float,
+ na_val: str,
+ max_distance: float | None,
+) -> pd.DataFrame:
+ loader = getattr(store, "get_target_label_evidence", None)
+ if not callable(loader):
+ raise TypeError("store does not provide mapping evidence data")
+ return loader(
+ result,
+ reference_class_group=reference_class_group,
+ reference=reference,
+ query_assay=query_assay,
+ threshold_fraction=threshold_fraction,
+ na_val=na_val,
+ max_distance=max_distance,
+ ).copy()
+
+
+def mapping_evidence(
+ store: Any,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None = None,
+ reference_class_group: str,
+ target_groups: Sequence[Any] | np.ndarray | None = None,
+ metrics: Sequence[str] = (
+ "voteFraction",
+ "topTwoMargin",
+ "voteEntropy",
+ "referenceDistancePercentile",
+ ),
+ kind: Literal["histogram", "box", "embedding"] = "histogram",
+ reference_layout_key: str | None = None,
+ bins: int = 30,
+ query_assay: str | None = None,
+ threshold_fraction: float = 0.5,
+ na_val: str = "NA",
+ max_distance: float | None = None,
+ categorical_scale: CategoricalScale | None = None,
+ color_scale: ColorScale | None = None,
+ point_size: float | None = None,
+ show_unknown: bool = True,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+) -> PlotResult:
+ """Plot query-level label-transfer evidence."""
+ if kind not in ("histogram", "box", "embedding"):
+ raise ValueError("kind must be 'histogram', 'box', or 'embedding'")
+ if kind == "embedding" and reference_layout_key is None:
+ raise ValueError("reference_layout_key is required for an evidence embedding")
+ if bins < 1:
+ raise ValueError("bins must be positive")
+ mapping = _mapping_result(
+ store,
+ result,
+ reference=reference,
+ query_assay=query_assay,
+ )
+ assert mapping.reference is not None
+ evidence = _label_evidence(
+ store,
+ mapping,
+ reference=mapping.reference,
+ reference_class_group=reference_class_group,
+ query_assay=None,
+ threshold_fraction=threshold_fraction,
+ na_val=na_val,
+ max_distance=max_distance,
+ )
+ groups = _external_groups(
+ target_groups,
+ len(evidence),
+ default=mapping.mapping_name,
+ argument_name="target_groups",
+ )
+ evidence["group"] = groups
+ requested_metrics = list(metrics)
+ if not requested_metrics:
+ raise ValueError("metrics must be non-empty")
+ missing = [metric for metric in requested_metrics if metric not in evidence]
+ if missing:
+ raise KeyError("Unknown evidence metrics: " + ", ".join(missing))
+ for metric in requested_metrics:
+ evidence[metric] = pd.to_numeric(evidence[metric], errors="coerce")
+ order: list[Any] = []
+ palette: dict[Any, str] = {}
+ resolved_categorical: CategoricalScale | None = None
+ reference_layout: np.ndarray | None = None
+ projected: np.ndarray | None = None
+ visible_unknown: np.ndarray | None = None
+ if kind == "embedding":
+ assert reference_layout_key is not None
+ reference_layout = _reference_layout(
+ mapping.reference,
+ reference_layout_key,
+ )
+ projected = _mapping_reference_embedding(
+ store,
+ mapping,
+ reference=mapping.reference,
+ reference_layout_key=reference_layout_key,
+ query_assay=None,
+ )
+ if len(projected) != len(evidence):
+ raise ValueError("Projected coordinates do not match label evidence")
+ color_scale = color_scale or ColorScale(
+ cmap="viridis",
+ quantiles=(0.01, 0.99),
+ )
+ else:
+ order, palette, resolved_categorical = _categorical_contract(
+ groups,
+ categorical_scale,
+ )
+ if figsize is None and target is None:
+ panel_width = 3.5 if kind == "embedding" else 4.0
+ figsize = (panel_width * min(len(requested_metrics), 3), 3.3)
+ _, mpl = require_matplotlib()
+ legend_specs: list[LegendSpec] = []
+ with theme_context(theme):
+ figure, axes, owns = normalize_axes_target(
+ target,
+ panel_keys=requested_metrics,
+ figsize=figsize,
+ n_columns=min(len(requested_metrics), 3),
+ )
+ if kind == "embedding":
+ assert reference_layout is not None and projected is not None
+ assert color_scale is not None
+ combined = np.vstack((reference_layout, projected))
+ xlim, ylim = _axis_limits(combined[:, 0], combined[:, 1])
+ shared_limits = (
+ _continuous_limits(
+ evidence[requested_metrics].to_numpy(dtype=np.float64),
+ color_scale,
+ )
+ if color_scale.scope == "shared"
+ else None
+ )
+ resolved_point_size = (
+ float(point_size)
+ if point_size is not None
+ else default_point_size(len(projected))
+ )
+ unknown = evidence["isUnknown"].to_numpy(dtype=bool)
+ projected_finite = np.isfinite(projected).all(axis=1)
+ reference_finite = np.isfinite(reference_layout).all(axis=1)
+ visible_unknown = unknown & projected_finite
+ for metric_index, metric in enumerate(requested_metrics):
+ ax = axes[metric]
+ if kind == "embedding":
+ assert reference_layout is not None and projected is not None
+ assert visible_unknown is not None
+ assert color_scale is not None
+ values = evidence[metric].to_numpy(dtype=np.float64)
+ limits = shared_limits or _continuous_limits(values, color_scale)
+ norm = continuous_norm(
+ mpl,
+ vmin=limits[0],
+ vmax=limits[1],
+ vcenter=color_scale.vcenter,
+ )
+ ax.scatter(
+ reference_layout[reference_finite, 0],
+ reference_layout[reference_finite, 1],
+ s=resolved_point_size * 0.55,
+ c="#bdbdbd",
+ alpha=0.2,
+ linewidths=0,
+ rasterized=int(reference_finite.sum()) >= 50_000,
+ zorder=0,
+ )
+ visible = projected_finite & np.isfinite(values)
+ draw_order = np.flatnonzero(visible)
+ draw_order = draw_order[np.argsort(values[draw_order])]
+ artist = ax.scatter(
+ projected[draw_order, 0],
+ projected[draw_order, 1],
+ c=values[draw_order],
+ s=resolved_point_size,
+ cmap=color_scale.cmap or "viridis",
+ norm=norm,
+ edgecolors="none",
+ rasterized=len(projected) >= 50_000,
+ zorder=1,
+ )
+ if show_unknown and visible_unknown.any():
+ ax.scatter(
+ projected[visible_unknown, 0],
+ projected[visible_unknown, 1],
+ s=resolved_point_size * 1.6,
+ marker="x",
+ c=scatter_edgecolor(theme),
+ linewidths=0.7,
+ rasterized=len(projected) >= 50_000,
+ zorder=2,
+ )
+ finish_embedding_axes(
+ ax,
+ xlim=xlim,
+ ylim=ylim,
+ title=metric,
+ frame="minimal",
+ )
+ if show_legend:
+ colorbar = figure.colorbar(
+ artist,
+ ax=ax,
+ location="top",
+ orientation="horizontal",
+ shrink=0.8,
+ fraction=0.06,
+ pad=0.03,
+ )
+ colorbar.set_label(metric)
+ legend_specs.append(
+ LegendSpec(
+ kind="colorbar",
+ label=metric,
+ extras={"vmin": limits[0], "vmax": limits[1]},
+ )
+ )
+ elif kind == "histogram":
+ all_values = evidence[metric].to_numpy(dtype=np.float64)
+ all_values = all_values[np.isfinite(all_values)]
+ shared_bins: int | np.ndarray = (
+ np.histogram_bin_edges(all_values, bins=bins)
+ if len(all_values)
+ else bins
+ )
+ for group in order:
+ values = evidence.loc[
+ evidence["group"] == group,
+ metric,
+ ].to_numpy(dtype=np.float64)
+ ax.hist(
+ values[np.isfinite(values)],
+ bins=shared_bins,
+ histtype="step",
+ linewidth=1.4,
+ color=palette[group],
+ label=str(group),
+ )
+ ax.set_ylabel("Mapped cells")
+ else:
+ grouped_values = [
+ evidence.loc[
+ evidence["group"] == group,
+ metric,
+ ].dropna()
+ for group in order
+ ]
+ boxes = ax.boxplot(
+ grouped_values,
+ tick_labels=[str(group) for group in order],
+ patch_artist=True,
+ showfliers=False,
+ )
+ for patch, group in zip(
+ boxes["boxes"],
+ order,
+ strict=True,
+ ):
+ patch.set_facecolor(palette[group])
+ patch.set_alpha(0.8)
+ ax.tick_params(axis="x", labelrotation=45)
+ if kind != "embedding":
+ ax.set_xlabel(metric)
+ if show_legend and kind == "histogram" and metric_index == 0:
+ ax.legend(frameon=False, title="Query group")
+ apply_figure_chrome(figure, theme)
+ if kind != "embedding":
+ legend_specs.append(LegendSpec(kind="categorical", label="Query group"))
+ elif show_unknown and visible_unknown is not None and visible_unknown.any():
+ legend_specs.append(
+ LegendSpec(
+ kind="marker",
+ label="Transfer status",
+ extras={"values": ["Unknown"], "markers": ["x"]},
+ )
+ )
+ if kind == "embedding":
+ assert color_scale is not None
+ result_scales: tuple[Any, ...] = (color_scale,)
+ else:
+ assert resolved_categorical is not None
+ result_scales = (resolved_categorical,)
+ plot_result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables={"evidence": evidence},
+ legends=tuple(legend_specs),
+ scales=result_scales,
+ provenance=PlotProvenance(
+ assay=mapping.ref.assay,
+ cell_key=None,
+ n_cells=len(evidence),
+ renderer="matplotlib",
+ notes=("mapping_evidence", kind),
+ extras={
+ "mapping_name": mapping.mapping_name,
+ "reference_class_group": reference_class_group,
+ "metrics": requested_metrics,
+ "reference_layout_key": reference_layout_key,
+ "threshold_fraction": threshold_fraction,
+ "max_distance": max_distance,
+ },
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ plot_result.show()
+ return plot_result
+
+
+def mapping_confusion(
+ store: Any,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None = None,
+ reference_class_group: str,
+ known_labels: Sequence[Any] | np.ndarray,
+ normalize: Literal["none", "true", "predicted", "all"] = "true",
+ known_order: Sequence[Any] | None = None,
+ predicted_order: Sequence[Any] | None = None,
+ query_assay: str | None = None,
+ threshold_fraction: float = 0.5,
+ na_val: str = "NA",
+ max_distance: float | None = None,
+ color_scale: ColorScale | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+) -> PlotResult:
+ """Plot known query labels against transferred labels."""
+ if normalize not in ("none", "true", "predicted", "all"):
+ raise ValueError("normalize must be 'none', 'true', 'predicted', or 'all'")
+ mapping = _mapping_result(
+ store,
+ result,
+ reference=reference,
+ query_assay=query_assay,
+ )
+ assert mapping.reference is not None
+ evidence = _label_evidence(
+ store,
+ mapping,
+ reference=mapping.reference,
+ reference_class_group=reference_class_group,
+ query_assay=None,
+ threshold_fraction=threshold_fraction,
+ na_val=na_val,
+ max_distance=max_distance,
+ )
+ known = np.asarray(known_labels, dtype=object)
+ if known.ndim != 1 or len(known) != len(evidence):
+ raise ValueError("known_labels must have one value per mapped cell")
+ valid = pd.notna(known)
+ truth = known[valid]
+ predicted = evidence.loc[valid, "label"].to_numpy(dtype=object)
+ observed_known = sort_categories(list(pd.unique(truth)))
+ observed_predicted = sort_categories(list(pd.unique(predicted)))
+ rows = list(known_order) if known_order is not None else observed_known
+ columns = (
+ list(predicted_order) if predicted_order is not None else observed_predicted
+ )
+ if any(value not in rows for value in observed_known):
+ raise ValueError("known_order is missing observed labels")
+ if any(value not in columns for value in observed_predicted):
+ raise ValueError("predicted_order is missing observed labels")
+ counts = pd.crosstab(
+ pd.Series(truth, name="known"),
+ pd.Series(predicted, name="predicted"),
+ ).reindex(index=rows, columns=columns, fill_value=0)
+ class_rows = []
+ for label in dict.fromkeys([*rows, *columns]):
+ true_positive = (
+ int(counts.loc[label, label])
+ if label in counts.index and label in counts.columns
+ else 0
+ )
+ support = int(counts.loc[label].sum()) if label in counts.index else 0
+ predicted_count = int(counts[label].sum()) if label in counts.columns else 0
+ class_rows.append(
+ {
+ "label": label,
+ "precision": (
+ true_positive / predicted_count if predicted_count else np.nan
+ ),
+ "recall": true_positive / support if support else np.nan,
+ "support": support,
+ "predicted": predicted_count,
+ }
+ )
+ per_class = pd.DataFrame(class_rows)
+ display = counts.astype(np.float64)
+ if normalize == "true":
+ display = display.div(display.sum(axis=1).replace(0, np.nan), axis=0)
+ elif normalize == "predicted":
+ display = display.div(display.sum(axis=0).replace(0, np.nan), axis=1)
+ elif normalize == "all":
+ display = display / max(float(display.to_numpy().sum()), 1.0)
+ display = display.fillna(0)
+ color_scale = color_scale or ColorScale(cmap="Blues", vmin=0)
+ limits = _continuous_limits(display.to_numpy(), color_scale)
+ _, mpl = require_matplotlib()
+ with theme_context(theme):
+ figure, axes, owns = normalize_axes_target(
+ target,
+ panel_keys=["mapping_confusion"],
+ figsize=figsize or ((5.2, 4.6) if target is None else None),
+ )
+ ax = axes["mapping_confusion"]
+ norm = continuous_norm(
+ mpl,
+ vmin=limits[0],
+ vmax=limits[1],
+ vcenter=color_scale.vcenter,
+ )
+ image = ax.imshow(
+ display.to_numpy(),
+ cmap=color_scale.cmap or "Blues",
+ norm=norm,
+ aspect="auto",
+ interpolation="nearest",
+ )
+ ax.set_xticks(range(len(columns)))
+ ax.set_xticklabels(columns, rotation=45, ha="right")
+ ax.set_yticks(range(len(rows)))
+ ax.set_yticklabels(rows)
+ ax.set_xlabel("Predicted label")
+ ax.set_ylabel("Known label")
+ midpoint = 0.5 * sum(limits)
+ for row_index in range(len(rows)):
+ for column_index in range(len(columns)):
+ value = float(display.iloc[row_index, column_index])
+ label = f"{value:.0f}" if normalize == "none" else f"{value:.0%}"
+ ax.text(
+ column_index,
+ row_index,
+ label,
+ ha="center",
+ va="center",
+ fontsize=7,
+ color="#ffffff" if value > midpoint else "#222222",
+ )
+ if show_legend:
+ colorbar = figure.colorbar(
+ image,
+ ax=ax,
+ location="top",
+ orientation="horizontal",
+ shrink=0.75,
+ fraction=0.06,
+ pad=0.04,
+ )
+ colorbar.set_label("Cells" if normalize == "none" else "Fraction of cells")
+ apply_figure_chrome(figure, theme)
+ plot_result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables={
+ "counts": counts.reset_index(),
+ "matrix": display.reset_index(),
+ "perClass": per_class,
+ "evidence": evidence.assign(knownLabel=known),
+ },
+ legends=(
+ LegendSpec(
+ kind="colorbar",
+ label="Cells" if normalize == "none" else "Fraction of cells",
+ extras={"vmin": limits[0], "vmax": limits[1]},
+ ),
+ ),
+ scales=(color_scale,),
+ provenance=PlotProvenance(
+ assay=mapping.ref.assay,
+ cell_key=None,
+ n_cells=int(valid.sum()),
+ renderer="matplotlib",
+ notes=("mapping_confusion",),
+ extras={
+ "mapping_name": mapping.mapping_name,
+ "reference_class_group": reference_class_group,
+ "normalize": normalize,
+ "threshold_fraction": threshold_fraction,
+ "dropped_missing_truth": int((~valid).sum()),
+ },
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ plot_result.show()
+ return plot_result
+
+
+def mapping_calibration(
+ store: Any,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None = None,
+ reference_class_group: str,
+ known_labels: Sequence[Any] | np.ndarray,
+ metric: str = "voteFraction",
+ direction: Literal["auto", "higher", "lower"] = "auto",
+ thresholds: Sequence[float] | np.ndarray | None = None,
+ n_thresholds: int = 50,
+ chosen_threshold: float | None = None,
+ query_assay: str | None = None,
+ na_val: str = "NA",
+ max_distance: float | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show: bool = True,
+) -> PlotResult:
+ """Plot held-out label accuracy against retained mapping coverage."""
+ if n_thresholds < 2:
+ raise ValueError("n_thresholds must be at least 2")
+ if direction not in ("auto", "higher", "lower"):
+ raise ValueError("direction must be 'auto', 'higher', or 'lower'")
+ if direction == "auto":
+ if metric in {"voteFraction", "topTwoMargin"}:
+ resolved_direction = "higher"
+ elif metric in {
+ "voteEntropy",
+ "referenceDistancePercentile",
+ "meanNeighborDistance",
+ }:
+ resolved_direction = "lower"
+ else:
+ raise ValueError(
+ f"Cannot infer threshold direction for metric {metric!r}; "
+ "pass direction='higher' or direction='lower'"
+ )
+ else:
+ resolved_direction = direction
+ mapping = _mapping_result(
+ store,
+ result,
+ reference=reference,
+ query_assay=query_assay,
+ )
+ assert mapping.reference is not None
+ evidence = _label_evidence(
+ store,
+ mapping,
+ reference=mapping.reference,
+ reference_class_group=reference_class_group,
+ query_assay=None,
+ threshold_fraction=0.0,
+ na_val=na_val,
+ max_distance=max_distance,
+ )
+ if metric not in evidence:
+ raise KeyError(f"Evidence has no threshold metric {metric!r}")
+ known = np.asarray(known_labels, dtype=object)
+ if known.ndim != 1 or len(known) != len(evidence):
+ raise ValueError("known_labels must have one value per mapped cell")
+ metric_values = pd.to_numeric(
+ evidence[metric],
+ errors="coerce",
+ ).to_numpy(dtype=np.float64)
+ predicted = evidence["label"].to_numpy(dtype=object)
+ valid = pd.notna(known) & np.isfinite(metric_values)
+ values = metric_values[valid]
+ correct = predicted[valid] == known[valid]
+ correct_all = np.zeros(len(evidence), dtype=bool)
+ correct_all[valid] = correct
+ informative = ~evidence.loc[valid, "isUnknown"].to_numpy(dtype=bool)
+ if len(values) == 0:
+ raise ValueError("No finite metric values with known labels")
+ textual_matches = np.fromiter(
+ (
+ str(known_value) == str(predicted_value)
+ for known_value, predicted_value in zip(
+ known[valid],
+ predicted[valid],
+ strict=True,
+ )
+ ),
+ dtype=bool,
+ count=len(values),
+ )
+ if np.any(textual_matches & ~correct):
+ raise ValueError(
+ "Some known_labels only equal their transferred labels after text "
+ "conversion. Convert known_labels to the value type stored "
+ f"in {reference_class_group!r}"
+ )
+ if thresholds is None:
+ resolved_thresholds = np.unique(
+ np.quantile(values, np.linspace(0, 1, n_thresholds))
+ )
+ else:
+ resolved_thresholds = np.unique(np.asarray(thresholds, dtype=np.float64))
+ if (
+ resolved_thresholds.ndim != 1
+ or len(resolved_thresholds) == 0
+ or not np.isfinite(resolved_thresholds).all()
+ ):
+ raise ValueError("thresholds must contain finite numeric values")
+ if chosen_threshold is not None:
+ if not np.isfinite(chosen_threshold):
+ raise ValueError("chosen_threshold must be finite")
+ resolved_thresholds = np.unique(
+ np.append(resolved_thresholds, chosen_threshold)
+ )
+ rows: list[dict[str, float | int]] = []
+ z_value = 1.959963984540054
+ for threshold in resolved_thresholds:
+ passes = (
+ values >= threshold
+ if resolved_direction == "higher"
+ else values <= threshold
+ )
+ accepted = informative & passes
+ n_accepted = int(accepted.sum())
+ if n_accepted == 0:
+ continue
+ accuracy = float(np.mean(correct[accepted]))
+ denominator = 1 + z_value**2 / n_accepted
+ center = (accuracy + z_value**2 / (2 * n_accepted)) / denominator
+ margin = (
+ z_value
+ * np.sqrt(
+ accuracy * (1 - accuracy) / n_accepted
+ + z_value**2 / (4 * n_accepted**2)
+ )
+ / denominator
+ )
+ rows.append(
+ {
+ "threshold": float(threshold),
+ "coverage": n_accepted / len(values),
+ "accuracy": accuracy,
+ "accuracyLower": max(0.0, center - margin),
+ "accuracyUpper": min(1.0, center + margin),
+ "nAccepted": n_accepted,
+ "nEvaluated": len(values),
+ }
+ )
+ calibration = pd.DataFrame(rows)
+ if calibration.empty:
+ raise ValueError("No threshold retained any mapped cells")
+ calibration = calibration.sort_values(
+ ["coverage", "threshold"],
+ kind="stable",
+ ).reset_index(drop=True)
+ if (
+ chosen_threshold is not None
+ and not np.isclose(
+ calibration["threshold"],
+ chosen_threshold,
+ ).any()
+ ):
+ warnings.warn(
+ f"chosen_threshold={chosen_threshold:g} retained no mapped cells; "
+ "the threshold marker was omitted",
+ RuntimeWarning,
+ stacklevel=2,
+ )
+ with theme_context(theme):
+ figure, axes, owns = normalize_axes_target(
+ target,
+ panel_keys=["mapping_calibration"],
+ figsize=figsize or ((4.5, 4.0) if target is None else None),
+ )
+ ax = axes["mapping_calibration"]
+ coverage = calibration["coverage"].to_numpy(dtype=np.float64)
+ accuracy = calibration["accuracy"].to_numpy(dtype=np.float64)
+ lower = calibration["accuracyLower"].to_numpy(dtype=np.float64)
+ upper = calibration["accuracyUpper"].to_numpy(dtype=np.float64)
+ ax.fill_between(
+ coverage,
+ lower,
+ upper,
+ color="#2b6cb0",
+ alpha=0.16,
+ linewidth=0,
+ )
+ ax.plot(
+ coverage,
+ accuracy,
+ color="#2b6cb0",
+ linewidth=1.4,
+ )
+ if chosen_threshold is not None:
+ chosen = calibration.loc[
+ np.isclose(calibration["threshold"], chosen_threshold)
+ ]
+ if not chosen.empty:
+ selected = chosen.iloc[0]
+ ax.scatter(
+ [selected["coverage"]],
+ [selected["accuracy"]],
+ s=48,
+ color="#d1495b",
+ edgecolor=scatter_edgecolor(theme),
+ linewidth=0.5,
+ zorder=3,
+ )
+ ax.annotate(
+ f"{metric} = {chosen_threshold:g}",
+ (
+ float(selected["coverage"]),
+ float(selected["accuracy"]),
+ ),
+ xytext=(5, 5),
+ textcoords="offset points",
+ fontsize=8,
+ )
+ # Pad the unit square so curves sitting at zero or one stay visible.
+ ax.set_xlim(-0.02, 1.02)
+ ax.set_ylim(-0.02, 1.02)
+ ax.set_xlabel("Retained coverage")
+ ax.set_ylabel("Held-out label accuracy")
+ ax.set_title(f"Threshold trade-off: {metric}")
+ apply_figure_chrome(figure, theme)
+ plot_result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables={
+ "calibration": calibration,
+ "evidence": evidence.assign(
+ knownLabel=known,
+ correct=correct_all,
+ ),
+ },
+ legends=(),
+ scales=(),
+ provenance=PlotProvenance(
+ assay=mapping.ref.assay,
+ cell_key=None,
+ n_cells=int(valid.sum()),
+ renderer="matplotlib",
+ notes=("mapping_calibration",),
+ extras={
+ "mapping_name": mapping.mapping_name,
+ "reference_class_group": reference_class_group,
+ "metric": metric,
+ "direction": resolved_direction,
+ "n_thresholds": len(resolved_thresholds),
+ "chosen_threshold": chosen_threshold,
+ },
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ plot_result.show()
+ return plot_result
+
+
+def mapping_projection(
+ store: Any,
+ result: MappingResult | ArtifactRef | str,
+ *,
+ reference: MappingReference | None = None,
+ reference_layout_key: str,
+ reference_groups: str | Sequence[Any] | np.ndarray | None = None,
+ target_groups: Sequence[Any] | np.ndarray | None = None,
+ reference_class_group: str | None = None,
+ threshold_fraction: float = 0.5,
+ na_val: str = "NA",
+ query_assay: str | None = None,
+ ref_name: str = "reference",
+ reference_mode: Literal["background", "groups"] = "background",
+ categorical_scale: CategoricalScale | None = None,
+ point_size: float | None = None,
+ reference_alpha: float = 0.22,
+ target_alpha: float = 0.95,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+) -> PlotResult:
+ """Plot query cells projected into an unchanged reference layout.
+
+ ``reference_mode="background"`` draws the reference as a light grey cloud and
+ colours only the query. ``reference_mode="groups"`` colours both layers with
+ the same categorical palette.
+ """
+ if reference_mode not in {"background", "groups"}:
+ raise ValueError("reference_mode must be 'background' or 'groups'")
+ if not 0 <= reference_alpha <= 1 or not 0 <= target_alpha <= 1:
+ raise ValueError("point alpha values must be between zero and one")
+ mapping = _mapping_result(
+ store,
+ result,
+ reference=reference,
+ query_assay=query_assay,
+ )
+ assert mapping.reference is not None
+ reference_layout = _reference_layout(
+ mapping.reference,
+ reference_layout_key,
+ )
+ projected = _mapping_reference_embedding(
+ store,
+ mapping,
+ reference=mapping.reference,
+ reference_layout_key=reference_layout_key,
+ query_assay=None,
+ )
+ reference_x = reference_layout[:, 0]
+ reference_y = reference_layout[:, 1]
+ if reference_groups is None and reference_class_group is None:
+ reference_values = np.full(len(reference_layout), ref_name, dtype=object)
+ query_values = _external_groups(
+ target_groups,
+ len(projected),
+ default=mapping.mapping_name,
+ argument_name="target_groups",
+ )
+ else:
+ group_source = (
+ reference_class_group
+ if reference_class_group is not None
+ else reference_groups
+ )
+ if isinstance(group_source, str):
+ reference_values = np.asarray(
+ mapping.reference.fetch_cell_column(group_source),
+ dtype=object,
+ )
+ if reference_values.shape != (len(reference_layout),):
+ raise ValueError(
+ "Reference labels must contain one value per reference cell"
+ )
+ if target_groups is None:
+ evidence = _label_evidence(
+ store,
+ mapping,
+ reference=mapping.reference,
+ reference_class_group=group_source,
+ query_assay=None,
+ threshold_fraction=threshold_fraction,
+ na_val=na_val,
+ max_distance=None,
+ )
+ query_values = evidence["label"].to_numpy(dtype=object)
+ else:
+ query_values = _external_groups(
+ target_groups,
+ len(projected),
+ default=mapping.mapping_name,
+ argument_name="target_groups",
+ )
+ else:
+ reference_values = np.asarray(group_source, dtype=object)
+ if reference_values.shape != (len(reference_layout),):
+ raise ValueError(
+ "reference_groups must have one value per reference cell"
+ )
+ query_values = _external_groups(
+ target_groups,
+ len(projected),
+ default=mapping.mapping_name,
+ argument_name="target_groups",
+ )
+ missing_label = (
+ categorical_scale.missing_label if categorical_scale is not None else "NA"
+ )
+ reference_values = np.asarray(
+ [missing_label if pd.isna(value) else value for value in reference_values],
+ dtype=object,
+ )
+ query_values = np.asarray(
+ [missing_label if pd.isna(value) else value for value in query_values],
+ dtype=object,
+ )
+ legend_values = (
+ query_values
+ if reference_mode == "background"
+ else np.concatenate((reference_values, query_values))
+ )
+ order, palette, resolved_scale = _categorical_contract(
+ legend_values,
+ categorical_scale,
+ )
+ combined_x = np.concatenate((reference_x, projected[:, 0]))
+ combined_y = np.concatenate((reference_y, projected[:, 1]))
+ xlim, ylim = _axis_limits(combined_x, combined_y)
+ reference_finite = np.isfinite(reference_layout).all(axis=1)
+ query_finite = np.isfinite(projected).all(axis=1)
+ size = (
+ float(point_size)
+ if point_size is not None
+ else default_point_size(len(combined_x), panel_area=6.5 * 5.0)
+ )
+ background_color = (
+ resolved_scale.missing_color if resolved_scale.missing_color else "#d0d0d0"
+ )
+ with theme_context(theme):
+ figure, axes, owns = normalize_axes_target(
+ target,
+ panel_keys=["mapping_projection"],
+ figsize=figsize or ((6.5, 5.0) if target is None else None),
+ )
+ ax = axes["mapping_projection"]
+ if reference_mode == "background":
+ ax.scatter(
+ reference_x[reference_finite],
+ reference_y[reference_finite],
+ c=background_color,
+ s=size * 0.7,
+ alpha=reference_alpha,
+ edgecolors="none",
+ rasterized=int(reference_finite.sum()) >= 50_000,
+ zorder=0,
+ )
+ else:
+ ax.scatter(
+ reference_x[reference_finite],
+ reference_y[reference_finite],
+ c=[palette[value] for value in reference_values[reference_finite]],
+ s=size,
+ alpha=reference_alpha,
+ edgecolors="none",
+ rasterized=int(reference_finite.sum()) >= 50_000,
+ zorder=0,
+ )
+ ax.scatter(
+ projected[query_finite, 0],
+ projected[query_finite, 1],
+ c=[palette[value] for value in query_values[query_finite]],
+ s=size * 1.45,
+ alpha=target_alpha,
+ edgecolors="none",
+ rasterized=int(query_finite.sum()) >= 50_000,
+ zorder=1,
+ )
+ finish_embedding_axes(
+ ax,
+ xlim=xlim,
+ ylim=ylim,
+ title=f"{mapping.mapping_name} on {reference_layout_key}",
+ frame="minimal",
+ )
+ if show_legend:
+ handles = []
+ if reference_mode == "background":
+ handles.append(
+ require_matplotlib()[1].lines.Line2D(
+ [],
+ [],
+ marker="o",
+ linestyle="",
+ markerfacecolor=background_color,
+ markeredgecolor="none",
+ markersize=5,
+ label=ref_name,
+ )
+ )
+ handles.extend(
+ require_matplotlib()[1].lines.Line2D(
+ [],
+ [],
+ marker="o",
+ linestyle="",
+ markerfacecolor=palette[value],
+ markeredgecolor="none",
+ markersize=5,
+ label=(
+ resolved_scale.labels.get(value, str(value))
+ if resolved_scale.labels is not None
+ else str(value)
+ ),
+ )
+ for value in order
+ )
+ if owns:
+ figure.legend(
+ handles=handles,
+ frameon=False,
+ loc="outside right center",
+ title="Cells",
+ )
+ else:
+ ax.legend(
+ handles=handles,
+ frameon=False,
+ loc="upper left",
+ bbox_to_anchor=(1.02, 1),
+ title="Cells",
+ )
+ apply_figure_chrome(figure, theme)
+ all_groups = np.concatenate((reference_values, query_values))
+ cells = pd.DataFrame(
+ {
+ "source": np.concatenate(
+ (
+ np.full(len(reference_layout), "reference", dtype=object),
+ np.full(len(projected), "query", dtype=object),
+ )
+ ),
+ "group": all_groups,
+ "x": combined_x,
+ "y": combined_y,
+ }
+ )
+ plot_result = PlotResult(
+ figure=figure,
+ axes=axes,
+ tables={"cells": cells},
+ legends=(LegendSpec(kind="categorical", label="Cells"),),
+ scales=(resolved_scale,),
+ provenance=PlotProvenance(
+ assay=mapping.ref.assay,
+ cell_key=None,
+ n_cells=len(all_groups),
+ renderer="matplotlib",
+ notes=("mapping_projection", reference_mode),
+ extras={
+ "mapping_name": mapping.mapping_name,
+ "reference_layout_key": reference_layout_key,
+ "n_reference": len(reference_layout),
+ "n_query": len(projected),
+ "reference_class_group": reference_class_group,
+ "threshold_fraction": threshold_fraction,
+ "reference_mode": reference_mode,
+ },
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ plot_result.show()
+ return plot_result
diff --git a/scarf/plotting/recipes.py b/scarf/plotting/recipes.py
new file mode 100644
index 00000000..043fb05e
--- /dev/null
+++ b/scarf/plotting/recipes.py
@@ -0,0 +1,807 @@
+"""Declarative plotting recipes and their sequential runner."""
+
+import json
+import inspect
+import re
+import tomllib
+import warnings as warning_module
+from collections.abc import Mapping, Sequence
+from dataclasses import dataclass, field, fields
+from importlib import import_module
+from pathlib import Path, PurePosixPath, PureWindowsPath
+from types import MappingProxyType
+from typing import Any
+
+from ..utils.logging import logger
+from ._contracts import (
+ CategoricalScale,
+ CellField,
+ ColorScale,
+ DensityOverlay,
+ FeatureRef,
+ Highlight,
+ NormalizationSpec,
+ SizeScale,
+ StudyDesign,
+)
+
+
+ALLOWED_PLOTS = frozenset(
+ {
+ "cluster_connectivity",
+ "cluster_tree",
+ "composition",
+ "distribution",
+ "dotplot",
+ "embedding",
+ "embedding_raster",
+ "marker_heatmap",
+ "mapping_calibration",
+ "mapping_confusion",
+ "mapping_evidence",
+ "mapping_projection",
+ "mapping_score",
+ "matrixplot",
+ "pseudotime_heatmap",
+ }
+)
+ALLOWED_OUTPUT_FORMATS = frozenset({"pdf", "png", "svg", "tif", "tiff"})
+
+_RECIPE_KEYS = frozenset({"steps"})
+_STEP_KEYS = frozenset(
+ {
+ "name",
+ "plot",
+ "kwargs",
+ "artifactKwargs",
+ "target",
+ "output",
+ "outputFilename",
+ }
+)
+_TARGET_KEYS = frozenset({"panel"})
+_OUTPUT_KEYS = frozenset({"filename", "dpi", "transparent", "exactSize"})
+_LOWER_CAMEL_KEY = re.compile(r"^[a-z][A-Za-z0-9]*$")
+
+
+def _validate_keys(
+ value: Mapping[Any, Any],
+ allowed: frozenset[str],
+ *,
+ context: str,
+) -> None:
+ unknown = [key for key in value if not isinstance(key, str) or key not in allowed]
+ if unknown:
+ names = ", ".join(sorted(repr(key) for key in unknown))
+ raise ValueError(f"Unknown {context} key(s): {names}")
+
+
+def _validate_output_filename(value: str) -> None:
+ if not isinstance(value, str):
+ raise TypeError("outputFilename must be a string")
+ if not value.strip():
+ raise ValueError("outputFilename must be non-empty")
+ if "\x00" in value:
+ raise ValueError("outputFilename must not contain a null byte")
+
+ posix_path = PurePosixPath(value)
+ windows_path = PureWindowsPath(value)
+ if (
+ posix_path.is_absolute()
+ or windows_path.is_absolute()
+ or bool(windows_path.drive)
+ ):
+ raise ValueError("outputFilename must be a relative path")
+ if ".." in posix_path.parts or ".." in windows_path.parts:
+ raise ValueError("outputFilename must not contain parent traversal")
+
+ suffix = Path(value).suffix.lower().lstrip(".")
+ if suffix not in ALLOWED_OUTPUT_FORMATS:
+ formats = ", ".join(sorted(ALLOWED_OUTPUT_FORMATS))
+ raise ValueError(
+ f"Unsupported output format {Path(value).suffix!r}; choose from {formats}"
+ )
+
+
+def _camel_to_snake(value: str) -> str:
+ return re.sub(r"(? dict[str, Any]:
+ if not isinstance(plot_name, str) or plot_name not in ALLOWED_PLOTS:
+ choices = ", ".join(sorted(ALLOWED_PLOTS))
+ raise ValueError(f"Unknown plot {plot_name!r}; choose from {choices}")
+ if not isinstance(value, Mapping):
+ raise TypeError(f"{context} must be a mapping")
+ converted: dict[str, Any] = {}
+ for key, item in value.items():
+ if not isinstance(key, str) or _LOWER_CAMEL_KEY.fullmatch(key) is None:
+ raise ValueError(f"{context} keys must use lower camelCase")
+ python_key = _camel_to_snake(key)
+ if python_key in converted:
+ raise ValueError(f"{context} contains duplicate key {python_key!r}")
+ converted[python_key] = item
+
+ plot = _resolve_plot(plot_name)
+ signature = inspect.signature(plot)
+ accepts_extra = any(
+ parameter.kind == inspect.Parameter.VAR_KEYWORD
+ for parameter in signature.parameters.values()
+ )
+ if not accepts_extra:
+ allowed = set(signature.parameters) - {"store"}
+ unknown = sorted(set(converted) - allowed)
+ if unknown:
+ raise ValueError(
+ f"Unknown kwargs for plot {plot_name!r}: "
+ + ", ".join(map(repr, unknown))
+ )
+ return converted
+
+
+def _contract_from_mapping(
+ contract: type[Any],
+ value: Any,
+ *,
+ context: str,
+) -> Any:
+ if isinstance(value, contract):
+ return value
+ if not isinstance(value, Mapping):
+ raise TypeError(f"{context} must be a mapping")
+ converted: dict[str, Any] = {}
+ for key, item in value.items():
+ if not isinstance(key, str) or _LOWER_CAMEL_KEY.fullmatch(key) is None:
+ raise ValueError(f"{context} keys must use lower camelCase")
+ converted[_camel_to_snake(key)] = item
+ allowed = {item.name for item in fields(contract)}
+ unknown = sorted(set(converted) - allowed)
+ if unknown:
+ raise ValueError(f"Unknown {context} key(s): " + ", ".join(map(repr, unknown)))
+ tuple_fields: dict[type[Any], tuple[str, ...]] = {
+ CategoricalScale: ("order",),
+ ColorScale: ("quantiles",),
+ DensityOverlay: ("levels", "groups"),
+ Highlight: ("groups", "indices"),
+ }
+ for name in tuple_fields.get(contract, ()):
+ item = converted.get(name)
+ if item is not None and not isinstance(item, int):
+ converted[name] = tuple(item)
+ try:
+ return contract(**converted)
+ except (TypeError, ValueError) as error:
+ raise type(error)(f"Invalid {context}: {error}") from error
+
+
+def _coerce_feature_reference(value: Any, *, context: str) -> Any:
+ if isinstance(value, Mapping):
+ return _contract_from_mapping(FeatureRef, value, context=context)
+ return value
+
+
+def _coerce_serialized_kwargs(kwargs: dict[str, Any]) -> dict[str, Any]:
+ out = dict(kwargs)
+ contract_kwargs: dict[str, type[Any]] = {
+ "categorical_scale": CategoricalScale,
+ "color_scale": ColorScale,
+ "density_overlay": DensityOverlay,
+ "highlight": Highlight,
+ "normalization": NormalizationSpec,
+ "size_scale": SizeScale,
+ "study_design": StudyDesign,
+ }
+ for name, contract in contract_kwargs.items():
+ if name in out and out[name] is not None:
+ out[name] = _contract_from_mapping(
+ contract,
+ out[name],
+ context=name,
+ )
+ if "color_by" in out:
+ color_by = out["color_by"]
+
+ def coerce_color(value: Any) -> Any:
+ if not isinstance(value, Mapping):
+ return value
+ if "key" in value:
+ return _contract_from_mapping(CellField, value, context="colorBy")
+ return _contract_from_mapping(FeatureRef, value, context="colorBy")
+
+ if isinstance(color_by, Sequence) and not isinstance(color_by, str | bytes):
+ out["color_by"] = [coerce_color(value) for value in color_by]
+ else:
+ out["color_by"] = coerce_color(color_by)
+ if "features" in out:
+ feature_value = out["features"]
+ if isinstance(feature_value, Mapping):
+ out["features"] = {
+ group: [
+ _coerce_feature_reference(
+ value,
+ context=f"features.{group}",
+ )
+ for value in values
+ ]
+ for group, values in feature_value.items()
+ }
+ elif isinstance(feature_value, Sequence) and not isinstance(
+ feature_value,
+ str | bytes,
+ ):
+ out["features"] = [
+ _coerce_feature_reference(value, context="features")
+ for value in feature_value
+ ]
+ return out
+
+
+@dataclass(frozen=True, slots=True)
+class PlotPanelTarget:
+ """Reference an axes supplied to the recipe runner by panel name."""
+
+ panel: str
+
+ def __post_init__(self) -> None:
+ if not isinstance(self.panel, str) or not self.panel.strip():
+ raise ValueError("Plot panel target must be a non-empty string")
+
+
+@dataclass(frozen=True, slots=True)
+class PlotOutputSettings:
+ """Per-step export settings resolved below the runner output directory."""
+
+ filename: str
+ dpi: int | None = None
+ transparent: bool = False
+ exact_size: bool = True
+
+ def __post_init__(self) -> None:
+ _validate_output_filename(self.filename)
+ if self.dpi is not None and (
+ isinstance(self.dpi, bool) or not isinstance(self.dpi, int) or self.dpi <= 0
+ ):
+ raise ValueError("Plot output dpi must be a positive integer or None")
+ if not isinstance(self.transparent, bool) or not isinstance(
+ self.exact_size,
+ bool,
+ ):
+ raise TypeError("Plot output flags must be boolean")
+
+
+@dataclass(frozen=True, slots=True)
+class PlotStep:
+ """One named plotting call in a recipe."""
+
+ name: str
+ plot: str
+ kwargs: Mapping[str, Any] = field(default_factory=dict)
+ artifact_kwargs: Mapping[str, str] = field(default_factory=dict)
+ target: PlotPanelTarget | None = None
+ output: PlotOutputSettings | None = None
+ output_filename: str | None = None
+
+ def __post_init__(self) -> None:
+ if not isinstance(self.name, str):
+ raise TypeError("Plot step name must be a string")
+ if not self.name.strip():
+ raise ValueError("Plot step name must be non-empty")
+ if not isinstance(self.plot, str):
+ raise TypeError("Plot step plot must be a string")
+ if self.plot not in ALLOWED_PLOTS:
+ choices = ", ".join(sorted(ALLOWED_PLOTS))
+ raise ValueError(f"Unknown plot {self.plot!r}; choose from {choices}")
+ if not isinstance(self.kwargs, Mapping):
+ raise TypeError(f"kwargs for step {self.name!r} must be a mapping")
+ if any(not isinstance(key, str) for key in self.kwargs):
+ raise TypeError(f"kwargs keys for step {self.name!r} must be strings")
+ object.__setattr__(
+ self,
+ "kwargs",
+ MappingProxyType(dict(self.kwargs)),
+ )
+ if not isinstance(self.artifact_kwargs, Mapping):
+ raise TypeError(f"artifact_kwargs for step {self.name!r} must be a mapping")
+ if any(
+ not isinstance(key, str)
+ or not isinstance(value, str)
+ or not key
+ or not value
+ for key, value in self.artifact_kwargs.items()
+ ):
+ raise TypeError(
+ "artifact_kwargs must map non-empty plot keyword names "
+ "to non-empty artifact names"
+ )
+ overlap = set(self.kwargs).intersection(self.artifact_kwargs)
+ if overlap:
+ raise ValueError(
+ "kwargs and artifact_kwargs cannot both define: "
+ + ", ".join(sorted(overlap))
+ )
+ object.__setattr__(
+ self,
+ "artifact_kwargs",
+ MappingProxyType(dict(self.artifact_kwargs)),
+ )
+ if self.target is not None and not isinstance(self.target, PlotPanelTarget):
+ raise TypeError("target must be a PlotPanelTarget or None")
+ if self.output is not None and not isinstance(
+ self.output,
+ PlotOutputSettings,
+ ):
+ raise TypeError("output must be PlotOutputSettings or None")
+ if self.output is not None and self.output_filename is not None:
+ raise ValueError("Set output or output_filename, not both")
+ if self.output_filename is not None:
+ _validate_output_filename(self.output_filename)
+ object.__setattr__(
+ self,
+ "output",
+ PlotOutputSettings(filename=self.output_filename),
+ )
+ elif self.output is not None:
+ object.__setattr__(self, "output_filename", self.output.filename)
+
+
+@dataclass(frozen=True, slots=True)
+class PlotRecipe:
+ """An ordered collection of plotting steps."""
+
+ steps: tuple[PlotStep, ...]
+
+ def __post_init__(self) -> None:
+ if not isinstance(self.steps, Sequence) or isinstance(
+ self.steps, (str, bytes, bytearray)
+ ):
+ raise TypeError("Plot recipe steps must be an ordered sequence")
+ steps = tuple(self.steps)
+ if any(not isinstance(step, PlotStep) for step in steps):
+ raise TypeError("Every recipe step must be a PlotStep")
+ if not steps:
+ raise ValueError("Plot recipe must contain at least one step")
+ names = [step.name for step in steps]
+ duplicates = sorted({name for name in names if names.count(name) > 1})
+ if duplicates:
+ joined = ", ".join(repr(name) for name in duplicates)
+ raise ValueError(f"Plot step names must be unique; duplicates: {joined}")
+ output_names = [
+ step.output_filename for step in steps if step.output_filename is not None
+ ]
+ duplicate_outputs = sorted(
+ {name for name in output_names if output_names.count(name) > 1}
+ )
+ if duplicate_outputs:
+ raise ValueError(
+ "Plot output filenames must be unique; duplicates: "
+ + ", ".join(map(repr, duplicate_outputs))
+ )
+ object.__setattr__(self, "steps", steps)
+
+ @classmethod
+ def from_dict(cls, value: Mapping[str, Any]) -> "PlotRecipe":
+ if not isinstance(value, Mapping):
+ raise TypeError("Plot recipe configuration must be a mapping")
+ _validate_keys(value, _RECIPE_KEYS, context="plot recipe")
+ if "steps" not in value:
+ raise ValueError("Plot recipe configuration requires 'steps'")
+
+ raw_steps = value["steps"]
+ if not isinstance(raw_steps, Sequence) or isinstance(
+ raw_steps, (str, bytes, bytearray)
+ ):
+ raise TypeError("Plot recipe 'steps' must be an ordered sequence")
+
+ steps: list[PlotStep] = []
+ for index, raw_step in enumerate(raw_steps):
+ if not isinstance(raw_step, Mapping):
+ raise TypeError(f"Plot recipe step {index} must be a mapping")
+ _validate_keys(raw_step, _STEP_KEYS, context=f"plot step {index}")
+ missing = {"name", "plot"}.difference(raw_step)
+ if missing:
+ names = ", ".join(sorted(missing))
+ raise ValueError(f"Plot recipe step {index} requires: {names}")
+ plot_name = raw_step["plot"]
+ kwargs = _coerce_serialized_kwargs(
+ _external_kwargs(
+ plot_name,
+ raw_step.get("kwargs", {}),
+ context=f"kwargs for plot step {index}",
+ )
+ )
+ artifact_kwargs = _external_kwargs(
+ plot_name,
+ raw_step.get("artifactKwargs", {}),
+ context=f"artifactKwargs for plot step {index}",
+ )
+ if any(not isinstance(value, str) for value in artifact_kwargs.values()):
+ raise TypeError("artifactKwargs values must be artifact names")
+ target_value = raw_step.get("target")
+ target = None
+ if target_value is not None:
+ if not isinstance(target_value, Mapping):
+ raise TypeError("target must be a mapping")
+ _validate_keys(
+ target_value,
+ _TARGET_KEYS,
+ context=f"plot step {index} target",
+ )
+ if "panel" not in target_value:
+ raise ValueError("target requires panel")
+ target = PlotPanelTarget(panel=target_value["panel"])
+ output_value = raw_step.get("output")
+ output_filename = raw_step.get("outputFilename")
+ if output_value is not None and output_filename is not None:
+ raise ValueError("Set output or outputFilename, not both")
+ output = None
+ if output_value is not None:
+ if not isinstance(output_value, Mapping):
+ raise TypeError("output must be a mapping")
+ _validate_keys(
+ output_value,
+ _OUTPUT_KEYS,
+ context=f"plot step {index} output",
+ )
+ if "filename" not in output_value:
+ raise ValueError("output requires filename")
+ output = PlotOutputSettings(
+ filename=output_value["filename"],
+ dpi=output_value.get("dpi"),
+ transparent=output_value.get("transparent", False),
+ exact_size=output_value.get("exactSize", True),
+ )
+ steps.append(
+ PlotStep(
+ name=raw_step["name"],
+ plot=plot_name,
+ kwargs=kwargs,
+ artifact_kwargs=artifact_kwargs,
+ target=target,
+ output=output,
+ output_filename=output_filename,
+ )
+ )
+ return cls(tuple(steps))
+
+ @classmethod
+ def from_json(
+ cls,
+ source: str | bytes | bytearray | Path,
+ ) -> "PlotRecipe":
+ raw = _read_json_source(source)
+ value = json.loads(
+ raw,
+ object_pairs_hook=_strict_json_object,
+ parse_constant=_reject_json_constant,
+ )
+ return cls.from_dict(value)
+
+ @classmethod
+ def from_toml(cls, source: str | bytes | Path) -> "PlotRecipe":
+ if isinstance(source, Path):
+ with source.open("rb") as stream:
+ value = tomllib.load(stream)
+ else:
+ raw = source.decode("utf-8") if isinstance(source, bytes) else source
+ path = _existing_config_path(raw)
+ if path is not None:
+ with path.open("rb") as stream:
+ value = tomllib.load(stream)
+ else:
+ value = tomllib.loads(raw)
+ return cls.from_dict(value)
+
+
+@dataclass(frozen=True, slots=True)
+class PlotOutput:
+ """The successful result of one recipe step."""
+
+ name: str
+ result: Any = field(repr=False)
+ path: Path | None = None
+
+ @property
+ def step_name(self) -> str:
+ return self.name
+
+ @property
+ def written_path(self) -> Path | None:
+ return self.path
+
+
+@dataclass(frozen=True, slots=True)
+class PlotRecipeResult:
+ """Results and diagnostics from a recipe execution."""
+
+ outputs: tuple[PlotOutput, ...] = ()
+ written_paths: tuple[Path, ...] = ()
+ warnings: tuple[str, ...] = ()
+ failures: Mapping[str, str] = field(default_factory=dict)
+
+ def __post_init__(self) -> None:
+ object.__setattr__(self, "outputs", tuple(self.outputs))
+ object.__setattr__(self, "written_paths", tuple(self.written_paths))
+ object.__setattr__(self, "warnings", tuple(self.warnings))
+ object.__setattr__(
+ self,
+ "failures",
+ MappingProxyType(dict(self.failures)),
+ )
+
+ @property
+ def results(self) -> tuple[Any, ...]:
+ return tuple(output.result for output in self.outputs)
+
+
+def _strict_json_object(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
+ value: dict[str, Any] = {}
+ for key, item in pairs:
+ if key in value:
+ raise ValueError(f"Duplicate JSON key: {key!r}")
+ value[key] = item
+ return value
+
+
+def _reject_json_constant(value: str) -> Any:
+ raise ValueError(f"Invalid JSON constant: {value}")
+
+
+def _existing_config_path(value: str) -> Path | None:
+ if "\n" in value or "\r" in value:
+ return None
+ try:
+ path = Path(value)
+ return path if path.is_file() else None
+ except OSError:
+ return None
+
+
+def _read_json_source(
+ source: str | bytes | bytearray | Path,
+) -> str | bytes | bytearray:
+ if isinstance(source, Path):
+ return source.read_text(encoding="utf-8")
+ if isinstance(source, str):
+ path = _existing_config_path(source)
+ if path is not None:
+ return path.read_text(encoding="utf-8")
+ return source
+
+
+def _load_recipe(recipe: PlotRecipe | str | Path) -> PlotRecipe:
+ if isinstance(recipe, PlotRecipe):
+ return recipe
+ if not isinstance(recipe, (str, Path)):
+ raise TypeError("recipe must be a PlotRecipe or a JSON/TOML path")
+ path = Path(recipe)
+ suffix = path.suffix.lower()
+ if suffix == ".json":
+ return PlotRecipe.from_json(path)
+ if suffix == ".toml":
+ return PlotRecipe.from_toml(path)
+ raise ValueError("Recipe path must use a .json or .toml extension")
+
+
+def _resolve_plot(name: str) -> Any:
+ plotting = import_module("scarf.plotting")
+ try:
+ plot = getattr(plotting, name)
+ except AttributeError as error:
+ raise RuntimeError(
+ f"Plot {name!r} is not available in this Scarf build"
+ ) from error
+ if not callable(plot):
+ raise RuntimeError(f"Plot {name!r} is not callable")
+ return plot
+
+
+def _safe_output_path(output_dir: Path, filename: str) -> Path:
+ path = output_dir / filename
+ root = output_dir.resolve()
+ resolved = path.resolve()
+ if not resolved.is_relative_to(root):
+ raise ValueError(f"Output path for {filename!r} escapes output_dir")
+ return path
+
+
+def _compact_failure(error: Exception) -> str:
+ detail = " ".join(str(error).split())
+ return f"{type(error).__name__}: {detail}" if detail else type(error).__name__
+
+
+def _log_failure(step_name: str, failure: str) -> None:
+ logger.exception(f"Plot recipe step {step_name!r} failed: {failure}")
+
+
+def _run_step(
+ store: Any,
+ step: PlotStep,
+ *,
+ output_path: Path | None,
+ show: bool,
+ artifacts: Mapping[str, Any] | None,
+ targets: Mapping[str, Any] | None,
+) -> PlotOutput:
+ plot = _resolve_plot(step.plot)
+ kwargs = dict(step.kwargs)
+ if step.artifact_kwargs:
+ if artifacts is None:
+ raise ValueError(f"Recipe step {step.name!r} requires an artifact mapping")
+ missing = [
+ artifact_name
+ for artifact_name in step.artifact_kwargs.values()
+ if artifact_name not in artifacts
+ ]
+ if missing:
+ raise KeyError(
+ f"Recipe step {step.name!r} is missing artifacts: "
+ + ", ".join(map(repr, missing))
+ )
+ kwargs.update(
+ {
+ plot_keyword: artifacts[artifact_name]
+ for plot_keyword, artifact_name in step.artifact_kwargs.items()
+ }
+ )
+ if step.target is not None:
+ if "target" in kwargs:
+ raise ValueError(f"Recipe step {step.name!r} defines target in two places")
+ if targets is None or step.target.panel not in targets:
+ raise KeyError(
+ f"Recipe step {step.name!r} requires panel "
+ f"{step.target.panel!r} in targets"
+ )
+ kwargs["target"] = targets[step.target.panel]
+ kwargs["show"] = False
+ plot_result = plot(store, **kwargs)
+ should_close = output_path is not None or show
+
+ try:
+ if output_path is not None:
+ assert step.output is not None
+ plot_result.save(
+ output_path,
+ dpi=step.output.dpi,
+ transparent=step.output.transparent,
+ exact_size=step.output.exact_size,
+ )
+ if show:
+ plot_result.show()
+ except Exception as error:
+ if should_close and getattr(plot_result, "owns_figure", False):
+ try:
+ plot_result.close()
+ except Exception as close_error:
+ error.add_note(
+ "The plot also failed to close cleanly: "
+ f"{type(close_error).__name__}: {close_error}"
+ )
+ raise
+
+ if should_close and getattr(plot_result, "owns_figure", False):
+ plot_result.close()
+ return PlotOutput(name=step.name, result=plot_result, path=output_path)
+
+
+def run_plot_recipe(
+ store: Any,
+ recipe: PlotRecipe | str | Path,
+ *,
+ artifacts: Mapping[str, Any] | None = None,
+ targets: Mapping[str, Any] | None = None,
+ output_dir: str | Path | None = None,
+ show: bool = False,
+ continue_on_error: bool = False,
+) -> PlotRecipeResult:
+ """Run recipe steps in order without allowing plots to display themselves."""
+
+ resolved_recipe = _load_recipe(recipe)
+ if not isinstance(show, bool):
+ raise TypeError("show must be a boolean")
+ if not isinstance(continue_on_error, bool):
+ raise TypeError("continue_on_error must be a boolean")
+ if artifacts is not None and not isinstance(artifacts, Mapping):
+ raise TypeError("artifacts must be a mapping or None")
+ if targets is not None and not isinstance(targets, Mapping):
+ raise TypeError("targets must be a mapping or None")
+
+ has_outputs = any(
+ step.output_filename is not None for step in resolved_recipe.steps
+ )
+ if has_outputs and output_dir is None:
+ raise ValueError("output_dir is required when a step declares outputFilename")
+
+ output_paths: dict[str, Path] = {}
+ if has_outputs:
+ assert output_dir is not None
+ output_root = Path(output_dir)
+ output_root.mkdir(parents=True, exist_ok=True)
+ output_paths = {
+ step.name: _safe_output_path(output_root, step.output_filename)
+ for step in resolved_recipe.steps
+ if step.output_filename is not None
+ }
+
+ outputs: list[PlotOutput] = []
+ written_paths: list[Path] = []
+ captured_warnings: list[str] = []
+ failures: dict[str, str] = {}
+
+ for step in resolved_recipe.steps:
+ recorded: list[warning_module.WarningMessage] = []
+ try:
+ with warning_module.catch_warnings(record=True) as recorded:
+ warning_module.simplefilter("always")
+ output = _run_step(
+ store,
+ step,
+ output_path=output_paths.get(step.name),
+ show=show,
+ artifacts=artifacts,
+ targets=targets,
+ )
+ except Exception as error:
+ captured_warnings.extend(
+ f"{step.name}: {warning.message}" for warning in recorded
+ )
+ if not continue_on_error:
+ raise
+ failure = _compact_failure(error)
+ failures[step.name] = failure
+ _log_failure(step.name, failure)
+ continue
+
+ captured_warnings.extend(
+ f"{step.name}: {warning.message}" for warning in recorded
+ )
+ outputs.append(output)
+ if output.path is not None:
+ written_paths.append(output.path)
+
+ return PlotRecipeResult(
+ outputs=tuple(outputs),
+ written_paths=tuple(written_paths),
+ warnings=tuple(captured_warnings),
+ failures=failures,
+ )
+
+
+def run_recipe(
+ store: Any,
+ recipe: PlotRecipe | str | Path,
+ *,
+ artifacts: Mapping[str, Any] | None = None,
+ targets: Mapping[str, Any] | None = None,
+ output_dir: str | Path | None = None,
+ show: bool = False,
+ continue_on_error: bool = False,
+) -> PlotRecipeResult:
+ """Run a plotting recipe after any analysis pipeline has completed."""
+ return run_plot_recipe(
+ store,
+ recipe,
+ artifacts=artifacts,
+ targets=targets,
+ output_dir=output_dir,
+ show=show,
+ continue_on_error=continue_on_error,
+ )
+
+
+__all__ = [
+ "ALLOWED_OUTPUT_FORMATS",
+ "ALLOWED_PLOTS",
+ "PlotOutput",
+ "PlotOutputSettings",
+ "PlotPanelTarget",
+ "PlotRecipe",
+ "PlotRecipeResult",
+ "PlotStep",
+ "run_recipe",
+ "run_plot_recipe",
+]
diff --git a/scarf/plotting/summary.py b/scarf/plotting/summary.py
new file mode 100644
index 00000000..feb1c4ec
--- /dev/null
+++ b/scarf/plotting/summary.py
@@ -0,0 +1,959 @@
+"""Dotplot and matrixplot."""
+
+from collections.abc import Mapping, Sequence
+import textwrap
+from typing import Any, Hashable
+
+import numpy as np
+import pandas as pd
+
+from ._contracts import (
+ CategoricalScale,
+ ColorScale,
+ FeatureRef,
+ NormalizationSpec,
+ PlotProvenance,
+ SizeScale,
+ StudyDesign,
+)
+from ._data import coerce_feature_list, resolve_feature, summarize_features_by_group
+from ._deps import require_matplotlib
+from ._display import resolve_categorical_scale
+from ._figure import LegendSpec, PlotResult, normalize_axes_target
+from ._heatmap_utils import (
+ annotation_colors,
+ draw_annotation_strips,
+ normalize_annotations,
+ order_heatmap,
+)
+from ._style import (
+ apply_figure_chrome,
+ capped_figsize,
+ continuous_norm,
+ scatter_edgecolor,
+ sort_categories,
+ theme_context,
+)
+
+
+def _wrap_tick_labels(values: Sequence[Any], width: int | None) -> list[str]:
+ labels = [str(value) for value in values]
+ if width is None:
+ return labels
+ if width < 1:
+ raise ValueError("label_wrap must be positive or None")
+ return [textwrap.fill(label, width=width) for label in labels]
+
+
+def _default_dot_size_scale(ax: Any, *, n_x: int, n_y: int) -> SizeScale:
+ """Fit default marker areas to the physical dot-grid cells."""
+ bounds = ax.get_position()
+ width_points = max(float(bounds.width * ax.figure.get_figwidth() * 72), 1.0)
+ height_points = max(float(bounds.height * ax.figure.get_figheight() * 72), 1.0)
+ slot_points = min(
+ width_points / max(n_x, 1),
+ height_points / max(n_y, 1),
+ )
+ maximum_diameter = float(np.clip(0.72 * slot_points, 2.5, 26.0))
+ minimum_diameter = float(np.clip(0.18 * maximum_diameter, 1.0, 3.0))
+ return SizeScale(
+ size_min=minimum_diameter**2,
+ size_max=maximum_diameter**2,
+ )
+
+
+def _draw_feature_group_brackets(
+ ax: Any,
+ feature_order: list[str],
+ feature_groups: Mapping[str, Any],
+ *,
+ swap_axes: bool,
+) -> int:
+ line_color = ax.xaxis.label.get_color()
+ grouped_ranges: list[tuple[str, int, int]] = []
+ start = 0
+ while start < len(feature_order):
+ group = feature_groups.get(feature_order[start])
+ end = start
+ while (
+ end + 1 < len(feature_order)
+ and feature_groups.get(feature_order[end + 1]) == group
+ ):
+ end += 1
+ if group is not None and not pd.isna(group):
+ grouped_ranges.append((str(group), start, end))
+ start = end + 1
+ left_geometry: tuple[Any, Any, float] | None = None
+ if grouped_ranges and not swap_axes:
+ from matplotlib.transforms import ScaledTranslation
+
+ ax.figure.canvas.draw()
+ renderer = ax.figure.canvas.get_renderer()
+ axes_width = max(float(ax.get_window_extent(renderer).width), 1.0)
+ label_width = max(
+ (
+ float(tick.get_window_extent(renderer).width)
+ for tick in ax.get_yticklabels()
+ if tick.get_visible() and tick.get_text()
+ ),
+ default=0.0,
+ )
+ bracket_transform = ax.get_yaxis_transform() + ScaledTranslation(
+ -(label_width / ax.figure.dpi + 16.0 / 72.0),
+ 0,
+ ax.figure.dpi_scale_trans,
+ )
+ text_transform = bracket_transform + ScaledTranslation(
+ -7.0 / 72.0,
+ 0,
+ ax.figure.dpi_scale_trans,
+ )
+ left_geometry = (
+ bracket_transform,
+ text_transform,
+ 4.0 * ax.figure.dpi / (72.0 * axes_width),
+ )
+ for label, start, end in grouped_ranges:
+ centre = (start + end) / 2
+ if swap_axes:
+ transform = ax.get_xaxis_transform()
+ line = ax.plot(
+ [start - 0.35, end + 0.35],
+ [1.03, 1.03],
+ transform=transform,
+ color=line_color,
+ linewidth=0.8,
+ clip_on=False,
+ )[0]
+ ax.plot(
+ [start - 0.35, start - 0.35, end + 0.35, end + 0.35],
+ [1.00, 1.03, 1.03, 1.00],
+ transform=transform,
+ color=line_color,
+ linewidth=0.8,
+ clip_on=False,
+ )
+ ax.text(
+ centre,
+ 1.07,
+ label,
+ transform=transform,
+ ha="center",
+ va="bottom",
+ fontsize=7,
+ clip_on=False,
+ )
+ else:
+ assert left_geometry is not None
+ bracket_transform, text_transform, cap_width = left_geometry
+ line = ax.plot(
+ [0, 0],
+ [start - 0.35, end + 0.35],
+ transform=bracket_transform,
+ color=line_color,
+ linewidth=0.8,
+ clip_on=False,
+ )[0]
+ ax.plot(
+ [
+ cap_width,
+ 0,
+ 0,
+ cap_width,
+ ],
+ [start - 0.35, start - 0.35, end + 0.35, end + 0.35],
+ transform=bracket_transform,
+ color=line_color,
+ linewidth=0.8,
+ clip_on=False,
+ )
+ group_label = ax.text(
+ 0,
+ centre,
+ label,
+ transform=text_transform,
+ ha="right",
+ va="center",
+ fontsize=7,
+ clip_on=False,
+ )
+ group_label.set_gid("feature-group-label")
+ line.set_gid("feature-group-bracket")
+ return len(grouped_ranges)
+
+
+def _standardize_feature(df: pd.DataFrame, value_col: str = "mean") -> pd.DataFrame:
+ out = df.copy()
+ means = out.groupby("feature", observed=False)[value_col].transform("mean")
+ stds = out.groupby("feature", observed=False)[value_col].transform("std")
+ stds = stds.replace(0, np.nan)
+ out[value_col] = (out[value_col] - means) / stds
+ return out
+
+
+def _group_axis_labels(df: pd.DataFrame, group_keys: tuple[str, ...]) -> pd.Series:
+ if len(group_keys) == 1:
+ return df[group_keys[0]].astype(str)
+ return df[list(group_keys)].astype(str).agg(" | ".join, axis=1)
+
+
+def _color_limits(values: np.ndarray, scale: ColorScale) -> tuple[float, float]:
+ finite = np.isfinite(values)
+ if finite.any():
+ if scale.quantiles is not None:
+ low, high = scale.quantiles
+ vmin = float(np.nanquantile(values[finite], low))
+ vmax = float(np.nanquantile(values[finite], high))
+ else:
+ vmin = float(np.nanmin(values[finite]))
+ vmax = float(np.nanmax(values[finite]))
+ else:
+ vmin, vmax = 0.0, 1.0
+ if scale.vmin is not None:
+ vmin = scale.vmin
+ if scale.vmax is not None:
+ vmax = scale.vmax
+ if vmax <= vmin:
+ if scale.vmin is not None or scale.vmax is not None:
+ raise ValueError("Color limits must satisfy vmin < vmax")
+ vmax = vmin + 1.0
+ return vmin, vmax
+
+
+def _sample_counts(
+ store: Any,
+ *,
+ cell_key: str,
+ sample_by: str | None,
+ study_design: StudyDesign | None,
+) -> tuple[int | None, int]:
+ sample_key = study_design.sample_by if study_design is not None else sample_by
+ if sample_key is None:
+ return None, 0
+ values = np.asarray(store.cells.fetch(sample_key, key=cell_key), dtype=object)
+ valid = pd.notna(values) & (values != "")
+ return int(pd.Series(values[valid]).nunique()), int((~valid).sum())
+
+
+def _feature_assays(
+ store: Any,
+ features: Sequence[str | FeatureRef] | Mapping[str, Sequence[str | FeatureRef]],
+ from_assay: str | None,
+) -> set[str]:
+ assays: set[str] = set()
+ for _, feature in coerce_feature_list(features):
+ assay = (
+ feature.assay
+ if isinstance(feature, FeatureRef) and feature.assay is not None
+ else from_assay or store._defaultAssay
+ )
+ assays.add(assay)
+ return assays
+
+
+def dotplot(
+ store: Any,
+ *,
+ features: Sequence[str | FeatureRef] | Mapping[str, Sequence[str | FeatureRef]],
+ group_by: str | tuple[str, ...],
+ cell_key: str = "I",
+ from_assay: str | None = None,
+ sample_by: str | None = None,
+ study_design: StudyDesign | None = None,
+ normalization: NormalizationSpec | None = None,
+ expression_cutoff: float = 0.0,
+ standardize: str = "none",
+ color_scale: ColorScale | None = None,
+ size_scale: SizeScale | None = None,
+ categorical_scale: CategoricalScale | None = None,
+ group_order: Sequence[Any] | None = None,
+ feature_order: Sequence[str] | None = None,
+ swap_axes: bool = False,
+ marker_edgecolor: str | None = None,
+ marker_linewidth: float = 0.3,
+ label_wrap: int | None = None,
+ italicize_features: bool = False,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ max_figure_width: float | None = 7.5,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+) -> PlotResult:
+ """Dotplot of expression by group.
+
+ Color is the mean value in the group. Dot size is the fraction of cells
+ above ``expression_cutoff``. Pass ``features`` as a list of genes, or as a
+ mapping of group name to gene list when you want gene-set brackets.
+
+ With ``sample_by`` (or ``study_design.sample_by``), each sample contributes
+ equally: Scarf first summarizes within sample, then averages across samples.
+ Without it, every cell contributes equally.
+ """
+ _, mpl = require_matplotlib()
+ color_scale = color_scale or ColorScale(cmap="viridis")
+ if color_scale.scale != "linear":
+ raise NotImplementedError("dotplot currently supports only linear color scales")
+ size_scale_is_explicit = size_scale is not None
+ normalization = normalization or NormalizationSpec()
+ if marker_linewidth < 0:
+ raise ValueError("marker_linewidth must be non-negative")
+ if isinstance(group_by, str):
+ categorical_scale = resolve_categorical_scale(
+ store,
+ group_by,
+ categorical_scale,
+ )
+ requested_feature_order = list(
+ dict.fromkeys(
+ resolve_feature(store, feature, from_assay=from_assay).label
+ for _, feature in coerce_feature_list(features)
+ )
+ )
+
+ aggregate, per_sample = summarize_features_by_group(
+ store,
+ features=features,
+ group_by=group_by,
+ cell_key=cell_key,
+ from_assay=from_assay,
+ sample_by=sample_by,
+ study_design=study_design,
+ normalization=normalization,
+ expression_cutoff=expression_cutoff,
+ )
+ if standardize == "feature":
+ aggregate = _standardize_feature(aggregate, "mean")
+ elif standardize not in ("none", "feature"):
+ raise ValueError("standardize must be 'none' or 'feature'")
+
+ group_keys = (group_by,) if isinstance(group_by, str) else tuple(group_by)
+ plot_df = aggregate.copy()
+ plot_df["group_label"] = _group_axis_labels(plot_df, group_keys)
+ # Preserve feature order from input
+ observed_features = [
+ value
+ for value in requested_feature_order
+ if value in set(plot_df["feature"].tolist())
+ ]
+ observed_groups = list(dict.fromkeys(plot_df["group_label"].tolist()))
+ if feature_order is None:
+ resolved_feature_order = observed_features
+ else:
+ resolved_feature_order = [str(value) for value in feature_order]
+ missing = [
+ value for value in observed_features if value not in resolved_feature_order
+ ]
+ if missing:
+ raise ValueError(
+ "feature_order is missing observed features: "
+ + ", ".join(map(str, missing[:10]))
+ )
+ if group_order is not None:
+ resolved_group_order = [str(value) for value in group_order]
+ elif categorical_scale is not None and categorical_scale.order is not None:
+ resolved_group_order = [str(value) for value in categorical_scale.order]
+ else:
+ resolved_group_order = sort_categories(observed_groups)
+ missing_groups = [
+ value for value in observed_groups if value not in resolved_group_order
+ ]
+ if missing_groups:
+ raise ValueError(
+ "group order is missing observed groups: "
+ + ", ".join(map(str, missing_groups[:10]))
+ )
+ feature_group_map = (
+ plot_df[["feature", "feature_group"]]
+ .drop_duplicates("feature")
+ .set_index("feature")["feature_group"]
+ .to_dict()
+ )
+ plot_df["feature"] = pd.Categorical(
+ plot_df["feature"], categories=resolved_feature_order, ordered=True
+ )
+ plot_df["group_label"] = pd.Categorical(
+ plot_df["group_label"], categories=resolved_group_order, ordered=True
+ )
+
+ panel_key: Hashable = "dotplot"
+ resolved_figsize = figsize
+ if resolved_figsize is None and target is None:
+ x_count = (
+ len(resolved_feature_order) if swap_axes else len(resolved_group_order)
+ )
+ y_count = (
+ len(resolved_group_order) if swap_axes else len(resolved_feature_order)
+ )
+ resolved_figsize = (
+ max(4.6, 0.42 * x_count + 2.6),
+ max(3.4, 0.34 * y_count + 2.4),
+ )
+ resolved_figsize = capped_figsize(
+ *resolved_figsize,
+ max_width=max_figure_width,
+ )
+ with theme_context(theme):
+ fig, axes, owns = normalize_axes_target(
+ target,
+ panel_keys=[panel_key],
+ figsize=resolved_figsize,
+ )
+ ax = axes[panel_key]
+ if size_scale is None:
+ size_scale = _default_dot_size_scale(
+ ax,
+ n_x=(
+ len(resolved_feature_order)
+ if swap_axes
+ else len(resolved_group_order)
+ ),
+ n_y=(
+ len(resolved_group_order)
+ if swap_axes
+ else len(resolved_feature_order)
+ ),
+ )
+
+ vals = plot_df["mean"].to_numpy(dtype=np.float64)
+ vmin, vmax = _color_limits(vals, color_scale)
+ norm = continuous_norm(
+ mpl,
+ vmin=vmin,
+ vmax=vmax,
+ vcenter=color_scale.vcenter,
+ )
+
+ areas = size_scale.areas(plot_df["fraction"].to_numpy(dtype=np.float64))
+ x = plot_df["group_label"].cat.codes.to_numpy()
+ y = plot_df["feature"].cat.codes.to_numpy()
+
+ edgecolor = marker_edgecolor or scatter_edgecolor(theme)
+ x_values = x if not swap_axes else y
+ y_values = y if not swap_axes else x
+ sc = ax.scatter(
+ x_values,
+ y_values,
+ c=vals,
+ s=areas,
+ cmap=color_scale.cmap or "viridis",
+ norm=norm,
+ edgecolors=edgecolor,
+ linewidths=marker_linewidth,
+ rasterized=False,
+ )
+ if swap_axes:
+ ax.set_xticks(range(len(resolved_feature_order)))
+ ax.set_xticklabels(
+ _wrap_tick_labels(resolved_feature_order, label_wrap),
+ rotation=45,
+ ha="right",
+ fontstyle=("italic" if italicize_features else "normal"),
+ )
+ ax.set_yticks(range(len(resolved_group_order)))
+ ax.set_yticklabels(_wrap_tick_labels(resolved_group_order, label_wrap))
+ ax.set_xlim(-0.5, len(resolved_feature_order) - 0.5)
+ ax.set_ylim(-0.5, len(resolved_group_order) - 0.5)
+ ax.set_xlabel("")
+ ax.set_ylabel(" / ".join(group_keys))
+ else:
+ ax.set_xticks(range(len(resolved_group_order)))
+ ax.set_xticklabels(
+ _wrap_tick_labels(resolved_group_order, label_wrap),
+ rotation=45,
+ ha="right",
+ )
+ ax.set_yticks(range(len(resolved_feature_order)))
+ ax.set_yticklabels(
+ _wrap_tick_labels(resolved_feature_order, label_wrap),
+ fontstyle=("italic" if italicize_features else "normal"),
+ )
+ ax.set_xlim(-0.5, len(resolved_group_order) - 0.5)
+ ax.set_ylim(-0.5, len(resolved_feature_order) - 0.5)
+ ax.invert_yaxis()
+ ax.set_xlabel(" / ".join(group_keys))
+ ax.set_ylabel("")
+ bracket_count = _draw_feature_group_brackets(
+ ax,
+ resolved_feature_order,
+ feature_group_map,
+ swap_axes=swap_axes,
+ )
+ colorbar_label = (
+ "Mean expression"
+ if standardize == "none"
+ else "Standardized mean expression"
+ )
+ if show_legend:
+ cb = fig.colorbar(
+ sc,
+ ax=ax,
+ location=("bottom" if swap_axes else "top"),
+ orientation="horizontal",
+ shrink=0.55,
+ pad=(0.08 if swap_axes else 0.04),
+ fraction=0.05,
+ )
+ cb.set_label(colorbar_label)
+ legend_values = np.array([0.25, 0.5, 0.75, 1.0])
+ legend_areas = size_scale.areas(legend_values)
+ legend_area_factor = min(
+ 1.0,
+ 180.0 / max(float(legend_areas.max()), 1.0),
+ )
+ handles = [
+ ax.scatter(
+ [],
+ [],
+ s=area * legend_area_factor,
+ facecolor="#bdbdbd",
+ edgecolor=edgecolor,
+ linewidth=marker_linewidth,
+ )
+ for area in legend_areas
+ ]
+ # Keep fraction sizes beside the colorbar, away from x tick labels.
+ if show_legend:
+ legend_kwargs = {
+ "handles": handles,
+ "labels": [f"{value:.0%}" for value in legend_values],
+ "title": "Detected cells",
+ "frameon": False,
+ "borderaxespad": 0.4,
+ "handletextpad": 0.4,
+ "labelspacing": 1.1,
+ }
+ if owns:
+ fig.legend(loc="outside right center", **legend_kwargs)
+ else:
+ ax.legend(
+ loc="upper left",
+ bbox_to_anchor=(1.02, 1),
+ **legend_kwargs,
+ )
+ apply_figure_chrome(fig, theme)
+ if not size_scale_is_explicit:
+ fig.canvas.draw()
+ size_scale = _default_dot_size_scale(
+ ax,
+ n_x=(
+ len(resolved_feature_order)
+ if swap_axes
+ else len(resolved_group_order)
+ ),
+ n_y=(
+ len(resolved_group_order)
+ if swap_axes
+ else len(resolved_feature_order)
+ ),
+ )
+ areas = size_scale.areas(plot_df["fraction"].to_numpy(dtype=np.float64))
+ sc.set_sizes(areas)
+ legend_areas = size_scale.areas(legend_values)
+ legend_area_factor = min(
+ 1.0,
+ 180.0 / max(float(legend_areas.max()), 1.0),
+ )
+ for handle, area in zip(handles, legend_areas, strict=True):
+ handle.set_sizes([area * legend_area_factor])
+
+ tables = {"aggregate": aggregate}
+ if per_sample is not None:
+ tables["per_sample"] = per_sample
+ n_samples, dropped_sample_cells = _sample_counts(
+ store,
+ cell_key=cell_key,
+ sample_by=sample_by,
+ study_design=study_design,
+ )
+ assays = _feature_assays(store, features, from_assay)
+
+ result = PlotResult(
+ figure=fig,
+ axes=axes,
+ tables=tables,
+ legends=(
+ LegendSpec(
+ kind="colorbar",
+ label=colorbar_label,
+ extras={"vmin": vmin, "vmax": vmax},
+ ),
+ LegendSpec(
+ kind="size",
+ label="Detected cells",
+ extras={"domain": [0.0, 1.0]},
+ ),
+ ),
+ scales=(
+ color_scale,
+ size_scale,
+ CategoricalScale(
+ order=tuple(resolved_group_order),
+ palette=(
+ {
+ str(value): color
+ for value, color in categorical_scale.palette.items()
+ }
+ if categorical_scale is not None
+ and categorical_scale.palette is not None
+ else None
+ ),
+ labels=(
+ {
+ str(value): label
+ for value, label in categorical_scale.labels.items()
+ }
+ if categorical_scale is not None
+ and categorical_scale.labels is not None
+ else None
+ ),
+ palette_name=(
+ categorical_scale.palette_name
+ if categorical_scale is not None
+ else "default"
+ ),
+ ),
+ ),
+ provenance=PlotProvenance(
+ assay=next(iter(assays)) if len(assays) == 1 else None,
+ cell_key=cell_key,
+ n_cells=len(store.cells.active_index(cell_key)),
+ n_samples=n_samples,
+ renderer="matplotlib",
+ notes=("dotplot",),
+ extras={
+ "group_by": list(group_keys),
+ "sample_by": (
+ study_design.sample_by if study_design is not None else sample_by
+ ),
+ "expression_cutoff": expression_cutoff,
+ "dropped_sample_cells": dropped_sample_cells,
+ "normalization": {
+ "source": normalization.source,
+ "transform": normalization.transform,
+ },
+ "assays": sorted(assays),
+ "feature_group_brackets": bracket_count,
+ "swap_axes": swap_axes,
+ "group_order": resolved_group_order,
+ "feature_order": resolved_feature_order,
+ "size_scale_source": (
+ "explicit" if size_scale_is_explicit else "panel"
+ ),
+ "size_range": [size_scale.size_min, size_scale.size_max],
+ },
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ result.show()
+ return result
+
+
+def matrixplot(
+ store: Any,
+ *,
+ features: Sequence[str | FeatureRef] | Mapping[str, Sequence[str | FeatureRef]],
+ group_by: str | tuple[str, ...],
+ cell_key: str = "I",
+ from_assay: str | None = None,
+ sample_by: str | None = None,
+ study_design: StudyDesign | None = None,
+ normalization: NormalizationSpec | None = None,
+ expression_cutoff: float = 0.0,
+ value: str = "mean",
+ standardize: str = "none",
+ color_scale: ColorScale | None = None,
+ feature_order: Sequence[Any] | None = None,
+ group_order: Sequence[Any] | None = None,
+ cluster_features: bool = False,
+ cluster_groups: bool = False,
+ cluster_method: str = "average",
+ cluster_metric: str = "euclidean",
+ row_annotations: Mapping[
+ str,
+ Mapping[Any, Any] | Sequence[Any],
+ ]
+ | None = None,
+ column_annotations: Mapping[
+ str,
+ Mapping[Any, Any] | Sequence[Any],
+ ]
+ | None = None,
+ annotation_scales: Mapping[str, CategoricalScale] | None = None,
+ target: Any | None = None,
+ figsize: tuple[float, float] | None = None,
+ theme: str = "notebook",
+ show_legend: bool = True,
+ show: bool = True,
+) -> PlotResult:
+ """Heatmap of mean expression or detection fraction by group.
+
+ Rows and columns preserve input order by default. Set explicit orders or
+ enable hierarchical clustering independently for either axis.
+ ``value="mean"`` colors by average expression; ``value="fraction"`` colors
+ by the share of cells above ``expression_cutoff``. ``sample_by`` has the
+ same equal-sample weighting behavior as :func:`dotplot`.
+ """
+ _, mpl = require_matplotlib()
+ if value not in ("mean", "fraction"):
+ raise ValueError("value must be 'mean' or 'fraction'")
+ color_scale = color_scale or ColorScale(cmap="viridis")
+ if color_scale.scale != "linear":
+ raise NotImplementedError(
+ "matrixplot currently supports only linear color scales"
+ )
+ normalization = normalization or NormalizationSpec()
+
+ aggregate, per_sample = summarize_features_by_group(
+ store,
+ features=features,
+ group_by=group_by,
+ cell_key=cell_key,
+ from_assay=from_assay,
+ sample_by=sample_by,
+ study_design=study_design,
+ normalization=normalization,
+ expression_cutoff=expression_cutoff,
+ )
+ plot_df = aggregate.copy()
+ if value == "mean" and standardize == "feature":
+ plot_df = _standardize_feature(plot_df, "mean")
+ elif standardize not in ("none", "feature"):
+ raise ValueError("standardize must be 'none' or 'feature'")
+
+ group_keys = (group_by,) if isinstance(group_by, str) else tuple(group_by)
+ plot_df["group_label"] = _group_axis_labels(plot_df, group_keys)
+ requested_feature_order = list(
+ dict.fromkeys(
+ resolve_feature(store, feature, from_assay=from_assay).label
+ for _, feature in coerce_feature_list(features)
+ )
+ )
+ summarized_features = list(dict.fromkeys(plot_df["feature"].tolist()))
+ observed_feature_order = [
+ feature
+ for feature in requested_feature_order
+ if feature in set(summarized_features)
+ ]
+ observed_feature_order += [
+ feature
+ for feature in summarized_features
+ if feature not in set(observed_feature_order)
+ ]
+ observed_group_order = list(dict.fromkeys(plot_df["group_label"].tolist()))
+ mat = plot_df.pivot_table(
+ index="feature",
+ columns="group_label",
+ values=value,
+ observed=False,
+ ).reindex(index=observed_feature_order, columns=observed_group_order)
+ row_annotation_values = normalize_annotations(
+ list(mat.index),
+ row_annotations,
+ axis_name="row",
+ )
+ column_annotation_values = normalize_annotations(
+ list(mat.columns),
+ column_annotations,
+ axis_name="column",
+ )
+ mat, row_linkage, column_linkage = order_heatmap(
+ mat,
+ row_order=feature_order,
+ column_order=group_order,
+ cluster_rows=cluster_features,
+ cluster_columns=cluster_groups,
+ method=cluster_method,
+ metric=cluster_metric,
+ )
+ resolved_feature_order = list(mat.index)
+ resolved_group_order = list(mat.columns)
+ row_annotation_values = row_annotation_values.reindex(resolved_feature_order)
+ column_annotation_values = column_annotation_values.reindex(resolved_group_order)
+ row_colors, row_annotation_scales = annotation_colors(
+ row_annotation_values,
+ annotation_scales,
+ )
+ column_colors, column_annotation_scales = annotation_colors(
+ column_annotation_values,
+ annotation_scales,
+ )
+ resolved_annotation_scales = row_annotation_scales + column_annotation_scales
+
+ panel_key: Hashable = "matrixplot"
+ resolved_figsize = figsize
+ if resolved_figsize is None and target is None:
+ # Extra height keeps rotated group labels from clipping.
+ resolved_figsize = (
+ max(4.5, 0.45 * len(resolved_group_order) + 2.2),
+ max(4.2, 0.45 * len(resolved_feature_order) + 2.8),
+ )
+ data = mat.to_numpy(dtype=np.float64)
+ vmin, vmax = _color_limits(data, color_scale)
+ norm = continuous_norm(
+ mpl,
+ vmin=vmin,
+ vmax=vmax,
+ vcenter=color_scale.vcenter,
+ )
+
+ with theme_context(theme):
+ fig, axes, owns = normalize_axes_target(
+ target,
+ panel_keys=[panel_key],
+ figsize=resolved_figsize,
+ )
+ ax = axes[panel_key]
+ im = ax.imshow(
+ data,
+ aspect="auto",
+ cmap=color_scale.cmap or "viridis",
+ norm=norm,
+ interpolation="nearest",
+ )
+ ax.set_xticks(range(len(resolved_group_order)))
+ ax.set_xticklabels(resolved_group_order, rotation=45, ha="right")
+ ax.set_yticks(range(len(resolved_feature_order)))
+ ax.set_yticklabels(resolved_feature_order)
+ annotation_xlim, annotation_ylim = draw_annotation_strips(
+ ax,
+ row_colors=row_colors,
+ column_colors=column_colors,
+ n_rows=len(resolved_feature_order),
+ n_columns=len(resolved_group_order),
+ )
+ ax.set_xlim(annotation_xlim)
+ ax.set_ylim(annotation_ylim)
+ colorbar_label = (
+ "Mean expression"
+ if value == "mean" and standardize == "none"
+ else "Standardized mean expression"
+ if value == "mean"
+ else "Detected cells"
+ )
+ if show_legend:
+ cb = fig.colorbar(
+ im,
+ ax=ax,
+ location="top",
+ orientation="horizontal",
+ shrink=0.8,
+ fraction=0.06,
+ pad=0.04,
+ )
+ cb.set_label(colorbar_label)
+ annotation_handles: list[Any] = []
+ for name, scale in zip(
+ list(row_annotation_values.columns)
+ + list(column_annotation_values.columns),
+ resolved_annotation_scales,
+ strict=True,
+ ):
+ if scale.order is None or scale.palette is None:
+ continue
+ annotation_handles.extend(
+ mpl.lines.Line2D(
+ [],
+ [],
+ marker="s",
+ linestyle="",
+ markerfacecolor=scale.palette[item],
+ markeredgecolor="none",
+ markersize=5,
+ label=(
+ f"{name}: "
+ + (
+ scale.labels.get(item, str(item))
+ if scale.labels is not None
+ else str(item)
+ )
+ ),
+ )
+ for item in scale.order
+ )
+ if annotation_handles:
+ legend_kwargs = {
+ "handles": annotation_handles,
+ "title": "Annotations",
+ "frameon": False,
+ }
+ if owns:
+ fig.legend(loc="outside right center", **legend_kwargs)
+ else:
+ ax.legend(
+ loc="upper left",
+ bbox_to_anchor=(1.02, 1),
+ borderaxespad=0,
+ **legend_kwargs,
+ )
+ apply_figure_chrome(fig, theme)
+
+ tables = {"aggregate": aggregate, "matrix": mat.reset_index()}
+ if per_sample is not None:
+ tables["per_sample"] = per_sample
+ if not row_annotation_values.empty:
+ tables["row_annotations"] = row_annotation_values.reset_index()
+ if not column_annotation_values.empty:
+ tables["column_annotations"] = column_annotation_values.reset_index()
+ n_samples, dropped_sample_cells = _sample_counts(
+ store,
+ cell_key=cell_key,
+ sample_by=sample_by,
+ study_design=study_design,
+ )
+ assays = _feature_assays(store, features, from_assay)
+
+ result = PlotResult(
+ figure=fig,
+ axes=axes,
+ tables=tables,
+ legends=(
+ LegendSpec(
+ kind="colorbar",
+ label=colorbar_label,
+ extras={"vmin": vmin, "vmax": vmax},
+ ),
+ *(
+ LegendSpec(kind="categorical", label=name)
+ for name in list(row_annotation_values.columns)
+ + list(column_annotation_values.columns)
+ ),
+ ),
+ scales=(color_scale, *resolved_annotation_scales),
+ provenance=PlotProvenance(
+ assay=next(iter(assays)) if len(assays) == 1 else None,
+ cell_key=cell_key,
+ n_cells=len(store.cells.active_index(cell_key)),
+ n_samples=n_samples,
+ renderer="matplotlib",
+ notes=("matrixplot", value),
+ extras={
+ "group_by": list(group_keys),
+ "sample_by": (
+ study_design.sample_by if study_design is not None else sample_by
+ ),
+ "expression_cutoff": expression_cutoff,
+ "dropped_sample_cells": dropped_sample_cells,
+ "normalization": {
+ "source": normalization.source,
+ "transform": normalization.transform,
+ },
+ "assays": sorted(assays),
+ "feature_order": resolved_feature_order,
+ "group_order": resolved_group_order,
+ "cluster_features": row_linkage is not None,
+ "cluster_groups": column_linkage is not None,
+ "cluster_method": cluster_method,
+ "cluster_metric": cluster_metric,
+ "row_annotations": list(row_annotation_values.columns),
+ "column_annotations": list(column_annotation_values.columns),
+ },
+ ),
+ owns_figure=owns,
+ theme=theme,
+ )
+ if show:
+ result.show()
+ return result
diff --git a/scarf/quality_control/__init__.py b/scarf/quality_control/__init__.py
new file mode 100644
index 00000000..90022b32
--- /dev/null
+++ b/scarf/quality_control/__init__.py
@@ -0,0 +1,20 @@
+from .cell_cycle import assign_cell_cycle_phase
+from .cell_cycle_genes import g2m_phase_genes, s_phase_genes
+from .doublets import (
+ sample_cluster_pool,
+ simulate_doublet_pairs,
+ write_doublet_target_zarr,
+)
+from .filtering import gaussian_quantile_bounds
+from .hto import hto_demux
+
+__all__ = [
+ "assign_cell_cycle_phase",
+ "g2m_phase_genes",
+ "gaussian_quantile_bounds",
+ "hto_demux",
+ "sample_cluster_pool",
+ "s_phase_genes",
+ "simulate_doublet_pairs",
+ "write_doublet_target_zarr",
+]
diff --git a/scarf/quality_control/cell_cycle.py b/scarf/quality_control/cell_cycle.py
new file mode 100644
index 00000000..776b04f3
--- /dev/null
+++ b/scarf/quality_control/cell_cycle.py
@@ -0,0 +1,13 @@
+import numpy as np
+
+__all__ = ["assign_cell_cycle_phase"]
+
+
+def assign_cell_cycle_phase(
+ s_score: np.ndarray,
+ g2m_score: np.ndarray,
+) -> np.ndarray:
+ phase = np.full(np.asarray(s_score).shape, "S", dtype=object)
+ phase[np.asarray(g2m_score) > np.asarray(s_score)] = "G2M"
+ phase[(np.asarray(g2m_score) < 0) & (np.asarray(s_score) < 0)] = "G1"
+ return phase
diff --git a/scarf/bio_data.py b/scarf/quality_control/cell_cycle_genes.py
similarity index 88%
rename from scarf/bio_data.py
rename to scarf/quality_control/cell_cycle_genes.py
index b6b75952..0812f04f 100644
--- a/scarf/bio_data.py
+++ b/scarf/quality_control/cell_cycle_genes.py
@@ -1,13 +1,6 @@
-"""Contains definitions of biological information (e.
-
-g. cell cycle genes).
-"""
-
-from typing import List
-
__all__ = ["s_phase_genes", "g2m_phase_genes"]
-s_phase_genes: List[str] = [
+s_phase_genes: list[str] = [
"MCM5",
"PCNA",
"TYMS",
@@ -53,7 +46,7 @@
"E2F8",
]
-g2m_phase_genes: List[str] = [
+g2m_phase_genes: list[str] = [
"HMGB2",
"CDK1",
"NUSAP1",
diff --git a/scarf/quality_control/doublets.py b/scarf/quality_control/doublets.py
new file mode 100644
index 00000000..eae843aa
--- /dev/null
+++ b/scarf/quality_control/doublets.py
@@ -0,0 +1,110 @@
+"""Utilities for synthetic doublet detection."""
+
+import numpy as np
+import zarr
+from numpy.typing import NDArray
+from scipy.sparse import csr_matrix
+
+from ..storage.profiles import StorageProfile
+from ..utils.logging import logger
+
+__all__ = [
+ "sample_cluster_pool",
+ "simulate_doublet_pairs",
+ "write_doublet_target_zarr",
+]
+
+
+def sample_cluster_pool(
+ clusters: NDArray,
+ fraction: float,
+ max_per_cluster: int,
+ rng: np.random.Generator,
+) -> NDArray[np.int64]:
+ """Draw a per-cluster subsample of cell positions to seed doublet simulation."""
+ pool = []
+ for c in np.unique(clusters):
+ idx = np.where(clusters == c)[0]
+ n = min(int(np.ceil(len(idx) * fraction)), max_per_cluster, len(idx))
+ if n <= 0:
+ continue
+ pool.append(rng.choice(idx, size=n, replace=False))
+ if len(pool) == 0:
+ raise ValueError("ERROR: No cells could be sampled to simulate doublets")
+ return np.sort(np.concatenate(pool))
+
+
+def simulate_doublet_pairs(
+ pool_clusters: NDArray,
+ n_sim: int,
+ heterotypic_fraction: float,
+ rng: np.random.Generator,
+ max_tries: int = 20,
+) -> tuple[NDArray[np.int64], NDArray[np.int64]]:
+ """Generate index pairs into the candidate pool for simulated doublets."""
+ pool_size = len(pool_clusters)
+ left = rng.integers(0, pool_size, size=n_sim)
+ right = rng.integers(0, pool_size, size=n_sim)
+ if heterotypic_fraction > 0 and len(np.unique(pool_clusters)) > 1:
+ want_hetero = rng.random(n_sim) < heterotypic_fraction
+ for _ in range(max_tries):
+ clash = want_hetero & (pool_clusters[left] == pool_clusters[right])
+ if not clash.any():
+ break
+ right[clash] = rng.integers(0, pool_size, size=int(clash.sum()))
+ return left, right
+
+
+def write_doublet_target_zarr(
+ zarr_loc: str,
+ assay_name: str,
+ sim_counts: csr_matrix,
+ feat_ids: NDArray,
+ feat_names: NDArray,
+ dtype: str = "uint32",
+ mem_budget: int | str | None = None,
+ nthreads: int | None = None,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+) -> zarr.Group:
+ """Materialise simulated doublet counts as a minimal Scarf Zarr hierarchy."""
+ from ..storage.schema import (
+ create_cell_data,
+ create_zarr_count_assay,
+ load_count_array,
+ validate_assay_name,
+ )
+ from ..storage.sharding import write_dense_in_shard_rows
+ from ..storage.budget import resolve_budget
+ from ..storage.profiles import resolve_storage_profile
+ from ..storage.stores import load_zarr
+
+ validate_assay_name(assay_name)
+ resources = resolve_budget(mem_budget, nthreads)
+ resolved_profile = resolve_storage_profile(zarr_loc, profile)
+ n_sim = sim_counts.shape[0]
+ z = load_zarr(zarr_loc=zarr_loc, mode="w")
+ ids = np.array([f"doublet_{i}" for i in range(n_sim)])
+ create_cell_data(z, workspace=None, ids=ids, names=ids)
+ create_zarr_count_assay(
+ z=z,
+ assay_name=assay_name,
+ workspace=None,
+ n_cells=n_sim,
+ feat_ids=np.asarray(feat_ids),
+ feat_names=np.asarray(feat_names),
+ dtype=dtype,
+ profile=resolved_profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ )
+ store = load_count_array(z, assay_name, None)
+ write_dense_in_shard_rows(
+ store,
+ lambda s, e: sim_counts[s:e].toarray().astype(dtype),
+ msg="Writing simulated doublets",
+ resources=resources,
+ )
+ logger.debug(f"Wrote {n_sim} simulated doublets to {zarr_loc}")
+ return z
diff --git a/scarf/quality_control/filtering.py b/scarf/quality_control/filtering.py
new file mode 100644
index 00000000..5834f9d9
--- /dev/null
+++ b/scarf/quality_control/filtering.py
@@ -0,0 +1,320 @@
+from typing import TypedDict
+
+import numpy as np
+import pandas as pd
+from scipy.stats import norm
+
+__all__ = ["gaussian_quantile_bounds"]
+
+_MAD_SCALE = 1.4826
+_COUNT_SUFFIXES = ("nCounts", "nFeatures")
+_PERCENT_SUFFIXES = ("percentMito", "percentRibo")
+
+
+class _MadProvenance(TypedDict):
+ mad_scale: float
+ metric_policies: dict[str, dict[str, str]]
+ sample_sizes: dict[str, int]
+ skip_reasons: dict[str, str]
+ resolved_bounds: dict[str, dict[str, dict[str, object]]]
+ warnings: list[str]
+
+
+def gaussian_quantile_bounds(
+ values: np.ndarray,
+ min_p: float = 0.01,
+ max_p: float = 0.99,
+) -> tuple[float, float]:
+ dist = norm(np.median(values), np.std(values))
+ return float(dist.ppf(min_p)), float(dist.ppf(max_p))
+
+
+def _mad_bounds(values: np.ndarray, n_mads: float) -> tuple[float, float, float]:
+ """Return ``(low, high, scaled_mad)`` for robust per-sample thresholds.
+
+ ``scaled_mad`` is ``1.4826 * MAD``. Callers must treat a zero scaled MAD as a
+ skip condition rather than applying a zero-width exclusive interval.
+ """
+ if not np.isfinite(n_mads) or n_mads <= 0:
+ raise ValueError("n_mads must be finite and greater than 0")
+ if not np.isfinite(values).all():
+ raise ValueError("MAD input values must all be finite")
+ median = float(np.median(values))
+ scaled_mad = float(_MAD_SCALE * np.median(np.abs(values - median)))
+ with np.errstate(over="ignore", invalid="ignore"):
+ distance = float(n_mads * scaled_mad)
+ low = float(median - distance)
+ high = float(median + distance)
+ if not np.isfinite([median, scaled_mad, low, high]).all():
+ raise ValueError("n_mads produces non-finite MAD bounds")
+ return low, high, scaled_mad
+
+
+def _metric_policy(attr: str) -> dict[str, str]:
+ """Return transform and bound direction for a QC metadata column."""
+ for suffix in _COUNT_SUFFIXES:
+ if attr == suffix or attr.endswith(f"_{suffix}"):
+ return {"transform": "log1p", "bound_direction": "two_sided"}
+ for suffix in _PERCENT_SUFFIXES:
+ if attr == suffix or attr.endswith(f"_{suffix}"):
+ return {"transform": "identity", "bound_direction": "upper"}
+ return {"transform": "identity", "bound_direction": "two_sided"}
+
+
+def _validated_work_scale(
+ values: np.ndarray,
+ *,
+ attr: str,
+ transform: str,
+) -> np.ndarray:
+ raw = np.asarray(values, dtype=float)
+ if not np.isfinite(raw).all():
+ raise ValueError(f"QC values in '{attr}' contain non-finite entries")
+ if transform == "log1p":
+ if (raw < 0).any():
+ raise ValueError(f"QC values in '{attr}' must be non-negative before log1p")
+ work = np.asarray(np.log1p(raw), dtype=float)
+ else:
+ work = raw
+ if not np.isfinite(work).all():
+ raise ValueError(
+ f"QC values in '{attr}' contain non-finite entries after {transform}"
+ )
+ return work
+
+
+def _from_work_scale(bound: float, transform: str) -> float:
+ if transform == "log1p":
+ with np.errstate(over="ignore", invalid="ignore"):
+ resolved = float(np.expm1(bound))
+ else:
+ resolved = float(bound)
+ if not np.isfinite(resolved):
+ raise ValueError(
+ f"MAD bound is non-finite after converting from the {transform} scale"
+ )
+ return resolved
+
+
+def _clamp_metric_bound(
+ bound: float,
+ *,
+ transform: str,
+ is_percent: bool,
+) -> float:
+ if not np.isfinite(bound):
+ raise ValueError("Resolved MAD bounds must be finite")
+ if transform == "log1p":
+ resolved = max(0.0, bound)
+ elif is_percent:
+ resolved = min(100.0, max(0.0, bound))
+ else:
+ resolved = bound
+ if not np.isfinite(resolved):
+ raise ValueError("Resolved MAD bounds must be finite")
+ return resolved
+
+
+def _validated_sample_labels(
+ sample_labels: np.ndarray,
+ active: np.ndarray,
+ *,
+ label_name: str = "sample labels",
+) -> np.ndarray:
+ labels = np.asarray(sample_labels)
+ active_mask = np.asarray(active)
+ if labels.ndim != 1 or active_mask.ndim != 1 or labels.shape != active_mask.shape:
+ raise ValueError("Sample labels and active selection must be aligned vectors")
+ normalized = labels.astype(object, copy=True)
+ kinds: set[str] = set()
+ for index in np.flatnonzero(active_mask):
+ value = labels[index]
+ if isinstance(value, np.generic):
+ value = value.item()
+ missing = pd.isna(value)
+ if isinstance(missing, bool | np.bool_) and bool(missing):
+ raise ValueError(f"{label_name} contains missing labels among active cells")
+ if isinstance(value, bytes):
+ try:
+ decoded = value.decode("utf-8")
+ except UnicodeDecodeError as exc:
+ raise ValueError(
+ f"{label_name} contains a non-UTF-8 bytes label"
+ ) from exc
+ if decoded.strip() == "":
+ raise ValueError(
+ f"{label_name} contains missing labels among active cells"
+ )
+ kind = "bytes"
+ elif isinstance(value, str):
+ if value.strip() == "":
+ raise ValueError(
+ f"{label_name} contains missing labels among active cells"
+ )
+ kind = "str"
+ elif isinstance(value, bool):
+ kind = "bool"
+ elif isinstance(value, int):
+ kind = "int"
+ elif isinstance(value, float):
+ if not np.isfinite(value):
+ raise ValueError(f"{label_name} must contain finite labels")
+ kind = "float"
+ else:
+ raise TypeError(
+ f"{label_name} contains unsupported label type {type(value).__name__!r}"
+ )
+ normalized[index] = value
+ kinds.add(kind)
+ if len(kinds) > 1:
+ raise ValueError(
+ f"{label_name} must use one consistent label type among active cells"
+ )
+ return normalized
+
+
+def _apply_exclusive_bounds(
+ values: np.ndarray,
+ low: float | None,
+ high: float | None,
+) -> np.ndarray:
+ keep = np.ones(values.shape[0], dtype=bool)
+ if low is not None:
+ keep &= values > low
+ if high is not None:
+ keep &= values < high
+ return keep
+
+
+def _sample_aware_mad_mask(
+ *,
+ values_by_attr: dict[str, np.ndarray],
+ sample_labels: np.ndarray,
+ active: np.ndarray,
+ n_mads: float,
+ min_cells_per_sample: int,
+ attrs: list[str],
+) -> tuple[np.ndarray, _MadProvenance]:
+ """Build one cell mask from per-sample MAD bounds.
+
+ Inactive cells are left ``True`` in the returned mask so callers can
+ intersect with the current selection via ``update_key``.
+ """
+ n_cells = active.shape[0]
+ sample_labels = _validated_sample_labels(sample_labels, active)
+ keep = np.ones(n_cells, dtype=bool)
+ policies = {attr: _metric_policy(attr) for attr in attrs}
+ raw_values_by_attr: dict[str, np.ndarray] = {}
+ work_values_by_attr: dict[str, np.ndarray] = {}
+ for attr in attrs:
+ raw = np.asarray(values_by_attr[attr], dtype=float)
+ raw_values_by_attr[attr] = raw
+ work = np.empty(n_cells, dtype=float)
+ work[active] = _validated_work_scale(
+ raw[active],
+ attr=attr,
+ transform=policies[attr]["transform"],
+ )
+ work_values_by_attr[attr] = work
+ sample_sizes: dict[str, int] = {}
+ skip_reasons: dict[str, str] = {}
+ resolved_bounds: dict[str, dict[str, dict[str, object]]] = {}
+ warnings: list[str] = []
+
+ # Preserve first-seen label order for deterministic provenance.
+ ordered_samples: list[object] = []
+ seen: set[object] = set()
+ for label in sample_labels[active]:
+ key = label if not isinstance(label, np.generic) else label.item()
+ if key not in seen:
+ seen.add(key)
+ ordered_samples.append(key)
+
+ for sample in ordered_samples:
+ sample_key = (
+ sample.decode("utf-8") if isinstance(sample, bytes) else str(sample)
+ )
+ if sample_key in sample_sizes:
+ raise ValueError(
+ "Sample labels collide after deterministic provenance encoding"
+ )
+ sample_mask = np.zeros(n_cells, dtype=bool)
+ sample_mask[active] = sample_labels[active] == sample
+ sample_idx = np.flatnonzero(sample_mask)
+ sample_sizes[sample_key] = int(sample_idx.shape[0])
+ resolved_bounds[sample_key] = {}
+
+ if sample_idx.shape[0] < min_cells_per_sample:
+ skip_reasons[sample_key] = "insufficient_cells"
+ warnings.append(
+ f"Sample '{sample_key}' has fewer than {min_cells_per_sample} "
+ "active cells; retaining all of its cells without MAD filtering"
+ )
+ continue
+
+ sample_keep = np.ones(sample_idx.shape[0], dtype=bool)
+ for attr in attrs:
+ policy = policies[attr]
+ transform = policy["transform"]
+ direction = policy["bound_direction"]
+ is_percent = direction == "upper" and transform == "identity"
+ raw = raw_values_by_attr[attr][sample_idx]
+ work = work_values_by_attr[attr][sample_idx]
+ low_t, high_t, scaled_mad = _mad_bounds(work, n_mads)
+ if scaled_mad == 0.0:
+ resolved_bounds[sample_key][attr] = {
+ "low": None,
+ "high": None,
+ "skip_reason": "zero_mad",
+ "transform": transform,
+ "bound_direction": direction,
+ "scaled_mad": 0.0,
+ }
+ warnings.append(
+ f"Sample '{sample_key}' has zero MAD for '{attr}'; "
+ "retaining cells for this metric"
+ )
+ continue
+
+ low: float | None
+ high: float | None
+ if direction == "upper":
+ low = None
+ high = _clamp_metric_bound(
+ _from_work_scale(high_t, transform),
+ transform=transform,
+ is_percent=is_percent,
+ )
+ else:
+ low = _clamp_metric_bound(
+ _from_work_scale(low_t, transform),
+ transform=transform,
+ is_percent=is_percent,
+ )
+ high = _clamp_metric_bound(
+ _from_work_scale(high_t, transform),
+ transform=transform,
+ is_percent=is_percent,
+ )
+
+ resolved_bounds[sample_key][attr] = {
+ "low": low,
+ "high": high,
+ "skip_reason": None,
+ "transform": transform,
+ "bound_direction": direction,
+ "scaled_mad": scaled_mad,
+ }
+ sample_keep &= _apply_exclusive_bounds(raw, low, high)
+
+ keep[sample_idx] = sample_keep
+
+ provenance: _MadProvenance = {
+ "mad_scale": _MAD_SCALE,
+ "metric_policies": policies,
+ "sample_sizes": sample_sizes,
+ "skip_reasons": skip_reasons,
+ "resolved_bounds": resolved_bounds,
+ "warnings": warnings,
+ }
+ return keep, provenance
diff --git a/scarf/quality_control/hto.py b/scarf/quality_control/hto.py
new file mode 100644
index 00000000..9c0e1e8c
--- /dev/null
+++ b/scarf/quality_control/hto.py
@@ -0,0 +1,298 @@
+import warnings
+from collections.abc import Mapping
+
+import numpy as np
+import pandas as pd
+from pandas.api.types import is_numeric_dtype
+
+__all__ = ["hto_demux"]
+
+_NORMALIZATION_METHOD = "clr_per_hto"
+_CLUSTERING_METHOD = "kmeans"
+_KMEANS_INIT = "random"
+_KMEANS_N_STARTS = 100
+_CLUSTER_COUNT_RULE = "n_htos_plus_one"
+_BACKGROUND_STATISTIC = "raw_mean"
+_CUTOFF_DISTRIBUTION = "negative_binomial_nb2"
+_POSITIVE_QUANTILE = 0.99
+_CUTOFF_LOCATION = 0
+_CUTOFF_COMPARISON = "strictly_greater"
+_SINGLET_ASSIGNMENT = "clr_argmax"
+_RESERVED_IDENTITIES = frozenset({"Negative", "Singlet", "Doublet"})
+
+
+def _hto_demux_method() -> dict[str, object]:
+ return {
+ "normalization": _NORMALIZATION_METHOD,
+ "clustering": {
+ "method": _CLUSTERING_METHOD,
+ "init": _KMEANS_INIT,
+ "n_starts": _KMEANS_N_STARTS,
+ "cluster_count": _CLUSTER_COUNT_RULE,
+ },
+ "background": _BACKGROUND_STATISTIC,
+ "cutoff": {
+ "distribution": _CUTOFF_DISTRIBUTION,
+ "quantile": _POSITIVE_QUANTILE,
+ "location": _CUTOFF_LOCATION,
+ "comparison": _CUTOFF_COMPARISON,
+ },
+ "singlet_assignment": _SINGLET_ASSIGNMENT,
+ }
+
+
+def _validated_hto_counts(hto_counts: pd.DataFrame) -> pd.DataFrame:
+ if not isinstance(hto_counts, pd.DataFrame):
+ raise TypeError("hto_counts must be a pandas DataFrame")
+ if hto_counts.shape[1] == 0:
+ raise ValueError("hto_counts must contain at least one HTO")
+ if not hto_counts.index.is_unique:
+ raise ValueError("hto_counts cell index must be unique")
+ if not hto_counts.columns.is_unique:
+ raise ValueError("HTO IDs must be unique")
+
+ hto_names = hto_counts.columns.tolist()
+ if any(not isinstance(name, str) or not name.strip() for name in hto_names):
+ raise ValueError("HTO IDs must be non-empty strings")
+ reserved = sorted(_RESERVED_IDENTITIES.intersection(hto_names))
+ if reserved:
+ raise ValueError(
+ "HTO IDs conflict with reserved identity labels: "
+ + ", ".join(repr(name) for name in reserved)
+ )
+ if not all(is_numeric_dtype(dtype) for dtype in hto_counts.dtypes):
+ raise TypeError("hto_counts must contain only numeric raw counts")
+
+ raw_counts = hto_counts.to_numpy()
+ if np.iscomplexobj(raw_counts):
+ raise TypeError("hto_counts must contain only real numeric raw counts")
+ counts = raw_counts.astype(float)
+ if not np.all(np.isfinite(counts)):
+ raise ValueError("hto_counts must contain only finite raw counts")
+ if np.any(counts < 0):
+ raise ValueError("hto_counts must contain only nonnegative raw counts")
+ if np.any(counts != np.floor(counts)):
+ raise ValueError("hto_counts must contain integer-valued raw counts")
+
+ required_cells = hto_counts.shape[1] + 1
+ if hto_counts.shape[0] < required_cells:
+ raise ValueError(
+ f"HTO demultiplexing requires at least {required_cells} selected cells"
+ )
+
+ empty_htos = [
+ hto_names[index]
+ for index in np.flatnonzero(np.all(counts == 0, axis=0)).tolist()
+ ]
+ if empty_htos:
+ raise ValueError(
+ "HTOs with no positive counts cannot be demultiplexed: "
+ + ", ".join(repr(name) for name in empty_htos)
+ )
+ return pd.DataFrame(counts, index=hto_counts.index, columns=hto_counts.columns)
+
+
+def _clr_normalize(hto_counts: pd.DataFrame) -> pd.DataFrame:
+ scale = np.exp(np.log1p(hto_counts).sum(axis=0) / len(hto_counts))
+ normalized = np.log1p(hto_counts / scale)
+ if not np.all(np.isfinite(normalized.to_numpy())):
+ raise ValueError("CLR normalization produced non-finite HTO values")
+ return normalized
+
+
+def _cluster_labels(normalized: pd.DataFrame, random_seed: int) -> np.ndarray:
+ from sklearn.cluster import KMeans
+
+ n_centers = normalized.shape[1] + 1
+ unique_profiles = np.unique(normalized.to_numpy(), axis=0).shape[0]
+ if unique_profiles < n_centers:
+ raise ValueError(
+ "HTO demultiplexing requires at least "
+ f"{n_centers} distinct normalized cell profiles; found {unique_profiles}"
+ )
+ labels = np.asarray(
+ KMeans(
+ n_clusters=n_centers,
+ init=_KMEANS_INIT,
+ n_init=_KMEANS_N_STARTS,
+ random_state=random_seed,
+ ).fit_predict(normalized)
+ )
+ occupied = np.unique(labels).size
+ if occupied != n_centers:
+ raise ValueError(
+ f"HTO clustering produced {occupied} occupied clusters; "
+ f"expected {n_centers}"
+ )
+ return labels
+
+
+def _background_clusters(
+ hto_counts: pd.DataFrame,
+ cluster_labels: np.ndarray,
+) -> pd.Series:
+ cluster_means = hto_counts.groupby(cluster_labels, sort=True).mean()
+ return cluster_means.idxmin(axis=0)
+
+
+def _fit_negative_binomial_parameters(
+ values: np.ndarray,
+ hto_name: str,
+) -> tuple[float, float]:
+ from statsmodels.discrete.discrete_model import NegativeBinomial
+
+ exog = np.ones((len(values), 1), dtype=float)
+ try:
+ with warnings.catch_warnings():
+ warnings.simplefilter("ignore")
+ fit = NegativeBinomial(
+ values,
+ exog,
+ loglike_method="nb2",
+ ).fit(
+ start_params=np.asarray([1.0, 1.0]),
+ disp=0,
+ full_output=True,
+ )
+ except Exception as exc: # noqa: BLE001
+ raise ValueError(
+ f"Negative-binomial background fit failed for HTO {hto_name!r}"
+ ) from exc
+
+ fit_details = getattr(fit, "mle_retvals", None)
+ convergence = (
+ fit_details.get("converged") if isinstance(fit_details, Mapping) else None
+ )
+ if not isinstance(convergence, bool | np.bool_) or not bool(convergence):
+ raise ValueError(
+ f"Negative-binomial background fit did not converge for HTO {hto_name!r}"
+ )
+ try:
+ parameters = np.asarray(fit.params, dtype=float)
+ except (TypeError, ValueError) as exc:
+ raise ValueError(
+ f"Negative-binomial background fit returned invalid parameters "
+ f"for HTO {hto_name!r}"
+ ) from exc
+ if parameters.size != 2:
+ raise ValueError(
+ f"Negative-binomial background fit returned invalid parameters "
+ f"for HTO {hto_name!r}"
+ )
+ with np.errstate(over="ignore", invalid="ignore"):
+ mu = float(np.exp(parameters[0]))
+ alpha = float(parameters[-1])
+ if not np.isfinite(mu) or mu <= 0:
+ raise ValueError(
+ f"Negative-binomial background fit returned invalid mean "
+ f"for HTO {hto_name!r}"
+ )
+ if not np.isfinite(alpha) or alpha <= 0:
+ raise ValueError(
+ f"Negative-binomial background fit returned invalid dispersion "
+ f"for HTO {hto_name!r}"
+ )
+ return mu, alpha
+
+
+def _negative_binomial_cutoff(
+ mu: float,
+ alpha: float,
+ quantile: float = _POSITIVE_QUANTILE,
+) -> int:
+ from scipy.stats import nbinom
+
+ if not np.isfinite(mu) or mu <= 0:
+ raise ValueError("Negative-binomial mean must be finite and greater than 0")
+ if not np.isfinite(alpha) or alpha <= 0:
+ raise ValueError(
+ "Negative-binomial dispersion must be finite and greater than 0"
+ )
+ if not np.isfinite(quantile) or not 0 < quantile < 1:
+ raise ValueError("Negative-binomial quantile must be between 0 and 1")
+
+ with np.errstate(over="ignore", divide="ignore", invalid="ignore"):
+ size = 1 / alpha
+ probability = 1 / (1 + alpha * mu)
+ if (
+ not np.isfinite(size)
+ or size <= 0
+ or not np.isfinite(probability)
+ or not 0 < probability <= 1
+ ):
+ raise ValueError(
+ "Negative-binomial parameters produced an invalid distribution"
+ )
+
+ cutoff = float(nbinom.ppf(quantile, n=size, p=probability))
+ if not np.isfinite(cutoff) or not cutoff.is_integer():
+ raise ValueError("Negative-binomial cutoff must be a finite integer")
+ return int(cutoff)
+
+
+def _positive_hto_calls(
+ hto_counts: pd.DataFrame,
+ cluster_labels: np.ndarray,
+) -> pd.DataFrame:
+ background_clusters = _background_clusters(hto_counts, cluster_labels)
+ cutoffs: dict[str, int] = {}
+ for hto_name in hto_counts.columns:
+ background = hto_counts.loc[
+ cluster_labels == background_clusters[hto_name],
+ hto_name,
+ ].to_numpy()
+ if len(background) < 2:
+ raise ValueError(
+ f"Background cluster for HTO {hto_name!r} must contain "
+ "at least two cells"
+ )
+ if not np.any(background > 0):
+ raise ValueError(
+ f"Background cluster for HTO {hto_name!r} contains only zero counts"
+ )
+ mu, alpha = _fit_negative_binomial_parameters(background, hto_name)
+ try:
+ cutoffs[hto_name] = _negative_binomial_cutoff(mu, alpha)
+ except ValueError as exc:
+ raise ValueError(
+ f"Negative-binomial cutoff is invalid for HTO {hto_name!r}"
+ ) from exc
+ return hto_counts > pd.Series(cutoffs)
+
+
+def _classify_hto_identities(
+ normalized: pd.DataFrame,
+ positive_calls: pd.DataFrame,
+) -> pd.Series:
+ positive_count = positive_calls.sum(axis=1)
+ identities = positive_count.map(
+ lambda count: (
+ "Negative" if count == 0 else "Singlet" if count == 1 else "Doublet"
+ )
+ )
+ singlets = positive_count == 1
+ identities.loc[singlets] = normalized.loc[singlets].idxmax(axis=1)
+ return identities
+
+
+def hto_demux(
+ hto_counts: pd.DataFrame,
+ *,
+ random_seed: int = 0,
+) -> pd.Series:
+ """Assigns HTO identity to each cell based on the HTO count distribution.
+ The algorithm is adapted from the Seurat package's HTOdemux function [Satija15]_.
+
+ Args:
+ hto_counts: A dataframe containing the raw HTO counts for each cell.
+
+ Returns:
+ A series containing the HTO identity for each cell.
+ """
+ if isinstance(random_seed, bool) or not isinstance(random_seed, int):
+ raise TypeError("random_seed must be an integer")
+ counts = _validated_hto_counts(hto_counts)
+ normalized = _clr_normalize(counts)
+ cluster_labels = _cluster_labels(normalized, random_seed)
+ positive_calls = _positive_hto_calls(counts, cluster_labels)
+ return _classify_hto_identities(normalized, positive_calls)
diff --git a/scarf/readers.py b/scarf/readers.py
deleted file mode 100644
index e1618c7f..00000000
--- a/scarf/readers.py
+++ /dev/null
@@ -1,1295 +0,0 @@
-"""A collection of classes for reading in different data formats.
-
-- Classes:
- - CrH5Reader: A class to read in CellRanger (Cr) data, in the form of an H5 file.
- - CrDirReader: A class to read in CellRanger (Cr) data, in the form of a directory.
- - CrReader: A class to read in CellRanger (Cr) data.
- - H5adReader: A class to read in data in the form of a H5ad file (h5 file with AnnData information).
- - NaboH5Reader: A class to read in data in the form of a Nabo H5 file.
- - LoomReader: A class to read in data in the form of a Loom file.
-"""
-
-import math
-import os
-from abc import ABC, abstractmethod
-from typing import IO, Dict, Generator, List, Optional, Tuple, Union
-
-import h5py
-import numpy as np
-import pandas as pd
-import polars as pl
-from scipy.sparse import coo_matrix
-
-from .utils import logger, tqdmbar
-
-__all__ = [
- "CrH5Reader",
- "CrDirReader",
- "CrReader",
- "H5adReader",
- "NaboH5Reader",
- "LoomReader",
- "CSVReader",
-]
-
-
-def get_file_handle(fn: str) -> IO:
- """Returns a file object for the given file name.
-
- Args:
- fn: The path to the file (file name).
- """
- import gzip
-
- try:
- if fn.rsplit(".", 1)[-1] == "gz":
- return gzip.open(fn, mode="rt")
- else:
- return open(fn, "r")
- except (OSError, IOError, FileNotFoundError):
- raise FileNotFoundError("ERROR: FILE NOT FOUND: %s" % fn)
-
-
-def read_file(fn: str):
- """Yields the lines from the file the given file name points to.
-
- Args:
- fn: The path to the file (file name).
- """
- fh = get_file_handle(fn)
- for line in fh:
- yield line.rstrip()
-
-
-class CrReader(ABC):
- """A class to read in CellRanger (Cr) data.
-
- Args:
- grp_names (Dict): A dictionary that specifies where to find the matrix, features and barcodes.
-
- Attributes:
- autoNames: Specifies if the data is from RNA or ATAC sequencing.
- grpNames: A dictionary that specifies where to find the matrix, features and barcodes.
- nFeatures: Number of features in dataset.
- nCells: Number of cells in dataset.
- assayFeats: A DataFrame with information about the features in the assay.
- """
-
- def __init__(self, grp_names):
- self.autoNames = {
- "Gene Expression": "RNA",
- "Peaks": "ATAC",
- "Antibody Capture": "ADT",
- "RNA": "RNA",
- "ADT": "ADT",
- "HTO": "HTO",
- }
- self.grpNames: Dict = grp_names
- self.nFeatures: int = len(self.feature_names())
- self.nCells: int = len(self.cell_names())
- self.assayFeats = self._make_feat_table()
- self._auto_rename_assay_names()
-
- @abstractmethod
- def _handle_version(self):
- pass
-
- @abstractmethod
- def _read_dataset(self, key: Optional[str] = None) -> List:
- pass
-
- @abstractmethod
- def consume(self, batch_size: int, lines_in_mem: int):
- """Returns a generator that yield chunks of data."""
- pass
-
- def _subset_by_assay(self, v, assay) -> List:
- if assay is None:
- return v
- elif assay not in self.assayFeats:
- raise ValueError(f"ERROR: Assay ID {assay} is not valid")
- if len(self.assayFeats[assay].shape) == 2:
- ret_val = []
- for i in self.assayFeats[assay].values[1:3].T:
- ret_val.extend(list(v[i[0] : i[1]]))
- return ret_val
- elif len(self.assayFeats[assay].shape) == 1:
- idx = self.assayFeats[assay]
- return v[idx.start : idx.end]
- else:
- raise ValueError(
- "ERROR: assay feats is 3D. Something went really wrong. Create a github issue"
- )
-
- def _make_feat_table(self) -> pd.DataFrame:
- s = self.feature_types()
- span: List[Tuple] = []
- last = s[0]
- last_n: int = 0
- for n, i in enumerate(s[1:], 1):
- if i != last:
- span.append((last, last_n, n))
- last_n = n
- elif n == len(s) - 1:
- span.append((last, last_n, n + 1))
- last = i
- df = pd.DataFrame(span, columns=["type", "start", "end"])
- df.index = ["ASSAY%s" % str(x + 1) for x in df.index]
- df["nFeatures"] = df.end - df.start
- return df.T
-
- def _auto_rename_assay_names(self):
- new_names = []
- for k, v in self.assayFeats.T["type"].to_dict().items():
- if v in self.autoNames:
- new_names.append(self.autoNames[v])
- else:
- new_names.append(k)
- self.assayFeats.columns = new_names
-
- def rename_assays(self, name_map: Dict[str, str]) -> None:
- """Renames specified assays in the Reader.
-
- Args:
- name_map: A Dictionary containing current name as key and new name as value.
- """
- self.assayFeats.rename(columns=name_map, inplace=True)
-
- def feature_ids(self, assay: str = None) -> List[str]:
- """Returns a list of feature IDs in a specified assay.
-
- Args:
- assay: Select which assay to retrieve feature IDs from.
- """
- return self._subset_by_assay(self._read_dataset("feature_ids"), assay)
-
- def feature_names(self, assay: str = None) -> List[str]:
- """Returns a list of features in the dataset.
-
- Args:
- assay: Select which assay to retrieve features from.
- """
- vals = self._read_dataset("feature_names")
- if vals is None:
- logger.warning("Feature names extraction failed using feature IDs")
- vals = self._read_dataset("feature_ids")
- return self._subset_by_assay(vals, assay)
-
- def feature_types(self) -> List[str]:
- """Returns a list of feature types in the dataset."""
- if self.grpNames["feature_types"] is not None:
- ret_val = self._read_dataset("feature_types")
- if ret_val is not None:
- return ret_val
- default_name = list(self.autoNames.keys())[0]
- return [default_name for _ in range(self.nFeatures)]
-
- def cell_names(self) -> List[str]:
- """Returns a list of names of the cells in the dataset."""
- return self._read_dataset("cell_names")
-
-
-class CrH5Reader(CrReader):
- # noinspection PyUnresolvedReferences
- """A class to read in CellRanger (Cr) data, in the form of an H5 file.
-
- Subclass of CrReader.
-
- Args:
- h5_fn: File name for the h5 file.
-
- Attributes:
- autoNames: Specifies if the data is from RNA or ATAC sequencing.
- grpNames: A dictionary that specifies where to find the matrix, features and barcodes.
- nFeatures: Number of features in dataset.
- nCells: Number of cells in dataset.
- assayFeats: A DataFrame with information about the features in the assay.
- h5obj: A File object from the h5py package.
- grp: Current active group in the hierarchy.
- """
-
- def __init__(self, h5_fn, is_filtered: bool = True, filtering_cutoff: int = 500):
- self.h5obj = h5py.File(h5_fn, mode="r")
- self.grp = None
- self.validBarcodeIdx = None
- super().__init__(self._handle_version())
- if is_filtered:
- self.validBarcodeIdx = np.array(range(self.nCells))
- else:
- self.validBarcodeIdx = self._get_valid_barcodes(filtering_cutoff)
- self.nCells = len(self.validBarcodeIdx)
-
- def _handle_version(self):
- root_key = list(self.h5obj.keys())[0]
- self.grp = self.h5obj[root_key]
- if root_key == "matrix":
- grps = {
- "feature_ids": "features/id",
- "feature_names": "features/name",
- "feature_types": "features/feature_type",
- "cell_names": "barcodes",
- }
- else:
- grps = {
- "feature_ids": "genes",
- "feature_names": "gene_names",
- "feature_types": None,
- "cell_names": "barcodes",
- }
- return grps
-
- def _get_valid_barcodes(
- self, filtering_cutoff: int, batch_size: int = 1000
- ) -> np.ndarray:
- valid_idx = []
- test_counter = 0
- indptr = self.grp["indptr"][:]
- for s in tqdmbar(
- range(0, len(indptr) - 1, batch_size),
- desc=f"Filtering out background barcodes", # noqa: F541
- ):
- idx = indptr[s : s + batch_size + 1]
- data = self.grp["data"][idx[0] : idx[-1]]
- indices = self.grp["indices"][idx[0] : idx[-1]]
- cell_idx = np.repeat(range(len(idx) - 1), np.diff(idx))
- mat = coo_matrix(
- (data, (cell_idx, indices)), shape=(len(idx) - 1, self.nFeatures)
- )
- valid_idx.append(np.array(mat.sum(axis=1)).T[0] > filtering_cutoff)
- test_counter += data.shape[0]
- assert test_counter == self.grp["data"].shape[0]
- assert len(indptr) == (s + len(idx))
- return np.where(np.hstack(valid_idx))[0]
-
- def _read_dataset(self, key: Optional[str] = None):
- return [x.decode("UTF-8") for x in self.grp[self.grpNames[key]][:]]
-
- def cell_names(self) -> List[str]:
- """Returns a list of names of the cells in the dataset."""
- vals = np.array(self._read_dataset("cell_names"))
- if self.validBarcodeIdx is not None:
- vals = vals[self.validBarcodeIdx]
- return list(vals)
-
- # noinspection DuplicatedCode
- def consume(
- self, batch_size: int, lines_in_mem: int = None
- ) -> Generator[coo_matrix, None, None]:
- indptr = self.grp["indptr"][:]
- for s in range(0, len(self.validBarcodeIdx), batch_size):
- v_pos = self.validBarcodeIdx[s : s + batch_size]
- idx = [np.arange(x, y) for x, y in zip(indptr[v_pos], indptr[v_pos + 1])]
- cell_idx = np.repeat(np.arange(len(idx)), [len(x) for x in idx])
- idx = np.hstack(idx)
- data = self.grp["data"][idx[0] : idx[-1] + 1]
- data = data[idx - idx[0]]
- indices = self.grp["indices"][idx[0] : idx[-1] + 1]
- indices = indices[idx - idx[0]]
- yield coo_matrix(
- (data, (cell_idx, indices)), shape=(len(v_pos), self.nFeatures)
- )
-
- def close(self) -> None:
- """Closes file connection."""
- self.h5obj.close()
-
-
-class CrDirReader(CrReader):
- """A class to read in CellRanger (Cr) data, in the form of a directory.
-
- Subclass of CrReader.
-
- Args:
- loc (str): Path for the directory containing the cellranger output.
- mtx_separator (str): Column delimiter in the MTX file (Default value: ' ')
- index_offset (int): This value is added to each feature index (Default value: -1)
-
- Attributes:
- loc: Path for the directory containing the cellranger output.
- matFn: The file name for the matrix file.
- sep (str): Column delimiter in the MTX file (Default value: ' ')
- indexOffset (int): This value is added to each feature index (Default value: -1)
- """
-
- def __init__(
- self,
- loc,
- mtx_separator: str = " ",
- index_offset: int = -1,
- is_filtered: bool = True,
- filtering_cutoff: int = 500,
- ):
- self.loc: str = loc.rstrip("/") + "/"
- self.matFn = None
- self.sep = mtx_separator
- self.indexOffset = index_offset
- self.validBarcodeIdx = None
- super().__init__(self._handle_version())
- if is_filtered:
- self.validBarcodeIdx = np.array(range(self.nCells))
- self.validBarcodeIdx -= self.indexOffset
- else:
- self.validBarcodeIdx = self._get_valid_barcodes(filtering_cutoff)
- self.nCells = len(self.validBarcodeIdx)
-
- def _handle_version(self):
- show_error = False
- if os.path.isfile(self.loc + "matrix.mtx.gz"):
- self.matFn = self.loc + "matrix.mtx.gz"
- elif os.path.isfile(self.loc + "matrix.mtx"):
- self.matFn = self.loc + "matrix.mtx"
- else:
- show_error = True
- if os.path.isfile(self.loc + "features.tsv.gz"):
- feat_fn = "features.tsv.gz"
- elif os.path.isfile(self.loc + "features.tsv"):
- feat_fn = "features.tsv"
- elif os.path.isfile(self.loc + "genes.tsv.gz"):
- feat_fn = "genes.tsv.gz"
- elif os.path.isfile(self.loc + "genes.tsv"):
- feat_fn = "genes.tsv"
- elif os.path.isfile(self.loc + "peaks.bed"):
- feat_fn = "peaks.bed"
- elif os.path.isfile(self.loc + "peaks.bed.gz"):
- feat_fn = "peaks.bed.gz"
- else:
- feat_fn = None
- show_error = True
- if os.path.isfile(self.loc + "barcodes.tsv.gz"):
- cell_fn = "barcodes.tsv.gz"
- elif os.path.isfile(self.loc + "barcodes.tsv"):
- cell_fn = "barcodes.tsv"
- else:
- cell_fn = None
- show_error = True
- if show_error:
- raise IOError(
- "ERROR: Couldn't find either of these expected combinations of files:\n"
- "\t- matrix.mtx, barcodes.tsv and genes.tsv\n"
- "\t- matrix.mtx.gz, barcodes.tsv.gz and features.tsv.gz\n"
- "Please make sure that you have not compressed or uncompressed the Cellranger output files "
- "manually"
- )
- return {
- "feature_ids": (feat_fn, 0),
- "feature_names": (feat_fn, 1),
- "feature_types": (feat_fn, 2),
- "cell_names": (cell_fn, 0),
- }
-
- def _read_dataset(self, key: Optional[str] = None):
- try:
- vals = [
- x.split("\t")[self.grpNames[key][1]]
- for x in read_file(self.loc + self.grpNames[key][0])
- ]
- except IndexError:
- logger.warning(
- f"{key} extraction failed from {self.grpNames[key][0]} "
- f"in column {self.grpNames[key][1]}",
- flush=True,
- )
- vals = None
- return vals
-
- def read_header(self) -> pd.DataFrame:
- header = pd.read_csv(
- self.matFn,
- comment="%",
- sep=self.sep,
- header=None,
- nrows=1,
- names=["nFeatures", "nCells", "nCounts"],
- )
- if header["nCells"][0] == 0 and self.nCells > 0:
- raise ValueError(
- "ERROR: Barcode count in MTX header is 0 but barcodes are present in the barcodes file"
- )
- if header["nCells"][0] > 0 and self.nCells == 0:
- raise ValueError(
- "ERROR: Barcode count in MTX header is greater than 0 but no barcodes are present in the barcodes file"
- )
- if header["nCells"][0] == 0 and self.nCells == 0:
- raise ValueError(
- "ERROR: Barcode count in MTX header and barcodes file is 0. No data to read"
- )
- return header
-
- def process_batch(self, dfs: List[pd.DataFrame], filtering_cutoff: int) -> np.array:
- """Returns a list of valid barcodes after filtering out background barcodes for a given batch.
-
- Args:
- dfs: A Polar DataFrame containing a chunk of data from the MTX file.
- filtering_cutoff: The cutoff value for filtering out background barcodes
- """
- pl_dfs = [pl.DataFrame(df) for df in dfs]
- pl_dfs = pl.concat(pl_dfs)
- dfs_ = pl_dfs.group_by("barcode").agg(pl.sum("count"))
- dfs_ = dfs_.filter(pl.col("count") > filtering_cutoff)
- return np.sort(dfs_["barcode"])
-
- def _get_valid_barcodes(
- self,
- filtering_cutoff: int,
- batch_size: int = int(10e3),
- lines_in_mem: int = int(10e5),
- ) -> np.ndarray:
- """Returns a list of valid barcodes after filtering out background barcodes.
-
- Args:
- filtering_cutoff: The cutoff value for filtering out background barcodes.
- batch_size: The number of barcodes to process in each batch.
- lines_in_mem: The number of lines to read into memory
- """
- test_counter = 0
- matrixIO = pd.read_csv(
- self.matFn,
- comment="%",
- sep=self.sep,
- header=0,
- chunksize=lines_in_mem,
- names=["gene", "barcode", "count"],
- )
-
- header = self.read_header()
- nChunks = math.ceil(header["nCounts"][0] / lines_in_mem)
- test_counter = 0
- valid_idx = []
- start = 1
-
- dfs = []
- for chunk in tqdmbar(
- # range(nChunks),
- matrixIO,
- total=nChunks,
- desc="Filtering out background barcodes",
- ):
- if (
- (chunk.iloc[-1]["barcode"] - start) >= batch_size
- ): # If the last "cell id" is greater than the start + batch size
- # Filter rows in the current chunk that belong to the current batch
- idx = np.array(
- chunk["barcode"].values < (batch_size + start)
- ) # This is the crucial line. This makes sure that if any cell ID is spread over multiple chunks, it is not missed, as any cell ID that is less than the batch size + start is included.
- # If no rows belong to the current batch, move to the next batch.
- if idx.sum() == 0:
- dfs.append(chunk)
- start += batch_size
- test_counter += len(chunk)
- continue
- # Process the rows belonging to the current batch
- mask_pos = np.where(idx)[0]
- mask_neg = np.where(~idx)[0]
- dfs.append(chunk.iloc[mask_pos])
- valid_idx.append(self.process_batch(dfs, filtering_cutoff))
- # Prepare for the next batch
- del dfs
- dfs = [chunk.iloc[mask_neg]]
- start += batch_size
- else:
- # If we haven't reached the batch boundary, accumulate the chunk
- dfs.append(chunk)
- test_counter += len(chunk)
- # Process any remaining data after the main loop
- if len(dfs) > 0:
- valid_idx.append(self.process_batch(dfs, filtering_cutoff))
- # Verify that all rows were processed
- assert test_counter == header["nCounts"][0]
- return np.sort(np.unique(np.hstack(valid_idx)))
-
- def to_sparse(self, a: np.ndarray, dtype) -> coo_matrix:
- """Returns the input data as a sparse (COO) matrix.
-
- Args:
- a: Sparse matrix, contains a chunk of data from the MTX file.
- dtype:
- """
- c = (a[:, 1] - a[0, 1]).astype(int)
- return coo_matrix(
- (
- a[:, 2],
- (
- c,
- (a[:, 0] + self.indexOffset).astype(int),
- ),
- ),
- shape=(c[-1] + 1, self.nFeatures),
- dtype=dtype,
- )
-
- def cell_names(self) -> List[str]:
- """Returns a list of names of the cells in the dataset."""
- vals = np.array(self._read_dataset("cell_names"))
- if self.validBarcodeIdx is not None:
- vals = vals[(self.validBarcodeIdx + self.indexOffset)]
- return list(vals)
-
- def rename_batches(self, collect: List[pd.DataFrame]) -> List:
- collect = [pl.DataFrame(df) for df in collect]
- df = pl.concat(collect)
- barcodes = np.array(df["barcode"])
- count_hash = {}
- for i, x in enumerate(np.unique(barcodes)):
- count_hash[x] = i
- cell_idx = np.array([count_hash[x] for x in barcodes])
- df = df.with_columns([pl.Series("barcode", cell_idx)])
- return np.array(df)
-
- # noinspection DuplicatedCode
- def consume(
- self,
- batch_size: int,
- lines_in_mem: int = int(1e5),
- dtype=np.uint32,
- ) -> Generator[coo_matrix, None, None]:
- """Yields chunks of data from the MTX file.
-
- Args:
- batch_size: The number of barcodes to process in each batch.
- lines_in_mem: The number of lines to read into memory.
- dtype: The data type of the matrix.
- """
- matrixIO = pd.read_csv(
- self.matFn,
- comment="%",
- sep=self.sep,
- header=0,
- chunksize=lines_in_mem,
- names=["gene", "barcode", "count"],
- )
- unique_list = []
- collect = []
- for chunk in matrixIO:
- chunk = chunk[chunk["barcode"].isin(self.validBarcodeIdx)]
- in_uniques = np.unique(chunk["barcode"].values)
- unique_list.extend(in_uniques)
- unique_list = list(set(unique_list))
- if len(unique_list) > batch_size:
- diff = batch_size - (len(unique_list) - len(in_uniques))
- mask_pos = in_uniques[:diff]
- mask_neg = in_uniques[diff:]
- extra = chunk[chunk["barcode"].isin(mask_pos)]
- collect.append(extra)
- collect = self.rename_batches(collect)
- mtx = self.to_sparse(np.array(collect), dtype=dtype)
- yield mtx
- left_out = chunk[chunk["barcode"].isin(mask_neg)]
- collect = []
- unique_list = list(mask_neg)
- collect.append(left_out)
- else:
- collect.append(chunk)
- if len(collect) > 0:
- collect = self.rename_batches(collect)
- mtx = self.to_sparse(np.array(collect), dtype=dtype)
- yield mtx
-
-
-class H5adReader:
- """A class to read in data from a H5ad file (h5 file with AnnData
- information).
-
- Args:
- h5ad_fn: Path to H5AD file
- cell_attrs_key: H5 group under which cell attributes are saved.(Default value: 'obs')
- feature_attrs_key: H5 group under which feature attributes are saved.(Default value: 'var')
- cell_ids_key: Key in `obs` group that contains unique cell IDs. By default the index will be used.
- feature_ids_key: Key in `var` group that contains unique feature IDs. By default the index will be used.
- feature_name_key: Key in `var` group that contains feature names. (Default: gene_short_name)
- matrix_key: Group where in the sparse matrix resides (default: 'X')
- category_names_key: Looks up this group and replaces the values in `var` and 'obs' child datasets with the
- corresponding index value within this group.
- dtype: Numpy dtype of the matrix data. This dtype is enforced when streaming the data through `consume`
- method. (Default value: Automatically determined)
-
- Attributes:
- h5: A File object from the h5py package.
- matrixKey: Group where in the sparse matrix resides (default: 'X')
- cellAttrsKey: Group wherein the cell attributes are present
- featureAttrsKey: Group wherein the feature attributes are present
- groupCodes: Used to ensure compatibility with different AnnData versions.
- nFeatures: Number of features in dataset.
- nCells: Number of cells in dataset.
- cellIdsKey: Key in `obs` group that contains unique cell IDs. By default the index will be used.
- featIdsKey: Key in `var` group that contains unique feature IDs. By default the index will be used.
- featNamesKey: Key in `var` group that contains feature names. (Default: gene_short_name)
- catNamesKey: Looks up this group and replaces the values in `var` and 'obs' child datasets with the
- corresponding index value within this group.
- matrixDtype: dtype of the matrix containing the data (as indicated by matrix_key)
- """
-
- def __init__(
- self,
- h5ad_fn: str,
- cell_attrs_key: str = "obs",
- cell_ids_key: str = "_index",
- feature_attrs_key: str = "var",
- feature_ids_key: str = "_index",
- feature_name_key: str = "gene_short_name",
- matrix_key: str = "X",
- obsm_attrs_key: str = "obsm",
- category_names_key: str = "__categories",
- dtype: str = None,
- ):
- self.h5 = h5py.File(h5ad_fn, mode="r")
- self.matrixKey = matrix_key
- self.cellAttrsKey, self.featureAttrsKey, self.obsmAttrsKey = (
- cell_attrs_key,
- feature_attrs_key,
- obsm_attrs_key,
- )
- self.groupCodes = {
- self.cellAttrsKey: self._validate_group(self.cellAttrsKey),
- self.featureAttrsKey: self._validate_group(self.featureAttrsKey),
- self.obsmAttrsKey: self._validate_group(self.obsmAttrsKey),
- self.matrixKey: self._validate_group(self.matrixKey),
- }
- self.nCells, self.nFeatures = (
- self._get_n(self.cellAttrsKey),
- self._get_n(self.featureAttrsKey),
- )
- self.cellIdsKey = self._fix_name_key(self.cellAttrsKey, cell_ids_key)
- self.featIdsKey = self._fix_name_key(self.featureAttrsKey, feature_ids_key)
- self.featNamesKey = feature_name_key
- self.catNamesKey = category_names_key
- self.matrixDtype = self._get_matrix_dtype() if dtype is None else dtype
-
- def _validate_group(self, group: str) -> int:
- if group not in self.h5:
- logger.warning(f"`{group}` group not found in the H5ad file")
- ret_val = 0
- elif isinstance(self.h5[group], h5py.Dataset):
- ret_val = 1
- elif isinstance(self.h5[group], h5py.Group):
- ret_val = 2
- else:
- logger.warning(
- f"`{group}` slot in H5ad file is not of Dataset or Group type. "
- f"Due to this, no information in `{group}` can be used"
- )
- ret_val = 0
- if ret_val == 2:
- if len(self.h5[group].keys()) == 0:
- logger.warning(f"`{group}` slot in H5ad file is empty.")
- ret_val = 0
- elif (
- len(
- set(
- [
- self.h5[group][x].shape[0]
- for x in self.h5[group].keys()
- if isinstance(self.h5[group][x], h5py.Dataset)
- ]
- )
- )
- > 1
- ):
- if sorted(self.h5[group].keys()) != ["data", "indices", "indptr"]:
- logger.info(
- f"`{group}` slot in H5ad file has unequal sized child groups"
- )
- return ret_val
-
- def _get_matrix_dtype(self):
- if self.groupCodes[self.matrixKey] == 1:
- return self.h5[self.matrixKey].dtype
- elif self.groupCodes[self.matrixKey] == 2:
- return self.h5[self.matrixKey]["data"].dtype
- else:
- raise ValueError(
- f"ERROR: {self.matrixKey} is neither Dataset or Group type. Will not consume data"
- )
-
- def _check_exists(self, group: str, key: str) -> bool:
- if group in self.groupCodes:
- group_code = self.groupCodes[group]
- else:
- group_code = self._validate_group(group)
- self.groupCodes[group] = group_code
- if group_code == 1:
- if key in list(self.h5[group].dtype.names):
- return True
- if group_code == 2:
- if key in self.h5[group].keys():
- return True
- return False
-
- def _fix_name_key(self, group: str, key: str) -> str:
- if self._check_exists(group, key) is False:
- if key.startswith("_"):
- temp_key = key[1:]
- if self._check_exists(group, temp_key):
- return temp_key
- return key
-
- def _get_n(self, group: str) -> int:
- if self.groupCodes[group] == 0:
- if self._check_exists(self.matrixKey, "shape"):
- return self.h5[self.matrixKey]["shape"][0]
- else:
- raise KeyError(
- f"ERROR: `{group}` not found and `shape` key is missing in the {self.matrixKey} group. "
- f"Aborting read process."
- )
- elif self.groupCodes[group] == 1:
- return self.h5[group].shape[0]
- else:
- for i in self.h5[group].keys():
- if isinstance(self.h5[group][i], h5py.Dataset):
- return self.h5[group][i].shape[0]
- raise KeyError(
- f"ERROR: `{group}` key doesn't contain any child node of Dataset type."
- f"Aborting because unexpected H5ad format."
- )
-
- def cell_ids(self) -> np.ndarray:
- """Returns a list of cell IDs."""
- if self._check_exists(self.cellAttrsKey, self.cellIdsKey):
- if self.groupCodes[self.cellAttrsKey] == 1:
- return self.h5[self.cellAttrsKey][self.cellIdsKey]
- else:
- return self.h5[self.cellAttrsKey][self.cellIdsKey][:]
- logger.warning(f"Could not find cells ids key: {self.cellIdsKey} in `obs`.")
- return np.array([f"cell_{x}" for x in range(self.nCells)])
-
- # noinspection DuplicatedCode
- def feat_ids(self) -> np.ndarray:
- """Returns a list of feature IDs."""
- if self._check_exists(self.featureAttrsKey, self.featIdsKey):
- if self.groupCodes[self.featureAttrsKey] == 1:
- return self.h5[self.featureAttrsKey][self.featIdsKey]
- else:
- return self.h5[self.featureAttrsKey][self.featIdsKey][:]
- logger.warning(
- f"Could not find feature ids key: {self.featIdsKey} in {self.featureAttrsKey}."
- )
- return np.array([f"feature_{x}" for x in range(self.nFeatures)])
-
- # noinspection DuplicatedCode
- def feat_names(self) -> np.ndarray:
- """Returns a list of feature names."""
- if self._check_exists(self.featureAttrsKey, self.featNamesKey):
- values = self.h5[self.featureAttrsKey][self.featNamesKey]
- return self._replace_category_values(
- values, self.featNamesKey, self.featureAttrsKey
- ).astype(object)
- logger.warning(
- f"Could not find feature names key: {self.featNamesKey} in self.featureAttrsKey."
- )
- return self.feat_ids()
-
- def _replace_category_values(
- self, v: Union[np.ndarray, h5py.Group, h5py.Dataset], key: str, group: str
- ) -> np.ndarray:
- # check if v is a Group with codes + categories structure
- if isinstance(v, h5py.Group):
- if "codes" in v and "categories" in v:
- codes = v["codes"][:]
- categories = v["categories"][:]
- try:
- return np.array([categories[x] for x in codes])
- except (IndexError, TypeError):
- logger.warning(f"Failed to decode categorical data for {key}")
- return np.array([f"feature_{x}" for x in range(len(codes))])
- else:
- # It's a Group but doesn't have the expected structure, try to read it as dataset
- logger.warning(f"{key} is a Group but missing 'codes' or 'categories', attempting to extract data")
- return v[:]
-
- # if v is a Dataset
- if isinstance(v, h5py.Dataset):
- v = v[:]
-
- if self.catNamesKey is not None:
- if self._check_exists(group, self.catNamesKey):
- cat_g = self.h5[group][self.catNamesKey]
- if isinstance(cat_g, h5py.Group):
- if key in cat_g:
- c = cat_g[key][:]
- try:
- return np.array([c[x] for x in v])
- except (IndexError, TypeError):
- return v
- if "uns" in self.h5:
- if key + "_categories" in self.h5["uns"]:
- c = self.h5["uns"][key + "_categories"][:]
- try:
- return np.array([c[x] for x in v])
- except (IndexError, TypeError):
- return v
- return v
-
- def _get_col_data(
- self, group: str, ignore_keys: List[str]
- ) -> Generator[Tuple[str, np.ndarray], None, None]:
- if self.groupCodes[group] == 1:
- for i in tqdmbar(
- self.h5[group].dtype.names,
- desc=f"Reading attributes from group {group}",
- ):
- if i in ignore_keys:
- continue
- yield i, self._replace_category_values(self.h5[group][i][:], i, group)
- if self.groupCodes[group] == 2:
- for i in tqdmbar(
- self.h5[group].keys(), desc=f"Reading attributes from group {group}"
- ):
- if i in ignore_keys:
- continue
- if isinstance(self.h5[group][i], h5py.Dataset):
- yield (
- i,
- self._replace_category_values(self.h5[group][i][:], i, group),
- )
-
- def _get_obsm_data(
- self, group: str
- ) -> Generator[Tuple[str, np.ndarray], None, None]:
- if self.groupCodes[group] == 2:
- for i in tqdmbar(
- self.h5[group].keys(), desc=f"Reading attributes from group {group}"
- ):
- g = self.h5[group][i]
- if g.shape[0] != self.nCells:
- logger.error(
- f"Dimension of {i}({g.shape}) is not correct."
- f" Will not save this specific slot into Zarr."
- )
- continue
- if isinstance(g, h5py.Dataset):
- for j in range(g.shape[1]):
- yield f"{i}{j+1}", g[:, j]
- else:
- logger.warning(
- f"Reading of obsm failed because it either does not exist or is not in expected format" # noqa: F541
- )
-
- def get_cell_columns(self) -> Generator[Tuple[str, np.ndarray], None, None]:
- """Creates a Generator that yields the cell columns."""
- for i, j in self._get_col_data(self.cellAttrsKey, [self.cellIdsKey]):
- yield i, j
- for i, j in self._get_obsm_data(self.obsmAttrsKey):
- yield i, j
-
- def get_feat_columns(self) -> Generator[Tuple[str, np.ndarray], None, None]:
- """Creates a Generator that yields the feature columns."""
- for i, j in self._get_col_data(
- self.featureAttrsKey, [self.featIdsKey, self.featNamesKey]
- ):
- yield i, j
-
- # noinspection DuplicatedCode
- def consume_dataset(
- self, batch_size: int = 1000
- ) -> Generator[coo_matrix, None, None]:
- """Returns a generator that yield chunks of data."""
- dset = self.h5[self.matrixKey]
- s = 0
- for e in range(batch_size, dset.shape[0] + batch_size, batch_size):
- if e > dset.shape[0]:
- e = dset.shape[0]
- yield coo_matrix(dset[s:e])
- s = e
-
- def consume_group(self, batch_size: int) -> Generator[coo_matrix, None, None]:
- """Returns a generator that yield chunks of data."""
- grp = self.h5[self.matrixKey]
- s = 0
- for ind_n in range(0, self.nCells, batch_size):
- i = grp["indptr"][ind_n : ind_n + batch_size]
- e = i[-1]
- if s != 0:
- idx = np.array([s] + list(i))
- idx = idx - idx[0]
- else:
- idx = np.array(i)
- n = idx.shape[0] - 1
- nidx = np.repeat(range(n), np.diff(idx).astype("int32"))
- yield coo_matrix(
- (grp["data"][s:e], (nidx, grp["indices"][s:e])),
- shape=(n, self.nFeatures),
- )
- s = e
-
- def consume(self, batch_size: int):
- """Returns a generator that yield chunks of data."""
- if self.groupCodes[self.matrixKey] == 1:
- return self.consume_dataset(batch_size)
- elif self.groupCodes[self.matrixKey] == 2:
- return self.consume_group(batch_size)
-
-
-class NaboH5Reader:
- """A class to read in data in the form of a Nabo H5 file.
-
- Args:
- h5_fn: Path to H5 file.
-
- Attributes:
- h5: A File object from the h5py package.
- nCells: Number of cells in dataset.
- nFeatures: Number of features in dataset.
- """
-
- def __init__(self, h5_fn: str):
- self.h5 = h5py.File(h5_fn, mode="r")
- self._check_integrity()
- self.nCells = self.h5["names"]["cells"].shape[0]
- self.nFeatures = self.h5["names"]["genes"].shape[0]
-
- def _check_integrity(self) -> bool:
- for i in ["cell_data", "gene_data", "names"]:
- if i not in self.h5:
- raise KeyError(f"ERROR: Expected group: {i} is missing in the H5 file")
- return True
-
- def cell_ids(self) -> List[str]:
- """Returns a list of cell IDs."""
- return [x.decode("UTF-8") for x in self.h5["names"]["cells"][:]]
-
- def feat_ids(self) -> np.ndarray:
- """Returns a list of feature IDs."""
- return np.array([f"feature_{x}" for x in range(self.nFeatures)])
-
- def feat_names(self) -> List[str]:
- """Returns a list of feature names."""
- return [
- x.decode("UTF-8").rsplit("_", 1)[0] for x in self.h5["names"]["genes"][:]
- ]
-
- def consume(self, batch_size: int = 100) -> Generator[np.ndarray, None, None]:
- """Returns a generator that yield chunks of data."""
- batch = []
- for i in self.h5["cell_data"]:
- a = np.zeros(self.nFeatures).astype(int)
- v = self.h5["cell_data"][i][:][::-1]
- a[v["idx"]] = v["val"]
- batch.append(a)
- if len(batch) >= batch_size:
- yield np.array(batch)
- batch = []
- if len(batch) > 0:
- yield np.array(batch)
-
-
-class LoomReader:
- """A class to read in data in the form of a Loom file.
-
- Args:
- loom_fn: Path to loom format file.
- matrix_key: Child node under HDF5 file root wherein the chunked matrix is stored. (Default value: matrix).
- This matrix is expected to be of form (nFeatures x nCells)
- cell_attrs_key: Child node under the HDF5 file wherein the cell attributes are stored.
- (Default value: col_attrs)
- cell_names_key: Child node under the `cell_attrs_key` wherein the cell names are stored.
- (Default value: obs_names)
- feature_attrs_key: Child node under the HDF5 file wherein the feature/gene attributes are stored.
- (Default value: row_attrs)
- feature_names_key: Child node under the `feature_attrs_key` wherein the feature/gene names are stored.
- (Default value: var_names)
- feature_ids_key: Child node under the `feature_attrs_key` wherein the feature/gene ids are stored.
- (Default value: None)
- dtype: Numpy dtype of the matrix data. This dtype is enforced when streaming the data through `consume`
- method. (Default value: Automatically determined)
-
- Attributes:
- h5: A File object from the h5py package.
- matrixKey: Child node under HDF5 file root wherein the chunked matrix is stored.
- cellAttrsKey: Child node under the HDF5 file wherein the cell attributes are stored.
- featureAttrsKey: Child node under the HDF5 file wherein the feature/gene attributes are stored.
- cellNamesKey: Child node under the `cell_attrs_key` wherein the cell names are stored.
- featureNamesKey: Child node under the `feature_attrs_key` wherein the feature/gene names are stored.
- featureIdsKey: Child node under the `feature_attrs_key` wherein the feature/gene ids are stored.
- matrixDtype: Numpy dtype of the matrix data.
- nFeatures: Number of features in dataset.
- nCells: Number of cells in dataset.
- """
-
- def __init__(
- self,
- loom_fn: str,
- matrix_key: str = "matrix",
- cell_attrs_key="col_attrs",
- cell_names_key: str = "obs_names",
- feature_attrs_key: str = "row_attrs",
- feature_names_key: str = "var_names",
- feature_ids_key: str = None,
- dtype: str = None,
- ) -> None:
- self.h5 = h5py.File(loom_fn, mode="r")
- self.matrixKey = matrix_key
- self.cellAttrsKey, self.featureAttrsKey = cell_attrs_key, feature_attrs_key
- self.cellNamesKey, self.featureNamesKey = cell_names_key, feature_names_key
- self.featureIdsKey = feature_ids_key
- self.matrixDtype = self.h5[self.matrixKey].dtype if dtype is None else dtype
- self._check_integrity()
- self.nFeatures, self.nCells = self.h5[self.matrixKey].shape
-
- def _check_integrity(self) -> bool:
- if self.matrixKey not in self.h5:
- raise KeyError(
- f"ERROR: Matrix key (location): {self.matrixKey} is missing in the H5 file"
- )
- if self.cellAttrsKey not in self.h5:
- logger.warning(
- f"Cell attributes are missing. Key {self.cellAttrsKey} was not found"
- )
- if self.featureAttrsKey not in self.h5:
- logger.warning(
- f"Feature attributes are missing. Key {self.featureAttrsKey} was not found"
- )
- return True
-
- def cell_names(self) -> List[str]:
- """Returns a list of names of the cells in the dataset."""
- if self.cellAttrsKey not in self.h5:
- pass
- elif self.cellNamesKey not in self.h5[self.cellAttrsKey]:
- logger.warning(
- f"Cell names/ids key ({self.cellNamesKey}) is missing in attributes"
- )
- else:
- return self.h5[self.cellAttrsKey][self.cellNamesKey][:]
- return [f"cell_{x}" for x in range(self.nCells)]
-
- def cell_ids(self) -> List[str]:
- """Returns a list of cell IDs."""
- return self.cell_names()
-
- def _stream_attrs(
- self, key, ignore
- ) -> Generator[Tuple[str, np.ndarray], None, None]:
- if key in self.h5:
- for i in tqdmbar(self.h5[key].keys(), desc=f"Reading {key} attributes"):
- if i in [ignore]:
- continue
- vals = self.h5[key][i][:]
- if vals.dtype.names is None:
- yield i, vals
- else:
- # Attribute is a structured array
- for j in vals.dtype.names:
- yield i + "_" + str(j), vals[j]
-
- def get_cell_attrs(self) -> Generator[Tuple[str, np.ndarray], None, None]:
- """Returns a Generator that yields the cells' attributes."""
- return self._stream_attrs(self.cellAttrsKey, [self.cellNamesKey])
-
- def feature_names(self) -> List[str]:
- """Returns a list of feature names."""
- if self.featureAttrsKey not in self.h5:
- pass
- elif self.featureNamesKey not in self.h5[self.featureAttrsKey]:
- logger.warning(
- f"Feature names key ({self.featureNamesKey}) is missing in attributes"
- )
- else:
- return self.h5[self.featureAttrsKey][self.featureNamesKey][:]
- return [f"feature_{x}" for x in range(self.nFeatures)]
-
- def feature_ids(self) -> List[str]:
- """Returns a list of feature IDs."""
- if self.featureAttrsKey not in self.h5:
- pass
- elif self.featureIdsKey is None:
- pass
- elif self.featureIdsKey not in self.h5[self.featureAttrsKey]:
- logger.warning(
- f"Feature names key ({self.featureIdsKey}) is missing in attributes"
- )
- else:
- return self.h5[self.featureAttrsKey][self.featureIdsKey][:]
- return [f"feature_{x}" for x in range(self.nFeatures)]
-
- def get_feature_attrs(self) -> Generator[Tuple[str, np.ndarray], None, None]:
- """Returns a Generator that yields the features' attributes."""
- return self._stream_attrs(
- self.featureAttrsKey, [self.featureIdsKey, self.featureNamesKey]
- )
-
- def consume(self, batch_size: int = 1000) -> Generator[np.ndarray, None, None]:
- """Returns a generator that yield chunks of data."""
- dset = self.h5[self.matrixKey]
- s = 0
- for e in range(batch_size, dset.shape[1] + batch_size, batch_size):
- if e > dset.shape[1]:
- e = dset.shape[1]
- yield coo_matrix(dset[:, s:e]).T.astype(self.matrixDtype)
- s = e
-
-
-class CSVReader:
- """A class to read in data from a CSV file.
-
- Args:
- csv_fn: Path to the CSV file
- has_header: Does the CSV file has a header. (Default value: True)
- id_column: The column number which contains row name. (Default value: None)
- rows_are_cells: If True then each row represents a cell and hence each column is a feature. If False then each
- row is feature and each column in a cell
- sep: The column separator in the CSV file (Default value: ',')
- skip_rows: Number of rows to skip from the top of the file. (Default value: 0)
- skip_cols: Names of columns to skip. Must be provided as a list even if just one column.
- (Default value: None)
- cell_data_cols: Names of columns to include in cell metadata rather than count matrix. Must be provided as a
- list even if just one column. (Default value: None)
- batch_size: Number of lines to read at a time. Decrease this value if you have too many columns.
- (Default value: 50,000)
- pandas_kwargs: A dictionary of keyword arguments to be passed to Pandas read_csv function.
-
- Attributes:
- nFeatures: Number of features in dataset.
- nCells: Number of cells in dataset.
- """
-
- def __init__(
- self,
- csv_fn: str,
- has_header: bool = True,
- id_column: Optional[int] = None,
- rows_are_cells: bool = True,
- sep: str = ",",
- skip_rows: int = 0,
- skip_cols: Optional[List[str]] = None,
- cell_data_cols: Optional[List[str]] = None,
- batch_size=10000,
- pandas_kwargs: Optional[dict] = None,
- ):
- self._fn = csv_fn
- if rows_are_cells is False:
- raise NotImplementedError(
- "Currently Scarf supports only those CSV files where cells are along the rows"
- )
- if pandas_kwargs is None:
- pandas_kwargs = {}
- else:
- if not isinstance(pandas_kwargs, dict):
- logger.error("")
- if has_header is False:
- has_header = None
- else:
- has_header = 0
- self.pandas_kwargs = pandas_kwargs
- self.pandas_kwargs["sep"] = sep
- self.pandas_kwargs["header"] = has_header
- self.pandas_kwargs["skiprows"] = skip_rows
- self.pandas_kwargs["chunksize"] = batch_size
- self.pandas_kwargs["index_col"] = id_column
-
- if skip_cols is None:
- self.skipCols = []
- else:
- self.skipCols = skip_cols
- if cell_data_cols is None:
- self.cellDataCols = []
- else:
- self.cellDataCols = cell_data_cols
- (
- self.nCells,
- self.nFeatures,
- self.cellIds,
- self.featureIds,
- self.keepCols,
- self.cellDataDtypes,
- self.cellDataIdx,
- ) = self._consistency_check()
-
- def _get_streamer(self) -> Generator:
- return pd.read_csv(self._fn, **self.pandas_kwargs)
-
- def _consistency_check(
- self,
- ) -> Tuple[
- int,
- int,
- Optional[np.ndarray],
- Optional[np.ndarray],
- Optional[List[int]],
- Optional[List[np.dtype]],
- Optional[List[int]],
- ]:
- stream = self._get_streamer()
- n_cells = 0
- n_features = 0
- feature_ids = None
- cell_data_dtypes = None
- cell_data_idx = None
- if self.pandas_kwargs["index_col"] is None:
- cell_ids = None
- else:
- cell_ids = []
- for df in tqdmbar(stream, desc="Performing CSV file consistency check"):
- n_cells += df.shape[0]
- if n_features == 0:
- n_features = df.shape[1]
- if self.pandas_kwargs["header"] is not None:
- feature_ids = df.columns.values
- if len(feature_ids) != n_features:
- raise ValueError(
- "Header length not same as number of features. This can happen if you did not"
- " skip the right number of rows."
- )
- if len(self.cellDataCols) > 0:
- cell_data_dtypes = list(df[self.cellDataCols].dtypes.values)
- cell_data_idx = [
- n
- for n, x in enumerate(feature_ids)
- if x in self.cellDataCols
- ]
- else:
- if n_features != df.shape[1]:
- raise ValueError(
- "Number of columns changed in the CSV during consistency check."
- " Maybe a problem with the delimiter."
- )
- if cell_ids is not None:
- cell_ids = np.ndarray(cell_ids)
- keep_cols = None
- if feature_ids is not None:
- skip_names = list(set(self.skipCols).union(self.cellDataCols))
- if len(skip_names) > 0:
- keep_cols = [
- n for n, x in enumerate(feature_ids) if x not in skip_names
- ]
- feature_ids = feature_ids[keep_cols]
- n_features = len(keep_cols)
- return (
- n_cells,
- n_features,
- cell_ids,
- feature_ids,
- keep_cols,
- cell_data_dtypes,
- cell_data_idx,
- )
-
- def _n_features(self):
- return np.array([f"feature_{x}" for x in range(self.nFeatures)])
-
- def cell_ids(self) -> np.ndarray:
- """Returns a list of cell IDs."""
- if self.cellIds is None:
- return np.array([f"cell_{x}" for x in range(self.nCells)])
- else:
- return self.cellIds
-
- def feature_ids(self) -> np.ndarray:
- """Returns a list of feature IDs."""
- if self.featureIds is None:
- return np.array([f"feature_{x}" for x in range(self.nFeatures)])
- else:
- return self.featureIds
-
- def consume(self) -> Generator[Tuple[np.ndarray, Optional[np.ndarray]], None, None]:
- """Returns a generator that yield chunks of data."""
- stream = self._get_streamer()
- if self.keepCols is None:
- for df in stream:
- yield df.values, None
- else:
- if self.cellDataIdx is not None:
- for df in stream:
- yield df.values[:, self.keepCols], df.values[:, self.cellDataIdx]
- else:
- for df in stream:
- yield df.values[:, self.keepCols], None
diff --git a/scarf/readers/__init__.py b/scarf/readers/__init__.py
new file mode 100644
index 00000000..d9ec4c01
--- /dev/null
+++ b/scarf/readers/__init__.py
@@ -0,0 +1,96 @@
+"""A collection of classes for reading in different data formats.
+
+- Classes:
+ - CrH5Reader: A class to read in CellRanger (Cr) data, in the form of an H5 file.
+ - CrDirReader: A class to read in CellRanger (Cr) data, in the form of a directory.
+ - CrReader: A class to read in CellRanger (Cr) data.
+ - H5adReader: A class to read in data in the form of a H5ad file (h5 file with AnnData information).
+ - LoomReader: A class to read in data in the form of a Loom file.
+"""
+
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+from ._text import get_file_handle as get_file_handle
+from ._text import read_file as read_file
+
+if TYPE_CHECKING:
+ from .cellranger import CrDirReader, CrH5Reader, CrReader
+ from .csv import CSVReader
+ from .h5ad import H5adInspectResult, H5adReader, inspect_h5ad
+ from .loom import LoomReader
+ from .mtx import MtxCandidate, MtxReader, inspect_mtx
+ from .seurat import SeuratInspectResult, SeuratReader, inspect_seurat
+
+__all__ = [
+ "CrH5Reader",
+ "CrDirReader",
+ "CrReader",
+ "H5adInspectResult",
+ "H5adReader",
+ "inspect_h5ad",
+ "MtxCandidate",
+ "MtxReader",
+ "inspect_mtx",
+ "SeuratInspectResult",
+ "SeuratReader",
+ "inspect_seurat",
+ "LoomReader",
+ "CSVReader",
+]
+
+_LAZY_EXPORTS = {
+ "CrDirReader": "cellranger",
+ "CrH5Reader": "cellranger",
+ "CrReader": "cellranger",
+ "CSVReader": "csv",
+ "H5adInspectResult": "h5ad",
+ "H5adReader": "h5ad",
+ "inspect_h5ad": "h5ad",
+ "MtxCandidate": "mtx",
+ "MtxReader": "mtx",
+ "inspect_mtx": "mtx",
+ "SeuratInspectResult": "seurat",
+ "SeuratReader": "seurat",
+ "inspect_seurat": "seurat",
+ "LoomReader": "loom",
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+
+def _normalize_export_metadata(value: Any) -> None:
+ value.__module__ = __name__
+ if not isinstance(value, type):
+ return
+ for descriptor in value.__dict__.values():
+ if isinstance(descriptor, classmethod | staticmethod):
+ descriptor.__func__.__module__ = __name__
+ elif callable(descriptor) and hasattr(descriptor, "__module__"):
+ descriptor.__module__ = __name__
+
+
+def __getattr__(name: str) -> Any:
+ module_name = _LAZY_EXPORTS.get(name)
+ if module_name is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ module = import_module(f".{module_name}", __name__)
+ for export_name, export_module in _LAZY_EXPORTS.items():
+ if export_module != module_name:
+ continue
+ value = getattr(module, export_name)
+ _normalize_export_metadata(value)
+ globals()[export_name] = value
+ return globals()[name]
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()) | set(__all__))
+
+
+for _function in (get_file_handle, read_file):
+ _function.__module__ = __name__
+del _function
diff --git a/scarf/readers/_assay_names.py b/scarf/readers/_assay_names.py
new file mode 100644
index 00000000..0ace5412
--- /dev/null
+++ b/scarf/readers/_assay_names.py
@@ -0,0 +1,59 @@
+from collections.abc import Mapping, Sequence
+
+import pandas as pd
+
+
+AUTO_ASSAY_NAMES: dict[str, str] = {
+ "Gene Expression": "RNA",
+ "mRNA": "RNA",
+ "Peaks": "ATAC",
+ "Antibody Capture": "ADT",
+ "AbSeq": "ADT",
+ "CRISPR Guide Capture": "CRISPR",
+ "CRISPR": "CRISPR",
+ "CRISPR Direct Capture": "CRISPR",
+ "Multiplexing Capture": "HTO",
+ "Antigen Capture": "ANTIGEN",
+ "ANTIGEN": "ANTIGEN",
+ "Custom": "CUSTOM",
+ "CUSTOM": "CUSTOM",
+ "RNA": "RNA",
+ "ADT": "ADT",
+ "HTO": "HTO",
+}
+
+
+def make_feat_table_from_types(feature_types: Sequence[str]) -> pd.DataFrame:
+ if len(feature_types) == 0:
+ raise ValueError("Cannot build an assay table without features")
+
+ spans: list[tuple[str, int, int]] = []
+ previous = feature_types[0]
+ start = 0
+ for index, feature_type in enumerate(feature_types[1:], 1):
+ if feature_type != previous:
+ spans.append((previous, start, index))
+ start = index
+ previous = feature_type
+ spans.append((previous, start, len(feature_types)))
+
+ table = pd.DataFrame(spans, columns=["type", "start", "end"])
+ table.index = [f"ASSAY{index + 1}" for index in table.index]
+ table["nFeatures"] = table.end - table.start
+ return table.T
+
+
+def auto_name_feat_table(
+ assay_feats: pd.DataFrame,
+ name_map: Mapping[str, str] | None = None,
+) -> pd.DataFrame:
+ mapping = dict(AUTO_ASSAY_NAMES)
+ if name_map is not None:
+ mapping.update(name_map)
+
+ renamed = assay_feats.copy()
+ renamed.columns = [
+ mapping.get(str(feature_type), mapping.get(str(key), str(key)))
+ for key, feature_type in renamed.T["type"].items()
+ ]
+ return renamed
diff --git a/scarf/readers/_h5ad_inspect.py b/scarf/readers/_h5ad_inspect.py
new file mode 100644
index 00000000..d82cb086
--- /dev/null
+++ b/scarf/readers/_h5ad_inspect.py
@@ -0,0 +1,640 @@
+from dataclasses import dataclass
+from typing import Any
+
+import h5py
+import numpy as np
+
+from ..utils.logging import logger
+from ._assay_names import auto_name_feat_table, make_feat_table_from_types
+
+
+_FEATURE_ID_KEYS = (
+ "_index",
+ "gene_ids",
+ "gene_id",
+ "ensembl_id",
+ "feature_ids",
+ "feature_id",
+ "id",
+ "index",
+)
+_FEATURE_NAME_KEYS = (
+ "gene_symbol",
+ "gene_symbols",
+ "gene_name",
+ "feature_name",
+ "gene_short_name",
+ "name",
+ "index",
+)
+_CELL_ID_KEYS = (
+ "_index",
+ "index",
+ "cell_id",
+ "cell_ids",
+ "barcode",
+ "barcodes",
+)
+_MATRIX_COMPONENTS = frozenset({"data", "indices", "indptr"})
+_NON_MATRIX_PREFIXES = (
+ "obs/",
+ "var/",
+ "obsm/",
+ "varm/",
+ "obsp/",
+ "varp/",
+ "uns/",
+ "raw/var/",
+ "raw/varm/",
+)
+
+
+@dataclass(frozen=True)
+class H5adInspectResult:
+ h5adFn: str
+ matrixKey: str
+ matrixCandidates: tuple[str, ...]
+ matrixEncoding: str
+ cellAttrsKey: str
+ cellIdsKey: str
+ featureAttrsKey: str
+ featureIdsKey: str
+ featureNameKey: str
+ categoryNamesKey: str
+ assaySplitKey: str | None
+ suggestedAssays: dict[str, int]
+ layers: tuple[str, ...]
+ title: str | None
+ description: str | None
+ nCells: int
+ nFeatures: int
+
+ def to_reader_kwargs(self) -> dict[str, Any]:
+ return {
+ "h5ad_fn": self.h5adFn,
+ "cell_attrs_key": self.cellAttrsKey,
+ "cell_ids_key": self.cellIdsKey,
+ "feature_attrs_key": self.featureAttrsKey,
+ "feature_ids_key": self.featureIdsKey,
+ "feature_name_key": self.featureNameKey,
+ "matrix_key": self.matrixKey,
+ "category_names_key": self.categoryNamesKey,
+ }
+
+
+@dataclass(frozen=True)
+class _MatrixCandidate:
+ key: str
+ encoding: str
+ shape: tuple[int, int]
+ integerLike: bool
+
+ @property
+ def isSparse(self) -> bool:
+ return self.encoding in {"csr", "csc"}
+
+
+def _as_text(value: Any) -> str:
+ if isinstance(value, bytes | np.bytes_):
+ return value.decode("utf-8")
+ return str(value)
+
+
+def _node_length(node: h5py.Group | h5py.Dataset | None) -> int | None:
+ if node is None:
+ return None
+ if isinstance(node, h5py.Dataset):
+ return int(node.shape[0]) if node.shape else None
+
+ index_key = node.attrs.get("_index")
+ if index_key is not None:
+ index_name = _as_text(index_key)
+ if index_name in node:
+ values = node[index_name]
+ if isinstance(values, h5py.Dataset) and values.shape:
+ return int(values.shape[0])
+ if isinstance(values, h5py.Group) and "codes" in values:
+ return int(values["codes"].shape[0])
+
+ for key in ("_index", "index"):
+ if key in node:
+ values = node[key]
+ if isinstance(values, h5py.Dataset) and values.shape:
+ return int(values.shape[0])
+ if isinstance(values, h5py.Group) and "codes" in values:
+ return int(values["codes"].shape[0])
+
+ for values in node.values():
+ if isinstance(values, h5py.Dataset) and values.shape:
+ return int(values.shape[0])
+ if (
+ isinstance(values, h5py.Group)
+ and "codes" in values
+ and isinstance(values["codes"], h5py.Dataset)
+ ):
+ return int(values["codes"].shape[0])
+ return None
+
+
+def _sparse_encoding(group: h5py.Group) -> str | None:
+ encoding = group.attrs.get("encoding-type")
+ if encoding is None:
+ encoding = group.attrs.get("h5sparse_format")
+ if encoding is None:
+ return "csr"
+
+ normalized = _as_text(encoding).lower()
+ if normalized in {"csr", "csr_matrix"}:
+ return "csr"
+ if normalized in {"csc", "csc_matrix"}:
+ return "csc"
+ return None
+
+
+def _stored_shape(group: h5py.Group) -> tuple[int, int] | None:
+ shape: Any = group.attrs.get("shape")
+ if shape is None:
+ shape = group.attrs.get("h5sparse_shape")
+ if shape is None and "shape" in group and isinstance(group["shape"], h5py.Dataset):
+ shape = group["shape"][:]
+ if shape is None:
+ return None
+ values = np.asarray(shape).reshape(-1)
+ if values.size != 2:
+ return None
+ return int(values[0]), int(values[1])
+
+
+def _infer_sparse_shape(
+ h5: h5py.File,
+ key: str,
+ group: h5py.Group,
+ encoding: str,
+) -> tuple[int, int]:
+ stored = _stored_shape(group)
+ if stored is not None:
+ return stored
+
+ compressed_axis = int(group["indptr"].shape[0] - 1)
+ indices = group["indices"]
+ observed_axis = int(np.max(indices[:])) + 1 if indices.shape[0] else 0
+ obs_length = _node_length(h5.get("obs"))
+ feature_group = h5.get("raw/var" if key.startswith("raw/") else "var")
+ feature_length = _node_length(feature_group)
+ if encoding == "csr":
+ return compressed_axis, feature_length or observed_axis
+ return obs_length or observed_axis, compressed_axis
+
+
+def _is_integer_like(dataset: h5py.Dataset) -> bool:
+ if np.issubdtype(dataset.dtype, np.integer):
+ return True
+ sample = np.asarray(dataset[: min(101, dataset.shape[0])])
+ if sample.size == 0 or not np.issubdtype(sample.dtype, np.number):
+ return False
+ return bool(
+ np.all(np.isfinite(sample))
+ and np.allclose(sample, np.round(sample), rtol=0, atol=1e-8)
+ )
+
+
+def _dense_is_integer_like(dataset: h5py.Dataset) -> bool:
+ if np.issubdtype(dataset.dtype, np.integer):
+ return True
+ rows = min(10, dataset.shape[0])
+ columns = min(100, dataset.shape[1])
+ sample = np.asarray(dataset[:rows, :columns])
+ if sample.size == 0 or not np.issubdtype(sample.dtype, np.number):
+ return False
+ return bool(
+ np.all(np.isfinite(sample))
+ and np.allclose(sample, np.round(sample), rtol=0, atol=1e-8)
+ )
+
+
+def _is_matrix_path(key: str) -> bool:
+ return not key.startswith(_NON_MATRIX_PREFIXES)
+
+
+def _matrix_candidates(h5: h5py.File) -> list[_MatrixCandidate]:
+ candidates: list[_MatrixCandidate] = []
+
+ def visit(key: str, node: h5py.Group | h5py.Dataset) -> None:
+ if not _is_matrix_path(key):
+ return
+ if isinstance(node, h5py.Group) and _MATRIX_COMPONENTS.issubset(node.keys()):
+ encoding = _sparse_encoding(node)
+ if encoding is None:
+ logger.warning(
+ f"Ignoring sparse matrix candidate with unknown encoding: {key}"
+ )
+ return
+ candidates.append(
+ _MatrixCandidate(
+ key=key,
+ encoding=encoding,
+ shape=_infer_sparse_shape(h5, key, node, encoding),
+ integerLike=_is_integer_like(node["data"]),
+ )
+ )
+ elif isinstance(node, h5py.Dataset) and len(node.shape) == 2:
+ if not np.issubdtype(node.dtype, np.number):
+ return
+ candidates.append(
+ _MatrixCandidate(
+ key=key,
+ encoding="dense",
+ shape=(int(node.shape[0]), int(node.shape[1])),
+ integerLike=_dense_is_integer_like(node),
+ )
+ )
+
+ h5.visititems(visit)
+
+ def rank(candidate: _MatrixCandidate) -> tuple[int, int, int, int, str]:
+ # Integer-like values signal raw counts and take priority over storage
+ # layout, so a dense count matrix outranks a sparse transformed layer.
+ integer = 0 if candidate.integerLike else 1
+ raw = 0 if candidate.key.startswith("raw/") else 1
+ canonical = 0 if candidate.key == "X" else 1
+ sparse = 0 if candidate.isSparse else 1
+ return integer, raw, canonical, sparse, candidate.key
+
+ return sorted(candidates, key=rank)
+
+
+def _read_column(
+ node: h5py.Group | h5py.Dataset,
+ key: str,
+) -> np.ndarray | None:
+ if isinstance(node, h5py.Dataset):
+ if node.dtype.names is None or key not in node.dtype.names:
+ return None
+ return np.asarray(node[key])
+ if key not in node:
+ return None
+
+ values = node[key]
+ if isinstance(values, h5py.Dataset):
+ raw = np.asarray(values[:])
+ for category_group_name in ("__categories", "categories"):
+ if category_group_name not in node:
+ continue
+ category_group = node[category_group_name]
+ if isinstance(category_group, h5py.Group) and key in category_group:
+ if not np.issubdtype(raw.dtype, np.integer):
+ return raw
+ categories = np.asarray(category_group[key][:])
+ valid = (raw >= 0) & (raw < len(categories))
+ decoded = np.empty(raw.shape, dtype=object)
+ decoded[valid] = categories[raw[valid]]
+ decoded[~valid] = None
+ return decoded
+ return raw
+
+ if isinstance(values, h5py.Group) and {"codes", "categories"}.issubset(
+ values.keys()
+ ):
+ codes = np.asarray(values["codes"][:])
+ categories = np.asarray(values["categories"][:])
+ valid = (codes >= 0) & (codes < len(categories))
+ decoded = np.empty(codes.shape, dtype=object)
+ decoded[valid] = categories[codes[valid]]
+ decoded[~valid] = None
+ return decoded
+ return None
+
+
+def _column_names(node: h5py.Group | h5py.Dataset) -> list[str]:
+ if isinstance(node, h5py.Dataset):
+ return list(node.dtype.names or ())
+ return [
+ key
+ for key in node.keys()
+ if key not in {"__categories", "categories"}
+ and (
+ isinstance(node[key], h5py.Dataset)
+ or (
+ isinstance(node[key], h5py.Group)
+ and {"codes", "categories"}.issubset(node[key].keys())
+ )
+ )
+ ]
+
+
+def _index_key(node: h5py.Group | h5py.Dataset) -> str | None:
+ """Return the dataframe index dataset name recorded by AnnData."""
+ if not isinstance(node, h5py.Group):
+ return None
+ index_attr = node.attrs.get("_index")
+ if index_attr is None:
+ return None
+ return _as_text(index_attr)
+
+
+def _matching_key(names: list[str], preferences: tuple[str, ...]) -> str | None:
+ normalized = {name.lower(): name for name in names}
+ for preferred in preferences:
+ if preferred in normalized:
+ return normalized[preferred]
+ return None
+
+
+def _is_string_column(values: np.ndarray) -> bool:
+ if values.dtype.kind in {"S", "U"}:
+ return True
+ if values.dtype.kind != "O":
+ return False
+ return all(
+ value is None or isinstance(value, str | bytes | np.str_ | np.bytes_)
+ for value in values[:100]
+ )
+
+
+def _is_unique(values: np.ndarray, expected_length: int) -> bool:
+ if values.ndim != 1 or len(values) != expected_length:
+ return False
+ normalized = np.asarray(
+ [None if value is None else _as_text(value) for value in values],
+ dtype=object,
+ )
+ return len(set(normalized.tolist())) == expected_length
+
+
+def _find_cell_ids(
+ node: h5py.Group | h5py.Dataset | None,
+ n_cells: int,
+) -> str:
+ if node is None:
+ return "_index"
+ names = _column_names(node)
+ index_key = _index_key(node)
+ if index_key is not None and index_key in names:
+ values = _read_column(node, index_key)
+ if values is not None and _is_unique(values, n_cells):
+ return index_key
+ preferred = _matching_key(names, _CELL_ID_KEYS)
+ if preferred is not None:
+ values = _read_column(node, preferred)
+ if values is not None and _is_unique(values, n_cells):
+ return preferred
+
+ candidates: list[tuple[str, bool]] = []
+ for name in names:
+ values = _read_column(node, name)
+ if values is None or not _is_unique(values, n_cells):
+ continue
+ candidates.append((name, _is_string_column(values)))
+ if candidates:
+ candidates.sort(key=lambda item: (not item[1], item[0]))
+ return candidates[0][0]
+
+ logger.warning("No unique cell ID column found; generated IDs will be used")
+ return "_index"
+
+
+def _mean_text_length(values: np.ndarray) -> float:
+ sample = [
+ _as_text(value)
+ for value in values[: min(100, len(values))]
+ if value is not None
+ ]
+ if not sample:
+ return 0
+ return float(np.mean([len(value) for value in sample]))
+
+
+def _find_features(
+ node: h5py.Group | h5py.Dataset,
+ n_features: int,
+) -> tuple[str, str]:
+ names = _column_names(node)
+ index_key = _index_key(node)
+ id_key: str | None = None
+ if index_key is not None and index_key in names:
+ values = _read_column(node, index_key)
+ if values is not None and _is_unique(values, n_features):
+ id_key = index_key
+ if id_key is None:
+ id_key = _matching_key(names, _FEATURE_ID_KEYS)
+ if id_key is not None:
+ values = _read_column(node, id_key)
+ if values is None or not _is_unique(values, n_features):
+ id_key = None
+
+ name_key = _matching_key(names, _FEATURE_NAME_KEYS)
+ if name_key is not None:
+ values = _read_column(node, name_key)
+ if (
+ values is None
+ or values.ndim != 1
+ or len(values) != n_features
+ or not _is_string_column(values)
+ ):
+ name_key = None
+
+ string_columns: list[tuple[str, float, bool]] = []
+ for name in names:
+ values = _read_column(node, name)
+ if (
+ values is None
+ or values.ndim != 1
+ or len(values) != n_features
+ or not _is_string_column(values)
+ ):
+ continue
+ string_columns.append(
+ (name, _mean_text_length(values), _is_unique(values, n_features))
+ )
+
+ if id_key is None:
+ unique_columns = [column for column in string_columns if column[2]]
+ if unique_columns:
+ id_key = max(unique_columns, key=lambda column: (column[1], column[0]))[0]
+ if name_key is None and string_columns:
+ alternatives = [
+ column for column in string_columns if column[0] != id_key
+ ] or string_columns
+ name_key = min(alternatives, key=lambda column: (column[1], column[0]))[0]
+
+ if id_key is None and name_key is None:
+ logger.warning("No feature ID or name column found; generated IDs will be used")
+ return "_index", "_index"
+ if id_key is None:
+ id_key = name_key
+ if name_key is None:
+ name_key = id_key
+ assert id_key is not None
+ assert name_key is not None
+ return id_key, name_key
+
+
+def _category_names_key(
+ *nodes: h5py.Group | h5py.Dataset | None,
+) -> str:
+ for node in nodes:
+ if isinstance(node, h5py.Group) and "categories" in node:
+ if isinstance(node["categories"], h5py.Group):
+ return "categories"
+ return "__categories"
+
+
+def _read_text_scalar(
+ h5: h5py.File,
+ key: str,
+ max_length: int,
+) -> str | None:
+ if key not in h5:
+ return None
+ node = h5[key]
+ if not isinstance(node, h5py.Dataset):
+ return None
+ value = node[()]
+ text = _as_text(value)
+ return text[:max_length]
+
+
+def _feature_group_for(key: str) -> str:
+ return "raw/var" if key.startswith("raw/") else "var"
+
+
+def _select_matrix(
+ h5: h5py.File,
+ candidates: list[_MatrixCandidate],
+) -> tuple[_MatrixCandidate, str]:
+ obs_length = _node_length(h5.get("obs"))
+ lengths = {
+ "raw/var": _node_length(h5.get("raw/var")),
+ "var": _node_length(h5.get("var")),
+ }
+
+ for candidate in candidates:
+ n_cells, n_features = candidate.shape
+ if obs_length is not None and obs_length != n_cells:
+ logger.warning(
+ f"Ignoring matrix candidate {candidate.key}: "
+ f"{n_cells} rows do not match obs length {obs_length}"
+ )
+ continue
+
+ own_key = _feature_group_for(candidate.key)
+ own_length = lengths[own_key]
+ if own_length == n_features:
+ return candidate, own_key
+ if own_length is not None:
+ logger.warning(
+ f"Ignoring matrix candidate {candidate.key}: feature metadata "
+ f"group `{own_key}` length {own_length} does not match feature "
+ f"count {n_features}"
+ )
+ continue
+
+ # The conventional feature group for this matrix is absent. Fall back to
+ # a dimension-matched group, otherwise keep the conventional key so the
+ # reader generates feature IDs rather than borrowing an unrelated table.
+ other_key = "var" if own_key == "raw/var" else "raw/var"
+ if lengths[other_key] == n_features:
+ return candidate, other_key
+ if lengths[other_key] is None:
+ return candidate, own_key
+ logger.warning(
+ f"Ignoring matrix candidate {candidate.key}: no feature metadata "
+ f"group has {n_features} rows"
+ )
+
+ raise ValueError("No matrix candidate matches the obs and var dimensions")
+
+
+def inspect_h5ad(h5ad_fn: str) -> H5adInspectResult:
+ """Report the matrix and metadata layout of an H5AD file.
+
+ Args:
+ h5ad_fn: Path to the H5AD file.
+
+ Returns:
+ Keys, shape, and column names needed to configure
+ :class:`~scarf.readers.H5adReader`.
+ """
+ with h5py.File(h5ad_fn, mode="r") as h5:
+ candidates = _matrix_candidates(h5)
+ if not candidates:
+ raise ValueError("No sparse or numeric 2D matrix found in the H5AD file")
+
+ matrix, feature_attrs_key = _select_matrix(h5, candidates)
+ n_cells, n_features = matrix.shape
+ cell_node = h5.get("obs")
+ feature_node = h5.get(feature_attrs_key)
+ if feature_node is None or not isinstance(
+ feature_node, h5py.Group | h5py.Dataset
+ ):
+ feature_ids_key = "_index"
+ feature_name_key = "_index"
+ else:
+ feature_ids_key, feature_name_key = _find_features(feature_node, n_features)
+
+ cell_ids_key = _find_cell_ids(
+ cell_node if isinstance(cell_node, h5py.Group | h5py.Dataset) else None,
+ n_cells,
+ )
+ category_names_key = _category_names_key(
+ cell_node if isinstance(cell_node, h5py.Group | h5py.Dataset) else None,
+ feature_node
+ if isinstance(feature_node, h5py.Group | h5py.Dataset)
+ else None,
+ )
+
+ assay_split_key = None
+ suggested_assays: dict[str, int] = {}
+ if isinstance(feature_node, h5py.Group | h5py.Dataset):
+ feature_columns = _column_names(feature_node)
+ assay_split_key = _matching_key(
+ feature_columns, ("feature_types", "feature_type")
+ )
+ if assay_split_key is not None:
+ values = _read_column(feature_node, assay_split_key)
+ if values is None or len(values) != n_features:
+ assay_split_key = None
+ else:
+ feature_types = [_as_text(value) for value in values]
+ assay_table = auto_name_feat_table(
+ make_feat_table_from_types(feature_types)
+ )
+ for assay_name in dict.fromkeys(assay_table.columns):
+ selected = assay_table[assay_name]
+ count = (
+ int(selected.loc["nFeatures"].sum())
+ if selected.ndim == 2
+ else int(selected.loc["nFeatures"])
+ )
+ suggested_assays[str(assay_name)] = count
+
+ layers_node = h5.get("layers")
+ layers = (
+ tuple(sorted(layers_node.keys()))
+ if isinstance(layers_node, h5py.Group)
+ else ()
+ )
+ title = _read_text_scalar(h5, "uns/title", 500)
+ description = _read_text_scalar(h5, "uns/description", 10_000)
+ if description is None:
+ description = _read_text_scalar(h5, "uns/citation", 10_000)
+
+ return H5adInspectResult(
+ h5adFn=h5ad_fn,
+ matrixKey=matrix.key,
+ matrixCandidates=tuple(candidate.key for candidate in candidates),
+ matrixEncoding=matrix.encoding,
+ cellAttrsKey="obs",
+ cellIdsKey=cell_ids_key,
+ featureAttrsKey=feature_attrs_key,
+ featureIdsKey=feature_ids_key,
+ featureNameKey=feature_name_key,
+ categoryNamesKey=category_names_key,
+ assaySplitKey=assay_split_key,
+ suggestedAssays=suggested_assays,
+ layers=layers,
+ title=title,
+ description=description,
+ nCells=n_cells,
+ nFeatures=n_features,
+ )
diff --git a/scarf/readers/_rds/__init__.py b/scarf/readers/_rds/__init__.py
new file mode 100644
index 00000000..f214d86c
--- /dev/null
+++ b/scarf/readers/_rds/__init__.py
@@ -0,0 +1,73 @@
+"""Private bounded-memory R serialization reader."""
+
+from ._lazy import LazyAtomicVector, LazyStringVector, LazyVector
+from ._model import (
+ AltRepValue,
+ BytecodeValue,
+ ClosureValue,
+ EnvironmentValue,
+ ExternalPointerValue,
+ PairValue,
+ PersistentValue,
+ PromiseValue,
+ RNode,
+ RdsDocument,
+ WeakReferenceValue,
+ get_attribute,
+ get_named,
+ get_slot,
+ iter_attributes,
+ iter_named,
+ iter_pairlist,
+ symbol_name,
+)
+from ._parser import open_rds
+from ._types import (
+ R_INT_NA,
+ RdsClosedError,
+ RdsCompression,
+ RdsEncoding,
+ RdsError,
+ RdsFormatError,
+ RdsLimitError,
+ RdsLimits,
+ RdsMetadata,
+ RdsSourceMetadata,
+ RType,
+)
+
+__all__ = [
+ "R_INT_NA",
+ "AltRepValue",
+ "BytecodeValue",
+ "ClosureValue",
+ "EnvironmentValue",
+ "ExternalPointerValue",
+ "LazyAtomicVector",
+ "LazyStringVector",
+ "LazyVector",
+ "PairValue",
+ "PersistentValue",
+ "PromiseValue",
+ "RNode",
+ "RType",
+ "RdsClosedError",
+ "RdsCompression",
+ "RdsDocument",
+ "RdsEncoding",
+ "RdsError",
+ "RdsFormatError",
+ "RdsLimitError",
+ "RdsLimits",
+ "RdsMetadata",
+ "RdsSourceMetadata",
+ "WeakReferenceValue",
+ "get_attribute",
+ "get_named",
+ "get_slot",
+ "iter_attributes",
+ "iter_named",
+ "iter_pairlist",
+ "open_rds",
+ "symbol_name",
+]
diff --git a/scarf/readers/_rds/_lazy.py b/scarf/readers/_rds/_lazy.py
new file mode 100644
index 00000000..7451db86
--- /dev/null
+++ b/scarf/readers/_rds/_lazy.py
@@ -0,0 +1,281 @@
+import codecs
+import struct
+from collections.abc import Iterator, Sequence
+from typing import Any, overload
+
+import numpy as np
+import numpy.typing as npt
+
+from ._storage import RandomAccessStorage
+from ._types import RType
+
+
+_STRING_DESCRIPTOR = struct.Struct(" int:
+ raise NotImplementedError
+
+ def __len__(self) -> int:
+ return self.length
+
+
+class LazyAtomicVector(LazyVector):
+ """A numeric or raw vector backed by a byte range."""
+
+ _DTYPES: dict[RType, str] = {
+ RType.LOGICAL: "i4",
+ RType.INTEGER: "i4",
+ RType.REAL: "f8",
+ RType.COMPLEX: "c16",
+ RType.RAW: "u1",
+ }
+
+ def __init__(
+ self,
+ storage: RandomAccessStorage,
+ *,
+ offset: int,
+ length: int,
+ r_type: RType,
+ byte_order: str,
+ path: str,
+ ) -> None:
+ if r_type not in self._DTYPES:
+ raise ValueError(f"{r_type.name} is not an atomic vector type")
+ self._storage = storage
+ self.offset = offset
+ self._length = length
+ self.r_type = r_type
+ self.byte_order = byte_order
+ self.path = path
+ self._dtype = np.dtype(self._DTYPES[r_type])
+
+ @property
+ def length(self) -> int:
+ return self._length
+
+ @property
+ def dtype(self) -> np.dtype[Any]:
+ return self._dtype
+
+ @property
+ def item_size(self) -> int:
+ return self._dtype.itemsize
+
+ @property
+ def nbytes(self) -> int:
+ return self.length * self.item_size
+
+ def read_block(self, start: int, stop: int) -> npt.NDArray[Any]:
+ """Read a half-open element range into a native-order array."""
+ if start < 0 or stop < start or stop > self.length:
+ raise IndexError(
+ f"invalid block [{start}:{stop}] for vector of length {self.length}"
+ )
+ count = stop - start
+ data = self._storage.read_at(
+ self.offset + start * self.item_size,
+ count * self.item_size,
+ path=self.path,
+ )
+ wire_dtype = self._dtype.newbyteorder("<" if self.byte_order == "<" else ">")
+ return np.frombuffer(data, dtype=wire_dtype, count=count).astype(
+ self._dtype,
+ copy=True,
+ )
+
+ def iter_blocks(self, block_size: int) -> Iterator[npt.NDArray[Any]]:
+ if block_size <= 0:
+ raise ValueError("block_size must be positive")
+ for start in range(0, self.length, block_size):
+ yield self.read_block(start, min(self.length, start + block_size))
+
+ def materialize(self) -> npt.NDArray[Any]:
+ """Read the complete vector when the caller explicitly requests it."""
+ return self.read_block(0, self.length)
+
+ @overload
+ def __getitem__(self, key: int) -> Any: ...
+
+ @overload
+ def __getitem__(self, key: slice) -> npt.NDArray[Any]: ...
+
+ def __getitem__(self, key: int | slice) -> Any | npt.NDArray[Any]:
+ if isinstance(key, slice):
+ start, stop, step = key.indices(self.length)
+ if step != 1:
+ raise ValueError("lazy vector slices require a step of 1")
+ return self.read_block(start, stop)
+ index = key
+ if index < 0:
+ index += self.length
+ if index < 0 or index >= self.length:
+ raise IndexError("vector index out of range")
+ return self.read_block(index, index + 1)[0].item()
+
+
+class LazyStringVector(LazyVector, Sequence[str | bytes | None]):
+ """A character vector backed by a fixed-width descriptor index."""
+
+ def __init__(
+ self,
+ index_storage: RandomAccessStorage,
+ *,
+ descriptor_offset: int,
+ length: int,
+ payload_storage: RandomAccessStorage,
+ decoded_storage: RandomAccessStorage | None,
+ default_encoding: str | None,
+ path: str,
+ element_attributes: dict[int, Any] | None = None,
+ ) -> None:
+ self._index_storage = index_storage
+ self.descriptor_offset = descriptor_offset
+ self._length = length
+ self._payload_storage = payload_storage
+ self._decoded_storage = decoded_storage
+ self.default_encoding = default_encoding
+ self.path = path
+ self._element_attributes = element_attributes or {}
+
+ @property
+ def length(self) -> int:
+ return self._length
+
+ def _descriptor(self, index: int) -> tuple[int, int, int, int]:
+ data = self._index_storage.read_at(
+ self.descriptor_offset + index * STRING_DESCRIPTOR_BYTES,
+ STRING_DESCRIPTOR_BYTES,
+ path=f"{self.path}[{index}]",
+ )
+ return _STRING_DESCRIPTOR.unpack(data)
+
+ def raw(self, index: int) -> bytes | None:
+ """Read one string without decoding it."""
+ if index < 0:
+ index += self.length
+ if index < 0 or index >= self.length:
+ raise IndexError("string vector index out of range")
+ source, offset, length, _gp = self._descriptor(index)
+ if length == -1:
+ return None
+ storage = (
+ self._payload_storage
+ if source == STRING_SOURCE_PAYLOAD
+ else self._decoded_storage
+ )
+ if storage is None:
+ raise RuntimeError("decoded string storage is unavailable")
+ return storage.read_at(offset, length, path=f"{self.path}[{index}]")
+
+ def gp_flags(self, index: int) -> int:
+ """Return the serialized general-purpose character flags."""
+ if index < 0:
+ index += self.length
+ if index < 0 or index >= self.length:
+ raise IndexError("string vector index out of range")
+ return self._descriptor(index)[3]
+
+ def element_attributes(self, index: int) -> Any | None:
+ """Return uncommon attributes attached to a character element."""
+ if index < 0:
+ index += self.length
+ if index < 0 or index >= self.length:
+ raise IndexError("string vector index out of range")
+ return self._element_attributes.get(index)
+
+ def _decode(self, raw: bytes | None, gp: int) -> str | bytes | None:
+ if raw is None:
+ return None
+ if gp & (1 << 1):
+ return raw
+ if gp & (1 << 2):
+ return raw.decode("latin-1")
+ if gp & (1 << 3):
+ return raw.decode("utf-8", errors="surrogateescape")
+ if gp & (1 << 6):
+ return raw.decode("ascii", errors="surrogateescape")
+ encoding = self.default_encoding or "utf-8"
+ try:
+ codecs.lookup(encoding)
+ except LookupError:
+ encoding = "utf-8"
+ return raw.decode(encoding, errors="surrogateescape")
+
+ def read_block(self, start: int, stop: int) -> list[str | bytes | None]:
+ """Read and decode a half-open string range."""
+ if start < 0 or stop < start or stop > self.length:
+ raise IndexError(
+ f"invalid block [{start}:{stop}] for vector of length {self.length}"
+ )
+ result: list[str | bytes | None] = []
+ for index in range(start, stop):
+ source, offset, length, gp = self._descriptor(index)
+ if length == -1:
+ result.append(None)
+ continue
+ storage = (
+ self._payload_storage
+ if source == STRING_SOURCE_PAYLOAD
+ else self._decoded_storage
+ )
+ if storage is None:
+ raise RuntimeError("decoded string storage is unavailable")
+ raw = storage.read_at(offset, length, path=f"{self.path}[{index}]")
+ result.append(self._decode(raw, gp))
+ return result
+
+ @overload
+ def __getitem__(self, key: int) -> str | bytes | None: ...
+
+ @overload
+ def __getitem__(self, key: slice) -> list[str | bytes | None]: ...
+
+ def __getitem__(
+ self,
+ key: int | slice,
+ ) -> str | bytes | None | list[str | bytes | None]:
+ if isinstance(key, slice):
+ start, stop, step = key.indices(self.length)
+ if step != 1:
+ raise ValueError("lazy string slices require a step of 1")
+ return self.read_block(start, stop)
+ index = key
+ if index < 0:
+ index += self.length
+ if index < 0 or index >= self.length:
+ raise IndexError("string vector index out of range")
+ source, offset, length, gp = self._descriptor(index)
+ if length == -1:
+ return None
+ storage = (
+ self._payload_storage
+ if source == STRING_SOURCE_PAYLOAD
+ else self._decoded_storage
+ )
+ if storage is None:
+ raise RuntimeError("decoded string storage is unavailable")
+ raw = storage.read_at(offset, length, path=f"{self.path}[{index}]")
+ return self._decode(raw, gp)
+
+ def __iter__(self) -> Iterator[str | bytes | None]:
+ for index in range(self.length):
+ yield self[index]
+
+
+def pack_string_descriptor(
+ source: int,
+ offset: int,
+ length: int,
+ gp: int,
+) -> bytes:
+ """Pack one internal string descriptor."""
+ return _STRING_DESCRIPTOR.pack(source, offset, length, gp)
diff --git a/scarf/readers/_rds/_model.py b/scarf/readers/_rds/_model.py
new file mode 100644
index 00000000..d59a3b26
--- /dev/null
+++ b/scarf/readers/_rds/_model.py
@@ -0,0 +1,263 @@
+from collections.abc import Iterator, Sequence
+from dataclasses import dataclass
+from typing import Any
+
+from ._lazy import LazyStringVector
+from ._storage import RandomAccessStorage, TempManager
+from ._types import (
+ RdsClosedError,
+ RdsMetadata,
+ RdsSourceMetadata,
+ RType,
+)
+
+
+@dataclass(frozen=True, slots=True)
+class PairValue:
+ """The CAR and CDR of a pairlist or language node."""
+
+ car: "RNode"
+ cdr: "RNode"
+
+
+@dataclass(frozen=True, slots=True)
+class ClosureValue:
+ """A closure represented without evaluation."""
+
+ environment: "RNode | None"
+ formals: "RNode"
+ body: "RNode"
+
+
+@dataclass(frozen=True, slots=True)
+class PromiseValue:
+ """A promise represented without forcing it."""
+
+ environment: "RNode | None"
+ value: "RNode"
+ expression: "RNode"
+
+
+@dataclass(frozen=True, slots=True)
+class EnvironmentValue:
+ """Serialized fields of an environment."""
+
+ locked: bool
+ enclosure: "RNode"
+ frame: "RNode"
+ hash_table: "RNode"
+
+
+@dataclass(frozen=True, slots=True)
+class ExternalPointerValue:
+ """Opaque external-pointer fields that are safe to inspect."""
+
+ protected: "RNode"
+ tag: "RNode"
+
+
+@dataclass(frozen=True, slots=True)
+class WeakReferenceValue:
+ """Opaque weak-reference state."""
+
+ ready_to_finalize: bool
+ finalize_on_exit: bool
+
+
+@dataclass(frozen=True, slots=True)
+class PersistentValue:
+ """A persistent-hook name represented as inert data."""
+
+ names: LazyStringVector
+
+
+@dataclass(frozen=True, slots=True)
+class BytecodeValue:
+ """Serialized bytecode and constants without execution."""
+
+ code: "RNode"
+ constants: "tuple[RNode, ...]"
+
+
+@dataclass(frozen=True, slots=True)
+class AltRepValue:
+ """An ALTREP constructor description represented as inert data."""
+
+ info: "RNode"
+ state: "RNode"
+ class_name: str | None
+ package_name: str | None
+ known: bool
+
+
+@dataclass(eq=False, slots=True)
+class RNode:
+ """One identity-bearing node in an R serialization graph."""
+
+ type: RType
+ value: Any = None
+ attributes: "RNode | None" = None
+ tag: "RNode | None" = None
+ object: bool = False
+ gp: int = 0
+ path: str = "$"
+ offset: int = 0
+
+ @property
+ def is_null(self) -> bool:
+ return self.type in {RType.NIL, RType.NIL_VALUE}
+
+ def attribute(self, name: str) -> "RNode | None":
+ return get_attribute(self, name)
+
+ def slot(self, name: str) -> "RNode | None":
+ return get_slot(self, name)
+
+ def named(self, name: str) -> "RNode | None":
+ return get_named(self, name)
+
+ def attribute_items(self) -> "Iterator[tuple[str, RNode]]":
+ return iter_attributes(self)
+
+ def named_items(self) -> "Iterator[tuple[str | bytes | None, RNode]]":
+ return iter_named(self)
+
+
+def symbol_name(node: RNode | None) -> str | None:
+ """Return the text carried by a symbol node."""
+ if node is None or node.is_null or node.type is not RType.SYMBOL:
+ return None
+ value = node.value
+ if not isinstance(value, RNode) or value.type is not RType.CHAR:
+ return None
+ if isinstance(value.value, bytes):
+ return value.value.decode("utf-8", errors="surrogateescape")
+ return value.value if isinstance(value.value, str) else None
+
+
+def iter_pairlist(node: RNode | None) -> Iterator[RNode]:
+ """Iterate pairlist cells while rejecting malformed tails and cycles."""
+ current = node
+ seen: set[int] = set()
+ while current is not None and not current.is_null:
+ if current.type not in {RType.PAIRLIST, RType.LANGUAGE, RType.DOTS}:
+ raise TypeError(
+ f"expected a pairlist at {current.path}, found {current.type.name}"
+ )
+ identity = id(current)
+ if identity in seen:
+ raise ValueError(f"cycle in pairlist at {current.path}")
+ seen.add(identity)
+ if not isinstance(current.value, PairValue):
+ raise TypeError(f"pairlist node at {current.path} has no pair value")
+ yield current
+ current = current.value.cdr
+
+
+def iter_attributes(node: RNode) -> Iterator[tuple[str, RNode]]:
+ """Iterate named attributes or S4 slots in serialized order."""
+ for cell in iter_pairlist(node.attributes):
+ name = symbol_name(cell.tag)
+ if name is None:
+ continue
+ pair = cell.value
+ if not isinstance(pair, PairValue):
+ continue
+ yield name, pair.car
+
+
+def get_attribute(node: RNode, name: str) -> RNode | None:
+ """Look up an attribute without materializing unrelated values."""
+ for candidate, value in iter_attributes(node):
+ if candidate == name:
+ return value
+ return None
+
+
+def get_slot(node: RNode, name: str) -> RNode | None:
+ """Look up one S4 slot stored in the attribute pairlist."""
+ return get_attribute(node, name)
+
+
+def _named_values(node: RNode) -> Sequence[RNode]:
+ if node.type not in {RType.VECTOR, RType.EXPRESSION}:
+ raise TypeError(f"{node.type.name} is not a generic named vector")
+ if not isinstance(node.value, tuple):
+ raise TypeError(f"vector node at {node.path} has no node sequence")
+ return node.value
+
+
+def iter_named(node: RNode) -> Iterator[tuple[str | bytes | None, RNode]]:
+ """Iterate a generic vector together with its names attribute."""
+ values = _named_values(node)
+ names_node = get_attribute(node, "names")
+ if names_node is None or names_node.type is not RType.STRING:
+ raise ValueError(f"named vector at {node.path} has no character names")
+ names = names_node.value
+ if not isinstance(names, LazyStringVector):
+ raise TypeError(f"names at {names_node.path} are not lazily indexed")
+ if len(names) != len(values):
+ raise ValueError(
+ f"name count {len(names)} does not match value count {len(values)} "
+ f"at {node.path}"
+ )
+ for index, value in enumerate(values):
+ yield names[index], value
+
+
+def get_named(node: RNode, name: str) -> RNode | None:
+ """Look up the first value with an exact serialized name."""
+ for candidate, value in iter_named(node):
+ if candidate == name:
+ return value
+ return None
+
+
+class RdsDocument:
+ """Own an indexed RDS graph and every backing resource it uses."""
+
+ def __init__(
+ self,
+ root: RNode,
+ *,
+ metadata: RdsMetadata,
+ source: RdsSourceMetadata,
+ temp_manager: TempManager,
+ direct_storage: RandomAccessStorage | None,
+ ) -> None:
+ self.root = root
+ self.metadata = metadata
+ self.source = source
+ self._temp_manager = temp_manager
+ self._direct_storage = direct_storage
+ self._closed = False
+
+ @property
+ def closed(self) -> bool:
+ return self._closed
+
+ @property
+ def temp_paths(self) -> tuple[str, ...]:
+ return tuple(str(path) for path in self._temp_manager.paths)
+
+ def close(self) -> None:
+ """Close all backings and remove owned temporary files."""
+ if self._closed:
+ return
+ self._closed = True
+ if self._direct_storage is not None:
+ self._direct_storage.close()
+ self._temp_manager.close()
+
+ def __enter__(self) -> "RdsDocument":
+ if self.closed:
+ raise RdsClosedError("RDS document is closed", path="$")
+ return self
+
+ def __exit__(
+ self,
+ exc_type: type[BaseException] | None,
+ exc_value: BaseException | None,
+ traceback: object | None,
+ ) -> None:
+ self.close()
diff --git a/scarf/readers/_rds/_parser.py b/scarf/readers/_rds/_parser.py
new file mode 100644
index 00000000..559ea88a
--- /dev/null
+++ b/scarf/readers/_rds/_parser.py
@@ -0,0 +1,1230 @@
+import codecs
+import math
+import os
+import struct
+from dataclasses import dataclass
+
+from ._lazy import (
+ STRING_SOURCE_DECODED,
+ STRING_SOURCE_PAYLOAD,
+ LazyAtomicVector,
+ LazyStringVector,
+ pack_string_descriptor,
+)
+from ._model import (
+ AltRepValue,
+ BytecodeValue,
+ ClosureValue,
+ EnvironmentValue,
+ ExternalPointerValue,
+ PairValue,
+ PersistentValue,
+ PromiseValue,
+ RNode,
+ RdsDocument,
+ WeakReferenceValue,
+ symbol_name,
+)
+from ._storage import (
+ BufferedTempWriter,
+ RandomAccessStorage,
+ RdsInput,
+ TempManager,
+ prepare_payload,
+)
+from ._types import (
+ R_INT_NA,
+ RdsEncoding,
+ RdsFormatError,
+ RdsLimitError,
+ RdsLimits,
+ RdsMetadata,
+ RType,
+)
+
+
+_RDATA_HEADERS = {
+ b"RDA2\n",
+ b"RDA3\n",
+ b"RDB2\n",
+ b"RDB3\n",
+ b"RDX2\n",
+ b"RDX3\n",
+}
+_PAIR_TYPES = {
+ RType.PAIRLIST,
+ RType.LANGUAGE,
+ RType.CLOSURE,
+ RType.PROMISE,
+ RType.DOTS,
+}
+_SINGLETON_TYPES = {
+ RType.NIL,
+ RType.NIL_VALUE,
+ RType.EMPTY_ENVIRONMENT,
+ RType.BASE_ENVIRONMENT,
+ RType.GLOBAL_ENVIRONMENT,
+ RType.UNBOUND_VALUE,
+ RType.MISSING_ARGUMENT,
+ RType.BASE_NAMESPACE,
+}
+_KNOWN_ALTREP = {
+ "compact_intseq",
+ "compact_realseq",
+ "deferred_string",
+ "wrap_complex",
+ "wrap_integer",
+ "wrap_logical",
+ "wrap_raw",
+ "wrap_real",
+ "wrap_string",
+}
+
+
+@dataclass(frozen=True, slots=True)
+class _ObjectInfo:
+ type: RType
+ object: bool
+ has_attributes: bool
+ has_tag: bool
+ gp: int
+
+
+class _WireReader:
+ def __init__(
+ self,
+ storage: RandomAccessStorage,
+ *,
+ position: int,
+ encoding: RdsEncoding,
+ byte_order: str | None,
+ ) -> None:
+ self.storage = storage
+ self.position = position
+ self.end = storage.size()
+ self.encoding = encoding
+ self.byte_order = byte_order
+ self.native_encoding: str | None = None
+ self._buffer = b""
+ self._buffer_start = position
+
+ def tell(self) -> int:
+ return self.position
+
+ def remaining(self) -> int:
+ return self.end - self.position
+
+ def _fill_buffer(self) -> None:
+ count = min(64 * 1024, self.remaining())
+ if count <= 0:
+ self._buffer = b""
+ self._buffer_start = self.position
+ return
+ self._buffer_start = self.position
+ self._buffer = self.storage.read_at(self.position, count)
+
+ def read_byte(self, *, path: str) -> int:
+ relative = self.position - self._buffer_start
+ if relative < 0 or relative >= len(self._buffer):
+ self._fill_buffer()
+ relative = 0
+ if not self._buffer:
+ raise RdsFormatError(
+ "unexpected end of stream",
+ path=path,
+ offset=self.position,
+ )
+ result = self._buffer[relative]
+ self.position += 1
+ return result
+
+ def read_exact(self, size: int, *, path: str) -> bytes:
+ if size < 0:
+ raise ValueError("size must be nonnegative")
+ if size > self.remaining():
+ raise RdsFormatError(
+ f"unexpected end of stream while reading {size} bytes",
+ path=path,
+ offset=self.position,
+ )
+ if size == 0:
+ return b""
+ relative = self.position - self._buffer_start
+ if 0 <= relative and relative + size <= len(self._buffer):
+ result = self._buffer[relative : relative + size]
+ else:
+ result = self.storage.read_at(self.position, size, path=path)
+ self.position += size
+ return result
+
+ def skip(self, size: int, *, path: str) -> int:
+ if size < 0:
+ raise ValueError("size must be nonnegative")
+ offset = self.position
+ if size > self.remaining():
+ raise RdsFormatError(
+ f"unexpected end of stream while skipping {size} bytes",
+ path=path,
+ offset=offset,
+ )
+ self.position += size
+ return offset
+
+ def read_word(self, *, path: str) -> bytes:
+ while True:
+ value = self.read_byte(path=path)
+ if not chr(value).isspace():
+ break
+ result = bytearray()
+ while not chr(value).isspace():
+ result.append(value)
+ if len(result) > 127:
+ raise RdsFormatError(
+ "ASCII token exceeds 127 bytes",
+ path=path,
+ offset=self.position - len(result),
+ )
+ value = self.read_byte(path=path)
+ return bytes(result)
+
+ def read_int(self, *, path: str) -> int:
+ offset = self.position
+ if self.encoding is RdsEncoding.ASCII:
+ word = self.read_word(path=path)
+ if word == b"NA":
+ return R_INT_NA
+ try:
+ return int(word, 10)
+ except ValueError as error:
+ raise RdsFormatError(
+ f"invalid ASCII integer {word!r}",
+ path=path,
+ offset=offset,
+ ) from error
+ order = ">" if self.encoding is RdsEncoding.XDR else self.byte_order
+ if order not in {"<", ">"}:
+ raise RdsFormatError(
+ "native binary byte order is unknown",
+ path=path,
+ offset=offset,
+ )
+ return int(struct.unpack(f"{order}i", self.read_exact(4, path=path))[0])
+
+ def read_real(self, *, path: str) -> float:
+ offset = self.position
+ if self.encoding is RdsEncoding.ASCII:
+ word = self.read_word(path=path)
+ if word == b"NA":
+ return math.nan
+ if word == b"NaN":
+ return math.nan
+ if word == b"Inf":
+ return math.inf
+ if word == b"-Inf":
+ return -math.inf
+ try:
+ text = word.decode("ascii")
+ return (
+ float.fromhex(text)
+ if text.lower().startswith(("0x", "-0x"))
+ else float(text)
+ )
+ except (UnicodeDecodeError, ValueError) as error:
+ raise RdsFormatError(
+ f"invalid ASCII real {word!r}",
+ path=path,
+ offset=offset,
+ ) from error
+ order = ">" if self.encoding is RdsEncoding.XDR else self.byte_order
+ if order not in {"<", ">"}:
+ raise RdsFormatError(
+ "native binary byte order is unknown",
+ path=path,
+ offset=offset,
+ )
+ return float(struct.unpack(f"{order}d", self.read_exact(8, path=path))[0])
+
+ def read_ascii_string(self, length: int, *, path: str) -> bytes:
+ if length == 0:
+ return b""
+ while True:
+ value = self.read_byte(path=path)
+ if not chr(value).isspace():
+ break
+ result = bytearray()
+ escaped = {
+ ord("n"): ord("\n"),
+ ord("t"): ord("\t"),
+ ord("v"): ord("\v"),
+ ord("b"): ord("\b"),
+ ord("r"): ord("\r"),
+ ord("f"): ord("\f"),
+ ord("a"): ord("\a"),
+ ord("\\"): ord("\\"),
+ ord("?"): ord("?"),
+ ord("'"): ord("'"),
+ ord('"'): ord('"'),
+ }
+ while len(result) < length:
+ if value != ord("\\"):
+ result.append(value)
+ else:
+ value = self.read_byte(path=path)
+ mapped = escaped.get(value)
+ if mapped is not None:
+ result.append(mapped)
+ elif ord("0") <= value <= ord("7"):
+ digits = [value]
+ while len(digits) < 3:
+ candidate = self.read_byte(path=path)
+ if not ord("0") <= candidate <= ord("7"):
+ self.position -= 1
+ break
+ digits.append(candidate)
+ result.append(int(bytes(digits), 8))
+ else:
+ result.append(value)
+ if len(result) < length:
+ value = self.read_byte(path=path)
+ return bytes(result)
+
+ def read_string(self, length: int, *, path: str) -> bytes:
+ if self.encoding is RdsEncoding.ASCII:
+ return self.read_ascii_string(length, path=path)
+ return self.read_exact(length, path=path)
+
+
+class _Parser:
+ def __init__(
+ self,
+ storage: RandomAccessStorage,
+ temp_manager: TempManager,
+ limits: RdsLimits,
+ ) -> None:
+ self.storage = storage
+ self.temp_manager = temp_manager
+ self.limits = limits
+ self.reader: _WireReader
+ self.metadata: RdsMetadata
+ self._objects = 0
+ self._references: list[RNode] = []
+ self._singletons: dict[RType, RNode] = {}
+
+ def parse(self) -> RNode:
+ self.reader, self.metadata = self._parse_header()
+ root = self._parse_node("$", 0)
+ self._check_complete()
+ self.temp_manager.flush()
+ return root
+
+ def _parse_header(self) -> tuple[_WireReader, RdsMetadata]:
+ start = self.storage.tell()
+ available = min(5, self.storage.size() - start)
+ prefix = self.storage.read_at(start, available, path="$header")
+ if prefix in _RDATA_HEADERS:
+ raise RdsFormatError(
+ "RData workspace header is not valid for RDS input",
+ path="$header",
+ offset=start,
+ )
+ if prefix.startswith(b"A\r\n"):
+ encoding = RdsEncoding.ASCII
+ position = start + 3
+ byte_order = None
+ elif prefix.startswith(b"A\n"):
+ encoding = RdsEncoding.ASCII
+ position = start + 2
+ byte_order = None
+ elif prefix.startswith(b"X\n"):
+ encoding = RdsEncoding.XDR
+ position = start + 2
+ byte_order = ">"
+ elif prefix.startswith(b"B\n"):
+ encoding = RdsEncoding.NATIVE
+ position = start + 2
+ marker = self.storage.read_at(position, 4, path="$header.formatVersion")
+ little = int.from_bytes(marker, "little", signed=True)
+ big = int.from_bytes(marker, "big", signed=True)
+ if little in {2, 3} and big not in {2, 3}:
+ byte_order = "<"
+ elif big in {2, 3} and little not in {2, 3}:
+ byte_order = ">"
+ else:
+ raise RdsFormatError(
+ "cannot determine native binary byte order",
+ path="$header",
+ offset=position,
+ )
+ else:
+ raise RdsFormatError(
+ "unknown R serialization format marker",
+ path="$header",
+ offset=start,
+ )
+
+ reader = _WireReader(
+ self.storage,
+ position=position,
+ encoding=encoding,
+ byte_order=byte_order,
+ )
+ format_version = reader.read_int(path="$header.formatVersion")
+ writer_version = reader.read_int(path="$header.writerVersion")
+ minimum_version = reader.read_int(path="$header.minimumReaderVersion")
+ if format_version not in {2, 3}:
+ raise RdsFormatError(
+ f"unsupported R serialization version {format_version}",
+ path="$header.formatVersion",
+ offset=position,
+ )
+ native_encoding: str | None = None
+ if format_version == 3:
+ length = reader.read_int(path="$header.nativeEncoding.length")
+ self._check_string_length(
+ length,
+ path="$header.nativeEncoding",
+ offset=reader.tell(),
+ allow_missing=False,
+ )
+ raw = reader.read_string(length, path="$header.nativeEncoding")
+ try:
+ native_encoding = raw.decode("ascii")
+ except UnicodeDecodeError as error:
+ raise RdsFormatError(
+ "native encoding name is not ASCII",
+ path="$header.nativeEncoding",
+ offset=reader.tell() - length,
+ ) from error
+ reader.native_encoding = native_encoding
+ return reader, RdsMetadata(
+ encoding=encoding,
+ format_version=format_version,
+ writer_version=writer_version,
+ minimum_reader_version=minimum_version,
+ native_encoding=native_encoding,
+ byte_order=byte_order,
+ )
+
+ def _check_complete(self) -> None:
+ if self.reader.encoding is not RdsEncoding.ASCII:
+ if self.reader.remaining() != 0:
+ raise RdsFormatError(
+ f"{self.reader.remaining()} trailing bytes after root object",
+ path="$",
+ offset=self.reader.tell(),
+ )
+ return
+ while self.reader.remaining():
+ offset = self.reader.tell()
+ chunk = self.reader.read_exact(
+ min(64 * 1024, self.reader.remaining()),
+ path="$",
+ )
+ if chunk.strip():
+ raise RdsFormatError(
+ "non-whitespace data follows the root object",
+ path="$",
+ offset=offset,
+ )
+
+ def _start_object(self, path: str, depth: int) -> int:
+ offset = self.reader.tell()
+ if depth > self.limits.max_depth:
+ raise RdsLimitError(
+ "max_depth",
+ depth,
+ self.limits.max_depth,
+ path=path,
+ offset=offset,
+ )
+ self._objects += 1
+ if self._objects > self.limits.max_objects:
+ raise RdsLimitError(
+ "max_objects",
+ self._objects,
+ self.limits.max_objects,
+ path=path,
+ offset=offset,
+ )
+ return offset
+
+ def _parse_info(self, flags: int, *, path: str, offset: int) -> _ObjectInfo:
+ type_code = flags & 0xFF
+ try:
+ r_type = RType(type_code)
+ except ValueError as error:
+ raise RdsFormatError(
+ f"unknown R node type {type_code}",
+ path=path,
+ offset=offset,
+ ) from error
+ if r_type in _SINGLETON_TYPES or r_type is RType.REFERENCE:
+ return _ObjectInfo(r_type, False, False, False, 0)
+ return _ObjectInfo(
+ type=r_type,
+ object=bool(flags & (1 << 8)),
+ has_attributes=bool(flags & (1 << 9)),
+ has_tag=bool(flags & (1 << 10)),
+ gp=(flags >> 12) & 0xFFFF,
+ )
+
+ def _parse_node(self, path: str, depth: int) -> RNode:
+ offset = self._start_object(path, depth)
+ flags = self.reader.read_int(path=path)
+ info = self._parse_info(flags, path=path, offset=offset)
+ r_type = info.type
+
+ if r_type in _SINGLETON_TYPES:
+ return self._singleton(r_type, path=path, offset=offset)
+ if r_type is RType.REFERENCE:
+ return self._read_reference(flags, path=path, offset=offset)
+ if r_type in {RType.CLASS_REFERENCE, RType.GENERIC_REFERENCE}:
+ raise RdsFormatError(
+ f"{r_type.name.lower()} is not a readable R serialization node",
+ path=path,
+ offset=offset,
+ )
+ if r_type is RType.SYMBOL:
+ name = self._parse_node(f"{path}.name", depth + 1)
+ node = RNode(
+ r_type,
+ value=name,
+ object=info.object,
+ gp=info.gp,
+ path=path,
+ offset=offset,
+ )
+ self._references.append(node)
+ return node
+ if r_type in {RType.PACKAGE, RType.NAMESPACE, RType.PERSISTENT}:
+ names = self._parse_persistent_strings(f"{path}.names", depth + 1)
+ node = RNode(
+ r_type,
+ value=PersistentValue(names),
+ object=info.object,
+ gp=info.gp,
+ path=path,
+ offset=offset,
+ )
+ self._references.append(node)
+ return node
+ if r_type is RType.ENVIRONMENT:
+ return self._parse_environment(info, path=path, depth=depth, offset=offset)
+ if r_type in _PAIR_TYPES:
+ return self._parse_pair(info, path=path, depth=depth, offset=offset)
+ if r_type is RType.ALTREP:
+ return self._parse_altrep(info, path=path, depth=depth, offset=offset)
+
+ node = RNode(
+ r_type,
+ object=info.object,
+ gp=info.gp,
+ path=path,
+ offset=offset,
+ )
+ if r_type in {RType.SPECIAL, RType.BUILTIN}:
+ length = self.reader.read_int(path=f"{path}.length")
+ self._check_string_length(
+ length,
+ path=path,
+ offset=self.reader.tell(),
+ allow_missing=False,
+ )
+ raw = self.reader.read_string(length, path=path)
+ node.value = raw.decode("utf-8", errors="surrogateescape")
+ elif r_type is RType.CHAR:
+ node.value = self._parse_char_value(info.gp, path=path)
+ elif r_type in {
+ RType.LOGICAL,
+ RType.INTEGER,
+ RType.REAL,
+ RType.COMPLEX,
+ RType.RAW,
+ }:
+ node.value = self._parse_atomic_vector(r_type, path=path)
+ elif r_type is RType.STRING:
+ node.value = self._parse_string_vector(path=path, depth=depth)
+ elif r_type in {RType.VECTOR, RType.EXPRESSION}:
+ length = self._read_length(path=f"{path}.length")
+ self._check_materialized_children(length, path=path)
+ values = [
+ self._parse_node(f"{path}[{index}]", depth + 1)
+ for index in range(length)
+ ]
+ node.value = tuple(values)
+ elif r_type is RType.BYTECODE:
+ node.value = self._parse_bytecode(path=path, depth=depth)
+ elif r_type is RType.EXTERNAL_POINTER:
+ self._references.append(node)
+ protected = self._parse_node(f"{path}.protected", depth + 1)
+ tag = self._parse_node(f"{path}.externalTag", depth + 1)
+ node.value = ExternalPointerValue(protected, tag)
+ elif r_type is RType.WEAK_REFERENCE:
+ self._references.append(node)
+ node.value = WeakReferenceValue(
+ ready_to_finalize=bool(info.gp & 1),
+ finalize_on_exit=bool(info.gp & 2),
+ )
+ elif r_type is RType.S4:
+ node.value = None
+ elif r_type is RType.ANY:
+ node.value = None
+ else:
+ raise RdsFormatError(
+ f"unsupported R node type {r_type.name}",
+ path=path,
+ offset=offset,
+ )
+
+ if info.has_tag:
+ node.tag = self._null_to_none(self._parse_node(f"{path}.tag", depth + 1))
+ if info.has_attributes:
+ node.attributes = self._null_to_none(
+ self._parse_node(f"{path}.attributes", depth + 1)
+ )
+ return node
+
+ def _singleton(self, r_type: RType, *, path: str, offset: int) -> RNode:
+ node = self._singletons.get(r_type)
+ if node is None:
+ node = RNode(r_type, path=path, offset=offset)
+ self._singletons[r_type] = node
+ return node
+
+ def _read_reference(self, flags: int, *, path: str, offset: int) -> RNode:
+ index = flags >> 8
+ if index == 0:
+ index = self.reader.read_int(path=f"{path}.reference")
+ if index <= 0 or index > len(self._references):
+ raise RdsFormatError(
+ f"reference index {index} is out of range",
+ path=path,
+ offset=offset,
+ )
+ return self._references[index - 1]
+
+ def _parse_environment(
+ self,
+ info: _ObjectInfo,
+ *,
+ path: str,
+ depth: int,
+ offset: int,
+ ) -> RNode:
+ node = RNode(
+ RType.ENVIRONMENT,
+ object=True,
+ gp=info.gp,
+ path=path,
+ offset=offset,
+ )
+ self._references.append(node)
+ locked = self.reader.read_int(path=f"{path}.locked")
+ if locked not in {0, 1}:
+ raise RdsFormatError(
+ f"environment lock flag must be 0 or 1, found {locked}",
+ path=f"{path}.locked",
+ offset=self.reader.tell() - 4,
+ )
+ enclosure = self._parse_node(f"{path}.enclosure", depth + 1)
+ frame = self._parse_node(f"{path}.frame", depth + 1)
+ hash_table = self._parse_node(f"{path}.hashTable", depth + 1)
+ attributes = self._parse_node(f"{path}.attributes", depth + 1)
+ node.value = EnvironmentValue(locked == 1, enclosure, frame, hash_table)
+ node.attributes = self._null_to_none(attributes)
+ return node
+
+ def _parse_pair(
+ self,
+ info: _ObjectInfo,
+ *,
+ path: str,
+ depth: int,
+ offset: int,
+ ) -> RNode:
+ attributes = (
+ self._parse_node(f"{path}.attributes", depth + 1)
+ if info.has_attributes
+ else None
+ )
+ tag = self._parse_node(f"{path}.tag", depth + 1) if info.has_tag else None
+ car = self._parse_node(f"{path}.car", depth + 1)
+ cdr = self._parse_node(f"{path}.cdr", depth + 1)
+ normalized_tag = self._null_to_none(tag)
+ node = RNode(
+ info.type,
+ attributes=self._null_to_none(attributes),
+ tag=normalized_tag,
+ object=info.object,
+ gp=info.gp,
+ path=path,
+ offset=offset,
+ )
+ if info.type is RType.CLOSURE:
+ node.value = ClosureValue(normalized_tag, car, cdr)
+ elif info.type is RType.PROMISE:
+ node.value = PromiseValue(normalized_tag, car, cdr)
+ else:
+ node.value = PairValue(car, cdr)
+ return node
+
+ def _parse_altrep(
+ self,
+ info: _ObjectInfo,
+ *,
+ path: str,
+ depth: int,
+ offset: int,
+ ) -> RNode:
+ altrep_info = self._parse_node(f"{path}.info", depth + 1)
+ state = self._parse_node(f"{path}.state", depth + 1)
+ attributes = self._parse_node(f"{path}.attributes", depth + 1)
+ class_name, package_name = self._altrep_names(altrep_info)
+ return RNode(
+ RType.ALTREP,
+ value=AltRepValue(
+ info=altrep_info,
+ state=state,
+ class_name=class_name,
+ package_name=package_name,
+ known=class_name in _KNOWN_ALTREP,
+ ),
+ attributes=self._null_to_none(attributes),
+ object=info.object,
+ gp=info.gp,
+ path=path,
+ offset=offset,
+ )
+
+ def _altrep_names(self, info: RNode) -> tuple[str | None, str | None]:
+ if info.type is not RType.PAIRLIST or not isinstance(info.value, PairValue):
+ return None, None
+ class_name = symbol_name(info.value.car)
+ tail = info.value.cdr
+ if tail.type is not RType.PAIRLIST or not isinstance(tail.value, PairValue):
+ return class_name, None
+ return class_name, symbol_name(tail.value.car)
+
+ def _parse_char_value(self, gp: int, *, path: str) -> str | bytes | None:
+ length = self.reader.read_int(path=f"{path}.length")
+ self._check_string_length(
+ length,
+ path=path,
+ offset=self.reader.tell(),
+ allow_missing=True,
+ )
+ if length == -1:
+ return None
+ raw = self.reader.read_string(length, path=path)
+ return self._decode_char(raw, gp)
+
+ def _decode_char(self, raw: bytes, gp: int) -> str | bytes:
+ if gp & (1 << 1):
+ return raw
+ if gp & (1 << 2):
+ return raw.decode("latin-1")
+ if gp & (1 << 3):
+ return raw.decode("utf-8", errors="surrogateescape")
+ if gp & (1 << 6):
+ return raw.decode("ascii", errors="surrogateescape")
+ encoding = self.metadata.native_encoding or "utf-8"
+ try:
+ codecs.lookup(encoding)
+ except LookupError:
+ encoding = "utf-8"
+ return raw.decode(encoding, errors="surrogateescape")
+
+ def _parse_atomic_vector(self, r_type: RType, *, path: str) -> LazyAtomicVector:
+ length = self._read_length(path=f"{path}.length")
+ item_size = {
+ RType.LOGICAL: 4,
+ RType.INTEGER: 4,
+ RType.REAL: 8,
+ RType.COMPLEX: 16,
+ RType.RAW: 1,
+ }[r_type]
+ if self.reader.encoding is not RdsEncoding.ASCII:
+ size = length * item_size
+ offset = self.reader.skip(size, path=path)
+ byte_order = (
+ ">"
+ if self.reader.encoding is RdsEncoding.XDR
+ else self.metadata.byte_order
+ )
+ if byte_order is None:
+ raise AssertionError("native byte order was not initialized")
+ return LazyAtomicVector(
+ self.storage,
+ offset=offset,
+ length=length,
+ r_type=r_type,
+ byte_order=byte_order,
+ path=path,
+ )
+
+ backing = self.temp_manager.shared("ascii-vector")
+ writer = BufferedTempWriter(self.temp_manager, backing)
+ offset = writer.position
+ for index in range(length):
+ item_path = f"{path}[{index}]"
+ if r_type in {RType.LOGICAL, RType.INTEGER}:
+ data = struct.pack(" 255:
+ raise RdsFormatError(
+ f"ASCII raw byte {value} is out of range",
+ path=item_path,
+ offset=self.reader.tell() - len(word),
+ )
+ data = bytes((value,))
+ writer.write(data, path=item_path)
+ writer.flush(path=path)
+ backing.flush()
+ return LazyAtomicVector(
+ backing,
+ offset=offset,
+ length=length,
+ r_type=r_type,
+ byte_order="<",
+ path=path,
+ )
+
+ def _parse_string_vector(self, *, path: str, depth: int) -> LazyStringVector:
+ length = self._read_length(path=f"{path}.length")
+ self._check_materialized_children(length, path=path)
+ index_storage = self.temp_manager.shared("string-index")
+ descriptor_writer = BufferedTempWriter(self.temp_manager, index_storage)
+ descriptor_offset = descriptor_writer.position
+ decoded_storage: RandomAccessStorage | None = None
+ decoded_writer: BufferedTempWriter | None = None
+ element_attributes: dict[int, RNode] = {}
+
+ for index in range(length):
+ item_path = f"{path}[{index}]"
+ item_offset = self._start_object(item_path, depth + 1)
+ flags = self.reader.read_int(path=item_path)
+ info = self._parse_info(flags, path=item_path, offset=item_offset)
+ if info.type is RType.REFERENCE:
+ referenced = self._read_reference(
+ flags,
+ path=item_path,
+ offset=item_offset,
+ )
+ if referenced.type is not RType.CHAR:
+ raise RdsFormatError(
+ "character vector reference does not target CHAR",
+ path=item_path,
+ offset=item_offset,
+ )
+ raw, gp = self._encoded_materialized_char(referenced, item_path)
+ if raw is None:
+ descriptor = pack_string_descriptor(0, 0, -1, gp)
+ else:
+ if decoded_storage is None:
+ decoded_storage = self.temp_manager.shared("string-data")
+ decoded_writer = BufferedTempWriter(
+ self.temp_manager,
+ decoded_storage,
+ )
+ if decoded_writer is None:
+ raise AssertionError("decoded writer was not initialized")
+ data_offset = decoded_writer.write(raw, path=item_path)
+ descriptor = pack_string_descriptor(
+ STRING_SOURCE_DECODED,
+ data_offset,
+ len(raw),
+ gp,
+ )
+ if referenced.attributes is not None:
+ element_attributes[index] = referenced.attributes
+ elif info.type is RType.CHAR:
+ string_length = self.reader.read_int(path=f"{item_path}.length")
+ self._check_string_length(
+ string_length,
+ path=item_path,
+ offset=self.reader.tell(),
+ allow_missing=True,
+ )
+ if string_length == -1:
+ descriptor = pack_string_descriptor(0, 0, -1, info.gp)
+ elif self.reader.encoding is RdsEncoding.ASCII:
+ raw = self.reader.read_ascii_string(
+ string_length,
+ path=item_path,
+ )
+ if decoded_storage is None:
+ decoded_storage = self.temp_manager.shared("string-data")
+ decoded_writer = BufferedTempWriter(
+ self.temp_manager,
+ decoded_storage,
+ )
+ if decoded_writer is None:
+ raise AssertionError("decoded writer was not initialized")
+ data_offset = decoded_writer.write(raw, path=item_path)
+ descriptor = pack_string_descriptor(
+ STRING_SOURCE_DECODED,
+ data_offset,
+ string_length,
+ info.gp,
+ )
+ else:
+ data_offset = self.reader.skip(string_length, path=item_path)
+ descriptor = pack_string_descriptor(
+ STRING_SOURCE_PAYLOAD,
+ data_offset,
+ string_length,
+ info.gp,
+ )
+ if info.has_attributes:
+ attributes = self._parse_node(
+ f"{item_path}.attributes",
+ depth + 2,
+ )
+ normalized = self._null_to_none(attributes)
+ if normalized is not None:
+ element_attributes[index] = normalized
+ if info.has_tag:
+ raise RdsFormatError(
+ "CHAR node cannot have a tag",
+ path=item_path,
+ offset=item_offset,
+ )
+ else:
+ raise RdsFormatError(
+ f"character vector element has type {info.type.name}",
+ path=item_path,
+ offset=item_offset,
+ )
+ descriptor_writer.write(descriptor, path=item_path)
+
+ descriptor_writer.flush(path=path)
+ if decoded_writer is not None:
+ decoded_writer.flush(path=path)
+ index_storage.flush()
+ if decoded_storage is not None:
+ decoded_storage.flush()
+ return LazyStringVector(
+ index_storage,
+ descriptor_offset=descriptor_offset,
+ length=length,
+ payload_storage=self.storage,
+ decoded_storage=decoded_storage,
+ default_encoding=self.metadata.native_encoding,
+ path=path,
+ element_attributes=element_attributes,
+ )
+
+ def _encoded_materialized_char(
+ self,
+ node: RNode,
+ path: str,
+ ) -> tuple[bytes | None, int]:
+ value = node.value
+ if value is None:
+ return None, node.gp
+ if isinstance(value, bytes):
+ return value, node.gp
+ if isinstance(value, str):
+ encoding = self.metadata.native_encoding or "utf-8"
+ try:
+ return value.encode(encoding, errors="surrogateescape"), node.gp
+ except LookupError:
+ return value.encode("utf-8", errors="surrogateescape"), node.gp
+ raise RdsFormatError(
+ "referenced CHAR has no string value",
+ path=path,
+ offset=node.offset,
+ )
+
+ def _parse_persistent_strings(
+ self,
+ path: str,
+ depth: int,
+ ) -> LazyStringVector:
+ placeholder = self.reader.read_int(path=f"{path}.placeholder")
+ if placeholder != 0:
+ raise RdsFormatError(
+ f"persistent string placeholder must be 0, found {placeholder}",
+ path=f"{path}.placeholder",
+ offset=self.reader.tell() - 4,
+ )
+ return self._parse_string_vector(path=path, depth=depth)
+
+ def _parse_bytecode(self, *, path: str, depth: int) -> BytecodeValue:
+ repeated = self.reader.read_int(path=f"{path}.repeatedCount")
+ self._check_count(repeated, path=f"{path}.repeatedCount")
+ self._check_materialized_children(
+ repeated,
+ path=f"{path}.repeatedCount",
+ )
+ repetitions: list[RNode | None] = [None] * repeated
+ return self._parse_bytecode_body(
+ repetitions,
+ path=path,
+ depth=depth + 1,
+ )
+
+ def _parse_bytecode_body(
+ self,
+ repetitions: list[RNode | None],
+ *,
+ path: str,
+ depth: int,
+ ) -> BytecodeValue:
+ code = self._parse_node(f"{path}.code", depth + 1)
+ count = self.reader.read_int(path=f"{path}.constantCount")
+ self._check_count(count, path=f"{path}.constantCount")
+ self._check_materialized_children(count, path=f"{path}.constants")
+ constants: list[RNode] = []
+ for index in range(count):
+ constant_path = f"{path}.constants[{index}]"
+ marker_offset = self.reader.tell()
+ marker = self.reader.read_int(path=f"{constant_path}.marker")
+ if marker == RType.BYTECODE:
+ self._start_object(constant_path, depth + 1)
+ value = RNode(
+ RType.BYTECODE,
+ value=self._parse_bytecode_body(
+ repetitions,
+ path=constant_path,
+ depth=depth + 1,
+ ),
+ path=constant_path,
+ offset=marker_offset,
+ )
+ elif marker in {
+ RType.LANGUAGE,
+ RType.PAIRLIST,
+ RType.BYTECODE_DEFINITION,
+ RType.BYTECODE_REFERENCE,
+ RType.BYTECODE_ATTR_LANGUAGE,
+ RType.BYTECODE_ATTR_PAIRLIST,
+ }:
+ value = self._parse_bytecode_language(
+ int(marker),
+ repetitions,
+ path=constant_path,
+ depth=depth + 1,
+ )
+ else:
+ value = self._parse_node(constant_path, depth + 1)
+ constants.append(value)
+ return BytecodeValue(code=code, constants=tuple(constants))
+
+ def _parse_bytecode_language(
+ self,
+ marker: int,
+ repetitions: list[RNode | None],
+ *,
+ path: str,
+ depth: int,
+ ) -> RNode:
+ special_markers = {
+ RType.LANGUAGE,
+ RType.PAIRLIST,
+ RType.BYTECODE_DEFINITION,
+ RType.BYTECODE_REFERENCE,
+ RType.BYTECODE_ATTR_LANGUAGE,
+ RType.BYTECODE_ATTR_PAIRLIST,
+ }
+ if marker not in special_markers:
+ return self._parse_node(path, depth)
+
+ marker_offset = self._start_object(path, depth)
+ if marker == RType.BYTECODE_REFERENCE:
+ position = self.reader.read_int(path=f"{path}.position")
+ if (
+ position < 0
+ or position >= len(repetitions)
+ or repetitions[position] is None
+ ):
+ raise RdsFormatError(
+ f"bytecode reference {position} is out of range",
+ path=path,
+ offset=marker_offset,
+ )
+ result = repetitions[position]
+ if result is None:
+ raise AssertionError("checked bytecode reference is missing")
+ return result
+
+ repetition_position: int | None = None
+ if marker == RType.BYTECODE_DEFINITION:
+ repetition_position = self.reader.read_int(path=f"{path}.position")
+ if repetition_position < 0 or repetition_position >= len(repetitions):
+ raise RdsFormatError(
+ f"bytecode definition {repetition_position} is out of range",
+ path=path,
+ offset=marker_offset,
+ )
+ marker = self.reader.read_int(path=f"{path}.type")
+
+ has_attributes = marker in {
+ RType.BYTECODE_ATTR_LANGUAGE,
+ RType.BYTECODE_ATTR_PAIRLIST,
+ }
+ if marker in {RType.LANGUAGE, RType.BYTECODE_ATTR_LANGUAGE}:
+ r_type = RType.LANGUAGE
+ elif marker in {RType.PAIRLIST, RType.BYTECODE_ATTR_PAIRLIST}:
+ r_type = RType.PAIRLIST
+ else:
+ raise RdsFormatError(
+ f"invalid bytecode pair definition type {marker}",
+ path=path,
+ offset=marker_offset,
+ )
+
+ node = RNode(r_type, path=path, offset=marker_offset)
+ if repetition_position is not None:
+ repetitions[repetition_position] = node
+ if has_attributes:
+ node.attributes = self._null_to_none(
+ self._parse_node(f"{path}.attributes", depth + 1)
+ )
+ node.tag = self._null_to_none(self._parse_node(f"{path}.tag", depth + 1))
+ car_marker = self.reader.read_int(path=f"{path}.car.marker")
+ car = self._parse_bytecode_language(
+ car_marker,
+ repetitions,
+ path=f"{path}.car",
+ depth=depth + 1,
+ )
+ cdr_marker = self.reader.read_int(path=f"{path}.cdr.marker")
+ cdr = self._parse_bytecode_language(
+ cdr_marker,
+ repetitions,
+ path=f"{path}.cdr",
+ depth=depth + 1,
+ )
+ node.value = PairValue(car, cdr)
+ return node
+
+ def _read_length(self, *, path: str) -> int:
+ offset = self.reader.tell()
+ length = self.reader.read_int(path=path)
+ if length == -1:
+ upper = self.reader.read_int(path=f"{path}.upper") & 0xFFFFFFFF
+ lower = self.reader.read_int(path=f"{path}.lower") & 0xFFFFFFFF
+ if upper > 65536:
+ raise RdsFormatError(
+ f"long-vector upper length {upper} exceeds R's wire limit",
+ path=path,
+ offset=offset,
+ )
+ length = (upper << 32) | lower
+ elif length < 0:
+ raise RdsFormatError(
+ f"negative vector length {length}",
+ path=path,
+ offset=offset,
+ )
+ self._check_count(length, path=path, offset=offset)
+ return length
+
+ def _check_count(
+ self,
+ count: int,
+ *,
+ path: str,
+ offset: int | None = None,
+ ) -> None:
+ if count < 0:
+ raise RdsFormatError(
+ f"negative count {count}",
+ path=path,
+ offset=self.reader.tell() if offset is None else offset,
+ )
+ if count > self.limits.max_vector_length:
+ raise RdsLimitError(
+ "max_vector_length",
+ count,
+ self.limits.max_vector_length,
+ path=path,
+ offset=self.reader.tell() if offset is None else offset,
+ )
+
+ def _check_materialized_children(self, count: int, *, path: str) -> None:
+ projected = self._objects + count
+ if projected > self.limits.max_objects:
+ raise RdsLimitError(
+ "max_objects",
+ projected,
+ self.limits.max_objects,
+ path=path,
+ offset=self.reader.tell(),
+ )
+
+ def _check_string_length(
+ self,
+ length: int,
+ *,
+ path: str,
+ offset: int,
+ allow_missing: bool,
+ ) -> None:
+ minimum = -1 if allow_missing else 0
+ if length < minimum:
+ raise RdsFormatError(
+ f"invalid string length {length}",
+ path=path,
+ offset=offset,
+ )
+ if length > self.limits.max_string_bytes:
+ raise RdsLimitError(
+ "max_string_bytes",
+ length,
+ self.limits.max_string_bytes,
+ path=path,
+ offset=offset,
+ )
+
+ @staticmethod
+ def _null_to_none(node: RNode | None) -> RNode | None:
+ if node is not None and node.is_null:
+ return None
+ return node
+
+
+def open_rds(
+ source: RdsInput,
+ *,
+ limits: RdsLimits | None = None,
+ temp_dir: str | os.PathLike[str] | None = None,
+) -> RdsDocument:
+ """Open and index one RDS stream without evaluating serialized code."""
+ active_limits = limits or RdsLimits()
+ prepared = prepare_payload(source, active_limits, temp_dir=temp_dir)
+ try:
+ parser = _Parser(
+ prepared.storage,
+ prepared.temp_manager,
+ active_limits,
+ )
+ root = parser.parse()
+ return RdsDocument(
+ root,
+ metadata=parser.metadata,
+ source=prepared.metadata,
+ temp_manager=prepared.temp_manager,
+ direct_storage=prepared.direct_storage,
+ )
+ except Exception:
+ if prepared.direct_storage is not None:
+ prepared.direct_storage.close()
+ prepared.temp_manager.close()
+ raise
diff --git a/scarf/readers/_rds/_storage.py b/scarf/readers/_rds/_storage.py
new file mode 100644
index 00000000..daa429a7
--- /dev/null
+++ b/scarf/readers/_rds/_storage.py
@@ -0,0 +1,492 @@
+import bz2
+import gzip
+import hashlib
+import io
+import lzma
+import os
+import pathlib
+import shutil
+import tempfile
+import threading
+from collections.abc import Buffer
+from dataclasses import dataclass
+from typing import Any, Protocol, cast
+
+from ._types import (
+ RdsClosedError,
+ RdsCompression,
+ RdsFormatError,
+ RdsLimitError,
+ RdsLimits,
+ RdsSourceMetadata,
+)
+
+
+class BinaryStream(Protocol):
+ def read(self, size: int = -1) -> bytes: ...
+
+ def seek(self, offset: int, whence: int = 0) -> int: ...
+
+ def tell(self) -> int: ...
+
+ def close(self) -> None: ...
+
+
+RdsInput = str | os.PathLike[str] | BinaryStream | Buffer
+
+
+class RandomAccessStorage:
+ """Synchronized random access to one binary backing stream."""
+
+ def __init__(
+ self,
+ file: BinaryStream,
+ *,
+ name: str,
+ close_file: bool,
+ restore_position: int | None = None,
+ ) -> None:
+ self._file = file
+ self.name = name
+ self._close_file = close_file
+ self._restore_position = restore_position
+ self._closed = False
+ self._lock = threading.RLock()
+
+ @property
+ def closed(self) -> bool:
+ return self._closed
+
+ def tell(self) -> int:
+ with self._lock:
+ self._ensure_open()
+ return self._file.tell()
+
+ def seek(self, offset: int, whence: int = 0) -> int:
+ with self._lock:
+ self._ensure_open()
+ return self._file.seek(offset, whence)
+
+ def read(self, size: int = -1) -> bytes:
+ with self._lock:
+ self._ensure_open()
+ return self._file.read(size)
+
+ def read_at(self, offset: int, size: int, *, path: str = "$") -> bytes:
+ if offset < 0 or size < 0:
+ raise ValueError("offset and size must be nonnegative")
+ with self._lock:
+ self._ensure_open(path)
+ current = self._file.tell()
+ try:
+ self._file.seek(offset)
+ data = self._file.read(size)
+ finally:
+ self._file.seek(current)
+ if len(data) != size:
+ raise RdsFormatError(
+ f"unexpected end of stream while reading {size} bytes",
+ path=path,
+ offset=offset,
+ )
+ return data
+
+ def size(self) -> int:
+ with self._lock:
+ self._ensure_open()
+ current = self._file.tell()
+ try:
+ return self._file.seek(0, os.SEEK_END)
+ finally:
+ self._file.seek(current)
+
+ def flush(self) -> None:
+ with self._lock:
+ self._ensure_open()
+ flush = getattr(self._file, "flush", None)
+ if flush is not None:
+ flush()
+
+ def _append_unchecked(self, data: bytes) -> int:
+ with self._lock:
+ self._ensure_open()
+ offset = self._file.seek(0, os.SEEK_END)
+ written = self._file.write(data) # type: ignore[attr-defined]
+ if written is not None and written != len(data):
+ raise OSError("short temporary-file write")
+ return offset
+
+ def _ensure_open(self, path: str = "$") -> None:
+ if self._closed:
+ raise RdsClosedError("RDS document is closed", path=path)
+
+ def close(self) -> None:
+ with self._lock:
+ if self._closed:
+ return
+ self._closed = True
+ if self._restore_position is not None:
+ try:
+ self._file.seek(self._restore_position)
+ except (OSError, ValueError):
+ pass
+ if self._close_file:
+ self._file.close()
+
+
+class TempManager:
+ """Own capped temporary files for one document."""
+
+ def __init__(
+ self,
+ limits: RdsLimits,
+ *,
+ temp_dir: str | os.PathLike[str] | None,
+ ) -> None:
+ self._maximum = limits.max_temp_bytes
+ self._used = 0
+ self._temp_dir = None if temp_dir is None else os.fspath(temp_dir)
+ self._storages: list[RandomAccessStorage] = []
+ self._paths: list[pathlib.Path] = []
+ self._shared: dict[str, RandomAccessStorage] = {}
+ self._closed = False
+ self._lock = threading.RLock()
+
+ @property
+ def used_bytes(self) -> int:
+ return self._used
+
+ @property
+ def paths(self) -> tuple[pathlib.Path, ...]:
+ return tuple(self._paths)
+
+ @property
+ def storages(self) -> tuple[RandomAccessStorage, ...]:
+ return tuple(self._storages)
+
+ def create(self, label: str) -> RandomAccessStorage:
+ with self._lock:
+ if self._closed:
+ raise RdsClosedError("RDS document is closed", path="$temp")
+ file = tempfile.NamedTemporaryFile(
+ mode="w+b",
+ prefix=f"scarf-rds-{label}-",
+ suffix=".tmp",
+ dir=self._temp_dir,
+ delete=False,
+ )
+ path = pathlib.Path(file.name)
+ storage = RandomAccessStorage(
+ file,
+ name=str(path),
+ close_file=True,
+ )
+ self._paths.append(path)
+ self._storages.append(storage)
+ return storage
+
+ def shared(self, label: str) -> RandomAccessStorage:
+ """Return one append-only temporary storage per label."""
+ with self._lock:
+ storage = self._shared.get(label)
+ if storage is not None:
+ return storage
+ storage = self.create(label)
+ self._shared[label] = storage
+ return storage
+
+ def append(self, storage: RandomAccessStorage, data: bytes, *, path: str) -> int:
+ with self._lock:
+ actual = self._used + len(data)
+ if actual > self._maximum:
+ raise RdsLimitError(
+ "max_temp_bytes",
+ actual,
+ self._maximum,
+ path=path,
+ )
+ try:
+ free_bytes = shutil.disk_usage(pathlib.Path(storage.name).parent).free
+ except OSError as error:
+ raise OSError(
+ f"Cannot inspect free scratch space for {storage.name}"
+ ) from error
+ if len(data) > free_bytes:
+ raise RdsLimitError(
+ "scratch_free_bytes",
+ len(data),
+ free_bytes,
+ path=path,
+ )
+ offset = storage._append_unchecked(data)
+ self._used = actual
+ return offset
+
+ def flush(self) -> None:
+ for storage in self._storages:
+ if not storage.closed:
+ storage.flush()
+
+ def close(self) -> None:
+ with self._lock:
+ if self._closed:
+ return
+ self._closed = True
+ for storage in reversed(self._storages):
+ storage.close()
+ for path in reversed(self._paths):
+ try:
+ path.unlink()
+ except FileNotFoundError:
+ pass
+
+
+class BufferedTempWriter:
+ """Chunk small records into bounded temporary-file writes."""
+
+ def __init__(
+ self,
+ manager: TempManager,
+ storage: RandomAccessStorage,
+ *,
+ chunk_bytes: int = 256 * 1024,
+ ) -> None:
+ self._manager = manager
+ self.storage = storage
+ self._chunk_bytes = chunk_bytes
+ self._buffer = bytearray()
+ self._base = storage.size()
+ self._committed = 0
+
+ @property
+ def position(self) -> int:
+ return self._base + self._committed + len(self._buffer)
+
+ def write(self, data: bytes, *, path: str) -> int:
+ offset = self.position
+ self._buffer.extend(data)
+ if len(self._buffer) >= self._chunk_bytes:
+ self.flush(path=path)
+ return offset
+
+ def flush(self, *, path: str) -> None:
+ if not self._buffer:
+ return
+ data = bytes(self._buffer)
+ actual = self._manager.append(self.storage, data, path=path)
+ expected = self._base + self._committed
+ if actual != expected:
+ raise RuntimeError("temporary writer was interleaved")
+ self._committed += len(data)
+ self._buffer.clear()
+
+
+@dataclass(slots=True)
+class PreparedPayload:
+ storage: RandomAccessStorage
+ metadata: RdsSourceMetadata
+ temp_manager: TempManager
+ direct_storage: RandomAccessStorage | None
+
+
+@dataclass(slots=True)
+class _OpenedInput:
+ file: BinaryStream
+ name: str
+ size: int | None
+ initial_position: int
+ owned: bool
+
+
+_COMPRESSION_MAGIC: tuple[tuple[bytes, RdsCompression], ...] = (
+ (b"\x1f\x8b", RdsCompression.GZIP),
+ (b"BZh", RdsCompression.BZIP2),
+ (b"\xfd7zXZ\x00", RdsCompression.XZ),
+ (b"\x28\xb5\x2f\xfd", RdsCompression.ZSTD),
+)
+
+
+def _open_input(source: RdsInput) -> _OpenedInput:
+ if isinstance(source, (str, os.PathLike)):
+ path = pathlib.Path(source)
+ path_file = path.open("rb")
+ return _OpenedInput(
+ file=path_file,
+ name=str(path),
+ size=path.stat().st_size,
+ initial_position=0,
+ owned=True,
+ )
+ if isinstance(source, Buffer):
+ data = bytes(source)
+ bytes_file = io.BytesIO(data)
+ return _OpenedInput(
+ file=bytes_file,
+ name="",
+ size=len(data),
+ initial_position=0,
+ owned=True,
+ )
+ try:
+ initial = source.tell()
+ end = source.seek(0, os.SEEK_END)
+ source.seek(initial)
+ except (AttributeError, OSError, ValueError) as error:
+ raise TypeError("RDS input stream must be seekable") from error
+ return _OpenedInput(
+ file=source,
+ name=str(getattr(source, "name", "")),
+ size=end,
+ initial_position=initial,
+ owned=False,
+ )
+
+
+def _detect_compression(prefix: bytes) -> RdsCompression:
+ for magic, compression in _COMPRESSION_MAGIC:
+ if prefix.startswith(magic):
+ return compression
+ return RdsCompression.NONE
+
+
+def _sha256_range(file: BinaryStream, start: int, stop: int) -> str:
+ digest = hashlib.sha256()
+ file.seek(start)
+ remaining = stop - start
+ while remaining:
+ chunk = file.read(min(1024 * 1024, remaining))
+ if not chunk:
+ raise OSError("Input stream ended while computing its digest")
+ digest.update(chunk)
+ remaining -= len(chunk)
+ file.seek(start)
+ return digest.hexdigest()
+
+
+def _decompressor(
+ compression: RdsCompression,
+ source: BinaryStream,
+) -> BinaryStream:
+ if compression is RdsCompression.GZIP:
+ return gzip.GzipFile(fileobj=source, mode="rb")
+ if compression is RdsCompression.BZIP2:
+ return cast(BinaryStream, bz2.BZ2File(cast(Any, source), mode="rb"))
+ if compression is RdsCompression.XZ:
+ return cast(BinaryStream, lzma.LZMAFile(cast(Any, source), mode="rb"))
+ if compression is RdsCompression.ZSTD:
+ import zstandard
+
+ return cast(
+ BinaryStream,
+ zstandard.ZstdDecompressor().stream_reader(
+ cast(Any, source),
+ read_across_frames=True,
+ closefd=False,
+ ),
+ )
+ raise AssertionError(f"no decompressor for {compression}")
+
+
+def prepare_payload(
+ source: RdsInput,
+ limits: RdsLimits,
+ *,
+ temp_dir: str | os.PathLike[str] | None,
+) -> PreparedPayload:
+ """Open an input and spool compressed payloads within the temp cap."""
+ manager = TempManager(limits, temp_dir=temp_dir)
+ opened = _open_input(source)
+ direct: RandomAccessStorage | None = None
+ try:
+ if opened.size is None:
+ raise AssertionError("seekable input size was not recorded")
+ source_bytes = opened.size - opened.initial_position
+ source_sha256 = _sha256_range(
+ opened.file,
+ opened.initial_position,
+ opened.size,
+ )
+ opened.file.seek(opened.initial_position)
+ prefix = opened.file.read(8)
+ opened.file.seek(opened.initial_position)
+ compression = _detect_compression(prefix)
+ if compression is RdsCompression.NONE:
+ direct = RandomAccessStorage(
+ opened.file,
+ name=opened.name,
+ close_file=opened.owned,
+ restore_position=None if opened.owned else opened.initial_position,
+ )
+ payload_bytes = (
+ direct.size() - opened.initial_position
+ if opened.size is None
+ else opened.size - opened.initial_position
+ )
+ return PreparedPayload(
+ storage=direct,
+ metadata=RdsSourceMetadata(
+ name=opened.name,
+ compression=compression,
+ source_bytes=source_bytes,
+ payload_bytes=payload_bytes,
+ source_sha256=source_sha256,
+ payload_sha256=source_sha256,
+ spooled=False,
+ ),
+ temp_manager=manager,
+ direct_storage=direct,
+ )
+
+ spool = manager.create("payload")
+ reader = _decompressor(compression, opened.file)
+ payload_bytes = 0
+ payload_digest = hashlib.sha256()
+ try:
+ while True:
+ chunk = reader.read(1024 * 1024)
+ if not chunk:
+ break
+ manager.append(spool, chunk, path="$source")
+ payload_bytes += len(chunk)
+ payload_digest.update(chunk)
+ except RdsLimitError:
+ raise
+ except Exception as error:
+ raise RdsFormatError(
+ f"invalid {compression.value} compressed stream",
+ path="$source",
+ ) from error
+ finally:
+ reader.close()
+ spool.flush()
+ spool.seek(0)
+ if opened.owned:
+ opened.file.close()
+ else:
+ opened.file.seek(opened.initial_position)
+ return PreparedPayload(
+ storage=spool,
+ metadata=RdsSourceMetadata(
+ name=opened.name,
+ compression=compression,
+ source_bytes=source_bytes,
+ payload_bytes=payload_bytes,
+ source_sha256=source_sha256,
+ payload_sha256=payload_digest.hexdigest(),
+ spooled=True,
+ ),
+ temp_manager=manager,
+ direct_storage=None,
+ )
+ except Exception:
+ if direct is not None:
+ direct.close()
+ elif opened.owned:
+ opened.file.close()
+ else:
+ try:
+ opened.file.seek(opened.initial_position)
+ except (OSError, ValueError):
+ pass
+ manager.close()
+ raise
diff --git a/scarf/readers/_rds/_types.py b/scarf/readers/_rds/_types.py
new file mode 100644
index 00000000..4331260a
--- /dev/null
+++ b/scarf/readers/_rds/_types.py
@@ -0,0 +1,192 @@
+import enum
+from dataclasses import dataclass
+
+
+class RType(enum.IntEnum):
+ """R serialization node types."""
+
+ NIL = 0
+ SYMBOL = 1
+ SYM = 1
+ PAIRLIST = 2
+ LIST = 2
+ CLOSURE = 3
+ CLO = 3
+ ENVIRONMENT = 4
+ ENV = 4
+ PROMISE = 5
+ PROM = 5
+ LANGUAGE = 6
+ LANG = 6
+ SPECIAL = 7
+ BUILTIN = 8
+ CHAR = 9
+ LOGICAL = 10
+ LGL = 10
+ INTEGER = 13
+ INT = 13
+ REAL = 14
+ COMPLEX = 15
+ CPLX = 15
+ STRING = 16
+ STR = 16
+ DOTS = 17
+ ANY = 18
+ VECTOR = 19
+ VEC = 19
+ EXPRESSION = 20
+ EXPR = 20
+ BYTECODE = 21
+ BCODE = 21
+ EXTERNAL_POINTER = 22
+ EXTPTR = 22
+ WEAK_REFERENCE = 23
+ WEAKREF = 23
+ RAW = 24
+ S4 = 25
+ ALTREP = 238
+ BYTECODE_ATTR_PAIRLIST = 239
+ BYTECODE_ATTR_LANGUAGE = 240
+ BASE_ENVIRONMENT = 241
+ EMPTY_ENVIRONMENT = 242
+ BYTECODE_REFERENCE = 243
+ BYTECODE_DEFINITION = 244
+ GENERIC_REFERENCE = 245
+ CLASS_REFERENCE = 246
+ PERSISTENT = 247
+ PACKAGE = 248
+ NAMESPACE = 249
+ BASE_NAMESPACE = 250
+ MISSING_ARGUMENT = 251
+ UNBOUND_VALUE = 252
+ GLOBAL_ENVIRONMENT = 253
+ NIL_VALUE = 254
+ REFERENCE = 255
+
+
+class RdsEncoding(enum.StrEnum):
+ """Serialization payload encoding."""
+
+ XDR = "xdr"
+ NATIVE = "native"
+ ASCII = "ascii"
+
+
+class RdsCompression(enum.StrEnum):
+ """Compression wrapped around an RDS payload."""
+
+ NONE = "none"
+ GZIP = "gzip"
+ BZIP2 = "bzip2"
+ XZ = "xz"
+ ZSTD = "zstd"
+
+
+@dataclass(frozen=True, slots=True)
+class RdsLimits:
+ """Finite parser and scratch-space limits."""
+
+ max_objects: int = 100_000_000
+ max_depth: int = 512
+ max_string_bytes: int = 256 * 1024**2
+ max_vector_length: int = 1 << 40
+ max_temp_bytes: int = 16 * 1024**3
+
+ def __post_init__(self) -> None:
+ for name, value in (
+ ("max_objects", self.max_objects),
+ ("max_depth", self.max_depth),
+ ("max_string_bytes", self.max_string_bytes),
+ ("max_vector_length", self.max_vector_length),
+ ("max_temp_bytes", self.max_temp_bytes),
+ ):
+ if value < 0:
+ raise ValueError(f"{name} must be nonnegative")
+
+
+@dataclass(frozen=True, slots=True)
+class RdsSourceMetadata:
+ """Input and decompression details."""
+
+ name: str
+ compression: RdsCompression
+ source_bytes: int | None
+ payload_bytes: int
+ source_sha256: str
+ payload_sha256: str
+ spooled: bool
+
+
+@dataclass(frozen=True, slots=True)
+class RdsMetadata:
+ """R serialization header details."""
+
+ encoding: RdsEncoding
+ format_version: int
+ writer_version: int
+ minimum_reader_version: int
+ native_encoding: str | None
+ byte_order: str | None
+
+ @staticmethod
+ def unpack_r_version(packed: int) -> tuple[int, int, int]:
+ """Return an R packed version as major, minor, and patch."""
+ major, remainder = divmod(packed, 65536)
+ minor, patch = divmod(remainder, 256)
+ return major, minor, patch
+
+ @property
+ def writer_version_tuple(self) -> tuple[int, int, int]:
+ return self.unpack_r_version(self.writer_version)
+
+ @property
+ def minimum_reader_version_tuple(self) -> tuple[int, int, int]:
+ return self.unpack_r_version(self.minimum_reader_version)
+
+
+class RdsError(Exception):
+ """Base class for deterministic RDS errors."""
+
+ def __init__(
+ self, message: str, *, path: str = "$", offset: int | None = None
+ ) -> None:
+ self.message = message
+ self.path = path
+ self.offset = offset
+ location = f" at {path}"
+ if offset is not None:
+ location += f" (byte {offset})"
+ super().__init__(f"{message}{location}")
+
+
+class RdsFormatError(RdsError):
+ """Malformed or unsupported wire data."""
+
+
+class RdsLimitError(RdsError):
+ """A configured parser limit was exceeded."""
+
+ def __init__(
+ self,
+ limit: str,
+ actual: int,
+ maximum: int,
+ *,
+ path: str,
+ offset: int | None = None,
+ ) -> None:
+ self.limit = limit
+ self.actual = actual
+ self.maximum = maximum
+ super().__init__(
+ f"{limit} exceeded: {actual} > {maximum}",
+ path=path,
+ offset=offset,
+ )
+
+
+class RdsClosedError(RdsError):
+ """Lazy data was accessed after its document closed."""
+
+
+R_INT_NA = -(1 << 31)
diff --git a/scarf/readers/_seurat/__init__.py b/scarf/readers/_seurat/__init__.py
new file mode 100644
index 00000000..6f80bd2d
--- /dev/null
+++ b/scarf/readers/_seurat/__init__.py
@@ -0,0 +1,211 @@
+from .bpcells import (
+ BPCellsDirMatrixSource,
+ BPCellsDirectoryMatrixSource,
+ BPCellsDirectorySource,
+ BPCellsH5MatrixSource,
+ BPCellsHDF5MatrixSource,
+ BPCellsHDF5Source,
+ BPCellsMatrixSource,
+ BPCellsMemoryMatrixSource,
+ decode_bp128,
+ decode_bp128_d1z,
+ decode_bp128_m1,
+)
+from .errors import (
+ MatrixSourceError,
+ ResourceLimitError,
+ UnsafeSidecarError,
+ UnsupportedMatrixOperation,
+)
+from .factory import (
+ matrix_source_from_mapping,
+ matrix_source_from_slots,
+ source_from_slot_mapping,
+)
+from .fragments import (
+ FRAGMENT_CAPABILITY_REGISTRY,
+ FragmentBlock,
+ FragmentCapability,
+ FragmentCapabilityRegistry,
+ FragmentDerivedMatrixSource,
+ FragmentSource,
+ StoredFragmentSource,
+ fragment_source_from_slots,
+)
+from .hdf5 import (
+ H5ADMatrixSource,
+ H5SparseMatrixSource,
+ H5adMatrixSource,
+ HDF5ArrayMatrixSource,
+ HDF5ArraySource,
+ HDF5CompressedMatrixSource,
+ HDF5DenseMatrixSource,
+ ReshapedHDF5ArrayMatrixSource,
+ TENxMatrixSource,
+ TenXMatrixSource,
+ TenxMatrixSource,
+)
+from .operations import (
+ DEFAULT_OPERATION_REGISTRY,
+ AxisMinimumMatrixSource,
+ BinaryMatrixSource,
+ BinaryTransformMatrixSource,
+ DelayedSubassignmentMatrixSource,
+ LocalBinaryMatrixSource,
+ LocalUnaryMatrixSource,
+ LinearResidualMatrixSource,
+ MaskMatrixSource,
+ MatrixMultiplySource,
+ MatrixOperation,
+ MatrixOperationRegistry,
+ MultiplyMatrixSource,
+ OperationCapability,
+ PearsonResidualMatrixSource,
+ RankMatrixSource,
+ ScaleShiftMatrixSource,
+ Subassignment,
+ SubassignmentMatrixSource,
+ UnaryMatrixSource,
+ UnaryTransformMatrixSource,
+ UnsupportedExecutionMatrixSource,
+ build_matrix_operation,
+)
+from .paths import (
+ SidecarPathResolver,
+ read_hdf5_names,
+ require_filesystem_path,
+ resolve_sidecar_path,
+ validate_hdf5_file,
+)
+from .sources import (
+ CSCMatrixSource,
+ DEFAULT_LIMITS,
+ BaseMatrixSource,
+ CastMatrixSource,
+ CellBindMatrixSource,
+ CellBindSource,
+ ColumnBindMatrixSource,
+ CscMatrixSource,
+ DenseMatrixSource,
+ DtypeMatrixSource,
+ FeatureBindMatrixSource,
+ FeatureBindSource,
+ InMemoryDenseMatrixSource,
+ LayerPlacement,
+ LayerStitchMatrixSource,
+ LayerStitchSource,
+ LazyDenseMatrixSource,
+ MappedMatrixSource,
+ MatrixBlock,
+ MatrixCscSource,
+ MatrixSource,
+ MemoryEstimate,
+ RColumnMajorMatrixSource,
+ RenameMatrixSource,
+ RenamedMatrixSource,
+ ReorderedMatrixSource,
+ RowBindMatrixSource,
+ SourceLimits,
+ SubsetMatrixSource,
+ TransposeMatrixSource,
+ TransposeSource,
+)
+
+__all__ = [
+ "BPCellsDirMatrixSource",
+ "BPCellsDirectoryMatrixSource",
+ "BPCellsDirectorySource",
+ "BPCellsH5MatrixSource",
+ "BPCellsHDF5MatrixSource",
+ "BPCellsHDF5Source",
+ "BPCellsMatrixSource",
+ "BPCellsMemoryMatrixSource",
+ "BaseMatrixSource",
+ "AxisMinimumMatrixSource",
+ "BinaryMatrixSource",
+ "BinaryTransformMatrixSource",
+ "CSCMatrixSource",
+ "CastMatrixSource",
+ "CellBindMatrixSource",
+ "CellBindSource",
+ "ColumnBindMatrixSource",
+ "CscMatrixSource",
+ "DEFAULT_LIMITS",
+ "DEFAULT_OPERATION_REGISTRY",
+ "DelayedSubassignmentMatrixSource",
+ "DenseMatrixSource",
+ "DtypeMatrixSource",
+ "FeatureBindMatrixSource",
+ "FeatureBindSource",
+ "FRAGMENT_CAPABILITY_REGISTRY",
+ "FragmentBlock",
+ "FragmentCapability",
+ "FragmentCapabilityRegistry",
+ "FragmentDerivedMatrixSource",
+ "FragmentSource",
+ "H5ADMatrixSource",
+ "H5SparseMatrixSource",
+ "H5adMatrixSource",
+ "HDF5ArrayMatrixSource",
+ "HDF5ArraySource",
+ "HDF5CompressedMatrixSource",
+ "HDF5DenseMatrixSource",
+ "ReshapedHDF5ArrayMatrixSource",
+ "InMemoryDenseMatrixSource",
+ "LayerPlacement",
+ "LayerStitchMatrixSource",
+ "LayerStitchSource",
+ "LazyDenseMatrixSource",
+ "LocalBinaryMatrixSource",
+ "LocalUnaryMatrixSource",
+ "LinearResidualMatrixSource",
+ "MappedMatrixSource",
+ "MaskMatrixSource",
+ "MatrixBlock",
+ "MatrixCscSource",
+ "MatrixMultiplySource",
+ "MatrixOperation",
+ "MatrixOperationRegistry",
+ "MatrixSource",
+ "MatrixSourceError",
+ "MemoryEstimate",
+ "MultiplyMatrixSource",
+ "OperationCapability",
+ "PearsonResidualMatrixSource",
+ "RColumnMajorMatrixSource",
+ "RankMatrixSource",
+ "ScaleShiftMatrixSource",
+ "RenameMatrixSource",
+ "RenamedMatrixSource",
+ "ReorderedMatrixSource",
+ "ResourceLimitError",
+ "RowBindMatrixSource",
+ "SidecarPathResolver",
+ "SourceLimits",
+ "Subassignment",
+ "SubassignmentMatrixSource",
+ "SubsetMatrixSource",
+ "StoredFragmentSource",
+ "TENxMatrixSource",
+ "TenXMatrixSource",
+ "TenxMatrixSource",
+ "TransposeMatrixSource",
+ "TransposeSource",
+ "UnaryMatrixSource",
+ "UnaryTransformMatrixSource",
+ "UnsafeSidecarError",
+ "UnsupportedExecutionMatrixSource",
+ "UnsupportedMatrixOperation",
+ "build_matrix_operation",
+ "decode_bp128",
+ "decode_bp128_d1z",
+ "decode_bp128_m1",
+ "fragment_source_from_slots",
+ "matrix_source_from_mapping",
+ "matrix_source_from_slots",
+ "read_hdf5_names",
+ "require_filesystem_path",
+ "resolve_sidecar_path",
+ "source_from_slot_mapping",
+ "validate_hdf5_file",
+]
diff --git a/scarf/readers/_seurat/bpcells.py b/scarf/readers/_seurat/bpcells.py
new file mode 100644
index 00000000..0ef1c194
--- /dev/null
+++ b/scarf/readers/_seurat/bpcells.py
@@ -0,0 +1,1124 @@
+import math
+import os
+import re
+from collections.abc import Mapping, Sequence
+from pathlib import Path
+from typing import Any, Protocol
+
+import h5py
+import numpy as np
+from numpy.typing import NDArray
+from scipy.sparse import coo_matrix, csr_matrix
+
+from .errors import MatrixSourceError, ResourceLimitError, UnsafeSidecarError
+from .paths import require_hdf5_group, validate_hdf5_file
+from .sources import (
+ DEFAULT_LIMITS,
+ BaseMatrixSource,
+ MemoryEstimate,
+ SourceLimits,
+ _validate_shape,
+)
+
+
+_NUMERIC_HEADERS: dict[bytes, np.dtype[Any]] = {
+ b"UINT32v1": np.dtype(" NDArray[np.uint32]:
+ if bits < 0 or bits > 32:
+ raise MatrixSourceError(f"BP128 bit width {bits} is outside [0, 32]")
+ expected = bits * 4
+ packed = np.asarray(words, dtype=np.uint32).reshape(-1)
+ if packed.size != expected:
+ raise MatrixSourceError(
+ f"BP128 block has {packed.size} words; expected {expected}"
+ )
+ output: NDArray[np.uint32] = np.zeros(128, dtype=np.uint32)
+ if bits == 0:
+ return output
+ vectors = packed.reshape(bits, 4).astype(np.uint64)
+ mask = np.uint64(0xFFFFFFFF if bits == 32 else (1 << bits) - 1)
+ for vector_index in range(32):
+ bit_position = vector_index * bits
+ word_index = bit_position // 32
+ shift = bit_position & 31
+ values = vectors[word_index] >> np.uint64(shift)
+ if shift + bits > 32:
+ values |= vectors[word_index + 1] << np.uint64(32 - shift)
+ output[vector_index * 4 : vector_index * 4 + 4] = (values & mask).astype(
+ np.uint32
+ )
+ return output
+
+
+def _expanded_bp128_indexes(
+ indexes: NDArray[Any],
+ index_offsets: NDArray[Any] | None,
+) -> NDArray[np.uint64]:
+ raw = np.asarray(indexes)
+ if raw.ndim != 1 or not np.issubdtype(raw.dtype, np.integer):
+ raise TypeError("BP128 idx must be a one-dimensional integer array")
+ raw_u64 = raw.astype(np.uint64, copy=False)
+ if index_offsets is None:
+ expanded = raw_u64.copy()
+ else:
+ offsets = np.asarray(index_offsets)
+ if offsets.ndim != 1 or not np.issubdtype(offsets.dtype, np.integer):
+ raise TypeError("BP128 idx_offsets must be a one-dimensional integer array")
+ offsets = offsets.astype(np.int64, copy=False)
+ if (
+ offsets.size < 2
+ or int(offsets[0]) != 0
+ or int(offsets[-1]) != raw.size
+ or np.any(offsets[1:] <= offsets[:-1])
+ ):
+ raise MatrixSourceError(
+ "BP128 idx_offsets must partition the complete idx array"
+ )
+ expanded = raw_u64.copy()
+ for segment in range(1, offsets.size - 1):
+ start = int(offsets[segment])
+ stop = int(offsets[segment + 1])
+ expanded[start:stop] += np.uint64(segment) << np.uint64(32)
+ if expanded.size and int(expanded[0]) != 0:
+ raise MatrixSourceError("BP128 idx must start at zero")
+ if expanded.size > 1 and np.any(expanded[1:] < expanded[:-1]):
+ raise MatrixSourceError("expanded BP128 idx must be nondecreasing")
+ return expanded
+
+
+def decode_bp128(
+ data: NDArray[Any],
+ indexes: NDArray[Any],
+ count: int,
+ *,
+ index_offsets: NDArray[Any] | None = None,
+ transform: str = "plain",
+ starts: NDArray[Any] | None = None,
+ start: int = 0,
+ stop: int | None = None,
+) -> NDArray[np.uint32]:
+ count = int(count)
+ if count < 0:
+ raise ValueError("BP128 count cannot be negative")
+ stop = count if stop is None else int(stop)
+ start = int(start)
+ if start < 0 or stop < start or stop > count:
+ raise IndexError(f"BP128 window [{start}, {stop}) is outside [0, {count})")
+ block_count = math.ceil(count / 128)
+ expanded = _expanded_bp128_indexes(indexes, index_offsets)
+ if expanded.shape != (block_count + 1,):
+ raise MatrixSourceError(
+ f"BP128 idx has length {expanded.size}; expected {block_count + 1}"
+ )
+ packed = np.asarray(data)
+ if packed.ndim != 1 or not np.issubdtype(packed.dtype, np.integer):
+ raise TypeError("BP128 data must be a one-dimensional integer array")
+ if int(expanded[-1]) != packed.size:
+ raise MatrixSourceError(
+ f"BP128 data has {packed.size} words; idx ends at {int(expanded[-1])}"
+ )
+ if transform not in {"plain", "m1", "d1", "d1z"}:
+ raise ValueError(f"unknown BP128 transform {transform!r}")
+ start_values: NDArray[Any] | None = None
+ if transform in {"d1", "d1z"}:
+ if starts is None:
+ raise MatrixSourceError(f"BP128 {transform} requires starts")
+ start_values = np.asarray(starts)
+ if (
+ start_values.ndim != 1
+ or start_values.size != block_count
+ or not np.issubdtype(start_values.dtype, np.integer)
+ ):
+ raise MatrixSourceError(f"BP128 starts must have length {block_count}")
+ if start == stop:
+ return np.empty(0, dtype=np.uint32)
+ first_block = start // 128
+ final_block = (stop - 1) // 128
+ decoded_blocks: list[NDArray[np.uint32]] = []
+ for block in range(first_block, final_block + 1):
+ word_start = int(expanded[block])
+ word_stop = int(expanded[block + 1])
+ word_count = word_stop - word_start
+ if word_count % 4:
+ raise MatrixSourceError(
+ f"BP128 block {block} word count {word_count} is not divisible by four"
+ )
+ values = _unpack_bp128_block(packed[word_start:word_stop], word_count // 4)
+ if transform == "m1":
+ widened = values.astype(np.uint64) + 1
+ if np.any(widened > np.iinfo(np.uint32).max):
+ raise MatrixSourceError("BP128 m1 decode overflows uint32")
+ values = widened.astype(np.uint32)
+ elif transform in {"d1", "d1z"}:
+ assert start_values is not None
+ if transform == "d1z":
+ encoded = values.astype(np.uint64)
+ deltas = (encoded >> np.uint64(1)).astype(np.int64) ^ -(
+ (encoded & np.uint64(1)).astype(np.int64)
+ )
+ else:
+ deltas = values.astype(np.int64)
+ decoded = np.cumsum(deltas, dtype=np.int64)
+ decoded += int(start_values[block])
+ if np.any(decoded < 0) or np.any(decoded > np.iinfo(np.uint32).max):
+ raise MatrixSourceError(f"BP128 {transform} decode leaves uint32 range")
+ values = decoded.astype(np.uint32)
+ decoded_blocks.append(values)
+ combined = np.concatenate(decoded_blocks)
+ local_start = start - first_block * 128
+ return combined[local_start : local_start + stop - start]
+
+
+def decode_bp128_m1(
+ data: NDArray[Any],
+ indexes: NDArray[Any],
+ count: int,
+ *,
+ index_offsets: NDArray[Any] | None = None,
+ start: int = 0,
+ stop: int | None = None,
+) -> NDArray[np.uint32]:
+ return decode_bp128(
+ data,
+ indexes,
+ count,
+ index_offsets=index_offsets,
+ transform="m1",
+ start=start,
+ stop=stop,
+ )
+
+
+def decode_bp128_d1z(
+ data: NDArray[Any],
+ indexes: NDArray[Any],
+ starts: NDArray[Any],
+ count: int,
+ *,
+ index_offsets: NDArray[Any] | None = None,
+ start: int = 0,
+ stop: int | None = None,
+) -> NDArray[np.uint32]:
+ return decode_bp128(
+ data,
+ indexes,
+ count,
+ index_offsets=index_offsets,
+ transform="d1z",
+ starts=starts,
+ start=start,
+ stop=stop,
+ )
+
+
+class _BPArrayStore(Protocol):
+ @property
+ def version(self) -> str: ...
+
+ def has(self, name: str) -> bool: ...
+
+ def numeric_info(self, name: str) -> tuple[np.dtype[Any], int]: ...
+
+ def read_numeric(
+ self,
+ name: str,
+ start: int = 0,
+ stop: int | None = None,
+ ) -> NDArray[Any]: ...
+
+ def read_text(self, name: str) -> tuple[str, ...]: ...
+
+
+def _read_vector_window(
+ values: Any,
+ start: int,
+ stop: int,
+) -> NDArray[Any]:
+ read_block = getattr(values, "read_block", None)
+ if callable(read_block):
+ return np.asarray(read_block(start, stop))
+ return np.asarray(values[start:stop])
+
+
+class _MemoryArrayStore:
+ def __init__(
+ self,
+ version: str,
+ arrays: Mapping[str, Any],
+ dtypes: Mapping[str, np.dtype[Any]],
+ *,
+ float_bit_arrays: frozenset[str],
+ text: Mapping[str, Sequence[str | bytes] | NDArray[Any] | None],
+ limits: SourceLimits,
+ ) -> None:
+ self._version = version
+ self._arrays = dict(arrays)
+ self._dtypes = dict(dtypes)
+ self._floatBitArrays = float_bit_arrays
+ self._text = dict(text)
+ self._limits = limits
+
+ @property
+ def version(self) -> str:
+ return self._version
+
+ def has(self, name: str) -> bool:
+ return name in self._arrays or (
+ name in self._text and self._text[name] is not None
+ )
+
+ def numeric_info(self, name: str) -> tuple[np.dtype[Any], int]:
+ if name not in self._arrays or name not in self._dtypes:
+ raise MatrixSourceError(f"BPCells memory array {name!r} is missing")
+ return self._dtypes[name], len(self._arrays[name])
+
+ def read_numeric(
+ self,
+ name: str,
+ start: int = 0,
+ stop: int | None = None,
+ ) -> NDArray[Any]:
+ dtype, length = self.numeric_info(name)
+ stop = length if stop is None else int(stop)
+ start = int(start)
+ if start < 0 or stop < start or stop > length:
+ raise IndexError(
+ f"BPCells memory array {name!r} window [{start}, {stop}) "
+ f"is outside [0, {length})"
+ )
+ values = _read_vector_window(self._arrays[name], start, stop)
+ if name in self._floatBitArrays:
+ if values.dtype.kind not in "iu" or values.dtype.itemsize != 4:
+ raise TypeError(
+ f"BPCells memory array {name!r} must contain 32-bit "
+ "floating-point bit patterns"
+ )
+ return np.ascontiguousarray(values, dtype=np.uint32).view(np.float32)
+ if dtype.kind in "ui":
+ if not np.issubdtype(values.dtype, np.number):
+ raise TypeError(f"BPCells memory array {name!r} must be numeric")
+ if values.dtype.kind == "f" and (
+ np.any(~np.isfinite(values))
+ or np.any(values != np.floor(values))
+ or np.any(values < 0)
+ ):
+ raise MatrixSourceError(
+ f"BPCells memory array {name!r} contains invalid integer values"
+ )
+ return values.astype(dtype, copy=False)
+
+ def read_text(self, name: str) -> tuple[str, ...]:
+ values = self._text.get(name)
+ if values is None:
+ raise MatrixSourceError(f"BPCells memory text array {name!r} is missing")
+ output: list[str] = []
+ used_bytes = 0
+ for start in range(0, len(values), 4096):
+ stop = min(len(values), start + 4096)
+ read_block = getattr(values, "read_block", None)
+ block = (
+ read_block(start, stop) if callable(read_block) else values[start:stop]
+ )
+ for value in block:
+ if isinstance(value, bytes):
+ try:
+ decoded = value.decode("utf-8")
+ except UnicodeDecodeError as error:
+ raise MatrixSourceError(
+ f"BPCells memory text array {name!r} is not valid UTF-8"
+ ) from error
+ elif isinstance(value, str):
+ decoded = value
+ else:
+ raise TypeError(
+ f"BPCells memory text array {name!r} must contain strings"
+ )
+ used_bytes += len(decoded.encode("utf-8")) + 8
+ if used_bytes > self._limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"BPCells memory text array {name!r} exceeds "
+ f"maxMetadataBytes={self._limits.maxMetadataBytes}"
+ )
+ output.append(decoded)
+ return tuple(output)
+
+
+class _DirectoryArrayStore:
+ def __init__(
+ self,
+ path: str | os.PathLike[str],
+ limits: SourceLimits,
+ ) -> None:
+ self.path = Path(path).expanduser().resolve(strict=False)
+ if not self.path.exists():
+ raise FileNotFoundError(self.path)
+ if not self.path.is_dir():
+ raise UnsafeSidecarError(
+ f"BPCells directory {self.path} is not a directory"
+ )
+ self._limits = limits
+ values = self.read_text("version")
+ if len(values) != 1:
+ raise MatrixSourceError(
+ "BPCells directory version must contain exactly one line"
+ )
+ self._version = values[0]
+
+ @property
+ def version(self) -> str:
+ return self._version
+
+ def _child(self, name: str) -> Path:
+ if "/" in name or name in {"", ".", ".."}:
+ raise UnsafeSidecarError(f"invalid BPCells array name {name!r}")
+ child = self.path / name
+ resolved = child.resolve(strict=False)
+ if not (resolved == self.path or resolved.is_relative_to(self.path)):
+ raise UnsafeSidecarError(f"BPCells array {name!r} escapes its directory")
+ return resolved
+
+ def has(self, name: str) -> bool:
+ child = self._child(name)
+ return child.exists() and child.is_file()
+
+ def numeric_info(self, name: str) -> tuple[np.dtype[Any], int]:
+ child = self._child(name)
+ if not child.exists():
+ raise MatrixSourceError(f"BPCells numeric array {name!r} is missing")
+ if not child.is_file():
+ raise UnsafeSidecarError(
+ f"BPCells numeric array {name!r} is not a regular file"
+ )
+ with child.open("rb") as handle:
+ header = handle.read(8)
+ if header not in _NUMERIC_HEADERS:
+ raise MatrixSourceError(
+ f"BPCells numeric array {name!r} has unknown 8-byte header {header!r}"
+ )
+ dtype = _NUMERIC_HEADERS[header]
+ payload_bytes = child.stat().st_size - 8
+ if payload_bytes < 0 or payload_bytes % dtype.itemsize:
+ raise MatrixSourceError(
+ f"BPCells numeric array {name!r} has a truncated payload"
+ )
+ return dtype, payload_bytes // dtype.itemsize
+
+ def read_numeric(
+ self,
+ name: str,
+ start: int = 0,
+ stop: int | None = None,
+ ) -> NDArray[Any]:
+ dtype, length = self.numeric_info(name)
+ stop = length if stop is None else int(stop)
+ start = int(start)
+ if start < 0 or stop < start or stop > length:
+ raise IndexError(
+ f"BPCells array {name!r} window [{start}, {stop}) "
+ f"is outside [0, {length})"
+ )
+ child = self._child(name)
+ with child.open("rb") as handle:
+ handle.seek(8 + start * dtype.itemsize)
+ payload = handle.read((stop - start) * dtype.itemsize)
+ if len(payload) != (stop - start) * dtype.itemsize:
+ raise MatrixSourceError(f"short read from BPCells numeric array {name!r}")
+ return np.frombuffer(payload, dtype=dtype).copy()
+
+ def read_text(self, name: str) -> tuple[str, ...]:
+ child = self._child(name)
+ if not child.exists():
+ raise MatrixSourceError(f"BPCells text array {name!r} is missing")
+ if not child.is_file():
+ raise UnsafeSidecarError(
+ f"BPCells text array {name!r} is not a regular file"
+ )
+ size = child.stat().st_size
+ if size > self._limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"BPCells text array {name!r} exceeds "
+ f"maxMetadataBytes={self._limits.maxMetadataBytes}"
+ )
+ try:
+ text = child.read_text(encoding="utf-8")
+ except UnicodeDecodeError as error:
+ raise MatrixSourceError(
+ f"BPCells text array {name!r} is not valid UTF-8"
+ ) from error
+ return tuple(text.splitlines())
+
+
+class _HDF5ArrayStore:
+ def __init__(
+ self,
+ path: str | os.PathLike[str] | Any,
+ group: str,
+ limits: SourceLimits,
+ ) -> None:
+ self.path = validate_hdf5_file(path, limits=limits)
+ self.group = "/" + group.strip("/")
+ self._limits = limits
+ with h5py.File(self.path, mode="r") as handle:
+ node = require_hdf5_group(handle, self.group)
+ if "version" not in node.attrs:
+ raise MatrixSourceError(
+ f"BPCells HDF5 group {self.group!r} has no version attribute"
+ )
+ value = np.asarray(node.attrs["version"]).reshape(-1)
+ if value.size != 1:
+ raise MatrixSourceError("BPCells HDF5 version attribute must be scalar")
+ scalar = value[0]
+ if isinstance(scalar, bytes | np.bytes_):
+ self._version = bytes(scalar).decode("utf-8")
+ elif isinstance(scalar, str | np.str_):
+ self._version = str(scalar)
+ else:
+ raise MatrixSourceError("BPCells HDF5 version attribute must be text")
+
+ @property
+ def version(self) -> str:
+ return self._version
+
+ def has(self, name: str) -> bool:
+ with h5py.File(self.path, mode="r") as handle:
+ group = require_hdf5_group(handle, self.group)
+ return name in group and isinstance(group[name], h5py.Dataset)
+
+ def numeric_info(self, name: str) -> tuple[np.dtype[Any], int]:
+ with h5py.File(self.path, mode="r") as handle:
+ group = require_hdf5_group(handle, self.group)
+ if name not in group or not isinstance(group[name], h5py.Dataset):
+ raise MatrixSourceError(
+ f"BPCells HDF5 numeric array {name!r} is missing"
+ )
+ node = group[name]
+ assert isinstance(node, h5py.Dataset)
+ if node.ndim != 1 or node.dtype.kind not in "uif":
+ raise TypeError(
+ f"BPCells HDF5 numeric array {name!r} must be numeric and "
+ "one-dimensional"
+ )
+ return np.dtype(node.dtype), int(node.size)
+
+ def read_numeric(
+ self,
+ name: str,
+ start: int = 0,
+ stop: int | None = None,
+ ) -> NDArray[Any]:
+ dtype, length = self.numeric_info(name)
+ stop = length if stop is None else int(stop)
+ start = int(start)
+ if start < 0 or stop < start or stop > length:
+ raise IndexError(
+ f"BPCells HDF5 array {name!r} window [{start}, {stop}) "
+ f"is outside [0, {length})"
+ )
+ with h5py.File(self.path, mode="r") as handle:
+ group = require_hdf5_group(handle, self.group)
+ node = group[name]
+ assert isinstance(node, h5py.Dataset)
+ return np.asarray(node[start:stop], dtype=dtype)
+
+ def read_text(self, name: str) -> tuple[str, ...]:
+ with h5py.File(self.path, mode="r") as handle:
+ group = require_hdf5_group(handle, self.group)
+ if name not in group or not isinstance(group[name], h5py.Dataset):
+ raise MatrixSourceError(f"BPCells HDF5 text array {name!r} is missing")
+ node = group[name]
+ assert isinstance(node, h5py.Dataset)
+ if node.ndim > 1:
+ raise MatrixSourceError(
+ f"BPCells HDF5 text array {name!r} must be scalar or 1D"
+ )
+ string_info = h5py.check_string_dtype(node.dtype)
+ if string_info is None:
+ raise MatrixSourceError(
+ f"BPCells HDF5 text array {name!r} must contain strings"
+ )
+ count = int(node.size)
+ if count * 8 > self._limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"BPCells HDF5 text array {name!r} exceeds "
+ f"maxMetadataBytes={self._limits.maxMetadataBytes}"
+ )
+ if (
+ string_info.length is not None
+ and count * (string_info.length + 8) > self._limits.maxMetadataBytes
+ ):
+ raise ResourceLimitError(
+ f"BPCells HDF5 text array {name!r} exceeds "
+ f"maxMetadataBytes={self._limits.maxMetadataBytes}"
+ )
+ output: list[str] = []
+ size = 0
+ for start in range(0, count, 4096):
+ stop = min(count, start + 4096)
+ values = np.asarray(
+ node[()] if node.ndim == 0 else node[start:stop]
+ ).reshape(-1)
+ for value in values:
+ if isinstance(value, bytes | np.bytes_):
+ try:
+ decoded = bytes(value).decode("utf-8")
+ except UnicodeDecodeError as error:
+ raise MatrixSourceError(
+ f"BPCells HDF5 text array {name!r} is not valid UTF-8"
+ ) from error
+ elif isinstance(value, str | np.str_):
+ decoded = str(value)
+ else:
+ raise MatrixSourceError(
+ f"BPCells HDF5 text array {name!r} must contain strings"
+ )
+ size += len(decoded.encode("utf-8")) + 8
+ if size > self._limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"BPCells HDF5 text array {name!r} exceeds "
+ f"maxMetadataBytes={self._limits.maxMetadataBytes}"
+ )
+ output.append(decoded)
+ return tuple(output)
+
+
+def _require_numeric_array(
+ store: _BPArrayStore,
+ name: str,
+ dtype: np.dtype[Any],
+ length: int | None = None,
+) -> int:
+ actual_dtype, actual_length = store.numeric_info(name)
+ if actual_dtype.kind != dtype.kind or actual_dtype.itemsize != dtype.itemsize:
+ raise MatrixSourceError(
+ f"BPCells array {name!r} has dtype {actual_dtype}; expected {dtype}"
+ )
+ if length is not None and actual_length != length:
+ raise MatrixSourceError(
+ f"BPCells array {name!r} has length {actual_length}; expected {length}"
+ )
+ return actual_length
+
+
+class _StoredBP128Array:
+ def __init__(
+ self,
+ store: _BPArrayStore,
+ prefix: str,
+ count: int,
+ transform: str,
+ *,
+ require_offsets: bool,
+ limits: SourceLimits,
+ ) -> None:
+ self.store = store
+ self.prefix = prefix
+ self.count = int(count)
+ self.transform = transform
+ self.blockCount = math.ceil(self.count / 128)
+ self.limits = limits
+ _require_numeric_array(
+ store, f"{prefix}_idx", np.dtype("uint32"), self.blockCount + 1
+ )
+ self.dataLength = _require_numeric_array(
+ store, f"{prefix}_data", np.dtype("uint32")
+ )
+ offset_name = f"{prefix}_idx_offsets"
+ if require_offsets:
+ if not store.has(offset_name):
+ raise MatrixSourceError(
+ f"BPCells packed v2 array {prefix!r} has no idx_offsets"
+ )
+ offset_length = _require_numeric_array(
+ store, offset_name, np.dtype("uint64")
+ )
+ if offset_length * np.dtype(np.uint64).itemsize > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"BPCells {offset_name} exceeds metadata limit"
+ )
+ self.indexOffsets = store.read_numeric(offset_name).astype(
+ np.int64, copy=False
+ )
+ else:
+ self.indexOffsets = np.array([0, self.blockCount + 1], dtype=np.int64)
+ if (
+ self.indexOffsets.size < 2
+ or int(self.indexOffsets[0]) != 0
+ or int(self.indexOffsets[-1]) != self.blockCount + 1
+ or np.any(self.indexOffsets[1:] <= self.indexOffsets[:-1])
+ ):
+ raise MatrixSourceError(f"BPCells {offset_name} does not partition idx")
+ if transform in {"d1", "d1z"}:
+ _require_numeric_array(
+ store,
+ f"{prefix}_starts",
+ np.dtype("uint32"),
+ self.blockCount,
+ )
+ self._validate_index_end()
+
+ def _idx_high(self, position: int) -> int:
+ boundaries = self.indexOffsets[1:-1]
+ return int(np.searchsorted(boundaries, position, side="right")) << 32
+
+ def _expanded_idx_pair(self, block: int) -> tuple[int, int]:
+ raw = self.store.read_numeric(f"{self.prefix}_idx", block, block + 2).astype(
+ np.uint64, copy=False
+ )
+ first = int(raw[0]) + self._idx_high(block)
+ second = int(raw[1]) + self._idx_high(block + 1)
+ if second < first:
+ raise MatrixSourceError(
+ f"BPCells {self.prefix}_idx decreases at block {block}"
+ )
+ return first, second
+
+ def _validate_index_end(self) -> None:
+ first = self.store.read_numeric(f"{self.prefix}_idx", 0, 1)
+ if first.size != 1 or int(first[0]) != 0:
+ raise MatrixSourceError(f"BPCells {self.prefix}_idx must start at zero")
+ raw_end = self.store.read_numeric(
+ f"{self.prefix}_idx", self.blockCount, self.blockCount + 1
+ )
+ expanded_end = int(raw_end[0]) + self._idx_high(self.blockCount)
+ if expanded_end != self.dataLength:
+ raise MatrixSourceError(
+ f"BPCells {self.prefix}_idx ends at {expanded_end}; "
+ f"data has length {self.dataLength}"
+ )
+
+ def read(self, start: int, stop: int) -> NDArray[np.uint32]:
+ start = int(start)
+ stop = int(stop)
+ if start < 0 or stop < start or stop > self.count:
+ raise IndexError(
+ f"BPCells packed window [{start}, {stop}) is outside [0, {self.count})"
+ )
+ if start == stop:
+ return np.empty(0, dtype=np.uint32)
+ first_block = start // 128
+ final_block = (stop - 1) // 128
+ output: list[NDArray[np.uint32]] = []
+ for block in range(first_block, final_block + 1):
+ word_start, word_stop = self._expanded_idx_pair(block)
+ word_count = word_stop - word_start
+ if word_count % 4:
+ raise MatrixSourceError(
+ f"BPCells {self.prefix} block {block} has invalid word count"
+ )
+ values = _unpack_bp128_block(
+ self.store.read_numeric(f"{self.prefix}_data", word_start, word_stop),
+ word_count // 4,
+ )
+ if self.transform == "m1":
+ widened = values.astype(np.uint64) + 1
+ if np.any(widened > np.iinfo(np.uint32).max):
+ raise MatrixSourceError("BP128 m1 decode overflows uint32")
+ values = widened.astype(np.uint32)
+ elif self.transform in {"d1", "d1z"}:
+ encoded = values.astype(np.uint64)
+ if self.transform == "d1z":
+ deltas = (encoded >> np.uint64(1)).astype(np.int64) ^ -(
+ (encoded & np.uint64(1)).astype(np.int64)
+ )
+ else:
+ deltas = encoded.astype(np.int64)
+ start_value = int(
+ self.store.read_numeric(f"{self.prefix}_starts", block, block + 1)[
+ 0
+ ]
+ )
+ decoded = np.cumsum(deltas, dtype=np.int64) + start_value
+ if np.any(decoded < 0) or np.any(decoded > np.iinfo(np.uint32).max):
+ raise MatrixSourceError(
+ f"BP128 {self.transform} decode leaves uint32 range"
+ )
+ values = decoded.astype(np.uint32)
+ output.append(values)
+ combined = np.concatenate(output)
+ local_start = start - first_block * 128
+ return combined[local_start : local_start + stop - start]
+
+
+class BPCellsMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ store: _BPArrayStore,
+ *,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ match = _VERSION_PATTERN.fullmatch(store.version)
+ if match is None:
+ raise MatrixSourceError(
+ f"unsupported BPCells matrix format {store.version!r}"
+ )
+ compression, datatype, version_text = match.groups()
+ self.store = store
+ self.compression = compression
+ self.datatype = datatype
+ self.formatVersion = int(version_text)
+ _require_numeric_array(store, "shape", np.dtype("uint32"), 2)
+ raw_shape = store.read_numeric("shape")
+ shape = _validate_shape(
+ (int(raw_shape[0]), int(raw_shape[1])),
+ limits,
+ )
+ order_values = store.read_text("storage_order")
+ if len(order_values) != 1 or order_values[0] not in {"row", "col"}:
+ raise MatrixSourceError(
+ "BPCells storage_order must contain exactly 'row' or 'col'"
+ )
+ self.storageOrder = order_values[0]
+ compressed_axis = shape[1] if self.storageOrder == "col" else shape[0]
+ pointer_dtype = np.dtype("uint32" if self.formatVersion == 1 else "uint64")
+ _require_numeric_array(store, "idxptr", pointer_dtype, compressed_axis + 1)
+ self._nnz = self._validate_pointers(compressed_axis, limits)
+ if self._nnz > limits.maxNnz:
+ raise ResourceLimitError(
+ f"BPCells nnz {self._nnz} exceeds maxNnz={limits.maxNnz}"
+ )
+ matrix_dtype: np.dtype[Any]
+ self._indexReader: _StoredBP128Array | None
+ self._valueReader: _StoredBP128Array | None
+ if datatype == "uint":
+ matrix_dtype = np.dtype(np.uint32)
+ elif datatype == "float":
+ matrix_dtype = np.dtype(np.float32)
+ else:
+ matrix_dtype = np.dtype(np.float64)
+ if compression == "packed":
+ self._indexReader = _StoredBP128Array(
+ store,
+ "index",
+ self._nnz,
+ "d1z",
+ require_offsets=self.formatVersion == 2,
+ limits=limits,
+ )
+ self._valueReader = (
+ _StoredBP128Array(
+ store,
+ "val",
+ self._nnz,
+ "m1",
+ require_offsets=self.formatVersion == 2,
+ limits=limits,
+ )
+ if datatype == "uint"
+ else None
+ )
+ if datatype != "uint":
+ _require_numeric_array(store, "val", matrix_dtype, self._nnz)
+ else:
+ self._indexReader = None
+ self._valueReader = None
+ _require_numeric_array(store, "index", np.dtype("uint32"), self._nnz)
+ _require_numeric_array(store, "val", matrix_dtype, self._nnz)
+ row_names = self._optional_names("row_names", shape[0])
+ column_names = self._optional_names("col_names", shape[1])
+ super().__init__(
+ shape,
+ matrix_dtype,
+ row_names=row_names,
+ column_names=column_names,
+ is_sparse=True,
+ limits=limits,
+ )
+ self._validate_indexes(limits)
+
+ def _optional_names(
+ self,
+ name: str,
+ length: int,
+ ) -> tuple[str, ...] | None:
+ if not self.store.has(name):
+ return None
+ values = self.store.read_text(name)
+ if len(values) == 0:
+ return None
+ if len(values) != length:
+ raise MatrixSourceError(
+ f"BPCells {name} has length {len(values)}; expected {length}"
+ )
+ return values
+
+ def _validate_pointers(
+ self,
+ compressed_axis: int,
+ limits: SourceLimits,
+ ) -> int:
+ previous: int | None = None
+ final = 0
+ chunk = max(1, min(compressed_axis + 1, limits.compressedChunkNnz))
+ for start in range(0, compressed_axis + 1, chunk):
+ stop = min(compressed_axis + 1, start + chunk)
+ values = self.store.read_numeric("idxptr", start, stop)
+ if values.size > 1 and np.any(values[1:] < values[:-1]):
+ raise MatrixSourceError("BPCells idxptr must be nondecreasing")
+ if previous is not None and values.size and int(values[0]) < previous:
+ raise MatrixSourceError("BPCells idxptr must be nondecreasing")
+ if start == 0 and (not values.size or int(values[0]) != 0):
+ raise MatrixSourceError("BPCells idxptr must start at zero")
+ if values.size:
+ previous = int(values[-1])
+ final = previous
+ return final
+
+ def _read_indexes(self, start: int, stop: int) -> NDArray[np.uint32]:
+ if self._indexReader is not None:
+ return self._indexReader.read(start, stop)
+ return self.store.read_numeric("index", start, stop).astype(
+ np.uint32, copy=False
+ )
+
+ def _read_values(self, start: int, stop: int) -> NDArray[Any]:
+ if self._valueReader is not None:
+ return self._valueReader.read(start, stop)
+ return self.store.read_numeric("val", start, stop).astype(
+ self.dtype, copy=False
+ )
+
+ def _validate_indexes(self, limits: SourceLimits) -> None:
+ minor_axis = self.shape[0] if self.storageOrder == "col" else self.shape[1]
+ for start in range(0, self._nnz, limits.compressedChunkNnz):
+ stop = min(self._nnz, start + limits.compressedChunkNnz)
+ indexes = self._read_indexes(start, stop)
+ if indexes.size and np.any(indexes >= minor_axis):
+ raise MatrixSourceError("BPCells index contains an out-of-range value")
+
+ @property
+ def nnz(self) -> int:
+ return self._nnz
+
+ @property
+ def resident_bytes(self) -> int:
+ offsets = 0
+ if self._indexReader is not None:
+ offsets += self._indexReader.indexOffsets.nbytes
+ if self._valueReader is not None:
+ offsets += self._valueReader.indexOffsets.nbytes
+ return int(super().resident_bytes + offsets)
+
+ def _bounds(
+ self,
+ start: int,
+ stop: int,
+ ) -> tuple[NDArray[np.int64], int, int]:
+ pointers = self.store.read_numeric("idxptr", start, stop + 1).astype(
+ np.int64, copy=False
+ )
+ data_start = int(pointers[0])
+ data_stop = int(pointers[-1])
+ return pointers - data_start, data_start, data_stop
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ index_size = np.dtype(np.int64).itemsize
+ if self.storageOrder == "col":
+ pointers, data_start, data_stop = self._bounds(start, stop)
+ nnz = data_stop - data_start
+ output = nnz * (self.dtype.itemsize + index_size) + pointers.nbytes
+ working = output + 128 * np.dtype(np.uint32).itemsize
+ return MemoryEstimate(self.resident_bytes, working, output)
+ max_output_nnz = min(self.nnz, (stop - start) * self.n_features)
+ output = max_output_nnz * (self.dtype.itemsize + 2 * index_size)
+ output += (stop - start + 1) * index_size
+ working = min(self.nnz, self._limits.compressedChunkNnz) * (
+ self.dtype.itemsize + 2 * index_size
+ )
+ return MemoryEstimate(self.resident_bytes, working, output)
+
+ def read_cells(self, start: int, stop: int) -> csr_matrix:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ if self.storageOrder == "col":
+ pointers, data_start, data_stop = self._bounds(start, stop)
+ return csr_matrix(
+ (
+ self._read_values(data_start, data_stop),
+ self._read_indexes(data_start, data_stop).astype(
+ np.int64, copy=False
+ ),
+ pointers,
+ ),
+ shape=(stop - start, self.n_features),
+ dtype=self.dtype,
+ )
+ return self._read_row_stored(start, stop)
+
+ def _read_row_stored(self, start: int, stop: int) -> csr_matrix:
+ if start == stop:
+ return csr_matrix((0, self.n_features), dtype=self.dtype)
+ data_parts: list[NDArray[Any]] = []
+ row_parts: list[NDArray[np.int64]] = []
+ column_parts: list[NDArray[np.int64]] = []
+ retained = 0
+ for feature in range(self.n_features):
+ bounds = self.store.read_numeric("idxptr", feature, feature + 2)
+ vector_start = int(bounds[0])
+ vector_stop = int(bounds[1])
+ for chunk_start in range(
+ vector_start,
+ vector_stop,
+ self._limits.compressedChunkNnz,
+ ):
+ chunk_stop = min(
+ vector_stop,
+ chunk_start + self._limits.compressedChunkNnz,
+ )
+ cells = self._read_indexes(chunk_start, chunk_stop).astype(
+ np.int64, copy=False
+ )
+ keep = (cells >= start) & (cells < stop)
+ count = int(np.count_nonzero(keep))
+ if count == 0:
+ continue
+ retained += count
+ required = retained * (
+ self.dtype.itemsize + 2 * np.dtype(np.int64).itemsize
+ )
+ if required > self._limits.maxBlockBytes:
+ raise ResourceLimitError(
+ "BPCells sparse block exceeds "
+ f"maxBlockBytes={self._limits.maxBlockBytes}"
+ )
+ data_parts.append(self._read_values(chunk_start, chunk_stop)[keep])
+ row_parts.append(cells[keep] - start)
+ column_parts.append(np.full(count, feature, dtype=np.int64))
+ if not data_parts:
+ return csr_matrix((stop - start, self.n_features), dtype=self.dtype)
+ return coo_matrix(
+ (
+ np.concatenate(data_parts),
+ (np.concatenate(row_parts), np.concatenate(column_parts)),
+ ),
+ shape=(stop - start, self.n_features),
+ dtype=self.dtype,
+ ).tocsr()
+
+
+class BPCellsMemoryMatrixSource(BPCellsMatrixSource):
+ def __init__(
+ self,
+ version: str,
+ arrays: Mapping[str, Any],
+ *,
+ shape: tuple[int, int],
+ storage_order: str,
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ float_bit_arrays: frozenset[str] = frozenset(),
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ match = _VERSION_PATTERN.fullmatch(version)
+ if match is None:
+ raise MatrixSourceError(f"unsupported BPCells matrix format {version!r}")
+ compression, datatype, version_text = match.groups()
+ if storage_order not in {"row", "col"}:
+ raise MatrixSourceError("BPCells memory storage order must be row or col")
+ pointer_dtype = np.dtype("uint32" if int(version_text) == 1 else "uint64")
+ value_dtypes: dict[str, np.dtype[Any]] = {
+ "uint": np.dtype(np.uint32),
+ "float": np.dtype(np.float32),
+ "double": np.dtype(np.float64),
+ }
+ value_dtype = value_dtypes[datatype]
+ normalized_arrays = {
+ **arrays,
+ "shape": np.asarray(shape, dtype=np.uint32),
+ }
+ dtypes: dict[str, np.dtype[Any]] = {
+ "shape": np.dtype(np.uint32),
+ "idxptr": pointer_dtype,
+ "index": np.dtype(np.uint32),
+ "index_data": np.dtype(np.uint32),
+ "index_starts": np.dtype(np.uint32),
+ "index_idx": np.dtype(np.uint32),
+ "index_idx_offsets": np.dtype(np.uint64),
+ "val": value_dtype,
+ "val_data": np.dtype(np.uint32),
+ "val_idx": np.dtype(np.uint32),
+ "val_idx_offsets": np.dtype(np.uint64),
+ }
+ if compression == "unpacked":
+ required = {"idxptr", "index", "val"}
+ elif datatype == "uint":
+ required = {
+ "idxptr",
+ "index_data",
+ "index_starts",
+ "index_idx",
+ "val_data",
+ "val_idx",
+ }
+ else:
+ required = {
+ "idxptr",
+ "index_data",
+ "index_starts",
+ "index_idx",
+ "val",
+ }
+ if int(version_text) == 2 and compression == "packed":
+ required.add("index_idx_offsets")
+ if datatype == "uint":
+ required.add("val_idx_offsets")
+ missing = sorted(required.difference(normalized_arrays))
+ if missing:
+ raise MatrixSourceError(
+ f"BPCells memory matrix is missing arrays {missing!r}"
+ )
+ store = _MemoryArrayStore(
+ version,
+ normalized_arrays,
+ dtypes,
+ float_bit_arrays=float_bit_arrays,
+ text={
+ "storage_order": (storage_order,),
+ "row_names": row_names,
+ "col_names": column_names,
+ },
+ limits=limits,
+ )
+ super().__init__(store, limits=limits)
+
+
+class BPCellsDirectoryMatrixSource(BPCellsMatrixSource):
+ def __init__(
+ self,
+ path: str | os.PathLike[str],
+ *,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ super().__init__(_DirectoryArrayStore(path, limits), limits=limits)
+ self.path = Path(path).expanduser().resolve(strict=False)
+
+
+class BPCellsHDF5MatrixSource(BPCellsMatrixSource):
+ def __init__(
+ self,
+ path: str | os.PathLike[str] | Any,
+ *,
+ group: str,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ store = _HDF5ArrayStore(path, group, limits)
+ super().__init__(store, limits=limits)
+ self.path = store.path
+ self.group = store.group
+
+
+BPCellsDirectorySource = BPCellsDirectoryMatrixSource
+BPCellsDirMatrixSource = BPCellsDirectoryMatrixSource
+BPCellsH5MatrixSource = BPCellsHDF5MatrixSource
+BPCellsHDF5Source = BPCellsHDF5MatrixSource
diff --git a/scarf/readers/_seurat/errors.py b/scarf/readers/_seurat/errors.py
new file mode 100644
index 00000000..a84f08a5
--- /dev/null
+++ b/scarf/readers/_seurat/errors.py
@@ -0,0 +1,34 @@
+class MatrixSourceError(ValueError):
+ """Base error for matrix source validation."""
+
+
+class ResourceLimitError(MatrixSourceError):
+ """A source or requested block exceeds configured limits."""
+
+
+class UnsafeSidecarError(MatrixSourceError):
+ """A sidecar uses a path or HDF5 feature that is not safe to load."""
+
+
+class UnsupportedMatrixOperation(NotImplementedError):
+ """A serialized matrix operation has no safe local implementation."""
+
+ def __init__(
+ self,
+ object_path: str,
+ operation: str,
+ class_name: str | None = None,
+ reason: str | None = None,
+ ) -> None:
+ details = f" at {object_path}: operation {operation!r}"
+ if class_name is not None:
+ details += f", class {class_name!r}"
+ if reason is not None:
+ details += f" ({reason})"
+ super().__init__("Unsupported matrix operation" + details)
+ self.objectPath = object_path
+ self.object_path = object_path
+ self.operation = operation
+ self.className = class_name
+ self.class_name = class_name
+ self.reason = reason
diff --git a/scarf/readers/_seurat/factory.py b/scarf/readers/_seurat/factory.py
new file mode 100644
index 00000000..c3f329a0
--- /dev/null
+++ b/scarf/readers/_seurat/factory.py
@@ -0,0 +1,1572 @@
+import os
+from collections.abc import Callable, Mapping, Sequence
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+from numpy.typing import NDArray
+
+from .bpcells import (
+ BPCellsDirectoryMatrixSource,
+ BPCellsHDF5MatrixSource,
+ BPCellsMemoryMatrixSource,
+)
+from .errors import (
+ MatrixSourceError,
+ UnsafeSidecarError,
+ UnsupportedMatrixOperation,
+)
+from .fragments import (
+ FRAGMENT_CAPABILITY_REGISTRY,
+ FragmentSource,
+ fragment_source_from_slots,
+)
+from .hdf5 import (
+ H5ADMatrixSource,
+ H5SparseMatrixSource,
+ HDF5ArrayMatrixSource,
+ ReshapedHDF5ArrayMatrixSource,
+ TENxMatrixSource,
+)
+from .operations import (
+ AxisMinimumMatrixSource,
+ LinearResidualMatrixSource,
+ PearsonResidualMatrixSource,
+ ScaleShiftMatrixSource,
+ build_matrix_operation,
+)
+from .paths import SidecarPathResolver, require_filesystem_path
+from .sources import (
+ DEFAULT_LIMITS,
+ CscMatrixSource,
+ DenseMatrixSource,
+ MatrixSource,
+ RenamedMatrixSource,
+ SourceLimits,
+ TransposeMatrixSource,
+)
+
+
+_DENSE_CLASSES = frozenset(
+ {
+ "matrix",
+ "array",
+ "denseMatrix",
+ "dgeMatrix",
+ "lgeMatrix",
+ "ngeMatrix",
+ "igeMatrix",
+ }
+)
+_CSC_CLASSES = frozenset(
+ {
+ "CsparseMatrix",
+ "dgCMatrix",
+ "lgCMatrix",
+ "ngCMatrix",
+ "igCMatrix",
+ "CSC",
+ }
+)
+_BPCELLS_DIRECTORY_CLASSES = frozenset({"MatrixDir", "BPCellsMatrixDir"})
+_BPCELLS_HDF5_CLASSES = frozenset({"MatrixH5", "BPCellsMatrixH5"})
+_BPCELLS_MEMORY_CLASSES: dict[str, tuple[str, str]] = {
+ "PackedMatrixMem_uint32_t": ("packed", "uint"),
+ "PackedMatrixMem_float": ("packed", "float"),
+ "PackedMatrixMem_double": ("packed", "double"),
+ "UnpackedMatrixMem_uint32_t": ("unpacked", "uint"),
+ "UnpackedMatrixMem_float": ("unpacked", "float"),
+ "UnpackedMatrixMem_double": ("unpacked", "double"),
+}
+_HDF5_DENSE_CLASSES = frozenset({"HDF5ArraySeed", "Dense_H5ADArraySeed"})
+_HDF5_RESHAPED_CLASSES = frozenset({"ReshapedHDF5ArraySeed"})
+_H5_SPARSE_CLASSES = frozenset(
+ {
+ "H5SparseMatrixSeed",
+ "CSC_H5SparseMatrixSeed",
+ "CSR_H5SparseMatrixSeed",
+ }
+)
+_H5AD_CLASSES = frozenset(
+ {
+ "H5ADMatrixSeed",
+ "Dense_H5ADMatrixSeed",
+ "CSC_H5ADMatrixSeed",
+ "CSR_H5ADMatrixSeed",
+ "AnnDataMatrixH5",
+ }
+)
+_TENX_CLASSES = frozenset({"TENxMatrixSeed", "10xMatrixH5"})
+_HDF5_WRAPPER_CLASSES = frozenset(
+ {
+ "HDF5Array",
+ "HDF5Matrix",
+ "H5SparseMatrix",
+ "H5ADMatrix",
+ "TENxMatrix",
+ }
+)
+
+
+def _class_names(value: Any) -> tuple[str, ...]:
+ if value is None:
+ return ()
+ if isinstance(value, str):
+ return (value,)
+ if isinstance(value, Sequence) and not isinstance(value, str | bytes):
+ result = tuple(str(item) for item in value)
+ if not result:
+ raise MatrixSourceError("matrix class vector cannot be empty")
+ return result
+ return (str(value),)
+
+
+def _first_value(slots: Mapping[str, Any], *names: str) -> Any:
+ for name in names:
+ if name in slots:
+ return slots[name]
+ raise MatrixSourceError(f"matrix slots are missing one of {names!r}")
+
+
+def _optional_value(
+ slots: Mapping[str, Any],
+ *names: str,
+ default: Any = None,
+) -> Any:
+ for name in names:
+ if name in slots:
+ return slots[name]
+ return default
+
+
+def _slot_array(value: Any, *, dtype: Any = None) -> NDArray[Any]:
+ read_block = getattr(value, "read_block", None)
+ raw = read_block(0, len(value)) if callable(read_block) else value
+ return np.asarray(raw, dtype=dtype)
+
+
+def _slot_shape(value: Any, *, object_path: str) -> tuple[int, int]:
+ values = _slot_array(value)
+ if (
+ values.ndim != 1
+ or values.size != 2
+ or not np.issubdtype(values.dtype, np.number)
+ or np.any(~np.isfinite(values))
+ or np.any(values != np.floor(values))
+ ):
+ raise MatrixSourceError(f"dim slot at {object_path} must contain two integers")
+ shape = (int(values[0]), int(values[1]))
+ if min(shape) < 0:
+ raise MatrixSourceError(f"dim slot at {object_path} cannot be negative")
+ return shape
+
+
+def _slot_text(value: Any, *, slot_name: str, object_path: str) -> str:
+ if isinstance(value, str):
+ return value
+ values = value.read_block(0, len(value)) if hasattr(value, "read_block") else value
+ if (
+ not isinstance(values, Sequence | np.ndarray)
+ or isinstance(values, bytes | bytearray)
+ or len(values) != 1
+ ):
+ raise MatrixSourceError(f"{slot_name} at {object_path} must contain one string")
+ item = values[0]
+ if isinstance(item, bytes):
+ try:
+ return item.decode("utf-8")
+ except UnicodeDecodeError as error:
+ raise MatrixSourceError(
+ f"{slot_name} at {object_path} is not valid UTF-8"
+ ) from error
+ if not isinstance(item, str):
+ raise MatrixSourceError(f"{slot_name} at {object_path} must contain one string")
+ return item
+
+
+def _slot_boolean(
+ value: Any,
+ *,
+ slot_name: str,
+ object_path: str,
+) -> bool:
+ values = _slot_array(value).reshape(-1)
+ if values.size != 1:
+ raise MatrixSourceError(
+ f"{slot_name} at {object_path} must contain one logical value"
+ )
+ return bool(values[0])
+
+
+def _parameter_matrix(
+ slots: Mapping[str, Any],
+ name: str,
+ *,
+ object_path: str,
+) -> NDArray[Any]:
+ value = slots.get(name)
+ if value is None:
+ return np.empty((0, 0), dtype=np.float64)
+ values = _slot_array(value)
+ if values.ndim == 1:
+ if values.size == 0:
+ return np.empty((0, 0), dtype=values.dtype)
+ return values.reshape(1, -1)
+ if values.ndim != 2:
+ raise MatrixSourceError(
+ f"{name} at {object_path} must be a two-dimensional matrix"
+ )
+ return values
+
+
+def _operation_arguments(
+ value: Any,
+ *,
+ object_path: str,
+) -> tuple[int | float | complex, ...]:
+ if value is None:
+ return ()
+ if isinstance(value, Mapping):
+ raw_values = tuple(value.values())
+ elif isinstance(value, Sequence) and not isinstance(value, str | bytes | bytearray):
+ raw_values = tuple(value)
+ else:
+ raw_values = (value,)
+ output: list[int | float | complex] = []
+ for index, raw in enumerate(raw_values):
+ values = _slot_array(raw).reshape(-1)
+ if values.size != 1:
+ raise UnsupportedMatrixOperation(
+ f"{object_path}[{index}]",
+ "delayed-function-argument",
+ None,
+ "only scalar numeric arguments are supported",
+ )
+ scalar = values[0]
+ if isinstance(scalar, np.generic):
+ scalar = scalar.item()
+ if isinstance(scalar, bool) or not isinstance(
+ scalar,
+ int | float | complex,
+ ):
+ raise UnsupportedMatrixOperation(
+ f"{object_path}[{index}]",
+ "delayed-function-argument",
+ None,
+ "only scalar numeric arguments are supported",
+ )
+ output.append(scalar)
+ return tuple(output)
+
+
+def _dimnames(
+ slots: Mapping[str, Any],
+) -> tuple[Sequence[str | bytes] | np.ndarray[Any, Any] | None, ...]:
+ row_names = _optional_value(slots, "rowNames", "row_names")
+ column_names = _optional_value(slots, "columnNames", "column_names")
+ if row_names is not None or column_names is not None:
+ return row_names, column_names
+ values = _optional_value(slots, "Dimnames", "dimnames")
+ if values is None:
+ return None, None
+ if (
+ not isinstance(values, Sequence)
+ or isinstance(values, str | bytes)
+ or len(values) != 2
+ ):
+ raise MatrixSourceError("dimnames must contain row and column names")
+ return values[0], values[1]
+
+
+def _sidecar_path(
+ value: Any,
+ *,
+ rds_path: str | os.PathLike[str] | None,
+ absolute_prefix_remaps: Mapping[str | os.PathLike[str], str | os.PathLike[str]]
+ | None,
+ expect: str,
+) -> Path:
+ if rds_path is None:
+ if expect == "file":
+ return require_filesystem_path(value)
+ if not isinstance(value, str | os.PathLike):
+ raise TypeError("sidecar directory must be a filesystem path")
+ path = Path(value).expanduser().resolve(strict=False)
+ if not path.exists():
+ raise FileNotFoundError(path)
+ if not path.is_dir():
+ raise UnsafeSidecarError(f"sidecar path {path} is not a directory")
+ return path
+ return SidecarPathResolver(
+ rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ ).resolve(value, expect=expect)
+
+
+def _finalize_slot_source(
+ source: MatrixSource,
+ slots: Mapping[str, Any],
+ row_names: Sequence[str | bytes] | np.ndarray[Any, Any] | None,
+ column_names: Sequence[str | bytes] | np.ndarray[Any, Any] | None,
+ limits: SourceLimits,
+) -> MatrixSource:
+ transpose_value = _optional_value(slots, "transpose", default=False)
+ transpose_array = (
+ np.asarray(transpose_value.read_block(0, len(transpose_value)))
+ if hasattr(transpose_value, "read_block")
+ else np.asarray(transpose_value)
+ )
+ if transpose_array.size != 1:
+ raise MatrixSourceError("transpose slot must be scalar")
+ expected_shape_value = _optional_value(slots, "Dim", "dim")
+ expected_shape: tuple[int, int] | None = None
+ if expected_shape_value is not None:
+ expected_array = (
+ np.asarray(expected_shape_value.read_block(0, len(expected_shape_value)))
+ if hasattr(expected_shape_value, "read_block")
+ else np.asarray(expected_shape_value)
+ )
+ if (
+ expected_array.ndim != 1
+ or expected_array.size != 2
+ or not np.issubdtype(expected_array.dtype, np.number)
+ or np.any(~np.isfinite(expected_array))
+ or np.any(expected_array != np.floor(expected_array))
+ ):
+ raise MatrixSourceError("dim slot must contain two integers")
+ expected_shape = (int(expected_array[0]), int(expected_array[1]))
+ if min(expected_shape) < 0:
+ raise MatrixSourceError("dim slot cannot contain negative values")
+ should_transpose = bool(transpose_array.reshape(-1)[0])
+ if expected_shape is not None:
+ if source.shape == expected_shape:
+ should_transpose = False
+ elif source.shape[::-1] == expected_shape:
+ should_transpose = True
+ else:
+ raise MatrixSourceError(
+ f"matrix source shape {source.shape} does not match dim slot "
+ f"{expected_shape}"
+ )
+ if should_transpose:
+ source = TransposeMatrixSource(source, limits=limits)
+ if row_names is not None or column_names is not None:
+ source = RenamedMatrixSource(
+ source,
+ row_names=row_names,
+ column_names=column_names,
+ limits=limits,
+ )
+ return source
+
+
+_UNARY_TRANSFORM_CLASSES = {
+ "TransformAbs": "abs",
+ "TransformExpm1": "expm1",
+ "TransformExpm1Slow": "expm1",
+ "TransformLog1p": "log1p",
+ "TransformLog1pSlow": "log1p",
+ "TransformNegate": "negative",
+ "TransformRound": "round",
+ "TransformSign": "sign",
+ "TransformSqrt": "sqrt",
+ "TransformSquare": "square",
+}
+
+
+def _r_selection(
+ slots: Mapping[str, Any],
+ name: str,
+ zero_dims: NDArray[np.bool_],
+ axis: int,
+) -> NDArray[np.int64] | None:
+ raw = _slot_array(slots.get(name, ()), dtype=np.int64).reshape(-1)
+ if raw.size:
+ if np.any(raw <= 0):
+ raise MatrixSourceError(f"{name} must contain positive R indexes")
+ return raw - 1
+ if zero_dims.size > axis and bool(zero_dims[axis]):
+ return np.empty(0, dtype=np.int64)
+ return None
+
+
+def _r_delayed_selection(
+ value: Any,
+ *,
+ slot_name: str,
+ object_path: str,
+) -> NDArray[np.int64] | None:
+ if value is None:
+ return None
+ indexes = _slot_array(value)
+ if indexes.ndim != 1 or not np.issubdtype(indexes.dtype, np.integer):
+ raise MatrixSourceError(
+ f"{slot_name} at {object_path} must be a one-dimensional integer vector"
+ )
+ indexes = indexes.astype(np.int64, copy=False)
+ if indexes.size and np.any(indexes <= 0):
+ raise MatrixSourceError(
+ f"{slot_name} at {object_path} contains a missing or nonpositive R index"
+ )
+ return indexes - 1
+
+
+def _r_scalar_integer(value: Any, *, slot_name: str, object_path: str) -> int:
+ values = _slot_array(value)
+ if (
+ values.size != 1
+ or not np.issubdtype(values.dtype, np.integer)
+ or isinstance(values.reshape(-1)[0], np.bool_)
+ ):
+ raise MatrixSourceError(
+ f"{slot_name} at {object_path} must contain one integer"
+ )
+ return int(values.reshape(-1)[0])
+
+
+def _inferred_operation_spec(
+ primary_class: str,
+ classes: tuple[str, ...],
+ slots: Mapping[str, Any],
+ *,
+ object_path: str,
+ resolve_source: Callable[[Any, str], MatrixSource],
+ resolve_fragment: Callable[[Any, str], FragmentSource],
+) -> dict[str, Any] | None:
+ base: dict[str, Any] = {"className": classes}
+ if primary_class in {"DelayedArray", "DelayedMatrix"} | _HDF5_WRAPPER_CLASSES:
+ return {
+ **base,
+ "operation": "rename",
+ "source": resolve_source(
+ _first_value(slots, "seed"),
+ f"{object_path}@seed",
+ ),
+ }
+ if primary_class == "DelayedSubset":
+ source = resolve_source(
+ _first_value(slots, "seed"),
+ f"{object_path}@seed",
+ )
+ index = _first_value(slots, "index")
+ if not isinstance(index, Sequence) or isinstance(
+ index, (str, bytes, bytearray)
+ ):
+ raise TypeError(f"index at {object_path} must be a sequence")
+ if len(index) != 2:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "subset",
+ primary_class,
+ "only two-dimensional DelayedSubset nodes are supported",
+ )
+ return {
+ **base,
+ "operation": "subset",
+ "source": source,
+ "featureIndices": _r_delayed_selection(
+ index[0],
+ slot_name="index[[1]]",
+ object_path=object_path,
+ ),
+ "cellIndices": _r_delayed_selection(
+ index[1],
+ slot_name="index[[2]]",
+ object_path=object_path,
+ ),
+ }
+ if primary_class in {"DelayedAperm", "SeedDimPicker"}:
+ permutation = _slot_array(_first_value(slots, "perm", "dim_combination"))
+ if (
+ permutation.ndim != 1
+ or permutation.size != 2
+ or not np.issubdtype(permutation.dtype, np.integer)
+ or np.any(permutation <= 0)
+ ):
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "aperm",
+ primary_class,
+ "only two-dimensional permutations without missing axes are supported",
+ )
+ return {
+ **base,
+ "operation": "aperm",
+ "source": resolve_source(
+ _first_value(slots, "seed"),
+ f"{object_path}@seed",
+ ),
+ "permutation": tuple(int(value) for value in permutation),
+ }
+ if primary_class in {"DelayedAbind", "SeedBinder"}:
+ values = _first_value(slots, "seeds")
+ if not isinstance(values, Sequence) or isinstance(
+ values, (str, bytes, bytearray)
+ ):
+ raise TypeError(f"seeds at {object_path} must be a sequence")
+ sources = [
+ resolve_source(value, f"{object_path}@seeds[{index}]")
+ for index, value in enumerate(values)
+ ]
+ along = _r_scalar_integer(
+ _first_value(slots, "along"),
+ slot_name="along",
+ object_path=object_path,
+ )
+ if along not in {1, 2}:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "abind",
+ primary_class,
+ "only two-dimensional row or column binding is supported",
+ )
+ return {
+ **base,
+ "operation": "feature_bind" if along == 1 else "cell_bind",
+ "sources": sources,
+ }
+ if primary_class == "DelayedSetDimnames":
+ return {
+ **base,
+ "operation": "rename",
+ "source": resolve_source(
+ _first_value(slots, "seed"),
+ f"{object_path}@seed",
+ ),
+ }
+ if primary_class == "DelayedSubassign":
+ index = _first_value(slots, "Lindex")
+ if not isinstance(index, Sequence) or isinstance(
+ index, (str, bytes, bytearray)
+ ):
+ raise TypeError(f"Lindex at {object_path} must be a sequence")
+ if len(index) != 2:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "subassignment",
+ primary_class,
+ "only two-dimensional DelayedSubassign nodes are supported",
+ )
+ source = resolve_source(
+ _first_value(slots, "seed"),
+ f"{object_path}@seed",
+ )
+ feature_indices = _r_delayed_selection(
+ index[0],
+ slot_name="Lindex[[1]]",
+ object_path=object_path,
+ )
+ cell_indices = _r_delayed_selection(
+ index[1],
+ slot_name="Lindex[[2]]",
+ object_path=object_path,
+ )
+ if feature_indices is None:
+ feature_indices = np.arange(source.shape[0], dtype=np.int64)
+ if cell_indices is None:
+ cell_indices = np.arange(source.shape[1], dtype=np.int64)
+ replacement = _first_value(slots, "Rvalue")
+ if not isinstance(replacement, MatrixSource):
+ replacement_values = _slot_array(replacement)
+ if replacement_values.size != 1:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "subassignment",
+ primary_class,
+ "ordinary replacement values must contain one numeric scalar",
+ )
+ replacement = replacement_values.reshape(-1)[0].item()
+ return {
+ **base,
+ "operation": "subassignment",
+ "source": source,
+ "assignments": (
+ {
+ "featureIndices": feature_indices,
+ "cellIndices": cell_indices,
+ "value": replacement,
+ },
+ ),
+ }
+ if primary_class == "MatrixSubset":
+ source = resolve_source(
+ _first_value(slots, "matrix", "source", "seed"),
+ f"{object_path}@matrix",
+ )
+ zero_dims = _slot_array(
+ slots.get("zero_dims", slots.get("zeroDims", (False, False))),
+ dtype=bool,
+ ).reshape(-1)
+ return {
+ **base,
+ "operation": "subset",
+ "source": source,
+ "featureIndices": _r_selection(slots, "row_selection", zero_dims, 0),
+ "cellIndices": _r_selection(slots, "col_selection", zero_dims, 1),
+ }
+ if primary_class in {"RowBindMatrices", "ColBindMatrices"}:
+ values = _first_value(slots, "matrix_list", "sources", "matrices")
+ if not isinstance(values, Sequence) or isinstance(
+ values, (str, bytes, bytearray)
+ ):
+ raise TypeError(f"matrix list at {object_path} must be a sequence")
+ sources = [
+ resolve_source(value, f"{object_path}@matrix_list[{index}]")
+ for index, value in enumerate(values)
+ ]
+ return {
+ **base,
+ "operation": (
+ "feature_bind" if primary_class == "RowBindMatrices" else "cell_bind"
+ ),
+ "sources": sources,
+ }
+ if primary_class == "RenameDims":
+ return {
+ **base,
+ "operation": "rename",
+ "source": resolve_source(
+ _first_value(slots, "matrix", "source", "seed"),
+ f"{object_path}@matrix",
+ ),
+ }
+ if primary_class == "ConvertMatrixType":
+ dtype_value = _first_value(slots, "type", "matrix_type", "matrixType", "dtype")
+ dtype_aliases = {
+ "uint32_t": np.dtype(np.uint32),
+ "float": np.dtype(np.float32),
+ "double": np.dtype(np.float64),
+ }
+ dtype = dtype_aliases.get(str(dtype_value))
+ if dtype is None:
+ dtype = np.dtype(dtype_value)
+ return {
+ **base,
+ "operation": "dtype",
+ "source": resolve_source(
+ _first_value(slots, "matrix", "source", "seed"),
+ f"{object_path}@matrix",
+ ),
+ "dtype": dtype,
+ }
+ if primary_class in _UNARY_TRANSFORM_CLASSES:
+ parameter: int | None = None
+ if primary_class == "TransformRound":
+ parameters = _slot_array(
+ slots.get("global_params", slots.get("globalParams", (0,)))
+ ).reshape(-1)
+ if (
+ parameters.size != 1
+ or not np.isfinite(parameters[0])
+ or parameters[0] != np.floor(parameters[0])
+ ):
+ raise MatrixSourceError(
+ f"{primary_class} at {object_path} requires one integer digit"
+ )
+ parameter = int(parameters[0])
+ result = {
+ **base,
+ "operation": "unary",
+ "source": resolve_source(
+ _first_value(slots, "matrix", "source", "seed"),
+ f"{object_path}@matrix",
+ ),
+ "function": _UNARY_TRANSFORM_CLASSES[primary_class],
+ }
+ if parameter is not None:
+ result["parameter"] = parameter
+ return result
+ if primary_class in {"TransformPow", "TransformMin"}:
+ parameters = _slot_array(
+ _first_value(slots, "global_params", "globalParams")
+ ).reshape(-1)
+ if parameters.size != 1:
+ raise MatrixSourceError(
+ f"{primary_class} at {object_path} requires one parameter"
+ )
+ return {
+ **base,
+ "operation": "binary",
+ "source": resolve_source(
+ _first_value(slots, "matrix", "source", "seed"),
+ f"{object_path}@matrix",
+ ),
+ "right": parameters[0].item(),
+ "function": ("power" if primary_class == "TransformPow" else "minimum"),
+ }
+ if primary_class == "TransformBinarize":
+ parameters = _slot_array(
+ _first_value(slots, "global_params", "globalParams")
+ ).reshape(-1)
+ if parameters.size != 2:
+ raise MatrixSourceError(
+ f"{primary_class} at {object_path} requires two parameters"
+ )
+ return {
+ **base,
+ "operation": "binary",
+ "source": resolve_source(
+ _first_value(slots, "matrix", "source", "seed"),
+ f"{object_path}@matrix",
+ ),
+ "right": parameters[0].item(),
+ "function": "greater" if bool(parameters[1]) else "greater_equal",
+ }
+ if primary_class == "MatrixAddition":
+ return {
+ **base,
+ "operation": "binary",
+ "source": resolve_source(
+ _first_value(slots, "left"), f"{object_path}@left"
+ ),
+ "right": resolve_source(
+ _first_value(slots, "right"), f"{object_path}@right"
+ ),
+ "function": "add",
+ }
+ if primary_class == "MatrixMask":
+ return {
+ **base,
+ "operation": "mask",
+ "source": resolve_source(
+ _first_value(slots, "matrix", "source"),
+ f"{object_path}@matrix",
+ ),
+ "mask": resolve_source(_first_value(slots, "mask"), f"{object_path}@mask"),
+ "invert": bool(slots.get("invert", False)),
+ }
+ if primary_class == "MatrixRankTransform":
+ return {
+ **base,
+ "operation": "rank",
+ "source": resolve_source(
+ _first_value(slots, "matrix", "source"),
+ f"{object_path}@matrix",
+ ),
+ "axis": "row" if bool(slots.get("transpose", False)) else "column",
+ }
+ if primary_class == "MatrixMultiply":
+ result = {
+ **base,
+ "operation": "multiply",
+ "source": resolve_source(
+ _first_value(slots, "left"), f"{object_path}@left"
+ ),
+ "right": resolve_source(
+ _first_value(slots, "right"), f"{object_path}@right"
+ ),
+ }
+ if "dim" in slots:
+ result["dim"] = slots["dim"]
+ elif "Dim" in slots:
+ result["Dim"] = slots["Dim"]
+ return result
+ if primary_class in {"PeakMatrix", "TileMatrix"}:
+ result = {
+ **base,
+ "operation": "fragment-derived",
+ "matrixType": primary_class,
+ "fragments": resolve_fragment(
+ _first_value(slots, "fragments"),
+ f"{object_path}@fragments",
+ ),
+ "chrId": _first_value(slots, "chr_id", "chrId"),
+ "start": _first_value(slots, "start"),
+ "end": _first_value(slots, "end"),
+ "chrLevels": _first_value(slots, "chr_levels", "chrLevels"),
+ "mode": _first_value(slots, "mode"),
+ "transpose": _optional_value(slots, "transpose", default=True),
+ "shape": _first_value(slots, "dim", "Dim", "shape"),
+ }
+ if primary_class == "TileMatrix":
+ result["tileWidths"] = _first_value(
+ slots,
+ "tile_width",
+ "tileWidths",
+ )
+ return result
+ return None
+
+
+def matrix_source_from_slots(
+ specification: Mapping[str, Any],
+ *,
+ object_path: str = "$",
+ class_name: str | None = None,
+ rds_path: str | os.PathLike[str] | None = None,
+ absolute_prefix_remaps: Mapping[str | os.PathLike[str], str | os.PathLike[str]]
+ | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+) -> MatrixSource:
+ if not isinstance(specification, Mapping):
+ raise TypeError("matrix source specification must be a mapping")
+ nested = specification.get("slots")
+ if nested is None:
+ slots = specification
+ elif isinstance(nested, Mapping):
+ slots = nested
+ else:
+ raise TypeError(f"matrix slots at {object_path} must be a mapping")
+ classes = _class_names(
+ class_name
+ if class_name is not None
+ else specification.get(
+ "className",
+ specification.get(
+ "class",
+ slots.get("className", slots.get("class")),
+ ),
+ )
+ )
+ primary_class = classes[0] if classes else None
+
+ def resolve_source(value: Any, path: str) -> MatrixSource:
+ if isinstance(value, MatrixSource):
+ return value
+ if isinstance(value, Mapping):
+ return matrix_source_from_slots(
+ value,
+ object_path=path,
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ limits=limits,
+ )
+ raise TypeError(f"matrix input at {path} must be a source or mapping")
+
+ def resolve_fragment(value: Any, path: str) -> FragmentSource:
+ if isinstance(value, FragmentSource):
+ return value
+ if isinstance(value, Mapping):
+ return fragment_source_from_slots(
+ value,
+ object_path=path,
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ limits=limits,
+ )
+ raise TypeError(f"fragment input at {path} must be a source or mapping")
+
+ operation = specification.get("operation", specification.get("op"))
+ if operation is not None:
+ operation_spec = dict(specification)
+ operation_spec.update(slots)
+ for key in ("source", "matrix", "seed", "left", "right", "mask"):
+ value = operation_spec.get(key)
+ if isinstance(value, Mapping):
+ operation_spec[key] = resolve_source(value, f"{object_path}@{key}")
+ fragment_value = operation_spec.get("fragments")
+ if isinstance(fragment_value, Mapping):
+ operation_spec["fragments"] = resolve_fragment(
+ fragment_value,
+ f"{object_path}@fragments",
+ )
+ for key in ("sources", "matrices", "matrix_list"):
+ values = operation_spec.get(key)
+ if isinstance(values, Sequence) and not isinstance(
+ values, (str, bytes, bytearray)
+ ):
+ operation_spec[key] = [
+ resolve_source(value, f"{object_path}@{key}[{index}]")
+ for index, value in enumerate(values)
+ ]
+ if primary_class is not None:
+ operation_spec["className"] = classes
+ return build_matrix_operation(
+ operation_spec,
+ object_path=object_path,
+ limits=limits,
+ )
+ if primary_class is None:
+ raise UnsupportedMatrixOperation(
+ object_path, "leaf", None, "matrix class is missing"
+ )
+ if FRAGMENT_CAPABILITY_REGISTRY.recognizes(primary_class):
+ FRAGMENT_CAPABILITY_REGISTRY.resolve(classes, object_path=object_path)
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "leaf",
+ primary_class,
+ "a fragment source cannot be used as a matrix",
+ )
+ row_names, column_names = _dimnames(slots)
+ if primary_class in {
+ "DelayedUnaryIsoOpStack",
+ "DelayedUnaryIsoOpWithArgs",
+ "DelayedNaryIsoOp",
+ }:
+ operation_value = _first_value(slots, "OP", "op", "function", "OPS")
+ if primary_class == "DelayedUnaryIsoOpStack":
+ if not isinstance(operation_value, Sequence) or isinstance(
+ operation_value,
+ str | bytes | bytearray,
+ ):
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "unary",
+ primary_class,
+ "OPS must be a sequence of recognized primitives",
+ )
+ operations = tuple(operation_value)
+ source = resolve_source(
+ _first_value(slots, "seed"),
+ f"{object_path}@seed",
+ )
+ transformed = source
+ for index, operation_name in enumerate(operations):
+ transformed = build_matrix_operation(
+ {
+ "operation": "unary",
+ "className": classes,
+ "source": transformed,
+ "function": operation_name,
+ },
+ object_path=f"{object_path}@OPS[{index}]",
+ limits=limits,
+ )
+ elif primary_class == "DelayedNaryIsoOp":
+ if not isinstance(operation_value, str):
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "binary",
+ primary_class,
+ "OP must be a recognized primitive",
+ )
+ values = _first_value(slots, "seeds")
+ if not isinstance(values, Sequence) or isinstance(
+ values,
+ str | bytes | bytearray,
+ ):
+ raise TypeError(f"seeds at {object_path} must be a sequence")
+ sources = [
+ resolve_source(value, f"{object_path}@seeds[{index}]")
+ for index, value in enumerate(values)
+ ]
+ if not sources:
+ raise MatrixSourceError(
+ f"DelayedNaryIsoOp at {object_path} has no seeds"
+ )
+ transformed = sources[0]
+ for index, right in enumerate(sources[1:], start=1):
+ transformed = build_matrix_operation(
+ {
+ "operation": "binary",
+ "className": classes,
+ "source": transformed,
+ "right": right,
+ "function": operation_value,
+ },
+ object_path=f"{object_path}@seeds[{index}]",
+ limits=limits,
+ )
+ for index, scalar_right in enumerate(
+ _operation_arguments(
+ slots.get("Rargs"),
+ object_path=f"{object_path}@Rargs",
+ )
+ ):
+ transformed = build_matrix_operation(
+ {
+ "operation": "binary",
+ "className": classes,
+ "source": transformed,
+ "right": scalar_right,
+ "function": operation_value,
+ },
+ object_path=f"{object_path}@Rargs[{index}]",
+ limits=limits,
+ )
+ else:
+ if not isinstance(operation_value, str):
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "unary",
+ primary_class,
+ "OP must be a recognized primitive",
+ )
+ source = resolve_source(
+ _first_value(slots, "seed"),
+ f"{object_path}@seed",
+ )
+ left_arguments = _operation_arguments(
+ slots.get("Largs"),
+ object_path=f"{object_path}@Largs",
+ )
+ right_arguments = _operation_arguments(
+ slots.get("Rargs"),
+ object_path=f"{object_path}@Rargs",
+ )
+ single_right = right_arguments[0] if len(right_arguments) == 1 else None
+ if len(left_arguments) + len(right_arguments) == 0:
+ transformed = build_matrix_operation(
+ {
+ "operation": "unary",
+ "className": classes,
+ "source": source,
+ "function": operation_value,
+ },
+ object_path=object_path,
+ limits=limits,
+ )
+ elif (
+ operation_value == "round"
+ and not left_arguments
+ and isinstance(single_right, int | float)
+ and float(single_right).is_integer()
+ ):
+ transformed = build_matrix_operation(
+ {
+ "operation": "unary",
+ "className": classes,
+ "source": source,
+ "function": "round",
+ "parameter": int(single_right),
+ },
+ object_path=object_path,
+ limits=limits,
+ )
+ elif (
+ operation_value == "log"
+ and not left_arguments
+ and isinstance(single_right, int | float)
+ and float(single_right) > 0
+ and float(single_right) != 1
+ ):
+ logged = build_matrix_operation(
+ {
+ "operation": "unary",
+ "className": classes,
+ "source": source,
+ "function": "log",
+ },
+ object_path=object_path,
+ limits=limits,
+ )
+ transformed = build_matrix_operation(
+ {
+ "operation": "binary",
+ "className": classes,
+ "source": logged,
+ "right": float(np.log(float(single_right))),
+ "function": "divide",
+ },
+ object_path=object_path,
+ limits=limits,
+ )
+ elif len(left_arguments) == 1 and not right_arguments:
+ transformed = build_matrix_operation(
+ {
+ "operation": "binary",
+ "className": classes,
+ "source": source,
+ "right": left_arguments[0],
+ "function": operation_value,
+ "reverse": True,
+ },
+ object_path=object_path,
+ limits=limits,
+ )
+ elif not left_arguments and len(right_arguments) == 1:
+ transformed = build_matrix_operation(
+ {
+ "operation": "binary",
+ "className": classes,
+ "source": source,
+ "right": right_arguments[0],
+ "function": operation_value,
+ },
+ object_path=object_path,
+ limits=limits,
+ )
+ else:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ operation_value,
+ primary_class,
+ "only one scalar left or right argument is supported",
+ )
+ return _finalize_slot_source(
+ transformed,
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in {"TransformMinByRow", "TransformMinByCol"}:
+ source = resolve_source(
+ _first_value(slots, "matrix", "source", "seed"),
+ f"{object_path}@matrix",
+ )
+ transposed = _slot_boolean(
+ _optional_value(slots, "transpose", default=False),
+ slot_name="transpose",
+ object_path=object_path,
+ )
+ parameter_name = (
+ "row_params" if primary_class == "TransformMinByRow" else "col_params"
+ )
+ parameters = _parameter_matrix(
+ slots,
+ parameter_name,
+ object_path=object_path,
+ )
+ axis = (
+ "cell"
+ if (primary_class == "TransformMinByRow") == transposed
+ else "feature"
+ )
+ return _finalize_slot_source(
+ AxisMinimumMatrixSource(
+ source,
+ parameters.reshape(-1),
+ axis=axis,
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class == "TransformScaleShift":
+ source = resolve_source(
+ _first_value(slots, "matrix", "source", "seed"),
+ f"{object_path}@matrix",
+ )
+ transposed = _slot_boolean(
+ _optional_value(slots, "transpose", default=False),
+ slot_name="transpose",
+ object_path=object_path,
+ )
+ active = _slot_array(
+ _first_value(slots, "active_transforms"),
+ dtype=bool,
+ )
+ if active.ndim == 1 and active.size == 6:
+ active = active.reshape((3, 2), order="F")
+ if active.shape != (3, 2):
+ raise MatrixSourceError(
+ f"active_transforms at {object_path} must have shape (3, 2)"
+ )
+ row_parameters = _parameter_matrix(
+ slots,
+ "row_params",
+ object_path=object_path,
+ )
+ column_parameters = _parameter_matrix(
+ slots,
+ "col_params",
+ object_path=object_path,
+ )
+ global_parameters = _slot_array(
+ slots.get("global_params", ()),
+ dtype=np.float64,
+ ).reshape(-1)
+
+ def active_parameters(
+ values: NDArray[Any],
+ parameter_row: int,
+ active_row: int,
+ ) -> NDArray[Any] | None:
+ if not active[active_row, parameter_row]:
+ return None
+ if values.ndim != 2 or values.shape[0] <= parameter_row:
+ raise MatrixSourceError(
+ f"TransformScaleShift parameters at {object_path} are incomplete"
+ )
+ return np.asarray(values[parameter_row])
+
+ row_scale = active_parameters(row_parameters, 0, 0)
+ column_scale = active_parameters(column_parameters, 0, 1)
+ row_shift = active_parameters(row_parameters, 1, 0)
+ column_shift = active_parameters(column_parameters, 1, 1)
+ if np.any(active[2]) and global_parameters.size < 2:
+ raise MatrixSourceError(
+ f"global_params at {object_path} must contain scale and shift"
+ )
+ feature_scale, cell_scale = (
+ (column_scale, row_scale) if transposed else (row_scale, column_scale)
+ )
+ feature_shift, cell_shift = (
+ (column_shift, row_shift) if transposed else (row_shift, column_shift)
+ )
+ return _finalize_slot_source(
+ ScaleShiftMatrixSource(
+ source,
+ feature_scale=feature_scale,
+ cell_scale=cell_scale,
+ global_scale=(float(global_parameters[0]) if active[2, 0] else 1.0),
+ feature_shift=feature_shift,
+ cell_shift=cell_shift,
+ global_shift=(float(global_parameters[1]) if active[2, 1] else 0.0),
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in {
+ "SCTransformPearson",
+ "SCTransformPearsonSlow",
+ "SCTransformPearsonTranspose",
+ "SCTransformPearsonTransposeSlow",
+ }:
+ source = resolve_source(
+ _first_value(slots, "matrix", "source", "seed"),
+ f"{object_path}@matrix",
+ )
+ row_parameters = _parameter_matrix(
+ slots,
+ "row_params",
+ object_path=object_path,
+ )
+ column_parameters = _parameter_matrix(
+ slots,
+ "col_params",
+ object_path=object_path,
+ )
+ transposed_kernel = "Transpose" in primary_class
+ feature_parameters, cell_parameters = (
+ (column_parameters, row_parameters)
+ if transposed_kernel
+ else (row_parameters, column_parameters)
+ )
+ if feature_parameters.shape[0] != 2 or cell_parameters.shape[0] != 1:
+ raise MatrixSourceError(
+ f"{primary_class} parameters at {object_path} have invalid shapes"
+ )
+ return _finalize_slot_source(
+ PearsonResidualMatrixSource(
+ source,
+ theta_inverse=feature_parameters[0],
+ gene_beta=feature_parameters[1],
+ cell_read_counts=cell_parameters[0],
+ global_parameters=_slot_array(
+ _first_value(slots, "global_params"),
+ ),
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class == "TransformLinearResidual":
+ source = resolve_source(
+ _first_value(slots, "matrix", "source", "seed"),
+ f"{object_path}@matrix",
+ )
+ row_parameters = _parameter_matrix(
+ slots,
+ "row_params",
+ object_path=object_path,
+ )
+ column_parameters = _parameter_matrix(
+ slots,
+ "col_params",
+ object_path=object_path,
+ )
+ transposed = _slot_boolean(
+ _optional_value(slots, "transpose", default=False),
+ slot_name="transpose",
+ object_path=object_path,
+ )
+ if row_parameters.size == 0 or column_parameters.size == 0:
+ residual_source: MatrixSource = source
+ else:
+ feature_parameters, cell_parameters = (
+ (column_parameters, row_parameters)
+ if transposed
+ else (row_parameters, column_parameters)
+ )
+ residual_source = LinearResidualMatrixSource(
+ source,
+ feature_parameters=feature_parameters,
+ cell_parameters=cell_parameters,
+ limits=limits,
+ )
+ return _finalize_slot_source(
+ residual_source,
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ inferred = _inferred_operation_spec(
+ primary_class,
+ classes,
+ slots,
+ object_path=object_path,
+ resolve_source=resolve_source,
+ resolve_fragment=resolve_fragment,
+ )
+ if inferred is not None:
+ return _finalize_slot_source(
+ build_matrix_operation(
+ inferred,
+ object_path=object_path,
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in _DENSE_CLASSES:
+ values = _first_value(slots, ".Data", "data", "values", "x")
+ shape = _first_value(slots, "dim", "shape")
+ return _finalize_slot_source(
+ DenseMatrixSource(
+ values,
+ shape,
+ dtype=_optional_value(slots, "dtype"),
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in _CSC_CLASSES:
+ return _finalize_slot_source(
+ CscMatrixSource(
+ _optional_value(slots, "x"),
+ _first_value(slots, "i"),
+ _first_value(slots, "p"),
+ _first_value(slots, "Dim", "dim", "shape"),
+ dtype=_optional_value(slots, "dtype"),
+ class_name=primary_class,
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class == "Iterable_dgCMatrix_wrapper":
+ wrapped = _first_value(slots, "mat")
+ if not isinstance(wrapped, MatrixSource):
+ wrapped = resolve_source(wrapped, f"{object_path}@mat")
+ return _finalize_slot_source(
+ wrapped,
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in _BPCELLS_MEMORY_CLASSES:
+ compression, datatype = _BPCELLS_MEMORY_CLASSES[primary_class]
+ version = _slot_text(
+ _first_value(slots, "version"),
+ slot_name="version",
+ object_path=object_path,
+ )
+ expected_prefix = f"{compression}-{datatype}-matrix-v"
+ if not version.startswith(expected_prefix):
+ raise MatrixSourceError(
+ f"BPCells class {primary_class!r} conflicts with format {version!r}"
+ )
+ shape = _slot_shape(
+ _first_value(slots, "dim", "shape"),
+ object_path=object_path,
+ )
+ transpose = _slot_array(
+ _optional_value(slots, "transpose", default=False)
+ ).reshape(-1)
+ if transpose.size != 1:
+ raise MatrixSourceError("transpose slot must be scalar")
+ array_names = {
+ "idxptr",
+ "index",
+ "index_data",
+ "index_starts",
+ "index_idx",
+ "index_idx_offsets",
+ "val",
+ "val_data",
+ "val_idx",
+ "val_idx_offsets",
+ }
+ arrays = {
+ name: slots[name]
+ for name in array_names
+ if name in slots and slots[name] is not None
+ }
+ return _finalize_slot_source(
+ BPCellsMemoryMatrixSource(
+ version,
+ arrays,
+ shape=shape,
+ storage_order="row" if bool(transpose[0]) else "col",
+ row_names=row_names,
+ column_names=column_names,
+ float_bit_arrays=(
+ frozenset({"val"}) if datatype == "float" else frozenset()
+ ),
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in _BPCELLS_DIRECTORY_CLASSES:
+ path = _sidecar_path(
+ _first_value(slots, "dir", "directory", "path"),
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ expect="directory",
+ )
+ return _finalize_slot_source(
+ BPCellsDirectoryMatrixSource(path, limits=limits),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in _BPCELLS_HDF5_CLASSES:
+ path = _sidecar_path(
+ _first_value(slots, "filepath", "path"),
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ expect="file",
+ )
+ return _finalize_slot_source(
+ BPCellsHDF5MatrixSource(
+ path,
+ group=str(_first_value(slots, "group")),
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in _HDF5_RESHAPED_CLASSES:
+ path = _sidecar_path(
+ _first_value(slots, "filepath", "path"),
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ expect="file",
+ )
+ reshaped = _slot_shape(
+ _first_value(slots, "reshaped_dim", "reshapedDim"),
+ object_path=object_path,
+ )
+ return _finalize_slot_source(
+ ReshapedHDF5ArrayMatrixSource(
+ path,
+ str(_first_value(slots, "name", "dataset")),
+ reshaped,
+ dtype=_optional_value(slots, "dtype"),
+ as_sparse=bool(
+ _optional_value(slots, "asSparse", "as_sparse", default=False)
+ ),
+ limits=limits,
+ ),
+ {**slots, "dim": reshaped},
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in _HDF5_DENSE_CLASSES:
+ path = _sidecar_path(
+ _first_value(slots, "filepath", "path"),
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ expect="file",
+ )
+ return _finalize_slot_source(
+ HDF5ArrayMatrixSource(
+ path,
+ str(_first_value(slots, "name", "dataset")),
+ dtype=_optional_value(slots, "dtype"),
+ as_sparse=bool(
+ _optional_value(slots, "asSparse", "as_sparse", default=False)
+ ),
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in _H5_SPARSE_CLASSES:
+ path = _sidecar_path(
+ _first_value(slots, "filepath", "path"),
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ expect="file",
+ )
+ sparse_layout = _optional_value(slots, "sparseLayout", "sparse_layout")
+ if sparse_layout is None and primary_class.startswith(("CSC_", "CSR_")):
+ sparse_layout = primary_class[:3]
+ return _finalize_slot_source(
+ H5SparseMatrixSource(
+ path,
+ str(_first_value(slots, "group")),
+ shape=_optional_value(slots, "dim", "shape"),
+ sparse_layout=sparse_layout,
+ dtype=_optional_value(slots, "dtype"),
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in _H5AD_CLASSES:
+ path = _sidecar_path(
+ _first_value(slots, "filepath", "path"),
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ expect="file",
+ )
+ return _finalize_slot_source(
+ H5ADMatrixSource(
+ path,
+ layer=(
+ _optional_value(slots, "layer")
+ if primary_class != "AnnDataMatrixH5"
+ else None
+ ),
+ matrix_path=(
+ _optional_value(slots, "group", default="X")
+ if primary_class == "AnnDataMatrixH5"
+ else None
+ ),
+ dtype=_optional_value(slots, "dtype"),
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ if primary_class in _TENX_CLASSES:
+ path = _sidecar_path(
+ _first_value(slots, "filepath", "path"),
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ expect="file",
+ )
+ return _finalize_slot_source(
+ TENxMatrixSource(
+ path,
+ group=str(_optional_value(slots, "group", default="matrix")),
+ limits=limits,
+ ),
+ slots,
+ row_names,
+ column_names,
+ limits,
+ )
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "leaf",
+ primary_class,
+ "unknown or custom matrix class",
+ )
+
+
+source_from_slot_mapping = matrix_source_from_slots
+matrix_source_from_mapping = matrix_source_from_slots
diff --git a/scarf/readers/_seurat/fragments.py b/scarf/readers/_seurat/fragments.py
new file mode 100644
index 00000000..9620779a
--- /dev/null
+++ b/scarf/readers/_seurat/fragments.py
@@ -0,0 +1,2189 @@
+import math
+import os
+import re
+from collections.abc import Iterator, Mapping, Sequence
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Protocol, cast, runtime_checkable
+
+import numpy as np
+from numpy.typing import NDArray
+from scipy.sparse import csr_matrix
+
+from .bpcells import (
+ _BPArrayStore,
+ _DirectoryArrayStore,
+ _HDF5ArrayStore,
+ _StoredBP128Array,
+ _require_numeric_array,
+)
+from .errors import (
+ MatrixSourceError,
+ ResourceLimitError,
+ UnsafeSidecarError,
+ UnsupportedMatrixOperation,
+)
+from .paths import SidecarPathResolver, require_filesystem_path
+from .sources import (
+ DEFAULT_LIMITS,
+ BaseMatrixSource,
+ MemoryEstimate,
+ SourceLimits,
+ _normalize_names,
+)
+
+
+_FRAGMENT_VERSION_PATTERN = re.compile(r"^(packed|unpacked)-fragments-v([12])$")
+_UINT32_MAX = int(np.iinfo(np.uint32).max)
+_DICT_ENTRY_BYTES = 160
+_FRAGMENT_BLOCK_SAFETY_FACTOR = 4
+
+
+@dataclass(frozen=True)
+class FragmentCapability:
+ className: str
+ accepted: bool
+ reason: str | None = None
+
+
+_FRAGMENT_CAPABILITIES = (
+ FragmentCapability("UnpackedMemFragments", True),
+ FragmentCapability("PackedMemFragments", True),
+ FragmentCapability("FragmentsDir", True),
+ FragmentCapability("FragmentsHDF5", True),
+ FragmentCapability(
+ "FragmentsTsv", False, "fragment TSV execution is not implemented"
+ ),
+ FragmentCapability("ShiftFragments", True),
+ FragmentCapability("SelectLength", True),
+ FragmentCapability("ChrSelectName", True),
+ FragmentCapability("ChrSelectIndex", True),
+ FragmentCapability("CellSelectName", True),
+ FragmentCapability("CellSelectIndex", True),
+ FragmentCapability("CellMerge", True),
+ FragmentCapability("ChrRename", True),
+ FragmentCapability("CellRename", True),
+ FragmentCapability("CellPrefix", True),
+ FragmentCapability("RegionSelect", True),
+ FragmentCapability("MergeFragments", True),
+ FragmentCapability(
+ "IterableFragments", False, "abstract fragment sources cannot execute"
+ ),
+)
+
+
+class FragmentCapabilityRegistry:
+ def __init__(self) -> None:
+ self.capabilities = {
+ capability.className: capability for capability in _FRAGMENT_CAPABILITIES
+ }
+
+ @property
+ def acceptedClasses(self) -> tuple[str, ...]:
+ return tuple(
+ capability.className
+ for capability in _FRAGMENT_CAPABILITIES
+ if capability.accepted
+ )
+
+ def recognizes(self, class_name: str | None) -> bool:
+ return class_name in self.capabilities
+
+ def resolve(
+ self,
+ classes: tuple[str, ...],
+ *,
+ object_path: str,
+ ) -> FragmentCapability:
+ if not classes:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "fragment-source",
+ None,
+ "fragment class is missing",
+ )
+ primary = classes[0]
+ capability = self.capabilities.get(primary)
+ if capability is None:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "fragment-source",
+ primary,
+ "unknown or custom fragment class",
+ )
+ unsupported_bases = [
+ class_name
+ for class_name in classes[1:]
+ if class_name != "IterableFragments"
+ ]
+ if unsupported_bases:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "fragment-source",
+ unsupported_bases[0],
+ "unknown or custom fragment base class",
+ )
+ if not capability.accepted:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "fragment-source",
+ primary,
+ capability.reason,
+ )
+ return capability
+
+
+FRAGMENT_CAPABILITY_REGISTRY = FragmentCapabilityRegistry()
+
+
+@dataclass(frozen=True)
+class FragmentBlock:
+ cellIds: NDArray[np.uint32]
+ starts: NDArray[np.uint32]
+ ends: NDArray[np.uint32]
+
+ @property
+ def size(self) -> int:
+ return int(self.cellIds.size)
+
+
+@runtime_checkable
+class FragmentSource(Protocol):
+ @property
+ def chromosomeNames(self) -> tuple[str, ...]: ...
+
+ @property
+ def cellNames(self) -> tuple[str, ...]: ...
+
+ @property
+ def recordCount(self) -> int: ...
+
+ @property
+ def residentBytes(self) -> int: ...
+
+ @property
+ def metadataBytes(self) -> int: ...
+
+ @property
+ def blockWorkingBytes(self) -> int: ...
+
+ def iter_chromosome(self, chromosome_id: int) -> Iterator[FragmentBlock]: ...
+
+
+class _FragmentWrapper:
+ def __init__(self, source: FragmentSource) -> None:
+ self.source = source
+
+ @property
+ def chromosomeNames(self) -> tuple[str, ...]:
+ return self.source.chromosomeNames
+
+ @property
+ def cellNames(self) -> tuple[str, ...]:
+ return self.source.cellNames
+
+ @property
+ def recordCount(self) -> int:
+ return self.source.recordCount
+
+ @property
+ def residentBytes(self) -> int:
+ return self.source.residentBytes
+
+ @property
+ def metadataBytes(self) -> int:
+ return self.source.metadataBytes
+
+ @property
+ def blockWorkingBytes(self) -> int:
+ return 2 * self.source.blockWorkingBytes
+
+
+class ShiftedFragmentSource(_FragmentWrapper):
+ def __init__(
+ self,
+ source: FragmentSource,
+ shift_start: int,
+ shift_end: int,
+ ) -> None:
+ super().__init__(source)
+ self.shiftStart = shift_start
+ self.shiftEnd = shift_end
+
+ def iter_chromosome(self, chromosome_id: int) -> Iterator[FragmentBlock]:
+ for block in self.source.iter_chromosome(chromosome_id):
+ starts = block.starts.astype(np.int64) + self.shiftStart
+ ends = block.ends.astype(np.int64) + self.shiftEnd
+ if (
+ np.any(starts < 0)
+ or np.any(ends < starts)
+ or np.any(starts > _UINT32_MAX)
+ or np.any(ends > _UINT32_MAX)
+ ):
+ raise MatrixSourceError(
+ "shifted fragment coordinates leave the valid uint32 range"
+ )
+ yield FragmentBlock(
+ block.cellIds,
+ starts.astype(np.uint32),
+ ends.astype(np.uint32),
+ )
+
+
+class LengthSelectedFragmentSource(_FragmentWrapper):
+ def __init__(
+ self,
+ source: FragmentSource,
+ minimum: int,
+ maximum: int,
+ ) -> None:
+ super().__init__(source)
+ if minimum < 0 or maximum < minimum:
+ raise MatrixSourceError("fragment length bounds are invalid")
+ self.minimum = minimum
+ self.maximum = maximum
+
+ def iter_chromosome(self, chromosome_id: int) -> Iterator[FragmentBlock]:
+ for block in self.source.iter_chromosome(chromosome_id):
+ lengths = block.ends.astype(np.uint64) - block.starts.astype(np.uint64)
+ keep = (lengths >= self.minimum) & (lengths <= self.maximum)
+ if np.any(keep):
+ yield FragmentBlock(
+ block.cellIds[keep],
+ block.starts[keep],
+ block.ends[keep],
+ )
+
+
+class ChromosomeSelectedFragmentSource(_FragmentWrapper):
+ def __init__(
+ self,
+ source: FragmentSource,
+ selection: Sequence[int],
+ names: Sequence[str],
+ ) -> None:
+ super().__init__(source)
+ self.selection = tuple(int(value) for value in selection)
+ self._chromosomeNames = tuple(names)
+
+ @property
+ def chromosomeNames(self) -> tuple[str, ...]:
+ return self._chromosomeNames
+
+ @property
+ def metadataBytes(self) -> int:
+ return self.source.metadataBytes + _metadata_bytes(self._chromosomeNames)
+
+ def iter_chromosome(self, chromosome_id: int) -> Iterator[FragmentBlock]:
+ if chromosome_id < 0 or chromosome_id >= len(self.selection):
+ raise IndexError("fragment chromosome ID is out of range")
+ yield from self.source.iter_chromosome(self.selection[chromosome_id])
+
+
+class CellMappedFragmentSource(_FragmentWrapper):
+ def __init__(
+ self,
+ source: FragmentSource,
+ mapping: NDArray[np.int64],
+ names: Sequence[str],
+ ) -> None:
+ super().__init__(source)
+ if mapping.shape != (len(source.cellNames),):
+ raise MatrixSourceError("fragment cell mapping has an invalid length")
+ self.mapping = mapping
+ self._cellNames = tuple(names)
+
+ @property
+ def cellNames(self) -> tuple[str, ...]:
+ return self._cellNames
+
+ @property
+ def residentBytes(self) -> int:
+ return self.source.residentBytes + self.mapping.nbytes
+
+ @property
+ def metadataBytes(self) -> int:
+ return (
+ self.source.metadataBytes
+ + self.mapping.nbytes
+ + _metadata_bytes(self._cellNames)
+ )
+
+ def iter_chromosome(self, chromosome_id: int) -> Iterator[FragmentBlock]:
+ for block in self.source.iter_chromosome(chromosome_id):
+ mapped = self.mapping[block.cellIds]
+ keep = mapped >= 0
+ if np.any(keep):
+ yield FragmentBlock(
+ mapped[keep].astype(np.uint32),
+ block.starts[keep],
+ block.ends[keep],
+ )
+
+
+class RenamedFragmentSource(_FragmentWrapper):
+ def __init__(
+ self,
+ source: FragmentSource,
+ *,
+ chromosome_names: Sequence[str] | None = None,
+ cell_names: Sequence[str] | None = None,
+ ) -> None:
+ super().__init__(source)
+ self._chromosomeNames = (
+ source.chromosomeNames
+ if chromosome_names is None
+ else tuple(chromosome_names)
+ )
+ self._cellNames = source.cellNames if cell_names is None else tuple(cell_names)
+ if len(self._chromosomeNames) != len(source.chromosomeNames):
+ raise MatrixSourceError(
+ "replacement chromosome names have an invalid length"
+ )
+ if len(self._cellNames) != len(source.cellNames):
+ raise MatrixSourceError("replacement cell names have an invalid length")
+
+ @property
+ def chromosomeNames(self) -> tuple[str, ...]:
+ return self._chromosomeNames
+
+ @property
+ def cellNames(self) -> tuple[str, ...]:
+ return self._cellNames
+
+ @property
+ def metadataBytes(self) -> int:
+ return (
+ self.source.metadataBytes
+ + _metadata_bytes(self._chromosomeNames)
+ + _metadata_bytes(self._cellNames)
+ )
+
+ def iter_chromosome(self, chromosome_id: int) -> Iterator[FragmentBlock]:
+ yield from self.source.iter_chromosome(chromosome_id)
+
+
+class RegionSelectedFragmentSource(_FragmentWrapper):
+ def __init__(
+ self,
+ source: FragmentSource,
+ regions: Mapping[int, tuple[NDArray[np.uint32], NDArray[np.uint32]]],
+ *,
+ invert: bool,
+ metadata_bytes: int,
+ ) -> None:
+ super().__init__(source)
+ self.regions = dict(regions)
+ self.invert = invert
+ self._regionMetadataBytes = metadata_bytes
+
+ @property
+ def residentBytes(self) -> int:
+ return self.source.residentBytes + self._regionMetadataBytes
+
+ @property
+ def metadataBytes(self) -> int:
+ return self.source.metadataBytes + self._regionMetadataBytes
+
+ def iter_chromosome(self, chromosome_id: int) -> Iterator[FragmentBlock]:
+ region = self.regions.get(chromosome_id)
+ for block in self.source.iter_chromosome(chromosome_id):
+ overlaps = np.zeros(block.size, dtype=bool)
+ if region is not None:
+ starts, ends = region
+ for region_start, region_end in zip(starts, ends, strict=True):
+ overlaps |= (block.starts <= region_end) & (
+ block.ends >= region_start
+ )
+ keep = ~overlaps if self.invert else overlaps
+ if np.any(keep):
+ yield FragmentBlock(
+ block.cellIds[keep],
+ block.starts[keep],
+ block.ends[keep],
+ )
+
+
+class MergedFragmentSource:
+ def __init__(self, sources: Sequence[FragmentSource]) -> None:
+ if not sources:
+ raise MatrixSourceError("fragment merge requires at least one source")
+ self.sources = tuple(sources)
+ chromosome_names: list[str] = []
+ for source in self.sources:
+ for name in source.chromosomeNames:
+ if name not in chromosome_names:
+ chromosome_names.append(name)
+ self._chromosomeNames = tuple(chromosome_names)
+ self._cellNames = tuple(
+ name for source in self.sources for name in source.cellNames
+ )
+ self._cellOffsets: list[int] = []
+ offset = 0
+ for source in self.sources:
+ self._cellOffsets.append(offset)
+ offset += len(source.cellNames)
+
+ @property
+ def chromosomeNames(self) -> tuple[str, ...]:
+ return self._chromosomeNames
+
+ @property
+ def cellNames(self) -> tuple[str, ...]:
+ return self._cellNames
+
+ @property
+ def recordCount(self) -> int:
+ return sum(source.recordCount for source in self.sources)
+
+ @property
+ def residentBytes(self) -> int:
+ return sum(source.residentBytes for source in self.sources)
+
+ @property
+ def metadataBytes(self) -> int:
+ return sum(source.metadataBytes for source in self.sources) + _metadata_bytes(
+ self._chromosomeNames + self._cellNames
+ )
+
+ @property
+ def blockWorkingBytes(self) -> int:
+ return 2 * max(source.blockWorkingBytes for source in self.sources)
+
+ def iter_chromosome(self, chromosome_id: int) -> Iterator[FragmentBlock]:
+ if chromosome_id < 0 or chromosome_id >= len(self.chromosomeNames):
+ raise IndexError("fragment chromosome ID is out of range")
+ name = self.chromosomeNames[chromosome_id]
+ for source, cell_offset in zip(
+ self.sources,
+ self._cellOffsets,
+ strict=True,
+ ):
+ if name not in source.chromosomeNames:
+ continue
+ source_id = source.chromosomeNames.index(name)
+ for block in source.iter_chromosome(source_id):
+ shifted_cells = block.cellIds.astype(np.uint64) + cell_offset
+ if np.any(shifted_cells > _UINT32_MAX):
+ raise MatrixSourceError("merged fragment cell IDs overflow uint32")
+ yield FragmentBlock(
+ shifted_cells.astype(np.uint32),
+ block.starts,
+ block.ends,
+ )
+
+
+def _class_names(value: Any) -> tuple[str, ...]:
+ if value is None:
+ return ()
+ if isinstance(value, str):
+ return (value,)
+ if isinstance(value, Sequence) and not isinstance(value, str | bytes):
+ result = tuple(str(item) for item in value)
+ if not result:
+ raise MatrixSourceError("fragment class vector cannot be empty")
+ return result
+ return (str(value),)
+
+
+def _slot_mapping(
+ specification: Mapping[str, Any], object_path: str
+) -> Mapping[str, Any]:
+ nested = specification.get("slots")
+ if nested is None:
+ return specification
+ if not isinstance(nested, Mapping):
+ raise TypeError(f"fragment slots at {object_path} must be a mapping")
+ return nested
+
+
+def _vector_length(value: Any, object_path: str) -> int:
+ try:
+ return len(value)
+ except TypeError:
+ shape = getattr(value, "shape", None)
+ if shape is None:
+ raise TypeError(f"vector at {object_path} has no bounded length") from None
+ normalized = tuple(int(item) for item in shape)
+ if len(normalized) != 1:
+ raise MatrixSourceError(f"vector at {object_path} must be one-dimensional")
+ return normalized[0]
+
+
+def _read_vector_slice(
+ value: Any,
+ start: int,
+ stop: int,
+ object_path: str,
+) -> NDArray[Any]:
+ if hasattr(value, "read_block"):
+ result = np.asarray(value.read_block(start, stop))
+ else:
+ try:
+ result = np.asarray(value[start:stop])
+ except (IndexError, TypeError, ValueError) as error:
+ raise MatrixSourceError(
+ f"vector at {object_path} does not support bounded slicing"
+ ) from error
+ if result.ndim != 1:
+ result = result.reshape(-1)
+ if result.size != stop - start:
+ raise MatrixSourceError(
+ f"vector at {object_path} returned {result.size} values; "
+ f"expected {stop - start}"
+ )
+ return result
+
+
+def _single_value(value: Any, object_path: str) -> Any:
+ if isinstance(value, os.PathLike):
+ return value
+ if isinstance(
+ value,
+ str
+ | bytes
+ | bool
+ | int
+ | float
+ | np.str_
+ | np.bytes_
+ | np.bool_
+ | np.integer
+ | np.floating,
+ ):
+ return value
+ length = _vector_length(value, object_path)
+ if length != 1:
+ raise MatrixSourceError(f"value at {object_path} must be scalar")
+ return _read_vector_slice(value, 0, 1, object_path)[0]
+
+
+def _decode_text(value: Any, object_path: str) -> str:
+ if isinstance(value, bytes | np.bytes_):
+ try:
+ result = bytes(value).decode("utf-8")
+ except UnicodeDecodeError as error:
+ raise MatrixSourceError(
+ f"text at {object_path} is not valid UTF-8"
+ ) from error
+ elif isinstance(value, str | np.str_):
+ result = str(value)
+ else:
+ raise TypeError(f"text at {object_path} must contain strings")
+ if "\x00" in result:
+ raise MatrixSourceError(f"text at {object_path} contains a NUL character")
+ return result
+
+
+def _text_scalar(value: Any, object_path: str) -> str:
+ return _decode_text(_single_value(value, object_path), object_path)
+
+
+def _bool_scalar(value: Any, object_path: str) -> bool:
+ scalar = _single_value(value, object_path)
+ if isinstance(scalar, bool | np.bool_):
+ return bool(scalar)
+ if isinstance(scalar, int | np.integer) and int(scalar) in {0, 1}:
+ return bool(scalar)
+ raise TypeError(f"logical value at {object_path} must be TRUE or FALSE")
+
+
+def _positive_int_scalar(value: Any, object_path: str) -> int:
+ scalar = _single_value(value, object_path)
+ if isinstance(scalar, bool | np.bool_) or not isinstance(
+ scalar, int | float | np.integer | np.floating
+ ):
+ raise TypeError(f"integer value at {object_path} must be numeric")
+ numeric = float(scalar)
+ if not math.isfinite(numeric) or not numeric.is_integer() or numeric <= 0:
+ raise MatrixSourceError(f"integer value at {object_path} must be positive")
+ return int(numeric)
+
+
+def _integer_scalar(
+ value: Any,
+ object_path: str,
+ *,
+ allow_missing: bool = False,
+) -> int | None:
+ scalar = _single_value(value, object_path)
+ if isinstance(scalar, bool | np.bool_) or not isinstance(
+ scalar, int | float | np.integer | np.floating
+ ):
+ raise TypeError(f"integer value at {object_path} must be numeric")
+ numeric = float(scalar)
+ if (
+ allow_missing
+ and isinstance(scalar, int | np.integer)
+ and int(scalar) == np.iinfo(np.int32).min
+ ):
+ return None
+ if (
+ not math.isfinite(numeric)
+ or not numeric.is_integer()
+ or numeric < np.iinfo(np.int32).min
+ or numeric > np.iinfo(np.int32).max
+ ):
+ raise MatrixSourceError(f"integer value at {object_path} is invalid")
+ return int(numeric)
+
+
+def _text_values(
+ value: Any,
+ object_path: str,
+ limits: SourceLimits,
+) -> tuple[str, ...]:
+ if isinstance(value, str | bytes | np.str_ | np.bytes_):
+ raw_values: Sequence[Any] = (value,)
+ else:
+ raw_values = value
+ length = _vector_length(raw_values, object_path)
+ if length * 8 > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"text at {object_path} exceeds maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ output: list[str] = []
+ total = 0
+ for start in range(0, length, 4096):
+ stop = min(length, start + 4096)
+ if hasattr(raw_values, "read_block"):
+ values = raw_values.read_block(start, stop)
+ else:
+ values = raw_values[start:stop]
+ for index, value_item in enumerate(values, start=start):
+ decoded = _decode_text(value_item, f"{object_path}[{index}]")
+ total += len(decoded.encode("utf-8")) + 8
+ if total > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"text at {object_path} exceeds "
+ f"maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ output.append(decoded)
+ return tuple(output)
+
+
+def _metadata_bytes(values: Sequence[str]) -> int:
+ return sum(len(value.encode("utf-8")) + 8 for value in values)
+
+
+def _resolve_sidecar(
+ value: Any,
+ *,
+ rds_path: str | os.PathLike[str] | None,
+ absolute_prefix_remaps: Mapping[str | os.PathLike[str], str | os.PathLike[str]]
+ | None,
+ expect: str,
+) -> Path:
+ if rds_path is not None:
+ return SidecarPathResolver(
+ rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ ).resolve(value, expect=expect)
+ if expect == "file":
+ return require_filesystem_path(value, "fragment HDF5 source")
+ if not isinstance(value, str | os.PathLike):
+ raise TypeError("fragment sidecar directory must be a filesystem path")
+ path = Path(value).expanduser().resolve(strict=False)
+ if not path.exists():
+ raise FileNotFoundError(path)
+ if not path.is_dir():
+ raise UnsafeSidecarError(f"fragment sidecar path {path} is not a directory")
+ return path
+
+
+def _coerce_unsigned(
+ values: NDArray[Any],
+ dtype: np.dtype[Any],
+ object_path: str,
+) -> NDArray[Any]:
+ raw = np.asarray(values)
+ if raw.dtype.kind == "i" and raw.dtype.itemsize == dtype.itemsize:
+ if dtype == np.dtype(np.uint32):
+ return raw.astype(np.int32, copy=False).view(np.uint32).copy()
+ if np.any(raw < 0):
+ raise MatrixSourceError(f"array at {object_path} contains a negative value")
+ return raw.astype(dtype, copy=False)
+ if raw.dtype.kind in "iu":
+ if raw.dtype.kind == "i" and np.any(raw < 0):
+ raise MatrixSourceError(f"array at {object_path} contains a negative value")
+ widened = raw.astype(object)
+ upper = int(np.iinfo(dtype).max)
+ if any(int(value) > upper for value in widened):
+ raise MatrixSourceError(f"array at {object_path} exceeds {dtype}")
+ return raw.astype(dtype)
+ if raw.dtype.kind == "f":
+ numeric = raw.astype(np.float64, copy=False)
+ if (
+ np.any(~np.isfinite(numeric))
+ or np.any(numeric < 0)
+ or np.any(numeric != np.floor(numeric))
+ ):
+ raise MatrixSourceError(
+ f"array at {object_path} must contain finite unsigned integers"
+ )
+ upper = min(int(np.iinfo(dtype).max), 2**53)
+ if np.any(numeric > upper):
+ raise MatrixSourceError(
+ f"array at {object_path} cannot be represented exactly as {dtype}"
+ )
+ return numeric.astype(dtype)
+ raise TypeError(f"array at {object_path} must contain numeric integers")
+
+
+class _MemoryArrayStore:
+ def __init__(
+ self,
+ slots: Mapping[str, Any],
+ version: str,
+ dtypes: Mapping[str, np.dtype[Any]],
+ limits: SourceLimits,
+ object_path: str,
+ ) -> None:
+ self.slots = slots
+ self._version = version
+ self.dtypes = dict(dtypes)
+ self._limits = limits
+ self.objectPath = object_path
+
+ @property
+ def version(self) -> str:
+ return self._version
+
+ @property
+ def residentBytes(self) -> int:
+ total = 0
+ for value in self.slots.values():
+ if isinstance(value, np.ndarray):
+ total += int(value.nbytes)
+ return total
+
+ def has(self, name: str) -> bool:
+ return name in self.slots
+
+ def numeric_info(self, name: str) -> tuple[np.dtype[Any], int]:
+ if name not in self.slots:
+ raise MatrixSourceError(f"BPCells numeric array {name!r} is missing")
+ if name not in self.dtypes:
+ raise MatrixSourceError(f"BPCells array {name!r} is not numeric")
+ value = self.slots[name]
+ length = _vector_length(value, f"{self.objectPath}@{name}")
+ return self.dtypes[name], length
+
+ def read_numeric(
+ self,
+ name: str,
+ start: int = 0,
+ stop: int | None = None,
+ ) -> NDArray[Any]:
+ dtype, length = self.numeric_info(name)
+ stop = length if stop is None else int(stop)
+ start = int(start)
+ if start < 0 or stop < start or stop > length:
+ raise IndexError(
+ f"BPCells memory array {name!r} window [{start}, {stop}) "
+ f"is outside [0, {length})"
+ )
+ values = _read_vector_slice(
+ self.slots[name],
+ start,
+ stop,
+ f"{self.objectPath}@{name}",
+ )
+ return _coerce_unsigned(values, dtype, f"{self.objectPath}@{name}")
+
+ def read_text(self, name: str) -> tuple[str, ...]:
+ if name not in self.slots:
+ raise MatrixSourceError(f"BPCells text array {name!r} is missing")
+ return _text_values(
+ self.slots[name],
+ f"{self.objectPath}@{name}",
+ self._limits,
+ )
+
+
+def _memory_dtypes(version: str) -> dict[str, np.dtype[Any]]:
+ match = _FRAGMENT_VERSION_PATTERN.fullmatch(version)
+ if match is None:
+ raise MatrixSourceError(f"unsupported BPCells fragment format {version!r}")
+ compression, version_text = match.groups()
+ pointer_dtype = np.dtype(np.uint32 if version_text == "1" else np.uint64)
+ result = {
+ "chr_ptr": pointer_dtype,
+ "end_max": np.dtype(np.uint32),
+ }
+ if compression == "unpacked":
+ result.update(
+ {
+ "cell": np.dtype(np.uint32),
+ "start": np.dtype(np.uint32),
+ "end": np.dtype(np.uint32),
+ }
+ )
+ else:
+ for prefix in ("cell", "start", "end"):
+ result[f"{prefix}_data"] = np.dtype(np.uint32)
+ result[f"{prefix}_idx"] = np.dtype(np.uint32)
+ result[f"{prefix}_idx_offsets"] = np.dtype(np.uint64)
+ result["start_starts"] = np.dtype(np.uint32)
+ return result
+
+
+class _PackedD1Reader:
+ def __init__(
+ self,
+ store: _BPArrayStore,
+ count: int,
+ *,
+ require_offsets: bool,
+ limits: SourceLimits,
+ ) -> None:
+ self.store = store
+ self.count = count
+ self.reader = _StoredBP128Array(
+ store,
+ "start",
+ count,
+ "plain",
+ require_offsets=require_offsets,
+ limits=limits,
+ )
+ self.blockCount = math.ceil(count / 128)
+ _require_numeric_array(
+ store,
+ "start_starts",
+ np.dtype(np.uint32),
+ self.blockCount,
+ )
+
+ @property
+ def indexOffsets(self) -> NDArray[np.int64]:
+ return self.reader.indexOffsets
+
+ def read(self, start: int, stop: int) -> NDArray[np.uint32]:
+ start = int(start)
+ stop = int(stop)
+ if start < 0 or stop < start or stop > self.count:
+ raise IndexError(
+ f"BPCells packed start window [{start}, {stop}) "
+ f"is outside [0, {self.count})"
+ )
+ if start == stop:
+ return np.empty(0, dtype=np.uint32)
+ first_block = start // 128
+ final_block = (stop - 1) // 128
+ decoded: list[NDArray[np.uint32]] = []
+ for block in range(first_block, final_block + 1):
+ block_start = block * 128
+ block_stop = min(self.count, block_start + 128)
+ deltas = self.reader.read(block_start, block_stop).astype(
+ np.uint64, copy=False
+ )
+ initial = int(self.store.read_numeric("start_starts", block, block + 1)[0])
+ values = (
+ np.cumsum(deltas, dtype=np.uint64) + np.uint64(initial)
+ ) & np.uint64(_UINT32_MAX)
+ decoded.append(values.astype(np.uint32))
+ combined = np.concatenate(decoded)
+ local_start = start - first_block * 128
+ return combined[local_start : local_start + stop - start]
+
+
+class StoredFragmentSource:
+ def __init__(
+ self,
+ store: _BPArrayStore,
+ *,
+ chromosome_names: Sequence[str] | None = None,
+ cell_names: Sequence[str] | None = None,
+ object_path: str,
+ class_name: str,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ match = _FRAGMENT_VERSION_PATTERN.fullmatch(store.version)
+ if match is None:
+ raise MatrixSourceError(
+ f"unsupported BPCells fragment format {store.version!r}"
+ )
+ compression, version_text = match.groups()
+ self.store = store
+ self.compression = compression
+ self.formatVersion = int(version_text)
+ self.objectPath = object_path
+ self.className = class_name
+ self._limits = limits
+ pointer_dtype = np.dtype(np.uint32 if self.formatVersion == 1 else np.uint64)
+ pointer_length = _require_numeric_array(store, "chr_ptr", pointer_dtype)
+ if pointer_length % 2:
+ raise MatrixSourceError("BPCells chr_ptr must contain start/end pairs")
+ pointer_bytes = pointer_length * pointer_dtype.itemsize
+ if pointer_bytes > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"BPCells chr_ptr exceeds maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ flat_pointers = store.read_numeric("chr_ptr").astype(np.uint64, copy=False)
+ self._chrPointers = flat_pointers.reshape(-1, 2)
+ self._validate_pointers()
+ self._recordCount = (
+ int(self._chrPointers[-1, 1]) if self._chrPointers.size else 0
+ )
+ if self._recordCount > limits.maxNnz:
+ raise ResourceLimitError(
+ f"fragment count {self._recordCount} exceeds maxNnz={limits.maxNnz}"
+ )
+ _require_numeric_array(
+ store,
+ "end_max",
+ np.dtype(np.uint32),
+ math.ceil(self._recordCount / 128),
+ )
+ source_chromosomes = (
+ tuple(chromosome_names)
+ if chromosome_names is not None
+ else store.read_text("chr_names")
+ )
+ source_cells = (
+ tuple(cell_names)
+ if cell_names is not None
+ else store.read_text("cell_names")
+ )
+ normalized_chromosomes = _normalize_names(
+ source_chromosomes,
+ self._chrPointers.shape[0],
+ "chromosome",
+ limits,
+ )
+ normalized_cells = _normalize_names(
+ source_cells,
+ len(source_cells),
+ "cell",
+ limits,
+ )
+ assert normalized_chromosomes is not None
+ assert normalized_cells is not None
+ self._chromosomeNames = normalized_chromosomes
+ self._cellNames = normalized_cells
+ self._cellReader: _StoredBP128Array | None = None
+ self._startReader: _PackedD1Reader | None = None
+ self._endReader: _StoredBP128Array | None = None
+ offset_bytes = 0
+ if compression == "unpacked":
+ for name in ("cell", "start", "end"):
+ _require_numeric_array(
+ store,
+ name,
+ np.dtype(np.uint32),
+ self._recordCount,
+ )
+ else:
+ require_offsets = self.formatVersion == 2
+ self._cellReader = _StoredBP128Array(
+ store,
+ "cell",
+ self._recordCount,
+ "plain",
+ require_offsets=require_offsets,
+ limits=limits,
+ )
+ self._startReader = _PackedD1Reader(
+ store,
+ self._recordCount,
+ require_offsets=require_offsets,
+ limits=limits,
+ )
+ self._endReader = _StoredBP128Array(
+ store,
+ "end",
+ self._recordCount,
+ "plain",
+ require_offsets=require_offsets,
+ limits=limits,
+ )
+ offset_bytes = (
+ self._cellReader.indexOffsets.nbytes
+ + self._startReader.indexOffsets.nbytes
+ + self._endReader.indexOffsets.nbytes
+ )
+ metadata_bytes = (
+ pointer_bytes
+ + offset_bytes
+ + _metadata_bytes(self._chromosomeNames)
+ + _metadata_bytes(self._cellNames)
+ )
+ if metadata_bytes > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"fragment metadata exceeds maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ self.metadataBytes = metadata_bytes
+ overhead = 4096 if compression == "packed" else 0
+ bytes_per_record = 128
+ minimum_block_bytes = (
+ overhead + bytes_per_record * _FRAGMENT_BLOCK_SAFETY_FACTOR
+ )
+ if limits.maxBlockBytes < minimum_block_bytes:
+ raise ResourceLimitError(
+ "fragment decoding requires at least "
+ f"{minimum_block_bytes} bytes; maxBlockBytes={limits.maxBlockBytes}"
+ )
+ self.blockRecords = min(
+ limits.compressedChunkNnz,
+ max(
+ 1,
+ (limits.maxBlockBytes - overhead)
+ // (bytes_per_record * _FRAGMENT_BLOCK_SAFETY_FACTOR),
+ ),
+ max(1, self._recordCount),
+ )
+ self._blockWorkingBytes = overhead + self.blockRecords * bytes_per_record
+ self._validate_records()
+
+ def _validate_pointers(self) -> None:
+ previous = 0
+ for chromosome_id, pair in enumerate(self._chrPointers):
+ start = int(pair[0])
+ stop = int(pair[1])
+ if start != previous:
+ raise MatrixSourceError(
+ f"BPCells chr_ptr chromosome {chromosome_id} starts at {start}; "
+ f"expected {previous}"
+ )
+ if stop < start:
+ raise MatrixSourceError(
+ f"BPCells chr_ptr chromosome {chromosome_id} has decreasing bounds"
+ )
+ previous = stop
+
+ def _read_records(self, start: int, stop: int) -> FragmentBlock:
+ if self.compression == "unpacked":
+ cells = self.store.read_numeric("cell", start, stop).astype(
+ np.uint32, copy=False
+ )
+ starts = self.store.read_numeric("start", start, stop).astype(
+ np.uint32, copy=False
+ )
+ ends = self.store.read_numeric("end", start, stop).astype(
+ np.uint32, copy=False
+ )
+ else:
+ assert self._cellReader is not None
+ assert self._startReader is not None
+ assert self._endReader is not None
+ cells = self._cellReader.read(start, stop)
+ starts = self._startReader.read(start, stop)
+ lengths = self._endReader.read(start, stop)
+ widened = starts.astype(np.uint64) + lengths.astype(np.uint64)
+ if np.any(widened > _UINT32_MAX):
+ raise MatrixSourceError(
+ "packed fragment end coordinate overflows uint32"
+ )
+ ends = widened.astype(np.uint32)
+ return FragmentBlock(cells, starts, ends)
+
+ def _validate_records(self) -> None:
+ expected_end_max: list[int] = []
+ expected_start = 0
+ current_end_max = 0
+ previous_end_max = 0
+ expected_index = 0
+
+ def flush_expected() -> None:
+ nonlocal expected_index
+ if not expected_end_max:
+ return
+ actual = self.store.read_numeric(
+ "end_max",
+ expected_index,
+ expected_index + len(expected_end_max),
+ ).astype(np.uint32, copy=False)
+ expected = np.asarray(expected_end_max, dtype=np.uint32)
+ if not np.array_equal(actual, expected):
+ mismatch = int(np.flatnonzero(actual != expected)[0])
+ raise MatrixSourceError(
+ "BPCells end_max is inconsistent at block "
+ f"{expected_index + mismatch}"
+ )
+ expected_index += len(expected_end_max)
+ expected_end_max.clear()
+
+ for chromosome_id in range(len(self._chromosomeNames)):
+ previous_end_max = max(previous_end_max, current_end_max)
+ current_end_max = 0
+ previous_start: int | None = None
+ pair = self._chrPointers[chromosome_id]
+ chromosome_start = int(pair[0])
+ chromosome_stop = int(pair[1])
+ for block_start in range(
+ chromosome_start,
+ chromosome_stop,
+ self.blockRecords,
+ ):
+ block_stop = min(chromosome_stop, block_start + self.blockRecords)
+ block = self._read_records(block_start, block_stop)
+ if block.size == 0:
+ continue
+ if np.any(block.cellIds >= len(self._cellNames)):
+ local = int(
+ np.flatnonzero(block.cellIds >= len(self._cellNames))[0]
+ )
+ raise MatrixSourceError(
+ f"fragment cell ID is out of range at record {block_start + local}"
+ )
+ if np.any(block.ends < block.starts):
+ local = int(np.flatnonzero(block.ends < block.starts)[0])
+ raise MatrixSourceError(
+ f"fragment end precedes start at record {block_start + local}"
+ )
+ if previous_start is not None and int(block.starts[0]) < previous_start:
+ raise MatrixSourceError(
+ f"fragment starts are not sorted on chromosome {chromosome_id}"
+ )
+ if block.starts.size > 1 and np.any(
+ block.starts[1:] < block.starts[:-1]
+ ):
+ raise MatrixSourceError(
+ f"fragment starts are not sorted on chromosome {chromosome_id}"
+ )
+ previous_start = int(block.starts[-1])
+ local_start = 0
+ while local_start < block.size:
+ global_position = block_start + local_start
+ take = min(block.size - local_start, 128 - global_position % 128)
+ local_stop = local_start + take
+ current_end_max = max(
+ current_end_max,
+ int(np.max(block.ends[local_start:local_stop])),
+ )
+ global_position += take
+ if global_position % 128 == 0:
+ expected_end_max.append(max(current_end_max, previous_end_max))
+ previous_end_max = 0
+ local_start = local_stop
+ flush_expected()
+ expected_start = block_stop
+ if expected_start != chromosome_stop:
+ raise MatrixSourceError(
+ f"fragment chromosome {chromosome_id} ended unexpectedly"
+ )
+ if self._recordCount % 128:
+ expected_end_max.append(max(current_end_max, previous_end_max))
+ flush_expected()
+ if expected_index != math.ceil(self._recordCount / 128):
+ raise MatrixSourceError("BPCells end_max has an invalid length")
+
+ @property
+ def chromosomeNames(self) -> tuple[str, ...]:
+ return self._chromosomeNames
+
+ @property
+ def cellNames(self) -> tuple[str, ...]:
+ return self._cellNames
+
+ @property
+ def recordCount(self) -> int:
+ return self._recordCount
+
+ @property
+ def residentBytes(self) -> int:
+ store_bytes = int(getattr(self.store, "residentBytes", 0))
+ return self.metadataBytes + store_bytes
+
+ @property
+ def blockWorkingBytes(self) -> int:
+ return self._blockWorkingBytes
+
+ def iter_chromosome(self, chromosome_id: int) -> Iterator[FragmentBlock]:
+ chromosome_id = int(chromosome_id)
+ if chromosome_id < 0 or chromosome_id >= len(self.chromosomeNames):
+ raise IndexError(
+ f"fragment chromosome ID {chromosome_id} is outside "
+ f"[0, {len(self.chromosomeNames)})"
+ )
+ pair = self._chrPointers[chromosome_id]
+ start = int(pair[0])
+ stop = int(pair[1])
+ for block_start in range(start, stop, self.blockRecords):
+ yield self._read_records(
+ block_start,
+ min(stop, block_start + self.blockRecords),
+ )
+
+
+def _optional_override_names(
+ slots: Mapping[str, Any],
+ name: str,
+ object_path: str,
+ limits: SourceLimits,
+) -> tuple[str, ...] | None:
+ value = slots.get(name)
+ if value is None:
+ return None
+ values = _text_values(value, f"{object_path}@{name}", limits)
+ return values or None
+
+
+def fragment_source_from_slots(
+ specification: Mapping[str, Any],
+ *,
+ object_path: str = "$",
+ rds_path: str | os.PathLike[str] | None = None,
+ absolute_prefix_remaps: Mapping[str | os.PathLike[str], str | os.PathLike[str]]
+ | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+) -> FragmentSource:
+ if not isinstance(specification, Mapping):
+ raise TypeError(f"fragment source at {object_path} must be a mapping")
+ slots = _slot_mapping(specification, object_path)
+ classes = _class_names(
+ specification.get(
+ "className",
+ specification.get(
+ "class",
+ slots.get("className", slots.get("class")),
+ ),
+ )
+ )
+ capability = FRAGMENT_CAPABILITY_REGISTRY.resolve(
+ classes,
+ object_path=object_path,
+ )
+ class_name = capability.className
+
+ def resolve_fragment(value: Any, path: str) -> FragmentSource:
+ if isinstance(value, FragmentSource):
+ return value
+ if isinstance(value, Mapping):
+ return fragment_source_from_slots(
+ value,
+ object_path=path,
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ limits=limits,
+ )
+ raise TypeError(f"fragment input at {path} must be a source or mapping")
+
+ if class_name == "MergeFragments":
+ values = slots.get("fragments_list")
+ if not isinstance(values, Sequence) or isinstance(
+ values, str | bytes | bytearray
+ ):
+ raise TypeError(f"fragments_list at {object_path} must be a sequence")
+ return MergedFragmentSource(
+ [
+ resolve_fragment(value, f"{object_path}@fragments_list[{index}]")
+ for index, value in enumerate(values)
+ ]
+ )
+ if class_name not in {
+ "UnpackedMemFragments",
+ "PackedMemFragments",
+ "FragmentsDir",
+ "FragmentsHDF5",
+ }:
+ source = resolve_fragment(
+ slots.get("fragments"),
+ f"{object_path}@fragments",
+ )
+ if class_name == "ShiftFragments":
+ shift_start = _integer_scalar(
+ slots.get("shift_start"),
+ f"{object_path}@shift_start",
+ )
+ shift_end = _integer_scalar(
+ slots.get("shift_end"),
+ f"{object_path}@shift_end",
+ )
+ assert shift_start is not None and shift_end is not None
+ return ShiftedFragmentSource(source, shift_start, shift_end)
+ if class_name == "SelectLength":
+ minimum = _integer_scalar(
+ slots.get("min_len"),
+ f"{object_path}@min_len",
+ allow_missing=True,
+ )
+ maximum = _integer_scalar(
+ slots.get("max_len"),
+ f"{object_path}@max_len",
+ allow_missing=True,
+ )
+ return LengthSelectedFragmentSource(
+ source,
+ 0 if minimum is None else minimum,
+ np.iinfo(np.int32).max if maximum is None else maximum,
+ )
+ if class_name in {"ChrSelectName", "ChrSelectIndex"}:
+ if class_name == "ChrSelectName":
+ names = _text_values(
+ slots.get("chr_names"),
+ f"{object_path}@chr_names",
+ limits,
+ )
+ if len(set(names)) != len(names):
+ raise MatrixSourceError(
+ "fragment chromosome selection contains duplicates"
+ )
+ missing = [name for name in names if name not in source.chromosomeNames]
+ if missing:
+ raise MatrixSourceError(
+ f"fragment chromosome selection contains unknown names {missing!r}"
+ )
+ selection = tuple(source.chromosomeNames.index(name) for name in names)
+ else:
+ raw = _unsigned_vector(
+ slots.get("chr_index_selection"),
+ "chr_index_selection",
+ object_path,
+ limits,
+ )
+ if np.any(raw == 0) or np.any(raw > len(source.chromosomeNames)):
+ raise MatrixSourceError(
+ "fragment chromosome selection contains an invalid R index"
+ )
+ selection = tuple(int(value) - 1 for value in raw)
+ if len(set(selection)) != len(selection):
+ raise MatrixSourceError(
+ "fragment chromosome selection contains duplicates"
+ )
+ names = tuple(source.chromosomeNames[index] for index in selection)
+ return ChromosomeSelectedFragmentSource(source, selection, names)
+ if class_name in {"CellSelectName", "CellSelectIndex"}:
+ if class_name == "CellSelectName":
+ names = _text_values(
+ slots.get("cell_names"),
+ f"{object_path}@cell_names",
+ limits,
+ )
+ if len(set(names)) != len(names):
+ raise MatrixSourceError(
+ "fragment cell selection contains duplicates"
+ )
+ missing = [name for name in names if name not in source.cellNames]
+ if missing:
+ raise MatrixSourceError(
+ f"fragment cell selection contains unknown names {missing!r}"
+ )
+ selection = tuple(source.cellNames.index(name) for name in names)
+ else:
+ raw = _unsigned_vector(
+ slots.get("cell_index_selection"),
+ "cell_index_selection",
+ object_path,
+ limits,
+ )
+ if np.any(raw == 0) or np.any(raw > len(source.cellNames)):
+ raise MatrixSourceError(
+ "fragment cell selection contains an invalid R index"
+ )
+ selection = tuple(int(value) - 1 for value in raw)
+ if len(set(selection)) != len(selection):
+ raise MatrixSourceError(
+ "fragment cell selection contains duplicates"
+ )
+ names = tuple(source.cellNames[index] for index in selection)
+ mapping = np.full(len(source.cellNames), -1, dtype=np.int64)
+ mapping[np.asarray(selection, dtype=np.int64)] = np.arange(
+ len(selection),
+ dtype=np.int64,
+ )
+ return CellMappedFragmentSource(source, mapping, names)
+ if class_name == "CellMerge":
+ group_names = _text_values(
+ slots.get("group_names"),
+ f"{object_path}@group_names",
+ limits,
+ )
+ group_ids = _unsigned_vector(
+ slots.get("group_ids"),
+ "group_ids",
+ object_path,
+ limits,
+ )
+ if group_ids.shape != (len(source.cellNames),) or np.any(
+ group_ids >= len(group_names)
+ ):
+ raise MatrixSourceError(
+ "fragment cell merge groups do not match the source cells"
+ )
+ return CellMappedFragmentSource(
+ source,
+ group_ids.astype(np.int64),
+ group_names,
+ )
+ if class_name in {"ChrRename", "CellRename", "CellPrefix"}:
+ if class_name == "ChrRename":
+ return RenamedFragmentSource(
+ source,
+ chromosome_names=_text_values(
+ slots.get("chr_names"),
+ f"{object_path}@chr_names",
+ limits,
+ ),
+ )
+ if class_name == "CellRename":
+ return RenamedFragmentSource(
+ source,
+ cell_names=_text_values(
+ slots.get("cell_names"),
+ f"{object_path}@cell_names",
+ limits,
+ ),
+ )
+ prefix = _text_scalar(
+ slots.get("prefix"),
+ f"{object_path}@prefix",
+ )
+ return RenamedFragmentSource(
+ source,
+ cell_names=tuple(prefix + name for name in source.cellNames),
+ )
+ if class_name == "RegionSelect":
+ chromosome_levels = _text_values(
+ slots.get("chr_levels"),
+ f"{object_path}@chr_levels",
+ limits,
+ )
+ chromosome_ids = _unsigned_vector(
+ slots.get("chr_id"),
+ "chr_id",
+ object_path,
+ limits,
+ )
+ starts = _unsigned_vector(
+ slots.get("start"),
+ "start",
+ object_path,
+ limits,
+ )
+ ends = _unsigned_vector(
+ slots.get("end"),
+ "end",
+ object_path,
+ limits,
+ )
+ if not (chromosome_ids.shape == starts.shape == ends.shape) or np.any(
+ chromosome_ids >= len(chromosome_levels)
+ ):
+ raise MatrixSourceError("fragment region metadata is inconsistent")
+ if np.any(ends < starts):
+ raise MatrixSourceError("fragment region end precedes its start")
+ source_regions: dict[
+ int,
+ tuple[list[np.uint32], list[np.uint32]],
+ ] = {}
+ for chromosome_id, start, end in zip(
+ chromosome_ids,
+ starts,
+ ends,
+ strict=True,
+ ):
+ name = chromosome_levels[int(chromosome_id)]
+ if name not in source.chromosomeNames:
+ continue
+ source_id = source.chromosomeNames.index(name)
+ region_starts, region_ends = source_regions.setdefault(
+ source_id,
+ ([], []),
+ )
+ region_starts.append(start)
+ region_ends.append(end)
+ normalized_regions = {
+ chromosome_id: (
+ np.asarray(region_starts, dtype=np.uint32),
+ np.asarray(region_ends, dtype=np.uint32),
+ )
+ for chromosome_id, (
+ region_starts,
+ region_ends,
+ ) in source_regions.items()
+ }
+ metadata_bytes = (
+ chromosome_ids.nbytes
+ + starts.nbytes
+ + ends.nbytes
+ + _metadata_bytes(chromosome_levels)
+ )
+ if metadata_bytes > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ "fragment region metadata exceeds "
+ f"maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ return RegionSelectedFragmentSource(
+ source,
+ normalized_regions,
+ invert=_bool_scalar(
+ slots.get("invert_selection", False),
+ f"{object_path}@invert_selection",
+ ),
+ metadata_bytes=metadata_bytes,
+ )
+ raise AssertionError(class_name)
+
+ store: _BPArrayStore
+ if class_name in {"UnpackedMemFragments", "PackedMemFragments"}:
+ if "version" not in slots:
+ raise MatrixSourceError(
+ f"fragment source at {object_path} has no version slot"
+ )
+ version = _text_scalar(slots["version"], f"{object_path}@version")
+ store = cast(
+ _BPArrayStore,
+ _MemoryArrayStore(
+ slots,
+ version,
+ _memory_dtypes(version),
+ limits,
+ object_path,
+ ),
+ )
+ chromosome_names = None
+ cell_names = None
+ elif class_name == "FragmentsDir":
+ if "dir" not in slots:
+ raise MatrixSourceError(f"fragment source at {object_path} has no dir slot")
+ path = _resolve_sidecar(
+ _single_value(slots["dir"], f"{object_path}@dir"),
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ expect="directory",
+ )
+ store = cast(_BPArrayStore, _DirectoryArrayStore(path, limits))
+ chromosome_names = _optional_override_names(
+ slots, "chr_names", object_path, limits
+ )
+ cell_names = _optional_override_names(slots, "cell_names", object_path, limits)
+ elif class_name == "FragmentsHDF5":
+ path_value = slots.get("path", slots.get("filepath"))
+ if path_value is None or "group" not in slots:
+ raise MatrixSourceError(
+ f"fragment source at {object_path} requires path and group slots"
+ )
+ path = _resolve_sidecar(
+ _single_value(path_value, f"{object_path}@path"),
+ rds_path=rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ expect="file",
+ )
+ group = _text_scalar(slots["group"], f"{object_path}@group")
+ store = cast(_BPArrayStore, _HDF5ArrayStore(path, group, limits))
+ chromosome_names = _optional_override_names(
+ slots, "chr_names", object_path, limits
+ )
+ cell_names = _optional_override_names(slots, "cell_names", object_path, limits)
+ else:
+ raise AssertionError(class_name)
+ match = _FRAGMENT_VERSION_PATTERN.fullmatch(store.version)
+ if match is None:
+ raise MatrixSourceError(
+ f"unsupported BPCells fragment format {store.version!r}"
+ )
+ compression, _version = match.groups()
+ expected_compression = {
+ "UnpackedMemFragments": "unpacked",
+ "PackedMemFragments": "packed",
+ }.get(class_name)
+ if expected_compression is not None and compression != expected_compression:
+ raise MatrixSourceError(f"{class_name} cannot contain {store.version!r}")
+ if class_name in {"FragmentsDir", "FragmentsHDF5"}:
+ if "compressed" not in slots:
+ raise MatrixSourceError(
+ f"fragment source at {object_path} has no compressed slot"
+ )
+ declared_compressed = _bool_scalar(
+ slots["compressed"],
+ f"{object_path}@compressed",
+ )
+ if declared_compressed != (compression == "packed"):
+ raise MatrixSourceError(
+ f"fragment compressed slot at {object_path} disagrees with "
+ f"format {store.version!r}"
+ )
+ if "buffer_size" in slots:
+ _positive_int_scalar(slots["buffer_size"], f"{object_path}@buffer_size")
+ return StoredFragmentSource(
+ store,
+ chromosome_names=chromosome_names,
+ cell_names=cell_names,
+ object_path=object_path,
+ class_name=class_name,
+ limits=limits,
+ )
+
+
+def _unsigned_vector(
+ value: Any,
+ name: str,
+ object_path: str,
+ limits: SourceLimits,
+) -> NDArray[np.uint32]:
+ length = _vector_length(value, f"{object_path}@{name}")
+ required = length * np.dtype(np.uint32).itemsize
+ if required > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"{name} at {object_path} exceeds "
+ f"maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ raw = _read_vector_slice(value, 0, length, f"{object_path}@{name}")
+ if raw.dtype.kind not in "iuf":
+ raise TypeError(f"{name} at {object_path} must contain integers")
+ if raw.dtype.kind == "f":
+ numeric = raw.astype(np.float64, copy=False)
+ invalid = (
+ ~np.isfinite(numeric)
+ | (numeric < 0)
+ | (numeric != np.floor(numeric))
+ | (numeric > _UINT32_MAX)
+ )
+ else:
+ numeric = raw
+ invalid = (numeric < 0) | (numeric > _UINT32_MAX)
+ if np.any(invalid):
+ raise MatrixSourceError(
+ f"{name} at {object_path} contains an invalid uint32 value"
+ )
+ return numeric.astype(np.uint32)
+
+
+def _shape_value(
+ value: Any,
+ object_path: str,
+) -> tuple[int, int]:
+ raw = _read_vector_slice(
+ value,
+ 0,
+ _vector_length(value, object_path),
+ object_path,
+ )
+ if raw.size != 2 or raw.dtype.kind not in "iuf":
+ raise MatrixSourceError(f"dim at {object_path} must contain two integers")
+ numeric = raw.astype(np.float64, copy=False)
+ if (
+ np.any(~np.isfinite(numeric))
+ or np.any(numeric < 0)
+ or np.any(numeric != np.floor(numeric))
+ ):
+ raise MatrixSourceError(f"dim at {object_path} must contain two integers")
+ return int(numeric[0]), int(numeric[1])
+
+
+class FragmentDerivedMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ fragments: FragmentSource,
+ *,
+ matrix_type: str,
+ chromosome_ids: Any,
+ starts: Any,
+ ends: Any,
+ chromosome_levels: Any,
+ mode: Any,
+ tile_widths: Any | None = None,
+ transpose: Any = True,
+ shape: Any | None = None,
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ object_path: str = "$",
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ if not isinstance(fragments, FragmentSource):
+ raise TypeError(
+ f"fragment input at {object_path}@fragments is not a FragmentSource"
+ )
+ if matrix_type not in {"PeakMatrix", "TileMatrix"}:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ "fragment-derived",
+ matrix_type,
+ "unknown fragment-derived matrix class",
+ )
+ self.fragments = fragments
+ self.matrixType = matrix_type
+ self.objectPath = object_path
+ self.operation = "fragment-derived"
+ self.logicalTranspose = _bool_scalar(transpose, f"{object_path}@transpose")
+ self.chromosomeIds = _unsigned_vector(
+ chromosome_ids, "chr_id", object_path, limits
+ )
+ self.starts = _unsigned_vector(starts, "start", object_path, limits)
+ self.ends = _unsigned_vector(ends, "end", object_path, limits)
+ if not (self.chromosomeIds.size == self.starts.size == self.ends.size):
+ raise MatrixSourceError(
+ f"{matrix_type} at {object_path} requires equal chr_id, start, and end lengths"
+ )
+ if np.any(self.ends < self.starts):
+ raise MatrixSourceError(
+ f"{matrix_type} at {object_path} contains an end before its start"
+ )
+ self.chromosomeLevels = _text_values(
+ chromosome_levels,
+ f"{object_path}@chr_levels",
+ limits,
+ )
+ if self.chromosomeLevels != fragments.chromosomeNames:
+ raise MatrixSourceError(
+ f"{matrix_type} chromosome levels at {object_path} do not match "
+ "the fragment source"
+ )
+ if self.chromosomeIds.size and np.any(
+ self.chromosomeIds >= len(self.chromosomeLevels)
+ ):
+ raise MatrixSourceError(
+ f"{matrix_type} at {object_path} has an out-of-range chromosome ID"
+ )
+ self.mode = _text_scalar(mode, f"{object_path}@mode")
+ self.tileWidths: NDArray[np.uint32] | None
+ if matrix_type == "PeakMatrix":
+ if self.mode not in {"insertions", "fragments", "overlaps"}:
+ raise MatrixSourceError(
+ f"PeakMatrix mode {self.mode!r} at {object_path} is invalid"
+ )
+ if tile_widths is not None:
+ raise MatrixSourceError("PeakMatrix cannot contain tile widths")
+ self.tileWidths = None
+ self._validate_peak_order()
+ derived_features = int(self.chromosomeIds.size)
+ self._featureOffsets = None
+ else:
+ if self.mode not in {"insertions", "fragments"}:
+ raise MatrixSourceError(
+ f"TileMatrix mode {self.mode!r} at {object_path} is invalid"
+ )
+ if tile_widths is None:
+ raise MatrixSourceError(
+ f"TileMatrix at {object_path} has no tile_width slot"
+ )
+ self.tileWidths = _unsigned_vector(
+ tile_widths, "tile_width", object_path, limits
+ )
+ if self.tileWidths.size != self.chromosomeIds.size:
+ raise MatrixSourceError(
+ f"TileMatrix at {object_path} requires one width per range"
+ )
+ if np.any(self.tileWidths == 0):
+ raise MatrixSourceError(
+ f"TileMatrix at {object_path} contains a zero tile width"
+ )
+ self._validate_tile_order()
+ widths = self.tileWidths.astype(np.uint64)
+ spans = self.ends.astype(np.uint64) - self.starts.astype(np.uint64)
+ tile_counts = (spans + widths - 1) // widths
+ total_tiles = int(np.sum(tile_counts, dtype=np.uint64))
+ if total_tiles > _UINT32_MAX:
+ raise ResourceLimitError(
+ f"TileMatrix at {object_path} exceeds uint32 feature capacity"
+ )
+ offsets = np.empty(tile_counts.size + 1, dtype=np.uint64)
+ offsets[0] = 0
+ np.cumsum(tile_counts, dtype=np.uint64, out=offsets[1:])
+ self._featureOffsets = offsets
+ derived_features = total_tiles
+ native_shape = (len(fragments.cellNames), derived_features)
+ logical_shape = native_shape[::-1] if self.logicalTranspose else native_shape
+ if shape is not None:
+ declared_shape = _shape_value(shape, f"{object_path}@dim")
+ if declared_shape != logical_shape:
+ raise MatrixSourceError(
+ f"{matrix_type} dim {declared_shape} at {object_path} does not "
+ f"match derived shape {logical_shape}"
+ )
+ region_metadata = (
+ self.chromosomeIds.nbytes
+ + self.starts.nbytes
+ + self.ends.nbytes
+ + (0 if self.tileWidths is None else self.tileWidths.nbytes)
+ + (0 if self._featureOffsets is None else self._featureOffsets.nbytes)
+ + _metadata_bytes(self.chromosomeLevels)
+ + self.chromosomeIds.size * np.dtype(np.int64).itemsize
+ )
+ if region_metadata + fragments.residentBytes > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ "fragment-derived metadata exceeds "
+ f"maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ self._build_chromosome_indexes()
+ self.regionMetadataBytes = int(region_metadata)
+ super().__init__(
+ logical_shape,
+ np.uint32,
+ row_names=row_names,
+ column_names=column_names,
+ is_sparse=True,
+ limits=limits,
+ )
+
+ def _validate_peak_order(self) -> None:
+ for index in range(1, self.chromosomeIds.size):
+ previous = (
+ int(self.chromosomeIds[index - 1]),
+ int(self.ends[index - 1]),
+ int(self.starts[index - 1]),
+ )
+ current = (
+ int(self.chromosomeIds[index]),
+ int(self.ends[index]),
+ int(self.starts[index]),
+ )
+ if current < previous:
+ raise MatrixSourceError(
+ f"PeakMatrix peaks at {self.objectPath} are not sorted by "
+ "(chr, end, start)"
+ )
+
+ def _validate_tile_order(self) -> None:
+ for index in range(1, self.chromosomeIds.size):
+ previous_chromosome = int(self.chromosomeIds[index - 1])
+ chromosome = int(self.chromosomeIds[index])
+ if chromosome < previous_chromosome:
+ raise MatrixSourceError(
+ f"TileMatrix ranges at {self.objectPath} are not sorted by chromosome"
+ )
+ if chromosome == previous_chromosome and int(self.ends[index - 1]) > int(
+ self.starts[index]
+ ):
+ raise MatrixSourceError(
+ f"TileMatrix ranges at {self.objectPath} overlap"
+ )
+
+ def _build_chromosome_indexes(self) -> None:
+ indexes: list[NDArray[np.int64]] = []
+ for chromosome_id in range(len(self.chromosomeLevels)):
+ selected = np.flatnonzero(self.chromosomeIds == chromosome_id).astype(
+ np.int64, copy=False
+ )
+ if self.matrixType == "PeakMatrix" and selected.size:
+ order = np.lexsort(
+ (
+ self.ends[selected],
+ self.starts[selected],
+ )
+ )
+ selected = selected[order]
+ indexes.append(selected)
+ self._chromosomeIndexes = tuple(indexes)
+
+ @property
+ def resident_bytes(self) -> int:
+ return int(
+ super().resident_bytes
+ + self.fragments.residentBytes
+ + self.regionMetadataBytes
+ )
+
+ def _selected_peak_indexes(
+ self,
+ chromosome_id: int,
+ start: int,
+ stop: int,
+ ) -> NDArray[np.int64]:
+ indexes = self._chromosomeIndexes[chromosome_id]
+ if self.logicalTranspose:
+ return indexes
+ return indexes[(indexes >= start) & (indexes < stop)]
+
+ def _peak_contributions(
+ self,
+ start: int,
+ stop: int,
+ ) -> Iterator[tuple[int, int, int]]:
+ for chromosome_id in range(len(self.chromosomeLevels)):
+ candidates = self._selected_peak_indexes(chromosome_id, start, stop)
+ next_candidate = 0
+ active: list[int] = []
+ for block in self.fragments.iter_chromosome(chromosome_id):
+ if block.size == 0:
+ continue
+ block_end_max = int(np.max(block.ends))
+ while (
+ next_candidate < candidates.size
+ and int(self.starts[candidates[next_candidate]]) < block_end_max
+ ):
+ active.append(int(candidates[next_candidate]))
+ next_candidate += 1
+ last_start = int(block.starts[-1])
+ if self.logicalTranspose:
+ keep_cells = (block.cellIds >= start) & (block.cellIds < stop)
+ if np.any(keep_cells):
+ cells = block.cellIds[keep_cells]
+ fragment_starts = block.starts[keep_cells]
+ fragment_ends = block.ends[keep_cells]
+ else:
+ cells = np.empty(0, dtype=np.uint32)
+ fragment_starts = np.empty(0, dtype=np.uint32)
+ fragment_ends = np.empty(0, dtype=np.uint32)
+ else:
+ cells = block.cellIds
+ fragment_starts = block.starts
+ fragment_ends = block.ends
+ retained: list[int] = []
+ for feature in active:
+ peak_start = int(self.starts[feature])
+ peak_end = int(self.ends[feature])
+ if cells.size:
+ if self.mode == "overlaps":
+ mask = (fragment_starts < peak_end) & (
+ fragment_ends > peak_start
+ )
+ counts = None
+ else:
+ start_overlap = (fragment_starts >= peak_start) & (
+ fragment_starts < peak_end
+ )
+ end_overlap = (fragment_ends > peak_start) & (
+ fragment_ends <= peak_end
+ )
+ mask = start_overlap | end_overlap
+ counts = (
+ start_overlap.astype(np.uint8)
+ + end_overlap.astype(np.uint8)
+ if self.mode == "insertions"
+ else None
+ )
+ for position in np.flatnonzero(mask):
+ cell_id = int(cells[position])
+ value = int(counts[position]) if counts is not None else 1
+ if self.logicalTranspose:
+ yield cell_id - start, feature, value
+ else:
+ yield feature - start, cell_id, value
+ if last_start < peak_end:
+ retained.append(feature)
+ active = retained
+
+ def _tile_region_for_start(
+ self,
+ chromosome_id: int,
+ position: int,
+ ) -> int | None:
+ regions = self._chromosomeIndexes[chromosome_id]
+ if regions.size == 0:
+ return None
+ first_region = int(regions[0])
+ region_starts = self.starts[first_region : first_region + regions.size]
+ local = int(np.searchsorted(region_starts, position, side="right") - 1)
+ if local < 0:
+ return None
+ region = first_region + local
+ return region if position < int(self.ends[region]) else None
+
+ def _tile_region_for_end(
+ self,
+ chromosome_id: int,
+ position: int,
+ ) -> int | None:
+ regions = self._chromosomeIndexes[chromosome_id]
+ if regions.size == 0:
+ return None
+ first_region = int(regions[0])
+ region_starts = self.starts[first_region : first_region + regions.size]
+ local = int(np.searchsorted(region_starts, position, side="left") - 1)
+ if local < 0:
+ return None
+ region = first_region + local
+ return region if position <= int(self.ends[region]) else None
+
+ def _tile_feature(self, region: int, position: int, *, end: bool) -> int:
+ assert self.tileWidths is not None
+ assert self._featureOffsets is not None
+ adjusted = position - 1 if end else position
+ return int(
+ self._featureOffsets[region]
+ + (adjusted - int(self.starts[region])) // int(self.tileWidths[region])
+ )
+
+ def _tile_contributions(
+ self,
+ start: int,
+ stop: int,
+ ) -> Iterator[tuple[int, int, int]]:
+ for chromosome_id in range(len(self.chromosomeLevels)):
+ for block in self.fragments.iter_chromosome(chromosome_id):
+ for position in range(block.size):
+ cell_id = int(block.cellIds[position])
+ if self.logicalTranspose and not start <= cell_id < stop:
+ continue
+ fragment_start = int(block.starts[position])
+ fragment_end = int(block.ends[position])
+ start_region = self._tile_region_for_start(
+ chromosome_id,
+ fragment_start,
+ )
+ end_region = self._tile_region_for_end(
+ chromosome_id,
+ fragment_end,
+ )
+ if start_region is not None:
+ feature = self._tile_feature(
+ start_region,
+ fragment_start,
+ end=False,
+ )
+ if self.logicalTranspose or start <= feature < stop:
+ if self.logicalTranspose:
+ yield cell_id - start, feature, 1
+ else:
+ yield feature - start, cell_id, 1
+ if end_region is not None and (
+ self.mode == "insertions" or end_region != start_region
+ ):
+ feature = self._tile_feature(
+ end_region,
+ fragment_end,
+ end=True,
+ )
+ if self.logicalTranspose or start <= feature < stop:
+ if self.logicalTranspose:
+ yield cell_id - start, feature, 1
+ else:
+ yield feature - start, cell_id, 1
+
+ def _contributions(
+ self,
+ start: int,
+ stop: int,
+ ) -> Iterator[tuple[int, int, int]]:
+ if self.matrixType == "PeakMatrix":
+ yield from self._peak_contributions(start, stop)
+ else:
+ yield from self._tile_contributions(start, stop)
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ rows = stop - start
+ base_working = self.fragments.blockWorkingBytes + rows * 64
+ if self.matrixType == "PeakMatrix":
+ base_working += self.n_features * 96
+ minimum_output = (rows + 1) * np.dtype(np.int64).itemsize
+ if rows == 0:
+ return MemoryEstimate(self.resident_bytes, base_working, minimum_output)
+ if base_working + minimum_output > self._limits.maxBlockBytes:
+ return MemoryEstimate(self.resident_bytes, base_working, minimum_output)
+ events = sum(1 for _event in self._contributions(start, stop))
+ possible_nnz = min(events, rows * self.n_features)
+ output = (
+ possible_nnz * (np.dtype(np.uint32).itemsize + np.dtype(np.int64).itemsize)
+ + (rows + 1) * np.dtype(np.int64).itemsize
+ )
+ working = base_working + possible_nnz * _DICT_ENTRY_BYTES
+ return MemoryEstimate(self.resident_bytes, working, output)
+
+ def read_cells(self, start: int, stop: int) -> csr_matrix:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ if start == stop:
+ return csr_matrix((0, self.n_features), dtype=np.uint32)
+ rows: list[dict[int, int]] = [{} for _index in range(stop - start)]
+ nnz = 0
+ for row, column, value in self._contributions(start, stop):
+ row_values = rows[row]
+ previous = row_values.get(column)
+ if previous is None:
+ if nnz >= self._limits.maxNnz:
+ raise ResourceLimitError(
+ f"fragment-derived block exceeds maxNnz={self._limits.maxNnz}"
+ )
+ row_values[column] = value
+ nnz += 1
+ dynamic_bytes = (
+ self.fragments.blockWorkingBytes
+ + nnz
+ * (
+ _DICT_ENTRY_BYTES
+ + np.dtype(np.uint32).itemsize
+ + np.dtype(np.int64).itemsize
+ )
+ + (len(rows) + 1) * np.dtype(np.int64).itemsize
+ )
+ if dynamic_bytes > self._limits.maxBlockBytes:
+ raise ResourceLimitError(
+ "fragment-derived block exceeds "
+ f"maxBlockBytes={self._limits.maxBlockBytes}"
+ )
+ else:
+ updated = previous + value
+ if updated > _UINT32_MAX:
+ raise MatrixSourceError("fragment-derived count overflows uint32")
+ row_values[column] = updated
+ data = np.empty(nnz, dtype=np.uint32)
+ indices = np.empty(nnz, dtype=np.int64)
+ indptr = np.empty(len(rows) + 1, dtype=np.int64)
+ indptr[0] = 0
+ position = 0
+ for row_index, row_values in enumerate(rows):
+ for column, value in row_values.items():
+ data[position] = value
+ indices[position] = column
+ position += 1
+ indptr[row_index + 1] = position
+ return csr_matrix(
+ (data, indices, indptr),
+ shape=(stop - start, self.n_features),
+ dtype=np.uint32,
+ copy=False,
+ )
+
+
+def build_fragment_matrix_source(
+ specification: Mapping[str, Any],
+ *,
+ object_path: str = "$",
+ limits: SourceLimits = DEFAULT_LIMITS,
+) -> FragmentDerivedMatrixSource:
+ class_value = specification.get(
+ "matrixType",
+ specification.get(
+ "matrix_type",
+ specification.get("className", specification.get("class")),
+ ),
+ )
+ classes = _class_names(class_value)
+ matrix_type = classes[0] if classes else ""
+ fragments = specification.get("fragments")
+ if not isinstance(fragments, FragmentSource):
+ raise TypeError(
+ f"fragment input at {object_path}@fragments is not a FragmentSource"
+ )
+ required = {
+ "chromosome_ids": specification.get(
+ "chrId",
+ specification.get("chr_id"),
+ ),
+ "starts": specification.get("start"),
+ "ends": specification.get("end"),
+ "chromosome_levels": specification.get(
+ "chrLevels",
+ specification.get("chr_levels"),
+ ),
+ "mode": specification.get("mode"),
+ }
+ missing = [name for name, value in required.items() if value is None]
+ if missing:
+ raise MatrixSourceError(
+ f"{matrix_type or 'fragment matrix'} at {object_path} is missing "
+ f"{', '.join(missing)}"
+ )
+ return FragmentDerivedMatrixSource(
+ fragments,
+ matrix_type=matrix_type,
+ chromosome_ids=required["chromosome_ids"],
+ starts=required["starts"],
+ ends=required["ends"],
+ chromosome_levels=required["chromosome_levels"],
+ mode=required["mode"],
+ tile_widths=specification.get(
+ "tileWidths",
+ specification.get("tile_width"),
+ ),
+ transpose=specification.get("transpose", True),
+ shape=specification.get(
+ "shape",
+ specification.get("Dim", specification.get("dim")),
+ ),
+ row_names=specification.get(
+ "rowNames",
+ specification.get("row_names"),
+ ),
+ column_names=specification.get(
+ "columnNames",
+ specification.get("column_names"),
+ ),
+ object_path=object_path,
+ limits=limits,
+ )
diff --git a/scarf/readers/_seurat/hdf5.py b/scarf/readers/_seurat/hdf5.py
new file mode 100644
index 00000000..b6a4450c
--- /dev/null
+++ b/scarf/readers/_seurat/hdf5.py
@@ -0,0 +1,861 @@
+import os
+from collections.abc import Sequence
+from typing import Any
+
+import h5py
+import numpy as np
+from numpy.typing import DTypeLike, NDArray
+from scipy.sparse import coo_matrix, csr_matrix
+
+from .errors import MatrixSourceError, ResourceLimitError
+from .paths import (
+ read_hdf5_names,
+ read_hdf5_shape,
+ require_hdf5_datasets,
+ require_hdf5_group,
+ validate_hdf5_file,
+)
+from .sources import (
+ DEFAULT_LIMITS,
+ BaseMatrixSource,
+ MatrixBlock,
+ MatrixSource,
+ MemoryEstimate,
+ SourceLimits,
+ _validate_shape,
+)
+
+
+def _text_attribute(value: Any, object_path: str) -> str:
+ array = np.asarray(value)
+ if array.size != 1:
+ raise MatrixSourceError(f"HDF5 attribute {object_path} must be scalar")
+ scalar = array.reshape(-1)[0]
+ if isinstance(scalar, bytes | np.bytes_):
+ try:
+ return bytes(scalar).decode("utf-8")
+ except UnicodeDecodeError as error:
+ raise MatrixSourceError(
+ f"HDF5 attribute {object_path} is not valid UTF-8"
+ ) from error
+ if isinstance(scalar, str | np.str_):
+ return str(scalar)
+ raise MatrixSourceError(f"HDF5 attribute {object_path} must be text")
+
+
+def _shape_from_group(group: h5py.Group) -> tuple[int, int] | None:
+ for key in ("shape", "h5sparse_shape", "dim"):
+ if key in group.attrs:
+ return read_hdf5_shape(group.attrs[key], f"{group.name}@{key}")
+ if key in group and isinstance(group[key], h5py.Dataset):
+ return read_hdf5_shape(group[key][:], f"{group.name}/{key}")
+ return None
+
+
+def _physical_sparse_layout(group: h5py.Group) -> str | None:
+ for key in ("encoding-type", "h5sparse_format", "sparse_layout", "layout"):
+ if key not in group.attrs:
+ continue
+ value = _text_attribute(group.attrs[key], f"{group.name}@{key}").lower()
+ if value in {"csr", "csr_matrix", "csr-matrix"}:
+ return "csr"
+ if value in {"csc", "csc_matrix", "csc-matrix"}:
+ return "csc"
+ return None
+
+
+def _h5ad_index_path(
+ handle: h5py.File,
+ group_path: str,
+) -> str | None:
+ normalized = "/" + group_path.strip("/")
+ if normalized not in handle:
+ return None
+ node = handle[normalized]
+ if not isinstance(node, h5py.Group):
+ return None
+ key = "_index"
+ if "_index" in node.attrs:
+ key = _text_attribute(node.attrs["_index"], f"{normalized}@_index")
+ if key not in node:
+ return None
+ return f"{normalized}/{key}"
+
+
+class HDF5DenseMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ path: str | os.PathLike[str] | Any,
+ dataset: str,
+ *,
+ r_transposed: bool = True,
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ row_names_path: str | None = None,
+ column_names_path: str | None = None,
+ dtype: DTypeLike | None = None,
+ as_sparse: bool = False,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self.path = validate_hdf5_file(path, limits=limits)
+ self.dataset = "/" + dataset.strip("/")
+ self.rTransposed = bool(r_transposed)
+ with h5py.File(self.path, mode="r") as handle:
+ if self.dataset not in handle:
+ raise MatrixSourceError(f"HDF5 dataset {self.dataset!r} is missing")
+ node = handle[self.dataset]
+ if not isinstance(node, h5py.Dataset):
+ raise MatrixSourceError(f"HDF5 path {self.dataset!r} is not a dataset")
+ if node.ndim != 2:
+ raise MatrixSourceError(
+ f"HDF5 dense dataset {self.dataset!r} must be two-dimensional"
+ )
+ if node.dtype.kind not in "biufc" or node.dtype.hasobject:
+ raise TypeError(
+ f"HDF5 dense dataset {self.dataset!r} has nonnumeric dtype "
+ f"{node.dtype}"
+ )
+ physical_shape = (int(node.shape[0]), int(node.shape[1]))
+ logical_shape = (
+ (physical_shape[1], physical_shape[0])
+ if self.rTransposed
+ else physical_shape
+ )
+ logical_shape = _validate_shape(logical_shape, limits)
+ source_dtype = node.dtype if dtype is None else np.dtype(dtype)
+ resolved_rows = (
+ row_names
+ if row_names is not None
+ else read_hdf5_names(
+ handle,
+ row_names_path,
+ logical_shape[0],
+ limits=limits,
+ )
+ )
+ resolved_columns = (
+ column_names
+ if column_names is not None
+ else read_hdf5_names(
+ handle,
+ column_names_path,
+ logical_shape[1],
+ limits=limits,
+ )
+ )
+ super().__init__(
+ logical_shape,
+ source_dtype,
+ row_names=resolved_rows,
+ column_names=resolved_columns,
+ is_sparse=as_sparse,
+ limits=limits,
+ )
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ if self.is_sparse:
+ output += (stop - start + 1 + (stop - start) * self.n_features) * np.dtype(
+ np.int64
+ ).itemsize
+ return MemoryEstimate(self.resident_bytes, output, output)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ with h5py.File(self.path, mode="r") as handle:
+ node = handle[self.dataset]
+ assert isinstance(node, h5py.Dataset)
+ if self.rTransposed:
+ values = np.asarray(node[start:stop, :], dtype=self.dtype)
+ else:
+ values = np.asarray(node[:, start:stop], dtype=self.dtype).T
+ values = np.ascontiguousarray(values)
+ if self.is_sparse:
+ return csr_matrix(values)
+ return values
+
+
+HDF5ArrayMatrixSource = HDF5DenseMatrixSource
+HDF5ArraySource = HDF5DenseMatrixSource
+
+
+class ReshapedHDF5ArrayMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ path: str | os.PathLike[str] | Any,
+ dataset: str,
+ shape: Sequence[int],
+ *,
+ dtype: DTypeLike | None = None,
+ as_sparse: bool = False,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self.path = validate_hdf5_file(path, limits=limits)
+ self.dataset = "/" + dataset.strip("/")
+ logical_shape = _validate_shape(shape, limits)
+ with h5py.File(self.path, mode="r") as handle:
+ if self.dataset not in handle:
+ raise MatrixSourceError(f"HDF5 dataset {self.dataset!r} is missing")
+ node = handle[self.dataset]
+ if not isinstance(node, h5py.Dataset):
+ raise MatrixSourceError(f"HDF5 path {self.dataset!r} is not a dataset")
+ if node.ndim == 0:
+ raise MatrixSourceError(
+ f"HDF5 dataset {self.dataset!r} cannot be reshaped from a scalar"
+ )
+ if node.dtype.kind not in "biufc" or node.dtype.hasobject:
+ raise TypeError(
+ f"HDF5 dataset {self.dataset!r} has nonnumeric dtype {node.dtype}"
+ )
+ if int(node.size) != logical_shape[0] * logical_shape[1]:
+ raise MatrixSourceError(
+ f"HDF5 dataset {self.dataset!r} has {node.size} values; "
+ f"reshaped matrix requires {logical_shape[0] * logical_shape[1]}"
+ )
+ self.physicalShape = tuple(int(value) for value in node.shape)
+ source_dtype = node.dtype if dtype is None else np.dtype(dtype)
+ super().__init__(
+ logical_shape,
+ source_dtype,
+ is_sparse=as_sparse,
+ limits=limits,
+ )
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ sparse_extra = 0
+ if self.is_sparse:
+ sparse_extra = (
+ stop - start + 1 + (stop - start) * self.n_features
+ ) * np.dtype(np.int64).itemsize
+ return MemoryEstimate(self.resident_bytes, output + sparse_extra, output)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ self._admit(self.estimate_read_memory(start, stop))
+ flat_start = start * self.n_features
+ flat_stop = stop * self.n_features
+ output = np.empty(flat_stop - flat_start, dtype=self.dtype)
+ with h5py.File(self.path, mode="r") as handle:
+ node = handle[self.dataset]
+ assert isinstance(node, h5py.Dataset)
+ position = flat_start
+ output_position = 0
+ while position < flat_stop:
+ coordinates = tuple(
+ int(value)
+ for value in np.unravel_index(
+ position,
+ self.physicalShape,
+ order="C",
+ )
+ )
+ run = int(
+ min(
+ flat_stop - position,
+ self.physicalShape[-1] - coordinates[-1],
+ )
+ )
+ selection: tuple[Any, ...] = coordinates[:-1] + (
+ slice(coordinates[-1], coordinates[-1] + run),
+ )
+ output[output_position : output_position + run] = np.asarray(
+ node[selection],
+ dtype=self.dtype,
+ ).reshape(-1)
+ position += run
+ output_position += run
+ values = output.reshape(stop - start, self.n_features)
+ return csr_matrix(values) if self.is_sparse else values
+
+
+class HDF5CompressedMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ path: str | os.PathLike[str] | Any,
+ group: str,
+ *,
+ physical_shape: Sequence[int],
+ physical_layout: str,
+ physical_order: str,
+ data_name: str = "data",
+ indices_name: str = "indices",
+ indptr_name: str = "indptr",
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ dtype: DTypeLike | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self.path = validate_hdf5_file(path, limits=limits)
+ self.group = "/" + group.strip("/")
+ layout = physical_layout.lower()
+ if layout not in {"csr", "csc"}:
+ raise MatrixSourceError("physical_layout must be 'csr' or 'csc'")
+ if physical_order not in {"cell_by_feature", "feature_by_cell"}:
+ raise MatrixSourceError(
+ "physical_order must be 'cell_by_feature' or 'feature_by_cell'"
+ )
+ if len(physical_shape) != 2:
+ raise MatrixSourceError("physical sparse shape must have length two")
+ self.physicalShape = (int(physical_shape[0]), int(physical_shape[1]))
+ if min(self.physicalShape) < 0:
+ raise MatrixSourceError("physical sparse shape cannot be negative")
+ self.physicalLayout = layout
+ self.physicalOrder = physical_order
+ self.dataName = data_name
+ self.indicesName = indices_name
+ self.indptrName = indptr_name
+ logical_shape = (
+ (self.physicalShape[1], self.physicalShape[0])
+ if physical_order == "cell_by_feature"
+ else self.physicalShape
+ )
+ logical_shape = _validate_shape(logical_shape, limits)
+ with h5py.File(self.path, mode="r") as handle:
+ sparse_group = require_hdf5_group(handle, self.group)
+ arrays = require_hdf5_datasets(
+ sparse_group, (data_name, indices_name, indptr_name)
+ )
+ data = arrays[data_name]
+ indices = arrays[indices_name]
+ indptr = arrays[indptr_name]
+ if data.ndim != 1 or indices.ndim != 1 or indptr.ndim != 1:
+ raise MatrixSourceError(
+ f"HDF5 sparse arrays under {self.group!r} must be one-dimensional"
+ )
+ if data.dtype.kind not in "biufc" or data.dtype.hasobject:
+ raise TypeError("HDF5 sparse data must have a numeric dtype")
+ if not np.issubdtype(indices.dtype, np.integer):
+ raise TypeError("HDF5 sparse indices must contain integers")
+ if not np.issubdtype(indptr.dtype, np.integer):
+ raise TypeError("HDF5 sparse indptr must contain integers")
+ compressed_axis = (
+ self.physicalShape[0] if layout == "csr" else self.physicalShape[1]
+ )
+ if indptr.shape != (compressed_axis + 1,):
+ raise MatrixSourceError(
+ f"HDF5 sparse indptr has shape {indptr.shape}; "
+ f"expected ({compressed_axis + 1},)"
+ )
+ self._nnz = self._validate_structure(data, indices, indptr, limits)
+ source_dtype = data.dtype if dtype is None else np.dtype(dtype)
+ self._direct = (layout == "csr" and physical_order == "cell_by_feature") or (
+ layout == "csc" and physical_order == "feature_by_cell"
+ )
+ super().__init__(
+ logical_shape,
+ source_dtype,
+ row_names=row_names,
+ column_names=column_names,
+ is_sparse=True,
+ limits=limits,
+ )
+
+ def _validate_structure(
+ self,
+ data: h5py.Dataset,
+ indices: h5py.Dataset,
+ indptr: h5py.Dataset,
+ limits: SourceLimits,
+ ) -> int:
+ previous: int | None = None
+ chunk = max(1, min(limits.compressedChunkNnz, int(indptr.size)))
+ final = 0
+ for start in range(0, int(indptr.size), chunk):
+ stop = min(int(indptr.size), start + chunk)
+ pointers = np.asarray(indptr[start:stop])
+ if pointers.size > 1 and np.any(pointers[1:] < pointers[:-1]):
+ raise MatrixSourceError("HDF5 sparse indptr must be nondecreasing")
+ if previous is not None and pointers.size and int(pointers[0]) < previous:
+ raise MatrixSourceError("HDF5 sparse indptr must be nondecreasing")
+ if start == 0 and (not pointers.size or int(pointers[0]) != 0):
+ raise MatrixSourceError("HDF5 sparse indptr must start at zero")
+ if pointers.size:
+ previous = int(pointers[-1])
+ final = previous
+ if final < 0:
+ raise MatrixSourceError("HDF5 sparse indptr contains negative offsets")
+ if final > limits.maxNnz:
+ raise ResourceLimitError(
+ f"HDF5 sparse nnz {final} exceeds maxNnz={limits.maxNnz}"
+ )
+ if int(data.size) != final or int(indices.size) != final:
+ raise MatrixSourceError(
+ "HDF5 sparse data, indices, and indptr lengths are inconsistent"
+ )
+ minor_axis = (
+ self.physicalShape[1]
+ if self.physicalLayout == "csr"
+ else self.physicalShape[0]
+ )
+ for start in range(0, final, limits.compressedChunkNnz):
+ stop = min(final, start + limits.compressedChunkNnz)
+ values = np.asarray(indices[start:stop])
+ if values.size and (np.any(values < 0) or np.any(values >= minor_axis)):
+ raise MatrixSourceError(
+ "HDF5 sparse indices contain an out-of-range value"
+ )
+ return final
+
+ @property
+ def nnz(self) -> int:
+ return self._nnz
+
+ def _direct_bounds(
+ self,
+ start: int,
+ stop: int,
+ ) -> tuple[NDArray[np.int64], int, int]:
+ with h5py.File(self.path, mode="r") as handle:
+ group = require_hdf5_group(handle, self.group)
+ node = group[self.indptrName]
+ assert isinstance(node, h5py.Dataset)
+ pointers = np.asarray(node[start : stop + 1], dtype=np.int64)
+ data_start = int(pointers[0])
+ data_stop = int(pointers[-1])
+ return pointers - data_start, data_start, data_stop
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ index_size = np.dtype(np.int64).itemsize
+ if self._direct:
+ pointers, data_start, data_stop = self._direct_bounds(start, stop)
+ nnz = data_stop - data_start
+ output = nnz * (self.dtype.itemsize + index_size) + pointers.nbytes
+ return MemoryEstimate(self.resident_bytes, output, output)
+ max_output_nnz = min(self.nnz, (stop - start) * self.n_features)
+ output = max_output_nnz * (self.dtype.itemsize + 2 * index_size)
+ output += (stop - start + 1) * index_size
+ working_nnz = min(self.nnz, self._limits.compressedChunkNnz)
+ working = working_nnz * (self.dtype.itemsize + 2 * index_size)
+ return MemoryEstimate(self.resident_bytes, working, output)
+
+ def read_cells(self, start: int, stop: int) -> csr_matrix:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ if self._direct:
+ return self._read_direct(start, stop)
+ return self._read_scanned(start, stop)
+
+ def _read_direct(self, start: int, stop: int) -> csr_matrix:
+ pointers, data_start, data_stop = self._direct_bounds(start, stop)
+ with h5py.File(self.path, mode="r") as handle:
+ group = require_hdf5_group(handle, self.group)
+ data_node = group[self.dataName]
+ index_node = group[self.indicesName]
+ assert isinstance(data_node, h5py.Dataset)
+ assert isinstance(index_node, h5py.Dataset)
+ data = np.asarray(data_node[data_start:data_stop], dtype=self.dtype)
+ indices = np.asarray(index_node[data_start:data_stop], dtype=np.int64)
+ return csr_matrix(
+ (data, indices, pointers),
+ shape=(stop - start, self.n_features),
+ dtype=self.dtype,
+ )
+
+ def _read_scanned(self, start: int, stop: int) -> csr_matrix:
+ if start == stop:
+ return csr_matrix((0, self.n_features), dtype=self.dtype)
+ data_parts: list[NDArray[Any]] = []
+ row_parts: list[NDArray[np.int64]] = []
+ column_parts: list[NDArray[np.int64]] = []
+ retained_nnz = 0
+ with h5py.File(self.path, mode="r") as handle:
+ group = require_hdf5_group(handle, self.group)
+ data_node = group[self.dataName]
+ index_node = group[self.indicesName]
+ pointer_node = group[self.indptrName]
+ assert isinstance(data_node, h5py.Dataset)
+ assert isinstance(index_node, h5py.Dataset)
+ assert isinstance(pointer_node, h5py.Dataset)
+ for feature in range(self.n_features):
+ bounds = np.asarray(pointer_node[feature : feature + 2], dtype=np.int64)
+ vector_start = int(bounds[0])
+ vector_stop = int(bounds[1])
+ for chunk_start in range(
+ vector_start,
+ vector_stop,
+ self._limits.compressedChunkNnz,
+ ):
+ chunk_stop = min(
+ vector_stop,
+ chunk_start + self._limits.compressedChunkNnz,
+ )
+ cell_indexes = np.asarray(
+ index_node[chunk_start:chunk_stop], dtype=np.int64
+ )
+ keep = (cell_indexes >= start) & (cell_indexes < stop)
+ count = int(np.count_nonzero(keep))
+ if count == 0:
+ continue
+ retained_nnz += count
+ required = retained_nnz * (
+ self.dtype.itemsize + 2 * np.dtype(np.int64).itemsize
+ )
+ if required > self._limits.maxBlockBytes:
+ raise ResourceLimitError(
+ "sparse block exceeds "
+ f"maxBlockBytes={self._limits.maxBlockBytes}"
+ )
+ data_parts.append(
+ np.asarray(
+ data_node[chunk_start:chunk_stop],
+ dtype=self.dtype,
+ )[keep]
+ )
+ row_parts.append(cell_indexes[keep] - start)
+ column_parts.append(np.full(count, feature, dtype=np.int64))
+ if not data_parts:
+ return csr_matrix((stop - start, self.n_features), dtype=self.dtype)
+ return coo_matrix(
+ (
+ np.concatenate(data_parts),
+ (np.concatenate(row_parts), np.concatenate(column_parts)),
+ ),
+ shape=(stop - start, self.n_features),
+ dtype=self.dtype,
+ ).tocsr()
+
+
+class H5SparseMatrixSource(HDF5CompressedMatrixSource):
+ def __init__(
+ self,
+ path: str | os.PathLike[str] | Any,
+ group: str,
+ *,
+ shape: Sequence[int] | None = None,
+ sparse_layout: str | None = None,
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ row_names_path: str | None = None,
+ column_names_path: str | None = None,
+ dtype: DTypeLike | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ resolved = validate_hdf5_file(path, limits=limits)
+ group_path = "/" + group.strip("/")
+ with h5py.File(resolved, mode="r") as handle:
+ sparse_group = require_hdf5_group(handle, group_path)
+ stored_shape = _shape_from_group(sparse_group)
+ if shape is None:
+ if stored_shape is None:
+ raise MatrixSourceError(
+ f"HDF5 sparse group {group_path!r} has no shape metadata"
+ )
+ physical_shape = stored_shape
+ logical_shape = (stored_shape[1], stored_shape[0])
+ else:
+ if len(shape) != 2:
+ raise MatrixSourceError(
+ "H5 sparse logical shape must have length two"
+ )
+ logical_shape = (int(shape[0]), int(shape[1]))
+ physical_shape = (logical_shape[1], logical_shape[0])
+ if stored_shape is not None and stored_shape != physical_shape:
+ raise MatrixSourceError(
+ f"HDF5 sparse stored shape {stored_shape} conflicts with "
+ f"logical shape {logical_shape}"
+ )
+ if sparse_layout is None:
+ physical_layout = _physical_sparse_layout(sparse_group)
+ if physical_layout is None:
+ pointer = sparse_group.get("indptr")
+ if not isinstance(pointer, h5py.Dataset):
+ raise MatrixSourceError(
+ f"HDF5 sparse group {group_path!r} has no indptr"
+ )
+ if int(pointer.size) == physical_shape[0] + 1:
+ physical_layout = "csr"
+ elif int(pointer.size) == physical_shape[1] + 1:
+ physical_layout = "csc"
+ else:
+ raise MatrixSourceError(
+ "cannot infer HDF5 sparse physical layout"
+ )
+ else:
+ logical_layout = sparse_layout.lower()
+ if logical_layout not in {"csr", "csc"}:
+ raise MatrixSourceError(
+ "sparse_layout must describe logical CSR or CSC storage"
+ )
+ physical_layout = "csc" if logical_layout == "csr" else "csr"
+ resolved_rows = (
+ row_names
+ if row_names is not None
+ else read_hdf5_names(
+ handle,
+ row_names_path,
+ logical_shape[0],
+ limits=limits,
+ )
+ )
+ resolved_columns = (
+ column_names
+ if column_names is not None
+ else read_hdf5_names(
+ handle,
+ column_names_path,
+ logical_shape[1],
+ limits=limits,
+ )
+ )
+ super().__init__(
+ resolved,
+ group_path,
+ physical_shape=physical_shape,
+ physical_layout=physical_layout,
+ physical_order="cell_by_feature",
+ row_names=resolved_rows,
+ column_names=resolved_columns,
+ dtype=dtype,
+ limits=limits,
+ )
+
+
+class _DelegatingMatrixSource:
+ _delegate: MatrixSource
+
+ @property
+ def shape(self) -> tuple[int, int]:
+ return self._delegate.shape
+
+ @property
+ def dtype(self) -> np.dtype[Any]:
+ return self._delegate.dtype
+
+ @property
+ def row_names(self) -> tuple[str, ...] | None:
+ return self._delegate.row_names
+
+ @property
+ def column_names(self) -> tuple[str, ...] | None:
+ return self._delegate.column_names
+
+ @property
+ def rowNames(self) -> tuple[str, ...] | None:
+ return self.row_names
+
+ @property
+ def columnNames(self) -> tuple[str, ...] | None:
+ return self.column_names
+
+ @property
+ def n_features(self) -> int:
+ return self.shape[0]
+
+ @property
+ def n_cells(self) -> int:
+ return self.shape[1]
+
+ @property
+ def is_sparse(self) -> bool:
+ return self._delegate.is_sparse
+
+ @property
+ def sparse(self) -> bool:
+ return self.is_sparse
+
+ @property
+ def zero_preserving(self) -> bool:
+ return self._delegate.zero_preserving
+
+ @property
+ def zeroPreserving(self) -> bool:
+ return self.zero_preserving
+
+ @property
+ def resident_bytes(self) -> int:
+ return self._delegate.resident_bytes
+
+ @property
+ def residentBytes(self) -> int:
+ return self.resident_bytes
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ return self._delegate.estimate_read_memory(start, stop)
+
+ def memory_estimate(self, start: int, stop: int) -> MemoryEstimate:
+ return self.estimate_read_memory(start, stop)
+
+ def estimate_read_bytes(self, start: int, stop: int) -> int:
+ return self.estimate_read_memory(start, stop).peakBytes
+
+ def estimate_memory(self, start: int, stop: int) -> MemoryEstimate:
+ return self.estimate_read_memory(start, stop)
+
+ def estimated_peak_bytes(self, start: int, stop: int) -> int:
+ return self.estimate_read_bytes(start, stop)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ return self._delegate.read_cells(start, stop)
+
+
+class H5ADMatrixSource(_DelegatingMatrixSource):
+ def __init__(
+ self,
+ path: str | os.PathLike[str] | Any,
+ *,
+ layer: str | None = None,
+ matrix_path: str | None = None,
+ row_names_path: str | None = None,
+ column_names_path: str | None = None,
+ dtype: DTypeLike | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ resolved = validate_hdf5_file(path, limits=limits)
+ if layer is not None and matrix_path is not None:
+ raise ValueError("provide layer or matrix_path, not both")
+ resolved_matrix_path = (
+ "/" + matrix_path.strip("/")
+ if matrix_path is not None
+ else ("/X" if layer is None else f"/layers/{layer.strip('/')}")
+ )
+ with h5py.File(resolved, mode="r") as handle:
+ if resolved_matrix_path not in handle:
+ raise MatrixSourceError(
+ f"H5AD matrix path {resolved_matrix_path!r} is missing"
+ )
+ matrix = handle[resolved_matrix_path]
+ if isinstance(matrix, h5py.Dataset):
+ if matrix.ndim != 2:
+ raise MatrixSourceError(
+ f"H5AD dense matrix {resolved_matrix_path!r} "
+ "must be two-dimensional"
+ )
+ physical_shape = (int(matrix.shape[0]), int(matrix.shape[1]))
+ layout = None
+ elif isinstance(matrix, h5py.Group):
+ group_shape = _shape_from_group(matrix)
+ if group_shape is None:
+ raise MatrixSourceError(
+ f"H5AD sparse matrix {resolved_matrix_path!r} "
+ "has no shape metadata"
+ )
+ physical_shape = group_shape
+ layout = _physical_sparse_layout(matrix)
+ if layout is None:
+ raise MatrixSourceError(
+ f"H5AD sparse matrix {resolved_matrix_path!r} "
+ "has no CSR/CSC encoding"
+ )
+ else:
+ raise MatrixSourceError(
+ f"H5AD matrix path {resolved_matrix_path!r} "
+ "has an unsupported node type"
+ )
+ logical_shape = (physical_shape[1], physical_shape[0])
+ feature_names_path = (
+ row_names_path
+ if row_names_path is not None
+ else _h5ad_index_path(handle, "/var")
+ )
+ cell_names_path = (
+ column_names_path
+ if column_names_path is not None
+ else _h5ad_index_path(handle, "/obs")
+ )
+ row_names = read_hdf5_names(
+ handle,
+ feature_names_path,
+ logical_shape[0],
+ limits=limits,
+ )
+ column_names = read_hdf5_names(
+ handle,
+ cell_names_path,
+ logical_shape[1],
+ limits=limits,
+ )
+ if layout is None:
+ self._delegate = HDF5DenseMatrixSource(
+ resolved,
+ resolved_matrix_path,
+ r_transposed=True,
+ row_names=row_names,
+ column_names=column_names,
+ dtype=dtype,
+ limits=limits,
+ )
+ else:
+ self._delegate = HDF5CompressedMatrixSource(
+ resolved,
+ resolved_matrix_path,
+ physical_shape=physical_shape,
+ physical_layout=layout,
+ physical_order="cell_by_feature",
+ row_names=row_names,
+ column_names=column_names,
+ dtype=dtype,
+ limits=limits,
+ )
+
+
+H5adMatrixSource = H5ADMatrixSource
+
+
+class TenXMatrixSource(HDF5CompressedMatrixSource):
+ def __init__(
+ self,
+ path: str | os.PathLike[str] | Any,
+ *,
+ group: str = "matrix",
+ feature_names: str = "name",
+ dtype: DTypeLike | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ resolved = validate_hdf5_file(path, limits=limits)
+ group_path = "/" + group.strip("/")
+ with h5py.File(resolved, mode="r") as handle:
+ matrix_group = require_hdf5_group(handle, group_path)
+ if "shape" not in matrix_group:
+ raise MatrixSourceError(
+ f"10x group {group_path!r} has no shape dataset"
+ )
+ shape_node = matrix_group["shape"]
+ if not isinstance(shape_node, h5py.Dataset):
+ raise MatrixSourceError("10x shape path must be a dataset")
+ physical_shape = read_hdf5_shape(shape_node[:], f"{group_path}/shape")
+ feature_candidates = (
+ f"{group_path}/features/{feature_names}",
+ f"{group_path}/features/id",
+ f"{group_path}/gene_names",
+ f"{group_path}/genes",
+ )
+ feature_path = next(
+ (candidate for candidate in feature_candidates if candidate in handle),
+ feature_candidates[0],
+ )
+ row_names = read_hdf5_names(
+ handle,
+ feature_path,
+ physical_shape[0],
+ required=True,
+ limits=limits,
+ )
+ column_names = read_hdf5_names(
+ handle,
+ f"{group_path}/barcodes",
+ physical_shape[1],
+ required=True,
+ limits=limits,
+ )
+ super().__init__(
+ resolved,
+ group_path,
+ physical_shape=physical_shape,
+ physical_layout="csc",
+ physical_order="feature_by_cell",
+ row_names=row_names,
+ column_names=column_names,
+ dtype=dtype,
+ limits=limits,
+ )
+
+
+TENxMatrixSource = TenXMatrixSource
+TenxMatrixSource = TenXMatrixSource
diff --git a/scarf/readers/_seurat/operations.py b/scarf/readers/_seurat/operations.py
new file mode 100644
index 00000000..d87ef7f1
--- /dev/null
+++ b/scarf/readers/_seurat/operations.py
@@ -0,0 +1,1827 @@
+from collections.abc import Mapping, Sequence
+from dataclasses import dataclass
+from enum import Enum
+from typing import Any, TypeGuard
+
+import numpy as np
+from numpy.typing import DTypeLike, NDArray
+from scipy.sparse import spmatrix
+
+from .errors import (
+ MatrixSourceError,
+ ResourceLimitError,
+ UnsupportedMatrixOperation,
+)
+from .fragments import build_fragment_matrix_source
+from .sources import (
+ DEFAULT_LIMITS,
+ BaseMatrixSource,
+ CellBindMatrixSource,
+ DtypeMatrixSource,
+ FeatureBindMatrixSource,
+ MappedMatrixSource,
+ MatrixBlock,
+ MatrixSource,
+ MemoryEstimate,
+ RenamedMatrixSource,
+ SourceLimits,
+ TransposeMatrixSource,
+ _block_to_csr,
+ _block_to_dense,
+ _normalize_indexes,
+ _read_selected_cells,
+)
+
+
+type NumericScalar = int | float | complex | np.number[Any]
+
+
+def _is_numeric_scalar(value: object) -> TypeGuard[NumericScalar]:
+ return isinstance(value, int | float | complex | np.number) and not isinstance(
+ value, bool | np.bool_
+ )
+
+
+class MatrixOperation(str, Enum):
+ SUBSET = "subset"
+ TRANSPOSE = "transpose"
+ APERM = "aperm"
+ ROW_BIND = "rbind"
+ COLUMN_BIND = "cbind"
+ RENAME = "dimnames"
+ DTYPE = "dtype"
+ UNARY = "unary"
+ BINARY = "binary"
+ MASK = "mask"
+ SUBASSIGNMENT = "subassignment"
+ RANK = "rank"
+ MULTIPLY = "multiply"
+ FRAGMENT = "fragment-derived"
+
+
+@dataclass(frozen=True)
+class OperationCapability:
+ operation: MatrixOperation
+ aliases: tuple[str, ...]
+ local: bool
+
+
+@dataclass(frozen=True)
+class _UnaryKernel:
+ function: Any
+ zeroPreserving: bool
+
+
+def _round(values: NDArray[Any]) -> NDArray[Any]:
+ return np.round(values)
+
+
+_UNARY_KERNELS: dict[str, _UnaryKernel] = {
+ "identity": _UnaryKernel(np.positive, True),
+ "positive": _UnaryKernel(np.positive, True),
+ "+": _UnaryKernel(np.positive, True),
+ "abs": _UnaryKernel(np.abs, True),
+ "absolute": _UnaryKernel(np.abs, True),
+ "negative": _UnaryKernel(np.negative, True),
+ "neg": _UnaryKernel(np.negative, True),
+ "sqrt": _UnaryKernel(np.sqrt, True),
+ "square": _UnaryKernel(np.square, True),
+ "sign": _UnaryKernel(np.sign, True),
+ "log1p": _UnaryKernel(np.log1p, True),
+ "expm1": _UnaryKernel(np.expm1, True),
+ "sin": _UnaryKernel(np.sin, True),
+ "sinh": _UnaryKernel(np.sinh, True),
+ "asin": _UnaryKernel(np.arcsin, True),
+ "asinh": _UnaryKernel(np.arcsinh, True),
+ "tan": _UnaryKernel(np.tan, True),
+ "tanh": _UnaryKernel(np.tanh, True),
+ "atan": _UnaryKernel(np.arctan, True),
+ "atanh": _UnaryKernel(np.arctanh, True),
+ "floor": _UnaryKernel(np.floor, True),
+ "ceil": _UnaryKernel(np.ceil, True),
+ "ceiling": _UnaryKernel(np.ceil, True),
+ "trunc": _UnaryKernel(np.trunc, True),
+ "round": _UnaryKernel(_round, True),
+ "log": _UnaryKernel(np.log, False),
+ "log2": _UnaryKernel(np.log2, False),
+ "log10": _UnaryKernel(np.log10, False),
+ "exp": _UnaryKernel(np.exp, False),
+ "cos": _UnaryKernel(np.cos, False),
+ "cosh": _UnaryKernel(np.cosh, False),
+ "acos": _UnaryKernel(np.arccos, False),
+ "acosh": _UnaryKernel(np.arccosh, False),
+ "!": _UnaryKernel(np.logical_not, False),
+ "is.na": _UnaryKernel(np.isnan, True),
+ "is.nan": _UnaryKernel(np.isnan, True),
+ "is.finite": _UnaryKernel(np.isfinite, False),
+ "is.infinite": _UnaryKernel(np.isinf, True),
+ "reciprocal": _UnaryKernel(np.reciprocal, False),
+}
+
+
+_BINARY_KERNELS: dict[str, Any] = {
+ "add": np.add,
+ "+": np.add,
+ "subtract": np.subtract,
+ "sub": np.subtract,
+ "-": np.subtract,
+ "multiply": np.multiply,
+ "mul": np.multiply,
+ "*": np.multiply,
+ "divide": np.divide,
+ "true_divide": np.divide,
+ "/": np.divide,
+ "floor_divide": np.floor_divide,
+ "%/%": np.floor_divide,
+ "remainder": np.remainder,
+ "%%": np.remainder,
+ "power": np.power,
+ "pow": np.power,
+ "^": np.power,
+ "minimum": np.minimum,
+ "pmin": np.minimum,
+ "maximum": np.maximum,
+ "pmax": np.maximum,
+ "greater": np.greater,
+ ">": np.greater,
+ "greater_equal": np.greater_equal,
+ ">=": np.greater_equal,
+ "less": np.less,
+ "<": np.less,
+ "less_equal": np.less_equal,
+ "<=": np.less_equal,
+ "equal": np.equal,
+ "==": np.equal,
+ "not_equal": np.not_equal,
+ "!=": np.not_equal,
+ "logical_and": np.logical_and,
+ "&": np.logical_and,
+ "logical_or": np.logical_or,
+ "|": np.logical_or,
+}
+
+
+def _result_dtype(
+ function: Any, left: np.dtype[Any], right: Any = None
+) -> np.dtype[Any]:
+ with np.errstate(all="ignore"):
+ left_value = np.zeros(1, dtype=left)
+ result = (
+ function(left_value)
+ if right is None
+ else function(left_value, np.asarray([right]))
+ )
+ dtype = np.asarray(result).dtype
+ if dtype.kind not in "biufc":
+ raise TypeError(f"operation produces unsupported dtype {dtype}")
+ return dtype
+
+
+def _zero_result(function: Any, right: Any = None) -> bool:
+ with np.errstate(all="ignore"):
+ result = (
+ function(np.asarray([0.0]))
+ if right is None
+ else function(np.asarray([0.0]), np.asarray([right]))
+ )
+ value = np.asarray(result).reshape(-1)[0]
+ return bool(np.isfinite(value) and value == 0)
+
+
+class UnaryTransformMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ operation: str,
+ *,
+ dtype: DTypeLike | None = None,
+ parameter: int | None = None,
+ object_path: str = "$",
+ class_name: str | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ if not isinstance(operation, str):
+ raise UnsupportedMatrixOperation(
+ object_path,
+ repr(operation),
+ class_name,
+ "custom functions are not executed",
+ )
+ normalized = operation.lower()
+ if normalized not in _UNARY_KERNELS:
+ raise UnsupportedMatrixOperation(
+ object_path, operation, class_name, "unknown unary function"
+ )
+ self.source = source
+ self.operation = normalized
+ self.kernel = (
+ _UnaryKernel(
+ lambda values: np.round(values, decimals=parameter),
+ True,
+ )
+ if normalized == "round" and parameter is not None
+ else _UNARY_KERNELS[normalized]
+ )
+ result_dtype = (
+ _result_dtype(self.kernel.function, source.dtype)
+ if dtype is None
+ else np.dtype(dtype)
+ )
+ zero_preserving = source.zero_preserving and self.kernel.zeroPreserving
+ super().__init__(
+ source.shape,
+ result_dtype,
+ row_names=source.row_names,
+ column_names=source.column_names,
+ is_sparse=source.is_sparse and zero_preserving,
+ zero_preserving=zero_preserving,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return super().resident_bytes + self.source.resident_bytes
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ child = self.source.estimate_read_memory(start, stop)
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(self.resident_bytes, child.peakBytes, output)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ block = self.source.read_cells(start, stop)
+ with np.errstate(all="ignore"):
+ if self.is_sparse:
+ result = _block_to_csr(block).astype(self.dtype, copy=True)
+ result.data = np.asarray(
+ self.kernel.function(result.data), dtype=self.dtype
+ )
+ result.eliminate_zeros()
+ return result
+ values = _block_to_dense(block)
+ return np.asarray(self.kernel.function(values), dtype=self.dtype)
+
+
+UnaryMatrixSource = UnaryTransformMatrixSource
+LocalUnaryMatrixSource = UnaryTransformMatrixSource
+
+
+class BinaryTransformMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ left: MatrixSource,
+ right: MatrixSource | NumericScalar,
+ operation: str,
+ *,
+ dtype: DTypeLike | None = None,
+ reverse: bool = False,
+ object_path: str = "$",
+ class_name: str | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ if not isinstance(operation, str):
+ raise UnsupportedMatrixOperation(
+ object_path,
+ repr(operation),
+ class_name,
+ "custom functions are not executed",
+ )
+ normalized = operation.lower()
+ if normalized not in _BINARY_KERNELS:
+ raise UnsupportedMatrixOperation(
+ object_path, operation, class_name, "unknown binary function"
+ )
+ if not isinstance(right, MatrixSource) and not _is_numeric_scalar(right):
+ raise TypeError(
+ "binary right operand must be a MatrixSource or numeric scalar"
+ )
+ self.left = left
+ self.right = right
+ self.operation = normalized
+ self.kernel = _BINARY_KERNELS[normalized]
+ self.reverse = bool(reverse)
+ if self.reverse and isinstance(right, MatrixSource):
+ raise MatrixSourceError(
+ "reversed binary operations require a scalar operand"
+ )
+ if isinstance(right, MatrixSource):
+ if right.shape != left.shape:
+ raise MatrixSourceError(
+ f"binary source shapes differ: {left.shape} and {right.shape}"
+ )
+ if (
+ left.row_names is not None
+ and right.row_names is not None
+ and left.row_names != right.row_names
+ ):
+ raise MatrixSourceError("binary source row names conflict")
+ if (
+ left.column_names is not None
+ and right.column_names is not None
+ and left.column_names != right.column_names
+ ):
+ raise MatrixSourceError("binary source column names conflict")
+ right_sample: Any = np.zeros(1, dtype=right.dtype)
+ with np.errstate(all="ignore"):
+ zero_value = self.kernel(np.asarray([0.0]), np.asarray([0.0]))[0]
+ zero_preserving = bool(np.isfinite(zero_value) and zero_value == 0)
+ sparse = left.is_sparse and right.is_sparse and zero_preserving
+ else:
+ right_sample = right
+ zero_preserving = (
+ _zero_result(self.kernel, right)
+ if not self.reverse
+ else _zero_result(lambda values: self.kernel(right, values))
+ )
+ sparse = left.is_sparse and zero_preserving
+ if dtype is None:
+ with np.errstate(all="ignore"):
+ left_sample = np.zeros(1, dtype=left.dtype)
+ result_dtype = np.asarray(
+ self.kernel(right_sample, left_sample)
+ if self.reverse
+ else self.kernel(left_sample, right_sample)
+ ).dtype
+ else:
+ result_dtype = np.dtype(dtype)
+ super().__init__(
+ left.shape,
+ result_dtype,
+ row_names=left.row_names,
+ column_names=left.column_names,
+ is_sparse=sparse,
+ zero_preserving=(
+ left.zero_preserving
+ and (right.zero_preserving if isinstance(right, MatrixSource) else True)
+ and zero_preserving
+ ),
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ right_bytes = (
+ self.right.resident_bytes if isinstance(self.right, MatrixSource) else 0
+ )
+ return super().resident_bytes + self.left.resident_bytes + right_bytes
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ child = self.left.estimate_read_memory(start, stop).peakBytes
+ if isinstance(self.right, MatrixSource):
+ child += self.right.estimate_read_memory(start, stop).peakBytes
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(self.resident_bytes, child, output)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ left_block = self.left.read_cells(start, stop)
+ right_block = (
+ self.right.read_cells(start, stop)
+ if isinstance(self.right, MatrixSource)
+ else self.right
+ )
+ with np.errstate(all="ignore"):
+ if self.is_sparse:
+ left_sparse = _block_to_csr(left_block, dtype=self.dtype)
+ if isinstance(self.right, MatrixSource):
+ right_sparse = _block_to_csr(right_block, dtype=self.dtype)
+ if self.operation in {"add", "+"}:
+ result = left_sparse + right_sparse
+ elif self.operation in {"subtract", "sub", "-"}:
+ result = left_sparse - right_sparse
+ elif self.operation in {"multiply", "mul", "*"}:
+ result = left_sparse.multiply(right_sparse)
+ elif self.operation in {"minimum", "pmin"}:
+ result = left_sparse.minimum(right_sparse)
+ elif self.operation in {"maximum", "pmax"}:
+ result = left_sparse.maximum(right_sparse)
+ else:
+ dense = self.kernel(
+ left_sparse.toarray(), right_sparse.toarray()
+ )
+ return _block_to_csr(
+ np.asarray(dense, dtype=self.dtype),
+ dtype=self.dtype,
+ )
+ result = result.tocsr().astype(self.dtype, copy=False)
+ result.eliminate_zeros()
+ return result
+ result = left_sparse.copy()
+ result.data = np.asarray(
+ (
+ self.kernel(self.right, result.data)
+ if self.reverse
+ else self.kernel(result.data, self.right)
+ ),
+ dtype=self.dtype,
+ )
+ result.eliminate_zeros()
+ return result
+ left_dense = _block_to_dense(left_block)
+ right_dense = (
+ _block_to_dense(right_block)
+ if isinstance(self.right, MatrixSource)
+ else self.right
+ )
+ return np.asarray(
+ (
+ self.kernel(right_dense, left_dense)
+ if self.reverse
+ else self.kernel(left_dense, right_dense)
+ ),
+ dtype=self.dtype,
+ )
+
+
+BinaryMatrixSource = BinaryTransformMatrixSource
+LocalBinaryMatrixSource = BinaryTransformMatrixSource
+
+
+class MaskMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ mask: MatrixSource,
+ *,
+ fill_value: int | float | complex = 0,
+ keep_nonzero: bool = True,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ if source.shape != mask.shape:
+ raise MatrixSourceError(
+ f"mask shape {mask.shape} does not match source {source.shape}"
+ )
+ if not np.isscalar(fill_value):
+ raise TypeError("mask fill_value must be numeric scalar")
+ self.source = source
+ self.mask = mask
+ self.fillValue = fill_value
+ self.keepNonzero = bool(keep_nonzero)
+ dtype = np.result_type(source.dtype, type(fill_value))
+ sparse = source.is_sparse and fill_value == 0
+ super().__init__(
+ source.shape,
+ dtype,
+ row_names=source.row_names,
+ column_names=source.column_names,
+ is_sparse=sparse,
+ zero_preserving=source.zero_preserving and fill_value == 0,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return (
+ super().resident_bytes
+ + self.source.resident_bytes
+ + self.mask.resident_bytes
+ )
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ child = self.source.estimate_read_memory(start, stop).peakBytes
+ child += self.mask.estimate_read_memory(start, stop).peakBytes
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(self.resident_bytes, child, output)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ values = self.source.read_cells(start, stop)
+ mask_block = self.mask.read_cells(start, stop)
+ if self.is_sparse:
+ source_sparse = _block_to_csr(values, dtype=self.dtype)
+ if self.keepNonzero:
+ selected_mask: Any = _block_to_csr(mask_block, dtype=bool)
+ else:
+ selected_mask = np.logical_not(_block_to_dense(mask_block, dtype=bool))
+ result = source_sparse.multiply(selected_mask).tocsr()
+ result.eliminate_zeros()
+ return result
+ dense = _block_to_dense(values, dtype=self.dtype)
+ mask_dense = _block_to_dense(mask_block, dtype=bool)
+ if not self.keepNonzero:
+ mask_dense = np.logical_not(mask_dense)
+ return np.where(mask_dense, dense, self.fillValue).astype(
+ self.dtype, copy=False
+ )
+
+
+@dataclass(frozen=True, init=False)
+class Subassignment:
+ featureIndices: Sequence[int] | NDArray[Any]
+ cellIndices: Sequence[int] | NDArray[Any]
+ value: MatrixSource | NDArray[Any] | NumericScalar
+
+ def __init__(
+ self,
+ feature_indices: Sequence[int] | NDArray[Any] | None = None,
+ cell_indices: Sequence[int] | NDArray[Any] | None = None,
+ value: MatrixSource | NDArray[Any] | NumericScalar | None = None,
+ *,
+ featureIndices: Sequence[int] | NDArray[Any] | None = None,
+ cellIndices: Sequence[int] | NDArray[Any] | None = None,
+ ) -> None:
+ if feature_indices is not None and featureIndices is not None:
+ raise TypeError("provide only one feature index spelling")
+ if cell_indices is not None and cellIndices is not None:
+ raise TypeError("provide only one cell index spelling")
+ resolved_features = (
+ feature_indices if featureIndices is None else featureIndices
+ )
+ resolved_cells = cell_indices if cellIndices is None else cellIndices
+ if resolved_features is None or resolved_cells is None or value is None:
+ raise TypeError(
+ "subassignment requires feature indexes, cell indexes, and value"
+ )
+ object.__setattr__(self, "featureIndices", resolved_features)
+ object.__setattr__(self, "cellIndices", resolved_cells)
+ object.__setattr__(self, "value", value)
+
+ @property
+ def feature_indices(self) -> Sequence[int] | NDArray[Any]:
+ return self.featureIndices
+
+ @property
+ def cell_indices(self) -> Sequence[int] | NDArray[Any]:
+ return self.cellIndices
+
+
+@dataclass(frozen=True)
+class _ResolvedSubassignment:
+ featureIndices: NDArray[np.int64]
+ cellIndices: NDArray[np.int64]
+ value: MatrixSource | NDArray[Any] | NumericScalar
+
+
+class DelayedSubassignmentMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ assignments: Sequence[Subassignment],
+ *,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ if not assignments:
+ raise ValueError("delayed subassignment requires at least one assignment")
+ self.source = source
+ resolved: list[_ResolvedSubassignment] = []
+ dtypes: list[np.dtype[Any]] = [source.dtype]
+ for index, assignment in enumerate(assignments):
+ features = _normalize_indexes(
+ assignment.featureIndices, source.shape[0], "feature"
+ )
+ cells = _normalize_indexes(assignment.cellIndices, source.shape[1], "cell")
+ value = assignment.value
+ if callable(value):
+ raise TypeError("subassignment values cannot be callable")
+ if isinstance(value, MatrixSource):
+ expected = (features.size, cells.size)
+ if value.shape != expected:
+ raise MatrixSourceError(
+ f"subassignment {index} source has shape {value.shape}; "
+ f"expected {expected}"
+ )
+ dtypes.append(value.dtype)
+ elif _is_numeric_scalar(value):
+ dtypes.append(np.asarray(value).dtype)
+ else:
+ array = np.asarray(value)
+ if array.shape != (features.size, cells.size):
+ raise MatrixSourceError(
+ f"subassignment {index} array has shape {array.shape}; "
+ f"expected {(features.size, cells.size)}"
+ )
+ value = array
+ dtypes.append(array.dtype)
+ resolved.append(_ResolvedSubassignment(features, cells, value))
+ self.assignments = tuple(resolved)
+ super().__init__(
+ source.shape,
+ np.result_type(*dtypes),
+ row_names=source.row_names,
+ column_names=source.column_names,
+ is_sparse=False,
+ zero_preserving=False,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ mappings = sum(
+ assignment.featureIndices.nbytes + assignment.cellIndices.nbytes
+ for assignment in self.assignments
+ )
+ arrays = sum(
+ assignment.value.nbytes
+ for assignment in self.assignments
+ if isinstance(assignment.value, np.ndarray)
+ )
+ source_values = sum(
+ assignment.value.resident_bytes
+ for assignment in self.assignments
+ if isinstance(assignment.value, MatrixSource)
+ )
+ return int(
+ super().resident_bytes
+ + self.source.resident_bytes
+ + mappings
+ + arrays
+ + source_values
+ )
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ child = self.source.estimate_read_memory(start, stop).peakBytes
+ for assignment in self.assignments:
+ if not isinstance(assignment.value, MatrixSource):
+ continue
+ selected = np.flatnonzero(
+ (assignment.cellIndices >= start) & (assignment.cellIndices < stop)
+ )
+ child += sum(
+ assignment.value.estimate_read_memory(
+ int(position), int(position) + 1
+ ).peakBytes
+ for position in selected
+ )
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(self.resident_bytes, child, output)
+
+ def read_cells(self, start: int, stop: int) -> NDArray[Any]:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ output = _block_to_dense(
+ self.source.read_cells(start, stop), dtype=self.dtype
+ ).copy()
+ for assignment in self.assignments:
+ selected_positions: NDArray[np.int64] = np.flatnonzero(
+ (assignment.cellIndices >= start) & (assignment.cellIndices < stop)
+ ).astype(np.int64, copy=False)
+ if selected_positions.size == 0 or assignment.featureIndices.size == 0:
+ continue
+ output_rows = (assignment.cellIndices[selected_positions] - start).astype(
+ np.int64, copy=False
+ )
+ value = assignment.value
+ if isinstance(value, MatrixSource):
+ assignment_values = _block_to_dense(
+ _read_selected_cells(value, selected_positions),
+ dtype=self.dtype,
+ )
+ elif _is_numeric_scalar(value):
+ assignment_values = np.asarray(value, dtype=self.dtype)
+ else:
+ assert isinstance(value, np.ndarray)
+ assignment_values = np.asarray(
+ value[:, selected_positions], dtype=self.dtype
+ ).T
+ output[np.ix_(output_rows, assignment.featureIndices)] = assignment_values
+ return output
+
+
+SubassignmentMatrixSource = DelayedSubassignmentMatrixSource
+
+
+def _parameter_vector(
+ values: Any,
+ length: int,
+ *,
+ name: str,
+ limits: SourceLimits,
+) -> NDArray[np.float64]:
+ result = np.asarray(values, dtype=np.float64).reshape(-1)
+ if result.shape != (length,):
+ raise MatrixSourceError(f"{name} has length {result.size}; expected {length}")
+ if result.nbytes > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"{name} exceeds maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ return result
+
+
+class AxisMinimumMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ parameters: Any,
+ *,
+ axis: str,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ if axis not in {"feature", "cell"}:
+ raise MatrixSourceError("minimum axis must be feature or cell")
+ self.source = source
+ self.axis = axis
+ self.parameters = _parameter_vector(
+ parameters,
+ source.shape[0] if axis == "feature" else source.shape[1],
+ name=f"{axis} minimum parameters",
+ limits=limits,
+ )
+ if np.any(~np.isfinite(self.parameters)) or np.any(self.parameters <= 0):
+ raise MatrixSourceError("minimum parameters must be finite and positive")
+ super().__init__(
+ source.shape,
+ np.float64,
+ row_names=source.row_names,
+ column_names=source.column_names,
+ is_sparse=source.is_sparse,
+ zero_preserving=True,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return (
+ super().resident_bytes + self.source.resident_bytes + self.parameters.nbytes
+ )
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ source = self.source.estimate_read_memory(start, stop)
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(
+ self.resident_bytes,
+ source.workingBytes + source.outputBytes + output,
+ output,
+ )
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ self._admit(self.estimate_read_memory(start, stop))
+ block = self.source.read_cells(start, stop)
+ if isinstance(block, spmatrix):
+ result = block.tocsr(copy=True).astype(np.float64)
+ if self.axis == "feature":
+ result.data = np.minimum(
+ result.data,
+ self.parameters[result.indices],
+ )
+ else:
+ for row in range(result.shape[0]):
+ row_start = int(result.indptr[row])
+ row_stop = int(result.indptr[row + 1])
+ result.data[row_start:row_stop] = np.minimum(
+ result.data[row_start:row_stop],
+ self.parameters[start + row],
+ )
+ result.eliminate_zeros()
+ return result
+ values = np.asarray(block, dtype=np.float64)
+ parameters = (
+ self.parameters[np.newaxis, :]
+ if self.axis == "feature"
+ else self.parameters[start:stop, np.newaxis]
+ )
+ return np.minimum(values, parameters)
+
+
+class ScaleShiftMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ *,
+ feature_scale: Any | None = None,
+ cell_scale: Any | None = None,
+ global_scale: float = 1.0,
+ feature_shift: Any | None = None,
+ cell_shift: Any | None = None,
+ global_shift: float = 0.0,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self.source = source
+ self.featureScale = (
+ None
+ if feature_scale is None
+ else _parameter_vector(
+ feature_scale,
+ source.shape[0],
+ name="feature scale parameters",
+ limits=limits,
+ )
+ )
+ self.cellScale = (
+ None
+ if cell_scale is None
+ else _parameter_vector(
+ cell_scale,
+ source.shape[1],
+ name="cell scale parameters",
+ limits=limits,
+ )
+ )
+ self.featureShift = (
+ None
+ if feature_shift is None
+ else _parameter_vector(
+ feature_shift,
+ source.shape[0],
+ name="feature shift parameters",
+ limits=limits,
+ )
+ )
+ self.cellShift = (
+ None
+ if cell_shift is None
+ else _parameter_vector(
+ cell_shift,
+ source.shape[1],
+ name="cell shift parameters",
+ limits=limits,
+ )
+ )
+ self.globalScale = float(global_scale)
+ self.globalShift = float(global_shift)
+ shifts_zero = self.globalShift == 0 and all(
+ values is None or not np.any(values)
+ for values in (self.featureShift, self.cellShift)
+ )
+ super().__init__(
+ source.shape,
+ np.float64,
+ row_names=source.row_names,
+ column_names=source.column_names,
+ is_sparse=source.is_sparse and shifts_zero,
+ zero_preserving=shifts_zero,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ parameters = sum(
+ 0 if values is None else values.nbytes
+ for values in (
+ self.featureScale,
+ self.cellScale,
+ self.featureShift,
+ self.cellShift,
+ )
+ )
+ return super().resident_bytes + self.source.resident_bytes + parameters
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ source = self.source.estimate_read_memory(start, stop)
+ dense_output = (stop - start) * self.n_features * self.dtype.itemsize
+ output = source.outputBytes if self.is_sparse else dense_output
+ return MemoryEstimate(
+ self.resident_bytes,
+ source.workingBytes + source.outputBytes + dense_output,
+ output,
+ )
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ self._admit(self.estimate_read_memory(start, stop))
+ block = self.source.read_cells(start, stop)
+ feature_scale: float | NDArray[np.float64] = self.globalScale
+ if self.featureScale is not None:
+ feature_scale = self.featureScale * feature_scale
+ if self.is_sparse and isinstance(block, spmatrix):
+ result = block.tocsr(copy=True).astype(np.float64)
+ result.data *= (
+ feature_scale
+ if isinstance(feature_scale, float)
+ else feature_scale[result.indices]
+ )
+ if self.cellScale is not None:
+ for row in range(result.shape[0]):
+ row_start = int(result.indptr[row])
+ row_stop = int(result.indptr[row + 1])
+ result.data[row_start:row_stop] *= self.cellScale[start + row]
+ result.eliminate_zeros()
+ return result
+ result = _block_to_dense(block, dtype=np.float64)
+ result *= feature_scale
+ if self.cellScale is not None:
+ result *= self.cellScale[start:stop, np.newaxis]
+ if self.featureShift is not None:
+ result += self.featureShift
+ if self.cellShift is not None:
+ result += self.cellShift[start:stop, np.newaxis]
+ result += self.globalShift
+ return result
+
+
+class PearsonResidualMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ *,
+ theta_inverse: Any,
+ gene_beta: Any,
+ cell_read_counts: Any,
+ global_parameters: Any,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self.source = source
+ self.thetaInverse = _parameter_vector(
+ theta_inverse,
+ source.shape[0],
+ name="SCTransform inverse theta",
+ limits=limits,
+ )
+ self.geneBeta = _parameter_vector(
+ gene_beta,
+ source.shape[0],
+ name="SCTransform gene beta",
+ limits=limits,
+ )
+ self.cellReadCounts = _parameter_vector(
+ cell_read_counts,
+ source.shape[1],
+ name="SCTransform cell read counts",
+ limits=limits,
+ )
+ global_values = np.asarray(global_parameters, dtype=np.float64).reshape(-1)
+ if global_values.shape != (3,):
+ raise MatrixSourceError(
+ "SCTransform global parameters must contain three values"
+ )
+ self.sdInverseMax = float(global_values[0])
+ self.clipMinimum = float(global_values[1])
+ self.clipMaximum = float(global_values[2])
+ if self.clipMinimum > self.clipMaximum:
+ raise MatrixSourceError("SCTransform clip bounds are reversed")
+ super().__init__(
+ source.shape,
+ np.float64,
+ row_names=source.row_names,
+ column_names=source.column_names,
+ is_sparse=False,
+ zero_preserving=False,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return (
+ super().resident_bytes
+ + self.source.resident_bytes
+ + self.thetaInverse.nbytes
+ + self.geneBeta.nbytes
+ + self.cellReadCounts.nbytes
+ )
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ source = self.source.estimate_read_memory(start, stop)
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(
+ self.resident_bytes,
+ source.workingBytes + source.outputBytes + 3 * output,
+ output,
+ )
+
+ def read_cells(self, start: int, stop: int) -> NDArray[np.float64]:
+ start, stop = self._window(start, stop)
+ self._admit(self.estimate_read_memory(start, stop))
+ values = _block_to_dense(
+ self.source.read_cells(start, stop),
+ dtype=np.float64,
+ )
+ mu = self.cellReadCounts[start:stop, np.newaxis] * self.geneBeta[np.newaxis, :]
+ variance = mu + mu * mu * self.thetaInverse[np.newaxis, :]
+ with np.errstate(divide="ignore", invalid="ignore"):
+ inverse_sd = np.minimum(self.sdInverseMax, 1.0 / np.sqrt(variance))
+ result = (values - mu) * inverse_sd
+ return np.clip(result, self.clipMinimum, self.clipMaximum)
+
+
+class LinearResidualMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ *,
+ feature_parameters: Any,
+ cell_parameters: Any,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self.source = source
+ self.featureParameters = np.asarray(
+ feature_parameters,
+ dtype=np.float64,
+ )
+ self.cellParameters = np.asarray(cell_parameters, dtype=np.float64)
+ if (
+ self.featureParameters.ndim != 2
+ or self.cellParameters.ndim != 2
+ or self.featureParameters.shape[1] != source.shape[0]
+ or self.cellParameters.shape[1] != source.shape[1]
+ or self.featureParameters.shape[0] != self.cellParameters.shape[0]
+ ):
+ raise MatrixSourceError(
+ "linear residual parameters do not match the matrix shape"
+ )
+ parameter_bytes = self.featureParameters.nbytes + self.cellParameters.nbytes
+ if parameter_bytes > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ "linear residual parameters exceed "
+ f"maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ super().__init__(
+ source.shape,
+ np.float64,
+ row_names=source.row_names,
+ column_names=source.column_names,
+ is_sparse=False,
+ zero_preserving=False,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return (
+ super().resident_bytes
+ + self.source.resident_bytes
+ + self.featureParameters.nbytes
+ + self.cellParameters.nbytes
+ )
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ source = self.source.estimate_read_memory(start, stop)
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(
+ self.resident_bytes,
+ source.workingBytes + source.outputBytes + 2 * output,
+ output,
+ )
+
+ def read_cells(self, start: int, stop: int) -> NDArray[np.float64]:
+ start, stop = self._window(start, stop)
+ self._admit(self.estimate_read_memory(start, stop))
+ values = _block_to_dense(
+ self.source.read_cells(start, stop),
+ dtype=np.float64,
+ )
+ prediction = self.cellParameters[:, start:stop].T @ self.featureParameters
+ return values - prediction
+
+
+class UnsupportedExecutionMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource | None,
+ operation: str,
+ *,
+ object_path: str,
+ class_name: str | None,
+ reason: str,
+ shape: Sequence[int] | None = None,
+ dtype: DTypeLike | None = None,
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ if source is None and (shape is None or dtype is None):
+ raise TypeError("unsupported source requires a source or shape and dtype")
+ resolved_shape = source.shape if shape is None and source is not None else shape
+ resolved_dtype = source.dtype if dtype is None and source is not None else dtype
+ if resolved_shape is None or resolved_dtype is None:
+ raise TypeError("unsupported source metadata is incomplete")
+ self.source = source
+ self.operation = operation
+ self.objectPath = object_path
+ self.className = class_name
+ self.reason = reason
+ super().__init__(
+ resolved_shape,
+ resolved_dtype,
+ row_names=(
+ source.row_names
+ if row_names is None and source is not None
+ else row_names
+ ),
+ column_names=(
+ source.column_names
+ if column_names is None and source is not None
+ else column_names
+ ),
+ is_sparse=source.is_sparse if source is not None else True,
+ zero_preserving=(source.zero_preserving if source is not None else True),
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ source_bytes = 0 if self.source is None else self.source.resident_bytes
+ return super().resident_bytes + source_bytes
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ self._window(start, stop)
+ if self.source is None:
+ return MemoryEstimate(self.resident_bytes)
+ return self.source.estimate_read_memory(start, stop)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ self._window(start, stop)
+ raise UnsupportedMatrixOperation(
+ self.objectPath,
+ self.operation,
+ self.className,
+ self.reason,
+ )
+
+
+def _rank_values(values: NDArray[Any], total_size: int) -> NDArray[np.float64]:
+ numeric = np.asarray(values)
+ if numeric.ndim != 1 or numeric.size > total_size:
+ raise MatrixSourceError("rank input has an invalid shape")
+ if not np.all(np.isfinite(numeric)):
+ raise MatrixSourceError("rank input contains non-finite values")
+ implicit_zeros = total_size - int(numeric.size)
+ explicit_zeros = int(np.count_nonzero(numeric == 0))
+ negative_count = int(np.count_nonzero(numeric < 0))
+ zero_count = implicit_zeros + explicit_zeros
+ zero_rank = negative_count + (1 + zero_count) / 2.0 if zero_count else 0.0
+ order = np.argsort(numeric, kind="stable")
+ sorted_values = numeric[order]
+ sorted_ranks = np.empty(numeric.size, dtype=np.float64)
+ start = 0
+ while start < numeric.size:
+ stop = start + 1
+ while stop < numeric.size and sorted_values[stop] == sorted_values[start]:
+ stop += 1
+ value = sorted_values[start]
+ if value == 0:
+ rank = 0.0
+ else:
+ rank = (start + 1 + stop) / 2.0 - zero_rank
+ if value > 0:
+ rank += implicit_zeros
+ sorted_ranks[start:stop] = rank
+ start = stop
+ result = np.empty(numeric.size, dtype=np.float64)
+ result[order] = sorted_ranks
+ return result
+
+
+class RankMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ *,
+ axis: str = "column",
+ object_path: str = "$",
+ class_name: str | None = "RankMatrix",
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ if axis not in {"column", "row"}:
+ raise MatrixSourceError("rank axis must be 'column' or 'row'")
+ self.source = source
+ self.axis = axis
+ self.operation = "rank"
+ self.objectPath = object_path
+ self.className = class_name
+ super().__init__(
+ source.shape,
+ np.float64,
+ row_names=source.row_names,
+ column_names=source.column_names,
+ is_sparse=source.is_sparse,
+ zero_preserving=True,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return super().resident_bytes + self.source.resident_bytes
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ source = self.source.estimate_read_memory(start, stop)
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ if self.axis == "row":
+ scan_peak = 0
+ for scan_start in range(0, self.n_cells, self._limits.tileCells):
+ scan_stop = min(scan_start + self._limits.tileCells, self.n_cells)
+ scan = self.source.estimate_read_memory(
+ scan_start,
+ scan_stop,
+ )
+ scan_peak = max(
+ scan_peak,
+ scan.workingBytes + scan.outputBytes,
+ )
+ counters = 2 * output
+ sparse_conversion = output if self.source.is_sparse else 0
+ return MemoryEstimate(
+ self.resident_bytes,
+ source.workingBytes
+ + source.outputBytes
+ + scan_peak
+ + counters
+ + sparse_conversion,
+ output,
+ )
+ return MemoryEstimate(
+ self.resident_bytes,
+ source.workingBytes + source.outputBytes + output,
+ output,
+ )
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ block = self.source.read_cells(start, stop)
+ if self.axis == "row":
+ target = _block_to_dense(block)
+ if not np.all(np.isfinite(target)):
+ raise MatrixSourceError("rank input contains non-finite values")
+ less = np.zeros(target.shape, dtype=np.int64)
+ equal = np.zeros(target.shape, dtype=np.int64)
+ negative_count = np.zeros(self.n_features, dtype=np.int64)
+ zero_count = np.zeros(self.n_features, dtype=np.int64)
+ for scan_start in range(0, self.n_cells, self._limits.tileCells):
+ scan_stop = min(scan_start + self._limits.tileCells, self.n_cells)
+ scan_block = self.source.read_cells(scan_start, scan_stop)
+ if isinstance(scan_block, spmatrix):
+ sparse_block = scan_block.tocsc(copy=False)
+ for feature in range(self.n_features):
+ data_start = int(sparse_block.indptr[feature])
+ data_stop = int(sparse_block.indptr[feature + 1])
+ values = np.asarray(sparse_block.data[data_start:data_stop])
+ if not np.all(np.isfinite(values)):
+ raise MatrixSourceError(
+ "rank input contains non-finite values"
+ )
+ ordered = np.sort(values)
+ implicit_zeros = (scan_stop - scan_start) - len(values)
+ targets = target[:, feature]
+ left = np.searchsorted(ordered, targets, side="left")
+ right = np.searchsorted(ordered, targets, side="right")
+ less[:, feature] += left
+ less[:, feature] += implicit_zeros * (targets > 0)
+ equal[:, feature] += right - left
+ equal[:, feature] += implicit_zeros * (targets == 0)
+ negative_count[feature] += int(
+ np.searchsorted(ordered, 0, side="left")
+ )
+ zero_count[feature] += int(
+ np.searchsorted(ordered, 0, side="right")
+ - np.searchsorted(ordered, 0, side="left")
+ + implicit_zeros
+ )
+ else:
+ dense_block = np.asarray(scan_block)
+ if not np.all(np.isfinite(dense_block)):
+ raise MatrixSourceError("rank input contains non-finite values")
+ for feature in range(self.n_features):
+ ordered = np.sort(dense_block[:, feature])
+ targets = target[:, feature]
+ left = np.searchsorted(ordered, targets, side="left")
+ right = np.searchsorted(ordered, targets, side="right")
+ less[:, feature] += left
+ equal[:, feature] += right - left
+ negative_count[feature] += int(
+ np.searchsorted(ordered, 0, side="left")
+ )
+ zero_count[feature] += int(
+ np.searchsorted(ordered, 0, side="right")
+ - np.searchsorted(ordered, 0, side="left")
+ )
+ standard_rank = less + (equal + 1) / 2.0
+ zero_rank = np.where(
+ zero_count > 0,
+ negative_count + (zero_count + 1) / 2.0,
+ 0.0,
+ )
+ result = np.where(
+ target == 0,
+ 0.0,
+ standard_rank - zero_rank[np.newaxis, :],
+ )
+ return _block_to_csr(result) if self.source.is_sparse else result
+ if isinstance(block, spmatrix):
+ result = block.tocsr(copy=True).astype(np.float64)
+ for row in range(result.shape[0]):
+ row_start = int(result.indptr[row])
+ row_stop = int(result.indptr[row + 1])
+ result.data[row_start:row_stop] = _rank_values(
+ result.data[row_start:row_stop],
+ self.n_features,
+ )
+ result.eliminate_zeros()
+ return result
+ values = np.asarray(block)
+ output = np.empty(values.shape, dtype=np.float64)
+ for row in range(values.shape[0]):
+ output[row] = _rank_values(values[row], self.n_features)
+ return output
+
+
+class MatrixMultiplySource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ *,
+ right: MatrixSource | None = None,
+ shape: Sequence[int] | None = None,
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ object_path: str = "$",
+ class_name: str | None = "MatrixMultiply",
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ if right is None:
+ raise MatrixSourceError(
+ f"matrix multiplication at {object_path} requires a right operand"
+ )
+ self.right = right
+ self.source = source
+ self.operation = "multiply"
+ self.objectPath = object_path
+ self.className = class_name
+ if source.shape[1] != right.shape[0]:
+ raise MatrixSourceError(
+ "matrix multiplication inner dimensions do not match"
+ )
+ inferred_shape = (source.shape[0], right.shape[1])
+ if shape is not None and tuple(int(value) for value in shape) != inferred_shape:
+ raise MatrixSourceError(
+ f"matrix multiplication shape {tuple(shape)} does not match "
+ f"inferred shape {inferred_shape}"
+ )
+ super().__init__(
+ inferred_shape,
+ np.result_type(source.dtype, right.dtype),
+ row_names=source.row_names if row_names is None else row_names,
+ column_names=right.column_names if column_names is None else column_names,
+ is_sparse=False,
+ zero_preserving=True,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return (
+ super().resident_bytes
+ + self.source.resident_bytes
+ + self.right.resident_bytes
+ )
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ rows = stop - start
+ inner = self.source.shape[1]
+ output = rows * self.n_features * self.dtype.itemsize
+ right_output = rows * inner * self.dtype.itemsize
+ right_estimate = self.right.estimate_read_memory(start, stop)
+ left_peak = 0
+ for inner_start in range(0, inner, self._limits.tileCells):
+ inner_stop = min(inner_start + self._limits.tileCells, inner)
+ estimate = self.source.estimate_read_memory(inner_start, inner_stop)
+ left_output = (
+ (inner_stop - inner_start) * self.n_features * self.dtype.itemsize
+ )
+ left_peak = max(
+ left_peak,
+ estimate.workingBytes + max(estimate.outputBytes, left_output),
+ )
+ working = (
+ right_estimate.workingBytes
+ + max(right_estimate.outputBytes, right_output)
+ + left_peak
+ + output
+ )
+ return MemoryEstimate(self.resident_bytes, working, output)
+
+ def read_cells(self, start: int, stop: int) -> NDArray[Any]:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ right_values = _block_to_dense(
+ self.right.read_cells(start, stop),
+ dtype=self.dtype,
+ )
+ output = np.zeros((stop - start, self.n_features), dtype=self.dtype)
+ for inner_start in range(0, self.source.shape[1], self._limits.tileCells):
+ inner_stop = min(
+ inner_start + self._limits.tileCells,
+ self.source.shape[1],
+ )
+ left_values = _block_to_dense(
+ self.source.read_cells(inner_start, inner_stop),
+ dtype=self.dtype,
+ )
+ output += right_values[:, inner_start:inner_stop] @ left_values
+ return output
+
+
+MultiplyMatrixSource = MatrixMultiplySource
+
+
+_CAPABILITIES = (
+ OperationCapability(MatrixOperation.SUBSET, ("subset", "[", "extract"), True),
+ OperationCapability(MatrixOperation.TRANSPOSE, ("transpose", "t"), True),
+ OperationCapability(MatrixOperation.APERM, ("aperm",), True),
+ OperationCapability(
+ MatrixOperation.ROW_BIND, ("rbind", "row_bind", "feature_bind"), True
+ ),
+ OperationCapability(
+ MatrixOperation.COLUMN_BIND, ("cbind", "column_bind", "cell_bind"), True
+ ),
+ OperationCapability(
+ MatrixOperation.RENAME, ("dimnames", "rename", "set_dimnames"), True
+ ),
+ OperationCapability(MatrixOperation.DTYPE, ("dtype", "cast", "convert_type"), True),
+ OperationCapability(MatrixOperation.UNARY, ("unary", "unary_transform"), True),
+ OperationCapability(MatrixOperation.BINARY, ("binary", "binary_transform"), True),
+ OperationCapability(MatrixOperation.MASK, ("mask",), True),
+ OperationCapability(
+ MatrixOperation.SUBASSIGNMENT,
+ ("subassignment", "subassign", "[<-"),
+ True,
+ ),
+ OperationCapability(MatrixOperation.RANK, ("rank",), False),
+ OperationCapability(
+ MatrixOperation.MULTIPLY,
+ ("multiply", "matrix_multiply", "matmul"),
+ False,
+ ),
+ OperationCapability(
+ MatrixOperation.FRAGMENT,
+ ("fragment-derived", "fragment_matrix"),
+ False,
+ ),
+)
+
+
+_KNOWN_CLASSES = frozenset(
+ {
+ "DelayedArray",
+ "DelayedMatrix",
+ "HDF5Array",
+ "HDF5Matrix",
+ "H5SparseMatrix",
+ "H5ADMatrix",
+ "TENxMatrix",
+ "DelayedOp",
+ "DelayedUnaryOp",
+ "DelayedUnaryIsoOp",
+ "DelayedNaryOp",
+ "DelayedSubset",
+ "DelayedAperm",
+ "SeedDimPicker",
+ "DelayedAbind",
+ "SeedBinder",
+ "DelayedSetDimnames",
+ "DelayedUnaryIsoOpStack",
+ "DelayedUnaryIsoOpWithArgs",
+ "DelayedNaryIsoOp",
+ "DelayedSubassign",
+ "IterableMatrix",
+ "TransformedMatrix",
+ "TransformLog1p",
+ "TransformLog1pSlow",
+ "TransformExpm1",
+ "TransformExpm1Slow",
+ "TransformAbs",
+ "TransformNegate",
+ "TransformSign",
+ "TransformSqrt",
+ "TransformSquare",
+ "TransformPow",
+ "TransformMin",
+ "TransformMinByRow",
+ "TransformMinByCol",
+ "TransformBinarize",
+ "TransformRound",
+ "TransformScaleShift",
+ "SCTransformPearson",
+ "SCTransformPearsonSlow",
+ "SCTransformPearsonTranspose",
+ "SCTransformPearsonTransposeSlow",
+ "TransformLinearResidual",
+ "MatrixAddition",
+ "MatrixMask",
+ "MatrixRankTransform",
+ "MatrixSubset",
+ "RenameDims",
+ "RowBindMatrices",
+ "ColBindMatrices",
+ "ConvertMatrixType",
+ "PeakMatrix",
+ "TileMatrix",
+ "SubsetMatrix",
+ "TransposeMatrix",
+ "ApermMatrix",
+ "RowBindMatrix",
+ "ColumnBindMatrix",
+ "RenameMatrix",
+ "DtypeMatrix",
+ "UnaryMatrix",
+ "BinaryMatrix",
+ "MaskMatrix",
+ "SubassignmentMatrix",
+ "RankMatrix",
+ "MatrixMultiply",
+ "FragmentMatrix",
+ "BPCellsMatrix",
+ }
+)
+
+
+class MatrixOperationRegistry:
+ def __init__(self) -> None:
+ aliases: dict[str, MatrixOperation] = {}
+ for capability in _CAPABILITIES:
+ for alias in capability.aliases:
+ aliases[alias.lower()] = capability.operation
+ self._aliases = aliases
+ self.capabilities = {
+ capability.operation: capability for capability in _CAPABILITIES
+ }
+
+ def resolve(
+ self,
+ operation: Any,
+ *,
+ object_path: str,
+ class_name: str | None,
+ ) -> MatrixOperation:
+ if not isinstance(operation, str):
+ raise UnsupportedMatrixOperation(
+ object_path,
+ repr(operation),
+ class_name,
+ "custom functions are not executed",
+ )
+ normalized = operation.lower()
+ if normalized not in self._aliases:
+ raise UnsupportedMatrixOperation(
+ object_path, operation, class_name, "unknown operation"
+ )
+ return self._aliases[normalized]
+
+ def build(
+ self,
+ specification: Mapping[str, Any],
+ *,
+ object_path: str = "$",
+ source: MatrixSource | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> MatrixSource:
+ if not isinstance(specification, Mapping):
+ raise TypeError("matrix operation specification must be a mapping")
+ operation_value = specification.get("operation", specification.get("op"))
+ class_value = specification.get("className", specification.get("class"))
+ class_names: tuple[str, ...]
+ if class_value is None:
+ class_names = ()
+ elif isinstance(class_value, str):
+ class_names = (class_value,)
+ elif isinstance(class_value, Sequence) and not isinstance(class_value, bytes):
+ class_names = tuple(str(value) for value in class_value)
+ if not class_names:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ str(operation_value),
+ None,
+ "empty class vector",
+ )
+ else:
+ class_names = (str(class_value),)
+ class_name = class_names[0] if class_names else None
+ unknown_classes = [
+ value for value in class_names if value not in _KNOWN_CLASSES
+ ]
+ if unknown_classes:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ str(operation_value),
+ unknown_classes[0],
+ "unknown or custom class",
+ )
+ operation = self.resolve(
+ operation_value,
+ object_path=object_path,
+ class_name=class_name,
+ )
+ if operation in {MatrixOperation.ROW_BIND, MatrixOperation.COLUMN_BIND}:
+ inputs = specification.get("sources")
+ if not isinstance(inputs, Sequence) or isinstance(inputs, str | bytes):
+ raise MatrixSourceError(
+ f"bind operation at {object_path} requires a source sequence"
+ )
+ sources = tuple(inputs)
+ if not all(isinstance(item, MatrixSource) for item in sources):
+ raise TypeError("bind sources must implement MatrixSource")
+ return (
+ FeatureBindMatrixSource(sources, limits=limits)
+ if operation == MatrixOperation.ROW_BIND
+ else CellBindMatrixSource(sources, limits=limits)
+ )
+ primary = specification.get("source", source)
+ if operation == MatrixOperation.FRAGMENT:
+ return build_fragment_matrix_source(
+ specification,
+ object_path=object_path,
+ limits=limits,
+ )
+ if not isinstance(primary, MatrixSource):
+ raise MatrixSourceError(
+ f"matrix operation at {object_path} has no MatrixSource input"
+ )
+ if operation == MatrixOperation.SUBSET:
+ return MappedMatrixSource(
+ primary,
+ feature_indices=specification.get(
+ "featureIndices",
+ specification.get(
+ "featureIndexes",
+ specification.get(
+ "feature_indices",
+ specification.get(
+ "feature_indexes", specification.get("rows")
+ ),
+ ),
+ ),
+ ),
+ cell_indices=specification.get(
+ "cellIndices",
+ specification.get(
+ "cellIndexes",
+ specification.get(
+ "cell_indices",
+ specification.get(
+ "cell_indexes", specification.get("columns")
+ ),
+ ),
+ ),
+ ),
+ limits=limits,
+ )
+ if operation in {MatrixOperation.TRANSPOSE, MatrixOperation.APERM}:
+ permutation = specification.get("permutation", (1, 0))
+ normalized_permutation = tuple(int(value) for value in permutation)
+ if normalized_permutation in {(0, 1), (1, 2)}:
+ return primary
+ if normalized_permutation not in {(1, 0), (2, 1)}:
+ raise UnsupportedMatrixOperation(
+ object_path,
+ operation.value,
+ class_name,
+ f"2D permutation {normalized_permutation!r} is invalid",
+ )
+ return TransposeMatrixSource(primary, limits=limits)
+ if operation == MatrixOperation.RENAME:
+ return RenamedMatrixSource(
+ primary,
+ row_names=specification.get("rowNames", specification.get("row_names")),
+ column_names=specification.get(
+ "columnNames", specification.get("column_names")
+ ),
+ limits=limits,
+ )
+ if operation == MatrixOperation.DTYPE:
+ if "dtype" not in specification:
+ raise MatrixSourceError(
+ f"dtype operation at {object_path} has no dtype"
+ )
+ return DtypeMatrixSource(primary, specification["dtype"], limits=limits)
+ if operation == MatrixOperation.UNARY:
+ function = specification.get("function", specification.get("name"))
+ return UnaryTransformMatrixSource(
+ primary,
+ function,
+ dtype=specification.get("dtype"),
+ parameter=specification.get("parameter"),
+ object_path=object_path,
+ class_name=class_name,
+ limits=limits,
+ )
+ if operation == MatrixOperation.BINARY:
+ if "right" not in specification:
+ raise MatrixSourceError(
+ f"binary operation at {object_path} has no right operand"
+ )
+ function = specification.get("function", specification.get("name"))
+ return BinaryTransformMatrixSource(
+ primary,
+ specification["right"],
+ function,
+ dtype=specification.get("dtype"),
+ reverse=bool(specification.get("reverse", False)),
+ object_path=object_path,
+ class_name=class_name,
+ limits=limits,
+ )
+ if operation == MatrixOperation.MASK:
+ mask = specification.get("mask")
+ if not isinstance(mask, MatrixSource):
+ raise MatrixSourceError(
+ f"mask operation at {object_path} has no MatrixSource mask"
+ )
+ return MaskMatrixSource(
+ primary,
+ mask,
+ fill_value=specification.get(
+ "fillValue", specification.get("fill_value", 0)
+ ),
+ keep_nonzero=specification.get(
+ "keepNonzero",
+ specification.get(
+ "keep_nonzero",
+ (
+ bool(specification.get("invert", False))
+ if class_name == "MatrixMask"
+ else True
+ ),
+ ),
+ ),
+ limits=limits,
+ )
+ if operation == MatrixOperation.SUBASSIGNMENT:
+ raw_assignments = specification.get("assignments")
+ if not isinstance(raw_assignments, Sequence) or isinstance(
+ raw_assignments, str | bytes
+ ):
+ raise MatrixSourceError(
+ f"subassignment at {object_path} requires assignments"
+ )
+ assignments: list[Subassignment] = []
+ for index, assignment in enumerate(raw_assignments):
+ if isinstance(assignment, Subassignment):
+ assignments.append(assignment)
+ continue
+ if not isinstance(assignment, Mapping):
+ raise TypeError(
+ f"subassignment {index} at {object_path} must be a mapping"
+ )
+ try:
+ assignments.append(
+ Subassignment(
+ assignment.get(
+ "featureIndices", assignment.get("feature_indices")
+ ),
+ assignment.get(
+ "cellIndices", assignment.get("cell_indices")
+ ),
+ assignment.get("value"),
+ )
+ )
+ except TypeError as error:
+ raise MatrixSourceError(
+ f"subassignment {index} at {object_path} is incomplete"
+ ) from error
+ return DelayedSubassignmentMatrixSource(primary, assignments, limits=limits)
+ if operation == MatrixOperation.RANK:
+ return RankMatrixSource(
+ primary,
+ axis=str(specification.get("axis", "column")),
+ object_path=object_path,
+ class_name=class_name,
+ limits=limits,
+ )
+ if operation == MatrixOperation.MULTIPLY:
+ right = specification.get("right")
+ if right is not None and not isinstance(right, MatrixSource):
+ raise TypeError(
+ "matrix multiplication right operand must implement MatrixSource"
+ )
+ return MatrixMultiplySource(
+ primary,
+ right=right,
+ shape=specification.get(
+ "shape",
+ specification.get("Dim", specification.get("dim")),
+ ),
+ row_names=specification.get("rowNames", specification.get("row_names")),
+ column_names=specification.get(
+ "columnNames", specification.get("column_names")
+ ),
+ object_path=object_path,
+ class_name=class_name,
+ limits=limits,
+ )
+ raise AssertionError(f"unhandled matrix operation {operation}")
+
+
+DEFAULT_OPERATION_REGISTRY = MatrixOperationRegistry()
+
+
+def build_matrix_operation(
+ specification: Mapping[str, Any],
+ *,
+ object_path: str = "$",
+ source: MatrixSource | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+) -> MatrixSource:
+ return DEFAULT_OPERATION_REGISTRY.build(
+ specification,
+ object_path=object_path,
+ source=source,
+ limits=limits,
+ )
diff --git a/scarf/readers/_seurat/paths.py b/scarf/readers/_seurat/paths.py
new file mode 100644
index 00000000..9c60f2ed
--- /dev/null
+++ b/scarf/readers/_seurat/paths.py
@@ -0,0 +1,415 @@
+import os
+from collections.abc import Mapping, Sequence
+from pathlib import Path, PurePath, PurePosixPath, PureWindowsPath
+from typing import Any
+
+import h5py
+import numpy as np
+
+from .errors import MatrixSourceError, ResourceLimitError, UnsafeSidecarError
+from .sources import DEFAULT_LIMITS, SourceLimits
+
+
+def _is_within(path: Path, root: Path) -> bool:
+ return path == root or path.is_relative_to(root)
+
+
+def _path_text(value: str | os.PathLike[str]) -> str:
+ text = os.fspath(value)
+ if not isinstance(text, str):
+ raise TypeError("sidecar paths must be text paths")
+ if "\x00" in text:
+ raise UnsafeSidecarError("sidecar path contains a NUL character")
+ return text
+
+
+def _serialized_absolute_path(text: str) -> PurePath | None:
+ posix = PurePosixPath(text)
+ if posix.is_absolute():
+ return posix
+ windows = PureWindowsPath(text)
+ if windows.is_absolute():
+ return windows
+ return None
+
+
+class SidecarPathResolver:
+ def __init__(
+ self,
+ rds_path: str | os.PathLike[str],
+ *,
+ absolute_prefix_remaps: Mapping[str | os.PathLike[str], str | os.PathLike[str]]
+ | None = None,
+ require_exists: bool = True,
+ ) -> None:
+ rds = Path(_path_text(rds_path)).expanduser().resolve(strict=False)
+ self.rdsPath = rds
+ self.anchor = rds.parent
+ self.requireExists = bool(require_exists)
+ remaps: list[tuple[PurePath, Path]] = []
+ for source, destination in (absolute_prefix_remaps or {}).items():
+ source_text = _path_text(source)
+ source_path = _serialized_absolute_path(source_text)
+ destination_path = Path(_path_text(destination)).expanduser()
+ if source_path is None or not destination_path.is_absolute():
+ raise ValueError("absolute path remaps require absolute prefixes")
+ destination_root = destination_path.resolve(strict=False)
+ remaps.append((source_path, destination_root))
+ self.absolutePrefixRemaps = tuple(
+ sorted(remaps, key=lambda item: len(item[0].parts), reverse=True)
+ )
+
+ def resolve(
+ self,
+ sidecar_path: str | os.PathLike[str],
+ *,
+ expect: str = "any",
+ ) -> Path:
+ text = _path_text(sidecar_path)
+ serialized_absolute = _serialized_absolute_path(text)
+ if serialized_absolute is not None:
+ candidate, root = self._remap_absolute(serialized_absolute)
+ else:
+ windows_relative = PureWindowsPath(text)
+ if windows_relative.drive:
+ raise UnsafeSidecarError(
+ f"drive-relative sidecar path {text!r} is rejected"
+ )
+ raw = (
+ Path(*windows_relative.parts)
+ if "\\" in text
+ else Path(text).expanduser()
+ )
+ candidate = self.anchor / raw
+ root = self.anchor
+ resolved = candidate.resolve(strict=False)
+ resolved_root = root.resolve(strict=False)
+ if not _is_within(resolved, resolved_root):
+ raise UnsafeSidecarError(
+ f"sidecar path {text!r} escapes allowed root {str(resolved_root)!r}"
+ )
+ if self.requireExists and not resolved.exists():
+ raise FileNotFoundError(resolved)
+ if expect == "file" and resolved.exists() and not resolved.is_file():
+ raise UnsafeSidecarError(f"sidecar path {resolved} is not a regular file")
+ if expect == "directory" and resolved.exists() and not resolved.is_dir():
+ raise UnsafeSidecarError(f"sidecar path {resolved} is not a directory")
+ if expect not in {"any", "file", "directory"}:
+ raise ValueError("expect must be 'any', 'file', or 'directory'")
+ return resolved
+
+ def _remap_absolute(self, path: PurePath) -> tuple[Path, Path]:
+ for source_prefix, destination_prefix in self.absolutePrefixRemaps:
+ if type(path) is not type(source_prefix):
+ continue
+ if path == source_prefix or path.is_relative_to(source_prefix):
+ relative = path.relative_to(source_prefix)
+ return (
+ destination_prefix.joinpath(*relative.parts),
+ destination_prefix,
+ )
+ raise UnsafeSidecarError(
+ f"absolute sidecar path {str(path)!r} has no explicit prefix remap"
+ )
+
+
+def resolve_sidecar_path(
+ sidecar_path: str | os.PathLike[str],
+ rds_path: str | os.PathLike[str],
+ *,
+ absolute_prefix_remaps: Mapping[str | os.PathLike[str], str | os.PathLike[str]]
+ | None = None,
+ require_exists: bool = True,
+ expect: str = "any",
+) -> Path:
+ return SidecarPathResolver(
+ rds_path,
+ absolute_prefix_remaps=absolute_prefix_remaps,
+ require_exists=require_exists,
+ ).resolve(sidecar_path, expect=expect)
+
+
+def require_filesystem_path(value: Any, description: str = "HDF5 source") -> Path:
+ if isinstance(value, h5py.File | h5py.Group | h5py.Dataset):
+ raise UnsafeSidecarError(
+ f"{description} must be a path; live HDF5 handles are rejected"
+ )
+ if not isinstance(value, str | os.PathLike):
+ raise TypeError(f"{description} must be a filesystem path")
+ path = Path(_path_text(value)).expanduser().resolve(strict=False)
+ if not path.exists():
+ raise FileNotFoundError(path)
+ if not path.is_file():
+ raise UnsafeSidecarError(f"{description} path {path} is not a regular file")
+ return path
+
+
+def _has_reference_dtype(dtype: np.dtype[Any]) -> bool:
+ if dtype.fields is not None:
+ return any(_has_reference_dtype(field[0]) for field in dtype.fields.values())
+ if dtype.subdtype is not None:
+ return _has_reference_dtype(dtype.subdtype[0])
+ try:
+ return h5py.check_dtype(ref=dtype) is not None
+ except TypeError:
+ return False
+
+
+def _validate_attribute_value(
+ value: Any,
+ object_path: str,
+ limits: SourceLimits,
+) -> None:
+ if isinstance(value, h5py.Reference | h5py.RegionReference):
+ raise UnsafeSidecarError(f"HDF5 references are rejected at {object_path}")
+ array = np.asarray(value)
+ payload_bytes = int(array.nbytes)
+ if array.dtype.kind in {"O", "S", "U"}:
+ for item in array.reshape(-1):
+ if isinstance(item, bytes | np.bytes_):
+ payload_bytes += len(bytes(item))
+ elif isinstance(item, str | np.str_):
+ payload_bytes += len(str(item).encode("utf-8"))
+ if payload_bytes > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"HDF5 attribute {object_path} exceeds "
+ f"maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ if _has_reference_dtype(array.dtype):
+ raise UnsafeSidecarError(f"HDF5 reference dtype is rejected at {object_path}")
+
+
+def _validate_filters(dataset: h5py.Dataset) -> None:
+ property_list = dataset.id.get_create_plist()
+ optional_flag = int(getattr(h5py.h5z, "FLAG_OPTIONAL", 1))
+ decode_flag = int(getattr(h5py.h5z, "FILTER_CONFIG_DECODE_ENABLED", 2))
+ for index in range(property_list.get_nfilters()):
+ filter_id, flags, _values, name = property_list.get_filter(index)
+ available = bool(h5py.h5z.filter_avail(filter_id))
+ decodable = False
+ if available:
+ try:
+ decodable = bool(h5py.h5z.get_filter_info(filter_id) & decode_flag)
+ except RuntimeError:
+ decodable = False
+ if not available or not decodable:
+ optional = bool(flags & optional_flag)
+ if not optional:
+ display = (
+ name.decode("utf-8", errors="replace")
+ if isinstance(name, bytes)
+ else str(name)
+ )
+ raise UnsafeSidecarError(
+ f"HDF5 dataset {dataset.name!r} requires unavailable "
+ f"filter {filter_id} ({display})"
+ )
+
+
+def _validate_dataset(dataset: h5py.Dataset) -> None:
+ if dataset.is_virtual:
+ raise UnsafeSidecarError(f"HDF5 virtual dataset {dataset.name!r} is rejected")
+ property_list = dataset.id.get_create_plist()
+ if property_list.get_external_count():
+ raise UnsafeSidecarError(
+ f"HDF5 external storage at {dataset.name!r} is rejected"
+ )
+ if _has_reference_dtype(dataset.dtype):
+ raise UnsafeSidecarError(f"HDF5 reference dataset {dataset.name!r} is rejected")
+ _validate_filters(dataset)
+
+
+def _validate_group_links(
+ group: h5py.Group,
+ visited: set[int],
+) -> None:
+ object_id = hash(group.id)
+ if object_id in visited:
+ return
+ visited.add(object_id)
+ for key in group.keys():
+ link = group.get(key, getlink=True)
+ link_path = f"{group.name.rstrip('/')}/{key}"
+ if isinstance(link, h5py.ExternalLink):
+ raise UnsafeSidecarError(f"HDF5 external link at {link_path!r} is rejected")
+ if not isinstance(link, h5py.HardLink):
+ continue
+ node = group.get(key, getlink=False)
+ if isinstance(node, h5py.Group):
+ _validate_group_links(node, visited)
+
+
+def validate_hdf5_file(
+ path: str | os.PathLike[str] | Any,
+ *,
+ limits: SourceLimits = DEFAULT_LIMITS,
+) -> Path:
+ resolved = require_filesystem_path(path)
+ try:
+ with h5py.File(resolved, mode="r") as handle:
+ _validate_group_links(handle, set())
+ metadata_bytes = 0
+
+ def inspect_attributes(node: h5py.Group | h5py.Dataset) -> None:
+ nonlocal metadata_bytes
+ for key in node.attrs:
+ attribute_id = node.attrs.get_id(key)
+ try:
+ storage_size = int(attribute_id.get_storage_size())
+ finally:
+ attribute_id.close()
+ metadata_bytes += storage_size + len(key.encode("utf-8")) + 8
+ if metadata_bytes > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ "HDF5 attributes exceed "
+ f"maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ _validate_attribute_value(
+ node.attrs[key],
+ f"{node.name}@{key}",
+ limits,
+ )
+
+ def inspect(name: str, node: h5py.Group | h5py.Dataset) -> None:
+ nonlocal metadata_bytes
+ metadata_bytes += len(name.encode("utf-8")) + 8
+ if metadata_bytes > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ "HDF5 object metadata exceeds "
+ f"maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ inspect_attributes(node)
+ if isinstance(node, h5py.Dataset):
+ _validate_dataset(node)
+
+ handle.visititems(inspect)
+ inspect_attributes(handle)
+ except OSError as error:
+ raise MatrixSourceError(f"cannot open HDF5 sidecar {resolved}") from error
+ return resolved
+
+
+def _decode_hdf5_text(value: Any, object_path: str) -> str:
+ if isinstance(value, bytes | np.bytes_):
+ try:
+ result = bytes(value).decode("utf-8")
+ except UnicodeDecodeError as error:
+ raise MatrixSourceError(
+ f"HDF5 text at {object_path} is not valid UTF-8"
+ ) from error
+ elif isinstance(value, str | np.str_):
+ result = str(value)
+ else:
+ raise MatrixSourceError(f"HDF5 metadata at {object_path} must contain strings")
+ if "\x00" in result:
+ raise MatrixSourceError(f"HDF5 text at {object_path} contains NUL")
+ return result
+
+
+def read_hdf5_names(
+ handle: h5py.File,
+ dataset_path: str | None,
+ expected_length: int,
+ *,
+ required: bool = False,
+ limits: SourceLimits = DEFAULT_LIMITS,
+) -> tuple[str, ...] | None:
+ if dataset_path is None:
+ return None
+ normalized_path = "/" + dataset_path.strip("/")
+ if normalized_path not in handle:
+ if required:
+ raise MatrixSourceError(
+ f"HDF5 names dataset {normalized_path!r} is missing"
+ )
+ return None
+ node = handle[normalized_path]
+ if not isinstance(node, h5py.Dataset):
+ raise MatrixSourceError(f"HDF5 names path {normalized_path!r} is not a dataset")
+ if node.ndim != 1 or int(node.shape[0]) != expected_length:
+ raise MatrixSourceError(
+ f"HDF5 names dataset {normalized_path!r} has shape {node.shape}; "
+ f"expected ({expected_length},)"
+ )
+ if _has_reference_dtype(node.dtype):
+ raise UnsafeSidecarError(
+ f"HDF5 reference names at {normalized_path!r} are rejected"
+ )
+ if expected_length * 8 > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"HDF5 names exceed maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ string_info = h5py.check_string_dtype(node.dtype)
+ if string_info is None:
+ raise MatrixSourceError(
+ f"HDF5 names dataset {normalized_path!r} must contain strings"
+ )
+ if string_info.length is not None:
+ declared_bytes = expected_length * (string_info.length + 8)
+ if declared_bytes > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"HDF5 names exceed maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ values: list[str] = []
+ size = 0
+ chunk = 4096
+ for start in range(0, expected_length, chunk):
+ stop = min(expected_length, start + chunk)
+ for value in np.asarray(node[start:stop]).reshape(-1):
+ decoded = _decode_hdf5_text(value, normalized_path)
+ size += len(decoded.encode("utf-8")) + 8
+ if size > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"HDF5 names exceed maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ values.append(decoded)
+ return tuple(values)
+
+
+def read_hdf5_shape(
+ value: Any,
+ object_path: str,
+) -> tuple[int, int]:
+ array = np.asarray(value)
+ if array.ndim != 1 or array.size != 2:
+ raise MatrixSourceError(
+ f"HDF5 shape at {object_path} must contain two integers"
+ )
+ if not np.issubdtype(array.dtype, np.integer):
+ raise TypeError(f"HDF5 shape at {object_path} must contain integers")
+ shape = (int(array[0]), int(array[1]))
+ if shape[0] < 0 or shape[1] < 0:
+ raise MatrixSourceError(f"HDF5 shape at {object_path} cannot contain negatives")
+ return shape
+
+
+def require_hdf5_group(
+ handle: h5py.File,
+ group_path: str,
+) -> h5py.Group:
+ normalized = "/" + group_path.strip("/")
+ if normalized not in handle:
+ raise MatrixSourceError(f"HDF5 group {normalized!r} is missing")
+ node = handle[normalized]
+ if not isinstance(node, h5py.Group):
+ raise MatrixSourceError(f"HDF5 path {normalized!r} is not a group")
+ return node
+
+
+def require_hdf5_datasets(
+ group: h5py.Group,
+ names: Sequence[str],
+) -> dict[str, h5py.Dataset]:
+ result: dict[str, h5py.Dataset] = {}
+ for name in names:
+ if name not in group:
+ raise MatrixSourceError(
+ f"HDF5 dataset {group.name.rstrip('/')}/{name} is missing"
+ )
+ node = group[name]
+ if not isinstance(node, h5py.Dataset):
+ raise MatrixSourceError(
+ f"HDF5 path {group.name.rstrip('/')}/{name} is not a dataset"
+ )
+ result[name] = node
+ return result
diff --git a/scarf/readers/_seurat/sources.py b/scarf/readers/_seurat/sources.py
new file mode 100644
index 00000000..2a7e4e75
--- /dev/null
+++ b/scarf/readers/_seurat/sources.py
@@ -0,0 +1,1412 @@
+from collections.abc import Sequence
+from dataclasses import dataclass
+from typing import Any, Protocol, cast, runtime_checkable
+
+import numpy as np
+from numpy.typing import DTypeLike, NDArray
+from scipy.sparse import (
+ coo_matrix,
+ csr_matrix,
+ hstack,
+ spmatrix,
+ vstack,
+)
+
+from .errors import MatrixSourceError, ResourceLimitError
+
+
+type MatrixBlock = NDArray[Any] | coo_matrix | csr_matrix
+
+
+@dataclass(frozen=True)
+class MemoryEstimate:
+ residentBytes: int = 0
+ workingBytes: int = 0
+ outputBytes: int = 0
+
+ @property
+ def peakBytes(self) -> int:
+ return self.residentBytes + self.workingBytes + self.outputBytes
+
+ @property
+ def resident_bytes(self) -> int:
+ return self.residentBytes
+
+ @property
+ def working_bytes(self) -> int:
+ return self.workingBytes
+
+ @property
+ def output_bytes(self) -> int:
+ return self.outputBytes
+
+ @property
+ def peak_bytes(self) -> int:
+ return self.peakBytes
+
+
+@dataclass(frozen=True)
+class SourceLimits:
+ maxFeatures: int = np.iinfo(np.int32).max
+ maxCells: int = np.iinfo(np.int32).max
+ maxNnz: int = np.iinfo(np.int64).max
+ maxBlockBytes: int = 512 * 1024 * 1024
+ maxMetadataBytes: int = 256 * 1024 * 1024
+ tileCells: int = 1024
+ compressedChunkNnz: int = 4 * 1024 * 1024
+
+ def __post_init__(self) -> None:
+ for field_name in (
+ "maxFeatures",
+ "maxCells",
+ "maxNnz",
+ "maxBlockBytes",
+ "maxMetadataBytes",
+ "tileCells",
+ "compressedChunkNnz",
+ ):
+ if getattr(self, field_name) <= 0:
+ raise ValueError(f"{field_name} must be positive")
+
+ @property
+ def max_features(self) -> int:
+ return self.maxFeatures
+
+ @property
+ def max_cells(self) -> int:
+ return self.maxCells
+
+ @property
+ def max_nnz(self) -> int:
+ return self.maxNnz
+
+ @property
+ def max_block_bytes(self) -> int:
+ return self.maxBlockBytes
+
+ @property
+ def max_metadata_bytes(self) -> int:
+ return self.maxMetadataBytes
+
+ @property
+ def tile_cells(self) -> int:
+ return self.tileCells
+
+ @property
+ def compressed_chunk_nnz(self) -> int:
+ return self.compressedChunkNnz
+
+
+DEFAULT_LIMITS = SourceLimits()
+
+
+@runtime_checkable
+class MatrixSource(Protocol):
+ @property
+ def shape(self) -> tuple[int, int]: ...
+
+ @property
+ def dtype(self) -> np.dtype[Any]: ...
+
+ @property
+ def row_names(self) -> tuple[str, ...] | None: ...
+
+ @property
+ def column_names(self) -> tuple[str, ...] | None: ...
+
+ @property
+ def is_sparse(self) -> bool: ...
+
+ @property
+ def zero_preserving(self) -> bool: ...
+
+ @property
+ def resident_bytes(self) -> int: ...
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock: ...
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate: ...
+
+
+def _metadata_bytes(names: tuple[str, ...] | None) -> int:
+ if names is None:
+ return 0
+ return sum(len(value.encode("utf-8")) + 8 for value in names)
+
+
+def _normalize_names(
+ names: Sequence[str | bytes] | NDArray[Any] | None,
+ length: int,
+ axis: str,
+ limits: SourceLimits,
+) -> tuple[str, ...] | None:
+ if names is None:
+ return None
+ if isinstance(names, str | bytes):
+ raise TypeError(f"{axis} names must be a sequence")
+ values: list[str] = []
+ size = 0
+ for value in names:
+ if isinstance(value, bytes | np.bytes_):
+ try:
+ decoded = bytes(value).decode("utf-8")
+ except UnicodeDecodeError as error:
+ raise MatrixSourceError(
+ f"{axis} names contain invalid UTF-8"
+ ) from error
+ elif isinstance(value, str | np.str_):
+ decoded = str(value)
+ else:
+ raise TypeError(f"{axis} names must contain only strings")
+ if "\x00" in decoded:
+ raise MatrixSourceError(f"{axis} names contain a NUL character")
+ size += len(decoded.encode("utf-8")) + 8
+ if size > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"{axis} names exceed maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ values.append(decoded)
+ if len(values) != length:
+ raise MatrixSourceError(
+ f"{axis} names have length {len(values)}; expected {length}"
+ )
+ return tuple(values)
+
+
+def _validate_shape(shape: Sequence[int], limits: SourceLimits) -> tuple[int, int]:
+ if len(shape) != 2:
+ raise MatrixSourceError("matrix shape must have exactly two dimensions")
+ n_features, n_cells = (int(shape[0]), int(shape[1]))
+ if n_features < 0 or n_cells < 0:
+ raise MatrixSourceError("matrix dimensions cannot be negative")
+ if n_features > limits.maxFeatures:
+ raise ResourceLimitError(
+ f"feature count {n_features} exceeds maxFeatures={limits.maxFeatures}"
+ )
+ if n_cells > limits.maxCells:
+ raise ResourceLimitError(
+ f"cell count {n_cells} exceeds maxCells={limits.maxCells}"
+ )
+ return n_features, n_cells
+
+
+def _validate_window(start: int, stop: int, n_cells: int) -> tuple[int, int]:
+ if isinstance(start, bool) or isinstance(stop, bool):
+ raise TypeError("cell bounds must be integers")
+ start = int(start)
+ stop = int(stop)
+ if start < 0 or stop < start or stop > n_cells:
+ raise IndexError(f"cell window [{start}, {stop}) is outside [0, {n_cells})")
+ return start, stop
+
+
+def _array_shape(values: Any) -> tuple[int, ...]:
+ shape = getattr(values, "shape", None)
+ if shape is None:
+ try:
+ return (len(values),)
+ except TypeError as error:
+ raise TypeError("array-like object must expose shape or len") from error
+ return tuple(int(value) for value in shape)
+
+
+def _array_dtype(values: Any) -> np.dtype[Any]:
+ dtype = getattr(values, "dtype", None)
+ if dtype is None:
+ return np.asarray(values).dtype
+ return cast(np.dtype[Any], np.dtype(dtype))
+
+
+def _array_resident_bytes(values: Any) -> int:
+ if isinstance(values, np.ndarray):
+ return int(values.nbytes)
+ return 0
+
+
+def _read_1d(
+ values: Any,
+ start: int,
+ stop: int,
+ *,
+ dtype: DTypeLike | None = None,
+) -> NDArray[Any]:
+ try:
+ result = np.asarray(values[start:stop])
+ except (IndexError, TypeError, ValueError) as error:
+ raise MatrixSourceError(
+ "array-like object does not support bounded slicing"
+ ) from error
+ if result.ndim != 1:
+ result = result.reshape(-1)
+ if result.size != stop - start:
+ raise MatrixSourceError(
+ f"bounded array read returned {result.size} values; expected {stop - start}"
+ )
+ if dtype is not None:
+ result = result.astype(dtype, copy=False)
+ return result
+
+
+def _normalize_indexes(
+ indexes: Sequence[int] | NDArray[Any],
+ upper_bound: int,
+ axis: str,
+) -> NDArray[np.int64]:
+ values = np.asarray(indexes)
+ if values.ndim != 1:
+ raise MatrixSourceError(f"{axis} indexes must be one-dimensional")
+ if not np.issubdtype(values.dtype, np.integer):
+ raise TypeError(f"{axis} indexes must contain integers")
+ values = values.astype(np.int64, copy=False)
+ if values.size and (np.any(values < 0) or np.any(values >= upper_bound)):
+ raise IndexError(f"{axis} indexes contain an out-of-range value")
+ return values
+
+
+def _block_to_csr(
+ block: MatrixBlock | spmatrix,
+ *,
+ dtype: DTypeLike | None = None,
+) -> csr_matrix:
+ if isinstance(block, spmatrix):
+ result = block.tocsr(copy=False)
+ if dtype is not None:
+ result = result.astype(dtype, copy=False)
+ return cast(csr_matrix, result)
+ values = np.asarray(block)
+ if values.ndim != 2:
+ raise MatrixSourceError("matrix block must be two-dimensional")
+ if dtype is not None:
+ values = values.astype(dtype, copy=False)
+ return csr_matrix(values)
+
+
+def _block_to_dense(
+ block: MatrixBlock | spmatrix,
+ *,
+ dtype: DTypeLike | None = None,
+) -> NDArray[Any]:
+ values = block.toarray() if isinstance(block, spmatrix) else np.asarray(block)
+ if values.ndim != 2:
+ raise MatrixSourceError("matrix block must be two-dimensional")
+ if dtype is not None:
+ values = values.astype(dtype, copy=False)
+ return values
+
+
+def _empty_block(
+ rows: int,
+ columns: int,
+ dtype: DTypeLike,
+ sparse: bool,
+) -> MatrixBlock:
+ if sparse:
+ return cast(MatrixBlock, csr_matrix((rows, columns), dtype=dtype))
+ return np.empty((rows, columns), dtype=dtype)
+
+
+def _read_selected_cells(
+ source: MatrixSource,
+ indexes: NDArray[np.int64],
+) -> MatrixBlock:
+ if indexes.size == 0:
+ return _empty_block(0, source.shape[0], source.dtype, source.is_sparse)
+ pieces: list[MatrixBlock] = []
+ run_start = 0
+ for position in range(1, indexes.size + 1):
+ run_finished = (
+ position == indexes.size or indexes[position] != indexes[position - 1] + 1
+ )
+ if not run_finished:
+ continue
+ source_start = int(indexes[run_start])
+ source_stop = int(indexes[position - 1]) + 1
+ pieces.append(source.read_cells(source_start, source_stop))
+ run_start = position
+ if source.is_sparse:
+ return cast(
+ MatrixBlock,
+ vstack(
+ [_block_to_csr(piece, dtype=source.dtype) for piece in pieces],
+ format="csr",
+ dtype=source.dtype,
+ ),
+ )
+ return np.vstack(
+ [_block_to_dense(piece, dtype=source.dtype) for piece in pieces],
+ dtype=source.dtype,
+ )
+
+
+class BaseMatrixSource:
+ def __init__(
+ self,
+ shape: Sequence[int],
+ dtype: DTypeLike,
+ *,
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ is_sparse: bool,
+ zero_preserving: bool = True,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self._limits = limits
+ self._shape = _validate_shape(shape, limits)
+ self._dtype: np.dtype[Any] = cast(np.dtype[Any], np.dtype(dtype))
+ if self._dtype.hasobject or self._dtype.fields is not None:
+ raise TypeError(f"matrix dtype {self._dtype} is not numeric")
+ if self._dtype.kind not in "biufc":
+ raise TypeError(f"matrix dtype {self._dtype} is not numeric")
+ self._row_names = _normalize_names(row_names, self._shape[0], "row", limits)
+ self._column_names = _normalize_names(
+ column_names, self._shape[1], "column", limits
+ )
+ metadata_size = _metadata_bytes(self._row_names) + _metadata_bytes(
+ self._column_names
+ )
+ if metadata_size > limits.maxMetadataBytes:
+ raise ResourceLimitError(
+ f"matrix names exceed maxMetadataBytes={limits.maxMetadataBytes}"
+ )
+ self._is_sparse = bool(is_sparse)
+ self._zero_preserving = bool(zero_preserving)
+
+ @property
+ def shape(self) -> tuple[int, int]:
+ return self._shape
+
+ @property
+ def dtype(self) -> np.dtype[Any]:
+ return self._dtype
+
+ @property
+ def row_names(self) -> tuple[str, ...] | None:
+ return self._row_names
+
+ @property
+ def column_names(self) -> tuple[str, ...] | None:
+ return self._column_names
+
+ @property
+ def rowNames(self) -> tuple[str, ...] | None:
+ return self.row_names
+
+ @property
+ def columnNames(self) -> tuple[str, ...] | None:
+ return self.column_names
+
+ @property
+ def is_sparse(self) -> bool:
+ return self._is_sparse
+
+ @property
+ def sparse(self) -> bool:
+ return self.is_sparse
+
+ @property
+ def zero_preserving(self) -> bool:
+ return self._zero_preserving
+
+ @property
+ def zeroPreserving(self) -> bool:
+ return self.zero_preserving
+
+ @property
+ def n_features(self) -> int:
+ return self.shape[0]
+
+ @property
+ def n_cells(self) -> int:
+ return self.shape[1]
+
+ @property
+ def resident_bytes(self) -> int:
+ return _metadata_bytes(self.row_names) + _metadata_bytes(self.column_names)
+
+ @property
+ def residentBytes(self) -> int:
+ return self.resident_bytes
+
+ def _window(self, start: int, stop: int) -> tuple[int, int]:
+ return _validate_window(start, stop, self.n_cells)
+
+ def _admit(self, estimate: MemoryEstimate) -> None:
+ if estimate.workingBytes + estimate.outputBytes > self._limits.maxBlockBytes:
+ raise ResourceLimitError(
+ "matrix block needs "
+ f"{estimate.workingBytes + estimate.outputBytes} bytes; "
+ f"maxBlockBytes={self._limits.maxBlockBytes}"
+ )
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ raise NotImplementedError
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ raise NotImplementedError
+
+ def memory_estimate(self, start: int, stop: int) -> MemoryEstimate:
+ return self.estimate_read_memory(start, stop)
+
+ def estimate_read_bytes(self, start: int, stop: int) -> int:
+ return self.estimate_read_memory(start, stop).peakBytes
+
+ def estimate_memory(self, start: int, stop: int) -> MemoryEstimate:
+ return self.estimate_read_memory(start, stop)
+
+ def estimated_peak_bytes(self, start: int, stop: int) -> int:
+ return self.estimate_read_bytes(start, stop)
+
+
+class DenseMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ values: Any,
+ shape: Sequence[int] | None = None,
+ *,
+ dtype: DTypeLike | None = None,
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ normalized_values = (
+ np.asarray(values)
+ if isinstance(values, Sequence) and not isinstance(values, str | bytes)
+ else values
+ )
+ source_shape = _array_shape(normalized_values)
+ if len(source_shape) == 1:
+ if shape is None:
+ raise MatrixSourceError(
+ "shape is required for a flat R column-major vector"
+ )
+ logical_shape = _validate_shape(shape, limits)
+ if source_shape[0] != logical_shape[0] * logical_shape[1]:
+ raise MatrixSourceError(
+ f"flat dense source has {source_shape[0]} values; "
+ f"expected {logical_shape[0] * logical_shape[1]}"
+ )
+ self._flat = True
+ elif len(source_shape) == 2:
+ logical_shape = _validate_shape(
+ source_shape if shape is None else shape, limits
+ )
+ if source_shape != logical_shape:
+ raise MatrixSourceError(
+ f"dense source shape {source_shape} does not match {logical_shape}"
+ )
+ self._flat = False
+ else:
+ raise MatrixSourceError(
+ "dense matrix source must be one or two-dimensional"
+ )
+ source_dtype = (
+ _array_dtype(normalized_values) if dtype is None else np.dtype(dtype)
+ )
+ self._values = normalized_values
+ super().__init__(
+ logical_shape,
+ source_dtype,
+ row_names=row_names,
+ column_names=column_names,
+ is_sparse=False,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return super().resident_bytes + _array_resident_bytes(self._values)
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(self.resident_bytes, output, output)
+
+ def read_cells(self, start: int, stop: int) -> NDArray[Any]:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ if self._flat:
+ flat = _read_1d(
+ self._values,
+ start * self.n_features,
+ stop * self.n_features,
+ dtype=self.dtype,
+ )
+ return np.ascontiguousarray(flat.reshape(stop - start, self.n_features))
+ try:
+ feature_by_cell = np.asarray(self._values[:, start:stop])
+ except (IndexError, TypeError, ValueError) as error:
+ raise MatrixSourceError(
+ "two-dimensional source does not support bounded column slicing"
+ ) from error
+ expected_shape = (self.n_features, stop - start)
+ if feature_by_cell.shape != expected_shape:
+ raise MatrixSourceError(
+ f"dense block has shape {feature_by_cell.shape}; "
+ f"expected {expected_shape}"
+ )
+ return np.ascontiguousarray(feature_by_cell.T, dtype=self.dtype)
+
+
+RColumnMajorMatrixSource = DenseMatrixSource
+InMemoryDenseMatrixSource = DenseMatrixSource
+LazyDenseMatrixSource = DenseMatrixSource
+
+
+class CscMatrixSource(BaseMatrixSource):
+ _SUPPORTED_CLASSES = frozenset(
+ {
+ "dgCMatrix",
+ "lgCMatrix",
+ "ngCMatrix",
+ "igCMatrix",
+ "CsparseMatrix",
+ "CSC",
+ }
+ )
+
+ def __init__(
+ self,
+ x: Any | None,
+ i: Any,
+ p: Any,
+ shape: Sequence[int],
+ *,
+ dtype: DTypeLike | None = None,
+ class_name: str = "dgCMatrix",
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ logical_shape = _validate_shape(shape, limits)
+ if class_name not in self._SUPPORTED_CLASSES:
+ raise MatrixSourceError(f"unsupported Matrix class {class_name!r}")
+ pointer_shape = _array_shape(p)
+ if pointer_shape != (logical_shape[1] + 1,):
+ raise MatrixSourceError(
+ f"CSC p slot has shape {pointer_shape}; "
+ f"expected ({logical_shape[1] + 1},)"
+ )
+ index_shape = _array_shape(i)
+ if len(index_shape) != 1:
+ raise MatrixSourceError("CSC i slot must be one-dimensional")
+ self._x = x
+ self._i = i
+ self._p = p
+ self.class_name = class_name
+ self._nnz = self._validate_structure(logical_shape, limits)
+ if x is None:
+ source_dtype = np.dtype(bool if dtype is None else dtype)
+ else:
+ value_shape = _array_shape(x)
+ if value_shape != (self._nnz,):
+ raise MatrixSourceError(
+ f"CSC x slot has shape {value_shape}; expected ({self._nnz},)"
+ )
+ source_dtype = _array_dtype(x) if dtype is None else np.dtype(dtype)
+ super().__init__(
+ logical_shape,
+ source_dtype,
+ row_names=row_names,
+ column_names=column_names,
+ is_sparse=True,
+ limits=limits,
+ )
+
+ def _validate_structure(
+ self,
+ shape: tuple[int, int],
+ limits: SourceLimits,
+ ) -> int:
+ chunk_size = max(1, min(limits.compressedChunkNnz, shape[1] + 1))
+ previous: int | None = None
+ last = 0
+ for start in range(0, shape[1] + 1, chunk_size):
+ stop = min(shape[1] + 1, start + chunk_size)
+ pointers = _read_1d(self._p, start, stop)
+ if not np.issubdtype(pointers.dtype, np.integer):
+ raise TypeError("CSC p slot must contain integers")
+ pointers = pointers.astype(np.int64, copy=False)
+ if previous is not None and pointers.size and int(pointers[0]) < previous:
+ raise MatrixSourceError("CSC p slot must be nondecreasing")
+ if pointers.size > 1 and np.any(pointers[1:] < pointers[:-1]):
+ raise MatrixSourceError("CSC p slot must be nondecreasing")
+ if start == 0 and (not pointers.size or int(pointers[0]) != 0):
+ raise MatrixSourceError("CSC p slot must start at zero")
+ if pointers.size:
+ previous = int(pointers[-1])
+ last = previous
+ if last < 0:
+ raise MatrixSourceError("CSC p slot cannot contain negative offsets")
+ if last > limits.maxNnz:
+ raise ResourceLimitError(f"CSC nnz {last} exceeds maxNnz={limits.maxNnz}")
+ if _array_shape(self._i) != (last,):
+ raise MatrixSourceError(
+ f"CSC i slot has length {_array_shape(self._i)[0]}; expected {last}"
+ )
+ for start in range(0, last, limits.compressedChunkNnz):
+ stop = min(last, start + limits.compressedChunkNnz)
+ indexes = _read_1d(self._i, start, stop)
+ if not np.issubdtype(indexes.dtype, np.integer):
+ raise TypeError("CSC i slot must contain integers")
+ if indexes.size and (np.any(indexes < 0) or np.any(indexes >= shape[0])):
+ raise MatrixSourceError("CSC i slot contains an out-of-range row")
+ return last
+
+ @property
+ def nnz(self) -> int:
+ return self._nnz
+
+ @property
+ def resident_bytes(self) -> int:
+ values = 0 if self._x is None else _array_resident_bytes(self._x)
+ return (
+ super().resident_bytes
+ + values
+ + _array_resident_bytes(self._i)
+ + _array_resident_bytes(self._p)
+ )
+
+ def _range(self, start: int, stop: int) -> tuple[NDArray[Any], int, int]:
+ pointers = _read_1d(self._p, start, stop + 1)
+ if not np.issubdtype(pointers.dtype, np.integer):
+ raise TypeError("CSC p slot must contain integers")
+ pointers = pointers.astype(np.int64, copy=False)
+ data_start = int(pointers[0])
+ data_stop = int(pointers[-1])
+ return pointers - data_start, data_start, data_stop
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ pointers, data_start, data_stop = self._range(start, stop)
+ nnz = data_stop - data_start
+ index_size = np.dtype(np.int64).itemsize
+ output = nnz * (self.dtype.itemsize + index_size) + pointers.nbytes
+ working = output
+ return MemoryEstimate(self.resident_bytes, working, output)
+
+ def read_cells(self, start: int, stop: int) -> csr_matrix:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ pointers, data_start, data_stop = self._range(start, stop)
+ indexes = _read_1d(self._i, data_start, data_stop, dtype=np.int64)
+ if self._x is None:
+ data = np.ones(data_stop - data_start, dtype=self.dtype)
+ else:
+ data = _read_1d(self._x, data_start, data_stop, dtype=self.dtype)
+ return csr_matrix(
+ (data, indexes, pointers),
+ shape=(stop - start, self.n_features),
+ dtype=self.dtype,
+ )
+
+
+CSCMatrixSource = CscMatrixSource
+MatrixCscSource = CscMatrixSource
+
+
+class MappedMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ *,
+ feature_indices: Sequence[int] | NDArray[Any] | None = None,
+ cell_indices: Sequence[int] | NDArray[Any] | None = None,
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self.source = source
+ self.feature_indices = (
+ None
+ if feature_indices is None
+ else _normalize_indexes(feature_indices, source.shape[0], "feature")
+ )
+ self.cell_indices = (
+ None
+ if cell_indices is None
+ else _normalize_indexes(cell_indices, source.shape[1], "cell")
+ )
+ n_features = (
+ source.shape[0]
+ if self.feature_indices is None
+ else int(self.feature_indices.size)
+ )
+ n_cells = (
+ source.shape[1]
+ if self.cell_indices is None
+ else int(self.cell_indices.size)
+ )
+ mapped_rows = row_names
+ if mapped_rows is None and source.row_names is not None:
+ mapped_rows = (
+ source.row_names
+ if self.feature_indices is None
+ else tuple(source.row_names[index] for index in self.feature_indices)
+ )
+ mapped_columns = column_names
+ if mapped_columns is None and source.column_names is not None:
+ mapped_columns = (
+ source.column_names
+ if self.cell_indices is None
+ else tuple(source.column_names[index] for index in self.cell_indices)
+ )
+ super().__init__(
+ (n_features, n_cells),
+ source.dtype,
+ row_names=mapped_rows,
+ column_names=mapped_columns,
+ is_sparse=source.is_sparse,
+ zero_preserving=source.zero_preserving,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ mapping_bytes = 0
+ if self.feature_indices is not None:
+ mapping_bytes += self.feature_indices.nbytes
+ if self.cell_indices is not None:
+ mapping_bytes += self.cell_indices.nbytes
+ return int(super().resident_bytes + self.source.resident_bytes + mapping_bytes)
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ rows = stop - start
+ if self.cell_indices is None:
+ source_estimate = self.source.estimate_read_memory(start, stop)
+ elif rows == 0:
+ source_estimate = MemoryEstimate()
+ else:
+ selected = self.cell_indices[start:stop]
+ source_estimate = MemoryEstimate(
+ workingBytes=sum(
+ self.source.estimate_read_memory(
+ int(index), int(index) + 1
+ ).peakBytes
+ for index in selected
+ )
+ )
+ output = rows * self.n_features * self.dtype.itemsize
+ if self.is_sparse:
+ output += rows * np.dtype(np.int64).itemsize
+ return MemoryEstimate(
+ self.resident_bytes,
+ source_estimate.workingBytes + output,
+ output,
+ )
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ if self.cell_indices is None:
+ block = self.source.read_cells(start, stop)
+ else:
+ block = _read_selected_cells(self.source, self.cell_indices[start:stop])
+ if self.feature_indices is None:
+ return block
+ if isinstance(block, spmatrix):
+ return cast(
+ MatrixBlock,
+ block.tocsr(copy=False)[:, self.feature_indices].tocsr(),
+ )
+ return np.ascontiguousarray(np.asarray(block)[:, self.feature_indices])
+
+
+SubsetMatrixSource = MappedMatrixSource
+ReorderedMatrixSource = MappedMatrixSource
+
+
+class TransposeMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ *,
+ tile_cells: int | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self.source = source
+ self.tile_cells = limits.tileCells if tile_cells is None else int(tile_cells)
+ if self.tile_cells <= 0:
+ raise ValueError("tile_cells must be positive")
+ super().__init__(
+ (source.shape[1], source.shape[0]),
+ source.dtype,
+ row_names=source.column_names,
+ column_names=source.row_names,
+ is_sparse=source.is_sparse,
+ zero_preserving=source.zero_preserving,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return super().resident_bytes + self.source.resident_bytes
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ tile = min(self.tile_cells, self.source.shape[1])
+ working = 0
+ for source_start in range(0, self.source.shape[1], max(1, tile)):
+ source_stop = min(self.source.shape[1], source_start + max(1, tile))
+ working = max(
+ working,
+ self.source.estimate_read_memory(source_start, source_stop).peakBytes,
+ )
+ return MemoryEstimate(self.resident_bytes, working, output)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ if start == stop:
+ return _empty_block(0, self.n_features, self.dtype, self.is_sparse)
+ if self.is_sparse:
+ pieces: list[csr_matrix] = []
+ for source_start in range(0, self.source.shape[1], self.tile_cells):
+ source_stop = min(self.source.shape[1], source_start + self.tile_cells)
+ block = _block_to_csr(
+ self.source.read_cells(source_start, source_stop),
+ dtype=self.dtype,
+ )
+ pieces.append(block[:, start:stop].T.tocsr())
+ if not pieces:
+ return csr_matrix((stop - start, self.n_features), dtype=self.dtype)
+ return cast(
+ MatrixBlock,
+ hstack(pieces, format="csr", dtype=self.dtype),
+ )
+ output = np.empty((stop - start, self.source.shape[1]), dtype=self.dtype)
+ for source_start in range(0, self.source.shape[1], self.tile_cells):
+ source_stop = min(self.source.shape[1], source_start + self.tile_cells)
+ block = _block_to_dense(
+ self.source.read_cells(source_start, source_stop),
+ dtype=self.dtype,
+ )
+ output[:, source_start:source_stop] = block[:, start:stop].T
+ return output
+
+
+TransposeSource = TransposeMatrixSource
+
+
+def _matching_names(
+ sources: Sequence[MatrixSource],
+ axis: str,
+) -> tuple[str, ...] | None:
+ if axis == "row_names":
+ first = sources[0].row_names
+ elif axis == "column_names":
+ first = sources[0].column_names
+ else:
+ raise ValueError("axis must be row_names or column_names")
+ for source in sources[1:]:
+ current = source.row_names if axis == "row_names" else source.column_names
+ if first is not None and current is not None and current != first:
+ raise MatrixSourceError(f"{axis.replace('_', ' ')} conflict across sources")
+ if first is None:
+ first = current
+ return first
+
+
+class FeatureBindMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ sources: Sequence[MatrixSource],
+ *,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ if not sources:
+ raise ValueError("feature bind requires at least one source")
+ self.sources = tuple(sources)
+ n_cells = self.sources[0].shape[1]
+ if any(source.shape[1] != n_cells for source in self.sources):
+ raise MatrixSourceError("feature bind sources must have equal cell counts")
+ column_names = _matching_names(self.sources, "column_names")
+ row_names = (
+ tuple(name for source in self.sources for name in source.row_names or ())
+ if all(source.row_names is not None for source in self.sources)
+ else None
+ )
+ dtype = np.result_type(*(source.dtype for source in self.sources))
+ super().__init__(
+ (sum(source.shape[0] for source in self.sources), n_cells),
+ dtype,
+ row_names=row_names,
+ column_names=column_names,
+ is_sparse=all(source.is_sparse for source in self.sources),
+ zero_preserving=all(source.zero_preserving for source in self.sources),
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return super().resident_bytes + sum(
+ source.resident_bytes for source in self.sources
+ )
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ child = sum(
+ source.estimate_read_memory(start, stop).peakBytes
+ for source in self.sources
+ )
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(self.resident_bytes, child, output)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ blocks = [source.read_cells(start, stop) for source in self.sources]
+ if self.is_sparse:
+ return cast(
+ MatrixBlock,
+ hstack(
+ [_block_to_csr(block, dtype=self.dtype) for block in blocks],
+ format="csr",
+ dtype=self.dtype,
+ ),
+ )
+ return np.hstack([_block_to_dense(block, dtype=self.dtype) for block in blocks])
+
+
+class CellBindMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ sources: Sequence[MatrixSource],
+ *,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ if not sources:
+ raise ValueError("cell bind requires at least one source")
+ self.sources = tuple(sources)
+ n_features = self.sources[0].shape[0]
+ if any(source.shape[0] != n_features for source in self.sources):
+ raise MatrixSourceError("cell bind sources must have equal feature counts")
+ row_names = _matching_names(self.sources, "row_names")
+ column_names = (
+ tuple(name for source in self.sources for name in source.column_names or ())
+ if all(source.column_names is not None for source in self.sources)
+ else None
+ )
+ dtype = np.result_type(*(source.dtype for source in self.sources))
+ self._offsets = np.cumsum(
+ [0, *(source.shape[1] for source in self.sources)],
+ dtype=np.int64,
+ )
+ super().__init__(
+ (n_features, int(self._offsets[-1])),
+ dtype,
+ row_names=row_names,
+ column_names=column_names,
+ is_sparse=all(source.is_sparse for source in self.sources),
+ zero_preserving=all(source.zero_preserving for source in self.sources),
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return int(
+ super().resident_bytes
+ + self._offsets.nbytes
+ + sum(source.resident_bytes for source in self.sources)
+ )
+
+ def _pieces(self, start: int, stop: int) -> list[tuple[MatrixSource, int, int]]:
+ pieces: list[tuple[MatrixSource, int, int]] = []
+ for index, source in enumerate(self.sources):
+ source_global_start = int(self._offsets[index])
+ source_global_stop = int(self._offsets[index + 1])
+ overlap_start = max(start, source_global_start)
+ overlap_stop = min(stop, source_global_stop)
+ if overlap_start < overlap_stop:
+ pieces.append(
+ (
+ source,
+ overlap_start - source_global_start,
+ overlap_stop - source_global_start,
+ )
+ )
+ return pieces
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ child = sum(
+ source.estimate_read_memory(local_start, local_stop).peakBytes
+ for source, local_start, local_stop in self._pieces(start, stop)
+ )
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(self.resident_bytes, child, output)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ blocks = [
+ source.read_cells(local_start, local_stop)
+ for source, local_start, local_stop in self._pieces(start, stop)
+ ]
+ if not blocks:
+ return _empty_block(0, self.n_features, self.dtype, self.is_sparse)
+ if self.is_sparse:
+ return cast(
+ MatrixBlock,
+ vstack(
+ [_block_to_csr(block, dtype=self.dtype) for block in blocks],
+ format="csr",
+ dtype=self.dtype,
+ ),
+ )
+ return np.vstack([_block_to_dense(block, dtype=self.dtype) for block in blocks])
+
+
+RowBindMatrixSource = FeatureBindMatrixSource
+ColumnBindMatrixSource = CellBindMatrixSource
+FeatureBindSource = FeatureBindMatrixSource
+CellBindSource = CellBindMatrixSource
+
+
+@dataclass(frozen=True, init=False)
+class LayerPlacement:
+ source: MatrixSource
+ featureIndices: Sequence[int] | NDArray[Any] | None = None
+ cellIndices: Sequence[int] | NDArray[Any] | None = None
+ name: str | None = None
+
+ def __init__(
+ self,
+ source: MatrixSource,
+ feature_indices: Sequence[int] | NDArray[Any] | None = None,
+ cell_indices: Sequence[int] | NDArray[Any] | None = None,
+ name: str | None = None,
+ *,
+ featureIndices: Sequence[int] | NDArray[Any] | None = None,
+ cellIndices: Sequence[int] | NDArray[Any] | None = None,
+ ) -> None:
+ if feature_indices is not None and featureIndices is not None:
+ raise TypeError("provide only one feature index spelling")
+ if cell_indices is not None and cellIndices is not None:
+ raise TypeError("provide only one cell index spelling")
+ object.__setattr__(self, "source", source)
+ object.__setattr__(
+ self,
+ "featureIndices",
+ feature_indices if featureIndices is None else featureIndices,
+ )
+ object.__setattr__(
+ self,
+ "cellIndices",
+ cell_indices if cellIndices is None else cellIndices,
+ )
+ object.__setattr__(self, "name", name)
+
+ @property
+ def feature_indices(self) -> Sequence[int] | NDArray[Any] | None:
+ return self.featureIndices
+
+ @property
+ def cell_indices(self) -> Sequence[int] | NDArray[Any] | None:
+ return self.cellIndices
+
+
+@dataclass(frozen=True)
+class _ResolvedLayerPlacement:
+ source: MatrixSource
+ featureIndices: NDArray[np.int64]
+ cellIndices: NDArray[np.int64]
+ name: str
+
+
+def _map_names(
+ local_names: tuple[str, ...] | None,
+ global_names: tuple[str, ...],
+ axis: str,
+) -> NDArray[np.int64]:
+ if local_names is None:
+ raise MatrixSourceError(
+ f"{axis} indexes are required when source names are absent"
+ )
+ if len(set(global_names)) != len(global_names):
+ raise MatrixSourceError(
+ f"global {axis} names must be unique for name-based stitching"
+ )
+ lookup = {name: index for index, name in enumerate(global_names)}
+ missing = [name for name in local_names if name not in lookup]
+ if missing:
+ raise MatrixSourceError(
+ f"source {axis} name {missing[0]!r} is absent from the global axis"
+ )
+ return np.asarray([lookup[name] for name in local_names], dtype=np.int64)
+
+
+class LayerStitchMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ layers: Sequence[LayerPlacement | MatrixSource],
+ *,
+ row_names: Sequence[str | bytes] | NDArray[Any],
+ column_names: Sequence[str | bytes] | NDArray[Any],
+ dtype: DTypeLike | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ raw_rows = _normalize_names(row_names, len(row_names), "row", limits)
+ raw_columns = _normalize_names(
+ column_names, len(column_names), "column", limits
+ )
+ assert raw_rows is not None
+ assert raw_columns is not None
+ if not layers:
+ raise ValueError("layer stitching requires at least one layer")
+ resolved: list[_ResolvedLayerPlacement] = []
+ for index, item in enumerate(layers):
+ placement = (
+ item if isinstance(item, LayerPlacement) else LayerPlacement(item)
+ )
+ features = (
+ _map_names(placement.source.row_names, raw_rows, "feature")
+ if placement.featureIndices is None
+ else _normalize_indexes(
+ placement.featureIndices, len(raw_rows), "feature"
+ )
+ )
+ cells = (
+ _map_names(placement.source.column_names, raw_columns, "cell")
+ if placement.cellIndices is None
+ else _normalize_indexes(placement.cellIndices, len(raw_columns), "cell")
+ )
+ if features.size != placement.source.shape[0]:
+ raise MatrixSourceError(
+ f"layer {index} maps {features.size} features; "
+ f"source has {placement.source.shape[0]}"
+ )
+ if cells.size != placement.source.shape[1]:
+ raise MatrixSourceError(
+ f"layer {index} maps {cells.size} cells; "
+ f"source has {placement.source.shape[1]}"
+ )
+ if np.unique(features).size != features.size:
+ raise MatrixSourceError(f"layer {index} repeats a global feature")
+ if np.unique(cells).size != cells.size:
+ raise MatrixSourceError(f"layer {index} repeats a global cell")
+ self._validate_layer_names(
+ placement.source,
+ features,
+ cells,
+ raw_rows,
+ raw_columns,
+ index,
+ )
+ resolved.append(
+ _ResolvedLayerPlacement(
+ placement.source,
+ features,
+ cells,
+ placement.name or f"layer[{index}]",
+ )
+ )
+ self._validate_conflicts(resolved)
+ self.layers = tuple(resolved)
+ result_dtype = (
+ np.result_type(*(layer.source.dtype for layer in resolved))
+ if dtype is None
+ else np.dtype(dtype)
+ )
+ super().__init__(
+ (len(raw_rows), len(raw_columns)),
+ result_dtype,
+ row_names=raw_rows,
+ column_names=raw_columns,
+ is_sparse=True,
+ zero_preserving=all(layer.source.zero_preserving for layer in resolved),
+ limits=limits,
+ )
+
+ @staticmethod
+ def _validate_layer_names(
+ source: MatrixSource,
+ feature_indexes: NDArray[np.int64],
+ cell_indexes: NDArray[np.int64],
+ row_names: tuple[str, ...],
+ column_names: tuple[str, ...],
+ index: int,
+ ) -> None:
+ if source.row_names is not None:
+ expected = tuple(row_names[position] for position in feature_indexes)
+ if source.row_names != expected:
+ raise MatrixSourceError(
+ f"layer {index} feature names conflict with global mapping"
+ )
+ if source.column_names is not None:
+ expected = tuple(column_names[position] for position in cell_indexes)
+ if source.column_names != expected:
+ raise MatrixSourceError(
+ f"layer {index} cell names conflict with global mapping"
+ )
+
+ @staticmethod
+ def _validate_conflicts(
+ layers: Sequence[_ResolvedLayerPlacement],
+ ) -> None:
+ feature_sets = [set(layer.featureIndices.tolist()) for layer in layers]
+ cell_sets = [set(layer.cellIndices.tolist()) for layer in layers]
+ for left in range(len(layers)):
+ for right in range(left + 1, len(layers)):
+ if feature_sets[left].intersection(feature_sets[right]) and cell_sets[
+ left
+ ].intersection(cell_sets[right]):
+ raise MatrixSourceError(
+ f"layer coordinate conflict between "
+ f"{layers[left].name!r} and {layers[right].name!r}"
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ mappings = sum(
+ layer.featureIndices.nbytes + layer.cellIndices.nbytes
+ for layer in self.layers
+ )
+ return int(
+ super().resident_bytes
+ + mappings
+ + sum(layer.source.resident_bytes for layer in self.layers)
+ )
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ child = 0
+ for layer in self.layers:
+ selected = np.flatnonzero(
+ (layer.cellIndices >= start) & (layer.cellIndices < stop)
+ )
+ child += sum(
+ layer.source.estimate_read_memory(
+ int(position), int(position) + 1
+ ).peakBytes
+ for position in selected
+ )
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(self.resident_bytes, child, output)
+
+ def read_cells(self, start: int, stop: int) -> csr_matrix:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ data_parts: list[NDArray[Any]] = []
+ row_parts: list[NDArray[np.int64]] = []
+ column_parts: list[NDArray[np.int64]] = []
+ for layer in self.layers:
+ local_cells: NDArray[np.int64] = np.flatnonzero(
+ (layer.cellIndices >= start) & (layer.cellIndices < stop)
+ ).astype(np.int64, copy=False)
+ if local_cells.size == 0:
+ continue
+ block = _read_selected_cells(layer.source, local_cells)
+ sparse_block = _block_to_csr(block, dtype=self.dtype).tocoo(copy=False)
+ if sparse_block.nnz == 0:
+ continue
+ global_rows = (
+ layer.cellIndices[local_cells[sparse_block.row]].astype(
+ np.int64, copy=False
+ )
+ - start
+ )
+ global_columns = layer.featureIndices[sparse_block.col].astype(
+ np.int64, copy=False
+ )
+ data_parts.append(np.asarray(sparse_block.data, dtype=self.dtype))
+ row_parts.append(global_rows)
+ column_parts.append(global_columns)
+ if not data_parts:
+ return csr_matrix((stop - start, self.n_features), dtype=self.dtype)
+ return coo_matrix(
+ (
+ np.concatenate(data_parts),
+ (np.concatenate(row_parts), np.concatenate(column_parts)),
+ ),
+ shape=(stop - start, self.n_features),
+ dtype=self.dtype,
+ ).tocsr()
+
+
+LayerStitchSource = LayerStitchMatrixSource
+
+
+class RenamedMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ *,
+ row_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ column_names: Sequence[str | bytes] | NDArray[Any] | None = None,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self.source = source
+ super().__init__(
+ source.shape,
+ source.dtype,
+ row_names=source.row_names if row_names is None else row_names,
+ column_names=(
+ source.column_names if column_names is None else column_names
+ ),
+ is_sparse=source.is_sparse,
+ zero_preserving=source.zero_preserving,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return super().resident_bytes + self.source.resident_bytes
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ return self.source.estimate_read_memory(start, stop)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ return self.source.read_cells(start, stop)
+
+
+class DtypeMatrixSource(BaseMatrixSource):
+ def __init__(
+ self,
+ source: MatrixSource,
+ dtype: DTypeLike,
+ *,
+ limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self.source = source
+ super().__init__(
+ source.shape,
+ dtype,
+ row_names=source.row_names,
+ column_names=source.column_names,
+ is_sparse=source.is_sparse,
+ zero_preserving=source.zero_preserving,
+ limits=limits,
+ )
+
+ @property
+ def resident_bytes(self) -> int:
+ return super().resident_bytes + self.source.resident_bytes
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ start, stop = self._window(start, stop)
+ child = self.source.estimate_read_memory(start, stop)
+ output = (stop - start) * self.n_features * self.dtype.itemsize
+ return MemoryEstimate(self.resident_bytes, child.peakBytes, output)
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ start, stop = self._window(start, stop)
+ estimate = self.estimate_read_memory(start, stop)
+ self._admit(estimate)
+ block = self.source.read_cells(start, stop)
+ if isinstance(block, spmatrix):
+ return cast(MatrixBlock, block.astype(self.dtype, copy=False))
+ return np.asarray(block).astype(self.dtype, copy=False)
+
+
+RenameMatrixSource = RenamedMatrixSource
+CastMatrixSource = DtypeMatrixSource
diff --git a/scarf/readers/_text.py b/scarf/readers/_text.py
new file mode 100644
index 00000000..1ddb4662
--- /dev/null
+++ b/scarf/readers/_text.py
@@ -0,0 +1,35 @@
+from collections.abc import Generator
+from typing import IO
+
+
+def get_file_handle(fn: str) -> IO:
+ """Returns a file object for the given file name.
+
+ Args:
+ fn: The path to the file (file name).
+ """
+ import gzip
+
+ try:
+ if fn.rsplit(".", 1)[-1] == "gz":
+ return gzip.open(fn, mode="rt")
+ else:
+ return open(fn, "r")
+ except (OSError, IOError, FileNotFoundError):
+ raise FileNotFoundError("ERROR: FILE NOT FOUND: %s" % fn)
+
+
+def read_file(fn: str) -> Generator[str, None, None]:
+ """Yields the lines from the file the given file name points to.
+
+ Args:
+ fn: The path to the file (file name).
+ """
+ from . import get_file_handle
+
+ fh = get_file_handle(fn)
+ try:
+ for line in fh:
+ yield line.rstrip()
+ finally:
+ fh.close()
diff --git a/scarf/readers/cellranger.py b/scarf/readers/cellranger.py
new file mode 100644
index 00000000..445e0283
--- /dev/null
+++ b/scarf/readers/cellranger.py
@@ -0,0 +1,761 @@
+import math
+from abc import ABC, abstractmethod
+from collections.abc import Generator, Iterator, Sequence
+from typing import Any
+
+import h5py
+import numpy as np
+import pandas as pd
+from numpy.typing import DTypeLike
+from scipy.sparse import coo_matrix
+
+from ..utils.logging import logger
+from ..utils.progress import iter_progress
+from ._assay_names import (
+ AUTO_ASSAY_NAMES,
+ auto_name_feat_table,
+ make_feat_table_from_types,
+)
+
+
+class CrReader(ABC):
+ """A class to read in CellRanger (Cr) data.
+
+ Args:
+ grp_names (Dict): A dictionary that specifies where to find the matrix, features and barcodes.
+
+ Attributes:
+ autoNames: Specifies if the data is from RNA or ATAC sequencing.
+ grpNames: A dictionary that specifies where to find the matrix, features and barcodes.
+ nFeatures: Number of features in dataset.
+ nCells: Number of cells in dataset.
+ assayFeats: A DataFrame with information about the features in the assay.
+ """
+
+ def __init__(self, grp_names: dict[str, Any]) -> None:
+ self.autoNames = dict(AUTO_ASSAY_NAMES)
+ self._featureTypeOverrides: dict[int, str] = {}
+ self._schemaCaptured = False
+ self.grpNames: dict[str, Any] = grp_names
+ self.nFeatures: int = len(self.feature_names())
+ self.nCells: int = len(self.cell_names())
+ self.assayFeats = self._make_feat_table()
+ self._auto_rename_assay_names()
+
+ @abstractmethod
+ def _handle_version(self) -> dict[str, Any]:
+ pass
+
+ @abstractmethod
+ def _read_dataset(self, key: str | None = None) -> list[Any] | None:
+ pass
+
+ @abstractmethod
+ def consume(
+ self, batch_size: int, lines_in_mem: int
+ ) -> Generator[coo_matrix, None, None]:
+ """Yield CSR matrix chunks of cell rows.
+
+ Args:
+ batch_size: Number of cells per yielded chunk.
+ lines_in_mem: MTX lines buffered in memory (CrDirReader only).
+
+ Yields:
+ scipy.sparse.coo_matrix chunks.
+ """
+ pass
+
+ def max_window_nnz(self, window_rows: int) -> int:
+ """Bound nnz in any source row window."""
+ if window_rows <= 0:
+ raise ValueError("window_rows must be positive")
+ return min(window_rows, self.nCells) * self.nFeatures
+
+ @property
+ def matrix_dtype(self) -> np.dtype[Any]:
+ """Return the count dtype yielded by the default consume call."""
+ return np.dtype(np.uint32)
+
+ def producer_staging_bytes(
+ self,
+ batch_size: int,
+ lines_in_mem: int,
+ ) -> int:
+ """Return source-reader bytes retained while a matrix batch is yielded."""
+ valid_idx = getattr(self, "validBarcodeIdx", None)
+ return int(valid_idx.nbytes) if isinstance(valid_idx, np.ndarray) else 0
+
+ def _prepare_sparse_import(self) -> None:
+ """Prepare optional reader-owned state used by import planning."""
+
+ def _release_sparse_import(self) -> None:
+ """Release optional reader-owned state created for one import."""
+
+ def _sparse_import_resident_bytes(self) -> int:
+ """Return reader arrays retained for the duration of an import."""
+ return 0
+
+ def _subset_by_assay(self, v: list[Any], assay: str | None) -> list[Any]:
+ if assay is None:
+ return v
+ elif assay not in self.assayFeats:
+ raise ValueError(f"ERROR: Assay ID {assay} is not valid")
+ if len(self.assayFeats[assay].shape) == 2:
+ ret_val: list[Any] = []
+ for i in self.assayFeats[assay].values[1:3].T:
+ ret_val.extend(list(v[i[0] : i[1]]))
+ return ret_val
+ elif len(self.assayFeats[assay].shape) == 1:
+ idx = self.assayFeats[assay]
+ return v[idx.start : idx.end]
+ else:
+ raise ValueError(
+ "ERROR: assay feats is 3D. Something went really wrong. Create a github issue"
+ )
+
+ @staticmethod
+ def _make_feat_table_from_types(feature_types: Sequence[str]) -> pd.DataFrame:
+ return make_feat_table_from_types(feature_types)
+
+ def _make_feat_table(self) -> pd.DataFrame:
+ return self._make_feat_table_from_types(self.feature_types())
+
+ def _auto_named_feat_table(self, assay_feats: pd.DataFrame) -> pd.DataFrame:
+ return auto_name_feat_table(assay_feats, self.autoNames)
+
+ def _auto_rename_assay_names(self) -> None:
+ self.assayFeats = self._auto_named_feat_table(self.assayFeats)
+
+ def _mark_schema_captured(self) -> None:
+ self._schemaCaptured = True
+
+ def reclassify_features(
+ self,
+ indexes: Sequence[int],
+ feature_type: str,
+ *,
+ require_previous: str | None = "Antibody Capture",
+ ) -> None:
+ """Reclassify global feature rows before a writer captures the schema."""
+ if self._schemaCaptured:
+ raise RuntimeError(
+ "Features cannot be reclassified after a writer captures the schema"
+ )
+ if not isinstance(feature_type, str) or feature_type == "":
+ raise ValueError("feature_type must be a non-empty string")
+ if isinstance(indexes, str):
+ raise TypeError("indexes must be a sequence of integer feature indexes")
+ index_array = np.asarray(indexes)
+ if index_array.ndim != 1:
+ raise ValueError("indexes must be one-dimensional")
+ if index_array.size == 0:
+ raise ValueError("indexes must contain at least one feature index")
+ if not np.issubdtype(index_array.dtype, np.integer):
+ raise TypeError("indexes must contain only integers")
+ index_array = index_array.astype(np.int64, copy=False)
+ if np.unique(index_array).size != index_array.size:
+ raise ValueError("indexes must contain unique feature indexes")
+ if np.any(index_array < 0) or np.any(index_array >= self.nFeatures):
+ raise IndexError("indexes contains an out-of-range feature index")
+
+ conflicting = [
+ int(index)
+ for index in index_array
+ if index in self._featureTypeOverrides
+ and self._featureTypeOverrides[int(index)] != feature_type
+ ]
+ if conflicting:
+ raise ValueError(
+ "Features already have a conflicting reclassification: "
+ + ", ".join(map(str, conflicting))
+ )
+
+ current_types = self.feature_types()
+ pending = np.asarray(
+ [
+ index
+ for index in index_array
+ if current_types[int(index)] != feature_type
+ ],
+ dtype=np.int64,
+ )
+ if pending.size == 0:
+ return None
+ if require_previous is not None:
+ invalid = [
+ int(index)
+ for index in pending
+ if current_types[int(index)] != require_previous
+ ]
+ if invalid:
+ raise ValueError(
+ f"Features must currently have type {require_previous!r}: "
+ + ", ".join(map(str, invalid))
+ )
+
+ updated_types = list(current_types)
+ updated_overrides = dict(self._featureTypeOverrides)
+ for index in pending:
+ updated_types[int(index)] = feature_type
+ updated_overrides[int(index)] = feature_type
+ updated_table = self._auto_named_feat_table(
+ self._make_feat_table_from_types(updated_types)
+ )
+
+ self._featureTypeOverrides = updated_overrides
+ self.assayFeats = updated_table
+ return None
+
+ def rename_assays(self, name_map: dict[str, str]) -> None:
+ """Renames specified assays in the Reader.
+
+ Args:
+ name_map: A Dictionary containing current name as key and new name as value.
+ """
+ self.assayFeats.rename(columns=name_map, inplace=True)
+
+ def feature_ids(self, assay: str | None = None) -> list[str]:
+ """Returns a list of feature IDs in a specified assay.
+
+ Args:
+ assay: Select which assay to retrieve feature IDs from.
+ """
+ vals = self._read_dataset("feature_ids")
+ if vals is None:
+ return []
+ return self._subset_by_assay(vals, assay)
+
+ def feature_names(self, assay: str | None = None) -> list[str]:
+ """Returns a list of features in the dataset.
+
+ Args:
+ assay: Select which assay to retrieve features from.
+ """
+ vals = self._read_dataset("feature_names")
+ if vals is None:
+ logger.warning("Feature names extraction failed using feature IDs")
+ vals = self._read_dataset("feature_ids")
+ if vals is None:
+ return []
+ return self._subset_by_assay(vals, assay)
+
+ def feature_types(self) -> list[str]:
+ """Returns a list of feature types in the dataset."""
+ if self.grpNames["feature_types"] is not None:
+ ret_val = self._read_dataset("feature_types")
+ if ret_val is not None:
+ feature_types = list(ret_val)
+ else:
+ feature_types = []
+ else:
+ feature_types = []
+ if not feature_types:
+ default_name = list(self.autoNames.keys())[0]
+ feature_types = [default_name for _ in range(self.nFeatures)]
+ for index, feature_type in self._featureTypeOverrides.items():
+ feature_types[index] = feature_type
+ return feature_types
+
+ def cell_names(self) -> list[str]:
+ """Returns a list of names of the cells in the dataset."""
+ vals = self._read_dataset("cell_names")
+ if vals is None:
+ return []
+ return vals
+
+ def get_cell_columns(self) -> Iterator[tuple[str, np.ndarray]]:
+ """Yield optional cell metadata columns supplied by the reader."""
+ yield from ()
+
+ def get_feature_columns(self) -> Iterator[tuple[str, np.ndarray]]:
+ """Yield source feature types and optional feature metadata."""
+ yield "feature_type", np.asarray(self.feature_types(), dtype=object)
+
+
+class CrH5Reader(CrReader):
+ # noinspection PyUnresolvedReferences
+ """A class to read in CellRanger (Cr) data, in the form of an H5 file.
+
+ Subclass of CrReader.
+
+ Args:
+ h5_fn: File name for the h5 file.
+
+ Attributes:
+ autoNames: Specifies if the data is from RNA or ATAC sequencing.
+ grpNames: A dictionary that specifies where to find the matrix, features and barcodes.
+ nFeatures: Number of features in dataset.
+ nCells: Number of cells in dataset.
+ assayFeats: A DataFrame with information about the features in the assay.
+ h5obj: A File object from the h5py package.
+ grp: Current active group in the hierarchy.
+ """
+
+ def __init__(
+ self,
+ h5_fn: str,
+ is_filtered: bool = True,
+ filtering_cutoff: int = 500,
+ ) -> None:
+ self.h5obj: h5py.File = h5py.File(h5_fn, mode="r")
+ self.grp: h5py.Group
+ self.validBarcodeIdx: np.ndarray | None = None
+ self._indptrCache: np.ndarray | None = None
+ self._cumulativeRowNnz: np.ndarray | None = None
+ super().__init__(self._handle_version())
+ if is_filtered:
+ self.validBarcodeIdx = np.array(range(self.nCells))
+ else:
+ self.validBarcodeIdx = self._get_valid_barcodes(filtering_cutoff)
+ self.nCells = len(self.validBarcodeIdx)
+
+ def _handle_version(self) -> dict[str, str | None]:
+ root_key = list(self.h5obj.keys())[0]
+ self.grp = self.h5obj[root_key]
+ if root_key == "matrix":
+ grps: dict[str, str | None] = {
+ "feature_ids": "features/id",
+ "feature_names": "features/name",
+ "feature_types": "features/feature_type",
+ "cell_names": "barcodes",
+ }
+ else:
+ grps = {
+ "feature_ids": "genes",
+ "feature_names": "gene_names",
+ "feature_types": None,
+ "cell_names": "barcodes",
+ }
+ return grps
+
+ def _get_valid_barcodes(
+ self, filtering_cutoff: int, batch_size: int = 1000
+ ) -> np.ndarray:
+ valid_idx = []
+ test_counter = 0
+ indptr = self._source_indptr()
+ for s in iter_progress(
+ range(0, len(indptr) - 1, batch_size),
+ desc="Filtering out background barcodes",
+ ):
+ idx = indptr[s : s + batch_size + 1]
+ data = self.grp["data"][idx[0] : idx[-1]]
+ indices = self.grp["indices"][idx[0] : idx[-1]]
+ cell_idx = np.repeat(range(len(idx) - 1), np.diff(idx))
+ mat = coo_matrix(
+ (data, (cell_idx, indices)), shape=(len(idx) - 1, self.nFeatures)
+ )
+ valid_idx.append(np.array(mat.sum(axis=1)).T[0] > filtering_cutoff)
+ test_counter += data.shape[0]
+ assert test_counter == self.grp["data"].shape[0]
+ assert len(indptr) == (s + len(idx))
+ return np.where(np.hstack(valid_idx))[0]
+
+ def _source_indptr(self) -> np.ndarray:
+ if self._indptrCache is None:
+ self._indptrCache = np.asarray(self.grp["indptr"][:])
+ return self._indptrCache
+
+ def _selected_cumulative_nnz(self) -> np.ndarray:
+ if self._cumulativeRowNnz is None:
+ valid_idx = self.validBarcodeIdx
+ assert valid_idx is not None
+ row_nnz = np.diff(self._source_indptr())[valid_idx]
+ cumulative = np.empty(row_nnz.size + 1, dtype=np.int64)
+ cumulative[0] = 0
+ np.cumsum(row_nnz, dtype=np.int64, out=cumulative[1:])
+ self._cumulativeRowNnz = cumulative
+ return self._cumulativeRowNnz
+
+ @property
+ def matrix_dtype(self) -> np.dtype[Any]:
+ dtype: np.dtype[Any] = np.dtype(self.grp["data"].dtype)
+ return dtype
+
+ def _read_dataset(self, key: str | None = None) -> list[str]:
+ if key is None:
+ raise ValueError("Dataset key must be provided")
+ grp_key = self.grpNames[key]
+ return [x.decode("UTF-8") for x in self.grp[grp_key][:]]
+
+ def cell_names(self) -> list[str]:
+ """Returns a list of names of the cells in the dataset."""
+ vals = np.array(self._read_dataset("cell_names"))
+ if self.validBarcodeIdx is not None:
+ vals = vals[self.validBarcodeIdx]
+ return list(vals)
+
+ def get_feature_columns(self) -> Iterator[tuple[str, np.ndarray]]:
+ yield from super().get_feature_columns()
+ if "features" not in self.grp:
+ return
+ features = self.grp["features"]
+ if not isinstance(features, h5py.Group) or "_all_tag_keys" not in features:
+ return
+ raw_keys = np.asarray(features["_all_tag_keys"][:]).reshape(-1)
+ for raw_key in raw_keys:
+ key = (
+ raw_key.decode("utf-8")
+ if isinstance(raw_key, bytes | np.bytes_)
+ else str(raw_key)
+ )
+ if key in {"id", "name", "feature_type"} or key not in features:
+ continue
+ values = features[key]
+ if not isinstance(values, h5py.Dataset):
+ continue
+ if values.ndim != 1 or int(values.shape[0]) != self.nFeatures:
+ raise ValueError(
+ f"10x feature tag {key!r} has shape {values.shape}; "
+ f"expected ({self.nFeatures},)"
+ )
+ yield key, np.asarray(values[:])
+
+ # noinspection DuplicatedCode
+ def consume(
+ self, batch_size: int, lines_in_mem: int | None = None
+ ) -> Generator[coo_matrix, None, None]:
+ """Yield CSR chunks from the Cell Ranger H5 matrix.
+
+ Args:
+ batch_size: Number of cells per chunk.
+ lines_in_mem: Unused; kept for CrReader API compatibility.
+ """
+ valid_idx = self.validBarcodeIdx
+ assert valid_idx is not None
+ indptr = self._source_indptr()
+ for s in range(0, len(valid_idx), batch_size):
+ v_pos = valid_idx[s : s + batch_size]
+ starts = indptr[v_pos]
+ ends = indptr[v_pos + 1]
+ counts = ends - starts
+ cell_idx = np.repeat(
+ np.arange(len(v_pos)),
+ counts,
+ )
+ nonempty = np.flatnonzero(counts)
+ idx = (
+ np.concatenate([np.arange(starts[i], ends[i]) for i in nonempty])
+ if nonempty.size
+ else np.array([], dtype=np.int64)
+ )
+ if idx.size == 0:
+ yield coo_matrix(
+ ([], ([], [])),
+ shape=(len(v_pos), self.nFeatures),
+ dtype=self.matrix_dtype,
+ )
+ continue
+ data = np.asarray(self.grp["data"][idx])
+ indices = np.asarray(self.grp["indices"][idx])
+ yield coo_matrix(
+ (data, (cell_idx, indices)), shape=(len(v_pos), self.nFeatures)
+ )
+
+ def max_window_nnz(self, window_rows: int) -> int:
+ """Return the largest selected-cell row-window nnz."""
+ if window_rows <= 0:
+ raise ValueError("window_rows must be positive")
+ valid_idx = self.validBarcodeIdx
+ assert valid_idx is not None
+ width = min(window_rows, len(valid_idx))
+ if width == 0:
+ return 0
+ cumulative = self._selected_cumulative_nnz()
+ return int(np.max(cumulative[width:] - cumulative[:-width]))
+
+ def producer_staging_bytes(
+ self,
+ batch_size: int,
+ lines_in_mem: int,
+ ) -> int:
+ """Count row-pointer arrays created while one H5 batch is produced."""
+ rows = min(max(1, int(batch_size)), self.nCells)
+ if rows == 0:
+ return 0
+ itemsize = self._source_indptr().dtype.itemsize
+ return int(rows * (3 * itemsize + np.dtype(np.int64).itemsize))
+
+ def _prepare_sparse_import(self) -> None:
+ self._source_indptr()
+ self._selected_cumulative_nnz()
+
+ def _sparse_import_resident_bytes(self) -> int:
+ arrays = (
+ self.validBarcodeIdx,
+ self._indptrCache,
+ self._cumulativeRowNnz,
+ )
+ return int(
+ sum(array.nbytes for array in arrays if isinstance(array, np.ndarray))
+ )
+
+ def close(self) -> None:
+ """Closes file connection."""
+ self.h5obj.close()
+
+
+class CrDirReader(CrReader):
+ """A class to read in CellRanger (Cr) data, in the form of a directory.
+
+ Subclass of CrReader.
+
+ Args:
+ loc (str): Path for the directory containing the cellranger output.
+ mtx_separator (str): Column delimiter in the MTX file (Default value: ' ')
+ index_offset (int): This value is added to each feature index (Default value: -1)
+
+ Attributes:
+ loc: Path for the directory containing the cellranger output.
+ matFn: The file name for the matrix file.
+ sep (str): Column delimiter in the MTX file (Default value: ' ')
+ indexOffset (int): This value is added to each feature index (Default value: -1)
+ """
+
+ def __init__(
+ self,
+ loc: str,
+ mtx_separator: str = " ",
+ index_offset: int = -1,
+ is_filtered: bool = True,
+ filtering_cutoff: int = 500,
+ ) -> None:
+ from .mtx import _MtxEngine, inspect_mtx
+
+ self.loc: str = loc.rstrip("/") + "/"
+ self.sep = mtx_separator
+ self.indexOffset = index_offset
+ candidates = inspect_mtx(loc)
+ if len(candidates) != 1:
+ raise ValueError(
+ "CrDirReader requires one complete Matrix Market triplet; "
+ "use inspect_mtx and MtxReader to select among candidates"
+ )
+ self._engine = _MtxEngine(
+ candidates[0],
+ cell_id_key=None,
+ separator=mtx_separator,
+ index_offset=index_offset,
+ is_filtered=is_filtered,
+ filtering_cutoff=filtering_cutoff,
+ temp_dir=None,
+ dtype=np.uint32,
+ )
+ self.matFn = self._engine.matrixPath
+ self.validBarcodeIdx = self._engine.validCellIndexes - self.indexOffset
+ self.matrixEntryCount = self._engine.matrixEntryCount
+ self.coordinateOrder = self._engine.coordinateOrder
+ CrReader.__init__(
+ self,
+ {
+ "feature_ids": "feature_ids",
+ "feature_names": "feature_names",
+ "feature_types": "feature_types",
+ "cell_names": "cell_names",
+ },
+ )
+ self.nCells = self._engine.nCells
+
+ def _handle_version(self) -> dict[str, str]:
+ return {
+ "feature_ids": "feature_ids",
+ "feature_names": "feature_names",
+ "feature_types": "feature_types",
+ "cell_names": "cell_names",
+ }
+
+ def _read_dataset(self, key: str | None = None) -> list[str]:
+ if key is None:
+ raise ValueError("Dataset key must be provided")
+ values = {
+ "feature_ids": self._engine.feature_ids,
+ "feature_names": self._engine.feature_names,
+ "feature_types": self._engine.feature_types,
+ "cell_names": self._engine.cell_names,
+ }
+ if key not in values:
+ raise KeyError(key)
+ return values[key]()
+
+ def read_header(self) -> pd.DataFrame:
+ return pd.DataFrame(
+ [
+ {
+ "nFeatures": self._engine.nFeatures,
+ "nCells": self._engine.rawCellCount,
+ "nCounts": self._engine.matrixEntryCount,
+ }
+ ]
+ )
+
+ def process_batch(
+ self, dfs: list[pd.DataFrame], filtering_cutoff: int
+ ) -> np.ndarray:
+ """Returns a list of valid barcodes after filtering out background barcodes for a given batch.
+
+ Args:
+ dfs: A Polar DataFrame containing a chunk of data from the MTX file.
+ filtering_cutoff: The cutoff value for filtering out background barcodes
+ """
+ merged = pd.concat(dfs, ignore_index=True)
+ summed = merged.groupby("barcode")["count"].sum()
+ valid = summed[summed > filtering_cutoff].index.to_numpy()
+ return np.sort(valid)
+
+ def _get_valid_barcodes(
+ self,
+ filtering_cutoff: int,
+ batch_size: int = int(10e3),
+ lines_in_mem: int = int(10e5),
+ ) -> np.ndarray:
+ """Returns a list of valid barcodes after filtering out background barcodes.
+
+ Args:
+ filtering_cutoff: The cutoff value for filtering out background barcodes.
+ batch_size: The number of barcodes to process in each batch.
+ lines_in_mem: The number of lines to read into memory
+ """
+ test_counter = 0
+ matrixIO = pd.read_csv(
+ self.matFn,
+ comment="%",
+ sep=self.sep,
+ header=0,
+ chunksize=lines_in_mem,
+ names=["gene", "barcode", "count"],
+ )
+
+ header = self.read_header()
+ nChunks = math.ceil(header["nCounts"][0] / lines_in_mem)
+ test_counter = 0
+ valid_idx = []
+ start = 1
+
+ dfs = []
+ for chunk in iter_progress(
+ # range(nChunks),
+ matrixIO,
+ total=nChunks,
+ desc="Filtering out background barcodes",
+ ):
+ if (
+ (chunk.iloc[-1]["barcode"] - start) >= batch_size
+ ): # If the last "cell id" is greater than the start + batch size
+ # Filter rows in the current chunk that belong to the current batch
+ idx = np.array(
+ chunk["barcode"].values < (batch_size + start)
+ ) # This is the crucial line. This makes sure that if any cell ID is spread over multiple chunks, it is not missed, as any cell ID that is less than the batch size + start is included.
+ # If no rows belong to the current batch, move to the next batch.
+ if idx.sum() == 0:
+ dfs.append(chunk)
+ start += batch_size
+ test_counter += len(chunk)
+ continue
+ # Process the rows belonging to the current batch
+ mask_pos = np.where(idx)[0]
+ mask_neg = np.where(~idx)[0]
+ dfs.append(chunk.iloc[mask_pos])
+ valid_idx.append(self.process_batch(dfs, filtering_cutoff))
+ # Prepare for the next batch
+ del dfs
+ dfs = [chunk.iloc[mask_neg]]
+ start += batch_size
+ else:
+ # If we haven't reached the batch boundary, accumulate the chunk
+ dfs.append(chunk)
+ test_counter += len(chunk)
+ # Process any remaining data after the main loop
+ if len(dfs) > 0:
+ valid_idx.append(self.process_batch(dfs, filtering_cutoff))
+ # Verify that all rows were processed
+ assert test_counter == header["nCounts"][0]
+ return np.sort(np.unique(np.hstack(valid_idx)))
+
+ def to_sparse(self, a: np.ndarray, dtype: DTypeLike) -> coo_matrix:
+ """Returns the input data as a sparse (COO) matrix.
+
+ Args:
+ a: Sparse matrix, contains a chunk of data from the MTX file.
+ dtype:
+ """
+ c = (a[:, 1] - a[0, 1]).astype(int)
+ return coo_matrix(
+ (
+ a[:, 2],
+ (
+ c,
+ (a[:, 0] + self.indexOffset).astype(int),
+ ),
+ ),
+ shape=(c[-1] + 1, self.nFeatures),
+ dtype=dtype,
+ )
+
+ def cell_names(self) -> list[str]:
+ """Returns a list of names of the cells in the dataset."""
+ return self._engine.cell_names()
+
+ def rename_batches(self, collect: list[pd.DataFrame]) -> np.ndarray:
+ df = pd.concat(collect, ignore_index=True)
+ barcodes = df["barcode"].to_numpy()
+ count_hash = {}
+ for i, x in enumerate(np.unique(barcodes)):
+ count_hash[x] = i
+ cell_idx = np.array([count_hash[x] for x in barcodes])
+ df = df.copy()
+ df["barcode"] = cell_idx
+ return np.array(df)
+
+ def producer_staging_bytes(
+ self,
+ batch_size: int,
+ lines_in_mem: int,
+ ) -> int:
+ return self._engine.producer_staging_bytes(
+ batch_size,
+ lines_in_mem,
+ )
+
+ def max_window_nnz(self, window_rows: int) -> int:
+ return self._engine.max_window_nnz(window_rows)
+
+ @property
+ def matrix_dtype(self) -> np.dtype[Any]:
+ return self._engine.matrixDtype
+
+ # noinspection DuplicatedCode
+ def consume(
+ self,
+ batch_size: int,
+ lines_in_mem: int = int(1e5),
+ dtype: DTypeLike = np.uint32,
+ ) -> Generator[coo_matrix, None, None]:
+ """Yields chunks of data from the MTX file.
+
+ Args:
+ batch_size: The number of barcodes to process in each batch.
+ lines_in_mem: The number of lines to read into memory.
+ dtype: The data type of the matrix.
+ """
+ yield from self._engine.consume(batch_size, lines_in_mem, dtype)
+
+ def get_cell_columns(self) -> Iterator[tuple[str, np.ndarray]]:
+ yield from self._engine.cell_columns()
+
+ def get_feature_columns(self) -> Iterator[tuple[str, np.ndarray]]:
+ yield from self._engine.feature_columns()
+
+ def _set_sparse_import_lines_in_mem(self, lines_in_mem: int) -> None:
+ self._engine.configure_import_lines(lines_in_mem)
+
+ def _prepare_sparse_import(self) -> None:
+ self._engine.prepare()
+
+ def _release_sparse_import(self) -> None:
+ self._engine.release()
+
+ def _sparse_import_resident_bytes(self) -> int:
+ return self._engine.resident_bytes()
+
+ def close(self) -> None:
+ self._engine.release()
diff --git a/scarf/readers/csv.py b/scarf/readers/csv.py
new file mode 100644
index 00000000..fd2427a7
--- /dev/null
+++ b/scarf/readers/csv.py
@@ -0,0 +1,185 @@
+from collections.abc import Generator
+from typing import Any
+
+import numpy as np
+import pandas as pd
+
+from ..utils.progress import iter_progress
+
+
+class CSVReader:
+ """A class to read in data from a CSV file.
+
+ Args:
+ csv_fn: Path to the CSV file
+ has_header: Does the CSV file has a header. (Default value: True)
+ id_column: The column number which contains row name. (Default value: None)
+ rows_are_cells: If True then each row represents a cell and hence each column is a feature. If False then each
+ row is feature and each column in a cell
+ sep: The column separator in the CSV file (Default value: ',')
+ skip_rows: Number of rows to skip from the top of the file. (Default value: 0)
+ skip_cols: Names of columns to skip. Must be provided as a list even if just one column.
+ (Default value: None)
+ cell_data_cols: Names of columns to include in cell metadata rather than count matrix. Must be provided as a
+ list even if just one column. (Default value: None)
+ batch_size: Number of lines to read at a time. Decrease this value if you have too many columns.
+ (Default value: 50,000)
+ pandas_kwargs: A dictionary of keyword arguments to be passed to Pandas read_csv function.
+
+ Attributes:
+ nFeatures: Number of features in dataset.
+ nCells: Number of cells in dataset.
+ """
+
+ def __init__(
+ self,
+ csv_fn: str,
+ has_header: bool = True,
+ id_column: int | None = None,
+ rows_are_cells: bool = True,
+ sep: str = ",",
+ skip_rows: int = 0,
+ skip_cols: list[str] | None = None,
+ cell_data_cols: list[str] | None = None,
+ batch_size: int = 10000,
+ pandas_kwargs: dict[str, Any] | None = None,
+ ) -> None:
+ self._fn = csv_fn
+ if rows_are_cells is False:
+ raise NotImplementedError(
+ "Currently Scarf supports only those CSV files where cells are along the rows"
+ )
+ if pandas_kwargs is None:
+ pandas_kwargs = {}
+ elif not isinstance(pandas_kwargs, dict):
+ raise TypeError("pandas_kwargs must be a dictionary")
+ header_row: int | None
+ if has_header is False:
+ header_row = None
+ else:
+ header_row = 0
+ self.pandas_kwargs: dict[str, Any] = pandas_kwargs
+ self.pandas_kwargs["sep"] = sep
+ self.pandas_kwargs["header"] = header_row
+ self.pandas_kwargs["skiprows"] = skip_rows
+ self.pandas_kwargs["chunksize"] = batch_size
+ self.pandas_kwargs["index_col"] = id_column
+
+ if skip_cols is None:
+ self.skipCols = []
+ else:
+ self.skipCols = skip_cols
+ if cell_data_cols is None:
+ self.cellDataCols = []
+ else:
+ self.cellDataCols = cell_data_cols
+ (
+ self.nCells,
+ self.nFeatures,
+ self.cellIds,
+ self.featureIds,
+ self.keepCols,
+ self.cellDataDtypes,
+ self.cellDataIdx,
+ ) = self._consistency_check()
+
+ def _get_streamer(self) -> Generator[pd.DataFrame, None, None]:
+ reader = pd.read_csv(self._fn, **self.pandas_kwargs)
+ yield from reader
+
+ def _consistency_check(
+ self,
+ ) -> tuple[
+ int,
+ int,
+ np.ndarray | None,
+ np.ndarray | None,
+ list[int] | None,
+ list[np.dtype] | None,
+ list[int] | None,
+ ]:
+ stream = self._get_streamer()
+ n_cells = 0
+ n_features = 0
+ feature_ids: np.ndarray | None = None
+ cell_data_dtypes: list[np.dtype] | None = None
+ cell_data_idx: list[int] | None = None
+ cell_ids: np.ndarray | None = None
+ collected_cell_ids: list[Any] | None = None
+ if self.pandas_kwargs["index_col"] is not None:
+ collected_cell_ids = []
+ for df in iter_progress(
+ stream,
+ desc="Checking CSV consistency",
+ ):
+ n_cells += df.shape[0]
+ if n_features == 0:
+ n_features = df.shape[1]
+ if self.pandas_kwargs["header"] is not None:
+ feature_ids = np.asarray(df.columns.values)
+ if len(feature_ids) != n_features:
+ raise ValueError(
+ "Header length not same as number of features. This can happen if you did not"
+ " skip the right number of rows."
+ )
+ if len(self.cellDataCols) > 0:
+ cell_data_dtypes = list(df[self.cellDataCols].dtypes.values)
+ cell_data_idx = [
+ n
+ for n, x in enumerate(feature_ids)
+ if x in self.cellDataCols
+ ]
+ else:
+ if n_features != df.shape[1]:
+ raise ValueError(
+ "Number of columns changed in the CSV during consistency check."
+ " Maybe a problem with the delimiter."
+ )
+ if collected_cell_ids is not None:
+ cell_ids = np.asarray(collected_cell_ids)
+ keep_cols: list[int] | None = None
+ if feature_ids is not None:
+ skip_names = list(set(self.skipCols).union(self.cellDataCols))
+ if len(skip_names) > 0:
+ keep_cols = [
+ n for n, x in enumerate(feature_ids) if x not in skip_names
+ ]
+ feature_ids = feature_ids[keep_cols]
+ n_features = len(keep_cols)
+ return (
+ n_cells,
+ n_features,
+ cell_ids,
+ feature_ids,
+ keep_cols,
+ cell_data_dtypes,
+ cell_data_idx,
+ )
+
+ def cell_ids(self) -> np.ndarray:
+ """Returns a list of cell IDs."""
+ if self.cellIds is None:
+ return np.array([f"cell_{x}" for x in range(self.nCells)])
+ else:
+ return self.cellIds
+
+ def feature_ids(self) -> np.ndarray:
+ """Returns a list of feature IDs."""
+ if self.featureIds is None:
+ return np.array([f"feature_{x}" for x in range(self.nFeatures)])
+ else:
+ return self.featureIds
+
+ def consume(self) -> Generator[tuple[np.ndarray, np.ndarray | None], None, None]:
+ """Returns a generator that yield chunks of data."""
+ stream = self._get_streamer()
+ if self.keepCols is None:
+ for df in stream:
+ yield df.values, None
+ else:
+ if self.cellDataIdx is not None:
+ for df in stream:
+ yield df.values[:, self.keepCols], df.values[:, self.cellDataIdx]
+ else:
+ for df in stream:
+ yield df.values[:, self.keepCols], None
diff --git a/scarf/readers/h5ad.py b/scarf/readers/h5ad.py
new file mode 100644
index 00000000..36c52fb7
--- /dev/null
+++ b/scarf/readers/h5ad.py
@@ -0,0 +1,829 @@
+from collections.abc import Generator, Mapping
+from dataclasses import dataclass
+from typing import Any
+
+import h5py
+import numpy as np
+from scipy.sparse import coo_matrix, csc_matrix, csr_matrix
+
+from ..utils.logging import logger
+from ..utils.progress import iter_progress
+from ._assay_names import auto_name_feat_table, make_feat_table_from_types
+from ._h5ad_inspect import H5adInspectResult, _as_text, inspect_h5ad as inspect_h5ad
+
+# AnnData writes a column as a group when it needs more than one array: a
+# categorical needs codes with categories, a pandas nullable dtype needs values
+# with a missingness mask.
+_CATEGORICAL_KEYS = frozenset({"codes", "categories"})
+_NULLABLE_KEYS = frozenset({"values", "mask"})
+
+
+def _column_encoding(node: h5py.Group) -> str:
+ encoding = node.attrs.get("encoding-type")
+ return "unknown" if encoding is None else _as_text(encoding)
+
+
+def _is_decodable_column(node: h5py.Group) -> bool:
+ keys = set(node.keys())
+ return _CATEGORICAL_KEYS.issubset(keys) or _NULLABLE_KEYS.issubset(keys)
+
+
+@dataclass(frozen=True)
+class _H5adAssayFeatures:
+ featureIndexes: np.ndarray
+ featureIds: np.ndarray
+ featureNames: np.ndarray
+
+
+class H5adReader:
+ """A class to read in data from a H5ad file (h5 file with AnnData
+ information).
+
+ Args:
+ h5ad_fn: Path to H5AD file
+ cell_attrs_key: H5 group under which cell attributes are saved.(Default value: 'obs')
+ feature_attrs_key: H5 group under which feature attributes are saved.(Default value: 'var')
+ cell_ids_key: Key in `obs` group that contains unique cell IDs. By default the index will be used.
+ feature_ids_key: Key in `var` group that contains unique feature IDs. By default the index will be used.
+ feature_name_key: Key in `var` group that contains feature names. (Default: gene_short_name)
+ matrix_key: Group where in the sparse matrix resides (default: 'X')
+ category_names_key: Looks up this group and replaces the values in `var` and 'obs' child datasets with the
+ corresponding index value within this group.
+ dtype: Numpy dtype of the matrix data. This dtype is enforced when streaming the data through `consume`
+ method. (Default value: Automatically determined)
+
+ Attributes:
+ h5: A File object from the h5py package.
+ matrixKey: Group where in the sparse matrix resides (default: 'X')
+ cellAttrsKey: Group wherein the cell attributes are present
+ featureAttrsKey: Group wherein the feature attributes are present
+ groupCodes: Used to ensure compatibility with different AnnData versions.
+ nFeatures: Number of features in dataset.
+ nCells: Number of cells in dataset.
+ cellIdsKey: Key in `obs` group that contains unique cell IDs. By default the index will be used.
+ featIdsKey: Key in `var` group that contains unique feature IDs. By default the index will be used.
+ featNamesKey: Key in `var` group that contains feature names. (Default: gene_short_name)
+ catNamesKey: Looks up this group and replaces the values in `var` and 'obs' child datasets with the
+ corresponding index value within this group.
+ matrixDtype: dtype of the matrix containing the data (as indicated by matrix_key)
+ """
+
+ def __init__(
+ self,
+ h5ad_fn: str,
+ cell_attrs_key: str = "obs",
+ cell_ids_key: str = "_index",
+ feature_attrs_key: str = "var",
+ feature_ids_key: str = "_index",
+ feature_name_key: str = "gene_short_name",
+ matrix_key: str = "X",
+ obsm_attrs_key: str = "obsm",
+ category_names_key: str = "__categories",
+ dtype: str | None = None,
+ ) -> None:
+ self.h5: h5py.File = h5py.File(h5ad_fn, mode="r")
+ self.matrixKey = matrix_key
+ self.cellAttrsKey, self.featureAttrsKey, self.obsmAttrsKey = (
+ cell_attrs_key,
+ feature_attrs_key,
+ obsm_attrs_key,
+ )
+ self.groupCodes: dict[str, int] = {
+ self.cellAttrsKey: self._validate_group(self.cellAttrsKey),
+ self.featureAttrsKey: self._validate_group(self.featureAttrsKey),
+ self.obsmAttrsKey: self._validate_group(self.obsmAttrsKey),
+ self.matrixKey: self._validate_group(self.matrixKey),
+ }
+ self.matrixOrientation = self._validate_sparse_matrix()
+ self._convertedCsr: csr_matrix | None = None
+ self._indptrCache: np.ndarray | None = None
+ self._cumulativeRowNnz: np.ndarray | None = None
+ self.nCells, self.nFeatures = (
+ self._get_n(self.cellAttrsKey),
+ self._get_n(self.featureAttrsKey),
+ )
+ self.cellIdsKey = self._fix_name_key(self.cellAttrsKey, cell_ids_key)
+ self.featIdsKey = self._fix_name_key(self.featureAttrsKey, feature_ids_key)
+ self.featNamesKey = feature_name_key
+ self.catNamesKey = category_names_key
+ self.sourceMatrixDtype: Any = self._get_matrix_dtype()
+ self.matrixDtype: Any = self.sourceMatrixDtype if dtype is None else dtype
+ self.storageDtype: Any = self.matrixDtype
+ self._dtypeOverridden = dtype is not None
+
+ @classmethod
+ def from_inspect(
+ cls,
+ inspection: H5adInspectResult,
+ **overrides: Any,
+ ) -> "H5adReader":
+ reader_kwargs = inspection.to_reader_kwargs()
+ reader_kwargs.update(overrides)
+ return cls(**reader_kwargs)
+
+ def _validate_sparse_matrix(self) -> str:
+ if self.groupCodes[self.matrixKey] != 2:
+ return "dense"
+
+ group = self.h5[self.matrixKey]
+ if not isinstance(group, h5py.Group):
+ return "dense"
+
+ required = {"data", "indices", "indptr"}
+ missing = required.difference(group.keys())
+ if missing:
+ raise ValueError(
+ f"ERROR: Sparse matrix group `{self.matrixKey}` is missing: "
+ f"{', '.join(sorted(missing))}"
+ )
+
+ encoding = group.attrs.get("encoding-type")
+ if encoding is None:
+ encoding = group.attrs.get("h5sparse_format")
+ if encoding is None:
+ logger.warning(
+ f"Sparse matrix group `{self.matrixKey}` has no sparse encoding; "
+ "assuming legacy CSR encoding"
+ )
+ return "csr"
+ if isinstance(encoding, bytes | np.bytes_):
+ encoding = encoding.decode("utf-8")
+ normalized = str(encoding).lower()
+ if normalized in {"csr", "csr_matrix"}:
+ return "csr"
+ if normalized in {"csc", "csc_matrix"}:
+ return "csc"
+ raise ValueError(
+ f"ERROR: Sparse matrix encoding `{encoding}` is not supported. "
+ "H5adReader supports CSR and CSC encoding."
+ )
+
+ def _validate_group(self, group: str) -> int:
+ if group not in self.h5:
+ logger.warning(f"`{group}` group not found in the H5ad file")
+ ret_val = 0
+ elif isinstance(self.h5[group], h5py.Dataset):
+ ret_val = 1
+ elif isinstance(self.h5[group], h5py.Group):
+ ret_val = 2
+ else:
+ logger.warning(
+ f"`{group}` slot in H5ad file is not of Dataset or Group type. "
+ f"Due to this, no information in `{group}` can be used"
+ )
+ ret_val = 0
+ if ret_val == 2:
+ if len(self.h5[group].keys()) == 0:
+ logger.warning(f"`{group}` slot in H5ad file is empty.")
+ ret_val = 0
+ elif (
+ len(
+ set(
+ [
+ self.h5[group][x].shape[0]
+ for x in self.h5[group].keys()
+ if isinstance(self.h5[group][x], h5py.Dataset)
+ ]
+ )
+ )
+ > 1
+ ):
+ if sorted(self.h5[group].keys()) != ["data", "indices", "indptr"]:
+ logger.warning(
+ f"`{group}` slot in H5ad file has unequal sized child groups"
+ )
+ return ret_val
+
+ def _get_matrix_dtype(self) -> Any:
+ if self.groupCodes[self.matrixKey] == 1:
+ return self.h5[self.matrixKey].dtype
+ elif self.groupCodes[self.matrixKey] == 2:
+ return self.h5[self.matrixKey]["data"].dtype
+ else:
+ raise ValueError(
+ f"ERROR: {self.matrixKey} is neither Dataset or Group type. Will not consume data"
+ )
+
+ def _matrix_shape(self) -> tuple[int, int]:
+ matrix = self.h5[self.matrixKey]
+ if isinstance(matrix, h5py.Dataset):
+ return int(matrix.shape[0]), int(matrix.shape[1])
+
+ shape: Any = matrix.attrs.get("shape")
+ if shape is None:
+ shape = matrix.attrs.get("h5sparse_shape")
+ if shape is None and "shape" in matrix:
+ shape_node = matrix["shape"]
+ if isinstance(shape_node, h5py.Dataset):
+ shape = shape_node[:]
+ if shape is not None:
+ values = np.asarray(shape).reshape(-1)
+ if values.size == 2:
+ return int(values[0]), int(values[1])
+
+ compressed_axis = int(matrix["indptr"].shape[0] - 1)
+ indices = matrix["indices"]
+ observed_axis = int(np.max(indices[:])) + 1 if indices.shape[0] else 0
+ if self.matrixOrientation == "csr":
+ return compressed_axis, observed_axis
+ return observed_axis, compressed_axis
+
+ def _check_exists(self, group: str, key: str) -> bool:
+ if group in self.groupCodes:
+ group_code = self.groupCodes[group]
+ else:
+ group_code = self._validate_group(group)
+ self.groupCodes[group] = group_code
+ if group_code == 1:
+ if key in list(self.h5[group].dtype.names):
+ return True
+ if group_code == 2:
+ if key in self.h5[group].keys():
+ return True
+ return False
+
+ def _fix_name_key(self, group: str, key: str) -> str:
+ if self._check_exists(group, key) is False:
+ if key.startswith("_"):
+ temp_key = key[1:]
+ if self._check_exists(group, temp_key):
+ return temp_key
+ return key
+
+ def _get_n(self, group: str) -> int:
+ if self.groupCodes[group] == 0:
+ matrix_shape = self._matrix_shape()
+ return matrix_shape[0 if group == self.cellAttrsKey else 1]
+ elif self.groupCodes[group] == 1:
+ return int(self.h5[group].shape[0])
+ else:
+ for i in self.h5[group].keys():
+ node = self.h5[group][i]
+ if isinstance(node, h5py.Dataset):
+ return int(node.shape[0])
+ if not isinstance(node, h5py.Group):
+ continue
+ # Group encoded columns carry the axis length in their codes
+ # (categorical) or values (pandas nullable) array.
+ if "codes" in node:
+ return int(node["codes"].shape[0])
+ if _NULLABLE_KEYS.issubset(node.keys()):
+ return int(node["values"].shape[0])
+ raise KeyError(
+ f"ERROR: `{group}` key doesn't contain any child node of Dataset type."
+ f"Aborting because unexpected H5ad format."
+ )
+
+ def cell_ids(self) -> np.ndarray:
+ """Returns a list of cell IDs."""
+ if self._check_exists(self.cellAttrsKey, self.cellIdsKey):
+ values = self.h5[self.cellAttrsKey][self.cellIdsKey]
+ return self._replace_category_values(
+ values, self.cellIdsKey, self.cellAttrsKey
+ ).astype(object)
+ logger.warning(
+ f"Cell ID key {self.cellIdsKey!r} was not found in H5AD obs; "
+ "generated IDs will be used"
+ )
+ return np.array([f"cell_{x}" for x in range(self.nCells)])
+
+ # noinspection DuplicatedCode
+ def feat_ids(self) -> np.ndarray:
+ """Returns a list of feature IDs."""
+ if self._check_exists(self.featureAttrsKey, self.featIdsKey):
+ values = self.h5[self.featureAttrsKey][self.featIdsKey]
+ return self._replace_category_values(
+ values, self.featIdsKey, self.featureAttrsKey
+ ).astype(object)
+ logger.warning(
+ f"Feature ID key {self.featIdsKey!r} was not found in "
+ f"{self.featureAttrsKey}; generated IDs will be used"
+ )
+ return np.array([f"feature_{x}" for x in range(self.nFeatures)])
+
+ # noinspection DuplicatedCode
+ def feat_names(self) -> np.ndarray:
+ """Returns a list of feature names."""
+ if self._check_exists(self.featureAttrsKey, self.featNamesKey):
+ values = self.h5[self.featureAttrsKey][self.featNamesKey]
+ return self._replace_category_values(
+ values, self.featNamesKey, self.featureAttrsKey
+ ).astype(object)
+ logger.warning(
+ f"Feature name key {self.featNamesKey!r} was not found in "
+ f"{self.featureAttrsKey}; feature IDs will be used"
+ )
+ return self.feat_ids()
+
+ def _replace_category_values(
+ self, v: np.ndarray | h5py.Group | h5py.Dataset, key: str, group: str
+ ) -> np.ndarray:
+ if isinstance(v, h5py.Group):
+ if _CATEGORICAL_KEYS.issubset(v.keys()):
+ codes = v["codes"][:]
+ categories = v["categories"][:]
+ valid = (codes >= 0) & (codes < len(categories))
+ decoded = np.empty(codes.shape, dtype=object)
+ decoded[valid] = categories[codes[valid]]
+ decoded[~valid] = None
+ return decoded
+ if _NULLABLE_KEYS.issubset(v.keys()):
+ return self._decode_nullable(v, key)
+ logger.warning(
+ f"Column {key!r} in {group} uses the H5AD encoding "
+ f"{_column_encoding(v)!r}, which cannot be decoded"
+ )
+ return np.array([], dtype=object)
+
+ # if v is a Dataset
+ if isinstance(v, h5py.Dataset):
+ v = v[:]
+
+ if self.catNamesKey is not None:
+ if self._check_exists(group, self.catNamesKey):
+ cat_g = self.h5[group][self.catNamesKey]
+ if isinstance(cat_g, h5py.Group):
+ if key in cat_g:
+ return self._decode_legacy_categories(v, cat_g[key][:])
+ if "uns" in self.h5:
+ if key + "_categories" in self.h5["uns"]:
+ categories = self.h5["uns"][key + "_categories"][:]
+ return self._decode_legacy_categories(v, categories)
+ return np.asarray(v)
+
+ @staticmethod
+ def _decode_nullable(v: h5py.Group, key: str) -> np.ndarray:
+ """Decode a pandas nullable column without losing numeric semantics."""
+ values = np.asarray(v["values"][:])
+ mask = np.asarray(v["mask"][:])
+ if mask.shape != values.shape:
+ logger.warning(
+ f"Column {key!r} has a missingness mask of shape {mask.shape} "
+ f"for {values.shape} values; the mask will be ignored"
+ )
+ return values
+ mask = mask.astype(bool, copy=False)
+ if not mask.any():
+ # Nothing is missing, so the native dtype survives the round trip.
+ return values
+ if values.dtype.kind in "iuf":
+ decoded = values.astype(np.float64)
+ decoded[mask] = np.nan
+ return decoded
+ decoded = np.empty(values.shape, dtype=object)
+ decoded[~mask] = values[~mask]
+ decoded[mask] = None
+ return decoded
+
+ @staticmethod
+ def _decode_legacy_categories(
+ codes: np.ndarray, categories: np.ndarray
+ ) -> np.ndarray:
+ values = np.asarray(codes)
+ if not np.issubdtype(values.dtype, np.integer):
+ return values
+ try:
+ # Negative codes mark missing values in legacy AnnData categoricals;
+ # they must decode to None rather than wrap to the final category.
+ return np.array([None if code < 0 else categories[code] for code in values])
+ except (IndexError, TypeError):
+ return values
+
+ def _get_col_data(
+ self, group: str, ignore_keys: list[str]
+ ) -> Generator[tuple[str, np.ndarray], None, None]:
+ if self.groupCodes[group] == 1:
+ for i in iter_progress(
+ self.h5[group].dtype.names,
+ desc=f"Reading attributes from group {group}",
+ ):
+ if i in ignore_keys:
+ continue
+ yield i, self._replace_category_values(self.h5[group][i][:], i, group)
+ if self.groupCodes[group] == 2:
+ for i in iter_progress(
+ self.h5[group].keys(), desc=f"Reading attributes from group {group}"
+ ):
+ if i in ignore_keys:
+ continue
+ values = self.h5[group][i]
+ if not isinstance(values, h5py.Dataset | h5py.Group):
+ continue
+ if isinstance(values, h5py.Group) and not _is_decodable_column(values):
+ logger.warning(
+ f"Skipping {group} column {i!r} because its H5AD encoding "
+ f"{_column_encoding(values)!r} is not supported"
+ )
+ continue
+ yield (
+ i,
+ self._replace_category_values(values, i, group),
+ )
+
+ def _get_obsm_data(
+ self, group: str
+ ) -> Generator[tuple[str, np.ndarray], None, None]:
+ if self.groupCodes[group] == 2:
+ for i in iter_progress(
+ self.h5[group].keys(), desc=f"Reading attributes from group {group}"
+ ):
+ g = self.h5[group][i]
+ if not isinstance(g, h5py.Dataset):
+ logger.warning(
+ f"Skipping H5AD slot {i!r} because only dense "
+ f"{group} arrays can be imported"
+ )
+ continue
+ if g.shape[0] != self.nCells:
+ logger.warning(
+ f"Skipping H5AD slot {i!r} with unexpected shape {g.shape}"
+ )
+ continue
+ for j in range(g.shape[1]):
+ yield f"{i}{j + 1}", g[:, j]
+ else:
+ logger.warning(
+ "H5AD obsm is missing or has an unsupported format; "
+ "embeddings will be skipped"
+ )
+
+ def get_cell_columns(self) -> Generator[tuple[str, np.ndarray], None, None]:
+ """Creates a Generator that yields the cell columns."""
+ for i, j in self._get_col_data(
+ self.cellAttrsKey, [self.cellIdsKey, self.catNamesKey]
+ ):
+ yield i, j
+ for i, j in self._get_obsm_data(self.obsmAttrsKey):
+ yield i, j
+
+ def get_feat_columns(self) -> Generator[tuple[str, np.ndarray], None, None]:
+ """Creates a Generator that yields the feature columns."""
+ for i, j in self._get_col_data(
+ self.featureAttrsKey,
+ [self.featIdsKey, self.featNamesKey, self.catNamesKey],
+ ):
+ yield i, j
+
+ def feature_types(self, key: str) -> list[str]:
+ """Return decoded feature types from a var column."""
+ if not self._check_exists(self.featureAttrsKey, key):
+ raise KeyError(
+ f"Feature type key `{key}` was not found in {self.featureAttrsKey}"
+ )
+ values = self._replace_category_values(
+ self.h5[self.featureAttrsKey][key],
+ key,
+ self.featureAttrsKey,
+ )
+ if values.ndim != 1 or len(values) != self.nFeatures:
+ raise ValueError(
+ f"Feature type key `{key}` has {len(values)} values; "
+ f"expected {self.nFeatures}"
+ )
+ return [
+ value.decode("utf-8")
+ if isinstance(value, bytes | np.bytes_)
+ else str(value)
+ for value in values
+ ]
+
+ def assay_feature_slices(
+ self,
+ key: str,
+ name_map: Mapping[str, str] | None = None,
+ ) -> dict[str, _H5adAssayFeatures]:
+ """Resolve feature ranges and metadata for each assay."""
+ assay_table = auto_name_feat_table(
+ make_feat_table_from_types(self.feature_types(key)),
+ name_map,
+ )
+ feature_ids = self.feat_ids()
+ feature_names = self.feat_names()
+ assays: dict[str, _H5adAssayFeatures] = {}
+ for assay_name in dict.fromkeys(assay_table.columns):
+ selected = assay_table[assay_name]
+ ranges: tuple[tuple[int, int], ...]
+ if selected.ndim == 1:
+ ranges = ((int(selected.loc["start"]), int(selected.loc["end"])),)
+ else:
+ ranges = tuple(
+ (int(start), int(end))
+ for start, end in zip(
+ selected.loc["start"],
+ selected.loc["end"],
+ strict=True,
+ )
+ )
+ indexes = np.concatenate(
+ [np.arange(start, end, dtype=np.int64) for start, end in ranges]
+ )
+ assays[str(assay_name)] = _H5adAssayFeatures(
+ featureIndexes=indexes,
+ featureIds=feature_ids[indexes],
+ featureNames=feature_names[indexes],
+ )
+ return assays
+
+ # noinspection DuplicatedCode
+ def consume_dataset(
+ self, batch_size: int = 1000
+ ) -> Generator[coo_matrix, None, None]:
+ """Returns a generator that yield chunks of data."""
+ dset = self.h5[self.matrixKey]
+ s = 0
+ for e in range(batch_size, dset.shape[0] + batch_size, batch_size):
+ if e > dset.shape[0]:
+ e = dset.shape[0]
+ yield coo_matrix(dset[s:e])
+ s = e
+
+ def _sparse_indices_are_strictly_sorted(self, maxValues: int) -> bool:
+ group = self.h5[self.matrixKey]
+ if not isinstance(group, h5py.Group) or maxValues < 1:
+ return False
+ indptr_node = group["indptr"]
+ indices_node = group["indices"]
+ compressed_size = int(indptr_node.size) - 1
+ start = 0
+ while start < compressed_size:
+ pointer_end = min(compressed_size, start + maxValues)
+ pointers = np.asarray(indptr_node[start : pointer_end + 1])
+ base = int(pointers[0])
+ relative = pointers - base
+ vectors = int(np.searchsorted(relative, maxValues, side="right") - 1)
+ if vectors < 1:
+ return False
+ pointers = pointers[: vectors + 1]
+ end = start + vectors
+ indices = np.asarray(indices_node[base : int(pointers[-1])])
+ offsets = pointers - base
+ for left, right in zip(offsets[:-1], offsets[1:], strict=True):
+ vector = indices[int(left) : int(right)]
+ if vector.size > 1 and np.any(vector[1:] <= vector[:-1]):
+ return False
+ start = end
+ return True
+
+ def infer_storage_dtype(self, maxScanBytes: int = 64 * 1024 * 1024) -> Any:
+ """Resolve the smallest lossless storage dtype."""
+ if (
+ self._dtypeOverridden
+ or self.groupCodes[self.matrixKey] != 2
+ or np.dtype(self.matrixDtype).kind != "f"
+ ):
+ return self.storageDtype
+ group = self.h5[self.matrixKey]
+ if not isinstance(group, h5py.Group):
+ return self.storageDtype
+
+ data_node = group["data"]
+ indices_node = group["indices"]
+ bytes_per_value = max(
+ 64,
+ 3 * int(data_node.dtype.itemsize)
+ + 3 * int(indices_node.dtype.itemsize)
+ + int(group["indptr"].dtype.itemsize),
+ )
+ check_values = min(
+ 1024 * 1024,
+ max(0, int(maxScanBytes)) // bytes_per_value,
+ )
+ if not self._sparse_indices_are_strictly_sorted(check_values):
+ logger.debug(
+ "Keeping the H5AD source dtype because sparse coordinates are "
+ "not canonical within the dtype-scan memory limit"
+ )
+ return self.storageDtype
+
+ finite = True
+ integral = True
+ minimum = np.inf
+ maximum = -np.inf
+ for start in range(0, data_node.size, check_values):
+ values = np.asarray(data_node[start : start + check_values])
+ if not values.size:
+ continue
+ finite = finite and bool(np.isfinite(values).all())
+ integral = integral and bool(np.equal(values, np.trunc(values)).all())
+ minimum = min(minimum, float(values.min()))
+ maximum = max(maximum, float(values.max()))
+
+ source_dtype = np.dtype(self.matrixDtype)
+ storage_dtype = source_dtype
+ if finite and integral and minimum >= 0:
+ for candidate in (
+ np.dtype("uint8"),
+ np.dtype("uint16"),
+ np.dtype("uint32"),
+ ):
+ if (
+ maximum <= np.iinfo(candidate).max
+ and candidate.itemsize < source_dtype.itemsize
+ ):
+ storage_dtype = candidate
+ break
+
+ self.storageDtype = storage_dtype
+ logger.debug(f"Resolved H5AD storage dtype={storage_dtype}")
+ return storage_dtype
+
+ def csc_conversion_peak_bytes(self) -> int:
+ """Return a conservative peak estimate for one CSC to CSR conversion."""
+ if self.matrixOrientation != "csc":
+ return 0
+ group = self.h5[self.matrixKey]
+ if not isinstance(group, h5py.Group):
+ raise TypeError("CSC matrix slot must be an HDF5 group")
+ data_node = group["data"]
+ indices_node = group["indices"]
+ indptr_node = group["indptr"]
+ source = sum(
+ int(dataset.size) * int(dataset.dtype.itemsize)
+ for dataset in (data_node, indices_node, indptr_node)
+ )
+ index_itemsize = int(indices_node.dtype.itemsize)
+ if (
+ max(self.nCells, self.nFeatures, int(data_node.size))
+ <= np.iinfo(np.int32).max
+ ):
+ index_itemsize = np.dtype("int32").itemsize
+ normalized = (
+ int(data_node.size) * np.dtype(self.storageDtype).itemsize
+ + int(indices_node.size) * index_itemsize
+ + int(indptr_node.size) * index_itemsize
+ )
+ canonical_value_itemsize = max(
+ int(data_node.dtype.itemsize),
+ np.dtype(np.int64).itemsize,
+ )
+ canonicalization = int(data_node.size) * (
+ 4 * canonical_value_itemsize + 6 * np.dtype(np.int64).itemsize + 4
+ )
+ destination = (
+ int(data_node.size)
+ * (np.dtype(self.storageDtype).itemsize + index_itemsize)
+ + (self.nCells + 1) * index_itemsize
+ )
+ return int(source + normalized + canonicalization + destination)
+
+ def materialized_csr_bytes(self) -> int:
+ """Return bytes retained by the materialized CSC-to-CSR conversion."""
+ if self._convertedCsr is None:
+ return 0
+ return int(
+ self._convertedCsr.data.nbytes
+ + self._convertedCsr.indices.nbytes
+ + self._convertedCsr.indptr.nbytes
+ )
+
+ def _csr_indptr(self) -> np.ndarray | None:
+ if self.matrixOrientation == "dense":
+ return None
+ if self._convertedCsr is not None:
+ return np.asarray(self._convertedCsr.indptr)
+ if self.matrixOrientation != "csr":
+ return None
+ if self._indptrCache is None:
+ self._indptrCache = np.asarray(self.h5[self.matrixKey]["indptr"][:])
+ return self._indptrCache
+
+ def _row_nnz_cumulative(self) -> np.ndarray | None:
+ indptr = self._csr_indptr()
+ if indptr is None:
+ return None
+ if self._cumulativeRowNnz is None:
+ cumulative = np.empty(self.nCells + 1, dtype=np.int64)
+ cumulative[0] = 0
+ np.cumsum(np.diff(indptr), dtype=np.int64, out=cumulative[1:])
+ self._cumulativeRowNnz = cumulative
+ return self._cumulativeRowNnz
+
+ def _prepare_sparse_import(self) -> None:
+ self._row_nnz_cumulative()
+
+ def _sparse_import_resident_bytes(self) -> int:
+ total = (
+ 0 if self._cumulativeRowNnz is None else int(self._cumulativeRowNnz.nbytes)
+ )
+ if self._convertedCsr is None and self._indptrCache is not None:
+ total += int(self._indptrCache.nbytes)
+ return total
+
+ def max_batch_nnz(self, batch_size: int) -> int:
+ """Return the largest contiguous row-window nnz without loading values."""
+ if batch_size <= 0:
+ raise ValueError("batch_size must be positive")
+ batch_rows = min(batch_size, self.nCells)
+ if self.matrixOrientation == "dense":
+ return int(batch_rows * self.nFeatures)
+ cumulative = self._row_nnz_cumulative()
+ if cumulative is None:
+ return int(batch_rows * self.nFeatures)
+
+ if self.nCells == 0:
+ return 0
+ return int(np.max(cumulative[batch_rows:] - cumulative[:-batch_rows]))
+
+ def max_batch_nnz_peak_bytes(self) -> int:
+ """Bound temporary row-pointer arrays used to plan sparse batches."""
+ if self.matrixOrientation == "dense":
+ return 0
+ self._prepare_sparse_import()
+ return self._sparse_import_resident_bytes()
+
+ def producer_batch_staging_bytes(self, batch_size: int) -> int:
+ """Bound sparse row pointers retained while one batch is produced."""
+ rows = min(max(1, int(batch_size)), self.nCells)
+ if rows == 0 or self.matrixOrientation == "dense":
+ return 0
+ if self._convertedCsr is not None:
+ itemsize = np.asarray(self._convertedCsr.indptr).dtype.itemsize
+ else:
+ itemsize = self.h5[self.matrixKey]["indptr"].dtype.itemsize
+ normalized_itemsize = np.dtype(np.int32).itemsize
+ return int((rows + 1) * (2 * itemsize + normalized_itemsize))
+
+ def materialize_csc(self) -> None:
+ """Convert the complete CSC source to CSR once."""
+ from ..utils.arrays import canonicalize_sparse
+
+ if self.matrixOrientation != "csc" or self._convertedCsr is not None:
+ return
+ group = self.h5[self.matrixKey]
+ if not isinstance(group, h5py.Group):
+ raise TypeError("CSC matrix slot must be an HDF5 group")
+ data_node = group["data"]
+ data = np.asarray(data_node[:])
+ indices = np.asarray(group["indices"][:])
+ indptr = np.asarray(group["indptr"][:])
+ if max(self.nCells, self.nFeatures, data.size) <= np.iinfo(np.int32).max:
+ indices = indices.astype(np.int32, copy=False)
+ indptr = indptr.astype(np.int32, copy=False)
+
+ source = csc_matrix(
+ (data, indices, indptr),
+ shape=(self.nCells, self.nFeatures),
+ )
+ canonical = canonicalize_sparse(
+ source.tocoo(copy=False),
+ self.storageDtype,
+ )
+ self._convertedCsr = canonical.tocsr()
+ logger.debug(
+ f"Materialized H5AD CSR matrix for conversion with "
+ f"dtype={self.storageDtype}"
+ )
+
+ def consume_group(self, batch_size: int) -> Generator[coo_matrix, None, None]:
+ """Returns a generator that yield chunks of data."""
+ if batch_size <= 0:
+ raise ValueError("batch_size must be positive")
+
+ if self._convertedCsr is not None or self.matrixOrientation == "csc":
+ yield from self._consume_converted_csr(batch_size)
+ return
+
+ grp = self.h5[self.matrixKey]
+ source_indptr = self._csr_indptr()
+ if source_indptr is None:
+ raise RuntimeError("CSR row pointers are unavailable")
+ for row_start in range(0, self.nCells, batch_size):
+ row_end = min(row_start + batch_size, self.nCells)
+ indptr = source_indptr[row_start : row_end + 1]
+ data_start = int(indptr[0])
+ data_end = int(indptr[-1])
+ local_indptr = indptr - data_start
+ n_rows = row_end - row_start
+ batch = csr_matrix(
+ (
+ np.asarray(grp["data"][data_start:data_end]),
+ np.asarray(grp["indices"][data_start:data_end]),
+ local_indptr,
+ ),
+ shape=(n_rows, self.nFeatures),
+ )
+ yield batch.tocoo(copy=False)
+
+ def _consume_converted_csr(
+ self,
+ batch_size: int,
+ ) -> Generator[coo_matrix, None, None]:
+ """Convert the complete CSC matrix once before yielding row batches."""
+ if self._convertedCsr is None:
+ self.materialize_csc()
+ if self._convertedCsr is None:
+ raise RuntimeError("CSC materialization did not produce a CSR matrix")
+
+ for row_start in range(0, self.nCells, batch_size):
+ row_end = min(row_start + batch_size, self.nCells)
+ yield self._convertedCsr[row_start:row_end].tocoo(copy=False)
+
+ def consume(self, batch_size: int) -> Generator[coo_matrix, None, None]:
+ """Returns a generator that yield chunks of data."""
+ if self.groupCodes[self.matrixKey] == 1:
+ return self.consume_dataset(batch_size)
+ elif self.groupCodes[self.matrixKey] == 2:
+ return self.consume_group(batch_size)
+ raise ValueError(
+ f"ERROR: {self.matrixKey} is neither Dataset or Group type. Will not consume data"
+ )
diff --git a/scarf/readers/loom.py b/scarf/readers/loom.py
new file mode 100644
index 00000000..8f98b82f
--- /dev/null
+++ b/scarf/readers/loom.py
@@ -0,0 +1,166 @@
+from collections.abc import Generator
+
+import h5py
+import numpy as np
+from scipy.sparse import coo_matrix
+
+from ..utils.logging import logger
+from ..utils.progress import iter_progress
+
+
+class LoomReader:
+ """A class to read in data in the form of a Loom file.
+
+ Args:
+ loom_fn: Path to loom format file.
+ matrix_key: Child node under HDF5 file root wherein the chunked matrix is stored. (Default value: matrix).
+ This matrix is expected to be of form (nFeatures x nCells)
+ cell_attrs_key: Child node under the HDF5 file wherein the cell attributes are stored.
+ (Default value: col_attrs)
+ cell_names_key: Child node under the `cell_attrs_key` wherein the cell names are stored.
+ (Default value: obs_names)
+ feature_attrs_key: Child node under the HDF5 file wherein the feature/gene attributes are stored.
+ (Default value: row_attrs)
+ feature_names_key: Child node under the `feature_attrs_key` wherein the feature/gene names are stored.
+ (Default value: var_names)
+ feature_ids_key: Child node under the `feature_attrs_key` wherein the feature/gene ids are stored.
+ (Default value: None)
+ dtype: Numpy dtype of the matrix data. This dtype is enforced when streaming the data through `consume`
+ method. (Default value: Automatically determined)
+
+ Attributes:
+ h5: A File object from the h5py package.
+ matrixKey: Child node under HDF5 file root wherein the chunked matrix is stored.
+ cellAttrsKey: Child node under the HDF5 file wherein the cell attributes are stored.
+ featureAttrsKey: Child node under the HDF5 file wherein the feature/gene attributes are stored.
+ cellNamesKey: Child node under the `cell_attrs_key` wherein the cell names are stored.
+ featureNamesKey: Child node under the `feature_attrs_key` wherein the feature/gene names are stored.
+ featureIdsKey: Child node under the `feature_attrs_key` wherein the feature/gene ids are stored.
+ matrixDtype: Numpy dtype of the matrix data.
+ nFeatures: Number of features in dataset.
+ nCells: Number of cells in dataset.
+ """
+
+ def __init__(
+ self,
+ loom_fn: str,
+ matrix_key: str = "matrix",
+ cell_attrs_key: str = "col_attrs",
+ cell_names_key: str = "obs_names",
+ feature_attrs_key: str = "row_attrs",
+ feature_names_key: str = "var_names",
+ feature_ids_key: str | None = None,
+ dtype: str | None = None,
+ ) -> None:
+ self.h5: h5py.File = h5py.File(loom_fn, mode="r")
+ self.matrixKey = matrix_key
+ self.cellAttrsKey, self.featureAttrsKey = cell_attrs_key, feature_attrs_key
+ self.cellNamesKey, self.featureNamesKey = cell_names_key, feature_names_key
+ self.featureIdsKey = feature_ids_key
+ self.sourceMatrixDtype = self.h5[self.matrixKey].dtype
+ self.matrixDtype = self.sourceMatrixDtype if dtype is None else dtype
+ self._check_integrity()
+ self.nFeatures, self.nCells = self.h5[self.matrixKey].shape
+
+ def _check_integrity(self) -> bool:
+ if self.matrixKey not in self.h5:
+ raise KeyError(
+ f"ERROR: Matrix key (location): {self.matrixKey} is missing in the H5 file"
+ )
+ if self.cellAttrsKey not in self.h5:
+ logger.warning(
+ f"Cell attributes are missing. Key {self.cellAttrsKey} was not found"
+ )
+ if self.featureAttrsKey not in self.h5:
+ logger.warning(
+ f"Feature attributes are missing. Key {self.featureAttrsKey} was not found"
+ )
+ return True
+
+ def cell_names(self) -> list[str]:
+ """Returns a list of names of the cells in the dataset."""
+ if self.cellAttrsKey not in self.h5:
+ pass
+ elif self.cellNamesKey not in self.h5[self.cellAttrsKey]:
+ logger.warning(
+ f"Cell names/ids key ({self.cellNamesKey}) is missing in attributes"
+ )
+ else:
+ return list(self.h5[self.cellAttrsKey][self.cellNamesKey][:])
+ return [f"cell_{x}" for x in range(self.nCells)]
+
+ def cell_ids(self) -> list[str]:
+ """Returns a list of cell IDs."""
+ return self.cell_names()
+
+ def _stream_attrs(
+ self, key: str, ignore: str | list[str] | None
+ ) -> Generator[tuple[str, np.ndarray], None, None]:
+ ignored: set[str]
+ if isinstance(ignore, str):
+ ignored = {ignore}
+ elif ignore is None:
+ ignored = set()
+ else:
+ ignored = {name for name in ignore if name is not None}
+ if key in self.h5:
+ for i in iter_progress(
+ self.h5[key].keys(),
+ desc=f"Reading {key} attributes",
+ ):
+ if i in ignored:
+ continue
+ vals = self.h5[key][i][:]
+ if vals.dtype.names is None:
+ yield i, vals
+ else:
+ # Attribute is a structured array
+ for j in vals.dtype.names:
+ yield i + "_" + str(j), vals[j]
+
+ def get_cell_attrs(self) -> Generator[tuple[str, np.ndarray], None, None]:
+ """Returns a Generator that yields the cells' attributes."""
+ return self._stream_attrs(self.cellAttrsKey, [self.cellNamesKey])
+
+ def feature_names(self) -> list[str]:
+ """Returns a list of feature names."""
+ if self.featureAttrsKey not in self.h5:
+ pass
+ elif self.featureNamesKey not in self.h5[self.featureAttrsKey]:
+ logger.warning(
+ f"Feature names key ({self.featureNamesKey}) is missing in attributes"
+ )
+ else:
+ return list(self.h5[self.featureAttrsKey][self.featureNamesKey][:])
+ return [f"feature_{x}" for x in range(self.nFeatures)]
+
+ def feature_ids(self) -> list[str]:
+ """Returns a list of feature IDs."""
+ if self.featureAttrsKey not in self.h5:
+ pass
+ elif self.featureIdsKey is None:
+ pass
+ elif self.featureIdsKey not in self.h5[self.featureAttrsKey]:
+ logger.warning(
+ f"Feature names key ({self.featureIdsKey}) is missing in attributes"
+ )
+ else:
+ return list(self.h5[self.featureAttrsKey][self.featureIdsKey][:])
+ return [f"feature_{x}" for x in range(self.nFeatures)]
+
+ def get_feature_attrs(self) -> Generator[tuple[str, np.ndarray], None, None]:
+ """Returns a Generator that yields the features' attributes."""
+ ignore_keys = [
+ key for key in (self.featureIdsKey, self.featureNamesKey) if key is not None
+ ]
+ return self._stream_attrs(self.featureAttrsKey, ignore_keys)
+
+ def consume(self, batch_size: int = 1000) -> Generator[np.ndarray, None, None]:
+ """Returns a generator that yield chunks of data."""
+ dset = self.h5[self.matrixKey]
+ s = 0
+ for e in range(batch_size, dset.shape[1] + batch_size, batch_size):
+ if e > dset.shape[1]:
+ e = dset.shape[1]
+ yield coo_matrix(dset[:, s:e]).T.astype(self.matrixDtype)
+ s = e
diff --git a/scarf/readers/mtx.py b/scarf/readers/mtx.py
new file mode 100644
index 00000000..895e3f47
--- /dev/null
+++ b/scarf/readers/mtx.py
@@ -0,0 +1,1601 @@
+import csv
+import gc
+import gzip
+import re
+import shutil
+import stat
+import tempfile
+import zipfile
+from collections.abc import Callable, Generator, Iterator
+from contextlib import contextmanager
+from dataclasses import dataclass, replace
+from io import TextIOWrapper
+from pathlib import Path, PurePosixPath
+from typing import Any, Literal, TextIO
+
+import numpy as np
+import pandas as pd
+from numpy.typing import DTypeLike
+from scipy.sparse import coo_matrix, csr_matrix
+
+from .cellranger import CrReader
+
+type MatrixOrientation = Literal["featuresByCells", "cellsByFeatures"]
+type CoordinateOrder = Literal["cellMajor", "featureMajor"]
+
+
+@dataclass(frozen=True, slots=True)
+class MtxCandidate:
+ """One complete Matrix Market matrix, feature, and cell selection."""
+
+ source: str
+ matrixPath: str
+ featurePath: str
+ cellPath: str
+ matrixOrientation: MatrixOrientation
+ nCells: int
+ nFeatures: int
+ nEntries: int
+ archivePath: str | None = None
+ cellMetadataPath: str | None = None
+ featureReferencePath: str | None = None
+ cellIdKeys: tuple[str, ...] = ()
+ relatedFiles: tuple[str, ...] = ()
+
+
+@dataclass(frozen=True, slots=True)
+class _MtxHeader:
+ field: Literal["integer", "real"]
+ nRows: int
+ nColumns: int
+ nEntries: int
+
+
+@dataclass(frozen=True, slots=True)
+class _ResolvedTriplet:
+ matrix: str
+ features: str
+ cells: str
+ parseLayout: bool
+ cellMetadata: str | None
+ featureReference: str | None
+ related: tuple[str, ...]
+
+
+def _without_gzip(name: str) -> str:
+ return name[:-3] if name.lower().endswith(".gz") else name
+
+
+def _matrix_basename(name: str) -> str:
+ return PurePosixPath(_without_gzip(name)).name.lower()
+
+
+def _is_matrix_path(name: str) -> bool:
+ lowered = name.lower()
+ return lowered.endswith(".mtx") or lowered.endswith(".mtx.gz")
+
+
+def _is_parse_matrix(name: str) -> bool:
+ return _matrix_basename(name) in {"count_matrix.mtx", "dge.mtx"}
+
+
+def _matrix_prefix(name: str) -> str:
+ stem = PurePosixPath(_without_gzip(name)).stem
+ lowered = stem.lower()
+ if lowered == "matrix":
+ return ""
+ if lowered.endswith("matrix"):
+ return stem[: -len("matrix")]
+ return ""
+
+
+def _same_parent_names(
+ names: tuple[str, ...],
+ matrix_name: str,
+) -> dict[str, str]:
+ parent = PurePosixPath(matrix_name).parent
+ return {
+ PurePosixPath(name).name.lower(): name
+ for name in names
+ if PurePosixPath(name).parent == parent
+ }
+
+
+def _first_name(
+ available: dict[str, str],
+ names: tuple[str, ...],
+) -> str | None:
+ for name in names:
+ match = available.get(name.lower())
+ if match is not None:
+ return match
+ return None
+
+
+def _related_files(available: dict[str, str]) -> tuple[str, ...]:
+ return tuple(
+ sorted(
+ value
+ for key, value in available.items()
+ if ("guide" in key or "protospacer" in key or "crispr_analysis" in key)
+ and key.endswith((".csv", ".csv.gz", ".tsv", ".tsv.gz"))
+ )
+ )
+
+
+def _resolve_triplet(
+ names: tuple[str, ...],
+ matrix_name: str,
+) -> _ResolvedTriplet | None:
+ available = _same_parent_names(names, matrix_name)
+ parse_layout = _is_parse_matrix(matrix_name)
+ if parse_layout:
+ features = _first_name(
+ available,
+ (
+ "all_genes.csv.gz",
+ "all_genes.csv",
+ "genes.csv.gz",
+ "genes.csv",
+ ),
+ )
+ cells = _first_name(
+ available,
+ ("cell_metadata.csv.gz", "cell_metadata.csv"),
+ )
+ if features is None or cells is None:
+ return None
+ return _ResolvedTriplet(
+ matrix=matrix_name,
+ features=features,
+ cells=cells,
+ parseLayout=True,
+ cellMetadata=cells,
+ featureReference=None,
+ related=_related_files(available),
+ )
+
+ prefix = _matrix_prefix(matrix_name).lower()
+ features = _first_name(
+ available,
+ tuple(
+ f"{prefix}{suffix}"
+ for suffix in (
+ "features.tsv.gz",
+ "features.tsv",
+ "genes.tsv.gz",
+ "genes.tsv",
+ "peaks.bed.gz",
+ "peaks.bed",
+ )
+ ),
+ )
+ cells = _first_name(
+ available,
+ (f"{prefix}barcodes.tsv.gz", f"{prefix}barcodes.tsv"),
+ )
+ if features is None or cells is None:
+ return None
+ feature_reference = _first_name(
+ available,
+ (
+ f"{prefix}feature_reference.csv.gz",
+ f"{prefix}feature_reference.csv",
+ "feature_reference.csv.gz",
+ "feature_reference.csv",
+ ),
+ )
+ return _ResolvedTriplet(
+ matrix=matrix_name,
+ features=features,
+ cells=cells,
+ parseLayout=False,
+ cellMetadata=None,
+ featureReference=feature_reference,
+ related=_related_files(available),
+ )
+
+
+@contextmanager
+def _open_path_text(path: str) -> Iterator[TextIO]:
+ if path.lower().endswith(".gz"):
+ with gzip.open(path, mode="rt", encoding="utf-8", newline="") as handle:
+ yield handle
+ else:
+ with open(path, mode="rt", encoding="utf-8", newline="") as handle:
+ yield handle
+
+
+def _zip_member_mode(info: zipfile.ZipInfo) -> int:
+ return int(info.external_attr >> 16)
+
+
+def _validated_zip_infos(archive: zipfile.ZipFile) -> dict[str, zipfile.ZipInfo]:
+ infos: dict[str, zipfile.ZipInfo] = {}
+ for info in archive.infolist():
+ member = PurePosixPath(info.filename)
+ if member.is_absolute() or ".." in member.parts:
+ raise ValueError(f"Unsafe ZIP archive member: {info.filename}")
+ normalized = str(member)
+ if normalized in infos:
+ raise ValueError(f"Duplicate ZIP archive member: {info.filename}")
+ if info.flag_bits & 0x1:
+ raise ValueError(f"Encrypted ZIP archive member: {info.filename}")
+ mode = _zip_member_mode(info)
+ if stat.S_ISLNK(mode):
+ raise ValueError(f"ZIP archive links are not supported: {info.filename}")
+ file_type = stat.S_IFMT(mode)
+ if file_type and not (stat.S_ISREG(mode) or stat.S_ISDIR(mode)):
+ raise ValueError(f"Unsupported ZIP archive member: {info.filename}")
+ if not info.is_dir() and normalized.lower().endswith(".zip"):
+ raise ValueError("A Matrix Market ZIP cannot contain another ZIP")
+ infos[normalized] = info
+ return infos
+
+
+@contextmanager
+def _open_zip_text(
+ archive: zipfile.ZipFile,
+ info: zipfile.ZipInfo,
+) -> Iterator[TextIO]:
+ with archive.open(info, mode="r") as raw:
+ if info.filename.lower().endswith(".gz"):
+ with gzip.GzipFile(fileobj=raw, mode="rb") as decoded:
+ with TextIOWrapper(decoded, encoding="utf-8", newline="") as text:
+ yield text
+ else:
+ with TextIOWrapper(raw, encoding="utf-8", newline="") as text:
+ yield text
+
+
+def _read_header(open_text: Callable[[], Any]) -> _MtxHeader:
+ with open_text() as handle:
+ banner = handle.readline().strip().split()
+ if len(banner) != 5 or [part.lower() for part in banner[:3]] != [
+ "%%matrixmarket",
+ "matrix",
+ "coordinate",
+ ]:
+ raise ValueError("Matrix Market input must use coordinate matrix format")
+ field = banner[3].lower()
+ symmetry = banner[4].lower()
+ if field not in {"integer", "real"}:
+ raise ValueError(
+ "Matrix Market counts must use integer or real coordinate values"
+ )
+ if symmetry != "general":
+ raise ValueError("Matrix Market counts must use general symmetry")
+ dimensions = ""
+ for line in handle:
+ stripped = line.strip()
+ if stripped and not stripped.startswith("%"):
+ dimensions = stripped
+ break
+ fields = dimensions.split()
+ if len(fields) != 3:
+ raise ValueError(
+ "Matrix Market dimensions line must contain three integers"
+ )
+ try:
+ n_rows, n_columns, n_entries = (int(value) for value in fields)
+ except ValueError as exc:
+ raise ValueError(
+ "Matrix Market dimensions line must contain three integers"
+ ) from exc
+ if n_rows < 0 or n_columns < 0 or n_entries < 0:
+ raise ValueError("Matrix Market dimensions cannot be negative")
+ return _MtxHeader(
+ field=field,
+ nRows=n_rows,
+ nColumns=n_columns,
+ nEntries=n_entries,
+ )
+
+
+def _count_sidecar_rows(
+ open_text: Callable[[], Any],
+ *,
+ header: bool,
+) -> int:
+ with open_text() as handle:
+ rows = sum(1 for line in handle if line.strip())
+ return max(0, rows - int(header))
+
+
+def _csv_header(open_text: Callable[[], Any]) -> tuple[str, ...]:
+ with open_text() as handle:
+ reader = csv.reader(handle)
+ try:
+ return tuple(str(value).strip() for value in next(reader))
+ except StopIteration:
+ return ()
+
+
+def _candidate_from_triplet(
+ *,
+ source: str,
+ archive_path: str | None,
+ triplet: _ResolvedTriplet,
+ open_text: Callable[[str], Any],
+) -> MtxCandidate:
+ header = _read_header(lambda: open_text(triplet.matrix))
+ feature_rows = _count_sidecar_rows(
+ lambda: open_text(triplet.features),
+ header=triplet.parseLayout,
+ )
+ cell_rows = _count_sidecar_rows(
+ lambda: open_text(triplet.cells),
+ header=triplet.parseLayout,
+ )
+ preferred: MatrixOrientation = (
+ "cellsByFeatures" if triplet.parseLayout else "featuresByCells"
+ )
+ features_by_cells = feature_rows == header.nRows and cell_rows == header.nColumns
+ cells_by_features = cell_rows == header.nRows and feature_rows == header.nColumns
+ if preferred == "featuresByCells" and features_by_cells:
+ orientation: MatrixOrientation = "featuresByCells"
+ elif preferred == "cellsByFeatures" and cells_by_features:
+ orientation = "cellsByFeatures"
+ elif features_by_cells and not cells_by_features:
+ orientation = "featuresByCells"
+ elif cells_by_features and not features_by_cells:
+ orientation = "cellsByFeatures"
+ else:
+ raise ValueError(
+ f"Matrix dimensions ({header.nRows}, {header.nColumns}) do not "
+ f"match {feature_rows} feature rows and {cell_rows} cell rows"
+ )
+ cell_id_keys = (
+ tuple(
+ key
+ for key in ("bc_wells", "bc_index")
+ if key in _csv_header(lambda: open_text(triplet.cells))
+ )
+ if triplet.parseLayout
+ else ()
+ )
+ n_cells = header.nColumns if orientation == "featuresByCells" else header.nRows
+ n_features = header.nRows if orientation == "featuresByCells" else header.nColumns
+ return MtxCandidate(
+ source=source,
+ matrixPath=triplet.matrix,
+ featurePath=triplet.features,
+ cellPath=triplet.cells,
+ matrixOrientation=orientation,
+ nCells=n_cells,
+ nFeatures=n_features,
+ nEntries=header.nEntries,
+ archivePath=archive_path,
+ cellMetadataPath=triplet.cellMetadata,
+ featureReferencePath=triplet.featureReference,
+ cellIdKeys=cell_id_keys,
+ relatedFiles=triplet.related,
+ )
+
+
+def inspect_mtx(source: str | Path) -> tuple[MtxCandidate, ...]:
+ """Return complete Matrix Market import candidates found in one source."""
+ source_path = Path(source).expanduser()
+ if not source_path.exists():
+ raise FileNotFoundError(source_path)
+
+ candidates: list[MtxCandidate] = []
+ if source_path.is_file() and source_path.suffix.lower() == ".zip":
+ with zipfile.ZipFile(source_path) as archive:
+ infos = _validated_zip_infos(archive)
+ names = tuple(
+ sorted(name for name, info in infos.items() if not info.is_dir())
+ )
+
+ @contextmanager
+ def open_member(name: str) -> Iterator[TextIO]:
+ with _open_zip_text(archive, infos[name]) as handle:
+ yield handle
+
+ for matrix_name in (name for name in names if _is_matrix_path(name)):
+ triplet = _resolve_triplet(names, matrix_name)
+ if triplet is None:
+ continue
+ candidates.append(
+ _candidate_from_triplet(
+ source=str(source_path),
+ archive_path=str(source_path),
+ triplet=triplet,
+ open_text=open_member,
+ )
+ )
+ else:
+ directory = source_path if source_path.is_dir() else source_path.parent
+ names = tuple(
+ sorted(
+ str(path.resolve()) for path in directory.iterdir() if path.is_file()
+ )
+ )
+ selected_matrix = str(source_path.resolve()) if source_path.is_file() else None
+ for matrix_name in (name for name in names if _is_matrix_path(name)):
+ if selected_matrix is not None and matrix_name != selected_matrix:
+ continue
+ triplet = _resolve_triplet(names, matrix_name)
+ if triplet is None:
+ continue
+ candidates.append(
+ _candidate_from_triplet(
+ source=str(source_path),
+ archive_path=None,
+ triplet=triplet,
+ open_text=lambda name: _open_path_text(name),
+ )
+ )
+
+ if not candidates:
+ raise ValueError(
+ "No complete Matrix Market matrix, feature, and cell triplet found"
+ )
+ return tuple(
+ sorted(
+ candidates,
+ key=lambda candidate: (
+ candidate.matrixPath,
+ candidate.featurePath,
+ candidate.cellPath,
+ ),
+ )
+ )
+
+
+def _explicit_candidate(
+ matrix_path: str,
+ feature_path: str,
+ cell_path: str,
+ *,
+ cell_metadata_path: str | None,
+ feature_reference_path: str | None,
+) -> MtxCandidate:
+ matrix = str(Path(matrix_path).expanduser().resolve())
+ features = str(Path(feature_path).expanduser().resolve())
+ cells = str(Path(cell_path).expanduser().resolve())
+ for path in (matrix, features, cells):
+ if not Path(path).is_file():
+ raise FileNotFoundError(path)
+ parse_layout = _is_parse_matrix(matrix)
+ triplet = _ResolvedTriplet(
+ matrix=matrix,
+ features=features,
+ cells=cells,
+ parseLayout=parse_layout,
+ cellMetadata=cell_metadata_path,
+ featureReference=feature_reference_path,
+ related=(),
+ )
+ return replace(
+ _candidate_from_triplet(
+ source=matrix,
+ archive_path=None,
+ triplet=triplet,
+ open_text=lambda name: _open_path_text(name),
+ ),
+ cellMetadataPath=(
+ str(Path(cell_metadata_path).expanduser().resolve())
+ if cell_metadata_path is not None
+ else None
+ ),
+ featureReferencePath=(
+ str(Path(feature_reference_path).expanduser().resolve())
+ if feature_reference_path is not None
+ else None
+ ),
+ )
+
+
+def _require_unique(values: np.ndarray, name: str) -> None:
+ normalized = np.asarray(values, dtype=str)
+ if normalized.ndim != 1 or np.any(np.char.strip(normalized) == ""):
+ raise ValueError(f"{name} must contain non-empty one-dimensional values")
+ if np.unique(normalized).size != normalized.size:
+ raise ValueError(f"{name} must contain unique values")
+
+
+def _column_name(values: tuple[str, ...], preferences: tuple[str, ...]) -> str:
+ normalized = {str(value).lower(): str(value) for value in values}
+ for preference in preferences:
+ if preference in normalized:
+ return normalized[preference]
+ raise ValueError(
+ "Could not identify a required column; available columns are "
+ + ", ".join(values)
+ )
+
+
+class _MtxEngine:
+ def __init__(
+ self,
+ candidate: MtxCandidate,
+ *,
+ cell_id_key: str | None,
+ separator: str,
+ index_offset: int,
+ is_filtered: bool,
+ filtering_cutoff: int,
+ temp_dir: str | None,
+ dtype: DTypeLike,
+ ) -> None:
+ self.candidate = candidate
+ self.separator = separator
+ self.indexOffset = int(index_offset)
+ self.isFiltered = bool(is_filtered)
+ self.filteringCutoff = int(filtering_cutoff)
+ self.tempDir = None if temp_dir is None else str(Path(temp_dir))
+ self.matrixDtype = np.dtype(dtype)
+ if self.matrixDtype.kind not in "iu":
+ raise TypeError("Matrix Market count dtype must be an integer dtype")
+ if self.filteringCutoff < 0:
+ raise ValueError("filtering_cutoff cannot be negative")
+ if self.tempDir is not None and not Path(self.tempDir).is_dir():
+ raise FileNotFoundError(self.tempDir)
+
+ self._archiveDirectory: Path | None = None
+ self._archiveTemporary: Any | None = None
+ self._archivePaths: dict[str, str] = {}
+ self._csrDirectory: Path | None = None
+ self._csrData: np.ndarray | np.memmap | None = None
+ self._csrIndices: np.ndarray | np.memmap | None = None
+ self._csrIndptr: np.ndarray | np.memmap | None = None
+ self._cellMap: np.ndarray | None = None
+ self._rowNnz: np.ndarray | None = None
+ self._cumulativeRowNnz: np.ndarray | None = None
+ self._importLinesInMem = 100_000
+ self.temporaryDiskBytes = 0
+ try:
+ self._ensure_archive_extracted()
+
+ self.matrixPath = self._local_path(candidate.matrixPath)
+ self.featurePath = self._local_path(candidate.featurePath)
+ self.cellPath = self._local_path(candidate.cellPath)
+ self.header = _read_header(lambda: _open_path_text(self.matrixPath))
+ self.matrixEntryCount = self.header.nEntries
+ self.rawCellCount = candidate.nCells
+ self.nFeatures = candidate.nFeatures
+ (
+ self._rawFeatureIds,
+ self._rawFeatureNames,
+ self._rawFeatureTypes,
+ ) = self._read_features()
+ (
+ self._rawCellNames,
+ self._rawCellColumns,
+ self.selectedCellIdKey,
+ ) = self._read_cells(cell_id_key)
+ _require_unique(self._rawFeatureIds, "Feature IDs")
+ _require_unique(self._rawCellNames, "Cell IDs")
+
+ self.coordinateOrder = self._probe_coordinate_order()
+ row_nnz: np.ndarray | None = None
+ if self.isFiltered:
+ valid = np.arange(self.rawCellCount, dtype=np.int64)
+ else:
+ row_nnz, cell_totals, order = self._scan_matrix(self._importLinesInMem)
+ if order != self.coordinateOrder:
+ raise RuntimeError(
+ "Matrix Market coordinate-order probes were inconsistent"
+ )
+ valid = np.flatnonzero(cell_totals > self.filteringCutoff).astype(
+ np.int64,
+ copy=False,
+ )
+ self._set_valid_cells(valid, row_nnz)
+ self._featureColumns = self._feature_reference_columns()
+ except BaseException:
+ self.release()
+ raise
+
+ def _ensure_archive_extracted(self) -> None:
+ if self.candidate.archivePath is None or self._archiveDirectory is not None:
+ return
+ temporary = tempfile.TemporaryDirectory(
+ prefix="scarf-mtx-archive-",
+ dir=self.tempDir,
+ )
+ directory = Path(temporary.name)
+ self._archiveTemporary = temporary
+ required = {
+ self.candidate.matrixPath,
+ self.candidate.featurePath,
+ self.candidate.cellPath,
+ self.candidate.cellMetadataPath,
+ self.candidate.featureReferencePath,
+ }
+ members = {name for name in required if name is not None}
+ try:
+ with zipfile.ZipFile(self.candidate.archivePath) as archive:
+ infos = _validated_zip_infos(archive)
+ missing = sorted(members.difference(infos))
+ if missing:
+ raise ValueError(
+ "ZIP archive is missing selected members: " + ", ".join(missing)
+ )
+ for index, member in enumerate(sorted(members)):
+ suffixes = "".join(PurePosixPath(member).suffixes)
+ destination = directory / f"{index}{suffixes}"
+ with archive.open(infos[member], mode="r") as source:
+ with destination.open("wb") as target:
+ shutil.copyfileobj(source, target)
+ self._archivePaths[member] = str(destination)
+ self._archiveDirectory = directory
+ except BaseException:
+ temporary.cleanup()
+ self._archiveTemporary = None
+ self._archivePaths.clear()
+ raise
+
+ def _local_path(self, path: str) -> str:
+ return self._archivePaths.get(path, path)
+
+ def _set_valid_cells(
+ self,
+ valid: np.ndarray,
+ row_nnz: np.ndarray | None,
+ ) -> None:
+ self.validCellIndexes = valid
+ self.nCells = int(valid.size)
+ self._cellNames = self._rawCellNames[valid]
+ self._cellColumns = {
+ name: values[valid] for name, values in self._rawCellColumns.items()
+ }
+ if row_nnz is not None:
+ self._set_row_nnz(row_nnz[valid])
+
+ def _set_row_nnz(self, row_nnz: np.ndarray) -> None:
+ self._rowNnz = np.asarray(row_nnz, dtype=np.int64)
+ cumulative = np.empty(self.nCells + 1, dtype=np.int64)
+ cumulative[0] = 0
+ np.cumsum(self._rowNnz, dtype=np.int64, out=cumulative[1:])
+ self._cumulativeRowNnz = cumulative
+
+ def _read_features(self) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ if _is_parse_matrix(self.candidate.matrixPath):
+ frame = pd.read_csv(self.featurePath, compression="infer")
+ columns = tuple(str(value) for value in frame.columns)
+ id_column = _column_name(
+ columns,
+ ("gene_id", "feature_id", "id", "gene"),
+ )
+ try:
+ name_column = _column_name(
+ columns,
+ ("gene_name", "feature_name", "name", "gene_symbol"),
+ )
+ except ValueError:
+ name_column = id_column
+ ids = frame[id_column].astype(str).to_numpy()
+ names = frame[name_column].astype(str).to_numpy()
+ types = np.full(ids.size, "Gene Expression", dtype=object)
+ else:
+ frame = pd.read_csv(
+ self.featurePath,
+ sep="\t",
+ header=None,
+ dtype=str,
+ keep_default_na=False,
+ compression="infer",
+ )
+ if frame.shape[1] < 1:
+ raise ValueError("Feature sidecar must contain at least one column")
+ ids = frame.iloc[:, 0].astype(str).to_numpy()
+ names = (
+ frame.iloc[:, 1].astype(str).to_numpy()
+ if frame.shape[1] > 1
+ else ids.copy()
+ )
+ types = (
+ frame.iloc[:, 2].astype(str).to_numpy()
+ if frame.shape[1] > 2
+ else np.full(ids.size, "Gene Expression", dtype=object)
+ )
+ if ids.size != self.nFeatures:
+ raise ValueError(
+ f"Feature sidecar has {ids.size} rows, expected {self.nFeatures}"
+ )
+ return ids, names, types
+
+ def _read_cells(
+ self,
+ cell_id_key: str | None,
+ ) -> tuple[np.ndarray, dict[str, np.ndarray], str | None]:
+ if _is_parse_matrix(self.candidate.matrixPath):
+ frame = pd.read_csv(self.cellPath, compression="infer")
+ aliases = [key for key in ("bc_wells", "bc_index") if key in frame.columns]
+ if cell_id_key is None:
+ if len(aliases) != 1:
+ reason = "absent" if not aliases else "ambiguous"
+ raise ValueError(
+ f"Parse cell-ID aliases are {reason}; pass cell_id_key "
+ "explicitly"
+ )
+ selected_key = aliases[0]
+ else:
+ selected_key = cell_id_key
+ if selected_key not in frame.columns:
+ raise KeyError(f"Cell ID column {selected_key!r} was not found")
+ names = frame[selected_key].astype(str).to_numpy()
+ columns = {
+ str(name): frame[name].to_numpy()
+ for name in frame.columns
+ if name != selected_key
+ }
+ else:
+ try:
+ frame = pd.read_csv(
+ self.cellPath,
+ sep="\t",
+ header=None,
+ dtype=str,
+ keep_default_na=False,
+ compression="infer",
+ )
+ except pd.errors.EmptyDataError:
+ frame = pd.DataFrame()
+ if frame.empty and self.rawCellCount == 0:
+ return np.empty(0, dtype=str), {}, None
+ if frame.shape[1] < 1:
+ raise ValueError("Cell sidecar must contain at least one column")
+ names = frame.iloc[:, 0].astype(str).to_numpy()
+ columns = {}
+ selected_key = None
+ if names.size != self.rawCellCount:
+ raise ValueError(
+ f"Cell sidecar has {names.size} rows, expected {self.rawCellCount}"
+ )
+ return names, columns, selected_key
+
+ def _feature_reference_columns(self) -> dict[str, np.ndarray]:
+ reference_path = self.candidate.featureReferencePath
+ if reference_path is None:
+ return {"feature_type": self._rawFeatureTypes}
+ path = self._local_path(reference_path)
+ frame = pd.read_csv(path, compression="infer")
+ columns = tuple(str(value) for value in frame.columns)
+ id_column = _column_name(
+ columns,
+ ("id", "feature_id", "gene_id"),
+ )
+ reference_ids = frame[id_column].astype(str).to_numpy()
+ _require_unique(reference_ids, "Feature-reference IDs")
+ positions = {value: index for index, value in enumerate(reference_ids)}
+ missing = [value for value in self._rawFeatureIds if value not in positions]
+ extras = set(reference_ids).difference(self._rawFeatureIds)
+ if missing or extras:
+ raise ValueError(
+ "Feature reference must contain exactly one row for every "
+ "matrix feature ID"
+ )
+ order = np.fromiter(
+ (positions[value] for value in self._rawFeatureIds),
+ dtype=np.int64,
+ count=self.nFeatures,
+ )
+ result = {"feature_type": self._rawFeatureTypes}
+ for name in frame.columns:
+ resolved_name = str(name)
+ if resolved_name == id_column:
+ continue
+ if "/" in resolved_name or "\\" in resolved_name:
+ raise ValueError(
+ f"Feature-reference column {resolved_name!r} contains a path separator"
+ )
+ result[resolved_name] = frame[name].to_numpy()[order]
+ return result
+
+ def _probe_coordinate_order(self) -> CoordinateOrder:
+ cell_previous: tuple[int, int] | None = None
+ feature_previous: tuple[int, int] | None = None
+ parsed = 0
+ dimensions_seen = False
+ with _open_path_text(self.matrixPath) as handle:
+ for line in handle:
+ stripped = line.strip()
+ if not stripped or stripped.startswith("%"):
+ continue
+ if not dimensions_seen:
+ dimensions_seen = True
+ continue
+ fields = (
+ stripped.split()
+ if self.separator == " "
+ else re.split(self.separator, stripped)
+ )
+ if len(fields) != 3:
+ raise ValueError("Could not parse Matrix Market coordinates")
+ try:
+ axis1 = int(fields[0])
+ axis2 = int(fields[1])
+ if self.header.field == "integer":
+ value: int | float = int(fields[2])
+ else:
+ value = float(fields[2])
+ except (OverflowError, ValueError) as exc:
+ raise ValueError(
+ "Could not parse Matrix Market coordinates"
+ ) from exc
+ if value < 0 or not np.isfinite(value) or value != np.floor(value):
+ raise ValueError(
+ "Matrix Market counts must be finite non-negative integers"
+ )
+ if self.candidate.matrixOrientation == "featuresByCells":
+ feature = axis1 + self.indexOffset
+ cell = axis2 + self.indexOffset
+ else:
+ cell = axis1 + self.indexOffset
+ feature = axis2 + self.indexOffset
+ if (
+ feature < 0
+ or feature >= self.nFeatures
+ or cell < 0
+ or cell >= self.rawCellCount
+ ):
+ raise ValueError(
+ "Matrix Market coordinate is outside the declared dimensions"
+ )
+ cell_key = (cell, feature)
+ feature_key = (feature, cell)
+ cell_invalid = cell_previous is not None and cell_key < cell_previous
+ feature_invalid = (
+ feature_previous is not None and feature_key < feature_previous
+ )
+ parsed += 1
+ if cell_invalid and feature_invalid:
+ raise ValueError(
+ "Matrix Market coordinates are neither cell-major nor "
+ f"feature-major at entry {parsed}"
+ )
+ if cell_invalid:
+ return "featureMajor"
+ if feature_invalid:
+ return "cellMajor"
+ cell_previous = cell_key
+ feature_previous = feature_key
+ if parsed != self.header.nEntries:
+ raise ValueError(
+ f"Matrix Market header declares {self.header.nEntries} entries, "
+ f"but {parsed} were read"
+ )
+ return "cellMajor"
+
+ def _raw_chunks(
+ self,
+ lines_in_mem: int,
+ ) -> Iterator[tuple[np.ndarray, np.ndarray, np.ndarray]]:
+ if lines_in_mem <= 0:
+ raise ValueError("lines_in_mem must be positive")
+ if self.header.nEntries == 0:
+ return
+ separator = r"\s+" if self.separator == " " else self.separator
+ count_dtype: Any = np.int64 if self.header.field == "integer" else np.float64
+ try:
+ stream = pd.read_csv(
+ self.matrixPath,
+ comment="%",
+ sep=separator,
+ header=0,
+ names=["axis1", "axis2", "count"],
+ chunksize=lines_in_mem,
+ dtype={
+ "axis1": np.int64,
+ "axis2": np.int64,
+ "count": count_dtype,
+ },
+ compression="infer",
+ )
+ for frame in stream:
+ axis1 = frame["axis1"].to_numpy(dtype=np.int64, copy=False)
+ axis2 = frame["axis2"].to_numpy(dtype=np.int64, copy=False)
+ raw_values = frame["count"].to_numpy(copy=False)
+ if (
+ not np.isfinite(raw_values).all()
+ or np.any(raw_values < 0)
+ or np.any(raw_values != np.floor(raw_values))
+ ):
+ raise ValueError(
+ "Matrix Market counts must be finite non-negative integers"
+ )
+ values = raw_values.astype(np.uint64, copy=False)
+ if self.candidate.matrixOrientation == "featuresByCells":
+ features = axis1 + self.indexOffset
+ cells = axis2 + self.indexOffset
+ else:
+ cells = axis1 + self.indexOffset
+ features = axis2 + self.indexOffset
+ if (
+ np.any(features < 0)
+ or np.any(features >= self.nFeatures)
+ or np.any(cells < 0)
+ or np.any(cells >= self.rawCellCount)
+ ):
+ raise ValueError(
+ "Matrix Market coordinate is outside the declared dimensions"
+ )
+ yield features, cells, values
+ except (TypeError, ValueError) as exc:
+ if isinstance(exc, ValueError) and str(exc).startswith("Matrix Market"):
+ raise
+ raise ValueError("Could not parse Matrix Market coordinates") from exc
+
+ @staticmethod
+ def _order_violation(
+ first: np.ndarray,
+ second: np.ndarray,
+ previous: tuple[int, int] | None,
+ ) -> int | None:
+ if first.size == 0:
+ return None
+ if previous is None:
+ joined_first = first
+ joined_second = second
+ first_new_offset = 1
+ else:
+ joined_first = np.concatenate(
+ (np.array([previous[0]], dtype=np.int64), first)
+ )
+ joined_second = np.concatenate(
+ (np.array([previous[1]], dtype=np.int64), second)
+ )
+ first_new_offset = 0
+ invalid = (joined_first[1:] < joined_first[:-1]) | (
+ (joined_first[1:] == joined_first[:-1])
+ & (joined_second[1:] < joined_second[:-1])
+ )
+ positions = np.flatnonzero(invalid)
+ return None if positions.size == 0 else int(positions[0] + first_new_offset)
+
+ def _scan_matrix(
+ self,
+ lines_in_mem: int,
+ ) -> tuple[np.ndarray, np.ndarray, CoordinateOrder]:
+ row_nnz = np.zeros(self.rawCellCount, dtype=np.int64)
+ cell_totals = np.zeros(self.rawCellCount, dtype=np.uint64)
+ previous: tuple[int, int] | None = None
+ pending: tuple[int, int, np.uint64] | None = None
+ parsed = 0
+ cell_violation: int | None = None
+ feature_violation: int | None = None
+
+ def account(
+ features: np.ndarray,
+ cells: np.ndarray,
+ values: np.ndarray,
+ ) -> None:
+ if cells.size:
+ np.add.at(row_nnz, cells, 1)
+ np.add.at(cell_totals, cells, values)
+
+ for features, cells, values in self._raw_chunks(lines_in_mem):
+ if features.size == 0:
+ continue
+ cell_previous = None if previous is None else (previous[1], previous[0])
+ cell_bad = self._order_violation(
+ cells,
+ features,
+ cell_previous,
+ )
+ feature_bad = self._order_violation(features, cells, previous)
+ if cell_violation is None and cell_bad is not None:
+ cell_violation = parsed + cell_bad
+ if feature_violation is None and feature_bad is not None:
+ feature_violation = parsed + feature_bad
+ if cell_violation is not None and feature_violation is not None:
+ failed_at = max(cell_violation, feature_violation) + 1
+ raise ValueError(
+ "Matrix Market coordinates are neither cell-major nor "
+ f"feature-major at entry {failed_at}"
+ )
+ previous = (int(features[-1]), int(cells[-1]))
+ parsed += int(features.size)
+
+ if pending is not None:
+ features = np.concatenate(
+ (np.array([pending[0]], dtype=np.int64), features)
+ )
+ cells = np.concatenate((np.array([pending[1]], dtype=np.int64), cells))
+ values = np.concatenate(
+ (np.array([pending[2]], dtype=np.uint64), values)
+ )
+ changes = (features[1:] != features[:-1]) | (cells[1:] != cells[:-1])
+ starts = np.concatenate(
+ (
+ np.array([0], dtype=np.int64),
+ np.flatnonzero(changes).astype(np.int64) + 1,
+ )
+ )
+ reduced = np.add.reduceat(values, starts)
+ complete = max(0, starts.size - 1)
+ if complete:
+ account(
+ features[starts[:complete]],
+ cells[starts[:complete]],
+ reduced[:complete],
+ )
+ last = int(starts[-1])
+ pending = (
+ int(features[last]),
+ int(cells[last]),
+ np.uint64(reduced[-1]),
+ )
+
+ if parsed != self.header.nEntries:
+ raise ValueError(
+ f"Matrix Market header declares {self.header.nEntries} entries, "
+ f"but {parsed} were read"
+ )
+ if pending is not None:
+ account(
+ np.array([pending[0]], dtype=np.int64),
+ np.array([pending[1]], dtype=np.int64),
+ np.array([pending[2]], dtype=np.uint64),
+ )
+ order: CoordinateOrder = (
+ "cellMajor" if cell_violation is None else "featureMajor"
+ )
+ return row_nnz, cell_totals, order
+
+ def _ensure_matrix_stats(self, lines_in_mem: int) -> None:
+ if self._rowNnz is not None:
+ return
+ row_nnz, _, order = self._scan_matrix(lines_in_mem)
+ if order != self.coordinateOrder:
+ raise RuntimeError(
+ "Matrix Market coordinate-order probes were inconsistent"
+ )
+ self._set_row_nnz(row_nnz[self.validCellIndexes])
+
+ def _coalesced_chunks(
+ self,
+ lines_in_mem: int,
+ ) -> Iterator[tuple[np.ndarray, np.ndarray, np.ndarray]]:
+ previous: tuple[int, int] | None = None
+ pending: tuple[int, int, np.uint64] | None = None
+ parsed = 0
+ for features, cells, values in self._raw_chunks(lines_in_mem):
+ if features.size == 0:
+ continue
+ first, second = (
+ (cells, features)
+ if self.coordinateOrder == "cellMajor"
+ else (features, cells)
+ )
+ invalid = self._order_violation(first, second, previous)
+ if invalid is not None:
+ raise ValueError(
+ f"Matrix Market {self.coordinateOrder} order is invalid at "
+ f"entry {parsed + invalid + 1}"
+ )
+ previous = (int(first[-1]), int(second[-1]))
+ parsed += int(features.size)
+
+ if pending is not None:
+ features = np.concatenate(
+ (np.array([pending[0]], dtype=np.int64), features)
+ )
+ cells = np.concatenate((np.array([pending[1]], dtype=np.int64), cells))
+ values = np.concatenate(
+ (np.array([pending[2]], dtype=np.uint64), values)
+ )
+ changes = (features[1:] != features[:-1]) | (cells[1:] != cells[:-1])
+ starts = np.concatenate(
+ (
+ np.array([0], dtype=np.int64),
+ np.flatnonzero(changes).astype(np.int64) + 1,
+ )
+ )
+ reduced = np.add.reduceat(values, starts)
+ complete = max(0, starts.size - 1)
+ if complete:
+ yield (
+ features[starts[:complete]],
+ cells[starts[:complete]],
+ self._checked_values(reduced[:complete]),
+ )
+ last = int(starts[-1])
+ pending = (
+ int(features[last]),
+ int(cells[last]),
+ np.uint64(reduced[-1]),
+ )
+ if parsed != self.header.nEntries:
+ raise ValueError(
+ f"Matrix Market header declares {self.header.nEntries} entries, "
+ f"but {parsed} were read"
+ )
+ if pending is not None:
+ yield (
+ np.array([pending[0]], dtype=np.int64),
+ np.array([pending[1]], dtype=np.int64),
+ self._checked_values(np.array([pending[2]], dtype=np.uint64)),
+ )
+
+ def _checked_values(self, values: np.ndarray) -> np.ndarray:
+ maximum = np.iinfo(self.matrixDtype).max
+ if values.size and int(values.max()) > maximum:
+ raise OverflowError(f"Matrix Market count exceeds dtype {self.matrixDtype}")
+ return values.astype(self.matrixDtype, copy=False)
+
+ def _prepare_feature_major(self, lines_in_mem: int = 1_000_000) -> None:
+ if self._csrIndptr is not None:
+ return
+ if self._rowNnz is None or self._cumulativeRowNnz is None:
+ raise RuntimeError(
+ "Feature-major Matrix Market row statistics are unavailable"
+ )
+ parent = (
+ Path(self.tempDir)
+ if self.tempDir is not None
+ else Path(tempfile.gettempdir())
+ )
+ retained_nnz = int(self._rowNnz.sum(dtype=np.int64))
+ index_dtype = (
+ np.int32
+ if max(self.nFeatures, retained_nnz) <= np.iinfo(np.int32).max
+ else np.int64
+ )
+ required = int(
+ retained_nnz * (self.matrixDtype.itemsize + np.dtype(index_dtype).itemsize)
+ + (self.nCells + 1) * np.dtype(np.int64).itemsize
+ )
+ self.temporaryDiskBytes = required
+ free = shutil.disk_usage(parent).free
+ if required > free:
+ raise OSError(
+ f"Feature-major Matrix Market preparation needs {required} "
+ f"temporary bytes, but {free} bytes are free in {parent}"
+ )
+ directory = Path(tempfile.mkdtemp(prefix="scarf-mtx-csr-", dir=str(parent)))
+ try:
+ indptr = np.memmap(
+ directory / "indptr.bin",
+ mode="w+",
+ dtype=np.int64,
+ shape=(self.nCells + 1,),
+ )
+ indptr[:] = self._cumulativeRowNnz
+ data: np.ndarray | np.memmap
+ indices: np.ndarray | np.memmap
+ if retained_nnz:
+ data = np.memmap(
+ directory / "data.bin",
+ mode="w+",
+ dtype=self.matrixDtype,
+ shape=(retained_nnz,),
+ )
+ indices = np.memmap(
+ directory / "indices.bin",
+ mode="w+",
+ dtype=index_dtype,
+ shape=(retained_nnz,),
+ )
+ else:
+ data = np.empty(0, dtype=self.matrixDtype)
+ indices = np.empty(0, dtype=index_dtype)
+ cell_map = np.full(self.rawCellCount, -1, dtype=np.int64)
+ cell_map[self.validCellIndexes] = np.arange(
+ self.nCells,
+ dtype=np.int64,
+ )
+ cursor = np.asarray(self._cumulativeRowNnz[:-1]).copy()
+ for features, cells, values in self._coalesced_chunks(lines_in_mem):
+ output_cells = cell_map[cells]
+ selected = output_cells >= 0
+ if not selected.any():
+ continue
+ output_cells = output_cells[selected]
+ selected_features = features[selected]
+ selected_values = values[selected]
+ order = np.argsort(output_cells, kind="stable")
+ grouped_cells = output_cells[order]
+ edges = np.concatenate(
+ (
+ np.array([0], dtype=np.int64),
+ np.flatnonzero(grouped_cells[1:] != grouped_cells[:-1]).astype(
+ np.int64
+ )
+ + 1,
+ np.array([grouped_cells.size], dtype=np.int64),
+ )
+ )
+ positions_sorted = np.empty(grouped_cells.size, dtype=np.int64)
+ for left, right in zip(edges[:-1], edges[1:], strict=True):
+ cell = int(grouped_cells[left])
+ width = int(right - left)
+ positions_sorted[left:right] = cursor[cell] + np.arange(
+ width, dtype=np.int64
+ )
+ cursor[cell] += width
+ positions = np.empty_like(positions_sorted)
+ positions[order] = positions_sorted
+ data[positions] = selected_values
+ indices[positions] = selected_features
+ if not np.array_equal(cursor, self._cumulativeRowNnz[1:]):
+ raise RuntimeError(
+ "Feature-major CSR preparation produced inconsistent row pointers"
+ )
+ for array in (data, indices, indptr):
+ if isinstance(array, np.memmap):
+ array.flush()
+ self._csrDirectory = directory
+ self._csrData = data
+ self._csrIndices = indices
+ self._csrIndptr = indptr
+ self._cellMap = cell_map
+ except BaseException:
+ shutil.rmtree(directory, ignore_errors=True)
+ raise
+
+ def configure_import_lines(self, lines_in_mem: int) -> None:
+ if lines_in_mem <= 0:
+ raise ValueError("lines_in_mem must be positive")
+ self._importLinesInMem = int(lines_in_mem)
+
+ def prepare(self, lines_in_mem: int | None = None) -> None:
+ resolved_lines = (
+ self._importLinesInMem if lines_in_mem is None else int(lines_in_mem)
+ )
+ self.configure_import_lines(resolved_lines)
+ self._ensure_archive_extracted()
+ if self.candidate.archivePath is not None:
+ self.matrixPath = self._local_path(self.candidate.matrixPath)
+ if self.coordinateOrder == "featureMajor":
+ try:
+ self._ensure_matrix_stats(resolved_lines)
+ self._prepare_feature_major(resolved_lines)
+ except BaseException:
+ self.release()
+ raise
+
+ @staticmethod
+ def _close_array(array: np.ndarray | np.memmap | None) -> None:
+ if isinstance(array, np.memmap):
+ mmap = getattr(array, "_mmap", None)
+ if mmap is not None:
+ mmap.close()
+
+ def release(self) -> None:
+ for array in (self._csrData, self._csrIndices, self._csrIndptr):
+ self._close_array(array)
+ self._csrData = None
+ self._csrIndices = None
+ self._csrIndptr = None
+ self._cellMap = None
+ gc.collect()
+ if self._csrDirectory is not None:
+ shutil.rmtree(self._csrDirectory, ignore_errors=True)
+ self._csrDirectory = None
+ if self._archiveDirectory is not None:
+ if self._archiveTemporary is not None:
+ self._archiveTemporary.cleanup()
+ else:
+ shutil.rmtree(self._archiveDirectory, ignore_errors=True)
+ self._archiveDirectory = None
+ self._archiveTemporary = None
+ self._archivePaths.clear()
+
+ def resident_bytes(self) -> int:
+ arrays = (
+ self.validCellIndexes,
+ self._rowNnz,
+ self._cumulativeRowNnz,
+ self._cellMap,
+ )
+ return int(
+ sum(array.nbytes for array in arrays if isinstance(array, np.ndarray))
+ )
+
+ def max_window_nnz(self, window_rows: int) -> int:
+ if window_rows <= 0:
+ raise ValueError("window_rows must be positive")
+ width = min(int(window_rows), self.nCells)
+ if width == 0:
+ return 0
+ if self._cumulativeRowNnz is None:
+ return min(self.matrixEntryCount, width * self.nFeatures)
+ return int(
+ np.max(self._cumulativeRowNnz[width:] - self._cumulativeRowNnz[:-width])
+ )
+
+ def producer_staging_bytes(
+ self,
+ batch_size: int,
+ lines_in_mem: int,
+ ) -> int:
+ if batch_size <= 0:
+ raise ValueError("batch_size must be positive")
+ if lines_in_mem <= 0:
+ raise ValueError("lines_in_mem must be positive")
+ parser_value_bytes = (
+ 2 * np.dtype(np.int64).itemsize + np.dtype(np.uint64).itemsize
+ )
+ source_nnz = self.max_window_nnz(batch_size)
+ cell_map_bytes = (
+ self.rawCellCount * np.dtype(np.int64).itemsize
+ if self.coordinateOrder == "cellMajor"
+ else 0
+ )
+ return int(
+ lines_in_mem * (4 * parser_value_bytes + np.dtype(np.bool_).itemsize)
+ + source_nnz * (2 * np.dtype(np.int64).itemsize + self.matrixDtype.itemsize)
+ + cell_map_bytes
+ )
+
+ def consume(
+ self,
+ batch_size: int,
+ lines_in_mem: int,
+ dtype: DTypeLike | None = None,
+ ) -> Generator[coo_matrix, None, None]:
+ if batch_size <= 0:
+ raise ValueError("batch_size must be positive")
+ if lines_in_mem <= 0:
+ raise ValueError("lines_in_mem must be positive")
+ requested_dtype = self.matrixDtype if dtype is None else np.dtype(dtype)
+ try:
+ if self.coordinateOrder == "featureMajor":
+ self.prepare(lines_in_mem)
+ source = self._consume_feature_major(batch_size)
+ else:
+ self._ensure_archive_extracted()
+ if self.candidate.archivePath is not None:
+ self.matrixPath = self._local_path(self.candidate.matrixPath)
+ source = self._consume_cell_major(batch_size, lines_in_mem)
+ for matrix in source:
+ yield (
+ matrix
+ if requested_dtype == self.matrixDtype
+ else matrix.astype(requested_dtype)
+ )
+ finally:
+ self.release()
+
+ def _consume_feature_major(
+ self,
+ batch_size: int,
+ ) -> Generator[coo_matrix, None, None]:
+ if self._csrData is None or self._csrIndices is None or self._csrIndptr is None:
+ raise RuntimeError("Feature-major CSR preparation is unavailable")
+ for start in range(0, self.nCells, batch_size):
+ end = min(start + batch_size, self.nCells)
+ pointers = np.asarray(self._csrIndptr[start : end + 1])
+ value_start = int(pointers[0])
+ value_end = int(pointers[-1])
+ local = pointers - value_start
+ matrix = csr_matrix(
+ (
+ np.array(self._csrData[value_start:value_end], copy=True),
+ np.array(self._csrIndices[value_start:value_end], copy=True),
+ local,
+ ),
+ shape=(end - start, self.nFeatures),
+ )
+ yield matrix.tocoo(copy=False)
+
+ def _consume_cell_major(
+ self,
+ batch_size: int,
+ lines_in_mem: int,
+ ) -> Generator[coo_matrix, None, None]:
+ cell_map = np.full(self.rawCellCount, -1, dtype=np.int64)
+ cell_map[self.validCellIndexes] = np.arange(
+ self.nCells,
+ dtype=np.int64,
+ )
+ batch_count = (self.nCells + batch_size - 1) // batch_size
+ current_batch = 0
+ row_parts: list[np.ndarray] = []
+ column_parts: list[np.ndarray] = []
+ value_parts: list[np.ndarray] = []
+
+ def matrix_for(batch_index: int) -> coo_matrix:
+ start = batch_index * batch_size
+ end = min(start + batch_size, self.nCells)
+ rows = (
+ np.concatenate(row_parts) if row_parts else np.empty(0, dtype=np.int64)
+ )
+ columns = (
+ np.concatenate(column_parts)
+ if column_parts
+ else np.empty(0, dtype=np.int64)
+ )
+ values = (
+ np.concatenate(value_parts)
+ if value_parts
+ else np.empty(0, dtype=self.matrixDtype)
+ )
+ return coo_matrix(
+ (values, (rows, columns)),
+ shape=(end - start, self.nFeatures),
+ dtype=self.matrixDtype,
+ )
+
+ for features, cells, values in self._coalesced_chunks(lines_in_mem):
+ output_cells = cell_map[cells]
+ selected = output_cells >= 0
+ output_cells = output_cells[selected]
+ features = features[selected]
+ values = values[selected]
+ if output_cells.size == 0:
+ continue
+ batch_ids = output_cells // batch_size
+ position = 0
+ while position < output_cells.size:
+ batch_id = int(batch_ids[position])
+ while current_batch < batch_id:
+ yield matrix_for(current_batch)
+ row_parts.clear()
+ column_parts.clear()
+ value_parts.clear()
+ current_batch += 1
+ end = int(np.searchsorted(batch_ids, batch_id, side="right"))
+ row_parts.append(
+ output_cells[position:end] - current_batch * batch_size
+ )
+ column_parts.append(features[position:end])
+ value_parts.append(values[position:end])
+ position = end
+ while current_batch < batch_count:
+ yield matrix_for(current_batch)
+ row_parts.clear()
+ column_parts.clear()
+ value_parts.clear()
+ current_batch += 1
+
+ def feature_ids(self) -> list[str]:
+ return [str(value) for value in self._rawFeatureIds]
+
+ def feature_names(self) -> list[str]:
+ return [str(value) for value in self._rawFeatureNames]
+
+ def feature_types(self) -> list[str]:
+ return [str(value) for value in self._rawFeatureTypes]
+
+ def cell_names(self) -> list[str]:
+ return [str(value) for value in self._cellNames]
+
+ def cell_columns(self) -> Iterator[tuple[str, np.ndarray]]:
+ yield from self._cellColumns.items()
+
+ def feature_columns(self) -> Iterator[tuple[str, np.ndarray]]:
+ yield from self._featureColumns.items()
+
+
+class MtxReader(CrReader):
+ """Read a selected Matrix Market triplet as cell-row sparse batches."""
+
+ def __init__(
+ self,
+ matrix_path: MtxCandidate | str,
+ feature_path: str | None = None,
+ cell_path: str | None = None,
+ *,
+ cell_metadata_path: str | None = None,
+ feature_reference_path: str | None = None,
+ cell_id_key: str | None = None,
+ mtx_separator: str = " ",
+ index_offset: int = -1,
+ is_filtered: bool = True,
+ filtering_cutoff: int = 500,
+ temp_dir: str | None = None,
+ dtype: DTypeLike = np.uint32,
+ ) -> None:
+ if isinstance(matrix_path, MtxCandidate):
+ if feature_path is not None or cell_path is not None:
+ raise ValueError("Do not pass explicit sidecars with an MtxCandidate")
+ candidate = replace(
+ matrix_path,
+ cellMetadataPath=(
+ cell_metadata_path
+ if cell_metadata_path is not None
+ else matrix_path.cellMetadataPath
+ ),
+ featureReferencePath=(
+ feature_reference_path
+ if feature_reference_path is not None
+ else matrix_path.featureReferencePath
+ ),
+ )
+ else:
+ if feature_path is None or cell_path is None:
+ raise ValueError(
+ "Explicit Matrix Market construction requires feature_path "
+ "and cell_path"
+ )
+ candidate = _explicit_candidate(
+ matrix_path,
+ feature_path,
+ cell_path,
+ cell_metadata_path=cell_metadata_path,
+ feature_reference_path=feature_reference_path,
+ )
+ self._engine = _MtxEngine(
+ candidate,
+ cell_id_key=cell_id_key,
+ separator=mtx_separator,
+ index_offset=index_offset,
+ is_filtered=is_filtered,
+ filtering_cutoff=filtering_cutoff,
+ temp_dir=temp_dir,
+ dtype=dtype,
+ )
+ self.validBarcodeIdx = self._engine.validCellIndexes
+ self.matrixEntryCount = self._engine.matrixEntryCount
+ self.coordinateOrder = self._engine.coordinateOrder
+ self.selectedCellIdKey = self._engine.selectedCellIdKey
+ self.temporaryDiskBytes = self._engine.temporaryDiskBytes
+ super().__init__(
+ {
+ "feature_ids": "feature_ids",
+ "feature_names": "feature_names",
+ "feature_types": "feature_types",
+ "cell_names": "cell_names",
+ }
+ )
+ self.nCells = self._engine.nCells
+
+ def _handle_version(self) -> dict[str, str]:
+ return {
+ "feature_ids": "feature_ids",
+ "feature_names": "feature_names",
+ "feature_types": "feature_types",
+ "cell_names": "cell_names",
+ }
+
+ def _read_dataset(self, key: str | None = None) -> list[str]:
+ if key is None:
+ raise ValueError("Dataset key must be provided")
+ values = {
+ "feature_ids": self._engine.feature_ids,
+ "feature_names": self._engine.feature_names,
+ "feature_types": self._engine.feature_types,
+ "cell_names": self._engine.cell_names,
+ }
+ if key not in values:
+ raise KeyError(key)
+ return values[key]()
+
+ @property
+ def matrix_dtype(self) -> np.dtype[Any]:
+ return self._engine.matrixDtype
+
+ def consume(
+ self,
+ batch_size: int,
+ lines_in_mem: int = 100000,
+ dtype: DTypeLike | None = None,
+ ) -> Generator[coo_matrix, None, None]:
+ yield from self._engine.consume(batch_size, lines_in_mem, dtype)
+
+ def max_window_nnz(self, window_rows: int) -> int:
+ return self._engine.max_window_nnz(window_rows)
+
+ def producer_staging_bytes(
+ self,
+ batch_size: int,
+ lines_in_mem: int,
+ ) -> int:
+ return self._engine.producer_staging_bytes(batch_size, lines_in_mem)
+
+ def get_cell_columns(self) -> Iterator[tuple[str, np.ndarray]]:
+ yield from self._engine.cell_columns()
+
+ def get_feature_columns(self) -> Iterator[tuple[str, np.ndarray]]:
+ yield from self._engine.feature_columns()
+
+ def _set_sparse_import_lines_in_mem(self, lines_in_mem: int) -> None:
+ self._engine.configure_import_lines(lines_in_mem)
+
+ def _prepare_sparse_import(self) -> None:
+ self._engine.prepare()
+ self.temporaryDiskBytes = self._engine.temporaryDiskBytes
+
+ def _release_sparse_import(self) -> None:
+ self._engine.release()
+
+ def _sparse_import_resident_bytes(self) -> int:
+ return self._engine.resident_bytes()
+
+ def close(self) -> None:
+ self._engine.release()
diff --git a/scarf/readers/seurat.py b/scarf/readers/seurat.py
new file mode 100644
index 00000000..f0c9c138
--- /dev/null
+++ b/scarf/readers/seurat.py
@@ -0,0 +1,3700 @@
+import os
+import re
+import shutil
+import sqlite3
+import tempfile
+from collections.abc import Iterator, Mapping, Sequence
+from dataclasses import dataclass
+from typing import Any, overload
+
+import numpy as np
+from numpy.typing import NDArray
+
+from ._rds import (
+ R_INT_NA,
+ AltRepValue,
+ LazyAtomicVector,
+ LazyStringVector,
+ PairValue,
+ RdsClosedError,
+ RdsDocument,
+ RdsLimits,
+ RNode,
+ RType,
+ get_attribute,
+ get_slot,
+ iter_attributes,
+ iter_named,
+ open_rds,
+)
+from ._seurat import (
+ CellBindMatrixSource,
+ DEFAULT_LIMITS,
+ FragmentSource,
+ LayerPlacement,
+ LayerStitchMatrixSource,
+ MatrixBlock,
+ MatrixSource,
+ MatrixSourceError,
+ MemoryEstimate,
+ SourceLimits,
+ UnsupportedMatrixOperation,
+ fragment_source_from_slots,
+ matrix_source_from_slots,
+)
+
+
+_VECTOR_BLOCK_SIZE = 65_536
+_SCALAR_MATRIX_SLOTS = frozenset(
+ {
+ "asSparse",
+ "as_sparse",
+ "dataset",
+ "dir",
+ "directory",
+ "dtype",
+ "along",
+ "buffer_size",
+ "compressed",
+ "filepath",
+ "function",
+ "group",
+ "layer",
+ "name",
+ "mode",
+ "op",
+ "operation",
+ "path",
+ "right",
+ "sparseLayout",
+ "sparse_layout",
+ "threads",
+ "tile_width",
+ "transpose",
+ "type",
+ "version",
+ "fillValue",
+ "fill_value",
+ "invert",
+ "keepNonzero",
+ "keep_nonzero",
+ }
+)
+_MATRIX_PARAMETER_SLOTS = frozenset(
+ {"Rvalue", "active_transforms", "col_params", "row_params"}
+)
+_SAFE_DELAYED_PRIMITIVES = frozenset(
+ {
+ "!",
+ "!=",
+ "%%",
+ "%/%",
+ "&",
+ "*",
+ "+",
+ "-",
+ "/",
+ "<",
+ "<=",
+ "==",
+ ">",
+ ">=",
+ "^",
+ "abs",
+ "acos",
+ "acosh",
+ "asin",
+ "asinh",
+ "atan",
+ "atanh",
+ "ceiling",
+ "cos",
+ "cosh",
+ "exp",
+ "expm1",
+ "floor",
+ "identity",
+ "is.finite",
+ "is.infinite",
+ "is.na",
+ "is.nan",
+ "log",
+ "log10",
+ "log1p",
+ "log2",
+ "pmax",
+ "pmin",
+ "round",
+ "sign",
+ "sin",
+ "sinh",
+ "sqrt",
+ "tan",
+ "tanh",
+ "trunc",
+ "|",
+ }
+)
+_ROOT_IMPORTED_SLOTS = frozenset(
+ {"active.assay", "active.ident", "assays", "class", "meta.data", "reductions"}
+)
+_ROOT_IGNORED_SLOTS = frozenset(
+ {
+ "commands",
+ "graphs",
+ "images",
+ "misc",
+ "neighbors",
+ "tools",
+ }
+)
+
+
+class SeuratImportError(ValueError):
+ def __init__(
+ self,
+ message: str,
+ *,
+ object_path: str,
+ code: str,
+ context: Mapping[str, Any] | None = None,
+ ) -> None:
+ self.message = message
+ self.objectPath = object_path
+ self.object_path = object_path
+ self.code = code
+ self.context = dict(context or {})
+ super().__init__(f"{message} at {object_path} [{code}]")
+
+
+@dataclass(frozen=True, slots=True)
+class SeuratDiagnostic:
+ code: str
+ message: str
+ objectPath: str
+ context: dict[str, Any]
+
+ @classmethod
+ def from_error(cls, error: SeuratImportError) -> "SeuratDiagnostic":
+ return cls(
+ code=error.code,
+ message=error.message,
+ objectPath=error.objectPath,
+ context=dict(error.context),
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class SeuratNotice:
+ code: str
+ message: str
+ objectPath: str
+ context: dict[str, Any]
+
+
+@dataclass(frozen=True, slots=True)
+class SeuratItemInspection:
+ name: str
+ importable: bool
+ sourceClass: str | None
+ dimensions: tuple[int, ...] | None
+ objectPath: str
+ dtype: str | None = None
+ backend: str | None = None
+ memoryEstimate: MemoryEstimate | None = None
+ blockingDiagnostic: SeuratDiagnostic | None = None
+ notices: tuple[SeuratNotice, ...] = ()
+
+
+@dataclass(frozen=True, slots=True)
+class SeuratInspectResult:
+ source: str
+ sourceDigest: str
+ payloadDigest: str
+ compression: str
+ activeAssay: str
+ nCells: int
+ assays: tuple[SeuratItemInspection, ...]
+ reductions: tuple[SeuratItemInspection, ...]
+ cellMetadata: SeuratItemInspection
+ activeIdentity: SeuratItemInspection
+ notices: tuple[SeuratNotice, ...] = ()
+
+ def assay(self, name: str) -> SeuratItemInspection:
+ for item in self.assays:
+ if item.name == name:
+ return item
+ raise KeyError(name)
+
+ def reduction(self, name: str) -> SeuratItemInspection:
+ for item in self.reductions:
+ if item.name == name:
+ return item
+ raise KeyError(name)
+
+
+@dataclass(frozen=True, slots=True)
+class SeuratColumnBlock:
+ values: NDArray[Any] | tuple[str | bytes | None, ...]
+ missing: NDArray[np.bool_]
+
+
+@dataclass(frozen=True, slots=True)
+class _CachedLayerSpec:
+ assay: str
+ layer: str
+ path: str
+ sourceClass: str
+ package: str
+ loader: str
+ objectPath: str
+
+
+def _parse_cache_loader_list(value: str, *, object_path: str) -> tuple[str, ...]:
+ result: list[str] = []
+ index = 0
+ while index < len(value):
+ while index < len(value) and value[index].isspace():
+ index += 1
+ if index >= len(value) or value[index] not in {"'", '"'}:
+ raise _error(
+ "SaveSeuratRds composite loader contains an invalid string list",
+ object_path=object_path,
+ code="unsupported_sidecar_cache_recipe",
+ )
+ quote = value[index]
+ index += 1
+ characters: list[str] = []
+ while index < len(value) and value[index] != quote:
+ character = value[index]
+ if character == "\\":
+ index += 1
+ if index >= len(value) or value[index] not in {quote, "\\"}:
+ raise _error(
+ "SaveSeuratRds composite loader uses an unsupported escape",
+ object_path=object_path,
+ code="unsupported_sidecar_cache_recipe",
+ )
+ character = value[index]
+ characters.append(character)
+ index += 1
+ if index >= len(value):
+ raise _error(
+ "SaveSeuratRds composite loader has an unterminated string",
+ object_path=object_path,
+ code="unsupported_sidecar_cache_recipe",
+ )
+ index += 1
+ result.append("".join(characters))
+ while index < len(value) and value[index].isspace():
+ index += 1
+ if index == len(value):
+ break
+ if value[index] != ",":
+ raise _error(
+ "SaveSeuratRds composite loader string list is malformed",
+ object_path=object_path,
+ code="unsupported_sidecar_cache_recipe",
+ )
+ index += 1
+ if not result:
+ raise _error(
+ "SaveSeuratRds composite loader has no leaf recipes",
+ object_path=object_path,
+ code="unsupported_sidecar_cache_recipe",
+ )
+ return tuple(result)
+
+
+class SeuratStringVector(Sequence[str]):
+ def __init__(
+ self,
+ values: LazyStringVector,
+ document: RdsDocument,
+ *,
+ object_path: str,
+ ) -> None:
+ self._values = values
+ self._document = document
+ self.objectPath = object_path
+
+ @property
+ def shape(self) -> tuple[int]:
+ return (len(self),)
+
+ def __len__(self) -> int:
+ return len(self._values)
+
+ def _ensure_open(self) -> None:
+ if self._document.closed:
+ raise RdsClosedError("RDS document is closed", path=self.objectPath)
+
+ def read_block(self, start: int, stop: int) -> tuple[str, ...]:
+ self._ensure_open()
+ if start < 0 or stop < start or stop > len(self):
+ raise IndexError(
+ f"identifier window [{start}, {stop}) is outside [0, {len(self)})"
+ )
+ return tuple(
+ _as_text(value, object_path=f"{self.objectPath}/{start + offset}")
+ for offset, value in enumerate(self._values.read_block(start, stop))
+ )
+
+ def iter_blocks(
+ self, block_size: int = _VECTOR_BLOCK_SIZE
+ ) -> Iterator[tuple[str, ...]]:
+ if block_size <= 0:
+ raise ValueError("block_size must be positive")
+ for start in range(0, len(self), block_size):
+ yield self.read_block(start, min(len(self), start + block_size))
+
+ @overload
+ def __getitem__(self, key: int) -> str: ...
+
+ @overload
+ def __getitem__(self, key: slice) -> tuple[str, ...]: ...
+
+ def __getitem__(self, key: int | slice) -> str | tuple[str, ...]:
+ if isinstance(key, slice):
+ start, stop, step = key.indices(len(self))
+ if step == 1:
+ return self.read_block(start, stop)
+ return tuple(self[index] for index in range(start, stop, step))
+ index = key + len(self) if key < 0 else key
+ if index < 0 or index >= len(self):
+ raise IndexError("identifier index out of range")
+ return self.read_block(index, index + 1)[0]
+
+ def __eq__(self, other: object) -> bool:
+ return (
+ isinstance(other, Sequence)
+ and not isinstance(other, str | bytes)
+ and _identifiers_equal(self, other)
+ )
+
+
+class _IndexedStringVector(Sequence[str]):
+ def __init__(
+ self,
+ values: Sequence[str],
+ indexes: NDArray[np.int64],
+ *,
+ object_path: str,
+ ) -> None:
+ self._values = values
+ self._indexes = indexes
+ self.objectPath = object_path
+
+ def __len__(self) -> int:
+ return int(self._indexes.size)
+
+ def read_block(self, start: int, stop: int) -> tuple[str, ...]:
+ if start < 0 or stop < start or stop > len(self):
+ raise IndexError(
+ f"identifier window [{start}, {stop}) is outside [0, {len(self)})"
+ )
+ return tuple(self._values[int(index)] for index in self._indexes[start:stop])
+
+ @overload
+ def __getitem__(self, key: int) -> str: ...
+
+ @overload
+ def __getitem__(self, key: slice) -> tuple[str, ...]: ...
+
+ def __getitem__(self, key: int | slice) -> str | tuple[str, ...]:
+ if isinstance(key, slice):
+ start, stop, step = key.indices(len(self))
+ if step == 1:
+ return self.read_block(start, stop)
+ return tuple(self[index] for index in range(start, stop, step))
+ index = key + len(self) if key < 0 else key
+ if index < 0 or index >= len(self):
+ raise IndexError("identifier index out of range")
+ return self._values[int(self._indexes[index])]
+
+ def __eq__(self, other: object) -> bool:
+ return (
+ isinstance(other, Sequence)
+ and not isinstance(other, str | bytes)
+ and _identifiers_equal(self, other)
+ )
+
+
+class SeuratMembership:
+ def __init__(
+ self,
+ length: int,
+ positions: NDArray[np.int64] | None = None,
+ ) -> None:
+ self._length = int(length)
+ if self._length < 0:
+ raise ValueError("membership length cannot be negative")
+ if positions is None:
+ self._positions = None
+ return
+ normalized = np.asarray(positions, dtype=np.int64)
+ if normalized.ndim != 1:
+ raise ValueError("membership positions must be one-dimensional")
+ if normalized.size and (
+ int(normalized.min()) < 0 or int(normalized.max()) >= self._length
+ ):
+ raise ValueError("membership position is out of range")
+ if normalized.size > 1 and np.any(normalized[1:] <= normalized[:-1]):
+ normalized = np.sort(normalized)
+ if np.any(normalized[1:] == normalized[:-1]):
+ raise ValueError("membership positions contain duplicates")
+ if normalized.size == self._length:
+ self._positions = None
+ else:
+ self._positions = normalized
+
+ @property
+ def allIncluded(self) -> bool:
+ return self._positions is None
+
+ def __len__(self) -> int:
+ return self._length
+
+ def read_block(self, start: int, stop: int) -> NDArray[np.bool_]:
+ if start < 0 or stop < start or stop > len(self):
+ raise IndexError(
+ f"membership window [{start}, {stop}) is outside [0, {len(self)})"
+ )
+ if self._positions is None:
+ return np.ones(stop - start, dtype=np.bool_)
+ result = np.zeros(stop - start, dtype=np.bool_)
+ left = int(np.searchsorted(self._positions, start, side="left"))
+ right = int(np.searchsorted(self._positions, stop, side="left"))
+ result[self._positions[left:right] - start] = True
+ return result
+
+ @overload
+ def __getitem__(self, key: int) -> bool: ...
+
+ @overload
+ def __getitem__(self, key: slice) -> NDArray[np.bool_]: ...
+
+ def __getitem__(self, key: int | slice) -> bool | NDArray[np.bool_]:
+ if isinstance(key, slice):
+ start, stop, step = key.indices(len(self))
+ block = self.read_block(start, stop)
+ return block if step == 1 else block[::step]
+ index = key + len(self) if key < 0 else key
+ if index < 0 or index >= len(self):
+ raise IndexError("membership index out of range")
+ return bool(self.read_block(index, index + 1)[0])
+
+ def __array__(
+ self,
+ dtype: Any | None = None,
+ copy: bool | None = None,
+ ) -> NDArray[Any]:
+ values = self.read_block(0, len(self))
+ if copy:
+ values = values.copy()
+ return values.astype(dtype, copy=False) if dtype is not None else values
+
+
+class SeuratMetadataColumn:
+ def __init__(
+ self,
+ *,
+ name: str,
+ kind: str,
+ values: LazyAtomicVector | LazyStringVector,
+ length: int,
+ document: RdsDocument,
+ source_indices: NDArray[np.int64] | None = None,
+ levels: tuple[str, ...] = (),
+ ordered: bool = False,
+ object_path: str,
+ ) -> None:
+ self.name = name
+ self.kind = kind
+ self.length = length
+ self.levels = levels
+ self.ordered = ordered
+ self.objectPath = object_path
+ self._values = values
+ self._document = document
+ self._sourceIndices = source_indices
+
+ @property
+ def sourceIndices(self) -> NDArray[np.int64] | None:
+ return self._sourceIndices
+
+ def _ensure_open(self) -> None:
+ if self._document.closed:
+ raise RdsClosedError("RDS document is closed", path=self.objectPath)
+
+ def _window(self, start: int, stop: int) -> tuple[int, int]:
+ if isinstance(start, bool) or isinstance(stop, bool):
+ raise TypeError("metadata bounds must be integers")
+ start = int(start)
+ stop = int(stop)
+ if start < 0 or stop < start or stop > self.length:
+ raise IndexError(
+ f"metadata window [{start}, {stop}) is outside [0, {self.length})"
+ )
+ return start, stop
+
+ def _read_atomic(self, start: int, stop: int) -> NDArray[Any]:
+ values = self._values
+ if not isinstance(values, LazyAtomicVector):
+ raise TypeError(f"{self.objectPath} is not an atomic column")
+ if self._sourceIndices is None:
+ return values.read_block(start, stop)
+ indexes = self._sourceIndices[start:stop]
+ output = np.empty(indexes.size, dtype=values.dtype)
+ run_start = 0
+ for position in range(1, indexes.size + 1):
+ finished = (
+ position == indexes.size
+ or indexes[position] != indexes[position - 1] + 1
+ )
+ if not finished:
+ continue
+ source_start = int(indexes[run_start])
+ source_stop = int(indexes[position - 1]) + 1
+ output[run_start:position] = values.read_block(source_start, source_stop)
+ run_start = position
+ return output
+
+ def _read_strings(self, start: int, stop: int) -> tuple[str | bytes | None, ...]:
+ values = self._values
+ if not isinstance(values, LazyStringVector):
+ raise TypeError(f"{self.objectPath} is not a character column")
+ if self._sourceIndices is None:
+ return tuple(values.read_block(start, stop))
+ return tuple(values[int(index)] for index in self._sourceIndices[start:stop])
+
+ def read_block(self, start: int, stop: int) -> SeuratColumnBlock:
+ self._ensure_open()
+ start, stop = self._window(start, stop)
+ if self.kind == "character":
+ string_values = self._read_strings(start, stop)
+ string_missing = np.fromiter(
+ (value is None for value in string_values),
+ dtype=np.bool_,
+ count=len(string_values),
+ )
+ return SeuratColumnBlock(string_values, string_missing)
+
+ atomic_values = self._read_atomic(start, stop)
+ if self.kind in {"integer", "factor"}:
+ atomic_missing = atomic_values == R_INT_NA
+ elif self.kind == "logical":
+ atomic_missing = atomic_values == R_INT_NA
+ atomic_values = atomic_values == 1
+ elif self.kind == "real":
+ atomic_missing = np.isnan(atomic_values)
+ else:
+ raise AssertionError(f"unknown metadata kind {self.kind!r}")
+ return SeuratColumnBlock(
+ atomic_values,
+ np.asarray(atomic_missing, dtype=np.bool_),
+ )
+
+ def read_decoded_block(
+ self, start: int, stop: int
+ ) -> tuple[str | bytes | int | float | bool | None, ...]:
+ block = self.read_block(start, stop)
+ if self.kind == "factor":
+ numeric = np.asarray(block.values)
+ return tuple(
+ None if block.missing[index] else self.levels[int(value) - 1]
+ for index, value in enumerate(numeric)
+ )
+ if isinstance(block.values, tuple):
+ return block.values
+ return tuple(
+ None if block.missing[index] else value.item()
+ for index, value in enumerate(block.values)
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class SeuratMetadata:
+ rowIds: Sequence[str]
+ columns: tuple[SeuratMetadataColumn, ...]
+ objectPath: str
+
+ @property
+ def columnNames(self) -> tuple[str, ...]:
+ return tuple(column.name for column in self.columns)
+
+ def column(self, name: str) -> SeuratMetadataColumn:
+ for column in self.columns:
+ if column.name == name:
+ return column
+ raise KeyError(name)
+
+
+class SeuratNumericVector:
+ def __init__(
+ self,
+ values: LazyAtomicVector,
+ document: RdsDocument,
+ *,
+ object_path: str,
+ ) -> None:
+ self._values = values
+ self._document = document
+ self.objectPath = object_path
+
+ @property
+ def length(self) -> int:
+ return len(self._values)
+
+ @property
+ def dtype(self) -> np.dtype[Any]:
+ return self._values.dtype
+
+ def __len__(self) -> int:
+ return self.length
+
+ def read_block(self, start: int, stop: int) -> NDArray[Any]:
+ if self._document.closed:
+ raise RdsClosedError("RDS document is closed", path=self.objectPath)
+ return self._values.read_block(start, stop)
+
+
+class SeuratRMatrix:
+ def __init__(
+ self,
+ values: LazyAtomicVector,
+ shape: tuple[int, int],
+ *,
+ row_ids: Sequence[str],
+ column_ids: Sequence[str],
+ document: RdsDocument,
+ object_path: str,
+ ) -> None:
+ if len(values) != shape[0] * shape[1]:
+ raise SeuratImportError(
+ f"matrix has {len(values)} values but shape {shape} needs "
+ f"{shape[0] * shape[1]}",
+ object_path=object_path,
+ code="matrix_length_mismatch",
+ )
+ self._values = values
+ self.shape = shape
+ self.rowIds = row_ids
+ self.columnIds = column_ids
+ self._document = document
+ self.objectPath = object_path
+
+ @property
+ def dtype(self) -> np.dtype[Any]:
+ return self._values.dtype
+
+ def read_rows(self, start: int, stop: int) -> NDArray[Any]:
+ if self._document.closed:
+ raise RdsClosedError("RDS document is closed", path=self.objectPath)
+ if isinstance(start, bool) or isinstance(stop, bool):
+ raise TypeError("matrix bounds must be integers")
+ start = int(start)
+ stop = int(stop)
+ if start < 0 or stop < start or stop > self.shape[0]:
+ raise IndexError(
+ f"matrix window [{start}, {stop}) is outside [0, {self.shape[0]})"
+ )
+ output = np.empty((stop - start, self.shape[1]), dtype=self.dtype)
+ for column in range(self.shape[1]):
+ offset = column * self.shape[0]
+ output[:, column] = self._values.read_block(offset + start, offset + stop)
+ return output
+
+ def read_cells(self, start: int, stop: int) -> NDArray[Any]:
+ return self.read_rows(start, stop)
+
+
+class _OwnedMatrixSource:
+ def __init__(
+ self,
+ source: MatrixSource,
+ document: RdsDocument,
+ *,
+ object_path: str,
+ ) -> None:
+ self._source = source
+ self._document = document
+ self.objectPath = object_path
+
+ def _ensure_open(self) -> None:
+ if self._document.closed:
+ raise RdsClosedError("RDS document is closed", path=self.objectPath)
+
+ @property
+ def shape(self) -> tuple[int, int]:
+ self._ensure_open()
+ return self._source.shape
+
+ @property
+ def dtype(self) -> np.dtype[Any]:
+ self._ensure_open()
+ return self._source.dtype
+
+ @property
+ def row_names(self) -> tuple[str, ...] | None:
+ self._ensure_open()
+ return self._source.row_names
+
+ @property
+ def column_names(self) -> tuple[str, ...] | None:
+ self._ensure_open()
+ return self._source.column_names
+
+ @property
+ def is_sparse(self) -> bool:
+ self._ensure_open()
+ return self._source.is_sparse
+
+ @property
+ def zero_preserving(self) -> bool:
+ self._ensure_open()
+ return self._source.zero_preserving
+
+ @property
+ def resident_bytes(self) -> int:
+ self._ensure_open()
+ return self._source.resident_bytes
+
+ def read_cells(self, start: int, stop: int) -> MatrixBlock:
+ self._ensure_open()
+ return self._source.read_cells(start, stop)
+
+ def estimate_read_memory(self, start: int, stop: int) -> MemoryEstimate:
+ self._ensure_open()
+ return self._source.estimate_read_memory(start, stop)
+
+
+@dataclass(frozen=True, slots=True)
+class SeuratAssay:
+ name: str
+ sourceClass: str
+ counts: MatrixSource
+ featureIds: Sequence[str]
+ cellIds: Sequence[str]
+ assayCellIds: Sequence[str]
+ cellMembership: SeuratMembership
+ featureMetadata: SeuratMetadata
+ notices: tuple[SeuratNotice, ...]
+ objectPath: str
+
+ @property
+ def matrix(self) -> MatrixSource:
+ return self.counts
+
+ @property
+ def dimensions(self) -> tuple[int, int]:
+ return self.counts.shape
+
+
+@dataclass(frozen=True, slots=True)
+class SeuratReduction:
+ name: str
+ sourceClass: str
+ role: str
+ cellEmbeddings: SeuratRMatrix
+ featureLoadings: SeuratRMatrix | None
+ stdev: SeuratNumericVector | None
+ assayUsed: str
+ key: str
+ globalReduction: bool
+ imported: bool
+ computedByScarf: bool
+ notices: tuple[SeuratNotice, ...]
+ objectPath: str
+
+ @property
+ def dimensions(self) -> tuple[int, int]:
+ return self.cellEmbeddings.shape
+
+
+@dataclass(frozen=True, slots=True)
+class _LogMap:
+ rowIds: Sequence[str]
+ layerNames: Sequence[str]
+ values: LazyAtomicVector
+ objectPath: str
+ maximumIndexBytes: int
+
+ def membership(self, layer_name: str) -> NDArray[np.int64]:
+ try:
+ column = self.layerNames.index(layer_name)
+ except ValueError as error:
+ raise SeuratImportError(
+ f"LogMap has no column for layer {layer_name!r}",
+ object_path=self.objectPath,
+ code="logmap_layer_missing",
+ context={"layer": layer_name},
+ ) from error
+ required_bytes = len(self.rowIds) * (
+ np.dtype(np.int32).itemsize + np.dtype(np.int64).itemsize
+ )
+ if required_bytes > self.maximumIndexBytes:
+ raise SeuratImportError(
+ "LogMap membership exceeds its memory budget",
+ object_path=f"{self.objectPath}/{layer_name}",
+ code="metadata_index_limit",
+ context={
+ "requiredBytes": required_bytes,
+ "maximumBytes": self.maximumIndexBytes,
+ },
+ )
+ start = column * len(self.rowIds)
+ values = self.values.read_block(start, start + len(self.rowIds))
+ invalid = (values != 0) & (values != 1)
+ if np.any(invalid):
+ offset = int(np.flatnonzero(invalid)[0])
+ raise SeuratImportError(
+ "LogMap membership must contain only TRUE or FALSE",
+ object_path=f"{self.objectPath}/{layer_name}",
+ code="invalid_logmap_membership",
+ context={"row": offset, "value": int(values[offset])},
+ )
+ return np.flatnonzero(values == 1).astype(np.int64, copy=False)
+
+
+def _error(
+ message: str,
+ *,
+ object_path: str,
+ code: str,
+ **context: Any,
+) -> SeuratImportError:
+ return SeuratImportError(
+ message,
+ object_path=object_path,
+ code=code,
+ context=context,
+ )
+
+
+def _as_text(value: str | bytes | None, *, object_path: str) -> str:
+ if value is None:
+ raise _error(
+ "identifier is missing",
+ object_path=object_path,
+ code="missing_id",
+ )
+ if isinstance(value, bytes):
+ try:
+ value = value.decode("utf-8")
+ except UnicodeDecodeError as error:
+ raise _error(
+ "identifier is not valid UTF-8",
+ object_path=object_path,
+ code="invalid_id_encoding",
+ ) from error
+ try:
+ value.encode("utf-8")
+ except UnicodeEncodeError as error:
+ raise _error(
+ "identifier is not valid Unicode",
+ object_path=object_path,
+ code="invalid_id_encoding",
+ ) from error
+ if not value:
+ raise _error(
+ "identifier is empty",
+ object_path=object_path,
+ code="missing_id",
+ )
+ if "\x00" in value:
+ raise _error(
+ "identifier contains a NUL character",
+ object_path=object_path,
+ code="invalid_id",
+ )
+ return value
+
+
+def _string_values(node: RNode, *, object_path: str) -> LazyStringVector:
+ if node.type is not RType.STRING or not isinstance(node.value, LazyStringVector):
+ raise _error(
+ "expected a character vector",
+ object_path=object_path,
+ code="invalid_character_vector",
+ rType=node.type.name,
+ )
+ return node.value
+
+
+def _read_text_vector(
+ node: RNode,
+ *,
+ object_path: str,
+ maximum_bytes: int = DEFAULT_LIMITS.maxMetadataBytes,
+) -> tuple[str, ...]:
+ values = _string_values(node, object_path=object_path)
+ result: list[str] = []
+ used_bytes = 0
+ for start in range(0, len(values), _VECTOR_BLOCK_SIZE):
+ block = values.read_block(start, min(len(values), start + _VECTOR_BLOCK_SIZE))
+ for offset, value in enumerate(block):
+ text = _as_text(value, object_path=f"{object_path}/{start + offset}")
+ used_bytes += len(text.encode("utf-8")) + 8
+ if used_bytes > maximum_bytes:
+ raise _error(
+ "text vector exceeds its metadata memory budget",
+ object_path=object_path,
+ code="metadata_index_limit",
+ requiredBytes=used_bytes,
+ maximumBytes=maximum_bytes,
+ )
+ result.append(text)
+ return tuple(result)
+
+
+def _identifier_block(
+ values: Sequence[str],
+ start: int,
+ stop: int,
+ *,
+ object_path: str,
+) -> tuple[str, ...]:
+ read_block = getattr(values, "read_block", None)
+ raw = read_block(start, stop) if callable(read_block) else values[start:stop]
+ if len(raw) != stop - start:
+ raise _error(
+ "identifier source returned an invalid block length",
+ object_path=object_path,
+ code="invalid_id_source",
+ expected=stop - start,
+ actual=len(raw),
+ )
+ return tuple(
+ _as_text(value, object_path=f"{object_path}/{start + offset}")
+ for offset, value in enumerate(raw)
+ )
+
+
+def _identifiers_equal(left: Sequence[str], right: Sequence[str]) -> bool:
+ if len(left) != len(right):
+ return False
+ for start in range(0, len(left), _VECTOR_BLOCK_SIZE):
+ stop = min(len(left), start + _VECTOR_BLOCK_SIZE)
+ if _identifier_block(
+ left,
+ start,
+ stop,
+ object_path="$left_ids",
+ ) != _identifier_block(
+ right,
+ start,
+ stop,
+ object_path="$right_ids",
+ ):
+ return False
+ return True
+
+
+def _identifier_database(
+ *,
+ scratch_dir: str | os.PathLike[str] | None,
+ maximum_bytes: int,
+ object_path: str,
+) -> tuple[sqlite3.Connection, str]:
+ if maximum_bytes < 8_192:
+ raise _error(
+ "identifier index budget is too small",
+ object_path=object_path,
+ code="metadata_index_limit",
+ maximumBytes=maximum_bytes,
+ )
+ file = tempfile.NamedTemporaryFile(
+ prefix="scarf-seurat-ids-",
+ suffix=".sqlite3",
+ dir=scratch_dir,
+ delete=False,
+ )
+ path = file.name
+ file.close()
+ try:
+ free_bytes = shutil.disk_usage(os.path.dirname(path)).free
+ allowed_bytes = min(maximum_bytes, free_bytes)
+ if allowed_bytes < 8_192:
+ raise _error(
+ "identifier index has insufficient scratch space",
+ object_path=object_path,
+ code="metadata_index_limit",
+ maximumBytes=maximum_bytes,
+ freeBytes=free_bytes,
+ )
+ connection = sqlite3.connect(path)
+ connection.execute("PRAGMA journal_mode=OFF")
+ connection.execute("PRAGMA synchronous=OFF")
+ connection.execute("PRAGMA temp_store=FILE")
+ page_size = int(connection.execute("PRAGMA page_size").fetchone()[0])
+ connection.execute(
+ f"PRAGMA max_page_count={max(2, allowed_bytes // page_size)}"
+ )
+ connection.execute(
+ "CREATE TABLE ids "
+ "(value TEXT PRIMARY KEY, position INTEGER NOT NULL, matched INTEGER NOT NULL) "
+ "WITHOUT ROWID"
+ )
+ return connection, path
+ except Exception:
+ try:
+ os.unlink(path)
+ except FileNotFoundError:
+ pass
+ raise
+
+
+def _close_identifier_database(connection: sqlite3.Connection, path: str) -> None:
+ connection.close()
+ try:
+ os.unlink(path)
+ except FileNotFoundError:
+ pass
+
+
+def _populate_identifier_database(
+ connection: sqlite3.Connection,
+ values: Sequence[str],
+ *,
+ object_path: str,
+) -> None:
+ try:
+ for start in range(0, len(values), _VECTOR_BLOCK_SIZE):
+ stop = min(len(values), start + _VECTOR_BLOCK_SIZE)
+ block = _identifier_block(
+ values,
+ start,
+ stop,
+ object_path=object_path,
+ )
+ connection.executemany(
+ "INSERT INTO ids(value, position, matched) VALUES (?, ?, 0)",
+ ((value, start + offset) for offset, value in enumerate(block)),
+ )
+ connection.commit()
+ except sqlite3.IntegrityError as error:
+ raise _error(
+ "identifiers are duplicated",
+ object_path=object_path,
+ code="duplicate_id",
+ ) from error
+ except sqlite3.OperationalError as error:
+ raise _error(
+ "identifier index exceeds its disk budget",
+ object_path=object_path,
+ code="metadata_index_limit",
+ ) from error
+
+
+def _validate_unique_ids(
+ values: Sequence[str],
+ *,
+ object_path: str,
+ scratch_dir: str | os.PathLike[str] | None,
+ maximum_bytes: int,
+) -> None:
+ if not values:
+ return
+ connection, path = _identifier_database(
+ scratch_dir=scratch_dir,
+ maximum_bytes=maximum_bytes,
+ object_path=object_path,
+ )
+ try:
+ _populate_identifier_database(
+ connection,
+ values,
+ object_path=object_path,
+ )
+ finally:
+ _close_identifier_database(connection, path)
+
+
+def _validate_unique(values: tuple[str, ...], *, object_path: str) -> None:
+ seen: set[str] = set()
+ for index, value in enumerate(values):
+ if value in seen:
+ raise _error(
+ f"identifier {value!r} is duplicated",
+ object_path=f"{object_path}/{index}",
+ code="duplicate_id",
+ id=value,
+ )
+ seen.add(value)
+
+
+def _class_names(node: RNode, *, object_path: str) -> tuple[str, ...]:
+ class_node = get_attribute(node, "class")
+ if class_node is None:
+ return ()
+ names = _read_text_vector(class_node, object_path=f"{object_path}/class")
+ if not names:
+ raise _error(
+ "class vector is empty",
+ object_path=f"{object_path}/class",
+ code="empty_class",
+ )
+ return names
+
+
+def _require_slot(node: RNode, name: str, *, object_path: str) -> RNode:
+ value = get_slot(node, name)
+ if value is None:
+ raise _error(
+ f"required slot {name!r} is missing",
+ object_path=f"{object_path}/{name}",
+ code="missing_slot",
+ slot=name,
+ )
+ return value
+
+
+def _named_nodes(node: RNode, *, object_path: str) -> tuple[tuple[str, RNode], ...]:
+ try:
+ raw = tuple(iter_named(node))
+ except (TypeError, ValueError) as error:
+ raise _error(
+ "expected a named list",
+ object_path=object_path,
+ code="invalid_named_list",
+ cause=str(error),
+ ) from error
+ result: list[tuple[str, RNode]] = []
+ names: list[str] = []
+ for index, (name, value) in enumerate(raw):
+ normalized = _as_text(name, object_path=f"{object_path}/names/{index}")
+ result.append((normalized, value))
+ names.append(normalized)
+ _validate_unique(tuple(names), object_path=f"{object_path}/names")
+ return tuple(result)
+
+
+def _read_integer_vector(
+ node: RNode,
+ *,
+ object_path: str,
+ expected_length: int | None = None,
+) -> NDArray[np.int64]:
+ if node.type not in {RType.INTEGER, RType.LOGICAL} or not isinstance(
+ node.value, LazyAtomicVector
+ ):
+ raise _error(
+ "expected an integer vector",
+ object_path=object_path,
+ code="invalid_integer_vector",
+ rType=node.type.name,
+ )
+ if expected_length is not None and len(node.value) != expected_length:
+ raise _error(
+ f"integer vector has length {len(node.value)}; expected {expected_length}",
+ object_path=object_path,
+ code="length_mismatch",
+ actual=len(node.value),
+ expected=expected_length,
+ )
+ return node.value.read_block(0, len(node.value)).astype(np.int64, copy=False)
+
+
+def _read_text_scalar(node: RNode, *, object_path: str) -> str:
+ values = _string_values(node, object_path=object_path)
+ if len(values) != 1:
+ raise _error(
+ f"expected one string, found {len(values)}",
+ object_path=object_path,
+ code="invalid_scalar",
+ actualLength=len(values),
+ )
+ return _as_text(values[0], object_path=object_path)
+
+
+def _read_logical_scalar(node: RNode, *, object_path: str) -> bool:
+ if node.type is not RType.LOGICAL or not isinstance(node.value, LazyAtomicVector):
+ raise _error(
+ "expected one logical value",
+ object_path=object_path,
+ code="invalid_logical_scalar",
+ rType=node.type.name,
+ )
+ if len(node.value) != 1:
+ raise _error(
+ f"expected one logical value, found {len(node.value)}",
+ object_path=object_path,
+ code="invalid_logical_scalar",
+ actualLength=len(node.value),
+ )
+ value = int(node.value[0])
+ if value not in {0, 1}:
+ raise _error(
+ "logical scalar cannot be missing",
+ object_path=object_path,
+ code="invalid_logical_scalar",
+ value=value,
+ )
+ return bool(value)
+
+
+def _node_has_content(node: RNode) -> bool:
+ if node.is_null:
+ return False
+ if isinstance(node.value, LazyAtomicVector | LazyStringVector | tuple):
+ return len(node.value) > 0
+ return True
+
+
+def _matrix_dimensions(node: RNode, *, object_path: str) -> tuple[int, int]:
+ dim = get_attribute(node, "Dim") or get_attribute(node, "dim")
+ if dim is None:
+ raise _error(
+ "matrix dimensions are missing",
+ object_path=f"{object_path}/dim",
+ code="missing_matrix_dimensions",
+ )
+ values = _read_integer_vector(
+ dim,
+ object_path=f"{object_path}/dim",
+ expected_length=2,
+ )
+ shape = (int(values[0]), int(values[1]))
+ if shape[0] < 0 or shape[1] < 0:
+ raise _error(
+ f"matrix dimensions cannot be negative: {shape}",
+ object_path=f"{object_path}/dim",
+ code="invalid_matrix_dimensions",
+ dimensions=shape,
+ )
+ return shape
+
+
+def _matrix_dimnames(
+ node: RNode,
+ *,
+ document: RdsDocument,
+ object_path: str,
+ require_both: bool,
+) -> tuple[Sequence[str] | None, Sequence[str] | None]:
+ dimnames = get_attribute(node, "Dimnames") or get_attribute(node, "dimnames")
+ if dimnames is None:
+ if require_both:
+ raise _error(
+ "matrix Dimnames are missing",
+ object_path=f"{object_path}/Dimnames",
+ code="missing_dimnames",
+ )
+ return None, None
+ if dimnames.type is not RType.VECTOR or not isinstance(dimnames.value, tuple):
+ raise _error(
+ "matrix Dimnames must contain two axes",
+ object_path=f"{object_path}/Dimnames",
+ code="invalid_dimnames",
+ )
+ if len(dimnames.value) != 2:
+ raise _error(
+ f"matrix Dimnames has {len(dimnames.value)} axes; expected 2",
+ object_path=f"{object_path}/Dimnames",
+ code="invalid_dimnames",
+ actualLength=len(dimnames.value),
+ )
+ axes: list[Sequence[str] | None] = []
+ for index, axis in enumerate(dimnames.value):
+ axis_path = f"{object_path}/Dimnames/{index}"
+ if axis.is_null:
+ if require_both:
+ raise _error(
+ "matrix axis identifiers are missing",
+ object_path=axis_path,
+ code="missing_dimnames",
+ )
+ axes.append(None)
+ continue
+ axes.append(
+ SeuratStringVector(
+ _string_values(axis, object_path=axis_path),
+ document,
+ object_path=axis_path,
+ )
+ )
+ return axes[0], axes[1]
+
+
+def _unwrap_atomic(node: RNode, *, object_path: str) -> LazyAtomicVector:
+ if node.type in {
+ RType.LOGICAL,
+ RType.INTEGER,
+ RType.REAL,
+ RType.COMPLEX,
+ RType.RAW,
+ } and isinstance(node.value, LazyAtomicVector):
+ return node.value
+ if node.type is RType.ALTREP and isinstance(node.value, AltRepValue):
+ altrep = node.value
+ if altrep.known and altrep.class_name in {
+ "wrap_complex",
+ "wrap_integer",
+ "wrap_logical",
+ "wrap_raw",
+ "wrap_real",
+ }:
+ state = altrep.state
+ if state.type is RType.PAIRLIST and isinstance(state.value, PairValue):
+ return _unwrap_atomic(state.value.car, object_path=object_path)
+ raise _error(
+ f"ALTREP class {altrep.class_name!r} is not usable as an atomic matrix",
+ object_path=object_path,
+ code="unsupported_altrep",
+ className=altrep.class_name,
+ packageName=altrep.package_name,
+ )
+ raise _error(
+ "expected an atomic vector",
+ object_path=object_path,
+ code="invalid_atomic_vector",
+ rType=node.type.name,
+ )
+
+
+def _matrix_slot_value(
+ node: RNode,
+ *,
+ name: str,
+ object_path: str,
+ reader: "SeuratReader",
+) -> Any:
+ if node.is_null:
+ return None
+ if node.type in {
+ RType.LOGICAL,
+ RType.INTEGER,
+ RType.REAL,
+ RType.COMPLEX,
+ RType.RAW,
+ }:
+ atomic_values = _unwrap_atomic(node, object_path=object_path)
+ if name in _MATRIX_PARAMETER_SLOTS:
+ dimensions = get_attribute(node, "dim")
+ if dimensions is not None:
+ raw_shape = _unwrap_atomic(
+ dimensions,
+ object_path=f"{object_path}/dim",
+ )
+ shape_values = raw_shape.read_block(0, len(raw_shape))
+ if (
+ shape_values.ndim != 1
+ or shape_values.size != 2
+ or np.any(shape_values < 0)
+ ):
+ raise _error(
+ "matrix parameter has invalid dimensions",
+ object_path=f"{object_path}/dim",
+ code="invalid_matrix_parameter",
+ )
+ shape = (int(shape_values[0]), int(shape_values[1]))
+ if shape[0] * shape[1] != len(atomic_values):
+ raise _error(
+ "matrix parameter dimensions do not match its values",
+ object_path=object_path,
+ code="invalid_matrix_parameter",
+ )
+ if atomic_values.nbytes > reader._maximumIndexBytes:
+ raise _error(
+ "matrix parameter exceeds its metadata memory budget",
+ object_path=object_path,
+ code="metadata_index_limit",
+ requiredBytes=atomic_values.nbytes,
+ maximumBytes=reader._maximumIndexBytes,
+ )
+ return atomic_values.read_block(0, len(atomic_values)).reshape(
+ shape,
+ order="F",
+ )
+ if name in _SCALAR_MATRIX_SLOTS and len(atomic_values) == 1:
+ return atomic_values[0]
+ return atomic_values
+ if node.type is RType.ALTREP:
+ return _unwrap_atomic(node, object_path=object_path)
+ if node.type is RType.STRING:
+ string_values = _string_values(node, object_path=object_path)
+ if name in _SCALAR_MATRIX_SLOTS and len(string_values) == 1:
+ return _as_text(string_values[0], object_path=object_path)
+ return string_values
+ if node.type is RType.CHAR:
+ if isinstance(node.value, str | bytes):
+ return _as_text(node.value, object_path=object_path)
+ return None
+ if node.type in {RType.BUILTIN, RType.SPECIAL}:
+ function_name = node.value
+ if isinstance(function_name, str) and function_name in _SAFE_DELAYED_PRIMITIVES:
+ return function_name
+ raise _error(
+ "matrix operation contains an unsupported R primitive",
+ object_path=object_path,
+ code="unsupported_matrix_function",
+ functionName=function_name,
+ )
+ if node.type in {
+ RType.CLOSURE,
+ RType.PROMISE,
+ RType.BYTECODE,
+ RType.BYTECODE_DEFINITION,
+ RType.BYTECODE_REFERENCE,
+ }:
+ raise _error(
+ "matrix operation contains executable R semantics",
+ object_path=object_path,
+ code="unsupported_matrix_function",
+ nodeType=node.type.name,
+ )
+ if node.type is RType.VECTOR and isinstance(node.value, tuple):
+ if name in {"Dimnames", "dimnames"}:
+ axes: list[LazyStringVector | None] = []
+ for index, axis in enumerate(node.value):
+ if axis.is_null:
+ axes.append(None)
+ else:
+ axes.append(
+ _string_values(
+ axis,
+ object_path=f"{object_path}/{index}",
+ )
+ )
+ return tuple(axes)
+ if name == "sources":
+ return tuple(
+ reader._matrix_source_from_node(
+ child,
+ object_path=f"{object_path}/{index}",
+ )
+ for index, child in enumerate(node.value)
+ )
+ if name == "fragments_list":
+ return tuple(
+ reader._fragment_source_from_node(
+ child,
+ object_path=f"{object_path}/{index}",
+ )
+ for index, child in enumerate(node.value)
+ )
+ names_node = get_attribute(node, "names")
+ if names_node is not None:
+ return {
+ child_name: _matrix_slot_value(
+ child,
+ name=child_name,
+ object_path=f"{object_path}/{child_name}",
+ reader=reader,
+ )
+ for child_name, child in _named_nodes(node, object_path=object_path)
+ }
+ return tuple(
+ _matrix_slot_value(
+ child,
+ name=name,
+ object_path=f"{object_path}/{index}",
+ reader=reader,
+ )
+ for index, child in enumerate(node.value)
+ )
+ if node.type is RType.S4:
+ if name == "fragments":
+ return reader._fragment_source_from_node(
+ node,
+ object_path=object_path,
+ )
+ return reader._matrix_source_from_node(node, object_path=object_path)
+ raise _error(
+ f"matrix slot uses unsupported R type {node.type.name}",
+ object_path=object_path,
+ code="unsupported_matrix_slot",
+ rType=node.type.name,
+ )
+
+
+def _frame_row_ids(
+ node: RNode,
+ *,
+ document: RdsDocument,
+ object_path: str,
+ expected_length: int,
+ compact_ids: Sequence[str] | None,
+) -> Sequence[str]:
+ row_names = get_attribute(node, "row.names")
+ if row_names is None:
+ raise _error(
+ "data frame row.names are missing",
+ object_path=f"{object_path}/row.names",
+ code="missing_row_names",
+ )
+ if row_names.type is RType.STRING:
+ result = SeuratStringVector(
+ _string_values(
+ row_names,
+ object_path=f"{object_path}/row.names",
+ ),
+ document,
+ object_path=f"{object_path}/row.names",
+ )
+ if len(result) != expected_length:
+ raise _error(
+ f"data frame has {len(result)} row names; expected {expected_length}",
+ object_path=f"{object_path}/row.names",
+ code="length_mismatch",
+ actual=len(result),
+ expected=expected_length,
+ )
+ return result
+ if row_names.type is RType.INTEGER:
+ compact = _read_integer_vector(
+ row_names,
+ object_path=f"{object_path}/row.names",
+ )
+ if (
+ compact_ids is not None
+ and compact.shape == (2,)
+ and int(compact[0]) == R_INT_NA
+ and abs(int(compact[1])) == expected_length
+ ):
+ return compact_ids
+ raise _error(
+ "data frame row.names do not carry explicit identifiers",
+ object_path=f"{object_path}/row.names",
+ code="invalid_row_names",
+ )
+
+
+def _alignment(
+ source_ids: Sequence[str],
+ target_ids: Sequence[str],
+ *,
+ object_path: str,
+ scratch_dir: str | os.PathLike[str] | None,
+ maximum_bytes: int,
+) -> NDArray[np.int64] | None:
+ if _identifiers_equal(source_ids, target_ids):
+ return None
+ if len(source_ids) * np.dtype(np.int64).itemsize > maximum_bytes:
+ raise _error(
+ "identifier alignment exceeds its memory budget",
+ object_path=object_path,
+ code="metadata_index_limit",
+ requiredBytes=len(source_ids) * np.dtype(np.int64).itemsize,
+ maximumBytes=maximum_bytes,
+ )
+ connection, path = _identifier_database(
+ scratch_dir=scratch_dir,
+ maximum_bytes=maximum_bytes,
+ object_path=object_path,
+ )
+ try:
+ _populate_identifier_database(
+ connection,
+ source_ids,
+ object_path=object_path,
+ )
+ connection.execute(
+ "CREATE TABLE target "
+ "(position INTEGER PRIMARY KEY, value TEXT NOT NULL UNIQUE)"
+ )
+ for start in range(0, len(target_ids), _VECTOR_BLOCK_SIZE):
+ stop = min(len(target_ids), start + _VECTOR_BLOCK_SIZE)
+ block = _identifier_block(
+ target_ids,
+ start,
+ stop,
+ object_path=object_path,
+ )
+ connection.executemany(
+ "INSERT INTO target(position, value) VALUES (?, ?)",
+ ((start + offset, value) for offset, value in enumerate(block)),
+ )
+ connection.commit()
+ missing = tuple(
+ str(row[0])
+ for row in connection.execute(
+ "SELECT target.value FROM target "
+ "LEFT JOIN ids ON ids.value = target.value "
+ "WHERE ids.value IS NULL LIMIT 5"
+ )
+ )
+ extra = tuple(
+ str(row[0])
+ for row in connection.execute(
+ "SELECT ids.value FROM ids "
+ "LEFT JOIN target ON target.value = ids.value "
+ "WHERE target.value IS NULL LIMIT 5"
+ )
+ )
+ if missing or extra or len(source_ids) != len(target_ids):
+ raise _error(
+ "metadata row identifiers conflict with the target axis",
+ object_path=object_path,
+ code="metadata_id_conflict",
+ missing=tuple(missing),
+ extra=extra,
+ )
+ result = np.empty(len(target_ids), dtype=np.int64)
+ cursor = connection.execute(
+ "SELECT ids.position FROM target "
+ "JOIN ids ON ids.value = target.value "
+ "ORDER BY target.position"
+ )
+ start = 0
+ while rows := cursor.fetchmany(_VECTOR_BLOCK_SIZE):
+ stop = start + len(rows)
+ result[start:stop] = np.fromiter(
+ (int(row[0]) for row in rows),
+ dtype=np.int64,
+ count=len(rows),
+ )
+ start = stop
+ if start != len(target_ids):
+ raise RuntimeError("identifier alignment row count changed")
+ return result
+ except sqlite3.IntegrityError as error:
+ raise _error(
+ "target identifiers are duplicated",
+ object_path=object_path,
+ code="duplicate_id",
+ ) from error
+ except sqlite3.OperationalError as error:
+ raise _error(
+ "identifier alignment exceeds its disk budget",
+ object_path=object_path,
+ code="metadata_index_limit",
+ ) from error
+ finally:
+ _close_identifier_database(connection, path)
+
+
+def _positions_in_target(
+ source_ids: Sequence[str],
+ target_ids: Sequence[str],
+ *,
+ object_path: str,
+ scratch_dir: str | os.PathLike[str] | None,
+ maximum_bytes: int,
+ missing_message: str = "Assay5 LogMap contains cells absent from global metadata",
+ missing_code: str = "assay_cell_id_conflict",
+) -> NDArray[np.int64]:
+ required_bytes = len(source_ids) * np.dtype(np.int64).itemsize
+ if required_bytes > maximum_bytes:
+ raise _error(
+ "identifier mapping exceeds its memory budget",
+ object_path=object_path,
+ code="metadata_index_limit",
+ requiredBytes=required_bytes,
+ maximumBytes=maximum_bytes,
+ )
+ if _identifiers_equal(source_ids, target_ids):
+ return np.arange(len(source_ids), dtype=np.int64)
+ connection, path = _identifier_database(
+ scratch_dir=scratch_dir,
+ maximum_bytes=maximum_bytes,
+ object_path=object_path,
+ )
+ try:
+ _populate_identifier_database(
+ connection,
+ target_ids,
+ object_path=object_path,
+ )
+ connection.execute(
+ "CREATE TABLE requested "
+ "(position INTEGER PRIMARY KEY, value TEXT NOT NULL UNIQUE)"
+ )
+ for start in range(0, len(source_ids), _VECTOR_BLOCK_SIZE):
+ stop = min(len(source_ids), start + _VECTOR_BLOCK_SIZE)
+ block = _identifier_block(
+ source_ids,
+ start,
+ stop,
+ object_path=object_path,
+ )
+ connection.executemany(
+ "INSERT INTO requested(position, value) VALUES (?, ?)",
+ ((start + offset, value) for offset, value in enumerate(block)),
+ )
+ connection.commit()
+ missing = tuple(
+ str(row[0])
+ for row in connection.execute(
+ "SELECT requested.value FROM requested "
+ "LEFT JOIN ids ON ids.value = requested.value "
+ "WHERE ids.value IS NULL LIMIT 5"
+ )
+ )
+ if missing:
+ raise _error(
+ missing_message,
+ object_path=object_path,
+ code=missing_code,
+ missing=missing,
+ )
+ result = np.empty(len(source_ids), dtype=np.int64)
+ cursor = connection.execute(
+ "SELECT ids.position FROM requested "
+ "JOIN ids ON ids.value = requested.value "
+ "ORDER BY requested.position"
+ )
+ start = 0
+ while rows := cursor.fetchmany(_VECTOR_BLOCK_SIZE):
+ stop = start + len(rows)
+ result[start:stop] = np.fromiter(
+ (int(row[0]) for row in rows),
+ dtype=np.int64,
+ count=len(rows),
+ )
+ start = stop
+ if start != len(source_ids):
+ raise RuntimeError("identifier mapping row count changed")
+ return result
+ except sqlite3.IntegrityError as error:
+ raise _error(
+ "source identifiers are duplicated",
+ object_path=object_path,
+ code="duplicate_id",
+ ) from error
+ except sqlite3.OperationalError as error:
+ raise _error(
+ "identifier mapping exceeds its disk budget",
+ object_path=object_path,
+ code="metadata_index_limit",
+ ) from error
+ finally:
+ _close_identifier_database(connection, path)
+
+
+def _validate_column_values(
+ values: LazyAtomicVector,
+ *,
+ kind: str,
+ levels: tuple[str, ...],
+ object_path: str,
+) -> None:
+ if kind not in {"logical", "factor"}:
+ return
+ for start in range(0, len(values), _VECTOR_BLOCK_SIZE):
+ block = values.read_block(start, min(len(values), start + _VECTOR_BLOCK_SIZE))
+ if kind == "logical":
+ invalid = (block != 0) & (block != 1) & (block != R_INT_NA)
+ else:
+ invalid = (block != R_INT_NA) & ((block < 1) | (block > len(levels)))
+ if np.any(invalid):
+ index = int(np.flatnonzero(invalid)[0])
+ raise _error(
+ f"{kind} column contains an invalid encoded value",
+ object_path=f"{object_path}/{start + index}",
+ code=f"invalid_{kind}_value",
+ value=int(block[index]),
+ )
+
+
+def _metadata_column(
+ node: RNode,
+ *,
+ name: str,
+ length: int,
+ alignment: NDArray[np.int64] | None,
+ document: RdsDocument,
+ object_path: str,
+ maximum_metadata_bytes: int,
+) -> SeuratMetadataColumn:
+ classes = _class_names(node, object_path=object_path)
+ levels: tuple[str, ...] = ()
+ ordered = "ordered" in classes
+ if "factor" in classes:
+ if node.type is not RType.INTEGER or not isinstance(
+ node.value, LazyAtomicVector
+ ):
+ raise _error(
+ "factor column must use integer codes",
+ object_path=object_path,
+ code="invalid_factor",
+ rType=node.type.name,
+ )
+ levels_node = get_attribute(node, "levels")
+ if levels_node is None:
+ raise _error(
+ "factor levels are missing",
+ object_path=f"{object_path}/levels",
+ code="missing_factor_levels",
+ )
+ levels = _read_text_vector(
+ levels_node,
+ object_path=f"{object_path}/levels",
+ maximum_bytes=maximum_metadata_bytes,
+ )
+ _validate_unique(levels, object_path=f"{object_path}/levels")
+ kind = "factor"
+ values: LazyAtomicVector | LazyStringVector = node.value
+ elif node.type is RType.LOGICAL and isinstance(node.value, LazyAtomicVector):
+ kind = "logical"
+ values = node.value
+ elif node.type is RType.INTEGER and isinstance(node.value, LazyAtomicVector):
+ kind = "integer"
+ values = node.value
+ elif node.type is RType.REAL and isinstance(node.value, LazyAtomicVector):
+ kind = "real"
+ values = node.value
+ elif node.type is RType.STRING and isinstance(node.value, LazyStringVector):
+ kind = "character"
+ values = node.value
+ else:
+ raise _error(
+ f"metadata column uses unsupported R type {node.type.name}",
+ object_path=object_path,
+ code="unsupported_metadata_type",
+ rType=node.type.name,
+ classNames=classes,
+ )
+ if len(values) != length:
+ raise _error(
+ f"metadata column has length {len(values)}; expected {length}",
+ object_path=object_path,
+ code="length_mismatch",
+ actual=len(values),
+ expected=length,
+ )
+ if isinstance(values, LazyAtomicVector):
+ _validate_column_values(
+ values,
+ kind=kind,
+ levels=levels,
+ object_path=object_path,
+ )
+ return SeuratMetadataColumn(
+ name=name,
+ kind=kind,
+ values=values,
+ length=length,
+ document=document,
+ source_indices=alignment,
+ levels=levels,
+ ordered=ordered,
+ object_path=object_path,
+ )
+
+
+def _data_frame(
+ node: RNode,
+ *,
+ target_ids: Sequence[str],
+ document: RdsDocument,
+ object_path: str,
+ allow_compact: bool,
+ scratch_dir: str | os.PathLike[str] | None,
+ maximum_index_bytes: int,
+) -> SeuratMetadata:
+ classes = _class_names(node, object_path=object_path)
+ if "data.frame" not in classes:
+ raise _error(
+ "metadata slot is not a data.frame",
+ object_path=object_path,
+ code="invalid_data_frame",
+ classNames=classes,
+ )
+ if node.type is not RType.VECTOR or not isinstance(node.value, tuple):
+ raise _error(
+ "data.frame columns are not stored as a generic vector",
+ object_path=object_path,
+ code="invalid_data_frame",
+ rType=node.type.name,
+ )
+ source_ids = _frame_row_ids(
+ node,
+ document=document,
+ object_path=object_path,
+ expected_length=len(target_ids),
+ compact_ids=target_ids if allow_compact else None,
+ )
+ index = _alignment(
+ source_ids,
+ target_ids,
+ object_path=f"{object_path}/row.names",
+ scratch_dir=scratch_dir,
+ maximum_bytes=maximum_index_bytes,
+ )
+ named_columns = _named_nodes(node, object_path=object_path)
+ columns = tuple(
+ _metadata_column(
+ column,
+ name=name,
+ length=len(target_ids),
+ alignment=index,
+ document=document,
+ object_path=f"{object_path}/{name}",
+ maximum_metadata_bytes=maximum_index_bytes,
+ )
+ for name, column in named_columns
+ )
+ return SeuratMetadata(
+ rowIds=target_ids,
+ columns=columns,
+ objectPath=object_path,
+ )
+
+
+class SeuratReader:
+ def __init__(
+ self,
+ source: Any,
+ *,
+ limits: RdsLimits | None = None,
+ temp_dir: str | os.PathLike[str] | None = None,
+ assays: Sequence[str] | None = None,
+ assay_layers: Mapping[str, Sequence[str]] | None = None,
+ reductions: Sequence[str] | None = None,
+ sidecar_path_remaps: Mapping[str | os.PathLike[str], str | os.PathLike[str]]
+ | None = None,
+ matrix_limits: SourceLimits = DEFAULT_LIMITS,
+ ) -> None:
+ self.source = source
+ self._sidecarPathRemaps = sidecar_path_remaps
+ self._matrixLimits = matrix_limits
+ self._scratchDir = temp_dir
+ self._maximumIndexBytes = matrix_limits.maxMetadataBytes
+ self._requestedAssayLayers = assay_layers
+ self._rdsPath = (
+ os.fspath(source) if isinstance(source, str | os.PathLike) else None
+ )
+ self._document = open_rds(source, limits=limits, temp_dir=temp_dir)
+ self._assayModels: dict[str, SeuratAssay] = {}
+ self._assayErrors: dict[str, SeuratImportError] = {}
+ self._reductionModels: dict[str, SeuratReduction] = {}
+ self._reductionErrors: dict[str, SeuratImportError] = {}
+ try:
+ self._initialize(assays=assays, reductions=reductions)
+ except Exception:
+ self._document.close()
+ raise
+
+ @property
+ def closed(self) -> bool:
+ return self._document.closed
+
+ @property
+ def document(self) -> RdsDocument:
+ self._ensure_open()
+ return self._document
+
+ @property
+ def tempPaths(self) -> tuple[str, ...]:
+ return self._document.temp_paths
+
+ @property
+ def inspection(self) -> SeuratInspectResult:
+ self._ensure_open()
+ return self._inspection
+
+ @property
+ def cellMetadata(self) -> SeuratMetadata:
+ self._ensure_open()
+ return self._cellMetadata
+
+ @property
+ def activeIdentity(self) -> SeuratMetadataColumn:
+ self._ensure_open()
+ if self._activeIdentityError is not None:
+ raise self._activeIdentityError
+ if self._activeIdentity is None:
+ raise AssertionError("active identity status is inconsistent")
+ return self._activeIdentity
+
+ @property
+ def assayNames(self) -> tuple[str, ...]:
+ return self._selectedAssays
+
+ @property
+ def reductionNames(self) -> tuple[str, ...]:
+ return self._selectedReductions
+
+ @property
+ def assays(self) -> tuple[SeuratAssay, ...]:
+ return tuple(self.get_assay(name) for name in self._selectedAssays)
+
+ @property
+ def reductions(self) -> tuple[SeuratReduction, ...]:
+ return tuple(self.get_reduction(name) for name in self._selectedReductions)
+
+ def _ensure_open(self) -> None:
+ if self._document.closed:
+ raise RdsClosedError("RDS document is closed", path="$")
+
+ def close(self) -> None:
+ self._document.close()
+
+ def __enter__(self) -> "SeuratReader":
+ self._ensure_open()
+ return self
+
+ def __exit__(
+ self,
+ exc_type: type[BaseException] | None,
+ exc_value: BaseException | None,
+ traceback: object | None,
+ ) -> None:
+ self.close()
+
+ def inspect(self) -> SeuratInspectResult:
+ return self.inspection
+
+ def get_assay(self, name: str | None = None) -> SeuratAssay:
+ self._ensure_open()
+ selected = self.activeAssay if name is None else name
+ if selected not in self._selectedAssays:
+ raise _error(
+ f"assay {selected!r} was not selected for inspection",
+ object_path=f"assays/{selected}",
+ code="assay_not_selected",
+ assay=selected,
+ )
+ error = self._assayErrors.get(selected)
+ if error is not None:
+ raise error
+ try:
+ return self._assayModels[selected]
+ except KeyError as error:
+ raise AssertionError("assay inspection status is inconsistent") from error
+
+ def get_reduction(self, name: str) -> SeuratReduction:
+ self._ensure_open()
+ if name not in self._selectedReductions:
+ raise _error(
+ f"reduction {name!r} was not selected for inspection",
+ object_path=f"reductions/{name}",
+ code="reduction_not_selected",
+ reduction=name,
+ )
+ error = self._reductionErrors.get(name)
+ if error is not None:
+ raise error
+ try:
+ return self._reductionModels[name]
+ except KeyError as error:
+ raise AssertionError(
+ "reduction inspection status is inconsistent"
+ ) from error
+
+ def _load_save_seurat_cache(self, root: RNode) -> None:
+ self._cachedLayers: dict[str, tuple[_CachedLayerSpec, ...]] = {}
+ self._saveCachePresent = False
+ self._saveCacheError: SeuratImportError | None = None
+ tools = get_slot(root, "tools")
+ if tools is None or not _node_has_content(tools):
+ return
+ try:
+ tool_nodes = _named_nodes(tools, object_path="tools")
+ except SeuratImportError:
+ return
+ cache_node = next(
+ (node for name, node in tool_nodes if name == "SaveSeuratRds"),
+ None,
+ )
+ if cache_node is None:
+ return
+ self._saveCachePresent = True
+ try:
+ classes = _class_names(
+ cache_node,
+ object_path="tools/SaveSeuratRds",
+ )
+ if "data.frame" not in classes:
+ raise _error(
+ "SaveSeuratRds layer cache is not a data.frame",
+ object_path="tools/SaveSeuratRds",
+ code="invalid_sidecar_cache",
+ classNames=classes,
+ )
+ columns = dict(
+ _named_nodes(
+ cache_node,
+ object_path="tools/SaveSeuratRds",
+ )
+ )
+ required = ("layer", "path", "class", "pkg", "fxn", "assay")
+ missing = tuple(name for name in required if name not in columns)
+ if missing:
+ raise _error(
+ "SaveSeuratRds layer cache is missing required columns",
+ object_path="tools/SaveSeuratRds",
+ code="invalid_sidecar_cache",
+ missingColumns=missing,
+ )
+ decoded: dict[str, tuple[str, ...]] = {}
+ remaining_bytes = self._maximumIndexBytes
+ for name in required:
+ values = _read_text_vector(
+ columns[name],
+ object_path=f"tools/SaveSeuratRds/{name}",
+ maximum_bytes=remaining_bytes,
+ )
+ decoded[name] = values
+ remaining_bytes -= sum(
+ len(value.encode("utf-8")) + 8 for value in values
+ )
+ lengths = {len(values) for values in decoded.values()}
+ if len(lengths) != 1:
+ raise _error(
+ "SaveSeuratRds layer cache columns have different lengths",
+ object_path="tools/SaveSeuratRds",
+ code="invalid_sidecar_cache",
+ columnLengths={
+ name: len(values) for name, values in decoded.items()
+ },
+ )
+ cached: dict[str, list[_CachedLayerSpec]] = {}
+ seen: set[tuple[str, str]] = set()
+ for index in range(lengths.pop()):
+ assay = decoded["assay"][index]
+ layer = decoded["layer"][index]
+ key = (assay, layer)
+ if key in seen:
+ raise _error(
+ "SaveSeuratRds layer cache contains a duplicate layer",
+ object_path=f"tools/SaveSeuratRds/{index}",
+ code="invalid_sidecar_cache",
+ assay=assay,
+ layer=layer,
+ )
+ seen.add(key)
+ if assay not in self._assayNodes:
+ continue
+ cached.setdefault(assay, []).append(
+ _CachedLayerSpec(
+ assay=assay,
+ layer=layer,
+ path=decoded["path"][index],
+ sourceClass=decoded["class"][index],
+ package=decoded["pkg"][index],
+ loader=decoded["fxn"][index],
+ objectPath=f"tools/SaveSeuratRds/{index}",
+ )
+ )
+ self._cachedLayers = {
+ assay: tuple(values) for assay, values in cached.items()
+ }
+ except SeuratImportError as error:
+ self._saveCacheError = error
+
+ def _cached_layer_source(self, cached: _CachedLayerSpec) -> MatrixSource:
+ loader = " ".join(cached.loader.split())
+ composite = re.fullmatch(
+ r"function\(x\) \{ paths <- unlist\(x = strsplit"
+ r"\(x = x, split = ','\)\); fxns <- list\(\s*(?P.*)\s*\); "
+ r"mats <- vector\(mode = 'list', length = length\(x = paths\)\); "
+ r"for \(i in seq_along\(paths\)\) \{ fn <- "
+ r"eval\(str2lang\(fxns\[\[i\]\]\)\); mats\[\[i\]\] <- "
+ r"fn\(paths\[i\]\); \}; return\(Reduce\(cbind, mats\)\); \}",
+ loader,
+ )
+ if composite is not None:
+ if cached.package != "BPCells":
+ raise _error(
+ "SaveSeuratRds composite loader has an unexpected package",
+ object_path=f"{cached.objectPath}/pkg",
+ code="unsupported_sidecar_cache_recipe",
+ package=cached.package,
+ )
+ loaders = _parse_cache_loader_list(
+ composite.group("loaders"),
+ object_path=f"{cached.objectPath}/fxn",
+ )
+ paths = tuple(path.strip() for path in cached.path.split(","))
+ if len(paths) != len(loaders) or any(not path for path in paths):
+ raise _error(
+ "SaveSeuratRds composite loader paths do not match its recipes",
+ object_path=f"{cached.objectPath}/path",
+ code="invalid_sidecar_cache",
+ pathCount=len(paths),
+ loaderCount=len(loaders),
+ )
+ sources = tuple(
+ self._cached_layer_source(
+ _CachedLayerSpec(
+ assay=cached.assay,
+ layer=cached.layer,
+ path=path,
+ sourceClass=cached.sourceClass,
+ package=cached.package,
+ loader=nested_loader,
+ objectPath=f"{cached.objectPath}/fxn/{index}",
+ )
+ )
+ for index, (path, nested_loader) in enumerate(zip(paths, loaders))
+ )
+ try:
+ return CellBindMatrixSource(sources, limits=self._matrixLimits)
+ except Exception as cause:
+ raise self._wrapped_item_error(
+ cause,
+ object_path=f"assays/{cached.assay}/layers/{cached.layer}",
+ code="invalid_matrix",
+ ) from cause
+ specification: dict[str, Any]
+ if cached.package == "BPCells" and re.fullmatch(
+ r"function\(x\) BPCells::open_matrix_dir\(dir = x\)",
+ loader,
+ ):
+ specification = {"class": "MatrixDir", "dir": cached.path}
+ elif cached.package == "BPCells" and (
+ match := re.fullmatch(
+ r"function\(x\) BPCells::open_matrix_hdf5"
+ r"\(path = x, group = (?P['\"])(?P[^'\"]+)"
+ r"(?P=quote)\)",
+ loader,
+ )
+ ):
+ specification = {
+ "class": "MatrixH5",
+ "filepath": cached.path,
+ "group": match.group("group"),
+ }
+ elif cached.package == "BPCells" and (
+ match := re.fullmatch(
+ r"function\(x\) BPCells::open_matrix_anndata_hdf5"
+ r"\(path = x, group = (?P['\"])(?P[^'\"]+)"
+ r"(?P=quote)\)",
+ loader,
+ )
+ ):
+ specification = {
+ "class": "AnnDataMatrixH5",
+ "filepath": cached.path,
+ "group": match.group("group"),
+ }
+ elif cached.package == "HDF5Array" and (
+ match := re.fullmatch(
+ r"function\(x\) HDF5Array::HDF5Array"
+ r"\(filepath = x, name = (?P['\"])(?P[^'\"]+)"
+ r"(?P=quote), as\.sparse = (?PTRUE|FALSE)\)",
+ loader,
+ )
+ ):
+ specification = {
+ "class": "HDF5ArraySeed",
+ "filepath": cached.path,
+ "name": match.group("name"),
+ "asSparse": match.group("sparse") == "TRUE",
+ }
+ elif cached.package == "HDF5Array" and (
+ match := re.fullmatch(
+ r"function\(x\) HDF5Array::H5ADMatrix"
+ r"\(filepath = x(?:, layer = (?P['\"])"
+ r"(?P[^'\"]+)(?P=quote))?\s*\)",
+ loader,
+ )
+ ):
+ specification = {
+ "class": "H5ADMatrixSeed",
+ "filepath": cached.path,
+ }
+ if match.group("layer") is not None:
+ specification["layer"] = match.group("layer")
+ else:
+ raise _error(
+ "SaveSeuratRds loader recipe is not in the safe built-in profile",
+ object_path=f"{cached.objectPath}/fxn",
+ code="unsupported_sidecar_cache_recipe",
+ assay=cached.assay,
+ layer=cached.layer,
+ sourceClass=cached.sourceClass,
+ package=cached.package,
+ )
+ try:
+ return matrix_source_from_slots(
+ specification,
+ object_path=f"assays/{cached.assay}/layers/{cached.layer}",
+ rds_path=self._rdsPath,
+ absolute_prefix_remaps=self._sidecarPathRemaps,
+ limits=self._matrixLimits,
+ )
+ except Exception as cause:
+ raise self._wrapped_item_error(
+ cause,
+ object_path=f"assays/{cached.assay}/layers/{cached.layer}",
+ code=(
+ "unsupported_matrix"
+ if isinstance(cause, UnsupportedMatrixOperation)
+ else "invalid_matrix"
+ ),
+ ) from cause
+
+ def _initialize(
+ self,
+ *,
+ assays: Sequence[str] | None,
+ reductions: Sequence[str] | None,
+ ) -> None:
+ root = self._document.root
+ root_classes = _class_names(root, object_path="$")
+ if "Seurat" not in root_classes:
+ raise _error(
+ "RDS root is not a Seurat object",
+ object_path="$",
+ code="not_seurat",
+ classNames=root_classes,
+ )
+ assay_list = _require_slot(root, "assays", object_path="")
+ reduction_list = _require_slot(root, "reductions", object_path="")
+ self._assayNodes = dict(_named_nodes(assay_list, object_path="assays"))
+ self._reductionNodes = dict(
+ _named_nodes(reduction_list, object_path="reductions")
+ )
+ self._load_save_seurat_cache(root)
+ active_node = _require_slot(root, "active.assay", object_path="")
+ self.activeAssay = _read_text_scalar(
+ active_node,
+ object_path="active.assay",
+ )
+ if self.activeAssay not in self._assayNodes:
+ raise _error(
+ f"active assay {self.activeAssay!r} does not exist",
+ object_path="active.assay",
+ code="active_assay_missing",
+ activeAssay=self.activeAssay,
+ availableAssays=tuple(self._assayNodes),
+ )
+
+ metadata_node = _require_slot(root, "meta.data", object_path="")
+ if metadata_node.type is not RType.VECTOR or not isinstance(
+ metadata_node.value, tuple
+ ):
+ raise _error(
+ "cell meta.data is not a data.frame vector",
+ object_path="meta.data",
+ code="invalid_data_frame",
+ )
+ self.cellIds = _frame_row_ids(
+ metadata_node,
+ document=self._document,
+ object_path="meta.data",
+ expected_length=len(metadata_node.value[0].value)
+ if metadata_node.value
+ and isinstance(
+ metadata_node.value[0].value,
+ LazyAtomicVector | LazyStringVector,
+ )
+ else self._frame_length(metadata_node, object_path="meta.data"),
+ compact_ids=None,
+ )
+ _validate_unique_ids(
+ self.cellIds,
+ object_path="meta.data/row.names",
+ scratch_dir=self._scratchDir,
+ maximum_bytes=self._maximumIndexBytes,
+ )
+ self._cellMetadata = _data_frame(
+ metadata_node,
+ target_ids=self.cellIds,
+ document=self._document,
+ object_path="meta.data",
+ allow_compact=False,
+ scratch_dir=self._scratchDir,
+ maximum_index_bytes=self._maximumIndexBytes,
+ )
+
+ self._activeIdentity: SeuratMetadataColumn | None = None
+ self._activeIdentityError: SeuratImportError | None = None
+ try:
+ self._activeIdentity = self._build_active_identity(root)
+ except SeuratImportError as error:
+ self._activeIdentityError = error
+
+ self._selectedAssays = self._selected_names(
+ assays,
+ available=tuple(self._assayNodes),
+ kind="assay",
+ )
+ self._assayLayers = self._layer_overrides(
+ self._requestedAssayLayers,
+ available=tuple(self._assayNodes),
+ )
+ self._selectedReductions = self._selected_names(
+ reductions,
+ available=tuple(self._reductionNodes),
+ kind="reduction",
+ )
+ assay_inspections = tuple(
+ self._inspect_assay(name) for name in self._selectedAssays
+ )
+ reduction_inspections = tuple(
+ self._inspect_reduction(name) for name in self._selectedReductions
+ )
+ root_notices = self._root_notices(root)
+ metadata_inspection = SeuratItemInspection(
+ name="meta.data",
+ importable=True,
+ sourceClass="data.frame",
+ dimensions=(len(self.cellIds), len(self._cellMetadata.columns)),
+ objectPath="meta.data",
+ dtype="mixed",
+ backend=type(self._cellMetadata).__name__,
+ )
+ active_inspection = (
+ SeuratItemInspection(
+ name="active.ident",
+ importable=False,
+ sourceClass=None,
+ dimensions=None,
+ objectPath="active.ident",
+ blockingDiagnostic=SeuratDiagnostic.from_error(
+ self._activeIdentityError
+ ),
+ )
+ if self._activeIdentityError is not None
+ else SeuratItemInspection(
+ name="active.ident",
+ importable=True,
+ sourceClass="factor",
+ dimensions=(len(self.cellIds),),
+ objectPath="active.ident",
+ dtype="factor",
+ backend=type(self._activeIdentity).__name__,
+ )
+ )
+ self._inspection = SeuratInspectResult(
+ source=self._document.source.name,
+ sourceDigest=self._document.source.source_sha256,
+ payloadDigest=self._document.source.payload_sha256,
+ compression=self._document.source.compression.value,
+ activeAssay=self.activeAssay,
+ nCells=len(self.cellIds),
+ assays=assay_inspections,
+ reductions=reduction_inspections,
+ cellMetadata=metadata_inspection,
+ activeIdentity=active_inspection,
+ notices=root_notices,
+ )
+
+ @staticmethod
+ def _frame_length(node: RNode, *, object_path: str) -> int:
+ if node.type is not RType.VECTOR or not isinstance(node.value, tuple):
+ raise _error(
+ "data.frame columns are not stored as a vector",
+ object_path=object_path,
+ code="invalid_data_frame",
+ )
+ if not node.value:
+ row_names = get_attribute(node, "row.names")
+ if row_names is None:
+ return 0
+ if row_names.type is RType.STRING and isinstance(
+ row_names.value, LazyStringVector
+ ):
+ return len(row_names.value)
+ compact = _read_integer_vector(
+ row_names,
+ object_path=f"{object_path}/row.names",
+ )
+ if (
+ compact.shape == (2,)
+ and int(compact[0]) == R_INT_NA
+ and int(compact[1]) <= 0
+ ):
+ return -int(compact[1])
+ raise _error(
+ "cannot determine data.frame row count",
+ object_path=object_path,
+ code="invalid_data_frame",
+ )
+ first = node.value[0].value
+ if isinstance(first, LazyAtomicVector | LazyStringVector):
+ return len(first)
+ raise _error(
+ "first data.frame column is not block-readable",
+ object_path=f"{object_path}/0",
+ code="unsupported_metadata_type",
+ )
+
+ @staticmethod
+ def _selected_names(
+ requested: Sequence[str] | None,
+ *,
+ available: tuple[str, ...],
+ kind: str,
+ ) -> tuple[str, ...]:
+ if requested is None:
+ return available
+ if isinstance(requested, str | bytes):
+ raise TypeError(f"{kind} selection must be a sequence of names")
+ result = tuple(str(value) for value in requested)
+ if len(set(result)) != len(result):
+ raise ValueError(f"{kind} selection contains duplicate names")
+ return result
+
+ @staticmethod
+ def _layer_overrides(
+ requested: Mapping[str, Sequence[str]] | None,
+ *,
+ available: tuple[str, ...],
+ ) -> dict[str, tuple[str, ...]]:
+ if requested is None:
+ return {}
+ if not isinstance(requested, Mapping):
+ raise TypeError("assay_layers must map assay names to layer-name sequences")
+ result: dict[str, tuple[str, ...]] = {}
+ for raw_assay, raw_layers in requested.items():
+ assay = str(raw_assay)
+ if assay not in available:
+ raise ValueError(
+ f"assay layer override references unknown assay {assay!r}"
+ )
+ if isinstance(raw_layers, str | bytes):
+ raise TypeError(
+ f"layer override for assay {assay!r} must be a sequence"
+ )
+ layers = tuple(str(value) for value in raw_layers)
+ if not layers:
+ raise ValueError(
+ f"layer override for assay {assay!r} must not be empty"
+ )
+ if len(set(layers)) != len(layers):
+ raise ValueError(
+ f"layer override for assay {assay!r} contains duplicates"
+ )
+ result[assay] = layers
+ return result
+
+ def _build_active_identity(self, root: RNode) -> SeuratMetadataColumn:
+ if "active.ident" in self._cellMetadata.columnNames:
+ raise _error(
+ "active.ident conflicts with a cell metadata column of the same name",
+ object_path="active.ident",
+ code="active_identity_conflict",
+ column="active.ident",
+ )
+ node = _require_slot(root, "active.ident", object_path="")
+ classes = _class_names(node, object_path="active.ident")
+ if "factor" not in classes:
+ raise _error(
+ "active.ident must be a factor",
+ object_path="active.ident",
+ code="invalid_active_identity",
+ classNames=classes,
+ )
+ names_node = get_attribute(node, "names")
+ if names_node is None:
+ raise _error(
+ "active.ident names are missing",
+ object_path="active.ident/names",
+ code="missing_active_identity_names",
+ )
+ source_ids = SeuratStringVector(
+ _string_values(
+ names_node,
+ object_path="active.ident/names",
+ ),
+ self._document,
+ object_path="active.ident/names",
+ )
+ index = _alignment(
+ source_ids,
+ self.cellIds,
+ object_path="active.ident/names",
+ scratch_dir=self._scratchDir,
+ maximum_bytes=self._maximumIndexBytes,
+ )
+ return _metadata_column(
+ node,
+ name="active.ident",
+ length=len(self.cellIds),
+ alignment=index,
+ document=self._document,
+ object_path="active.ident",
+ maximum_metadata_bytes=self._maximumIndexBytes,
+ )
+
+ def _root_notices(self, root: RNode) -> tuple[SeuratNotice, ...]:
+ notices: list[SeuratNotice] = []
+ for name, node in iter_attributes(root):
+ if name in _ROOT_IMPORTED_SLOTS or not _node_has_content(node):
+ continue
+ if name == "tools" and self._saveCachePresent:
+ notices.append(
+ SeuratNotice(
+ code="used_save_seurat_rds_cache",
+ message="recognized SaveSeuratRds sidecar cache was inspected",
+ objectPath="tools/SaveSeuratRds",
+ context={
+ "cachedAssays": tuple(self._cachedLayers),
+ "valid": self._saveCacheError is None,
+ },
+ )
+ )
+ continue
+ if name in _ROOT_IGNORED_SLOTS or name not in {
+ "project.name",
+ "version",
+ }:
+ notices.append(
+ SeuratNotice(
+ code="ignored_seurat_slot",
+ message=f"Seurat slot {name!r} is not imported",
+ objectPath=name,
+ context={"rType": node.type.name},
+ )
+ )
+ return tuple(notices)
+
+ def _inspect_assay(self, name: str) -> SeuratItemInspection:
+ object_path = f"assays/{name}"
+ node = self._assayNodes.get(name)
+ if node is None:
+ error = _error(
+ f"assay {name!r} does not exist",
+ object_path=object_path,
+ code="assay_not_found",
+ availableAssays=tuple(self._assayNodes),
+ )
+ self._assayErrors[name] = error
+ return SeuratItemInspection(
+ name=name,
+ importable=False,
+ sourceClass=None,
+ dimensions=None,
+ objectPath=object_path,
+ blockingDiagnostic=SeuratDiagnostic.from_error(error),
+ )
+ classes: tuple[str, ...] = ()
+ blocked_error: SeuratImportError
+ try:
+ classes = _class_names(node, object_path=object_path)
+ if "Assay5" in classes:
+ assay = self._build_assay5(name, node)
+ elif "Assay" in classes:
+ assay = self._build_legacy_assay(name, node)
+ else:
+ raise _error(
+ "assay does not expose legacy Assay or Assay5 capabilities",
+ object_path=object_path,
+ code="unsupported_assay_class",
+ classNames=classes,
+ )
+ self._assayModels[name] = assay
+ source = getattr(assay.counts, "_source", assay.counts)
+ estimate = assay.counts.estimate_read_memory(
+ 0,
+ min(1, assay.dimensions[1]),
+ )
+ return SeuratItemInspection(
+ name=name,
+ importable=True,
+ sourceClass=assay.sourceClass,
+ dimensions=assay.dimensions,
+ objectPath=object_path,
+ dtype=assay.counts.dtype.str,
+ backend=type(source).__name__,
+ memoryEstimate=estimate,
+ notices=assay.notices,
+ )
+ except SeuratImportError as error:
+ blocked_error = error
+ self._assayErrors[name] = blocked_error
+ except Exception as cause:
+ blocked_error = self._wrapped_item_error(
+ cause,
+ object_path=object_path,
+ code=(
+ "unsupported_matrix"
+ if isinstance(cause, UnsupportedMatrixOperation)
+ else "invalid_assay"
+ ),
+ )
+ self._assayErrors[name] = blocked_error
+ return SeuratItemInspection(
+ name=name,
+ importable=False,
+ sourceClass=classes[0] if classes else None,
+ dimensions=None,
+ objectPath=object_path,
+ blockingDiagnostic=SeuratDiagnostic.from_error(blocked_error),
+ )
+
+ def _inspect_reduction(self, name: str) -> SeuratItemInspection:
+ object_path = f"reductions/{name}"
+ node = self._reductionNodes.get(name)
+ if node is None:
+ error = _error(
+ f"reduction {name!r} does not exist",
+ object_path=object_path,
+ code="reduction_not_found",
+ availableReductions=tuple(self._reductionNodes),
+ )
+ self._reductionErrors[name] = error
+ return SeuratItemInspection(
+ name=name,
+ importable=False,
+ sourceClass=None,
+ dimensions=None,
+ objectPath=object_path,
+ blockingDiagnostic=SeuratDiagnostic.from_error(error),
+ )
+ classes: tuple[str, ...] = ()
+ blocked_error: SeuratImportError
+ try:
+ classes = _class_names(node, object_path=object_path)
+ if "DimReduc" not in classes:
+ raise _error(
+ "reduction does not expose DimReduc capabilities",
+ object_path=object_path,
+ code="unsupported_reduction_class",
+ classNames=classes,
+ )
+ reduction = self._build_reduction(name, node)
+ self._reductionModels[name] = reduction
+ output_bytes = (
+ min(1, reduction.dimensions[0])
+ * reduction.dimensions[1]
+ * reduction.cellEmbeddings.dtype.itemsize
+ )
+ return SeuratItemInspection(
+ name=name,
+ importable=True,
+ sourceClass=reduction.sourceClass,
+ dimensions=reduction.dimensions,
+ objectPath=object_path,
+ dtype=reduction.cellEmbeddings.dtype.str,
+ backend=type(reduction.cellEmbeddings).__name__,
+ memoryEstimate=MemoryEstimate(
+ workingBytes=output_bytes,
+ outputBytes=output_bytes,
+ ),
+ notices=reduction.notices,
+ )
+ except SeuratImportError as error:
+ blocked_error = error
+ self._reductionErrors[name] = blocked_error
+ except Exception as cause:
+ blocked_error = self._wrapped_item_error(
+ cause,
+ object_path=object_path,
+ code="invalid_reduction",
+ )
+ self._reductionErrors[name] = blocked_error
+ return SeuratItemInspection(
+ name=name,
+ importable=False,
+ sourceClass=classes[0] if classes else None,
+ dimensions=None,
+ objectPath=object_path,
+ blockingDiagnostic=SeuratDiagnostic.from_error(blocked_error),
+ )
+
+ @staticmethod
+ def _wrapped_item_error(
+ cause: Exception,
+ *,
+ object_path: str,
+ code: str,
+ ) -> SeuratImportError:
+ source_path = getattr(cause, "objectPath", object_path)
+ return _error(
+ str(cause),
+ object_path=str(source_path),
+ code=code,
+ causeType=type(cause).__name__,
+ )
+
+ def _fragment_source_from_node(
+ self,
+ node: RNode,
+ *,
+ object_path: str,
+ ) -> FragmentSource:
+ classes = _class_names(node, object_path=object_path)
+ if node.type is not RType.S4:
+ raise _error(
+ f"fragment source uses unsupported R type {node.type.name}",
+ object_path=object_path,
+ code="unsupported_fragment_structure",
+ rType=node.type.name,
+ classNames=classes,
+ )
+ slots: dict[str, Any] = {}
+ for name, value in iter_attributes(node):
+ if name == "class":
+ continue
+ slots[name] = _matrix_slot_value(
+ value,
+ name=name,
+ object_path=f"{object_path}/{name}",
+ reader=self,
+ )
+ try:
+ return fragment_source_from_slots(
+ {"class": classes, "slots": slots},
+ object_path=object_path,
+ rds_path=self._rdsPath,
+ absolute_prefix_remaps=self._sidecarPathRemaps,
+ limits=self._matrixLimits,
+ )
+ except Exception as cause:
+ raise self._wrapped_item_error(
+ cause,
+ object_path=object_path,
+ code=(
+ "unsupported_matrix"
+ if isinstance(cause, UnsupportedMatrixOperation)
+ else "invalid_matrix"
+ ),
+ ) from cause
+
+ def _matrix_source_from_node(
+ self,
+ node: RNode,
+ *,
+ object_path: str,
+ ) -> MatrixSource:
+ classes = _class_names(node, object_path=object_path)
+ if node.type in {
+ RType.LOGICAL,
+ RType.INTEGER,
+ RType.REAL,
+ RType.COMPLEX,
+ RType.RAW,
+ RType.ALTREP,
+ }:
+ values = _unwrap_atomic(node, object_path=object_path)
+ shape = _matrix_dimensions(node, object_path=object_path)
+ specification: dict[str, Any] = {
+ "class": classes or ("matrix", "array"),
+ ".Data": values,
+ "dim": shape,
+ "Dimnames": (None, None),
+ }
+ elif node.type is RType.S4:
+ slots: dict[str, Any] = {}
+ for name, value in iter_attributes(node):
+ if name in {"class", "Dimnames", "dimnames"}:
+ continue
+ slots[name] = _matrix_slot_value(
+ value,
+ name=name,
+ object_path=f"{object_path}/{name}",
+ reader=self,
+ )
+ if classes and classes[0] in {
+ "dgeMatrix",
+ "lgeMatrix",
+ "ngeMatrix",
+ "igeMatrix",
+ "denseMatrix",
+ }:
+ if "dim" not in slots and "Dim" in slots:
+ slots["dim"] = slots["Dim"]
+ specification = {"class": classes, "slots": slots}
+ else:
+ raise _error(
+ f"matrix uses unsupported R type {node.type.name}",
+ object_path=object_path,
+ code="unsupported_matrix_structure",
+ rType=node.type.name,
+ classNames=classes,
+ )
+ try:
+ return matrix_source_from_slots(
+ specification,
+ object_path=object_path,
+ rds_path=self._rdsPath,
+ absolute_prefix_remaps=self._sidecarPathRemaps,
+ limits=self._matrixLimits,
+ )
+ except SeuratImportError:
+ raise
+ except Exception as cause:
+ raise self._wrapped_item_error(
+ cause,
+ object_path=object_path,
+ code=(
+ "unsupported_matrix"
+ if isinstance(cause, UnsupportedMatrixOperation)
+ else "invalid_matrix"
+ ),
+ ) from cause
+
+ def _build_legacy_assay(self, name: str, node: RNode) -> SeuratAssay:
+ object_path = f"assays/{name}"
+ override = self._assayLayers.get(name)
+ if override is not None and override != ("counts",):
+ raise _error(
+ "legacy Assay layer override must select only 'counts'",
+ object_path=f"{object_path}/counts",
+ code="invalid_layer_override",
+ selectedLayers=override,
+ )
+ counts_node = _require_slot(node, "counts", object_path=object_path)
+ source = self._matrix_source_from_node(
+ counts_node,
+ object_path=f"{object_path}/counts",
+ )
+ feature_ids, cell_ids = _matrix_dimnames(
+ counts_node,
+ document=self._document,
+ object_path=f"{object_path}/counts",
+ require_both=True,
+ )
+ assert feature_ids is not None
+ assert cell_ids is not None
+ _validate_unique_ids(
+ feature_ids,
+ object_path=f"{object_path}/counts/Dimnames/0",
+ scratch_dir=self._scratchDir,
+ maximum_bytes=self._maximumIndexBytes,
+ )
+ source_row_conflict = source.row_names is not None and not _identifiers_equal(
+ source.row_names,
+ feature_ids,
+ )
+ source_column_conflict = (
+ source.column_names is not None
+ and not _identifiers_equal(source.column_names, cell_ids)
+ )
+ if source_row_conflict or source_column_conflict:
+ raise _error(
+ "matrix source identifiers conflict with serialized Dimnames",
+ object_path=f"{object_path}/counts/Dimnames",
+ code="matrix_id_conflict",
+ )
+ if not _identifiers_equal(cell_ids, self.cellIds):
+ raise _error(
+ "legacy assay cell identifiers conflict with global cell order",
+ object_path=f"{object_path}/counts/Dimnames/1",
+ code="assay_cell_id_conflict",
+ )
+ metadata_node = _require_slot(
+ node,
+ "meta.features",
+ object_path=object_path,
+ )
+ feature_metadata = _data_frame(
+ metadata_node,
+ target_ids=feature_ids,
+ document=self._document,
+ object_path=f"{object_path}/meta.features",
+ allow_compact=True,
+ scratch_dir=self._scratchDir,
+ maximum_index_bytes=self._maximumIndexBytes,
+ )
+ notices: list[SeuratNotice] = []
+ for slot_name, slot in iter_attributes(node):
+ if slot_name in {"data", "scale.data"} and _node_has_content(slot):
+ notices.append(
+ SeuratNotice(
+ code="ignored_normalized_layer",
+ message=f"normalized slot {slot_name!r} is not imported",
+ objectPath=f"{object_path}/{slot_name}",
+ context={},
+ )
+ )
+ elif slot_name not in {
+ "class",
+ "counts",
+ "meta.features",
+ } and _node_has_content(slot):
+ notices.append(
+ SeuratNotice(
+ code="ignored_assay_slot",
+ message=f"assay slot {slot_name!r} is not imported",
+ objectPath=f"{object_path}/{slot_name}",
+ context={"rType": slot.type.name},
+ )
+ )
+ owned = _OwnedMatrixSource(
+ source,
+ self._document,
+ object_path=f"{object_path}/counts",
+ )
+ return SeuratAssay(
+ name=name,
+ sourceClass=_class_names(node, object_path=object_path)[0],
+ counts=owned,
+ featureIds=feature_ids,
+ cellIds=self.cellIds,
+ assayCellIds=cell_ids,
+ cellMembership=SeuratMembership(len(self.cellIds)),
+ featureMetadata=feature_metadata,
+ notices=tuple(notices),
+ objectPath=object_path,
+ )
+
+ def _build_logmap(self, node: RNode, *, object_path: str) -> _LogMap:
+ classes = _class_names(node, object_path=object_path)
+ if "LogMap" not in classes:
+ raise _error(
+ "membership slot is not a LogMap",
+ object_path=object_path,
+ code="invalid_logmap",
+ classNames=classes,
+ )
+ if node.type is not RType.LOGICAL:
+ raise _error(
+ "LogMap values must be logical",
+ object_path=object_path,
+ code="invalid_logmap",
+ rType=node.type.name,
+ )
+ values = _unwrap_atomic(node, object_path=object_path)
+ shape = _matrix_dimensions(node, object_path=object_path)
+ row_ids, layer_names = _matrix_dimnames(
+ node,
+ document=self._document,
+ object_path=object_path,
+ require_both=True,
+ )
+ assert row_ids is not None
+ assert layer_names is not None
+ _validate_unique_ids(
+ row_ids,
+ object_path=f"{object_path}/Dimnames/0",
+ scratch_dir=self._scratchDir,
+ maximum_bytes=self._maximumIndexBytes,
+ )
+ _validate_unique_ids(
+ layer_names,
+ object_path=f"{object_path}/Dimnames/1",
+ scratch_dir=self._scratchDir,
+ maximum_bytes=self._maximumIndexBytes,
+ )
+ if shape != (len(row_ids), len(layer_names)):
+ raise _error(
+ "LogMap dimensions conflict with Dimnames",
+ object_path=f"{object_path}/dim",
+ code="logmap_dimension_mismatch",
+ dimensions=shape,
+ rowIds=len(row_ids),
+ layerNames=len(layer_names),
+ )
+ if len(values) != shape[0] * shape[1]:
+ raise _error(
+ "LogMap value count conflicts with dimensions",
+ object_path=object_path,
+ code="logmap_dimension_mismatch",
+ values=len(values),
+ dimensions=shape,
+ )
+ result = _LogMap(
+ rowIds=row_ids,
+ layerNames=layer_names,
+ values=values,
+ objectPath=object_path,
+ maximumIndexBytes=self._maximumIndexBytes,
+ )
+ return result
+
+ def _cached_axis_positions(
+ self,
+ source_ids: Sequence[str] | None,
+ target_ids: Sequence[str],
+ source_size: int,
+ *,
+ object_path: str,
+ ) -> NDArray[np.int64]:
+ if source_ids is None:
+ if source_size != len(target_ids):
+ raise _error(
+ "SaveSeuratRds removed the layer membership map and the "
+ "sidecar has no identifiers to reconstruct it",
+ object_path=object_path,
+ code="irrecoverable_sidecar_cache",
+ sourceSize=source_size,
+ targetSize=len(target_ids),
+ )
+ return np.arange(source_size, dtype=np.int64)
+ if len(source_ids) != source_size:
+ raise _error(
+ "sidecar identifier length conflicts with its matrix dimension",
+ object_path=object_path,
+ code="sidecar_dimension_mismatch",
+ sourceSize=source_size,
+ identifierCount=len(source_ids),
+ )
+ return _positions_in_target(
+ source_ids,
+ target_ids,
+ object_path=object_path,
+ scratch_dir=self._scratchDir,
+ maximum_bytes=self._maximumIndexBytes,
+ )
+
+ def _build_assay5(self, name: str, node: RNode) -> SeuratAssay:
+ object_path = f"assays/{name}"
+ layers_node = _require_slot(node, "layers", object_path=object_path)
+ layer_nodes = _named_nodes(
+ layers_node,
+ object_path=f"{object_path}/layers",
+ )
+ cached_layers = self._cachedLayers.get(name, ())
+ in_object_names = {layer_name for layer_name, _ in layer_nodes}
+ duplicates = tuple(
+ cached.layer for cached in cached_layers if cached.layer in in_object_names
+ )
+ if duplicates:
+ raise _error(
+ "SaveSeuratRds cache duplicates an in-object layer",
+ object_path="tools/SaveSeuratRds",
+ code="invalid_sidecar_cache",
+ assay=name,
+ duplicateLayers=duplicates,
+ )
+ layer_entries: tuple[
+ tuple[str, RNode | _CachedLayerSpec],
+ ...,
+ ] = (
+ *layer_nodes,
+ *((cached.layer, cached) for cached in cached_layers),
+ )
+ cells = self._build_logmap(
+ _require_slot(node, "cells", object_path=object_path),
+ object_path=f"{object_path}/cells",
+ )
+ features = self._build_logmap(
+ _require_slot(node, "features", object_path=object_path),
+ object_path=f"{object_path}/features",
+ )
+ global_positions = _positions_in_target(
+ cells.rowIds,
+ self.cellIds,
+ object_path=f"{object_path}/cells/Dimnames/0",
+ scratch_dir=self._scratchDir,
+ maximum_bytes=self._maximumIndexBytes,
+ )
+ if global_positions.size > 1 and np.any(
+ global_positions[1:] <= global_positions[:-1]
+ ):
+ raise _error(
+ "Assay5 cell identifiers do not follow global cell order",
+ object_path=f"{object_path}/cells/Dimnames/0",
+ code="assay_cell_order_conflict",
+ )
+ resident_index_bytes = int(global_positions.nbytes) + len(self.cellIds)
+ available_count_layers = tuple(
+ (layer_name, layer_value)
+ for layer_name, layer_value in layer_entries
+ if layer_name == "counts"
+ or (layer_name.startswith("counts.") and len(layer_name) > len("counts."))
+ )
+ if not available_count_layers and self._saveCacheError is not None:
+ raise self._saveCacheError
+ override = self._assayLayers.get(name)
+ if override is None:
+ count_layers = available_count_layers
+ else:
+ raw_count_names = {layer_name for layer_name, _ in available_count_layers}
+ invalid = tuple(
+ layer_name
+ for layer_name in override
+ if layer_name not in raw_count_names
+ )
+ if invalid:
+ raise _error(
+ "Assay5 layer override contains a non-count or missing layer",
+ object_path=f"{object_path}/layers",
+ code="invalid_layer_override",
+ invalidLayers=invalid,
+ availableCountLayers=tuple(
+ layer_name for layer_name, _ in available_count_layers
+ ),
+ )
+ selected = set(override)
+ count_layers = tuple(
+ (layer_name, layer_node)
+ for layer_name, layer_node in available_count_layers
+ if layer_name in selected
+ )
+ if not count_layers:
+ raise _error(
+ "Assay5 has no raw count-bearing layer",
+ object_path=f"{object_path}/layers",
+ code="counts_layer_missing",
+ )
+ for layer_name, layer_value in count_layers:
+ if (
+ not isinstance(layer_value, _CachedLayerSpec)
+ and layer_name not in cells.layerNames
+ ):
+ raise _error(
+ f"cell LogMap has no column for layer {layer_name!r}",
+ object_path=f"{object_path}/cells",
+ code="logmap_layer_missing",
+ layer=layer_name,
+ )
+ if (
+ not isinstance(layer_value, _CachedLayerSpec)
+ and layer_name not in features.layerNames
+ ):
+ raise _error(
+ f"feature LogMap has no column for layer {layer_name!r}",
+ object_path=f"{object_path}/features",
+ code="logmap_layer_missing",
+ layer=layer_name,
+ )
+ placements: list[LayerPlacement] = []
+ for layer_name, layer_value in count_layers:
+ layer_path = f"{object_path}/layers/{layer_name}"
+ if isinstance(layer_value, _CachedLayerSpec):
+ source = self._cached_layer_source(layer_value)
+ feature_indices = self._cached_axis_positions(
+ source.row_names,
+ features.rowIds,
+ source.shape[0],
+ object_path=f"{layer_path}/Dimnames/0",
+ )
+ assay_cell_indices = self._cached_axis_positions(
+ source.column_names,
+ cells.rowIds,
+ source.shape[1],
+ object_path=f"{layer_path}/Dimnames/1",
+ )
+ else:
+ source = self._matrix_source_from_node(
+ layer_value,
+ object_path=layer_path,
+ )
+ feature_indices = features.membership(layer_name)
+ assay_cell_indices = cells.membership(layer_name)
+ resident_index_bytes += int(
+ feature_indices.nbytes + assay_cell_indices.nbytes
+ )
+ if resident_index_bytes * 4 > self._maximumIndexBytes:
+ raise _error(
+ "Assay5 stitching indexes exceed their memory budget",
+ object_path=f"{object_path}/layers",
+ code="metadata_index_limit",
+ requiredBytes=resident_index_bytes * 4,
+ maximumBytes=self._maximumIndexBytes,
+ )
+ if source.shape != (feature_indices.size, assay_cell_indices.size):
+ raise _error(
+ f"layer shape {source.shape} does not match LogMap membership "
+ f"({feature_indices.size}, {assay_cell_indices.size})",
+ object_path=layer_path,
+ code="layer_membership_dimension_mismatch",
+ sourceDimensions=source.shape,
+ featureMembership=int(feature_indices.size),
+ cellMembership=int(assay_cell_indices.size),
+ )
+ expected_features = _IndexedStringVector(
+ features.rowIds,
+ feature_indices,
+ object_path=f"{layer_path}/Dimnames/0",
+ )
+ expected_cells = _IndexedStringVector(
+ cells.rowIds,
+ assay_cell_indices,
+ object_path=f"{layer_path}/Dimnames/1",
+ )
+ if source.row_names is not None and not _identifiers_equal(
+ source.row_names,
+ expected_features,
+ ):
+ raise _error(
+ "layer feature identifiers conflict with feature LogMap",
+ object_path=f"{layer_path}/Dimnames/0",
+ code="layer_feature_id_conflict",
+ )
+ if source.column_names is not None and not _identifiers_equal(
+ source.column_names, expected_cells
+ ):
+ raise _error(
+ "layer cell identifiers conflict with cell LogMap",
+ object_path=f"{layer_path}/Dimnames/1",
+ code="layer_cell_id_conflict",
+ )
+ layer_global_positions = global_positions[assay_cell_indices]
+ resident_index_bytes += int(layer_global_positions.nbytes)
+ if resident_index_bytes * 4 > self._maximumIndexBytes:
+ raise _error(
+ "Assay5 stitching indexes exceed their memory budget",
+ object_path=f"{object_path}/layers",
+ code="metadata_index_limit",
+ requiredBytes=resident_index_bytes * 4,
+ maximumBytes=self._maximumIndexBytes,
+ )
+ placements.append(
+ LayerPlacement(
+ source,
+ feature_indices=feature_indices,
+ cell_indices=layer_global_positions,
+ name=layer_name,
+ )
+ )
+ try:
+ stitched = LayerStitchMatrixSource(
+ placements,
+ row_names=features.rowIds,
+ column_names=self.cellIds,
+ limits=self._matrixLimits,
+ )
+ except MatrixSourceError as cause:
+ raise _error(
+ str(cause),
+ object_path=f"{object_path}/layers",
+ code="layer_stitch_conflict",
+ causeType=type(cause).__name__,
+ ) from cause
+
+ metadata_node = _require_slot(
+ node,
+ "meta.data",
+ object_path=object_path,
+ )
+ feature_metadata = _data_frame(
+ metadata_node,
+ target_ids=features.rowIds,
+ document=self._document,
+ object_path=f"{object_path}/meta.data",
+ allow_compact=True,
+ scratch_dir=self._scratchDir,
+ maximum_index_bytes=self._maximumIndexBytes,
+ )
+ notices: list[SeuratNotice] = []
+ selected_count_names = {layer_name for layer_name, _ in count_layers}
+ available_count_names = {layer_name for layer_name, _ in available_count_layers}
+ for layer_name, layer_value in count_layers:
+ if isinstance(layer_value, _CachedLayerSpec):
+ notices.append(
+ SeuratNotice(
+ code="restored_sidecar_cache_layer",
+ message=(
+ f"raw count layer {layer_name!r} was reconstructed "
+ "from a safe SaveSeuratRds cache recipe"
+ ),
+ objectPath=layer_value.objectPath,
+ context={
+ "package": layer_value.package,
+ "sourceClass": layer_value.sourceClass,
+ },
+ )
+ )
+ for layer_name, _ in layer_entries:
+ if layer_name in selected_count_names:
+ continue
+ normalized = (
+ layer_name == "data"
+ or layer_name.startswith("data.")
+ or layer_name == "scale.data"
+ or layer_name.startswith("scale.data.")
+ )
+ unselected_count = layer_name in available_count_names
+ notices.append(
+ SeuratNotice(
+ code=(
+ "ignored_unselected_count_layer"
+ if unselected_count
+ else (
+ "ignored_normalized_layer"
+ if normalized
+ else "ignored_non_count_layer"
+ )
+ ),
+ message=(
+ f"raw count layer {layer_name!r} was not selected"
+ if unselected_count
+ else f"layer {layer_name!r} is not a raw count-bearing layer"
+ ),
+ objectPath=f"{object_path}/layers/{layer_name}",
+ context={},
+ )
+ )
+ for slot_name, slot in iter_attributes(node):
+ if slot_name not in {
+ "cells",
+ "class",
+ "features",
+ "layers",
+ "meta.data",
+ } and _node_has_content(slot):
+ notices.append(
+ SeuratNotice(
+ code="ignored_assay_slot",
+ message=f"assay slot {slot_name!r} is not imported",
+ objectPath=f"{object_path}/{slot_name}",
+ context={"rType": slot.type.name},
+ )
+ )
+ owned = _OwnedMatrixSource(
+ stitched,
+ self._document,
+ object_path=f"{object_path}/layers",
+ )
+ return SeuratAssay(
+ name=name,
+ sourceClass=_class_names(node, object_path=object_path)[0],
+ counts=owned,
+ featureIds=features.rowIds,
+ cellIds=self.cellIds,
+ assayCellIds=cells.rowIds,
+ cellMembership=SeuratMembership(
+ len(self.cellIds),
+ global_positions,
+ ),
+ featureMetadata=feature_metadata,
+ notices=tuple(notices),
+ objectPath=object_path,
+ )
+
+ def _r_matrix(
+ self,
+ node: RNode,
+ *,
+ object_path: str,
+ require_names: bool,
+ ) -> SeuratRMatrix:
+ values = _unwrap_atomic(node, object_path=object_path)
+ shape = _matrix_dimensions(node, object_path=object_path)
+ rows, columns = _matrix_dimnames(
+ node,
+ document=self._document,
+ object_path=object_path,
+ require_both=require_names,
+ )
+ if rows is None:
+ rows = tuple(str(index) for index in range(shape[0]))
+ if columns is None:
+ columns = tuple(str(index) for index in range(shape[1]))
+ if len(rows) != shape[0] or len(columns) != shape[1]:
+ raise _error(
+ "matrix dimensions conflict with Dimnames",
+ object_path=f"{object_path}/Dimnames",
+ code="dimnames_length_mismatch",
+ dimensions=shape,
+ rowIds=len(rows),
+ columnIds=len(columns),
+ )
+ if require_names:
+ _validate_unique_ids(
+ rows,
+ object_path=f"{object_path}/Dimnames/0",
+ scratch_dir=self._scratchDir,
+ maximum_bytes=self._maximumIndexBytes,
+ )
+ _validate_unique_ids(
+ columns,
+ object_path=f"{object_path}/Dimnames/1",
+ scratch_dir=self._scratchDir,
+ maximum_bytes=self._maximumIndexBytes,
+ )
+ return SeuratRMatrix(
+ values,
+ shape,
+ row_ids=rows,
+ column_ids=columns,
+ document=self._document,
+ object_path=object_path,
+ )
+
+ def _assay_for_reduction(self, name: str) -> SeuratAssay:
+ if name in self._assayModels:
+ return self._assayModels[name]
+ if name in self._assayErrors:
+ raise self._assayErrors[name]
+ node = self._assayNodes.get(name)
+ if node is None:
+ raise _error(
+ f"reduction references missing assay {name!r}",
+ object_path="reductions",
+ code="reduction_assay_missing",
+ assay=name,
+ )
+ classes = _class_names(node, object_path=f"assays/{name}")
+ if "Assay5" in classes:
+ assay = self._build_assay5(name, node)
+ elif "Assay" in classes:
+ assay = self._build_legacy_assay(name, node)
+ else:
+ raise _error(
+ f"reduction assay {name!r} has an unsupported class",
+ object_path=f"assays/{name}",
+ code="unsupported_assay_class",
+ classNames=classes,
+ )
+ self._assayModels[name] = assay
+ return assay
+
+ def _build_reduction(self, name: str, node: RNode) -> SeuratReduction:
+ object_path = f"reductions/{name}"
+ embeddings = self._r_matrix(
+ _require_slot(node, "cell.embeddings", object_path=object_path),
+ object_path=f"{object_path}/cell.embeddings",
+ require_names=True,
+ )
+ if not _identifiers_equal(embeddings.rowIds, self.cellIds):
+ raise _error(
+ "reduction cell identifiers do not exactly match global cell order",
+ object_path=f"{object_path}/cell.embeddings/Dimnames/0",
+ code="reduction_cell_id_conflict",
+ )
+ assay_used = _read_text_scalar(
+ _require_slot(node, "assay.used", object_path=object_path),
+ object_path=f"{object_path}/assay.used",
+ )
+ assay = self._assay_for_reduction(assay_used)
+ key = _read_text_scalar(
+ _require_slot(node, "key", object_path=object_path),
+ object_path=f"{object_path}/key",
+ )
+ global_reduction = _read_logical_scalar(
+ _require_slot(node, "global", object_path=object_path),
+ object_path=f"{object_path}/global",
+ )
+
+ loadings_node = get_slot(node, "feature.loadings")
+ feature_loadings: SeuratRMatrix | None = None
+ if loadings_node is not None:
+ loading_values = _unwrap_atomic(
+ loadings_node,
+ object_path=f"{object_path}/feature.loadings",
+ )
+ loading_dim = get_attribute(loadings_node, "Dim") or get_attribute(
+ loadings_node, "dim"
+ )
+ loading_shape = (
+ (0, 0)
+ if loading_dim is None and len(loading_values) == 0
+ else _matrix_dimensions(
+ loadings_node,
+ object_path=f"{object_path}/feature.loadings",
+ )
+ )
+ if loading_shape[0] > 0 and loading_shape[1] > 0:
+ feature_loadings = self._r_matrix(
+ loadings_node,
+ object_path=f"{object_path}/feature.loadings",
+ require_names=True,
+ )
+ if not _identifiers_equal(
+ feature_loadings.columnIds,
+ embeddings.columnIds,
+ ):
+ raise _error(
+ "loading component identifiers conflict with embeddings",
+ object_path=(f"{object_path}/feature.loadings/Dimnames/1"),
+ code="loading_component_id_conflict",
+ )
+ _positions_in_target(
+ feature_loadings.rowIds,
+ assay.featureIds,
+ object_path=f"{object_path}/feature.loadings/Dimnames/0",
+ scratch_dir=self._scratchDir,
+ maximum_bytes=self._maximumIndexBytes,
+ missing_message=(
+ "loading features are absent from the referenced assay"
+ ),
+ missing_code="loading_feature_id_conflict",
+ )
+
+ stdev_node = get_slot(node, "stdev")
+ stdev: SeuratNumericVector | None = None
+ if stdev_node is not None:
+ stdev_values = _unwrap_atomic(
+ stdev_node,
+ object_path=f"{object_path}/stdev",
+ )
+ if len(stdev_values) not in {0, embeddings.shape[1]}:
+ raise _error(
+ f"stdev has length {len(stdev_values)}; expected "
+ f"{embeddings.shape[1]}",
+ object_path=f"{object_path}/stdev",
+ code="stdev_length_mismatch",
+ actual=len(stdev_values),
+ expected=embeddings.shape[1],
+ )
+ if len(stdev_values):
+ stdev = SeuratNumericVector(
+ stdev_values,
+ self._document,
+ object_path=f"{object_path}/stdev",
+ )
+
+ notices: list[SeuratNotice] = []
+ for slot_name, slot in iter_attributes(node):
+ if slot_name == "feature.loadings.projected" and _node_has_content(slot):
+ notices.append(
+ SeuratNotice(
+ code="ignored_projected_loadings",
+ message="projected feature loadings are not imported",
+ objectPath=f"{object_path}/{slot_name}",
+ context={},
+ )
+ )
+ elif slot_name not in {
+ "assay.used",
+ "cell.embeddings",
+ "class",
+ "feature.loadings",
+ "global",
+ "key",
+ "stdev",
+ } and _node_has_content(slot):
+ notices.append(
+ SeuratNotice(
+ code="ignored_reduction_slot",
+ message=f"reduction slot {slot_name!r} is not imported",
+ objectPath=f"{object_path}/{slot_name}",
+ context={"rType": slot.type.name},
+ )
+ )
+ normalized_name = name.casefold()
+ role = (
+ "displayEmbedding"
+ if normalized_name in {"umap", "tsne", "t-sne"}
+ else "graphCoordinates"
+ )
+ return SeuratReduction(
+ name=name,
+ sourceClass=_class_names(node, object_path=object_path)[0],
+ role=role,
+ cellEmbeddings=embeddings,
+ featureLoadings=feature_loadings,
+ stdev=stdev,
+ assayUsed=assay_used,
+ key=key,
+ globalReduction=global_reduction,
+ imported=True,
+ computedByScarf=False,
+ notices=tuple(notices),
+ objectPath=object_path,
+ )
+
+
+def inspect_seurat(
+ source: Any,
+ *,
+ limits: RdsLimits | None = None,
+ temp_dir: str | os.PathLike[str] | None = None,
+ assays: Sequence[str] | None = None,
+ assay_layers: Mapping[str, Sequence[str]] | None = None,
+ reductions: Sequence[str] | None = None,
+ sidecar_path_remaps: Mapping[str | os.PathLike[str], str | os.PathLike[str]]
+ | None = None,
+ matrix_limits: SourceLimits = DEFAULT_LIMITS,
+) -> SeuratInspectResult:
+ with SeuratReader(
+ source,
+ limits=limits,
+ temp_dir=temp_dir,
+ assays=assays,
+ assay_layers=assay_layers,
+ reductions=reductions,
+ sidecar_path_remaps=sidecar_path_remaps,
+ matrix_limits=matrix_limits,
+ ) as reader:
+ return reader.inspection
+
+
+__all__ = [
+ "SeuratAssay",
+ "SeuratColumnBlock",
+ "SeuratDiagnostic",
+ "SeuratImportError",
+ "SeuratInspectResult",
+ "SeuratItemInspection",
+ "SeuratMembership",
+ "SeuratMetadata",
+ "SeuratMetadataColumn",
+ "SeuratNotice",
+ "SeuratNumericVector",
+ "SeuratRMatrix",
+ "SeuratReader",
+ "SeuratReduction",
+ "inspect_seurat",
+]
diff --git a/scarf/storage/__init__.py b/scarf/storage/__init__.py
new file mode 100644
index 00000000..81d1a945
--- /dev/null
+++ b/scarf/storage/__init__.py
@@ -0,0 +1,23 @@
+"""Storage package public surface.
+
+Analyst-facing artifact types and inspection helpers live here. Path builders,
+fingerprint helpers, and artifact writers stay package-internal. Prefer the
+``DataStore`` wrappers (``list_artifacts``, ``inspect_artifact``,
+``load_artifact``, ``get_assay_state``) in analysis code.
+"""
+
+from .artifacts import (
+ ARTIFACT_KINDS as ARTIFACT_KINDS,
+ ArtifactRef as ArtifactRef,
+ ArtifactStatus as ArtifactStatus,
+ inspect_artifact as inspect_artifact,
+ list_artifacts as list_artifacts,
+)
+
+__all__ = [
+ "ARTIFACT_KINDS",
+ "ArtifactRef",
+ "ArtifactStatus",
+ "inspect_artifact",
+ "list_artifacts",
+]
diff --git a/scarf/storage/ann_index.py b/scarf/storage/ann_index.py
new file mode 100644
index 00000000..7f335e99
--- /dev/null
+++ b/scarf/storage/ann_index.py
@@ -0,0 +1,192 @@
+import hashlib
+import os
+import tempfile
+from typing import Any
+
+import numpy as np
+import zarr
+
+from .types import as_zarr_array
+from .layout import _group_zarr_format, get_compressors
+from .profiles import StorageProfile
+
+ANN_INDEX_ARRAY = "ann_idx_bytes"
+ANN_INDEX_CHUNK_BYTES = 8 * 1024 * 1024
+ANN_INDEX_FORMAT_VERSION = 1
+_ANN_INDEX_METADATA = (
+ "ann_index_format_version",
+ "metric",
+ "dimensions",
+ "element_count",
+ "payload_sha256",
+)
+
+
+def has_ann_index(group: zarr.Group, name: str = ANN_INDEX_ARRAY) -> bool:
+ """Return whether a group contains an ANN index byte array."""
+ return name in group
+
+
+def legacy_ann_index_path(zw_root: str | None, ann_loc: str) -> str | None:
+ """Return the legacy filesystem path for an ANN index."""
+ if zw_root is None:
+ return None
+ return os.path.join(zw_root, ann_loc, "ann_idx")
+
+
+def save_ann_index(
+ group: zarr.Group,
+ ann_idx: Any,
+ *,
+ profile: StorageProfile,
+ metric: str,
+ dimensions: int,
+ element_count: int,
+ name: str = ANN_INDEX_ARRAY,
+) -> None:
+ """Persist an hnswlib index as a chunked byte array."""
+ if int(ann_idx.get_current_count()) != int(element_count):
+ raise ValueError("ANN index element count does not match its coordinates")
+ with tempfile.NamedTemporaryFile(delete=False) as tmp:
+ path = tmp.name
+ try:
+ ann_idx.save_index(path)
+ byte_length = os.path.getsize(path)
+ if name in group:
+ del group[name]
+ chunk_size = min(ANN_INDEX_CHUNK_BYTES, max(byte_length, 1))
+ zarr_format = _group_zarr_format(group)
+ array = group.create_array(
+ name,
+ shape=(byte_length,),
+ chunks=(chunk_size,),
+ dtype="uint8",
+ overwrite=True,
+ compressors=get_compressors(
+ profile,
+ zarrFormat=zarr_format,
+ ),
+ )
+ digest = hashlib.sha256()
+ with open(path, "rb") as source:
+ for start in range(0, byte_length, ANN_INDEX_CHUNK_BYTES):
+ values = np.frombuffer(
+ source.read(min(ANN_INDEX_CHUNK_BYTES, byte_length - start)),
+ dtype=np.uint8,
+ )
+ digest.update(values)
+ array[start : start + len(values)] = values
+ array.attrs["byte_length"] = byte_length
+ array.attrs["ann_index_format_version"] = ANN_INDEX_FORMAT_VERSION
+ array.attrs["metric"] = str(metric)
+ array.attrs["dimensions"] = int(dimensions)
+ array.attrs["element_count"] = int(element_count)
+ array.attrs["payload_sha256"] = digest.hexdigest()
+ finally:
+ os.unlink(path)
+
+
+def load_ann_index(
+ group: zarr.Group,
+ space: str,
+ dim: int,
+ expected_count: int | None = None,
+ name: str = ANN_INDEX_ARRAY,
+) -> Any:
+ """Load an hnswlib index from a Zarr byte array."""
+ import hnswlib
+
+ if name not in group:
+ raise FileNotFoundError(f"ANN index array {name!r} not found in group")
+ source = as_zarr_array(group[name], name=name)
+ if source.ndim != 1 or np.dtype(source.dtype) != np.dtype(np.uint8):
+ raise ValueError("ANN index payload must be a one-dimensional uint8 array")
+ stored_byte_length = source.attrs.get("byte_length")
+ if stored_byte_length is not None:
+ if isinstance(stored_byte_length, bool) or not isinstance(
+ stored_byte_length, int
+ ):
+ raise ValueError("ANN index byte length is invalid")
+ if stored_byte_length != int(source.shape[0]):
+ raise ValueError("ANN index byte length does not match its payload")
+ present_metadata = {key for key in _ANN_INDEX_METADATA if key in source.attrs}
+ if present_metadata and present_metadata != set(_ANN_INDEX_METADATA):
+ raise ValueError("ANN index metadata is incomplete")
+ stored_count: int | None = None
+ stored_digest: str | None = None
+ if present_metadata:
+ stored_version = source.attrs["ann_index_format_version"]
+ stored_metric = source.attrs["metric"]
+ stored_dimensions = source.attrs["dimensions"]
+ stored_element_count = source.attrs["element_count"]
+ payload_digest = source.attrs["payload_sha256"]
+ if (
+ isinstance(stored_version, bool)
+ or not isinstance(stored_version, int)
+ or stored_version != ANN_INDEX_FORMAT_VERSION
+ ):
+ raise ValueError("ANN index format version is unsupported")
+ if not isinstance(stored_metric, str) or stored_metric != space:
+ raise ValueError("ANN index metric does not match artifact provenance")
+ if (
+ isinstance(stored_dimensions, bool)
+ or not isinstance(stored_dimensions, int)
+ or stored_dimensions != dim
+ ):
+ raise ValueError("ANN index dimensions do not match artifact provenance")
+ if isinstance(stored_element_count, bool) or not isinstance(
+ stored_element_count, int
+ ):
+ raise ValueError("ANN index element count is invalid")
+ stored_count = stored_element_count
+ if expected_count is not None and stored_count != int(expected_count):
+ raise ValueError("ANN index element count does not match coordinates")
+ if not isinstance(payload_digest, str):
+ raise ValueError("ANN index payload digest is invalid")
+ stored_digest = payload_digest
+ with tempfile.NamedTemporaryFile(delete=False) as tmp:
+ path = tmp.name
+ try:
+ digest = hashlib.sha256()
+ with open(path, "wb") as destination:
+ for start in range(0, int(source.shape[0]), ANN_INDEX_CHUNK_BYTES):
+ values = np.asarray(
+ source[
+ start : min(
+ start + ANN_INDEX_CHUNK_BYTES,
+ int(source.shape[0]),
+ )
+ ],
+ dtype=np.uint8,
+ )
+ digest.update(values)
+ destination.write(memoryview(values))
+ if stored_digest is not None and digest.hexdigest() != stored_digest:
+ raise ValueError("ANN index payload digest does not match its metadata")
+ index = hnswlib.Index(space=space, dim=dim)
+ index.load_index(path)
+ actual_count = int(index.get_current_count())
+ required_count = stored_count if expected_count is None else int(expected_count)
+ if required_count is not None and actual_count != required_count:
+ raise ValueError("ANN index element count does not match coordinates")
+ return index
+ finally:
+ os.unlink(path)
+
+
+def load_ann_index_from_path(
+ path: str,
+ space: str,
+ dim: int,
+ expected_count: int | None = None,
+) -> Any:
+ """Load an hnswlib index from a legacy filesystem path."""
+ import hnswlib
+
+ index = hnswlib.Index(space=space, dim=dim)
+ index.load_index(path)
+ if expected_count is not None and int(index.get_current_count()) != int(
+ expected_count
+ ):
+ raise ValueError("ANN index element count does not match coordinates")
+ return index
diff --git a/scarf/storage/arrays.py b/scarf/storage/arrays.py
new file mode 100644
index 00000000..64a71821
--- /dev/null
+++ b/scarf/storage/arrays.py
@@ -0,0 +1,259 @@
+from collections.abc import Iterable
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+import zarr
+
+from .layout import (
+ ZarrArraySpec,
+ _group_zarr_format,
+ get_compressors,
+ normalize_chunks,
+)
+from .profiles import StorageProfile, resolve_storage_profile
+
+_MISSING_COLUMN_PREFIX = "__scarf_missing__"
+
+
+@dataclass(frozen=True, slots=True)
+class MetadataBlock:
+ """A contiguous block for one metadata column."""
+
+ start: int
+ values: np.ndarray
+ missing: np.ndarray | None = None
+
+
+def _checked_shards(
+ shards: tuple[int, ...],
+ chunks: tuple[int, ...],
+) -> tuple[int, ...]:
+ """Return shard extents that hold a whole number of the resolved chunks."""
+ resolved = tuple(max(1, int(value)) for value in shards)
+ if len(resolved) != len(chunks):
+ raise ValueError(f"Array shards {resolved} do not match chunks {chunks}")
+ for shard, chunk in zip(resolved, chunks, strict=True):
+ if shard < chunk or shard % chunk:
+ raise ValueError(
+ f"Array shards {resolved} must hold whole chunks {chunks}; "
+ f"shard extent {shard} is not a multiple of chunk extent {chunk}"
+ )
+ return resolved
+
+
+def create_numeric_array(
+ group: zarr.Group,
+ name: str,
+ spec: ZarrArraySpec,
+) -> zarr.Array:
+ """Create a numeric Zarr array from a specification."""
+ zarrFormat = _group_zarr_format(group)
+ chunks = normalize_chunks(spec.chunks, spec.shape)
+ kwargs: dict[str, Any] = {
+ "shape": spec.shape,
+ "chunks": chunks,
+ "dtype": spec.dtype,
+ "compressors": (
+ spec.compressors
+ if zarrFormat >= 3
+ else get_compressors(
+ resolve_storage_profile(group.store),
+ zarrFormat=2,
+ )
+ ),
+ "overwrite": spec.overwrite,
+ }
+ if spec.shards is not None and zarrFormat >= 3:
+ kwargs["shards"] = _checked_shards(spec.shards, chunks)
+ if spec.fillValue is not None:
+ kwargs["fill_value"] = spec.fillValue
+ return group.create_array(name, **kwargs)
+
+
+def dtype_fix(dtype: Any, data: np.ndarray) -> Any:
+ """Infer or adjust a metadata dtype from sample values."""
+ if dtype is None or np.dtype(dtype).kind == "O":
+ return "U" + str(max(len(str(value)) for value in data))
+ if np.issubdtype(data.dtype, np.dtype("S")):
+ try:
+ decoded = data.astype("U")
+ except UnicodeDecodeError:
+ decoded = np.array([value.decode("UTF-8") for value in data]).astype("U")
+ return decoded.dtype
+ return dtype
+
+
+def create_metadata_column(
+ group: zarr.Group,
+ name: str,
+ data: np.ndarray | list[Any] | None = None,
+ dtype: Any = None,
+ overwrite: bool = True,
+ chunkSize: int | bool | None = None,
+ shape: int | None = None,
+ profile: StorageProfile | None = None,
+) -> zarr.Array:
+ """Create a metadata column, optionally from provided data."""
+ if chunkSize is None or chunkSize is False:
+ chunks: tuple[int, ...] | bool = False
+ else:
+ chunks = (chunkSize,)
+
+ resolved_profile = profile or resolve_storage_profile(group.store)
+ compressors = get_compressors(
+ resolved_profile,
+ zarrFormat=_group_zarr_format(group),
+ )
+
+ if data is not None:
+ values = np.array(data)
+ values = np.asarray(values, dtype=dtype_fix(dtype, values))
+ if chunks is False:
+ chunks = (len(values),)
+ return group.create_array(
+ name,
+ data=values,
+ chunks=chunks,
+ overwrite=overwrite,
+ compressors=compressors,
+ )
+
+ if shape is None:
+ raise ValueError("shape is required when data is None")
+ if chunks is False:
+ chunks = (shape,)
+ return group.create_array(
+ name,
+ shape=(shape,),
+ chunks=chunks,
+ dtype=dtype,
+ overwrite=overwrite,
+ compressors=compressors,
+ )
+
+
+def create_streamed_metadata_column(
+ group: zarr.Group,
+ name: str,
+ *,
+ shape: int,
+ dtype: Any,
+ blocks: Iterable[MetadataBlock],
+ overwrite: bool = True,
+ chunkSize: int = 100_000,
+ hasMissing: bool = False,
+ profile: StorageProfile | None = None,
+) -> zarr.Array:
+ """Create and fill a metadata column from bounded contiguous blocks."""
+ if shape < 0:
+ raise ValueError("shape must be non-negative")
+ if chunkSize < 1:
+ raise ValueError("chunkSize must be positive")
+ output = create_metadata_column(
+ group,
+ name,
+ dtype=dtype,
+ overwrite=overwrite,
+ chunkSize=chunkSize,
+ shape=shape,
+ profile=profile,
+ )
+ missing_output: zarr.Array | None = None
+ if hasMissing:
+ missing_name = f"{_MISSING_COLUMN_PREFIX}{name}"
+ missing_output = create_metadata_column(
+ group,
+ missing_name,
+ dtype=bool,
+ overwrite=overwrite,
+ chunkSize=chunkSize,
+ shape=shape,
+ profile=profile,
+ )
+ output.attrs["missing_mask"] = missing_name
+
+ next_row = 0
+ for block in blocks:
+ if block.start != next_row:
+ raise ValueError(
+ f"Metadata blocks must be contiguous; expected {next_row}, "
+ f"received {block.start}"
+ )
+ values = np.asarray(block.values)
+ if values.ndim != 1:
+ raise ValueError("Metadata blocks must be one-dimensional")
+ stop = block.start + len(values)
+ if stop > shape:
+ raise ValueError("Metadata block exceeds declared shape")
+ if values.dtype != np.dtype(dtype):
+ values = values.astype(dtype)
+ output[block.start : stop] = values
+
+ if block.missing is not None:
+ if missing_output is None:
+ raise ValueError("A missing mask was supplied but hasMissing is false")
+ missing = np.asarray(block.missing, dtype=bool)
+ if missing.shape != values.shape:
+ raise ValueError("Missing mask must align with metadata values")
+ missing_output[block.start : stop] = missing
+ elif missing_output is not None:
+ missing_output[block.start : stop] = False
+ next_row = stop
+
+ if next_row != shape:
+ raise ValueError(
+ f"Metadata column is incomplete: wrote {next_row} of {shape} rows"
+ )
+ return output
+
+
+def _normalize_chunks(chunks: tuple[int, ...] | int) -> tuple[int, ...]:
+ if isinstance(chunks, int):
+ return (chunks,)
+ return chunks
+
+
+def create_zarr_dataset(
+ group: zarr.Group,
+ name: str,
+ chunks: tuple[int, ...] | int,
+ dtype: Any,
+ shape: tuple[int, ...],
+ overwrite: bool = True,
+ profile: StorageProfile | None = None,
+) -> zarr.Array:
+ resolved_profile = profile or resolve_storage_profile(group.store)
+ spec = ZarrArraySpec(
+ shape=shape,
+ chunks=_normalize_chunks(chunks),
+ dtype=dtype,
+ compressors=get_compressors(
+ resolved_profile,
+ zarrFormat=_group_zarr_format(group),
+ ),
+ overwrite=overwrite,
+ )
+ return create_numeric_array(group, name, spec)
+
+
+def create_zarr_obj_array(
+ group: zarr.Group,
+ name: str,
+ data: Any,
+ dtype: str | Any = None,
+ overwrite: bool = True,
+ chunk_size: int = 100000,
+ shape: int | None = None,
+ profile: StorageProfile | None = None,
+) -> zarr.Array:
+ return create_metadata_column(
+ group,
+ name,
+ data=data,
+ dtype=dtype,
+ overwrite=overwrite,
+ chunkSize=chunk_size,
+ shape=shape if data is None else None,
+ profile=profile,
+ )
diff --git a/scarf/storage/artifact_writer.py b/scarf/storage/artifact_writer.py
new file mode 100644
index 00000000..8cd6b225
--- /dev/null
+++ b/scarf/storage/artifact_writer.py
@@ -0,0 +1,288 @@
+from dataclasses import dataclass
+from collections.abc import Callable
+import time
+from typing import Any
+
+import numpy as np
+import zarr
+
+from .artifacts import (
+ ArtifactRef,
+ ArtifactScope,
+ artifact_path,
+ canonical_bytes,
+ find_reusable_artifacts,
+ make_provenance,
+ new_artifact_id,
+ serialize_artifact_value,
+)
+
+
+@dataclass(frozen=True, slots=True)
+class PlannedArtifact:
+ ref: ArtifactRef
+ provenance: dict[str, Any]
+ execution_options: dict[str, Any]
+ reused: bool
+ required_arrays: tuple[Any, ...]
+ required_attributes: tuple[Any, ...]
+ reuse_validator: Callable[[ArtifactRef, zarr.Group], bool] | None
+
+ def invalidated(
+ self,
+ root: zarr.Group,
+ *,
+ required_arrays: tuple[Any, ...] | None = None,
+ required_attributes: tuple[Any, ...] | None = None,
+ reuse_validator: Callable[[ArtifactRef, zarr.Group], bool] | None = None,
+ ) -> "PlannedArtifact":
+ if not isinstance(self.provenance, dict):
+ raise TypeError("PlannedArtifact provenance must be a mapping")
+ operation = self.provenance.get("operation")
+ parameters = self.provenance.get("parameters")
+ inputs = self.provenance.get("inputs")
+ if (
+ not isinstance(operation, str)
+ or not isinstance(parameters, dict)
+ or not isinstance(inputs, dict)
+ ):
+ raise TypeError("PlannedArtifact provenance is incomplete for invalidation")
+ return plan_artifact(
+ root,
+ scope=self.ref.scope,
+ assay=self.ref.assay,
+ kind=self.ref.kind,
+ operation=operation,
+ parameters=parameters,
+ inputs=inputs,
+ execution_options=dict(self.execution_options),
+ invalidate_cache=True,
+ required_arrays=(
+ self.required_arrays if required_arrays is None else required_arrays
+ ),
+ required_attributes=(
+ self.required_attributes
+ if required_attributes is None
+ else required_attributes
+ ),
+ reuse_validator=(
+ self.reuse_validator if reuse_validator is None else reuse_validator
+ ),
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class ArrayRequirement:
+ name: str
+ shape: tuple[int | None, ...] | None = None
+ dtype_kind: str | None = None
+ dtype: Any | None = None
+
+ def matches(self, group: zarr.Group) -> bool:
+ from .types import as_zarr_array
+
+ if self.name not in group:
+ return False
+ try:
+ array = as_zarr_array(group[self.name], name=self.name)
+ except TypeError:
+ return False
+ if self.shape is not None:
+ if len(array.shape) != len(self.shape):
+ return False
+ if any(
+ expected is not None and int(actual) != expected
+ for actual, expected in zip(array.shape, self.shape, strict=True)
+ ):
+ return False
+ if self.dtype_kind is not None:
+ if np.dtype(array.dtype).kind != self.dtype_kind:
+ return False
+ if self.dtype is not None and np.dtype(array.dtype) != np.dtype(self.dtype):
+ return False
+ return True
+
+
+@dataclass(frozen=True, slots=True)
+class AttributeRequirement:
+ name: str
+ expected_types: tuple[type[Any], ...] | None = None
+ predicate: Callable[[Any], bool] | None = None
+
+ def matches(self, group: zarr.Group) -> bool:
+ if self.name not in group.attrs:
+ return False
+ value = group.attrs[self.name]
+ if self.expected_types is not None and not isinstance(
+ value,
+ self.expected_types,
+ ):
+ return False
+ return self.predicate(value) if self.predicate is not None else True
+
+
+def plan_artifact(
+ root: zarr.Group,
+ *,
+ scope: ArtifactScope,
+ kind: str,
+ operation: str,
+ parameters: dict[str, Any],
+ inputs: dict[str, Any],
+ execution_options: dict[str, Any],
+ assay: str | None = None,
+ invalidate_cache: bool = False,
+ required_arrays: tuple[str | ArrayRequirement, ...] = (),
+ required_attributes: tuple[str | AttributeRequirement, ...] = (),
+ reuse_validator: Callable[[ArtifactRef, zarr.Group], bool] | None = None,
+) -> PlannedArtifact:
+ provenance = make_provenance(
+ operation=operation,
+ parameters=parameters,
+ inputs=inputs,
+ )
+ stored_execution_options = serialize_artifact_value(execution_options)
+ if not isinstance(stored_execution_options, dict):
+ raise TypeError("execution_options must serialize to a mapping")
+ canonical_bytes(stored_execution_options)
+ candidates = find_reusable_artifacts(
+ root,
+ scope=scope,
+ assay=assay,
+ kind=kind,
+ provenance=provenance,
+ invalidate_cache=invalidate_cache,
+ )
+ requirements = tuple(
+ requirement
+ if isinstance(requirement, ArrayRequirement)
+ else ArrayRequirement(requirement)
+ for requirement in required_arrays
+ )
+ attribute_requirements = tuple(
+ requirement
+ if isinstance(requirement, AttributeRequirement)
+ else AttributeRequirement(requirement)
+ for requirement in required_attributes
+ )
+ from .artifacts import artifact_group
+
+ reused = None
+ for candidate in candidates:
+ try:
+ group = artifact_group(root, candidate)
+ except KeyError:
+ continue
+ if any(not requirement.matches(group) for requirement in requirements):
+ continue
+ if any(
+ not requirement.matches(group) for requirement in attribute_requirements
+ ):
+ continue
+ if reuse_validator is not None and not reuse_validator(
+ candidate,
+ group,
+ ):
+ continue
+ reused = candidate
+ break
+ if reused is not None:
+ return PlannedArtifact(
+ ref=reused,
+ provenance=provenance,
+ execution_options=stored_execution_options,
+ reused=True,
+ required_arrays=required_arrays,
+ required_attributes=required_attributes,
+ reuse_validator=reuse_validator,
+ )
+ while True:
+ ref = ArtifactRef(
+ scope=scope,
+ assay=assay,
+ kind=kind,
+ artifact_id=new_artifact_id(),
+ )
+ if artifact_path(ref) not in root:
+ break
+ return PlannedArtifact(
+ ref=ref,
+ provenance=provenance,
+ execution_options=stored_execution_options,
+ reused=False,
+ required_arrays=required_arrays,
+ required_attributes=required_attributes,
+ reuse_validator=reuse_validator,
+ )
+
+
+def start_artifact(root: zarr.Group, planned: PlannedArtifact) -> zarr.Group:
+ if planned.reused:
+ raise ValueError("Cannot start a reused artifact")
+ path = artifact_path(planned.ref)
+ if path in root:
+ raise FileExistsError(f"Artifact path already exists: {path}")
+ group = root.create_group(path)
+ group.attrs.update(
+ {
+ "artifact_id": planned.ref.artifact_id,
+ "kind": planned.ref.kind,
+ "provenance": planned.provenance,
+ "execution_options": planned.execution_options,
+ "created_at_ns": time.time_ns(),
+ "complete": False,
+ }
+ )
+ return group
+
+
+def finish_artifact(
+ group: zarr.Group,
+ planned: PlannedArtifact,
+) -> None:
+ if planned.reused:
+ raise ValueError("Cannot finish a reused artifact")
+ group.attrs["complete"] = False
+ if (
+ group.attrs.get("artifact_id") != planned.ref.artifact_id
+ or group.attrs.get("kind") != planned.ref.kind
+ ):
+ raise ValueError("Artifact group does not match its creation plan")
+ for requirement in planned.required_arrays:
+ resolved = (
+ requirement
+ if isinstance(requirement, ArrayRequirement)
+ else ArrayRequirement(str(requirement))
+ )
+ if not resolved.matches(group):
+ raise ValueError(
+ f"Artifact array {resolved.name!r} does not satisfy its contract"
+ )
+ for requirement in planned.required_attributes:
+ attribute_requirement = (
+ requirement
+ if isinstance(requirement, AttributeRequirement)
+ else AttributeRequirement(str(requirement))
+ )
+ if not attribute_requirement.matches(group):
+ raise ValueError(
+ f"Artifact attribute {attribute_requirement.name!r} "
+ "does not satisfy its contract"
+ )
+ if planned.reuse_validator is not None and not planned.reuse_validator(
+ planned.ref,
+ group,
+ ):
+ raise ValueError("Artifact payload does not satisfy its reuse contract")
+ group.attrs["complete"] = True
+
+
+def reused_artifact_group(
+ root: zarr.Group,
+ planned: PlannedArtifact,
+) -> zarr.Group:
+ from .artifacts import artifact_group
+
+ if not planned.reused:
+ raise ValueError("Artifact is not reused")
+ return artifact_group(root, planned.ref)
diff --git a/scarf/storage/artifacts.py b/scarf/storage/artifacts.py
new file mode 100644
index 00000000..f728fb92
--- /dev/null
+++ b/scarf/storage/artifacts.py
@@ -0,0 +1,604 @@
+import hashlib
+import inspect
+import json
+import math
+import secrets
+import struct
+from collections.abc import Iterable, Mapping, Sequence
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+import numpy as np
+import zarr
+
+from .geometry import array_geometry
+from .partition import row_band
+from .refs import (
+ ARTIFACT_KINDS as ARTIFACT_KINDS,
+ ArtifactRef as ArtifactRef,
+ ArtifactScope as ArtifactScope,
+ ExternalArtifactRef as ExternalArtifactRef,
+ _validate_artifact_kind,
+ _validate_name,
+ artifact_path as artifact_path,
+ parse_artifact_path as parse_artifact_path,
+)
+from .types import as_zarr_array, as_zarr_group
+
+
+def new_artifact_id() -> str:
+ return secrets.token_hex(32)
+
+
+def group_at(root: zarr.Group, path: str) -> zarr.Group:
+ return as_zarr_group(root[path], name=path)
+
+
+def artifact_group(root: zarr.Group, ref: ArtifactRef) -> zarr.Group:
+ return group_at(root, artifact_path(ref))
+
+
+def _canonical_node(value: Any) -> Any:
+ if isinstance(value, ArtifactRef | ExternalArtifactRef):
+ return _canonical_node(value.to_dict())
+ if isinstance(value, np.generic):
+ return _canonical_node(value.item())
+ if value is None:
+ return ["none"]
+ if isinstance(value, bool):
+ return ["bool", value]
+ if isinstance(value, int):
+ return ["int", str(value)]
+ if isinstance(value, float):
+ if not math.isfinite(value):
+ raise ValueError("Provenance cannot contain non-finite floats")
+ return ["float", struct.pack(">d", value).hex()]
+ if isinstance(value, str):
+ return ["str", value]
+ if isinstance(value, bytes):
+ return ["bytes", value.hex()]
+ if isinstance(value, Path):
+ return _canonical_node(str(value))
+ if isinstance(value, np.ndarray):
+ raise TypeError("Arrays must be represented by a value_fingerprint")
+ if isinstance(value, Mapping):
+ if any(not isinstance(key, str) for key in value):
+ raise TypeError("Provenance mappings require string keys")
+ return [
+ "mapping",
+ [[key, _canonical_node(value[key])] for key in sorted(value)],
+ ]
+ if isinstance(value, set | frozenset):
+ items = [_canonical_node(item) for item in value]
+ items.sort(key=lambda item: json.dumps(item, separators=(",", ":")))
+ return ["set", items]
+ if isinstance(value, Sequence):
+ return ["sequence", [_canonical_node(item) for item in value]]
+ raise TypeError(f"Unsupported provenance value: {type(value).__name__}")
+
+
+def canonical_bytes(value: Any) -> bytes:
+ return json.dumps(
+ _canonical_node(value),
+ ensure_ascii=False,
+ separators=(",", ":"),
+ ).encode("utf-8")
+
+
+def _canonical_dtype(dtype: np.dtype[Any]) -> np.dtype[Any]:
+ if dtype.subdtype is not None:
+ base, shape = dtype.subdtype
+ return np.dtype((_canonical_dtype(base), shape))
+ if dtype.fields is not None:
+ assert dtype.names is not None
+ return np.dtype(
+ [(name, _canonical_dtype(dtype.fields[name][0])) for name in dtype.names],
+ align=False,
+ )
+ return dtype.newbyteorder("<")
+
+
+def _dtype_descriptor(dtype: np.dtype[Any]) -> dict[str, Any]:
+ normalized = _canonical_dtype(dtype)
+ if normalized.fields is not None:
+ assert normalized.names is not None
+ return {
+ "fields": [
+ [name, _dtype_descriptor(normalized.fields[name][0])]
+ for name in normalized.names
+ ]
+ }
+ if normalized.subdtype is not None:
+ base, shape = normalized.subdtype
+ return {
+ "base": _dtype_descriptor(base),
+ "shape": list(shape),
+ }
+ return {"str": normalized.str}
+
+
+def _canonical_array(values: np.ndarray) -> np.ndarray:
+ array = np.asarray(values)
+ if array.dtype.hasobject:
+ raise TypeError("Object arrays require an explicit stable encoding")
+ dtype = _canonical_dtype(array.dtype)
+ if array.dtype.fields is None:
+ return np.ascontiguousarray(array.astype(dtype, copy=False))
+ packed = np.empty(array.shape, dtype=dtype)
+ assert array.dtype.names is not None
+ for name in array.dtype.names:
+ packed[name] = array[name]
+ return np.ascontiguousarray(packed)
+
+
+class ValueFingerprintBuilder:
+ def __init__(self) -> None:
+ self._digest = hashlib.blake2b(digest_size=32, person=b"scarf-values")
+ self._active_array: tuple[str, np.dtype[Any], tuple[int, ...], int] | None = (
+ None
+ )
+
+ def update_bytes(self, name: str, payload: bytes) -> None:
+ if self._active_array is not None:
+ raise RuntimeError("Finish the active array before adding another value")
+ encoded_name = name.encode("utf-8")
+ self._digest.update(len(encoded_name).to_bytes(8, "big"))
+ self._digest.update(encoded_name)
+ self._digest.update(len(payload).to_bytes(8, "big"))
+ self._digest.update(payload)
+
+ def update_array(self, name: str, values: np.ndarray) -> None:
+ array = np.asarray(values)
+ self.begin_array(name, array.shape, array.dtype)
+ if array.shape[0]:
+ self.update_array_block(name, (0,) * array.ndim, array)
+ self.end_array(name)
+
+ def begin_array(
+ self,
+ name: str,
+ shape: tuple[int, ...],
+ dtype: np.dtype[Any] | str,
+ ) -> None:
+ if self._active_array is not None:
+ raise RuntimeError("Finish the active array before starting another")
+ normalized_dtype = np.dtype(dtype)
+ if normalized_dtype.hasobject:
+ raise TypeError("Object arrays require an explicit stable encoding")
+ normalized_dtype = _canonical_dtype(normalized_dtype)
+ normalized_shape = tuple(int(size) for size in shape)
+ if not normalized_shape or any(size < 0 for size in normalized_shape):
+ raise ValueError("Array shape must contain non-negative dimensions")
+ self.update_bytes(
+ f"{name}:metadata",
+ canonical_bytes(
+ {
+ "dtype": _dtype_descriptor(normalized_dtype),
+ "shape": list(normalized_shape),
+ }
+ ),
+ )
+ self._active_array = (name, normalized_dtype, normalized_shape, 0)
+
+ def update_array_block(
+ self,
+ name: str,
+ offset: tuple[int, ...],
+ values: np.ndarray,
+ ) -> None:
+ if self._active_array is None:
+ raise RuntimeError("begin_array must be called before writing blocks")
+ active_name, dtype, shape, next_row = self._active_array
+ if name != active_name:
+ raise ValueError(f"Expected block for {active_name!r}, got {name!r}")
+ raw_array = np.asarray(values)
+ if _canonical_dtype(raw_array.dtype) != dtype:
+ raise TypeError(f"Expected dtype {dtype}, got {raw_array.dtype}")
+ array = _canonical_array(raw_array)
+ expected_offset = (next_row,) + (0,) * (len(shape) - 1)
+ if offset != expected_offset:
+ raise ValueError(f"Expected block offset {expected_offset}, got {offset}")
+ if array.ndim != len(shape) or array.shape[1:] != shape[1:]:
+ raise ValueError(
+ f"Block shape {array.shape} is incompatible with array shape {shape}"
+ )
+ stop = next_row + array.shape[0]
+ if stop > shape[0]:
+ raise ValueError("Array block exceeds declared shape")
+ self._digest.update(array.view(np.uint8).tobytes())
+ self._active_array = (active_name, dtype, shape, stop)
+
+ def end_array(self, name: str) -> None:
+ if self._active_array is None:
+ raise RuntimeError("No active array to finish")
+ active_name, _dtype, shape, next_row = self._active_array
+ if name != active_name:
+ raise ValueError(f"Expected to finish {active_name!r}, got {name!r}")
+ if next_row != shape[0]:
+ raise ValueError(
+ f"Array {name!r} is incomplete: wrote {next_row} of {shape[0]} rows"
+ )
+ self._active_array = None
+
+ def hexdigest(self) -> str:
+ if self._active_array is not None:
+ raise RuntimeError("Cannot finish fingerprint while an array is incomplete")
+ return self._digest.hexdigest()
+
+
+def fingerprint_array(values: np.ndarray) -> str:
+ builder = ValueFingerprintBuilder()
+ builder.update_array("values", values)
+ return builder.hexdigest()
+
+
+def _stored_array_chunk_rows(array: zarr.Array) -> int:
+ return row_band(array_geometry(array), unit="chunk", fallback=1)
+
+
+def fingerprint_stored_arrays(
+ group: zarr.Group,
+ names: Sequence[str],
+) -> str:
+ builder = ValueFingerprintBuilder()
+ for name in names:
+ array = as_zarr_array(group[name], name=name)
+ builder.begin_array(name, array.shape, array.dtype)
+ chunk_rows = _stored_array_chunk_rows(array)
+ for start in range(0, array.shape[0], chunk_rows):
+ stop = min(start + chunk_rows, array.shape[0])
+ block = np.asarray(array[start:stop])
+ builder.update_array_block(
+ name,
+ (start,) + (0,) * (array.ndim - 1),
+ block,
+ )
+ builder.end_array(name)
+ return builder.hexdigest()
+
+
+def fingerprint_stored_strings(array: zarr.Array) -> str:
+ """Fingerprint a stored string column without loading it in full."""
+ if array.ndim != 1:
+ raise ValueError("Stored string fingerprints require a one-dimensional array")
+
+ chunk_rows = _stored_array_chunk_rows(array)
+ source_dtype = np.dtype(array.dtype)
+ if source_dtype.hasobject:
+ max_length = 1
+ for start in range(0, array.shape[0], chunk_rows):
+ stop = min(start + chunk_rows, array.shape[0])
+ values = np.asarray(array[start:stop])
+ if values.size:
+ max_length = max(
+ max_length,
+ max(len(str(value)) for value in values),
+ )
+ string_dtype = np.dtype(f"U{max_length}")
+ else:
+ string_dtype = np.empty(0, dtype=source_dtype).astype(str).dtype
+
+ builder = ValueFingerprintBuilder()
+ builder.begin_array("values", array.shape, string_dtype)
+ for start in range(0, array.shape[0], chunk_rows):
+ stop = min(start + chunk_rows, array.shape[0])
+ block = np.asarray(array[start:stop]).astype(string_dtype)
+ builder.update_array_block("values", (start,), block)
+ builder.end_array("values")
+ return builder.hexdigest()
+
+
+def fingerprint_strings(values: np.ndarray) -> str:
+ strings = np.asarray(values).astype(str)
+ return fingerprint_array(strings)
+
+
+def fingerprint_string_blocks(
+ blocks: Iterable[tuple[int, np.ndarray]],
+ *,
+ length: int,
+ max_length: int,
+) -> str:
+ """Fingerprint ordered strings without collecting the full column."""
+ if length < 0:
+ raise ValueError("length must be non-negative")
+ if max_length < 1:
+ raise ValueError("max_length must be positive")
+ dtype = np.dtype(f"U{max_length}")
+ builder = ValueFingerprintBuilder()
+ builder.begin_array("values", (length,), dtype)
+ next_row = 0
+ for start, raw_values in blocks:
+ if int(start) != next_row:
+ raise ValueError(
+ f"String blocks must be contiguous; expected {next_row}, "
+ f"received {start}"
+ )
+ values = np.asarray(raw_values).astype(dtype)
+ if values.ndim != 1:
+ raise ValueError("String blocks must be one-dimensional")
+ builder.update_array_block("values", (next_row,), values)
+ next_row += len(values)
+ builder.end_array("values")
+ return builder.hexdigest()
+
+
+def callable_identity(value: Any) -> dict[str, str]:
+ explicit = getattr(value, "artifact_identity", None)
+ if explicit is not None:
+ return {"identity": str(explicit)}
+ qualname = str(getattr(value, "__qualname__", type(value).__qualname__))
+ if (
+ (not inspect.isfunction(value) and not inspect.isbuiltin(value))
+ or "" in qualname
+ or "" in qualname
+ or bool(getattr(value, "__closure__", None))
+ ):
+ raise ValueError("Dynamic or stateful callables must define artifact_identity")
+ return {
+ "module": str(getattr(value, "__module__", type(value).__module__)),
+ "qualname": qualname,
+ }
+
+
+def serialize_artifact_value(value: Any) -> Any:
+ if isinstance(value, ArtifactRef | ExternalArtifactRef):
+ return value.to_dict()
+ if isinstance(value, np.ndarray):
+ return {"value_fingerprint": fingerprint_array(value)}
+ if isinstance(value, np.generic):
+ return serialize_artifact_value(value.item())
+ if isinstance(value, float) and not math.isfinite(value):
+ if math.isnan(value):
+ return {"special_float": "nan"}
+ return {"special_float": "inf" if value > 0 else "-inf"}
+ if isinstance(value, Path):
+ return str(value)
+ if isinstance(value, bytes):
+ return {"bytes_hex": value.hex()}
+ if callable(value):
+ return {"external_hook": True, **callable_identity(value)}
+ if isinstance(value, Mapping):
+ return {str(key): serialize_artifact_value(item) for key, item in value.items()}
+ if isinstance(value, tuple | list):
+ return [serialize_artifact_value(item) for item in value]
+ if isinstance(value, set | frozenset):
+ values = [serialize_artifact_value(item) for item in value]
+ return sorted(values, key=canonical_bytes)
+ return value
+
+
+def make_provenance(
+ *,
+ operation: str,
+ parameters: Mapping[str, Any],
+ inputs: Mapping[str, Any],
+) -> dict[str, Any]:
+ _validate_name(operation, "operation")
+ provenance = {
+ "operation": operation,
+ "parameters": serialize_artifact_value(parameters),
+ "inputs": serialize_artifact_value(inputs),
+ }
+ canonical_bytes(provenance)
+ return provenance
+
+
+def provenance_hash(provenance: Mapping[str, Any]) -> str:
+ digest = hashlib.blake2b(digest_size=32, person=b"scarf-provenance")
+ digest.update(canonical_bytes(provenance))
+ return digest.hexdigest()
+
+
+@dataclass(frozen=True, slots=True)
+class ArtifactStatus:
+ ref: ArtifactRef
+ path: str
+ exists: bool
+ complete: bool
+ provenance: dict[str, Any] | None = None
+ execution_options: dict[str, Any] | None = None
+
+ @property
+ def operation(self) -> str | None:
+ if self.provenance is None:
+ return None
+ value = self.provenance.get("operation")
+ return value if isinstance(value, str) else None
+
+ @property
+ def parameters(self) -> dict[str, Any] | None:
+ if self.provenance is None:
+ return None
+ value = self.provenance.get("parameters")
+ return dict(value) if isinstance(value, Mapping) else None
+
+ @property
+ def inputs(self) -> dict[str, Any] | None:
+ if self.provenance is None:
+ return None
+ value = self.provenance.get("inputs")
+ return dict(value) if isinstance(value, Mapping) else None
+
+
+def _mapping_attr(group: zarr.Group, name: str) -> dict[str, Any] | None:
+ if name not in group.attrs:
+ return None
+ value = group.attrs[name]
+ if not isinstance(value, Mapping):
+ raise TypeError(f"Artifact attr {name!r} must be a mapping")
+ return dict(value)
+
+
+def require_complete_artifact(
+ root: zarr.Group,
+ ref: ArtifactRef,
+) -> ArtifactStatus:
+ status = inspect_artifact(root, ref)
+ if not status.exists:
+ raise KeyError(f"Artifact does not exist: {status.path}")
+ if not status.complete:
+ raise RuntimeError(f"Artifact is incomplete: {status.path}")
+ return status
+
+
+def inspect_artifact(root: zarr.Group, ref: ArtifactRef) -> ArtifactStatus:
+ path = artifact_path(ref)
+ if path not in root:
+ return ArtifactStatus(ref=ref, path=path, exists=False, complete=False)
+ group = group_at(root, path)
+ stored_id = group.attrs.get("artifact_id")
+ stored_kind = group.attrs.get("kind")
+ if stored_id is not None and stored_id != ref.artifact_id:
+ raise ValueError(f"Artifact at {path} has mismatched artifact_id")
+ if stored_kind is not None and stored_kind != ref.kind:
+ raise ValueError(f"Artifact at {path} has mismatched kind")
+ raw_complete = group.attrs.get("complete", False)
+ if not isinstance(raw_complete, bool):
+ raise TypeError(f"Artifact complete attr at {path} must be boolean")
+ complete = raw_complete
+ if complete:
+ required = {
+ "artifact_id",
+ "kind",
+ "provenance",
+ "execution_options",
+ "complete",
+ }
+ missing = required - set(group.attrs)
+ if missing:
+ raise KeyError(
+ f"Completed artifact at {path} is missing attrs: "
+ f"{', '.join(sorted(missing))}"
+ )
+ provenance = _mapping_attr(group, "provenance")
+ execution_options = _mapping_attr(group, "execution_options")
+ if complete:
+ if provenance is None or execution_options is None:
+ raise KeyError(f"Completed artifact at {path} has an incomplete record")
+ operation = provenance.get("operation")
+ parameters = provenance.get("parameters")
+ inputs = provenance.get("inputs")
+ if (
+ not isinstance(operation, str)
+ or not isinstance(parameters, Mapping)
+ or not isinstance(inputs, Mapping)
+ ):
+ raise TypeError(f"Artifact provenance at {path} is malformed")
+ _validate_name(operation, "operation")
+ canonical_bytes(provenance)
+ return ArtifactStatus(
+ ref=ref,
+ path=path,
+ exists=True,
+ complete=complete,
+ provenance=provenance,
+ execution_options=execution_options,
+ )
+
+
+def artifact_exists(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ *,
+ require_complete: bool = True,
+) -> bool:
+ status = inspect_artifact(root, ref)
+ return status.exists and (status.complete or not require_complete)
+
+
+def list_artifacts(
+ root: zarr.Group,
+ *,
+ scope: ArtifactScope,
+ assay: str | None = None,
+ kind: str | None = None,
+ complete_only: bool = False,
+) -> list[ArtifactRef]:
+ if scope not in {"assay", "datastore"}:
+ raise ValueError(f"Invalid artifact scope: {scope!r}")
+ if scope == "assay":
+ if assay is None or not assay or "/" in assay:
+ raise ValueError("assay is required for assay-scoped artifact listing")
+ base_path = f"{assay}/artifacts"
+ else:
+ if assay is not None:
+ raise ValueError("assay cannot be set for datastore-scoped listing")
+ base_path = "artifacts"
+ if kind is not None:
+ _validate_artifact_kind(kind)
+ if base_path not in root:
+ return []
+ base = as_zarr_group(root[base_path], name=base_path)
+ kinds = [kind] if kind is not None else sorted(base.group_keys())
+ refs = []
+ for artifact_kind in kinds:
+ if artifact_kind not in base:
+ continue
+ _validate_artifact_kind(artifact_kind)
+ kind_group = as_zarr_group(base[artifact_kind], name=artifact_kind)
+ for artifact_id in sorted(kind_group.group_keys()):
+ try:
+ ref = ArtifactRef(
+ scope=scope,
+ assay=assay,
+ kind=artifact_kind,
+ artifact_id=artifact_id,
+ )
+ except ValueError:
+ continue
+ if complete_only and not artifact_exists(root, ref):
+ continue
+ refs.append(ref)
+ return refs
+
+
+def find_reusable_artifacts(
+ root: zarr.Group,
+ *,
+ scope: ArtifactScope,
+ kind: str,
+ provenance: Mapping[str, Any],
+ assay: str | None = None,
+ invalidate_cache: bool = False,
+) -> list[ArtifactRef]:
+ if invalidate_cache:
+ return []
+ requested = make_provenance(
+ operation=str(provenance["operation"]),
+ parameters=provenance["parameters"],
+ inputs=provenance["inputs"],
+ )
+ requested_hash = provenance_hash(requested)
+ requested_bytes = canonical_bytes(requested)
+ reusable: list[tuple[int, ArtifactRef]] = []
+ for ref in list_artifacts(
+ root,
+ scope=scope,
+ assay=assay,
+ kind=kind,
+ ):
+ try:
+ status = inspect_artifact(root, ref)
+ except (KeyError, TypeError, ValueError):
+ continue
+ if not status.complete or status.provenance is None:
+ continue
+ if provenance_hash(status.provenance) != requested_hash:
+ continue
+ if canonical_bytes(status.provenance) == requested_bytes:
+ group = group_at(root, status.path)
+ raw_created = group.attrs.get("created_at_ns", 0)
+ created_at_ns = (
+ int(raw_created)
+ if not isinstance(raw_created, bool)
+ and isinstance(raw_created, (int, np.integer))
+ else 0
+ )
+ reusable.append((created_at_ns, ref))
+ reusable.sort(
+ key=lambda item: (item[0], item[1].artifact_id),
+ reverse=True,
+ )
+ return [ref for _, ref in reusable]
diff --git a/scarf/storage/budget.py b/scarf/storage/budget.py
new file mode 100644
index 00000000..8e28e0d6
--- /dev/null
+++ b/scarf/storage/budget.py
@@ -0,0 +1,315 @@
+"""Resource limits used by one storage or analysis operation."""
+
+import os
+from collections.abc import Callable
+from dataclasses import dataclass
+from pathlib import Path
+
+_FALLBACK_MEMORY_BYTES = 8 * 1024 * 1024 * 1024
+_SUFFIXES = {"K": 1024, "M": 1024**2, "G": 1024**3, "T": 1024**4}
+_MIN_RAW_BYTES = 1024 * 1024
+_CGROUP_UNLIMITED_THRESHOLD = 1 << 60
+
+
+@dataclass(frozen=True)
+class ResourceBudget:
+ """Memory and CPU limits for one operation."""
+
+ memoryBytes: int
+ workers: int
+
+
+def _read_int(path: str) -> int | None:
+ try:
+ value = Path(path).read_text().strip()
+ except OSError:
+ return None
+ if not value or value == "max":
+ return None
+ try:
+ parsed = int(value)
+ except ValueError:
+ return None
+ if parsed <= 0 or parsed >= _CGROUP_UNLIMITED_THRESHOLD:
+ return None
+ return parsed
+
+
+def _physical_memory_bytes() -> int:
+ try:
+ with open("/proc/meminfo") as handle:
+ for line in handle:
+ if line.startswith("MemTotal:"):
+ return int(line.split()[1]) * 1024
+ except OSError:
+ pass
+ try:
+ page_size = os.sysconf("SC_PAGE_SIZE")
+ total_pages = os.sysconf("SC_PHYS_PAGES")
+ if page_size > 0 and total_pages > 0:
+ return int(page_size) * int(total_pages)
+ except (ValueError, OSError, AttributeError):
+ pass
+ return _FALLBACK_MEMORY_BYTES
+
+
+def _process_cgroup_entry(controller: str) -> tuple[list[str], str] | None:
+ try:
+ lines = Path("/proc/self/cgroup").read_text().splitlines()
+ except OSError:
+ return None
+ for line in lines:
+ fields = line.split(":", 2)
+ if len(fields) != 3:
+ continue
+ controllers = fields[1].split(",") if fields[1] else []
+ if (not controllers and controller == "") or controller in controllers:
+ return controllers, fields[2].lstrip("/")
+ return None
+
+
+def _process_cgroup_path(controller: str) -> str | None:
+ entry = _process_cgroup_entry(controller)
+ return None if entry is None else entry[1]
+
+
+def _unified_cgroup_files(name: str) -> list[str]:
+ path = _process_cgroup_path("")
+ parts = Path(path).parts if path else ()
+ candidates = [
+ str(Path("/sys/fs/cgroup", *parts[:depth], name))
+ for depth in range(len(parts), -1, -1)
+ ]
+ return list(dict.fromkeys(candidates))
+
+
+def _legacy_cgroup_files(controller: str, name: str) -> list[str]:
+ entry = _process_cgroup_entry(controller)
+ controllers, path = ([], "") if entry is None else entry
+ parts = Path(path).parts if path else ()
+ mount_names = list(dict.fromkeys([",".join(controllers), controller]))
+ candidates = [
+ str(Path("/sys/fs/cgroup", mount_name, *parts[:depth], name))
+ for mount_name in mount_names
+ if mount_name
+ for depth in range(len(parts), -1, -1)
+ ]
+ return list(dict.fromkeys(candidates))
+
+
+def _cgroup_memory_bytes() -> int | None:
+ paths = _unified_cgroup_files("memory.max")
+ paths.extend(_legacy_cgroup_files("memory", "memory.limit_in_bytes"))
+ limits = [value for path in paths if (value := _read_int(path)) is not None]
+ return min(limits) if limits else None
+
+
+def detect_total_memory_bytes() -> int:
+ """Return the effective physical or cgroup memory limit."""
+ physical = _physical_memory_bytes()
+ cgroup = _cgroup_memory_bytes()
+ return physical if cgroup is None else min(physical, cgroup)
+
+
+def _cpu_quota_workers() -> int | None:
+ limits: list[int] = []
+ for path in _unified_cgroup_files("cpu.max"):
+ try:
+ quota_text, period_text = Path(path).read_text().strip().split()
+ except (OSError, ValueError):
+ continue
+ if quota_text != "max":
+ try:
+ quota = int(quota_text)
+ period = int(period_text)
+ except ValueError:
+ continue
+ if quota > 0 and period > 0:
+ limits.append(max(1, quota // period))
+
+ for legacy_quota_path in _legacy_cgroup_files("cpu", "cpu.cfs_quota_us"):
+ legacy_period_path = str(Path(legacy_quota_path).with_name("cpu.cfs_period_us"))
+ legacy_quota = _read_int(legacy_quota_path)
+ legacy_period = _read_int(legacy_period_path)
+ if legacy_quota is not None and legacy_period is not None:
+ limits.append(max(1, legacy_quota // legacy_period))
+ return min(limits) if limits else None
+
+
+def detect_workers() -> int:
+ limits = [max(1, os.cpu_count() or 1)]
+ try:
+ limits.append(max(1, len(os.sched_getaffinity(0))))
+ except (AttributeError, OSError):
+ pass
+ quota = _cpu_quota_workers()
+ if quota is not None:
+ limits.append(quota)
+ return min(limits)
+
+
+def _parse_memory_spec(spec: str | int | float) -> int:
+ guidance = (
+ "Use raw bytes, a suffixed size like '8G'/'512M', "
+ "or a fraction strictly between 0 and 1 like '0.6'."
+ )
+ if isinstance(spec, bool):
+ raise ValueError(f"Invalid memory spec: {spec!r}. {guidance}")
+ if isinstance(spec, int):
+ if spec <= 0:
+ raise ValueError(f"Memory budget must be positive, got {spec!r}.")
+ return spec
+
+ text = str(spec).strip()
+ if not text:
+ raise ValueError("Empty memory spec. " + guidance)
+ suffix = text[-1].upper()
+ if suffix in _SUFFIXES:
+ try:
+ value = float(text[:-1])
+ except ValueError as exc:
+ raise ValueError(f"Invalid memory spec: {spec!r}. {guidance}") from exc
+ if value <= 0:
+ raise ValueError(f"Memory budget must be positive, got {spec!r}.")
+ return max(1, int(value * _SUFFIXES[suffix]))
+
+ try:
+ number = float(text)
+ except ValueError as exc:
+ raise ValueError(f"Invalid memory spec: {spec!r}. {guidance}") from exc
+ if number <= 0:
+ raise ValueError(f"Memory budget must be positive, got {spec!r}.")
+ if number < 1:
+ return max(1, int(number * detect_total_memory_bytes()))
+ if number < _MIN_RAW_BYTES:
+ raise ValueError(
+ f"Ambiguous memory spec {spec!r}: a bare number >= 1 is read as bytes, "
+ f"which is implausibly small. {guidance}"
+ )
+ return int(number)
+
+
+def resolve_budget(
+ memory: int | str | None = None,
+ workers: int | None = None,
+) -> ResourceBudget:
+ """Build a :class:`ResourceBudget`, auto-detecting unset fields."""
+ if memory is None:
+ env_mem = os.environ.get("SCARF_MEM_BUDGET")
+ if env_mem:
+ memory_bytes = _parse_memory_spec(env_mem)
+ else:
+ memory_bytes = detect_total_memory_bytes()
+ else:
+ memory_bytes = _parse_memory_spec(memory)
+
+ if workers is None:
+ env_workers = os.environ.get("SCARF_WORKERS")
+ if env_workers:
+ try:
+ worker_count = int(env_workers)
+ except ValueError as exc:
+ raise ValueError(
+ f"Invalid SCARF_WORKERS={env_workers!r}; expected an integer."
+ ) from exc
+ else:
+ worker_count = detect_workers()
+ else:
+ worker_count = int(workers)
+
+ return ResourceBudget(
+ memoryBytes=max(1, memory_bytes),
+ workers=max(1, worker_count),
+ )
+
+
+def admitted_worker_count(
+ resources: ResourceBudget,
+ *,
+ taskBytes: int,
+ residentBytes: int = 0,
+ requested: int | None = None,
+) -> int:
+ """Bound concurrent tasks by CPU and live byte estimates."""
+ workers, _ = admitted_worker_split(
+ resources,
+ nTasks=resources.workers if requested is None else requested,
+ taskBytes=lambda _: taskBytes,
+ residentBytes=residentBytes,
+ requested=requested,
+ )
+ return workers
+
+
+def admitted_worker_split(
+ resources: ResourceBudget,
+ *,
+ nTasks: int,
+ taskBytes: Callable[[int], int],
+ residentBytes: int = 0,
+ requested: int | None = None,
+) -> tuple[int, int]:
+ """Split worker slots between outer tasks and each task's inner work."""
+ cpu = min(
+ resources.workers,
+ resources.workers if requested is None else max(1, int(requested)),
+ )
+ tasks = max(1, int(nTasks))
+ resident = max(0, int(residentBytes))
+ available = resources.memoryBytes - resident
+ if available <= 0:
+ raise MemoryError(
+ f"Resident data needs about {resident} bytes, but the operation "
+ f"limit is {resources.memoryBytes} bytes"
+ )
+
+ for outer in range(min(cpu, tasks), 0, -1):
+ for inner in range(max(1, cpu // outer), 0, -1):
+ per_task = max(1, int(taskBytes(inner)))
+ if outer * per_task <= available:
+ return outer, inner
+
+ one_task = max(1, int(taskBytes(1)))
+ raise MemoryError(
+ f"One task needs about {one_task} bytes in addition to {resident} "
+ f"resident bytes, but the operation limit is {resources.memoryBytes} bytes"
+ )
+
+
+DEFAULT_READ_AHEAD_BLOCKS = 2
+"""Read depth a public streaming API uses when the caller has not planned one."""
+
+
+@dataclass(frozen=True, slots=True)
+class StreamAdmission:
+ """Read depth and per-read concurrency admitted for one block stream."""
+
+ outerWorkers: int
+ ioConcurrency: int
+
+
+def admit_stream(
+ resources: ResourceBudget,
+ *,
+ nBlocks: int,
+ blockBytes: int,
+ decodeBytes: int = 0,
+ residentBytes: int = 0,
+ requested: int | None = None,
+) -> StreamAdmission:
+ """Admit read depth for a block stream that also decodes stored chunks.
+
+ Each in-flight read owns its block buffer plus as many decoded chunks as its
+ own concurrency allows, so ``decodeBytes`` is charged per concurrent decode
+ rather than once per read.
+ """
+ block = max(1, int(blockBytes))
+ decode = max(0, int(decodeBytes))
+ outer, inner = admitted_worker_split(
+ resources,
+ nTasks=nBlocks,
+ taskBytes=lambda concurrency: block + concurrency * decode,
+ residentBytes=residentBytes,
+ requested=requested,
+ )
+ return StreamAdmission(outerWorkers=outer, ioConcurrency=inner)
diff --git a/scarf/storage/copy.py b/scarf/storage/copy.py
new file mode 100644
index 00000000..51e4d5ec
--- /dev/null
+++ b/scarf/storage/copy.py
@@ -0,0 +1,134 @@
+import os
+from typing import Any
+
+import numpy as np
+import zarr
+
+from .types import as_zarr_array
+from .arrays import create_metadata_column, create_numeric_array, dtype_fix
+from .budget import ResourceBudget
+from .geometry import array_geometry
+from .layout import PROFILE_METADATA_CHUNK, normed_array_spec
+from .partition import row_band
+from .sharding import write_dense_in_shard_rows
+
+
+def copy_zarr_array(
+ src: zarr.Array,
+ dst: zarr.Array,
+ msg: str | None = None,
+ resources: ResourceBudget | None = None,
+) -> None:
+ """Stream-copy a 2D Zarr array in row blocks."""
+ if src.shape != dst.shape:
+ raise ValueError(f"Shape mismatch: src {src.shape} vs dst {dst.shape}")
+ if len(src.shape) != 2:
+ raise ValueError("copy_zarr_array only supports 2D arrays")
+ write_dense_in_shard_rows(
+ dst,
+ lambda start, end: np.asarray(src[start:end, :]),
+ msg=msg or "Copying Zarr array",
+ resources=resources,
+ )
+
+
+def _metadata_block_rows(array: zarr.Array) -> int:
+ return row_band(
+ array_geometry(array),
+ unit="chunk",
+ fallback=PROFILE_METADATA_CHUNK,
+ )
+
+
+def _resolve_metadata_dtype(
+ array: zarr.Array,
+ block_rows: int,
+) -> np.dtype[Any]:
+ dtype: np.dtype[Any] = np.dtype(array.dtype)
+ if not (dtype.kind in {"O", "S"} or dtype.hasobject):
+ return dtype
+ n_rows = int(array.shape[0])
+ if n_rows == 0:
+ return np.dtype("U1")
+ max_len = 1
+ for start in range(0, n_rows, block_rows):
+ stop = min(start + block_rows, n_rows)
+ block = np.asarray(array[start:stop])
+ if block.size == 0:
+ continue
+ resolved: np.dtype[Any] = np.dtype(dtype_fix(dtype, block))
+ if resolved.kind == "U":
+ max_len = max(max_len, resolved.itemsize // 4)
+ else:
+ return resolved
+ return np.dtype(f"U{max_len}")
+
+
+def _copy_metadata_array(
+ src: zarr.Array,
+ dst: zarr.Group,
+ name: str,
+ *,
+ overwrite: bool,
+) -> None:
+ if src.ndim != 1:
+ create_metadata_column(
+ dst,
+ name,
+ data=np.asarray(src[:]),
+ dtype=src.dtype,
+ overwrite=overwrite,
+ chunkSize=PROFILE_METADATA_CHUNK,
+ )
+ target = as_zarr_array(dst[name], name=name)
+ else:
+ block_rows = _metadata_block_rows(src)
+ dtype = _resolve_metadata_dtype(src, block_rows)
+ target = create_metadata_column(
+ dst,
+ name,
+ data=None,
+ dtype=dtype,
+ overwrite=overwrite,
+ chunkSize=PROFILE_METADATA_CHUNK,
+ shape=int(src.shape[0]),
+ )
+ n_rows = int(src.shape[0])
+ for start in range(0, n_rows, block_rows):
+ stop = min(start + block_rows, n_rows)
+ target[start:stop] = np.asarray(src[start:stop], dtype=dtype)
+
+ if "display" in src.attrs:
+ target.attrs["display"] = src.attrs["display"]
+
+
+def copy_zarr_group_tree(
+ src: zarr.Group,
+ dst: zarr.Group,
+ *,
+ overwrite: bool = True,
+) -> None:
+ """Recursively copy a Zarr group tree."""
+ for name, node in src.members():
+ if isinstance(node, zarr.Group):
+ child = dst.create_group(name, overwrite=overwrite)
+ copy_zarr_group_tree(node, child, overwrite=overwrite)
+ else:
+ array = as_zarr_array(node, name=name)
+ _copy_metadata_array(array, dst, name, overwrite=overwrite)
+
+
+def create_or_open_staged_normed_array(
+ cache_path: str,
+ shape: tuple[int, int],
+) -> zarr.Array:
+ """Open or create a reusable local normalized-data array."""
+ if os.path.exists(os.path.join(cache_path, "zarr.json")):
+ root = zarr.open_group(cache_path, mode="r+")
+ if "data" in root:
+ array = as_zarr_array(root["data"], name="data")
+ if tuple(array.shape) == tuple(shape):
+ return array
+ root = zarr.open_group(cache_path, mode="w")
+ spec = normed_array_spec(shape[0], shape[1], profile="fast_local")
+ return create_numeric_array(root, "data", spec)
diff --git a/scarf/storage/feature_stream.py b/scarf/storage/feature_stream.py
new file mode 100644
index 00000000..644aa49d
--- /dev/null
+++ b/scarf/storage/feature_stream.py
@@ -0,0 +1,204 @@
+"""Geometry-aware planning for feature-column streams."""
+
+import operator
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+
+from .budget import ResourceBudget, admit_stream
+from .geometry import ArrayGeometry, array_geometry
+from .partition import (
+ IndexBlock,
+ affordable_width,
+ checked_indices,
+ partition_indices,
+)
+
+type BlockBytes = Callable[[int], int]
+
+__all__ = [
+ "FeatureStreamPlan",
+ "feature_column_chunk",
+ "plan_feature_stream",
+]
+
+
+@dataclass(frozen=True, slots=True)
+class FeatureStreamPlan:
+ """Ordered feature blocks and their admitted read concurrency."""
+
+ geometry: ArrayGeometry
+ featureAxis: int
+ blocks: tuple[IndexBlock, ...]
+ readWorkers: int
+ ioConcurrency: int
+ repeatedDecodeCount: int
+
+
+def _axis(value: int, *, name: str) -> int:
+ resolved = operator.index(value)
+ if resolved not in (0, 1):
+ raise ValueError(f"{name} must be 0 or 1")
+ return int(resolved)
+
+
+def _plane(array: Any) -> ArrayGeometry:
+ geometry = array_geometry(array)
+ if geometry is None or len(geometry.shape) != 2:
+ raise ValueError("Feature streams require a chunked two-dimensional array")
+ return geometry
+
+
+def feature_column_chunk(array: Any, *, featureAxis: int) -> int:
+ """Return one physical feature-chunk width."""
+ return _plane(array).axisChunk(_axis(featureAxis, name="featureAxis"))
+
+
+def _positive_requested(value: int | None) -> int | None:
+ if value is None:
+ return None
+ if isinstance(value, (bool, np.bool_)):
+ raise TypeError("requestedBatchSize must be a positive integer")
+ try:
+ resolved = operator.index(value)
+ except TypeError:
+ raise TypeError("requestedBatchSize must be a positive integer") from None
+ if resolved < 1:
+ raise ValueError("requestedBatchSize must be greater than zero")
+ return int(resolved)
+
+
+def _owned_bytes(blockBytes: BlockBytes, width: int) -> int:
+ value = int(blockBytes(max(1, int(width))))
+ if value < 1:
+ raise ValueError("blockBytes must return a positive byte count")
+ return value
+
+
+def _repeated_decodes(
+ blocks: Sequence[IndexBlock],
+ *,
+ cell_bin_count: int,
+) -> int:
+ touches: dict[int, int] = {}
+ for block in blocks:
+ for feature_bin in block.bins:
+ touches[feature_bin] = touches.get(feature_bin, 0) + 1
+ return sum(max(0, count - 1) * cell_bin_count for count in touches.values())
+
+
+def plan_feature_stream(
+ array: Any,
+ *,
+ featureAxis: int,
+ cellAxis: int,
+ featureIndices: Sequence[int] | np.ndarray,
+ cellIndices: Sequence[int] | np.ndarray,
+ resources: ResourceBudget,
+ blockBytes: BlockBytes,
+ residentBytes: int = 0,
+ requestedBatchSize: int | None = None,
+) -> FeatureStreamPlan:
+ """Plan variable-width feature blocks from physical chunk geometry."""
+ feature_axis = _axis(featureAxis, name="featureAxis")
+ cell_axis = _axis(cellAxis, name="cellAxis")
+ if feature_axis == cell_axis:
+ raise ValueError("featureAxis and cellAxis must differ")
+
+ geometry = _plane(array)
+ feature_indices = checked_indices(
+ featureIndices,
+ limit=geometry.shape[feature_axis],
+ name="featureIndices",
+ )
+ cell_indices = checked_indices(
+ cellIndices,
+ limit=geometry.shape[cell_axis],
+ name="cellIndices",
+ )
+ requested = _positive_requested(requestedBatchSize)
+ resident = max(0, int(residentBytes))
+ available = resources.memoryBytes - resident
+ if available <= 0:
+ raise MemoryError(
+ f"Resident data needs {resident} bytes, but the operation limit is "
+ f"{resources.memoryBytes} bytes"
+ )
+
+ decode_bytes = geometry.nominalChunkBytes()
+
+ def fits(width: int) -> bool:
+ return _owned_bytes(blockBytes, width) + decode_bytes <= available
+
+ if feature_indices.size == 0:
+ return FeatureStreamPlan(
+ geometry=geometry,
+ featureAxis=feature_axis,
+ blocks=(),
+ readWorkers=1,
+ ioConcurrency=1,
+ repeatedDecodeCount=0,
+ )
+
+ if requested is not None:
+ blocks = partition_indices(
+ geometry,
+ feature_axis,
+ feature_indices,
+ maxWidth=requested,
+ )
+ if any(not fits(block.indices.size) for block in blocks):
+ raise MemoryError(
+ f"Requested feature batch width {requested} does not fit; "
+ f"the affordable width is {affordable_width(fits, requested)}"
+ )
+ else:
+ blocks = partition_indices(
+ geometry,
+ feature_axis,
+ feature_indices,
+ fits=fits,
+ )
+
+ block_bytes = max(_owned_bytes(blockBytes, block.indices.size) for block in blocks)
+ prefetchable = len(blocks) - 1
+ # A read-ahead stream holds the block being consumed while the next ones load.
+ # A single-block stream holds nothing before its own read.
+ held = resident + (block_bytes + decode_bytes if prefetchable else 0)
+ read_workers = 1
+ io_concurrency = 1
+ try:
+ admission = admit_stream(
+ resources,
+ nBlocks=max(1, prefetchable),
+ blockBytes=block_bytes,
+ decodeBytes=decode_bytes,
+ residentBytes=held,
+ )
+ except MemoryError:
+ # A second materialized block may not fit, while the current block can
+ # still use the remaining budget for concurrent chunk decodes.
+ current_admission = admit_stream(
+ resources,
+ nBlocks=1,
+ blockBytes=block_bytes,
+ decodeBytes=decode_bytes,
+ residentBytes=resident,
+ )
+ io_concurrency = current_admission.ioConcurrency
+ else:
+ io_concurrency = admission.ioConcurrency
+ if prefetchable:
+ read_workers = admission.outerWorkers
+
+ cell_bins = int(np.unique(geometry.binOf(cell_axis, cell_indices)).size)
+ return FeatureStreamPlan(
+ geometry=geometry,
+ featureAxis=feature_axis,
+ blocks=tuple(blocks),
+ readWorkers=read_workers,
+ ioConcurrency=io_concurrency,
+ repeatedDecodeCount=_repeated_decodes(blocks, cell_bin_count=cell_bins),
+ )
diff --git a/scarf/storage/geometry.py b/scarf/storage/geometry.py
new file mode 100644
index 00000000..d6ea4093
--- /dev/null
+++ b/scarf/storage/geometry.py
@@ -0,0 +1,85 @@
+"""Stored chunk geometry of a streamed array."""
+
+import operator
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+
+from .types import array_metadata_shards
+
+__all__ = ["ArrayGeometry", "array_geometry"]
+
+
+@dataclass(frozen=True, slots=True)
+class ArrayGeometry:
+ """Chunk and shard extents an array is physically stored in."""
+
+ shape: tuple[int, ...]
+ chunks: tuple[int, ...]
+ shards: tuple[int, ...] | None
+ itemsize: int
+
+ def axisChunk(self, axis: int) -> int:
+ """Return the chunk extent along one axis."""
+ return max(1, self.chunks[axis])
+
+ def axisShard(self, axis: int) -> int:
+ """Return the shard extent along one axis, or its chunk extent."""
+ if self.shards is None:
+ return self.axisChunk(axis)
+ return max(1, self.shards[axis])
+
+ def nominalChunkBytes(self) -> int:
+ """Return the bytes one whole chunk decodes to.
+
+ Zarr stores and decodes a full chunk even where the chunk overhangs the
+ array edge, so this deliberately ignores the clipped extent.
+ """
+ elements = 1
+ for extent in self.chunks:
+ elements *= max(1, extent)
+ return elements * self.itemsize
+
+ def binOf(self, axis: int, indices: np.ndarray) -> np.ndarray:
+ """Map indices along one axis to the chunk they live in."""
+ return np.asarray(indices, dtype=np.int64) // self.axisChunk(axis)
+
+
+def array_geometry(array: Any) -> ArrayGeometry | None:
+ """Read stored geometry, or ``None`` when the array has no chunk layout.
+
+ In-memory arrays have no chunks, so callers get one explicit place to decide
+ what an unchunked array should mean for them.
+ """
+ chunks = getattr(array, "chunks", None)
+ if chunks is None:
+ return None
+ resolved_chunks = tuple(operator.index(value) for value in chunks)
+ resolved_shape = tuple(operator.index(value) for value in array.shape)
+ if not resolved_chunks:
+ return None
+ if len(resolved_chunks) != len(resolved_shape):
+ raise ValueError(
+ f"Array chunks {resolved_chunks} do not match shape {resolved_shape}"
+ )
+ if any(value < 1 for value in resolved_chunks):
+ raise ValueError(f"Array chunks must be positive, got {resolved_chunks}")
+ if any(value < 0 for value in resolved_shape):
+ raise ValueError(f"Array dimensions cannot be negative, got {resolved_shape}")
+
+ raw_shards = array_metadata_shards(array)
+ shards = None
+ if raw_shards is not None:
+ shards = tuple(operator.index(value) for value in raw_shards)
+ if len(shards) != len(resolved_shape):
+ raise ValueError(
+ f"Array shards {shards} do not match shape {resolved_shape}"
+ )
+
+ return ArrayGeometry(
+ shape=resolved_shape,
+ chunks=resolved_chunks,
+ shards=shards,
+ itemsize=max(1, int(np.dtype(array.dtype).itemsize)),
+ )
diff --git a/scarf/storage/layout.py b/scarf/storage/layout.py
new file mode 100644
index 00000000..997bc755
--- /dev/null
+++ b/scarf/storage/layout.py
@@ -0,0 +1,372 @@
+from collections.abc import Iterator
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+import zarr
+from zarr.codecs import BloscCodec, ZstdCodec
+
+from .geometry import array_geometry
+from .partition import contiguous_ranges, row_band
+from .profiles import StorageProfile
+from .types import array_metadata_shards
+
+PROFILE_METADATA_CHUNK = 100_000
+_CODEC_MAX_BYTES = 2_147_483_647
+DEFAULT_TARGET_CHUNK_BYTES = 128 * 1024 * 1024
+DEFAULT_TARGET_SHARD_BYTES = 5 * DEFAULT_TARGET_CHUNK_BYTES
+
+
+def _divisors(value: int) -> tuple[int, ...]:
+ small: list[int] = []
+ large: list[int] = []
+ candidate = 1
+ while candidate * candidate <= value:
+ if value % candidate == 0:
+ small.append(candidate)
+ paired = value // candidate
+ if paired != candidate:
+ large.append(paired)
+ candidate += 1
+ return tuple(small + large[::-1])
+
+
+def _aligned_count_layout(
+ nCells: int,
+ nFeats: int,
+ itemsize: int,
+ *,
+ targetChunkBytes: int,
+ targetShardBytes: int,
+) -> tuple[tuple[int, int], tuple[int, int]]:
+ n_cells = max(1, int(nCells))
+ n_feats = max(1, int(nFeats))
+ stored_bytes = max(1, int(itemsize))
+ chunk_target = max(stored_bytes, int(targetChunkBytes))
+ shard_target = max(stored_bytes, int(targetShardBytes))
+ inner_limit = min(chunk_target, _CODEC_MAX_BYTES)
+
+ candidates = tuple(
+ divisor
+ for divisor in _divisors(n_feats)
+ if (n_feats // divisor) * stored_bytes <= inner_limit
+ )
+ if not candidates:
+ raise ValueError(
+ f"A single stored value requires {stored_bytes} bytes, exceeding "
+ f"the codec input limit of {inner_limit} bytes"
+ )
+ feature_chunks = min(
+ candidates,
+ key=lambda divisor: (
+ abs(divisor * chunk_target - shard_target),
+ divisor,
+ ),
+ )
+ feature_chunk = n_feats // feature_chunks
+ row_bytes = n_feats * stored_bytes
+ inner_row_bytes = feature_chunk * stored_bytes
+ shard_rows = max(
+ 1,
+ min(
+ n_cells,
+ inner_limit // inner_row_bytes,
+ shard_target // row_bytes,
+ ),
+ )
+ return (shard_rows, feature_chunk), (shard_rows, n_feats)
+
+
+@dataclass(frozen=True, slots=True)
+class ZarrArraySpec:
+ """Specification for creating a numeric Zarr array."""
+
+ shape: tuple[int, ...]
+ chunks: tuple[int, ...]
+ dtype: Any
+ compressors: tuple[Any, ...]
+ shards: tuple[int, ...] | None = None
+ fillValue: Any | None = None
+ overwrite: bool = True
+
+
+def get_compressors(
+ profile: StorageProfile,
+ zarrFormat: int = 3,
+) -> tuple[Any, ...]:
+ """Return codecs for a storage profile and Zarr format."""
+ if zarrFormat == 2:
+ from numcodecs import Blosc
+
+ return (Blosc(cname="lz4", clevel=5, shuffle=Blosc.BITSHUFFLE),)
+ if profile == "cloud":
+ return (ZstdCodec(level=3),)
+ return (BloscCodec(cname="lz4", clevel=5, shuffle="bitshuffle"),)
+
+
+def _group_zarr_format(group: zarr.Group) -> int:
+ metadata = getattr(group, "metadata", None)
+ if metadata is not None and getattr(metadata, "zarr_format", None) is not None:
+ return int(metadata.zarr_format)
+ return 3
+
+
+def normalize_chunks(
+ chunks: tuple[int, ...] | int,
+ shape: tuple[int, ...],
+) -> tuple[int, ...]:
+ """Map a chunk specification to the dimensions in ``shape``."""
+
+ def bounded_chunk(chunk: int, dimension: int) -> int:
+ return max(1, min(max(1, int(chunk)), max(1, int(dimension))))
+
+ if isinstance(chunks, int):
+ chunks = (chunks,)
+ if len(chunks) == len(shape):
+ return tuple(
+ bounded_chunk(chunk, dimension)
+ for chunk, dimension in zip(chunks, shape, strict=True)
+ )
+ if len(chunks) == 1 and len(shape) > 1:
+ return (bounded_chunk(chunks[0], shape[0]),) + tuple(
+ max(1, int(dimension)) for dimension in shape[1:]
+ )
+ if len(chunks) == 1:
+ return (bounded_chunk(chunks[0], shape[0]),)
+ raise ValueError(
+ f"Cannot map chunks {chunks} to array shape {shape}. "
+ "Provide one chunk size per dimension."
+ )
+
+
+def count_array_spec(
+ nCells: int,
+ nFeats: int,
+ dtype: Any = "uint32",
+ *,
+ profile: StorageProfile,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+ zarrFormat: int = 3,
+) -> ZarrArraySpec:
+ """Build an array specification for an assay count matrix."""
+ itemsize = int(np.dtype(dtype).itemsize)
+ chunk_target = (
+ DEFAULT_TARGET_CHUNK_BYTES
+ if targetChunkBytes is None
+ else int(targetChunkBytes)
+ )
+ shard_target = (
+ DEFAULT_TARGET_SHARD_BYTES
+ if targetShardBytes is None
+ else int(targetShardBytes)
+ )
+ if chunk_target <= 0 or shard_target <= 0:
+ raise ValueError("Chunk and shard targets must be positive")
+ chunks, shards = _aligned_count_layout(
+ nCells,
+ nFeats,
+ itemsize,
+ targetChunkBytes=chunk_target,
+ targetShardBytes=shard_target,
+ )
+ return ZarrArraySpec(
+ shape=(nCells, nFeats),
+ chunks=chunks,
+ shards=shards if zarrFormat >= 3 else None,
+ dtype=dtype,
+ compressors=get_compressors(profile, zarrFormat=zarrFormat),
+ fillValue=0,
+ )
+
+
+def normed_array_spec(
+ nCells: int,
+ nFeats: int,
+ *,
+ profile: StorageProfile,
+ targetChunkBytes: int | None = None,
+ zarrFormat: int = 3,
+) -> ZarrArraySpec:
+ """Build an array specification for normalized expression data."""
+ n_cells = max(1, int(nCells))
+ n_feats = max(1, int(nFeats))
+ row_bytes = n_feats * np.dtype("float32").itemsize
+ if row_bytes > _CODEC_MAX_BYTES:
+ raise ValueError(
+ f"One full-width row requires {row_bytes} bytes, exceeding "
+ f"the codec input limit of {_CODEC_MAX_BYTES} bytes"
+ )
+ requested_target = (
+ DEFAULT_TARGET_CHUNK_BYTES
+ if targetChunkBytes is None
+ else int(targetChunkBytes)
+ )
+ if requested_target <= 0:
+ raise ValueError("Chunk target must be positive")
+ target = min(requested_target, _CODEC_MAX_BYTES)
+ row_chunk = max(1, min(n_cells, target // row_bytes))
+ return ZarrArraySpec(
+ shape=(nCells, nFeats),
+ chunks=(row_chunk, n_feats),
+ shards=None,
+ dtype="float32",
+ compressors=get_compressors(profile, zarrFormat=zarrFormat),
+ fillValue=0.0,
+ )
+
+
+def row_sharded_array_spec(
+ shape: tuple[int, ...],
+ dtype: Any,
+ *,
+ profile: StorageProfile,
+ band_rows: int,
+ target_chunk_bytes: int | None = None,
+ zarr_format: int = 3,
+ fill_value: Any | None = 0,
+) -> ZarrArraySpec:
+ """Build a full-width array specification sharded along the first axis."""
+ if not shape:
+ raise ValueError("Row-sharded arrays require at least one dimension")
+ dimensions = tuple(int(value) for value in shape)
+ if any(value < 0 for value in dimensions):
+ raise ValueError("Array dimensions cannot be negative")
+ if int(band_rows) < 1:
+ raise ValueError("band_rows must be positive")
+ trailing = dimensions[1:]
+ trailing_values = int(np.prod(trailing, dtype=np.int64)) if trailing else 1
+ row_bytes = trailing_values * int(np.dtype(dtype).itemsize)
+ if row_bytes < 1 or row_bytes > _CODEC_MAX_BYTES:
+ raise ValueError(
+ f"One full-width row requires {row_bytes} bytes, exceeding "
+ f"the codec input limit of {_CODEC_MAX_BYTES} bytes"
+ )
+ requested_target = (
+ DEFAULT_TARGET_CHUNK_BYTES
+ if target_chunk_bytes is None
+ else int(target_chunk_bytes)
+ )
+ if requested_target < 1:
+ raise ValueError("Chunk target must be positive")
+ n_rows = max(1, dimensions[0])
+ shard_rows = min(n_rows, int(band_rows))
+ codec_rows = max(1, _CODEC_MAX_BYTES // row_bytes)
+ target_rows = max(
+ 1,
+ min(
+ shard_rows,
+ requested_target // row_bytes,
+ codec_rows,
+ ),
+ )
+ if zarr_format >= 3:
+ candidates = tuple(
+ value for value in _divisors(shard_rows) if value <= codec_rows
+ )
+ chunk_rows = min(
+ candidates,
+ key=lambda value: (abs(value - target_rows), value > target_rows),
+ default=1,
+ )
+ if chunk_rows == 1 and target_rows > 1:
+ shard_rows = target_rows
+ chunk_rows = target_rows
+ else:
+ chunk_rows = target_rows
+ shard_shape = (shard_rows, *tuple(max(1, value) for value in trailing))
+ chunk_shape = (
+ chunk_rows,
+ *tuple(max(1, value) for value in trailing),
+ )
+ return ZarrArraySpec(
+ shape=dimensions,
+ chunks=chunk_shape,
+ shards=shard_shape if zarr_format >= 3 else None,
+ dtype=dtype,
+ compressors=get_compressors(profile, zarrFormat=zarr_format),
+ fillValue=fill_value,
+ )
+
+
+def bounded_row_sharded_array_spec(
+ shape: tuple[int, ...],
+ dtype: Any,
+ *,
+ profile: StorageProfile,
+ target_chunk_bytes: int | None = None,
+ target_shard_bytes: int | None = None,
+ zarr_format: int = 3,
+ fill_value: Any | None = 0,
+) -> ZarrArraySpec:
+ """Build a full-width array with byte-bounded row shards."""
+ if not shape:
+ raise ValueError("Row-sharded arrays require at least one dimension")
+ dimensions = tuple(int(value) for value in shape)
+ if any(value < 0 for value in dimensions):
+ raise ValueError("Array dimensions cannot be negative")
+ trailing = dimensions[1:]
+ trailing_values = int(np.prod(trailing, dtype=np.int64)) if trailing else 1
+ row_bytes = trailing_values * int(np.dtype(dtype).itemsize)
+ chunk_target = (
+ DEFAULT_TARGET_CHUNK_BYTES
+ if target_chunk_bytes is None
+ else int(target_chunk_bytes)
+ )
+ shard_target = (
+ DEFAULT_TARGET_SHARD_BYTES
+ if target_shard_bytes is None
+ else int(target_shard_bytes)
+ )
+ if chunk_target < 1 or shard_target < 1:
+ raise ValueError("Chunk and shard targets must be positive")
+ n_rows = max(1, dimensions[0])
+ codec_rows = max(1, _CODEC_MAX_BYTES // max(1, row_bytes))
+ chunk_rows = max(
+ 1,
+ min(
+ n_rows,
+ chunk_target // max(1, row_bytes),
+ codec_rows,
+ ),
+ )
+ chunks_per_shard = max(
+ 1,
+ shard_target // max(1, chunk_rows * row_bytes),
+ )
+ band_rows = min(n_rows, chunk_rows * chunks_per_shard)
+ return row_sharded_array_spec(
+ dimensions,
+ dtype,
+ profile=profile,
+ band_rows=band_rows,
+ target_chunk_bytes=chunk_target,
+ zarr_format=zarr_format,
+ fill_value=fill_value,
+ )
+
+
+def array_shard_rows(array: zarr.Array) -> int:
+ """Return the row extent of a shard, chunk, or full array."""
+ return row_band(
+ array_geometry(array),
+ unit="shard",
+ fallback=max(int(array.shape[0]), 1),
+ )
+
+
+def iter_shard_row_slices(
+ n_rows: int,
+ shard_rows: int,
+) -> Iterator[tuple[int, int]]:
+ """Yield row slices aligned to a shard height."""
+ yield from contiguous_ranges(n_rows, shard_rows)
+
+
+def array_info(array: zarr.Array) -> str:
+ """Return a compact description of a Zarr array."""
+ parts = [f"shape={array.shape}", f"dtype={array.dtype}", f"chunks={array.chunks}"]
+ shards_meta = array_metadata_shards(array)
+ if shards_meta is not None:
+ parts.append(f"shards={shards_meta}")
+ return ", ".join(parts)
diff --git a/scarf/storage/materialize.py b/scarf/storage/materialize.py
new file mode 100644
index 00000000..450a8d18
--- /dev/null
+++ b/scarf/storage/materialize.py
@@ -0,0 +1,135 @@
+from typing import Any
+
+import numpy as np
+import zarr
+
+from ..utils.compute import controlled_compute
+from .arrays import create_numeric_array, create_zarr_dataset
+from .budget import ResourceBudget
+from .layout import normed_array_spec
+from .profiles import resolve_storage_profile
+from .sharding import write_dense_in_shard_rows
+
+
+def _feature_summary(block: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
+ return (
+ np.asarray(np.sum(block, axis=0, dtype=np.float64)),
+ np.asarray(
+ np.einsum(
+ "ij,ij->j",
+ block,
+ block,
+ dtype=np.float64,
+ optimize=True,
+ )
+ ),
+ )
+
+
+def _merge_feature_summaries(
+ accumulated: tuple[np.ndarray, np.ndarray],
+ current: tuple[np.ndarray, np.ndarray],
+) -> tuple[np.ndarray, np.ndarray]:
+ accumulated[0][...] += current[0]
+ accumulated[1][...] += current[1]
+ return accumulated
+
+
+def _write_feature_summaries(
+ group: zarr.Group | None,
+ summary: tuple[np.ndarray, np.ndarray] | None,
+) -> None:
+ if group is None or summary is None:
+ return
+ for name, values in zip(
+ ("feature_sum", "feature_squared_sum"),
+ summary,
+ strict=True,
+ ):
+ output = create_zarr_dataset(
+ group,
+ name,
+ (100_000,),
+ np.float64,
+ values.shape,
+ )
+ output[:] = values
+
+
+def write_renorm_subset_to_zarr(
+ assay: Any,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ root: zarr.Group,
+ loc: str,
+ nthreads: int,
+ log_transform: bool = False,
+ msg: str | None = None,
+ mirror: zarr.Array | None = None,
+ stats_group: zarr.Group | None = None,
+) -> None:
+ counts = assay.rawData[:, feat_idx][cell_idx, :]
+ if msg is None:
+ msg = f"Writing data to {loc}"
+ spec = normed_array_spec(
+ counts.shape[0],
+ counts.shape[1],
+ profile=resolve_storage_profile(root.store),
+ )
+ output = create_numeric_array(root, loc, spec)
+ scale_factor = assay.sf
+
+ def normalize_block(block: Any) -> np.ndarray:
+ block = np.asarray(block)
+ row_sum = block.sum(axis=1)
+ row_sum[row_sum == 0] = 1
+ normalized = scale_factor * block / row_sum[:, np.newaxis]
+ if log_transform:
+ normalized = np.log1p(normalized)
+ return np.asarray(normalized, dtype=np.float32)
+
+ summary = write_dense_in_shard_rows(
+ output,
+ lambda start, end: normalize_block(
+ controlled_compute(counts[start:end, :], nthreads)
+ ),
+ msg=msg,
+ also_write_to=mirror,
+ resources=assay.resources,
+ summarize=_feature_summary if stats_group is not None else None,
+ merge_summary=(_merge_feature_summaries if stats_group is not None else None),
+ )
+ _write_feature_summaries(stats_group, summary)
+
+
+def dask_to_zarr(
+ data: Any,
+ root: zarr.Group,
+ loc: str,
+ nthreads: int,
+ msg: str | None = None,
+ mirror: zarr.Array | None = None,
+ resources: ResourceBudget | None = None,
+ stats_group: zarr.Group | None = None,
+) -> None:
+ if msg is None:
+ msg = f"Writing data to {loc}"
+ spec = normed_array_spec(
+ data.shape[0],
+ data.shape[1],
+ profile=resolve_storage_profile(root.store),
+ )
+ output = create_numeric_array(root, loc, spec)
+ summary = write_dense_in_shard_rows(
+ output,
+ lambda start, end: controlled_compute(
+ data[start:end, :],
+ nthreads,
+ ).astype(np.float32, copy=False),
+ msg=msg,
+ also_write_to=mirror,
+ resources=resources,
+ summarize=_feature_summary if stats_group is not None else None,
+ merge_summary=(_merge_feature_summaries if stats_group is not None else None),
+ )
+ _write_feature_summaries(stats_group, summary)
diff --git a/scarf/storage/parallel.py b/scarf/storage/parallel.py
new file mode 100644
index 00000000..621cbf30
--- /dev/null
+++ b/scarf/storage/parallel.py
@@ -0,0 +1,187 @@
+import threading
+from collections import deque
+from collections.abc import Callable, Iterable, Iterator
+from concurrent.futures import Future, ThreadPoolExecutor
+from contextlib import contextmanager, nullcontext
+from typing import Any, Literal
+
+from threadpoolctl import threadpool_limits
+
+__all__ = ["map_shards", "stream_shards", "in_shard_context"]
+
+type Backend = Literal["thread", "serial"]
+type RangeProduce = Callable[[int, int, int], Any]
+
+_shard_ctx = threading.local()
+
+
+def in_shard_context() -> bool:
+ """Return whether the caller is already inside a shard worker."""
+ return bool(getattr(_shard_ctx, "active", False))
+
+
+@contextmanager
+def _shard_context() -> Iterator[None]:
+ previous = getattr(_shard_ctx, "active", False)
+ _shard_ctx.active = True
+ try:
+ yield
+ finally:
+ _shard_ctx.active = previous
+
+
+@contextmanager
+def _io_concurrency(io: int | None) -> Iterator[None]:
+ if io is None:
+ yield
+ return
+ import zarr
+
+ with zarr.config.set({"async.concurrency": max(1, int(io))}):
+ yield
+
+
+def _blas_limit(within: int | None) -> Any:
+ if within is None or within < 1:
+ return nullcontext()
+ return threadpool_limits(limits=within)
+
+
+def _close_iterator(iterator: Iterator[Any]) -> None:
+ close = getattr(iterator, "close", None)
+ if callable(close):
+ close()
+
+
+def _imap_ordered(
+ items: Iterable[Any],
+ fn: Callable[[Any], Any],
+ *,
+ workers: int,
+ within_block_threads: int | None,
+) -> Iterator[Any]:
+ def worker(item: Any) -> Any:
+ _shard_ctx.active = True
+ return fn(item)
+
+ iterator = iter(items)
+ with _blas_limit(within_block_threads):
+ executor = ThreadPoolExecutor(max_workers=workers)
+ pending: deque[Future[Any]] = deque()
+
+ def enqueue() -> bool:
+ try:
+ item = next(iterator)
+ except StopIteration:
+ return False
+ pending.append(executor.submit(worker, item))
+ return True
+
+ try:
+ for _ in range(workers):
+ if not enqueue():
+ break
+ while pending:
+ result = pending.popleft().result()
+ enqueue()
+ yield result
+ finally:
+ for future in pending:
+ future.cancel()
+ executor.shutdown(wait=True, cancel_futures=True)
+ _close_iterator(iterator)
+
+
+def _resolve_plan(
+ workers: int,
+ n_shards: int,
+ backend: Backend,
+) -> tuple[int, int, int]:
+ if backend == "serial" or in_shard_context():
+ return 1, 1, 1
+ outer_workers = min(max(1, int(workers)), max(1, int(n_shards)))
+ return outer_workers, 1, 1
+
+
+def _progress(
+ iterator: Iterator[Any],
+ msg: str | None,
+ total: int | None,
+) -> Iterator[Any]:
+ if msg is None:
+ yield from iterator
+ return
+ from ..utils.progress import iter_progress
+
+ yield from iter_progress(iterator, desc=msg, total=total)
+
+
+def stream_shards(
+ items: Iterable[Any],
+ fn: Callable[[Any], Any],
+ *,
+ workers: int,
+ within_block_threads: int | None = None,
+ io_concurrency: int | None = None,
+ msg: str | None = None,
+ total: int | None = None,
+ backend: Backend = "thread",
+) -> Iterator[Any]:
+ """Yield transformed items in order with bounded read-ahead."""
+ workers = max(1, int(workers))
+ if backend == "serial" or workers <= 1 or in_shard_context():
+ iterator = iter(items)
+ with _io_concurrency(io_concurrency), _blas_limit(within_block_threads):
+ try:
+ yield from _progress((fn(item) for item in iterator), msg, total)
+ finally:
+ _close_iterator(iterator)
+ return
+ with _io_concurrency(io_concurrency):
+ base = _imap_ordered(
+ items,
+ fn,
+ workers=workers,
+ within_block_threads=within_block_threads,
+ )
+ yield from _progress(base, msg, total)
+
+
+def map_shards(
+ ranges: list[tuple[int, int]],
+ produce: RangeProduce,
+ *,
+ workers: int,
+ msg: str | None = None,
+ backend: Backend = "thread",
+) -> list[Any]:
+ """Map row ranges in parallel while preserving input order."""
+ n_ranges = len(ranges)
+ if n_ranges == 0:
+ return []
+ worker_count, io_concurrency, within_block_threads = _resolve_plan(
+ workers,
+ n_ranges,
+ backend,
+ )
+ indexed = list(enumerate(ranges))
+
+ def call(item: tuple[int, tuple[int, int]]) -> Any:
+ index, (start, end) = item
+ return produce(index, start, end)
+
+ if worker_count <= 1:
+ with _io_concurrency(io_concurrency), _blas_limit(within_block_threads):
+ return list(_progress((call(item) for item in indexed), msg, n_ranges))
+
+ with (
+ _shard_context(),
+ _io_concurrency(io_concurrency),
+ ):
+ stream = _imap_ordered(
+ indexed,
+ call,
+ workers=worker_count,
+ within_block_threads=within_block_threads,
+ )
+ return list(_progress(stream, msg, n_ranges))
diff --git a/scarf/storage/partition.py b/scarf/storage/partition.py
new file mode 100644
index 00000000..71f04798
--- /dev/null
+++ b/scarf/storage/partition.py
@@ -0,0 +1,214 @@
+"""Row bands and chunk-aligned index blocks for streamed arrays."""
+
+import operator
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass
+from typing import Literal
+
+import numpy as np
+
+from .geometry import ArrayGeometry
+
+type Fits = Callable[[int], bool]
+
+__all__ = [
+ "IndexBlock",
+ "affordable_width",
+ "checked_indices",
+ "contiguous_ranges",
+ "partition_indices",
+ "row_band",
+]
+
+
+@dataclass(frozen=True, slots=True)
+class IndexBlock:
+ """Selected indices that share physical chunks, and where they belong."""
+
+ indices: np.ndarray
+ destinations: np.ndarray
+ bins: tuple[int, ...]
+
+
+def row_band(
+ geometry: ArrayGeometry | None,
+ *,
+ unit: Literal["shard", "chunk"] = "shard",
+ fallback: int,
+) -> int:
+ """Return rows per block from stored geometry, or ``fallback`` without it.
+
+ ``unit`` selects the stored extent to follow: ``shard`` for the object a
+ write lands in, ``chunk`` for the unit a read decodes.
+ """
+ if geometry is None:
+ return max(1, int(fallback))
+ extent = geometry.axisShard(0) if unit == "shard" else geometry.axisChunk(0)
+ return max(1, extent)
+
+
+def contiguous_ranges(nRows: int, band: int) -> list[tuple[int, int]]:
+ """Split a row axis into consecutive half-open ranges of ``band`` rows."""
+ rows = max(0, int(nRows))
+ step = max(1, int(band))
+ return [(start, min(start + step, rows)) for start in range(0, rows, step)]
+
+
+def affordable_width(fits: Fits, maxWidth: int) -> int:
+ """Return the largest width up to ``maxWidth`` that ``fits`` accepts."""
+ limit = max(0, int(maxWidth))
+ if limit == 0 or not fits(1):
+ return 0
+ low = 1
+ high = limit
+ while low < high:
+ candidate = (low + high + 1) // 2
+ if fits(candidate):
+ low = candidate
+ else:
+ high = candidate - 1
+ return low
+
+
+def checked_indices(
+ values: Sequence[int] | np.ndarray,
+ *,
+ limit: int,
+ name: str,
+) -> np.ndarray:
+ """Validate a one-dimensional selection of distinct in-range indices."""
+ indexes = np.asarray(values)
+ if indexes.ndim != 1:
+ raise ValueError(f"{name} must be one-dimensional")
+ if indexes.size == 0:
+ return np.empty(0, dtype=np.int64)
+ if not np.issubdtype(indexes.dtype, np.integer):
+ raise TypeError(f"{name} must contain integers")
+ indexes = indexes.astype(np.int64, copy=False)
+ if np.any(indexes < 0) or np.any(indexes >= limit):
+ raise IndexError(f"{name} contains an out-of-range index")
+ if np.unique(indexes).size != indexes.size:
+ raise ValueError(f"{name} cannot contain duplicate indexes")
+ return indexes
+
+
+def _bins(geometry: ArrayGeometry, axis: int, indices: np.ndarray) -> tuple[int, ...]:
+ return tuple(int(value) for value in np.unique(geometry.binOf(axis, indices)))
+
+
+def _bin_groups(
+ geometry: ArrayGeometry,
+ axis: int,
+ indices: np.ndarray,
+) -> list[tuple[int, np.ndarray]]:
+ """Group index positions by chunk, ascending, keeping request order inside."""
+ bins = geometry.binOf(axis, indices)
+ order = np.argsort(bins, kind="stable")
+ ordered = bins[order]
+ edges = np.concatenate(
+ (
+ np.array([0], dtype=np.int64),
+ (np.flatnonzero(np.diff(ordered)) + 1).astype(np.int64),
+ np.array([order.size], dtype=np.int64),
+ )
+ )
+ return [
+ (int(ordered[edges[position]]), order[edges[position] : edges[position + 1]])
+ for position in range(edges.size - 1)
+ ]
+
+
+def partition_indices(
+ geometry: ArrayGeometry,
+ axis: int,
+ indices: Sequence[int] | np.ndarray,
+ *,
+ maxWidth: int | None = None,
+ fits: Fits | None = None,
+) -> list[IndexBlock]:
+ """Group selected indices into blocks aligned to physical chunks.
+
+ With ``maxWidth`` the selection is cut into fixed-width blocks in request
+ order. With ``fits`` adjacent chunks are packed while the predicate accepts
+ the combined width, and a chunk too wide on its own is split. With neither,
+ every chunk becomes one block.
+ """
+ if maxWidth is not None and fits is not None:
+ raise ValueError("Pass either maxWidth or fits, not both")
+ resolved = checked_indices(indices, limit=geometry.shape[axis], name="indices")
+ if resolved.size == 0:
+ return []
+ positions = np.arange(resolved.size, dtype=np.int64)
+
+ if maxWidth is not None:
+ width = max(1, operator.index(maxWidth))
+ return [
+ IndexBlock(
+ indices=resolved[start : start + width],
+ destinations=positions[start : start + width],
+ bins=_bins(geometry, axis, resolved[start : start + width]),
+ )
+ for start in range(0, resolved.size, width)
+ ]
+
+ groups = _bin_groups(geometry, axis, resolved)
+ if fits is None:
+ return [
+ IndexBlock(
+ indices=resolved[member],
+ destinations=positions[member],
+ bins=(value,),
+ )
+ for value, member in groups
+ ]
+
+ blocks: list[IndexBlock] = []
+ pending: list[np.ndarray] = []
+ pending_bins: list[int] = []
+ pending_width = 0
+ previous_bin: int | None = None
+
+ def flush() -> None:
+ nonlocal pending_width, previous_bin
+ if not pending:
+ return
+ member = np.concatenate(pending)
+ blocks.append(
+ IndexBlock(
+ indices=resolved[member],
+ destinations=positions[member],
+ bins=tuple(pending_bins),
+ )
+ )
+ pending.clear()
+ pending_bins.clear()
+ pending_width = 0
+ previous_bin = None
+
+ for value, member in groups:
+ if not fits(member.size):
+ flush()
+ width = affordable_width(fits, member.size)
+ if width < 1:
+ raise MemoryError(
+ "One index of the selection does not fit the operation budget"
+ )
+ for start in range(0, member.size, width):
+ piece = member[start : start + width]
+ blocks.append(
+ IndexBlock(
+ indices=resolved[piece],
+ destinations=positions[piece],
+ bins=(value,),
+ )
+ )
+ continue
+ adjacent = previous_bin is None or value == previous_bin + 1
+ if pending and (not adjacent or not fits(pending_width + member.size)):
+ flush()
+ pending.append(member)
+ pending_bins.append(value)
+ pending_width += int(member.size)
+ previous_bin = value
+ flush()
+ return blocks
diff --git a/scarf/storage/profiles.py b/scarf/storage/profiles.py
new file mode 100644
index 00000000..5ae3f3d0
--- /dev/null
+++ b/scarf/storage/profiles.py
@@ -0,0 +1,38 @@
+import os
+from typing import Literal
+
+from zarr.abc.store import Store
+
+type StorageProfile = Literal["fast_local", "cloud"]
+type ZarrLocation = str | Store
+
+
+def is_remote_zarr_location(location: str) -> bool:
+ """Return whether a location is a non-local URI."""
+ if "://" not in location:
+ return False
+ return not location.startswith("file://")
+
+
+def is_local_zarr_path(location: ZarrLocation) -> bool:
+ """Return whether a location is a plain local filesystem path."""
+ return isinstance(location, str) and not is_remote_zarr_location(location)
+
+
+def resolve_storage_profile(
+ location: ZarrLocation,
+ requested: StorageProfile | None = None,
+) -> StorageProfile:
+ """Resolve a storage profile without changing process state."""
+ if requested is not None:
+ return requested
+ env_profile = os.environ.get("SCARF_ZARR_PROFILE")
+ if env_profile == "cloud":
+ return "cloud"
+ if env_profile == "fast_local":
+ return "fast_local"
+ if isinstance(location, str):
+ return "cloud" if is_remote_zarr_location(location) else "fast_local"
+ if type(location).__name__ in {"LocalStore", "MemoryStore"}:
+ return "fast_local"
+ return "cloud"
diff --git a/scarf/storage/refs.py b/scarf/storage/refs.py
new file mode 100644
index 00000000..a84ffad7
--- /dev/null
+++ b/scarf/storage/refs.py
@@ -0,0 +1,204 @@
+import re
+from collections.abc import Mapping
+from dataclasses import dataclass
+from typing import Any, Literal
+
+type ArtifactScope = Literal["assay", "datastore"]
+
+_NAME_PATTERN = re.compile(r"^[a-z][a-z0-9_]*$")
+_ARTIFACT_ID_PATTERN = re.compile(r"^[0-9a-f]{64}$")
+
+ARTIFACT_KINDS = frozenset(
+ {
+ "ann_index",
+ "batch_correction",
+ "cell_cycle",
+ "cell_selection",
+ "cluster_cut",
+ "cluster_hierarchy",
+ "cluster_labels",
+ "coalesced_tree",
+ "connectivity_map",
+ "dendrogram",
+ "diffusion_operator",
+ "doublet_score",
+ "embedding",
+ "embedding_initialization",
+ "enrichment_scores",
+ "fate_map",
+ "feature_scaling",
+ "feature_selection",
+ "feature_summary",
+ "hto_identity",
+ "integrated_graph",
+ "imported_coordinates",
+ "mapping_reference",
+ "marker_table",
+ "membership_strength",
+ "metadata_snapshot",
+ "neighbors",
+ "normalized",
+ "projection",
+ "pseudotime",
+ "pseudotime_aggregation",
+ "pseudotime_markers",
+ "quality_metric",
+ "reduction",
+ "sampling",
+ "smart_label",
+ "wnn_coordinates",
+ }
+)
+
+
+def _validate_name(value: str, label: str) -> None:
+ if _NAME_PATTERN.fullmatch(value) is None:
+ raise ValueError(f"{label} must be a snake_case identifier, got {value!r}")
+
+
+def _validate_artifact_kind(kind: str) -> None:
+ _validate_name(kind, "kind")
+ if kind not in ARTIFACT_KINDS:
+ raise ValueError(f"Unknown artifact kind: {kind!r}")
+
+
+@dataclass(frozen=True, slots=True)
+class ArtifactRef:
+ scope: ArtifactScope
+ kind: str
+ artifact_id: str
+ assay: str | None = None
+
+ def __post_init__(self) -> None:
+ if self.scope not in {"assay", "datastore"}:
+ raise ValueError(f"Invalid artifact scope: {self.scope!r}")
+ _validate_artifact_kind(self.kind)
+ if _ARTIFACT_ID_PATTERN.fullmatch(self.artifact_id) is None:
+ raise ValueError("artifact_id must be a 64-character lowercase hex token")
+ if self.scope == "assay":
+ if self.assay is None or not self.assay or "/" in self.assay:
+ raise ValueError("assay-scoped artifact references require an assay")
+ elif self.assay is not None:
+ raise ValueError("datastore-scoped artifact references cannot set assay")
+ if self.kind == "imported_coordinates" and self.scope != "assay":
+ raise ValueError("imported_coordinates artifacts must be assay-scoped")
+
+ def __repr__(self) -> str:
+ location = f"assay={self.assay!r}" if self.assay is not None else "datastore"
+ return (
+ f"ArtifactRef({location}, kind={self.kind!r}, "
+ f"artifact_id='{self.artifact_id[:12]}...')"
+ )
+
+ def to_dict(self) -> dict[str, str]:
+ value = {
+ "type": "artifact",
+ "scope": self.scope,
+ "kind": self.kind,
+ "artifact_id": self.artifact_id,
+ }
+ if self.assay is not None:
+ value["assay"] = self.assay
+ return value
+
+ @classmethod
+ def from_dict(cls, value: Mapping[str, Any]) -> "ArtifactRef":
+ if value.get("type") != "artifact":
+ raise ValueError("Artifact reference type must be 'artifact'")
+ scope = value.get("scope")
+ if scope not in {"assay", "datastore"}:
+ raise ValueError(f"Invalid artifact scope: {scope!r}")
+ kind = value.get("kind")
+ artifact_id = value.get("artifact_id")
+ assay = value.get("assay")
+ if not isinstance(kind, str) or not isinstance(artifact_id, str):
+ raise TypeError("Artifact reference kind and artifact_id must be strings")
+ if assay is not None and not isinstance(assay, str):
+ raise TypeError("Artifact reference assay must be a string or null")
+ return cls(
+ scope=scope,
+ assay=assay,
+ kind=kind,
+ artifact_id=artifact_id,
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class ExternalArtifactRef:
+ dataset_fingerprint: str
+ ref: ArtifactRef
+
+ def __post_init__(self) -> None:
+ if not isinstance(self.dataset_fingerprint, str):
+ raise TypeError("dataset_fingerprint must be a string")
+ if not self.dataset_fingerprint:
+ raise ValueError("dataset_fingerprint must be non-empty")
+ if not isinstance(self.ref, ArtifactRef):
+ raise TypeError("ref must be an ArtifactRef")
+ if self.ref.scope != "assay" or self.ref.assay is None:
+ raise ValueError(
+ "External artifact references require an assay-scoped ArtifactRef"
+ )
+
+ def to_dict(self) -> dict[str, Any]:
+ return {
+ "type": "external_artifact",
+ "dataset_fingerprint": self.dataset_fingerprint,
+ "ref": self.ref.to_dict(),
+ }
+
+ @classmethod
+ def from_dict(cls, value: Mapping[str, Any]) -> "ExternalArtifactRef":
+ if not isinstance(value, Mapping):
+ raise TypeError("External artifact reference must be a mapping")
+ expected_keys = {"type", "dataset_fingerprint", "ref"}
+ if set(value) != expected_keys:
+ raise ValueError(
+ "External artifact reference must contain exactly "
+ "'type', 'dataset_fingerprint', and 'ref'"
+ )
+ if value["type"] != "external_artifact":
+ raise ValueError(
+ "External artifact reference type must be 'external_artifact'"
+ )
+ dataset_fingerprint = value["dataset_fingerprint"]
+ if not isinstance(dataset_fingerprint, str):
+ raise TypeError("dataset_fingerprint must be a string")
+ raw_ref = value["ref"]
+ if not isinstance(raw_ref, Mapping):
+ raise TypeError("External artifact ref must be a mapping")
+ if set(raw_ref) != {"type", "scope", "kind", "artifact_id", "assay"}:
+ raise ValueError(
+ "External artifact ref must be a complete assay artifact reference"
+ )
+ return cls(
+ dataset_fingerprint=dataset_fingerprint,
+ ref=ArtifactRef.from_dict(raw_ref),
+ )
+
+
+type ArtifactLocator = ArtifactRef | ExternalArtifactRef
+
+
+def artifact_path(ref: ArtifactRef) -> str:
+ if ref.scope == "assay":
+ return f"{ref.assay}/artifacts/{ref.kind}/{ref.artifact_id}"
+ return f"artifacts/{ref.kind}/{ref.artifact_id}"
+
+
+def parse_artifact_path(path: str) -> ArtifactRef:
+ parts = path.strip("/").split("/")
+ if len(parts) == 3 and parts[0] == "artifacts":
+ return ArtifactRef(
+ scope="datastore",
+ kind=parts[1],
+ artifact_id=parts[2],
+ )
+ if len(parts) == 4 and parts[1] == "artifacts":
+ return ArtifactRef(
+ scope="assay",
+ assay=parts[0],
+ kind=parts[2],
+ artifact_id=parts[3],
+ )
+ raise ValueError(f"Not an artifact path: {path!r}")
diff --git a/scarf/storage/schema.py b/scarf/storage/schema.py
new file mode 100644
index 00000000..02469c37
--- /dev/null
+++ b/scarf/storage/schema.py
@@ -0,0 +1,260 @@
+from typing import Any
+
+import numpy as np
+import zarr
+
+from .types import array_metadata_shards, as_zarr_array
+from .arrays import (
+ create_metadata_column,
+ create_numeric_array,
+ create_zarr_obj_array,
+)
+from .layout import _group_zarr_format, count_array_spec
+from .profiles import StorageProfile, resolve_storage_profile
+
+_ASSAY_NAME_OWNERS = {
+ "artifacts": "datastore artifact storage",
+ "cellData": "datastore cell metadata",
+ "matrices": "workspace matrix storage",
+ "pipeline": "DataStore.pipeline",
+ "plots": "DataStore.plots",
+ "summary": "DataStore.summary",
+}
+RESERVED_ASSAY_NAMES = frozenset(_ASSAY_NAME_OWNERS)
+
+
+def validate_assay_name(assay_name: str) -> None:
+ """Reject invalid assay names and names reserved by the datastore layout."""
+ if not assay_name or not assay_name.strip():
+ raise ValueError("Assay names must be non-empty")
+ if "/" in assay_name or "\\" in assay_name:
+ raise ValueError(f"Assay name {assay_name!r} must not contain path separators")
+ if assay_name in RESERVED_ASSAY_NAMES:
+ owner = _ASSAY_NAME_OWNERS[assay_name]
+ raise ValueError(
+ f"Assay name {assay_name!r} is reserved for {owner}. "
+ "Choose another name, or explicitly migrate an existing assay before "
+ "opening the store with Scarf."
+ )
+
+
+def create_zarr_count_assay(
+ z: zarr.Group,
+ assay_name: str,
+ workspace: str | None,
+ n_cells: int,
+ feat_ids: np.ndarray | list[str],
+ feat_names: np.ndarray | list[str],
+ dtype: str = "uint32",
+ *,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+) -> zarr.Array:
+ validate_assay_name(assay_name)
+ if workspace is None:
+ group = z.create_group(assay_name, overwrite=True)
+ else:
+ group = z.create_group(f"{workspace}/{assay_name}", overwrite=True)
+ group.attrs["is_assay"] = True
+ group.attrs["misc"] = {}
+ resolved_profile = profile or resolve_storage_profile(group.store)
+ create_zarr_obj_array(
+ group,
+ "featureData/ids",
+ feat_ids,
+ profile=resolved_profile,
+ )
+ create_zarr_obj_array(
+ group,
+ "featureData/names",
+ feat_names,
+ profile=resolved_profile,
+ )
+ create_zarr_obj_array(
+ group,
+ "featureData/I",
+ [True for _ in range(len(feat_ids))],
+ "bool",
+ profile=resolved_profile,
+ )
+ if workspace is not None:
+ group = z.create_group(f"matrices/{assay_name}", overwrite=True)
+ n_feats = len(feat_ids)
+ zarr_format = _group_zarr_format(group)
+ spec = count_array_spec(
+ n_cells,
+ n_feats,
+ dtype=dtype,
+ profile=resolved_profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ zarrFormat=zarr_format,
+ )
+ counts = create_numeric_array(group, "counts", spec)
+ stored_shards = array_metadata_shards(counts)
+ group.attrs["scarf:zarr_spec"] = {
+ "profile": resolved_profile,
+ "dtype": np.dtype(dtype).str,
+ "chunks": list(counts.chunks),
+ "shards": None if stored_shards is None else list(stored_shards),
+ "zarr_format": zarr_format,
+ }
+ return counts
+
+
+def create_empty_zarr_count_assay(
+ z: zarr.Group,
+ assay_name: str,
+ workspace: str | None,
+ n_cells: int,
+ n_features: int,
+ feature_id_dtype: Any,
+ feature_name_dtype: Any,
+ dtype: Any = "uint32",
+ *,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+) -> tuple[zarr.Array, zarr.Group]:
+ """Create an assay whose feature metadata can be filled blockwise."""
+ validate_assay_name(assay_name)
+ if n_cells < 0 or n_features < 0:
+ raise ValueError("Assay dimensions must be non-negative")
+ assay_path = assay_name if workspace is None else f"{workspace}/{assay_name}"
+ assay_group = z.create_group(assay_path, overwrite=True)
+ assay_group.attrs["is_assay"] = True
+ assay_group.attrs["misc"] = {}
+ resolved_profile = profile or resolve_storage_profile(assay_group.store)
+ feature_group = assay_group.create_group("featureData")
+ create_metadata_column(
+ feature_group,
+ "ids",
+ dtype=feature_id_dtype,
+ shape=n_features,
+ chunkSize=100_000,
+ profile=resolved_profile,
+ )
+ create_metadata_column(
+ feature_group,
+ "names",
+ dtype=feature_name_dtype,
+ shape=n_features,
+ chunkSize=100_000,
+ profile=resolved_profile,
+ )
+ included = create_metadata_column(
+ feature_group,
+ "I",
+ dtype=bool,
+ shape=n_features,
+ chunkSize=100_000,
+ profile=resolved_profile,
+ )
+ included[:] = True
+
+ matrix_group = (
+ assay_group
+ if workspace is None
+ else z.create_group(f"matrices/{assay_name}", overwrite=True)
+ )
+ zarr_format = _group_zarr_format(matrix_group)
+ spec = count_array_spec(
+ n_cells,
+ n_features,
+ dtype=dtype,
+ profile=resolved_profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ zarrFormat=zarr_format,
+ )
+ counts = create_numeric_array(matrix_group, "counts", spec)
+ stored_shards = array_metadata_shards(counts)
+ matrix_group.attrs["scarf:zarr_spec"] = {
+ "profile": resolved_profile,
+ "dtype": np.dtype(dtype).str,
+ "chunks": list(counts.chunks),
+ "shards": None if stored_shards is None else list(stored_shards),
+ "zarr_format": zarr_format,
+ }
+ return counts, feature_group
+
+
+def load_count_array(
+ root: zarr.Group,
+ assay_name: str,
+ workspace: str | None,
+) -> zarr.Array:
+ if workspace is None:
+ return as_zarr_array(
+ root[f"{assay_name}/counts"],
+ name=f"{assay_name}/counts",
+ )
+ return as_zarr_array(
+ root[f"matrices/{assay_name}/counts"],
+ name=f"matrices/{assay_name}/counts",
+ )
+
+
+def create_cell_data(
+ root: zarr.Group,
+ workspace: str | None,
+ ids: np.ndarray,
+ names: np.ndarray,
+ profile: StorageProfile | None = None,
+) -> zarr.Group:
+ if workspace is None:
+ group = root.create_group("cellData")
+ else:
+ group = root.create_group(f"{workspace}/cellData")
+ create_zarr_obj_array(group, "ids", ids, ids.dtype, profile=profile)
+ create_zarr_obj_array(group, "names", names, names.dtype, profile=profile)
+ create_zarr_obj_array(
+ group,
+ "I",
+ [True for _ in range(len(ids))],
+ "bool",
+ profile=profile,
+ )
+ return group
+
+
+def create_empty_cell_data(
+ root: zarr.Group,
+ workspace: str | None,
+ n_cells: int,
+ id_dtype: Any,
+ name_dtype: Any,
+ profile: StorageProfile | None = None,
+) -> zarr.Group:
+ """Create cell metadata columns that can be filled blockwise."""
+ if n_cells < 0:
+ raise ValueError("n_cells must be non-negative")
+ path = "cellData" if workspace is None else f"{workspace}/cellData"
+ group = root.create_group(path)
+ create_metadata_column(
+ group,
+ "ids",
+ dtype=id_dtype,
+ shape=n_cells,
+ chunkSize=100_000,
+ profile=profile,
+ )
+ create_metadata_column(
+ group,
+ "names",
+ dtype=name_dtype,
+ shape=n_cells,
+ chunkSize=100_000,
+ profile=profile,
+ )
+ included = create_metadata_column(
+ group,
+ "I",
+ dtype=bool,
+ shape=n_cells,
+ chunkSize=100_000,
+ profile=profile,
+ )
+ included[:] = True
+ return group
diff --git a/scarf/storage/selections.py b/scarf/storage/selections.py
new file mode 100644
index 00000000..f091971e
--- /dev/null
+++ b/scarf/storage/selections.py
@@ -0,0 +1,359 @@
+from typing import Any
+
+import numpy as np
+import zarr
+
+from .arrays import create_metadata_column
+from .artifact_writer import (
+ ArrayRequirement,
+ finish_artifact,
+ plan_artifact,
+ reused_artifact_group,
+ start_artifact,
+)
+from .artifacts import (
+ ArtifactRef,
+ ArtifactScope,
+ ValueFingerprintBuilder,
+ fingerprint_array,
+ fingerprint_stored_strings,
+ fingerprint_strings,
+)
+from .geometry import array_geometry
+from .partition import row_band
+from .types import as_zarr_array, as_zarr_group
+
+
+def _stored_selection_fingerprint(array: zarr.Array) -> str:
+ if array.ndim != 1 or np.dtype(array.dtype) != np.dtype(bool):
+ raise TypeError("Stored selection columns must be one-dimensional booleans")
+ builder = ValueFingerprintBuilder()
+ builder.begin_array("values", array.shape, array.dtype)
+ block_rows = row_band(array_geometry(array), unit="chunk", fallback=1)
+ for start in range(0, int(array.shape[0]), block_rows):
+ stop = min(start + block_rows, int(array.shape[0]))
+ builder.update_array_block(
+ "values",
+ (start,),
+ np.asarray(array[start:stop], dtype=bool),
+ )
+ builder.end_array("values")
+ return builder.hexdigest()
+
+
+def fingerprint_selected_stored_strings(
+ ids: zarr.Array,
+ selection: zarr.Array,
+) -> tuple[str, int]:
+ """Fingerprint selected row IDs without materializing the selection."""
+ if ids.ndim != 1 or selection.ndim != 1 or ids.shape != selection.shape:
+ raise ValueError("Stored row IDs and selection must be aligned vectors")
+ if np.dtype(selection.dtype) != np.dtype(bool):
+ raise TypeError("Stored selection values must be booleans")
+ source_dtype = np.dtype(ids.dtype)
+ if source_dtype.kind not in {"O", "S", "U"}:
+ raise TypeError("Stored row IDs must contain strings")
+
+ block_rows = min(
+ row_band(array_geometry(ids), unit="chunk", fallback=1),
+ row_band(array_geometry(selection), unit="chunk", fallback=1),
+ )
+ selected_count = 0
+ if source_dtype.hasobject:
+ max_length = 1
+ for start in range(0, int(ids.shape[0]), block_rows):
+ stop = min(start + block_rows, int(ids.shape[0]))
+ mask = np.asarray(selection[start:stop], dtype=bool)
+ values = np.asarray(ids[start:stop])[mask]
+ selected_count += len(values)
+ if values.size:
+ max_length = max(max_length, max(len(str(value)) for value in values))
+ string_dtype = np.dtype(f"U{max_length}")
+ else:
+ for start in range(0, int(ids.shape[0]), block_rows):
+ stop = min(start + block_rows, int(ids.shape[0]))
+ selected_count += int(
+ np.count_nonzero(np.asarray(selection[start:stop], dtype=bool))
+ )
+ string_dtype = np.empty(0, dtype=source_dtype).astype(str).dtype
+
+ builder = ValueFingerprintBuilder()
+ builder.begin_array("values", (selected_count,), string_dtype)
+ selected_start = 0
+ for start in range(0, int(ids.shape[0]), block_rows):
+ stop = min(start + block_rows, int(ids.shape[0]))
+ mask = np.asarray(selection[start:stop], dtype=bool)
+ values = np.asarray(ids[start:stop])[mask].astype(string_dtype)
+ if values.size:
+ builder.update_array_block("values", (selected_start,), values)
+ selected_start += len(values)
+ builder.end_array("values")
+ return builder.hexdigest(), selected_count
+
+
+def resolve_stored_selection_artifact(
+ root: zarr.Group,
+ *,
+ table_path: str,
+ id_column: str,
+ source_column: str,
+ scope: ArtifactScope,
+ kind: str,
+ operation: str,
+ parameters: dict[str, Any],
+ inputs: dict[str, Any],
+ assay: str | None = None,
+ invalidate_cache: bool = False,
+) -> ArtifactRef:
+ """Create a selection artifact by copying a stored column blockwise."""
+ table = as_zarr_group(root[table_path], name=table_path)
+ source = as_zarr_array(table[source_column], name=source_column)
+ ids = as_zarr_array(table[id_column], name=id_column)
+ if source.ndim != 1 or np.dtype(source.dtype) != np.dtype(bool):
+ raise TypeError("Selection source column must be one-dimensional booleans")
+ if ids.ndim != 1 or ids.shape != source.shape:
+ raise ValueError("Selection row IDs must align with source values")
+ selection_inputs = dict(inputs)
+ selection_inputs.update(
+ {
+ "ordered_row_ids_fingerprint": fingerprint_stored_strings(ids),
+ "values_fingerprint": _stored_selection_fingerprint(source),
+ }
+ )
+ planned = plan_artifact(
+ root,
+ scope=scope,
+ assay=assay,
+ kind=kind,
+ operation=operation,
+ parameters=parameters,
+ inputs=selection_inputs,
+ execution_options={"source_column": source_column},
+ invalidate_cache=invalidate_cache,
+ required_arrays=(ArrayRequirement("values", shape=source.shape, dtype=bool),),
+ )
+ if planned.reused:
+ return planned.ref
+ group = start_artifact(root, planned)
+ output = create_metadata_column(
+ group,
+ "values",
+ dtype=bool,
+ shape=int(source.shape[0]),
+ chunkSize=row_band(array_geometry(source), unit="chunk", fallback=1),
+ overwrite=True,
+ )
+ block_rows = row_band(array_geometry(source), unit="chunk", fallback=1)
+ for start in range(0, int(source.shape[0]), block_rows):
+ stop = min(start + block_rows, int(source.shape[0]))
+ output[start:stop] = source[start:stop]
+ finish_artifact(group, planned)
+ return planned.ref
+
+
+def resolve_selection_artifact(
+ root: zarr.Group,
+ *,
+ scope: ArtifactScope,
+ kind: str,
+ values: np.ndarray,
+ row_ids: np.ndarray,
+ operation: str,
+ parameters: dict[str, Any],
+ inputs: dict[str, Any],
+ source_column: str,
+ assay: str | None = None,
+ invalidate_cache: bool = False,
+) -> ArtifactRef:
+ mask = np.asarray(values)
+ if mask.ndim != 1 or mask.dtype != bool:
+ raise TypeError("Selection values must be a one-dimensional boolean array")
+ rows = np.asarray(row_ids)
+ if rows.ndim != 1 or len(rows) != len(mask):
+ raise ValueError("Selection row IDs must align with selection values")
+ selection_inputs = dict(inputs)
+ selection_inputs.update(
+ {
+ "ordered_row_ids_fingerprint": fingerprint_strings(rows),
+ "values_fingerprint": fingerprint_array(mask),
+ }
+ )
+ planned = plan_artifact(
+ root,
+ scope=scope,
+ assay=assay,
+ kind=kind,
+ operation=operation,
+ parameters=parameters,
+ inputs=selection_inputs,
+ execution_options={"source_column": source_column},
+ invalidate_cache=invalidate_cache,
+ )
+ if planned.reused:
+ group = reused_artifact_group(root, planned)
+ if "values" in group:
+ try:
+ stored_values = np.asarray(
+ as_zarr_array(group["values"], name="values")[:]
+ )
+ except TypeError:
+ pass
+ else:
+ if (
+ stored_values.ndim == 1
+ and stored_values.dtype == np.dtype(bool)
+ and stored_values.shape == mask.shape
+ and np.array_equal(stored_values, mask)
+ ):
+ return planned.ref
+ planned = planned.invalidated(
+ root,
+ required_arrays=(
+ ArrayRequirement(
+ "values",
+ shape=mask.shape,
+ dtype_kind="b",
+ ),
+ ),
+ )
+ group = start_artifact(root, planned)
+ create_metadata_column(
+ group,
+ "values",
+ data=mask,
+ dtype=bool,
+ overwrite=True,
+ )
+ finish_artifact(group, planned)
+ return planned.ref
+
+
+def resolve_generated_selection_artifact(
+ root: zarr.Group,
+ *,
+ scope: ArtifactScope,
+ kind: str,
+ values: np.ndarray,
+ row_ids: np.ndarray,
+ operation: str,
+ parameters: dict[str, Any],
+ inputs: dict[str, Any],
+ source_column: str,
+ assay: str | None = None,
+ invalidate_cache: bool = False,
+) -> tuple[ArtifactRef, np.ndarray]:
+ mask = np.asarray(values)
+ if mask.ndim != 1 or mask.dtype != bool:
+ raise TypeError("Selection values must be a one-dimensional boolean array")
+ rows = np.asarray(row_ids)
+ if rows.ndim != 1 or len(rows) != len(mask):
+ raise ValueError("Selection row IDs must align with selection values")
+ selection_inputs = dict(inputs)
+ selection_inputs["ordered_row_ids_fingerprint"] = fingerprint_strings(rows)
+ planned = plan_artifact(
+ root,
+ scope=scope,
+ assay=assay,
+ kind=kind,
+ operation=operation,
+ parameters=parameters,
+ inputs=selection_inputs,
+ execution_options={"source_column": source_column},
+ invalidate_cache=invalidate_cache,
+ required_arrays=(
+ ArrayRequirement(
+ "values",
+ shape=mask.shape,
+ dtype_kind="b",
+ ),
+ ),
+ )
+ if planned.reused:
+ group = reused_artifact_group(root, planned)
+ stored_values = np.asarray(as_zarr_array(group["values"], name="values")[:])
+ if (
+ stored_values.ndim == 1
+ and stored_values.dtype == np.dtype(bool)
+ and stored_values.shape == mask.shape
+ ):
+ return planned.ref, stored_values
+ planned = planned.invalidated(root)
+ group = start_artifact(root, planned)
+ create_metadata_column(
+ group,
+ "values",
+ data=mask,
+ dtype=bool,
+ overwrite=True,
+ )
+ finish_artifact(group, planned)
+ return planned.ref, mask
+
+
+def resolve_metadata_snapshot(
+ root: zarr.Group,
+ *,
+ values: np.ndarray,
+ row_ids: np.ndarray,
+ operation: str,
+ parameters: dict[str, Any],
+ inputs: dict[str, Any],
+ source_columns: list[str],
+ invalidate_cache: bool = False,
+) -> ArtifactRef:
+ array = np.asarray(values)
+ rows = np.asarray(row_ids)
+ if array.ndim < 1 or rows.ndim != 1 or array.shape[0] != len(rows):
+ raise ValueError("Metadata values must align with row IDs")
+ values_fingerprint = (
+ fingerprint_strings(array.reshape(-1))
+ if array.dtype.kind in {"O", "S", "U"}
+ else fingerprint_array(array)
+ )
+ snapshot_inputs = dict(inputs)
+ snapshot_inputs.update(
+ {
+ "ordered_row_ids_fingerprint": fingerprint_strings(rows),
+ "values_fingerprint": values_fingerprint,
+ }
+ )
+ snapshot_parameters = dict(parameters)
+ snapshot_parameters["shape"] = list(array.shape)
+ planned = plan_artifact(
+ root,
+ scope="datastore",
+ kind="metadata_snapshot",
+ operation=operation,
+ parameters=snapshot_parameters,
+ inputs=snapshot_inputs,
+ execution_options={"source_columns": source_columns},
+ invalidate_cache=invalidate_cache,
+ )
+ flattened = array.reshape(-1)
+ if planned.reused:
+ group = reused_artifact_group(root, planned)
+ if "values" in group:
+ try:
+ stored_values = as_zarr_array(group["values"], name="values")[:]
+ except TypeError:
+ pass
+ else:
+ expected_values = (
+ flattened.astype(str)
+ if flattened.dtype.kind in {"O", "S", "U"}
+ else flattened
+ )
+ if np.array_equal(stored_values, expected_values):
+ return planned.ref
+ planned = planned.invalidated(root)
+ group = start_artifact(root, planned)
+ create_metadata_column(
+ group,
+ "values",
+ data=flattened.astype(str)
+ if flattened.dtype.kind in {"O", "S", "U"}
+ else flattened,
+ overwrite=True,
+ )
+ finish_artifact(group, planned)
+ return planned.ref
diff --git a/scarf/storage/sharding.py b/scarf/storage/sharding.py
new file mode 100644
index 00000000..ed8fb860
--- /dev/null
+++ b/scarf/storage/sharding.py
@@ -0,0 +1,922 @@
+import asyncio
+import contextvars
+import threading
+from collections import deque
+from collections.abc import Callable, Coroutine, Iterator, Sequence
+from dataclasses import dataclass
+from typing import Any
+
+import numpy as np
+import zarr
+
+from .arrays import create_numeric_array
+from .budget import (
+ ResourceBudget,
+ admitted_worker_split,
+ resolve_budget,
+)
+from .layout import (
+ ZarrArraySpec,
+ _group_zarr_format,
+ array_shard_rows,
+ get_compressors,
+ iter_shard_row_slices,
+)
+from .parallel import _close_iterator, stream_shards
+from .partition import affordable_width
+from .profiles import StorageProfile, resolve_storage_profile
+from ..utils.arrays import canonicalize_sparse, checked_sparse_cast
+
+
+def _run_async(factory: Callable[[], Coroutine[Any, Any, None]]) -> None:
+ try:
+ asyncio.get_running_loop()
+ except RuntimeError:
+ asyncio.run(factory())
+ return
+
+ error: list[BaseException] = []
+ context = contextvars.copy_context()
+
+ def run() -> None:
+ try:
+ context.run(lambda: asyncio.run(factory()))
+ except BaseException as exc:
+ error.append(exc)
+
+ thread = threading.Thread(target=run)
+ thread.start()
+ thread.join()
+ if error:
+ raise error[0]
+
+
+@dataclass(frozen=True, slots=True)
+class _DenseWriteBand:
+ start: int
+ end: int
+ values: np.ndarray
+
+
+@dataclass(frozen=True, slots=True)
+class SparseRowBand:
+ start: int
+ end: int
+ nColumns: int
+ row: np.ndarray
+ column: np.ndarray
+ data: np.ndarray
+ dtype: Any
+
+ @property
+ def sparseBytes(self) -> int:
+ return int(self.row.nbytes + self.column.nbytes + self.data.nbytes)
+
+ def dense(self) -> np.ndarray:
+ dense = np.zeros(
+ (self.end - self.start, self.nColumns),
+ dtype=self.dtype,
+ )
+ dense[self.row, self.column] = checked_sparse_cast(self.data, self.dtype)
+ return dense
+
+
+@dataclass(frozen=True, slots=True)
+class SparseWriteBand:
+ destination: zarr.Array
+ band: SparseRowBand
+ producerBytes: int = 0
+
+
+@dataclass(frozen=True, slots=True)
+class SparseImportPlan:
+ batchRows: int
+ producerReserveBytes: int
+ writeTasks: int
+
+
+def sparse_matrix_bytes(*matrices: Any) -> int:
+ """Return unique array bytes owned by SciPy sparse matrices."""
+ arrays = (
+ getattr(matrix, name, None)
+ for matrix in matrices
+ for name in ("data", "row", "col", "indices", "indptr")
+ )
+ unique = {id(array): array for array in arrays if isinstance(array, np.ndarray)}
+ return int(sum(array.nbytes for array in unique.values()))
+
+
+def sparse_producer_peak_bytes(
+ bufferedValues: int,
+ sourceValues: int,
+ valueItemsize: int,
+) -> int:
+ """Bound one sparse pull through projection, buffering, and band splitting."""
+ buffered_values = max(0, int(bufferedValues))
+ source_values = max(0, int(sourceValues))
+ value_bytes = max(1, int(valueItemsize))
+ index_bytes = np.dtype(np.int64).itemsize
+ mask_bytes = np.dtype(np.bool_).itemsize
+ triplet_bytes = value_bytes + 2 * index_bytes
+ buffered_bytes = buffered_values * (
+ 2 * triplet_bytes + index_bytes + 2 * mask_bytes
+ )
+ source_bytes = 2 * source_values * triplet_bytes
+ source_indexes = source_values * (2 * index_bytes + 2 * mask_bytes)
+ canonical_value_bytes = max(value_bytes, np.dtype(np.int64).itemsize)
+ canonicalization_bytes = source_values * (
+ 4 * canonical_value_bytes + 6 * index_bytes + 4
+ )
+ return buffered_bytes + source_bytes + source_indexes + canonicalization_bytes
+
+
+def sparse_write_task_count(
+ destinations: Sequence[zarr.Array],
+ nRows: int,
+) -> int:
+ """Return the exact number of destination row bands an import will write."""
+ rows = max(0, int(nRows))
+ if rows == 0:
+ return 0
+ return int(
+ sum(
+ (rows + array_shard_rows(destination) - 1) // array_shard_rows(destination)
+ for destination in destinations
+ )
+ )
+
+
+def resolve_sparse_import_batch(
+ destinations: Sequence[zarr.Array],
+ *,
+ nRows: int,
+ resources: ResourceBudget,
+ maxWindowNnz: Callable[[int], int],
+ sourceDtype: Any,
+ batchRows: int | None = None,
+ residentBytes: int = 0,
+ producerStagingBytes: Callable[[int], int] | None = None,
+ extraProducerBytes: Callable[[int], int] | None = None,
+) -> SparseImportPlan:
+ """Resolve source rows while admitting one complete destination write band."""
+ destination_list = tuple(destinations)
+ if not destination_list:
+ raise ValueError("At least one sparse import destination is required")
+ rows = int(nRows)
+ if rows < 0:
+ raise ValueError("nRows cannot be negative")
+ for destination in destination_list:
+ if int(destination.shape[0]) != rows:
+ raise ValueError(
+ "Sparse import destinations must match the source row count"
+ )
+ if batchRows is not None and int(batchRows) <= 0:
+ raise ValueError("batch_size must be positive")
+
+ source_dtype = np.dtype(sourceDtype)
+ value_itemsize = max(
+ source_dtype.itemsize,
+ *(np.dtype(destination.dtype).itemsize for destination in destination_list),
+ )
+ staging_bytes = producerStagingBytes or (lambda _: 0)
+ extra_bytes = extraProducerBytes or (lambda _: 0)
+ maximum_shard_rows = max(
+ array_shard_rows(destination) for destination in destination_list
+ )
+ write_tasks = sparse_write_task_count(destination_list, rows)
+
+ band_requirements: list[tuple[zarr.Array, int, int]] = []
+ if rows:
+ for destination in destination_list:
+ band_rows = min(rows, array_shard_rows(destination))
+ band_values = max(0, int(maxWindowNnz(band_rows)))
+ band_sparse_bytes = band_values * (
+ source_dtype.itemsize + 2 * np.dtype(np.int64).itemsize
+ )
+ band_requirements.append((destination, band_values, band_sparse_bytes))
+
+ def reserve(width: int) -> int:
+ source_values = max(0, int(maxWindowNnz(width)))
+ buffered_values = max(
+ 0,
+ int(maxWindowNnz(width + maximum_shard_rows)),
+ )
+ return int(
+ sparse_producer_peak_bytes(
+ buffered_values,
+ source_values,
+ value_itemsize,
+ )
+ + max(0, int(staging_bytes(width)))
+ + max(0, int(extra_bytes(width)))
+ )
+
+ def admit(width: int) -> int:
+ producer_reserve = reserve(width)
+ for destination, band_values, band_sparse_bytes in band_requirements:
+ dense_bytes, inner_bytes, n_chunks = _band_geometry(destination)
+ destination_dtype = np.dtype(destination.dtype)
+ conversion_bytes = (
+ band_values * destination_dtype.itemsize
+ if source_dtype != destination_dtype
+ else 0
+ )
+ if source_dtype.kind == "f" and destination_dtype.kind in "biu":
+ conversion_bytes += band_values * (source_dtype.itemsize + 1)
+ admitted_worker_split(
+ resources,
+ nTasks=1,
+ residentBytes=(
+ max(0, int(residentBytes)) + producer_reserve + band_sparse_bytes
+ ),
+ taskBytes=lambda inner: _row_band_task_peak(
+ sourceBytes=conversion_bytes,
+ denseBytes=dense_bytes,
+ innerChunkBytes=inner_bytes,
+ nChunks=n_chunks,
+ innerConcurrency=inner,
+ ),
+ requested=1,
+ )
+ if not band_requirements:
+ admitted_worker_split(
+ resources,
+ nTasks=1,
+ residentBytes=max(0, int(residentBytes)) + producer_reserve,
+ taskBytes=lambda _: 1,
+ requested=1,
+ )
+ return producer_reserve
+
+ if batchRows is not None:
+ resolved_rows = int(batchRows)
+ producer_reserve = admit(resolved_rows)
+ elif rows == 0:
+ resolved_rows = 1
+ producer_reserve = admit(resolved_rows)
+ else:
+ preferred_rows = min(
+ rows,
+ min(array_shard_rows(destination) for destination in destination_list),
+ )
+
+ def fits(width: int) -> bool:
+ try:
+ admit(width)
+ except MemoryError:
+ return False
+ return True
+
+ resolved_rows = affordable_width(fits, preferred_rows)
+ if resolved_rows < 1:
+ try:
+ admit(1)
+ except MemoryError as exc:
+ raise MemoryError(
+ "Automatic sparse import cannot fit one source row and one "
+ "complete destination row band. Increase mem_budget or use "
+ "smaller target chunk and shard byte limits."
+ ) from exc
+ raise RuntimeError("Sparse import admission failed unexpectedly")
+ producer_reserve = reserve(resolved_rows)
+
+ return SparseImportPlan(
+ batchRows=resolved_rows,
+ producerReserveBytes=producer_reserve,
+ writeTasks=write_tasks,
+ )
+
+
+class SparseShardBuffer:
+ """Accumulate sparse source batches into immutable destination row bands."""
+
+ def __init__(self, destination: zarr.Array) -> None:
+ self.nRows = max(0, int(destination.shape[0]))
+ self.nColumns = max(0, int(destination.shape[1]))
+ self.shardRows = array_shard_rows(destination)
+ self.dtype = np.dtype(destination.dtype)
+ self.rows = 0
+ self._nextFlush = self.shardRows
+ self._rows: list[np.ndarray] = []
+ self._columns: list[np.ndarray] = []
+ self._data: list[np.ndarray] = []
+
+ @property
+ def residentBytes(self) -> int:
+ arrays = (*self._rows, *self._columns, *self._data)
+ return int(sum(array.nbytes for array in arrays))
+
+ def add(self, batch: Any) -> Iterator[SparseRowBand]:
+ from scipy.sparse import coo_matrix
+
+ coo = batch if hasattr(batch, "row") else coo_matrix(batch)
+ if coo.shape[1] != self.nColumns:
+ raise ValueError(
+ f"Sparse batch has {coo.shape[1]} columns, expected {self.nColumns}"
+ )
+ if self.rows + int(coo.shape[0]) > self.nRows:
+ raise ValueError("Sparse batches contain more rows than the destination")
+ if coo.nnz:
+ coo = coo.tocoo(copy=False)
+ if not bool(getattr(coo, "has_canonical_format", False)):
+ coo = canonicalize_sparse(coo, self.dtype)
+ self._rows.append(np.asarray(coo.row, dtype=np.int64) + self.rows)
+ self._columns.append(np.asarray(coo.col, dtype=np.int64))
+ self._data.append(np.asarray(coo.data))
+ self.rows += int(coo.shape[0])
+ while self._nextFlush <= self.rows:
+ end = self._nextFlush
+ start = end - self.shardRows
+ self._nextFlush += self.shardRows
+ yield self._take(start, end)
+
+ def finish(self) -> Iterator[SparseRowBand]:
+ if self.rows != self.nRows:
+ raise ValueError(
+ f"Sparse stream contains {self.rows} rows, expected {self.nRows}"
+ )
+ trailing_start = self._nextFlush - self.shardRows
+ if trailing_start < self.rows:
+ yield self._take(trailing_start, self.rows)
+ elif self._rows or self._columns or self._data:
+ raise RuntimeError("Sparse coordinates remain after the final row band")
+
+ def _take(self, start: int, end: int) -> SparseRowBand:
+ row, column, data = self._drain()
+ selected = row < end
+ keep = ~selected
+ if keep.any():
+ self._rows = [row[keep]]
+ self._columns = [column[keep]]
+ self._data = [data[keep]]
+ local_rows = row[selected] - start
+ return SparseRowBand(
+ start=start,
+ end=end,
+ nColumns=self.nColumns,
+ row=local_rows,
+ column=column[selected],
+ data=data[selected],
+ dtype=self.dtype,
+ )
+
+ def _drain(self) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ def joined(parts: list[np.ndarray], dtype: Any) -> np.ndarray:
+ if not parts:
+ return np.array([], dtype=dtype)
+ return np.concatenate(parts)
+
+ values = (
+ joined(self._rows, np.int64),
+ joined(self._columns, np.int64),
+ joined(self._data, self.dtype),
+ )
+ self._rows = []
+ self._columns = []
+ self._data = []
+ return values
+
+
+def _band_geometry(
+ destination: zarr.Array,
+ nRows: int | None = None,
+) -> tuple[int, int, int]:
+ rows = array_shard_rows(destination) if nRows is None else max(1, int(nRows))
+ columns = max(1, int(destination.shape[1]))
+ itemsize = max(1, int(np.dtype(destination.dtype).itemsize))
+ chunks = tuple(int(value) for value in destination.chunks)
+ chunk_rows = min(rows, chunks[0])
+ chunk_columns = min(columns, chunks[1])
+ n_chunks = (
+ (rows + chunk_rows - 1)
+ // chunk_rows
+ * ((columns + chunk_columns - 1) // chunk_columns)
+ )
+ dense_bytes = rows * columns * itemsize
+ inner_bytes = chunk_rows * chunk_columns * itemsize
+ return dense_bytes, inner_bytes, n_chunks
+
+
+def _encoded_chunk_bound(rawBytes: int) -> int:
+ """Conservative encoded-size bound for the supported compressors."""
+ raw_bytes = max(0, int(rawBytes))
+ return raw_bytes + raw_bytes // 128 + 1024
+
+
+def _shard_index_bound(nChunks: int) -> int:
+ """Conservative encoded-size bound for a sharding index."""
+ return max(1, int(nChunks)) * 16 + 1024
+
+
+def _row_band_task_peak(
+ *,
+ sourceBytes: int,
+ denseBytes: int,
+ innerChunkBytes: int,
+ nChunks: int,
+ innerConcurrency: int,
+) -> int:
+ encoded_parts = max(1, int(nChunks)) * _encoded_chunk_bound(innerChunkBytes)
+ return (
+ max(0, int(sourceBytes))
+ + max(0, int(denseBytes))
+ + min(max(1, int(nChunks)), max(1, int(innerConcurrency)))
+ * max(0, int(innerChunkBytes))
+ + encoded_parts
+ + encoded_parts
+ + _shard_index_bound(nChunks)
+ )
+
+
+def _writer_count(
+ destination: zarr.Array,
+ resources: ResourceBudget,
+ nTasks: int,
+) -> tuple[int, int]:
+ dense_bytes, inner_bytes, n_chunks = _band_geometry(destination)
+ return admitted_worker_split(
+ resources,
+ nTasks=nTasks,
+ taskBytes=lambda inner: _row_band_task_peak(
+ sourceBytes=0,
+ denseBytes=dense_bytes,
+ innerChunkBytes=inner_bytes,
+ nChunks=n_chunks,
+ innerConcurrency=inner,
+ ),
+ )
+
+
+def _sparse_task_working_bytes(
+ write: SparseWriteBand,
+ innerConcurrency: int,
+) -> int:
+ dense_bytes, inner_bytes, n_chunks = _band_geometry(
+ write.destination,
+ write.band.end - write.band.start,
+ )
+ source_dtype = np.dtype(write.band.data.dtype)
+ destination_dtype = np.dtype(write.band.dtype)
+ values = int(write.band.data.size)
+ conversion_bytes = (
+ values * destination_dtype.itemsize if source_dtype != destination_dtype else 0
+ )
+ if source_dtype.kind == "f" and destination_dtype.kind in "biu":
+ conversion_bytes += values * (source_dtype.itemsize + 1)
+ return _row_band_task_peak(
+ sourceBytes=conversion_bytes,
+ denseBytes=dense_bytes,
+ innerChunkBytes=inner_bytes,
+ nChunks=n_chunks,
+ innerConcurrency=innerConcurrency,
+ )
+
+
+def _sparse_batch_plan(
+ pending: deque[SparseWriteBand],
+ resources: ResourceBudget,
+ residentBytes: int,
+ producerReserveBytes: int,
+ nTasks: int,
+) -> tuple[int, int]:
+ sparse_bytes = sum(write.band.sparseBytes for write in pending)
+ producer_bytes = max(write.producerBytes for write in pending)
+ return admitted_worker_split(
+ resources,
+ nTasks=nTasks,
+ residentBytes=(
+ residentBytes + max(producerReserveBytes, producer_bytes) + sparse_bytes
+ ),
+ taskBytes=lambda inner: max(
+ _sparse_task_working_bytes(write, inner) for write in pending
+ ),
+ )
+
+
+def write_sparse_bands(
+ writes: Iterator[SparseWriteBand],
+ *,
+ resources: ResourceBudget,
+ residentBytes: int = 0,
+ producerReserveBytes: int = 0,
+ msg: str | None = None,
+ total: int | None = None,
+) -> None:
+ """Densify and write complete sparse row bands within CPU and memory limits."""
+ pending: deque[SparseWriteBand] = deque()
+ source = iter(writes)
+ exhausted = False
+ progress = None
+ if msg is not None:
+ from ..utils.progress import tqdmbar
+
+ progress = tqdmbar(desc=msg, total=total)
+
+ def pull() -> SparseWriteBand:
+ sparse_bytes = sum(write.band.sparseBytes for write in pending)
+ producer_bytes = max(
+ producerReserveBytes,
+ max((write.producerBytes for write in pending), default=0),
+ )
+ reserved = residentBytes + producer_bytes + sparse_bytes
+ if reserved > resources.memoryBytes:
+ raise MemoryError(
+ f"Sparse producer needs about {reserved} bytes before a write "
+ f"task, but the operation limit is {resources.memoryBytes} bytes"
+ )
+ return next(source)
+
+ def fill() -> None:
+ nonlocal exhausted
+ if not pending and not exhausted:
+ try:
+ pending.append(pull())
+ except StopIteration:
+ exhausted = True
+ return
+ while not exhausted:
+ capacity, _ = _sparse_batch_plan(
+ pending,
+ resources,
+ residentBytes,
+ producerReserveBytes,
+ resources.workers,
+ )
+ if len(pending) >= capacity:
+ return
+ try:
+ pending.append(pull())
+ except StopIteration:
+ exhausted = True
+
+ def write_one(item: SparseWriteBand) -> None:
+ item.destination[item.band.start : item.band.end, :] = item.band.dense()
+
+ try:
+ fill()
+ while pending:
+ admitted, inner = _sparse_batch_plan(
+ pending,
+ resources,
+ residentBytes,
+ producerReserveBytes,
+ len(pending),
+ )
+ batch = [pending.popleft() for _ in range(admitted)]
+ for _ in stream_shards(
+ batch,
+ write_one,
+ workers=admitted,
+ within_block_threads=1,
+ io_concurrency=inner,
+ ):
+ if progress is not None:
+ progress.update()
+ del batch
+ fill()
+ finally:
+ _close_iterator(source)
+ if progress is not None:
+ progress.close()
+
+
+def write_dense_from_row_batches(
+ dst: zarr.Array,
+ batches: Iterator[np.ndarray],
+ *,
+ dtype: Any | None = None,
+ msg: str | None = None,
+ resources: ResourceBudget | None = None,
+) -> int:
+ """Align source batches to destination row bands and write them in parallel."""
+ resources = resources or resolve_budget()
+ source = iter(batches)
+ if int(dst.shape[0]) == 0:
+ try:
+ for batch in source:
+ values = np.asarray(batch)
+ if values.ndim != 2 or values.shape[1] != dst.shape[1]:
+ raise ValueError("Dense source batch has an invalid shape")
+ if values.shape[0]:
+ raise ValueError(
+ "Dense stream contains rows for an empty destination"
+ )
+ finally:
+ _close_iterator(source)
+ return 0
+ shard_rows = array_shard_rows(dst)
+
+ def aligned() -> Iterator[_DenseWriteBand]:
+ target_dtype = np.dtype(dst.dtype if dtype is None else dtype)
+ n_columns = int(dst.shape[1])
+ buffer = np.empty((shard_rows, n_columns), dtype=target_dtype)
+ buffered_rows = 0
+ position = 0
+ try:
+ for batch in source:
+ values = np.asarray(batch)
+ if values.ndim != 2 or values.shape[1] != dst.shape[1]:
+ raise ValueError("Dense source batch has an invalid shape")
+ if values.shape[0] == 0:
+ continue
+ source_start = 0
+ while source_start < int(values.shape[0]):
+ copied = min(
+ shard_rows - buffered_rows,
+ int(values.shape[0]) - source_start,
+ )
+ buffer[buffered_rows : buffered_rows + copied] = values[
+ source_start : source_start + copied
+ ]
+ buffered_rows += copied
+ source_start += copied
+ if buffered_rows == shard_rows:
+ yield _DenseWriteBand(
+ position,
+ position + shard_rows,
+ buffer,
+ )
+ position += shard_rows
+ buffer = np.empty(
+ (shard_rows, n_columns),
+ dtype=target_dtype,
+ )
+ buffered_rows = 0
+ if buffered_rows:
+ yield _DenseWriteBand(
+ position,
+ position + buffered_rows,
+ buffer[:buffered_rows],
+ )
+ finally:
+ _close_iterator(source)
+
+ n_bands = (int(dst.shape[0]) + shard_rows - 1) // shard_rows
+ try:
+ workers, inner = _writer_count(dst, resources, n_bands)
+ except BaseException:
+ _close_iterator(source)
+ raise
+
+ def write_band(band: _DenseWriteBand) -> int:
+ dst[band.start : band.end, :] = band.values
+ return band.end - band.start
+
+ total_rows = int(
+ sum(
+ stream_shards(
+ aligned(),
+ write_band,
+ workers=workers,
+ within_block_threads=1,
+ io_concurrency=inner,
+ msg=msg or "Writing Zarr array",
+ total=n_bands,
+ )
+ )
+ )
+ if total_rows != int(dst.shape[0]):
+ raise ValueError(
+ f"Dense stream contains {total_rows} rows, expected {dst.shape[0]}"
+ )
+ return total_rows
+
+
+def write_dense_in_shard_rows(
+ dst: zarr.Array,
+ produce: Callable[[int, int], np.ndarray],
+ *,
+ msg: str | None = None,
+ also_write_to: zarr.Array | None = None,
+ resources: ResourceBudget | None = None,
+ summarize: Callable[[np.ndarray], Any] | None = None,
+ merge_summary: Callable[[Any, Any], Any] | None = None,
+) -> Any | None:
+ """Produce and write complete destination row bands in the same worker."""
+ if (summarize is None) != (merge_summary is None):
+ raise ValueError("summarize and merge_summary must be provided together")
+ merger = merge_summary
+ resources = resources or resolve_budget()
+ n_rows = int(dst.shape[0])
+ if n_rows == 0:
+ return None
+ rows = array_shard_rows(dst)
+ if also_write_to is not None and (
+ tuple(also_write_to.shape) != tuple(dst.shape)
+ or array_shard_rows(also_write_to) != rows
+ ):
+ raise ValueError("Mirror array must have matching shape and row-band layout")
+ slices = list(iter_shard_row_slices(n_rows, rows))
+ workers, inner = _writer_count(dst, resources, len(slices))
+
+ def produce_and_write(bounds: tuple[int, int]) -> Any:
+ start, end = bounds
+ block = np.asarray(produce(start, end))
+ expected = (end - start, int(dst.shape[1]))
+ if block.shape != expected:
+ raise ValueError(
+ f"Dense producer returned shape {block.shape}, expected {expected}"
+ )
+ dst[start:end, :] = block
+ if also_write_to is not None:
+ also_write_to[start:end, :] = block
+ return None if summarize is None else summarize(block)
+
+ summary: Any | None = None
+ for result in stream_shards(
+ slices,
+ produce_and_write,
+ workers=workers,
+ within_block_threads=1,
+ io_concurrency=inner,
+ msg=msg or "Writing Zarr array",
+ total=len(slices),
+ ):
+ if summarize is not None:
+ assert merger is not None
+ summary = result if summary is None else merger(summary, result)
+ return summary
+
+
+def accumulate_sparse_to_shards(
+ dst: zarr.Array,
+ data_stream: Iterator[Any],
+ *,
+ resources: ResourceBudget | None = None,
+ residentBytes: int = 0,
+ producerReserveBytes: int,
+ msg: str | None = None,
+) -> int:
+ """Write one complete dense row band per sparse destination object."""
+ resources = resources or resolve_budget()
+ source = iter(data_stream)
+ if int(dst.shape[0]) == 0:
+ try:
+ for batch in source:
+ if len(batch.shape) != 2 or int(batch.shape[1]) != int(dst.shape[1]):
+ raise ValueError("Sparse source batch has an invalid shape")
+ if int(batch.shape[0]):
+ raise ValueError(
+ "Sparse stream contains rows for an empty destination"
+ )
+ finally:
+ _close_iterator(source)
+ return 0
+ buffer = SparseShardBuffer(dst)
+
+ def writes() -> Iterator[SparseWriteBand]:
+ from scipy.sparse import coo_matrix
+
+ try:
+ for batch in source:
+ coo = batch if hasattr(batch, "row") else coo_matrix(batch)
+ source_bytes = sparse_matrix_bytes(batch, coo)
+ for band in buffer.add(coo):
+ yield SparseWriteBand(
+ dst,
+ band,
+ source_bytes + buffer.residentBytes,
+ )
+ del batch, coo
+ for band in buffer.finish():
+ yield SparseWriteBand(dst, band, buffer.residentBytes)
+ finally:
+ _close_iterator(source)
+
+ try:
+ write_sparse_bands(
+ writes(),
+ resources=resources,
+ residentBytes=residentBytes,
+ producerReserveBytes=producerReserveBytes,
+ msg=msg,
+ total=(int(dst.shape[0]) + array_shard_rows(dst) - 1)
+ // array_shard_rows(dst),
+ )
+ finally:
+ _close_iterator(source)
+ return buffer.rows
+
+
+def write_counts_t(
+ counts: zarr.Array,
+ group: zarr.Group,
+ *,
+ profile: StorageProfile | None = None,
+ resources: ResourceBudget | None = None,
+ feature_major_layout: bool = False,
+) -> zarr.Array | None:
+ """Write the optional feature-major count matrix using public Zarr APIs."""
+ from ..utils.logging import logger
+ from ..utils.progress import tqdmbar
+
+ if _group_zarr_format(group) < 3:
+ return None
+ resources = resources or resolve_budget()
+ resolved_profile = profile or resolve_storage_profile(group.store)
+ n_cells = int(counts.shape[0])
+ n_feats = int(counts.shape[1])
+ itemsize = max(1, int(np.dtype(counts.dtype).itemsize))
+ if feature_major_layout:
+ chunk_elements = max(1, int(np.prod(counts.chunks, dtype=np.int64)))
+ cell_chunk = max(1, min(max(1, n_cells), chunk_elements))
+ feature_chunk = max(
+ 1,
+ min(
+ max(1, n_feats),
+ chunk_elements // cell_chunk,
+ ),
+ )
+ else:
+ source_rows = array_shard_rows(counts)
+ feature_chunk = max(1, min(max(1, n_feats), int(counts.chunks[1])))
+ cell_chunk = max(1, min(max(1, n_cells), source_rows))
+ inner_bytes = feature_chunk * cell_chunk * itemsize
+
+ counts_t = create_numeric_array(
+ group,
+ "countsT",
+ ZarrArraySpec(
+ shape=(n_feats, n_cells),
+ chunks=(feature_chunk, cell_chunk),
+ dtype=counts.dtype,
+ compressors=get_compressors(resolved_profile, zarrFormat=3),
+ shards=None,
+ fillValue=0,
+ overwrite=True,
+ ),
+ )
+ counts_t.attrs["complete"] = False
+ if n_cells == 0 or n_feats == 0:
+ counts_t.attrs["complete"] = True
+ return counts_t
+
+ n_tasks = (
+ (n_cells + cell_chunk - 1)
+ // cell_chunk
+ * ((n_feats + feature_chunk - 1) // feature_chunk)
+ )
+ workers, inner = admitted_worker_split(
+ resources,
+ nTasks=n_tasks,
+ taskBytes=lambda concurrency: _row_band_task_peak(
+ sourceBytes=inner_bytes,
+ denseBytes=inner_bytes,
+ innerChunkBytes=inner_bytes,
+ nChunks=1,
+ innerConcurrency=concurrency,
+ ),
+ )
+ logger.debug(
+ f"Writing countsT shape=({n_feats}, {n_cells}) "
+ f"chunks=({feature_chunk}, {cell_chunk}) "
+ f"tasks={n_tasks} workers={workers}"
+ )
+ progress = tqdmbar(desc="Writing transposed counts", total=n_tasks)
+
+ def inner_chunk_values(
+ task: tuple[int, int],
+ ) -> tuple[tuple[slice, slice], np.ndarray]:
+ feat_start, cell_start = task
+ feat_end = min(feat_start + feature_chunk, n_feats)
+ cell_end = min(cell_start + cell_chunk, n_cells)
+ block = np.asarray(
+ counts[cell_start:cell_end, feat_start:feat_end],
+ )
+ return (
+ (slice(feat_start, feat_end), slice(cell_start, cell_end)),
+ np.ascontiguousarray(block.T),
+ )
+
+ async def copy_all() -> None:
+ remaining = (
+ (feat_start, cell_start)
+ for cell_start in range(0, n_cells, cell_chunk)
+ for feat_start in range(0, n_feats, feature_chunk)
+ )
+
+ async def copy_worker() -> None:
+ for task in remaining:
+ selection, values = await asyncio.to_thread(
+ inner_chunk_values,
+ task,
+ )
+ await counts_t.async_array.setitem(selection, values)
+ del selection, values
+ progress.update()
+
+ async with asyncio.TaskGroup() as group_tasks:
+ for _ in range(workers):
+ group_tasks.create_task(copy_worker())
+
+ with zarr.config.set({"async.concurrency": inner}):
+ try:
+ _run_async(copy_all)
+ except ExceptionGroup as error:
+ if len(error.exceptions) == 1:
+ raise error.exceptions[0] from error
+ raise
+ finally:
+ progress.close()
+ counts_t.attrs["complete"] = True
+ return counts_t
diff --git a/scarf/storage/stores.py b/scarf/storage/stores.py
new file mode 100644
index 00000000..9df6c704
--- /dev/null
+++ b/scarf/storage/stores.py
@@ -0,0 +1,382 @@
+import os
+from typing import Any
+
+import numpy as np
+import zarr
+from zarr.abc.store import Store
+
+from .types import ZarrMode, as_zarr_array, as_zarr_group
+from .profiles import (
+ ZarrLocation,
+ is_remote_zarr_location,
+)
+
+MATRIX_SOURCE_ATTR = "matrixSource"
+_ASSAY_COPY_ATTRS = ("is_assay", "misc", "percentFeatures", "size_factor")
+_WORKSPACE_COPY_ATTRS = ("defaultAssay", "assayTypes")
+
+
+def zarr_group_root(group: zarr.Group, mode: ZarrMode = "r+") -> zarr.Group:
+ """Open the root Zarr group sharing the same store as ``group``."""
+ return zarr.open_group(store=group.store, mode=mode)
+
+
+def zarr_root_path(node: zarr.Group | zarr.Array) -> str | None:
+ """Return the filesystem path for a Zarr node when available."""
+ store = node.store
+ root = getattr(store, "root", None)
+ if root is not None:
+ return str(root)
+ storePath = getattr(node, "store_path", None)
+ if storePath and str(storePath).startswith("file://"):
+ return str(storePath)[7:]
+ return None
+
+
+def is_remote_datastore(
+ zarr_loc: ZarrLocation | None,
+ node: zarr.Group | zarr.Array,
+) -> bool:
+ """Return whether a datastore uses a remote or object backend."""
+ if isinstance(zarr_loc, str) and zarr_loc:
+ return is_remote_zarr_location(zarr_loc)
+ if zarr_root_path(node) is not None:
+ return False
+ store_name = type(node.store).__name__
+ if store_name in ("MemoryStore", "LocalStore"):
+ return False
+ return True
+
+
+def _is_obstore_native_store(obj: object) -> bool:
+ return type(obj).__module__.startswith("obstore.")
+
+
+def make_store(
+ location: ZarrLocation,
+ *,
+ storage_options: dict[str, Any] | None = None,
+ read_only: bool = False,
+) -> str | Store:
+ """Resolve a path, URI, or store for use with ``zarr.open_group``."""
+ if isinstance(location, Store):
+ return location
+
+ if _is_obstore_native_store(location):
+ from zarr.storage import ObjectStore
+
+ return ObjectStore(store=location, read_only=read_only) # type: ignore[type-var]
+
+ if isinstance(location, str):
+ if is_remote_zarr_location(location):
+ if location.startswith("hf://"):
+ from zarr.storage import FsspecStore
+
+ return FsspecStore.from_url(
+ location,
+ storage_options=storage_options,
+ read_only=read_only,
+ )
+ try:
+ from obstore.store import from_url as obstore_from_url
+ from zarr.storage import ObjectStore
+ except ImportError as exc:
+ raise ImportError("Remote Zarr stores require obstore.") from exc
+ obstore = obstore_from_url(location, **(storage_options or {}))
+ return ObjectStore(store=obstore, read_only=read_only) # type: ignore[type-var]
+ return location
+
+ raise TypeError(
+ f"zarr location must be a path string or zarr Store, got {type(location)!r}"
+ )
+
+
+def open_store(
+ path: ZarrLocation,
+ mode: ZarrMode = "r",
+ storage_options: dict[str, Any] | None = None,
+) -> zarr.Group:
+ """Open a Zarr group from a path, URI, or store object."""
+ store = make_store(path, storage_options=storage_options, read_only=(mode == "r"))
+ if isinstance(store, str):
+ return zarr.open_group(store, mode=mode)
+ return zarr.open_group(store=store, mode=mode)
+
+
+def load_zarr(
+ zarr_loc: ZarrLocation,
+ mode: ZarrMode,
+ storage_options: dict[str, Any] | None = None,
+) -> zarr.Group:
+ """Open a Zarr group through the compatibility entry point."""
+ return open_store(zarr_loc, mode=mode, storage_options=storage_options)
+
+
+def _persistable_location(source: str) -> str:
+ """Return a location that resolves identically from any working directory."""
+ if "://" in source:
+ return source
+ return os.path.abspath(source)
+
+
+def _discard_target(target: zarr.Group, at: ZarrLocation) -> None:
+ """Delete a target created by this call so the mount can be retried."""
+ from zarr.core.sync import sync
+
+ from ..utils.logging import logger
+
+ try:
+ sync(target.store_path.delete_dir())
+ except Exception as exc:
+ logger.warning(
+ f"Could not remove the incomplete mount target at {at}: {exc}. "
+ "Delete it before mounting again."
+ )
+
+
+def _workspace_group(root: zarr.Group, workspace: str | None) -> zarr.Group:
+ if workspace is None:
+ return root
+ return as_zarr_group(root[workspace], name=workspace)
+
+
+def _list_assay_names(root: zarr.Group, workspace: str | None) -> list[str]:
+ zw = _workspace_group(root, workspace)
+ assays: list[str] = []
+ for name in sorted(dict.fromkeys(zw.group_keys())):
+ node = zw[name]
+ if isinstance(node, zarr.Group) and "is_assay" in node.attrs:
+ assays.append(name)
+ return assays
+
+
+def _assay_identity(
+ source_root: zarr.Group,
+ assay: zarr.Group,
+ assay_name: str,
+ workspace: str | None,
+ *,
+ cell_ids_fingerprint: str | None,
+) -> dict[str, Any]:
+ from .artifacts import fingerprint_stored_strings
+ from .schema import load_count_array
+
+ counts = load_count_array(source_root, assay_name, workspace)
+ entry: dict[str, Any] = {
+ "shape": [int(counts.shape[0]), int(counts.shape[1])],
+ "dtype": np.dtype(counts.dtype).str,
+ "datasetFingerprint": None,
+ "cellIdsFingerprint": None,
+ "featureIdsFingerprint": None,
+ }
+ fingerprint = assay.attrs.get("dataset_fingerprint")
+ if fingerprint is not None:
+ entry["datasetFingerprint"] = str(fingerprint)
+ return entry
+ if cell_ids_fingerprint is None:
+ raise RuntimeError("Cell identifier fingerprint was not calculated")
+ feature_data = as_zarr_group(
+ assay["featureData"],
+ name=f"{assay_name}/featureData",
+ )
+ feature_ids = as_zarr_array(feature_data["ids"], name="ids")
+ entry["cellIdsFingerprint"] = cell_ids_fingerprint
+ entry["featureIdsFingerprint"] = fingerprint_stored_strings(feature_ids)
+ return entry
+
+
+def _validate_assay_identity(
+ source_root: zarr.Group,
+ assay: zarr.Group,
+ assay_name: str,
+ workspace: str | None,
+ expected: dict[str, Any],
+ *,
+ cell_ids_fingerprint: str | None,
+) -> None:
+ from .artifacts import fingerprint_stored_strings
+ from .schema import load_count_array
+
+ counts = load_count_array(source_root, assay_name, workspace)
+ shape = [int(counts.shape[0]), int(counts.shape[1])]
+ dtype = np.dtype(counts.dtype).str
+ if shape != list(expected.get("shape", [])) or dtype != expected.get("dtype"):
+ raise ValueError(
+ f"Matrix source assay {assay_name!r} no longer matches the mounted "
+ "count matrix identity"
+ )
+ expected_fingerprint = expected.get("datasetFingerprint")
+ if expected_fingerprint is not None:
+ current = assay.attrs.get("dataset_fingerprint")
+ if current is None or str(current) != str(expected_fingerprint):
+ raise ValueError(
+ f"Matrix source assay {assay_name!r} dataset fingerprint no longer "
+ "matches the mounted store"
+ )
+ return
+ if cell_ids_fingerprint is None:
+ raise RuntimeError("Cell identifier fingerprint was not calculated")
+ feature_data = as_zarr_group(
+ assay["featureData"],
+ name=f"{assay_name}/featureData",
+ )
+ feature_ids = as_zarr_array(feature_data["ids"], name="ids")
+ if cell_ids_fingerprint != expected.get(
+ "cellIdsFingerprint"
+ ) or fingerprint_stored_strings(feature_ids) != expected.get(
+ "featureIdsFingerprint"
+ ):
+ raise ValueError(
+ f"Matrix source assay {assay_name!r} cell or feature identifiers no "
+ "longer match the mounted store"
+ )
+
+
+def create_matrix_source(
+ source: str,
+ at: ZarrLocation,
+ *,
+ workspace: str | None = None,
+ storage_options: dict[str, Any] | None = None,
+) -> zarr.Group:
+ """Create a writable store that mounts count matrices from ``source``."""
+ from .copy import copy_zarr_group_tree
+
+ if not isinstance(source, str) or not source:
+ raise TypeError("Matrix source location must be a non-empty string")
+ source = _persistable_location(source)
+ source_root = load_zarr(
+ source,
+ mode="r",
+ storage_options=storage_options,
+ )
+ assay_names = _list_assay_names(source_root, workspace)
+ if not assay_names:
+ raise ValueError("No assays found in the matrix source")
+
+ source_zw = _workspace_group(source_root, workspace)
+ source_cell_data = as_zarr_group(
+ source_zw["cellData"],
+ name="cellData",
+ )
+ cell_ids = as_zarr_array(source_cell_data["ids"], name="ids")
+ source_assays: dict[str, zarr.Group] = {}
+ needs_id_fingerprint = False
+ for assay_name in assay_names:
+ source_assay = as_zarr_group(source_zw[assay_name], name=assay_name)
+ feature_data = as_zarr_group(
+ source_assay["featureData"],
+ name=f"{assay_name}/featureData",
+ )
+ as_zarr_array(feature_data["ids"], name="ids")
+ source_assays[assay_name] = source_assay
+ needs_id_fingerprint = (
+ needs_id_fingerprint
+ or source_assay.attrs.get("dataset_fingerprint") is None
+ )
+
+ cell_ids_fingerprint: str | None = None
+ if needs_id_fingerprint:
+ from .artifacts import fingerprint_stored_strings
+
+ cell_ids_fingerprint = fingerprint_stored_strings(cell_ids)
+
+ assay_manifest = {
+ assay_name: _assay_identity(
+ source_root,
+ source_assays[assay_name],
+ assay_name,
+ workspace,
+ cell_ids_fingerprint=cell_ids_fingerprint,
+ )
+ for assay_name in assay_names
+ }
+
+ target = load_zarr(at, mode="w-", storage_options=storage_options)
+ try:
+ target_zw = target if workspace is None else target.create_group(workspace)
+ for key in _WORKSPACE_COPY_ATTRS:
+ if key in source_zw.attrs:
+ target_zw.attrs[key] = source_zw.attrs[key]
+
+ cell_data = target_zw.create_group("cellData")
+ copy_zarr_group_tree(source_cell_data, cell_data)
+ for assay_name in assay_names:
+ source_assay = source_assays[assay_name]
+ target_assay = target_zw.create_group(assay_name)
+ for key in _ASSAY_COPY_ATTRS:
+ if key in source_assay.attrs:
+ target_assay.attrs[key] = source_assay.attrs[key]
+ feature_data = target_assay.create_group("featureData")
+ copy_zarr_group_tree(
+ as_zarr_group(source_assay["featureData"], name="featureData"),
+ feature_data,
+ )
+
+ target.attrs[MATRIX_SOURCE_ATTR] = {
+ "location": source,
+ "workspace": workspace,
+ "assays": assay_manifest,
+ }
+ except BaseException:
+ _discard_target(target, at)
+ raise
+ return target
+
+
+def resolve_matrix_source(
+ root: zarr.Group,
+ *,
+ storage_options: dict[str, Any] | None = None,
+) -> tuple[zarr.Group, str | None] | None:
+ """Open and validate a mounted matrix source, if present."""
+ raw = root.attrs.get(MATRIX_SOURCE_ATTR)
+ if raw is None:
+ return None
+ if not isinstance(raw, dict):
+ raise ValueError("matrixSource attribute must be a mapping")
+ location = raw.get("location")
+ if not isinstance(location, str) or not location:
+ raise ValueError("matrixSource.location must be a non-empty string")
+ workspace = raw.get("workspace")
+ if workspace is not None and not isinstance(workspace, str):
+ raise ValueError("matrixSource.workspace must be a string or null")
+ assays = raw.get("assays")
+ if not isinstance(assays, dict) or not assays:
+ raise ValueError("matrixSource.assays must be a non-empty mapping")
+
+ source_root = load_zarr(
+ location,
+ mode="r",
+ storage_options=storage_options,
+ )
+ entries: list[tuple[str, dict[str, Any]]] = []
+ for assay_name, expected in assays.items():
+ if not isinstance(assay_name, str):
+ raise ValueError("matrixSource assay names must be strings")
+ if not isinstance(expected, dict):
+ raise ValueError(
+ f"matrixSource assay entry for {assay_name!r} must be a mapping"
+ )
+ entries.append((assay_name, expected))
+
+ source_zw = _workspace_group(source_root, workspace)
+ cell_ids_fingerprint: str | None = None
+ if any(expected.get("datasetFingerprint") is None for _, expected in entries):
+ from .artifacts import fingerprint_stored_strings
+
+ cell_data = as_zarr_group(source_zw["cellData"], name="cellData")
+ cell_ids = as_zarr_array(cell_data["ids"], name="ids")
+ cell_ids_fingerprint = fingerprint_stored_strings(cell_ids)
+
+ for assay_name, expected in entries:
+ source_assay = as_zarr_group(source_zw[assay_name], name=assay_name)
+ _validate_assay_identity(
+ source_root,
+ source_assay,
+ assay_name,
+ workspace,
+ expected,
+ cell_ids_fingerprint=cell_ids_fingerprint,
+ )
+ return source_root, workspace
diff --git a/scarf/storage/types.py b/scarf/storage/types.py
new file mode 100644
index 00000000..df3d8377
--- /dev/null
+++ b/scarf/storage/types.py
@@ -0,0 +1,32 @@
+from typing import Literal
+
+import zarr
+
+
+type ZarrMode = Literal["r", "r+", "a", "w", "w-"]
+
+
+def as_zarr_array(
+ node: zarr.Array | zarr.Group,
+ *,
+ name: str = "",
+) -> zarr.Array:
+ if isinstance(node, zarr.Array):
+ return node
+ label = f" at {name!r}" if name else ""
+ raise TypeError(f"Expected Zarr array{label}, got {type(node).__name__}")
+
+
+def as_zarr_group(
+ node: zarr.Array | zarr.Group,
+ *,
+ name: str = "",
+) -> zarr.Group:
+ if isinstance(node, zarr.Group):
+ return node
+ label = f" at {name!r}" if name else ""
+ raise TypeError(f"Expected Zarr group{label}, got {type(node).__name__}")
+
+
+def array_metadata_shards(array: zarr.Array) -> tuple[int, ...] | None:
+ return getattr(array.metadata, "shards", None)
diff --git a/scarf/tests/__init__.py b/scarf/tests/__init__.py
deleted file mode 100644
index 8794fadc..00000000
--- a/scarf/tests/__init__.py
+++ /dev/null
@@ -1,26 +0,0 @@
-import os
-import shutil
-import sys
-
-from ..utils import logger
-
-logger.remove()
-logger.add(sys.stderr, level="ERROR")
-
-__all__ = ["full_path", "remove"]
-
-
-def full_path(fn, *args):
- if fn == "" or fn is None:
- return os.path.join("scarf", "tests", "datasets")
- else:
- return os.path.join("scarf", "tests", "datasets", fn, *args)
-
-
-def remove(dir_path):
- if os.path.isdir(dir_path):
- shutil.rmtree(dir_path)
- elif os.path.exists(dir_path):
- os.unlink(dir_path)
- else:
- pass
diff --git a/scarf/tests/conftest.py b/scarf/tests/conftest.py
deleted file mode 100644
index 1895758a..00000000
--- a/scarf/tests/conftest.py
+++ /dev/null
@@ -1,4 +0,0 @@
-from .fixtures_downloader import *
-from .fixtures_readers import *
-from .fixtures_datastore import *
-
diff --git a/scarf/tests/datasets/1K_pbmc_citeseq.h5 b/scarf/tests/datasets/1K_pbmc_citeseq.h5
deleted file mode 100644
index 81315733..00000000
Binary files a/scarf/tests/datasets/1K_pbmc_citeseq.h5 and /dev/null differ
diff --git a/scarf/tests/datasets/1K_pbmc_citeseq.zarr.tar.gz b/scarf/tests/datasets/1K_pbmc_citeseq.zarr.tar.gz
deleted file mode 100644
index b8666399..00000000
Binary files a/scarf/tests/datasets/1K_pbmc_citeseq.zarr.tar.gz and /dev/null differ
diff --git a/scarf/tests/datasets/500_pbmc_atac.zarr.tar.gz b/scarf/tests/datasets/500_pbmc_atac.zarr.tar.gz
deleted file mode 100644
index eb718d46..00000000
Binary files a/scarf/tests/datasets/500_pbmc_atac.zarr.tar.gz and /dev/null differ
diff --git a/scarf/tests/datasets/aggregated_df_top_10.npy b/scarf/tests/datasets/aggregated_df_top_10.npy
deleted file mode 100644
index 6009594f..00000000
Binary files a/scarf/tests/datasets/aggregated_df_top_10.npy and /dev/null differ
diff --git a/scarf/tests/datasets/aggregated_feat_idx.npy b/scarf/tests/datasets/aggregated_feat_idx.npy
deleted file mode 100644
index 3b66bada..00000000
Binary files a/scarf/tests/datasets/aggregated_feat_idx.npy and /dev/null differ
diff --git a/scarf/tests/datasets/atac_knn_distances.npy b/scarf/tests/datasets/atac_knn_distances.npy
deleted file mode 100644
index 325bc480..00000000
Binary files a/scarf/tests/datasets/atac_knn_distances.npy and /dev/null differ
diff --git a/scarf/tests/datasets/atac_knn_indices.npy b/scarf/tests/datasets/atac_knn_indices.npy
deleted file mode 100644
index e7853d17..00000000
Binary files a/scarf/tests/datasets/atac_knn_indices.npy and /dev/null differ
diff --git a/scarf/tests/datasets/cell_attributes.csv b/scarf/tests/datasets/cell_attributes.csv
deleted file mode 100644
index 313fe368..00000000
--- a/scarf/tests/datasets/cell_attributes.csv
+++ /dev/null
@@ -1,809 +0,0 @@
-,I,ids,names,PTIME_MODULES_nCounts,PTIME_MODULES_nFeatures,RNA_G2M_score,RNA_S_score,RNA_UMAP1,RNA_UMAP2,RNA_balanced_clusters,RNA_cell_cycle_phase,RNA_cell_density,RNA_cluster,RNA_leiden_cluster,RNA_nCounts,RNA_nFeatures,RNA_percentMito,RNA_percentRibo,RNA_pseudotime,RNA_sketch_seeds,RNA_sketched,RNA_snn_value,RNA_unified_UMAP1,RNA_unified_UMAP2,assay2_nCounts,assay2_nFeatures,target_classes,mapping_scores
-0,True,AATCACGAGCAGCCCT-1,AATCACGAGCAGCCCT-1,1.9146028840632483,15.0,-0.09262390055430042,-0.019684164407220917,2.6459084,6.8622274,6,G1,1635.3163743019104,1,1,3058.0,1020.0,6.8999345977763245,41.79202092871158,0.6149374478797888,False,False,3.4,10.589231,10.660447,1651.0,9.0,1,0.8054144824310264
-1,True,AATCACGAGGAACTCG-1,AATCACGAGGAACTCG-1,1.9429760410381736,15.0,-0.025279845838156366,-0.02296731799515923,1.8557484,8.659906,27,G1,1331.7302141338587,8,1,2176.0,975.0,18.841911764705884,14.108455882352942,0.6176216869864952,False,False,3.0,10.040825,12.057594,788.0,9.0,8,0.8054144824310264
-2,True,AATCACGCACTACCGG-1,AATCACGCACTACCGG-1,1.8578043969550482,15.0,-0.08864835706481089,-0.0076830777206780415,9.69295,-11.369626,21,G1,1278.1951800137758,7,6,5536.0,2120.0,5.816473988439307,19.436416184971097,0.04650833633018682,False,False,6.799999999999999,12.0010605,-10.761246,1305.0,9.0,7,0.8054144824310264
-3,True,AATCACGCATGAATAG-1,AATCACGCATGAATAG-1,1.880634640338231,15.0,-0.11152050902681884,-0.006583769451145836,6.977848,2.8306186,8,G1,1306.5597768723965,9,9,5077.0,1462.0,4.648414417963364,45.02659050620445,0.6112025923359019,False,False,2.3,13.427173,7.700363,1007.0,9.0,9,0.9808893821763591
-4,True,AATCACGGTATAGGAT-1,AATCACGGTATAGGAT-1,1.901891912646523,15.0,-0.10505539901191005,-0.0037932092540785417,8.251231,-2.7979677,22,G1,1164.8476556241512,6,8,7386.0,2097.0,8.434876793934471,38.044949905226105,0.45369430937882754,False,False,6.8999999999999995,14.396282,0.054674238,928.0,9.0,6,1.3206616453813353
-5,True,AATCACGGTCGAATGG-1,AATCACGGTCGAATGG-1,1.9259363130711664,15.0,-0.07483857271001484,-0.012962702620913924,2.8696203,6.2859616,5,G1,1541.134561829269,1,1,3966.0,1333.0,14.246091780131115,30.963187090267272,0.6116533853041001,False,False,2.7,9.603099,8.927263,1428.0,10.0,1,0.8054144824310264
-6,True,AATCACGGTGTCTAAC-1,AATCACGGTGTCTAAC-1,1.879618727631841,15.0,-0.10944414568326681,0.0010908939374005798,2.4051206,7.9692683,5,S,1546.4653607159853,1,1,6357.0,1541.0,5.647317917256568,44.81673745477426,0.612388298156135,False,False,2.9,8.690099,10.703329,1077.0,9.0,1,1.0102850184689025
-7,True,AATCACGTCCACGTAA-1,AATCACGTCCACGTAA-1,1.8973074885383008,15.0,-0.09167494807424922,-0.00853403836255738,5.4800663,6.8413477,12,G1,1352.1825319975615,1,1,5737.0,1506.0,5.508105281506014,43.907965835802685,0.6130000705872367,False,False,4.800000000000001,9.873918,7.695063,895.0,9.0,1,0.9817706108534437
-8,True,AATCACGTCCCGTTGT-1,AATCACGTCCCGTTGT-1,1.9033433854258006,15.0,-0.0902786072188888,-0.012823941075421428,5.4946275,0.32944006,22,G1,1254.204477339983,6,8,5332.0,1742.0,9.808702175543885,33.514628657164295,0.5770699260800276,False,False,4.1000000000000005,14.220285,5.354528,1610.0,9.0,6,1.1559159954332647
-9,True,AATCACGTCGCCTTGT-1,AATCACGTCGCCTTGT-1,1.9503836684894997,15.0,-0.0797848840078176,-0.023488315476172913,-9.845075,-5.89527,10,G1,1177.667477890849,3,7,9873.0,3263.0,5.8138357135622405,12.731692494682468,0.9765974809887463,False,False,3.2,-13.166267,-6.8673778,2566.0,9.0,3,1.2323730773035082
-10,True,AATCACGTCGTTTACT-1,AATCACGTCGTTTACT-1,1.9527865874278352,15.0,-0.0910403049385046,-0.024601078042838773,-5.093038,-12.03094,24,G1,1226.4370895773172,10,7,11441.0,3275.0,11.275238178480903,17.515951402849403,0.9631948135077659,False,False,4.0,-8.788839,-10.761385,2437.0,9.0,10,1.0149610188844926
-11,True,AATCACGTCTCACTCG-1,AATCACGTCTCACTCG-1,2.0295826256267238,15.0,-0.03935599284436494,-0.03079673293265527,-0.06250891,8.262804,5,G1,1324.4120434373617,1,1,1875.0,906.0,29.493333333333332,2.506666666666667,0.6268871961484299,False,False,6.3,9.200972,12.995318,896.0,9.0,1,1.5766106259950199
-12,True,ACAGAAAAGACCATTC-1,ACAGAAAAGACCATTC-1,1.9302843479752636,15.0,-0.0854586650258603,-0.027278620576574104,-3.1279614,-13.404911,24,G1,1325.8092089295387,10,7,15316.0,3934.0,5.791329328806477,14.958213632802298,0.9594748838539864,False,False,6.499999999999999,-7.4478264,-11.790138,3837.0,9.0,10,0.9706953108584262
-13,True,ACAGAAAAGCGACATG-1,ACAGAAAAGCGACATG-1,1.9809942190057146,15.0,-0.09065688322869574,-0.03489471870873216,-3.8341582,-6.9765625,3,G1,1216.2443554103374,2,2,11838.0,3446.0,12.341611758743031,21.51545869234668,0.9546716355560309,False,False,3.4,-15.039748,-6.8850813,1651.0,9.0,2,1.0131033238105016
-14,True,ACAGAAAAGGCCTAGA-1,ACAGAAAAGGCCTAGA-1,1.894572635466383,15.0,-0.09623704331549857,-0.021230952960169847,8.448699,-12.429867,21,G1,1194.048403546214,7,6,2195.0,936.0,10.979498861047835,25.10250569476082,0.021458791239320978,False,False,5.5,11.008296,-11.105168,2073.0,9.0,7,0.9877630652298343
-15,True,ACAGAAAAGGTACAGC-1,ACAGAAAAGGTACAGC-1,1.923602176252625,15.0,-0.103005989491423,-0.0035347646575937808,5.3981943,4.3785796,12,G1,1593.3654461428523,1,1,5611.0,1428.0,6.612012119051863,48.72571734093744,0.6113470470095177,False,False,3.1,11.269537,7.7245784,1538.0,9.0,1,1.2110532970525933
-16,True,ACAGAAAAGGTTACAA-1,ACAGAAAAGGTTACAA-1,1.896852546598276,15.0,-0.0979899688715361,-0.012741612046525205,6.908522,10.887742,15,G1,1136.3607185781002,8,10,6738.0,1636.0,6.188780053428317,46.734936182843576,0.6196351581036357,True,True,4.6,15.006257,9.488196,2006.0,9.0,8,0.948632716953345
-17,True,ACAGAAACAATAGTAG-1,ACAGAAACAATAGTAG-1,1.8795965414294988,15.0,-0.10176110771026943,-0.020960593163283227,9.025863,5.55703,26,G1,1130.8305401653051,9,9,4515.0,1495.0,3.521594684385382,40.8859357696567,0.6120454900655272,False,False,4.8999999999999995,10.484926,5.837624,2010.0,9.0,9,1.2591180974451068
-18,True,ACAGAAACACAGAGAC-1,ACAGAAACACAGAGAC-1,1.879690088918888,15.0,-0.09917952087317958,-0.014060675249673244,7.934319,-3.7038977,23,G1,1180.102633073926,6,8,9486.0,2776.0,5.418511490617752,30.212945393211047,0.42960329282772103,False,True,7.1,14.962476,-0.45513263,1368.0,9.0,6,0.8054144824310264
-19,True,ACAGAAACAGACCTAT-1,ACAGAAACAGACCTAT-1,1.9694872213793844,15.0,-0.08439067125224588,-0.014616310813025158,-4.891262,-0.7927549,20,G1,1066.216839298606,4,4,8594.0,2799.0,9.750989062136375,15.289737025831975,0.9644182857009779,False,False,5.4,-17.608807,-0.9948211,4236.0,9.0,4,0.8054144824310264
-20,True,ACAGAAACAGCTACTA-1,ACAGAAACAGCTACTA-1,1.9776234695529329,15.0,-0.08198230238747423,-0.026254721246912242,-9.305638,-1.1854355,7,G1,1635.7103563845158,3,3,12756.0,3422.0,8.490122295390405,18.179680150517402,0.983336135120444,False,False,4.500000000000001,-16.334785,-1.4771385,3079.0,9.0,3,1.118423185428613
-21,True,ACAGAAACATCAGTGT-1,ACAGAAACATCAGTGT-1,1.9143367614442166,15.0,-0.09235357012593028,-0.02194865077749555,11.414043,-8.609801,11,G1,1215.8651418685913,7,6,5485.0,1562.0,8.113035551504103,37.50227894257065,0.15002312810456642,False,True,7.1000000000000005,12.914139,-7.0806255,2756.0,9.0,7,1.2066315300929993
-22,True,ACAGAAAGTCGAATGG-1,ACAGAAAGTCGAATGG-1,1.876593629845063,15.0,-0.08351020492251578,-0.022082887904243932,8.529866,-12.983943,21,G1,1119.1940445750952,7,6,3647.0,1512.0,8.225939128050452,15.958321908417878,0.016788197122685228,False,True,4.3,10.64558,-11.4172125,1972.0,10.0,7,1.1127213340393742
-23,True,ACAGAAAGTCGTATGT-1,ACAGAAAGTCGTATGT-1,1.9328029834907183,15.0,-0.12795666865434305,-0.011959013841320164,0.501181,12.729525,2,G1,850.2403893768787,5,5,5880.0,1650.0,8.503401360544217,42.993197278911566,0.7255904165010446,False,False,6.7,-0.29323268,16.415169,1136.0,9.0,5,1.3311446034738328
-24,True,ACAGAAAGTTGGACCC-1,ACAGAAAGTTGGACCC-1,1.9605230741291146,15.0,-0.06752153878920938,-0.02738148039330419,-7.8785524,0.0218554,14,G1,1255.818872153759,4,4,6151.0,2233.0,9.218013331165665,12.0955942123232,0.9495228655063732,False,False,3.6,-13.418969,-0.72872823,2761.0,9.0,4,1.0511520956857803
-25,True,ACAGAAATCGACGCGT-1,ACAGAAATCGACGCGT-1,1.8916691281157154,15.0,-0.10229398333612602,-0.027564728792835322,7.4776497,-2.934058,22,G1,1195.7869687974453,6,8,5669.0,1804.0,9.931204798024343,31.910389839477862,0.45520109551390225,False,False,7.7,14.815071,0.08478602,1417.0,9.0,6,0.8054144824310264
-26,True,ACAGAAATCTCGAGTA-1,ACAGAAATCTCGAGTA-1,1.883192495165775,15.0,-0.10148617035915784,-0.017039383860827505,7.696917,-1.9375056,22,G1,1027.250883370638,6,8,5980.0,1875.0,7.909698996655519,33.64548494983278,0.49610373452423573,False,False,3.0000000000000004,15.035936,1.5720447,1173.0,9.0,6,0.8054144824310264
-28,True,ACAGAAATCTTAGCCC-1,ACAGAAATCTTAGCCC-1,1.9256684515903995,15.0,-0.05988375713951127,-0.022115305158513822,-4.178427,-13.458362,24,G1,1180.4352160394192,10,7,13998.0,3630.0,8.48692670381483,16.959565652236034,0.9597494317807175,False,True,5.4,-7.9204836,-11.824245,3124.0,9.0,10,0.9558447829765444
-29,True,ACCCTTGAGACGTCCC-1,ACCCTTGAGACGTCCC-1,1.9266154671094529,15.0,-0.07311668226149834,-0.01455771261430732,-1.699633,-13.507107,24,G1,1307.2199267148972,10,7,14056.0,3696.0,8.708025042686398,17.551223676721683,0.9584301895311081,False,False,7.799999999999999,-6.587843,-12.211588,4017.0,9.0,10,0.9625685164062322
-30,True,ACCCTTGAGACGTCGA-1,ACCCTTGAGACGTCGA-1,1.9084909421261187,15.0,-0.10509415851892019,-0.010426675300685513,6.8856034,2.7047584,8,G1,1451.9405464082956,9,9,5813.0,1532.0,10.562532255289868,38.929984517460866,0.6108684693951845,False,False,4.0,12.857927,6.8620844,1475.0,9.0,9,1.3537176247944693
-31,True,ACCCTTGAGAGAGCGG-1,ACCCTTGAGAGAGCGG-1,1.9377355562266423,15.0,-0.09460539585616512,-0.018024137525203203,-5.6261396,-12.498869,10,G1,1201.8003843277693,3,7,16098.0,3771.0,9.995030438563797,19.673251335569635,0.9614684243398492,False,False,4.6,-9.069668,-10.986452,2305.0,9.0,3,1.1040216349363394
-33,True,ACCCTTGCACCAGCGT-1,ACCCTTGCACCAGCGT-1,1.899544289246287,15.0,-0.0642534312454616,-0.027580614813772115,-2.4103203,-12.326489,24,G1,1315.186081185937,10,7,13814.0,3541.0,6.696105400318517,14.383958303170697,0.9593821126702171,False,False,7.3999999999999995,-6.9891562,-12.658615,2127.0,9.0,10,0.9788750406701542
-34,True,ACCCTTGGTCACGCTG-1,ACCCTTGGTCACGCTG-1,1.8989244497204025,15.0,-0.06780221761292611,-0.03225494307002549,7.8903446,7.59997,27,G1,1256.2189148664474,8,10,3010.0,1070.0,11.096345514950166,33.15614617940199,0.613327551644923,False,True,4.800000000000001,13.773765,9.959809,1318.0,9.0,8,1.4094532797118868
-35,True,ACCCTTGGTCGGTGTC-1,ACCCTTGGTCGGTGTC-1,1.9922674162496778,15.0,-0.08400483963710707,-0.02735226372102239,-7.097548,-8.520551,3,G1,1306.474305421114,2,2,8960.0,3017.0,12.979910714285714,9.955357142857142,0.9559370864853503,False,False,3.1,-14.188237,-8.294819,2628.0,9.0,2,1.2088955930832395
-36,True,ACCCTTGGTGACGCCT-1,ACCCTTGGTGACGCCT-1,1.8684989813807944,15.0,-0.0907860473696177,-7.483610892146213e-05,5.740291,6.7807717,27,G1,1492.8751185759902,8,1,5560.0,1461.0,4.586330935251799,46.960431654676256,0.6127383762212906,False,False,3.1,11.156091,9.784362,2019.0,9.0,8,1.1784599979468142
-37,True,ACCCTTGGTTACACAC-1,ACCCTTGGTTACACAC-1,1.8794470496798685,15.0,-0.07666811817694673,0.0005727753746162557,2.5971267,2.7670872,12,S,1493.7553976029158,1,1,7103.0,2156.0,5.406166408559764,38.11065746867521,0.6118240648050127,False,False,3.1,10.048934,9.633621,1277.0,9.0,1,0.8054144824310264
-38,True,ACCCTTGTCCTACAAG-1,ACCCTTGTCCTACAAG-1,1.9353466112242022,15.0,-0.09056816741099151,-0.014568511852282574,-8.446106,-7.3514495,10,G1,1280.0189653784037,3,7,14712.0,3840.0,8.571234366503534,18.196030451332245,0.9675504536868252,False,False,3.3000000000000003,-13.166993,-7.5069847,3598.0,9.0,3,1.2658502293765688
-39,True,ACCCTTGTCGTCGGGT-1,ACCCTTGTCGTCGGGT-1,1.8924303247646337,15.0,-0.09846455306976869,-0.018815781254158715,4.208197,3.0059917,12,G1,1867.3819949030876,1,1,6757.0,1915.0,6.1713778303981055,41.423708746485126,0.6093151266184236,False,False,6.5,11.923405,9.056196,1347.0,9.0,1,1.63685188706804
-40,True,ACCCTTGTCTCACTCG-1,ACCCTTGTCTCACTCG-1,1.9612492536510562,15.0,-0.06978511811162805,-0.019192048864489088,-5.9530315,-5.712263,3,G1,1662.0810243785381,2,2,8265.0,2831.0,6.860254083484573,9.437386569872958,0.9637888864530519,False,False,3.3000000000000003,-15.239969,-5.843939,3303.0,9.0,2,1.059224453113974
-41,True,ACGTCCTAGATCACCT-1,ACGTCCTAGATCACCT-1,1.890923984907585,15.0,-0.07057484182913082,-0.004168466686372149,10.007795,-11.226237,21,G1,1289.3786257356405,7,6,4514.0,1841.0,13.712893221089942,16.415595923792644,0.053870042186144114,False,False,7.499999999999999,11.95194,-9.918121,865.0,9.0,7,1.212127791019733
-43,True,ACGTCCTAGCGAATGC-1,ACGTCCTAGCGAATGC-1,1.8843828604974306,15.0,-0.0963429395844203,0.008325421482058715,1.2262518,5.757637,5,S,1764.2370253801346,1,1,6847.0,1839.0,5.213962319263911,41.857747918796555,0.6140408407370542,True,True,7.4,9.921135,11.217627,1683.0,9.0,1,1.3553743959662323
-44,True,ACGTCCTAGGGCGAAG-1,ACGTCCTAGGGCGAAG-1,1.9626061492248548,15.0,-0.05250621617119254,-0.028860630781521773,-5.7333927,-3.655653,9,G1,1541.5463787019253,2,2,10241.0,3318.0,10.955961331901182,16.355824626501317,0.9504781298590307,False,False,3.6999999999999997,-15.761517,-3.6664236,3113.0,9.0,2,0.9501407005745005
-45,True,ACGTCCTAGGTTGCCC-1,ACGTCCTAGGTTGCCC-1,1.8411991286863043,15.0,-0.08153035609353625,-0.012079119397240289,9.381279,-13.654902,21,G1,1066.690640091896,7,6,3571.0,1547.0,6.048725847101652,22.5707084850182,0.0,True,True,8.1,10.070463,-12.128085,1563.0,9.0,7,0.9781724383891092
-46,True,ACGTCCTCAAACCATC-1,ACGTCCTCAAACCATC-1,2.1240603444743913,15.0,-0.06839945280437756,-0.03299701757725744,-4.0088377,3.7243028,14,G1,1197.2542121261358,4,4,765.0,512.0,20.130718954248366,3.9215686274509802,0.8777616830484557,False,False,5.0,-13.106527,2.793371,837.0,9.0,4,1.0786012599433636
-47,True,ACGTCCTCAAGAGTAT-1,ACGTCCTCAAGAGTAT-1,1.9940406175301597,15.0,-0.07262111129113716,-0.02540544507298371,-9.376629,-0.38546473,7,G1,1603.7265069186687,3,3,9683.0,3074.0,10.131157699060209,14.592584942683052,0.9806804810927897,False,False,4.7,-16.228443,-1.5504993,6456.0,9.0,3,1.3354959871622751
-49,True,ACGTCCTCATTATGCG-1,ACGTCCTCATTATGCG-1,2.034448925587455,15.0,-0.05711323462871902,-0.027609150700334782,-4.974606,-0.079187624,19,G1,1507.6808382868767,2,4,4975.0,1861.0,11.678391959798995,13.78894472361809,0.9543663654580443,False,False,2.9999999999999996,-14.459675,-0.69358426,1204.0,9.0,2,1.0799877553826405
-50,True,ACGTCCTGTATGCGTT-1,ACGTCCTGTATGCGTT-1,1.9037954077126373,15.0,-0.10632762413983757,-0.004796169795629876,0.9615184,6.489793,5,G1,1742.8540508449078,1,1,4945.0,1262.0,7.906976744186046,47.664307381193126,0.6156071771135834,False,False,6.8,9.659729,11.509306,1155.0,9.0,1,1.8163536210883573
-51,True,ACGTCCTGTCCCGTGA-1,ACGTCCTGTCCCGTGA-1,1.9482561961961429,15.0,-0.08340208602750185,-0.00889055747472582,-6.5086055,-4.7722297,13,G1,1664.770869165659,2,2,7795.0,2570.0,6.837716484926235,16.856959589480436,0.9658141163972863,False,False,3.9999999999999996,-16.368217,-5.299693,2170.0,9.0,2,1.069160195042647
-52,True,ACGTCCTGTGCACAAG-1,ACGTCCTGTGCACAAG-1,1.911524318285445,15.0,-0.12208339707426523,-0.012633811495195116,2.7054572,1.9183406,12,G1,1411.3157752752304,1,1,6876.0,1564.0,7.5770796974985455,50.174520069808025,0.6116662581149102,False,False,4.3999999999999995,11.59077,8.909735,758.0,9.0,1,1.170453247554156
-53,True,ACGTCCTTCCCAAGCG-1,ACGTCCTTCCCAAGCG-1,1.9231493720785193,15.0,-0.09072517514696551,-0.02057011127366228,6.8668537,8.437247,27,G1,1240.402009472251,8,10,5475.0,1391.0,9.077625570776256,46.10045662100457,0.6165011538181364,False,False,3.9,13.2112465,10.936267,1220.0,9.0,8,1.3014299323702554
-54,True,ACGTCCTTCGCGGTAC-1,ACGTCCTTCGCGGTAC-1,1.8911260399407708,15.0,-0.10555465517854065,-0.022654199451553374,3.8456054,1.5156982,12,G1,1318.4750853031874,1,1,5806.0,1740.0,6.4071650017223565,40.4753703065794,0.6109226755978073,False,False,3.2,12.465912,8.845622,858.0,9.0,1,0.8054144824310264
-55,True,ACGTCCTTCTGCCCTA-1,ACGTCCTTCTGCCCTA-1,1.8593612057089002,15.0,-0.08281755893925226,-0.022818658115809423,7.375444,2.2778807,8,G1,1443.4430965036154,9,9,7229.0,2373.0,5.118273620141099,31.17996956702172,0.606216996821714,False,False,3.9999999999999996,12.999833,6.5133967,1268.0,9.0,9,1.319020352364041
-56,True,ACGTCCTTCTGTGCGG-1,ACGTCCTTCTGTGCGG-1,1.9978282107380063,15.0,-0.1291482250750845,-0.016104516056073763,1.9032989,19.121885,18,G1,1221.429703116417,5,5,4309.0,1184.0,10.95381759108842,43.420747273149225,0.7231815237262016,False,False,8.8,-4.5405436,15.926964,1167.0,9.0,5,0.8054144824310264
-57,True,ACGTCCTTCTTAGCCC-1,ACGTCCTTCTTAGCCC-1,1.8795040235680138,15.0,-0.10417673935115795,-0.012292111806674916,2.491866,2.831029,12,G1,1674.4842343777418,1,1,5920.0,1719.0,5.236486486486487,41.84121621621622,0.611295414830068,False,False,4.0,10.894972,8.740703,1295.0,9.0,1,1.1922775842747282
-58,True,ACTATTCAGCAATTAG-1,ACTATTCAGCAATTAG-1,1.8716250208397454,15.0,-0.09873974141291997,-0.002366597299575949,3.032851,5.5111866,12,G1,1637.1956877335906,1,1,6095.0,1868.0,4.446267432321575,39.96718621821165,0.6112315477687476,False,False,2.9,9.682619,9.356861,1495.0,9.0,1,0.9875659855422305
-59,True,ACTATTCAGGCCTAGA-1,ACTATTCAGGCCTAGA-1,1.9822908718248735,15.0,-0.07645455319873923,-0.015195259796012724,-3.9619758,-3.8943422,17,G1,1745.468310981989,2,2,9800.0,2898.0,8.448979591836734,17.336734693877553,0.9573610689035008,False,False,5.3,-14.678455,-4.0126634,2488.0,9.0,2,2.3005542990075627
-60,True,ACTATTCAGTAATTGG-1,ACTATTCAGTAATTGG-1,1.9494896771752799,15.0,-0.10974118484265478,-0.010726607001107828,7.0587835,9.195922,15,G1,1266.4743803143501,8,10,6784.0,1656.0,13.413915094339623,44.23643867924528,0.6187331211021164,False,False,3.8,14.198982,10.02298,2064.0,9.0,8,1.1339888656653048
-61,True,ACTATTCCAATCTCGA-1,ACTATTCCAATCTCGA-1,1.989709565229192,15.0,-0.09595017547036037,-0.01685608385535281,-10.3224325,-7.364411,10,G1,1196.57967813313,3,7,9841.0,2873.0,9.236866172136978,16.380449141347423,0.9747064063278525,False,False,3.5,-12.972481,-8.096525,2289.0,9.0,3,1.3745075350529723
-62,True,ACTATTCCAATCTGCA-1,ACTATTCCAATCTGCA-1,1.8640990373110142,15.0,-0.07787698925620688,-0.013662516468510067,9.99852,-7.2948914,23,G1,1183.6172689944506,6,8,5536.0,1933.0,7.22543352601156,26.28251445086705,0.1917290687991576,False,False,3.6999999999999997,13.940358,-6.1188426,1038.0,9.0,6,0.8054144824310264
-63,True,ACTATTCCACTTCAGA-1,ACTATTCCACTTCAGA-1,1.8587843246922044,15.0,-0.08531783024404366,-0.014554262927601043,9.2084,-1.940629,23,G1,1009.1727796345949,6,8,12128.0,3367.0,5.722295514511873,28.91655672823219,0.46669974421030835,False,False,3.5000000000000004,14.163376,0.57997674,1364.0,9.0,6,0.8054144824310264
-64,True,ACTATTCCATCCTTCG-1,ACTATTCCATCCTTCG-1,1.8657433984472056,15.0,-0.06689007824535846,-0.0226436777151306,8.529536,-0.15996589,22,G1,1041.2424451112747,6,8,4649.0,1960.0,8.216820821682083,16.390621639062164,0.5387803478399062,False,False,4.3,13.964401,2.8545892,1390.0,9.0,6,1.508013832392216
-65,True,ACTATTCGTAACTTCG-1,ACTATTCGTAACTTCG-1,1.9316580492555695,15.0,-0.10560344996018134,-0.008961632216795152,6.348596,7.5691085,27,G1,1479.0055207759142,8,10,7630.0,1685.0,9.698558322411534,47.929226736566186,0.6141173864443679,False,False,3.5999999999999996,12.718884,10.227739,1071.0,9.0,8,1.4603486448954297
-66,True,ACTATTCGTATGCGTT-1,ACTATTCGTATGCGTT-1,2.033584666481111,15.0,-0.060911720310231995,-0.023422260347474202,-8.126636,-0.1539,14,G1,1594.126646950841,4,3,6794.0,2379.0,17.868707683249927,12.275537238740064,0.9730471548675709,False,False,3.5,-15.053409,-0.92805713,2106.0,9.0,4,1.3211486736114244
-67,True,ACTATTCGTCACGCTG-1,ACTATTCGTCACGCTG-1,2.0202172791313604,15.0,-0.06407273849134315,-0.023838165308345618,-4.5382743,-4.9442253,10,G1,1714.09015481174,3,2,5698.0,2233.0,13.75921375921376,11.53036153036153,0.9601567997651083,False,False,4.4,-14.511831,-4.1597834,1819.0,9.0,3,0.8054144824310264
-68,True,ACTATTCGTGCCTTCT-1,ACTATTCGTGCCTTCT-1,1.9752282667226142,15.0,-0.05783954470884885,-0.02764999751763641,-12.387285,-4.460261,7,G1,990.4759607613087,3,3,7459.0,2720.0,9.760021450596595,10.202440005362648,0.9932245030405271,False,False,3.7,-18.842728,-4.447149,2673.0,10.0,3,1.1258223601919735
-69,True,ACTATTCGTTGGGATG-1,ACTATTCGTTGGGATG-1,1.9373940662589682,15.0,-0.10287765338407372,-0.023579749884573845,8.379806,-7.64282,11,G1,958.1159641593695,7,6,15005.0,3775.0,8.250583138953681,21.552815728090636,0.2640752669995253,True,True,4.6,14.483362,-7.0831995,3889.0,9.0,7,0.9478035426099897
-70,True,ACTATTCTCACAAGAA-1,ACTATTCTCACAAGAA-1,2.000246832278422,15.0,-0.06294029999704105,-0.015643994021842748,-3.1175537,-1.7770852,19,G1,1478.637172743678,2,2,7134.0,2451.0,7.429212223156714,14.872441827866554,0.9477372583707036,False,False,3.5,-14.024736,-2.2256036,1971.0,9.0,2,1.4944681780875912
-71,True,ACTATTCTCCGGACGT-1,ACTATTCTCCGGACGT-1,1.933901768862953,15.0,-0.10583325437329355,-0.013787787504780007,0.6444152,13.119549,2,G1,1015.092177465558,5,5,14462.0,3138.0,8.207716774996543,41.76462453325958,0.7239050480111444,False,False,5.199999999999999,-0.69497895,16.233822,1006.0,9.0,5,1.0829206595588365
-72,True,ACTATTCTCGCGGTAC-1,ACTATTCTCGCGGTAC-1,1.9592700498166693,15.0,0.01474474761334097,-0.04241681591101894,0.62224597,6.7831917,5,G2M,1621.0989419817924,1,1,1648.0,807.0,23.361650485436893,4.368932038834951,0.620333207693447,False,True,5.0,9.962697,12.011572,1295.0,9.0,1,1.2352060533237517
-73,True,ACTATTCTCTCCAATT-1,ACTATTCTCTCCAATT-1,1.8863013495422911,15.0,-0.10731155866669423,-0.014151722775435874,9.522674,-0.0152956275,25,G1,995.6393796652555,6,8,9866.0,2763.0,5.98013379282384,30.356780863571863,0.5459177420541588,False,True,3.6000000000000005,13.737263,2.547135,1663.0,9.0,6,0.9413669634110602
-74,True,ACTATTCTCTCGAGTA-1,ACTATTCTCTCGAGTA-1,2.0265210746159714,15.0,-0.06351269107622752,-0.03299969503730104,-10.213072,-2.7101417,7,G1,1647.141313701868,3,3,8532.0,2692.0,11.75574308485701,10.864978902953586,0.9867050689808768,False,False,4.5,-16.388884,-3.3517702,2676.0,9.0,3,1.462783751316627
-75,True,ACTATTCTCTGCCCTA-1,ACTATTCTCTGCCCTA-1,1.879849804057742,15.0,-0.07067451416056399,-0.01706827905185314,8.046637,2.115622,8,G1,1262.433500006795,9,9,10515.0,3083.0,6.533523537803139,20.542082738944366,0.62643940436049,False,False,4.6000000000000005,13.081439,5.5957775,1724.0,9.0,9,0.9630688780490345
-77,True,ACTGTGAAGAGAGCGG-1,ACTGTGAAGAGAGCGG-1,1.9724923648909782,15.0,-0.10025597602350028,-0.020843644461494906,-4.2718983,-6.330891,9,G1,1388.5975793004036,2,2,11851.0,3092.0,10.395747194329592,27.70230360307147,0.935009614164887,False,False,3.0,-15.903538,-6.389505,1096.0,9.0,2,1.1645165933714983
-78,True,ACTGTGAAGCAGGTCA-1,ACTGTGAAGCAGGTCA-1,1.9947385828717523,15.0,-0.057442459041111574,-0.02543645849304551,-3.15415,-0.69599515,19,G1,1275.5968184620142,2,4,6296.0,2337.0,9.831639135959339,12.53176620076239,0.9446992692606417,False,False,3.8,-13.189667,-1.1440012,1180.0,9.0,2,1.0525539774112422
-81,True,ACTGTGAAGGATCACG-1,ACTGTGAAGGATCACG-1,1.9266578281674922,15.0,-0.09814519374711683,-0.016400717829896935,8.743321,4.0973234,26,G1,1305.6378846168518,9,9,7783.0,1881.0,10.073236541179494,41.68058589232944,0.6179944914865506,False,False,4.999999999999999,11.59279,6.0011744,1886.0,9.0,9,0.8054144824310264
-82,True,ACTGTGAAGGCGTTAG-1,ACTGTGAAGGCGTTAG-1,1.8881178355891592,15.0,-0.07841461162284163,-0.01923006035929569,9.20645,-14.0381365,21,G1,1090.366791844368,7,6,7287.0,2630.0,10.37463976945245,20.296418279127213,0.0006390215367841526,False,False,8.6,9.978135,-12.121716,970.0,9.0,7,1.6743943618986665
-83,True,ACTGTGAAGGCTCAAG-1,ACTGTGAAGGCTCAAG-1,1.9475653696109883,15.0,-0.08942293937292428,-0.00784013955448407,-5.9713006,-1.4599253,9,G1,1445.5336554348469,2,2,14577.0,3689.0,6.997324552377032,25.17664814433697,0.9640010456652016,False,False,2.5,-15.794286,-1.888785,3456.0,9.0,2,1.0949590491706929
-85,True,ACTGTGAAGTGGCCTC-1,ACTGTGAAGTGGCCTC-1,1.965260042299594,15.0,-0.07500620368312884,-0.01460627014926029,-4.0513477,-5.9482374,19,G1,1425.8415170162916,2,2,6207.0,2303.0,10.165941678749798,16.15917512485903,0.9557728335364338,False,False,3.5,-14.791169,-6.24024,2325.0,9.0,2,0.8054144824310264
-86,True,ACTGTGACAATTAGGA-1,ACTGTGACAATTAGGA-1,1.9327927618413883,15.0,-0.0998966036500815,-0.0156442656982727,0.9952308,14.753332,8,G1,1109.8356473222375,9,5,9303.0,2672.0,11.856390411695152,33.47307320219284,0.697008395033082,False,True,3.0,-1.5004996,14.2016325,2140.0,9.0,9,0.8054144824310264
-87,True,ACTGTGACACAAATCC-1,ACTGTGACACAAATCC-1,1.9112776409873067,15.0,-0.10435065737854013,-0.013943357410080494,6.662921,8.214513,27,G1,1393.8291646540165,8,10,6333.0,1711.0,7.468814148113059,42.74435496605084,0.6154113585046815,False,False,3.2,13.2977705,10.739697,914.0,10.0,8,1.2151900768914132
-88,True,ACTGTGACACCAGCGT-1,ACTGTGACACCAGCGT-1,1.8843854288840542,15.0,-0.031549382327760445,0.0666209967199984,9.221009,-13.117173,21,S,1133.3619707673788,7,6,3043.0,1549.0,14.919487348011831,9.924416694051923,0.005305961053718569,False,False,6.8999999999999995,10.457469,-11.725558,1411.0,9.0,7,1.3975645918688595
-89,True,ACTGTGACACTATGTG-1,ACTGTGACACTATGTG-1,1.8773673584448123,15.0,-0.08073465970932901,-0.01179629071497552,7.3555627,-1.408197,22,G1,1102.344379708171,6,8,5918.0,1991.0,8.110848259547144,30.111524163568774,0.4989222204860849,False,False,5.3999999999999995,14.456866,1.6121007,1759.0,9.0,6,1.382121204918996
-90,True,ACTGTGACAGCTGCCA-1,ACTGTGACAGCTGCCA-1,1.9733803584821725,15.0,-0.0904900306933119,-0.034761915743397646,-4.5596476,-7.0749164,19,G1,1306.0351961255074,2,2,8420.0,2891.0,10.225653206650831,15.344418052256533,0.9559341471175475,False,False,3.5999999999999996,-15.021027,-7.0312304,2326.0,9.0,2,0.9591061456682548
-91,True,ACTGTGACATCCTATT-1,ACTGTGACATCCTATT-1,1.9484008453405508,15.0,-0.09501456956603493,-0.006045972293970855,2.3803263,16.69263,2,G1,1016.6927028298378,5,5,13443.0,3371.0,7.275161794242357,36.40556423417392,0.7205631256568056,False,True,4.8999999999999995,-3.5152087,14.122371,1548.0,9.0,5,0.9575994643424515
-92,True,ACTGTGACATCCTTCG-1,ACTGTGACATCCTTCG-1,1.9115047521735686,15.0,-0.09690825422658138,-0.013000148693138524,7.1932964,4.5039196,8,G1,1397.0810600668192,9,1,6604.0,1925.0,10.902483343428225,36.2961841308298,0.6097175198073198,False,True,4.3999999999999995,12.802611,7.3782597,1581.0,9.0,9,0.8054144824310264
-93,True,ACTGTGACATTATGCG-1,ACTGTGACATTATGCG-1,1.867015604702736,15.0,-0.09069105388635482,-0.014058363906306028,10.109635,-9.319949,21,G1,1220.7012396007776,7,6,5242.0,2032.0,6.52422739412438,24.13201068294544,0.1263544505636979,False,False,6.199999999999999,13.481164,-8.171996,1071.0,9.0,7,0.8054144824310264
-94,True,ACTGTGAGTATGAGCG-1,ACTGTGAGTATGAGCG-1,1.9193074794065383,15.0,-0.0808251038682122,-0.018059408229311142,8.985912,-8.727263,11,G1,1072.5891362279654,7,6,15744.0,3906.0,9.711636178861788,18.39430894308943,0.18548127965580266,False,False,3.4,14.480761,-8.508059,3126.0,10.0,7,0.8054144824310264
-95,True,ACTGTGAGTCAAGCCC-1,ACTGTGAGTCAAGCCC-1,1.9380479095353644,15.0,-0.046582736536601534,-0.008014468924024364,0.013417046,6.093354,5,G1,1168.5311177372932,1,1,2717.0,1359.0,16.930437983069563,8.870077291129922,0.6105760287826422,False,False,1.5000000000000002,8.699713,10.604262,1577.0,9.0,1,0.8054144824310264
-96,True,ACTGTGAGTCCACACG-1,ACTGTGAGTCCACACG-1,1.9218758682230412,15.0,-0.11418510520261387,-0.005908749779647763,0.97859776,15.535449,2,G1,1104.8479398041964,5,5,5657.0,1582.0,6.487537564079901,42.425313770549764,0.719205159237297,False,False,8.5,-1.6820934,14.997838,1397.0,9.0,5,0.9586635921345452
-97,True,ACTGTGAGTGCACAAG-1,ACTGTGAGTGCACAAG-1,2.0076344269936413,15.0,-0.06553093191795308,-0.025316597127778916,-6.0604887,1.2074281,14,G1,1563.576756849885,4,4,6786.0,2329.0,13.188918361332155,13.144709696433834,0.9558605311241337,True,True,4.8999999999999995,-15.19085,0.29295868,1902.0,9.0,4,1.6224621336919873
-98,True,ACTGTGAGTTCCGTTC-1,ACTGTGAGTTCCGTTC-1,2.0513904483473366,15.0,-0.04704167494865169,-0.030341946199551377,-5.058761,2.9278603,14,G1,1476.4966849833727,4,4,6750.0,2508.0,13.6,10.592592592592593,0.9028007977447524,False,False,5.699999999999999,-13.447811,1.908975,1940.0,9.0,4,0.8054144824310264
-99,True,ACTGTGATCAATCCAG-1,ACTGTGATCAATCCAG-1,1.9677889995869646,15.0,-0.07736155571630421,-0.03320218996062414,-3.877977,-3.0950725,17,G1,1658.9715236574411,2,2,11826.0,3527.0,5.361068831388466,16.007102993404363,0.958462505027154,False,False,4.1,-14.66154,-3.169422,2539.0,9.0,2,1.5869652682386162
-100,True,ACTGTGATCTAGTCAG-1,ACTGTGATCTAGTCAG-1,1.9466893161646306,15.0,-0.09076098358083046,-0.023074522113504914,-7.0152774,-5.3941374,13,G1,1143.41269184649,2,2,12805.0,3644.0,7.254978524014057,16.876220226474032,0.9634746731867095,False,False,3.1000000000000005,-16.693508,-6.4157534,2625.0,9.0,2,0.8054144824310264
-101,True,ACTGTGATCTCATAGG-1,ACTGTGATCTCATAGG-1,2.124130229296572,15.0,-0.0554255920982218,-0.034575277293723894,-8.716258,-12.356273,16,G1,642.6370834708214,2,2,1404.0,481.0,11.253561253561253,2.92022792022792,0.9321543439904747,True,True,6.3999999999999995,-15.184017,-11.536053,271.0,9.0,2,1.234266557605285
-102,True,ACTGTGATCTCTAGGA-1,ACTGTGATCTCTAGGA-1,1.9517632949957404,15.0,-0.10409763850698597,-0.017588243472378882,-0.20712575,12.735458,2,G1,853.510957852006,5,5,10099.0,2507.0,9.456381819982177,40.568373106248146,0.7269787266345247,False,False,6.5,-0.04312111,16.949722,951.0,9.0,5,1.0581504017760734
-103,True,ACTTTCAAGAAATTGC-1,ACTTTCAAGAAATTGC-1,1.8964340325655307,15.0,-0.06780674499543277,-0.0030607945515295637,10.020077,-11.785579,21,G1,1239.8609247505665,7,6,3610.0,1547.0,11.495844875346261,18.44875346260388,0.03854061697907293,False,False,5.599999999999999,12.014874,-11.329532,637.0,9.0,7,0.8054144824310264
-104,True,ACTTTCAAGATGCTTC-1,ACTTTCAAGATGCTTC-1,1.8942135405054608,15.0,-0.10400421670057675,-0.013261345579848823,6.6327767,10.295502,15,G1,1066.0915660709143,8,10,6582.0,1911.0,6.7760559100577336,41.44636888483743,0.6307472862668468,False,False,5.7,15.171917,9.373586,857.0,9.0,8,0.8054144824310264
-105,True,ACTTTCAAGCACTCTA-1,ACTTTCAAGCACTCTA-1,1.892639797434566,15.0,-0.1008618131353074,-0.008951953980983942,9.652359,3.5414011,8,G1,1293.3827391117811,9,9,6657.0,1898.0,7.766261078563918,35.2410995944119,0.6068761448463132,False,True,5.1000000000000005,12.074815,4.759704,1853.0,9.0,9,1.3232212714969998
-106,True,ACTTTCAAGCCTCTGG-1,ACTTTCAAGCCTCTGG-1,1.8963406512652279,15.0,-0.11331042571751636,-0.010063135235411736,7.260237,6.2923594,25,G1,1345.7832995951176,6,10,6023.0,1691.0,4.51602191598871,44.59571642038851,0.6067439093096848,False,False,4.5,14.117527,8.71284,1344.0,9.0,6,0.8054144824310264
-107,True,ACTTTCAAGCGACATG-1,ACTTTCAAGCGACATG-1,1.9197130280788057,15.0,-0.09652048342705816,-0.01932593184048986,8.416509,10.292564,15,G1,1147.810196891427,8,10,3818.0,1022.0,7.778941854374017,45.52121529596648,0.6185634924820022,False,False,3.6,14.383933,10.498906,1110.0,9.0,8,0.8054144824310264
-108,True,ACTTTCAAGGTTGGAC-1,ACTTTCAAGGTTGGAC-1,1.9582891171491301,15.0,-0.07747401135661072,-0.027039160317768565,-3.6416376,-1.9445916,19,G1,1529.2768681645393,2,2,8149.0,2774.0,7.240152165909928,17.315007976438828,0.9542288044611094,False,False,2.6,-14.080324,-2.116249,1936.0,9.0,2,1.256546251028587
-109,True,ACTTTCAAGTCATCCA-1,ACTTTCAAGTCATCCA-1,2.0408762159858713,15.0,-0.062080735990089245,-0.017013267061355183,-5.266631,2.7807066,14,G1,1482.9733168929815,4,4,10549.0,3125.0,8.49369608493696,15.555976869845482,0.9356444355957817,False,False,5.3999999999999995,-14.663025,1.9532293,1672.0,9.0,4,1.1702966058509467
-110,True,ACTTTCACAAATGCGG-1,ACTTTCACAAATGCGG-1,1.901485540160068,15.0,-0.058256046977676286,-0.006145748115123761,1.2737167,7.560882,5,G1,1675.4764136373997,1,1,1497.0,666.0,19.57247828991316,8.082832331329325,0.6142047373708482,False,False,6.299999999999999,9.092782,11.775453,1395.0,9.0,1,1.1855478963496768
-111,True,ACTTTCACAAGAGAGA-1,ACTTTCACAAGAGAGA-1,1.9881571699752574,15.0,-0.07865278525264902,-0.01690831252383268,-10.430816,-3.6354675,1,G1,1231.4007664173841,3,3,10959.0,3140.0,9.289168719773702,16.379231681722786,0.9905785193245781,False,False,3.5,-14.445804,-4.201788,2687.0,9.0,3,1.397034968469201
-112,True,ACTTTCACAAGCCATT-1,ACTTTCACAAGCCATT-1,1.88816766793761,15.0,-0.09418699997783833,-0.01256697558650215,6.0996795,2.8216445,12,G1,1482.4616336226463,1,9,9256.0,2690.0,7.443820224719101,28.900172860847018,0.6157231479458831,False,False,3.9,11.829315,7.551596,1023.0,9.0,1,0.8054144824310264
-113,True,ACTTTCACAAGTAGTA-1,ACTTTCACAAGTAGTA-1,1.991366036019676,15.0,-0.07184883561454661,-0.02700686157381342,-8.155318,-8.60291,10,G1,1154.6786435991526,3,7,8787.0,2864.0,12.029133947877547,14.191419141914192,0.9645025588677265,False,False,3.6999999999999997,-12.797674,-9.291117,2492.0,9.0,3,1.0189772375382224
-114,True,ACTTTCACACTAAACC-1,ACTTTCACACTAAACC-1,1.9636512200179288,15.0,-0.05073034646695112,-0.028989124171488693,-5.852978,-4.9155946,10,G1,1677.563086733222,3,2,9092.0,3038.0,10.679718433787945,10.481742190937087,0.9600669699873128,False,False,4.500000000000001,-14.552683,-4.9078383,3471.0,9.0,3,1.0245455485087864
-115,True,ACTTTCACACTTCAGA-1,ACTTTCACACTTCAGA-1,1.8664332532387522,15.0,-0.08044743458992354,-0.01644939384433121,9.4461355,-10.887075,21,G1,1301.580148652196,7,6,4629.0,1736.0,10.153380859796933,19.356232447612875,0.05701591663650824,False,False,7.499999999999999,12.121291,-10.262825,1426.0,9.0,7,1.6106171771038273
-116,True,ACTTTCACAGGGAATC-1,ACTTTCACAGGGAATC-1,1.9010610756800377,15.0,-0.10819248165349397,-0.02128464435717434,3.5717866,6.3991256,6,G1,1656.1327080205083,1,1,7548.0,1981.0,7.180710121886593,43.693693693693696,0.6126634191717819,False,False,3.5999999999999996,9.67276,9.791462,1221.0,9.0,1,1.1916692492517824
-117,True,ACTTTCACATCAGTGT-1,ACTTTCACATCAGTGT-1,1.9663578487792606,15.0,-0.07449996334128788,-0.027096054738262126,-4.3381352,-5.009088,17,G1,1671.6289958059788,2,2,9833.0,3013.0,8.868097223634699,15.051357673141462,0.9590278667574874,False,False,4.8,-14.919808,-5.258017,1150.0,9.0,2,1.0538656653031109
-118,True,ACTTTCAGTAGTGTGG-1,ACTTTCAGTAGTGTGG-1,2.0028762645080698,15.0,-0.07890328216804067,-0.032740991478855556,-6.6934385,1.6961279,14,G1,1490.9386226385832,4,4,6864.0,2438.0,7.328088578088578,15.224358974358974,0.9579899192793372,False,False,4.6000000000000005,-16.009384,0.57214844,2694.0,9.0,4,1.1225541884034926
-119,True,ACTTTCAGTATGCGTT-1,ACTTTCAGTATGCGTT-1,1.9699978206779976,15.0,-0.0662849908772989,-0.026505389467882976,-7.35479,-2.8719203,10,G1,1413.7459429949522,3,3,7966.0,2669.0,10.369068541300527,13.670600050213407,0.9622293142616853,False,False,3.1999999999999997,-13.959063,-2.7390842,3405.0,9.0,3,1.369311926511253
-120,True,ACTTTCAGTGATTGGG-1,ACTTTCAGTGATTGGG-1,1.9241425848798015,15.0,-0.1011549593459831,-0.012651399997797734,12.66525,-9.2177105,11,G1,1136.7876921594143,7,6,11924.0,2679.0,10.45790003354579,40.77490774907749,0.13605310054438416,False,False,7.6,12.360453,-6.8366046,1724.0,9.0,7,0.9378839828532496
-121,True,ACTTTCAGTGTCTAAC-1,ACTTTCAGTGTCTAAC-1,1.9702176253231498,15.0,-0.07251054342496285,-0.02948290900829872,-7.330685,-0.59754944,10,G1,1540.2221986353397,3,4,11030.0,3523.0,10.28105167724388,13.880326382592928,0.9620596072637166,False,False,3.6999999999999997,-14.191189,-1.4582981,1724.0,9.0,3,1.282510015549939
-122,True,ACTTTCAGTTACACAC-1,ACTTTCAGTTACACAC-1,1.9002754240094941,15.0,-0.08767007353670596,-0.013301066421987593,9.55026,-4.678605,23,G1,1138.1989960074425,6,8,4887.0,1550.0,10.272150603642316,34.8475547370575,0.33490607246167725,False,False,6.8999999999999995,14.273514,-2.7547255,1639.0,9.0,6,1.15795119741862
-123,True,ACTTTCATCAAAGGTA-1,ACTTTCATCAAAGGTA-1,2.0053525793108604,15.0,-0.045531502407777744,-0.03391797980135789,-6.6274514,-5.3536544,1,G1,1493.3843783438206,3,2,8958.0,2960.0,10.069211877651261,12.51395400759098,0.9665439751153807,False,False,3.8000000000000003,-15.950292,-5.7020864,2041.0,9.0,3,1.0671095548959224
-124,True,ACTTTCATCTGGCTGG-1,ACTTTCATCTGGCTGG-1,1.9458778588056291,15.0,-0.08240510296775536,-0.03113490966173404,-9.386459,-6.8633184,10,G1,1263.0734204798937,3,7,11502.0,3637.0,7.311771865762476,14.154060163449834,0.9705164495937822,False,False,4.2,-12.398723,-7.5927157,3132.0,9.0,3,1.4321073132235262
-125,True,ACTTTCATCTGTGCGG-1,ACTTTCATCTGTGCGG-1,1.9705704044143308,15.0,-0.06566028373629774,-0.023336295207385723,-6.857171,-6.8440337,13,G1,1225.849198102951,2,2,8816.0,3031.0,10.401542649727768,11.195553539019963,0.968040228924521,False,False,3.9,-17.115747,-6.469414,1505.0,9.0,2,0.9813075068299141
-126,True,AGACACTAGACCATTC-1,AGACACTAGACCATTC-1,1.9655597333551948,15.0,-0.06527099550355364,-0.026739638390123832,-7.673124,-4.6967716,10,G1,1194.2258914858103,3,2,9620.0,3147.0,7.401247401247401,13.212058212058212,0.969023276892191,False,False,3.3,-15.258705,-5.503738,2726.0,9.0,3,1.3517423825432624
-127,True,AGACACTAGCAATTAG-1,AGACACTAGCAATTAG-1,1.9302685623358062,15.0,-0.053527832754029434,-0.0183329024064674,9.244272,-9.792043,11,G1,1273.247906446457,7,6,3122.0,1424.0,19.218449711723256,11.915438821268417,0.08989631586234813,False,False,6.0,13.009545,-9.209364,1897.0,9.0,7,1.1896827199692022
-128,True,AGACACTAGGGAGGTG-1,AGACACTAGGGAGGTG-1,1.9417296881973234,15.0,-0.08105302678946089,-0.01725997842502697,8.13858,-3.4224825,22,G1,1204.7062613368034,6,8,3375.0,1156.0,14.42962962962963,31.644444444444446,0.43964569569115763,False,False,7.1000000000000005,14.278927,-0.22619778,1320.0,9.0,6,1.5275692225903867
-129,True,AGACACTAGTATAACG-1,AGACACTAGTATAACG-1,1.8929122814307386,15.0,-0.08905174202953123,0.001448738968399449,0.30511925,5.0811133,5,S,1037.2865631952882,1,1,5074.0,1441.0,4.237288135593221,46.68900275916437,0.6116234266318952,False,False,3.5,7.9972086,9.589724,1390.0,9.0,1,0.9684021274307453
-130,True,AGACACTCAACGCATT-1,AGACACTCAACGCATT-1,1.893928517980608,15.0,-0.10179908867452041,-0.0051530811888401,5.120653,0.25168934,22,G1,1296.669214606285,6,8,6666.0,1985.0,8.04080408040804,36.108610861086106,0.576448042763361,False,False,4.2,13.659927,6.323415,1144.0,9.0,6,1.1557748892262265
-132,True,AGACACTCACATGAAA-1,AGACACTCACATGAAA-1,1.968532508658971,15.0,-0.08101819371944596,-0.02549711287575365,-10.421326,-3.6499476,1,G1,1371.0099867135286,3,3,10582.0,3281.0,10.584010584010583,13.362313362313362,0.9921049946843041,False,False,3.0,-17.10485,-4.2063203,3704.0,9.0,3,1.0146453417455004
-133,True,AGACACTCACCCTCTA-1,AGACACTCACCCTCTA-1,1.884263406559217,15.0,-0.06645754877474479,-0.00555304604059699,9.640648,-5.695478,23,G1,1161.5730865597725,6,8,4271.0,1575.0,8.054319831421212,27.604776398969797,0.2893233672292946,False,False,7.0,14.281711,-4.0700774,958.0,9.0,6,1.3433065262464856
-134,True,AGACACTCACTACGGC-1,AGACACTCACTACGGC-1,1.9759764583256292,15.0,-0.07223219903202369,-0.030847459369281738,-7.0190253,-0.9695707,14,G1,1170.822343185544,4,4,4138.0,1869.0,11.213146447559208,10.850652489125181,0.9575753820613987,False,False,3.1,-14.04022,-1.755754,2475.0,9.0,4,0.8054144824310264
-135,True,AGACACTCATCCTATT-1,AGACACTCATCCTATT-1,1.983532850304823,15.0,-0.06614508442306528,-0.021979802477902946,-5.896441,-5.383041,13,G1,1703.8182702213526,2,2,9867.0,3152.0,10.823958650045606,16.205533596837945,0.9641172484931678,False,False,4.0,-16.53302,-5.7506332,1979.0,9.0,2,1.3862740530887567
-136,True,AGACACTGTAATGCTC-1,AGACACTGTAATGCTC-1,1.8954734374426594,15.0,-0.1091875587277592,-0.011293781999864553,7.464264,3.8092983,8,G1,1386.8500839993358,9,9,7086.0,1827.0,6.209427039232289,46.401354784081285,0.6104120718317447,False,False,3.8,12.260687,6.1861205,1187.0,9.0,9,0.8054144824310264
-137,True,AGACACTGTACGGGAT-1,AGACACTGTACGGGAT-1,1.872303862166951,15.0,-0.08881781772910148,-0.022078941752193076,6.116325,-0.75833714,22,G1,1027.3127206414938,6,8,4703.0,1604.0,7.761003614714013,33.638103338294705,0.5435426938549931,False,False,4.199999999999999,14.553701,3.1371253,1258.0,9.0,6,0.8054144824310264
-138,True,AGACACTGTAGCGTAG-1,AGACACTGTAGCGTAG-1,2.068224709385442,15.0,-0.07726599635021306,-0.02468847820666064,-9.665532,-3.0413587,1,G1,1485.682854756713,3,3,5601.0,2093.0,18.425281199785754,9.891090876629173,0.9857468177817974,False,False,4.1,-15.293498,-3.3079426,2934.0,9.0,3,0.9391727281283155
-139,True,AGACACTGTATACCTG-1,AGACACTGTATACCTG-1,2.0337654921907915,15.0,-0.035614698010714776,-0.039291236786737314,0.4117912,8.699431,5,G1,1240.5807669907808,1,1,1532.0,771.0,30.026109660574413,7.506527415143603,0.6270422431854095,False,False,4.699999999999999,8.644056,12.775502,1534.0,9.0,1,0.8054144824310264
-141,True,AGACACTGTGTCTAAC-1,AGACACTGTGTCTAAC-1,1.8820001822323045,15.0,-0.07368941567341505,-0.015534136508612188,9.900773,-6.8847203,23,G1,1165.6580424457788,6,8,4744.0,1700.0,9.064080944350758,22.84991568296796,0.2225738408939001,False,False,5.800000000000001,13.998683,-5.293639,1188.0,9.0,6,1.3391534867442854
-142,True,AGACACTTCAAAGGTA-1,AGACACTTCAAAGGTA-1,1.9269118204392879,15.0,-0.08089074840278257,-0.015076211964687398,0.5829504,5.2405825,5,G1,1041.7845131754875,1,1,5667.0,1459.0,9.175930827598377,44.150344097406034,0.6105809437307363,False,False,4.199999999999999,8.668133,9.110955,690.0,9.0,1,1.1192238731153785
-143,True,AGACACTTCTTAGCAG-1,AGACACTTCTTAGCAG-1,1.9441010837991037,15.0,-0.11096538416494359,-0.004182940305674839,-0.6257613,15.682435,2,G1,1148.8032605946064,5,5,5891.0,1753.0,8.962824647767782,40.1120353080971,0.7262852959990129,False,False,5.1,-0.7066064,14.042106,1299.0,9.0,5,0.8054144824310264
-144,True,AGGGAGTAGCACCTGC-1,AGGGAGTAGCACCTGC-1,1.957833556890565,15.0,-0.1010389775011694,-0.013971709839682628,-7.7296343,-8.169029,3,G1,1251.8619155734777,2,7,10688.0,3002.0,10.086077844311378,16.88809880239521,0.9601241395770446,False,False,3.7999999999999994,-13.7501955,-8.747741,3048.0,9.0,2,1.3713055388634732
-145,True,AGGGAGTAGCACTCTA-1,AGGGAGTAGCACTCTA-1,1.906197269797348,15.0,-0.09310814853762016,-0.02396000367308873,10.165238,-5.8065104,23,G1,1141.7077158242464,6,8,4355.0,1426.0,7.715269804822044,34.489092996555684,0.25257303345337373,False,False,6.900000000000001,13.918837,-4.66509,923.0,9.0,6,1.3360037577711967
-146,True,AGGGAGTAGGATCACG-1,AGGGAGTAGGATCACG-1,1.9059903968497969,15.0,-0.10153118831231757,-0.007672518885136112,1.7252922,2.5899122,12,G1,1354.4148722738028,1,1,4929.0,1301.0,4.9502941773179145,51.57232704402516,0.6187537231918308,False,False,3.7,10.973508,9.19703,1519.0,9.0,1,1.0069201340880614
-147,True,AGGGAGTAGGGCGAAG-1,AGGGAGTAGGGCGAAG-1,1.9824154310308426,15.0,-0.06755085394336002,-0.012481272802121905,-8.289623,-1.2659498,10,G1,1503.642642363906,3,3,7649.0,2512.0,8.301738789384233,17.505556281866912,0.9666225764192599,False,False,4.199999999999999,-13.826553,-2.6719792,2334.0,9.0,3,0.8054144824310264
-148,True,AGGGAGTAGGTTACAA-1,AGGGAGTAGGTTACAA-1,1.9916368734247434,15.0,-0.08289469966228921,-0.019197495020508572,-7.44249,-3.4943151,10,G1,1436.41731454432,3,3,8188.0,2670.0,8.121641426477773,14.521250610649732,0.9715883030295386,True,True,3.6000000000000005,-15.73977,-3.5475662,1623.0,9.0,3,1.4399161372346798
-149,True,AGGGAGTAGGTTGGAC-1,AGGGAGTAGGTTGGAC-1,2.045645307965738,15.0,-0.06708119969053972,-0.00813391840403057,-7.186732,1.3808044,4,G1,1546.6160161197186,4,4,5178.0,1981.0,9.096176129779838,11.181923522595596,0.9567986740215486,False,False,5.4,-14.513669,0.23000188,1889.0,9.0,4,1.370249510524696
-150,True,AGGGAGTAGTAGTCTC-1,AGGGAGTAGTAGTCTC-1,1.9407983827117277,15.0,-0.09045136318029473,-0.017473515399601017,-12.124252,-2.9278862,7,G1,998.6132479161024,3,3,12745.0,3429.0,6.786975284425265,17.779521380933698,0.9999341963321966,True,True,4.7,-18.799204,-3.3198898,3176.0,9.0,3,0.937235816181699
-152,True,AGGGAGTGTAGCGTAG-1,AGGGAGTGTAGCGTAG-1,2.0300153716423677,15.0,-0.06741954001546069,-0.029307386527127985,-9.151705,-1.4656878,7,G1,1656.9634977132082,3,3,7288.0,2474.0,14.297475301866081,13.80351262349067,0.9844431669873921,False,False,5.199999999999999,-15.536969,-2.2467148,4014.0,9.0,3,1.7076582637188416
-154,True,AGGGAGTGTTCGGCTG-1,AGGGAGTGTTCGGCTG-1,1.94650394794739,15.0,-0.10189891699436926,-0.019138538736746657,-7.066377,-9.656598,3,G1,1184.8417666703463,2,2,9938.0,2639.0,8.200845240491045,23.032803380961965,0.949329502399926,False,False,3.1000000000000005,-14.686593,-9.002657,2290.0,9.0,2,1.110800675967786
-155,True,AGGGAGTTCAAGAAAC-1,AGGGAGTTCAAGAAAC-1,1.9470315343977591,15.0,-0.1221571468020825,-0.009736829450266707,2.057646,15.849435,2,G1,1073.5921186804771,5,5,5826.0,1741.0,6.2306900102986615,43.97528321318229,0.7194423108915827,False,True,9.3,-2.6589324,14.116144,821.0,9.0,5,0.9575994643424515
-156,True,AGGGAGTTCATGCCCT-1,AGGGAGTTCATGCCCT-1,1.8969043107268002,15.0,-0.08766494638777043,-0.020550939876704404,7.7138553,1.3971434,22,G1,1253.4215150773525,6,9,4605.0,1465.0,8.816503800217156,36.82953311617807,0.586740977593362,False,False,3.6999999999999997,13.878795,5.502858,825.0,9.0,6,0.8054144824310264
-157,True,AGGGAGTTCCCAACTC-1,AGGGAGTTCCCAACTC-1,1.9219937610651605,15.0,-0.1123819906081104,0.0019493241913706681,1.8143669,5.374422,5,S,1424.89253911376,1,1,6285.0,1448.0,8.035003977724742,51.20127287191726,0.6114980857402794,False,False,3.1999999999999997,9.387725,9.979434,1119.0,9.0,1,0.8054144824310264
-158,True,AGGGAGTTCCGCTTAC-1,AGGGAGTTCCGCTTAC-1,2.0088915619024723,15.0,-0.06757332338727688,-0.01647575850072382,-6.2522187,3.467735,14,G1,1441.8648898303509,4,4,5180.0,1926.0,9.45945945945946,13.32046332046332,0.9383917666508493,False,False,7.0,-14.71522,2.3763268,2353.0,9.0,4,1.6507563310617184
-160,True,AGGGAGTTCTCGAGTA-1,AGGGAGTTCTCGAGTA-1,1.8915453986936526,15.0,-0.10407496477364017,-0.010294432715070396,1.2022097,6.2667117,5,G1,1752.7356855422258,1,1,3485.0,1022.0,6.743185078909613,45.423242467718794,0.613937168097773,False,False,7.199999999999999,9.478535,11.157132,1007.0,9.0,1,1.2119072064054222
-161,True,AGGGAGTTCTCTAGGA-1,AGGGAGTTCTCTAGGA-1,1.957867669246695,15.0,-0.10691768231697926,-0.014312429421570091,2.7061987,18.901005,18,G1,1196.3447940945625,5,5,4406.0,1522.0,7.149341806627326,36.63186563776668,0.7231791866237086,False,False,9.1,-3.704866,15.647907,958.0,9.0,5,0.9980207379784404
-162,True,AGGGTGAAGCACCTGC-1,AGGGTGAAGCACCTGC-1,2.0397586146031923,15.0,-0.01996084022351966,-0.036177820359488175,-5.1781807,3.2959056,14,G1,1437.028485968709,4,4,6856.0,2588.0,12.689614935822638,10.064177362893815,0.927258647654456,False,False,6.800000000000001,-14.027452,2.148273,1711.0,9.0,4,2.239958061671911
-163,True,AGGGTGAAGCTTAAGA-1,AGGGTGAAGCTTAAGA-1,1.9802107172895436,15.0,-0.10999061447410213,-0.026561380306396588,2.1311727,18.807083,18,G1,1207.519254386425,5,5,3509.0,1244.0,14.44856084354517,34.65374750641208,0.7234316659834791,False,False,9.100000000000001,-3.6191807,15.690668,767.0,9.0,5,1.5744992908810693
-164,True,AGGGTGAAGGCCTAGA-1,AGGGTGAAGGCCTAGA-1,1.9990835068917752,15.0,-0.07236404544577993,-0.023745254319913447,-6.5546765,0.42638955,4,G1,1621.4113033562899,4,4,6172.0,2254.0,9.105638366817887,13.204795852235904,0.9625973898219361,False,False,4.3999999999999995,-15.174699,-0.32964537,1397.0,9.0,4,1.817077281330816
-165,True,AGGGTGAAGGCGTTAG-1,AGGGTGAAGGCGTTAG-1,1.9620204041931226,15.0,-0.12935582014355124,-0.01120823409532776,0.9282501,15.69259,2,G1,1107.087795048952,5,5,6146.0,1525.0,9.192971038073544,47.81972014318256,0.7186362275638372,False,False,9.4,-2.0567956,14.492488,993.0,9.0,5,1.352031409542768
-166,True,AGGGTGAAGGGAGGTG-1,AGGGTGAAGGGAGGTG-1,2.0138349006251075,15.0,-0.07452376690657511,-0.02074191522205611,-4.779829,1.095592,14,G1,1437.709598749876,4,4,5287.0,1941.0,13.48590883298657,13.410251560431247,0.9466798625089519,False,False,4.800000000000001,-14.562924,0.4363549,1552.0,9.0,4,0.8054144824310264
-167,True,AGGGTGAAGTCATCCA-1,AGGGTGAAGTCATCCA-1,1.8778471941564088,15.0,-0.10061880565477688,-0.020631584506698004,10.136134,-5.513826,23,G1,1140.2399310171604,6,8,4587.0,1622.0,6.780030521037715,32.962720732504906,0.2707591250544456,False,False,6.8,14.111893,-4.0814323,1602.0,9.0,6,1.4945853239801525
-168,True,AGGGTGACACTACGGC-1,AGGGTGACACTACGGC-1,1.9297264228309974,15.0,-0.0646902203118325,-0.024332676351071853,7.919957,-0.12238291,22,G1,1071.0502562969923,6,8,2793.0,1036.0,15.216612960973864,28.82205513784461,0.5444924458367872,False,False,4.699999999999999,13.967702,3.0003886,1149.0,9.0,6,0.986549540679356
-169,True,AGGGTGACAGCAGTGA-1,AGGGTGACAGCAGTGA-1,1.984137542044802,15.0,-0.0756494689965495,-0.02336710021705623,-8.917856,-2.380756,7,G1,1725.3096423447132,3,3,12338.0,3211.0,6.921705300697034,17.036796887664128,0.982287713185272,False,False,3.1,-16.2113,-3.0082824,1873.0,9.0,3,1.3383842755014106
-170,True,AGGGTGACATCAGTGT-1,AGGGTGACATCAGTGT-1,1.9097789450653049,15.0,-0.06911803679929535,-0.004740299571824414,8.035425,8.021349,27,G1,1185.869840875268,8,10,4896.0,1584.0,10.008169934640524,36.580882352941174,0.6139213389126777,False,False,5.2,13.851757,10.124836,1606.0,9.0,8,1.3006826524250614
-171,True,AGGGTGACATCCTTCG-1,AGGGTGACATCCTTCG-1,1.9009450620398822,15.0,-0.09437997948404876,-0.019304455536605078,7.2996416,-2.4540176,22,G1,1187.2551356852055,6,8,4876.0,1591.0,10.787530762920426,33.326497128794095,0.4635339802439194,False,False,7.1000000000000005,14.387135,0.54983985,1176.0,9.0,6,1.5689939908624575
-172,True,AGGGTGAGTCAAGCGA-1,AGGGTGAGTCAAGCGA-1,1.9075905945102112,15.0,-0.11464967359113308,-0.019072546692231993,2.863798,7.134179,5,G1,1250.507382042706,1,1,4785.0,1241.0,5.287356321839081,49.61337513061651,0.6118634172165451,False,False,3.1000000000000005,8.636889,9.681277,1540.0,9.0,1,1.545777420304805
-173,True,AGGGTGAGTTAAGTCC-1,AGGGTGAGTTAAGTCC-1,1.9017889465225057,15.0,-0.0794714482447183,-0.0039144269649442585,9.339613,-8.561738,11,G1,1211.5488028079271,7,6,5303.0,1857.0,11.578351876296436,26.89043937393928,0.1171388206921545,False,False,6.2,13.480804,-8.712111,1976.0,9.0,7,1.344406567033444
-174,True,AGGGTGATCATCGCTC-1,AGGGTGATCATCGCTC-1,1.8779006157448968,15.0,-0.08281205911869458,-0.011025243240351604,10.231219,-8.225378,11,G1,1215.2654928863049,7,6,6378.0,1915.0,6.945751019128253,32.659140796487925,0.14250351890000915,False,False,6.3999999999999995,13.771712,-7.422075,1455.0,9.0,7,1.3259779288380678
-175,True,AGGGTGATCCATTGCC-1,AGGGTGATCCATTGCC-1,2.018844681362616,15.0,-0.03302024198990963,-0.021773368194770347,-3.3272517,-5.6114087,3,G1,1426.1243579685688,2,2,4614.0,1955.0,18.053749458170785,11.009969657563936,0.9554010779097449,False,False,2.7000000000000006,-15.034717,-4.8673525,1900.0,10.0,2,1.2280874339147625
-176,True,AGGGTGATCCGAGAAG-1,AGGGTGATCCGAGAAG-1,1.9273932051682094,15.0,-0.07758609583544984,0.004610722989903136,8.081143,-1.1625338,22,S,1136.9777319580317,6,8,5184.0,1451.0,9.567901234567902,39.891975308641975,0.5030672911493764,False,False,4.6000000000000005,14.282672,1.9146088,1764.0,9.0,6,1.285062833113634
-177,True,AGGGTGATCCGCTTAC-1,AGGGTGATCCGCTTAC-1,1.9787561670628313,15.0,-0.08542220027141212,-0.027226092661780523,-10.037198,-8.245152,10,G1,1075.686592578888,3,7,13532.0,3308.0,10.30889742831806,18.238250073898907,0.9719571896436185,False,False,4.699999999999999,-12.228919,-8.838771,3168.0,9.0,3,0.8054144824310264
-178,True,AGGGTGATCGAACGCC-1,AGGGTGATCGAACGCC-1,1.9381192303650645,15.0,-0.07732169139685383,-0.014798374711734053,-4.8147006,-6.299513,19,G1,1351.0896033793688,2,2,12156.0,3578.0,6.5893385982231,13.984863441921684,0.9580476582948024,False,False,4.1,-15.775162,-6.420506,3278.0,9.0,2,1.23753771102208
-179,True,AGTAACCAGCTCACTA-1,AGTAACCAGCTCACTA-1,2.008151482078859,15.0,-0.0339328893908614,-0.02808084933825431,-6.2390175,-2.2200975,14,G1,1354.292534172535,4,2,7552.0,2712.0,14.009533898305085,9.229343220338983,0.9703430967145152,False,False,2.1,-16.745632,-2.1484218,2363.0,9.0,4,0.8054144824310264
-180,True,AGTAACCAGTGTTGAA-1,AGTAACCAGTGTTGAA-1,1.8992957169099052,15.0,-0.11292960516013482,-0.01974836440731568,-3.2247796,12.846995,2,G1,931.7019924521446,5,5,11717.0,3220.0,5.6499103866177345,30.152769480242384,0.7226957996154734,False,False,3.3,2.964976,13.1519785,1184.0,9.0,5,1.153800596728569
-181,True,AGTAACCCAAACCATC-1,AGTAACCCAAACCATC-1,1.9896787941171463,15.0,-0.07095069331114916,-0.02781177677856159,-8.459632,-1.586563,4,G1,1593.954465046525,4,3,15044.0,3524.0,7.577771869183728,13.659930869449614,0.9825523099230157,False,False,4.0,-14.851536,-2.3525853,2157.0,9.0,4,1.8145340755406008
-182,True,AGTAACCCACCGTGAC-1,AGTAACCCACCGTGAC-1,2.009012127312308,15.0,-0.06729689107636799,-0.04253783513799518,-7.039367,0.11341624,4,G1,1565.6348773539066,4,4,5569.0,2237.0,13.14419105764051,8.888489854551985,0.9591302073557765,False,False,3.5999999999999996,-14.823204,-0.9303894,2067.0,9.0,4,1.112774701128965
-183,True,AGTAACCCACGCGCAT-1,AGTAACCCACGCGCAT-1,1.9032648073584948,15.0,-0.1240869825472873,-0.01533688901170209,5.6774244,7.6212363,27,G1,1136.2247373312712,8,1,6264.0,1590.0,7.263729246487867,46.45593869731801,0.6142725716950314,True,True,2.0,11.633839,11.094412,1861.0,9.0,8,0.8054144824310264
-184,True,AGTAACCCACTAAACC-1,AGTAACCCACTAAACC-1,1.8797399209536272,15.0,-0.09205308541206794,-0.01619016115624707,8.812552,9.291855,15,G1,1135.8423383831978,8,10,2847.0,1053.0,3.582718651211802,41.55251141552512,0.6182550155039122,False,False,5.5,13.908773,10.854307,1920.0,9.0,8,1.333463942409766
-185,True,AGTAACCCACTACACA-1,AGTAACCCACTACACA-1,1.9129600248697474,15.0,-0.06965951546884966,-0.019162463805250857,-2.582553,-13.376397,24,G1,1323.8105005770922,10,7,17067.0,3698.0,9.222476123513212,19.558211753676687,0.9592329285962357,False,True,6.8,-6.501388,-11.774815,2446.0,9.0,10,1.07800882493837
-186,True,AGTAACCGTAGCGTAG-1,AGTAACCGTAGCGTAG-1,2.0479107710387545,15.0,-0.0637804137801157,-0.011294328907686692,-5.208605,3.8224652,14,G1,1323.0723382532597,4,4,3901.0,1714.0,15.175596001025378,15.252499359138682,0.9155700482144069,False,False,6.3,-13.824158,2.919669,1775.0,10.0,4,1.044757763005983
-187,True,AGTAACCGTCGAATGG-1,AGTAACCGTCGAATGG-1,1.966959372870528,15.0,-0.09017281089723979,-0.02652180244756926,-6.880208,-9.277692,3,G1,1202.3733336031437,2,2,8362.0,2783.0,11.95886151638364,16.443434585027504,0.9504781875501616,False,False,4.7,-14.952475,-8.613107,2918.0,9.0,2,1.1075219258728686
-188,True,AGTAACCGTCGTGCCA-1,AGTAACCGTCGTGCCA-1,1.9742828157154368,15.0,-0.06954572183639268,-0.025004564484149224,-2.6510036,-4.710017,19,G1,1416.8228048682213,2,2,6928.0,2396.0,10.825635103926096,16.35392609699769,0.9536384157550776,False,False,4.4,-14.879069,-4.730261,2968.0,9.0,2,1.4744857463590444
-189,True,AGTAACCGTCTAGGTT-1,AGTAACCGTCTAGGTT-1,2.0197237760144637,15.0,-0.0577421903198252,-0.028726369794203486,-6.947303,1.1455452,14,G1,1529.4890487343073,4,4,9901.0,3008.0,8.514291485708513,12.311887688112312,0.9548686289515712,False,False,4.2,-15.197692,0.34337413,2003.0,9.0,4,1.4081095231790977
-190,True,AGTAACCGTTGGACCC-1,AGTAACCGTTGGACCC-1,1.9735451985040848,15.0,-0.10391885525440755,-0.007243338666475434,-1.988978,15.579804,2,G1,1234.7291979640722,5,5,6421.0,1725.0,10.247624980532628,40.10278772776826,0.7321124975607457,False,False,7.1,0.26219994,14.023526,1491.0,9.0,5,0.8054144824310264
-191,True,AGTAACCTCAAGAAAC-1,AGTAACCTCAAGAAAC-1,1.944006566182653,15.0,-0.04604359597004596,-0.032834760215155585,-9.059354,-8.093079,10,G1,1133.6020619124174,3,7,7968.0,2791.0,8.170180722891565,12.625502008032129,0.9678084716665716,False,False,3.8000000000000003,-13.192566,-8.37319,3920.0,9.0,3,1.4165992585769842
-192,True,AGTAACCTCGCGGTAC-1,AGTAACCTCGCGGTAC-1,1.9684426437387044,15.0,-0.06981902319686387,-0.030591401535327117,-5.111448,-3.6757283,13,G1,985.5017040669918,2,2,10520.0,3287.0,9.923954372623575,16.378326996197718,0.9590778097497064,False,False,3.6999999999999997,-16.837934,-4.378158,4817.0,9.0,2,1.3205567218415608
-193,True,AGTAACCTCTCCAATT-1,AGTAACCTCTCCAATT-1,1.8979634203759799,15.0,-0.059929151473997605,-0.019841625535233037,8.9587145,-11.733198,21,G1,1175.9114208370447,7,6,3791.0,1521.0,11.9493537325244,20.047480875758374,0.04120311929676538,False,True,4.1,11.924159,-11.164298,1336.0,9.0,7,0.8054144824310264
-194,True,AGTAACCTCTGGCTGG-1,AGTAACCTCTGGCTGG-1,1.8859184198541095,15.0,-0.07800812547929072,-0.01609809708634754,0.5149173,7.7141066,5,G1,1611.2110636085272,1,1,3587.0,1252.0,7.081126289378311,37.106216894340676,0.620537055212367,False,False,6.3,9.579427,12.532116,1578.0,9.0,1,1.0519082176203531
-195,True,AGTAGTCAGACCAGAC-1,AGTAGTCAGACCAGAC-1,2.039567564149452,15.0,-0.04368265590547907,-0.0221483942414175,-4.107729,1.497595,14,G1,1393.0325990766287,4,4,5805.0,2179.0,10.180878552971576,11.541774332472007,0.9392906681895427,False,False,5.2,-15.035425,0.99208266,1983.0,9.0,4,1.062821352303575
-196,True,AGTAGTCAGCGAATGC-1,AGTAGTCAGCGAATGC-1,2.015483401242211,15.0,-0.06146055654566803,-0.023648989718452237,-2.581624,-4.78026,19,G1,1411.7691136449575,2,2,10670.0,3180.0,9.14714151827554,16.016869728209933,0.9556042426675108,True,True,4.2,-15.53265,-4.7096996,1886.0,9.0,2,1.1330446603053392
-197,True,AGTAGTCAGCTTGTTG-1,AGTAGTCAGCTTGTTG-1,1.912242012441276,15.0,-0.08679128439163991,-0.02344206497143412,-2.797753,-12.44052,24,G1,1319.2929827868938,10,7,13188.0,3003.0,7.089778586593873,21.40582347588717,0.959287740048452,False,True,7.599999999999999,-6.940486,-11.6441765,2032.0,9.0,10,1.5709628405879854
-198,True,AGTAGTCAGGAACTCG-1,AGTAGTCAGGAACTCG-1,1.9873961084059872,15.0,-0.07536777742113586,-0.03180078686106828,-10.436971,-2.4028747,20,G1,1467.7244146764278,4,3,10149.0,3191.0,7.350477879594049,12.602226820376393,0.9916565024382628,False,False,4.6000000000000005,-17.714714,-2.9400742,2963.0,9.0,4,1.427718777406776
-201,True,AGTAGTCCAGCTGCCA-1,AGTAGTCCAGCTGCCA-1,1.9790214285723737,15.0,-0.06411211940006553,-0.022578176937646598,-9.694992,-6.67721,10,G1,1045.421784967184,3,7,11389.0,3182.0,7.928703134603565,7.99894635174291,0.9767350652129011,True,True,3.4,-13.352134,-7.103583,2499.0,9.0,3,0.8054144824310264
-202,True,AGTAGTCGTATGCGTT-1,AGTAGTCGTATGCGTT-1,1.8886523578399372,15.0,-0.11435510093828517,-0.008910542891125414,3.207613,3.4545557,12,G1,1844.081807360053,1,1,4680.0,1333.0,4.358974358974359,47.67094017094017,0.612063794400627,False,False,5.0,11.042665,9.332835,1043.0,9.0,1,1.6482295406833074
-203,True,AGTAGTCGTGATTGGG-1,AGTAGTCGTGATTGGG-1,1.9784014444950915,15.0,-0.06917126923529704,-0.023325660312419394,-7.671123,-7.6788607,3,G1,1329.9316990971565,2,2,9153.0,2976.0,11.001857314541681,12.45493280891511,0.9609477704302996,False,False,3.3999999999999995,-14.183264,-7.749397,3278.0,9.0,2,1.204339849734597
-204,True,AGTAGTCGTGGCTTAT-1,AGTAGTCGTGGCTTAT-1,1.937370792346772,15.0,-0.0852476690319341,-0.018331858335112583,-6.3143053,-11.665253,10,G1,1225.1400522440672,3,7,13843.0,3341.0,10.879144694069204,22.849093404608826,0.9621849601356224,False,False,4.3999999999999995,-9.601259,-10.608301,1938.0,9.0,3,1.0986191307828845
-205,True,AGTAGTCGTTCGAAGG-1,AGTAGTCGTTCGAAGG-1,1.9867834125907478,15.0,-0.06619035020003189,-0.027515078422428816,-9.52604,-1.9999309,1,G1,1493.4433780908585,3,3,9858.0,3273.0,8.734023128423615,12.507608034083992,0.9818048226334961,False,False,2.5999999999999996,-16.759935,-2.970612,1612.0,9.0,3,1.0120189530495938
-206,True,AGTAGTCTCATGCCCT-1,AGTAGTCTCATGCCCT-1,1.9759440166839446,15.0,-0.08981887448122243,-0.01769604859279354,-1.9479306,-2.5904353,14,G1,1044.4829179495573,4,2,9980.0,2953.0,9.829659318637274,25.29058116232465,0.8811024647326025,False,False,1.2000000000000002,-12.60621,-0.90279055,2606.0,9.0,4,0.9657673329367392
-207,True,AGTAGTCTCCCGTTGT-1,AGTAGTCTCCCGTTGT-1,1.9566393133094453,15.0,-0.09019056175573137,-0.02395964750923653,-5.616607,-7.8448715,3,G1,987.6026282757521,2,2,14605.0,3763.0,11.235878123930162,18.062307428962683,0.9454580692173806,False,False,2.4000000000000004,-16.21682,-8.240849,2635.0,9.0,2,0.9275761967036148
-208,True,AGTAGTCTCGCGGTAC-1,AGTAGTCTCGCGGTAC-1,1.8898046552053585,15.0,-0.08483936527603522,-0.020618995842050045,6.899056,-0.1468433,22,G1,1085.7298173457384,6,8,2518.0,965.0,10.007942811755361,35.18665607625099,0.5505351534293874,False,False,4.7,14.320609,3.734429,869.0,9.0,6,1.1081635770280527
-209,True,AGTAGTCTCGTAGAGG-1,AGTAGTCTCGTAGAGG-1,1.9105795564275383,15.0,-0.1131286876545566,-0.0035259013425582297,6.028894,8.085403,27,G1,1195.5219519138336,8,1,7133.0,1710.0,5.719893452965092,46.43207626524604,0.6150576209990805,False,False,3.3,11.183523,11.403489,1545.0,9.0,8,1.1719784193997438
-211,True,AGTGACTAGCATTGAA-1,AGTGACTAGCATTGAA-1,2.0034961839355736,15.0,-0.06685575998449146,-0.020980087342572083,-5.5544257,0.60409164,20,G1,1493.7320757359266,4,4,11818.0,3387.0,7.471653410052462,15.180233542054493,0.9563478267741482,False,False,4.6000000000000005,-16.24631,0.39875418,2402.0,9.0,4,1.8390623816332334
-212,True,AGTGACTAGCGAATGC-1,AGTGACTAGCGAATGC-1,1.9231910118938516,15.0,-0.09023858606266301,-0.024295603591455953,3.8922818,3.4011533,12,G1,1893.378217563033,1,1,4135.0,1251.0,11.825876662636034,38.30713422007255,0.6092172455330276,False,False,6.3,11.309192,9.340478,1862.0,9.0,1,1.6886958137388428
-213,True,AGTGACTAGGAACTCG-1,AGTGACTAGGAACTCG-1,1.920071075335419,15.0,-0.087238866424578,-0.018162970712132895,9.093022,-11.370275,21,G1,1245.0428095906973,7,6,4513.0,1648.0,12.231331708397962,22.26900066474629,0.04398950042688977,False,False,5.599999999999999,11.672203,-10.8875475,1219.0,9.0,7,1.4585607180548472
-214,True,AGTGACTAGGATCACG-1,AGTGACTAGGATCACG-1,1.9270208770563118,15.0,-0.09104138284243529,-0.024443387068289624,7.24496,0.76392496,22,G1,1207.7217156589031,6,8,4110.0,1422.0,14.549878345498783,26.32603406326034,0.5707021834581947,True,True,3.7,13.807022,4.9449224,1256.0,9.0,6,1.626654214416418
-216,True,AGTGACTAGTAGGAAG-1,AGTGACTAGTAGGAAG-1,1.9988659218296208,15.0,-0.08538139330770252,-0.030859069005625843,-2.5866919,-1.3580492,19,G1,1293.0350689888,2,2,4908.0,1986.0,8.822330888345558,17.685411572942137,0.9307242582898891,False,False,5.000000000000001,-12.996251,-2.3639677,1886.0,9.0,2,1.174574202767167
-217,True,AGTGACTAGTGTTGAA-1,AGTGACTAGTGTTGAA-1,1.9622873449948424,15.0,-0.09720421545130667,-0.0011106592041833022,-1.1784297,15.602561,2,G1,1209.9243125095963,5,5,6452.0,1773.0,11.174829510229387,36.26782393056417,0.7355953260871654,False,True,6.3,-0.11782662,13.948348,1100.0,9.0,5,1.2269461008431268
-218,True,AGTGACTCAATCTGCA-1,AGTGACTCAATCTGCA-1,1.9880115183805345,15.0,-0.06021523059274649,-0.02290354079722359,-9.072821,-0.38701987,20,G1,1396.35428789258,4,3,8920.0,2808.0,10.448430493273543,17.286995515695068,0.9779221227610093,False,False,3.3000000000000003,-17.1895,-1.0579098,1932.0,9.0,4,1.4431469775576604
-220,True,AGTGACTCATTATGCG-1,AGTGACTCATTATGCG-1,2.040474633948175,15.0,-0.03810735375768781,-0.015159116883418108,-9.975807,-1.0048057,7,G1,1558.08698579669,3,3,5023.0,1971.0,14.433605415090584,9.655584312164045,0.9839381123530931,False,False,3.9000000000000004,-16.572664,-1.4191266,10865.0,9.0,3,1.2926364428612958
-221,True,AGTGACTGTACGGGAT-1,AGTGACTGTACGGGAT-1,1.8658606667391968,15.0,-0.08483662106648691,-0.01990163608674161,9.00566,-9.225144,11,G1,1213.5385919809341,7,6,4509.0,1841.0,6.941672211133289,25.371479263694834,0.10769841955126792,False,False,6.1,13.443702,-8.845428,1306.0,9.0,7,1.0188893454215837
-222,True,AGTGACTGTAGTGTGG-1,AGTGACTGTAGTGTGG-1,1.944628520832997,15.0,-0.08391270472692411,-0.01967614359802309,-9.82459,-9.567346,10,G1,1095.5074007064104,3,7,11807.0,3525.0,8.063013466587618,15.109680697891081,0.9663011058020665,False,False,4.9,-11.612929,-9.865841,4917.0,9.0,3,0.9932064799220534
-223,True,AGTGACTGTGGCTTAT-1,AGTGACTGTGGCTTAT-1,2.124227584016134,15.0,-0.05961756698736719,-0.026519813201065764,-7.2831745,2.9054945,14,G1,1362.1653456538916,4,4,2343.0,1042.0,24.029022620571915,6.274007682458387,0.9097585860908094,False,False,3.1000000000000005,-13.562403,1.0645906,2140.0,9.0,4,1.08477779533976
-224,True,AGTGACTTCATTTGGG-1,AGTGACTTCATTTGGG-1,1.9660326938011705,15.0,-0.08647985928774644,-0.03467858394938445,-6.658715,-2.4828436,10,G1,1413.482744127512,3,3,7671.0,2771.0,8.916699256941728,15.47386259940034,0.9711161437698143,False,False,2.9000000000000004,-15.778843,-3.0696752,2211.0,9.0,3,0.8054144824310264
-225,True,AGTGACTTCCCAAGCG-1,AGTGACTTCCCAAGCG-1,1.9718846907674594,15.0,-0.10440350520140344,0.004274095929688174,-2.1045136,15.088611,2,S,1268.0870485082269,5,5,5019.0,1396.0,10.639569635385534,39.84857541342897,0.7329628686899524,False,False,7.2,0.67419714,14.312525,1213.0,9.0,5,1.5243698758630269
-226,True,AGTGACTTCGCCCAGA-1,AGTGACTTCGCCCAGA-1,2.1548967503416705,15.0,-0.056704552548730205,-0.040784983478895365,-6.0760794,3.2444549,14,G1,1443.4711663722992,4,4,3047.0,1327.0,24.71283229405973,5.218247456514605,0.9353626808878867,False,True,5.8999999999999995,-14.593301,2.5032594,2193.0,9.0,4,1.8220154132044484
-227,True,AGTGACTTCGTTTACT-1,AGTGACTTCGTTTACT-1,2.009436536363936,15.0,-0.057015311068813475,-0.02540834467297301,-4.2355447,0.96951485,14,G1,1450.369052067399,4,4,16407.0,3999.0,7.679648930334613,14.512098494545011,0.9464330930069054,False,False,4.3,-14.66405,0.6021441,1621.0,10.0,4,1.5543627449492794
-228,True,AGTGACTTCTAGTCAG-1,AGTGACTTCTAGTCAG-1,1.9639098841540013,15.0,-0.09826708870864945,-0.023261302572870433,-10.631753,-5.9143066,10,G1,1275.491935417056,3,3,9389.0,2846.0,8.158483331558207,16.338268186175313,0.9809677971901972,False,False,2.6,-12.736212,-7.151743,2268.0,9.0,3,0.8054144824310264
-229,True,AGTGACTTCTCTAGGA-1,AGTGACTTCTCTAGGA-1,1.9670531743471937,15.0,-0.07983617822789546,-0.02857715553333101,-3.3725173,-4.60185,19,G1,1543.4820107519627,2,2,7217.0,2654.0,7.038935845919357,13.731467368712762,0.9505352015279241,False,False,4.4,-14.686379,-4.291832,2830.0,9.0,2,0.8054144824310264
-230,True,ATGAGTCAGATACCAA-1,ATGAGTCAGATACCAA-1,1.910694998021482,15.0,-0.10755965562966284,-0.013925016427268452,7.507246,10.222009,15,G1,1099.2311912626028,8,10,7178.0,1791.0,7.564781276121482,45.36082474226804,0.6203969153168982,False,False,6.0,14.726801,10.089775,829.0,9.0,8,1.2945102574612721
-231,True,ATGAGTCAGGCCTAGA-1,ATGAGTCAGGCCTAGA-1,2.033929660476258,15.0,-0.06412187179771714,-0.0274650730306219,-7.8592477,1.0067067,14,G1,1383.5188588351011,4,4,5868.0,2013.0,13.173142467620995,9.423994546693933,0.9562336110848553,False,False,3.7,-13.515028,-0.41962734,1220.0,9.0,4,1.7615150195516065
-232,True,ATGAGTCAGTCCCAAT-1,ATGAGTCAGTCCCAAT-1,1.9382044608620894,15.0,-0.09087765020396948,-0.028752725636322026,-10.60772,-9.36925,10,G1,1148.315005555749,3,7,9014.0,3051.0,6.76725094297759,13.94497448413579,0.9715380091634137,False,False,5.999999999999999,-11.760579,-9.03043,3289.0,9.0,3,0.8054144824310264
-233,True,ATGAGTCCACATGAAA-1,ATGAGTCCACATGAAA-1,1.9862839619938597,15.0,-0.07988446316382732,-0.023590422386047312,-11.081515,-4.263356,7,G1,1269.61087885499,3,3,10428.0,3251.0,9.810126582278482,13.147295742232451,0.9930996329851969,False,False,3.4999999999999996,-18.020067,-4.368959,3314.0,9.0,3,1.4214155226461038
-234,True,ATGAGTCCATGGCTAT-1,ATGAGTCCATGGCTAT-1,1.9493445477216347,15.0,-0.03149080352974349,-0.03386769022352675,8.595844,5.521287,26,G1,1239.9330451861024,9,9,1548.0,737.0,25.0,6.976744186046512,0.6120179367616124,False,False,4.3,10.468837,5.819747,788.0,9.0,9,0.8054144824310264
-235,True,ATGAGTCGTATAGGAT-1,ATGAGTCGTATAGGAT-1,1.961525541381548,15.0,-0.1014056741883011,-0.011787332641642868,-6.0236382,-5.2765417,7,G1,1391.0980592221022,3,2,15877.0,3724.0,8.99414247023997,29.615166593185112,0.9620517735004553,False,False,3.3,-13.4641695,-5.423077,3006.0,9.0,3,1.3790528332138858
-236,True,ATGAGTCGTCACTCGG-1,ATGAGTCGTCACTCGG-1,1.9709729163855587,15.0,-0.09059679680429394,-0.0049051334565988515,-1.5361401,15.654716,2,G1,1192.3622210547328,5,5,7691.0,2240.0,9.894682095956313,31.543362371603173,0.7321771078566929,False,True,6.0,-0.31161445,13.993011,2057.0,9.0,5,1.3295008470439955
-237,True,ATGAGTCGTCGTGCCA-1,ATGAGTCGTCGTGCCA-1,1.9942356383646915,15.0,-0.07587572435515191,-0.025380229263724408,-6.7467384,-5.7821975,13,G1,1359.2146485298872,2,2,7800.0,2720.0,10.884615384615385,12.23076923076923,0.9607920176334105,False,False,2.5,-16.573494,-6.039786,3063.0,9.0,2,1.1541723609248176
-238,True,ATGAGTCGTGCACAAG-1,ATGAGTCGTGCACAAG-1,1.888700455832798,15.0,-0.08005214608204642,-0.017425486144527158,10.030667,-10.439313,21,G1,1267.19097648561,7,6,4704.0,1692.0,8.60969387755102,23.724489795918366,0.07761839504007188,False,False,5.499999999999999,12.448465,-9.778137,1078.0,9.0,7,1.1840045921432583
-239,True,ATGAGTCGTTAAGCAA-1,ATGAGTCGTTAAGCAA-1,1.9532027439023074,15.0,-0.08129605292827986,-0.029628611792187492,-10.010821,-4.804679,10,G1,1558.1191532462835,3,3,9044.0,2782.0,6.037151702786378,14.849624060150376,0.9828022972400092,False,False,3.3000000000000003,-15.278864,-4.9805813,2068.0,9.0,3,1.2131002172767453
-240,True,ATGAGTCGTTTGATCG-1,ATGAGTCGTTTGATCG-1,1.9230040837798366,15.0,-0.0732228360365308,-0.01127877926514623,-3.2522728,-12.142946,24,G1,1296.7913182526827,10,7,8899.0,2551.0,10.85515226429936,19.79997752556467,0.9595654502660406,True,True,6.8,-6.96506,-11.7019615,2412.0,9.0,10,1.3913939848529155
-241,True,ATGAGTCTCAATCCAG-1,ATGAGTCTCAATCCAG-1,1.8972339045678766,15.0,-0.10934662097452795,-0.011753021461759326,2.545958,4.2550983,12,G1,1813.8059920966625,1,1,6105.0,1702.0,4.307944307944308,45.06142506142506,0.607718519717103,False,False,4.1,10.436038,9.445456,1412.0,9.0,1,1.2470856525873835
-242,True,ATGAGTCTCATCGCTC-1,ATGAGTCTCATCGCTC-1,1.9485914996478833,15.0,-0.0907139737301823,-0.016847045384850625,-6.5174108,-10.106189,3,G1,930.5267608761787,2,2,13068.0,3861.0,7.621671258034894,19.589837771655954,0.9445292153188589,False,False,4.1,-15.194768,-9.461049,1470.0,9.0,2,0.8054144824310264
-243,True,ATGAGTCTCCTTATCA-1,ATGAGTCTCCTTATCA-1,1.9185561604857253,15.0,-0.11519032569688643,-0.003575447063018964,9.837386,1.5639766,25,G1,962.2551143467426,6,9,10409.0,2277.0,6.936305120568739,48.05456816216736,0.5912641326524077,True,True,4.2,13.338084,4.0977106,571.0,9.0,6,0.9000524015178514
-244,True,ATGAGTCTCCTTCACG-1,ATGAGTCTCCTTCACG-1,1.909566683084526,15.0,-0.11364707552075888,-0.009640580438490386,5.000597,5.7478833,12,G1,1517.232741497457,1,1,5630.0,1567.0,6.980461811722913,44.95559502664298,0.6112424642399018,False,False,4.0,9.9637575,7.8894196,1088.0,9.0,1,1.6399485244002727
-245,True,ATGAGTCTCGTTTACT-1,ATGAGTCTCGTTTACT-1,1.8788403126855981,15.0,-0.07333624725693144,-0.023709955968888252,8.862596,-13.644124,21,G1,1139.287091806531,7,6,3970.0,1558.0,9.294710327455919,18.035264483627206,0.0034460559644219056,False,False,7.3,10.017577,-11.7891035,729.0,9.0,7,1.5382125678571381
-246,True,CACAGATAGCAGGTCA-1,CACAGATAGCAGGTCA-1,1.898174051312316,15.0,-0.11803585446996301,-0.021375723641095808,7.808576,9.525522,15,G1,1122.682582437992,8,10,7425.0,2019.0,5.5353535353535355,44.0,0.6201380679485924,False,False,5.2,14.875615,10.343692,1560.0,9.0,8,1.389617141286779
-247,True,CACAGATAGCCTCTGG-1,CACAGATAGCCTCTGG-1,2.0934914231895725,15.0,-0.05426606088277783,-0.04197641196463149,-7.9298472,-12.2404375,16,G1,641.9509965926409,2,2,1434.0,629.0,6.066945606694561,2.510460251046025,0.9315334324549855,False,False,6.1,-15.350565,-11.500702,1062.0,9.0,2,0.9125033401313234
-248,True,CACAGATAGCGACATG-1,CACAGATAGCGACATG-1,1.905899827817694,15.0,-0.13339191047176205,-0.008636840495927807,5.919783,4.639442,6,G1,1597.5391537323594,1,1,6568.0,1566.0,5.67904993909866,49.84774665042631,0.6096368825558802,False,False,3.1,11.957175,7.931293,1447.0,9.0,1,1.3606883983701565
-249,True,CACAGATAGTCCCAAT-1,CACAGATAGTCCCAAT-1,1.8858406616845709,15.0,-0.10830045531129179,-0.0066014400381299194,9.584839,3.9964867,8,G1,1128.0994460433722,9,9,6303.0,2069.0,6.615897191813422,36.12565445026178,0.6109226635081283,False,False,3.5000000000000004,12.084686,5.3174706,1755.0,9.0,9,0.9577368584765362
-250,True,CACAGATCAAGAGAGA-1,CACAGATCAAGAGAGA-1,1.9741766965528793,15.0,-0.08120035503877043,-0.021487682362265646,-3.2732892,-1.6592981,19,G1,1356.8133256584406,2,2,11087.0,3201.0,7.495264724452061,19.806981149093534,0.9440600790098771,False,False,3.8,-13.2692995,-2.2195203,2308.0,9.0,2,1.6533085336883095
-252,True,CACAGATCAATTAGGA-1,CACAGATCAATTAGGA-1,1.9628979902706531,15.0,-0.08293546945017305,-0.02460502592472946,-11.262711,-5.558702,1,G1,1187.7571196556091,3,3,8166.0,2577.0,12.417340191036002,14.952240999265246,0.9869834782863756,False,False,3.4000000000000004,-17.816526,-5.113067,2501.0,9.0,3,1.3445335173275872
-253,True,CACAGATCACGCGCAT-1,CACAGATCACGCGCAT-1,1.9091945917701743,15.0,-0.0866817229890888,-0.027372228265579807,-2.8760626,-13.046076,24,G1,1323.607300773263,10,7,17358.0,4060.0,6.988132273303376,20.831893075239083,0.959061183069423,False,False,7.5,-6.6627893,-11.961482,3433.0,9.0,10,1.2535270090677104
-254,True,CACAGATCACTAAACC-1,CACAGATCACTAAACC-1,1.909125968005788,15.0,-0.10948072022806724,-0.00038473302413622595,5.4973087,5.610584,12,G1,1744.458917349577,1,1,7210.0,1684.0,6.5325936199722605,50.48543689320388,0.6121234524306127,False,False,4.8,11.83121,9.444659,1164.0,9.0,1,1.2614974210295193
-255,True,CACAGATCAGCTGCCA-1,CACAGATCAGCTGCCA-1,1.9471749863806758,15.0,-0.10124475217216014,-0.014079676828782604,-2.498184,14.80123,2,G1,1252.6248314082623,5,5,8892.0,2106.0,7.658569500674764,43.71345029239766,0.735898360129958,False,False,6.8999999999999995,1.2036521,14.435313,986.0,9.0,5,1.0689573863442006
-256,True,CACAGATCAGGGAATC-1,CACAGATCAGGGAATC-1,1.8613033638757588,15.0,-0.08065933242879653,-0.0037298348822198575,10.259737,-6.2967033,23,G1,1131.5720804184675,6,8,4140.0,1652.0,8.88888888888889,22.8743961352657,0.21764991458768254,False,False,6.1000000000000005,13.817165,-5.373298,1376.0,9.0,6,0.9697530029822112
-257,True,CACAGATCATGTTCGA-1,CACAGATCATGTTCGA-1,1.9616032279032383,15.0,-0.08706609813750579,-0.01821969736669793,-6.4216986,-2.4200215,9,G1,1351.7011956125498,2,2,8734.0,2608.0,8.953514998855049,27.83375314861461,0.9653114918597391,False,False,3.1999999999999997,-16.749008,-2.6551,2857.0,9.0,2,1.0305418786578064
-258,True,CACAGATGTCAAGCCC-1,CACAGATGTCAAGCCC-1,1.9519790683451264,15.0,-0.08635927679051328,-0.022463477557814162,-4.115569,-12.60929,24,G1,1305.9845884144306,10,7,14559.0,3744.0,10.454014698811731,17.350092726148773,0.9600976374165852,False,False,5.699999999999999,-7.8798165,-11.245723,1921.0,9.0,10,1.408997248830342
-259,True,CACAGATGTCACTCGG-1,CACAGATGTCACTCGG-1,1.8868835987017194,15.0,-0.10545702514469354,-0.01035548293414312,8.120695,10.056861,15,G1,1057.6837232112885,8,10,6603.0,1843.0,6.860517946388005,42.526124488868696,0.6191842498840736,False,False,4.8999999999999995,15.10931,10.704387,1041.0,9.0,8,0.9497939589833029
-260,True,CACAGATGTCCTTAAG-1,CACAGATGTCCTTAAG-1,1.9533978814964625,15.0,-0.07155278635282526,-0.020770586335033665,-6.764694,-8.085549,3,G1,1363.4560393095016,2,2,13384.0,3768.0,9.982068141063957,15.189778840406456,0.9561299280158662,False,False,4.3,-15.0842285,-7.612093,3105.0,9.0,2,1.6488763139858706
-261,True,CACAGATGTGCACAAG-1,CACAGATGTGCACAAG-1,1.9779090884904582,15.0,-0.08287318316838081,-0.019529119772589674,-7.2426553,0.97742075,14,G1,1312.5899412930012,4,4,8075.0,2627.0,7.306501547987616,14.402476780185758,0.9550102196527467,False,False,4.4,-13.86885,-0.24455814,1853.0,9.0,4,1.204438876733728
-262,True,CACAGATGTTAAGACA-1,CACAGATGTTAAGACA-1,1.9830480234972334,15.0,-0.09527086247401946,-0.02628467528379961,-6.5058575,-9.116389,3,G1,1066.927904829383,2,2,13737.0,3766.0,10.912135109558127,14.049646938924074,0.9425982751423084,False,False,4.0,-15.082209,-9.436545,3322.0,9.0,2,1.3223600882395572
-263,True,CACAGATGTTCCACGG-1,CACAGATGTTCCACGG-1,1.8958746942479703,15.0,-0.10356223633511522,-0.03891124802105785,6.307671,6.438841,12,G1,1365.1214649826288,1,1,2545.0,970.0,9.587426326129666,28.33005893909627,0.6146709326254021,False,False,5.0,10.134251,7.1058884,1158.0,9.0,1,1.4180889356972706
-264,True,CACAGATGTTGGAGGT-1,CACAGATGTTGGAGGT-1,1.9748341599065642,15.0,-0.08506303035137558,-0.018668447231858683,-9.223729,-10.1375065,10,G1,1181.183958351612,3,7,12110.0,3456.0,12.601156069364162,17.159372419488026,0.964871546327314,False,False,5.6,-10.804631,-9.927038,3207.0,9.0,3,1.640550436902247
-265,True,CACAGATTCAAGAAAC-1,CACAGATTCAAGAAAC-1,1.9134856020367523,15.0,-0.08419378288376399,-0.0051882798562666,9.0042515,-13.854578,21,G1,1120.239810794592,7,6,4099.0,1651.0,14.174188826543059,16.467431080751403,0.0035375530519038798,False,False,6.499999999999999,10.436758,-11.754163,1036.0,9.0,7,1.2323484104496285
-266,True,CACAGATTCATGCCCT-1,CACAGATTCATGCCCT-1,1.9481054751152984,15.0,-0.08472481176051987,-0.026824819284657166,-8.722018,-10.356675,10,G1,1164.206478729844,3,7,16573.0,3997.0,10.861039039401437,19.549870270922586,0.9657253455844346,False,False,6.000000000000001,-10.887927,-10.057448,3835.0,9.0,3,1.499041005359969
-267,True,CACAGATTCCATTGCC-1,CACAGATTCCATTGCC-1,1.8581455679214434,15.0,-0.06409620076133328,-0.020523672998327427,10.495759,-8.958761,11,G1,1226.2552442848682,7,6,5154.0,1951.0,8.110205665502523,20.391928599146294,0.13442962685379053,False,False,6.6,13.345274,-7.510192,2116.0,9.0,7,0.8054144824310264
-268,True,CACAGATTCCCAACTC-1,CACAGATTCCCAACTC-1,1.9093803348390341,15.0,-0.08828690165819103,-0.015869867935006395,-2.3800576,-12.490969,24,G1,1348.863736987114,10,7,13022.0,3330.0,7.917370603593918,19.682076485946858,0.9589984515008175,False,False,8.5,-6.640193,-11.979427,4485.0,9.0,10,0.9918336898289957
-269,True,CACAGATTCCGTATGA-1,CACAGATTCCGTATGA-1,1.9632509929798163,15.0,-0.08433901556778228,-0.01315216855900474,-10.7482815,-6.2994127,1,G1,1097.8861177563667,3,7,12746.0,3692.0,8.857680841048172,14.231915895182803,0.984780307491724,False,False,3.1000000000000005,-13.469548,-7.102729,3581.0,9.0,3,0.8054144824310264
-270,True,CACAGATTCGTAGAGG-1,CACAGATTCGTAGAGG-1,1.9704229461530953,15.0,-0.05080116182891875,-0.026927952515174564,-4.3699207,-3.3322155,17,G1,1594.0050113648176,2,2,7750.0,2523.0,11.161290322580646,20.361290322580643,0.9570453162069913,False,False,4.3,-15.56773,-4.5900908,2082.0,9.0,2,1.0532599957420288
-272,True,CAGCAGCAGCTCTGTA-1,CAGCAGCAGCTCTGTA-1,1.9041296265439265,15.0,-0.09331506362679048,-0.008525065331510281,5.530037,8.750011,27,G1,1074.45114479959,8,1,7050.0,1868.0,9.21985815602837,41.98581560283688,0.617227765200647,False,False,2.9999999999999996,12.094755,11.459143,1082.0,9.0,8,0.8054144824310264
-273,True,CAGCAGCAGGCTCAAG-1,CAGCAGCAGGCTCAAG-1,1.8957838652224246,15.0,-0.09483975432631953,-0.011754775011588291,8.277005,8.820984,27,G1,1137.3976351469755,8,10,4582.0,1364.0,5.216062854648625,45.1113051069402,0.6170648577776312,False,False,5.300000000000001,14.075966,11.035069,1682.0,9.0,8,0.9446843091192515
-274,True,CAGCAGCAGGTTGCCC-1,CAGCAGCAGGTTGCCC-1,1.8935367023713838,15.0,-0.1032021445427114,-0.014970336586120484,7.4085307,8.899925,15,G1,1075.2252583503723,8,10,8329.0,2111.0,5.97910913675111,42.886300876455756,0.6181604676216871,False,False,3.4999999999999996,14.334815,11.316506,1864.0,9.0,8,0.9558057699680218
-275,True,CAGCAGCAGTCATCCA-1,CAGCAGCAGTCATCCA-1,2.072274772597774,15.0,-0.06417259968063777,-0.021043355766754472,-5.4121737,1.8840361,4,G1,1510.6264656484127,4,4,7450.0,2329.0,4.87248322147651,14.885906040268456,0.9430660231426132,False,False,4.7,-14.232456,1.0581014,1508.0,9.0,4,1.1296894359186769
-276,True,CAGCAGCCACTTCAGA-1,CAGCAGCCACTTCAGA-1,1.8454301616095674,15.0,-0.06400566788695715,-0.01830058918378252,9.411259,-12.613887,21,G1,1174.005507990718,7,6,4954.0,1998.0,7.246669358094469,16.4715381509891,0.02473113873654927,False,False,4.9,11.068652,-11.770066,1928.0,9.0,7,0.8054144824310264
-277,True,CAGCAGCCAGGGAATC-1,CAGCAGCCAGGGAATC-1,1.9643646667892538,15.0,-0.09424605911132186,-0.01275678373867263,0.71400756,18.093674,18,G1,1233.3037815615535,5,5,6161.0,1798.0,9.203051452686251,37.73738029540659,0.7248604763817115,False,False,6.999999999999999,-3.1797683,14.928721,1292.0,9.0,5,0.9949148082922232
-278,True,CAGCAGCCATCCTTCG-1,CAGCAGCCATCCTTCG-1,1.8965761332749582,15.0,-0.05928563429183141,-0.02003089588702408,-3.5908296,-12.916389,24,G1,1336.944579616189,10,7,13322.0,3500.0,8.024320672571687,14.975228944602913,0.9594495181822962,False,False,7.699999999999999,-7.3358603,-11.754679,2063.0,9.0,10,1.5971664117153936
-279,True,CAGCAGCCATGAAGGC-1,CAGCAGCCATGAAGGC-1,1.9773807084516883,15.0,-0.076204586491683,-0.023704974820893447,-5.8436213,-6.9199953,19,G1,1262.0959270745516,2,2,12835.0,3591.0,11.554343591741333,12.458122321776393,0.9605003297729222,False,False,3.5999999999999996,-15.267097,-7.049223,3452.0,9.0,2,1.3619548679656055
-280,True,CAGCAGCCATGGCTAT-1,CAGCAGCCATGGCTAT-1,1.9441397009563925,15.0,-0.09201889140150872,-0.007170106728072034,6.8749213,0.9597962,22,G1,1250.126637890935,6,8,5816.0,1652.0,16.007565337001374,35.14442916093535,0.5815675208758194,False,False,3.6,13.705687,5.19386,11101.0,9.0,6,1.1114423203132549
-281,True,CAGCAGCGTAACTTCG-1,CAGCAGCGTAACTTCG-1,1.9806747537682547,15.0,-0.07609799531464159,-0.012452881275423062,-5.6983137,-8.902907,3,G1,1088.1675548553467,2,2,9842.0,3111.0,12.121520016256857,13.04612883560252,0.9538352937924222,False,False,3.0000000000000004,-15.2246275,-8.665625,2453.0,9.0,2,0.8054144824310264
-282,True,CAGCAGCGTCCCGGTA-1,CAGCAGCGTCCCGGTA-1,1.9423026879724503,15.0,-0.13482675327219729,0.011121327107799502,0.7916824,12.804564,2,S,931.3062853962183,5,5,8443.0,1795.0,5.614118204429705,51.28508823877769,0.7242427935823973,False,False,6.0,-0.43756908,16.642206,1933.0,9.0,5,0.9267070380396117
-283,True,CAGCAGCGTCTAGGTT-1,CAGCAGCGTCTAGGTT-1,1.927168968087493,15.0,-0.081771351342875,-0.01874244640139112,8.133891,-2.9813418,22,G1,1206.0587933510542,6,8,4297.0,1440.0,13.939958110309519,29.346055387479637,0.4483497738408061,False,False,8.0,14.494095,0.5356392,797.0,9.0,6,1.4808010897496513
-284,True,CAGCAGCGTTACACAC-1,CAGCAGCGTTACACAC-1,1.9933168150936433,15.0,-0.07354621550077928,-0.024670686003292447,-12.163645,-5.6390243,7,G1,1127.6028670668602,3,3,10899.0,3290.0,10.945958344802275,19.130195430773465,0.9866743050630565,False,False,3.0,-18.30229,-5.217289,3344.0,9.0,3,1.1068635329128147
-285,True,CAGCAGCGTTAGGAGC-1,CAGCAGCGTTAGGAGC-1,1.9614114783098342,15.0,-0.08590065499104034,-0.018690333814772885,-4.679022,-4.8975286,7,G1,1455.666338995099,3,2,10253.0,3069.0,9.382619721057251,20.257485613966644,0.9639772603702849,False,False,3.0000000000000004,-13.507075,-3.893383,3385.0,9.0,3,1.0536968663288746
-286,True,CAGCAGCGTTCCGTTC-1,CAGCAGCGTTCCGTTC-1,1.8868101806629753,15.0,-0.06614894689721659,0.001800733246639475,0.6283982,6.515158,5,S,1733.8930397331715,1,1,4647.0,1472.0,7.617817947062621,38.99289864428663,0.6172750011358452,False,False,7.800000000000001,9.575956,11.817798,1410.0,9.0,1,1.7325161573380234
-288,True,CAGCAGCTCCGTATGA-1,CAGCAGCTCCGTATGA-1,2.005260149478001,15.0,-0.06572555767546823,-0.020850894637302404,-4.183463,-6.409666,19,G1,1308.2072361707687,2,2,7215.0,2461.0,14.636174636174637,14.594594594594595,0.9558262532214236,False,False,2.8000000000000003,-15.511817,-6.504503,1733.0,9.0,2,1.3352325228116744
-289,True,CAGCAGCTCGCCACTT-1,CAGCAGCTCGCCACTT-1,1.9762233244970147,15.0,-0.07864669622313997,-0.02712270646952304,-8.091279,0.44261587,14,G1,1290.582909077406,4,4,5279.0,2074.0,9.376775904527372,12.710740670581549,0.9524410896749995,False,False,3.0,-13.279324,-0.44757164,1676.0,9.0,4,0.8054144824310264
-290,True,CAGCGTGAGACCAGAC-1,CAGCGTGAGACCAGAC-1,1.902265976409908,15.0,-0.05953971100141961,-0.011441873822386466,11.453809,-8.923208,11,G1,1224.022267267108,7,6,4279.0,1543.0,11.614863285814442,23.930824959102594,0.14103469852712305,True,True,6.7,12.806714,-6.968602,1031.0,9.0,7,1.5775854309189588
-291,True,CAGCGTGAGATCGCCC-1,CAGCGTGAGATCGCCC-1,1.9575179707264967,15.0,-0.10398170452890014,-0.0272639450252243,-9.376102,-8.196126,10,G1,1201.007884055376,3,7,9435.0,2993.0,8.48966613672496,17.62586115527292,0.9691332643703283,False,False,4.5,-12.143355,-8.286798,4712.0,9.0,3,1.2880316103699065
-293,True,CAGCGTGAGGTTGGAC-1,CAGCGTGAGGTTGGAC-1,1.989439819868378,15.0,-0.0748291746597491,-0.02027836820707675,-8.823982,-7.309765,10,G1,1298.15692999959,3,7,9979.0,2823.0,10.241507165046597,16.484617697164044,0.9721469326833069,False,True,3.4000000000000004,-13.497178,-6.6546316,2570.0,9.0,3,1.3600510211168857
-294,True,CAGCGTGCACACCGCA-1,CAGCGTGCACACCGCA-1,1.922676370551629,15.0,-0.08577007694082489,-0.027942945251438287,-3.6864176,-12.391164,24,G1,1307.993331104517,10,7,12801.0,3513.0,9.358643855948754,17.38145457386142,0.9595455706767301,False,False,5.999999999999999,-7.8238006,-11.267099,3263.0,9.0,10,0.960975136721149
-296,True,CAGCGTGCATCCTATT-1,CAGCGTGCATCCTATT-1,1.951849503647278,15.0,-0.09066779002598309,-0.017390136006736452,-6.196989,-3.9396236,13,G1,1026.6741499304771,2,2,9903.0,3017.0,9.532464909623346,17.39876805008583,0.9554242747719779,False,False,3.4,-16.678116,-5.279139,5092.0,9.0,2,1.1209096610163056
-297,True,CAGCGTGCATGGCTAT-1,CAGCGTGCATGGCTAT-1,1.892774074268755,15.0,-0.09850368165593097,-0.014555628445107105,8.893151,4.6832576,8,G1,1143.2166037932038,9,9,5533.0,1808.0,6.000361467558287,35.85758178203506,0.6278893379130539,False,False,2.7999999999999994,11.622207,5.7896385,922.0,9.0,9,0.9577368584765362
-298,True,CAGCGTGGTAACTTCG-1,CAGCGTGGTAACTTCG-1,1.9977278688987814,15.0,-0.06451288872262749,-0.026235365520615003,-5.3351364,-3.4496443,9,G1,1687.5481162667274,2,2,8952.0,2942.0,7.138069705093834,14.622430741733691,0.9600106439296674,False,False,5.5,-15.958025,-3.952847,2138.0,9.0,2,1.4147295055549005
-299,True,CAGCGTGGTATAGGAT-1,CAGCGTGGTATAGGAT-1,1.9126620669578083,15.0,-0.11613802464099122,-0.0006447460591793436,4.610061,5.0987163,12,G1,1785.9470813572407,1,1,3657.0,1214.0,6.808859721082855,45.720535958435875,0.6109482903198683,False,False,3.6999999999999993,11.068561,8.518081,1004.0,9.0,1,1.4923312286500765
-300,True,CAGCGTGGTCCCGGTA-1,CAGCGTGGTCCCGGTA-1,2.03503668905526,15.0,-0.07391438206166984,-0.022177277319707055,-4.7095237,1.4924757,14,G1,1520.8275038152933,4,4,8249.0,2542.0,8.158564674506001,15.504909686022549,0.9457719391270027,False,False,5.3,-15.187228,1.0397762,3315.0,9.0,4,1.8467524754042035
-302,True,CAGCGTGGTCTTCATT-1,CAGCGTGGTCTTCATT-1,1.9378528759246565,15.0,-0.08606832575932068,-0.0220839790089908,-9.19775,-9.1499,10,G1,1082.4515704661608,3,7,12722.0,3389.0,7.506681339412042,20.853639364879736,0.9633561871043795,False,False,3.5,-12.401261,-9.858042,3063.0,9.0,3,1.2567936061319265
-303,True,CAGCGTGGTTAAGACA-1,CAGCGTGGTTAAGACA-1,2.014017201127487,15.0,-0.05613298556990644,-0.022134534567274797,-6.3368793,-0.31831452,20,G1,1345.7171936035156,4,4,9821.0,3108.0,7.881071174014866,15.711231035536096,0.9628520860093824,False,False,2.9999999999999996,-16.871943,-0.18761344,1993.0,9.0,4,0.8054144824310264
-304,True,CAGCGTGTCATCGCTC-1,CAGCGTGTCATCGCTC-1,1.9952659635229375,15.0,-0.08690162934515255,-0.029227433505201517,-6.6559978,-6.100761,3,G1,1501.5781627446413,2,2,6294.0,2405.0,13.806800127105179,13.918017159199238,0.9619277487970058,False,False,4.1,-14.033578,-5.929706,3818.0,9.0,2,1.6086021212745887
-305,True,CAGCGTGTCATGCCCT-1,CAGCGTGTCATGCCCT-1,1.8619634010403365,15.0,-0.09089002966320059,-0.040935151299475,-2.813361,13.258792,2,G1,1078.6742936745286,5,5,10753.0,3528.0,4.268576211289872,19.027248209801915,0.7172978787920485,False,False,4.1,2.3110642,13.378363,1025.0,9.0,5,0.940005096626502
-306,True,CAGCGTGTCCCGTTGT-1,CAGCGTGTCCCGTTGT-1,1.9514984517065972,15.0,-0.09573396297560784,-0.02120014336307013,-10.043522,-2.9829772,10,G1,1483.6279841512442,3,3,14949.0,3630.0,9.385243160077597,31.139206635895377,0.9878772700083635,False,False,4.0,-17.208666,-3.6935847,3503.0,9.0,3,1.0221414249343657
-307,True,CAGCGTGTCCGCTTAC-1,CAGCGTGTCCGCTTAC-1,1.9112246080232664,15.0,-0.11820340664587489,-0.009856909470963772,5.586106,6.5548515,12,G1,1275.0759824216366,1,1,6543.0,1575.0,5.639614855570839,50.16047684548372,0.6126790384106436,False,False,4.6000000000000005,9.451321,7.795113,1270.0,9.0,1,1.3337054561921877
-308,True,CAGCGTGTCGCCTTGT-1,CAGCGTGTCGCCTTGT-1,1.9220335536937052,15.0,-0.10558230889789288,-0.006182985005825939,0.06741738,7.2858453,5,G1,1575.2278412282467,1,1,4778.0,1467.0,7.262452909167015,42.50732524068648,0.6219886162432513,False,False,6.4,9.18971,12.592957,896.0,9.0,1,1.4486019550069733
-309,True,CAGCGTGTCGTCGGGT-1,CAGCGTGTCGTCGGGT-1,1.9426612801465168,15.0,-0.09233395466434968,-0.02811671006448634,-6.829375,-11.165282,3,G1,939.3805213272572,2,2,13914.0,3831.0,9.386229696708352,23.472761247664224,0.9400135920156864,False,False,2.9000000000000004,-15.631174,-10.014842,775.0,9.0,2,0.8054144824310264
-310,True,CAGCGTGTCTAGTCAG-1,CAGCGTGTCTAGTCAG-1,2.0051052040743786,15.0,-0.07092550118734878,-0.024939706140948467,-4.237031,-2.1202898,19,G1,1528.693746805191,2,2,9925.0,2842.0,13.914357682619647,18.811083123425693,0.9572126128390144,False,False,2.8,-14.317675,-2.1408563,2098.0,9.0,2,1.040309407416091
-311,True,CAGCGTGTCTGCCCTA-1,CAGCGTGTCTGCCCTA-1,1.911065684907062,15.0,-0.11789446957105151,-0.008927707456882056,7.943603,10.745313,15,G1,1204.3262564241886,8,10,7224.0,1670.0,6.589147286821706,49.18327796234773,0.6219049513748113,False,False,4.199999999999999,13.94672,10.561295,1066.0,9.0,8,0.9750606631853395
-312,True,CAGGCCAAGATCACCT-1,CAGGCCAAGATCACCT-1,1.8811423824383284,15.0,-0.09675253411856541,-0.016698010619568195,1.7498485,2.791531,12,G1,1452.9915368556976,1,1,7063.0,1863.0,3.9784793996885175,46.45334843550899,0.6107499234706271,False,False,3.3000000000000003,9.990693,8.528086,1937.0,9.0,1,1.3277792842586493
-313,True,CAGGCCAAGGTACAGC-1,CAGGCCAAGGTACAGC-1,2.005267164425277,15.0,-0.05823907393061884,-0.03150587522969367,-6.54021,-1.1740099,4,G1,1600.7668046802282,4,2,10244.0,3196.0,9.263959390862944,9.966809839906286,0.9669574089611214,False,False,2.6,-15.41965,-1.9626967,1895.0,9.0,4,1.4879203446320601
-314,True,CAGGCCAAGTGGCCTC-1,CAGGCCAAGTGGCCTC-1,1.8953359367369411,15.0,-0.11851701929787144,-0.015495060141080975,4.716559,8.064844,27,G1,1320.572511434555,8,1,6576.0,1763.0,5.428832116788321,44.95133819951338,0.6136000585475705,False,False,2.9000000000000004,10.216022,9.458859,1438.0,9.0,8,1.1884528478630074
-315,True,CAGGCCACAAGAGAGA-1,CAGGCCACAAGAGAGA-1,1.8657808204188817,15.0,-0.07057706028518514,-0.020719613018814247,10.879068,-7.935388,11,G1,1234.894474685192,7,6,4987.0,1903.0,5.694806496891919,22.05734910767997,0.14506607725757495,False,False,6.9,13.357323,-7.399864,1510.0,9.0,7,1.5646005659517708
-316,True,CAGGCCACACATGAAA-1,CAGGCCACACATGAAA-1,1.8920698811488235,15.0,-0.09906502698505831,-0.01432935114547684,4.238044,7.274736,12,G1,1253.5317683145404,1,1,7482.0,1930.0,7.805399625768511,43.58460304731355,0.6124788164865407,False,False,4.0,9.091192,8.634979,847.0,9.0,1,1.3291322357508284
-317,True,CAGGCCACATATCTGG-1,CAGGCCACATATCTGG-1,1.9210098054712659,15.0,-0.10327225517098934,-0.012701853060707672,6.263997,4.153552,12,G1,1591.1747448891401,1,9,6952.0,1728.0,6.947640966628309,48.69102416570771,0.6106789253906755,False,False,3.6999999999999997,11.4604645,7.2857785,1604.0,9.0,1,1.5244458930159492
-318,True,CAGGCCAGTACTGAGG-1,CAGGCCAGTACTGAGG-1,1.845746308970741,15.0,-0.06858948890852842,-0.020546391187058824,9.691778,-13.867936,21,G1,1147.7593779861927,7,6,4273.0,1803.0,9.735548794757781,12.941727123800609,0.00379941574657688,False,False,7.3999999999999995,9.880914,-11.997215,1443.0,9.0,7,1.6092122864742475
-319,True,CAGGCCAGTCACGCTG-1,CAGGCCAGTCACGCTG-1,1.9302089866905756,15.0,-0.05125052003304121,-0.025258912393140726,7.4866066,-7.107141,11,G1,948.3033172041178,7,6,12179.0,3937.0,7.283028163231792,12.094589046719763,0.4036574471293984,False,False,3.6000000000000005,14.8980255,-6.969359,3068.0,9.0,7,1.1208407825868383
-320,True,CAGGCCAGTCGGTGTC-1,CAGGCCAGTCGGTGTC-1,1.9309372709956314,15.0,-0.09891338355748591,-0.023404686498884154,-5.556116,-11.733703,25,G1,1131.9383014887571,6,7,12253.0,3482.0,8.389782094181017,15.02489186321717,0.9615941712576579,False,True,2.1000000000000005,-9.389994,-10.947856,3529.0,9.0,6,0.8054144824310264
-321,True,CAGGCCAGTTAAGTCC-1,CAGGCCAGTTAAGTCC-1,1.9096363724609307,15.0,-0.1078625317013818,-0.008665797694832575,7.501019,5.5559645,27,G1,1431.6936453133821,8,10,8504.0,2065.0,8.3137347130762,42.4035747883349,0.6064092113141658,False,False,3.7,13.371239,8.513163,2002.0,9.0,8,1.3225285162333158
-322,True,CATACAGAGAAATTGC-1,CATACAGAGAAATTGC-1,1.8937425165336728,15.0,-0.09812440428399043,-0.024606299212598427,7.143212,9.289116,15,G1,1401.8904049396515,8,10,8128.0,2313.0,5.167322834645669,41.22785433070866,0.6181134282600291,False,False,2.7000000000000006,14.296753,9.592731,961.0,9.0,8,0.9393306927058855
-323,True,CATACAGAGCAGGTCA-1,CATACAGAGCAGGTCA-1,1.933137208453172,15.0,-0.07702682734358052,-0.0015669925895179342,2.4695525,7.5136595,5,G1,1489.3200511932373,1,1,3983.0,1186.0,12.553351744915892,42.229475269897065,0.6127720778852997,False,False,3.7999999999999994,8.25029,11.642545,986.0,9.0,1,1.0102850184689025
-324,True,CATACAGAGCGACATG-1,CATACAGAGCGACATG-1,1.888774982085284,15.0,-0.07747857006367286,-0.025796163406969656,12.031397,-8.980037,11,G1,1198.9395132660866,7,6,2484.0,918.0,6.884057971014493,30.95813204508857,0.13526571132841314,False,False,6.499999999999999,12.723535,-7.5385222,985.0,9.0,7,0.9962154886322245
-325,True,CATACAGAGCTTAAGA-1,CATACAGAGCTTAAGA-1,1.8975941600817567,15.0,-0.10971987881362615,-0.00020798144104976907,6.5040245,4.1625996,8,G1,1520.7485368549824,9,1,8447.0,2147.0,5.789037528116491,42.39374926009234,0.6089036031296702,False,False,2.9000000000000004,12.57322,7.748602,1876.0,10.0,9,1.1982591168648236
-326,True,CATACAGAGGTTACAA-1,CATACAGAGGTTACAA-1,1.8813890824811015,15.0,-0.10194224968979472,-0.01668604452402199,7.716195,6.6074505,27,G1,1543.5538897812366,8,10,5788.0,1605.0,3.1962681409813407,46.45818935729095,0.6112718993951893,False,False,4.1,13.524274,9.127428,1957.0,9.0,8,1.237002859343705
-327,True,CATACAGCAAGAGAGA-1,CATACAGCAAGAGAGA-1,2.055712856022889,15.0,-0.08439762082682306,-0.019729103390157827,-6.140167,3.2012618,14,G1,1418.0869150608778,4,4,6320.0,2219.0,9.746835443037975,14.794303797468354,0.9329942909921487,False,False,6.100000000000001,-13.928124,2.0932927,3268.0,9.0,4,1.7527888243976026
-328,True,CATACAGCACCAGCGT-1,CATACAGCACCAGCGT-1,1.920380542338816,15.0,-0.1171352115077595,-0.00694441476083987,6.562728,3.236687,8,G1,1497.458046272397,9,9,7463.0,1859.0,8.522042074232882,44.15114565188262,0.6098702594880665,False,False,3.9000000000000004,12.360581,6.562899,964.0,9.0,9,1.5678310544261775
-330,True,CATACAGCAGCTGCCA-1,CATACAGCAGCTGCCA-1,1.9741548829208306,15.0,-0.06321792851389152,-0.010616690339016754,-8.669186,-0.65960777,7,G1,1581.8901606947184,3,3,7895.0,2818.0,8.803039898670045,12.260924635845472,0.9742986893517108,False,False,4.199999999999999,-15.230895,-2.5533588,3090.0,9.0,3,1.1385112939403332
-331,True,CATACAGCATATCTGG-1,CATACAGCATATCTGG-1,1.960414884617253,15.0,-0.10450853757498785,-0.019925577966295883,1.1727306,16.09752,2,G1,1099.15285089612,5,5,5847.0,1764.0,11.065503677099366,37.55772190867111,0.7191933505331903,False,True,9.5,-2.4869375,14.272452,1013.0,9.0,5,1.3598537505011798
-332,True,CATACAGGTCACTCGG-1,CATACAGGTCACTCGG-1,1.9888506562836559,15.0,-0.057456271119061826,-0.027679626327337836,-5.7840734,2.8080409,14,G1,1459.046968266368,4,4,4032.0,1783.0,7.192460317460317,10.66468253968254,0.9396685559347789,False,False,5.7,-14.941632,2.0607872,2474.0,9.0,4,1.3937894669484336
-333,True,CATACAGGTGTCTAAC-1,CATACAGGTGTCTAAC-1,1.8999020713830626,15.0,-0.11597629978128718,-0.009389648946489342,6.5228305,0.58743936,22,G1,1278.6056822538376,6,8,9296.0,2187.0,7.669965576592083,42.10413080895009,0.5763361859966477,False,False,4.3999999999999995,13.538184,5.8749685,2214.0,9.0,6,1.1721280544238497
-334,True,CATACAGGTTGGACCC-1,CATACAGGTTGGACCC-1,1.9939380892502347,15.0,-0.0778389144805403,-0.011263797547426308,-11.732965,-4.9690776,1,G1,1006.5759285539389,3,3,8603.0,2817.0,8.671393699872137,15.715448099500174,0.9911677128853777,False,False,3.6000000000000005,-18.748152,-5.123764,2309.0,9.0,3,1.0292877503014077
-335,True,CATACAGGTTGGGATG-1,CATACAGGTTGGGATG-1,2.0158538887071504,15.0,-0.0653749145908374,-0.020320012824726657,-3.8350995,-4.4519796,3,G1,1574.6938680708408,2,2,9119.0,2949.0,14.277881346638885,11.02094527908762,0.9571298945422917,False,False,3.5,-15.401074,-4.7322435,2011.0,9.0,2,1.2521096874567261
-337,True,CCCTCTCAGACGTCCC-1,CCCTCTCAGACGTCCC-1,1.9650613238965864,15.0,-0.08677111002692398,-0.028949691085613415,-6.724183,-10.43392,3,G1,963.5958538353443,2,2,8250.0,2664.0,8.4,18.8,0.9482690413867253,False,False,2.8,-15.596768,-9.6329365,3747.0,9.0,2,1.1647220741717665
-338,True,CCCTCTCAGCAGGTCA-1,CCCTCTCAGCAGGTCA-1,2.0727647988638918,15.0,-0.06519335245248425,-0.02680771821094998,-4.99498,4.0468736,14,G1,1355.4826240837574,4,4,7514.0,2347.0,9.901517167953154,14.559488953952622,0.9134535085992458,False,False,6.7,-13.468655,2.5225534,1774.0,9.0,4,1.282823347179086
-339,True,CCCTCTCAGGTCTACT-1,CCCTCTCAGGTCTACT-1,1.8789012904058646,15.0,-0.09957808110434618,-0.00921749936833019,3.788646,6.412645,12,G1,1684.116705223918,1,1,3837.0,1318.0,6.385196768308575,36.87776909043524,0.6121939119350903,False,False,3.8000000000000003,9.478539,9.137184,1156.0,9.0,1,1.0180081103546588
-340,True,CCCTCTCAGTCATCCA-1,CCCTCTCAGTCATCCA-1,1.9540887572581178,15.0,-0.07753709340731804,-0.022227524200471874,-6.854687,-7.053111,3,G1,1432.4177133589983,2,2,11109.0,3671.0,8.794670987487622,8.88468809073724,0.9597970431774242,False,False,3.6999999999999997,-15.808394,-6.734242,2277.0,10.0,2,1.66271512327643
-341,True,CCCTCTCCAAATGCGG-1,CCCTCTCCAAATGCGG-1,1.9916698723749264,15.0,-0.07193492331417284,-0.029654503549457034,-10.929289,-0.8748561,7,G1,1456.9272047281265,3,3,14633.0,3780.0,7.223399166268024,17.282853823549512,0.9938070510641033,False,False,4.799999999999999,-17.777576,-1.7393465,2720.0,9.0,3,1.3198912675453556
-342,True,CCCTCTCCACACCGCA-1,CCCTCTCCACACCGCA-1,1.895991602905573,15.0,-0.10983177527136297,-0.013013711488243589,5.3682623,3.131261,12,G1,1547.5188335478306,1,9,5812.0,1663.0,5.6951135581555405,41.41431520991053,0.611858401463065,False,False,3.7,11.946625,7.704663,1086.0,9.0,1,1.0097827324533222
-343,True,CCCTCTCCACAGAGAC-1,CCCTCTCCACAGAGAC-1,1.9459408824495976,15.0,-0.11883757560003565,-0.022099715125589958,-1.7543505,13.569383,2,G1,1141.1378885358572,5,5,6414.0,1680.0,8.481446835048331,41.98628001247272,0.7336333754896308,False,False,6.2,1.0240138,15.433463,1143.0,9.0,5,1.0750721142572695
-344,True,CCCTCTCCATTATGCG-1,CCCTCTCCATTATGCG-1,1.8632198530128166,15.0,-0.0820160166634955,0.0006393675329864276,9.748156,-6.4506683,23,S,1159.1340185254812,6,8,3977.0,1673.0,6.462157405079205,22.831279859190346,0.22839643503859605,False,False,5.800000000000001,14.124119,-5.0394874,1328.0,9.0,6,1.1919946538905841
-345,True,CCCTCTCGTACGGGAT-1,CCCTCTCGTACGGGAT-1,2.0487020078335343,15.0,-0.06140721209687111,-0.02636932951525803,-7.163423,3.3625908,14,G1,1403.782047227025,4,4,4178.0,1726.0,14.576352321685016,10.172331258975586,0.9415186649904357,False,False,5.1,-13.907275,1.833342,1590.0,9.0,4,1.1353175038090075
-346,True,CCCTCTCGTATACCTG-1,CCCTCTCGTATACCTG-1,1.8800198998827067,15.0,-0.06546841278065069,-0.026770819385812213,8.830024,-13.982556,21,G1,1083.0468273460865,7,6,5490.0,2287.0,12.313296903460838,10.418943533697632,0.0018762984288668262,False,False,7.8999999999999995,9.764536,-12.200286,1438.0,9.0,7,1.2260956028822898
-347,True,CCCTCTCGTCACTCGG-1,CCCTCTCGTCACTCGG-1,1.9035041147763876,15.0,-0.10452354600712949,-0.011091586979857827,7.801524,-1.9930842,22,G1,1083.1298670172691,6,8,6294.0,1859.0,8.102955195424213,37.639021290117576,0.4763128614103582,False,True,4.3999999999999995,14.436056,1.5302564,1204.0,9.0,6,1.1137790287560545
-348,True,CCCTCTCGTCTAGGTT-1,CCCTCTCGTCTAGGTT-1,1.9936208127115704,15.0,-0.07179089925946022,-0.02228598383635436,-5.979312,-2.2953367,17,G1,1536.17895257473,2,2,8466.0,3028.0,9.225135837467517,10.713442003307348,0.9614207277145181,False,False,2.8000000000000003,-15.343706,-2.7011223,2318.0,9.0,2,0.8054144824310264
-349,True,CCCTCTCGTGACGCCT-1,CCCTCTCGTGACGCCT-1,2.250942876960828,14.0,0.03210432397785415,-0.021817388458601504,-8.475535,-12.748832,16,G2M,720.7980148494244,2,2,534.0,256.0,22.846441947565545,4.119850187265918,0.9058916952231162,False,False,2.8000000000000003,-15.072312,-11.330073,963.0,9.0,2,0.8054144824310264
-350,True,CCCTCTCGTGCACAAG-1,CCCTCTCGTGCACAAG-1,1.878782848250622,15.0,-0.07834038157508054,-0.017564270309774054,9.318883,-12.701311,21,G1,1198.8450797349215,7,6,8723.0,3031.0,8.655279147082426,19.546027742749054,0.017439526960035093,False,False,5.1,10.951147,-11.4422245,1824.0,9.0,7,1.438770127615809
-351,True,CCCTCTCGTGCGGTAA-1,CCCTCTCGTGCGGTAA-1,1.9513802363668515,15.0,-0.08521237331828534,-0.03961122512999926,-10.3360615,-4.4438515,1,G1,1195.213123768568,3,3,7304.0,2535.0,7.8450164293537785,10.008214676889375,0.9862659104939607,False,False,3.4,-13.385398,-4.4662795,3395.0,9.0,3,1.0539205889133303
-352,True,CCCTCTCGTTCCACGG-1,CCCTCTCGTTCCACGG-1,1.8524357921032255,15.0,-0.0707747344177394,-0.00869661136451734,10.068282,-5.129393,23,G1,1142.5617601424456,6,8,5348.0,1969.0,6.020942408376963,25.523560209424083,0.3006275194622053,False,False,7.199999999999999,14.246238,-3.644408,1369.0,9.0,6,1.3308283858636578
-353,True,CCCTCTCGTTGGACCC-1,CCCTCTCGTTGGACCC-1,1.925535229860219,15.0,-0.04278015105758644,-0.0023465772801228857,0.66477543,7.7751274,5,G1,1629.3074842989445,1,1,2049.0,877.0,15.617374328940947,19.960956564177646,0.6201020830043741,False,False,6.4,9.281414,12.343484,1799.0,9.0,1,1.6335883792588684
-354,True,CCCTCTCTCCCAAGCG-1,CCCTCTCTCCCAAGCG-1,1.8829287457467798,15.0,-0.09685613933496715,-0.017014119937965273,1.1897258,4.18516,12,G1,1714.21916821599,1,1,5190.0,1622.0,6.377649325626204,37.39884393063584,0.6125050375925161,False,False,4.5,10.176049,10.429145,2161.0,9.0,1,1.0502393374153849
-355,True,CCCTCTCTCCGCACGA-1,CCCTCTCTCCGCACGA-1,1.9065998780812494,15.0,-0.07727762053326268,-0.020660977063113203,-3.0104413,-12.65278,24,G1,1322.6958612799644,10,7,10828.0,2798.0,8.330254894717399,21.001108237901736,0.9592728554700117,False,False,7.3,-6.539811,-11.624563,3020.0,9.0,10,0.9758455406552953
-356,True,CCCTCTCTCGAACGCC-1,CCCTCTCTCGAACGCC-1,1.9166645223951313,15.0,-0.10481363001813558,-0.0031597127915674664,4.0462704,3.5157738,12,G1,1858.6506461501122,1,1,4887.0,1456.0,7.612031921424187,42.09126253325148,0.608022887381211,False,False,5.800000000000001,11.739289,9.2189665,1459.0,9.0,1,1.302400829963068
-357,True,CCCTCTCTCTCCGAGG-1,CCCTCTCTCTCCGAGG-1,1.9033908828207595,15.0,-0.08222446710415256,-0.00648670306096745,5.1122775,1.6823483,12,G1,1631.7853568941355,1,8,4034.0,1252.0,10.436291522062469,38.795240456122954,0.591153983761569,False,False,3.1999999999999997,12.626924,6.7854333,811.0,9.0,1,1.0874029793228943
-358,True,CCCTCTCTCTCTAGGA-1,CCCTCTCTCTCTAGGA-1,1.9028863695431333,15.0,-0.09770354478510528,-0.007140147121334773,5.7871876,6.718869,27,G1,1550.562692336738,8,1,9434.0,2356.0,6.667373330506678,44.42442230231079,0.6124064567429208,False,False,3.9,11.92023,9.726447,788.0,9.0,8,1.2166313146772585
-359,True,CGGCAGTAGAGAGCGG-1,CGGCAGTAGAGAGCGG-1,1.9189803764903852,15.0,-0.0784759164468842,-0.015197015770302465,10.020934,-8.294735,11,G1,1207.5650903135538,7,6,4183.0,1452.0,12.048768826201291,30.289266076978244,0.14968138158519695,False,False,6.5,13.454297,-7.468745,1261.0,9.0,7,1.026932343957817
-360,True,CGGCAGTCACACCGCA-1,CGGCAGTCACACCGCA-1,1.91861939466199,15.0,-0.08618632591090962,-0.009278795979878313,-2.9263196,-13.675326,24,G1,1295.115891739726,10,7,14564.0,3572.0,5.8019774787146385,17.865970887118923,0.9593405261657042,False,False,6.8,-7.4710293,-12.578688,3003.0,9.0,10,0.9403405532365913
-361,True,CGGCAGTCACCAGCGT-1,CGGCAGTCACCAGCGT-1,1.953151411849151,15.0,-0.08565734879984924,-0.0266721496939675,-8.200189,-9.872453,10,G1,995.4691574722528,3,7,8587.0,2873.0,11.342727378595551,14.708279958076162,0.9587158119403458,False,False,3.6,-13.263959,-10.044881,2524.0,9.0,3,0.8054144824310264
-362,True,CGGCAGTCAGTGTGCC-1,CGGCAGTCAGTGTGCC-1,1.9193662858855807,15.0,-0.09321135289020749,-0.011265385281897343,-2.616632,-11.732062,24,G1,1269.4341300725937,10,7,10506.0,2515.0,7.766990291262136,25.785265562535695,0.9596313292574161,False,False,6.4,-7.1672173,-11.947493,2040.0,9.0,10,1.0015219091544367
-364,True,CGGCAGTGTAGAATGT-1,CGGCAGTGTAGAATGT-1,1.9514264651818216,15.0,-0.08176782883259245,-0.022586705633852473,-8.533034,-4.816835,10,G1,1122.3878692090511,3,7,9258.0,3069.0,8.425145819831497,13.728667098725426,0.9713576694284423,False,False,3.5999999999999996,-13.489421,-6.1341186,3208.0,9.0,3,1.2264156125468055
-365,True,CGGCAGTGTAGGTTTC-1,CGGCAGTGTAGGTTTC-1,1.9080313332936878,15.0,-0.11161794103824668,-0.008990922748479999,4.553734,0.99636346,22,G1,1385.6881998628378,6,1,10459.0,2333.0,6.367721579500908,46.96433693469739,0.6002382226582099,False,False,3.4000000000000004,13.426334,7.512413,1512.0,9.0,6,0.9904197242749064
-366,True,CGGCAGTGTATAGGAT-1,CGGCAGTGTATAGGAT-1,1.9269289403375431,15.0,-0.071968103287653,-0.022021385476749714,-2.2821681,-13.615139,24,G1,1283.0497619509697,10,7,10094.0,2580.0,10.768773528829007,21.983356449375865,0.959278819710406,False,False,7.2,-6.744525,-11.4052,1734.0,9.0,10,0.8054144824310264
-367,True,CGGCAGTGTATGCGTT-1,CGGCAGTGTATGCGTT-1,1.9104318405110468,15.0,-0.1266591064961649,-0.012929130369766702,2.7881358,7.0473595,5,G1,1695.3633325397968,1,1,8632.0,1982.0,7.599629286376274,48.58665430954588,0.6122969479783753,False,False,3.5999999999999996,9.068408,10.808477,919.0,9.0,1,1.112152250731469
-368,True,CGGCAGTGTCCACACG-1,CGGCAGTGTCCACACG-1,1.941869323756232,15.0,-0.08978439722216792,-0.01607590487942517,-2.08065,-13.0179405,24,G1,1287.983282983303,10,7,11259.0,2848.0,9.59232613908873,22.9061195488054,0.9593924149137347,False,False,6.8999999999999995,-6.5958343,-12.141531,2934.0,10.0,10,0.9788750406701542
-369,True,CGGCAGTGTCGAATGG-1,CGGCAGTGTCGAATGG-1,1.91850348856739,15.0,-0.11757049017796165,-0.01524024293440331,2.04788,5.8373904,12,G1,1442.5792770534754,1,1,5063.0,1374.0,5.8660873000197515,48.92356310487853,0.6322546940490276,False,True,2.6999999999999997,11.942179,11.090333,1558.0,9.0,1,1.0399810749184109
-370,True,CGGCAGTGTGCCTTCT-1,CGGCAGTGTGCCTTCT-1,1.882804736068339,15.0,-0.11011395776279331,-0.004462429786456331,11.505131,-9.56628,11,G1,1249.715460985899,7,6,5636.0,1762.0,8.481192334989354,35.87650816181689,0.1171481816461192,False,False,5.699999999999999,12.66915,-8.075925,1960.0,9.0,7,1.0690030998731175
-372,True,CGGCAGTGTTGGAGGT-1,CGGCAGTGTTGGAGGT-1,1.9614205628243575,15.0,-0.07241067824685067,-0.026341848121577267,-7.840459,-3.9509761,10,G1,1370.053646236658,3,3,12089.0,3469.0,5.782115973198776,18.810488874183143,0.966293900942943,False,False,3.1999999999999997,-13.442972,-4.2361517,1904.0,9.0,3,1.8507823626714903
-374,True,CGGCAGTTCCGGACGT-1,CGGCAGTTCCGGACGT-1,1.959377445367422,15.0,-0.09586438028343827,-0.0025033617645135627,-1.3511472,16.022533,2,G1,1241.5621903464198,5,5,4617.0,1461.0,9.075157028373402,38.05501407840589,0.7317128039812371,True,True,7.6000000000000005,0.05219317,14.3996935,1190.0,9.0,5,1.6723068237584109
-376,True,CGTCCATAGTGTTGAA-1,CGTCCATAGTGTTGAA-1,1.9551381239704457,15.0,-0.07168697944855575,-0.018093196109716485,-9.749792,-1.4847265,7,G1,1611.5776236951351,3,3,8445.0,2917.0,7.34162226169331,11.095322676139727,0.979703315802304,False,False,4.800000000000001,-15.030734,-1.3498517,3338.0,9.0,3,1.1077349154430558
-377,True,CGTCCATCACGCGCAT-1,CGTCCATCACGCGCAT-1,1.897839474835368,15.0,-0.09967999113264568,-0.02567779431363437,10.080686,-4.849407,23,G1,1145.8119096755981,6,8,5369.0,1715.0,10.29986962190352,30.974110635127584,0.31663482625370276,False,False,7.199999999999999,14.228688,-3.1542192,2116.0,9.0,6,1.1834856120867876
-378,True,CGTCCATCACTACACA-1,CGTCCATCACTACACA-1,1.9848601002011872,15.0,-0.09237949349557881,-0.0035592240437654563,-0.07276739,17.431839,18,G1,1263.0185985416174,5,5,13444.0,3134.0,12.845879202618269,34.58048199940494,0.7278801626185175,False,False,5.1,-1.8403897,14.943713,981.0,9.0,5,0.9949148082922232
-379,True,CGTCCATCAGCAGTGA-1,CGTCCATCAGCAGTGA-1,2.0148173549837534,15.0,-0.060994026555763414,-0.0379385941725445,-5.2425203,0.57739353,20,G1,1581.4556798487902,4,4,10363.0,3166.0,7.266235646048441,13.01746598475345,0.9520007705325964,False,False,4.5,-15.222205,-0.08154499,1964.0,9.0,4,1.8882570788227317
-380,True,CGTCCATCAGGGAATC-1,CGTCCATCAGGGAATC-1,1.9400373078917716,15.0,-0.08939361012202382,-0.013584944432013473,-4.3653855,-13.656964,24,G1,1135.3985050618649,10,7,14957.0,3640.0,8.892157518218895,20.45864812462392,0.9600429195778493,True,True,4.8999999999999995,-8.228996,-12.017874,3088.0,9.0,10,1.0690719696785353
-381,True,CGTCCATCATATCTGG-1,CGTCCATCATATCTGG-1,1.8738944654810383,15.0,-0.1256616595215034,0.005305883618873346,7.735076,-2.4962428,22,S,1159.927320331335,6,8,4043.0,1179.0,5.1694286420974525,47.58842443729903,0.46251057498825315,False,False,6.6000000000000005,13.938397,0.116523124,896.0,9.0,6,1.0088873921243666
-383,True,CGTCCATGTCAAGCGA-1,CGTCCATGTCAAGCGA-1,1.9604554209452123,15.0,-0.10939808854586157,-0.005156510169734916,-4.0254354,3.0926068,14,G1,1235.2945231050253,4,4,3667.0,973.0,0.436323970548132,39.29642759749114,0.8732016156226156,False,False,4.5,-13.154397,2.3658345,1195.0,9.0,4,1.2492640120368883
-384,True,CGTCCATGTGCACAAG-1,CGTCCATGTGCACAAG-1,1.9638201520581517,15.0,-0.0834095996571717,-0.01841503737227947,-5.208086,-5.389476,13,G1,1562.7849843651056,2,2,9746.0,3194.0,8.629181202544634,19.19761953621999,0.9563291590313003,False,False,3.4999999999999996,-16.275427,-5.350124,2809.0,9.0,2,1.4381038299180309
-385,True,CGTCCATGTTAGGAGC-1,CGTCCATGTTAGGAGC-1,1.968595662266925,15.0,-0.07768042787538369,-0.014567289483195842,7.766992,-7.5480494,11,G1,923.4268907010555,7,6,11219.0,3281.0,10.170246902575988,15.545057491755058,0.3366313829837774,False,False,3.5000000000000004,14.793212,-7.4846296,3174.0,9.0,7,1.2201809404287483
-386,True,CGTCCATGTTCGGCTG-1,CGTCCATGTTCGGCTG-1,1.8974652750913374,15.0,-0.10751379422265497,-0.0064373789333848335,8.290299,4.7089496,26,G1,1392.7411314621568,9,9,6320.0,1735.0,5.965189873417722,43.18037974683544,0.6113259043998333,False,False,4.3999999999999995,11.012638,6.1099777,1461.0,9.0,9,1.4851538553532957
-387,True,CGTCCATTCTGCCCTA-1,CGTCCATTCTGCCCTA-1,2.0557147611182756,15.0,-0.08016922090770714,-0.032040677634204366,-6.8965273,3.938574,14,G1,1426.5106876343489,4,4,3362.0,1622.0,12.671029149315883,8.149910767400357,0.9398242550489379,False,False,5.8999999999999995,-13.651529,2.621331,1777.0,9.0,4,1.1353175038090075
-388,True,GAGGGTAAGACCAGAC-1,GAGGGTAAGACCAGAC-1,1.961650454423746,15.0,-0.09057114498523394,-0.018522757706469654,-8.651427,-1.7656965,7,G1,1608.3845952004194,3,3,9270.0,3170.0,10.35598705501618,13.829557713052859,0.9843163147244981,False,False,5.2,-15.911959,-1.9679083,3383.0,9.0,3,1.4878660176210219
-389,True,GAGGGTAAGCACCTGC-1,GAGGGTAAGCACCTGC-1,1.8986477362807002,15.0,-0.07663836868420623,-0.01777441273712556,4.0701923,5.6515675,12,G1,1781.244106695056,1,1,2484.0,939.0,12.077294685990339,29.871175523349436,0.6114210926081723,False,False,5.2,10.004462,8.897074,1245.0,9.0,1,1.4120562125104608
-390,True,GAGGGTAAGCGACATG-1,GAGGGTAAGCGACATG-1,1.9163577732868302,15.0,-0.11475349280132699,0.0007243675014130188,5.050214,8.841273,27,S,1051.6663328632712,8,1,6893.0,1835.0,8.356303496300594,45.32134049035253,0.6292875876114878,False,False,3.3000000000000003,11.489351,11.575405,659.0,9.0,8,0.9411293383160607
-391,True,GAGGGTAAGGCCTAGA-1,GAGGGTAAGGCCTAGA-1,1.990867429296621,15.0,-0.07021314267921142,-0.028225240193827626,-9.756679,0.1068885,14,G1,1469.558833822608,4,3,7325.0,2538.0,11.890784982935154,13.924914675767917,0.982692616376982,False,False,3.9000000000000004,-17.020735,-1.1734837,1049.0,9.0,4,1.2603903856899734
-392,True,GAGGGTAAGGGAGGTG-1,GAGGGTAAGGGAGGTG-1,2.0732674181043933,15.0,-0.06703346250589129,-0.025798240559993805,-4.4784493,2.9041028,14,G1,1424.530461370945,4,4,3387.0,1439.0,13.404192500738116,12.370829642751698,0.9264838684732848,False,False,5.799999999999999,-14.152505,2.172565,1554.0,9.0,4,1.1702966058509467
-393,True,GAGGGTAAGGTCTACT-1,GAGGGTAAGGTCTACT-1,1.9823189307958076,15.0,-0.06490851249615567,-0.027769919090017622,-5.814096,-2.1105,17,G1,1264.9400390088558,2,2,12060.0,3366.0,6.865671641791045,16.558872305140962,0.9653027370680748,False,False,4.1,-17.103138,-2.7483957,3334.0,9.0,2,1.0984708838805604
-395,True,GAGGGTAAGTGTTGAA-1,GAGGGTAAGTGTTGAA-1,1.987437486753611,15.0,-0.09709189018285534,-0.037129601901632676,-8.714432,-5.3014216,3,G1,1330.4915774613619,2,2,10407.0,3056.0,10.2527145190737,14.394157778418371,0.9774933477979837,False,False,2.5,-16.38343,-5.1587996,2354.0,9.0,2,1.1941258902583858
-396,True,GAGGGTACAAATGCGG-1,GAGGGTACAAATGCGG-1,1.9245385070443923,15.0,-0.11499893922686662,-0.0010491908031318685,8.327663,9.585501,15,G1,1103.3345563858747,8,10,5464.0,1456.0,9.132503660322108,46.92532942898975,0.6207294734795556,False,False,5.299999999999999,15.300105,9.9095125,990.0,9.0,8,0.9562600999765033
-397,True,GAGGGTACAGGTTCAT-1,GAGGGTACAGGTTCAT-1,1.9629175723863408,15.0,-0.09853542184795729,-0.018727753158252806,-9.179546,-9.686361,10,G1,1123.5468605011702,3,7,9840.0,3102.0,5.813008130081301,13.729674796747968,0.9647556179414448,False,False,3.8999999999999995,-11.644603,-9.733569,1813.0,9.0,3,1.0246186277677674
-398,True,GAGGGTAGTAGTGTGG-1,GAGGGTAGTAGTGTGG-1,1.9058988445789706,15.0,-0.10856950260136691,-0.03075753357360396,-2.1277275,11.959253,2,G1,877.4017400145531,5,10,13032.0,3557.0,6.645181092694905,27.892879066912215,0.6632566775498758,False,False,3.4,3.8058274,13.140235,1321.0,9.0,5,0.8054144824310264
-399,True,GAGGGTAGTTAGGAGC-1,GAGGGTAGTTAGGAGC-1,1.8933691371161094,15.0,-0.11271625356750414,0.00490816098070174,7.0387516,6.3973045,27,S,1403.34445861727,8,10,7950.0,1978.0,4.50314465408805,46.77987421383648,0.6120070955716002,False,False,3.1,12.7371025,9.14054,1180.0,9.0,8,1.2452746909952663
-400,True,GAGGGTAGTTCGAAGG-1,GAGGGTAGTTCGAAGG-1,1.898319146576067,15.0,-0.09131883972891351,-0.007515213013949716,0.8755439,5.854018,5,G1,1738.541418850422,1,1,3931.0,1311.0,13.96591198168405,35.43627575680488,0.6138376051784263,False,False,7.1,9.105708,11.2875,970.0,9.0,1,1.2501517667997706
-401,True,GAGGGTAGTTCGGCTG-1,GAGGGTAGTTCGGCTG-1,1.9848565863362932,15.0,-0.07146221461776396,-0.025524540378327947,-6.6536975,-3.356771,10,G1,1482.3660599440336,3,3,9185.0,2907.0,9.461077844311378,12.923244420250409,0.9699474686308385,False,False,3.8000000000000003,-15.409359,-3.8414571,4520.0,9.0,3,1.1082909943197194
-402,True,GAGGGTAGTTGGACCC-1,GAGGGTAGTTGGACCC-1,1.9296106587218,15.0,-0.07105703094917791,-0.006625854381588168,6.868426,6.020367,8,G1,1367.8595857471228,9,1,2484.0,931.0,20.531400966183575,16.70692431561997,0.61463879256387,False,False,3.6,11.247502,7.6633477,1145.0,9.0,9,1.1236781032398344
-403,True,GAGGGTATCAGGGATG-1,GAGGGTATCAGGGATG-1,1.9629355320280748,15.0,-0.07350796750894377,-0.02769243867690122,-8.162191,-5.9455304,10,G1,1458.1405840069056,3,3,9481.0,3085.0,11.275181942833035,15.937137432760258,0.9690401899612058,False,False,4.3,-12.933802,-6.197296,2810.0,9.0,3,1.2974956605861356
-404,True,GAGGGTATCCGGACGT-1,GAGGGTATCCGGACGT-1,1.8898340910420204,15.0,-0.11660011022389004,-0.016778144530808867,7.796214,7.3390613,27,G1,1252.5737234055996,8,10,6018.0,1548.0,3.6556995679627784,51.41242937853107,0.6111643043872729,True,True,4.3,13.84853,9.450892,916.0,9.0,8,1.3781356827645914
-405,True,GAGGGTATCGAACGCC-1,GAGGGTATCGAACGCC-1,1.9508317339695467,15.0,-0.08935355801733412,-0.03022650797262577,-9.965252,-9.140249,10,G1,1163.1111931204796,3,7,12331.0,3685.0,9.147676587462493,13.064633849647231,0.9694202839666316,False,False,5.6000000000000005,-11.540826,-9.053819,4456.0,9.0,3,0.997428114413575
-406,True,GAGGGTATCGTTCCTG-1,GAGGGTATCGTTCCTG-1,1.9644084838018074,15.0,-0.0967348858100616,-0.0020386241074561544,0.43639222,15.343253,2,G1,1111.9533635675907,5,5,6622.0,1706.0,11.3711869525823,39.972817879794626,0.7187888891670337,False,False,7.4,-1.3966873,14.906741,750.0,9.0,5,1.072372799549762
-407,True,GAGGGTATCTCCGAGG-1,GAGGGTATCTCCGAGG-1,1.875858905758202,15.0,-0.10640138014538868,-0.02340394824606911,5.405017,8.068325,27,G1,1233.4610465466976,8,1,4978.0,1430.0,5.04218561671354,46.082764162314184,0.6144507283104254,False,False,3.6999999999999997,11.001662,11.630738,1341.0,9.0,8,0.8054144824310264
-409,True,GAGTTGTAGCCTCTTC-1,GAGTTGTAGCCTCTTC-1,1.8773614782874604,15.0,-0.10355691354429412,0.008183198519617577,10.644986,-5.539434,23,S,1123.8978370130062,6,8,4644.0,1559.0,6.2446167097329885,31.93367786391042,0.2659509262947879,False,False,7.1,14.032845,-4.4738197,1490.0,9.0,6,0.9492563449557461
-411,True,GAGTTGTCAAGTCGTT-1,GAGTTGTCAAGTCGTT-1,2.028373827987996,15.0,-0.09164821624446946,-0.0286453428308876,-9.145013,-6.962429,10,G1,1288.5011351555586,3,7,5184.0,2052.0,13.194444444444445,10.185185185185185,0.9738773330789481,False,False,3.5000000000000004,-13.448934,-7.1246886,2824.0,9.0,3,1.3786711877855171
-412,True,GAGTTGTCACTAAACC-1,GAGTTGTCACTAAACC-1,2.0154160160939285,15.0,-0.07948892051760836,-0.016113737678474408,0.922361,14.672169,2,G1,1095.8173262178898,5,5,2740.0,986.0,16.532846715328468,28.868613138686133,0.7193508806257586,False,False,7.8999999999999995,-1.2680078,15.483536,740.0,9.0,5,0.9638800466103177
-414,True,GAGTTGTCATGAATAG-1,GAGTTGTCATGAATAG-1,1.9100172876826065,15.0,-0.07600377193149989,-0.004251723874867004,9.746234,2.331165,25,G1,1147.4085248559713,6,9,5339.0,1709.0,8.203783480052444,32.10339014796779,0.5980212558898053,False,True,4.5,12.633586,4.377047,1097.0,9.0,6,1.1051376087517883
-415,True,GAGTTGTGTCGTGCCA-1,GAGTTGTGTCGTGCCA-1,1.9125066123734022,15.0,-0.09826782597365227,0.020372985589480138,8.943073,-10.758092,21,S,1282.264548331499,7,6,3295.0,1206.0,12.200303490136571,25.91805766312595,0.06510220068375193,False,False,7.1,11.97742,-10.226901,610.0,9.0,7,1.0599277571389403
-416,True,GAGTTGTGTCTAGGTT-1,GAGTTGTGTCTAGGTT-1,1.967556475422572,15.0,-0.07718401855791882,-0.031144757074560894,-11.43003,-4.152301,1,G1,1129.0501042604446,3,3,8197.0,2763.0,11.101622544833475,15.042088568988655,0.9966516297245044,False,False,3.7999999999999994,-18.00638,-4.1720934,1851.0,9.0,3,1.4881230776743295
-417,True,GAGTTGTGTGCCTTCT-1,GAGTTGTGTGCCTTCT-1,1.9677195338591296,15.0,-0.10566230511047225,-0.02343660398621728,1.8988804,18.920614,18,G1,1217.0519406348467,5,5,3894.0,1214.0,8.988186954288649,40.67796610169491,0.7234115696821902,False,False,9.3,-4.1586204,15.429235,763.0,9.0,5,1.0823755123601224
-418,True,GAGTTGTGTTCCACGG-1,GAGTTGTGTTCCACGG-1,1.9132403727830813,15.0,-0.10102529482374442,-0.004359421835149991,7.302568,6.375039,27,G1,1369.2804308831692,8,10,4995.0,1329.0,8.308308308308309,46.106106106106104,0.610111596381938,False,True,4.2,13.669567,8.868969,1196.0,9.0,8,0.9890392966214314
-419,True,GAGTTGTGTTCGAAGG-1,GAGTTGTGTTCGAAGG-1,2.0386602431705247,15.0,-0.05961259423863303,-0.023619161781378774,-9.844534,0.7698294,14,G1,1511.0530198812485,4,3,5054.0,2104.0,8.686189157103284,10.645033636723387,0.9800896687828496,False,False,4.1,-16.90356,-0.52645314,1937.0,9.0,4,1.4542779131353019
-420,True,GAGTTGTTCCGCTTAC-1,GAGTTGTTCCGCTTAC-1,1.915573832293061,15.0,-0.12527298225866573,-0.015351458169805448,1.7728759,5.2887044,5,G1,1709.9839492291212,1,1,6764.0,1604.0,8.32347723240686,48.97989355410999,0.6118866943028299,False,False,5.0,9.357945,10.472148,766.0,9.0,1,1.0031359453897972
-421,True,GCGTTTCAGATACCAA-1,GCGTTTCAGATACCAA-1,1.9935422477078797,15.0,-0.07048484157529494,-0.030364865849914845,-5.154694,-1.1153615,20,G1,1318.4492756575346,4,4,10262.0,3047.0,9.033326836873904,20.82440070161762,0.9611667235667967,False,False,3.3999999999999995,-16.523664,-0.1643319,1572.0,9.0,4,1.2626360339929414
-422,True,GCGTTTCAGCTCACTA-1,GCGTTTCAGCTCACTA-1,1.9451207673588513,15.0,-0.07359374538849026,-0.02143062591452305,-9.243341,-8.483466,10,G1,1059.0613911151886,3,7,9395.0,2989.0,7.993613624268228,13.507184672698244,0.9678401698107225,False,False,4.4,-12.7232065,-9.0594015,1899.0,9.0,3,1.0544564466971402
-423,True,GCGTTTCAGCTCTGTA-1,GCGTTTCAGCTCTGTA-1,1.8885949606387378,15.0,-0.09092886999863745,-0.024147839681820264,10.927044,-8.351941,11,G1,1233.2414600104094,7,6,4884.0,1664.0,7.186732186732187,28.74692874692875,0.1349628976429639,False,False,6.700000000000001,13.1919985,-7.434416,1179.0,9.0,7,1.5330612562039885
-424,True,GCGTTTCAGCTTGTTG-1,GCGTTTCAGCTTGTTG-1,2.0242732011430613,15.0,-0.07358062730616917,-0.030891240220429692,-9.330983,0.078763604,7,G1,1574.6309480667114,3,3,10590.0,3104.0,8.90462700661001,15.55240793201133,0.9796309612741497,False,False,3.8999999999999995,-16.248022,-1.3389395,1253.0,9.0,3,1.0793149764938728
-425,True,GCGTTTCAGGAACTCG-1,GCGTTTCAGGAACTCG-1,2.0017909000755822,15.0,-0.07444851807912907,-0.015773218114720135,-2.5234683,-4.344702,14,G1,1137.6243809461594,4,2,13348.0,3690.0,11.342523224453101,21.988312855858556,0.9206999030063117,False,False,1.6,-14.203999,-2.2558563,1924.0,9.0,4,1.0733894727636921
-426,True,GCGTTTCAGGATCACG-1,GCGTTTCAGGATCACG-1,1.967709755304794,15.0,-0.0919551651354266,-0.023417228755453354,-6.0315757,-9.812909,3,G1,1010.7253363877535,2,2,11666.0,3417.0,11.100634321961255,16.87810732041831,0.9511336335320776,False,False,3.0000000000000004,-15.330246,-9.283644,3214.0,9.0,2,1.0982061776526664
-427,True,GCGTTTCAGGTACAGC-1,GCGTTTCAGGTACAGC-1,1.8754535604837101,15.0,-0.09080379540837237,-0.003860186503558627,7.579875,9.429963,15,G1,1223.3936507850885,8,10,5641.0,1448.0,3.846835667434852,49.03385924481475,0.61855496032972,False,False,4.300000000000001,14.348682,10.16381,934.0,9.0,8,1.6166475945876415
-428,True,GCGTTTCAGGTTGCCC-1,GCGTTTCAGGTTGCCC-1,1.9499029409800286,15.0,-0.08448712077026158,-0.022439205193524662,-7.4234653,-6.185965,13,G1,1174.9671227037907,2,2,9597.0,3310.0,7.929561321246223,12.691466083150985,0.9604204617739387,False,False,3.4,-16.938093,-6.5025816,2978.0,9.0,2,0.8054144824310264
-429,True,GCGTTTCCACCGTGAC-1,GCGTTTCCACCGTGAC-1,2.081643073576501,15.0,-0.03609347463168062,-0.028820001080750043,-2.4651778,-5.6973424,19,G1,1286.2833405286074,2,2,2156.0,1020.0,27.040816326530614,4.962894248608534,0.9535516721691608,False,False,3.4999999999999996,-15.453573,-5.3314967,1249.0,9.0,2,0.8054144824310264
-430,True,GCGTTTCCAGCTGCCA-1,GCGTTTCCAGCTGCCA-1,1.9466911368327433,15.0,-0.12395255413645995,0.006007376929910137,-2.2944522,15.464206,2,S,1232.1421486735344,5,5,4302.0,1339.0,6.206415620641562,41.95722919572292,0.743193423594481,False,False,6.8999999999999995,-0.01046834,13.6125765,1294.0,9.0,5,0.8054144824310264
-431,True,GCGTTTCCATGGCTAT-1,GCGTTTCCATGGCTAT-1,1.9287115479219177,15.0,-0.09740269308868213,-0.02314803178512972,-10.3707695,-8.06818,10,G1,1157.9712169617414,3,7,15175.0,3987.0,6.9851729818780885,18.082372322899506,0.9740795629935742,False,False,4.0,-12.121913,-8.435523,2697.0,9.0,3,1.174952204829321
-433,True,GCGTTTCGTAGCGTAG-1,GCGTTTCGTAGCGTAG-1,1.9520738389292305,15.0,-0.09680622860382827,-0.0195520644068004,-6.0812874,-8.873951,25,G1,1082.416327252984,6,2,13225.0,3406.0,7.6521739130434785,17.83742911153119,0.9496429268315969,False,True,2.6,-14.478318,-8.829839,3362.0,9.0,6,1.0379934821247077
-434,True,GCGTTTCGTAGTGTGG-1,GCGTTTCGTAGTGTGG-1,1.8848002923439118,15.0,-0.09624085052171281,-0.00034100745024551876,10.313253,4.767892,26,G1,1036.6052235662937,9,9,5423.0,1564.0,4.33339479992624,45.4176654988014,0.6133969323817307,False,True,5.6,11.092388,4.607266,1174.0,9.0,9,1.115336184313886
-435,True,GCGTTTCGTATGCGTT-1,GCGTTTCGTATGCGTT-1,1.8908286834053516,15.0,-0.11207805738856084,-0.012879267882239948,1.4292498,3.4994023,12,G1,1632.7694101035595,1,1,7007.0,1801.0,5.794205794205794,45.46881689738833,0.611889558333002,False,False,4.199999999999999,10.062279,9.412356,1513.0,9.0,1,1.4750446060631406
-436,True,GCGTTTCGTCAAGCCC-1,GCGTTTCGTCAAGCCC-1,1.9793244788433022,15.0,-0.08567365803445987,-0.021554174450831583,-3.1899264,-1.9526285,19,G1,1395.3430116772652,2,2,11707.0,3289.0,10.344238489792431,15.905014094131715,0.9404946987332147,False,False,4.6,-13.329576,-2.34084,2294.0,9.0,2,1.6978102219916276
-437,True,GCGTTTCGTCGAATGG-1,GCGTTTCGTCGAATGG-1,1.9783355780166196,15.0,-0.08531803459645494,-0.030593654032190852,-8.064227,-7.8188386,10,G1,1308.6128604263067,3,2,9203.0,2953.0,11.887428012604586,12.593719439313267,0.9646991267972062,False,False,2.2,-14.2399435,-7.539477,2103.0,9.0,3,0.8054144824310264
-438,True,GCGTTTCGTCTAGGTT-1,GCGTTTCGTCTAGGTT-1,2.0039633883954897,15.0,-0.05874877798209569,-0.02572278559513492,-5.056116,-3.1967916,17,G1,1757.1379400640726,2,2,15829.0,4120.0,9.097226609387832,13.28574136079348,0.9650562207040994,False,False,4.7,-15.416959,-3.5250683,3852.0,9.0,2,1.802129890940226
-439,True,GCGTTTCGTGACAGGT-1,GCGTTTCGTGACAGGT-1,2.023416206185012,15.0,-0.06602568606362606,-0.029331534332560937,-5.504479,1.3208313,20,G1,1466.0097438246012,4,4,4965.0,2010.0,9.365558912386707,9.32527693856999,0.9522775733085661,False,False,5.7,-15.767346,0.952871,1718.0,9.0,4,1.290021396948959
-440,True,GCGTTTCGTTCGAAGG-1,GCGTTTCGTTCGAAGG-1,1.9752644734464588,15.0,-0.1363490248236485,-0.016630579698463037,1.5513514,15.252966,2,G1,1099.15285089612,5,5,4990.0,1392.0,9.39879759519038,42.28456913827655,0.7195369930497734,False,False,9.5,-2.099129,14.7687645,1387.0,9.0,5,1.4011142273600008
-441,True,GCGTTTCTCATTTGGG-1,GCGTTTCTCATTTGGG-1,1.886492977321364,15.0,-0.08472741832345376,0.005045431454759875,7.9674616,-2.7226562,22,S,1187.39734955132,6,8,3482.0,1397.0,8.271108558299828,26.3641585295807,0.46863271789780175,False,False,7.6000000000000005,14.983665,0.7728801,962.0,9.0,6,1.0169260766413897
-442,True,GCGTTTCTCCCAAGCG-1,GCGTTTCTCCCAAGCG-1,1.9015581455662387,15.0,-0.10609862603335811,-0.0028105496237303892,-2.0484564,14.427796,2,G1,1222.2123991549015,5,5,12699.0,3208.0,6.047720292936452,35.750846523348294,0.7326081582877823,False,False,6.999999999999998,0.71638346,14.672622,2130.0,9.0,5,1.076455644882097
-443,True,GCGTTTCTCCCGTTGT-1,GCGTTTCTCCCGTTGT-1,1.9506298066663137,15.0,-0.07136089107287204,-0.02116898394788601,-10.2033615,-6.3424973,10,G1,1139.548169568181,3,7,9854.0,3085.0,10.47290440430282,11.87335092348285,0.9777478723750416,False,False,3.1000000000000005,-12.08442,-7.3487835,2026.0,9.0,3,1.0611395816907838
-444,True,GCGTTTCTCTAGTCAG-1,GCGTTTCTCTAGTCAG-1,1.9964784082581721,15.0,-0.055700906815747374,-0.03759711496674763,-3.9054937,-2.2442648,19,G1,1543.690078958869,2,2,10767.0,3324.0,9.129748305006038,16.095476920219188,0.9532809659095467,False,False,2.6,-14.816743,-2.501882,1227.0,9.0,2,1.3051359729982455
-445,True,GCGTTTCTCTGCCCTA-1,GCGTTTCTCTGCCCTA-1,1.9656316325190635,15.0,-0.06069256120459873,-0.015190145644023312,-3.9933107,-3.5019958,13,G1,1099.6918728351593,2,2,9992.0,3250.0,7.7261809447558045,13.55084067253803,0.9573252455260762,False,False,2.9,-16.758545,-4.7086496,1533.0,9.0,2,1.1997971929725997
-446,True,GGTAACTAGATGCTTC-1,GGTAACTAGATGCTTC-1,1.8918720019597555,15.0,-0.09330599793876822,-0.0184775282050157,6.512094,5.2294383,12,G1,1636.7178554832935,1,9,5437.0,1542.0,4.027956593709766,45.42946477837042,0.611311013982174,False,False,3.8,10.914086,7.3337765,756.0,9.0,1,1.2396093642962331
-447,True,GGTAACTAGCCTCTGG-1,GGTAACTAGCCTCTGG-1,1.8887532887360579,15.0,-0.10111408230485952,-0.004433987388836122,2.8902223,1.7041141,12,G1,1200.6996416896582,1,1,4911.0,1342.0,4.846263490124211,46.97617593158216,0.6107310827294637,False,False,3.5,11.312934,8.372095,877.0,9.0,1,1.1652119799398735
-448,True,GGTAACTAGCTTAAGA-1,GGTAACTAGCTTAAGA-1,1.9113925680708361,15.0,-0.09511497961675365,-0.025200291393513645,-3.2451751,-14.225075,24,G1,1167.204173207283,10,7,12211.0,2947.0,8.754401768896896,23.70813201212022,0.9594379946591277,False,False,5.1,-7.8526325,-12.614308,2785.0,9.0,10,0.9400161588187672
-449,True,GGTAACTAGGGCGAAG-1,GGTAACTAGGGCGAAG-1,1.9699440082783706,15.0,-0.10878233422103392,-0.0013726262335715736,2.7056437,17.858658,18,G1,1063.50942184031,5,5,4547.0,1250.0,9.698702441170003,45.876402023312075,0.7218999254377241,False,False,5.700000000000001,-4.2892904,14.367057,649.0,9.0,5,1.2208709111950036
-450,True,GGTAACTAGTGCGTCC-1,GGTAACTAGTGCGTCC-1,1.9551212659862627,15.0,-0.09889392862430793,-0.01867137512225979,-9.528202,-9.777505,10,G1,1109.2821858525276,3,7,14708.0,3511.0,8.457982050584716,18.36415556159913,0.9687595297406814,False,False,5.2,-10.783125,-9.626292,4962.0,9.0,3,0.9702743922760952
-451,True,GGTAACTCACAAATCC-1,GGTAACTCACAAATCC-1,1.8759669373683747,15.0,-0.06802325170378722,-0.027163102640709903,5.75147,3.3351243,6,G1,1610.2430288791656,1,1,3627.0,1422.0,7.719878687620623,26.30272952853598,0.6032408273219441,True,True,4.700000000000001,12.920521,8.166678,1854.0,9.0,1,1.3799657090043111
-452,True,GGTAACTCAGACCTAT-1,GGTAACTCAGACCTAT-1,1.944626235055211,15.0,-0.09372083375991666,-0.023242899869066753,-7.742586,-9.708252,3,G1,1201.3187436759472,2,2,14031.0,3646.0,8.944480079823249,18.34509300833868,0.9535681952909236,False,False,4.1000000000000005,-13.738346,-9.0563,3166.0,9.0,2,1.0202116580598752
-453,True,GGTAACTCATCTCCCA-1,GGTAACTCATCTCCCA-1,2.033483917345862,15.0,-0.05485199446504899,-0.028545861247874088,-5.5598354,2.843624,14,G1,1399.1558936983347,4,4,4998.0,1969.0,13.505402160864346,10.504201680672269,0.9353594581749559,False,False,4.5,-15.4124,2.0603826,1233.0,9.0,4,1.0871745436712557
-454,True,GGTAACTGTCACGCTG-1,GGTAACTGTCACGCTG-1,1.9858006348937287,15.0,-0.08536124802156364,-0.02321540134089462,-7.772422,-1.0053236,9,G1,1360.5417627096176,2,2,10702.0,3188.0,9.091758549803775,16.65109325359746,0.9713656551610899,False,False,2.9,-15.644869,-1.7741287,1927.0,9.0,2,1.0390015412436735
-455,True,GGTAACTGTGACAGGT-1,GGTAACTGTGACAGGT-1,1.957304290672029,15.0,-0.11518228418070908,-0.007208030261447656,-1.4474683,14.716389,2,G1,1220.5084094703197,5,5,7530.0,1883.0,8.725099601593625,43.266932270916335,0.7371193816948719,False,False,7.799999999999999,0.42819965,13.944037,943.0,9.0,5,0.9538903929007907
-456,True,GGTAACTGTTACACAC-1,GGTAACTGTTACACAC-1,1.8857906178341237,15.0,-0.10188288408584617,-0.017670495192377116,6.6575985,4.4825907,12,G1,1251.1253600344062,1,9,9445.0,2382.0,5.897300158814187,43.23980942297512,0.6114860164741535,False,False,2.5000000000000004,10.531204,7.4353104,1691.0,9.0,1,0.9803736437095918
-457,True,GGTAACTGTTGGACCC-1,GGTAACTGTTGGACCC-1,1.9024341966254572,15.0,-0.11726949711445836,-0.013460392563490109,7.229759,10.083213,15,G1,1102.1117427647114,8,10,6993.0,1716.0,5.677105677105677,49.735449735449734,0.6190201929917489,False,False,4.4,14.629374,9.827219,1572.0,9.0,8,1.116711645410346
-458,True,GGTAACTTCCAATCCC-1,GGTAACTTCCAATCCC-1,1.99096553162365,15.0,-0.07060965099387748,-0.021836893934825553,-12.452853,-2.9943953,7,G1,1060.496095508337,3,3,9108.0,2988.0,9.541062801932368,13.790074659639878,0.999557244302952,False,False,4.7,-18.599064,-3.3331532,2449.0,9.0,3,1.134797514700434
-459,True,GGTAATCAGACGTCGA-1,GGTAATCAGACGTCGA-1,1.9408863585421237,15.0,-0.10409274709085438,-0.025884827615922645,-6.6967583,-8.433001,3,G1,1256.2316560298204,2,2,11513.0,3149.0,8.11256840093807,23.034830191956917,0.9499418506323816,False,False,5.0,-15.525059,-8.063196,2160.0,9.0,2,0.9503529382560777
-462,True,GGTAATCAGCTCTGTA-1,GGTAATCAGCTCTGTA-1,1.965317714071137,15.0,-0.08592764464281849,-0.025145371680025147,-8.6891775,-5.20088,13,G1,1294.8012250065804,2,3,12120.0,3504.0,8.556105610561056,15.825082508250825,0.9787106808347231,False,False,2.7000000000000006,-17.600815,-5.52944,3234.0,9.0,2,1.3323129786654517
-463,True,GGTAATCAGTCATCCA-1,GGTAATCAGTCATCCA-1,1.8952141853397164,15.0,-0.09300914755616117,-0.004979058477798833,1.5237654,4.588611,5,G1,1717.7054914981127,1,1,6509.0,1668.0,5.853433707174681,45.644492241511756,0.614626622547069,False,False,6.4,10.565291,11.511914,1246.0,9.0,1,0.8054144824310264
-464,True,GGTAATCAGTGCGTCC-1,GGTAATCAGTGCGTCC-1,1.9770971143775944,15.0,-0.0786451734747061,-0.023425388681303474,-10.664176,-1.3330064,7,G1,1573.1423325687647,3,3,7622.0,2759.0,8.121228024140645,13.185515612700078,0.9857140015934845,False,False,5.3,-16.834873,-2.2501268,3398.0,9.0,3,0.8054144824310264
-465,True,GGTAATCCAAGTAGTA-1,GGTAATCCAAGTAGTA-1,1.9326041014073199,15.0,-0.0775483355132815,-0.014841732257145607,-4.677714,-6.7852545,9,G1,1397.9825575202703,2,2,12974.0,3490.0,8.463080006166178,26.7303838446123,0.9449327314949466,False,False,2.4,-15.056797,-5.310502,2938.0,9.0,2,1.0123512893774396
-466,True,GGTAATCCACTTCAGA-1,GGTAATCCACTTCAGA-1,1.8828935913609999,15.0,-0.11616157770470616,-0.0029771863100975064,6.031983,3.5819085,6,G1,1624.7798773348331,1,1,7283.0,1870.0,3.7621859123987367,44.21254977344501,0.6069775310629637,False,False,4.5,12.706373,7.540646,743.0,9.0,1,1.7619741443073946
-467,True,GGTAATCGTATGAGCG-1,GGTAATCGTATGAGCG-1,1.9380398841722883,15.0,-0.09745939004839455,-0.0025259732473302335,0.72355986,6.9153223,5,G1,1656.4171461164951,1,1,4777.0,1248.0,9.776010048147374,47.079757169771824,0.6161805199712311,False,False,5.699999999999999,9.01557,11.998518,1159.0,9.0,1,1.2084786706331958
-468,True,GGTAATCGTCAAGCCC-1,GGTAATCGTCAAGCCC-1,1.9685775780255599,15.0,-0.08444370898767585,-0.015750375563740007,-2.285971,-0.46520284,14,G1,1134.3154958933592,4,4,10843.0,3073.0,10.98404500599465,23.1393525776999,0.8443196535411321,False,False,1.2,-12.340751,-1.3298858,4183.0,9.0,4,0.8054144824310264
-469,True,GGTAATCGTTCCGTTC-1,GGTAATCGTTCCGTTC-1,1.8857330027703727,15.0,-0.08848705846036635,0.008527493307551669,9.35844,-4.143062,23,S,1142.1055243462324,6,8,5942.0,1881.0,8.58296869740828,33.82699427802087,0.3696090746887515,False,False,7.1,14.362684,-2.0169702,1110.0,9.0,6,1.0050707685337341
-470,True,GGTAATCGTTCGGCTG-1,GGTAATCGTTCGGCTG-1,1.9011098495195558,15.0,-0.1102274807087149,-0.004595743901713335,7.9309607,8.98313,15,G1,1151.9602027982473,8,10,5050.0,1618.0,8.792079207920793,36.89108910891089,0.614236099290194,False,True,3.4000000000000004,14.814207,9.09316,1631.0,9.0,8,0.8054144824310264
-471,True,GGTAATCTCGTTCCTG-1,GGTAATCTCGTTCCTG-1,2.0074381757225765,15.0,-0.07160783159108604,-0.021307182482217968,-8.447385,-3.8579357,1,G1,1638.6017775088549,3,3,5444.0,2083.0,10.855988243938281,11.590742101396032,0.9794415867767325,False,True,3.0999999999999996,-16.25864,-4.5356255,1804.0,9.0,3,1.4347729404197185
-472,True,GGTAATCTCTGTGCGG-1,GGTAATCTCTGTGCGG-1,1.9822614779493652,15.0,-0.06902023507763573,-0.03355305170417033,-10.487968,-1.567702,7,G1,1599.094700023532,3,3,8970.0,2904.0,9.777034559643255,11.616499442586399,0.994213337549173,False,False,6.1,-17.235723,-2.6015809,1746.0,9.0,3,1.0861494330169923
-474,True,GTCAAGTAGATCGCCC-1,GTCAAGTAGATCGCCC-1,1.8860958970489563,15.0,-0.10069835224853681,-0.01753380282270008,10.865987,-9.377702,11,G1,1265.883071884513,7,6,3818.0,1450.0,8.486118386589837,29.36092194866422,0.11905398888001079,False,True,4.5,13.098978,-8.013861,1830.0,9.0,7,1.1589866241916746
-475,True,GTCAAGTAGATGCTTC-1,GTCAAGTAGATGCTTC-1,1.9905038578368874,15.0,-0.07783534140317036,-0.03123930066951593,-5.543269,-2.9932883,17,G1,1545.7968413829803,2,2,8002.0,2821.0,8.697825543614096,12.859285178705324,0.9658095944146652,False,True,3.6000000000000005,-16.012985,-3.5600214,2125.0,9.0,2,1.4691424885910218
-476,True,GTCAAGTAGCACCTGC-1,GTCAAGTAGCACCTGC-1,1.9624055661996864,15.0,-0.11022108785403112,-0.011743538550786092,1.856404,17.832495,18,G1,1080.1587940752506,5,5,3925.0,1210.0,8.815286624203821,41.65605095541401,0.7222010970037678,False,False,5.6,-4.436091,14.096253,954.0,9.0,5,1.2345429384636424
-477,True,GTCAAGTAGCAGCCCT-1,GTCAAGTAGCAGCCCT-1,1.9088212166387093,15.0,-0.11604828307173348,-0.010455415053863281,8.492172,9.163621,15,G1,1073.6456372737885,8,10,4780.0,1214.0,6.02510460251046,52.19665271966527,0.6185828505349793,False,False,5.4,14.44918,10.60913,1206.0,9.0,8,1.6163638299966483
-478,True,GTCAAGTAGCTCTGTA-1,GTCAAGTAGCTCTGTA-1,1.97917494630792,15.0,-0.07422637953570234,-0.023599039776816627,-7.651654,-8.341952,3,G1,1072.2969840168953,2,2,11247.0,3358.0,10.518360451676003,13.870365430781542,0.9593039965025305,False,True,2.8000000000000003,-14.225913,-8.402383,2737.0,9.0,2,0.9909435072959271
-479,True,GTCAAGTCAAACCATC-1,GTCAAGTCAAACCATC-1,2.04939635981785,15.0,-0.04240666293488034,-0.019039121246425105,-4.3301945,2.9644606,14,G1,1424.2926710993052,4,4,6855.0,2336.0,7.425237053245806,12.399708242159008,0.9252923667364973,False,False,5.4,-14.317543,1.6900156,1943.0,9.0,4,0.8054144824310264
-480,True,GTCAAGTCAAGAGAGA-1,GTCAAGTCAAGAGAGA-1,1.919275865534939,15.0,-0.11798687717667462,-0.004135494270163862,5.0957813,5.6288114,12,G1,1417.8654056414962,1,1,5735.0,1363.0,6.800348735832607,49.92153443766347,0.6122956471328568,False,False,3.8000000000000003,10.449599,7.40204,953.0,9.0,1,0.8054144824310264
-481,True,GTCAAGTCACACCGCA-1,GTCAAGTCACACCGCA-1,1.8850537697774505,15.0,-0.06765571117998781,-0.011254783574593906,5.6949844,8.3197365,27,G1,1145.0007433518767,8,1,7928.0,2207.0,7.341069626639758,40.03531786074672,0.6147984363891109,False,False,3.7,11.028669,10.825708,1945.0,9.0,8,1.2674650979792708
-482,True,GTCAAGTCACATATGC-1,GTCAAGTCACATATGC-1,1.8977190777700486,15.0,-0.1115759828288394,-0.009959368449102519,8.396104,3.1766622,8,G1,1441.383401542902,9,9,9688.0,2344.0,5.150701899256813,45.10734929810074,0.6046778460570154,False,False,4.3,12.33405,5.7299876,1363.0,9.0,9,1.3799404569406668
-483,True,GTCAAGTCACCGTGAC-1,GTCAAGTCACCGTGAC-1,1.964401215485108,15.0,-0.10967592202740256,0.013630768593817427,2.9664392,18.162455,18,S,1138.8892051428556,5,5,4355.0,1390.0,9.621125143513202,37.06084959816303,0.7228925540646978,False,False,7.899999999999999,-4.6785583,14.862218,1007.0,9.0,5,0.8054144824310264
-484,True,GTCAAGTGTAGCGTAG-1,GTCAAGTGTAGCGTAG-1,1.9139499800989541,15.0,-0.1257756163005199,-0.0102496757435221,3.0825953,3.6570723,12,G1,1668.5989705175161,1,1,5963.0,1451.0,8.049639443233271,48.51584772765386,0.6129318776376159,False,False,4.6,12.091324,10.181129,1527.0,10.0,1,1.3007900415296285
-486,True,GTCAAGTGTTAGGAGC-1,GTCAAGTGTTAGGAGC-1,1.8924152110949277,15.0,-0.12482950534273707,0.004481500966268292,4.391797,6.3245444,6,S,1582.544979646802,1,1,5220.0,1314.0,3.888888888888889,50.67049808429119,0.6133216662756666,False,False,4.0,11.274971,9.943995,1466.0,9.0,1,1.0001014312127572
-487,True,GTCAAGTGTTCCGTTC-1,GTCAAGTGTTCCGTTC-1,1.9069767154067787,15.0,-0.1052975754189846,-0.03128890701251808,-3.3508227,12.602101,2,G1,903.6031815707684,5,5,11537.0,3304.0,5.53870156886539,23.949033544248937,0.7087709872152613,False,False,3.8000000000000003,2.794634,13.691138,1068.0,9.0,5,1.134040899293891
-488,True,GTCAAGTTCAAGAAAC-1,GTCAAGTTCAAGAAAC-1,1.918895222138211,15.0,-0.08396854273225432,-0.006926246730530085,9.434865,3.3941958,8,G1,1236.7346387654543,9,9,7556.0,1952.0,8.33774483853891,42.53573319216517,0.6132792530800618,True,True,5.2,12.2731,5.1158533,902.0,9.0,9,0.8054144824310264
-489,True,GTCAAGTTCATGCCCT-1,GTCAAGTTCATGCCCT-1,1.897972502998224,15.0,-0.06448290394276891,-0.029420417769932334,3.7359707,2.811946,12,G1,1842.284537166357,1,1,2046.0,898.0,17.986314760508307,14.418377321603128,0.6122201402317655,False,False,6.1,12.005209,10.118835,954.0,9.0,1,1.398320174884873
-490,True,GTCAAGTTCTCCAATT-1,GTCAAGTTCTCCAATT-1,1.9500180174581772,15.0,-0.08085428073155704,-0.025439587224867298,-6.857682,-5.5567646,3,G1,1380.4451939761639,2,2,14326.0,3752.0,8.07622504537205,15.328772860533297,0.9662996002885605,False,False,3.1,-14.607156,-6.270057,2908.0,9.0,2,0.8054144824310264
-491,True,GTGAGCCAGATACCAA-1,GTGAGCCAGATACCAA-1,1.8864953418277608,15.0,-0.10688150428474202,-0.021090325731994403,7.9417014,0.3081697,22,G1,1145.4485174566507,6,8,5660.0,1843.0,6.07773851590106,36.64310954063604,0.5555179510166546,False,True,3.0,13.880096,3.928719,2654.0,9.0,6,0.9897589645476249
-493,True,GTGAGCCAGCCTCTGG-1,GTGAGCCAGCCTCTGG-1,2.0204990265130056,15.0,-0.0608487719228573,-0.027998946909922388,-3.11558,-2.6331613,19,G1,1558.7233407348394,2,2,5946.0,2241.0,15.943491422805247,10.99899091826438,0.9565500961035296,False,False,4.2,-13.559421,-3.2194192,2053.0,9.0,2,0.8054144824310264
-494,True,GTGAGCCAGCTTAAGA-1,GTGAGCCAGCTTAAGA-1,2.0626545595272177,15.0,-0.027137138960004872,-0.044183477704784604,-5.9080186,4.0868554,14,G1,1434.5245258659124,4,4,1670.0,918.0,12.694610778443113,11.676646706586826,0.9359884740289985,True,True,7.199999999999999,-14.530601,2.5754066,1570.0,9.0,4,1.6289097711356149
-495,True,GTGAGCCAGGCGTTAG-1,GTGAGCCAGGCGTTAG-1,1.9266850908724502,15.0,-0.09916891525734861,-0.020030571159376544,0.77162814,3.986483,5,G1,1261.3708607554436,1,1,5087.0,1515.0,12.305877727540791,39.76803617063102,0.6106950019701213,False,False,2.3000000000000003,9.002518,9.665691,2327.0,9.0,1,1.0888327821872623
-496,True,GTGAGCCAGTAATTGG-1,GTGAGCCAGTAATTGG-1,1.9786785267100906,15.0,-0.09231397536989219,-0.027625933913842642,-3.4729223,-2.5521114,19,G1,1296.7094856798649,2,2,7099.0,2368.0,7.719397098182843,12.818706860121145,0.9516894071636389,False,False,3.8,-13.104668,-3.0961804,1818.0,9.0,2,1.5037717087205094
-497,True,GTGAGCCAGTAGGAAG-1,GTGAGCCAGTAGGAAG-1,1.890445784398436,15.0,-0.11430747036121161,-0.011055639979457318,8.328998,2.7240727,8,G1,1287.1207217425108,9,9,7109.0,1882.0,4.2481361654241105,45.57603038402026,0.6044298993131737,False,False,2.9000000000000004,13.961166,6.5160375,1842.0,9.0,9,0.9808893821763591
-498,True,GTGAGCCCAAGAGTAT-1,GTGAGCCCAAGAGTAT-1,1.945627458555895,15.0,-0.11813746498517849,-0.011189775913954448,0.2500917,13.828904,2,G1,1061.5513131320477,5,5,5681.0,1544.0,8.537229361027988,44.023939447280405,0.7242434966343649,False,False,5.3,-0.6297741,15.750689,936.0,9.0,5,1.192661877989499
-499,True,GTGAGCCCACATGAAA-1,GTGAGCCCACATGAAA-1,1.9196008945223666,15.0,-0.10445829080042718,-0.014666280093300007,5.4838424,3.9967427,6,G1,1648.1316543370485,1,1,7080.0,1657.0,7.641242937853107,48.53107344632768,0.6109533780962834,False,False,4.0,12.212603,8.41571,730.0,9.0,1,1.426458888211635
-500,True,GTGAGCCGTATACCTG-1,GTGAGCCGTATACCTG-1,1.9644814176313266,15.0,-0.08252482170938098,-0.026433977582147362,-5.722054,-6.724826,13,G1,1288.336611866951,2,2,10555.0,3235.0,9.872098531501658,15.082899099952629,0.957426628017822,False,False,2.2,-16.362658,-6.6698437,5970.0,9.0,2,1.199813723524251
-501,True,GTGAGCCGTATAGGAT-1,GTGAGCCGTATAGGAT-1,2.1717708876313946,15.0,-0.05503991842504721,-0.02031723917744206,-8.440418,-12.124357,16,G1,684.5364937782288,2,2,669.0,302.0,12.406576980568012,3.7369207772795217,0.933438177065736,False,False,5.7,-15.858946,-10.6613865,736.0,9.0,2,0.9819739476362334
-502,True,GTGAGCCGTCTAGGTT-1,GTGAGCCGTCTAGGTT-1,1.9107601068615223,15.0,-0.10310638501222733,-0.01215203107053856,9.873085,4.8288255,26,G1,1018.3185731694102,9,9,4592.0,1255.0,8.362369337979095,45.55749128919861,0.6119398124069392,False,False,4.6,10.596211,4.9502316,889.0,9.0,9,1.239219353275973
-503,True,GTGAGCCGTCTTCATT-1,GTGAGCCGTCTTCATT-1,1.8942735911901045,15.0,-0.10409534957338576,-0.009337739650576762,6.560102,-0.15046738,22,G1,1021.5159673392773,6,8,6075.0,1757.0,9.82716049382716,38.91358024691358,0.5478307341893253,False,False,4.6,14.354403,3.5571644,874.0,9.0,6,1.262248996011977
-504,True,GTGAGCCGTTAAGACA-1,GTGAGCCGTTAAGACA-1,2.0044293905509303,15.0,-0.05317253293895029,-0.02665303465098171,-4.4044046,-0.82890046,20,G1,1124.474106490612,4,4,6810.0,2594.0,13.230543318649046,13.729809104258443,0.9613247874494422,False,False,5.6000000000000005,-16.811478,-0.8569928,2560.0,9.0,4,1.8367803788359192
-505,True,GTGAGCCGTTAAGTCC-1,GTGAGCCGTTAAGTCC-1,1.914767750731165,15.0,-0.10096652523816445,0.0033735242389664327,1.4475539,5.9705586,5,S,1792.288076132536,1,1,5934.0,1524.0,6.69025952140209,49.02258173238962,0.6132308121872807,False,False,6.8,9.276881,11.293825,1381.0,9.0,1,1.5813781746189925
-506,True,GTGAGCCGTTACACAC-1,GTGAGCCGTTACACAC-1,1.9605998441621446,15.0,-0.12868866280809435,-0.024523156710320453,-4.7320776,-7.8737206,3,G1,986.720964372158,2,2,15853.0,3454.0,7.613700876805652,29.268908093105406,0.9017510779184013,False,False,2.0,-15.885158,-8.332598,1453.0,9.0,2,0.8054144824310264
-507,True,GTGAGCCGTTGGAGGT-1,GTGAGCCGTTGGAGGT-1,1.8978907362289528,15.0,-0.10660163966647013,-0.005471927958434491,3.8709202,4.9065924,12,G1,1873.5644485205412,1,1,7697.0,1807.0,6.275172144991555,49.915551513576716,0.6111906024541025,False,False,5.2,10.541565,9.040163,1447.0,10.0,1,1.9461565517866377
-508,True,GTGAGCCGTTGGGATG-1,GTGAGCCGTTGGGATG-1,1.9733737451215703,15.0,-0.1105438870728522,-0.007930439457631415,2.3079946,18.245754,18,G1,1207.519254386425,5,5,4279.0,1338.0,14.839915868193502,35.68590792241178,0.7237064369412606,False,False,9.100000000000001,-4.127724,15.02495,1499.0,9.0,5,1.2445225742530512
-510,True,GTGAGCCTCCCAACTC-1,GTGAGCCTCCCAACTC-1,1.8874373387668735,15.0,-0.07495158141631046,-0.013568111921847679,9.652743,-10.709131,21,G1,1300.8082700371742,7,6,4324.0,1761.0,8.996299722479186,17.252543940795558,0.05792982617213052,False,False,8.2,12.332295,-10.453076,2493.0,9.0,7,1.479914115214732
-511,True,GTGAGCCTCCCAAGCG-1,GTGAGCCTCCCAAGCG-1,1.8870914520730193,15.0,-0.10625762921185479,0.01747026557468347,10.144069,4.105806,26,S,1006.3072529733181,9,9,8328.0,2135.0,6.015850144092219,41.63064361191162,0.6131626569486126,False,True,4.2,11.442899,4.7176294,1687.0,9.0,9,0.9716577852028643
-512,True,GTGAGCCTCCTGCTAC-1,GTGAGCCTCCTGCTAC-1,2.027946381367446,15.0,-0.06112059697144727,-0.028189268907615306,-9.917135,-0.9529053,7,G1,1653.1824572235346,3,3,9883.0,3035.0,12.182535667307498,9.410098148335525,0.9893603513057588,False,False,5.9,-16.203405,-1.6806772,2665.0,9.0,3,2.1011954019149104
-513,True,GTGAGCCTCGTCGGGT-1,GTGAGCCTCGTCGGGT-1,1.8787686266259958,15.0,-0.10079029489393893,-0.02220596211881282,8.413511,-0.5823542,23,G1,988.9731565564871,6,8,4699.0,1569.0,9.065758672057884,35.11385401149181,0.5340997388898594,False,False,3.8000000000000003,14.351809,2.8623846,1410.0,9.0,6,0.8054144824310264
-514,True,GTGAGCCTCTGAATCG-1,GTGAGCCTCTGAATCG-1,1.9389098333995145,15.0,-0.11087559964963423,-0.007738641329086898,0.11604611,12.331433,2,G1,844.5019671469927,5,5,5323.0,1566.0,6.443734736051099,41.72459139582942,0.7259674662331613,False,False,6.500000000000001,-0.070908725,16.66124,1134.0,9.0,5,1.069266113299076
-515,True,GTGGAGAAGAGAGCGG-1,GTGGAGAAGAGAGCGG-1,1.9051926906015735,15.0,-0.08338686161820022,-0.020147266981143615,9.13239,-4.3947196,23,G1,1164.9217701405287,6,8,4119.0,1441.0,10.827870842437484,31.731002670551106,0.3810027725046905,False,False,6.6000000000000005,14.4549465,-1.6591017,1395.0,9.0,6,1.1817649573553104
-516,True,GTGGAGAAGATCACCT-1,GTGGAGAAGATCACCT-1,1.9487917914529382,15.0,-0.07178846788883948,-0.014996942887340471,-6.4327035,-8.791237,3,G1,1268.0536492466927,2,2,12254.0,3594.0,8.413579239432023,16.802676677003426,0.9504671554960653,False,False,5.6,-14.978763,-8.331528,1954.0,9.0,2,1.4251293084786487
-517,True,GTGGAGAAGCACCTGC-1,GTGGAGAAGCACCTGC-1,1.8955718492896854,15.0,-0.1138256320830245,0.004793552970105145,5.8336267,5.072006,12,S,1514.9389950037003,1,1,7735.0,1769.0,4.201680672268908,51.58371040723982,0.6110450730456021,False,False,3.0999999999999996,11.265764,8.122241,1162.0,9.0,1,1.123968205432449
-518,True,GTGGAGAAGCATTGAA-1,GTGGAGAAGCATTGAA-1,1.9748978528624122,15.0,-0.06163184071320128,-0.02209429655247542,-5.0642986,-0.93358815,20,G1,954.7243008762598,4,4,6018.0,2299.0,10.784313725490197,9.92023928215354,0.9620537058419285,False,False,5.599999999999999,-17.091925,-1.2080588,1781.0,9.0,4,1.0389605241795559
-519,True,GTGGAGAAGGCTCAAG-1,GTGGAGAAGGCTCAAG-1,2.0280536439457797,15.0,-0.026460580675274025,-0.035783902893787724,0.24478208,8.634653,5,G1,1302.0978266447783,1,1,1031.0,492.0,28.51600387972842,4.364694471387003,0.6253365333800049,False,False,5.1000000000000005,9.403603,13.532088,1474.0,9.0,1,0.8054144824310264
-520,True,GTGGAGAAGGTTACAA-1,GTGGAGAAGGTTACAA-1,1.9724585499627925,15.0,-0.06316860220907766,-0.023095004999359615,-7.701872,-6.6284986,3,G1,1555.6911355406046,2,2,10988.0,3286.0,10.029122679286495,13.423734983618493,0.9672610567941193,False,False,3.7,-14.701832,-6.4220695,3242.0,9.0,2,1.263166790240676
-521,True,GTGGAGAAGTAGGAAG-1,GTGGAGAAGTAGGAAG-1,1.9184892948150047,15.0,-0.08483097292212322,-0.011536865645160429,5.630493,-0.20171976,22,G1,1150.6198026835918,6,8,4917.0,1593.0,12.34492576774456,34.49257677445597,0.5631527153179208,False,False,3.4,14.445616,4.261913,1246.0,9.0,6,1.1458554135230128
-522,True,GTGGAGAAGTGGCCTC-1,GTGGAGAAGTGGCCTC-1,1.9860356116974116,15.0,-0.116305814143599,-0.015209171830115546,-2.700012,15.363356,2,G1,1271.5897273421288,5,5,9253.0,2206.0,11.855614395331244,40.24640657084189,0.7337904098723382,False,False,7.8999999999999995,0.8834708,14.180544,1160.0,9.0,5,1.4035771029350492
-523,True,GTGGAGACAAGTCGTT-1,GTGGAGACAAGTCGTT-1,1.8805150405594686,15.0,-0.09878444559674233,-0.010786905637578444,7.5359607,4.9781437,8,G1,1399.1317790672183,9,9,5306.0,1440.0,4.730493780625706,48.17188088955899,0.6083959190058399,False,False,2.9,12.684891,6.5576515,1398.0,9.0,9,0.9967247498089045
-524,True,GTGGAGACACATATGC-1,GTGGAGACACATATGC-1,2.0066052882125085,15.0,-0.053609957457820705,-0.014353807780450457,-7.471446,-0.051164076,4,G1,1580.6797027885914,4,4,9157.0,2757.0,10.64759200611554,15.72567434749372,0.9677103027375437,False,False,4.6000000000000005,-14.6141,-0.7442251,2408.0,9.0,4,1.8819233464060614
-525,True,GTGGAGACATCTCCCA-1,GTGGAGACATCTCCCA-1,1.9852881352167047,15.0,-0.0736353704445755,-0.02417197623207743,-11.48046,-3.5127802,7,G1,1324.2157233208418,3,3,7268.0,2655.0,9.314804623004953,9.906439185470555,0.9946185058522786,False,False,3.6,-17.974112,-4.031796,2928.0,9.0,3,1.00956676208859
-526,True,GTGGAGACATGAATAG-1,GTGGAGACATGAATAG-1,1.997862569371393,15.0,-0.06730090589835278,-0.029342615380665415,-8.288498,0.0060452307,4,G1,1540.8689799308777,4,4,7941.0,2596.0,8.563153255257523,16.97519204130462,0.9733637503200613,False,False,5.6000000000000005,-14.546636,-1.1862334,2720.0,9.0,4,1.734008289080401
-527,True,GTGGAGAGTAGAATGT-1,GTGGAGAGTAGAATGT-1,1.9560694131007152,15.0,-0.06594978881131797,-0.03122968854355757,-8.0574045,-6.5586324,3,G1,1199.490616068244,2,2,8983.0,2981.0,6.801736613603473,11.699877546476678,0.9715443352494672,False,False,3.8000000000000003,-16.753887,-6.2603874,3489.0,9.0,2,1.0220909052009652
-528,True,GTGGAGAGTAGTGTGG-1,GTGGAGAGTAGTGTGG-1,1.8999575605977002,15.0,-0.09220859135698137,-0.04515692029803567,4.327864,7.6955724,27,G1,1488.3171306401491,8,1,2881.0,1237.0,15.688996876084692,18.986463033668866,0.6155038681506834,False,False,2.8000000000000003,11.787191,11.041236,1021.0,9.0,8,1.1006157928494924
-529,True,GTGGAGAGTATGCGTT-1,GTGGAGAGTATGCGTT-1,1.8804644330004883,15.0,-0.0881052091734523,-0.00470920264190363,10.160702,-4.251098,23,G1,1130.5492237210274,6,8,4516.0,1635.0,8.990256864481843,32.108060230292296,0.35541232237950837,False,False,7.3999999999999995,14.166001,-2.366508,991.0,9.0,6,1.5280308569241976
-530,True,GTGGAGAGTGACAGGT-1,GTGGAGAGTGACAGGT-1,2.047773550500772,15.0,-0.08181792310538147,-0.02665656847094947,-10.7379265,-0.41238794,7,G1,1447.9372851401567,3,3,3604.0,1615.0,11.542730299667037,10.238623751387347,0.9903124113072069,False,False,4.3,-17.57844,-1.2917508,2446.0,9.0,3,1.012522421032248
-532,True,GTGGAGATCCACGTAA-1,GTGGAGATCCACGTAA-1,1.9954623538143408,15.0,-0.05989603663214681,-0.01465790959585751,-9.261654,-3.8173409,10,G1,1401.7336284518242,3,3,8639.0,2591.0,10.429447852760736,13.022340548674615,0.9812697916904074,False,False,3.8999999999999995,-13.35116,-3.589532,2241.0,9.0,3,1.066574391796639
-533,True,GTGGAGATCCGCACGA-1,GTGGAGATCCGCACGA-1,1.9450201123214939,15.0,-0.08565160450134508,-0.005132451772211586,3.1629772,5.51316,12,G1,1803.017857760191,1,1,4621.0,1370.0,17.29062973382385,35.77147803505735,0.6124952109151263,False,False,3.8,10.8035,10.618077,1056.0,9.0,1,1.1966467743803582
-534,True,GTGGAGATCCGGACGT-1,GTGGAGATCCGGACGT-1,1.9309597455804457,15.0,-0.11911343678722212,0.004484732554154325,-1.1289706,13.687366,2,S,1070.9903070628643,5,5,5876.0,1657.0,10.704560925799864,38.49557522123894,0.7300678558589978,False,False,4.7,0.36199176,15.556907,2092.0,9.0,5,0.8054144824310264
-535,True,GTGGAGATCTTAGCCC-1,GTGGAGATCTTAGCCC-1,1.8789197958739425,15.0,-0.09872703747451962,-0.007904698459155852,-0.31033337,5.143897,5,G1,1028.9701055586338,1,1,3556.0,1155.0,6.664791901012373,41.451068616422944,0.6108977404328603,False,False,3.3,7.9720926,10.035867,1128.0,9.0,1,0.8054144824310264
-536,True,GTGTGGCAGCCTCTTC-1,GTGTGGCAGCCTCTTC-1,1.904967098237018,15.0,-0.0840913642643487,-0.0028232116808229943,8.688718,2.853584,8,G1,1368.1487836837769,9,9,5797.0,1679.0,9.315163015352768,38.53717440055201,0.6060852461586613,False,False,4.3,12.684597,5.222799,978.0,9.0,9,1.421333812828499
-537,True,GTGTGGCAGCTCACTA-1,GTGTGGCAGCTCACTA-1,1.8886636395485266,15.0,-0.10406222915004641,-0.012192039896912315,7.634615,4.164181,26,G1,1448.109961770475,9,9,14201.0,2934.0,6.098162101260475,45.37708612069572,0.6098099341195842,False,False,3.8000000000000003,11.730461,6.532735,3125.0,9.0,9,0.8054144824310264
-538,True,GTGTGGCAGGAACTCG-1,GTGTGGCAGGAACTCG-1,2.01876031145061,15.0,-0.07467242880283059,-0.032897264808150965,-7.04266,-2.6689494,10,G1,1562.22102047503,3,3,7201.0,2483.0,14.94236911540064,13.23427301763644,0.9677294751702443,False,False,3.3,-14.328333,-3.0531821,2386.0,9.0,3,1.6698757083431846
-540,True,GTGTGGCAGTAATTGG-1,GTGTGGCAGTAATTGG-1,1.920737611051871,15.0,-0.10248168876213459,-0.00952584170337291,3.3508363,5.0465765,12,G1,1762.616995356977,1,1,4732.0,1333.0,7.016060862214708,48.24598478444632,0.6113735147627062,False,False,3.8,9.96698,8.411963,1374.0,9.0,1,1.1008992261743455
-541,True,GTGTGGCAGTGCGTCC-1,GTGTGGCAGTGCGTCC-1,1.967393129967805,15.0,-0.09907354421797154,0.006165758383099809,-2.3485239,13.811834,2,S,1191.0935328900814,5,5,6246.0,1677.0,11.959654178674352,41.80275376240794,0.732807797049985,False,False,7.4,1.5323181,14.820442,758.0,9.0,5,1.4166384903767528
-542,True,GTGTGGCAGTGTTGAA-1,GTGTGGCAGTGTTGAA-1,1.8873270360965113,15.0,-0.1403465605547684,-0.016909035125758524,0.7772391,8.044735,5,G1,1485.3836473077536,1,1,5482.0,1444.0,3.6483035388544325,50.41955490696826,0.6202882594422758,False,False,3.9999999999999996,9.927231,12.392451,1257.0,9.0,1,1.228392641989862
-543,True,GTGTGGCCAAGCCATT-1,GTGTGGCCAAGCCATT-1,2.009801123880846,15.0,-0.07602417251844534,-0.028042864733199467,-7.952174,-1.8129474,10,G1,1464.262759566307,3,3,8375.0,2852.0,13.444776119402984,13.217910447761193,0.9335728120019586,False,False,2.7,-14.813199,-2.0232449,3824.0,9.0,3,0.8054144824310264
-544,True,GTGTGGCCACAGAGAC-1,GTGTGGCCACAGAGAC-1,1.9008589957140591,15.0,-0.1313728864340083,-0.00778138987355392,4.4841113,3.3045228,12,G1,1717.1841997355223,1,1,9221.0,2100.0,5.7260600802515995,48.91009651881575,0.6114296400514194,False,False,3.3,11.194004,8.22794,1592.0,9.0,1,0.8054144824310264
-545,True,GTGTGGCCACATGAAA-1,GTGTGGCCACATGAAA-1,1.8764146877219223,15.0,-0.1111497549668062,-0.02424807204054658,2.8081772,7.0896916,5,G1,1286.135250851512,1,1,4084.0,1235.0,3.5749265426052887,45.372184133202744,0.6121397593791077,False,False,3.3000000000000003,8.564429,9.542634,1834.0,9.0,1,1.4387270348037065
-546,True,GTGTGGCCACTAAACC-1,GTGTGGCCACTAAACC-1,2.014304614125087,15.0,-0.05784216087117759,-0.026449333673292868,-6.3294683,-0.306418,14,G1,1557.8175098001957,4,4,7670.0,2595.0,9.98696219035202,13.142112125162972,0.958767441528049,False,False,2.4,-14.441423,-0.8024009,1655.0,9.0,4,0.9418883927432015
-547,True,GTGTGGCCACTTCAGA-1,GTGTGGCCACTTCAGA-1,1.8888507061887982,15.0,-0.11525514814857202,-0.0168870824062931,3.973424,3.9524112,12,G1,1394.3165978342295,1,1,6409.0,1673.0,5.6639101263847715,45.779372757060386,0.6266805072490624,True,True,3.5999999999999996,12.958389,9.747328,1894.0,9.0,1,1.2098189097854344
-548,True,GTGTGGCCAGTTTCGA-1,GTGTGGCCAGTTTCGA-1,2.008340912481727,15.0,-0.06582109352721566,-0.013183690830575565,-4.7509894,-1.4714333,20,G1,1551.6061856150627,4,4,10629.0,3096.0,9.031893875246967,15.043748235958228,0.9582177526234296,False,False,2.6,-15.549908,-1.4873714,1986.0,9.0,4,1.4525959895187106
-549,True,GTGTGGCCATGGCTAT-1,GTGTGGCCATGGCTAT-1,1.8840520130588523,15.0,-0.08726258060538192,-0.02046114660062633,9.28119,1.8148487,25,G1,986.5468703061342,6,9,5764.0,1920.0,8.102012491325468,28.140180430256766,0.5938800897565121,False,False,2.7,14.540578,5.4934134,927.0,9.0,6,1.0959107728037938
-550,True,GTGTGGCGTAACTTCG-1,GTGTGGCGTAACTTCG-1,2.0461532256235286,15.0,-0.08431246607638532,-0.028595160609485422,-7.873993,-4.2421603,10,G1,1299.1468257904053,3,3,2852.0,1238.0,20.021037868162693,10.659186535764375,0.9678727529836058,False,False,3.9000000000000004,-13.141097,-4.565519,1146.0,9.0,3,1.5600077572158508
-551,True,GTGTGGCGTATACCTG-1,GTGTGGCGTATACCTG-1,1.9230146274608166,15.0,-0.10571806265852737,0.0027762165843775513,-3.05646,11.806054,2,S,859.8493028283119,5,5,15737.0,3525.0,7.599923746584483,35.432420410497556,0.7153854706216032,False,False,1.8,3.4538765,12.742705,602.0,9.0,5,0.947274088417254
-552,True,GTGTGGCGTATAGGAT-1,GTGTGGCGTATAGGAT-1,2.014939091010808,15.0,-0.07562550251156273,-0.025462812452320023,-5.2605324,-0.526607,20,G1,1147.337303698063,4,4,6442.0,2339.0,13.054951878298665,13.272275690779262,0.9627560828322599,False,False,4.5,-16.246864,-0.9355467,1005.0,9.0,4,1.2852959077809885
-553,True,GTGTGGCGTCGAATGG-1,GTGTGGCGTCGAATGG-1,1.8882892463343817,15.0,-0.11626429934828691,-0.013652043520326246,1.4797767,6.8057137,5,G1,1721.6973308771849,1,1,5652.0,1452.0,4.476291578202407,50.424628450106155,0.6124197063239225,False,False,5.5,8.66766,11.105161,1065.0,9.0,1,1.3540715914304267
-554,True,GTGTGGCGTGTCTAAC-1,GTGTGGCGTGTCTAAC-1,1.9149036336515632,15.0,-0.08855117830416426,-0.006953402057463436,11.800872,-8.146275,11,G1,1137.555390626192,7,6,4601.0,1447.0,8.128667680938927,33.62312540752011,0.16766685263510545,False,False,6.200000000000001,12.753118,-6.58073,2078.0,9.0,7,0.985777900473622
-555,True,GTGTGGCGTTAGGAGC-1,GTGTGGCGTTAGGAGC-1,1.9105927124897024,15.0,-0.0994945914022317,-0.0010649388000283332,4.870056,4.032449,12,G1,1865.770537301898,1,1,6729.0,1698.0,6.4199732501114575,46.64883340763858,0.6108701562817306,False,False,5.1,11.386353,8.931483,1255.0,9.0,1,1.5686620003807206
-556,True,GTGTGGCTCGCCACTT-1,GTGTGGCTCGCCACTT-1,1.9262843668692167,15.0,-0.10491581209580271,-0.01341335167224365,6.450916,-1.3414365,22,G1,1022.8307742029428,6,8,6728.0,1783.0,11.682520808561236,37.96076099881094,0.5130630349402919,False,False,4.6,14.868992,2.4050767,1318.0,9.0,6,1.4666392467974971
-557,True,GTGTGGCTCGCCCAGA-1,GTGTGGCTCGCCCAGA-1,1.9004333706317915,15.0,-0.07589867480913785,-0.03944408800525256,8.41546,-12.476289,21,G1,1180.560720384121,7,6,3889.0,1599.0,11.828233479043456,17.125224993571614,0.018074712648886805,False,False,5.1000000000000005,11.4219475,-11.503132,1248.0,9.0,7,1.0258942873036565
-558,True,GTGTGGCTCGCGGTAC-1,GTGTGGCTCGCGGTAC-1,1.8933313254289574,15.0,-0.09862561041729072,-0.006638174968545541,8.335781,8.421789,15,G1,1032.4547358453274,8,10,5112.0,1381.0,6.4358372456964,47.04616588419405,0.6152289422858586,False,False,4.7,14.856622,11.298916,939.0,9.0,8,0.9558057699680218
-559,True,TAACACGAGACCATTC-1,TAACACGAGACCATTC-1,1.9033489921237152,15.0,-0.07455198671300772,-0.015561541737120597,-2.8152425,-13.030462,24,G1,1348.863736987114,10,7,15835.0,3894.0,6.86454057467635,17.221345121566152,0.9590593710458568,False,True,8.0,-6.920044,-11.725984,7706.0,10.0,10,1.4736247443789987
-560,True,TAACACGAGAGAGCGG-1,TAACACGAGAGAGCGG-1,1.9519468010960916,15.0,-0.11083144011262405,-0.030922217594768354,-6.886939,-1.6452265,20,G1,997.6873728185892,4,4,8712.0,2712.0,9.630394857667586,18.296602387511477,0.9669014308476728,False,True,3.9,-17.573704,-1.1268713,4199.0,9.0,4,1.0266563462822664
-561,True,TAACACGAGCACTCTA-1,TAACACGAGCACTCTA-1,1.8800594229410257,15.0,-0.10651561863833017,-0.01115114314866432,10.218443,-8.092923,11,G1,1207.382659688592,7,6,5170.0,1867.0,7.988394584139265,28.007736943907158,0.15314046665345904,False,False,5.999999999999999,13.461852,-7.689159,1354.0,9.0,7,1.3202140573738887
-562,True,TAACACGAGGCCTAGA-1,TAACACGAGGCCTAGA-1,1.9066860947618471,15.0,-0.08276749664614409,-0.02189224227870877,-1.940731,-13.490963,24,G1,1322.960656479001,10,7,13921.0,3520.0,8.081315997413979,17.778895194310753,0.9591470330235248,True,True,7.800000000000001,-6.241779,-12.402616,2666.0,9.0,10,0.9625685164062322
-564,True,TAACACGCAAGAGTAT-1,TAACACGCAAGAGTAT-1,1.9681571149831516,15.0,-0.1309402366484819,-0.02042872298829933,-1.5339167,13.3768425,2,G1,1069.8489926308393,5,5,9075.0,2272.0,9.234159779614325,38.53443526170799,0.7323363832717652,False,False,5.6,0.8702816,15.740313,1844.0,9.0,5,0.8054144824310264
-565,True,TAACACGCAATAGTAG-1,TAACACGCAATAGTAG-1,2.1425855976425883,15.0,-0.03134326993069457,-0.017434321641441382,-8.287345,-12.267354,16,G1,664.1168532222509,2,2,2673.0,761.0,11.485222596333708,2.05761316872428,0.9315222778269401,False,False,6.3,-15.337466,-11.092378,889.0,9.0,2,1.2895372537594163
-566,True,TAACACGCACTACACA-1,TAACACGCACTACACA-1,1.9202316397211454,15.0,-0.07498771809876684,-0.01825879797603801,-3.2467015,-12.256461,24,G1,1328.758084088564,10,7,16238.0,3638.0,7.89506096809952,21.203350166276635,0.9565544187994006,False,False,7.499999999999999,-7.074737,-11.0302725,3104.0,9.0,10,0.8054144824310264
-567,True,TAACACGCAGCTGCCA-1,TAACACGCAGCTGCCA-1,1.9600644415830377,15.0,-0.07321547751492852,-0.026157096311316054,-8.797946,-6.445177,10,G1,1274.639593333006,3,7,10315.0,3226.0,6.185167232186137,15.472612699951528,0.9719181562955227,False,False,3.1000000000000005,-12.837567,-6.6821237,3020.0,10.0,3,1.0586414463989005
-568,True,TAACACGCAGGTTCAT-1,TAACACGCAGGTTCAT-1,1.9014838718921419,15.0,-0.11782215530580094,-0.013612492429579435,0.5158483,3.83589,5,G1,1213.1601306647062,1,1,6239.0,1624.0,5.465619490302934,48.1968264144895,0.6099970246009617,False,False,2.6999999999999997,9.081205,8.622112,1430.0,9.0,1,0.8054144824310264
-569,True,TAACACGCATTATGCG-1,TAACACGCATTATGCG-1,1.8909426466267778,15.0,-0.11224465863155332,-0.00932828551071655,4.847835,1.59169,12,G1,1717.5075565129519,1,1,5873.0,1454.0,5.125148986889154,48.271752085816445,0.6069933511159287,False,False,5.3,12.610448,8.843966,1247.0,9.0,1,1.0898713919160035
-570,True,TAACACGGTCAAGCCC-1,TAACACGGTCAAGCCC-1,1.93274444151844,15.0,-0.0813911012586325,-0.010181242724866955,-4.103703,-14.271429,24,G1,1070.1442987173796,10,7,16706.0,3907.0,9.631270202322519,18.711840057464386,0.9598217932824393,False,False,3.5,-8.470448,-12.166748,4555.0,9.0,10,0.9358706416958789
-571,True,TAACACGGTGATTGGG-1,TAACACGGTGATTGGG-1,1.8990788546206345,15.0,-0.10789622622727958,-0.005769823017110282,9.915226,-8.788749,11,G1,1218.828633338213,7,6,5032.0,1620.0,7.452305246422894,35.59220985691574,0.11851404160697072,False,False,6.4,13.047045,-8.476495,1750.0,9.0,7,1.3469965854545554
-573,True,TAACACGGTTCGGCTG-1,TAACACGGTTCGGCTG-1,1.953453608866699,15.0,-0.09244408803350262,-0.02760640022106376,-11.70976,-3.587752,7,G1,996.6828173696995,3,3,1740.0,1106.0,2.586206896551724,8.218390804597702,0.9908771412137711,False,False,2.3,-18.703491,-2.6982446,5385.0,9.0,3,0.8054144824310264
-575,True,TAACACGTCAGGGATG-1,TAACACGTCAGGGATG-1,1.8740479403842105,15.0,-0.08824078591520453,-0.004802221630700598,7.6794186,-1.6484816,22,G1,1083.5563035309315,6,8,5940.0,1882.0,7.878787878787879,33.686868686868685,0.4853548748010588,False,False,5.1000000000000005,14.6604185,1.7181122,2001.0,9.0,6,1.2451788854830004
-576,True,TAACACGTCTGAATCG-1,TAACACGTCTGAATCG-1,1.9261569551853854,15.0,-0.11690768121041767,-0.016804736692211393,4.802303,2.3117132,12,G1,1822.1864954680204,1,1,4245.0,1161.0,10.789163722025913,42.94464075382803,0.6063914099100421,False,False,5.999999999999999,12.159952,8.022453,1398.0,9.0,1,1.4804076122355372
-577,True,TAGGAGGAGACGTCCC-1,TAGGAGGAGACGTCCC-1,1.9936098467109646,15.0,-0.08939515979007723,-0.026621780743491974,-11.501834,-6.559144,7,G1,1071.2746607661247,3,3,8895.0,2895.0,14.592467678471051,12.906127037661607,0.9749858074559021,False,False,2.6999999999999997,-17.884352,-6.1094074,3012.0,9.0,3,0.9661959181781773
-578,True,TAGGAGGAGCACCTGC-1,TAGGAGGAGCACCTGC-1,1.9632201369032927,15.0,-0.07868706192628899,-0.018419557634556504,-6.017016,-3.182112,7,G1,1567.4161136895418,3,2,15647.0,3804.0,7.400779702179332,24.113248546047167,0.968668689195569,False,False,3.3999999999999995,-13.986904,-3.6690094,2423.0,9.0,3,1.2558742398439324
-579,True,TAGGAGGAGGGCGAAG-1,TAGGAGGAGGGCGAAG-1,2.0329549554834716,15.0,-0.07078358665253386,-0.028795454987645196,-3.5725312,-5.272411,19,G1,1556.816163316369,2,2,7485.0,2516.0,14.762859051436205,15.537742150968604,0.95561288461176,False,True,3.7,-14.98978,-5.333673,2976.0,9.0,2,1.5613533066321073
-580,True,TAGGAGGAGTAGGAAG-1,TAGGAGGAGTAGGAAG-1,1.9445393533258113,15.0,-0.09305220036230434,-0.006706019165013826,2.2003303,2.010643,12,G1,1562.44801735878,1,1,5274.0,1634.0,14.125900644671976,33.10580204778157,0.6187772805208653,False,False,3.6999999999999997,11.38515,9.805123,1235.0,9.0,1,1.0069201340880614
-581,True,TAGGAGGAGTATAACG-1,TAGGAGGAGTATAACG-1,1.971208971634281,15.0,-0.0814286077177887,-0.016238080822293572,-6.0400753,-5.4865317,3,G1,1639.4385280311108,2,2,9108.0,2886.0,10.342555994729908,16.458058849363198,0.9610227431590741,False,False,4.2,-14.379485,-5.628166,2814.0,10.0,2,1.5492031698581938
-582,True,TAGGAGGAGTCATCCA-1,TAGGAGGAGTCATCCA-1,1.958917505708069,15.0,-0.08892664863720064,-0.020712163391854548,-10.122102,-7.210983,10,G1,1112.0317127853632,3,7,9770.0,2860.0,6.366427840327534,18.075742067553737,0.9765035255073727,False,False,4.0,-12.841159,-7.9210706,2648.0,9.0,3,1.5293952261509267
-583,True,TAGGAGGCAACCAATC-1,TAGGAGGCAACCAATC-1,1.9915012917264392,15.0,-0.08112713176649827,-0.025905524979639962,-8.806339,-5.3607373,10,G1,1223.1636317968369,3,3,7251.0,2469.0,9.695214453178872,15.694386981106055,0.9715453745571917,False,False,4.6000000000000005,-12.349511,-5.6623096,2531.0,9.0,3,1.280267109300333
-584,True,TAGGAGGCAACGCATT-1,TAGGAGGCAACGCATT-1,1.951175288911056,15.0,-0.09538257082041787,-0.022832975516875437,-10.283912,-8.639964,10,G1,1165.3721543550491,3,7,14046.0,3796.0,9.917414210451375,19.78499216858892,0.9718389029432524,False,False,3.7,-12.824407,-8.980664,3240.0,9.0,3,0.9545945411403118
-585,True,TAGGAGGCAAGTCGTT-1,TAGGAGGCAAGTCGTT-1,1.898853196175605,15.0,-0.10753085614695354,-0.004224803338899363,4.8282394,6.975618,12,G1,1354.5474991127849,1,1,5953.0,1696.0,4.4851335461112045,43.80984377624727,0.6126961983460109,False,False,5.1000000000000005,9.677133,7.8861136,1426.0,9.0,1,1.3709377976051438
-586,True,TAGGAGGCAATAGTAG-1,TAGGAGGCAATAGTAG-1,1.906703705366873,15.0,-0.11098770043313871,-0.02622781748995341,8.667967,-4.3739247,23,G1,1162.123053625226,6,8,4368.0,1453.0,11.057692307692308,34.40934065934066,0.38323220618685905,False,False,5.8,14.554498,-1.511154,844.0,9.0,6,1.0077115079891399
-587,True,TAGGAGGCACTACGGC-1,TAGGAGGCACTACGGC-1,1.8945286432571844,15.0,-0.11814864150735731,-0.010970622461545834,8.968799,1.4595244,22,G1,1235.2577348947525,6,9,7910.0,1916.0,6.8773704171934265,44.424778761061944,0.584689245662746,False,True,3.0,13.288368,4.29623,1516.0,9.0,6,1.2394139250314031
-589,True,TAGGAGGCATGGCTAT-1,TAGGAGGCATGGCTAT-1,1.9957486620315232,15.0,-0.07517682883846892,-0.024963075247329908,-4.832475,-2.2758834,17,G1,1531.5107227116823,2,2,9449.0,2955.0,10.91120753518891,13.38766006984866,0.9612734498808503,False,False,4.2,-16.416534,-3.5820715,2802.0,9.0,2,1.0928298243498888
-590,True,TAGGAGGGTAACGCGA-1,TAGGAGGGTAACGCGA-1,1.8943114395229252,15.0,-0.11367024448419799,-0.007417834544337951,8.9664135,4.5975585,26,G1,1318.4765589386225,9,9,5360.0,1483.0,7.201492537313433,44.14179104477612,0.6118644776664354,True,True,4.5,11.157627,6.006687,1589.0,9.0,9,0.9762847362179835
-591,True,TAGGAGGGTACTGAGG-1,TAGGAGGGTACTGAGG-1,1.9111135075418189,15.0,-0.0975639193894592,0.003496586344868515,7.855893,2.9994626,8,S,1228.6807295084,9,9,9229.0,2243.0,7.043016578177484,46.96066746126341,0.6070583258567167,True,True,2.3000000000000003,13.324407,6.3535943,1253.0,9.0,9,1.1295934572259603
-592,True,TAGGAGGGTATAGGAT-1,TAGGAGGGTATAGGAT-1,1.9047388680279134,15.0,-0.10867092448911385,-0.009207974178863342,8.05196,6.2594175,26,G1,1264.2727644443512,9,9,5051.0,1580.0,7.463868540882993,41.39774302118392,0.6121872918068108,False,False,2.8999999999999995,10.622946,6.5859957,899.0,9.0,9,0.8054144824310264
-593,True,TAGGAGGGTTAAGCAA-1,TAGGAGGGTTAAGCAA-1,2.0047318657593602,15.0,-0.09627396671465178,-0.014319834763420546,-0.5014523,12.927036,2,G1,913.4835133701563,5,5,7558.0,1997.0,15.347975654935167,38.54194231278116,0.7272525849375067,False,False,6.0,0.30898312,16.49458,804.0,9.0,5,0.9434651752928881
-594,True,TAGGAGGGTTAAGTCC-1,TAGGAGGGTTAAGTCC-1,1.9131667171362303,15.0,-0.07055254913272051,-0.017225989924727063,12.152752,-9.678508,11,G1,1167.1029457896948,7,6,4788.0,1717.0,10.609857978279031,26.900584795321638,0.11715038552180444,False,False,4.4,12.32061,-8.3084345,597.0,9.0,7,0.8054144824310264
-595,True,TAGGAGGGTTACACAC-1,TAGGAGGGTTACACAC-1,1.9477946450344197,15.0,-0.06521006433082752,-0.015573832814642547,3.1688619,2.8093562,12,G1,1536.741395458579,1,1,2645.0,1238.0,21.134215500945178,9.338374291115311,0.6149674078587354,False,False,4.800000000000001,12.145481,10.631927,1582.0,9.0,1,1.2584565298571562
-596,True,TAGGAGGGTTCCACGG-1,TAGGAGGGTTCCACGG-1,1.9007278658713505,15.0,-0.12436534128104718,-0.010672957361665816,3.4676182,4.428808,12,G1,1885.0434751063585,1,1,5315.0,1486.0,6.942615239887112,45.90780809031044,0.6109452680758359,False,False,5.1,10.94628,9.562157,1157.0,9.0,1,1.8075590031723183
-597,True,TAGGAGGTCAAAGGTA-1,TAGGAGGTCAAAGGTA-1,1.8939617317346136,15.0,-0.12180518643418639,-0.0012523704505757322,3.2198503,5.6159115,12,G1,1813.1293862983584,1,1,6128.0,1544.0,4.29177545691906,49.05352480417755,0.61148884990741,False,False,4.6000000000000005,9.8759775,9.728944,1231.0,9.0,1,1.393284096196813
-598,True,TAGGAGGTCAAGAAAC-1,TAGGAGGTCAAGAAAC-1,1.9742953775730374,15.0,-0.07485548599031507,-0.018939021833864894,-8.661267,0.9183273,14,G1,1355.3689723163843,4,3,8683.0,2799.0,9.466774156397559,15.340320165841298,0.9689671405036082,False,False,2.2,-16.601643,0.55356663,3248.0,9.0,4,1.0072247322945997
-599,True,TAGGAGGTCACAAGAA-1,TAGGAGGTCACAAGAA-1,2.1201900482191594,15.0,-0.05359627452650709,-0.03421174294960702,-10.597267,0.3448813,14,G1,1317.2918828427792,4,3,2100.0,957.0,29.38095238095238,6.904761904761905,0.9841858844032881,False,False,4.0,-17.349266,-0.29371935,2502.0,9.0,4,0.8054144824310264
-600,True,TAGGAGGTCCCAAGCG-1,TAGGAGGTCCCAAGCG-1,1.9988233354442089,15.0,-0.08232238881495414,-0.013305361277619525,-10.023526,-0.89700824,7,G1,1636.024218633771,3,3,8857.0,2730.0,10.116292198261263,13.300214519589026,0.9905414574355751,False,False,5.8,-16.8863,-1.8074914,3159.0,9.0,3,1.8620749307639513
-601,True,TAGGAGGTCTGAATCG-1,TAGGAGGTCTGAATCG-1,1.9755390933012411,15.0,-0.05818782099568162,-0.025867726700074832,-7.6753497,-5.6118655,3,G1,1525.7555863261223,2,3,7317.0,2562.0,12.641793084597513,13.953806204728714,0.9687611809688031,False,False,3.9999999999999996,-13.435364,-5.7192535,3470.0,9.0,2,1.503104343376727
-602,True,TATTGCTAGATACCAA-1,TATTGCTAGATACCAA-1,1.9070866497110317,15.0,-0.07991389060340508,-0.004506051176124416,9.32593,-3.0909312,23,G1,1135.5415417701006,6,8,9234.0,2404.0,7.894736842105263,38.650638943036604,0.4181061881323877,False,False,5.500000000000001,13.976861,-0.9212981,1089.0,9.0,6,1.2251818304999142
-603,True,TATTGCTAGATCACCT-1,TATTGCTAGATCACCT-1,1.9266722344769016,15.0,-0.08562675880102187,-0.012331060568886415,-7.2983637,-10.608932,3,G1,951.5102065950632,2,2,15998.0,4165.0,6.5070633829228655,15.226903362920366,0.9463928509981301,False,False,3.8000000000000003,-14.682517,-10.087637,2349.0,9.0,2,0.8054144824310264
-604,True,TATTGCTAGATGCTTC-1,TATTGCTAGATGCTTC-1,1.9743303445289562,15.0,-0.09102187898879661,-0.022934394242392728,-9.331853,-2.2512615,7,G1,1617.8079160153866,3,3,8908.0,2801.0,9.508307139649753,19.21867983834755,0.9820632590015091,False,False,3.4,-14.601277,-2.2881668,2320.0,9.0,3,1.3319839568445446
-605,True,TATTGCTAGCCTCTGG-1,TATTGCTAGCCTCTGG-1,1.8800771881632905,15.0,-0.0904163885242908,0.01165367959318427,4.6101127,6.668374,6,S,1341.8668562993407,1,1,5558.0,1890.0,6.009355883411299,31.30622526088521,0.611862629929599,False,False,2.3000000000000003,8.805046,8.334858,951.0,9.0,1,0.8054144824310264
-606,True,TATTGCTAGGCGTTAG-1,TATTGCTAGGCGTTAG-1,1.9859935626688883,15.0,-0.06646771039174042,-0.018050337245951932,-8.655784,-9.622894,10,G1,1014.7960696518421,3,7,17281.0,4125.0,12.823331983102829,15.132226144320352,0.962557742606879,False,False,3.6,-13.089811,-9.804615,2224.0,9.0,3,0.9486621989407168
-607,True,TATTGCTAGGGAGGTG-1,TATTGCTAGGGAGGTG-1,1.9061415325160354,15.0,-0.11683510376426946,-0.015768572681362794,4.17502,4.838518,12,G1,1885.7940989285707,1,1,7019.0,1573.0,4.744265564895284,51.56005128935746,0.6107605971140642,False,False,6.1000000000000005,11.098937,9.626065,620.0,9.0,1,1.6478300539323947
-608,True,TATTGCTAGGGCGAAG-1,TATTGCTAGGGCGAAG-1,2.027804671298693,15.0,-0.05233977798114638,-0.020878774474114525,-2.7238657,-3.794747,19,G1,1479.1154086738825,2,2,6437.0,2274.0,16.047848376572937,15.224483455025632,0.955874405266003,False,False,3.5999999999999996,-15.67815,-4.382188,1251.0,9.0,2,0.8054144824310264
-610,True,TATTGCTCAAGAGAGA-1,TATTGCTCAAGAGAGA-1,1.891930605834647,15.0,-0.11433378879416412,-0.011763889330034735,3.0062764,5.314927,12,G1,1824.0052290707827,1,1,6681.0,1628.0,3.2031133063912587,49.453674599610835,0.6119541482747788,False,False,4.3,10.297705,9.8838005,1205.0,9.0,1,1.8122603952358871
-611,True,TATTGCTCAATCTCGA-1,TATTGCTCAATCTCGA-1,1.9150316969527776,15.0,-0.07658623405117282,-0.006658361360799661,1.161219,5.803264,5,G1,1776.2699012160301,1,1,3930.0,1266.0,8.854961832061068,35.64885496183206,0.6148259145101724,False,True,6.999999999999999,9.851053,11.085645,1190.0,9.0,1,2.0300141177107736
-612,True,TATTGCTCAATCTGCA-1,TATTGCTCAATCTGCA-1,1.9669135554056636,15.0,-0.11489039881107223,-0.025672567217626487,0.77447677,15.4919405,2,G1,1103.8704384416342,5,5,6031.0,1844.0,7.345382192007959,37.07511192173769,0.7199471630829142,False,False,9.0,-2.1658297,14.3880825,853.0,9.0,5,1.0104332441636088
-614,True,TATTGCTCACTACCGG-1,TATTGCTCACTACCGG-1,1.8718859420492193,15.0,-0.09403471657618005,-0.02402908506789862,8.236633,-3.712775,23,G1,1183.099345177412,6,8,5899.0,1942.0,8.255636548567553,28.818443804034583,0.42929088800269155,True,True,6.099999999999999,14.692297,-0.43960133,1117.0,9.0,6,0.9821697393258806
-615,True,TATTGCTCAGCTGCCA-1,TATTGCTCAGCTGCCA-1,1.8737555261088272,15.0,-0.0604862293968781,-0.052713425222367466,1.7309469,7.3114247,27,G1,1403.9662496596575,8,1,2090.0,1122.0,17.99043062200957,4.784688995215311,0.6143930943732607,False,False,2.9000000000000004,8.399804,10.903099,2308.0,9.0,8,0.8054144824310264
-616,True,TATTGCTGTAACTTCG-1,TATTGCTGTAACTTCG-1,1.8948126947674881,15.0,-0.08535820721714361,-0.011225621312497363,4.356797,8.601131,27,G1,1214.0728960633278,8,1,7302.0,2001.0,6.628321007943029,44.23445631333881,0.6163432031403224,False,False,2.4000000000000004,10.990289,10.99788,756.0,9.0,8,1.0948064196618672
-617,True,TATTGCTGTAGCGTAG-1,TATTGCTGTAGCGTAG-1,1.9797172225717925,15.0,-0.07859703572788469,-0.02484835319084938,-9.045951,-4.4229503,10,G1,1245.0829603374004,3,3,8300.0,2715.0,10.421686746987952,13.975903614457831,0.9742969554402228,False,False,3.5,-12.618501,-4.9787755,3585.0,9.0,3,1.2018442938375378
-618,True,TATTGCTTCAGGGATG-1,TATTGCTTCAGGGATG-1,1.9089545029234216,15.0,-0.10762696836472022,-0.00938552879731359,4.163219,5.40932,12,G1,1825.055550545454,1,1,5118.0,1301.0,4.962876123485737,48.57366158655725,0.6113091972989818,False,False,5.6,10.245847,9.419679,729.0,10.0,1,0.8054144824310264
-619,True,TATTGCTTCCCGTTGT-1,TATTGCTTCCCGTTGT-1,2.0015330038093033,15.0,-0.0751844801092407,-0.03339808022846287,-5.5073676,-2.3917763,17,G1,1339.216727644205,2,2,8415.0,2716.0,9.257278669043375,13.475935828877006,0.9637132842311372,False,False,3.6999999999999997,-16.223825,-3.291715,2194.0,9.0,2,1.353259659144308
-620,True,TATTGCTTCGAACGCC-1,TATTGCTTCGAACGCC-1,1.9174056017507883,15.0,-0.12154031483893259,-0.019023988110820424,0.13392748,6.919147,5,G1,1593.244992658496,1,1,7677.0,1853.0,7.19030871434154,43.376318874560376,0.6214142120706734,False,False,4.6,10.100385,12.098377,1136.0,9.0,1,0.8054144824310264
-621,True,TATTGCTTCGACGCGT-1,TATTGCTTCGACGCGT-1,1.9324273332814896,15.0,-0.08494708678375285,-0.019088442039509566,-5.406357,-12.82145,10,G1,1129.1717528998852,3,7,9889.0,2925.0,7.472949742137729,18.293052887046212,0.9623369145433637,False,False,3.3,-9.209503,-11.29541,750.0,9.0,3,1.0673286967617488
-622,True,TATTGCTTCTCACTCG-1,TATTGCTTCTCACTCG-1,1.8928693454525043,15.0,-0.07926553404055146,-0.021331143006628188,9.19427,-3.3381388,23,G1,1149.6492844969034,6,8,5802.0,2062.0,10.806618407445708,24.57773181661496,0.4018455621903199,False,False,5.800000000000001,14.405481,-1.367888,1221.0,9.0,6,0.8054144824310264
-623,True,TCACGCTAGACCATTC-1,TCACGCTAGACCATTC-1,1.9358764036019767,15.0,-0.08450114542784468,-0.0024136032772996314,-0.87585026,14.758426,2,G1,1154.8566451966763,5,5,6187.0,1753.0,7.919831905608534,37.72426054630677,0.7304825662210855,False,False,4.4,-0.17387588,14.949865,1230.0,9.0,5,1.0708122203630395
-624,True,TCACGCTAGCTTGTTG-1,TCACGCTAGCTTGTTG-1,1.9295983379944566,15.0,-0.06831844721252084,-0.023258220815318944,8.287484,-7.3376813,11,G1,977.9713559746742,7,6,8617.0,3021.0,9.156318904491123,16.21213879540443,0.2550873974521062,False,False,4.5,14.770875,-6.873608,1894.0,9.0,7,1.204322462390436
-625,True,TCACGCTAGGTTACAA-1,TCACGCTAGGTTACAA-1,1.917598370403247,15.0,-0.11697190679436241,-0.02246702580766679,8.767438,-2.447984,22,G1,1082.6714586913586,6,8,6696.0,1895.0,9.214456391875746,40.59139784946237,0.4526181719319594,False,False,4.199999999999999,14.087333,0.79129505,707.0,9.0,6,1.224782928867908
-627,True,TCACGCTAGTCTTGGT-1,TCACGCTAGTCTTGGT-1,1.8558379357477308,15.0,-0.0692642745552534,-0.012321931323616392,10.518392,-6.497881,23,G1,1139.1971760690212,6,8,5583.0,2112.0,6.591438294823572,22.246104245029553,0.21041616705454178,False,False,5.800000000000001,14.226497,-5.623344,1564.0,9.0,6,0.8054144824310264
-628,True,TCACGCTCAAGAGTAT-1,TCACGCTCAAGAGTAT-1,1.952266103973654,15.0,-0.09151796129347158,-0.019058812591293044,-6.535925,-7.4793057,13,G1,1186.745100170374,2,2,16141.0,4043.0,7.973483675113066,19.39780682733412,0.9497117703169925,False,False,3.4,-16.34276,-7.2142253,2635.0,9.0,2,1.0662710483703537
-629,True,TCACGCTCAAGCCATT-1,TCACGCTCAAGCCATT-1,1.9417782664019234,15.0,-0.10417431173726105,0.006501996768671518,-3.1189823,13.856475,2,S,1151.8440171182156,5,5,5923.0,1762.0,8.678034779672464,37.41347290224548,0.7348773420002048,False,False,7.300000000000001,1.8395907,14.770291,1230.0,9.0,5,0.8054144824310264
-630,True,TCACGCTCAATCTGCA-1,TCACGCTCAATCTGCA-1,1.8984707797511062,15.0,-0.1094967910917838,-0.018822807808201263,7.747655,3.5896783,8,G1,1443.2493682354689,9,9,9051.0,2136.0,4.364158656502044,48.558170367915146,0.6092311620882579,False,False,4.6000000000000005,12.745442,6.0176916,1453.0,9.0,9,1.3141759901229113
-631,True,TCACGCTCACGCGCAT-1,TCACGCTCACGCGCAT-1,1.8688515322456323,15.0,-0.05465622822248078,-0.0167632394739267,8.283907,1.8873179,8,G1,1200.660037189722,9,9,8158.0,2798.0,8.936013728855112,22.309389556263792,0.5865466855217216,False,False,3.8000000000000003,13.770477,4.6235995,912.0,9.0,9,0.8054144824310264
-632,True,TCACGCTCAGACCTAT-1,TCACGCTCAGACCTAT-1,1.962214653271303,15.0,-0.07175594726235997,-0.03103892953169236,-7.1814423,-3.7455363,10,G1,1551.9388055950403,3,3,9351.0,3144.0,8.352047909314512,11.014864720350765,0.9702417474092274,False,True,2.9,-14.4363365,-3.954416,2827.0,9.0,3,1.609387040090457
-633,True,TCACGCTGTAATGCTC-1,TCACGCTGTAATGCTC-1,1.8917791795721983,15.0,-0.10519531707732785,-0.011351542282766337,9.372717,0.6513048,25,G1,1017.8341751247644,6,9,14442.0,3030.0,6.141808613765407,44.17670682730924,0.5730458820512577,False,False,3.9999999999999996,13.579727,3.9031665,2490.0,9.0,6,0.8054144824310264
-634,True,TCACGCTGTAGAATGT-1,TCACGCTGTAGAATGT-1,1.8636722261231293,15.0,-0.08966961231343117,-0.023263776639935483,9.217827,-1.5559161,25,G1,1069.0836197584867,6,8,5320.0,1825.0,7.293233082706767,31.93609022556391,0.48043261311745494,False,True,3.4,13.87444,1.517037,654.0,9.0,6,0.9413669634110602
-635,True,TCACGCTGTCAAGCCC-1,TCACGCTGTCAAGCCC-1,1.9083080410775757,15.0,-0.10433090146984136,0.0005656873589959949,4.2985473,4.856627,12,S,1853.3862182945013,1,1,5394.0,1491.0,6.3589173155357805,43.65962180200223,0.6104745356767937,False,False,5.8,10.889554,8.2876005,2058.0,9.0,1,0.8054144824310264
-636,True,TCACGCTGTCACGCTG-1,TCACGCTGTCACGCTG-1,1.960140040102747,15.0,-0.08536591248924275,-0.005447677980111584,1.7747018,16.22458,2,G1,1027.7385633736849,5,5,5686.0,1848.0,10.288427717200141,35.262047133309885,0.7183090376603622,False,False,6.6000000000000005,-2.614674,14.514374,815.0,9.0,5,1.0057120305185743
-638,True,TCACGCTGTTCGGCTG-1,TCACGCTGTTCGGCTG-1,2.1258546835451124,15.0,-0.015836319610745643,-0.04032383138434816,-8.188996,2.2816377,14,G1,1291.507243514061,4,4,1252.0,657.0,18.929712460063897,3.2747603833865813,0.9368169472494425,False,False,3.7,-13.461947,0.78384954,1452.0,9.0,4,1.4186430005075945
-639,True,TCACGCTGTTTGATCG-1,TCACGCTGTTTGATCG-1,1.8436882688256773,15.0,-0.08463564517246266,-0.016615330478779243,8.283134,4.271799,26,G1,1379.3460582569242,9,9,3211.0,1327.0,4.1420118343195265,34.25724073497353,0.610683651748799,False,False,4.3999999999999995,11.175912,6.1369376,970.0,9.0,9,1.4960005175783853
-640,True,TCACGCTTCCACGTAA-1,TCACGCTTCCACGTAA-1,2.012815984319007,15.0,-0.037597914368494155,-0.03428053722499051,-0.31822318,8.333308,5,G1,1276.149196445942,1,1,2379.0,1063.0,26.944094157208912,7.608238755779739,0.6250809681324047,False,False,5.7,8.998982,13.171636,899.0,10.0,1,0.9996232605453769
-641,True,TCACGCTTCGAACGCC-1,TCACGCTTCGAACGCC-1,1.909177500977344,15.0,-0.07979266758936981,-0.02429935140142176,0.3332711,7.3696394,5,G1,1577.436096638441,1,1,4454.0,1501.0,6.668163448585541,37.49438706780422,0.619000727623201,False,False,5.2,9.313559,12.228039,1205.0,9.0,1,1.2283639736538412
-643,True,TCCAGAAAGACCAGAC-1,TCCAGAAAGACCAGAC-1,1.9671669679117965,15.0,-0.11353691683342047,-0.012261029454732683,-1.8289365,15.900776,2,G1,1238.1832092776895,5,5,6527.0,1792.0,8.043511567335683,40.32480465757622,0.7320123422731535,False,False,7.7,0.33985654,14.037793,1711.0,9.0,5,1.559218234163678
-644,True,TCCAGAAAGGGAGGTG-1,TCCAGAAAGGGAGGTG-1,1.9578764122202192,15.0,-0.07879975530602298,-0.01667196662465959,-5.8836956,-7.1240983,3,G1,1400.6363264769316,2,2,12365.0,3454.0,9.640113222806308,18.803073190456935,0.9539930412932003,False,False,4.0,-14.618614,-7.2080727,2729.0,9.0,2,1.5319548011191668
-645,True,TCCAGAAAGGTTGGAC-1,TCCAGAAAGGTTGGAC-1,1.938653331955835,15.0,-0.09216120237900936,-0.01414828275912917,2.6460013,8.381447,5,G1,1443.043089300394,1,1,6653.0,1593.0,10.43138433789268,46.309935367503385,0.6122619410711685,False,False,3.1,7.8777804,10.73781,1169.0,9.0,1,0.8054144824310264
-646,True,TCCAGAAAGTAGTCTC-1,TCCAGAAAGTAGTCTC-1,1.9916160105803282,15.0,-0.10376633590206912,-0.006064753843304488,2.3826606,18.708853,18,G1,1213.2228570729494,5,5,4955.0,1260.0,9.8284561049445,45.18668012108981,0.7230214252264942,False,False,8.6,-4.2176175,15.082658,914.0,9.0,5,0.9587819215788999
-647,True,TCCAGAAAGTGGCCTC-1,TCCAGAAAGTGGCCTC-1,1.954206815115506,15.0,-0.09373638509279127,-0.02735922612548038,-8.988244,-8.687489,10,G1,1217.1231657117605,3,7,14404.0,3767.0,7.4076645376284365,18.994723687864482,0.9672262047850572,False,False,3.3,-13.471094,-9.081588,2558.0,9.0,3,0.8054144824310264
-649,True,TCCAGAAGTATACCTG-1,TCCAGAAGTATACCTG-1,1.903186077016388,15.0,-0.097704007739599,-0.009026670163672576,8.680523,-10.458825,21,G1,1304.3006233423948,7,6,4881.0,1887.0,8.809670149559517,22.249539028887522,0.06566872208947795,False,False,7.5,12.618439,-10.119822,1036.0,9.0,7,1.4355230139629744
-650,True,TCCAGAAGTCACGCTG-1,TCCAGAAGTCACGCTG-1,1.9572352480394377,15.0,-0.06952807388472185,-0.022515713445253563,-6.2285995,-8.825859,3,G1,1223.0178946256638,2,2,10281.0,3206.0,9.678046882598968,13.578445676490613,0.9532483018940371,False,False,4.0,-15.585749,-8.358909,3435.0,9.0,2,0.8054144824310264
-651,True,TCCAGAAGTCACTCGG-1,TCCAGAAGTCACTCGG-1,1.8650512867665967,15.0,-0.07362575234719484,-0.016134839281838995,8.839259,-14.046198,21,G1,1097.0887193381786,7,6,4086.0,1839.0,10.15663240332844,14.464023494860498,0.00575559420795318,False,False,7.0,10.032454,-11.33612,1589.0,9.0,7,1.214596760873807
-652,True,TCCAGAAGTCTAGGTT-1,TCCAGAAGTCTAGGTT-1,2.018551011489666,15.0,-0.07512960332076651,-0.03440862582959197,-4.158024,-1.0393155,20,G1,1380.711725756526,4,4,7054.0,2481.0,12.673660334561951,12.999716472923165,0.9557956457188042,False,False,2.5,-14.918872,-0.76789075,2188.0,9.0,4,0.8054144824310264
-653,True,TCCAGAAGTTAAGTCC-1,TCCAGAAGTTAAGTCC-1,1.9910924971871835,15.0,-0.08917010411829576,-0.03159354519438886,-9.643506,-4.760633,13,G1,1217.815705358982,2,3,10937.0,3315.0,8.676968089969828,11.813111456523727,0.9828030493649047,False,False,3.3,-17.817225,-5.0195236,2219.0,9.0,2,1.0240089423977876
-654,True,TCCAGAATCATGCCCT-1,TCCAGAATCATGCCCT-1,1.8954514134437437,15.0,-0.0929413257637359,-0.0007940169568589686,7.6231637,2.8512356,8,G1,1364.1116490364075,9,9,7744.0,2151.0,7.541322314049586,37.93904958677686,0.6022683540136744,False,False,3.2999999999999994,12.904878,5.889982,1967.0,9.0,9,1.225133459151207
-656,True,TCCAGAATCGTAGAGG-1,TCCAGAATCGTAGAGG-1,1.962747883194215,15.0,-0.11401272039655382,-0.018218124723560457,-5.831869,-9.005712,3,G1,1136.7000368535519,2,2,17368.0,3771.0,9.269921695071396,27.17065868263473,0.9182308512756282,False,False,3.6,-15.260987,-8.353245,1571.0,9.0,2,1.072176509407569
-657,True,TCGAAGTAGCAATTAG-1,TCGAAGTAGCAATTAG-1,1.9142504941346463,15.0,-0.09306017224591881,-0.0011448702063032272,4.5495787,3.400083,6,G1,1542.6872509717941,1,1,5815.0,1698.0,8.684436801375753,38.194325021496134,0.6123457894354274,False,False,3.7,12.15024,8.964586,1077.0,9.0,1,0.8054144824310264
-658,True,TCGAAGTAGCCTCTGG-1,TCGAAGTAGCCTCTGG-1,1.9576220762778798,15.0,-0.11516109029266706,-0.00217381924561226,2.592434,19.485168,18,G1,1178.422138467431,5,5,8422.0,2142.0,7.967228686772738,41.771550700546186,0.7228105467532835,False,False,8.7,-4.423855,14.877431,1040.0,9.0,5,1.0082126749131124
-659,True,TCGAAGTAGGGAGGTG-1,TCGAAGTAGGGAGGTG-1,1.9095473483024348,15.0,-0.0661230813162079,-0.029031545509051324,12.019052,-8.318773,11,G1,1163.4477026462555,7,6,4615.0,1685.0,10.617551462621885,23.640303358613217,0.1699325442630335,False,False,7.2,13.137146,-6.6960993,1372.0,9.0,7,1.2307612572131277
-660,True,TCGAAGTCAAGAGTAT-1,TCGAAGTCAAGAGTAT-1,1.9783141262982273,15.0,-0.06533020095886305,-0.025355717379881616,-6.6145425,0.3183884,19,G1,1559.392406836152,2,4,9609.0,3060.0,10.56301384119055,13.050265376209802,0.9612092251506734,False,False,3.8999999999999995,-14.595691,-0.19346547,2444.0,9.0,2,1.3220812003669467
-661,True,TCGAAGTCAATCTCGA-1,TCGAAGTCAATCTCGA-1,2.015697292783301,15.0,-0.06056201550387597,-0.04719525350593312,9.20022,10.491923,15,G1,923.0555487126112,8,10,576.0,390.0,14.756944444444445,6.770833333333333,0.6266298216159808,False,False,4.5,15.438016,11.267436,879.0,9.0,8,0.8054144824310264
-662,True,TCGAAGTCACTACCGG-1,TCGAAGTCACTACCGG-1,2.000728791398067,15.0,-0.0724778671032713,-0.026515023578759577,-4.880422,-3.9350815,17,G1,1787.0381335318089,2,2,14003.0,3596.0,9.719345854459759,15.41812468756695,0.9632898205706526,False,False,5.2,-15.248407,-3.9588563,2795.0,9.0,2,1.8189474974340007
-663,True,TCGAAGTCATCTCCCA-1,TCGAAGTCATCTCCCA-1,1.9969715342896053,15.0,-0.06641693897611647,-0.019525290409677737,-5.168773,-1.9871955,20,G1,1194.5363852232695,4,4,12547.0,3583.0,11.74782816609548,12.4093408782976,0.9620218194993418,False,False,3.4999999999999996,-16.502924,-1.695164,4685.0,9.0,4,1.2828164428263784
-664,True,TCGAAGTCATTATGCG-1,TCGAAGTCATTATGCG-1,2.0017895851902914,15.0,-0.08718142454447704,-0.028609049996148786,-8.29914,-3.910493,10,G1,1203.6675989627838,3,3,2647.0,1185.0,19.304873441632036,10.011333585190782,0.9611119309724508,False,False,3.4,-12.756589,-3.9409451,2461.0,9.0,3,0.8054144824310264
-665,True,TCGAAGTGTCGTGCCA-1,TCGAAGTGTCGTGCCA-1,1.9002038975433235,15.0,-0.09260646131949266,-0.011750756394079144,9.132883,0.88479084,25,G1,1057.9851561188698,6,8,5697.0,1578.0,8.987186238371072,41.583289450588026,0.5724357493804056,False,False,3.7,14.52157,4.3227115,1887.0,9.0,6,0.8054144824310264
-666,True,TCGAAGTGTTAAGTCC-1,TCGAAGTGTTAAGTCC-1,1.9581467258610044,15.0,-0.11575102614331105,-0.016375943738647925,-2.6423528,14.797153,2,G1,1261.1751947253942,5,5,7408.0,1926.0,8.57181425485961,41.68466522678186,0.7318645753897091,False,False,7.7,1.2543212,14.202694,1134.0,9.0,5,1.1494006263450627
-668,True,TCGAAGTTCCATTGCC-1,TCGAAGTTCCATTGCC-1,1.9131149369327187,15.0,-0.1021042767540304,-0.0022509083619328893,7.99233,3.2500844,8,G1,1479.69571647048,9,9,9407.0,2231.0,7.345593706814075,43.371957053258214,0.6060463991473884,False,False,4.6,12.742836,6.085818,744.0,9.0,9,1.6225209498318787
-669,True,TCGAAGTTCCCAAGCG-1,TCGAAGTTCCCAAGCG-1,2.001869844770349,15.0,-0.06227077772769557,-0.013176373797655894,-3.2197552,-3.6598146,19,G1,1611.394123479724,2,2,11077.0,3103.0,10.318678342511511,16.737383768168275,0.9518225742479185,False,False,3.9999999999999996,-14.172669,-3.8595345,1749.0,9.0,2,1.5626794437614846
-670,True,TCGAAGTTCCTGCTAC-1,TCGAAGTTCCTGCTAC-1,2.0096767830436675,15.0,-0.061887638107150304,-0.03238751758761231,-5.2481937,0.2756698,4,G1,1526.6131178736687,4,4,5904.0,2342.0,16.073848238482384,10.060975609756097,0.9551937846575944,False,False,3.5999999999999996,-15.643944,0.25495446,1234.0,9.0,4,0.8054144824310264
-671,True,TCGAAGTTCTCACTCG-1,TCGAAGTTCTCACTCG-1,1.9790515183943522,15.0,-0.08357898465763129,-0.006456082864719697,-9.685635,-2.7006888,7,G1,1650.723553493619,3,3,11608.0,3236.0,6.874569262577532,14.19710544452102,0.993364325254691,False,False,5.6,-16.081768,-3.3927326,2230.0,9.0,3,1.8409411948210852
-672,True,TCGCTCAAGACGTCCC-1,TCGCTCAAGACGTCCC-1,1.9395417124089476,15.0,-0.09385538323093848,-0.015406723961333498,-4.695982,-11.585086,24,G1,1232.1851411163807,10,7,15523.0,3767.0,9.695290858725762,21.168588546028474,0.9619650459054542,False,False,4.400000000000001,-8.383047,-11.058127,2844.0,9.0,10,0.9962928677473688
-673,True,TCGCTCAAGATACCAA-1,TCGCTCAAGATACCAA-1,1.9485850268717886,15.0,-0.09436240659567915,-0.031114155875754168,-7.680683,-8.824037,3,G1,1238.53900629282,2,2,12370.0,3598.0,9.636216653193209,12.732417138237672,0.9579377055391868,False,False,3.9999999999999996,-14.338801,-8.451429,3087.0,9.0,2,1.3600913283917158
-674,True,TCGCTCAAGGATCACG-1,TCGCTCAAGGATCACG-1,1.8836556668506712,15.0,-0.10304514102279431,0.003789515169429228,4.6296215,6.688089,6,S,1551.5108725130558,1,1,5551.0,1649.0,5.5665645829580255,41.77625653035489,0.6124705495319507,False,False,3.6,10.415406,9.107108,1871.0,9.0,1,1.2197254860671147
-676,True,TCGCTCAAGTGTTGAA-1,TCGCTCAAGTGTTGAA-1,2.0199891448432687,15.0,-0.0818180911727505,-0.019644481840486373,-7.5972724,1.5781083,14,G1,1481.3613579571247,4,4,6399.0,1934.0,15.783716205657134,21.456477574621033,0.9524045432428312,False,False,4.1,-13.965091,-0.15573514,2268.0,9.0,4,1.5247417702332517
-677,True,TCGCTCACAATCTCGA-1,TCGCTCACAATCTCGA-1,1.9164413078043057,15.0,-0.10201732364444481,0.016263038282517912,6.478319,1.3585752,22,S,1310.566964611411,6,8,5623.0,1645.0,10.759381113284723,35.51484972434643,0.5826036892839831,False,True,3.3000000000000003,13.689241,6.2164187,1370.0,9.0,6,1.0197972111781748
-678,True,TCGCTCACACAAATCC-1,TCGCTCACACAAATCC-1,1.9852890712440843,15.0,-0.07646879043523469,-0.03245260567379723,-4.888874,-5.9145927,3,G1,1481.2204222232103,2,2,7180.0,2570.0,12.994428969359332,14.275766016713092,0.9579855244904801,False,False,3.8000000000000003,-14.941083,-5.9198976,2375.0,9.0,2,1.0158687321103217
-679,True,TCGCTCACACACCGCA-1,TCGCTCACACACCGCA-1,1.999204376925453,15.0,-0.058879037154677336,-0.011391860194289102,-12.018182,-3.5038552,7,G1,1007.3005505949259,3,3,6469.0,2435.0,11.284588035245015,10.61987942494976,1.0,False,False,5.0,-18.617264,-3.765828,1437.0,9.0,3,1.3173431684307069
-680,True,TCGCTCACACCCTCTA-1,TCGCTCACACCCTCTA-1,1.9462624274374065,15.0,-0.07382400620499448,-0.02416847470426486,-10.089146,-7.669301,10,G1,1105.5173516124487,3,7,13574.0,3631.0,6.954471784293502,19.640489170472964,0.9735823773914273,False,False,4.3,-12.637971,-8.288453,3433.0,9.0,3,1.6205254676928134
-681,True,TCGCTCACAGTTTCGA-1,TCGCTCACAGTTTCGA-1,1.8915648650614891,15.0,-0.09822328738044928,-0.016743998254108945,2.6616688,7.17605,5,G1,1477.4511660709977,1,1,5652.0,1535.0,5.201698513800425,46.77990092002831,0.6123430291435416,False,False,2.8000000000000003,8.828321,10.089671,970.0,9.0,1,1.1621394467763864
-683,True,TCGCTCAGTGCACAAG-1,TCGCTCAGTGCACAAG-1,2.006497646816203,15.0,-0.07879427002602445,-0.017901833013694555,-4.1339726,-4.0310006,17,G1,1740.5769593566656,2,2,8073.0,2700.0,15.384615384615385,9.537966059705191,0.9608805594016727,False,False,6.0,-14.290732,-4.621388,3458.0,9.0,2,0.8054144824310264
-684,True,TCGCTCAGTTAAGCAA-1,TCGCTCAGTTAAGCAA-1,1.9700584801713001,15.0,-0.07325021632501696,-0.026615847539000412,-7.657188,-2.0054142,9,G1,1325.4932126104832,2,2,12272.0,3472.0,11.3754889178618,21.789439374185136,0.971923813219599,False,False,2.8,-16.554491,-2.907848,2817.0,9.0,2,1.0305418786578064
-685,True,TCGCTCATCCTACAAG-1,TCGCTCATCCTACAAG-1,1.9785556257865555,15.0,-0.10125969936482485,-0.007976703808120569,1.6027129,15.568839,2,G1,1073.5921186804771,5,5,3947.0,1450.0,11.933113757284014,29.896123638206234,0.7192258740665514,False,False,9.0,-2.1932206,14.391827,923.0,9.0,5,1.1006156182254136
-686,True,TCGCTCATCCTTCACG-1,TCGCTCATCCTTCACG-1,1.973522896201295,15.0,-0.08369975095855868,-0.025903591671914604,-6.7747817,-4.5814643,13,G1,972.1545389294624,2,2,7737.0,2786.0,12.252811167119038,13.532376890267546,0.9587773208116617,False,False,2.7,-15.998331,-5.0962353,1726.0,9.0,2,1.0003902924206178
-687,True,TGCTCGTAGCACCTGC-1,TGCTCGTAGCACCTGC-1,1.8579265201413198,15.0,-0.10467112089784639,-0.035214631880570134,-2.7362394,12.5479765,2,G1,926.105884462595,5,5,13684.0,3697.0,3.7489038292896812,25.285004384682843,0.7028736497788481,False,False,3.8000000000000003,3.0671618,13.237379,1453.0,9.0,5,1.2201613949944972
-688,True,TGCTCGTAGCATTGAA-1,TGCTCGTAGCATTGAA-1,1.9046380056557495,15.0,-0.09523622334719847,-0.014615991451283568,8.05597,10.511884,15,G1,1107.310365691781,8,10,5643.0,1599.0,5.954279638490164,42.79638490164806,0.6297563560813222,False,False,6.3,15.06465,10.066391,1389.0,9.0,8,0.8054144824310264
-689,True,TGCTCGTAGCCTCTTC-1,TGCTCGTAGCCTCTTC-1,2.2012553001677158,15.0,-0.07729852691092999,-0.011064088733991647,-8.425517,-11.59289,16,G1,667.5909079015255,2,2,702.0,260.0,13.532763532763532,3.988603988603989,0.9361578595849888,False,True,5.5,-15.562806,-10.768108,502.0,9.0,2,0.9485778766191572
-690,True,TGCTCGTAGGCCTAGA-1,TGCTCGTAGGCCTAGA-1,1.8787996298189602,15.0,-0.0973596903829462,-0.008228954137414608,5.1466784,2.4985347,6,G1,1722.5439319610596,1,1,5180.0,1547.0,5.945945945945946,42.181467181467184,0.6028290183325881,False,False,4.2,12.601604,7.737241,775.0,9.0,1,1.5043685629660082
-692,True,TGCTCGTAGTATAACG-1,TGCTCGTAGTATAACG-1,1.9029590986195781,15.0,-0.10208298418587183,-0.009544096901998844,2.575842,5.2503543,12,G1,1693.3400313705206,1,1,6554.0,1710.0,7.155935306682942,45.69728410131218,0.6123075764130359,False,False,3.0999999999999996,11.253841,10.7593775,1036.0,9.0,1,0.8054144824310264
-693,True,TGCTCGTAGTGGCCTC-1,TGCTCGTAGTGGCCTC-1,1.9527954111443704,15.0,-0.0934065169378935,-0.023240545455601846,1.6277683,7.7308974,5,G1,1644.9857279360294,1,1,3342.0,998.0,15.320167564332735,36.65469778575703,0.6138171769359551,False,False,5.8999999999999995,8.81556,11.399123,1009.0,9.0,1,1.185433140001456
-694,True,TGCTCGTCAACGCATT-1,TGCTCGTCAACGCATT-1,1.9764199496084418,15.0,-0.10814717703308083,-0.018484488885105763,1.1048962,19.067942,18,G1,1236.2221630066633,5,5,5540.0,1727.0,10.667870036101084,36.35379061371841,0.7243678254301371,False,False,7.2,-4.5346007,15.811884,1484.0,9.0,5,0.8054144824310264
-696,True,TGCTCGTCACCGTGAC-1,TGCTCGTCACCGTGAC-1,2.017170651269734,15.0,-0.08169076698592634,-0.038972218980803656,-11.237457,-2.5500104,14,G1,1364.8111157864332,4,3,6763.0,2642.0,11.932574301345557,10.217359160136034,0.994136009278356,False,False,4.0,-17.435148,-2.7646334,2525.0,9.0,4,0.8054144824310264
-697,True,TGCTCGTCATGAATAG-1,TGCTCGTCATGAATAG-1,1.9623875537805024,15.0,-0.07235568731138896,-0.02553361979556653,-11.752244,-2.450853,7,G1,1144.3104189932346,3,3,10623.0,3377.0,10.11955191565471,12.830650475383601,0.998351773163012,False,True,4.0,-18.351603,-2.8916817,3126.0,9.0,3,1.1935534825687564
-698,True,TGCTCGTGTACTGAGG-1,TGCTCGTGTACTGAGG-1,1.9858896821608998,15.0,-0.07411883945372134,-0.009942708842368295,-5.8816123,-6.8266935,25,G1,849.8574360609055,6,2,7649.0,2699.0,14.537848084716957,15.07386586481893,0.9155101123985143,False,True,3.4000000000000004,-16.259775,-7.330807,2182.0,9.0,6,0.899061679436515
-699,True,TGCTCGTGTCAAGCCC-1,TGCTCGTGTCAAGCCC-1,1.8748626623162896,15.0,-0.09138556262179273,-0.014298765448484407,6.425994,-1.1642622,22,G1,1049.7662628293037,6,8,5225.0,1654.0,5.645933014354067,40.17224880382775,0.5205849077250941,False,False,4.199999999999999,14.694947,2.7314882,1632.0,9.0,6,0.9895195640099048
-700,True,TGCTCGTGTCTAGGTT-1,TGCTCGTGTCTAGGTT-1,2.054112133552775,15.0,-0.0640941049821096,-0.036757635508340034,-5.275005,3.516658,14,G1,1464.3138327896595,4,4,4582.0,1903.0,9.602793539938892,10.650371017023135,0.9385021061215706,False,False,5.699999999999999,-14.6442995,1.6346612,1597.0,9.0,4,0.8054144824310264
-701,True,TGCTCGTGTTCCACGG-1,TGCTCGTGTTCCACGG-1,2.023594136425409,15.0,-0.07564997548061698,-0.024741966661985977,-4.052011,0.07990292,14,G1,1440.4397929161787,4,4,11260.0,3313.0,8.943161634103019,17.841918294849023,0.9472986526985316,False,False,4.699999999999999,-14.706867,-0.23539524,1189.0,9.0,4,1.0506509094123777
-702,True,TGCTCGTGTTGGGATG-1,TGCTCGTGTTGGGATG-1,1.9134716533111513,15.0,-0.09999172554141952,-0.0024481513064159525,1.6725556,5.78431,5,G1,1779.306372821331,1,1,5722.0,1418.0,7.829430269136665,49.108703250611676,0.61331692087955,False,False,6.6,9.656051,10.708947,1647.0,9.0,1,1.4837678568491623
-703,True,TGCTCGTTCATGCCCT-1,TGCTCGTTCATGCCCT-1,1.881934595308163,15.0,-0.11252771860175467,-0.010632253407736582,5.493402,4.9906754,12,G1,1506.100800678134,1,1,6979.0,1763.0,4.585184123799971,45.794526436452216,0.6079674215041945,False,False,2.9,11.018494,7.689675,910.0,9.0,1,1.0613058845862255
-704,True,TGCTCGTTCCGAGAAG-1,TGCTCGTTCCGAGAAG-1,1.9029217099536027,15.0,-0.09267053623989505,-0.014833936924749738,11.21525,-6.7385354,11,G1,1160.5451955497265,7,6,6573.0,1938.0,9.736802069070439,33.91145595618439,0.22840552035639494,False,False,4.6000000000000005,13.62536,-5.5181084,1464.0,9.0,7,0.8054144824310264
-705,True,TGCTCGTTCGTCGGGT-1,TGCTCGTTCGTCGGGT-1,1.9369948068584733,15.0,-0.08932438612635764,-0.02553557796394498,8.273165,-7.599273,11,G1,992.153044089675,7,6,12315.0,3459.0,8.932196508323184,19.53714981729598,0.2539868940414648,False,False,4.2,14.435333,-7.403716,1448.0,9.0,7,1.2294035465817903
-706,True,TGCTCGTTCGTTTACT-1,TGCTCGTTCGTTTACT-1,1.9277159086446656,15.0,-0.07036862584310155,-0.0184053210417798,-3.7497492,-11.8405,24,G1,1286.8815755099058,10,7,11728.0,3299.0,8.995566166439291,17.573328785811732,0.9600926057890252,False,False,6.299999999999999,-7.61334,-11.685133,1441.0,9.0,10,1.1476447307389888
-707,True,TGGTACAAGACGTCGA-1,TGGTACAAGACGTCGA-1,2.004295999361444,15.0,-0.07562541684067492,-0.02299019299881922,-5.9322824,-2.6855078,9,G1,1719.1389792710543,2,2,9184.0,2770.0,11.770470383275262,19.52308362369338,0.964511877220525,False,False,3.9999999999999996,-15.622446,-2.7776437,2082.0,9.0,2,1.7048563686127152
-708,True,TGGTACAAGCACTCTA-1,TGGTACAAGCACTCTA-1,2.0643182157987736,15.0,-0.06740167147916143,-0.029723474391037633,-5.4121566,3.4130397,14,G1,1457.841810375452,4,4,5797.0,1973.0,12.937726410212179,14.352251164395376,0.9282918028979709,False,False,6.0,-14.124995,1.4777582,781.0,9.0,4,1.895003129573888
-709,True,TGGTACAAGCAGCCCT-1,TGGTACAAGCAGCCCT-1,1.9026842717308456,15.0,-0.10187769054590697,-0.001048259636526199,3.9137306,6.1722093,12,G1,1354.2896008491516,1,1,7233.0,1958.0,7.286050048389327,40.32904742154016,0.6119232935801731,False,False,2.5000000000000004,10.355409,9.417555,1499.0,9.0,1,0.9803736437095918
-710,True,TGGTACAAGGTCTACT-1,TGGTACAAGGTCTACT-1,1.8911326893078662,15.0,-0.07767227803005977,-0.028746873407067573,10.118851,-12.190447,21,G1,1217.8547803610563,7,6,3900.0,1627.0,10.23076923076923,17.46153846153846,0.028254068653785484,False,False,5.0,11.34842,-11.202848,1267.0,9.0,7,0.8054144824310264
-711,True,TGGTACACAAGTCGTT-1,TGGTACACAAGTCGTT-1,1.9103384377242905,15.0,-0.13589125098307445,-0.016662831306742443,3.7740633,3.0617135,12,G1,1586.2375886291265,1,1,5563.0,1304.0,4.2782671220564445,51.86050692072623,0.6135241267667881,False,False,4.7,12.616341,9.805595,1417.0,9.0,1,1.0211365684291407
-712,True,TGGTACACAATCAAGA-1,TGGTACACAATCAAGA-1,1.9062022761246855,15.0,-0.06721210176296277,-0.017700977226286645,9.301143,-10.953219,21,G1,1296.3735474646091,7,6,4192.0,1667.0,12.95324427480916,18.606870229007633,0.04739248647864181,False,False,6.6,11.804786,-10.518646,961.0,9.0,7,1.890189192948571
-713,True,TGGTACACATGGCTAT-1,TGGTACACATGGCTAT-1,1.9939949435103408,15.0,-0.06535287668070865,-0.016464587714196233,-5.233852,-1.9723355,9,G1,1663.3033603727818,2,2,15500.0,3830.0,10.019354838709678,16.36774193548387,0.9603734595332298,False,False,2.7,-14.778332,-2.5045917,4188.0,10.0,2,1.5302818959712985
-714,True,TGGTACACATGTTCGA-1,TGGTACACATGTTCGA-1,2.0816411151407244,15.0,-0.054978283577986696,0.0077873192932065086,10.477175,2.6624699,25,S,808.9151745140553,6,9,564.0,355.0,28.54609929078014,6.560283687943262,0.6328648247165117,False,True,2.0,11.687203,3.3552747,2381.0,9.0,6,0.8054144824310264
-715,True,TGGTACAGTAACTTCG-1,TGGTACAGTAACTTCG-1,1.944688244505543,15.0,-0.0718212767220713,-0.024312997577003663,-8.757403,-5.6888995,10,G1,1309.0606059283018,3,3,9521.0,2998.0,9.515807163113118,15.51307635752547,0.9688419358431407,False,False,3.8000000000000003,-12.672276,-6.2802315,2813.0,9.0,3,1.0999347589582273
-716,True,TGGTACAGTCGTGCCA-1,TGGTACAGTCGTGCCA-1,1.9743895223305992,15.0,-0.1180362110078446,-0.013022978013333922,1.1733916,15.332442,2,G1,1106.8139469772577,5,5,4505.0,1467.0,10.366259711431743,36.958934517203105,0.7195504465318351,False,False,9.200000000000001,-1.959967,14.826358,560.0,9.0,5,1.248793220324594
-717,True,TGGTACAGTTAAGTCC-1,TGGTACAGTTAAGTCC-1,2.0052163478038985,15.0,-0.08232316880676963,-0.029941410615868413,-6.121143,-0.07703727,19,G1,1549.5485635846853,2,4,6970.0,2489.0,16.499282639885223,6.829268292682927,0.9664943867705973,False,False,3.3,-14.506975,-1.1525476,2504.0,9.0,2,1.0796593453050356
-718,True,TGGTACATCACAAGAA-1,TGGTACATCACAAGAA-1,1.9087888615193525,15.0,-0.07991571616344173,-0.015331485282082164,4.5097537,3.3156636,12,G1,1812.7740164548159,1,1,5253.0,1568.0,9.308966304968589,38.14962878355226,0.6111591756128197,False,False,5.500000000000001,12.351362,9.53583,1436.0,9.0,1,1.7382680596526896
-719,True,TGGTACATCCTGCTAC-1,TGGTACATCCTGCTAC-1,1.8876826057782135,15.0,-0.10865942957313936,-0.015512719646262219,4.648127,4.322025,6,G1,1702.2992860525846,1,1,6193.0,1731.0,5.603100274503472,43.72678830938156,0.6119303235052337,False,False,2.9,11.735623,9.060129,1099.0,9.0,1,1.4065931698728518
-720,True,TGGTACATCCTTCACG-1,TGGTACATCCTTCACG-1,1.9974978974785114,15.0,-0.08004887991688274,-0.011995778393580799,-9.755837,-3.6560767,10,G1,1269.9728217571974,3,3,9373.0,2957.0,6.828123332977702,8.791208791208792,0.9850314569033785,False,False,3.8000000000000003,-13.145911,-5.1307683,2334.0,9.0,3,1.3905506255957503
-721,True,TGGTACATCGCGGTAC-1,TGGTACATCGCGGTAC-1,1.9861702677708417,15.0,-0.08086381434844428,-0.0170025602232003,-3.921139,-3.8080227,19,G1,1563.0256024301052,2,2,7706.0,2315.0,9.680768232546068,15.792888658188424,0.9503294105291532,False,False,4.0,-13.875796,-4.3224316,784.0,9.0,2,1.2764001638668243
-722,True,TGGTACATCGTAGAGG-1,TGGTACATCGTAGAGG-1,2.0379557063454006,15.0,-0.044605027775162405,-0.021193761101783587,-5.155376,-5.1119237,3,G1,1101.6976715177298,2,2,3800.0,1660.0,21.86842105263158,11.078947368421053,0.9498818601705068,False,False,2.4,-17.015408,-5.9378247,1734.0,9.0,2,0.8054144824310264
-723,True,TGGTACATCTGTGCGG-1,TGGTACATCTGTGCGG-1,1.8860542898113724,15.0,-0.1084213719553249,-0.02642675688499331,6.8260283,-2.3086681,22,G1,1128.3589890897274,6,8,10194.0,2540.0,7.318030213851285,40.955463998430446,0.4796580700076769,False,False,6.1,14.870863,1.4870076,2343.0,9.0,6,0.8054144824310264
-724,True,TGGTACATCTTAGCCC-1,TGGTACATCTTAGCCC-1,1.917364680268752,15.0,-0.07432804270267496,-0.02057432293736497,-3.411143,-12.941483,24,G1,1334.5839346796274,10,7,11723.0,2862.0,8.410816343939265,18.015866245841508,0.9594457652444184,False,True,7.0,-7.72156,-11.5654745,2027.0,9.0,10,1.357837327667586
-725,True,TGTCCTGAGACCAGAC-1,TGTCCTGAGACCAGAC-1,1.9440843478251237,15.0,-0.08817014152731359,-0.02356311469871993,-2.1106577,-4.3707504,19,G1,1251.6893403232098,2,2,12461.0,3445.0,7.655886365460236,27.975282882593692,0.9505496029097,False,True,3.4000000000000004,-14.984768,-3.716016,3853.0,9.0,2,0.8054144824310264
-726,True,TGTCCTGAGATCACCT-1,TGTCCTGAGATCACCT-1,1.9656132902217094,15.0,-0.091232798940644,-0.027955299018585695,0.45000005,15.490623,2,G1,1105.997310757637,5,5,4237.0,1416.0,9.794666037290536,34.41113995751711,0.7203067558848951,False,False,8.3,-1.930388,14.350856,607.0,9.0,5,1.4729340716123187
-727,True,TGTCCTGAGCAGCCCT-1,TGTCCTGAGCAGCCCT-1,1.9734430199328175,15.0,-0.12466789045697432,-0.012944363877971479,-4.765042,-5.4361906,9,G1,1279.3510907292366,2,2,14915.0,3096.0,10.3855179349648,29.782098558498156,0.945398541154095,False,False,2.8,-16.278988,-4.7119308,1386.0,9.0,2,0.8054144824310264
-728,True,TGTCCTGAGCTCACTA-1,TGTCCTGAGCTCACTA-1,2.0129199589867244,15.0,-0.046455346585956465,-0.027795147866941032,-3.833991,1.2910342,14,G1,1428.452069401741,4,4,15384.0,3738.0,7.722308892355694,16.978679147165888,0.9415303607827716,False,False,5.800000000000001,-14.903009,0.63063896,1937.0,9.0,4,1.5524679448293264
-731,True,TGTCCTGCAAGAGTAT-1,TGTCCTGCAAGAGTAT-1,1.880229815854178,15.0,-0.08976118113084655,-0.010429224770543453,8.716928,2.0649524,8,G1,1258.5271293222904,9,9,5949.0,1851.0,7.060010085728694,35.56900319381408,0.6026359538116105,False,True,4.6,13.277973,5.3792305,1005.0,9.0,9,1.5090271454508366
-732,True,TGTCCTGCAATCAAGA-1,TGTCCTGCAATCAAGA-1,1.9121122231911436,15.0,-0.10220777520834244,-0.00827667083883044,9.207056,5.1361117,26,G1,1121.9906817749143,9,9,4100.0,1179.0,10.439024390243903,43.4390243902439,0.6132679830460128,False,False,3.4000000000000004,11.08734,5.405723,1516.0,9.0,9,0.994405497775594
-733,True,TGTCCTGCAGCAGTGA-1,TGTCCTGCAGCAGTGA-1,1.9741747581081432,15.0,-0.09282845928220478,-0.027166726210659103,-8.237478,-2.2110348,20,G1,1318.79146258533,4,3,6557.0,2306.0,9.791062986121702,13.25301204819277,0.9759200356607102,False,True,2.5999999999999996,-17.649857,-2.1510115,2227.0,9.0,4,1.0266563462822664
-734,True,TGTCCTGCAGGTTCAT-1,TGTCCTGCAGGTTCAT-1,2.011345775358929,15.0,-0.08083217385542966,-0.02359120563004058,-6.9259295,1.5645906,4,G1,1491.2352100163698,4,4,10800.0,3106.0,8.36111111111111,13.703703703703704,0.9553014294977409,False,False,4.9,-14.776884,0.6592473,2221.0,9.0,4,1.7596138170802211
-735,True,TGTCCTGCATCAGTGT-1,TGTCCTGCATCAGTGT-1,2.0371746282503764,15.0,-0.06859238007127956,-0.020697888361983174,-6.848836,-6.4982953,3,G1,1340.3999814540148,2,2,6118.0,2264.0,16.181758744687805,11.768551814318405,0.9647265414578146,False,False,4.2,-15.285648,-6.284294,1849.0,9.0,2,1.2565453708589873
-736,True,TGTCCTGCATCCTATT-1,TGTCCTGCATCCTATT-1,1.8925517654975101,15.0,-0.07222736615686572,-0.03439407005768594,9.25675,-9.098481,11,G1,1232.3332973569632,7,6,3839.0,1548.0,14.11825996353217,13.961969262828863,0.1103943871471451,False,False,6.300000000000001,13.458703,-9.040423,1174.0,9.0,7,1.1538380923351783
-737,True,TGTCCTGCATGAAGGC-1,TGTCCTGCATGAAGGC-1,1.9236508661044558,15.0,-0.11994642799308824,-0.0072494272686586345,3.3905487,2.6621463,12,G1,1779.5438902527094,1,1,7825.0,1646.0,8.447284345047922,49.78913738019169,0.611382959772176,False,False,5.4,12.046296,9.274032,838.0,9.0,1,1.5503286204324531
-738,True,TGTCCTGGTAACTTCG-1,TGTCCTGGTAACTTCG-1,1.9935006662436916,15.0,-0.08749817673860144,-0.027152448235304504,-8.838496,-3.423599,7,G1,1656.2593463212252,3,3,8115.0,2590.0,9.661121380160196,19.71657424522489,0.9820539815447719,False,False,4.2,-14.19843,-3.4327896,2818.0,9.0,3,1.4709352270695035
-739,True,TGTCCTGGTAGAATGT-1,TGTCCTGGTAGAATGT-1,1.9252233422202916,15.0,-0.0851615540486835,-0.007476365747280431,5.4570036,8.590933,27,G1,1059.1375079900026,8,1,6932.0,1864.0,8.799769186381997,44.0132717830352,0.615222212255545,False,False,2.7,11.137352,11.336961,911.0,9.0,8,1.4194593160291944
-740,True,TGTCCTGGTAGCGTAG-1,TGTCCTGGTAGCGTAG-1,1.957688685325323,15.0,-0.11743777365381364,-0.0016742928003249218,0.29628354,12.34783,2,G1,874.7393051683903,5,5,5136.0,1768.0,9.871495327102803,33.31386292834891,0.7250754621375203,False,False,7.1000000000000005,-0.48804674,16.727488,1163.0,9.0,5,0.9362699326799151
-741,True,TGTCCTGGTATAGGAT-1,TGTCCTGGTATAGGAT-1,1.9754912911559004,15.0,-0.11724692527246106,-0.003286104085647206,2.1683464,18.888487,18,G1,1217.0519406348467,5,5,3672.0,1199.0,9.667755991285404,38.3442265795207,0.7233762333126846,False,False,9.3,-3.9285257,15.4664755,738.0,9.0,5,1.7829168942005256
-742,True,TGTCCTGGTCACTCGG-1,TGTCCTGGTCACTCGG-1,1.8877548610826922,15.0,-0.10858152879149881,-0.004433987388836122,1.9156111,3.575112,12,G1,1723.6746903955936,1,1,4911.0,1373.0,6.373447363062513,46.42638973732437,0.6109081962773073,False,False,3.8,10.680849,9.2193365,1216.0,9.0,1,1.2449347700414877
-743,True,TGTCCTGGTCCACACG-1,TGTCCTGGTCCACACG-1,1.8651994322952643,15.0,-0.05714878313462741,-0.012393644667660218,8.948169,-3.853235,23,G1,1172.0634506344795,6,8,4025.0,1751.0,6.136645962732919,20.29813664596273,0.385913429808859,False,False,6.000000000000001,14.361672,-1.8630714,1197.0,9.0,6,1.3984666828444978
-744,True,TGTCCTGGTGACAGGT-1,TGTCCTGGTGACAGGT-1,2.1044371472376504,15.0,-0.04664805145368088,-0.02866048077956508,-6.0115957,3.9681327,14,G1,1396.805547952652,4,4,2710.0,1193.0,20.66420664206642,9.59409594095941,0.932409247898045,False,False,6.8,-14.557359,2.8455603,1966.0,9.0,4,0.8054144824310264
-745,True,TGTCCTGGTGCCTTCT-1,TGTCCTGGTGCCTTCT-1,1.8599432686230368,15.0,-0.08198562118747262,-0.038779218602953164,9.035441,-13.116464,21,G1,1100.6179761737585,7,6,4944.0,2067.0,8.353559870550162,15.29126213592233,0.01138698590317793,False,False,6.0,11.052999,-11.904284,890.0,9.0,7,1.1903291071671906
-747,True,TGTCCTGTCCTGCTAC-1,TGTCCTGTCCTGCTAC-1,1.9438201957052987,15.0,-0.08914412301135688,-0.02854755328545661,-2.8329115,-2.476909,19,G1,1356.29573084414,2,2,9895.0,3083.0,8.29711975745326,13.623041940373927,0.9495302697137441,False,False,3.5000000000000004,-14.351428,-2.7654119,2573.0,9.0,2,0.8054144824310264
-748,True,TGTCCTGTCCTTATCA-1,TGTCCTGTCCTTATCA-1,1.8739426976336102,15.0,-0.09084828777012971,-0.02692042923361035,10.124462,-9.719194,21,G1,1273.2722176611423,7,6,4867.0,1887.0,7.643312101910828,18.24532566262585,0.08095612650161749,False,False,6.1,12.764945,-9.372049,1108.0,9.0,7,0.8054144824310264
-749,True,TGTCCTGTCGCCTTGT-1,TGTCCTGTCGCCTTGT-1,2.0391777688168444,15.0,-0.06475327887057554,-0.01786127512728626,-8.414596,0.8401632,4,G1,1488.4632907807827,4,4,3748.0,1612.0,14.541088580576307,10.005336179295623,0.9729194677056379,False,False,5.6,-14.48177,-0.31520638,2084.0,9.0,4,1.1399911048344968
-750,True,TGTCCTGTCGTCGGGT-1,TGTCCTGTCGTCGGGT-1,2.0455329391093113,15.0,-0.0664465838119449,-0.027784874808380176,-4.7377,-1.5307406,20,G1,1450.8899179548025,4,4,6080.0,2233.0,10.707236842105264,13.388157894736842,0.9588598092610706,False,False,2.3000000000000007,-15.264599,-2.0068157,7171.0,9.0,4,1.6103570683267916
-751,True,TGTCCTGTCTCGAGTA-1,TGTCCTGTCTCGAGTA-1,1.8962142807611084,15.0,-0.12022368531598933,-0.011267283412213851,2.4431252,7.9659977,5,G1,1650.7070639953017,1,1,7275.0,1830.0,4.769759450171821,46.87285223367697,0.6145300802073601,False,False,4.6000000000000005,9.472733,10.841999,1164.0,9.0,1,1.4207972118459975
-752,True,TGTGATGAGACCATTC-1,TGTGATGAGACCATTC-1,1.8699770888531382,15.0,-0.08138970090325326,0.002360002747190691,9.8293085,-6.079503,23,S,1153.779224023223,6,8,5015.0,1953.0,3.7088733798604188,21.136590229312063,0.2531702832926107,False,False,6.3999999999999995,14.244544,-4.742673,1210.0,9.0,6,1.2099737981140368
-753,True,TGTGATGAGACGTCCC-1,TGTGATGAGACGTCCC-1,1.8594184193400594,15.0,-0.06296257322249116,-0.0422615648201028,11.88913,-9.04346,11,G1,1224.3050207346678,7,6,1700.0,935.0,9.764705882352942,16.294117647058822,0.13286077154027318,False,False,6.9,12.769146,-7.235843,3087.0,10.0,7,1.4578952213257133
-754,True,TGTGATGAGACGTCGA-1,TGTGATGAGACGTCGA-1,2.002670043717114,15.0,-0.059429990320731874,-0.022190872163394604,-8.268753,1.2638988,4,G1,1478.0252715051174,4,4,6996.0,2529.0,14.122355631789594,11.120640365923386,0.9716872707156552,False,False,5.8999999999999995,-15.148985,0.22915505,2515.0,9.0,4,1.5463095259133282
-755,True,TGTGATGAGATCACCT-1,TGTGATGAGATCACCT-1,1.9902697802956906,15.0,-0.04358081790299998,-0.022092633763587374,-5.172722,-0.6400332,20,G1,1163.6595047563314,4,4,7115.0,2570.0,11.581166549543218,12.902319044272664,0.9607989690671437,True,True,5.000000000000001,-16.96524,-0.15379454,2676.0,9.0,4,1.0223161199091562
-756,True,TGTGATGAGCACCTGC-1,TGTGATGAGCACCTGC-1,1.9423891097703267,15.0,-0.09052049708754564,-0.01584507720106633,-6.913187,-12.242573,10,G1,1071.421182692051,3,7,10927.0,3137.0,12.656721881577743,18.586986364052347,0.9631460197112968,False,False,3.8000000000000003,-10.0325365,-10.7817745,1827.0,9.0,3,0.9637775905036999
-757,True,TGTGATGAGCCTCTTC-1,TGTGATGAGCCTCTTC-1,1.8915394344338388,15.0,-0.08891244813036746,-0.015114335502685016,3.545871,8.349058,5,G1,1360.532907217741,1,1,3692.0,1156.0,7.719393282773565,43.066088840736725,0.612930847596162,False,False,3.4999999999999996,9.195339,10.307538,1299.0,9.0,1,0.8054144824310264
-759,True,TGTGATGAGTAGTCTC-1,TGTGATGAGTAGTCTC-1,2.0258883651591555,15.0,0.022383412193512642,-0.03590509565117481,0.003162016,8.216598,5,G2M,1296.9681415706873,1,1,1352.0,710.0,24.70414201183432,4.881656804733728,0.6278250436898888,False,False,5.0,9.689995,12.692268,896.0,9.0,1,1.3328911693489227
-760,True,TGTGATGAGTATAACG-1,TGTGATGAGTATAACG-1,1.8958107075448165,15.0,-0.10549974771799459,-0.01036072880733075,3.114544,2.4391558,12,G1,1580.433781400323,1,1,5850.0,1488.0,5.965811965811966,47.21367521367522,0.6135218189775006,False,False,5.0,12.491536,10.150838,1091.0,9.0,1,1.186046434490902
-761,True,TGTGATGAGTCATCCA-1,TGTGATGAGTCATCCA-1,1.9191157579869884,15.0,-0.07850085128313056,-0.019075964964028923,5.742293,1.8336173,12,G1,1786.7683380842209,1,1,3536.0,1147.0,16.68552036199095,29.0158371040724,0.5992554581262096,False,False,6.5,12.816557,8.282912,1502.0,9.0,1,1.287099085379463
-762,True,TGTGATGAGTCCCAAT-1,TGTGATGAGTCCCAAT-1,2.0127671490312786,15.0,-0.04936384706084413,-0.030599954815243722,-5.9047256,0.8160577,20,G1,1473.191466987133,4,4,7725.0,2545.0,13.229773462783172,10.964401294498382,0.9578311293628247,False,True,4.8,-16.260683,0.6389205,2463.0,9.0,4,1.7559983261623684
-765,True,TGTGATGCAATCTGCA-1,TGTGATGCAATCTGCA-1,1.943544232085012,15.0,-0.06534355853981896,-0.018846317303197455,8.087464,-8.607146,11,G1,1169.5885694473982,7,6,12582.0,3963.0,9.871244635193133,12.056906692099826,0.1782334255817215,False,False,4.6,14.091203,-8.498599,2775.0,9.0,7,0.9362956489179584
-768,True,TGTGATGGTAACTTCG-1,TGTGATGGTAACTTCG-1,1.8896757760310963,15.0,-0.08874529609013322,-0.0074656569207062675,5.9505515,3.9363832,6,G1,1535.353196874261,1,1,6558.0,1755.0,5.85544373284538,47.04178103080207,0.6103428685909125,False,False,4.3999999999999995,12.151866,7.619268,1150.0,9.0,1,0.8054144824310264
-770,True,TGTGATGGTGCACAAG-1,TGTGATGGTGCACAAG-1,1.9746013971306806,15.0,-0.056877750106204286,-0.03754468346130253,-7.821018,-2.9351847,10,G1,1455.0421804487705,3,3,7344.0,2735.0,10.049019607843137,9.191176470588236,0.9720968309468662,False,True,2.8000000000000003,-14.23063,-3.0232525,3109.0,9.0,3,1.2811098899469326
-772,True,TGTGATGTCATCGCTC-1,TGTGATGTCATCGCTC-1,1.971919922367477,15.0,-0.11586427455100505,-0.0076923973472884325,2.838526,18.93781,18,G1,1169.9681016132236,5,5,5355.0,1481.0,8.571428571428571,40.26143790849673,0.7227137404306239,False,False,8.1,-3.9903052,14.759049,964.0,9.0,5,1.1492905295886513
-773,True,TGTGATGTCCCGTTGT-1,TGTGATGTCCCGTTGT-1,1.8979202933055017,15.0,-0.10313909647706189,0.003933925223191009,8.23399,4.512122,26,S,1440.2268758416176,9,9,6828.0,1754.0,6.5172817809021675,46.42647920328061,0.6100413467122258,False,False,4.3,11.591798,6.469926,871.0,9.0,9,1.3965182310137936
-774,True,TGTGATGTCCGCTTAC-1,TGTGATGTCCGCTTAC-1,1.8668055891437116,15.0,-0.10038819217327795,-0.021287043358448166,9.860882,-9.349485,11,G1,1226.159728512168,7,6,4057.0,1595.0,6.630515158984471,21.838797140744393,0.10715825320928203,False,False,5.2,12.834761,-8.610558,1029.0,9.0,7,1.3336802027849282
-775,True,TGTGATGTCCTACAAG-1,TGTGATGTCCTACAAG-1,1.918207204784291,15.0,-0.11055075472171733,-0.011230227963632176,3.5585155,6.8085895,6,G1,1346.4516777023673,1,1,7863.0,1933.0,9.105939208953325,45.28805799313239,0.6126006382095321,False,False,3.8000000000000003,8.998334,8.670845,1188.0,9.0,1,0.9768199251844366
-776,True,TGTGATGTCGCGGTAC-1,TGTGATGTCGCGGTAC-1,1.969157764004461,15.0,-0.13616527293340647,0.0008666349362728663,0.43775743,12.073963,2,S,874.7393051683903,5,5,8002.0,1790.0,9.585103724068983,44.23894026493377,0.7250018967120385,False,False,7.1000000000000005,-0.32569414,17.088692,991.0,9.0,5,0.9267070380396117
-777,True,TTACAGGAGCTTGTTG-1,TTACAGGAGCTTGTTG-1,1.9144210910157125,15.0,-0.12063061890787728,-0.003286641118509707,8.298939,-2.3887007,22,G1,1189.415604159236,6,8,4853.0,1439.0,8.530805687203792,41.211621677313005,0.4649355255540338,False,False,7.6,14.589276,0.6786306,895.0,9.0,6,1.26166821458594
-779,True,TTACAGGCAAGTCGTT-1,TTACAGGCAAGTCGTT-1,1.9547158435558727,15.0,-0.07583643431629818,-0.029920067662846515,-4.1658845,-3.999729,17,G1,1760.917803823948,2,2,12054.0,3468.0,7.375145180023229,18.234610917537747,0.9581938335193284,False,False,5.1,-14.5362215,-4.2232075,2724.0,9.0,2,1.6548521468330977
-781,True,TTACAGGCACCGTGAC-1,TTACAGGCACCGTGAC-1,2.0211588464387775,15.0,-0.06458810275685482,-0.03025165062459449,-2.8186924,-3.4168575,19,G1,1506.8784712404013,2,2,12226.0,3270.0,9.487976443644692,17.16015049893669,0.9560814977192231,False,False,3.8999999999999995,-15.387221,-3.4632876,1654.0,9.0,2,1.060198703168392
-782,True,TTACAGGCACGCGCAT-1,TTACAGGCACGCGCAT-1,1.880100179222348,15.0,-0.07976150557628482,-0.02639488317693825,11.414364,-8.977948,11,G1,1231.6576531976461,7,6,5351.0,1896.0,8.521771631470752,20.07101476359559,0.13161221951324903,False,False,6.1,13.517597,-7.434969,1158.0,9.0,7,0.9625823252815406
-783,True,TTACAGGCACTACGGC-1,TTACAGGCACTACGGC-1,1.863786659967178,15.0,-0.07376606373169986,-0.02068025537412085,8.456794,-14.064933,21,G1,1084.4156861752272,7,6,8209.0,3048.0,7.784139359239859,14.045559751492265,0.0023921021241353115,False,False,7.6,10.099889,-12.412533,1169.0,9.0,7,1.054449906360884
-784,True,TTACAGGCACTATGTG-1,TTACAGGCACTATGTG-1,1.8820304905342313,15.0,-0.1016387705800872,-0.014795039472315835,3.9309125,5.466453,12,G1,1817.991869442165,1,1,7184.0,1839.0,3.8557906458797326,49.05345211581292,0.6101786568235836,False,False,4.3,10.082329,8.650741,1661.0,9.0,1,1.3891576830101735
-787,True,TTACAGGGTCTAGGTT-1,TTACAGGGTCTAGGTT-1,1.869406827501072,15.0,-0.10160865458638463,0.0008477900810101996,9.676507,-11.391399,21,S,1289.3595440238714,7,6,4199.0,1732.0,9.764229578471065,18.599666587282687,0.04268725381887684,False,False,6.6,11.69186,-10.466122,2919.0,9.0,7,0.8054144824310264
-788,True,TTACAGGGTGCCTTCT-1,TTACAGGGTGCCTTCT-1,1.9140753111259359,15.0,-0.10169341340039735,-0.007594760717570141,0.9084095,4.3206205,5,G1,1440.4715542197227,1,1,9058.0,2143.0,8.423493044822257,44.53521748730404,0.6024070038649685,False,False,2.6999999999999997,9.224276,10.115904,884.0,9.0,1,0.9413450291278668
-790,True,TTACAGGTCACAAGAA-1,TTACAGGTCACAAGAA-1,1.9107201860887741,15.0,-0.09848695059892426,-0.015341848100337173,9.091842,1.4287473,25,G1,951.530439004302,6,9,5596.0,1689.0,9.88205861329521,36.02573266619014,0.5940503294929078,False,False,3.2,14.292254,5.3474603,1469.0,9.0,6,0.9746864115033027
-791,True,TTACAGGTCCAATCCC-1,TTACAGGTCCAATCCC-1,1.983734173066752,15.0,-0.08118050682909767,-0.020626071384939347,-4.5361385,-3.844927,17,G1,1724.290115788579,2,2,9033.0,2880.0,7.539023580205912,13.572456548212111,0.9587155700839172,False,False,4.4,-14.843795,-4.109962,2756.0,9.0,2,1.6863565313395337
-792,True,TTACAGGTCGTAGAGG-1,TTACAGGTCGTAGAGG-1,1.9949702387845065,15.0,-0.11751151307077841,-0.013825701204634812,-1.9741431,15.74029,2,G1,1245.386822938919,5,5,6631.0,1815.0,13.84406575177198,36.66113708339617,0.7327370943618671,False,False,8.299999999999999,0.90784925,13.792503,1115.0,9.0,5,0.8054144824310264
-793,True,TTACAGGTCGTTTACT-1,TTACAGGTCGTTTACT-1,1.9890036483227598,15.0,-0.07626212465988462,-0.02915275709703739,-5.29986,-1.3441175,20,G1,1171.1466239392757,4,4,7176.0,2486.0,12.639353400222966,14.241917502787068,0.9626546477152669,False,False,4.300000000000001,-16.491022,-1.3243436,1407.0,9.0,4,1.104877227849968
-794,True,TTCGATTAGCCTCTGG-1,TTCGATTAGCCTCTGG-1,1.9631693251041176,15.0,-0.06865708469557708,-0.01428093165622459,-7.780041,-1.2791023,14,G1,1343.2738090604544,4,4,14500.0,3792.0,7.337931034482758,23.76551724137931,0.9479123612226927,False,False,1.1,-13.474938,-0.9305661,2745.0,9.0,4,1.0926648690157201
-795,True,TTCGATTAGGCGTTAG-1,TTCGATTAGGCGTTAG-1,1.928713646918808,15.0,-0.07403403075879844,-0.018897249880283722,8.851331,-11.595025,21,G1,1249.209385380149,7,6,2515.0,1150.0,17.852882703777336,14.671968190854871,0.03526252521358326,False,False,5.8999999999999995,11.472835,-10.496805,1319.0,9.0,7,1.0149907358058574
-796,True,TTCGATTAGGTTGCCC-1,TTCGATTAGGTTGCCC-1,1.8649231107748268,15.0,-0.08106329206572681,-0.0008553770251572661,0.14592372,4.6286373,5,G1,975.1607788205147,1,1,4432.0,1552.0,4.30956678700361,36.28158844765343,0.6101413900688135,False,False,3.3000000000000003,8.256551,9.4347725,1459.0,9.0,1,1.1256431281235757
-797,True,TTCGATTAGTATAACG-1,TTCGATTAGTATAACG-1,1.9179810431212303,15.0,-0.1089560915722368,-0.008929451068204801,3.7961442,2.0741343,12,G1,1526.6916943192482,1,1,7818.0,1805.0,7.802507035047327,46.763878229726274,0.6117926384130513,False,False,4.3,11.994957,8.831274,897.0,9.0,1,0.9404045018181607
-798,True,TTCGATTCAATCTGCA-1,TTCGATTCAATCTGCA-1,1.9608337696806817,15.0,-0.08077660865412815,-0.007930530694502825,-12.441348,-4.0289507,7,G1,964.5888964086771,3,3,12079.0,3651.0,9.421309711068798,15.05919364185777,0.9952075944258093,False,False,4.8,-18.631914,-4.345091,2261.0,9.0,3,1.1682968396501654
-799,True,TTCGATTCATTATGCG-1,TTCGATTCATTATGCG-1,2.0118543827454745,15.0,-0.07824926123351353,-0.027633763608679678,-5.8079047,1.9708694,14,G1,1556.311996564269,4,4,9379.0,2777.0,10.864697728968974,17.7950741017166,0.9461969530887814,False,True,4.3,-14.733806,0.78425884,1493.0,9.0,4,0.8054144824310264
-800,True,TTCGATTGTAACGCGA-1,TTCGATTGTAACGCGA-1,2.0344376272162523,15.0,-0.06042445240137878,-0.030891970544974143,-5.219139,1.6673121,14,G1,1489.2060773521662,4,4,8297.0,2750.0,10.087983608533206,15.258527178498252,0.9447871588863059,False,False,4.699999999999999,-14.991589,1.0256742,1905.0,9.0,4,1.848549175898292
-802,True,TTCGATTGTGCGGTAA-1,TTCGATTGTGCGGTAA-1,1.9888991161998046,15.0,-0.08881710451137034,-0.02906812130343859,-6.7208543,-4.2697744,13,G1,1164.5691663473845,2,2,6311.0,2487.0,16.19394707653304,11.376960861987007,0.9562796949061462,False,False,2.2,-16.034702,-4.3789635,3529.0,9.0,2,1.0208956556010638
-803,True,TTCGATTGTGTCTAAC-1,TTCGATTGTGTCTAAC-1,1.9477074839190385,15.0,-0.10874677100114714,-0.018153062733224132,2.0897775,17.661932,18,G1,1068.0044005140662,5,5,9171.0,2237.0,7.34925308036201,41.020608439646715,0.7222859859982317,True,True,4.8,-4.106,14.503065,1622.0,9.0,5,0.9552628918617548
-804,True,TTCGATTGTTAAGACA-1,TTCGATTGTTAAGACA-1,1.9195328507802165,15.0,-0.10269830108318244,-0.0023951703948935695,0.78249884,4.268289,5,G1,1215.2527044862509,1,1,7856.0,1933.0,9.521384928716904,42.90987780040733,0.6113422542057891,False,False,2.3000000000000003,9.402512,8.975839,793.0,9.0,1,0.8054144824310264
-805,True,TTCGATTGTTAAGCAA-1,TTCGATTGTTAAGCAA-1,1.9421704727415126,15.0,-0.08238245495608054,-0.012615184329428642,-12.081141,-6.5755415,7,G1,992.9331075549126,3,3,10452.0,3310.0,9.672789896670494,16.054343666283966,0.9798407488628106,False,False,3.1999999999999997,-18.472147,-5.788819,2927.0,9.0,3,0.8054144824310264
-806,True,TTCGATTGTTCGGCTG-1,TTCGATTGTTCGGCTG-1,1.903802507693129,15.0,-0.0968745433938719,-0.009759256367627749,7.413578,-3.3116705,22,G1,1199.1119232177734,6,8,3624.0,1368.0,10.596026490066226,31.76048565121413,0.44256553865472414,False,False,8.0,14.787608,0.11021995,1569.0,9.0,6,1.0169260766413897
-807,True,TTCGATTTCACAAGAA-1,TTCGATTTCACAAGAA-1,1.9640852845711783,15.0,-0.07740569991184637,-0.027162351538518852,-6.9134884,-6.79006,3,G1,1308.4515990763903,2,2,11540.0,3527.0,9.939341421143848,14.428076256499134,0.9633406149641547,False,False,3.5,-15.62653,-7.1037297,3259.0,9.0,2,1.4677254263487154
-808,True,TTCGATTTCCAATCCC-1,TTCGATTTCCAATCCC-1,1.9125397362007335,15.0,-0.10178809368463583,-0.02359250685130612,2.3287635,7.083042,5,G1,1227.3833480924368,1,1,4609.0,1340.0,12.280321110870037,35.86461271425472,0.6121793169715376,False,False,3.5,8.380452,9.634605,870.0,9.0,1,1.340469844704727
-809,True,TTCGATTTCGAACGCC-1,TTCGATTTCGAACGCC-1,1.8854513209431758,15.0,-0.1193423568330109,-0.0059031673451434685,4.001051,7.340846,12,G1,1269.300905406475,1,1,6563.0,1834.0,5.1653207374676215,43.821423129666314,0.6127036409071476,False,False,3.5000000000000004,8.850339,9.3605,1656.0,9.0,1,1.3494763739696543
-810,True,TTGTTTGAGAAATTGC-1,TTGTTTGAGAAATTGC-1,1.9752689028906927,15.0,-0.07757841973821714,-0.00987810292818464,-10.772302,-3.7890031,10,G1,1330.34066426754,3,3,5050.0,1978.0,15.069306930693068,10.732673267326733,0.9914607104775305,False,False,3.2,-17.594032,-3.3788261,1761.0,10.0,3,1.1917359633853428
-811,True,TTGTTTGAGACCAGAC-1,TTGTTTGAGACCAGAC-1,1.9621097761570383,15.0,-0.09456534419079017,-0.02008473801119061,-6.5687246,-9.546865,3,G1,1010.1624644100666,2,2,13551.0,3755.0,8.892332669175707,15.984060216958158,0.9511266457105118,False,False,3.0,-15.020468,-9.309875,3123.0,9.0,2,1.3374992685758762
-813,True,TTGTTTGAGACGTCCC-1,TTGTTTGAGACGTCCC-1,1.8745730638202727,15.0,-0.06834019932176122,-0.030724741947241542,9.180972,-13.146715,21,G1,1132.0540813356638,7,6,4943.0,1991.0,10.277159619664172,13.857980983208577,0.005732670789113113,False,False,6.899999999999999,10.251482,-11.901026,1999.0,9.0,7,1.396173189026539
-815,True,TTGTTTGAGCAATTAG-1,TTGTTTGAGCAATTAG-1,1.9066917558300296,15.0,-0.1054992652371328,-0.018887705732656974,5.51581,3.0671074,6,G1,1684.0923893153667,1,1,9507.0,2223.0,7.100031555695803,43.78878720942463,0.6069726364709669,False,False,4.2,12.87091,7.9099436,1433.0,9.0,1,1.188516974947064
-816,True,TTGTTTGAGGCCTAGA-1,TTGTTTGAGGCCTAGA-1,1.9181357430858224,15.0,-0.07599780166017484,-0.01865549315817125,-4.6159506,-12.828959,24,G1,1297.5579494386911,10,7,11028.0,2983.0,8.07943416757345,15.034457743924555,0.9601773092099093,False,False,5.6,-7.7111163,-11.129664,1154.0,9.0,10,0.9416146684201823
-817,True,TTGTTTGCAAGTCGTT-1,TTGTTTGCAAGTCGTT-1,1.9533856827931142,15.0,-0.12885978272673784,-0.008880581728370654,-2.6169894,14.133298,2,G1,1164.3194821029902,5,5,9933.0,2589.0,7.701600724856539,34.34007852612504,0.7328779266116664,False,False,7.3,1.3515564,15.070348,1501.0,9.0,5,1.4685209615728756
-818,True,TTGTTTGCACCCTCTA-1,TTGTTTGCACCCTCTA-1,1.9217212087577784,15.0,-0.11378021075890364,-0.000840733228175742,2.281904,19.181974,18,G1,1198.0498885288835,5,5,5609.0,1868.0,4.671064360848636,37.22588696737386,0.7229715840619367,False,False,9.600000000000001,-4.176912,15.232005,737.0,9.0,5,1.5835220671435046
-819,True,TTGTTTGCACTATGTG-1,TTGTTTGCACTATGTG-1,1.9473365061266577,15.0,-0.08515531677873708,-0.028869038328033735,-8.072493,-5.4398336,13,G1,1411.8426660001278,2,2,12440.0,3714.0,7.564308681672026,12.733118971061094,0.9742945039929677,False,False,2.8000000000000003,-16.999773,-5.9771724,2409.0,9.0,2,1.3707553553494534
-820,True,TTGTTTGCACTTCAGA-1,TTGTTTGCACTTCAGA-1,1.8671083641448456,15.0,-0.05843897425594144,-0.031179197758492774,7.938827,-13.788934,21,G1,1058.2730616033077,7,6,3704.0,1693.0,8.450323974082073,15.955723542116631,0.0054462222247503205,False,True,6.7,10.610644,-12.3141575,521.0,9.0,7,0.8054144824310264
-821,True,TTGTTTGCAGCTACTA-1,TTGTTTGCAGCTACTA-1,1.9479615766758482,15.0,-0.09976606577679925,0.019993425155632117,-0.34441575,13.141485,2,S,964.2033444046974,5,5,5330.0,1752.0,8.03001876172608,36.02251407129456,0.7270697192371115,False,False,5.1000000000000005,-0.08312444,16.123432,1308.0,9.0,5,1.1779758086943841
-822,True,TTGTTTGCATATCTGG-1,TTGTTTGCATATCTGG-1,1.985089671216524,15.0,-0.05410006347740781,-0.03239042250160321,-2.3342252,-3.2354357,19,G1,1494.2676855921745,2,2,4960.0,2126.0,19.43548387096774,7.237903225806452,0.9544236710401232,False,False,4.8,-14.051211,-3.4206548,2035.0,9.0,2,1.0525581538937765
-823,True,TTGTTTGCATCCTATT-1,TTGTTTGCATCCTATT-1,1.8909349033372356,15.0,-0.07287427077146383,-0.012455177568919184,10.540299,-7.8922668,11,G1,1215.4865544587374,7,6,5063.0,1846.0,6.814141813154256,27.078807031404306,0.13658278960963938,False,False,7.5,13.823376,-7.8621225,2083.0,9.0,7,0.8054144824310264
-824,True,TTGTTTGGTACGGGAT-1,TTGTTTGGTACGGGAT-1,1.9202850900238484,15.0,-0.09669055217116364,-0.0021071062286991994,7.4346137,10.0420685,15,G1,1204.904662579298,8,10,8930.0,1953.0,7.435610302351623,48.21948488241881,0.6222434056885917,False,False,5.5,14.833162,10.197359,909.0,9.0,8,1.6211560017212299
-825,True,TTGTTTGGTATACCTG-1,TTGTTTGGTATACCTG-1,1.9343771314970037,15.0,-0.0874661068835826,-0.0235254712585281,-9.707651,-9.559587,10,G1,1172.2345714718103,3,7,15244.0,3730.0,7.340593020204671,18.80739963264235,0.9667254728563515,False,False,5.6,-11.290097,-9.605325,2939.0,9.0,3,1.6623647202889213
-826,True,TTGTTTGGTCACGCTG-1,TTGTTTGGTCACGCTG-1,1.910520151675248,15.0,-0.09986865582294008,-0.0199584924426294,9.609835,-3.6800518,23,G1,1143.5191202908754,6,8,6662.0,2023.0,9.606724707295106,37.00090063044131,0.3955592402813177,False,False,6.900000000000001,14.519898,-1.3914303,1453.0,9.0,6,1.0970140788841232
-827,True,TTGTTTGGTTTGATCG-1,TTGTTTGGTTTGATCG-1,1.915807264806834,15.0,-0.105964352439357,-0.011693154419245698,4.383486,4.548897,12,G1,1778.920767262578,1,1,8382.0,1896.0,7.217847769028872,47.51849200668099,0.6116652448150776,False,False,4.6000000000000005,10.368675,8.270589,1128.0,9.0,1,1.1008992261743455
-829,True,TTGTTTGTCCCGTTGT-1,TTGTTTGTCCCGTTGT-1,1.8904271189518058,15.0,-0.11790194059342561,-0.007929377743535616,6.647358,3.584762,6,G1,1616.9419396817684,1,1,7154.0,1761.0,6.13642717360917,46.12804025719877,0.6092344315439678,False,False,4.300000000000001,12.4094925,8.071194,1140.0,9.0,1,1.2636853133742638
-830,True,TTGTTTGTCCGCTTAC-1,TTGTTTGTCCGCTTAC-1,1.9340631091893716,15.0,-0.08316946058226787,0.007226235772029667,-0.59491754,16.386585,2,S,1120.9773846194148,5,5,7012.0,2021.0,8.214489446662864,38.861950941243585,0.7260288282457544,False,True,3.9000000000000004,-0.9130639,13.869253,712.0,9.0,5,0.8054144824310264
-831,True,TTGTTTGTCTAGTCAG-1,TTGTTTGTCTAGTCAG-1,1.914348065114423,15.0,-0.11822268268567233,-0.016390210968518143,2.1490452,4.5141597,12,G1,1790.6346159130335,1,1,5774.0,1673.0,9.005888465535158,39.729823346033946,0.6138953933954113,False,False,4.1,10.697361,10.452252,1207.0,9.0,1,1.4296175557143471
-832,True,TTGTTTGTCTCCGAGG-1,TTGTTTGTCTCCGAGG-1,1.9746648258488386,15.0,-0.03403378221692178,-0.03391155545281759,-7.931644,-12.821559,16,G1,678.1539006382227,2,2,4608.0,1253.0,5.403645833333333,1.6710069444444444,0.9320526574632143,False,False,6.000000000000001,-15.255273,-11.320599,418.0,10.0,2,1.0639007614998948
-833,True,TTTCATGAGACCATTC-1,TTTCATGAGACCATTC-1,1.9053436661977547,15.0,-0.0826095216364206,-0.009610521892234585,1.5492808,6.737165,5,G1,1748.7209868729115,1,1,7009.0,1884.0,6.9910115565701245,43.629619061207016,0.6136561007123282,False,False,7.3,9.397441,11.10729,764.0,9.0,1,1.8353995103829035
-834,True,TTTCATGAGCTCACTA-1,TTTCATGAGCTCACTA-1,1.902431747507048,15.0,-0.09963029218843172,-0.010319001386962555,5.0693865,5.910128,12,G1,1696.854793831706,1,1,6250.0,1690.0,6.96,44.032,0.6108920397404606,False,False,5.199999999999999,10.163379,8.36053,779.0,9.0,1,0.9936018591465023
-835,True,TTTCATGAGGAACTCG-1,TTTCATGAGGAACTCG-1,2.081869246235219,15.0,-0.025958231548530022,-0.02486455454979241,-12.472483,-1.6728506,7,G1,1073.500611051917,3,3,3239.0,1399.0,30.56498919419574,2.686014201914171,0.9983722905717957,False,False,3.8000000000000003,-18.925875,-2.2101033,1974.0,9.0,3,0.8054144824310264
-837,True,TTTCATGCAATAGTAG-1,TTTCATGCAATAGTAG-1,1.980117458915072,15.0,-0.06346314162452374,-0.0315566840222714,-10.975349,-2.4686308,7,G1,1359.0012575536966,3,3,12135.0,3529.0,10.045323444581788,12.632880098887515,0.9965198728365181,False,True,3.0000000000000004,-18.049793,-3.0058286,2661.0,9.0,3,1.354153753751116
-838,True,TTTCATGCACAAATCC-1,TTTCATGCACAAATCC-1,2.0865810434795247,15.0,-0.05635228353817205,-0.010165918643523786,-4.786767,2.4467435,14,G1,1192.5043985396624,4,4,1619.0,855.0,23.65657813465102,5.435453983940704,0.8818081429272452,False,False,3.3,-13.075273,1.6400137,3321.0,9.0,4,0.8054144824310264
-839,True,TTTCATGCACAGAGAC-1,TTTCATGCACAGAGAC-1,1.9827361549109466,15.0,-0.0607892369610962,-0.019737441285118423,-7.1416144,2.125142,14,G1,1416.560026422143,4,4,12117.0,3454.0,8.021787571180985,14.549806057605018,0.9444012609014811,False,False,4.2,-13.788753,0.52584934,2347.0,9.0,4,1.5214671427815627
-840,True,TTTCATGGTATGAGCG-1,TTTCATGGTATGAGCG-1,1.9848093043658221,15.0,-0.09059345451060567,-0.0014275532219360197,1.7681144,15.977026,2,G1,1099.15285089612,5,5,5376.0,1536.0,12.239583333333334,39.78794642857143,0.7192568907062931,True,True,9.700000000000001,-2.2076035,14.47884,1325.0,9.0,5,1.1657563932436377
-841,True,TTTCATGGTCTTCATT-1,TTTCATGGTCTTCATT-1,1.899735812440576,15.0,-0.08175444927533435,-0.012624052354469926,5.698633,6.0660357,12,G1,1468.4821402207017,1,1,7416.0,2046.0,4.854368932038835,43.67583603020496,0.6117084621413412,False,False,3.9,10.459781,8.147521,1262.0,8.0,1,0.9882727834584122
-842,True,TTTCATGGTGCACAAG-1,TTTCATGGTGCACAAG-1,1.9004130326272624,15.0,-0.0918545229977413,-0.010409649779120872,8.287515,-1.3116577,22,G1,1103.3079520314932,6,8,6844.0,2034.0,8.781414377556985,35.8562244301578,0.4981501824944171,False,False,5.1,14.564746,1.8483461,1747.0,9.0,6,1.0027619583455896
-843,True,TTTCATGGTGCCTTCT-1,TTTCATGGTGCCTTCT-1,1.9497846825193248,15.0,-0.07771081713371006,-0.026227772326966575,-3.0179029,-5.781449,19,G1,1474.7950672507286,2,2,9843.0,3384.0,5.61820583155542,14.62968607131972,0.9512974980485656,False,True,4.1,-14.781908,-5.7593703,1971.0,9.0,2,1.496729092924822
-844,True,TTTCATGGTTAAGACA-1,TTTCATGGTTAAGACA-1,1.900858909556776,15.0,-0.06961057296364562,-0.012668994643159244,8.709948,10.139294,15,G1,1061.6687054932117,8,10,5930.0,1996.0,6.020236087689713,32.78246205733558,0.6229006318903403,False,False,5.8999999999999995,15.312605,10.760088,1491.0,9.0,8,0.8933758631233296
-845,True,TTTCATGGTTGGGATG-1,TTTCATGGTTGGGATG-1,1.9735293007275374,15.0,-0.13416815742397137,-0.006083169087767962,1.7496735,18.55981,18,G1,1217.0519406348467,5,5,4940.0,1247.0,10.34412955465587,43.48178137651822,0.7233114173337123,False,False,9.3,-4.405904,15.23818,886.0,9.0,5,1.3995114052562618
-846,True,TTTCATGGTTTGATCG-1,TTTCATGGTTTGATCG-1,2.00550161956267,15.0,-0.07137386898171123,-0.025626175483850946,-7.2914724,0.9384664,14,G1,1536.9589839726686,4,4,4052.0,1743.0,16.33761105626851,10.069101678183612,0.9602837923338556,False,False,4.4,-14.240705,-0.54623425,2197.0,9.0,4,1.3002162541380378
-847,True,TTTCATGTCCACGTAA-1,TTTCATGTCCACGTAA-1,1.8869538571397328,15.0,-0.05614630630870724,-0.02393392858219023,9.560914,-14.147546,21,G1,1090.366791844368,7,6,4296.0,1881.0,7.937616387337058,12.965549348230912,0.0004933585999722467,False,False,8.9,9.5675535,-12.099556,729.0,9.0,7,1.0564091990052558
-848,True,TTTCATGTCTCATAGG-1,TTTCATGTCTCATAGG-1,1.8661443215062412,15.0,-0.0718229829660782,-0.00399756258204758,10.023972,-10.23923,21,G1,1293.6926913708448,7,6,3377.0,1521.0,8.380219129404797,17.145395321291087,0.06935945092627059,False,False,7.1000000000000005,12.298812,-9.924648,1332.0,9.0,7,1.3748487670888325
-849,True,TTTCATGTCTCTAGGA-1,TTTCATGTCTCTAGGA-1,1.9809075896782677,15.0,-0.06153621579901342,-0.01526941759027743,-9.512626,-4.242979,1,G1,1413.3102297782898,3,3,4036.0,1559.0,12.314172447968286,11.719524281466798,0.9856237457178274,False,False,4.4,-13.542432,-4.7362223,1807.0,9.0,3,1.537307872645059
-850,True,TTTCGATAGCAATTAG-1,TTTCGATAGCAATTAG-1,1.956032534047517,15.0,-0.10356088419049281,-0.011195574606825912,2.6174123,19.53445,18,G1,1196.3447940945625,5,5,4592.0,1577.0,9.277003484320558,36.106271777003485,0.722755055273252,False,False,9.8,-3.7965312,15.82559,630.0,9.0,5,0.9980207379784404
-851,True,TTTCGATAGCAGCCCT-1,TTTCGATAGCAGCCCT-1,1.9783649555935634,15.0,-0.07733700303184136,-0.026364638522549277,-4.890395,-4.507329,13,G1,1228.7539929002523,2,2,15498.0,4121.0,9.568976642147375,19.11859594786424,0.9545851307045427,False,False,3.1000000000000005,-16.634544,-5.424287,680.0,9.0,2,1.3227646984289116
-852,True,TTTCGATAGGCTCAAG-1,TTTCGATAGGCTCAAG-1,1.9281777510328753,15.0,-0.11685922562183546,-0.008410210926966546,0.5070734,14.122722,2,G1,1069.4703082293272,5,5,5761.0,1727.0,8.140947752126367,40.67002256552682,0.7231650662258335,False,False,6.199999999999998,-0.9211489,15.650139,1039.0,9.0,5,0.8054144824310264
-854,True,TTTCGATAGGTTACAA-1,TTTCGATAGGTTACAA-1,1.9834395145658077,15.0,-0.09702810349193229,-0.018276666472641277,-3.4579217,13.970264,14,G1,1054.8793014734983,4,5,12319.0,3260.0,11.307736017533891,27.883756798441432,0.7744462018865857,False,False,4.4,1.3054883,13.664667,2180.0,9.0,4,0.8054144824310264
-855,True,TTTCGATAGTATAACG-1,TTTCGATAGTATAACG-1,1.9778467375833202,15.0,-0.04323153209971339,-0.023362595884264925,-3.2459311,-0.27868858,19,G1,1389.2521254718304,2,4,3469.0,1579.0,11.761314499855866,11.040645719227443,0.9476087568427997,False,False,3.1999999999999997,-13.188602,-0.64472383,1996.0,9.0,2,0.8054144824310264
-856,True,TTTCGATAGTCATCCA-1,TTTCGATAGTCATCCA-1,1.9246239818181294,15.0,-0.10216006134427047,-0.008844625754924033,2.0101724,5.810622,12,G1,1779.5804100111127,1,1,5316.0,1487.0,11.963882618510159,40.331075996990215,0.6131774238886664,False,False,4.8999999999999995,9.586488,10.659369,927.0,9.0,1,1.0108431860397664
-857,True,TTTCGATAGTGGCCTC-1,TTTCGATAGTGGCCTC-1,2.0252024060360094,15.0,-0.05089159468233808,-0.024863508278896745,-8.975499,-0.09873688,20,G1,1496.6306136399508,4,3,9154.0,2894.0,9.17631636443085,15.479571771902993,0.9766590446020111,False,False,3.0,-17.47184,-1.0116367,1433.0,9.0,4,1.314136678557241
-858,True,TTTCGATCAACGCATT-1,TTTCGATCAACGCATT-1,1.9678142474000782,15.0,-0.07942279407544897,-0.02854721781866202,-3.8061025,-5.9308567,3,G1,1405.0495891273022,2,2,13092.0,3667.0,11.083104185762297,14.634891536816376,0.9544199724810613,True,True,3.5000000000000004,-15.34601,-5.980405,2282.0,9.0,2,1.0131033238105016
-859,True,TTTCGATCAATTAGGA-1,TTTCGATCAATTAGGA-1,1.9976063899980538,15.0,-0.06975534555776722,-0.018812394644858735,-5.245734,1.2630126,20,G1,1509.1067515611649,4,4,6655.0,2216.0,8.685199098422238,14.154770848985725,0.9498201497478881,False,False,5.7,-16.052538,1.3479481,1543.0,9.0,4,1.1298925429918807
-860,True,TTTCGATCACCAGCGT-1,TTTCGATCACCAGCGT-1,2.025678871959194,15.0,-0.06966074956002373,-0.03189665857404402,-11.600535,-1.1176513,7,G1,1367.1188849061728,3,3,3454.0,1633.0,16.47365373480023,5.67458019687319,0.9950585460661505,False,False,5.3,-18.401112,-1.9424376,1829.0,9.0,3,0.8054144824310264
-861,True,TTTCGATCAGGTTCAT-1,TTTCGATCAGGTTCAT-1,1.9774142722357326,15.0,-0.06653119721498757,-0.02353947072103861,-4.546644,-2.496202,17,G1,1570.1034287959337,2,2,8844.0,2834.0,11.79330619629127,13.319764812302125,0.9607875744580767,False,False,3.1000000000000005,-16.156998,-2.4776654,2580.0,9.0,2,1.298738306364051
-862,True,TTTCGATCATCAGTGT-1,TTTCGATCATCAGTGT-1,1.9358900144835494,15.0,-0.11175001430945306,-0.008746293391300711,3.559014,3.1816068,12,G1,1846.6502016484737,1,1,3938.0,1067.0,9.217877094972067,45.606907059421026,0.6044411598690631,False,False,5.1000000000000005,11.84136,8.207761,930.0,9.0,1,1.2179190971276364
-863,True,TTTCGATGTATGAGCG-1,TTTCGATGTATGAGCG-1,1.928528420475325,15.0,-0.1043877832768613,-0.014673192928056472,-2.7567112,15.052473,2,G1,1265.617069043219,5,5,12433.0,2849.0,7.4640070779377465,44.06820558191909,0.7295941395490527,False,False,7.8,1.2518011,13.926078,2014.0,9.0,5,0.8054144824310264
-864,True,TTTCGATGTCCCGTGA-1,TTTCGATGTCCCGTGA-1,1.9433387732719194,15.0,-0.08336764534356744,-0.02615581120603276,-10.345494,-7.386103,10,G1,1131.2083248347044,3,7,12730.0,3427.0,9.057344854673998,18.735271013354282,0.9755312349074768,False,False,3.9000000000000004,-12.350902,-8.013443,2445.0,9.0,3,1.5083956432650072
-866,True,TTTCGATGTTCCGTTC-1,TTTCGATGTTCCGTTC-1,2.0693157343073265,15.0,-0.05383839834697074,-0.030110677832313446,-4.050998,-4.589004,19,G1,1609.2243176698685,2,2,6028.0,2288.0,27.488387524883876,8.775713337757134,0.9566639519110978,False,False,4.1,-13.734276,-4.868857,2217.0,9.0,2,0.8054144824310264
-867,True,TTTCGATTCCCGTTGT-1,TTTCGATTCCCGTTGT-1,1.8855441123116619,15.0,-0.11588649012662205,-0.009914594012233583,4.051968,5.3594155,6,G1,1666.8162060678005,1,1,5134.0,1524.0,5.960264900662252,42.34514998052201,0.6123927158888267,False,False,4.199999999999999,10.575139,8.686182,927.0,9.0,1,1.7161832713700895
-868,True,TTTCGATTCCGAGAAG-1,TTTCGATTCCGAGAAG-1,2.008966448093938,15.0,-0.04678429160923411,-0.030891768557708723,-5.420791,-2.58762,9,G1,1625.4862473756075,2,2,7007.0,2579.0,13.686313686313687,11.859569002426145,0.959927520721118,False,False,4.0,-16.170618,-2.930263,985.0,9.0,2,1.8487134598031176
-869,True,TTTCGATTCCTGCTAC-1,TTTCGATTCCTGCTAC-1,1.9603115358658523,15.0,-0.0770408588158617,-0.01344687302879709,1.7376593,18.634333,18,G1,1216.3890205323696,5,5,5807.0,1776.0,7.938694678835888,38.7807818150508,0.7235056831280424,False,False,9.000000000000002,-3.4221091,15.927235,783.0,9.0,5,1.26054803309828
-870,True,TTTCGATTCTCTAGGA-1,TTTCGATTCTCTAGGA-1,1.9778148053057394,15.0,-0.051016845639497994,-0.02739020176319023,-5.3302994,-4.2790384,17,G1,1786.0439134985209,2,2,10914.0,3333.0,7.669048927982408,15.869525380245555,0.9624274387504391,False,False,4.700000000000001,-14.7514,-4.531816,1843.0,9.0,2,1.7006603210004854
-871,True,TTTCGATTCTGTGCGG-1,TTTCGATTCTGTGCGG-1,2.2338719272078404,15.0,-0.042323489482747646,-0.02457908320019663,-7.8920197,-12.066318,16,G1,678.5019699335098,2,2,553.0,290.0,13.200723327305607,6.509945750452079,0.9357306342565149,False,False,5.9,-15.480569,-11.224276,632.0,9.0,2,0.9485778766191572
-872,True,TTTGTTGAGACGTCCC-1,TTTGTTGAGACGTCCC-1,2.027454786020684,15.0,-0.03272929838927921,-0.009741436391826452,9.341307,9.809969,15,G1,1038.162241205573,8,10,2036.0,981.0,28.68369351669941,7.170923379174853,0.6229043973488843,False,False,4.699999999999999,15.100548,11.025036,1113.0,10.0,8,0.8937778591737116
-874,True,TTTGTTGAGCACCTGC-1,TTTGTTGAGCACCTGC-1,2.0016144480294282,15.0,-0.0571163442666586,-0.03435010721624269,-6.068311,2.2698145,14,G1,1506.447009652853,4,4,8634.0,2853.0,6.636553161917998,13.458420199212416,0.9459527397869153,False,False,4.8,-14.5372925,1.1269181,2632.0,9.0,4,1.5993908315924596
-875,True,TTTGTTGAGCTCACTA-1,TTTGTTGAGCTCACTA-1,1.9736209768879647,15.0,-0.06479071304707006,-0.03400256284493233,-5.314267,-8.256545,13,G1,1169.837094038725,2,2,10706.0,3336.0,8.910891089108912,14.272370633289745,0.9528409114681339,False,False,3.0000000000000004,-16.47763,-7.799421,3107.0,9.0,2,0.9526172425755786
-876,True,TTTGTTGAGGGAGGTG-1,TTTGTTGAGGGAGGTG-1,2.0138076763775508,15.0,-0.07509019714854476,-0.02770749938230492,-7.732722,0.25982577,4,G1,1509.3713083565235,4,4,5533.0,2242.0,9.958431230797036,11.115127417314296,0.9694846788513838,False,False,4.8,-15.318817,-0.3536881,2133.0,9.0,4,1.5553820261544036
-877,True,TTTGTTGAGTGGCCTC-1,TTTGTTGAGTGGCCTC-1,1.9104197268944285,15.0,-0.11032744758679827,0.0037365960671110303,5.007131,6.5467052,12,S,1298.165541909635,1,1,4182.0,1270.0,7.388809182209469,45.2415112386418,0.6121347732725128,False,False,4.8,9.621283,6.9951615,1172.0,9.0,1,0.8054144824310264
-878,True,TTTGTTGCAAGAGAGA-1,TTTGTTGCAAGAGAGA-1,1.9019245679658685,15.0,-0.09791754972812704,-0.015742645667559795,2.6460757,4.100134,12,G1,1842.1758461147547,1,1,3389.0,1126.0,7.170256712894659,42.01829448214813,0.6133079864408679,False,False,4.2,11.495862,10.216359,1800.0,9.0,1,0.8054144824310264
-879,True,TTTGTTGCAAGTAGTA-1,TTTGTTGCAAGTAGTA-1,1.968728554080061,15.0,-0.08221014213225711,-0.02256306442708976,-8.123048,-3.0154252,4,G1,1697.3799031078815,4,3,11872.0,3368.0,8.372641509433961,16.307277628032345,0.9745183450741869,False,False,3.2,-14.509166,-3.350533,2405.0,9.0,4,0.8054144824310264
-881,True,TTTGTTGCAGGTTCAT-1,TTTGTTGCAGGTTCAT-1,1.9157857543664985,15.0,-0.076752258059448,-0.0290014820493784,7.9043465,-6.919629,11,G1,950.9687501788139,7,6,14878.0,4223.0,6.9565801855088045,17.838419142357843,0.3030912308265342,False,True,3.6,14.842916,-6.789481,2122.0,9.0,7,1.1231825187212259
-882,True,TTTGTTGGTAACGCGA-1,TTTGTTGGTAACGCGA-1,1.943312629895224,15.0,-0.0932854322139371,-0.014591335418098445,-8.301857,-3.7428346,10,G1,1434.5371922850609,3,3,8238.0,2842.0,7.853848021364409,15.768390386016023,0.970242342942307,False,False,3.0999999999999996,-14.013627,-3.7452972,4123.0,9.0,3,1.0447717255928088
-883,True,TTTGTTGTCATCGCTC-1,TTTGTTGTCATCGCTC-1,1.957806326704775,15.0,-0.06847312759986651,-0.016089844252105388,-6.030591,-1.6460114,9,G1,1472.9912744164467,2,2,16973.0,4248.0,5.962410887880751,19.324809992340775,0.9619333826362116,False,False,2.9000000000000004,-15.946691,-1.0646045,1441.0,9.0,2,1.2441389063273753
-884,True,TTTGTTGTCCAATCCC-1,TTTGTTGTCCAATCCC-1,1.9075688473238475,15.0,-0.10661440192694789,-0.015417194682835741,8.255973,1.1994703,8,G1,1101.9708822071552,9,9,7113.0,1939.0,9.475608041613945,37.48066919724448,0.5886626988052546,False,False,2.7,13.960351,5.5284853,1020.0,9.0,9,0.8054144824310264
-885,True,TTTGTTGTCCATTGCC-1,TTTGTTGTCCATTGCC-1,1.9504314105868148,15.0,-0.06561495271036624,-0.021414982917128764,-9.2922535,-2.7916152,7,G1,1550.6583121269941,3,3,11578.0,3219.0,6.555536362065987,16.773190533770943,0.9863150086644521,False,False,4.199999999999999,-15.460365,-3.6372786,2273.0,9.0,3,1.4568611182082167
-886,True,TTTGTTGTCCTACAAG-1,TTTGTTGTCCTACAAG-1,1.9251247027646166,15.0,-0.1145327893365303,0.0015631880818967638,5.6546464,4.15667,6,S,1612.6362722963095,1,1,7246.0,1800.0,9.205078664090532,43.058239028429476,0.610292287787697,False,False,4.5,12.751787,7.777711,954.0,9.0,1,1.4332994794648768
-887,True,TTTGTTGTCCTTCACG-1,TTTGTTGTCCTTCACG-1,2.029579151840361,15.0,-0.046387429305931836,-0.04263561169921185,-0.17627867,7.842231,5,G1,1249.5995611995459,1,1,1594.0,861.0,21.64366373902133,6.900878293601004,0.6584404132656012,False,False,4.3999999999999995,8.735247,13.291577,1036.0,9.0,1,1.4515487513862504
-888,True,TTTGTTGTCGACGCGT-1,TTTGTTGTCGACGCGT-1,1.9939973652769272,15.0,-0.0635679237988607,-0.016249270797186542,-6.9197288,2.4502618,4,G1,1479.2349963188171,4,4,5164.0,2090.0,13.129357087529048,13.381099922540667,0.951128413152932,False,False,5.300000000000001,-15.084896,1.495179,1729.0,9.0,4,1.4047135774338826
-889,True,TTTGTTGTCGCCACTT-1,TTTGTTGTCGCCACTT-1,1.9445120367680255,15.0,-0.0839984809134657,-0.02202600184367002,-5.9126496,-7.3615575,3,G1,1484.7777071297169,2,2,14166.0,3808.0,6.1979387265283075,16.476069462092333,0.9518926392563122,False,False,4.6000000000000005,-15.102908,-7.110796,2616.0,9.0,2,0.8054144824310264
-890,True,TTTGTTGTCGTAGAGG-1,TTTGTTGTCGTAGAGG-1,1.9710957290948403,15.0,-0.11861214520476623,-0.018631847448560477,-2.5691123,14.208008,2,G1,1241.0094733685255,5,5,6851.0,1744.0,11.210042329586921,41.789519778134576,0.7322556311436602,False,False,7.6,1.1887627,14.585867,824.0,9.0,5,1.3272521773497132
-891,True,TTTGTTGTCTCTAGGA-1,TTTGTTGTCTCTAGGA-1,1.9628245783546727,15.0,-0.06286823249838676,-0.028605509095688937,-9.306996,-3.4692883,1,G1,1559.950357273221,3,3,11197.0,3392.0,6.001607573457176,14.584263642046977,0.9871437083080061,False,True,3.2,-16.988478,-4.3092875,2439.0,9.0,3,1.8941064638830696
diff --git a/scarf/tests/datasets/knn_distances.npy b/scarf/tests/datasets/knn_distances.npy
deleted file mode 100644
index 27278c7f..00000000
Binary files a/scarf/tests/datasets/knn_distances.npy and /dev/null differ
diff --git a/scarf/tests/datasets/knn_indices.npy b/scarf/tests/datasets/knn_indices.npy
deleted file mode 100644
index e7e30175..00000000
Binary files a/scarf/tests/datasets/knn_indices.npy and /dev/null differ
diff --git a/scarf/tests/datasets/knn_weights.npy b/scarf/tests/datasets/knn_weights.npy
deleted file mode 100644
index 8ab43758..00000000
Binary files a/scarf/tests/datasets/knn_weights.npy and /dev/null differ
diff --git a/scarf/tests/datasets/markers_cluster1.csv b/scarf/tests/datasets/markers_cluster1.csv
deleted file mode 100644
index b4db3e65..00000000
--- a/scarf/tests/datasets/markers_cluster1.csv
+++ /dev/null
@@ -1,23 +0,0 @@
-,group_id,feature_name,feature_index,score,mean,mean_rest,frac_exp,frac_exp_rest,fold_change,p_value
-0,1,LEF1,8752.0,0.33122,0.2967,0.04111,0.73571,0.12725,7.21684,1.3468492583919777e-56
-1,1,CCR7,29040.0,0.31948,0.31051,0.05481,0.71429,0.1482,5.6654,1.303081203640238e-48
-2,1,PASK,5916.0,0.3078,0.08755,0.0126,0.27857,0.06587,6.94808,2.959313972605772e-15
-3,1,FAM153CP,11107.0,0.30597,0.09362,0.01476,0.3,0.08084,6.34314,9.01971057038007e-15
-4,1,CAMK4,10283.0,0.3032,0.12619,0.02217,0.44286,0.11078,5.69199,6.432216159402198e-24
-5,1,OXNAD1,6124.0,0.29315,0.2405,0.04098,0.60714,0.17814,5.86813,4.561360581110357e-32
-6,1,PDE3B,19299.0,0.28502,0.16879,0.03557,0.51429,0.18713,4.74522,4.607808821916747e-21
-7,1,RCAN3,547.0,0.28094,0.22673,0.0463,0.64286,0.21407,4.89703,1.1497416494441125e-31
-8,1,PIK3IP1,34866.0,0.27789,0.31654,0.06082,0.72143,0.26347,5.20433,3.204752190325667e-35
-9,1,PCED1B,21585.0,0.27194,0.0985,0.01763,0.31429,0.09132,5.58611,3.962141555008794e-14
-10,1,ANK3,18064.0,0.26989,0.05188,0.01045,0.21429,0.0494,4.9669,8.042095391236527e-12
-11,1,FHIT,6642.0,0.26883,0.06146,0.01148,0.26429,0.06437,5.35291,6.378882432475336e-14
-12,1,LDLRAP1,568.0,0.26668,0.1414,0.03687,0.52857,0.17066,3.83499,4.9506042495638135e-23
-13,1,TCF7,10484.0,0.26362,0.35899,0.06701,0.75714,0.22455,5.35724,1.18335482474118e-40
-14,1,TRABD2A,4310.0,0.26222,0.11259,0.0285,0.47857,0.1003,3.95082,4.288612022936954e-28
-15,1,TRAT1,6877.0,0.2622,0.1261,0.0252,0.42143,0.11527,5.00337,5.385352981156799e-20
-16,1,CHRM3-AS2,3269.0,0.2597,0.07844,0.01529,0.3,0.05988,5.13003,3.340274876456977e-18
-17,1,TBC1D4,23357.0,0.25868,0.05019,0.01044,0.2,0.07036,4.806,1.9327088042662477e-07
-18,1,PRKCQ-AS1,17615.0,0.25857,0.36278,0.07586,0.76429,0.21108,4.78202,6.628889398177008e-42
-19,1,TMEM204,26536.0,0.25476,0.05645,0.01091,0.27857,0.06287,5.17677,2.2915637229641518e-15
-20,1,NELL2,21563.0,0.25295,0.06383,0.01107,0.3,0.05389,5.76786,1.776913554122277e-19
-21,1,BCL11B,24832.0,0.2506,0.54998,0.12834,0.88571,0.26198,4.2854,1.6507082668410988e-48
diff --git a/scarf/tests/datasets/pseudotime_clusters.npy b/scarf/tests/datasets/pseudotime_clusters.npy
deleted file mode 100644
index 71a2743b..00000000
Binary files a/scarf/tests/datasets/pseudotime_clusters.npy and /dev/null differ
diff --git a/scarf/tests/datasets/pseudotime_markers_r_values.csv b/scarf/tests/datasets/pseudotime_markers_r_values.csv
deleted file mode 100644
index e4dbec64..00000000
--- a/scarf/tests/datasets/pseudotime_markers_r_values.csv
+++ /dev/null
@@ -1,10737 +0,0 @@
-,names,I__RNA_pseudotime__r
-6,AL627309.5,0.1230549217056546
-14,LINC01409,0.02697594296065394
-16,LINC01128,-0.06488185841951406
-24,NOC2L,-0.005567253956048132
-29,HES4,0.06671517782535455
-30,ISG15,0.23636363748713451
-42,TNFRSF18,-0.21417934390153354
-43,TNFRSF4,-0.15897884988130603
-44,SDF4,-0.039933597761962
-45,B3GALT6,-0.040967828909551175
-48,UBE2J2,-0.014120672687761925
-51,ACAP3,0.13718386481836156
-52,PUSL1,0.0068323665641420415
-53,INTS11,-0.02113976481616301
-55,CPTP,-0.06969205159494302
-57,DVL1,0.023184020126135272
-59,AURKAIP1,-0.029504706616703496
-60,CCNL2,0.045257420497981184
-61,MRPL20-AS1,-0.07333460013050093
-62,MRPL20,-0.05875118335163438
-70,ATAD3B,0.007633271807662926
-71,ATAD3A,0.019501792053005115
-73,SSU72,-0.00031632196672365924
-74,AL645728.1,-0.06300234164385085
-75,FNDC10,0.05980393758077972
-77,AL691432.2,0.029970291678462907
-78,MIB2,-0.24014760143053754
-80,CDK11B,-0.05215958375284411
-82,SLC35E2B,-0.05736428867890881
-83,CDK11A,-0.11250500821854219
-84,SLC35E2A,-0.041969797992089164
-85,NADK,0.3777765390075238
-86,GNB1,0.2506066088252328
-97,FAAP20,0.012513316457367767
-99,SKI,-0.0033973857506125623
-104,RER1,-0.09063835287590319
-105,PEX10,-0.0003860865930343741
-109,PANK4,0.06402702012057145
-111,AL139246.5,-0.11380095908753837
-112,TNFRSF14-AS1,-0.02571978055231639
-113,TNFRSF14,0.09163652045622896
-115,PRXL2B,-0.039661811088409285
-136,TPRG1L,0.03398381277567639
-137,WRAP73,-0.02672456817716085
-143,LRRC47,0.007645878335275087
-145,CEP104,0.06615876347907368
-146,DFFB,-0.05457637807639354
-147,C1orf174,-0.049844654360918395
-168,KCNAB2,-0.04808576132692299
-170,RPL22,-0.236210517827778
-173,ICMT,0.034638232433495564
-177,ACOT7,-0.09698521249600359
-182,TNFRSF25,-0.15352758246249806
-184,NOL9,0.0005619370160748977
-186,ZBTB48,0.024292385599682133
-187,KLHL21,-0.019966391627239993
-188,PHF13,-0.019463482818329037
-189,THAP3,0.0019628436548927773
-190,DNAJC11,0.004571992880053765
-194,CAMTA1,0.10346889792931625
-202,VAMP3,0.21812487861858856
-204,PER3,-0.04522825714047427
-208,PARK7,-0.10012148047778027
-216,RERE,0.22735127970261543
-219,ENO1,0.1391947774208346
-227,H6PD,0.045854756434876395
-232,SLC25A33,-0.025720955285396578
-234,PIK3CD,0.06495424022804244
-237,CLSTN1,-0.10360510382397015
-238,CTNNBIP1,-0.09460321131278472
-241,LZIC,-0.005256521828738424
-243,NMNAT1,0.07758944078179397
-245,RBP7,0.3554503453866769
-246,UBE4B,0.08170302928451648
-247,KIF1B,0.17589793182073735
-250,PGD,0.4400119901074925
-254,DFFA,0.05948297214194947
-256,PEX14,-0.02054688471217005
-258,CASZ1,0.04452592790114171
-261,TARDBP,-0.053910779294433095
-263,AL109811.2,-0.08985000133016134
-264,SRM,-0.0933477759816086
-265,EXOSC10,-0.016006769020427632
-268,MTOR,0.03613929608744564
-271,UBIAD1,0.04181058517497577
-278,FBXO44,-0.143642892475068
-279,FBXO6,-0.008588845313193313
-280,MAD2L2,-0.08598847881205167
-282,AGTRAP,0.2985823142605977
-285,MTHFR,0.13992756662582434
-286,CLCN6,0.13029517829098325
-289,AL021155.5,0.2118140977687788
-290,KIAA2013,0.07206094552080566
-291,PLOD1,0.11796094290853949
-293,MFN2,0.11952713801693134
-294,MIIP,0.11659202671286495
-295,TNFRSF8,0.1435668960400651
-297,TNFRSF1B,0.47907507726847964
-298,VPS13D,0.02450165123222427
-300,DHRS3,-0.3001979867946025
-338,PRDM2,0.026335074461534404
-344,TMEM51,0.19520669774466087
-350,EFHD2,-0.04198061434469568
-354,CASP9,-0.007058581171555149
-355,DNAJC16,0.0858229787284598
-357,AGMAT,-0.07363005354763731
-359,DDI2,-0.012351059041843245
-362,PLEKHM2,0.11798320093760765
-369,AL450998.2,-0.08571108857869655
-370,SPEN,0.11381785156085569
-371,ZBTB17,0.03377566474099116
-385,FBXO42,0.08648134145750665
-386,SZRD1,-0.033362919469690396
-388,NECAP2,-0.03635147647835925
-393,NBPF1,-0.04882800173144501
-398,CROCC,-0.07694291369395005
-401,BX284668.5,0.022571240396238732
-406,ATP13A2,-0.03738901234356289
-407,SDHB,0.05017363273521305
-408,PADI2,0.1373171029292057
-413,PADI4,0.23251315570147651
-416,RCC2,0.13444352641100238
-417,ARHGEF10L,0.27667702832331054
-429,IFFO2,-0.11203573504000285
-431,UBR4,0.09485488144299467
-433,EMC1,0.015196811205233338
-434,MRTO4,-0.012466719343090384
-436,AKR7A2,-0.041727984072374484
-437,SLC66A1,0.09528782708719437
-438,CAPZB,0.08932824478362296
-439,MICOS10,-0.01386066241508532
-443,TMCO4,-0.02853236100540018
-446,OTUD3,-0.06596742216326937
-465,MUL1,0.0058776915639362996
-467,CDA,0.37783580502154923
-468,PINK1,0.13922519884111875
-470,DDOST,-0.13505339050878262
-474,HP1BP3,-0.010565498951759492
-475,EIF4G3,0.048648410701711185
-477,ECE1,0.1545263184326597
-485,USP48,0.1355832894754495
-491,LINC00339,0.044434902727545335
-492,CDC42,0.05970505965291489
-494,CDC42-IT1,0.11079686982431751
-497,ZBTB40,-0.06667110701980225
-507,KDM1A,-0.007217550325998549
-509,LUZP1,-0.09447161858203226
-511,LINC01355,-0.009535755163188747
-513,HNRNPR,0.04513786171893914
-517,TCEA3,-0.0723311604405331
-521,ID3,-0.07075408554332689
-523,MDS2,-0.0624758255795342
-524,RPL11,-0.3521296662998126
-526,ELOA,0.04975837717528601
-527,PITHD1,-0.16402276074599387
-528,LYPLA2,-0.021297180180469135
-529,GALE,0.011795680860376063
-530,HMGCL,-0.041319989705999725
-531,FUCA1,0.1365353249929212
-532,CNR2,-0.04311906877377416
-534,PNRC2,-0.03739976099394955
-535,SRSF10,-0.03870922905211816
-545,NIPAL3,-0.11299906558104254
-547,RCAN3,-0.1546415624599943
-548,NCMAP-DT,-0.07271399828589718
-550,SRRM1,-0.06357989463992358
-552,CLIC4,0.2505700865392326
-553,RUNX3,-0.443379061726743
-559,SYF2,0.013113557526173039
-562,RSRP1,0.07523694483873336
-565,TMEM50A,0.05938580537035053
-567,MACO1,-0.0977749993723366
-568,LDLRAP1,-0.13546153921067158
-570,MAN1C1,-0.06109922662660458
-571,AL031280.1,-0.05276257299899762
-572,SELENON,0.09269587918026835
-574,MTFR1L,0.065471317426969
-578,PAQR7,0.08276066475676322
-579,STMN1,-0.15678612376166642
-580,PAFAH2,-0.12707976235154347
-585,PDIK1L,0.008603173774090778
-589,ZNF593,0.0345500296356332
-592,CEP85,-0.0203318387036094
-593,SH3BGRL3,0.11805054308154371
-594,UBXN11,0.19531207547041204
-595,CD52,-0.2859481438086418
-599,DHDDS,0.046911952631789905
-601,HMGN2,0.015894416800225814
-602,RPS6KA1,0.1341334836714293
-604,ARID1A,0.19969715471983238
-605,PIGV,-0.0013645436924121595
-606,ZDHHC18,-0.04197998375283504
-608,GPN2,0.060770212078623674
-610,GPATCH3,0.0443430456718138
-611,NUDC,-0.16531331508572075
-616,SLC9A1,0.045098465474636244
-618,WDTC1,-0.018210496908345957
-619,TMEM222,-0.1396495053789517
-620,SYTL1,-0.3043601546708943
-621,MAP3K6,-0.05117118588239713
-625,WASF2,0.18987379362159923
-628,AHDC1,-0.0590461543721751
-629,FGR,0.42475906254103013
-631,IFI6,0.333186637849025
-634,FAM76A,-0.03839817624399984
-635,STX12,0.2404357866351109
-639,PPP1R8,-0.05479117674744419
-640,THEMIS2,0.4007152306320804
-641,RPA2,-0.2954480735175474
-644,XKR8,0.08750986952815397
-645,EYA3,0.012092667850428788
-646,PTAFR,0.41544314993122294
-647,DNAJC8,-0.12634769603595938
-648,ATP5IF1,-0.3213303137954154
-649,AL353622.1,-0.08396144513454108
-651,SESN2,0.0492211928820459
-652,MED18,-0.042886918170366294
-653,PHACTR4,0.05081690291893618
-654,RCC1,0.04433811240705384
-655,SNHG3,-0.0908858955753079
-656,TRNAU1AP,-0.12519955633062738
-657,SNHG12,-0.16567442822269965
-658,TAF12,-0.052789562928356244
-661,AL360012.1,0.10478296033865367
-662,GMEB1,-0.03805970938047825
-663,YTHDF2,0.030946087428810144
-666,EPB41,-0.2153544543816789
-669,SRSF4,0.055411410681447004
-670,MECR,0.0036921106172176027
-681,MATN1-AS1,-0.038909331258823815
-683,LAPTM5,0.4679250936659717
-687,SDC3,0.13638723709006054
-688,PUM1,0.21661077512036042
-690,SNRNP40,-0.10983494996911104
-691,ZCCHC17,-0.029476186162603703
-699,PEF1,-0.04820455599839965
-708,PTP4A2,0.23356626881075868
-711,KHDRBS1,0.11244545932092129
-713,TMEM39B,0.07029023545338424
-714,KPNA6,0.049332727871123895
-716,TXLNA,0.01727459129312241
-717,CCDC28B,-0.2593965388814996
-721,TMEM234,-0.03346104239695377
-722,EIF3I,-0.09509860821216531
-724,LCK,-0.5051305776916664
-725,HDAC1,-0.1273041572505737
-726,MARCKSL1,0.009874320863173346
-730,BSDC1,-0.03265118353167306
-733,ZBTB8OS,-0.09063856688000357
-734,RBBP4,0.08154028097827945
-735,SYNC,-0.11583267395200326
-738,YARS,-0.044084655044928096
-739,S100PBP,-0.11940850106959405
-744,RNF19B,0.2297778434021784
-746,AK2,0.1353866446983729
-750,ZNF362,0.09899506971809227
-754,PHC2,0.15142815141994492
-762,SMIM12,0.01369673908911601
-770,TMEM35B,-0.2428063657375646
-771,ZMYM6,0.12480569910449067
-772,ZMYM1,-0.0024400541977930602
-774,SFPQ,0.13612612993620762
-775,ZMYM4,0.12354360535253361
-777,KIAA0319L,0.16819319865842008
-778,NCDN,0.054407980509276886
-779,AC004865.2,-0.14652531514469286
-781,PSMB2,0.054687176724772045
-782,C1orf216,-0.1291107641125408
-785,AGO4,0.3096421783111556
-786,AGO1,0.11630040401184202
-789,AGO3,0.10565962888205979
-792,ADPRHL2,0.03131540761274148
-793,COL8A2,0.13310814576698693
-794,TRAPPC3,-0.042783151968809725
-795,MAP7D1,-0.01873385505899189
-796,THRAP3,-0.10663277815072107
-798,EVA1B,0.0866028233653545
-800,STK40,0.09130010089409536
-801,LSM10,0.0765447131511328
-803,MRPS15,-0.008835141781899059
-804,CSF3R,0.47582557218353894
-812,ZC3H12A,-0.025846121989861387
-813,MEAF6,-0.18295529910853137
-814,SNIP1,-0.06727665190497625
-816,GNL2,0.04362277410521528
-819,C1orf109,-0.09753718506652714
-824,YRDC,-0.09098009807422307
-825,C1orf122,-0.022112928580704076
-826,MTF1,0.2103586886500932
-828,INPP5B,0.02001845868562048
-829,SF3A3,0.014406568206325493
-830,FHL3,0.09395212917496838
-831,UTP11,0.058845182653001996
-840,RRAGC,0.11156200179882095
-842,MYCBP,0.1444173490634703
-845,AKIRIN1,-0.016466751358024133
-846,NDUFS5,-0.010276694420487004
-847,MACF1,-0.1596830646954101
-852,PABPC4,0.14099640859043872
-858,PPIE,-0.05975264866179944
-865,TRIT1,0.12568773772199582
-866,MYCL,0.25104435123173013
-868,MFSD2A,0.15427884316991602
-869,CAP1,0.29972506709527996
-870,PPT1,0.47124542163685446
-871,RLF,0.10199497539192409
-873,AL050341.2,-0.026682253912128324
-874,ZMPSTE24,0.1409732880330726
-875,COL9A2,0.15403674716452606
-876,SMAP2,0.3283344183536616
-880,ZFP69,-0.09571657298990982
-882,EXO5,-0.007230376951609572
-885,ZNF684,0.03358600257179506
-891,NFYC,0.10164193001459686
-893,CITED4,0.0671532015304515
-896,CTPS1,-0.009482371046693593
-899,SCMH1,-0.006574362503734779
-905,HIVEP3,0.030880578831552933
-911,FOXJ3,-0.11705452427200572
-915,PPCS,0.05503323930736351
-916,CCDC30,0.039944925314208936
-917,PPIH,-0.0400107740439354
-920,YBX1,0.2331984163639987
-922,P3H1,0.07453255867459331
-923,C1orf50,-0.08665257411753087
-926,SVBP,0.0515949412664668
-927,ERMAP,0.1337533285394462
-929,ZNF691,-0.05423299666950909
-930,SLC2A1,-0.1778396979504289
-934,EBNA1BP2,-0.09828858473076153
-943,ELOVL1,-0.01715667661499036
-944,MED8,-0.05799311093153786
-946,SZT2,0.08845231213928195
-948,HYI,0.021303071367521617
-951,KDM4A,0.007750325510282202
-953,ST3GAL3,-0.02473636082489289
-957,IPO13,0.06429319488868504
-959,DPH2,-0.01887336098927653
-960,ATP6V0B,0.443302897104421
-967,DMAP1,-0.04350089222311067
-968,ERI3,-0.049605732310712136
-970,RNF220,0.02523068446703389
-972,ARMH1,0.01681325468282223
-975,RPS8,-0.2885976428911654
-977,PLK3,0.18626530558304363
-980,PTCH2,0.13418766076727154
-981,EIF2B3,-0.039138482790820335
-982,HECTD3,0.06099141970403076
-983,UROD,0.054041784768249834
-987,MUTYH,-0.06434839777990788
-988,TOE1,-0.0923645041874432
-989,TESK2,0.09681289816837337
-992,PRDX1,0.12519723911666208
-993,AKR1A1,0.15040454677555024
-994,NASP,-0.025558913454575078
-996,GPBP1L1,0.062343207846760985
-997,TMEM69,0.0530467414137923
-998,IPP,-0.07626732315920767
-1000,MAST2,0.10202662452714799
-1006,POMGNT1,-0.046330196126079644
-1009,LRRC41,0.03501280236425198
-1010,UQCRH,-0.032844634720913875
-1011,NSUN4,0.10587899156409974
-1012,FAAH,0.1010766404625462
-1018,MKNK1,0.04853177818365361
-1019,MOB3C,0.08893456036103724
-1020,ATPAF1,-0.004066493768567328
-1023,EFCAB14,0.07602766451248576
-1035,CMPK1,0.0012916334326167204
-1049,SPATA6,0.1246181054072658
-1052,BEND5,-0.00955397115151429
-1064,FAF1,-0.07116391230445514
-1066,CDKN2C,0.0014144130834979418
-1069,RNF11,0.06246464199054348
-1073,EPS15,0.1410045347063142
-1076,OSBPL9,0.07742288060247088
-1077,NRDC,-0.023256606583312578
-1081,TXNDC12,-0.0009449990125919163
-1082,KTI12,-0.04527880425174545
-1085,BTF3L4,0.14844410630317828
-1087,CC2D1B,0.06256685192176688
-1091,PRPF38A,-0.11039641872673985
-1092,TUT4,-0.24507557300975935
-1094,GPX7,-0.11596291754669838
-1095,SHISAL2A,-0.1677011635017399
-1096,COA7,0.061975290665355266
-1097,ZYG11B,0.07909048850700386
-1100,ECHDC2,-0.1314668856512397
-1101,SCP2,-0.09057797001358026
-1106,CPT2,0.11366083312481434
-1108,CZIB,-0.08376016732814294
-1110,MAGOH,-0.1866368534115611
-1112,LRP8,0.0257608418375831
-1119,LINC02812,0.009279852303847261
-1122,NDC1,0.0344100920185222
-1123,YIPF1,0.09828560500743282
-1125,HSPB11,-0.1743519216965921
-1126,LRRC42,-0.013176444210681868
-1128,TMEM59,-0.10451064508961974
-1129,TCEANC2,0.09901024894936104
-1133,MRPL37,0.05143282698286113
-1134,SSBP3,-0.16879238009449665
-1144,TTC4,-0.06268326975161274
-1158,USP24,0.028682784058302036
-1177,OMA1,-0.0033358509274704923
-1180,MYSM1,0.015969381795219444
-1182,JUN,0.17302807853647978
-1194,HOOK1,-0.06492444063724286
-1200,NFIA,0.07857369488388166
-1205,TM2D1,0.02044531563015189
-1207,PATJ,-0.11912565494359523
-1210,USP1,-0.013546011283409638
-1211,DOCK7,0.03292647946419866
-1214,ATG4C,0.05427593296523082
-1225,ALG6,-0.03155681713450471
-1226,ITGB3BP,-0.0015932796199634237
-1230,PGM1,0.021121173652675938
-1237,JAK1,-0.34740017651946964
-1243,LEPROT,0.24197799687232788
-1247,PDE4B,-0.01707736008319981
-1255,MIER1,-0.020165101226946647
-1256,SLC35D1,-0.13687504872128373
-1261,SERBP1,-0.023938849952812435
-1263,GADD45A,-0.0032990416727488392
-1267,WLS,0.20800104481321738
-1282,LRRC40,-0.05493551620636063
-1283,SRSF11,-0.012560099802708335
-1285,ANKRD13C,0.06008367441392752
-1296,ZRANB2,-0.19056029390311033
-1309,FPGT,0.08818669165704399
-1317,CRYZ,-0.1321077867094611
-1318,TYW3,-0.019571042315263146
-1325,ACADM,0.01435775165835368
-1326,RABGGTB,0.040324105773069996
-1336,PIGK,-0.07806500507221757
-1338,AK5,-0.20843807640171724
-1341,AC118549.1,-0.024320200975218458
-1342,USP33,0.09864250692309813
-1343,MIGA1,-0.0455560268593408
-1345,NEXN,0.16677504094961393
-1346,FUBP1,-0.05836851139080973
-1347,DNAJB4,-0.00339277528733327
-1349,AC103591.3,0.15735806081551837
-1353,IFI44L,0.3711302764621523
-1354,IFI44,0.3176279459708511
-1363,LINC01781,0.0023198944259790436
-1380,PRKACB,-0.2939579677030131
-1384,RPF1,0.004090458107404308
-1385,GNG5,0.2096155130025133
-1387,CTBS,0.18811612213587883
-1391,SSX2IP,-0.056520620429776895
-1394,MCOLN2,-0.08071082658680018
-1398,C1orf52,-0.03074797426699041
-1399,BCL10,0.07941219693710325
-1407,ZNHIT6,-0.1365869382564246
-1410,ODF2L,-0.19670874297287622
-1416,SH3GLB1,0.08096949321535866
-1418,SELENOF,-0.007795034541650271
-1419,HS2ST1,0.0044825080046648254
-1423,LMO4,0.08979342870561986
-1427,PKN2,0.18748452500106594
-1428,GTF2B,-0.013183305851795935
-1429,KYAT3,0.034803642363612296
-1430,RBMXL1,-0.008284572822916704
-1431,GBP3,0.09720532035249377
-1432,GBP1,0.3452450852926635
-1433,GBP2,0.3842171606000256
-1436,GBP4,0.2815849589682886
-1437,AC099063.4,-0.11368677068902676
-1438,GBP5,0.2148605754384967
-1442,LRRC8B,0.13903818375912283
-1443,LRRC8C-DT,-0.031858937725204255
-1445,LRRC8C,0.12869878397355683
-1448,LRRC8D,0.17409105524279866
-1450,ZNF326,0.006280514757241575
-1461,ZNF644,-0.037404425867191525
-1464,TGFBR3,-0.3882886524779984
-1468,BTBD8,0.016618276547262802
-1471,GLMN,-0.16769656701719501
-1472,RPAP2,-0.17054093115556165
-1474,EVI5,0.3270945056160331
-1475,RPL5,-0.40871562680410606
-1476,DIPK1A,-0.09176863521795961
-1478,MTF2,-0.007298385309597535
-1479,TMED5,-0.004971397653712314
-1480,CCDC18,0.06141494126681908
-1481,CCDC18-AS1,-0.04965677191389404
-1482,DR1,0.21147914981841964
-1487,DNTTIP2,0.03625851701669107
-1488,GCLM,-0.08431685828155337
-1495,ABCD3,-0.0795500226354738
-1501,ALG14,-0.04841466236474992
-1506,RWDD3,-0.09622041185693504
-1517,PTBP2,0.0469629302625801
-1518,DPYD,0.42093917703796907
-1533,AGL,-0.08797839003289
-1535,SLC35A3,0.0008301271666048649
-1536,MFSD14A,0.006444876810803581
-1538,SASS6,-0.05279770415367116
-1539,TRMT13,-0.08198597956628349
-1541,DBT,0.007177681428351698
-1543,RTCA-AS1,-0.005616975329124829
-1544,RTCA,-0.007978571026859296
-1546,CDC14A,-0.22848396969594262
-1554,SLC30A7,0.05069320517747298
-1555,DPH5,-0.06570813626434331
-1557,AC093157.1,-0.07494748785469671
-1559,S1PR1,-0.2789643669697766
-1568,RNPC3,-0.030718090568228166
-1579,PRMT6,-0.05600066592478537
-1581,VAV3,-0.034130438428482
-1583,LINC02785,0.14030445998429134
-1584,SLC25A24,0.19941852070687732
-1590,FAM102B,0.15704029117160623
-1591,HENMT1,-0.2689955328157962
-1593,PRPF38B,-0.05953467960000291
-1596,STXBP3,0.15170639908596212
-1599,GPSM2,-0.04897909357660194
-1600,CLCC1,0.07019459452104725
-1601,WDR47,0.02561704876145906
-1602,TAF13,0.040861776965014945
-1604,TMEM167B,0.19430794902606022
-1608,SARS,-0.09259653454824265
-1610,PSRC1,0.07198337826651541
-1612,SORT1,0.39178755007903593
-1613,PSMA5,-0.09406513953471722
-1616,CYB561D1,-0.04673108425680487
-1620,GNAI3,0.12653083304636528
-1623,AMPD2,0.16160137362618554
-1625,GSTM4,0.024584267797337054
-1626,GSTM2,-0.04788791597569748
-1631,GSTM3,-0.011592687260680391
-1637,AHCYL1,0.17203308213531657
-1638,STRIP1,0.09552756211274767
-1649,RBM15,-0.007477965111677607
-1650,LAMTOR5-AS1,-0.03219690439689876
-1651,SLC16A4,-0.1216738372603351
-1652,AL355488.1,-0.0016838267738990538
-1653,LAMTOR5,0.054566764431779186
-1660,AL365361.1,-0.12803915775743088
-1661,KCNA3,-0.04688197748573514
-1662,CD53,-0.012033041593474353
-1665,LRIF1,-0.013335750290024352
-1667,DRAM2,0.13626764958487877
-1668,CEPT1,0.14355736191537305
-1670,AL355816.2,-0.07066290090154864
-1671,DENND2D,-0.2609870158109731
-1672,CHI3L2,-0.1642259535439208
-1679,WDR77,0.043476972105719616
-1680,ATP5PB,0.05076359413922439
-1681,C1orf162,0.32358069131373585
-1684,RAP1A,0.0648204341243554
-1686,INKA2,0.12976797528194542
-1689,DDX20,0.01571916120808659
-1695,CTTNBP2NL,0.11808386363050709
-1698,ST7L,-0.07055033741854601
-1699,CAPZA1,0.24801354494737946
-1700,MOV10,0.005570535761832372
-1702,RHOC,-0.09836513341351112
-1708,SLC16A1,0.01977664646162683
-1709,SLC16A1-AS1,-0.004133305562075322
-1712,LRIG2,0.02439341004747944
-1714,PHTF1,-0.008778105460234855
-1715,RSBN1,-0.10157379024789476
-1717,PTPN22,-0.2617828463659572
-1720,AP4B1,-0.046857163519514605
-1721,DCLRE1B,0.09871411248439374
-1723,HIPK1,0.12616833798293883
-1728,TRIM33,0.027473447984524054
-1729,BCAS2,-0.2249348311907223
-1732,NRAS,0.12593563696865961
-1733,CSDE1,0.25285887303437243
-1734,SIKE1,-0.029430252612593844
-1737,TSPAN2,-0.1400589886846058
-1747,SLC22A15,0.19020909409331463
-1752,ATP1A1,0.06946492102908475
-1758,CD58,0.06525207555902474
-1761,CD2,-0.5266817468161709
-1764,CD101,0.20343048492344443
-1766,TTF2,0.0005019683276468543
-1771,MAN1A2,0.034123043802422336
-1773,TENT5C,-0.014318861365760108
-1774,GDAP2,0.16932417788174703
-1775,WDR3,-0.01589971377240118
-1781,WARS2,-0.025428709984878
-1783,WARS2-AS1,-0.04212085717849835
-1791,ZNF697,0.12483226137362713
-1796,NOTCH2,0.4089965092621748
-1798,SEC22B,0.04171545223585034
-1800,NBPF26,0.15122286685324252
-1801,AC253572.2,0.2986804356320298
-1803,LINC00623,-0.2625158459883076
-1804,FCGR1B,0.3552205391869643
-1809,SRGAP2C,0.16080571026469473
-1817,LINC02591,-0.0765436503762333
-1824,AC246785.3,-0.05939496089189511
-1825,NBPF15,-0.02269229530885977
-1829,SRGAP2B,0.26392149067081144
-1832,AC245014.3,0.204468411095803
-1836,GPR89A,-0.047921651623902266
-1838,CD160,-0.4951034362314438
-1839,RNF115,-0.03081419986969479
-1840,POLR3C,0.08751987705660158
-1842,PIAS3,0.013175344665962159
-1846,PEX11B,0.004207484145735683
-1847,RBM8A,-0.12444921121517695
-1849,LIX1L,-0.044890131801122536
-1852,POLR3GL,-0.2548760056962633
-1853,TXNIP,-0.24210423614979326
-1855,AC239799.2,0.0634334418648291
-1857,NBPF10,0.12048690528121034
-1858,NOTCH2NLA,0.16453720979387315
-1861,AC245407.2,-0.11947067215730695
-1862,NBPF12,0.055632387021188175
-1864,PRKAB2,-0.0065533442302265115
-1865,AC242426.2,0.002825090664763407
-1866,FMO5,0.08214312810076006
-1869,BCL9,-0.05028112375203732
-1876,GPR89B,-0.08353380783317926
-1879,NBPF11,0.12609815162116803
-1883,LINC01138,-0.1749206619066541
-1888,NBPF14,0.2257552393934731
-1891,PDE4DIP,-0.016118141731702028
-1895,NBPF9,0.18769652811402443
-1896,AC245297.3,-0.1354711362358376
-1898,NOTCH2NLC,0.08212060214956937
-1899,NBPF19,0.12812289911578142
-1902,FCGR1A,0.43532475104214685
-1903,HIST2H2BF,-0.03729379313211941
-1913,HIST2H2BE,0.056901789297599745
-1914,HIST2H2AC,-0.05654517395153597
-1915,HIST2H2AB,-0.14449138908436177
-1916,BOLA1,-0.12978335393237952
-1918,SF3B4,0.004470151857384013
-1919,MTMR11,0.26113387315503706
-1922,VPS45,0.07332625068950144
-1923,PLEKHO1,0.5035809583724987
-1925,ANP32E,-0.1448910181400107
-1928,APH1A,0.11889163030811739
-1931,MRPS21,-0.1477068320927416
-1932,PRPF3,-0.046749354700098486
-1933,RPRD2,-0.07684441423228272
-1934,TARS2,-0.09294316925868414
-1938,ADAMTSL4,0.22561964408183047
-1939,ADAMTSL4-AS1,0.12324593615304487
-1940,MCL1,0.45410926181443145
-1941,ENSA,-0.14670010229929212
-1942,GOLPH3L,-0.11232339432405065
-1944,CTSS,0.7016822384868344
-1946,ARNT,0.12235242872920422
-1948,SETDB1,0.13972906723236872
-1949,CERS2,0.0946153830700544
-1953,MINDY1,0.07831826555655008
-1954,PRUNE1,0.015155368313324855
-1956,C1orf56,-0.20452132747565646
-1957,CDC42SE1,-0.19397441252160788
-1958,MLLT11,-0.051638691297096216
-1959,GABPB2,-0.07900349480879348
-1962,TNFAIP8L2,0.17242359258641593
-1963,SCNM1,0.14902543763491177
-1966,VPS72,-0.04640002607605422
-1967,PIP5K1A,0.035279485370556
-1968,PSMD4,0.04430310918092479
-1970,ZNF687,0.033404100282053754
-1971,PI4KB,0.10213972351438531
-1973,RFX5,0.03521461839861062
-1977,PSMB4,0.1012183517334879
-1978,POGZ,0.0973029162487224
-1982,SNX27,0.35976951818733477
-1988,MRPL9,-0.20609886985516268
-2001,THEM4,-0.18269974958141202
-2004,S100A10,0.19870528705485824
-2006,S100A11,0.536212225924074
-2059,S100A9,0.5074430864285135
-2060,S100A12,0.40200166667951426
-2061,S100A8,0.4765702106780999
-2066,S100A6,0.5399221492483436
-2068,S100A4,0.2932608217106712
-2077,CHTOP,0.05779476378190596
-2078,SNAPIN,0.019168281098890315
-2079,ILF2,0.05608711831287201
-2081,INTS3,0.06674777206914573
-2084,SLC27A3,0.12471434634770238
-2085,GATAD2B,0.004875924208183606
-2087,DENND4B,0.11132729511041824
-2088,CRTC2,0.07374641216329506
-2089,SLC39A1,0.0908701527519854
-2093,JTB,-0.07881385870807925
-2095,RAB13,0.15789865143941761
-2096,RPS27,-0.49257792739220135
-2098,TPM3,0.31019575411692885
-2100,C1orf43,0.12506035358677903
-2101,UBAP2L,0.0507413123925315
-2102,HAX1,-0.13077926931336073
-2104,ATP8B2,-0.18498202039051417
-2106,IL6R,0.3786466269332097
-2110,UBE2Q1,0.03336837878167554
-2115,ADAR,0.23997727247682285
-2118,PMVK,0.01916927035246405
-2120,PBXIP1,-0.18518987106886758
-2121,PYGO2,0.000350504509218759
-2123,SHC1,0.08051180991311827
-2124,CKS1B,-0.006346006852151962
-2125,FLAD1,-0.06676813530462025
-2127,ZBTB7B,0.1937344392317168
-2131,ADAM15,0.2698429998196673
-2132,EFNA4,0.017678175173785653
-2135,SLC50A1,-0.02381871369881321
-2136,DPM3,-0.1487236991814259
-2137,KRTCAP2,-0.34959091963485306
-2142,MTX1,0.0702802280728636
-2144,GBA,0.08548295597950745
-2145,FAM189B,0.06410095901777403
-2146,SCAMP3,-0.053420844463528516
-2147,CLK2,0.03290016710382531
-2150,FDPS,-0.18936473510429577
-2152,RUSC1,0.06808246497822917
-2153,ASH1L,0.029922077661316816
-2155,ASH1L-AS1,-0.07522566647440038
-2157,MSTO1,0.00035104913729497626
-2158,AL353807.5,0.15508956394962561
-2159,YY1AP1,0.16689286889271882
-2160,DAP3,0.01991744758431806
-2162,GON4L,-0.036525003544427885
-2163,SYT11,-0.16860317897762028
-2164,RIT1,0.10349327932433873
-2165,KHDC4,0.0422320880936085
-2167,ARHGEF2,0.1045097225971159
-2171,SSR2,-0.25070702515277415
-2172,UBQLN4,-0.06268489532515444
-2173,LAMTOR2,0.16372942043983033
-2176,LMNA,0.09418577461310396
-2177,SEMA4A,0.30737722757855546
-2178,SLC25A44,0.0847269952331866
-2179,PMF1,-0.07148415264766075
-2182,SMG5,0.1329782847509423
-2183,TMEM79,0.0288319434873534
-2184,GLMP,0.21195185373623224
-2186,CCT3,-0.19757054724937154
-2191,MEF2D,0.2022872030608336
-2195,NAXE,-0.03592580600654372
-2196,GPATCH4,-0.08265738669012052
-2208,ISG20L2,0.013252835097596833
-2209,RRNAD1,0.06116424414000072
-2210,MRPL24,0.01018196975219451
-2211,HDGF,0.1932889802417052
-2212,PRCC,0.007149244776527536
-2214,SH2D2A,-0.2695707108933275
-2219,ARHGEF11,0.26651935844426017
-2221,ETV3,0.03490235604361974
-2226,FCRL3,-0.20306752676376374
-2228,FCRL2,0.0021332422619659203
-2229,FCRL1,0.003260296669874765
-2235,CD1D,0.3880787716406388
-2238,CD1C,0.039309488838070665
-2256,MNDA,0.6103464035725646
-2257,PYHIN1,-0.46446402106841145
-2258,IFI16,0.28799312532403404
-2259,AIM2,0.11261876797872315
-2270,DUSP23,0.0255029204554246
-2271,FCRL6,-0.4813954644286315
-2272,SLAMF8,0.04624542381872483
-2279,TAGLN2,0.11058681980376629
-2284,PIGM,0.0818174715772197
-2287,IGSF8,-0.06593090157630456
-2292,PEA15,0.3091459412331339
-2293,DCAF8,0.07009558422977734
-2295,PEX19,-0.05275311018735095
-2296,COPA,0.21163562775555472
-2297,NCSTN,0.24063169801346934
-2300,SLAMF6,-0.2838325890442219
-2302,CD84,-0.020470458921117086
-2303,SLAMF1,-0.13014205753368674
-2305,CD48,-0.18166784976267028
-2306,SLAMF7,-0.10734083005559832
-2307,LY9,-0.23279420352094643
-2308,CD244,-0.0799922686481144
-2313,F11R,0.059095283046884474
-2314,TSTD1,-0.3562796528538796
-2315,USF1,0.16430394066957113
-2316,ARHGAP30,-0.008954666478600807
-2320,PFDN2,-0.06931157372193703
-2321,NIT1,0.13454206573391478
-2322,DEDD,0.09065258926315638
-2323,UFC1,-0.19065077817485168
-2325,USP21,-0.06413586595843046
-2326,PPOX,-0.09507901908891667
-2327,B4GALT3,-0.09531757609884006
-2329,NDUFS2,0.08819974195614023
-2330,FCER1G,0.2607122945415529
-2332,TOMM40L,0.058199777115089765
-2336,SDHC,0.09127013667871443
-2339,AL592295.5,-0.058755705286668417
-2342,FCGR2A,0.4593963961128281
-2343,HSPA6,0.233480061049543
-2344,FCGR3A,-0.0866751069698364
-2347,FCGR2B,0.10018697425120651
-2349,FCRLA,0.0077574604543496125
-2351,DUSP12,-0.035307786765881054
-2353,ATF6,0.17396640582183373
-2361,SH2D1B,-0.32208325088305384
-2362,UHMK1,-0.0030093162884387787
-2364,UAP1,-0.16826555439903954
-2367,HSD17B7,-0.017539997011195654
-2386,MGST3,-0.06747917179805098
-2387,ALDH9A1,-0.09526376487182348
-2389,TMCO1,-0.1686291586202359
-2391,UCK2,0.07758930624507468
-2399,POGK,0.10113262556452483
-2400,TADA1,0.05459198315985523
-2408,POU2F1,0.04452830775678509
-2410,CD247,-0.581358808773943
-2414,CREG1,0.28045813067953285
-2416,RCSD1,0.2454420969241478
-2417,MPZL1,0.17472890809005834
-2420,MPC2,-0.18897351022546624
-2421,DCAF6,0.06855878501632345
-2423,TIPRL,-0.03403461669123291
-2424,SFT2D2,0.1091208966596912
-2425,TBX19,0.08965819909234402
-2428,XCL2,-0.3814920766751192
-2437,ATP1B1,0.06073766206605843
-2438,NME7,-0.13567365333903478
-2440,BLZF1,0.035448013477034475
-2442,SLC19A2,0.004802856522455602
-2444,F5,0.23010572923522174
-2447,SELL,0.20437501705633612
-2450,METTL18,-0.17636009612853087
-2451,SCYL3,0.0441387856953231
-2452,KIFAP3,-0.1633688968564747
-2458,GORAB,-0.02749608912653107
-2470,PRRC2C,-0.04137563053559955
-2473,VAMP4,0.04812367682301079
-2474,EEF1AKNMT,0.04538029445542833
-2479,PIGC,-0.07643512093828045
-2481,SUCO,-0.021143732245071326
-2482,FASLG,-0.36174047576710305
-2488,AL645568.1,-0.08119931678700155
-2489,PRDX6,0.028381445153257006
-2494,KLHL20,-0.030795769367784637
-2496,DARS2,0.046426709212237585
-2497,GAS5,-0.22958885816893873
-2499,ZBTB37,-0.04692600130520321
-2501,RC3H1,0.18937372379989345
-2505,RABGAP1L,0.20490919963600185
-2507,RABGAP1L-IT1,0.09553714830653236
-2509,Z99127.4,-0.11581889882433677
-2510,CACYBP,-0.020410913170189598
-2511,MRPS14,-0.05151218458039087
-2513,KIAA0040,0.09733236923811879
-2520,COP1,0.2695341228865061
-2539,RALGPS2,0.0022057827131025044
-2541,FAM20B,0.13231448586876446
-2542,TOR3A,0.09441927776775924
-2543,ABL2,0.10687282542950279
-2544,SOAT1,0.19878226049958972
-2554,TOR1AIP2,0.08479940949564885
-2555,AL353708.3,-0.007045640928199204
-2556,TOR1AIP1,0.12300196671192276
-2559,CEP350,0.0596150279731648
-2561,QSOX1,0.2335794213817601
-2563,ACBD6,-0.11499880430399381
-2565,XPR1,0.0021591449509549983
-2570,STX6,0.09645612123800804
-2571,MR1,0.1073155280412313
-2572,IER5,0.22740771063023182
-2584,GLUL,0.3477604759053914
-2588,RNASEL,0.08851394613149052
-2593,NPL,0.3081697173817382
-2595,DHX9,0.010309444360711338
-2604,SMG7,0.1545805077784018
-2605,NCF2,0.5781328370976107
-2607,ARPC5,0.33842355292186055
-2608,RGL1,0.0762295581741006
-2612,TSEN15,-0.11132630796356957
-2616,C1orf21,-0.4313548556218341
-2620,EDEM3,0.2264722358386208
-2621,NIBAN1,0.23316843641784424
-2623,RNF2,0.03025746282929334
-2624,TRMT1L,0.06652849890303991
-2625,SWT1,0.01320426407401006
-2626,IVNS1ABP,-0.04476941047630166
-2634,TPR,-0.030959887806552216
-2635,ODR4,0.10583958935923386
-2640,PTGS2,0.14402180014402652
-2642,PLA2G4A,0.14873972773371713
-2663,RGS18,0.15234894685806716
-2670,RGS2,0.43624749487557524
-2671,UCHL5,-0.07152796038531664
-2672,RO60,-0.0022874226813659354
-2673,GLRX2,0.07190935755074561
-2674,CDC73,0.03706545142160386
-2694,ZBTB41,0.00935444606159036
-2697,DENND1B,0.030651288654708467
-2701,NEK7,0.0858757120853416
-2705,PTPRC,0.19103389122331113
-2707,MIR181A1HG,0.01606529288973126
-2717,ZNF281,0.11153371508439994
-2720,DDX59,-0.002074153995109934
-2722,CAMSAP2,0.09513672566895462
-2725,KIF21B,-0.10493497960439388
-2730,TMEM9,-0.03422086399314643
-2740,CSRP1,0.08653702143920076
-2742,NAV1,0.03932930323350387
-2746,IPO9,0.07186873777468501
-2750,TIMM17A,0.06741776610742044
-2751,RNPEP,0.2873980809884269
-2752,ELF3-AS1,0.028745756759732587
-2756,ARL8A,0.16128821663620085
-2757,PTPN7,-0.2997093998179809
-2760,PPP1R12B,0.054907595249520484
-2764,KDM5B,0.00207345774736278
-2767,RABIF,0.03522407119599279
-2768,KLHL12,0.00370141859174803
-2769,ADIPOR1,0.1444662116974638
-2770,CYB5R1,0.0609601928295955
-2771,TMEM183A,-0.02827917475594576
-2782,BTG2,-0.025148117354092073
-2786,ATP2B4,-0.1831404376826135
-2787,LAX1,-0.22049819120376699
-2790,SNRPE,-0.09428362807079878
-2805,PPP1R15B,0.13631971468425322
-2807,MDM4,0.09955700624706107
-2816,RBBP5,0.05015830330410572
-2817,DSTYK,-0.04341631248796663
-2820,NUAK2,0.04981101523444204
-2828,ELK4,0.01609934909433947
-2830,NUCKS1,-0.32761944783489594
-2831,RAB29,-0.16329123972693962
-2833,SLC41A1,-0.0643129797679448
-2845,FAM72A,0.07991391194108509
-2846,SRGAP2,0.2846267102548133
-2847,IKBKE,0.004501519865573392
-2850,RASSF5,-0.08693293098035197
-2851,EIF2D,0.04154642059937186
-2854,MAPKAPK2,0.045183094273143674
-2859,FCMR,-0.24049494144065728
-2863,PFKFB2,0.10955470652784356
-2864,YOD1,0.09506763257831813
-2870,CD55,0.20451648041819132
-2873,CR1,0.3213608644833712
-2877,CD46,0.18917072781270175
-2878,MIR29B2CHG,0.05425892933478768
-2901,TRAF3IP3,-0.3111210882364119
-2904,UTP25,-0.011666727107108326
-2915,RCOR3,0.1014036905826955
-2916,TRAF5,-0.22203860494991748
-2918,LINC00467,0.07988275588003411
-2921,SLC30A1,0.17601902972490063
-2928,LPGAT1,0.17781847629838943
-2930,INTS7,0.001120743578174862
-2934,PPP2R5A,0.04529788799522194
-2938,PACC1,0.03614122645537616
-2940,NENF,0.016909525415405757
-2944,AC092803.1,0.062366120790481816
-2946,ATF3,0.17499832398630055
-2950,BATF3,0.18059565493785826
-2951,NSL1,0.016941119888240267
-2952,TATDN3,0.14807611136163945
-2954,FLVCR1-DT,0.02382782024614989
-2955,FLVCR1,0.04496229960365304
-2958,ANGEL2,0.026540109672115286
-2959,RPS6KC1,0.14096581041533307
-2968,SMYD2,-0.02571246882853043
-2975,KCTD3,0.11962953277017511
-2981,GPATCH2,-0.06201082832308754
-2988,RRP15,-0.08004689528591243
-2995,LYPLAL1,0.05114835257961428
-3003,EPRS,-0.014298652679386223
-3004,BPNT1,-0.10051535832830272
-3005,IARS2,0.056864119732748374
-3006,RAB3GAP2,0.03253595098370535
-3013,MARC1,0.19299996422153068
-3018,HLX,0.2165596505459172
-3023,DUSP10,0.024548714726290437
-3032,TAF1A,-0.035873926320862
-3034,MIA3,0.013521437000894997
-3036,AIDA,-0.038880744028908364
-3037,BROX,0.09117446275608249
-3040,AL392172.1,-0.07223377721398516
-3043,TLR5,0.2826812743567339
-3049,CAPN2,0.19625122610931447
-3050,TP53BP2,0.1117240869950755
-3052,FBXO28,0.029261838631335725
-3053,DEGS1,-0.04954735404353803
-3056,NVL,-0.11204561668560711
-3057,CNIH4,0.07201930567275709
-3058,WDR26,0.1406390839401906
-3066,LBR,-0.011494977809536932
-3072,SRP9,-0.11932823312038646
-3074,AL591895.1,-0.030760960331218206
-3075,TMEM63A,-0.038264610445633444
-3077,PYCR2,-0.09656732567954394
-3080,SDE2,0.006915828785865056
-3082,H3F3A,0.31219859156186963
-3084,ACBD3,0.08676520033196737
-3088,PARP1,-0.1628218373952822
-3091,ITPKB,-0.08372264869312128
-3096,COQ8A,-0.02335154952953042
-3104,ZNF678,-0.027561789367112878
-3105,SNAP47,-0.0574076502348705
-3106,JMJD4,-0.029068982179845545
-3113,ARF1,0.14954922777535104
-3114,C1orf35,-0.1091235954529634
-3115,MRPL55,-0.07091029785252198
-3117,GUK1,-0.18044255028466294
-3120,IBA57,-0.06070782225859193
-3122,OBSCN,-0.09119137820754586
-3128,TRIM11,0.09161928789455985
-3133,HIST3H2A,-0.1302108360097372
-3135,RNF187,-0.04339836874641278
-3136,RHOU,0.12810038446785527
-3142,RAB4A,-0.023267083004019938
-3145,CCSAP,0.14871290288920863
-3147,NUP133,0.022677007038566885
-3150,ABCB10,0.04323381651073919
-3151,TAF5L,0.05844823918601125
-3152,URB2,-0.007987565975430795
-3155,GALNT2,0.1647507396125177
-3161,COG2,0.08348217018601198
-3171,TTC13,0.046932541212893406
-3172,ARV1,-0.006768021387820978
-3173,FAM89A,-0.08360351862004672
-3176,C1orf131,-0.05175518793055274
-3177,GNPAT,0.05725110530673056
-3178,EXOC8,0.054360126587478186
-3179,SPRTN,-0.06751262584704222
-3180,EGLN1,0.04293022046917649
-3181,AL445524.1,0.18177431665948565
-3182,TSNAX,0.12085659144621962
-3184,DISC1,0.17176477972372958
-3185,AL136171.2,-0.028451417960747866
-3189,SIPA1L2,0.14732949103072254
-3192,MAP10,0.05074014711500616
-3194,NTPCR,0.04755314317810244
-3195,PCNX2,-0.04052772296670836
-3208,COA6-AS1,-0.08511833648103066
-3209,COA6,-0.007196621804926549
-3210,TARBP1,0.011993347924302266
-3215,IRF2BP2,0.33161363445674846
-3231,TOMM20,-0.09433600685416231
-3232,RBM34,-0.06466649489043075
-3233,ARID4B,-0.07558317044885751
-3234,GGPS1,-0.0237949367492587
-3235,TBCE,-0.0641743227631362
-3237,TBCE,0.025111434324045025
-3238,B3GALNT2,0.010847647888431648
-3240,LYST,0.5217539007628255
-3244,NID1,0.2131189610729805
-3246,GPR137B,0.06266813067107095
-3247,ERO1B,-0.029892056221831633
-3249,LGALS8,0.1954343269941745
-3253,HEATR1,-0.051818717012853
-3255,MTR,0.00826902736187243
-3269,CHRM3-AS2,-0.1114454813986403
-3285,FH,-0.04048071042476534
-3286,KMO,0.17382615151067712
-3287,OPN3,0.15581939218254587
-3288,CHML,0.13286149532797714
-3298,CEP170,0.2537026965845072
-3301,SDCCAG8,0.1531648026754695
-3302,AKT3,-0.1787330775878146
-3307,ZBTB18,0.18130532370974803
-3314,ADSS,-0.08760146084099567
-3316,DESI2,-0.14293508710582215
-3320,COX20,-0.23755173569275495
-3321,HNRNPU,0.27180755258143374
-3323,AL356512.1,0.08812581768213701
-3324,EFCAB2,0.1358505659727487
-3329,SMYD3,-0.08818902825731513
-3334,TFB2M,-0.025947786448639782
-3335,CNST,-0.04372285549354217
-3339,SCCPDH,0.027553193021135635
-3342,AHCTF1,0.06945245138609928
-3345,ZNF669,-0.09752332652381723
-3348,ZNF124,0.030181367717348733
-3349,AL390728.6,0.074162199873565
-3351,ZNF496,0.04033314136445934
-3354,NLRP3,0.3616454098142198
-3405,SH3BP5L,0.02132343751735445
-3406,ZNF672,0.07356180964484776
-3407,ZNF692,-0.011021527287068263
-3409,PGBD2,-0.03275262341449746
-3412,SH3YL1,-0.16006899189392845
-3413,ACP1,-0.11009330127735954
-3425,TMEM18,-0.08984148556373082
-3458,EIPR1,-0.08674912590100226
-3460,TRAPPC12,0.055621680156723696
-3463,ADI1,0.13063561477880672
-3466,RNASEH1,-0.10735327116359063
-3467,RNASEH1-AS1,-0.05513901699326858
-3469,RPS7,-0.4635382002999135
-3493,CMPK2,0.25701933343604016
-3494,RSAD2,0.19895108578175624
-3497,RNF144A,-0.11249665500192177
-3503,LINC01871,-0.3620338644473577
-3513,ID2,-0.30591570271374846
-3514,KIDINS220,0.15546387253322616
-3520,ASAP2,0.12860927224878216
-3521,ITGB1BP1,-0.13752729317218254
-3522,CPSF3,0.08892898802574133
-3523,IAH1,-0.11751400990522555
-3524,ADAM17,0.1741955881862438
-3527,YWHAQ,-0.08893844508620935
-3531,TAF1B,-0.10846354861040636
-3532,AC010969.2,0.03902686217265188
-3537,AC104794.3,0.047737819642685676
-3538,KLF11,0.1623847008523021
-3544,HPCAL1,0.08202523513510757
-3545,ODC1,0.0020127810120394145
-3547,NOL10,-0.028854637975822453
-3553,PDIA6,0.017086264794617542
-3559,SLC66A3,0.14240521510206508
-3560,ROCK2,0.17940821124126907
-3565,E2F6,-0.0348011328299306
-3569,LPIN1,-0.1502631858066601
-3577,TRIB2,-0.17754096923675694
-3587,NBAS,0.050725083443742366
-3589,DDX1,0.07784556060108243
-3606,FAM49A,0.31008234711093946
-3611,SMC6,0.03199372889279875
-3612,GEN1,0.02632413281637072
-3618,RDH14,-0.1307125036857916
-3627,LINC00954,-0.03386369337683092
-3628,TTC32,0.06005370548849525
-3633,LAPTM4A,0.30339338884424283
-3637,PUM2,0.1474562523634391
-3638,RHOB,0.3287228550574142
-3641,HS1BP3,0.03239179903610253
-3646,LDAH,0.016637613353321846
-3668,ATAD2B,0.11367114720878344
-3669,UBXN2A,0.003468605385645744
-3671,WDCP,-0.08229346622688569
-3673,SF3B6,-0.06467705109316257
-3674,FAM228B,-0.002792465144689234
-3675,TP53I3,0.13478038831744965
-3680,ITSN2,0.10586386655722342
-3683,NCOA1,0.05488287230771231
-3684,PTRHD1,-0.018586892611111235
-3686,ADCY3,0.06157199148701501
-3688,DNAJC27,-0.033938193669163445
-3689,DNAJC27-AS1,-0.01230295463428027
-3692,POMC,0.014085576885204603
-3694,DNMT3A,-0.07839222307608121
-3696,DTNB,-0.03119622232392976
-3698,ASXL2,0.01983426412097881
-3700,RAB10,0.39343835375978176
-3702,HADHA,0.1673893777364338
-3703,HADHB,0.21277159118093
-3706,SELENOI,0.014179199400509072
-3713,SLC35F6,0.17557493678184938
-3719,TMEM214,0.06840104968588048
-3725,OST4,-0.18774865237272534
-3730,PREB,0.027903406645029806
-3733,SLC5A6,0.04615662270577039
-3734,ATRAID,-0.11110235701860098
-3735,CAD,-0.08375773825524688
-3740,MPV17,0.023559819125910098
-3741,GTF3C2,0.11230828330218272
-3744,EIF2B4,0.024715326922401713
-3745,SNX17,0.09184579441525452
-3746,ZNF513,0.0564950169020854
-3747,PPM1G,-0.08779081974559691
-3748,NRBP1,0.02739472440681292
-3754,ZNF512,0.021506683846262895
-3756,GPN1,-0.003501832597439905
-3757,SUPT7L,-0.017521550570058855
-3758,SLC4A1AP,0.03938627789764893
-3760,MRPL33,0.04069894028986808
-3762,BABAM2,0.003027383042324261
-3765,FOSL2,0.217738791768323
-3766,AC104695.4,0.10327841039808457
-3769,PLB1,0.11115947110580157
-3772,PPP1CB,0.26420026278680453
-3775,TRMT61B,-0.0012996156343653786
-3776,WDR43,-0.05123204172042152
-3780,CLIP4,0.08288698149146724
-3786,YPEL5,0.15701981310087892
-3787,LBH,-0.4090660759378844
-3790,LCLAT1,-0.08960387642591332
-3802,MEMO1,0.11670310138779728
-3803,DPY30,-0.11365094241936294
-3805,SPAST,0.09361190799741402
-3807,SLC30A6,0.058476413137746666
-3808,NLRC4,0.20021140935348625
-3809,YIPF4,0.19617175398979228
-3810,AL133245.1,0.07315426731555935
-3811,BIRC6,0.057851166335426
-3813,AL133243.2,0.006253101105526019
-3814,AL133243.3,0.03222639283386975
-3816,TTC27,-0.05841830236078515
-3821,RASGRP3,0.0026779429021710163
-3823,FAM98A,-0.015802962222383705
-3835,FEZ2,0.1547711184020215
-3837,STRN,-0.028778843865757287
-3838,HEATR5B,-0.030138959769695235
-3839,GPATCH11,-0.08009306309497742
-3840,EIF2AK2,0.2115477088370233
-3842,CEBPZOS,-0.056737914629396875
-3843,CEBPZ,-0.0765714095569197
-3845,NDUFAF7,0.037230688416023415
-3846,PRKD3,0.011543929523673039
-3849,QPCT,0.2642356264703035
-3851,AC006369.1,-0.22956883640888479
-3852,CDC42EP3,0.3807748695090205
-3855,RMDN2,0.06121595549122663
-3857,CYP1B1,0.290401376929399
-3858,CYP1B1-AS1,0.08410953786632651
-3862,ATL2,-0.01276119276789875
-3866,HNRNPLL,0.02760944534660607
-3868,GALM,-0.11535794539835806
-3870,SRSF7,-0.19436001750157023
-3871,GEMIN6,-0.09418006310423045
-3872,DHX57,0.1642377554152741
-3874,MORN2,0.06123886373638641
-3877,SOS1,0.05146416070065335
-3880,MAP4K3,0.027750373563033563
-3882,MAP4K3-DT,0.029279254529265646
-3884,THUMPD2,-0.0436092259533727
-3886,SLC8A1,0.44469684924270564
-3898,EML4,0.021793877286907196
-3899,COX7A2L,-0.04187021932932055
-3901,MTA3,-0.027628068839137456
-3903,HAAO,0.10665160308251519
-3909,AC010883.3,-0.052273995642736264
-3910,ZFP36L2,0.24720541853290165
-3911,LINC01126,0.054866070696253745
-3913,THADA,0.014436350907164653
-3919,LRPPRC,0.01866472128930627
-3921,PPM1B,0.08753802145019343
-3923,PREPL,-0.0789427730341284
-3924,CAMKMT,-0.07778475477112191
-3935,SRBD1,0.10504586762087802
-3936,PRKCE,0.08138696827697625
-3944,ATP6V1E2,-0.005695870471239839
-3946,RHOQ,0.3019773418275863
-3948,PIGF,0.013086211189250579
-3949,CRIPT,-0.05429220541656803
-3951,SOCS5,0.034599325834306346
-3955,MCFD2,0.0698155476124835
-3956,TTC7A,0.28515906906879596
-3961,CALM2,0.410165090685854
-3965,MSH2,-0.14335149396932065
-3968,MSH6,0.044902513205472304
-3969,FBXO11,0.2122844626724248
-3972,FOXN2,0.2466077168016932
-3974,PPP1R21,0.05702710942541635
-3994,ASB3,0.01631883848777843
-3996,ERLEC1,-0.12511380111590242
-3998,PSME4,-0.03540280329369151
-3999,ACYP2,-0.03331740638333218
-4003,SPTBN1,-0.18360997417445407
-4010,RTN4,0.33312330062204215
-4013,RPS27A,-0.46776282378813483
-4014,MTIF2,0.09559200925036916
-4016,CCDC88A,0.4263579213881338
-4017,CFAP36,-0.13242520309969005
-4018,PPP4R3B,0.010671270342786391
-4019,AC015982.2,-0.016208403181401262
-4020,PNPT1,0.08966696078321537
-4028,VRK2,0.09624863697662284
-4030,FANCL,0.04039729049566903
-4042,BCL11A,0.1423755293211823
-4044,PAPOLG,0.040791182983521745
-4046,REL,0.21799143447725058
-4048,PUS10,-0.04515966599101342
-4049,PEX13,-0.012532522126643719
-4050,KIAA1841,-0.04542126657653033
-4051,AC016747.1,-0.022554775658569757
-4052,C2orf74,-0.09107428318218894
-4054,USP34,-0.029857915793299078
-4056,AC016727.1,0.036497116993240805
-4057,XPO1,0.058771076896417675
-4060,CCT4,-0.11614678481381732
-4063,COMMD1,-0.025803905615719266
-4065,B3GNT2,0.012881785206636106
-4069,EHBP1,-0.01736133303487485
-4075,WDPCP,-0.05909754478427741
-4076,MDH1,-0.09369245857188405
-4077,UGP2,-0.05024958134494417
-4078,VPS54,0.08689395054618619
-4080,PELI1,0.30856519339411626
-4081,AC012368.1,0.1586711540566311
-4088,AC008074.2,0.08116128826815029
-4089,AFTPH,0.17219484498080917
-4091,SERTAD2,0.20446363241055152
-4095,LINC02245,0.003384222618208172
-4096,SLC1A4,0.1063937086863396
-4098,CEP68,-0.06970327483602243
-4099,RAB1A,0.22420928835497989
-4100,ACTR2,0.5026462265115831
-4121,ETAA1,-0.037012827131026604
-4126,C1D,-0.055574063948737915
-4127,WDR92,-0.020372939580815323
-4128,PNO1,-0.024620604459946845
-4129,PPP3R1,0.09618334511835591
-4134,PLEK,0.026695200262293522
-4135,FBXO48,-0.060521811904619534
-4138,ARHGAP25,-0.03061435589184341
-4146,GFPT1,-0.05526287138077426
-4147,NFU1,-0.007669675903745408
-4148,AAK1,-0.2511040941702531
-4149,ANXA4,0.21547978828581796
-4151,GMCL1,-0.0570178929269787
-4152,SNRNP27,-0.04246709358522554
-4153,MXD1,0.2299151565144025
-4155,PCBP1-AS1,0.030092742124526915
-4156,PCBP1,0.2900059724630289
-4159,C2orf42,-0.03871977551024012
-4160,TIA1,0.04108368243579689
-4161,PCYOX1,-0.039808208993005724
-4162,SNRPG,-0.05787209480278358
-4163,FAM136A,-0.008663350289473579
-4178,TEX261,0.03376052767588121
-4181,NAGK,0.5086751419051943
-4183,MCEE,-0.0457008824013247
-4185,MPHOSPH10,-0.039499003591504725
-4186,PAIP2B,-0.057965784386090506
-4187,ZNF638,0.07998297235540185
-4188,AC007878.1,0.015943322449660065
-4189,DYSF,0.3393801687525607
-4191,EXOC6B,-0.0059567709331388346
-4195,SFXN5,0.16320188695574417
-4199,SMYD5,-0.019899461782203514
-4200,PRADC1,-0.05788594655318647
-4201,CCT7,-0.12428551662985415
-4202,FBXO41,0.008261412967854173
-4206,ALMS1,-0.03707543693491771
-4210,TPRKB,0.020329619847435074
-4212,DUSP11,-0.06104778183430893
-4215,STAMBP,0.034156389769550495
-4218,DGUOK,-0.1017342771172798
-4221,TET3,0.30168003597520643
-4223,BOLA3,-0.23269411092440553
-4225,MOB1A,0.33359902600860414
-4227,MTHFD2,0.2214112014055077
-4229,DCTN1,0.0569050298067693
-4233,WDR54,-0.15135946347578744
-4235,INO80B,-0.020225032490778977
-4236,WBP1,-0.11689232557652773
-4237,MOGS,-0.007659187736146792
-4239,MRPL53,0.04099193528279888
-4240,CCDC142,0.05539280725906855
-4241,TTC31,0.00956260301473889
-4245,PCGF1,-0.06524123934293295
-4248,AUP1,0.011873417854754377
-4249,HTRA2,-0.034665336578651305
-4250,LOXL3,0.1690342403702416
-4251,DOK1,0.19480423190117604
-4257,HK2,0.19135054568212706
-4261,POLE4,0.13546112899063228
-4267,MRPL19,-0.05448638337995482
-4268,GCFC2,-0.012210371021707196
-4308,SUCLG1,-0.052361462863966035
-4310,TRABD2A,-0.12125818491843157
-4311,TMSB10,0.10511678429860147
-4313,KCMF1,0.06897375632923691
-4317,TGOLN2,0.14193972598407162
-4318,RETSAT,-0.06965441749648942
-4319,ELMOD3,-0.009605995038643598
-4321,CAPG,0.39819518401501885
-4324,PARTICL,-0.012077891107724227
-4325,MAT2A,0.12491002150096266
-4326,GGCX,0.05167848831284616
-4327,VAMP8,0.19430916195554676
-4328,VAMP5,0.37044902003129837
-4329,RNF181,0.042258896534659016
-4330,TMEM150A,0.11713627284205845
-4331,USP39,0.051563916578957356
-4332,C2orf68,0.030728681533663885
-4334,GNLY,-0.6456208330327069
-4337,ST3GAL5,0.18927816930157162
-4341,POLR1A,-0.00814619172315291
-4342,PTCD3,-0.04451824739656674
-4343,IMMT,0.05739336032240736
-4345,MRPL35,-0.015404888647798844
-4347,KDM3A,-0.0703627867451164
-4348,CHMP3,0.15758528141422729
-4350,RNF103,0.007702334174074196
-4351,RMND5A,0.0288682706510757
-4352,CD8A,-0.3927922730204935
-4353,CD8B,-0.2941808878372046
-4361,CYTOR,-0.14576467832204215
-4366,KRCC1,0.031384448399978754
-4373,EIF2AK3,0.00043979394160324525
-4376,RPIA,-0.012512995766950318
-4377,IGKC,0.004433955207262528
-4467,MAL,-0.16693436215844384
-4470,MRPS5,0.06769632386994065
-4471,ZNF514,-0.03097011689171506
-4472,ZNF2,-0.022110502950129403
-4476,FAHD2A,-0.04089313020981877
-4482,LINC00342,-0.17409399395707942
-4483,ANKRD36C,-0.17175873441492687
-4487,DUSP2,-0.23343147463350644
-4489,STARD7,0.16019061691596562
-4490,STARD7-AS1,0.00036397700774475344
-4491,TMEM127,0.24727779961684934
-4492,CIAO1,0.07088949637079486
-4493,SNRNP200,-0.07776059565631566
-4499,ARID5A,-0.06028055929905449
-4500,KANSL3,-0.0060451367022552665
-4502,LMAN2L,0.08296289653540581
-4503,CNNM4,-0.022755686298961582
-4505,CNNM3,-0.0730093236077145
-4507,ANKRD39,-0.11083294422376948
-4515,ANKRD36,-0.1802082348263284
-4526,AC092683.1,-0.12063412582080885
-4528,ANKRD36B,-0.09244314010759168
-4530,COX5B,0.11880736445233586
-4531,ACTR1B,-0.01583497833321699
-4533,ZAP70,-0.3382505419733231
-4534,TMEM131,0.18820974287003625
-4539,INPP4A,-0.07390163233919003
-4540,COA5,-0.049222852716025486
-4541,UNC50,-0.094134625796727
-4542,MGAT4A,-0.26995030936395475
-4543,LINC02611,-0.10896881605106465
-4545,TSGA10,-0.09447508385077935
-4548,LIPT1,-0.08459387584802684
-4549,MITD1,-0.0021143595665774452
-4550,MRPL30,-0.005083280960182778
-4553,TXNDC9,-0.14311279361246065
-4554,EIF5B,-0.0867957290704963
-4555,REV1,0.03786488898527051
-4557,AFF3,0.020790249715019672
-4563,PDCL3,-0.032031070334309485
-4573,RPL31,-0.2938736772344408
-4574,TBC1D8,0.20365145813327895
-4576,CNOT11,0.07241937650040461
-4577,RNF149,0.3164022093274636
-4583,MAP4K4,0.11731124094382614
-4584,LINC01127,0.1614642796196719
-4591,IL18R1,-0.19329431086952406
-4597,MFSD9,0.014349378598865713
-4624,MRPS9,-0.1768479465010364
-4629,TGFBRAP1,0.0425990529993937
-4631,AC012360.3,-0.08701997384161983
-4632,C2orf49,0.005396850188201053
-4637,NCK2,-0.0676131732856634
-4641,UXS1,-0.05709204988057144
-4665,GCC2,-0.17120874534936711
-4667,LIMS1,0.14082866296849894
-4669,RANBP2,0.1287029004447842
-4670,CCDC138,-0.09149703754870317
-4674,SEPTIN10,0.14282066916433478
-4683,MTLN,-0.09738506461705812
-4692,MIR4435-2HG,-0.05686565099142644
-4694,BCL2L11,0.16107200894757417
-4704,ANAPC1,-0.015283095252375067
-4705,MERTK,0.08473776574428828
-4707,TMEM87B,0.02753274833949147
-4710,ZC3H8,-0.0942175047030343
-4712,ZC3H6,-0.06832982545254067
-4714,TTL,0.03351373859470746
-4715,POLR1B,-0.029892551469070057
-4716,CHCHD5,-0.06706636300575922
-4719,AC079922.2,-0.039498104722798
-4720,SLC20A1,0.14775275111776467
-4725,IL1B,0.20046008864420162
-4732,IL1RN,0.2914690910372998
-4733,PSD4,-0.01806737625204614
-4735,PAX8-AS1,0.072276066613422
-4739,AC016745.2,0.11735504216447588
-4740,CBWD2,0.18815116325450226
-4746,RABL2A,0.003982795751804928
-4749,SLC35F5,0.0675069952281692
-4752,AC110769.2,0.006613956185903748
-4753,ACTR3,0.1530758077912592
-4766,DDX18,-0.13331040571831226
-4767,AC009404.1,0.03284228371523337
-4768,CCDC93,-0.06710299088746549
-4771,INSIG2,-0.04382207526407806
-4775,MARCO,0.2593486096736799
-4780,C2orf76,-0.06028071024401163
-4781,DBI,-0.25967891956495637
-4786,TMEM177,-0.08481461282592179
-4787,PTPN4,-0.4231067016779627
-4790,TMEM185B,-0.02121982914868197
-4791,RALB,0.32575359504413287
-4802,CLASP1,0.015220056526168311
-4804,NIFK-AS1,-0.08602634491339216
-4805,NIFK,-0.10891052573118402
-4806,TSN,-0.012797269157050404
-4822,GYPC,-0.4254197571930013
-4827,BIN1,-0.2959177513571566
-4830,ERCC3,-0.04262119664847299
-4832,MAP3K2,0.2117573735460183
-4833,MAP3K2-DT,-0.04908510473081728
-4836,IWS1,0.03321975511773485
-4842,WDR33,-0.1793531105757658
-4843,SFT2D3,0.005344919026924573
-4844,POLR2D,-0.0333509076333456
-4845,AMMECR1L,0.07313551661793095
-4848,SAP130,0.0751770276930104
-4849,UGGT1,0.10453795406567438
-4850,HS6ST1,-0.04878736244497769
-4861,SMPD4,0.04433160912503014
-4862,MZT2B,-0.2765311081509429
-4864,CCDC115,-0.008387022811329605
-4865,IMP4,0.03255449987951845
-4866,PTPN18,0.11299825451971336
-4880,FAM168B,0.04260967209131954
-4881,PLEKHB2,0.23669481632397926
-4887,MZT2A,-0.2574199263071514
-4892,C2orf27A,-0.0817096206264859
-4912,MGAT5,-0.07062390097911331
-4917,CCNT2,0.07234231368104103
-4919,RAB3GAP1,0.06962222379253569
-4921,R3HDM1,0.011978123978646735
-4922,UBXN4,0.01195382676636464
-4925,MCM6,-0.053444331277463675
-4926,DARS,0.027354713990457934
-4929,CXCR4,-0.08198311890682579
-4931,HNMT,0.4129075382011426
-4935,SPOPL,0.22738199377286847
-4936,AC092620.1,0.14347391152484842
-4948,KYNU,0.3205239516745874
-4949,ARHGAP15,-0.219470869284263
-4958,GTDC1,-0.019364252730579987
-4959,ZEB2,0.42941281787047997
-4961,ZEB2-AS1,0.17816364725875877
-4962,AC009951.6,0.08372700709644637
-4975,ACVR2A,0.09599326876755765
-4977,ORC4,-0.03870598844891137
-4978,MBD5,-0.06769588174230495
-4979,EPC2,-0.012423341131737748
-4986,MMADHC,-0.03755742825449173
-5000,RBM43,0.015129039185554024
-5001,NMI,0.2907077607867926
-5003,RIF1,0.024608752921740123
-5005,ARL5A,0.07144847770623049
-5006,AC097448.1,-0.1422282519186739
-5009,STAM2,0.1543183599797885
-5011,FMNL2,0.13259759711166658
-5014,PRPF40A,0.14093923750314447
-5015,ARL6IP6,-0.03649236911093291
-5029,NR4A2,0.16893717288810844
-5030,GPD2,0.1056911724118194
-5036,ERMN,-0.011614114184017189
-5037,CYTIP,-0.24640192624625856
-5040,ACVR1,0.08809779382684907
-5047,PKP4,0.04622997727267622
-5050,DAPL1,0.09485197255552884
-5053,WDSUB1,-0.03688961501766596
-5054,BAZ2B,0.24890965474457125
-5058,MARCH7,0.025531658064503866
-5059,CD302,0.5399192042869607
-5061,LY75,0.046945774163250804
-5066,RBMS1,0.25540480463919807
-5068,TANK,0.15920496004135914
-5071,PSMD14,-0.13664350681631096
-5077,DPP4,-0.11068521700992327
-5083,IFIH1,0.2620116285712006
-5084,GCA,0.489368089724997
-5090,COBLL1,-0.0025727328650093457
-5097,GALNT3,0.04113529903925252
-5101,TTC21B,-0.09082315733224029
-5112,STK39,-0.20246678620747088
-5113,CERS6,0.17178221715437197
-5119,DHRS9,0.19744685694031366
-5124,FASTKD1,0.07311124238666677
-5125,PPIG,0.0002505464931794843
-5129,SSB,-0.009679196134159615
-5130,METTL5,-0.15529330238036532
-5131,UBR3,0.07953848708522533
-5144,GORASP2,0.09201932295812312
-5145,TLK1,-0.019581937678105857
-5146,METTL8,-0.022465964280784616
-5147,DCAF17,-0.011337722471608764
-5148,CYBRD1,0.1863426317051021
-5149,DYNC1I2,0.038717953163053966
-5150,SLC25A12,-0.011993069205146223
-5151,HAT1,-0.06489950198500737
-5159,ITGA6,-0.24398284216679267
-5163,PDK1,-0.08061295954675789
-5166,MAP3K20,0.2697862173499763
-5170,SP3,0.12480435449953335
-5174,OLA1,-0.05464936512146665
-5177,CIR1,0.08720303787149489
-5178,SCRN3,-0.0025548819986496536
-5179,GPR155,0.07982413841548451
-5182,WIPF1,0.05988687629192841
-5186,ATF2,0.03642358553438445
-5188,ATP5MC3,-0.0286142603193885
-5192,LNPK,-0.017047456150680505
-5207,MTX2,0.04208625759256071
-5212,LINC01116,-0.003878895750669932
-5217,HNRNPA3,0.010693205267121414
-5218,NFE2L2,0.2131394768047896
-5224,AGPS,0.1034183100979215
-5231,RBM45,0.028838007878824966
-5233,CHROMR,0.09850750426624871
-5234,PRKRA,-0.0058557667768903456
-5237,PLEKHA3,-0.023802941948306167
-5238,TTN-AS1,0.03463388401386204
-5239,TTN,-0.0663577310186719
-5248,CCDC141,-0.07280206706223759
-5249,SESTD1,0.22718159287244136
-5252,CWC22,0.03857277205863466
-5255,UBE2E3,-0.0040030126804667795
-5258,LINC01934,-0.17232507629166566
-5260,ITGA4,0.28950650493469077
-5266,ITPRID2,0.25644543744641424
-5270,DNAJC10,0.14469632624102918
-5275,NUP35,-0.03367358962099626
-5283,AC096667.1,0.3163818370423029
-5292,ZC3H15,-0.0006381372710959301
-5293,ITGAV,0.05467440080706245
-5298,CALCRL,0.13416381858119159
-5307,WDR75,-0.03635407060162787
-5309,SLC40A1,0.1254005341731333
-5310,ASNSD1,-0.03331085597738606
-5312,ANKAR,-0.10987591391273722
-5313,OSGEPL1,-0.1723311180930337
-5316,ORMDL1,-0.1469543810961903
-5317,PMS1,-0.03234105629566249
-5320,HIBCH,-0.09736449109092435
-5321,INPP1,0.07812806173929018
-5322,MFSD6,-0.02492145186787968
-5324,NEMP2,-0.07300007705945748
-5327,NAB1,0.10344981720220653
-5330,GLS,-0.19362557608421654
-5331,STAT1,0.40935573011306764
-5334,STAT4,-0.34614794781041197
-5338,NABP1,0.1543672277215687
-5357,SLC39A10,-0.051765363597972146
-5359,STK17B,0.24042173706318834
-5360,AC114760.2,-0.007156091582770477
-5365,GTF3C3,0.05194476023637918
-5367,PGAP1,0.0837724922928979
-5368,ANKRD44,0.11112876728335262
-5369,AC013264.1,-0.09817014896814934
-5370,ANKRD44-IT1,0.0993455208801252
-5372,SF3B1,0.022214948718384076
-5373,COQ10B,-0.0960194085426279
-5374,HSPD1,-0.040237999018100726
-5375,HSPE1,-0.27152652807082167
-5376,MOB4,-0.038043880935277594
-5380,MARS2,-0.0007600662652021993
-5382,PLCL1,0.00012264653918536508
-5392,FTCDNL1,0.06444927168869853
-5394,C2orf69,-0.008251990795072823
-5395,TYW5,-0.011491863779485011
-5396,MAIP1,0.06183998734601139
-5398,SPATS2L,0.13240529575165064
-5400,KCTD18,0.004120302087460976
-5401,SGO2,-0.05104322296089025
-5405,BZW1,-0.04522940711714569
-5406,CLK1,0.030217637066243256
-5407,PPIL3,-0.08780461247344927
-5408,NIF3L1,-0.009948628700267829
-5409,ORC2,-0.032431173568487945
-5411,FAM126B,0.06880638062339724
-5412,NDUFB3,-0.015421633554819816
-5414,CFLAR,0.12607551922016647
-5417,CASP10,0.04624955456359288
-5418,CASP8,-0.15949584023182406
-5420,TRAK2,0.027474322702503582
-5421,STRADB,0.03761843879896187
-5426,ALS2,-0.011152624121986872
-5434,SUMO1,-0.14497788954020224
-5435,NOP58,-0.08997909378040309
-5439,BMPR2,0.11542812019290696
-5440,FAM117B,0.0983104192386103
-5441,ICA1L,-0.024984362459194356
-5442,WDR12,0.06383397841542364
-5443,CARF,-0.038604294146606453
-5444,NBEAL1,-0.05621868029841968
-5445,CYP20A1,-0.04950784739282875
-5446,ABI2,-0.06313512240125142
-5448,RAPH1,0.07672777777286625
-5449,CD28,-0.11116021424184325
-5451,ICOS,-0.1232642236739617
-5461,INO80D,-0.048122773970618524
-5463,NDUFS1,0.020898702980611264
-5465,EEF1B2,-0.35296911979411216
-5475,FASTKD2,0.006172083426191514
-5479,KLF7,0.16248304506695405
-5489,CREB1,0.11252366015517465
-5490,METTL21A,-0.0010038478968682114
-5491,LINC01857,0.0009700633220263451
-5492,CCNYL1,0.04752871163893172
-5495,PLEKHM3,0.06568424857879364
-5502,IDH1,0.19561861786975432
-5504,PIKFYVE,0.1759581635702402
-5511,RPE,0.016160805918103687
-5512,KANSL1L,0.07602093153687971
-5513,AC007038.2,-0.06493560434894792
-5514,AC007038.1,-0.003484132274783141
-5519,LANCL1,-0.06050587219896246
-5527,IKZF2,-0.19287257428035567
-5532,SPAG16,-0.10890761264297867
-5536,BARD1,-0.0011186205105511641
-5542,ATIC,-0.022702159896116368
-5555,PECR,-0.089135759186657
-5557,XRCC5,-0.02636907049752998
-5563,SMARCAL1,-0.030086231191846873
-5566,RPL37A,-0.31849345521420946
-5586,RUFY4,0.14810831288406728
-5587,CXCR2,-0.08070905421403834
-5589,ARPC2,0.046524424706771796
-5591,GPBAR1,0.3534707026572445
-5592,AAMP,0.13391720692976372
-5593,PNKD,-0.10462724776755428
-5594,TMBIM1,0.03178276597040644
-5598,SLC11A1,0.4591498509085583
-5599,CTDSP1,-0.007853821594101633
-5602,USP37,-0.0012569756709990508
-5603,CNOT9,0.07728536052331775
-5606,ZNF142,0.023564750730567453
-5607,BCS1L,-0.07528568616605288
-5608,RNF25,-0.014149962510668102
-5610,TTLL4,0.1421609739100783
-5611,CYP27A1,0.197752506728845
-5628,NHEJ1,-0.016751614403786072
-5630,CNPPD1,0.0109611933789947
-5631,RETREG2,0.0206309901773498
-5632,ZFAND2B,-0.13017278977177188
-5635,ATG9A,0.07816477152716568
-5636,ANKZF1,-0.10660728720717898
-5637,GLB1L,0.042533577694340075
-5638,STK16,-0.11203416684514324
-5639,TUBA4A,-0.013254021537091263
-5641,DNAJB2,-0.015064621582468525
-5645,DNPEP,0.04396448657294435
-5653,GMPPA,0.09588256320308595
-5661,STK11IP,0.10897987057376098
-5681,FARSB,0.03730172414519749
-5683,ACSL3,0.03321838441853163
-5689,AP1S3,0.046363714173252364
-5690,WDFY1,0.25566508233386326
-5691,MRPL44,0.025946357951968962
-5696,CUL3,0.05222726019541283
-5698,DOCK10,-0.11198441791369425
-5705,RHBDD1,0.12245676051210869
-5709,MFF,-0.106181150557881
-5712,AGFG1,0.2826093665430173
-5729,PID1,0.3309874846085023
-5732,TRIP12,0.20528332299509283
-5736,AC009950.1,0.0034088712445427597
-5737,SP110,0.2512184784377923
-5738,SP140,-0.13614296578582724
-5739,SP140L,-0.07127820281578406
-5740,SP100,0.045172093978010835
-5743,CAB39,0.07814788762131054
-5744,ITM2C,-0.04121955916339977
-5753,PSMD1,0.02119228228316989
-5759,NCL,-0.2310185495483895
-5765,PTMA,-0.21818948652115516
-5766,PDE6D,-0.04320163610563805
-5767,COPS7B,0.008921226119707863
-5770,DIS3L2,0.006398698981453612
-5781,TIGD1,-0.06371410118493212
-5782,EIF4E2,0.20780173296897317
-5785,GIGYF2,0.07149967908357367
-5792,INPP5D,0.07640959949574978
-5793,ATG16L1,-0.08404461534429475
-5796,DGKD,-0.0744629083518803
-5797,USP40,0.08391998732773222
-5819,ARL4C,-0.685015874503874
-5838,COPS8,-0.10398237101010757
-5849,LRRFIP1,0.35627895437561113
-5850,AC012076.1,0.024231959766023475
-5853,UBE2F,-0.010075779999750047
-5854,SCLY,-0.06305899695555076
-5858,ILKAP,-0.05275108298518827
-5862,PER2,0.019313476281270014
-5865,TRAF3IP1,-0.04616062229228058
-5866,ASB1,0.05289874843038447
-5875,HDAC4,0.10620308256180644
-5878,HDAC4-AS1,-0.036767666078768656
-5886,NDUFA10,0.057966391141766674
-5889,COPS9,-0.05482475000443644
-5896,ANKMY1,0.014693392055531284
-5898,DUSP28,-0.0435432764334454
-5899,RNPEPL1,-0.004665242092667018
-5901,CAPN10,-0.025501208098884242
-5902,GPR35,0.0842954401897453
-5915,MTERF4,-0.04096169833445607
-5916,PASK,-0.08886289503607818
-5917,PPP1R7,-0.09604944871903572
-5919,HDLBP,0.17513439014075374
-5920,SEPTIN2,0.16171059118932682
-5923,FARP2,0.07813253415131766
-5925,STK25,-0.016273960482534842
-5929,THAP4,0.033247397718683444
-5930,ATG4B,0.11095940278995874
-5931,DTYMK,-0.06836377255904803
-5932,ING5,-0.028531716194131865
-5934,D2HGDH,0.04372068576748159
-5968,TRNT1,-0.10005108890473387
-5969,CRBN,-0.0876690300362131
-5970,AC024060.2,-0.07426546374778237
-5972,SUMF1,0.1622295599641124
-5978,ITPR1,0.13911831758404106
-5982,BHLHE40,-0.0718955709791119
-5983,ARL8B,0.13601054145669433
-5986,EDEM1,0.05839403671334348
-6005,RAD18,-0.07640321764261367
-6013,THUMPD3-AS1,-0.17598276282566694
-6014,THUMPD3,-0.06455072409178854
-6015,SETD5,0.057808427670185875
-6017,MTMR14,0.16311172731283954
-6020,BRPF1,0.05178854117925912
-6021,OGG1,-0.02490625543493382
-6022,CAMK1,0.2962426389419097
-6023,TADA3,-0.1450514986046815
-6024,ARPC4,0.1356463579961666
-6025,TTLL3,0.07234096597008108
-6027,RPUSD3,-0.10562099938592777
-6029,JAGN1,-0.014869180761198376
-6031,IL17RC,0.04813439230917524
-6032,CRELD1,-0.09930804909671635
-6034,PRRT3,-0.0422897109899705
-6037,EMC3,0.06891643857756247
-6039,FANCD2,-0.06933708585522848
-6041,BRK1,-0.0029824045633320665
-6042,VHL,0.034069792683263656
-6045,TATDN2,0.010537486310325351
-6047,GHRL,0.02470874411638928
-6049,SEC13,0.08741774316885838
-6062,ATG7,0.23969180199284698
-6063,VGLL4,-0.04577224509155311
-6066,TAMM41,-0.06132021942171135
-6070,TSEN2,-0.07303196023966414
-6072,MKRN2,-0.03883971015793185
-6073,RAF1,0.132813161656117
-6077,RPL32,-0.3566993481227219
-6079,IQSEC1,0.16353178595116996
-6080,NUP210,-0.06993155324981747
-6087,CHCHD4,0.027167985469379454
-6088,TMEM43,0.03428092270166224
-6090,XPC,-0.005766611938844573
-6091,LSM3,-0.0741792959923476
-6094,SLC6A6,0.24466459979191366
-6098,CCDC174,-0.06355533806444673
-6104,FGD5-AS1,0.011896762419865294
-6105,NR2C2,-0.04584541409296224
-6106,MRPS25,0.00517342696588761
-6107,RBSN,-0.01011323413113225
-6108,CAPN7,0.044872199809795836
-6109,SH3BP5-AS1,-0.013076439789103484
-6110,SH3BP5,-0.19488257646695253
-6111,METTL6,-0.034435714054204966
-6112,EAF1,0.061106753929537225
-6116,HACL1,-0.01829169173487047
-6117,BTD,-0.084299697134102
-6118,ANKRD28,-0.03230280053824899
-6123,DPH3,0.06662549507685807
-6124,OXNAD1,-0.19535441806933646
-6125,RFTN1,-0.07147287023928901
-6135,PLCL2,-0.07284839790501497
-6138,TBC1D5,0.1495630944238689
-6142,SATB1,-0.08264098886324818
-6144,SATB1-AS1,-0.09853011881843977
-6148,RAB5A,0.06737399917522668
-6150,KAT2B,0.10879441887117611
-6159,UBE2E2,0.22116763507874734
-6161,UBE2E1,0.20362435426230005
-6163,RPL15,-0.3250755682054314
-6164,NR1D2,0.034407018830139545
-6172,TOP2B,-0.009258026198045832
-6173,NGLY1,-0.10241020941817416
-6174,OXSM,-0.047704194266270254
-6180,SLC4A7,-0.11715909962662147
-6181,LINC02084,-0.286041457210978
-6182,EOMES,-0.3280097017747177
-6186,CMC1,-0.40827925305800034
-6187,AZI2,0.20249452490344413
-6196,TGFBR2,0.04778129140034522
-6199,STT3B,0.13972137224505138
-6200,OSBPL10,0.006676115856927773
-6203,GPD1L,0.03442377886463393
-6205,CMTM8,-0.04318364031815813
-6206,CMTM7,0.10355379194134019
-6207,CMTM6,0.43680908863923756
-6208,DYNC1LI1,0.17490076203436541
-6209,CNOT10,-0.003674394693368801
-6213,GLB1,0.2411909776254347
-6215,CRTAP,0.33906209158305606
-6219,UBP1,0.07954175950628808
-6220,CLASP2,0.1697390785849842
-6222,PDCD6IP,0.20689750829113837
-6233,TRANK1,-0.051081733759387384
-6235,EPM2AIP1,-0.0011684890225714898
-6236,MLH1,0.04001890114363165
-6237,LRRFIP2,0.115934349925854
-6241,GOLGA4,0.008431351740577913
-6246,VILL,-0.04954213860370474
-6247,PLCD1,0.01607322817237409
-6249,ACAA1,0.1556104970775927
-6250,MYD88,0.3448393024322693
-6251,OXSR1,0.06015917240115966
-6257,EXOG,-0.102548562610109
-6262,WDR48,0.005564726709157084
-6263,GORASP1,0.1519471659237973
-6265,CSRNP1,0.1509874168285493
-6272,CX3CR1,-0.11800183592014461
-6275,SLC25A38,-0.17339476728892025
-6276,RPSA,-0.4582348206170158
-6283,EIF1B,-0.11362148128540735
-6284,ENTPD3-AS1,-0.04875542645395083
-6286,RPL14,-0.45959316299594466
-6287,ZNF619,0.03058260012866769
-6289,ZNF621,-0.0754574791128353
-6295,CTNNB1,0.14643043863790256
-6296,ULK4,-0.03701719738033826
-6297,TRAK1,0.1711535611311382
-6301,VIPR1,0.05554590886247903
-6303,SEC22C,-0.05104905434289899
-6304,SS18L2,-0.15862750169889037
-6305,NKTR,-0.03644883907510698
-6307,ZBTB47,0.14709148585357892
-6316,HIGD1A,-0.0021656968313763145
-6328,SNRK,-0.03990695087679992
-6330,ANO10,0.15861378766364354
-6331,ABHD5,0.24408958308957465
-6338,TCAIM,0.06200735311846255
-6340,ZNF445,0.07562141601675866
-6345,ZNF197,-0.048632287363702335
-6347,ZNF35,0.08123194920825982
-6351,KIAA1143,-0.021297684147432713
-6353,TMEM42,-0.05623502679081698
-6356,ZDHHC3,0.13069441253280012
-6357,EXOSC7,-0.0581488496176031
-6361,LARS2,-0.009513738196134143
-6363,LIMD1,0.1755194689653379
-6364,LIMD1-AS1,-0.004447227624271822
-6365,SACM1L,-0.05432091062515962
-6367,LZTFL1,-0.0025208691879609305
-6370,FYCO1,-0.018047778712513732
-6374,CCR1,0.41681256005515305
-6375,CCR2,0.35186475891786034
-6377,CCR5,-0.03161454950852866
-6378,CCRL2,0.1087903445526514
-6392,CCDC12,-0.1100679731927772
-6393,NBEAL2,0.19662623604526874
-6394,SETD2,0.051079272947896395
-6396,KIF9,-0.001676570743669881
-6397,KLHL18,0.10784476173573891
-6399,PTPN23,0.099050592122006
-6400,SCAP,0.013728050186675587
-6401,ELP6,-0.14122910552290271
-6403,SMARCC1,0.037110457489542566
-6405,DHX30,-0.030347426743956307
-6406,MAP4,-0.0029683829919831418
-6409,ZNF589,0.05292474020954821
-6410,NME6,-0.03717484051965727
-6414,CCDC51,-0.04090594241188569
-6415,TMA7,-0.3764204712162413
-6417,ATRIP,-0.07801072499967239
-6419,SHISA5,0.043163759490757216
-6420,PFKFB4,0.20918311960423117
-6423,UQCRC1,0.24557644421017674
-6425,SLC26A6,0.04949173922421481
-6429,NCKIPSD,0.02535293090001398
-6430,IP6K2,0.014418436131147239
-6431,PRKAR2A,0.1328930698026181
-6433,SLC25A20,-0.06090121350483151
-6434,ARIH2OS,-0.013194514307734329
-6435,ARIH2,-0.03880924605721326
-6439,P4HTM,-0.048958618818814685
-6440,WDR6,0.0183104626841483
-6441,DALRD3,-0.014836008850243578
-6442,NDUFAF3,-0.10550018247233973
-6443,IMPDH2,0.0030221265551924536
-6445,QRICH1,0.012136183832800943
-6446,QARS,0.15237836846531821
-6447,USP19,0.028491345614576537
-6449,CCDC71,-0.016098841904453307
-6450,KLHDC8B,0.09335835064367189
-6454,C3orf62,0.08122931455634039
-6455,USP4,0.1377986090794329
-6456,RHOA,0.39031143962745757
-6458,TCTA,-0.0023103426854125925
-6459,AMT,-0.06902839039532721
-6460,NICN1,-0.041613038410481175
-6461,DAG1,0.061265908591655135
-6465,APEH,0.014615290267910584
-6467,RNF123,0.1307628062461675
-6469,GMPPB,0.0081000888136635
-6470,IP6K1,-0.009419697657605699
-6473,UBA7,0.11949059585623165
-6479,MON1A,-0.056803715976849294
-6480,RBM6,0.05341029766312029
-6481,RBM5,0.14901615316965666
-6487,GNAI2,0.512234404096363
-6488,U73169.1,0.048130590251431904
-6489,U73166.1,0.04340274437230322
-6493,IFRD2,-0.13023003771715735
-6495,NAA80,-0.07761140273211664
-6497,HYAL2,0.06389450351106202
-6498,TUSC2,-0.07204141775346486
-6499,RASSF1,-0.16102148720142268
-6503,NPRL2,-0.19853940123691335
-6504,CYB561D2,-0.13533614577564312
-6505,TMEM115,-0.009306232863818336
-6507,CYB561D2,-0.13861554195844666
-6509,HEMK1,-0.10418858636313308
-6511,CISH,-0.11638152542145888
-6512,MAPKAPK3,0.3088702042808796
-6515,MANF,-0.2612868770619518
-6516,RBM15B,0.030104681403456353
-6517,DCAF1,0.03765411834436746
-6518,RAD54L2,0.020142063036616094
-6519,TEX264,-0.17145992612636596
-6529,RRP9,-0.13284685537188062
-6530,PARP3,0.004244643906438069
-6532,PCBP4,-0.12018051882861956
-6533,ABHD14B,-0.05256857326304062
-6534,ABHD14A,-0.22044816348631507
-6535,ACY1,0.039999747812647285
-6536,RPL29,-0.2942906330883844
-6538,DUSP7,0.12583175951312206
-6541,ALAS1,0.11869720706825349
-6543,TWF2,0.07155606125764012
-6545,PPM1M,-0.017085675214676153
-6546,WDR82,-0.05681741732447938
-6547,GLYCTK,0.01491265614575483
-6549,DNAH1,0.04806097401748904
-6550,BAP1,-0.005766887784979344
-6554,NISCH,0.0840637495429126
-6555,STAB1,0.3253981742764791
-6556,NT5DC2,0.042966851749628186
-6557,SMIM4,-0.018530050942995938
-6558,PBRM1,0.1498776473672137
-6559,GNL3,-0.10908007353643528
-6560,GLT8D1,-0.003448969946638408
-6561,SPCS1,-0.12546446360763797
-6562,NEK4,0.13233713466829142
-6565,ITIH4,0.0908174639550908
-6568,STIMATE,0.13595783128691158
-6569,SFMBT1,-0.044319552253120426
-6570,RFT1,0.043109540808546516
-6571,PRKCD,0.39220184209034
-6573,TKT,0.5766376785150495
-6574,DCP1A,-0.03391549928777609
-6580,ACTR8,0.024245989077569353
-6581,SELENOK,-0.1928214716020276
-6584,CACNA2D3,0.2060818499059535
-6599,CCDC66,-0.1290034462829639
-6600,TASOR,0.04565943280602091
-6601,ARHGEF3,-0.18288910085399243
-6607,APPL1,0.08050203055986004
-6612,PDE12,0.03563479064200176
-6613,ARF4,0.054877082068322015
-6615,DENND6A,0.07765442833085774
-6618,SLMAP,0.11110446799154736
-6619,FLNB,-0.052583966586005115
-6623,ABHD6,0.05522064288300451
-6624,RPP14,0.08665596707126812
-6627,PXK,0.11546692928335169
-6628,PDHB,-0.014450873941211726
-6631,KCTD6,0.05599606426840042
-6642,FHIT,-0.10240897045615076
-6646,C3orf14,0.08152931258074846
-6654,THOC7,-0.09410655774875987
-6656,AC104162.2,0.0040765342014775115
-6657,ATXN7,0.10786818654187925
-6659,PSMD6-AS2,0.0331558937529959
-6661,PSMD6,0.005531523745687
-6679,SLC25A26,-0.10646719238894516
-6680,LRIG1,-0.0354580281337802
-6684,KBTBD8,0.02203936763590275
-6686,SUCLG2,-0.042806813097001344
-6693,TMF1,0.05287850597532191
-6694,UBA3,0.11707481659540438
-6695,ARL6IP5,-0.04170957238325369
-6696,LMOD3,-0.07513397549067974
-6697,FRMD4B,0.21382806958859332
-6700,MITF,0.14534589677836526
-6705,FOXP1,-0.056182589478104616
-6708,FOXP1-IT1,0.10556785090695295
-6710,AC097634.1,0.05698554987326558
-6711,EIF4E3,0.18611090765841765
-6712,GPR27,0.12461737878877578
-6713,PROK2,0.014800187923346846
-6715,LINC00877,0.21527880702541577
-6723,RYBP,0.13198521548928974
-6726,SHQ1,0.021324107079007868
-6728,PPP4R2,0.015174872297331023
-6729,EBLN2,0.07026956178076388
-6739,ZNF717,-0.011947412496628371
-6754,GBE1,0.1005808812257288
-6771,CHMP2B,0.09479976909020917
-6775,CGGBP1,0.06971127581645581
-6776,ZNF654,0.039569574463366154
-6777,C3orf38,0.13561393777517397
-6782,ARL13B,-0.02857867362978895
-6784,DHFR2,0.008148123749056112
-6785,NSUN3,-0.02452256114624699
-6788,MTRNR2L12,-0.12696177158581673
-6794,CRYBG3,0.2627357829874388
-6796,RIOX2,0.06680335010895797
-6811,CLDND1,-0.11526256642300903
-6815,CPOX,0.045090541048961395
-6817,ST3GAL6,0.20330858775591976
-6826,CMSS1,-0.05904571465724358
-6829,TBC1D23,0.0545778893207297
-6830,NIT2,-0.15542051165664694
-6831,TOMM70,0.015241046247176279
-6835,TFG,0.09185326358254985
-6838,SENP7,-0.12361619453300957
-6839,TRMT10C,-0.07174346199354123
-6840,PCNP,-0.05483049654843696
-6841,ZBTB11,0.05513193866773714
-6842,ZBTB11-AS1,-0.04694357832456806
-6843,RPL24,-0.3463543721789223
-6844,AC084198.2,0.04111028945958516
-6845,CEP97,-0.013193444684471416
-6846,NXPE3,0.049999908428275376
-6848,NFKBIZ,0.3099720661379123
-6854,ALCAM,0.1466667906748037
-6855,CBLB,-0.2840950579508489
-6865,BBX,-0.05814168307003777
-6868,AC012020.1,-0.10134826576987642
-6869,CD47,-0.23392678279998744
-6870,LINC01215,-0.07822911950518098
-6871,IFT57,0.0002441525285279204
-6875,DZIP3,-0.1696729307949329
-6877,TRAT1,-0.21142478028066802
-6891,CD96,-0.3298393283710896
-6894,PLCXD2,-0.1493422793453618
-6898,ABHD10,0.032686685932377904
-6902,GCSAM,-0.12312142161664018
-6908,BTLA,-0.0626375707889975
-6909,ATG3,0.3719448554256399
-6910,SLC35A5,0.04756579942357771
-6915,CD200R1,-0.03177491668120734
-6917,GTPBP8,-0.09434297853610697
-6918,NEPRO,-0.0019256151550404912
-6926,SPICE1,-0.018792632699123835
-6927,SIDT1,-0.17618521065755607
-6929,USF3,-0.01616420228921921
-6930,NAA50,0.127257189908437
-6932,ATP6V1A,0.3912174928843707
-6936,CCDC191,-0.035381814036257994
-6939,QTRT2,0.10704448366042783
-6943,TIGIT,-0.3180023508591249
-6945,ZBTB20,-0.09694176099631982
-6970,B4GALT4,-0.09289000548023012
-6972,ARHGAP31,0.13437904835068998
-6974,TMEM39A,-0.022211393233863227
-6975,POGLUT1,0.016476828265475495
-6977,TIMMDC1,-0.010157695416537365
-6980,ADPRH,0.2038699637588854
-6983,COX17,-0.009449530397669745
-6988,GSK3B,0.07079449136527222
-6991,LRRC58,0.1855215527856485
-6995,NDUFB4,-0.1272574228660213
-6997,RABL3,0.0472606169605412
-6998,GTF2E1,0.060837993735569834
-7007,HCLS1,0.2180887919276411
-7008,GOLGB1,0.08454740170228527
-7009,IQCB1,-0.03819478515813517
-7010,EAF2,0.1172487196093727
-7011,SLC15A2,0.11315407260592256
-7013,CD86,0.3830432019936958
-7015,CSTA,0.538812369967757
-7016,CCDC58,-0.04665697257649865
-7017,FAM162A,-0.18459786622852475
-7018,WDR5B,-0.02562546426652532
-7019,AC083798.2,-0.06464831596700422
-7020,KPNA1,0.07770147808182942
-7022,PARP9,0.36068732869043574
-7023,DTX3L,0.268906341490131
-7024,PARP15,-0.24740233627320288
-7025,PARP14,0.38948928978046393
-7026,HSPBAP1,0.08190698173496547
-7027,SLC49A4,0.10081770769162576
-7028,LINC02035,0.10782595110549306
-7030,PDIA5,0.05469773302640452
-7031,SEC22A,-0.10030050365105227
-7035,HACD2,0.016655431925331865
-7040,CCDC14,-0.07755159432544767
-7044,UMPS,-0.017294083523716777
-7048,HEG1,-0.04259908306968886
-7051,ZNF148,0.050280740037581644
-7052,SNX4,0.05327770645168671
-7053,OSBPL11,0.2904304569876975
-7061,SLC41A3,-0.0777862993783193
-7074,ZXDC,0.06725086700539419
-7076,CHST13,0.14523045876492496
-7080,CHCHD6,-0.11193315818549622
-7091,TPRA1,0.023354326668765402
-7095,ABTB1,0.1497097000342315
-7096,MGLL,0.07977982079655425
-7099,SEC61A1,0.13934061791887928
-7100,RUVBL1,-0.15300058755158277
-7102,EEFSEC,0.015292689090951907
-7110,RPN1,0.268358581972632
-7111,RAB7A,0.2627714089678231
-7115,ACAD9,0.02698973443027827
-7119,RAB43,0.018262809986359006
-7121,ISY1,-0.052027911111745144
-7122,AC108673.3,0.062164884315621435
-7123,CNBP,0.16299019954350777
-7124,COPG1,0.07218597427981636
-7126,HMCES,-0.20503749393058066
-7127,H1FX,0.18987962629791935
-7130,MBD4,0.08383365282788875
-7134,PLXND1,0.06599379933874772
-7135,TMCC1,0.05521125439759841
-7136,AC083799.1,0.07576719426671351
-7146,PIK3R4,-0.019157014869791853
-7148,ATP2C1,0.006761059098519362
-7150,ASTE1,-0.11706401174699627
-7154,NUDT16,0.3692021472919239
-7156,MRPL3,-0.059426826708525754
-7159,ACPP,0.2026473574649871
-7160,DNAJC13,0.2801019219264978
-7161,ACAD11,-0.017997962946168584
-7163,UBA5,-0.07818455353057066
-7164,NPHP3,-0.02159132125339917
-7172,CDV3,0.25503406285068864
-7173,TOPBP1,0.001953806909620098
-7175,SRPRB,-0.08226353853090743
-7180,RYK,-0.026828359549367443
-7184,ANAPC13,0.04393427581776424
-7185,CEP63,0.0753219140701674
-7192,MSL2,0.0863941001373537
-7193,PCCB,0.07630791055536718
-7194,STAG1,-0.009718435316195345
-7198,NCK1-DT,-0.058110065696591114
-7200,NCK1,-0.1317089323200445
-7210,DBR1,0.09401905947904608
-7211,ARMC8,0.025269293829554305
-7215,CEP70,-0.060275143200359024
-7216,FAIM,-0.05139959462187447
-7217,PIK3CB,0.1885222515506593
-7222,MRPS22,-0.03204771361000072
-7229,COPB2,0.07011134466509587
-7244,SLC25A36,0.05495185837623647
-7248,PXYLP1,-0.1363211967428355
-7251,ZBTB38,-0.1523220663028639
-7252,RASA2,-0.10512453927847507
-7254,RNF7,-0.035274285787458684
-7257,ATP1B3,0.1746794834781296
-7259,TFDP2,-0.3476406743144561
-7260,GK5,-0.20777417328631934
-7261,XRN1,0.143239665380622
-7262,ATR,0.0021916339947794213
-7272,U2SURP,-0.11252220412472282
-7275,CHST2,-0.189454613356155
-7277,SLC9A9,0.003955268590882305
-7281,DIPK2A,0.12498751711026554
-7290,PLSCR1,0.441178468285024
-7313,GYG1,0.15048337264860054
-7314,HLTF,-0.1746662730652299
-7316,HPS3,0.09269771108107942
-7328,COMMD2,-0.03411051854727958
-7330,RNF13,0.3380430032503425
-7339,TSC22D2,0.02536664791427782
-7340,SERP1,0.3342955342003293
-7341,EIF2A,0.004813205416853589
-7342,SELENOT,0.022633600923706865
-7346,SIAH2,0.06624969897122307
-7353,GPR171,-0.21084689154953273
-7354,P2RY14,0.04807091508064669
-7357,P2RY13,0.46935510061977054
-7366,MBNL1,0.07923086005929666
-7367,MBNL1-AS1,-0.033095880963079076
-7373,RAP2B,-0.03758313229546962
-7381,DHX36,-0.15791334351065717
-7396,SLC33A1,-0.02168769476166392
-7398,GMPS,-0.06574168809186295
-7401,AC091607.2,-0.031994250823241914
-7405,SSR3,0.14848859641342418
-7406,TIPARP-AS1,0.043740681005824095
-7407,TIPARP,0.10230268950564061
-7413,CCNL1,0.16806236154649035
-7415,AC092944.1,0.014294342118282824
-7424,RSRC1,-0.00956529623881545
-7425,AC106707.1,-0.10005478162927858
-7427,GFM1,0.07094759819674977
-7432,MFSD1,0.37064646720120886
-7445,IFT80,-0.09930618231271594
-7446,SMC4,-0.025513525289153478
-7447,TRIM59,-0.15134349958193208
-7448,KPNA4,0.023644981079845123
-7451,PPM1L,-0.018441618886219122
-7453,NMD3,0.02219675874774093
-7476,PDCD10,-0.04509044977140075
-7477,SERPINI1,-0.13906059893554462
-7480,GOLIM4,0.12648975235844787
-7493,MYNN,-0.052273843498801625
-7502,SEC62,-0.08315254946338868
-7504,GPR160,0.039110049866373855
-7505,PHC3,0.011810899085349932
-7506,PRKCI,0.08498552399058511
-7508,SKIL,0.1587353200852427
-7516,RPL22L1,-0.10723863910263402
-7517,EIF5A2,-0.10575625633934109
-7520,TNIK,-0.3100952216320265
-7522,PLD1,0.17614469886365788
-7527,FNDC3B,0.3168949286189844
-7529,TNFSF10,0.47179610998837734
-7531,NCEH1,0.16516112664553556
-7533,ECT2,-0.0022270858911402147
-7546,TBL1XR1,0.007863257376923786
-7561,ZMAT3,0.09164988586400866
-7563,PIK3CA,0.0999233283556968
-7565,ZNF639,-0.08547805404941429
-7567,MFN1,0.08075141976181638
-7568,GNB4,0.3350091192656033
-7571,ACTL6A,0.022471472587136214
-7574,MRPL47,-0.08067578329788538
-7575,NDUFB5,0.0995882765962361
-7576,USP13,-0.06858065137716073
-7584,TTC14,-0.023000610088468623
-7588,FXR1,-0.04180995531997341
-7589,DNAJC19,-0.2241652869286366
-7605,ATP11B,0.008031728264086416
-7607,DCUN1D1,0.10418191817229754
-7608,MCCC1,0.01951907885490116
-7613,KLHL6,0.10458419216008188
-7615,KLHL24,0.03215380045644416
-7616,YEATS2,0.08930620113328208
-7619,PARL,0.0853590169039849
-7620,ABCC5,-0.029854798409276715
-7628,EIF2B5,0.036240597412022436
-7629,DVL3,0.11274198967317883
-7630,AP2M1,0.1432847851815708
-7631,ABCF3,0.013283741602712633
-7633,ALG3,0.062195014269337684
-7634,EEF1AKMT4,0.04130351800826052
-7637,PSMD2,-0.035525368322156664
-7638,EIF4G1,0.14740265807525735
-7639,FAM131A,0.07011632686534414
-7641,POLR2H,-0.17484804351464106
-7648,MAGEF1,-0.03299498165649885
-7653,VPS8,0.0552202255089508
-7657,MAP3K13,0.04681896374992779
-7658,TMEM41A,0.033419826992887994
-7661,SENP2,0.056195634274698134
-7662,IGF2BP2,0.18714472760217774
-7664,TRA2B,0.062273717478252395
-7667,DGKG,0.17057278321731723
-7672,TBCCD1,-0.01408936167992921
-7673,DNAJB11,-0.016529335681374487
-7680,EIF4A2,-0.21838666160304201
-7681,RFC4,-0.1415640563325324
-7686,ST6GAL1,-0.12010243549605883
-7687,RPL39L,-0.15257752425697382
-7692,RTP4,0.03331610005359277
-7697,BCL6,0.31467297420020207
-7698,AC072022.2,0.04439728161462636
-7706,LPP,0.03524967355883985
-7717,IL1RAP,0.06475082973315258
-7724,CCDC50,0.08849463317993045
-7740,OPA1,0.046994702749263643
-7760,ATP13A3,0.09246605418932524
-7763,TMEM44-AS1,-0.022859473512958695
-7767,LSG1,-0.021696762127801316
-7768,FAM43A,-0.10906360439641337
-7774,XXYLT1,0.12957730480560997
-7778,ACAP2,0.35657672563303494
-7780,PPP1R2,-0.18541240220394817
-7787,TNK2,0.1677951983049691
-7791,TFRC,0.05813489087366686
-7795,PCYT1A,0.1687166513413388
-7797,TCTEX1D2,-0.1706366556965615
-7800,UBXN7,0.1189950000570333
-7802,RNF168,-0.053978754744615866
-7805,WDR53,0.03635469567729678
-7806,FBXO45,0.020421760250475412
-7809,NRROS,0.08955175117675451
-7810,PIGX,0.06288115749472387
-7811,CEP19,-0.0043717881840739945
-7812,PAK2,0.24895929343088646
-7813,SENP5,0.015851659365943804
-7814,AC016949.1,0.027348414067564473
-7815,NCBP2,-0.04042797896839922
-7817,NCBP2AS2,-0.09789919983333517
-7822,DLG1,0.07361977016103016
-7829,BDH1,-0.10173139666045705
-7833,AC024560.5,-0.12111554610552858
-7834,RUBCN,0.017799369550588764
-7835,FYTTD1,0.05912384774628677
-7837,LRCH3,0.019895612146205822
-7839,IQCG,0.014317310199941117
-7840,RPL35A,-0.33231654081919315
-7844,ZNF595,-0.2557248339741949
-7845,ZNF718,-0.08351596941659868
-7849,ZNF141,-0.10937376370953847
-7852,ZNF721,-0.15365225799076573
-7853,PIGG,-0.037753287660822095
-7856,PDE6B,-0.039941253055822426
-7858,AC107464.3,-0.09266566351093118
-7859,ATP5ME,-0.03669247686478203
-7860,MYL5,-0.05550870404124113
-7861,SLC49A3,0.15093793253697213
-7862,PCGF3,0.09529437700554118
-7865,AC139887.2,-0.11968974816310574
-7870,GAK,0.1397398416519899
-7871,TMEM175,0.09341397416230153
-7872,DGKQ,-0.15788260740674917
-7874,IDUA,0.0710357289878999
-7880,SPON2,-0.48071704642517776
-7882,CTBP1-AS,-0.11297272035637582
-7883,CTBP1,0.0442368716626115
-7884,CTBP1-DT,-0.17329889537177706
-7885,MAEA,0.003056819445456474
-7886,UVSSA,0.01914545287789604
-7892,AC147067.1,0.030461242320274577
-7894,SLBP,0.012796455085166468
-7895,AC016773.1,-0.027146614007508493
-7896,TACC3,0.11272904131076288
-7897,TMEM129,-0.04625575880255265
-7899,LETM1,0.03702535124244077
-7900,NSD2,0.00485122031960335
-7901,NELFA,-0.06922697318408323
-7902,C4orf48,0.3639117159138969
-7905,HAUS3,-0.07652999958947583
-7907,MXD4,-0.11846387082984386
-7908,ZFYVE28,-0.21441403384746602
-7911,RNF4,-0.03607679798797039
-7913,FAM193A,0.030650047719264932
-7914,TNIP2,-0.022060997140104805
-7915,SH3BP2,0.28653210330838264
-7916,ADD1,-0.019190422594685137
-7918,MFSD10,-0.062116713187654674
-7919,NOP14-AS1,0.01600654643221263
-7920,NOP14,-0.02812139070100071
-7922,HTT,0.07484198483334817
-7925,RGS12,0.06800068352776323
-7929,LRPAP1,0.001473413198868886
-7937,TMEM128,0.04898331730805961
-7938,LYAR,-0.3620016222492085
-7939,ZBTB49,0.003709218692613713
-7941,NSG1,-0.16130576680505176
-7942,STX18,-0.03660670400655665
-7944,STX18-AS1,-0.0718758550233838
-7961,MAN2B2,-0.013058033733027507
-7962,MRFAP1,0.019292683797569415
-7963,LINC02482,-0.17992980675536882
-7964,AC093323.1,-0.1868958118635085
-7967,MRFAP1L1,-0.16493886675591127
-7968,BLOC1S4,-0.21700519336578422
-7970,KIAA0232,0.08425469832165998
-7971,TBC1D14,0.17306443201873434
-7972,AC097382.3,-0.00949998333195558
-7976,TADA2B,0.0675229275855305
-7977,GRPEL1,-0.04817375961962366
-7982,AFAP1,0.008938869944643997
-7988,SH3TC1,0.23623667147126984
-7992,ACOX3,0.04095302851528818
-7993,TRMT44,-0.0010979485475590218
-8026,SLC2A9,0.135545666940438
-8030,AC005674.2,0.013960747349936856
-8031,WDR1,0.023982692771557085
-8032,ZNF518B,0.06916300883187824
-8046,RAB28,0.04778104695260649
-8051,BOD1L1,0.17714709994149935
-8061,CPEB2,0.16731245482695875
-8067,FBXL5,0.44990816664335953
-8068,FAM200B,0.14723585150773583
-8069,BST1,0.39924380577488
-8070,CD38,-0.16314595300359014
-8072,FGFBP2,-0.5828075944632736
-8075,TAPT1,-0.027987328776005103
-8084,QDPR,-0.010949322086279752
-8086,LAP3,0.5374225873507635
-8088,MED28,0.062419290732269234
-8089,FAM184B,-0.045399955790954356
-8090,DCAF16,-0.024849766755293278
-8092,LCORL,0.08567512983780899
-8100,PACRGL,-0.06968656199460115
-8116,DHX15,0.12405447760313115
-8119,CCDC149,0.15023111009813092
-8121,SEPSECS,0.06440617053015057
-8122,SEPSECS-AS1,-0.09203944473728912
-8123,PI4K2B,0.044158047373477366
-8125,ZCCHC4,-0.08244340267408425
-8126,ANAPC4,-0.04012922658750458
-8129,SEL1L3,-0.0607127786342901
-8131,SMIM20,-0.09519634624577003
-8134,RBPJ,0.30593245632076643
-8136,TBC1D19,-0.1607849160289094
-8137,STIM2,-0.1685436268513922
-8171,ARAP2,-0.027128195333848106
-8173,DTHD1,-0.3473618260488518
-8183,RELL1,0.1566244049100894
-8184,PGM2,0.09651402297033389
-8185,TBC1D1,0.17533968949299847
-8193,KLF3-AS1,-0.07032831639009414
-8195,KLF3,0.053028555163336065
-8196,TLR10,0.018983849588030625
-8197,TLR1,0.28898819567306944
-8198,TLR6,0.2487776803244024
-8199,FAM114A1,0.11504561438630577
-8200,TMEM156,-0.14067140214282994
-8201,KLHL5,0.24417292081088934
-8202,AC079921.1,-0.03942145785822678
-8204,WDR19,-0.04419070959347006
-8205,RFC1,-0.11343782648613328
-8207,RPL9,-0.25453036243987115
-8208,LIAS,0.00543571549738609
-8209,UGDH,0.02063054094345541
-8212,SMIM14,0.241822237063139
-8215,UBE2K,0.038528061857272144
-8216,PDS5A,0.11603307855994144
-8217,N4BP2,0.01665957809616221
-8220,RHOH,-0.27630204127706415
-8224,RBM47,0.4186543275379054
-8240,TMEM33,0.22529611786195156
-8243,SLC30A9,0.06338276870761768
-8248,ATP8A1,-0.12042684234864885
-8260,GUF1,0.019491560182482763
-8261,GNPDA2,-0.11184508974863377
-8273,COMMD8,0.0380255416956829
-8276,ATP10D,0.14759877776884014
-8281,NFXL1,-0.024659045234960885
-8284,TXK,-0.3636447380010776
-8285,TEC,0.08952946055759095
-8286,SLAIN2,0.01290911627498759
-8289,FRYL,-0.046428930581832624
-8290,OCIAD1,-0.1569493108820384
-8292,OCIAD2,-0.31064925673803956
-8294,DCUN1D4,-0.002321658435658164
-8297,SGCB,0.005125976854425557
-8301,USP46,-0.07738567869629907
-8305,DANCR,-0.1968295140088441
-8309,SCFD2,0.016864328648440115
-8311,FIP1L1,-0.049764598655415326
-8318,CHIC2,-0.03791103461297669
-8334,SRD5A3,0.08992769253746191
-8337,TMEM165,0.18001868630092605
-8338,CLOCK,-0.07480267292919437
-8343,EXOC1,0.08115873406967014
-8346,CEP135,-0.013422874378893347
-8349,AASDH,-0.09681401344760757
-8351,PPAT,-0.06152330847150748
-8353,PAICS,-0.04685787786101414
-8354,SRP72,0.07181552162201517
-8357,HOPX,-0.5975595586450315
-8359,REST,-0.08192771486051624
-8361,NOA1,-0.03154078691757527
-8362,POLR2B,-0.07101917247044925
-8363,IGFBP7,-0.25958151684637115
-8403,CENPC,-0.12902539459522763
-8404,STAP1,-0.0583177440171262
-8405,UBA6,0.07509569420849879
-8406,UBA6-AS1,-0.09069596953570164
-8412,YTHDC1,-0.07399088206806383
-8426,SULT1B1,-0.019969368650050022
-8446,JCHAIN,-0.003159642791163291
-8447,UTP3,0.06686421121063885
-8449,RUFY3,0.2364354231845043
-8450,GRSF1,0.12376367667355322
-8451,MOB1B,0.14602435542136802
-8452,DCK,-0.02122930130583101
-8459,COX18,-0.00917209921010763
-8460,ANKRD17,0.11411458098680037
-8473,PPBP,0.07294720841112784
-8479,MTHFD2L,-0.006668669045757595
-8497,RCHY1,0.05342555144474422
-8498,THAP6,0.02221927382656926
-8501,G3BP2,0.042936141294763856
-8502,USO1,0.05360952946701822
-8505,NAAA,0.27781849874248143
-8506,SDAD1,-0.03384151632627905
-8510,CXCL10,0.12674831277412144
-8512,NUP54,-0.03665651136139037
-8514,SCARB2,0.28653195731484377
-8523,SEPTIN11,-0.07790059983072599
-8524,CCNI,0.18510294780340222
-8525,CCNG2,0.1345406767486242
-8528,CNOT6L,-0.22989090833013565
-8529,MRPL1,-0.12691863326844058
-8535,BMP2K,0.2913022880365029
-8536,PAQR3,-0.01673565161123878
-8544,ANTXR2,0.1319121058729463
-8552,RASGEF1B,0.12208591607867092
-8553,AC124016.2,-0.0975908720171235
-8554,HNRNPD,0.10268392599307069
-8556,HNRNPDL,-0.13837665920398942
-8557,ENOPH1,0.029989187732309817
-8562,SEC31A,0.12451693279434232
-8563,THAP9-AS1,0.06045745142830188
-8565,LIN54,-0.015763654210187304
-8566,COPS4,-0.03825017279136875
-8568,PLAC8,0.03455122065783618
-8570,COQ2,0.15894500453904692
-8571,HPSE,0.2575503191871873
-8572,HELQ,-0.03652389226971568
-8573,MRPS18C,-0.015075117340350504
-8574,ABRAXAS1,-0.1073220094379943
-8575,GPAT3,0.20936269689606937
-8581,WDFY3,0.3343081921670511
-8585,ARHGAP24,0.13072630809864705
-8592,AFF1,0.0042335881957839844
-8593,KLHL8,0.22815952328487832
-8596,HSD17B11,0.2322567550113275
-8597,NUDT9,-0.028397094797728287
-8607,PKD2,0.06504342887245046
-8609,PPM1K,-0.05471252192769501
-8613,HERC6,0.009331652102970272
-8614,HERC5,0.19300398285790835
-8615,PYURF,-0.14032502973836816
-8618,HERC3,0.13330867391367338
-8620,FAM13A-AS1,0.14552942408981412
-8621,FAM13A,0.1485194416301357
-8624,GPRIN3,-0.26813474769006307
-8626,SNCA,0.10664373989338326
-8645,SMARCAD1,-0.09890742331829755
-8648,PDLIM5,0.232200750250601
-8662,RAP1GDS1,0.011933117257256336
-8663,TSPAN5,-0.17013915250416103
-8666,EIF4E,-0.04067310606907543
-8669,METAP1,0.014116223831166973
-8671,ADH5,-0.13726438199313767
-8680,TRMT10A,-0.020108668039219423
-8684,DAPP1,0.33220644520276904
-8685,LAMTOR3,0.06647548207780622
-8686,DNAJB14,-0.04794347647529722
-8687,H2AFZ,-0.09059264560912982
-8696,PPP3CA,0.10111825730436656
-8698,BANK1,0.041520452666429175
-8701,SLC39A8,0.0028618497768396495
-8704,NFKB1,0.11593712481652886
-8705,MANBA,0.234257286808729
-8706,UBE2D3,0.27033789465591873
-8709,CISD2,0.09610095956879171
-8712,BDH2,-0.12462475284132918
-8726,AC004069.1,0.0544744403302805
-8727,TET2,0.3883786010147286
-8729,PPA2,0.02455614132444616
-8733,INTS12,0.013050218102545164
-8734,GSTCD,0.002356531391327379
-8739,TBCK,0.019635978850678342
-8740,AIMP1,-0.006074509683966785
-8746,PAPSS1,0.2080143860856814
-8747,SGMS2,0.22840844767178153
-8749,CYP2U1,0.03465759213904795
-8751,HADH,-0.012899667150280939
-8752,LEF1,-0.16497044636273214
-8753,LEF1-AS1,-0.060511953011381625
-8755,RPL34,-0.3603574988471146
-8756,OSTC,-0.15022698502057122
-8762,SEC24B,0.09615301931509411
-8763,MCUB,0.004201788943754147
-8764,CASP6,-0.06155793788705939
-8766,PLA2G12A,0.015537464354280321
-8769,GAR1,-0.06098586737068718
-8783,FAM241A,0.18953866489379934
-8785,AP1AR,0.11561256824734918
-8787,TIFA,0.08992494793817805
-8788,ALPK1,0.23847109580332967
-8792,LARP7,0.04477124052575982
-8799,CAMK2D,0.13037247235760502
-8817,SNHG8,-0.13237856001743845
-8820,METTL14,0.011503791767176518
-8821,SEC24D,0.19201159980033725
-8824,USP53,-0.10794962232557892
-8825,C4orf3,0.07668630455356111
-8842,ANXA5,0.5118032212254995
-8845,EXOSC9,-0.03942307299114636
-8847,BBS7,-0.046493583483013357
-8850,KIAA1109,0.15228055545836655
-8860,SPATA5,-0.06583745606028897
-8871,ANKRD50,0.1981390246785086
-8885,MFSD8,-0.0735236886622235
-8886,ABHD18,0.030284138612674587
-8887,LARP1B,-0.04157643373165685
-8888,PGRMC2,-0.09165253133656012
-8894,JADE1,-0.023909245909942395
-8895,SCLT1,0.22316127983351852
-8896,C4orf33,0.13145024715077228
-8950,ELF2,0.07136312258496365
-8952,NDUFC1,-0.010320841382506716
-8953,NAA15,-0.008896705709240067
-8954,AC097376.3,-0.03116642458941748
-8955,RAB33B,0.025588098475199026
-8956,SETD7,0.20315312222845858
-8960,MGST2,0.21221242965209908
-8961,MAML3,0.24493992600920225
-8964,SCOC,-0.08427909417563965
-8968,ELMOD2,-0.06012443393530113
-8971,TBC1D9,0.2798739521030415
-8973,AC096733.2,0.05532096879259489
-8975,ZNF330,0.1150279179280062
-8977,LINC02432,0.2839267407919806
-8980,IL15,0.2941714947542316
-8981,INPP4B,-0.20265006555337528
-8985,USP38,0.08469975027333947
-8987,GAB1,0.10162298846647567
-8990,SMARCA5,-0.05226610523163035
-9001,ANAPC10,-0.09657492529596612
-9003,ABCE1,0.003868165331208794
-9004,OTUD4,0.04039430056564711
-9007,SMAD1,0.13013720187568836
-9010,MMAA,-0.02627590855326912
-9013,ZNF827,-0.1827434913180039
-9017,LSM6,-0.04962830854905692
-9020,SLC10A7,0.0037503260056823014
-9030,TMEM184C,0.04343601434862572
-9031,PRMT9,-0.0947351195029817
-9032,ARHGAP10,-0.05327011173902323
-9047,LRBA,-0.14272003604438963
-9050,RPS3A,-0.3717912701357127
-9057,GATB,0.043504862687923515
-9067,LINC02273,-0.11626413936582426
-9073,FBXW7,0.12331296630635194
-9076,MIR4453HG,-0.04761949420524798
-9077,TMEM154,0.3679510078718408
-9080,ARFIP1,0.12670040501920646
-9086,TMEM131L,0.10559417145266227
-9088,TLR2,0.4833328269096885
-9096,PLRG1,-0.05109249341589774
-9107,MAP9,-0.16124294526667554
-9110,GUCY1A1,0.08853628696584581
-9115,CTSO,0.13471003442612337
-9120,PDGFC,0.1381299927056326
-9129,GASK1B,0.41440993012438154
-9130,FAM198B-AS1,0.24082348602031126
-9133,TMEM144,0.23641842576283786
-9138,ETFDH,0.07973840591399668
-9139,PPID,-0.07345662263995426
-9140,FNIP2,0.22290107247243734
-9142,RAPGEF2,0.2206835138423049
-9152,NAF1,-0.008095854158378327
-9156,TMA16,0.0591194416522419
-9157,MARCH1,0.5550058576168719
-9167,TMEM192,0.0715548558340797
-9168,KLHL2,0.06764056641025815
-9170,MSMO1,-0.20094911259961407
-9181,DDX60,0.1426493653665371
-9182,DDX60L,0.25864499814520137
-9183,PALLD,-0.03222368909733777
-9186,CBR4,0.004656681663342312
-9188,SH3RF1,0.1351772958893287
-9190,NEK1,-0.043287491963760714
-9192,CLCN3,0.0252748054490372
-9193,HPF1,-0.12065732009620599
-9213,GALNT7,0.1349321112656198
-9215,HMGB2,0.1932853024734827
-9217,SAP30,0.09799847329061172
-9231,FBXO8,-0.011594114292824224
-9232,CEP44,-0.01581313918469531
-9234,HPGD,-0.10851806030406265
-9242,GPM6A,-0.0818182445351569
-9248,SPCS3,0.08982415186411752
-9257,AGA,-0.09120042530040848
-9278,DCTD,-0.05439932850565246
-9283,WWC2,0.0906456931182519
-9287,CDKN2AIP,-0.13028452752757752
-9289,ING2,0.015889021645517473
-9291,RWDD4,-0.06677907537744683
-9292,TRAPPC11,0.09824312563969445
-9301,IRF2,0.03011608163316896
-9303,AC099343.3,-0.010922953220372131
-9307,CASP3,-0.01159279428873254
-9308,PRIMPOL,0.018009444498588396
-9310,ACSL1,0.3625625318835063
-9319,SLC25A4,-0.20443545541202757
-9320,CFAP97,-0.19575778214738573
-9324,ANKRD37,-0.07239662278183315
-9325,UFSP2,-0.07852214375096214
-9336,CYP4V2,-0.0453516840211674
-9373,FRG1,-0.061354953001137136
-9380,CCDC127,-0.11057793991517334
-9381,SDHA,-0.0565404982845407
-9383,PDCD6,-0.06212444760431276
-9386,EXOC3,0.07506368885795876
-9398,BRD9,0.025524015015550183
-9406,SLC12A7,0.22235671436591942
-9412,CLPTM1L,0.07359237432651365
-9416,LPCAT1,-0.05827830933310792
-9420,MRPL36,-0.18845251348076067
-9421,NDUFS6,0.08440409366900098
-9462,ICE1,-0.02456376492012195
-9467,MED10,-0.22059087142639694
-9471,NSUN2,0.057835095930869045
-9472,SRD5A1,0.21407466672190287
-9474,TENT4A,0.014145768227013269
-9490,MTRR,0.039652200561634356
-9491,FASTKD3,0.0016640570961655075
-9498,MIR4458HG,-0.024993874989696546
-9517,ATPSCKMT,-0.014068391563088572
-9519,CCT5,0.0018418517430147155
-9524,MARCH6,-0.05469635257319415
-9533,DAP,0.09353433793994097
-9543,TRIO,0.1923157091031804
-9545,OTULINL,0.22241007968826798
-9547,OTULIN,-0.04586570094278257
-9548,ANKH,-0.12880528874018565
-9561,ZNF622,-0.08047502148715067
-9562,RETREG1,-0.09544771489826655
-9566,BASP1,0.203975883951686
-9640,DROSHA,0.03464928739995989
-9641,C5orf22,0.023383119026536628
-9650,GOLPH3,0.13707891647436699
-9651,AC025181.2,0.08957611512538774
-9652,MTMR12,0.05372020946300318
-9653,ZFR,0.11880055301398144
-9654,SUB1,-0.234780484326541
-9664,TARS,-0.04102602394799863
-9669,AMACR,-0.09272053410389752
-9678,RAD1,-0.03869185519715522
-9679,BRIX1,-0.11444775552951975
-9680,DNAJC21,-0.03542760008076016
-9682,PRLR,0.1382044295702888
-9683,SPEF2,-0.07405165748399473
-9685,IL7R,-0.2756148018799732
-9690,LMBRD2,0.022769795622443446
-9691,SKP2,0.002500504320984449
-9692,NADK2,0.07115524637848102
-9700,NIPBL-DT,-0.015792393215115
-9701,NIPBL,0.2500225978151325
-9702,CPLANE1,6.627438886899169e-05
-9704,NUP155,0.03693227509258943
-9705,WDR70,-0.011330037703217994
-9725,RICTOR,0.06379158646921144
-9727,FYB1,0.4189755857334603
-9729,DAB2,-0.030263772984356223
-9737,TTC33,0.05952100202633664
-9738,PTGER4,0.02895233338556173
-9739,PRKAA1,-0.040420673229577146
-9740,RPL37,-0.3402963352426163
-9741,CARD6,0.22916433182626336
-9746,OXCT1,-0.19049131626112367
-9749,C5orf51,-0.018168401856678805
-9750,FBXO4,-0.10069891482515343
-9753,CCDC152,-0.1508827038554329
-9757,AC008875.3,-0.1562613384700555
-9760,AC025171.1,-0.09126610623578028
-9761,AC025171.3,0.07119746766452283
-9763,ANXA2R,-0.31574893999629333
-9764,AC025171.2,-0.0034908300719247334
-9766,ZNF131,-0.09963580622694469
-9768,HMGCS1,0.03601428669326155
-9770,CCL28,-0.10919698890558795
-9771,TMEM267,-0.025807472997087508
-9776,PAIP1,0.03081737012723098
-9777,NNT-AS1,-0.056550945841641886
-9778,NNT,-0.00011754478004528617
-9786,MRPS30,-0.051573871295513905
-9792,EMB,-0.041608314086915746
-9794,PARP8,0.01788748264105334
-9806,PELO,0.015955211197060042
-9812,MOCS2,-0.05322878788610112
-9816,NDUFS4,-0.03468097753778765
-9819,ARL15,0.08823524255386923
-9823,SNX18,0.20420181140999588
-9828,GZMK,-0.24551020504266943
-9831,GZMA,-0.7259227340063925
-9837,DHX29,0.06281993952389373
-9838,MTREX,0.025343197845177298
-9840,SLC38A9,0.0021200394672540633
-9842,IL31RA,0.11910519141279716
-9843,IL6ST,0.1468331229530067
-9847,ANKRD55,-0.04580043323158002
-9849,LINC01948,0.055738650830793046
-9854,MAP3K1,0.32352226413969093
-9857,SETD9,0.002442097699980177
-9858,MIER3,0.01189246713407343
-9861,GPBP1,-0.06046436123109149
-9871,GAPT,0.24844856892830633
-9876,PDE4D,-0.039181898600936536
-9884,ERCC8,-0.001752141811791387
-9886,NDUFAF2,-0.035189744351579816
-9887,SMIM15,0.06378219242112718
-9890,ZSWIM6,0.26910696619141317
-9895,KIF2A,-0.11506221752419098
-9896,DIMT1,-0.07081600705912144
-9897,IPO11,-0.04626997748070661
-9905,SREK1IP1,0.0010952620871878074
-9906,CWC27,0.008257848041443933
-9909,ADAMTS6,-0.06255819446778847
-9911,CENPK,-0.14720474515909618
-9912,PPWD1,-0.045127335848536156
-9913,TRIM23,0.006866768245080089
-9914,SHLD3,-0.048101993695340635
-9915,TRAPPC13,0.05892572566056909
-9916,SGTB,0.12278769942591286
-9917,NLN,0.2731696484321267
-9919,ERBIN,0.1759528399153463
-9920,AC010359.3,0.03465877507893711
-9922,SREK1,0.0778724901598099
-9928,MAST4,-0.03726860978753565
-9931,CD180,0.18729429183486862
-9940,PIK3R1,-0.20053862960898544
-9949,SLC30A5,0.04179779824698753
-9952,CENPH,-0.04885013749474409
-9953,MRPS36,-0.03196255748837771
-9954,CDK7,0.06506031715009025
-9955,CCDC125,-0.024380812704890977
-9956,AK6,-0.12493786533172602
-9957,TAF9,0.04438355229228321
-9958,RAD17,-0.10603632334877561
-9962,GTF2H2C,-0.08402060627970048
-9968,SMN1,-0.02427826062021582
-9970,NAIP,0.42644825143404175
-9971,GTF2H2,-0.0338546116637848
-9975,BDP1,0.007056255293629182
-9976,MCCC2,0.04220984391257312
-9980,MRPS27,-0.07814389137255126
-9981,PTCD2,-0.01687304073906856
-9987,TNPO1,0.03674957137227263
-9990,FCHO2,0.07068442224356453
-10002,BTF3,-0.23566633322543284
-10003,ANKRA2,-0.10243325611575906
-10004,UTP15,0.014025537213829854
-10013,ENC1,0.16104735997885497
-10014,HEXB,0.21646052630306914
-10015,GFM2,-0.005768463043974085
-10016,NSA2,-0.10549972217642359
-10024,HMGCR,0.09656357055034062
-10025,CERT1,0.2772337892254688
-10027,POLK,0.3048614422222934
-10031,POC5,-0.051655995570818306
-10035,IQGAP2,0.09496511562683138
-10039,F2R,-0.1765380548035368
-10041,S100Z,0.282542503258058
-10043,AGGF1,0.07853066538544547
-10044,ZBED3,0.06870404801081462
-10048,WDR41,0.06481501307557008
-10050,TBCA,0.004524309428383565
-10053,AP3B1,0.040693817201430775
-10054,SCAMP1-AS1,-0.1098681044678993
-10055,SCAMP1,-0.014561332578324896
-10056,LHFPL2,0.18911055378151756
-10057,ARSB,0.15101871162213837
-10061,JMY,-0.09820523650569578
-10062,HOMER1,0.07614162291971284
-10063,TENT2,0.12782909113048763
-10068,MTX3,-0.04550119947534074
-10071,SERINC5,0.09980329857322869
-10074,ZFYVE16,0.22466685464558797
-10075,AC008771.1,0.02464531525801111
-10079,DHFR,0.03702420884517438
-10081,MSH3,0.04583386119279253
-10083,RASGRF2,-0.09103788841342782
-10085,CKMT2-AS1,-0.023615010863680674
-10087,ZCCHC9,-0.03218260005083224
-10090,SSBP2,-0.03148240274215509
-10093,ATG10,-0.018513103576581653
-10096,RPS23,-0.33853312160200005
-10106,TMEM167A,0.25388544907663185
-10107,XRCC4,0.15082959017291536
-10108,VCAN,0.5447241166878943
-10123,COX7C,-0.19405324994468143
-10128,RASA1,0.040238867740552846
-10129,CCNH,-0.03648320026490081
-10135,TMEM161B,-0.02602966355180124
-10136,TMEM161B-AS1,-0.1746449084669817
-10143,MEF2C,0.31891936728419024
-10151,CETN3,0.010408444057457673
-10153,MBLAC2,0.0024780938359980896
-10155,LYSMD3,0.07754366939258296
-10157,LUCAT1,0.1612691207704238
-10164,ARRDC3,-0.03972461582723815
-10177,FAM172A,-0.15475281595658597
-10179,POU5F2,0.059923805263661124
-10182,AC114980.1,0.051592723973884334
-10186,SLF1,-0.11965822114438103
-10187,MCTP1,0.3348628271564881
-10191,TTC37,-0.023305375430933476
-10192,ARSK,0.014575085133763164
-10198,GLRX,0.30237290376973214
-10203,ELL2,0.11865461231888035
-10210,CAST,0.2665870414644574
-10212,ERAP1,0.08080912906142873
-10213,AC008906.1,-0.07495013477839331
-10215,AC009126.1,-0.017029334165875185
-10216,ERAP2,0.14748903069404756
-10217,LNPEP,-0.048686005785236366
-10220,RIOK2,-0.030224787604367642
-10231,CHD1,0.310122148569758
-10236,FAM174A,-0.03313766645550207
-10238,ST8SIA4,0.14358132712235538
-10240,SLCO4C1,-0.1453546350217272
-10247,PAM,0.008034666922983292
-10248,GIN1,-0.00693913062854687
-10249,PPIP5K2,0.08972209095322684
-10263,FBXL17,0.04658005448104118
-10266,FER,0.11435460515892379
-10269,AC008467.1,-0.13207578061271233
-10270,PJA2,0.1440904532193461
-10275,MAN2A1,0.08203822192005618
-10278,SLC25A46,-0.014819915550813327
-10282,WDR36,0.017226481038140885
-10283,CAMK4,-0.17544323380881435
-10286,STARD4,-0.03491721860977338
-10288,NREP,0.1103313122100939
-10291,EPB41L4A-AS1,-0.1736897494717603
-10297,APC,0.09434653445798703
-10298,SRP19,-0.1798331896145121
-10299,REEP5,0.157724198206602
-10300,DCP2,0.17719898083333516
-10304,YTHDC2,0.07367892601303198
-10314,PGGT1B,0.13181712940802207
-10316,CCDC112,0.10445335359422832
-10317,FEM1C,0.11075353376690045
-10318,TICAM2,0.286558893764155
-10320,TMED7,0.17311634274722112
-10324,ATG12,-0.012467185082513883
-10325,AP3S1,0.08489588526181406
-10330,AC034236.2,-0.023147117799557554
-10331,COMMD10,0.06942978929397024
-10348,DTWD2,-0.0626947895585135
-10350,DMXL1,0.14224027824037183
-10351,TNFAIP8,-0.20466119283230633
-10352,HSD17B4,0.2416710013591333
-10360,SRFBP1,0.020292559587711825
-10372,SNX2,0.33110948232597237
-10374,SNX24,-0.02468280409805274
-10379,CEP120,-0.013845748193812778
-10380,CSNK1G3,-0.03761387885566911
-10398,GRAMD2B,-0.04069633167971626
-10400,ALDH7A1,-0.06889406868849487
-10401,PHAX,-0.10017491963416397
-10404,LMNB1,0.2582016948212292
-10406,C5orf63,-0.07335698053949319
-10411,PRRC1,0.04156730327490468
-10417,LINC01184,-0.057433365551225524
-10418,SLC12A2,-0.03320845910978571
-10419,FBN2,0.183314634310229
-10422,ISOC1,-0.08331274495086644
-10430,HINT1,-0.3090507505377452
-10431,LYRM7,-0.022667304738541883
-10432,CDC42SE2,-0.17959971186437437
-10433,RAPGEF6,-0.15172693594816677
-10434,FNIP1,0.09474507924174846
-10437,ACSL6,-0.10796752468950034
-10446,SLC22A4,0.11211384950177256
-10447,MIR3936HG,-0.037402110633886966
-10448,SLC22A5,-0.008743935586233879
-10449,C5orf56,-0.23735637750329341
-10451,AC116366.1,0.07950195174097047
-10452,IRF1,0.3105527065024831
-10454,RAD50,0.017921301862891866
-10459,KIF3A,-0.10123516939596498
-10464,SHROOM1,0.17926559743063258
-10466,UQCRQ,0.0016507028249110921
-10467,LEAP2,-0.14276444417686215
-10468,AFF4,0.1028564371697716
-10470,ZCCHC10,-0.008919091431181338
-10473,HSPA4,0.05727388963114862
-10481,C5orf15,0.06861474942683912
-10482,VDAC1,0.05439266272755987
-10484,TCF7,-0.23019217203841352
-10485,SKP1,-0.28780950864862875
-10486,PPP2CA,0.08436685826841102
-10489,UBE2B,0.037609242151351585
-10491,CDKN2AIPNL,-0.10252055926300262
-10495,JADE2,-0.15773527400624657
-10496,SAR1B,0.018775394335782007
-10497,SEC24A,0.1272758030348309
-10498,CAMLG,-0.11839094562247555
-10499,DDX46,-0.09477058310302884
-10500,C5orf24,0.044220476108599714
-10501,TXNDC15,-0.12476860928109904
-10502,PCBD2,0.058095997316516025
-10511,H2AFY,0.4938109885533153
-10514,TIFAB,0.032848640568854
-10526,TGFBI,0.47068365675691337
-10528,SMAD5,0.12914420395042667
-10538,HNRNPA0,-0.043428366246590036
-10540,AC106791.1,-0.0562661167347277
-10543,FAM13B,0.05868733576851504
-10548,BRD8,-0.02208544261082086
-10550,CDC23,-0.08603856108612147
-10553,FAM53C,0.08419920280426607
-10555,KDM3B,0.09697187840167527
-10557,EGR1,0.09283376869232902
-10558,ETF1,0.08897152280741188
-10559,HSPA9,-0.01845783036423724
-10560,CTNNA1,0.3125820896743361
-10563,SIL1,0.14335374061225942
-10566,SNHG4,0.01357922659955517
-10567,MATR3,-0.038675781272228336
-10568,PAIP2,0.003605507669096111
-10571,MZB1,-0.0230421107806085
-10577,TMEM173,-0.04355430724311801
-10579,UBE2D2,-0.20401467442542368
-10581,CXXC5,-0.08759578800620378
-10591,PURA,-0.12227534507317812
-10592,IGIP,-0.09911045065156056
-10594,CYSTM1,0.15490502861275376
-10596,PFDN1,-0.04531210268368414
-10597,HBEGF,0.1229371763312422
-10601,ANKHD1,0.18525660997289728
-10602,SRA1,0.04953966180638912
-10603,EIF4EBP3,0.02372822771450144
-10604,APBB3,0.09543373634216107
-10605,SLC35A4,0.03369582918686398
-10606,CD14,0.5080679217268551
-10607,NDUFA2,0.021395690432759525
-10608,TMCO6,-0.08788081961065398
-10609,IK,-0.09376273454042168
-10610,WDR55,0.07846689450984087
-10612,HARS,-0.09741453647649226
-10613,HARS2,0.05971100537286767
-10614,ZMAT2,0.029073743287795082
-10660,TAF7,-0.14626166945815236
-10689,DIAPH1,0.0629602709082795
-10692,HDAC3,-0.03178305288728103
-10693,RELL2,0.028993950046158255
-10694,FCHSD1,-0.0405139450906083
-10695,ARAP3,0.12115913436365347
-10701,DELE1,0.01941655676678613
-10704,RNF14,-0.08590313213902627
-10706,GNPDA1,0.05818643504098866
-10707,NDFIP1,-0.08908742383487987
-10714,ARHGAP26,0.3087472287029177
-10717,NR3C1,0.11554100052263513
-10722,YIPF5,-0.003550040154877264
-10734,LARS,-0.044860057418616296
-10735,RBM27,-0.032636365093806824
-10740,TCERG1,0.07297884858403882
-10743,PPP2R2B,-0.3182173378029108
-10766,FBXO38,0.11473597914117417
-10770,ADRB2,0.00600503988037304
-10779,GRPEL2,-0.05087695502706438
-10781,PCYOX1L,0.06536723026605835
-10786,CSNK1A1,0.2586571789398263
-10789,PPARGC1B,0.09925770601341626
-10791,SLC26A2,0.019583944140373937
-10793,HMGXB3,0.029032520817934112
-10794,CSF1R,0.3926171073725566
-10800,TCOF1,-0.038952748115286644
-10801,CD74,0.4691360291137732
-10803,RPS14,-0.4259171431788925
-10805,NDST1,0.16666861359269894
-10811,RBM22,-0.030124591224404702
-10812,DCTN4,0.08446883077956441
-10813,SMIM3,0.0648143388782345
-10817,TNIP1,0.14896836051482748
-10818,ANXA6,-0.3552770461096477
-10820,CCDC69,0.18004183861358433
-10821,GM2A,0.3106246124783482
-10824,SLC36A1,0.1960702279141166
-10832,ATOX1,0.16292710296131555
-10835,G3BP1,-0.09630860188023635
-10844,FAM114A2,-0.11189161701099806
-10845,MFAP3,-0.015909000250526256
-10846,GALNT10,-0.12770582986679685
-10847,SAP30L-AS1,-0.07291185951613664
-10849,SAP30L,0.01603238120495835
-10852,LARP1,0.20996809962264076
-10854,CNOT8,0.08828414289277911
-10855,GEMIN5,0.033143881892104675
-10856,MRPL22,-0.0602866134198469
-10865,HAVCR2,-0.008527980939963362
-10867,MED7,-0.1391589991652118
-10868,ITK,-0.20612273858985003
-10872,CYFIP2,-0.28363265194049964
-10876,ADAM19,0.05803381356592184
-10881,THG1L,0.05919679076971172
-10882,LSM11,0.0015639278348555394
-10883,CLINT1,0.09929657348131447
-10893,EBF1,0.0003836930723564931
-10897,RNF145,-0.020139631819097368
-10900,UBLCP1,-0.026700349913052052
-10907,TTC1,0.0020682382048955047
-10908,PWWP2A,0.12382678019548991
-10914,SLU7,0.10127273713322668
-10915,PTTG1,-0.14740033863370716
-10928,CCNG1,0.014025561161623469
-10929,NUDCD2,-0.14408348637264842
-10932,MAT2B,0.010631853696208005
-10956,RARS,0.0933033282067228
-10958,PANK3,0.0698014333699749
-10964,SPDL1,-0.01352216741355714
-10965,DOCK2,0.3019055768285492
-10971,LCP2,0.17740700355333322
-10975,KCNMB1,0.19930417539370762
-10984,NPM1,-0.3348995917529061
-10989,FBXW11,-0.011456012510275407
-10990,STK10,-0.02353344791046729
-10992,UBTD2,0.1082536971138293
-11000,DUSP1,0.5522389153473157
-11002,ERGIC1,0.13135023200354962
-11005,RPL26L1,-0.04952263983877344
-11006,ATP6V0E1,-0.04996345362408512
-11007,CREBRF,0.06933759513315234
-11009,BNIP1,-0.07909147499752892
-11017,BOD1,-0.08871150045801167
-11023,CPEB4,0.1768676612950889
-11039,SFXN1,-0.13941198444242275
-11040,HRH2,0.35162970806166777
-11043,THOC3,-0.150659871802907
-11051,SIMC1,0.012148079701287643
-11052,KIAA1191,-0.019894080518962923
-11056,NOP16,-0.05241471176063997
-11057,HIGD2A,0.08610340523227816
-11058,CLTB,0.02416232674566593
-11059,FAF2,0.03015472060947058
-11060,RNF44,-0.01084136459038185
-11065,TSPAN17,0.1211017537303782
-11070,HK3,0.34479492301608183
-11071,UIMC1,-0.021402235014553437
-11072,ZNF346,-0.03310133714745422
-11075,NSD1,0.05612067411481097
-11077,RAB24,0.21896725869029202
-11078,MXD3,0.19704387688999844
-11079,PRELID1,0.19893588815067298
-11080,LMAN2,-0.1356186637044752
-11081,RGS14,-0.00010161673683425503
-11085,GRK6,-0.017070951216506196
-11086,PRR7-AS1,-0.05944415659203566
-11087,PRR7,-0.13155416122297853
-11089,PDLIM7,0.20327815037725858
-11091,DOK3,0.33347074805877563
-11092,DDX41,0.10635567463648758
-11093,FAM193B,0.014722310214336782
-11095,TMED9,-0.16647269201506815
-11096,B4GALT7,-0.015990567277507466
-11107,FAM153CP,-0.09333714030476857
-11109,RMND5B,0.04086143421443778
-11110,NHP2,-0.12856434515630322
-11112,HNRNPAB,0.08286914468774217
-11113,PHYKPL,0.04642654367720394
-11117,CLK4,-0.015416598810287193
-11120,ZNF354A,0.04921270136933359
-11121,ZNF354B,-0.02278787901988919
-11134,RUFY1,0.08083241978826565
-11136,HNRNPH1,0.02211514095776117
-11140,CANX,0.2913573127728438
-11141,MAML1,0.0011805506122212809
-11142,LTC4S,-0.02200845989088334
-11143,MGAT4B,0.04095115318230873
-11144,SQSTM1,0.2693533028781302
-11145,MRNIP,-0.10104395430625833
-11147,TBC1D9B,0.10674263587791102
-11148,RNF130,0.5504993325310872
-11152,MAPK9,0.014435959731860295
-11156,CNOT6,0.05118348108642177
-11157,SCGB3A1,-0.09997052670755482
-11161,MGAT1,0.2889420098831457
-11162,HEIH,0.03282800023035768
-11163,LINC00847,-0.04266834345205731
-11165,ZFP62,-0.01125515510723453
-11173,TRIM7,0.15331965196667005
-11177,TRIM41,-0.04868157919517649
-11179,RACK1,-0.25424707393903084
-11180,AC008443.1,0.019923188262109637
-11181,TRIM52,-0.15127758789595575
-11182,TRIM52-AS1,-0.11469710538982857
-11191,AL365272.1,-0.0654813693949249
-11192,DUSP22,0.12321331885962675
-11193,IRF4,0.09090946209015646
-11195,EXOC2,-0.01675358789359656
-11215,GMDS,0.0001572142582250036
-11217,GMDS-DT,0.010060280455120724
-11224,WRNIP1,0.039889899941959954
-11225,SERPINB1,0.34462483755570067
-11226,SERPINB9P1,-0.0005386572634956997
-11227,SERPINB9,0.226851796082813
-11229,SERPINB6,0.07428435617521982
-11231,NQO2,0.08937608632916019
-11234,RIPK1,-0.0026137830867676835
-11236,BPHL,-0.07452808264611017
-11241,PSMG4,-0.05015158614994918
-11249,FAM50B,-0.09233210746995017
-11253,PRPF4B,-0.043634510808353864
-11256,ECI2,-0.07656876553948243
-11267,CDYL,-0.03822979064481488
-11270,RPP40,-0.04651823478810567
-11274,LYRM4,0.04894035692019667
-11275,FARS2,-0.05046636544311181
-11280,F13A1,0.11690477401143773
-11281,LY86-AS1,-0.01061651403300217
-11283,LY86,0.4019093580645466
-11291,RREB1,0.17146913376954456
-11294,SSR1,0.23331652266747352
-11297,RIOK1,-0.07011854279984857
-11300,SNRNP48,-0.027063483209351886
-11302,TXNDC5,0.0409181695394987
-11303,BLOC1S5,-0.08787958833759497
-11304,EEF1E1,-0.2176155099817312
-11307,SLC35B3,0.019369359001502377
-11318,GCNT2,0.19807446903132908
-11322,PAK1IP1,-0.06774570463583948
-11323,TMEM14C,0.155329479540436
-11325,TMEM14B,-0.01862010860149528
-11332,SMIM13,0.04127984780050288
-11335,NEDD9,-0.02493865772826464
-11339,TMEM170B,0.3301564782971275
-11341,ADTRP,-0.07558025185631542
-11346,HIVEP1,0.09407933925287597
-11349,PHACTR1,0.019163579553288564
-11351,TBC1D7,0.022501195816621217
-11353,GFOD1,-0.1747954244471815
-11355,SIRT5,-0.018732201178285043
-11357,NOL7,-0.12976268502275415
-11358,RANBP9,0.05409152643608273
-11360,MCUR1,0.04841784400261389
-11364,CD83,0.0828113152999169
-11375,JARID2,0.16752180307460512
-11377,DTNBP1,-0.0017776054385242844
-11380,MYLIP,-0.055555657701375244
-11381,GMPR,0.07620018313620487
-11383,ATXN1,0.11227734380048357
-11394,FAM8A1,0.05996478442838732
-11395,NUP153,0.03716693716919704
-11397,KIF13A,0.2542271660897792
-11399,TPMT,0.16348417991560513
-11400,KDM1B,0.3357327439258133
-11401,DEK,0.19703650393160352
-11402,RNF144B,0.282997377002949
-11408,MBOAT1,0.20734126486854276
-11413,E2F3,0.1899350420689022
-11415,CDKAL1,-0.05550607111957562
-11423,SOX4,0.18380393671355855
-11435,MRS2,-0.03894665547602275
-11439,TDP2,0.06857083158260589
-11440,ACOT13,-0.16849085407387
-11442,C6orf62,0.2243148262296918
-11445,GMNN,-0.03656053600241357
-11447,RIPOR2,0.19729393406116283
-11452,CARMIL1,0.010375909436478467
-11461,TRIM38,0.2794770018388663
-11473,HIST1H1C,-0.08092785144245707
-11474,HFE,0.14423246825761124
-11475,HIST1H4C,-0.3158420390113054
-11477,HIST1H2BC,-0.08497181883902365
-11478,HIST1H2AC,0.04827755148475003
-11479,HIST1H1E,-0.2569550783240646
-11480,HIST1H2BD,-0.05258485219392434
-11482,HIST1H2BE,0.06403558131276957
-11484,HIST1H3D,-0.2690337830694853
-11486,HIST1H2BF,-0.12616255590165196
-11487,HIST1H4E,-0.06794785110575903
-11488,HIST1H2BG,-0.08577552687376576
-11491,HIST1H1D,-0.30055799564734365
-11498,HIST1H4H,-0.011617585712890809
-11500,BTN3A2,-0.1749371125850679
-11501,BTN2A2,0.12853650224030566
-11502,BTN3A1,-0.10836088274463125
-11503,BTN3A3,0.0010036835901295735
-11504,BTN2A1,-0.025381618943381687
-11506,HCG11,0.002067537379040825
-11507,HMGN4,-0.17359354211622924
-11509,ABT1,-0.16739434568866093
-11511,ZNF322,-0.07323635180585257
-11519,HIST1H2AG,-0.11144300401075902
-11521,HIST1H2BK,-0.083675172848063
-11522,HIST1H2AH,-0.08896185002297805
-11527,ZNF184,-0.06154893794783762
-11536,HIST1H3H,0.03193609512932894
-11541,HIST1H2BN,-0.06584442354599132
-11542,HIST1H2AK,-0.10244498776511368
-11543,HIST1H2AL,-0.018456285088178596
-11544,HIST1H1B,-0.027832640805136737
-11552,ZSCAN16-AS1,-0.027224941969419623
-11553,AL121944.1,-0.030435189215415947
-11555,ZSCAN16,-0.08861182404597831
-11557,ZKSCAN8,0.05129489808358666
-11559,ZKSCAN4,-0.016504414127732743
-11561,ZSCAN26,-0.05758380844403336
-11578,TRIM27,0.11791560724000767
-11603,GABBR1,0.17391520790220688
-11607,HLA-F,-0.22730034493794057
-11608,HLA-F-AS1,-0.06415901059510223
-11611,HLA-G,-0.005161636364033287
-11613,HLA-A,-0.12644030550309887
-11615,ZNRD1,-0.16873068163210037
-11616,PPP1R11,-0.00019745808684776047
-11623,TRIM26,0.09020224565215716
-11625,HCG18,-0.09931231065499051
-11626,TRIM39,0.023116089475021105
-11627,RPP21,-0.10244439435155561
-11629,HLA-E,-0.16384184004563826
-11631,GNL1,-0.08563310068595284
-11632,PRR3,-0.003595888217099152
-11633,ABCF1,-0.11616060314825062
-11634,PPP1R10,0.008861502806101543
-11635,MRPS18B,0.029807968196769074
-11636,ATAT1,-0.055317945029788075
-11637,C6orf136,0.058786586359915106
-11638,DHX16,0.037674748127929855
-11639,PPP1R18,-0.006734619076842505
-11640,NRM,0.08559044217011667
-11641,MDC1,-0.09768889664669746
-11643,TUBB,-0.13158489512011418
-11645,FLOT1,-0.06596722501700701
-11647,IER3,0.1592694224250493
-11649,LINC00243,-0.06874426065930547
-11653,GTF2H4,-0.027536969766216862
-11654,VARS2,-0.05808811018442122
-11665,CCHCR1,-0.018540366861316063
-11669,AL662844.4,-0.11717251044962904
-11670,HCG27,0.08931719276701422
-11672,HLA-C,-0.4122193592548705
-11673,HLA-B,-0.06697473725911002
-11677,MICA,-0.010325431524538728
-11678,HCP5,-0.06490755604771822
-11681,MICB,0.09384355344238278
-11683,DDX39B,-0.012436180062345635
-11686,NFKBIL1,0.05615364793150125
-11687,LTA,-0.04580097078591991
-11688,TNF,0.011138854860911169
-11689,LTB,-0.23433321871698196
-11690,LST1,0.5409240723457159
-11691,NCR3,-0.3006064792255714
-11692,AIF1,0.5853156448251817
-11693,PRRC2A,0.07620529444811798
-11694,BAG6,0.23865459608159734
-11695,APOM,0.019649087530381464
-11696,C6orf47,-0.06441691917839973
-11698,GPANK1,-0.056848824563403996
-11699,CSNK2B,-0.0005491540270379827
-11700,LY6G5B,0.1067211655924904
-11702,ABHD16A,0.027764321713739524
-11708,DDAH2,0.06275593285125275
-11709,CLIC1,-0.18676007367745312
-11710,MSH5,-0.0124605469545582
-11714,VARS,0.02817272060844086
-11715,LSM2,-0.2320971788676682
-11716,HSPA1L,0.051756976137314635
-11717,HSPA1A,0.3353227505968411
-11718,HSPA1B,0.10037363870299539
-11719,SNHG32,-0.17258390697226236
-11720,NEU1,0.11078431595063026
-11723,EHMT2,-0.0934966585491835
-11724,C2,0.15896800170141034
-11728,NELFE,-0.009443547986180986
-11729,SKIV2L,0.01228948519311532
-11730,DXO,0.024640579023799392
-11731,STK19,0.022557764349302636
-11739,ATF6B,0.025687609367132058
-11740,FKBPL,-0.05718865340689243
-11745,AGPAT1,-0.027264302469529555
-11746,RNF5,-0.057213310301569384
-11747,AGER,0.041992108127618444
-11749,PBX2,0.1166543411505163
-11750,GPSM3,0.03020388860588708
-11755,HLA-DRA,0.43960153410735775
-11756,HLA-DRB5,0.35522884494380425
-11757,HLA-DRB1,0.3878502119264251
-11758,HLA-DQA1,0.1852044898899332
-11759,HLA-DQB1,0.43106821055994726
-11762,HLA-DQA2,0.09938931594474905
-11764,HLA-DOB,0.02066282880927653
-11765,TAP2,0.06622759995061245
-11766,PSMB8,-0.030856897537992593
-11767,PSMB8-AS1,-0.08798621242250616
-11768,PSMB9,0.022432464122749768
-11769,TAP1,0.16600572960582125
-11770,HLA-DMB,0.4105450898279144
-11771,HLA-DMA,0.39092187576677645
-11772,BRD2,0.02869929828882225
-11775,HLA-DOA,0.040228656903720725
-11776,HLA-DPA1,0.3922130511318096
-11777,HLA-DPB1,0.31755410703491915
-11780,RXRB,0.037328445509694295
-11781,SLC39A7,0.017407026303973172
-11782,HSD17B8,-0.057113235186413584
-11783,RING1,-0.030604240032516435
-11785,HCG25,-0.11852728306783658
-11786,VPS52,0.040267420856655446
-11787,RPS18,-0.40950690558885433
-11788,B3GALT4,-0.1502206565279803
-11789,WDR46,0.018402442044937392
-11790,PFDN6,-0.04252567034922947
-11791,RGL2,0.04649997522328876
-11792,TAPBP,0.001534045348179779
-11793,ZBTB22,0.08515570064680798
-11794,DAXX,-0.06094398336947779
-11797,PHF1,-0.08567586645444292
-11798,CUTA,-0.25124936175538587
-11799,SYNGAP1,-0.0924020224458478
-11802,BAK1,-0.023396448973160098
-11804,ITPR3,-0.07233094724500506
-11805,UQCC2,-0.05866303140914682
-11807,LEMD2,4.440148520017228e-05
-11813,HMGA1,0.08129644775694203
-11814,SMIM29,-0.07178941007285593
-11815,AL354740.1,-0.07954954299227598
-11816,NUDT3,0.18814420337079474
-11818,RPS10,-0.3603488262800206
-11821,ILRUN,0.017692311330843136
-11822,AL451165.2,-0.061554845242276585
-11823,SNRPC,-0.1195153663329829
-11824,UHRF1BP1,0.060483000472832664
-11825,TAF11,0.013121396872692304
-11826,ANKS1A,0.0958289162466081
-11831,ZNF76,-0.09379261372002959
-11832,DEF6,-0.12729606264899165
-11833,PPARD,-0.02029017329485081
-11835,RPL10A,-0.39999279054932707
-11839,FKBP5,0.20424586558962743
-11847,SRPK1,-0.060867695281061764
-11849,MAPK14,0.25916111668770303
-11850,MAPK13,-0.20780752899480628
-11852,BRPF3,-0.06796062961832197
-11855,ETV7,0.17464271783819754
-11858,KCTD20,0.14231206925964707
-11859,STK38,0.07721058161214746
-11860,SRSF3,-0.034665993850797536
-11862,CDKN1A,0.15070580364802058
-11864,RAB44,0.10545155554223078
-11865,CPNE5,-0.004763406957357991
-11868,PPIL1,-0.0718445142575103
-11869,C6orf89,0.07122477462648173
-11872,MTCH1,0.10500726230022837
-11873,FGD2,0.3795543545537403
-11874,PIM1,-0.2305110727332045
-11877,TBC1D22B,0.0544739355825061
-11879,RNF8,-0.0842703441036873
-11880,CMTR1,0.10757534562479602
-11881,CCDC167,-0.20779767777684635
-11886,ZFAND3,0.1466773484806254
-11887,BTBD9,-0.024528721591463312
-11889,GLO1,-0.06939846052013916
-11894,SAYSD1,-0.005985193326146888
-11911,OARD1,-0.0245461571095206
-11914,NFYA,-0.004069581127069538
-11920,TREM1,0.25841042024412436
-11924,FOXP4,-0.011656077486653663
-11926,TFEB,0.1874169790573949
-11932,TOMM6,-0.04977447418381408
-11933,USP49,0.11768560782794761
-11936,MED20,0.0623954790939905
-11937,BYSL,-0.06041364648408359
-11938,CCND3,-0.2006442300121072
-11940,TAF8,0.04465906725577354
-11947,MRPS10,-0.040500588091265256
-11948,TRERF1,0.008252894793613078
-11949,UBR2,0.12312457267647117
-11951,TBCC,-0.1793746544442286
-11952,BICRAL,-0.10784354027938164
-11953,RPL7L1,0.020199908390761693
-11954,C6orf226,-0.080138897237308
-11957,AL035587.2,-0.06393564250255024
-11958,CNPY3,0.36830506719260525
-11961,PEX6,-0.04116800509046296
-11962,PPP2R5D,-0.037275658644011445
-11963,MEA1,-0.1505748441350925
-11964,KLHDC3,-0.015585460581387304
-11965,RRP36,-0.08296710143941888
-11967,CUL7,0.012503745447659314
-11968,KLC4,0.05793300152621204
-11969,MRPL2,-0.027709267322542336
-11972,SRF,0.06896375959446287
-11973,CUL9,0.02099311945477612
-11975,DNPH1,-0.009421400192511545
-11979,ZNF318,0.06901945196521557
-11982,ABCC10,0.0003646692927031053
-11984,TJAP1,-0.005142648396769716
-11986,POLR1C,-0.05375416492775467
-11987,YIPF3,0.14441469442288898
-11989,XPO5,0.007613556586484379
-11990,POLH,-0.06485008488460954
-11992,GTPBP2,0.1439816146319614
-11993,MAD2L1BP,-0.13934291642898103
-11995,MRPS18A,-0.05806566951424983
-11997,VEGFA,0.19038158265827732
-12006,MRPL14,-0.005205230918064725
-12007,TMEM63B,-0.037562260101882135
-12010,SLC29A1,0.10456251689343035
-12011,HSP90AB1,-0.23430476637559525
-12012,SLC35B2,-0.1063950418863676
-12013,NFKBIE,0.1277587928996387
-12016,AARS2,-0.010955800104834278
-12018,CDC5L,0.07371754425607792
-12021,SUPT3H,-0.10425075916445478
-12022,RUNX2,-0.02905881043402387
-12023,AL096865.1,-0.09649248001709412
-12027,ENPP4,-0.16295410144900016
-12035,PLA2G7,0.21838439533638263
-12045,CD2AP,0.00868338322192783
-12053,MMUT,0.08092750914899755
-12054,CENPQ,-0.03368655177766245
-12080,MCM3,-0.1392645768887951
-12081,PAQR8,0.05449344538080491
-12082,EFHC1,0.04860014288652461
-12083,TRAM2,0.05301168682128765
-12085,TMEM14A,-0.10336328625916014
-12091,ICK,-0.04467833012642645
-12092,FBXO9,0.09545344306827747
-12095,ELOVL5,0.1282246147058144
-12097,GCLC,0.09603712934781912
-12121,DST,0.21644531805002665
-12124,KIAA1586,-0.01933687885316447
-12125,ZNF451,0.06267216605090126
-12127,BAG2,-0.08814969125056128
-12129,PRIM2,-0.10611836842938192
-12134,AL021368.2,-0.07069777211345367
-12146,PHF3,0.07765717714448168
-12160,LMBRD1,-0.04412596312218934
-12161,COL19A1,-0.0009497305200056002
-12167,FAM135A,0.11388727444450071
-12168,SDHAF4,0.05128664456858182
-12170,SMAP1,-0.07055000855952664
-12171,B3GAT2,-0.12433217180350946
-12175,OGFRL1,0.44487459634023
-12193,CGAS,0.1667668957533457
-12194,MTO1,0.07449790175415633
-12196,EEF1A1,-0.13468604573941695
-12198,SLC17A5,0.09054266105637956
-12205,COX7A2,-0.21340482096503902
-12206,TMEM30A,0.17652580578450178
-12211,SENP6,0.1046426528636844
-12222,PHIP,0.193895525870499
-12224,HMGN3,-0.13130467871458038
-12239,BCKDHB,-0.04002765282179185
-12243,TENT5A,0.33148346029421627
-12250,IBTK,0.16107349799711174
-12253,UBE3D,-0.09768763080213423
-12255,DOP1A,0.03078438920008741
-12256,PGM3,0.010831349210028046
-12264,CYB5R4,0.205565767781044
-12268,CEP162,0.039008028029141495
-12274,NT5E,-0.078156098880204
-12275,SNX14,0.030513357783434544
-12276,SYNCRIP,0.17268057841252013
-12277,SNHG5,0.018565318176196395
-12282,ZNF292,-0.033285011918599075
-12284,SMIM8,-0.15639044727740628
-12287,SLC35A1,0.1293618054672034
-12288,RARS2,-0.10331262697209047
-12289,ORC3,-0.039448148646386544
-12290,AKIRIN2,0.2424139043681663
-12296,RNGTT,-0.07894527095470326
-12298,PNRC1,-0.11001955788983918
-12301,PM20D2,-0.04296371493162517
-12304,UBE2J1,0.34577344017362555
-12305,RRAGD,0.14833142107058603
-12308,LYRM2,0.014680365125971873
-12309,MDN1,-0.06764237392511098
-12311,CASP8AP2,-0.09650806935212407
-12313,BACH2,-0.09327603122348106
-12317,MAP3K7,0.03953306658770415
-12330,MANEA-DT,-0.014721755994818838
-12331,MANEA,0.04354058073080794
-12335,UFL1,-0.06113085085069638
-12339,NDUFAF4,-0.14466636092894108
-12341,MMS22L,0.025720654353611566
-12347,FBXL4,0.05866914114378326
-12350,PNISR,-0.1286565510170302
-12352,USP45,-0.0016593713971922038
-12354,CCNC,-0.12396374742724302
-12360,ASCC3,0.07739977727757169
-12367,HACE1,-0.055873844811875344
-12374,PREP,-0.010663024634884263
-12379,PRDM1,-0.23525671399364217
-12380,ATG5,-0.08799555567355744
-12383,CRYBG1,0.18927391313866462
-12385,RTN4IP1,0.03551442802946081
-12386,QRSL1,0.043203219693010386
-12389,CD24,0.001552616401447988
-12390,MTRES1,-0.08257418076290672
-12392,PDSS2,-0.0016343782659793036
-12395,SCML4,-0.18957029566981654
-12396,SEC63,0.11200148028034013
-12399,OSTM1,0.036764684581119464
-12402,SNX3,0.2081279875952875
-12407,FOXO3,0.12010471576008692
-12412,SESN1,-0.04683716355513902
-12414,CEP57L1,0.04765648725603105
-12415,CD164,-0.014906093964857271
-12418,SMPD2,-0.05656203986947075
-12419,MICAL1,0.0660783838338012
-12420,ZBTB24,-0.12106689420028495
-12422,AK9,-0.019790281346589507
-12423,FIG4,0.14241487494718064
-12428,CDC40,-0.0008581369812742201
-12434,CDK19,0.21165468032991291
-12435,AMD1,0.14303512352721756
-12436,GTF3C6,-0.050275599785319716
-12437,RPF2,-0.08821041810158126
-12440,MFSD4B,0.003747647653473639
-12442,REV3L,0.08395532266862046
-12445,TRAF3IP2-AS1,-0.04507006415693108
-12447,TRAF3IP2,0.006663300676140891
-12449,FYN,-0.2287649042396327
-12452,TUBE1,-0.09933106929362796
-12468,MARCKS,0.4748309081872568
-12471,HDAC2,0.1055641924308629
-12480,NT5DC1,-0.04709973868739114
-12483,TSPYL4,-0.08694560049554403
-12484,DSE,0.2566638928913937
-12485,TSPYL1,-0.0013594555210165134
-12487,CALHM6,0.4336534525343163
-12492,RWDD1,-0.19078071642411573
-12494,ZUP1,0.07668811570785575
-12495,KPNA5,-0.05483883503673501
-12503,DCBLD1,0.007633514349912768
-12504,GOPC,-0.05611543852231789
-12505,NUS1,0.05548648331116138
-12507,CEP85L,0.03686972632321423
-12510,MCM9,-0.010854234210688448
-12511,ASF1A,-0.08863618455415352
-12514,MAN1A1,-0.09113933750364449
-12518,TBC1D32,-0.04584081747102371
-12520,HSF2,-0.10242285019377982
-12521,SERINC1,0.1373363552493411
-12527,SMPDL3A,0.1756527264258438
-12537,RNF217,0.18704797747010413
-12539,HDDC2,-0.21202859407984442
-12545,NCOA7,-0.11156067056330013
-12547,HINT3,0.11644864395035111
-12548,TRMT11,-0.12944123278590466
-12549,CENPW,0.1602983352972755
-12552,RNF146,0.1309539846771969
-12553,ECHDC1,0.16438285587861656
-12558,THEMIS,-0.20151626970711947
-12565,ARHGAP18,0.05371064507934863
-12567,L3MBTL3,0.010271932460842846
-12569,SAMD3,-0.4880562324908462
-12573,EPB41L2,0.11115947210955159
-12574,AKAP7,-0.021463098022222352
-12576,MED23,0.05108061337318742
-12587,STX7,0.314733230009896
-12595,VNN1,0.1785492300149834
-12596,VNN3,0.17656904232706824
-12597,VNN2,0.28407707586933145
-12599,SLC18B1,0.10985739638124466
-12600,RPS12,-0.4234553996684834
-12610,TBPL1,-0.15729905330912092
-12613,SGK1,0.28966492099374685
-12624,HBS1L,-0.07732743895298481
-12629,AHI1,-0.11459731521264878
-12640,BCLAF1,0.13596644713763312
-12644,MAP3K5,0.20477774850063574
-12646,AL121933.2,0.05763045494847853
-12647,PEX7,-0.02978744231502777
-12653,IFNGR1,0.3178221529288002
-12658,WAKMAR2,-0.0013973350753450908
-12659,TNFAIP3,0.1716545830914423
-12667,HEBP2,0.19214067161304516
-12669,AL590617.2,0.13581373639574335
-12671,CCDC28A,0.028317981325394734
-12673,REPS1,0.09035918270316869
-12674,ABRACL,0.006962819177918652
-12675,HECA,0.059441001652246125
-12679,CITED2,0.23787480844796816
-12692,VTA1,0.011406749751515554
-12698,HIVEP2,-0.09647206518955533
-12703,AIG1,-0.02581316867239546
-12705,ADAT2,-0.049658999681759665
-12706,PEX3,-0.09837462905337786
-12707,FUCA2,0.2312971392763744
-12708,PHACTR2,0.11781799193457741
-12710,LTV1,-0.007514199718944303
-12712,PLAGL1,0.1725763907591368
-12713,HYMAI,0.12339114904021307
-12714,SF3B5,-0.06942382828219076
-12715,STX11,0.4550476863909979
-12716,UTRN,0.06690367635563772
-12719,EPM2A,0.031040741068740246
-12722,AL356599.1,-0.04613289597386885
-12724,FBXO30,0.06525406353742327
-12725,SHPRH,-0.01906736362810956
-12727,RAB32,0.336939313743553
-12734,STXBP5,-0.018344292428998397
-12743,SASH1,0.19017107021895702
-12748,TAB2,0.14801341723442382
-12751,SUMO4,-0.06337180209376386
-12752,ZC3H12D,-0.023491470409928453
-12753,PPIL4,-0.06553510867868588
-12755,GINM1,0.1274797536820652
-12758,KATNA1,-0.05926633025373322
-12759,LATS1,0.028293050275495536
-12761,NUP43,0.008530212704371688
-12762,PCMT1,0.09646246233058732
-12786,ZBTB2,0.015064667460426482
-12787,RMND1,-0.07931568370330895
-12788,ARMT1,0.1182054373550856
-12789,CCDC170,0.20669144427626765
-12794,SYNE1,-0.28820077550534734
-12801,FBXO5,-0.06685525636255213
-12802,AL080276.2,0.01060973880776447
-12803,MTRF1L,0.02303761167228734
-12808,IPCEF1,-0.014998211188250555
-12812,SCAF8,-0.043803355013312444
-12813,TIAM2,-0.022072059022330507
-12815,TFB1M,-0.03539282642958573
-12823,ARID1B,0.07149011048964438
-12827,TMEM242,-0.09830044410013439
-12830,ZDHHC14,-0.07366520213839733
-12831,SNX9,0.03747486182405059
-12835,SYNJ2,-0.17928397095035187
-12838,SERAC1,0.03207157497870114
-12839,GTF2H5,-0.08769267014192501
-12840,TULP4,-0.13167126629942805
-12844,TMEM181,-0.056646049108006316
-12845,DYNLT1,0.33180523118051397
-12846,SYTL3,-0.28542678017482775
-12847,EZR,-0.07874008048397793
-12851,RSPH3,0.15543487748410983
-12852,AL035530.2,-0.031359395621327994
-12853,TAGAP,-0.07024354902391727
-12863,SOD2,0.43859754649079913
-12864,WTAP,0.03975936473095453
-12866,ACAT2,-0.06232817359940024
-12867,TCP1,-0.051505702541987754
-12868,MRPL18,-0.09663811176004312
-12872,IGF2R,0.01439890502964196
-12887,MAP3K4,-0.05382735882965386
-12888,AGPAT4,0.039142591561966984
-12900,CAHM,0.036244325589924804
-12901,QKI,0.4154516194538947
-12924,SFT2D1,0.002669184311372087
-12926,MPC1,-0.023237355869822316
-12933,RNASET2,0.3154937726874482
-12936,FGFR1OP,0.016204969528314763
-12937,CCR6,-0.0315531428400254
-12985,WDR27,0.00814987450248758
-12986,C6orf120,0.051774472317391816
-12987,PHF10,0.022142878289382786
-12990,TCTE3,0.014989674646892104
-12991,ERMARD,-0.08013264721407924
-13005,FAM120B,0.00024412687895734633
-13008,PSMB1,-0.11293757176799415
-13009,TBP,-0.009934978566937761
-13010,PDCD2,-0.11003415218921925
-13021,AC093627.4,0.055556603945071475
-13025,FAM20C,0.2322615888032983
-13034,PRKAR1B,0.02493623656424336
-13036,AC147651.4,-0.05671743579211542
-13037,DNAAF5,0.03488202485098887
-13038,SUN1,0.031924444241578115
-13040,AC073957.3,0.0373281656692187
-13041,ADAP1,0.08554667386125643
-13042,COX19,0.022916832863271334
-13044,C7orf50,0.016550341315539063
-13051,ZFAND2A,-0.018971431966819847
-13058,INTS1,0.060391823842752726
-13060,MAFK,0.08332639439414509
-13062,PSMG3,-0.0317697284027638
-13069,MAD1L1,-0.14060205977739632
-13071,MRM2,0.05250564297193384
-13072,NUDT1,-0.00999738064026218
-13073,SNX8,0.1644771041532406
-13074,EIF3B,0.09832587172870273
-13075,CHST12,-0.23056320927495658
-13077,LFNG,0.19071792474134822
-13078,BRAT1,0.07134435039353659
-13079,IQCE,0.10223838486971337
-13080,TTYH3,0.33209479823542193
-13082,GNA12,0.08012815678794863
-13083,CARD11,-0.17930395908551067
-13093,FOXK1,-0.011580399117005742
-13094,AC092428.1,0.03517530622981927
-13095,AP5Z1,0.10099048684338904
-13099,RBAK,0.04595714061469482
-13101,WIPI2,0.12173575186588752
-13103,TNRC18,0.27370138159415336
-13109,FBXL18,0.04324395564083048
-13111,ACTB,0.38205674383007293
-13114,RNF216,-0.059868111988492054
-13117,CCZ1,0.045186231768016595
-13119,PMS2,-0.0594055274248284
-13120,AIMP2,0.10071852292720263
-13121,EIF2AK1,-0.010159069944090722
-13124,USP42,-0.01897702008246388
-13125,CYTH3,-0.06302706596254802
-13128,RAC1,0.45854287515051273
-13129,DAGLB,0.13996083518757726
-13130,KDELR2,-0.13639944211103633
-13132,ZDHHC4,0.0656511022075625
-13134,C7orf26,-0.02865213033952365
-13136,ZNF316,0.0928786274943815
-13138,ZNF12,0.00357701963597003
-13140,CCZ1B,0.00473937350415067
-13142,C1GALT1,0.020105652220666445
-13147,MIOS,0.0776924013337717
-13149,RPA3,-0.1789940785626688
-13150,UMAD1,0.05973604827489943
-13153,GLCCI1,-0.22721267409249565
-13166,NDUFA4,-0.20905486678005686
-13168,PHF14,-0.013689772789620844
-13172,TMEM106B,-0.046062834012108984
-13178,ARL4A,0.09154496634235505
-13199,ANKMY2,-0.03507757917531693
-13201,BZW2,0.011985075739520058
-13203,TSPAN13,0.0008167856157863705
-13206,AHR,0.3694582116275263
-13207,AC073332.1,-0.03910147083894978
-13211,SNX13,0.17247276600364728
-13213,HDAC9,0.22164518972137381
-13222,TWISTNB,-0.1314144875651408
-13237,SP4,-0.12107499641456825
-13239,CDCA7L,-0.012610837299446051
-13246,TOMM7,-0.3439058501110881
-13249,FAM126A,0.10891301756800502
-13251,KLHL7,-0.025447699427020066
-13252,NUP42,-0.0819814790851332
-13256,MALSU1,0.00651779311070209
-13257,IGF2BP3,0.08392942536585951
-13258,TRA2A,-0.014096018988383108
-13259,CCDC126,0.1141750553231946
-13260,FAM221A,-0.0986167224648322
-13267,MPP6,-0.06936673597050731
-13270,OSBPL3,-0.06913830165627287
-13271,CYCS,-0.06623433293197378
-13272,C7orf31,-0.006002265911625004
-13283,NFE2L3,0.11634317200695846
-13284,HNRNPA2B1,0.12035220041343064
-13285,CBX3,-0.01487742122278506
-13287,SNX10,0.47774073152725155
-13294,SKAP2,0.4588597141580946
-13296,HOTAIRM1,0.22042043209364914
-13319,HIBADH,0.008944644576935041
-13322,TAX1BP1,0.23114119313077708
-13323,JAZF1,0.06423682149437063
-13326,CREB5,0.363138659063299
-13331,CPVL,0.5212343486809282
-13335,CHN2,0.16355557557632427
-13341,SCRN1,0.09421849576236582
-13343,FKBP14,-0.013692038618007416
-13344,PLEKHA8,-0.024029877165050763
-13347,MTURN,0.06448572857490546
-13351,ZNRF2,0.08482362583932018
-13352,LINC01176,0.021507848657618938
-13353,NOD1,0.07002693277771016
-13355,GGCT,-0.043415686221850824
-13356,GARS-DT,0.21048851003031788
-13359,GARS,0.07157663585961666
-13374,LSM5,-0.10042645468216331
-13375,AVL9,0.0948377613785641
-13379,KBTBD2,0.0691291236042368
-13381,NT5C3A,0.08882576255975812
-13382,RP9,-0.03120151929627315
-13383,BBS9,0.0038772142885927017
-13393,DPY19L1,0.0815263143520744
-13401,HERPUD2,0.007327860358877633
-13402,AC018647.2,-0.006761125603123456
-13405,SEPTIN7,-0.38173408972323714
-13411,EEPD1,0.034678585830476324
-13417,AOAH,0.23809054463548285
-13423,ELMO1,0.0852785833851046
-13425,GPR141,0.044497046749514886
-13427,NME8,0.0020187635115578167
-13429,STARD3NL,-0.19264911904750737
-13430,TRGC2,-0.35382316635441435
-13433,TRGC1,-0.3171340194063383
-13446,TRG-AS1,-0.3632820477838271
-13454,VPS41,0.13229707832846485
-13459,YAE1,-0.10772569948034831
-13462,RALA,-0.015999362286110364
-13466,CDK13,0.01415233478641313
-13467,MPLKIP,-0.04520005560353257
-13479,C7orf25,-0.022392749901879564
-13480,PSMA2,0.00959389282547803
-13481,MRPL32,-0.10280332124807672
-13487,STK17A,-0.48025602268786344
-13488,COA1,-0.019931012968653575
-13489,BLVRA,0.35720472351173194
-13490,MRPS24,-0.035646622601907506
-13491,URGCP,-0.01696908146969466
-13492,UBE2D4,0.02463899122678616
-13494,AC004951.1,0.11369354096799017
-13496,DBNL,0.1922798795133673
-13499,POLM,-0.02901925417380457
-13501,POLD2,-0.0838999005426601
-13504,YKT6,0.03675912142208982
-13506,NUDCD3,-0.08245147972249174
-13508,DDX56,-0.036743788838676406
-13509,TMED4,-0.18317589432738562
-13511,OGDH,0.16152404138528256
-13512,ZMIZ2,0.06658269053058326
-13514,PPIA,-0.15930174280975545
-13515,H2AFV,0.017323903650532654
-13517,PURB,0.12042668708283341
-13519,AC004847.1,0.024059196336367855
-13520,MYO1G,0.1635237254292081
-13521,SNHG15,-0.0617514575991848
-13522,CCM2,-0.03864558221355197
-13524,TBRG4,0.020341639926629736
-13543,TNS3,0.3280933108171088
-13549,HUS1,-0.0035828449894639985
-13552,UPP1,0.10453774242719835
-13566,IKZF1,-0.19361835467371585
-13568,FIGNL1,-0.03663828479588089
-13594,SEC61G,-0.09223517611023255
-13599,LANCL2,-0.0057965403154514325
-13600,VOPP1,-0.03782505618723591
-13606,NIPSNAP2,0.18990042673994564
-13607,MRPS17,-0.005804350676632239
-13609,CCT6A,0.041739717112138496
-13610,SUMF2,-0.05075985347045136
-13611,PHKG1,-0.0001923464055490477
-13612,CHCHD2,-0.18648919066675823
-13647,ZNF736,-0.03288382625296454
-13649,ZNF680,-0.06171385701199001
-13652,ZNF107,-0.11654270015674756
-13653,ZNF138,-0.10318626571446785
-13656,ZNF273,-0.02025748356354717
-13659,ZNF117,0.058249738720933524
-13660,ERV3-1,-0.002918440033053542
-13662,ZNF92,-0.06645037041078392
-13665,VKORC1L1,0.009824780175880394
-13666,GUSB,0.17411755898888084
-13667,ASL,0.07238522766868205
-13668,CRCP,-0.06749792920743759
-13671,LINC00174,-0.04112848616938316
-13672,AC008267.5,-0.05681930628990915
-13673,KCTD7,-0.05573911098728714
-13675,RABGEF1,0.03135341554410758
-13676,AC027644.3,-0.03258274084301084
-13678,TMEM248,0.05824688367092614
-13679,SBDS,-0.16730407681821874
-13680,TYW1,-0.12287070678798512
-13682,AC006480.2,-0.09747313157297728
-13692,AUTS2,-0.22314510948419639
-13696,POM121,0.057371341397404296
-13703,NSUN5,-0.10137362466045961
-13707,BAZ1B,0.03472736981198916
-13708,BCL7B,-0.0516704950025438
-13709,TBL2,0.06170408347292175
-13713,DNAJC30,-0.01726633979533826
-13714,BUD23,-0.062378818389358874
-13716,ABHD11,0.07162711429005872
-13724,LIMK1,0.10083731525191442
-13725,EIF4H,0.12875945155935067
-13726,LAT2,0.33553141321080665
-13727,RFC2,0.010379208816570312
-13729,CLIP2,0.10919407736576144
-13732,GTF2I,0.19886416046633953
-13734,NCF1,0.5327004006334234
-13735,GTF2IRD2,0.03733242642054482
-13736,CASTOR2,0.0148573012776657
-13737,RCC1L,-0.0705139621943372
-13738,GTF2IRD2B,-0.05811618648755576
-13744,TRIM73,-0.14730254380261987
-13745,POM121C,-0.06453417316319929
-13747,HIP1,0.15699939248797648
-13750,RHBDD2,-0.01839990844752051
-13751,POR,0.1315505337761809
-13752,TMEM120A,0.11010895276950292
-13753,STYXL1,0.18093259466539155
-13754,MDH2,-0.009357368586367846
-13756,HSPB1,-0.11047216307320828
-13757,YWHAG,0.2766447885381468
-13760,DTX2,0.13914190534948273
-13764,POMZP3,-0.07635218163447757
-13768,CCDC146,-0.09994539837578088
-13769,FGL2,0.6408791505041104
-13771,GSAP,0.1290372255358975
-13774,PTPN12,0.010744985067530547
-13775,APTR,-0.06781869615263801
-13776,RSBN1L,-0.1596158260403237
-13777,TMEM60,0.055175808035912315
-13778,PHTF2,0.003300825573823123
-13791,CD36,0.5343755959160029
-13793,SEMA3C,0.1493745489334839
-13795,HGF,0.2312998305174824
-13800,SEMA3E,-0.11978819525169808
-13812,DMTF1,0.02191270710346686
-13813,TMEM243,-0.13648105634657154
-13814,TP53TG1,-0.20404995175556678
-13815,CROT,-0.10419945200379917
-13817,ABCB1,-0.17516320057218437
-13819,SLC25A40,0.2037877699774616
-13820,DBF4,-0.13570257697511223
-13822,SRI,-0.10048823426489123
-13825,STEAP4,0.26218955372886527
-13838,GTPBP10,-0.07688880005840153
-13840,CDK14,0.06301461930542789
-13843,FZD1,0.1788272559716132
-13847,MTERF1,-0.0008208408218580085
-13849,AKAP9,0.07111690264506179
-13850,CYP51A1,-0.03562050557711421
-13853,KRIT1,-0.04927419415348741
-13855,ANKIB1,0.08108811628988165
-13856,GATAD1,0.20792244730110035
-13857,AC007566.1,-0.011397946985350123
-13859,PEX1,-0.09296440305975309
-13860,RBM48,-0.11916206266445561
-13861,FAM133B,0.02204711831976267
-13862,CDK6,-0.1129935861017938
-13866,SAMD9,0.0652148055741235
-13867,SAMD9L,0.2493869946134059
-13869,VPS50,-0.04129098115763396
-13875,BET1,-0.0403296237956616
-13882,CASD1,-0.07333942389445777
-13889,PON2,-0.008045297159997434
-13894,PDK4,0.15650497449676518
-13896,SLC25A13,0.10741465966721418
-13897,SEM1,-0.05391049775483415
-13901,SDHAF3,-0.1411683037118089
-13904,AC079781.5,0.002288204763999821
-13906,LMTK2,0.08987146022181361
-13908,TECPR1,0.07806135687573329
-13909,BRI3,0.5929423390012232
-13914,TRRAP,0.02743785636133895
-13916,SMURF1,0.06981955090393477
-13919,ARPC1A,0.08769230095803439
-13920,ARPC1B,0.3086372795216774
-13921,PDAP1,-0.1282790592106239
-13922,BUD31,-0.08225120989600748
-13923,PTCD1,0.029807706707495753
-13924,CPSF4,-0.0547693909537511
-13925,ATP5MF,-0.04287208209624747
-13926,ZNF789,-0.0695408068426332
-13927,ZNF394,-0.03099162644240946
-13928,ZKSCAN5,0.008738243813965072
-13929,FAM200A,-0.12902858858494465
-13930,ZNF655,0.05984765273617852
-13932,ZSCAN25,-0.0441937167031541
-13939,TRIM4,-0.022661218534172634
-13945,ZKSCAN1,0.07996386510366887
-13946,ZSCAN21,-0.02260175091572192
-13947,ZNF3,0.028565101615624575
-13948,COPS6,-0.1674081870551585
-13949,MCM7,-0.0818769470243342
-13950,AP4M1,0.04948378656432484
-13951,TAF6,-0.061503161010755175
-13953,CNPY4,-0.14333556947663975
-13956,LAMTOR4,0.21805762564846207
-13957,MAP11,-0.004226141002199539
-13960,STAG3,-0.011306846022116545
-13964,PILRB,-0.16821324265236162
-13965,PILRA,0.4919103834617942
-13967,ZCWPW1,0.03613880317513852
-13968,MEPCE,0.10715944338540778
-13970,PPP1R35,-0.1990147071911632
-13973,TSC22D4,-0.020432650414041364
-13978,SAP25,0.08159396220038069
-13979,LRCH4,0.14598431837259854
-13983,MOSPD3,-0.0946569817114081
-13986,GNB2,0.30466144149555496
-13987,GIGYF1,0.029386805159284168
-13988,POP7,-0.011143089038235495
-13992,SLC12A9,0.20815620731247472
-13994,TRIP6,0.001360815656005385
-13995,SRRT,-0.003079208758314381
-14003,TRIM56,0.17687374307960887
-14005,AP1S1,0.03909386171336043
-14009,PLOD3,0.04645250025387641
-14010,ZNHIT1,-0.02143176058367997
-14011,CLDN15,0.0526722052240324
-14012,FIS1,-0.08164793301539988
-14015,IFT22,-0.10590793029129655
-14019,CUX1,0.24293822591740677
-14021,AC005072.1,0.06529591150535306
-14023,SH2B2,0.12781444406532586
-14025,PRKRIP1,-0.09475863344725634
-14027,ORAI2,0.06438537998066217
-14028,ALKBH4,-0.0499956562690127
-14029,LRWD1,-0.007353733965700329
-14030,POLR2J,-0.06408326729733639
-14034,POLR2J3,0.060870425009104216
-14035,RASA4,-0.002952021711425245
-14037,UPK3BL1,0.0196561354134744
-14039,POLR2J2,0.009991237311547193
-14041,FAM185A,-0.013415790707059738
-14046,ARMC10,-0.06103258364154545
-14047,NAPEPLD,-0.02024132266840311
-14048,PMPCB,-0.00614956727986268
-14049,DNAJC2,0.01632741409619347
-14050,PSMC2,0.01955316813909229
-14054,ORC5,-0.02129991783976482
-14058,AC007384.1,-0.01632513136310346
-14059,LINC01004,-0.02767343252790393
-14060,KMT2E-AS1,-0.0710341531922858
-14061,KMT2E,-0.024959198499096193
-14063,SRPK2,-0.19492195540335513
-14065,PUS7,-0.04898906927161266
-14066,RINT1,-0.00914931382268785
-14069,ATXN7L1,-0.05033523742551421
-14071,SYPL1,0.022521638720633664
-14072,NAMPT,0.38806139430295516
-14078,CCDC71L,0.020285162975084342
-14080,PIK3CG,0.13923076427406145
-14081,PRKAR2B,0.08028384905492163
-14083,HBP1,0.08886878539621168
-14085,COG5,0.07031452198000675
-14087,DUS4L,-0.027106601768869906
-14089,BCAP29,0.13112143907041318
-14092,AC002467.1,0.02556802325730258
-14093,CBLL1,-0.061543131910136675
-14095,DLD,0.08998098980312624
-14100,PNPLA8,0.08696200050518646
-14101,THAP5,-0.0777712933740775
-14102,DNAJB9,-0.0945451712530431
-14110,IMMP2L,-0.0503261010247636
-14112,LRRN3,-0.07736249111713876
-14116,ZNF277,0.03741577017150385
-14118,IFRD1,0.1576847321167617
-14121,TMEM168,0.014014874707286152
-14122,BMT2,0.08487201655343803
-14126,SMIM30,-0.05312816828228915
-14132,MDFIC,-0.10761289876467431
-14138,TFEC,0.420304083668713
-14139,TES,-0.08926242503920472
-14149,CAPZA2,0.3939500064162285
-14151,ST7,-0.03426944461116379
-14165,LSM8,-0.07293313794426189
-14174,ING3,0.011112445050813272
-14175,CPED1,0.21894357324654076
-14178,FAM3C,-0.05209682607331782
-14194,NDUFA5,-0.15153445962313422
-14198,WASL,0.06811772938384414
-14199,AC006333.2,0.033606446485685136
-14210,POT1,0.0021641589926755496
-14220,ZNF800,-0.018586213275143648
-14223,GCC1,-0.044692751084875595
-14224,ARF5,0.03452085948559413
-14228,SND1,0.11601726620733523
-14229,SND1-IT1,0.11153026537565867
-14230,LRRC4,0.040377203517164166
-14233,RBM28,-0.06519586984645273
-14235,IMPDH1,0.2853518213217939
-14237,METTL2B,0.024294213904573658
-14238,AC090114.2,0.04389096918162272
-14243,CALU,0.05775247251888534
-14249,ATP6V1F,0.11741899706299838
-14251,IRF5,0.33174177892935025
-14252,TNPO3,0.06887141489109071
-14254,TSPAN33,0.06042561755158105
-14257,AHCYL2,0.043967298235094314
-14261,AC078846.1,-0.014526590555504234
-14262,NRF1,-0.023180308778967127
-14265,UBE2H,-0.019470461750245305
-14267,ZC3HC1,0.015391276358088344
-14268,KLHDC10,0.07846440695843258
-14270,TMEM209,-0.029990828781915592
-14277,CEP41,-0.01362219475239565
-14282,COPG2,0.027315486268521438
-14290,LINC00513,-0.12699672856809557
-14291,AC016831.1,-0.017459692229993735
-14292,AC016831.5,0.0020827109176350406
-14293,AC058791.1,-0.16967102386058483
-14294,LINC-PINT,-0.10593307812620817
-14295,MKLN1,-0.012103680059401442
-14305,CHCHD3,0.004937235520289676
-14307,EXOC4,0.011204999071187714
-14310,SLC35B4,-0.07135203218894119
-14313,AKR1B1,-0.10428154418504275
-14316,BPGM,-0.00825639654394721
-14321,CYREN,-0.0453821119208539
-14322,TMEM140,0.07450175748230417
-14324,AC083862.3,0.059929383767026424
-14325,WDR91,0.06011754933892031
-14328,CNOT4,-0.05744915547134006
-14329,NUP205,0.04411404177493006
-14330,STMP1,0.1502721266209643
-14335,MTPN,0.17113039584047954
-14345,CREB3L2,0.025638277114908535
-14349,TRIM24,0.0600548840167011
-14355,ZC3HAV1,-0.004929727179426924
-14357,UBN2,0.1137620644118442
-14358,FMC1,-0.030290653241511194
-14359,LUC7L2,0.01999734855048235
-14363,HIPK2,-0.046808218469810536
-14364,TBXAS1,0.4057193078876284
-14365,PARP12,0.17774342149683162
-14366,KDM7A,0.27298286056295945
-14367,KDM7A-DT,-0.0050453024608292174
-14368,SLC37A3,1.7326317287833265e-05
-14370,MKRN1,0.15874006310418826
-14373,ADCK2,0.05129410688732933
-14374,NDUFB2,-0.20812068235811432
-14376,BRAF,0.023790393942656893
-14377,MRPS33,-0.11200742756758769
-14384,AGK,-0.144123919471838
-14386,DENND11,-0.07880661936272804
-14390,SSBP1,-0.2598444456130206
-14397,CLEC5A,0.1464791667331892
-14473,TRBC1,-0.34335317890229017
-14482,TRBC2,-0.4495940349105629
-14484,EPHB6,-0.006222053489366056
-14497,GSTK1,-0.08495508114630985
-14500,CASP2,0.012382925908716189
-14504,AC093673.1,-0.03298396036249652
-14505,ZYX,0.28668734909740096
-14507,EPHA1-AS1,-0.04872053713314457
-14514,TCAF1,-0.09160727754051945
-14525,AC004889.1,-0.0569615086458969
-14533,TPK1,0.0849477392364641
-14543,CUL1,0.2555377648050709
-14544,EZH2,-0.00437175729736677
-14548,PDIA4,-0.06546746023324009
-14549,ZNF786,-0.10575814142817841
-14551,ZNF398,-0.012081046105539898
-14552,ZNF282,0.01709870745701297
-14553,ZNF212,0.004421068116417172
-14554,ZNF783,0.014445830068486026
-14557,ZNF777,-0.10531129529181038
-14558,ZNF746,0.03820015926029236
-14559,KRBA1,-0.052901508833551776
-14560,ZNF467,0.30335834339183476
-14562,ZNF862,-0.005040193663076114
-14565,ATP6V0E2,-0.15269707239619426
-14573,LRRC61,-0.055573751098293815
-14577,REPIN1,0.06612968907061113
-14578,ZNF775,-0.007939905833975943
-14580,AC073111.4,-0.10494467512741888
-14582,GIMAP8,0.3266484342210795
-14583,GIMAP7,-0.4362809120034142
-14584,GIMAP4,-0.03486247236426622
-14585,GIMAP6,-0.1000483566363222
-14586,GIMAP2,-0.09391203834764296
-14587,GIMAP1,-0.15408854322725124
-14588,GIMAP5,-0.25469082135185506
-14589,TMEM176B,0.6062786219200049
-14590,TMEM176A,0.49298264155668725
-14595,ABCB8,-0.06696521311322191
-14598,CDK5,-0.0563937375529358
-14599,SLC4A2,0.09839324550426784
-14601,FASTK,0.02462534931446743
-14602,TMUB1,-0.1722229701960042
-14603,AGAP3,0.20425075938504234
-14608,ABCF2,0.009449673152185449
-14609,CHPF2,0.058284779483643856
-14610,SMARCD3,0.2739239995891697
-14613,NUB1,0.05293501451344952
-14618,RHEB,-0.02370280367904367
-14619,PRKAG2,0.26479486840716876
-14621,PRKAG2-AS1,-0.05894425888190446
-14624,GALNT11,-0.10337409484808309
-14626,KMT2C,0.1585030891962354
-14627,LINC01003,-0.014673774320867193
-14629,ACTR3B,-0.042611846951298035
-14640,PAXIP1-AS2,0.025454847596540823
-14641,PAXIP1,0.018158999318099053
-14644,PAXIP1-AS1,-0.10861944954458404
-14651,AC144652.1,0.022323968555957606
-14652,INSIG1,0.0014629544317080834
-14663,AC009403.1,-0.08597615753858319
-14664,RBM33,0.01818647717553289
-14671,LMBR1,-0.012901369461386024
-14673,NOM1,0.01075673491378051
-14679,UBE3C,0.08677833986843328
-14681,DNAJB6,0.014600386204787022
-14685,PTPRN2,-0.1253649657589
-14695,ESYT2,-0.12232962188890387
-14696,WDR60,-0.07707410223686335
-14709,FBXO25,-0.014033216868964509
-14712,ERICH1,0.14739166290820835
-14724,CLN8,0.014727549364313086
-14726,AC100810.1,0.04208675497574862
-14732,KBTBD11,0.17836623204709823
-14756,AC016065.1,-0.04499711141239748
-14757,MCPH1,-0.06360578082670582
-14758,ANGPT2,-0.07705791337066427
-14762,AGPAT5,0.08576469302973438
-14811,PRAG1,-0.08986368949422041
-14820,MFHAS1,-0.00901823880054148
-14821,ERI1,-0.0250461061579325
-14822,PPP1R3B,0.06915741805330139
-14831,TNKS,-0.08077596147024398
-14835,MSRA,0.07658149321114542
-14848,PINX1,-0.058883426560885646
-14852,AF131215.6,-0.13628812045829197
-14859,MTMR9,0.07865131699029104
-14866,BLK,-0.004261140118608898
-14873,FDFT1,-0.14056249328361375
-14875,CTSB,0.5638287195418196
-14893,AC068587.4,-0.04207906131479726
-14895,LONRF1,0.12476946017684043
-14911,MSR1,0.12037026457706172
-14915,MICU3,-0.02589664106480993
-14917,ZDHHC2,0.0071099863329955725
-14918,CNOT7,-0.012907562995147412
-14919,VPS37A,0.05461609232490086
-14932,PCM1,-0.04085704760254485
-14933,ASAH1,0.46813410115461807
-14936,NAT1,0.01994627860501322
-14947,CSGALNACT1,0.054785705724431866
-14949,INTS10,-0.0076663793374784775
-14953,ATP6V1B2,0.46890604333902786
-14967,DOK2,0.038517535479958696
-14968,XPO7,0.04222541692686593
-14972,FAM160B2,-0.04071430445790874
-14973,NUDT18,-0.0065378165264758726
-14975,REEP4,0.028156548958161064
-14982,POLR3D,-0.01870639956553032
-14986,PPP3CC,-0.13428508636468467
-14987,AC087854.1,-0.04525630352093478
-14988,SORBS3,-0.039991711567672866
-14991,PDLIM2,-0.06431656932021035
-14992,C8orf58,0.04379833728688639
-14993,CCAR2,0.031149328909388724
-14995,BIN3,0.010349993646950897
-15004,RHOBTB2,0.014804249253153231
-15005,TNFRSF10B,0.19728780361753226
-15008,TNFRSF10C,0.10526896221018568
-15009,TNFRSF10D,0.09379842351446671
-15011,TNFRSF10A,0.03053484439196692
-15012,AC100861.1,0.015225782753031455
-15014,CHMP7,-0.03026848333299535
-15015,R3HCC1,0.006386283470475656
-15018,ENTPD4,0.11927616946969877
-15022,SLC25A37,0.22230814075653663
-15029,ADAM28,0.026111147285910886
-15041,DOCK5,0.24070998207837035
-15044,KCTD9,-0.013491806170431858
-15051,PPP2R2A,0.11610351072140766
-15052,BNIP3L,0.38144564161105604
-15056,DPYSL2,0.4496348353094356
-15061,TRIM35,-0.006352772380694282
-15062,PTK2B,0.27874519650494806
-15064,EPHX2,-0.06672179019398752
-15065,CLU,0.0720999759937889
-15068,CCDC25,-0.1411873395853276
-15074,ELP3,0.019536714948370493
-15076,PNOC,0.02817161926211222
-15077,ZNF395,0.032562999957788406
-15083,EXTL3,0.07368155715686178
-15085,INTS9,0.005887385493624601
-15087,HMBOX1,0.08834881696849982
-15090,KIF13B,0.04024885886390301
-15104,SARAF,-0.19435156999490125
-15105,AC044849.1,-0.040270323763823834
-15106,LEPROTL1,-0.14345933807242886
-15108,AC026979.2,-0.05972517276349259
-15109,DCTN6,0.0700400377772202
-15116,GTF2E2,-0.07100629041220008
-15119,GSR,0.11862764773460868
-15120,UBXN8,0.0391705693106835
-15121,PPP2CB,0.18658180559145357
-15124,WRN,-0.05085849351637301
-15130,NRG1,0.23845954488385399
-15135,AC090204.1,0.08563566208880108
-15138,TTI2,-0.05497235848853798
-15139,MAK16,-0.021643534697808064
-15165,ZNF703,0.24501788075255385
-15168,ERLIN2,0.04254809078119926
-15169,PLPBP,-0.0363282728083786
-15171,BRF2,-0.11656960732130014
-15172,RAB11FIP1,0.355306997188352
-15175,EIF4EBP1,0.2678837908246338
-15179,ASH2L,0.053073802419534376
-15183,LSM1,-0.07603971418851112
-15184,BAG4,0.1422611177558962
-15186,DDHD2,-0.05338596224266712
-15187,PLPP5,0.028211997533375416
-15188,NSD3,-0.0450738590877014
-15199,TACC1,0.14705552817777787
-15202,PLEKHA2,0.14881455287747647
-15203,AC108863.2,-0.028197857780893416
-15205,TM2D2,0.01941621814521725
-15206,ADAM9,0.23273751952496333
-15210,IDO1,0.07067472108003615
-15227,GOLGA7,0.025370106991360206
-15232,GPAT4,0.03289443254728756
-15238,KAT6A,-0.0376926450177663
-15241,AP3M2,-0.12972870425970057
-15244,IKBKB,0.15167223224630075
-15245,POLB,0.10564639787808146
-15247,VDAC3,-0.10929490909334222
-15248,SLC20A2,-0.05906776534236991
-15250,SMIM19,-0.15919032537858258
-15254,THAP1,-0.02359135128141385
-15256,RNF170,0.025513638527093072
-15257,HOOK3,0.29357513350040226
-15258,FNTA,-0.05200813173997912
-15260,HGSNAT,0.15482789200482996
-15269,SPIDR,0.11940187969753385
-15271,CEBPD,0.4730481440089084
-15272,PRKDC,-0.09012880019769175
-15274,MCM4,-0.022326710088419945
-15275,UBE2V2,-0.07861526660881323
-15299,PCMTD1,0.07190565030180229
-15301,AC064807.1,-0.048497161961780805
-15310,RB1CC1,0.17366728599434753
-15319,ATP6V1H,0.07268112490823424
-15322,TCEA1,-0.0676811289364914
-15324,LYPLA1,0.21930143020934556
-15325,MRPL15,0.0002446258412600052
-15334,TMEM68,-0.07442095887650187
-15335,TGS1,-0.06363333837630952
-15336,LYN,0.5111123375735166
-15338,RPS20,-0.21250244930138604
-15342,CHCHD7,-0.031769185065381476
-15350,IMPAD1,0.12104738154267801
-15367,UBXN2B,0.053392523419757365
-15369,SDCBP,0.5458952176419276
-15370,NSMAF,0.06175427155228683
-15379,RAB2A,0.0008551887505038311
-15383,CHD7,-0.0812517991523084
-15387,AC022182.2,-0.04178619732701987
-15391,ASPH,0.054424831585775074
-15406,YTHDF3,0.19710673339517307
-15426,ARMC1,-0.005440031871671812
-15427,MTFR1,0.027812171507230952
-15429,PDE7A,-0.09749587080937559
-15438,RRS1,-0.009484300067401344
-15439,ADHFE1,-0.018931735827987576
-15441,MYBL1,-0.48179434222886086
-15442,VCPIP1,0.14223580582571127
-15443,SGK3,0.1339579347612261
-15445,SNHG6,-0.2259009472186224
-15448,COPS5,-0.07954641159101788
-15449,CSPP1,-0.004656571442109276
-15450,ARFGEF1,0.05941499373329395
-15468,NCOA2,0.16141507510424374
-15472,TRAM1,0.09340087357361417
-15474,LACTB2,0.006265391290975773
-15489,TERF1,-0.10921402067379593
-15495,RPL7,0.022542482188135503
-15501,STAU2,-0.03503103086479262
-15503,UBE2W,0.18980831318943345
-15505,ELOC,0.012457157893260481
-15506,TMEM70,0.03365881168214859
-15507,LY96,0.3104066507976824
-15512,GDAP1,0.07491480755289383
-15531,PEX2,-0.11075148927348354
-15535,PKIA,-0.0809365167635285
-15538,ZC2HC1A,0.08973615138117916
-15549,MRPS28,-0.13323533285331918
-15552,TPD52,-0.07077941222485042
-15556,ZBTB10,0.017435769375845952
-15561,PAG1,0.047475674286944554
-15567,FABP5,-0.04378814887194666
-15574,IMPA1,-0.09628515798481986
-15576,ZFAND1,-0.08115835597064987
-15579,SNX16,0.05496059444168725
-15588,LRRCC1,-0.0384273037189938
-15592,RBIS,-0.09567708082703881
-15599,CA2,0.12263140190946123
-15610,WWP1,-0.02291755971451913
-15611,RMDN1,0.07522744407076012
-15612,CPNE3,-0.07214260176804271
-15624,AF117829.1,0.06273792821733026
-15625,RIPK2,0.0999578422727975
-15626,OSGIN2,0.09185981114889528
-15627,NBN,0.131908304481789
-15628,DECR1,0.012523354850781019
-15633,TMEM64,-0.05444550293371845
-15638,PIP4P2,0.21573260441007322
-15641,OTUD6B-AS1,-0.21218813351942295
-15642,OTUD6B,0.00814545257829341
-15657,TRIQK,0.20675220026431435
-15662,FAM92A,0.026486592311893268
-15665,RBM12B,-0.055833270027567494
-15670,PDP1,0.2431645380805785
-15677,VIRMA,0.003409144576845264
-15682,DPY19L4,-0.005710400908779035
-15684,INTS8,-0.02771899727581531
-15687,NDUFAF6,-0.025913454152980485
-15688,TP53INP1,-0.00012239121662552632
-15693,PLEKHF2,0.15491223925918288
-15701,UQCRB,-0.20864943701059002
-15703,MTERF3,0.018481342406220094
-15704,PTDSS1,0.10470427554307134
-15707,CPQ,0.0887029451073689
-15713,MTDH,0.25509857444716394
-15716,RPL30,-0.38754924128844914
-15719,RIDA,-0.07971427498165501
-15720,POP1,0.00936474075841994
-15721,NIPAL2,0.10184758472264979
-15722,STK3,0.23370397391593656
-15728,VPS13B-DT,0.011130969655228052
-15730,VPS13B,0.03152850839457429
-15734,COX6C,-0.3300557156688226
-15738,POLR2K,-0.14481643843237196
-15740,RNF19A,-0.08740901937344196
-15746,ANKRD46,-0.0880452085044099
-15749,PABPC1,0.580471917136222
-15751,YWHAZ,-0.014898048538747898
-15757,ZNF706,0.043795998398307986
-15766,NCALD,-0.3694654897276612
-15768,RRM2B,0.20820676000197255
-15769,UBR5-AS1,0.11494008819671071
-15770,UBR5,0.0750859325335054
-15774,KLF10,0.24415365044287501
-15777,AZIN1,0.12324913200438335
-15782,ATP6V1C1,0.15653897529352123
-15792,SLC25A32,-0.014953950407730298
-15793,DCAF13,-0.0954737252625998
-15811,OXR1,0.2058963413065251
-15820,EIF3E,-0.007006096349347619
-15822,EMC2,-0.06651113264301926
-15827,NUDCD1,0.04105063852370665
-15829,ENY2,0.18695666909701164
-15831,EBAG9,-0.08198630590722997
-15850,TRPS1,0.2605023182271036
-15855,EIF3H,0.1291093504955743
-15856,UTP23,-0.02509881749521224
-15857,RAD21,0.0077757139540157744
-15863,MED30,-0.09931392505057442
-15864,EXT1,0.14438596293469289
-15877,TAF2,0.016658346372611346
-15884,MRPL13,0.020743723696357713
-15886,SNTB1,0.16535517047013676
-15900,ZHX2,-0.0806703247185642
-15904,DERL1,-0.09224203734398652
-15905,TBC1D31,-0.1374497175768171
-15910,C8orf76,-0.024586374819897314
-15911,ZHX1,0.07574973918507635
-15912,ATAD2,-0.007460607320211881
-15913,WDYHV1,-0.04663037717659234
-15914,FBXO32,-0.11367743053724826
-15919,FAM91A1,0.21887616832155707
-15927,TMEM65,0.051207428741783835
-15928,TRMT12,0.035362382618958556
-15930,RNF139,0.05147933020683845
-15931,TATDN1,-0.07435024811554367
-15933,NDUFB9,-0.10148970399450173
-15934,MTSS1,0.1215410269543963
-15941,SQLE,-0.03595950552741126
-15942,WASHC5,0.01672914946486995
-15944,NSMCE2,-0.032740921208839986
-15946,TRIB1,0.2102773978837797
-15950,LINC00861,-0.41647572386722814
-15968,MYC,-0.06304902816880521
-15969,PVT1,-0.02073077676880838
-15971,LINC00824,-0.08067318261748124
-15978,FAM49B,0.014718177328808132
-15982,ASAP1,0.32601074230497157
-15983,ASAP1-IT2,0.12857676377480987
-15992,EFR3A,0.17480342956553638
-15998,TMEM71,-0.009281772050564564
-16000,PHF20L1,0.2115370380341481
-16004,SLA,0.10152236715134304
-16005,AF305872.2,-0.10800030690727154
-16007,NDRG1,0.02340891835397803
-16008,ST3GAL1,-0.1072513114477305
-16009,AC103706.1,0.006059776210871094
-16015,ZFAT,0.007101128241731751
-16038,TRAPPC9,-0.07774861315514986
-16042,CHRAC1,0.025110146674775482
-16043,AGO2,0.03535543149605497
-16045,PTK2,0.08535707870190515
-16047,DENND3,0.2592345483135793
-16048,AC040970.1,0.0902962255770358
-16049,SLC45A4,-0.009042107457041388
-16058,PTP4A3,0.03362529406251483
-16072,JRK,0.02357515487295564
-16076,THEM6,-0.09191901973363914
-16090,LY6E,0.19411562315529418
-16096,GLI4,-0.0787608297163088
-16097,MINCR,0.0021024051898156644
-16098,ZNF696,0.018022243027139947
-16100,TOP1MT,0.004160720458322825
-16106,ZC3H3,0.06886954374955247
-16107,AC067930.8,0.0853396426349724
-16108,AC067930.3,0.07153998019303666
-16109,GSDMD,0.0641985488923712
-16110,MROH6,0.11553681081862265
-16112,NAPRT,0.15512935728417285
-16114,EEF1D,-0.269885856779195
-16115,TIGD5,-0.07433890412644191
-16117,TSTA3,-0.08387551195820614
-16119,ZNF623,-0.0064148378075763604
-16120,ZNF707,-0.0775369076864922
-16129,SCRIB,0.011698228440835033
-16130,PUF60,-0.05660800779439883
-16133,NRBP2,-0.005675956805606265
-16137,PLEC,0.24281573808191503
-16138,PARP10,0.029477737176169354
-16139,GRINA,0.42863813728358996
-16141,OPLAH,0.10753092689117316
-16143,EXOSC4,0.0998929911649747
-16144,GPAA1,-0.06393757948907151
-16145,CYC1,-0.03807732365863437
-16146,SHARPIN,0.010169135329407155
-16147,MAF1,-0.03777768085088901
-16149,HGH1,-0.03993595886095165
-16150,MROH1,0.09287927195326985
-16151,BOP1,0.05022968511238855
-16153,HSF1,-0.0057466007822539675
-16154,DGAT1,0.1525476113491698
-16158,SLC52A2,-0.08016774596042524
-16159,FBXL6,0.0009166667206273223
-16160,ADCK5,0.05019255965178351
-16161,CPSF1,0.03925162262138767
-16163,SLC39A4,0.010996514042876153
-16164,VPS28,-0.06521226972722952
-16167,CYHR1,-0.011846269724896427
-16169,KIFC2,-0.0026401568148306186
-16171,PPP1R16A,0.03281878101016887
-16174,MFSD3,0.020619323494558153
-16177,LRRC14,-0.038405187986184794
-16179,C8orf82,0.04440355710592018
-16185,ZNF34,-0.0494766986423985
-16186,RPL8,-0.17422767260302557
-16188,ZNF7,-0.042224094305788544
-16189,COMMD5,0.04908373532551785
-16191,ZNF250,-0.01478520530406901
-16192,ZNF16,0.06285205253198688
-16195,C8orf33,-0.1033988775656388
-16196,WASHC1,0.0573022212690693
-16203,CBWD1,0.2310039159436834
-16205,DOCK8-AS1,0.1371486793644156
-16206,DOCK8,0.2640184772090517
-16219,SMARCA2,-0.034748489276527565
-16226,PUM3,-0.06225655023301824
-16228,RFX3,0.0020182841969002777
-16239,PLPP6,-0.03159462516675009
-16241,CDC37L1,-0.12790838535739252
-16242,AK3,-0.0819014191420425
-16244,RCL1,-0.05454299622440559
-16246,JAK2,0.4814347645083829
-16253,PLGRKT,-0.19367940645147122
-16254,CD274,-0.010960684766602771
-16255,AL162253.2,-0.023167542932122035
-16258,RIC1,0.1810394582963478
-16260,ERMP1,-0.13206254493595804
-16261,KIAA2026,-0.006991134923188925
-16263,RANBP6,-0.10169634670013
-16267,UHRF2,0.11971263422903548
-16273,KDM4C,0.06766464376803026
-16278,DMAC1,-0.09826214288958908
-16308,ZDHHC21,-0.020096407671421162
-16313,TTC39B,-0.0710513271255881
-16314,SNAPC3,0.09603151622511592
-16315,PSIP1,-0.20128536128079255
-16324,CNTLN,0.24230619633013958
-16329,RRAGA,0.004344801882255528
-16330,HAUS6,-0.08428914093304345
-16331,PLIN2,0.03974895553571597
-16332,DENND4C,0.01788224963226478
-16336,RPS6,-0.37351426933592863
-16343,MLLT3,-0.10364578234818571
-16344,FOCAD,-0.023787984998558193
-16346,HACD4,0.21850395011143373
-16357,KLHL9,0.017444725531084636
-16366,MTAP,0.01659749900081322
-16394,CAAP1,-0.020123992698479722
-16395,PLAA,0.1737607368518723
-16396,IFT74,-0.02232700218647212
-16402,MOB3B,0.12844214789662317
-16407,C9orf72,0.4122227419502586
-16415,ACO1,0.08887258105640497
-16416,DDX58,0.11339329806235637
-16417,TOPORS,0.09323964556719308
-16418,SMIM27,-0.1752118094218135
-16419,NDUFB6,-0.011482745852226596
-16425,APTX,0.036974665767506694
-16426,DNAJA1,0.20076784128803843
-16427,SMU1,0.029135933618138654
-16428,B4GALT1,0.01510605280992788
-16431,BAG1,0.13000183502301116
-16432,CHMP5,0.10302924421208495
-16433,NFX1,-0.007042922274874806
-16436,AQP3,-0.1667163605314475
-16437,NOL6,-0.025769496195006448
-16457,UBE2R2,0.29174442603188194
-16458,UBAP2,-0.014971044951748861
-16460,DCAF12,0.26013394039358423
-16461,UBAP1,0.06176065746240694
-16463,NUDT2,-0.0027146505497668357
-16466,FAM219A,-0.004461559275290206
-16472,RPP25L,-0.09714609309308718
-16473,DCTN3,-0.09666969180901337
-16475,SIGMAR1,0.020629009847125394
-16476,GALT,-0.04645566868023879
-16477,IL11RA,-0.015805893986901614
-16492,VCP,0.03470796209673025
-16494,PIGO,-0.016479921050741114
-16496,STOML2,-0.10374463172282537
-16497,FAM214B,0.2337674547677745
-16502,TESK1,0.00962918541384455
-16503,CD72,-0.03325087205258261
-16506,SIT1,-0.22559288657161272
-16507,RMRP,0.019314104179592062
-16508,CCDC107,-0.3117959231251323
-16511,TPM2,-0.04518491046225067
-16512,TLN1,0.10480917158077571
-16513,CREB3,-0.0561269651482597
-16514,GBA2,0.07398688996776685
-16515,RGP1,-0.055632548132990525
-16521,HINT2,-0.14438959682965113
-16522,TMEM8B,0.05355625100895447
-16529,RECK,0.02999953362547164
-16530,GLIPR2,0.28261662283216166
-16532,CLTA,0.17552431950190647
-16533,GNE,0.05973781387099065
-16534,RNF38,0.0908373455881667
-16537,PAX5,-0.0011864672393094122
-16542,EBLN3P,-0.07371272481237595
-16544,ZCCHC7,-0.020371950268269722
-16546,GRHPR,-0.03197522192272978
-16547,ZBTB5,-0.10028476421674196
-16548,POLR1E,-0.0951658192271379
-16551,TOMM5,-0.19294736820237232
-16553,TRMT10B,-0.02483556593352457
-16554,EXOSC3,0.00822135689364488
-16555,DCAF10,0.11181040956160501
-16556,SLC25A51,-0.07483551893447792
-16616,LINC01410,0.16769864375286636
-16617,AL512625.3,0.04960059944003158
-16629,CBWD5,0.12523753194871706
-16644,CBWD3,0.08326007778946164
-16654,LINC01506,0.1650143031592395
-16655,PIP5K1B,0.12603188166555473
-16656,FAM122A,0.022299308093889397
-16659,FXN,-0.05872874173579522
-16660,TJP2,0.12152354790336688
-16667,PTAR1,0.03555220389880905
-16673,SMC5,0.0059435498552428455
-16675,KLF9,-0.019466205211129475
-16679,CEMIP2,-0.05585379743251632
-16680,ABHD17B,0.036042592138128736
-16681,C9orf85,-0.034243770175841376
-16686,LINC01504,0.1159265678498077
-16687,ZFAND5,0.42305151192705004
-16690,ALDH1A1,0.312957669512858
-16691,ANXA1,0.16129703562789677
-16699,C9orf40,-0.10975259317989935
-16701,CARNMT1,-0.1366350714635006
-16703,NMRK1,-0.17856077408773727
-16704,OSTF1,-0.18905884260246048
-16706,PCSK5,-0.047304784238119735
-16707,RFK,0.10046353620883045
-16708,GCNT1,0.09878870827491151
-16709,PRUNE2,0.08571358200378429
-16714,VPS13A,-0.11368873561975416
-16717,GNAQ,0.4019054639563037
-16718,CEP78,-0.29794667532944397
-16722,TLE4,0.07777803271468176
-16728,TLE1,-0.10910601906426252
-16741,FRMD3,0.1543350217645114
-16743,IDNK,-0.19171248556065837
-16744,UBQLN1,-0.08219382297095022
-16746,GKAP1,-0.040125239259316776
-16749,KIF27,0.003378370490274408
-16750,C9orf64,0.052794768113914536
-16751,HNRNPK,0.19183360671815375
-16752,AL354733.3,-0.07987674287082884
-16753,RMI1,0.10905001393954541
-16762,AGTPBP1,0.17417630126317596
-16765,NAA35,0.03849357967191493
-16766,GOLM1,-0.015373381271131742
-16769,ISCA1,-0.05756505535725553
-16770,TUT7,0.17004701789532076
-16779,DAPK1,0.27858644253872494
-16780,DAPK1-IT1,0.1106576844739865
-16781,CTSL,0.1486512078664596
-16788,SPIN1,0.008733322080257278
-16793,S1PR3,0.28607846157326317
-16796,CKS2,-0.0056756753369955676
-16797,SECISBP2,0.07470288356804698
-16798,SEMA4D,0.0071893527651549
-16809,SYK,0.45345014635781516
-16818,AUH,-0.11985014831713493
-16819,NFIL3,0.08064728876417362
-16822,SPTLC1,0.08219817420990258
-16824,IARS,-0.032382752284690376
-16825,NOL8,0.022765686997347025
-16833,BICD2,0.16535114391852254
-16835,ZNF484,-0.09143808414089566
-16836,FGD3,-0.0520726151424728
-16837,SUSD3,-0.11422944733985242
-16839,CARD19,0.0976227251103223
-16840,NINJ1,0.23841412455373465
-16845,FAM120AOS,-0.02256203115343713
-16846,FAM120A,0.41531042673999946
-16847,PHF2,0.14871257922196254
-16853,AL158152.1,0.04811755172201807
-16855,ZNF169,0.04523433884308847
-16858,MFSD14B,0.22090964481217795
-16862,FBP1,0.20984672679278452
-16864,AOPEP,-0.015267483165135563
-16872,PTCH1,-0.2632660310146329
-16878,LINC00476,-0.18363326252348044
-16880,ERCC6L2,0.01311042854063016
-16888,SLC35D2,0.0508384094146397
-16892,HABP4,-0.058810925127508545
-16894,PRXL2C,0.03039963194553589
-16896,ZNF510,-0.08670233519162106
-16897,ZNF782,0.009246864908597342
-16898,MFSD14C,0.10981601800024969
-16906,TDRD7,0.142260605336273
-16909,TSTD2,0.02186920112598491
-16910,NCBP1,-0.016142599609168144
-16912,XPA,-0.13487074056852574
-16916,TRMO,0.01593491884316468
-16917,AL499604.1,-0.057697210493837726
-16919,ANP32B,-0.04286385044007101
-16921,NANS,0.18488444957375316
-16922,TRIM14,0.054073917828060765
-16924,TBC1D2,0.21167133193440318
-16932,TGFBR1,-0.11394186061352608
-16934,ALG2,0.06483418568611451
-16935,SEC61B,-0.09122981878795491
-16942,STX17,0.03342289368898685
-16944,ERP44,0.14782931241250516
-16945,INVS,-0.04954284649905045
-16946,TEX10,0.023098166125344346
-16947,MSANTD3,-0.13250026321640937
-16952,MRPL50,-0.02728518664214158
-16953,ZNF189,-0.007840086846918191
-16957,RNF20,0.0361986916076985
-16966,SMC2,-0.01616936236332549
-16976,NIPSNAP3A,0.22480693370986976
-16978,ABCA1,0.12736316667813158
-16982,SLC44A1,0.19896412013448447
-16983,FSD1L,0.0382301288568355
-16985,FKTN,0.0075369331143332954
-16987,TMEM38B,-0.021439565708087464
-16996,RAD23B,0.1680718258619556
-16999,KLF4,0.41803199473962294
-17004,ELP1,-0.021855310614702077
-17005,ABITRAM,0.0149771866544706
-17007,TMEM245,0.0038898638531032798
-17011,PALM2-AKAP2,-0.12031022448229813
-17014,TXN,0.07026800832004533
-17019,LPAR1,0.18924640862437628
-17022,ECPAS,0.11217089275204661
-17026,DNAJC25,-0.004767884957281456
-17027,GNG10,0.2885673389249256
-17029,UGCG,-0.06699347478864658
-17030,AL138756.1,0.03649927353266406
-17031,SUSD1,0.14173505000319556
-17032,PTBP3,0.2386150184399167
-17033,HSDL2,0.01975670649522166
-17035,KIAA1958,0.1360447984600948
-17036,INIP,-0.017313341922795798
-17038,SNX30,0.2440991465546767
-17039,SLC46A2,0.18981932859130737
-17045,SLC31A2,0.46077018602934144
-17046,FKBP15,0.3673622642629094
-17047,SLC31A1,0.29265845718738814
-17048,CDC26,-0.187609474779088
-17049,PRPF4,0.020209913491461194
-17053,HDHD3,-0.05930962968711863
-17054,ALAD,-0.05027148815465617
-17055,POLE3,-0.03507548805607259
-17057,RGS3,-0.06322099911594879
-17061,ZNF618,0.0661063875487482
-17067,AKNA,-0.07120080633778955
-17070,ATP6V1G1,0.0632655283479321
-17071,TMEM268,0.15500213740305638
-17076,TNFSF8,0.005804833039789274
-17094,TLR4,0.4363536197732099
-17095,AL160272.1,0.12335879512286768
-17103,CDK5RAP2,-0.010168005516368536
-17104,MEGF9,0.4237204842606992
-17105,FBXW2,-0.013381850164708753
-17107,PSMD5,0.010009117539642601
-17108,PHF19,-0.023560254152269942
-17109,TRAF1,-0.10676707807155354
-17111,CNTRL,0.11079313435160905
-17112,RAB14,0.1491152760577986
-17113,GSN,0.272751204098456
-17116,STOM,-0.1329712877275919
-17123,NDUFA8,-0.0669437318694824
-17126,RBM18,-0.0800156856109139
-17127,MRRF,0.004255480485594553
-17128,PTGS1,0.07673341177914784
-17145,PDCL,-0.027551780511597578
-17147,RC3H2,0.010101680890106693
-17148,ZBTB6,0.005341507918057116
-17149,ZBTB26,0.038958007270143445
-17152,RABGAP1,0.15418632980743044
-17155,STRBP,-0.06802434654777878
-17157,DENND1A,0.17542852955666693
-17164,NEK6,0.12165729230025779
-17167,PSMB7,0.010246210017396487
-17176,RPL35,-0.42430452662372914
-17177,ARPC5L,-0.4614954470686954
-17178,GOLGA1,0.05447057025401439
-17179,SCAI,-0.03341842846163689
-17180,PPP6C,0.08924778900070039
-17181,RABEPK,0.0015786075045640675
-17182,HSPA5,-0.0026543470884354764
-17184,GAPVD1,0.13013379090671506
-17186,MAPKAP1,0.02484288187971097
-17189,PBX3,0.09042188633664154
-17192,MVB12B,-0.037905756256142335
-17199,ZBTB43,0.020098284803826326
-17200,ZBTB34,0.1333482866527246
-17201,RALGPS1,-0.010573610734245705
-17206,SLC2A8,0.07907346567114239
-17208,RPL12,-0.25253463545464994
-17209,LRSAM1,0.06324761095981357
-17210,NIBAN2,0.20089961920254254
-17213,PTRH1,-0.1891132710024512
-17215,TTC16,-0.25223360800841244
-17216,TOR2A,0.06813422715987175
-17217,SH2D3C,0.09354247736113898
-17219,CDK9,0.022209406398668474
-17220,FPGS,-0.057218156037516496
-17221,ENG,0.015710052461669413
-17224,ST6GALNAC6,-0.1160392102969456
-17225,ST6GALNAC4,-0.06999404863683151
-17228,DPM2,-0.05433136937868011
-17229,FAM102A,-0.17068344127071627
-17231,NAIF1,0.007523029830542751
-17232,SLC25A25,0.0018074757813384232
-17235,PTGES2,0.012583777832665736
-17238,C9orf16,-0.16620956792478705
-17239,CIZ1,0.019668928851991178
-17241,GOLGA2,-0.032792089801065086
-17242,SWI5,-0.030478719328281556
-17243,TRUB2,-0.03075864711868829
-17245,COQ4,-0.043956306655086035
-17246,SLC27A4,0.06934819828989487
-17247,URM1,-0.015646922767620138
-17252,ODF2,-0.030069520897109958
-17254,GLE1,0.08147433081503692
-17256,SPTAN1,-0.1536650372942373
-17261,SET,-0.0886147267999705
-17263,ZDHHC12,0.020959789377359286
-17264,AL441992.1,-0.014988377117970561
-17265,ZER1,-0.06581672181895937
-17266,TBC1D13,0.025061116109697766
-17267,ENDOG,0.025929916804332754
-17268,SPOUT1,-0.05778121264863809
-17270,LRRC8A,-0.06576669038869293
-17272,DOLK,-0.06783950597512671
-17273,NUP188,-0.08600622770337309
-17275,SH3GLB2,-0.1431116624541421
-17276,MIGA2,-0.0015822294292692329
-17278,DOLPP1,0.027757136503004563
-17279,CRAT,0.06998882338183131
-17281,PTPA,0.11924718062674537
-17283,IER5L,0.1460330857704713
-17291,LINC01503,0.2093558700535392
-17294,LINC00963,0.1873502904508959
-17298,NTMT1,-0.04837404168337894
-17300,ASB6,-0.06998771032861047
-17305,TOR1B,0.09885499933847051
-17306,TOR1A,0.03152869682678888
-17307,C9orf78,-0.08935611183400687
-17308,USP20,-0.1299724288007405
-17309,FNBP1,-0.02458964372572022
-17312,GPR107,0.07996713489251031
-17317,FUBP3,-0.043344249602362425
-17319,EXOSC2,-0.02300505342874723
-17320,ABL1,0.10709298995272831
-17326,NUP214,0.44934322043963976
-17328,AL157938.3,-0.04451578249373185
-17329,FAM78A,-0.018771502636346735
-17332,PRRC2B,-0.059560114080283305
-17334,POMT1,-0.017183883988665664
-17336,UCK1,-0.03764293694697955
-17338,RAPGEF1,0.04260546200886125
-17340,MED27,-0.010502879928343369
-17341,NTNG2,0.0732372261464684
-17342,SETX,0.17209584962197397
-17343,TTF1,-0.0016515826312056127
-17346,DDX31,-0.05035030687963848
-17347,GTF3C4,0.016608201370136123
-17351,TSC1,-0.010753250974619575
-17353,GTF3C5,0.0381141052979645
-17355,RALGDS,-0.11494204675651419
-17356,GBGT1,0.17441314393721366
-17359,SURF6,-0.051026093115419016
-17360,MED22,0.013793020705897815
-17361,RPL7A,-0.42641037484390787
-17362,SURF1,0.012672200931069301
-17363,SURF2,-0.13323087743271295
-17364,SURF4,0.09735648797977052
-17366,REXO4,-0.10518253100914689
-17369,SLC2A6,0.32446490054015475
-17377,VAV2,0.13431794837913835
-17378,BRD3OS,-0.03388965227783579
-17379,BRD3,0.16266767991973463
-17382,WDR5,-0.0038193914439589066
-17385,RXRA,0.4535930064561945
-17393,FCN1,0.6623251550744285
-17403,MRPS2,-0.053140071607945115
-17414,CAMSAP1,0.06267241445376809
-17416,UBAC1,0.1443406557689419
-17417,NACC2,0.16325327256006508
-17419,TMEM250,-0.02600658320961421
-17422,QSOX2,0.011837909716217013
-17428,CARD9,0.21145468091456213
-17429,SNAPC4,-0.02575980482172485
-17430,ENTR1,-0.022921036640764832
-17431,PMPCA,-0.0064177941572479075
-17432,INPP5E,0.026474259791812223
-17433,SEC16A,0.07943219561620703
-17435,NOTCH1,0.08010988260474733
-17441,AGPAT2,0.1900083581331242
-17443,SNHG7,-0.053321912981856186
-17448,TMEM141,-0.04403599058531108
-17451,RABL6,0.1473669765051469
-17453,PHPT1,0.010445500568069359
-17455,EDF1,-0.182204429150686
-17456,TRAF2,-0.10845930353272905
-17458,FBXW5,-0.03267706069251738
-17464,PAXX,-0.3636491115526496
-17465,CLIC3,-0.46495419593195675
-17466,ABCA2,-0.20708973916392212
-17467,C9orf139,0.18619537285280308
-17468,FUT7,0.0018966897695336389
-17469,NPDC1,-0.1229610054766528
-17475,UAP1L1,0.1628161987200344
-17476,MAN1B1-DT,0.032149026203244196
-17477,MAN1B1,0.0187090193406489
-17478,DPP7,-0.04908958204458912
-17482,ANAPC2,-0.08353428991116893
-17483,SSNA1,-0.17216597489425084
-17484,TPRN,0.054624385477444325
-17485,TMEM203,-0.06959782081658114
-17486,NDOR1,0.08798229408614168
-17487,BX255925.3,0.04797137121421901
-17492,TUBB4B,0.03190414908803812
-17496,NELFB,0.040927442576703645
-17497,TOR4A,0.18535915842614215
-17501,EXD3,-0.007825933910929917
-17504,NSMF,0.02022501232228107
-17506,MRPL41,-0.06301030776154401
-17507,DPH7,-0.10132361841073532
-17508,ZMYND19,0.05352399025031016
-17509,ARRDC1,0.17371330736344412
-17510,ARRDC1-AS1,-0.02265126461801077
-17511,EHMT1,-0.07483973360620694
-17517,ZMYND11,-0.1331557134043959
-17523,LARP4B,0.22684135386982565
-17526,GTPBP4,0.030836925230149737
-17529,IDI1,-0.144246519596361
-17530,WDR37,-0.04146446816784678
-17550,PFKP,-0.025014736996475037
-17551,PITRM1,-0.0008910640732139745
-17562,KLF6,0.41510776161555424
-17580,AKR1C3,-0.3119510496931781
-17598,ASB13,0.1259938387180018
-17599,TASOR2,0.02963532016547818
-17601,GDI2,0.40594383163638526
-17603,ANKRD16,-0.07188675560219739
-17604,FBH1,0.053032478706055174
-17606,IL15RA,0.1945041542869609
-17609,RBM17,0.06128543748202891
-17610,PFKFB3,0.16905213156491422
-17614,PRKCQ,-0.17706953426915456
-17615,PRKCQ-AS1,-0.286689481023805
-17622,SFMBT2,-0.1036938373187183
-17627,KIN,-0.005474053149997461
-17628,ATP5F1C,0.03814583728438443
-17629,TAF3,-0.011958450728279208
-17630,GATA3,-0.24578285050878979
-17647,CELF2,0.07337387597304555
-17655,USP6NL,0.14127589558210338
-17658,ECHDC3,0.1193851927345667
-17661,UPF2,-0.006851720109430953
-17662,DHTKD1,0.11679205080714881
-17663,SEC61A2,-0.10771287136245178
-17664,NUDT5,-0.12042121098395876
-17665,CDC123,-0.07518577311988882
-17667,CAMK1D,-0.04075669841904801
-17671,OPTN,-0.29803518611190144
-17674,PHYH,0.018514583978413133
-17675,SEPHS1,-0.009703460636304357
-17678,BEND7,-0.00794017247303368
-17680,PRPF18,-0.0010697495920261465
-17681,AL157392.3,0.015289138305637056
-17689,FAM107B,-0.30381419319462916
-17692,HSPA14,0.03823839450138374
-17693,HSPA14,-0.008792636777554716
-17696,DCLRE1C,0.14447249051900987
-17701,RPP38,-0.08055415242847853
-17702,NMT2,-0.15672989025330997
-17706,MINDY3,-0.06864921862236932
-17708,PTER,0.03608884960635379
-17711,RSU1,0.02541818682801264
-17716,TRDMT1,-0.009731682560438491
-17717,VIM-AS1,0.051344889735500825
-17718,VIM,0.441139503792532
-17724,STAM,-0.009501052327159643
-17735,NSUN6,-0.0717693249381971
-17736,ARL5B,0.05532066794897357
-17742,PLXDC2,0.48356234941578824
-17751,MLLT10,0.11583491806178778
-17753,DNAJC1,-0.22235212630798826
-17759,COMMD3,0.020522705618199938
-17760,BMI1,-0.05677578692935239
-17768,PIP4K2A,-0.2132315966249254
-17771,MSRB2,0.18526736413939043
-17775,OTUD1,0.1426686420449799
-17780,ARHGAP21,0.10470099956324087
-17793,APBB1IP,0.004735864568469695
-17795,PDSS1,0.08592269844509427
-17797,ABI1,0.027370211891719155
-17798,ANKRD26,0.005778675967888142
-17799,YME1L1,0.1048900897790491
-17800,MASTL,0.09754383646223705
-17801,ACBD5,0.042918740718071155
-17804,RAB18,0.02148849378402855
-17810,MPP7,0.15936615529008763
-17814,WAC-AS1,0.04296971510201533
-17815,WAC,0.2571914689562798
-17822,SVIL,0.07133611604846567
-17825,MTPAP,0.005804243042680137
-17827,MAP3K8,0.08057148636520227
-17834,ZNF438,0.09758398912503435
-17839,ZEB1,-0.10201401748513858
-17843,ARHGAP12,-0.131904706770046
-17844,KIF5B,0.21077033585230967
-17846,EPC1,-0.10066180040868875
-17851,CCDC7,-0.11040203019657013
-17853,ITGB1,-0.04212412248776227
-17867,CUL2,-0.06085864363059017
-17869,CREM,-0.1679865377259609
-17872,CCNY,0.24769785415400372
-17873,AL117336.2,-0.03832366757854332
-17888,ZNF248,-0.06669690262411529
-17891,ZNF25,0.14089731583297802
-17893,ZNF33A,-0.016197230293094216
-17894,ZNF37A,-0.039420756673049534
-17899,ZNF33B,-0.0773662156993662
-17904,BMS1,-0.04023500301944034
-17909,CSGALNACT2,0.128068776585733
-17910,RASGEF1A,-0.22361568000913185
-17914,HNRNPF,-0.04049612577497503
-17916,ZNF487,-0.01228147015098897
-17922,ZNF32,-0.20298352485940818
-17943,RASSF4,0.3258261122077328
-17946,ZNF22,-0.11134809240409911
-17949,ALOX5,0.44984820228012085
-17951,MARCH8,-0.01560988223990755
-17952,ZFAND4,-0.0050044075444715475
-17953,WASHC2C,0.19364275591273186
-17954,AGAP4,0.014570594747317029
-17955,TIMM23,0.00535457109134166
-17956,NCOA4,0.3501338288333509
-17976,AGAP9,0.009633005662362169
-17985,MAPK8,0.10723692111031974
-17986,ARHGAP22,0.09256029288678874
-17990,WDFY4,0.15393132656838365
-18000,TMEM273,-0.16196381795408593
-18005,ERCC6,-0.04535614182275999
-18010,PARG,0.011197419563643488
-18011,TIMM23B,0.04857971027410341
-18013,AL442003.1,-0.06511508860724605
-18015,WASHC2A,0.11692748652632108
-18017,SGMS1,0.1920891779820649
-18020,SGMS1-AS1,0.009219293251806312
-18030,CSTF2T,0.02130663310857633
-18047,IPMK,0.16537323431743273
-18048,CISD1,-0.09395100687804897
-18051,UBE2D1,0.48068476699338136
-18052,TFAM,-0.040382987017451695
-18062,CCDC6,0.05888483638392855
-18064,ANK3,-0.07198706473741581
-18077,ARID5B,-0.18865912946787375
-18078,RTKN2,-0.06714374516672347
-18084,ADO,0.1556785606060878
-18087,NRBF2,0.19529517627566623
-18088,JMJD1C,0.23108392367163497
-18090,REEP3,0.22165158651182335
-18104,SIRT1,-0.0025828426625690154
-18105,HERC4,0.04821538324786836
-18110,HNRNPH3,0.010137958008189916
-18111,RUFY2,-0.014408442100450747
-18113,SLC25A16,0.006164317076077996
-18116,CCAR1,0.08366495247394905
-18119,DDX50,-0.05254064575415995
-18120,DDX21,0.1991741957108964
-18121,KIF1BP,0.06612614717666722
-18122,SRGN,0.3321637744853449
-18123,VPS26A,-0.025186509889724814
-18124,SUPV3L1,-0.10183990114876369
-18128,HK1,0.2602926885610502
-18140,TYSND1,-0.017503151879502438
-18141,SAR1A,-0.09090309501885459
-18142,PPA1,0.046378260334525544
-18145,EIF4EBP2,0.19668521552582421
-18148,PRF1,-0.6303805567480577
-18151,SGPL1,0.09810589303310394
-18152,PCBD1,-0.002104819897090438
-18158,SLC29A3,0.13300856212571835
-18159,CDH23,0.22190873218122406
-18162,VSIR,0.4937203692200924
-18163,PSAP,0.6918177435832032
-18167,SPOCK2,-0.31907804226686
-18168,ASCC1,-0.027771059738999033
-18169,ANAPC16,-0.2772315530101726
-18170,DDIT4,-0.36914138483742776
-18172,DNAJB12,0.15012493656230166
-18173,MICU1,0.2220302172387926
-18176,MCU,0.05113540097892335
-18180,P4HA1,0.13615881882242187
-18185,ECD,0.05341943836447738
-18186,FAM149B1,0.0636715742220393
-18187,DNAJC9,-0.17150530128797706
-18189,MRPS16,-0.04149518427167257
-18192,ANXA7,0.13944664535410692
-18195,PPP3CB,0.062217206483549524
-18196,PPP3CB-AS1,0.06180024337474125
-18205,SEC24C,0.07661927755319518
-18206,FUT11,-0.14076037267790772
-18207,CHCHD1,-0.04899743647108823
-18208,ZSWIM8,0.051572260213969774
-18210,NDST2,0.04151033269807608
-18211,CAMK2G,-0.10792460447074911
-18216,VCL,0.11026387239728366
-18218,AP3M1,0.022344378715502655
-18219,ADK,0.0659852275597187
-18222,KAT6B,-0.1402907379091531
-18228,SAMD8,0.1696967055627371
-18229,VDAC2,-0.06613500432408419
-18230,COMTD1,-0.07887003722546913
-18233,ZNF503,0.15912095052903186
-18238,LRMDA,0.24756598184928824
-18240,AL589863.1,0.1716688654415001
-18253,POLR3A,0.0049740271901556445
-18254,RPS24,-0.2562759188284217
-18265,ZMIZ1,0.2061938426977334
-18266,PPIF,0.19296829972437027
-18267,ZCCHC24,0.1426819682177154
-18275,NUTM2B-AS1,-0.004651975151571467
-18278,AL135925.1,0.0028756392649428955
-18284,TMEM254,0.009926164249396434
-18286,ANXA11,-0.02448634994028409
-18292,PRXL2A,-0.027055126377883373
-18293,TSPAN14,0.05808320032218789
-18306,GHITM,0.09116246213889521
-18314,CCSER2,-0.2094546912232221
-18325,WAPL,0.10612758095605485
-18338,GLUD1,0.1662978768343788
-18339,SHLD2,-0.03325879804377034
-18341,NUTM2A-AS1,-0.011641294748843227
-18342,AL157893.2,0.11659195010674439
-18344,LINC00863,0.030917944320759966
-18348,MINPP1,0.04445052606803459
-18352,PAPSS2,0.11004643316010364
-18353,ATAD1,-0.08682693218373426
-18355,PTEN,0.2697662439531439
-18363,ANKRD22,0.30479748489905
-18364,STAMBPL1,-0.08240049640482404
-18366,ACTA2,0.06767875175812892
-18368,FAS,0.13906932603210517
-18371,LIPA,0.29691104898738013
-18372,IFIT2,0.2919537609063751
-18374,IFIT3,0.3368219300391523
-18376,IFIT1,0.26865301371694067
-18377,IFIT5,0.033647146012278126
-18385,KIF20B,-0.03582644232697067
-18395,RPP30,-0.10014737261804635
-18401,PCGF5,0.03710997179949014
-18405,TNKS2,0.1479696123481216
-18408,BTAF1,0.12826969313520192
-18409,CPEB3,0.14836925573096993
-18410,MARCH5,-0.005222874392288071
-18412,IDE,0.10866643489075455
-18414,HHEX,0.3933737977071099
-18415,EXOC6,0.11206357244373022
-18420,MYOF,0.42173157307111436
-18425,FRA10AC1,0.041879978285644846
-18429,SLC35G1,-0.06243599776660673
-18430,PLCE1,-0.06372395662075625
-18436,NOC3L,-0.06145354712384491
-18437,TBC1D12,0.07002199513391665
-18438,HELLS,-0.17523761929286938
-18447,PDLIM1,0.0391109698003592
-18449,ALDH18A1,-0.1049371210655948
-18450,TCTN3,-0.018768784601144214
-18451,ENTPD1,0.3821247260086059
-18452,ENTPD1-AS1,0.010144577004227065
-18455,CCNJ,0.11651135886569113
-18457,ZNF518A,0.11590531103327743
-18458,BLNK,0.014523555138250235
-18463,TM9SF3,0.06288146309057618
-18464,PIK3AP1,0.39854149833589153
-18465,LCOR,0.06256149846104511
-18469,ARHGAP19,0.08134437845102795
-18470,FRAT1,0.17756836280267463
-18471,FRAT2,0.2801437263488721
-18473,RRP12,0.16062810430440735
-18475,PGAM1,0.1495521143937348
-18476,EXOSC1,-0.048973880817327904
-18477,ZDHHC16,-0.01247457318452201
-18478,MMS19,0.05180158598446748
-18479,UBTD1,0.10241561152457976
-18484,PI4K2A,-0.05473746871587504
-18486,MARVELD1,0.17809934127478427
-18487,ZFYVE27,-0.032763535965375366
-18492,R3HCC1L,-0.0225935085793385
-18495,PYROXD2,-0.047288852666927186
-18496,HPS1,0.05118768731755814
-18500,GOT1,0.029720015236656225
-18505,SLC25A28,0.11180755878991425
-18508,ENTPD7,0.056168060072201334
-18509,CUTC,-0.07484618769993981
-18510,COX15,0.13576907226936005
-18512,DNMBP,0.09998911385157541
-18515,ERLIN1,0.08959052930845339
-18516,CHUK,0.12791085837826555
-18519,CWF19L1,0.05159289405345375
-18520,BLOC1S2,0.08521437436681563
-18526,SEC31B,0.030368952620376883
-18527,NDUFB8,-0.21896537944299
-18528,HIF1AN,0.02925709848873631
-18531,SLF2,-0.01999555176985012
-18533,MRPL43,-0.1315458937737174
-18536,TWNK,-0.05998350500014628
-18537,LZTS2,-0.011386423348763407
-18539,SFXN3,0.00328346874239551
-18550,BTRC,-0.048038441149106464
-18551,DPCD,0.03592210302130425
-18552,POLL,0.044546463654392826
-18553,FBXW4,-0.0303407091070271
-18557,NPM3,-0.09004024055674364
-18558,OGA,0.05103362747227972
-18561,ARMH3,0.07445840979239822
-18562,HPS6,-0.017840814765435323
-18563,LDB1,-0.03312536893604388
-18564,PPRC1,-0.02758925737285019
-18565,NOLC1,-0.05374197297280441
-18568,GBF1,0.033795015321117304
-18569,NFKB2,0.11336617291060999
-18571,FBXL15,-0.10411917492782194
-18572,CUEDC2,-0.05923252533784757
-18573,RPARP-AS1,-0.15085139156287927
-18574,C10orf95,-0.08421420289948686
-18575,MFSD13A,-0.012616112260765791
-18576,ACTR1A,0.10976948405778202
-18578,SUFU,0.04493502590286473
-18579,TRIM8,0.2406700493146933
-18580,AL391121.1,0.061920738492469866
-18581,ARL3,0.023101306578890715
-18583,WBP1L,0.054807601608595415
-18586,BORCS7,-0.013794449930898374
-18590,CNNM2,-0.052832835769748926
-18591,NT5C2,0.19608679436379
-18594,PCGF6,-0.040754314396366086
-18595,TAF5,-0.015740105067766606
-18596,ATP5MD,-0.11544713741697457
-18597,PDCD11,-0.00429674076859938
-18598,CALHM2,0.10458863921405098
-18604,NEURL1,0.06238065198607893
-18611,STN1,-0.07804299833925973
-18612,SLK,0.065031968185481
-18614,SFR1,0.029628834607309737
-18616,GSTO1,0.1949013065182691
-18618,ITPRIP,0.14049058857834978
-18636,XPNPEP1,0.07832587549001162
-18638,ADD3,-0.17136240224415403
-18639,MXI1,0.08885493142239931
-18641,SMNDC1,0.03496795630099661
-18643,DUSP5,0.10230429602555073
-18644,SMC3,-0.12581988155751808
-18647,PDCD4-AS1,-0.021510429981656153
-18648,PDCD4,-0.19824250129641693
-18649,BBIP1,-0.05358411405131711
-18652,SHOC2,0.0721898043036948
-18656,GPAM,-0.0033548153397215657
-18658,ACSL5,-0.007644543744852196
-18660,ZDHHC6,0.06533376519572479
-18661,VTI1A,0.09601841331162286
-18666,TCF7L2,0.3235471074008205
-18671,CASP7,0.10274593768038327
-18675,NHLRC2,0.03238817871230418
-18679,CCDC186,-0.08711494061770478
-18684,ABLIM1,-0.17926499164443746
-18686,FAM160B1,0.06555002854678038
-18687,TRUB1,-0.017228027197545237
-18701,SHTN1,0.34809318578413684
-18709,PDZD8,0.020681959734492397
-18717,RAB11FIP2,0.0051000276582719144
-18721,FAM204A,-0.017757874315977955
-18724,CACUL1,0.1666667552617453
-18728,EIF3A,0.24581817113503024
-18729,DENND10,0.2743278790552892
-18730,SFXN4,0.0009052181967413782
-18731,PRDX3,0.20820475653808468
-18732,GRK5,0.029349835947104008
-18735,RGS10,0.0515992803555402
-18736,TIAL1,-0.036101395372614135
-18737,BAG3,-0.029366304211636562
-18738,INPP5F,0.06324097727231752
-18739,MCMBP,0.2708572906656682
-18740,SEC23IP,0.11218150328349473
-18746,WDR11,0.1937901443051379
-18754,ATE1,0.03350315678352363
-18756,NSMCE4A,-0.047298302331269984
-18762,PLEKHA1,-0.2413358207658806
-18770,FAM24B,-0.10277829675290938
-18772,C10orf88,0.01671071278501519
-18773,PSTK,0.04150622148372544
-18774,IKZF5,0.018102467992657842
-18775,ACADSB,-0.013143594335543301
-18778,BUB3,-0.15089677611570196
-18788,CHST15,0.3908173121758276
-18789,OAT,0.10005191094814314
-18792,LHPP,0.03678945759717482
-18793,FAM53B,-0.06678803055963245
-18796,EEF1AKMT2,-0.048449876549804274
-18797,ABRAXAS2,0.039207337124889684
-18799,ZRANB1,0.06060632905340739
-18801,CTBP2,0.1809529404255962
-18809,EDRF1,-0.014113877017413437
-18812,UROS,0.013210038645847483
-18813,BCCIP,0.06123063628006903
-18814,DHX32,0.10762432071862264
-18829,PTPRE,0.41108564982292223
-18841,MGMT,-0.1907235525178869
-18852,GLRX3,-0.023711275385969142
-18860,PPP2R2D,0.008166542334603994
-18861,BNIP3,-0.15587741470456573
-18867,STK32C,0.12391488874310408
-18868,LRRC27,0.139017419132747
-18869,PWWP2B,0.09290135103352538
-18875,INPP5A,0.06177765131991048
-18884,UTF1,0.1386711370106563
-18885,VENTX,0.19311241999161233
-18888,ADAM8,-0.01456399859359469
-18889,TUBGCP2,0.14553834175363123
-18891,ZNF511,-0.0012008746164594325
-18895,FUOM,0.211196817822023
-18896,ECHS1,0.025830909484593854
-18898,PAOX,0.03973611195289591
-18899,MTG1,-0.07899269120264053
-18910,LINC01001,0.14804041907088428
-18912,BET1L,-0.07586722289663174
-18916,RIC8A,0.005429934796453689
-18917,SIRT3,-0.017654487905257515
-18918,PSMD13,-0.024538659441892634
-18920,AC136475.3,-0.0680152608312674
-18921,PGGHG,-0.043239334469585146
-18923,IFITM2,-0.23742325784772905
-18925,IFITM1,-0.3907499165205584
-18927,IFITM3,0.36138997314940363
-18935,SIGIRR,-0.27140473191148934
-18937,PTDSS2,0.028510575488980887
-18939,RNH1,0.1996769740467373
-18942,HRAS,-0.09793911767674232
-18946,RASSF7,-0.12361981403756715
-18948,PHRF1,-0.038506778214087906
-18949,IRF7,0.18740147189889123
-18953,DEAF1,0.022625131110136427
-18956,TMEM80,-0.07092494440437083
-18958,TALDO1,0.5038668106294175
-18959,GATD1,-0.014614700484698229
-18960,AP006621.3,-0.0022391764697869336
-18963,SLC25A22,-0.09217548476524887
-18965,PIDD1,0.00764017240883055
-18966,RPLP2,-0.3230827448971952
-18967,PNPLA2,0.12499525305968633
-18970,CD151,0.07381552344241997
-18971,POLR2L,-0.10463537529093703
-18972,TSPAN4,0.2254425203466002
-18974,CHID1,0.006907804682562459
-18975,AP2A2,-0.00746196835593728
-18983,TOLLIP,0.15401292355123042
-18989,MOB2,-0.1576646118337146
-19003,CTSD,0.39846955614576074
-19009,TNNI2,0.1727476845659177
-19010,LSP1,0.16215378331993094
-19015,MRPL23,0.07232868328942182
-19025,ASCL2,-0.11199813382328867
-19026,C11orf21,0.0689958580161797
-19027,TSPAN32,-0.09283319048225618
-19029,CD81,-0.4735847317785035
-19030,TSSC4,-0.031895002913410596
-19033,KCNQ1,0.1831059822971918
-19034,KCNQ1OT1,0.02314591463629525
-19037,CDKN1C,0.13540056413484605
-19039,SLC22A18,0.102149131687314
-19041,NAP1L4,-0.23417211222373033
-19043,CARS,0.05534658679779262
-19046,OSBPL5,-0.0108896214833014
-19053,ZNF195,-0.025198872493402438
-19061,NUP98,0.10915987489319111
-19062,PGAP2,-0.1084989858096363
-19063,RHOG,0.3191549448804016
-19065,STIM1,-0.029527430468067235
-19067,RRM1,-0.08774089660486535
-19076,TRIM21,0.10944480880164048
-19128,TRIM34,0.12306847344619529
-19129,TRIM5,0.09204491046746109
-19130,TRIM22,0.23485033379465398
-19151,FAM160A2,0.1330101107312938
-19157,SMPD1,0.06150104396777696
-19158,APBB1,-0.08620815722890156
-19161,ARFIP2,-0.09162504844200589
-19162,TIMM10B,-0.07186931589306382
-19163,DNHD1,-0.018219068212561335
-19164,RRP8,-0.030953521567409135
-19166,ILK,0.08725539496840566
-19167,TAF10,0.001253795200467858
-19169,TPP1,0.4177603649041034
-19174,MRPL17,-0.04837702259362486
-19192,PPFIBP2,0.10408381844020764
-19205,EIF3F,-0.10387027134581321
-19210,RIC3,-0.0884071433262797
-19214,TRIM66,-0.011862213565263888
-19216,RPL27A,-0.4191699965531535
-19220,AKIP1,0.015611031326309182
-19223,TMEM9B,0.04160970602266767
-19224,TMEM9B-AS1,0.006363031185635776
-19229,DENND5A,0.31629020923382134
-19231,TMEM41B,-0.0606396072446105
-19232,IPO7,0.12106573600865858
-19234,AC132192.2,0.04573410586058191
-19235,ZNF143,-0.016782573944628298
-19237,SWAP70,0.12432584344836703
-19240,SBF2,0.2563541439167
-19246,AMPD3,0.13049326806636144
-19247,MTRNR2L8,0.11453112369079015
-19248,RNF141,0.23524232254589036
-19251,MRVI1,0.14142222772599863
-19252,CTR9,0.056513837452686566
-19253,EIF4G2,0.2552654937416828
-19255,ZBED5,0.022715898011284073
-19256,ZBED5-AS1,-0.006605500662088019
-19265,USP47,0.013904753859520353
-19270,MICAL2,0.22942688041210224
-19284,ARNTL,-0.06627123863650206
-19285,BTBD10,-0.013774962666160599
-19288,FAR1,0.27443102953386866
-19296,RRAS2,-0.2219334290148006
-19297,COPB1,-0.023742975284947846
-19298,PSMA1,0.01587352892838994
-19299,PDE3B,-0.08888367926758675
-19300,CYP2R1,-0.04196535098852471
-19312,C11orf58,-0.00033711028724264455
-19314,RPS13,-0.20201370935733956
-19315,PIK3C2A,0.06165808627625503
-19316,NUCB2,-0.23327652286194214
-19328,KCNC1,-0.06767203705223826
-19329,SERGEF,-0.04943475069364794
-19331,SAAL1,0.03605587374650138
-19339,HPS5,0.10204700054460464
-19340,GTF2H1,0.08467980131675827
-19341,LDHA,-0.15371620938084282
-19345,TSG101,-0.02862630058354597
-19347,UEVLD,0.02387639670266285
-19349,SPTY2D1,-0.01357638718913906
-19358,ZDHHC13,-0.003230170837080514
-19372,HTATIP2,0.014675878724024772
-19373,PRMT3,0.04272284683024075
-19389,FANCF,-0.009820126368774659
-19391,SVIP,-0.08905958387328737
-19412,LIN7C,0.09032516884196
-19419,METTL15,-0.021451172252794574
-19433,ARL14EP,-0.14258172388529802
-19440,DNAJC24,-0.08324048785545259
-19441,IMMP1L,-0.22335542204647657
-19442,ELP4,-0.023492095431620896
-19449,RCN1,0.004250110078035782
-19454,EIF3M,0.1625123421002368
-19456,PRRG4,0.263996222403453
-19457,QSER1,0.16079090046284014
-19459,TCP11L1,0.015361824774063584
-19460,LINC00294,0.1290133445892948
-19461,CSTF3,-0.026037263617189957
-19463,HIPK3,0.30316908102883544
-19467,CD59,-0.05425924248583709
-19468,FBXO3,-0.05295970525500742
-19474,LMO2,0.4989865891914998
-19477,CAPRIN1,0.135860662546562
-19478,NAT10,0.021558955086264157
-19480,CAT,0.3217553402875283
-19485,APIP,0.03476228049049834
-19486,PDHX,-0.0006864488525333186
-19488,CD44,0.4441434647336419
-19490,AL133330.1,0.09610680754139461
-19498,TRIM44,0.1353259737187674
-19502,LDLRAD3,0.10583493582058487
-19504,COMMD9,0.1452971254406108
-19505,PRR5L,-0.24198348457355867
-19509,TRAF6,0.06365521662633239
-19512,C11orf74,0.09357449880792061
-19526,API5,0.04793897630221417
-19528,TTC17,0.05071509260101876
-19534,HSD17B12,0.02497195408062195
-19537,ALKBH3,-0.046859373144318
-19543,ACCS,0.06808232839891874
-19545,EXT2,-0.029229823725529517
-19548,CD82,0.03495922788747541
-19553,TSPAN18,-0.02836124232456483
-19554,TP53I11,0.023964803905711686
-19574,SLC35C1,0.02026635821401355
-19576,CRY2,0.028100639474035273
-19580,PEX16,-0.038610404763586216
-19582,PHF21A,0.19373290723857386
-19589,DGKZ,-0.06086487805595503
-19592,AMBRA1,-0.01314457630096873
-19595,ATG13,0.05552308004638984
-19596,ARHGAP1,0.01305768665450671
-19597,ZNF408,0.02732772440029107
-19599,CKAP5,-0.09477054109104215
-19603,C11orf49,-0.060046737681408266
-19605,ARFGAP2,0.005454908631164057
-19608,DDB2,-0.004946507972569303
-19610,ACP2,0.1391369700249091
-19611,NR1H3,0.06324175749443887
-19612,MADD,0.006929653297533514
-19615,SPI1,0.5540274766715332
-19616,AC090559.1,0.4678953088672918
-19617,SLC39A13,-0.10193922289752724
-19618,PSMC3,-0.01893037705873554
-19620,CELF1,0.2087740943949832
-19621,AC090559.2,0.061663631367116604
-19622,NDUFS3,-0.09750122568994272
-19623,PTPMT1,-0.035579123270754735
-19624,KBTBD4,-0.0703336040611831
-19628,MTCH2,0.06365779551122361
-19630,FNBP4,-0.02191415626318299
-19631,NUP160,0.005825070789321831
-19632,PTPRJ,0.1795390670223972
-19706,SSRP1,-0.014298930028497513
-19710,SLC43A3,0.25446117851181244
-19714,TIMM10,-0.0876211822710909
-19716,UBE2L6,0.3316697002476264
-19717,SERPING1,0.28952596028745303
-19720,CLP1,0.053659106022051865
-19721,ZDHHC5,0.05069167780036573
-19722,MED19,0.008799487135132948
-19723,TMX2,-0.006734204340582269
-19724,SELENOH,-0.024343687065284017
-19726,CTNND1,0.22618398665673128
-19743,LPXN,-0.04357044872395328
-19744,ZFP91,0.08518424842870324
-19754,FAM111A-DT,-0.044393830246932095
-19755,FAM111A,0.1135596137362594
-19757,MPEG1,0.6144417103619213
-19769,OSBP,0.024064112267347438
-19770,PATL1,0.15932932256544002
-19773,STX3,0.17673276212441683
-19775,MRPL16,-0.003977268446022285
-19786,MS4A6A,0.5789648145817967
-19787,MS4A4A,0.14905448661335477
-19788,MS4A4E,0.11248004914954275
-19790,MS4A7,0.33702590596470544
-19791,MS4A14,0.18879159898132908
-19793,MS4A1,0.00294563934022396
-19803,CCDC86,0.01652995411840721
-19806,PRPF19,0.10075942763127214
-19809,TMEM109,-0.1674553880764869
-19812,SLC15A3,0.3866673344228871
-19813,CD6,-0.23046737502584166
-19815,CD5,-0.15787798222306196
-19816,VPS37C,0.1548025542233081
-19821,DDB1,0.07549183888366592
-19822,TKFC,0.1135190169409826
-19823,CYB561A3,0.0856050009384049
-19824,TMEM138,-0.0038949633667783416
-19825,TMEM216,-0.04625714836535943
-19826,CPSF7,0.055424542624655834
-19827,SDHAF2,-0.15063056908840883
-19838,TMEM258,-0.05702922412317555
-19839,FEN1,-0.09746159677212642
-19841,FADS1,0.09549866506760614
-19844,BEST1,0.19392116138817195
-19845,FTH1,0.5444989229675227
-19848,INCENP,-0.02963901643785724
-19856,ASRGL1,0.19786390265719583
-19860,AHNAK,0.4131594106500716
-19863,EEF1G,-0.23753576848956984
-19864,TUT1,0.032183224473402425
-19865,MTA2,-0.05947397852686207
-19866,EML3,0.05772437062322118
-19869,B3GAT3,-0.1375436941596079
-19870,GANAB,-0.007756182433194822
-19871,INTS5,0.07037107512606458
-19874,CSKMT,-0.07867518924375701
-19875,UQCC3,0.03640047411016783
-19876,UBXN1,-0.16535694479280136
-19878,BSCL2,0.08705722190409504
-19880,HNRNPUL2,0.05286897836735127
-19881,TTC9C,-0.0581617422074711
-19882,ZBTB3,-0.04256451448010103
-19883,POLR2G,-0.22135469790825757
-19885,TAF6L,0.009856386295340806
-19886,TMEM223,-0.17883893677520915
-19888,TMEM179B,0.010543700783413709
-19889,NXF1,0.11997157564503945
-19890,STX5,0.044606188816401146
-19894,WDR74,0.14861033166273024
-19895,SNHG1,-0.049732780671883456
-19896,SLC3A2,-0.1581077400627601
-19907,LGALS12,0.09560934021901282
-19908,PLAAT4,-0.5115175979745045
-19910,PLAAT3,-0.3975792569086689
-19912,ATL3,-0.005659262783206555
-19914,RTN3,0.3542886878232023
-19917,MARK2,0.05693101057577632
-19919,NAA40,-0.1101457570440746
-19920,COX8A,0.021576902869347492
-19921,OTUB1,-0.033855456680874065
-19927,STIP1,0.0195288731639285
-19928,FERMT3,0.14261018287040295
-19929,TRPT1,0.021004756729487244
-19930,NUDT22,-0.008372503570229508
-19932,DNAJC4,0.09014453811757539
-19933,VEGFB,-0.15204151933010845
-19934,FKBP2,-0.0922395714211586
-19936,PPP1R14B,0.006578450467125189
-19939,PLCB3,0.1354732287968626
-19940,BAD,-0.08441309375693941
-19941,GPR137,0.03580261446708925
-19944,ESRRA,0.1831158060834596
-19945,TRMT112,-0.15232646081772278
-19946,PRDX5,0.15738556778518645
-19947,AP003774.2,-0.10581033507807502
-19948,CCDC88B,0.2295289941273771
-19949,RPS6KA4,0.2807656849121126
-19950,LINC02723,0.2056480907908956
-19959,RASGRP2,-0.06504568932569062
-19961,SF1,-0.0448705212315294
-19962,AP001462.1,-0.09589253809480854
-19963,MAP4K2,-0.09166736957925249
-19964,MEN1,-0.019592647465563447
-19966,EHD1,-0.061665298872491635
-19968,ATG2A,0.033184915598578724
-19969,PPP2R5B,0.06662640392559996
-19972,BATF2,0.15478366192444584
-19973,ARL2,-0.09869496740504831
-19974,SNX15,0.03725952051776163
-19975,SAC3D1,0.10242313346448026
-19976,NAALADL1,-0.07984592603446104
-19978,ZFPL1,-0.015227235881554866
-19980,VPS51,-0.11270361678688869
-19982,TM7SF2,-0.15388609135088213
-19983,ZNHIT2,-0.06681438056877803
-19985,FAU,-0.3234247014491553
-19986,SYVN1,0.026085461067443992
-19987,MRPL49,-0.038167368895736985
-19989,AP003068.2,0.04447360884758029
-19990,CAPN1,0.06705267627623027
-19992,POLA2,0.0230598372444871
-19994,CDC42EP2,0.23144502131796402
-19995,DPF2,-0.041371530288142305
-19998,SLC25A45,-0.14613925313490192
-19999,FRMD8,0.07942969554240047
-20000,NEAT1,0.5571314586244224
-20003,MALAT1,-0.5055356425709229
-20005,SCYL1,0.08766208063891465
-20006,LTBP3,-0.20066259664821354
-20008,ZNRD2,-0.05484660174285046
-20009,FAM89B,-0.06515793560976336
-20010,EHBP1L1,0.3503181785885314
-20012,MAP3K11,0.30291990908375765
-20013,PCNX3,-0.028276416015635537
-20014,SIPA1,0.0061828279807890595
-20015,RELA,0.05082182006591791
-20016,RELA-DT,-0.007030477935012477
-20017,KAT5,0.06896643686842208
-20018,RNASEH2C,-0.044902592050751304
-20020,AP5B1,0.3054173893574381
-20024,CFL1,-0.1040882681024588
-20026,MUS81,-0.04742392520615546
-20028,CTSW,-0.6916774773740407
-20029,FIBP,-0.02527153502933001
-20030,CCDC85B,-0.333943140229961
-20032,C11orf68,-0.0027258126365736417
-20033,DRAP1,-0.06997193334041597
-20035,SART1,0.046834600120971855
-20036,EIF1AD,0.031000201104731576
-20037,BANF1,-0.12565129914890916
-20039,CATSPER1,0.11289109347637902
-20042,SF3B2,0.0027957092853720966
-20044,PACS1,-0.10955806930718161
-20050,RAB1B,0.07461445328947859
-20054,YIF1A,-0.01412972541676579
-20058,RIN1,0.19053252109894434
-20060,BRMS1,0.043628878311067334
-20066,MRPL11,-0.14254576498838106
-20068,AP002748.4,-0.021039561300432955
-20069,DPP3,0.09916437019434562
-20072,ZDHHC24,-0.08762693745131411
-20077,CCS,0.009453939217286864
-20078,RBM14,0.031870048210695695
-20079,RBM4,-0.05610352556075242
-20080,RBM4B,-0.043329890787875004
-20083,RCE1,0.032548919588494186
-20089,KDM2A,0.11661937604147041
-20091,GRK2,0.28878565217242874
-20092,ANKRD13D,0.16556698564993305
-20093,SSH3,0.10046780043140055
-20095,RAD9A,-0.08335242177462203
-20096,POLD4,0.07237772244574371
-20100,PPP1CA,-0.0854893070653526
-20101,TBC1D10C,-0.3868673195089521
-20103,RPS6KB2,0.062267209870485375
-20106,CORO1B,-0.04448552108640447
-20109,TMEM134,-0.13596762606278406
-20110,AIP,-0.1687516958023577
-20111,PITPNM1,0.028837514714785448
-20112,CDK2AP2,-0.14920121081457502
-20114,GSTP1,0.27121515560227627
-20117,NDUFV1,0.04171155816857096
-20118,NUDT8,-0.11047772154772323
-20123,UNC93B1,0.38634262046797635
-20124,ALDH3B1,0.2973779529364544
-20126,NDUFS8,-0.036032761205392325
-20127,TCIRG1,0.3708308278649609
-20128,AP002807.1,0.10293890097189178
-20129,CHKA,0.1236834253458203
-20131,KMT5B,0.056687559400938524
-20132,C11orf24,0.04490551813456378
-20135,PPP6R3,0.029147691419350984
-20139,CPT1A,0.10842486292008367
-20141,MRPL21,-0.044830274402101036
-20142,IGHMBP2,0.04590139260972633
-20149,TPCN2,0.12495626964015524
-20162,LTO1,0.03707812867713271
-20175,FADD,-0.1483655346326286
-20177,PPFIA1,0.2587821086419941
-20191,AP002387.2,-0.19308681387388457
-20192,NADSYN1,0.006095754988053231
-20205,AP002495.1,-0.035240113877617656
-20207,RNF121,-0.0762808357609514
-20208,IL18BP,0.0012863370641786494
-20209,NUMA1,-0.03361368563705233
-20212,LAMTOR1,0.24057465708659773
-20214,ANAPC15,0.06459404143450564
-20215,FOLR3,0.19119229021856313
-20218,FOLR2,0.1931975896756216
-20219,INPPL1,0.3429744192187583
-20222,CLPB,0.18962970864989012
-20231,ARAP1,0.36706164877906855
-20235,STARD10,0.06878033803397725
-20236,AP002381.2,0.020903252530860773
-20237,ATG16L2,0.25111888408570426
-20238,FCHSD2,0.07600285519450177
-20241,P2RY2,0.13411977824818488
-20246,RELT,0.28553249459507196
-20248,FAM168A,0.0639541207645654
-20249,AP000763.3,0.07278408463698317
-20251,RAB6A,0.059980987554998935
-20254,MRPL48,-0.11304920822095064
-20255,COA4,-0.03148626696981227
-20256,PAAF1,-0.0583699692080336
-20259,UCP2,-0.06909169982930884
-20260,AP003717.4,0.04733886682427651
-20262,C2CD3,-0.04452123691640412
-20263,PPME1,-0.05659971366006085
-20266,PGM2L1,-0.04801722776540931
-20269,KCNE3,0.42337969727804614
-20273,POLD3,0.05944717332645988
-20275,RNF169,0.08143778227445574
-20276,XRRA1,-0.1644667929436452
-20278,SPCS2,-0.29934843599336114
-20279,NEU3,0.0591054424430676
-20285,ARRB1,0.18008187195323805
-20288,RPS3,-0.523421830545113
-20290,GDPD5,0.06363454884830647
-20299,DGAT2,0.11289859234647767
-20303,UVRAG,0.14239062621462995
-20309,THAP12,-0.0629192452858953
-20311,AP002360.1,-0.03659938444335159
-20312,EMSY,-0.020550302569906746
-20323,ACER3,0.30953382255382056
-20330,PAK1,0.4260727986607704
-20332,CLNS1A,-0.009124343727343468
-20334,RSF1,-0.008983323916190796
-20337,AAMDC,-0.025151045093802376
-20341,INTS4,0.07677963640433204
-20346,NDUFC2,-0.09317652424065098
-20347,ALG8,-0.008413677483455133
-20349,KCTD21,-0.00925988869944478
-20351,GAB2,0.22111446991393463
-20355,NARS2,-0.009808515068517633
-20371,PRCP,0.16134500963641488
-20372,DDIAS,0.08902305793871719
-20374,RAB30,-0.020203221253536283
-20375,RAB30-DT,-0.04729784803994817
-20377,PCF11,0.03938866239924769
-20379,AP000873.2,-0.040901453141206384
-20381,ANKRD42,0.09917745943859464
-20383,CCDC90B,-0.08675945409279451
-20389,TMEM126B,-0.10057875615519514
-20390,TMEM126A,-0.11876048626658083
-20391,CREBZF,-0.07676690482477437
-20393,SYTL2,-0.2649636879343972
-20396,PICALM,0.36535944097807993
-20398,EED,-0.015087757672847426
-20399,HIKESHI,-0.049544391971277746
-20405,PRSS23,-0.4629806773808759
-20410,TMEM135,-0.04735513681570954
-20417,CTSC,-0.19496866283728823
-20435,CHORDC1,-0.057229629269066315
-20445,SLC36A4,0.15923322522693864
-20448,SMCO4,0.39893011495129477
-20449,CEP295,0.027051499204989688
-20451,TAF1D,-0.08363736965960343
-20452,C11orf54,0.10093498409195552
-20453,MED17,-0.03968497448627527
-20456,PANX1,-0.016946063196655967
-20460,MRE11,0.03310084630401161
-20462,ANKRD49,-0.1437782401798437
-20464,FUT4,0.1333455484981591
-20465,PIWIL4,0.15217604645600383
-20471,CWC15,-0.10228906083865485
-20475,SRSF8,-0.0936739493208235
-20476,ENDOD1,0.02876905770216907
-20477,AP000787.1,-0.06184447735162529
-20478,SESN3,-0.016409855896251414
-20482,FAM76B,0.0017179847336905474
-20483,CEP57,-0.10686245854841372
-20484,MTMR2,-0.01172780968194301
-20485,MAML2,-0.08509172228824155
-20487,CCDC82,-0.17620280772026184
-20488,JRKL,-0.0005907785668878563
-20509,BIRC3,0.011647903935635018
-20510,BIRC2,0.039164320870201476
-20511,TMEM123,0.2814957959806534
-20525,DCUN1D5,-0.07505569367724609
-20530,PDGFD,-0.34669340029088586
-20534,CASP4,0.1002707815599281
-20535,CASP5,0.14592508571538426
-20536,CASP1,0.4860737521221709
-20537,CARD16,0.29001557262053396
-20542,MSANTD4,0.0456306905199446
-20543,KBTBD3,-0.08793854448350484
-20544,AASDHPPT,-0.09812119751986541
-20550,AP000766.1,0.06657652310204538
-20551,CWF19L2,-0.1456180562798286
-20552,ALKBH8,0.06392093525023765
-20556,RAB39A,0.16154589877122155
-20557,AP003307.1,0.015671781344196698
-20558,CUL5,0.05801884008904992
-20560,ACAT1,-0.030235112657573348
-20562,NPAT,-0.06606514721675859
-20563,ATM,-0.27644529393878675
-20565,POGLUT3,-0.06939891989582889
-20567,DDX10,-0.007574454952692451
-20573,RDX,0.11794929154843552
-20577,FDX1,-0.04298932860524714
-20587,POU2AF1,-0.03265327483975255
-20594,SIK2,0.026663377977295215
-20595,PPP2R1B,0.06222781933120025
-20597,ALG9,-0.033415517168244306
-20599,FDXACB1,-0.08247232352329142
-20600,C11orf1,-0.10468482806956358
-20606,DLAT,0.07336801909002322
-20608,NKAPD1,-0.016454721505367406
-20609,TIMM8B,0.03839623731696977
-20610,SDHD,-0.010145357149858621
-20611,IL18,0.18974168315140544
-20615,PTS,-0.02054152278836124
-20618,LINC02762,-0.0464913702152086
-20635,ZW10,0.022631132188686936
-20638,USP28,-0.1598984676624617
-20642,ZBTB16,-0.03803520133366927
-20647,C11orf71,-0.03729299356292925
-20648,RBM7,-0.009302535490006475
-20649,REXO2,-0.22733511957816857
-20668,BUD13,0.0011590580872502011
-20670,ZPR1,0.02703812316145751
-20678,SIK3,0.19562626729111704
-20680,PAFAH1B2,0.1935961242547043
-20681,SIDT2,0.24422202128400025
-20682,TAGLN,0.08833118754414952
-20683,PCSK7,-0.056831832902951934
-20684,RNF214,-0.0333102495031367
-20685,BACE1,0.1330011720768745
-20688,CEP164,0.0025378489839460055
-20695,FXYD6,0.13863312120106994
-20697,IL10RA,0.1374789563808918
-20702,JAML,0.523090009768654
-20703,MPZL3,0.03582522844673006
-20704,MPZL2,0.17948594573656587
-20705,CD3E,-0.5179951831000079
-20706,CD3D,-0.5499808404341607
-20707,CD3G,-0.5094756891565168
-20708,UBE4A,0.15824328035473922
-20710,ATP5MG,-0.03282093658565245
-20713,KMT2A,-0.1147201077979173
-20716,TMEM25,-0.11136172179094236
-20718,ARCN1,0.018048308638113946
-20724,DDX6,-0.1883428701088667
-20725,AP004609.3,0.06798721317099041
-20726,CXCR5,-0.03674818639563134
-20728,BCL9L,-0.18039329041388913
-20732,CCDC84,0.022159946413940906
-20734,RPS25,-0.40461934370482894
-20735,TRAPPC4,-0.1535264766992612
-20738,HYOU1,-0.013986684438139425
-20741,VPS11,-0.006163546835705636
-20742,HMBS,0.04004067889615311
-20743,H2AFX,0.0753092237395292
-20744,DPAGT1,0.057843208431215404
-20745,C2CD2L,0.19385629062006085
-20746,HINFP,-0.008051977765786618
-20748,NLRX1,0.13649046359007036
-20751,CBL,0.17184499709925383
-20753,RNF26,-0.010144229449672052
-20774,OAF,0.2778100347191818
-20778,ARHGEF12,-0.07493743763280578
-20781,TBCEL,0.042545505146076884
-20783,SC5D,-0.04963324479494893
-20785,SORL1,0.029836867205009734
-20793,UBASH3B,0.07707213070981836
-20796,CRTAM,-0.08884781440469197
-20799,HSPA8,-0.09221003989110811
-20804,GRAMD1B,0.1766114302182796
-20820,VWA5A,0.04744208057399015
-20834,TBRG1,0.08622707017875555
-20835,SIAE,0.06469103616574429
-20836,SPA17,0.017705693353504397
-20837,NRGN,0.0886177352943339
-20842,MSANTD2,0.007630122864252142
-20852,CCDC15,-0.03365646619143951
-20853,SLC37A2,0.18864908169132094
-20854,TMEM218,-0.08213961973945169
-20862,EI24,-0.0011898009126339259
-20863,STT3A,0.04400899417887879
-20871,HYLS1,-0.06831044589054541
-20872,PUS3,-0.00014854049636098827
-20881,RPUSD4,0.0021319042920243344
-20883,FAM118B,0.12392377076044077
-20884,SRPRA,-0.16136104639244972
-20885,FOXRED1,0.024390089489178337
-20886,TIRAP,0.011331974213742917
-20889,DCPS,-0.08801026531045306
-20891,ST3GAL4,-0.11100684382431902
-20904,ETS1,-0.4255789239920279
-20907,FLI1,0.2005768430794699
-20908,SENCR,0.07838092761514205
-20913,ARHGAP32,0.047839881743787345
-20919,NFRKB,-0.03406873347204011
-20920,PRDM10,0.05610810345510277
-20922,APLP2,0.5568059081804788
-20923,ST14,0.11421931469796098
-20924,ZBTB44,-0.03734445029191079
-20932,SNX19,0.13218983495439615
-20955,NCAPD3,-0.06631110280443879
-20957,VPS26B,0.002775067140356687
-20958,THYN1,-0.05787925237563759
-20959,ACAD8,-0.14377948067815582
-20976,AC026369.3,0.15497679633388703
-20980,SLC6A12,0.13563120241751855
-20985,AC007406.5,0.004780548977436207
-20986,KDM5A,0.039966546060556635
-20987,CCDC77,0.00047344439351108396
-20989,NINJ2,0.13029048817767344
-20992,AC021054.1,0.09570220856723811
-20994,WNK1,0.10630825642596287
-20998,ERC1,0.12339389646242917
-21004,ADIPOR2,0.08011372573125808
-21006,CACNA2D4,0.15327470237367538
-21011,DCP1B,-0.004633188409142895
-21025,FKBP4,-0.02076444538768315
-21028,ITFG2,-0.0822445860068775
-21032,RHNO1,-0.0039027817049014334
-21046,CRACR2A,-0.11260738698924062
-21048,PARP11,-0.030040813879280072
-21049,AC005842.1,-0.07612939668657463
-21052,CCND2,-0.36403308405191054
-21053,TIGAR,0.022440467183111257
-21056,C12orf4,0.0832250136627174
-21058,DYRK4,-0.0765500121713936
-21061,NDUFA9,0.06302774509795779
-21083,CD9,0.06439634033856508
-21085,TNFRSF1A,0.25541912508893094
-21087,LTBR,0.3536699661379874
-21089,CD27-AS1,0.05777121969062087
-21090,CD27,-0.23746478494746837
-21091,TAPBPL,0.010047637910648643
-21092,VAMP1,-0.06007271827829546
-21094,MRPL51,-0.10372042513508938
-21095,NCAPD2,-0.03830229312271937
-21098,GAPDH,0.43329671072455406
-21100,IFFO1,0.10497979134061865
-21102,NOP2,-0.010085765089664253
-21103,CHD4,0.1065831284936
-21105,LPAR5,-0.1252842650758512
-21106,ACRBP,0.06210974366378561
-21107,ING4,-0.0655999163625656
-21109,ZNF384,0.07111983932276403
-21112,COPS7A,0.17095150909692253
-21114,MLF2,0.0550729888502223
-21115,PTMS,-0.20352710543941835
-21116,LAG3,-0.21891129880200003
-21117,CD4,0.3894900315687765
-21118,GPR162,0.12497889610522868
-21122,USP5,0.01815515945790449
-21123,TPI1,0.25721941711617824
-21124,SPSB2,-0.10788965589649328
-21125,LRRC23,0.02621785797123883
-21128,ATN1,0.002599687020264149
-21129,C12orf57,-0.36325821168864936
-21131,PTPN6,0.2961990731962203
-21134,PHB2,-0.09096590205621256
-21135,EMG1,-0.07277006649937882
-21136,LPCAT3,0.20745998540731558
-21139,C1RL,0.052560600991145505
-21140,C1RL-AS1,0.0007225019022496364
-21146,PEX5,-0.008902622144263423
-21149,CD163,0.35833514799241606
-21158,SLC2A3,0.27671693954235227
-21161,FOXJ2,0.11033424101991832
-21162,C3AR1,0.29732733133828715
-21163,NECAP1,0.024863230056420382
-21164,CLEC4A,0.39492430480455143
-21173,LINC00937,0.17635492526835603
-21176,CLEC4D,0.22846419042587007
-21177,CLEC4E,0.39004209374662885
-21182,RIMKLB,0.09859969057762534
-21188,PHC1,-0.04236356303442087
-21189,M6PR,0.15018181764738828
-21190,KLRG1,-0.512217696971922
-21193,A2M-AS1,-0.2164540448372424
-21201,AC092821.3,-0.23566601500911094
-21202,KLRB1,-0.48295031064683497
-21204,AC010186.3,-0.03568806494170384
-21205,CLEC2D,-0.2870304135743258
-21207,CLECL1,0.04769083668941615
-21208,CD69,-0.2070519145491415
-21209,KLRF1,-0.5480128293580124
-21210,CLEC2B,-0.20965576358339227
-21216,CLEC12A,0.463070641732107
-21218,CLEC12B,0.132473723271978
-21222,CLEC7A,0.5266630193089675
-21225,GABARAPL1,0.030986557690675244
-21227,KLRD1,-0.663250730132468
-21231,KLRK1,-0.5421517990233045
-21232,KLRC4,-0.3353950048455591
-21237,LINC02446,-0.16538319196113793
-21238,MAGOHB,-0.10085356107644863
-21240,YBX3,0.3967968108548073
-21245,PRH1,-0.19282397255224962
-21250,TAS2R14,0.020166534024819764
-21259,SMIM10L1,-0.11343269804010145
-21269,ETV6,0.28666737998339037
-21274,BORCS5,-0.07304322186596207
-21275,DUSP16,-0.09904384567743277
-21277,CREBL2,0.1367773650220527
-21281,CDKN1B,-0.16226020512723052
-21282,AC008115.3,-0.04389354517046442
-21284,APOLD1,-0.07875370244522564
-21285,DDX47,-0.008081112561730995
-21290,HEBP1,0.13711478869905097
-21302,ATF7IP,-0.032531901218656116
-21303,PLBD1,0.43485634852715027
-21305,PLBD1-AS1,0.13961412932761366
-21308,AC010168.2,-0.03937096902242478
-21309,HIST4H4,0.02054524876902399
-21310,H2AFJ,0.11389885796292515
-21311,WBP11,-0.004331064046617098
-21318,ERP27,-0.03463980659226871
-21319,ARHGDIB,-0.05581388705916833
-21325,PTPRO,0.13521935133486962
-21326,EPS8,0.13300860392653763
-21328,STRAP,-0.10447178252216646
-21329,DERA,0.024522930108648797
-21331,MGST1,0.3032634459197431
-21346,PLEKHA5,-0.1257062333784477
-21348,AEBP2,-0.003190862568166542
-21362,PYROXD1,-0.06712649517165514
-21363,RECQL,-0.04041117192113058
-21364,GOLT1B,-0.06315999441761296
-21367,LDHB,-0.29581063394273366
-21374,CMAS,-0.056370085787584574
-21379,C2CD5,0.15409468632621184
-21382,ETNK1,0.14251621359022631
-21392,BCAT1,0.14461359237384766
-21396,LRMP,0.042699411329066295
-21400,ETFRF1,-0.09661855763182146
-21401,KRAS,-0.050804051339566746
-21404,AC087239.1,0.0028399121811434583
-21421,ITPR2,0.18545204765652318
-21424,INTS13,0.02088449232245551
-21425,FGFR1OP2,-0.12217660169634867
-21426,TM7SF3,0.06558255765120986
-21428,MED21,-0.13931085779507016
-21429,AC092747.4,0.06189664131428119
-21432,STK38L,0.22937531080863083
-21442,MRPS35,-0.04319932877124301
-21445,KLHL42,0.006123777225452617
-21449,CCDC91,-0.11999035425024149
-21455,FAR2,0.12455167184397843
-21461,ERGIC2,-0.1814312735192353
-21464,TMTC1,0.126373275575872
-21470,IPO8,0.066864774405687
-21472,CAPRIN2,-0.09137842916395254
-21477,DDX11,0.026732627054944887
-21479,SINHCAF,-0.10701596103849763
-21486,ETFBKMT,0.036192393120956866
-21487,AMN1,-0.05371974757803318
-21488,AC023157.3,0.06549679203935113
-21490,LINC02422,-0.007467381683618546
-21491,RESF1,-0.13364328342331544
-21492,AC016957.2,-0.08258432987850442
-21494,BICD1,-0.08936525622798047
-21498,FGD4,0.37062211943167406
-21499,DNM1L,0.08454608406581038
-21501,YARS2,-0.07593818562964473
-21517,ALG10B,-0.006762367204263286
-21519,CPNE8,0.20722070721798583
-21522,KIF21A,-0.25100591133011557
-21524,ABCD2,-0.11364056580776948
-21527,SLC2A13,0.045497949035202705
-21530,AC079630.1,0.13817884473829783
-21531,LRRK2,0.47588773778263405
-21541,GXYLT1,0.0563927050092358
-21542,YAF2,-0.07716729056555983
-21543,PPHLN1,-0.1138529184005133
-21544,ZCRB1,-0.05629101539829176
-21546,PRICKLE1,0.10675546642024002
-21556,PUS7L,-0.061943071018065746
-21557,IRAK4,-0.0012978409105380818
-21558,TWF1,0.04570886220156662
-21563,NELL2,-0.1331882083932435
-21567,ANO6,0.19248217115617314
-21570,AC008124.1,-0.1525850501898857
-21571,ARID2,0.0377190267141931
-21573,SCAF11,0.20063525274404573
-21574,SLC38A1,-0.4079176563871824
-21576,SLC38A2,0.19604088819479804
-21578,AC008014.1,-0.18667324148104422
-21584,AMIGO2,-0.10451511863818272
-21585,PCED1B,-0.15587649278349808
-21586,PCED1B-AS1,-0.5116247310983569
-21592,RPAP3,0.01937312121227107
-21594,AC004241.1,0.04964707836417914
-21598,SLC48A1,0.05485317121510071
-21600,HDAC7,-0.014786316028804806
-21605,VDR,0.18574487673099307
-21608,TMEM106C,-0.030301706236229967
-21614,SENP1,0.002431590741562832
-21616,ASB8,0.0002339652354432201
-21619,AC024257.3,0.08971775542217836
-21624,ZNF641,0.09705860666982036
-21633,KANSL2,-0.03874830198923701
-21634,CCNT1,0.008078628796799955
-21640,DDX23,0.08556824905392482
-21642,CCDC65,-0.18882279365469085
-21643,FKBP11,-0.4356197225185384
-21644,ARF3,0.21986898799963467
-21650,PRKAG1,0.0831325792677331
-21651,KMT2D,0.1040628597983048
-21655,LMBR1L,-0.010787576894686006
-21656,TUBA1B,0.10687119960217736
-21657,AC011603.2,0.02570198430706221
-21658,TUBA1A,0.35187610753100684
-21659,TUBA1C,0.060556805629965064
-21667,SPATS2,-0.05107359645630419
-21670,MCRS1,0.06503733602351223
-21674,FMNL3,-0.10298376763792177
-21675,TMBIM6,0.22375808500864927
-21676,NCKAP5L,0.16390160679456608
-21678,BCDIN3D,-0.12636239815156017
-21690,SMARCD1,-0.07124335615189634
-21692,COX14,-0.011990642888803938
-21693,AC074032.1,-0.022116889294847942
-21694,CERS5,-0.10170548339335317
-21695,LIMA1,-0.18106087958375303
-21699,LARP4,0.055435396942871064
-21700,DIP2B,0.1773302518692157
-21701,ATF1,-0.027955704232594514
-21703,METTL7A,0.3899848837729131
-21707,SLC11A2,0.14101007234153387
-21708,LETMD1,0.014849862412177962
-21709,CSRNP2,-0.06397959897190883
-21710,TFCP2,0.09093828021115129
-21713,DAZAP2,0.39237357453087324
-21714,SMAGP,-0.18539744744198303
-21715,BIN2,-0.08719778665481231
-21716,CELA1,-0.038868494061932664
-21717,GALNT6,0.07419157360100684
-21718,SLC4A8,0.04542846915430432
-21729,ACVR1B,0.08539100914165987
-21730,GRASP,-0.018106013619713878
-21731,NR4A1,0.11301923819520564
-21733,ATG101,0.10625652785396539
-21774,EIF4B,0.08331320810379668
-21778,SPRYD3,-0.10690436712623297
-21781,CSAD,0.055106861024039354
-21783,ZNF740,0.047392031268654954
-21784,ITGB7,-0.2464080845461938
-21785,RARG,-0.08507147630781434
-21787,MFSD5,0.01689151102353968
-21789,PFDN5,0.09233245295420803
-21791,C12orf10,-0.045227615264727454
-21792,AAAS,0.025620019146816615
-21794,SP1,0.28256476809294606
-21796,PRR13,0.23530356202057637
-21797,PCBP2,0.2285864562770191
-21799,MAP3K12,-0.1492607313294053
-21800,AC023509.3,-0.11412026605691367
-21801,TARBP2,-0.033040345323718856
-21803,ATF7,0.05289695282281804
-21806,ATP5MC2,0.05680989045427299
-21807,CALCOCO1,0.0838128199134062
-21829,SMUG1,-0.10747205188058746
-21834,CBX5,-0.11841949091199114
-21837,HNRNPA1,-0.11997132231330936
-21838,NFE2,0.3054217978197107
-21839,COPZ1,0.010745056787968004
-21843,ZNF385A,0.4217580215314069
-21844,ITGA5,0.1599829197048369
-21847,NCKAP1L,0.2767290512574127
-21855,TESPA1,-0.17824815227064236
-21877,BLOC1S1,0.1694064189919557
-21878,RDH5,-0.0004694093347357976
-21879,CD63,-0.08371926883663108
-21880,AC009779.2,0.029200900075630844
-21881,GDF11,-0.15966487408568747
-21882,SARNP,-0.022306631441980394
-21884,ORMDL2,-0.13034291819841148
-21885,DNAJC14,-0.02730995319932911
-21887,PYM1,-0.04996833879596995
-21888,DGKA,-0.1381898448123688
-21893,RAB5B,0.043021634235029
-21894,SUOX,0.12308720530321297
-21898,RPS26,-0.4510464020959033
-21900,PA2G4,-0.102169269474222
-21902,RPL41,-0.45088131380848384
-21903,ESYT1,-0.12651582855229632
-21904,ZC3H10,0.029954876761394595
-21908,MYL6B,-0.044487807923595114
-21909,MYL6,0.041785533012089614
-21910,SMARCC2,0.039182412314743474
-21912,RNF41,0.12329368154113904
-21913,NABP2,-0.06335535043992963
-21915,ANKRD52,-0.007362521791697226
-21916,COQ10A,-0.02313949607054571
-21918,CS,0.16788788525733295
-21919,AC073896.2,-0.09366051076579371
-21921,CNPY2,-0.08543667751536517
-21922,PAN2,0.05498392702466293
-21923,IL23A,-0.048626226199400824
-21924,STAT2,0.3963967267682941
-21927,TIMELESS,0.08225657128202253
-21929,SPRYD4,-0.007432775932653606
-21931,RBMS2,0.07902186472627473
-21932,BAZ2A,0.18980599429019734
-21933,ATP5F1B,0.18117399296861803
-21934,PTGES3,0.003736538965223948
-21935,NACA,-0.19221933761251933
-21936,PRIM1,-0.09170699079926482
-21945,NEMP1,-0.07823432445669569
-21946,NAB2,0.039547269080873
-21947,STAT6,0.2547639021301489
-21948,LRP1,0.49085561867726285
-21952,SHMT2,0.008441279107623193
-21954,STAC3,0.23892999456640313
-21955,R3HDM2,-0.00011282745306473956
-21961,ARHGAP9,0.10031089097162672
-21962,MARS,0.06399088052234177
-21963,DDIT3,0.025800039364418194
-21965,MBD6,0.09180044721829356
-21966,DCTN2,0.010130223181847812
-21968,PIP4K2C,-0.0013137739414035468
-21974,OS9,0.27218945461595906
-21976,AGAP2,-0.10395594793412462
-21978,TSPAN31,-0.03377366904343264
-21979,CDK4,-0.06252782259015469
-21980,MARCH9,-0.12245815036275408
-21982,METTL1,-0.015099604190497475
-21984,TSFM,-0.057312775454137044
-21985,AVIL,0.05576755326085151
-21988,CTDSP2,0.1717905272552526
-21992,ATP23,0.004817620836468258
-21993,GIHCG,-0.09258195546939228
-22003,SLC16A7,-0.007983596638652373
-22007,TAFA2,0.0565418115935738
-22009,USP15,0.3938832213538252
-22010,MON2,0.13414957669750496
-22013,AC048341.1,0.025331087801763682
-22021,RXYLT1,-0.05215774679653021
-22029,C12orf66,0.06861025114190822
-22031,XPOT,0.016573446643822178
-22033,TBK1,0.00047704433704692903
-22034,RASSF3,0.27083475719237926
-22039,GNS,0.4516135593028999
-22047,LEMD3,-0.05244931637168186
-22060,LLPH,0.014250412924925222
-22062,TMBIM4,0.05851000276888205
-22063,IRAK3,0.4668471282627401
-22065,HELB,-0.07949036251592125
-22068,CAND1,-0.007939814568265953
-22073,DYRK2,-0.21274961416650412
-22077,IFNG-AS1,-0.07859793429566161
-22081,MDM1,-0.02998485352810523
-22085,RAP1B,-0.15427302068149373
-22087,NUP107,-0.006792060766742977
-22088,SLC35E3,0.026084017162832842
-22089,MDM2,0.09343325412340961
-22092,CPM,0.1999659222489329
-22094,CPSF6,-0.004589331003036799
-22096,LYZ,0.6568269678477596
-22097,AC020656.1,0.530462022903405
-22098,YEATS4,-0.05610333060504444
-22100,FRS2,0.06843442992928672
-22101,CCT2,-0.11972768038960087
-22105,RAB3IP,-0.032858651709614
-22107,AC025159.1,0.044569972119298376
-22112,CNOT2,0.06844015497615183
-22125,ZFC3H1,0.07892815147410649
-22126,THAP2,-0.09797454650466504
-22127,TMEM19,0.02090649692385106
-22129,RAB21,0.07140121024239493
-22131,TBC1D15,0.0025852657886687724
-22145,ATXN7L3B,-0.11235663191921587
-22154,GLIPR1,0.36774417941598475
-22156,KRR1,-0.14102319392445345
-22164,NAP1L1,0.18792990746612834
-22166,BBS10,0.05311211065850219
-22167,OSBPL8,0.21826776654601043
-22170,ZDHHC17,0.02114089572243999
-22189,PAWR,-0.019944868409438453
-22191,PPP1R12A,0.023104894325917297
-22201,LIN7A,0.1994632108997382
-22208,CCDC59,-0.07227953535119949
-22209,METTL25,0.09520665981981918
-22214,TMTC2,0.1969874310887349
-22232,C12orf29,-0.07384267668175838
-22233,CEP290,-0.06180120993999262
-22234,TMTC3,0.03597201756841866
-22239,DUSP6,0.5393563631284451
-22240,AC024909.1,0.12433724520245486
-22244,POC1B,-0.016813867556951956
-22245,GALNT4,0.044559801851576354
-22246,POC1B-AS1,0.007042715941496941
-22248,ATP2B1,0.1500796708273719
-22249,ATP2B1-AS1,0.3119648261663
-22273,BTG1,-0.22822789690035283
-22274,AC025164.1,-0.01930726889166697
-22278,LINC02397,0.002014410998348205
-22280,EEA1,0.129437737162462
-22286,NUDT4,0.02036526484950702
-22287,UBE2N,-0.06250293697785606
-22288,MRPL42,-0.1061892130952146
-22290,SOCS2,-0.10028015293543324
-22291,CRADD,-0.078763835179653
-22296,PLXNC1,0.4002705179051726
-22301,CEP83,-0.02333008532748212
-22304,TMCC3,0.0007319838365632424
-22305,NDUFA12,-0.27163713814524076
-22306,NR2C1,0.06366718084492752
-22307,FGD6,0.1798260754335749
-22308,VEZT,-0.025400388874695302
-22310,METAP2,0.04141191661545094
-22311,USP44,-0.05272180368477049
-22315,SNRPF,-0.18255294685571039
-22318,HAL,0.14439910091156505
-22320,LTA4H,0.4717068352776794
-22323,ELK3,0.13191271460438542
-22326,CDK17,-0.053882124605952306
-22329,NEDD1,0.02209719744639508
-22338,TMPO-AS1,-0.0028361782203879305
-22339,TMPO,0.08342952842789311
-22340,SLC25A3,0.02772483140544145
-22341,IKBIP,0.00016516037400131798
-22342,APAF1,0.3014723024041789
-22348,UHRF1BP1L,0.2403548814901552
-22350,ACTR6,-0.17227690161109802
-22352,SCYL2,0.09341857227039699
-22360,UTP20,-0.0339833155299539
-22361,ARL1,-0.1113904046301516
-22367,CHPT1,0.19629214224712116
-22369,GNPTAB,-0.2963656335445928
-22370,DRAM1,0.17189687623885083
-22373,WASHC3,-0.08042553552584707
-22375,NUP37,-0.06895903106700178
-22398,HSP90B1,0.030257493853070968
-22399,C12orf73,-0.0844925999398108
-22400,TDG,-0.010465129836635166
-22402,HCFC2,-0.005727726276448376
-22403,NFYB,-0.14858737626788884
-22406,TXNRD1,0.06423778945310389
-22408,EID3,-0.0784084622979844
-22409,CHST11,0.004333469601310279
-22412,C12orf45,-0.029506614304755534
-22414,WASHC4,0.288816192084943
-22415,APPL2,-0.060759585101191685
-22416,C12orf75,-0.36609596690440976
-22424,CKAP4,0.30634033284465007
-22427,TCP11L2,0.019232738759505333
-22428,POLR3B,-0.032826502013028856
-22433,RIC8B,-0.007153667884471171
-22436,TMEM263,-0.15319310275819403
-22438,CRY1,-0.05151326872932001
-22441,BTBD11,0.01717571767487454
-22443,PWP1,-0.13504483510948817
-22444,PRDM4,-0.043162631247190844
-22449,CMKLR1,0.10340007637167456
-22451,FICD,-0.03368992510223088
-22452,SART3,0.0048895809153892975
-22454,ISCU,-0.14495256727189623
-22456,SELPLG,0.09407899120385727
-22458,CORO1C,0.3357856457355609
-22459,SSH1,0.090421890863024
-22463,USP30,0.037820831049363894
-22464,USP30-AS1,-0.10247780593699769
-22465,ALKBH2,-0.02518446395801906
-22466,UNG,-0.057102022878956216
-22473,KCTD10,-0.08532390202984376
-22474,UBE3B,-0.03058944465511286
-22475,MMAB,-0.11358823213660325
-22476,MVK,-0.053346818972692556
-22480,GLTP,0.15124752527185958
-22482,TCHP,-0.041101173815256804
-22483,GIT2,0.19247188935950169
-22485,ANKRD13A,0.1949908553312892
-22486,C12orf76,-0.08214126968575844
-22489,ATP2A2,0.13017864124012835
-22490,ANAPC7,-0.027885949491582757
-22491,ARPC3,0.31738104044958587
-22492,GPN3,0.03920886566837917
-22493,FAM216A,-0.046747860815701425
-22494,VPS29,0.1492221209024229
-22496,PPTC7,0.16844534694844251
-22497,TCTN1,0.0211118114522862
-22498,HVCN1,0.23400653244388978
-22499,PPP1CC,0.10118027466011704
-22506,PHETA1,0.12564059272269226
-22508,SH2B3,0.33200595797855154
-22509,ATXN2,0.10812227647598237
-22511,BRAP,0.06336104314107525
-22512,ACAD10,0.011702948456193223
-22513,ALDH2,0.4100538729263276
-22514,MAPKAPK5-AS1,-0.1618041923241122
-22515,MAPKAPK5,0.05128867721090215
-22516,TMEM116,-0.12367061008137088
-22517,ERP29,0.013344542927944585
-22519,NAA25,0.09887083160644658
-22521,TRAFD1,0.12228898869693061
-22522,HECTD4,0.06487120087557811
-22524,RPL6,-0.32628059949294347
-22525,PTPN11,0.03756725604115155
-22526,RPH3A,0.12986641850665462
-22527,OAS1,0.33891408270724843
-22529,OAS3,0.2569797283051684
-22530,OAS2,0.31500613397151067
-22534,DDX54,-0.009373615523114486
-22535,RITA1,-0.10822323432469373
-22538,TPCN1,0.1845249704711962
-22540,SLC8B1,-0.002325298470588354
-22541,PLBD2,0.04253983819449033
-22543,SDSL,0.14385154383371168
-22547,RBM19,-0.038410774436891046
-22572,MED13L,0.3167671667476376
-22574,AC130895.1,0.11848922275080111
-22578,LINC00173,0.1089647552842107
-22584,C12orf49,-0.06903545106326447
-22588,FBXW8,-0.07755125067374383
-22592,TESC,0.0993650291597447
-22594,FBXO21,0.010889397883407958
-22599,RFC5,-0.0369141068139143
-22601,WSB2,0.15078243830333213
-22603,VSIG10,0.052291169259747125
-22605,PEBP1,-0.24502183631813187
-22606,TAOK3,0.07503901939717428
-22608,SUDS3,-0.033628744656487596
-22627,PRKAB1,0.05040466039720388
-22631,RAB35,0.01073317449128033
-22633,GCN1,-0.015269825183901504
-22634,RPLP0,-0.14554545458359733
-22635,PXN-AS1,-0.05457176326048392
-22636,PXN,0.03775361450817629
-22643,COX6A1,-0.1644052786080898
-22644,TRIAP1,0.002019436475278672
-22645,GATC,-0.006700199163114931
-22646,SRSF9,0.04463608481253223
-22647,DYNLL1,0.0758171800646341
-22650,COQ5,-0.09132403728220322
-22651,RNF10,0.11101942039338461
-22652,POP5,-0.19694968720322273
-22655,MLEC,0.02029328513232541
-22659,UNC119B,0.014675999024567073
-22661,ACADS,-0.0414408935296935
-22663,SPPL3,0.06425991058554391
-22667,C12orf43,-0.03710816326449729
-22668,OASL,0.1824532331853922
-22669,P2RX7,0.27057448157308206
-22672,P2RX4,0.17909668659013805
-22673,CAMKK2,0.3506817293654365
-22674,ANAPC5,0.04212612149266465
-22676,RNF34,0.01818086082047082
-22677,KDM2B,-0.02201248366345357
-22679,ORAI1,-0.12028952048548257
-22680,MORN3,-0.12331369736773645
-22682,TMEM120B,-0.24574946758335614
-22683,RHOF,-0.33389607370361385
-22684,LINC01089,-0.11554908434755719
-22687,SETD1B,0.07633962078950446
-22692,PSMD9,-0.055623086562140156
-22695,BCL7A,0.004968070407209331
-22697,MLXIP,0.14097398337642061
-22703,VPS33A,0.033331736391025046
-22704,CLIP1,0.2257646472857074
-22706,ZCCHC8,0.03038157531007844
-22708,RSRC2,-0.06921936832108887
-22713,HCAR3,0.1347113100157907
-22715,DENR,-0.06106048878658337
-22717,HIP1R,-0.14110848333886528
-22718,VPS37B,-0.023781002253704386
-22721,OGFOD2,0.05375207102690644
-22722,ARL6IP4,-0.048077474480150775
-22723,PITPNM2,-0.11253254890984733
-22725,MPHOSPH9,-0.09052996480269945
-22726,C12orf65,-0.21482390173740876
-22727,CDK2AP1,0.194253187031726
-22729,SBNO1,0.004352751313517531
-22730,SBNO1-AS1,-0.08148074401726779
-22731,KMT5A,-0.031002747671030687
-22732,RILPL2,0.29633054129573133
-22733,SNRNP35,-0.047678216993655605
-22738,TMED2,0.07081217040185278
-22739,DDX55,-0.03980389331593487
-22740,EIF2B1,-0.025177054619194623
-22741,GTF2H3,0.010067202674307878
-22744,ATP6V0A2,-0.0008711828938363033
-22747,CCDC92,-0.07368348025885461
-22755,ZNF664,-0.001989591340246505
-22759,NCOR2,0.12350322450197793
-22761,SCARB1,0.1625747863990096
-22763,UBC,0.08432043257301677
-22764,DHX37,0.009428857437134553
-22765,BRI3BP,0.14456188856534186
-22767,AACS,0.0013682555703732828
-22811,SLC15A4,0.014074944528508939
-22813,GLT1D1,0.30547916824800897
-22834,STX2,-0.0448598822015389
-22837,RAN,-0.23618950802420582
-22855,SFSWAP,0.033912383733790394
-22857,MMP17,0.11690353026263209
-22859,ULK1,0.10947297237286546
-22861,PUS1,-0.03028641823351722
-22863,EP400,-0.017095619940553114
-22866,DDX51,-0.03223782759338769
-22867,NOC4L,0.0075874309496618605
-22869,LINC02361,-0.0886379678196706
-22880,FBRSL1,0.017520874583127396
-22884,POLE,0.09792517226701412
-22885,PXMP2,-0.059435881821421885
-22886,PGAM5,-0.056852527727555745
-22887,ANKLE2,0.036783716678347
-22888,GOLGA3,0.0654266753349734
-22889,CHFR,0.04810805019118071
-22892,ZNF605,-0.06808614725541182
-22893,ZNF26,-0.017219018490127545
-22895,ZNF84,-0.07022532151630452
-22896,ZNF140,0.010064155673106095
-22897,ZNF891,-0.005074548573398705
-22900,ZNF268,-0.024722482619675034
-22917,MPHOSPH8,-0.15595886630566907
-22919,PSPC1,0.07894775922028638
-22920,ZMYM5,-0.023826064057386125
-22922,AL355001.2,0.040904659628822275
-22923,ZMYM2,0.09764122161261657
-22929,CRYL1,0.07599533076642732
-22931,IFT88,-0.02493036581853062
-22934,EEF1AKMT1,-0.037502289040832214
-22936,XPO4,0.005447205152475542
-22937,LATS2,0.015657918174035515
-22939,SAP18,-0.11802694158846512
-22941,MRPL57,-0.1767163940752586
-22944,ZDHHC20,0.21026851694204965
-22946,MICU2,-0.05982214155573794
-22956,SACS,-0.12698393078823825
-22961,MIPEP,-0.03724786605077322
-22965,SPATA13,-0.06719958407240725
-22971,PARP4,0.08991935885218452
-22978,CENPJ,0.02917672899407747
-22984,MTMR6,0.1922459093466855
-22986,NUP58,0.14580457071523684
-22990,RNF6,0.004619222324426253
-22992,CDK8,0.05108787744112902
-22999,USP12,-0.007812782089077824
-23004,RPL21,-0.36728767962570874
-23007,GTF3A,-0.24791401083160844
-23008,MTIF3,-0.013014728774946865
-23009,LNX2,-0.010707571693772197
-23010,POLR1D,0.041105711841808695
-23018,FLT3,0.08859374323377055
-23020,PAN3,0.08232958753985099
-23022,POMP,0.0649677892679364
-23023,SLC46A3,-0.0359752438886336
-23027,SLC7A1,-0.006802517187793167
-23028,UBL3,0.1390217130690458
-23036,KATNAL1,-0.022677904231752876
-23038,LINC00426,-0.13275264521526042
-23042,HMGB1,-0.02118834077430362
-23044,USPL1,0.0016317340944231719
-23045,ALOX5AP,-0.08616972733706839
-23054,HSPH1,0.04972018968234379
-23059,FRY,0.2370659495476278
-23062,BRCA2,0.05499554921639215
-23063,N4BP2L1,0.09065046998747069
-23064,N4BP2L2,-0.05466055263024956
-23065,N4BP2L2-IT2,0.029078439019696842
-23066,PDS5B,0.09359829199324357
-23081,RFC3,0.039521869273225395
-23094,SPART,0.13267954671777027
-23098,RFXAP,-0.033779902900899404
-23101,ALG5,-0.14584265931455404
-23102,EXOSC8,-0.1442877344860003
-23103,SUPT20H,0.011611148619173636
-23112,UFM1,-0.04042671306365861
-23118,PROSER1,0.04880138838076566
-23119,NHLRC3,0.11859635741400162
-23122,COG6,-0.008676956252451485
-23125,FOXO1,-0.07187198766674929
-23126,MRPS31,-0.06823831316539723
-23129,ELF1,0.08916408769400332
-23130,WBP4,-0.004158411365261806
-23133,KBTBD7,0.05478450482887091
-23134,MTRF1,-0.07150086459596537
-23136,NAA16,-0.08529512915864557
-23137,RGCC,-0.1346653221469959
-23138,VWA8,-0.021860059402911546
-23140,DGKH,-0.01653467459560601
-23142,AKAP11,0.0391463641664753
-23148,EPSTI1,0.3319847904178926
-23149,DNAJC15,-0.03335127729460259
-23159,LACC1,0.12049576197802245
-23170,TSC22D1,0.0323678700183047
-23175,NUFIP1,-0.010811065479450961
-23177,GPALPP1,0.0437045354993785
-23178,GTF2F2,0.0035937779587471553
-23180,TPT1,-0.1577217390767606
-23182,TPT1-AS1,-0.04379996630146568
-23184,SLC25A30,0.07437954772013139
-23186,COG3,0.004495207982536085
-23191,ZC3H13,-0.04506265180323886
-23194,LCP1,0.40025154706565896
-23198,RUBCNL,0.05226202823517762
-23201,LRCH1,0.05870596794360781
-23203,ESD,0.09560512592948646
-23206,SUCLA2,-0.007849221292655353
-23211,NUDT15,0.04588232157132476
-23213,MED4,-0.15409340197490715
-23215,ITM2B,0.33045798493174583
-23217,RB1,0.25183231082954555
-23219,LPAR6,0.2221860349395595
-23220,RCBTB2,0.1480283049253099
-23225,CYSLTR2,0.127267081273152
-23227,FNDC3A,0.10080183863324987
-23229,CDADC1,-0.05556553581501588
-23231,SETDB2,0.03344720311669455
-23232,PHF11,-0.01036611950666961
-23233,RCBTB1,0.04128550524263277
-23234,ARL11,0.26306943061917376
-23235,EBPL,-0.10632461656837114
-23236,KPNA3,-0.013639090769669845
-23238,SPRYD7,-0.08087698186529266
-23239,DLEU2,0.24421159980781446
-23240,TRIM13,-0.0007962121772204151
-23242,DLEU1,0.017528602079513157
-23245,DLEU7,0.11258854154777677
-23247,RNASEH2B,0.04714351511104639
-23252,INTS6,0.06133030555488785
-23253,INTS6-AS1,-0.03921787669666852
-23254,WDFY2,0.2881841554423391
-23255,DHRS12,-0.012266835203142892
-23260,ALG11,-0.02809920182987339
-23264,NEK3,0.07748809888098687
-23267,VPS36,-0.07272875910624879
-23269,CKAP2,0.06307019017533026
-23273,SUGT1,-0.0008576930583994259
-23304,TDRD3,-0.03258979257579914
-23342,MZT1,-0.055202901314407804
-23343,BORA,0.024573431058617513
-23344,DIS3,-0.03521598910230758
-23345,PIBF1,-0.10623520495286609
-23349,KLF12,-0.275422491074582
-23351,LINC00402,-0.09164268445495052
-23357,TBC1D4,-0.0584872986283178
-23359,COMMD6,-0.26478422085593584
-23360,UCHL3,0.0065285031542474435
-23362,LMO7,-0.1045395037532416
-23371,KCTD12,0.5287730474559125
-23374,CLN5,0.002338548280737813
-23375,FBXL3,0.14835400947521704
-23378,MYCBP2,0.1709807041386299
-23384,SLAIN1,-0.10461923948671874
-23394,OBI1,-0.052789742488880735
-23396,RBM26,-0.027684360732798178
-23397,RBM26-AS1,-0.05747343244370152
-23467,TGDS,-0.01235925283377822
-23468,GPR180,-0.0021542155341851606
-23474,ABCC4,0.015098251619274795
-23478,DNAJC3-DT,-0.046459788231880565
-23479,DNAJC3,0.08549281578936115
-23481,UGGT2,0.17888263889890646
-23486,MBNL2,0.05576881264692654
-23489,RAP2A,-0.044031898068277425
-23495,IPO5,0.02344952679260517
-23501,STK24,0.1367443204360159
-23504,DOCK9,-0.14434611355709928
-23508,UBAC2,-0.12679306615982217
-23509,GPR18,-0.17334994854285957
-23510,GPR183,-0.12971396426043563
-23512,TM9SF2,0.3450050376442936
-23513,LINC01232,0.026176461062279586
-23518,CLYBL,-0.06187144476433837
-23527,PCCA,0.040341891862668074
-23530,GGACT,-0.006637730807733414
-23533,TMTC4,0.03353060055925764
-23545,TPP2,0.01401769600032713
-23550,TEX30,0.014357281377092383
-23552,BIVM,-0.06592914489961725
-23553,ERCC5,0.02076975757384117
-23571,ARGLU1,-0.1751502825301782
-23583,LIG4,-0.007581112983281692
-23584,ABHD13,0.12116458031746383
-23585,TNFSF13B,0.5554960368902753
-23597,IRS2,0.23987441380957214
-23609,RAB20,0.19792411164985482
-23611,NAXD,0.009111635443866282
-23612,CARS2,0.23775207825053454
-23613,ING1,0.060727887588383045
-23616,ANKRD10,-0.06005304192310985
-23617,ANKRD10-IT1,0.10322021382375064
-23622,ARHGEF7,0.04728262062811765
-23638,TUBGCP3,0.08884077421901877
-23641,AL139384.1,0.0980878826488004
-23642,ATP11A,0.3092050412615604
-23656,PCID2,-0.17387577007597838
-23657,CUL4A,0.10216308002250965
-23658,LAMP1,0.18166597184400227
-23663,DCUN1D2,0.0511790876019179
-23665,TMCO3,0.13381957374703415
-23668,TFDP1,0.04825121224806244
-23672,GAS6-AS1,-0.007973849028939526
-23673,GAS6,0.1256100792101876
-23681,RASA3,-0.16061745510072967
-23685,CDC16,0.06014004073957379
-23687,UPF3A,-0.17362891672357594
-23688,CHAMP1,0.0982028852029366
-23723,TTC5,-0.10943010822246442
-23726,CCNB1IP1,-0.09506541684816867
-23728,AL355075.4,0.14473172052506597
-23729,PARP2,-0.07378717072256276
-23731,TEP1,0.2095254536441842
-23733,OSGEP,-0.009887642174720185
-23735,APEX1,0.015537722173594157
-23736,PIP4P1,0.001312347296304576
-23737,PNP,-0.012081451077565343
-23746,RNASE4,0.1448955368911046
-23750,RNASE6,0.33247568303857705
-23759,RNASE2,0.3000753656990925
-23760,METTL17,-0.060374270872579315
-23763,NDRG2,0.0065247154384860066
-23770,ARHGEF40,0.25498177594323584
-23775,LINC00641,0.0159539691695258
-23776,HNRNPC,0.09608760347631559
-23778,SUPT16H,-0.0689748964951129
-23780,CHD8,0.0865803415549337
-23781,RAB2B,-0.041727648697638536
-23782,TOX4,-0.025430146875065333
-23783,METTL3,0.018394112644001953
-23856,TRDC,-0.4432270050942198
-23918,TRAC,-0.41779355214879693
-23921,DAD1,-0.34131032628538277
-23923,ABHD4,0.052067186339854084
-23926,OXA1L,0.24706276065851723
-23927,SLC7A7,0.5209382899485507
-23928,MRPL52,-0.019109396065519722
-23930,LRP10,-0.1276566299795672
-23932,RBM23,0.024959477245818148
-23934,PRMT5,0.026605467157413777
-23936,HAUS4,0.15892654940615852
-23940,PSMB5,-0.00548895605329621
-23943,ACIN1,0.09694403712957538
-23944,C14orf119,-0.11100546766031771
-23949,HOMEZ,-0.02819436096795833
-23950,PPP1R3E,-0.05713216517396781
-23951,BCL2L2,0.09953210109292
-23953,PABPN1,0.10414513424973768
-23962,NGDN,-0.0701355462975708
-23963,ZFHX2-AS1,0.04504668457500709
-23967,AP1G2,-0.012079312454778566
-23968,AL135999.1,0.1207600658287935
-23975,DHRS4-AS1,-0.024943259249550553
-23976,DHRS4,0.08800617365994265
-23977,DHRS4L2,0.15337394199847448
-23978,AL136419.3,-0.04223620781949071
-23982,PCK2,0.1566397098698699
-23983,DCAF11,0.056564048899384045
-23985,PSME1,-0.05238183519850339
-23986,EMC9,-0.006364582607511505
-23987,AL136295.2,0.02779406082446981
-23988,PSME2,0.21884556810496417
-23989,RNF31,0.022867907269680653
-23990,IRF9,-0.05455137289273327
-23991,REC8,0.01570316421049715
-23992,IPO4,-0.09234251179868919
-23994,TM9SF1,0.07573368949101048
-23996,AL136295.7,0.04149195264620275
-23998,CHMP4A,-0.11694504470158798
-23999,MDP1,-0.007357888186055674
-24000,NEDD8,-0.05488159422906632
-24001,GMPR2,0.0307204639933895
-24002,TINF2,-0.03782710120325198
-24004,RABGGTA,0.024218874248777812
-24005,AL096870.8,0.09431989606487816
-24007,DHRS1,-0.04310261727555816
-24008,NOP9,0.04921978442932396
-24009,CIDEB,0.10675228143134
-24011,LTB4R,0.2062196702481875
-24013,RIPK3,0.07900262510675554
-24017,KHNYN,0.06585628592986975
-24018,SDR39U1,0.0006458342697223737
-24023,GZMH,-0.5536619822167544
-24024,GZMB,-0.443754516694892
-24059,G2E3,0.04405012123423198
-24060,SCFD1,0.03788228944142783
-24064,STRN3,0.042975663209326054
-24065,AP4S1,-0.08218191189748812
-24066,HECTD1,0.04640434751622811
-24068,HEATR5A,0.15798719186122165
-24070,DTD2,-0.023792098873768423
-24072,NUBPL,-0.021579581506898284
-24079,ARHGAP5,-0.018478796545169658
-24089,SPTSSA,0.2236969301146083
-24090,EAPP,-0.0927887865127147
-24093,SNX6,0.005846249443486201
-24094,CFL2,-0.05194025467428462
-24095,BAZ1A,0.26033448101268303
-24096,AL121603.2,-0.030537098639451903
-24097,SRP54,0.003499377956158874
-24098,FAM177A1,-0.13064160500220684
-24099,PPP2R3C,0.016917404639837506
-24100,PRORP,-0.14873458093425296
-24101,PSMA6,0.04067559296744325
-24103,NFKBIA,0.14828573360451947
-24106,RALGAPA1,-0.027898588265523126
-24116,MBIP,-0.0315891961918352
-24142,SEC23A,0.09910995529021897
-24144,GEMIN2,-0.09855802332751235
-24145,TRAPPC6B,-0.036973124840162186
-24146,AL109628.2,-0.08604012141523898
-24148,PNN,-0.2163903501424646
-24149,MIA2,-0.017216719396031623
-24151,FBXO33,-0.05266369132560366
-24169,C14orf28,-0.0006024447428455648
-24171,KLHL28,-0.10919885495315615
-24172,TOGARAM1,-0.0515094315687257
-24175,PRPF39,-0.061215571196866554
-24176,FKBP3,-0.1532526320279989
-24177,FANCM,-0.02423848626489299
-24178,MIS18BP1,0.37994441900437137
-24191,RPS29,-0.4677588183229408
-24192,LRR1,0.047994518241196056
-24193,RPL36AL,-0.4264820701918506
-24194,MGAT2,0.09409094547021447
-24196,DNAAF2,-0.06271618201272719
-24199,KLHDC1,-0.10436183948512424
-24201,KLHDC2,-0.12494211137682637
-24202,NEMF,0.01565336746599139
-24203,AL627171.2,-0.1330674309029542
-24204,AL627171.1,-0.024404894263626027
-24205,ARF6,0.0481114749283949
-24206,LINC01588,-0.08062059852220588
-24207,VCPKMT,-0.09196301705141645
-24208,SOS2,0.19058191458072857
-24209,L2HGDH,-0.025018462833670315
-24210,DMAC2L,-0.0980513752185987
-24214,MAP4K5,-0.06409539149135572
-24217,SAV1,0.07683250134941223
-24219,NIN,0.21321270507923235
-24220,AL606834.2,-0.025350638479527704
-24224,PYGL,0.37561057542784254
-24231,TMX1,0.03947625842908451
-24242,GNG2,-0.32498401323320564
-24246,RTRAF,-0.0172790656177639
-24249,PTGDR,-0.4932800085432136
-24250,PTGER2,-0.22949150542548818
-24251,TXNDC16,0.049163541242125595
-24253,ERO1A,0.09950216356730868
-24254,AL133453.1,-0.11778181564402171
-24255,PSMC6,0.10443879200248346
-24256,STYX,0.05826602660390091
-24257,GNPNAT1,-0.09003589644911765
-24263,DDHD1,-0.1006331397421344
-24273,CNIH1,0.03402668508469494
-24275,GMFB,0.12231901361127269
-24276,CGRRF1,-0.06418919260861326
-24277,SAMD4A,0.2882838146052272
-24279,GCH1,0.29188713562053337
-24281,SOCS4,0.08818450499930111
-24282,MAPK1IP1L,0.11604955387915139
-24283,LGALS3,0.4902832599853588
-24287,FBXO34,0.013970256973195707
-24289,ATG14,-0.03032103196466237
-24292,KTN1,-0.003278183297914829
-24297,PELI2,0.19762428061665352
-24301,TMEM260,0.05573689892174415
-24311,EXOC5,0.07577879310995174
-24312,AP5M1,0.025118543672903886
-24315,NAA30,-0.09890282367965991
-24321,ACTR10,0.022483284223890624
-24322,PSMA3,-0.05244585534290409
-24324,PSMA3-AS1,0.051697372823453026
-24326,ARID4A,0.0027205948167370263
-24330,TIMM9,-0.010164854576848731
-24331,KIAA0586,-0.1348213849344729
-24334,DAAM1,0.047319117018554874
-24336,L3HYPDH,0.0019954337712946857
-24337,JKAMP,0.1489165863208888
-24339,RTN1,0.31846988896428263
-24343,PCNX4,-0.045981434072008984
-24344,DHRS7,-0.2197551806079215
-24346,PPM1A,0.026162842553936107
-24354,MNAT1,-0.16039195305539683
-24355,TRMT5,0.03231146949521868
-24356,SLC38A6,0.07796239915804207
-24357,PRKCH,-0.46501477191095125
-24359,AL359220.1,-0.1615944383504819
-24364,HIF1A,0.29594912182385563
-24366,SNAPC1,-0.0028165810257682975
-24376,PPP2R5E,0.06837658551326721
-24378,WDR89,-0.04555502092315619
-24380,AL136038.5,-0.09080614198654254
-24381,SGPP1,-0.03432451529836185
-24382,SYNE2,-0.4706948147746654
-24386,MTHFD1,0.0015335370544508572
-24389,ZBTB25,-0.08413483973932435
-24391,ZBTB1,-0.10109372925797455
-24396,PLEKHG3,-0.11928931299745797
-24398,CHURC1,-0.061304569098423536
-24401,FNTB,0.07805899776115822
-24403,MAX,0.06660754494530159
-24407,FUT8,-0.09681718369650931
-24415,GPHN,-0.002640979800932271
-24418,MPP5,-0.01822523267293198
-24420,ATP6V1D,0.04455376106814802
-24421,EIF2S1,-0.10724196144687363
-24423,TMEM229B,0.07854769432491258
-24425,PIGH,0.005009482201986922
-24428,VTI1B,0.10092486765925879
-24429,RDH11,0.011139754286568682
-24431,ZFYVE26,0.10897269843872842
-24432,RAD51B,-0.04123281329730689
-24439,ZFP36L1,0.2761897193142464
-24440,ACTN1,0.10802762567773079
-24442,DCAF5,-0.1152316451175439
-24448,ERH,-0.044211394509914466
-24449,SLC39A9,0.0697168567468582
-24454,SUSD6,0.1580146079006017
-24456,SRSF5,-0.28366889495198827
-24462,COX16,-0.040534027149561315
-24463,SYNJ2BP,0.010697242306934331
-24467,MED6,-0.11474965830086567
-24469,TTC9,-0.05243081345170204
-24475,PCNX1,-0.0044946596987970894
-24483,SIPA1L1,0.17942460790873052
-24492,DCAF4,-0.06741178952507909
-24493,ZFYVE1,0.03181452784717924
-24495,RBM25,0.02244228041166494
-24496,PSEN1,0.21589560326126395
-24500,NUMB,0.3884754215126958
-24501,AC005280.2,0.181565389789517
-24504,RIOX1,-0.025805316160678247
-24512,DNAL1,-0.017623242977451103
-24514,PNMA1,-0.09291989726278747
-24515,ELMSAN1,0.09796626842180327
-24520,ZNF410,0.010631161158520492
-24524,COQ6,-0.03569864144603815
-24525,ENTPD5,-0.0006777129960349319
-24527,ALDH6A1,0.023676640204071907
-24528,LIN52,-0.0798785430158817
-24530,ABCD4,0.005627243026549298
-24536,NPC2,0.45503343517912787
-24537,ISCA2,-0.11936970345983858
-24541,AREL1,0.06295478591646737
-24542,FCF1,-0.02568828067098305
-24543,YLPM1,0.046654249671624175
-24545,DLST,-0.026231362931345644
-24548,EIF2B2,-0.03132633708154661
-24550,MLH3,0.023742786697707033
-24551,ACYP1,-0.14066750603850947
-24553,NEK9,0.09326480137366087
-24555,TMED10,-0.10352288970538037
-24559,FOS,0.549191793045046
-24560,LINC01220,0.09579021966496895
-24562,JDP2,0.3089209876908251
-24563,BATF,-0.3087929618284475
-24565,FLVCR2,0.23062019420059401
-24566,TTLL5,0.042856662847958096
-24567,ERG28,-0.20032858128597708
-24569,IFT43,-0.08819279826642033
-24573,GPATCH2L,0.13600838538975404
-24579,VASH1,-0.017478452512383516
-24583,ANGEL1,0.010987282009322365
-24589,IRF2BPL,0.05484301831202878
-24590,AC007686.3,-0.09772881842456223
-24595,CIPC,0.07411224794856404
-24601,GSTZ1,0.10011912029243192
-24602,TMED8,0.10070917875256459
-24605,VIPAS39,-0.049815510268478504
-24606,AHSA1,-0.07079461110311731
-24608,SPTLC2,0.37599428560830656
-24609,ALKBH1,0.05946800749950338
-24610,SLIRP,0.08563652404935797
-24611,SNW1,-0.02969183638373312
-24613,ADCK1,0.02287344753400586
-24628,CEP128,-0.12885602097035967
-24632,GTF2A1,0.05867013728398205
-24636,SEL1L,0.15405362953228746
-24653,LINC02328,-0.06727656401253686
-24666,GALC,-0.01402169921246165
-24667,GPR65,-0.07284198692534044
-24679,ZC3H14,-0.0887604687510747
-24685,FOXN3,0.14977875692204515
-24694,TDP1,-0.06998459137922512
-24696,PSMC1,-0.02912030974031472
-24697,NRDE2,-0.003252318096075279
-24700,CALM1,-0.6217863472376204
-24712,RPS6KA5,-0.004063739706309966
-24713,DGLUCY,0.06202495164884017
-24714,GPR68,-0.2386198759636363
-24718,CCDC88C,-0.14298694469313608
-24720,PPP4R3A,0.09295066196030864
-24722,AL121839.2,-0.13675827781705774
-24723,TC2N,-0.2990860099702903
-24724,FBLN5,-0.14187232036897665
-24725,TRIP11,-0.056979595166694895
-24726,ATXN3,0.1429843883969752
-24727,NDUFB1,0.05848675786207022
-24728,CPSF2,0.22284135104794714
-24731,SLC24A4,0.3131347650849459
-24732,RIN3,0.179372006580688
-24734,GOLGA5,0.02786919216274897
-24738,ITPK1,0.09685347797586809
-24740,MOAP1,-0.15253927327305988
-24741,TMEM251,-0.02063006374356418
-24742,GON7,-0.12020447708395575
-24743,UBR7,-0.04059090403656086
-24744,BTBD7,0.09150825355196623
-24755,DDX24,-0.22594684588882158
-24756,IFI27L1,-0.021990202176041106
-24758,IFI27L2,0.0006536865170978967
-24762,SERPINA1,0.581038044877951
-24777,DICER1,0.36964056257268546
-24779,CLMN,0.26552431924276226
-24782,SYNE3,0.1135773441956131
-24784,SNHG10,-0.124279507745358
-24785,GLRX5,-0.06547205229047051
-24791,TCL1A,-0.005460200500739307
-24802,ATG2B,0.024352003229613464
-24803,GSKIP,-0.017620462731507563
-24806,PAPOLA,0.16036762932659954
-24811,VRK1,-0.046024366231067565
-24823,LINC01550,-0.1237689250696923
-24832,BCL11B,-0.3528237166713455
-24835,SETD3,-0.010585988694159021
-24836,CCNK,0.11513790614162864
-24844,EVL,-0.486154955423475
-24849,YY1,0.07469029805700765
-24852,SLC25A29,0.022423075685010747
-24856,WARS,0.4793037820785906
-24858,WDR25,0.03850603141955708
-24873,AL132709.8,0.15223582595183993
-24888,PPP2R5C,-0.34623470134053064
-24891,AL118558.3,0.04306768954840655
-24893,DYNC1H1,0.13557704259544606
-24894,HSP90AA1,-0.09521162064347231
-24895,WDR20,0.051755669373016115
-24897,ZNF839,-0.06829000833049077
-24898,CINP,0.015018772483321759
-24899,TECPR2,0.06335215575622612
-24902,RCOR1,0.25949352068966214
-24904,TRAF3,0.06104826009042491
-24906,CDC42BPB,0.1826656175444856
-24912,TNFAIP2,0.5357083197902209
-24916,EIF5,-0.0015985867785226924
-24917,MARK3,0.07280208115302365
-24918,CKB,0.09635212768494683
-24920,TRMT61A,-0.018993381796098108
-24922,BAG5,-0.05148794513952124
-24923,KLC1,0.014285874652697638
-24924,COA8,-0.03591622280150519
-24925,AL049840.2,-0.14334589306323317
-24928,AL049840.4,0.08410092819786949
-24929,AL049840.5,0.027423885805232265
-24931,ZFYVE21,0.001465979229785489
-24936,ATP5MPL,0.15466098941384754
-24951,INF2,0.05533925514473654
-24954,SIVA1,-0.09412278780943635
-24956,AKT1,0.07731147095761819
-24963,PLD4,0.08139557289161403
-24966,CDCA4,-0.08462345668540229
-24967,GPR132,0.0422285440797503
-24972,NUDT14,0.019272454824882886
-24973,BRF1,0.06718654163746812
-24974,BTBD6,-0.0570401214224055
-24975,PACS2,0.008087848174612947
-24978,MTA1,-0.008605931340664048
-24980,CRIP2,-0.06142817295386809
-24981,CRIP1,-0.1173329045429076
-24982,TEDC1,0.03283532993584841
-24990,IGHA1,0.001376310731389642
-24992,IGHG1,0.0041913495631793075
-24994,IGHD,0.0032296670779140314
-24995,IGHM,0.005060546986489158
-25032,FAM30A,0.0025524104207324816
-25215,NIPA2,0.027934432707054254
-25216,CYFIP1,0.3222511710087893
-25217,TUBGCP5,0.017728734015281983
-25234,SNRPN,-0.19075504002690963
-25237,SNHG14,-0.1239570388719663
-25241,UBE3A,0.02551487167465503
-25244,ATP10A,-0.04349063225670071
-25268,HERC2,-0.08517565722159179
-25275,APBA2,-0.17199831238104465
-25278,NSMCE3,-0.19166050809171878
-25293,AC127502.2,-0.0404864598698089
-25302,FAN1,0.014190491568181383
-25303,MTMR10,0.21061012447084468
-25308,KLF13,-0.16578920284061924
-25313,GOLGA8O,-0.08223099116495437
-25314,LINC02256,0.05339475941745256
-25324,GREM1,-0.023037896832709032
-25325,FMN1,0.11913416512244093
-25331,AVEN,-0.08705682772038617
-25335,EMC7,-0.1544096652928877
-25336,PGBD4,-0.04813454209955951
-25337,KATNBL1,-0.061717371142200654
-25338,EMC4,-0.09332504603912777
-25339,SLC12A6,0.19500926154263207
-25341,NOP10,0.01793109496657908
-25343,LPCAT4,-0.010230140912476106
-25344,GOLGA8A,-0.09786148155408259
-25345,GOLGA8B,-0.14553969689655824
-25351,AQR,0.11311511906943271
-25352,ZNF770,0.057614473989943724
-25357,DPH6,-0.02658173319175746
-25374,SPRED1,0.17904893900946445
-25375,FAM98B,0.01540378820199794
-25376,RASGRP1,-0.28464135573336297
-25377,AC116158.3,-0.038560520431531596
-25390,THBS1,0.23469365859946922
-25398,EIF2AK4,0.13745700229987406
-25399,SRP14,-0.31992003608316194
-25400,SRP14-AS1,-0.04879158841586476
-25403,BMF,0.16846648966470182
-25410,PLCB2,0.26960157492678494
-25412,AC020658.5,0.025977511665124024
-25415,INAFM2,0.09743624715710102
-25420,KNSTRN,0.023742936265745983
-25421,IVD,0.015079512947701922
-25422,BAHD1,0.014487908483166142
-25425,CHST14,-0.03883218612264312
-25427,CCDC32,0.014913390553927316
-25428,RPUSD2,-0.07696488949894478
-25431,RAD51-AS1,-0.10299959484550524
-25433,RMDN3,-0.09259728096534948
-25434,GCHFR,-0.18341057671572045
-25435,DNAJC17,-0.00869257384059681
-25437,ZFYVE19,-0.045085742123413396
-25440,SPINT1,0.1449224008317388
-25443,VPS18,0.10686111198018466
-25447,INO80,-0.03730764487312162
-25453,CHP1,0.3187333121546876
-25454,OIP5-AS1,-0.036431708820879975
-25458,NDUFAF1,-0.02214676175802048
-25459,RTF1,-0.008276065519951345
-25462,RPAP1,0.03580469530934378
-25465,MGA,0.02759058431749396
-25467,MAPKBP1,0.01144535631897271
-25469,JMJD7,-0.022325094313575974
-25470,PLA2G4B,0.049968858883198644
-25472,AC020659.1,-0.10602264510561561
-25473,EHD4,0.08286183311387428
-25481,VPS39,0.1023002538584113
-25483,TMEM87A,0.016350739763664385
-25484,GANC,0.12701897485909566
-25485,CAPN3,0.1059600702897305
-25486,ZNF106,0.385054602471033
-25487,SNAP23,0.2576641622702352
-25488,AC018362.1,0.029809226725046347
-25489,LRRC57,0.039881926810299555
-25490,HAUS2,0.010781418024107526
-25493,CDAN1,0.04669589362995065
-25496,TTBK2,0.08678647703716498
-25499,UBR1,-0.03542311829434252
-25501,TMEM62,-0.02132728466677624
-25503,CCNDBP1,0.0027926571943871423
-25509,ADAL,-0.04862848351660725
-25510,ZSCAN29,0.02701800793307928
-25511,TUBGCP4,-0.028918811196153433
-25512,TP53BP1,-0.025846387625162906
-25517,CATSPER2,-0.051481594740372456
-25520,PDIA3,-0.02751398042918921
-25522,SERF2,0.0916615051200225
-25524,HYPK,-0.0009051630066517335
-25525,MFAP1,-0.14992666742650768
-25526,WDR76,-0.15807328671809506
-25528,CASC4,0.006491123758753707
-25531,CTDSPL2,-0.04710950735907245
-25533,EIF3J-DT,-0.0899957447528574
-25534,EIF3J,0.006443977178074167
-25536,SPG11,0.25257256794931854
-25537,PATL2,-0.24214041618187768
-25538,B2M,-0.5369649636930913
-25539,TRIM69,0.020493319247339253
-25543,SORD,0.09750271511380622
-25560,SPATA5L1,0.04552409377186406
-25567,BLOC1S6,0.20250513370383122
-25568,SQOR,0.35006613386269075
-25592,DUT,-0.06332858899365397
-25598,CEP152,0.07672818161990078
-25603,EID1,0.02690826703473762
-25605,SECISBP2L,0.04718164326938551
-25607,COPS2,0.014690242075485612
-25608,GALK2,0.05384795726616768
-25613,DTWD1,0.045406827506057935
-25614,ATP8B4,0.06996144984429967
-25618,GABPB1,0.0032460634848345796
-25619,GABPB1-IT1,-0.15241532089575804
-25620,GABPB1-AS1,-0.061411300902162666
-25622,USP8,0.2004038788679323
-25626,TRPM7,-0.0014238281658055085
-25628,SPPL2A,0.214549282881397
-25631,AP4E1,0.07403695283242552
-25641,DMXL2,0.47586571599475047
-25646,LYSMD2,0.1497837175676061
-25647,TMOD2,0.15327913571059265
-25649,TMOD3,0.1498070893295176
-25653,LEO1,-0.16149200172113842
-25654,MAPK6,0.07853414653106013
-25661,GNB5,-0.046593468094457316
-25666,MYO5A,0.21181173434378467
-25667,ARPP19,-0.05322348465233663
-25669,FAM214A,-0.03111477189162768
-25681,RSL24D1,-0.11392899239216694
-25682,RAB27A,0.11455818744277363
-25684,PIGBOS1,0.025617944086838866
-25685,PIGB,0.00016309917482848697
-25686,CCPG1,0.1508790683984243
-25695,NEDD4,-0.055991395533003106
-25696,RFX7,0.011112356879408512
-25704,ZNF280D,0.043843438086042005
-25709,TCF12,0.07762663268563225
-25710,LINC00926,0.0023857535011409386
-25718,POLR2M,0.024021607732555794
-25723,AQP9,0.2179631719343011
-25730,ADAM10,0.19986360362144145
-25733,MINDY2,0.0063282890944537816
-25736,RNF111,0.10687735443781166
-25737,SLTM,0.02667933877365067
-25742,MYO1E,0.08024060350518318
-25748,GTF2A2,-0.06404966218096529
-25750,BNIP2,0.263017503807131
-25756,ANXA2,0.4282615176972279
-25757,ICE2,-0.07216600376750355
-25759,RORA,-0.4025895588690783
-25775,VPS13C,0.20039518298161219
-25790,TPM1,0.0011211601356654213
-25794,LACTB,0.34248906388506695
-25795,RPS27L,-0.11763880024758716
-25796,RAB8B,0.026282353473929904
-25797,APH1B,0.12486602948808544
-25802,USP3,0.320004025856026
-25805,HERC1,0.06021797434355575
-25809,CIAO2A,0.19104471777451026
-25810,SNX1,0.15642617899387493
-25811,SNX22,0.01623217532536695
-25812,PPIB,-0.2478568138999639
-25813,CSNK1G1,0.016251061842784256
-25815,TRIP4,-0.04284327373613202
-25816,ZNF609,-0.029326206183571873
-25818,OAZ2,0.23840218923472625
-25824,PLEKHO2,0.13142929348387103
-25828,AC103691.1,0.061277683233824085
-25829,SPG21,0.27936838848809376
-25830,MTFMT,-0.0974363541543447
-25837,PDCD7,-0.10052989796824276
-25838,CLPX,0.03778143686127174
-25840,PARP16,-0.11869249885728729
-25843,DPP8,0.07562170198787156
-25844,HACD3,-0.16629170893136444
-25845,INTS14,-0.0247590957608781
-25848,DENND4A,0.0781085504949888
-25849,RAB11A,0.03072150799749508
-25850,AC011939.2,0.08758387088846523
-25853,DIS3L,0.010777806637033859
-25855,TIPIN,-0.06917950579769504
-25856,MAP2K1,0.12433656272418275
-25858,SNAPC5,-0.07504902752703872
-25859,RPL4,-0.015811692471643872
-25860,ZWILCH,-0.027719112846320358
-25869,SMAD3,0.04588156272862513
-25871,AAGAB,-0.0470386973869125
-25874,C15orf61,-0.09380897395834203
-25876,MAP2K5,0.035625245232437076
-25880,PIAS1,0.17840295123884517
-25881,CALML4,0.20001378420585925
-25882,CLN6,0.06376404389018368
-25884,FEM1B,0.05388742145403629
-25888,ANP32A,0.103827980614209
-25896,GLCE,0.13398920055854277
-25903,RPLP1,-0.0911501595211071
-25908,TLE3,0.16452726110897173
-25916,UACA,0.08228959777006828
-25931,MYO9A,-0.1013865850641592
-25936,PKM,0.30944265670238696
-25937,PARP6,0.013677384741234814
-25940,HEXA,0.10570147564824604
-25945,ARIH1,0.08753376858900491
-25951,BBS4,-0.024868928348049543
-25952,ADPGK,0.255627787894761
-25960,NPTN,0.22693960561699264
-25970,STOML1,-0.0003294988097500927
-25971,PML,0.18578631800283502
-25988,UBL7,0.08920572657934524
-25989,UBL7-AS1,-0.07747750209535935
-25991,ARID3B,-0.007675038944442213
-25992,CLK3,-0.02990800643123474
-25995,EDC3,-0.0034969616279444432
-25998,CSK,0.12396815588228209
-26001,ULK3,-0.05433234901933931
-26002,SCAMP2,0.058705693330581295
-26003,MPI,-0.01455945555993314
-26004,FAM219B,0.038418686159268654
-26005,COX5A,0.014693309593388559
-26006,RPP25,0.05896642854297799
-26008,PPCDC,0.14169254142789986
-26009,C15orf39,0.3047362099810903
-26014,COMMD4,-0.09180774276135888
-26016,NEIL1,-0.12847189450015656
-26017,MAN2C1,0.1258768713563349
-26019,SIN3A,0.07023347590473919
-26021,PTPN9,0.11525482319742088
-26022,SNUPN,-0.08480373178043074
-26025,IMP3,-0.19592828866654874
-26036,UBE2Q2,-0.14377818387888505
-26037,FBXO22,-0.004914121694427078
-26041,ETFA,0.037989328533312994
-26046,SCAPER,0.025049913956425682
-26047,RCN2,-0.17408693680778498
-26048,PSTPIP1,0.08475086975602363
-26049,TSPAN3,-0.09711978271389377
-26052,PEAK1,0.01829479037262229
-26053,HMG20A,-0.04904934549272156
-26062,TBC1D2B,0.11753970981388273
-26066,IDH3A,0.1547979234137336
-26070,DNAJA4,-0.010211102060169393
-26071,WDR61,0.02409409333603322
-26076,IREB2,0.10828317316594512
-26079,PSMA4,0.3695954885424227
-26086,MORF4L1,0.03053538045694253
-26087,CTSH,0.4383468841234031
-26092,TMED3,-0.04255579007385585
-26096,MTHFS,0.05819558578519993
-26098,AC015871.7,0.022041249556681474
-26099,AC015871.1,0.06464208748197245
-26100,AC015871.4,0.072235849219462
-26101,ST20-AS1,0.11858051843683985
-26102,BCL2A1,0.21590440434575525
-26103,ZFAND6,-0.04330763277303298
-26104,FAH,0.08825473852441844
-26119,MESD,-0.08321713503579957
-26122,TLNRD1,0.04466067430834274
-26124,IL16,-0.13042800768679358
-26138,EFL1,0.07190623119262693
-26143,RPS17,0.0848281922187431
-26150,SNHG21,-0.04099893950596842
-26152,WHAMM,-0.03948280149928515
-26155,RAMAC,-0.03371514257238476
-26156,C15orf40,-0.007927958838703633
-26158,BTBD1,0.09245666471036927
-26163,TM6SF1,0.11217570611620169
-26180,WDR73,-0.10261529761646358
-26182,SEC11A,0.1483345807058228
-26185,ZNF592,0.07029721770595893
-26190,PDE8A,0.13154111494064347
-26191,AKAP13,0.14898660155401994
-26213,MRPL46,-0.06248585457546668
-26214,MRPS11,-0.016079301471821506
-26215,DET1,-0.025775492012712915
-26217,AEN,-0.05128588576196223
-26218,ISG20,-0.25374779014362225
-26221,HAPLN3,0.010534002349670429
-26225,ABHD2,0.19307974273346643
-26228,POLG,0.08574526717709359
-26229,AC124068.2,-0.01536183778947383
-26245,ANPEP,0.3202607699498158
-26246,AP3S2,0.10748200096800493
-26248,ZNF710,0.19811830335442854
-26249,AC087284.1,0.13770321784889575
-26251,IDH2,-0.12766544731647134
-26253,SEMA4B,0.1513872123780156
-26254,CIB1,-0.19188568439701803
-26257,NGRN,-0.10915791346905572
-26260,IQGAP1,0.37135157823589643
-26262,CRTC3,-0.002318832194335914
-26269,BLM,-0.03236295373195504
-26272,FURIN,0.13295971883130933
-26273,FES,0.3650120843889582
-26274,MAN2A2,0.10495235081772805
-26276,HDDC3,-0.08798281389619844
-26277,UNC45A,-0.005671814228487222
-26279,RCCD1,0.0602647824159408
-26282,VPS33B,-0.018178873932089105
-26291,SLCO3A1,0.19637600013585446
-26306,LINC01578,-0.044120905350981426
-26307,CHD2,-0.06303028992119396
-26320,MCTP2,-0.27273199276518834
-26359,ARRDC4,0.16536727316292393
-26370,IGF1R,-0.04308167431129211
-26382,LRRC28,-0.013358137002473515
-26388,MEF2A,0.25953412473334025
-26400,LINS1,-0.09214308779543837
-26401,ASB7,0.0588098451738566
-26408,LRRK1,0.21632433373192586
-26414,CHSY1,0.17441780073622204
-26415,SELENOS,-0.07201641306754449
-26416,SNRPA1,-0.17847528790513068
-26424,TM2D3,-0.06354123081186865
-26425,TARSL2,-0.09331209888362138
-26432,POLR3K,-0.06676775426244724
-26433,SNRNP25,-0.07488470820752127
-26435,MPG,-0.0008920081042516031
-26436,NPRL3,-0.17482757805491986
-26445,LUC7L,0.015282549943118773
-26446,FAM234A,0.048967717643651945
-26450,AXIN1,-0.1121454649116017
-26451,MRPL28,0.05710073507886904
-26452,TMEM8A,0.035928753125324876
-26453,NME4,-0.040227091139414486
-26458,CAPN15,-0.07872829879736021
-26461,PIGQ,0.004195932696578058
-26463,RAB40C,0.06113232337576417
-26465,METTL26,-0.06878881388262403
-26466,MCRIP2,0.07616763831063719
-26470,RHOT2,0.007916273535858534
-26473,STUB1,-0.13905531009133731
-26474,JMJD8,0.03285280994940569
-26475,WDR24,0.007452709400800616
-26479,METRN,-0.17997501534057184
-26480,ANTKMT,-0.11836630159542957
-26482,HAGHL,-0.08482764829390427
-26483,CIAO3,0.01083876216951363
-26486,RPUSD1,-0.025955090718147743
-26491,LMF1,0.0029984194516948157
-26496,AC009041.2,-0.05589689120309244
-26498,AC009041.1,-0.08242567756759402
-26517,UBE2I,-0.010186275827846846
-26520,TSR3,-0.0898795650429188
-26521,GNPTG,-0.01023735632020332
-26523,UNKL,0.026178040415071813
-26525,C16orf91,-0.11636499007677173
-26528,CLCN7,0.16358433009652718
-26533,TELO2,0.07645204389996005
-26536,TMEM204,-0.14908527163574206
-26539,CRAMP1,0.07016508473922502
-26541,JPT2,-0.15325915889815667
-26542,MAPK8IP3,-0.01545777465052421
-26546,NME3,-0.13105094375248752
-26547,MRPS34,-0.09319603963345797
-26548,EME2,0.1122700827393825
-26549,SPSB3,-0.07422458005851354
-26550,NUBP2,-0.11792470400472053
-26552,HAGH,-0.041709200851220596
-26553,FAHD1,0.020074703063081213
-26559,MSRB1,0.3858030726535792
-26561,NDUFB10,-0.04670761577312382
-26562,RPS2,-0.3120828307523026
-26563,SNHG9,-0.10722316161637811
-26566,TBL3,-0.044782929274875184
-26568,GFER,-0.09077026756594714
-26572,ZNF598,0.021614339713359434
-26575,NTHL1,-0.047688529952330826
-26576,TSC2,0.0895012370271144
-26577,PKD1,0.01776614326119405
-26583,SNHG19,-0.060633240018411805
-26584,TRAF7,0.11971830271879062
-26586,MLST8,0.054891898052666546
-26588,PGP,-0.013861895380423848
-26590,E4F1,-0.03811011041269553
-26593,ECI1,0.07497399182471054
-26595,RNPS1,-0.013522950474164889
-26596,AC009065.4,0.007152854536643655
-26605,TBC1D24,-0.042958962985864636
-26607,ATP6V0C,0.4250653520470912
-26608,AMDHD2,-0.007999593224881389
-26610,PDPK1,0.1482892649729168
-26614,AC093525.4,0.0874450904089583
-26618,ERVK13-1,0.012135658562587824
-26619,KCTD5,0.026685720333433267
-26622,SRRM2,0.054229352206989226
-26623,ELOB,0.048127618017754135
-26628,ZG16B,0.1291861429798555
-26632,FLYWCH2,-0.019756582227152743
-26633,FLYWCH1,-0.018300631288732837
-26645,HCFC1R1,0.07847731626134329
-26646,THOC6,0.0698816294220081
-26649,MMP25-AS1,-0.2631575182135891
-26650,MMP25,0.09039181603871817
-26651,IL32,-0.5992539811341191
-26652,AC108134.3,0.1615358425390794
-26655,ZNF213-AS1,-0.07807359999053319
-26657,ZNF213,0.05092359019460287
-26658,AC108134.2,-0.026021780181178948
-26662,ZNF200,-0.033686296433250884
-26663,MEFV,0.3799120279170299
-26664,LINC00921,-0.1587147717929219
-26665,ZNF263,-0.04868272917161979
-26666,TIGD7,0.004385518098368692
-26667,ZNF75A,-0.10702121301305495
-26672,ZSCAN32,-0.0055247293746679095
-26673,ZNF174,-0.027597446884745507
-26675,NAA60,0.1102682227991141
-26677,CLUAP1,-0.12837129440053074
-26678,NLRC3,-0.2330906642365416
-26681,SLX4,0.06236023260585784
-26682,DNASE1,0.02275852565361759
-26684,TRAP1,0.023660330134081836
-26686,CREBBP,0.14744910805188105
-26688,ADCY9,0.036972917387534006
-26693,TFAP4,-0.038624774486344035
-26696,PAM16,-0.010809719504095308
-26698,CORO7,0.05095123673416621
-26700,DNAJA3,-0.07255522242114028
-26703,NMRAL1,-0.029151670737215755
-26704,HMOX2,-0.3328433651191357
-26705,CDIP1,-0.029267611141778938
-26709,UBALD1,-0.009620371515636737
-26710,MGRN1,0.19398324017709573
-26712,NUDT16L1,-0.15979962362865083
-26713,ANKS3,-0.07956367097566586
-26715,ZNF500,-0.002360588337711178
-26719,ROGDI,0.2766780575989286
-26720,GLYR1,0.20366154068900927
-26722,UBN1,0.03518892194748477
-26726,NAGPA,0.02658536187913145
-26727,ALG1,0.018989735423042768
-26729,EEF2KMT,-0.03867785049092098
-26748,METTL22,0.14773094973825798
-26749,ABAT,0.10545816519205203
-26750,TMEM186,-0.11004248401401107
-26751,PMM2,0.028007549242462067
-26754,CARHSP1,0.006920953670035123
-26758,USP7,0.16997434551352425
-26761,C16orf72,0.0666935421889067
-26763,AC087190.1,-0.12991838733529767
-26775,ATF7IP2,-0.09196570208353415
-26783,NUBP1,0.09384273798892313
-26786,CIITA,0.3903328492129584
-26787,DEXI,-0.24678130293007844
-26788,CLEC16A,0.027337152894568378
-26794,SOCS1,-0.15871024643990203
-26804,LITAF,-0.23102400679289659
-26805,SNN,0.2070736827673547
-26806,TXNDC11,0.09816462926333296
-26807,AC007613.1,-0.03758757586267762
-26808,ZC3H7A,0.1641058083946313
-26809,AC010654.1,0.1137897479619533
-26811,RSL1D1,0.01924568444711439
-26813,GSPT1,0.036635090831496894
-26818,SNX29,0.07353281044506299
-26828,CPPED1,0.49125612227272725
-26836,ERCC4,0.0009258669276089052
-26841,MRTFB,-0.02767515523720865
-26847,PARN,0.019363533550117728
-26848,BFAR,-0.04964169320512445
-26854,NOMO1,0.04640240716593706
-26858,PDXDC1,0.02461832394134021
-26860,NTAN1,0.015382909195192173
-26861,RRN3,-0.06860756611842493
-26866,MARF1,0.08662461580034163
-26868,NDE1,0.03349279821735798
-26870,AC026401.3,0.09152487439970243
-26871,MYH11,-0.1074381922675661
-26874,FOPNL,-0.014145817918515054
-26876,ABCC1,0.10857367963206861
-26879,NOMO3,0.045931666007088624
-26886,XYLT1,0.06960485750098687
-26901,RPS15A,-0.4809170298218713
-26902,ARL6IP1,-0.18911846050802067
-26904,SMG1,0.14139000935115792
-26909,COQ7,0.027106517028787042
-26912,ITPRIPL2,0.3151937448710807
-26913,AC099518.5,0.06044685591530981
-26924,GDE1,0.0735668716232841
-26925,CCP110,0.012932847745804714
-26926,VPS35L,0.06374441866257355
-26928,KNOP1,-0.06284192116319848
-26946,THUMPD1,-0.05763705409703946
-26948,ERI2,0.1266586875958378
-26950,DCUN1D3,0.07162749487257927
-26951,LYRM1,-0.013198128538330191
-26954,TMEM159,0.023364461984510546
-26962,METTL9,0.275554298876023
-26963,AC005632.3,0.27327766150015703
-26964,IGSF6,0.553611927171271
-26967,NPIPB4,0.00452826017902411
-26969,UQCRC2,0.06072462659657088
-26972,MOSMO,0.008595852891854563
-26975,SDR42E2,0.024665376278127676
-26977,EEF2K,-0.061627118048231296
-26979,POLR3E,-0.077918973655638
-26980,AC092338.3,-0.0779391431932411
-26981,CDR2,-0.1029495957177935
-26984,NPIPB5,0.06678516433383769
-26991,USP31,0.09215444450207937
-26996,COG7,0.031856323235905135
-26999,GGA2,-0.018629095583677948
-27001,UBFD1,0.0009707882064210534
-27003,NDUFAB1,-0.0988707815880854
-27004,PALB2,-0.01923005721917182
-27006,DCTN5,0.08687220833952551
-27013,PRKCB,0.29439188538385486
-27017,RBBP6,0.10173644299236
-27018,TNRC6A,-0.018137037731782792
-27022,ARHGAP17,0.17834821472663157
-27026,LCMT1,-0.020002904947697533
-27031,ZKSCAN2-DT,0.040933392736454674
-27044,NSMCE1,-0.12586010300040873
-27047,IL4R,0.12519519940609736
-27048,IL21R,-0.14987755268005967
-27050,GTF3C1,-0.08943956885286104
-27051,KIAA0556,0.025639766845348497
-27056,XPO6,0.07415834497605289
-27058,SBK1,-0.2264098987740095
-27064,CLN3,0.10149380448182926
-27065,APOBR,0.2703050446156773
-27066,IL27,0.138904447512716
-27069,SGF29,-0.09866063213809986
-27071,SULT1A1,0.2521539450630462
-27073,EIF3C,-0.048056581025733514
-27077,AC145285.6,0.08076745509437741
-27078,ATXN2L,0.0018015215324526108
-27080,TUFM,0.020326717261169136
-27081,SH2B1,-0.016346496655757416
-27085,RABEP2,-0.0037281109533286298
-27086,CD19,0.002737580656794063
-27087,NFATC2IP,0.05997897091984059
-27090,SPNS1,0.1180126550802935
-27091,LAT,-0.3216148372783685
-27093,AC009093.2,0.10353696891381388
-27099,BOLA2,0.05693353248001038
-27103,SPN,-0.12548630452637383
-27105,C16orf54,-0.1268518966205119
-27108,KIF22,0.0036715270831419356
-27110,MAZ,-0.11342720026908526
-27112,AC009133.1,-0.022629878026831158
-27114,PAGR1,-0.0385842770220981
-27115,MVP,0.29745289538635816
-27116,CDIPT,-0.027762825545286182
-27120,KCTD13,-0.01608985479249315
-27122,TMEM219,0.10707916617667887
-27123,TAOK2,0.019349658236987817
-27124,HIRIP3,-0.09674185449542423
-27125,INO80E,-0.041659761557944375
-27130,ALDOA,0.1112685777199023
-27132,PPP4C,0.09507573164685157
-27134,YPEL3,-0.015357967886082916
-27139,MAPK3,0.13372580376097642
-27140,CORO1A,-0.1894589862833235
-27141,AC012645.3,-0.004753114076037877
-27142,BOLA2B,-0.005537531223348526
-27144,SULT1A3,0.018258998048317838
-27146,CD2BP2,-0.12877864399659208
-27148,TBC1D10B,0.087204369368159
-27150,MYLPF,-0.10258340019614348
-27151,ZNF48,-0.09343272630054196
-27152,SEPTIN1,-0.2743122238441577
-27154,DCTPP1,-0.08826959042691997
-27155,SEPHS2,0.014538549179987337
-27156,ITGAL,-0.06477847513239135
-27160,ZNF768,0.07415240447728648
-27161,ZNF747,0.03982498197470129
-27162,AC002310.1,-0.0688653766168248
-27163,ZNF764,-0.1307125914046994
-27164,ZNF688,0.004610792635558847
-27166,ZNF785,0.016201651312839245
-27168,ZNF689,0.010559769634751345
-27169,PRR14,0.0850385535900256
-27170,FBRS,0.0034774587798551563
-27172,SRCAP,0.017311956322510452
-27173,TMEM265,0.03559613042767861
-27174,PHKG2,-0.020889487672112107
-27176,RNF40,0.08522057922833999
-27177,ZNF629,0.11907686219643084
-27179,BCL7C,-0.17483132298406487
-27180,MIR762HG,-0.034632825915316724
-27184,FBXL19,0.03859163423427806
-27185,ORAI3,0.09944367802901938
-27187,SETD1A,0.025128291089612325
-27190,STX4,-0.028023637516444763
-27193,ZNF668,-0.0273223792011297
-27195,ZNF646,0.04028621741099114
-27197,VKORC1,0.010369966679856634
-27198,BCKDK,0.26517074622434905
-27199,KAT8,0.023363853176432407
-27201,AC135050.6,0.11753508253235335
-27205,FUS,-0.2046418644637964
-27208,PYCARD,0.38801408830229034
-27209,PYCARD-AS1,0.009909161505865094
-27212,ITGAM,0.24471222843639823
-27213,ITGAX,0.2660869909436584
-27220,AC026471.1,-0.0017559471333466622
-27221,ARMC5,0.05744057492825251
-27225,C16orf58,-0.004783310981821651
-27233,ZNF720,-0.20068374871786182
-27235,ZNF267,0.12311847978108505
-27300,VPS35,0.19101318161326464
-27304,C16orf87,0.05152711434475544
-27306,DNAJA2,-0.0013109693763321624
-27309,NETO2,0.19504470383834774
-27311,ITFG1,0.07580614954563736
-27314,PHKB,0.16806534673936763
-27323,LONP2,0.00618778966498193
-27324,SIAH1,0.020788227956047917
-27326,N4BP1,0.002804522538507819
-27344,CNEP1R1,0.005316351509095013
-27346,HEATR3,0.16323009296873647
-27349,TENT4B,0.11525950152336596
-27350,ADCY7,0.1922512190818673
-27351,BRD7,0.04870762676948827
-27359,SNX20,0.14068922450875684
-27360,NOD2,0.3120451976711432
-27362,AC007728.2,0.06838087325413517
-27363,CYLD,-0.05589543589633147
-27377,HNRNPA1P48,-0.024702226365985753
-27393,CHD9,0.04692989713178436
-27397,RBL2,-0.20929937133430557
-27399,AKTIP,-0.12948518096613057
-27403,FTO,0.0479170727016732
-27430,LPCAT2,0.3980496478955719
-27431,AC007336.1,0.1459468133370059
-27434,CES1,0.24899232327234105
-27443,AMFR,0.24240466387021833
-27445,NUDT21,0.009580268625292153
-27446,OGFOD1,-0.01556632853916149
-27448,BBS2,-0.02866988617461374
-27451,MT2A,0.04682989686340158
-27453,MT1E,-0.10787346053805776
-27457,MT1F,-0.17031917784078765
-27460,MT1X,-0.05106072792113603
-27462,NUP93,0.07631951922889105
-27464,HERPUD1,0.1746939600920673
-27466,AC012181.1,0.16074200492116933
-27468,NLRC5,0.036586606429117326
-27471,CPNE2,0.10372313339754277
-27472,FAM192A,0.05519392422023706
-27474,RSPRY1,0.041750268475965586
-27475,ARL2BP,-0.025299883520478322
-27482,CIAPIN1,-0.033857403346138085
-27483,COQ9,0.020777066685910655
-27484,POLR2C,-0.13454827248363335
-27487,ADGRG5,-0.24569912679618
-27488,ADGRG1,-0.5529476251190266
-27493,KATNB1,0.06917141104286284
-27499,ZNF319,-0.1333372904059314
-27500,USB1,0.07367441162241628
-27502,CFAP20,-0.09065733907530249
-27504,AC009107.2,-0.1374440064039286
-27505,CSNK2A2,0.02701142001075578
-27514,SETD6,-0.06756680971187969
-27515,CNOT1,0.08393853094715213
-27517,SLC38A7,0.11311580278710898
-27518,GOT2,-0.06856035910877033
-27551,TK2,0.22113606519299303
-27554,CKLF,-0.04699994015733551
-27559,CMTM3,0.12066905659688054
-27560,CMTM4,0.10095455221646125
-27561,DYNC1LI2,0.16721350845807761
-27567,NAE1,-0.2234142061538945
-27573,CIAO2B,-0.11463992788021762
-27574,CES2,0.07713664782484553
-27576,CES4A,0.055837692338412424
-27577,CBFB,0.04214236484889593
-27578,C16orf70,0.19230505932635047
-27580,TRADD,-0.1804003261149847
-27581,FBXL8,-0.02394730408752877
-27583,NOL3,0.10368042059924289
-27585,EXOC3L1,0.09456361753430581
-27586,E2F4,0.044073818393296535
-27589,TMEM208,-0.010777936806413726
-27590,FHOD1,0.08702208922183463
-27595,TPPP3,0.19106634161216854
-27596,ZDHHC1,0.17951024783533542
-27599,ATP6V0D1,0.35608961452967225
-27600,AC009061.2,-0.04717296036634246
-27603,RIPOR1,0.20071243562298866
-27609,CTCF,-0.018041735091442523
-27611,CARMIL2,-0.09465388055707603
-27612,ACD,-0.030394305141260583
-27613,PARD6A,-0.08726761569815772
-27615,C16orf86,0.04827805908038292
-27616,GFOD2,-0.026545587059352737
-27617,RANBP10,0.08158684968669534
-27620,CENPT,-0.023295358900151233
-27621,THAP11,-0.15596049389299022
-27622,NUTF2,-0.03453320002397389
-27623,EDC4,0.0027152274519555005
-27625,PSKH1,-0.028043878970041642
-27626,CTRL,5.28302859665958e-05
-27627,PSMB10,0.10418263289937499
-27629,SLC12A4,0.05808134973993063
-27631,DPEP2,0.14859973602357918
-27632,DUS2,0.09445086863391289
-27634,DDX28,0.01876238507125474
-27635,NFATC3,-0.10186018256966206
-27642,PLA2G15,0.10808571427146858
-27644,SLC7A6,-0.0799618955465541
-27645,SLC7A6OS,-0.07336063487146663
-27646,PRMT7,-0.08635077774290385
-27648,SMPD3,-0.005608997786833638
-27651,ZFP90,-0.06479728537312868
-27658,TANGO6,0.07422607316485175
-27664,UTP4,-0.0017049088669329666
-27665,SNTB2,-0.10156638807728792
-27667,VPS4A,-0.055205433679790444
-27668,COG8,0.0007161694061137913
-27670,NIP7,-0.07786047559178445
-27672,TERF2,0.00599771931680675
-27673,CYB5B,-0.02993058685750725
-27675,NFAT5,0.02710128005161709
-27681,NOB1,-0.08582764132903502
-27682,WWP2,0.11280547706861017
-27684,PDPR,0.03879271903263291
-27685,AC009060.1,-0.1104145094860375
-27687,EXOSC6,-0.08662455123901514
-27688,AARS,-0.04018584422731528
-27689,DDX19B,-0.0186041366414901
-27691,DDX19A,-0.0268894791331326
-27693,ST3GAL2,0.06570403208780204
-27695,FCSK,0.00507536476798478
-27696,COG4,-0.04903471100717209
-27697,SF3B3,0.0034562194166762503
-27700,VAC14,0.06708339316327185
-27707,CMTR2,0.008317219036082856
-27712,ZNF23,-0.11076355055777089
-27722,PHLPP2,-0.028691963377915696
-27726,AP1G1,0.08276843403752461
-27727,ATXN1L,0.006268609869103615
-27728,IST1,0.060995959096272895
-27729,ZNF821,-0.14374327282954014
-27732,DHODH,-0.026144367823301814
-27733,TXNL4B,0.11348555708692773
-27734,HP,0.13843401264689356
-27737,DHX38,-0.030186106170892676
-27744,ZFHX3,0.2913000746318796
-27758,PSMD7,0.08512860433189232
-27759,AC009120.2,0.01280472808703465
-27765,GLG1,0.021179547788733828
-27766,RFWD3,-0.009219027135465623
-27767,MLKL,0.2063217951669762
-27769,WDR59,-0.010410691779688508
-27770,ZNRF1,0.08877153786799732
-27774,ZFP1,-0.06337094915417402
-27778,CFDP1,-0.05952601066038639
-27782,TMEM170A,-0.07645646170901389
-27790,GABARAPL2,0.01334160273037219
-27792,ADAT1,0.06279907432602193
-27793,KARS,0.21442770000166078
-27794,TERF2IP,-0.16638326594873434
-27806,MON1B,0.08187538821851838
-27807,SYCE1L,-0.04881412482920661
-27813,NUDT7,1.446389596093469e-05
-27819,WWOX,-0.0027837029688729197
-27826,AC027279.1,-0.013449606185649837
-27835,MAF,-0.1742236507126264
-27853,CMC2,-0.0017348627754152546
-27857,ATMIN,0.06717703012624969
-27862,GCSH,-0.15088300837056365
-27868,CMIP,0.2800653761608766
-27870,PLCG2,0.058169658056884725
-27875,MPHOSPH6,-0.03992547082164812
-27887,HSBP1,0.3112332915098072
-27888,MLYCD,0.05144981062058001
-27894,MBTPS1,0.02589693005700959
-27897,HSDL1,0.028152621474498554
-27899,TAF1C,0.08851982996067409
-27909,MEAK7,0.036088235227908445
-27911,COTL1,0.6449018824252062
-27913,KLHL36,-0.08102998110360343
-27914,USP10,0.10751093680751014
-27916,CRISPLD2,0.27383946296931483
-27920,ZDHHC7,0.32123683281482407
-27921,KIAA0513,0.2872865286468898
-27925,GSE1,0.02186259269769299
-27933,C16orf74,0.019209687629395858
-27937,EMC8,0.07501546938301266
-27940,COX4I1,0.0071771550562804275
-27942,IRF8,0.15061802817365752
-27959,MTHFSD,-0.06804434135297711
-27979,FBXO31,-0.012152795478609186
-27981,MAP1LC3B,-0.012948659211568722
-27983,ZCCHC14,-0.005324309352818687
-27990,KLHDC4,-0.21451715996239595
-27994,SLC7A5,0.0006055999998700876
-27997,BANP,0.11105480858885317
-28007,ZFPM1,-0.10797333817453393
-28009,ZC3H18,-0.060364288399185374
-28011,CYBA,0.023491546699130988
-28012,MVD,-0.13492899138679967
-28014,SNAI3,-0.01924073819628172
-28015,RNF166,0.058413955014904
-28017,CTU2,-0.04268740662906544
-28019,PIEZO1,0.16598446098250957
-28025,APRT,-0.21566579017010848
-28026,GALNS,0.08532735777081674
-28027,TRAPPC2L,-0.11196700966505735
-28029,CBFA2T3,0.23282353781036982
-28035,ACSF3,0.028848461363345092
-28042,ZNF778,0.01696676361630112
-28044,ANKRD11,0.02420377281837444
-28050,SPG7,0.0043348475672417
-28052,RPL13,-0.379650387620572
-28055,CHMP1A,0.020777399741650007
-28056,SPATA33,-0.008081037786290513
-28058,CDK10,-0.04707379475204761
-28060,SPATA2L,-0.029525893265202458
-28061,VPS9D1,0.08709697999296247
-28063,ZNF276,-0.19088732636575872
-28064,FANCA,-0.005724797873586295
-28066,TCF25,-0.08381047297551046
-28070,DEF8,0.13238884375919288
-28071,CENPBD1,0.0977061715128153
-28076,FAM157C,0.08063876008690675
-28092,RFLNB,0.13324462098390583
-28094,VPS53,0.14941663958899715
-28098,GEMIN4,0.04176298178423316
-28099,GLOD4,-0.12266838241455508
-28101,MRM3,-0.034495930160341
-28107,TIMM22,-0.006881402634772371
-28108,ABR,0.09375525980493098
-28113,YWHAE,0.18617735305714717
-28114,CRK,0.20390568321977698
-28116,MYO1C,0.0016332315669936694
-28117,INPP5K,0.044846720277879025
-28118,PITPNA-AS1,-0.1578224638176211
-28119,PITPNA,0.18423491415785273
-28120,SLC43A2,0.38707774370419745
-28121,SCARF1,0.14720349509934988
-28122,RILP,0.16825120949612934
-28123,PRPF8,0.0433063331666016
-28126,MIR22HG,0.1638009632950121
-28127,WDR81,0.03090501831153099
-28131,SMYD4,-0.03993711424674722
-28132,RPA1,0.0013593009489968513
-28143,SMG6,-0.07781474373847247
-28148,SRR,0.007407728364680493
-28149,TSR1,0.01291367019042185
-28150,SGSM2,-0.010055662454699603
-28153,MNT,0.17054740271948746
-28155,METTL16,-0.006014805949925352
-28156,PAFAH1B1,-0.042842787618229884
-28159,CLUH,0.0191904333475791
-28163,RAP1GAP2,-0.1256216633228646
-28184,CTNS,0.0965024806994986
-28186,TAX1BP3,0.12456706936641156
-28187,EMC6,-0.08992146619235009
-28188,P2RX5,-0.024618116156905723
-28189,ITGAE,-0.040537534953785474
-28193,NCBP3,0.05664054685313245
-28195,P2RX1,0.13341116020123348
-28196,ATP2A3,-0.20975424401827333
-28198,ZZEF1,0.029039505512369772
-28199,CYB5D2,0.01586206092318869
-28200,ANKFY1,0.1838075882088267
-28203,UBE2G1,-0.06893954015838775
-28204,SPNS3,-0.10773119842430319
-28208,MYBBP1A,-0.021354061587053445
-28214,PELP1,-0.12225916567239442
-28217,ARRB2,0.38737697487226996
-28218,MED11,-0.07376160561576227
-28220,CXCL16,0.258948184569968
-28225,PSMB6,-0.02302179167658578
-28227,PLD2,0.08129703672163371
-28228,MINK1,-0.011213201280503618
-28232,SLC25A11,-0.035040654023612706
-28233,RNF167,-0.140693572731259
-28234,PFN1,-0.2947688120848683
-28236,SPAG7,-0.07928591359249981
-28237,CAMTA2,0.012374506767292317
-28243,KIF1C,0.07273163778674276
-28249,AC012146.1,-0.18872164721371237
-28251,ZNF594,-0.06701187638738429
-28252,AC087500.1,-0.005575167327440764
-28253,SCIMP,0.4405069174601143
-28255,RABEP1,-0.09372666883099996
-28256,NUP88,0.004925831896168433
-28258,RPAIN,-0.0054968094333777895
-28260,C1QBP,-0.22710586293236343
-28261,DHX33,0.043699926311250685
-28263,DERL2,-0.12931010289137648
-28264,MIS12,0.016306321366610924
-28265,NLRP1,0.19311562367076285
-28275,KIAA0753,0.0028302377263369673
-28277,TXNDC17,0.11514527494045172
-28278,MED31,-0.09915419888469301
-28282,XAF1,0.3680480425615637
-28285,ALOX12-AS1,-0.06613809201648807
-28287,AC040977.1,-0.06239384376406633
-28289,RNASEK,0.24310625207330328
-28290,C17orf49,-0.10098653198517729
-28295,CLEC10A,0.15564897663500313
-28296,ASGR2,0.2406599885272594
-28297,ASGR1,0.41642031921525563
-28298,DLG4,0.12393148436593161
-28299,ACADVL,0.10742288354023972
-28300,DVL2,-0.020895053449600094
-28301,PHF23,0.15604419295256722
-28302,GABARAP,0.5949705412899813
-28303,CTDNEP1,0.18198820016815964
-28304,ELP5,-0.0667950767813744
-28305,CLDN7,0.006674807577290234
-28309,EIF5A,-0.016140916743537476
-28310,GPS2,-0.09139642562706769
-28311,NEURL4,0.036573021144471626
-28312,ACAP1,-0.37340915013361553
-28313,KCTD11,0.021579609895219624
-28317,PLSCR3,-0.08483222376401998
-28318,TMEM256,-0.0695769756093746
-28323,TMEM102,-0.0139491245132376
-28327,CHRNB1,-0.05050400931200543
-28328,ZBTB4,-0.06264024763151671
-28330,POLR2A,0.06913383536121505
-28331,TNFSF12,0.013531501591389078
-28333,TNFSF13,0.44559302737974704
-28334,SENP3,0.08430011272965446
-28335,EIF4A1,0.4470983969993135
-28336,CD68,0.5790302135530349
-28337,AC016876.1,-0.008948892484521248
-28338,MPDU1,0.0794988982009983
-28340,FXR2,0.06846923409147897
-28342,SAT2,0.28456779901528123
-28344,TP53,0.07733216921328805
-28345,WRAP53,-0.09482202468867755
-28348,KDM6B,0.24917118261208396
-28350,NAA38,0.02627965580504318
-28352,CHD3,-0.1301629122914781
-28355,TRAPPC1,-0.01686528648806172
-28356,CNTROB,0.06853565398559715
-28364,PER1,0.20766673845689063
-28365,VAMP2,-0.31113326871976626
-28366,TMEM107,0.03227391230891459
-28368,BORCS6,-0.05486143225744201
-28370,LINC00324,0.016827725335529193
-28371,CTC1,-0.08490985801800388
-28372,PFAS,-0.060089228239925775
-28374,SLC25A35,0.03409234690711927
-28375,RANGRF,-0.14532802028454128
-28380,KRBA2,-0.03313176634550867
-28382,RPL26,-0.38003525680900463
-28384,NDEL1,0.031613051391641354
-28389,PIK3R6,0.06932877739285964
-28390,PIK3R5,-0.018870523262968127
-28396,STX8,-0.12675286128928107
-28412,GAS7,0.40247546496403397
-28424,SCO1,-0.0009508317948412415
-28426,ADPRM,-0.08257222061775367
-28427,TMEM220,-0.02928065270426795
-28441,MAP2K4,0.11446259103014687
-28452,ELAC2,0.00277145586841732
-28456,COX10-AS1,-0.07573539129063928
-28460,COX10,-0.009221943482897118
-28463,HS3ST3B1,-0.06916910986523961
-28465,AC005224.2,-0.17298461337090912
-28482,TVP23C,-0.11235458840335397
-28485,TRIM16,-0.03998316343770373
-28487,ZNF286A,-0.029577203919171055
-28495,ZSWIM7,0.01130851076829975
-28496,TTC19,-0.11173631914177516
-28498,NCOR1,-0.02514750035659653
-28500,PIGL,-0.03072423012764634
-28502,UBB,-0.42422002782385376
-28505,TRPV2,-0.02995777427574715
-28506,SNHG29,-0.19222336184750696
-28508,LRRC75A,-0.023975918696104822
-28510,ZNF287,-0.06764896503345559
-28512,ZNF624,-0.024484856906459903
-28514,CCDC144A,0.05570348049317722
-28519,TNFRSF13B,-0.0028541480269414785
-28525,MPRIP,-0.13452143701671232
-28529,FLCN,0.04735050634645326
-28531,COPS3,0.011224443400469996
-28533,MED9,-0.03760067373329203
-28536,PEMT,-0.09329323604036718
-28540,RAI1,0.045284336972167494
-28542,SMCR5,-0.010755995325956443
-28544,SREBF1,0.0202480705896021
-28545,TOM1L2,-0.06747830667192799
-28548,ATPAF2,0.07267651041171497
-28549,GID4,0.039593339561058406
-28550,DRG2,-0.030477088797785108
-28553,ALKBH5,0.08281131362713746
-28555,FLII,0.23182961038063354
-28556,MIEF2,-0.048445933970426736
-28558,TOP3A,0.17926573233051885
-28559,SMCR8,0.1513360995754915
-28560,SHMT1,-0.09242517055797991
-28565,FAM106A,0.08714710206275082
-28570,TVP23B,0.0012647651707896935
-28572,PRPSAP2,0.02145767376677765
-28577,GRAP,-0.06205793801654866
-28581,AC007952.4,0.32276576768009546
-28594,MAPK7,0.019094156371118724
-28604,ALDH3A2,0.17589087158794522
-28610,ULK2,0.1276670299922992
-28613,AKAP10,0.18466992120414227
-28616,SPECC1,0.24287519938053356
-28631,USP22,0.11358745169646543
-28634,DHRS7B,0.07498031012593215
-28635,TMEM11,0.012652023806330617
-28637,NATD1,0.09797905691483336
-28638,MAP2K3,0.2337812490795383
-28658,WSB1,0.4059887985898241
-28660,KSR1,0.213649306363172
-28663,LGALS9,0.40303764616356036
-28666,LYRM9,-0.07416279665961634
-28669,NLK,0.1106233526309159
-28677,IFT20,0.012342969164119347
-28678,TNFAIP1,0.03614466738363178
-28679,POLDIP2,0.05631395264979939
-28680,TMEM199,0.00464677277433587
-28683,SARM1,-0.11737113217696696
-28688,AC005726.1,0.05902538163794158
-28691,UNC119,0.17414954922857964
-28692,PIGS,0.0015386354720492012
-28698,AC005726.5,-0.01380322901504367
-28699,RSKR,0.07389182625983647
-28700,KIAA0100,0.15513938690447646
-28702,SDF2,-0.11242658343810218
-28703,SUPT6H,0.015557589681525015
-28705,PROCA1,-0.014776262701025295
-28706,RAB34,0.2932789007436222
-28707,RPL23A,-0.49248144919570147
-28710,NEK8,0.0012902981971409926
-28713,TRAF4,-0.053817498057877014
-28716,FAM222B,-0.08891620090918112
-28717,ERAL1,-0.03133427409144012
-28719,FLOT2,0.07671966579120881
-28722,DHRS13,-0.08132049610188534
-28723,PHF12,0.07583301043736344
-28730,MYO18A,0.03137754367060607
-28734,NUFIP2,0.11558972451360143
-28737,TAOK1,0.22839894629533103
-28738,ABHD15,0.001414217574478183
-28740,TP53I13,-0.13801881150939188
-28742,GIT1,0.026978257441226033
-28747,SSH2,0.20373647996579447
-28755,NSRP1,-0.0034463280185889743
-28757,AC104984.3,-0.014819621864399592
-28762,BLMH,0.055559009738255415
-28764,CPD,0.20579651860282044
-28765,GOSR1,-0.04494782780708301
-28770,AC005562.1,-0.044928070251883785
-28773,CRLF3,0.05258355800315717
-28777,ATAD5,-0.10199869320360562
-28780,TEFM,-0.03637988848986805
-28782,ADAP2,0.35979941654072356
-28786,RNF135,0.2506276472280922
-28790,NF1,0.0038404070207577966
-28793,EVI2B,0.4739086791307863
-28794,EVI2A,0.37306710781051244
-28795,RAB11FIP4,0.05178251363929947
-28801,COPRS,0.09290791305340897
-28802,UTP6,0.007217682943783794
-28804,SUZ12,0.04077170860544699
-28805,LRRC37B,0.07390331668671321
-28806,AC116407.2,-0.1363278980021331
-28808,RHOT1,0.20118810183286528
-28812,C17orf75,-0.017080348047639877
-28814,ZNF207,0.2203669235559745
-28820,PSMD11,0.018046863031604295
-28864,ZNF830,-0.09417999244107103
-28865,LIG3,0.0015254059725900693
-28866,RFFL,0.012596035252175944
-28869,RAD51D,0.023851667265106142
-28875,AC022916.1,-0.0722502753894175
-28876,SLFN5,-0.31455556286199626
-28877,AC022706.1,-0.17682813544226508
-28878,SLFN11,0.27695012874786284
-28881,SLFN12,0.2321317128414462
-28882,SLFN13,-0.2008967704546021
-28883,SLFN12L,-0.28076420899655086
-28890,SNHG30,-0.11196372821542364
-28892,AP2B1,-0.07604062457293632
-28893,TAF15,-0.014266734208516471
-28904,CCL5,-0.3895054495446396
-28916,CCL3,-0.16182435770748868
-28917,CCL4,-0.5487030308056304
-28928,ZNHIT3,-0.11693509852713946
-28929,MYO19,0.007148429036600317
-28930,PIGW,0.024338365454496583
-28931,GGNBP2,0.05582639342407516
-28940,AATF,0.00907704967144788
-28945,ACACA,-0.049887455577310426
-28948,TADA2A,0.017686256069708448
-28951,SYNRG,-0.27941768421319085
-28953,DDX52,0.04986109044801632
-28964,MRPL45,0.02696595548062086
-28966,SOCS7,0.008768393400151274
-28972,AC006449.6,-0.018371821564133247
-28974,MLLT6,-0.3188676176031505
-28976,CISD3,-0.16480613581560058
-28979,PSMB3,0.17623410958000654
-28980,PIP4K2B,0.005366200019161357
-28981,CWC25,0.02260442228644678
-28983,RPL23,0.09583059075172443
-28984,LASP1,0.16106545361663413
-28987,LINC00672,0.026597463013333357
-28996,RPL19,-0.39055179747218893
-28998,FBXL20,0.10348604054954594
-29000,MED1,0.054949687918370266
-29001,CDK12,-0.003962624656591785
-29006,STARD3,0.024130908512555047
-29011,MIEN1,-0.1657557499026066
-29013,IKZF3,-0.39011303787819784
-29016,ORMDL3,-0.10632291120371194
-29020,PSMD3,-0.021086102056174824
-29023,MED24,0.0052394227462900945
-29024,THRA,-0.16504239745115123
-29025,NR1D1,-0.024918336190212872
-29026,MSL1,0.050169279941743
-29027,CASC3,0.02772298457432426
-29029,WIPF2,0.15995911586461908
-29031,RARA,0.22634670518715747
-29033,RARA-AS1,0.17220307140567478
-29040,CCR7,-0.15267723022230426
-29041,SMARCE1,-0.09260763056045418
-29050,KRT10,-0.28427158458538965
-29051,TMEM99,-0.11962519371762298
-29113,EIF1,0.10734095833992004
-29116,JUP,0.20840079795017466
-29119,NT5C3B,-0.020588994759147713
-29122,KLHL11,-0.01212673112785834
-29123,ACLY,0.09649934987968516
-29126,CNP,0.09684915797031313
-29127,DNAJC7,0.13862856002547086
-29128,NKIRAS2,0.14880901859003748
-29131,DHX58,0.0803334825804443
-29132,KAT2A,-0.011998765942081418
-29134,RAB5C,0.26650335455999247
-29138,GHDC,0.029032924885914538
-29139,STAT5B,0.006809950416130562
-29143,STAT5A,0.2501080892025065
-29144,STAT3,0.11634399501733392
-29146,ATP6V0A1,0.32458195217481944
-29148,AC067852.3,0.14335224447985725
-29149,NAGLU,0.006028640917903033
-29151,AC067852.2,0.012133568799447292
-29152,COASY,0.04229502080269777
-29153,MLX,0.049113975311345044
-29156,RETREG3,-0.027336473518499858
-29157,TUBG1,-0.018741683433546735
-29164,EZH1,0.02506239419314696
-29168,VPS25,0.0049781638342720255
-29170,COA3,0.0662400314371293
-29172,BECN1,0.14972090406480937
-29173,PSME3,0.13122113767838278
-29178,AARSD1,-0.0017298636968969415
-29180,RUNDC1,0.10055045283883754
-29181,RPL27,-0.32574072036610596
-29182,IFI35,0.2230454414710495
-29183,VAT1,0.0448870643714995
-29185,BRCA1,0.04801308515343091
-29186,NBR2,0.049378659662991244
-29187,AC060780.1,-0.07018454822152122
-29188,NBR1,0.14922578120822516
-29189,TMEM106A,0.1799924790711993
-29194,DHX8,0.09341671235084419
-29199,DUSP3,0.3108952743309675
-29212,TMEM101,0.07111605891491253
-29213,LSM12,0.008452548924795628
-29214,G6PC3,0.04034344120674666
-29215,HDAC5,0.07325138136423937
-29219,ASB16-AS1,-0.06550179268355967
-29220,TMUB2,0.09563920696475384
-29221,ATXN7L3,0.039713064480927394
-29223,UBTF,-0.19075571274826486
-29224,AC003102.1,-0.012412798404537058
-29229,SLC25A39,-0.030085628629322556
-29231,GRN,0.5837404658907327
-29234,GPATCH8,-0.039968240052099695
-29235,AC103703.1,0.0451019817567721
-29236,FZD2,0.20870529977489521
-29239,CCDC43,0.04185193976653619
-29247,EFTUD2,0.12387665434878256
-29255,DCAKD,0.0312047384499698
-29256,NMT1,-0.003992903822258022
-29258,ACBD4,0.029746500030353774
-29259,AC142472.1,0.031244591989815485
-29260,HEXIM1,-0.10956629682594235
-29261,AC138150.1,-0.02557970443252392
-29264,AC008105.3,0.031579402805491354
-29265,FMNL1,0.23252986290746452
-29267,MAP3K14-AS1,-0.08295774912799425
-29269,MAP3K14,0.04093735133848416
-29271,ARHGAP27,0.2443953383573664
-29273,PLEKHM1,0.17319783825585824
-29287,KANSL1,-0.025001203211738657
-29289,KANSL1-AS1,-0.1269477873067899
-29290,ARL17B,-0.0014278937846124533
-29292,LRRC37A2,0.022182968027297106
-29293,ARL17A,0.021829322953823994
-29294,FAM215B,-0.06741013852821934
-29295,NSF,0.10774082708133836
-29300,GOSR2,0.032225411575979857
-29302,AC005670.3,-0.06967643241651947
-29303,CDC27,-0.002290646796638563
-29304,AC002558.3,0.025122303358124657
-29310,EFCAB13,0.010021656080757422
-29312,NPEPPS,0.016145212711839482
-29314,KPNB1,0.18924319513325505
-29317,TBX21,-0.4925665285968005
-29318,OSBPL7,-0.11903381425263998
-29319,MRPL10,-0.11333716103572195
-29321,SCRN2,-0.0006931123414162293
-29325,AC018521.5,-0.08556261433372324
-29326,SP2,-0.0001473832028796394
-29329,PNPO,0.10075820282741475
-29332,CDK5RAP3,0.13108575846385864
-29336,NFE2L1,0.0937956749037937
-29339,CBX1,0.05293088641588362
-29340,SNX11,0.09885808535644625
-29341,SKAP1,-0.48390850802955565
-29347,HOXB2,-0.18720078744765206
-29348,HOXB-AS1,-0.10506377725484611
-29352,HOXB4,-0.08897334334974168
-29367,CALCOCO2,0.11127291319929863
-29368,ATP5MC1,-0.00969278825695676
-29369,UBE2Z,0.16206722769090076
-29370,SNF8,0.1153437111991633
-29377,GNGT2,-0.05169549034599504
-29378,ABI3,-0.11380278686636996
-29381,ZNF652,-0.05664566888262394
-29385,PHB,-0.08033862355884633
-29392,SPOP,0.04828729240777117
-29393,SLC35B1,0.006045221730528744
-29395,FAM117A,-0.13110157857190158
-29396,KAT7,0.0293299692675375
-29408,PDK2,0.024091548185708217
-29412,PPP1R9B,0.22280643439933445
-29422,XYLT2,0.008935033792208148
-29423,MRPL27,0.029939199919014923
-29425,LRRC59,0.09408812283603933
-29427,ACSF2,0.09896291366315299
-29430,RSAD1,0.015740560322200225
-29435,SPATA20,0.050586881955237986
-29440,ABCC3,0.09169062777399316
-29441,ANKRD40,0.02892526256506855
-29442,AC005921.2,-0.10008682963919759
-29443,LUC7L3,-0.0716964319305759
-29448,TOB1,-0.014536213079538706
-29449,TOB1-AS1,-0.07594146225187459
-29453,SPAG9,0.15539328316794437
-29455,NME1,-0.05463363918105303
-29456,NME2,-4.7405416270140093e-05
-29457,MBTD1,0.0927008343994553
-29458,AC006141.1,-0.02131946904889507
-29459,UTP18,-0.014102798095837064
-29472,COX11,-0.12683034445594465
-29478,MMD,0.03439800855208993
-29481,PCTP,0.13664854901642978
-29484,NOG,-0.0669469861351763
-29485,C17orf67,-0.10332284248647747
-29487,DGKE,-0.030289765852386
-29488,TRIM25,0.20724332434290296
-29490,AC015912.3,0.1326973072728416
-29491,COIL,-0.0921795312291121
-29493,SCPEP1,0.44731185313663135
-29496,AKAP1,0.03521421288064543
-29500,MSI2,-0.1153250030667342
-29505,MRPS23,-0.08969170207954606
-29506,CUEDC1,0.14812831887555175
-29509,VEZF1,0.047519919604447784
-29510,AC015813.1,0.06204054467759284
-29511,SRSF1,0.07103805141789205
-29514,DYNLL2,-0.011799837376640797
-29519,MKS1,0.006260155003981651
-29524,TSPOAP1,-0.17008014723642845
-29525,TSPOAP1-AS1,-0.12864023438303224
-29526,AC004687.1,-0.19981279699859744
-29527,SUPT4H1,0.0894578274442009
-29530,MTMR4,-0.01769266358991587
-29535,RAD51C,-0.06749772902435576
-29539,TRIM37,0.05828650362486212
-29541,SKA2,-0.17174988430276175
-29542,PRR11,0.15183056480676296
-29544,SMG8,0.05648838696782295
-29545,GDPD1,-0.0706473462904482
-29546,YPEL2,0.18545617297971886
-29549,DHX40,0.16107922345236572
-29550,AC091271.1,0.06873018733457116
-29551,CLTC,0.2752454376901087
-29552,PTRH2,0.020363180913386566
-29553,VMP1,0.5114263765086708
-29554,AC040904.1,0.10294153650232675
-29555,TUBD1,-0.01660092455645779
-29556,RPS6KB1,0.03919538237146148
-29557,RNFT1,0.07562766331568159
-29560,HEATR6,0.007815110143440945
-29566,USP32,0.2349002259811325
-29568,APPBP2,0.11781936397065229
-29571,PPM1D,0.16883007258198873
-29572,BCAS3,-0.08594722882669627
-29584,NACA2,0.015602036751957905
-29586,INTS2,0.031549698718770114
-29587,MED13,0.12019785892847676
-29590,METTL2A,-0.07939884479598833
-29591,TLK2,0.04123626036755261
-29601,TANC2,-0.03912804849520525
-29608,CYB561,-0.04656549645429482
-29612,DCAF7,0.19614974549212655
-29613,TACO1,-0.056212000730661765
-29614,MAP3K3,0.23864651608239743
-29615,STRADA,0.050251818111827745
-29616,LIMD2,-0.08787897325349546
-29617,CCDC47,0.008611560535457134
-29618,DDX42,0.045151835717452384
-29619,FTSJ3,0.04409805608424497
-29620,PSMC5,-0.136845019997253
-29621,SMARCD2,0.19742590006208635
-29627,CD79B,0.0017018593067092487
-29632,ICAM2,-0.14536647692131163
-29633,ERN1,-0.11537046818711166
-29635,SNHG25,-0.03846092487670565
-29636,TEX2,0.03170291074773936
-29637,PECAM1,0.41531274612965763
-29638,MILR1,0.21435991964329368
-29639,POLG2,-0.04874965942036321
-29640,DDX5,-0.24079844935966077
-29641,CEP95,-0.11880647369232228
-29642,SMURF2,-0.028795556357833763
-29647,GNA13,0.1396747056441267
-29651,AXIN2,-0.06601406445964515
-29654,PRKCA,-0.10523483020396723
-29662,HELZ,0.146195339343326
-29666,PSMD12,-0.004437577318845547
-29667,PITPNC1,-0.18662602057170244
-29670,NOL11,-0.10007733638399913
-29671,BPTF,0.08782017259619096
-29673,C17orf58,-0.022288042310685856
-29674,KPNA2,0.02213598529886254
-29676,AC005332.6,-0.028375954291767837
-29681,AC005332.4,-0.15169377757742122
-29683,AMZ2,-0.0048767579979976505
-29684,ARSG,-0.028401816032840057
-29685,SLC16A6,0.15637123717423865
-29687,WIPI1,0.004260808710499494
-29688,PRKAR1A,0.3505707216704414
-29699,ABCA5,-0.12129523573154513
-29700,MAP2K6,-0.0038112246090537547
-29708,KCNJ2,0.22002123209954658
-29724,SLC39A11,0.15149825292205019
-29726,SSTR2,0.04855863341955264
-29727,COG1,-0.0008353542345324099
-29729,FAM104A,-0.016627078854906545
-29730,C17orf80,-0.052588618334346915
-29733,CDC42EP4,0.10358451060347354
-29746,RPL38,-0.30333048193604956
-29747,AC100786.1,0.09842062207681992
-29748,TTYH2,0.14770751167019722
-29756,CD300A,-0.004785879087058221
-29757,CD300LB,0.23581279316351667
-29758,CD300C,0.25226038052719113
-29761,AC064805.1,0.13560148274539321
-29762,CD300E,0.43023627659117575
-29764,RAB37,-0.16642638435808282
-29765,CD300LF,0.36049273728669695
-29767,SLC9A3R1,-0.3557144782136761
-29768,NAT9,0.0036490754762026987
-29769,TMEM104,-0.054050309706362584
-29779,MRPL58,-0.0808468383083359
-29780,KCTD2,0.006664594982584208
-29781,ATP5PD,0.010869051297057742
-29782,SLC16A5,0.14075783702385603
-29783,ARMC7,-0.0010378349753500248
-29784,NT5C,-0.18697405840370313
-29785,JPT1,-0.03784532133451042
-29788,SUMO2,-0.15692782405257868
-29789,NUP85,0.06489860530985478
-29790,GGA3,0.16489944185366393
-29791,MRPS7,-0.04713092394664405
-29792,MIF4GD,0.05628106425518815
-29794,SLC25A19,0.03349017938367286
-29795,GRB2,0.4002827709079115
-29797,AC011933.2,0.11668389011685044
-29799,TMEM94,-0.0032343282518143357
-29801,TSEN54,-0.15158463896006702
-29802,LLGL2,-0.3486481574110104
-29803,MYO15B,0.14893497250007648
-29804,RECQL5,0.056401443019760425
-29807,SAP30BP,-0.02664163335131818
-29811,GALK1,0.1266144143670613
-29812,H3F3B,0.19937003779855997
-29813,UNK,-0.08763567724241951
-29815,UNC13D,0.08939661231970186
-29816,WBP2,0.06088880047252751
-29819,TRIM65,-0.025357485440235042
-29821,MRPL38,-0.09615108679489845
-29823,ACOX1,0.2184041075596728
-29825,TEN1,-0.009646143070720361
-29828,SRP68,0.005385315663332447
-29831,EXOC7,0.029742726492036648
-29834,RNF157,-0.12834032273056187
-29835,UBALD2,-0.03811481028564505
-29837,PRPSAP1,-0.09118447332739436
-29838,SPHK1,0.09752823771068861
-29839,UBE2O,0.024104554305255002
-29841,RHBDF2,0.12011814542415772
-29847,SNHG16,-0.006046510789781613
-29848,AC015802.6,-0.029081595180918626
-29854,MXRA7,-0.14288645643453898
-29856,JMJD6,-0.00795379701355044
-29857,METTL23,-0.2200532817331246
-29858,SRSF2,0.0024327615319899346
-29859,MFSD11,-0.021392961473202673
-29866,SNHG20,0.007023831273817482
-29867,SEC14L1,0.23227915356336298
-29870,SEPTIN9,0.05863121086615484
-29882,TNRC6C,-0.17829438132667214
-29883,TNRC6C-AS1,-0.08545095677424935
-29884,TMC6,-0.20867855675571267
-29885,TMC8,-0.14697550855657743
-29887,SYNGR2,0.2650575707929611
-29889,AFMID,0.06723866009190946
-29895,SOCS3,0.11749761249584534
-29897,PGS1,0.07612735389456857
-29901,CYTH1,-0.06287028213029033
-29902,USP36,-0.07936481554816806
-29904,TIMP2,0.46090492740499367
-29908,LGALS3BP,0.0474185508239192
-29909,CANT1,0.09909253623141387
-29912,ENGASE,0.04869766223679429
-29919,CBX8,0.022329353920541938
-29922,CBX4,0.005408389243087992
-29931,GAA,0.32940284695212296
-29932,EIF4A3,-0.02560844865092839
-29935,AC087741.1,0.08706623070858817
-29936,SGSH,0.06367368302140418
-29937,SLC26A11,-0.08758729744188977
-29938,RNF213,0.28243262948808556
-29939,AC124319.1,0.03278657183234808
-29941,RNF213-AS1,-0.09028819098938398
-29942,ENDOV,0.027604223676588085
-29945,RPTOR,-0.05052774517704679
-29951,CHMP6,-0.05831916487672244
-29955,BAIAP2-DT,0.14174860290479036
-29956,BAIAP2,0.09342812279993751
-29960,CEP131,-0.03161072701655159
-29962,TEPSIN,0.04706537888440792
-29964,NDUFAF8,-0.1766703631222184
-29965,SLC38A10,0.192800627932345
-29969,AC027601.6,0.08514656172102691
-29977,ACTG1,-0.2601270089849833
-29980,FAAP100,0.07299634477060343
-29981,NPLOC4,0.07279022377363227
-29984,OXLD1,-0.07677931566313649
-29985,CCDC137,-0.09036417192404508
-29986,ARL16,-0.01723511098562315
-29987,HGS,0.11045259321393248
-29989,AC139530.1,-0.1116572210444537
-29990,MRPL12,0.024725165962076502
-29993,MCRIP1,-0.15015252331154813
-29995,P4HB,0.06870424544842003
-29998,ARHGDIA,0.16443992121814863
-29999,AC145207.5,-0.02528120894777612
-30001,ALYREF,0.0784392933237438
-30002,ANAPC11,-0.03529150549085427
-30004,PCYT2,-0.08922436457233603
-30005,SIRT7,0.003951690052154913
-30006,MAFG,0.20582185368643005
-30015,ASPSCR1,0.10298201041792791
-30016,CENPX,0.00953071577935663
-30017,LRRC45,0.023355602250866997
-30019,DCXR,-0.10778257778879984
-30021,RFNG,0.07403469367904214
-30022,GPS1,0.000986237171079841
-30023,DUS1L,-0.06038099341038449
-30024,FASN,0.011021622758800436
-30025,CCDC57,-0.10923150327605707
-30029,SLC16A3,0.3169609626030677
-30030,CSNK1D,0.038723418731561196
-30034,AC132872.1,0.01394614796834652
-30035,CD7,-0.5213912807999432
-30036,SECTM1,0.39040388729537323
-30040,OGFOD3,-0.01703646257672244
-30042,HEXD,-0.023885350413789978
-30044,CYBC1,0.0477823581788127
-30047,NARF,-0.05611928366575018
-30050,FOXK2,0.015538771037811622
-30053,AC124283.1,-0.055990704220251444
-30054,WDR45B,-0.01748602323458011
-30057,FN3KRP,-0.14747573724281293
-30061,TBCD,0.05687862520162272
-30065,B3GNTL1,-0.03130920546029623
-30067,METRNL,0.0055678760809782565
-30076,USP14,-0.0584164289744974
-30077,THOC1,-0.06212778733634682
-30089,ENOSF1,-0.05828419931186647
-30091,YES1,-0.2358047277868204
-30108,METTL4,-0.09174668130247367
-30111,SMCHD1,0.25296313343789434
-30113,EMILIN2,0.4637577697720384
-30114,LPIN2,-0.094806889354541
-30120,AP005329.3,-0.03514016414626985
-30121,MYL12A,-0.43069035821611684
-30123,MYL12B,-0.35404343292664675
-30127,TGIF1,0.010616111795568274
-30130,DLGAP1-AS1,-0.12111927827898333
-30144,LINC00667,-0.006157696660024606
-30145,ZBTB14,-0.12360974981348974
-30148,EPB41L3,0.37092315205485255
-30181,RAB12,0.17870328367081253
-30188,NDUFV2,-0.10400834069091713
-30190,NDUFV2-AS1,-0.07844317484316636
-30191,ANKRD12,-0.1490367032628711
-30194,TWSG1,0.0501488110283556
-30197,RALBP1,0.010338880756371214
-30199,PPP4R1,0.07745988744384306
-30203,RAB31,0.510882582097072
-30207,VAPA,0.11637866366779837
-30216,NAPG,-0.03335606997060493
-30235,CHMP1B,0.2373891342431829
-30238,MPPE1,0.04643758341491856
-30241,IMPA2,0.2965362227857429
-30251,AFG3L2,0.06901233607767542
-30255,SPIRE1,-0.030191885683153997
-30256,PSMG2,0.004661040354672355
-30261,PTPN2,0.14598410981563864
-30262,SEH1L,0.055431744205987565
-30264,CEP192,0.042711137178141005
-30267,LDLRAD4,0.0015305604060419722
-30280,FAM210A,-0.05609223497738798
-30281,RNMT,-0.08107452789669242
-30284,ZNF519,-0.030675557385139773
-30302,ROCK1,0.20084631013733178
-30307,ESCO1,-0.09226318497656723
-30308,SNRPD1,-0.149080618104254
-30309,ABHD3,0.07124603869299165
-30312,MIB1,0.024348427025118833
-30326,RBBP8,0.20389853338111597
-30331,RIOK3,0.19047072288602498
-30332,RMC1,-0.015253428273397956
-30333,NPC1,-0.02677283872825151
-30337,TTC39C,-0.18235501216327904
-30343,OSBPL1A,0.1377928165491098
-30346,IMPACT,0.13363279736035055
-30359,SS18,0.04067150275511898
-30361,TAF4B,0.007106588972560252
-30390,DSC2,0.21106027418046666
-30405,TRAPPC8,0.11056303691147773
-30409,RNF125,-0.19960751372906763
-30412,RNF138,0.10669720269647233
-30434,MAPRE2,-0.10461128920062783
-30436,ZNF397,0.01903943523880937
-30437,ZSCAN30,-0.06357331150055422
-30441,ZNF24,0.030587521553278416
-30445,INO80C,-0.00999955815797596
-30446,GALNT1,0.1376961751502604
-30450,C18orf21,0.07069677639947826
-30451,RPRD1A,0.06210485626403813
-30452,SLC39A6,0.058464253617439975
-30453,ELP2,-0.052925590107770225
-30459,TPGS2,-0.04943464774391598
-30460,KIAA1328,-0.08367108057179036
-30478,PIK3C3,0.06323047160602818
-30489,SETBP1,-0.10202501585427232
-30501,EPG5,0.10011133044825735
-30502,PSTPIP2,0.35852933435297585
-30503,ATP5F1A,0.08710627889761507
-30504,HAUS1,-0.025977593506058168
-30505,C18orf25,0.05771903033351364
-30512,PIAS2,-0.02695592773620664
-30520,HDHD2,0.02622277847557584
-30521,IER3IP1,-0.01625272812257469
-30526,SMAD2,0.044391741650018766
-30533,SMAD7,-0.1996031731613397
-30536,DYM,0.06502115223417233
-30541,C18orf32,-0.08468784093008934
-30544,RPL17,-0.43723191775761533
-30548,ACAA2,-0.11834767204457149
-30554,MBD1,-0.0003689330199229311
-30555,CXXC1,-0.0014236415897923844
-30561,ME2,0.18187841289963463
-30562,ELAC1,-0.039285348756322905
-30563,SMAD4,0.045858980473147605
-30564,MEX3C,-0.1642039374844774
-30572,MBD2,0.3149769632273726
-30574,POLI,0.09602021267763398
-30583,TCF4,0.08720219013952728
-30598,TXNL1,-0.02351574883985405
-30599,WDR7,0.05521461814256959
-30610,FECH,0.015745901316836404
-30612,NARS,0.13742078983461972
-30613,AC027097.1,0.012570986668468152
-30614,AC027097.2,0.026249347900171464
-30630,MALT1,0.05514006752937136
-30635,ZNF532,0.05962936465457614
-30637,SEC11C,-0.23983193225075378
-30641,LMAN1,-0.1373674930836892
-30649,PMAIP1,0.08073464146296416
-30663,PIGN,0.06798760994772365
-30664,RELCH,0.1694674457271096
-30669,ZCCHC2,0.07261659922467725
-30671,PHLPP1,0.01566569871205602
-30672,BCL2,-0.20810073392773012
-30675,KDSR,-0.011805649226762558
-30677,VPS4B,0.08687252052658473
-30690,SERPINB8,0.20228876038860363
-30714,TMX3,-0.03539526088523696
-30726,CD226,-0.0994483019642023
-30727,RTTN,-0.05012608721682985
-30752,TIMM21,-0.0010010026294522212
-30754,CYB5A,-0.06958694542726047
-30760,CNDP2,0.23813321095898604
-30764,LINC00909,-0.03398228600931874
-30765,ZNF407,-0.01836971961267781
-30769,ZADH2,0.13266603492506243
-30770,TSHZ1,-0.007890201340364776
-30790,ZNF516,0.280247578251059
-30800,ZNF236,0.009644516461562438
-30802,MBP,-0.23389964621359957
-30822,ATP9B,0.028170237637116676
-30827,NFATC1,0.06967618267863604
-30835,CTDP1,0.0894667863592019
-30841,SLC66A2,0.19379518098955192
-30843,HSBP1L1,-0.10425478095243834
-30844,TXNL4A,-0.07275217784203224
-30846,RBFA,-0.06875323189799067
-30848,ADNP2,0.04313852176112203
-30857,LINC01002,0.09765487977501704
-30860,MIER2,0.11201575733055241
-30870,TPGS1,-0.11391155319922065
-30871,CDC34,0.022008172200241424
-30872,GZMM,-0.7117981140891726
-30874,BSG,0.030431517145494143
-30876,POLRMT,-0.0011340424265016595
-30879,RNF126,-0.18957921357681679
-30881,FSTL3,0.048855885849198796
-30887,PTBP1,0.027454299968303293
-30892,CFD,0.4388240476417986
-30893,MED16,0.09926980930079517
-30894,R3HDM4,0.06876870642969928
-30896,ARID3A,0.2839754311737155
-30899,WDR18,0.04873943497441964
-30902,TMEM259,0.003924894600470161
-30904,CNN2,-0.153092145771622
-30905,ABCA7,0.03578808177027476
-30906,ARHGAP45,0.10412621184400268
-30907,POLR2E,0.045038999579883825
-30908,GPX4,0.14811320181792445
-30909,SBNO2,0.12163792364319431
-30910,STK11,0.17784336540314877
-30913,ATP5F1D,-0.04714941737690732
-30914,MIDN,0.19347678505008556
-30915,CIRBP,-0.2824979325570062
-30917,FAM174C,-0.06781201867183118
-30920,PWWP3A,-0.0505483970266064
-30922,NDUFS7,-0.0255369068517168
-30925,GAMT,-0.19942513730410197
-30926,DAZAP1,0.04765146715713794
-30927,RPS15,-0.4258083815799846
-30931,C19orf25,-0.13738547995822514
-30938,MBD3,0.06582693022566156
-30939,UQCR11,-0.012884278442878545
-30940,TCF3,-0.04021713084760583
-30944,REXO1,-0.03793079515517945
-30948,KLF16,0.048852239290230165
-30950,AC012615.1,-0.08830989552882407
-30952,ABHD17A,-0.42720059624502
-30953,SCAMP4,0.10843719990119619
-30955,CSNK1G2,-0.007751526753989596
-30957,BTBD2,0.06929542803647402
-30958,AC005306.1,-0.031864709789036806
-30959,MKNK2,0.0357253002481179
-30960,MOB3A,0.08020608785302234
-30961,IZUMO4,0.029381324149679314
-30962,AP3D1,0.07456803947628225
-30963,DOT1L,0.09920094885258933
-30965,PLEKHJ1,-0.06921263272215583
-30966,SF3A2,-0.14281965105877642
-30969,OAZ1,0.41709203162338826
-30970,PEAK3,0.10134278406725587
-30972,AC104530.1,0.11584412258295715
-30974,LSM7,-0.1145310342348809
-30975,SPPL2B,-0.042521430189456345
-30977,TIMM13,0.044297797469370896
-30978,LMNB2,-0.010715675168916925
-30980,GADD45B,0.05251130642970659
-30981,GNG7,-0.02057556622030177
-30989,SLC39A3,0.02131233775822227
-30990,SGTA,-0.005147196762112073
-30991,THOP1,-0.10333700394605168
-31003,TLE5,-0.4924447936489909
-31004,GNA11,0.09740600543615255
-31007,GNA15,0.22314863651478373
-31009,S1PR4,-0.19555178236839704
-31010,NCLN,0.07986175613578307
-31013,NFIC,0.2083165988852818
-31016,DOHH,-0.002742685622112337
-31017,FZR1,0.08094455239937301
-31018,MFSD12,0.17756047356647117
-31022,HMG20B,0.014734853724084026
-31026,CACTIN,-0.07528501119051573
-31027,PIP5K1C,-0.001174067540666709
-31030,APBA3,-0.02753770778571712
-31033,MRPL54,-0.071951271286789
-31035,MATK,-0.5266600887298329
-31039,DAPK3,0.05676666630088119
-31040,EEF2,0.17954580891721253
-31041,PIAS4,0.1120661566689739
-31042,ZBTB7A,0.12664244330945587
-31044,MAP2K2,-0.13695057945497432
-31046,SIRT6,-0.007589154444630507
-31049,YJU2,-0.05745396880398525
-31051,TMIGD2,-0.2032493067663352
-31055,MPND,0.0410351674751817
-31058,SH3GL1,0.030267205550108256
-31060,CHAF1A,-0.06389230874486573
-31062,UBXN6,0.1608672435058871
-31066,HDGFL2,-0.06769780869605017
-31069,LRG1,0.09907691162918633
-31071,TNFAIP8L1,-0.047757244377345076
-31072,MYDGF,-0.16818443580666476
-31074,DPP9,0.03400779689036546
-31079,FEM1A,-0.0018465175124071354
-31080,AC005523.2,-0.08860005091789469
-31081,TICAM1,0.11473791397424264
-31083,PLIN3,0.30732177453409637
-31084,ARRDC5,-0.010939254755992129
-31086,KDM4B,0.11589485341348299
-31092,SAFB2,0.04889812040716548
-31093,SAFB,0.0467386567632858
-31094,RPL36,-0.34285884727434196
-31095,MICOS13,-0.10551402910374587
-31097,LONP1,-0.026955048258564054
-31100,DUS3L,-0.009209815143457527
-31106,NDUFA11,-0.02754102420809061
-31108,VMAC,-0.07014026897572678
-31109,AC104532.2,-0.04079004482643299
-31110,CAPS,-0.03827876980276227
-31111,RANBP3,-0.023666800554247112
-31113,RFX2,-0.005539376843644482
-31117,MLLT1,0.08591475445078493
-31121,CLPP,-0.1029997496635962
-31122,ALKBH7,-0.09438006008039201
-31124,GTF2F1,-0.016532763590493153
-31126,KHSRP,0.08946835578680566
-31130,DENND1C,-0.0631514777101265
-31137,TNFSF14,-0.24026868804707424
-31140,GPR108,-0.017638684197555807
-31143,SH2D3A,-0.13981190801934704
-31144,VAV1,0.005549716611536318
-31145,ADGRE1,0.20886095612962832
-31152,ZNF557,0.015372732604838343
-31153,INSR,0.23888996734106893
-31154,AC119396.1,-0.04649224018398573
-31156,ARHGEF18,-0.040285968587946475
-31159,ZNF358,0.03189583587824733
-31161,MCOLN1,0.23469110069844779
-31162,PNPLA6,0.26665067002719206
-31164,XAB2,0.08409394405748832
-31165,PET100,-0.04944275252119753
-31168,STXBP2,0.33278214963812247
-31169,RETN,0.20389351856261365
-31170,MCEMP1,0.24183670384144526
-31171,TRAPPC5,0.266224290574946
-31172,FCER2,0.042075756339993
-31176,EVI5L,-0.005972718546825741
-31181,MAP2K7,0.09806687717047902
-31185,SNAPC2,0.09068222763639927
-31188,TIMM44,-0.08901197162693703
-31190,ELAVL1,0.12914611971295617
-31195,CERS4,-0.13146188610823537
-31197,CD320,-0.31369976016333556
-31198,NDUFA7,0.0004887957795309435
-31199,RPS28,-0.3633181786067579
-31202,RAB11B-AS1,0.016921299505551952
-31203,RAB11B,0.04494177434351722
-31204,MARCH2,0.06474673120485043
-31206,HNRNPM,-0.08329322272634641
-31207,PRAM1,0.4457871200154126
-31208,ZNF414,-0.05485206663716139
-31209,MYO1F,0.42610713490795626
-31225,ZNF317,0.0709288495562222
-31229,ZNF699,0.005289367882042942
-31231,ZNF559,-0.07608149469728523
-31234,ZNF266,-0.0028812299989037942
-31239,ZNF426,-0.07515173376751925
-31241,ZNF121,0.017104672653521356
-31242,ZNF561,-0.053177144303925006
-31243,ZNF561-AS1,0.014585786829079423
-31244,ZNF562,-0.01295386753840543
-31247,ZNF846,-0.09368642169072028
-31249,FBXL12,0.04725454704426189
-31250,UBL5,-0.11760595959681121
-31252,PIN1,-0.20772623941570076
-31253,OLFM2,-0.10120502636071091
-31257,SHFL,-0.025333972430423794
-31260,PPAN,-0.13210826071791285
-31261,P2RY11,-0.12846315083137752
-31262,EIF3G,-0.23097406622449454
-31263,DNMT1,-0.09054399538439031
-31264,S1PR2,-0.02923908571784921
-31265,MRPL4,0.0022704439733217075
-31268,ICAM1,0.19077284558637642
-31270,ICAM4,0.12060080751489448
-31272,ZGLP1,0.08412294548033067
-31273,FDX2,-0.031365170654358246
-31274,RAVER1,-0.008060609898603154
-31276,ICAM3,-0.21833652886352714
-31277,TYK2,0.17787759480227439
-31278,CDC37,-0.04100968819229878
-31279,PDE4A,0.02117703069426015
-31280,KEAP1,0.010099235773471353
-31282,S1PR5,-0.4767393934303378
-31283,ATG4D,-0.14451954378451481
-31284,KRI1,-0.011162672397646399
-31285,CDKN2D,-0.060429058821040896
-31287,SLC44A2,-0.12486136890649328
-31288,ILF3-DT,-0.09369270579525567
-31289,ILF3,0.013020920434189958
-31290,QTRT1,-0.0260989453204583
-31291,DNM2,0.1350888119999751
-31292,TMED1,0.025329003468411573
-31293,C19orf38,0.44477404120201064
-31294,CARM1,0.08494825843873853
-31295,YIPF2,0.019846922287627425
-31296,TIMM29,-0.01453524295891114
-31297,SMARCA4,0.11067970787510817
-31299,LDLR,-0.03432936410047263
-31307,AC011472.2,0.16721598308024066
-31308,RAB3D,0.3568979708938869
-31310,TMEM205,0.1295297256549926
-31311,CCDC159,0.06271841904431381
-31312,PLPPR2,0.2210082673433411
-31314,SWSAP1,0.04014088484992133
-31315,EPOR,0.07343990708177918
-31317,CCDC151,0.08538128544148937
-31318,PRKCSH,0.09546248955099337
-31321,ECSIT,0.0275401437693144
-31323,ELOF1,-0.1278501441666159
-31325,ACP5,0.008612216152143564
-31329,ZNF441,0.0012294714117935736
-31332,ZNF440,0.042935486238613194
-31334,ZNF439,-0.04970298919558966
-31336,ZNF700,0.034802486712442474
-31338,ZNF763,0.007908062256851632
-31339,ZNF433-AS1,-0.11090153135127731
-31342,ZNF844,-0.00583209002840183
-31345,ZNF136,-0.03483652947195865
-31346,ZNF44,-0.060908656140050886
-31347,ZNF563,-0.04079664763895247
-31349,ZNF799,-0.010409496934756186
-31353,ZNF564,0.044188256600740314
-31356,ZNF490,0.027363196717447504
-31357,ZNF791,0.04680702348274843
-31358,MAN2B1,0.3417859299779304
-31359,WDR83,-0.09507874147175518
-31360,WDR83OS,0.007072534570887276
-31362,DHPS,-0.030228188476042948
-31367,TNPO2,0.03416482016423421
-31369,TRIR,-0.18353052120976857
-31370,GET3,-0.06126246823339832
-31372,HOOK2,-0.001294360742541508
-31374,JUNB,0.3995613460207629
-31375,PRDX2,-0.3274434630000776
-31382,DNASE2,0.04193373565447344
-31385,GCDH,-0.00911016611469565
-31387,FARSA,0.0907883798106322
-31389,CALR,-0.18994576813056716
-31391,RAD23A,-0.045238460663472514
-31392,GADD45GIP1,-0.22566936557151734
-31394,NFIX,0.14531960895224297
-31397,LYL1,0.2498402686960637
-31398,TRMT1,0.18832790912659317
-31399,NACC1,0.059006235016781164
-31402,STX10,0.1351880175342034
-31403,IER2,0.15898067707456745
-31406,CACNA1A,0.042813000206062
-31407,CCDC130,-0.04848994368189351
-31408,MRI1,-0.027051545573511936
-31410,C19orf53,-0.22859768671760936
-31413,AC020916.1,0.38929196185665743
-31416,CC2D1A,0.030796264791063146
-31418,DCAF15,0.01835846415627249
-31419,RFX1,0.0502062709708439
-31423,IL27RA,-0.038603633264951975
-31427,SAMD1,-0.014064761891464266
-31428,PRKACA,0.3141673073894181
-31430,ASF1B,0.12562043659145508
-31431,AC022098.1,0.10015644598392774
-31438,ADGRE5,0.14273418560327716
-31440,DDX39A,-0.06874839649783668
-31441,PKN1,0.10945961159030045
-31442,AC008569.2,0.024879373038568887
-31444,GIPC1,-0.26154959217273027
-31445,DNAJB1,-0.07986900623288473
-31446,TECR,-0.21114529274896549
-31447,NDUFB7,-0.19244778624300385
-31449,ADGRE3,0.1251456142390486
-31450,ZNF333,0.06552144492361331
-31451,ADGRE2,0.3030011988963641
-31462,ILVBL,-0.019269374290022633
-31467,BRD4,0.08105032249643747
-31469,AKAP8,0.0028213172148801634
-31471,AKAP8L,-0.06985271802176311
-31472,WIZ,0.0158775271268742
-31473,RASAL3,-0.26715922139048026
-31493,TPM4,0.1493049402813677
-31495,RAB8A,0.11745412375252372
-31497,HSH2D,-0.009671066626280428
-31499,FAM32A,0.11833400217719174
-31502,AP1M1,0.0250687150933312
-31503,AC020911.2,0.03644919743283876
-31504,KLF2,-0.4097822679201057
-31506,EPS15L1,-0.022051717260823424
-31510,CHERP,-0.008758875397076085
-31512,SLC35E1,0.13703489991932097
-31515,AC008764.8,-0.01226526906665209
-31516,MED26,0.033504522226259985
-31520,SMIM7,0.012038404166505045
-31521,AC024075.2,0.0037743758630177926
-31522,AC024075.1,0.05007543731983098
-31523,TMEM38A,0.11090665607628587
-31526,SIN3B,0.17898355884285197
-31529,CPAMD8,0.1087982094068722
-31531,HAUS8,0.014981395947817652
-31533,MYO9B,0.18766963282452417
-31538,USE1,-0.05737141881545378
-31539,OCEL1,0.011660619317782691
-31540,NR2F6,-0.028797712455809546
-31542,BABAM1,0.07950485557934954
-31544,ABHD8,0.06532039447504383
-31545,MRPL34,-0.15872177808491097
-31547,DDA1,-0.043685214315626754
-31549,GTPBP3,-0.05235870468098454
-31553,BST2,0.16828229510891166
-31554,BISPR,0.06740727868975814
-31555,MVB12A,0.07037667779322243
-31561,SLC27A1,0.08857266616944441
-31562,AC010618.3,-0.0040096533699229015
-31563,PGLS,0.2636204844603219
-31565,NIBAN3,-0.0017700997423094644
-31566,COLGALT1,0.1830971763387983
-31569,MAP1S,0.0720093013155348
-31571,FCHO1,0.053858515795995175
-31574,JAK3,0.016065070703849067
-31575,RPL18A,-0.3162564983387396
-31577,CCDC124,-0.0009635317375965617
-31579,ARRDC2,0.006660863537132654
-31580,IL12RB1,-0.10101571297594338
-31581,MAST3,-0.01811978418053457
-31583,PIK3R2,-0.08039662247119884
-31584,IFI30,0.6237941191812438
-31585,MPV17L2,0.06728848394200511
-31591,JUND,0.22644734300707328
-31592,LSM4,-0.02418958292169447
-31593,PGPEP1,0.07180077077484628
-31595,LRRC25,0.5038229640086286
-31596,SSBP4,-0.09851256931158504
-31599,ELL,0.033809621239378695
-31601,FKBP8,-0.035221409065881316
-31603,KXD1,0.04437575000630813
-31604,AC005253.1,-0.09136033978565117
-31606,UBA52,-0.126403816687493
-31608,REX1BD,-0.0999793311013489
-31610,KLHL26,0.010312637590687358
-31611,CRTC1,0.06841228821130187
-31613,UPF1,0.0951631736862715
-31617,COPE,0.05055030647448062
-31618,DDX49,0.039480983114709725
-31619,HOMER3,0.21410550459433944
-31621,SUGP2,0.031973271421632754
-31622,ARMC6,0.024586289893449274
-31623,SLC25A42,-0.2020322789847506
-31624,TMEM161A,-0.06538189433547686
-31625,MEF2B,0.06409140652451283
-31626,BORCS8,0.040658591560110084
-31627,RFXANK,-0.08161961405142455
-31628,NR2C2AP,-0.20510087418007067
-31632,SUGP1,-0.050285306267945785
-31633,MAU2,0.11361780100140009
-31634,GATAD2A,-0.04762828533898853
-31637,NDUFA13,-0.11591688501905228
-31640,PBX4,-0.13668011255049697
-31642,LPAR2,0.05118411217283183
-31643,GMIP,0.14455249427076222
-31644,ATP13A1,0.1326709209663949
-31645,ZNF101,-0.14448618796292562
-31646,ZNF14,-0.035874370328821355
-31650,ZNF506,-0.07926340354986511
-31652,ZNF253,-0.14596660946513912
-31654,ZNF93,-0.029976454168199666
-31655,ZNF682,-0.01613238576387384
-31656,ZNF90,-0.03996224534238795
-31658,ZNF486,-0.06945192532893422
-31660,ZNF737,-0.16031448359429443
-31662,ZNF626,-0.17324388517577963
-31664,ZNF66,-0.12813646476324725
-31665,ZNF85,0.0005763561050066928
-31667,ZNF430,-0.15780697159450774
-31668,ZNF714,-0.008432933691523219
-31669,ZNF431,-0.07237162390174497
-31670,ZNF708,-0.15583098386056435
-31671,ZNF738,-0.03305618942015001
-31672,ZNF493,-0.09006807717487159
-31675,ZNF429,-0.06778311503642494
-31679,ZNF100,0.08320834088212788
-31681,ZNF43,-0.11987045214294696
-31708,AC092329.4,-0.08871083951258994
-31709,ZNF91,-0.22287959914946018
-31712,ZNF675,-0.10348319288884156
-31713,ZNF681,-0.06820268660503845
-31719,AC092279.1,-0.033873894398646125
-31720,ZNF254,-0.10672310083328763
-31723,LINC00662,-0.008620262804195774
-31725,AC006504.5,0.022972368767200907
-31745,UQCRFS1,0.1235476709602609
-31754,POP4,-0.006019566694016204
-31755,PLEKHF1,-0.4761473789868448
-31756,C19orf12,-0.16655901919306307
-31760,URI1,-0.1549749458773366
-31771,TSHZ3,0.18600415255852434
-31784,ZNF507,-0.10395369992926845
-31786,DPY19L3,0.007084217379098639
-31787,PDCD5,-0.173639221561781
-31788,ANKRD27,0.10934371299103203
-31792,NUDT19,-0.02574817074278208
-31797,CEP89,-0.012784926150710434
-31800,GPATCH1,-0.07935437963137386
-31809,CEBPA,0.3634370940935467
-31813,CEBPG,0.1800281524544441
-31814,PEPD,0.0476412215844334
-31817,KCTD15,0.14902446793463248
-31820,LSM14A,-0.0663323511683215
-31821,KIAA0355,-0.0804370600571544
-31823,GPI,0.07635692116929493
-31824,PDCD2L,-0.09723393885666662
-31826,UBA2,-0.04206664610935096
-31831,ZNF302,-0.0350730002418068
-31833,ZNF181,-0.09113347727602326
-31838,AC008555.4,-0.1612896287784488
-31844,GRAMD1A,-0.03413029924226517
-31853,FXYD1,-0.07185136114060132
-31854,FXYD7,-0.07916156776848313
-31855,FXYD5,-0.005895298691166301
-31857,LSR,-0.07806730993054828
-31860,USF2,0.1670804432187585
-31863,CD22,0.003610205060165965
-31871,FFAR2,0.2179821336904092
-31876,TMEM147-AS1,-0.012573931093417473
-31877,TMEM147,0.03965237033367385
-31883,HAUS5,-0.06947995307765059
-31884,RBM42,0.09733151238643391
-31886,COX6B1,0.001565525719653916
-31890,KMT2B,-0.010269634250548604
-31891,IGFLR1,0.11536194514900783
-31892,U2AF1L4,-0.08278945486564067
-31893,PSENEN,0.0008356083915174178
-31895,LIN37,-0.06974899385101709
-31898,PROSER3,-0.03601416221803384
-31905,NFKBID,0.06543714444743251
-31906,HCST,-0.618117235436427
-31907,TYROBP,0.499782930990521
-31911,SDHAF1,-0.064625951044657
-31913,ALKBH6,0.002348916311648211
-31919,POLR2I,-0.1427551326856941
-31920,TBCB,-0.18471209033767136
-31921,CAPNS1,0.20042597075132665
-31924,ZNF565,-0.07304952188759797
-31925,ZNF146,0.005755521176976411
-31928,ZFP14,-0.15168983408123665
-31931,ZFP82,-0.0733790735405141
-31932,ZNF566,-0.0349579624363996
-31934,ZNF260,-0.08954449353197029
-31936,ZNF529,-0.07093431126328682
-31937,ZNF529-AS1,-0.035888583496940414
-31941,LINC01534,-0.07037266897088554
-31942,ZNF567,-0.09676917361782726
-31946,ZNF790-AS1,-0.09774251181438844
-31947,ZNF790,-0.03802420597111008
-31948,ZNF345,-0.07941314562416371
-31950,ZNF568,-0.1475930268736371
-31951,ZNF420,-0.0783680410988408
-31953,AC010632.2,0.011666740172326497
-31954,ZNF585A,-0.1299103569707002
-31956,ZNF585B,-0.0316491744771851
-31957,ZNF383,-0.05662064198962002
-31961,ZNF875,-0.015987526209675182
-31964,ZNF527,-0.06235951873806962
-31965,ZNF569,-0.1895958010799119
-31966,ZNF570,-0.148131813734073
-31973,ZNF571,-0.05988936261161383
-31979,ZNF573,-0.06331142052712975
-31981,SIPA1L3,0.04619607385054292
-31987,SPINT2,0.24927897378477856
-31988,PPP1R14A,0.005871506473285547
-31990,YIF1B,0.057027200087653634
-31992,KCNK6,0.2974694940402477
-31995,PSMD8,-0.1395477709648004
-31999,FAM98C,-0.0119018074684586
-32000,RASGRP4,0.26674306796423425
-32004,MAP4K1,-0.10107452088330182
-32006,EIF3K,-0.16391887141911216
-32007,ACTN4,-0.022409796352164613
-32009,CAPN12,-0.19283076893676807
-32014,ECH1,-0.18624135092942307
-32016,HNRNPL,0.02841509519945201
-32017,AC008982.2,0.044520859041188694
-32018,RINL,-0.156352468445766
-32020,SIRT2,0.031444183093185196
-32021,NFKBIB,-0.05115723804212018
-32023,SARS2,-0.03252398591291689
-32024,MRPS12,-0.16821664982055756
-32031,PAK4,-0.06149385368046109
-32040,GMFG,0.10167816357434321
-32041,AC011445.2,-0.04281754735015724
-32042,SAMD4B,0.10938547144431593
-32043,PAF1,0.017255193330975135
-32044,MED29,-0.01306256611603622
-32045,ZFP36,0.44383104750015906
-32046,PLEKHG2,0.10970828448298278
-32047,RPS16,-0.32902273788469977
-32048,SUPT5H,-0.008483274468254183
-32049,TIMM50,0.015372174268973377
-32052,EID2B,-0.08503188788536152
-32054,EID2,-0.04748485207514284
-32061,DYRK1B,-0.017685700260643936
-32062,FBL,-0.04960264693598085
-32064,PSMC4,-0.06989785823298526
-32066,ZNF780B,-0.03194845160921013
-32067,ZNF780A,0.07190294182016572
-32069,MAP3K10,0.02012223148291495
-32072,AKT2,-0.01718771323185104
-32075,PLD3,0.1737921756478847
-32078,SERTAD1,-0.0029207610776385254
-32080,SERTAD3,0.1387073153094127
-32082,BLVRB,0.4008905313873433
-32084,SHKBP1,0.21454671924545904
-32085,LTBP4,-0.09195744938312614
-32087,COQ8B,0.034323975813448294
-32089,C19orf54,0.00360475851182852
-32090,SNRPA,-0.0968383825920439
-32092,RAB4B,-0.015864893506000382
-32094,EGLN2,-0.07504881316765784
-32102,CYP2S1,0.11414589939231427
-32105,HNRNPUL1,0.08369823848505878
-32106,TGFB1,0.09751890521106922
-32107,AC011462.5,0.016167519763653344
-32108,CCDC97,-0.0666800840148708
-32109,TMEM91,0.1735690347302328
-32110,B9D2,-0.038735001927438194
-32112,EXOSC5,-0.03294538510884622
-32113,BCKDHA,0.14938990925189738
-32116,B3GNT8,0.10644831801651342
-32117,DMAC2,-0.0658677007184274
-32122,AC243960.1,-0.25903784004099717
-32123,CEACAM21,-0.18036934240575425
-32124,CEACAM4,0.21082515923156997
-32133,RPS19,-0.4226967192773783
-32134,CD79A,-0.0019253042287418554
-32135,ARHGEF1,-0.11214422691698868
-32137,RABAC1,-0.21658250147707933
-32140,ZNF574,0.003543295186825496
-32141,POU2F2,0.48450014408393316
-32142,AC010247.1,0.04229366277077955
-32143,AC010247.2,-0.006061831947437014
-32144,DEDD2,0.05988488704796078
-32145,ZNF526,-0.07809215468084886
-32146,GSK3A,0.04404504913258795
-32147,ERF,0.060272369590380355
-32148,CIC,0.0010301456141940946
-32149,PAFAH1B3,-0.09416127026751828
-32154,LIPE-AS1,0.006685501725951815
-32178,PHLDB3,-0.06113654536392681
-32179,ETHE1,-0.01723729779735348
-32181,XRCC1,0.056753258903423934
-32185,IRGQ,0.10608956955378095
-32186,ZNF576,-0.015391872045781501
-32188,ZNF428,-0.2541572257845223
-32190,PLAUR,0.3268837399256171
-32192,SMG9,0.020565660050864087
-32193,KCNN4,0.10771178132336083
-32196,ZNF283,-0.08624200727938172
-32201,ZNF45,-0.03502627630580012
-32207,ZNF230,0.04223143201010498
-32210,ZNF223,-0.028611764869357213
-32212,ZNF224,-0.08708607045370112
-32214,ZNF225,0.03739096175172539
-32215,ZNF234,-0.041189027547638445
-32217,ZNF226,-0.10766386581592226
-32218,ZNF227,-0.06934576046754497
-32224,ZNF180,0.0596346647232715
-32229,PVR,0.14657136228734896
-32232,BCL3,0.14871742955352324
-32236,NECTIN2,0.1345122404433781
-32238,TOMM40,-0.03331164416265992
-32244,CLPTM1,-0.015675461263087345
-32245,RELB,0.07968476819197824
-32246,CLASRP,0.06207086595122042
-32247,ZNF296,-0.06833559890756101
-32249,GEMIN7,-0.08955648895689865
-32250,MARK4,-0.006011197516319457
-32251,PPP1R37,-0.028690641331322507
-32254,TRAPPC6A,-0.10150313634594844
-32255,BLOC1S3,0.004656632353599937
-32259,ERCC2,0.04461797951548292
-32261,CD3EAP,0.0592942972516841
-32262,ERCC1,0.10340344636912104
-32263,FOSB,0.3561430509162172
-32265,PPM1N,0.04944430238888332
-32266,VASP,0.39355325326584933
-32267,OPA3,0.004567980749931156
-32269,EML2,-0.052833915918888516
-32272,SNRPD2,-0.30146639370071154
-32274,FBXO46,-0.01821936937680887
-32280,DMWD,-0.1401948226644394
-32282,SYMPK,0.20369585622041686
-32285,IRF2BP1,-0.05312731928965722
-32286,MYPOP,0.00804985194248701
-32289,CCDC61,-0.027686931714163452
-32303,PPP5C,-0.11914719167144713
-32313,CALM3,0.07244827881392694
-32315,PTGIR,0.1220353745709846
-32319,PRKD2,-0.026898661144039212
-32320,STRN4,0.19868615469029266
-32322,FKRP,0.02995815499084765
-32323,SLC1A5,0.2101038846383286
-32325,AP2S1,0.2802905082002303
-32326,ARHGAP35,-0.05489633930964868
-32328,TMEM160,-0.1272721416370933
-32329,ZC3H4,0.12577265297695797
-32330,SAE1,-0.02992926109433244
-32331,BBC3,-0.049853954360721035
-32333,CCDC9,0.10419907536034059
-32334,INAFM1,0.021645667475411714
-32335,C5AR1,0.3942667682576428
-32336,C5AR2,0.21583150177070795
-32337,DHX34,0.034905001621365395
-32341,KPTN,0.04716889776539971
-32343,NAPA,0.19862981387298037
-32346,AC010331.1,0.013501058484520701
-32347,BICRA,-0.019132606499963293
-32350,NOP53,-0.28357810249038773
-32352,SELENOW,-0.27020370615665124
-32362,LIG1,-0.12966365138764163
-32365,ZSWIM9,-0.1007983198138403
-32366,CARD8,0.1045795422715315
-32368,CARD8-AS1,0.06068250575122434
-32372,EMP3,0.009757156294930656
-32375,KDELR1,0.21245923533516486
-32377,GRWD1,-0.05616759217788919
-32380,CYTH2,-0.018713115367737565
-32386,RPL18,-0.3547427850634799
-32388,SPHK2,-0.0499246583407313
-32389,DBP,-0.1090362552410183
-32390,CA11,-0.05064957923822667
-32398,BCAT2,-0.027416577733006674
-32402,PPP1R15A,0.22872787646632162
-32404,NUCB1,0.12089254358268575
-32408,BAX,0.05845702648789727
-32410,FTL,0.6374974310854091
-32411,GYS1,-0.02542789873816336
-32412,RUVBL2,0.06956288932132489
-32425,SNRNP70,0.009704776828545475
-32426,LIN7B,-0.08194739656337958
-32430,TRPM4,0.08595725079125315
-32433,CD37,0.10346569559217304
-32442,PIH1D1,-0.09727487331508887
-32443,ALDH16A1,0.09544554140645196
-32444,FLT3LG,-0.2035531135626205
-32445,RPL13A,-0.33253412071951116
-32446,RPS11,0.009686066671817439
-32447,FCGRT,0.5127993134407272
-32448,RCN3,0.0359454098309363
-32449,NOSIP,-0.23770639382262324
-32451,PRR12,-0.0693855342094637
-32453,RRAS,0.195470251497156
-32454,SCAF1,0.045905569571642484
-32455,IRF3,-0.15956186368605285
-32456,BCL2L12,-0.05922692992177569
-32457,PRMT1,-0.09692555968369058
-32462,AP2A1,0.18424020688471637
-32465,MED25,0.03845434815345377
-32467,PTOV1,0.06652173583571051
-32470,PNKP,-0.038121287307721276
-32471,AKT1S1,0.07610268188207998
-32472,TBC1D17,-0.005330685565194641
-32473,IL4I1,0.06792516250890812
-32474,NUP62,0.21246509873698632
-32475,ATF5,0.2781807170572091
-32477,VRK3,-0.06505752404383458
-32486,KCNC3,0.1620790319837647
-32487,NR1H2,0.07809808466066888
-32490,POLD1,0.06666410556385555
-32491,SPIB,-0.0031068452004839363
-32494,EMC10,0.02556348841420001
-32497,JOSD2,0.01578102542918136
-32504,CLEC11A,-0.00042664262395290857
-32509,C19orf48,-0.08858309518271353
-32531,CTU1,-0.050687556575005344
-32532,SIGLEC9,0.2749154663970839
-32533,SIGLEC7,0.1919727072137674
-32535,CD33,0.4043759326283114
-32543,ETFB,0.0050710691452455926
-32546,NKG7,-0.7176278152118245
-32550,SIGLEC10,0.22327360003051858
-32558,ZNF175,0.0017489961277721918
-32560,SIGLEC5,0.17886932651295256
-32561,AC018755.4,0.2736287019335654
-32562,SIGLEC14,0.28533483772082086
-32564,SPACA6,-0.03473357780931659
-32566,FPR1,0.5036563013288317
-32567,FPR2,0.30973738236965753
-32570,ZNF577,0.06592353122153803
-32573,ZNF649,-0.04336056043116191
-32576,ZNF350,-0.017462068469388175
-32577,ZNF615,-0.06905122977614009
-32578,ZNF614,-0.004206120466334758
-32579,ZNF432,-0.07623536409334875
-32582,ZNF841,-0.004454798106023448
-32583,ZNF616,-0.007054728958797083
-32585,ZNF836,-0.04816095754985506
-32587,PPP2R1A,0.06373892039716032
-32589,ZNF766,-0.08188557827044271
-32591,ZNF480,-0.09391667951986128
-32594,ZNF880,-0.12693325454682122
-32596,ZNF528,-0.03030346944433703
-32601,ZNF808,-0.028858636938363536
-32602,ZNF701,-0.12038694598091414
-32604,ZNF83,-0.06345759891889749
-32607,ZNF611,0.016634248599295187
-32608,ZNF600,-0.1966410577814949
-32609,ZNF28,-0.015599875909800732
-32610,ZNF468,-0.10970067990486805
-32614,ZNF816,-0.04636530670017045
-32618,ZNF160,0.03860286342120598
-32620,ZNF347,-0.08117101339462345
-32623,ZNF677,-0.027535321569049323
-32628,ZNF845,-0.08387057880078955
-32629,ZNF525,-0.020644799790504282
-32630,ZNF765,-0.003347773654948124
-32631,AC022137.4,-0.06615745406730145
-32632,ZNF761,-0.032253622245429575
-32633,ZNF813,-0.021659830826964536
-32634,ZNF331,-0.10071701872762803
-32639,NLRP12,0.2410521769397425
-32641,MYADM,0.34018457875654123
-32647,VSTM1,0.2576155598742958
-32649,OSCAR,0.4254590514698113
-32650,NDUFA3,-0.1929014445285984
-32651,TFPT,-0.05891838569458354
-32652,PRPF31,-0.03485026704862188
-32654,CNOT3,0.09981598477925194
-32655,LENG1,-0.1346142517599469
-32657,MBOAT7,0.3202875747877984
-32658,TSEN34,0.22412672815787904
-32660,RPS9,-0.11771619882305247
-32661,LILRB3,0.4839725897488994
-32663,LILRA6,0.28854216427751833
-32665,LILRB2,0.44513010726363805
-32666,LILRA5,0.41658450513929257
-32668,LAIR1,0.12347744368583147
-32674,LENG8-AS1,-0.033229018273330205
-32675,LENG8,0.061666947286139376
-32676,LENG9,0.05321188594090975
-32678,LAIR2,-0.5151897741300032
-32679,LILRA2,0.4131598600743399
-32681,LILRA1,0.35888601939447917
-32682,LILRB1,0.34491129444110163
-32684,LILRB4,0.2786039039256724
-32691,FCAR,0.16785327440574863
-32693,NCR1,-0.3451475818973619
-32698,RDH13,-0.11842324714297327
-32700,PPP1R12C,0.11783979257121381
-32710,TMEM86B,0.019491369612753908
-32711,PPP6R1,0.002919625945301138
-32712,HSPBP1,-0.05194909343182161
-32715,TMEM150B,0.22251638736231272
-32723,TMEM238,-0.17153448334716254
-32724,RPL28,-0.23298331593317062
-32725,UBE2S,-0.041224232616449544
-32727,ISOC2,0.025697756139371117
-32730,ZNF628,0.03109761236812665
-32735,ZNF579,-0.006841653518799298
-32736,FIZ1,0.04943386866878747
-32737,ZNF524,0.06924827186738192
-32740,ZNF784,0.075385056836982
-32741,ZNF580,0.04814713455089357
-32742,ZNF581,0.044275415097354644
-32743,CCDC106,-0.047639767435009917
-32744,U2AF2,0.049003502322394595
-32747,EPN1,0.1437364387282177
-32758,ZNF787,-0.02966762068989088
-32759,ZNF444,-0.06642035270169803
-32763,ZSCAN5A,0.039908569014639274
-32774,ZNF583,-0.05992896659331572
-32793,ZNF264,-0.08926846727833887
-32795,ZNF805,-0.056900753174720334
-32798,ZNF460,-0.03296317298757854
-32799,AC005261.1,0.05027831412872918
-32800,ZNF543,-0.03152543910870507
-32804,TRAPPC2B,-0.1086692905637788
-32805,ZNF548,-0.005805253944344559
-32806,ZNF17,-0.02957545835794791
-32807,ZNF749,-0.05058573589327389
-32812,ZNF419,-0.02296668059394845
-32813,ZNF773,-0.08221088087535426
-32814,ZNF549,-0.012902466534819603
-32815,ZNF550,-0.057020720534077696
-32819,ZNF134,-0.03916144498355634
-32821,ZNF211,-0.06436902607702083
-32823,ZNF551,-0.0062691292904613945
-32824,ZNF154,-0.025043988527475372
-32825,ZNF671,0.04460645894387295
-32826,ZNF776,-0.01033690071443204
-32827,ZNF586,-0.008896999838353644
-32828,ZNF552,-0.028946019721934373
-32829,ZNF587B,-0.024178310239804286
-32830,ZNF814,-0.0338911526924936
-32831,ZNF587,0.057928150125513826
-32833,ZNF417,0.07558106618055226
-32837,ZNF606,-0.03036796353239018
-32838,AC008969.1,-0.03145962166725035
-32842,ZSCAN18,-0.12379772892326617
-32845,ZNF274,-0.06582135241830754
-32846,ZNF544,-0.11886854575139093
-32847,AC020915.2,-0.05415108591081656
-32849,ZNF8,-0.0009094378300607214
-32850,ERVK3-1,-0.12678397681353518
-32851,AC010642.2,0.014691683919854084
-32853,A1BG,0.07777714113245605
-32854,A1BG-AS1,0.0343627058825945
-32859,RPS5,-0.3738638649396007
-32868,ZNF324,-0.06250554776426766
-32869,ZNF446,0.007245630744569584
-32871,SLC27A5,-0.052256294278881996
-32873,ZBTB45,0.006550286637606361
-32874,TRIM28,-0.008216263166316063
-32875,CHMP2A,0.14088498920545547
-32876,UBE2M,-0.0732752071900736
-32878,MZF1,-0.011145556996079475
-32888,ZCCHC3,-0.038264486680300226
-32893,TRIB3,-0.08266925702535155
-32894,RBCK1,-0.021349260325694046
-32895,TBC1D20,-0.058074063568101846
-32896,CSNK2A1,0.09977675466039354
-32898,SRXN1,0.04598986035945822
-32901,FAM110A,0.10834795461587071
-32905,PSMF1,0.029314387220824002
-32911,SDCBP2-AS1,-0.09210392560039739
-32913,FKBP1A,0.4260539132431566
-32914,NSFL1C,0.12421633223394976
-32915,SIRPB2,0.2908556877618683
-32917,SIRPB1,0.36400652611259265
-32918,SIRPG,-0.1521852198097185
-32921,AL117335.1,0.1397608704297619
-32922,SIRPA,0.39066136197060314
-32925,STK35,-0.019287377864260966
-32932,SNRPB,-0.08978490908049311
-32935,NOP56,-0.08363031618994733
-32936,IDH3B,0.03235206869105252
-32942,PCED1A,0.00020164088539985757
-32943,VPS16,0.003049998910440176
-32944,PTPRA,-0.10481631241456571
-32946,MRPS26,-0.1354512308186686
-32950,UBOX5,-0.022237388488355464
-32951,FASTKD5,0.03628732003023615
-32953,DDRGK1,-0.08642910079986042
-32954,ITPA,0.013909625520884924
-32957,C20orf194,0.1273737826026809
-32958,ATRN,0.11691559327761362
-32961,SIGLEC1,0.268028309348012
-32963,C20orf27,0.21674172263773095
-32965,CENPB,0.10280583499088493
-32966,CDC25B,-0.26665874810982376
-32968,AP5S1,-0.019817485792756884
-32969,MAVS,0.16969979591771453
-32971,PANK2,0.22506006704258077
-32973,RNF24,0.24180944232717158
-32980,PRNP,0.20263837814231186
-32984,RASSF2,0.3861532313106453
-32985,SLC23A2,0.04580065331416403
-32990,TMEM230,-0.11744139007241078
-32991,PCNA,-0.1459135753159158
-32992,CDS2,-0.01821433331460174
-33001,GPCPD1,0.3437906256926762
-33002,SHLD1,-0.04114939835932811
-33004,TRMT6,-0.09122394479584933
-33008,CRLS1,0.099186590925187
-33021,TMX4,0.0005365971411373112
-33023,PLCB1,0.05576670932792182
-33039,MKKS,-0.03188950679934006
-33042,SLX4IP,-0.00790322149584744
-33055,BTBD3,0.09939031999788872
-33068,TASP1,-0.08789192863354021
-33069,ESF1,-0.05183728517083539
-33070,NDUFAF5,-0.09313009701367857
-33081,KIF16B,-0.031541021696684886
-33085,SNRPB2,-0.06359982482240352
-33093,DSTN,-0.36392115530094743
-33094,RRBP1,0.14169551513400563
-33097,SNX5,0.11376116845990303
-33099,MGME1,0.0011846025053525028
-33101,PET117,-0.04627086440031584
-33102,KAT14,0.06051022260146624
-33104,ZNF133,-0.03550271323938985
-33109,POLR3F,-0.08995796343677091
-33110,RBBP9,0.04684825144360244
-33111,SEC23B,0.09127015503023388
-33112,SMIM26,-0.034129229698698665
-33113,DTD1,-0.1340374929132813
-33116,LINC00653,-0.09778899380347598
-33126,RIN2,0.25566897499972785
-33127,NAA20,-0.12080403901404399
-33128,CRNKL1,-0.03311319666182666
-33134,RALGAPA2,0.10180090545062374
-33138,KIZ,-0.05860980488025344
-33142,XRN2,0.15289148352414822
-33165,THBD,0.12519541614382645
-33166,CD93,0.4752463252996572
-33171,NXT1,-0.07123337779459044
-33173,GZF1,0.06007505268620964
-33174,NAPB,0.04148198784015737
-33183,CST3,0.6165501647697734
-33201,CST7,-0.6955115050508607
-33202,APMAP,-0.34440480377301197
-33204,ACSS1,0.020879348209302154
-33209,ENTPD6,0.06620089521604658
-33212,PYGB,0.12504439559573657
-33215,ABHD12,0.05677590556020591
-33219,ZNF337-AS1,-0.05307708834770062
-33220,ZNF337,-0.06225787420361267
-33246,HM13,0.27736450090569753
-33254,BCL2L1,-0.12101298356796716
-33262,PDRG1,-0.057272057831270935
-33267,HCK,0.5129525037736945
-33268,TM9SF4,0.17204919840448885
-33270,PLAGL2,0.17986025687729734
-33271,POFUT1,0.05516600801890599
-33273,KIF3B,0.04028581408164596
-33275,ASXL1,-0.0767827979081858
-33276,NOL4L,-0.11852869496293349
-33284,COMMD7,-0.059357434882394054
-33286,MAPRE1,0.07467941685863422
-33288,AL035071.1,0.04032775312790983
-33300,CDK5RAP1,0.013573962900382564
-33302,CBFA2T2,0.06503033416379261
-33304,NECAB3,0.004911519548317953
-33309,PXMP4,0.0027554785428261877
-33313,CHMP4B,0.09229319686331611
-33315,RALY,0.04774697063227365
-33317,EIF2S2,0.01086729102916085
-33321,AHCY,0.05571401178857257
-33323,ITCH,0.06670588273434178
-33327,DYNLRB1,-0.16704686873175575
-33330,NCOA6,0.09931453139459671
-33334,ACSS2,0.15763221507484862
-33335,GSS,-0.015542372282696981
-33337,TRPC4AP,0.09427000655696426
-33338,EDEM2,0.09285025363707719
-33341,MMP24OS,-0.020845788964450718
-33343,EIF6,-0.016275211635762114
-33346,UQCC1,-0.022268699899575195
-33349,CEP250,-0.16430905099319928
-33350,FO393401.1,-0.07353028109233621
-33352,ERGIC3,0.16844430688831744
-33354,CPNE1,-0.06655877109558068
-33355,RBM12,0.1563751463742801
-33356,NFS1,0.02066396761319031
-33357,ROMO1,-0.09757160428802135
-33358,RBM39,-0.05260671919096399
-33359,PHF20,-0.04654887000218286
-33360,SCAND1,0.06286933018698675
-33361,CNBD2,-0.11820831658076096
-33362,NORAD,0.14951759903456646
-33369,AAR2,0.059301063899838836
-33370,DLGAP4,0.050938196654254496
-33373,TGIF2,0.048219873090084096
-33374,RAB5IF,0.32561993325898003
-33375,SLA2,-0.04067083012808659
-33376,NDRG3,-0.09069604938582204
-33377,DSN1,-0.046149686877852224
-33378,SOGA1,0.14051500418095353
-33380,SAMHD1,0.5029263522946887
-33381,RBL1,-0.08777815628295421
-33384,RPN2,0.07728306065668428
-33386,MANBAL,0.02839641960017401
-33388,SRC,0.18450570640970837
-33389,BLCAP,0.1440561244299066
-33395,CTNNBL1,-0.009647528129932548
-33397,TTI1,0.01651320714707415
-33398,RPRD1B,0.07052607864850564
-33406,BPI,0.1418391052475024
-33411,SNHG17,0.024000950450715
-33412,SNHG11,-0.009410717679729815
-33413,RALGAPB,0.033760450402542865
-33417,ACTR5,0.003628973029260694
-33418,PPP1R16B,-0.11804940599743889
-33422,DHX35,-0.010643874978891597
-33431,MAFB,0.4540097405077347
-33434,TOP1,0.07512399338424833
-33436,PLCG1,-0.130916113812551
-33437,ZHX3,0.006314743176119382
-33440,CHD6,-0.08061366971453235
-33448,SRSF6,0.10345153935382997
-33449,L3MBTL1,-0.00857038737732093
-33453,IFT52,-0.00026295693645678116
-33460,OSER1,0.010994546840666706
-33461,OSER1-DT,-0.06803669637664572
-33470,TTPAL,-0.06659036715308726
-33471,SERINC3,0.10097096800336099
-33472,PKIG,-0.022992196060406207
-33473,ADA,-0.17842467873989812
-33481,YWHAB,0.1512086528650779
-33482,PABPC1L,-0.005007286169522669
-33483,TOMM34,-0.044043930790170084
-33485,STK4,-0.2064789196548999
-33497,SYS1,-0.07312992319445817
-33499,DBNDD2,-0.11335173637379023
-33500,PIGT,0.03481084551201677
-33515,DNTTIP1,0.18036476488349534
-33519,ACOT8,0.022742825757917133
-33524,CTSA,0.17592011404657468
-33527,AL162458.1,0.009725740823100648
-33528,PCIF1,0.028471631202786802
-33529,ZNF335,-0.017779278479471812
-33533,NCOA5,0.060558881926342636
-33534,CD40,0.11806730212721815
-33537,SLC35C2,0.014665539120493275
-33539,ELMO2,0.03215984032671182
-33544,TP53RK,-0.0582368687878638
-33549,ZMYND8,-0.042728534802699975
-33554,NCOA3,0.1992519554119081
-33555,SULF2,0.4071200273168332
-33573,PREX1,-0.010329648402502377
-33575,AL133342.1,0.005977100774265159
-33576,ARFGEF2,0.06421494028119643
-33578,CSE1L,0.08786385830413886
-33579,STAU1,0.039157323777401214
-33580,DDX27,-0.013928994794249862
-33581,ZNFX1,0.23134706775163172
-33582,ZFAS1,-0.017334809462123058
-33585,B4GALT5,0.11023113994291807
-33586,SLC9A8,0.007348246856439122
-33587,SPATA2,0.02665968524330142
-33588,RNF114,0.13325241219353715
-33591,UBE2V1,-0.11445701862311955
-33592,TMEM189,0.0704645481229059
-33597,CEBPB,0.5125736013245366
-33598,SMIM25,0.35157191236559915
-33601,PTPN1,0.045143278176158354
-33606,BCAS4,-0.14074292753966125
-33607,ADNP,-0.0314847826147417
-33609,DPM1,-0.052063143133141886
-33610,MOCS3,0.023960781420406953
-33613,NFATC2,-0.2811095641392935
-33617,ZFP64,0.020020757877190216
-33625,TSHZ2,-0.0595194646194698
-33633,ZNF217,0.22634003059967767
-33641,PFDN4,-0.06641007133183398
-33652,FAM210B,0.05552081340881139
-33654,CSTF1,0.004178037159590319
-33656,RTF2,0.004697064181192724
-33671,RAE1,0.06666009968208837
-33674,RBM38,-0.1568672671632006
-33679,ZBP1,-0.10648216829962344
-33680,PMEPA1,-0.02375603267766945
-33687,RAB22A,-0.014609982463377963
-33688,VAPB,-0.011726974220921926
-33693,STX16,-0.039211988591182974
-33694,NPEPL1,0.10265170485657485
-33700,GNAS,0.12376023819326097
-33704,NELFCD,-0.081511491099719
-33705,CTSZ,0.5859786895246021
-33707,ATP5F1E,0.20897362209781434
-33708,PRELID3B,-0.032443385641819686
-33709,ZNF831,-0.14427243100029993
-33718,SYCP2,-0.018582885305542654
-33719,FAM217B,0.019683100369238017
-33720,PPP1R3D,0.12575185422538304
-33724,MIR646HG,-0.05278908611026418
-33735,TAF4,0.06921532318016844
-33737,LSM14B,0.03509138105694297
-33738,PSMA7,-0.03800255910584888
-33739,SS18L1,0.08077019062147775
-33740,MTG2,-0.00868175588553859
-33742,OSBPL2,0.05107214818859964
-33743,ADRM1,0.0013063265087261673
-33748,RPS21,-0.4035822896009248
-33771,MRGBP,0.008907096217259556
-33773,OGFR,0.22220010552369066
-33775,TCFL5,0.030158830553638105
-33776,DIDO1,-0.07026644673674895
-33777,AL117379.1,-0.03503212994536042
-33778,GID8,0.07174157842014701
-33779,SLC17A9,-0.06577945621482548
-33790,YTHDF1,0.09961491201731776
-33795,ARFGAP1,0.09989024106638145
-33804,PPDPF,-0.27795821649951685
-33809,HELZ2,0.1905619977993025
-33810,GMEB2,-0.08369747912725113
-33811,MHENCR,-0.08824558461795633
-33812,STMN3,-0.17457935148972992
-33813,RTEL1,0.019713306100954658
-33815,ARFRP1,-0.05453783559086323
-33816,ZGPAT,-0.08309747694327564
-33818,LIME1,-0.4620159031189959
-33819,SLC2A4RG,-0.23098766797404094
-33825,TPD52L2,0.24525054062807325
-33826,DNAJC5,0.12406903585440344
-33827,UCKL1,-0.08479452127564441
-33830,SAMD10,-0.09134613294682145
-33831,PRPF6,0.019561598068707994
-33832,C20orf204,-0.08143946115801075
-33834,TCEA2,-0.01794718873605022
-33836,RGS19,0.16329361845124024
-33837,OPRL1,0.11427594523732691
-33842,PCMTD2,-0.048181027451935886
-33847,GATD3B,-0.01983436580918249
-33848,FP565260.1,-0.03426034450381565
-33849,FP565260.6,0.049976375466418775
-33864,U2AF1L5,-0.02092328765537915
-33906,HSPA13,0.06852050447013289
-33907,SAMSN1,0.08469991718310328
-33913,NRIP1,0.24010517838157275
-33919,USP25,0.20290739163411203
-33929,BTG3,-0.0465692843348301
-33933,C21orf91,0.015229766452800768
-33973,LINC01684,-0.16572847808316077
-33985,MRPL39,-0.045057902454196344
-33988,ATP5PF,-0.049847265631823394
-33989,GABPA,0.10965217186630918
-33990,APP,0.2348799667046799
-34002,ADAMTS5,0.1338566468292555
-34013,N6AMT1,-0.03690858125396815
-34014,LTN1,0.03680942195371345
-34015,RWDD2B,-0.0008333606377459166
-34017,USP16,-0.11225533976195284
-34018,CCT8,0.03288119440906685
-34021,MAP3K7CL,0.08960395463881679
-34024,BACH1,0.42510269116510724
-34026,AP000240.1,0.1070479245797848
-34070,TIAM1,0.14208135941819094
-34074,SOD1,-0.2849073658012511
-34076,SCAF4,0.06535088944949845
-34082,MIS18A,-0.06902776978808553
-34086,URB1,-0.01783209521301879
-34087,URB1-AS1,-0.0636920111017173
-34090,CFAP298,-0.06649225088379025
-34091,SYNJ1,0.07372008798158938
-34092,PAXBP1-AS1,0.10740627757638532
-34093,PAXBP1,0.07776648646394853
-34094,C21orf62-AS1,-0.05517554808974308
-34101,OLIG1,0.16900516406782592
-34105,IFNAR2,0.05452563010830844
-34106,IL10RB-DT,0.09857755423510545
-34107,IL10RB,0.2222028891771736
-34108,IFNAR1,0.19445827729166806
-34109,IFNGR2,0.46718109642303157
-34110,TMEM50B,-0.03900453458453563
-34113,GART,-0.04931417636340137
-34114,SON,-0.028840434374763656
-34116,CRYZL1,0.004609477422699713
-34117,ITSN1,0.2303435732048944
-34118,ATP5PO,-0.08858090865510501
-34119,LINC00649,-0.18854634930154518
-34121,MRPS6,-0.178681087929345
-34122,SLC5A3,-0.09615451130845445
-34132,KCNE1,0.16261078728753509
-34133,RCAN1,0.07616180911495354
-34137,RUNX1,0.1915297078175525
-34142,SETD4,-0.0586258345741801
-34145,CBR1,0.1756534574258446
-34148,CBR3-AS1,-0.004765142071064847
-34149,CBR3,-0.0639197497966682
-34150,DOP1B,0.06887384224303326
-34152,MORC3,0.03980122046371531
-34153,AP000692.1,0.032034322103322084
-34163,HLCS,-0.08637426587655608
-34167,PIGP,-0.14483727627140608
-34168,TTC3,0.011016844317695194
-34173,VPS26C,-0.002892073715047047
-34177,DYRK1A,0.05213427284143468
-34191,ETS2,0.25409822615327926
-34199,PSMG1,-0.04804196266522157
-34200,BRWD1,0.04993141901050437
-34205,HMGN1,-0.20140291245135047
-34206,GET1,0.02272666862008927
-34221,BACE2,0.005623152101491478
-34225,MX2,0.3228232246404514
-34226,MX1,0.3424892530622765
-34237,PRDM15,0.06122833500485892
-34240,C2CD2,0.13823466535368356
-34241,ZBTB21,0.05977222871708139
-34245,ABCG1,-0.10340424965882929
-34250,UBASH3A,-0.1803020690084753
-34252,SLC37A1,0.012492887803213054
-34263,WDR4,0.032727090387464845
-34264,NDUFV3,0.06504713826480317
-34267,PKNOX1,0.018464366455873495
-34283,RRP1B,-0.13213570429553714
-34284,PDXK,0.23181341365986038
-34286,CSTB,0.0043099000532800585
-34287,RRP1,-0.03114298420693273
-34288,AATBC,0.2088504657079268
-34289,AGPAT3,0.21920993907219685
-34290,TRAPPC10,-0.17927996141571462
-34293,LINC01678,0.04445819544577478
-34297,ICOSLG,0.061754530417389354
-34303,PFKL,0.11507128385606527
-34305,CFAP410,-0.007088546575787054
-34309,TRPM2,0.09651709875149192
-34336,UBE2G2,0.0436678595393529
-34338,SUMO3,0.18193316972185256
-34339,PTTG1IP,0.2691393198848392
-34340,ITGB2,0.3096056337895797
-34341,ITGB2-AS1,0.03259373099249871
-34344,AL844908.1,0.12131215717000814
-34345,FAM207A,-0.05339948178670664
-34350,ADARB1,-0.14531310682522106
-34353,POFUT2,0.03062971145596596
-34354,LINC00205,0.03300632795614516
-34361,COL18A1,-0.04736471482139425
-34364,SLC19A1,0.19140625315964221
-34380,COL6A2,-0.2783773833205781
-34383,SPATC1L,-0.03410973372175591
-34385,LSS,-0.023833373122181978
-34387,MCM3AP-AS1,-0.13316569719586957
-34388,MCM3AP,0.026374267943741137
-34391,YBEY,-0.12105245394275924
-34393,PCNT,-0.02230030532492002
-34395,DIP2A,-0.1472630929645011
-34397,S100B,-0.1260152750224746
-34398,PRMT2,-0.3296861739436104
-34429,IL17RA,0.43388077897646854
-34430,TMEM121B,0.11715975128709188
-34432,HDHD5,0.07398971798884516
-34434,ADA2,0.5010115227194754
-34440,ATP6V1E1,0.1626638635200167
-34441,BCL2L13,0.0998718268276393
-34442,BID,0.34370814189719456
-34443,LINC00528,0.012593763667371957
-34444,MICAL3,-0.0543104618491647
-34450,PEX26,0.07657236332025706
-34453,USP18,0.05553421695957406
-34461,TMEM191B,-0.09857379675736212
-34473,DGCR2,0.09052959022565664
-34478,ESS2,0.021300105165626828
-34482,SLC25A1,0.06013144228042225
-34486,HIRA,0.05306356490662157
-34487,C22orf39,-0.02459521204485379
-34488,MRPL40,-0.12587873596780275
-34490,UFD1,-0.12546044454930236
-34502,RTL10,-0.046566366304405
-34503,TXNRD2,0.17139978459610603
-34504,COMT,0.18644362497132008
-34506,TANGO2,0.09076060236938618
-34509,DGCR8,-0.09810128928327279
-34511,TRMT2A,0.03166789711869719
-34512,RANBP1,-0.04005276571280924
-34513,ZDHHC8,-0.005015417041529991
-34518,DGCR6L,-0.0842436198559175
-34527,KLHL22,-0.05170196022596826
-34528,MED15,0.09812160287141665
-34530,PI4KA,0.04406189329535103
-34532,SNAP29,0.06414940284928164
-34534,CRKL,0.04300908944074044
-34535,LINC01637,0.17078187903879505
-34536,AIFM3,0.11934011636644545
-34538,LZTR1,0.034192587789753606
-34539,THAP7,-0.05698002150295869
-34540,THAP7-AS1,-0.06171685486351674
-34554,HIC2,0.07873324000457946
-34555,TMEM191C,-0.021802846060731212
-34557,UBE2L3,-0.014937520419435177
-34558,YDJC,-0.04436589100604576
-34561,SDF2L1,-0.15709536695552986
-34563,PPIL2,-0.06961398401456685
-34564,YPEL1,-0.2891883718247456
-34567,MAPK1,0.024874830990564728
-34568,PPM1F,0.3189781710975638
-34569,AC245452.1,0.053875905102213566
-34570,TOP3B,0.009311350455389437
-34592,AC245060.6,-0.14072949097527207
-34593,AC245060.5,0.022973152855409004
-34657,IGLC1,0.0009006594513248822
-34659,IGLC2,-0.0019122625168397835
-34661,IGLC3,-0.002100552377528111
-34673,BCR,-0.03744896982895483
-34684,RGL4,-0.24493553639887755
-34686,VPREB3,-0.0004450065624121173
-34688,CHCHD10,0.1408593156086575
-34690,SMARCB1,-0.0024567883120507543
-34693,SLC2A11,-0.05012434988747945
-34695,MIF,-0.40895651616237144
-34701,DDT,-0.041335861917279525
-34703,CABIN1,-0.06147499952260878
-34707,SPECC1L,0.022023728875807596
-34708,ADORA2A,-0.11850193420064367
-34712,GUCD1,0.10565221213556876
-34713,SNRPD3,-0.08321120661156972
-34740,GRK3,0.3894594919076004
-34750,ASPHD2,0.12879406457540543
-34751,HPS4,-0.025726505569745515
-34752,SRRD,-0.02087012055790631
-34753,TFIP11,-0.08170730857198093
-34755,TPST2,-0.1906620213714818
-34759,MIAT,-0.12443060721575026
-34761,MIATNB,-0.036809148641056985
-34784,PITPNB,-0.0656383027639427
-34785,TTC28-AS1,-0.04670659393723048
-34787,CHEK2,0.0014714971185082457
-34788,HSCB,-0.07857833565166841
-34789,CCDC117,0.03547952468636783
-34790,XBP1,-0.29367903767304965
-34791,Z93930.2,-0.11104432704683281
-34797,KREMEN1,0.12592233388340898
-34800,RHBDD3,0.03850334083491867
-34802,EWSR1,0.14605952317976853
-34803,GAS2L1,0.07357936119808767
-34806,AP1B1,0.20446054601210234
-34811,THOC5,-0.03336444620181149
-34813,NIPSNAP1,-0.09860044539413709
-34814,NF2,0.0024910342270484543
-34818,ZMAT5,-0.09269154097496289
-34820,UQCR10,-0.043443171612326366
-34821,ASCC2,-0.0350610474424689
-34822,MTMR3,0.20129109729002448
-34823,AC003681.1,0.019419827215052032
-34831,CASTOR1,-0.07540311390766388
-34832,TBC1D10A,-0.046245239711307784
-34833,SF3A1,0.040162923961105886
-34835,RNF215,0.07138064512001963
-34839,MTFP1,-0.1770275105727103
-34846,PES1,0.10882882912412344
-34848,TCN2,0.1832523680518305
-34849,SLC35E4,0.11566154673590068
-34850,DUSP18,0.15179512788630165
-34852,MORC2-AS1,-0.11579561964602356
-34853,MORC2,-0.08778712568915611
-34854,TUG1,0.03508569764133396
-34860,SELENOM,-0.15188868245498682
-34863,RNF185,0.08081741871230366
-34865,LIMK2,0.06302408027638935
-34866,PIK3IP1,-0.1880593319901471
-34868,PATZ1,0.017490650027270235
-34870,DRG1,0.06837787642992106
-34871,EIF4ENIF1,0.02477937544764609
-34873,SFI1,-0.14286195205153274
-34875,PISD,0.2557163183670793
-34876,PRR14L,0.10943801027169456
-34877,DEPDC5,0.04275627648709521
-34879,YWHAH,0.18145613099705868
-34897,RTCB,0.07955451471754496
-34899,FBXO7,0.0422517357976952
-34920,HMGXB4,-0.0032853277213624288
-34922,TOM1,0.20116106223473595
-34925,HMOX1,0.30079402166788577
-34926,MCM5,-0.06301776784496195
-34931,APOL6,0.26153584676681796
-34939,APOL3,0.13016533705986208
-34940,APOL4,0.12992234367059471
-34941,APOL2,0.038810707359676455
-34942,APOL1,0.14487920014457706
-34943,MYH9,0.03501868389781158
-34944,Z82215.1,-0.07533372383781967
-34947,TXN2,-0.04295879252795731
-34949,EIF3D,0.00017899378329054558
-34955,IFT27,-0.022022720138437757
-34960,NCF4,0.3414342641207019
-34961,CSF2RB,0.3656156815967857
-34964,TST,0.17530008349436776
-34965,MPST,0.052787038547516756
-34967,KCTD17,-0.03181301498046146
-34970,IL2RB,-0.5235493378666132
-34974,RAC2,-0.37444575811586617
-34975,CYTH4,0.12118768054539698
-34980,MFNG,-0.062061279669090426
-34983,CDC42EP1,0.13650087694743587
-34984,LGALS2,0.44657564059567123
-34985,GGA1,-0.03132754377667306
-34986,SH3BP1,0.16654537115068743
-34988,PDXP,-0.1559580098117067
-34989,LGALS1,0.3125401585419577
-34990,NOL12,-0.04414852858539877
-34991,TRIOBP,0.3053932088643465
-34992,H1F0,0.13294573930601894
-34995,ANKRD54,-0.047407925857035176
-34996,EIF3L,0.18397309254292488
-34997,AL022311.1,0.029473179255736632
-34998,MICALL1,0.10053302313463222
-35001,POLR2F,-0.036320940317953625
-35005,PICK1,-0.049055893842665836
-35012,MAFF,-0.07508208256574025
-35013,TMEM184B,-0.012953263947394685
-35015,CSNK1E,-0.06948379745121323
-35022,DDX17,0.1785408468710487
-35025,CBY1,-0.03772030919487548
-35029,TOMM22,-0.05088302908168821
-35030,JOSD1,-0.03246195693818418
-35031,GTPBP1,0.01950288647298539
-35033,SUN2,-0.1748310244891453
-35039,DNAL4,-0.08324533614606679
-35041,CBX6,0.07015719198924399
-35044,APOBEC3A,0.31063989864452374
-35045,APOBEC3B,0.12342624062116413
-35047,APOBEC3C,-0.12552448348236883
-35048,APOBEC3D,-0.11045041144446999
-35049,APOBEC3F,-0.03717198657235934
-35050,APOBEC3G,-0.4997838702541362
-35052,CBX7,-0.06453437621644571
-35053,AL031846.2,-0.12121640746869401
-35057,RPL3,-0.5027895159148994
-35058,SYNGR1,0.042924541502080873
-35059,TAB1,0.006597289956135167
-35062,MIEF1,0.06551103272794925
-35064,ATF4,0.09579595677467698
-35065,RPS19BP1,-0.1377509767466812
-35068,GRAP2,-0.157443998898656
-35069,Z82206.1,-0.14250922965145452
-35072,TNRC6B,-0.02213244277317093
-35073,ADSL,0.028174939555305624
-35074,SGSM3,-0.08941394420470837
-35076,MRTFA,0.09796961841700996
-35080,SLC25A17,-0.03969252581492708
-35081,ST13,-0.09525256505575476
-35082,XPNPEP3,0.008678279648643003
-35084,RBX1,0.03994061215670347
-35085,EP300,0.232690342841137
-35088,L3MBTL2,-0.006240436065619491
-35089,AL035681.1,-0.07618351560124804
-35091,RANGAP1,0.004256296499987717
-35092,ZC3H7B,-0.012277865365205764
-35095,TOB2,0.056105207369955515
-35096,PHF5A,-0.014325019681326387
-35097,ACO2,0.1309227376579434
-35098,POLR3H,-0.0415937900417423
-35101,PMM1,-0.06460240772220857
-35102,DESI1,0.12919273472483114
-35103,XRCC6,0.14275079681270397
-35104,SNU13,-0.10487517185833437
-35105,MEI1,-0.11022277117053646
-35108,SREBF2,0.007081169454938743
-35110,TNFRSF13C,0.0005674993646916973
-35116,NAGA,0.34540604780599343
-35119,SMDT1,-0.12886934661434335
-35120,NDUFA6,-0.09250754708542933
-35126,TCF20,0.0027897052570357538
-35129,NFAM1,0.37648274241820906
-35131,RRP7A,0.08896548615371004
-35133,POLDIP3,0.15081980953370627
-35134,Z93241.1,0.11733284397498123
-35135,CYB5R3,0.15776056634814667
-35138,ARFGAP3,0.02252758992613604
-35139,PACSIN2,0.12571392362249367
-35141,TTLL1,-0.03784201549247648
-35143,MCAT,-0.028381826943949192
-35144,TSPO,0.4797375783076793
-35145,TTLL12,0.007013329555074291
-35159,SAMM50,-0.014837100576021917
-35160,PARVB,0.14676275496751592
-35162,AL031595.3,0.1356353197848045
-35163,PARVG,0.2298317747247783
-35168,RTL6,-0.006903231264990642
-35171,PRR5,-0.4247655472445546
-35175,NUP50-DT,0.045353437698136996
-35177,NUP50,0.10279694332924239
-35179,KIAA0930,0.40347372708124074
-35181,UPK3A,0.12815711015451836
-35182,FAM118A,0.01230336896131335
-35188,Z95331.1,0.02212963588674078
-35189,ATXN10,0.010679840713482892
-35197,PRR34-AS1,0.02436944929859854
-35198,MIRLET7BHG,0.06469757262578363
-35201,PPARA,0.052659454007134676
-35205,TTC38,-0.27906951829876336
-35208,TRMU,-0.12681866920087778
-35211,GRAMD4,0.1124741467344477
-35212,CERK,0.0005538290483002792
-35213,AL118516.1,0.009267474037349094
-35214,TBC1D22A,0.12113940497606132
-35236,C22orf34,0.0023688943029304107
-35241,Z97192.3,0.02811934009564145
-35244,BRD1,-0.03267653809561524
-35248,ZBED4,0.012028153740499257
-35249,ALG12,0.017929424876171326
-35251,CRELD2,-0.1140985993112582
-35253,PIM3,0.07411715076900174
-35256,MLC1,-0.09215423596234038
-35260,TRABD,0.056056850313986224
-35263,SELENOO,0.12374654945856219
-35266,TUBGCP6,-0.034983665046279214
-35267,HDAC10,0.01265463364728645
-35270,PLXNB2,0.38316648031158246
-35271,DENND6B,0.16048979787495188
-35274,PPP6R2,-0.038890768285675476
-35275,SBF1,-0.05366552458875341
-35278,LMF2,0.004530179291987092
-35279,NCAPH2,-0.049695173497394524
-35280,SCO2,0.464504217552165
-35282,TYMP,0.654644856651401
-35283,ODF3B,0.37314140775711396
-35284,U62317.4,0.14530863597270208
-35286,KLHDC7B,0.11311599989114351
-35288,CPT1B,-0.0025186762173096456
-35289,CHKB,0.02226768580231088
-35290,CHKB-DT,-0.044169942562104
-35293,ARSA,0.029197008570256904
-35299,RABL2B,-0.01912856736955762
-35300,PLCXD1,0.04739973230233509
-35301,GTPBP6,-0.06227397230940443
-35302,LINC00685,-0.009111763706615835
-35303,PPP2R3B,0.005725740992937892
-35309,CSF2RA,0.30870945606331035
-35310,IL3RA,0.03442975780504273
-35311,SLC25A6,0.2131137867036301
-35314,ASMTL,-0.003488245228472566
-35315,P2RY8,-0.00494814440889657
-35316,AKAP17A,0.0054587450930288365
-35320,DHRSX,0.025972156811990908
-35322,ZBED1,0.07102916795915164
-35324,CD99,-0.363791018620875
-35328,ARSD,0.2545245520388918
-35335,PRKX,0.10401557941533845
-35337,BX890604.2,-0.14160721163239176
-35343,PUDP,0.13917303651162222
-35344,STS,0.11931544685043015
-35346,PNPLA4,-0.07524037381320169
-35357,TBL1X,0.17819895041463085
-35361,WWC3,0.0117059012809619
-35367,HCCS,0.11305556954872607
-35371,MSL3,0.14923043392492857
-35374,PRPS2,0.03654078981035567
-35375,TLR7,0.28604757052674357
-35377,TLR8,0.38973048183690434
-35378,TMSB4X,-0.06957524206187676
-35386,TCEANC,-0.020719414649394273
-35387,RAB9A,-0.13789682605243564
-35388,TRAPPC2,-0.17942833432599664
-35389,OFD1,-0.20979861172233394
-35390,GPM6B,-0.11545519326632082
-35393,GEMIN8,-0.05875381702389909
-35395,FANCB,-0.024422204397564748
-35396,MOSPD2,0.24399327125875944
-35399,PIGA,-0.001787303375573558
-35406,CA5B,-0.052354266850231354
-35408,ZRSR2,0.01711554705060783
-35409,AP1S2,0.5429509810913761
-35415,SYAP1,-0.08174724384253611
-35416,TXLNG,-0.13629449765474078
-35417,RBBP7,-0.10702185138500248
-35418,REPS2,0.12801302416796684
-35422,SCML1,-0.0019557353990805595
-35427,CDKL5,0.059973051989471435
-35432,PHKA2,0.08270300660627995
-35435,PDHA1,-0.01698568292137743
-35437,SH3KBP1,0.09335036396922089
-35438,BCLAF3,0.028574963609682553
-35440,EIF1AX,4.06957554152503e-05
-35442,RPS6KA3,0.11575617593947234
-35447,CNKSR2,-0.04100334447045688
-35450,MBTPS2,-0.0160192577508903
-35452,SMS,0.1045348530596026
-35461,PRDX4,-0.027077379292058487
-35462,ACOT9,0.20698625495415873
-35464,SAT1,0.5103736169292116
-35465,APOO,-0.052109225597550435
-35467,KLHL15,0.03917601367462345
-35468,EIF2S3,0.17119337473886836
-35470,ZFX,-0.03394199786247643
-35471,PDK3,0.06966308179039123
-35474,POLA1,0.009933551112192147
-35496,CXorf21,0.29831980850891304
-35497,GK,0.28497433820900525
-35501,TAB3,0.14582310704538423
-35522,CYBB,0.6601776060218929
-35523,DYNLT3,-0.07747422824204775
-35530,RPGR,0.02003579565126187
-35535,MID1IP1,0.21251537878726431
-35544,BCOR,0.09105695887487353
-35546,ATP6AP2,0.2466257559484598
-35549,CXorf38,0.169532408985871
-35550,MED14,-0.019185276924918757
-35553,USP9X,0.028871558200980932
-35555,DDX3X,0.2460736487093401
-35557,CASK,-0.013626098677877039
-35559,GPR34,0.06985276048395408
-35560,GPR82,-0.08867338227832014
-35572,FUNDC1,-0.0002343093397555715
-35574,KDM6A,0.06252159632678769
-35578,MIR222HG,0.06686493384925506
-35584,KRBOX4,-0.037258160331320754
-35585,ZNF674,-0.03583075151897401
-35587,CHST7,0.015638030688596604
-35588,SLC9A7,0.006461674913111695
-35589,RP2,0.33692592206941935
-35593,NDUFB11,-0.13113301661003054
-35594,RBM10,0.022928227172891984
-35595,UBA1,0.09198466838300111
-35597,CDK16,0.024497510826011485
-35598,USP11,-0.19739746048477377
-35601,ZNF41,-0.05270501229343836
-35603,ARAF,0.11756585414703227
-35605,TIMP1,0.21373810751940797
-35606,CFP,0.5352854131375986
-35607,ELK1,0.08749124488841512
-35608,UXT,-0.26786631686118817
-35611,ZNF182,0.02748663142957317
-35624,FTSJ1,-0.030343045264646866
-35626,PORCN,-0.024153258375088693
-35627,EBP,-0.24614702099531455
-35628,TBC1D25,0.0013610714673084874
-35630,RBM3,0.1541019198037292
-35632,WDR13,0.08204394348621619
-35633,WAS,0.27937805476487076
-35638,HDAC6,0.012882395973393747
-35640,PCSK1N,-0.06520732704228259
-35641,PQBP1,-0.04521604121096473
-35642,TIMM17B,-0.028352872160821482
-35643,SLC35A2,-0.07645557014553267
-35644,PIM2,-0.1675942567303808
-35645,OTUD5,0.007005504037236396
-35647,GRIPAP1,0.12622448470300082
-35648,TFE3,0.08024507290936034
-35650,PRAF2,-0.035933587050234105
-35651,WDR45,-0.015964861336888955
-35652,GPKOW,-0.045330383190864736
-35654,PLP2,0.1155199221396374
-35655,PRICKLE3,0.10862106794282611
-35659,CCDC22,0.037521487056724805
-35662,PPP1R3F,-0.09062941613007315
-35682,CLCN5,0.15685039598846437
-35699,GSPT2,-0.10184666957179156
-35700,MAGED1,-0.06634031268378532
-35720,TSPYL2,-0.046212342333704755
-35722,KANTR,-0.003407885320216794
-35723,KDM5C,0.022714394473556734
-35725,IQSEC2,0.06313429945063183
-35726,SMC1A,0.10739923487371174
-35728,HSD17B10,-0.05251045288137271
-35730,HUWE1,0.015805562436102275
-35731,PHF8,0.034776966285901
-35732,FAM120C,-0.044626117573182585
-35734,TSR2,-0.08761855112005228
-35736,GNL3L,0.11779939453886554
-35738,MAGED2,-0.1063667476696755
-35741,APEX2,0.0878386757276833
-35745,FAM104B,-0.0643164467178141
-35749,MAGEH1,-0.13791145188816412
-35752,RRAGB,-0.026364166447073716
-35753,AL050309.1,-0.021856650967682942
-35755,UBQLN2,-0.00987352469210277
-35757,NBDY,-0.1289183736531296
-35758,SPIN3,0.032501822323812976
-35760,SPIN2B,-0.10372450049232503
-35768,LINC01278,-0.1311666968034566
-35771,ARHGEF9,-0.14202121757316624
-35779,LAS1L,-0.03172493610750527
-35780,MSN,0.22291230751390131
-35782,AL034397.3,0.3060337767602539
-35789,YIPF6,-0.03544949587444724
-35791,EFNB1,0.08438312114422793
-35792,PJA1,-0.021070127297787095
-35800,IGBP1,-0.045389126345570095
-35808,PDZD11,-0.11477123503027983
-35816,SNX12,0.040342273909205494
-35817,FOXO4,0.030404931673340647
-35819,IL2RG,-0.5077542020478681
-35820,MED12,-0.005006011637045277
-35822,AL590764.1,0.009431898118805059
-35824,ZMYM3,0.07056598940234207
-35825,NONO,0.1253530348127399
-35827,TAF1,0.030856203881058756
-35828,OGT,0.048449125897759224
-35829,GCNA,-0.11341479580639711
-35830,CXCR3,-0.12573867768282787
-35835,NHSL2,0.2609505623888201
-35837,PIN4,-0.13237513534700698
-35839,RPS4X,-0.4376301958725731
-35841,HDAC8,0.09666948948917642
-35857,CHIC1,-0.10557901227431997
-35858,TSIX,-0.07540962765801186
-35859,XIST,0.174946738088447
-35860,FTX,0.059693640723665585
-35861,JPX,-0.13573237602675284
-35864,Z83843.1,0.11636433620034156
-35867,RLIM,0.06161774328142752
-35869,ABCB7,0.019509755977693596
-35870,UPRT,0.041498282272999495
-35874,PBDC1,-0.2255659140419986
-35878,ATRX,-0.015445512176775564
-35879,MAGT1,0.10818998124882202
-35880,COX7B,-0.05028048921684737
-35881,ATP7A,-0.03803392386551635
-35882,PGK1,0.3332238142324831
-35884,TAF9B,0.030060716947849713
-35885,CYSLTR1,0.15144012989799277
-35888,P2RY10,-0.11293449921641152
-35889,GPR174,-0.3727144693705558
-35890,ITM2A,-0.30672077789057667
-35893,BRWD3,0.06321570231388744
-35896,SH3BGRL,0.331660521726671
-35901,HDX,0.08082825223085172
-35902,APOOL,0.09095632116282656
-35907,CHM,0.01941754033287935
-35921,DIAPH2,0.33562947766696766
-35930,CSTF2,0.05622232372631561
-35934,TRMT2B,-0.04937554214112421
-35940,TIMM8A,-0.03262026151701962
-35941,BTK,0.34401329286418225
-35942,RPL36A,-0.2761376645015826
-35943,GLA,0.13836071925241783
-35944,HNRNPH2,0.200119853182454
-35945,ARMCX4,-0.0073516203939472235
-35947,ARMCX6,-0.010860430555805272
-35948,ARMCX3,0.03102273190840908
-35954,ZMAT1,-0.004761958782683418
-35958,BEX5,-0.12834870858082917
-35964,ARMCX5,0.0301854412697916
-35965,ARMCX5-GPRASP2,0.038898341691745296
-35967,GPRASP1,-0.050031136784660035
-35979,BEX4,-0.15799599734855693
-35980,TCEAL8,-0.14517935604916143
-35982,BEX2,-0.1282908085447422
-35985,BEX3,0.03492436998229401
-35990,TCEAL4,0.03352949242201337
-35991,TCEAL3,-0.08250287303159938
-35993,TCEAL1,-0.037761926052411014
-35994,MORF4L2,-0.008536726738135022
-36005,SLC25A53,-0.0863914771962448
-36007,FAM199X,-0.07002712421046318
-36017,MORC4,-0.06982116096218996
-36020,RBM41,0.040183043757895665
-36025,PRPS1,-0.18768163478408906
-36026,TSC22D3,0.25326400008697164
-36031,VSIG1,-0.06652667519447146
-36032,PSMD10,0.15412583930418505
-36033,ATG4A,0.023625181133854006
-36040,NXT2,-0.031727938277939446
-36042,ACSL4,0.29406624916485036
-36043,TMEM164,0.15986708490450832
-36044,AMMECR1,-0.0910681051750612
-36053,ALG13,0.002476471548591849
-36077,WDR44,0.062352190195953804
-36078,DOCK11,0.06136977954593533
-36079,IL13RA1,0.3823073616449476
-36085,PGRMC1,0.049596349580456445
-36088,SLC25A43,0.02207063955248995
-36091,SLC25A5,0.16612043302854604
-36092,CXorf56,0.057296842772735605
-36093,UBE2A,0.06159219101781632
-36094,NKRF,-0.006874845784467984
-36095,SEPTIN6,-0.25638900479868143
-36096,SOWAHD,0.1319643260093204
-36097,RPL39,-0.27385701787901545
-36099,UPF3B,-0.13088353355680798
-36100,RNF113A,-0.09225850380948132
-36101,NDUFA1,-0.057825045454590676
-36103,NKAP,-0.04320457991071741
-36109,ZBTB33,-0.08851770053637799
-36112,LAMP2,0.32024694900053174
-36113,CUL4B,0.056464585647043965
-36114,MCTS1,-0.017514411735311097
-36115,C1GALT1C1,0.008658660622103583
-36136,THOC2,0.019688789326329923
-36137,XIAP,0.09113457794874814
-36140,STAG2,0.1305546160035711
-36141,SH2D1A,-0.3403161278438673
-36143,TENM1,-0.08746769217519429
-36153,OCRL,0.07658429817986681
-36157,SASH3,0.003189174780768219
-36161,UTP14A,-0.011488068165585437
-36162,BCORL1,0.08681316702000823
-36163,ELF4,0.14316260133938186
-36164,AIFM1,0.0031237928636960325
-36166,RAB33A,-0.19454146636271746
-36167,ZNF280C,-0.021493206865591606
-36168,SLC25A14,-0.003845846028703127
-36170,RBMX2,-0.0012407087083213565
-36172,ENOX2,0.039022792745724365
-36180,STK26,-0.1116239780819637
-36182,RAP2C,0.09417722487420792
-36183,RAP2C-AS1,0.04347425081350844
-36184,MBNL3,0.05991826385066163
-36193,PHF6,-0.016663114111778007
-36194,HPRT1,-0.07667093613078446
-36199,FAM122B,-0.04107829583340085
-36200,FAM122C,0.005850004781467156
-36201,MOSPD1,-0.09066729550396772
-36205,RTL8C,0.12550992460052116
-36207,RTL8A,0.015642948957280925
-36213,ZNF75D,0.03909597802180093
-36217,ZNF449,-0.023512611423003735
-36220,INTS6L,-0.01308934913563798
-36231,MMGT1,-0.04205020146914362
-36232,SLC9A6,0.03914484612593535
-36233,FHL1,0.013520470929521052
-36234,MAP7D3,-0.0861421641434301
-36237,HTATSF1,-0.048470512650640785
-36239,LINC00892,-0.14384545740845223
-36240,CD40LG,-0.11488578968703432
-36241,ARHGEF6,0.07775940449308251
-36242,AL683813.1,0.04230004856113724
-36243,RBMX,-0.031173288768002728
-36253,ATP11C,0.052980882315783795
-36277,SLITRK4,0.08993152092297647
-36291,FMR1,0.09891100427356428
-36298,IDS,0.03484956175376978
-36300,LINC00893,-0.10179336937252985
-36301,CXorf40A,0.01535874959460581
-36305,TMEM185A,-0.022860665732262885
-36312,CXorf40B,-0.022939904398138183
-36314,LINC00894,0.018290843775173163
-36316,MTM1,0.11509877311116452
-36317,MTMR1,0.017337791296368355
-36318,CD99L2,-0.024007168517867144
-36325,VMA21,0.0529273311055695
-36350,CETN2,-0.05046666227092573
-36351,NSDHL,-0.005385879399780633
-36352,ZNF185,0.1137903162455039
-36361,ZNF275,0.002195621867406697
-36365,HAUS7,-0.010464602860764428
-36368,CCNQ,-0.0570211054934531
-36372,BCAP31,0.001367519703077213
-36373,ABCD1,0.17498489889346436
-36377,IDH3G,0.04507735227470654
-36378,SSR4,-0.16424921207518037
-36383,ARHGAP4,0.12136591932111544
-36384,NAA10,-0.0625297386053286
-36385,RENBP,0.18343726964268253
-36386,HCFC1,0.15730041968229688
-36388,TMEM187,-0.12549968792736327
-36389,IRAK1,0.17801139437453892
-36390,MECP2,-0.1254045089825172
-36398,FLNA,0.17306855793601833
-36400,EMD,-0.11574939524629656
-36401,RPL10,-0.45345748330014896
-36402,AC245140.2,-0.021671330694357407
-36403,DNASE1L1,0.20590733741384087
-36404,TAZ,0.022704676160825468
-36405,AC244090.1,-0.018072116965695273
-36406,ATP6AP1,0.2245343975713343
-36407,GDI1,-0.007709860825332988
-36408,FAM50A,0.17247138707019877
-36409,PLXNA3,-0.0472701186202171
-36410,LAGE3,-0.10557754631661923
-36411,UBL4A,-0.07173422511071681
-36412,SLC10A3,0.010611353886525432
-36413,FAM3A,-0.10641377360493236
-36415,G6PD,0.15388172775844292
-36416,IKBKG,0.026137575590987654
-36422,GAB3,0.004893896016511426
-36423,DKC1,-0.039806249494750326
-36424,MPP1,0.17538813677227022
-36428,F8A1,-0.01250896017211151
-36429,FUNDC2,-0.05390832676648117
-36430,CMC4,-0.1268850337267403
-36432,BRCC3,-0.04396885725839461
-36433,VBP1,-0.1412414577478851
-36434,RAB39B,-0.05765490943971836
-36435,CLIC2,0.09520302947566933
-36443,TMLHE,0.11707919502954454
-36445,VAMP7,0.12476331483037964
-36559,MT-ND1,0.3683588058016804
-36560,MT-ND2,0.3157932653637021
-36561,MT-CO1,0.28779693192577344
-36562,MT-CO2,0.05264224281556177
-36563,MT-ATP8,0.12261611803814773
-36564,MT-ATP6,-0.1649120810326163
-36565,MT-CO3,0.05734669419739229
-36566,MT-ND3,0.1041293995087942
-36567,MT-ND4L,0.18812367227530583
-36568,MT-ND4,0.21306351177526348
-36569,MT-ND5,-0.07216012020140243
-36570,MT-ND6,-0.18792858240812466
-36571,MT-CYB,-0.08945658334898675
-36580,AL592183.1,0.019248635055742164
-36581,AC240274.1,-0.0007150145677195353
-36584,AC004556.3,0.0263942054157376
diff --git a/scarf/tests/datasets/ptime_modules_group_1.npy b/scarf/tests/datasets/ptime_modules_group_1.npy
deleted file mode 100644
index 39f6913e..00000000
Binary files a/scarf/tests/datasets/ptime_modules_group_1.npy and /dev/null differ
diff --git a/scarf/tests/datasets/sympathetic.loom b/scarf/tests/datasets/sympathetic.loom
deleted file mode 100644
index f1dc5119..00000000
Binary files a/scarf/tests/datasets/sympathetic.loom and /dev/null differ
diff --git a/scarf/tests/datasets/toy_cr_dir.tar.gz b/scarf/tests/datasets/toy_cr_dir.tar.gz
deleted file mode 100644
index fe82c753..00000000
Binary files a/scarf/tests/datasets/toy_cr_dir.tar.gz and /dev/null differ
diff --git a/scarf/tests/datasets/toy_cr_dir_empty.tar.gz b/scarf/tests/datasets/toy_cr_dir_empty.tar.gz
deleted file mode 100644
index 26e59c8b..00000000
Binary files a/scarf/tests/datasets/toy_cr_dir_empty.tar.gz and /dev/null differ
diff --git a/scarf/tests/datasets/unified_UMAP_coords.npy b/scarf/tests/datasets/unified_UMAP_coords.npy
deleted file mode 100644
index 7ae4ba56..00000000
Binary files a/scarf/tests/datasets/unified_UMAP_coords.npy and /dev/null differ
diff --git a/scarf/tests/fixtures_datastore.py b/scarf/tests/fixtures_datastore.py
deleted file mode 100644
index 2f4111d4..00000000
--- a/scarf/tests/fixtures_datastore.py
+++ /dev/null
@@ -1,217 +0,0 @@
-import numpy as np
-import pandas as pd
-import pytest
-
-from . import full_path, remove
-
-
-@pytest.fixture(scope="session")
-def toy_crdir_writer(toy_crdir_reader):
- from ..writers import CrToZarr
-
- out_fn = full_path("toy_crdir.zarr")
-
- writer = CrToZarr(toy_crdir_reader, out_fn)
- writer.dump()
- yield out_fn
- remove(out_fn)
-
-
-@pytest.fixture(scope="session")
-def toy_crdir_ds(toy_crdir_writer):
- from ..datastore.datastore import DataStore
-
- yield DataStore(toy_crdir_writer, default_assay="RNA")
-
-
-@pytest.fixture(scope="session")
-def datastore():
- import tarfile
-
- from ..datastore.datastore import DataStore
-
- fn = full_path("1K_pbmc_citeseq.zarr.tar.gz")
- out_fn = fn.replace(".tar.gz", "")
- remove(out_fn)
- tar = tarfile.open(fn, "r:gz")
- tar.extractall(out_fn)
- yield DataStore(out_fn, default_assay="RNA")
- remove(out_fn)
-
-
-@pytest.fixture
-def datastore_ephemeral():
- import tarfile
-
- from ..datastore.datastore import DataStore
-
- fn = full_path("1K_pbmc_citeseq.zarr.tar.gz")
- out_fn = fn.replace(".tar.gz", "")
- remove(out_fn)
- tar = tarfile.open(fn, "r:gz")
- tar.extractall(out_fn)
- yield DataStore(out_fn, default_assay="RNA")
- remove(out_fn)
-
-
-@pytest.fixture(scope="class")
-def auto_filter_cells(datastore):
- datastore.auto_filter_cells(show_qc_plots=False)
-
-
-@pytest.fixture(scope="class")
-def mark_hvgs(auto_filter_cells, datastore):
- datastore.mark_hvgs(top_n=100, show_plot=False)
-
-
-@pytest.fixture(scope="class")
-def make_graph(mark_hvgs, datastore):
- datastore.make_graph(feat_key="hvgs")
- graph_loc = datastore._get_latest_graph_loc(
- from_assay="RNA", cell_key="I", feat_key="hvgs"
- )
- yield graph_loc.rsplit("/", 1)[0]
-
-
-@pytest.fixture(scope="class")
-def leiden_clustering(make_graph, datastore):
- datastore.run_leiden_clustering()
- yield datastore.cells.fetch("RNA_leiden_cluster")
-
-
-@pytest.fixture(scope="class")
-def paris_clustering(make_graph, datastore):
- datastore.run_clustering(n_clusters=10)
- yield datastore.cells.fetch("RNA_cluster")
-
-
-@pytest.fixture(scope="class")
-def paris_clustering_balanced(make_graph, datastore):
- datastore.run_clustering(
- balanced_cut=True, max_size=100, min_size=10, label="balanced_clusters"
- )
- yield datastore.cells.fetch("RNA_balanced_clusters")
-
-
-@pytest.fixture(scope="class")
-def umap(make_graph, datastore):
- datastore.run_umap(n_epochs=50)
- yield np.array(
- [datastore.cells.fetch("RNA_UMAP1"), datastore.cells.fetch("RNA_UMAP2")]
- ).T
-
-
-@pytest.fixture(scope="class")
-def marker_search(datastore):
- # Testing this with Paris clusters rather than Leiden clusters because of reproducibility.
- datastore.run_marker_search(group_key="RNA_cluster")
-
-
-@pytest.fixture(scope="class")
-def pseudotime_scoring(datastore, leiden_clustering):
- datastore.run_pseudotime_scoring(
- source_sink_key="RNA_leiden_cluster", sources=[6], sinks=[3]
- )
- yield datastore.cells.fetch("RNA_pseudotime")
-
-
-@pytest.fixture(scope="class")
-def pseudotime_markers(datastore, pseudotime_scoring):
- datastore.run_pseudotime_marker_search(pseudotime_key="RNA_pseudotime")
- df = datastore.RNA.feats.to_pandas_dataframe(
- ["names", "I__RNA_pseudotime__r"], key="I"
- )
- yield df
-
-
-@pytest.fixture(scope="class")
-def pseudotime_aggregation(datastore, pseudotime_scoring):
- datastore.run_pseudotime_aggregation(
- pseudotime_key="RNA_pseudotime",
- cluster_label="pseudotime_clusters",
- n_clusters=15,
- window_size=50,
- chunk_size=10,
- )
-
-
-@pytest.fixture(scope="class")
-def grouped_assay(datastore, pseudotime_aggregation):
- datastore.add_grouped_assay(
- group_key="pseudotime_clusters", assay_label="PTIME_MODULES"
- )
-
-
-@pytest.fixture(scope="class")
-def run_mapping(make_graph, datastore):
- datastore.run_mapping(
- target_assay=datastore.RNA,
- target_name="selfmap",
- target_feat_key="hvgs_self",
- save_k=3,
- )
-
-
-@pytest.fixture(scope="class")
-def run_mapping_coral(make_graph, datastore):
- datastore.run_mapping(
- target_assay=datastore.RNA,
- target_name="selfmap_coral",
- target_feat_key="hvgs_self2",
- save_k=3,
- run_coral=True,
- )
-
-
-@pytest.fixture(scope="class")
-def run_unified_umap(run_mapping, datastore):
- datastore.run_unified_umap(target_names=["selfmap"])
-
-
-@pytest.fixture(scope="class")
-def cell_cycle_scoring(datastore):
- datastore.run_cell_cycle_scoring()
- return datastore.cells.fetch("RNA_cell_cycle_phase")
-
-
-@pytest.fixture(scope="class")
-def topacedo_sampler(paris_clustering, datastore):
- try:
- import topacedo
- except ImportError:
- pytest.skip("topacedo package not installed")
- datastore.run_topacedo_sampler(cluster_key="RNA_cluster")
- return datastore.cells.fetch("RNA_sketched")
-
-
-@pytest.fixture(scope="class")
-def cell_attrs():
- return pd.read_csv(full_path("cell_attributes.csv"), index_col=0)
-
-
-@pytest.fixture(scope="session")
-def atac_datastore():
- from ..datastore.datastore import DataStore
- import tarfile
-
- fn = full_path("500_pbmc_atac.zarr.tar.gz")
- out_fn = fn.replace(".tar.gz", "")
- remove(out_fn)
- tar = tarfile.open(fn, "r:gz")
- tar.extractall(out_fn)
- yield DataStore(out_fn)
- remove(out_fn)
-
-
-@pytest.fixture(scope="class")
-def mark_prevalent_peaks(atac_datastore):
- atac_datastore.mark_prevalent_peaks(top_n=5000)
-
-
-@pytest.fixture(scope="class")
-def make_atac_graph(mark_prevalent_peaks, atac_datastore):
- atac_datastore.make_graph(feat_key="prevalent_peaks")
- graph_loc = atac_datastore._get_latest_graph_loc(
- from_assay="ATAC", cell_key="I", feat_key="prevalent_peaks"
- )
- yield graph_loc.rsplit("/", 1)[0]
diff --git a/scarf/tests/fixtures_downloader.py b/scarf/tests/fixtures_downloader.py
deleted file mode 100644
index 63428b4a..00000000
--- a/scarf/tests/fixtures_downloader.py
+++ /dev/null
@@ -1,13 +0,0 @@
-import pytest
-
-from . import full_path, remove
-
-
-@pytest.fixture(scope="session")
-def bastidas_ponce_data():
- from ..downloader import fetch_dataset
-
- sample = "bastidas-ponce_4K_pancreas-d15_rnaseq"
- fetch_dataset(sample, full_path(None))
- yield full_path(sample, "data.h5ad")
- remove(full_path(sample))
diff --git a/scarf/tests/fixtures_readers.py b/scarf/tests/fixtures_readers.py
deleted file mode 100644
index db8cdc8a..00000000
--- a/scarf/tests/fixtures_readers.py
+++ /dev/null
@@ -1,79 +0,0 @@
-import pytest
-
-from . import full_path, remove
-
-
-@pytest.fixture(scope="session")
-def toy_crdir_reader():
- from ..readers import CrDirReader
- import tarfile
-
- fn = full_path("toy_cr_dir.tar.gz")
- out_fn = fn.replace(".tar.gz", "")
- remove(out_fn)
- tar = tarfile.open(fn, "r:gz")
- tar.extractall(out_fn)
- reader = CrDirReader(out_fn)
- reader.rename_assays({"ASSAY4": "HTO"})
- yield reader
- remove(out_fn)
-
-@pytest.fixture(scope="session")
-def toy_crdir_empty():
- from ..readers import CrDirReader
- import tarfile
-
- fn = full_path("toy_cr_dir_empty.tar.gz")
- out_fn = fn.replace(".tar.gz", "")
- remove(out_fn)
- tar = tarfile.open(fn, "r:gz")
- tar.extractall(out_fn)
- reader = CrDirReader(out_fn)
- yield reader
- remove(out_fn)
-
-@pytest.fixture(scope="session")
-def crh5_reader():
- from ..readers import CrH5Reader
-
- return CrH5Reader(full_path("1K_pbmc_citeseq.h5"))
-
-
-@pytest.fixture(scope="session")
-def mtx_dir(datastore):
- from ..writers import to_mtx
-
- fn = full_path("1K_pbmc_citeseq_dir")
- to_mtx(datastore.RNA, fn, compress=True)
- yield fn
- remove(fn)
-
-
-@pytest.fixture(scope="session")
-def crdir_reader(datastore, mtx_dir):
- from ..readers import CrDirReader
-
- reader = CrDirReader(mtx_dir)
- yield reader
-
-
-@pytest.fixture(scope="session")
-def h5ad_reader(bastidas_ponce_data):
- from ..readers import H5adReader
-
- reader = H5adReader(bastidas_ponce_data)
- yield reader
- reader.h5.close()
-
-
-@pytest.fixture(scope="session")
-def loom_reader():
- from ..readers import LoomReader
-
- reader = LoomReader(
- full_path("sympathetic.loom"),
- cell_names_key="Cell_id",
- feature_names_key="Gene",
- )
- yield reader
- reader.h5.close()
diff --git a/scarf/tests/requirements.txt b/scarf/tests/requirements.txt
deleted file mode 100644
index 1e824230..00000000
--- a/scarf/tests/requirements.txt
+++ /dev/null
@@ -1,6 +0,0 @@
-pytest
-pytest-cov
-topacedo
-anndata==0.11.4
-black
-scalene
diff --git a/scarf/tests/test_datastore.py b/scarf/tests/test_datastore.py
deleted file mode 100644
index 3e3d4769..00000000
--- a/scarf/tests/test_datastore.py
+++ /dev/null
@@ -1,279 +0,0 @@
-import numpy as np
-import pandas as pd
-
-from . import full_path, remove
-
-
-class TestToyDataStore:
- def test_toy_crdir_metadata(self, toy_crdir_ds):
- assert np.all(
- toy_crdir_ds.RNA.feats.fetch_all("ids") == ["g1", "g2", "g3", "g4"]
- )
- assert np.all(toy_crdir_ds.ADT.feats.fetch_all("ids") == ["a1", "a2"])
- assert np.all(toy_crdir_ds.HTO.feats.fetch_all("ids") == ["h1"])
- assert np.all(toy_crdir_ds.cells.fetch_all("ids") == ["b1", "b2", "b3"])
-
- def test_toy_crdir_rawdata(self, toy_crdir_ds):
- assert np.all(
- toy_crdir_ds.RNA.rawData.compute()
- == [[5, 0, 0, 2], [3, 3, 0, 7], [3, 3, 0, 7]]
- )
- assert np.all(
- toy_crdir_ds.ADT.rawData.compute() == [[30, 40], [30, 50], [0, 50]]
- )
- assert np.all(toy_crdir_ds.HTO.rawData.compute() == [[200], [100], [100]])
-
-
-class TestDataStore:
- def test_init_wrong_zarr_mode(self):
- import pytest
- import tarfile
-
- from ..datastore.datastore import DataStore
-
- fn = full_path("1K_pbmc_citeseq.zarr.tar.gz")
- out_fn = fn.replace(".tar.gz", "")
- remove(out_fn)
- tar = tarfile.open(fn, "r:gz")
- tar.extractall(out_fn)
- with pytest.raises(ValueError):
- ds = DataStore(out_fn, zarr_mode="wrong", default_assay="RNA")
- remove(out_fn)
-
- def test_auto_filter_cells(self, datastore_ephemeral):
- assert (
- datastore_ephemeral.auto_filter_cells(
- attrs=["nCounts", "nFeatures", "non_existing_column"],
- show_qc_plots=False,
- )
- is None
- )
- # show_qc_plots=True
- # howto test plots?
-
- def test_filter_cells(self, datastore_ephemeral):
- assert (
- datastore_ephemeral.filter_cells(
- attrs=["nCounts", "nFeatures", "non_existing_column"],
- lows=[None, None, None],
- highs=[None, None, None],
- reset_previous=True,
- )
- is None
- )
- # still doesn't access `if j is None:` cases for j and k
-
- def test_graph_indices(self, make_graph, datastore):
- a = np.load(full_path("knn_indices.npy"))
- b = datastore.z[make_graph]["indices"][:]
- assert np.array_equal(a, b)
-
- def test_graph_distances(self, make_graph, datastore):
- a = np.load(full_path("knn_distances.npy"))
- b = datastore.z[make_graph]["distances"][:]
- assert np.all((a - b) < 1e-3)
-
- def test_graph_weights(self, make_graph, datastore):
- a = np.load(full_path("knn_weights.npy"))
- b = datastore.z[make_graph]["graph__1.0__1.5"]["weights"][:]
- assert np.all((a - b) < 1e-5)
-
- def test_atac_graph_indices(self, make_atac_graph, atac_datastore):
- a = np.load(full_path("atac_knn_indices.npy"))
- b = atac_datastore.z[make_atac_graph]["indices"][:]
- assert a.shape == b.shape
-
- # TODO: activate this when this PR is merged and released in gensim
- # https://github.com/RaRe-Technologies/gensim/pull/3194
- # assert np.array_equal(a, b)
-
- def test_atac_graph_distances(self, make_atac_graph, atac_datastore):
- a = np.load(full_path("atac_knn_distances.npy"))
- b = atac_datastore.z[make_atac_graph]["distances"][:]
- assert a.shape == b.shape
-
- # TODO: activate this when this PR is merged and released in gensim
- # https://github.com/RaRe-Technologies/gensim/pull/3194
- # assert np.all((a - b) < 1e-5)
-
- def test_leiden_values(self, leiden_clustering, cell_attrs):
- assert len(set(leiden_clustering)) == 10
- # Disabled the following test because failing on CI
- # assert np.array_equal(leiden_clustering, cell_attrs['RNA_leiden_cluster'].values)
-
- def test_paris_values(self, paris_clustering, cell_attrs):
- assert np.array_equal(paris_clustering, cell_attrs["RNA_cluster"].values)
-
- def test_paris_balanced_values(self, paris_clustering_balanced, cell_attrs):
- assert np.array_equal(
- paris_clustering_balanced, cell_attrs["RNA_balanced_clusters"].values
- )
-
- def test_run_cell_cycle_scoring(self, cell_cycle_scoring, cell_attrs):
- assert np.array_equal(
- cell_cycle_scoring, cell_attrs["RNA_cell_cycle_phase"].values
- )
-
- def test_umap_values(self, umap, cell_attrs):
- precalc_umap = cell_attrs[["RNA_UMAP1", "RNA_UMAP2"]].values
- assert umap.shape == precalc_umap.shape
- # Disabled the following test because failing on CI
- # assert np.all((umap - precalc_umap) < 0.1)
-
- def test_get_markers(self, marker_search, paris_clustering, datastore):
- precalc_markers = pd.read_csv(full_path("markers_cluster1.csv"), index_col=0)
- markers = datastore.get_markers(group_key="RNA_cluster", group_id=1)
-
- # Check feature names and scores (always required)
- assert markers.feature_name.equals(precalc_markers.feature_name)
- diff = (markers.score - precalc_markers.score).values
- assert np.all(diff < 1e-3)
-
- # Check p_values only if they exist in reference data (backward compatible)
- if 'p_value' in precalc_markers.columns:
- assert 'p_value' in markers.columns, "p_value column missing in output"
- # P-values should match within reasonable tolerance
- p_diff = (markers.p_value - precalc_markers.p_value).values
- assert np.all(np.abs(p_diff) < 1e-3), "p_values differ from reference"
-
- def test_export_markers_to_csv(self, marker_search, paris_clustering, datastore):
- precalc_markers = pd.read_csv(full_path("markers_all_clusters.csv"))
- out_file = full_path("test_values_markers.csv")
- datastore.export_markers_to_csv(group_key="RNA_cluster", csv_filename=out_file)
- markers = pd.read_csv(out_file)
- assert markers.equals(precalc_markers)
- remove(out_file)
-
- def test_run_unified_umap(self, run_unified_umap, datastore):
- coords = datastore.z["RNA"].projections["unified_UMAP"][:]
- precalc_coords = np.load(full_path("unified_UMAP_coords.npy"))
- assert coords.shape == precalc_coords.shape
-
- def test_get_target_classes(
- self, run_mapping, paris_clustering, cell_attrs, datastore
- ):
- classes = datastore.get_target_classes(
- target_name="selfmap", reference_class_group="RNA_cluster"
- )
- assert np.array_equal(classes.values, cell_attrs["target_classes"].values)
-
- def test_get_mapping_score(self, run_mapping, cell_attrs, datastore):
- scores = next(datastore.get_mapping_score(target_name="selfmap"))[1]
- diff = scores - cell_attrs["mapping_scores"].values
- assert np.all(diff < 1e-2)
-
- def test_coral_mapping_score(self, run_mapping_coral, cell_attrs, datastore):
- # TODO: add test values for coral
- assert 1 == 1
-
- def test_repr(self, datastore):
- # TODO: Test if the expected values are printed
- print(datastore)
-
- def test_get_imputed(self, datastore):
- # TODO: Test the output values
- values = datastore.get_imputed(feature_name="CD4")
- assert values.shape == datastore.cells.fetch("I").shape
-
- def test_run_pseudotime_scoring(self, pseudotime_scoring, cell_attrs):
- diff = pseudotime_scoring - cell_attrs["RNA_pseudotime"].values
- assert np.all(diff < 1e-3)
-
- def test_run_pseudotime_marker_search(self, pseudotime_markers):
- precalc_markers = pd.read_csv(
- full_path("pseudotime_markers_r_values.csv"), index_col=0
- )
- assert np.all(precalc_markers.index == pseudotime_markers.index)
- assert np.all(precalc_markers.names.values == pseudotime_markers.names.values)
- assert np.allclose(
- precalc_markers.I__RNA_pseudotime__r.values,
- pseudotime_markers.I__RNA_pseudotime__r.values,
- )
-
- def test_run_pseudotime_aggregation(self, pseudotime_aggregation, datastore):
- precalc_values = np.load(full_path("aggregated_feat_idx.npy"))
- test_values = datastore.z.RNA.aggregated_I_I_RNA_pseudotime.feature_indices[:]
- assert np.array_equal(precalc_values.astype(np.int64), test_values.astype(np.int64))
-
- precalc_values = np.load(full_path("aggregated_df_top_10.npy"))
- test_values = datastore.z.RNA.aggregated_I_I_RNA_pseudotime.data[:10]
- assert np.all(precalc_values == test_values)
-
- precalc_values = np.load(full_path("pseudotime_clusters.npy"))
- test_values = datastore.RNA.feats.fetch_all("pseudotime_clusters")
- assert np.all(precalc_values == test_values)
-
- def test_add_grouped_assay(self, grouped_assay, datastore):
- precalc_values = np.load(full_path("ptime_modules_group_1.npy"))
- test_values = datastore.get_cell_vals(
- from_assay="PTIME_MODULES", cell_key="I", k="group_1"
- )
- assert np.allclose(precalc_values, test_values)
-
- def test_make_bulk(self, leiden_clustering, datastore):
- df = datastore.make_bulk(group_key="RNA_leiden_cluster")
- assert df.shape == (18850, 10)
- assert hash(tuple((df.values.flatten()))) == -1872129810056415572
-
- def test_to_anndata(self, datastore):
- # TODO: Check if all the attributes copied to anndata
- datastore.to_anndata()
-
- def test_run_topacedo_sampler(self, cell_attrs, topacedo_sampler):
- assert np.all(topacedo_sampler == cell_attrs["RNA_sketched"])
-
- def test_plot_cells_dists(self, datastore):
- datastore.plot_cells_dists(show_fig=False)
-
- def test_plot_layout(self, umap, paris_clustering, datastore):
- datastore.plot_layout(
- layout_key="RNA_UMAP", color_by="RNA_cluster", show_fig=False
- )
-
- # def test_plot_layout_shade(self, umap, paris_clustering, datastore):
- # datastore.plot_layout(
- # layout_key="RNA_UMAP",
- # color_by="RNA_cluster",
- # show_fig=False,
- # do_shading=True,
- # )
-
- def test_plot_cluster_tree(self, datastore):
- datastore.plot_cluster_tree(cluster_key="RNA_cluster", show_fig=False)
-
- def test_plot_marker_heatmap(self, marker_search, datastore):
- datastore.plot_marker_heatmap(group_key="RNA_cluster", show_fig=False)
-
- def test_plot_unified_layout(self, run_unified_umap, datastore):
- datastore.plot_unified_layout(layout_key="unified_UMAP", show_fig=False)
-
- def test_plot_pseudotime_heatmap(self, pseudotime_aggregation, datastore):
- datastore.plot_pseudotime_heatmap(
- cell_key="I",
- feat_key="I",
- feature_cluster_key="pseudotime_clusters",
- pseudotime_key="RNA_pseudotime",
- show_features=["Wsb1", "Rest"],
- show_fig=False,
- )
-
- def test_mark_hvgs_with_atac_assay(self, atac_datastore):
- import pytest
-
- with pytest.raises(TypeError):
- atac_datastore.mark_hvgs()
-
- def test_mark_prevalent_peaks_with_rna_assay(self, datastore):
- import pytest
-
- with pytest.raises(TypeError):
- datastore.mark_prevalent_peaks()
-
- def test_run_marker_search_with_no_groupkey(self, datastore):
- import pytest
-
- with pytest.raises(ValueError):
- datastore.run_marker_search(group_key=None)
-
- def test_run_marker_search_with_cellkey(self, datastore, paris_clustering):
- datastore.run_marker_search(group_key="RNA_cluster", cell_key="I")
diff --git a/scarf/tests/test_downloader.py b/scarf/tests/test_downloader.py
deleted file mode 100644
index a1192073..00000000
--- a/scarf/tests/test_downloader.py
+++ /dev/null
@@ -1,31 +0,0 @@
-from . import full_path, remove
-
-
-def test_downloader():
- from ..downloader import osfd, OSFdownloader
-
- if osfd is None:
- osfd = OSFdownloader("zeupv")
- # Making sure that there are at least 15 datasets in the list
- assert len(osfd.datasets) > 15
-
-
-def test_show_available_datasets():
- from ..downloader import show_available_datasets
-
- show_available_datasets()
-
-
-def test_fetch_dataset(bastidas_ponce_data):
- # TODO: check if correct file was downloaded.
- pass
-
-
-def test_downloader_as_zarr():
- from ..downloader import fetch_dataset
-
- # TODO: check if only the zarr file was downloaded
-
- sample = "tenx_5K_pbmc_rnaseq"
- fetch_dataset(sample, as_zarr=True, save_path=full_path(None))
- remove(full_path(sample))
diff --git a/scarf/tests/test_merger.py b/scarf/tests/test_merger.py
deleted file mode 100644
index e85ce1d3..00000000
--- a/scarf/tests/test_merger.py
+++ /dev/null
@@ -1,112 +0,0 @@
-import zarr
-
-from . import full_path, remove
-
-
-def test_assay_merge(datastore):
- # TODO: Evaluate the resulting merged file
- """
- Test the AssayMerge class by merging two assays of same type from the same dataset.
- """
- from ..merge import AssayMerge
-
- fn = full_path("merged_zarr.zarr")
- writer = AssayMerge(
- zarr_path=fn,
- assays=[datastore.RNA, datastore.RNA],
- names=["self1", "self2"],
- merge_assay_name="RNA",
- prepend_text="",
- overwrite=True,
- )
- writer.dump()
- tmp = zarr.open(fn + "/RNA/counts")
- assert tmp.shape[0] == 2 * datastore.cells.N
- assert int(tmp[...].sum()) == int(datastore.RNA.rawData.compute().sum() * 2)
- remove(fn)
-
-
-def test_dataset_merge_2(datastore):
- """
- Test the DatasetMerge class by merging two datasets. This will test the merging of all assays in the dataset.
- """
- from ..merge import DatasetMerge
-
- fn = full_path("merged_zarr.zarr")
- writer = DatasetMerge(
- zarr_path=fn,
- datasets=[datastore, datastore],
- names=["self1", "self2"],
- prepend_text="",
- overwrite=True
- )
- writer.dump()
- # Check if the merged file has the correct shape and counts
- # Load the merged counts file
- rna_count = zarr.open(fn + "/RNA/counts")
- assay2_count = zarr.open(fn + "/assay2/counts")
- # We merged only two datasets, so the shape should be 2*original_shape
- # Check the number of cells
- assert rna_count.shape[0] == 2 * datastore.cells.N
- assert assay2_count.shape[0] == 2 * datastore.cells.N
- # Check the count values through sum
- assert int(rna_count[...].sum()) == int(datastore.RNA.rawData.compute().sum() * 2)
- assert int(assay2_count[...].sum()) == int(
- datastore.assay2.rawData.compute().sum() * 2
- )
- remove(fn)
-
-
-def test_dataset_merge_3(datastore):
- """
- Test the DatasetMerge class by merging two datasets. This will test the merging of all assays in the dataset.
- If merging is successful for more than two datasets, it should work for any number of datasets.
- """
- from ..merge import DatasetMerge
-
- fn = full_path("merged_zarr.zarr")
- writer = DatasetMerge(
- zarr_path=fn,
- datasets=[datastore, datastore, datastore],
- names=["self1", "self2", "self3"],
- prepend_text="",
- overwrite=True
- )
- writer.dump()
- # Check if the merged file has the correct shape and counts
- # Load the merged counts file
- rna_count = zarr.open(fn + "/RNA/counts")
- assay2_count = zarr.open(fn + "/assay2/counts")
- # We merged only three datasets, so the shape should be 3*original_shape
- # Check the number of cells
- assert rna_count.shape[0] == 3 * datastore.cells.N
- assert assay2_count.shape[0] == 3 * datastore.cells.N
- # Check the count values through sum
- assert int(rna_count[...].sum()) == int(datastore.RNA.rawData.compute().sum() * 3)
- assert int(assay2_count[...].sum()) == int(
- datastore.assay2.rawData.compute().sum() * 3
- )
- remove(fn)
-
-def test_dataset_merge_cells(datastore):
- from ..merge import DatasetMerge
- from ..datastore.datastore import DataStore
-
- fn = full_path("merged_zarr.zarr")
- writer = DatasetMerge(
- zarr_path=fn,
- datasets=[datastore, datastore],
- names=["self1", "self2"],
- prepend_text="orig",
- overwrite=True,
- )
- writer.dump()
-
- ds = DataStore(
- fn,
- default_assay="RNA",
- )
-
- df = ds.cells.to_pandas_dataframe(ds.cells.columns)
- df_diff = df[df['orig_RNA_nCounts'] != df['RNA_nCounts']]
- assert len(df_diff) == 0
\ No newline at end of file
diff --git a/scarf/tests/test_metadata.py b/scarf/tests/test_metadata.py
deleted file mode 100644
index c5ffc0cd..00000000
--- a/scarf/tests/test_metadata.py
+++ /dev/null
@@ -1,39 +0,0 @@
-import numpy as np
-import pytest
-
-from . import full_path, remove
-
-
-@pytest.fixture
-def dummy_metadata():
- import zarr
- from ..metadata import MetaData
-
- fn = full_path("dummy_metadata.zarr")
- remove(fn)
- g = zarr.open(fn)
- data = np.array([1, 1, 1, 1, 0, 0, 1, 1, 1]).astype(bool)
- g.create_dataset(
- "I", data=data, chunks=(100000,), shape=len(data), dtype=data.dtype
- )
- yield MetaData(g)
- remove(fn)
-
-
-def test_metadata_attrs(dummy_metadata):
- assert dummy_metadata.N == 9
- assert np.all(dummy_metadata.index == np.array(range(9)))
-
-
-def test_metadata_fetch(dummy_metadata):
- assert len(dummy_metadata.fetch("I")) == 7
- assert len(dummy_metadata.fetch_all("I")) == 9
-
-
-def test_metadata_verify_bool(dummy_metadata):
- assert dummy_metadata._verify_bool("I") is True
-
-
-def test_metadata_active_index(dummy_metadata):
- a = np.array([0, 1, 2, 3, 6, 7, 8])
- assert np.all(dummy_metadata.active_index(key="I") == a)
diff --git a/scarf/tests/test_metrics.py b/scarf/tests/test_metrics.py
deleted file mode 100644
index 6cbb277b..00000000
--- a/scarf/tests/test_metrics.py
+++ /dev/null
@@ -1,49 +0,0 @@
-import numpy as np
-
-
-def test_metric_lisi(datastore, make_graph):
- lables = np.random.randint(0, 2, datastore.cells.N)
- datastore.cells.insert(
- column_name="samples_id",
- values=lables,
- overwrite=True,
- )
- lisi = datastore.metric_lisi(
- label_colnames=["samples_id"], save_result=False, return_lisi=True
- )
- assert len(lisi[0][1]) == len(datastore.cells.active_index("I"))
-
-
-def test_metric_silhouette(datastore, make_graph, leiden_clustering):
- _ = datastore.metric_silhouette()
-
-
-def test_metric_integration_ari(datastore, make_graph, leiden_clustering):
- lables1 = np.random.randint(0, 2, datastore.cells.N)
- lables2 = np.random.randint(0, 2, datastore.cells.N)
- datastore.cells.insert(
- column_name="lables1",
- values=lables1,
- overwrite=True,
- )
- datastore.cells.insert(
- column_name="lables2",
- values=lables2,
- overwrite=True,
- )
- _ = datastore.metric_integration(batch_labels=["lables1", "lables2"], metric="ari")
-
-def test_metric_integration_nmi(datastore, make_graph, leiden_clustering):
- lables1 = np.random.randint(0, 2, datastore.cells.N)
- lables2 = np.random.randint(0, 2, datastore.cells.N)
- datastore.cells.insert(
- column_name="lables1",
- values=lables1,
- overwrite=True,
- )
- datastore.cells.insert(
- column_name="lables2",
- values=lables2,
- overwrite=True,
- )
- _ = datastore.metric_integration(batch_labels=["lables1", "lables2"], metric="nmi")
\ No newline at end of file
diff --git a/scarf/tests/test_misc.py b/scarf/tests/test_misc.py
deleted file mode 100644
index d20bc822..00000000
--- a/scarf/tests/test_misc.py
+++ /dev/null
@@ -1,17 +0,0 @@
-from . import full_path, remove
-
-
-def test_export_knn_to_mtx(datastore, make_graph):
- from ..knn_utils import export_knn_to_mtx
-
- graph = datastore.load_graph(
- from_assay="RNA",
- cell_key="I",
- feat_key="hvgs",
- symmetric=False,
- upper_only=False,
- )
- fn = full_path("test_export_mtx_from_graph.mtx")
- ret_val = export_knn_to_mtx(fn, graph)
- assert ret_val is None
- remove(fn)
diff --git a/scarf/tests/test_readers.py b/scarf/tests/test_readers.py
deleted file mode 100644
index 52c2d4e7..00000000
--- a/scarf/tests/test_readers.py
+++ /dev/null
@@ -1,80 +0,0 @@
-import numpy as np
-
-
-def test_toy_crdir_assay_feats_table(toy_crdir_reader):
- assert np.all(
- toy_crdir_reader.assayFeats.columns
- == np.array(["RNA", "ADT", "RNA", "HTO", "RNA"])
- )
- assert np.all(
- toy_crdir_reader.assayFeats.values[1:]
- == [[0, 1, 3, 5, 6], [1, 3, 5, 6, 7], [1, 2, 2, 1, 1]]
- )
-
-
-def test_toy_crdir_reader_cells_feats(toy_crdir_reader):
- assert toy_crdir_reader.nCells == 3
- assert toy_crdir_reader.nFeatures == 8
- assert toy_crdir_reader.cell_names() == ["b1", "b2", "b3"]
- assert toy_crdir_reader.feature_names() == [
- "g1",
- "a1",
- "a2",
- "g2",
- "g3",
- "h1",
- "g4",
- "a3",
- ]
- assert toy_crdir_reader.feature_ids() == [
- "g1",
- "a1",
- "a2",
- "g2",
- "g3",
- "h1",
- "g4",
- "a3",
- ]
-
-def test_toy_crdir_empty(toy_crdir_empty):
- assert toy_crdir_empty.nCells == 0
- assert toy_crdir_empty.nFeatures == 4
- assert toy_crdir_empty.feature_names() == [
- "g1",
- "a1",
- "a2",
- "g2",
- ]
- assert toy_crdir_empty.feature_ids() == [
- "g1",
- "a1",
- "a2",
- "g2",
- ]
- # check for raise ValueError
- try:
- toy_crdir_empty.read_header()
- except ValueError:
- pass
-
-def test_crh5reader(crh5_reader):
- assert crh5_reader.nCells == 892
- assert crh5_reader.nFeatures == 36611
- n_assay_feats = list(crh5_reader.assayFeats.T.nFeatures.values)
- assert n_assay_feats == [36601, 10]
-
-
-def test_crdir_reader(crdir_reader):
- assert crdir_reader.nCells == 892
- assert crdir_reader.nFeatures == 36601 # Does not contain 10 ADTs
-
-
-def test_h5ad_reader(h5ad_reader):
- assert h5ad_reader.nCells == 3696 == len(h5ad_reader.cell_ids())
- assert h5ad_reader.nFeatures == 27998 == len(h5ad_reader.feat_names())
-
-
-def test_loom_reader(loom_reader):
- assert loom_reader.nCells == 298 == len(loom_reader.cell_ids())
- assert loom_reader.nFeatures == 16892 == len(loom_reader.feature_names())
diff --git a/scarf/tests/test_writers.py b/scarf/tests/test_writers.py
deleted file mode 100644
index 1be67ac2..00000000
--- a/scarf/tests/test_writers.py
+++ /dev/null
@@ -1,92 +0,0 @@
-import numpy as np
-
-from . import full_path, remove
-
-
-def test_crtozarr(crh5_reader):
- from ..writers import CrToZarr
-
- fn = full_path("dummy_1K_pbmc_citeseq.zarr")
- writer = CrToZarr(crh5_reader, zarr_loc=fn)
- writer.dump()
- remove(fn)
-
-
-def test_crtozarr_fromdir(crdir_reader):
- from ..writers import CrToZarr
-
- fn = full_path("1K_pbmc_citeseq_dir.zarr")
- writer = CrToZarr(crdir_reader, zarr_loc=fn)
- writer.dump()
- remove(fn)
-
-
-def test_h5adtozarr(h5ad_reader):
- from ..writers import H5adToZarr
-
- fn = full_path("bastidas.zarr")
- writer = H5adToZarr(h5ad_reader, zarr_loc=fn)
- writer.dump()
- remove(fn)
-
-
-def test_loomtozarr(loom_reader):
- from ..writers import LoomToZarr
-
- fn = full_path("sympathetic.zarr")
- writer = LoomToZarr(loom_reader, zarr_loc=fn)
- writer.dump()
- remove(fn)
-
-
-def test_sparsetozarr():
- from ..writers import SparseToZarr
- from scipy.sparse import csr_matrix
-
- cols = [1, 3, 8, 2, 3, 1, 2, 8, 9]
- rows = [0, 0, 0, 1, 1, 1, 2, 2, 2]
- data = [1, 10, 15, 10, 20, 2, 3, 1, 5]
- mat = (data, (rows, cols))
- mat = csr_matrix(mat, shape=(3, 10))
-
- fn = full_path("dummy_sparse.zarr")
-
- writer = SparseToZarr(
- mat,
- zarr_loc=fn,
- cell_ids=[f"cell_{x}" for x in range(3)],
- feature_ids=[f"feat_{x}" for x in range(10)],
- )
- writer.dump()
- remove(fn)
-
-
-def test_to_h5ad(datastore):
- # TODO: Evaluate the resulting H5ad file
- from ..writers import to_h5ad
-
- fn = full_path("test_1K_pbmc_citeseq.h5ad")
- to_h5ad(datastore.RNA, fn)
- remove(fn)
-
-
-def test_to_mtx(datastore):
- # TODO: Evaluate the resulting MTX directory
- from ..writers import to_mtx
-
- fn = full_path("test_1K_pbmc_citeseq_dir")
- to_mtx(datastore.RNA, fn)
- remove(fn)
-
-
-def test_zarr_subset(datastore):
- # TODO: Evaluate the resulting subsetted file
-
- from ..writers import SubsetZarr
-
- zarr_path = full_path("subset.zarr")
- writer = SubsetZarr(
- zarr_loc=zarr_path, assays=[datastore.RNA], cell_idx=np.array([1, 10, 100, 500])
- )
- writer.dump()
- remove(zarr_path)
diff --git a/scarf/tools/__init__.py b/scarf/tools/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/scarf/tools/repack_zarr.py b/scarf/tools/repack_zarr.py
new file mode 100644
index 00000000..bd316a0a
--- /dev/null
+++ b/scarf/tools/repack_zarr.py
@@ -0,0 +1,233 @@
+"""Repack a Zarr store from v2 to v3 with optional sharding."""
+
+import argparse
+import posixpath
+from pathlib import Path, PurePosixPath
+from urllib.parse import urlsplit, urlunsplit
+
+import numpy as np
+import zarr
+
+from scarf.storage.arrays import create_numeric_array
+from scarf.storage.layout import (
+ array_info,
+ count_array_spec,
+ get_compressors,
+ normalize_chunks,
+)
+from scarf.storage.profiles import StorageProfile
+from scarf.storage.sharding import write_counts_t, write_dense_in_shard_rows
+from scarf.storage.stores import open_store
+from scarf.storage.types import array_metadata_shards, as_zarr_array, as_zarr_group
+
+
+def _location_identity(location: str) -> tuple[str, str]:
+ parsed = urlsplit(location)
+ if parsed.scheme in ("", "file"):
+ path = parsed.path if parsed.scheme == "file" else location
+ return "file", str(Path(path).expanduser().resolve())
+ normalized_path = posixpath.normpath(parsed.path or "/")
+ return "uri", urlunsplit(
+ (
+ parsed.scheme.lower(),
+ parsed.netloc,
+ normalized_path,
+ parsed.query,
+ parsed.fragment,
+ )
+ )
+
+
+def _locations_overlap(first: str, second: str) -> bool:
+ first_kind, first_identity = _location_identity(first)
+ second_kind, second_identity = _location_identity(second)
+ if (first_kind, first_identity) == (second_kind, second_identity):
+ return True
+ if first_kind != second_kind:
+ return False
+ if first_kind == "file":
+ first_path: Path | PurePosixPath = Path(first_identity)
+ second_path: Path | PurePosixPath = Path(second_identity)
+ else:
+ first_uri = urlsplit(first_identity)
+ second_uri = urlsplit(second_identity)
+ if (first_uri.scheme, first_uri.netloc) != (
+ second_uri.scheme,
+ second_uri.netloc,
+ ):
+ return False
+ first_path = PurePosixPath(first_uri.path)
+ second_path = PurePosixPath(second_uri.path)
+ if first_path == second_path:
+ return True
+ return first_path in second_path.parents or second_path in first_path.parents
+
+
+def _count_assays(store: zarr.Group) -> list[tuple[str, str | None]]:
+ assays: list[tuple[str, str | None]] = []
+ for name in store.group_keys():
+ group = as_zarr_group(store[name], name=name)
+ if group.attrs.get("is_assay") and "counts" in group:
+ assays.append((name, None))
+
+ if "matrices" not in store:
+ return assays
+ matrices = as_zarr_group(store["matrices"], name="matrices")
+ workspace_assays: set[str] = set()
+
+ def visit(group: zarr.Group, path: str) -> None:
+ for name in group.group_keys():
+ if path == "" and name == "matrices":
+ continue
+ child_path = f"{path}/{name}" if path else name
+ child = as_zarr_group(group[name], name=child_path)
+ if child.attrs.get("is_assay"):
+ if path == "" and "counts" in child:
+ continue
+ if (
+ name not in workspace_assays
+ and name in matrices
+ and "counts" in as_zarr_group(matrices[name], name=name)
+ ):
+ workspace_assays.add(name)
+ assays.append((name, path or "."))
+ continue
+ visit(child, child_path)
+
+ visit(store, "")
+ return assays
+
+
+def _copy_group(
+ src: zarr.Group,
+ dst: zarr.Group,
+ profile: StorageProfile,
+ *,
+ path: str = "",
+ shardedCounts: frozenset[str] = frozenset(),
+) -> None:
+ for key in src.keys():
+ node = src[key]
+ child_path = f"{path}/{key}" if path else key
+ if isinstance(node, zarr.Group):
+ new_group = dst.create_group(key, overwrite=True)
+ for attr_key, attr_val in node.attrs.items():
+ new_group.attrs[attr_key] = attr_val
+ _copy_group(
+ node,
+ new_group,
+ profile,
+ path=child_path,
+ shardedCounts=shardedCounts,
+ )
+ continue
+ if child_path in shardedCounts:
+ spec = count_array_spec(
+ int(node.shape[0]),
+ int(node.shape[1]),
+ dtype=node.dtype,
+ profile=profile,
+ )
+ dst_array = create_numeric_array(dst, key, spec)
+ write_dense_in_shard_rows(
+ dst_array,
+ lambda start, end: np.asarray(node[start:end, :]),
+ msg=f"Repacking {child_path}",
+ )
+ stored_shards = array_metadata_shards(dst_array)
+ dst.attrs["scarf:zarr_spec"] = {
+ "profile": profile,
+ "dtype": np.dtype(node.dtype).str,
+ "chunks": list(dst_array.chunks),
+ "shards": None if stored_shards is None else list(stored_shards),
+ "zarr_format": 3,
+ }
+ else:
+ chunks = normalize_chunks(node.chunks, node.shape)
+ dst_array = dst.create_array(
+ key,
+ data=np.asarray(node[...]),
+ chunks=chunks,
+ compressors=get_compressors(profile, zarrFormat=3),
+ overwrite=True,
+ )
+ for attr_key, attr_val in node.attrs.items():
+ dst_array.attrs[attr_key] = attr_val
+
+
+def repack_store(
+ input_path: str,
+ output_path: str,
+ profile: StorageProfile = "fast_local",
+ shard_counts: bool = True,
+ storage_options: dict | None = None,
+) -> None:
+ """Copy a Zarr store to a new path and optionally shard count arrays.
+
+ Args:
+ input_path: Source Zarr directory.
+ output_path: Destination Zarr directory (created or overwritten).
+ profile: Storage profile for compressors and shard sizes.
+ shard_counts: Repack assay count arrays to sharded layout when True.
+ """
+ if _locations_overlap(input_path, output_path):
+ raise ValueError(
+ "input_path and output_path must refer to different stores "
+ "and must not overlap"
+ )
+
+ src = open_store(input_path, mode="r", storage_options=storage_options)
+ dst = open_store(output_path, mode="w", storage_options=storage_options)
+ assays = _count_assays(src)
+ count_paths = frozenset(
+ f"{assay_name}/counts" if workspace is None else f"matrices/{assay_name}/counts"
+ for assay_name, workspace in assays
+ )
+ _copy_group(
+ src,
+ dst,
+ profile,
+ shardedCounts=count_paths if shard_counts else frozenset(),
+ )
+ if not shard_counts:
+ return
+ for assay_name, workspace in assays:
+ counts_path = (
+ f"{assay_name}/counts"
+ if workspace is None
+ else f"matrices/{assay_name}/counts"
+ )
+ group_path = assay_name if workspace is None else f"matrices/{assay_name}"
+ counts = as_zarr_array(dst[counts_path], name=counts_path)
+ write_counts_t(
+ counts,
+ as_zarr_group(dst[group_path], name=group_path),
+ profile=profile,
+ )
+ print(f" {counts_path}: {array_info(counts)}")
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser(
+ description="Repack Zarr v2 stores to v3 with sharding"
+ )
+ parser.add_argument("input", type=Path)
+ parser.add_argument("output", type=Path)
+ parser.add_argument(
+ "--profile",
+ choices=["fast_local", "cloud"],
+ default="fast_local",
+ )
+ parser.add_argument("--no-shard-counts", action="store_true")
+ args = parser.parse_args()
+ repack_store(
+ str(args.input),
+ str(args.output),
+ profile=args.profile,
+ shard_counts=not args.no_shard_counts,
+ )
+ print(f"Repacked {args.input} -> {args.output}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scarf/trajectory/__init__.py b/scarf/trajectory/__init__.py
new file mode 100644
index 00000000..14f952b0
--- /dev/null
+++ b/scarf/trajectory/__init__.py
@@ -0,0 +1,106 @@
+from importlib import import_module as _import_module
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from .feature_dynamics import (
+ aggregate_feature_profiles as aggregate_feature_profiles,
+ scatter_feature_clusters as scatter_feature_clusters,
+ validate_pseudotime_regressor as validate_pseudotime_regressor,
+ )
+ from .pseudotime import (
+ make_source_sink_vector as make_source_sink_vector,
+ random_walk_laplacian_transpose as random_walk_laplacian_transpose,
+ select_pseudotime_component as select_pseudotime_component,
+ truncated_pba_potential as truncated_pba_potential,
+ validate_source_sink_labels as validate_source_sink_labels,
+ validate_source_sink_vector as validate_source_sink_vector,
+ )
+ from .results import (
+ FateMappingResult as FateMappingResult,
+ PseudotimeAggregationResult as PseudotimeAggregationResult,
+ PseudotimeMarkerResult as PseudotimeMarkerResult,
+ PseudotimeScoreResult as PseudotimeScoreResult,
+ )
+
+__all__ = [
+ "FateMappingResult",
+ "PseudotimeAggregationResult",
+ "PseudotimeMarkerResult",
+ "PseudotimeScoreResult",
+ "aggregate_feature_profiles",
+ "make_source_sink_vector",
+ "random_walk_laplacian_transpose",
+ "select_pseudotime_component",
+ "scatter_feature_clusters",
+ "truncated_pba_potential",
+ "validate_source_sink_labels",
+ "validate_source_sink_vector",
+ "validate_pseudotime_regressor",
+]
+
+_LAZY_EXPORTS = {
+ "aggregate_feature_profiles": (
+ ".feature_dynamics",
+ "aggregate_feature_profiles",
+ ),
+ "make_source_sink_vector": (
+ ".pseudotime",
+ "make_source_sink_vector",
+ ),
+ "random_walk_laplacian_transpose": (
+ ".pseudotime",
+ "random_walk_laplacian_transpose",
+ ),
+ "select_pseudotime_component": (
+ ".pseudotime",
+ "select_pseudotime_component",
+ ),
+ "scatter_feature_clusters": (
+ ".feature_dynamics",
+ "scatter_feature_clusters",
+ ),
+ "truncated_pba_potential": (
+ ".pseudotime",
+ "truncated_pba_potential",
+ ),
+ "validate_source_sink_labels": (
+ ".pseudotime",
+ "validate_source_sink_labels",
+ ),
+ "validate_source_sink_vector": (
+ ".pseudotime",
+ "validate_source_sink_vector",
+ ),
+ "validate_pseudotime_regressor": (
+ ".feature_dynamics",
+ "validate_pseudotime_regressor",
+ ),
+ "FateMappingResult": (
+ ".results",
+ "FateMappingResult",
+ ),
+ "PseudotimeAggregationResult": (
+ ".results",
+ "PseudotimeAggregationResult",
+ ),
+ "PseudotimeMarkerResult": (".results", "PseudotimeMarkerResult"),
+ "PseudotimeScoreResult": (".results", "PseudotimeScoreResult"),
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+
+def __getattr__(name: str) -> Any:
+ export = _LAZY_EXPORTS.get(name)
+ if export is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ module_name, attribute_name = export
+ value = getattr(_import_module(module_name, __name__), attribute_name)
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()).union(_LAZY_EXPORTS))
diff --git a/scarf/trajectory/fate.py b/scarf/trajectory/fate.py
new file mode 100644
index 00000000..550b1c8d
--- /dev/null
+++ b/scarf/trajectory/fate.py
@@ -0,0 +1,483 @@
+import math
+import re
+from typing import Any
+
+import numpy as np
+from numba import njit
+from scipy.sparse import csr_matrix
+from scipy.sparse.csgraph import connected_components
+from scipy.sparse.linalg import LinearOperator, gmres
+
+from ..utils.logging import logger
+
+
+@njit(cache=True, inline="always")
+def _log_biased_weight(
+ weight: float,
+ pseudotime_difference: float,
+ beta: float,
+) -> float:
+ value = math.log(weight)
+ if pseudotime_difference > 0.0 and beta > 0.0:
+ exponent = beta * pseudotime_difference
+ value += math.log(2.0) - exponent - math.log1p(math.exp(-exponent))
+ return value
+
+
+@njit(cache=True)
+def _bias_and_normalize_rows(
+ data: np.ndarray,
+ indices: np.ndarray,
+ indptr: np.ndarray,
+ pseudotime: np.ndarray,
+ absorbing: np.ndarray,
+ beta: float,
+) -> int:
+ """Bias backward edges and normalize CSR rows in place."""
+ isolated_transient_count = 0
+ tiny = np.finfo(np.float64).tiny
+ for row in range(pseudotime.shape[0]):
+ maximum_log_weight = -math.inf
+ for offset in range(indptr[row], indptr[row + 1]):
+ col = indices[offset]
+ if col == row:
+ data[offset] = 0.0
+ continue
+ log_weight = _log_biased_weight(
+ data[offset],
+ pseudotime[row] - pseudotime[col],
+ beta,
+ )
+ maximum_log_weight = max(maximum_log_weight, log_weight)
+
+ if maximum_log_weight == -math.inf:
+ if not absorbing[row]:
+ isolated_transient_count += 1
+ continue
+
+ row_sum = 0.0
+ for offset in range(indptr[row], indptr[row + 1]):
+ col = indices[offset]
+ if col == row:
+ continue
+ log_weight = _log_biased_weight(
+ data[offset],
+ pseudotime[row] - pseudotime[col],
+ beta,
+ )
+ scaled_weight = math.exp(log_weight - maximum_log_weight)
+ if scaled_weight == 0.0:
+ scaled_weight = tiny
+ data[offset] = scaled_weight
+ row_sum += scaled_weight
+
+ inverse_sum = 1.0 / row_sum
+ for offset in range(indptr[row], indptr[row + 1]):
+ data[offset] *= inverse_sum
+ return isolated_transient_count
+
+
+@njit(cache=True)
+def _has_symmetric_support(
+ indices: np.ndarray,
+ indptr: np.ndarray,
+) -> bool:
+ for row in range(indptr.shape[0] - 1):
+ for offset in range(indptr[row], indptr[row + 1]):
+ col = indices[offset]
+ if col == row:
+ continue
+ lower = indptr[col]
+ upper = indptr[col + 1]
+ while lower < upper:
+ middle = (lower + upper) // 2
+ candidate = indices[middle]
+ if candidate < row:
+ lower = middle + 1
+ else:
+ upper = middle
+ if lower >= indptr[col + 1] or indices[lower] != row:
+ return False
+ return True
+
+
+def _validate_sink_groups(
+ labels: np.ndarray,
+ sinks: list[Any],
+) -> tuple[tuple[Any, ...], np.ndarray]:
+ if not isinstance(sinks, list):
+ raise TypeError("sinks must be a list")
+ if not sinks:
+ raise ValueError("At least one sink label must be provided")
+ if any(np.ndim(sink) != 0 for sink in sinks):
+ raise TypeError("Sink labels must be hashable scalar values")
+ try:
+ if len(set(sinks)) != len(sinks):
+ raise ValueError("Sink labels must be unique")
+ except TypeError as exc:
+ raise TypeError("Sink labels must be hashable scalar values") from exc
+
+ sink_labels = tuple(sinks)
+ sink_groups = np.full(labels.shape[0], -1, dtype=np.int32)
+ missing: list[Any] = []
+ for group, sink in enumerate(sink_labels):
+ try:
+ matches = np.asarray(labels == sink)
+ except (TypeError, ValueError) as exc:
+ raise TypeError("Sink labels must be comparable scalar values") from exc
+ if matches.ndim != 1 or matches.dtype.kind != "b":
+ raise TypeError("Sink labels must be comparable scalar values")
+ if not matches.any():
+ missing.append(sink)
+ continue
+ if np.any(sink_groups[matches] >= 0):
+ raise ValueError("Sink groups must be disjoint")
+ sink_groups[matches] = group
+ if missing:
+ raise ValueError(f"Sink labels were not found in the selected cells: {missing}")
+ return sink_labels, sink_groups
+
+
+def make_sink_tokens(sinks: tuple[Any, ...]) -> tuple[str, ...]:
+ """Create deterministic metadata-safe tokens for sink labels."""
+ tokens: list[str] = []
+ used: set[str] = set()
+ for index, sink in enumerate(sinks):
+ base = re.sub(r"[^0-9A-Za-z_.-]+", "_", str(sink)).strip("_.-")
+ if not base:
+ base = f"sink_{index + 1}"
+ token = base
+ suffix = 2
+ while token in used:
+ token = f"{base}_{suffix}"
+ suffix += 1
+ used.add(token)
+ tokens.append(token)
+ return tuple(tokens)
+
+
+def _normalize_pseudotime(values: np.ndarray) -> np.ndarray:
+ try:
+ pseudotime = np.asarray(values, dtype=np.float64)
+ except (TypeError, ValueError) as exc:
+ raise TypeError("Pseudotime values must be numeric") from exc
+ if pseudotime.ndim != 1:
+ raise ValueError("Pseudotime values must be one-dimensional")
+ if not np.isfinite(pseudotime).all():
+ raise ValueError("Pseudotime values must be finite")
+ minimum = float(pseudotime.min())
+ maximum = float(pseudotime.max())
+ scale = max(1.0, abs(minimum), abs(maximum))
+ scaled_minimum = minimum / scale
+ value_range = (maximum / scale) - scaled_minimum
+ if value_range <= np.finfo(np.float64).eps:
+ raise ValueError("Pseudotime values must contain at least two distinct values")
+ normalized = pseudotime / scale
+ normalized -= scaled_minimum
+ normalized /= value_range
+ if not np.isfinite(normalized).all():
+ raise ValueError("Pseudotime values could not be normalized safely")
+ np.clip(normalized, 0.0, 1.0, out=normalized)
+ return normalized
+
+
+def _validate_graph(
+ graph: csr_matrix,
+ n_cells: int,
+) -> None:
+ if not isinstance(graph, csr_matrix):
+ raise TypeError("graph must be a scipy.sparse.csr_matrix")
+ try:
+ graph.check_format(full_check=True)
+ except ValueError as exc:
+ raise ValueError("graph has invalid CSR structure") from exc
+ if graph.shape != (n_cells, n_cells):
+ raise ValueError(
+ f"Graph shape {graph.shape} does not match {n_cells} selected cells"
+ )
+ if graph.data.dtype.kind not in "buif":
+ raise TypeError("Graph weights must be real numeric values")
+ if not np.isfinite(graph.data).all():
+ raise ValueError("Graph weights must be finite")
+ if np.any(graph.data < 0):
+ raise ValueError("Graph weights must be non-negative")
+
+
+def _make_transition(
+ graph: csr_matrix,
+ pseudotime: np.ndarray,
+ absorbing: np.ndarray,
+ beta: float,
+) -> csr_matrix:
+ transition = graph
+ transition.sum_duplicates()
+ transition.eliminate_zeros()
+ if transition.dtype != np.float64:
+ with np.errstate(over="ignore", invalid="ignore"):
+ transition.data = transition.data.astype(np.float64)
+ if not np.isfinite(transition.data).all():
+ raise ValueError("Graph weights must remain finite when converted to float64")
+ if np.any(transition.data <= 0.0):
+ raise ValueError("Graph weights must remain positive when converted to float64")
+ transition.sort_indices()
+ isolated_transient_count = _bias_and_normalize_rows(
+ transition.data,
+ transition.indices,
+ transition.indptr,
+ pseudotime,
+ absorbing,
+ beta,
+ )
+ transition.eliminate_zeros()
+ if isolated_transient_count:
+ raise ValueError(
+ "The directed graph contains "
+ f"{isolated_transient_count} isolated transient cells"
+ )
+ return transition
+
+
+def _dirichlet_operator(
+ transition: csr_matrix,
+ absorbing: np.ndarray,
+) -> LinearOperator:
+ n_cells = transition.shape[0]
+
+ def matvec(values: np.ndarray) -> np.ndarray:
+ result = np.asarray(values, dtype=np.float64) - transition.dot(values)
+ result[absorbing] = values[absorbing]
+ return np.asarray(result)
+
+ return LinearOperator(
+ shape=(n_cells, n_cells),
+ matvec=matvec,
+ dtype=np.dtype(np.float64),
+ )
+
+
+def _bellman_residual(
+ transition: csr_matrix,
+ probabilities: np.ndarray,
+ sink_groups: np.ndarray,
+ group: int,
+) -> float:
+ boundary = np.asarray(sink_groups == group, dtype=np.float64)
+ residual = probabilities[:, group] - transition.dot(probabilities[:, group])
+ absorbing = sink_groups >= 0
+ residual[absorbing] = probabilities[absorbing, group] - boundary[absorbing]
+ minimum = float(residual.min(initial=0.0))
+ maximum = float(residual.max(initial=0.0))
+ return max(-minimum, maximum)
+
+
+def _solve_fates(
+ transition: csr_matrix,
+ sink_groups: np.ndarray,
+ n_sinks: int,
+ solver_tol: float,
+ max_iterations: int,
+) -> np.ndarray:
+ n_cells = transition.shape[0]
+ probabilities = np.zeros((n_cells, n_sinks), dtype=np.float32)
+ last_probability = np.ones(n_cells, dtype=np.float64)
+ absorbing = sink_groups >= 0
+
+ operator = _dirichlet_operator(transition, absorbing)
+ for group in range(n_sinks - 1):
+ boundary = np.asarray(sink_groups == group, dtype=np.float64)
+ iterations = 0
+
+ def count_iteration(_residual: float) -> None:
+ nonlocal iterations
+ iterations += 1
+
+ solution, info = gmres(
+ operator,
+ boundary,
+ x0=boundary,
+ rtol=solver_tol,
+ atol=0.0,
+ restart=20,
+ maxiter=max_iterations,
+ callback=count_iteration,
+ # This mode makes maxiter count inner iterations, not restart cycles.
+ callback_type="legacy",
+ )
+ iteration_unit = "iteration" if iterations == 1 else "iterations"
+ if info != 0:
+ reason = "broke down" if info < 0 else "did not converge"
+ raise RuntimeError(
+ f"Fate probability solve for sink index {group} {reason} "
+ f"after {iterations} {iteration_unit}"
+ )
+ if not np.isfinite(solution).all():
+ raise RuntimeError(
+ f"Fate probability solve for sink index {group} produced "
+ "non-finite values"
+ )
+ probabilities[:, group] = solution
+ last_probability -= solution
+ logger.debug(
+ f"Fate mapping: sink {group + 1}/{n_sinks} converged "
+ f"in {iterations} {iteration_unit}"
+ )
+
+ probabilities[:, -1] = last_probability
+ del boundary, last_probability, solution
+ probabilities[absorbing] = 0.0
+ probabilities[absorbing, sink_groups[absorbing]] = 1.0
+
+ validation_scale = 10.0 * solver_tol * max(1, n_sinks - 1)
+ float32_tolerance = 5.0 * float(np.finfo(np.float32).eps)
+ check_tol = max(float32_tolerance, min(1e-3, validation_scale))
+ if not np.isfinite(probabilities).all():
+ raise RuntimeError("Fate probability calculation produced non-finite values")
+ minimum = float(probabilities.min())
+ maximum = float(probabilities.max())
+ if minimum < -check_tol or maximum > 1.0 + check_tol:
+ raise RuntimeError(
+ "Fate probabilities exceeded numerical bounds "
+ f"(minimum={minimum:.3e}, maximum={maximum:.3e})"
+ )
+ row_sums = probabilities.sum(axis=1, dtype=np.float64)
+ if not np.allclose(row_sums, 1.0, rtol=0.0, atol=check_tol):
+ deviation = float(np.max(np.abs(row_sums - 1.0)))
+ raise RuntimeError(
+ f"Fate probabilities do not sum to one (maximum deviation={deviation:.3e})"
+ )
+
+ np.clip(probabilities, 0.0, 1.0, out=probabilities)
+ row_sums = probabilities.sum(axis=1, dtype=np.float64)
+ probabilities /= row_sums[:, None]
+ probabilities[absorbing] = 0.0
+ probabilities[absorbing, sink_groups[absorbing]] = 1.0
+
+ residual_limit = max(float32_tolerance, min(1e-3, validation_scale))
+ residuals = [
+ _bellman_residual(transition, probabilities, sink_groups, group)
+ for group in range(n_sinks)
+ ]
+ if max(residuals) > residual_limit:
+ raise RuntimeError(
+ "Fate probabilities failed the Bellman residual check "
+ f"(maximum={max(residuals):.3e}, limit={residual_limit:.3e})"
+ )
+ return probabilities
+
+
+def compute_fate_probabilities(
+ graph: csr_matrix,
+ pseudotime: np.ndarray,
+ labels: np.ndarray,
+ sinks: list[Any],
+ *,
+ beta: float = 10.0,
+ solver_tol: float = 1e-6,
+ max_iterations: int = 1000,
+ _copy_graph: bool = True,
+) -> tuple[np.ndarray, np.ndarray, tuple[Any, ...]]:
+ """Compute grouped absorption probabilities on a pseudotime-biased graph."""
+ try:
+ beta = float(beta)
+ except (TypeError, ValueError) as exc:
+ raise TypeError("beta must be numeric") from exc
+ if not np.isfinite(beta) or beta < 0:
+ raise ValueError("beta must be finite and non-negative")
+ try:
+ solver_tol = float(solver_tol)
+ except (TypeError, ValueError) as exc:
+ raise TypeError("solver_tol must be numeric") from exc
+ if not np.isfinite(solver_tol) or not 0.0 < solver_tol < 1.0:
+ raise ValueError("solver_tol must be finite and between 0 and 1")
+ if not isinstance(max_iterations, int) or isinstance(max_iterations, bool):
+ raise TypeError("max_iterations must be an integer")
+ if max_iterations < 1:
+ raise ValueError("max_iterations must be at least 1")
+
+ labels = np.asarray(labels)
+ if labels.ndim != 1:
+ raise ValueError("Sink labels must be one-dimensional")
+ _validate_graph(graph, labels.shape[0])
+ if graph.shape[0] == 0:
+ raise ValueError("No cells were selected for fate mapping")
+ try:
+ pseudotime_values = np.asarray(pseudotime, dtype=np.float64)
+ except (TypeError, ValueError) as exc:
+ raise TypeError("Pseudotime values must be numeric") from exc
+ if pseudotime_values.shape != (labels.shape[0],):
+ raise ValueError("Pseudotime values must align with the selected cells")
+ if not np.isfinite(pseudotime_values).all():
+ raise ValueError("Pseudotime values must be finite")
+
+ sink_labels, sink_groups = _validate_sink_groups(labels, sinks)
+ component_graph = graph
+ if not graph.has_canonical_format or np.count_nonzero(graph.data) != graph.nnz:
+ component_graph = graph.copy()
+ component_graph.sum_duplicates()
+ component_graph.eliminate_zeros()
+ if not np.isfinite(component_graph.data).all():
+ raise ValueError("Graph weights became non-finite when duplicates were summed")
+ if not _has_symmetric_support(
+ component_graph.indices,
+ component_graph.indptr,
+ ):
+ raise ValueError("Graph support must be symmetric")
+ n_components, component_labels = connected_components(
+ component_graph,
+ directed=False,
+ return_labels=True,
+ )
+ retained_components = np.zeros(n_components, dtype=bool)
+ retained_components[component_labels[sink_groups >= 0]] = True
+ valid = retained_components[component_labels]
+ all_components_retained = bool(retained_components.all())
+ if not all_components_retained:
+ component_sizes = np.bincount(component_labels, minlength=n_components)
+ omitted_sizes = component_sizes[~retained_components]
+ displayed_sizes = omitted_sizes[:20].astype(int).tolist()
+ display_suffix = (
+ "" if omitted_sizes.size <= 20 else f" (showing 20 of {omitted_sizes.size})"
+ )
+ logger.warning(
+ "Fate mapping: omitting sinkless graph components with sizes "
+ f"{displayed_sizes}{display_suffix}; "
+ f"{int(omitted_sizes.sum())} cells marked invalid"
+ )
+
+ retained_pseudotime = _normalize_pseudotime(pseudotime_values[valid])
+ if len(sink_labels) == 1:
+ probabilities = np.full((labels.shape[0], 1), np.nan, dtype=np.float32)
+ probabilities[valid, 0] = 1.0
+ return probabilities, valid, sink_labels
+
+ retained_sink_groups = sink_groups[valid]
+ if all_components_retained:
+ retained_graph = component_graph
+ if retained_graph is graph and _copy_graph:
+ retained_graph = graph.copy()
+ else:
+ retained_graph = component_graph[valid][:, valid].tocsr()
+ transition = _make_transition(
+ retained_graph,
+ retained_pseudotime,
+ retained_sink_groups >= 0,
+ beta,
+ )
+ retained_probabilities = _solve_fates(
+ transition,
+ retained_sink_groups,
+ len(sink_labels),
+ solver_tol,
+ max_iterations,
+ )
+
+ if all_components_retained:
+ probabilities = retained_probabilities
+ else:
+ probabilities = np.full(
+ (labels.shape[0], len(sink_labels)),
+ np.nan,
+ dtype=np.float32,
+ )
+ probabilities[valid] = retained_probabilities
+ return probabilities, valid, sink_labels
diff --git a/scarf/trajectory/feature_dynamics.py b/scarf/trajectory/feature_dynamics.py
new file mode 100644
index 00000000..40c9e25d
--- /dev/null
+++ b/scarf/trajectory/feature_dynamics.py
@@ -0,0 +1,277 @@
+from typing import Any
+
+import numpy as np
+import scarf
+
+from ..utils.arrays import rolling_window
+from ..utils.logging import logger
+
+__all__ = [
+ "aggregate_feature_profiles",
+ "knn_clustering",
+ "scatter_feature_clusters",
+ "validate_pseudotime_regressor",
+]
+
+
+def validate_pseudotime_regressor(
+ values: object,
+ expected_size: int,
+ pseudotime_key: str,
+ cell_key: str,
+ *,
+ has_validity_column: bool,
+) -> np.ndarray:
+ try:
+ pseudotime = np.asarray(values, dtype=float)
+ except (TypeError, ValueError) as exc:
+ raise TypeError(
+ f"Pseudotime column '{pseudotime_key}' must be numeric"
+ ) from exc
+
+ if pseudotime.ndim != 1:
+ raise ValueError(
+ f"Pseudotime column '{pseudotime_key}' must be one-dimensional"
+ )
+ if pseudotime.shape[0] != expected_size:
+ raise ValueError(
+ f"Pseudotime column '{pseudotime_key}' has {pseudotime.shape[0]} values, "
+ f"but cell_key '{cell_key}' selects {expected_size} cells"
+ )
+ if not np.isfinite(pseudotime).all():
+ validity_key = f"{pseudotime_key}__valid"
+ if has_validity_column:
+ raise ValueError(
+ f"Pseudotime column '{pseudotime_key}' contains unscored cells. "
+ f"Use cell_key='{validity_key}' for downstream analysis"
+ )
+ raise ValueError(
+ f"Pseudotime column '{pseudotime_key}' contains non-finite values"
+ )
+ if pseudotime.size < 2 or np.unique(pseudotime).size < 2:
+ raise ValueError(
+ f"Pseudotime column '{pseudotime_key}' must contain at least two distinct values"
+ )
+ return pseudotime
+
+
+def scatter_feature_clusters(
+ n_features: int,
+ feature_indices: np.ndarray,
+ clusters: np.ndarray,
+ unassigned_value: int,
+) -> np.ndarray:
+ feature_indices = np.asarray(feature_indices, dtype=int)
+ clusters = np.asarray(clusters, dtype=int)
+ if feature_indices.shape != clusters.shape:
+ raise ValueError("Feature indices and cluster assignments are misaligned")
+ if unassigned_value in clusters:
+ raise ValueError("unassigned_value conflicts with an assigned feature cluster")
+ values = np.full(n_features, unassigned_value, dtype=int)
+ values[feature_indices] = clusters
+ return values
+
+
+def aggregate_feature_profiles(
+ values: np.ndarray,
+ ordering_indices: np.ndarray,
+ feature_indices: np.ndarray,
+ *,
+ min_expression: float,
+ window_size: int,
+ n_bins: int,
+ smooth: bool,
+ z_scale: bool,
+) -> tuple[np.ndarray, np.ndarray]:
+ """Order, smooth, filter, and bin normalized feature profiles."""
+ ordered = np.asarray(values, dtype=float)[ordering_indices]
+ if not np.isfinite(ordered).all():
+ invalid_columns = np.asarray(feature_indices)[~np.isfinite(ordered).all(axis=0)]
+ raise ValueError(
+ f"Normalized features contain non-finite values: {invalid_columns.tolist()}"
+ )
+ if smooth:
+ ordered = rolling_window(ordered, window_size)
+ if not np.isfinite(ordered).all():
+ raise ValueError("Smoothed feature profiles contain non-finite values")
+
+ mean_expression = np.asarray(values, dtype=float).mean(axis=0)
+ standard_deviation = ordered.std(axis=0)
+ valid_features = (
+ (mean_expression > min_expression)
+ & np.isfinite(standard_deviation)
+ & (standard_deviation > np.finfo(float).eps)
+ )
+ if z_scale:
+ processed = np.zeros_like(ordered, dtype=float)
+ processed[:, valid_features] = (
+ ordered[:, valid_features] - ordered[:, valid_features].mean(axis=0)
+ ) / standard_deviation[valid_features]
+ else:
+ processed = ordered.copy()
+ processed[:, ~valid_features] = 0.0
+
+ binned = np.stack(
+ [
+ bin_values.mean(axis=0)
+ for bin_values in np.array_split(processed, n_bins, axis=0)
+ ],
+ axis=1,
+ )
+ if not np.isfinite(binned).all():
+ raise ValueError("Binned feature profiles contain non-finite values")
+ return binned, valid_features
+
+
+def knn_clustering(
+ d_array: "scarf.matrix.ChunkedArray",
+ n_neighbours: int,
+ n_clusters: int,
+ n_threads: int,
+ ann_params: dict[str, Any] | None = None,
+) -> np.ndarray:
+ """Cluster pseudotime-ordered feature profiles."""
+ import pandas as pd
+ from scipy.sparse import csr_matrix
+
+ from ..neighbors.index import fix_knn_query, instantiate_knn_index
+ from ..utils.compute import show_dask_progress
+
+ if len(d_array.shape) != 2:
+ raise ValueError("d_array must be two-dimensional")
+ n_genes = int(d_array.shape[0])
+ if n_genes < 2:
+ raise ValueError("At least two retained genes are required for clustering")
+ if isinstance(n_neighbours, (bool, np.bool_)) or not isinstance(
+ n_neighbours,
+ (int, np.integer),
+ ):
+ raise TypeError("n_neighbours must be an integer")
+ n_neighbours = int(n_neighbours)
+ if not 1 <= n_neighbours < n_genes:
+ raise ValueError(f"n_neighbours must satisfy 1 <= n_neighbours < {n_genes}")
+ if isinstance(n_clusters, (bool, np.bool_)) or not isinstance(
+ n_clusters,
+ (int, np.integer),
+ ):
+ raise TypeError("n_clusters must be an integer")
+ n_clusters = int(n_clusters)
+ if not 1 <= n_clusters <= n_genes:
+ raise ValueError(f"n_clusters must satisfy 1 <= n_clusters <= {n_genes}")
+
+ def make_knn_mat(
+ data: "scarf.matrix.ChunkedArray",
+ k: int,
+ threads: int,
+ ) -> "csr_matrix":
+ logger.debug("Pseudotime modules: fitting feature KNN")
+ for block in data.stream_blocks(nthreads=threads, msg="Fitting feature KNN"):
+ if not np.isfinite(block).all():
+ raise ValueError("Feature profiles must contain only finite values")
+ ann_idx.add_items(block)
+ start, end = 0, 0
+ neighbor_indices: list[np.ndarray] = []
+ logger.debug("Pseudotime modules: querying feature KNN")
+ for block in data.stream_blocks(
+ nthreads=threads,
+ msg="Identifying feature neighbors",
+ ):
+ end += block.shape[0]
+ indices, distances = ann_idx.knn_query(block, k=k + 1)
+ indices, _, _ = fix_knn_query(
+ indices,
+ distances,
+ np.arange(start, end),
+ )
+ neighbor_indices.append(indices)
+ start = end
+ indices_mat = np.vstack(neighbor_indices)
+ assert indices_mat.shape[0] == data.shape[0]
+
+ return csr_matrix(
+ (
+ np.ones(indices_mat.shape[0] * indices_mat.shape[1]),
+ (
+ np.repeat(
+ np.arange(indices_mat.shape[0]),
+ indices_mat.shape[1],
+ ),
+ indices_mat.flatten(),
+ ),
+ ),
+ shape=(indices_mat.shape[0], indices_mat.shape[0]),
+ )
+
+ def make_clusters(matrix: "csr_matrix", n_cluster: int) -> np.ndarray:
+ from ..clustering.paris import (
+ fit_paris_hierarchy,
+ hierarchy_to_dendrogram,
+ straight_cut,
+ )
+
+ logger.debug("Pseudotime modules: clustering modules")
+ hierarchy = fit_paris_hierarchy(
+ matrix,
+ n_threads=n_threads,
+ )
+ dendrogram = hierarchy_to_dendrogram(hierarchy)
+ return straight_cut(dendrogram, n_cluster)
+
+ def fix_cluster_order(
+ data: "scarf.matrix.ChunkedArray",
+ clusters: np.ndarray,
+ threads: int,
+ ) -> np.ndarray:
+ idxmax = show_dask_progress(
+ data.argmax(axis=1),
+ "Sorting clusters",
+ threads,
+ )
+ cluster_medians = (
+ pd.DataFrame([idxmax, clusters]).T.groupby(1).median()[0].sort_values()
+ )
+ return np.asarray(
+ pd.Series(clusters)
+ .replace(
+ dict(
+ zip(
+ cluster_medians.index,
+ range(1, 1 + len(cluster_medians)),
+ strict=True,
+ )
+ )
+ )
+ .values
+ )
+
+ default_ann_params: dict[str, Any] = {
+ "space": "l2",
+ "dim": d_array.shape[1],
+ "max_elements": d_array.shape[0],
+ "ef_construction": 80,
+ "M": 50,
+ "random_seed": 444,
+ "ef": 80,
+ "num_threads": 1,
+ }
+ if ann_params is None:
+ ann_params = {}
+ default_ann_params.update(ann_params)
+ ann_idx = instantiate_knn_index(
+ space=str(default_ann_params["space"]),
+ dim=int(default_ann_params["dim"]),
+ max_elements=int(default_ann_params["max_elements"]),
+ ef_construction=int(default_ann_params["ef_construction"]),
+ M=int(default_ann_params["M"]),
+ random_seed=int(default_ann_params["random_seed"]),
+ ef=int(default_ann_params["ef"]),
+ num_threads=int(default_ann_params["num_threads"]),
+ )
+ return fix_cluster_order(
+ d_array,
+ make_clusters(
+ make_knn_mat(d_array, n_neighbours, n_threads),
+ n_clusters,
+ ),
+ n_threads,
+ )
diff --git a/scarf/trajectory/pseudotime.py b/scarf/trajectory/pseudotime.py
new file mode 100644
index 00000000..81d88e22
--- /dev/null
+++ b/scarf/trajectory/pseudotime.py
@@ -0,0 +1,178 @@
+from typing import Any, Literal
+
+import numpy as np
+import pandas as pd
+from scipy.sparse import csr_matrix
+from scipy.sparse.csgraph import connected_components
+from scipy.sparse.linalg import ArpackNoConvergence, svds
+
+from ..utils.logging import logger
+
+
+def validate_source_sink_labels(
+ labels: pd.Series,
+ sources: list[Any],
+ sinks: list[Any],
+ context: str,
+) -> None:
+ overlap = sorted(set(sources) & set(sinks))
+ if overlap:
+ raise ValueError(f"Source and sink labels overlap in {context}: {overlap}")
+
+ present = set(pd.unique(labels))
+ missing_sources = [source for source in sources if source not in present]
+ missing_sinks = [sink for sink in sinks if sink not in present]
+ if missing_sources or missing_sinks:
+ raise ValueError(
+ f"Source/sink labels were not found in {context}. "
+ f"Missing sources: {missing_sources}; missing sinks: {missing_sinks}"
+ )
+
+
+def make_source_sink_vector(
+ labels: pd.Series,
+ sources: list[Any],
+ sinks: list[Any],
+) -> np.ndarray:
+ sink_mask = labels.isin(sinks).to_numpy(dtype=bool)
+ source_mask = labels.isin(sources).to_numpy(dtype=bool)
+ labelled = sink_mask | source_mask
+ if labelled.all():
+ raise ValueError(
+ "All selected cells are labelled as sources or sinks, so the "
+ "source/sink vector cannot be balanced over unlabelled cells"
+ )
+
+ vector = np.zeros(labels.shape[0], dtype=float)
+ vector[sink_mask] = 1.0
+ vector[source_mask] = -1.0
+ vector[~labelled] = -vector.sum() / int((~labelled).sum())
+ return vector
+
+
+def validate_source_sink_vector(
+ values: np.ndarray,
+ n_cells: int,
+ context: str,
+) -> np.ndarray:
+ try:
+ vector = np.asarray(values, dtype=float)
+ except (TypeError, ValueError) as exc:
+ raise TypeError(f"{context} must contain numeric values") from exc
+
+ if vector.ndim == 2 and vector.shape == (n_cells, 1):
+ vector = vector[:, 0]
+ elif vector.ndim != 1:
+ raise ValueError(
+ f"{context} must be one-dimensional or have shape ({n_cells}, 1)"
+ )
+
+ if vector.shape[0] != n_cells:
+ raise ValueError(
+ f"Size mismatch between {context} ({vector.shape[0]}) and graph ({n_cells})"
+ )
+ if not np.isfinite(vector).all():
+ raise ValueError(f"{context} must contain only finite values")
+
+ tolerance = 1e-10 * max(1.0, float(np.abs(vector).sum()))
+ if not np.isclose(vector.sum(), 0.0, atol=tolerance, rtol=0.0):
+ raise ValueError(f"The values in {context} must sum to zero")
+ return vector
+
+
+def select_pseudotime_component(
+ graph: csr_matrix,
+ selected_cell_indices: np.ndarray,
+ component_policy: Literal["largest", "error"],
+) -> tuple[np.ndarray, list[int]]:
+ if component_policy not in {"largest", "error"}:
+ raise ValueError("component_policy must be either 'largest' or 'error'")
+
+ n_components, component_labels = connected_components(
+ graph,
+ directed=False,
+ return_labels=True,
+ )
+ sizes = np.bincount(component_labels, minlength=n_components).astype(int).tolist()
+ if n_components == 1:
+ return np.ones(graph.shape[0], dtype=bool), sizes
+ if component_policy == "error":
+ raise ValueError(
+ f"The selected graph has {n_components} connected components with sizes {sizes}"
+ )
+
+ retained_component = min(
+ range(n_components),
+ key=lambda component_id: (
+ -sizes[component_id],
+ int(selected_cell_indices[component_labels == component_id].min()),
+ ),
+ )
+ return np.asarray(component_labels == retained_component, dtype=bool), sizes
+
+
+def random_walk_laplacian_transpose(graph: csr_matrix) -> csr_matrix:
+ degree = np.asarray(graph.sum(axis=1), dtype=float).ravel()
+ if np.any(degree <= 0):
+ raise ValueError("The retained graph contains isolated cells")
+ n_cells = graph.shape[0]
+ inverse_degree = csr_matrix(
+ (1.0 / degree, (range(n_cells), range(n_cells))),
+ shape=(n_cells, n_cells),
+ )
+ identity = csr_matrix(
+ (np.ones(n_cells), (range(n_cells), range(n_cells))),
+ shape=(n_cells, n_cells),
+ )
+ return identity - graph.dot(inverse_degree)
+
+
+def truncated_pba_potential(
+ laplacian_transpose: csr_matrix,
+ n_singular_vals: int,
+ random_seed: int,
+ source_sink: np.ndarray,
+) -> np.ndarray:
+ random_state = np.random.RandomState(random_seed)
+ initial_vector = random_state.rand(laplacian_transpose.shape[0])
+ logger.debug(
+ f"Pseudotime scoring: calculating SVD "
+ f"(shape={laplacian_transpose.shape}, nnz={laplacian_transpose.nnz}, "
+ f"k={n_singular_vals})"
+ )
+ try:
+ left_vectors, singular_values, right_vectors_t = svds(
+ laplacian_transpose,
+ k=n_singular_vals,
+ which="SM",
+ v0=initial_vector,
+ )
+ except ArpackNoConvergence as exc:
+ raise RuntimeError(
+ "Pseudotime SVD did not converge. Try a smaller n_singular_vals value"
+ ) from exc
+
+ order = np.argsort(singular_values)
+ singular_values = singular_values[order]
+ left_vectors = left_vectors[:, order]
+ right_vectors = right_vectors_t[order, :].T
+
+ rank_tolerance = (
+ max(laplacian_transpose.shape)
+ * np.finfo(singular_values.dtype).eps
+ * max(1.0, float(singular_values.max()))
+ )
+ if singular_values[0] > max(1e-8, rank_tolerance):
+ raise ValueError("The graph Laplacian does not contain the expected null mode")
+ if np.any(singular_values[1:] <= rank_tolerance):
+ raise ValueError(
+ "The graph Laplacian contains additional near-zero singular modes"
+ )
+
+ inverse_singular_values = 1.0 / singular_values[1:]
+ left_vectors = left_vectors[:, 1:]
+ right_vectors = right_vectors[:, 1:]
+ return np.asarray(
+ left_vectors
+ @ (inverse_singular_values * (right_vectors.T @ source_sink).ravel())
+ )
diff --git a/scarf/trajectory/results.py b/scarf/trajectory/results.py
new file mode 100644
index 00000000..79473411
--- /dev/null
+++ b/scarf/trajectory/results.py
@@ -0,0 +1,119 @@
+from dataclasses import dataclass, field
+from typing import Any
+
+import numpy as np
+import pandas as pd
+
+from ..matrix import ChunkedArray
+
+
+@dataclass(frozen=True, slots=True, eq=False)
+class FateMappingResult:
+ """Saved fate probabilities aligned to cells selected by result_cell_key."""
+
+ fate_keys: tuple[str, ...]
+ validity_key: str
+ sink_labels: tuple[Any, ...]
+ assay: str
+ graph_cell_key: str
+ result_cell_key: str
+ feature_key: str
+ pseudotime_key: str
+ sink_key: str
+ values: np.ndarray = field(repr=False)
+ valid: np.ndarray = field(repr=False)
+
+ def __post_init__(self) -> None:
+ if self.values.ndim != 2:
+ raise ValueError("Fate probabilities must be two-dimensional")
+ if self.valid.ndim != 1:
+ raise ValueError("Fate validity must be one-dimensional")
+ if self.values.shape[0] != self.valid.shape[0]:
+ raise ValueError("Fate probabilities and validity rows must align")
+ n_sinks = self.values.shape[1]
+ if n_sinks != len(self.fate_keys) or n_sinks != len(self.sink_labels):
+ raise ValueError(
+ "Fate probability columns, keys, and sink labels must align"
+ )
+
+
+@dataclass(frozen=True, slots=True, eq=False)
+class PseudotimeScoreResult:
+ """Saved pseudotime values and their metadata keys."""
+
+ pseudotime_key: str
+ validity_key: str
+ assay: str
+ graph_cell_key: str
+ result_cell_key: str
+ feature_key: str
+ values: np.ndarray = field(repr=False)
+ valid: np.ndarray = field(repr=False)
+
+ def __post_init__(self) -> None:
+ if self.values.ndim != 1 or self.valid.ndim != 1:
+ raise ValueError("Pseudotime values and validity must be one-dimensional")
+ if self.values.shape != self.valid.shape:
+ raise ValueError("Pseudotime values and validity must have the same shape")
+
+
+@dataclass(frozen=True, slots=True, eq=False)
+class PseudotimeMarkerResult:
+ """Pseudotime correlation table and saved feature-metadata keys."""
+
+ table: pd.DataFrame = field(repr=False)
+ correlation_key: str
+ p_value_key: str
+ assay: str
+ cell_key: str
+ feature_key: str
+ pseudotime_key: str
+
+ def __post_init__(self) -> None:
+ required = {
+ "feature_index",
+ "feature_name",
+ "r_value",
+ "p_value",
+ }
+ missing = required.difference(self.table.columns)
+ if missing:
+ raise ValueError(
+ "Pseudotime marker table is missing columns: "
+ + ", ".join(sorted(missing))
+ )
+
+ @property
+ def p_value_adjusted_key(self) -> str | None:
+ if (
+ "p_value_adjusted" not in self.table.columns
+ or not self.p_value_key.endswith("__p")
+ ):
+ return None
+ return f"{self.p_value_key[:-3]}__padj"
+
+
+@dataclass(frozen=True, slots=True, eq=False)
+class PseudotimeAggregationResult:
+ """Lazy pseudotime aggregation with aligned feature metadata."""
+
+ data: ChunkedArray = field(repr=False)
+ feature_indices: np.ndarray = field(repr=False)
+ feature_clusters: np.ndarray = field(repr=False)
+ cluster_key: str
+ storage_path: str
+ assay: str
+ cell_key: str
+ feature_key: str
+ pseudotime_key: str
+
+ def __post_init__(self) -> None:
+ if len(self.data.shape) != 2:
+ raise ValueError("Pseudotime aggregation data must be two-dimensional")
+ if self.feature_indices.ndim != 1 or self.feature_clusters.ndim != 1:
+ raise ValueError("Feature indices and clusters must be one-dimensional")
+ n_features = int(self.data.shape[0])
+ if len(self.feature_indices) != n_features:
+ raise ValueError("Feature indices do not align with aggregation rows")
+ if len(self.feature_clusters) != n_features:
+ raise ValueError("Feature clusters do not align with aggregation rows")
diff --git a/scarf/utils.py b/scarf/utils.py
deleted file mode 100644
index 3915728f..00000000
--- a/scarf/utils.py
+++ /dev/null
@@ -1,276 +0,0 @@
-"""Utility methods.
-
-- Methods:
- - clean_array: returns input array with nan and infinite values removed
- - controlled_compute: performs computation with Dask
- - rescale_array: performs edge trimming on values of the input vector
- - show_progress: performs computation with Dask and shows progress bar
- - system_call: executes a command in the underlying operative system
- - rolling_window: applies rolling window mean over a vector
-"""
-
-import sys
-from typing import List, Optional, TypeAlias, Union
-
-import numpy as np
-import zarr
-from dask.array.core import Array
-from loguru import logger
-from numba import jit
-from tqdm.dask import TqdmCallback
-
-__all__ = [
- "logger",
- "tqdmbar",
- "tqdm_params",
- "set_verbosity",
- "get_log_level",
- "system_call",
- "rescale_array",
- "clean_array",
- "load_zarr",
- "permute_into_chunks",
- "show_dask_progress",
- "controlled_compute",
- "rolling_window",
-]
-
-logger.remove()
-logger.add(
- sys.stdout, colorize=True, format="{level} : {message}", level="INFO"
-)
-
-tqdm_params = {
- "bar_format": "{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]",
- "ncols": 500,
- "colour": "#34abeb",
-}
-
-ZARRLOC: TypeAlias = Union[str, zarr.LRUStoreCache]
-
-
-def get_log_level():
- # noinspection PyUnresolvedReferences
- return logger._core.min_level # type: ignore
-
-
-def is_notebook() -> bool:
- try:
- shell = get_ipython().__class__.__name__ # type: ignore
- if shell == "ZMQInteractiveShell":
- return True
- elif shell == "TerminalInteractiveShell":
- return False
- else:
- return False
- except NameError:
- return False
-
-
-def tqdmbar(*args, **kwargs):
- params = dict(tqdm_params)
- for i in kwargs:
- if i in params:
- del params[i]
- if "disable" not in kwargs and "disable" not in params:
- if get_log_level() <= 20:
- params["disable"] = False
- else:
- params["disable"] = True
- if is_notebook():
- from tqdm import tqdm_notebook
-
- return tqdm_notebook(*args, **kwargs, **params)
- else:
- from tqdm.auto import tqdm
-
- return tqdm(*args, **kwargs, **params)
-
-
-def set_verbosity(level: Optional[str] = None, filepath: Optional[str] = None):
- """Set verbosity level of Scarf's output. Setting value of level='CRITICAL'
- should silence all logs. Progress bars are automatically disabled for
- levels above 'INFO'.
-
- Args:
- level: A valid level name. Run without any parameter to see available options
- filepath: The output file path. All logs will be saved to this file. If no file path is
- is provided then all the logs are printed on standard output.
-
- Returns:
- """
- # noinspection PyUnresolvedReferences
- available_levels = logger._core.levels.keys() # type: ignore
-
- if level is None or level not in available_levels:
- raise ValueError(
- f"Please provide a value for level: {', '.join(available_levels)}"
- )
- logger.remove()
- if filepath is None:
- filepath = sys.stdout # type: ignore
- logger.add(
- filepath, # type: ignore
- colorize=True,
- format="{level} : {message}",
- level=level,
- )
-
-
-def rescale_array(a: np.ndarray, frac: float = 0.9) -> np.ndarray:
- """Performs edge trimming on values of the input vector.
-
- Performs edge trimming on values of the input vector and constraints them between within frac and 1-frac density of
- normal distribution created with the sample mean and std. dev. of a.
-
- Args:
- a: numeric vector
- frac: Value between 0 and 1.
-
- Return:
- The input array, edge trimmed and constrained.
- """
- from scipy.stats import norm
-
- loc = (np.median(a) + np.median(a[::-1])) / 2
- dist = norm(loc, np.std(a))
- minv, maxv = dist.ppf(1 - frac), dist.ppf(frac)
- a[a < minv] = minv
- a[a > maxv] = maxv
- return a
-
-
-def clean_array(x, fill_val: int = 0):
- """Returns input array with nan and infinite values removed.
-
- Args:
- x (np.ndarray): input array
- fill_val: value to fill zero values with (default: 0)
- """
- x = np.nan_to_num(x, copy=True)
- x[(x == np.inf) | (x == -np.inf)] = 0
- x[x == 0] = fill_val
- return x
-
-
-def load_zarr(
- zarr_loc: ZARRLOC, mode: str, synchronizer: Optional[zarr.ThreadSynchronizer] = None
-) -> zarr.Group:
- if synchronizer is None:
- synchronizer = zarr.ThreadSynchronizer()
- if isinstance(zarr_loc, str):
- return zarr.open_group(zarr_loc, mode=mode, synchronizer=synchronizer)
- else:
- return zarr.group(zarr_loc, synchronizer=synchronizer)
-
-
-def controlled_compute(arr, nthreads):
- """Performs computation with Dask.
-
- Args:
- arr:
- nthreads: number of threads to use for computation
-
- Returns:
- Result of computation.
- """
- import dask
-
- try:
- # Multiprocessing may be faster, but it throws exception if SemLock is not implemented.
- # For example, multiprocessing won't work on AWS Lambda, in those scenarios we switch ThreadPoolExecutor
- from multiprocessing.pool import ThreadPool
-
- pool = ThreadPool(nthreads)
- except Exception:
- from concurrent.futures import ThreadPoolExecutor
-
- pool = ThreadPoolExecutor(nthreads)
-
- with dask.config.set(schedular="threads", pool=pool): # type: ignore
- res = arr.compute()
- return res
-
-
-def show_dask_progress(arr: Array, msg: Optional[str] = None, nthreads: int = 1):
- """Performs computation with Dask and shows progress bar.
-
- Args:
- arr: A Dask array
- msg: message to log, default None
- nthreads: number of threads to use for computation, default 1
-
- Returns:
- Result of computation.
- """
-
- params = dict(tqdm_params)
- if "disable" not in params:
- if get_log_level() <= 20:
- params["disable"] = False
- else:
- params["disable"] = True
- with TqdmCallback(desc=msg, **params):
- res = controlled_compute(arr, nthreads)
- return res
-
-
-def system_call(command):
- """Executes a command in the underlying operative system."""
- import shlex
- import subprocess
-
- process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE)
- while True:
- output = process.stdout.readline() # type: ignore
- if process.poll() is not None:
- break
- if output:
- logger.info(output.strip())
- process.poll()
- return None
-
-
-@jit(nopython=True)
-def rolling_window(a, w):
- n, m = a.shape
- b = np.zeros(shape=(n, m))
- for i in range(n):
- if i < w:
- x = i
- y = w - i
- elif (n - i) < w:
- x = w - (n - i)
- y = n - i
- else:
- x = w // 2
- y = w // 2
- x = i - x
- y = i + y
- for j in range(m):
- b[i, j] = a[x:y, j].mean()
- return b
-
-
-def permute_into_chunks(size: int, chunk_size: int, seed: int = 42) -> List[np.ndarray]:
- """
- Permute the chunks of an array of the given size.
- Args:
- size: The size of the array to be permuted
- chunk_size: The size of the chunks to permute
- Returns:
- A permuted array of the given size
- Examples:
- >>> permute_into_chunks(10, 3)
- [array([2, 1, 0]), array([3, 5, 4]), array([7, 8, 6]), array([9])]
- """
- rng = np.random.default_rng(seed=seed)
- arr = np.arange(size)
- start = 0
- end = len(arr) - len(arr) % chunk_size
- chunks = [arr[i : i + chunk_size] for i in range(start, end, chunk_size)]
- p_values = [rng.permutation(chunk) for chunk in chunks]
- # add the remaining elements
- if end < len(arr):
- p_values.append(rng.permutation(arr[end:]))
- return p_values
diff --git a/scarf/utils/__init__.py b/scarf/utils/__init__.py
new file mode 100644
index 00000000..a85f7942
--- /dev/null
+++ b/scarf/utils/__init__.py
@@ -0,0 +1,98 @@
+from importlib import import_module as _import_module
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from .arrays import (
+ array_digest as array_digest,
+ clean_array as clean_array,
+ permute_into_chunks as permute_into_chunks,
+ rescale_array as rescale_array,
+ rolling_window as rolling_window,
+ )
+ from .compute import (
+ controlled_compute as controlled_compute,
+ show_dask_progress as show_dask_progress,
+ )
+ from .logging import (
+ configure_output as configure_output,
+ get_log_level as get_log_level,
+ logger as logger,
+ set_verbosity as set_verbosity,
+ )
+ from .prefetch import (
+ iter_column_blocks as iter_column_blocks,
+ )
+ from .process import (
+ process_rss_mb as process_rss_mb,
+ rss_peak_tracker as rss_peak_tracker,
+ system_call as system_call,
+ )
+ from .progress import (
+ tqdm_params as tqdm_params,
+ tqdmbar as tqdmbar,
+ )
+ from ..storage.stores import load_zarr as load_zarr
+
+__all__ = [
+ "logger",
+ "tqdmbar",
+ "tqdm_params",
+ "configure_output",
+ "set_verbosity",
+ "get_log_level",
+ "system_call",
+ "rescale_array",
+ "clean_array",
+ "load_zarr",
+ "permute_into_chunks",
+ "show_dask_progress",
+ "controlled_compute",
+ "iter_column_blocks",
+ "process_rss_mb",
+ "rss_peak_tracker",
+ "array_digest",
+ "rolling_window",
+]
+
+_LAZY_EXPORTS: dict[str, tuple[str, str]] = {
+ "logger": (".logging", "logger"),
+ "tqdmbar": (".progress", "tqdmbar"),
+ "tqdm_params": (".progress", "tqdm_params"),
+ "configure_output": (".logging", "configure_output"),
+ "set_verbosity": (".logging", "set_verbosity"),
+ "get_log_level": (".logging", "get_log_level"),
+ "system_call": (".process", "system_call"),
+ "rescale_array": (".arrays", "rescale_array"),
+ "clean_array": (".arrays", "clean_array"),
+ "load_zarr": ("..storage.stores", "load_zarr"),
+ "permute_into_chunks": (".arrays", "permute_into_chunks"),
+ "show_dask_progress": (".compute", "show_dask_progress"),
+ "controlled_compute": (".compute", "controlled_compute"),
+ "iter_column_blocks": (".prefetch", "iter_column_blocks"),
+ "process_rss_mb": (".process", "process_rss_mb"),
+ "rss_peak_tracker": (".process", "rss_peak_tracker"),
+ "array_digest": (".arrays", "array_digest"),
+ "rolling_window": (".arrays", "rolling_window"),
+ "stdout_is_interactive": (".logging", "stdout_is_interactive"),
+ "is_notebook": (".progress", "is_notebook"),
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+
+def __getattr__(name: str) -> Any:
+ export = _LAZY_EXPORTS.get(name)
+ if export is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ module_name, attribute_name = export
+ value = getattr(_import_module(module_name, __name__), attribute_name)
+ if name in __all__ and hasattr(value, "__module__"):
+ value.__module__ = __name__
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()).union(_LAZY_EXPORTS))
diff --git a/scarf/utils/arrays.py b/scarf/utils/arrays.py
new file mode 100644
index 00000000..dc227c13
--- /dev/null
+++ b/scarf/utils/arrays.py
@@ -0,0 +1,196 @@
+import hashlib
+from typing import Any
+
+import numpy as np
+from numpy.typing import NDArray
+
+
+def checked_sparse_cast(values: np.ndarray, dtype: Any) -> np.ndarray:
+ destination_dtype = np.dtype(dtype)
+ if destination_dtype.kind in "biu" and values.size:
+ if values.dtype.kind == "c":
+ raise OverflowError(
+ "Complex sparse values cannot use an integer destination dtype"
+ )
+ if values.dtype.kind == "f":
+ if (
+ not np.isfinite(values).all()
+ or not np.equal(values, np.trunc(values)).all()
+ ):
+ raise OverflowError(
+ "Sparse values cannot be represented by the destination dtype"
+ )
+ if destination_dtype.kind == "b":
+ lower, upper = 0, 1
+ else:
+ limits = np.iinfo(destination_dtype)
+ lower, upper = limits.min, limits.max
+ if values.min() < lower or values.max() > upper:
+ raise OverflowError("Sparse values exceed the destination dtype")
+ return values.astype(destination_dtype, copy=False)
+
+
+def _canonical_64bit_integer_sparse(coo: Any) -> Any:
+ from scipy.sparse import coo_matrix
+
+ row = np.asarray(coo.row)
+ column = np.asarray(coo.col)
+ data = np.asarray(coo.data)
+ if data.size == 0:
+ canonical = coo_matrix(coo.shape, dtype=data.dtype)
+ canonical.has_canonical_format = True
+ return canonical
+ order = np.lexsort((column, row))
+ row = row[order]
+ column = column[order]
+ data = data[order]
+ starts = np.flatnonzero(
+ np.concatenate(
+ (
+ np.array([True]),
+ (row[1:] != row[:-1]) | (column[1:] != column[:-1]),
+ )
+ )
+ )
+ ends = np.append(starts[1:], data.size)
+ if starts.size == data.size:
+ summed = data
+ else:
+ limits = np.iinfo(data.dtype)
+ summed = np.empty(starts.size, dtype=data.dtype)
+ for index, (start, end) in enumerate(zip(starts, ends, strict=True)):
+ total = sum(int(value) for value in data[start:end])
+ if total < limits.min or total > limits.max:
+ raise OverflowError(
+ "Duplicate sparse values exceed supported integer dtypes"
+ )
+ summed[index] = total
+ canonical = coo_matrix(
+ (summed, (row[starts], column[starts])),
+ shape=coo.shape,
+ )
+ canonical.has_canonical_format = True
+ return canonical
+
+
+def canonicalize_sparse(coo: Any, dtype: Any | None = None) -> Any:
+ from scipy.sparse import coo_matrix
+
+ if bool(getattr(coo, "has_canonical_format", False)):
+ if dtype is not None:
+ coo.data = checked_sparse_cast(np.asarray(coo.data), dtype)
+ return coo
+ data = np.asarray(coo.data)
+ if data.dtype.kind in "biu":
+ if data.dtype.itemsize >= 8:
+ canonical = _canonical_64bit_integer_sparse(coo)
+ else:
+ accumulator_dtype = np.uint64 if data.dtype.kind in "bu" else np.int64
+ data = data.astype(accumulator_dtype)
+ canonical = coo_matrix(
+ (data, (coo.row, coo.col)),
+ shape=coo.shape,
+ )
+ canonical.sum_duplicates()
+ else:
+ canonical = coo_matrix(
+ (data, (coo.row, coo.col)),
+ shape=coo.shape,
+ )
+ canonical.sum_duplicates()
+ if dtype is not None:
+ canonical.data = checked_sparse_cast(canonical.data, dtype)
+ return canonical
+
+
+def rescale_array(a: np.ndarray, frac: float = 0.9) -> np.ndarray:
+ """Trim extreme values using a fitted normal distribution."""
+ from scipy.stats import norm
+
+ location = (np.median(a) + np.median(a[::-1])) / 2
+ distribution = norm(location, np.std(a))
+ minimum, maximum = distribution.ppf(1 - frac), distribution.ppf(frac)
+ a[a < minimum] = minimum
+ a[a > maximum] = maximum
+ return a
+
+
+def clean_array(
+ x: NDArray[Any] | list[Any],
+ fill_val: int | float = 0,
+) -> NDArray[Any]:
+ """Replace non-finite and zero values in a numeric array."""
+ array = np.asarray(x, dtype=np.float64)
+ array = np.nan_to_num(array, copy=True)
+ array[(array == np.inf) | (array == -np.inf)] = 0
+ array[array == 0] = fill_val
+ return array
+
+
+def array_digest(values: np.ndarray) -> str:
+ """Return a deterministic digest for a non-object NumPy array."""
+ array = np.ascontiguousarray(values)
+ if array.dtype.hasobject:
+ raise TypeError("Cannot create a deterministic digest for object arrays")
+ digest = hashlib.blake2b(digest_size=16)
+ digest.update(array.dtype.str.encode())
+ digest.update(np.asarray(array.shape, dtype=np.int64).tobytes())
+ digest.update(array.view(np.uint8).tobytes())
+ return digest.hexdigest()
+
+
+def permute_into_chunks(
+ size: int,
+ chunk_size: int,
+ seed: int = 42,
+) -> list[np.ndarray]:
+ """Split sequential indices into independently permuted chunks."""
+ rng = np.random.default_rng(seed=seed)
+ array = np.arange(size)
+ end = len(array) - len(array) % chunk_size
+ chunks = [array[index : index + chunk_size] for index in range(0, end, chunk_size)]
+ permuted = [rng.permutation(chunk) for chunk in chunks]
+ if end < len(array):
+ permuted.append(rng.permutation(array[end:]))
+ return permuted
+
+
+def _rolling_window_kernel(a: np.ndarray, w: int) -> np.ndarray:
+ if a.ndim != 2:
+ raise ValueError("a must be a two-dimensional array")
+ if w <= 0:
+ raise ValueError("w must be greater than zero")
+
+ n, m = a.shape
+ if n == 0:
+ raise ValueError("a must contain at least one row")
+ w = min(w, n)
+ left = (w - 1) // 2
+ right = w // 2
+ result = np.empty((n, m), dtype=np.float64)
+
+ for column in range(m):
+ cumulative = np.empty(n + 1, dtype=np.float64)
+ cumulative[0] = 0.0
+ for row in range(n):
+ cumulative[row + 1] = cumulative[row] + a[row, column]
+ for row in range(n):
+ start = max(0, row - left)
+ stop = min(n, row + right + 1)
+ result[row, column] = (cumulative[stop] - cumulative[start]) / (
+ stop - start
+ )
+ return result
+
+
+_rollingWindowImpl: Any = None
+
+
+def rolling_window(a: np.ndarray, w: int) -> np.ndarray:
+ """Apply a centered rolling mean along the first axis."""
+ global _rollingWindowImpl
+ if _rollingWindowImpl is None:
+ from numba import jit
+
+ _rollingWindowImpl = jit(nopython=True)(_rolling_window_kernel)
+ return np.asarray(_rollingWindowImpl(a, w))
diff --git a/scarf/utils/compute.py b/scarf/utils/compute.py
new file mode 100644
index 00000000..36e0d330
--- /dev/null
+++ b/scarf/utils/compute.py
@@ -0,0 +1,24 @@
+from typing import Any
+
+import numpy as np
+
+from .logging import progress_enabled
+
+
+def controlled_compute(arr: Any, nthreads: int) -> np.ndarray:
+ """Materialize a deferred array with a bounded thread count."""
+ if hasattr(arr, "compute"):
+ return np.asarray(arr.compute(nthreads))
+ return np.asarray(arr)
+
+
+def show_dask_progress(
+ arr: Any,
+ msg: str | None = None,
+ nthreads: int = 1,
+) -> np.ndarray:
+ """Materialize a deferred array while reporting progress."""
+ if hasattr(arr, "compute"):
+ progress_msg = msg if progress_enabled() else None
+ return np.asarray(arr.compute(nthreads, progress_msg))
+ return np.asarray(arr)
diff --git a/scarf/utils/logging.py b/scarf/utils/logging.py
new file mode 100644
index 00000000..b056b6e0
--- /dev/null
+++ b/scarf/utils/logging.py
@@ -0,0 +1,125 @@
+import sys
+from dataclasses import dataclass
+from typing import Any
+
+from loguru import logger
+
+
+@dataclass(slots=True)
+class _OutputConfig:
+ level: str = "INFO"
+ progress: bool = True
+ timestamps: bool = False
+ filepath: str | None = None
+
+
+_config = _OutputConfig()
+_handler_id: int | None = None
+
+
+def stdout_is_interactive() -> bool:
+ """Return whether stdout is an interactive terminal."""
+ return hasattr(sys.stdout, "isatty") and sys.stdout.isatty()
+
+
+def _flushing_stdout_sink(message: Any) -> None:
+ sys.stdout.write(str(message))
+ sys.stdout.flush()
+
+
+def _log_format() -> str:
+ if _config.timestamps:
+ return "{time:YYYY-MM-DDTHH:mm:ss.SSSZ} | {level} | {message}"
+ return "{level} : {message}"
+
+
+def _remove_scarf_handler() -> None:
+ global _handler_id
+ if _handler_id is None:
+ return
+ try:
+ logger.remove(_handler_id)
+ except ValueError:
+ pass
+ _handler_id = None
+
+
+def _install_scarf_handler() -> None:
+ global _handler_id
+ _remove_scarf_handler()
+ if _config.filepath is None:
+ _handler_id = logger.add(
+ _flushing_stdout_sink,
+ colorize=None,
+ format=_log_format(),
+ level=_config.level,
+ )
+ return
+ _handler_id = logger.add(
+ _config.filepath,
+ colorize=False,
+ format=_log_format(),
+ level=_config.level,
+ )
+
+
+def _resolve_level(level: str | None) -> str:
+ if level is None:
+ raise ValueError("Please provide a value for level recognized by Loguru")
+ try:
+ return logger.level(level).name
+ except ValueError as error:
+ raise ValueError(
+ f"Please provide a value for level recognized by Loguru: {level!r}"
+ ) from error
+
+
+try:
+ logger.remove(0)
+except ValueError:
+ pass
+_install_scarf_handler()
+
+
+def get_log_level() -> int:
+ """Return the current minimum Scarf log level."""
+ return logger.level(_config.level).no
+
+
+def progress_enabled() -> bool:
+ """Return whether Scarf progress reporting is enabled."""
+ return _config.progress
+
+
+def configure_output(
+ *,
+ level: str | None = None,
+ progress: bool | None = None,
+ timestamps: bool | None = None,
+) -> None:
+ """Configure Scarf's logging and progress output.
+
+ Unspecified settings retain their current values.
+ """
+ update_handler = False
+ if level is not None:
+ _config.level = _resolve_level(level)
+ update_handler = True
+ if progress is not None:
+ if not isinstance(progress, bool):
+ raise TypeError("progress must be a bool")
+ _config.progress = progress
+ if timestamps is not None:
+ if not isinstance(timestamps, bool):
+ raise TypeError("timestamps must be a bool")
+ _config.timestamps = timestamps
+ update_handler = True
+ if update_handler:
+ _install_scarf_handler()
+
+
+def set_verbosity(level: str | None = None, filepath: str | None = None) -> None:
+ """Set Scarf's logging level and optional output file."""
+ _config.level = _resolve_level(level)
+ _config.filepath = filepath
+ _install_scarf_handler()
diff --git a/scarf/utils/numba.py b/scarf/utils/numba.py
new file mode 100644
index 00000000..718417b1
--- /dev/null
+++ b/scarf/utils/numba.py
@@ -0,0 +1,18 @@
+from collections.abc import Callable
+from functools import wraps
+
+
+def restore_numba_threads[**P, R](fn: Callable[P, R]) -> Callable[P, R]:
+ """Restore Numba's process thread count after a call."""
+
+ @wraps(fn)
+ def wrapped(*args: P.args, **kwargs: P.kwargs) -> R:
+ import numba
+
+ previous = numba.get_num_threads()
+ try:
+ return fn(*args, **kwargs)
+ finally:
+ numba.set_num_threads(previous)
+
+ return wrapped
diff --git a/scarf/utils/prefetch.py b/scarf/utils/prefetch.py
new file mode 100644
index 00000000..14adf0f5
--- /dev/null
+++ b/scarf/utils/prefetch.py
@@ -0,0 +1,48 @@
+import time
+from collections.abc import Callable, Iterator
+
+import numpy as np
+
+from ..storage.budget import DEFAULT_READ_AHEAD_BLOCKS
+from ..storage.parallel import stream_shards
+from .logging import logger
+from .process import process_rss_mb
+
+
+def iter_column_blocks(
+ n_blocks: int,
+ read_block: Callable[[int], np.ndarray],
+ *,
+ workers: int = DEFAULT_READ_AHEAD_BLOCKS,
+ io_concurrency: int | None = None,
+ msg: str | None = None,
+) -> Iterator[tuple[int, np.ndarray, float, str]]:
+ """Yield column blocks in order with caller-planned read-ahead."""
+ if n_blocks <= 0:
+ return
+ worker_budget = max(1, int(workers))
+ outer_workers = min(worker_budget, n_blocks)
+ resolved_io = (
+ max(1, worker_budget // outer_workers)
+ if io_concurrency is None
+ else max(1, int(io_concurrency))
+ )
+
+ def timed_read(block_idx: int) -> tuple[int, np.ndarray, float, str]:
+ if msg:
+ logger.debug(
+ f"{msg}: reading block {block_idx + 1}/{n_blocks} "
+ f"(rss {process_rss_mb():.0f} MiB)"
+ )
+ started = time.perf_counter()
+ array = np.asarray(read_block(block_idx))
+ return block_idx, array, time.perf_counter() - started, "direct"
+
+ yield from stream_shards(
+ range(n_blocks),
+ timed_read,
+ workers=outer_workers,
+ io_concurrency=resolved_io,
+ msg=msg,
+ total=n_blocks,
+ )
diff --git a/scarf/utils/process.py b/scarf/utils/process.py
new file mode 100644
index 00000000..db830232
--- /dev/null
+++ b/scarf/utils/process.py
@@ -0,0 +1,55 @@
+import resource
+import shlex
+import subprocess
+import threading
+from collections.abc import Callable, Iterator
+from contextlib import contextmanager
+
+from .logging import logger
+
+
+def system_call(command: str) -> None:
+ """Run a command and forward its output to Scarf's logger."""
+ process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE)
+ while True:
+ output = process.stdout.readline() # type: ignore[union-attr]
+ if process.poll() is not None:
+ break
+ if output:
+ logger.debug(output.strip())
+ process.poll()
+
+
+def process_rss_mb() -> float:
+ """Return this process's resident memory in MiB."""
+ try:
+ with open("/proc/self/status") as handle:
+ for line in handle:
+ if line.startswith("VmRSS:"):
+ return int(line.split()[1]) / 1024.0
+ except OSError:
+ pass
+ return resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1024.0
+
+
+@contextmanager
+def rss_peak_tracker(
+ interval_s: float = 0.25,
+) -> Iterator[Callable[[], float]]:
+ """Track peak resident memory while the context is active."""
+ peak = process_rss_mb()
+ stop = threading.Event()
+
+ def sample_loop() -> None:
+ nonlocal peak
+ while not stop.wait(interval_s):
+ peak = max(peak, process_rss_mb())
+
+ thread = threading.Thread(target=sample_loop, daemon=True)
+ thread.start()
+ try:
+ yield lambda: peak
+ finally:
+ stop.set()
+ thread.join(timeout=2.0)
+ peak = max(peak, process_rss_mb())
diff --git a/scarf/utils/progress.py b/scarf/utils/progress.py
new file mode 100644
index 00000000..dddd1251
--- /dev/null
+++ b/scarf/utils/progress.py
@@ -0,0 +1,69 @@
+from collections.abc import Iterable, Iterator
+from typing import Any
+
+from .logging import progress_enabled
+
+tqdm_params = {
+ "bar_format": "{desc}: {percentage:3.0f}%| {bar} {n_fmt}/{total_fmt} [{elapsed}]",
+ "dynamic_ncols": True,
+ "colour": "#34abeb",
+}
+
+
+def is_notebook() -> bool:
+ """Return whether the current shell is a Jupyter kernel."""
+ try:
+ shell = get_ipython().__class__.__name__ # type: ignore[name-defined]
+ if shell == "ZMQInteractiveShell":
+ return True
+ if shell == "TerminalInteractiveShell":
+ return False
+ return False
+ except NameError:
+ return False
+
+
+def tqdmbar(*args: Any, **kwargs: Any) -> Any:
+ """Create a progress bar using Scarf's display defaults."""
+ params = dict(tqdm_params)
+ for name in kwargs:
+ if name in params:
+ del params[name]
+ if "disable" in kwargs:
+ kwargs["disable"] = not progress_enabled() or bool(kwargs["disable"])
+ else:
+ params["disable"] = not progress_enabled()
+ from tqdm.auto import tqdm
+
+ return tqdm(*args, **kwargs, **params)
+
+
+def iter_progress[T](
+ iterable: Iterable[T],
+ *,
+ desc: str | None = None,
+ total: int | None = None,
+ disable: bool | None = None,
+) -> Iterator[T]:
+ """Yield values while updating and reliably closing a progress bar."""
+ iterator = iter(iterable)
+ kwargs: dict[str, Any] = {"desc": desc, "total": total}
+ if disable is not None:
+ kwargs["disable"] = disable
+ progress = tqdmbar(**kwargs)
+ try:
+ while True:
+ try:
+ item = next(iterator)
+ except StopIteration:
+ return
+ try:
+ yield item
+ finally:
+ del item
+ progress.update()
+ finally:
+ close = getattr(iterator, "close", None)
+ if callable(close):
+ close()
+ progress.close()
diff --git a/scarf/writers.py b/scarf/writers.py
deleted file mode 100644
index be4fa1ee..00000000
--- a/scarf/writers.py
+++ /dev/null
@@ -1,1395 +0,0 @@
-"""Methods and classes for writing data to disk.
-
-- Methods:
- - create_zarr_dataset: Creates and returns a Zarr hierarchy/dataset.
- - create_zarr_obj_array: Creates and returns a Zarr object array.
- - create_zarr_count_assay: Creates and returns a Zarr array with name 'counts'.
- - subset_assay_zarr: Selects a subset of the data in an assay in the specified Zarr hierarchy.
- - dask_to_zarr: Creates a Zarr hierarchy from a Dask array.
- - to_h5ad: Convert a Zarr file to H5ad format
- - to_mtx: Convert a Zarr file to MTX format
-
-- Classes:
- - ZarrMerge: Merge multiple Zarr files into a single Zarr file.
- - SubsetZarr: Extracts a subset of cells from the given Zarr file and saves into a new Zarr file.
- - CrToZarr: A class for converting data in the Cellranger format to a Zarr hierarchy.
- - H5adToZarr: A class for converting data in the Cellranger Matrix Market format to a Zarr hierarchy.
- - NaboH5ToZarr: A class for converting data in a h5 file generated by Nabo, to a Zarr hierarchy.
- - LoomToZarr: A class for converting data in a Loom file to a Zarr hierarchy.
-"""
-
-import os
-from typing import Any, Tuple, List, Union, Dict, Optional
-
-import numpy as np
-import pandas as pd
-import polars as pl
-import zarr
-from scipy.sparse import csr_matrix, coo_matrix
-
-from .readers import CrReader, H5adReader, NaboH5Reader, LoomReader, CSVReader
-from .utils import (
- controlled_compute,
- logger,
- tqdmbar,
- show_dask_progress,
- load_zarr,
- ZARRLOC,
-)
-
-__all__ = [
- "create_zarr_dataset",
- "create_zarr_obj_array",
- "create_zarr_count_assay",
- "subset_assay_zarr",
- "dask_to_zarr",
- "SubsetZarr",
- "CrToZarr",
- "H5adToZarr",
- "NaboH5ToZarr",
- "LoomToZarr",
- "SparseToZarr",
- "to_h5ad",
- "to_mtx",
- "CSVtoZarr",
-]
-
-
-def create_zarr_dataset(
- g: zarr.Group,
- name: str,
- chunks: tuple,
- dtype: Any,
- shape: Tuple,
- overwrite: bool = True,
-) -> zarr.Array:
- """Creates and returns a Zarr array.
-
- Args:
- g (zarr.hierarchy):
- name (str):
- chunks (tuple):
- dtype (Any):
- shape (Tuple):
- overwrite (bool):
-
- Returns:
- A Zarr Array.
- """
- from numcodecs import Blosc
-
- compressor = Blosc(cname="lz4", clevel=5, shuffle=Blosc.BITSHUFFLE)
- return g.create_dataset(
- name,
- chunks=chunks,
- dtype=dtype,
- shape=shape,
- compressor=compressor,
- overwrite=overwrite,
- )
-
-
-def dtype_fix(dtype, data: np.ndarray):
- if dtype is None or dtype == object:
- return "U" + str(max([len(str(x)) for x in data]))
- if np.issubdtype(data.dtype, np.dtype("S")):
- try:
- adata = data.astype("U")
- except UnicodeDecodeError:
- adata = np.array([x.decode("UTF-8") for x in data]).astype("U")
- return adata.dtype
- return dtype
-
-
-def create_zarr_obj_array(
- g: zarr.Group,
- name: str,
- data,
- dtype: Union[str, Any] = None,
- overwrite: bool = True,
- chunk_size: int = 100000,
- shape: Optional[int] = None,
-) -> zarr.Array:
- """Creates and returns a Zarr object array.
-
- A Zarr object array can contain any type of object.
- https://zarr.readthedocs.io/en/stable/tutorial.html#object-arrays
-
- Args:
- g (zarr.hierarchy):
- name (str):
- data ():
- dtype (Union[str, Any]):
- overwrite (bool):
- chunk_size (int):
- shape:
-
- Returns:
- A Zarr object Array.
- """
-
- from numcodecs import Blosc
-
- compressor = Blosc(cname="lz4", clevel=5, shuffle=Blosc.BITSHUFFLE)
-
- if chunk_size is None or chunk_size is False:
- chunks = False
- else:
- chunks = (chunk_size,)
-
- if data is not None:
- data = np.array(data)
- dtype = dtype_fix(dtype, data)
-
- return g.create_dataset(
- name,
- data=data,
- chunks=chunks,
- shape=len(data),
- dtype=dtype,
- overwrite=overwrite,
- compressor=compressor,
- )
- else:
- return g.create_dataset(
- name,
- chunks=chunks,
- shape=shape,
- dtype=dtype,
- overwrite=overwrite,
- compressor=compressor,
- )
-
-
-def create_zarr_count_assay(
- z: zarr.Group,
- assay_name: str,
- workspace: Union[str, None],
- chunk_size: Tuple[int, int],
- n_cells: int,
- feat_ids: Union[np.ndarray, List[str]],
- feat_names: Union[np.ndarray, List[str]],
- dtype: str = "uint32",
-) -> zarr.Array:
- """Creates and returns a Zarr array with name 'counts'.
-
- Args:
- z (zarr.Group):
- assay_name (str):
- workspace (Union[str, None]):
- chunk_size (Tuple[int, int]):
- n_cells (int):
- feat_ids (Union[np.ndarray, List[str]]):
- feat_names (Union[np.ndarray, List[str]]):
- dtype (str = 'uint32'):
-
- Returns:
- A Zarr array.
- """
- if workspace is None:
- g = z.create_group(assay_name, overwrite=True)
- else:
- g = z.create_group(f"{workspace}/{assay_name}", overwrite=True)
- g.attrs["is_assay"] = True
- g.attrs["misc"] = {}
- create_zarr_obj_array(g, "featureData/ids", feat_ids)
- create_zarr_obj_array(g, "featureData/names", feat_names)
- create_zarr_obj_array(
- g, "featureData/I", [True for _ in range(len(feat_ids))], "bool"
- )
- if workspace is not None:
- g = z.create_group(f"matrices/{assay_name}", overwrite=True)
- return create_zarr_dataset(
- g, "counts", chunk_size, dtype, (n_cells, len(feat_ids)), overwrite=True
- )
-
-
-def load_count_store(
- z: zarr.Group, assay_name: str, workspace: Union[str, None]
-) -> zarr.Array:
- if workspace is None:
- return z[f"{assay_name}/counts"] # type: ignore
- else:
- return z[f"matrices/{assay_name}/counts"] # type: ignore
-
-
-def create_cell_data(
- z: zarr.Group, workspace: Union[str, None], ids: np.ndarray, names: np.ndarray
-) -> zarr.Group:
- if workspace is None:
- g = z.create_group("cellData")
- else:
- g = z.create_group(f"{workspace}/cellData")
- create_zarr_obj_array(g, "ids", ids, ids.dtype)
- create_zarr_obj_array(g, "names", names, names.dtype)
- create_zarr_obj_array(g, "I", [True for _ in range(len(ids))], "bool")
- return g
-
-
-def sparse_writer(store: zarr.Array, data_stream, n_cells: int, batch_size: int) -> int:
- (
- s,
- e,
- ) = (
- 0,
- 0,
- )
- n_chunks = n_cells // batch_size + 1
- for a in tqdmbar(data_stream, total=n_chunks):
- e += a.shape[0]
- store.set_coordinate_selection((a.row + s, a.col), a.data)
- s = e
- return e
-
-
-class CrToZarr:
- """A class for converting data in the Cellranger format to a Zarr
- hierarchy.
-
- Args:
- cr: A CrReader object, containing the Cellranger data.
- zarr_loc: The file name for the Zarr hierarchy or a store
- chunk_size: The requested size of chunks to load into memory and process.
- dtype: the dtype of the data.
-
- Attributes:
- cr: A CrReader object, containing the Cellranger data.
- chunkSizes: The requested size of chunks to load into memory and process.
- z: The Zarr hierarchy (array or group).
- """
-
- def __init__(
- self,
- cr: CrReader,
- zarr_loc: ZARRLOC,
- chunk_size=(1000, 1000),
- dtype: str = "uint32",
- workspace: Optional[str] = None,
- ):
- self.cr = cr
- self.chunkSizes = chunk_size
- self.workspace = workspace
- self.z = load_zarr(zarr_loc=zarr_loc, mode="w")
- create_cell_data(
- z=self.z,
- workspace=self.workspace,
- ids=np.array(self.cr.cell_names()),
- names=np.array(self.cr.cell_names()),
- )
- for assay_name in self.cr.assayFeats.columns:
- create_zarr_count_assay(
- z=self.z,
- assay_name=assay_name,
- workspace=workspace,
- chunk_size=chunk_size,
- n_cells=self.cr.nCells,
- feat_ids=self.cr.feature_ids(assay_name),
- feat_names=self.cr.feature_names(assay_name),
- dtype=dtype,
- )
-
- @staticmethod
- def _prep_assay_input_ranges(af: pd.DataFrame) -> Dict[str, List[List[int]]]:
- assay_order = (
- af.T.nFeatures.groupby(af.columns).sum().sort_values(ascending=False).index
- )
- ranges = {}
- for assay in assay_order:
- temp = []
- if len(af[assay].shape) == 2:
- for i in af[assay].values[1:3].T:
- temp.append([i[0], i[1]])
- else:
- idx = af[assay]
- temp = [[idx.start, idx.end]]
- ranges[assay] = temp
- return ranges
-
- @staticmethod
- def _prep_feat_index_offset(
- ranges: Dict[str, List[List[int]]]
- ) -> Dict[str, List[int]]:
- feat_offset = {}
- for i in ranges:
- feat_offset[i] = []
- lv = 0
- for j in ranges[i]:
- feat_offset[i].append(-j[0] + lv)
- lv += j[1] - j[0]
- return feat_offset
-
- def dump(self, batch_size: int = 1000, lines_in_mem: int = 100000) -> None:
- """Writes the count values into the Zarr matrix.
-
- Args:
- batch_size: Number of cells to save at a time. (Default value: 1000)
- lines_in_mem: Number of lines to read at a time from MTX file (only used for CrDirReader)
- (Default value: 100000)
-
- Raises:
- AssertionError: Catches eventual bugs in the class, if number of cells does not match after transformation.
-
- Returns:
- None
- """
- input_ranges = self._prep_assay_input_ranges(self.cr.assayFeats)
- stores = {x: load_count_store(self.z, x, self.workspace) for x in input_ranges}
- feat_offset = self._prep_feat_index_offset(input_ranges)
- s = 0
- n_chunks = self.cr.nCells // batch_size + 1
- for a in tqdmbar(self.cr.consume(batch_size, lines_in_mem), total=n_chunks):
- for assay in input_ranges:
- idx = np.zeros(a.col.shape[0]).astype(bool)
- feat_coords = a.col.copy()
- for r, of in zip(input_ranges[assay], feat_offset[assay]):
- temp = (a.col >= r[0]) & (a.col < r[1])
- if of != 0:
- feat_coords[temp] = (
- feat_coords[temp] + of
- ) # of is already a negative value
- idx = idx | temp
- if idx.sum() > 0:
- stores[assay].set_coordinate_selection(
- (s + a.row[idx], feat_coords[idx]), a.data[idx]
- )
- else:
- logger.warning(
- f"No feature captured from chunk {s} to {s+a.shape[0]} for assay: {assay}"
- )
- s += a.shape[0]
- if s != self.cr.nCells:
- raise AssertionError(
- "ERROR: This is a bug in CrToZarr. All cells might not have been successfully "
- "written into the zarr file. Please report this issue"
- )
-
-
-class H5adToZarr:
- """A class for converting data in anndata's H5ad format to Zarr hierarchy.
-
- Args:
- h5ad: A H5adReader object, containing the Cellranger data.
- zarr_loc: The file name for the Zarr hierarchy or a store
- assay_name: the name of the assay (e. g. 'RNA')
- workspace: An optional workspace id.
- chunk_size: The requested size of chunks to load into memory and process.
-
- Attributes:
- h5ad: A h5ad object (h5 file with added AnnData structure).
- chunkSizes: The requested size of chunks to load into memory and process.
- assayName: The Zarr hierarchy (array or group).
- z: The Zarr hierarchy (array or group).
- """
-
- def __init__(
- self,
- h5ad: H5adReader,
- zarr_loc: ZARRLOC,
- assay_name: Optional[str] = None,
- workspace: Union[str, None] = None,
- chunk_size=(1000, 1000),
- ):
- # TODO: support for multiple assay. One of the `var` datasets can be used to group features in separate assays
- self.h5ad = h5ad
- self.chunkSizes = chunk_size
- self.workspace = workspace
- if assay_name is None:
- logger.info(
- f"No value provided for assay names. Will use default value: 'RNA'"
- )
- self.assayName = "RNA"
- else:
- self.assayName = assay_name
- self.z = load_zarr(zarr_loc=zarr_loc, mode="w")
- self._ini_cell_data()
- create_zarr_count_assay(
- z=self.z,
- assay_name=self.assayName,
- workspace=workspace,
- chunk_size=chunk_size,
- n_cells=self.h5ad.nCells,
- feat_ids=self.h5ad.feat_ids(),
- feat_names=self.h5ad.feat_names(),
- dtype=self.h5ad.matrixDtype,
- )
- self._ini_feature_data()
-
- def _ini_cell_data(self):
- ids = self.h5ad.cell_ids()
- g = create_cell_data(
- z=self.z,
- workspace=self.workspace,
- ids=ids,
- names=ids,
- )
- for i, j in self.h5ad.get_cell_columns():
- create_zarr_obj_array(g, i, j, j.dtype)
-
- def _ini_feature_data(self):
- if self.workspace is None:
- g: zarr.Group = self.z[f"{self.assayName}/featureData"] # type: ignore
- else:
- g: zarr.Group = self.z[f"{self.workspace}/{self.assayName}/featureData"] # type: ignore
- for i, j in self.h5ad.get_feat_columns():
- if i not in g:
- create_zarr_obj_array(g, i, j, j.dtype)
-
- def dump(self, batch_size: int = 1000) -> None:
- # TODO: add informed description to docstring
- """
- Raises:
- AssertionError: Catches eventual bugs in the class, if number of cells does not match after transformation.
-
- Returns:
- None
- """
- store = load_count_store(self.z, self.assayName, self.workspace)
- total_cells_written = sparse_writer(
- store=store,
- data_stream=self.h5ad.consume(batch_size),
- n_cells=self.h5ad.nCells,
- batch_size=batch_size,
- )
- if total_cells_written != self.h5ad.nCells:
- raise AssertionError(
- "ERROR: This is a bug in H5adToZarr. All cells might not have been successfully "
- "written into the zarr file. Please report this issue"
- )
-
-
-class NaboH5ToZarr:
- """A class for converting data in a h5 file generated by Nabo, to a Zarr
- hierarchy.
-
- Args:
- h5: A Nabo h5 object containing the data.
- zarr_loc: The file name for the Zarr hierarchy.
- assay_name: the name of the assay (e. g. 'RNA')
- chunk_size: The requested size of chunks to load into memory and process.
- dtype: the dtype of the data.
-
- Attributes:
- h5: A Nabo h5 object.
- zarr_loc: The file name for the Zarr hierarchy.
- chunkSizes: The requested size of chunks to load into memory and process.
- assayName: The Zarr hierarchy (array or group).
- z: The Zarr hierarchy (array or group).
- """
-
- def __init__(
- self,
- h5: NaboH5Reader,
- zarr_loc: ZARRLOC,
- assay_name: Optional[str] = None,
- workspace: Union[str, None] = None,
- chunk_size=(1000, 1000),
- dtype: str = "uint32",
- ):
- self.h5 = h5
- self.chunkSizes = chunk_size
- self.workspace = workspace
- if assay_name is None:
- logger.info(
- f"No value provided for assay names. Will use default value: 'RNA'"
- )
- self.assayName = "RNA"
- else:
- self.assayName = assay_name
- self.z = load_zarr(zarr_loc, mode="w") # type: ignore
- self._ini_cell_data()
- create_zarr_count_assay(
- z=self.z,
- assay_name=self.assayName,
- workspace=workspace,
- chunk_size=chunk_size,
- n_cells=self.h5.nCells,
- feat_ids=self.h5.feat_ids(),
- feat_names=self.h5.feat_names(),
- dtype=dtype,
- )
-
- def _ini_cell_data(self):
- ids = self.h5.cell_ids()
- _ = create_cell_data(
- z=self.z,
- workspace=self.workspace,
- ids=np.array(ids),
- names=np.array(ids),
- )
-
- def dump(self, batch_size: int = 500) -> None:
- # TODO: add informed description to docstring
- """
- Raises:
- AssertionError: Catches eventual bugs in the class, if number of cells does not match after transformation.
-
- Returns:
- None
- """
- store = load_count_store(self.z, self.assayName, self.workspace)
- (
- s,
- e,
- ) = (
- 0,
- 0,
- )
- n_chunks = self.h5.nCells // batch_size + 1
- for a in tqdmbar(self.h5.consume(batch_size), total=n_chunks):
- e += a.shape[0]
- store[s:e] = a
- s = e
- if e != self.h5.nCells:
- raise AssertionError(
- "ERROR: This is a bug in NaboH5ToZarr. All cells might not have been successfully "
- "written into the zarr file. Please report this issue"
- )
-
-
-class LoomToZarr:
- """A class for converting data in a Loom file to a Zarr hierarchy. Converts
- a Loom file read using scarf.LoomReader into Scarf's Zarr format.
-
- Args:
- loom: LoomReader object used to open Loom format file
- zarr_loc: Output Zarr filename with path
- assay_name: Name for the output assay. If not provided then automatically set to RNA
- chunk_size: Chunk size for the count matrix saved in Zarr file.
-
- Attributes:
- loom: A scarf.LoomReader object used to open Loom format file.
- fn: The file name for the Zarr hierarchy.
- chunkSizes: The requested size of chunks to load into memory and process.
- assayName: The Zarr hierarchy (array or group).
- z: The Zarr hierarchy (array or group).
- """
-
- def __init__(
- self,
- loom: LoomReader,
- zarr_loc: ZARRLOC,
- assay_name: Optional[str] = None,
- workspace: Union[str, None] = None,
- chunk_size=(1000, 1000),
- ):
- # TODO: support for multiple assay. Data from within individual layers can be treated as separate assays
- self.loom = loom
- self.chunkSizes = chunk_size
- self.workspace = workspace
- if assay_name is None:
- logger.info(
- f"No value provided for assay names. Will use default value: 'RNA'"
- )
- self.assayName = "RNA"
- else:
- self.assayName = assay_name
- self.z = load_zarr(zarr_loc, mode="w")
- self._ini_cell_data()
- create_zarr_count_assay(
- z=self.z,
- assay_name=self.assayName,
- workspace=workspace,
- chunk_size=chunk_size,
- n_cells=self.loom.nCells,
- feat_ids=self.loom.feature_ids(),
- feat_names=self.loom.feature_names(),
- dtype=self.loom.matrixDtype,
- )
- self._ini_feature_data()
-
- def _ini_cell_data(self):
- ids = np.array(self.loom.cell_ids())
- g = create_cell_data(
- z=self.z,
- workspace=self.workspace,
- ids=ids,
- names=ids,
- )
- for i, j in self.loom.get_cell_attrs():
- try:
- create_zarr_obj_array(g, i, j, j.dtype)
- except UnicodeDecodeError:
- logger.warning(f"Could not import {i} cell(column) attribute")
-
- def _ini_feature_data(self):
- if self.workspace is None:
- g: zarr.Group = self.z[f"{self.assayName}/featureData"] # type: ignore
- else:
- g: zarr.Group = self.z[f"{self.workspace}/{self.assayName}/featureData"] # type: ignore
- for i, j in self.loom.get_feature_attrs():
- create_zarr_obj_array(g, i, j, j.dtype)
-
- def dump(self, batch_size: int = 1000) -> None:
- # TODO: add informed description to docstring
- """
- Raises:
- AssertionError: Catches eventual bugs in the class, if number of cells does not match after transformation.
-
- Returns:
- None
- """
- store = load_count_store(self.z, self.assayName, self.workspace)
- total_cells_written = sparse_writer(
- store=store,
- data_stream=self.loom.consume(batch_size),
- n_cells=self.loom.nCells,
- batch_size=batch_size,
- )
- if total_cells_written != self.loom.nCells:
- raise AssertionError(
- "ERROR: This is a bug in LoomToZarr. All cells might not have been successfully "
- "written into the zarr file. Please report this issue"
- )
-
-
-class SparseToZarr:
- """A class for converting data in a sparse matrix to a Zarr hierarchy.
-
- Args:
- csr_mat: A CSR format sparse matrix
- zarr_loc: Output Zarr filename with path
- cell_ids: Cell IDs for the cells in the dataset.
- feature_ids: Feature IDs for the features in the dataset.
- assay_name: Name for the output assay. If not provided then automatically set to RNA.
- chunk_size: The requested size of chunks to load into memory and process.
-
- Raises:
- ValueError: Raised if number of input cell or feature IDs does not match the matrix.
- AssertionError: Catches eventual bugs in the class, if number of cells does not match after transformation.
-
- Attributes:
- mat: Input CSR matrix
- fn: The file name for the Zarr hierarchy.
- chunkSizes: The requested size of chunks to load into memory and process.
- assayName: The Zarr hierarchy (array or group).
- z: The Zarr hierarchy (array or group).
- """
-
- def __init__(
- self,
- csr_mat: csr_matrix,
- zarr_loc: ZARRLOC,
- cell_ids: Union[np.ndarray, List[str]],
- feature_ids: Union[np.ndarray, List[str]],
- assay_name: Optional[str] = None,
- workspace: Union[str, None] = None,
- feature_names: Union[np.ndarray, List[str], None] = None,
- chunk_size=(1000, 1000),
- matrix_dtype: Optional[np.dtype] = None,
- ):
- self.mat = csr_mat
- self.chunkSizes = chunk_size
- self.workspace = workspace
- cell_ids = np.array(cell_ids)
- if matrix_dtype is None:
- self.matrixDtype = self.mat.dtype
- else:
- self.matrixDtype = matrix_dtype
- if assay_name is None:
- logger.info(
- f"No value provided for assay names. Will use default value: 'RNA'"
- )
- self.assayName = "RNA"
- else:
- self.assayName = assay_name
- self.nCells, self.nFeatures = self.mat.shape
- if len(cell_ids) != self.nCells:
- raise ValueError(
- "ERROR: Number of cell ids are not same as number of cells in the matrix"
- )
- if len(feature_ids) != self.nFeatures:
- raise ValueError(
- "ERROR: Number of feature ids are not same as number of features in the matrix"
- )
-
- self.z = load_zarr(zarr_loc, mode="w")
- _ = create_cell_data(
- z=self.z,
- workspace=self.workspace,
- ids=cell_ids,
- names=cell_ids,
- )
- if feature_names is None:
- feature_names = feature_ids
- create_zarr_count_assay(
- z=self.z,
- assay_name=self.assayName,
- workspace=workspace,
- chunk_size=chunk_size,
- n_cells=self.nCells,
- feat_ids=feature_ids,
- feat_names=feature_names,
- dtype=str(self.matrixDtype),
- )
-
- def dump(self, batch_size: Optional[int] = None) -> None:
- """Write out the data matrix into the Zarr hierarchy.
-
- Args:
- batch_size: Number of cells to be written in one go. By default, this value will automatically be chosen
- based on the chunk size in the cell dimension.
-
- Raises:
- ValueError: Raised if there is any unexpected errors when writing to the Zarr hierarchy.
- AssertionError: Catches eventual bugs in the class, if number of cells does not match after transformation.
-
- Returns:
- None
- """
-
- store = load_count_store(self.z, self.assayName, self.workspace)
- if batch_size is None:
- batch_size = int(store.chunks[0])
- (
- s,
- e,
- ) = (
- 0,
- 0,
- )
- n_chunks = self.nCells // batch_size + 1
- for e in tqdmbar(
- range(batch_size, self.nCells + batch_size, batch_size),
- total=n_chunks,
- desc="Writing data matrix",
- ):
- if s == self.nCells:
- raise ValueError(
- "Unexpected error encountered in writing to Zarr. The last iteration has failed. "
- "Please report this issue."
- )
- if e > self.nCells:
- e = self.nCells
- a = self.mat[s:e].tocoo() # type: ignore
- store.set_coordinate_selection(
- (a.row + s, a.col), a.data.astype(self.matrixDtype)
- )
- s = e
- if e != self.nCells:
- raise AssertionError(
- "ERROR: This is a bug in SparseToZarr. All cells might not have been successfully "
- "written into the zarr file. Please report this issue"
- )
-
-
-class CSVtoZarr:
- """A class for converting data from CSV format to a Zarr hierarchy.
-
- Args:
- cr: A CSVReader object
- zarr_loc: The file name for the Zarr hierarchy.
- assay_name: A label for the assay. Ex. "RNA" or "ATAC"
- chunk_size: The requested size of chunks to load into memory and process.
- dtype: the dtype of the data.
-
- Attributes:
- csvr: A CSVReader object
- fn: The file name for the Zarr hierarchy.
- chunkSizes: The requested size of chunks to store in Zarr file
- z: The Zarr hierarchy (array or group).
- """
-
- def __init__(
- self,
- cr: CSVReader,
- zarr_loc: ZARRLOC,
- assay_name: str,
- chunk_size=(1000, 1000),
- workspace: Union[str, None] = None,
- dtype: Optional[np.dtype] = None,
- ):
- self.csvr = cr
- self.assayName = assay_name
- self.chunkSizes = chunk_size
- self.workspace = workspace
- self.z = load_zarr(zarr_loc, mode="w")
- if dtype is not None:
- self.dtype = dtype
- else:
- self.dtype = next(self.csvr.consume())[0].dtype
- cell_ids = self.csvr.cell_ids()
- _ = create_cell_data(
- z=self.z,
- workspace=workspace,
- ids=cell_ids,
- names=cell_ids,
- )
- create_zarr_count_assay(
- z=self.z,
- assay_name=self.assayName,
- workspace=workspace,
- chunk_size=chunk_size,
- n_cells=self.csvr.nCells,
- feat_ids=self.csvr.feature_ids(),
- feat_names=self.csvr.feature_ids(),
- dtype=str(self.dtype),
- )
-
- def dump(self) -> None:
- """Writes the count values into the Zarr matrix.
-
- Args:
-
- Raises:
- AssertionError: Catches eventual bugs in the class, if number of cells does not match after transformation.
-
- Returns:
- None
- """
-
- store = load_count_store(self.z, self.assayName, self.workspace)
- cell_data_grp: zarr.Group = self.z["cellData"] # type: ignore
- cell_data = [
- create_zarr_obj_array(
- cell_data_grp, name=x, data=None, dtype=y, shape=self.csvr.nCells
- )
- for x, y in zip(self.csvr.cellDataCols, self.csvr.cellDataDtypes) # type: ignore
- ]
- batch_size = self.csvr.pandas_kwargs["chunksize"]
- (
- s,
- e,
- ) = (
- 0,
- 0,
- )
- if self.csvr.nCells % batch_size == 0:
- n_chunks = int(self.csvr.nCells / batch_size)
- else:
- n_chunks = (self.csvr.nCells // batch_size) + 1
- for a, c in tqdmbar(self.csvr.consume(), total=n_chunks):
- e += a.shape[0]
- if self.dtype is not None:
- store[s:e] = a.astype(self.dtype)
- else:
- store[s:e] = a
- if c is not None:
- for n, i in enumerate(c.T):
- cell_data[n][s:e] = i
- s = e
- if e != self.csvr.nCells:
- raise AssertionError(
- "ERROR: This is a bug in LoomToZarr. All cells might not have been successfully "
- "written into the zarr file. Please report this issue"
- )
-
-
-def subset_assay_zarr(
- zarr_loc: ZARRLOC,
- in_grp: str,
- out_grp: str,
- cells_idx: np.ndarray,
- feat_idx: np.ndarray,
- chunk_size: tuple,
-):
- """Selects a subset of the data in an assay in the specified Zarr
- hierarchy.
-
- For the arguments `cells_idx` and `feat_idx`, refer to the documentation for numpy.split:
- https://numpy.org/doc/stable/reference/generated/numpy.split.html
-
- Args:
- zarr_loc: The file name for the Zarr hierarchy.
- in_grp: Group in Zarr hierarchy to subset.
- out_grp: Group name in Zarr hierarchy to write subsetted assay to.
- cells_idx: A list of cell indices to (keep | drop ?).
- feat_idx: A list of feature indices to (keep | drop ?).
- chunk_size: The requested size of chunks to load into memory and process.
-
- Returns:
- None
- """
- z = load_zarr(zarr_loc, "r+")
- ig: zarr.Array = z[in_grp] # type: ignore
- og = create_zarr_dataset(
- z, out_grp, chunk_size, "uint32", (len(cells_idx), len(feat_idx))
- )
- pos_start, pos_end = 0, 0
- for i in tqdmbar(np.array_split(cells_idx, len(cells_idx) // chunk_size[0] + 1)):
- pos_end += len(i)
- og[pos_start:pos_end, :] = ig.get_orthogonal_selection((i, feat_idx))
- pos_start = pos_end
- return None
-
-
-def dask_to_zarr(df, z, loc, chunk_size, nthreads: int, msg: Optional[str] = None):
- # TODO: perhaps change name of Dask array so it does not get confused with a dataframe
- """Creates a Zarr hierarchy from a Dask array.
-
- Args:
- df (): Dask array.
- z (): Zarr hierarchy.
- loc (): Location to write data/Zarr hierarchy to.
- chunk_size (): Size of chunks to load into memory and process.
- nthreads (int): Number of threads to use.
- msg (str): Message to use with progress bar (Default: f"Writing data to {loc}").
- """
- if msg is None:
- msg = f"Writing data to {loc}"
- og = create_zarr_dataset(z, loc, chunk_size, "float64", df.shape)
- pos_start, pos_end = 0, 0
- for i in tqdmbar(df.blocks, total=df.numblocks[0], desc=msg):
- pos_end += i.shape[0]
- og[pos_start:pos_end, :] = controlled_compute(i, nthreads)
- pos_start = pos_end
- return None
-
-
-class SubsetZarr:
- """Split Zarr file using a subset of cells.
-
- Args:
- zarr_loc: Path for the output (subsetted) Zarr file
- assays: Source assays to be subsetted. These assays must be from the same dataset
- cell_key: Name of a boolean column in cell metadata. The cells with value True are included in the
- subset. Only used when cell_idx is None.
- cell_idx: Indices of the cells to be included in the subsetted.
- reset_cell_filter: If True, then the cell filtering information is removed, i.e. even the filtered out cells
- are set as True as in the 'I' column. To keep the filtering information set the value for
- this parameter to False. (Default value: True)
- overwrite_existing_file: If True, then overwrites the existing data. (Default value: False)
- overwrite_cell_data: If True, then overwrites cell data (Default value: True)
- """
-
- def __init__(
- self,
- zarr_loc: ZARRLOC,
- assays: list,
- in_workspace: Union[str, None] = None,
- out_workspace: Union[str, None] = None,
- cell_key: Optional[str] = None,
- cell_idx: Optional[np.ndarray] = None,
- reset_cell_filter: bool = True,
- overwrite_existing_file: bool = False,
- overwrite_cell_data: bool = False,
- ) -> None:
- self.resetCells = reset_cell_filter
- self.overFn = overwrite_existing_file
- self.overCells = overwrite_cell_data
- self.inWorkspace = in_workspace
- self.outWorkspace = out_workspace
- self.z = self._check_files(zarr_loc)
- self.assays = self._check_assays(assays)
- self.cellIdx = self._check_idx(cell_key, cell_idx)
-
- def _check_files(self, zarr_loc: ZARRLOC):
- if (
- isinstance(zarr_loc, str)
- and os.path.isdir(zarr_loc)
- and self.overFn is False
- ):
- raise ValueError(
- f"Zarr file with name: {zarr_loc} already exists.\n"
- f"If you want to overwrite it then please set overwrite_existing_file to True. "
- f"No subsetting was performed."
- )
- return load_zarr(zarr_loc=zarr_loc, mode="w")
-
- @staticmethod
- def _check_assays(assays):
- # if type(assays) != list:
- if isinstance(assays, list) is False:
- raise TypeError(
- "Value for parameter `assays` should be a list. For example, `[ds.RNA]`"
- )
- n = []
- for assay in assays:
- try:
- n.append(assay.cells.N)
- except AttributeError:
- raise ValueError(
- "Please make sure you are passing actual assay objects and not assay names. "
- "For example, `[ds.RNA]`"
- )
- if len(set(n)) != 1:
- raise ValueError(
- f"ERROR: Provided assays do not have the same numer of cells. Please make " # noqa: F541
- f"sure that the assays are from the same DataStore." # noqa: F541
- )
- return assays
-
- def _check_idx(self, cell_key, cell_idx):
- if cell_key is None and cell_idx is None:
- raise ValueError("Both `cell_key` and `cell_idx` parameters cannot be None")
- if cell_idx is None:
- for assay in self.assays:
- try:
- idx = assay.cells.fetch_all(cell_key)
- except KeyError:
- raise ValueError(
- f"ERROR: Provided cell_key {cell_key} was not found in the assay: {assay.name}"
- )
- if idx.dtype != bool:
- raise ValueError(
- f"ERROR: {cell_key} is not of boolean type. Cannot perform subsetting"
- )
- if cell_idx is None:
- cell_idx = idx
- else:
- if np.all(cell_idx == idx) is False:
- raise ValueError(
- f"ERROR: Provided cell_key {cell_key} is not consistent across the assays. "
- f"Please make sure that the assays are from the same DataStore."
- )
- cell_idx = np.where(cell_idx)[0]
- else:
- cell_idx = np.array(cell_idx)
- if np.issubdtype(cell_idx.dtype, np.integer) is False:
- raise ValueError(
- f"ERROR: `cell_idx` must be of integer type. Provided array has a dtype: {cell_idx.dtype}"
- )
- if max(cell_idx) >= self.assays[0].cells.N:
- raise ValueError(
- f"ERROR: `cell_idx` max value is larger than the number of cells in the data." # noqa: F541
- )
- return cell_idx
-
- def _prep_cell_data(self):
- if self.outWorkspace is None:
- cell_slot = "cellData"
- else:
- cell_slot = f"{self.outWorkspace}/cellData"
- if cell_slot in self.z:
- g: zarr.Group = self.z[cell_slot] # type: ignore
- else:
- g: zarr.Group = self.z.create_group(cell_slot)
-
- if self.outWorkspace is None:
- cell_data = self.assays[0].z["/cellData"]
- else:
- cell_data = self.assays[0].z[f"/{self.inWorkspace}/cellData"]
-
- n_cells = len(self.cellIdx)
- for i in cell_data.keys():
- if i in g and self.overCells is False:
- continue
- if i in ["I"] and self.resetCells:
- create_zarr_obj_array(g, "I", [True for _ in range(n_cells)], "bool")
- continue
- v = cell_data[i][:][self.cellIdx]
- create_zarr_obj_array(g, i, v, dtype=v.dtype)
-
- def _prep_counts(self):
- n_cells = len(self.cellIdx)
- for assay in self.assays:
- create_zarr_count_assay(
- z=self.z,
- assay_name=assay.name,
- workspace=self.outWorkspace,
- chunk_size=assay.rawData.chunksize,
- n_cells=n_cells,
- feat_ids=assay.feats.fetch_all("ids"),
- feat_names=assay.feats.fetch_all("names"),
- dtype=assay.rawData.dtype,
- )
-
- def dump(self):
- self._prep_cell_data()
- self._prep_counts()
- for assay in self.assays:
- raw_data = assay.rawData[self.cellIdx]
- if self.outWorkspace is None:
- store = self.z[f"{assay.name}/counts"]
- else:
- store = self.z[f"matrices/{assay.name}/counts"]
- (
- s,
- e,
- ) = (
- 0,
- 0,
- )
- for a in tqdmbar(
- raw_data.blocks,
- desc=f"Subsetting assay: {assay.name}",
- total=raw_data.numblocks[0],
- ):
- if a.shape[0] > 0:
- e += a.shape[0]
- store[s:e] = a.compute()
- s = e
-
-
-def to_h5ad(
- assay,
- h5ad_filename: str,
- embeddings_cols: Optional[List[str]] = None,
- skip_recalc_nfeats: bool = True,
- n_threads: int = 4,
-) -> None:
- """Save an assay as H5ad file.
-
- Args:
- assay: Assay to save in H5ad format
- h5ad_filename: Name for the H5ad file to be created.
- embeddings_cols: Columns in cell metadata to be treated as embeddings e. UMAP, tSNE
- (Default value: ['UMAP', 'tSNE'])
- skip_recalc_nfeats: Skip recalculating nFeatures per cell. (Default value: True)
- n_threads: Number of processing threads to use (Default value: 4)
-
- Returns:
- None
- """
- import h5py
- from dask import array as da
-
- def save_attr(group, col, scarf_col, md):
- d = md.fetch_all(scarf_col)
- d_type = d.dtype
- if np.issubdtype(d_type, np.number) or np.issubdtype(d_type, bool):
- pass
- else:
- d_type = h5py.special_dtype(vlen=str)
- try:
- h5[group].create_dataset(col, data=d.astype(d_type)) # type: ignore
- except TypeError:
- logger.warning(f"Dtype issue in {col}, {d.type} ({d_type})")
-
- h5 = h5py.File(h5ad_filename, "w")
- for i in ["X", "obs", "var", "obsm"]:
- h5.create_group(i)
-
- # Recalculating nFeature here just to avoid potential issues with stale data.
- if skip_recalc_nfeats is False:
- assay.cells.insert(
- f"{assay.name}_nFeatures",
- show_dask_progress(
- da.count_nonzero(assay.rawData, axis=1), # type: ignore
- msg="Preflight: recalculating nFeatures",
- nthreads=n_threads,
- ),
- overwrite=True,
- )
-
- n_feats_per_cell = assay.cells.fetch_all(f"{assay.name}_nFeatures").astype(int)
- tot_counts = int(n_feats_per_cell.sum())
-
- for i, s in zip(
- ["indptr", "indices", "data"], [assay.cells.N + 1, tot_counts, tot_counts]
- ):
- if i == "data":
- mat_dtype = assay.rawData.dtype
- else:
- mat_dtype = int
- h5["X"].create_dataset( # type: ignore
- i, (s,), chunks=True, compression="gzip", dtype=mat_dtype
- )
-
- h5["X/indptr"][:] = np.array([0] + list(n_feats_per_cell.cumsum())).astype(int) # type: ignore
-
- s, e = 0, 0
- for i in tqdmbar(
- assay.rawData.blocks,
- total=assay.rawData.numblocks[0],
- desc="Writing raw counts",
- ):
- i = csr_matrix(i.compute())
- e += i.data.shape[0]
- h5["X/data"][s:e] = i.data # type: ignore
- h5["X/indices"][s:e] = i.indices # type: ignore
- s = e
- attrs = {
- "encoding-type": "csr_matrix",
- "encoding-version": "0.1.0",
- "shape": np.array([assay.cells.N, assay.feats.N]),
- }
- for i, j in attrs.items():
- h5["X"].attrs[i] = j
-
- out_cols = []
- emb_cols = []
- if embeddings_cols is None:
- embeddings_cols = ["UMAP", "tSNE"]
- for i in assay.cells.columns:
- if i == "ids":
- save_attr("obs", "_index", "ids", assay.cells)
- out_cols.append("_index")
- else:
- is_emb = False
- if len(embeddings_cols) > 0:
- for j in embeddings_cols:
- if i.startswith(f"{assay.name}_{j}"):
- emb_cols.append(i)
- is_emb = True
- break
- if is_emb is False:
- save_attr("obs", i, i, assay.cells)
- out_cols.append(i)
-
- attrs = {
- "_index": "_index",
- "column-order": np.array(out_cols, dtype=object),
- "encoding-type": "dataframe",
- "encoding-version": "0.1.0",
- }
- for i, j in attrs.items():
- h5["obs"].attrs[i] = j
-
- out_cols = []
- for i in assay.feats.columns:
- if i == "ids":
- save_attr("var", "_index", "ids", assay.feats)
- out_cols.append("_index")
- elif i == "names":
- save_attr("var", "gene_short_name", "names", assay.feats)
- out_cols.append("gene_short_name")
- else:
- save_attr("var", i, i, assay.feats)
- out_cols.append(i)
-
- attrs = {
- "_index": "_index",
- "column-order": np.array(out_cols, dtype=object),
- "encoding-type": "dataframe",
- "encoding-version": "0.1.0",
- }
- for i, j in attrs.items():
- h5["var"].attrs[i] = j
-
- if len(emb_cols) > 0:
- emb_cols = np.array(emb_cols)
- c = pd.Series([x[:-1] for x in emb_cols])
- for i in c.unique():
- data = np.array([assay.cells.fetch_all(x) for x in emb_cols[c == i]]).T
- h5["obsm"].create_dataset( # type: ignore
- i.lower().replace(f"{assay.name.lower()}_", "X_"), data=data
- )
-
- h5.close()
- return None
-
-
-def to_mtx(assay, mtx_directory: str, compress: bool = False):
- """Save an assay as a Matrix Market directory.
-
- Args:
- assay: Scarf assay. For example: `ds.RNA`
- mtx_directory: Out directory where MTX file will be saved along with barcodes and features file
- compress: If True, then the files are compressed and saved with .gz extension. (Default value: False).
-
- Returns:
- None
- """
- from scipy.sparse import coo_matrix
- import gzip
-
- if os.path.isdir(mtx_directory) is False:
- os.mkdir(mtx_directory)
-
- n_feats_per_cell = assay.cells.fetch_all(f"{assay.name}_nFeatures").astype(int)
- tot_counts = int(n_feats_per_cell.sum())
- if compress:
- barcodes_fn = "barcodes.tsv.gz"
- features_fn = "features.tsv.gz"
- h = gzip.open(os.path.join(mtx_directory, "matrix.mtx.gz"), "wt")
- else:
- barcodes_fn = "barcodes.tsv"
- features_fn = "genes.tsv"
- h = open(os.path.join(mtx_directory, "matrix.mtx"), "w")
- h.write("%%MatrixMarket matrix coordinate integer general\n% Generated by Scarf\n")
- h.write(f"{assay.feats.N} {assay.cells.N} {tot_counts}\n")
- s = 0
- for i in tqdmbar(assay.rawData.blocks, total=assay.rawData.numblocks[0]):
- i = coo_matrix((i.compute()))
- df = pd.DataFrame({"col": i.col + 1, "row": i.row + s + 1, "d": i.data})
- df.to_csv(h, sep=" ", header=False, index=False, mode="a", lineterminator="\n")
- s += i.shape[0]
- h.close()
- assay.cells.to_pandas_dataframe(["ids"]).to_csv(
- os.path.join(mtx_directory, barcodes_fn), sep="\t", header=False, index=False
- )
-
- assay.feats.to_pandas_dataframe(["ids", "names"]).to_csv(
- os.path.join(mtx_directory, features_fn), sep="\t", header=False, index=False
- )
-
-
-def bed_to_sparse_array(
- bed_fn: str,
- bin_size: int,
- chrom_sizes: Dict[str, int],
- min_counts_per_cell: int = 500,
- read_chunk_size=1e6,
- sep: str = "\t",
- chrom_col: int = 0,
- start_col: int = 1,
- end_col: int = 2,
- barcode_col: int = 3,
- count_col: int = 4,
- comments_startswith: str = "#",
- disable_tqdm: bool = False,
- chrom_modifier=None,
-):
- """
-
- Args:
- bed_fn:
- bin_size:
- chrom_sizes:
- min_counts_per_cell:
- read_chunk_size:
- sep:
- chrom_col:
- start_col:
- end_col:
- barcode_col:
- count_col:
- comments_startswith:
- disable_tqdm:
- chrom_modifier:
-
- Returns:
-
- """
- import gc
-
- def feat_mapper(x):
- return feat_idx.get(x, n_feats)
-
- def default_chrom_modifier(x):
- return x + "_"
-
- feat_idx = {}
- for i in tqdmbar(chrom_sizes, disable=disable_tqdm, desc="Calculating bin indices"):
- for j in range((chrom_sizes[i] // bin_size) + 1):
- feat_idx[f"{i}_{j}"] = len(feat_idx)
- cell_idx = {}
- mat = []
- n_feats = len(feat_idx)
- if chrom_modifier is None:
- chrom_modifier = default_chrom_modifier
-
- stream = pd.read_csv(
- bed_fn,
- sep=sep,
- header=None,
- comment=comments_startswith,
- usecols=[chrom_col, start_col, end_col, barcode_col, count_col],
- chunksize=int(read_chunk_size),
- )
- for df in tqdmbar(
- stream, disable=disable_tqdm, desc="Building in memory sparse matrix"
- ):
- df[chrom_col] = df[chrom_col].map(chrom_modifier) + (
- (df[start_col] + (df[end_col] - df[start_col]) // 2).values // bin_size
- ).astype(str)
- for i in df[barcode_col].unique():
- if i not in cell_idx:
- cell_idx[i] = len(cell_idx)
- mat.append(
- np.vstack(
- [
- np.fromiter(map(cell_idx.get, df[barcode_col].values), dtype=int),
- np.fromiter(map(feat_mapper, df[chrom_col].values), dtype=int),
- df[count_col].values,
- ]
- ).T
- )
- mat = np.vstack(mat)
- gc.collect()
- mat = csr_matrix(
- (mat[:, 2], (mat[:, 0], mat[:, 1])), shape=(len(cell_idx), n_feats + 1)
- )
- gc.collect()
- idx = np.array(mat.sum(axis=1))[:, 0] > min_counts_per_cell
- return mat[idx, :-1], pd.Series(cell_idx.keys())[idx], pd.Series(feat_idx.keys())
diff --git a/scarf/writers/__init__.py b/scarf/writers/__init__.py
new file mode 100644
index 00000000..fffd1637
--- /dev/null
+++ b/scarf/writers/__init__.py
@@ -0,0 +1,110 @@
+"""Methods and classes for writing data to disk."""
+
+from importlib import import_module
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ from ._materialize import dask_to_zarr, write_renorm_subset_to_zarr
+ from ._store import (
+ create_cell_data as create_cell_data,
+ create_zarr_count_assay,
+ create_zarr_dataset,
+ create_zarr_obj_array,
+ load_count_store as load_count_store,
+ load_zarr as load_zarr,
+ )
+ from .cellranger import CrToZarr, MtxToZarr
+ from .csv import CSVtoZarr
+ from .export import to_h5ad, to_mtx
+ from .h5ad import H5adToZarr
+ from .loom import LoomToZarr
+ from .seurat import SeuratImportResult, SeuratToZarr
+ from .sparse import SparseToZarr, bed_to_sparse_array as bed_to_sparse_array
+ from .subset import SubsetZarr, subset_assay_zarr
+
+__all__ = [
+ "create_zarr_dataset",
+ "create_zarr_obj_array",
+ "create_zarr_count_assay",
+ "subset_assay_zarr",
+ "dask_to_zarr",
+ "write_renorm_subset_to_zarr",
+ "SubsetZarr",
+ "CrToZarr",
+ "MtxToZarr",
+ "H5adToZarr",
+ "LoomToZarr",
+ "SeuratImportResult",
+ "SeuratToZarr",
+ "SparseToZarr",
+ "to_h5ad",
+ "to_mtx",
+ "CSVtoZarr",
+]
+
+_LAZY_EXPORTS = {
+ "create_cell_data": "_store",
+ "create_zarr_count_assay": "_store",
+ "create_zarr_dataset": "_store",
+ "create_zarr_obj_array": "_store",
+ "load_count_store": "_store",
+ "load_zarr": "_store",
+ "dask_to_zarr": "_materialize",
+ "write_renorm_subset_to_zarr": "_materialize",
+ "CrToZarr": "cellranger",
+ "MtxToZarr": "cellranger",
+ "CSVtoZarr": "csv",
+ "to_h5ad": "export",
+ "to_mtx": "export",
+ "H5adToZarr": "h5ad",
+ "LoomToZarr": "loom",
+ "SeuratImportResult": "seurat",
+ "SeuratToZarr": "seurat",
+ "SparseToZarr": "sparse",
+ "bed_to_sparse_array": "sparse",
+ "SubsetZarr": "subset",
+ "subset_assay_zarr": "subset",
+}
+
+for _export_name in _LAZY_EXPORTS:
+ globals().pop(_export_name, None)
+del _export_name
+
+_METADATA_EXPORTS = frozenset(_LAZY_EXPORTS) - {"load_zarr"}
+
+
+def _normalize_export_metadata(value: Any) -> None:
+ value.__module__ = __name__
+ if not isinstance(value, type):
+ return
+ for descriptor in value.__dict__.values():
+ if isinstance(descriptor, (classmethod, staticmethod)):
+ method = descriptor.__func__
+ else:
+ method = descriptor
+ if callable(method) and hasattr(method, "__module__"):
+ method.__module__ = __name__
+
+
+def _cache_module_exports(module_name: str, module: Any) -> None:
+ for export_name, export_module in _LAZY_EXPORTS.items():
+ if export_module != module_name:
+ continue
+ value = getattr(module, export_name)
+ if export_name in _METADATA_EXPORTS:
+ _normalize_export_metadata(value)
+ globals().setdefault(export_name, value)
+
+
+def __getattr__(name: str) -> Any:
+ module_name = _LAZY_EXPORTS.get(name)
+ if module_name is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+
+ module = import_module(f".{module_name}", __name__)
+ _cache_module_exports(module_name, module)
+ return globals()[name]
+
+
+def __dir__() -> list[str]:
+ return sorted(set(globals()) | set(_LAZY_EXPORTS))
diff --git a/scarf/writers/_materialize.py b/scarf/writers/_materialize.py
new file mode 100644
index 00000000..3df7497d
--- /dev/null
+++ b/scarf/writers/_materialize.py
@@ -0,0 +1,70 @@
+from typing import Any
+
+import numpy as np
+import zarr
+
+from ..storage.budget import ResourceBudget
+from ..storage.materialize import (
+ dask_to_zarr as _dask_to_zarr,
+ write_renorm_subset_to_zarr as _write_renorm_subset_to_zarr,
+)
+
+
+def write_renorm_subset_to_zarr(
+ assay: Any,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ z: zarr.Group,
+ loc: str,
+ nthreads: int,
+ log_transform: bool = False,
+ msg: str | None = None,
+ mirror: zarr.Array | None = None,
+) -> None:
+ """Write library-size normalized subset data in a single scattered read pass.
+
+ For HVG subsets the per-cell scale factor is the row sum within each block,
+ so a separate ``counts.sum(axis=1)`` pass is not needed before writing.
+ """
+ return _write_renorm_subset_to_zarr(
+ assay,
+ cell_idx,
+ feat_idx,
+ z,
+ loc,
+ nthreads,
+ log_transform,
+ msg=msg,
+ mirror=mirror,
+ )
+
+
+def dask_to_zarr(
+ df: Any,
+ z: zarr.Group,
+ loc: str,
+ nthreads: int,
+ msg: str | None = None,
+ mirror: zarr.Array | None = None,
+ resources: ResourceBudget | None = None,
+) -> None:
+ """Creates a Zarr hierarchy from a chunked array.
+
+ Args:
+ df: ChunkedArray to materialize and write.
+ z: Root Zarr group.
+ loc: Array path within the group.
+ nthreads: Threads for block compute.
+ msg: Progress bar message (default: ``Writing data to {loc}``).
+ mirror: Optional second array of the same shape to write each band into
+ during the same pass (local staging cache).
+ """
+ return _dask_to_zarr(
+ df,
+ z,
+ loc,
+ nthreads,
+ msg=msg,
+ mirror=mirror,
+ resources=resources,
+ )
diff --git a/scarf/writers/_store.py b/scarf/writers/_store.py
new file mode 100644
index 00000000..80a5e664
--- /dev/null
+++ b/scarf/writers/_store.py
@@ -0,0 +1,122 @@
+from typing import Any
+
+import numpy as np
+import zarr
+
+from ..storage.arrays import (
+ create_zarr_dataset as _create_zarr_dataset,
+ create_zarr_obj_array as _create_zarr_obj_array,
+)
+from ..storage.schema import (
+ create_cell_data as _create_cell_data,
+ create_zarr_count_assay as _create_zarr_count_assay,
+ load_count_array as _load_count_array,
+)
+from ..storage.profiles import StorageProfile
+from ..storage.stores import load_zarr as load_zarr
+
+
+def create_zarr_dataset(
+ g: zarr.Group,
+ name: str,
+ chunks: tuple[int, ...] | int,
+ dtype: Any,
+ shape: tuple[int, ...],
+ overwrite: bool = True,
+) -> zarr.Array:
+ """Creates and returns a Zarr array.
+
+ Args:
+ g (zarr.hierarchy):
+ name (str):
+ chunks (tuple):
+ dtype (Any):
+ shape (Tuple):
+ overwrite (bool):
+
+ Returns:
+ A Zarr Array.
+ """
+ return _create_zarr_dataset(g, name, chunks, dtype, shape, overwrite)
+
+
+def create_zarr_obj_array(
+ g: zarr.Group,
+ name: str,
+ data: Any,
+ dtype: str | Any = None,
+ overwrite: bool = True,
+ chunk_size: int = 100000,
+ shape: int | None = None,
+) -> zarr.Array:
+ """Creates and returns a metadata column array."""
+ return _create_zarr_obj_array(
+ g,
+ name,
+ data,
+ dtype,
+ overwrite,
+ chunk_size,
+ shape,
+ )
+
+
+def create_zarr_count_assay(
+ z: zarr.Group,
+ assay_name: str,
+ workspace: str | None,
+ n_cells: int,
+ feat_ids: np.ndarray | list[str],
+ feat_names: np.ndarray | list[str],
+ dtype: str = "uint32",
+ *,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+) -> zarr.Array:
+ """Creates and returns a Zarr array with name 'counts'.
+
+ Args:
+ z (zarr.Group):
+ assay_name (str):
+ workspace (str | None):
+ n_cells (int):
+ feat_ids (np.ndarray | list[str]):
+ feat_names (np.ndarray | list[str]):
+ dtype (str = 'uint32'):
+ targetChunkBytes: Optional inner-chunk byte target.
+ targetShardBytes: Optional full-width shard byte target.
+
+ Returns:
+ A Zarr array.
+ """
+ return _create_zarr_count_assay(
+ z,
+ assay_name,
+ workspace,
+ n_cells,
+ feat_ids,
+ feat_names,
+ dtype,
+ profile=profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ )
+
+
+def load_count_store(
+ z: zarr.Group, assay_name: str, workspace: str | None
+) -> zarr.Array:
+ """Return the counts Zarr array for an assay in the given workspace."""
+ return _load_count_array(z, assay_name, workspace)
+
+
+def create_cell_data(
+ z: zarr.Group,
+ workspace: str | None,
+ ids: np.ndarray,
+ names: np.ndarray,
+ profile: StorageProfile | None = None,
+) -> zarr.Group:
+ """Create the cellData group with ids, names, and default ``I`` filter column."""
+ return _create_cell_data(z, workspace, ids, names, profile=profile)
diff --git a/scarf/writers/cellranger.py b/scarf/writers/cellranger.py
new file mode 100644
index 00000000..6afd2d0b
--- /dev/null
+++ b/scarf/writers/cellranger.py
@@ -0,0 +1,359 @@
+from collections.abc import Iterator
+from typing import Any
+
+import numpy as np
+import pandas as pd
+
+from ..readers import CrReader
+from ..storage.profiles import (
+ StorageProfile,
+ ZarrLocation,
+ resolve_storage_profile,
+)
+from ..utils.logging import logger
+from ..utils.progress import iter_progress
+
+
+class CrToZarr:
+ """A class for converting data in the Cellranger format to a Zarr
+ hierarchy.
+
+ Args:
+ cr: A CrReader object, containing the Cellranger data.
+ zarr_loc: The file name for the Zarr hierarchy or a store
+ dtype: the dtype of the data.
+ mem_budget: Memory available to the conversion. Accepts bytes, a
+ suffixed size (e.g. '8G'), or a fraction of total system memory (e.g. '0.6').
+ nthreads: Worker count for write-time concurrency. When None, auto-detected.
+
+ Attributes:
+ cr: A CrReader object, containing the Cellranger data.
+ z: The Zarr hierarchy (array or group).
+ """
+
+ def __init__(
+ self,
+ cr: CrReader,
+ zarr_loc: ZarrLocation,
+ dtype: str = "uint32",
+ workspace: str | None = None,
+ storage_options: dict[str, Any] | None = None,
+ mem_budget: int | str | None = None,
+ nthreads: int | None = None,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+ ) -> None:
+ from ..storage.budget import resolve_budget
+ from ..storage.schema import (
+ create_cell_data,
+ create_zarr_count_assay,
+ validate_assay_name,
+ )
+ from ..storage.stores import load_zarr
+
+ self.resources = resolve_budget(mem_budget, nthreads)
+ self.profile = resolve_storage_profile(zarr_loc, profile)
+ self.cr = cr
+ mark_schema_captured = getattr(self.cr, "_mark_schema_captured", None)
+ if callable(mark_schema_captured):
+ mark_schema_captured()
+ self.workspace = workspace
+ self.storage_options = storage_options
+ assay_names = tuple(dict.fromkeys(self.cr.assayFeats.columns))
+ for assay_name in assay_names:
+ validate_assay_name(assay_name)
+ self.z = load_zarr(zarr_loc=zarr_loc, mode="w", storage_options=storage_options)
+ cell_group = create_cell_data(
+ root=self.z,
+ workspace=self.workspace,
+ ids=np.array(self.cr.cell_names()),
+ names=np.array(self.cr.cell_names()),
+ profile=self.profile,
+ )
+ for assay_name in assay_names:
+ create_zarr_count_assay(
+ z=self.z,
+ assay_name=assay_name,
+ workspace=workspace,
+ n_cells=self.cr.nCells,
+ feat_ids=self.cr.feature_ids(assay_name),
+ feat_names=self.cr.feature_names(assay_name),
+ dtype=dtype,
+ profile=self.profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ )
+ self._write_reader_metadata(cell_group, assay_names)
+
+ def _write_reader_metadata(
+ self,
+ cell_group: Any,
+ assay_names: tuple[str, ...],
+ ) -> None:
+ from ..storage.arrays import create_zarr_obj_array
+ from ..storage.types import as_zarr_group
+
+ cell_columns = getattr(self.cr, "get_cell_columns", None)
+ if callable(cell_columns):
+ for name, raw_values in cell_columns():
+ if name in {"I", "ids", "names"} or name in cell_group:
+ continue
+ values = np.asarray(raw_values)
+ if values.ndim != 1 or values.size != self.cr.nCells:
+ raise ValueError(
+ f"Cell metadata column {name!r} has shape {values.shape}; "
+ f"expected ({self.cr.nCells},)"
+ )
+ create_zarr_obj_array(
+ cell_group,
+ name,
+ values,
+ values.dtype,
+ profile=self.profile,
+ )
+
+ feature_columns = getattr(self.cr, "get_feature_columns", None)
+ if not callable(feature_columns):
+ return
+ ranges = self._prep_assay_input_ranges(self.cr.assayFeats)
+ targets: dict[str, tuple[Any, np.ndarray]] = {}
+ for assay_name in assay_names:
+ indexes = np.concatenate(
+ [
+ np.arange(start, end, dtype=np.int64)
+ for start, end in ranges[assay_name]
+ ]
+ )
+ group_path = (
+ f"{assay_name}/featureData"
+ if self.workspace is None
+ else f"{self.workspace}/{assay_name}/featureData"
+ )
+ targets[assay_name] = (
+ as_zarr_group(self.z[group_path], name=group_path),
+ indexes,
+ )
+ for name, raw_values in feature_columns():
+ values = np.asarray(raw_values)
+ if values.ndim != 1 or values.size != self.cr.nFeatures:
+ raise ValueError(
+ f"Feature metadata column {name!r} has shape {values.shape}; "
+ f"expected ({self.cr.nFeatures},)"
+ )
+ for group, indexes in targets.values():
+ if name in {"I", "ids", "names"} or name in group:
+ continue
+ selected = values[indexes]
+ create_zarr_obj_array(
+ group,
+ name,
+ selected,
+ selected.dtype,
+ profile=self.profile,
+ )
+
+ @staticmethod
+ def _prep_assay_input_ranges(af: pd.DataFrame) -> dict[str, list[list[int]]]:
+ assay_order = (
+ af.T.nFeatures.groupby(af.columns).sum().sort_values(ascending=False).index
+ )
+ ranges = {}
+ for assay in assay_order:
+ temp = []
+ if len(af[assay].shape) == 2:
+ for i in af[assay].values[1:3].T:
+ temp.append([i[0], i[1]])
+ else:
+ idx = af[assay]
+ temp = [[idx.start, idx.end]]
+ ranges[assay] = temp
+ return ranges
+
+ @staticmethod
+ def _prep_feat_index_offset(
+ ranges: dict[str, list[list[int]]],
+ ) -> dict[str, list[int]]:
+ feat_offset: dict[str, list[int]] = {}
+ for i in ranges:
+ feat_offset[i] = []
+ lv = 0
+ for j in ranges[i]:
+ feat_offset[i].append(-j[0] + lv)
+ lv += j[1] - j[0]
+ return feat_offset
+
+ def dump(
+ self,
+ batch_size: int | None = None,
+ lines_in_mem: int = 100000,
+ ) -> None:
+ """Writes the count values into the Zarr matrix.
+
+ Args:
+ batch_size: Number of source cells per batch. By default, a
+ destination-aligned value is selected within the memory budget.
+ lines_in_mem: Number of lines to read at a time from MTX file (only used for CrDirReader)
+ (Default value: 100000)
+
+ Raises:
+ AssertionError: Catches eventual bugs in the class, if number of cells does not match after transformation.
+
+ Returns:
+ None
+ """
+ from scipy.sparse import coo_matrix
+
+ from ..storage.schema import load_count_array
+ from ..storage.sharding import (
+ SparseShardBuffer,
+ SparseWriteBand,
+ resolve_sparse_import_batch,
+ sparse_matrix_bytes,
+ write_sparse_bands,
+ )
+
+ if batch_size is not None and batch_size <= 0:
+ raise ValueError("batch_size must be positive")
+ if lines_in_mem <= 0:
+ raise ValueError("lines_in_mem must be positive")
+ input_ranges = self._prep_assay_input_ranges(self.cr.assayFeats)
+ stores = {
+ assay: load_count_array(self.z, assay, self.workspace)
+ for assay in input_ranges
+ }
+ buffers = {assay: SparseShardBuffer(store) for assay, store in stores.items()}
+ feat_offset = self._prep_feat_index_offset(input_ranges)
+ configure_lines = getattr(
+ self.cr,
+ "_set_sparse_import_lines_in_mem",
+ None,
+ )
+ prepare = getattr(self.cr, "_prepare_sparse_import", None)
+ release = getattr(self.cr, "_release_sparse_import", None)
+
+ try:
+ if callable(configure_lines):
+ configure_lines(lines_in_mem)
+ if callable(prepare):
+ prepare()
+ resident_reader_bytes = 0
+ reader_resident = getattr(
+ self.cr,
+ "_sparse_import_resident_bytes",
+ None,
+ )
+ if callable(reader_resident):
+ resident_reader_bytes = max(0, int(reader_resident()))
+ projection_value_bytes = max(
+ self.cr.matrix_dtype.itemsize,
+ *(store.dtype.itemsize for store in stores.values()),
+ )
+
+ def projection_staging_bytes(rows: int) -> int:
+ source_values = max(0, int(self.cr.max_window_nnz(rows)))
+ return source_values * (
+ projection_value_bytes
+ + 3 * np.dtype(np.int64).itemsize
+ + 2 * np.dtype(np.bool_).itemsize
+ )
+
+ plan = resolve_sparse_import_batch(
+ tuple(stores.values()),
+ nRows=self.cr.nCells,
+ resources=self.resources,
+ maxWindowNnz=self.cr.max_window_nnz,
+ sourceDtype=self.cr.matrix_dtype,
+ batchRows=batch_size,
+ residentBytes=resident_reader_bytes,
+ producerStagingBytes=lambda rows: self.cr.producer_staging_bytes(
+ rows,
+ lines_in_mem,
+ ),
+ extraProducerBytes=projection_staging_bytes,
+ )
+ self._lastImportPlan = plan
+ resolved_batch_rows = plan.batchRows
+ logger.debug(
+ f"Resolved Cell Ranger source batch rows={resolved_batch_rows} "
+ f"write_tasks={plan.writeTasks}"
+ )
+
+ def writes() -> Iterator[SparseWriteBand]:
+ n_chunks = (
+ self.cr.nCells + resolved_batch_rows - 1
+ ) // resolved_batch_rows
+ source = iter_progress(
+ self.cr.consume(resolved_batch_rows, lines_in_mem),
+ total=n_chunks,
+ desc="Writing counts",
+ )
+ for matrix in source:
+ chunk = matrix.tocoo(copy=False)
+ source_bytes = sparse_matrix_bytes(matrix, chunk)
+ for assay in input_ranges:
+ selected = np.zeros(chunk.col.shape[0], dtype=bool)
+ columns = chunk.col.copy()
+ for bounds, offset in zip(
+ input_ranges[assay],
+ feat_offset[assay],
+ strict=True,
+ ):
+ inside = (chunk.col >= bounds[0]) & (chunk.col < bounds[1])
+ columns[inside] += offset
+ selected |= inside
+ projected = coo_matrix(
+ (
+ chunk.data[selected],
+ (chunk.row[selected], columns[selected]),
+ ),
+ shape=(chunk.shape[0], buffers[assay].nColumns),
+ )
+ for band in buffers[assay].add(projected):
+ producer_bytes = (
+ source_bytes
+ + columns.nbytes
+ + 2 * selected.nbytes
+ + sparse_matrix_bytes(projected)
+ + sum(item.residentBytes for item in buffers.values())
+ )
+ yield SparseWriteBand(
+ stores[assay],
+ band,
+ producer_bytes,
+ )
+ if self.cr.nCells:
+ del matrix, chunk, selected, columns, inside, projected
+ for assay, buffer in buffers.items():
+ for band in buffer.finish():
+ producer_bytes = sum(
+ item.residentBytes for item in buffers.values()
+ )
+ yield SparseWriteBand(
+ stores[assay],
+ band,
+ producer_bytes,
+ )
+
+ write_sparse_bands(
+ writes(),
+ resources=self.resources,
+ residentBytes=resident_reader_bytes,
+ producerReserveBytes=plan.producerReserveBytes,
+ total=plan.writeTasks,
+ )
+ if any(buffer.rows != self.cr.nCells for buffer in buffers.values()):
+ raise AssertionError(
+ "Cell Ranger conversion did not write every source row"
+ )
+ logger.info(
+ f"Wrote {self.cr.nCells} cells and "
+ f"{sum(buffer.nColumns for buffer in buffers.values())} features "
+ f"from Cell Ranger to {len(stores)} assay(s)"
+ )
+ finally:
+ if callable(release):
+ release()
+
+
+MtxToZarr = CrToZarr
diff --git a/scarf/writers/csv.py b/scarf/writers/csv.py
new file mode 100644
index 00000000..9a2c6004
--- /dev/null
+++ b/scarf/writers/csv.py
@@ -0,0 +1,148 @@
+from collections.abc import Iterator
+from typing import Any
+
+import numpy as np
+
+from ..storage.types import as_zarr_group
+from ..readers import CSVReader
+from ..storage.profiles import (
+ StorageProfile,
+ ZarrLocation,
+ resolve_storage_profile,
+)
+from ..storage.sharding import write_dense_from_row_batches
+from ..utils.logging import logger
+
+
+class CSVtoZarr:
+ """A class for converting data from CSV format to a Zarr hierarchy.
+
+ Args:
+ cr: A CSVReader object
+ zarr_loc: The file name for the Zarr hierarchy.
+ assay_name: A label for the assay. Ex. "RNA" or "ATAC"
+ dtype: the dtype of the data.
+
+ Attributes:
+ csvr: A CSVReader object
+ fn: The file name for the Zarr hierarchy.
+ z: The Zarr hierarchy (array or group).
+ """
+
+ def __init__(
+ self,
+ cr: CSVReader,
+ zarr_loc: ZarrLocation,
+ assay_name: str,
+ workspace: str | None = None,
+ dtype: np.dtype | None = None,
+ storage_options: dict[str, Any] | None = None,
+ mem_budget: int | str | None = None,
+ nthreads: int | None = None,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+ ) -> None:
+ from ..storage.budget import resolve_budget
+ from ..storage.schema import (
+ create_cell_data,
+ create_zarr_count_assay,
+ validate_assay_name,
+ )
+ from ..storage.stores import load_zarr
+
+ self.csvr = cr
+ self.assayName = assay_name
+ validate_assay_name(self.assayName)
+ self.resources = resolve_budget(mem_budget, nthreads)
+ self.profile = resolve_storage_profile(zarr_loc, profile)
+ self.workspace = workspace
+ self.storage_options = storage_options
+ self.z = load_zarr(zarr_loc, mode="w", storage_options=storage_options)
+ if dtype is not None:
+ self.dtype = dtype
+ else:
+ self.dtype = next(self.csvr.consume())[0].dtype
+ cell_ids = self.csvr.cell_ids()
+ _ = create_cell_data(
+ root=self.z,
+ workspace=workspace,
+ ids=cell_ids,
+ names=cell_ids,
+ profile=self.profile,
+ )
+ create_zarr_count_assay(
+ z=self.z,
+ assay_name=self.assayName,
+ workspace=workspace,
+ n_cells=self.csvr.nCells,
+ feat_ids=self.csvr.feature_ids(),
+ feat_names=self.csvr.feature_ids(),
+ dtype=str(self.dtype),
+ profile=self.profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ )
+
+ def dump(self) -> None:
+ """Writes the count values into the Zarr matrix.
+
+ Args:
+
+ Raises:
+ AssertionError: Catches eventual bugs in the class, if number of cells does not match after transformation.
+
+ Returns:
+ None
+ """
+ from ..storage.arrays import create_zarr_obj_array
+ from ..storage.schema import load_count_array
+
+ store = load_count_array(self.z, self.assayName, self.workspace)
+ cell_data_path = (
+ "cellData" if self.workspace is None else f"{self.workspace}/cellData"
+ )
+ cell_data_grp = as_zarr_group(
+ self.z[cell_data_path],
+ name=cell_data_path,
+ )
+ cell_data = [
+ create_zarr_obj_array(
+ cell_data_grp,
+ name=x,
+ data=None,
+ dtype=y,
+ shape=self.csvr.nCells,
+ profile=self.profile,
+ )
+ for x, y in zip(self.csvr.cellDataCols, self.csvr.cellDataDtypes or [])
+ ]
+
+ def count_batches() -> Iterator[np.ndarray]:
+ s = 0
+ for a, c in self.csvr.consume():
+ e = s + a.shape[0]
+ if c is not None:
+ for n, i in enumerate(c.T):
+ cell_data[n][s:e] = i
+ s = e
+ if self.dtype is not None:
+ yield a.astype(self.dtype)
+ else:
+ yield a
+
+ e = write_dense_from_row_batches(
+ store,
+ count_batches(),
+ msg="Writing CSV counts",
+ resources=self.resources,
+ )
+ if e != self.csvr.nCells:
+ raise AssertionError(
+ "ERROR: This is a bug in CSVtoZarr. All cells might not have been successfully "
+ "written into the zarr file. Please report this issue"
+ )
+ logger.info(
+ f"Wrote {self.csvr.nCells} cells and {self.csvr.nFeatures} features "
+ f"from CSV to assay {self.assayName}"
+ )
diff --git a/scarf/writers/export.py b/scarf/writers/export.py
new file mode 100644
index 00000000..ccb7a92c
--- /dev/null
+++ b/scarf/writers/export.py
@@ -0,0 +1,221 @@
+import os
+from typing import Any
+
+import numpy as np
+import pandas as pd
+from scipy.sparse import csr_matrix
+
+from ..utils.compute import show_dask_progress
+from ..utils.logging import logger
+
+
+def to_h5ad(
+ assay: Any,
+ h5ad_filename: str,
+ embeddings_cols: list[str] | None = None,
+ skip_recalc_nfeats: bool = True,
+ n_threads: int = 4,
+) -> None:
+ """Save an assay as H5ad file.
+
+ Args:
+ assay: Assay to save in H5ad format
+ h5ad_filename: Name for the H5ad file to be created.
+ embeddings_cols: Columns in cell metadata to be treated as embeddings e. UMAP, tSNE
+ (Default value: ['UMAP', 'tSNE'])
+ skip_recalc_nfeats: Skip recalculating nFeatures per cell. (Default value: True)
+ n_threads: Number of processing threads to use (Default value: 4)
+
+ Returns:
+ None
+ """
+ import h5py
+
+ def save_attr(group: str, col: str, scarf_col: str, md: Any) -> None:
+ d = md.fetch_all(scarf_col)
+ d_type = d.dtype
+ if np.issubdtype(d_type, np.number) or np.issubdtype(d_type, bool):
+ pass
+ else:
+ d_type = h5py.special_dtype(vlen=str)
+ try:
+ h5[group].create_dataset(col, data=d.astype(d_type))
+ except TypeError:
+ logger.warning(
+ f"Skipping metadata column {col!r} with unsupported dtype {d.dtype}"
+ )
+
+ h5 = h5py.File(h5ad_filename, "w")
+ for i in ["X", "obs", "var", "obsm"]:
+ h5.create_group(i)
+
+ # Recalculating nFeature here just to avoid potential issues with stale data.
+ if skip_recalc_nfeats is False:
+ assay.cells.insert(
+ f"{assay.name}_nFeatures",
+ show_dask_progress(
+ assay.rawData.count_nonzero(axis=1),
+ msg="Recalculating detected feature counts",
+ nthreads=n_threads,
+ ),
+ overwrite=True,
+ )
+
+ n_feats_per_cell = assay.cells.fetch_all(f"{assay.name}_nFeatures").astype(int)
+ tot_counts = int(n_feats_per_cell.sum())
+
+ for i, s in zip(
+ ["indptr", "indices", "data"], [assay.cells.N + 1, tot_counts, tot_counts]
+ ):
+ if i == "data":
+ mat_dtype = assay.rawData.dtype
+ else:
+ mat_dtype = int
+ h5["X"].create_dataset(
+ i, (s,), chunks=True, compression="gzip", dtype=mat_dtype
+ )
+
+ h5["X/indptr"][:] = np.array([0] + list(n_feats_per_cell.cumsum())).astype(int)
+
+ s, e = 0, 0
+ for values in assay.rawData.stream_blocks(
+ nthreads=n_threads,
+ msg="Writing raw counts",
+ ):
+ block = csr_matrix(values)
+ e += block.data.shape[0]
+ h5["X/data"][s:e] = block.data
+ h5["X/indices"][s:e] = block.indices
+ s = e
+ attrs = {
+ "encoding-type": "csr_matrix",
+ "encoding-version": "0.1.0",
+ "shape": np.array([assay.cells.N, assay.feats.N]),
+ }
+ for i, j in attrs.items():
+ h5["X"].attrs[i] = j
+
+ out_cols = []
+ emb_cols = []
+ if embeddings_cols is None:
+ embeddings_cols = ["UMAP", "tSNE"]
+ for i in assay.cells.columns:
+ if i == "ids":
+ save_attr("obs", "_index", "ids", assay.cells)
+ out_cols.append("_index")
+ else:
+ is_emb = False
+ if len(embeddings_cols) > 0:
+ for j in embeddings_cols:
+ if i.startswith(f"{assay.name}_{j}"):
+ emb_cols.append(i)
+ is_emb = True
+ break
+ if is_emb is False:
+ save_attr("obs", i, i, assay.cells)
+ out_cols.append(i)
+
+ attrs = {
+ "_index": "_index",
+ "column-order": np.array(out_cols, dtype=object),
+ "encoding-type": "dataframe",
+ "encoding-version": "0.1.0",
+ }
+ for i, j in attrs.items():
+ h5["obs"].attrs[i] = j
+
+ out_cols = []
+ for i in assay.feats.columns:
+ if i == "ids":
+ save_attr("var", "_index", "ids", assay.feats)
+ out_cols.append("_index")
+ elif i == "names":
+ save_attr("var", "gene_short_name", "names", assay.feats)
+ out_cols.append("gene_short_name")
+ else:
+ save_attr("var", i, i, assay.feats)
+ out_cols.append(i)
+
+ attrs = {
+ "_index": "_index",
+ "column-order": np.array(out_cols, dtype=object),
+ "encoding-type": "dataframe",
+ "encoding-version": "0.1.0",
+ }
+ for i, j in attrs.items():
+ h5["var"].attrs[i] = j
+
+ if len(emb_cols) > 0:
+ emb_names = np.array(emb_cols)
+ c = pd.Series([x[:-1] for x in emb_names])
+ for i in c.unique():
+ data = np.array([assay.cells.fetch_all(x) for x in emb_names[c == i]]).T
+ h5["obsm"].create_dataset(
+ i.lower().replace(f"{assay.name.lower()}_", "X_"), data=data
+ )
+
+ h5.close()
+ logger.info(
+ f"Exported {assay.cells.N} cells and {assay.feats.N} features "
+ f"to {h5ad_filename}"
+ )
+ return None
+
+
+def to_mtx(assay: Any, mtx_directory: str, compress: bool = False) -> None:
+ """Save an assay as a Matrix Market directory.
+
+ Args:
+ assay: Scarf assay. For example: `ds.RNA`
+ mtx_directory: Out directory where MTX file will be saved along with barcodes and features file
+ compress: If True, then the files are compressed and saved with .gz extension. (Default value: False).
+
+ Returns:
+ None
+ """
+ from scipy.sparse import coo_matrix
+
+ import gzip
+
+ if os.path.isdir(mtx_directory) is False:
+ os.mkdir(mtx_directory)
+
+ n_feats_per_cell = assay.cells.fetch_all(f"{assay.name}_nFeatures").astype(int)
+ tot_counts = int(n_feats_per_cell.sum())
+ if compress:
+ barcodes_fn = "barcodes.tsv.gz"
+ features_fn = "features.tsv.gz"
+ h = gzip.open(os.path.join(mtx_directory, "matrix.mtx.gz"), "wt")
+ else:
+ barcodes_fn = "barcodes.tsv"
+ features_fn = "genes.tsv"
+ h = open(os.path.join(mtx_directory, "matrix.mtx"), "w")
+ h.write("%%MatrixMarket matrix coordinate integer general\n% Generated by Scarf\n")
+ h.write(f"{assay.feats.N} {assay.cells.N} {tot_counts}\n")
+ s = 0
+ for values in assay.rawData.stream_blocks(
+ nthreads=assay.nthreads,
+ msg="Writing Matrix Market counts",
+ ):
+ block = coo_matrix(values)
+ df = pd.DataFrame(
+ {
+ "col": block.col + 1,
+ "row": block.row + s + 1,
+ "d": block.data,
+ }
+ )
+ df.to_csv(h, sep=" ", header=False, index=False, mode="a", lineterminator="\n")
+ s += block.shape[0]
+ h.close()
+ assay.cells.to_pandas_dataframe(["ids"]).to_csv(
+ os.path.join(mtx_directory, barcodes_fn), sep="\t", header=False, index=False
+ )
+
+ assay.feats.to_pandas_dataframe(["ids", "names"]).to_csv(
+ os.path.join(mtx_directory, features_fn), sep="\t", header=False, index=False
+ )
+ logger.info(
+ f"Exported {assay.cells.N} cells and {assay.feats.N} features "
+ f"to {mtx_directory}"
+ )
diff --git a/scarf/writers/h5ad.py b/scarf/writers/h5ad.py
new file mode 100644
index 00000000..f1a81e91
--- /dev/null
+++ b/scarf/writers/h5ad.py
@@ -0,0 +1,392 @@
+import time
+from collections.abc import Iterator
+from typing import Any
+
+import numpy as np
+from scipy.sparse import coo_matrix
+
+from ..storage.types import as_zarr_group
+from ..readers import H5adReader
+from ..readers.h5ad import _H5adAssayFeatures
+from ..storage.profiles import (
+ StorageProfile,
+ ZarrLocation,
+ resolve_storage_profile,
+)
+from ..utils.logging import logger
+from ..utils.progress import iter_progress
+
+
+def _validate_assay_names(names: tuple[str, ...]) -> None:
+ from ..storage.schema import validate_assay_name
+
+ for name in names:
+ validate_assay_name(name)
+
+
+class H5adToZarr:
+ """A class for converting data in anndata's H5ad format to Zarr hierarchy.
+
+ Args:
+ h5ad: Reader for the source H5AD file.
+ zarr_loc: The file name for the Zarr hierarchy or a store
+ assay_name: the name of the assay (e. g. 'RNA')
+ assay_split_key: A var column used to split features into assays.
+ assay_name_map: Feature type to assay name overrides.
+ workspace: An optional workspace id.
+ mem_budget: Memory available to the conversion. Accepts bytes, a
+ suffixed size (e.g. '8G'), or a fraction of total system memory (e.g. '0.6').
+ nthreads: Worker count for write-time concurrency. When None, auto-detected.
+
+ Attributes:
+ h5ad: A h5ad object (h5 file with added AnnData structure).
+ assayName: The Zarr hierarchy (array or group).
+ z: The Zarr hierarchy (array or group).
+ """
+
+ def __init__(
+ self,
+ h5ad: H5adReader,
+ zarr_loc: ZarrLocation,
+ assay_name: str | None = None,
+ workspace: str | None = None,
+ storage_options: dict[str, Any] | None = None,
+ mem_budget: int | str | None = None,
+ nthreads: int | None = None,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+ assay_split_key: str | None = None,
+ assay_name_map: dict[str, str] | None = None,
+ ) -> None:
+ from ..storage.budget import resolve_budget
+ from ..storage.schema import create_zarr_count_assay
+ from ..storage.stores import load_zarr
+
+ self.h5ad = h5ad
+ self.workspace = workspace
+ self.storage_options = storage_options
+ self.assaySplitKey = assay_split_key
+ self.assayNameMap = assay_name_map
+ self.assayFeatures: dict[str, _H5adAssayFeatures] | None
+ if assay_split_key is not None:
+ if assay_name is not None:
+ logger.warning(
+ "`assay_name` is ignored when `assay_split_key` is provided"
+ )
+ self.assayName = None
+ self.assayFeatures = self.h5ad.assay_feature_slices(
+ assay_split_key,
+ assay_name_map,
+ )
+ self.assayNames = tuple(self.assayFeatures)
+ elif assay_name is None:
+ logger.debug("Using RNA as the default assay name")
+ self.assayName = "RNA"
+ self.assayFeatures = None
+ self.assayNames = (self.assayName,)
+ else:
+ self.assayName = assay_name
+ self.assayFeatures = None
+ self.assayNames = (self.assayName,)
+ _validate_assay_names(self.assayNames)
+ self.resources = resolve_budget(mem_budget, nthreads)
+ self.profile = resolve_storage_profile(zarr_loc, profile)
+ self.h5ad.infer_storage_dtype(self.resources.memoryBytes)
+ csc_peak = self.h5ad.csc_conversion_peak_bytes()
+ if csc_peak > self.resources.memoryBytes:
+ raise MemoryError(
+ f"CSC to CSR conversion needs about {csc_peak} bytes, but the "
+ f"conversion memory limit is {self.resources.memoryBytes} bytes"
+ )
+ if csc_peak:
+ self.h5ad.materialize_csc()
+ self.storageDtype = getattr(
+ self.h5ad,
+ "storageDtype",
+ self.h5ad.matrixDtype,
+ )
+ self.z = load_zarr(zarr_loc=zarr_loc, mode="w", storage_options=storage_options)
+ self._ini_cell_data()
+ for resolved_assay_name in self.assayNames:
+ if self.assayFeatures is None:
+ feature_ids = self.h5ad.feat_ids()
+ feature_names = self.h5ad.feat_names()
+ else:
+ features = self.assayFeatures[resolved_assay_name]
+ feature_ids = features.featureIds
+ feature_names = features.featureNames
+ create_zarr_count_assay(
+ z=self.z,
+ assay_name=resolved_assay_name,
+ workspace=workspace,
+ n_cells=self.h5ad.nCells,
+ feat_ids=feature_ids,
+ feat_names=feature_names,
+ dtype=self.storageDtype,
+ profile=self.profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ )
+ self._ini_feature_data()
+
+ def _ini_cell_data(self) -> None:
+ from ..storage.arrays import create_zarr_obj_array
+ from ..storage.schema import create_cell_data
+
+ ids = self.h5ad.cell_ids()
+ g = create_cell_data(
+ root=self.z,
+ workspace=self.workspace,
+ ids=ids,
+ names=ids,
+ profile=self.profile,
+ )
+ for i, j in self.h5ad.get_cell_columns():
+ create_zarr_obj_array(g, i, j, j.dtype, profile=self.profile)
+
+ def _ini_feature_data(self) -> None:
+ from ..storage.arrays import create_zarr_obj_array
+
+ targets: list[tuple[Any, np.ndarray | None]] = []
+ for assay_name in self.assayNames:
+ if self.workspace is None:
+ group_path = f"{assay_name}/featureData"
+ else:
+ group_path = f"{self.workspace}/{assay_name}/featureData"
+ feat_group = as_zarr_group(self.z[group_path], name=group_path)
+ feature_indexes = (
+ None
+ if self.assayFeatures is None
+ else self.assayFeatures[assay_name].featureIndexes
+ )
+ targets.append((feat_group, feature_indexes))
+
+ # Stream one column at a time so a single decoded var column is held in
+ # memory rather than every column for the full feature axis at once.
+ for column_name, values in self.h5ad.get_feat_columns():
+ for feat_group, feature_indexes in targets:
+ if column_name in feat_group:
+ continue
+ selected = (
+ values if feature_indexes is None else values[feature_indexes]
+ )
+ create_zarr_obj_array(
+ feat_group,
+ column_name,
+ selected,
+ selected.dtype,
+ profile=self.profile,
+ )
+
+ def dump(self, batch_size: int | None = None) -> None:
+ """Write h5ad matrix data into the Zarr counts array.
+
+ Args:
+ batch_size: Number of source cells per batch. By default, a
+ destination-aligned value is selected within the memory budget.
+
+ Raises:
+ AssertionError: If written cell count does not match expected nCells.
+
+ Returns:
+ None
+ """
+ from ..storage.sharding import (
+ SparseShardBuffer,
+ resolve_sparse_import_batch,
+ write_sparse_bands,
+ )
+ from ..storage.schema import load_count_array
+
+ if batch_size is not None and batch_size <= 0:
+ raise ValueError("batch_size must be positive")
+
+ destinations = {
+ assay_name: load_count_array(self.z, assay_name, self.workspace)
+ for assay_name in self.assayNames
+ }
+ buffers = {
+ assay_name: SparseShardBuffer(destination)
+ for assay_name, destination in destinations.items()
+ }
+ logger.debug(
+ f"Writing counts with up to {self.resources.workers} row-band writer(s)"
+ )
+ started = time.perf_counter()
+ resident_source_bytes = self.h5ad.materialized_csr_bytes()
+ if self.assayFeatures is None:
+ feature_index_bytes = 0
+ else:
+ feature_index_bytes = sum(
+ assay.featureIndexes.nbytes for assay in self.assayFeatures.values()
+ )
+ projection = (
+ None if self.assayFeatures is None else self._assay_feature_projection()
+ )
+ if projection is not None:
+ resident_source_bytes += sum(array.nbytes for array in projection)
+ resident_source_bytes += feature_index_bytes
+ prepare = getattr(self.h5ad, "_prepare_sparse_import", None)
+ if callable(prepare):
+ prepare()
+ reader_resident = getattr(
+ self.h5ad,
+ "_sparse_import_resident_bytes",
+ None,
+ )
+ if callable(reader_resident):
+ resident_source_bytes += max(0, int(reader_resident()))
+ source_itemsize = np.dtype(self.h5ad.sourceMatrixDtype).itemsize
+ projection_value_bytes = max(
+ source_itemsize,
+ *(destination.dtype.itemsize for destination in destinations.values()),
+ )
+
+ def extra_producer_bytes(rows: int) -> int:
+ source_rows = min(rows, self.h5ad.nCells)
+ extra = (
+ source_rows * self.h5ad.nFeatures * source_itemsize
+ if self.h5ad.matrixOrientation == "dense"
+ else 0
+ )
+ if self.assayFeatures is not None:
+ source_values = max(0, int(self.h5ad.max_batch_nnz(rows)))
+ extra += source_values * (
+ projection_value_bytes
+ + 4 * np.dtype(np.int64).itemsize
+ + np.dtype(np.bool_).itemsize
+ )
+ return extra
+
+ plan = resolve_sparse_import_batch(
+ tuple(destinations.values()),
+ nRows=self.h5ad.nCells,
+ resources=self.resources,
+ maxWindowNnz=self.h5ad.max_batch_nnz,
+ sourceDtype=self.h5ad.sourceMatrixDtype,
+ batchRows=batch_size,
+ residentBytes=resident_source_bytes,
+ producerStagingBytes=self.h5ad.producer_batch_staging_bytes,
+ extraProducerBytes=extra_producer_bytes,
+ )
+ self._lastImportPlan = plan
+ resolved_batch_rows = plan.batchRows
+ logger.debug(
+ f"Resolved H5AD source batch rows={resolved_batch_rows} "
+ f"write_tasks={plan.writeTasks}"
+ )
+ write_sparse_bands(
+ self._count_shard_tasks(
+ resolved_batch_rows,
+ buffers,
+ destinations,
+ projection,
+ ),
+ resources=self.resources,
+ residentBytes=resident_source_bytes,
+ producerReserveBytes=plan.producerReserveBytes,
+ total=plan.writeTasks,
+ )
+ counts_seconds = time.perf_counter() - started
+
+ for assay_name, buffer in buffers.items():
+ if buffer.rows != self.h5ad.nCells:
+ raise AssertionError(
+ "ERROR: This is a bug in H5adToZarr. All cells might not have been "
+ f"successfully written into the {assay_name} counts array. "
+ "Please report this issue"
+ )
+ # counts is the durable physical orientation for H5AD imports. Assay
+ # readers use it directly when the optional derived countsT is absent.
+ logger.debug(f"Counts written in {counts_seconds:.1f}s")
+ logger.info(
+ f"Wrote {self.h5ad.nCells} cells and {self.h5ad.nFeatures} features "
+ f"from H5AD to {len(destinations)} assay(s)"
+ )
+
+ def _count_shard_tasks(
+ self,
+ batch_size: int,
+ buffers: dict[str, Any],
+ destinations: dict[str, Any],
+ projection: tuple[np.ndarray, np.ndarray] | None,
+ ) -> Iterator[Any]:
+ """Yield complete row-band writes from one serial pass over the source."""
+ from ..storage.sharding import SparseWriteBand, sparse_matrix_bytes
+
+ n_batches = (self.h5ad.nCells + batch_size - 1) // batch_size
+ stream = iter_progress(
+ self.h5ad.consume(batch_size),
+ total=n_batches,
+ desc="Writing counts",
+ )
+ if self.assayFeatures is None:
+ buffer = buffers[self.assayNames[0]]
+ destination = destinations[self.assayNames[0]]
+ for matrix in stream:
+ chunk = matrix.tocoo(copy=False)
+ source_bytes = sparse_matrix_bytes(matrix, chunk)
+ for band in buffer.add(chunk):
+ producer_bytes = source_bytes + sum(
+ item.residentBytes for item in buffers.values()
+ )
+ yield SparseWriteBand(destination, band, producer_bytes)
+ if self.h5ad.nCells:
+ del matrix, chunk
+ else:
+ if projection is None:
+ raise RuntimeError("Multi-assay projection was not initialized")
+ codes, columns = projection
+ for matrix in stream:
+ chunk = matrix.tocoo(copy=False)
+ source_bytes = sparse_matrix_bytes(matrix, chunk)
+ batch_codes = codes[chunk.col]
+ batch_columns = columns[chunk.col]
+ source_bytes += batch_codes.nbytes + batch_columns.nbytes
+ for code, assay_name in enumerate(self.assayNames):
+ buffer = buffers[assay_name]
+ selected = batch_codes == code
+ # Every assay sees every batch, including one with no
+ # values, so all buffers share the same row offsets.
+ projected = coo_matrix(
+ (
+ chunk.data[selected],
+ (chunk.row[selected], batch_columns[selected]),
+ ),
+ shape=(chunk.shape[0], buffer.nColumns),
+ )
+ for band in buffer.add(projected):
+ producer_bytes = (
+ source_bytes
+ + selected.nbytes
+ + sparse_matrix_bytes(projected)
+ + sum(item.residentBytes for item in buffers.values())
+ )
+ yield SparseWriteBand(
+ destinations[assay_name],
+ band,
+ producer_bytes,
+ )
+ if self.h5ad.nCells:
+ del matrix, chunk, batch_codes, batch_columns, selected, projected
+ for assay_name, buffer in buffers.items():
+ for band in buffer.finish():
+ producer_bytes = sum(item.residentBytes for item in buffers.values())
+ yield SparseWriteBand(
+ destinations[assay_name],
+ band,
+ producer_bytes,
+ )
+
+ def _assay_feature_projection(self) -> tuple[np.ndarray, np.ndarray]:
+ """Map each source feature to its assay code and assay-local column."""
+ if self.assayFeatures is None:
+ raise RuntimeError("Multi-assay features have not been initialized")
+ codes = np.full(int(self.h5ad.nFeatures), -1, dtype=np.int64)
+ columns = np.zeros(int(self.h5ad.nFeatures), dtype=np.int64)
+ for code, assay_name in enumerate(self.assayNames):
+ indexes = self.assayFeatures[assay_name].featureIndexes
+ codes[indexes] = code
+ columns[indexes] = np.arange(indexes.size, dtype=np.int64)
+ return codes, columns
diff --git a/scarf/writers/loom.py b/scarf/writers/loom.py
new file mode 100644
index 00000000..cb78895e
--- /dev/null
+++ b/scarf/writers/loom.py
@@ -0,0 +1,175 @@
+from typing import Any
+
+import numpy as np
+
+from ..storage.types import as_zarr_group
+from ..readers import LoomReader
+from ..storage.profiles import (
+ StorageProfile,
+ ZarrLocation,
+ resolve_storage_profile,
+)
+from ..utils.logging import logger
+
+
+class LoomToZarr:
+ """A class for converting data in a Loom file to a Zarr hierarchy. Converts
+ a Loom file read using scarf.LoomReader into Scarf's Zarr format.
+
+ Args:
+ loom: LoomReader object used to open Loom format file
+ zarr_loc: Output Zarr filename with path
+ assay_name: Name for the output assay. If not provided then automatically set to RNA
+
+ Attributes:
+ loom: A scarf.LoomReader object used to open Loom format file.
+ fn: The file name for the Zarr hierarchy.
+ assayName: The Zarr hierarchy (array or group).
+ z: The Zarr hierarchy (array or group).
+ """
+
+ def __init__(
+ self,
+ loom: LoomReader,
+ zarr_loc: ZarrLocation,
+ assay_name: str | None = None,
+ workspace: str | None = None,
+ storage_options: dict[str, Any] | None = None,
+ mem_budget: int | str | None = None,
+ nthreads: int | None = None,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+ ) -> None:
+ from ..storage.budget import resolve_budget
+ from ..storage.schema import create_zarr_count_assay, validate_assay_name
+ from ..storage.stores import load_zarr
+
+ # TODO: support for multiple assay. Data from within individual layers can be treated as separate assays
+ self.loom = loom
+ self.resources = resolve_budget(mem_budget, nthreads)
+ self.profile = resolve_storage_profile(zarr_loc, profile)
+ self.workspace = workspace
+ self.storage_options = storage_options
+ if assay_name is None:
+ logger.debug("Using RNA as the default assay name")
+ self.assayName = "RNA"
+ else:
+ self.assayName = assay_name
+ validate_assay_name(self.assayName)
+ self.z = load_zarr(zarr_loc, mode="w", storage_options=storage_options)
+ self._ini_cell_data()
+ create_zarr_count_assay(
+ z=self.z,
+ assay_name=self.assayName,
+ workspace=workspace,
+ n_cells=self.loom.nCells,
+ feat_ids=self.loom.feature_ids(),
+ feat_names=self.loom.feature_names(),
+ dtype=self.loom.matrixDtype,
+ profile=self.profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ )
+ self._ini_feature_data()
+
+ def _ini_cell_data(self) -> None:
+ from ..storage.arrays import create_zarr_obj_array
+ from ..storage.schema import create_cell_data
+
+ ids = np.array(self.loom.cell_ids())
+ cell_group = create_cell_data(
+ root=self.z,
+ workspace=self.workspace,
+ ids=ids,
+ names=ids,
+ profile=self.profile,
+ )
+ for i, j in self.loom.get_cell_attrs():
+ try:
+ create_zarr_obj_array(
+ cell_group,
+ i,
+ j,
+ j.dtype,
+ profile=self.profile,
+ )
+ except UnicodeDecodeError:
+ logger.warning(f"Could not import {i} cell(column) attribute")
+
+ def _ini_feature_data(self) -> None:
+ from ..storage.arrays import create_zarr_obj_array
+
+ if self.workspace is None:
+ feat_group = as_zarr_group(
+ self.z[f"{self.assayName}/featureData"],
+ name=f"{self.assayName}/featureData",
+ )
+ else:
+ feat_group = as_zarr_group(
+ self.z[f"{self.workspace}/{self.assayName}/featureData"],
+ name=f"{self.workspace}/{self.assayName}/featureData",
+ )
+ for i, j in self.loom.get_feature_attrs():
+ create_zarr_obj_array(
+ feat_group,
+ i,
+ j,
+ j.dtype,
+ profile=self.profile,
+ )
+
+ def dump(self, batch_size: int = 1000) -> None:
+ """Write Loom matrix data into the Zarr counts array.
+
+ Args:
+ batch_size: Number of cells read from the source per batch.
+
+ Raises:
+ AssertionError: If written cell count does not match expected nCells.
+
+ Returns:
+ None
+ """
+ from ..storage.layout import array_shard_rows
+ from ..storage.sharding import (
+ accumulate_sparse_to_shards,
+ sparse_producer_peak_bytes,
+ )
+ from ..storage.schema import load_count_array
+
+ store = load_count_array(self.z, self.assayName, self.workspace)
+ source_rows = min(batch_size, self.loom.nCells)
+ buffered_rows = min(
+ self.loom.nCells,
+ batch_size + array_shard_rows(store),
+ )
+ source_nnz = source_rows * self.loom.nFeatures
+ buffered_nnz = buffered_rows * self.loom.nFeatures
+ value_bytes = max(
+ np.dtype(self.loom.sourceMatrixDtype).itemsize,
+ np.dtype(self.loom.matrixDtype).itemsize,
+ np.dtype(store.dtype).itemsize,
+ )
+ dense_source_bytes = source_nnz * np.dtype(self.loom.sourceMatrixDtype).itemsize
+ total_cells_written = accumulate_sparse_to_shards(
+ store,
+ self.loom.consume(batch_size),
+ resources=self.resources,
+ producerReserveBytes=sparse_producer_peak_bytes(
+ buffered_nnz,
+ source_nnz,
+ value_bytes,
+ )
+ + dense_source_bytes,
+ msg="Writing Loom counts",
+ )
+ if total_cells_written != self.loom.nCells:
+ raise AssertionError(
+ "ERROR: This is a bug in LoomToZarr. All cells might not have been successfully "
+ "written into the zarr file. Please report this issue"
+ )
+ logger.info(
+ f"Wrote {self.loom.nCells} cells and {self.loom.nFeatures} features "
+ f"from Loom to assay {self.assayName}"
+ )
diff --git a/scarf/writers/seurat.py b/scarf/writers/seurat.py
new file mode 100644
index 00000000..a3f95f68
--- /dev/null
+++ b/scarf/writers/seurat.py
@@ -0,0 +1,999 @@
+import re
+from collections.abc import Iterator, Mapping, Sequence
+from dataclasses import dataclass
+from types import MappingProxyType
+from typing import Any
+
+import numpy as np
+import zarr
+from scipy.sparse import coo_matrix, issparse
+
+from ..readers import SeuratReader
+from ..readers.seurat import (
+ SeuratAssay,
+ SeuratMetadata,
+ SeuratMetadataColumn,
+ SeuratMembership,
+ SeuratNotice,
+ SeuratNumericVector,
+ SeuratRMatrix,
+ SeuratReduction,
+)
+from ..storage.profiles import StorageProfile, ZarrLocation
+from ..storage.refs import ArtifactRef
+
+
+_RESERVED_METADATA_COLUMNS = frozenset({"I", "ids", "names"})
+_DEFAULT_BLOCK_ROWS = 65_536
+
+
+def _named_result_key(name: str) -> str:
+ normalized = re.sub(r"[^a-z0-9]+", "_", name.casefold()).strip("_")
+ if not normalized:
+ normalized = "imported_result"
+ elif not normalized[0].isalpha():
+ normalized = f"imported_{normalized}"
+ return normalized
+
+
+@dataclass(frozen=True, slots=True)
+class SeuratImportResult:
+ assayNames: tuple[str, ...]
+ defaultAssay: str
+ cellSelection: ArtifactRef
+ reductionArtifacts: Mapping[str, ArtifactRef]
+ notices: tuple[SeuratNotice, ...] = ()
+
+ def __post_init__(self) -> None:
+ object.__setattr__(
+ self,
+ "reductionArtifacts",
+ MappingProxyType(dict(self.reductionArtifacts)),
+ )
+
+ @property
+ def artifactRefs(self) -> tuple[ArtifactRef, ...]:
+ return (self.cellSelection, *self.reductionArtifacts.values())
+
+
+def _string_blocks(
+ values: Sequence[str],
+ block_rows: int,
+) -> Iterator[tuple[str, ...]]:
+ read_block = getattr(values, "read_block", None)
+ for start in range(0, len(values), block_rows):
+ stop = min(start + block_rows, len(values))
+ block = read_block(start, stop) if callable(read_block) else values[start:stop]
+ yield tuple(_decode_text(value) for value in block)
+
+
+def _bounded_string_dtype(
+ values: Sequence[str],
+ block_rows: int,
+) -> np.dtype[Any]:
+ maximum = 1
+ for block in _string_blocks(values, block_rows):
+ maximum = max(maximum, max((len(value) for value in block), default=1))
+ return np.dtype(f"U{maximum}")
+
+
+def _decode_text(value: str | bytes | None) -> str:
+ if value is None:
+ return ""
+ if isinstance(value, bytes):
+ return value.decode("utf-8")
+ return value
+
+
+class SeuratToZarr:
+ def __init__(
+ self,
+ reader: SeuratReader,
+ zarr_loc: ZarrLocation,
+ workspace: str | None = None,
+ storage_options: dict[str, Any] | None = None,
+ mem_budget: int | str | None = None,
+ nthreads: int | None = None,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+ ) -> None:
+ from ..storage.budget import resolve_budget
+ from ..storage.schema import (
+ create_empty_cell_data,
+ create_empty_zarr_count_assay,
+ validate_assay_name,
+ )
+ from ..storage.stores import load_zarr
+
+ resources = resolve_budget(mem_budget, nthreads)
+ inspection = reader.inspection
+ assay_names = tuple(reader.assayNames)
+ for assay_name in assay_names:
+ validate_assay_name(assay_name)
+ if inspection.activeAssay not in assay_names:
+ raise ValueError(
+ f"Active assay {inspection.activeAssay!r} is not selected for import"
+ )
+
+ assays = tuple(reader.get_assay(name) for name in assay_names)
+ blocked_reductions = tuple(
+ item for item in inspection.reductions if not item.importable
+ )
+ if blocked_reductions:
+ details = ", ".join(
+ f"{item.name} "
+ f"({item.blockingDiagnostic.code if item.blockingDiagnostic is not None else 'unsupported'})"
+ for item in blocked_reductions
+ )
+ raise ValueError(f"Selected reductions cannot be imported: {details}")
+ reductions = tuple(
+ reader.get_reduction(item.name) for item in inspection.reductions
+ )
+ reduction_result_names = {
+ reduction.name: _named_result_key(reduction.name)
+ for reduction in reductions
+ }
+ if len(set(reduction_result_names.values())) != len(reduction_result_names):
+ raise ValueError(
+ "Selected reduction names collide after conversion to snake case"
+ )
+ imported_assays = set(assay_names)
+ missing_reduction_assays = tuple(
+ reduction.name
+ for reduction in reductions
+ if reduction.assayUsed not in imported_assays
+ )
+ if missing_reduction_assays:
+ raise ValueError(
+ "Selected reductions reference assays that are not selected for "
+ f"import: {', '.join(missing_reduction_assays)}"
+ )
+ active_identity = reader.activeIdentity
+ self._validate_metadata_names(reader.cellMetadata, "cell")
+ self._validate_metadata_name(active_identity.name, "cell")
+ membership_names = {
+ f"{assay.name}_I"
+ for assay in assays
+ if assay.sourceClass == "Assay5" and not assay.cellMembership.allIncluded
+ }
+ cell_names = set(reader.cellMetadata.columnNames)
+ conflicts = sorted(cell_names.intersection(membership_names))
+ if conflicts:
+ raise ValueError(
+ "Assay membership columns conflict with cell metadata: "
+ + ", ".join(conflicts)
+ )
+ for assay in assays:
+ self._validate_metadata_names(
+ assay.featureMetadata,
+ f"{assay.name} feature",
+ )
+ dtype = np.dtype(assay.counts.dtype)
+ if dtype.kind not in "biuf":
+ raise TypeError(
+ f"Assay {assay.name!r} counts use unsupported dtype {dtype}"
+ )
+ expected_shape = (len(assay.featureIds), len(reader.cellIds))
+ if tuple(assay.counts.shape) != expected_shape:
+ raise ValueError(
+ f"Assay {assay.name!r} has shape {assay.counts.shape}, "
+ f"expected {expected_shape}"
+ )
+
+ source_digest = bytes.fromhex(reader.document.source.source_sha256)
+ if len(source_digest) != 32:
+ raise ValueError("Seurat source SHA-256 digest must contain 32 bytes")
+ string_block_rows = max(
+ 1,
+ min(
+ _DEFAULT_BLOCK_ROWS,
+ int(resources.memoryBytes) // (8 * 64),
+ ),
+ )
+ cell_dtype = _bounded_string_dtype(reader.cellIds, string_block_rows)
+ feature_dtypes = {
+ assay.name: _bounded_string_dtype(assay.featureIds, string_block_rows)
+ for assay in assays
+ }
+
+ self.reader = reader
+ self.workspace = workspace
+ self.storageOptions = storage_options
+ self.resources = resources
+ from ..storage.profiles import resolve_storage_profile
+
+ self.profile = resolve_storage_profile(zarr_loc, profile)
+ self.assayNames = assay_names
+ self.defaultAssay = inspection.activeAssay
+ self._assays = assays
+ self._reductions = reductions
+ self._reductionResultNames = reduction_result_names
+ self._activeIdentity = active_identity
+ self._sourceDigest = source_digest
+ self._notices = self._collect_notices(inspection.notices, assays, reductions)
+ self._residentSourceBytes = sum(
+ max(0, int(assay.counts.resident_bytes)) for assay in assays
+ )
+ self._lastImportPlans: dict[str, Any] = {}
+ self._lastDenseBatchRows: dict[str, int] = {}
+
+ self.z = load_zarr(
+ zarr_loc=zarr_loc,
+ mode="w",
+ storage_options=storage_options,
+ )
+ self.root = (
+ self.z
+ if workspace is None
+ else self.z.create_group(workspace, overwrite=True)
+ )
+ self.root.attrs["complete"] = False
+ self.root.attrs["scarf:import_source"] = "seurat"
+ self.root.attrs["scarf:import_complete"] = False
+ self.root.attrs["scarf:import_source_sha256"] = (
+ reader.document.source.source_sha256
+ )
+ self.root.attrs["scarf:import_payload_sha256"] = (
+ reader.document.source.payload_sha256
+ )
+
+ self.cellData = create_empty_cell_data(
+ self.z,
+ workspace,
+ len(reader.cellIds),
+ cell_dtype,
+ cell_dtype,
+ profile=self.profile,
+ )
+ self.counts: dict[str, zarr.Array] = {}
+ self.featureData: dict[str, zarr.Group] = {}
+ for assay in assays:
+ feature_dtype = feature_dtypes[assay.name]
+ counts, feature_data = create_empty_zarr_count_assay(
+ self.z,
+ assay.name,
+ workspace,
+ len(reader.cellIds),
+ len(assay.featureIds),
+ feature_dtype,
+ feature_dtype,
+ dtype=assay.counts.dtype,
+ profile=self.profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ )
+ self.counts[assay.name] = counts
+ self.featureData[assay.name] = feature_data
+ self.root.attrs["defaultAssay"] = self.defaultAssay
+
+ @staticmethod
+ def _validate_metadata_name(name: str, axis: str) -> None:
+ if name in _RESERVED_METADATA_COLUMNS:
+ raise ValueError(f"{axis} metadata column {name!r} is reserved")
+ if name.startswith("__scarf_missing__"):
+ raise ValueError(
+ f"{axis} metadata column {name!r} uses Scarf's internal prefix"
+ )
+
+ @classmethod
+ def _validate_metadata_names(cls, metadata: SeuratMetadata, axis: str) -> None:
+ names = metadata.columnNames
+ if len(set(names)) != len(names):
+ raise ValueError(f"{axis} metadata contains duplicate column names")
+ for name in names:
+ cls._validate_metadata_name(name, axis)
+ generated_masks = {f"__scarf_missing__{name}" for name in names}
+ overlap = generated_masks.intersection(names)
+ if overlap:
+ raise ValueError(
+ f"{axis} metadata conflicts with generated missing masks: "
+ + ", ".join(sorted(overlap))
+ )
+
+ @staticmethod
+ def _collect_notices(
+ root_notices: tuple[SeuratNotice, ...],
+ assays: tuple[SeuratAssay, ...],
+ reductions: tuple[SeuratReduction, ...],
+ ) -> tuple[SeuratNotice, ...]:
+ return (
+ *root_notices,
+ *(notice for assay in assays for notice in assay.notices),
+ *(notice for reduction in reductions for notice in reduction.notices),
+ )
+
+ def dump(self, batch_size: int | None = None) -> SeuratImportResult:
+ if batch_size is not None and (
+ isinstance(batch_size, bool) or int(batch_size) <= 0
+ ):
+ raise ValueError("batch_size must be positive")
+ requested_rows = None if batch_size is None else int(batch_size)
+
+ self.reader.inspection
+ self.root.attrs["complete"] = False
+ self.root.attrs["scarf:import_complete"] = False
+ try:
+ metadata_rows = self._bounded_block_rows(
+ requested_rows,
+ row_bytes=64,
+ )
+ self._write_cell_data(metadata_rows)
+ for assay in self._assays:
+ self._write_feature_data(assay, metadata_rows)
+ for assay in self._assays:
+ self._write_counts(assay, requested_rows)
+ cell_selection = self._write_cell_selection()
+ reduction_artifacts = self._write_reductions(
+ cell_selection,
+ requested_rows,
+ )
+ except BaseException:
+ self.root.attrs["complete"] = False
+ self.root.attrs["scarf:import_complete"] = False
+ raise
+ self.root.attrs["complete"] = True
+ self.root.attrs["scarf:import_complete"] = True
+ return SeuratImportResult(
+ assayNames=self.assayNames,
+ defaultAssay=self.defaultAssay,
+ cellSelection=cell_selection,
+ reductionArtifacts=reduction_artifacts,
+ notices=self._notices,
+ )
+
+ def _bounded_block_rows(
+ self,
+ requested: int | None,
+ *,
+ row_bytes: int,
+ ) -> int:
+ bytes_per_row = max(1, int(row_bytes))
+ memory_rows = max(
+ 1,
+ int(self.resources.memoryBytes) // (8 * bytes_per_row),
+ )
+ preferred = _DEFAULT_BLOCK_ROWS if requested is None else requested
+ return int(max(1, min(int(preferred), memory_rows)))
+
+ def _write_cell_data(self, block_rows: int) -> None:
+ self._write_string_axis(
+ self.cellData["ids"],
+ self.cellData["names"],
+ self.reader.cellIds,
+ block_rows,
+ )
+ for column in self.reader.cellMetadata.columns:
+ self._write_metadata_column(self.cellData, column, block_rows)
+ self._write_metadata_column(
+ self.cellData,
+ self._activeIdentity,
+ block_rows,
+ )
+ for assay in self._assays:
+ if assay.sourceClass != "Assay5" or assay.cellMembership.allIncluded:
+ continue
+ column_name = f"{assay.name}_I"
+ output = self._create_boolean_column(
+ self.cellData,
+ column_name,
+ assay.cellMembership,
+ block_rows,
+ )
+ output.attrs["assay"] = assay.name
+ output.attrs["role"] = "assay_membership"
+
+ def _write_feature_data(self, assay: SeuratAssay, block_rows: int) -> None:
+ group = self.featureData[assay.name]
+ self._write_string_axis(
+ group["ids"],
+ group["names"],
+ assay.featureIds,
+ block_rows,
+ )
+ for column in assay.featureMetadata.columns:
+ self._write_metadata_column(group, column, block_rows)
+
+ @staticmethod
+ def _write_string_axis(
+ ids: Any,
+ names: Any,
+ values: Sequence[str],
+ block_rows: int,
+ ) -> None:
+ if int(ids.shape[0]) != len(values) or int(names.shape[0]) != len(values):
+ raise ValueError("String axis length does not match its destination")
+ start = 0
+ for values_block in _string_blocks(values, block_rows):
+ stop = start + len(values_block)
+ block = np.asarray(values_block, dtype=ids.dtype)
+ ids[start:stop] = block
+ names[start:stop] = block.astype(names.dtype, copy=False)
+ start = stop
+
+ def _metadata_dtype(
+ self,
+ column: SeuratMetadataColumn,
+ block_rows: int,
+ ) -> np.dtype[Any]:
+ if column.kind == "logical":
+ return np.dtype(bool)
+ if column.kind == "integer":
+ return np.dtype(np.int64)
+ if column.kind == "real":
+ return np.dtype(np.float64)
+ if column.kind == "factor":
+ return _bounded_string_dtype(column.levels, block_rows)
+ if column.kind != "character":
+ raise TypeError(f"Unsupported Seurat metadata kind {column.kind!r}")
+ maximum = 1
+ for start in range(0, column.length, block_rows):
+ stop = min(start + block_rows, column.length)
+ block = column.read_block(start, stop)
+ values = block.values
+ if not isinstance(values, tuple):
+ raise TypeError("Character metadata did not return string values")
+ maximum = max(
+ maximum,
+ max((len(_decode_text(value)) for value in values), default=1),
+ )
+ return np.dtype(f"U{maximum}")
+
+ def _metadata_blocks(
+ self,
+ column: SeuratMetadataColumn,
+ dtype: np.dtype[Any],
+ block_rows: int,
+ ) -> Iterator[Any]:
+ from ..storage.arrays import MetadataBlock
+
+ for start in range(0, column.length, block_rows):
+ stop = min(start + block_rows, column.length)
+ block = column.read_block(start, stop)
+ missing = np.asarray(block.missing, dtype=bool)
+ if column.kind == "character":
+ if not isinstance(block.values, tuple):
+ raise TypeError("Character metadata did not return string values")
+ values = np.asarray(
+ [_decode_text(value) for value in block.values],
+ dtype=dtype,
+ )
+ elif column.kind == "factor":
+ codes = np.asarray(block.values)
+ values = np.asarray(
+ [
+ "" if missing[index] else column.levels[int(code) - 1]
+ for index, code in enumerate(codes)
+ ],
+ dtype=dtype,
+ )
+ else:
+ values = np.asarray(block.values, dtype=dtype)
+ if column.kind == "logical":
+ values[missing] = False
+ elif column.kind == "integer":
+ values[missing] = 0
+ elif column.kind == "real":
+ values[missing] = np.nan
+ yield MetadataBlock(start, values, missing)
+
+ def _write_metadata_column(
+ self,
+ group: zarr.Group,
+ column: SeuratMetadataColumn,
+ block_rows: int,
+ ) -> zarr.Array:
+ from ..storage.arrays import create_streamed_metadata_column
+
+ dtype = self._metadata_dtype(column, block_rows)
+ output = create_streamed_metadata_column(
+ group,
+ column.name,
+ shape=column.length,
+ dtype=dtype,
+ blocks=self._metadata_blocks(column, dtype, block_rows),
+ overwrite=True,
+ chunkSize=min(_DEFAULT_BLOCK_ROWS, max(1, column.length)),
+ hasMissing=True,
+ profile=self.profile,
+ )
+ if column.kind == "factor":
+ output.attrs["levels"] = list(column.levels)
+ output.attrs["ordered"] = bool(column.ordered)
+ return output
+
+ def _create_boolean_column(
+ self,
+ group: zarr.Group,
+ name: str,
+ values: SeuratMembership,
+ block_rows: int,
+ ) -> zarr.Array:
+ from ..storage.arrays import MetadataBlock, create_streamed_metadata_column
+
+ read_block = values.read_block
+ return create_streamed_metadata_column(
+ group,
+ name,
+ shape=len(values),
+ dtype=bool,
+ blocks=(
+ MetadataBlock(
+ start,
+ read_block(start, min(start + block_rows, len(values))),
+ )
+ for start in range(0, len(values), block_rows)
+ ),
+ overwrite=True,
+ chunkSize=min(_DEFAULT_BLOCK_ROWS, max(1, len(values))),
+ profile=self.profile,
+ )
+
+ def _source_staging_peak(
+ self,
+ source: Any,
+ rows: int,
+ ) -> int:
+ n_cells = int(source.shape[1])
+ width = max(1, min(int(rows), max(1, n_cells)))
+ peak = 0
+ for start in range(0, n_cells, width):
+ stop = min(start + width, n_cells)
+ estimate = source.estimate_read_memory(start, stop)
+ peak = max(
+ peak,
+ max(0, int(estimate.workingBytes)) + max(0, int(estimate.outputBytes)),
+ )
+ return peak
+
+ def _write_counts(
+ self,
+ assay: SeuratAssay,
+ requested_rows: int | None,
+ ) -> None:
+ source = assay.counts
+ destination = self.counts[assay.name]
+ n_cells = len(self.reader.cellIds)
+ if n_cells == 0:
+ return
+ if source.is_sparse:
+ self._write_sparse_counts(
+ assay.name,
+ source,
+ destination,
+ requested_rows,
+ )
+ else:
+ self._write_dense_counts(
+ assay.name,
+ source,
+ destination,
+ requested_rows,
+ )
+
+ def _write_sparse_counts(
+ self,
+ assay_name: str,
+ source: Any,
+ destination: zarr.Array,
+ requested_rows: int | None,
+ ) -> None:
+ from ..storage.sharding import (
+ accumulate_sparse_to_shards,
+ resolve_sparse_import_batch,
+ )
+
+ n_cells = int(source.shape[1])
+ n_features = int(source.shape[0])
+ staging_cache: dict[int, int] = {}
+
+ def staging(rows: int) -> int:
+ width = max(1, min(int(rows), max(1, n_cells)))
+ if width not in staging_cache:
+ staging_cache[width] = self._source_staging_peak(source, width)
+ return staging_cache[width]
+
+ def max_window_nnz(rows: int) -> int:
+ width = max(0, min(int(rows), n_cells))
+ if width == 0:
+ return 0
+ dense_bound = width * n_features
+ estimated_values = (
+ staging(width) + max(1, source.dtype.itemsize) - 1
+ ) // max(1, source.dtype.itemsize)
+ return int(min(dense_bound, max(0, estimated_values)))
+
+ plan = resolve_sparse_import_batch(
+ (destination,),
+ nRows=n_cells,
+ resources=self.resources,
+ maxWindowNnz=max_window_nnz,
+ sourceDtype=source.dtype,
+ batchRows=requested_rows,
+ residentBytes=self._residentSourceBytes,
+ producerStagingBytes=staging,
+ )
+ self._lastImportPlans[assay_name] = plan
+ self._lastImportPlan = plan
+
+ def batches() -> Iterator[coo_matrix]:
+ for start in range(0, n_cells, plan.batchRows):
+ stop = min(start + plan.batchRows, n_cells)
+ raw = source.read_cells(start, stop)
+ block = (
+ raw.tocoo(copy=False)
+ if issparse(raw)
+ else coo_matrix(np.asarray(raw))
+ )
+ expected = (stop - start, int(destination.shape[1]))
+ if block.shape != expected:
+ raise ValueError(
+ f"Assay {assay_name!r} source returned shape "
+ f"{block.shape}, expected {expected}"
+ )
+ yield block
+
+ rows = accumulate_sparse_to_shards(
+ destination,
+ batches(),
+ resources=self.resources,
+ residentBytes=self._residentSourceBytes,
+ producerReserveBytes=plan.producerReserveBytes,
+ msg=f"Writing {assay_name} counts",
+ )
+ if rows != n_cells:
+ raise ValueError(
+ f"Assay {assay_name!r} wrote {rows} count rows, expected {n_cells}"
+ )
+
+ @staticmethod
+ def _dense_write_reserve(destination: zarr.Array) -> int:
+ from ..storage.layout import array_shard_rows
+
+ rows = min(int(destination.shape[0]), array_shard_rows(destination))
+ columns = max(1, int(destination.shape[1]))
+ itemsize = max(1, int(np.dtype(destination.dtype).itemsize))
+ chunk_rows = min(max(1, rows), int(destination.chunks[0]))
+ chunk_columns = min(columns, int(destination.chunks[1]))
+ chunks = (
+ (max(1, rows) + chunk_rows - 1)
+ // chunk_rows
+ * ((columns + chunk_columns - 1) // chunk_columns)
+ )
+ band_bytes = max(1, rows) * columns * itemsize
+ chunk_bytes = chunk_rows * chunk_columns * itemsize
+ encoded_chunk = chunk_bytes + chunk_bytes // 128 + 1024
+ return int(
+ band_bytes + chunk_bytes + 2 * chunks * encoded_chunk + chunks * 16 + 1024
+ )
+
+ def _resolve_dense_batch_rows(
+ self,
+ source: Any,
+ destination: zarr.Array,
+ requested_rows: int | None,
+ ) -> tuple[int, int]:
+ from ..storage.layout import array_shard_rows
+ from ..storage.partition import affordable_width
+
+ n_cells = int(source.shape[1])
+ task_reserve = self._dense_write_reserve(destination)
+ staging_cache: dict[int, int] = {}
+
+ def staging(rows: int) -> int:
+ width = max(1, min(int(rows), n_cells))
+ if width not in staging_cache:
+ staging_cache[width] = self._source_staging_peak(source, width)
+ return staging_cache[width]
+
+ def fits(rows: int) -> bool:
+ required = self._residentSourceBytes + staging(rows) + task_reserve
+ return bool(required <= int(self.resources.memoryBytes))
+
+ if requested_rows is not None:
+ rows = min(requested_rows, n_cells)
+ if not fits(rows):
+ raise MemoryError(
+ "Dense Seurat import cannot fit the requested source batch and "
+ "one destination row band within mem_budget"
+ )
+ else:
+ preferred = min(n_cells, array_shard_rows(destination))
+ rows = affordable_width(fits, preferred)
+ if rows < 1:
+ raise MemoryError(
+ "Dense Seurat import cannot fit one source row and one "
+ "destination row band within mem_budget"
+ )
+ return rows, staging(rows)
+
+ def _write_dense_counts(
+ self,
+ assay_name: str,
+ source: Any,
+ destination: zarr.Array,
+ requested_rows: int | None,
+ ) -> None:
+ from ..storage.budget import ResourceBudget
+ from ..storage.sharding import write_dense_from_row_batches
+
+ n_cells = int(source.shape[1])
+ rows, producer_reserve = self._resolve_dense_batch_rows(
+ source,
+ destination,
+ requested_rows,
+ )
+ self._lastDenseBatchRows[assay_name] = rows
+ writer_memory = (
+ self.resources.memoryBytes - self._residentSourceBytes - producer_reserve
+ )
+ if writer_memory < 1:
+ raise MemoryError("Dense Seurat import has no memory left for Zarr writes")
+ writer_resources = ResourceBudget(
+ memoryBytes=writer_memory,
+ workers=self.resources.workers,
+ )
+
+ def batches() -> Iterator[np.ndarray]:
+ for start in range(0, n_cells, rows):
+ stop = min(start + rows, n_cells)
+ raw = source.read_cells(start, stop)
+ block = raw.toarray() if issparse(raw) else np.asarray(raw)
+ expected = (stop - start, int(destination.shape[1]))
+ if block.shape != expected:
+ raise ValueError(
+ f"Assay {assay_name!r} source returned shape "
+ f"{block.shape}, expected {expected}"
+ )
+ yield np.ascontiguousarray(block, dtype=destination.dtype)
+
+ written = write_dense_from_row_batches(
+ destination,
+ batches(),
+ dtype=destination.dtype,
+ msg=f"Writing {assay_name} counts",
+ resources=writer_resources,
+ )
+ if written != n_cells:
+ raise ValueError(
+ f"Assay {assay_name!r} wrote {written} count rows, expected {n_cells}"
+ )
+
+ def _write_cell_selection(self) -> ArtifactRef:
+ from ..storage.selections import resolve_stored_selection_artifact
+
+ ref = resolve_stored_selection_artifact(
+ self.root,
+ table_path="cellData",
+ id_column="ids",
+ source_column="I",
+ scope="datastore",
+ kind="cell_selection",
+ operation="import_cell_selection",
+ parameters={"source": "seurat"},
+ inputs={"source_digest": self._sourceDigest},
+ )
+ included = self.cellData["I"]
+ included.attrs["source_artifact"] = ref.to_dict()
+ included.attrs["source_value"] = "values"
+ return ref
+
+ @staticmethod
+ def _floating_dtype(dtype: Any) -> np.dtype[Any]:
+ source: np.dtype[Any] = np.dtype(dtype)
+ if source.kind == "f":
+ return np.dtype(source.str)
+ if source.kind in "biu":
+ return np.dtype(np.float64)
+ raise TypeError(f"Reduction payload uses unsupported dtype {source}")
+
+ @staticmethod
+ def _matrix_blocks(
+ matrix: SeuratRMatrix,
+ block_rows: int,
+ dtype: np.dtype[Any],
+ ) -> Iterator[np.ndarray]:
+ for start in range(0, matrix.shape[0], block_rows):
+ stop = min(start + block_rows, matrix.shape[0])
+ yield np.asarray(matrix.read_rows(start, stop), dtype=dtype)
+
+ @staticmethod
+ def _vector_blocks(
+ vector: SeuratNumericVector,
+ block_rows: int,
+ dtype: np.dtype[Any],
+ ) -> Iterator[np.ndarray]:
+ for start in range(0, vector.length, block_rows):
+ stop = min(start + block_rows, vector.length)
+ yield np.asarray(vector.read_block(start, stop), dtype=dtype)
+
+ @classmethod
+ def _fingerprint_matrix(
+ cls,
+ matrix: SeuratRMatrix,
+ block_rows: int,
+ dtype: np.dtype[Any],
+ ) -> str:
+ from ..storage.artifacts import ValueFingerprintBuilder
+
+ builder = ValueFingerprintBuilder()
+ builder.begin_array("values", matrix.shape, dtype)
+ start = 0
+ for block in cls._matrix_blocks(matrix, block_rows, dtype):
+ builder.update_array_block("values", (start, 0), block)
+ start += int(block.shape[0])
+ builder.end_array("values")
+ return str(builder.hexdigest())
+
+ @classmethod
+ def _fingerprint_vector(
+ cls,
+ vector: SeuratNumericVector,
+ block_rows: int,
+ dtype: np.dtype[Any],
+ ) -> str:
+ from ..storage.artifacts import ValueFingerprintBuilder
+
+ builder = ValueFingerprintBuilder()
+ builder.begin_array("values", (vector.length,), dtype)
+ start = 0
+ for block in cls._vector_blocks(vector, block_rows, dtype):
+ builder.update_array_block("values", (start,), block)
+ start += int(block.shape[0])
+ builder.end_array("values")
+ return str(builder.hexdigest())
+
+ @staticmethod
+ def _fingerprint_feature_ids(
+ values: Sequence[str],
+ block_rows: int,
+ ) -> str:
+ from ..storage.artifacts import ValueFingerprintBuilder
+
+ dtype = _bounded_string_dtype(values, block_rows)
+ builder = ValueFingerprintBuilder()
+ builder.begin_array("values", (len(values),), dtype)
+ for start in range(0, len(values), block_rows):
+ stop = min(start + block_rows, len(values))
+ builder.update_array_block(
+ "values",
+ (start,),
+ np.asarray(values[start:stop], dtype=dtype),
+ )
+ builder.end_array("values")
+ return str(builder.hexdigest())
+
+ def _reduction_block_rows(
+ self,
+ reduction: SeuratReduction,
+ requested_rows: int | None,
+ ) -> int:
+ row_bytes = max(1, reduction.cellEmbeddings.shape[1]) * max(
+ 1, reduction.cellEmbeddings.dtype.itemsize
+ )
+ return self._bounded_block_rows(requested_rows, row_bytes=row_bytes)
+
+ def _write_reductions(
+ self,
+ cell_selection: ArtifactRef,
+ requested_rows: int | None,
+ ) -> dict[str, ArtifactRef]:
+ from ..embeddings.imported import (
+ write_imported_coordinates,
+ write_imported_embedding,
+ )
+
+ artifacts: dict[str, ArtifactRef] = {}
+ for reduction in self._reductions:
+ block_rows = self._reduction_block_rows(reduction, requested_rows)
+ coordinate_dtype = self._floating_dtype(reduction.cellEmbeddings.dtype)
+ coordinate_fingerprint = self._fingerprint_matrix(
+ reduction.cellEmbeddings,
+ block_rows,
+ coordinate_dtype,
+ )
+
+ def coordinate_blocks(
+ matrix: SeuratRMatrix = reduction.cellEmbeddings,
+ rows: int = block_rows,
+ dtype: np.dtype[Any] = coordinate_dtype,
+ ) -> Iterator[np.ndarray]:
+ return self._matrix_blocks(matrix, rows, dtype)
+
+ normalized_name = reduction.name.casefold()
+ if normalized_name in {"umap", "tsne", "t-sne"}:
+ role = "umap" if normalized_name == "umap" else "tsne"
+ ref = write_imported_embedding(
+ self.root,
+ assay=reduction.assayUsed,
+ dimreduc_key=reduction.name,
+ role=role,
+ coordinates=coordinate_blocks,
+ coordinate_shape=reduction.cellEmbeddings.shape,
+ coordinate_dtype=coordinate_dtype,
+ source_digest=self._sourceDigest,
+ payload_fingerprints={"values": coordinate_fingerprint},
+ source_cell_ids=self.reader.cellIds,
+ cell_selection=cell_selection,
+ cell_key="I",
+ cell_data=self.cellData,
+ named_result=self._reductionResultNames[reduction.name],
+ feat_key="I",
+ block_rows=block_rows,
+ )
+ else:
+ payload_fingerprints = {"data": coordinate_fingerprint}
+ loadings = reduction.featureLoadings
+ loading_blocks = None
+ loading_shape = None
+ loading_dtype = None
+ feature_ids: Sequence[str] | None = None
+ if loadings is not None:
+ loading_dtype = self._floating_dtype(loadings.dtype)
+ payload_fingerprints["loadings"] = self._fingerprint_matrix(
+ loadings,
+ block_rows,
+ loading_dtype,
+ )
+ feature_ids = loadings.rowIds
+ payload_fingerprints["feature_ids"] = self._fingerprint_feature_ids(
+ feature_ids, block_rows
+ )
+ loading_shape = loadings.shape
+
+ def loading_blocks(
+ matrix: SeuratRMatrix = loadings,
+ rows: int = block_rows,
+ dtype: np.dtype[Any] = loading_dtype,
+ ) -> Iterator[np.ndarray]:
+ return self._matrix_blocks(matrix, rows, dtype)
+
+ stdev = reduction.stdev
+ stdev_blocks = None
+ stdev_shape = None
+ stdev_dtype = None
+ if stdev is not None:
+ stdev_dtype = self._floating_dtype(stdev.dtype)
+ payload_fingerprints["stdev"] = self._fingerprint_vector(
+ stdev,
+ block_rows,
+ stdev_dtype,
+ )
+ stdev_shape = (stdev.length,)
+
+ def stdev_blocks(
+ vector: SeuratNumericVector = stdev,
+ rows: int = block_rows,
+ dtype: np.dtype[Any] = stdev_dtype,
+ ) -> Iterator[np.ndarray]:
+ return self._vector_blocks(vector, rows, dtype)
+
+ ref = write_imported_coordinates(
+ self.root,
+ assay=reduction.assayUsed,
+ dimreduc_key=reduction.name,
+ role=reduction.role,
+ coordinates=coordinate_blocks,
+ coordinate_shape=reduction.cellEmbeddings.shape,
+ coordinate_dtype=coordinate_dtype,
+ source_digest=self._sourceDigest,
+ payload_fingerprints=payload_fingerprints,
+ source_cell_ids=self.reader.cellIds,
+ cell_selection=cell_selection,
+ cell_key="I",
+ loadings=loading_blocks,
+ loadings_shape=loading_shape,
+ loadings_dtype=loading_dtype,
+ feature_ids=feature_ids,
+ stdev=stdev_blocks,
+ stdev_shape=stdev_shape,
+ stdev_dtype=stdev_dtype,
+ named_result=self._reductionResultNames[reduction.name],
+ feat_key="I",
+ block_rows=block_rows,
+ )
+ artifacts[reduction.name] = ref
+ return artifacts
+
+
+__all__ = ["SeuratImportResult", "SeuratToZarr"]
diff --git a/scarf/writers/sparse.py b/scarf/writers/sparse.py
new file mode 100644
index 00000000..c6c8a70c
--- /dev/null
+++ b/scarf/writers/sparse.py
@@ -0,0 +1,288 @@
+from collections.abc import Iterator
+from typing import Any
+
+import numpy as np
+import pandas as pd
+from scipy.sparse import coo_matrix, csr_matrix
+
+from ..storage.profiles import (
+ StorageProfile,
+ ZarrLocation,
+ resolve_storage_profile,
+)
+from ..storage.sharding import accumulate_sparse_to_shards
+from ..utils.logging import logger
+from ..utils.progress import iter_progress
+
+
+class SparseToZarr:
+ """A class for converting data in a sparse matrix to a Zarr hierarchy.
+
+ Args:
+ csr_mat: A CSR format sparse matrix
+ zarr_loc: Output Zarr filename with path
+ cell_ids: Cell IDs for the cells in the dataset.
+ feature_ids: Feature IDs for the features in the dataset.
+ assay_name: Name for the output assay. If not provided then automatically set to RNA.
+
+ Raises:
+ ValueError: Raised if number of input cell or feature IDs does not match the matrix.
+ AssertionError: Catches eventual bugs in the class, if number of cells does not match after transformation.
+
+ Attributes:
+ mat: Input CSR matrix
+ fn: The file name for the Zarr hierarchy.
+ assayName: The Zarr hierarchy (array or group).
+ z: The Zarr hierarchy (array or group).
+ """
+
+ def __init__(
+ self,
+ csr_mat: csr_matrix,
+ zarr_loc: ZarrLocation,
+ cell_ids: np.ndarray | list[str],
+ feature_ids: np.ndarray | list[str],
+ assay_name: str | None = None,
+ workspace: str | None = None,
+ feature_names: np.ndarray | list[str] | None = None,
+ matrix_dtype: np.dtype | None = None,
+ storage_options: dict[str, Any] | None = None,
+ mem_budget: int | str | None = None,
+ nthreads: int | None = None,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+ ) -> None:
+ from ..storage.budget import resolve_budget
+ from ..storage.schema import (
+ create_cell_data,
+ create_zarr_count_assay,
+ validate_assay_name,
+ )
+ from ..storage.stores import load_zarr
+
+ self.mat = csr_mat
+ self.resources = resolve_budget(mem_budget, nthreads)
+ self.profile = resolve_storage_profile(zarr_loc, profile)
+ self.workspace = workspace
+ self.storage_options = storage_options
+ cell_ids = np.array(cell_ids)
+ if matrix_dtype is None:
+ self.matrixDtype = self.mat.dtype
+ else:
+ self.matrixDtype = matrix_dtype
+ if assay_name is None:
+ logger.debug("Using RNA as the default assay name")
+ self.assayName = "RNA"
+ else:
+ self.assayName = assay_name
+ validate_assay_name(self.assayName)
+ self.nCells, self.nFeatures = self.mat.shape
+ if len(cell_ids) != self.nCells:
+ raise ValueError(
+ "ERROR: Number of cell ids are not same as number of cells in the matrix"
+ )
+ if len(feature_ids) != self.nFeatures:
+ raise ValueError(
+ "ERROR: Number of feature ids are not same as number of features in the matrix"
+ )
+
+ self.z = load_zarr(zarr_loc, mode="w", storage_options=storage_options)
+ _ = create_cell_data(
+ root=self.z,
+ workspace=self.workspace,
+ ids=cell_ids,
+ names=cell_ids,
+ profile=self.profile,
+ )
+ if feature_names is None:
+ feature_names = feature_ids
+ create_zarr_count_assay(
+ z=self.z,
+ assay_name=self.assayName,
+ workspace=workspace,
+ n_cells=self.nCells,
+ feat_ids=feature_ids,
+ feat_names=feature_names,
+ dtype=str(self.matrixDtype),
+ profile=self.profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ )
+
+ def dump(self, batch_size: int | None = None) -> None:
+ """Write out the data matrix into the Zarr hierarchy.
+
+ Args:
+ batch_size: Number of source cells per batch. By default, a
+ destination-aligned value is selected within the memory budget.
+
+ Raises:
+ ValueError: Raised if there is any unexpected errors when writing to the Zarr hierarchy.
+ AssertionError: Catches eventual bugs in the class, if number of cells does not match after transformation.
+
+ Returns:
+ None
+ """
+ from ..storage.schema import load_count_array
+ from ..storage.sharding import (
+ resolve_sparse_import_batch,
+ sparse_matrix_bytes,
+ )
+
+ if batch_size is not None and batch_size <= 0:
+ raise ValueError("batch_size must be positive")
+ store = load_count_array(self.z, self.assayName, self.workspace)
+ resident_bytes = sparse_matrix_bytes(self.mat)
+ indptr = np.asarray(self.mat.indptr)
+
+ def max_window_nnz(window_rows: int) -> int:
+ width = min(window_rows, self.nCells)
+ if width == 0:
+ return 0
+ return int(np.max(indptr[width:] - indptr[:-width]))
+
+ plan = resolve_sparse_import_batch(
+ (store,),
+ nRows=self.nCells,
+ resources=self.resources,
+ maxWindowNnz=max_window_nnz,
+ sourceDtype=self.mat.dtype,
+ batchRows=batch_size,
+ residentBytes=resident_bytes,
+ )
+ self._lastImportPlan = plan
+ resolved_batch_rows = plan.batchRows
+ logger.debug(
+ f"Resolved sparse source batch rows={resolved_batch_rows} "
+ f"write_tasks={plan.writeTasks}"
+ )
+
+ def row_batches() -> Iterator[coo_matrix]:
+ s = 0
+ for end in range(
+ resolved_batch_rows,
+ self.nCells + resolved_batch_rows,
+ resolved_batch_rows,
+ ):
+ if s == self.nCells:
+ break
+ if end > self.nCells:
+ end = self.nCells
+ yield self.mat[s:end].tocoo()
+ s = end
+
+ e = accumulate_sparse_to_shards(
+ store,
+ row_batches(),
+ resources=self.resources,
+ residentBytes=resident_bytes,
+ producerReserveBytes=plan.producerReserveBytes,
+ msg="Writing sparse counts",
+ )
+ if e != self.nCells:
+ raise AssertionError(
+ "ERROR: This is a bug in SparseToZarr. All cells might not have been successfully "
+ "written into the zarr file. Please report this issue"
+ )
+ logger.info(
+ f"Wrote {self.nCells} cells and {self.nFeatures} features "
+ f"to assay {self.assayName}"
+ )
+
+
+def bed_to_sparse_array(
+ bed_fn: str,
+ bin_size: int,
+ chrom_sizes: dict[str, int],
+ min_counts_per_cell: int = 500,
+ read_chunk_size: float = 1e6,
+ sep: str = "\t",
+ chrom_col: int = 0,
+ start_col: int = 1,
+ end_col: int = 2,
+ barcode_col: int = 3,
+ count_col: int = 4,
+ comments_startswith: str = "#",
+ disable_tqdm: bool = False,
+ chrom_modifier: Any = None,
+) -> tuple[csr_matrix, pd.Series, pd.Series]:
+ """
+
+ Args:
+ bed_fn:
+ bin_size:
+ chrom_sizes:
+ min_counts_per_cell:
+ read_chunk_size:
+ sep:
+ chrom_col:
+ start_col:
+ end_col:
+ barcode_col:
+ count_col:
+ comments_startswith:
+ disable_tqdm:
+ chrom_modifier:
+
+ Returns:
+
+ """
+ import gc
+
+ def feat_mapper(x: str) -> int:
+ return feat_idx.get(x, n_feats)
+
+ def default_chrom_modifier(x: str) -> str:
+ return x + "_"
+
+ feat_idx: dict[str, int] = {}
+ for i in iter_progress(
+ chrom_sizes,
+ disable=disable_tqdm,
+ desc="Calculating bin indices",
+ total=len(chrom_sizes),
+ ):
+ for j in range((chrom_sizes[i] // bin_size) + 1):
+ feat_idx[f"{i}_{j}"] = len(feat_idx)
+ cell_idx: dict[Any, int] = {}
+ mat_chunks: list[np.ndarray] = []
+ n_feats = len(feat_idx)
+ if chrom_modifier is None:
+ chrom_modifier = default_chrom_modifier
+
+ stream = pd.read_csv(
+ bed_fn,
+ sep=sep,
+ header=None,
+ comment=comments_startswith,
+ usecols=[chrom_col, start_col, end_col, barcode_col, count_col],
+ chunksize=int(read_chunk_size),
+ )
+ for df in iter_progress(
+ stream, disable=disable_tqdm, desc="Building in memory sparse matrix"
+ ):
+ df[chrom_col] = df[chrom_col].map(chrom_modifier) + (
+ (df[start_col] + (df[end_col] - df[start_col]) // 2).values // bin_size
+ ).astype(str)
+ for i in df[barcode_col].unique():
+ if i not in cell_idx:
+ cell_idx[i] = len(cell_idx)
+ mat_chunks.append(
+ np.vstack(
+ [
+ np.fromiter(map(cell_idx.get, df[barcode_col].values), dtype=int),
+ np.fromiter(map(feat_mapper, df[chrom_col].values), dtype=int),
+ df[count_col].values,
+ ]
+ ).T
+ )
+ mat_arr = np.vstack(mat_chunks)
+ gc.collect()
+ mat = csr_matrix(
+ (mat_arr[:, 2], (mat_arr[:, 0], mat_arr[:, 1])),
+ shape=(len(cell_idx), n_feats + 1),
+ )
+ gc.collect()
+ idx = np.array(mat.sum(axis=1))[:, 0] > min_counts_per_cell
+ return mat[idx, :-1], pd.Series(cell_idx.keys())[idx], pd.Series(feat_idx.keys())
diff --git a/scarf/writers/subset.py b/scarf/writers/subset.py
new file mode 100644
index 00000000..833f48ae
--- /dev/null
+++ b/scarf/writers/subset.py
@@ -0,0 +1,293 @@
+import os
+from typing import Any
+
+import numpy as np
+import zarr
+
+from ..storage.types import as_zarr_array, as_zarr_group
+from ..storage.arrays import create_numeric_array, create_zarr_obj_array
+from ..storage.layout import count_array_spec
+from ..storage.profiles import (
+ StorageProfile,
+ ZarrLocation,
+ is_local_zarr_path,
+ resolve_storage_profile,
+)
+from ..storage.schema import create_zarr_count_assay
+from ..storage.sharding import write_dense_in_shard_rows
+from ..storage.stores import load_zarr
+from ..utils.logging import logger
+
+
+def subset_assay_zarr(
+ zarr_loc: ZarrLocation,
+ in_grp: str,
+ out_grp: str,
+ cells_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ storage_options: dict[str, Any] | None = None,
+ mem_budget: int | str | None = None,
+ nthreads: int | None = None,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+) -> None:
+ """Selects a subset of the data in an assay in the specified Zarr
+ hierarchy.
+
+ For the arguments `cells_idx` and `feat_idx`, refer to the documentation for numpy.split:
+ https://numpy.org/doc/stable/reference/generated/numpy.split.html
+
+ Args:
+ zarr_loc: The file name for the Zarr hierarchy.
+ in_grp: Group in Zarr hierarchy to subset.
+ out_grp: Group name in Zarr hierarchy to write subsetted assay to.
+ cells_idx: Indices of cells to keep in the subset.
+ feat_idx: Indices of features to keep in the subset.
+ Returns:
+ None
+ """
+ from ..storage.budget import resolve_budget
+
+ resources = resolve_budget(mem_budget, nthreads)
+ resolved_profile = resolve_storage_profile(zarr_loc, profile)
+ z = load_zarr(zarr_loc, "r+", storage_options=storage_options)
+ ig = as_zarr_array(z[in_grp], name=in_grp)
+ spec = count_array_spec(
+ len(cells_idx),
+ len(feat_idx),
+ dtype="uint32",
+ profile=resolved_profile,
+ targetChunkBytes=targetChunkBytes,
+ targetShardBytes=targetShardBytes,
+ )
+ og = create_numeric_array(z, out_grp, spec)
+ write_dense_in_shard_rows(
+ og,
+ lambda start, end: np.asarray(
+ ig.get_orthogonal_selection((cells_idx[start:end], feat_idx))
+ ),
+ msg="Subsetting assay",
+ resources=resources,
+ )
+ return None
+
+
+class SubsetZarr:
+ """Split Zarr file using a subset of cells.
+
+ Args:
+ zarr_loc: Path for the output (subsetted) Zarr file
+ assays: Source assays to be subsetted. These assays must be from the same dataset
+ in_workspace: Source workspace name (None for legacy layout).
+ out_workspace: Target workspace name in the output Zarr file.
+ cell_key: Name of a boolean column in cell metadata. The cells with value True are included in the
+ subset. Only used when cell_idx is None.
+ cell_idx: Explicit indices of cells to include in the subset.
+ reset_cell_filter: If True, then the cell filtering information is removed, i.e. even the filtered out cells
+ are set as True as in the 'I' column. To keep the filtering information set the value for
+ this parameter to False. (Default value: True)
+ overwrite_existing_file: If True, then overwrites the existing data. (Default value: False)
+ overwrite_cell_data: If True, then overwrites cell data (Default value: False)
+ """
+
+ def __init__(
+ self,
+ zarr_loc: ZarrLocation,
+ assays: list[Any],
+ in_workspace: str | None = None,
+ out_workspace: str | None = None,
+ cell_key: str | None = None,
+ cell_idx: np.ndarray | None = None,
+ reset_cell_filter: bool = True,
+ overwrite_existing_file: bool = False,
+ overwrite_cell_data: bool = False,
+ storage_options: dict[str, Any] | None = None,
+ mem_budget: int | str | None = None,
+ nthreads: int | None = None,
+ profile: StorageProfile | None = None,
+ targetChunkBytes: int | None = None,
+ targetShardBytes: int | None = None,
+ ) -> None:
+ from ..storage.budget import resolve_budget
+
+ self.resetCells = reset_cell_filter
+ self.overFn = overwrite_existing_file
+ self.overCells = overwrite_cell_data
+ self.inWorkspace = in_workspace
+ self.outWorkspace = out_workspace
+ self.storage_options = storage_options
+ assay_resources = [
+ assay.resources for assay in assays if hasattr(assay, "resources")
+ ]
+ self.resources = resolve_budget(
+ (
+ mem_budget
+ if mem_budget is not None
+ else (
+ min(resource.memoryBytes for resource in assay_resources)
+ if assay_resources
+ else None
+ )
+ ),
+ (
+ nthreads
+ if nthreads is not None
+ else (
+ min(resource.workers for resource in assay_resources)
+ if assay_resources
+ else None
+ )
+ ),
+ )
+ self.profile = resolve_storage_profile(zarr_loc, profile)
+ self.targetChunkBytes = targetChunkBytes
+ self.targetShardBytes = targetShardBytes
+ self.z = self._check_files(zarr_loc)
+ self.assays = self._check_assays(assays)
+ self.cellIdx = self._check_idx(cell_key, cell_idx)
+
+ def _check_files(self, zarr_loc: ZarrLocation) -> zarr.Group:
+ if (
+ is_local_zarr_path(zarr_loc)
+ and isinstance(zarr_loc, str)
+ and os.path.isdir(zarr_loc)
+ and self.overFn is False
+ ):
+ raise ValueError(
+ f"Zarr file with name: {zarr_loc} already exists.\n"
+ f"If you want to overwrite it then please set overwrite_existing_file to True. "
+ f"No subsetting was performed."
+ )
+ return load_zarr(
+ zarr_loc=zarr_loc, mode="w", storage_options=self.storage_options
+ )
+
+ @staticmethod
+ def _check_assays(assays: list[Any]) -> list[Any]:
+ # if type(assays) != list:
+ if isinstance(assays, list) is False:
+ raise TypeError(
+ "Value for parameter `assays` should be a list. For example, `[ds.RNA]`"
+ )
+ n = []
+ for assay in assays:
+ try:
+ n.append(assay.cells.N)
+ except AttributeError:
+ raise ValueError(
+ "Please make sure you are passing actual assay objects and not assay names. "
+ "For example, `[ds.RNA]`"
+ )
+ if len(set(n)) != 1:
+ raise ValueError(
+ f"ERROR: Provided assays do not have the same numer of cells. Please make " # noqa: F541
+ f"sure that the assays are from the same DataStore." # noqa: F541
+ )
+ return assays
+
+ def _check_idx(
+ self, cell_key: str | None, cell_idx: np.ndarray | None
+ ) -> np.ndarray:
+ if cell_key is None and cell_idx is None:
+ raise ValueError("Both `cell_key` and `cell_idx` parameters cannot be None")
+ if cell_idx is None:
+ resolved: np.ndarray | None = None
+ for assay in self.assays:
+ try:
+ idx = assay.cells.fetch_all(cell_key)
+ except KeyError:
+ raise ValueError(
+ f"ERROR: Provided cell_key {cell_key} was not found in the assay: {assay.name}"
+ )
+ if idx.dtype != bool:
+ raise ValueError(
+ f"ERROR: {cell_key} is not of boolean type. Cannot perform subsetting"
+ )
+ if resolved is None:
+ resolved = idx
+ elif np.all(resolved == idx) is False:
+ raise ValueError(
+ f"ERROR: Provided cell_key {cell_key} is not consistent across the assays. "
+ f"Please make sure that the assays are from the same DataStore."
+ )
+ if resolved is None:
+ raise ValueError("No assays available for cell index resolution")
+ cell_idx = np.where(resolved)[0]
+ else:
+ cell_idx = np.array(cell_idx)
+ if np.issubdtype(cell_idx.dtype, np.integer) is False:
+ raise ValueError(
+ f"ERROR: `cell_idx` must be of integer type. Provided array has a dtype: {cell_idx.dtype}"
+ )
+ if max(cell_idx) >= self.assays[0].cells.N:
+ raise ValueError(
+ f"ERROR: `cell_idx` max value is larger than the number of cells in the data." # noqa: F541
+ )
+ return cell_idx
+
+ def _prep_cell_data(self) -> None:
+ if self.outWorkspace is None:
+ cell_slot = "cellData"
+ else:
+ cell_slot = f"{self.outWorkspace}/cellData"
+ if cell_slot in self.z:
+ cell_group = as_zarr_group(self.z[cell_slot], name=cell_slot)
+ else:
+ cell_group = self.z.create_group(cell_slot)
+
+ cell_data = self.assays[0].cells.locations["primary"]
+
+ n_cells = len(self.cellIdx)
+ for i in cell_data.keys():
+ if i in cell_group and self.overCells is False:
+ continue
+ if i in ["I"] and self.resetCells:
+ create_zarr_obj_array(
+ cell_group, "I", [True for _ in range(n_cells)], "bool"
+ )
+ continue
+ v = cell_data[i][:][self.cellIdx]
+ create_zarr_obj_array(cell_group, i, v, dtype=v.dtype)
+
+ def _prep_counts(self) -> None:
+ n_cells = len(self.cellIdx)
+ for assay in self.assays:
+ create_zarr_count_assay(
+ z=self.z,
+ assay_name=assay.name,
+ workspace=self.outWorkspace,
+ n_cells=n_cells,
+ feat_ids=assay.feats.fetch_all("ids"),
+ feat_names=assay.feats.fetch_all("names"),
+ dtype=assay.rawData.dtype,
+ profile=self.profile,
+ targetChunkBytes=self.targetChunkBytes,
+ targetShardBytes=self.targetShardBytes,
+ )
+
+ def dump(self) -> None:
+ self._prep_cell_data()
+ self._prep_counts()
+ for assay in self.assays:
+ raw_data = assay.rawData[self.cellIdx]
+ if self.outWorkspace is None:
+ store = as_zarr_array(
+ self.z[f"{assay.name}/counts"],
+ name=f"{assay.name}/counts",
+ )
+ else:
+ store = as_zarr_array(
+ self.z[f"matrices/{assay.name}/counts"],
+ name=f"matrices/{assay.name}/counts",
+ )
+ write_dense_in_shard_rows(
+ store,
+ lambda start, end: raw_data[start:end, :].compute(),
+ msg=f"Subsetting assay: {assay.name}",
+ resources=self.resources,
+ )
+ logger.info(
+ f"Wrote a subset of {len(self.cellIdx)} cells across "
+ f"{len(self.assays)} assay(s)"
+ )
diff --git a/scripts/export_seurat_hto_golden.R b/scripts/export_seurat_hto_golden.R
new file mode 100644
index 00000000..9684f5cf
--- /dev/null
+++ b/scripts/export_seurat_hto_golden.R
@@ -0,0 +1,94 @@
+#!/usr/bin/env Rscript
+# Produces the Seurat side of tests/seurat_hto_5_5_1_golden.json.
+# Driven by scripts/export_seurat_hto_golden.py, which supplies a deterministic
+# subset of raw GSE245108 HTO counts and records the returned cell identities.
+
+suppressPackageStartupMessages({
+ library(Matrix)
+ library(Seurat)
+ library(SeuratObject)
+})
+
+args <- commandArgs(trailingOnly = TRUE)
+if (length(args) != 2L) {
+ stop("usage: export_seurat_hto_golden.R ")
+}
+work <- args[[1]]
+random_seed <- as.integer(args[[2]])
+
+counts_by_cell <- read.delim(
+ file.path(work, "hto_counts.tsv"),
+ row.names = 1,
+ check.names = FALSE
+)
+hto <- t(as.matrix(counts_by_cell))
+storage.mode(hto) <- "numeric"
+
+# Seurat replaces underscores in feature names. Temporary safe names preserve
+# an unambiguous mapping back to the names stored in the source matrix.
+original_names <- rownames(hto)
+safe_names <- sprintf("HTO%02d", seq_along(original_names))
+rownames(hto) <- safe_names
+
+object <- CreateSeuratObject(
+ counts = Matrix(hto, sparse = TRUE),
+ assay = "HTO",
+ project = "GSE245108"
+)
+object <- NormalizeData(
+ object,
+ assay = "HTO",
+ normalization.method = "CLR",
+ margin = 1,
+ verbose = FALSE
+)
+object <- HTODemux(
+ object,
+ assay = "HTO",
+ positive.quantile = 0.99,
+ init = length(safe_names) + 1,
+ nstarts = 100,
+ kfunc = "kmeans",
+ seed = random_seed,
+ verbose = FALSE
+)
+
+labels <- as.character(object$hash.ID)
+matched <- match(labels, safe_names)
+labels[!is.na(matched)] <- original_names[matched[!is.na(matched)]]
+global_column <- "HTO_classification.global"
+if (!(global_column %in% colnames(object[[]]))) {
+ stop("Seurat did not produce HTO_classification.global")
+}
+
+calls <- data.frame(
+ barcode = colnames(object),
+ hashId = labels,
+ classificationGlobal = as.character(object[[global_column, drop = TRUE]]),
+ stringsAsFactors = FALSE
+)
+write.table(
+ calls,
+ file.path(work, "seurat_calls.tsv"),
+ sep = "\t",
+ quote = FALSE,
+ row.names = FALSE
+)
+
+versions <- data.frame(
+ key = c("r", "seurat", "seuratObject", "fitdistrplus"),
+ value = c(
+ R.version$version.string,
+ as.character(packageVersion("Seurat")),
+ as.character(packageVersion("SeuratObject")),
+ as.character(packageVersion("fitdistrplus"))
+ )
+)
+write.table(
+ versions,
+ file.path(work, "versions.tsv"),
+ sep = "\t",
+ quote = FALSE,
+ row.names = FALSE,
+ col.names = FALSE
+)
diff --git a/scripts/export_seurat_hto_golden.py b/scripts/export_seurat_hto_golden.py
new file mode 100644
index 00000000..a4d874ec
--- /dev/null
+++ b/scripts/export_seurat_hto_golden.py
@@ -0,0 +1,180 @@
+#!/usr/bin/env python
+"""Regenerate tests/seurat_hto_5_5_1_golden.json.
+
+Reads the GSE245108 TNC-1-2-1 filtered 10x H5 matrix, takes a deterministic
+cell subset, and hands the raw HTO counts to scripts/export_seurat_hto_golden.R.
+The Seurat calls and the exact input counts are written to a compact fixture, so
+normal CI does not require R, Seurat, network access, or the full GEO matrix.
+
+Download the source matrix from GEO, then provide an R installation carrying
+Seurat 5.5.1. For example:
+
+ uv run python scripts/export_seurat_hto_golden.py \
+ --h5 /path/to/GSE245108_TNC-1-2-1_filtered_feature_bc_matrix.h5 \
+ --rscript /path/to/seurat-environment/bin/Rscript
+"""
+
+import argparse
+import hashlib
+import json
+import subprocess
+import tempfile
+from pathlib import Path
+
+import h5py
+import numpy as np
+import pandas as pd
+from scipy.sparse import csc_matrix
+
+REPO = Path(__file__).resolve().parent.parent
+DEFAULT_FIXTURE = REPO / "tests/seurat_hto_5_5_1_golden.json"
+R_SCRIPT = Path(__file__).resolve().parent / "export_seurat_hto_golden.R"
+DATASET_URL = (
+ "https://ftp.ncbi.nlm.nih.gov/geo/series/GSE245nnn/GSE245108/suppl/"
+ "GSE245108_TNC-1-2-1_filtered_feature_bc_matrix.h5"
+)
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--h5", type=Path, required=True)
+ parser.add_argument("--rscript", type=Path, default=Path("Rscript"))
+ parser.add_argument("--output", type=Path, default=DEFAULT_FIXTURE)
+ parser.add_argument("--cells", type=int, default=1000)
+ parser.add_argument("--sample-seed", type=int, default=20260731)
+ parser.add_argument("--random-seed", type=int, default=0)
+ return parser.parse_args()
+
+
+def sha256_file(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ while chunk := handle.read(1024 * 1024):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def input_fingerprint(inputs: dict[str, object]) -> str:
+ encoded = json.dumps(
+ inputs,
+ separators=(",", ":"),
+ sort_keys=True,
+ ).encode()
+ return hashlib.sha256(encoded).hexdigest()
+
+
+def read_hto_counts(path: Path) -> pd.DataFrame:
+ with h5py.File(path) as handle:
+ matrix = handle["matrix"]
+ sparse = csc_matrix(
+ (
+ matrix["data"][:],
+ matrix["indices"][:],
+ matrix["indptr"][:],
+ ),
+ shape=tuple(int(value) for value in matrix["shape"][:]),
+ )
+ names = np.asarray(matrix["features/name"]).astype(str)
+ feature_types = np.asarray(matrix["features/feature_type"]).astype(str)
+ barcodes = np.asarray(matrix["barcodes"]).astype(str)
+
+ selected = (feature_types == "Antibody Capture") & np.char.startswith(names, "TNC_")
+ if selected.sum() != 10:
+ raise ValueError(
+ "Expected exactly 10 TNC_ Antibody Capture features; "
+ f"found {int(selected.sum())}"
+ )
+ return pd.DataFrame(
+ sparse[np.flatnonzero(selected)].T.toarray(),
+ index=barcodes,
+ columns=names[selected],
+ )
+
+
+def main() -> None:
+ args = parse_args()
+ if not args.h5.is_file():
+ raise FileNotFoundError(args.h5)
+
+ counts = read_hto_counts(args.h5)
+ if args.cells > len(counts):
+ raise ValueError(
+ f"Matrix holds {len(counts)} cells, cannot sample {args.cells}"
+ )
+ generator = np.random.default_rng(args.sample_seed)
+ selected = np.sort(generator.choice(len(counts), size=args.cells, replace=False))
+ subset = counts.iloc[selected].astype(np.int64)
+
+ with tempfile.TemporaryDirectory(prefix="scarf-hto-golden-") as raw_work:
+ work = Path(raw_work)
+ subset.to_csv(work / "hto_counts.tsv", sep="\t", index_label="barcode")
+ subprocess.run(
+ [
+ str(args.rscript),
+ str(R_SCRIPT),
+ str(work),
+ str(args.random_seed),
+ ],
+ check=True,
+ )
+ calls = pd.read_csv(work / "seurat_calls.tsv", sep="\t")
+ versions = dict(
+ line.split("\t")
+ for line in (work / "versions.tsv").read_text().splitlines()
+ )
+
+ barcodes = subset.index.tolist()
+ if calls["barcode"].tolist() != barcodes:
+ raise RuntimeError("Seurat returned calls in an unexpected cell order")
+
+ input_values: dict[str, object] = {
+ "barcodes": barcodes,
+ "htoNames": subset.columns.tolist(),
+ "rawCounts": subset.to_numpy().tolist(),
+ }
+ fixture = {
+ "provenance": {
+ "package": "Seurat",
+ "packageVersion": versions["seurat"],
+ "seuratObjectVersion": versions["seuratObject"],
+ "fitdistrplusVersion": versions["fitdistrplus"],
+ "rVersion": versions["r"],
+ "sourceRepository": "https://github.com/satijalab/seurat",
+ "dataset": "GSE245108",
+ "datasetSource": DATASET_URL,
+ "sourceFile": args.h5.name,
+ "sourceSha256": sha256_file(args.h5),
+ "generator": "scripts/export_seurat_hto_golden.py",
+ "nCells": int(args.cells),
+ "nHtos": int(subset.shape[1]),
+ "cellSampleSeed": int(args.sample_seed),
+ "randomSeed": int(args.random_seed),
+ "normalization": {"method": "CLR", "margin": 1},
+ "clustering": {
+ "method": "kmeans",
+ "clusterCount": int(subset.shape[1] + 1),
+ "nStarts": 100,
+ },
+ "cutoff": {
+ "distribution": "negative_binomial",
+ "positiveQuantile": 0.99,
+ "comparison": "strictly_greater",
+ },
+ "referenceFunction": "Seurat::HTODemux",
+ },
+ "inputs": {
+ **input_values,
+ "sha256": input_fingerprint(input_values),
+ },
+ "seurat": {
+ "hashId": calls["hashId"].tolist(),
+ "classificationGlobal": calls["classificationGlobal"].tolist(),
+ },
+ }
+ args.output.parent.mkdir(parents=True, exist_ok=True)
+ args.output.write_text(json.dumps(fixture, indent=1) + "\n")
+ print(f"wrote {args.output} ({args.output.stat().st_size / 1024:.0f} KB)")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/export_seurat_wnn_golden.R b/scripts/export_seurat_wnn_golden.R
new file mode 100644
index 00000000..053ed796
--- /dev/null
+++ b/scripts/export_seurat_wnn_golden.R
@@ -0,0 +1,195 @@
+#!/usr/bin/env Rscript
+# Produces the Seurat side of tests/seurat_wnn_5_5_1_golden.json.
+# Driven by scripts/export_seurat_wnn_golden.py, which supplies the shared
+# embeddings and neighbour indices and consumes the tables written here.
+#
+# The matched tables come from Seurat's own routines applied to Scarf's
+# candidate pool and bandwidth: L2Norm, PredictAssay, impute_dist and NNdist do
+# the arithmetic, and the kernel, score and softmax expressions are copied from
+# the body of FindModalityWeights. The default tables come from the public
+# FindMultiModalNeighbors with its shipped defaults, which Scarf does not try to
+# reproduce.
+
+suppressPackageStartupMessages({
+ library(Seurat)
+ library(SeuratObject)
+})
+
+args <- commandArgs(trailingOnly = TRUE)
+if (length(args) != 1L) {
+ stop("usage: export_seurat_wnn_golden.R ")
+}
+work <- args[[1]]
+
+read_matrix <- function(name) {
+ as.matrix(read.table(file.path(work, name), header = FALSE, colClasses = "numeric"))
+}
+
+write_matrix <- function(values, name) {
+ values <- as.matrix(values)
+ lines <- apply(values, 1, function(row) paste(sprintf("%.17g", row), collapse = "\t"))
+ writeLines(lines, file.path(work, name))
+}
+
+rna <- read_matrix("rna_embedding.tsv")
+adt <- read_matrix("adt_embedding.tsv")
+# Python writes zero-based, self-free neighbour rows.
+rna_idx <- read_matrix("rna_indices.tsv") + 1
+adt_idx <- read_matrix("adt_indices.tsv") + 1
+
+n_cells <- nrow(rna)
+k <- ncol(rna_idx)
+cell_names <- paste0("cell", seq_len(n_cells))
+rownames(rna) <- rownames(adt) <- cell_names
+colnames(rna) <- paste0("PC_", seq_len(ncol(rna)))
+colnames(adt) <- paste0("APC_", seq_len(ncol(adt)))
+
+counts <- matrix(
+ 1,
+ nrow = 5,
+ ncol = n_cells,
+ dimnames = list(paste0("gene", 1:5), cell_names)
+)
+object <- CreateSeuratObject(counts = counts, assay = "RNA")
+object[["pca"]] <- CreateDimReducObject(embeddings = rna, key = "PC_", assay = "RNA")
+object[["apca"]] <- CreateDimReducObject(embeddings = adt, key = "APC_", assay = "RNA")
+
+reductions <- c("pca", "apca")
+dims <- list(pca = seq_len(ncol(rna)), apca = seq_len(ncol(adt)))
+indices <- list(pca = rna_idx, apca = adt_idx)
+
+# Seurat's own row normalization, matching l2_normalize=True on the Scarf side.
+normed <- lapply(reductions, function(r) Seurat:::L2Norm(mat = Embeddings(object, r)[, dims[[r]]]))
+names(normed) <- reductions
+for (r in reductions) {
+ object[[paste0(r, ".norm")]] <- CreateDimReducObject(
+ embeddings = normed[[r]],
+ key = paste0("norm", Key(object[[r]])),
+ assay = DefaultAssay(object[[r]])
+ )
+}
+
+# nearest.dist is the closest non-self neighbour and sigma is the k-th non-self
+# neighbour minus that, which is the bandwidth Scarf uses. Seurat reaches the
+# same two quantities as Distances(nn)[, 2] and Distances(nn)[, sigma.idx] on a
+# neighbour object whose first column is the cell itself.
+own_distances <- lapply(reductions, function(r) {
+ embedding <- normed[[r]]
+ t(sapply(seq_len(n_cells), function(i) {
+ sqrt(rowSums((embedding[indices[[r]][i, ], , drop = FALSE] - rep(embedding[i, ], each = k))^2))
+ }))
+})
+names(own_distances) <- reductions
+nearest_dist <- lapply(own_distances, function(d) apply(d, 1, min))
+sigma <- lapply(reductions, function(r) {
+ apply(own_distances[[r]], 1, function(row) sort(row)[k]) - nearest_dist[[r]]
+})
+names(sigma) <- reductions
+
+within_impute <- lapply(reductions, function(r) {
+ PredictAssay(
+ object = object,
+ nn.idx = indices[[r]],
+ assay = DefaultAssay(object[[r]]),
+ reduction = paste0(r, ".norm"),
+ dims = dims[[r]],
+ return.assay = FALSE,
+ verbose = FALSE
+ )
+})
+names(within_impute) <- reductions
+cross_impute <- lapply(reductions, function(r) {
+ other <- setdiff(reductions, r)
+ PredictAssay(
+ object = object,
+ nn.idx = indices[[other]],
+ assay = DefaultAssay(object[[r]]),
+ reduction = paste0(r, ".norm"),
+ dims = dims[[r]],
+ return.assay = FALSE,
+ verbose = FALSE
+ )
+})
+names(cross_impute) <- reductions
+
+within_dist <- lapply(reductions, function(r) {
+ Seurat:::impute_dist(
+ x = normed[[r]],
+ y = t(within_impute[[r]]),
+ nearest.dist = nearest_dist[[r]]
+ )
+})
+cross_dist <- lapply(reductions, function(r) {
+ Seurat:::impute_dist(
+ x = normed[[r]],
+ y = t(cross_impute[[r]]),
+ nearest.dist = nearest_dist[[r]]
+ )
+})
+names(within_dist) <- names(cross_dist) <- reductions
+
+# The next three expressions are copied from the body of FindModalityWeights.
+within_kernel <- lapply(reductions, function(r) exp(-1 * (within_dist[[r]] / sigma[[r]])))
+cross_kernel <- lapply(reductions, function(r) exp(-1 * (cross_dist[[r]] / sigma[[r]])))
+names(within_kernel) <- names(cross_kernel) <- reductions
+score <- lapply(reductions, function(r) {
+ MinMax(data = within_kernel[[r]] / (cross_kernel[[r]] + 1e-04), min = 0, max = 200)
+})
+names(score) <- reductions
+score_total <- rowSums(exp(Reduce(cbind, score)))
+weights <- lapply(reductions, function(r) exp(score[[r]]) / score_total)
+names(weights) <- reductions
+
+# Blend the two kernels over the union pool and keep the k strongest, breaking
+# ties by ascending cell index the way Scarf's lexsort does.
+matched_indices <- matrix(0L, nrow = n_cells, ncol = k)
+matched_affinity <- matrix(0, nrow = n_cells, ncol = k)
+for (i in seq_len(n_cells)) {
+ pool <- sort(union(indices$pca[i, ], indices$apca[i, ]))
+ affinity <- rep(0, length(pool))
+ for (r in reductions) {
+ adjusted <- Seurat:::NNdist(
+ nn.idx = matrix(pool, nrow = 1),
+ embeddings = normed[[r]],
+ query.embeddings = normed[[r]][i, , drop = FALSE],
+ nearest.dist = nearest_dist[[r]][i]
+ )[[1]]
+ affinity <- affinity + weights[[r]][i] * exp(-1 * (adjusted / sigma[[r]][i]))
+ }
+ chosen <- order(-affinity, pool)[seq_len(k)]
+ matched_indices[i, ] <- pool[chosen] - 1L
+ matched_affinity[i, ] <- affinity[chosen]
+}
+
+write_matrix(matched_indices, "matched_indices.tsv")
+write_matrix(matched_affinity, "matched_affinities.tsv")
+write_matrix(cbind(weights$pca, weights$apca), "matched_modality_weights.tsv")
+
+# Public defaults. Scarf deliberately differs here, so this table only supports
+# a drift check on how far apart the two stay.
+default_object <- FindMultiModalNeighbors(
+ object = object,
+ reduction.list = list("pca", "apca"),
+ dims.list = list(dims$pca, dims$apca),
+ k.nn = k,
+ modality.weight.name = c("pca.weight", "apca.weight"),
+ verbose = FALSE
+)
+default_nn <- default_object[["weighted.nn"]]
+default_indices <- Indices(default_nn)
+write_matrix(default_indices - 1L, "default_indices.tsv")
+write_matrix(
+ cbind(default_object$pca.weight, default_object$apca.weight),
+ "default_modality_weights.tsv"
+)
+
+writeLines(
+ c(
+ paste0("seurat\t", as.character(packageVersion("Seurat"))),
+ paste0("seuratObject\t", as.character(packageVersion("SeuratObject"))),
+ paste0("r\t", paste(R.version$major, R.version$minor, sep = ".")),
+ paste0("kNn\t", k),
+ paste0("nCells\t", n_cells)
+ ),
+ file.path(work, "versions.tsv")
+)
diff --git a/scripts/export_seurat_wnn_golden.py b/scripts/export_seurat_wnn_golden.py
new file mode 100644
index 00000000..9bd4212f
--- /dev/null
+++ b/scripts/export_seurat_wnn_golden.py
@@ -0,0 +1,167 @@
+#!/usr/bin/env python
+"""Regenerate tests/seurat_wnn_5_5_1_golden.json.
+
+Pulls RNA and ADT reductions from the public tenx_8K_pbmc_citeseq store, takes a
+deterministic cell subset, computes exact self-free neighbour rows, and hands
+both to scripts/export_seurat_wnn_golden.R. The Seurat tables come back as
+tab-separated text and are written out as the golden fixture.
+
+Needs an R installation carrying Seurat. Point --rscript at it, for example a
+conda environment created with `conda create -p /tmp/scarf-seurat-env
+-c conda-forge r-base r-seurat`.
+"""
+
+import argparse
+import json
+import subprocess
+import tempfile
+from pathlib import Path
+
+import numpy as np
+
+from scarf.storage.types import as_zarr_array
+
+REPO = Path(__file__).resolve().parent.parent
+DEFAULT_STORE = (
+ REPO / "docs/source/tutorials/scarf_datasets/tenx_8K_pbmc_citeseq/data.zarr"
+)
+DEFAULT_FIXTURE = REPO / "tests/seurat_wnn_5_5_1_golden.json"
+R_SCRIPT = Path(__file__).resolve().parent / "export_seurat_wnn_golden.R"
+
+
+def parse_args() -> argparse.Namespace:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--store", type=Path, default=DEFAULT_STORE)
+ parser.add_argument("--rscript", type=Path, default=Path("Rscript"))
+ parser.add_argument("--output", type=Path, default=DEFAULT_FIXTURE)
+ parser.add_argument("--cells", type=int, default=300)
+ parser.add_argument("--k", type=int, default=20)
+ parser.add_argument("--seed", type=int, default=20210601)
+ return parser.parse_args()
+
+
+def load_embeddings(store: Path, assays: tuple[str, str]) -> dict[str, np.ndarray]:
+ import scarf
+
+ scarf.configure_output(level="ERROR", progress=False)
+ datastore = scarf.DataStore(str(store), default_assay=assays[0], nthreads=4)
+ embeddings = {}
+ for assay in assays:
+ state = datastore.get_assay_state(assay)
+ if state is None or state.reduction is None:
+ raise RuntimeError(f"Assay {assay!r} has no reduction artifact")
+ reference = state.batch_correction or state.reduction
+ group = datastore.load_artifact(reference)
+ embeddings[assay] = np.asarray(as_zarr_array(group["data"])[:])
+ return embeddings
+
+
+def exact_neighbors(embedding: np.ndarray, k: int) -> np.ndarray:
+ """Self-free neighbour rows ordered by ascending float64 distance."""
+ gram = embedding @ embedding.T
+ square = np.diag(gram)
+ distances = np.sqrt(np.maximum(square[:, None] - 2.0 * gram + square[None, :], 0.0))
+ np.fill_diagonal(distances, np.inf)
+ return np.argsort(distances, axis=1, kind="stable")[:, :k].astype(np.uint32)
+
+
+def write_table(path: Path, values: np.ndarray) -> None:
+ fmt = "%d" if np.issubdtype(values.dtype, np.integer) else "%.17g"
+ np.savetxt(path, np.atleast_2d(values), fmt=fmt, delimiter="\t")
+
+
+def read_table(path: Path) -> np.ndarray:
+ return np.atleast_2d(np.loadtxt(path, dtype=np.float64, delimiter="\t"))
+
+
+def main() -> None:
+ args = parse_args()
+ embeddings = load_embeddings(args.store, ("RNA", "ADT"))
+ generator = np.random.default_rng(args.seed)
+ total = embeddings["RNA"].shape[0]
+ if args.cells > total:
+ raise ValueError(f"Store holds {total} cells, cannot sample {args.cells}")
+ selected = np.sort(generator.choice(total, size=args.cells, replace=False))
+
+ # float32 storage values widened to float64 so R and Python start from
+ # identical numbers and recompute identical distances.
+ rna = np.asarray(embeddings["RNA"][selected], dtype=np.float64)
+ adt = np.asarray(embeddings["ADT"][selected], dtype=np.float64)
+ rna_indices = exact_neighbors(rna, args.k)
+ adt_indices = exact_neighbors(adt, args.k)
+
+ with tempfile.TemporaryDirectory(prefix="scarf-wnn-golden-") as raw_work:
+ work = Path(raw_work)
+ write_table(work / "rna_embedding.tsv", rna)
+ write_table(work / "adt_embedding.tsv", adt)
+ write_table(work / "rna_indices.tsv", rna_indices)
+ write_table(work / "adt_indices.tsv", adt_indices)
+ subprocess.run(
+ [str(args.rscript), str(R_SCRIPT), str(work)],
+ check=True,
+ )
+ versions = dict(
+ line.split("\t")
+ for line in (work / "versions.tsv").read_text().splitlines()
+ )
+ matched_indices = read_table(work / "matched_indices.tsv")
+ matched_affinities = read_table(work / "matched_affinities.tsv")
+ matched_weights = read_table(work / "matched_modality_weights.tsv")
+ default_indices = read_table(work / "default_indices.tsv")
+ default_weights = read_table(work / "default_modality_weights.tsv")
+
+ fixture = {
+ "provenance": {
+ "package": "Seurat",
+ "packageVersion": versions["seurat"],
+ "seuratObjectVersion": versions["seuratObject"],
+ "rVersion": versions["r"],
+ "sourceRepository": "https://github.com/satijalab/seurat",
+ "dataset": "tenx_8K_pbmc_citeseq",
+ "datasetSource": "scarf.cytebase.connect('scarf_docs')",
+ "generator": "scripts/export_seurat_wnn_golden.py",
+ "nCells": int(args.cells),
+ "kNn": int(args.k),
+ "cellSampleSeed": int(args.seed),
+ "l2Normalize": True,
+ "matchedFunctions": [
+ "Seurat:::L2Norm",
+ "Seurat:::PredictAssay",
+ "Seurat:::impute_dist",
+ "Seurat:::NNdist",
+ "Seurat::MinMax",
+ ],
+ "matchedNote": (
+ "Seurat routines evaluated on Scarf's union candidate pool and "
+ "k-th non-self bandwidth. The kernel, score and softmax "
+ "expressions are copied from the body of FindModalityWeights."
+ ),
+ "defaultFunction": "Seurat::FindMultiModalNeighbors",
+ "defaultNote": (
+ "Shipped defaults, including the wider knn.range search and the "
+ "SNN-far bandwidth. Scarf does not reproduce these, so the "
+ "comparison is statistical."
+ ),
+ },
+ "inputs": {
+ "rnaEmbedding": rna.tolist(),
+ "adtEmbedding": adt.tolist(),
+ "rnaIndices": rna_indices.astype(int).tolist(),
+ "adtIndices": adt_indices.astype(int).tolist(),
+ },
+ "matched": {
+ "modalityWeights": matched_weights.tolist(),
+ "neighborIndices": matched_indices.astype(int).tolist(),
+ "neighborAffinities": matched_affinities.tolist(),
+ },
+ "seuratDefault": {
+ "modalityWeights": default_weights.tolist(),
+ "neighborIndices": default_indices.astype(int).tolist(),
+ },
+ }
+ args.output.write_text(json.dumps(fixture, indent=1) + "\n")
+ print(f"wrote {args.output} ({args.output.stat().st_size / 1024:.0f} KB)")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/generate_plotting_showcase.py b/scripts/generate_plotting_showcase.py
new file mode 100644
index 00000000..3cedb352
--- /dev/null
+++ b/scripts/generate_plotting_showcase.py
@@ -0,0 +1,571 @@
+"""Generate publication plotting acceptance figures from the 1K PBMC fixture."""
+
+import argparse
+from collections.abc import Hashable, Sequence
+from pathlib import Path
+import shutil
+import tarfile
+import tempfile
+
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+
+from scarf import DataStore
+import scarf.plotting as splt
+
+
+_FIXTURE = Path("tests/datasets/1K_pbmc_citeseq.zarr.tar.gz")
+_LAYOUT_FIXTURE = Path("tests/visual/showcase/plotting_showcase_layout.npz")
+_MARKER_SETS = {
+ "T cells": ("CD3D", "IL7R", "LTB"),
+ "B cells": ("MS4A1", "CD79A", "CD37"),
+ "Myeloid": ("LST1", "FCER1G", "CTSS"),
+ "Cytotoxic": ("NKG7", "GNLY", "GZMB"),
+}
+_CLUSTER_KEY = "RNA_leiden_cluster"
+_CELL_CYCLE_KEY = "RNA_cell_cycle_phase"
+
+
+def _zarr_root(directory: Path) -> Path:
+ if (directory / "zarr.json").exists() or (directory / ".zgroup").exists():
+ return directory
+ candidates = [
+ path
+ for path in directory.iterdir()
+ if path.is_dir()
+ and ((path / "zarr.json").exists() or (path / ".zgroup").exists())
+ ]
+ if len(candidates) != 1:
+ raise ValueError(f"Could not identify one Zarr store under {directory}")
+ return candidates[0]
+
+
+def _copy_fixture(source: Path, destination: Path) -> Path:
+ if source.is_dir():
+ target = destination / "showcase.zarr"
+ shutil.copytree(source, target)
+ return target
+ if not source.exists():
+ raise FileNotFoundError(
+ f"Fixture not found at {source}. Download the existing test fixtures first."
+ )
+ with tarfile.open(source, "r:gz") as archive:
+ archive.extractall(destination, filter="data")
+ return _zarr_root(destination)
+
+
+def _build_graph(store: DataStore) -> None:
+ normalized = store.run_normalization(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ update_state=False,
+ )
+ reduction = store.run_pca(
+ normalized,
+ dims=11,
+ pca_cell_key="I",
+ update_state=False,
+ local_cache=False,
+ )
+ ann_index = store.build_ann_index(
+ reduction,
+ ann_efc=50,
+ ann_ef=50,
+ ann_m=48,
+ rand_state=4466,
+ update_state=False,
+ )
+ store.build_embedding_initialization(
+ reduction,
+ n_centroids=1000,
+ rand_state=4466,
+ )
+ neighbors = store.query_neighbors(
+ ann_index,
+ coordinates=reduction,
+ k=11,
+ update_state=False,
+ )
+ store.build_connectivity_map(neighbors)
+
+
+def _apply_fixed_layout(store: DataStore, source: Path) -> None:
+ with np.load(source, allow_pickle=False) as fixture:
+ cell_ids = fixture["cellIds"].astype(str)
+ umap = fixture["umap"]
+
+ fixture_index = pd.Index(cell_ids)
+ current_ids = np.asarray(store.cells.fetch("ids")).astype(str)
+ positions = fixture_index.get_indexer(current_ids)
+ if (
+ not fixture_index.is_unique
+ or len(fixture_index) != len(current_ids)
+ or np.any(positions < 0)
+ ):
+ raise ValueError(
+ f"Layout fixture at {source} does not match the selected datastore cells"
+ )
+ if umap.shape != (len(fixture_index), 2):
+ raise ValueError(f"Layout fixture at {source} must contain two UMAP columns")
+
+ store.cells.insert(
+ "RNA_UMAP1",
+ umap[positions, 0],
+ key="I",
+ overwrite=True,
+ )
+ store.cells.insert(
+ "RNA_UMAP2",
+ umap[positions, 1],
+ key="I",
+ overwrite=True,
+ )
+
+
+def _prepare_store(
+ source: Path,
+ work_directory: Path,
+ layout_fixture: Path = _LAYOUT_FIXTURE,
+) -> DataStore:
+ store = DataStore(str(_copy_fixture(source, work_directory)), default_assay="RNA")
+ store.auto_filter_cells(show_qc_plots=False)
+ if "hvgs" not in store._get_assay("RNA").feats.columns:
+ store.mark_hvgs(
+ top_n=100,
+ show_plot=False,
+ bin_strategy="fixed",
+ min_cells=int(0.01 * store.cells.N),
+ max_cells=np.inf,
+ blacklist="^MT-|^RPS|^RPL|^MRPS|^MRPL|^CCN|^HLA-|^H2-|^HIST",
+ )
+ try:
+ store.get_latest_graph_loc(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ )
+ except KeyError:
+ _build_graph(store)
+ if _CLUSTER_KEY not in store.cells.columns:
+ store.run_leiden_clustering()
+ _apply_fixed_layout(store, layout_fixture)
+ return store
+
+
+def _marker_sets(store: DataStore) -> dict[str, list[str]]:
+ features = store._get_assay("RNA").feats
+ available = {str(value) for value in features.fetch_all("names")}
+ selected = {
+ group: [gene for gene in genes if gene in available]
+ for group, genes in _MARKER_SETS.items()
+ }
+ selected = {group: genes for group, genes in selected.items() if genes}
+ if selected:
+ return selected
+ fallback = [str(value) for value in features.fetch_all("names")[:8]]
+ midpoint = max(1, len(fallback) // 2)
+ return {"Set 1": fallback[:midpoint], "Set 2": fallback[midpoint:]}
+
+
+def _first_continuous_feature(store: DataStore, markers: dict[str, list[str]]) -> str:
+ for genes in markers.values():
+ if genes:
+ return genes[0]
+ return "RNA_nCounts"
+
+
+def _save(result: splt.PlotResult, path: Path) -> Path:
+ output = result.save(path, dpi=180, exact_size=True)
+ result.close()
+ return output
+
+
+def _normalize_svg(path: Path) -> Path:
+ lines = path.read_text(encoding="utf-8").splitlines()
+ path.write_text(
+ "\n".join(line.rstrip() for line in lines) + "\n",
+ encoding="utf-8",
+ )
+ return path
+
+
+def _relative_cycling_share_per_cluster(store: DataStore) -> dict[object, str]:
+ """Group clusters by their relative share of cells outside G1."""
+ clusters = pd.Series(np.asarray(store.cells.fetch(_CLUSTER_KEY)))
+ phases = pd.Series(np.asarray(store.cells.fetch(_CELL_CYCLE_KEY)).astype(str))
+ share = (phases != "G1").groupby(clusters).mean()
+ n_groups = min(3, len(share))
+ labels = ("low", "medium", "high")[:n_groups]
+ bins = pd.qcut(
+ share.rank(method="first"),
+ q=n_groups,
+ labels=labels,
+ )
+ return {label: str(value) for label, value in bins.items()}
+
+
+def _expression_matrix_figures(
+ store: DataStore,
+ output_directory: Path,
+ markers: dict[str, list[str]],
+) -> list[Path]:
+ outputs: list[Path] = []
+ store.run_marker_search(group_key=_CLUSTER_KEY)
+ cycling_share = _relative_cycling_share_per_cluster(store)
+ cycling_scale = splt.CategoricalScale(order=("low", "medium", "high"))
+ heatmap = splt.marker_heatmap(
+ store,
+ group_key=_CLUSTER_KEY,
+ topn=4,
+ figsize=(6.0, 7.5),
+ fontsize=8,
+ cluster_columns=True,
+ color_scale=splt.ColorScale(
+ cmap="RdBu_r",
+ vmin=-1.0,
+ vcenter=0.0,
+ vmax=2.0,
+ ),
+ column_annotations={"relative cycling share": cycling_share},
+ annotation_scales={"relative cycling share": cycling_scale},
+ theme="paper",
+ show=False,
+ )
+ heatmap.axes["heatmap"].set_xlabel("Leiden cluster")
+ outputs.append(_save(heatmap, output_directory / "marker_heatmap.png"))
+
+ matrix = splt.matrixplot(
+ store,
+ features=markers,
+ group_by=_CLUSTER_KEY,
+ normalization=splt.NormalizationSpec(transform="log1p"),
+ standardize="feature",
+ cluster_groups=True,
+ color_scale=splt.ColorScale(cmap="RdBu_r", vcenter=0.0),
+ column_annotations={
+ "relative cycling share": {
+ str(label): value for label, value in cycling_share.items()
+ }
+ },
+ annotation_scales={"relative cycling share": cycling_scale},
+ figsize=(6.0, 5.0),
+ theme="paper",
+ show=False,
+ )
+ outputs.append(_save(matrix, output_directory / "matrix_plot.png"))
+ return outputs
+
+
+def _additional_real_data_figures(
+ store: DataStore,
+ output_directory: Path,
+) -> list[Path]:
+ outputs: list[Path] = []
+ boxes = splt.distribution(
+ store,
+ keys=("RNA_S_score", "RNA_G2M_score"),
+ group_by=_CELL_CYCLE_KEY,
+ kind="box",
+ share_y=True,
+ figsize=(7.2, 3.8),
+ theme="paper",
+ show=False,
+ )
+ for axis, title in zip(
+ boxes.axes.values(),
+ ("S-phase score", "G2M-phase score"),
+ strict=True,
+ ):
+ axis.set_title(title)
+ axis.set_xlabel("Cell-cycle phase")
+ axis.set_ylabel("score")
+ outputs.append(_save(boxes, output_directory / "cell_cycle_scores.png"))
+
+ clusters = np.asarray(store.cells.fetch(_CLUSTER_KEY))
+ highlighted_group = pd.Series(clusters).value_counts().index[0]
+ highlighted = splt.embedding(
+ store,
+ layout_key="RNA_UMAP",
+ color_by=None,
+ default_color="#bdbdbd",
+ point_alpha=0.4,
+ highlight=splt.Highlight(
+ by=_CLUSTER_KEY,
+ groups=(highlighted_group,),
+ color="#d62728",
+ dim_alpha=0.12,
+ size_multiplier=1.35,
+ halo_width=0.4,
+ ),
+ show_titles=False,
+ theme="paper",
+ show=False,
+ )
+ next(iter(highlighted.axes.values())).set_title(
+ f"Cluster {highlighted_group} highlighted"
+ )
+ outputs.append(_save(highlighted, output_directory / "highlighted_embedding.png"))
+ return outputs
+
+
+def _dark_theme_figure(
+ store: DataStore,
+ output_directory: Path,
+ feature: str,
+) -> Path:
+ dark = splt.embedding(
+ store,
+ layout_key="RNA_UMAP",
+ color_by=[_CLUSTER_KEY, feature],
+ legend_loc="on_data",
+ color_scale=splt.ColorScale(cmap="magma", quantiles=(0, 0.99)),
+ theme="dark",
+ show=False,
+ )
+ next(iter(dark.axes.values())).set_title("Leiden clusters")
+ return _save(dark, output_directory / "dark_embedding.png")
+
+
+def generate_showcase(store: DataStore, output_directory: Path) -> list[Path]:
+ output_directory.mkdir(parents=True, exist_ok=True)
+ markers = _marker_sets(store)
+ feature = _first_continuous_feature(store, markers)
+ cluster_key = _CLUSTER_KEY
+ cell_cycle_key = _CELL_CYCLE_KEY
+ if cell_cycle_key not in store.cells.columns:
+ store.run_cell_cycle_scoring()
+
+ outputs: list[Path] = []
+ categorical = splt.embedding(
+ store,
+ layout_key="RNA_UMAP",
+ color_by=cluster_key,
+ legend_loc="on_data",
+ theme="paper",
+ show=False,
+ )
+ next(iter(categorical.axes.values())).set_title("Leiden clusters")
+ outputs.append(_save(categorical, output_directory / "categorical_embedding.png"))
+
+ continuous = splt.embedding(
+ store,
+ layout_key="RNA_UMAP",
+ color_by=feature,
+ sort_values=True,
+ color_scale=splt.ColorScale(cmap="magma", quantiles=(0, 0.99)),
+ density_overlay=splt.DensityOverlay(
+ statistic="mean",
+ pixels=72,
+ sigma=5.0,
+ min_support=0.35,
+ levels=(0.9,),
+ max_hotspots=1,
+ color="#303030",
+ alpha=0.9,
+ linewidth=1.2,
+ ),
+ show_titles=False,
+ theme="paper",
+ show=False,
+ )
+ outputs.append(_save(continuous, output_directory / "continuous_embedding.png"))
+
+ connectivity = splt.cluster_connectivity(
+ store,
+ group_by=cluster_key,
+ layout_key="RNA_UMAP",
+ feat_key="hvgs",
+ show_cells=True,
+ cell_alpha=0.3,
+ theme="paper",
+ show=False,
+ )
+ outputs.append(_save(connectivity, output_directory / "cluster_connectivity.png"))
+
+ dotplot = splt.dotplot(
+ store,
+ features=markers,
+ group_by=cluster_key,
+ normalization=splt.NormalizationSpec(transform="log1p"),
+ theme="paper",
+ show=False,
+ )
+ dotplot.axes["dotplot"].set_xlabel("Leiden cluster")
+ outputs.append(_save(dotplot, output_directory / "grouped_dotplot.png"))
+
+ violin = splt.distribution(
+ store,
+ keys=[gene for genes in markers.values() for gene in genes[:1]],
+ group_by=cluster_key,
+ normalization=splt.NormalizationSpec(transform="log1p"),
+ kind="stacked_violin",
+ share_y=True,
+ max_points=600,
+ point_alpha=0.18,
+ theme="paper",
+ show=False,
+ )
+ for axis in violin.axes.values():
+ if axis.get_xlabel():
+ axis.set_xlabel("Leiden cluster")
+ outputs.append(_save(violin, output_directory / "stacked_violin.png"))
+
+ composition = splt.composition(
+ store,
+ category_by=cell_cycle_key,
+ sample_by=cluster_key,
+ segment_linewidth=0.7,
+ show_percent_labels=True,
+ label_min_fraction=0.06,
+ theme="paper",
+ show=False,
+ )
+ composition.axes["composition"].set_xlabel("Leiden cluster")
+ for legend in composition.figure.legends:
+ if legend.get_title().get_text() == cell_cycle_key:
+ legend.set_title("Cell-cycle phase")
+ outputs.append(_save(composition, output_directory / "composition.png"))
+
+ outputs.extend(_expression_matrix_figures(store, output_directory, markers))
+ outputs.extend(_additional_real_data_figures(store, output_directory))
+ outputs.append(_dark_theme_figure(store, output_directory, feature))
+
+ figure, axes = plt.subplot_mosaic(
+ [
+ ["embedding", "continuous", "connectivity"],
+ ["dotplot", "dotplot", "composition"],
+ ],
+ figsize=(11, 6.5),
+ gridspec_kw={"width_ratios": [1.0, 1.0, 0.9], "height_ratios": [1, 0.9]},
+ layout="constrained",
+ )
+ children: dict[Hashable, splt.PlotResult] = {
+ "embedding": splt.embedding(
+ store,
+ layout_key="RNA_UMAP",
+ color_by=cluster_key,
+ legend_loc="on_data",
+ show_legend=False,
+ show_titles=False,
+ target=axes["embedding"],
+ theme="paper",
+ show=False,
+ ),
+ "continuous": splt.embedding(
+ store,
+ layout_key="RNA_UMAP",
+ color_by=feature,
+ color_scale=splt.ColorScale(cmap="magma", quantiles=(0, 0.99)),
+ density_overlay=splt.DensityOverlay(
+ statistic="mean",
+ pixels=60,
+ sigma=4.2,
+ min_support=0.35,
+ levels=(0.9,),
+ max_hotspots=1,
+ color="#303030",
+ alpha=0.9,
+ linewidth=0.95,
+ ),
+ show_legend=False,
+ show_titles=False,
+ target=axes["continuous"],
+ theme="paper",
+ show=False,
+ ),
+ "connectivity": splt.cluster_connectivity(
+ store,
+ group_by=cluster_key,
+ layout_key="RNA_UMAP",
+ feat_key="hvgs",
+ show_cells=True,
+ cell_alpha=0.3,
+ target=axes["connectivity"],
+ theme="paper",
+ show=False,
+ ),
+ "dotplot": splt.dotplot(
+ store,
+ features=markers,
+ group_by=cluster_key,
+ normalization=splt.NormalizationSpec(transform="log1p"),
+ swap_axes=True,
+ show_legend=False,
+ target=axes["dotplot"],
+ theme="paper",
+ show=False,
+ ),
+ "composition": splt.composition(
+ store,
+ category_by=cell_cycle_key,
+ sample_by=cluster_key,
+ segment_linewidth=0.7,
+ show_percent_labels=False,
+ label_min_fraction=0.06,
+ show_legend=False,
+ target=axes["composition"],
+ theme="paper",
+ show=False,
+ ),
+ }
+ axes["dotplot"].set_ylabel("Leiden cluster")
+ axes["composition"].set_xlabel("Leiden cluster")
+ composite = splt.compose_results(figure, children, shared_legends=True)
+ for legend in figure.legends:
+ title = legend.get_title().get_text()
+ if title == cluster_key:
+ legend.set_title("Leiden clusters")
+ elif title == cell_cycle_key:
+ legend.set_title("Cell-cycle phase")
+ outputs.append(
+ composite.save(
+ output_directory / "publication_composite.png",
+ dpi=220,
+ exact_size=True,
+ )
+ )
+ outputs.append(
+ _normalize_svg(
+ composite.save(
+ output_directory / "publication_composite.svg",
+ exact_size=True,
+ )
+ )
+ )
+ plt.close(figure)
+ return outputs
+
+
+def _parser() -> argparse.ArgumentParser:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--fixture", type=Path, default=_FIXTURE)
+ parser.add_argument(
+ "--layout-fixture",
+ type=Path,
+ default=_LAYOUT_FIXTURE,
+ )
+ parser.add_argument(
+ "--output-dir",
+ type=Path,
+ default=Path("plotting_showcase"),
+ )
+ return parser
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+ arguments = _parser().parse_args(argv)
+ with tempfile.TemporaryDirectory(prefix="scarf_plot_showcase_") as temporary:
+ store = _prepare_store(
+ arguments.fixture,
+ Path(temporary),
+ arguments.layout_fixture,
+ )
+ outputs = generate_showcase(store, arguments.output_dir)
+ print("\n".join(str(path) for path in outputs))
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/scripts/publish_docs_datasets.py b/scripts/publish_docs_datasets.py
new file mode 100644
index 00000000..32b8e2d3
--- /dev/null
+++ b/scripts/publish_docs_datasets.py
@@ -0,0 +1,203 @@
+#!/usr/bin/env python3
+"""Publish the rebuilt documentation stores to the Cytebase bucket.
+
+Each dataset is swapped in place: the archive currently at
+`/data.zarr.tar.gz` is preserved as `_legacy_master` and the
+freshly built archive takes its place. Preservation is a server-side copy by
+xet hash, so nothing is downloaded or re-uploaded, and it never overwrites an
+existing legacy snapshot.
+
+Run without `--apply` to print the plan. Uploading needs a write token for the
+bucket.
+
+Example:
+ uv run python scripts/publish_docs_datasets.py
+ uv run python scripts/publish_docs_datasets.py --apply tenx_5K_pbmc_rnaseq
+"""
+
+import argparse
+import hashlib
+import json
+import sys
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from huggingface_hub import BucketFile, batch_bucket_files, list_bucket_tree
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+MANIFEST_DIR = REPO_ROOT / "docs/source/developers/dataset_manifests"
+BUCKET_ID = "Nygen/cytebase"
+REPOSITORY = "scarf_docs"
+STORE_NAME = "data.zarr"
+ARCHIVE_NAME = f"{STORE_NAME}.tar.gz"
+LEGACY_SUFFIX = "_legacy_master"
+
+# Datasets whose unpacked store backs the remote `open_zarr` example.
+UNPACKED_DATASETS = frozenset({"tenx_5K_pbmc_rnaseq"})
+
+
+@dataclass(slots=True)
+class Plan:
+ """Every bucket mutation needed to publish one dataset."""
+
+ dataset: str
+ preserve: list[tuple[str, str, str, str]] = field(default_factory=list)
+ upload: list[tuple[Path, str]] = field(default_factory=list)
+ remove: list[str] = field(default_factory=list)
+ notes: list[str] = field(default_factory=list)
+
+ @property
+ def empty(self) -> bool:
+ return not (self.preserve or self.upload or self.remove)
+
+
+def _published_files(prefix: str) -> dict[str, BucketFile]:
+ files = {}
+ for item in list_bucket_tree(
+ BUCKET_ID,
+ prefix=f"{REPOSITORY}/{prefix}",
+ recursive=True,
+ token=False,
+ ):
+ if isinstance(item, BucketFile):
+ files[item.path] = item
+ return files
+
+
+def _file_digest(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for block in iter(lambda: handle.read(1 << 20), b""):
+ digest.update(block)
+ return digest.hexdigest()
+
+
+def _plan_dataset(dataset: str, build_root: Path) -> Plan:
+ manifest_path = MANIFEST_DIR / f"{dataset}_manifest.json"
+ if not manifest_path.is_file():
+ raise FileNotFoundError(f"No manifest for {dataset}, rebuild it first")
+ manifest = json.loads(manifest_path.read_text())
+
+ archive = build_root / dataset / ARCHIVE_NAME
+ if not archive.is_file():
+ raise FileNotFoundError(f"No archive at {archive}, rebuild {dataset} first")
+ digest = _file_digest(archive)
+ if digest != manifest["archiveSha256"]:
+ raise RuntimeError(
+ f"{archive} does not match its manifest checksum. Rebuild {dataset} "
+ "so the manifest and the archive describe the same store."
+ )
+
+ plan = Plan(dataset=dataset)
+ published = _published_files(dataset)
+ current = published.get(f"{REPOSITORY}/{dataset}/{ARCHIVE_NAME}")
+ legacy_path = f"{REPOSITORY}/{dataset}{LEGACY_SUFFIX}/{ARCHIVE_NAME}"
+ legacy_exists = bool(_published_files(f"{dataset}{LEGACY_SUFFIX}"))
+
+ if current is None:
+ plan.notes.append("No archive is published yet, nothing to preserve")
+ elif legacy_exists:
+ plan.notes.append(f"{dataset}{LEGACY_SUFFIX} already exists, leaving it alone")
+ else:
+ plan.preserve.append(("bucket", BUCKET_ID, current.xet_hash, legacy_path))
+
+ plan.upload.append((archive, f"{REPOSITORY}/{dataset}/{ARCHIVE_NAME}"))
+
+ if dataset in UNPACKED_DATASETS:
+ store = build_root / dataset / STORE_NAME
+ if not store.is_dir():
+ raise FileNotFoundError(f"No unpacked store at {store}")
+ wanted = set()
+ for item in sorted(store.rglob("*")):
+ if not item.is_file():
+ continue
+ relative = item.relative_to(store).as_posix()
+ remote = f"{REPOSITORY}/{dataset}/{STORE_NAME}/{relative}"
+ wanted.add(remote)
+ plan.upload.append((item, remote))
+ stale = [
+ path
+ for path in published
+ if path.startswith(f"{REPOSITORY}/{dataset}/{STORE_NAME}/")
+ and path not in wanted
+ ]
+ plan.remove.extend(sorted(stale))
+
+ return plan
+
+
+def _describe(plan: Plan) -> None:
+ print(plan.dataset)
+ for note in plan.notes:
+ print(f" note {note}")
+ for _, _, _, destination in plan.preserve:
+ print(f" preserve {destination} (server-side copy)")
+ archives = [item for item in plan.upload if item[1].endswith(ARCHIVE_NAME)]
+ members = [item for item in plan.upload if not item[1].endswith(ARCHIVE_NAME)]
+ for source, destination in archives:
+ print(f" upload {destination} ({source.stat().st_size / 1e6:.1f} MB)")
+ if members:
+ total = sum(source.stat().st_size for source, _ in members)
+ print(f" upload {len(members)} unpacked files ({total / 1e6:.1f} MB)")
+ if plan.remove:
+ print(f" delete {len(plan.remove)} stale unpacked file(s)")
+
+
+def _apply(plan: Plan) -> None:
+ if plan.preserve:
+ batch_bucket_files(BUCKET_ID, copy=plan.preserve)
+ print(f" preserved {len(plan.preserve)} archive(s)")
+ if plan.upload:
+ batch_bucket_files(
+ BUCKET_ID,
+ add=[(str(source), destination) for source, destination in plan.upload],
+ )
+ print(f" uploaded {len(plan.upload)} file(s)")
+ if plan.remove:
+ batch_bucket_files(BUCKET_ID, delete=plan.remove)
+ print(f" deleted {len(plan.remove)} stale file(s)")
+
+
+def main(argv: list[str] | None = None) -> int:
+ available = sorted(
+ path.name.removesuffix("_manifest.json")
+ for path in MANIFEST_DIR.glob("*_manifest.json")
+ )
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "datasets",
+ nargs="*",
+ choices=[*available, []],
+ help="Datasets to publish, defaults to every rebuilt dataset",
+ )
+ parser.add_argument(
+ "--apply",
+ action="store_true",
+ help="Perform the plan instead of printing it",
+ )
+ parser.add_argument(
+ "--build-root",
+ type=Path,
+ default=REPO_ROOT / "build/cytebase",
+ help="Directory holding the rebuilt stores and archives",
+ )
+ args = parser.parse_args(argv)
+
+ selected = list(args.datasets) or available
+ if not selected:
+ parser.error("no manifests found, rebuild a dataset first")
+
+ plans = [_plan_dataset(dataset, args.build_root) for dataset in selected]
+ for plan in plans:
+ _describe(plan)
+ if args.apply and not plan.empty:
+ _apply(plan)
+ print()
+
+ if not args.apply:
+ print("Dry run. Pass --apply to publish, which needs a bucket write token.")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/regenerate_docs_datasets.py b/scripts/regenerate_docs_datasets.py
new file mode 100644
index 00000000..70a7f779
--- /dev/null
+++ b/scripts/regenerate_docs_datasets.py
@@ -0,0 +1,502 @@
+#!/usr/bin/env python3
+"""Rebuild the analyzed Zarr stores that Cytebase publishes for the documentation.
+
+Each store is built from the dataset's raw counts, so the result uses the
+current Zarr layout and carries artifact provenance. This script only writes to
+`build/cytebase`; `scripts/publish_docs_datasets.py` uploads the result.
+
+Example:
+ uv run python scripts/regenerate_docs_datasets.py tenx_5K_pbmc_rnaseq
+ uv run python scripts/regenerate_docs_datasets.py --all
+"""
+
+import argparse
+import hashlib
+import json
+import shutil
+import subprocess
+import sys
+import tarfile
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Any
+
+REPO_ROOT = Path(__file__).resolve().parents[1]
+MANIFEST_DIR = REPO_ROOT / "docs/source/developers/dataset_manifests"
+STORE_NAME = "data.zarr"
+ARCHIVE_NAME = f"{STORE_NAME}.tar.gz"
+
+PBMC_FILTERS = {
+ "method": "manual",
+ "attrs": ["RNA_nCounts", "RNA_nFeatures", "RNA_percentMito"],
+ "highs": [15000, 4000, 15],
+ "lows": [1000, 500, 0],
+}
+
+
+def _convert_cellranger_h5(source: Path, store: Path) -> None:
+ import scarf
+
+ reader = scarf.CrH5Reader(str(source / "data.h5"))
+ scarf.CrToZarr(reader, zarr_loc=str(store)).dump()
+
+
+def _convert_cellranger_mtx(source: Path, store: Path) -> None:
+ import scarf
+
+ reader = scarf.CrDirReader(str(source))
+ scarf.CrToZarr(reader, zarr_loc=str(store)).dump()
+
+
+def _convert_h5ad(source: Path, store: Path) -> None:
+ import scarf
+
+ inspection = scarf.inspect_h5ad(str(source / "data.h5ad"))
+ reader = scarf.H5adReader.from_inspect(inspection)
+ scarf.H5adToZarr(reader, zarr_loc=str(store)).dump()
+
+
+def _analyze_pbmc(store: Any) -> None:
+ store.pipeline.run(
+ filtering=PBMC_FILTERS,
+ highly_variable_features={
+ "min_cells": 20,
+ "top_n": 500,
+ "min_mean": -3,
+ "max_mean": 2,
+ "max_var": 6,
+ },
+ pca={"dims": 15, "n_centroids": 100},
+ neighbors={"k": 11},
+ umap={"n_epochs": 250, "spread": 5, "min_dist": 1, "parallel": True},
+ leiden={0.5: {"label": "leiden_cluster"}},
+ paris=False,
+ markers={},
+ )
+ store.run_paris_clustering(n_clusters=15)
+
+
+def _analyze_pancreas(store: Any) -> None:
+ store.pipeline.run(
+ filtering=False,
+ cell_cycle_scoring=False,
+ highly_variable_features={"min_cells": 20, "top_n": 2000},
+ pca={"dims": 15, "n_centroids": 100},
+ neighbors={"k": 11},
+ umap={"n_epochs": 250, "spread": 5, "min_dist": 1, "parallel": True},
+ leiden={0.5: {"label": "leiden_cluster"}},
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ )
+ store.run_marker_search(group_key="clusters")
+
+
+def _analyze_kang(store: Any) -> None:
+ store.pipeline.run(
+ filtering={
+ "method": "manual",
+ "attrs": ["RNA_nCounts", "RNA_nFeatures"],
+ "highs": [15000, 4000],
+ "lows": [500, 200],
+ },
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "min_cells": 10,
+ "top_n": 2000,
+ "min_mean": -3,
+ "max_mean": 2,
+ "max_var": 6,
+ },
+ pca={"dims": 25, "n_centroids": 100},
+ neighbors={"k": 21},
+ umap={"n_epochs": 250, "spread": 5, "min_dist": 1, "parallel": True},
+ leiden={1.0: {"label": "leiden_cluster"}},
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ )
+
+
+def _analyze_citeseq(store: Any) -> None:
+ import numpy as np
+
+ store.auto_filter_cells(show_qc_plots=False)
+ store.pipeline.run(
+ filtering=False,
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "min_cells": 20,
+ "top_n": 1000,
+ "min_mean": -3,
+ "max_mean": 2,
+ "max_var": 6,
+ },
+ pca={"dims": 15, "n_centroids": 100},
+ neighbors={"k": 21},
+ umap={"n_epochs": 250, "spread": 5, "min_dist": 1, "parallel": True},
+ leiden={1.0: {"label": "leiden_cluster"}},
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ )
+
+ names = np.asarray(store.ADT.feats.fetch_all("names")).astype(str)
+ is_control = np.char.find(np.char.lower(names), "control") >= 0
+ store.ADT.feats.update_key(~is_control, "I")
+
+ normalized = store.run_normalization(from_assay="ADT", feat_key="I")
+ n_features = int(store.load_artifact(normalized)["data"].shape[1])
+ reduction = store.run_custom_reduction(
+ np.eye(n_features, dtype=np.float64),
+ normalized,
+ from_assay="ADT",
+ )
+ store.build_embedding_initialization(reduction, n_centroids=100)
+ ann = store.build_ann_index(reduction)
+ neighbors = store.query_neighbors(ann, k=21)
+ graph = store.build_connectivity_map(neighbors)
+ store.run_umap(graph, n_epochs=250, spread=5, min_dist=1, parallel=True)
+ store.run_leiden_clustering(graph, resolution=1.0, label="leiden_cluster")
+
+ for label, method in (("RNA+ADT", "snn"), ("RNA+ADT_wnn", "wnn")):
+ integrated = store.integrate_assays(
+ assays=["RNA", "ADT"],
+ label=label,
+ method=method,
+ )
+ store.run_umap(integrated, n_epochs=250, spread=5, min_dist=1, parallel=True)
+ store.run_leiden_clustering(integrated, resolution=1.75, label="leiden_cluster")
+
+
+def _analyze_atac(store: Any) -> None:
+ store.auto_filter_cells(show_qc_plots=False)
+ store.mark_prevalent_peaks(top_n=25000)
+ normalized = store.run_normalization(feat_key="prevalent_peaks")
+ reduction = store.run_lsi(normalized, dims=50, skip_first=True)
+ store.build_embedding_initialization(reduction)
+ ann = store.build_ann_index(reduction)
+ neighbors = store.query_neighbors(ann, k=21)
+ graph = store.build_connectivity_map(neighbors)
+ store.run_umap(graph, n_epochs=500, min_dist=0.1, spread=1, parallel=True)
+ store.run_leiden_clustering(graph, resolution=0.6, label="leiden_cluster")
+
+
+@dataclass(frozen=True, slots=True)
+class DatasetRecipe:
+ """One publishable store: where its counts come from and how it is analyzed."""
+
+ sources: tuple[str, ...]
+ convert: Callable[[Path, Path], None]
+ analyze: Callable[[Any], None]
+ summary: str
+ default_assay: str = "RNA"
+ carry_columns: tuple[str, ...] = ()
+ drop_columns: tuple[str, ...] = ()
+
+
+RECIPES: dict[str, DatasetRecipe] = {
+ "tenx_5K_pbmc_rnaseq": DatasetRecipe(
+ sources=("data.h5",),
+ convert=_convert_cellranger_h5,
+ analyze=_analyze_pbmc,
+ summary=(
+ "Manual QC filter, 500 HVGs, PCA 15, k=11 graph, UMAP, "
+ "Leiden 0.5, Paris, doublet scores, markers"
+ ),
+ ),
+ "bastidas-ponce_4K_pancreas-d15_rnaseq": DatasetRecipe(
+ sources=("data.h5ad",),
+ convert=_convert_h5ad,
+ analyze=_analyze_pancreas,
+ summary=(
+ "2000 HVGs, PCA 15, k=11 graph, UMAP, Leiden 0.5, markers on the "
+ "published cell-type annotations"
+ ),
+ drop_columns=("X_pca*",),
+ ),
+ "tenx_8K_pbmc_citeseq": DatasetRecipe(
+ sources=("data.h5",),
+ convert=_convert_cellranger_h5,
+ analyze=_analyze_citeseq,
+ summary=(
+ "RNA and ADT chains with UMAP and Leiden, plus SNN and WNN "
+ "integrated graphs"
+ ),
+ ),
+ "kang_15K_pbmc_rnaseq": DatasetRecipe(
+ sources=("matrix.mtx.gz", "features.tsv.gz", "barcodes.tsv.gz"),
+ convert=_convert_cellranger_mtx,
+ analyze=_analyze_kang,
+ summary="2000 HVGs, PCA 25, k=21 graph, UMAP, Leiden 1.0",
+ carry_columns=("cluster_labels",),
+ ),
+ "kang_14K_ifnb-pbmc_rnaseq": DatasetRecipe(
+ sources=("matrix.mtx.gz", "features.tsv.gz", "barcodes.tsv.gz"),
+ convert=_convert_cellranger_mtx,
+ analyze=_analyze_kang,
+ summary="2000 HVGs, PCA 25, k=21 graph, UMAP, Leiden 1.0",
+ carry_columns=("cluster_labels",),
+ ),
+ "tenx_10K_pbmc-v1_atacseq": DatasetRecipe(
+ sources=("data.h5",),
+ convert=_convert_cellranger_h5,
+ analyze=_analyze_atac,
+ summary=(
+ "25000 prevalent peaks, TF-IDF normalization, LSI 50, k=21 graph, "
+ "UMAP, Leiden 0.6"
+ ),
+ default_assay="ATAC",
+ ),
+}
+
+
+def _git_commit() -> str:
+ try:
+ return subprocess.check_output(
+ ["git", "rev-parse", "HEAD"],
+ cwd=REPO_ROOT,
+ text=True,
+ ).strip()
+ except (OSError, subprocess.CalledProcessError):
+ return "unknown"
+
+
+def _artifact_inventory(store: Any) -> list[dict[str, object]]:
+ refs = list(store.list_artifacts(scope="datastore", complete_only=True))
+ for assay in store.assay_names:
+ refs.extend(
+ store.list_artifacts(
+ from_assay=assay,
+ complete_only=True,
+ )
+ )
+ inventory: list[dict[str, object]] = []
+ for ref in refs:
+ status = store.inspect_artifact(ref)
+ inventory.append(
+ {
+ "kind": ref.kind,
+ "artifactId": ref.artifact_id,
+ "assay": ref.assay,
+ "scope": ref.scope,
+ "operation": status.operation,
+ "path": status.path,
+ }
+ )
+ return inventory
+
+
+def _directory_bytes(path: Path) -> int:
+ return sum(item.stat().st_size for item in path.rglob("*") if item.is_file())
+
+
+def _file_digest(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ for block in iter(lambda: handle.read(1 << 20), b""):
+ digest.update(block)
+ return digest.hexdigest()
+
+
+def _drop_columns(
+ *,
+ store: Path,
+ patterns: Sequence[str],
+ default_assay: str,
+) -> None:
+ """Remove imported columns that would only add noise to metadata tables."""
+ from fnmatch import fnmatch
+
+ from scarf import DataStore
+
+ target = DataStore(str(store), default_assay=default_assay, nthreads=1)
+ dropped = [
+ column
+ for column in target.cells.columns
+ if any(fnmatch(column, pattern) for pattern in patterns)
+ ]
+ for column in dropped:
+ target.cells.drop(column)
+ print(f"Dropped {len(dropped)} imported column(s) from {store}")
+
+
+def _carry_columns(
+ *,
+ repository: Any,
+ dataset: str,
+ store: Path,
+ columns: Sequence[str],
+ work: Path,
+ default_assay: str,
+) -> None:
+ """Copy author-provided annotations that only exist in the published store."""
+ import numpy as np
+ import zarr
+
+ from scarf import DataStore
+ from scarf.storage.types import as_zarr_array, as_zarr_group
+
+ repository.download_dataset(dataset, destination=str(work), zarr=True)
+ legacy = zarr.open_group(str(work / dataset / STORE_NAME), mode="r")
+ legacy_cells = as_zarr_group(legacy["cellData"], name="cellData")
+ legacy_ids = np.asarray(as_zarr_array(legacy_cells["ids"], name="ids")[:]).astype(
+ str
+ )
+ positions = {value: index for index, value in enumerate(legacy_ids)}
+
+ target = DataStore(str(store), default_assay=default_assay, nthreads=1)
+ target_ids = np.asarray(target.cells.fetch_all("ids")).astype(str)
+ missing = [value for value in target_ids if value not in positions]
+ if missing:
+ raise RuntimeError(
+ f"{len(missing)} cells of {dataset} are absent from the published "
+ f"store, first missing id {missing[0]!r}"
+ )
+ order = np.array([positions[value] for value in target_ids])
+ for column in columns:
+ if column not in legacy_cells:
+ raise RuntimeError(f"{dataset} has no published column {column!r}")
+ source = as_zarr_array(legacy_cells[column], name=column)
+ values = np.asarray(source[:])[order]
+ target.cells.insert(column, values, overwrite=True)
+ print(f"Carried {len(columns)} published column(s) into {store}")
+
+
+def build_store(
+ *,
+ dataset: str,
+ recipe: DatasetRecipe,
+ destination: Path,
+ repository_name: str,
+) -> Path:
+ import scarf
+ from scarf import DataStore
+
+ started = datetime.now(UTC)
+ work = destination / "_source"
+ repository = scarf.cytebase.connect(repository_name)
+ for name in recipe.sources:
+ repository.download(f"{dataset}/{name}", destination=str(work))
+
+ output = destination / dataset
+ output.mkdir(parents=True, exist_ok=True)
+ store = output / STORE_NAME
+ if store.exists():
+ shutil.rmtree(store)
+ recipe.convert(work / dataset, store)
+
+ if recipe.drop_columns:
+ _drop_columns(
+ store=store,
+ patterns=recipe.drop_columns,
+ default_assay=recipe.default_assay,
+ )
+
+ if recipe.carry_columns:
+ _carry_columns(
+ repository=repository,
+ dataset=dataset,
+ store=store,
+ columns=recipe.carry_columns,
+ work=work,
+ default_assay=recipe.default_assay,
+ )
+
+ datastore = DataStore(
+ str(store),
+ default_assay=recipe.default_assay,
+ nthreads=4,
+ )
+ recipe.analyze(datastore)
+
+ archive = output / ARCHIVE_NAME
+ with tarfile.open(archive, "w:gz") as handle:
+ handle.add(store, arcname=STORE_NAME)
+
+ store_bytes = _directory_bytes(store)
+ archive_bytes = archive.stat().st_size
+ cells_total = int(datastore.cells.N)
+ cells_active = int(datastore.cells.active_index("I").size)
+ artifacts = _artifact_inventory(datastore)
+
+ manifest = {
+ "dataset": dataset,
+ "repository": repository_name,
+ "recipe": recipe.summary,
+ "generatedAt": started.isoformat(),
+ "generatorCommit": _git_commit(),
+ "scarfVersion": getattr(scarf, "__version__", "unknown"),
+ "sourceFiles": list(recipe.sources),
+ "carriedColumns": list(recipe.carry_columns),
+ "nCellsTotal": cells_total,
+ "nCellsActive": cells_active,
+ "storeBytes": store_bytes,
+ "archiveBytes": archive_bytes,
+ "archiveSha256": _file_digest(archive),
+ "cellColumns": sorted(datastore.cells.columns),
+ "artifacts": artifacts,
+ "publishNotes": [
+ f"Publish with: uv run python scripts/publish_docs_datasets.py {dataset}",
+ f"That preserves the published archive as {dataset}_legacy_master "
+ f"and swaps {ARCHIVE_NAME} in place.",
+ ],
+ }
+ MANIFEST_DIR.mkdir(parents=True, exist_ok=True)
+ manifest_path = MANIFEST_DIR / f"{dataset}_manifest.json"
+ manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n")
+
+ elapsed = (datetime.now(UTC) - started).total_seconds()
+ print(f"Built {dataset} in {elapsed:.0f}s")
+ print(f" store {store} ({store_bytes / 1e6:.1f} MB)")
+ print(f" archive {archive} ({archive_bytes / 1e6:.1f} MB)")
+ print(f" manifest {manifest_path}")
+ print(f" cells {cells_active} of {cells_total} active")
+ print(f" results {len(artifacts)} complete artifact(s)")
+ return store
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "datasets",
+ nargs="*",
+ choices=[*RECIPES, []],
+ help="Datasets to rebuild",
+ )
+ parser.add_argument(
+ "--all",
+ action="store_true",
+ help="Rebuild every dataset in the recipe table",
+ )
+ parser.add_argument(
+ "--repository",
+ default="scarf_docs",
+ help="Cytebase repository holding the raw counts",
+ )
+ parser.add_argument(
+ "--destination",
+ type=Path,
+ default=REPO_ROOT / "build/cytebase",
+ help="Output directory for stores, archives, and downloads",
+ )
+ args = parser.parse_args(argv)
+
+ selected = list(RECIPES) if args.all else list(args.datasets)
+ if not selected:
+ parser.error("name at least one dataset or pass --all")
+ args.destination.mkdir(parents=True, exist_ok=True)
+ for dataset in selected:
+ build_store(
+ dataset=dataset,
+ recipe=RECIPES[dataset],
+ destination=args.destination,
+ repository_name=args.repository,
+ )
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/setup.py b/setup.py
index 4b86e972..6ab280cd 100644
--- a/setup.py
+++ b/setup.py
@@ -1,24 +1,12 @@
-from setuptools import setup, find_packages
-from setuptools.command.install import install
-import os
import glob
+import os
-
-def read(f_name):
- with open(os.path.join(os.path.dirname(__file__), f_name)) as fp:
- return fp.read().rstrip("\n")
-
-
-def read_lines(f_name):
- return read(f_name).split("\n")
+from setuptools import setup
+from setuptools.command.install import install
class PostInstallCommand(install):
- """
- Post-installation for installation mode.
- From here:
- https://github.com/benfred/py-spy/blob/290584dde76834599d66d74b64165dfe9a357ef5/setup.py#L42
- """
+ """Copy bundled binaries from bin/ into the install scripts directory."""
def run(self):
install.run(self)
@@ -32,38 +20,7 @@ def run(self):
os.remove(target)
with open(source, "rb") as src, open(target, "wb") as dst:
dst.write(src.read())
- os.chmod(target, 0o655)
-
+ os.chmod(target, 0o755)
-if __name__ == "__main__":
- version = read("VERSION").rstrip("\n")
- core_requirements = read_lines("requirements.txt")
- extra_requirements = read_lines("requirements_extra.txt")
- setup(
- name="scarf",
- version=version,
- python_requires=">=3.11",
- description="Scarf: A scalable tool for single-cell omics data analysis",
- long_description=read("pypi_README.rst"),
- long_description_content_type="text/x-rst",
- author="Parashar Dhapola",
- author_email="parashar.dhapola@gmail.com",
- url="https://github.com/parashardhapola/scarf",
- license="BSD 3-Clause",
- classifiers=[
- "Development Status :: 4 - Beta",
- "License :: OSI Approved :: BSD License",
- "Natural Language :: English",
- "Programming Language :: Python :: 3",
- "Programming Language :: Python :: 3.12",
- ],
- keywords=["single-cell"],
- install_requires=core_requirements,
- extras_require={
- "extra": extra_requirements,
- },
- packages=find_packages(exclude=["tests*"]),
- include_package_data=False,
- cmdclass={"install": PostInstallCommand},
- )
+setup(cmdclass={"install": PostInstallCommand})
diff --git a/tests/__init__.py b/tests/__init__.py
new file mode 100644
index 00000000..1afaae52
--- /dev/null
+++ b/tests/__init__.py
@@ -0,0 +1,26 @@
+import os
+import shutil
+
+__all__ = ["full_path", "remove", "dask_total_sum"]
+
+_DATASETS_DIR = os.path.join(os.path.dirname(__file__), "datasets")
+
+
+def dask_total_sum(raw_data) -> int:
+ total = 0
+ for block in raw_data.blocks:
+ total += int(block.compute().sum())
+ return total
+
+
+def full_path(fn, *args):
+ if fn == "" or fn is None:
+ return _DATASETS_DIR
+ return os.path.join(_DATASETS_DIR, fn, *args)
+
+
+def remove(dir_path):
+ if os.path.isdir(dir_path):
+ shutil.rmtree(dir_path)
+ elif os.path.exists(dir_path):
+ os.unlink(dir_path)
diff --git a/tests/conftest.py b/tests/conftest.py
new file mode 100644
index 00000000..3d9e142f
--- /dev/null
+++ b/tests/conftest.py
@@ -0,0 +1,19 @@
+import sys
+
+import pytest
+
+from scarf.utils import configure_output, logger
+
+pytest_plugins = [
+ "tests.fixtures_downloader",
+ "tests.fixtures_readers",
+ "tests.fixtures_datastore",
+]
+
+
+@pytest.fixture(scope="session", autouse=True)
+def _quiet_test_logs() -> None:
+ """Keep pytest output readable while standalone CLIs retain INFO logs."""
+ configure_output(progress=False)
+ logger.remove()
+ logger.add(sys.stderr, level="ERROR")
diff --git a/scarf/tests/datasets/markers_all_clusters.csv b/tests/datasets/markers_all_clusters.csv
similarity index 94%
rename from scarf/tests/datasets/markers_all_clusters.csv
rename to tests/datasets/markers_all_clusters.csv
index 0f83c217..db781006 100644
--- a/scarf/tests/datasets/markers_all_clusters.csv
+++ b/tests/datasets/markers_all_clusters.csv
@@ -29,8 +29,8 @@ BCL11B,SIRPA,CD163,PYGL,PDLIM1,CST7,TRDC,PDK1,PGM2L1,CSF1R
,ACER3,HPSE,PGD,TCL1A,MAP4K1,PRSS23,NELL2,CD28,TNNI2
,CACNA2D3,FGL2,VCAN,PLPP5,CMC1,CCL5,SH3YL1,CD226,DAPK1
,PLXDC2,TLR8,PLBD1,BLNK,FKBP11,C1orf21,SNHG14,,LRRC25
-,HPSE,TMEM176A,LRRK2,CD24,SYNE2,XCL2,CAMK2G,,LILRB1
-,NIBAN2,DUSP6,RETN,CD40,BDH2,CTSW,DYRK2,,LINC02432
+,HPSE,TMEM176A,LRRK2,CD24,SYNE2,XCL2,CAMK2G,,LINC02432
+,NIBAN2,DUSP6,RETN,CD40,BDH2,CTSW,DYRK2,,LILRB1
,SORT1,NLRP3,AQP9,BCL11A,CD3D,PLEKHF1,,,CXCL10
,INPPL1,IMPA2,SLC11A1,COBLL1,PATL2,KLRB1,,,RNF144B
,RTN1,DSC2,ALDH2,CD72,CD69,MCTP2,,,CDK2AP1
@@ -56,8 +56,8 @@ BCL11B,SIRPA,CD163,PYGL,PDLIM1,CST7,TRDC,PDK1,PGM2L1,CSF1R
,RGS18,LILRA5,GM2A,EAF2,,TFDP2,,,CALHM6
,DOK3,NOD2,SMARCD3,MZB1,,USP28,,,TPPP3
,PELI2,MAFB,NAIP,CYB561A3,,HENMT1,,,LST1
-,NFAM1,LTBR,S1PR3,SWAP70,,GK5,,,DUSP5
-,ALDH1A1,SECTM1,VSTM1,FCGR2B,,GPR174,,,FCER1G
+,NFAM1,SECTM1,S1PR3,SWAP70,,GK5,,,DUSP5
+,ALDH1A1,LTBR,VSTM1,FCGR2B,,GPR174,,,FCER1G
,PLA2G7,NBPF14,RBP7,BCL7A,,CD38,,,HCAR3
,CRTAP,AGAP3,DENND6B,MICAL3,,ZAP70,,,TNFRSF8
,OAF,IFIT2,NLRP12,FCRL3,,DDIT4,,,SIDT2
@@ -126,10 +126,10 @@ BCL11B,SIRPA,CD163,PYGL,PDLIM1,CST7,TRDC,PDK1,PGM2L1,CSF1R
,FNDC3B,GRN,PTAFR,TCF3,,SYNE1,,,MAFB
,CYBB,KCNK6,SHTN1,SETBP1,,LYAR,,,IGSF6
,SGMS2,TNFSF13,AL355075.4,LINC01215,,GPRIN3,,,ADGRE1
-,RAB31,ATP6V1B2,TNFSF13,ZNF318,,F2R,,,CD86
-,,LPCAT2,ACAP3,SMARCB1,,LPCAT1,,,IFIT3
-,,C2CD2L,KLF4,MEF2C,,SH2D1A,,,APH1B
-,,HOMER3,HOMER3,RIC3,,STAT4,,,PLAGL2
+,RAB31,ATP6V1B2,TNFSF13,SMARCB1,,F2R,,,CD86
+,,LPCAT2,ACAP3,ZNF318,,LPCAT1,,,IFIT3
+,,C2CD2L,KLF4,MEF2C,,SH2D1A,,,PLAGL2
+,,HOMER3,HOMER3,RIC3,,STAT4,,,APH1B
,,C3AR1,CD101,ODC1,,EOMES,,,KCNJ2
,,MYOF,CD302,LINC01781,,KLHDC4,,,HSBP1
,,EMILIN2,INPPL1,TIFA,,ATP2A3,,,TTYH3
@@ -139,8 +139,8 @@ BCL11B,SIRPA,CD163,PYGL,PDLIM1,CST7,TRDC,PDK1,PGM2L1,CSF1R
,,NFE2,AGO4,SLC9A7,,TNIK,,,PIK3AP1
,,KCTD12,LIN7A,FUT8,,EBP,,,AP2A1
,,LRRK2,FCN1,BEND5,,MEX3C,,,TOR4A
-,,P2RX7,MAFG,AC008969.1,,SNTB2,,,SIGLEC5
-,,CLEC7A,BCL6,HLA-DRA,,MRPL10,,,LINC00467
+,,P2RX7,MAFG,AC008969.1,,SNTB2,,,LINC00467
+,,CLEC7A,BCL6,HLA-DRA,,MRPL10,,,SIGLEC5
,,IFI44L,GAS7,HLA-DMA,,PTPN7,,,HGF
,,MSRB1,APOBR,IGHG1,,CD7,,,SPI1
,,MPZL2,CARS2,IL4R,,CYTOR,,,PPM1N
@@ -182,8 +182,8 @@ BCL11B,SIRPA,CD163,PYGL,PDLIM1,CST7,TRDC,PDK1,PGM2L1,CSF1R
,,CXCL10,ZNF317,,,,,,TUBA1B
,,PER1,OSCAR,,,,,,TIMP1
,,RXRA,FOSL2,,,,,,TRIQK
-,,INPPL1,TIMP2,,,,,,SGPP1
-,,OAS3,ZNF516,,,,,,HLA-DRB5
+,,OAS3,TIMP2,,,,,,SGPP1
+,,INPPL1,ZNF516,,,,,,HLA-DRB5
,,JAK2,VENTX,,,,,,AC147067.1
,,FCN1,APLP2,,,,,,TTYH2
,,LILRA1,MXD1,,,,,,CARD9
diff --git a/tests/download_fixtures.py b/tests/download_fixtures.py
new file mode 100644
index 00000000..74c0dada
--- /dev/null
+++ b/tests/download_fixtures.py
@@ -0,0 +1,233 @@
+"""Download bundled test datasets for CI and local development."""
+
+import argparse
+import gzip
+import hashlib
+import sys
+import tarfile
+import tempfile
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+import h5py
+import pandas as pd
+from scipy.sparse import csr_matrix
+
+_H5AD_DOWNLOAD_ATTEMPTS = 3
+
+_FIXTURES_BASE_URL = "https://raw.githubusercontent.com/parashardhapola/scarf/master/scarf/tests/datasets"
+_EXTERNAL_FIXTURES = {
+ "seurat_assay5_synthetic.rds": (
+ "https://raw.githubusercontent.com/harryhaller001/readseurat/"
+ "8a688b47df27f90e98a4c57ddd9e47c0e5ded01e/tests/data/synthetic.rds",
+ "f1e6f6fd3e1959452a9ef7e72571a86e1b27a061d8cf00cd28932d8757cdac7c",
+ ),
+ "seurat_v4_1_3_pbmc_mye.rds": (
+ "https://zenodo.org/api/records/10944066/files/"
+ "pbmc10k_mye_small_velocyto.rds/content",
+ "f84adf523a78aeb6e6681cf09e06a2a2fcd4e3fe857fdd89b17e90a1782fac3d",
+ ),
+}
+
+FIXTURE_FILES = (
+ "1K_pbmc_citeseq.h5",
+ "1K_pbmc_citeseq.zarr.tar.gz",
+ "500_pbmc_atac.zarr.tar.gz",
+ "toy_cr_dir.tar.gz",
+ "toy_cr_dir_empty.tar.gz",
+ "sympathetic.loom",
+ "cell_attributes.csv",
+ "knn_indices.npy",
+ "knn_distances.npy",
+ "knn_weights.npy",
+ "atac_knn_indices.npy",
+ "atac_knn_distances.npy",
+ "markers_cluster1.csv",
+ "pseudotime_markers_r_values.csv",
+ "aggregated_feat_idx.npy",
+ "aggregated_df_top_10.npy",
+ "pseudotime_clusters.npy",
+ "ptime_modules_group_1.npy",
+ "seurat_assay5_synthetic.rds",
+ "seurat_v4_1_3_pbmc_mye.rds",
+)
+
+
+def datasets_dir() -> Path:
+ return Path(__file__).resolve().parent / "datasets"
+
+
+def _sha256(path: Path) -> str:
+ digest = hashlib.sha256()
+ with path.open("rb") as handle:
+ while block := handle.read(1024 * 1024):
+ digest.update(block)
+ return digest.hexdigest()
+
+
+def download_fixtures(*, force: bool = False) -> None:
+ target = datasets_dir()
+ target.mkdir(parents=True, exist_ok=True)
+
+ missing: list[tuple[str, urllib.error.HTTPError]] = []
+ for name in FIXTURE_FILES:
+ dest = target / name
+ external = _EXTERNAL_FIXTURES.get(name)
+ if dest.is_file() and not force:
+ if external is None or _sha256(dest) == external[1]:
+ continue
+ dest.unlink()
+ url = f"{_FIXTURES_BASE_URL}/{name}" if external is None else external[0]
+ try:
+ urllib.request.urlretrieve(url, dest)
+ except urllib.error.HTTPError as exc:
+ missing.append((name, exc))
+ continue
+ if external is not None:
+ digest = _sha256(dest)
+ if digest != external[1]:
+ dest.unlink()
+ raise RuntimeError(
+ f"Fixture {name} has SHA-256 {digest}, expected {external[1]}"
+ )
+
+ if missing:
+ details = "\n".join(f" {name}: HTTP {exc.code}" for name, exc in missing)
+ msg = f"Failed to download {len(missing)} fixture(s):\n{details}"
+ raise RuntimeError(msg)
+
+
+def build_mtx_dir_fixture() -> None:
+ archive = datasets_dir() / "1K_pbmc_citeseq_dir.tar.gz"
+ if archive.is_file():
+ return
+
+ h5_path = datasets_dir() / "1K_pbmc_citeseq.h5"
+ if not h5_path.is_file():
+ msg = (
+ "Cannot build 1K_pbmc_citeseq_dir.tar.gz without "
+ f"{h5_path.name}; download core fixtures first."
+ )
+ raise RuntimeError(msg)
+
+ from scarf.readers import CrDirReader, CrH5Reader
+
+ reader = CrH5Reader(str(h5_path))
+ rna = "RNA"
+ assay = reader.assayFeats[rna]
+ start, end = int(assay["start"]), int(assay["end"])
+ feat_ids = reader.feature_ids(rna)
+ feat_names = reader.feature_names(rna)
+ feat_types = reader.feature_types()
+ cells = reader.cell_names()
+
+ with tempfile.TemporaryDirectory(prefix="scarf_mtx_fixture_") as tmp:
+ mtx_dir = Path(tmp) / "1K_pbmc_citeseq_dir"
+ mtx_dir.mkdir()
+
+ with gzip.open(mtx_dir / "features.tsv.gz", "wt") as handle:
+ for idx in range(start, end):
+ handle.write(
+ f"{feat_ids[idx - start]}\t{feat_names[idx - start]}\t{feat_types[idx]}\n"
+ )
+
+ with gzip.open(mtx_dir / "barcodes.tsv.gz", "wt") as handle:
+ for cell in cells:
+ handle.write(f"{cell}\n")
+
+ with h5py.File(h5_path) as h5:
+ matrix = h5["matrix"]
+ mat = csr_matrix(
+ (matrix["data"][:], matrix["indices"][:], matrix["indptr"][:]),
+ shape=(len(cells), reader.nFeatures),
+ )
+ rna_mat = mat[:, start:end].T.tocoo()
+
+ with gzip.open(mtx_dir / "matrix.mtx.gz", "wt") as handle:
+ handle.write(
+ "%%MatrixMarket matrix coordinate integer general\n% Generated by Scarf\n"
+ )
+ handle.write(f"{rna_mat.shape[0]} {rna_mat.shape[1]} {rna_mat.nnz}\n")
+ pd.DataFrame(
+ {"row": rna_mat.row + 1, "col": rna_mat.col + 1, "d": rna_mat.data}
+ ).to_csv(
+ handle,
+ sep=" ",
+ header=False,
+ index=False,
+ mode="a",
+ lineterminator="\n",
+ )
+
+ CrDirReader(str(mtx_dir))
+
+ with tarfile.open(archive, "w:gz") as tar:
+ tar.add(mtx_dir, arcname=mtx_dir.name)
+
+
+def download_optional_h5ad(*, attempts: int = _H5AD_DOWNLOAD_ATTEMPTS) -> bool:
+ sample = "bastidas-ponce_4K_pancreas-d15_rnaseq"
+ local_h5ad = datasets_dir() / sample / "data.h5ad"
+ if local_h5ad.is_file():
+ return True
+
+ from scarf import cytebase
+
+ last_error: BaseException | None = None
+ for attempt in range(1, attempts + 1):
+ try:
+ cytebase.connect("scarf_docs").download_dataset(
+ sample,
+ destination=datasets_dir(),
+ )
+ if local_h5ad.is_file():
+ return True
+ last_error = FileNotFoundError(
+ f"Cytebase download finished without creating {local_h5ad}"
+ )
+ except Exception as exc:
+ last_error = exc
+
+ if attempt >= attempts:
+ break
+ delay = min(60.0, 2.0**attempt)
+ print(
+ f"Warning: h5ad download attempt {attempt}/{attempts} failed "
+ f"({last_error}); retrying in {delay:.0f}s",
+ file=sys.stderr,
+ )
+ time.sleep(delay)
+
+ print(
+ f"Warning: skipping optional h5ad fixture after {attempts} attempts "
+ f"({last_error}). Tests that need {local_h5ad} will be skipped.",
+ file=sys.stderr,
+ )
+ return False
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument(
+ "--force",
+ action="store_true",
+ help="Re-download fixtures even when files already exist.",
+ )
+ parser.add_argument(
+ "--with-h5ad",
+ action="store_true",
+ help="Also download the bastidas-ponce h5ad from Cytebase.",
+ )
+ args = parser.parse_args(argv)
+
+ download_fixtures(force=args.force)
+ build_mtx_dir_fixture()
+ if args.with_h5ad:
+ download_optional_h5ad()
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tests/fixtures_datastore.py b/tests/fixtures_datastore.py
new file mode 100644
index 00000000..418c00fc
--- /dev/null
+++ b/tests/fixtures_datastore.py
@@ -0,0 +1,417 @@
+import os
+import shutil
+import tempfile
+
+import numpy as np
+import pandas as pd
+import pytest
+
+from . import full_path, remove, dask_total_sum
+
+
+def build_neighbourhood_graph(
+ datastore,
+ *,
+ from_assay: str | None = None,
+ cell_key: str = "I",
+ feat_key: str,
+ reduction_method: str = "pca",
+ dims: int = 11,
+ pca_cell_key: str | None = None,
+ k: int = 11,
+ ann_metric: str = "l2",
+ ann_efc: int | None = None,
+ ann_ef: int | None = None,
+ ann_m: int | None = None,
+ ann_parallel: bool = False,
+ rand_state: int = 4466,
+ n_centroids: int = 1000,
+ batch_size: int | None = None,
+ log_transform: bool | None = None,
+ renormalize_subset: bool | None = None,
+ local_connectivity: float = 1.0,
+ bandwidth: float = 1.5,
+ feat_scaling: bool = True,
+ lsi_skip_first: bool = True,
+ harmonize: bool = False,
+ batch_columns: list[str] | None = None,
+ harmony_params: dict | None = None,
+ local_cache: bool | str = "auto",
+ update_state: bool = True,
+ invalidate_cache: bool = False,
+):
+ normalized = datastore.run_normalization(
+ from_assay=from_assay,
+ cell_key=cell_key,
+ feat_key=feat_key,
+ log_transform=log_transform,
+ renormalize_subset=renormalize_subset,
+ update_state=False,
+ invalidate_cache=invalidate_cache,
+ )
+ if reduction_method == "lsi":
+ reduction = datastore.run_lsi(
+ normalized,
+ dims=dims,
+ skip_first=lsi_skip_first,
+ batch_size=batch_size,
+ local_cache=local_cache,
+ update_state=False,
+ invalidate_cache=invalidate_cache,
+ )
+ elif reduction_method == "pca":
+ reduction = datastore.run_pca(
+ normalized,
+ dims=dims,
+ pca_cell_key=pca_cell_key or cell_key,
+ feat_scaling=feat_scaling,
+ batch_size=batch_size,
+ local_cache=local_cache,
+ update_state=False,
+ invalidate_cache=invalidate_cache,
+ )
+ else:
+ raise ValueError(f"Unsupported test reduction method: {reduction_method}")
+ coordinates = reduction
+ if harmonize:
+ coordinates = datastore.run_harmony(
+ batch_columns or [],
+ reduction,
+ harmony_params=harmony_params,
+ batch_size=batch_size,
+ update_state=False,
+ invalidate_cache=invalidate_cache,
+ )
+ effective_ann_efc = ann_efc or min(100, max(k * 3, 50))
+ effective_ann_ef = ann_ef or min(100, max(k * 3, 50))
+ effective_ann_m = ann_m or min(max(48, int(dims * 1.5)), 64)
+ ann_index = datastore.build_ann_index(
+ coordinates,
+ ann_metric=ann_metric,
+ ann_efc=effective_ann_efc,
+ ann_ef=effective_ann_ef,
+ ann_m=effective_ann_m,
+ ann_parallel=ann_parallel,
+ rand_state=rand_state,
+ batch_size=batch_size,
+ update_state=False,
+ invalidate_cache=invalidate_cache,
+ )
+ datastore.build_embedding_initialization(
+ reduction,
+ n_centroids=n_centroids,
+ rand_state=rand_state,
+ batch_size=batch_size,
+ update_state=update_state,
+ invalidate_cache=invalidate_cache,
+ )
+ neighbors = datastore.query_neighbors(
+ ann_index,
+ coordinates=coordinates,
+ k=k,
+ batch_size=batch_size,
+ update_state=False,
+ invalidate_cache=invalidate_cache,
+ )
+ connectivity = datastore.build_connectivity_map(
+ neighbors,
+ local_connectivity=local_connectivity,
+ bandwidth=bandwidth,
+ update_state=update_state,
+ invalidate_cache=invalidate_cache,
+ )
+ return connectivity
+
+
+def _extract_zarr_fixture(tar_path: str, prefix: str) -> tuple[str, str]:
+ import tarfile
+
+ temp_dir = tempfile.mkdtemp(prefix=prefix)
+ with tarfile.open(tar_path, "r:gz") as tar:
+ tar.extractall(temp_dir, filter="data")
+ if os.path.isfile(os.path.join(temp_dir, ".zgroup")) or os.path.isfile(
+ os.path.join(temp_dir, "zarr.json")
+ ):
+ return temp_dir, temp_dir
+ for name in sorted(os.listdir(temp_dir)):
+ candidate = os.path.join(temp_dir, name)
+ if os.path.isdir(candidate) and (
+ os.path.isfile(os.path.join(candidate, ".zgroup"))
+ or os.path.isfile(os.path.join(candidate, "zarr.json"))
+ ):
+ return temp_dir, candidate
+ return temp_dir, temp_dir
+
+
+def _datastore_tar_path() -> str:
+ return full_path("1K_pbmc_citeseq.zarr.tar.gz")
+
+
+def _has_graph(datastore) -> bool:
+ try:
+ datastore.get_latest_graph_loc(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ )
+ return True
+ except KeyError:
+ return False
+
+
+def _cell_has(datastore, column: str) -> bool:
+ return column in datastore.cells.columns
+
+
+@pytest.fixture(scope="session")
+def toy_crdir_writer(toy_crdir_reader, tmp_path_factory):
+ from scarf.writers import CrToZarr
+
+ out_fn = tmp_path_factory.mktemp("toy_crdir") / "toy_crdir.zarr"
+ writer = CrToZarr(toy_crdir_reader, str(out_fn))
+ writer.dump()
+ yield str(out_fn)
+
+
+@pytest.fixture(scope="session")
+def toy_crdir_ds(toy_crdir_writer):
+ from scarf.datastore.datastore import DataStore
+
+ yield DataStore(toy_crdir_writer, default_assay="RNA")
+
+
+@pytest.fixture(scope="session")
+def datastore_zarr_root():
+ temp_dir, zarr_root = _extract_zarr_fixture(
+ _datastore_tar_path(), "scarf_session_1K_pbmc_"
+ )
+ yield zarr_root
+ remove(temp_dir)
+
+
+@pytest.fixture(scope="session")
+def datastore(datastore_zarr_root):
+ from scarf.datastore.datastore import DataStore
+
+ temp_dir = tempfile.mkdtemp(prefix="scarf_session_working_1K_pbmc_")
+ shutil.copytree(datastore_zarr_root, temp_dir, dirs_exist_ok=True)
+ yield DataStore(temp_dir, default_assay="RNA")
+ remove(temp_dir)
+
+
+@pytest.fixture(scope="session")
+def rna_raw_total(datastore):
+ return dask_total_sum(datastore.RNA.rawData)
+
+
+@pytest.fixture(scope="session")
+def assay2_raw_total(datastore):
+ return dask_total_sum(datastore.assay2.rawData)
+
+
+@pytest.fixture
+def datastore_ephemeral(datastore_zarr_root):
+ from scarf.datastore.datastore import DataStore
+
+ temp_dir = tempfile.mkdtemp(prefix="scarf_ephemeral_1K_pbmc_")
+ shutil.copytree(datastore_zarr_root, temp_dir, dirs_exist_ok=True)
+ yield DataStore(temp_dir, default_assay="RNA")
+ remove(temp_dir)
+
+
+@pytest.fixture(scope="session")
+def analyzed_datastore_zarr_root(datastore_zarr_root, tmp_path_factory):
+ from scarf.datastore.datastore import DataStore
+
+ zarr_root = tmp_path_factory.mktemp("scarf_analyzed_1K_pbmc_") / "data.zarr"
+ shutil.copytree(datastore_zarr_root, zarr_root)
+ datastore = DataStore(str(zarr_root), default_assay="RNA")
+ datastore.auto_filter_cells(show_qc_plots=False)
+ datastore.mark_hvgs(
+ top_n=100,
+ show_plot=False,
+ bin_strategy="fixed",
+ min_cells=int(0.01 * datastore.cells.N),
+ max_cells=np.inf,
+ blacklist="^MT-|^RPS|^RPL|^MRPS|^MRPL|^CCN|^HLA-|^H2-|^HIST",
+ )
+ build_neighbourhood_graph(
+ datastore,
+ feat_key="hvgs",
+ local_cache=False,
+ )
+ state = datastore.get_assay_state("RNA")
+ assert state is not None and state.connectivity_map is not None
+ return str(zarr_root)
+
+
+@pytest.fixture
+def analyzed_datastore_ephemeral(analyzed_datastore_zarr_root):
+ from scarf.datastore.datastore import DataStore
+
+ temp_dir = tempfile.mkdtemp(prefix="scarf_ephemeral_analyzed_1K_pbmc_")
+ shutil.copytree(analyzed_datastore_zarr_root, temp_dir, dirs_exist_ok=True)
+ yield DataStore(temp_dir, default_assay="RNA")
+ remove(temp_dir)
+
+
+@pytest.fixture(scope="session")
+def auto_filter_cells(datastore):
+ datastore.auto_filter_cells(show_qc_plots=False)
+
+
+@pytest.fixture(scope="session")
+def mark_hvgs(auto_filter_cells, datastore):
+ datastore.mark_hvgs(
+ top_n=100,
+ show_plot=False,
+ bin_strategy="fixed",
+ min_cells=int(0.01 * datastore.cells.N),
+ max_cells=np.inf,
+ blacklist="^MT-|^RPS|^RPL|^MRPS|^MRPL|^CCN|^HLA-|^H2-|^HIST",
+ )
+
+
+@pytest.fixture(scope="session")
+def graph_artifacts(mark_hvgs, datastore):
+ build_neighbourhood_graph(datastore, feat_key="hvgs")
+ state = datastore.get_assay_state("RNA")
+ assert state is not None and state.neighbors is not None
+ yield datastore.inspect_artifact(state.neighbors).path
+
+
+@pytest.fixture(scope="session")
+def leiden_clustering(graph_artifacts, datastore):
+ if not _cell_has(datastore, "RNA_leiden_cluster"):
+ datastore.run_leiden_clustering()
+ yield datastore.cells.fetch("RNA_leiden_cluster")
+
+
+@pytest.fixture(scope="session")
+def paris_clustering(graph_artifacts, datastore):
+ if not _cell_has(datastore, "RNA_cluster"):
+ datastore.run_paris_clustering(n_clusters=10, label="cluster")
+ yield datastore.cells.fetch("RNA_cluster")
+
+
+@pytest.fixture(scope="session")
+def paris_clustering_auto(graph_artifacts, datastore):
+ if not _cell_has(datastore, "RNA_adaptive_clusters"):
+ datastore.run_paris_clustering(
+ n_clusters="auto",
+ min_cluster_size=10,
+ label="adaptive_clusters",
+ )
+ yield datastore.cells.fetch("RNA_adaptive_clusters")
+
+
+@pytest.fixture(scope="session")
+def umap(graph_artifacts, datastore):
+ if not _cell_has(datastore, "RNA_UMAP1"):
+ datastore.run_umap(n_epochs=50)
+ yield np.array(
+ [datastore.cells.fetch("RNA_UMAP1"), datastore.cells.fetch("RNA_UMAP2")]
+ ).T
+
+
+@pytest.fixture(scope="session")
+def marker_search(datastore, paris_clustering):
+ if (
+ "markers" not in datastore.z["RNA"]
+ or "I__RNA_cluster" not in datastore.z["RNA"]["markers"]
+ ):
+ datastore.run_marker_search(group_key="RNA_cluster")
+
+
+@pytest.fixture(scope="session")
+def pseudotime_scoring(datastore, leiden_clustering):
+ if not _cell_has(datastore, "RNA_pseudotime"):
+ datastore.run_pseudotime_scoring(
+ source_sink_key="RNA_leiden_cluster", sources=[6], sinks=[3]
+ )
+ yield datastore.cells.fetch("RNA_pseudotime")
+
+
+@pytest.fixture(scope="session")
+def pseudotime_markers(datastore, pseudotime_scoring):
+ marker_key = "I__RNA_pseudotime__r"
+ if (
+ marker_key not in datastore.RNA.feats.columns
+ or "source_artifact" not in datastore.RNA.z["featureData"][marker_key].attrs
+ ):
+ datastore.run_pseudotime_marker_search(pseudotime_key="RNA_pseudotime")
+ df = datastore.RNA.feats.to_pandas_dataframe(
+ ["names", "I__RNA_pseudotime__r"], key="I"
+ )
+ yield df
+
+
+@pytest.fixture(scope="session")
+def pseudotime_aggregation(datastore, pseudotime_scoring):
+ result = datastore.run_pseudotime_aggregation(
+ pseudotime_key="RNA_pseudotime",
+ cluster_label="pseudotime_clusters",
+ n_clusters=15,
+ window_size=50,
+ chunk_size=10,
+ )
+ yield result
+
+
+@pytest.fixture(scope="session")
+def grouped_assay(datastore, pseudotime_aggregation):
+ datastore.add_grouped_assay(
+ group_key="pseudotime_clusters", assay_label="PTIME_MODULES"
+ )
+
+
+@pytest.fixture(scope="session")
+def cell_cycle_scoring(auto_filter_cells, datastore):
+ del auto_filter_cells
+ if not _cell_has(datastore, "RNA_cell_cycle_phase"):
+ datastore.run_cell_cycle_scoring()
+ return datastore.cells.fetch("RNA_cell_cycle_phase")
+
+
+@pytest.fixture(scope="session")
+def topacedo_sampler(paris_clustering, datastore):
+ import importlib.util
+
+ if importlib.util.find_spec("topacedo") is None:
+ pytest.skip("topacedo package not installed")
+ if not _cell_has(datastore, "RNA_sketched"):
+ datastore.run_topacedo_sampler(cluster_key="RNA_cluster")
+ return datastore.cells.fetch("RNA_sketched")
+
+
+@pytest.fixture(scope="session")
+def cell_attrs():
+ return pd.read_csv(full_path("cell_attributes.csv"), index_col=0)
+
+
+@pytest.fixture(scope="session")
+def atac_datastore():
+ from scarf.datastore.datastore import DataStore
+
+ fn = full_path("500_pbmc_atac.zarr.tar.gz")
+ temp_dir, zarr_root = _extract_zarr_fixture(fn, "scarf_session_atac_")
+ yield DataStore(zarr_root)
+ remove(temp_dir)
+
+
+@pytest.fixture(scope="session")
+def mark_prevalent_peaks(atac_datastore):
+ atac_datastore.mark_prevalent_peaks(top_n=5000)
+
+
+@pytest.fixture(scope="session")
+def make_atac_graph(mark_prevalent_peaks, atac_datastore):
+ build_neighbourhood_graph(
+ atac_datastore,
+ feat_key="prevalent_peaks",
+ reduction_method="lsi",
+ feat_scaling=False,
+ )
+ state = atac_datastore.get_assay_state("ATAC")
+ assert state is not None and state.neighbors is not None
+ yield atac_datastore.inspect_artifact(state.neighbors).path
diff --git a/tests/fixtures_downloader.py b/tests/fixtures_downloader.py
new file mode 100644
index 00000000..afa2e79f
--- /dev/null
+++ b/tests/fixtures_downloader.py
@@ -0,0 +1,18 @@
+import os
+
+import pytest
+
+from . import full_path
+
+
+@pytest.fixture(scope="session")
+def bastidas_ponce_data():
+ sample = "bastidas-ponce_4K_pancreas-d15_rnaseq"
+ local_h5ad = full_path(sample, "data.h5ad")
+ if not os.path.isfile(local_h5ad):
+ pytest.skip(
+ f"Bundled test data missing at {local_h5ad}. "
+ "Download via `python -m tests.download_fixtures --with-h5ad` "
+ "(transient Hugging Face failures are skipped after retries)."
+ )
+ yield local_h5ad
diff --git a/tests/fixtures_readers.py b/tests/fixtures_readers.py
new file mode 100644
index 00000000..fd16f412
--- /dev/null
+++ b/tests/fixtures_readers.py
@@ -0,0 +1,80 @@
+import os
+import tarfile
+
+import pytest
+
+from . import full_path
+
+
+@pytest.fixture(scope="session")
+def toy_crdir_reader(tmp_path_factory):
+ from scarf.readers import CrDirReader
+
+ out_fn = tmp_path_factory.mktemp("toy_cr_dir")
+ with tarfile.open(full_path("toy_cr_dir.tar.gz"), "r:gz") as tar:
+ tar.extractall(out_fn, filter="data")
+ reader = CrDirReader(str(out_fn))
+ reader.rename_assays({"ASSAY4": "HTO"})
+ yield reader
+
+
+@pytest.fixture(scope="session")
+def toy_crdir_empty(tmp_path_factory):
+ from scarf.readers import CrDirReader
+
+ out_fn = tmp_path_factory.mktemp("toy_cr_dir_empty")
+ with tarfile.open(full_path("toy_cr_dir_empty.tar.gz"), "r:gz") as tar:
+ tar.extractall(out_fn, filter="data")
+ reader = CrDirReader(str(out_fn))
+ yield reader
+
+
+@pytest.fixture(scope="session")
+def crh5_reader():
+ from scarf.readers import CrH5Reader
+
+ return CrH5Reader(full_path("1K_pbmc_citeseq.h5"))
+
+
+@pytest.fixture(scope="session")
+def mtx_dir(tmp_path_factory):
+ fn = full_path("1K_pbmc_citeseq_dir.tar.gz")
+ if not os.path.isfile(fn):
+ pytest.skip(
+ f"Bundled MTX fixture missing at {fn}. "
+ "Add 1K_pbmc_citeseq_dir.tar.gz under tests/datasets/."
+ )
+ base = tmp_path_factory.mktemp("mtx_dir")
+ with tarfile.open(fn, "r:gz") as tar:
+ tar.extractall(base, filter="data")
+ yield str(base / "1K_pbmc_citeseq_dir")
+
+
+@pytest.fixture(scope="session")
+def crdir_reader(mtx_dir):
+ from scarf.readers import CrDirReader
+
+ reader = CrDirReader(mtx_dir)
+ yield reader
+
+
+@pytest.fixture(scope="session")
+def h5ad_reader(bastidas_ponce_data):
+ from scarf.readers import H5adReader
+
+ reader = H5adReader(bastidas_ponce_data)
+ yield reader
+ reader.h5.close()
+
+
+@pytest.fixture(scope="session")
+def loom_reader():
+ from scarf.readers import LoomReader
+
+ reader = LoomReader(
+ full_path("sympathetic.loom"),
+ cell_names_key="Cell_id",
+ feature_names_key="Gene",
+ )
+ yield reader
+ reader.h5.close()
diff --git a/tests/paris_quality_gate.py b/tests/paris_quality_gate.py
new file mode 100644
index 00000000..8f1ae5ca
--- /dev/null
+++ b/tests/paris_quality_gate.py
@@ -0,0 +1,1230 @@
+import argparse
+import json
+import math
+from dataclasses import asdict, dataclass
+
+import numpy as np
+from scipy.sparse import csr_matrix, triu
+from sklearn.metrics import adjusted_rand_score
+
+from scarf.clustering._paris_core import ParisHierarchy
+from scarf.clustering._paris_modularity import modularity_split_gains
+from scarf.clustering.paris import (
+ fit_paris_hierarchy,
+ hierarchy_to_dendrogram,
+ straight_cut,
+)
+from scarf.clustering.paris_multiscale import (
+ PlateauForest,
+ adaptive_cut,
+ collapse_equal_height_plateaus,
+)
+
+DEFAULT_SEED_COUNT = 8
+MIN_RARE_CLUSTER_F1 = 0.9
+MAX_ADAPTIVE_CLUSTERS = 5
+MIN_ACCEPTED_SEED_FRACTION = 0.8
+CPM_RESOLUTIONS = tuple(np.geomspace(0.002, 0.2, 12))
+# The guarded cut must recover structure as well as the oracle horizontal cut
+# that scans every cluster count, otherwise it has silently degraded.
+ORACLE_ARI_TOLERANCE = 0.05
+PAIR_F1_ORACLE_TOLERANCE = 1e-12
+MAX_DINKELBACH_ITERATIONS = 100
+MAX_EXHAUSTIVE_ORACLE_LEAVES = 20
+SUPPORTED_ALPHA_BETA_EXPONENTS = tuple(range(-12, 13))
+SUPPORTED_ALPHA_BETA_RATIOS = tuple(
+ 2.0**exponent for exponent in SUPPORTED_ALPHA_BETA_EXPONENTS
+)
+
+
+@dataclass(frozen=True, slots=True)
+class PairMetrics:
+ true_positive_pairs: int
+ false_positive_pairs: int
+ false_negative_pairs: int
+ true_negative_pairs: int
+ precision: float
+ recall: float
+ f1: float
+
+ @property
+ def predicted_positive_pairs(self) -> int:
+ return self.true_positive_pairs + self.false_positive_pairs
+
+ @property
+ def truth_positive_pairs(self) -> int:
+ return self.true_positive_pairs + self.false_negative_pairs
+
+ @property
+ def total_pairs(self) -> int:
+ return (
+ self.true_positive_pairs
+ + self.false_positive_pairs
+ + self.false_negative_pairs
+ + self.true_negative_pairs
+ )
+
+
+@dataclass(frozen=True, slots=True)
+class PairCut:
+ selected_nodes: tuple[int, ...]
+ metrics: PairMetrics
+
+ @property
+ def n_clusters(self) -> int:
+ return len(self.selected_nodes)
+
+
+@dataclass(frozen=True, slots=True)
+class PairF1OracleResult:
+ cut: PairCut
+ iterations: int
+ tolerance: float
+ residual: float
+
+
+@dataclass(frozen=True, slots=True)
+class PairErrorOracleResult:
+ alpha: int | float
+ beta: int | float
+ weighted_error: int | float
+ cut: PairCut
+
+ @property
+ def alpha_beta_ratio(self) -> float:
+ return float(self.alpha) / float(self.beta)
+
+
+@dataclass(frozen=True, slots=True)
+class HierarchyHeadroomEvaluation:
+ min_cluster_size: int
+ baseline: PairMetrics
+ maximum_pair_f1: PairF1OracleResult
+ supported_cuts: tuple[PairErrorOracleResult, ...]
+ supported_frontier: tuple[PairErrorOracleResult, ...]
+ pair_f1_headroom: float
+ has_measurable_headroom: bool
+
+
+@dataclass(frozen=True, slots=True)
+class _EventPairStatistics:
+ same_pairs: tuple[int, ...]
+ cross_pairs: tuple[int, ...]
+ all_pairs: tuple[int, ...]
+ truth_pairs: int
+ total_pairs: int
+ evaluated_items: int
+
+
+def _choose_two(size: int) -> int:
+ return size * (size - 1) // 2
+
+
+def _label_array(values: np.ndarray, name: str) -> np.ndarray:
+ labels = np.asarray(values)
+ if labels.ndim != 1:
+ raise ValueError(f"{name} must be one-dimensional")
+ if labels.size == 0:
+ raise ValueError(f"{name} must not be empty")
+ return labels
+
+
+def _normalized_label_strings(values: np.ndarray) -> np.ndarray:
+ normalized = np.empty(values.size, dtype=object)
+ for index, raw_value in enumerate(values):
+ value = raw_value.item() if isinstance(raw_value, np.generic) else raw_value
+ normalized[index] = str(value)
+ return normalized.astype(str)
+
+
+def _encoded_labels(values: np.ndarray) -> np.ndarray:
+ labels = _label_array(values, "labels")
+ normalized = (
+ _normalized_label_strings(labels)
+ if labels.dtype.kind == "O"
+ else labels.astype(str)
+ )
+ _unique, codes = np.unique(normalized, return_inverse=True)
+ return codes.astype(np.int32, copy=False)
+
+
+def _encoded_truth(values: np.ndarray) -> tuple[np.ndarray, int]:
+ truth = _label_array(values, "truth")
+ valid = np.ones(truth.size, dtype=bool)
+ normalized = np.empty(truth.size, dtype=object)
+ for index, raw_value in enumerate(truth):
+ value = raw_value.item() if isinstance(raw_value, np.generic) else raw_value
+ missing = value is None
+ if not missing:
+ try:
+ missing = bool(value != value)
+ except (TypeError, ValueError):
+ missing = False
+ label = "" if missing else str(value).strip()
+ if missing or not label or label.casefold() == "unknown":
+ valid[index] = False
+ normalized[index] = ""
+ else:
+ normalized[index] = label
+
+ codes = np.full(truth.size, -1, dtype=np.int32)
+ if np.any(valid):
+ _unique, valid_codes = np.unique(
+ normalized[valid].astype(str),
+ return_inverse=True,
+ )
+ codes[valid] = valid_codes.astype(np.int32, copy=False)
+ return codes, int(np.count_nonzero(valid))
+
+
+def _pair_metrics_from_counts(
+ *,
+ n_items: int,
+ true_positive_pairs: int,
+ predicted_positive_pairs: int,
+ truth_positive_pairs: int,
+) -> PairMetrics:
+ false_positive_pairs = predicted_positive_pairs - true_positive_pairs
+ false_negative_pairs = truth_positive_pairs - true_positive_pairs
+ true_negative_pairs = (
+ _choose_two(n_items)
+ - true_positive_pairs
+ - false_positive_pairs
+ - false_negative_pairs
+ )
+ if (
+ min(
+ true_positive_pairs,
+ false_positive_pairs,
+ false_negative_pairs,
+ true_negative_pairs,
+ )
+ < 0
+ ):
+ raise ValueError("pair counts are inconsistent")
+ precision = (
+ true_positive_pairs / predicted_positive_pairs
+ if predicted_positive_pairs
+ else 0.0
+ )
+ recall = true_positive_pairs / truth_positive_pairs if truth_positive_pairs else 0.0
+ denominator = predicted_positive_pairs + truth_positive_pairs
+ f1 = 2 * true_positive_pairs / denominator if denominator else 0.0
+ return PairMetrics(
+ true_positive_pairs=true_positive_pairs,
+ false_positive_pairs=false_positive_pairs,
+ false_negative_pairs=false_negative_pairs,
+ true_negative_pairs=true_negative_pairs,
+ precision=precision,
+ recall=recall,
+ f1=f1,
+ )
+
+
+def exact_pair_metrics(labels: np.ndarray, truth: np.ndarray) -> PairMetrics:
+ predicted = _label_array(labels, "labels")
+ expected = _label_array(truth, "truth")
+ if predicted.size != expected.size:
+ raise ValueError("labels and truth must have the same length")
+
+ truth_codes, evaluated_items = _encoded_truth(expected)
+ valid = truth_codes >= 0
+ if evaluated_items == 0:
+ return _pair_metrics_from_counts(
+ n_items=0,
+ true_positive_pairs=0,
+ predicted_positive_pairs=0,
+ truth_positive_pairs=0,
+ )
+ predicted_codes = _encoded_labels(predicted[valid])
+ valid_truth_codes = truth_codes[valid]
+ predicted_counts = np.bincount(predicted_codes).astype(np.int64, copy=False)
+ truth_counts = np.bincount(valid_truth_codes).astype(np.int64, copy=False)
+ n_predicted = max(1, predicted_counts.size)
+ joint_keys = valid_truth_codes.astype(
+ np.int64
+ ) * n_predicted + predicted_codes.astype(np.int64)
+ _joint_values, joint_counts = np.unique(joint_keys, return_counts=True)
+ true_positive_pairs = int(
+ np.sum(joint_counts * (joint_counts - 1) // 2, dtype=np.int64)
+ )
+ predicted_positive_pairs = int(
+ np.sum(predicted_counts * (predicted_counts - 1) // 2, dtype=np.int64)
+ )
+ truth_positive_pairs = int(
+ np.sum(truth_counts * (truth_counts - 1) // 2, dtype=np.int64)
+ )
+ return _pair_metrics_from_counts(
+ n_items=evaluated_items,
+ true_positive_pairs=true_positive_pairs,
+ predicted_positive_pairs=predicted_positive_pairs,
+ truth_positive_pairs=truth_positive_pairs,
+ )
+
+
+def _validate_min_cluster_size(min_cluster_size: int) -> None:
+ if isinstance(min_cluster_size, bool) or not isinstance(min_cluster_size, int):
+ raise TypeError("min_cluster_size must be an integer")
+ if min_cluster_size < 2:
+ raise ValueError("min_cluster_size must be at least 2")
+
+
+def _event_pair_statistics(
+ forest: PlateauForest,
+ truth: np.ndarray,
+) -> _EventPairStatistics:
+ expected = _label_array(truth, "truth")
+ if expected.size != forest.n_leaves:
+ raise ValueError("truth and plateau forest have different leaf counts")
+ truth_codes, evaluated_items = _encoded_truth(expected)
+ n_classes = int(truth_codes.max()) + 1 if evaluated_items else 0
+ dense_bytes = (
+ int(forest.representatives.size) * n_classes * np.dtype(np.int64).itemsize
+ )
+ if dense_bytes < 64 * 1024**2:
+ return _dense_event_pair_statistics(
+ forest,
+ truth_codes,
+ evaluated_items,
+ n_classes,
+ )
+ return _sparse_event_pair_statistics(
+ forest,
+ truth_codes,
+ evaluated_items,
+ )
+
+
+def _statistics_from_event_counts(
+ event_counts: np.ndarray,
+ evaluated_items: int,
+) -> _EventPairStatistics:
+ event_sizes = event_counts.sum(axis=1, dtype=np.int64)
+ same_pairs_array = np.sum(
+ event_counts * (event_counts - 1) // 2,
+ axis=1,
+ dtype=np.int64,
+ )
+ all_pairs_array = event_sizes * (event_sizes - 1) // 2
+ truth_counts = event_counts[-1] if event_counts.shape[0] else np.empty(0)
+ return _EventPairStatistics(
+ same_pairs=tuple(int(value) for value in same_pairs_array),
+ cross_pairs=tuple(int(value) for value in all_pairs_array - same_pairs_array),
+ all_pairs=tuple(int(value) for value in all_pairs_array),
+ truth_pairs=int(np.sum(truth_counts * (truth_counts - 1) // 2, dtype=np.int64)),
+ total_pairs=_choose_two(evaluated_items),
+ evaluated_items=evaluated_items,
+ )
+
+
+def _dense_event_pair_statistics(
+ forest: PlateauForest,
+ truth_codes: np.ndarray,
+ evaluated_items: int,
+ n_classes: int,
+) -> _EventPairStatistics:
+ counts = np.zeros(
+ (forest.representatives.size, n_classes),
+ dtype=np.int64,
+ )
+ for event in range(forest.representatives.size):
+ for offset in range(
+ int(forest.child_offsets[event]),
+ int(forest.child_offsets[event + 1]),
+ ):
+ child_ref = int(forest.child_refs[offset])
+ if child_ref < 0:
+ leaf = -child_ref - 1
+ if leaf < 0 or leaf >= forest.n_leaves:
+ raise ValueError(
+ "plateau forest contains an invalid leaf reference"
+ )
+ code = int(truth_codes[leaf])
+ if code >= 0:
+ counts[event, code] += 1
+ else:
+ if child_ref >= event:
+ raise ValueError("plateau forest events must be in postorder")
+ counts[event] += counts[child_ref]
+ root_counts = np.zeros(n_classes, dtype=np.int64)
+ for root_ref_value in forest.component_roots:
+ root_ref = int(root_ref_value)
+ if root_ref >= 0:
+ root_counts += counts[root_ref]
+ else:
+ code = int(truth_codes[-root_ref - 1])
+ if code >= 0:
+ root_counts[code] += 1
+ statistics = _statistics_from_event_counts(counts, evaluated_items)
+ return _EventPairStatistics(
+ same_pairs=statistics.same_pairs,
+ cross_pairs=statistics.cross_pairs,
+ all_pairs=statistics.all_pairs,
+ truth_pairs=int(np.sum(root_counts * (root_counts - 1) // 2, dtype=np.int64)),
+ total_pairs=statistics.total_pairs,
+ evaluated_items=evaluated_items,
+ )
+
+
+def _merge_sparse_counts(
+ left_ids: np.ndarray,
+ left_counts: np.ndarray,
+ right_ids: np.ndarray,
+ right_counts: np.ndarray,
+) -> tuple[np.ndarray, np.ndarray]:
+ merged_ids = np.empty(left_ids.size + right_ids.size, dtype=np.int32)
+ merged_counts = np.empty(left_ids.size + right_ids.size, dtype=np.int64)
+ left = 0
+ right = 0
+ output = 0
+ while left < left_ids.size and right < right_ids.size:
+ left_id = int(left_ids[left])
+ right_id = int(right_ids[right])
+ if left_id < right_id:
+ merged_ids[output] = left_id
+ merged_counts[output] = left_counts[left]
+ left += 1
+ elif right_id < left_id:
+ merged_ids[output] = right_id
+ merged_counts[output] = right_counts[right]
+ right += 1
+ else:
+ merged_ids[output] = left_id
+ merged_counts[output] = left_counts[left] + right_counts[right]
+ left += 1
+ right += 1
+ output += 1
+ if left < left_ids.size:
+ count = left_ids.size - left
+ merged_ids[output : output + count] = left_ids[left:]
+ merged_counts[output : output + count] = left_counts[left:]
+ output += count
+ if right < right_ids.size:
+ count = right_ids.size - right
+ merged_ids[output : output + count] = right_ids[right:]
+ merged_counts[output : output + count] = right_counts[right:]
+ output += count
+ return merged_ids[:output], merged_counts[:output]
+
+
+def _sparse_event_pair_statistics(
+ forest: PlateauForest,
+ truth_codes: np.ndarray,
+ evaluated_items: int,
+) -> _EventPairStatistics:
+ event_ids: list[np.ndarray | None] = []
+ event_counts: list[np.ndarray | None] = []
+ same_pairs: list[int] = []
+ cross_pairs: list[int] = []
+ all_pairs: list[int] = []
+
+ for event in range(forest.representatives.size):
+ child_events: list[int] = []
+ leaf_codes: list[int] = []
+ for offset in range(
+ int(forest.child_offsets[event]),
+ int(forest.child_offsets[event + 1]),
+ ):
+ child_ref = int(forest.child_refs[offset])
+ if child_ref < 0:
+ leaf = -child_ref - 1
+ if leaf < 0 or leaf >= forest.n_leaves:
+ raise ValueError(
+ "plateau forest contains an invalid leaf reference"
+ )
+ code = int(truth_codes[leaf])
+ if code >= 0:
+ leaf_codes.append(code)
+ continue
+ if child_ref >= event:
+ raise ValueError("plateau forest events must be in postorder")
+ child_events.append(child_ref)
+
+ if child_events:
+ largest_child = max(
+ child_events,
+ key=lambda child: len(
+ event_ids[child] if event_ids[child] is not None else ()
+ ),
+ )
+ ids = event_ids[largest_child]
+ counts = event_counts[largest_child]
+ if ids is None or counts is None:
+ raise ValueError("plateau event has more than one parent")
+ event_ids[largest_child] = None
+ event_counts[largest_child] = None
+ for child in child_events:
+ if child == largest_child:
+ continue
+ child_ids = event_ids[child]
+ child_counts = event_counts[child]
+ if child_ids is None or child_counts is None:
+ raise ValueError("plateau event has more than one parent")
+ ids, counts = _merge_sparse_counts(
+ ids,
+ counts,
+ child_ids,
+ child_counts,
+ )
+ event_ids[child] = None
+ event_counts[child] = None
+ else:
+ ids = np.empty(0, dtype=np.int32)
+ counts = np.empty(0, dtype=np.int64)
+ if leaf_codes:
+ leaf_ids, leaf_counts = np.unique(
+ np.asarray(leaf_codes, dtype=np.int32),
+ return_counts=True,
+ )
+ ids, counts = _merge_sparse_counts(
+ ids,
+ counts,
+ leaf_ids,
+ leaf_counts.astype(np.int64, copy=False),
+ )
+ event_ids.append(ids)
+ event_counts.append(counts)
+ event_same_pairs = int(np.sum(counts * (counts - 1) // 2, dtype=np.int64))
+ size = int(counts.sum(dtype=np.int64))
+ event_all_pairs = _choose_two(size)
+ same_pairs.append(event_same_pairs)
+ cross_pairs.append(event_all_pairs - event_same_pairs)
+ all_pairs.append(event_all_pairs)
+
+ valid_truth = truth_codes[truth_codes >= 0]
+ truth_counts = np.bincount(valid_truth).astype(np.int64, copy=False)
+ return _EventPairStatistics(
+ same_pairs=tuple(same_pairs),
+ cross_pairs=tuple(cross_pairs),
+ all_pairs=tuple(all_pairs),
+ truth_pairs=int(np.sum(truth_counts * (truth_counts - 1) // 2, dtype=np.int64)),
+ total_pairs=_choose_two(evaluated_items),
+ evaluated_items=evaluated_items,
+ )
+
+
+def _selected_ref_key(forest: PlateauForest, selected_ref: int) -> int:
+ return (
+ -selected_ref - 1 if selected_ref < 0 else int(forest.min_leaves[selected_ref])
+ )
+
+
+def _selected_nodes(
+ forest: PlateauForest,
+ selected_refs: tuple[int, ...],
+) -> tuple[int, ...]:
+ ordered_refs = sorted(
+ selected_refs,
+ key=lambda selected_ref: _selected_ref_key(forest, selected_ref),
+ )
+ return tuple(
+ -selected_ref - 1
+ if selected_ref < 0
+ else int(forest.representatives[selected_ref])
+ for selected_ref in ordered_refs
+ )
+
+
+def _pair_cut_from_refs(
+ forest: PlateauForest,
+ statistics: _EventPairStatistics,
+ selected_refs: tuple[int, ...],
+) -> PairCut:
+ true_positive_pairs = sum(
+ statistics.same_pairs[selected_ref]
+ for selected_ref in selected_refs
+ if selected_ref >= 0
+ )
+ predicted_positive_pairs = sum(
+ statistics.all_pairs[selected_ref]
+ for selected_ref in selected_refs
+ if selected_ref >= 0
+ )
+ return PairCut(
+ selected_nodes=_selected_nodes(forest, selected_refs),
+ metrics=_pair_metrics_from_counts(
+ n_items=statistics.evaluated_items,
+ true_positive_pairs=true_positive_pairs,
+ predicted_positive_pairs=predicted_positive_pairs,
+ truth_positive_pairs=statistics.truth_pairs,
+ ),
+ )
+
+
+def _optimize_additive_antichain(
+ forest: PlateauForest,
+ node_values: tuple[int | float, ...],
+ min_cluster_size: int,
+ *,
+ maximize: bool,
+) -> tuple[int | float, tuple[int, ...]]:
+ _validate_min_cluster_size(min_cluster_size)
+ n_events = forest.representatives.size
+ if len(node_values) != n_events:
+ raise ValueError("node_values must have one value per plateau event")
+
+ best_values: list[int | float | None] = [None] * n_events
+ split_events = [False] * n_events
+ for event in range(n_events):
+ if int(forest.sizes[event]) < min_cluster_size:
+ continue
+ keep_value = node_values[event]
+ split_value: int | float = 0
+ split_feasible = True
+ for offset in range(
+ int(forest.child_offsets[event]),
+ int(forest.child_offsets[event + 1]),
+ ):
+ child_ref = int(forest.child_refs[offset])
+ if child_ref < 0 or best_values[child_ref] is None:
+ split_feasible = False
+ break
+ split_value += best_values[child_ref]
+
+ prefer_split = split_feasible and (
+ split_value > keep_value if maximize else split_value < keep_value
+ )
+ if prefer_split:
+ best_values[event] = split_value
+ split_events[event] = True
+ else:
+ best_values[event] = keep_value
+
+ total_value: int | float = 0
+ selected_refs: list[int] = []
+ stack: list[int] = []
+ for root_ref_value in forest.component_roots:
+ root_ref = int(root_ref_value)
+ if root_ref < 0:
+ selected_refs.append(root_ref)
+ continue
+ if root_ref >= n_events:
+ raise ValueError("plateau forest contains an invalid component root")
+ if best_values[root_ref] is None:
+ total_value += node_values[root_ref]
+ selected_refs.append(root_ref)
+ continue
+ total_value += best_values[root_ref]
+ stack.append(root_ref)
+ while stack:
+ event = stack.pop()
+ if not split_events[event]:
+ selected_refs.append(event)
+ continue
+ for offset in range(
+ int(forest.child_offsets[event + 1]) - 1,
+ int(forest.child_offsets[event]) - 1,
+ -1,
+ ):
+ child_ref = int(forest.child_refs[offset])
+ if child_ref < 0:
+ raise RuntimeError("a feasible oracle split contains a leaf")
+ stack.append(child_ref)
+
+ selected_refs.sort(key=lambda selected_ref: _selected_ref_key(forest, selected_ref))
+ return total_value, tuple(selected_refs)
+
+
+def _maximum_pair_f1_from_statistics(
+ forest: PlateauForest,
+ statistics: _EventPairStatistics,
+ min_cluster_size: int,
+ *,
+ tolerance: float,
+ max_iterations: int,
+ initial_numerator: int = 0,
+ initial_denominator: int = 1,
+) -> PairF1OracleResult:
+ if not math.isfinite(tolerance) or tolerance <= 0:
+ raise ValueError("tolerance must be a positive finite number")
+ if isinstance(max_iterations, bool) or not isinstance(max_iterations, int):
+ raise TypeError("max_iterations must be an integer")
+ if max_iterations < 1:
+ raise ValueError("max_iterations must be positive")
+ if initial_numerator < 0 or initial_denominator < 1:
+ raise ValueError("initial pair-F1 quotient is invalid")
+
+ quotient_numerator = initial_numerator
+ quotient_denominator = initial_denominator
+ for iteration in range(1, max_iterations + 1):
+ node_values = tuple(
+ 2 * same_pairs * quotient_denominator - quotient_numerator * all_pairs
+ for same_pairs, all_pairs in zip(
+ statistics.same_pairs,
+ statistics.all_pairs,
+ strict=True,
+ )
+ )
+ _objective, selected_refs = _optimize_additive_antichain(
+ forest,
+ node_values,
+ min_cluster_size,
+ maximize=True,
+ )
+ cut = _pair_cut_from_refs(forest, statistics, selected_refs)
+ numerator = 2 * cut.metrics.true_positive_pairs
+ denominator = cut.metrics.predicted_positive_pairs + statistics.truth_pairs
+ if denominator == 0:
+ return PairF1OracleResult(
+ cut=cut,
+ iterations=iteration,
+ tolerance=tolerance,
+ residual=0.0,
+ )
+ residual_numerator = (
+ numerator * quotient_denominator - quotient_numerator * denominator
+ )
+ residual = residual_numerator / quotient_denominator
+ if abs(residual) <= tolerance * denominator:
+ return PairF1OracleResult(
+ cut=cut,
+ iterations=iteration,
+ tolerance=tolerance,
+ residual=max(0.0, residual),
+ )
+ divisor = math.gcd(numerator, denominator)
+ quotient_numerator = numerator // divisor
+ quotient_denominator = denominator // divisor
+ raise RuntimeError("pair-F1 Dinkelbach iterations did not converge")
+
+
+def maximum_pair_f1_cut(
+ forest: PlateauForest,
+ truth: np.ndarray,
+ min_cluster_size: int = 2,
+ *,
+ tolerance: float = PAIR_F1_ORACLE_TOLERANCE,
+ max_iterations: int = MAX_DINKELBACH_ITERATIONS,
+) -> PairF1OracleResult:
+ statistics = _event_pair_statistics(forest, truth)
+ return _maximum_pair_f1_from_statistics(
+ forest,
+ statistics,
+ min_cluster_size,
+ tolerance=tolerance,
+ max_iterations=max_iterations,
+ )
+
+
+def _minimum_pair_error_from_statistics(
+ forest: PlateauForest,
+ statistics: _EventPairStatistics,
+ *,
+ alpha: int | float,
+ beta: int | float,
+ min_cluster_size: int,
+) -> PairErrorOracleResult:
+ for name, value in (("alpha", alpha), ("beta", beta)):
+ if isinstance(value, bool) or not isinstance(value, (int, float)):
+ raise TypeError(f"{name} must be numeric")
+ if not math.isfinite(float(value)) or value <= 0:
+ raise ValueError(f"{name} must be a positive finite number")
+ node_values = tuple(
+ alpha * cross_pairs - beta * same_pairs
+ for cross_pairs, same_pairs in zip(
+ statistics.cross_pairs,
+ statistics.same_pairs,
+ strict=True,
+ )
+ )
+ _objective, selected_refs = _optimize_additive_antichain(
+ forest,
+ node_values,
+ min_cluster_size,
+ maximize=False,
+ )
+ cut = _pair_cut_from_refs(forest, statistics, selected_refs)
+ weighted_error = (
+ alpha * cut.metrics.false_positive_pairs
+ + beta * cut.metrics.false_negative_pairs
+ )
+ return PairErrorOracleResult(
+ alpha=alpha,
+ beta=beta,
+ weighted_error=weighted_error,
+ cut=cut,
+ )
+
+
+def minimum_pair_error_cut(
+ forest: PlateauForest,
+ truth: np.ndarray,
+ *,
+ alpha: int | float,
+ beta: int | float,
+ min_cluster_size: int = 2,
+) -> PairErrorOracleResult:
+ statistics = _event_pair_statistics(forest, truth)
+ return _minimum_pair_error_from_statistics(
+ forest,
+ statistics,
+ alpha=alpha,
+ beta=beta,
+ min_cluster_size=min_cluster_size,
+ )
+
+
+def _supported_pair_cuts_from_statistics(
+ forest: PlateauForest,
+ statistics: _EventPairStatistics,
+ min_cluster_size: int,
+) -> tuple[PairErrorOracleResult, ...]:
+ cuts: list[PairErrorOracleResult] = []
+ for exponent in SUPPORTED_ALPHA_BETA_EXPONENTS:
+ if exponent < 0:
+ alpha, beta = 1, 1 << -exponent
+ else:
+ alpha, beta = 1 << exponent, 1
+ cuts.append(
+ _minimum_pair_error_from_statistics(
+ forest,
+ statistics,
+ alpha=alpha,
+ beta=beta,
+ min_cluster_size=min_cluster_size,
+ )
+ )
+ return tuple(cuts)
+
+
+def supported_pair_cuts(
+ forest: PlateauForest,
+ truth: np.ndarray,
+ min_cluster_size: int = 2,
+) -> tuple[PairErrorOracleResult, ...]:
+ statistics = _event_pair_statistics(forest, truth)
+ return _supported_pair_cuts_from_statistics(
+ forest,
+ statistics,
+ min_cluster_size,
+ )
+
+
+def nondominated_supported_cuts(
+ cuts: tuple[PairErrorOracleResult, ...],
+) -> tuple[PairErrorOracleResult, ...]:
+ frontier: list[PairErrorOracleResult] = []
+ seen_metrics: set[tuple[int, int, int]] = set()
+ for candidate in cuts:
+ metrics = candidate.cut.metrics
+ key = (
+ metrics.true_positive_pairs,
+ metrics.false_positive_pairs,
+ metrics.false_negative_pairs,
+ )
+ if key in seen_metrics:
+ continue
+ seen_metrics.add(key)
+ dominated = any(
+ other.cut.metrics.precision >= metrics.precision
+ and other.cut.metrics.recall >= metrics.recall
+ and (
+ other.cut.metrics.precision > metrics.precision
+ or other.cut.metrics.recall > metrics.recall
+ )
+ for other in cuts
+ )
+ if not dominated:
+ frontier.append(candidate)
+ return tuple(frontier)
+
+
+def _enumerate_feasible_refs(
+ forest: PlateauForest,
+ min_cluster_size: int,
+) -> tuple[tuple[int, ...], ...]:
+ _validate_min_cluster_size(min_cluster_size)
+ if forest.n_leaves > MAX_EXHAUSTIVE_ORACLE_LEAVES:
+ raise ValueError(
+ "exhaustive antichain enumeration is limited to "
+ f"{MAX_EXHAUSTIVE_ORACLE_LEAVES} leaves"
+ )
+
+ event_leaf_masks: list[int] = []
+ for event in range(forest.representatives.size):
+ leaf_mask = 0
+ for offset in range(
+ int(forest.child_offsets[event]),
+ int(forest.child_offsets[event + 1]),
+ ):
+ child_ref = int(forest.child_refs[offset])
+ if child_ref < 0:
+ leaf = -child_ref - 1
+ if leaf < 0 or leaf >= forest.n_leaves:
+ raise ValueError(
+ "plateau forest contains an invalid leaf reference"
+ )
+ child_mask = 1 << leaf
+ else:
+ if child_ref >= event:
+ raise ValueError("plateau forest events must be in postorder")
+ child_mask = event_leaf_masks[child_ref]
+ if leaf_mask & child_mask:
+ raise ValueError("plateau event children overlap")
+ leaf_mask |= child_mask
+ if leaf_mask.bit_count() != int(forest.sizes[event]):
+ raise ValueError("plateau event size does not match its children")
+ event_leaf_masks.append(leaf_mask)
+
+ root_events: set[int] = set()
+ fixed_refs: list[int] = []
+ fixed_mask = 0
+ for root_ref_value in forest.component_roots:
+ root_ref = int(root_ref_value)
+ if root_ref < 0:
+ leaf = -root_ref - 1
+ if leaf < 0 or leaf >= forest.n_leaves:
+ raise ValueError("plateau forest contains an invalid component root")
+ leaf_mask = 1 << leaf
+ if fixed_mask & leaf_mask:
+ raise ValueError("plateau forest component roots overlap")
+ fixed_mask |= leaf_mask
+ fixed_refs.append(root_ref)
+ else:
+ if root_ref >= forest.representatives.size:
+ raise ValueError("plateau forest contains an invalid component root")
+ root_events.add(root_ref)
+ if not forest.component_roots.size:
+ raise ValueError("plateau forest must contain a component root")
+
+ candidate_events = tuple(
+ event
+ for event in range(forest.representatives.size)
+ if int(forest.sizes[event]) >= min_cluster_size or event in root_events
+ )
+ target_mask = (1 << forest.n_leaves) - 1
+ antichains: list[tuple[int, ...]] = []
+ for subset in range(1 << len(candidate_events)):
+ coverage = fixed_mask
+ selected_refs = fixed_refs.copy()
+ for candidate_index, event in enumerate(candidate_events):
+ if not subset & (1 << candidate_index):
+ continue
+ leaf_mask = event_leaf_masks[event]
+ if coverage & leaf_mask:
+ break
+ coverage |= leaf_mask
+ selected_refs.append(event)
+ else:
+ if coverage != target_mask:
+ continue
+ antichains.append(
+ tuple(
+ sorted(
+ selected_refs,
+ key=lambda selected_ref: _selected_ref_key(
+ forest,
+ selected_ref,
+ ),
+ )
+ )
+ )
+ if not antichains:
+ raise RuntimeError("plateau forest has no feasible antichain")
+ return tuple(antichains)
+
+
+def enumerate_feasible_antichains(
+ forest: PlateauForest,
+ min_cluster_size: int = 2,
+) -> tuple[tuple[int, ...], ...]:
+ return tuple(
+ _selected_nodes(forest, selected_refs)
+ for selected_refs in _enumerate_feasible_refs(forest, min_cluster_size)
+ )
+
+
+def exhaustive_pair_cuts(
+ forest: PlateauForest,
+ truth: np.ndarray,
+ min_cluster_size: int = 2,
+) -> tuple[PairCut, ...]:
+ statistics = _event_pair_statistics(forest, truth)
+ return tuple(
+ _pair_cut_from_refs(forest, statistics, selected_refs)
+ for selected_refs in _enumerate_feasible_refs(forest, min_cluster_size)
+ )
+
+
+def exhaustive_maximum_pair_f1_cut(
+ forest: PlateauForest,
+ truth: np.ndarray,
+ min_cluster_size: int = 2,
+) -> PairCut:
+ cuts = exhaustive_pair_cuts(forest, truth, min_cluster_size)
+ return max(cuts, key=lambda cut: cut.metrics.f1)
+
+
+def exhaustive_minimum_pair_error_cut(
+ forest: PlateauForest,
+ truth: np.ndarray,
+ *,
+ alpha: int | float,
+ beta: int | float,
+ min_cluster_size: int = 2,
+) -> PairCut:
+ cuts = exhaustive_pair_cuts(forest, truth, min_cluster_size)
+ return min(
+ cuts,
+ key=lambda cut: (
+ alpha * cut.metrics.false_positive_pairs
+ + beta * cut.metrics.false_negative_pairs
+ ),
+ )
+
+
+def oracle_gap_recovery(
+ scorer_pair_f1: float,
+ baseline_pair_f1: float,
+ maximum_pair_f1: float,
+ *,
+ tolerance: float = PAIR_F1_ORACLE_TOLERANCE,
+) -> float:
+ if not math.isfinite(tolerance) or tolerance <= 0:
+ raise ValueError("tolerance must be a positive finite number")
+ values = (scorer_pair_f1, baseline_pair_f1, maximum_pair_f1)
+ if not all(math.isfinite(value) for value in values):
+ raise ValueError("pair-F1 values must be finite")
+ headroom = maximum_pair_f1 - baseline_pair_f1
+ if headroom <= tolerance:
+ return 0.0
+ return (scorer_pair_f1 - baseline_pair_f1) / headroom
+
+
+def evaluate_hierarchy_headroom(
+ hierarchy: ParisHierarchy,
+ truth: np.ndarray,
+ min_cluster_size: int = 2,
+ *,
+ plateau_forest: PlateauForest | None = None,
+ graph: csr_matrix | None = None,
+ baseline_labels: np.ndarray | None = None,
+ tolerance: float = PAIR_F1_ORACLE_TOLERANCE,
+) -> HierarchyHeadroomEvaluation:
+ forest = (
+ collapse_equal_height_plateaus(hierarchy)
+ if plateau_forest is None
+ else plateau_forest
+ )
+ if forest.n_leaves != hierarchy.n_leaves:
+ raise ValueError("plateau forest and hierarchy have different leaf counts")
+ statistics = _event_pair_statistics(forest, truth)
+ if baseline_labels is None:
+ if graph is None:
+ raise ValueError("graph is required when baseline_labels are not provided")
+ split_gate = modularity_split_gains(hierarchy, forest, graph)
+ baseline_labels = adaptive_cut(
+ hierarchy,
+ min_cluster_size,
+ plateau_forest=forest,
+ split_gate=split_gate,
+ ).labels
+ baseline = exact_pair_metrics(baseline_labels, truth)
+ maximum = _maximum_pair_f1_from_statistics(
+ forest,
+ statistics,
+ min_cluster_size,
+ tolerance=tolerance,
+ max_iterations=MAX_DINKELBACH_ITERATIONS,
+ initial_numerator=2 * baseline.true_positive_pairs,
+ initial_denominator=max(
+ 1,
+ baseline.predicted_positive_pairs + statistics.truth_pairs,
+ ),
+ )
+ supported = _supported_pair_cuts_from_statistics(
+ forest,
+ statistics,
+ min_cluster_size,
+ )
+ headroom = maximum.cut.metrics.f1 - baseline.f1
+ return HierarchyHeadroomEvaluation(
+ min_cluster_size=min_cluster_size,
+ baseline=baseline,
+ maximum_pair_f1=maximum,
+ supported_cuts=supported,
+ supported_frontier=nondominated_supported_cuts(supported),
+ pair_f1_headroom=headroom,
+ has_measurable_headroom=headroom > tolerance,
+ )
+
+
+@dataclass(frozen=True)
+class SeedQuality:
+ seed: int
+ adaptive_clusters: int
+ adaptive_ari: float
+ adaptive_rare_f1: float
+ unguarded_clusters: int
+ unguarded_ari: float
+ best_global_ari: float
+ best_global_clusters: int
+ best_cpm_ari: float
+ best_cpm_resolution: float
+ accepted: bool
+
+
+@dataclass(frozen=True)
+class QualityGateReport:
+ seed_count: int
+ accepted_seed_fraction: float
+ passed: bool
+ seeds: tuple[SeedQuality, ...]
+
+
+def unequal_depth_block_graph(seed: int) -> tuple[csr_matrix, np.ndarray]:
+ rng = np.random.default_rng(seed)
+ subgroup_sizes = (45, 45, 35, 35, 20)
+ subgroup = np.repeat(np.arange(len(subgroup_sizes)), subgroup_sizes)
+ labels = np.where(subgroup < 2, 0, np.where(subgroup < 4, 1, 2))
+ probabilities = np.full((5, 5), 0.006)
+ np.fill_diagonal(probabilities, (0.19, 0.19, 0.15, 0.15, 0.38))
+ probabilities[0, 1] = probabilities[1, 0] = 0.10
+ probabilities[2, 3] = probabilities[3, 2] = 0.115
+ probabilities[:2, 2:4] = probabilities[2:4, :2] = 0.012
+ probabilities[4, :4] = probabilities[:4, 4] = 0.003
+
+ rows, columns = np.triu_indices(labels.size, k=1)
+ present = rng.random(rows.size) < probabilities[subgroup[rows], subgroup[columns]]
+ weights = rng.uniform(0.7, 1.3, int(present.sum()))
+ upper = csr_matrix(
+ (weights, (rows[present], columns[present])),
+ shape=(labels.size, labels.size),
+ )
+ return csr_matrix(upper + upper.T), labels
+
+
+def _rare_cluster_f1(labels: np.ndarray, truth: np.ndarray) -> float:
+ rare = truth == 2
+ best = 0.0
+ for cluster in np.unique(labels):
+ predicted = labels == cluster
+ score = (
+ 2
+ * int(np.count_nonzero(predicted & rare))
+ / (int(np.count_nonzero(predicted)) + int(np.count_nonzero(rare)))
+ )
+ best = max(best, score)
+ return best
+
+
+def _best_global_cut(
+ dendrogram: np.ndarray,
+ truth: np.ndarray,
+) -> tuple[float, int]:
+ best_ari = -1.0
+ best_clusters = 0
+ for n_clusters in range(2, len(truth) + 1):
+ labels = straight_cut(dendrogram, n_clusters)
+ ari = float(adjusted_rand_score(truth, labels))
+ if ari > best_ari:
+ best_ari = ari
+ best_clusters = n_clusters
+ return best_ari, best_clusters
+
+
+def _best_cpm_partition(
+ graph: csr_matrix,
+ truth: np.ndarray,
+ seed: int,
+) -> tuple[float, float]:
+ import igraph
+ import leidenalg
+
+ upper = triu(graph, k=1).tocoo()
+ igraph_graph = igraph.Graph(n=graph.shape[0])
+ igraph_graph.add_edges(
+ list(zip(upper.row.tolist(), upper.col.tolist(), strict=True))
+ )
+ igraph_graph.es["weight"] = upper.data.tolist()
+ best_ari = -1.0
+ best_resolution = 0.0
+ for resolution in CPM_RESOLUTIONS:
+ partition = leidenalg.find_partition(
+ igraph_graph,
+ leidenalg.CPMVertexPartition,
+ weights="weight",
+ resolution_parameter=float(resolution),
+ seed=seed,
+ )
+ ari = float(adjusted_rand_score(truth, partition.membership))
+ if ari > best_ari:
+ best_ari = ari
+ best_resolution = float(resolution)
+ return best_ari, best_resolution
+
+
+def evaluate_quality_gate(
+ seed_count: int = DEFAULT_SEED_COUNT,
+ *,
+ n_threads: int = 4,
+) -> QualityGateReport:
+ if seed_count < 1:
+ raise ValueError("seed_count must be positive")
+ outcomes: list[SeedQuality] = []
+ for seed in range(seed_count):
+ graph, truth = unequal_depth_block_graph(seed)
+ hierarchy = fit_paris_hierarchy(graph, n_threads=n_threads)
+ forest = collapse_equal_height_plateaus(hierarchy)
+ split_gate = modularity_split_gains(hierarchy, forest, graph)
+ adaptive = adaptive_cut(
+ hierarchy,
+ min_cluster_size=8,
+ plateau_forest=forest,
+ split_gate=split_gate,
+ )
+ unguarded = adaptive_cut(
+ hierarchy,
+ min_cluster_size=8,
+ plateau_forest=forest,
+ )
+ dendrogram = hierarchy_to_dendrogram(hierarchy)
+ adaptive_ari = float(adjusted_rand_score(truth, adaptive.labels))
+ adaptive_rare_f1 = _rare_cluster_f1(adaptive.labels, truth)
+ unguarded_ari = float(adjusted_rand_score(truth, unguarded.labels))
+ global_ari, global_clusters = _best_global_cut(dendrogram, truth)
+ cpm_ari, cpm_resolution = _best_cpm_partition(graph, truth, seed)
+ accepted = (
+ adaptive_rare_f1 >= MIN_RARE_CLUSTER_F1
+ and adaptive.n_clusters <= MAX_ADAPTIVE_CLUSTERS
+ and adaptive_ari >= global_ari - ORACLE_ARI_TOLERANCE
+ and adaptive.n_clusters <= unguarded.n_clusters
+ and adaptive_ari >= unguarded_ari - ORACLE_ARI_TOLERANCE
+ )
+ outcomes.append(
+ SeedQuality(
+ seed=seed,
+ adaptive_clusters=adaptive.n_clusters,
+ adaptive_ari=adaptive_ari,
+ adaptive_rare_f1=adaptive_rare_f1,
+ unguarded_clusters=unguarded.n_clusters,
+ unguarded_ari=unguarded_ari,
+ best_global_ari=global_ari,
+ best_global_clusters=global_clusters,
+ best_cpm_ari=cpm_ari,
+ best_cpm_resolution=cpm_resolution,
+ accepted=accepted,
+ )
+ )
+ accepted_fraction = sum(item.accepted for item in outcomes) / seed_count
+ return QualityGateReport(
+ seed_count=seed_count,
+ accepted_seed_fraction=accepted_fraction,
+ passed=accepted_fraction >= MIN_ACCEPTED_SEED_FRACTION,
+ seeds=tuple(outcomes),
+ )
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--seeds", type=int, default=DEFAULT_SEED_COUNT)
+ parser.add_argument("--threads", type=int, default=4)
+ arguments = parser.parse_args()
+ report = evaluate_quality_gate(arguments.seeds, n_threads=arguments.threads)
+ print(json.dumps(asdict(report), indent=2))
+ if not report.passed:
+ raise SystemExit(1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/seurat_hto_5_5_1_golden.json b/tests/seurat_hto_5_5_1_golden.json
new file mode 100644
index 00000000..f269f144
--- /dev/null
+++ b/tests/seurat_hto_5_5_1_golden.json
@@ -0,0 +1,15059 @@
+{
+ "provenance": {
+ "package": "Seurat",
+ "packageVersion": "5.5.1",
+ "seuratObjectVersion": "5.4.0",
+ "fitdistrplusVersion": "1.2.6",
+ "rVersion": "R version 4.4.3 (2025-02-28)",
+ "sourceRepository": "https://github.com/satijalab/seurat",
+ "dataset": "GSE245108",
+ "datasetSource": "https://ftp.ncbi.nlm.nih.gov/geo/series/GSE245nnn/GSE245108/suppl/GSE245108_TNC-1-2-1_filtered_feature_bc_matrix.h5",
+ "sourceFile": "GSE245108_TNC-1-2-1_filtered_feature_bc_matrix.h5",
+ "sourceSha256": "22d64482c0fb04a7685e641d762efdd95a6498156ac22e7b0264925b0907b589",
+ "generator": "scripts/export_seurat_hto_golden.py",
+ "nCells": 1000,
+ "nHtos": 10,
+ "cellSampleSeed": 20260731,
+ "randomSeed": 0,
+ "normalization": {
+ "method": "CLR",
+ "margin": 1
+ },
+ "clustering": {
+ "method": "kmeans",
+ "clusterCount": 11,
+ "nStarts": 100
+ },
+ "cutoff": {
+ "distribution": "negative_binomial",
+ "positiveQuantile": 0.99,
+ "comparison": "strictly_greater"
+ },
+ "referenceFunction": "Seurat::HTODemux"
+ },
+ "inputs": {
+ "barcodes": [
+ "AAACCCACAAGCGAGT-1",
+ "AAACCCAGTCGTGTTA-1",
+ "AAACCCATCCTAACAG-1",
+ "AAAGAACCACGCGTGT-1",
+ "AAAGGGCCAGTGCGCT-1",
+ "AAAGGTAAGGGCCAAT-1",
+ "AAAGTCCCAACGGCTC-1",
+ "AAAGTGATCTTCTAAC-1",
+ "AAATGGAGTATTCCTT-1",
+ "AAATGGATCAGCCTTC-1",
+ "AACAAAGAGCAGCACA-1",
+ "AACAACCGTGAGTAAT-1",
+ "AACAAGAGTGAGCTCC-1",
+ "AACACACCAAAGCTAA-1",
+ "AACACACCAGTTTCGA-1",
+ "AACCATGTCAAGCCCG-1",
+ "AACCCAAAGTAGGGTC-1",
+ "AACCTGAGTGCTTATG-1",
+ "AACCTGATCTTGCAGA-1",
+ "AACCTTTAGACATAAC-1",
+ "AACCTTTAGTTCTACG-1",
+ "AACGAAAGTCACGACC-1",
+ "AACGGGATCTCCCTAG-1",
+ "AACGTCAGTGAGTAGC-1",
+ "AAGAACACACCAGCGT-1",
+ "AAGACAAGTGTCCGGT-1",
+ "AAGACTCTCTTGGCTC-1",
+ "AAGATAGCAATAACCC-1",
+ "AAGATAGGTCCTATAG-1",
+ "AAGATAGGTTACGTAC-1",
+ "AAGATAGGTTTCGCTC-1",
+ "AAGGTAAGTCCCTCAT-1",
+ "AAGTACCAGAGGATCC-1",
+ "AAGTCGTAGACGAGCT-1",
+ "AAGTTCGGTATATGGA-1",
+ "AAGTTCGGTCACCGCA-1",
+ "AATAGAGGTCTAGGCC-1",
+ "AATAGAGTCTATCCAT-1",
+ "AATCGACAGGTCATTC-1",
+ "AATGGCTCAATACCCA-1",
+ "AATTCCTCAAACTAGA-1",
+ "AATTCCTCAACCAGAG-1",
+ "AATTCCTTCTCAGGCG-1",
+ "ACAAGCTAGGCTGTAG-1",
+ "ACAAGCTCATTAGGAA-1",
+ "ACACAGTGTTCTCGCT-1",
+ "ACACGCGAGGTCATTC-1",
+ "ACACTGACAAAGCTCT-1",
+ "ACAGGGAAGCGCCTAC-1",
+ "ACAGGGAGTAGATCCT-1",
+ "ACATCCCCAAGTGGCA-1",
+ "ACATCCCTCATGACAC-1",
+ "ACATGCAAGTCACTCA-1",
+ "ACATGCACAGTAACGG-1",
+ "ACCAAACTCCGGCAGT-1",
+ "ACCACAACACTTCAGA-1",
+ "ACCATTTGTAGTTACC-1",
+ "ACCATTTTCATCGGGC-1",
+ "ACCCAAATCGTTCATT-1",
+ "ACCCTTGAGACGACGT-1",
+ "ACCCTTGGTTGTAAAG-1",
+ "ACCCTTGTCAAGTGTC-1",
+ "ACCGTTCGTATGGTAA-1",
+ "ACCTACCAGGAGTACC-1",
+ "ACCTACCGTTAGTCGT-1",
+ "ACCTACCTCCAATCTT-1",
+ "ACCTGAAAGCAAACAT-1",
+ "ACCTGAACAATGAGCG-1",
+ "ACCTGAATCTAGCCTC-1",
+ "ACCTGTCTCCACGTAA-1",
+ "ACGATCAAGGTTCAGG-1",
+ "ACGATCAGTACAACGG-1",
+ "ACGATCATCCCTCTCC-1",
+ "ACGCACGCAGCATTGT-1",
+ "ACGCACGCATCGATGT-1",
+ "ACGCACGGTCAGCGTC-1",
+ "ACGGAAGAGTTGTACC-1",
+ "ACGGAAGCATCTGTTT-1",
+ "ACGGGTCAGGTGGCTA-1",
+ "ACGGTCGGTATGGGAC-1",
+ "ACGGTTAAGCCTGGAA-1",
+ "ACGTACAAGACCGCCT-1",
+ "ACGTACAAGTGCTACT-1",
+ "ACGTACACAGCTATAC-1",
+ "ACGTACATCGCAACAT-1",
+ "ACGTCCTTCGGCATTA-1",
+ "ACGTTCCCACTTGGGC-1",
+ "ACGTTCCGTGGAGAAA-1",
+ "ACGTTCCTCCGAGGCT-1",
+ "ACTATTCCAGGGACTA-1",
+ "ACTCCCACAAAGGTTA-1",
+ "ACTCCCACAGTCTACA-1",
+ "ACTCTCGAGAAACACT-1",
+ "ACTGCAACAGAGATTA-1",
+ "ACTGTCCAGAATAACC-1",
+ "ACTTATCCACAAGCCC-1",
+ "ACTTCGCCACTTGGCG-1",
+ "ACTTCGCGTGGCGTAA-1",
+ "ACTTCGCTCTGAGATC-1",
+ "ACTTTCAAGGGCTGAT-1",
+ "ACTTTGTAGAAGGCTC-1",
+ "AGAACAAGTGGACCTC-1",
+ "AGAACCTAGAGGTTTA-1",
+ "AGAACCTAGCGTACAG-1",
+ "AGAACCTTCTTTCCAA-1",
+ "AGAAGTATCACAACCA-1",
+ "AGACAAACATGACACT-1",
+ "AGACACTAGAGAGCGG-1",
+ "AGACACTAGTCCCAGC-1",
+ "AGACACTCAAGTAGTA-1",
+ "AGACACTCAGGAACCA-1",
+ "AGACACTGTTATTCTC-1",
+ "AGACAGGAGATGATTG-1",
+ "AGACAGGAGCGTATAA-1",
+ "AGACAGGTCCGTGGGT-1",
+ "AGACAGGTCTCAATCT-1",
+ "AGACTCAGTAGAGTTA-1",
+ "AGAGAATTCCTAGCGG-1",
+ "AGATAGACAACTTGGT-1",
+ "AGATGAACATGGCGCT-1",
+ "AGATGCTCACAACGTT-1",
+ "AGATGCTCACATGGTT-1",
+ "AGCCAATTCACATCAG-1",
+ "AGCCAGCGTAATACCC-1",
+ "AGCGATTCACGAAAGC-1",
+ "AGCGCCAGTAACGGTG-1",
+ "AGCGTATAGTCCTACA-1",
+ "AGCGTATGTTACCTTT-1",
+ "AGCGTATGTTGCTCAA-1",
+ "AGCGTCGGTACCTAAC-1",
+ "AGCTACACAGCCGGTT-1",
+ "AGGAAATGTGGTCTGC-1",
+ "AGGAATAAGTCTGGAG-1",
+ "AGGACGATCAAGGAGC-1",
+ "AGGACGATCCTCATAT-1",
+ "AGGATCTAGAGGTATT-1",
+ "AGGATCTCATATGAAG-1",
+ "AGGCATTAGCCTATTG-1",
+ "AGGCATTGTCTCGGGT-1",
+ "AGGCCACTCTCCCATG-1",
+ "AGGGCTCTCTATCGGA-1",
+ "AGGGTCCGTGCCCACA-1",
+ "AGGGTCCTCCTGTAAG-1",
+ "AGGGTGAAGACGGTTG-1",
+ "AGGGTGAAGACTCTAC-1",
+ "AGGGTTTAGATTGTGA-1",
+ "AGGGTTTTCCACCTCA-1",
+ "AGGTCATCAGAGGAAA-1",
+ "AGGTCTAAGGATACGC-1",
+ "AGGTCTACATCAGCGC-1",
+ "AGGTGTTTCTAACGGT-1",
+ "AGGTTACGTATGTCTG-1",
+ "AGGTTGTCAGGTCAAG-1",
+ "AGTACTGAGGTGCAGT-1",
+ "AGTACTGGTCTTCCGT-1",
+ "AGTACTGTCATCCTAT-1",
+ "AGTACTGTCCGTAGTA-1",
+ "AGTAGCTCATGAGAAT-1",
+ "AGTAGTCCAATGCTCA-1",
+ "AGTAGTCGTAGGAGGG-1",
+ "AGTAGTCTCATGCATG-1",
+ "AGTCAACAGCCTATCA-1",
+ "AGTCAACGTCATCACA-1",
+ "AGTCATGTCACGTAGT-1",
+ "AGTCTCCAGGAAACGA-1",
+ "AGTGACTGTCCACAGC-1",
+ "AGTGACTTCTTCGCTG-1",
+ "AGTGCCGAGCCATCCG-1",
+ "AGTGCCGGTCAGATTC-1",
+ "AGTGCCGGTCGTTCAA-1",
+ "AGTTAGCGTGCCTATA-1",
+ "AGTTCGATCTTAAGGC-1",
+ "ATAGACCTCTGGGAGA-1",
+ "ATAGACCTCTTAGCTT-1",
+ "ATCACAGTCGTCAGAT-1",
+ "ATCACTTCAAACTGCT-1",
+ "ATCACTTGTCTCACGG-1",
+ "ATCACTTTCCATGCAA-1",
+ "ATCAGGTAGCACCCAC-1",
+ "ATCAGGTGTCGAGCAA-1",
+ "ATCATTCGTTCCGCAG-1",
+ "ATCCACCTCAACTGGT-1",
+ "ATCCATTAGTATGGCG-1",
+ "ATCCATTCACGTACTA-1",
+ "ATCCATTGTAGCGTTT-1",
+ "ATCCCTGTCCACATAG-1",
+ "ATCCCTGTCTCTCTAA-1",
+ "ATCCGTCAGCACTCTA-1",
+ "ATCCGTCCAACAGCTT-1",
+ "ATCCGTCCACTGGCGT-1",
+ "ATCCGTCGTTCGGCTG-1",
+ "ATCCGTCTCTCAACCC-1",
+ "ATCCGTCTCTCAATCT-1",
+ "ATCGATGTCACTTCTA-1",
+ "ATCGCCTAGACTCAAA-1",
+ "ATCGCCTCAAATAAGC-1",
+ "ATCGGATGTGTGTACT-1",
+ "ATCGGATTCGGACCAC-1",
+ "ATCGGATTCTGACAGT-1",
+ "ATCGGCGTCTTCTGTA-1",
+ "ATCGTAGAGCCAGACA-1",
+ "ATCGTCCGTACATACC-1",
+ "ATCGTGATCTTGGCTC-1",
+ "ATCTCTAGTTAGAGTA-1",
+ "ATCTTCATCCCATAGA-1",
+ "ATGAAAGTCAGCTCTC-1",
+ "ATGACCAGTTAGCGGA-1",
+ "ATGAGGGGTATACCTG-1",
+ "ATGAGTCTCACTGTTT-1",
+ "ATGATCGCAAACCACT-1",
+ "ATGATCGGTAGTCTTG-1",
+ "ATGCATGTCGGTCAGC-1",
+ "ATGCCTCTCTCAGTCC-1",
+ "ATGGAGGAGAATAGTC-1",
+ "ATGGAGGTCACCATAG-1",
+ "ATGGTTGGTTACCTGA-1",
+ "ATGGTTGTCGTTAGTG-1",
+ "ATGTCCCAGTAGACAT-1",
+ "ATTACCTCAGAGATGC-1",
+ "ATTACCTTCGGTTCAA-1",
+ "ATTCACTAGTGCTCGC-1",
+ "ATTCACTTCGCATTAG-1",
+ "ATTCAGGTCTTCCCAG-1",
+ "ATTCATCGTCAAGGCA-1",
+ "ATTCCTAGTAGGAGTC-1",
+ "ATTCTTGAGTAACGTA-1",
+ "ATTGGGTTCAGCAATC-1",
+ "ATTTACCGTAGTTAGA-1",
+ "ATTTACCGTCTGCCTT-1",
+ "ATTTCACGTAGGCAAC-1",
+ "ATTTCTGTCTACACAG-1",
+ "ATTTCTGTCTGACAGT-1",
+ "CAACAACCAGGTGACA-1",
+ "CAACAACGTCACTTCC-1",
+ "CAACAGTAGGACTAAT-1",
+ "CAACAGTGTAACAGTA-1",
+ "CAACCAATCTCTGCTG-1",
+ "CAAGGGAGTCTCACAA-1",
+ "CAATTTCGTCACTAGT-1",
+ "CACAACAAGGGATGTC-1",
+ "CACAGGCCATCTAACG-1",
+ "CACAGGCGTCGGCCTA-1",
+ "CACAGGCTCAACCTTT-1",
+ "CACATGATCTCTCCGA-1",
+ "CACCAAACAAATCGGG-1",
+ "CACCGTTCATAAGATG-1",
+ "CACGAATAGAGCAGCT-1",
+ "CACGAATGTGTAGTGG-1",
+ "CACGGGTGTGATAGTA-1",
+ "CACGGGTTCCGTTGGG-1",
+ "CACGTTCGTAAGGCCA-1",
+ "CACTAAGGTAGGTCAG-1",
+ "CACTGAAAGTCATACC-1",
+ "CACTGAAGTGGATGAC-1",
+ "CACTGGGTCGTAACAC-1",
+ "CACTGTCAGTTGTAAG-1",
+ "CACTGTCGTTTACGAC-1",
+ "CACTGTCGTTTAGACC-1",
+ "CACTTCGCATGTGCCG-1",
+ "CAGATACCAGCACAGA-1",
+ "CAGATCACAAGGTCGA-1",
+ "CAGATCATCATCTACT-1",
+ "CAGATTGTCTAACACG-1",
+ "CAGCAATAGCGGATCA-1",
+ "CAGCAGCAGTAAACGT-1",
+ "CAGCAGCAGTACCCTA-1",
+ "CAGCAGCAGTTCCGGC-1",
+ "CAGCGTGTCACCGGGT-1",
+ "CAGGTATAGCTCTATG-1",
+ "CAGTTAGTCATTCATC-1",
+ "CAGTTCCGTGTGATGG-1",
+ "CATACCCCAGAGATTA-1",
+ "CATACCCTCATTTCGT-1",
+ "CATACTTAGTAAATGC-1",
+ "CATACTTGTCTTCGAA-1",
+ "CATAGACAGAAACCCG-1",
+ "CATCCACAGCATAGGC-1",
+ "CATCCACCAATTTCTC-1",
+ "CATCCACCATCATTTC-1",
+ "CATCCGTAGCAACAAT-1",
+ "CATCGTCAGCATCTTG-1",
+ "CATCGTCCACCTGCTT-1",
+ "CATGCAATCCTGTTAT-1",
+ "CATGCCTTCTTAGCAG-1",
+ "CATGGATTCCGCGATG-1",
+ "CATTCATGTTAGTCGT-1",
+ "CATTCCGGTGCCTAAT-1",
+ "CATTCCGTCACTTATC-1",
+ "CATTGCCCATGAGGGT-1",
+ "CATTGTTCAAAGAGTT-1",
+ "CATTGTTTCGTCCATC-1",
+ "CCAAGCGAGTGTTCCA-1",
+ "CCAAGCGCACGAGGTA-1",
+ "CCAAGCGTCAAGATAG-1",
+ "CCAATGAGTAGGCTGA-1",
+ "CCAATGATCAAATGAG-1",
+ "CCAATTTTCTAGTACG-1",
+ "CCACAAAAGAATCGCG-1",
+ "CCACAAATCGGTCGAC-1",
+ "CCACACTAGGATTTGA-1",
+ "CCACCATAGGGTGGGA-1",
+ "CCACCATTCCCGTTCA-1",
+ "CCACGAGGTACAGGTG-1",
+ "CCACGTTAGCAACAGC-1",
+ "CCACGTTGTGGTTCTA-1",
+ "CCACTTGCAGTATACC-1",
+ "CCACTTGGTACAAACA-1",
+ "CCATAAGTCGTCCTCA-1",
+ "CCCATTGGTGACTATC-1",
+ "CCCATTGTCAAGTCTG-1",
+ "CCCGGAAAGATTAGAC-1",
+ "CCCGGAATCTTGGAAC-1",
+ "CCCTCAACAAGACTGG-1",
+ "CCCTGATGTGCCTATA-1",
+ "CCCTTAGAGACCATAA-1",
+ "CCCTTAGGTGGCGTAA-1",
+ "CCGATCTCAAAGCACG-1",
+ "CCGATGGCAATGTCTG-1",
+ "CCGCAAGAGGTATAGT-1",
+ "CCGCAAGCAGCAGATG-1",
+ "CCGCAAGTCTACTGAG-1",
+ "CCGGACAAGCGTTCAT-1",
+ "CCGGTAGCAAGTGGCA-1",
+ "CCGGTGACAGACCAAG-1",
+ "CCGGTGACAGTTCACA-1",
+ "CCGTAGGCAGAAACCG-1",
+ "CCGTAGGCAGACGCTC-1",
+ "CCGTGAGCAGCCTATA-1",
+ "CCGTTCACAGGAACCA-1",
+ "CCGTTCAGTTAGAAGT-1",
+ "CCTACGTCACCAAATC-1",
+ "CCTCAGTAGGACTGGT-1",
+ "CCTCAGTCAACCGTGC-1",
+ "CCTCATGAGTGGTGAC-1",
+ "CCTCATGCAGTCAACT-1",
+ "CCTCCAATCTGGCCGA-1",
+ "CCTCCTCGTTTGTTGG-1",
+ "CCTCTAGCAACTCCAA-1",
+ "CCTCTAGGTTTGCCGG-1",
+ "CCTCTCCAGAGGATCC-1",
+ "CCTGCATGTGCCTAAT-1",
+ "CCTTGTGGTGTTCGTA-1",
+ "CCTTTGGCAGGCTACC-1",
+ "CCTTTGGTCACAATGC-1",
+ "CGAAGGAAGCTGACCC-1",
+ "CGAAGGAGTACGACAG-1",
+ "CGAAGGAGTAGTACGG-1",
+ "CGAAGTTAGGATGCGT-1",
+ "CGACAGCTCTTCGGTC-1",
+ "CGAGAAGTCACCGGGT-1",
+ "CGAGGCTGTCACAGTT-1",
+ "CGAGGCTTCAGGAAAT-1",
+ "CGAGGCTTCGTGCATA-1",
+ "CGATCGGTCAGTGTCA-1",
+ "CGATGCGCAATGGCAG-1",
+ "CGATGGCAGTTCTCTT-1",
+ "CGATGGCCATCGATGT-1",
+ "CGCAGGTCAAGGAGTC-1",
+ "CGCAGGTTCATGTCAG-1",
+ "CGCATAACACAGCCTG-1",
+ "CGCATAAGTCAGACGA-1",
+ "CGCATAAGTGTTTGCA-1",
+ "CGCATGGCAAGTGCAG-1",
+ "CGCATGGGTCTTCTAT-1",
+ "CGCCATTGTCGTTTCC-1",
+ "CGCCATTGTTGAGGAC-1",
+ "CGCGTGAAGCTTGTGT-1",
+ "CGGAACCCACACACTA-1",
+ "CGGAACCTCGCCGAAC-1",
+ "CGGACACAGCGAATGC-1",
+ "CGGACACCACAATTCG-1",
+ "CGGACACTCATGGCCG-1",
+ "CGGAGAAAGTAGGCCA-1",
+ "CGGAGAACAGTCAGTT-1",
+ "CGGGCATCATGACACT-1",
+ "CGGGTCACAACGCATT-1",
+ "CGGGTCACACCTTCGT-1",
+ "CGGTCAGCATCATTTC-1",
+ "CGGTCAGTCCGTGGGT-1",
+ "CGTAAGTGTTGAGTCT-1",
+ "CGTAATGCAACGGTAG-1",
+ "CGTGAATGTATCCTTT-1",
+ "CGTTAGAGTGACACAG-1",
+ "CGTTAGATCTTCGATT-1",
+ "CGTTCTGAGTACTGGG-1",
+ "CGTTCTGCAACGATCT-1",
+ "CGTTGGGGTTATTCCT-1",
+ "CGTTGGGTCAGACATC-1",
+ "CTAACCCTCGCTCTAC-1",
+ "CTAAGTGGTACTGTTG-1",
+ "CTACAGACACCAATTG-1",
+ "CTACCCAAGGAGCAAA-1",
+ "CTACCTGAGTATAACG-1",
+ "CTAGACAAGGGAGTGG-1",
+ "CTAGGTATCTCTGCTG-1",
+ "CTATAGGAGGAAGTAG-1",
+ "CTCAACCAGGCAGTCA-1",
+ "CTCAACCTCGTAGCCG-1",
+ "CTCAAGAAGTATGACA-1",
+ "CTCAATTAGTCTGCGC-1",
+ "CTCAATTGTATGGAGC-1",
+ "CTCAGGGCACTTCAAG-1",
+ "CTCAGGGGTTATGTCG-1",
+ "CTCAGTCTCGATGCTA-1",
+ "CTCCAACGTATTCCGA-1",
+ "CTCCCAAAGGCCTTCG-1",
+ "CTCCCAAAGTTACGGG-1",
+ "CTCCCAAGTCGATTCA-1",
+ "CTCCGATAGTCTAACC-1",
+ "CTCCGATAGTGGACTG-1",
+ "CTCCGATCAAATCAAG-1",
+ "CTCCGATTCTATCGGA-1",
+ "CTCCTCCAGCTCCGAC-1",
+ "CTCCTCCCACTCATAG-1",
+ "CTCCTCCCAGTGGCTC-1",
+ "CTCCTCCGTGAACGGT-1",
+ "CTCCTCCTCTCTCCTC-1",
+ "CTCCTTTGTGAACTAA-1",
+ "CTCCTTTTCCACGAAT-1",
+ "CTCGAGGAGTGCCTCG-1",
+ "CTCTCGAAGCACTGGA-1",
+ "CTCTCGACAGAGTTGG-1",
+ "CTCTCGATCATGAAAG-1",
+ "CTGAATGGTAGCTGTT-1",
+ "CTGAATGTCAGGACGA-1",
+ "CTGAATGTCTTCACGC-1",
+ "CTGAGCGCAATTGCCA-1",
+ "CTGAGCGTCGAAATCC-1",
+ "CTGCAGGGTTTCGTGA-1",
+ "CTGCAGGTCTATCCAT-1",
+ "CTGCATCAGCAAGTGC-1",
+ "CTGCATCAGCATTGTC-1",
+ "CTGCCATGTGGACCTC-1",
+ "CTGCCTAGTCATATGC-1",
+ "CTGGACGAGTGATAAC-1",
+ "CTGGCAGAGGATTTAG-1",
+ "CTGGCAGCATCAACCA-1",
+ "CTGTACCAGAACAAGG-1",
+ "CTGTATTGTTGTCAGT-1",
+ "CTGTGAAAGAAGGGAT-1",
+ "CTGTGAAAGGTTCTTG-1",
+ "CTGTGAATCATCAGTG-1",
+ "CTGTGGGCAAGTTGGG-1",
+ "CTTACCGGTCAACACT-1",
+ "CTTAGGATCATCTCTA-1",
+ "CTTAGGATCGCCAGTG-1",
+ "CTTCCGACATAGGTTC-1",
+ "CTTCCTTTCCAATGCA-1",
+ "CTTCGGTAGCGTGCTC-1",
+ "CTTCGGTCACTATCGA-1",
+ "CTTCGGTGTCACGCTG-1",
+ "CTTCTAAAGTAGCAAT-1",
+ "CTTGAGACACATAACC-1",
+ "CTTTCGGAGTCGAAAT-1",
+ "GAAATGAAGCCAGTAG-1",
+ "GAAATGACAGCTATAC-1",
+ "GAAATGATCCCGTGAG-1",
+ "GAACACTCACTTACAG-1",
+ "GAACGTTGTATTCCTT-1",
+ "GAACTGTTCACGTAGT-1",
+ "GAACTGTTCCATTCAT-1",
+ "GAAGCCCGTGAAGCGT-1",
+ "GAAGCCCTCCAAGCAT-1",
+ "GAAGCGATCCGAGAAG-1",
+ "GAAGGACTCTGGACCG-1",
+ "GAAGGGTAGAAGGGAT-1",
+ "GAAGGGTAGATGTTAG-1",
+ "GAAGGGTCAACCGCCA-1",
+ "GAAGTAACATCCAACA-1",
+ "GAAGTAAGTAATACCC-1",
+ "GAATAGATCGTGGACC-1",
+ "GAATCACAGGATTTGA-1",
+ "GAATCACTCCGACAGC-1",
+ "GACAGCCGTGAGAACC-1",
+ "GACAGCCTCGTTATCT-1",
+ "GACATCACAGCTACAT-1",
+ "GACATCATCGAAACAA-1",
+ "GACCAATAGGCTTAGG-1",
+ "GACCCAGTCAATCGGT-1",
+ "GACCCTTAGAAACTAC-1",
+ "GACCCTTTCGTCTAAG-1",
+ "GACCGTGCACAGAGAC-1",
+ "GACCGTGGTTCGGTTA-1",
+ "GACGCTGCACGCCACA-1",
+ "GACGCTGTCACTCCGT-1",
+ "GACGCTGTCACTTGTT-1",
+ "GACTATGGTGTGTCGC-1",
+ "GACTATGTCAACGAGG-1",
+ "GACTCTCCACAGCTTA-1",
+ "GACTCTCGTCTGTAGT-1",
+ "GACTCTCGTGAGTAGC-1",
+ "GACTTCCCAGGGCTTC-1",
+ "GAGAAATAGACAAGCC-1",
+ "GAGATGGTCGCTTGAA-1",
+ "GAGCCTGCATGACAGG-1",
+ "GAGGCAATCGCCTCTA-1",
+ "GAGGCCTAGCGTTCAT-1",
+ "GAGGGATTCACCTCGT-1",
+ "GAGTGAGCAGCGGTTC-1",
+ "GAGTTACAGCCTCAGC-1",
+ "GAGTTGTTCATGAGGG-1",
+ "GAGTTTGAGGCTCTAT-1",
+ "GATCAGTAGGCTTAAA-1",
+ "GATCGTAAGGACCCAA-1",
+ "GATCGTAGTACCCAGC-1",
+ "GATGAGGCAGCGAACA-1",
+ "GATGCTATCGTGGGAA-1",
+ "GATGGAGAGATGACCG-1",
+ "GATGGAGCAATGACCT-1",
+ "GATGGAGCATAGACTC-1",
+ "GATTCGAAGGCGATAC-1",
+ "GATTCGATCGCGCCAA-1",
+ "GATTTCTGTCACGCTG-1",
+ "GCAACCGAGAACGTGC-1",
+ "GCAACCGAGTTAGAAC-1",
+ "GCAACCGCATGATCTG-1",
+ "GCAACCGGTTAGCGGA-1",
+ "GCACTAATCGGTGTAT-1",
+ "GCAGCCAAGAGGTGCT-1",
+ "GCAGCCAGTTCGAACT-1",
+ "GCAGCTGAGCTCTATG-1",
+ "GCATCGGGTACCCAGC-1",
+ "GCATGATGTCTTCGAA-1",
+ "GCATTAGCATACCACA-1",
+ "GCATTAGGTATCCCAA-1",
+ "GCCAACGCAAGTATAG-1",
+ "GCCAACGCACGTCTCT-1",
+ "GCCAGCAGTAGAGTTA-1",
+ "GCCAGTGCAGTCAGAG-1",
+ "GCCATGGAGTGAGTTA-1",
+ "GCCATTCGTTCGGTAT-1",
+ "GCCCAGAAGGGCCCTT-1",
+ "GCCCAGAAGTTCCATG-1",
+ "GCCCGAACACCTGTCT-1",
+ "GCCCGAACATTAGGAA-1",
+ "GCCCGAATCTCCATAT-1",
+ "GCGATCGGTTGCATCA-1",
+ "GCGGAAACACGGTGCT-1",
+ "GCGGATCAGATTTGCC-1",
+ "GCGGATCAGCTAAATG-1",
+ "GCGGATCAGTAAGAGG-1",
+ "GCGGATCCAAACCGGA-1",
+ "GCGGATCCAATCGCCG-1",
+ "GCGGATCGTTCCAAAC-1",
+ "GCGTTTCTCAAAGAAC-1",
+ "GCGTTTCTCGCTTTAT-1",
+ "GCTACCTCAACAGATA-1",
+ "GCTACCTCAAGGTACG-1",
+ "GCTACCTCACGCGGTT-1",
+ "GCTCAAACAGCGTTTA-1",
+ "GCTCAAACATACTGTG-1",
+ "GCTGAATCAGCTGGTC-1",
+ "GCTGCAGAGAGTTGAT-1",
+ "GCTGCAGCAGACTCTA-1",
+ "GCTGCAGCATCGAAGG-1",
+ "GCTTCACGTCCCGCAA-1",
+ "GCTTTCGCAGCTGTTA-1",
+ "GCTTTCGGTCCAGTTA-1",
+ "GGAACCCCAATTGCAC-1",
+ "GGAACCCTCTCATTAC-1",
+ "GGAAGTGGTCTTGAGT-1",
+ "GGAAGTGGTGGCTAGA-1",
+ "GGAATCTGTAACACCT-1",
+ "GGACGTCAGATGGGCT-1",
+ "GGAGGATGTAGTGCGA-1",
+ "GGAGGTAGTGATCATC-1",
+ "GGATGTTCACTCGATA-1",
+ "GGCAGTCAGATCGCCC-1",
+ "GGCGTCATCTAAGAAG-1",
+ "GGCTGTGAGCACAAAT-1",
+ "GGCTGTGAGCGTGAAC-1",
+ "GGCTGTGAGGCAGCTA-1",
+ "GGCTGTGGTGATTCTG-1",
+ "GGCTGTGTCCTAGCTC-1",
+ "GGCTTGGTCGTTACCC-1",
+ "GGCTTTCGTTTCAGAC-1",
+ "GGGAAGTCAATTTCGG-1",
+ "GGGACAATCTCGTTTA-1",
+ "GGGAGATAGCATGATA-1",
+ "GGGATCCCAGTGTGGA-1",
+ "GGGCGTTCAAGCGCAA-1",
+ "GGGCGTTCACAGTCAT-1",
+ "GGGCGTTTCAGCTCTC-1",
+ "GGGCTACGTTGCTAGT-1",
+ "GGGCTCAAGGCTCACC-1",
+ "GGGTATTAGGCACAAC-1",
+ "GGGTCACCATAACTCG-1",
+ "GGGTCACGTCCCTGAG-1",
+ "GGGTCACTCGCAACAT-1",
+ "GGGTCTGCACGTCTCT-1",
+ "GGGTGAAAGATAGGGA-1",
+ "GGGTGAAAGCTACTAC-1",
+ "GGGTGTCAGCGTCAGA-1",
+ "GGGTTTACAACGTTAC-1",
+ "GGTAACTGTTCGTACA-1",
+ "GGTAATCAGCATAGGC-1",
+ "GGTAGAGCACGGATCC-1",
+ "GGTAGAGGTAGCGTTT-1",
+ "GGTGATTTCGCTTGCT-1",
+ "GGTGGCTCAAGCAATA-1",
+ "GGTGTCGAGTGAACAT-1",
+ "GGTGTTAAGAGGCCAT-1",
+ "GGTTAACTCCCTCTCC-1",
+ "GGTTCTCCATGAAGCG-1",
+ "GGTTGTAAGTAGAGTT-1",
+ "GGTTGTACAACCGTAT-1",
+ "GGTTGTACATGTCTAG-1",
+ "GTAACCAGTATCCTTT-1",
+ "GTAACCAGTGTGAGCA-1",
+ "GTAAGTCAGGAGAATG-1",
+ "GTAGAGGCATGTCGTA-1",
+ "GTAGAGGGTAACACGG-1",
+ "GTAGAGGTCCCATTCG-1",
+ "GTAGAGGTCCGAACGC-1",
+ "GTAGATCTCATCGTAG-1",
+ "GTAGATCTCTAGACCA-1",
+ "GTAGCTAAGAAGATCT-1",
+ "GTAGCTAGTTGATCGT-1",
+ "GTAGGAGGTGAGCCAA-1",
+ "GTAGTACTCCACTTTA-1",
+ "GTATTGGAGCGGTATG-1",
+ "GTATTTCAGTGGTCAG-1",
+ "GTATTTCGTAAGTTAG-1",
+ "GTCAAACTCGTCAACA-1",
+ "GTCAAGTCACATTACG-1",
+ "GTCAAGTCAGATTTCG-1",
+ "GTCAAGTGTAGGTCAG-1",
+ "GTCACGGAGGCTGGAT-1",
+ "GTCACTCGTAACAAGT-1",
+ "GTCAGCGAGCCACAAG-1",
+ "GTCAGCGAGCGTTGTT-1",
+ "GTCATCCCAAGACAAT-1",
+ "GTCATTTAGTTTGAGA-1",
+ "GTCATTTGTCGGTGTC-1",
+ "GTCCACTAGTAAACTG-1",
+ "GTCGCGATCGATACTG-1",
+ "GTCGTAAAGACGCCCT-1",
+ "GTCGTTCAGTGATAGT-1",
+ "GTCTACCCAGGTGTGA-1",
+ "GTCTACCGTAAGCGGT-1",
+ "GTCTACCTCACCCTTG-1",
+ "GTCTAGAAGTCCGTCG-1",
+ "GTCTCACGTCCAATCA-1",
+ "GTCTGTCTCTCATGCC-1",
+ "GTCTTTAGTTATAGCC-1",
+ "GTGACGCAGTTACTCG-1",
+ "GTGAGGAAGTATGATG-1",
+ "GTGAGGAGTGCGAACA-1",
+ "GTGAGTTAGATGCTAA-1",
+ "GTGAGTTGTTGAGGAC-1",
+ "GTGATGTAGAATCGCG-1",
+ "GTGATGTCAGTAGAAT-1",
+ "GTGATGTTCAGTCAGT-1",
+ "GTGCACGGTGTCATCA-1",
+ "GTGCACGTCTTGGGCG-1",
+ "GTGCAGCGTGCAGGAT-1",
+ "GTGCAGCTCCGCACTT-1",
+ "GTGCTGGCAGCGTGAA-1",
+ "GTGCTGGGTGGTATGG-1",
+ "GTGCTTCAGTAGGATT-1",
+ "GTGGAAGGTAATTAGG-1",
+ "GTGGAGACAGCAGTTT-1",
+ "GTGGAGAGTCGTCATA-1",
+ "GTGGAGAGTTCTAAGC-1",
+ "GTGGCGTCAGAACGCA-1",
+ "GTGGCGTGTAACCAGG-1",
+ "GTGGGAAAGTTGCCCG-1",
+ "GTGGGAATCTCCATAT-1",
+ "GTGTAACGTCGTCTCT-1",
+ "GTGTGATCAGCACACC-1",
+ "GTGTGATGTTCCCAAA-1",
+ "GTGTGATTCAGGAGAC-1",
+ "GTGTGGCGTCGTCATA-1",
+ "GTTACAGGTTCGGCGT-1",
+ "GTTACGAGTAGTCCTA-1",
+ "GTTAGACGTTCGGTAT-1",
+ "GTTAGTGCACGGTAGA-1",
+ "GTTATGGCAAACGTGG-1",
+ "GTTCATTAGACCTTTG-1",
+ "GTTCCGTTCCCGAGGT-1",
+ "GTTCCGTTCTTTGGAG-1",
+ "GTTCGCTAGATGACAT-1",
+ "GTTGCGGAGAATACAC-1",
+ "GTTGCGGGTACCGTCG-1",
+ "GTTGCGGTCACTGTTT-1",
+ "GTTGCTCTCTTAGCTT-1",
+ "GTTGTAGAGTTAACGA-1",
+ "GTTGTGAAGGCACAAC-1",
+ "GTTGTGAGTCATATGC-1",
+ "TAACGACTCCTAGAGT-1",
+ "TAAGCACGTATTGCCA-1",
+ "TAAGCACTCGGTCACG-1",
+ "TAAGCGTCATAGAAAC-1",
+ "TAAGTCGAGAGCATTA-1",
+ "TAAGTCGGTAGATTAG-1",
+ "TACAGGTAGTCTTCGA-1",
+ "TACAGGTGTCGCTTGG-1",
+ "TACATTCAGCGCATCC-1",
+ "TACCCGTAGATGGTAT-1",
+ "TACCGAAAGAACTTCC-1",
+ "TACCGAACAATTCGTG-1",
+ "TACCGGGCAATAGGAT-1",
+ "TACCGGGTCGTCAACA-1",
+ "TACGGGCGTCTCACGG-1",
+ "TACGGTAGTCATTGCA-1",
+ "TACTTACAGCCTATCA-1",
+ "TACTTACCAATGAGCG-1",
+ "TACTTACTCAGCTGAT-1",
+ "TACTTCAGTCCCTAAA-1",
+ "TACTTGTAGCAGTACG-1",
+ "TACTTGTCAGATCATC-1",
+ "TAGACCATCCACGTCT-1",
+ "TAGAGTCCAATCACGT-1",
+ "TAGATCGGTCGTGGAA-1",
+ "TAGATCGGTGGATACG-1",
+ "TAGCACAGTAGCCCTG-1",
+ "TAGCACATCCGGACGT-1",
+ "TAGGAGGAGCGCATCC-1",
+ "TAGGAGGTCACCCTCA-1",
+ "TAGGTTGAGAAGATCT-1",
+ "TATACCTAGGATATAC-1",
+ "TATACCTCACCTGCTT-1",
+ "TATATCCCAACTGTGT-1",
+ "TATATCCCACAGACGA-1",
+ "TATATCCCAGGCTATT-1",
+ "TATATCCGTATACGGG-1",
+ "TATATCCTCCAAGCAT-1",
+ "TATATCCTCGTTGTGA-1",
+ "TATCCTACACAAACGG-1",
+ "TATCTGTCAGACTCTA-1",
+ "TATCTTGCACACCGCA-1",
+ "TATGTTCCAGATCACT-1",
+ "TATGTTCTCCATCTGC-1",
+ "TATTCCATCTTCGATT-1",
+ "TATTGCTAGACATCCT-1",
+ "TATTGGGGTGCAGTGA-1",
+ "TATTGGGTCCGTGTCT-1",
+ "TATTGGGTCGATGCTA-1",
+ "TATTGGGTCGCTTGCT-1",
+ "TCAAGACGTTAGGGTG-1",
+ "TCAAGCACAGGATTCT-1",
+ "TCAAGTGAGGTAAGTT-1",
+ "TCAAGTGGTCGTATTG-1",
+ "TCACAAGCACCACTGG-1",
+ "TCACAAGCAGTCCGTG-1",
+ "TCACAAGTCCATTTGT-1",
+ "TCACATTAGTAGACCG-1",
+ "TCACATTCAGTCCCGA-1",
+ "TCACATTGTGGTCAAG-1",
+ "TCACATTGTGTGCTTA-1",
+ "TCACGGGCACATGAAA-1",
+ "TCACGGGGTGACTCGC-1",
+ "TCACGGGGTGGCGTAA-1",
+ "TCACTATAGCGATGCA-1",
+ "TCACTATAGGATGTTA-1",
+ "TCACTCGAGACTGGGT-1",
+ "TCACTCGGTTCTCAGA-1",
+ "TCAGCAAAGAGTTGTA-1",
+ "TCAGCAAGTGATTGGG-1",
+ "TCAGCCTAGAACTGAT-1",
+ "TCAGCCTAGAAGGGAT-1",
+ "TCAGGTACATCCAATG-1",
+ "TCAGTCCGTATGAAAC-1",
+ "TCAGTCCTCATTTGGG-1",
+ "TCAGTTTAGCGTCTCG-1",
+ "TCATATCGTAAGTAGT-1",
+ "TCATATCGTGCATGAG-1",
+ "TCATATCGTGTTACAC-1",
+ "TCATATCTCCCATAAG-1",
+ "TCATCATCACAATCTG-1",
+ "TCATCATCACTTCTCG-1",
+ "TCATCCGAGCCAGAGT-1",
+ "TCATCCGTCGCAGTGC-1",
+ "TCATGAGGTCTGTAGT-1",
+ "TCATGCCCACCTGCAG-1",
+ "TCATGTTGTGGTCAAG-1",
+ "TCATTCATCGGAGTGA-1",
+ "TCATTGTCAGCATTGT-1",
+ "TCCACCAAGACTTGTC-1",
+ "TCCACGTGTGAGAACC-1",
+ "TCCACGTGTTATGGTC-1",
+ "TCCACGTTCGTGTGGC-1",
+ "TCCACGTTCTTGATTC-1",
+ "TCCAGAACAAGTGACG-1",
+ "TCCATCGAGATGGCGT-1",
+ "TCCATCGGTATGCTTG-1",
+ "TCCCACACAGAGATGC-1",
+ "TCCCACATCACCTTGC-1",
+ "TCCCAGTGTAAGGCTG-1",
+ "TCCCATGAGATTGCGG-1",
+ "TCCGAAAAGTCCTGCG-1",
+ "TCCGAAATCCTGTAGA-1",
+ "TCCGTGTGTCCACATA-1",
+ "TCCTCCCGTGGAAGTC-1",
+ "TCCTCGAAGCCATTGT-1",
+ "TCCTCGAAGCGCCATC-1",
+ "TCCTCGATCCGGCAGT-1",
+ "TCCTCGATCGTGTTCC-1",
+ "TCCTGCAAGAGCTTTC-1",
+ "TCCTGCATCCCAGGCA-1",
+ "TCCTTTCAGCCTCTCT-1",
+ "TCGAACAAGCATGAAT-1",
+ "TCGAACACATGAGTAA-1",
+ "TCGAACAGTACTGGGA-1",
+ "TCGAACATCTACGCGG-1",
+ "TCGAAGTGTGGTCTCG-1",
+ "TCGACGGTCTTAGCCC-1",
+ "TCGATTTGTGTTGAGG-1",
+ "TCGCACTAGTTGGGAC-1",
+ "TCGCACTCAAGGGTCA-1",
+ "TCGCACTGTAGTGTGG-1",
+ "TCGCTCATCGCAACAT-1",
+ "TCGCTCATCGCAGTTA-1",
+ "TCGCTTGAGACTCCGC-1",
+ "TCGCTTGGTGTCCAAT-1",
+ "TCGGATACACCTATCC-1",
+ "TCGGATAGTTTGGGAG-1",
+ "TCGGGACTCTTCTGTA-1",
+ "TCGGGCACACCAGACC-1",
+ "TCGGGCATCTATTTCG-1",
+ "TCGGGTGCAAATTAGG-1",
+ "TCGGTCTCAACTGATC-1",
+ "TCGGTCTTCCACGTCT-1",
+ "TCGTAGACAAGTGACG-1",
+ "TCGTAGATCACACGAT-1",
+ "TCGTAGATCCGAGATT-1",
+ "TCGTAGATCGACATAC-1",
+ "TCGTCCACAAATCAGA-1",
+ "TCGTGCTTCTCCGTGT-1",
+ "TCTAACTGTAGCTTGT-1",
+ "TCTACCGAGAGGTCGT-1",
+ "TCTACCGTCTGCGGGT-1",
+ "TCTATACTCATTGTGG-1",
+ "TCTATACTCCGTAGGC-1",
+ "TCTCACGCATCATTTC-1",
+ "TCTCACGGTTGGAGAC-1",
+ "TCTCAGCCACCAGACC-1",
+ "TCTCAGCGTGAGATTA-1",
+ "TCTCTGGCATCTCAAG-1",
+ "TCTGCCACATAGGAGC-1",
+ "TCTGCCAGTGCTGCAC-1",
+ "TCTGGCTGTCCTCCTA-1",
+ "TCTGTCGCACTTGTCC-1",
+ "TCTGTCGTCAAGAGTA-1",
+ "TCTGTCGTCCTAAACG-1",
+ "TCTTAGTGTGCAGTGA-1",
+ "TCTTCCTTCGACGTCG-1",
+ "TCTTGCGCAATGTGGG-1",
+ "TCTTTGACAGAGTCTT-1",
+ "TCTTTGATCATGGGAG-1",
+ "TGAATCGGTACTAGCT-1",
+ "TGAATGCAGCTTTCTT-1",
+ "TGAATGCCATGACTGT-1",
+ "TGAATGCGTTGAAGTA-1",
+ "TGACTCCCATTGCCGG-1",
+ "TGACTCCGTGTTGATC-1",
+ "TGAGCATCAACTTGGT-1",
+ "TGAGCATGTCCGGATC-1",
+ "TGAGCGCAGAGGGTCT-1",
+ "TGAGGAGTCCTCTAAT-1",
+ "TGAGGGACATTCCTCG-1",
+ "TGAGTCACAGTCAACT-1",
+ "TGAGTCAGTATGGAGC-1",
+ "TGATCTTAGCTGCCTG-1",
+ "TGATGCAGTAGGCAGT-1",
+ "TGATTCTAGCACGTCC-1",
+ "TGATTTCGTAGCCAGA-1",
+ "TGCACGGAGGACGCAT-1",
+ "TGCACGGGTGTGGACA-1",
+ "TGCAGATCACAGCTTA-1",
+ "TGCAGATGTGCAGATG-1",
+ "TGCAGATTCGGTTAGT-1",
+ "TGCATCCGTGGATGAC-1",
+ "TGCCGAGAGGAGCAAA-1",
+ "TGCGACGGTACATTGC-1",
+ "TGCGATAAGCAAATCA-1",
+ "TGCGGCACAGGTCCCA-1",
+ "TGCGGGTCAGGTTCGC-1",
+ "TGCGGGTTCACCGCTT-1",
+ "TGCGGGTTCCTTATCA-1",
+ "TGCTCCAGTCACGACC-1",
+ "TGCTCCAGTCTCGGAC-1",
+ "TGCTCCATCCAGTGCG-1",
+ "TGCTCGTCACAAGCAG-1",
+ "TGCTCGTCACGGTGAA-1",
+ "TGCTGAAGTGTTCCAA-1",
+ "TGCTTCGTCTCACTCG-1",
+ "TGGAGAGGTGATTCTG-1",
+ "TGGAGGAAGGGTACAC-1",
+ "TGGGAGAAGATTGACA-1",
+ "TGGGATTCATATCTCT-1",
+ "TGGGATTCATGACTCA-1",
+ "TGGGATTTCAGAGCGA-1",
+ "TGGGCGTCAGGAGACT-1",
+ "TGGGCGTTCATGAGGG-1",
+ "TGGTACACACGACAGA-1",
+ "TGGTAGTCACGATAGG-1",
+ "TGGTGATCACGCGCAT-1",
+ "TGGTTAGCAACACGAG-1",
+ "TGGTTAGCAGGAACCA-1",
+ "TGTAACGAGGACTGGT-1",
+ "TGTAACGCAGCTTTGA-1",
+ "TGTAACGGTGGAACAC-1",
+ "TGTAAGCGTGTCACAT-1",
+ "TGTCAGAGTCTTCAAG-1",
+ "TGTCCCAGTTGAGTCT-1",
+ "TGTCCTGCAACCGTGC-1",
+ "TGTCCTGCACGAAGAC-1",
+ "TGTGATGGTCATGGCC-1",
+ "TGTGATGTCGTAACTG-1",
+ "TGTGATGTCTCCGCAT-1",
+ "TGTGGCGGTCTGATCA-1",
+ "TGTGTGACAATCGCGC-1",
+ "TGTTCCGTCGTAACCA-1",
+ "TGTTGGAAGAGTGAAG-1",
+ "TGTTGGATCGAGTCCG-1",
+ "TGTTTGTGTAATGCGG-1",
+ "TTACAGGCACTCTGCT-1",
+ "TTACAGGGTCCAAATC-1",
+ "TTACCATAGCTAATCC-1",
+ "TTACCATTCAAGCTGT-1",
+ "TTACCATTCCAACTAG-1",
+ "TTACCGCAGATTGACA-1",
+ "TTACCGCTCTCTCTTC-1",
+ "TTACGTTGTTCGTTCC-1",
+ "TTAGGCAAGTCTGTAC-1",
+ "TTAGGCACAAGTATAG-1",
+ "TTAGTCTTCCCTAGGG-1",
+ "TTATTGCAGGAGTCTG-1",
+ "TTATTGCCATAGAAAC-1",
+ "TTCAATCTCGATTCCC-1",
+ "TTCACCGAGATAGGGA-1",
+ "TTCACCGAGTATTAGG-1",
+ "TTCACGCGTGCACGCT-1",
+ "TTCACGCTCGCCGAGT-1",
+ "TTCAGGATCATTACTC-1",
+ "TTCATGTTCATGGTAC-1",
+ "TTCATTGAGTCAGCGA-1",
+ "TTCATTGCACCAGCGT-1",
+ "TTCCAATCACATCATG-1",
+ "TTCCAATCACTCCTTG-1",
+ "TTCCAATGTTCGGCCA-1",
+ "TTCCACGTCTAAACGC-1",
+ "TTCCGGTTCAGCGCAC-1",
+ "TTCCGGTTCGCCTATC-1",
+ "TTCCGGTTCTCGGCTT-1",
+ "TTCCGTGAGGGAGATA-1",
+ "TTCCGTGAGTTTGAGA-1",
+ "TTCCGTGGTAGTTAGA-1",
+ "TTCCTAAAGCCGATCC-1",
+ "TTCCTCTGTTAGGGAC-1",
+ "TTCCTTCAGCTAGTTC-1",
+ "TTCTAGTAGACTCGAG-1",
+ "TTCTAGTTCGTTGTTT-1",
+ "TTCTCTCAGATCCTAC-1",
+ "TTCTCTCCATCCGGCA-1",
+ "TTCTCTCGTTTGTTGG-1",
+ "TTCTGTAAGCTCGACC-1",
+ "TTCTTCCTCGCCATAA-1",
+ "TTGAACGTCCATTTAC-1",
+ "TTGAGTGAGGTATCTC-1",
+ "TTGAGTGCATCCAACA-1",
+ "TTGCATTCAAAGGCGT-1",
+ "TTGCGTCCAGTCGCTG-1",
+ "TTGCGTCTCATACGAC-1",
+ "TTGGATGAGTGTAGAT-1",
+ "TTGGATGTCATTGTGG-1",
+ "TTGGGATTCGTCGACG-1",
+ "TTGGGCGTCTCAGAAC-1",
+ "TTGGTTTAGGCACAAC-1",
+ "TTGTGGAAGGTCACAG-1",
+ "TTGTGGAGTAACGATA-1",
+ "TTGTTCAAGAATGTTG-1",
+ "TTGTTCAGTCTAACGT-1",
+ "TTGTTGTCACACGCCA-1",
+ "TTGTTGTGTCAGATTC-1",
+ "TTGTTGTGTGGAACAC-1",
+ "TTGTTTGTCATCGCTC-1",
+ "TTGTTTGTCTGCTTAT-1",
+ "TTTACCACATCCAACA-1",
+ "TTTACTGAGACGTCCC-1",
+ "TTTACTGTCCGATCTC-1",
+ "TTTATGCTCGGAAACG-1",
+ "TTTCACAGTCCGAAAG-1",
+ "TTTCACATCATACGGT-1",
+ "TTTCAGTAGACAACTA-1",
+ "TTTCAGTAGGGTACAC-1",
+ "TTTCAGTCAACAGCCC-1",
+ "TTTCAGTTCGCTGCGA-1",
+ "TTTCATGAGAACTTCC-1",
+ "TTTCCTCGTGAATTAG-1",
+ "TTTCCTCGTGGCTGCT-1",
+ "TTTCGATTCGTGCACG-1",
+ "TTTGACTGTCCACTTC-1",
+ "TTTGGAGAGGCCTAAG-1",
+ "TTTGGAGAGGTGCCTC-1",
+ "TTTGGAGTCCGTGACG-1",
+ "TTTGGTTAGCTACTAC-1",
+ "TTTGGTTGTCGATGCC-1",
+ "TTTGTTGTCCTGGGAC-1"
+ ],
+ "htoNames": [
+ "TNC_1_0_25x",
+ "TNC_1_0_5x",
+ "TNC_1_1x",
+ "TNC_1_2x",
+ "TNC_1_4x",
+ "TNC_2_0_25x",
+ "TNC_2_0_5x",
+ "TNC_2_1x",
+ "TNC_2_2x",
+ "TNC_2_4x"
+ ],
+ "rawCounts": [
+ [
+ 32,
+ 6,
+ 13,
+ 10,
+ 243,
+ 6,
+ 5,
+ 2,
+ 6,
+ 2
+ ],
+ [
+ 5,
+ 3,
+ 1,
+ 9,
+ 8,
+ 1,
+ 7,
+ 1,
+ 3,
+ 140
+ ],
+ [
+ 13,
+ 7,
+ 4,
+ 89,
+ 8,
+ 1,
+ 3,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 10,
+ 9,
+ 8,
+ 10,
+ 158,
+ 3,
+ 7,
+ 3,
+ 5,
+ 4
+ ],
+ [
+ 18,
+ 6,
+ 4,
+ 11,
+ 11,
+ 6,
+ 3,
+ 3,
+ 71,
+ 2
+ ],
+ [
+ 14,
+ 3,
+ 9,
+ 7,
+ 205,
+ 2,
+ 2,
+ 5,
+ 6,
+ 1
+ ],
+ [
+ 16,
+ 4,
+ 10,
+ 15,
+ 17,
+ 240,
+ 6,
+ 8,
+ 8,
+ 1
+ ],
+ [
+ 10,
+ 5,
+ 9,
+ 8,
+ 7,
+ 273,
+ 1,
+ 1,
+ 2,
+ 4
+ ],
+ [
+ 24,
+ 2,
+ 7,
+ 14,
+ 117,
+ 3,
+ 4,
+ 4,
+ 3,
+ 1
+ ],
+ [
+ 17,
+ 4,
+ 9,
+ 182,
+ 6,
+ 2,
+ 2,
+ 0,
+ 8,
+ 0
+ ],
+ [
+ 11,
+ 11,
+ 120,
+ 132,
+ 6,
+ 8,
+ 5,
+ 1,
+ 11,
+ 0
+ ],
+ [
+ 16,
+ 10,
+ 6,
+ 10,
+ 8,
+ 84,
+ 15,
+ 3,
+ 1,
+ 3
+ ],
+ [
+ 25,
+ 11,
+ 8,
+ 729,
+ 19,
+ 8,
+ 6,
+ 7,
+ 8,
+ 0
+ ],
+ [
+ 174,
+ 4,
+ 11,
+ 11,
+ 7,
+ 4,
+ 7,
+ 2,
+ 4,
+ 0
+ ],
+ [
+ 10,
+ 7,
+ 8,
+ 12,
+ 12,
+ 3,
+ 2,
+ 139,
+ 8,
+ 1
+ ],
+ [
+ 16,
+ 254,
+ 10,
+ 6,
+ 6,
+ 3,
+ 4,
+ 1,
+ 2,
+ 1
+ ],
+ [
+ 12,
+ 6,
+ 3,
+ 9,
+ 10,
+ 5,
+ 142,
+ 5,
+ 7,
+ 0
+ ],
+ [
+ 13,
+ 6,
+ 9,
+ 12,
+ 210,
+ 6,
+ 4,
+ 5,
+ 10,
+ 3
+ ],
+ [
+ 12,
+ 4,
+ 10,
+ 12,
+ 11,
+ 97,
+ 7,
+ 1,
+ 9,
+ 2
+ ],
+ [
+ 17,
+ 7,
+ 11,
+ 13,
+ 9,
+ 4,
+ 7,
+ 159,
+ 4,
+ 1
+ ],
+ [
+ 9,
+ 0,
+ 4,
+ 8,
+ 12,
+ 2,
+ 2,
+ 205,
+ 3,
+ 0
+ ],
+ [
+ 18,
+ 3,
+ 109,
+ 9,
+ 12,
+ 8,
+ 2,
+ 4,
+ 8,
+ 4
+ ],
+ [
+ 10,
+ 9,
+ 6,
+ 20,
+ 15,
+ 4,
+ 4,
+ 3,
+ 156,
+ 3
+ ],
+ [
+ 11,
+ 242,
+ 10,
+ 14,
+ 8,
+ 7,
+ 3,
+ 2,
+ 6,
+ 1
+ ],
+ [
+ 16,
+ 10,
+ 6,
+ 16,
+ 349,
+ 3,
+ 4,
+ 4,
+ 1,
+ 2
+ ],
+ [
+ 21,
+ 4,
+ 8,
+ 6,
+ 10,
+ 6,
+ 3,
+ 4,
+ 7,
+ 57
+ ],
+ [
+ 14,
+ 8,
+ 13,
+ 8,
+ 9,
+ 1,
+ 5,
+ 142,
+ 11,
+ 2
+ ],
+ [
+ 83,
+ 2,
+ 3,
+ 7,
+ 8,
+ 5,
+ 1,
+ 2,
+ 73,
+ 1
+ ],
+ [
+ 19,
+ 7,
+ 8,
+ 58,
+ 4,
+ 1,
+ 6,
+ 3,
+ 10,
+ 4
+ ],
+ [
+ 9,
+ 4,
+ 9,
+ 104,
+ 9,
+ 9,
+ 4,
+ 2,
+ 3,
+ 1
+ ],
+ [
+ 3,
+ 4,
+ 74,
+ 12,
+ 5,
+ 3,
+ 2,
+ 2,
+ 10,
+ 1
+ ],
+ [
+ 7,
+ 59,
+ 3,
+ 8,
+ 7,
+ 3,
+ 27,
+ 0,
+ 3,
+ 0
+ ],
+ [
+ 9,
+ 95,
+ 7,
+ 8,
+ 13,
+ 2,
+ 3,
+ 0,
+ 2,
+ 0
+ ],
+ [
+ 7,
+ 3,
+ 4,
+ 4,
+ 3,
+ 1,
+ 1,
+ 0,
+ 49,
+ 50
+ ],
+ [
+ 21,
+ 8,
+ 101,
+ 16,
+ 10,
+ 8,
+ 4,
+ 3,
+ 16,
+ 3
+ ],
+ [
+ 8,
+ 11,
+ 6,
+ 14,
+ 8,
+ 2,
+ 3,
+ 105,
+ 6,
+ 3
+ ],
+ [
+ 5,
+ 4,
+ 5,
+ 7,
+ 376,
+ 4,
+ 4,
+ 7,
+ 8,
+ 0
+ ],
+ [
+ 13,
+ 7,
+ 8,
+ 61,
+ 8,
+ 265,
+ 1,
+ 4,
+ 10,
+ 1
+ ],
+ [
+ 19,
+ 2,
+ 7,
+ 6,
+ 8,
+ 3,
+ 3,
+ 133,
+ 2,
+ 0
+ ],
+ [
+ 22,
+ 4,
+ 8,
+ 118,
+ 9,
+ 3,
+ 4,
+ 10,
+ 7,
+ 3
+ ],
+ [
+ 225,
+ 5,
+ 11,
+ 13,
+ 11,
+ 4,
+ 5,
+ 5,
+ 7,
+ 1
+ ],
+ [
+ 12,
+ 2,
+ 12,
+ 115,
+ 9,
+ 5,
+ 5,
+ 3,
+ 9,
+ 1
+ ],
+ [
+ 12,
+ 2,
+ 6,
+ 12,
+ 14,
+ 4,
+ 2,
+ 5,
+ 106,
+ 1
+ ],
+ [
+ 14,
+ 4,
+ 8,
+ 6,
+ 246,
+ 6,
+ 2,
+ 3,
+ 11,
+ 0
+ ],
+ [
+ 18,
+ 3,
+ 7,
+ 149,
+ 15,
+ 9,
+ 4,
+ 6,
+ 10,
+ 0
+ ],
+ [
+ 34,
+ 14,
+ 15,
+ 244,
+ 13,
+ 7,
+ 5,
+ 5,
+ 12,
+ 3
+ ],
+ [
+ 19,
+ 6,
+ 7,
+ 7,
+ 9,
+ 3,
+ 4,
+ 1,
+ 321,
+ 1
+ ],
+ [
+ 10,
+ 4,
+ 242,
+ 13,
+ 8,
+ 2,
+ 3,
+ 1,
+ 7,
+ 0
+ ],
+ [
+ 12,
+ 6,
+ 7,
+ 122,
+ 12,
+ 7,
+ 1,
+ 6,
+ 6,
+ 3
+ ],
+ [
+ 9,
+ 3,
+ 3,
+ 7,
+ 7,
+ 2,
+ 6,
+ 132,
+ 8,
+ 1
+ ],
+ [
+ 16,
+ 10,
+ 12,
+ 9,
+ 6,
+ 172,
+ 2,
+ 3,
+ 8,
+ 2
+ ],
+ [
+ 18,
+ 6,
+ 13,
+ 11,
+ 4,
+ 3,
+ 101,
+ 1,
+ 3,
+ 2
+ ],
+ [
+ 13,
+ 16,
+ 17,
+ 17,
+ 9,
+ 7,
+ 118,
+ 2,
+ 6,
+ 0
+ ],
+ [
+ 19,
+ 9,
+ 13,
+ 19,
+ 9,
+ 46,
+ 6,
+ 2,
+ 9,
+ 1
+ ],
+ [
+ 15,
+ 5,
+ 10,
+ 10,
+ 8,
+ 6,
+ 233,
+ 8,
+ 5,
+ 0
+ ],
+ [
+ 18,
+ 8,
+ 9,
+ 11,
+ 14,
+ 5,
+ 4,
+ 7,
+ 97,
+ 2
+ ],
+ [
+ 15,
+ 180,
+ 8,
+ 17,
+ 11,
+ 10,
+ 5,
+ 6,
+ 5,
+ 0
+ ],
+ [
+ 15,
+ 4,
+ 10,
+ 17,
+ 11,
+ 217,
+ 5,
+ 3,
+ 6,
+ 2
+ ],
+ [
+ 18,
+ 8,
+ 17,
+ 16,
+ 18,
+ 27,
+ 6,
+ 183,
+ 7,
+ 1
+ ],
+ [
+ 12,
+ 8,
+ 215,
+ 10,
+ 10,
+ 5,
+ 4,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 225,
+ 5,
+ 8,
+ 8,
+ 11,
+ 6,
+ 6,
+ 2,
+ 3,
+ 2
+ ],
+ [
+ 17,
+ 5,
+ 7,
+ 10,
+ 13,
+ 4,
+ 4,
+ 65,
+ 7,
+ 4
+ ],
+ [
+ 15,
+ 199,
+ 8,
+ 10,
+ 2,
+ 4,
+ 8,
+ 4,
+ 3,
+ 0
+ ],
+ [
+ 17,
+ 12,
+ 7,
+ 6,
+ 5,
+ 2,
+ 5,
+ 2,
+ 64,
+ 2
+ ],
+ [
+ 21,
+ 6,
+ 12,
+ 15,
+ 21,
+ 3,
+ 2,
+ 7,
+ 107,
+ 2
+ ],
+ [
+ 14,
+ 4,
+ 6,
+ 119,
+ 13,
+ 2,
+ 3,
+ 4,
+ 1,
+ 1
+ ],
+ [
+ 24,
+ 11,
+ 176,
+ 12,
+ 9,
+ 10,
+ 3,
+ 6,
+ 8,
+ 0
+ ],
+ [
+ 25,
+ 9,
+ 7,
+ 11,
+ 10,
+ 417,
+ 4,
+ 3,
+ 6,
+ 0
+ ],
+ [
+ 14,
+ 7,
+ 9,
+ 6,
+ 10,
+ 5,
+ 3,
+ 4,
+ 164,
+ 1
+ ],
+ [
+ 17,
+ 9,
+ 4,
+ 15,
+ 289,
+ 4,
+ 6,
+ 3,
+ 11,
+ 0
+ ],
+ [
+ 7,
+ 3,
+ 8,
+ 139,
+ 7,
+ 4,
+ 2,
+ 1,
+ 4,
+ 1
+ ],
+ [
+ 23,
+ 8,
+ 8,
+ 11,
+ 21,
+ 137,
+ 4,
+ 3,
+ 5,
+ 7
+ ],
+ [
+ 14,
+ 4,
+ 8,
+ 9,
+ 6,
+ 10,
+ 71,
+ 3,
+ 5,
+ 0
+ ],
+ [
+ 19,
+ 7,
+ 9,
+ 8,
+ 7,
+ 7,
+ 4,
+ 57,
+ 3,
+ 0
+ ],
+ [
+ 16,
+ 10,
+ 10,
+ 18,
+ 11,
+ 3,
+ 6,
+ 90,
+ 11,
+ 1
+ ],
+ [
+ 7,
+ 3,
+ 7,
+ 8,
+ 6,
+ 6,
+ 64,
+ 3,
+ 4,
+ 0
+ ],
+ [
+ 5,
+ 247,
+ 6,
+ 10,
+ 4,
+ 2,
+ 3,
+ 2,
+ 6,
+ 0
+ ],
+ [
+ 13,
+ 2,
+ 4,
+ 9,
+ 9,
+ 926,
+ 4,
+ 3,
+ 10,
+ 2
+ ],
+ [
+ 17,
+ 7,
+ 6,
+ 14,
+ 181,
+ 3,
+ 3,
+ 4,
+ 6,
+ 0
+ ],
+ [
+ 161,
+ 3,
+ 5,
+ 5,
+ 12,
+ 6,
+ 7,
+ 3,
+ 6,
+ 0
+ ],
+ [
+ 13,
+ 13,
+ 10,
+ 132,
+ 13,
+ 6,
+ 2,
+ 5,
+ 8,
+ 0
+ ],
+ [
+ 13,
+ 6,
+ 4,
+ 11,
+ 234,
+ 1,
+ 2,
+ 5,
+ 11,
+ 1
+ ],
+ [
+ 13,
+ 118,
+ 9,
+ 6,
+ 4,
+ 0,
+ 1,
+ 2,
+ 2,
+ 0
+ ],
+ [
+ 14,
+ 9,
+ 5,
+ 13,
+ 9,
+ 87,
+ 0,
+ 1,
+ 8,
+ 1
+ ],
+ [
+ 11,
+ 5,
+ 157,
+ 7,
+ 3,
+ 141,
+ 1,
+ 4,
+ 1,
+ 3
+ ],
+ [
+ 19,
+ 8,
+ 11,
+ 17,
+ 248,
+ 5,
+ 3,
+ 4,
+ 7,
+ 2
+ ],
+ [
+ 10,
+ 2,
+ 4,
+ 7,
+ 10,
+ 1,
+ 2,
+ 7,
+ 67,
+ 0
+ ],
+ [
+ 16,
+ 5,
+ 10,
+ 8,
+ 7,
+ 67,
+ 8,
+ 6,
+ 13,
+ 2
+ ],
+ [
+ 7,
+ 5,
+ 8,
+ 10,
+ 316,
+ 159,
+ 5,
+ 2,
+ 7,
+ 2
+ ],
+ [
+ 10,
+ 11,
+ 7,
+ 9,
+ 5,
+ 3,
+ 81,
+ 2,
+ 4,
+ 1
+ ],
+ [
+ 14,
+ 7,
+ 18,
+ 15,
+ 374,
+ 8,
+ 4,
+ 8,
+ 7,
+ 2
+ ],
+ [
+ 19,
+ 345,
+ 9,
+ 9,
+ 4,
+ 2,
+ 2,
+ 8,
+ 13,
+ 1
+ ],
+ [
+ 16,
+ 2,
+ 3,
+ 8,
+ 831,
+ 1,
+ 0,
+ 1,
+ 3,
+ 0
+ ],
+ [
+ 10,
+ 3,
+ 128,
+ 9,
+ 1,
+ 4,
+ 4,
+ 2,
+ 92,
+ 1
+ ],
+ [
+ 14,
+ 2,
+ 8,
+ 6,
+ 9,
+ 3,
+ 64,
+ 2,
+ 5,
+ 0
+ ],
+ [
+ 23,
+ 6,
+ 12,
+ 14,
+ 11,
+ 9,
+ 4,
+ 1,
+ 8,
+ 134
+ ],
+ [
+ 177,
+ 4,
+ 9,
+ 9,
+ 5,
+ 6,
+ 7,
+ 1,
+ 6,
+ 2
+ ],
+ [
+ 8,
+ 7,
+ 8,
+ 14,
+ 222,
+ 5,
+ 2,
+ 2,
+ 19,
+ 0
+ ],
+ [
+ 15,
+ 9,
+ 203,
+ 10,
+ 4,
+ 6,
+ 5,
+ 5,
+ 9,
+ 0
+ ],
+ [
+ 17,
+ 6,
+ 102,
+ 6,
+ 6,
+ 5,
+ 3,
+ 2,
+ 3,
+ 0
+ ],
+ [
+ 17,
+ 7,
+ 7,
+ 10,
+ 6,
+ 4,
+ 80,
+ 3,
+ 2,
+ 2
+ ],
+ [
+ 19,
+ 101,
+ 6,
+ 10,
+ 1,
+ 4,
+ 3,
+ 9,
+ 2,
+ 0
+ ],
+ [
+ 25,
+ 10,
+ 19,
+ 16,
+ 78,
+ 11,
+ 2,
+ 3,
+ 9,
+ 2
+ ],
+ [
+ 15,
+ 7,
+ 6,
+ 105,
+ 8,
+ 6,
+ 3,
+ 4,
+ 9,
+ 4
+ ],
+ [
+ 18,
+ 9,
+ 723,
+ 8,
+ 8,
+ 1,
+ 18,
+ 1,
+ 9,
+ 1
+ ],
+ [
+ 16,
+ 11,
+ 11,
+ 18,
+ 6,
+ 7,
+ 1,
+ 94,
+ 4,
+ 0
+ ],
+ [
+ 8,
+ 6,
+ 12,
+ 12,
+ 7,
+ 4,
+ 6,
+ 7,
+ 8,
+ 102
+ ],
+ [
+ 14,
+ 169,
+ 9,
+ 12,
+ 10,
+ 4,
+ 3,
+ 2,
+ 8,
+ 2
+ ],
+ [
+ 19,
+ 3,
+ 9,
+ 14,
+ 14,
+ 4,
+ 74,
+ 2,
+ 5,
+ 1
+ ],
+ [
+ 15,
+ 7,
+ 11,
+ 16,
+ 16,
+ 10,
+ 1,
+ 4,
+ 8,
+ 136
+ ],
+ [
+ 16,
+ 3,
+ 3,
+ 11,
+ 8,
+ 3,
+ 58,
+ 65,
+ 3,
+ 2
+ ],
+ [
+ 22,
+ 5,
+ 7,
+ 15,
+ 11,
+ 4,
+ 6,
+ 153,
+ 7,
+ 1
+ ],
+ [
+ 9,
+ 5,
+ 9,
+ 10,
+ 11,
+ 5,
+ 3,
+ 4,
+ 9,
+ 166
+ ],
+ [
+ 12,
+ 5,
+ 8,
+ 11,
+ 15,
+ 8,
+ 1,
+ 397,
+ 6,
+ 0
+ ],
+ [
+ 225,
+ 9,
+ 7,
+ 11,
+ 8,
+ 3,
+ 4,
+ 2,
+ 8,
+ 2
+ ],
+ [
+ 7,
+ 1,
+ 2,
+ 11,
+ 3,
+ 6,
+ 1,
+ 142,
+ 11,
+ 2
+ ],
+ [
+ 15,
+ 3,
+ 9,
+ 156,
+ 13,
+ 5,
+ 2,
+ 4,
+ 4,
+ 0
+ ],
+ [
+ 11,
+ 9,
+ 6,
+ 5,
+ 6,
+ 3,
+ 2,
+ 120,
+ 19,
+ 3
+ ],
+ [
+ 10,
+ 4,
+ 6,
+ 323,
+ 8,
+ 9,
+ 3,
+ 8,
+ 5,
+ 4
+ ],
+ [
+ 14,
+ 4,
+ 259,
+ 13,
+ 11,
+ 3,
+ 3,
+ 4,
+ 13,
+ 0
+ ],
+ [
+ 16,
+ 1,
+ 5,
+ 9,
+ 11,
+ 2,
+ 4,
+ 87,
+ 8,
+ 1
+ ],
+ [
+ 13,
+ 9,
+ 5,
+ 16,
+ 7,
+ 100,
+ 1,
+ 4,
+ 10,
+ 0
+ ],
+ [
+ 10,
+ 3,
+ 3,
+ 6,
+ 5,
+ 0,
+ 35,
+ 2,
+ 4,
+ 1
+ ],
+ [
+ 10,
+ 4,
+ 11,
+ 6,
+ 160,
+ 2,
+ 2,
+ 3,
+ 4,
+ 0
+ ],
+ [
+ 10,
+ 4,
+ 13,
+ 7,
+ 8,
+ 5,
+ 5,
+ 0,
+ 72,
+ 0
+ ],
+ [
+ 22,
+ 4,
+ 8,
+ 7,
+ 11,
+ 4,
+ 88,
+ 4,
+ 6,
+ 1
+ ],
+ [
+ 12,
+ 6,
+ 6,
+ 8,
+ 365,
+ 6,
+ 5,
+ 4,
+ 7,
+ 0
+ ],
+ [
+ 12,
+ 4,
+ 10,
+ 8,
+ 9,
+ 4,
+ 46,
+ 3,
+ 8,
+ 3
+ ],
+ [
+ 11,
+ 3,
+ 2,
+ 12,
+ 199,
+ 3,
+ 2,
+ 2,
+ 7,
+ 5
+ ],
+ [
+ 5,
+ 4,
+ 6,
+ 5,
+ 1,
+ 135,
+ 0,
+ 3,
+ 1,
+ 2
+ ],
+ [
+ 15,
+ 5,
+ 8,
+ 12,
+ 248,
+ 2,
+ 7,
+ 4,
+ 8,
+ 0
+ ],
+ [
+ 19,
+ 5,
+ 7,
+ 6,
+ 269,
+ 6,
+ 4,
+ 3,
+ 6,
+ 0
+ ],
+ [
+ 16,
+ 1,
+ 7,
+ 11,
+ 5,
+ 214,
+ 4,
+ 7,
+ 5,
+ 2
+ ],
+ [
+ 19,
+ 4,
+ 9,
+ 2,
+ 215,
+ 4,
+ 3,
+ 4,
+ 6,
+ 1
+ ],
+ [
+ 13,
+ 7,
+ 8,
+ 12,
+ 2,
+ 3,
+ 1,
+ 1,
+ 117,
+ 2
+ ],
+ [
+ 15,
+ 4,
+ 7,
+ 6,
+ 7,
+ 2,
+ 3,
+ 71,
+ 13,
+ 3
+ ],
+ [
+ 22,
+ 6,
+ 11,
+ 14,
+ 14,
+ 264,
+ 7,
+ 3,
+ 8,
+ 2
+ ],
+ [
+ 8,
+ 5,
+ 11,
+ 15,
+ 11,
+ 75,
+ 2,
+ 6,
+ 4,
+ 0
+ ],
+ [
+ 121,
+ 4,
+ 4,
+ 11,
+ 4,
+ 3,
+ 0,
+ 3,
+ 5,
+ 2
+ ],
+ [
+ 9,
+ 3,
+ 7,
+ 115,
+ 10,
+ 5,
+ 0,
+ 3,
+ 4,
+ 1
+ ],
+ [
+ 78,
+ 1,
+ 1,
+ 0,
+ 1,
+ 0,
+ 1,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 12,
+ 5,
+ 7,
+ 11,
+ 6,
+ 4,
+ 4,
+ 2,
+ 69,
+ 1
+ ],
+ [
+ 11,
+ 4,
+ 6,
+ 10,
+ 11,
+ 3,
+ 112,
+ 3,
+ 6,
+ 0
+ ],
+ [
+ 23,
+ 4,
+ 7,
+ 11,
+ 9,
+ 351,
+ 6,
+ 5,
+ 8,
+ 5
+ ],
+ [
+ 278,
+ 3,
+ 5,
+ 7,
+ 6,
+ 5,
+ 4,
+ 0,
+ 5,
+ 2
+ ],
+ [
+ 17,
+ 7,
+ 6,
+ 10,
+ 6,
+ 118,
+ 1,
+ 3,
+ 7,
+ 2
+ ],
+ [
+ 15,
+ 3,
+ 12,
+ 16,
+ 8,
+ 10,
+ 3,
+ 3,
+ 67,
+ 2
+ ],
+ [
+ 21,
+ 7,
+ 9,
+ 161,
+ 12,
+ 4,
+ 5,
+ 4,
+ 5,
+ 0
+ ],
+ [
+ 22,
+ 5,
+ 12,
+ 272,
+ 3,
+ 4,
+ 1,
+ 12,
+ 6,
+ 1
+ ],
+ [
+ 14,
+ 4,
+ 6,
+ 143,
+ 14,
+ 5,
+ 1,
+ 3,
+ 5,
+ 1
+ ],
+ [
+ 27,
+ 7,
+ 4,
+ 18,
+ 8,
+ 5,
+ 5,
+ 98,
+ 12,
+ 2
+ ],
+ [
+ 11,
+ 5,
+ 9,
+ 14,
+ 14,
+ 8,
+ 96,
+ 0,
+ 4,
+ 1
+ ],
+ [
+ 16,
+ 8,
+ 8,
+ 16,
+ 10,
+ 143,
+ 3,
+ 7,
+ 7,
+ 1
+ ],
+ [
+ 17,
+ 9,
+ 10,
+ 13,
+ 8,
+ 4,
+ 0,
+ 3,
+ 187,
+ 1
+ ],
+ [
+ 23,
+ 8,
+ 9,
+ 17,
+ 9,
+ 57,
+ 2,
+ 5,
+ 8,
+ 0
+ ],
+ [
+ 13,
+ 9,
+ 13,
+ 11,
+ 11,
+ 5,
+ 3,
+ 4,
+ 77,
+ 6
+ ],
+ [
+ 12,
+ 70,
+ 5,
+ 13,
+ 10,
+ 1,
+ 2,
+ 2,
+ 8,
+ 0
+ ],
+ [
+ 19,
+ 7,
+ 9,
+ 12,
+ 10,
+ 2,
+ 2,
+ 143,
+ 4,
+ 1
+ ],
+ [
+ 3,
+ 3,
+ 81,
+ 4,
+ 9,
+ 4,
+ 1,
+ 0,
+ 2,
+ 136
+ ],
+ [
+ 33,
+ 5,
+ 10,
+ 8,
+ 12,
+ 1,
+ 6,
+ 5,
+ 52,
+ 4
+ ],
+ [
+ 210,
+ 9,
+ 5,
+ 10,
+ 2,
+ 4,
+ 5,
+ 1,
+ 5,
+ 1
+ ],
+ [
+ 15,
+ 4,
+ 12,
+ 15,
+ 11,
+ 4,
+ 6,
+ 254,
+ 8,
+ 3
+ ],
+ [
+ 21,
+ 12,
+ 9,
+ 10,
+ 49,
+ 3,
+ 6,
+ 3,
+ 9,
+ 147
+ ],
+ [
+ 17,
+ 2,
+ 6,
+ 20,
+ 6,
+ 6,
+ 3,
+ 97,
+ 6,
+ 6
+ ],
+ [
+ 15,
+ 4,
+ 80,
+ 5,
+ 15,
+ 1,
+ 4,
+ 1,
+ 9,
+ 0
+ ],
+ [
+ 22,
+ 153,
+ 6,
+ 13,
+ 12,
+ 1,
+ 3,
+ 5,
+ 11,
+ 0
+ ],
+ [
+ 13,
+ 3,
+ 13,
+ 6,
+ 6,
+ 8,
+ 73,
+ 5,
+ 3,
+ 0
+ ],
+ [
+ 116,
+ 19,
+ 14,
+ 14,
+ 22,
+ 5,
+ 10,
+ 2,
+ 7,
+ 1
+ ],
+ [
+ 110,
+ 5,
+ 6,
+ 10,
+ 9,
+ 6,
+ 1,
+ 1,
+ 8,
+ 0
+ ],
+ [
+ 10,
+ 10,
+ 145,
+ 5,
+ 6,
+ 5,
+ 5,
+ 2,
+ 6,
+ 0
+ ],
+ [
+ 367,
+ 8,
+ 8,
+ 9,
+ 84,
+ 6,
+ 5,
+ 4,
+ 3,
+ 1
+ ],
+ [
+ 19,
+ 136,
+ 10,
+ 13,
+ 16,
+ 6,
+ 2,
+ 6,
+ 8,
+ 1
+ ],
+ [
+ 18,
+ 10,
+ 7,
+ 228,
+ 11,
+ 7,
+ 10,
+ 1,
+ 4,
+ 1
+ ],
+ [
+ 18,
+ 3,
+ 3,
+ 6,
+ 16,
+ 8,
+ 1,
+ 3,
+ 4,
+ 135
+ ],
+ [
+ 23,
+ 7,
+ 1,
+ 15,
+ 7,
+ 3,
+ 50,
+ 4,
+ 10,
+ 2
+ ],
+ [
+ 248,
+ 11,
+ 12,
+ 12,
+ 14,
+ 6,
+ 2,
+ 1,
+ 5,
+ 1
+ ],
+ [
+ 14,
+ 6,
+ 10,
+ 8,
+ 10,
+ 124,
+ 3,
+ 1,
+ 13,
+ 0
+ ],
+ [
+ 11,
+ 5,
+ 13,
+ 98,
+ 3,
+ 8,
+ 4,
+ 5,
+ 11,
+ 0
+ ],
+ [
+ 138,
+ 3,
+ 8,
+ 9,
+ 8,
+ 4,
+ 1,
+ 2,
+ 9,
+ 0
+ ],
+ [
+ 12,
+ 9,
+ 15,
+ 214,
+ 18,
+ 7,
+ 4,
+ 9,
+ 6,
+ 0
+ ],
+ [
+ 11,
+ 122,
+ 4,
+ 6,
+ 3,
+ 4,
+ 1,
+ 5,
+ 2,
+ 2
+ ],
+ [
+ 116,
+ 16,
+ 14,
+ 251,
+ 19,
+ 8,
+ 10,
+ 2,
+ 10,
+ 2
+ ],
+ [
+ 20,
+ 10,
+ 7,
+ 9,
+ 9,
+ 12,
+ 105,
+ 4,
+ 6,
+ 1
+ ],
+ [
+ 14,
+ 154,
+ 9,
+ 6,
+ 5,
+ 3,
+ 3,
+ 3,
+ 5,
+ 2
+ ],
+ [
+ 221,
+ 7,
+ 5,
+ 12,
+ 6,
+ 2,
+ 1,
+ 3,
+ 6,
+ 1
+ ],
+ [
+ 16,
+ 10,
+ 12,
+ 8,
+ 9,
+ 4,
+ 2,
+ 4,
+ 138,
+ 0
+ ],
+ [
+ 12,
+ 4,
+ 10,
+ 217,
+ 9,
+ 8,
+ 5,
+ 4,
+ 4,
+ 1
+ ],
+ [
+ 17,
+ 15,
+ 12,
+ 12,
+ 17,
+ 5,
+ 6,
+ 7,
+ 57,
+ 2
+ ],
+ [
+ 22,
+ 3,
+ 278,
+ 9,
+ 11,
+ 4,
+ 1,
+ 1,
+ 3,
+ 1
+ ],
+ [
+ 23,
+ 10,
+ 20,
+ 11,
+ 6,
+ 5,
+ 7,
+ 57,
+ 11,
+ 1
+ ],
+ [
+ 10,
+ 2,
+ 7,
+ 13,
+ 14,
+ 3,
+ 63,
+ 9,
+ 2,
+ 2
+ ],
+ [
+ 22,
+ 10,
+ 13,
+ 15,
+ 17,
+ 3,
+ 2,
+ 5,
+ 13,
+ 3
+ ],
+ [
+ 14,
+ 8,
+ 4,
+ 12,
+ 7,
+ 3,
+ 117,
+ 4,
+ 3,
+ 1
+ ],
+ [
+ 12,
+ 13,
+ 12,
+ 19,
+ 11,
+ 8,
+ 8,
+ 2,
+ 10,
+ 72
+ ],
+ [
+ 49,
+ 16,
+ 13,
+ 27,
+ 168,
+ 43,
+ 61,
+ 4,
+ 13,
+ 1
+ ],
+ [
+ 152,
+ 16,
+ 10,
+ 18,
+ 17,
+ 5,
+ 3,
+ 1,
+ 6,
+ 1
+ ],
+ [
+ 158,
+ 9,
+ 14,
+ 80,
+ 18,
+ 5,
+ 7,
+ 6,
+ 9,
+ 4
+ ],
+ [
+ 12,
+ 4,
+ 4,
+ 199,
+ 210,
+ 1,
+ 3,
+ 35,
+ 3,
+ 1
+ ],
+ [
+ 15,
+ 9,
+ 12,
+ 12,
+ 14,
+ 5,
+ 3,
+ 6,
+ 13,
+ 150
+ ],
+ [
+ 5,
+ 3,
+ 7,
+ 107,
+ 2,
+ 4,
+ 0,
+ 1,
+ 1,
+ 0
+ ],
+ [
+ 14,
+ 192,
+ 9,
+ 21,
+ 11,
+ 2,
+ 4,
+ 3,
+ 10,
+ 0
+ ],
+ [
+ 33,
+ 3,
+ 174,
+ 15,
+ 11,
+ 5,
+ 2,
+ 8,
+ 5,
+ 4
+ ],
+ [
+ 22,
+ 3,
+ 14,
+ 17,
+ 14,
+ 6,
+ 6,
+ 96,
+ 7,
+ 2
+ ],
+ [
+ 131,
+ 6,
+ 29,
+ 11,
+ 12,
+ 2,
+ 3,
+ 4,
+ 8,
+ 2
+ ],
+ [
+ 11,
+ 8,
+ 7,
+ 10,
+ 9,
+ 6,
+ 2,
+ 281,
+ 6,
+ 0
+ ],
+ [
+ 13,
+ 3,
+ 5,
+ 11,
+ 183,
+ 4,
+ 2,
+ 0,
+ 126,
+ 2
+ ],
+ [
+ 17,
+ 2,
+ 9,
+ 5,
+ 223,
+ 4,
+ 4,
+ 2,
+ 10,
+ 0
+ ],
+ [
+ 13,
+ 6,
+ 5,
+ 9,
+ 296,
+ 0,
+ 6,
+ 3,
+ 6,
+ 1
+ ],
+ [
+ 33,
+ 21,
+ 15,
+ 23,
+ 19,
+ 7,
+ 8,
+ 10,
+ 127,
+ 1
+ ],
+ [
+ 9,
+ 164,
+ 15,
+ 11,
+ 7,
+ 5,
+ 8,
+ 1,
+ 4,
+ 0
+ ],
+ [
+ 13,
+ 13,
+ 155,
+ 18,
+ 13,
+ 2,
+ 3,
+ 5,
+ 11,
+ 1
+ ],
+ [
+ 16,
+ 2,
+ 6,
+ 52,
+ 5,
+ 1,
+ 5,
+ 1,
+ 5,
+ 3
+ ],
+ [
+ 10,
+ 6,
+ 112,
+ 11,
+ 7,
+ 4,
+ 4,
+ 1,
+ 8,
+ 0
+ ],
+ [
+ 14,
+ 8,
+ 5,
+ 14,
+ 8,
+ 5,
+ 5,
+ 126,
+ 9,
+ 0
+ ],
+ [
+ 18,
+ 8,
+ 9,
+ 11,
+ 9,
+ 112,
+ 5,
+ 3,
+ 4,
+ 2
+ ],
+ [
+ 17,
+ 4,
+ 9,
+ 81,
+ 10,
+ 3,
+ 5,
+ 4,
+ 7,
+ 2
+ ],
+ [
+ 15,
+ 9,
+ 10,
+ 138,
+ 9,
+ 7,
+ 1,
+ 5,
+ 7,
+ 3
+ ],
+ [
+ 21,
+ 7,
+ 139,
+ 17,
+ 14,
+ 8,
+ 1,
+ 3,
+ 5,
+ 1
+ ],
+ [
+ 10,
+ 165,
+ 6,
+ 6,
+ 10,
+ 3,
+ 0,
+ 2,
+ 9,
+ 1
+ ],
+ [
+ 16,
+ 8,
+ 10,
+ 8,
+ 14,
+ 6,
+ 5,
+ 134,
+ 8,
+ 2
+ ],
+ [
+ 19,
+ 2,
+ 5,
+ 5,
+ 8,
+ 282,
+ 3,
+ 1,
+ 7,
+ 2
+ ],
+ [
+ 20,
+ 9,
+ 14,
+ 19,
+ 16,
+ 153,
+ 3,
+ 5,
+ 7,
+ 2
+ ],
+ [
+ 12,
+ 9,
+ 6,
+ 12,
+ 242,
+ 3,
+ 1,
+ 2,
+ 4,
+ 1
+ ],
+ [
+ 29,
+ 10,
+ 11,
+ 16,
+ 10,
+ 4,
+ 123,
+ 6,
+ 13,
+ 4
+ ],
+ [
+ 13,
+ 8,
+ 6,
+ 15,
+ 5,
+ 3,
+ 2,
+ 97,
+ 5,
+ 2
+ ],
+ [
+ 16,
+ 3,
+ 16,
+ 6,
+ 8,
+ 3,
+ 1,
+ 1,
+ 62,
+ 1
+ ],
+ [
+ 21,
+ 8,
+ 15,
+ 15,
+ 13,
+ 7,
+ 1,
+ 264,
+ 7,
+ 1
+ ],
+ [
+ 11,
+ 3,
+ 4,
+ 6,
+ 4,
+ 4,
+ 2,
+ 4,
+ 107,
+ 2
+ ],
+ [
+ 35,
+ 10,
+ 15,
+ 25,
+ 313,
+ 7,
+ 10,
+ 5,
+ 18,
+ 2
+ ],
+ [
+ 17,
+ 3,
+ 8,
+ 10,
+ 13,
+ 4,
+ 4,
+ 99,
+ 7,
+ 0
+ ],
+ [
+ 19,
+ 7,
+ 118,
+ 9,
+ 10,
+ 8,
+ 1,
+ 3,
+ 3,
+ 0
+ ],
+ [
+ 14,
+ 7,
+ 9,
+ 11,
+ 8,
+ 2,
+ 2,
+ 5,
+ 7,
+ 70
+ ],
+ [
+ 26,
+ 12,
+ 10,
+ 13,
+ 279,
+ 5,
+ 2,
+ 4,
+ 10,
+ 3
+ ],
+ [
+ 19,
+ 4,
+ 3,
+ 9,
+ 264,
+ 3,
+ 2,
+ 3,
+ 1,
+ 1
+ ],
+ [
+ 6,
+ 8,
+ 7,
+ 13,
+ 8,
+ 10,
+ 78,
+ 6,
+ 5,
+ 2
+ ],
+ [
+ 17,
+ 6,
+ 6,
+ 15,
+ 10,
+ 272,
+ 6,
+ 3,
+ 3,
+ 2
+ ],
+ [
+ 9,
+ 227,
+ 3,
+ 9,
+ 6,
+ 4,
+ 1,
+ 5,
+ 7,
+ 0
+ ],
+ [
+ 6,
+ 99,
+ 6,
+ 2,
+ 9,
+ 4,
+ 0,
+ 2,
+ 14,
+ 0
+ ],
+ [
+ 11,
+ 254,
+ 8,
+ 12,
+ 13,
+ 6,
+ 6,
+ 5,
+ 5,
+ 4
+ ],
+ [
+ 11,
+ 273,
+ 2,
+ 9,
+ 11,
+ 7,
+ 4,
+ 1,
+ 5,
+ 0
+ ],
+ [
+ 17,
+ 0,
+ 5,
+ 10,
+ 9,
+ 9,
+ 75,
+ 7,
+ 10,
+ 2
+ ],
+ [
+ 14,
+ 11,
+ 9,
+ 14,
+ 5,
+ 5,
+ 4,
+ 2,
+ 64,
+ 1
+ ],
+ [
+ 25,
+ 29,
+ 17,
+ 16,
+ 16,
+ 9,
+ 5,
+ 6,
+ 11,
+ 2
+ ],
+ [
+ 21,
+ 8,
+ 12,
+ 23,
+ 18,
+ 5,
+ 7,
+ 4,
+ 66,
+ 0
+ ],
+ [
+ 14,
+ 4,
+ 7,
+ 11,
+ 2,
+ 9,
+ 5,
+ 1,
+ 5,
+ 77
+ ],
+ [
+ 12,
+ 12,
+ 5,
+ 7,
+ 13,
+ 10,
+ 4,
+ 6,
+ 10,
+ 228
+ ],
+ [
+ 17,
+ 196,
+ 10,
+ 17,
+ 5,
+ 5,
+ 2,
+ 5,
+ 2,
+ 2
+ ],
+ [
+ 12,
+ 228,
+ 5,
+ 15,
+ 12,
+ 1,
+ 1,
+ 3,
+ 8,
+ 0
+ ],
+ [
+ 212,
+ 8,
+ 7,
+ 7,
+ 7,
+ 4,
+ 4,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 13,
+ 3,
+ 3,
+ 180,
+ 7,
+ 1,
+ 8,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 9,
+ 11,
+ 11,
+ 13,
+ 202,
+ 9,
+ 4,
+ 6,
+ 4,
+ 2
+ ],
+ [
+ 18,
+ 8,
+ 141,
+ 10,
+ 16,
+ 3,
+ 4,
+ 4,
+ 9,
+ 0
+ ],
+ [
+ 21,
+ 7,
+ 16,
+ 17,
+ 11,
+ 7,
+ 11,
+ 7,
+ 121,
+ 2
+ ],
+ [
+ 13,
+ 5,
+ 4,
+ 217,
+ 10,
+ 2,
+ 6,
+ 5,
+ 4,
+ 0
+ ],
+ [
+ 12,
+ 1,
+ 4,
+ 12,
+ 5,
+ 7,
+ 4,
+ 7,
+ 209,
+ 0
+ ],
+ [
+ 184,
+ 11,
+ 5,
+ 8,
+ 11,
+ 6,
+ 5,
+ 4,
+ 8,
+ 2
+ ],
+ [
+ 18,
+ 7,
+ 5,
+ 6,
+ 6,
+ 3,
+ 4,
+ 0,
+ 4,
+ 260
+ ],
+ [
+ 5,
+ 5,
+ 3,
+ 1,
+ 2,
+ 1,
+ 0,
+ 17,
+ 1,
+ 1
+ ],
+ [
+ 7,
+ 5,
+ 8,
+ 11,
+ 6,
+ 98,
+ 9,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 15,
+ 3,
+ 6,
+ 11,
+ 299,
+ 7,
+ 1,
+ 117,
+ 3,
+ 1
+ ],
+ [
+ 9,
+ 23,
+ 10,
+ 13,
+ 6,
+ 4,
+ 61,
+ 7,
+ 7,
+ 0
+ ],
+ [
+ 14,
+ 1,
+ 13,
+ 5,
+ 6,
+ 4,
+ 1,
+ 0,
+ 2,
+ 68
+ ],
+ [
+ 310,
+ 8,
+ 12,
+ 5,
+ 14,
+ 2,
+ 3,
+ 4,
+ 7,
+ 1
+ ],
+ [
+ 9,
+ 5,
+ 7,
+ 11,
+ 11,
+ 1,
+ 110,
+ 2,
+ 3,
+ 2
+ ],
+ [
+ 11,
+ 5,
+ 9,
+ 8,
+ 5,
+ 6,
+ 3,
+ 2,
+ 10,
+ 24
+ ],
+ [
+ 20,
+ 7,
+ 4,
+ 10,
+ 12,
+ 128,
+ 10,
+ 4,
+ 116,
+ 1
+ ],
+ [
+ 11,
+ 2,
+ 11,
+ 52,
+ 6,
+ 7,
+ 1,
+ 3,
+ 5,
+ 1
+ ],
+ [
+ 16,
+ 319,
+ 6,
+ 7,
+ 157,
+ 2,
+ 2,
+ 4,
+ 7,
+ 1
+ ],
+ [
+ 14,
+ 3,
+ 6,
+ 11,
+ 6,
+ 4,
+ 2,
+ 103,
+ 10,
+ 3
+ ],
+ [
+ 27,
+ 8,
+ 177,
+ 12,
+ 9,
+ 4,
+ 8,
+ 3,
+ 6,
+ 4
+ ],
+ [
+ 18,
+ 7,
+ 5,
+ 67,
+ 12,
+ 6,
+ 3,
+ 1,
+ 3,
+ 3
+ ],
+ [
+ 10,
+ 5,
+ 12,
+ 8,
+ 7,
+ 4,
+ 1,
+ 3,
+ 7,
+ 71
+ ],
+ [
+ 11,
+ 4,
+ 12,
+ 8,
+ 9,
+ 8,
+ 3,
+ 2,
+ 7,
+ 51
+ ],
+ [
+ 15,
+ 4,
+ 15,
+ 12,
+ 17,
+ 223,
+ 5,
+ 3,
+ 2,
+ 3
+ ],
+ [
+ 12,
+ 6,
+ 61,
+ 8,
+ 6,
+ 1,
+ 1,
+ 3,
+ 1,
+ 1
+ ],
+ [
+ 14,
+ 10,
+ 12,
+ 10,
+ 13,
+ 108,
+ 2,
+ 3,
+ 7,
+ 1
+ ],
+ [
+ 9,
+ 7,
+ 81,
+ 11,
+ 3,
+ 1,
+ 4,
+ 186,
+ 6,
+ 0
+ ],
+ [
+ 11,
+ 10,
+ 9,
+ 17,
+ 12,
+ 14,
+ 3,
+ 91,
+ 10,
+ 2
+ ],
+ [
+ 9,
+ 6,
+ 7,
+ 9,
+ 5,
+ 4,
+ 3,
+ 71,
+ 8,
+ 1
+ ],
+ [
+ 15,
+ 6,
+ 13,
+ 12,
+ 11,
+ 3,
+ 155,
+ 10,
+ 12,
+ 2
+ ],
+ [
+ 13,
+ 7,
+ 91,
+ 12,
+ 12,
+ 2,
+ 0,
+ 8,
+ 2,
+ 2
+ ],
+ [
+ 36,
+ 4,
+ 9,
+ 6,
+ 17,
+ 9,
+ 3,
+ 4,
+ 5,
+ 103
+ ],
+ [
+ 9,
+ 9,
+ 13,
+ 10,
+ 8,
+ 4,
+ 3,
+ 2,
+ 110,
+ 3
+ ],
+ [
+ 13,
+ 6,
+ 6,
+ 19,
+ 12,
+ 4,
+ 5,
+ 4,
+ 3,
+ 55
+ ],
+ [
+ 163,
+ 8,
+ 7,
+ 9,
+ 9,
+ 5,
+ 1,
+ 3,
+ 6,
+ 0
+ ],
+ [
+ 14,
+ 11,
+ 19,
+ 7,
+ 8,
+ 2,
+ 85,
+ 1,
+ 15,
+ 0
+ ],
+ [
+ 17,
+ 3,
+ 5,
+ 13,
+ 205,
+ 4,
+ 3,
+ 9,
+ 2,
+ 0
+ ],
+ [
+ 16,
+ 5,
+ 103,
+ 19,
+ 8,
+ 1,
+ 4,
+ 4,
+ 5,
+ 0
+ ],
+ [
+ 12,
+ 8,
+ 8,
+ 13,
+ 11,
+ 6,
+ 1,
+ 182,
+ 7,
+ 12
+ ],
+ [
+ 14,
+ 5,
+ 5,
+ 34,
+ 156,
+ 4,
+ 3,
+ 3,
+ 4,
+ 2
+ ],
+ [
+ 227,
+ 252,
+ 6,
+ 10,
+ 7,
+ 3,
+ 4,
+ 4,
+ 3,
+ 0
+ ],
+ [
+ 22,
+ 4,
+ 16,
+ 6,
+ 16,
+ 4,
+ 7,
+ 4,
+ 12,
+ 166
+ ],
+ [
+ 18,
+ 4,
+ 14,
+ 7,
+ 8,
+ 2,
+ 58,
+ 2,
+ 5,
+ 1
+ ],
+ [
+ 11,
+ 3,
+ 12,
+ 12,
+ 7,
+ 146,
+ 4,
+ 5,
+ 5,
+ 1
+ ],
+ [
+ 19,
+ 6,
+ 11,
+ 14,
+ 198,
+ 6,
+ 2,
+ 3,
+ 7,
+ 0
+ ],
+ [
+ 128,
+ 6,
+ 15,
+ 14,
+ 9,
+ 4,
+ 0,
+ 4,
+ 16,
+ 1
+ ],
+ [
+ 13,
+ 177,
+ 8,
+ 8,
+ 7,
+ 4,
+ 7,
+ 5,
+ 11,
+ 2
+ ],
+ [
+ 10,
+ 6,
+ 128,
+ 10,
+ 4,
+ 4,
+ 6,
+ 23,
+ 4,
+ 1
+ ],
+ [
+ 15,
+ 2,
+ 10,
+ 134,
+ 17,
+ 1,
+ 7,
+ 4,
+ 6,
+ 0
+ ],
+ [
+ 12,
+ 11,
+ 12,
+ 309,
+ 7,
+ 9,
+ 3,
+ 2,
+ 5,
+ 2
+ ],
+ [
+ 9,
+ 6,
+ 6,
+ 7,
+ 6,
+ 3,
+ 5,
+ 2,
+ 4,
+ 75
+ ],
+ [
+ 15,
+ 5,
+ 9,
+ 155,
+ 9,
+ 5,
+ 3,
+ 6,
+ 10,
+ 3
+ ],
+ [
+ 15,
+ 110,
+ 7,
+ 15,
+ 10,
+ 1,
+ 2,
+ 2,
+ 6,
+ 1
+ ],
+ [
+ 11,
+ 283,
+ 11,
+ 9,
+ 15,
+ 122,
+ 8,
+ 1,
+ 7,
+ 1
+ ],
+ [
+ 18,
+ 7,
+ 6,
+ 84,
+ 7,
+ 3,
+ 3,
+ 5,
+ 4,
+ 0
+ ],
+ [
+ 22,
+ 5,
+ 12,
+ 24,
+ 15,
+ 107,
+ 3,
+ 7,
+ 7,
+ 2
+ ],
+ [
+ 18,
+ 4,
+ 7,
+ 10,
+ 10,
+ 6,
+ 2,
+ 126,
+ 1,
+ 0
+ ],
+ [
+ 16,
+ 9,
+ 12,
+ 11,
+ 13,
+ 3,
+ 106,
+ 4,
+ 5,
+ 0
+ ],
+ [
+ 8,
+ 2,
+ 9,
+ 6,
+ 378,
+ 1,
+ 3,
+ 0,
+ 2,
+ 6
+ ],
+ [
+ 9,
+ 4,
+ 15,
+ 8,
+ 143,
+ 5,
+ 5,
+ 3,
+ 9,
+ 0
+ ],
+ [
+ 8,
+ 2,
+ 8,
+ 13,
+ 13,
+ 4,
+ 1,
+ 4,
+ 92,
+ 2
+ ],
+ [
+ 12,
+ 7,
+ 8,
+ 10,
+ 9,
+ 125,
+ 8,
+ 4,
+ 5,
+ 2
+ ],
+ [
+ 12,
+ 9,
+ 11,
+ 12,
+ 6,
+ 4,
+ 0,
+ 3,
+ 76,
+ 2
+ ],
+ [
+ 23,
+ 7,
+ 18,
+ 15,
+ 289,
+ 7,
+ 4,
+ 4,
+ 10,
+ 0
+ ],
+ [
+ 79,
+ 17,
+ 688,
+ 10,
+ 6,
+ 4,
+ 4,
+ 2,
+ 4,
+ 0
+ ],
+ [
+ 11,
+ 3,
+ 5,
+ 6,
+ 5,
+ 2,
+ 4,
+ 2,
+ 66,
+ 3
+ ],
+ [
+ 16,
+ 3,
+ 6,
+ 14,
+ 13,
+ 2,
+ 1,
+ 124,
+ 53,
+ 0
+ ],
+ [
+ 12,
+ 2,
+ 7,
+ 8,
+ 13,
+ 5,
+ 104,
+ 1,
+ 6,
+ 2
+ ],
+ [
+ 15,
+ 7,
+ 12,
+ 10,
+ 122,
+ 7,
+ 3,
+ 3,
+ 4,
+ 1
+ ],
+ [
+ 29,
+ 280,
+ 14,
+ 34,
+ 25,
+ 9,
+ 8,
+ 11,
+ 14,
+ 1
+ ],
+ [
+ 24,
+ 7,
+ 13,
+ 19,
+ 12,
+ 7,
+ 8,
+ 6,
+ 9,
+ 181
+ ],
+ [
+ 13,
+ 201,
+ 8,
+ 13,
+ 9,
+ 7,
+ 4,
+ 4,
+ 7,
+ 1
+ ],
+ [
+ 3,
+ 3,
+ 7,
+ 10,
+ 14,
+ 1,
+ 1,
+ 76,
+ 10,
+ 0
+ ],
+ [
+ 10,
+ 170,
+ 8,
+ 7,
+ 11,
+ 5,
+ 2,
+ 6,
+ 9,
+ 1
+ ],
+ [
+ 12,
+ 6,
+ 5,
+ 11,
+ 4,
+ 3,
+ 61,
+ 2,
+ 7,
+ 6
+ ],
+ [
+ 9,
+ 7,
+ 12,
+ 12,
+ 13,
+ 2,
+ 2,
+ 2,
+ 6,
+ 74
+ ],
+ [
+ 14,
+ 9,
+ 17,
+ 6,
+ 11,
+ 10,
+ 6,
+ 0,
+ 85,
+ 2
+ ],
+ [
+ 14,
+ 17,
+ 17,
+ 15,
+ 6,
+ 1,
+ 3,
+ 4,
+ 297,
+ 1
+ ],
+ [
+ 143,
+ 4,
+ 11,
+ 13,
+ 14,
+ 2,
+ 1,
+ 2,
+ 3,
+ 2
+ ],
+ [
+ 19,
+ 7,
+ 13,
+ 207,
+ 15,
+ 2,
+ 5,
+ 4,
+ 8,
+ 0
+ ],
+ [
+ 9,
+ 6,
+ 99,
+ 12,
+ 6,
+ 6,
+ 4,
+ 3,
+ 6,
+ 1
+ ],
+ [
+ 18,
+ 2,
+ 8,
+ 14,
+ 20,
+ 4,
+ 5,
+ 2,
+ 7,
+ 52
+ ],
+ [
+ 15,
+ 9,
+ 9,
+ 14,
+ 13,
+ 2,
+ 2,
+ 1,
+ 3,
+ 99
+ ],
+ [
+ 15,
+ 5,
+ 5,
+ 3,
+ 14,
+ 4,
+ 1,
+ 78,
+ 6,
+ 0
+ ],
+ [
+ 16,
+ 8,
+ 13,
+ 5,
+ 98,
+ 6,
+ 3,
+ 3,
+ 3,
+ 2
+ ],
+ [
+ 15,
+ 3,
+ 6,
+ 9,
+ 6,
+ 3,
+ 3,
+ 2,
+ 20,
+ 2
+ ],
+ [
+ 14,
+ 6,
+ 13,
+ 9,
+ 12,
+ 3,
+ 6,
+ 119,
+ 4,
+ 1
+ ],
+ [
+ 38,
+ 12,
+ 24,
+ 51,
+ 14,
+ 9,
+ 7,
+ 1,
+ 16,
+ 2
+ ],
+ [
+ 15,
+ 4,
+ 7,
+ 11,
+ 402,
+ 6,
+ 0,
+ 2,
+ 9,
+ 1
+ ],
+ [
+ 19,
+ 2,
+ 6,
+ 9,
+ 5,
+ 3,
+ 0,
+ 3,
+ 52,
+ 2
+ ],
+ [
+ 15,
+ 2,
+ 5,
+ 13,
+ 10,
+ 6,
+ 2,
+ 2,
+ 137,
+ 1
+ ],
+ [
+ 13,
+ 181,
+ 5,
+ 3,
+ 9,
+ 3,
+ 14,
+ 3,
+ 2,
+ 0
+ ],
+ [
+ 14,
+ 169,
+ 5,
+ 17,
+ 8,
+ 8,
+ 5,
+ 6,
+ 3,
+ 2
+ ],
+ [
+ 12,
+ 235,
+ 8,
+ 9,
+ 11,
+ 5,
+ 5,
+ 2,
+ 6,
+ 2
+ ],
+ [
+ 21,
+ 6,
+ 5,
+ 10,
+ 19,
+ 398,
+ 5,
+ 7,
+ 10,
+ 0
+ ],
+ [
+ 21,
+ 2,
+ 218,
+ 8,
+ 46,
+ 5,
+ 4,
+ 3,
+ 6,
+ 0
+ ],
+ [
+ 17,
+ 8,
+ 243,
+ 11,
+ 9,
+ 7,
+ 3,
+ 5,
+ 6,
+ 6
+ ],
+ [
+ 233,
+ 6,
+ 5,
+ 2,
+ 7,
+ 4,
+ 4,
+ 4,
+ 3,
+ 1
+ ],
+ [
+ 19,
+ 5,
+ 10,
+ 19,
+ 193,
+ 3,
+ 2,
+ 4,
+ 6,
+ 0
+ ],
+ [
+ 24,
+ 13,
+ 10,
+ 138,
+ 15,
+ 9,
+ 2,
+ 4,
+ 15,
+ 1
+ ],
+ [
+ 19,
+ 12,
+ 9,
+ 12,
+ 7,
+ 4,
+ 2,
+ 5,
+ 12,
+ 91
+ ],
+ [
+ 8,
+ 2,
+ 10,
+ 9,
+ 141,
+ 4,
+ 4,
+ 0,
+ 7,
+ 0
+ ],
+ [
+ 28,
+ 9,
+ 11,
+ 207,
+ 18,
+ 6,
+ 6,
+ 3,
+ 9,
+ 1
+ ],
+ [
+ 18,
+ 9,
+ 9,
+ 79,
+ 10,
+ 5,
+ 5,
+ 5,
+ 7,
+ 3
+ ],
+ [
+ 27,
+ 8,
+ 10,
+ 6,
+ 8,
+ 114,
+ 3,
+ 3,
+ 3,
+ 0
+ ],
+ [
+ 307,
+ 9,
+ 3,
+ 14,
+ 6,
+ 3,
+ 4,
+ 3,
+ 2,
+ 1
+ ],
+ [
+ 16,
+ 2,
+ 7,
+ 7,
+ 10,
+ 6,
+ 1,
+ 119,
+ 6,
+ 4
+ ],
+ [
+ 29,
+ 8,
+ 11,
+ 21,
+ 482,
+ 5,
+ 4,
+ 8,
+ 9,
+ 3
+ ],
+ [
+ 12,
+ 15,
+ 5,
+ 6,
+ 10,
+ 4,
+ 67,
+ 3,
+ 5,
+ 1
+ ],
+ [
+ 20,
+ 6,
+ 16,
+ 14,
+ 358,
+ 5,
+ 4,
+ 3,
+ 6,
+ 2
+ ],
+ [
+ 16,
+ 10,
+ 6,
+ 8,
+ 10,
+ 6,
+ 167,
+ 110,
+ 7,
+ 1
+ ],
+ [
+ 20,
+ 4,
+ 23,
+ 15,
+ 12,
+ 2,
+ 16,
+ 3,
+ 4,
+ 6
+ ],
+ [
+ 20,
+ 14,
+ 6,
+ 22,
+ 18,
+ 5,
+ 6,
+ 5,
+ 153,
+ 1
+ ],
+ [
+ 15,
+ 3,
+ 13,
+ 158,
+ 6,
+ 3,
+ 2,
+ 6,
+ 4,
+ 0
+ ],
+ [
+ 13,
+ 5,
+ 3,
+ 248,
+ 6,
+ 14,
+ 4,
+ 3,
+ 10,
+ 0
+ ],
+ [
+ 6,
+ 4,
+ 4,
+ 15,
+ 5,
+ 3,
+ 2,
+ 1,
+ 7,
+ 47
+ ],
+ [
+ 12,
+ 7,
+ 6,
+ 16,
+ 169,
+ 7,
+ 13,
+ 2,
+ 8,
+ 1
+ ],
+ [
+ 17,
+ 10,
+ 258,
+ 11,
+ 6,
+ 3,
+ 2,
+ 3,
+ 7,
+ 0
+ ],
+ [
+ 17,
+ 6,
+ 211,
+ 12,
+ 8,
+ 4,
+ 2,
+ 4,
+ 4,
+ 0
+ ],
+ [
+ 14,
+ 12,
+ 10,
+ 13,
+ 8,
+ 4,
+ 1,
+ 1,
+ 3,
+ 130
+ ],
+ [
+ 12,
+ 5,
+ 11,
+ 8,
+ 6,
+ 5,
+ 9,
+ 87,
+ 4,
+ 0
+ ],
+ [
+ 7,
+ 279,
+ 15,
+ 8,
+ 7,
+ 0,
+ 1,
+ 3,
+ 111,
+ 0
+ ],
+ [
+ 19,
+ 6,
+ 9,
+ 15,
+ 20,
+ 3,
+ 2,
+ 3,
+ 125,
+ 2
+ ],
+ [
+ 12,
+ 8,
+ 8,
+ 131,
+ 11,
+ 9,
+ 3,
+ 2,
+ 4,
+ 2
+ ],
+ [
+ 21,
+ 5,
+ 10,
+ 27,
+ 16,
+ 5,
+ 5,
+ 5,
+ 135,
+ 2
+ ],
+ [
+ 18,
+ 5,
+ 108,
+ 7,
+ 12,
+ 4,
+ 0,
+ 12,
+ 10,
+ 1
+ ],
+ [
+ 10,
+ 6,
+ 12,
+ 11,
+ 293,
+ 0,
+ 3,
+ 3,
+ 8,
+ 1
+ ],
+ [
+ 10,
+ 150,
+ 6,
+ 11,
+ 12,
+ 6,
+ 3,
+ 4,
+ 11,
+ 1
+ ],
+ [
+ 13,
+ 7,
+ 4,
+ 8,
+ 10,
+ 4,
+ 4,
+ 2,
+ 9,
+ 287
+ ],
+ [
+ 18,
+ 12,
+ 212,
+ 17,
+ 10,
+ 3,
+ 2,
+ 8,
+ 10,
+ 3
+ ],
+ [
+ 15,
+ 5,
+ 7,
+ 5,
+ 7,
+ 4,
+ 0,
+ 3,
+ 79,
+ 0
+ ],
+ [
+ 17,
+ 5,
+ 4,
+ 10,
+ 405,
+ 2,
+ 4,
+ 4,
+ 6,
+ 2
+ ],
+ [
+ 227,
+ 9,
+ 4,
+ 20,
+ 10,
+ 4,
+ 5,
+ 6,
+ 11,
+ 0
+ ],
+ [
+ 15,
+ 5,
+ 160,
+ 6,
+ 12,
+ 2,
+ 6,
+ 3,
+ 5,
+ 0
+ ],
+ [
+ 13,
+ 6,
+ 9,
+ 14,
+ 9,
+ 7,
+ 2,
+ 4,
+ 78,
+ 1
+ ],
+ [
+ 29,
+ 4,
+ 11,
+ 17,
+ 11,
+ 2,
+ 155,
+ 6,
+ 10,
+ 3
+ ],
+ [
+ 166,
+ 4,
+ 4,
+ 8,
+ 11,
+ 2,
+ 1,
+ 3,
+ 1,
+ 0
+ ],
+ [
+ 250,
+ 2,
+ 4,
+ 16,
+ 10,
+ 4,
+ 1,
+ 4,
+ 6,
+ 0
+ ],
+ [
+ 10,
+ 5,
+ 8,
+ 6,
+ 17,
+ 1,
+ 0,
+ 110,
+ 6,
+ 0
+ ],
+ [
+ 18,
+ 10,
+ 11,
+ 11,
+ 11,
+ 3,
+ 6,
+ 8,
+ 39,
+ 7
+ ],
+ [
+ 22,
+ 5,
+ 10,
+ 13,
+ 231,
+ 5,
+ 5,
+ 6,
+ 9,
+ 4
+ ],
+ [
+ 14,
+ 621,
+ 5,
+ 8,
+ 8,
+ 3,
+ 4,
+ 3,
+ 7,
+ 0
+ ],
+ [
+ 16,
+ 110,
+ 11,
+ 13,
+ 14,
+ 9,
+ 1,
+ 4,
+ 8,
+ 4
+ ],
+ [
+ 19,
+ 64,
+ 8,
+ 12,
+ 13,
+ 3,
+ 1,
+ 4,
+ 11,
+ 1
+ ],
+ [
+ 18,
+ 4,
+ 11,
+ 19,
+ 28,
+ 7,
+ 5,
+ 3,
+ 7,
+ 4
+ ],
+ [
+ 9,
+ 4,
+ 5,
+ 9,
+ 8,
+ 4,
+ 39,
+ 1,
+ 4,
+ 1
+ ],
+ [
+ 35,
+ 8,
+ 99,
+ 21,
+ 25,
+ 8,
+ 5,
+ 8,
+ 8,
+ 3
+ ],
+ [
+ 13,
+ 4,
+ 9,
+ 9,
+ 9,
+ 4,
+ 9,
+ 2,
+ 108,
+ 2
+ ],
+ [
+ 16,
+ 6,
+ 8,
+ 9,
+ 14,
+ 3,
+ 49,
+ 3,
+ 4,
+ 1
+ ],
+ [
+ 7,
+ 2,
+ 3,
+ 5,
+ 6,
+ 185,
+ 0,
+ 2,
+ 4,
+ 0
+ ],
+ [
+ 14,
+ 4,
+ 12,
+ 11,
+ 14,
+ 3,
+ 1,
+ 5,
+ 6,
+ 101
+ ],
+ [
+ 26,
+ 10,
+ 7,
+ 11,
+ 12,
+ 93,
+ 1,
+ 2,
+ 15,
+ 1
+ ],
+ [
+ 15,
+ 9,
+ 10,
+ 6,
+ 11,
+ 5,
+ 8,
+ 4,
+ 6,
+ 263
+ ],
+ [
+ 21,
+ 5,
+ 11,
+ 143,
+ 11,
+ 3,
+ 0,
+ 6,
+ 4,
+ 2
+ ],
+ [
+ 13,
+ 8,
+ 6,
+ 6,
+ 4,
+ 162,
+ 2,
+ 4,
+ 5,
+ 2
+ ],
+ [
+ 18,
+ 10,
+ 10,
+ 10,
+ 257,
+ 7,
+ 3,
+ 3,
+ 4,
+ 0
+ ],
+ [
+ 25,
+ 11,
+ 11,
+ 18,
+ 16,
+ 9,
+ 7,
+ 6,
+ 13,
+ 5
+ ],
+ [
+ 13,
+ 170,
+ 7,
+ 6,
+ 14,
+ 7,
+ 2,
+ 2,
+ 6,
+ 0
+ ],
+ [
+ 18,
+ 2,
+ 10,
+ 4,
+ 9,
+ 3,
+ 4,
+ 160,
+ 10,
+ 1
+ ],
+ [
+ 9,
+ 4,
+ 8,
+ 7,
+ 10,
+ 92,
+ 1,
+ 72,
+ 2,
+ 1
+ ],
+ [
+ 8,
+ 96,
+ 5,
+ 7,
+ 6,
+ 0,
+ 3,
+ 1,
+ 3,
+ 1
+ ],
+ [
+ 20,
+ 87,
+ 6,
+ 10,
+ 8,
+ 9,
+ 3,
+ 5,
+ 6,
+ 0
+ ],
+ [
+ 12,
+ 5,
+ 7,
+ 89,
+ 11,
+ 4,
+ 6,
+ 2,
+ 6,
+ 0
+ ],
+ [
+ 18,
+ 4,
+ 194,
+ 6,
+ 4,
+ 4,
+ 2,
+ 3,
+ 3,
+ 0
+ ],
+ [
+ 18,
+ 3,
+ 8,
+ 9,
+ 5,
+ 0,
+ 37,
+ 1,
+ 2,
+ 14
+ ],
+ [
+ 133,
+ 4,
+ 8,
+ 18,
+ 14,
+ 4,
+ 7,
+ 4,
+ 7,
+ 6
+ ],
+ [
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0
+ ],
+ [
+ 16,
+ 4,
+ 6,
+ 9,
+ 17,
+ 145,
+ 4,
+ 4,
+ 6,
+ 2
+ ],
+ [
+ 9,
+ 2,
+ 6,
+ 10,
+ 6,
+ 1,
+ 1,
+ 1,
+ 6,
+ 60
+ ],
+ [
+ 115,
+ 15,
+ 26,
+ 35,
+ 33,
+ 15,
+ 14,
+ 5,
+ 7,
+ 3
+ ],
+ [
+ 9,
+ 4,
+ 4,
+ 6,
+ 14,
+ 3,
+ 4,
+ 98,
+ 6,
+ 1
+ ],
+ [
+ 15,
+ 8,
+ 5,
+ 12,
+ 6,
+ 4,
+ 57,
+ 6,
+ 1,
+ 2
+ ],
+ [
+ 16,
+ 7,
+ 8,
+ 18,
+ 12,
+ 5,
+ 10,
+ 11,
+ 220,
+ 0
+ ],
+ [
+ 22,
+ 3,
+ 6,
+ 16,
+ 136,
+ 0,
+ 3,
+ 2,
+ 5,
+ 3
+ ],
+ [
+ 15,
+ 5,
+ 11,
+ 13,
+ 7,
+ 5,
+ 4,
+ 9,
+ 187,
+ 2
+ ],
+ [
+ 12,
+ 5,
+ 7,
+ 9,
+ 3,
+ 3,
+ 169,
+ 5,
+ 6,
+ 6
+ ],
+ [
+ 16,
+ 6,
+ 9,
+ 12,
+ 12,
+ 4,
+ 3,
+ 2,
+ 82,
+ 1
+ ],
+ [
+ 5,
+ 3,
+ 3,
+ 7,
+ 4,
+ 1,
+ 1,
+ 0,
+ 0,
+ 1
+ ],
+ [
+ 145,
+ 5,
+ 7,
+ 3,
+ 5,
+ 3,
+ 9,
+ 4,
+ 4,
+ 1
+ ],
+ [
+ 5,
+ 1,
+ 6,
+ 4,
+ 140,
+ 4,
+ 2,
+ 0,
+ 3,
+ 1
+ ],
+ [
+ 27,
+ 9,
+ 8,
+ 10,
+ 9,
+ 13,
+ 87,
+ 3,
+ 6,
+ 2
+ ],
+ [
+ 19,
+ 10,
+ 10,
+ 13,
+ 11,
+ 344,
+ 23,
+ 4,
+ 5,
+ 2
+ ],
+ [
+ 17,
+ 5,
+ 126,
+ 12,
+ 9,
+ 185,
+ 4,
+ 8,
+ 7,
+ 2
+ ],
+ [
+ 9,
+ 3,
+ 7,
+ 8,
+ 8,
+ 6,
+ 5,
+ 58,
+ 7,
+ 1
+ ],
+ [
+ 21,
+ 2,
+ 6,
+ 10,
+ 3,
+ 184,
+ 1,
+ 2,
+ 8,
+ 0
+ ],
+ [
+ 14,
+ 11,
+ 12,
+ 12,
+ 12,
+ 92,
+ 3,
+ 2,
+ 9,
+ 0
+ ],
+ [
+ 23,
+ 4,
+ 9,
+ 12,
+ 6,
+ 4,
+ 3,
+ 1,
+ 91,
+ 0
+ ],
+ [
+ 235,
+ 5,
+ 2,
+ 7,
+ 5,
+ 3,
+ 1,
+ 1,
+ 4,
+ 0
+ ],
+ [
+ 187,
+ 9,
+ 9,
+ 19,
+ 361,
+ 6,
+ 5,
+ 3,
+ 9,
+ 0
+ ],
+ [
+ 17,
+ 5,
+ 190,
+ 15,
+ 10,
+ 5,
+ 5,
+ 10,
+ 8,
+ 2
+ ],
+ [
+ 17,
+ 4,
+ 149,
+ 19,
+ 8,
+ 3,
+ 2,
+ 3,
+ 3,
+ 1
+ ],
+ [
+ 5,
+ 4,
+ 5,
+ 14,
+ 281,
+ 7,
+ 5,
+ 5,
+ 11,
+ 2
+ ],
+ [
+ 7,
+ 6,
+ 3,
+ 33,
+ 8,
+ 3,
+ 2,
+ 1,
+ 4,
+ 0
+ ],
+ [
+ 16,
+ 13,
+ 13,
+ 19,
+ 355,
+ 10,
+ 2,
+ 6,
+ 8,
+ 0
+ ],
+ [
+ 11,
+ 5,
+ 4,
+ 19,
+ 182,
+ 3,
+ 3,
+ 4,
+ 6,
+ 1
+ ],
+ [
+ 14,
+ 4,
+ 5,
+ 12,
+ 7,
+ 8,
+ 3,
+ 5,
+ 119,
+ 1
+ ],
+ [
+ 6,
+ 4,
+ 10,
+ 5,
+ 8,
+ 164,
+ 4,
+ 2,
+ 57,
+ 1
+ ],
+ [
+ 17,
+ 4,
+ 3,
+ 4,
+ 6,
+ 3,
+ 1,
+ 19,
+ 0,
+ 3
+ ],
+ [
+ 219,
+ 8,
+ 2,
+ 3,
+ 8,
+ 3,
+ 2,
+ 4,
+ 1,
+ 2
+ ],
+ [
+ 16,
+ 4,
+ 9,
+ 17,
+ 333,
+ 4,
+ 3,
+ 5,
+ 3,
+ 2
+ ],
+ [
+ 12,
+ 4,
+ 14,
+ 69,
+ 14,
+ 3,
+ 6,
+ 6,
+ 3,
+ 0
+ ],
+ [
+ 19,
+ 8,
+ 10,
+ 19,
+ 8,
+ 3,
+ 3,
+ 156,
+ 10,
+ 2
+ ],
+ [
+ 14,
+ 218,
+ 19,
+ 15,
+ 121,
+ 5,
+ 0,
+ 4,
+ 11,
+ 4
+ ],
+ [
+ 13,
+ 6,
+ 4,
+ 10,
+ 304,
+ 5,
+ 2,
+ 4,
+ 7,
+ 1
+ ],
+ [
+ 15,
+ 6,
+ 7,
+ 6,
+ 146,
+ 5,
+ 4,
+ 2,
+ 6,
+ 2
+ ],
+ [
+ 11,
+ 9,
+ 8,
+ 10,
+ 8,
+ 4,
+ 4,
+ 63,
+ 5,
+ 0
+ ],
+ [
+ 504,
+ 12,
+ 9,
+ 22,
+ 17,
+ 8,
+ 4,
+ 9,
+ 6,
+ 7
+ ],
+ [
+ 16,
+ 11,
+ 7,
+ 12,
+ 10,
+ 1,
+ 4,
+ 2,
+ 79,
+ 0
+ ],
+ [
+ 14,
+ 3,
+ 11,
+ 10,
+ 12,
+ 412,
+ 4,
+ 6,
+ 7,
+ 0
+ ],
+ [
+ 15,
+ 3,
+ 4,
+ 7,
+ 13,
+ 3,
+ 1,
+ 110,
+ 2,
+ 0
+ ],
+ [
+ 16,
+ 6,
+ 9,
+ 11,
+ 14,
+ 1,
+ 4,
+ 5,
+ 13,
+ 184
+ ],
+ [
+ 242,
+ 7,
+ 6,
+ 8,
+ 10,
+ 5,
+ 3,
+ 3,
+ 1,
+ 0
+ ],
+ [
+ 19,
+ 6,
+ 12,
+ 6,
+ 13,
+ 5,
+ 4,
+ 38,
+ 5,
+ 1
+ ],
+ [
+ 141,
+ 4,
+ 14,
+ 15,
+ 12,
+ 5,
+ 4,
+ 5,
+ 1,
+ 0
+ ],
+ [
+ 14,
+ 9,
+ 14,
+ 9,
+ 12,
+ 4,
+ 4,
+ 1,
+ 125,
+ 3
+ ],
+ [
+ 19,
+ 0,
+ 7,
+ 12,
+ 7,
+ 1,
+ 45,
+ 2,
+ 10,
+ 5
+ ],
+ [
+ 20,
+ 195,
+ 11,
+ 18,
+ 10,
+ 5,
+ 6,
+ 9,
+ 7,
+ 2
+ ],
+ [
+ 11,
+ 3,
+ 137,
+ 15,
+ 8,
+ 2,
+ 3,
+ 4,
+ 2,
+ 1
+ ],
+ [
+ 18,
+ 106,
+ 11,
+ 13,
+ 10,
+ 8,
+ 4,
+ 7,
+ 15,
+ 3
+ ],
+ [
+ 13,
+ 6,
+ 5,
+ 6,
+ 8,
+ 1,
+ 3,
+ 86,
+ 3,
+ 0
+ ],
+ [
+ 11,
+ 7,
+ 8,
+ 11,
+ 518,
+ 3,
+ 8,
+ 0,
+ 1,
+ 0
+ ],
+ [
+ 18,
+ 12,
+ 18,
+ 22,
+ 551,
+ 7,
+ 2,
+ 7,
+ 4,
+ 1
+ ],
+ [
+ 17,
+ 6,
+ 14,
+ 4,
+ 2,
+ 8,
+ 137,
+ 0,
+ 7,
+ 1
+ ],
+ [
+ 12,
+ 4,
+ 6,
+ 11,
+ 269,
+ 9,
+ 3,
+ 7,
+ 8,
+ 3
+ ],
+ [
+ 12,
+ 3,
+ 4,
+ 15,
+ 9,
+ 1,
+ 58,
+ 1,
+ 6,
+ 1
+ ],
+ [
+ 13,
+ 303,
+ 7,
+ 5,
+ 11,
+ 4,
+ 3,
+ 5,
+ 16,
+ 0
+ ],
+ [
+ 12,
+ 8,
+ 7,
+ 3,
+ 7,
+ 9,
+ 149,
+ 3,
+ 4,
+ 0
+ ],
+ [
+ 229,
+ 9,
+ 8,
+ 8,
+ 10,
+ 6,
+ 2,
+ 2,
+ 7,
+ 3
+ ],
+ [
+ 5,
+ 2,
+ 5,
+ 3,
+ 300,
+ 6,
+ 1,
+ 6,
+ 6,
+ 2
+ ],
+ [
+ 13,
+ 224,
+ 12,
+ 18,
+ 18,
+ 7,
+ 5,
+ 4,
+ 13,
+ 3
+ ],
+ [
+ 25,
+ 171,
+ 13,
+ 11,
+ 9,
+ 4,
+ 4,
+ 3,
+ 4,
+ 3
+ ],
+ [
+ 26,
+ 13,
+ 103,
+ 23,
+ 12,
+ 27,
+ 13,
+ 4,
+ 23,
+ 1
+ ],
+ [
+ 16,
+ 6,
+ 9,
+ 17,
+ 280,
+ 9,
+ 2,
+ 4,
+ 4,
+ 0
+ ],
+ [
+ 4,
+ 2,
+ 11,
+ 155,
+ 6,
+ 2,
+ 2,
+ 6,
+ 2,
+ 2
+ ],
+ [
+ 307,
+ 9,
+ 20,
+ 21,
+ 18,
+ 8,
+ 9,
+ 6,
+ 10,
+ 63
+ ],
+ [
+ 455,
+ 7,
+ 14,
+ 8,
+ 14,
+ 6,
+ 7,
+ 71,
+ 5,
+ 0
+ ],
+ [
+ 10,
+ 6,
+ 3,
+ 8,
+ 254,
+ 1,
+ 1,
+ 1,
+ 5,
+ 0
+ ],
+ [
+ 13,
+ 6,
+ 6,
+ 6,
+ 7,
+ 7,
+ 32,
+ 1,
+ 2,
+ 3
+ ],
+ [
+ 18,
+ 6,
+ 9,
+ 13,
+ 9,
+ 5,
+ 4,
+ 165,
+ 9,
+ 2
+ ],
+ [
+ 24,
+ 7,
+ 6,
+ 7,
+ 9,
+ 9,
+ 4,
+ 71,
+ 4,
+ 0
+ ],
+ [
+ 196,
+ 6,
+ 12,
+ 12,
+ 15,
+ 7,
+ 0,
+ 4,
+ 7,
+ 2
+ ],
+ [
+ 13,
+ 7,
+ 9,
+ 9,
+ 4,
+ 2,
+ 4,
+ 3,
+ 2,
+ 38
+ ],
+ [
+ 19,
+ 14,
+ 12,
+ 17,
+ 16,
+ 4,
+ 123,
+ 3,
+ 5,
+ 1
+ ],
+ [
+ 24,
+ 9,
+ 5,
+ 169,
+ 13,
+ 8,
+ 2,
+ 4,
+ 19,
+ 3
+ ],
+ [
+ 17,
+ 10,
+ 7,
+ 14,
+ 7,
+ 7,
+ 3,
+ 80,
+ 10,
+ 0
+ ],
+ [
+ 12,
+ 5,
+ 9,
+ 6,
+ 5,
+ 100,
+ 0,
+ 2,
+ 6,
+ 1
+ ],
+ [
+ 20,
+ 4,
+ 12,
+ 11,
+ 13,
+ 3,
+ 6,
+ 102,
+ 6,
+ 3
+ ],
+ [
+ 123,
+ 9,
+ 2,
+ 5,
+ 6,
+ 2,
+ 5,
+ 3,
+ 8,
+ 0
+ ],
+ [
+ 21,
+ 2,
+ 7,
+ 12,
+ 6,
+ 9,
+ 6,
+ 5,
+ 8,
+ 104
+ ],
+ [
+ 16,
+ 7,
+ 8,
+ 13,
+ 9,
+ 4,
+ 3,
+ 5,
+ 4,
+ 70
+ ],
+ [
+ 167,
+ 7,
+ 11,
+ 10,
+ 9,
+ 2,
+ 3,
+ 78,
+ 4,
+ 0
+ ],
+ [
+ 15,
+ 169,
+ 15,
+ 11,
+ 7,
+ 2,
+ 5,
+ 7,
+ 5,
+ 3
+ ],
+ [
+ 11,
+ 4,
+ 107,
+ 12,
+ 13,
+ 2,
+ 4,
+ 3,
+ 8,
+ 3
+ ],
+ [
+ 13,
+ 4,
+ 8,
+ 12,
+ 10,
+ 6,
+ 4,
+ 2,
+ 1,
+ 26
+ ],
+ [
+ 7,
+ 3,
+ 14,
+ 11,
+ 245,
+ 7,
+ 0,
+ 2,
+ 4,
+ 2
+ ],
+ [
+ 16,
+ 4,
+ 12,
+ 193,
+ 15,
+ 7,
+ 5,
+ 5,
+ 5,
+ 3
+ ],
+ [
+ 12,
+ 5,
+ 221,
+ 8,
+ 5,
+ 3,
+ 8,
+ 5,
+ 4,
+ 1
+ ],
+ [
+ 17,
+ 334,
+ 10,
+ 14,
+ 11,
+ 6,
+ 6,
+ 6,
+ 4,
+ 1
+ ],
+ [
+ 31,
+ 4,
+ 14,
+ 13,
+ 11,
+ 8,
+ 3,
+ 327,
+ 8,
+ 5
+ ],
+ [
+ 15,
+ 6,
+ 21,
+ 95,
+ 15,
+ 5,
+ 4,
+ 3,
+ 8,
+ 0
+ ],
+ [
+ 14,
+ 5,
+ 4,
+ 8,
+ 450,
+ 2,
+ 6,
+ 1,
+ 6,
+ 2
+ ],
+ [
+ 13,
+ 96,
+ 6,
+ 9,
+ 14,
+ 3,
+ 2,
+ 2,
+ 2,
+ 4
+ ],
+ [
+ 457,
+ 9,
+ 7,
+ 10,
+ 13,
+ 6,
+ 0,
+ 3,
+ 1,
+ 0
+ ],
+ [
+ 17,
+ 7,
+ 7,
+ 10,
+ 391,
+ 4,
+ 3,
+ 4,
+ 8,
+ 4
+ ],
+ [
+ 12,
+ 5,
+ 134,
+ 7,
+ 6,
+ 2,
+ 3,
+ 105,
+ 5,
+ 0
+ ],
+ [
+ 9,
+ 0,
+ 12,
+ 79,
+ 30,
+ 5,
+ 5,
+ 3,
+ 7,
+ 0
+ ],
+ [
+ 18,
+ 9,
+ 9,
+ 56,
+ 8,
+ 7,
+ 2,
+ 4,
+ 9,
+ 3
+ ],
+ [
+ 29,
+ 4,
+ 11,
+ 9,
+ 7,
+ 3,
+ 0,
+ 133,
+ 7,
+ 1
+ ],
+ [
+ 29,
+ 5,
+ 9,
+ 15,
+ 18,
+ 6,
+ 6,
+ 8,
+ 10,
+ 46
+ ],
+ [
+ 18,
+ 112,
+ 8,
+ 14,
+ 255,
+ 8,
+ 1,
+ 4,
+ 8,
+ 1
+ ],
+ [
+ 17,
+ 8,
+ 152,
+ 7,
+ 7,
+ 2,
+ 8,
+ 6,
+ 7,
+ 2
+ ],
+ [
+ 26,
+ 12,
+ 13,
+ 22,
+ 276,
+ 19,
+ 43,
+ 4,
+ 13,
+ 4
+ ],
+ [
+ 10,
+ 7,
+ 6,
+ 8,
+ 4,
+ 156,
+ 2,
+ 2,
+ 7,
+ 1
+ ],
+ [
+ 7,
+ 7,
+ 245,
+ 14,
+ 4,
+ 1,
+ 2,
+ 3,
+ 4,
+ 0
+ ],
+ [
+ 12,
+ 8,
+ 11,
+ 19,
+ 92,
+ 4,
+ 7,
+ 6,
+ 10,
+ 5
+ ],
+ [
+ 20,
+ 5,
+ 8,
+ 12,
+ 5,
+ 5,
+ 5,
+ 155,
+ 10,
+ 1
+ ],
+ [
+ 14,
+ 9,
+ 70,
+ 8,
+ 7,
+ 2,
+ 2,
+ 2,
+ 93,
+ 0
+ ],
+ [
+ 14,
+ 4,
+ 3,
+ 4,
+ 12,
+ 2,
+ 3,
+ 0,
+ 81,
+ 2
+ ],
+ [
+ 38,
+ 3,
+ 6,
+ 92,
+ 11,
+ 2,
+ 1,
+ 4,
+ 18,
+ 0
+ ],
+ [
+ 45,
+ 5,
+ 2,
+ 9,
+ 10,
+ 5,
+ 58,
+ 3,
+ 3,
+ 1
+ ],
+ [
+ 8,
+ 4,
+ 5,
+ 114,
+ 4,
+ 2,
+ 2,
+ 0,
+ 4,
+ 1
+ ],
+ [
+ 17,
+ 6,
+ 5,
+ 12,
+ 6,
+ 1,
+ 1,
+ 191,
+ 6,
+ 0
+ ],
+ [
+ 14,
+ 7,
+ 13,
+ 6,
+ 143,
+ 1,
+ 4,
+ 1,
+ 8,
+ 2
+ ],
+ [
+ 17,
+ 11,
+ 9,
+ 19,
+ 6,
+ 3,
+ 7,
+ 6,
+ 3,
+ 119
+ ],
+ [
+ 219,
+ 6,
+ 12,
+ 16,
+ 8,
+ 10,
+ 6,
+ 4,
+ 7,
+ 0
+ ],
+ [
+ 20,
+ 7,
+ 8,
+ 19,
+ 10,
+ 173,
+ 4,
+ 2,
+ 6,
+ 0
+ ],
+ [
+ 25,
+ 13,
+ 10,
+ 20,
+ 12,
+ 9,
+ 9,
+ 366,
+ 10,
+ 2
+ ],
+ [
+ 211,
+ 6,
+ 7,
+ 5,
+ 7,
+ 3,
+ 3,
+ 6,
+ 6,
+ 0
+ ],
+ [
+ 20,
+ 3,
+ 5,
+ 13,
+ 12,
+ 4,
+ 3,
+ 1,
+ 93,
+ 1
+ ],
+ [
+ 11,
+ 5,
+ 13,
+ 8,
+ 10,
+ 2,
+ 43,
+ 5,
+ 3,
+ 4
+ ],
+ [
+ 22,
+ 3,
+ 8,
+ 6,
+ 11,
+ 6,
+ 4,
+ 174,
+ 7,
+ 4
+ ],
+ [
+ 23,
+ 7,
+ 9,
+ 11,
+ 7,
+ 2,
+ 111,
+ 0,
+ 9,
+ 4
+ ],
+ [
+ 9,
+ 4,
+ 89,
+ 5,
+ 6,
+ 5,
+ 1,
+ 2,
+ 0,
+ 3
+ ],
+ [
+ 20,
+ 1,
+ 7,
+ 8,
+ 6,
+ 8,
+ 3,
+ 5,
+ 8,
+ 62
+ ],
+ [
+ 14,
+ 6,
+ 6,
+ 8,
+ 6,
+ 4,
+ 144,
+ 3,
+ 7,
+ 1
+ ],
+ [
+ 9,
+ 10,
+ 5,
+ 10,
+ 6,
+ 5,
+ 83,
+ 2,
+ 5,
+ 1
+ ],
+ [
+ 350,
+ 157,
+ 9,
+ 17,
+ 8,
+ 8,
+ 4,
+ 5,
+ 5,
+ 0
+ ],
+ [
+ 6,
+ 140,
+ 2,
+ 8,
+ 8,
+ 2,
+ 3,
+ 2,
+ 4,
+ 0
+ ],
+ [
+ 14,
+ 4,
+ 7,
+ 135,
+ 15,
+ 3,
+ 6,
+ 0,
+ 2,
+ 4
+ ],
+ [
+ 10,
+ 4,
+ 6,
+ 14,
+ 324,
+ 3,
+ 2,
+ 11,
+ 6,
+ 2
+ ],
+ [
+ 15,
+ 5,
+ 12,
+ 7,
+ 11,
+ 304,
+ 4,
+ 1,
+ 2,
+ 0
+ ],
+ [
+ 17,
+ 4,
+ 8,
+ 9,
+ 435,
+ 5,
+ 6,
+ 6,
+ 6,
+ 1
+ ],
+ [
+ 24,
+ 11,
+ 17,
+ 101,
+ 12,
+ 4,
+ 14,
+ 6,
+ 4,
+ 6
+ ],
+ [
+ 17,
+ 6,
+ 20,
+ 102,
+ 9,
+ 4,
+ 3,
+ 2,
+ 5,
+ 2
+ ],
+ [
+ 13,
+ 2,
+ 9,
+ 133,
+ 11,
+ 7,
+ 3,
+ 1,
+ 5,
+ 2
+ ],
+ [
+ 9,
+ 5,
+ 9,
+ 13,
+ 2,
+ 4,
+ 8,
+ 4,
+ 79,
+ 1
+ ],
+ [
+ 19,
+ 5,
+ 95,
+ 15,
+ 6,
+ 6,
+ 5,
+ 0,
+ 7,
+ 0
+ ],
+ [
+ 14,
+ 76,
+ 8,
+ 10,
+ 7,
+ 5,
+ 3,
+ 8,
+ 4,
+ 0
+ ],
+ [
+ 17,
+ 6,
+ 5,
+ 226,
+ 10,
+ 2,
+ 3,
+ 6,
+ 5,
+ 2
+ ],
+ [
+ 14,
+ 1,
+ 4,
+ 10,
+ 7,
+ 4,
+ 0,
+ 98,
+ 64,
+ 5
+ ],
+ [
+ 15,
+ 82,
+ 7,
+ 5,
+ 4,
+ 5,
+ 3,
+ 3,
+ 5,
+ 2
+ ],
+ [
+ 14,
+ 7,
+ 6,
+ 9,
+ 7,
+ 2,
+ 4,
+ 5,
+ 69,
+ 0
+ ],
+ [
+ 254,
+ 10,
+ 10,
+ 8,
+ 10,
+ 6,
+ 6,
+ 5,
+ 7,
+ 0
+ ],
+ [
+ 14,
+ 6,
+ 6,
+ 107,
+ 9,
+ 3,
+ 2,
+ 2,
+ 5,
+ 0
+ ],
+ [
+ 14,
+ 104,
+ 8,
+ 10,
+ 4,
+ 3,
+ 5,
+ 7,
+ 4,
+ 1
+ ],
+ [
+ 14,
+ 6,
+ 11,
+ 14,
+ 11,
+ 5,
+ 10,
+ 89,
+ 6,
+ 3
+ ],
+ [
+ 11,
+ 7,
+ 6,
+ 13,
+ 10,
+ 3,
+ 3,
+ 2,
+ 4,
+ 133
+ ],
+ [
+ 13,
+ 8,
+ 5,
+ 30,
+ 14,
+ 11,
+ 3,
+ 122,
+ 14,
+ 2
+ ],
+ [
+ 11,
+ 2,
+ 105,
+ 6,
+ 9,
+ 3,
+ 3,
+ 1,
+ 2,
+ 0
+ ],
+ [
+ 28,
+ 10,
+ 308,
+ 18,
+ 12,
+ 5,
+ 3,
+ 4,
+ 16,
+ 0
+ ],
+ [
+ 18,
+ 6,
+ 63,
+ 14,
+ 7,
+ 3,
+ 3,
+ 2,
+ 2,
+ 4
+ ],
+ [
+ 24,
+ 17,
+ 12,
+ 19,
+ 11,
+ 9,
+ 10,
+ 136,
+ 28,
+ 5
+ ],
+ [
+ 17,
+ 6,
+ 10,
+ 13,
+ 5,
+ 3,
+ 2,
+ 4,
+ 123,
+ 1
+ ],
+ [
+ 10,
+ 11,
+ 7,
+ 19,
+ 14,
+ 3,
+ 3,
+ 107,
+ 7,
+ 0
+ ],
+ [
+ 11,
+ 2,
+ 4,
+ 13,
+ 9,
+ 1,
+ 1,
+ 5,
+ 132,
+ 0
+ ],
+ [
+ 19,
+ 182,
+ 6,
+ 7,
+ 8,
+ 2,
+ 8,
+ 4,
+ 7,
+ 2
+ ],
+ [
+ 23,
+ 8,
+ 14,
+ 98,
+ 17,
+ 6,
+ 4,
+ 6,
+ 12,
+ 1
+ ],
+ [
+ 15,
+ 7,
+ 10,
+ 9,
+ 9,
+ 4,
+ 1,
+ 7,
+ 8,
+ 124
+ ],
+ [
+ 15,
+ 83,
+ 4,
+ 10,
+ 6,
+ 4,
+ 147,
+ 2,
+ 2,
+ 1
+ ],
+ [
+ 9,
+ 3,
+ 4,
+ 10,
+ 4,
+ 3,
+ 8,
+ 0,
+ 73,
+ 0
+ ],
+ [
+ 13,
+ 2,
+ 9,
+ 10,
+ 14,
+ 186,
+ 4,
+ 5,
+ 3,
+ 0
+ ],
+ [
+ 10,
+ 7,
+ 10,
+ 85,
+ 9,
+ 3,
+ 2,
+ 4,
+ 3,
+ 2
+ ],
+ [
+ 8,
+ 5,
+ 5,
+ 9,
+ 13,
+ 170,
+ 5,
+ 3,
+ 5,
+ 2
+ ],
+ [
+ 218,
+ 7,
+ 11,
+ 9,
+ 9,
+ 8,
+ 3,
+ 5,
+ 4,
+ 1
+ ],
+ [
+ 17,
+ 10,
+ 16,
+ 16,
+ 95,
+ 9,
+ 7,
+ 2,
+ 16,
+ 78
+ ],
+ [
+ 9,
+ 6,
+ 7,
+ 154,
+ 11,
+ 2,
+ 5,
+ 3,
+ 8,
+ 0
+ ],
+ [
+ 20,
+ 8,
+ 13,
+ 12,
+ 18,
+ 4,
+ 5,
+ 2,
+ 182,
+ 0
+ ],
+ [
+ 24,
+ 7,
+ 9,
+ 17,
+ 10,
+ 3,
+ 3,
+ 85,
+ 6,
+ 2
+ ],
+ [
+ 423,
+ 9,
+ 12,
+ 15,
+ 38,
+ 2,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 15,
+ 88,
+ 8,
+ 13,
+ 11,
+ 3,
+ 3,
+ 5,
+ 2,
+ 1
+ ],
+ [
+ 38,
+ 14,
+ 27,
+ 33,
+ 835,
+ 7,
+ 2,
+ 5,
+ 20,
+ 2
+ ],
+ [
+ 18,
+ 8,
+ 13,
+ 172,
+ 8,
+ 5,
+ 2,
+ 5,
+ 5,
+ 1
+ ],
+ [
+ 175,
+ 6,
+ 7,
+ 13,
+ 6,
+ 3,
+ 6,
+ 5,
+ 10,
+ 3
+ ],
+ [
+ 12,
+ 4,
+ 4,
+ 14,
+ 8,
+ 8,
+ 189,
+ 4,
+ 2,
+ 1
+ ],
+ [
+ 24,
+ 4,
+ 10,
+ 10,
+ 14,
+ 4,
+ 272,
+ 3,
+ 9,
+ 5
+ ],
+ [
+ 19,
+ 9,
+ 5,
+ 13,
+ 12,
+ 7,
+ 6,
+ 1,
+ 173,
+ 0
+ ],
+ [
+ 9,
+ 2,
+ 15,
+ 6,
+ 10,
+ 3,
+ 1,
+ 3,
+ 6,
+ 224
+ ],
+ [
+ 30,
+ 20,
+ 20,
+ 40,
+ 389,
+ 10,
+ 3,
+ 9,
+ 15,
+ 6
+ ],
+ [
+ 15,
+ 6,
+ 10,
+ 11,
+ 6,
+ 8,
+ 58,
+ 3,
+ 8,
+ 2
+ ],
+ [
+ 26,
+ 6,
+ 10,
+ 13,
+ 8,
+ 328,
+ 5,
+ 5,
+ 7,
+ 6
+ ],
+ [
+ 8,
+ 104,
+ 3,
+ 12,
+ 6,
+ 6,
+ 2,
+ 161,
+ 5,
+ 0
+ ],
+ [
+ 8,
+ 5,
+ 8,
+ 7,
+ 14,
+ 56,
+ 111,
+ 0,
+ 9,
+ 3
+ ],
+ [
+ 13,
+ 8,
+ 8,
+ 14,
+ 4,
+ 201,
+ 2,
+ 4,
+ 6,
+ 0
+ ],
+ [
+ 18,
+ 4,
+ 8,
+ 11,
+ 265,
+ 5,
+ 3,
+ 6,
+ 6,
+ 4
+ ],
+ [
+ 165,
+ 5,
+ 3,
+ 7,
+ 8,
+ 4,
+ 2,
+ 2,
+ 5,
+ 1
+ ],
+ [
+ 11,
+ 4,
+ 10,
+ 14,
+ 7,
+ 136,
+ 3,
+ 3,
+ 51,
+ 0
+ ],
+ [
+ 15,
+ 84,
+ 2,
+ 11,
+ 168,
+ 6,
+ 5,
+ 3,
+ 6,
+ 0
+ ],
+ [
+ 13,
+ 7,
+ 4,
+ 36,
+ 7,
+ 2,
+ 2,
+ 1,
+ 3,
+ 171
+ ],
+ [
+ 14,
+ 9,
+ 167,
+ 11,
+ 9,
+ 7,
+ 8,
+ 8,
+ 5,
+ 1
+ ],
+ [
+ 17,
+ 3,
+ 12,
+ 7,
+ 16,
+ 5,
+ 167,
+ 3,
+ 8,
+ 0
+ ],
+ [
+ 16,
+ 12,
+ 12,
+ 13,
+ 13,
+ 6,
+ 5,
+ 11,
+ 90,
+ 1
+ ],
+ [
+ 8,
+ 4,
+ 5,
+ 9,
+ 240,
+ 3,
+ 3,
+ 0,
+ 5,
+ 1
+ ],
+ [
+ 12,
+ 6,
+ 3,
+ 6,
+ 7,
+ 2,
+ 3,
+ 3,
+ 51,
+ 1
+ ],
+ [
+ 10,
+ 5,
+ 10,
+ 9,
+ 257,
+ 3,
+ 3,
+ 2,
+ 9,
+ 2
+ ],
+ [
+ 119,
+ 3,
+ 2,
+ 6,
+ 11,
+ 1,
+ 3,
+ 6,
+ 7,
+ 1
+ ],
+ [
+ 15,
+ 2,
+ 7,
+ 10,
+ 286,
+ 2,
+ 0,
+ 2,
+ 0,
+ 0
+ ],
+ [
+ 7,
+ 69,
+ 2,
+ 8,
+ 6,
+ 1,
+ 1,
+ 0,
+ 2,
+ 2
+ ],
+ [
+ 16,
+ 9,
+ 6,
+ 9,
+ 5,
+ 7,
+ 3,
+ 31,
+ 8,
+ 1
+ ],
+ [
+ 18,
+ 235,
+ 2,
+ 13,
+ 10,
+ 3,
+ 5,
+ 4,
+ 8,
+ 0
+ ],
+ [
+ 10,
+ 130,
+ 7,
+ 9,
+ 9,
+ 1,
+ 3,
+ 0,
+ 2,
+ 1
+ ],
+ [
+ 228,
+ 9,
+ 11,
+ 8,
+ 14,
+ 7,
+ 4,
+ 7,
+ 3,
+ 1
+ ],
+ [
+ 13,
+ 4,
+ 5,
+ 26,
+ 14,
+ 5,
+ 4,
+ 0,
+ 6,
+ 0
+ ],
+ [
+ 9,
+ 7,
+ 9,
+ 16,
+ 13,
+ 5,
+ 0,
+ 2,
+ 6,
+ 96
+ ],
+ [
+ 10,
+ 4,
+ 9,
+ 9,
+ 13,
+ 7,
+ 6,
+ 4,
+ 54,
+ 0
+ ],
+ [
+ 15,
+ 3,
+ 5,
+ 10,
+ 10,
+ 3,
+ 4,
+ 216,
+ 4,
+ 1
+ ],
+ [
+ 17,
+ 5,
+ 11,
+ 12,
+ 9,
+ 7,
+ 5,
+ 7,
+ 9,
+ 98
+ ],
+ [
+ 17,
+ 6,
+ 6,
+ 11,
+ 5,
+ 3,
+ 58,
+ 3,
+ 4,
+ 1
+ ],
+ [
+ 14,
+ 41,
+ 9,
+ 14,
+ 25,
+ 3,
+ 3,
+ 3,
+ 3,
+ 4
+ ],
+ [
+ 12,
+ 6,
+ 12,
+ 12,
+ 5,
+ 140,
+ 3,
+ 1,
+ 7,
+ 3
+ ],
+ [
+ 17,
+ 8,
+ 5,
+ 14,
+ 8,
+ 5,
+ 159,
+ 4,
+ 8,
+ 2
+ ],
+ [
+ 7,
+ 6,
+ 6,
+ 4,
+ 8,
+ 1,
+ 2,
+ 3,
+ 54,
+ 3
+ ],
+ [
+ 13,
+ 6,
+ 8,
+ 6,
+ 6,
+ 1,
+ 6,
+ 4,
+ 52,
+ 4
+ ],
+ [
+ 14,
+ 4,
+ 8,
+ 117,
+ 12,
+ 3,
+ 3,
+ 2,
+ 3,
+ 1
+ ],
+ [
+ 12,
+ 5,
+ 8,
+ 7,
+ 174,
+ 6,
+ 1,
+ 4,
+ 3,
+ 0
+ ],
+ [
+ 12,
+ 4,
+ 5,
+ 16,
+ 22,
+ 393,
+ 4,
+ 5,
+ 11,
+ 1
+ ],
+ [
+ 7,
+ 3,
+ 2,
+ 3,
+ 83,
+ 2,
+ 2,
+ 3,
+ 5,
+ 1
+ ],
+ [
+ 159,
+ 13,
+ 12,
+ 18,
+ 4,
+ 4,
+ 2,
+ 4,
+ 8,
+ 0
+ ],
+ [
+ 16,
+ 5,
+ 7,
+ 15,
+ 8,
+ 7,
+ 0,
+ 4,
+ 59,
+ 1
+ ],
+ [
+ 6,
+ 3,
+ 10,
+ 5,
+ 360,
+ 7,
+ 0,
+ 3,
+ 1,
+ 0
+ ],
+ [
+ 12,
+ 174,
+ 6,
+ 13,
+ 10,
+ 3,
+ 3,
+ 2,
+ 4,
+ 2
+ ],
+ [
+ 10,
+ 4,
+ 4,
+ 10,
+ 5,
+ 6,
+ 4,
+ 0,
+ 9,
+ 100
+ ],
+ [
+ 14,
+ 114,
+ 7,
+ 10,
+ 12,
+ 6,
+ 3,
+ 2,
+ 2,
+ 0
+ ],
+ [
+ 11,
+ 2,
+ 10,
+ 9,
+ 6,
+ 1,
+ 2,
+ 95,
+ 3,
+ 1
+ ],
+ [
+ 22,
+ 9,
+ 8,
+ 16,
+ 13,
+ 3,
+ 6,
+ 7,
+ 39,
+ 0
+ ],
+ [
+ 17,
+ 6,
+ 11,
+ 10,
+ 10,
+ 4,
+ 2,
+ 196,
+ 5,
+ 1
+ ],
+ [
+ 23,
+ 5,
+ 9,
+ 14,
+ 11,
+ 7,
+ 1,
+ 157,
+ 11,
+ 0
+ ],
+ [
+ 38,
+ 11,
+ 21,
+ 34,
+ 22,
+ 24,
+ 94,
+ 14,
+ 16,
+ 77
+ ],
+ [
+ 194,
+ 5,
+ 4,
+ 6,
+ 5,
+ 1,
+ 2,
+ 2,
+ 113,
+ 1
+ ],
+ [
+ 29,
+ 2,
+ 19,
+ 25,
+ 194,
+ 9,
+ 31,
+ 4,
+ 10,
+ 2
+ ],
+ [
+ 19,
+ 4,
+ 12,
+ 6,
+ 4,
+ 135,
+ 3,
+ 2,
+ 6,
+ 3
+ ],
+ [
+ 14,
+ 368,
+ 8,
+ 7,
+ 4,
+ 4,
+ 3,
+ 0,
+ 2,
+ 5
+ ],
+ [
+ 21,
+ 7,
+ 6,
+ 13,
+ 9,
+ 335,
+ 5,
+ 6,
+ 5,
+ 0
+ ],
+ [
+ 13,
+ 4,
+ 7,
+ 13,
+ 3,
+ 6,
+ 2,
+ 3,
+ 425,
+ 1
+ ],
+ [
+ 11,
+ 5,
+ 10,
+ 13,
+ 12,
+ 9,
+ 5,
+ 4,
+ 106,
+ 1
+ ],
+ [
+ 11,
+ 4,
+ 6,
+ 9,
+ 8,
+ 169,
+ 3,
+ 3,
+ 2,
+ 2
+ ],
+ [
+ 16,
+ 7,
+ 4,
+ 612,
+ 13,
+ 1,
+ 5,
+ 4,
+ 6,
+ 1
+ ],
+ [
+ 20,
+ 7,
+ 7,
+ 12,
+ 10,
+ 10,
+ 5,
+ 6,
+ 7,
+ 101
+ ],
+ [
+ 24,
+ 3,
+ 10,
+ 16,
+ 15,
+ 171,
+ 7,
+ 7,
+ 4,
+ 4
+ ],
+ [
+ 13,
+ 6,
+ 7,
+ 16,
+ 17,
+ 4,
+ 2,
+ 0,
+ 158,
+ 0
+ ],
+ [
+ 14,
+ 4,
+ 11,
+ 16,
+ 8,
+ 5,
+ 8,
+ 4,
+ 64,
+ 0
+ ],
+ [
+ 22,
+ 12,
+ 14,
+ 25,
+ 14,
+ 161,
+ 7,
+ 10,
+ 9,
+ 0
+ ],
+ [
+ 267,
+ 9,
+ 5,
+ 13,
+ 11,
+ 2,
+ 1,
+ 1,
+ 3,
+ 1
+ ],
+ [
+ 16,
+ 5,
+ 6,
+ 100,
+ 5,
+ 0,
+ 5,
+ 0,
+ 5,
+ 0
+ ],
+ [
+ 9,
+ 8,
+ 10,
+ 10,
+ 6,
+ 269,
+ 7,
+ 3,
+ 4,
+ 0
+ ],
+ [
+ 20,
+ 6,
+ 9,
+ 7,
+ 11,
+ 171,
+ 4,
+ 1,
+ 5,
+ 4
+ ],
+ [
+ 12,
+ 3,
+ 180,
+ 11,
+ 3,
+ 1,
+ 8,
+ 2,
+ 2,
+ 1
+ ],
+ [
+ 17,
+ 3,
+ 12,
+ 8,
+ 8,
+ 6,
+ 86,
+ 2,
+ 3,
+ 1
+ ],
+ [
+ 16,
+ 3,
+ 9,
+ 8,
+ 3,
+ 2,
+ 2,
+ 1,
+ 72,
+ 2
+ ],
+ [
+ 10,
+ 9,
+ 8,
+ 6,
+ 259,
+ 7,
+ 5,
+ 3,
+ 5,
+ 2
+ ],
+ [
+ 18,
+ 10,
+ 6,
+ 6,
+ 5,
+ 3,
+ 5,
+ 78,
+ 6,
+ 2
+ ],
+ [
+ 14,
+ 3,
+ 6,
+ 13,
+ 13,
+ 6,
+ 47,
+ 2,
+ 78,
+ 1
+ ],
+ [
+ 12,
+ 6,
+ 7,
+ 12,
+ 5,
+ 1,
+ 3,
+ 2,
+ 49,
+ 2
+ ],
+ [
+ 10,
+ 5,
+ 9,
+ 8,
+ 6,
+ 179,
+ 2,
+ 1,
+ 6,
+ 0
+ ],
+ [
+ 12,
+ 2,
+ 5,
+ 7,
+ 216,
+ 3,
+ 1,
+ 3,
+ 6,
+ 1
+ ],
+ [
+ 173,
+ 2,
+ 8,
+ 10,
+ 6,
+ 2,
+ 3,
+ 0,
+ 1,
+ 0
+ ],
+ [
+ 22,
+ 2,
+ 7,
+ 11,
+ 10,
+ 5,
+ 115,
+ 2,
+ 5,
+ 2
+ ],
+ [
+ 12,
+ 5,
+ 9,
+ 82,
+ 6,
+ 3,
+ 6,
+ 3,
+ 6,
+ 6
+ ],
+ [
+ 29,
+ 9,
+ 310,
+ 18,
+ 8,
+ 7,
+ 3,
+ 6,
+ 7,
+ 0
+ ],
+ [
+ 11,
+ 5,
+ 156,
+ 8,
+ 8,
+ 1,
+ 4,
+ 209,
+ 7,
+ 4
+ ],
+ [
+ 417,
+ 1,
+ 4,
+ 4,
+ 4,
+ 5,
+ 3,
+ 3,
+ 3,
+ 1
+ ],
+ [
+ 16,
+ 3,
+ 4,
+ 11,
+ 13,
+ 6,
+ 6,
+ 4,
+ 64,
+ 2
+ ],
+ [
+ 21,
+ 5,
+ 9,
+ 8,
+ 8,
+ 413,
+ 4,
+ 4,
+ 9,
+ 1
+ ],
+ [
+ 11,
+ 8,
+ 6,
+ 108,
+ 6,
+ 5,
+ 6,
+ 4,
+ 4,
+ 2
+ ],
+ [
+ 14,
+ 4,
+ 2,
+ 7,
+ 10,
+ 2,
+ 2,
+ 4,
+ 92,
+ 2
+ ],
+ [
+ 360,
+ 6,
+ 5,
+ 5,
+ 12,
+ 6,
+ 1,
+ 4,
+ 5,
+ 4
+ ],
+ [
+ 8,
+ 2,
+ 5,
+ 7,
+ 7,
+ 2,
+ 2,
+ 137,
+ 8,
+ 1
+ ],
+ [
+ 21,
+ 208,
+ 9,
+ 16,
+ 23,
+ 7,
+ 7,
+ 9,
+ 10,
+ 3
+ ],
+ [
+ 12,
+ 7,
+ 6,
+ 14,
+ 9,
+ 3,
+ 95,
+ 1,
+ 5,
+ 1
+ ],
+ [
+ 12,
+ 2,
+ 6,
+ 8,
+ 13,
+ 162,
+ 1,
+ 1,
+ 5,
+ 0
+ ],
+ [
+ 11,
+ 0,
+ 7,
+ 10,
+ 145,
+ 2,
+ 5,
+ 2,
+ 10,
+ 1
+ ],
+ [
+ 12,
+ 5,
+ 5,
+ 11,
+ 209,
+ 6,
+ 3,
+ 17,
+ 3,
+ 0
+ ],
+ [
+ 21,
+ 10,
+ 10,
+ 12,
+ 12,
+ 6,
+ 7,
+ 508,
+ 5,
+ 1
+ ],
+ [
+ 17,
+ 6,
+ 175,
+ 13,
+ 13,
+ 10,
+ 4,
+ 4,
+ 4,
+ 0
+ ],
+ [
+ 14,
+ 4,
+ 6,
+ 129,
+ 82,
+ 2,
+ 2,
+ 3,
+ 9,
+ 1
+ ],
+ [
+ 14,
+ 5,
+ 11,
+ 18,
+ 10,
+ 3,
+ 3,
+ 4,
+ 191,
+ 2
+ ],
+ [
+ 179,
+ 2,
+ 2,
+ 3,
+ 7,
+ 2,
+ 4,
+ 8,
+ 5,
+ 1
+ ],
+ [
+ 15,
+ 5,
+ 113,
+ 11,
+ 7,
+ 8,
+ 0,
+ 2,
+ 7,
+ 1
+ ],
+ [
+ 14,
+ 7,
+ 6,
+ 8,
+ 7,
+ 4,
+ 0,
+ 0,
+ 56,
+ 2
+ ],
+ [
+ 7,
+ 5,
+ 12,
+ 7,
+ 8,
+ 1,
+ 79,
+ 9,
+ 6,
+ 0
+ ],
+ [
+ 13,
+ 2,
+ 13,
+ 9,
+ 7,
+ 6,
+ 223,
+ 3,
+ 2,
+ 1
+ ],
+ [
+ 11,
+ 5,
+ 15,
+ 86,
+ 15,
+ 5,
+ 8,
+ 4,
+ 11,
+ 6
+ ],
+ [
+ 16,
+ 5,
+ 140,
+ 124,
+ 13,
+ 3,
+ 5,
+ 4,
+ 6,
+ 4
+ ],
+ [
+ 10,
+ 3,
+ 7,
+ 11,
+ 19,
+ 272,
+ 6,
+ 2,
+ 2,
+ 0
+ ],
+ [
+ 9,
+ 6,
+ 8,
+ 114,
+ 4,
+ 2,
+ 4,
+ 10,
+ 4,
+ 7
+ ],
+ [
+ 15,
+ 4,
+ 5,
+ 9,
+ 7,
+ 48,
+ 3,
+ 8,
+ 59,
+ 4
+ ],
+ [
+ 17,
+ 2,
+ 7,
+ 14,
+ 5,
+ 4,
+ 124,
+ 2,
+ 5,
+ 2
+ ],
+ [
+ 16,
+ 4,
+ 8,
+ 111,
+ 5,
+ 3,
+ 4,
+ 4,
+ 6,
+ 0
+ ],
+ [
+ 18,
+ 7,
+ 8,
+ 8,
+ 12,
+ 7,
+ 6,
+ 2,
+ 5,
+ 62
+ ],
+ [
+ 22,
+ 6,
+ 9,
+ 6,
+ 9,
+ 10,
+ 4,
+ 67,
+ 7,
+ 2
+ ],
+ [
+ 131,
+ 4,
+ 9,
+ 6,
+ 6,
+ 2,
+ 1,
+ 4,
+ 1,
+ 2
+ ],
+ [
+ 12,
+ 5,
+ 6,
+ 9,
+ 5,
+ 102,
+ 1,
+ 5,
+ 10,
+ 1
+ ],
+ [
+ 211,
+ 1,
+ 7,
+ 8,
+ 5,
+ 4,
+ 149,
+ 1,
+ 3,
+ 0
+ ],
+ [
+ 12,
+ 8,
+ 11,
+ 12,
+ 9,
+ 9,
+ 5,
+ 81,
+ 9,
+ 2
+ ],
+ [
+ 9,
+ 10,
+ 5,
+ 12,
+ 10,
+ 6,
+ 8,
+ 273,
+ 3,
+ 1
+ ],
+ [
+ 31,
+ 127,
+ 12,
+ 14,
+ 19,
+ 15,
+ 3,
+ 2,
+ 10,
+ 3
+ ],
+ [
+ 16,
+ 3,
+ 7,
+ 12,
+ 407,
+ 5,
+ 2,
+ 4,
+ 9,
+ 1
+ ],
+ [
+ 10,
+ 4,
+ 59,
+ 14,
+ 9,
+ 4,
+ 2,
+ 1,
+ 5,
+ 1
+ ],
+ [
+ 12,
+ 3,
+ 9,
+ 12,
+ 283,
+ 1,
+ 4,
+ 4,
+ 5,
+ 3
+ ],
+ [
+ 103,
+ 3,
+ 208,
+ 5,
+ 4,
+ 3,
+ 2,
+ 1,
+ 6,
+ 1
+ ],
+ [
+ 6,
+ 9,
+ 2,
+ 4,
+ 6,
+ 10,
+ 4,
+ 0,
+ 135,
+ 0
+ ],
+ [
+ 30,
+ 6,
+ 12,
+ 18,
+ 130,
+ 161,
+ 1,
+ 7,
+ 4,
+ 0
+ ],
+ [
+ 15,
+ 95,
+ 7,
+ 9,
+ 7,
+ 2,
+ 2,
+ 2,
+ 12,
+ 1
+ ],
+ [
+ 15,
+ 5,
+ 3,
+ 9,
+ 7,
+ 3,
+ 4,
+ 119,
+ 8,
+ 0
+ ],
+ [
+ 11,
+ 6,
+ 8,
+ 14,
+ 177,
+ 5,
+ 4,
+ 5,
+ 10,
+ 1
+ ],
+ [
+ 8,
+ 5,
+ 3,
+ 7,
+ 228,
+ 3,
+ 3,
+ 3,
+ 2,
+ 0
+ ],
+ [
+ 11,
+ 4,
+ 10,
+ 72,
+ 8,
+ 5,
+ 1,
+ 4,
+ 4,
+ 1
+ ],
+ [
+ 27,
+ 11,
+ 16,
+ 17,
+ 12,
+ 13,
+ 200,
+ 4,
+ 6,
+ 2
+ ],
+ [
+ 24,
+ 8,
+ 8,
+ 12,
+ 9,
+ 3,
+ 173,
+ 2,
+ 6,
+ 0
+ ],
+ [
+ 9,
+ 1,
+ 11,
+ 8,
+ 7,
+ 176,
+ 3,
+ 0,
+ 5,
+ 2
+ ],
+ [
+ 11,
+ 3,
+ 8,
+ 6,
+ 108,
+ 0,
+ 4,
+ 2,
+ 6,
+ 2
+ ],
+ [
+ 18,
+ 23,
+ 5,
+ 14,
+ 5,
+ 4,
+ 82,
+ 2,
+ 11,
+ 1
+ ],
+ [
+ 13,
+ 7,
+ 131,
+ 5,
+ 8,
+ 3,
+ 4,
+ 5,
+ 9,
+ 1
+ ],
+ [
+ 186,
+ 13,
+ 11,
+ 10,
+ 12,
+ 0,
+ 3,
+ 5,
+ 10,
+ 4
+ ],
+ [
+ 11,
+ 7,
+ 5,
+ 6,
+ 14,
+ 6,
+ 1,
+ 35,
+ 8,
+ 1
+ ],
+ [
+ 14,
+ 272,
+ 16,
+ 18,
+ 9,
+ 1,
+ 3,
+ 6,
+ 13,
+ 2
+ ],
+ [
+ 8,
+ 3,
+ 6,
+ 14,
+ 8,
+ 5,
+ 2,
+ 141,
+ 7,
+ 2
+ ],
+ [
+ 4,
+ 4,
+ 7,
+ 2,
+ 7,
+ 1,
+ 4,
+ 126,
+ 6,
+ 0
+ ],
+ [
+ 17,
+ 6,
+ 4,
+ 17,
+ 347,
+ 2,
+ 2,
+ 8,
+ 4,
+ 0
+ ],
+ [
+ 12,
+ 4,
+ 2,
+ 8,
+ 5,
+ 3,
+ 2,
+ 2,
+ 5,
+ 52
+ ],
+ [
+ 22,
+ 3,
+ 8,
+ 90,
+ 8,
+ 4,
+ 8,
+ 2,
+ 12,
+ 1
+ ],
+ [
+ 22,
+ 3,
+ 4,
+ 6,
+ 350,
+ 2,
+ 3,
+ 5,
+ 5,
+ 1
+ ],
+ [
+ 239,
+ 5,
+ 12,
+ 11,
+ 9,
+ 8,
+ 4,
+ 2,
+ 6,
+ 2
+ ],
+ [
+ 6,
+ 6,
+ 7,
+ 6,
+ 4,
+ 4,
+ 2,
+ 34,
+ 60,
+ 1
+ ],
+ [
+ 16,
+ 1,
+ 9,
+ 9,
+ 459,
+ 5,
+ 4,
+ 4,
+ 14,
+ 1
+ ],
+ [
+ 9,
+ 2,
+ 6,
+ 65,
+ 6,
+ 1,
+ 2,
+ 1,
+ 4,
+ 0
+ ],
+ [
+ 17,
+ 170,
+ 16,
+ 27,
+ 21,
+ 7,
+ 2,
+ 10,
+ 10,
+ 0
+ ],
+ [
+ 10,
+ 3,
+ 4,
+ 171,
+ 6,
+ 2,
+ 1,
+ 1,
+ 3,
+ 1
+ ],
+ [
+ 243,
+ 8,
+ 7,
+ 16,
+ 14,
+ 5,
+ 4,
+ 5,
+ 16,
+ 4
+ ],
+ [
+ 21,
+ 11,
+ 8,
+ 13,
+ 5,
+ 7,
+ 6,
+ 4,
+ 2,
+ 2
+ ],
+ [
+ 14,
+ 5,
+ 12,
+ 21,
+ 8,
+ 146,
+ 7,
+ 3,
+ 14,
+ 1
+ ],
+ [
+ 27,
+ 14,
+ 12,
+ 14,
+ 182,
+ 5,
+ 5,
+ 4,
+ 5,
+ 3
+ ],
+ [
+ 12,
+ 3,
+ 2,
+ 11,
+ 9,
+ 2,
+ 2,
+ 6,
+ 4,
+ 127
+ ],
+ [
+ 18,
+ 187,
+ 7,
+ 14,
+ 13,
+ 5,
+ 5,
+ 8,
+ 12,
+ 0
+ ],
+ [
+ 22,
+ 5,
+ 9,
+ 72,
+ 11,
+ 11,
+ 0,
+ 1,
+ 8,
+ 0
+ ],
+ [
+ 11,
+ 3,
+ 8,
+ 11,
+ 9,
+ 2,
+ 6,
+ 120,
+ 8,
+ 0
+ ],
+ [
+ 13,
+ 3,
+ 152,
+ 7,
+ 14,
+ 6,
+ 2,
+ 3,
+ 6,
+ 0
+ ],
+ [
+ 14,
+ 4,
+ 9,
+ 9,
+ 15,
+ 163,
+ 3,
+ 10,
+ 5,
+ 1
+ ],
+ [
+ 12,
+ 4,
+ 10,
+ 9,
+ 11,
+ 4,
+ 5,
+ 3,
+ 3,
+ 76
+ ],
+ [
+ 13,
+ 4,
+ 6,
+ 179,
+ 8,
+ 14,
+ 7,
+ 5,
+ 4,
+ 3
+ ],
+ [
+ 12,
+ 7,
+ 84,
+ 7,
+ 8,
+ 3,
+ 7,
+ 2,
+ 4,
+ 3
+ ],
+ [
+ 171,
+ 5,
+ 137,
+ 10,
+ 7,
+ 4,
+ 2,
+ 6,
+ 6,
+ 2
+ ],
+ [
+ 11,
+ 3,
+ 6,
+ 9,
+ 7,
+ 6,
+ 4,
+ 7,
+ 72,
+ 1
+ ],
+ [
+ 11,
+ 5,
+ 5,
+ 11,
+ 133,
+ 8,
+ 0,
+ 1,
+ 15,
+ 0
+ ],
+ [
+ 11,
+ 9,
+ 15,
+ 14,
+ 11,
+ 3,
+ 77,
+ 1,
+ 8,
+ 2
+ ],
+ [
+ 11,
+ 10,
+ 12,
+ 12,
+ 170,
+ 6,
+ 3,
+ 3,
+ 3,
+ 0
+ ],
+ [
+ 15,
+ 180,
+ 14,
+ 10,
+ 9,
+ 4,
+ 2,
+ 2,
+ 6,
+ 2
+ ],
+ [
+ 13,
+ 7,
+ 17,
+ 12,
+ 6,
+ 7,
+ 4,
+ 3,
+ 213,
+ 1
+ ],
+ [
+ 13,
+ 5,
+ 12,
+ 108,
+ 6,
+ 4,
+ 0,
+ 198,
+ 6,
+ 1
+ ],
+ [
+ 14,
+ 7,
+ 9,
+ 95,
+ 8,
+ 1,
+ 2,
+ 3,
+ 1,
+ 0
+ ],
+ [
+ 16,
+ 7,
+ 140,
+ 11,
+ 11,
+ 6,
+ 4,
+ 4,
+ 6,
+ 3
+ ],
+ [
+ 11,
+ 3,
+ 5,
+ 112,
+ 11,
+ 6,
+ 2,
+ 125,
+ 6,
+ 0
+ ],
+ [
+ 12,
+ 4,
+ 15,
+ 7,
+ 362,
+ 7,
+ 2,
+ 5,
+ 4,
+ 1
+ ],
+ [
+ 51,
+ 3,
+ 4,
+ 17,
+ 358,
+ 3,
+ 2,
+ 3,
+ 6,
+ 2
+ ],
+ [
+ 11,
+ 4,
+ 8,
+ 15,
+ 6,
+ 6,
+ 93,
+ 72,
+ 11,
+ 0
+ ],
+ [
+ 19,
+ 4,
+ 11,
+ 8,
+ 9,
+ 5,
+ 4,
+ 0,
+ 108,
+ 1
+ ],
+ [
+ 10,
+ 6,
+ 247,
+ 4,
+ 11,
+ 2,
+ 1,
+ 7,
+ 9,
+ 1
+ ],
+ [
+ 6,
+ 7,
+ 4,
+ 11,
+ 208,
+ 2,
+ 24,
+ 0,
+ 3,
+ 0
+ ],
+ [
+ 6,
+ 7,
+ 5,
+ 8,
+ 6,
+ 7,
+ 1,
+ 72,
+ 3,
+ 0
+ ],
+ [
+ 346,
+ 6,
+ 3,
+ 8,
+ 10,
+ 1,
+ 0,
+ 1,
+ 5,
+ 0
+ ],
+ [
+ 13,
+ 6,
+ 9,
+ 10,
+ 10,
+ 2,
+ 2,
+ 99,
+ 9,
+ 4
+ ],
+ [
+ 7,
+ 2,
+ 5,
+ 164,
+ 13,
+ 4,
+ 2,
+ 4,
+ 6,
+ 0
+ ],
+ [
+ 15,
+ 6,
+ 12,
+ 7,
+ 10,
+ 3,
+ 58,
+ 5,
+ 7,
+ 1
+ ],
+ [
+ 17,
+ 5,
+ 5,
+ 10,
+ 10,
+ 3,
+ 43,
+ 5,
+ 6,
+ 3
+ ],
+ [
+ 15,
+ 3,
+ 12,
+ 13,
+ 18,
+ 4,
+ 4,
+ 8,
+ 9,
+ 108
+ ],
+ [
+ 9,
+ 5,
+ 2,
+ 12,
+ 3,
+ 130,
+ 4,
+ 6,
+ 5,
+ 1
+ ],
+ [
+ 116,
+ 6,
+ 4,
+ 8,
+ 6,
+ 7,
+ 5,
+ 0,
+ 8,
+ 2
+ ],
+ [
+ 22,
+ 10,
+ 16,
+ 11,
+ 8,
+ 230,
+ 4,
+ 2,
+ 9,
+ 1
+ ],
+ [
+ 17,
+ 9,
+ 14,
+ 13,
+ 13,
+ 9,
+ 13,
+ 66,
+ 3,
+ 0
+ ],
+ [
+ 6,
+ 7,
+ 8,
+ 12,
+ 380,
+ 2,
+ 6,
+ 4,
+ 3,
+ 3
+ ],
+ [
+ 14,
+ 4,
+ 7,
+ 11,
+ 4,
+ 3,
+ 2,
+ 2,
+ 9,
+ 94
+ ],
+ [
+ 11,
+ 113,
+ 6,
+ 8,
+ 7,
+ 3,
+ 35,
+ 2,
+ 6,
+ 0
+ ],
+ [
+ 23,
+ 7,
+ 12,
+ 23,
+ 17,
+ 49,
+ 4,
+ 4,
+ 2,
+ 5
+ ],
+ [
+ 10,
+ 2,
+ 3,
+ 1,
+ 6,
+ 4,
+ 1,
+ 3,
+ 54,
+ 0
+ ],
+ [
+ 10,
+ 5,
+ 9,
+ 6,
+ 2,
+ 8,
+ 0,
+ 3,
+ 59,
+ 0
+ ],
+ [
+ 10,
+ 3,
+ 8,
+ 13,
+ 5,
+ 164,
+ 2,
+ 1,
+ 10,
+ 4
+ ],
+ [
+ 25,
+ 8,
+ 14,
+ 13,
+ 17,
+ 11,
+ 3,
+ 2,
+ 14,
+ 112
+ ],
+ [
+ 6,
+ 1,
+ 11,
+ 14,
+ 7,
+ 4,
+ 1,
+ 7,
+ 209,
+ 1
+ ],
+ [
+ 11,
+ 2,
+ 11,
+ 9,
+ 8,
+ 4,
+ 80,
+ 6,
+ 1,
+ 0
+ ],
+ [
+ 15,
+ 3,
+ 8,
+ 6,
+ 12,
+ 2,
+ 6,
+ 20,
+ 4,
+ 0
+ ],
+ [
+ 9,
+ 3,
+ 11,
+ 5,
+ 295,
+ 5,
+ 2,
+ 5,
+ 6,
+ 3
+ ],
+ [
+ 14,
+ 129,
+ 7,
+ 10,
+ 9,
+ 4,
+ 2,
+ 2,
+ 7,
+ 2
+ ],
+ [
+ 5,
+ 6,
+ 2,
+ 9,
+ 294,
+ 4,
+ 4,
+ 1,
+ 6,
+ 1
+ ],
+ [
+ 17,
+ 6,
+ 8,
+ 13,
+ 10,
+ 3,
+ 3,
+ 0,
+ 61,
+ 0
+ ],
+ [
+ 16,
+ 9,
+ 13,
+ 6,
+ 9,
+ 258,
+ 1,
+ 3,
+ 3,
+ 1
+ ],
+ [
+ 22,
+ 4,
+ 9,
+ 102,
+ 8,
+ 2,
+ 5,
+ 3,
+ 2,
+ 1
+ ],
+ [
+ 17,
+ 4,
+ 8,
+ 69,
+ 8,
+ 2,
+ 5,
+ 2,
+ 6,
+ 2
+ ],
+ [
+ 16,
+ 4,
+ 3,
+ 18,
+ 263,
+ 5,
+ 4,
+ 1,
+ 7,
+ 63
+ ],
+ [
+ 8,
+ 3,
+ 8,
+ 15,
+ 8,
+ 1,
+ 3,
+ 163,
+ 4,
+ 3
+ ],
+ [
+ 19,
+ 95,
+ 15,
+ 16,
+ 12,
+ 8,
+ 8,
+ 7,
+ 6,
+ 0
+ ],
+ [
+ 16,
+ 6,
+ 9,
+ 112,
+ 9,
+ 1,
+ 4,
+ 3,
+ 5,
+ 1
+ ],
+ [
+ 19,
+ 3,
+ 3,
+ 11,
+ 195,
+ 4,
+ 2,
+ 4,
+ 6,
+ 1
+ ],
+ [
+ 14,
+ 92,
+ 8,
+ 12,
+ 7,
+ 1,
+ 3,
+ 2,
+ 103,
+ 2
+ ],
+ [
+ 15,
+ 4,
+ 6,
+ 12,
+ 18,
+ 5,
+ 97,
+ 4,
+ 10,
+ 1
+ ],
+ [
+ 14,
+ 3,
+ 136,
+ 12,
+ 11,
+ 4,
+ 2,
+ 3,
+ 10,
+ 2
+ ],
+ [
+ 184,
+ 5,
+ 4,
+ 4,
+ 9,
+ 4,
+ 2,
+ 2,
+ 8,
+ 0
+ ],
+ [
+ 16,
+ 13,
+ 9,
+ 7,
+ 8,
+ 7,
+ 4,
+ 1,
+ 126,
+ 3
+ ],
+ [
+ 10,
+ 1,
+ 10,
+ 5,
+ 9,
+ 6,
+ 1,
+ 3,
+ 61,
+ 3
+ ],
+ [
+ 18,
+ 3,
+ 11,
+ 9,
+ 5,
+ 1,
+ 98,
+ 2,
+ 6,
+ 1
+ ],
+ [
+ 27,
+ 7,
+ 14,
+ 8,
+ 11,
+ 7,
+ 3,
+ 3,
+ 12,
+ 107
+ ],
+ [
+ 12,
+ 4,
+ 12,
+ 10,
+ 6,
+ 3,
+ 5,
+ 4,
+ 112,
+ 2
+ ],
+ [
+ 11,
+ 4,
+ 189,
+ 15,
+ 8,
+ 5,
+ 8,
+ 5,
+ 116,
+ 3
+ ],
+ [
+ 11,
+ 10,
+ 178,
+ 13,
+ 8,
+ 7,
+ 3,
+ 6,
+ 8,
+ 2
+ ],
+ [
+ 10,
+ 5,
+ 67,
+ 10,
+ 5,
+ 9,
+ 1,
+ 1,
+ 5,
+ 0
+ ],
+ [
+ 21,
+ 342,
+ 11,
+ 16,
+ 7,
+ 7,
+ 4,
+ 4,
+ 9,
+ 0
+ ],
+ [
+ 12,
+ 180,
+ 5,
+ 6,
+ 7,
+ 3,
+ 6,
+ 2,
+ 6,
+ 0
+ ],
+ [
+ 159,
+ 6,
+ 12,
+ 16,
+ 7,
+ 4,
+ 3,
+ 1,
+ 2,
+ 0
+ ],
+ [
+ 224,
+ 9,
+ 8,
+ 7,
+ 7,
+ 4,
+ 5,
+ 3,
+ 5,
+ 1
+ ],
+ [
+ 16,
+ 12,
+ 7,
+ 15,
+ 291,
+ 3,
+ 5,
+ 4,
+ 10,
+ 0
+ ],
+ [
+ 7,
+ 8,
+ 1,
+ 10,
+ 6,
+ 5,
+ 123,
+ 3,
+ 4,
+ 1
+ ],
+ [
+ 263,
+ 5,
+ 4,
+ 11,
+ 7,
+ 1,
+ 2,
+ 2,
+ 4,
+ 1
+ ],
+ [
+ 14,
+ 2,
+ 3,
+ 9,
+ 3,
+ 2,
+ 117,
+ 3,
+ 5,
+ 2
+ ],
+ [
+ 15,
+ 5,
+ 7,
+ 5,
+ 5,
+ 188,
+ 3,
+ 0,
+ 4,
+ 1
+ ],
+ [
+ 19,
+ 44,
+ 9,
+ 96,
+ 10,
+ 5,
+ 3,
+ 7,
+ 5,
+ 3
+ ],
+ [
+ 12,
+ 5,
+ 13,
+ 11,
+ 7,
+ 3,
+ 2,
+ 265,
+ 7,
+ 0
+ ],
+ [
+ 25,
+ 13,
+ 16,
+ 22,
+ 23,
+ 4,
+ 102,
+ 10,
+ 16,
+ 119
+ ],
+ [
+ 13,
+ 9,
+ 7,
+ 179,
+ 8,
+ 7,
+ 2,
+ 2,
+ 6,
+ 0
+ ],
+ [
+ 14,
+ 2,
+ 14,
+ 10,
+ 14,
+ 3,
+ 3,
+ 74,
+ 13,
+ 1
+ ],
+ [
+ 367,
+ 8,
+ 13,
+ 20,
+ 6,
+ 4,
+ 3,
+ 4,
+ 3,
+ 4
+ ],
+ [
+ 13,
+ 7,
+ 4,
+ 10,
+ 8,
+ 7,
+ 40,
+ 1,
+ 9,
+ 1
+ ],
+ [
+ 15,
+ 4,
+ 8,
+ 13,
+ 11,
+ 4,
+ 3,
+ 5,
+ 44,
+ 0
+ ],
+ [
+ 12,
+ 2,
+ 6,
+ 129,
+ 9,
+ 5,
+ 6,
+ 2,
+ 4,
+ 3
+ ],
+ [
+ 16,
+ 4,
+ 3,
+ 112,
+ 3,
+ 3,
+ 1,
+ 2,
+ 5,
+ 0
+ ],
+ [
+ 17,
+ 7,
+ 10,
+ 11,
+ 17,
+ 5,
+ 107,
+ 3,
+ 4,
+ 2
+ ],
+ [
+ 19,
+ 6,
+ 8,
+ 10,
+ 295,
+ 11,
+ 4,
+ 3,
+ 11,
+ 3
+ ],
+ [
+ 24,
+ 7,
+ 9,
+ 9,
+ 8,
+ 69,
+ 3,
+ 5,
+ 6,
+ 1
+ ],
+ [
+ 13,
+ 67,
+ 11,
+ 12,
+ 8,
+ 6,
+ 3,
+ 3,
+ 6,
+ 1
+ ],
+ [
+ 12,
+ 6,
+ 8,
+ 2,
+ 5,
+ 5,
+ 2,
+ 162,
+ 5,
+ 0
+ ],
+ [
+ 27,
+ 5,
+ 15,
+ 11,
+ 6,
+ 7,
+ 54,
+ 4,
+ 15,
+ 1
+ ],
+ [
+ 9,
+ 3,
+ 7,
+ 13,
+ 9,
+ 3,
+ 4,
+ 52,
+ 7,
+ 2
+ ],
+ [
+ 17,
+ 7,
+ 7,
+ 6,
+ 8,
+ 60,
+ 4,
+ 5,
+ 7,
+ 2
+ ],
+ [
+ 11,
+ 1,
+ 8,
+ 12,
+ 7,
+ 5,
+ 9,
+ 3,
+ 101,
+ 0
+ ],
+ [
+ 14,
+ 6,
+ 10,
+ 16,
+ 9,
+ 9,
+ 4,
+ 179,
+ 11,
+ 3
+ ],
+ [
+ 21,
+ 2,
+ 8,
+ 10,
+ 16,
+ 4,
+ 0,
+ 4,
+ 7,
+ 58
+ ],
+ [
+ 10,
+ 3,
+ 5,
+ 8,
+ 14,
+ 217,
+ 3,
+ 1,
+ 5,
+ 2
+ ],
+ [
+ 12,
+ 8,
+ 3,
+ 19,
+ 8,
+ 1,
+ 89,
+ 9,
+ 6,
+ 3
+ ],
+ [
+ 5,
+ 4,
+ 7,
+ 9,
+ 11,
+ 2,
+ 2,
+ 2,
+ 49,
+ 0
+ ],
+ [
+ 174,
+ 2,
+ 7,
+ 11,
+ 9,
+ 2,
+ 5,
+ 3,
+ 6,
+ 2
+ ],
+ [
+ 14,
+ 3,
+ 111,
+ 8,
+ 226,
+ 4,
+ 173,
+ 4,
+ 5,
+ 2
+ ],
+ [
+ 18,
+ 8,
+ 156,
+ 14,
+ 3,
+ 3,
+ 1,
+ 5,
+ 8,
+ 1
+ ],
+ [
+ 11,
+ 156,
+ 10,
+ 18,
+ 6,
+ 5,
+ 1,
+ 2,
+ 5,
+ 2
+ ],
+ [
+ 16,
+ 8,
+ 11,
+ 9,
+ 9,
+ 7,
+ 197,
+ 7,
+ 1,
+ 1
+ ],
+ [
+ 13,
+ 214,
+ 6,
+ 8,
+ 15,
+ 2,
+ 4,
+ 1,
+ 2,
+ 0
+ ],
+ [
+ 13,
+ 2,
+ 8,
+ 9,
+ 1174,
+ 5,
+ 5,
+ 7,
+ 5,
+ 0
+ ],
+ [
+ 24,
+ 7,
+ 9,
+ 17,
+ 12,
+ 4,
+ 0,
+ 11,
+ 6,
+ 117
+ ],
+ [
+ 20,
+ 6,
+ 3,
+ 13,
+ 182,
+ 4,
+ 3,
+ 4,
+ 11,
+ 2
+ ],
+ [
+ 14,
+ 5,
+ 206,
+ 8,
+ 6,
+ 6,
+ 1,
+ 0,
+ 5,
+ 2
+ ],
+ [
+ 13,
+ 6,
+ 4,
+ 16,
+ 9,
+ 2,
+ 114,
+ 3,
+ 4,
+ 0
+ ],
+ [
+ 146,
+ 10,
+ 10,
+ 10,
+ 15,
+ 2,
+ 3,
+ 1,
+ 6,
+ 0
+ ],
+ [
+ 9,
+ 5,
+ 179,
+ 14,
+ 6,
+ 3,
+ 4,
+ 0,
+ 4,
+ 1
+ ],
+ [
+ 12,
+ 7,
+ 6,
+ 239,
+ 7,
+ 8,
+ 5,
+ 2,
+ 43,
+ 1
+ ],
+ [
+ 27,
+ 6,
+ 4,
+ 10,
+ 8,
+ 5,
+ 5,
+ 2,
+ 72,
+ 3
+ ],
+ [
+ 12,
+ 2,
+ 6,
+ 14,
+ 9,
+ 5,
+ 3,
+ 236,
+ 10,
+ 3
+ ],
+ [
+ 16,
+ 9,
+ 1,
+ 11,
+ 7,
+ 11,
+ 2,
+ 1,
+ 106,
+ 1
+ ],
+ [
+ 11,
+ 4,
+ 4,
+ 34,
+ 202,
+ 2,
+ 3,
+ 2,
+ 8,
+ 1
+ ],
+ [
+ 17,
+ 4,
+ 10,
+ 15,
+ 13,
+ 69,
+ 4,
+ 97,
+ 6,
+ 2
+ ],
+ [
+ 15,
+ 14,
+ 7,
+ 16,
+ 13,
+ 7,
+ 4,
+ 208,
+ 7,
+ 0
+ ],
+ [
+ 18,
+ 9,
+ 14,
+ 18,
+ 149,
+ 8,
+ 4,
+ 3,
+ 9,
+ 0
+ ],
+ [
+ 19,
+ 8,
+ 344,
+ 11,
+ 7,
+ 7,
+ 5,
+ 5,
+ 9,
+ 2
+ ],
+ [
+ 29,
+ 15,
+ 15,
+ 170,
+ 15,
+ 18,
+ 5,
+ 29,
+ 10,
+ 0
+ ],
+ [
+ 12,
+ 8,
+ 240,
+ 10,
+ 7,
+ 2,
+ 1,
+ 2,
+ 2,
+ 0
+ ],
+ [
+ 10,
+ 4,
+ 4,
+ 12,
+ 9,
+ 1,
+ 5,
+ 86,
+ 2,
+ 1
+ ],
+ [
+ 19,
+ 91,
+ 4,
+ 3,
+ 5,
+ 2,
+ 2,
+ 2,
+ 4,
+ 1
+ ],
+ [
+ 8,
+ 3,
+ 123,
+ 5,
+ 8,
+ 10,
+ 2,
+ 1,
+ 3,
+ 3
+ ],
+ [
+ 12,
+ 5,
+ 11,
+ 18,
+ 12,
+ 3,
+ 2,
+ 102,
+ 3,
+ 1
+ ],
+ [
+ 412,
+ 8,
+ 5,
+ 8,
+ 3,
+ 3,
+ 2,
+ 2,
+ 5,
+ 0
+ ],
+ [
+ 7,
+ 4,
+ 5,
+ 11,
+ 9,
+ 9,
+ 5,
+ 1,
+ 61,
+ 1
+ ],
+ [
+ 19,
+ 6,
+ 9,
+ 9,
+ 6,
+ 5,
+ 2,
+ 3,
+ 7,
+ 47
+ ],
+ [
+ 20,
+ 3,
+ 9,
+ 8,
+ 11,
+ 6,
+ 8,
+ 86,
+ 11,
+ 1
+ ],
+ [
+ 8,
+ 1,
+ 127,
+ 6,
+ 7,
+ 3,
+ 3,
+ 5,
+ 1,
+ 1
+ ],
+ [
+ 7,
+ 160,
+ 4,
+ 33,
+ 8,
+ 1,
+ 2,
+ 5,
+ 6,
+ 0
+ ],
+ [
+ 32,
+ 3,
+ 8,
+ 9,
+ 8,
+ 1,
+ 2,
+ 2,
+ 45,
+ 0
+ ],
+ [
+ 14,
+ 6,
+ 212,
+ 5,
+ 6,
+ 6,
+ 3,
+ 6,
+ 9,
+ 2
+ ],
+ [
+ 19,
+ 5,
+ 8,
+ 8,
+ 9,
+ 7,
+ 6,
+ 3,
+ 7,
+ 149
+ ],
+ [
+ 11,
+ 2,
+ 16,
+ 12,
+ 10,
+ 4,
+ 4,
+ 9,
+ 41,
+ 3
+ ],
+ [
+ 11,
+ 5,
+ 3,
+ 13,
+ 181,
+ 2,
+ 0,
+ 1,
+ 1,
+ 1
+ ],
+ [
+ 13,
+ 5,
+ 12,
+ 12,
+ 9,
+ 113,
+ 6,
+ 9,
+ 2,
+ 0
+ ],
+ [
+ 13,
+ 6,
+ 7,
+ 14,
+ 5,
+ 6,
+ 3,
+ 3,
+ 8,
+ 89
+ ],
+ [
+ 7,
+ 3,
+ 3,
+ 187,
+ 8,
+ 3,
+ 5,
+ 1,
+ 1,
+ 4
+ ],
+ [
+ 21,
+ 1,
+ 7,
+ 11,
+ 10,
+ 5,
+ 5,
+ 4,
+ 1,
+ 107
+ ],
+ [
+ 11,
+ 6,
+ 6,
+ 12,
+ 9,
+ 4,
+ 67,
+ 8,
+ 4,
+ 1
+ ],
+ [
+ 30,
+ 11,
+ 17,
+ 30,
+ 22,
+ 7,
+ 5,
+ 7,
+ 11,
+ 120
+ ],
+ [
+ 20,
+ 6,
+ 6,
+ 18,
+ 12,
+ 6,
+ 1,
+ 1,
+ 97,
+ 0
+ ],
+ [
+ 19,
+ 93,
+ 10,
+ 13,
+ 20,
+ 0,
+ 4,
+ 4,
+ 13,
+ 1
+ ],
+ [
+ 17,
+ 5,
+ 7,
+ 177,
+ 5,
+ 2,
+ 1,
+ 5,
+ 4,
+ 1
+ ],
+ [
+ 14,
+ 5,
+ 9,
+ 9,
+ 3,
+ 3,
+ 5,
+ 3,
+ 4,
+ 78
+ ],
+ [
+ 12,
+ 7,
+ 5,
+ 16,
+ 8,
+ 3,
+ 5,
+ 2,
+ 4,
+ 65
+ ],
+ [
+ 25,
+ 11,
+ 16,
+ 32,
+ 17,
+ 142,
+ 13,
+ 8,
+ 8,
+ 0
+ ],
+ [
+ 19,
+ 89,
+ 18,
+ 7,
+ 5,
+ 5,
+ 3,
+ 2,
+ 7,
+ 1
+ ],
+ [
+ 18,
+ 6,
+ 10,
+ 17,
+ 5,
+ 7,
+ 127,
+ 4,
+ 11,
+ 3
+ ],
+ [
+ 519,
+ 8,
+ 7,
+ 12,
+ 8,
+ 4,
+ 4,
+ 2,
+ 9,
+ 1
+ ],
+ [
+ 10,
+ 8,
+ 11,
+ 11,
+ 7,
+ 136,
+ 6,
+ 5,
+ 4,
+ 0
+ ],
+ [
+ 13,
+ 6,
+ 5,
+ 9,
+ 5,
+ 4,
+ 83,
+ 1,
+ 4,
+ 1
+ ],
+ [
+ 10,
+ 3,
+ 10,
+ 12,
+ 9,
+ 3,
+ 3,
+ 11,
+ 3,
+ 71
+ ],
+ [
+ 11,
+ 76,
+ 8,
+ 8,
+ 10,
+ 3,
+ 3,
+ 3,
+ 3,
+ 0
+ ],
+ [
+ 21,
+ 9,
+ 6,
+ 17,
+ 181,
+ 3,
+ 5,
+ 2,
+ 6,
+ 2
+ ],
+ [
+ 8,
+ 9,
+ 5,
+ 6,
+ 155,
+ 5,
+ 4,
+ 8,
+ 3,
+ 1
+ ],
+ [
+ 11,
+ 2,
+ 9,
+ 6,
+ 4,
+ 5,
+ 3,
+ 91,
+ 1,
+ 0
+ ],
+ [
+ 13,
+ 4,
+ 149,
+ 9,
+ 10,
+ 7,
+ 0,
+ 4,
+ 5,
+ 1
+ ],
+ [
+ 26,
+ 199,
+ 19,
+ 35,
+ 24,
+ 11,
+ 5,
+ 6,
+ 14,
+ 0
+ ],
+ [
+ 13,
+ 7,
+ 9,
+ 16,
+ 213,
+ 4,
+ 5,
+ 2,
+ 13,
+ 0
+ ],
+ [
+ 10,
+ 6,
+ 12,
+ 12,
+ 261,
+ 3,
+ 2,
+ 2,
+ 9,
+ 1
+ ],
+ [
+ 20,
+ 4,
+ 5,
+ 9,
+ 10,
+ 3,
+ 65,
+ 3,
+ 9,
+ 2
+ ],
+ [
+ 13,
+ 7,
+ 2,
+ 10,
+ 12,
+ 1,
+ 4,
+ 3,
+ 4,
+ 101
+ ],
+ [
+ 20,
+ 27,
+ 118,
+ 8,
+ 5,
+ 1,
+ 4,
+ 3,
+ 7,
+ 1
+ ],
+ [
+ 13,
+ 4,
+ 8,
+ 5,
+ 5,
+ 1,
+ 90,
+ 5,
+ 4,
+ 0
+ ],
+ [
+ 16,
+ 4,
+ 5,
+ 7,
+ 13,
+ 5,
+ 0,
+ 5,
+ 138,
+ 0
+ ],
+ [
+ 20,
+ 4,
+ 210,
+ 9,
+ 7,
+ 4,
+ 3,
+ 6,
+ 4,
+ 0
+ ],
+ [
+ 12,
+ 5,
+ 7,
+ 16,
+ 7,
+ 3,
+ 5,
+ 4,
+ 104,
+ 0
+ ],
+ [
+ 17,
+ 3,
+ 11,
+ 11,
+ 13,
+ 7,
+ 158,
+ 4,
+ 7,
+ 0
+ ],
+ [
+ 13,
+ 6,
+ 4,
+ 14,
+ 7,
+ 124,
+ 3,
+ 5,
+ 5,
+ 1
+ ],
+ [
+ 13,
+ 9,
+ 8,
+ 12,
+ 295,
+ 2,
+ 4,
+ 4,
+ 3,
+ 1
+ ],
+ [
+ 6,
+ 134,
+ 5,
+ 4,
+ 13,
+ 1,
+ 3,
+ 0,
+ 4,
+ 0
+ ],
+ [
+ 21,
+ 173,
+ 6,
+ 6,
+ 10,
+ 4,
+ 4,
+ 5,
+ 8,
+ 3
+ ],
+ [
+ 235,
+ 4,
+ 1,
+ 8,
+ 2,
+ 3,
+ 0,
+ 2,
+ 1,
+ 2
+ ],
+ [
+ 7,
+ 3,
+ 6,
+ 8,
+ 6,
+ 3,
+ 2,
+ 5,
+ 6,
+ 36
+ ],
+ [
+ 8,
+ 6,
+ 9,
+ 85,
+ 14,
+ 7,
+ 4,
+ 3,
+ 10,
+ 0
+ ],
+ [
+ 13,
+ 4,
+ 153,
+ 13,
+ 5,
+ 2,
+ 95,
+ 1,
+ 6,
+ 0
+ ],
+ [
+ 27,
+ 16,
+ 22,
+ 301,
+ 32,
+ 7,
+ 6,
+ 11,
+ 9,
+ 4
+ ],
+ [
+ 6,
+ 7,
+ 8,
+ 88,
+ 5,
+ 4,
+ 5,
+ 3,
+ 4,
+ 4
+ ],
+ [
+ 16,
+ 7,
+ 11,
+ 7,
+ 6,
+ 1,
+ 2,
+ 2,
+ 197,
+ 1
+ ],
+ [
+ 29,
+ 7,
+ 10,
+ 15,
+ 21,
+ 4,
+ 9,
+ 117,
+ 14,
+ 3
+ ],
+ [
+ 19,
+ 3,
+ 6,
+ 10,
+ 204,
+ 11,
+ 3,
+ 4,
+ 5,
+ 3
+ ],
+ [
+ 14,
+ 6,
+ 10,
+ 17,
+ 6,
+ 257,
+ 4,
+ 4,
+ 7,
+ 1
+ ],
+ [
+ 17,
+ 11,
+ 14,
+ 84,
+ 14,
+ 6,
+ 9,
+ 12,
+ 11,
+ 4
+ ],
+ [
+ 17,
+ 16,
+ 68,
+ 9,
+ 10,
+ 6,
+ 3,
+ 7,
+ 5,
+ 1
+ ],
+ [
+ 11,
+ 4,
+ 3,
+ 16,
+ 9,
+ 79,
+ 0,
+ 1,
+ 6,
+ 3
+ ],
+ [
+ 16,
+ 119,
+ 12,
+ 6,
+ 6,
+ 6,
+ 2,
+ 2,
+ 5,
+ 3
+ ],
+ [
+ 23,
+ 5,
+ 9,
+ 6,
+ 5,
+ 5,
+ 3,
+ 3,
+ 4,
+ 121
+ ],
+ [
+ 193,
+ 6,
+ 8,
+ 12,
+ 9,
+ 10,
+ 1,
+ 3,
+ 4,
+ 0
+ ],
+ [
+ 6,
+ 6,
+ 5,
+ 7,
+ 292,
+ 4,
+ 3,
+ 2,
+ 3,
+ 0
+ ],
+ [
+ 18,
+ 4,
+ 11,
+ 16,
+ 7,
+ 4,
+ 6,
+ 4,
+ 6,
+ 158
+ ],
+ [
+ 11,
+ 2,
+ 4,
+ 7,
+ 401,
+ 1,
+ 3,
+ 6,
+ 6,
+ 0
+ ],
+ [
+ 15,
+ 70,
+ 8,
+ 13,
+ 7,
+ 8,
+ 2,
+ 9,
+ 6,
+ 6
+ ],
+ [
+ 12,
+ 7,
+ 198,
+ 82,
+ 3,
+ 2,
+ 2,
+ 2,
+ 0,
+ 2
+ ],
+ [
+ 9,
+ 4,
+ 9,
+ 10,
+ 4,
+ 225,
+ 8,
+ 0,
+ 9,
+ 1
+ ],
+ [
+ 15,
+ 10,
+ 8,
+ 12,
+ 11,
+ 3,
+ 3,
+ 3,
+ 208,
+ 0
+ ],
+ [
+ 131,
+ 2,
+ 2,
+ 7,
+ 5,
+ 1,
+ 1,
+ 1,
+ 6,
+ 0
+ ],
+ [
+ 8,
+ 54,
+ 5,
+ 10,
+ 5,
+ 2,
+ 4,
+ 8,
+ 3,
+ 2
+ ],
+ [
+ 14,
+ 7,
+ 11,
+ 7,
+ 12,
+ 2,
+ 6,
+ 2,
+ 5,
+ 69
+ ],
+ [
+ 3,
+ 5,
+ 3,
+ 7,
+ 154,
+ 3,
+ 3,
+ 2,
+ 2,
+ 1
+ ],
+ [
+ 7,
+ 3,
+ 6,
+ 5,
+ 4,
+ 2,
+ 5,
+ 5,
+ 94,
+ 0
+ ],
+ [
+ 18,
+ 8,
+ 223,
+ 6,
+ 11,
+ 2,
+ 2,
+ 7,
+ 11,
+ 1
+ ],
+ [
+ 15,
+ 12,
+ 4,
+ 14,
+ 6,
+ 8,
+ 53,
+ 2,
+ 5,
+ 1
+ ],
+ [
+ 12,
+ 3,
+ 8,
+ 6,
+ 245,
+ 3,
+ 2,
+ 4,
+ 140,
+ 1
+ ],
+ [
+ 11,
+ 185,
+ 7,
+ 13,
+ 9,
+ 2,
+ 3,
+ 7,
+ 10,
+ 1
+ ],
+ [
+ 7,
+ 7,
+ 9,
+ 6,
+ 7,
+ 5,
+ 93,
+ 1,
+ 3,
+ 2
+ ],
+ [
+ 19,
+ 7,
+ 10,
+ 17,
+ 13,
+ 3,
+ 8,
+ 1,
+ 7,
+ 124
+ ],
+ [
+ 14,
+ 2,
+ 11,
+ 20,
+ 296,
+ 4,
+ 5,
+ 3,
+ 8,
+ 3
+ ],
+ [
+ 10,
+ 5,
+ 9,
+ 8,
+ 10,
+ 5,
+ 81,
+ 5,
+ 6,
+ 0
+ ],
+ [
+ 16,
+ 5,
+ 5,
+ 7,
+ 258,
+ 6,
+ 2,
+ 3,
+ 8,
+ 0
+ ],
+ [
+ 17,
+ 5,
+ 11,
+ 9,
+ 4,
+ 8,
+ 68,
+ 1,
+ 26,
+ 2
+ ],
+ [
+ 17,
+ 6,
+ 8,
+ 10,
+ 8,
+ 2,
+ 79,
+ 7,
+ 2,
+ 0
+ ],
+ [
+ 13,
+ 7,
+ 189,
+ 15,
+ 5,
+ 5,
+ 4,
+ 5,
+ 1,
+ 2
+ ],
+ [
+ 16,
+ 217,
+ 9,
+ 8,
+ 6,
+ 2,
+ 2,
+ 4,
+ 3,
+ 1
+ ],
+ [
+ 9,
+ 128,
+ 2,
+ 11,
+ 4,
+ 0,
+ 4,
+ 2,
+ 5,
+ 0
+ ],
+ [
+ 9,
+ 7,
+ 2,
+ 14,
+ 11,
+ 88,
+ 4,
+ 5,
+ 3,
+ 12
+ ],
+ [
+ 14,
+ 4,
+ 13,
+ 8,
+ 185,
+ 4,
+ 3,
+ 2,
+ 8,
+ 0
+ ],
+ [
+ 15,
+ 2,
+ 6,
+ 6,
+ 2,
+ 5,
+ 63,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 9,
+ 5,
+ 5,
+ 8,
+ 4,
+ 23,
+ 1,
+ 3,
+ 8,
+ 1
+ ],
+ [
+ 11,
+ 1,
+ 8,
+ 16,
+ 8,
+ 4,
+ 34,
+ 4,
+ 4,
+ 3
+ ],
+ [
+ 18,
+ 12,
+ 7,
+ 13,
+ 8,
+ 120,
+ 5,
+ 3,
+ 4,
+ 0
+ ],
+ [
+ 18,
+ 5,
+ 4,
+ 12,
+ 9,
+ 258,
+ 6,
+ 3,
+ 6,
+ 1
+ ],
+ [
+ 12,
+ 5,
+ 7,
+ 12,
+ 14,
+ 5,
+ 8,
+ 4,
+ 3,
+ 54
+ ],
+ [
+ 7,
+ 3,
+ 8,
+ 6,
+ 4,
+ 3,
+ 1,
+ 5,
+ 92,
+ 1
+ ],
+ [
+ 9,
+ 213,
+ 5,
+ 7,
+ 9,
+ 1,
+ 5,
+ 3,
+ 4,
+ 0
+ ],
+ [
+ 20,
+ 4,
+ 19,
+ 17,
+ 360,
+ 6,
+ 5,
+ 4,
+ 5,
+ 2
+ ],
+ [
+ 8,
+ 5,
+ 6,
+ 19,
+ 14,
+ 5,
+ 117,
+ 8,
+ 10,
+ 3
+ ],
+ [
+ 13,
+ 10,
+ 10,
+ 16,
+ 12,
+ 346,
+ 7,
+ 6,
+ 4,
+ 2
+ ],
+ [
+ 11,
+ 4,
+ 5,
+ 2,
+ 9,
+ 4,
+ 5,
+ 1,
+ 79,
+ 1
+ ],
+ [
+ 8,
+ 4,
+ 3,
+ 231,
+ 12,
+ 5,
+ 4,
+ 2,
+ 9,
+ 3
+ ],
+ [
+ 14,
+ 4,
+ 5,
+ 13,
+ 286,
+ 2,
+ 2,
+ 4,
+ 8,
+ 1
+ ],
+ [
+ 6,
+ 3,
+ 6,
+ 11,
+ 3,
+ 2,
+ 2,
+ 3,
+ 111,
+ 1
+ ],
+ [
+ 10,
+ 3,
+ 9,
+ 9,
+ 357,
+ 4,
+ 3,
+ 1,
+ 145,
+ 0
+ ],
+ [
+ 22,
+ 0,
+ 6,
+ 131,
+ 7,
+ 7,
+ 3,
+ 3,
+ 4,
+ 2
+ ],
+ [
+ 93,
+ 10,
+ 11,
+ 15,
+ 4,
+ 2,
+ 3,
+ 2,
+ 4,
+ 3
+ ],
+ [
+ 8,
+ 3,
+ 124,
+ 11,
+ 11,
+ 1,
+ 0,
+ 0,
+ 106,
+ 0
+ ],
+ [
+ 18,
+ 12,
+ 9,
+ 9,
+ 11,
+ 16,
+ 64,
+ 5,
+ 8,
+ 42
+ ],
+ [
+ 17,
+ 4,
+ 11,
+ 13,
+ 21,
+ 7,
+ 1,
+ 155,
+ 9,
+ 0
+ ],
+ [
+ 16,
+ 3,
+ 7,
+ 3,
+ 259,
+ 3,
+ 1,
+ 3,
+ 5,
+ 9
+ ],
+ [
+ 19,
+ 7,
+ 192,
+ 14,
+ 5,
+ 8,
+ 9,
+ 4,
+ 9,
+ 1
+ ]
+ ],
+ "sha256": "c59ef3eaf08bde4988997bae90c5988ef81be30ca2be397e3fb1f661e2b1f0b8"
+ },
+ "seurat": {
+ "hashId": [
+ "TNC_1_4x",
+ "TNC_2_4x",
+ "TNC_1_2x",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "TNC_1_4x",
+ "TNC_2_0_25x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "TNC_1_2x",
+ "Doublet",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_1_0_25x",
+ "TNC_2_1x",
+ "TNC_1_0_5x",
+ "TNC_2_0_5x",
+ "TNC_1_4x",
+ "TNC_2_0_25x",
+ "TNC_2_1x",
+ "TNC_2_1x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_1_4x",
+ "TNC_2_4x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_1_2x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "Doublet",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_1_2x",
+ "TNC_1_0_25x",
+ "TNC_1_2x",
+ "TNC_2_2x",
+ "TNC_1_4x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_1_1x",
+ "TNC_1_2x",
+ "TNC_2_1x",
+ "TNC_2_0_25x",
+ "TNC_2_0_5x",
+ "Doublet",
+ "TNC_2_0_25x",
+ "TNC_2_0_5x",
+ "TNC_2_2x",
+ "TNC_1_0_5x",
+ "TNC_2_0_25x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_1_0_25x",
+ "TNC_2_1x",
+ "TNC_1_0_5x",
+ "TNC_2_2x",
+ "TNC_2_2x",
+ "TNC_1_2x",
+ "TNC_1_1x",
+ "TNC_2_0_25x",
+ "TNC_2_2x",
+ "TNC_1_4x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_2_1x",
+ "TNC_2_1x",
+ "TNC_2_0_5x",
+ "TNC_1_0_5x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "TNC_1_0_25x",
+ "TNC_1_2x",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "TNC_2_0_25x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "TNC_2_0_25x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_2_4x",
+ "TNC_1_0_25x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_1_1x",
+ "TNC_2_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_4x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_2_4x",
+ "TNC_1_0_5x",
+ "TNC_2_0_5x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_2_4x",
+ "TNC_2_1x",
+ "TNC_1_0_25x",
+ "TNC_2_1x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_1_1x",
+ "TNC_2_1x",
+ "TNC_2_0_25x",
+ "TNC_2_0_5x",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "TNC_2_0_5x",
+ "TNC_1_4x",
+ "TNC_2_0_5x",
+ "TNC_1_4x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "TNC_1_4x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "TNC_2_1x",
+ "TNC_2_0_25x",
+ "TNC_2_0_25x",
+ "TNC_1_0_25x",
+ "TNC_1_2x",
+ "TNC_1_0_25x",
+ "TNC_2_2x",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "TNC_1_0_25x",
+ "TNC_2_0_25x",
+ "TNC_2_2x",
+ "TNC_1_2x",
+ "TNC_1_2x",
+ "TNC_1_2x",
+ "TNC_2_1x",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "TNC_2_2x",
+ "TNC_2_0_25x",
+ "TNC_2_2x",
+ "TNC_1_0_5x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_1_0_25x",
+ "TNC_2_1x",
+ "Doublet",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_1_0_5x",
+ "TNC_2_0_5x",
+ "Doublet",
+ "TNC_1_0_25x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_1_2x",
+ "TNC_2_4x",
+ "TNC_2_0_5x",
+ "TNC_1_0_25x",
+ "TNC_2_0_25x",
+ "TNC_1_2x",
+ "TNC_1_0_25x",
+ "TNC_1_2x",
+ "TNC_1_0_5x",
+ "Doublet",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_1_0_25x",
+ "TNC_2_2x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_2_1x",
+ "TNC_2_0_5x",
+ "Negative",
+ "TNC_2_0_5x",
+ "TNC_2_4x",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "TNC_2_4x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_1_1x",
+ "TNC_1_2x",
+ "TNC_1_1x",
+ "TNC_2_1x",
+ "TNC_2_0_25x",
+ "TNC_1_2x",
+ "TNC_1_2x",
+ "TNC_1_1x",
+ "TNC_1_0_5x",
+ "TNC_2_1x",
+ "TNC_2_0_25x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "TNC_2_0_5x",
+ "TNC_2_1x",
+ "TNC_2_2x",
+ "TNC_2_1x",
+ "TNC_2_2x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_1_1x",
+ "TNC_2_4x",
+ "TNC_1_4x",
+ "TNC_1_4x",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "TNC_2_0_5x",
+ "TNC_2_2x",
+ "TNC_1_0_5x",
+ "Doublet",
+ "TNC_2_4x",
+ "TNC_2_4x",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_0_25x",
+ "TNC_1_2x",
+ "TNC_1_4x",
+ "TNC_1_1x",
+ "TNC_2_2x",
+ "TNC_1_2x",
+ "TNC_2_2x",
+ "TNC_1_0_25x",
+ "TNC_2_4x",
+ "TNC_2_1x",
+ "TNC_2_0_25x",
+ "Doublet",
+ "Doublet",
+ "TNC_2_4x",
+ "TNC_1_0_25x",
+ "TNC_2_0_5x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_1_1x",
+ "TNC_1_2x",
+ "TNC_2_4x",
+ "TNC_2_4x",
+ "TNC_2_0_25x",
+ "TNC_1_1x",
+ "TNC_2_0_25x",
+ "Doublet",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_2_0_5x",
+ "TNC_1_1x",
+ "TNC_2_4x",
+ "TNC_2_2x",
+ "TNC_2_4x",
+ "TNC_1_0_25x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_1_1x",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "TNC_2_4x",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_1_2x",
+ "TNC_2_4x",
+ "TNC_1_2x",
+ "TNC_1_0_5x",
+ "Doublet",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_2_0_5x",
+ "TNC_1_4x",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "TNC_2_0_25x",
+ "TNC_2_2x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_2_2x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_2_4x",
+ "TNC_1_0_5x",
+ "TNC_2_1x",
+ "TNC_1_0_5x",
+ "TNC_2_0_5x",
+ "TNC_2_4x",
+ "TNC_2_2x",
+ "Doublet",
+ "TNC_1_0_25x",
+ "TNC_1_2x",
+ "TNC_1_1x",
+ "TNC_2_4x",
+ "TNC_2_4x",
+ "TNC_2_1x",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "TNC_2_2x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "TNC_2_0_25x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_1_0_25x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_2_4x",
+ "TNC_1_4x",
+ "TNC_1_2x",
+ "TNC_1_2x",
+ "TNC_2_0_25x",
+ "TNC_1_0_25x",
+ "TNC_2_1x",
+ "Doublet",
+ "Doublet",
+ "TNC_1_4x",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_1_1x",
+ "TNC_2_4x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "TNC_2_4x",
+ "TNC_1_1x",
+ "TNC_2_2x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_2_2x",
+ "TNC_2_0_5x",
+ "TNC_1_0_25x",
+ "TNC_1_0_25x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_4x",
+ "TNC_2_0_5x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_2_4x",
+ "TNC_1_2x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "Negative",
+ "TNC_1_0_5x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_2x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_1_0_25x",
+ "Negative",
+ "TNC_2_0_25x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_2_0_5x",
+ "TNC_2_2x",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "TNC_2_0_5x",
+ "TNC_2_2x",
+ "Negative",
+ "TNC_1_0_25x",
+ "TNC_1_4x",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_2_0_25x",
+ "TNC_2_0_25x",
+ "TNC_2_2x",
+ "TNC_1_0_25x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_1_1x",
+ "TNC_1_4x",
+ "TNC_1_2x",
+ "TNC_1_4x",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_1_0_25x",
+ "TNC_1_4x",
+ "TNC_1_2x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_1_4x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_2_0_25x",
+ "TNC_2_1x",
+ "TNC_2_4x",
+ "TNC_1_0_25x",
+ "TNC_2_1x",
+ "TNC_1_0_25x",
+ "TNC_2_2x",
+ "TNC_2_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_1_4x",
+ "TNC_2_0_5x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_1_0_25x",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_1_2x",
+ "Doublet",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_2_0_5x",
+ "TNC_2_1x",
+ "TNC_2_1x",
+ "TNC_1_0_25x",
+ "TNC_2_4x",
+ "Doublet",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_2_0_25x",
+ "TNC_2_1x",
+ "TNC_1_0_25x",
+ "TNC_2_4x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_1_1x",
+ "TNC_2_4x",
+ "TNC_1_4x",
+ "TNC_1_2x",
+ "TNC_1_1x",
+ "TNC_1_0_5x",
+ "TNC_2_1x",
+ "TNC_1_2x",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "TNC_1_0_25x",
+ "TNC_1_4x",
+ "Doublet",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_2_1x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_2_0_25x",
+ "TNC_1_1x",
+ "TNC_1_4x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_2_2x",
+ "Doublet",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_2_1x",
+ "TNC_1_4x",
+ "TNC_2_4x",
+ "TNC_1_0_25x",
+ "TNC_2_0_25x",
+ "Doublet",
+ "TNC_1_0_25x",
+ "TNC_2_2x",
+ "TNC_2_0_5x",
+ "TNC_2_1x",
+ "TNC_2_0_5x",
+ "TNC_1_1x",
+ "TNC_2_4x",
+ "TNC_2_0_5x",
+ "TNC_2_0_5x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_1_2x",
+ "TNC_1_4x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_1_2x",
+ "TNC_2_2x",
+ "TNC_1_1x",
+ "TNC_1_0_5x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_2_2x",
+ "TNC_1_0_25x",
+ "TNC_1_2x",
+ "TNC_1_0_5x",
+ "TNC_2_1x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_2_1x",
+ "TNC_2_2x",
+ "TNC_1_0_5x",
+ "TNC_1_2x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_2_0_25x",
+ "TNC_1_2x",
+ "TNC_2_0_25x",
+ "TNC_1_0_25x",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_2_2x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_1_0_25x",
+ "TNC_2_0_5x",
+ "TNC_2_0_5x",
+ "TNC_2_2x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "Doublet",
+ "Doublet",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "TNC_1_0_25x",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_2_0_5x",
+ "TNC_2_2x",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "TNC_1_4x",
+ "TNC_1_0_25x",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "TNC_2_1x",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_0_25x",
+ "TNC_1_2x",
+ "TNC_2_4x",
+ "TNC_2_2x",
+ "TNC_2_1x",
+ "TNC_2_4x",
+ "TNC_2_0_5x",
+ "Doublet",
+ "TNC_2_0_25x",
+ "TNC_2_0_5x",
+ "TNC_2_2x",
+ "TNC_2_2x",
+ "TNC_1_2x",
+ "TNC_1_4x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "TNC_1_0_25x",
+ "TNC_2_2x",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "TNC_2_4x",
+ "TNC_1_0_5x",
+ "TNC_2_1x",
+ "TNC_2_2x",
+ "TNC_2_1x",
+ "TNC_2_1x",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "TNC_2_0_25x",
+ "TNC_1_0_5x",
+ "TNC_2_0_25x",
+ "TNC_2_2x",
+ "TNC_2_2x",
+ "TNC_2_0_25x",
+ "TNC_1_2x",
+ "TNC_2_4x",
+ "TNC_2_0_25x",
+ "TNC_2_2x",
+ "TNC_2_2x",
+ "Doublet",
+ "TNC_1_0_25x",
+ "TNC_1_2x",
+ "TNC_2_0_25x",
+ "TNC_2_0_25x",
+ "TNC_1_1x",
+ "TNC_2_0_5x",
+ "TNC_2_2x",
+ "TNC_1_4x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "TNC_1_0_25x",
+ "TNC_2_0_5x",
+ "TNC_1_2x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_1_0_25x",
+ "TNC_2_2x",
+ "TNC_2_0_25x",
+ "TNC_1_2x",
+ "TNC_2_2x",
+ "TNC_1_0_25x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_1_0_25x",
+ "TNC_1_1x",
+ "TNC_2_2x",
+ "TNC_2_0_5x",
+ "TNC_2_0_5x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_2_0_25x",
+ "Doublet",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_1_2x",
+ "TNC_2_4x",
+ "TNC_2_1x",
+ "TNC_1_0_25x",
+ "TNC_2_0_25x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_1_1x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_2_2x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_2_1x",
+ "TNC_1_4x",
+ "TNC_1_4x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_1_0_25x",
+ "TNC_2_1x",
+ "TNC_1_0_5x",
+ "TNC_2_1x",
+ "TNC_2_1x",
+ "TNC_1_4x",
+ "TNC_2_4x",
+ "TNC_1_2x",
+ "TNC_1_4x",
+ "TNC_1_0_25x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_1_2x",
+ "Doublet",
+ "Negative",
+ "Doublet",
+ "Doublet",
+ "TNC_2_4x",
+ "TNC_1_0_5x",
+ "TNC_1_2x",
+ "TNC_2_1x",
+ "TNC_1_1x",
+ "TNC_2_0_25x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_2_2x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "TNC_2_2x",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_1_4x",
+ "Doublet",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_1_0_25x",
+ "TNC_2_1x",
+ "TNC_1_2x",
+ "TNC_2_0_5x",
+ "TNC_2_0_5x",
+ "TNC_2_4x",
+ "TNC_2_0_25x",
+ "TNC_1_0_25x",
+ "TNC_2_0_25x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_2_4x",
+ "Doublet",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_2_2x",
+ "TNC_2_0_25x",
+ "TNC_2_4x",
+ "TNC_2_2x",
+ "TNC_2_0_5x",
+ "TNC_2_1x",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "TNC_2_0_25x",
+ "TNC_1_2x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_1_0_5x",
+ "TNC_1_2x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_1_1x",
+ "TNC_1_0_25x",
+ "TNC_2_2x",
+ "TNC_2_2x",
+ "TNC_2_0_5x",
+ "TNC_2_4x",
+ "TNC_2_2x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_1_1x",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_0_25x",
+ "TNC_1_0_25x",
+ "TNC_1_4x",
+ "TNC_2_0_5x",
+ "TNC_1_0_25x",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "Doublet",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_2_2x",
+ "TNC_1_2x",
+ "TNC_1_2x",
+ "TNC_2_0_5x",
+ "TNC_1_4x",
+ "TNC_2_0_25x",
+ "TNC_1_0_5x",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_2_1x",
+ "TNC_2_0_25x",
+ "TNC_2_2x",
+ "TNC_2_1x",
+ "TNC_2_4x",
+ "TNC_2_0_25x",
+ "TNC_2_0_5x",
+ "TNC_2_2x",
+ "TNC_1_0_25x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_1_0_5x",
+ "TNC_2_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_4x",
+ "TNC_2_4x",
+ "TNC_1_4x",
+ "TNC_1_1x",
+ "TNC_2_0_5x",
+ "TNC_1_0_25x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_2_1x",
+ "TNC_2_2x",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_1_1x",
+ "TNC_2_1x",
+ "TNC_1_0_5x",
+ "TNC_1_1x",
+ "TNC_2_1x",
+ "TNC_1_0_25x",
+ "TNC_2_2x",
+ "TNC_2_4x",
+ "TNC_2_1x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_1_1x",
+ "TNC_2_4x",
+ "TNC_2_2x",
+ "TNC_1_4x",
+ "TNC_2_0_25x",
+ "TNC_2_4x",
+ "TNC_1_2x",
+ "TNC_2_4x",
+ "TNC_2_0_5x",
+ "Doublet",
+ "TNC_2_2x",
+ "TNC_1_0_5x",
+ "TNC_1_2x",
+ "TNC_2_4x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_2_0_5x",
+ "TNC_1_0_25x",
+ "TNC_2_0_25x",
+ "TNC_2_0_5x",
+ "TNC_2_4x",
+ "TNC_1_0_5x",
+ "TNC_1_4x",
+ "TNC_1_4x",
+ "TNC_2_1x",
+ "TNC_1_1x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_1_4x",
+ "TNC_2_0_5x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_2_2x",
+ "TNC_1_1x",
+ "TNC_2_2x",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "TNC_1_0_25x",
+ "TNC_2_4x",
+ "TNC_1_2x",
+ "Doublet",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_2_2x",
+ "TNC_2_1x",
+ "TNC_1_4x",
+ "TNC_2_0_25x",
+ "TNC_1_2x",
+ "Doublet",
+ "TNC_2_0_25x",
+ "TNC_1_0_5x",
+ "TNC_2_4x",
+ "TNC_1_0_25x",
+ "TNC_1_4x",
+ "TNC_2_4x",
+ "TNC_1_4x",
+ "TNC_1_0_5x",
+ "Doublet",
+ "TNC_2_0_25x",
+ "TNC_2_2x",
+ "TNC_1_0_25x",
+ "TNC_1_0_5x",
+ "TNC_2_4x",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "TNC_1_1x",
+ "TNC_2_0_5x",
+ "Doublet",
+ "TNC_1_0_5x",
+ "TNC_2_0_5x",
+ "TNC_2_4x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_1_4x",
+ "Doublet",
+ "TNC_2_0_5x",
+ "TNC_1_1x",
+ "TNC_1_0_5x",
+ "TNC_1_0_5x",
+ "Doublet",
+ "TNC_1_4x",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "TNC_2_0_25x",
+ "TNC_2_4x",
+ "TNC_2_2x",
+ "TNC_1_0_5x",
+ "TNC_1_4x",
+ "TNC_2_0_5x",
+ "TNC_2_0_25x",
+ "TNC_2_2x",
+ "TNC_1_2x",
+ "TNC_1_4x",
+ "TNC_2_2x",
+ "Doublet",
+ "TNC_1_2x",
+ "TNC_1_0_25x",
+ "Doublet",
+ "Doublet",
+ "TNC_2_1x",
+ "Doublet",
+ "TNC_1_1x"
+ ],
+ "classificationGlobal": [
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Negative",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Negative",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Negative",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Negative",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Negative",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Singlet",
+ "Singlet",
+ "Doublet",
+ "Doublet",
+ "Singlet",
+ "Doublet",
+ "Singlet"
+ ]
+ }
+}
diff --git a/tests/seurat_wnn_5_5_1_golden.json b/tests/seurat_wnn_5_5_1_golden.json
new file mode 100644
index 00000000..9fb15408
--- /dev/null
+++ b/tests/seurat_wnn_5_5_1_golden.json
@@ -0,0 +1,45350 @@
+{
+ "provenance": {
+ "package": "Seurat",
+ "packageVersion": "5.5.1",
+ "seuratObjectVersion": "5.4.0",
+ "rVersion": "4.4.3",
+ "sourceRepository": "https://github.com/satijalab/seurat",
+ "dataset": "tenx_8K_pbmc_citeseq",
+ "datasetSource": "scarf.cytebase.connect('scarf_docs')",
+ "generator": "scripts/export_seurat_wnn_golden.py",
+ "nCells": 300,
+ "kNn": 20,
+ "cellSampleSeed": 20210601,
+ "l2Normalize": true,
+ "matchedFunctions": [
+ "Seurat:::L2Norm",
+ "Seurat:::PredictAssay",
+ "Seurat:::impute_dist",
+ "Seurat:::NNdist",
+ "Seurat::MinMax"
+ ],
+ "matchedNote": "Seurat routines evaluated on Scarf's union candidate pool and k-th non-self bandwidth. The kernel, score and softmax expressions are copied from the body of FindModalityWeights.",
+ "defaultFunction": "Seurat::FindMultiModalNeighbors",
+ "defaultNote": "Shipped defaults, including the wider knn.range search and the SNN-far bandwidth. Scarf does not reproduce these, so the comparison is statistical."
+ },
+ "inputs": {
+ "rnaEmbedding": [
+ [
+ -6.862426280975342,
+ -3.3749632835388184,
+ -5.401952266693115,
+ -0.15382786095142365,
+ -4.2711076736450195,
+ -0.007886423729360104,
+ 0.21317283809185028,
+ 0.11642634868621826,
+ -0.47073060274124146,
+ 1.0991243124008179,
+ 0.08116507530212402,
+ 0.23402294516563416,
+ 0.020684346556663513,
+ 2.4195361137390137,
+ -0.7335403561592102
+ ],
+ [
+ -4.533507823944092,
+ -2.282576560974121,
+ -5.008650779724121,
+ 0.18614356219768524,
+ 2.387432336807251,
+ 3.97452712059021,
+ -2.013277292251587,
+ -0.9306365251541138,
+ 0.054879672825336456,
+ -0.9745799899101257,
+ 0.3211608827114105,
+ -0.9913449287414551,
+ -0.5761488080024719,
+ 0.33216744661331177,
+ 1.69247305393219
+ ],
+ [
+ -5.220303058624268,
+ 11.251313209533691,
+ 5.402013301849365,
+ -0.6690139174461365,
+ -4.657553195953369,
+ 1.1078311204910278,
+ -1.308544635772705,
+ -1.3671351671218872,
+ -0.5079149603843689,
+ -1.0529006719589233,
+ -1.2302958965301514,
+ -1.6640228033065796,
+ 0.5573424100875854,
+ -0.08587249368429184,
+ -0.273770809173584
+ ],
+ [
+ -5.1097307205200195,
+ -2.2430763244628906,
+ -4.161198139190674,
+ 0.35004493594169617,
+ 2.3380649089813232,
+ 3.9920058250427246,
+ -1.886628270149231,
+ -2.4233696460723877,
+ 0.22308628261089325,
+ -2.017882823944092,
+ -0.30942782759666443,
+ -0.24928200244903564,
+ 0.5946006774902344,
+ -0.03905932977795601,
+ 1.0441380739212036
+ ],
+ [
+ 14.466516494750977,
+ -0.014108388684689999,
+ -0.9453933835029602,
+ 1.5274907350540161,
+ -0.04774315282702446,
+ -0.05276859179139137,
+ -0.9525335431098938,
+ 0.36960116028785706,
+ 0.291767418384552,
+ -0.5899267196655273,
+ 1.4973913431167603,
+ 1.2808518409729004,
+ 1.8262594938278198,
+ -2.1261959075927734,
+ -0.4350801706314087
+ ],
+ [
+ 16.724102020263672,
+ 0.042855944484472275,
+ -1.0625267028808594,
+ -0.23215287923812866,
+ 0.763384997844696,
+ 0.7823343873023987,
+ 2.1331076622009277,
+ -0.14936307072639465,
+ -0.37886637449264526,
+ -0.5720008015632629,
+ 2.7679126262664795,
+ -1.4428454637527466,
+ -1.6377507448196411,
+ -1.6133463382720947,
+ -0.43422964215278625
+ ],
+ [
+ -3.7319746017456055,
+ -3.0349252223968506,
+ -3.943357467651367,
+ -0.36633196473121643,
+ 1.965852975845337,
+ 2.163123846054077,
+ -1.629744052886963,
+ -0.3702146112918854,
+ -0.10311228781938553,
+ -1.1914255619049072,
+ -0.6098885536193848,
+ -0.2957827150821686,
+ -1.486481785774231,
+ -1.7827144861221313,
+ 1.4671051502227783
+ ],
+ [
+ -6.458469390869141,
+ 2.2094905376434326,
+ -2.6897974014282227,
+ 0.035224366933107376,
+ 7.155047416687012,
+ -1.2816879749298096,
+ 0.9380318522453308,
+ 1.0235261917114258,
+ -0.23294365406036377,
+ -0.4392356276512146,
+ 0.2725569009780884,
+ -0.6199442148208618,
+ 0.05874163284897804,
+ -0.5971851348876953,
+ 2.141587495803833
+ ],
+ [
+ -5.295546531677246,
+ -2.7921478748321533,
+ -3.549241781234741,
+ -0.7519335746765137,
+ 2.0144317150115967,
+ 0.6734527349472046,
+ -1.1351374387741089,
+ -1.098777413368225,
+ 0.5236518979072571,
+ -1.2117623090744019,
+ -0.7060422301292419,
+ 0.4084591567516327,
+ -1.155539631843567,
+ -2.1360220909118652,
+ -0.029097171500325203
+ ],
+ [
+ -4.9732465744018555,
+ 15.231152534484863,
+ 7.854043960571289,
+ -0.5242583155632019,
+ -4.862489700317383,
+ 1.6659572124481201,
+ 0.11515721678733826,
+ -0.3982374370098114,
+ 0.18059803545475006,
+ -1.4005249738693237,
+ 0.5269710421562195,
+ -0.42879098653793335,
+ -0.1261538565158844,
+ 0.6900517344474792,
+ 0.1938886195421219
+ ],
+ [
+ -6.342941761016846,
+ -4.251736640930176,
+ -5.50806999206543,
+ 0.5164388418197632,
+ -4.186404705047607,
+ -0.5550045967102051,
+ -0.3797577917575836,
+ 0.1189500242471695,
+ -0.1250302642583847,
+ -0.8415070176124573,
+ 1.3040313720703125,
+ -0.22650180757045746,
+ 0.6396699547767639,
+ -0.35036715865135193,
+ -1.389693021774292
+ ],
+ [
+ -1.5926311016082764,
+ -10.881962776184082,
+ 9.882366180419922,
+ -2.478710889816284,
+ 2.460641384124756,
+ -1.0588082075119019,
+ 1.1184548139572144,
+ 0.09320013970136642,
+ -0.3070906400680542,
+ -1.4944759607315063,
+ 1.2157319784164429,
+ 0.28358516097068787,
+ -1.2847648859024048,
+ -1.753907322883606,
+ -0.40163347125053406
+ ],
+ [
+ -4.781680107116699,
+ 5.414392471313477,
+ 0.7349257469177246,
+ -0.37366363406181335,
+ 2.169193983078003,
+ 0.018314717337489128,
+ -0.0773385614156723,
+ -0.7332291603088379,
+ 0.42656877636909485,
+ -0.10419675707817078,
+ -0.675670862197876,
+ -1.7865763902664185,
+ 0.21628500521183014,
+ -0.07827585935592651,
+ -1.7866740226745605
+ ],
+ [
+ 16.369789123535156,
+ -0.21235142648220062,
+ -1.1945347785949707,
+ -0.6162688732147217,
+ 0.14115247130393982,
+ -1.8950960636138916,
+ -1.5615475177764893,
+ 0.4653532803058624,
+ -0.6612560153007507,
+ -0.6376727223396301,
+ -1.0051372051239014,
+ 0.1992236226797104,
+ 0.20956629514694214,
+ -0.5998852849006653,
+ -0.6687213182449341
+ ],
+ [
+ -5.109886169433594,
+ -1.4709714651107788,
+ -2.8844974040985107,
+ -0.1321423351764679,
+ 3.314218044281006,
+ 7.183156967163086,
+ -1.69542396068573,
+ -3.3842530250549316,
+ 0.33985525369644165,
+ 0.5084229707717896,
+ 0.31693390011787415,
+ 3.643266439437866,
+ -1.1228622198104858,
+ -0.8727076053619385,
+ -1.664806842803955
+ ],
+ [
+ 15.708404541015625,
+ 1.339893102645874,
+ -1.409623146057129,
+ 0.326123982667923,
+ 1.341237187385559,
+ 1.2896792888641357,
+ 2.0171010494232178,
+ -0.16351865231990814,
+ -0.48706087470054626,
+ -0.24676302075386047,
+ 2.2532284259796143,
+ -1.167142629623413,
+ -0.28143712878227234,
+ 0.10215424746274948,
+ 0.6453988552093506
+ ],
+ [
+ -5.950135231018066,
+ -0.956806480884552,
+ -3.58147931098938,
+ 0.26101982593536377,
+ 5.3209733963012695,
+ 2.9546549320220947,
+ -0.8047458529472351,
+ -1.8074040412902832,
+ 0.25301218032836914,
+ -0.6402029991149902,
+ 0.3760581910610199,
+ 0.33291107416152954,
+ -0.16307637095451355,
+ 0.20328480005264282,
+ 1.9576839208602905
+ ],
+ [
+ 15.867429733276367,
+ 0.18602438271045685,
+ -1.2999224662780762,
+ -0.5051133632659912,
+ 0.12176451086997986,
+ -0.873372495174408,
+ -1.9939042329788208,
+ -0.24550354480743408,
+ -0.3220718801021576,
+ -0.7410950660705566,
+ -1.6397662162780762,
+ 0.8067479729652405,
+ -1.2729617357254028,
+ -1.3217276334762573,
+ 1.4059644937515259
+ ],
+ [
+ 16.436138153076172,
+ 0.20578768849372864,
+ -0.7785106897354126,
+ 0.04282413050532341,
+ 0.7477620244026184,
+ 0.8835130929946899,
+ 1.38152015209198,
+ -0.37157848477363586,
+ -0.5687206983566284,
+ 0.7899201512336731,
+ 0.6101014018058777,
+ -0.6011698842048645,
+ -0.9289954900741577,
+ -0.805939793586731,
+ -0.027477072551846504
+ ],
+ [
+ -6.890591144561768,
+ 6.075699806213379,
+ 1.39870285987854,
+ 0.12147247046232224,
+ 6.495605945587158,
+ -4.437543869018555,
+ 1.9807475805282593,
+ 1.298646330833435,
+ -0.403984010219574,
+ -2.66467022895813,
+ -0.3822888135910034,
+ -1.3003431558609009,
+ 2.0239548683166504,
+ 4.311764717102051,
+ -0.07758191972970963
+ ],
+ [
+ -6.942126750946045,
+ 2.3066837787628174,
+ -2.1514482498168945,
+ 0.051777563989162445,
+ 8.789236068725586,
+ -3.965360164642334,
+ 2.071518898010254,
+ 0.07355764508247375,
+ 0.17715327441692352,
+ -1.4994763135910034,
+ -0.7525551915168762,
+ 1.4011934995651245,
+ 0.09368690103292465,
+ 0.6676381230354309,
+ -0.5915248394012451
+ ],
+ [
+ -4.80228853225708,
+ -4.074131488800049,
+ -2.6196842193603516,
+ 0.3094523251056671,
+ -3.539212226867676,
+ -0.7718569040298462,
+ 0.1337812840938568,
+ 0.24859526753425598,
+ 0.09953183680772781,
+ -1.510522723197937,
+ 0.07212912291288376,
+ -0.6493706107139587,
+ 0.6937844753265381,
+ -1.7507671117782593,
+ -1.1651710271835327
+ ],
+ [
+ 15.080215454101562,
+ 0.06074795126914978,
+ -0.24571363627910614,
+ 0.4491477906703949,
+ 0.8011071085929871,
+ 0.7502331137657166,
+ 1.37602698802948,
+ -0.6502664089202881,
+ -0.2265588790178299,
+ -0.83139568567276,
+ 3.669407606124878,
+ -1.3994303941726685,
+ -1.8734437227249146,
+ -1.7633874416351318,
+ 0.4991828203201294
+ ],
+ [
+ -7.010314464569092,
+ 3.5383474826812744,
+ -0.6988250017166138,
+ -0.08811669796705246,
+ 8.75991439819336,
+ -4.990825176239014,
+ 1.2492995262145996,
+ 1.3930376768112183,
+ -0.2952728271484375,
+ -0.3429676294326782,
+ -1.9285550117492676,
+ -1.0055493116378784,
+ 0.5565334558486938,
+ -0.7406623363494873,
+ -2.034463882446289
+ ],
+ [
+ -6.505236625671387,
+ 5.666015625,
+ 1.0762635469436646,
+ -0.3102800250053406,
+ 4.542480945587158,
+ -2.5698494911193848,
+ 1.3810248374938965,
+ 1.1575875282287598,
+ -0.11296544969081879,
+ 1.817895531654358,
+ 1.0242359638214111,
+ 3.348438262939453,
+ -0.6028278470039368,
+ 3.240760326385498,
+ -0.8784909844398499
+ ],
+ [
+ -5.216909885406494,
+ 9.7903470993042,
+ 5.166380405426025,
+ -0.28777560591697693,
+ -0.2217828333377838,
+ 0.6546372771263123,
+ -1.3741341829299927,
+ -0.06905987113714218,
+ -0.2470952719449997,
+ -2.2180323600769043,
+ -0.8327293395996094,
+ 1.2453497648239136,
+ 0.5926279425621033,
+ -1.6068185567855835,
+ -1.0254945755004883
+ ],
+ [
+ -6.326718807220459,
+ -4.639016151428223,
+ -4.542400360107422,
+ 0.4990499019622803,
+ -3.177238702774048,
+ -0.4241127073764801,
+ -0.1261882781982422,
+ 0.2598721385002136,
+ 0.4305728077888489,
+ -1.2703700065612793,
+ 0.31710106134414673,
+ -0.2681090831756592,
+ -0.5246409773826599,
+ 0.29372555017471313,
+ 0.9901218414306641
+ ],
+ [
+ 16.242389678955078,
+ 0.6297402381896973,
+ -1.7059905529022217,
+ -0.0549151711165905,
+ -0.34362682700157166,
+ -1.6732800006866455,
+ -3.051926612854004,
+ 1.4733350276947021,
+ 3.6234004497528076,
+ -0.22836175560951233,
+ -0.440629243850708,
+ -0.7063243985176086,
+ -1.0713413953781128,
+ 1.8751275539398193,
+ -0.03918338567018509
+ ],
+ [
+ -5.085692405700684,
+ 8.451180458068848,
+ 2.9798641204833984,
+ 0.2730255424976349,
+ 0.5276965498924255,
+ 1.5645930767059326,
+ -1.0717202425003052,
+ -0.11924038827419281,
+ -0.2377212643623352,
+ 3.920626640319824,
+ 0.8080355525016785,
+ -0.3927246332168579,
+ 0.67253577709198,
+ -1.6299381256103516,
+ 3.127800703048706
+ ],
+ [
+ -6.707473278045654,
+ -5.124661922454834,
+ -4.393282890319824,
+ 0.17494529485702515,
+ -2.683598756790161,
+ -2.769467830657959,
+ 0.5065394043922424,
+ 2.44641375541687,
+ -0.4258648157119751,
+ 1.762669563293457,
+ 0.12325850129127502,
+ 0.24289527535438538,
+ -1.3454374074935913,
+ -2.418065309524536,
+ 2.665771245956421
+ ],
+ [
+ -0.14083901047706604,
+ -9.37141227722168,
+ 14.517510414123535,
+ 2.3304924964904785,
+ 2.85042405128479,
+ 5.69500207901001,
+ 1.2898708581924438,
+ 6.259385585784912,
+ -1.300935983657837,
+ 0.0976652055978775,
+ 1.8388869762420654,
+ -1.776641845703125,
+ -0.43710508942604065,
+ 0.17136313021183014,
+ -1.3662117719650269
+ ],
+ [
+ -6.297425270080566,
+ 5.592845439910889,
+ 1.1806894540786743,
+ -0.38654688000679016,
+ 3.2900760173797607,
+ -1.4468146562576294,
+ 0.6100004315376282,
+ -0.5198421478271484,
+ -0.4323853850364685,
+ 5.152219772338867,
+ 1.3174995183944702,
+ 1.007414698600769,
+ -0.9000331163406372,
+ -0.05419003218412399,
+ -1.5231432914733887
+ ],
+ [
+ -6.456739902496338,
+ 5.197171211242676,
+ 0.14399752020835876,
+ -0.12529103457927704,
+ 8.440364837646484,
+ -2.600539207458496,
+ 3.9868228435516357,
+ 0.9177753925323486,
+ 0.24229921400547028,
+ 0.26818034052848816,
+ -1.3830994367599487,
+ 0.26142290234565735,
+ 0.2066006064414978,
+ 1.4369996786117554,
+ -2.144101858139038
+ ],
+ [
+ -2.528944969177246,
+ -11.639944076538086,
+ 12.095565795898438,
+ -0.5768436193466187,
+ -1.8936468362808228,
+ -2.339414358139038,
+ 0.4160315990447998,
+ -1.8174149990081787,
+ -0.21540291607379913,
+ -0.032230570912361145,
+ -0.7594102025032043,
+ 1.390856385231018,
+ 0.8787968158721924,
+ 0.43751659989356995,
+ -1.7597041130065918
+ ],
+ [
+ -4.65050745010376,
+ 10.544567108154297,
+ 4.646190643310547,
+ -0.11188820749521255,
+ -3.332406520843506,
+ 1.1126848459243774,
+ -0.5961830019950867,
+ 0.33941370248794556,
+ -0.38746178150177,
+ -0.09199847280979156,
+ -0.24880442023277283,
+ -0.3612813651561737,
+ 0.8004976511001587,
+ 0.11529719084501266,
+ 0.7940204739570618
+ ],
+ [
+ -2.1943278312683105,
+ -13.550276756286621,
+ 14.205401420593262,
+ -3.018681049346924,
+ 0.8631725907325745,
+ 2.4047107696533203,
+ -0.8960325717926025,
+ 4.227734088897705,
+ -0.43762335181236267,
+ -0.39123278856277466,
+ -1.142282247543335,
+ -0.6107078790664673,
+ -1.0329631567001343,
+ 1.1459721326828003,
+ -0.3556163012981415
+ ],
+ [
+ -6.2317705154418945,
+ 3.6727864742279053,
+ -1.3771032094955444,
+ -0.7047655582427979,
+ 7.146742820739746,
+ -5.798803806304932,
+ 2.7892417907714844,
+ 2.247192144393921,
+ -0.4616563618183136,
+ 0.21587316691875458,
+ 0.4115142524242401,
+ -1.1963955163955688,
+ 0.41764742136001587,
+ 0.5095536708831787,
+ -0.899638831615448
+ ],
+ [
+ -5.373989105224609,
+ 13.335378646850586,
+ 6.189030170440674,
+ 0.14043709635734558,
+ -2.387554883956909,
+ 1.5305428504943848,
+ 0.01027913112193346,
+ -0.14637736976146698,
+ -0.1187424510717392,
+ -0.7535691857337952,
+ 0.4314129054546356,
+ -1.2134536504745483,
+ -0.04901670292019844,
+ -0.6371603012084961,
+ 0.5591298341751099
+ ],
+ [
+ 17.821823120117188,
+ -0.25485149025917053,
+ -1.9803484678268433,
+ -0.5177862048149109,
+ -0.37415066361427307,
+ -0.5199428796768188,
+ -1.0882985591888428,
+ 0.4661526679992676,
+ 1.5783404111862183,
+ 0.2077028602361679,
+ 0.5663413405418396,
+ -0.4671148657798767,
+ -0.31087231636047363,
+ -0.8558345437049866,
+ -0.139353409409523
+ ],
+ [
+ 15.310354232788086,
+ 0.053190331906080246,
+ -1.0172150135040283,
+ -0.18213698267936707,
+ 0.13723047077655792,
+ -0.23507803678512573,
+ 0.19518843293190002,
+ 0.22078821063041687,
+ -0.8314805030822754,
+ -0.4297834634780884,
+ 1.8084313869476318,
+ -0.645946204662323,
+ -0.23242446780204773,
+ -0.02405226230621338,
+ -0.7864153385162354
+ ],
+ [
+ -6.70077657699585,
+ -3.4132466316223145,
+ -5.130453109741211,
+ -0.04329381510615349,
+ -2.6830921173095703,
+ -1.1003797054290771,
+ 0.7244337201118469,
+ 0.8376031517982483,
+ -0.4182494878768921,
+ -0.15311050415039062,
+ -0.00729042524471879,
+ 0.5862588882446289,
+ -0.2081400752067566,
+ 1.4174212217330933,
+ 0.6463549137115479
+ ],
+ [
+ -5.859715938568115,
+ 13.72706413269043,
+ 6.868424415588379,
+ -0.6869797706604004,
+ -2.3427467346191406,
+ -0.09417792409658432,
+ -0.0006599111366085708,
+ -0.47457894682884216,
+ -0.409212589263916,
+ 0.3001571595668793,
+ -0.3158469796180725,
+ -0.3952411711215973,
+ -0.6819210648536682,
+ -0.3258408308029175,
+ 0.6583566069602966
+ ],
+ [
+ 13.100767135620117,
+ 0.12599419057369232,
+ -0.6458606719970703,
+ 0.4434346854686737,
+ 1.470230221748352,
+ 0.6573288440704346,
+ 0.9330585598945618,
+ -0.18910011649131775,
+ -0.30485260486602783,
+ -0.8939131498336792,
+ 2.8151795864105225,
+ -1.1449055671691895,
+ -0.6575067043304443,
+ -1.3823187351226807,
+ 0.16978812217712402
+ ],
+ [
+ -4.8839521408081055,
+ -1.908344030380249,
+ -3.301401376724243,
+ 0.27744343876838684,
+ 2.218602418899536,
+ 1.6649351119995117,
+ -0.3770425319671631,
+ -1.072985053062439,
+ 0.1569482684135437,
+ -2.633009195327759,
+ 0.1388104110956192,
+ -0.9358571171760559,
+ -0.6474556922912598,
+ -1.3343490362167358,
+ 1.9104560613632202
+ ],
+ [
+ -6.014501571655273,
+ 4.092717170715332,
+ -0.9475468397140503,
+ -0.0755738839507103,
+ 6.917782783508301,
+ -3.7359681129455566,
+ 1.709553599357605,
+ 1.4159787893295288,
+ -0.4279165267944336,
+ -1.460460901260376,
+ -0.4610905647277832,
+ -1.881361722946167,
+ 1.318948745727539,
+ 1.823029637336731,
+ -1.6606550216674805
+ ],
+ [
+ -6.674319744110107,
+ 4.0125579833984375,
+ -0.7074280381202698,
+ 0.1789102852344513,
+ 9.122469902038574,
+ -4.062293529510498,
+ 1.4414114952087402,
+ 0.9908096790313721,
+ -0.3715239465236664,
+ 1.2750837802886963,
+ -1.1277999877929688,
+ -2.394761323928833,
+ 2.1140379905700684,
+ 2.528718948364258,
+ -1.989930510520935
+ ],
+ [
+ -5.434725284576416,
+ -0.9983865022659302,
+ -2.7392823696136475,
+ -0.001873499946668744,
+ 3.4648499488830566,
+ 6.911930561065674,
+ -3.020899534225464,
+ -4.122612953186035,
+ 0.3871079981327057,
+ 1.1812399625778198,
+ -0.7201542854309082,
+ 1.7566767930984497,
+ -0.43885576725006104,
+ 0.4902428090572357,
+ -0.9788342714309692
+ ],
+ [
+ -6.1821794509887695,
+ 8.72192096710205,
+ 3.274310827255249,
+ 0.06498304754495621,
+ 0.2917725741863251,
+ 2.142866373062134,
+ 0.15465059876441956,
+ -0.5309908986091614,
+ -0.2937319874763489,
+ 5.975095748901367,
+ -0.24037501215934753,
+ -1.3785063028335571,
+ 0.9252731800079346,
+ 1.38113534450531,
+ 0.09166807681322098
+ ],
+ [
+ -5.14548397064209,
+ -1.262101411819458,
+ -3.385972499847412,
+ -0.6729592084884644,
+ 2.753697395324707,
+ 4.237046241760254,
+ -1.944983720779419,
+ -3.0554721355438232,
+ 0.31687620282173157,
+ -0.013525580987334251,
+ -1.282463550567627,
+ -0.8039599061012268,
+ -1.2317630052566528,
+ -2.7767951488494873,
+ -0.735011100769043
+ ],
+ [
+ -4.454718112945557,
+ 2.738574266433716,
+ -1.8245248794555664,
+ -0.5964228510856628,
+ 7.907241344451904,
+ -3.483403444290161,
+ 1.06597101688385,
+ 0.7319890856742859,
+ 0.34719765186309814,
+ -1.7459906339645386,
+ -1.8961195945739746,
+ -0.7094967365264893,
+ -0.07113759964704514,
+ -0.8017848134040833,
+ -1.5151560306549072
+ ],
+ [
+ -3.6847455501556396,
+ -16.987323760986328,
+ 18.471031188964844,
+ -0.6944491267204285,
+ 1.0128029584884644,
+ 0.8281464576721191,
+ -1.486168384552002,
+ 5.446848392486572,
+ -0.08592332899570465,
+ -0.3188522160053253,
+ -1.0257080793380737,
+ 0.07065144926309586,
+ 1.4136335849761963,
+ -0.6332101821899414,
+ 1.0500080585479736
+ ],
+ [
+ -6.079846382141113,
+ 7.932499408721924,
+ 2.7653872966766357,
+ -0.4469989240169525,
+ 2.0020670890808105,
+ -0.2600184381008148,
+ -0.6714902520179749,
+ -0.42929795384407043,
+ -0.7865199446678162,
+ 7.137324810028076,
+ 0.18354478478431702,
+ 0.09730832278728485,
+ 0.6589099168777466,
+ -0.42563244700431824,
+ -0.5812025666236877
+ ],
+ [
+ -4.932258605957031,
+ 8.840142250061035,
+ 4.810303688049316,
+ 0.1172158420085907,
+ -0.6045385003089905,
+ -4.5635986328125,
+ 3.6588735580444336,
+ 3.8742294311523438,
+ 0.7749407291412354,
+ -8.474267959594727,
+ 6.5513081550598145,
+ 17.290620803833008,
+ -3.6812405586242676,
+ 7.544482231140137,
+ 2.5210959911346436
+ ],
+ [
+ -6.1930694580078125,
+ 1.693129539489746,
+ -1.1575472354888916,
+ -0.2916656732559204,
+ 6.849514484405518,
+ -4.934143543243408,
+ 1.2764300107955933,
+ 1.4114242792129517,
+ 0.23381470143795013,
+ 1.1167176961898804,
+ -2.0630404949188232,
+ -1.0687884092330933,
+ 0.18624411523342133,
+ -3.0956170558929443,
+ -2.137526512145996
+ ],
+ [
+ -6.033767223358154,
+ 12.409158706665039,
+ 8.114327430725098,
+ -0.76006019115448,
+ -3.4427430629730225,
+ 0.07314804196357727,
+ -1.3588935136795044,
+ 1.0366511344909668,
+ -0.6144195795059204,
+ -1.8855735063552856,
+ 0.3895631432533264,
+ -1.1193015575408936,
+ 0.10657630860805511,
+ -0.8922220468521118,
+ -0.179920494556427
+ ],
+ [
+ 16.85222625732422,
+ 0.1270725131034851,
+ -1.311887264251709,
+ -0.8193671703338623,
+ 0.6831282377243042,
+ -0.9571661353111267,
+ -2.0330052375793457,
+ -0.0926409661769867,
+ -0.8936906456947327,
+ -0.06749191135168076,
+ -0.24934914708137512,
+ -0.21551217138767242,
+ -0.47030553221702576,
+ -0.94297856092453,
+ -0.191094309091568
+ ],
+ [
+ -6.5200581550598145,
+ -2.6751952171325684,
+ -5.161164283752441,
+ -0.026537606492638588,
+ -2.0543389320373535,
+ -1.9100395441055298,
+ 1.3580313920974731,
+ 1.8200111389160156,
+ -0.7371600270271301,
+ 3.149224281311035,
+ 0.9781128168106079,
+ -0.5707170367240906,
+ -0.8324666619300842,
+ -0.8338364362716675,
+ 2.2553799152374268
+ ],
+ [
+ 16.63842010498047,
+ -0.5326351523399353,
+ -1.242287039756775,
+ -1.077056646347046,
+ -1.0682452917099,
+ -0.5691231489181519,
+ 0.010454259812831879,
+ -0.22671139240264893,
+ 2.0644047260284424,
+ 0.5023987293243408,
+ -2.920330762863159,
+ 0.6401874423027039,
+ 0.11253856122493744,
+ 0.9814208745956421,
+ 0.39489689469337463
+ ],
+ [
+ -5.735812664031982,
+ 3.61704683303833,
+ -0.5007290840148926,
+ -0.7631713151931763,
+ 7.253269672393799,
+ -4.151494979858398,
+ 1.9537160396575928,
+ 0.7676513195037842,
+ 0.1492743045091629,
+ 0.8771468997001648,
+ -0.2416204810142517,
+ -1.7191267013549805,
+ -0.4840770661830902,
+ -0.7733525633811951,
+ -1.535552978515625
+ ],
+ [
+ -7.677181720733643,
+ -4.415409564971924,
+ -5.1213603019714355,
+ -0.7844926714897156,
+ -2.3259036540985107,
+ 0.7468432784080505,
+ -1.0090713500976562,
+ -0.24943479895591736,
+ -0.18898536264896393,
+ 0.24669647216796875,
+ -0.2904670834541321,
+ -1.4001328945159912,
+ 0.055736199021339417,
+ 1.7272063493728638,
+ -0.14972659945487976
+ ],
+ [
+ 16.69170570373535,
+ 0.27603060007095337,
+ -0.4758330285549164,
+ 0.311064213514328,
+ -0.5128851532936096,
+ -0.2568327784538269,
+ 0.8708244562149048,
+ 0.2853836417198181,
+ 1.1428744792938232,
+ 0.1107705608010292,
+ -0.4721565842628479,
+ 0.3228705823421478,
+ -0.7256032824516296,
+ 0.5934153199195862,
+ 0.3359329402446747
+ ],
+ [
+ -6.792538166046143,
+ 4.124615669250488,
+ -0.394855797290802,
+ 0.21304890513420105,
+ 8.653674125671387,
+ -5.7984619140625,
+ 2.6319403648376465,
+ 1.5454316139221191,
+ -0.33840450644493103,
+ -2.1033852100372314,
+ -1.4219125509262085,
+ -1.1137155294418335,
+ 1.7965186834335327,
+ 0.4929806888103485,
+ -1.9859925508499146
+ ],
+ [
+ 15.842928886413574,
+ 0.5634710192680359,
+ -1.0533418655395508,
+ -0.43699344992637634,
+ -0.5049417018890381,
+ 1.1188185214996338,
+ 2.02581787109375,
+ -0.6816791296005249,
+ -0.4230431914329529,
+ -0.15731070935726166,
+ 0.4310527443885803,
+ -0.391789048910141,
+ -0.7412664294242859,
+ -0.2580079138278961,
+ 0.8541924953460693
+ ],
+ [
+ 16.910144805908203,
+ -0.1750563383102417,
+ -1.7566258907318115,
+ 1.0199686288833618,
+ 1.2065545320510864,
+ -0.3336258828639984,
+ 2.2726805210113525,
+ 0.1662144511938095,
+ 1.877784252166748,
+ -1.2252163887023926,
+ 0.9345066547393799,
+ -0.730570375919342,
+ -2.2481296062469482,
+ -1.8144524097442627,
+ 1.6883021593093872
+ ],
+ [
+ -4.065316677093506,
+ 8.165868759155273,
+ 3.6280627250671387,
+ -0.2332184910774231,
+ 0.37974312901496887,
+ 3.231814384460449,
+ -1.530228853225708,
+ -2.6377124786376953,
+ 0.584044873714447,
+ 2.562239170074463,
+ -0.08086249232292175,
+ 2.6250410079956055,
+ -1.725292682647705,
+ -0.746849000453949,
+ -0.7185608148574829
+ ],
+ [
+ -6.839229583740234,
+ 4.605571269989014,
+ 0.10555824637413025,
+ -0.48352620005607605,
+ 5.963280200958252,
+ -4.138943672180176,
+ 2.4601426124572754,
+ 0.8729100227355957,
+ -0.15987426042556763,
+ 0.057914964854717255,
+ -0.37883055210113525,
+ 1.071068525314331,
+ -1.755721092224121,
+ -0.6571905016899109,
+ -0.6874585151672363
+ ],
+ [
+ -6.193687915802002,
+ -0.24159087240695953,
+ -2.9683361053466797,
+ 0.1900593638420105,
+ 6.1905622482299805,
+ 0.1694948673248291,
+ -0.6006805896759033,
+ -0.7131437063217163,
+ -0.32946643233299255,
+ -1.0060927867889404,
+ -1.275110125541687,
+ -1.9658399820327759,
+ -6.399033009074628e-05,
+ -0.38161179423332214,
+ 0.7297350168228149
+ ],
+ [
+ -6.34757661819458,
+ -1.8202141523361206,
+ -3.834160566329956,
+ 0.29470711946487427,
+ 0.355691134929657,
+ 4.22029447555542,
+ -0.30862993001937866,
+ -1.6806085109710693,
+ 0.0040648761205375195,
+ 0.5231242775917053,
+ 0.40121015906333923,
+ 0.9768931865692139,
+ 0.6358144283294678,
+ 2.353426694869995,
+ -1.5278141498565674
+ ],
+ [
+ -6.12180233001709,
+ -0.7110816240310669,
+ -3.421389102935791,
+ -0.22498725354671478,
+ 4.0608086585998535,
+ 2.1706740856170654,
+ -0.8811895251274109,
+ -0.784561812877655,
+ 0.007135624065995216,
+ -1.0600026845932007,
+ -1.5403295755386353,
+ 0.1056235283613205,
+ -0.7503244876861572,
+ 0.781173050403595,
+ -0.49507826566696167
+ ],
+ [
+ -2.126591682434082,
+ -16.35332489013672,
+ 18.200510025024414,
+ -3.4105846881866455,
+ -0.1943015605211258,
+ -4.452500343322754,
+ 1.0011343955993652,
+ -5.117233753204346,
+ 0.07751704752445221,
+ -0.3601195812225342,
+ 0.09072856605052948,
+ -1.208369493484497,
+ -2.37139892578125,
+ 0.3882855772972107,
+ 1.4134467840194702
+ ],
+ [
+ -3.2271580696105957,
+ -12.704505920410156,
+ 12.380252838134766,
+ 1.4149694442749023,
+ 0.5859320163726807,
+ -2.441592216491699,
+ -0.19059066474437714,
+ -2.9037063121795654,
+ 0.5068321824073792,
+ -0.5785152316093445,
+ 0.19580970704555511,
+ 0.62396240234375,
+ -2.2243778705596924,
+ -3.456479787826538,
+ -0.07903476059436798
+ ],
+ [
+ 15.728907585144043,
+ 0.845786988735199,
+ -0.9989268779754639,
+ -0.026321599259972572,
+ 0.47805890440940857,
+ -0.37811169028282166,
+ -2.3856215476989746,
+ -0.029168277978897095,
+ -0.49119988083839417,
+ 0.2061157375574112,
+ 1.0140366554260254,
+ -0.19297462701797485,
+ 0.7871918678283691,
+ 1.3983629941940308,
+ 0.884014904499054
+ ],
+ [
+ -5.397256851196289,
+ -2.150092840194702,
+ -3.9420409202575684,
+ -0.5168575048446655,
+ 1.6236424446105957,
+ 0.7218470573425293,
+ -0.6026673316955566,
+ -0.7342190742492676,
+ -0.09276070445775986,
+ -0.5423229932785034,
+ 0.33622708916664124,
+ 1.3078808784484863,
+ -1.2183208465576172,
+ 0.6911581158638,
+ 0.08431994915008545
+ ],
+ [
+ -5.2851057052612305,
+ 0.702862024307251,
+ -3.125762701034546,
+ 0.5145718455314636,
+ 4.907112121582031,
+ 3.061554431915283,
+ -0.6382738947868347,
+ -2.2976479530334473,
+ 0.6576557159423828,
+ -3.09958553314209,
+ 1.018552303314209,
+ 1.8902299404144287,
+ -0.7691378593444824,
+ -0.2935553193092346,
+ 0.8778378367424011
+ ],
+ [
+ 15.76735782623291,
+ -0.17001917958259583,
+ -1.619789958000183,
+ -0.5488831996917725,
+ 0.4390573799610138,
+ -0.6335156559944153,
+ 0.1011955663561821,
+ -0.53117835521698,
+ 0.06588815152645111,
+ -0.7152013778686523,
+ 1.0647810697555542,
+ -0.5942334532737732,
+ -1.6026884317398071,
+ -1.2119481563568115,
+ 0.851684033870697
+ ],
+ [
+ -5.142185211181641,
+ -1.4328631162643433,
+ -3.18361759185791,
+ 1.3164862394332886,
+ 3.2097702026367188,
+ 6.1521735191345215,
+ -2.6366891860961914,
+ -3.102351427078247,
+ 0.35630983114242554,
+ 0.5910196304321289,
+ 0.6236009001731873,
+ 4.9129252433776855,
+ 0.7043859362602234,
+ -1.7294515371322632,
+ -4.215325355529785
+ ],
+ [
+ -5.486556053161621,
+ -1.3769551515579224,
+ -3.868558645248413,
+ 0.06542803347110748,
+ 0.5563022494316101,
+ 2.8160407543182373,
+ -1.1585392951965332,
+ -1.556806206703186,
+ 1.617812156677246,
+ -1.703887701034546,
+ 1.0620797872543335,
+ -0.28914347290992737,
+ 1.327103853225708,
+ 0.10540936142206192,
+ 1.4475270509719849
+ ],
+ [
+ -6.021926403045654,
+ 14.172562599182129,
+ 7.810983657836914,
+ -0.7395527958869934,
+ -3.465106964111328,
+ 0.2021714299917221,
+ -0.9633735418319702,
+ -0.40604886412620544,
+ -0.9637922048568726,
+ -0.7410628199577332,
+ -0.25685495138168335,
+ -2.4222538471221924,
+ 0.6457655429840088,
+ -2.188843250274658,
+ -0.10309528559446335
+ ],
+ [
+ -5.555271148681641,
+ -3.4205212593078613,
+ -3.890643835067749,
+ -0.09440476447343826,
+ 0.1802571415901184,
+ 2.252962112426758,
+ -1.8675806522369385,
+ 0.6152818202972412,
+ 0.04719618335366249,
+ -2.3474011421203613,
+ -0.6850166320800781,
+ -0.41462522745132446,
+ -1.4515678882598877,
+ -2.5200178623199463,
+ 1.4840788841247559
+ ],
+ [
+ -6.198551177978516,
+ -3.760737419128418,
+ -3.6509737968444824,
+ 0.26112988591194153,
+ -0.09634777903556824,
+ 1.7367209196090698,
+ -1.160361409187317,
+ -1.767102837562561,
+ 0.20315411686897278,
+ -1.2807830572128296,
+ -0.9761962294578552,
+ -0.038148555904626846,
+ -0.7406179308891296,
+ -3.2363038063049316,
+ 2.4157629013061523
+ ],
+ [
+ -5.813241004943848,
+ -3.8303000926971436,
+ -5.530303001403809,
+ 0.2740092873573303,
+ -0.9503493905067444,
+ 1.7983520030975342,
+ -0.5298846960067749,
+ -0.18277309834957123,
+ -0.28443455696105957,
+ -0.4401709735393524,
+ -0.22149065136909485,
+ -0.2389329969882965,
+ -0.06158684566617012,
+ 0.7658049464225769,
+ -0.6598126888275146
+ ],
+ [
+ -6.463676929473877,
+ -4.200480937957764,
+ -5.2344207763671875,
+ 0.3273541331291199,
+ -5.860932350158691,
+ -0.3781096041202545,
+ -0.5746645331382751,
+ 1.763120174407959,
+ -0.5198701620101929,
+ -0.15312014520168304,
+ -0.39248883724212646,
+ -1.110715627670288,
+ 1.700247049331665,
+ 1.0498038530349731,
+ -2.5028703212738037
+ ],
+ [
+ -6.1627326011657715,
+ -3.148367166519165,
+ -3.9814841747283936,
+ 0.33024829626083374,
+ -2.397129774093628,
+ -1.2450470924377441,
+ 1.393389105796814,
+ 0.7845653891563416,
+ -0.40267881751060486,
+ 2.434230089187622,
+ -0.1080431416630745,
+ -1.1420389413833618,
+ 0.4305382966995239,
+ 0.5266884565353394,
+ 0.20497022569179535
+ ],
+ [
+ 12.454237937927246,
+ -0.09287191182374954,
+ 1.0154423713684082,
+ 0.35131216049194336,
+ -1.1626118421554565,
+ 1.7326133251190186,
+ 6.313076019287109,
+ 0.45990416407585144,
+ -0.15590161085128784,
+ 0.9266138076782227,
+ -1.6149449348449707,
+ 0.44704562425613403,
+ -0.860653281211853,
+ -1.5649281740188599,
+ -1.2199493646621704
+ ],
+ [
+ -6.563511848449707,
+ 5.740373611450195,
+ 1.0224210023880005,
+ -0.40317416191101074,
+ 2.036006450653076,
+ -0.3096613585948944,
+ -0.2475014179944992,
+ -0.3447610139846802,
+ -0.39851391315460205,
+ 3.893097162246704,
+ 0.8642324209213257,
+ 1.2770129442214966,
+ 0.11976855248212814,
+ -1.3359696865081787,
+ -0.4605255126953125
+ ],
+ [
+ -5.632588863372803,
+ 8.069982528686523,
+ 3.0253942012786865,
+ 0.7629379630088806,
+ 0.48239651322364807,
+ 0.6835309267044067,
+ -1.540412425994873,
+ -1.3323733806610107,
+ -0.41174259781837463,
+ 6.609964847564697,
+ -0.8329766988754272,
+ 0.27316558361053467,
+ -0.7548940777778625,
+ -2.4467339515686035,
+ -1.7154620885849
+ ],
+ [
+ 16.055044174194336,
+ 0.7364309430122375,
+ -1.7055799961090088,
+ -0.2601950466632843,
+ 0.8317641019821167,
+ -0.5609895586967468,
+ -1.3501547574996948,
+ 0.3687279522418976,
+ -0.46842893958091736,
+ -0.1582537144422531,
+ 1.4598114490509033,
+ -0.49709194898605347,
+ -0.5156151056289673,
+ -0.24076053500175476,
+ 0.548172652721405
+ ],
+ [
+ -6.953810214996338,
+ -3.923348903656006,
+ -4.965150833129883,
+ -0.5229616761207581,
+ -1.518310546875,
+ -0.5291378498077393,
+ -0.38771021366119385,
+ 0.17787009477615356,
+ -0.3060648739337921,
+ -0.7512204647064209,
+ -0.4637655019760132,
+ 0.5357893109321594,
+ -0.29572853446006775,
+ 0.95058673620224,
+ -0.7995700836181641
+ ],
+ [
+ 16.47468376159668,
+ 0.5520190000534058,
+ -1.032327651977539,
+ 0.20777960121631622,
+ -0.4530528783798218,
+ -0.2023552507162094,
+ -0.9703431725502014,
+ 0.10853497684001923,
+ -1.0648045539855957,
+ -0.1463509202003479,
+ -0.1375129520893097,
+ 0.1273230016231537,
+ 1.0367685556411743,
+ 1.071170449256897,
+ 0.33179691433906555
+ ],
+ [
+ 15.717649459838867,
+ 1.4680194854736328,
+ -0.9806771278381348,
+ -0.3491624593734741,
+ -0.06795182824134827,
+ 0.3484379053115845,
+ -2.1277031898498535,
+ -0.0933847427368164,
+ -0.8735061287879944,
+ 0.20862671732902527,
+ 0.7582827210426331,
+ -0.39785128831863403,
+ 0.29983091354370117,
+ 1.3906135559082031,
+ 0.32917457818984985
+ ],
+ [
+ -7.060929775238037,
+ 13.572473526000977,
+ 7.498504161834717,
+ -0.16396132111549377,
+ -4.7448906898498535,
+ -2.2645530700683594,
+ 1.819135069847107,
+ 1.7375386953353882,
+ 0.5861872434616089,
+ -3.770596742630005,
+ 0.8259909152984619,
+ 1.6960102319717407,
+ -1.4602059125900269,
+ 0.4731135368347168,
+ 0.8878863453865051
+ ],
+ [
+ 17.37494468688965,
+ 0.39975646138191223,
+ -1.3164888620376587,
+ -0.2859053313732147,
+ 0.766148567199707,
+ 0.31579381227493286,
+ 1.1933690309524536,
+ -0.19997043907642365,
+ -0.6372908353805542,
+ -0.24248650670051575,
+ 0.5569870471954346,
+ 0.15446263551712036,
+ -0.5389887094497681,
+ 0.17224562168121338,
+ 1.6045938730239868
+ ],
+ [
+ -7.048969745635986,
+ -4.7444233894348145,
+ -4.886315822601318,
+ 0.6044252514839172,
+ -3.383493661880493,
+ -0.6911393404006958,
+ -0.24253389239311218,
+ 0.1761212795972824,
+ 0.2086455225944519,
+ -1.6408298015594482,
+ -0.26187586784362793,
+ -0.7196176648139954,
+ 0.0006670141592621803,
+ -0.1941906064748764,
+ -0.439665287733078
+ ],
+ [
+ -6.453530311584473,
+ -4.5186381340026855,
+ -5.807698726654053,
+ -0.41978949308395386,
+ -3.131531238555908,
+ -3.6113526821136475,
+ 1.0095123052597046,
+ 2.202704429626465,
+ -0.2935839891433716,
+ 2.1989667415618896,
+ 0.9091111421585083,
+ -0.08632414788007736,
+ -2.3427798748016357,
+ -2.77911639213562,
+ 2.7345948219299316
+ ],
+ [
+ -6.453597068786621,
+ -6.174638748168945,
+ -4.033801555633545,
+ 0.09753026813268661,
+ -1.8342828750610352,
+ -0.8856537938117981,
+ -0.3094361126422882,
+ 0.6346542835235596,
+ -0.16729845106601715,
+ -1.0864558219909668,
+ 0.393062949180603,
+ -0.6750361323356628,
+ -1.665480613708496,
+ -1.5235743522644043,
+ 0.25121742486953735
+ ],
+ [
+ -5.2807536125183105,
+ 5.199624538421631,
+ -0.6607793569564819,
+ -0.3725581169128418,
+ 6.967741966247559,
+ -1.7224106788635254,
+ 1.8026896715164185,
+ 0.8987315893173218,
+ -0.39361727237701416,
+ 1.4493111371994019,
+ 0.15259094536304474,
+ -2.0621323585510254,
+ 1.7183526754379272,
+ 0.9045999646186829,
+ 0.6019555330276489
+ ],
+ [
+ -5.44573450088501,
+ -1.8003910779953003,
+ -4.866376876831055,
+ -0.25944212079048157,
+ 2.2298812866210938,
+ 1.1123679876327515,
+ -0.532491147518158,
+ -0.6712276935577393,
+ 0.08461049944162369,
+ -1.6131587028503418,
+ -1.6509366035461426,
+ -1.7553420066833496,
+ 0.9089667201042175,
+ 1.2235372066497803,
+ 2.07614803314209
+ ],
+ [
+ -6.614140033721924,
+ -3.5204784870147705,
+ -5.141568183898926,
+ -0.07863745838403702,
+ -1.895725131034851,
+ -2.478670358657837,
+ 0.465487003326416,
+ 1.3462426662445068,
+ -0.3629695773124695,
+ 1.3632006645202637,
+ 0.49485209584236145,
+ -0.547432005405426,
+ -0.9093806147575378,
+ -2.2987465858459473,
+ 1.544570803642273
+ ],
+ [
+ -8.411136627197266,
+ -5.433815956115723,
+ -5.165317535400391,
+ 0.1597031205892563,
+ -3.500607967376709,
+ -0.4639797508716583,
+ -0.5162348747253418,
+ -0.1489281952381134,
+ 0.0336957685649395,
+ -1.3949940204620361,
+ 0.18362222611904144,
+ -1.2701555490493774,
+ 0.9533156752586365,
+ -0.5044304728507996,
+ -2.847531318664551
+ ],
+ [
+ -6.22203254699707,
+ 8.652993202209473,
+ 5.057187080383301,
+ -0.022827886044979095,
+ -1.95346999168396,
+ -3.041724443435669,
+ 1.2338248491287231,
+ 2.736158847808838,
+ 0.09104279428720474,
+ -3.2643039226531982,
+ 1.6099902391433716,
+ 5.666271686553955,
+ -3.309936761856079,
+ 1.327452301979065,
+ -0.08587050437927246
+ ],
+ [
+ -5.1187357902526855,
+ -0.9845396876335144,
+ -3.4617769718170166,
+ -0.366971492767334,
+ 3.851454257965088,
+ -0.724642813205719,
+ -0.16165126860141754,
+ 0.05612725391983986,
+ 0.3005759119987488,
+ 0.2369498759508133,
+ 0.6757530570030212,
+ 0.34368896484375,
+ -2.3127520084381104,
+ -3.7815043926239014,
+ 1.5165438652038574
+ ],
+ [
+ -5.747828006744385,
+ 13.191741943359375,
+ 7.4492082595825195,
+ 0.5313394665718079,
+ -3.7755792140960693,
+ 0.5770216584205627,
+ -1.4898253679275513,
+ -0.2630039155483246,
+ 0.19070200622081757,
+ 1.287041425704956,
+ 0.13310754299163818,
+ -1.2174794673919678,
+ 0.10592062771320343,
+ -1.3768612146377563,
+ -0.5701101422309875
+ ],
+ [
+ 16.886709213256836,
+ 0.430945485830307,
+ -1.0052765607833862,
+ -0.5038477182388306,
+ -0.0057471743784844875,
+ -0.9125931859016418,
+ -0.2598317563533783,
+ 0.29799649119377136,
+ -0.3420410752296448,
+ -0.08929479867219925,
+ 0.48862385749816895,
+ -0.3963267207145691,
+ 0.2045430839061737,
+ 0.48300236463546753,
+ -0.1408403217792511
+ ],
+ [
+ -7.524351596832275,
+ -5.105806827545166,
+ -5.30363130569458,
+ -0.03719491884112358,
+ -5.062799453735352,
+ -1.4689756631851196,
+ 0.509407103061676,
+ 0.8482363820075989,
+ -0.14915066957473755,
+ 1.0597683191299438,
+ -0.13827642798423767,
+ -1.5753674507141113,
+ 2.096705436706543,
+ 0.5985457301139832,
+ -1.391709804534912
+ ],
+ [
+ -6.869634628295898,
+ -4.139954090118408,
+ -3.850037097930908,
+ -0.5634969472885132,
+ -0.43848350644111633,
+ 1.4113686084747314,
+ -0.4059918224811554,
+ -1.4047906398773193,
+ 0.5753624439239502,
+ -1.83465576171875,
+ 0.5360394716262817,
+ 0.6204566955566406,
+ 0.1099749356508255,
+ -0.8460798859596252,
+ 1.4408271312713623
+ ],
+ [
+ 14.925935745239258,
+ -0.4074998199939728,
+ -1.4738237857818604,
+ -0.31285902857780457,
+ 0.20664726197719574,
+ -1.0716842412948608,
+ 0.5480701923370361,
+ 0.4761698842048645,
+ -0.15526659786701202,
+ -0.03797939047217369,
+ 1.6520509719848633,
+ -0.14115051925182343,
+ -1.2078596353530884,
+ -2.7829349040985107,
+ -0.3369837701320648
+ ],
+ [
+ -7.381149768829346,
+ -2.7750444412231445,
+ -5.675482273101807,
+ 0.5271058678627014,
+ 0.9061923623085022,
+ 1.5820975303649902,
+ -0.8034591674804688,
+ -1.0321238040924072,
+ 0.13378068804740906,
+ -2.7102177143096924,
+ -0.8823903799057007,
+ -0.2126581072807312,
+ -1.3668711185455322,
+ -1.0316190719604492,
+ 1.057979941368103
+ ],
+ [
+ -8.00684928894043,
+ 8.750338554382324,
+ 2.4455113410949707,
+ 0.2227385938167572,
+ -2.939723014831543,
+ 0.3272297978401184,
+ -0.5329667329788208,
+ 0.2927182912826538,
+ -0.8044423460960388,
+ -1.2536293268203735,
+ 0.8706832528114319,
+ -1.463799238204956,
+ 0.17332510650157928,
+ -0.3485916256904602,
+ 0.693541944026947
+ ],
+ [
+ -1.4843552112579346,
+ -14.451223373413086,
+ 14.614763259887695,
+ -0.7820783853530884,
+ -0.03890696540474892,
+ -1.95494544506073,
+ -0.4242260158061981,
+ -4.18278694152832,
+ -0.08530568331480026,
+ 0.7584887146949768,
+ -1.6289374828338623,
+ -0.23417527973651886,
+ -1.8760066032409668,
+ -0.5010749697685242,
+ 0.4189968705177307
+ ],
+ [
+ 17.19890785217285,
+ -0.0810442641377449,
+ -0.12003548443317413,
+ -0.8817811012268066,
+ -1.341315507888794,
+ -0.1502331644296646,
+ 1.3370479345321655,
+ 0.309091717004776,
+ -0.2204398363828659,
+ 0.09604229778051376,
+ -0.33026185631752014,
+ 0.38782039284706116,
+ 0.08168800920248032,
+ -0.7545603513717651,
+ -0.5315138101577759
+ ],
+ [
+ -6.0849928855896,
+ -3.2056069374084473,
+ -3.998224973678589,
+ 0.038113225251436234,
+ 0.041428279131650925,
+ 0.7153939008712769,
+ -1.3557159900665283,
+ -0.8790354132652283,
+ -0.4173247516155243,
+ -0.4184249937534332,
+ -0.9139978289604187,
+ -0.3091578483581543,
+ 0.09026209264993668,
+ -1.4589803218841553,
+ -0.0782725065946579
+ ],
+ [
+ 15.5570068359375,
+ -0.25444740056991577,
+ -0.8665013313293457,
+ -0.6254184246063232,
+ -0.6039519309997559,
+ -2.786053419113159,
+ -0.9795847535133362,
+ -0.05604514107108116,
+ -0.3830505609512329,
+ -0.40762537717819214,
+ -2.926361560821533,
+ 0.6275385022163391,
+ -0.7668434381484985,
+ -1.2301853895187378,
+ -0.8546693325042725
+ ],
+ [
+ 17.58441734313965,
+ 0.37326887249946594,
+ -1.3148761987686157,
+ -0.8148215413093567,
+ -0.399021714925766,
+ -1.5881764888763428,
+ -3.564412832260132,
+ 0.034354809671640396,
+ -0.6326979994773865,
+ 0.4899085760116577,
+ -0.6580378413200378,
+ 0.8099148869514465,
+ 0.7185613512992859,
+ 1.2419477701187134,
+ 0.5197818875312805
+ ],
+ [
+ -6.28576135635376,
+ -4.536136627197266,
+ -6.06014347076416,
+ -0.251470148563385,
+ -2.140618324279785,
+ -0.3298644721508026,
+ -0.33550775051116943,
+ 1.087011456489563,
+ -0.3260304033756256,
+ -1.2823131084442139,
+ -0.2834818959236145,
+ -0.303975909948349,
+ -0.6538190245628357,
+ -0.18441249430179596,
+ 0.9065616726875305
+ ],
+ [
+ -4.872636795043945,
+ 6.820934772491455,
+ 1.7536709308624268,
+ -0.15123134851455688,
+ 1.1518877744674683,
+ 2.2417616844177246,
+ -1.7583619356155396,
+ -1.7983477115631104,
+ 0.5114458799362183,
+ -1.3062365055084229,
+ -0.7234511375427246,
+ 4.291016578674316,
+ -2.2845311164855957,
+ -0.5545896291732788,
+ -2.670872926712036
+ ],
+ [
+ -5.160970687866211,
+ -2.074739456176758,
+ -3.272693634033203,
+ 1.1178474426269531,
+ 3.629728078842163,
+ 5.443666458129883,
+ -2.7877445220947266,
+ -1.927201509475708,
+ 0.2564472556114197,
+ -0.3967757821083069,
+ 0.7764869928359985,
+ 1.9794913530349731,
+ -0.08380229771137238,
+ -0.3363972008228302,
+ -2.1833081245422363
+ ],
+ [
+ -6.141955852508545,
+ 6.1545329093933105,
+ 1.19171142578125,
+ 0.20730042457580566,
+ 4.783782958984375,
+ -1.0015720129013062,
+ 1.1161302328109741,
+ 0.4406246840953827,
+ 0.047709885984659195,
+ 3.518287181854248,
+ 0.06478435546159744,
+ 1.3730493783950806,
+ 1.0073041915893555,
+ 0.6883945465087891,
+ -0.8127768635749817
+ ],
+ [
+ 9.234298706054688,
+ 2.392899513244629,
+ -1.23753023147583,
+ -0.6154603958129883,
+ 1.471860647201538,
+ -0.4303331971168518,
+ -1.1232966184616089,
+ -0.07292213290929794,
+ -0.1174088716506958,
+ -0.1466832309961319,
+ 0.6318751573562622,
+ 3.2080295085906982,
+ -0.9979432225227356,
+ 0.03259330242872238,
+ 1.9949012994766235
+ ],
+ [
+ 15.118252754211426,
+ 0.5416136980056763,
+ -1.5819413661956787,
+ 0.07873179018497467,
+ 0.7751050591468811,
+ -0.06468062847852707,
+ 1.0371202230453491,
+ -0.6629642248153687,
+ -0.23144283890724182,
+ -0.5694121718406677,
+ 2.0237784385681152,
+ 0.050239384174346924,
+ -0.7915604710578918,
+ 0.1399315744638443,
+ -0.5963683128356934
+ ],
+ [
+ -7.368793487548828,
+ -5.195271015167236,
+ -5.421992301940918,
+ 0.014620660804212093,
+ -4.0812811851501465,
+ -0.40707406401634216,
+ 0.3908478617668152,
+ 0.7213388681411743,
+ -0.3492436110973358,
+ -0.7954406142234802,
+ 0.16042093932628632,
+ -0.9433107972145081,
+ -0.18954627215862274,
+ 0.2907215356826782,
+ 0.19131653010845184
+ ],
+ [
+ -6.7479963302612305,
+ 6.492605209350586,
+ -0.2652687728404999,
+ -0.38964149355888367,
+ 5.879523754119873,
+ -3.609097957611084,
+ 1.254077672958374,
+ 1.3584873676300049,
+ 0.29477113485336304,
+ -0.03912695124745369,
+ 1.0431238412857056,
+ 4.164698600769043,
+ -1.2134745121002197,
+ 4.124520778656006,
+ -0.7492610216140747
+ ],
+ [
+ 16.55554962158203,
+ 0.8154615759849548,
+ -1.8081904649734497,
+ -0.2746868431568146,
+ -0.004903312772512436,
+ -0.9734160304069519,
+ -0.07400308549404144,
+ 0.4034442603588104,
+ -0.6995950937271118,
+ -0.16584272682666779,
+ 0.6635292768478394,
+ 0.5704999566078186,
+ 0.07075589150190353,
+ 1.3086507320404053,
+ 0.11759471893310547
+ ],
+ [
+ -6.581644535064697,
+ -2.6804306507110596,
+ -3.6151294708251953,
+ -0.423348993062973,
+ -2.5976686477661133,
+ -0.8196505308151245,
+ 0.965988278388977,
+ 2.3603904247283936,
+ -0.16017208993434906,
+ 3.597660779953003,
+ 0.4711466133594513,
+ 0.9850522875785828,
+ 0.9617981910705566,
+ -1.2653627395629883,
+ 2.790325403213501
+ ],
+ [
+ 15.279899597167969,
+ 0.6962617039680481,
+ -0.571417510509491,
+ 0.05474783107638359,
+ 0.18591764569282532,
+ 1.6707340478897095,
+ 1.5761053562164307,
+ -0.3427935242652893,
+ -0.6173341274261475,
+ 0.4618779718875885,
+ 1.5787231922149658,
+ -0.7776580452919006,
+ -0.09195568412542343,
+ 0.7547168731689453,
+ 0.47802576422691345
+ ],
+ [
+ 15.996201515197754,
+ 1.1725062131881714,
+ -1.0615843534469604,
+ -0.9764143824577332,
+ -0.4292951822280884,
+ 0.35146528482437134,
+ 1.9544240236282349,
+ -0.45521974563598633,
+ -0.093844935297966,
+ 0.8419609665870667,
+ -1.382050633430481,
+ -0.3658134639263153,
+ 0.36194297671318054,
+ -0.22937455773353577,
+ 0.32468536496162415
+ ],
+ [
+ -5.417383670806885,
+ -2.818613052368164,
+ -4.685920715332031,
+ 0.7374101877212524,
+ 1.1419175863265991,
+ 1.9975084066390991,
+ -0.057666290551424026,
+ -0.34550175070762634,
+ -0.14376184344291687,
+ -2.8444597721099854,
+ -0.4295022487640381,
+ -0.5495659708976746,
+ -0.8620116114616394,
+ -0.6380635499954224,
+ 3.2620086669921875
+ ],
+ [
+ 11.501131057739258,
+ -0.7672261595726013,
+ 1.2208549976348877,
+ 3.1452932357788086,
+ 0.7648532390594482,
+ 2.8339107036590576,
+ 4.563808917999268,
+ 0.8631947040557861,
+ -1.020896077156067,
+ -0.20339006185531616,
+ 14.318796157836914,
+ -3.725689649581909,
+ 0.2653120458126068,
+ 1.2506954669952393,
+ -6.719168186187744
+ ],
+ [
+ -5.36683988571167,
+ 4.108222484588623,
+ -0.06259981542825699,
+ 0.10032312572002411,
+ 8.400591850280762,
+ -5.355870723724365,
+ 2.053061008453369,
+ 1.199594497680664,
+ -0.07363522052764893,
+ 0.045987147837877274,
+ -1.3434216976165771,
+ -2.4783387184143066,
+ 1.0184440612792969,
+ 0.8838633298873901,
+ -1.3048652410507202
+ ],
+ [
+ 16.859058380126953,
+ 0.18271297216415405,
+ -0.6091774106025696,
+ -0.38606876134872437,
+ -0.4618530869483948,
+ -1.4552910327911377,
+ -0.3090725541114807,
+ -0.007869375869631767,
+ -0.3481723368167877,
+ -0.004233176354318857,
+ -1.127031922340393,
+ 0.22830325365066528,
+ 0.20028750598430634,
+ -0.5324158072471619,
+ -0.5372936129570007
+ ],
+ [
+ -5.826787948608398,
+ 11.367147445678711,
+ 5.868143558502197,
+ -0.228071391582489,
+ -1.6722501516342163,
+ 0.09428392350673676,
+ -0.6344202756881714,
+ -0.11633660644292831,
+ -0.5443027019500732,
+ -0.5492139458656311,
+ -0.2483176440000534,
+ -1.4035032987594604,
+ 0.6200864911079407,
+ -0.18879108130931854,
+ 0.447535902261734
+ ],
+ [
+ -5.386014938354492,
+ 3.375802755355835,
+ -0.8702671527862549,
+ -0.31144946813583374,
+ 3.910175323486328,
+ 2.03147554397583,
+ -1.0265295505523682,
+ -1.6475725173950195,
+ 0.4237372875213623,
+ 3.173679828643799,
+ -0.2508271038532257,
+ -0.10177332907915115,
+ 0.6949865818023682,
+ -0.15194419026374817,
+ -0.6307065486907959
+ ],
+ [
+ 16.394615173339844,
+ 0.2659149169921875,
+ -0.3357762396335602,
+ -0.18157914280891418,
+ -0.24385181069374084,
+ 0.9219197630882263,
+ 0.5896589159965515,
+ -0.22082433104515076,
+ -0.2512436807155609,
+ 0.5003365278244019,
+ 1.0188803672790527,
+ -1.2100757360458374,
+ -0.3206968307495117,
+ -0.6016821265220642,
+ 0.06487230956554413
+ ],
+ [
+ -7.331903457641602,
+ -3.2044765949249268,
+ -5.396541118621826,
+ 0.0027803441043943167,
+ -2.4967451095581055,
+ -0.1993507742881775,
+ 0.1999778002500534,
+ 1.0773028135299683,
+ -0.2123643308877945,
+ -1.2595369815826416,
+ 0.3230036795139313,
+ -0.6352567076683044,
+ -0.4259500205516815,
+ 0.7701215744018555,
+ 0.18796978890895844
+ ],
+ [
+ -5.914208889007568,
+ 14.346423149108887,
+ 6.78137731552124,
+ 0.06276730448007584,
+ -3.448211908340454,
+ 1.0262383222579956,
+ -1.0991966724395752,
+ -0.5940549969673157,
+ -0.4859949052333832,
+ -1.2414205074310303,
+ 0.21872079372406006,
+ -0.8738023638725281,
+ -0.5549202561378479,
+ -0.22040921449661255,
+ 0.2910454273223877
+ ],
+ [
+ 13.503969192504883,
+ 0.1413235068321228,
+ -2.313829183578491,
+ -0.0800095871090889,
+ 0.6515366435050964,
+ 0.08354594558477402,
+ -2.4160614013671875,
+ -0.36947181820869446,
+ -0.5229675769805908,
+ -0.471690833568573,
+ 0.9363477230072021,
+ -0.3156624436378479,
+ 1.0759471654891968,
+ 1.16020929813385,
+ 1.0745742321014404
+ ],
+ [
+ -6.7679643630981445,
+ 4.004916667938232,
+ -0.05965685099363327,
+ -0.05944298207759857,
+ 6.585826873779297,
+ -4.777495384216309,
+ 2.093461275100708,
+ 1.0655291080474854,
+ -0.35533562302589417,
+ 0.979006826877594,
+ 0.04529023543000221,
+ -1.1027817726135254,
+ 0.5973372459411621,
+ 0.5644313097000122,
+ -0.8433760404586792
+ ],
+ [
+ -5.373757362365723,
+ 1.4737225770950317,
+ -1.522688388824463,
+ -0.5385514497756958,
+ 5.611554145812988,
+ -4.455031394958496,
+ 1.727871298789978,
+ 2.0655105113983154,
+ -0.16636447608470917,
+ -2.4985122680664062,
+ 0.4269803464412689,
+ -0.04414469748735428,
+ -0.8147880434989929,
+ -1.0624991655349731,
+ -0.2164226770401001
+ ],
+ [
+ 15.313660621643066,
+ 0.699398398399353,
+ -0.017433706670999527,
+ -0.2833401560783386,
+ -1.0091381072998047,
+ 0.9403733611106873,
+ 0.4416939318180084,
+ 0.13470441102981567,
+ -0.12380918860435486,
+ -0.3785853683948517,
+ -0.8412065505981445,
+ -0.500156581401825,
+ 0.12162832170724869,
+ -0.9573531746864319,
+ 0.5355179309844971
+ ],
+ [
+ -4.281583786010742,
+ 12.882100105285645,
+ 6.964280128479004,
+ -0.3018713891506195,
+ -3.061074733734131,
+ 1.3300954103469849,
+ 0.4284987449645996,
+ 0.8029297590255737,
+ 0.10029462724924088,
+ -1.2664140462875366,
+ -1.0315101146697998,
+ -2.0341694355010986,
+ 0.9003301858901978,
+ -0.4190327525138855,
+ -0.3948363959789276
+ ],
+ [
+ -1.2500795125961304,
+ -10.68753433227539,
+ 12.946601867675781,
+ -2.123971939086914,
+ 0.9700509905815125,
+ 3.663595199584961,
+ -1.4947175979614258,
+ 3.4506301879882812,
+ -0.22909648716449738,
+ 0.6409482359886169,
+ -1.0287599563598633,
+ 1.3846304416656494,
+ -1.9578843116760254,
+ -3.550295352935791,
+ -1.6108314990997314
+ ],
+ [
+ -7.6905107498168945,
+ -4.06799840927124,
+ -4.622145652770996,
+ 0.4274730980396271,
+ -5.406381607055664,
+ -1.8483288288116455,
+ 2.26678729057312,
+ 0.6814031600952148,
+ 0.3127233684062958,
+ -0.8189547061920166,
+ -1.9091731309890747,
+ -0.07897820323705673,
+ 0.027494965121150017,
+ -2.0938780307769775,
+ -3.025592565536499
+ ],
+ [
+ -5.315820217132568,
+ 0.8708693981170654,
+ -2.9041709899902344,
+ -0.0762491449713707,
+ 5.350043296813965,
+ -0.9130520224571228,
+ 1.0123212337493896,
+ -0.534661054611206,
+ 0.3767729699611664,
+ -1.098626971244812,
+ -0.75087571144104,
+ -2.5179948806762695,
+ 0.7236995697021484,
+ 0.9275928735733032,
+ 0.6163321137428284
+ ],
+ [
+ -6.598985195159912,
+ -3.7572877407073975,
+ -4.6457085609436035,
+ 0.018884122371673584,
+ -1.3105138540267944,
+ 2.256241798400879,
+ -0.1920509934425354,
+ -1.1520471572875977,
+ -0.3878231942653656,
+ -0.190560981631279,
+ -0.26007652282714844,
+ -0.3436100482940674,
+ 0.05565290525555611,
+ 1.8951059579849243,
+ -0.6436958312988281
+ ],
+ [
+ -4.600231170654297,
+ -3.162822961807251,
+ -3.551182746887207,
+ -0.13015061616897583,
+ 2.1080260276794434,
+ 2.6189024448394775,
+ -0.7048113346099854,
+ -1.9455394744873047,
+ 0.3540889322757721,
+ -3.001779794692993,
+ -0.9470803141593933,
+ 0.26709458231925964,
+ 0.4966341555118561,
+ -0.6748549342155457,
+ 1.2352863550186157
+ ],
+ [
+ -4.615818023681641,
+ 9.347359657287598,
+ 5.341259479522705,
+ -0.6127427220344543,
+ -3.4046127796173096,
+ -4.90465784072876,
+ -0.21042653918266296,
+ 3.983189582824707,
+ 25.007095336914062,
+ -3.9341423511505127,
+ 0.7415090799331665,
+ 2.9939441680908203,
+ -5.162024974822998,
+ 4.407095909118652,
+ -4.088258266448975
+ ],
+ [
+ 17.102569580078125,
+ 1.1297510862350464,
+ -1.4648122787475586,
+ -0.35679659247398376,
+ 0.06497315317392349,
+ -0.45602506399154663,
+ -1.3658151626586914,
+ 0.11316290497779846,
+ -0.4466085135936737,
+ -0.08553099632263184,
+ -0.04114395007491112,
+ 0.527037501335144,
+ -0.6459896564483643,
+ -0.0664970651268959,
+ 1.4958083629608154
+ ],
+ [
+ -6.184813022613525,
+ 7.9065375328063965,
+ 2.545173168182373,
+ 0.3063340485095978,
+ -0.3733074963092804,
+ -0.6903794407844543,
+ 0.16255195438861847,
+ 0.5221481919288635,
+ 1.2158558368682861,
+ 3.700249433517456,
+ 0.43516507744789124,
+ 3.734369993209839,
+ -1.5860105752944946,
+ 0.5706697106361389,
+ -2.5257794857025146
+ ],
+ [
+ -4.976676940917969,
+ -1.3449783325195312,
+ -2.8286001682281494,
+ -0.07347513735294342,
+ 6.264525890350342,
+ -0.5560336709022522,
+ -0.7395226955413818,
+ -0.45485854148864746,
+ 0.2170872837305069,
+ -4.129927158355713,
+ -1.1635513305664062,
+ -1.2348246574401855,
+ -1.31836998462677,
+ -3.557253360748291,
+ -0.9474031925201416
+ ],
+ [
+ 14.391002655029297,
+ 0.07539080828428268,
+ -1.4267610311508179,
+ -0.7369380593299866,
+ 0.6644213199615479,
+ -0.19788677990436554,
+ 0.4635968804359436,
+ -0.3099837601184845,
+ 1.5987117290496826,
+ -0.2629241943359375,
+ 2.787721872329712,
+ -0.9706678986549377,
+ -1.4905846118927002,
+ -0.2793314754962921,
+ -0.3204517662525177
+ ],
+ [
+ -6.591704368591309,
+ -3.952514410018921,
+ -5.396167278289795,
+ 0.15521956980228424,
+ -1.2330482006072998,
+ 0.08562842011451721,
+ 0.5654276609420776,
+ 0.5310455560684204,
+ -0.3284682631492615,
+ -0.41161367297172546,
+ 0.8810088634490967,
+ -0.09545514732599258,
+ -0.8268296122550964,
+ 0.1819719523191452,
+ -0.23854102194309235
+ ],
+ [
+ 15.348514556884766,
+ -0.12318442761898041,
+ -1.4094529151916504,
+ 0.11577431112527847,
+ -0.010065248236060143,
+ -1.3660989999771118,
+ -2.0384929180145264,
+ 0.8178081512451172,
+ 2.43208384513855,
+ 0.4734187126159668,
+ 0.6365599036216736,
+ -0.06012937054038048,
+ 1.0061110258102417,
+ 1.4513602256774902,
+ 0.5953408479690552
+ ],
+ [
+ -5.954479694366455,
+ -3.8916141986846924,
+ -5.163046836853027,
+ -0.5312680602073669,
+ -4.6248579025268555,
+ -0.7909315228462219,
+ 0.8929393887519836,
+ 0.7275758981704712,
+ -0.11722560226917267,
+ -0.212258979678154,
+ -0.7436587810516357,
+ -1.0322000980377197,
+ 1.5617783069610596,
+ 1.3996336460113525,
+ -1.1171494722366333
+ ],
+ [
+ -6.448538780212402,
+ -4.5111823081970215,
+ -4.6458539962768555,
+ -0.3479244112968445,
+ -2.8649239540100098,
+ -1.5503325462341309,
+ -0.795987069606781,
+ 1.6032171249389648,
+ -0.4762885868549347,
+ -0.9234229922294617,
+ -0.7172319293022156,
+ -0.563379168510437,
+ -0.9619304537773132,
+ -0.12964513897895813,
+ -0.3518214225769043
+ ],
+ [
+ -6.629207134246826,
+ -3.3067867755889893,
+ -4.081345558166504,
+ 1.3129019737243652,
+ -3.6552071571350098,
+ -0.10967211425304413,
+ -0.6427248120307922,
+ -0.3429478704929352,
+ -0.05184370279312134,
+ 1.9944301843643188,
+ 0.02649965137243271,
+ 1.1875909566879272,
+ 2.347770929336548,
+ 1.703338623046875,
+ -2.5857841968536377
+ ],
+ [
+ -6.143009662628174,
+ 2.7287564277648926,
+ -1.4027280807495117,
+ -0.16674847900867462,
+ 7.05440616607666,
+ -4.049267768859863,
+ 0.8007529377937317,
+ 1.1251388788223267,
+ -0.1471835970878601,
+ -1.2861535549163818,
+ -1.053173542022705,
+ -2.496473789215088,
+ 0.885018527507782,
+ 0.9555114507675171,
+ -0.7988075017929077
+ ],
+ [
+ -7.908029556274414,
+ -6.1812849044799805,
+ -5.704342842102051,
+ -0.5218104124069214,
+ -5.242709159851074,
+ -2.2886412143707275,
+ 0.3473987579345703,
+ 0.03891615942120552,
+ -0.09657543152570724,
+ -2.4194998741149902,
+ -0.1268320530653,
+ -2.803461790084839,
+ 0.2231648564338684,
+ -0.8405543565750122,
+ -0.04769641533493996
+ ],
+ [
+ -4.711842060089111,
+ 7.55436897277832,
+ 3.3896310329437256,
+ -0.2687242031097412,
+ -1.6266106367111206,
+ 1.3415195941925049,
+ -0.7071898579597473,
+ -0.8578293919563293,
+ 0.5229393839836121,
+ -2.317815065383911,
+ -1.2514042854309082,
+ -0.6815071105957031,
+ 0.324026495218277,
+ -0.9267949461936951,
+ -1.134938359260559
+ ],
+ [
+ -8.120716094970703,
+ -4.335094451904297,
+ -5.645213603973389,
+ 0.3677743971347809,
+ -3.2917768955230713,
+ -0.3064234256744385,
+ -0.4712369441986084,
+ 0.36187177896499634,
+ 0.35335075855255127,
+ -0.592014491558075,
+ 0.1871861070394516,
+ -1.196291208267212,
+ 0.02895771525800228,
+ 0.8528544306755066,
+ 0.9744959473609924
+ ],
+ [
+ -5.880974769592285,
+ -0.3670935332775116,
+ -2.4090466499328613,
+ 0.48119646310806274,
+ -0.9269082546234131,
+ 3.0234975814819336,
+ -0.9428695440292358,
+ -0.8672193884849548,
+ -0.3591618835926056,
+ 2.949249744415283,
+ 1.520254135131836,
+ 0.903583824634552,
+ 0.680966317653656,
+ 3.914811134338379,
+ -1.6413958072662354
+ ],
+ [
+ -6.998774528503418,
+ -4.374878883361816,
+ -4.474905014038086,
+ 2.3528892993927,
+ -3.4421610832214355,
+ -1.224019169807434,
+ 0.5164800882339478,
+ 1.30836820602417,
+ -0.4301678538322449,
+ 3.8623580932617188,
+ 1.2893476486206055,
+ 1.691411018371582,
+ -1.276428461074829,
+ -1.9213443994522095,
+ 1.8252136707305908
+ ],
+ [
+ 16.342836380004883,
+ 0.5079377889633179,
+ -1.2915213108062744,
+ -0.48686516284942627,
+ 0.0076100463047623634,
+ -1.6208908557891846,
+ -1.1608275175094604,
+ 0.3737008273601532,
+ -0.7286328077316284,
+ 0.13449415564537048,
+ -1.9173510074615479,
+ 0.9259527325630188,
+ 0.27347084879875183,
+ 0.7196119427680969,
+ 0.37734493613243103
+ ],
+ [
+ -6.060410499572754,
+ -3.8269269466400146,
+ -5.349358081817627,
+ -0.01565159671008587,
+ 0.41567209362983704,
+ 1.8115620613098145,
+ -1.4353262186050415,
+ -1.588179349899292,
+ 0.19438281655311584,
+ -1.153036117553711,
+ -0.07157095521688461,
+ -0.8317083716392517,
+ -1.2015817165374756,
+ -1.584916114807129,
+ 0.25163349509239197
+ ],
+ [
+ -5.3356032371521,
+ 10.898844718933105,
+ 4.7295145988464355,
+ 0.7095488905906677,
+ -1.8105360269546509,
+ 1.0235750675201416,
+ 0.04352681338787079,
+ -0.9405864477157593,
+ 0.15568877756595612,
+ 0.13484495878219604,
+ -1.2121975421905518,
+ -0.6832903027534485,
+ -1.2783591747283936,
+ 0.23165731132030487,
+ 0.28848057985305786
+ ],
+ [
+ 15.197884559631348,
+ 0.2615242600440979,
+ -1.3064310550689697,
+ 2.0120091438293457,
+ 0.6069387793540955,
+ -0.7562810778617859,
+ -2.041457414627075,
+ -0.03376109153032303,
+ -0.22705939412117004,
+ 0.4391613304615021,
+ 3.900758981704712,
+ -0.8604841828346252,
+ 1.171325922012329,
+ 1.2827467918395996,
+ -0.04904978349804878
+ ],
+ [
+ -6.510202407836914,
+ 7.814167022705078,
+ 2.4003524780273438,
+ 0.8736487627029419,
+ -3.830332040786743,
+ 1.018011450767517,
+ -0.5830634832382202,
+ -0.030634313821792603,
+ -0.12075601518154144,
+ -1.2960249185562134,
+ 0.6665583848953247,
+ 0.05568848177790642,
+ -0.11123475432395935,
+ 2.039278745651245,
+ 0.46275582909584045
+ ],
+ [
+ -7.10549783706665,
+ -4.803020477294922,
+ -5.006899833679199,
+ 2.138461112976074,
+ -4.95953369140625,
+ -2.1354379653930664,
+ 0.389918714761734,
+ 0.798849880695343,
+ -0.3647337555885315,
+ -0.7648957371711731,
+ -0.8313960433006287,
+ -0.8962271213531494,
+ -1.4100801944732666,
+ -1.759772777557373,
+ -0.9369853138923645
+ ],
+ [
+ -6.480328559875488,
+ -4.1455769538879395,
+ -3.8551201820373535,
+ 0.41581639647483826,
+ -2.9452550411224365,
+ 0.8703669905662537,
+ 0.5872851610183716,
+ 0.29847538471221924,
+ -0.15152958035469055,
+ 0.06933263689279556,
+ 0.20140309631824493,
+ 0.42233529686927795,
+ 1.535245656967163,
+ 1.7842042446136475,
+ -0.19458693265914917
+ ],
+ [
+ -4.905895709991455,
+ 9.918049812316895,
+ 4.756811618804932,
+ 0.5317060947418213,
+ 0.5507330894470215,
+ -2.5775487422943115,
+ 1.6600838899612427,
+ 0.682368814945221,
+ 0.03741747885942459,
+ -2.6927003860473633,
+ 1.1372499465942383,
+ 3.2438089847564697,
+ -1.4996825456619263,
+ 2.4341602325439453,
+ 0.20408697426319122
+ ],
+ [
+ 14.603556632995605,
+ 0.4518592357635498,
+ -1.686720609664917,
+ -0.22545838356018066,
+ 0.24844618141651154,
+ -0.17753185331821442,
+ -0.12948383390903473,
+ 0.13808351755142212,
+ 1.851628303527832,
+ 0.7129397988319397,
+ 2.157816171646118,
+ -0.11418919265270233,
+ -0.30216795206069946,
+ -0.01868320070207119,
+ -0.5167483687400818
+ ],
+ [
+ -5.854101657867432,
+ -1.9088225364685059,
+ -4.041264057159424,
+ -0.31440961360931396,
+ 3.5374433994293213,
+ 0.889319121837616,
+ -0.7817341089248657,
+ -0.43372172117233276,
+ 0.07150017470121384,
+ 0.021922988817095757,
+ -1.0299962759017944,
+ -0.9412795901298523,
+ -0.16720059514045715,
+ -1.0668143033981323,
+ 0.5788196921348572
+ ],
+ [
+ -4.882711410522461,
+ -2.2223639488220215,
+ 17.194150924682617,
+ -1.2252739667892456,
+ 0.7118556499481201,
+ 2.058480978012085,
+ -2.451023578643799,
+ 2.7517335414886475,
+ 0.9755515456199646,
+ 1.360844373703003,
+ -1.1085257530212402,
+ -1.327622413635254,
+ -0.6160000562667847,
+ 0.5330684781074524,
+ 1.7370065450668335
+ ],
+ [
+ -3.279736042022705,
+ -14.209259033203125,
+ 14.40494155883789,
+ -0.6951738595962524,
+ -0.7899683713912964,
+ -2.055312156677246,
+ 0.887896716594696,
+ -5.892005920410156,
+ 2.5284578800201416,
+ 0.22934314608573914,
+ 1.38213050365448,
+ 0.6089062690734863,
+ 1.6349722146987915,
+ 0.23385000228881836,
+ -1.6554076671600342
+ ],
+ [
+ -6.633031845092773,
+ -4.710548400878906,
+ -3.9331347942352295,
+ 2.122498035430908,
+ -3.4610869884490967,
+ 0.37038806080818176,
+ -1.1668025255203247,
+ -0.023927858099341393,
+ -0.0039329687133431435,
+ 0.1696121245622635,
+ -1.9923278093338013,
+ -1.283496618270874,
+ -0.2518565058708191,
+ -0.18193419277668,
+ -2.407423496246338
+ ],
+ [
+ -6.7707037925720215,
+ -0.02049800381064415,
+ -2.9092395305633545,
+ 0.10438992828130722,
+ 0.33034420013427734,
+ -2.6344707012176514,
+ 0.7101739645004272,
+ 1.5019316673278809,
+ -0.5193276405334473,
+ 3.2251436710357666,
+ 0.926478922367096,
+ 1.3138591051101685,
+ -1.1560633182525635,
+ -0.7247028350830078,
+ 1.277480959892273
+ ],
+ [
+ -6.346324920654297,
+ -3.0712578296661377,
+ -3.9589152336120605,
+ 1.3239221572875977,
+ -2.1389081478118896,
+ 0.720106840133667,
+ -0.07943042367696762,
+ 0.40552255511283875,
+ -0.16582681238651276,
+ -1.7012275457382202,
+ -0.2805171012878418,
+ -0.39177650213241577,
+ 0.07540342956781387,
+ 0.7645885944366455,
+ 0.23567278683185577
+ ],
+ [
+ 13.804797172546387,
+ 0.3623347580432892,
+ 0.46241530776023865,
+ 0.5661116242408752,
+ 0.13731516897678375,
+ 1.928476095199585,
+ 5.276743412017822,
+ -0.015138063579797745,
+ -0.3951400816440582,
+ -0.6959807276725769,
+ 1.8166042566299438,
+ -0.49044322967529297,
+ -0.09260746836662292,
+ -0.5163337588310242,
+ 0.25299549102783203
+ ],
+ [
+ -6.5857954025268555,
+ 11.538633346557617,
+ 6.916597366333008,
+ -0.034145940095186234,
+ -3.0373787879943848,
+ 1.5458464622497559,
+ -1.482081413269043,
+ 0.8587384223937988,
+ -0.6591081619262695,
+ 0.9805246591567993,
+ -1.2945560216903687,
+ -2.1394832134246826,
+ 0.3758484721183777,
+ -2.373839855194092,
+ 0.21602465212345123
+ ],
+ [
+ 10.631915092468262,
+ 2.9713528156280518,
+ -1.9708739519119263,
+ -0.7804402112960815,
+ 2.3933732509613037,
+ -2.582388401031494,
+ -0.75424724817276,
+ 1.3414348363876343,
+ -0.2752961814403534,
+ 0.890378475189209,
+ -1.7859374284744263,
+ 0.1485324651002884,
+ 0.06210426613688469,
+ 0.34111982583999634,
+ -0.321896493434906
+ ],
+ [
+ -7.065332412719727,
+ -5.17409086227417,
+ -5.174910545349121,
+ -0.4979696571826935,
+ -3.9807562828063965,
+ -1.071084976196289,
+ 0.6349092721939087,
+ 0.44708988070487976,
+ 0.08550722897052765,
+ -1.603600263595581,
+ -0.6004006862640381,
+ -1.8206074237823486,
+ -0.5566458702087402,
+ -0.410028338432312,
+ -0.24181970953941345
+ ],
+ [
+ -5.072853088378906,
+ 11.885273933410645,
+ 7.13008975982666,
+ -0.9047028422355652,
+ -3.7634646892547607,
+ 0.5458245277404785,
+ 0.27958813309669495,
+ 0.19300906360149384,
+ 0.1546933799982071,
+ 0.8597409129142761,
+ -0.10468994826078415,
+ -0.9900985956192017,
+ -0.0802856832742691,
+ -0.2954711318016052,
+ -0.39123740792274475
+ ],
+ [
+ -2.2959165573120117,
+ -13.591452598571777,
+ 15.500890731811523,
+ -0.013831814751029015,
+ 0.05452581122517586,
+ 1.8694772720336914,
+ 0.6506286859512329,
+ -0.009574650786817074,
+ -0.46518468856811523,
+ 1.3939846754074097,
+ 0.8806840181350708,
+ -1.322688102722168,
+ 0.9747006893157959,
+ 1.819898247718811,
+ 1.959892749786377
+ ],
+ [
+ -6.935782432556152,
+ -3.541311740875244,
+ -4.456383228302002,
+ -0.07430724054574966,
+ -3.4466419219970703,
+ 0.1676584780216217,
+ 0.2099786400794983,
+ 0.10795062780380249,
+ -0.5206305384635925,
+ 1.541839838027954,
+ 0.14256364107131958,
+ -0.015470422804355621,
+ 0.741610586643219,
+ 2.578979730606079,
+ -1.4457194805145264
+ ],
+ [
+ 17.404340744018555,
+ 0.39841020107269287,
+ -0.9337308406829834,
+ -0.39530205726623535,
+ -0.5581492781639099,
+ -0.610517144203186,
+ -0.04846830293536186,
+ -0.17271195352077484,
+ -0.45810505747795105,
+ 0.13301117718219757,
+ 0.5532274842262268,
+ 0.15852680802345276,
+ -0.2785075008869171,
+ 0.23434460163116455,
+ -0.765234649181366
+ ],
+ [
+ -5.915950775146484,
+ 1.5354317426681519,
+ -2.333698034286499,
+ -0.3393474817276001,
+ 7.194741249084473,
+ -2.119023323059082,
+ 1.024801254272461,
+ 0.6118509769439697,
+ 0.12715208530426025,
+ -2.252159833908081,
+ 0.2316802740097046,
+ -1.7755224704742432,
+ -0.011273633688688278,
+ -1.349552869796753,
+ -0.228363499045372
+ ],
+ [
+ 16.091808319091797,
+ 0.13688872754573822,
+ -1.3607317209243774,
+ -0.6114463806152344,
+ -0.41346484422683716,
+ -3.682649850845337,
+ -3.5649712085723877,
+ 0.26666638255119324,
+ -1.0104358196258545,
+ -0.47902339696884155,
+ -2.6395249366760254,
+ 1.5015807151794434,
+ 1.0135904550552368,
+ 1.0721237659454346,
+ -1.230551838874817
+ ],
+ [
+ 16.59830665588379,
+ 0.4300448000431061,
+ -1.2387993335723877,
+ -0.24021530151367188,
+ 0.023953618481755257,
+ 0.6204151511192322,
+ 1.1967992782592773,
+ 0.3118155598640442,
+ -0.5549460053443909,
+ 0.11880111694335938,
+ 0.37932923436164856,
+ 0.27386167645454407,
+ -0.5981679558753967,
+ 0.3505535423755646,
+ 1.487138032913208
+ ],
+ [
+ -1.3084070682525635,
+ -12.297880172729492,
+ 11.09471321105957,
+ 0.305716872215271,
+ 2.1178572177886963,
+ 4.145359039306641,
+ -1.8143569231033325,
+ 10.424415588378906,
+ -1.388187289237976,
+ -1.1978607177734375,
+ -0.8731321692466736,
+ -0.46229320764541626,
+ 0.814496636390686,
+ -1.7034133672714233,
+ -0.9586772918701172
+ ],
+ [
+ -6.430109977722168,
+ -3.271021842956543,
+ -4.572028636932373,
+ 0.3631417155265808,
+ -0.8798067569732666,
+ 0.8441181182861328,
+ -0.388458251953125,
+ -0.45486146211624146,
+ 0.7894196510314941,
+ 0.455885648727417,
+ 0.3334960341453552,
+ 1.3697117567062378,
+ 1.6643719673156738,
+ -1.0224260091781616,
+ -1.403250813484192
+ ],
+ [
+ -6.49962043762207,
+ -0.45692217350006104,
+ -3.622687339782715,
+ 0.3394929766654968,
+ 6.06002950668335,
+ 2.0702478885650635,
+ -0.29025521874427795,
+ -1.262398362159729,
+ 0.38432830572128296,
+ -2.4310250282287598,
+ 0.48529404401779175,
+ 0.46411582827568054,
+ 0.520784318447113,
+ -0.008954841643571854,
+ 1.2721083164215088
+ ],
+ [
+ -5.68439245223999,
+ 11.167657852172852,
+ 5.8115763664245605,
+ -0.20993679761886597,
+ -3.1285252571105957,
+ 2.211216688156128,
+ -0.9384390711784363,
+ -0.588556170463562,
+ -0.41864484548568726,
+ -0.46031516790390015,
+ -0.1168997511267662,
+ -1.1101247072219849,
+ -1.069313406944275,
+ -1.3167625665664673,
+ -1.8780213594436646
+ ],
+ [
+ -7.479487419128418,
+ -4.613443851470947,
+ -5.166690826416016,
+ -0.4069637954235077,
+ -3.2189550399780273,
+ 0.044903241097927094,
+ 0.21926328539848328,
+ 0.31391432881355286,
+ 0.017145520076155663,
+ -1.4062927961349487,
+ 0.05131641775369644,
+ -0.2786791920661926,
+ -0.5209872126579285,
+ -1.5775574445724487,
+ 0.8041971325874329
+ ],
+ [
+ -6.66842794418335,
+ 6.531101703643799,
+ 2.1764655113220215,
+ 0.5428754687309265,
+ -0.3222517967224121,
+ -0.10063676536083221,
+ 0.8725026249885559,
+ 0.150613933801651,
+ -0.6010708212852478,
+ 6.20190954208374,
+ 0.8109442591667175,
+ 1.0208653211593628,
+ 1.5127489566802979,
+ 1.8117544651031494,
+ -0.8865102529525757
+ ],
+ [
+ -2.478512763977051,
+ -16.180892944335938,
+ 18.044273376464844,
+ -4.064462184906006,
+ -0.46531668305397034,
+ -0.1454036682844162,
+ -0.09138303250074387,
+ 1.965491533279419,
+ -0.4185629189014435,
+ -0.46352165937423706,
+ 0.23098570108413696,
+ -0.27447715401649475,
+ -0.5247368812561035,
+ -0.9289137125015259,
+ -1.750327467918396
+ ],
+ [
+ -5.447409629821777,
+ -2.076162099838257,
+ -4.344316482543945,
+ 1.5971858501434326,
+ 1.1051886081695557,
+ 3.5759482383728027,
+ -0.8039531111717224,
+ -1.5088238716125488,
+ 0.0441010519862175,
+ -1.324963092803955,
+ 1.550406575202942,
+ 0.5344802737236023,
+ -0.782378613948822,
+ 1.1846884489059448,
+ 2.528050422668457
+ ],
+ [
+ -5.630663871765137,
+ 12.533702850341797,
+ 6.23322868347168,
+ 0.04751843214035034,
+ -3.0239779949188232,
+ -3.031585693359375,
+ 1.5364348888397217,
+ 1.4549788236618042,
+ 0.06350533664226532,
+ -4.895161151885986,
+ 0.8156552910804749,
+ 2.812267303466797,
+ -0.7577130794525146,
+ 2.7860100269317627,
+ -0.8287935853004456
+ ],
+ [
+ -7.791927814483643,
+ -4.8703508377075195,
+ -3.679556131362915,
+ -0.48408517241477966,
+ -3.3475894927978516,
+ -1.583886742591858,
+ 0.5284850597381592,
+ -0.8230766654014587,
+ 0.022779617458581924,
+ -1.1726802587509155,
+ -1.8748058080673218,
+ -1.252226710319519,
+ -0.9364314675331116,
+ -1.246066689491272,
+ -2.6306052207946777
+ ],
+ [
+ -2.291104555130005,
+ -13.061836242675781,
+ 14.402167320251465,
+ -0.1429309993982315,
+ 2.8811264038085938,
+ 7.199743270874023,
+ -2.38653564453125,
+ 12.696192741394043,
+ -0.924425482749939,
+ -0.3050547242164612,
+ 1.3858680725097656,
+ -1.7162210941314697,
+ 0.2283399999141693,
+ -0.5868731141090393,
+ -0.2882992923259735
+ ],
+ [
+ -5.9022297859191895,
+ 12.92559814453125,
+ 6.55058479309082,
+ 0.3339094817638397,
+ -3.2592599391937256,
+ 1.4221041202545166,
+ -0.8242647647857666,
+ -0.3591609597206116,
+ -0.3012116551399231,
+ 0.01829279214143753,
+ 0.5673591494560242,
+ -1.4940037727355957,
+ 0.6341957449913025,
+ 0.38729721307754517,
+ 0.7728603482246399
+ ],
+ [
+ -3.9878621101379395,
+ -0.5290510654449463,
+ -3.044506788253784,
+ 0.21571201086044312,
+ 4.827415466308594,
+ 8.016499519348145,
+ -3.523212432861328,
+ -4.653190612792969,
+ 0.8904099464416504,
+ 1.2144001722335815,
+ -0.7137830257415771,
+ 4.021688461303711,
+ 0.4177875220775604,
+ -0.37794822454452515,
+ -4.94310188293457
+ ],
+ [
+ -6.1021294593811035,
+ 2.7890055179595947,
+ -1.6568065881729126,
+ -0.03851001337170601,
+ 8.133480072021484,
+ -3.228820562362671,
+ 1.0343797206878662,
+ 1.4934390783309937,
+ -0.014339433051645756,
+ -1.1809924840927124,
+ -0.2421381026506424,
+ -0.4704233407974243,
+ 0.6633786559104919,
+ 2.3128154277801514,
+ -0.607623815536499
+ ],
+ [
+ -6.945530891418457,
+ -4.052828311920166,
+ -4.517100811004639,
+ 0.08029808104038239,
+ -4.831117630004883,
+ -2.162205934524536,
+ -0.24815519154071808,
+ 1.514981985092163,
+ -0.07990355789661407,
+ 0.7973329424858093,
+ 0.3900177478790283,
+ -0.34134891629219055,
+ -0.9423345327377319,
+ -1.145782232284546,
+ -1.330998420715332
+ ],
+ [
+ -7.017252445220947,
+ -3.718937635421753,
+ -4.669196605682373,
+ 0.32154178619384766,
+ -2.2396390438079834,
+ -1.9931856393814087,
+ 0.5184475183486938,
+ 1.5591081380844116,
+ -0.3359176218509674,
+ 1.449782133102417,
+ 0.7808154821395874,
+ 0.73558509349823,
+ -1.571465015411377,
+ -2.7460081577301025,
+ 3.4523682594299316
+ ],
+ [
+ 12.70518684387207,
+ 1.105376958847046,
+ 0.46443885564804077,
+ 0.006108684930950403,
+ -2.178523063659668,
+ 7.4828925132751465,
+ 15.545074462890625,
+ -0.5486260056495667,
+ 3.5271387100219727,
+ 1.0451549291610718,
+ -7.873144626617432,
+ 0.8020601868629456,
+ 0.763676643371582,
+ -1.0674582719802856,
+ -1.2201184034347534
+ ],
+ [
+ -5.700778961181641,
+ 12.155426025390625,
+ 6.647866725921631,
+ -0.6507703065872192,
+ -2.2717182636260986,
+ 1.2642405033111572,
+ -0.9206889867782593,
+ -0.8717304468154907,
+ -0.6469022631645203,
+ -0.9301763772964478,
+ -1.1721569299697876,
+ -2.0312952995300293,
+ 0.8208807706832886,
+ -0.40883055329322815,
+ -0.17387880384922028
+ ],
+ [
+ -5.086932182312012,
+ -2.2743642330169678,
+ -3.7345104217529297,
+ 0.816969096660614,
+ 1.8481322526931763,
+ 3.6211490631103516,
+ -1.938193917274475,
+ -2.4763588905334473,
+ -0.1535286009311676,
+ -1.519432783126831,
+ 0.12146995216608047,
+ 0.5037128329277039,
+ 0.516011655330658,
+ -0.5917330384254456,
+ 0.5921635627746582
+ ],
+ [
+ -7.316657066345215,
+ 2.562756061553955,
+ -1.1275848150253296,
+ -0.1805962175130844,
+ 3.186422824859619,
+ 0.1768350452184677,
+ 0.01161456760019064,
+ 0.007710997015237808,
+ -0.34776148200035095,
+ 5.967932224273682,
+ 1.2801789045333862,
+ 2.245292901992798,
+ -2.2617173194885254,
+ -1.628810167312622,
+ -1.082520842552185
+ ],
+ [
+ 15.696523666381836,
+ 0.7627936005592346,
+ 0.15198983252048492,
+ -0.9573447704315186,
+ -0.2635559141635895,
+ -0.8562425374984741,
+ -2.3888051509857178,
+ 1.0123579502105713,
+ -0.9281941056251526,
+ 0.336731493473053,
+ -3.3900439739227295,
+ 1.3121342658996582,
+ 0.09629403054714203,
+ 0.27128398418426514,
+ -0.6069579720497131
+ ],
+ [
+ -6.727656364440918,
+ 4.048370361328125,
+ 0.20942409336566925,
+ -0.2926693856716156,
+ 6.385499954223633,
+ -5.693485736846924,
+ 2.538562297821045,
+ 1.8860918283462524,
+ -0.4288187026977539,
+ 0.6820266842842102,
+ -0.3381219208240509,
+ -1.0059707164764404,
+ 0.9352676868438721,
+ 0.33742737770080566,
+ -2.545661687850952
+ ],
+ [
+ -5.980621337890625,
+ -3.997999429702759,
+ -4.98711633682251,
+ 0.24229000508785248,
+ -3.303849458694458,
+ 0.1655520498752594,
+ 0.3048363924026489,
+ 0.0592426173388958,
+ -0.23530490696430206,
+ -0.5366261005401611,
+ -0.2757284343242645,
+ -0.8656066060066223,
+ 0.3633071184158325,
+ -0.45731356739997864,
+ -1.215511679649353
+ ],
+ [
+ -7.438937664031982,
+ -4.56027889251709,
+ -4.888825416564941,
+ 0.49093884229660034,
+ -2.644442081451416,
+ -2.131178617477417,
+ 2.109267234802246,
+ 1.000357747077942,
+ -0.12623676657676697,
+ 1.638617753982544,
+ 0.7005933523178101,
+ 0.09732241928577423,
+ -0.8304453492164612,
+ -0.9294398427009583,
+ 1.7243679761886597
+ ],
+ [
+ -6.0656609535217285,
+ -2.270077705383301,
+ -4.085136413574219,
+ -0.10518162697553635,
+ -0.09069894254207611,
+ 0.06278130412101746,
+ -0.11426981538534164,
+ -0.2945103645324707,
+ -0.12845854461193085,
+ 2.43989634513855,
+ -0.03523989021778107,
+ 1.5752842426300049,
+ -1.627826452255249,
+ -0.10466018319129944,
+ -1.4241315126419067
+ ],
+ [
+ -5.765082359313965,
+ 11.487556457519531,
+ 5.7546892166137695,
+ 0.016675537452101707,
+ -3.688321590423584,
+ 1.6755340099334717,
+ -0.8479875922203064,
+ -1.121994137763977,
+ -0.3696088492870331,
+ 0.12996453046798706,
+ 0.0355529822409153,
+ -2.0037403106689453,
+ 0.6693429946899414,
+ 0.44086068868637085,
+ 0.5783438086509705
+ ],
+ [
+ -6.089733600616455,
+ 13.87205982208252,
+ 7.262159824371338,
+ -0.35948702692985535,
+ -2.9167728424072266,
+ 1.8577951192855835,
+ -0.47354331612586975,
+ -0.8900036811828613,
+ -0.2620699405670166,
+ -1.1244406700134277,
+ -0.7470769882202148,
+ -0.7905363440513611,
+ -0.5980591773986816,
+ -0.2446945160627365,
+ 0.9068487286567688
+ ],
+ [
+ 13.732917785644531,
+ -0.43437838554382324,
+ -0.6249669790267944,
+ 1.7151765823364258,
+ 0.4943024218082428,
+ 2.058339834213257,
+ 2.6219570636749268,
+ 0.03283187001943588,
+ -0.9114151000976562,
+ -0.8340620994567871,
+ 8.516166687011719,
+ -2.498161792755127,
+ -1.3327449560165405,
+ -1.0513575077056885,
+ -3.2517800331115723
+ ],
+ [
+ -6.339433193206787,
+ -2.1785330772399902,
+ -3.5437073707580566,
+ -0.7653146386146545,
+ -0.15360388159751892,
+ 1.416759729385376,
+ -0.48551493883132935,
+ -0.9400314688682556,
+ 0.06254110485315323,
+ 0.5760915875434875,
+ -0.7095029950141907,
+ 1.7436562776565552,
+ -1.1189439296722412,
+ -0.3678286373615265,
+ -2.2614386081695557
+ ],
+ [
+ -6.797684192657471,
+ 13.798863410949707,
+ 6.294400691986084,
+ -0.5679174661636353,
+ -1.8912429809570312,
+ -1.3460578918457031,
+ 0.19070222973823547,
+ 0.3973613977432251,
+ 1.3991913795471191,
+ -2.1795384883880615,
+ 0.9994227290153503,
+ 2.6301205158233643,
+ -1.572855830192566,
+ 2.691922903060913,
+ 0.9809936285018921
+ ],
+ [
+ -5.757458686828613,
+ -4.220179557800293,
+ -4.859086036682129,
+ 0.5714593529701233,
+ -4.403603553771973,
+ -0.3711105287075043,
+ 0.6259642243385315,
+ 0.29165977239608765,
+ -0.20353034138679504,
+ 0.4925423264503479,
+ 1.030312418937683,
+ -0.40660056471824646,
+ 2.4093384742736816,
+ 2.681809902191162,
+ -0.5850101709365845
+ ],
+ [
+ -5.104196071624756,
+ 10.743734359741211,
+ 4.897531509399414,
+ -0.5247437357902527,
+ -1.4643065929412842,
+ 0.5741134881973267,
+ 1.377345085144043,
+ -0.14909154176712036,
+ -0.558527946472168,
+ 1.5443559885025024,
+ -0.5456503629684448,
+ -0.8983170390129089,
+ 0.12367919087409973,
+ 0.23191863298416138,
+ -0.3351898491382599
+ ],
+ [
+ -6.188399314880371,
+ 6.326329708099365,
+ 0.773901641368866,
+ -0.23110637068748474,
+ 2.4968466758728027,
+ 0.42012202739715576,
+ -0.44387325644493103,
+ -0.9131253361701965,
+ -0.3476557433605194,
+ 3.492542266845703,
+ 0.7810912728309631,
+ 2.046994924545288,
+ -0.5956247448921204,
+ 1.4118949174880981,
+ -0.08625523000955582
+ ],
+ [
+ -7.620984077453613,
+ 2.0888242721557617,
+ -2.0531115531921387,
+ 0.07210119813680649,
+ -0.0033177295699715614,
+ 1.517013430595398,
+ -0.07913161814212799,
+ -0.11665050685405731,
+ -0.13873159885406494,
+ 2.2769079208374023,
+ 0.5773445963859558,
+ 1.2186094522476196,
+ -0.7978529930114746,
+ 1.275191068649292,
+ -0.08977816253900528
+ ],
+ [
+ -6.255349636077881,
+ 2.121272325515747,
+ -1.878645658493042,
+ -0.024361448362469673,
+ 7.282958507537842,
+ -2.1559531688690186,
+ 2.1553571224212646,
+ 0.13814009726047516,
+ 0.07358822226524353,
+ -0.8670037984848022,
+ 1.9388540983200073,
+ 5.0528998374938965,
+ -0.566911518573761,
+ 3.7543795108795166,
+ -0.042845357209444046
+ ],
+ [
+ -6.026260852813721,
+ 12.034859657287598,
+ 6.135149002075195,
+ 0.8699914216995239,
+ -3.436173915863037,
+ -0.1928558349609375,
+ -1.1331976652145386,
+ -0.2982964515686035,
+ -0.38778194785118103,
+ 0.36411207914352417,
+ -0.5022473931312561,
+ -1.8706458806991577,
+ -0.599449634552002,
+ -0.3201214671134949,
+ 0.25424981117248535
+ ],
+ [
+ -5.602170944213867,
+ -4.292830467224121,
+ -4.831594467163086,
+ 0.1802518516778946,
+ -2.9897522926330566,
+ 0.8406361937522888,
+ 1.7832081317901611,
+ 1.5639214515686035,
+ -0.026532089337706566,
+ -1.3539882898330688,
+ -1.6044265031814575,
+ -0.28894251585006714,
+ 0.6404204964637756,
+ 0.44369497895240784,
+ 0.38703271746635437
+ ],
+ [
+ -5.434932708740234,
+ 11.52717113494873,
+ 5.446648120880127,
+ -0.22112129628658295,
+ -2.0511932373046875,
+ 0.815772533416748,
+ -1.2517374753952026,
+ -0.7927738428115845,
+ -0.4825977683067322,
+ 0.7583417296409607,
+ -0.7095592617988586,
+ -2.0682222843170166,
+ 1.1820802688598633,
+ 0.13331486284732819,
+ 1.348726749420166
+ ],
+ [
+ -6.666961669921875,
+ 1.872323989868164,
+ -1.3750282526016235,
+ -0.6270864605903625,
+ 1.3675918579101562,
+ -3.1123876571655273,
+ 1.4452173709869385,
+ 1.2984651327133179,
+ 0.11844184994697571,
+ 2.931164503097534,
+ 1.6566526889801025,
+ 4.900057792663574,
+ -2.322136878967285,
+ -0.19765561819076538,
+ 1.3267816305160522
+ ],
+ [
+ -4.966930866241455,
+ 0.0881548821926117,
+ 15.919828414916992,
+ -1.6008784770965576,
+ -2.661015033721924,
+ -1.4928714036941528,
+ 0.5743482708930969,
+ -2.994516134262085,
+ -0.38667774200439453,
+ -0.2896711528301239,
+ 2.5278561115264893,
+ -1.7012395858764648,
+ 0.46226099133491516,
+ -1.0481940507888794,
+ 1.6261000633239746
+ ],
+ [
+ -5.190653324127197,
+ 3.900385618209839,
+ -0.4485127627849579,
+ -0.2626904249191284,
+ 8.368755340576172,
+ -4.749380111694336,
+ 2.630518913269043,
+ 0.7549716234207153,
+ -0.03948593512177467,
+ -0.19504866003990173,
+ -0.9936651587486267,
+ -0.6250144839286804,
+ 2.086578607559204,
+ 1.1859568357467651,
+ -2.295984983444214
+ ],
+ [
+ -6.789980888366699,
+ -3.412338972091675,
+ -4.595227241516113,
+ -0.2965986132621765,
+ -2.994529962539673,
+ -0.6274967789649963,
+ 0.01063049491494894,
+ -0.2900548279285431,
+ -0.17455337941646576,
+ -0.7836854457855225,
+ -0.680665135383606,
+ -0.29079124331474304,
+ -0.6397696733474731,
+ 0.5734148621559143,
+ -1.219370722770691
+ ],
+ [
+ 12.458173751831055,
+ 0.9928799867630005,
+ 0.5661988854408264,
+ 1.0641676187515259,
+ -2.238077163696289,
+ 8.026447296142578,
+ 19.298439025878906,
+ -1.1008973121643066,
+ 1.8517982959747314,
+ 0.9945985078811646,
+ -8.217056274414062,
+ 1.5194100141525269,
+ 2.184067726135254,
+ 0.44887474179267883,
+ -1.475339651107788
+ ],
+ [
+ -6.150762557983398,
+ 7.645421981811523,
+ 1.968967318534851,
+ -0.6401703357696533,
+ 1.1364065408706665,
+ 0.6626666188240051,
+ -0.5434557795524597,
+ -0.5008590221405029,
+ -0.7167225480079651,
+ 5.318227767944336,
+ 0.3315293788909912,
+ 0.4773620367050171,
+ 0.20791013538837433,
+ -0.5136129856109619,
+ -1.01117742061615
+ ],
+ [
+ 14.14163875579834,
+ 1.0966260433197021,
+ -0.21283715963363647,
+ -0.0082848547026515,
+ -1.187068223953247,
+ 2.8856191635131836,
+ 7.175673484802246,
+ -0.10285797715187073,
+ -0.157830610871315,
+ 0.28306642174720764,
+ -1.10685133934021,
+ -0.568964421749115,
+ 0.034882619976997375,
+ -0.07423487305641174,
+ -0.6391218900680542
+ ],
+ [
+ -5.449928283691406,
+ -1.331894874572754,
+ -2.757129192352295,
+ -0.6270777583122253,
+ 1.8425058126449585,
+ 6.538372039794922,
+ -2.9426589012145996,
+ -4.057281970977783,
+ 0.39316609501838684,
+ -0.5152219533920288,
+ -1.2901997566223145,
+ 3.809587240219116,
+ -2.658989429473877,
+ -3.818325996398926,
+ -4.446944236755371
+ ],
+ [
+ -2.3904988765716553,
+ -15.74937629699707,
+ 16.863431930541992,
+ -2.6391594409942627,
+ 1.504044532775879,
+ 3.956631660461426,
+ -1.3550394773483276,
+ 8.43813705444336,
+ -0.7382986545562744,
+ 0.016093885526061058,
+ -0.31090953946113586,
+ -0.8905919194221497,
+ 0.3063087463378906,
+ -2.6425814628601074,
+ -0.8235694169998169
+ ],
+ [
+ 16.23615837097168,
+ 0.6645206212997437,
+ -0.7617345452308655,
+ -0.2835473418235779,
+ 0.011683493852615356,
+ -0.7405539155006409,
+ -0.4511185586452484,
+ 0.3138611316680908,
+ -0.5949645638465881,
+ 0.057277996093034744,
+ 0.9081122279167175,
+ -0.9759302139282227,
+ 0.9061576724052429,
+ 0.4022362530231476,
+ -0.9708775281906128
+ ],
+ [
+ 18.343862533569336,
+ 0.07675272226333618,
+ -1.8176493644714355,
+ -0.2769220769405365,
+ -0.10537555813789368,
+ -1.0792196989059448,
+ -1.0152643918991089,
+ 0.4399084150791168,
+ -0.609952986240387,
+ -0.5864500999450684,
+ 1.162947654724121,
+ -0.828974187374115,
+ -0.6866984963417053,
+ -0.31936198472976685,
+ 0.43322238326072693
+ ],
+ [
+ -4.560738563537598,
+ 10.814248085021973,
+ 4.9330902099609375,
+ 0.334751695394516,
+ -3.142925262451172,
+ 1.080179214477539,
+ -1.3614506721496582,
+ -0.7903503179550171,
+ -0.1278296560049057,
+ 0.8188009858131409,
+ 0.10064570605754852,
+ -2.3810577392578125,
+ 0.6785773634910583,
+ -0.8411354422569275,
+ 0.3903875946998596
+ ],
+ [
+ -5.951234340667725,
+ 13.173152923583984,
+ 6.1014084815979,
+ -0.48816537857055664,
+ -2.7613472938537598,
+ 0.32044118642807007,
+ -0.907184898853302,
+ -0.18239310383796692,
+ 0.27147501707077026,
+ -2.5999114513397217,
+ -0.3297966718673706,
+ 0.1728261113166809,
+ -0.9232636094093323,
+ -1.6993590593338013,
+ 0.20625866949558258
+ ],
+ [
+ 15.57251262664795,
+ 0.08797601610422134,
+ -1.3900195360183716,
+ -0.20082174241542816,
+ -0.1162174716591835,
+ -1.7898186445236206,
+ -2.5040948390960693,
+ 0.14733608067035675,
+ 3.298449754714966,
+ -0.051637064665555954,
+ -0.7214210033416748,
+ -0.08210410177707672,
+ -0.11126982420682907,
+ 0.7695987224578857,
+ -0.32426077127456665
+ ],
+ [
+ -7.088982105255127,
+ -6.174103736877441,
+ -5.021151065826416,
+ -0.2828860282897949,
+ -3.9586546421051025,
+ -0.14257483184337616,
+ -0.21211481094360352,
+ 1.893849492073059,
+ -0.34052813053131104,
+ -3.182173013687134,
+ -1.1871626377105713,
+ -0.41524025797843933,
+ -1.5617260932922363,
+ -0.4104822278022766,
+ -1.0941965579986572
+ ],
+ [
+ 15.880690574645996,
+ 0.8026187419891357,
+ -0.5276344418525696,
+ -0.38032016158103943,
+ -0.18843767046928406,
+ 0.8458476662635803,
+ 1.5060819387435913,
+ -0.20492680370807648,
+ -0.3685515522956848,
+ -0.36231252551078796,
+ 1.9305822849273682,
+ -0.9753183722496033,
+ -0.6145795583724976,
+ 0.23460577428340912,
+ -0.22291429340839386
+ ],
+ [
+ -6.361682891845703,
+ 3.219177007675171,
+ -2.0330557823181152,
+ -0.4735466241836548,
+ 7.754646301269531,
+ -4.380991458892822,
+ 1.412617802619934,
+ 0.8815485835075378,
+ 0.5002526044845581,
+ -2.300358772277832,
+ -0.14812257885932922,
+ 0.8741014003753662,
+ 0.7017986178398132,
+ 0.507710337638855,
+ 0.4232455790042877
+ ],
+ [
+ -6.066108703613281,
+ 7.160762786865234,
+ 1.161568284034729,
+ -0.24863214790821075,
+ -4.235307693481445,
+ 0.18797945976257324,
+ -0.09072237461805344,
+ 0.6166948080062866,
+ -0.5953987240791321,
+ -0.22188417613506317,
+ 1.2365245819091797,
+ -0.15427552163600922,
+ -0.2902108430862427,
+ 0.6293447613716125,
+ 0.9842432141304016
+ ],
+ [
+ -7.3408050537109375,
+ -4.797475814819336,
+ -4.794350624084473,
+ 0.14486566185951233,
+ -4.824836254119873,
+ -0.5007281303405762,
+ 0.2443666309118271,
+ -0.41302046179771423,
+ 0.2990414500236511,
+ -0.35543590784072876,
+ -0.6826984882354736,
+ 2.1069772243499756,
+ 3.9895408153533936,
+ 2.135657548904419,
+ -5.87898063659668
+ ],
+ [
+ -4.875540256500244,
+ 10.718949317932129,
+ 5.983212947845459,
+ -0.11701803654432297,
+ -4.9712958335876465,
+ 0.0836150199174881,
+ 0.8680984973907471,
+ -0.4831905961036682,
+ -0.4637221693992615,
+ -2.8177003860473633,
+ 0.24882400035858154,
+ 1.1051568984985352,
+ -1.5586003065109253,
+ 0.2694137692451477,
+ -0.09742088615894318
+ ],
+ [
+ -5.851585388183594,
+ 2.582256555557251,
+ 0.22286534309387207,
+ -0.8527863025665283,
+ 1.7842249870300293,
+ -4.656991958618164,
+ 2.0828399658203125,
+ 2.0207765102386475,
+ 0.09121420234441757,
+ 2.7791500091552734,
+ 1.8178304433822632,
+ 2.1813371181488037,
+ 0.15667535364627838,
+ 2.6607015132904053,
+ -0.0993545725941658
+ ],
+ [
+ -2.6678831577301025,
+ -14.65042495727539,
+ 15.662988662719727,
+ -3.6235530376434326,
+ -0.17167150974273682,
+ -3.9498746395111084,
+ 0.8044977188110352,
+ -7.719770908355713,
+ -0.020934388041496277,
+ -0.19814150035381317,
+ 0.8770599961280823,
+ 0.18226462602615356,
+ -1.7058148384094238,
+ 0.23512688279151917,
+ -0.4161730706691742
+ ],
+ [
+ -5.162461280822754,
+ 1.3584612607955933,
+ -1.6012245416641235,
+ 0.8597280383110046,
+ 6.674613952636719,
+ 0.4208642840385437,
+ 0.19211256504058838,
+ -0.6879403591156006,
+ 0.1113719791173935,
+ -1.5325052738189697,
+ -0.64009028673172,
+ -0.8738364577293396,
+ 0.7858895659446716,
+ 0.8170205950737,
+ 1.3355096578598022
+ ],
+ [
+ 12.179642677307129,
+ -0.12274681031703949,
+ -2.3750791549682617,
+ 0.055637139827013016,
+ -0.404527872800827,
+ -0.8479369878768921,
+ -0.20463591814041138,
+ -0.2346992939710617,
+ -0.4325162470340729,
+ -0.15284684300422668,
+ 1.7864497900009155,
+ -0.8519548773765564,
+ 0.0384695939719677,
+ 0.37783512473106384,
+ 0.5891197323799133
+ ],
+ [
+ -0.9325080513954163,
+ -12.961133003234863,
+ 15.693190574645996,
+ 5.172044277191162,
+ 3.5036284923553467,
+ 5.647056579589844,
+ -1.5056716203689575,
+ 9.482507705688477,
+ -1.5466662645339966,
+ -1.4268571138381958,
+ 0.629973292350769,
+ -0.1934894174337387,
+ -0.8044611215591431,
+ -1.5135233402252197,
+ 0.33773142099380493
+ ],
+ [
+ -6.1214518547058105,
+ -3.3240959644317627,
+ -5.008976936340332,
+ 0.06606898456811905,
+ -3.9215962886810303,
+ 1.449435830116272,
+ -0.7985987067222595,
+ -0.8267103433609009,
+ 0.06930509209632874,
+ 0.8817352056503296,
+ 0.3161417245864868,
+ 2.2780449390411377,
+ 4.335684776306152,
+ 3.551471471786499,
+ -5.759546279907227
+ ],
+ [
+ -6.9705047607421875,
+ -5.136174201965332,
+ -4.513982772827148,
+ -0.3649926483631134,
+ -1.4481322765350342,
+ 0.010722089558839798,
+ -1.267285943031311,
+ -0.051102932542562485,
+ 0.5598697066307068,
+ -1.9063827991485596,
+ -1.4770725965499878,
+ -1.9059206247329712,
+ 0.44697874784469604,
+ -2.7497174739837646,
+ 0.5507687926292419
+ ],
+ [
+ -6.280230522155762,
+ -4.762950420379639,
+ -4.389538288116455,
+ -0.4687071144580841,
+ -3.7348577976226807,
+ -0.6476107835769653,
+ -0.479973167181015,
+ 1.0686129331588745,
+ -0.3489128351211548,
+ -1.4642921686172485,
+ -0.1880936622619629,
+ -0.22345216572284698,
+ 0.41826313734054565,
+ -0.04909781366586685,
+ -3.0921638011932373
+ ],
+ [
+ -6.835711479187012,
+ -2.7260327339172363,
+ -3.3980841636657715,
+ -0.06251862645149231,
+ -1.7048448324203491,
+ 1.0574171543121338,
+ 0.28930777311325073,
+ -0.6351366639137268,
+ -0.45026668906211853,
+ 0.30037224292755127,
+ 0.5967055559158325,
+ -0.05326603725552559,
+ 0.6062237620353699,
+ 2.7284984588623047,
+ -1.5833712816238403
+ ],
+ [
+ 17.13951301574707,
+ 0.2902541756629944,
+ -1.1870598793029785,
+ -0.06530554592609406,
+ 0.026257459074258804,
+ -0.31616538763046265,
+ -0.9351834654808044,
+ -0.6088644862174988,
+ -0.03547847270965576,
+ -0.007532288786023855,
+ -0.3075263500213623,
+ -0.009201611392199993,
+ 0.211130753159523,
+ 0.7352514863014221,
+ 1.325519323348999
+ ],
+ [
+ 15.187187194824219,
+ 0.757494330406189,
+ 0.09178885817527771,
+ -0.07737278193235397,
+ -1.2665880918502808,
+ 0.2526005804538727,
+ 0.10819513350725174,
+ 0.6413975954055786,
+ -0.5436862111091614,
+ -0.734383225440979,
+ 1.3499523401260376,
+ 0.845566987991333,
+ -0.6593477725982666,
+ 0.40862157940864563,
+ 0.796596109867096
+ ],
+ [
+ -7.41516637802124,
+ -4.527830123901367,
+ -5.472275257110596,
+ -0.14972688257694244,
+ -3.844280242919922,
+ -1.7551053762435913,
+ 0.7586268782615662,
+ 0.8673347234725952,
+ -0.07116465270519257,
+ 0.05273078382015228,
+ -0.4985381066799164,
+ -1.3311123847961426,
+ 0.10669252276420593,
+ 1.3641259670257568,
+ 0.20912937819957733
+ ],
+ [
+ -6.0446062088012695,
+ 4.762195110321045,
+ 1.5721702575683594,
+ -0.448029488325119,
+ 0.2129586935043335,
+ -2.9641735553741455,
+ 1.11492919921875,
+ 0.56430584192276,
+ 0.9394159913063049,
+ -0.9345411062240601,
+ 3.2149155139923096,
+ 11.607451438903809,
+ 0.12991297245025635,
+ 0.3247561752796173,
+ 1.7887933254241943
+ ],
+ [
+ -5.883874893188477,
+ 4.9470391273498535,
+ 1.8837604522705078,
+ -0.2320719212293625,
+ 1.675734281539917,
+ -0.518809974193573,
+ 0.5250863432884216,
+ 0.2572382092475891,
+ -0.8255153298377991,
+ 8.803028106689453,
+ 1.5651146173477173,
+ 0.9968246221542358,
+ -0.3760044276714325,
+ -1.1907284259796143,
+ -1.3961575031280518
+ ],
+ [
+ -6.034341335296631,
+ -4.305602073669434,
+ -5.190322399139404,
+ -0.13480496406555176,
+ -2.8267557621002197,
+ -0.80681973695755,
+ 0.9597901105880737,
+ 1.1068496704101562,
+ -0.14398778975009918,
+ -1.280863642692566,
+ -0.15198944509029388,
+ -0.9252493381500244,
+ -1.279813528060913,
+ -1.4752174615859985,
+ 1.599778175354004
+ ],
+ [
+ 15.730146408081055,
+ 0.5321155786514282,
+ -0.5668128728866577,
+ -0.12107297033071518,
+ -0.0819493904709816,
+ 0.4375878870487213,
+ 0.3130713105201721,
+ -0.3232412338256836,
+ -0.5597994327545166,
+ 0.6905936598777771,
+ 1.6605350971221924,
+ -1.1296932697296143,
+ 0.11435055732727051,
+ 0.6684394478797913,
+ 0.2037631869316101
+ ],
+ [
+ -5.2079176902771,
+ 13.315241813659668,
+ 6.467434406280518,
+ -0.7666963934898376,
+ -3.839055061340332,
+ -0.572628378868103,
+ -0.22336189448833466,
+ -0.03795431926846504,
+ 0.009081266820430756,
+ -1.8523200750350952,
+ 1.0594722032546997,
+ 0.5713392496109009,
+ 0.7977897524833679,
+ 2.1415436267852783,
+ 0.2139623612165451
+ ],
+ [
+ -6.073563098907471,
+ -3.846618890762329,
+ -5.260581970214844,
+ 0.20958884060382843,
+ -1.9875439405441284,
+ 0.4676087498664856,
+ -0.18513613939285278,
+ -0.025853663682937622,
+ -0.23123323917388916,
+ -0.8724439144134521,
+ 0.45256906747817993,
+ -1.0812145471572876,
+ 0.38371598720550537,
+ 1.2785496711730957,
+ -0.7292386889457703
+ ],
+ [
+ -6.395683765411377,
+ -2.2049005031585693,
+ -4.289514064788818,
+ -0.4084542393684387,
+ 1.5580873489379883,
+ 1.7633846998214722,
+ -1.461719274520874,
+ -1.5344667434692383,
+ -0.002609308809041977,
+ -2.989838123321533,
+ -1.0505965948104858,
+ 0.4598299264907837,
+ -2.038743734359741,
+ -3.134530544281006,
+ 0.4749079644680023
+ ],
+ [
+ -5.3982977867126465,
+ 13.590713500976562,
+ 6.474094867706299,
+ -0.4380223751068115,
+ -2.378664255142212,
+ 0.061584994196891785,
+ -0.22715353965759277,
+ 0.3393128216266632,
+ -0.7389751076698303,
+ -1.2102748155593872,
+ 0.5933757424354553,
+ 0.20830947160720825,
+ 0.18980886042118073,
+ 1.2637603282928467,
+ 0.21045511960983276
+ ],
+ [
+ -5.783041477203369,
+ -1.754094123840332,
+ -4.430751323699951,
+ -0.5563341379165649,
+ 1.8422584533691406,
+ -0.20826588571071625,
+ -0.20856934785842896,
+ 0.033841196447610855,
+ -0.035257354378700256,
+ 0.6816293001174927,
+ -0.04582776501774788,
+ 0.3999321460723877,
+ -3.3668909072875977,
+ -3.8692431449890137,
+ 3.078887462615967
+ ],
+ [
+ -6.482924461364746,
+ 3.2555813789367676,
+ -0.08492658287286758,
+ -0.30264830589294434,
+ 2.1925861835479736,
+ -1.6547521352767944,
+ 1.080346703529358,
+ 1.6622753143310547,
+ 0.6655005216598511,
+ 2.4442129135131836,
+ 2.5460164546966553,
+ 7.012863636016846,
+ -2.487018346786499,
+ 0.08551935106515884,
+ 0.42132312059402466
+ ],
+ [
+ -2.4216322898864746,
+ -14.592395782470703,
+ 16.943883895874023,
+ 0.34237536787986755,
+ 1.4241620302200317,
+ -1.83762788772583,
+ 1.7845478057861328,
+ -3.602642059326172,
+ -0.058047402650117874,
+ -0.0459340363740921,
+ 2.8129937648773193,
+ -1.1095263957977295,
+ 1.7641550302505493,
+ 0.9482302665710449,
+ -0.4517807960510254
+ ],
+ [
+ 17.44760513305664,
+ 0.18731439113616943,
+ -0.1252410113811493,
+ 0.09500889480113983,
+ -0.29461029171943665,
+ -0.23728272318840027,
+ 0.7350441813468933,
+ 0.25818103551864624,
+ -0.1076945960521698,
+ -0.15350200235843658,
+ -0.9126036167144775,
+ 0.1732773631811142,
+ 1.009062647819519,
+ 0.5453808307647705,
+ -0.26853278279304504
+ ],
+ [
+ -5.834136486053467,
+ 12.155327796936035,
+ 5.81230354309082,
+ 0.0527985580265522,
+ -3.1274564266204834,
+ 0.8186353445053101,
+ -0.5731974244117737,
+ -0.432409405708313,
+ -0.502465009689331,
+ -1.0649375915527344,
+ 0.9272364974021912,
+ 0.9894016981124878,
+ -0.5595642328262329,
+ 0.7520011067390442,
+ 0.29200854897499084
+ ],
+ [
+ -6.587196350097656,
+ -5.120001316070557,
+ -4.0292158126831055,
+ -0.9150570631027222,
+ -2.1185948848724365,
+ -1.055741310119629,
+ 0.13707555830478668,
+ -1.0340059995651245,
+ 0.7263143658638,
+ -2.374215602874756,
+ -1.2529559135437012,
+ -0.11139648407697678,
+ -2.6360795497894287,
+ -3.772892951965332,
+ -2.200453042984009
+ ],
+ [
+ -5.092447757720947,
+ 3.952857255935669,
+ 0.21226562559604645,
+ -0.03330252319574356,
+ 3.17830228805542,
+ 0.00100678950548172,
+ -1.0244731903076172,
+ -1.6944031715393066,
+ -0.03894034028053284,
+ 0.8403973579406738,
+ -0.34397393465042114,
+ 0.060561440885066986,
+ -0.9336060285568237,
+ -0.5222085118293762,
+ 0.005402968730777502
+ ],
+ [
+ -7.806102275848389,
+ -3.6733527183532715,
+ -5.526575088500977,
+ -0.036682840436697006,
+ -2.5519659519195557,
+ -2.055598020553589,
+ 0.33756858110427856,
+ 1.4821784496307373,
+ -0.11858178675174713,
+ 1.821948766708374,
+ 0.38264918327331543,
+ -0.6009384989738464,
+ -1.2855019569396973,
+ -0.761405885219574,
+ 2.493947744369507
+ ],
+ [
+ -5.609742641448975,
+ -2.7174999713897705,
+ -5.010997772216797,
+ -0.06320977956056595,
+ 0.5887398719787598,
+ 1.023790717124939,
+ -1.126497745513916,
+ -0.36586323380470276,
+ -0.12590500712394714,
+ -1.0618464946746826,
+ -0.1380961686372757,
+ 1.4946081638336182,
+ -1.0294137001037598,
+ 0.4324985146522522,
+ -0.7891609072685242
+ ],
+ [
+ -6.7961649894714355,
+ -3.1865170001983643,
+ -4.186081886291504,
+ -0.15249837934970856,
+ -4.014158725738525,
+ -0.6254463791847229,
+ 0.19289949536323547,
+ 0.23242773115634918,
+ -0.15024052560329437,
+ 0.0739053413271904,
+ 0.22420424222946167,
+ 0.3124925196170807,
+ -0.018837884068489075,
+ 2.4454588890075684,
+ -0.15498822927474976
+ ],
+ [
+ -6.620680809020996,
+ -3.3968517780303955,
+ -5.359402656555176,
+ 2.2100260257720947,
+ -1.2100486755371094,
+ -0.37816646695137024,
+ -0.5914230942726135,
+ 0.1684741973876953,
+ 0.13219667971134186,
+ -0.7456404566764832,
+ -0.24651676416397095,
+ 0.5916925668716431,
+ -1.1824599504470825,
+ -1.1586835384368896,
+ -0.5679850578308105
+ ],
+ [
+ -5.5756707191467285,
+ 3.30572509765625,
+ -0.28041431307792664,
+ 0.771645188331604,
+ 7.619564056396484,
+ -2.490936517715454,
+ 1.2512855529785156,
+ 0.8593441247940063,
+ 0.2591637372970581,
+ -1.508784532546997,
+ 0.025733204558491707,
+ -1.3403161764144897,
+ 0.5446876287460327,
+ 0.0027767582796514034,
+ -1.132686972618103
+ ],
+ [
+ -5.875317096710205,
+ 11.844175338745117,
+ 5.431635856628418,
+ -0.6728940606117249,
+ -1.2616236209869385,
+ 0.08941620588302612,
+ 1.6531895399093628,
+ 0.6917409896850586,
+ -0.0777929350733757,
+ -0.589515745639801,
+ -2.36767840385437,
+ 0.9150326251983643,
+ 0.9835535883903503,
+ -0.4933379888534546,
+ 0.36374741792678833
+ ],
+ [
+ -5.7548041343688965,
+ 12.401386260986328,
+ 6.432449817657471,
+ -0.15789858996868134,
+ -2.208817720413208,
+ -0.3638131618499756,
+ -0.45353230834007263,
+ -0.28809046745300293,
+ 0.21622787415981293,
+ 0.3671993911266327,
+ 0.3708276152610779,
+ 0.5621020197868347,
+ 1.4108452796936035,
+ 0.3120458126068115,
+ -1.5206315517425537
+ ],
+ [
+ 16.953887939453125,
+ 0.3568756580352783,
+ -1.5085558891296387,
+ -0.26158407330513,
+ -0.289008766412735,
+ -0.08170934021472931,
+ -0.26564186811447144,
+ 0.44369402527809143,
+ -0.42134034633636475,
+ 0.08346591889858246,
+ -0.21282491087913513,
+ 0.40806615352630615,
+ 0.07503722608089447,
+ -0.10082324594259262,
+ 0.3933199346065521
+ ],
+ [
+ -5.701622009277344,
+ 9.903303146362305,
+ 4.506409168243408,
+ 0.1177029088139534,
+ -0.6854044795036316,
+ -0.05685915797948837,
+ -0.7557153701782227,
+ -0.36272186040878296,
+ -0.7173914909362793,
+ 0.7834107875823975,
+ -0.5783897042274475,
+ 0.07463803142309189,
+ 0.23051045835018158,
+ -0.1040203869342804,
+ -0.16527576744556427
+ ],
+ [
+ -1.9546335935592651,
+ -14.448707580566406,
+ 16.07667350769043,
+ -1.465821385383606,
+ -0.9865174293518066,
+ -2.4181385040283203,
+ 0.32989591360092163,
+ -3.0165092945098877,
+ -0.3809855282306671,
+ 0.5750214457511902,
+ 0.19309604167938232,
+ -1.5880630016326904,
+ 0.061342015862464905,
+ -1.1758278608322144,
+ -3.212981939315796
+ ],
+ [
+ -3.055765390396118,
+ 8.030560493469238,
+ 4.855647087097168,
+ -0.9612423777580261,
+ -1.411350965499878,
+ 1.2918970584869385,
+ -1.1534048318862915,
+ -0.6059806942939758,
+ -0.10075324028730392,
+ 1.128818392753601,
+ -0.6344068050384521,
+ -1.9126962423324585,
+ 0.5415311455726624,
+ -2.0281970500946045,
+ 0.04516582563519478
+ ],
+ [
+ -7.2383012771606445,
+ -4.285314559936523,
+ -4.634127140045166,
+ -0.6623633503913879,
+ -2.776782512664795,
+ -0.7991152405738831,
+ 0.4331893026828766,
+ -0.1436343640089035,
+ -0.08059569448232651,
+ -0.4286653995513916,
+ 0.2052445262670517,
+ -1.969322681427002,
+ -0.32280221581459045,
+ -0.09131314605474472,
+ 0.4486905038356781
+ ],
+ [
+ -8.390392303466797,
+ -4.150084972381592,
+ -4.879299640655518,
+ -0.16836602985858917,
+ -2.4146530628204346,
+ -2.556464195251465,
+ 0.9004931449890137,
+ 1.1251652240753174,
+ 0.22162796556949615,
+ -1.0654157400131226,
+ 0.13042396306991577,
+ 0.47588858008384705,
+ -1.6081558465957642,
+ -1.9476381540298462,
+ -1.4114985466003418
+ ],
+ [
+ 0.03182445839047432,
+ -12.754207611083984,
+ 17.1997013092041,
+ 1.792147159576416,
+ 2.3960354328155518,
+ 7.0978498458862305,
+ -0.6223981976509094,
+ 9.433928489685059,
+ -1.496492624282837,
+ -0.3681964874267578,
+ 1.9502346515655518,
+ -1.3324201107025146,
+ -0.2752528190612793,
+ 2.0039496421813965,
+ -0.7128958702087402
+ ],
+ [
+ -5.246722221374512,
+ 13.310433387756348,
+ 6.20261287689209,
+ -0.05930429324507713,
+ -2.71423077583313,
+ 1.8864401578903198,
+ -1.0929769277572632,
+ -1.020850658416748,
+ -0.1883726865053177,
+ 0.13205845654010773,
+ -0.23299860954284668,
+ -1.2711858749389648,
+ 3.5967085361480713,
+ -2.0314621925354004,
+ 1.4053467512130737
+ ],
+ [
+ 17.72433090209961,
+ 1.1251739263534546,
+ -1.405901551246643,
+ -0.6390752196311951,
+ -0.580747127532959,
+ 0.15610720217227936,
+ 2.119433879852295,
+ -0.4004031717777252,
+ -0.17649440467357635,
+ -0.4631305932998657,
+ -1.5668816566467285,
+ -0.3027191162109375,
+ -1.56236732006073,
+ -1.3841357231140137,
+ 2.041842460632324
+ ],
+ [
+ -3.109299898147583,
+ -11.327393531799316,
+ 12.377817153930664,
+ -0.8400132060050964,
+ 2.6565539836883545,
+ 5.593794345855713,
+ -2.5959367752075195,
+ 12.44808292388916,
+ -1.075226068496704,
+ 0.08454342186450958,
+ -1.387001872062683,
+ -1.3608940839767456,
+ 0.5651962161064148,
+ 1.1027905941009521,
+ 1.0814998149871826
+ ],
+ [
+ -7.794628620147705,
+ -3.647747755050659,
+ -6.266522407531738,
+ -0.22173920273780823,
+ -2.2974045276641846,
+ -2.377558708190918,
+ 0.6823127865791321,
+ 1.3951449394226074,
+ -0.19108177721500397,
+ 0.3917752206325531,
+ 0.47401753067970276,
+ -1.1457408666610718,
+ -0.8914557099342346,
+ -0.5573046803474426,
+ 2.753009557723999
+ ],
+ [
+ 12.198766708374023,
+ -0.8882254958152771,
+ -3.3408703804016113,
+ -0.5749514698982239,
+ -1.1658008098602295,
+ 0.009045272134244442,
+ -1.822765588760376,
+ 0.7777139544487,
+ 0.15112896263599396,
+ 0.10767723619937897,
+ 0.6080081462860107,
+ -0.918311595916748,
+ -0.7204992771148682,
+ -0.0556701235473156,
+ 1.3792905807495117
+ ],
+ [
+ -6.378169536590576,
+ -2.3875925540924072,
+ -5.211617946624756,
+ -0.3236759901046753,
+ -1.0279154777526855,
+ 1.9642255306243896,
+ -0.0968950167298317,
+ -1.116280436515808,
+ -0.12979188561439514,
+ -0.13191083073616028,
+ 0.468875527381897,
+ 0.37450796365737915,
+ -0.3838207721710205,
+ 0.9104065299034119,
+ 1.0360482931137085
+ ],
+ [
+ -5.385272979736328,
+ 13.828802108764648,
+ 7.2639923095703125,
+ -0.4631432592868805,
+ -2.803455114364624,
+ -0.03698118403553963,
+ -0.07153591513633728,
+ -0.277224063873291,
+ 0.26904118061065674,
+ -1.7567626237869263,
+ -0.2619616389274597,
+ -0.14050883054733276,
+ 0.11592387408018112,
+ 1.0537855625152588,
+ 0.995434045791626
+ ],
+ [
+ -7.520326614379883,
+ -3.001997470855713,
+ -4.388489723205566,
+ 0.23098652064800262,
+ -2.888960361480713,
+ -1.154165267944336,
+ 0.5133972764015198,
+ 1.4732340574264526,
+ -0.5925716757774353,
+ 0.611148476600647,
+ 0.53006911277771,
+ 0.734542727470398,
+ 1.4422513246536255,
+ 2.689650774002075,
+ -1.5800451040267944
+ ],
+ [
+ -7.526669025421143,
+ -5.133892059326172,
+ -4.5747551918029785,
+ -0.14790070056915283,
+ -4.36122989654541,
+ -0.3135836720466614,
+ -0.033772632479667664,
+ 0.2402026653289795,
+ -0.24801406264305115,
+ 0.3176944851875305,
+ 0.5898320078849792,
+ -0.5752222537994385,
+ 1.0058118104934692,
+ -0.3828313648700714,
+ -2.743025541305542
+ ],
+ [
+ -7.158897876739502,
+ -4.795564651489258,
+ -5.759509563446045,
+ 0.524413526058197,
+ -3.6136317253112793,
+ -0.9520716667175293,
+ -0.13840936124324799,
+ 1.303300380706787,
+ 0.0039553591050207615,
+ -1.6595197916030884,
+ 0.25799447298049927,
+ -0.49236130714416504,
+ 1.3190375566482544,
+ -0.7646784782409668,
+ -0.7353445291519165
+ ],
+ [
+ -5.7398786544799805,
+ -2.3636245727539062,
+ -3.733139753341675,
+ 0.18981531262397766,
+ 0.6379444003105164,
+ 2.749208927154541,
+ -2.701148271560669,
+ -1.815198540687561,
+ 2.4555513858795166,
+ -0.07848959416151047,
+ -1.2806971073150635,
+ -0.031318169087171555,
+ 0.4933904707431793,
+ 1.0689598321914673,
+ -0.09746333211660385
+ ],
+ [
+ -4.532369136810303,
+ 12.643762588500977,
+ 6.361972808837891,
+ 0.7135351896286011,
+ -3.728808641433716,
+ 0.3537982702255249,
+ 0.3629753291606903,
+ 0.535239040851593,
+ 0.5627918243408203,
+ -1.099459171295166,
+ 0.10881971567869186,
+ 0.8114083409309387,
+ -0.5001876950263977,
+ 1.7933348417282104,
+ 0.6289044618606567
+ ],
+ [
+ -2.9009897708892822,
+ -17.06219482421875,
+ 18.30048942565918,
+ -2.8186521530151367,
+ -1.3712546825408936,
+ -4.028502941131592,
+ 0.5207890272140503,
+ -6.210597515106201,
+ -0.1275671422481537,
+ 0.7556989789009094,
+ -0.7033094763755798,
+ -1.1479538679122925,
+ -0.08936275541782379,
+ -1.9270058870315552,
+ 1.5569343566894531
+ ],
+ [
+ 14.24770736694336,
+ 0.07185164839029312,
+ -0.659893274307251,
+ -0.07882166653871536,
+ -0.05070047453045845,
+ 0.48449987173080444,
+ 2.579801559448242,
+ -0.7146349549293518,
+ -0.3262321949005127,
+ -0.20972901582717896,
+ 1.2568861246109009,
+ -1.0470867156982422,
+ -1.5853444337844849,
+ -1.6707404851913452,
+ 0.4702533781528473
+ ],
+ [
+ 15.702459335327148,
+ 0.1923084855079651,
+ -1.0252901315689087,
+ -1.2759329080581665,
+ -0.6311899423599243,
+ -1.9468648433685303,
+ -4.1576738357543945,
+ -0.6534209847450256,
+ -0.7933971285820007,
+ 0.49015384912490845,
+ -4.26126766204834,
+ 1.7315831184387207,
+ 1.1545073986053467,
+ 1.1272165775299072,
+ -0.6785593032836914
+ ]
+ ],
+ "adtEmbedding": [
+ [
+ 2.8318028450012207,
+ 2.6889729499816895,
+ 0.5272998809814453,
+ 0.4912741482257843,
+ 0.7322114706039429,
+ 0.5299299359321594,
+ 0.5384441018104553,
+ 0.7266590595245361,
+ 0.5153290033340454,
+ 1.8630807399749756,
+ 0.20727482438087463,
+ 0.6125134229660034,
+ 0.7601904273033142,
+ 1.9784611463546753
+ ],
+ [
+ 2.6769590377807617,
+ 3.210934638977051,
+ 0.45825818181037903,
+ 0.5147606730461121,
+ 0.7856290936470032,
+ 0.3402591049671173,
+ 0.6415525078773499,
+ 0.23668336868286133,
+ 1.5764379501342773,
+ 1.3512948751449585,
+ 2.0810484886169434,
+ 0.9334067702293396,
+ 0.8613783121109009,
+ 1.55366051197052
+ ],
+ [
+ 0.0621153749525547,
+ 0.09608898311853409,
+ 0.21689648926258087,
+ 0.3557475507259369,
+ 0.6016442179679871,
+ 2.8201401233673096,
+ 2.382474422454834,
+ 0.33687201142311096,
+ 0.4838624894618988,
+ 2.038573741912842,
+ 0.09825781732797623,
+ 0.6322313547134399,
+ 1.3570367097854614,
+ 0.44588133692741394
+ ],
+ [
+ 2.681356430053711,
+ 2.781632900238037,
+ 0.3730353116989136,
+ 0.434226930141449,
+ 0.6890870928764343,
+ 0.5299299359321594,
+ 0.5563841462135315,
+ 0.33687201142311096,
+ 1.0164127349853516,
+ 0.1573103368282318,
+ 2.1637630462646484,
+ 0.5718713402748108,
+ 0.4745543897151947,
+ 1.9277541637420654
+ ],
+ [
+ 0.18215049803256989,
+ 1.2777798175811768,
+ 0.6937257051467896,
+ 3.045278549194336,
+ 0.8514364361763,
+ 0.9636500477790833,
+ 0.5740079879760742,
+ 0.33687201142311096,
+ 1.3351269960403442,
+ 1.2662312984466553,
+ 1.228000521659851,
+ 1.003153681755066,
+ 0.8929534554481506,
+ 0.34261754155158997
+ ],
+ [
+ 0.06917575001716614,
+ 1.2528811693191528,
+ 0.26758623123168945,
+ 3.1559698581695557,
+ 0.644018828868866,
+ 0.4440770447254181,
+ 0.501568615436554,
+ 0.12532778084278107,
+ 0.3097064793109894,
+ 0.2432185411453247,
+ 1.7096437215805054,
+ 0.36526960134506226,
+ 0.4261729419231415,
+ 0.12017736583948135
+ ],
+ [
+ 2.41329288482666,
+ 2.750974655151367,
+ 0.555489718914032,
+ 0.41731157898902893,
+ 0.5624087452888489,
+ 0.2892536222934723,
+ 0.33874937891960144,
+ 0.0,
+ 2.0263454914093018,
+ 0.09083104133605957,
+ 2.3677608966827393,
+ 0.5509136915206909,
+ 0.3217654228210449,
+ 1.3768919706344604
+ ],
+ [
+ 2.699769973754883,
+ 0.21473664045333862,
+ 5.206839561462402,
+ 0.5377082228660583,
+ 1.188525915145874,
+ 0.7032298445701599,
+ 0.8606293797492981,
+ 0.7266590595245361,
+ 1.2789974212646484,
+ 0.41805219650268555,
+ 2.3267502784729004,
+ 1.6674801111221313,
+ 1.0639537572860718,
+ 2.7212882041931152
+ ],
+ [
+ 2.2256433963775635,
+ 2.598163366317749,
+ 0.32753586769104004,
+ 0.4087453782558441,
+ 0.7022207975387573,
+ 0.601360559463501,
+ 0.3163984417915344,
+ 0.5113852620124817,
+ 1.0711884498596191,
+ 0.13851656019687653,
+ 2.2976465225219727,
+ 1.0165376663208008,
+ 0.5648149847984314,
+ 2.128995418548584
+ ],
+ [
+ 0.12059704214334488,
+ 0.06112862005829811,
+ 0.3158300220966339,
+ 0.21993105113506317,
+ 0.6393982172012329,
+ 2.7896952629089355,
+ 2.3910279273986816,
+ 0.5113852620124817,
+ 0.4838624894618988,
+ 2.324460506439209,
+ 0.22645559906959534,
+ 0.43898242712020874,
+ 1.5190480947494507,
+ 0.20689572393894196
+ ],
+ [
+ 2.7373201847076416,
+ 2.7943060398101807,
+ 0.47847554087638855,
+ 0.4912741482257843,
+ 0.6713027358055115,
+ 0.309969037771225,
+ 0.4826081097126007,
+ 0.7266590595245361,
+ 0.6321395635604858,
+ 1.8640025854110718,
+ 0.15425117313861847,
+ 0.48526886105537415,
+ 0.4261729419231415,
+ 1.1620206832885742
+ ],
+ [
+ 0.055004797875881195,
+ 0.05319083109498024,
+ 0.3391062617301941,
+ 0.2804097831249237,
+ 0.6713027358055115,
+ 0.26809996366500854,
+ 0.38200613856315613,
+ 4.567813396453857,
+ 1.2347309589385986,
+ 1.966107726097107,
+ 0.0910329520702362,
+ 0.5295073390007019,
+ 0.2651671767234802,
+ 0.34261754155158997
+ ],
+ [
+ 1.6736137866973877,
+ 1.7642056941986084,
+ 3.7958948612213135,
+ 0.38259753584861755,
+ 2.664438009262085,
+ 3.93196439743042,
+ 1.4557183980941772,
+ 0.7266590595245361,
+ 0.8969447612762451,
+ 1.5421433448791504,
+ 2.4142839908599854,
+ 1.4516199827194214,
+ 2.55863881111145,
+ 1.2893855571746826
+ ],
+ [
+ 0.27222785353660583,
+ 1.0001801252365112,
+ 0.2551526427268982,
+ 2.996680498123169,
+ 0.7406176328659058,
+ 0.43507203459739685,
+ 0.46328112483024597,
+ 0.12532778084278107,
+ 0.6857970356941223,
+ 0.13851656019687653,
+ 1.6625701189041138,
+ 0.6322313547134399,
+ 0.5207027196884155,
+ 0.20689572393894196
+ ],
+ [
+ 2.5689237117767334,
+ 2.535914421081543,
+ 0.38409391045570374,
+ 0.3281567096710205,
+ 0.6531964540481567,
+ 0.25735294818878174,
+ 0.4435732364654541,
+ 0.42793068289756775,
+ 2.8437774181365967,
+ 0.11230579763650894,
+ 2.175555467605591,
+ 1.485459327697754,
+ 2.6802191734313965,
+ 0.3605857789516449
+ ],
+ [
+ 0.04784329608082771,
+ 0.9096441268920898,
+ 0.21689648926258087,
+ 3.1021549701690674,
+ 0.6713027358055115,
+ 0.4530017077922821,
+ 0.7048219442367554,
+ 0.7892204523086548,
+ 0.3470487594604492,
+ 1.2959394454956055,
+ 0.29379579424858093,
+ 0.5509136915206909,
+ 0.3217654228210449,
+ 0.3056824803352356
+ ],
+ [
+ 2.871636152267456,
+ 3.648634195327759,
+ 2.961536169052124,
+ 0.8135258555412292,
+ 0.8989907503128052,
+ 1.4517183303833008,
+ 0.6577491164207458,
+ 1.006408929824829,
+ 1.1562992334365845,
+ 0.15900148451328278,
+ 2.9987785816192627,
+ 2.8147366046905518,
+ 1.793459177017212,
+ 3.1753149032592773
+ ],
+ [
+ 0.13716620206832886,
+ 0.798808753490448,
+ 0.24256250262260437,
+ 2.6545629501342773,
+ 0.6064421534538269,
+ 0.23550604283809662,
+ 0.22163675725460052,
+ 0.12532778084278107,
+ 1.249705195426941,
+ 0.14023974537849426,
+ 1.3401044607162476,
+ 0.7253791689872742,
+ 0.4261729419231415,
+ 0.34261754155158997
+ ],
+ [
+ 0.0621153749525547,
+ 1.0261175632476807,
+ 2.2574079036712646,
+ 2.8424477577209473,
+ 0.7694947123527527,
+ 0.4618474245071411,
+ 0.5384441018104553,
+ 0.5113852620124817,
+ 1.053261160850525,
+ 0.4799940884113312,
+ 0.9692894816398621,
+ 0.5923988223075867,
+ 0.37533119320869446,
+ 0.1425720900297165
+ ],
+ [
+ 2.9244191646575928,
+ 0.169662207365036,
+ 2.0641908645629883,
+ 0.373727411031723,
+ 0.6016442179679871,
+ 0.35995450615882874,
+ 0.924148678779602,
+ 0.33687201142311096,
+ 0.8969447612762451,
+ 0.267766535282135,
+ 1.4975436925888062,
+ 0.6515679955482483,
+ 0.6865549683570862,
+ 2.046710968017578
+ ],
+ [
+ 2.2963645458221436,
+ 0.04919818416237831,
+ 2.75124454498291,
+ 0.29014328122138977,
+ 0.7447943091392517,
+ 0.6089913249015808,
+ 1.850900650024414,
+ 0.33687201142311096,
+ 0.7851776480674744,
+ 0.3151254951953888,
+ 1.7910218238830566,
+ 1.7601286172866821,
+ 0.7240503430366516,
+ 2.4398343563079834
+ ],
+ [
+ 2.593459367752075,
+ 2.1588222980499268,
+ 0.26758623123168945,
+ 0.38259753584861755,
+ 0.7406176328659058,
+ 1.5120444297790527,
+ 0.1706608682870865,
+ 0.7892204523086548,
+ 0.38304662704467773,
+ 1.7513058185577393,
+ 0.15425117313861847,
+ 0.5718713402748108,
+ 0.4261729419231415,
+ 0.7410764694213867
+ ],
+ [
+ 0.05856640636920929,
+ 0.9011473655700684,
+ 0.16349944472312927,
+ 2.8348734378814697,
+ 0.4789937138557434,
+ 0.39821842312812805,
+ 0.42346909642219543,
+ 0.23668336868286133,
+ 0.6593281030654907,
+ 1.102149248123169,
+ 0.35131219029426575,
+ 0.4150101840496063,
+ 0.3217654228210449,
+ 0.324320524930954
+ ],
+ [
+ 1.7731399536132812,
+ 0.07291840016841888,
+ 3.7984306812286377,
+ 0.2804097831249237,
+ 1.0813579559326172,
+ 1.773543357849121,
+ 1.1540570259094238,
+ 0.5113852620124817,
+ 1.0350066423416138,
+ 1.1775339841842651,
+ 1.5255708694458008,
+ 0.8584281802177429,
+ 0.8929534554481506,
+ 1.3239538669586182
+ ],
+ [
+ 2.6367554664611816,
+ 0.16610930860042572,
+ 0.13569219410419464,
+ 0.40010517835617065,
+ 0.6890870928764343,
+ 0.5381233096122742,
+ 0.2701396346092224,
+ 0.7266590595245361,
+ 0.41779354214668274,
+ 0.15561634302139282,
+ 2.3071787357330322,
+ 0.4150101840496063,
+ 0.37533119320869446,
+ 0.765129029750824
+ ],
+ [
+ 0.04784329608082771,
+ 0.07291840016841888,
+ 0.1214931309223175,
+ 0.16657860577106476,
+ 0.49517327547073364,
+ 2.262739658355713,
+ 1.715448260307312,
+ 0.12532778084278107,
+ 0.3097064793109894,
+ 1.766318917274475,
+ 0.0837555006146431,
+ 0.4623934328556061,
+ 1.0639537572860718,
+ 0.26733070611953735
+ ],
+ [
+ 2.459455728530884,
+ 2.479792356491089,
+ 0.41655433177948,
+ 0.45907527208328247,
+ 0.6757785677909851,
+ 0.26809996366500854,
+ 0.5563841462135315,
+ 0.5113852620124817,
+ 0.575438916683197,
+ 1.9175230264663696,
+ 0.20079860091209412,
+ 1.2523192167282104,
+ 0.5207027196884155,
+ 1.3960429430007935
+ ],
+ [
+ 0.05856640636920929,
+ 1.1479154825210571,
+ 0.27986711263656616,
+ 3.0489680767059326,
+ 0.6713027358055115,
+ 0.37926948070526123,
+ 0.4435732364654541,
+ 0.23668336868286133,
+ 0.05056070536375046,
+ 0.12811455130577087,
+ 1.6020312309265137,
+ 0.33943966031074524,
+ 0.3217654228210449,
+ 0.12017736583948135
+ ],
+ [
+ 1.231887936592102,
+ 0.06112862005829811,
+ 4.566332817077637,
+ 0.24049971997737885,
+ 0.6159695386886597,
+ 1.0504170656204224,
+ 2.5579757690429688,
+ 0.0,
+ 0.545835554599762,
+ 1.8258110284805298,
+ 0.22010280191898346,
+ 0.6515679955482483,
+ 1.5988694429397583,
+ 0.1859109103679657
+ ],
+ [
+ 1.837587594985962,
+ 0.09608898311853409,
+ 4.331662654876709,
+ 0.373727411031723,
+ 0.7022207975387573,
+ 0.5133389234542847,
+ 0.3606116473674774,
+ 0.7266590595245361,
+ 0.27091550827026367,
+ 2.1037275791168213,
+ 0.18111440539360046,
+ 0.7074308395385742,
+ 0.5648149847984314,
+ 1.3833165168762207
+ ],
+ [
+ 0.09692908078432083,
+ 0.07681763917207718,
+ 0.32753586769104004,
+ 0.29014328122138977,
+ 0.6486181616783142,
+ 0.33026403188705444,
+ 0.6893759369850159,
+ 4.982709884643555,
+ 0.8085528016090393,
+ 2.974761486053467,
+ 0.17446593940258026,
+ 0.7941148281097412,
+ 0.5648149847984314,
+ 0.39558178186416626
+ ],
+ [
+ 1.893602728843689,
+ 0.08070173114538193,
+ 4.104313850402832,
+ 0.3281567096710205,
+ 0.6668068170547485,
+ 0.2892536222934723,
+ 0.46328112483024597,
+ 0.5113852620124817,
+ 0.3097064793109894,
+ 0.16405780613422394,
+ 2.2367680072784424,
+ 0.6515679955482483,
+ 2.66975736618042,
+ 0.8450298309326172
+ ],
+ [
+ 1.7647839784622192,
+ 0.07681763917207718,
+ 2.7716548442840576,
+ 0.16657860577106476,
+ 0.6300925016403198,
+ 0.4259851574897766,
+ 1.1734294891357422,
+ 0.23668336868286133,
+ 0.3097064793109894,
+ 0.28430038690567017,
+ 1.2441518306732178,
+ 0.43898242712020874,
+ 0.3217654228210449,
+ 1.7884501218795776
+ ],
+ [
+ 0.1884147673845291,
+ 0.19071851670742035,
+ 0.21689648926258087,
+ 0.21993105113506317,
+ 0.5773030519485474,
+ 0.26809996366500854,
+ 0.4435732364654541,
+ 4.3958306312561035,
+ 0.6593281030654907,
+ 1.8786377906799316,
+ 0.10543086379766464,
+ 0.5076327323913574,
+ 0.14134716987609863,
+ 0.28669044375419617
+ ],
+ [
+ 0.04063013568520546,
+ 0.04518952965736389,
+ 0.1214931309223175,
+ 0.2506275475025177,
+ 0.587110698223114,
+ 2.3583736419677734,
+ 1.8060595989227295,
+ 0.23668336868286133,
+ 0.545835554599762,
+ 2.1791152954101562,
+ 0.20079860091209412,
+ 0.5295073390007019,
+ 1.1636052131652832,
+ 0.324320524930954
+ ],
+ [
+ 0.05143045634031296,
+ 0.07291840016841888,
+ 0.17711839079856873,
+ 0.21993105113506317,
+ 0.4789937138557434,
+ 0.33026403188705444,
+ 0.3163984417915344,
+ 4.213999271392822,
+ 1.1884138584136963,
+ 1.714897632598877,
+ 0.09825781732797623,
+ 0.4150101840496063,
+ 0.3217654228210449,
+ 0.24758876860141754
+ ],
+ [
+ 1.46786630153656,
+ 0.05716760456562042,
+ 4.552812099456787,
+ 0.20948584377765656,
+ 0.6531964540481567,
+ 0.2892536222934723,
+ 1.0292103290557861,
+ 0.7266590595245361,
+ 0.5153290033340454,
+ 0.36447873711586,
+ 1.1303409337997437,
+ 0.7253791689872742,
+ 0.4745543897151947,
+ 2.219001054763794
+ ],
+ [
+ 0.10374870896339417,
+ 0.11124304682016373,
+ 0.14969247579574585,
+ 0.2506275475025177,
+ 0.4401911795139313,
+ 3.08687424659729,
+ 2.6364941596984863,
+ 0.5113852620124817,
+ 0.38304662704467773,
+ 2.4337704181671143,
+ 0.21370938420295715,
+ 0.6125134229660034,
+ 1.551745057106018,
+ 0.42939433455467224
+ ],
+ [
+ 0.1003447026014328,
+ 1.1585261821746826,
+ 0.48843297362327576,
+ 2.887449264526367,
+ 0.7065606117248535,
+ 0.9636500477790833,
+ 0.33874937891960144,
+ 0.33687201142311096,
+ 0.230558842420578,
+ 0.5724232792854309,
+ 0.7808703184127808,
+ 0.43898242712020874,
+ 0.2651671767234802,
+ 0.24758876860141754
+ ],
+ [
+ 0.08661163598299026,
+ 1.5004640817642212,
+ 0.27986711263656616,
+ 3.530587673187256,
+ 0.8439154028892517,
+ 1.3038479089736938,
+ 0.501568615436554,
+ 0.33687201142311096,
+ 0.4513735771179199,
+ 0.2723029553890228,
+ 1.3667775392532349,
+ 0.9477507472038269,
+ 0.5207027196884155,
+ 0.3056824803352356
+ ],
+ [
+ 0.5290957689285278,
+ 0.04919818416237831,
+ 0.17711839079856873,
+ 0.21993105113506317,
+ 0.5773030519485474,
+ 0.25735294818878174,
+ 0.42346909642219543,
+ 0.6599210500717163,
+ 0.575438916683197,
+ 1.887977123260498,
+ 0.14054401218891144,
+ 0.4150101840496063,
+ 0.37533119320869446,
+ 1.268056035041809
+ ],
+ [
+ 0.09006262570619583,
+ 0.09226430952548981,
+ 0.16349944472312927,
+ 0.15555782616138458,
+ 0.5968231558799744,
+ 2.024040460586548,
+ 2.1587886810302734,
+ 0.33687201142311096,
+ 0.6321395635604858,
+ 2.0470499992370605,
+ 0.1877189427614212,
+ 0.36526960134506226,
+ 2.123875379562378,
+ 0.1859109103679657
+ ],
+ [
+ 0.09692908078432083,
+ 0.9445626735687256,
+ 0.32753586769104004,
+ 2.4992337226867676,
+ 0.5968231558799744,
+ 0.4440770447254181,
+ 0.4435732364654541,
+ 0.42793068289756775,
+ 0.41779354214668274,
+ 0.9678483605384827,
+ 0.2452753782272339,
+ 0.3129248023033142,
+ 0.2051723748445511,
+ 0.16447626054286957
+ ],
+ [
+ 2.4487648010253906,
+ 2.5689730644226074,
+ 0.39503154158592224,
+ 0.27058058977127075,
+ 0.620699405670166,
+ 0.4440770447254181,
+ 0.2701396346092224,
+ 0.12532778084278107,
+ 1.1884138584136963,
+ 0.07073231786489487,
+ 2.1987311840057373,
+ 1.76642906665802,
+ 1.0374081134796143,
+ 2.2134323120117188
+ ],
+ [
+ 2.7803142070770264,
+ 0.08070173114538193,
+ 0.3158300220966339,
+ 0.38259753584861755,
+ 0.6668068170547485,
+ 0.20181387662887573,
+ 0.4435732364654541,
+ 0.5884084701538086,
+ 0.4513735771179199,
+ 0.4775395393371582,
+ 1.2418603897094727,
+ 0.4150101840496063,
+ 0.5207027196884155,
+ 2.3587067127227783
+ ],
+ [
+ 1.8065022230148315,
+ 0.06507401168346405,
+ 3.2629241943359375,
+ 0.30933061242103577,
+ 0.6757785677909851,
+ 0.24648915231227875,
+ 0.2701396346092224,
+ 0.12532778084278107,
+ 0.41779354214668274,
+ 0.07808776944875717,
+ 1.9083762168884277,
+ 0.5718713402748108,
+ 0.3217654228210449,
+ 0.7531750202178955
+ ],
+ [
+ 2.2733421325683594,
+ 2.306926965713501,
+ 0.3391062617301941,
+ 0.40010517835617065,
+ 0.587110698223114,
+ 0.32016801834106445,
+ 0.3606116473674774,
+ 0.6599210500717163,
+ 0.575438916683197,
+ 0.09624312072992325,
+ 2.0133936405181885,
+ 1.8390928506851196,
+ 1.8060106039047241,
+ 0.41263097524642944
+ ],
+ [
+ 1.9990612268447876,
+ 0.14815296232700348,
+ 3.7460689544677734,
+ 0.3466349244117737,
+ 0.644018828868866,
+ 0.3501552939414978,
+ 0.38200613856315613,
+ 0.6599210500717163,
+ 0.8313940167427063,
+ 1.5767111778259277,
+ 0.1610349416732788,
+ 0.8738803863525391,
+ 1.8546979427337646,
+ 0.24758876860141754
+ ],
+ [
+ 1.0685805082321167,
+ 2.332537889480591,
+ 0.41655433177948,
+ 0.3557475507259369,
+ 0.6159695386886597,
+ 0.4530017077922821,
+ 0.40295252203941345,
+ 0.6599210500717163,
+ 0.14460423588752747,
+ 0.08356906473636627,
+ 2.0620367527008057,
+ 0.48526886105537415,
+ 0.14134716987609863,
+ 0.6782853007316589
+ ],
+ [
+ 2.07049298286438,
+ 0.07681763917207718,
+ 3.9996321201324463,
+ 0.3281567096710205,
+ 1.4607630968093872,
+ 3.8371896743774414,
+ 2.0649967193603516,
+ 0.5113852620124817,
+ 0.736721396446228,
+ 0.26015982031822205,
+ 1.5511515140533447,
+ 0.7603374719619751,
+ 0.37533119320869446,
+ 1.7536875009536743
+ ],
+ [
+ 0.11725001037120819,
+ 0.09226430952548981,
+ 0.2919990122318268,
+ 0.3374384939670563,
+ 0.6393982172012329,
+ 0.2996649742126465,
+ 0.4435732364654541,
+ 4.588390350341797,
+ 1.5764379501342773,
+ 2.3317980766296387,
+ 0.14054401218891144,
+ 0.8738803863525391,
+ 0.4261729419231415,
+ 0.324320524930954
+ ],
+ [
+ 1.8065022230148315,
+ 0.08070173114538193,
+ 4.845461845397949,
+ 0.27058058977127075,
+ 0.715184211730957,
+ 0.5216687917709351,
+ 0.5384441018104553,
+ 0.5884084701538086,
+ 0.4838624894618988,
+ 0.7158493399620056,
+ 1.0964316129684448,
+ 0.48526886105537415,
+ 1.8060106039047241,
+ 0.20689572393894196
+ ],
+ [
+ 0.029711894690990448,
+ 0.028992263600230217,
+ 1.294240117073059,
+ 0.29978296160697937,
+ 0.6622905731201172,
+ 1.1437031030654907,
+ 3.538140058517456,
+ 0.5113852620124817,
+ 0.6857970356941223,
+ 0.1505170315504074,
+ 0.7131527066230774,
+ 0.3904491066932678,
+ 0.8929534554481506,
+ 0.7288296818733215
+ ],
+ [
+ 1.5756323337554932,
+ 0.06507401168346405,
+ 3.3811240196228027,
+ 0.26065385341644287,
+ 0.6486181616783142,
+ 0.2996649742126465,
+ 0.38200613856315613,
+ 0.23668336868286133,
+ 0.7612429857254028,
+ 0.27831965684890747,
+ 1.2845240831375122,
+ 0.43898242712020874,
+ 0.37533119320869446,
+ 1.7267987728118896
+ ],
+ [
+ 0.04063013568520546,
+ 0.041164740920066833,
+ 0.30398550629615784,
+ 0.26065385341644287,
+ 0.5919787287712097,
+ 1.910759687423706,
+ 1.6871566772460938,
+ 0.6599210500717163,
+ 0.545835554599762,
+ 1.8837701082229614,
+ 0.11962441354990005,
+ 0.5509136915206909,
+ 1.46790611743927,
+ 0.28669044375419617
+ ],
+ [
+ 0.08661163598299026,
+ 0.7140223383903503,
+ 0.8707502484321594,
+ 2.9228827953338623,
+ 0.6064421534538269,
+ 0.601360559463501,
+ 1.9527260065078735,
+ 0.12532778084278107,
+ 1.7706902027130127,
+ 0.3309005796909332,
+ 1.3169655799865723,
+ 0.4150101840496063,
+ 0.4745543897151947,
+ 0.24758876860141754
+ ],
+ [
+ 1.8816146850585938,
+ 0.11873482167720795,
+ 4.4841461181640625,
+ 0.20948584377765656,
+ 0.7406176328659058,
+ 0.2787327170372009,
+ 0.6250892281532288,
+ 0.5884084701538086,
+ 0.4513735771179199,
+ 1.9921128749847412,
+ 0.14054401218891144,
+ 0.7074308395385742,
+ 0.5207027196884155,
+ 1.5697332620620728
+ ],
+ [
+ 0.05856640636920929,
+ 0.9280867576599121,
+ 0.09247544407844543,
+ 2.693845272064209,
+ 0.6978620290756226,
+ 0.41681498289108276,
+ 0.293536514043808,
+ 1.006408929824829,
+ 0.545835554599762,
+ 0.1922372430562973,
+ 1.1556644439697266,
+ 0.2856876850128174,
+ 0.5207027196884155,
+ 0.1425720900297165
+ ],
+ [
+ 1.9054486751556396,
+ 0.1224597692489624,
+ 4.391001224517822,
+ 0.21993105113506317,
+ 0.6486181616783142,
+ 0.2996649742126465,
+ 1.5587304830551147,
+ 0.42793068289756775,
+ 0.3097064793109894,
+ 0.31945252418518066,
+ 1.6534916162490845,
+ 0.7773687243461609,
+ 0.37533119320869446,
+ 1.784170389175415
+ ],
+ [
+ 2.415652275085449,
+ 2.671400785446167,
+ 0.3158300220966339,
+ 0.45907527208328247,
+ 0.6934841871261597,
+ 0.2787327170372009,
+ 0.33874937891960144,
+ 0.42793068289756775,
+ 0.9974663853645325,
+ 0.5555179715156555,
+ 1.0018935203552246,
+ 0.4150101840496063,
+ 0.14134716987609863,
+ 1.636542797088623
+ ],
+ [
+ 0.037003953009843826,
+ 0.7700414061546326,
+ 0.3391062617301941,
+ 3.0114457607269287,
+ 0.6757785677909851,
+ 0.5623083114624023,
+ 0.6736875176429749,
+ 0.12532778084278107,
+ 0.38304662704467773,
+ 0.17741768062114716,
+ 1.559535026550293,
+ 0.48526886105537415,
+ 0.5648149847984314,
+ 0.20689572393894196
+ ],
+ [
+ 1.8175841569900513,
+ 0.041164740920066833,
+ 3.9474029541015625,
+ 0.29014328122138977,
+ 0.5919787287712097,
+ 0.22440098226070404,
+ 1.764439582824707,
+ 0.5113852620124817,
+ 0.3470487594604492,
+ 0.2478674352169037,
+ 1.7168028354644775,
+ 0.810585081577301,
+ 0.6070632934570312,
+ 1.3573671579360962
+ ],
+ [
+ 0.13716620206832886,
+ 1.2707289457321167,
+ 0.48843297362327576,
+ 2.9882373809814453,
+ 1.2681728601455688,
+ 0.8328189849853516,
+ 0.6736875176429749,
+ 0.7266590595245361,
+ 1.1884138584136963,
+ 0.29909756779670715,
+ 1.8625801801681519,
+ 1.003153681755066,
+ 0.8929534554481506,
+ 0.37823686003685
+ ],
+ [
+ 0.07268733531236649,
+ 1.1015199422836304,
+ 0.2038121521472931,
+ 2.8677799701690674,
+ 0.5163466930389404,
+ 0.3501552939414978,
+ 0.4826081097126007,
+ 0.33687201142311096,
+ 0.3470487594604492,
+ 0.38759905099868774,
+ 0.9261941313743591,
+ 0.33943966031074524,
+ 0.37533119320869446,
+ 0.09726960211992264
+ ],
+ [
+ 1.8034019470214844,
+ 0.07681763917207718,
+ 3.899188280105591,
+ 0.19893039762973785,
+ 0.6016442179679871,
+ 0.5049390196800232,
+ 1.4337785243988037,
+ 0.0,
+ 0.4513735771179199,
+ 0.20037640631198883,
+ 0.7881190180778503,
+ 0.43898242712020874,
+ 1.3373017311096191,
+ 0.20689572393894196
+ ],
+ [
+ 3.223759412765503,
+ 0.14087916910648346,
+ 2.041388511657715,
+ 0.41731157898902893,
+ 0.5624087452888489,
+ 0.32016801834106445,
+ 0.24618223309516907,
+ 0.23668336868286133,
+ 0.575438916683197,
+ 0.1906014233827591,
+ 2.03230357170105,
+ 0.5295073390007019,
+ 0.7601904273033142,
+ 1.9277541637420654
+ ],
+ [
+ 2.44452166557312,
+ 3.029153823852539,
+ 0.5272998809814453,
+ 0.5224685072898865,
+ 0.7406176328659058,
+ 0.5381233096122742,
+ 0.46328112483024597,
+ 0.5113852620124817,
+ 0.6857970356941223,
+ 0.16068977117538452,
+ 2.276676654815674,
+ 1.1759603023529053,
+ 0.3217654228210449,
+ 2.551375150680542
+ ],
+ [
+ 1.8873411417007446,
+ 2.3972814083099365,
+ 0.26758623123168945,
+ 0.41731157898902893,
+ 0.6347562074661255,
+ 0.25735294818878174,
+ 0.38200613856315613,
+ 0.5113852620124817,
+ 0.4513735771179199,
+ 0.10162606835365295,
+ 2.0059421062469482,
+ 1.9599751234054565,
+ 1.8546979427337646,
+ 0.939187228679657
+ ],
+ [
+ 1.9851776361465454,
+ 2.9536216259002686,
+ 0.5272998809814453,
+ 0.3557475507259369,
+ 0.680234432220459,
+ 0.35995450615882874,
+ 0.3606116473674774,
+ 0.6599210500717163,
+ 1.0164127349853516,
+ 0.08902048319578171,
+ 2.433976411819458,
+ 1.4165953397750854,
+ 0.4745543897151947,
+ 2.270411491394043
+ ],
+ [
+ 0.05856640636920929,
+ 0.04919818416237831,
+ 0.10708953440189362,
+ 0.3557475507259369,
+ 0.5968231558799744,
+ 0.24648915231227875,
+ 0.4435732364654541,
+ 4.385904788970947,
+ 0.3470487594604492,
+ 1.8695158958435059,
+ 0.1610349416732788,
+ 0.5076327323913574,
+ 0.2651671767234802,
+ 0.34261754155158997
+ ],
+ [
+ 0.06917575001716614,
+ 0.08842495083808899,
+ 0.42714449763298035,
+ 0.2804097831249237,
+ 0.5822188854217529,
+ 0.2892536222934723,
+ 0.6250892281532288,
+ 4.432991981506348,
+ 1.2347309589385986,
+ 1.7673344612121582,
+ 0.20727482438087463,
+ 0.4623934328556061,
+ 0.5207027196884155,
+ 0.324320524930954
+ ],
+ [
+ 0.07618662714958191,
+ 0.674139678478241,
+ 0.14969247579574585,
+ 2.3867859840393066,
+ 0.7896223664283752,
+ 0.9529289603233337,
+ 0.293536514043808,
+ 0.42793068289756775,
+ 0.4838624894618988,
+ 0.5325236320495605,
+ 0.9324654936790466,
+ 0.5923988223075867,
+ 0.4261729419231415,
+ 0.2274492233991623
+ ],
+ [
+ 1.52607262134552,
+ 2.837069511413574,
+ 0.3505443334579468,
+ 0.434226930141449,
+ 0.7322114706039429,
+ 0.37926948070526123,
+ 0.42346909642219543,
+ 0.5113852620124817,
+ 0.917873740196228,
+ 0.12985575199127197,
+ 1.7776867151260376,
+ 1.342635989189148,
+ 0.3217654228210449,
+ 2.2409706115722656
+ ],
+ [
+ 1.9980393648147583,
+ 2.865233898162842,
+ 0.3505443334579468,
+ 0.45907527208328247,
+ 0.7816198468208313,
+ 0.7169710397720337,
+ 0.5563841462135315,
+ 0.5884084701538086,
+ 0.7851776480674744,
+ 0.23385542631149292,
+ 1.4142255783081055,
+ 2.449422597885132,
+ 0.37533119320869446,
+ 1.8664687871932983
+ ],
+ [
+ 0.06917575001716614,
+ 1.0789188146591187,
+ 0.41655433177948,
+ 2.808682441711426,
+ 0.7022207975387573,
+ 0.309969037771225,
+ 0.42346909642219543,
+ 0.42793068289756775,
+ 1.5326108932495117,
+ 0.44887059926986694,
+ 1.1803624629974365,
+ 1.3329943418502808,
+ 0.2651671767234802,
+ 0.24758876860141754
+ ],
+ [
+ 2.2421977519989014,
+ 2.22135853767395,
+ 0.4058508574962616,
+ 0.30933061242103577,
+ 0.6300925016403198,
+ 0.32016801834106445,
+ 0.38200613856315613,
+ 0.33687201142311096,
+ 3.2255373001098633,
+ 0.09983497112989426,
+ 1.7776867151260376,
+ 0.7941148281097412,
+ 0.9532615542411804,
+ 0.5688196420669556
+ ],
+ [
+ 1.7353079319000244,
+ 2.3442957401275635,
+ 0.27986711263656616,
+ 0.3913896679878235,
+ 0.6486181616783142,
+ 0.3402591049671173,
+ 0.293536514043808,
+ 0.5113852620124817,
+ 1.5764379501342773,
+ 0.07808776944875717,
+ 1.8784843683242798,
+ 0.33943966031074524,
+ 0.4261729419231415,
+ 1.4210200309753418
+ ],
+ [
+ 0.02604582905769348,
+ 0.05716760456562042,
+ 0.22981183230876923,
+ 0.27058058977127075,
+ 0.680234432220459,
+ 2.7591798305511475,
+ 2.991673707962036,
+ 0.5113852620124817,
+ 0.545835554599762,
+ 2.4162437915802,
+ 0.12664635479450226,
+ 0.6125134229660034,
+ 1.485245704650879,
+ 0.39558178186416626
+ ],
+ [
+ 2.129570484161377,
+ 2.9345943927764893,
+ 0.4058508574962616,
+ 0.5147606730461121,
+ 0.7816198468208313,
+ 0.7829799056053162,
+ 0.24618223309516907,
+ 0.7266590595245361,
+ 1.4387359619140625,
+ 0.10341396182775497,
+ 2.700737953186035,
+ 2.0299277305603027,
+ 0.7601904273033142,
+ 1.636542797088623
+ ],
+ [
+ 1.6721985340118408,
+ 2.42331600189209,
+ 0.3391062617301941,
+ 0.3913896679878235,
+ 0.5624087452888489,
+ 0.43507203459739685,
+ 0.3606116473674774,
+ 0.5113852620124817,
+ 1.1061067581176758,
+ 0.07991821318864822,
+ 2.13224720954895,
+ 0.8427334427833557,
+ 1.5988694429397583,
+ 1.4866544008255005
+ ],
+ [
+ 2.2465927600860596,
+ 2.508589029312134,
+ 0.21689648926258087,
+ 0.4425784945487976,
+ 0.7022207975387573,
+ 0.3501552939414978,
+ 0.4435732364654541,
+ 0.12532778084278107,
+ 0.7851776480674744,
+ 0.12811455130577087,
+ 1.7504708766937256,
+ 1.3802992105484009,
+ 1.3570367097854614,
+ 1.475036859512329
+ ],
+ [
+ 2.424367666244507,
+ 2.2896230220794678,
+ 0.3158300220966339,
+ 0.36477789282798767,
+ 0.5723628997802734,
+ 0.3887888491153717,
+ 0.5913266539573669,
+ 0.5884084701538086,
+ 0.3470487594604492,
+ 1.3200392723083496,
+ 0.17446593940258026,
+ 0.6515679955482483,
+ 0.6070632934570312,
+ 1.4271680116653442
+ ],
+ [
+ 1.5227893590927124,
+ 0.16254374384880066,
+ 4.7360711097717285,
+ 0.38259753584861755,
+ 0.7406176328659058,
+ 0.26809996366500854,
+ 0.46328112483024597,
+ 0.33687201142311096,
+ 0.4513735771179199,
+ 1.8119947910308838,
+ 0.22645559906959534,
+ 0.6515679955482483,
+ 0.7950698137283325,
+ 0.939187228679657
+ ],
+ [
+ 0.08314869552850723,
+ 0.5728067755699158,
+ 0.10708953440189362,
+ 2.142667531967163,
+ 0.3265842795372009,
+ 0.24648915231227875,
+ 0.1706608682870865,
+ 0.12532778084278107,
+ 0.09868758171796799,
+ 0.14195998013019562,
+ 0.6064555048942566,
+ 0.13712148368358612,
+ 0.14134716987609863,
+ 0.049817007035017014
+ ],
+ [
+ 2.16306471824646,
+ 0.11873482167720795,
+ 4.788907527923584,
+ 0.27058058977127075,
+ 0.8325268030166626,
+ 0.6892971992492676,
+ 0.6415525078773499,
+ 0.6599210500717163,
+ 0.6321395635604858,
+ 0.7584969997406006,
+ 1.0857622623443604,
+ 1.361644983291626,
+ 1.700890064239502,
+ 1.112892985343933
+ ],
+ [
+ 2.121018886566162,
+ 0.06900390237569809,
+ 4.068508148193359,
+ 0.31878796219825745,
+ 0.5110952258110046,
+ 0.33026403188705444,
+ 0.6893759369850159,
+ 0.6599210500717163,
+ 0.3470487594604492,
+ 0.12111927568912506,
+ 2.5631535053253174,
+ 0.5923988223075867,
+ 2.259477376937866,
+ 0.20689572393894196
+ ],
+ [
+ 0.20697550475597382,
+ 1.1236214637756348,
+ 0.3505443334579468,
+ 2.6699235439300537,
+ 0.9779547452926636,
+ 1.2651658058166504,
+ 0.6736875176429749,
+ 0.7266590595245361,
+ 1.0350066423416138,
+ 0.8912374973297119,
+ 0.9417994022369385,
+ 1.0297449827194214,
+ 0.6070632934570312,
+ 0.324320524930954
+ ],
+ [
+ 2.4555814266204834,
+ 2.5225138664245605,
+ 0.43762364983558655,
+ 0.4087453782558441,
+ 0.6347562074661255,
+ 0.39821842312812805,
+ 0.33874937891960144,
+ 0.33687201142311096,
+ 0.38304662704467773,
+ 1.390519142150879,
+ 0.22010280191898346,
+ 0.36526960134506226,
+ 0.5207027196884155,
+ 1.809578776359558
+ ],
+ [
+ 0.07268733531236649,
+ 0.703680157661438,
+ 0.2919990122318268,
+ 2.6003053188323975,
+ 0.6347562074661255,
+ 0.6165642142295837,
+ 1.372803807258606,
+ 0.903699517250061,
+ 0.917873740196228,
+ 0.7388208508491516,
+ 0.847699761390686,
+ 0.4623934328556061,
+ 0.3217654228210449,
+ 0.24758876860141754
+ ],
+ [
+ 0.04784329608082771,
+ 0.5488225221633911,
+ 0.47847554087638855,
+ 2.376016855239868,
+ 0.6846705675125122,
+ 0.601360559463501,
+ 0.293536514043808,
+ 0.23668336868286133,
+ 0.8313940167427063,
+ 0.17242862284183502,
+ 1.1754714250564575,
+ 0.3904491066932678,
+ 0.4261729419231415,
+ 0.2274492233991623
+ ],
+ [
+ 0.09006262570619583,
+ 0.08070173114538193,
+ 0.26758623123168945,
+ 0.29014328122138977,
+ 0.6713027358055115,
+ 2.0691704750061035,
+ 2.0125110149383545,
+ 0.42793068289756775,
+ 0.4513735771179199,
+ 1.9549524784088135,
+ 0.11255282163619995,
+ 0.43898242712020874,
+ 0.5207027196884155,
+ 0.3605857789516449
+ ],
+ [
+ 0.055004797875881195,
+ 1.1807069778442383,
+ 0.8079805374145508,
+ 3.164208173751831,
+ 0.6531964540481567,
+ 0.862752377986908,
+ 0.4435732364654541,
+ 0.5113852620124817,
+ 0.7851776480674744,
+ 0.7369266152381897,
+ 0.9812688231468201,
+ 1.4076440334320068,
+ 0.4745543897151947,
+ 0.26733070611953735
+ ],
+ [
+ 2.7744522094726562,
+ 2.4698874950408936,
+ 0.32753586769104004,
+ 0.5601409673690796,
+ 0.715184211730957,
+ 0.2996649742126465,
+ 0.40295252203941345,
+ 0.6599210500717163,
+ 0.38304662704467773,
+ 1.1787536144256592,
+ 0.38429826498031616,
+ 0.5923988223075867,
+ 0.7601904273033142,
+ 1.3768919706344604
+ ],
+ [
+ 2.215824604034424,
+ 0.12617088854312897,
+ 5.140074253082275,
+ 0.36477789282798767,
+ 0.8476830124855042,
+ 0.2892536222934723,
+ 0.608350396156311,
+ 0.23668336868286133,
+ 0.7851776480674744,
+ 1.8264490365982056,
+ 0.3287010192871094,
+ 0.7074308395385742,
+ 0.6475988030433655,
+ 1.282326102256775
+ ],
+ [
+ 2.413630247116089,
+ 2.244338035583496,
+ 0.4058508574962616,
+ 0.2804097831249237,
+ 0.5624087452888489,
+ 0.32016801834106445,
+ 0.38200613856315613,
+ 0.42793068289756775,
+ 0.41779354214668274,
+ 1.1646369695663452,
+ 0.23904138803482056,
+ 0.6125134229660034,
+ 0.3217654228210449,
+ 1.813751459121704
+ ],
+ [
+ 1.7319785356521606,
+ 0.033066242933273315,
+ 3.0623810291290283,
+ 0.2506275475025177,
+ 0.587110698223114,
+ 0.5049390196800232,
+ 0.608350396156311,
+ 0.5113852620124817,
+ 0.4838624894618988,
+ 0.2571008503437042,
+ 1.5099023580551147,
+ 0.5718713402748108,
+ 0.5207027196884155,
+ 1.935157299041748
+ ],
+ [
+ 2.659964084625244,
+ 2.9666852951049805,
+ 0.32753586769104004,
+ 0.5452418327331543,
+ 0.5005088448524475,
+ 0.3402591049671173,
+ 0.24618223309516907,
+ 0.6599210500717163,
+ 1.053261160850525,
+ 0.148811474442482,
+ 2.0364577770233154,
+ 0.43898242712020874,
+ 0.37533119320869446,
+ 2.2106363773345947
+ ],
+ [
+ 2.187891721725464,
+ 0.10369472205638885,
+ 4.768271446228027,
+ 0.31878796219825745,
+ 0.8210069537162781,
+ 0.32016801834106445,
+ 0.5563841462135315,
+ 0.903699517250061,
+ 0.7612429857254028,
+ 2.0790157318115234,
+ 0.22010280191898346,
+ 0.8890974521636963,
+ 0.7950698137283325,
+ 1.4332784414291382
+ ],
+ [
+ 2.3789637088775635,
+ 2.2034225463867188,
+ 0.8433377742767334,
+ 0.3913896679878235,
+ 0.6577538251876831,
+ 0.35995450615882874,
+ 0.293536514043808,
+ 0.33687201142311096,
+ 2.224175214767456,
+ 1.548479676246643,
+ 0.1610349416732788,
+ 0.4623934328556061,
+ 0.3217654228210449,
+ 0.8450298309326172
+ ],
+ [
+ 0.0621153749525547,
+ 0.06900390237569809,
+ 1.9560775756835938,
+ 0.2804097831249237,
+ 0.6159695386886597,
+ 3.0817577838897705,
+ 3.136383533477783,
+ 0.42793068289756775,
+ 0.545835554599762,
+ 2.4878756999969482,
+ 0.19428016245365143,
+ 0.7603374719619751,
+ 1.2966234683990479,
+ 0.39558178186416626
+ ],
+ [
+ 1.7952959537506104,
+ 0.10747600346803665,
+ 4.579507350921631,
+ 0.2804097831249237,
+ 0.715184211730957,
+ 0.9256123304367065,
+ 0.501568615436554,
+ 0.5113852620124817,
+ 0.7851776480674744,
+ 0.6383705139160156,
+ 0.8612175583839417,
+ 1.3986119031906128,
+ 0.9235620498657227,
+ 1.0787469148635864
+ ],
+ [
+ 0.05856640636920929,
+ 0.03712369129061699,
+ 0.07764460891485214,
+ 0.23026826977729797,
+ 0.5919787287712097,
+ 2.418644428253174,
+ 2.7009990215301514,
+ 0.33687201142311096,
+ 0.38304662704467773,
+ 2.012176036834717,
+ 0.09825781732797623,
+ 0.7074308395385742,
+ 1.209938645362854,
+ 0.28669044375419617
+ ],
+ [
+ 0.12059704214334488,
+ 0.6676944494247437,
+ 0.2551526427268982,
+ 2.676180601119995,
+ 0.6300925016403198,
+ 0.4879245460033417,
+ 0.3606116473674774,
+ 0.5884084701538086,
+ 0.7612429857254028,
+ 0.21326282620429993,
+ 1.4705350399017334,
+ 1.1413614749908447,
+ 0.2651671767234802,
+ 0.539472758769989
+ ],
+ [
+ 2.215824604034424,
+ 2.3953680992126465,
+ 0.45825818181037903,
+ 0.38259753584861755,
+ 0.5523545145988464,
+ 0.26809996366500854,
+ 0.3606116473674774,
+ 0.6599210500717163,
+ 0.4838624894618988,
+ 1.4456806182861328,
+ 0.17446593940258026,
+ 0.4150101840496063,
+ 0.6070632934570312,
+ 0.41263097524642944
+ ],
+ [
+ 2.109196662902832,
+ 2.6280441284179688,
+ 0.5647124648094177,
+ 0.3466349244117737,
+ 0.6668068170547485,
+ 0.309969037771225,
+ 0.293536514043808,
+ 0.42793068289756775,
+ 0.6321395635604858,
+ 0.9457827806472778,
+ 0.31140071153640747,
+ 0.5076327323913574,
+ 0.4261729419231415,
+ 1.6011239290237427
+ ],
+ [
+ 0.07268733531236649,
+ 1.1195144653320312,
+ 0.4058508574962616,
+ 3.278811454772949,
+ 0.6531964540481567,
+ 0.5299299359321594,
+ 1.2473710775375366,
+ 0.23668336868286133,
+ 0.545835554599762,
+ 1.486823320388794,
+ 0.3287010192871094,
+ 0.6125134229660034,
+ 0.3217654228210449,
+ 0.28669044375419617
+ ],
+ [
+ 1.935280680656433,
+ 2.0328152179718018,
+ 0.27986711263656616,
+ 0.29978296160697937,
+ 0.48980912566185,
+ 0.11848179996013641,
+ 0.14416414499282837,
+ 0.23668336868286133,
+ 0.18850478529930115,
+ 0.21166107058525085,
+ 0.9479739665985107,
+ 0.33943966031074524,
+ 0.2051723748445511,
+ 1.4210200309753418
+ ],
+ [
+ 2.490859270095825,
+ 2.3519248962402344,
+ 0.3391062617301941,
+ 0.434226930141449,
+ 0.7237340807914734,
+ 2.6115808486938477,
+ 2.4622433185577393,
+ 0.5113852620124817,
+ 0.7115833163261414,
+ 2.7564566135406494,
+ 0.23276829719543457,
+ 0.7941148281097412,
+ 1.5988694429397583,
+ 1.7969552278518677
+ ],
+ [
+ 0.1071411594748497,
+ 0.09989908337593079,
+ 0.14969247579574585,
+ 0.17747923731803894,
+ 0.5163466930389404,
+ 0.2787327170372009,
+ 0.4435732364654541,
+ 4.4314045906066895,
+ 0.41779354214668274,
+ 1.6504735946655273,
+ 0.12664635479450226,
+ 0.3904491066932678,
+ 0.37533119320869446,
+ 0.20689572393894196
+ ],
+ [
+ 0.07967371493577957,
+ 0.7912179827690125,
+ 0.26758623123168945,
+ 3.1160261631011963,
+ 0.8287015557289124,
+ 0.5049390196800232,
+ 1.1637901067733765,
+ 0.23668336868286133,
+ 0.38304662704467773,
+ 0.10519866645336151,
+ 1.7628097534179688,
+ 0.43898242712020874,
+ 0.5207027196884155,
+ 0.24758876860141754
+ ],
+ [
+ 1.3939769268035889,
+ 1.9923820495605469,
+ 0.13569219410419464,
+ 0.21993105113506317,
+ 0.43452298641204834,
+ 0.6751676797866821,
+ 0.1706608682870865,
+ 0.33687201142311096,
+ 1.0350066423416138,
+ 0.03312096372246742,
+ 1.7096437215805054,
+ 0.22888152301311493,
+ 0.14134716987609863,
+ 1.137758493423462
+ ],
+ [
+ 0.07618662714958191,
+ 1.0440295934677124,
+ 0.30398550629615784,
+ 2.7777702808380127,
+ 0.7613288760185242,
+ 0.6089913249015808,
+ 0.3606116473674774,
+ 0.23668336868286133,
+ 0.230558842420578,
+ 0.0872066468000412,
+ 1.135457158088684,
+ 0.5076327323913574,
+ 0.4745543897151947,
+ 0.16447626054286957
+ ],
+ [
+ 0.17900358140468597,
+ 0.919745922088623,
+ 0.17711839079856873,
+ 2.7120633125305176,
+ 0.748953640460968,
+ 0.47061556577682495,
+ 0.5384441018104553,
+ 0.5884084701538086,
+ 0.7115833163261414,
+ 0.30203092098236084,
+ 1.0192394256591797,
+ 0.36526960134506226,
+ 0.2651671767234802,
+ 0.3056824803352356
+ ],
+ [
+ 1.6855623722076416,
+ 2.046480655670166,
+ 0.21689648926258087,
+ 0.26065385341644287,
+ 0.38200926780700684,
+ 0.19032607972621918,
+ 0.293536514043808,
+ 0.12532778084278107,
+ 0.230558842420578,
+ 0.3165699243545532,
+ 0.575671911239624,
+ 0.22888152301311493,
+ 0.14134716987609863,
+ 0.703876793384552
+ ],
+ [
+ 0.9269273281097412,
+ 1.7967873811721802,
+ 2.7166292667388916,
+ 0.5224685072898865,
+ 2.981792449951172,
+ 4.664264678955078,
+ 2.244533061981201,
+ 1.1430498361587524,
+ 2.350471019744873,
+ 1.5709636211395264,
+ 1.960028052330017,
+ 1.6030418872833252,
+ 2.0287222862243652,
+ 0.7531750202178955
+ ],
+ [
+ 1.7563575506210327,
+ 2.568006753921509,
+ 0.38409391045570374,
+ 0.24049971997737885,
+ 0.6713027358055115,
+ 0.5381233096122742,
+ 0.293536514043808,
+ 0.23668336868286133,
+ 1.4011292457580566,
+ 0.06146126985549927,
+ 1.9973576068878174,
+ 1.7280162572860718,
+ 1.7678735256195068,
+ 0.41263097524642944
+ ],
+ [
+ 2.4842798709869385,
+ 0.09608898311853409,
+ 3.0684053897857666,
+ 0.26065385341644287,
+ 0.567398190498352,
+ 0.23550604283809662,
+ 0.3606116473674774,
+ 0.33687201142311096,
+ 0.575438916683197,
+ 0.10519866645336151,
+ 1.7365800142288208,
+ 0.5295073390007019,
+ 0.2651671767234802,
+ 0.7288296818733215
+ ],
+ [
+ 1.3272708654403687,
+ 1.1572059392929077,
+ 4.8189544677734375,
+ 2.8685152530670166,
+ 1.508493185043335,
+ 1.756798505783081,
+ 0.8340471982955933,
+ 0.33687201142311096,
+ 1.5656598806381226,
+ 0.7631253004074097,
+ 1.9897851943969727,
+ 1.0165376663208008,
+ 1.0374081134796143,
+ 1.1212503910064697
+ ],
+ [
+ 0.05143045634031296,
+ 1.3401964902877808,
+ 0.32753586769104004,
+ 3.0526440143585205,
+ 0.7065606117248535,
+ 1.035642147064209,
+ 1.0725810527801514,
+ 0.5884084701538086,
+ 0.545835554599762,
+ 1.5763016939163208,
+ 0.2452753782272339,
+ 0.48526886105537415,
+ 0.5648149847984314,
+ 0.28669044375419617
+ ],
+ [
+ 2.535144329071045,
+ 2.3539230823516846,
+ 0.32753586769104004,
+ 0.3557475507259369,
+ 0.587110698223114,
+ 0.601360559463501,
+ 0.40295252203941345,
+ 0.33687201142311096,
+ 0.4838624894618988,
+ 1.972465991973877,
+ 0.20079860091209412,
+ 0.6891544461250305,
+ 0.6865549683570862,
+ 1.779872179031372
+ ],
+ [
+ 3.212744951248169,
+ 0.14452268183231354,
+ 2.2971529960632324,
+ 0.3374384939670563,
+ 0.55739426612854,
+ 0.309969037771225,
+ 0.42346909642219543,
+ 0.5884084701538086,
+ 0.4838624894618988,
+ 0.15391945838928223,
+ 2.1406867504119873,
+ 0.33943966031074524,
+ 0.6070632934570312,
+ 1.6944866180419922
+ ],
+ [
+ 0.10374870896339417,
+ 1.3790379762649536,
+ 0.86396723985672,
+ 2.9631409645080566,
+ 0.6531964540481567,
+ 0.6751676797866821,
+ 2.565157413482666,
+ 0.5113852620124817,
+ 2.1239633560180664,
+ 0.43100881576538086,
+ 1.4595239162445068,
+ 0.6705377697944641,
+ 0.4261729419231415,
+ 0.24758876860141754
+ ],
+ [
+ 2.5543770790100098,
+ 0.16610930860042572,
+ 5.027149677276611,
+ 0.4425784945487976,
+ 0.7735527753829956,
+ 0.5936712026596069,
+ 0.7497782707214355,
+ 0.5884084701538086,
+ 0.5153290033340454,
+ 2.1761975288391113,
+ 0.20727482438087463,
+ 0.9477507472038269,
+ 0.7240503430366516,
+ 1.4981385469436646
+ ],
+ [
+ 0.24309439957141876,
+ 1.0703099966049194,
+ 0.27986711263656616,
+ 3.162017822265625,
+ 0.6159695386886597,
+ 0.37926948070526123,
+ 0.5384441018104553,
+ 0.23668336868286133,
+ 0.27091550827026367,
+ 0.828366756439209,
+ 0.6193638443946838,
+ 1.231090784072876,
+ 0.3217654228210449,
+ 0.1859109103679657
+ ],
+ [
+ 0.0935017466545105,
+ 1.1768282651901245,
+ 0.13569219410419464,
+ 3.0654044151306152,
+ 0.7364233732223511,
+ 0.601360559463501,
+ 0.6250892281532288,
+ 0.5113852620124817,
+ 1.2040923833847046,
+ 0.782333493232727,
+ 0.9722977876663208,
+ 0.5509136915206909,
+ 0.4261729419231415,
+ 0.28669044375419617
+ ],
+ [
+ 2.238185167312622,
+ 2.638310194015503,
+ 0.3730353116989136,
+ 0.4912741482257843,
+ 0.6531964540481567,
+ 0.3501552939414978,
+ 0.4435732364654541,
+ 0.5113852620124817,
+ 1.6286513805389404,
+ 0.07991821318864822,
+ 2.436758279800415,
+ 0.5509136915206909,
+ 0.6475988030433655,
+ 2.0367655754089355
+ ],
+ [
+ 0.1469775140285492,
+ 0.8286082744598389,
+ 0.45825818181037903,
+ 0.31878796219825745,
+ 0.6393982172012329,
+ 0.5133389234542847,
+ 1.2473710775375366,
+ 0.6599210500717163,
+ 0.38304662704467773,
+ 0.29172635078430176,
+ 0.7400348782539368,
+ 0.3904491066932678,
+ 0.6070632934570312,
+ 0.049817007035017014
+ ],
+ [
+ 1.9363685846328735,
+ 0.129868283867836,
+ 4.318844318389893,
+ 0.31878796219825745,
+ 0.5968231558799744,
+ 0.15504899621009827,
+ 1.324501872062683,
+ 0.33687201142311096,
+ 0.6593281030654907,
+ 0.5101780891418457,
+ 0.9931064248085022,
+ 0.5076327323913574,
+ 0.5207027196884155,
+ 1.9127813577651978
+ ],
+ [
+ 0.13716620206832886,
+ 1.2395871877670288,
+ 0.7562811374664307,
+ 2.9325833320617676,
+ 0.7065606117248535,
+ 0.309969037771225,
+ 0.3606116473674774,
+ 0.23668336868286133,
+ 0.14460423588752747,
+ 0.11230579763650894,
+ 1.6980814933776855,
+ 0.33943966031074524,
+ 0.5207027196884155,
+ 0.1425720900297165
+ ],
+ [
+ 0.08661163598299026,
+ 0.04518952965736389,
+ 0.4058508574962616,
+ 0.29978296160697937,
+ 0.6016442179679871,
+ 3.17651104927063,
+ 2.773284435272217,
+ 0.33687201142311096,
+ 0.38304662704467773,
+ 2.431509256362915,
+ 0.15425117313861847,
+ 0.9040864706039429,
+ 1.7678735256195068,
+ 0.20689572393894196
+ ],
+ [
+ 2.2096383571624756,
+ 2.7303664684295654,
+ 0.4479941427707672,
+ 0.40010517835617065,
+ 0.765420138835907,
+ 0.40755993127822876,
+ 0.520176351070404,
+ 0.5113852620124817,
+ 0.4513735771179199,
+ 0.11584051698446274,
+ 2.6975326538085938,
+ 2.6806440353393555,
+ 2.1147572994232178,
+ 0.5092385411262512
+ ],
+ [
+ 0.055004797875881195,
+ 0.9955323338508606,
+ 0.30398550629615784,
+ 3.3142685890197754,
+ 0.7237340807914734,
+ 0.309969037771225,
+ 0.5740079879760742,
+ 0.33687201142311096,
+ 1.3074560165405273,
+ 0.148811474442482,
+ 1.9453784227371216,
+ 0.8427334427833557,
+ 0.5648149847984314,
+ 0.26733070611953735
+ ],
+ [
+ 2.649360179901123,
+ 2.557640314102173,
+ 0.7562811374664307,
+ 0.4425784945487976,
+ 0.7572208046913147,
+ 0.37926948070526123,
+ 0.520176351070404,
+ 0.5884084701538086,
+ 0.41779354214668274,
+ 2.0268983840942383,
+ 0.14742107689380646,
+ 0.6515679955482483,
+ 0.7601904273033142,
+ 1.2535792589187622
+ ],
+ [
+ 0.09692908078432083,
+ 0.08457080274820328,
+ 0.1214931309223175,
+ 0.16657860577106476,
+ 0.5370810627937317,
+ 3.0951333045959473,
+ 2.421778678894043,
+ 0.7892204523086548,
+ 0.4513735771179199,
+ 2.594998836517334,
+ 0.16777299344539642,
+ 0.5509136915206909,
+ 1.7416160106658936,
+ 0.324320524930954
+ ],
+ [
+ 1.975813627243042,
+ 2.637408494949341,
+ 0.6007756590843201,
+ 3.0266239643096924,
+ 0.7279817461967468,
+ 0.37926948070526123,
+ 0.608350396156311,
+ 0.5113852620124817,
+ 1.3752487897872925,
+ 1.2246136665344238,
+ 2.184532880783081,
+ 2.230408191680908,
+ 0.9235620498657227,
+ 2.2078325748443604
+ ],
+ [
+ 1.1659753322601318,
+ 0.04518952965736389,
+ 1.8425465822219849,
+ 0.16657860577106476,
+ 0.2611686885356903,
+ 0.14300790429115295,
+ 0.6415525078773499,
+ 0.0,
+ 0.09868758171796799,
+ 0.13679037988185883,
+ 0.9842414259910583,
+ 0.3129248023033142,
+ 0.07316888868808746,
+ 0.7886165976524353
+ ],
+ [
+ 1.630262851715088,
+ 0.10747600346803665,
+ 2.8521692752838135,
+ 0.29978296160697937,
+ 0.6016442179679871,
+ 0.6165642142295837,
+ 0.3163984417915344,
+ 0.23668336868286133,
+ 0.575438916683197,
+ 0.1807299554347992,
+ 1.5561901330947876,
+ 0.7603374719619751,
+ 0.37533119320869446,
+ 2.053286552429199
+ ],
+ [
+ 0.08314869552850723,
+ 0.47321227192878723,
+ 0.3730353116989136,
+ 2.78498911857605,
+ 0.748953640460968,
+ 0.8019618391990662,
+ 0.4435732364654541,
+ 0.42793068289756775,
+ 0.38304662704467773,
+ 0.09083104133605957,
+ 1.4613674879074097,
+ 0.5923988223075867,
+ 0.37533119320869446,
+ 0.09726960211992264
+ ],
+ [
+ 0.12393291294574738,
+ 0.09989908337593079,
+ 0.5080552101135254,
+ 0.29014328122138977,
+ 0.6486181616783142,
+ 3.243614435195923,
+ 2.690532922744751,
+ 0.5113852620124817,
+ 0.7115833163261414,
+ 2.430812358856201,
+ 0.19428016245365143,
+ 0.5923988223075867,
+ 1.8546979427337646,
+ 0.3056824803352356
+ ],
+ [
+ 0.06917575001716614,
+ 0.033066242933273315,
+ 0.17711839079856873,
+ 0.1444142460823059,
+ 0.33916914463043213,
+ 0.16694684326648712,
+ 0.2701396346092224,
+ 4.523896217346191,
+ 0.41779354214668274,
+ 1.3295189142227173,
+ 0.061599891632795334,
+ 0.6515679955482483,
+ 0.3217654228210449,
+ 0.1425720900297165
+ ],
+ [
+ 2.7501726150512695,
+ 2.3634583950042725,
+ 0.4479941427707672,
+ 0.45907527208328247,
+ 0.7613288760185242,
+ 0.309969037771225,
+ 0.5384441018104553,
+ 0.33687201142311096,
+ 0.545835554599762,
+ 1.6625683307647705,
+ 0.22010280191898346,
+ 0.33943966031074524,
+ 0.7950698137283325,
+ 1.4924129247665405
+ ],
+ [
+ 2.14115571975708,
+ 2.750706195831299,
+ 0.32753586769104004,
+ 0.434226930141449,
+ 0.5163466930389404,
+ 0.5623083114624023,
+ 0.501568615436554,
+ 0.33687201142311096,
+ 1.053261160850525,
+ 0.17076006531715393,
+ 1.6473932266235352,
+ 1.5806055068969727,
+ 0.37533119320869446,
+ 2.199373483657837
+ ],
+ [
+ 2.3459441661834717,
+ 2.6632301807403564,
+ 0.555489718914032,
+ 0.434226930141449,
+ 0.6577538251876831,
+ 0.24648915231227875,
+ 0.38200613856315613,
+ 0.12532778084278107,
+ 0.9781542420387268,
+ 0.3849066197872162,
+ 1.1226171255111694,
+ 0.6515679955482483,
+ 0.7601904273033142,
+ 1.50949227809906
+ ],
+ [
+ 2.1398258209228516,
+ 2.380707263946533,
+ 0.4058508574962616,
+ 0.4087453782558441,
+ 0.5919787287712097,
+ 0.9529289603233337,
+ 0.3606116473674774,
+ 0.12532778084278107,
+ 1.9986201524734497,
+ 0.41019687056541443,
+ 0.4967121481895447,
+ 0.33943966031074524,
+ 0.4261729419231415,
+ 1.8743380308151245
+ ],
+ [
+ 0.3522104322910309,
+ 0.2960927188396454,
+ 0.8774875402450562,
+ 0.6785493493080139,
+ 2.934401512145996,
+ 4.789186477661133,
+ 2.835247755050659,
+ 1.184740662574768,
+ 1.4630436897277832,
+ 2.4217100143432617,
+ 1.4741785526275635,
+ 1.1983784437179565,
+ 1.4140070676803589,
+ 0.9087759256362915
+ ],
+ [
+ 0.06917575001716614,
+ 0.7484066486358643,
+ 0.13569219410419464,
+ 2.9927926063537598,
+ 0.6668068170547485,
+ 0.32016801834106445,
+ 0.7497782707214355,
+ 0.33687201142311096,
+ 0.6041910648345947,
+ 0.39028429985046387,
+ 1.1803624629974365,
+ 0.3904491066932678,
+ 0.4745543897151947,
+ 0.28669044375419617
+ ],
+ [
+ 2.241396427154541,
+ 0.09989908337593079,
+ 4.290937423706055,
+ 0.30933061242103577,
+ 0.6622905731201172,
+ 0.4618474245071411,
+ 0.8067390322685242,
+ 0.5113852620124817,
+ 0.545835554599762,
+ 2.016404628753662,
+ 0.20727482438087463,
+ 1.1983784437179565,
+ 2.1147572994232178,
+ 0.3056824803352356
+ ],
+ [
+ 1.9309169054031372,
+ 2.6940999031066895,
+ 2.3447108268737793,
+ 0.4087453782558441,
+ 0.5919787287712097,
+ 0.3501552939414978,
+ 0.38200613856315613,
+ 0.5113852620124817,
+ 0.7851776480674744,
+ 0.04833567887544632,
+ 2.131304979324341,
+ 1.1413614749908447,
+ 0.14134716987609863,
+ 2.150038957595825
+ ],
+ [
+ 0.04784329608082771,
+ 1.0246104001998901,
+ 0.24256250262260437,
+ 2.914492607116699,
+ 0.715184211730957,
+ 0.37926948070526123,
+ 0.33874937891960144,
+ 0.5113852620124817,
+ 0.38304662704467773,
+ 0.3309005796909332,
+ 1.0502792596817017,
+ 0.48526886105537415,
+ 0.2051723748445511,
+ 0.1859109103679657
+ ],
+ [
+ 2.2469913959503174,
+ 2.873103380203247,
+ 0.43762364983558655,
+ 0.4912741482257843,
+ 0.680234432220459,
+ 0.47930750250816345,
+ 0.24618223309516907,
+ 0.42793068289756775,
+ 0.3097064793109894,
+ 0.148811474442482,
+ 1.477808952331543,
+ 1.0297449827194214,
+ 0.37533119320869446,
+ 0.703876793384552
+ ],
+ [
+ 0.11389173567295074,
+ 0.7283240556716919,
+ 0.5367850661277771,
+ 2.3528940677642822,
+ 0.6622905731201172,
+ 0.7893473505973816,
+ 0.3606116473674774,
+ 1.370530366897583,
+ 1.4509637355804443,
+ 0.5172886252403259,
+ 0.7092527151107788,
+ 0.2856876850128174,
+ 0.37533119320869446,
+ 0.1859109103679657
+ ],
+ [
+ 2.093522787094116,
+ 2.452676296234131,
+ 0.3391062617301941,
+ 0.4991644322872162,
+ 0.6622905731201172,
+ 0.21317119896411896,
+ 0.46328112483024597,
+ 0.33687201142311096,
+ 0.3470487594604492,
+ 1.681546926498413,
+ 0.20079860091209412,
+ 0.7253791689872742,
+ 0.37533119320869446,
+ 0.8115651607513428
+ ],
+ [
+ 2.216646432876587,
+ 2.436267375946045,
+ 0.42714449763298035,
+ 0.45907527208328247,
+ 0.5919787287712097,
+ 0.2892536222934723,
+ 0.40295252203941345,
+ 0.33687201142311096,
+ 0.230558842420578,
+ 1.223448634147644,
+ 0.3287010192871094,
+ 0.43898242712020874,
+ 0.2651671767234802,
+ 1.1620206832885742
+ ],
+ [
+ 2.7831156253814697,
+ 2.4255483150482178,
+ 0.3505443334579468,
+ 0.40010517835617065,
+ 0.8093528151512146,
+ 0.5216687917709351,
+ 0.42346909642219543,
+ 0.33687201142311096,
+ 0.3470487594604492,
+ 1.1332435607910156,
+ 0.2452753782272339,
+ 0.5509136915206909,
+ 0.3217654228210449,
+ 1.6513465642929077
+ ],
+ [
+ 1.44057035446167,
+ 0.06112862005829811,
+ 2.9083471298217773,
+ 0.2506275475025177,
+ 0.7447943091392517,
+ 0.36965861916542053,
+ 1.8212311267852783,
+ 0.5113852620124817,
+ 0.27091550827026367,
+ 0.21486201882362366,
+ 1.5477783679962158,
+ 0.5509136915206909,
+ 0.4261729419231415,
+ 1.4453884363174438
+ ],
+ [
+ 2.7633376121520996,
+ 2.6549923419952393,
+ 0.4058508574962616,
+ 0.4991644322872162,
+ 0.7896223664283752,
+ 0.8686330318450928,
+ 0.38200613856315613,
+ 0.33687201142311096,
+ 0.4513735771179199,
+ 2.1063826084136963,
+ 0.16777299344539642,
+ 0.5718713402748108,
+ 0.6475988030433655,
+ 1.985499620437622
+ ],
+ [
+ 1.9160468578338623,
+ 1.9767879247665405,
+ 4.062135696411133,
+ 0.5748213529586792,
+ 2.9050424098968506,
+ 4.204631805419922,
+ 1.7265444993972778,
+ 0.8480972647666931,
+ 0.8313940167427063,
+ 1.720940351486206,
+ 2.7160871028900146,
+ 1.6948384046554565,
+ 2.690572500228882,
+ 1.4924129247665405
+ ],
+ [
+ 2.630800247192383,
+ 2.3551199436187744,
+ 0.30398550629615784,
+ 0.4258050322532654,
+ 0.7530957460403442,
+ 0.3402591049671173,
+ 0.293536514043808,
+ 0.5113852620124817,
+ 0.3097064793109894,
+ 1.7963430881500244,
+ 0.17446593940258026,
+ 0.6125134229660034,
+ 0.7240503430366516,
+ 1.7176728248596191
+ ],
+ [
+ 2.299016237258911,
+ 1.0291253328323364,
+ 0.3391062617301941,
+ 0.2804097831249237,
+ 0.7322114706039429,
+ 0.40755993127822876,
+ 0.6415525078773499,
+ 0.23668336868286133,
+ 0.41779354214668274,
+ 2.5474560260772705,
+ 0.17446593940258026,
+ 0.6515679955482483,
+ 2.2434990406036377,
+ 0.41263097524642944
+ ],
+ [
+ 1.8399852514266968,
+ 0.3389178514480591,
+ 4.53772497177124,
+ 0.26065385341644287,
+ 0.6300925016403198,
+ 0.309969037771225,
+ 0.5913266539573669,
+ 0.6599210500717163,
+ 0.6321395635604858,
+ 1.705224633216858,
+ 0.2452753782272339,
+ 0.5076327323913574,
+ 0.6865549683570862,
+ 1.112892985343933
+ ],
+ [
+ 0.0935017466545105,
+ 0.8156801462173462,
+ 0.27986711263656616,
+ 2.467904806137085,
+ 0.8325268030166626,
+ 0.5702417492866516,
+ 0.6415525078773499,
+ 0.5113852620124817,
+ 0.8313940167427063,
+ 0.15391945838928223,
+ 1.5477783679962158,
+ 0.43898242712020874,
+ 0.5648149847984314,
+ 0.20689572393894196
+ ],
+ [
+ 2.6450870037078857,
+ 2.722665548324585,
+ 0.42714449763298035,
+ 0.434226930141449,
+ 0.6531964540481567,
+ 0.41681498289108276,
+ 0.38200613856315613,
+ 0.42793068289756775,
+ 0.7612429857254028,
+ 0.11760319769382477,
+ 2.6414096355438232,
+ 0.9477507472038269,
+ 1.432298183441162,
+ 1.3374534845352173
+ ],
+ [
+ 0.0621153749525547,
+ 0.06112862005829811,
+ 0.21689648926258087,
+ 0.1444142460823059,
+ 0.5822188854217529,
+ 2.561073064804077,
+ 1.957116961479187,
+ 0.33687201142311096,
+ 0.230558842420578,
+ 2.2365996837615967,
+ 0.1610349416732788,
+ 0.4623934328556061,
+ 1.3570367097854614,
+ 0.1859109103679657
+ ],
+ [
+ 0.029711894690990448,
+ 0.705757200717926,
+ 0.14969247579574585,
+ 1.7335790395736694,
+ 0.5163466930389404,
+ 0.309969037771225,
+ 1.1829767227172852,
+ 0.33687201142311096,
+ 0.4838624894618988,
+ 0.6570274233818054,
+ 0.5205897092819214,
+ 0.33943966031074524,
+ 0.37533119320869446,
+ 0.1859109103679657
+ ],
+ [
+ 2.812520980834961,
+ 2.542217493057251,
+ 0.5177238583564758,
+ 0.4753043055534363,
+ 0.7935997247695923,
+ 3.4431889057159424,
+ 2.645339012145996,
+ 0.5884084701538086,
+ 0.6857970356941223,
+ 2.83355450630188,
+ 0.20727482438087463,
+ 0.810585081577301,
+ 2.0484910011291504,
+ 1.469177007675171
+ ],
+ [
+ 2.4112658500671387,
+ 2.3826487064361572,
+ 0.39503154158592224,
+ 0.40010517835617065,
+ 0.6757785677909851,
+ 0.5623083114624023,
+ 0.42346909642219543,
+ 0.33687201142311096,
+ 0.545835554599762,
+ 1.4899535179138184,
+ 0.23276829719543457,
+ 0.6322313547134399,
+ 0.4261729419231415,
+ 1.1778745651245117
+ ],
+ [
+ 2.4210245609283447,
+ 2.2972567081451416,
+ 0.30398550629615784,
+ 0.40010517835617065,
+ 0.6064421534538269,
+ 0.6165642142295837,
+ 0.33874937891960144,
+ 0.42793068289756775,
+ 0.27091550827026367,
+ 0.281314492225647,
+ 1.061331868171692,
+ 1.5881403684616089,
+ 0.2651671767234802,
+ 0.41263097524642944
+ ],
+ [
+ 0.05856640636920929,
+ 0.08842495083808899,
+ 0.26758623123168945,
+ 0.26065385341644287,
+ 0.7065606117248535,
+ 2.8916537761688232,
+ 1.9123151302337646,
+ 0.42793068289756775,
+ 0.4838624894618988,
+ 2.3632352352142334,
+ 0.1610349416732788,
+ 0.33943966031074524,
+ 1.5355302095413208,
+ 0.3056824803352356
+ ],
+ [
+ 0.17900358140468597,
+ 0.5142478346824646,
+ 0.13569219410419464,
+ 2.4524219036102295,
+ 0.6016442179679871,
+ 0.22440098226070404,
+ 2.1939735412597656,
+ 0.33687201142311096,
+ 0.38304662704467773,
+ 0.9587767720222473,
+ 0.334401935338974,
+ 0.4150101840496063,
+ 0.6865549683570862,
+ 0.16447626054286957
+ ],
+ [
+ 1.8839092254638672,
+ 2.5776283740997314,
+ 0.30398550629615784,
+ 0.3466349244117737,
+ 0.5421981811523438,
+ 0.3501552939414978,
+ 0.14416414499282837,
+ 0.23668336868286133,
+ 1.053261160850525,
+ 0.04833567887544632,
+ 2.0720880031585693,
+ 0.7603374719619751,
+ 1.089812994003296,
+ 1.4573534727096558
+ ],
+ [
+ 0.07268733531236649,
+ 0.12617088854312897,
+ 0.2038121521472931,
+ 0.5069929361343384,
+ 0.7735527753829956,
+ 2.83502197265625,
+ 2.7092936038970947,
+ 4.436159133911133,
+ 1.1398468017578125,
+ 2.9358839988708496,
+ 0.25147074460983276,
+ 1.0297449827194214,
+ 1.842746615409851,
+ 0.3605857789516449
+ ],
+ [
+ 0.08314869552850723,
+ 0.04919818416237831,
+ 0.09247544407844543,
+ 0.27058058977127075,
+ 0.5370810627937317,
+ 0.2892536222934723,
+ 0.38200613856315613,
+ 4.828633785247803,
+ 0.4513735771179199,
+ 2.0234999656677246,
+ 0.12664635479450226,
+ 0.6322313547134399,
+ 0.3217654228210449,
+ 0.3056824803352356
+ ],
+ [
+ 2.481761932373047,
+ 2.082732677459717,
+ 0.3391062617301941,
+ 0.36477789282798767,
+ 0.6064421534538269,
+ 0.40755993127822876,
+ 0.42346909642219543,
+ 0.42793068289756775,
+ 2.098849296569824,
+ 1.4124678373336792,
+ 0.20079860091209412,
+ 0.4150101840496063,
+ 0.3217654228210449,
+ 0.7164310812950134
+ ],
+ [
+ 0.38608792424201965,
+ 0.06112862005829811,
+ 4.514835357666016,
+ 0.17747923731803894,
+ 0.6668068170547485,
+ 0.25735294818878174,
+ 0.608350396156311,
+ 0.5113852620124817,
+ 0.9974663853645325,
+ 0.17575742304325104,
+ 1.457676887512207,
+ 0.8584281802177429,
+ 0.5207027196884155,
+ 0.9190161228179932
+ ],
+ [
+ 2.526141405105591,
+ 2.552089214324951,
+ 0.3158300220966339,
+ 0.5452418327331543,
+ 0.6531964540481567,
+ 0.32016801834106445,
+ 0.33874937891960144,
+ 0.5113852620124817,
+ 0.5153290033340454,
+ 1.7673344612121582,
+ 0.21370938420295715,
+ 0.7253791689872742,
+ 0.6070632934570312,
+ 1.408609390258789
+ ],
+ [
+ 0.1469775140285492,
+ 0.9671822190284729,
+ 0.5738509297370911,
+ 3.1964995861053467,
+ 0.715184211730957,
+ 0.5049390196800232,
+ 0.4435732364654541,
+ 1.184740662574768,
+ 0.4513735771179199,
+ 0.6549715399742126,
+ 0.847699761390686,
+ 0.3129248023033142,
+ 1.1396077871322632,
+ 0.28669044375419617
+ ],
+ [
+ 0.308875173330307,
+ 0.27397215366363525,
+ 0.26758623123168945,
+ 0.21993105113506317,
+ 0.6934841871261597,
+ 2.6876256465911865,
+ 1.8984736204147339,
+ 0.5884084701538086,
+ 0.4838624894618988,
+ 2.2110965251922607,
+ 0.1877189427614212,
+ 0.4623934328556061,
+ 1.6869375705718994,
+ 0.37823686003685
+ ],
+ [
+ 2.093987226486206,
+ 0.9955323338508606,
+ 4.661231994628906,
+ 2.9973270893096924,
+ 2.052233934402466,
+ 3.7314794063568115,
+ 3.0752739906311035,
+ 1.006408929824829,
+ 1.4387359619140625,
+ 0.41019687056541443,
+ 2.3213086128234863,
+ 1.5498850345611572,
+ 1.0639537572860718,
+ 2.0300800800323486
+ ],
+ [
+ 2.4958434104919434,
+ 2.686972141265869,
+ 0.5461810827255249,
+ 0.36477789282798767,
+ 0.7022207975387573,
+ 0.2787327170372009,
+ 0.4826081097126007,
+ 0.6599210500717163,
+ 0.38304662704467773,
+ 1.445213794708252,
+ 0.2996985912322998,
+ 0.6125134229660034,
+ 0.5207027196884155,
+ 1.7403335571289062
+ ],
+ [
+ 0.01867312751710415,
+ 0.020794173702597618,
+ 0.9424756765365601,
+ 0.08675385266542435,
+ 0.3009292185306549,
+ 1.9473083019256592,
+ 1.2562408447265625,
+ 0.0,
+ 0.09868758171796799,
+ 1.4701135158538818,
+ 0.05410424619913101,
+ 0.43898242712020874,
+ 0.6475988030433655,
+ 0.09726960211992264
+ ],
+ [
+ 0.0442432202398777,
+ 0.05319083109498024,
+ 0.13569219410419464,
+ 0.2506275475025177,
+ 0.6978620290756226,
+ 0.39821842312812805,
+ 0.4435732364654541,
+ 4.1338725090026855,
+ 1.8389281034469604,
+ 2.1474666595458984,
+ 0.11255282163619995,
+ 0.6891544461250305,
+ 0.4745543897151947,
+ 0.34261754155158997
+ ],
+ [
+ 2.1682519912719727,
+ 2.490644931793213,
+ 0.26758623123168945,
+ 0.41731157898902893,
+ 0.48441606760025024,
+ 0.17870478332042694,
+ 0.501568615436554,
+ 0.5113852620124817,
+ 0.41779354214668274,
+ 1.6109154224395752,
+ 0.13361932337284088,
+ 0.4623934328556061,
+ 0.3217654228210449,
+ 1.402345895767212
+ ],
+ [
+ 0.1566934883594513,
+ 1.2940410375595093,
+ 0.24256250262260437,
+ 2.934649705886841,
+ 1.1723496913909912,
+ 0.2892536222934723,
+ 1.6219455003738403,
+ 0.42793068289756775,
+ 0.575438916683197,
+ 0.25863152742385864,
+ 1.348387598991394,
+ 1.1759603023529053,
+ 0.6865549683570862,
+ 0.37823686003685
+ ],
+ [
+ 1.597996473312378,
+ 2.2340400218963623,
+ 0.17711839079856873,
+ 0.3281567096710205,
+ 0.47354185581207275,
+ 0.36965861916542053,
+ 0.14416414499282837,
+ 0.0,
+ 0.4513735771179199,
+ 0.12985575199127197,
+ 1.590766429901123,
+ 0.5076327323913574,
+ 0.37533119320869446,
+ 2.158923864364624
+ ],
+ [
+ 0.07268733531236649,
+ 0.08457080274820328,
+ 0.47847554087638855,
+ 2.582850217819214,
+ 0.5723628997802734,
+ 0.5381233096122742,
+ 0.40295252203941345,
+ 0.33687201142311096,
+ 1.5099552869796753,
+ 0.08356906473636627,
+ 1.4219192266464233,
+ 0.8584281802177429,
+ 0.14134716987609863,
+ 0.2274492233991623
+ ],
+ [
+ 0.11052214354276657,
+ 0.5066808462142944,
+ 0.24256250262260437,
+ 1.5088554620742798,
+ 0.4789937138557434,
+ 0.19032607972621918,
+ 0.24618223309516907,
+ 0.42793068289756775,
+ 0.14460423588752747,
+ 0.281314492225647,
+ 0.452240914106369,
+ 0.36526960134506226,
+ 0.5648149847984314,
+ 0.20689572393894196
+ ],
+ [
+ 0.0621153749525547,
+ 0.05716760456562042,
+ 0.09247544407844543,
+ 0.21993105113506317,
+ 0.5005088448524475,
+ 0.36965861916542053,
+ 0.5563841462135315,
+ 4.4121575355529785,
+ 1.053261160850525,
+ 2.0723049640655518,
+ 0.11255282163619995,
+ 0.8427334427833557,
+ 0.4261729419231415,
+ 0.20689572393894196
+ ],
+ [
+ 2.1916897296905518,
+ 2.5725083351135254,
+ 0.48843297362327576,
+ 0.6035509705543518,
+ 1.0356773138046265,
+ 0.7101240754127502,
+ 0.8340471982955933,
+ 0.33687201142311096,
+ 0.575438916683197,
+ 1.3163284063339233,
+ 0.3055667579174042,
+ 1.153027892112732,
+ 0.6475988030433655,
+ 1.6707500219345093
+ ],
+ [
+ 1.4635138511657715,
+ 2.4941208362579346,
+ 0.41655433177948,
+ 0.29014328122138977,
+ 0.5110952258110046,
+ 0.06754773110151291,
+ 0.2701396346092224,
+ 0.5113852620124817,
+ 1.2195290327072144,
+ 0.1140747219324112,
+ 1.7253267765045166,
+ 0.5923988223075867,
+ 0.3217654228210449,
+ 1.838428258895874
+ ],
+ [
+ 0.09006262570619583,
+ 0.04518952965736389,
+ 0.26758623123168945,
+ 0.16657860577106476,
+ 0.547289252281189,
+ 2.063861131668091,
+ 2.0728323459625244,
+ 0.23668336868286133,
+ 0.38304662704467773,
+ 1.7696998119354248,
+ 0.14054401218891144,
+ 0.4623934328556061,
+ 1.089812994003296,
+ 0.24758876860141754
+ ],
+ [
+ 2.810481548309326,
+ 2.9481277465820312,
+ 0.4479941427707672,
+ 0.4508608877658844,
+ 0.5773030519485474,
+ 0.25735294818878174,
+ 0.3606116473674774,
+ 0.33687201142311096,
+ 2.224175214767456,
+ 1.3226815462112427,
+ 1.2712469100952148,
+ 0.8267884254455566,
+ 1.2542200088500977,
+ 1.1934809684753418
+ ],
+ [
+ 1.9716235399246216,
+ 0.07681763917207718,
+ 4.799998760223389,
+ 0.3374384939670563,
+ 0.8093528151512146,
+ 0.33026403188705444,
+ 0.6415525078773499,
+ 0.42793068289756775,
+ 0.6593281030654907,
+ 2.2737770080566406,
+ 0.20079860091209412,
+ 1.0297449827194214,
+ 1.8897184133529663,
+ 0.3056824803352356
+ ],
+ [
+ 0.029711894690990448,
+ 0.04919818416237831,
+ 0.10708953440189362,
+ 0.23026826977729797,
+ 0.5523545145988464,
+ 0.26809996366500854,
+ 0.3606116473674774,
+ 4.42981481552124,
+ 2.0399248600006104,
+ 1.8921664953231812,
+ 0.17446593940258026,
+ 0.33943966031074524,
+ 0.2651671767234802,
+ 0.1859109103679657
+ ],
+ [
+ 2.6280815601348877,
+ 2.779548406600952,
+ 0.39503154158592224,
+ 0.5224685072898865,
+ 0.6757785677909851,
+ 0.26809996366500854,
+ 0.24618223309516907,
+ 0.23668336868286133,
+ 0.7115833163261414,
+ 0.09624312072992325,
+ 2.5551652908325195,
+ 0.8427334427833557,
+ 0.6070632934570312,
+ 1.4453884363174438
+ ],
+ [
+ 0.06917575001716614,
+ 0.07681763917207718,
+ 0.6095919013023376,
+ 0.27058058977127075,
+ 0.5319376587867737,
+ 1.7174378633499146,
+ 2.3881847858428955,
+ 0.23668336868286133,
+ 0.3470487594604492,
+ 2.215644598007202,
+ 0.1610349416732788,
+ 0.6515679955482483,
+ 0.7950698137283325,
+ 0.20689572393894196
+ ],
+ [
+ 2.2814605236053467,
+ 1.9918087720870972,
+ 0.27986711263656616,
+ 0.40010517835617065,
+ 0.7775944471359253,
+ 1.1211847066879272,
+ 0.24618223309516907,
+ 0.42793068289756775,
+ 2.5184507369995117,
+ 0.846372663974762,
+ 0.25147074460983276,
+ 0.48526886105537415,
+ 0.4745543897151947,
+ 0.5244699120521545
+ ],
+ [
+ 0.11052214354276657,
+ 0.08457080274820328,
+ 0.1214931309223175,
+ 0.3374384939670563,
+ 0.611217200756073,
+ 0.36965861916542053,
+ 0.3163984417915344,
+ 4.553856372833252,
+ 0.41779354214668274,
+ 1.952987551689148,
+ 0.13361932337284088,
+ 0.5923988223075867,
+ 0.4745543897151947,
+ 0.324320524930954
+ ],
+ [
+ 0.04784329608082771,
+ 0.08070173114538193,
+ 0.16349944472312927,
+ 0.2506275475025177,
+ 0.611217200756073,
+ 2.4528965950012207,
+ 2.4753758907318115,
+ 0.5884084701538086,
+ 0.38304662704467773,
+ 2.227245569229126,
+ 0.14742107689380646,
+ 0.48526886105537415,
+ 1.2966234683990479,
+ 0.20689572393894196
+ ],
+ [
+ 1.9752908945083618,
+ 2.6754610538482666,
+ 0.3730353116989136,
+ 0.4672226905822754,
+ 0.6347562074661255,
+ 0.21317119896411896,
+ 0.33874937891960144,
+ 0.5884084701538086,
+ 3.0721702575683594,
+ 0.07073231786489487,
+ 2.0260398387908936,
+ 0.7941148281097412,
+ 1.7282241582870483,
+ 0.44588133692741394
+ ],
+ [
+ 2.1963119506835938,
+ 0.04518952965736389,
+ 2.6013681888580322,
+ 0.29978296160697937,
+ 0.5215707421302795,
+ 0.3501552939414978,
+ 0.608350396156311,
+ 0.0,
+ 0.4838624894618988,
+ 0.281314492225647,
+ 1.4084162712097168,
+ 0.5295073390007019,
+ 0.14134716987609863,
+ 1.7884501218795776
+ ],
+ [
+ 2.416325330734253,
+ 0.08842495083808899,
+ 4.5447821617126465,
+ 0.3281567096710205,
+ 0.5919787287712097,
+ 0.26809996366500854,
+ 0.46328112483024597,
+ 0.42793068289756775,
+ 0.4513735771179199,
+ 1.1100139617919922,
+ 0.29379579424858093,
+ 0.6125134229660034,
+ 0.828773558139801,
+ 1.389699935913086
+ ],
+ [
+ 2.212942361831665,
+ 0.11873482167720795,
+ 4.277134418487549,
+ 0.38259753584861755,
+ 0.801507294178009,
+ 0.39821842312812805,
+ 0.46328112483024597,
+ 0.8480972647666931,
+ 0.4513735771179199,
+ 2.1133482456207275,
+ 0.16777299344539642,
+ 0.6515679955482483,
+ 0.7240503430366516,
+ 1.2752164602279663
+ ],
+ [
+ 0.0935017466545105,
+ 1.2801190614700317,
+ 0.45825818181037903,
+ 1.5645431280136108,
+ 2.8451359272003174,
+ 4.332710266113281,
+ 0.5563841462135315,
+ 0.903699517250061,
+ 0.7851776480674744,
+ 0.6621487140655518,
+ 1.3338468074798584,
+ 0.5923988223075867,
+ 0.8929534554481506,
+ 0.4937715530395508
+ ],
+ [
+ 0.04063013568520546,
+ 0.04919818416237831,
+ 0.22981183230876923,
+ 0.21993105113506317,
+ 0.5773030519485474,
+ 2.65574049949646,
+ 2.1800477504730225,
+ 0.23668336868286133,
+ 0.3470487594604492,
+ 2.052915334701538,
+ 0.11255282163619995,
+ 0.33943966031074524,
+ 1.0101385116577148,
+ 0.37823686003685
+ ],
+ [
+ 1.7104036808013916,
+ 2.762183904647827,
+ 0.47847554087638855,
+ 0.41731157898902893,
+ 0.6668068170547485,
+ 0.3402591049671173,
+ 0.1964735984802246,
+ 1.006408929824829,
+ 1.8053910732269287,
+ 0.12985575199127197,
+ 1.9235738515853882,
+ 0.3904491066932678,
+ 0.4261729419231415,
+ 2.238250732421875
+ ],
+ [
+ 2.2409956455230713,
+ 0.13355207443237305,
+ 4.641807556152344,
+ 0.4258050322532654,
+ 0.7530957460403442,
+ 0.35995450615882874,
+ 0.5740079879760742,
+ 0.5884084701538086,
+ 0.6593281030654907,
+ 0.12287268787622452,
+ 3.2246177196502686,
+ 2.474485397338867,
+ 1.4502606391906738,
+ 0.5688196420669556
+ ],
+ [
+ 0.14371776580810547,
+ 0.5632820129394531,
+ 0.2919990122318268,
+ 2.4298789501190186,
+ 0.715184211730957,
+ 0.32016801834106445,
+ 0.33874937891960144,
+ 0.23668336868286133,
+ 0.4513735771179199,
+ 0.0872066468000412,
+ 1.4103564023971558,
+ 0.5718713402748108,
+ 0.4261729419231415,
+ 0.42939433455467224
+ ],
+ [
+ 0.944672167301178,
+ 0.11124304682016373,
+ 3.7044196128845215,
+ 0.20948584377765656,
+ 0.6064421534538269,
+ 0.20181387662887573,
+ 0.42346909642219543,
+ 0.5884084701538086,
+ 0.27091550827026367,
+ 0.2401072382926941,
+ 1.3525035381317139,
+ 0.6125134229660034,
+ 0.3217654228210449,
+ 1.145911455154419
+ ],
+ [
+ 2.285686731338501,
+ 2.541224956512451,
+ 0.26758623123168945,
+ 0.41731157898902893,
+ 0.7108817100524902,
+ 0.6389449238777161,
+ 0.5563841462135315,
+ 0.5884084701538086,
+ 0.575438916683197,
+ 2.00553297996521,
+ 0.11255282163619995,
+ 0.7074308395385742,
+ 0.5207027196884155,
+ 1.537325143814087
+ ],
+ [
+ 2.0618979930877686,
+ 0.11124304682016373,
+ 3.79734468460083,
+ 0.26065385341644287,
+ 0.6254069805145264,
+ 0.2892536222934723,
+ 0.4435732364654541,
+ 0.42793068289756775,
+ 0.6593281030654907,
+ 1.6124964952468872,
+ 0.16777299344539642,
+ 0.48526886105537415,
+ 0.4745543897151947,
+ 0.939187228679657
+ ],
+ [
+ 1.7530975341796875,
+ 0.06112862005829811,
+ 0.13569219410419464,
+ 0.15555782616138458,
+ 0.48980912566185,
+ 0.3501552939414978,
+ 0.293536514043808,
+ 0.42793068289756775,
+ 0.38304662704467773,
+ 1.6085389852523804,
+ 0.05410424619913101,
+ 0.36526960134506226,
+ 0.5648149847984314,
+ 0.47806161642074585
+ ],
+ [
+ 0.0656517967581749,
+ 0.04919818416237831,
+ 0.5367850661277771,
+ 0.16657860577106476,
+ 0.6757785677909851,
+ 2.9785003662109375,
+ 2.6926350593566895,
+ 0.6599210500717163,
+ 0.4513735771179199,
+ 2.3281359672546387,
+ 0.1610349416732788,
+ 0.6515679955482483,
+ 1.3953752517700195,
+ 0.3056824803352356
+ ],
+ [
+ 0.0621153749525547,
+ 0.028992263600230217,
+ 0.14969247579574585,
+ 0.16657860577106476,
+ 0.48441606760025024,
+ 2.646865129470825,
+ 2.581716299057007,
+ 0.5884084701538086,
+ 0.3097064793109894,
+ 2.2340571880340576,
+ 0.14054401218891144,
+ 0.4150101840496063,
+ 1.5022896528244019,
+ 0.3056824803352356
+ ],
+ [
+ 0.1003447026014328,
+ 0.9607715606689453,
+ 0.13569219410419464,
+ 1.4679425954818726,
+ 0.4173230230808258,
+ 0.25735294818878174,
+ 0.14416414499282837,
+ 0.0,
+ 0.4838624894618988,
+ 0.5463835597038269,
+ 0.20727482438087463,
+ 0.25768792629241943,
+ 0.2051723748445511,
+ 0.1859109103679657
+ ],
+ [
+ 2.095379590988159,
+ 2.131772994995117,
+ 0.32753586769104004,
+ 0.3281567096710205,
+ 0.5319376587867737,
+ 0.20181387662887573,
+ 0.14416414499282837,
+ 0.5113852620124817,
+ 1.7347418069839478,
+ 0.27831965684890747,
+ 0.7209073305130005,
+ 0.5076327323913574,
+ 1.3570367097854614,
+ 0.47806161642074585
+ ],
+ [
+ 0.09006262570619583,
+ 0.04518952965736389,
+ 0.17711839079856873,
+ 0.31878796219825745,
+ 0.748953640460968,
+ 3.812486171722412,
+ 2.927367925643921,
+ 0.5113852620124817,
+ 0.6593281030654907,
+ 3.019174575805664,
+ 0.20727482438087463,
+ 0.8267884254455566,
+ 1.8897184133529663,
+ 0.37823686003685
+ ],
+ [
+ 2.0604584217071533,
+ 2.5051631927490234,
+ 0.3158300220966339,
+ 0.434226930141449,
+ 0.715184211730957,
+ 0.3887888491153717,
+ 0.42346909642219543,
+ 0.23668336868286133,
+ 0.575438916683197,
+ 1.8435229063034058,
+ 0.10543086379766464,
+ 0.6891544461250305,
+ 0.14134716987609863,
+ 1.0523498058319092
+ ],
+ [
+ 0.04063013568520546,
+ 0.07291840016841888,
+ 0.06259050220251083,
+ 0.19893039762973785,
+ 0.5370810627937317,
+ 2.6586813926696777,
+ 2.717519998550415,
+ 0.5113852620124817,
+ 0.3470487594604492,
+ 2.2991416454315186,
+ 0.16777299344539642,
+ 0.5718713402748108,
+ 1.7282241582870483,
+ 0.3056824803352356
+ ],
+ [
+ 2.308804512023926,
+ 0.7343912720680237,
+ 4.552301406860352,
+ 0.31878796219825745,
+ 0.8093528151512146,
+ 0.47061556577682495,
+ 0.5384441018104553,
+ 0.12532778084278107,
+ 0.6593281030654907,
+ 0.2662498354911804,
+ 1.0585801601409912,
+ 2.27670955657959,
+ 1.46790611743927,
+ 0.6520216464996338
+ ],
+ [
+ 2.884779453277588,
+ 2.644896984100342,
+ 4.246404647827148,
+ 0.4425784945487976,
+ 0.6934841871261597,
+ 0.3501552939414978,
+ 0.6893759369850159,
+ 0.7892204523086548,
+ 0.5153290033340454,
+ 0.540631890296936,
+ 2.3321621417999268,
+ 2.1395180225372314,
+ 2.675001859664917,
+ 0.37823686003685
+ ],
+ [
+ 1.9127122163772583,
+ 0.09989908337593079,
+ 0.3391062617301941,
+ 0.29014328122138977,
+ 0.7108817100524902,
+ 0.26809996366500854,
+ 0.1706608682870865,
+ 0.23668336868286133,
+ 0.18850478529930115,
+ 0.13851656019687653,
+ 1.4408982992172241,
+ 1.361644983291626,
+ 0.4261729419231415,
+ 1.55366051197052
+ ],
+ [
+ 0.055004797875881195,
+ 0.033066242933273315,
+ 0.22981183230876923,
+ 0.24049971997737885,
+ 0.5319376587867737,
+ 2.4790091514587402,
+ 2.1800477504730225,
+ 0.42793068289756775,
+ 0.38304662704467773,
+ 2.1979868412017822,
+ 0.061599891632795334,
+ 0.48526886105537415,
+ 1.5988694429397583,
+ 0.24758876860141754
+ ],
+ [
+ 2.326627254486084,
+ 2.473790407180786,
+ 0.3391062617301941,
+ 0.41731157898902893,
+ 0.6713027358055115,
+ 0.37926948070526123,
+ 0.42346909642219543,
+ 0.8480972647666931,
+ 0.41779354214668274,
+ 1.8378639221191406,
+ 0.22010280191898346,
+ 0.810585081577301,
+ 0.2651671767234802,
+ 1.1934809684753418
+ ],
+ [
+ 0.08661163598299026,
+ 0.09989908337593079,
+ 0.17711839079856873,
+ 0.3374384939670563,
+ 0.6159695386886597,
+ 3.256080389022827,
+ 3.120084285736084,
+ 0.8480972647666931,
+ 0.7612429857254028,
+ 2.5646743774414062,
+ 0.19428016245365143,
+ 0.5923988223075867,
+ 1.7282241582870483,
+ 0.28669044375419617
+ ],
+ [
+ 1.693218469619751,
+ 0.08842495083808899,
+ 4.61386775970459,
+ 0.3466349244117737,
+ 0.6890870928764343,
+ 0.3887888491153717,
+ 0.7350162267684937,
+ 0.7892204523086548,
+ 0.4838624894618988,
+ 0.8665614724159241,
+ 0.6193638443946838,
+ 0.5509136915206909,
+ 1.232324481010437,
+ 1.112892985343933
+ ],
+ [
+ 0.14371776580810547,
+ 0.129868283867836,
+ 0.7861500978469849,
+ 0.4087453782558441,
+ 0.9910730123519897,
+ 3.182326555252075,
+ 2.667114734649658,
+ 4.304543495178223,
+ 0.917873740196228,
+ 2.7919304370880127,
+ 0.3287010192871094,
+ 0.7773687243461609,
+ 1.3570367097854614,
+ 0.6386265158653259
+ ],
+ [
+ 1.939083456993103,
+ 0.03712369129061699,
+ 0.14969247579574585,
+ 0.19893039762973785,
+ 0.715184211730957,
+ 0.20181387662887573,
+ 0.2701396346092224,
+ 0.42793068289756775,
+ 0.6041910648345947,
+ 0.1889629364013672,
+ 1.3808506727218628,
+ 0.5076327323913574,
+ 0.2651671767234802,
+ 1.2462615966796875
+ ],
+ [
+ 2.4422295093536377,
+ 2.372903823852539,
+ 0.32753586769104004,
+ 0.45907527208328247,
+ 0.680234432220459,
+ 0.7893473505973816,
+ 0.3163984417915344,
+ 0.5884084701538086,
+ 0.3097064793109894,
+ 1.6826521158218384,
+ 0.18111440539360046,
+ 0.6125134229660034,
+ 0.5207027196884155,
+ 1.408609390258789
+ ],
+ [
+ 0.16950300335884094,
+ 1.2186881303787231,
+ 0.5829066038131714,
+ 0.5069929361343384,
+ 0.611217200756073,
+ 3.5724925994873047,
+ 0.42346909642219543,
+ 0.33687201142311096,
+ 0.41779354214668274,
+ 2.3745434284210205,
+ 0.20079860091209412,
+ 0.7253791689872742,
+ 0.4745543897151947,
+ 0.42939433455467224
+ ],
+ [
+ 2.2810752391815186,
+ 0.13355207443237305,
+ 4.301712512969971,
+ 0.30933061242103577,
+ 0.7279817461967468,
+ 0.5623083114624023,
+ 0.6415525078773499,
+ 0.5113852620124817,
+ 0.4838624894618988,
+ 0.21486201882362366,
+ 2.7521936893463135,
+ 0.6891544461250305,
+ 2.564497947692871,
+ 0.3056824803352356
+ ],
+ [
+ 0.1071411594748497,
+ 1.4563939571380615,
+ 1.131144404411316,
+ 3.354408025741577,
+ 0.6577538251876831,
+ 1.165725588798523,
+ 0.5740079879760742,
+ 0.7266590595245361,
+ 0.3470487594604492,
+ 0.4738464057445526,
+ 1.6565269231796265,
+ 0.5295073390007019,
+ 0.4261729419231415,
+ 0.20689572393894196
+ ],
+ [
+ 1.091620922088623,
+ 2.5884222984313965,
+ 0.4058508574962616,
+ 0.3913896679878235,
+ 0.7896223664283752,
+ 0.7305260300636292,
+ 0.3606116473674774,
+ 0.42793068289756775,
+ 4.230941295623779,
+ 0.06703420728445053,
+ 2.061026096343994,
+ 1.5420552492141724,
+ 2.077430248260498,
+ 0.6520216464996338
+ ],
+ [
+ 0.11052214354276657,
+ 0.14452268183231354,
+ 0.07764460891485214,
+ 0.29978296160697937,
+ 0.680234432220459,
+ 0.22440098226070404,
+ 0.5740079879760742,
+ 4.584308624267578,
+ 1.5437489748001099,
+ 2.0690577030181885,
+ 0.15425117313861847,
+ 0.5923988223075867,
+ 0.6070632934570312,
+ 0.34261754155158997
+ ],
+ [
+ 0.17267990112304688,
+ 0.4055202305316925,
+ 0.555489718914032,
+ 2.048759937286377,
+ 0.6393982172012329,
+ 0.25735294818878174,
+ 0.6893759369850159,
+ 0.23668336868286133,
+ 0.3470487594604492,
+ 1.1197586059570312,
+ 0.18111440539360046,
+ 0.43898242712020874,
+ 0.37533119320869446,
+ 0.26733070611953735
+ ],
+ [
+ 0.04784329608082771,
+ 0.661207377910614,
+ 0.30398550629615784,
+ 2.4218673706054688,
+ 0.5267676115036011,
+ 0.2787327170372009,
+ 0.46328112483024597,
+ 0.23668336868286133,
+ 1.1724854707717896,
+ 0.4128221869468689,
+ 1.0391032695770264,
+ 0.5923988223075867,
+ 0.3217654228210449,
+ 0.20689572393894196
+ ],
+ [
+ 0.04784329608082771,
+ 0.033066242933273315,
+ 0.10708953440189362,
+ 0.17747923731803894,
+ 0.42308926582336426,
+ 2.316534996032715,
+ 2.247814655303955,
+ 0.23668336868286133,
+ 0.3470487594604492,
+ 2.1205034255981445,
+ 0.12664635479450226,
+ 0.48526886105537415,
+ 1.3570367097854614,
+ 0.28669044375419617
+ ],
+ [
+ 0.08661163598299026,
+ 0.10369472205638885,
+ 0.3730353116989136,
+ 0.2506275475025177,
+ 0.7322114706039429,
+ 2.682859420776367,
+ 2.567539691925049,
+ 0.5113852620124817,
+ 0.6041910648345947,
+ 2.2756104469299316,
+ 0.2698298394680023,
+ 0.6705377697944641,
+ 1.6584346294403076,
+ 0.324320524930954
+ ],
+ [
+ 0.18215049803256989,
+ 1.0139955282211304,
+ 0.30398550629615784,
+ 2.855194091796875,
+ 0.765420138835907,
+ 1.13923978805542,
+ 0.7497782707214355,
+ 0.23668336868286133,
+ 0.575438916683197,
+ 0.3165699243545532,
+ 1.582641839981079,
+ 0.5295073390007019,
+ 0.2651671767234802,
+ 0.24758876860141754
+ ],
+ [
+ 2.531553030014038,
+ 2.5154058933258057,
+ 0.3730353116989136,
+ 0.4753043055534363,
+ 0.6486181616783142,
+ 0.26809996366500854,
+ 0.5563841462135315,
+ 0.7266590595245361,
+ 0.575438916683197,
+ 2.0866658687591553,
+ 0.10543086379766464,
+ 0.3904491066932678,
+ 0.7601904273033142,
+ 1.1539984941482544
+ ],
+ [
+ 0.1003447026014328,
+ 0.882196307182312,
+ 0.26758623123168945,
+ 2.3565807342529297,
+ 0.644018828868866,
+ 0.5702417492866516,
+ 0.33874937891960144,
+ 0.12532778084278107,
+ 0.230558842420578,
+ 0.5172886252403259,
+ 0.37342333793640137,
+ 0.5718713402748108,
+ 0.2651671767234802,
+ 0.24758876860141754
+ ],
+ [
+ 2.0972328186035156,
+ 0.06507401168346405,
+ 3.953927993774414,
+ 0.24049971997737885,
+ 0.6978620290756226,
+ 0.2892536222934723,
+ 2.0125110149383545,
+ 0.6599210500717163,
+ 0.9781542420387268,
+ 0.28876256942749023,
+ 1.7477080821990967,
+ 0.6125134229660034,
+ 0.5207027196884155,
+ 2.1735587120056152
+ ],
+ [
+ 2.2624142169952393,
+ 2.3547210693359375,
+ 0.5918810963630676,
+ 0.40010517835617065,
+ 0.6890870928764343,
+ 3.0009961128234863,
+ 2.9405641555786133,
+ 0.7892204523086548,
+ 0.6321395635604858,
+ 2.7414984703063965,
+ 0.14742107689380646,
+ 1.0297449827194214,
+ 1.9454909563064575,
+ 1.5482449531555176
+ ],
+ [
+ 2.5669000148773193,
+ 2.3693721294403076,
+ 0.3391062617301941,
+ 0.3374384939670563,
+ 0.587110698223114,
+ 0.22440098226070404,
+ 0.42346909642219543,
+ 0.42793068289756775,
+ 0.4838624894618988,
+ 1.3751137256622314,
+ 0.13361932337284088,
+ 0.5295073390007019,
+ 0.37533119320869446,
+ 0.8001567125320435
+ ],
+ [
+ 0.09692908078432083,
+ 0.11499594897031784,
+ 2.289001941680908,
+ 0.30933061242103577,
+ 0.7194682955741882,
+ 3.462653636932373,
+ 2.6989145278930664,
+ 0.5884084701538086,
+ 0.18850478529930115,
+ 2.230018377304077,
+ 0.14054401218891144,
+ 0.6515679955482483,
+ 0.3217654228210449,
+ 0.39558178186416626
+ ],
+ [
+ 1.0607808828353882,
+ 0.08457080274820328,
+ 4.235246658325195,
+ 0.11021846532821655,
+ 0.6890870928764343,
+ 0.4259851574897766,
+ 0.3163984417915344,
+ 0.33687201142311096,
+ 0.38304662704467773,
+ 0.7463622689247131,
+ 0.562183141708374,
+ 1.4076440334320068,
+ 0.9235620498657227,
+ 0.703876793384552
+ ],
+ [
+ 0.05856640636920929,
+ 0.04919818416237831,
+ 0.06259050220251083,
+ 0.18826234340667725,
+ 0.5919787287712097,
+ 0.24648915231227875,
+ 0.4826081097126007,
+ 4.336490154266357,
+ 0.6041910648345947,
+ 1.7767627239227295,
+ 0.10543086379766464,
+ 0.33943966031074524,
+ 0.4745543897151947,
+ 0.26733070611953735
+ ],
+ [
+ 1.2017440795898438,
+ 0.09226430952548981,
+ 4.036803245544434,
+ 0.24049971997737885,
+ 0.6846705675125122,
+ 0.24648915231227875,
+ 0.5913266539573669,
+ 0.7266590595245361,
+ 0.575438916683197,
+ 0.2707931101322174,
+ 0.9872051477432251,
+ 1.4254671335220337,
+ 0.7240503430366516,
+ 1.8465204238891602
+ ],
+ [
+ 2.9511818885803223,
+ 2.5504508018493652,
+ 0.9915869235992432,
+ 2.6654303073883057,
+ 3.121152400970459,
+ 4.607342720031738,
+ 1.403755784034729,
+ 0.903699517250061,
+ 1.1231189966201782,
+ 1.7444063425064087,
+ 1.7009845972061157,
+ 1.0427800416946411,
+ 1.1396077871322632,
+ 1.389699935913086
+ ],
+ [
+ 0.11389173567295074,
+ 0.08457080274820328,
+ 0.2919990122318268,
+ 0.3374384939670563,
+ 0.6347562074661255,
+ 0.36965861916542053,
+ 0.6736875176429749,
+ 5.2578444480896,
+ 1.8471394777297974,
+ 2.5983939170837402,
+ 0.20727482438087463,
+ 0.9040864706039429,
+ 0.9235620498657227,
+ 0.5092385411262512
+ ],
+ [
+ 1.9637194871902466,
+ 1.9779514074325562,
+ 0.17711839079856873,
+ 0.2804097831249237,
+ 0.3577543795108795,
+ 0.23550604283809662,
+ 0.22163675725460052,
+ 0.0,
+ 0.09868758171796799,
+ 0.8573588132858276,
+ 0.0837555006146431,
+ 0.5923988223075867,
+ 0.3217654228210449,
+ 0.7886165976524353
+ ],
+ [
+ 2.068112850189209,
+ 2.5194737911224365,
+ 0.24256250262260437,
+ 0.3281567096710205,
+ 0.55739426612854,
+ 0.2787327170372009,
+ 0.4435732364654541,
+ 0.42793068289756775,
+ 0.545835554599762,
+ 0.41805219650268555,
+ 0.8909805417060852,
+ 0.4150101840496063,
+ 0.4261729419231415,
+ 1.223981499671936
+ ],
+ [
+ 2.539318323135376,
+ 2.470242977142334,
+ 0.3391062617301941,
+ 0.3557475507259369,
+ 0.7322114706039429,
+ 0.5216687917709351,
+ 0.520176351070404,
+ 0.5884084701538086,
+ 0.41779354214668274,
+ 1.9999184608459473,
+ 0.0910329520702362,
+ 0.6891544461250305,
+ 0.37533119320869446,
+ 0.8001567125320435
+ ],
+ [
+ 2.747277021408081,
+ 2.3606865406036377,
+ 0.3730353116989136,
+ 0.4753043055534363,
+ 0.6577538251876831,
+ 0.309969037771225,
+ 0.3163984417915344,
+ 0.6599210500717163,
+ 0.3470487594604492,
+ 1.4244565963745117,
+ 0.20727482438087463,
+ 0.6322313547134399,
+ 0.4745543897151947,
+ 0.5973297953605652
+ ],
+ [
+ 0.0621153749525547,
+ 0.7262934446334839,
+ 0.22981183230876923,
+ 2.941507339477539,
+ 0.5319376587867737,
+ 0.23550604283809662,
+ 1.1924337148666382,
+ 0.42793068289756775,
+ 1.1398468017578125,
+ 0.2401072382926941,
+ 1.1754714250564575,
+ 0.7253791689872742,
+ 0.2651671767234802,
+ 0.2274492233991623
+ ],
+ [
+ 0.07268733531236649,
+ 0.6974231004714966,
+ 0.2551526427268982,
+ 2.3602540493011475,
+ 0.5370810627937317,
+ 0.32016801834106445,
+ 0.24618223309516907,
+ 0.33687201142311096,
+ 0.3470487594604492,
+ 0.2052282989025116,
+ 1.091111183166504,
+ 0.4150101840496063,
+ 0.37533119320869446,
+ 0.20689572393894196
+ ],
+ [
+ 2.110567808151245,
+ 2.66469407081604,
+ 0.38409391045570374,
+ 0.434226930141449,
+ 0.6668068170547485,
+ 0.2996649742126465,
+ 0.42346909642219543,
+ 0.33687201142311096,
+ 0.3097064793109894,
+ 1.6405377388000488,
+ 0.16777299344539642,
+ 0.48526886105537415,
+ 0.6475988030433655,
+ 1.4332784414291382
+ ],
+ [
+ 0.5725551247596741,
+ 0.14815296232700348,
+ 2.2195329666137695,
+ 0.2506275475025177,
+ 0.6159695386886597,
+ 0.21317119896411896,
+ 2.7216079235076904,
+ 0.6599210500717163,
+ 0.8313940167427063,
+ 0.4128221869468689,
+ 1.1876544952392578,
+ 1.22030508518219,
+ 1.1396077871322632,
+ 0.8879755139350891
+ ],
+ [
+ 2.071443557739258,
+ 0.12617088854312897,
+ 4.571188926696777,
+ 0.4087453782558441,
+ 0.7194682955741882,
+ 0.4440770447254181,
+ 0.4435732364654541,
+ 0.42793068289756775,
+ 0.4838624894618988,
+ 0.15900148451328278,
+ 1.9000961780548096,
+ 1.7850960493087769,
+ 2.1595382690429688,
+ 0.42939433455467224
+ ],
+ [
+ 2.2529513835906982,
+ 2.4336905479431152,
+ 0.2038121521472931,
+ 0.3281567096710205,
+ 0.6531964540481567,
+ 0.23550604283809662,
+ 0.2701396346092224,
+ 0.6599210500717163,
+ 0.6041910648345947,
+ 1.439595341682434,
+ 0.21370938420295715,
+ 0.2856876850128174,
+ 0.4261729419231415,
+ 1.9713728427886963
+ ],
+ [
+ 0.09006262570619583,
+ 0.919745922088623,
+ 0.32753586769104004,
+ 3.0654044151306152,
+ 0.6668068170547485,
+ 0.8328189849853516,
+ 0.38200613856315613,
+ 0.12532778084278107,
+ 1.3351269960403442,
+ 0.12287268787622452,
+ 2.1360068321228027,
+ 1.105522632598877,
+ 0.5207027196884155,
+ 0.26733070611953735
+ ],
+ [
+ 0.0621153749525547,
+ 0.07681763917207718,
+ 0.4684179425239563,
+ 0.2804097831249237,
+ 0.611217200756073,
+ 1.3805137872695923,
+ 2.421778678894043,
+ 0.33687201142311096,
+ 0.27091550827026367,
+ 2.0770320892333984,
+ 0.1610349416732788,
+ 0.6125134229660034,
+ 1.46790611743927,
+ 0.5092385411262512
+ ],
+ [
+ 2.211291790008545,
+ 2.363853693008423,
+ 0.36185306310653687,
+ 0.4672226905822754,
+ 0.611217200756073,
+ 0.2996649742126465,
+ 0.33874937891960144,
+ 0.12532778084278107,
+ 0.4513735771179199,
+ 1.3142015933990479,
+ 0.20727482438087463,
+ 0.48526886105537415,
+ 0.37533119320869446,
+ 1.1778745651245117
+ ],
+ [
+ 2.6253554821014404,
+ 2.6594138145446777,
+ 0.47847554087638855,
+ 0.4912741482257843,
+ 0.7108817100524902,
+ 0.4259851574897766,
+ 0.4826081097126007,
+ 0.33687201142311096,
+ 0.9584617614746094,
+ 0.12462302297353745,
+ 2.647615909576416,
+ 1.361644983291626,
+ 2.086893320083618,
+ 1.813751459121704
+ ],
+ [
+ 0.07268733531236649,
+ 0.06112862005829811,
+ 0.07764460891485214,
+ 0.2506275475025177,
+ 0.6159695386886597,
+ 2.361013889312744,
+ 2.4805808067321777,
+ 0.5884084701538086,
+ 0.4513735771179199,
+ 2.231295585632324,
+ 0.1610349416732788,
+ 0.4623934328556061,
+ 1.6140966415405273,
+ 0.2274492233991623
+ ],
+ [
+ 1.2675375938415527,
+ 0.09608898311853409,
+ 4.0764851570129395,
+ 0.20948584377765656,
+ 0.7975613474845886,
+ 0.32016801834106445,
+ 0.40295252203941345,
+ 0.5884084701538086,
+ 0.6593281030654907,
+ 0.1436772644519806,
+ 1.7587136030197144,
+ 1.003153681755066,
+ 1.432298183441162,
+ 1.4573534727096558
+ ],
+ [
+ 1.2219406366348267,
+ 0.08457080274820328,
+ 4.826618671417236,
+ 0.3374384939670563,
+ 0.6846705675125122,
+ 0.25735294818878174,
+ 0.4435732364654541,
+ 0.42793068289756775,
+ 0.575438916683197,
+ 0.7985090017318726,
+ 0.6363197565078735,
+ 0.4623934328556061,
+ 1.1150202751159668,
+ 0.8450298309326172
+ ],
+ [
+ 0.04784329608082771,
+ 0.033066242933273315,
+ 0.06259050220251083,
+ 0.2804097831249237,
+ 0.611217200756073,
+ 0.21317119896411896,
+ 0.520176351070404,
+ 4.629620552062988,
+ 0.8085528016090393,
+ 2.2871434688568115,
+ 0.11962441354990005,
+ 0.7603374719619751,
+ 0.7240503430366516,
+ 0.42939433455467224
+ ],
+ [
+ 0.12393291294574738,
+ 0.661207377910614,
+ 0.6269944906234741,
+ 2.5490241050720215,
+ 0.748953640460968,
+ 0.40755993127822876,
+ 0.3606116473674774,
+ 0.33687201142311096,
+ 0.8313940167427063,
+ 0.2052282989025116,
+ 0.9479739665985107,
+ 0.43898242712020874,
+ 0.3217654228210449,
+ 0.20689572393894196
+ ],
+ [
+ 0.03336457163095474,
+ 0.06507401168346405,
+ 0.41655433177948,
+ 0.30933061242103577,
+ 0.5523545145988464,
+ 2.6177186965942383,
+ 2.9600374698638916,
+ 0.7266590595245361,
+ 0.5153290033340454,
+ 2.4084339141845703,
+ 0.1610349416732788,
+ 0.7074308395385742,
+ 1.9879722595214844,
+ 0.3056824803352356
+ ],
+ [
+ 2.291802167892456,
+ 2.5031020641326904,
+ 0.36185306310653687,
+ 0.40010517835617065,
+ 0.7022207975387573,
+ 0.25735294818878174,
+ 0.4435732364654541,
+ 0.6599210500717163,
+ 2.310119152069092,
+ 1.1274864673614502,
+ 0.14742107689380646,
+ 0.5509136915206909,
+ 0.9235620498657227,
+ 0.6782853007316589
+ ],
+ [
+ 2.233752965927124,
+ 2.615215539932251,
+ 0.39503154158592224,
+ 0.4912741482257843,
+ 0.7406176328659058,
+ 0.8205901384353638,
+ 1.759113073348999,
+ 0.33687201142311096,
+ 0.3470487594604492,
+ 1.2986459732055664,
+ 0.6571182012557983,
+ 0.33943966031074524,
+ 0.4745543897151947,
+ 0.7288296818733215
+ ],
+ [
+ 2.193372964859009,
+ 0.11499594897031784,
+ 4.698983669281006,
+ 0.24049971997737885,
+ 0.680234432220459,
+ 0.5623083114624023,
+ 0.5740079879760742,
+ 0.33687201142311096,
+ 0.5153290033340454,
+ 2.1030023097991943,
+ 0.2698298394680023,
+ 0.9188540577888489,
+ 0.5648149847984314,
+ 1.6897841691970825
+ ],
+ [
+ 2.0157723426818848,
+ 2.284501314163208,
+ 0.24256250262260437,
+ 0.31878796219825745,
+ 0.567398190498352,
+ 0.2996649742126465,
+ 0.4435732364654541,
+ 0.5113852620124817,
+ 0.3470487594604492,
+ 0.09444234520196915,
+ 1.9408273696899414,
+ 1.342635989189148,
+ 0.4261729419231415,
+ 0.8339989185333252
+ ],
+ [
+ 2.2199275493621826,
+ 2.4541220664978027,
+ 0.2919990122318268,
+ 0.3466349244117737,
+ 0.6531964540481567,
+ 1.0649769306182861,
+ 0.5563841462135315,
+ 0.5884084701538086,
+ 0.6321395635604858,
+ 1.4877187013626099,
+ 0.1610349416732788,
+ 0.5076327323913574,
+ 0.2051723748445511,
+ 1.112892985343933
+ ],
+ [
+ 2.17897367477417,
+ 2.5215015411376953,
+ 0.3158300220966339,
+ 0.38259753584861755,
+ 0.547289252281189,
+ 0.2787327170372009,
+ 0.1706608682870865,
+ 0.7892204523086548,
+ 1.0350066423416138,
+ 0.08174530416727066,
+ 2.182743787765503,
+ 1.6812528371810913,
+ 1.1150202751159668,
+ 1.5697332620620728
+ ],
+ [
+ 1.7864923477172852,
+ 0.07291840016841888,
+ 0.4982922375202179,
+ 0.434226930141449,
+ 1.0990616083145142,
+ 0.6680273413658142,
+ 1.0067986249923706,
+ 0.8480972647666931,
+ 0.575438916683197,
+ 0.267766535282135,
+ 1.8784843683242798,
+ 0.9477507472038269,
+ 0.6865549683570862,
+ 1.7884501218795776
+ ],
+ [
+ 0.19153223931789398,
+ 0.08842495083808899,
+ 0.9855782985687256,
+ 0.29014328122138977,
+ 0.6978620290756226,
+ 3.0243561267852783,
+ 2.5507423877716064,
+ 0.5113852620124817,
+ 0.4838624894618988,
+ 2.3015241622924805,
+ 0.12664635479450226,
+ 0.5718713402748108,
+ 1.5834068059921265,
+ 0.34261754155158997
+ ],
+ [
+ 0.11389173567295074,
+ 0.08457080274820328,
+ 0.10708953440189362,
+ 0.20948584377765656,
+ 0.6064421534538269,
+ 2.7627146244049072,
+ 2.801926374435425,
+ 0.5113852620124817,
+ 0.4838624894618988,
+ 2.449798822402954,
+ 0.16777299344539642,
+ 0.5923988223075867,
+ 1.46790611743927,
+ 0.34261754155158997
+ ],
+ [
+ 0.19464004039764404,
+ 1.1505787372589111,
+ 0.21689648926258087,
+ 3.055696964263916,
+ 0.7935997247695923,
+ 0.7893473505973816,
+ 0.4826081097126007,
+ 0.23668336868286133,
+ 0.575438916683197,
+ 0.5934343338012695,
+ 1.321212649345398,
+ 0.7941148281097412,
+ 0.3217654228210449,
+ 0.24758876860141754
+ ],
+ [
+ 0.0621153749525547,
+ 0.09226430952548981,
+ 0.16349944472312927,
+ 0.48332110047340393,
+ 0.9477939605712891,
+ 2.7776005268096924,
+ 2.600311040878296,
+ 0.5113852620124817,
+ 0.7115833163261414,
+ 2.301722526550293,
+ 0.20079860091209412,
+ 0.8584281802177429,
+ 1.232324481010437,
+ 0.5688196420669556
+ ],
+ [
+ 0.037003953009843826,
+ 0.05716760456562042,
+ 0.2038121521472931,
+ 0.24049971997737885,
+ 0.7065606117248535,
+ 0.41681498289108276,
+ 0.5384441018104553,
+ 4.4314045906066895,
+ 0.545835554599762,
+ 2.1495463848114014,
+ 0.14742107689380646,
+ 0.3904491066932678,
+ 0.6475988030433655,
+ 0.3056824803352356
+ ],
+ [
+ 0.055004797875881195,
+ 0.07291840016841888,
+ 0.14969247579574585,
+ 0.21993105113506317,
+ 0.5110952258110046,
+ 2.0147647857666016,
+ 1.7320468425750732,
+ 0.5113852620124817,
+ 0.3470487594604492,
+ 1.7485517263412476,
+ 0.16777299344539642,
+ 0.3904491066932678,
+ 1.376389741897583,
+ 0.1425720900297165
+ ],
+ [
+ 2.1327028274536133,
+ 2.3531243801116943,
+ 0.2551526427268982,
+ 0.3557475507259369,
+ 0.5267676115036011,
+ 0.2892536222934723,
+ 0.3606116473674774,
+ 0.903699517250061,
+ 0.3470487594604492,
+ 1.7946984767913818,
+ 0.10543086379766464,
+ 0.48526886105537415,
+ 0.4745543897151947,
+ 1.6513465642929077
+ ],
+ [
+ 2.966824531555176,
+ 2.559267282485962,
+ 0.36185306310653687,
+ 0.3466349244117737,
+ 0.6064421534538269,
+ 0.4879245460033417,
+ 0.38200613856315613,
+ 0.5113852620124817,
+ 0.5153290033340454,
+ 1.6053615808486938,
+ 0.15425117313861847,
+ 0.5509136915206909,
+ 0.4745543897151947,
+ 1.784170389175415
+ ],
+ [
+ 0.06917575001716614,
+ 0.06112862005829811,
+ 0.22981183230876923,
+ 0.2804097831249237,
+ 0.6668068170547485,
+ 0.2892536222934723,
+ 0.520176351070404,
+ 3.6336705684661865,
+ 1.5976516008377075,
+ 2.173947334289551,
+ 0.18111440539360046,
+ 0.5076327323913574,
+ 0.37533119320869446,
+ 0.34261754155158997
+ ],
+ [
+ 0.1071411594748497,
+ 0.13355207443237305,
+ 0.13569219410419464,
+ 0.16657860577106476,
+ 0.6254069805145264,
+ 2.9206955432891846,
+ 2.8425040245056152,
+ 0.42793068289756775,
+ 0.5153290033340454,
+ 2.1497771739959717,
+ 0.16777299344539642,
+ 0.5295073390007019,
+ 1.842746615409851,
+ 0.39558178186416626
+ ],
+ [
+ 0.037003953009843826,
+ 0.9526999592781067,
+ 0.14969247579574585,
+ 2.7291104793548584,
+ 0.55739426612854,
+ 0.862752377986908,
+ 0.33874937891960144,
+ 0.33687201142311096,
+ 0.545835554599762,
+ 0.328050822019577,
+ 0.7286022901535034,
+ 0.36526960134506226,
+ 0.2051723748445511,
+ 0.26733070611953735
+ ],
+ [
+ 0.09692908078432083,
+ 0.041164740920066833,
+ 0.5367850661277771,
+ 0.23026826977729797,
+ 0.46806007623672485,
+ 0.26809996366500854,
+ 0.501568615436554,
+ 4.358943939208984,
+ 0.8537251353263855,
+ 1.9169405698776245,
+ 0.20079860091209412,
+ 1.9122755527496338,
+ 0.4745543897151947,
+ 0.324320524930954
+ ],
+ [
+ 2.07759952545166,
+ 0.08070173114538193,
+ 4.6022748947143555,
+ 0.18826234340667725,
+ 0.6713027358055115,
+ 0.24648915231227875,
+ 0.38200613856315613,
+ 0.5113852620124817,
+ 0.41779354214668274,
+ 1.5909372568130493,
+ 0.22010280191898346,
+ 0.6515679955482483,
+ 0.4261729419231415,
+ 1.8821457624435425
+ ],
+ [
+ 2.4461557865142822,
+ 2.4663262367248535,
+ 0.7254924774169922,
+ 2.9530584812164307,
+ 0.8882156610488892,
+ 0.7636306881904602,
+ 0.5913266539573669,
+ 0.5884084701538086,
+ 1.053261160850525,
+ 1.1559457778930664,
+ 1.3907829523086548,
+ 0.7074308395385742,
+ 0.4261729419231415,
+ 1.4513888359069824
+ ],
+ [
+ 1.8770097494125366,
+ 2.4819018840789795,
+ 0.41655433177948,
+ 0.3557475507259369,
+ 0.5370810627937317,
+ 0.24648915231227875,
+ 0.3606116473674774,
+ 0.5884084701538086,
+ 0.38304662704467773,
+ 0.06703420728445053,
+ 2.042656660079956,
+ 0.9895880222320557,
+ 1.793459177017212,
+ 1.112892985343933
+ ],
+ [
+ 0.055004797875881195,
+ 0.033066242933273315,
+ 0.1214931309223175,
+ 0.27058058977127075,
+ 0.620699405670166,
+ 2.3728086948394775,
+ 2.4622433185577393,
+ 0.5113852620124817,
+ 0.27091550827026367,
+ 2.1873738765716553,
+ 0.09825781732797623,
+ 0.5076327323913574,
+ 1.0374081134796143,
+ 0.1425720900297165
+ ],
+ [
+ 2.372657299041748,
+ 2.7669496536254883,
+ 0.4058508574962616,
+ 0.3557475507259369,
+ 0.6254069805145264,
+ 0.41681498289108276,
+ 0.6250892281532288,
+ 0.42793068289756775,
+ 0.4513735771179199,
+ 1.280094861984253,
+ 0.2698298394680023,
+ 0.6125134229660034,
+ 0.6475988030433655,
+ 1.6164573431015015
+ ],
+ [
+ 1.8752774000167847,
+ 1.987210988998413,
+ 0.22981183230876923,
+ 0.24049971997737885,
+ 0.48980912566185,
+ 0.5543115139007568,
+ 0.38200613856315613,
+ 0.33687201142311096,
+ 1.9181571006774902,
+ 1.2740185260772705,
+ 0.10543086379766464,
+ 0.43898242712020874,
+ 0.5207027196884155,
+ 0.44588133692741394
+ ],
+ [
+ 2.588939905166626,
+ 2.6006617546081543,
+ 0.38409391045570374,
+ 0.434226930141449,
+ 0.611217200756073,
+ 0.24648915231227875,
+ 0.6250892281532288,
+ 0.7892204523086548,
+ 0.4838624894618988,
+ 1.6826521158218384,
+ 0.20079860091209412,
+ 0.5076327323913574,
+ 0.2651671767234802,
+ 1.3833165168762207
+ ],
+ [
+ 2.1304664611816406,
+ 2.53157901763916,
+ 0.26758623123168945,
+ 0.3281567096710205,
+ 0.4789937138557434,
+ 0.5216687917709351,
+ 0.24618223309516907,
+ 0.42793068289756775,
+ 1.3213871717453003,
+ 0.11053374409675598,
+ 2.0498404502868652,
+ 0.7773687243461609,
+ 0.5648149847984314,
+ 1.5855517387390137
+ ],
+ [
+ 0.06917575001716614,
+ 0.041164740920066833,
+ 0.16349944472312927,
+ 0.18826234340667725,
+ 0.48980912566185,
+ 2.875234842300415,
+ 1.9701752662658691,
+ 0.23668336868286133,
+ 0.4838624894618988,
+ 2.1080684661865234,
+ 0.0910329520702362,
+ 0.5076327323913574,
+ 1.209938645362854,
+ 0.28669044375419617
+ ],
+ [
+ 0.02604582905769348,
+ 0.02490161918103695,
+ 0.17711839079856873,
+ 0.1444142460823059,
+ 0.611217200756073,
+ 0.5936712026596069,
+ 0.2701396346092224,
+ 4.365752220153809,
+ 0.575438916683197,
+ 1.47782564163208,
+ 0.12664635479450226,
+ 0.33943966031074524,
+ 0.2051723748445511,
+ 0.1425720900297165
+ ],
+ [
+ 0.0656517967581749,
+ 1.2456518411636353,
+ 0.07764460891485214,
+ 2.84546160697937,
+ 0.6531964540481567,
+ 0.7101240754127502,
+ 0.520176351070404,
+ 0.23668336868286133,
+ 0.38304662704467773,
+ 0.4022793471813202,
+ 0.9692894816398621,
+ 0.6125134229660034,
+ 0.3217654228210449,
+ 0.20689572393894196
+ ],
+ [
+ 0.22820307314395905,
+ 0.48363012075424194,
+ 0.677455723285675,
+ 2.433293104171753,
+ 0.6890870928764343,
+ 0.20181387662887573,
+ 0.293536514043808,
+ 0.12532778084278107,
+ 0.6857970356941223,
+ 0.10341396182775497,
+ 1.5409976243972778,
+ 0.2856876850128174,
+ 0.4261729419231415,
+ 0.1425720900297165
+ ]
+ ],
+ "rnaIndices": [
+ [
+ 181,
+ 273,
+ 40,
+ 166,
+ 292,
+ 151,
+ 216,
+ 227,
+ 255,
+ 59,
+ 261,
+ 132,
+ 119,
+ 252,
+ 87,
+ 208,
+ 157,
+ 10,
+ 153,
+ 82
+ ],
+ [
+ 3,
+ 204,
+ 193,
+ 6,
+ 76,
+ 43,
+ 143,
+ 125,
+ 96,
+ 16,
+ 161,
+ 169,
+ 295,
+ 72,
+ 68,
+ 78,
+ 106,
+ 290,
+ 272,
+ 8
+ ],
+ [
+ 211,
+ 203,
+ 235,
+ 34,
+ 221,
+ 189,
+ 223,
+ 129,
+ 197,
+ 179,
+ 138,
+ 37,
+ 133,
+ 162,
+ 212,
+ 268,
+ 101,
+ 54,
+ 176,
+ 236
+ ],
+ [
+ 204,
+ 1,
+ 143,
+ 76,
+ 43,
+ 193,
+ 295,
+ 16,
+ 6,
+ 96,
+ 68,
+ 161,
+ 125,
+ 48,
+ 106,
+ 169,
+ 8,
+ 115,
+ 72,
+ 110
+ ],
+ [
+ 39,
+ 105,
+ 168,
+ 233,
+ 86,
+ 254,
+ 278,
+ 118,
+ 137,
+ 88,
+ 42,
+ 102,
+ 13,
+ 134,
+ 131,
+ 128,
+ 71,
+ 259,
+ 74,
+ 247
+ ],
+ [
+ 22,
+ 239,
+ 18,
+ 15,
+ 298,
+ 131,
+ 118,
+ 74,
+ 39,
+ 62,
+ 105,
+ 148,
+ 123,
+ 259,
+ 42,
+ 91,
+ 63,
+ 185,
+ 182,
+ 86
+ ],
+ [
+ 8,
+ 78,
+ 43,
+ 1,
+ 143,
+ 161,
+ 125,
+ 169,
+ 262,
+ 79,
+ 110,
+ 204,
+ 72,
+ 3,
+ 272,
+ 106,
+ 48,
+ 96,
+ 104,
+ 76
+ ],
+ [
+ 183,
+ 246,
+ 141,
+ 66,
+ 240,
+ 275,
+ 199,
+ 154,
+ 20,
+ 95,
+ 136,
+ 49,
+ 188,
+ 58,
+ 135,
+ 44,
+ 65,
+ 36,
+ 100,
+ 23
+ ],
+ [
+ 6,
+ 262,
+ 110,
+ 169,
+ 72,
+ 161,
+ 43,
+ 272,
+ 78,
+ 143,
+ 79,
+ 106,
+ 104,
+ 214,
+ 204,
+ 100,
+ 68,
+ 48,
+ 125,
+ 3
+ ],
+ [
+ 133,
+ 291,
+ 212,
+ 263,
+ 37,
+ 260,
+ 197,
+ 296,
+ 41,
+ 138,
+ 179,
+ 268,
+ 54,
+ 77,
+ 101,
+ 236,
+ 203,
+ 211,
+ 221,
+ 2
+ ],
+ [
+ 208,
+ 294,
+ 92,
+ 293,
+ 119,
+ 251,
+ 227,
+ 261,
+ 26,
+ 98,
+ 132,
+ 151,
+ 200,
+ 178,
+ 190,
+ 157,
+ 81,
+ 282,
+ 149,
+ 152
+ ],
+ [
+ 70,
+ 33,
+ 139,
+ 35,
+ 108,
+ 180,
+ 280,
+ 171,
+ 266,
+ 192,
+ 245,
+ 30,
+ 69,
+ 186,
+ 50,
+ 232,
+ 297,
+ 170,
+ 225,
+ 248
+ ],
+ [
+ 270,
+ 130,
+ 84,
+ 156,
+ 218,
+ 116,
+ 95,
+ 229,
+ 31,
+ 275,
+ 281,
+ 279,
+ 141,
+ 219,
+ 58,
+ 246,
+ 44,
+ 114,
+ 65,
+ 135
+ ],
+ [
+ 128,
+ 55,
+ 160,
+ 102,
+ 111,
+ 278,
+ 17,
+ 182,
+ 88,
+ 233,
+ 145,
+ 121,
+ 253,
+ 86,
+ 112,
+ 234,
+ 267,
+ 206,
+ 38,
+ 184
+ ],
+ [
+ 46,
+ 115,
+ 75,
+ 198,
+ 231,
+ 48,
+ 204,
+ 67,
+ 3,
+ 73,
+ 16,
+ 68,
+ 1,
+ 214,
+ 295,
+ 193,
+ 143,
+ 72,
+ 76,
+ 8
+ ],
+ [
+ 123,
+ 239,
+ 118,
+ 18,
+ 259,
+ 62,
+ 5,
+ 131,
+ 91,
+ 185,
+ 39,
+ 22,
+ 298,
+ 42,
+ 148,
+ 74,
+ 86,
+ 168,
+ 233,
+ 102
+ ],
+ [
+ 188,
+ 68,
+ 73,
+ 169,
+ 3,
+ 1,
+ 204,
+ 66,
+ 43,
+ 193,
+ 246,
+ 143,
+ 72,
+ 96,
+ 76,
+ 6,
+ 8,
+ 48,
+ 125,
+ 115
+ ],
+ [
+ 145,
+ 55,
+ 13,
+ 160,
+ 128,
+ 278,
+ 253,
+ 111,
+ 74,
+ 86,
+ 206,
+ 137,
+ 102,
+ 88,
+ 112,
+ 38,
+ 60,
+ 234,
+ 182,
+ 121
+ ],
+ [
+ 131,
+ 62,
+ 239,
+ 185,
+ 123,
+ 91,
+ 259,
+ 15,
+ 118,
+ 39,
+ 5,
+ 74,
+ 278,
+ 182,
+ 298,
+ 124,
+ 60,
+ 102,
+ 137,
+ 109
+ ],
+ [
+ 44,
+ 61,
+ 199,
+ 135,
+ 127,
+ 226,
+ 154,
+ 45,
+ 32,
+ 275,
+ 207,
+ 240,
+ 36,
+ 95,
+ 120,
+ 23,
+ 65,
+ 58,
+ 20,
+ 24
+ ],
+ [
+ 240,
+ 199,
+ 49,
+ 23,
+ 275,
+ 226,
+ 154,
+ 61,
+ 183,
+ 44,
+ 36,
+ 136,
+ 65,
+ 135,
+ 7,
+ 32,
+ 58,
+ 127,
+ 207,
+ 220
+ ],
+ [
+ 208,
+ 92,
+ 251,
+ 10,
+ 26,
+ 227,
+ 152,
+ 294,
+ 174,
+ 178,
+ 94,
+ 190,
+ 282,
+ 293,
+ 258,
+ 200,
+ 195,
+ 261,
+ 222,
+ 151
+ ],
+ [
+ 5,
+ 42,
+ 298,
+ 239,
+ 148,
+ 15,
+ 118,
+ 74,
+ 105,
+ 39,
+ 131,
+ 18,
+ 123,
+ 259,
+ 62,
+ 63,
+ 168,
+ 86,
+ 254,
+ 91
+ ],
+ [
+ 61,
+ 127,
+ 58,
+ 226,
+ 154,
+ 207,
+ 135,
+ 49,
+ 53,
+ 36,
+ 275,
+ 44,
+ 45,
+ 20,
+ 199,
+ 240,
+ 32,
+ 65,
+ 183,
+ 136
+ ],
+ [
+ 120,
+ 116,
+ 244,
+ 218,
+ 31,
+ 65,
+ 220,
+ 135,
+ 265,
+ 32,
+ 84,
+ 146,
+ 207,
+ 224,
+ 95,
+ 270,
+ 199,
+ 19,
+ 44,
+ 58
+ ],
+ [
+ 279,
+ 156,
+ 129,
+ 276,
+ 236,
+ 34,
+ 162,
+ 189,
+ 277,
+ 203,
+ 268,
+ 281,
+ 217,
+ 37,
+ 223,
+ 2,
+ 263,
+ 235,
+ 54,
+ 138
+ ],
+ [
+ 92,
+ 119,
+ 113,
+ 157,
+ 190,
+ 132,
+ 282,
+ 174,
+ 258,
+ 40,
+ 152,
+ 149,
+ 208,
+ 227,
+ 178,
+ 261,
+ 94,
+ 87,
+ 10,
+ 294
+ ],
+ [
+ 237,
+ 150,
+ 38,
+ 112,
+ 71,
+ 60,
+ 57,
+ 253,
+ 168,
+ 102,
+ 145,
+ 86,
+ 160,
+ 13,
+ 89,
+ 121,
+ 278,
+ 55,
+ 234,
+ 128
+ ],
+ [
+ 229,
+ 47,
+ 51,
+ 279,
+ 281,
+ 84,
+ 85,
+ 217,
+ 218,
+ 64,
+ 223,
+ 235,
+ 34,
+ 191,
+ 162,
+ 129,
+ 107,
+ 241,
+ 31,
+ 116
+ ],
+ [
+ 201,
+ 93,
+ 97,
+ 271,
+ 209,
+ 56,
+ 288,
+ 159,
+ 258,
+ 122,
+ 94,
+ 200,
+ 152,
+ 190,
+ 113,
+ 82,
+ 283,
+ 26,
+ 40,
+ 282
+ ],
+ [
+ 284,
+ 248,
+ 186,
+ 196,
+ 139,
+ 35,
+ 287,
+ 232,
+ 180,
+ 11,
+ 50,
+ 170,
+ 192,
+ 33,
+ 266,
+ 70,
+ 280,
+ 108,
+ 171,
+ 225
+ ],
+ [
+ 84,
+ 116,
+ 218,
+ 229,
+ 257,
+ 51,
+ 205,
+ 191,
+ 24,
+ 270,
+ 146,
+ 130,
+ 85,
+ 12,
+ 244,
+ 65,
+ 47,
+ 135,
+ 219,
+ 95
+ ],
+ [
+ 226,
+ 135,
+ 44,
+ 58,
+ 127,
+ 207,
+ 65,
+ 275,
+ 45,
+ 23,
+ 61,
+ 199,
+ 20,
+ 36,
+ 95,
+ 49,
+ 240,
+ 154,
+ 19,
+ 116
+ ],
+ [
+ 108,
+ 280,
+ 70,
+ 171,
+ 11,
+ 180,
+ 266,
+ 245,
+ 35,
+ 192,
+ 139,
+ 69,
+ 297,
+ 50,
+ 225,
+ 30,
+ 170,
+ 232,
+ 186,
+ 248
+ ],
+ [
+ 235,
+ 211,
+ 129,
+ 223,
+ 162,
+ 2,
+ 268,
+ 279,
+ 217,
+ 197,
+ 179,
+ 221,
+ 37,
+ 203,
+ 296,
+ 189,
+ 138,
+ 277,
+ 263,
+ 276
+ ],
+ [
+ 139,
+ 192,
+ 180,
+ 232,
+ 50,
+ 11,
+ 186,
+ 30,
+ 33,
+ 284,
+ 108,
+ 287,
+ 280,
+ 196,
+ 70,
+ 248,
+ 266,
+ 171,
+ 170,
+ 69
+ ],
+ [
+ 135,
+ 207,
+ 58,
+ 127,
+ 44,
+ 226,
+ 154,
+ 23,
+ 61,
+ 199,
+ 240,
+ 65,
+ 275,
+ 136,
+ 45,
+ 20,
+ 49,
+ 32,
+ 95,
+ 53
+ ],
+ [
+ 197,
+ 133,
+ 212,
+ 41,
+ 129,
+ 203,
+ 138,
+ 263,
+ 291,
+ 211,
+ 268,
+ 221,
+ 179,
+ 236,
+ 101,
+ 223,
+ 77,
+ 54,
+ 189,
+ 34
+ ],
+ [
+ 234,
+ 278,
+ 102,
+ 182,
+ 55,
+ 253,
+ 86,
+ 145,
+ 128,
+ 60,
+ 74,
+ 13,
+ 233,
+ 131,
+ 121,
+ 109,
+ 168,
+ 267,
+ 237,
+ 150
+ ],
+ [
+ 118,
+ 259,
+ 233,
+ 239,
+ 102,
+ 131,
+ 86,
+ 182,
+ 74,
+ 121,
+ 18,
+ 254,
+ 123,
+ 168,
+ 278,
+ 148,
+ 105,
+ 88,
+ 15,
+ 62
+ ],
+ [
+ 132,
+ 273,
+ 87,
+ 255,
+ 149,
+ 26,
+ 227,
+ 0,
+ 113,
+ 157,
+ 119,
+ 166,
+ 261,
+ 152,
+ 282,
+ 174,
+ 292,
+ 92,
+ 208,
+ 181
+ ],
+ [
+ 37,
+ 212,
+ 133,
+ 291,
+ 263,
+ 179,
+ 197,
+ 221,
+ 129,
+ 101,
+ 268,
+ 203,
+ 236,
+ 277,
+ 77,
+ 54,
+ 223,
+ 138,
+ 296,
+ 211
+ ],
+ [
+ 22,
+ 298,
+ 148,
+ 118,
+ 39,
+ 15,
+ 105,
+ 239,
+ 247,
+ 123,
+ 5,
+ 74,
+ 168,
+ 259,
+ 131,
+ 18,
+ 86,
+ 62,
+ 254,
+ 4
+ ],
+ [
+ 125,
+ 143,
+ 6,
+ 8,
+ 169,
+ 3,
+ 78,
+ 204,
+ 76,
+ 262,
+ 1,
+ 96,
+ 106,
+ 161,
+ 79,
+ 72,
+ 193,
+ 110,
+ 104,
+ 16
+ ],
+ [
+ 154,
+ 199,
+ 275,
+ 226,
+ 127,
+ 135,
+ 61,
+ 36,
+ 45,
+ 207,
+ 58,
+ 23,
+ 240,
+ 49,
+ 95,
+ 19,
+ 32,
+ 20,
+ 183,
+ 136
+ ],
+ [
+ 127,
+ 226,
+ 44,
+ 135,
+ 154,
+ 199,
+ 23,
+ 61,
+ 207,
+ 32,
+ 36,
+ 95,
+ 58,
+ 275,
+ 20,
+ 49,
+ 19,
+ 240,
+ 53,
+ 65
+ ],
+ [
+ 14,
+ 115,
+ 198,
+ 48,
+ 75,
+ 204,
+ 3,
+ 67,
+ 295,
+ 16,
+ 231,
+ 68,
+ 1,
+ 73,
+ 193,
+ 76,
+ 143,
+ 130,
+ 214,
+ 72
+ ],
+ [
+ 229,
+ 51,
+ 191,
+ 28,
+ 85,
+ 217,
+ 218,
+ 279,
+ 84,
+ 31,
+ 223,
+ 257,
+ 64,
+ 281,
+ 162,
+ 235,
+ 116,
+ 146,
+ 34,
+ 211
+ ],
+ [
+ 204,
+ 3,
+ 8,
+ 6,
+ 262,
+ 169,
+ 68,
+ 1,
+ 115,
+ 161,
+ 46,
+ 143,
+ 43,
+ 16,
+ 295,
+ 110,
+ 14,
+ 79,
+ 78,
+ 76
+ ],
+ [
+ 275,
+ 154,
+ 183,
+ 23,
+ 58,
+ 240,
+ 199,
+ 20,
+ 136,
+ 44,
+ 226,
+ 127,
+ 53,
+ 61,
+ 36,
+ 135,
+ 7,
+ 65,
+ 141,
+ 207
+ ],
+ [
+ 232,
+ 192,
+ 35,
+ 180,
+ 139,
+ 284,
+ 248,
+ 186,
+ 280,
+ 266,
+ 108,
+ 196,
+ 287,
+ 30,
+ 33,
+ 70,
+ 69,
+ 11,
+ 297,
+ 171
+ ],
+ [
+ 229,
+ 85,
+ 47,
+ 257,
+ 191,
+ 84,
+ 31,
+ 218,
+ 28,
+ 116,
+ 146,
+ 279,
+ 64,
+ 130,
+ 217,
+ 205,
+ 270,
+ 281,
+ 12,
+ 24
+ ],
+ [
+ 256,
+ 99,
+ 167,
+ 194,
+ 215,
+ 265,
+ 120,
+ 90,
+ 220,
+ 24,
+ 243,
+ 224,
+ 296,
+ 260,
+ 114,
+ 146,
+ 244,
+ 268,
+ 164,
+ 263
+ ],
+ [
+ 23,
+ 58,
+ 49,
+ 207,
+ 135,
+ 154,
+ 36,
+ 127,
+ 136,
+ 275,
+ 183,
+ 65,
+ 226,
+ 61,
+ 20,
+ 44,
+ 240,
+ 199,
+ 7,
+ 45
+ ],
+ [
+ 77,
+ 133,
+ 236,
+ 203,
+ 291,
+ 138,
+ 37,
+ 263,
+ 197,
+ 179,
+ 129,
+ 212,
+ 101,
+ 221,
+ 41,
+ 268,
+ 176,
+ 260,
+ 277,
+ 189
+ ],
+ [
+ 13,
+ 86,
+ 145,
+ 128,
+ 102,
+ 278,
+ 234,
+ 17,
+ 182,
+ 253,
+ 160,
+ 233,
+ 38,
+ 88,
+ 74,
+ 121,
+ 112,
+ 71,
+ 89,
+ 39
+ ],
+ [
+ 271,
+ 97,
+ 209,
+ 122,
+ 201,
+ 82,
+ 288,
+ 29,
+ 93,
+ 159,
+ 173,
+ 40,
+ 258,
+ 149,
+ 282,
+ 255,
+ 200,
+ 132,
+ 113,
+ 152
+ ],
+ [
+ 60,
+ 160,
+ 128,
+ 267,
+ 253,
+ 278,
+ 124,
+ 237,
+ 109,
+ 111,
+ 38,
+ 102,
+ 13,
+ 137,
+ 182,
+ 88,
+ 206,
+ 17,
+ 150,
+ 145
+ ],
+ [
+ 135,
+ 36,
+ 207,
+ 127,
+ 23,
+ 275,
+ 154,
+ 65,
+ 53,
+ 226,
+ 49,
+ 44,
+ 95,
+ 32,
+ 199,
+ 183,
+ 45,
+ 61,
+ 136,
+ 240
+ ],
+ [
+ 261,
+ 157,
+ 142,
+ 87,
+ 282,
+ 132,
+ 227,
+ 80,
+ 181,
+ 119,
+ 0,
+ 166,
+ 273,
+ 149,
+ 255,
+ 208,
+ 92,
+ 26,
+ 40,
+ 113
+ ],
+ [
+ 267,
+ 278,
+ 185,
+ 182,
+ 102,
+ 109,
+ 253,
+ 128,
+ 91,
+ 121,
+ 137,
+ 131,
+ 124,
+ 62,
+ 18,
+ 254,
+ 88,
+ 145,
+ 57,
+ 38
+ ],
+ [
+ 226,
+ 23,
+ 127,
+ 44,
+ 207,
+ 36,
+ 154,
+ 135,
+ 240,
+ 275,
+ 45,
+ 199,
+ 20,
+ 32,
+ 49,
+ 58,
+ 19,
+ 65,
+ 53,
+ 136
+ ],
+ [
+ 185,
+ 18,
+ 239,
+ 123,
+ 131,
+ 91,
+ 124,
+ 137,
+ 298,
+ 259,
+ 15,
+ 118,
+ 60,
+ 74,
+ 109,
+ 278,
+ 254,
+ 39,
+ 286,
+ 182
+ ],
+ [
+ 74,
+ 5,
+ 91,
+ 298,
+ 18,
+ 286,
+ 185,
+ 62,
+ 60,
+ 22,
+ 105,
+ 15,
+ 148,
+ 118,
+ 131,
+ 239,
+ 38,
+ 234,
+ 86,
+ 278
+ ],
+ [
+ 114,
+ 229,
+ 85,
+ 279,
+ 218,
+ 146,
+ 28,
+ 281,
+ 162,
+ 84,
+ 25,
+ 156,
+ 217,
+ 47,
+ 51,
+ 189,
+ 270,
+ 34,
+ 235,
+ 12
+ ],
+ [
+ 135,
+ 58,
+ 207,
+ 36,
+ 32,
+ 240,
+ 23,
+ 275,
+ 136,
+ 20,
+ 49,
+ 44,
+ 199,
+ 226,
+ 53,
+ 127,
+ 24,
+ 154,
+ 61,
+ 95
+ ],
+ [
+ 141,
+ 246,
+ 169,
+ 188,
+ 68,
+ 183,
+ 16,
+ 7,
+ 147,
+ 96,
+ 43,
+ 100,
+ 8,
+ 73,
+ 154,
+ 275,
+ 49,
+ 72,
+ 143,
+ 6
+ ],
+ [
+ 142,
+ 158,
+ 252,
+ 290,
+ 295,
+ 214,
+ 80,
+ 272,
+ 204,
+ 72,
+ 76,
+ 193,
+ 3,
+ 187,
+ 115,
+ 68,
+ 1,
+ 210,
+ 166,
+ 261
+ ],
+ [
+ 169,
+ 16,
+ 188,
+ 72,
+ 66,
+ 3,
+ 96,
+ 8,
+ 204,
+ 43,
+ 73,
+ 272,
+ 143,
+ 1,
+ 295,
+ 246,
+ 6,
+ 48,
+ 141,
+ 106
+ ],
+ [
+ 297,
+ 245,
+ 108,
+ 280,
+ 266,
+ 171,
+ 192,
+ 70,
+ 180,
+ 33,
+ 11,
+ 35,
+ 50,
+ 139,
+ 232,
+ 225,
+ 170,
+ 30,
+ 284,
+ 186
+ ],
+ [
+ 108,
+ 33,
+ 11,
+ 280,
+ 171,
+ 266,
+ 245,
+ 180,
+ 69,
+ 297,
+ 139,
+ 192,
+ 35,
+ 50,
+ 30,
+ 225,
+ 232,
+ 170,
+ 186,
+ 248
+ ],
+ [
+ 89,
+ 88,
+ 86,
+ 253,
+ 134,
+ 121,
+ 102,
+ 233,
+ 145,
+ 259,
+ 112,
+ 150,
+ 278,
+ 55,
+ 39,
+ 163,
+ 160,
+ 182,
+ 254,
+ 13
+ ],
+ [
+ 272,
+ 8,
+ 214,
+ 290,
+ 110,
+ 169,
+ 210,
+ 68,
+ 80,
+ 6,
+ 104,
+ 161,
+ 43,
+ 149,
+ 204,
+ 87,
+ 106,
+ 143,
+ 193,
+ 142
+ ],
+ [
+ 188,
+ 16,
+ 68,
+ 204,
+ 3,
+ 43,
+ 143,
+ 246,
+ 193,
+ 76,
+ 115,
+ 72,
+ 1,
+ 66,
+ 169,
+ 262,
+ 8,
+ 48,
+ 6,
+ 125
+ ],
+ [
+ 86,
+ 105,
+ 118,
+ 39,
+ 18,
+ 131,
+ 148,
+ 91,
+ 185,
+ 102,
+ 62,
+ 278,
+ 234,
+ 55,
+ 145,
+ 298,
+ 239,
+ 182,
+ 259,
+ 5
+ ],
+ [
+ 14,
+ 198,
+ 115,
+ 231,
+ 46,
+ 204,
+ 67,
+ 48,
+ 214,
+ 3,
+ 73,
+ 295,
+ 68,
+ 272,
+ 187,
+ 16,
+ 72,
+ 8,
+ 143,
+ 1
+ ],
+ [
+ 3,
+ 204,
+ 193,
+ 143,
+ 43,
+ 295,
+ 1,
+ 104,
+ 290,
+ 96,
+ 125,
+ 72,
+ 161,
+ 110,
+ 272,
+ 106,
+ 6,
+ 67,
+ 8,
+ 80
+ ],
+ [
+ 54,
+ 133,
+ 101,
+ 203,
+ 212,
+ 37,
+ 41,
+ 197,
+ 138,
+ 176,
+ 221,
+ 179,
+ 236,
+ 129,
+ 285,
+ 291,
+ 9,
+ 211,
+ 2,
+ 263
+ ],
+ [
+ 6,
+ 79,
+ 161,
+ 262,
+ 125,
+ 106,
+ 8,
+ 110,
+ 43,
+ 104,
+ 143,
+ 250,
+ 272,
+ 1,
+ 204,
+ 80,
+ 169,
+ 113,
+ 174,
+ 274
+ ],
+ [
+ 78,
+ 161,
+ 104,
+ 110,
+ 262,
+ 8,
+ 106,
+ 6,
+ 125,
+ 43,
+ 250,
+ 143,
+ 204,
+ 264,
+ 290,
+ 190,
+ 3,
+ 76,
+ 169,
+ 274
+ ],
+ [
+ 142,
+ 261,
+ 149,
+ 290,
+ 87,
+ 272,
+ 174,
+ 59,
+ 208,
+ 110,
+ 132,
+ 113,
+ 161,
+ 227,
+ 187,
+ 166,
+ 252,
+ 274,
+ 26,
+ 104
+ ],
+ [
+ 151,
+ 103,
+ 293,
+ 251,
+ 10,
+ 216,
+ 294,
+ 208,
+ 0,
+ 119,
+ 181,
+ 98,
+ 255,
+ 200,
+ 92,
+ 273,
+ 292,
+ 227,
+ 172,
+ 153
+ ],
+ [
+ 56,
+ 40,
+ 209,
+ 181,
+ 255,
+ 282,
+ 273,
+ 166,
+ 0,
+ 97,
+ 208,
+ 271,
+ 149,
+ 151,
+ 227,
+ 292,
+ 122,
+ 132,
+ 261,
+ 200
+ ],
+ [
+ 230,
+ 175,
+ 298,
+ 124,
+ 62,
+ 137,
+ 18,
+ 123,
+ 109,
+ 239,
+ 60,
+ 131,
+ 185,
+ 42,
+ 118,
+ 15,
+ 254,
+ 105,
+ 286,
+ 5
+ ],
+ [
+ 31,
+ 218,
+ 229,
+ 116,
+ 51,
+ 270,
+ 191,
+ 130,
+ 205,
+ 257,
+ 85,
+ 146,
+ 12,
+ 28,
+ 219,
+ 47,
+ 24,
+ 64,
+ 279,
+ 224
+ ],
+ [
+ 229,
+ 51,
+ 47,
+ 84,
+ 257,
+ 28,
+ 64,
+ 191,
+ 31,
+ 146,
+ 218,
+ 279,
+ 281,
+ 217,
+ 116,
+ 162,
+ 130,
+ 270,
+ 205,
+ 12
+ ],
+ [
+ 102,
+ 145,
+ 55,
+ 39,
+ 71,
+ 74,
+ 234,
+ 121,
+ 278,
+ 233,
+ 89,
+ 259,
+ 253,
+ 182,
+ 88,
+ 118,
+ 131,
+ 38,
+ 91,
+ 168
+ ],
+ [
+ 227,
+ 149,
+ 132,
+ 261,
+ 40,
+ 113,
+ 152,
+ 80,
+ 59,
+ 92,
+ 208,
+ 174,
+ 26,
+ 273,
+ 282,
+ 142,
+ 166,
+ 272,
+ 157,
+ 274
+ ],
+ [
+ 278,
+ 253,
+ 102,
+ 121,
+ 89,
+ 233,
+ 71,
+ 182,
+ 267,
+ 160,
+ 145,
+ 128,
+ 259,
+ 86,
+ 13,
+ 185,
+ 55,
+ 39,
+ 254,
+ 60
+ ],
+ [
+ 71,
+ 88,
+ 86,
+ 233,
+ 253,
+ 102,
+ 121,
+ 259,
+ 145,
+ 134,
+ 278,
+ 182,
+ 55,
+ 39,
+ 254,
+ 112,
+ 131,
+ 160,
+ 185,
+ 234
+ ],
+ [
+ 194,
+ 215,
+ 260,
+ 243,
+ 291,
+ 296,
+ 263,
+ 236,
+ 268,
+ 54,
+ 9,
+ 133,
+ 41,
+ 212,
+ 37,
+ 179,
+ 276,
+ 277,
+ 138,
+ 221
+ ],
+ [
+ 185,
+ 278,
+ 62,
+ 18,
+ 253,
+ 102,
+ 145,
+ 121,
+ 131,
+ 60,
+ 182,
+ 74,
+ 239,
+ 259,
+ 15,
+ 123,
+ 267,
+ 86,
+ 286,
+ 234
+ ],
+ [
+ 26,
+ 119,
+ 178,
+ 294,
+ 208,
+ 227,
+ 190,
+ 10,
+ 157,
+ 152,
+ 282,
+ 132,
+ 113,
+ 261,
+ 87,
+ 174,
+ 255,
+ 98,
+ 251,
+ 149
+ ],
+ [
+ 29,
+ 201,
+ 97,
+ 271,
+ 209,
+ 56,
+ 288,
+ 159,
+ 258,
+ 122,
+ 200,
+ 283,
+ 94,
+ 152,
+ 82,
+ 190,
+ 113,
+ 165,
+ 282,
+ 40
+ ],
+ [
+ 152,
+ 190,
+ 258,
+ 26,
+ 113,
+ 92,
+ 282,
+ 149,
+ 178,
+ 119,
+ 250,
+ 283,
+ 208,
+ 274,
+ 238,
+ 87,
+ 227,
+ 132,
+ 294,
+ 21
+ ],
+ [
+ 135,
+ 44,
+ 275,
+ 58,
+ 45,
+ 154,
+ 127,
+ 199,
+ 226,
+ 7,
+ 32,
+ 36,
+ 116,
+ 207,
+ 141,
+ 65,
+ 246,
+ 240,
+ 23,
+ 49
+ ],
+ [
+ 169,
+ 125,
+ 43,
+ 143,
+ 1,
+ 3,
+ 68,
+ 76,
+ 106,
+ 72,
+ 110,
+ 6,
+ 290,
+ 204,
+ 16,
+ 295,
+ 66,
+ 161,
+ 8,
+ 272
+ ],
+ [
+ 271,
+ 201,
+ 29,
+ 209,
+ 56,
+ 288,
+ 93,
+ 258,
+ 82,
+ 282,
+ 152,
+ 113,
+ 200,
+ 190,
+ 149,
+ 94,
+ 283,
+ 122,
+ 40,
+ 159
+ ],
+ [
+ 293,
+ 251,
+ 92,
+ 294,
+ 10,
+ 208,
+ 195,
+ 178,
+ 119,
+ 227,
+ 103,
+ 172,
+ 282,
+ 261,
+ 81,
+ 157,
+ 190,
+ 155,
+ 87,
+ 59
+ ],
+ [
+ 167,
+ 194,
+ 215,
+ 90,
+ 243,
+ 268,
+ 296,
+ 25,
+ 146,
+ 164,
+ 260,
+ 263,
+ 276,
+ 241,
+ 279,
+ 236,
+ 256,
+ 277,
+ 162,
+ 107
+ ],
+ [
+ 264,
+ 8,
+ 169,
+ 6,
+ 43,
+ 262,
+ 72,
+ 66,
+ 147,
+ 16,
+ 110,
+ 78,
+ 79,
+ 7,
+ 173,
+ 68,
+ 188,
+ 161,
+ 125,
+ 183
+ ],
+ [
+ 221,
+ 179,
+ 197,
+ 77,
+ 133,
+ 176,
+ 41,
+ 37,
+ 211,
+ 54,
+ 277,
+ 203,
+ 212,
+ 189,
+ 235,
+ 129,
+ 138,
+ 223,
+ 2,
+ 268
+ ],
+ [
+ 182,
+ 233,
+ 121,
+ 278,
+ 88,
+ 128,
+ 253,
+ 86,
+ 267,
+ 234,
+ 39,
+ 259,
+ 145,
+ 55,
+ 60,
+ 13,
+ 131,
+ 185,
+ 91,
+ 38
+ ],
+ [
+ 151,
+ 293,
+ 81,
+ 255,
+ 119,
+ 216,
+ 294,
+ 10,
+ 98,
+ 208,
+ 200,
+ 181,
+ 0,
+ 178,
+ 92,
+ 157,
+ 282,
+ 251,
+ 292,
+ 273
+ ],
+ [
+ 161,
+ 110,
+ 79,
+ 290,
+ 106,
+ 80,
+ 174,
+ 76,
+ 78,
+ 190,
+ 272,
+ 149,
+ 125,
+ 143,
+ 26,
+ 8,
+ 142,
+ 72,
+ 87,
+ 187
+ ],
+ [
+ 74,
+ 39,
+ 298,
+ 118,
+ 148,
+ 86,
+ 18,
+ 5,
+ 168,
+ 42,
+ 131,
+ 22,
+ 239,
+ 4,
+ 55,
+ 102,
+ 278,
+ 182,
+ 62,
+ 38
+ ],
+ [
+ 161,
+ 262,
+ 125,
+ 78,
+ 104,
+ 110,
+ 272,
+ 79,
+ 43,
+ 8,
+ 290,
+ 143,
+ 274,
+ 80,
+ 72,
+ 6,
+ 96,
+ 169,
+ 3,
+ 204
+ ],
+ [
+ 164,
+ 241,
+ 34,
+ 279,
+ 129,
+ 156,
+ 162,
+ 235,
+ 211,
+ 223,
+ 221,
+ 2,
+ 268,
+ 217,
+ 189,
+ 25,
+ 197,
+ 203,
+ 37,
+ 243
+ ],
+ [
+ 280,
+ 70,
+ 245,
+ 69,
+ 33,
+ 171,
+ 297,
+ 266,
+ 180,
+ 192,
+ 11,
+ 35,
+ 139,
+ 50,
+ 232,
+ 30,
+ 170,
+ 225,
+ 186,
+ 248
+ ],
+ [
+ 267,
+ 182,
+ 128,
+ 278,
+ 60,
+ 137,
+ 102,
+ 124,
+ 131,
+ 185,
+ 62,
+ 18,
+ 233,
+ 91,
+ 239,
+ 254,
+ 121,
+ 88,
+ 39,
+ 253
+ ],
+ [
+ 161,
+ 8,
+ 272,
+ 80,
+ 104,
+ 187,
+ 79,
+ 274,
+ 78,
+ 72,
+ 106,
+ 87,
+ 169,
+ 214,
+ 290,
+ 250,
+ 174,
+ 149,
+ 6,
+ 142
+ ],
+ [
+ 13,
+ 128,
+ 160,
+ 17,
+ 206,
+ 55,
+ 184,
+ 57,
+ 278,
+ 102,
+ 109,
+ 182,
+ 299,
+ 267,
+ 60,
+ 145,
+ 137,
+ 253,
+ 88,
+ 237
+ ],
+ [
+ 160,
+ 71,
+ 253,
+ 88,
+ 145,
+ 55,
+ 13,
+ 89,
+ 184,
+ 102,
+ 278,
+ 128,
+ 121,
+ 206,
+ 182,
+ 234,
+ 17,
+ 299,
+ 86,
+ 150
+ ],
+ [
+ 132,
+ 26,
+ 258,
+ 152,
+ 190,
+ 149,
+ 92,
+ 119,
+ 87,
+ 157,
+ 40,
+ 282,
+ 261,
+ 178,
+ 208,
+ 94,
+ 294,
+ 227,
+ 174,
+ 222
+ ],
+ [
+ 64,
+ 156,
+ 25,
+ 270,
+ 146,
+ 218,
+ 12,
+ 279,
+ 84,
+ 164,
+ 162,
+ 229,
+ 130,
+ 167,
+ 281,
+ 116,
+ 241,
+ 219,
+ 189,
+ 31
+ ],
+ [
+ 14,
+ 46,
+ 75,
+ 204,
+ 3,
+ 48,
+ 67,
+ 68,
+ 1,
+ 16,
+ 73,
+ 295,
+ 198,
+ 143,
+ 72,
+ 193,
+ 272,
+ 231,
+ 76,
+ 214
+ ],
+ [
+ 31,
+ 218,
+ 84,
+ 24,
+ 229,
+ 95,
+ 270,
+ 130,
+ 51,
+ 191,
+ 135,
+ 65,
+ 12,
+ 32,
+ 58,
+ 146,
+ 257,
+ 205,
+ 120,
+ 244
+ ],
+ [
+ 177,
+ 247,
+ 134,
+ 289,
+ 42,
+ 254,
+ 168,
+ 118,
+ 4,
+ 17,
+ 71,
+ 148,
+ 89,
+ 86,
+ 74,
+ 105,
+ 39,
+ 137,
+ 298,
+ 145
+ ],
+ [
+ 39,
+ 239,
+ 148,
+ 15,
+ 259,
+ 74,
+ 123,
+ 168,
+ 18,
+ 62,
+ 86,
+ 131,
+ 121,
+ 42,
+ 102,
+ 5,
+ 185,
+ 233,
+ 298,
+ 182
+ ],
+ [
+ 178,
+ 92,
+ 157,
+ 255,
+ 26,
+ 282,
+ 190,
+ 294,
+ 132,
+ 208,
+ 113,
+ 10,
+ 152,
+ 227,
+ 40,
+ 258,
+ 261,
+ 151,
+ 59,
+ 149
+ ],
+ [
+ 24,
+ 220,
+ 65,
+ 32,
+ 116,
+ 244,
+ 199,
+ 240,
+ 19,
+ 135,
+ 20,
+ 44,
+ 218,
+ 265,
+ 207,
+ 36,
+ 226,
+ 275,
+ 95,
+ 31
+ ],
+ [
+ 102,
+ 278,
+ 88,
+ 182,
+ 233,
+ 253,
+ 145,
+ 86,
+ 185,
+ 39,
+ 89,
+ 91,
+ 160,
+ 71,
+ 128,
+ 259,
+ 267,
+ 60,
+ 234,
+ 118
+ ],
+ [
+ 56,
+ 201,
+ 271,
+ 82,
+ 209,
+ 97,
+ 159,
+ 29,
+ 173,
+ 288,
+ 93,
+ 40,
+ 258,
+ 166,
+ 200,
+ 149,
+ 210,
+ 273,
+ 282,
+ 26
+ ],
+ [
+ 239,
+ 259,
+ 15,
+ 62,
+ 131,
+ 18,
+ 185,
+ 118,
+ 39,
+ 91,
+ 254,
+ 298,
+ 137,
+ 124,
+ 233,
+ 102,
+ 5,
+ 60,
+ 278,
+ 42
+ ],
+ [
+ 62,
+ 137,
+ 109,
+ 185,
+ 267,
+ 60,
+ 18,
+ 278,
+ 131,
+ 128,
+ 286,
+ 91,
+ 102,
+ 182,
+ 123,
+ 259,
+ 239,
+ 253,
+ 88,
+ 121
+ ],
+ [
+ 43,
+ 106,
+ 143,
+ 78,
+ 6,
+ 96,
+ 193,
+ 79,
+ 104,
+ 1,
+ 161,
+ 76,
+ 3,
+ 262,
+ 290,
+ 204,
+ 8,
+ 110,
+ 72,
+ 169
+ ],
+ [
+ 213,
+ 22,
+ 42,
+ 175,
+ 163,
+ 5,
+ 148,
+ 15,
+ 239,
+ 118,
+ 39,
+ 123,
+ 168,
+ 259,
+ 298,
+ 247,
+ 131,
+ 105,
+ 233,
+ 4
+ ],
+ [
+ 226,
+ 23,
+ 61,
+ 45,
+ 44,
+ 135,
+ 154,
+ 58,
+ 36,
+ 207,
+ 275,
+ 199,
+ 49,
+ 32,
+ 95,
+ 240,
+ 53,
+ 20,
+ 65,
+ 19
+ ],
+ [
+ 13,
+ 278,
+ 182,
+ 102,
+ 267,
+ 160,
+ 109,
+ 55,
+ 88,
+ 253,
+ 233,
+ 60,
+ 111,
+ 121,
+ 145,
+ 137,
+ 38,
+ 17,
+ 124,
+ 234
+ ],
+ [
+ 223,
+ 203,
+ 221,
+ 37,
+ 211,
+ 197,
+ 34,
+ 279,
+ 235,
+ 162,
+ 41,
+ 217,
+ 179,
+ 277,
+ 263,
+ 268,
+ 138,
+ 2,
+ 54,
+ 291
+ ],
+ [
+ 270,
+ 84,
+ 218,
+ 12,
+ 116,
+ 219,
+ 31,
+ 205,
+ 229,
+ 246,
+ 68,
+ 16,
+ 95,
+ 66,
+ 141,
+ 169,
+ 48,
+ 7,
+ 51,
+ 158
+ ],
+ [
+ 18,
+ 259,
+ 239,
+ 62,
+ 123,
+ 39,
+ 137,
+ 185,
+ 182,
+ 102,
+ 233,
+ 278,
+ 91,
+ 109,
+ 74,
+ 60,
+ 118,
+ 15,
+ 86,
+ 5
+ ],
+ [
+ 40,
+ 149,
+ 157,
+ 113,
+ 261,
+ 87,
+ 26,
+ 227,
+ 174,
+ 92,
+ 119,
+ 282,
+ 152,
+ 208,
+ 190,
+ 255,
+ 273,
+ 59,
+ 294,
+ 178
+ ],
+ [
+ 212,
+ 37,
+ 197,
+ 41,
+ 291,
+ 9,
+ 263,
+ 236,
+ 77,
+ 268,
+ 54,
+ 101,
+ 221,
+ 203,
+ 211,
+ 138,
+ 179,
+ 129,
+ 260,
+ 296
+ ],
+ [
+ 71,
+ 89,
+ 247,
+ 86,
+ 289,
+ 88,
+ 150,
+ 39,
+ 253,
+ 163,
+ 121,
+ 259,
+ 233,
+ 168,
+ 102,
+ 145,
+ 278,
+ 118,
+ 55,
+ 254
+ ],
+ [
+ 207,
+ 36,
+ 58,
+ 127,
+ 44,
+ 226,
+ 65,
+ 154,
+ 23,
+ 275,
+ 95,
+ 45,
+ 199,
+ 61,
+ 32,
+ 240,
+ 20,
+ 53,
+ 49,
+ 136
+ ],
+ [
+ 183,
+ 240,
+ 49,
+ 275,
+ 154,
+ 36,
+ 65,
+ 20,
+ 58,
+ 199,
+ 7,
+ 44,
+ 135,
+ 53,
+ 23,
+ 207,
+ 141,
+ 61,
+ 127,
+ 226
+ ],
+ [
+ 131,
+ 62,
+ 124,
+ 109,
+ 278,
+ 60,
+ 254,
+ 185,
+ 267,
+ 18,
+ 128,
+ 259,
+ 102,
+ 239,
+ 88,
+ 123,
+ 253,
+ 182,
+ 39,
+ 233
+ ],
+ [
+ 37,
+ 203,
+ 179,
+ 197,
+ 212,
+ 129,
+ 54,
+ 291,
+ 133,
+ 77,
+ 211,
+ 263,
+ 41,
+ 2,
+ 221,
+ 34,
+ 101,
+ 9,
+ 236,
+ 223
+ ],
+ [
+ 35,
+ 11,
+ 232,
+ 186,
+ 30,
+ 192,
+ 180,
+ 33,
+ 70,
+ 50,
+ 108,
+ 248,
+ 280,
+ 287,
+ 284,
+ 170,
+ 196,
+ 266,
+ 171,
+ 245
+ ],
+ [
+ 195,
+ 165,
+ 200,
+ 283,
+ 293,
+ 251,
+ 178,
+ 208,
+ 98,
+ 294,
+ 92,
+ 227,
+ 103,
+ 151,
+ 10,
+ 21,
+ 81,
+ 119,
+ 172,
+ 255
+ ],
+ [
+ 66,
+ 246,
+ 183,
+ 7,
+ 154,
+ 169,
+ 188,
+ 68,
+ 275,
+ 199,
+ 96,
+ 49,
+ 16,
+ 44,
+ 95,
+ 136,
+ 240,
+ 43,
+ 58,
+ 20
+ ],
+ [
+ 80,
+ 290,
+ 261,
+ 252,
+ 59,
+ 166,
+ 87,
+ 174,
+ 149,
+ 227,
+ 67,
+ 181,
+ 272,
+ 208,
+ 132,
+ 110,
+ 0,
+ 273,
+ 104,
+ 187
+ ],
+ [
+ 3,
+ 43,
+ 204,
+ 6,
+ 125,
+ 8,
+ 76,
+ 1,
+ 104,
+ 96,
+ 262,
+ 161,
+ 106,
+ 78,
+ 79,
+ 110,
+ 72,
+ 169,
+ 193,
+ 68
+ ],
+ [
+ 215,
+ 99,
+ 194,
+ 167,
+ 90,
+ 146,
+ 296,
+ 260,
+ 243,
+ 277,
+ 236,
+ 291,
+ 156,
+ 164,
+ 162,
+ 114,
+ 179,
+ 268,
+ 25,
+ 276
+ ],
+ [
+ 253,
+ 278,
+ 86,
+ 55,
+ 102,
+ 121,
+ 88,
+ 234,
+ 17,
+ 160,
+ 91,
+ 182,
+ 185,
+ 89,
+ 71,
+ 128,
+ 13,
+ 74,
+ 38,
+ 60
+ ],
+ [
+ 229,
+ 218,
+ 84,
+ 191,
+ 31,
+ 64,
+ 85,
+ 279,
+ 51,
+ 116,
+ 114,
+ 24,
+ 217,
+ 47,
+ 257,
+ 265,
+ 162,
+ 241,
+ 164,
+ 28
+ ],
+ [
+ 66,
+ 183,
+ 8,
+ 100,
+ 169,
+ 43,
+ 188,
+ 262,
+ 68,
+ 141,
+ 246,
+ 49,
+ 6,
+ 136,
+ 73,
+ 143,
+ 7,
+ 16,
+ 48,
+ 106
+ ],
+ [
+ 168,
+ 118,
+ 74,
+ 39,
+ 42,
+ 239,
+ 22,
+ 105,
+ 259,
+ 298,
+ 86,
+ 5,
+ 131,
+ 15,
+ 247,
+ 123,
+ 18,
+ 62,
+ 102,
+ 254
+ ],
+ [
+ 132,
+ 87,
+ 261,
+ 113,
+ 80,
+ 40,
+ 26,
+ 227,
+ 208,
+ 282,
+ 274,
+ 174,
+ 92,
+ 190,
+ 290,
+ 119,
+ 152,
+ 157,
+ 258,
+ 59
+ ],
+ [
+ 237,
+ 27,
+ 71,
+ 168,
+ 253,
+ 102,
+ 38,
+ 86,
+ 134,
+ 88,
+ 89,
+ 121,
+ 233,
+ 60,
+ 278,
+ 112,
+ 160,
+ 145,
+ 13,
+ 163
+ ],
+ [
+ 103,
+ 81,
+ 255,
+ 216,
+ 208,
+ 0,
+ 273,
+ 119,
+ 181,
+ 227,
+ 166,
+ 10,
+ 251,
+ 261,
+ 293,
+ 92,
+ 222,
+ 40,
+ 294,
+ 178
+ ],
+ [
+ 113,
+ 92,
+ 227,
+ 178,
+ 26,
+ 87,
+ 119,
+ 132,
+ 94,
+ 282,
+ 208,
+ 258,
+ 255,
+ 40,
+ 190,
+ 294,
+ 200,
+ 251,
+ 149,
+ 238
+ ],
+ [
+ 181,
+ 292,
+ 166,
+ 216,
+ 0,
+ 252,
+ 273,
+ 293,
+ 187,
+ 151,
+ 81,
+ 10,
+ 103,
+ 208,
+ 249,
+ 227,
+ 172,
+ 242,
+ 261,
+ 82
+ ],
+ [
+ 44,
+ 199,
+ 275,
+ 127,
+ 49,
+ 23,
+ 135,
+ 58,
+ 183,
+ 36,
+ 240,
+ 226,
+ 61,
+ 45,
+ 207,
+ 136,
+ 20,
+ 141,
+ 95,
+ 7
+ ],
+ [
+ 178,
+ 119,
+ 92,
+ 294,
+ 282,
+ 255,
+ 98,
+ 190,
+ 165,
+ 157,
+ 238,
+ 103,
+ 10,
+ 152,
+ 195,
+ 26,
+ 258,
+ 208,
+ 293,
+ 151
+ ],
+ [
+ 25,
+ 281,
+ 279,
+ 34,
+ 162,
+ 164,
+ 107,
+ 189,
+ 129,
+ 2,
+ 235,
+ 12,
+ 217,
+ 241,
+ 203,
+ 211,
+ 223,
+ 276,
+ 243,
+ 268
+ ],
+ [
+ 119,
+ 132,
+ 282,
+ 26,
+ 92,
+ 255,
+ 59,
+ 113,
+ 190,
+ 40,
+ 178,
+ 261,
+ 294,
+ 227,
+ 149,
+ 87,
+ 273,
+ 208,
+ 152,
+ 174
+ ],
+ [
+ 67,
+ 252,
+ 219,
+ 142,
+ 181,
+ 210,
+ 153,
+ 290,
+ 214,
+ 166,
+ 295,
+ 72,
+ 292,
+ 80,
+ 0,
+ 273,
+ 187,
+ 272,
+ 261,
+ 76
+ ],
+ [
+ 209,
+ 201,
+ 29,
+ 56,
+ 122,
+ 271,
+ 97,
+ 93,
+ 82,
+ 200,
+ 288,
+ 40,
+ 274,
+ 149,
+ 258,
+ 165,
+ 26,
+ 210,
+ 173,
+ 190
+ ],
+ [
+ 13,
+ 128,
+ 88,
+ 278,
+ 253,
+ 206,
+ 145,
+ 121,
+ 102,
+ 112,
+ 55,
+ 17,
+ 111,
+ 267,
+ 182,
+ 57,
+ 60,
+ 184,
+ 71,
+ 233
+ ],
+ [
+ 110,
+ 106,
+ 104,
+ 8,
+ 78,
+ 79,
+ 80,
+ 6,
+ 272,
+ 262,
+ 290,
+ 204,
+ 43,
+ 125,
+ 274,
+ 72,
+ 143,
+ 149,
+ 250,
+ 1
+ ],
+ [
+ 279,
+ 217,
+ 129,
+ 34,
+ 221,
+ 223,
+ 211,
+ 235,
+ 189,
+ 37,
+ 268,
+ 203,
+ 41,
+ 179,
+ 197,
+ 2,
+ 276,
+ 296,
+ 212,
+ 277
+ ],
+ [
+ 71,
+ 86,
+ 259,
+ 39,
+ 134,
+ 89,
+ 233,
+ 168,
+ 150,
+ 118,
+ 247,
+ 88,
+ 121,
+ 102,
+ 4,
+ 148,
+ 123,
+ 239,
+ 254,
+ 131
+ ],
+ [
+ 241,
+ 107,
+ 34,
+ 156,
+ 162,
+ 279,
+ 268,
+ 211,
+ 243,
+ 2,
+ 235,
+ 129,
+ 217,
+ 296,
+ 223,
+ 189,
+ 221,
+ 25,
+ 197,
+ 260
+ ],
+ [
+ 200,
+ 92,
+ 178,
+ 140,
+ 119,
+ 152,
+ 294,
+ 283,
+ 10,
+ 208,
+ 195,
+ 190,
+ 258,
+ 172,
+ 26,
+ 255,
+ 155,
+ 227,
+ 238,
+ 94
+ ],
+ [
+ 181,
+ 273,
+ 216,
+ 261,
+ 252,
+ 0,
+ 174,
+ 40,
+ 142,
+ 292,
+ 208,
+ 151,
+ 87,
+ 26,
+ 222,
+ 59,
+ 227,
+ 132,
+ 80,
+ 149
+ ],
+ [
+ 99,
+ 194,
+ 215,
+ 296,
+ 268,
+ 279,
+ 25,
+ 243,
+ 263,
+ 276,
+ 260,
+ 162,
+ 277,
+ 129,
+ 217,
+ 90,
+ 291,
+ 164,
+ 34,
+ 236
+ ],
+ [
+ 148,
+ 118,
+ 39,
+ 259,
+ 86,
+ 74,
+ 239,
+ 150,
+ 233,
+ 131,
+ 102,
+ 105,
+ 247,
+ 38,
+ 123,
+ 60,
+ 121,
+ 182,
+ 254,
+ 18
+ ],
+ [
+ 8,
+ 68,
+ 66,
+ 96,
+ 43,
+ 6,
+ 72,
+ 110,
+ 16,
+ 161,
+ 1,
+ 143,
+ 100,
+ 272,
+ 106,
+ 188,
+ 3,
+ 204,
+ 262,
+ 125
+ ],
+ [
+ 225,
+ 139,
+ 35,
+ 30,
+ 180,
+ 33,
+ 11,
+ 287,
+ 70,
+ 50,
+ 108,
+ 192,
+ 186,
+ 284,
+ 232,
+ 280,
+ 266,
+ 248,
+ 196,
+ 171
+ ],
+ [
+ 266,
+ 280,
+ 245,
+ 108,
+ 33,
+ 70,
+ 297,
+ 69,
+ 180,
+ 11,
+ 192,
+ 35,
+ 139,
+ 50,
+ 225,
+ 30,
+ 170,
+ 232,
+ 186,
+ 284
+ ],
+ [
+ 208,
+ 92,
+ 227,
+ 251,
+ 293,
+ 195,
+ 98,
+ 165,
+ 152,
+ 174,
+ 261,
+ 10,
+ 274,
+ 81,
+ 119,
+ 21,
+ 59,
+ 87,
+ 178,
+ 80
+ ],
+ [
+ 56,
+ 224,
+ 210,
+ 122,
+ 97,
+ 219,
+ 82,
+ 201,
+ 271,
+ 209,
+ 244,
+ 264,
+ 100,
+ 205,
+ 159,
+ 72,
+ 40,
+ 29,
+ 288,
+ 149
+ ],
+ [
+ 261,
+ 132,
+ 26,
+ 92,
+ 80,
+ 227,
+ 208,
+ 149,
+ 166,
+ 87,
+ 222,
+ 40,
+ 142,
+ 113,
+ 274,
+ 157,
+ 290,
+ 273,
+ 282,
+ 152
+ ],
+ [
+ 298,
+ 230,
+ 123,
+ 15,
+ 62,
+ 239,
+ 83,
+ 42,
+ 22,
+ 5,
+ 18,
+ 118,
+ 131,
+ 185,
+ 259,
+ 137,
+ 124,
+ 39,
+ 254,
+ 91
+ ],
+ [
+ 101,
+ 221,
+ 203,
+ 129,
+ 235,
+ 223,
+ 179,
+ 77,
+ 189,
+ 211,
+ 197,
+ 37,
+ 54,
+ 138,
+ 2,
+ 212,
+ 41,
+ 34,
+ 285,
+ 133
+ ],
+ [
+ 117,
+ 247,
+ 134,
+ 160,
+ 289,
+ 206,
+ 111,
+ 13,
+ 168,
+ 86,
+ 71,
+ 89,
+ 121,
+ 17,
+ 150,
+ 233,
+ 237,
+ 39,
+ 118,
+ 124
+ ],
+ [
+ 119,
+ 92,
+ 282,
+ 190,
+ 155,
+ 255,
+ 152,
+ 26,
+ 294,
+ 208,
+ 258,
+ 157,
+ 227,
+ 113,
+ 238,
+ 132,
+ 10,
+ 195,
+ 94,
+ 165
+ ],
+ [
+ 101,
+ 41,
+ 221,
+ 197,
+ 138,
+ 37,
+ 211,
+ 129,
+ 277,
+ 203,
+ 34,
+ 217,
+ 189,
+ 54,
+ 235,
+ 268,
+ 2,
+ 263,
+ 212,
+ 133
+ ],
+ [
+ 266,
+ 35,
+ 108,
+ 192,
+ 33,
+ 280,
+ 50,
+ 171,
+ 70,
+ 11,
+ 139,
+ 69,
+ 30,
+ 297,
+ 232,
+ 245,
+ 284,
+ 170,
+ 248,
+ 186
+ ],
+ [
+ 0,
+ 273,
+ 292,
+ 166,
+ 252,
+ 216,
+ 153,
+ 151,
+ 59,
+ 227,
+ 40,
+ 261,
+ 82,
+ 87,
+ 142,
+ 255,
+ 208,
+ 293,
+ 103,
+ 132
+ ],
+ [
+ 102,
+ 278,
+ 121,
+ 128,
+ 233,
+ 109,
+ 267,
+ 88,
+ 234,
+ 253,
+ 60,
+ 131,
+ 39,
+ 259,
+ 145,
+ 55,
+ 38,
+ 185,
+ 86,
+ 91
+ ],
+ [
+ 7,
+ 275,
+ 49,
+ 136,
+ 154,
+ 141,
+ 240,
+ 66,
+ 246,
+ 199,
+ 20,
+ 58,
+ 44,
+ 147,
+ 23,
+ 53,
+ 36,
+ 135,
+ 188,
+ 95
+ ],
+ [
+ 299,
+ 160,
+ 112,
+ 206,
+ 13,
+ 111,
+ 128,
+ 55,
+ 88,
+ 17,
+ 237,
+ 121,
+ 102,
+ 71,
+ 253,
+ 145,
+ 278,
+ 150,
+ 57,
+ 233
+ ],
+ [
+ 91,
+ 62,
+ 278,
+ 18,
+ 60,
+ 131,
+ 123,
+ 121,
+ 253,
+ 102,
+ 259,
+ 239,
+ 145,
+ 254,
+ 182,
+ 124,
+ 137,
+ 267,
+ 88,
+ 74
+ ],
+ [
+ 287,
+ 196,
+ 248,
+ 232,
+ 30,
+ 284,
+ 35,
+ 139,
+ 50,
+ 11,
+ 180,
+ 192,
+ 33,
+ 170,
+ 70,
+ 280,
+ 108,
+ 266,
+ 171,
+ 69
+ ],
+ [
+ 110,
+ 80,
+ 214,
+ 87,
+ 272,
+ 149,
+ 208,
+ 274,
+ 166,
+ 261,
+ 290,
+ 210,
+ 142,
+ 104,
+ 174,
+ 227,
+ 252,
+ 153,
+ 10,
+ 72
+ ],
+ [
+ 16,
+ 73,
+ 68,
+ 66,
+ 246,
+ 169,
+ 43,
+ 3,
+ 141,
+ 143,
+ 204,
+ 96,
+ 7,
+ 1,
+ 72,
+ 76,
+ 8,
+ 183,
+ 193,
+ 147
+ ],
+ [
+ 203,
+ 2,
+ 211,
+ 179,
+ 37,
+ 221,
+ 162,
+ 235,
+ 129,
+ 176,
+ 101,
+ 268,
+ 197,
+ 34,
+ 236,
+ 133,
+ 212,
+ 138,
+ 54,
+ 277
+ ],
+ [
+ 258,
+ 92,
+ 119,
+ 113,
+ 26,
+ 178,
+ 282,
+ 132,
+ 157,
+ 294,
+ 94,
+ 208,
+ 152,
+ 149,
+ 227,
+ 10,
+ 250,
+ 87,
+ 40,
+ 174
+ ],
+ [
+ 229,
+ 51,
+ 47,
+ 84,
+ 218,
+ 257,
+ 31,
+ 146,
+ 116,
+ 85,
+ 28,
+ 279,
+ 217,
+ 219,
+ 24,
+ 130,
+ 205,
+ 244,
+ 241,
+ 64
+ ],
+ [
+ 50,
+ 35,
+ 280,
+ 180,
+ 232,
+ 108,
+ 266,
+ 69,
+ 139,
+ 33,
+ 297,
+ 171,
+ 11,
+ 70,
+ 245,
+ 30,
+ 284,
+ 186,
+ 248,
+ 196
+ ],
+ [
+ 1,
+ 76,
+ 204,
+ 3,
+ 125,
+ 290,
+ 43,
+ 72,
+ 143,
+ 16,
+ 104,
+ 106,
+ 272,
+ 6,
+ 161,
+ 67,
+ 96,
+ 80,
+ 295,
+ 142
+ ],
+ [
+ 90,
+ 215,
+ 167,
+ 260,
+ 243,
+ 296,
+ 99,
+ 263,
+ 291,
+ 268,
+ 236,
+ 277,
+ 54,
+ 276,
+ 41,
+ 133,
+ 9,
+ 129,
+ 37,
+ 25
+ ],
+ [
+ 227,
+ 178,
+ 98,
+ 140,
+ 92,
+ 251,
+ 269,
+ 283,
+ 208,
+ 293,
+ 172,
+ 152,
+ 165,
+ 282,
+ 200,
+ 21,
+ 238,
+ 190,
+ 119,
+ 87
+ ],
+ [
+ 287,
+ 186,
+ 284,
+ 248,
+ 232,
+ 30,
+ 35,
+ 50,
+ 139,
+ 180,
+ 192,
+ 11,
+ 170,
+ 33,
+ 70,
+ 280,
+ 266,
+ 108,
+ 171,
+ 69
+ ],
+ [
+ 211,
+ 37,
+ 133,
+ 221,
+ 212,
+ 203,
+ 129,
+ 223,
+ 101,
+ 41,
+ 179,
+ 263,
+ 268,
+ 291,
+ 235,
+ 138,
+ 34,
+ 2,
+ 54,
+ 277
+ ],
+ [
+ 75,
+ 14,
+ 46,
+ 115,
+ 231,
+ 48,
+ 67,
+ 204,
+ 3,
+ 73,
+ 68,
+ 295,
+ 16,
+ 130,
+ 214,
+ 1,
+ 143,
+ 158,
+ 188,
+ 72
+ ],
+ [
+ 44,
+ 154,
+ 240,
+ 275,
+ 20,
+ 226,
+ 49,
+ 135,
+ 36,
+ 45,
+ 127,
+ 23,
+ 183,
+ 61,
+ 58,
+ 7,
+ 32,
+ 95,
+ 136,
+ 207
+ ],
+ [
+ 165,
+ 152,
+ 10,
+ 293,
+ 119,
+ 283,
+ 208,
+ 255,
+ 92,
+ 227,
+ 251,
+ 178,
+ 294,
+ 103,
+ 282,
+ 97,
+ 190,
+ 26,
+ 81,
+ 140
+ ],
+ [
+ 29,
+ 97,
+ 271,
+ 93,
+ 209,
+ 56,
+ 288,
+ 159,
+ 122,
+ 258,
+ 190,
+ 94,
+ 113,
+ 40,
+ 26,
+ 82,
+ 149,
+ 283,
+ 282,
+ 152
+ ],
+ [
+ 228,
+ 230,
+ 83,
+ 175,
+ 124,
+ 62,
+ 298,
+ 286,
+ 137,
+ 109,
+ 123,
+ 57,
+ 18,
+ 60,
+ 267,
+ 185,
+ 15,
+ 239,
+ 63,
+ 131
+ ],
+ [
+ 129,
+ 211,
+ 223,
+ 37,
+ 138,
+ 197,
+ 2,
+ 212,
+ 221,
+ 133,
+ 77,
+ 179,
+ 189,
+ 41,
+ 176,
+ 235,
+ 54,
+ 101,
+ 291,
+ 34
+ ],
+ [
+ 3,
+ 143,
+ 1,
+ 76,
+ 193,
+ 43,
+ 6,
+ 295,
+ 161,
+ 115,
+ 48,
+ 8,
+ 16,
+ 110,
+ 72,
+ 272,
+ 68,
+ 104,
+ 169,
+ 125
+ ],
+ [
+ 31,
+ 84,
+ 257,
+ 130,
+ 219,
+ 218,
+ 224,
+ 173,
+ 116,
+ 270,
+ 265,
+ 229,
+ 210,
+ 51,
+ 191,
+ 24,
+ 100,
+ 244,
+ 146,
+ 85
+ ],
+ [
+ 160,
+ 299,
+ 111,
+ 13,
+ 128,
+ 184,
+ 17,
+ 112,
+ 55,
+ 88,
+ 278,
+ 57,
+ 145,
+ 267,
+ 253,
+ 102,
+ 137,
+ 89,
+ 182,
+ 121
+ ],
+ [
+ 135,
+ 36,
+ 58,
+ 127,
+ 226,
+ 23,
+ 44,
+ 61,
+ 65,
+ 154,
+ 45,
+ 32,
+ 275,
+ 53,
+ 199,
+ 240,
+ 49,
+ 95,
+ 136,
+ 20
+ ],
+ [
+ 227,
+ 92,
+ 10,
+ 261,
+ 119,
+ 282,
+ 294,
+ 251,
+ 26,
+ 151,
+ 178,
+ 132,
+ 293,
+ 149,
+ 87,
+ 174,
+ 190,
+ 152,
+ 21,
+ 113
+ ],
+ [
+ 271,
+ 97,
+ 56,
+ 288,
+ 29,
+ 201,
+ 82,
+ 93,
+ 40,
+ 258,
+ 159,
+ 255,
+ 282,
+ 149,
+ 119,
+ 122,
+ 190,
+ 26,
+ 200,
+ 283
+ ],
+ [
+ 214,
+ 72,
+ 272,
+ 187,
+ 149,
+ 87,
+ 110,
+ 290,
+ 274,
+ 80,
+ 252,
+ 82,
+ 227,
+ 173,
+ 142,
+ 40,
+ 8,
+ 181,
+ 261,
+ 219
+ ],
+ [
+ 197,
+ 223,
+ 235,
+ 2,
+ 203,
+ 221,
+ 34,
+ 129,
+ 37,
+ 179,
+ 162,
+ 189,
+ 212,
+ 268,
+ 133,
+ 101,
+ 138,
+ 176,
+ 41,
+ 217
+ ],
+ [
+ 133,
+ 37,
+ 41,
+ 197,
+ 291,
+ 203,
+ 236,
+ 9,
+ 263,
+ 77,
+ 268,
+ 138,
+ 211,
+ 54,
+ 221,
+ 179,
+ 101,
+ 129,
+ 2,
+ 296
+ ],
+ [
+ 22,
+ 42,
+ 5,
+ 148,
+ 15,
+ 126,
+ 239,
+ 118,
+ 175,
+ 39,
+ 298,
+ 123,
+ 163,
+ 168,
+ 259,
+ 105,
+ 131,
+ 247,
+ 18,
+ 74
+ ],
+ [
+ 210,
+ 272,
+ 72,
+ 187,
+ 110,
+ 80,
+ 87,
+ 8,
+ 142,
+ 290,
+ 252,
+ 67,
+ 227,
+ 149,
+ 274,
+ 161,
+ 295,
+ 104,
+ 174,
+ 261
+ ],
+ [
+ 260,
+ 263,
+ 291,
+ 296,
+ 194,
+ 90,
+ 268,
+ 41,
+ 167,
+ 236,
+ 277,
+ 133,
+ 243,
+ 212,
+ 37,
+ 9,
+ 276,
+ 129,
+ 54,
+ 197
+ ],
+ [
+ 166,
+ 151,
+ 181,
+ 0,
+ 273,
+ 292,
+ 103,
+ 153,
+ 81,
+ 261,
+ 10,
+ 255,
+ 40,
+ 208,
+ 119,
+ 252,
+ 293,
+ 82,
+ 157,
+ 26
+ ],
+ [
+ 279,
+ 162,
+ 129,
+ 34,
+ 179,
+ 223,
+ 276,
+ 235,
+ 277,
+ 211,
+ 221,
+ 41,
+ 37,
+ 203,
+ 189,
+ 197,
+ 268,
+ 138,
+ 281,
+ 263
+ ],
+ [
+ 84,
+ 31,
+ 116,
+ 229,
+ 270,
+ 191,
+ 130,
+ 24,
+ 51,
+ 146,
+ 219,
+ 12,
+ 205,
+ 47,
+ 64,
+ 28,
+ 257,
+ 85,
+ 279,
+ 114
+ ],
+ [
+ 158,
+ 210,
+ 173,
+ 130,
+ 214,
+ 270,
+ 67,
+ 218,
+ 72,
+ 205,
+ 252,
+ 290,
+ 84,
+ 68,
+ 272,
+ 224,
+ 76,
+ 187,
+ 169,
+ 142
+ ],
+ [
+ 120,
+ 20,
+ 24,
+ 240,
+ 199,
+ 65,
+ 7,
+ 32,
+ 265,
+ 136,
+ 275,
+ 246,
+ 244,
+ 135,
+ 44,
+ 224,
+ 226,
+ 36,
+ 49,
+ 188
+ ],
+ [
+ 197,
+ 129,
+ 211,
+ 101,
+ 179,
+ 235,
+ 223,
+ 41,
+ 203,
+ 37,
+ 133,
+ 162,
+ 2,
+ 176,
+ 34,
+ 189,
+ 212,
+ 268,
+ 54,
+ 77
+ ],
+ [
+ 208,
+ 174,
+ 113,
+ 26,
+ 166,
+ 132,
+ 40,
+ 151,
+ 119,
+ 92,
+ 261,
+ 258,
+ 227,
+ 178,
+ 190,
+ 149,
+ 294,
+ 87,
+ 152,
+ 255
+ ],
+ [
+ 129,
+ 211,
+ 235,
+ 203,
+ 197,
+ 34,
+ 221,
+ 162,
+ 37,
+ 2,
+ 279,
+ 217,
+ 176,
+ 179,
+ 41,
+ 285,
+ 138,
+ 101,
+ 212,
+ 277
+ ],
+ [
+ 265,
+ 173,
+ 244,
+ 205,
+ 219,
+ 24,
+ 84,
+ 31,
+ 218,
+ 210,
+ 65,
+ 100,
+ 270,
+ 220,
+ 116,
+ 120,
+ 72,
+ 122,
+ 264,
+ 136
+ ],
+ [
+ 170,
+ 33,
+ 11,
+ 281,
+ 70,
+ 180,
+ 179,
+ 54,
+ 171,
+ 280,
+ 129,
+ 266,
+ 243,
+ 156,
+ 279,
+ 211,
+ 108,
+ 235,
+ 25,
+ 203
+ ],
+ [
+ 127,
+ 61,
+ 44,
+ 45,
+ 207,
+ 23,
+ 135,
+ 36,
+ 32,
+ 58,
+ 199,
+ 154,
+ 275,
+ 49,
+ 20,
+ 240,
+ 95,
+ 65,
+ 53,
+ 19
+ ],
+ [
+ 87,
+ 208,
+ 92,
+ 132,
+ 261,
+ 152,
+ 282,
+ 40,
+ 273,
+ 26,
+ 149,
+ 174,
+ 119,
+ 251,
+ 10,
+ 178,
+ 59,
+ 255,
+ 0,
+ 113
+ ],
+ [
+ 202,
+ 230,
+ 83,
+ 175,
+ 124,
+ 62,
+ 298,
+ 286,
+ 123,
+ 137,
+ 109,
+ 18,
+ 15,
+ 60,
+ 267,
+ 185,
+ 57,
+ 239,
+ 63,
+ 91
+ ],
+ [
+ 51,
+ 84,
+ 85,
+ 47,
+ 191,
+ 218,
+ 31,
+ 257,
+ 28,
+ 116,
+ 146,
+ 64,
+ 279,
+ 130,
+ 217,
+ 12,
+ 270,
+ 281,
+ 205,
+ 162
+ ],
+ [
+ 83,
+ 175,
+ 62,
+ 124,
+ 298,
+ 123,
+ 239,
+ 18,
+ 15,
+ 137,
+ 109,
+ 185,
+ 131,
+ 286,
+ 60,
+ 5,
+ 118,
+ 267,
+ 259,
+ 91
+ ],
+ [
+ 14,
+ 75,
+ 198,
+ 115,
+ 46,
+ 48,
+ 214,
+ 204,
+ 262,
+ 8,
+ 3,
+ 67,
+ 68,
+ 295,
+ 73,
+ 272,
+ 161,
+ 6,
+ 143,
+ 72
+ ],
+ [
+ 50,
+ 35,
+ 196,
+ 186,
+ 192,
+ 284,
+ 139,
+ 287,
+ 248,
+ 30,
+ 180,
+ 11,
+ 280,
+ 33,
+ 108,
+ 266,
+ 70,
+ 170,
+ 69,
+ 171
+ ],
+ [
+ 102,
+ 39,
+ 182,
+ 259,
+ 88,
+ 121,
+ 278,
+ 86,
+ 131,
+ 89,
+ 128,
+ 267,
+ 71,
+ 55,
+ 13,
+ 239,
+ 253,
+ 234,
+ 118,
+ 109
+ ],
+ [
+ 102,
+ 182,
+ 86,
+ 38,
+ 145,
+ 278,
+ 55,
+ 253,
+ 121,
+ 74,
+ 233,
+ 91,
+ 128,
+ 13,
+ 88,
+ 39,
+ 131,
+ 185,
+ 259,
+ 71
+ ],
+ [
+ 211,
+ 223,
+ 34,
+ 221,
+ 129,
+ 2,
+ 197,
+ 203,
+ 162,
+ 179,
+ 189,
+ 37,
+ 176,
+ 281,
+ 217,
+ 101,
+ 279,
+ 138,
+ 285,
+ 268
+ ],
+ [
+ 133,
+ 212,
+ 37,
+ 54,
+ 291,
+ 268,
+ 41,
+ 263,
+ 129,
+ 203,
+ 77,
+ 221,
+ 189,
+ 138,
+ 197,
+ 243,
+ 296,
+ 179,
+ 2,
+ 101
+ ],
+ [
+ 27,
+ 150,
+ 38,
+ 57,
+ 13,
+ 168,
+ 253,
+ 60,
+ 160,
+ 128,
+ 102,
+ 71,
+ 112,
+ 55,
+ 17,
+ 86,
+ 278,
+ 145,
+ 233,
+ 121
+ ],
+ [
+ 178,
+ 92,
+ 152,
+ 119,
+ 251,
+ 190,
+ 294,
+ 113,
+ 94,
+ 26,
+ 227,
+ 258,
+ 208,
+ 165,
+ 98,
+ 132,
+ 155,
+ 195,
+ 222,
+ 87
+ ],
+ [
+ 123,
+ 259,
+ 131,
+ 62,
+ 39,
+ 15,
+ 118,
+ 18,
+ 5,
+ 185,
+ 254,
+ 298,
+ 91,
+ 233,
+ 102,
+ 182,
+ 148,
+ 74,
+ 22,
+ 137
+ ],
+ [
+ 20,
+ 199,
+ 49,
+ 154,
+ 275,
+ 136,
+ 44,
+ 183,
+ 36,
+ 61,
+ 23,
+ 7,
+ 226,
+ 135,
+ 65,
+ 58,
+ 127,
+ 207,
+ 32,
+ 95
+ ],
+ [
+ 164,
+ 107,
+ 34,
+ 156,
+ 279,
+ 162,
+ 235,
+ 217,
+ 211,
+ 243,
+ 268,
+ 129,
+ 281,
+ 2,
+ 223,
+ 221,
+ 28,
+ 189,
+ 25,
+ 12
+ ],
+ [
+ 249,
+ 153,
+ 293,
+ 81,
+ 251,
+ 292,
+ 98,
+ 151,
+ 181,
+ 216,
+ 103,
+ 10,
+ 166,
+ 0,
+ 252,
+ 140,
+ 208,
+ 294,
+ 227,
+ 273
+ ],
+ [
+ 268,
+ 296,
+ 236,
+ 260,
+ 34,
+ 263,
+ 2,
+ 291,
+ 179,
+ 189,
+ 54,
+ 162,
+ 133,
+ 37,
+ 90,
+ 221,
+ 129,
+ 212,
+ 41,
+ 211
+ ],
+ [
+ 24,
+ 224,
+ 173,
+ 31,
+ 265,
+ 135,
+ 116,
+ 120,
+ 65,
+ 207,
+ 218,
+ 36,
+ 84,
+ 219,
+ 220,
+ 136,
+ 191,
+ 58,
+ 95,
+ 205
+ ],
+ [
+ 69,
+ 297,
+ 108,
+ 171,
+ 280,
+ 266,
+ 70,
+ 33,
+ 192,
+ 180,
+ 11,
+ 35,
+ 139,
+ 50,
+ 225,
+ 170,
+ 232,
+ 30,
+ 186,
+ 284
+ ],
+ [
+ 141,
+ 66,
+ 7,
+ 188,
+ 183,
+ 16,
+ 275,
+ 68,
+ 199,
+ 73,
+ 154,
+ 169,
+ 49,
+ 270,
+ 95,
+ 240,
+ 20,
+ 43,
+ 44,
+ 96
+ ],
+ [
+ 289,
+ 134,
+ 39,
+ 168,
+ 42,
+ 118,
+ 148,
+ 259,
+ 86,
+ 74,
+ 254,
+ 105,
+ 123,
+ 71,
+ 298,
+ 233,
+ 89,
+ 239,
+ 121,
+ 163
+ ],
+ [
+ 284,
+ 196,
+ 30,
+ 186,
+ 287,
+ 232,
+ 50,
+ 35,
+ 139,
+ 180,
+ 192,
+ 11,
+ 170,
+ 70,
+ 33,
+ 266,
+ 108,
+ 280,
+ 171,
+ 69
+ ],
+ [
+ 242,
+ 153,
+ 181,
+ 292,
+ 216,
+ 252,
+ 81,
+ 166,
+ 0,
+ 293,
+ 251,
+ 151,
+ 187,
+ 273,
+ 142,
+ 10,
+ 158,
+ 67,
+ 261,
+ 103
+ ],
+ [
+ 190,
+ 94,
+ 110,
+ 92,
+ 113,
+ 282,
+ 178,
+ 258,
+ 161,
+ 79,
+ 78,
+ 104,
+ 152,
+ 208,
+ 26,
+ 294,
+ 21,
+ 157,
+ 269,
+ 274
+ ],
+ [
+ 293,
+ 208,
+ 10,
+ 98,
+ 227,
+ 92,
+ 294,
+ 152,
+ 81,
+ 151,
+ 21,
+ 87,
+ 178,
+ 195,
+ 119,
+ 238,
+ 261,
+ 200,
+ 172,
+ 132
+ ],
+ [
+ 142,
+ 181,
+ 166,
+ 261,
+ 273,
+ 292,
+ 0,
+ 87,
+ 80,
+ 59,
+ 227,
+ 174,
+ 290,
+ 67,
+ 153,
+ 149,
+ 40,
+ 132,
+ 216,
+ 208
+ ],
+ [
+ 145,
+ 278,
+ 88,
+ 102,
+ 121,
+ 182,
+ 91,
+ 71,
+ 185,
+ 160,
+ 267,
+ 60,
+ 128,
+ 89,
+ 86,
+ 55,
+ 234,
+ 233,
+ 38,
+ 259
+ ],
+ [
+ 185,
+ 39,
+ 239,
+ 259,
+ 137,
+ 278,
+ 121,
+ 62,
+ 60,
+ 102,
+ 131,
+ 88,
+ 123,
+ 182,
+ 118,
+ 86,
+ 89,
+ 145,
+ 233,
+ 109
+ ],
+ [
+ 119,
+ 157,
+ 282,
+ 40,
+ 178,
+ 151,
+ 132,
+ 92,
+ 103,
+ 273,
+ 152,
+ 227,
+ 0,
+ 26,
+ 294,
+ 208,
+ 59,
+ 113,
+ 87,
+ 261
+ ],
+ [
+ 265,
+ 224,
+ 99,
+ 24,
+ 120,
+ 167,
+ 244,
+ 146,
+ 220,
+ 114,
+ 218,
+ 84,
+ 116,
+ 65,
+ 219,
+ 173,
+ 31,
+ 241,
+ 270,
+ 64
+ ],
+ [
+ 51,
+ 31,
+ 229,
+ 191,
+ 84,
+ 205,
+ 85,
+ 218,
+ 116,
+ 47,
+ 146,
+ 130,
+ 28,
+ 270,
+ 219,
+ 244,
+ 24,
+ 224,
+ 64,
+ 173
+ ],
+ [
+ 190,
+ 113,
+ 26,
+ 282,
+ 178,
+ 94,
+ 152,
+ 119,
+ 132,
+ 92,
+ 288,
+ 149,
+ 97,
+ 208,
+ 222,
+ 40,
+ 157,
+ 209,
+ 271,
+ 294
+ ],
+ [
+ 131,
+ 123,
+ 239,
+ 39,
+ 233,
+ 102,
+ 18,
+ 118,
+ 185,
+ 62,
+ 182,
+ 86,
+ 15,
+ 89,
+ 121,
+ 88,
+ 254,
+ 278,
+ 91,
+ 71
+ ],
+ [
+ 263,
+ 291,
+ 296,
+ 268,
+ 9,
+ 277,
+ 133,
+ 197,
+ 41,
+ 215,
+ 54,
+ 37,
+ 243,
+ 179,
+ 129,
+ 212,
+ 236,
+ 138,
+ 34,
+ 211
+ ],
+ [
+ 80,
+ 208,
+ 149,
+ 132,
+ 87,
+ 174,
+ 59,
+ 227,
+ 142,
+ 166,
+ 92,
+ 26,
+ 282,
+ 113,
+ 157,
+ 10,
+ 40,
+ 252,
+ 119,
+ 273
+ ],
+ [
+ 8,
+ 106,
+ 78,
+ 161,
+ 79,
+ 43,
+ 6,
+ 143,
+ 110,
+ 125,
+ 104,
+ 204,
+ 272,
+ 169,
+ 48,
+ 3,
+ 72,
+ 264,
+ 68,
+ 1
+ ],
+ [
+ 291,
+ 260,
+ 268,
+ 41,
+ 296,
+ 133,
+ 37,
+ 197,
+ 277,
+ 129,
+ 212,
+ 54,
+ 236,
+ 9,
+ 179,
+ 138,
+ 203,
+ 221,
+ 34,
+ 211
+ ],
+ [
+ 100,
+ 8,
+ 6,
+ 79,
+ 262,
+ 169,
+ 78,
+ 201,
+ 43,
+ 97,
+ 161,
+ 125,
+ 72,
+ 110,
+ 106,
+ 173,
+ 210,
+ 104,
+ 272,
+ 274
+ ],
+ [
+ 224,
+ 24,
+ 244,
+ 205,
+ 256,
+ 218,
+ 84,
+ 173,
+ 120,
+ 146,
+ 31,
+ 220,
+ 219,
+ 116,
+ 65,
+ 270,
+ 114,
+ 257,
+ 210,
+ 191
+ ],
+ [
+ 171,
+ 280,
+ 180,
+ 108,
+ 69,
+ 245,
+ 297,
+ 33,
+ 70,
+ 192,
+ 11,
+ 35,
+ 50,
+ 139,
+ 30,
+ 232,
+ 225,
+ 170,
+ 284,
+ 248
+ ],
+ [
+ 128,
+ 109,
+ 278,
+ 60,
+ 102,
+ 182,
+ 88,
+ 253,
+ 233,
+ 121,
+ 124,
+ 185,
+ 137,
+ 160,
+ 91,
+ 131,
+ 13,
+ 18,
+ 259,
+ 145
+ ],
+ [
+ 263,
+ 296,
+ 260,
+ 37,
+ 197,
+ 133,
+ 291,
+ 41,
+ 34,
+ 129,
+ 277,
+ 236,
+ 243,
+ 212,
+ 211,
+ 221,
+ 179,
+ 162,
+ 189,
+ 54
+ ],
+ [
+ 195,
+ 283,
+ 94,
+ 250,
+ 190,
+ 21,
+ 238,
+ 178,
+ 92,
+ 227,
+ 152,
+ 274,
+ 258,
+ 208,
+ 165,
+ 251,
+ 140,
+ 110,
+ 98,
+ 161
+ ],
+ [
+ 12,
+ 130,
+ 84,
+ 218,
+ 116,
+ 31,
+ 219,
+ 246,
+ 141,
+ 229,
+ 66,
+ 95,
+ 68,
+ 7,
+ 65,
+ 275,
+ 205,
+ 114,
+ 58,
+ 24
+ ],
+ [
+ 288,
+ 97,
+ 209,
+ 56,
+ 201,
+ 29,
+ 93,
+ 258,
+ 40,
+ 82,
+ 282,
+ 157,
+ 255,
+ 122,
+ 113,
+ 159,
+ 132,
+ 190,
+ 152,
+ 119
+ ],
+ [
+ 72,
+ 80,
+ 214,
+ 110,
+ 290,
+ 87,
+ 161,
+ 8,
+ 149,
+ 274,
+ 142,
+ 106,
+ 210,
+ 187,
+ 104,
+ 174,
+ 261,
+ 6,
+ 227,
+ 204
+ ],
+ [
+ 0,
+ 40,
+ 181,
+ 166,
+ 227,
+ 292,
+ 255,
+ 132,
+ 151,
+ 216,
+ 87,
+ 26,
+ 261,
+ 252,
+ 157,
+ 119,
+ 59,
+ 174,
+ 208,
+ 92
+ ],
+ [
+ 149,
+ 174,
+ 87,
+ 113,
+ 92,
+ 110,
+ 80,
+ 227,
+ 132,
+ 26,
+ 208,
+ 272,
+ 152,
+ 187,
+ 190,
+ 261,
+ 94,
+ 161,
+ 40,
+ 258
+ ],
+ [
+ 154,
+ 44,
+ 199,
+ 49,
+ 183,
+ 58,
+ 135,
+ 240,
+ 127,
+ 226,
+ 23,
+ 20,
+ 95,
+ 61,
+ 136,
+ 36,
+ 7,
+ 32,
+ 207,
+ 246
+ ],
+ [
+ 217,
+ 129,
+ 279,
+ 162,
+ 41,
+ 277,
+ 34,
+ 263,
+ 203,
+ 138,
+ 37,
+ 291,
+ 25,
+ 179,
+ 236,
+ 268,
+ 223,
+ 296,
+ 212,
+ 221
+ ],
+ [
+ 263,
+ 129,
+ 179,
+ 268,
+ 41,
+ 197,
+ 101,
+ 260,
+ 37,
+ 291,
+ 203,
+ 221,
+ 279,
+ 217,
+ 34,
+ 296,
+ 133,
+ 211,
+ 54,
+ 276
+ ],
+ [
+ 102,
+ 182,
+ 253,
+ 145,
+ 88,
+ 121,
+ 128,
+ 185,
+ 267,
+ 60,
+ 91,
+ 55,
+ 109,
+ 86,
+ 160,
+ 233,
+ 234,
+ 38,
+ 13,
+ 131
+ ],
+ [
+ 217,
+ 162,
+ 129,
+ 34,
+ 223,
+ 25,
+ 277,
+ 235,
+ 276,
+ 221,
+ 268,
+ 281,
+ 203,
+ 211,
+ 189,
+ 156,
+ 179,
+ 107,
+ 41,
+ 37
+ ],
+ [
+ 108,
+ 266,
+ 171,
+ 33,
+ 245,
+ 192,
+ 297,
+ 69,
+ 70,
+ 180,
+ 11,
+ 35,
+ 50,
+ 139,
+ 232,
+ 30,
+ 225,
+ 170,
+ 186,
+ 284
+ ],
+ [
+ 235,
+ 279,
+ 156,
+ 34,
+ 217,
+ 223,
+ 129,
+ 162,
+ 189,
+ 25,
+ 211,
+ 28,
+ 203,
+ 176,
+ 2,
+ 179,
+ 221,
+ 64,
+ 138,
+ 37
+ ],
+ [
+ 178,
+ 119,
+ 157,
+ 92,
+ 26,
+ 132,
+ 255,
+ 190,
+ 227,
+ 208,
+ 258,
+ 59,
+ 113,
+ 261,
+ 152,
+ 149,
+ 40,
+ 87,
+ 94,
+ 294
+ ],
+ [
+ 200,
+ 152,
+ 190,
+ 195,
+ 92,
+ 94,
+ 165,
+ 227,
+ 178,
+ 294,
+ 258,
+ 149,
+ 274,
+ 97,
+ 87,
+ 132,
+ 282,
+ 140,
+ 119,
+ 269
+ ],
+ [
+ 248,
+ 196,
+ 30,
+ 287,
+ 232,
+ 186,
+ 35,
+ 50,
+ 139,
+ 180,
+ 192,
+ 170,
+ 11,
+ 266,
+ 33,
+ 280,
+ 108,
+ 70,
+ 171,
+ 69
+ ],
+ [
+ 223,
+ 197,
+ 203,
+ 37,
+ 211,
+ 77,
+ 235,
+ 129,
+ 101,
+ 176,
+ 138,
+ 212,
+ 34,
+ 133,
+ 2,
+ 41,
+ 277,
+ 179,
+ 221,
+ 291
+ ],
+ [
+ 91,
+ 62,
+ 185,
+ 124,
+ 60,
+ 109,
+ 278,
+ 18,
+ 137,
+ 145,
+ 74,
+ 63,
+ 253,
+ 131,
+ 267,
+ 128,
+ 182,
+ 17,
+ 102,
+ 298
+ ],
+ [
+ 196,
+ 186,
+ 284,
+ 248,
+ 232,
+ 30,
+ 35,
+ 139,
+ 50,
+ 180,
+ 170,
+ 192,
+ 11,
+ 33,
+ 70,
+ 108,
+ 280,
+ 266,
+ 171,
+ 225
+ ],
+ [
+ 271,
+ 97,
+ 209,
+ 258,
+ 56,
+ 201,
+ 282,
+ 29,
+ 113,
+ 157,
+ 255,
+ 40,
+ 93,
+ 132,
+ 190,
+ 119,
+ 26,
+ 152,
+ 149,
+ 178
+ ],
+ [
+ 247,
+ 134,
+ 168,
+ 86,
+ 74,
+ 148,
+ 150,
+ 39,
+ 71,
+ 89,
+ 105,
+ 254,
+ 17,
+ 259,
+ 137,
+ 278,
+ 42,
+ 118,
+ 88,
+ 121
+ ],
+ [
+ 142,
+ 80,
+ 149,
+ 272,
+ 104,
+ 261,
+ 174,
+ 72,
+ 87,
+ 110,
+ 132,
+ 161,
+ 59,
+ 252,
+ 193,
+ 76,
+ 40,
+ 166,
+ 187,
+ 106
+ ],
+ [
+ 263,
+ 260,
+ 133,
+ 41,
+ 212,
+ 296,
+ 37,
+ 268,
+ 9,
+ 197,
+ 236,
+ 54,
+ 138,
+ 129,
+ 203,
+ 179,
+ 277,
+ 221,
+ 211,
+ 77
+ ],
+ [
+ 181,
+ 273,
+ 0,
+ 166,
+ 40,
+ 216,
+ 252,
+ 153,
+ 151,
+ 87,
+ 255,
+ 227,
+ 132,
+ 261,
+ 82,
+ 103,
+ 293,
+ 81,
+ 10,
+ 149
+ ],
+ [
+ 98,
+ 10,
+ 251,
+ 103,
+ 208,
+ 81,
+ 294,
+ 92,
+ 119,
+ 200,
+ 151,
+ 227,
+ 181,
+ 178,
+ 172,
+ 195,
+ 0,
+ 282,
+ 255,
+ 166
+ ],
+ [
+ 92,
+ 10,
+ 119,
+ 208,
+ 178,
+ 190,
+ 98,
+ 251,
+ 132,
+ 152,
+ 26,
+ 157,
+ 113,
+ 293,
+ 255,
+ 282,
+ 151,
+ 227,
+ 261,
+ 103
+ ],
+ [
+ 76,
+ 3,
+ 204,
+ 67,
+ 110,
+ 1,
+ 272,
+ 142,
+ 290,
+ 72,
+ 80,
+ 161,
+ 143,
+ 214,
+ 68,
+ 8,
+ 96,
+ 104,
+ 169,
+ 187
+ ],
+ [
+ 268,
+ 260,
+ 263,
+ 291,
+ 243,
+ 37,
+ 197,
+ 179,
+ 41,
+ 133,
+ 34,
+ 9,
+ 277,
+ 212,
+ 138,
+ 221,
+ 162,
+ 129,
+ 236,
+ 215
+ ],
+ [
+ 69,
+ 245,
+ 108,
+ 280,
+ 171,
+ 266,
+ 192,
+ 70,
+ 33,
+ 180,
+ 50,
+ 11,
+ 35,
+ 139,
+ 232,
+ 225,
+ 170,
+ 30,
+ 186,
+ 284
+ ],
+ [
+ 62,
+ 22,
+ 18,
+ 239,
+ 42,
+ 5,
+ 74,
+ 118,
+ 105,
+ 131,
+ 123,
+ 15,
+ 39,
+ 148,
+ 175,
+ 137,
+ 259,
+ 185,
+ 124,
+ 254
+ ],
+ [
+ 184,
+ 206,
+ 160,
+ 112,
+ 111,
+ 13,
+ 17,
+ 128,
+ 57,
+ 55,
+ 88,
+ 237,
+ 253,
+ 71,
+ 145,
+ 89,
+ 278,
+ 267,
+ 102,
+ 150
+ ]
+ ],
+ "adtIndices": [
+ [
+ 283,
+ 155,
+ 157,
+ 178,
+ 119,
+ 174,
+ 140,
+ 132,
+ 208,
+ 87,
+ 294,
+ 292,
+ 10,
+ 227,
+ 238,
+ 92,
+ 282,
+ 258,
+ 81,
+ 26
+ ],
+ [
+ 190,
+ 3,
+ 125,
+ 142,
+ 6,
+ 295,
+ 193,
+ 8,
+ 96,
+ 161,
+ 59,
+ 169,
+ 80,
+ 141,
+ 43,
+ 68,
+ 66,
+ 274,
+ 76,
+ 78
+ ],
+ [
+ 9,
+ 296,
+ 203,
+ 236,
+ 221,
+ 212,
+ 197,
+ 101,
+ 37,
+ 279,
+ 235,
+ 277,
+ 217,
+ 211,
+ 162,
+ 263,
+ 167,
+ 291,
+ 285,
+ 77
+ ],
+ [
+ 96,
+ 8,
+ 193,
+ 125,
+ 295,
+ 66,
+ 169,
+ 68,
+ 141,
+ 142,
+ 6,
+ 161,
+ 59,
+ 76,
+ 43,
+ 188,
+ 80,
+ 72,
+ 204,
+ 1
+ ],
+ [
+ 86,
+ 91,
+ 124,
+ 278,
+ 62,
+ 74,
+ 39,
+ 118,
+ 237,
+ 131,
+ 259,
+ 123,
+ 298,
+ 38,
+ 230,
+ 13,
+ 105,
+ 234,
+ 22,
+ 17
+ ],
+ [
+ 27,
+ 60,
+ 13,
+ 128,
+ 111,
+ 278,
+ 109,
+ 148,
+ 63,
+ 145,
+ 298,
+ 237,
+ 137,
+ 160,
+ 112,
+ 206,
+ 254,
+ 131,
+ 38,
+ 39
+ ],
+ [
+ 125,
+ 295,
+ 76,
+ 3,
+ 193,
+ 169,
+ 8,
+ 96,
+ 204,
+ 188,
+ 1,
+ 142,
+ 68,
+ 161,
+ 75,
+ 59,
+ 43,
+ 141,
+ 79,
+ 110
+ ],
+ [
+ 84,
+ 240,
+ 58,
+ 36,
+ 127,
+ 264,
+ 100,
+ 246,
+ 205,
+ 61,
+ 23,
+ 20,
+ 218,
+ 257,
+ 224,
+ 95,
+ 200,
+ 53,
+ 93,
+ 288
+ ],
+ [
+ 68,
+ 295,
+ 3,
+ 125,
+ 66,
+ 141,
+ 96,
+ 43,
+ 72,
+ 193,
+ 169,
+ 274,
+ 188,
+ 204,
+ 76,
+ 80,
+ 78,
+ 161,
+ 79,
+ 183
+ ],
+ [
+ 236,
+ 212,
+ 197,
+ 221,
+ 277,
+ 211,
+ 263,
+ 2,
+ 167,
+ 217,
+ 37,
+ 133,
+ 162,
+ 176,
+ 285,
+ 77,
+ 296,
+ 235,
+ 291,
+ 138
+ ],
+ [
+ 294,
+ 238,
+ 174,
+ 132,
+ 251,
+ 222,
+ 140,
+ 208,
+ 165,
+ 283,
+ 178,
+ 181,
+ 242,
+ 227,
+ 157,
+ 252,
+ 255,
+ 0,
+ 26,
+ 92
+ ],
+ [
+ 70,
+ 35,
+ 186,
+ 232,
+ 50,
+ 33,
+ 266,
+ 245,
+ 180,
+ 280,
+ 171,
+ 196,
+ 192,
+ 69,
+ 108,
+ 297,
+ 284,
+ 139,
+ 30,
+ 248
+ ],
+ [
+ 156,
+ 114,
+ 49,
+ 23,
+ 177,
+ 117,
+ 247,
+ 144,
+ 84,
+ 219,
+ 202,
+ 229,
+ 100,
+ 257,
+ 16,
+ 7,
+ 31,
+ 218,
+ 264,
+ 164
+ ],
+ [
+ 60,
+ 5,
+ 278,
+ 160,
+ 27,
+ 145,
+ 111,
+ 131,
+ 237,
+ 17,
+ 109,
+ 206,
+ 128,
+ 137,
+ 148,
+ 112,
+ 298,
+ 102,
+ 299,
+ 63
+ ],
+ [
+ 198,
+ 115,
+ 75,
+ 231,
+ 190,
+ 46,
+ 262,
+ 214,
+ 79,
+ 161,
+ 67,
+ 274,
+ 80,
+ 130,
+ 169,
+ 290,
+ 6,
+ 78,
+ 1,
+ 43
+ ],
+ [
+ 22,
+ 42,
+ 105,
+ 118,
+ 123,
+ 38,
+ 124,
+ 63,
+ 88,
+ 239,
+ 148,
+ 175,
+ 298,
+ 286,
+ 233,
+ 112,
+ 145,
+ 71,
+ 278,
+ 86
+ ],
+ [
+ 147,
+ 78,
+ 262,
+ 43,
+ 66,
+ 68,
+ 219,
+ 73,
+ 274,
+ 8,
+ 141,
+ 161,
+ 130,
+ 134,
+ 72,
+ 1,
+ 3,
+ 125,
+ 80,
+ 193
+ ],
+ [
+ 234,
+ 89,
+ 267,
+ 13,
+ 102,
+ 206,
+ 160,
+ 74,
+ 184,
+ 299,
+ 112,
+ 131,
+ 145,
+ 254,
+ 253,
+ 148,
+ 60,
+ 124,
+ 259,
+ 278
+ ],
+ [
+ 267,
+ 91,
+ 230,
+ 299,
+ 89,
+ 128,
+ 4,
+ 38,
+ 74,
+ 234,
+ 150,
+ 175,
+ 112,
+ 148,
+ 160,
+ 124,
+ 111,
+ 17,
+ 137,
+ 278
+ ],
+ [
+ 65,
+ 120,
+ 199,
+ 32,
+ 95,
+ 136,
+ 20,
+ 116,
+ 44,
+ 53,
+ 154,
+ 275,
+ 45,
+ 240,
+ 220,
+ 226,
+ 61,
+ 135,
+ 127,
+ 24
+ ],
+ [
+ 240,
+ 19,
+ 32,
+ 154,
+ 95,
+ 61,
+ 136,
+ 199,
+ 58,
+ 127,
+ 53,
+ 246,
+ 65,
+ 120,
+ 23,
+ 36,
+ 264,
+ 116,
+ 256,
+ 275
+ ],
+ [
+ 273,
+ 227,
+ 251,
+ 165,
+ 252,
+ 222,
+ 208,
+ 242,
+ 103,
+ 174,
+ 81,
+ 216,
+ 10,
+ 132,
+ 238,
+ 151,
+ 119,
+ 294,
+ 261,
+ 157
+ ],
+ [
+ 42,
+ 15,
+ 239,
+ 286,
+ 63,
+ 124,
+ 123,
+ 38,
+ 105,
+ 298,
+ 233,
+ 112,
+ 148,
+ 145,
+ 71,
+ 234,
+ 118,
+ 267,
+ 278,
+ 254
+ ],
+ [
+ 100,
+ 84,
+ 95,
+ 224,
+ 61,
+ 58,
+ 127,
+ 264,
+ 53,
+ 240,
+ 246,
+ 36,
+ 154,
+ 136,
+ 32,
+ 159,
+ 209,
+ 244,
+ 64,
+ 207
+ ],
+ [
+ 226,
+ 220,
+ 275,
+ 44,
+ 65,
+ 272,
+ 120,
+ 110,
+ 106,
+ 19,
+ 48,
+ 76,
+ 295,
+ 166,
+ 193,
+ 135,
+ 113,
+ 169,
+ 210,
+ 250
+ ],
+ [
+ 189,
+ 34,
+ 281,
+ 203,
+ 235,
+ 162,
+ 296,
+ 90,
+ 54,
+ 221,
+ 291,
+ 2,
+ 197,
+ 101,
+ 167,
+ 176,
+ 194,
+ 263,
+ 179,
+ 9
+ ],
+ [
+ 174,
+ 222,
+ 208,
+ 132,
+ 119,
+ 165,
+ 157,
+ 294,
+ 251,
+ 81,
+ 216,
+ 227,
+ 10,
+ 178,
+ 151,
+ 181,
+ 238,
+ 255,
+ 187,
+ 292
+ ],
+ [
+ 5,
+ 128,
+ 60,
+ 111,
+ 148,
+ 13,
+ 63,
+ 298,
+ 109,
+ 145,
+ 137,
+ 278,
+ 206,
+ 237,
+ 254,
+ 112,
+ 160,
+ 57,
+ 38,
+ 299
+ ],
+ [
+ 146,
+ 64,
+ 191,
+ 82,
+ 224,
+ 159,
+ 51,
+ 47,
+ 265,
+ 93,
+ 23,
+ 56,
+ 244,
+ 209,
+ 84,
+ 100,
+ 122,
+ 201,
+ 271,
+ 97
+ ],
+ [
+ 201,
+ 56,
+ 159,
+ 271,
+ 97,
+ 288,
+ 82,
+ 209,
+ 122,
+ 93,
+ 200,
+ 224,
+ 265,
+ 191,
+ 100,
+ 244,
+ 47,
+ 146,
+ 127,
+ 84
+ ],
+ [
+ 266,
+ 50,
+ 280,
+ 171,
+ 186,
+ 248,
+ 196,
+ 11,
+ 232,
+ 33,
+ 70,
+ 69,
+ 245,
+ 108,
+ 180,
+ 35,
+ 287,
+ 284,
+ 297,
+ 192
+ ],
+ [
+ 85,
+ 229,
+ 257,
+ 264,
+ 51,
+ 84,
+ 64,
+ 224,
+ 205,
+ 265,
+ 61,
+ 45,
+ 218,
+ 100,
+ 207,
+ 95,
+ 246,
+ 116,
+ 244,
+ 47
+ ],
+ [
+ 95,
+ 199,
+ 154,
+ 136,
+ 53,
+ 61,
+ 19,
+ 45,
+ 207,
+ 116,
+ 127,
+ 135,
+ 58,
+ 240,
+ 20,
+ 246,
+ 36,
+ 120,
+ 65,
+ 264
+ ],
+ [
+ 69,
+ 108,
+ 245,
+ 196,
+ 171,
+ 297,
+ 35,
+ 280,
+ 11,
+ 186,
+ 139,
+ 70,
+ 266,
+ 232,
+ 50,
+ 284,
+ 180,
+ 30,
+ 192,
+ 287
+ ],
+ [
+ 162,
+ 25,
+ 235,
+ 296,
+ 203,
+ 189,
+ 221,
+ 281,
+ 167,
+ 54,
+ 291,
+ 197,
+ 2,
+ 176,
+ 90,
+ 263,
+ 9,
+ 101,
+ 212,
+ 236
+ ],
+ [
+ 70,
+ 11,
+ 33,
+ 186,
+ 245,
+ 297,
+ 232,
+ 108,
+ 180,
+ 284,
+ 192,
+ 69,
+ 196,
+ 280,
+ 139,
+ 50,
+ 266,
+ 171,
+ 287,
+ 30
+ ],
+ [
+ 127,
+ 58,
+ 246,
+ 61,
+ 240,
+ 53,
+ 224,
+ 207,
+ 264,
+ 95,
+ 100,
+ 288,
+ 265,
+ 200,
+ 173,
+ 32,
+ 136,
+ 84,
+ 154,
+ 159
+ ],
+ [
+ 277,
+ 133,
+ 217,
+ 285,
+ 211,
+ 9,
+ 212,
+ 129,
+ 77,
+ 236,
+ 138,
+ 2,
+ 223,
+ 197,
+ 268,
+ 263,
+ 279,
+ 221,
+ 167,
+ 276
+ ],
+ [
+ 286,
+ 298,
+ 111,
+ 148,
+ 63,
+ 278,
+ 71,
+ 239,
+ 112,
+ 42,
+ 237,
+ 267,
+ 22,
+ 254,
+ 145,
+ 123,
+ 137,
+ 27,
+ 60,
+ 124
+ ],
+ [
+ 278,
+ 237,
+ 230,
+ 298,
+ 91,
+ 38,
+ 5,
+ 111,
+ 13,
+ 60,
+ 62,
+ 27,
+ 148,
+ 86,
+ 124,
+ 286,
+ 137,
+ 109,
+ 63,
+ 259
+ ],
+ [
+ 210,
+ 233,
+ 126,
+ 213,
+ 185,
+ 163,
+ 226,
+ 54,
+ 179,
+ 90,
+ 281,
+ 260,
+ 25,
+ 249,
+ 189,
+ 220,
+ 194,
+ 135,
+ 282,
+ 34
+ ],
+ [
+ 221,
+ 263,
+ 235,
+ 54,
+ 281,
+ 176,
+ 236,
+ 217,
+ 162,
+ 197,
+ 9,
+ 212,
+ 34,
+ 189,
+ 260,
+ 167,
+ 285,
+ 2,
+ 101,
+ 268
+ ],
+ [
+ 22,
+ 239,
+ 233,
+ 15,
+ 286,
+ 38,
+ 63,
+ 71,
+ 112,
+ 148,
+ 298,
+ 163,
+ 267,
+ 254,
+ 83,
+ 123,
+ 213,
+ 105,
+ 234,
+ 145
+ ],
+ [
+ 274,
+ 8,
+ 141,
+ 68,
+ 80,
+ 78,
+ 66,
+ 262,
+ 295,
+ 169,
+ 72,
+ 3,
+ 161,
+ 193,
+ 125,
+ 79,
+ 73,
+ 96,
+ 142,
+ 67
+ ],
+ [
+ 226,
+ 220,
+ 275,
+ 19,
+ 24,
+ 65,
+ 120,
+ 106,
+ 199,
+ 94,
+ 183,
+ 210,
+ 153,
+ 81,
+ 59,
+ 250,
+ 92,
+ 258,
+ 32,
+ 142
+ ],
+ [
+ 116,
+ 53,
+ 207,
+ 95,
+ 199,
+ 136,
+ 32,
+ 264,
+ 61,
+ 154,
+ 135,
+ 120,
+ 58,
+ 246,
+ 64,
+ 173,
+ 127,
+ 244,
+ 100,
+ 65
+ ],
+ [
+ 67,
+ 115,
+ 290,
+ 130,
+ 80,
+ 274,
+ 272,
+ 161,
+ 79,
+ 262,
+ 169,
+ 149,
+ 166,
+ 193,
+ 78,
+ 43,
+ 295,
+ 214,
+ 142,
+ 73
+ ],
+ [
+ 146,
+ 191,
+ 209,
+ 159,
+ 224,
+ 82,
+ 51,
+ 201,
+ 200,
+ 244,
+ 265,
+ 84,
+ 29,
+ 97,
+ 100,
+ 56,
+ 64,
+ 93,
+ 271,
+ 122
+ ],
+ [
+ 110,
+ 272,
+ 149,
+ 188,
+ 106,
+ 250,
+ 76,
+ 113,
+ 169,
+ 183,
+ 295,
+ 290,
+ 79,
+ 166,
+ 193,
+ 72,
+ 142,
+ 8,
+ 59,
+ 80
+ ],
+ [
+ 23,
+ 177,
+ 12,
+ 240,
+ 100,
+ 58,
+ 20,
+ 61,
+ 154,
+ 95,
+ 32,
+ 127,
+ 36,
+ 136,
+ 156,
+ 243,
+ 84,
+ 7,
+ 64,
+ 117
+ ],
+ [
+ 232,
+ 180,
+ 11,
+ 186,
+ 70,
+ 266,
+ 192,
+ 248,
+ 35,
+ 284,
+ 30,
+ 33,
+ 280,
+ 171,
+ 196,
+ 245,
+ 287,
+ 69,
+ 108,
+ 297
+ ],
+ [
+ 265,
+ 224,
+ 84,
+ 64,
+ 100,
+ 257,
+ 47,
+ 244,
+ 31,
+ 85,
+ 146,
+ 191,
+ 264,
+ 200,
+ 82,
+ 159,
+ 229,
+ 218,
+ 93,
+ 173
+ ],
+ [
+ 256,
+ 260,
+ 194,
+ 189,
+ 126,
+ 101,
+ 90,
+ 281,
+ 54,
+ 291,
+ 197,
+ 235,
+ 179,
+ 25,
+ 263,
+ 203,
+ 2,
+ 41,
+ 236,
+ 154
+ ],
+ [
+ 95,
+ 136,
+ 207,
+ 32,
+ 199,
+ 45,
+ 246,
+ 127,
+ 116,
+ 36,
+ 264,
+ 154,
+ 58,
+ 61,
+ 173,
+ 100,
+ 240,
+ 224,
+ 209,
+ 19
+ ],
+ [
+ 281,
+ 189,
+ 34,
+ 25,
+ 221,
+ 235,
+ 162,
+ 41,
+ 176,
+ 263,
+ 260,
+ 90,
+ 197,
+ 291,
+ 203,
+ 296,
+ 167,
+ 194,
+ 2,
+ 9
+ ],
+ [
+ 121,
+ 253,
+ 88,
+ 109,
+ 124,
+ 234,
+ 182,
+ 131,
+ 160,
+ 145,
+ 184,
+ 237,
+ 74,
+ 267,
+ 60,
+ 17,
+ 112,
+ 4,
+ 62,
+ 13
+ ],
+ [
+ 29,
+ 271,
+ 288,
+ 201,
+ 159,
+ 97,
+ 82,
+ 93,
+ 122,
+ 209,
+ 200,
+ 224,
+ 265,
+ 127,
+ 100,
+ 191,
+ 244,
+ 36,
+ 47,
+ 146
+ ],
+ [
+ 112,
+ 148,
+ 63,
+ 254,
+ 160,
+ 111,
+ 145,
+ 286,
+ 267,
+ 137,
+ 71,
+ 298,
+ 206,
+ 89,
+ 175,
+ 27,
+ 60,
+ 13,
+ 102,
+ 234
+ ],
+ [
+ 61,
+ 127,
+ 36,
+ 240,
+ 154,
+ 246,
+ 95,
+ 53,
+ 32,
+ 207,
+ 264,
+ 224,
+ 100,
+ 45,
+ 136,
+ 200,
+ 84,
+ 173,
+ 199,
+ 23
+ ],
+ [
+ 142,
+ 250,
+ 104,
+ 94,
+ 106,
+ 153,
+ 292,
+ 152,
+ 92,
+ 96,
+ 87,
+ 3,
+ 188,
+ 258,
+ 295,
+ 178,
+ 261,
+ 76,
+ 143,
+ 165
+ ],
+ [
+ 13,
+ 109,
+ 5,
+ 145,
+ 137,
+ 27,
+ 111,
+ 237,
+ 160,
+ 206,
+ 128,
+ 278,
+ 148,
+ 63,
+ 298,
+ 112,
+ 299,
+ 254,
+ 89,
+ 267
+ ],
+ [
+ 58,
+ 240,
+ 154,
+ 127,
+ 36,
+ 32,
+ 95,
+ 207,
+ 246,
+ 53,
+ 264,
+ 64,
+ 45,
+ 224,
+ 116,
+ 100,
+ 20,
+ 199,
+ 136,
+ 173
+ ],
+ [
+ 131,
+ 259,
+ 160,
+ 13,
+ 86,
+ 278,
+ 4,
+ 237,
+ 39,
+ 102,
+ 74,
+ 124,
+ 182,
+ 91,
+ 60,
+ 109,
+ 230,
+ 17,
+ 137,
+ 112
+ ],
+ [
+ 148,
+ 298,
+ 111,
+ 112,
+ 145,
+ 286,
+ 254,
+ 38,
+ 27,
+ 57,
+ 278,
+ 60,
+ 5,
+ 267,
+ 239,
+ 13,
+ 71,
+ 42,
+ 22,
+ 206
+ ],
+ [
+ 51,
+ 224,
+ 265,
+ 61,
+ 244,
+ 100,
+ 127,
+ 84,
+ 45,
+ 207,
+ 47,
+ 154,
+ 264,
+ 200,
+ 58,
+ 116,
+ 209,
+ 53,
+ 32,
+ 85
+ ],
+ [
+ 120,
+ 19,
+ 199,
+ 116,
+ 136,
+ 95,
+ 44,
+ 32,
+ 45,
+ 53,
+ 24,
+ 20,
+ 275,
+ 220,
+ 226,
+ 154,
+ 135,
+ 61,
+ 240,
+ 264
+ ],
+ [
+ 68,
+ 8,
+ 3,
+ 96,
+ 141,
+ 72,
+ 43,
+ 193,
+ 125,
+ 295,
+ 274,
+ 78,
+ 169,
+ 188,
+ 161,
+ 204,
+ 183,
+ 80,
+ 73,
+ 142
+ ],
+ [
+ 46,
+ 290,
+ 80,
+ 115,
+ 130,
+ 274,
+ 79,
+ 272,
+ 262,
+ 161,
+ 169,
+ 43,
+ 149,
+ 78,
+ 193,
+ 73,
+ 166,
+ 295,
+ 8,
+ 141
+ ],
+ [
+ 8,
+ 66,
+ 72,
+ 141,
+ 43,
+ 78,
+ 274,
+ 125,
+ 295,
+ 3,
+ 96,
+ 193,
+ 169,
+ 188,
+ 204,
+ 73,
+ 80,
+ 161,
+ 79,
+ 76
+ ],
+ [
+ 196,
+ 108,
+ 33,
+ 245,
+ 171,
+ 280,
+ 297,
+ 139,
+ 266,
+ 186,
+ 35,
+ 11,
+ 70,
+ 232,
+ 50,
+ 30,
+ 284,
+ 180,
+ 287,
+ 192
+ ],
+ [
+ 11,
+ 35,
+ 232,
+ 186,
+ 245,
+ 33,
+ 50,
+ 280,
+ 266,
+ 180,
+ 108,
+ 196,
+ 192,
+ 297,
+ 69,
+ 284,
+ 171,
+ 139,
+ 30,
+ 248
+ ],
+ [
+ 286,
+ 89,
+ 254,
+ 112,
+ 239,
+ 298,
+ 38,
+ 111,
+ 137,
+ 267,
+ 148,
+ 206,
+ 160,
+ 278,
+ 63,
+ 57,
+ 234,
+ 145,
+ 237,
+ 42
+ ],
+ [
+ 141,
+ 68,
+ 8,
+ 188,
+ 66,
+ 295,
+ 183,
+ 73,
+ 169,
+ 204,
+ 43,
+ 274,
+ 125,
+ 3,
+ 96,
+ 76,
+ 80,
+ 142,
+ 78,
+ 79
+ ],
+ [
+ 141,
+ 72,
+ 274,
+ 43,
+ 68,
+ 78,
+ 80,
+ 66,
+ 8,
+ 272,
+ 149,
+ 166,
+ 295,
+ 67,
+ 142,
+ 187,
+ 169,
+ 3,
+ 188,
+ 193
+ ],
+ [
+ 17,
+ 234,
+ 102,
+ 124,
+ 91,
+ 131,
+ 253,
+ 184,
+ 259,
+ 278,
+ 267,
+ 13,
+ 112,
+ 4,
+ 86,
+ 160,
+ 62,
+ 89,
+ 145,
+ 148
+ ],
+ [
+ 198,
+ 6,
+ 214,
+ 14,
+ 190,
+ 76,
+ 195,
+ 231,
+ 269,
+ 295,
+ 115,
+ 125,
+ 143,
+ 169,
+ 172,
+ 79,
+ 204,
+ 293,
+ 98,
+ 1
+ ],
+ [
+ 295,
+ 188,
+ 110,
+ 169,
+ 125,
+ 204,
+ 6,
+ 8,
+ 3,
+ 79,
+ 142,
+ 59,
+ 96,
+ 183,
+ 250,
+ 72,
+ 193,
+ 141,
+ 272,
+ 274
+ ],
+ [
+ 277,
+ 217,
+ 236,
+ 285,
+ 211,
+ 37,
+ 212,
+ 268,
+ 9,
+ 279,
+ 129,
+ 197,
+ 223,
+ 101,
+ 263,
+ 2,
+ 138,
+ 133,
+ 291,
+ 276
+ ],
+ [
+ 274,
+ 68,
+ 43,
+ 8,
+ 141,
+ 295,
+ 73,
+ 161,
+ 72,
+ 66,
+ 193,
+ 125,
+ 80,
+ 169,
+ 262,
+ 79,
+ 3,
+ 115,
+ 272,
+ 6
+ ],
+ [
+ 169,
+ 290,
+ 80,
+ 274,
+ 295,
+ 161,
+ 8,
+ 76,
+ 262,
+ 125,
+ 67,
+ 188,
+ 115,
+ 193,
+ 43,
+ 142,
+ 68,
+ 3,
+ 272,
+ 72
+ ],
+ [
+ 169,
+ 274,
+ 79,
+ 290,
+ 43,
+ 161,
+ 67,
+ 142,
+ 295,
+ 262,
+ 272,
+ 8,
+ 193,
+ 141,
+ 46,
+ 3,
+ 115,
+ 149,
+ 72,
+ 68
+ ],
+ [
+ 165,
+ 292,
+ 92,
+ 94,
+ 181,
+ 178,
+ 227,
+ 174,
+ 87,
+ 152,
+ 294,
+ 255,
+ 153,
+ 261,
+ 157,
+ 140,
+ 282,
+ 242,
+ 222,
+ 283
+ ],
+ [
+ 159,
+ 56,
+ 29,
+ 93,
+ 201,
+ 97,
+ 271,
+ 209,
+ 265,
+ 288,
+ 200,
+ 224,
+ 122,
+ 191,
+ 244,
+ 100,
+ 47,
+ 146,
+ 51,
+ 84
+ ],
+ [
+ 254,
+ 185,
+ 239,
+ 213,
+ 63,
+ 286,
+ 206,
+ 89,
+ 267,
+ 42,
+ 111,
+ 71,
+ 148,
+ 112,
+ 163,
+ 299,
+ 234,
+ 298,
+ 233,
+ 57
+ ],
+ [
+ 100,
+ 224,
+ 51,
+ 257,
+ 218,
+ 265,
+ 200,
+ 264,
+ 244,
+ 246,
+ 146,
+ 159,
+ 93,
+ 97,
+ 47,
+ 82,
+ 127,
+ 31,
+ 191,
+ 23
+ ],
+ [
+ 229,
+ 31,
+ 257,
+ 51,
+ 264,
+ 84,
+ 64,
+ 205,
+ 45,
+ 61,
+ 116,
+ 224,
+ 218,
+ 100,
+ 265,
+ 207,
+ 58,
+ 95,
+ 173,
+ 244
+ ],
+ [
+ 4,
+ 124,
+ 91,
+ 278,
+ 71,
+ 237,
+ 88,
+ 62,
+ 298,
+ 38,
+ 112,
+ 74,
+ 286,
+ 118,
+ 160,
+ 39,
+ 102,
+ 150,
+ 123,
+ 267
+ ],
+ [
+ 178,
+ 94,
+ 292,
+ 258,
+ 153,
+ 255,
+ 157,
+ 283,
+ 81,
+ 140,
+ 104,
+ 181,
+ 174,
+ 227,
+ 165,
+ 119,
+ 261,
+ 92,
+ 152,
+ 294
+ ],
+ [
+ 253,
+ 112,
+ 124,
+ 145,
+ 163,
+ 234,
+ 160,
+ 86,
+ 15,
+ 150,
+ 267,
+ 71,
+ 42,
+ 286,
+ 57,
+ 63,
+ 22,
+ 168,
+ 148,
+ 298
+ ],
+ [
+ 267,
+ 206,
+ 299,
+ 254,
+ 234,
+ 160,
+ 137,
+ 71,
+ 112,
+ 17,
+ 111,
+ 286,
+ 145,
+ 148,
+ 60,
+ 102,
+ 13,
+ 63,
+ 184,
+ 57
+ ],
+ [
+ 189,
+ 25,
+ 194,
+ 34,
+ 203,
+ 291,
+ 281,
+ 235,
+ 197,
+ 54,
+ 162,
+ 101,
+ 296,
+ 221,
+ 2,
+ 263,
+ 260,
+ 212,
+ 9,
+ 179
+ ],
+ [
+ 278,
+ 86,
+ 123,
+ 4,
+ 74,
+ 39,
+ 124,
+ 38,
+ 102,
+ 298,
+ 230,
+ 148,
+ 237,
+ 62,
+ 71,
+ 111,
+ 286,
+ 267,
+ 112,
+ 13
+ ],
+ [
+ 81,
+ 153,
+ 178,
+ 292,
+ 140,
+ 174,
+ 87,
+ 157,
+ 165,
+ 283,
+ 294,
+ 227,
+ 94,
+ 242,
+ 152,
+ 252,
+ 261,
+ 255,
+ 10,
+ 104
+ ],
+ [
+ 271,
+ 122,
+ 97,
+ 159,
+ 82,
+ 56,
+ 288,
+ 200,
+ 201,
+ 29,
+ 209,
+ 224,
+ 265,
+ 191,
+ 100,
+ 84,
+ 127,
+ 146,
+ 244,
+ 47
+ ],
+ [
+ 87,
+ 153,
+ 81,
+ 258,
+ 178,
+ 104,
+ 292,
+ 181,
+ 152,
+ 261,
+ 165,
+ 283,
+ 157,
+ 92,
+ 227,
+ 255,
+ 282,
+ 174,
+ 294,
+ 140
+ ],
+ [
+ 136,
+ 53,
+ 32,
+ 199,
+ 207,
+ 154,
+ 45,
+ 116,
+ 246,
+ 127,
+ 61,
+ 264,
+ 19,
+ 36,
+ 58,
+ 240,
+ 120,
+ 65,
+ 20,
+ 135
+ ],
+ [
+ 3,
+ 8,
+ 125,
+ 66,
+ 295,
+ 193,
+ 68,
+ 204,
+ 59,
+ 141,
+ 188,
+ 142,
+ 169,
+ 76,
+ 72,
+ 6,
+ 183,
+ 161,
+ 43,
+ 274
+ ],
+ [
+ 201,
+ 122,
+ 56,
+ 271,
+ 93,
+ 159,
+ 29,
+ 288,
+ 82,
+ 200,
+ 209,
+ 224,
+ 191,
+ 100,
+ 146,
+ 265,
+ 84,
+ 47,
+ 127,
+ 244
+ ],
+ [
+ 172,
+ 269,
+ 293,
+ 195,
+ 143,
+ 165,
+ 190,
+ 242,
+ 216,
+ 10,
+ 273,
+ 103,
+ 261,
+ 214,
+ 104,
+ 238,
+ 140,
+ 174,
+ 251,
+ 151
+ ],
+ [
+ 276,
+ 243,
+ 211,
+ 138,
+ 129,
+ 77,
+ 268,
+ 236,
+ 9,
+ 223,
+ 37,
+ 277,
+ 279,
+ 285,
+ 2,
+ 212,
+ 217,
+ 197,
+ 133,
+ 215
+ ],
+ [
+ 84,
+ 244,
+ 224,
+ 265,
+ 246,
+ 200,
+ 218,
+ 264,
+ 51,
+ 159,
+ 36,
+ 23,
+ 82,
+ 127,
+ 93,
+ 288,
+ 173,
+ 209,
+ 97,
+ 271
+ ],
+ [
+ 291,
+ 197,
+ 235,
+ 2,
+ 212,
+ 263,
+ 277,
+ 217,
+ 221,
+ 203,
+ 77,
+ 236,
+ 9,
+ 279,
+ 189,
+ 162,
+ 285,
+ 211,
+ 296,
+ 37
+ ],
+ [
+ 206,
+ 17,
+ 137,
+ 160,
+ 13,
+ 112,
+ 278,
+ 234,
+ 89,
+ 148,
+ 267,
+ 74,
+ 145,
+ 254,
+ 60,
+ 184,
+ 71,
+ 111,
+ 253,
+ 57
+ ],
+ [
+ 242,
+ 252,
+ 151,
+ 251,
+ 165,
+ 152,
+ 261,
+ 222,
+ 216,
+ 238,
+ 181,
+ 81,
+ 10,
+ 255,
+ 174,
+ 294,
+ 273,
+ 227,
+ 92,
+ 132
+ ],
+ [
+ 292,
+ 94,
+ 87,
+ 152,
+ 261,
+ 178,
+ 258,
+ 81,
+ 255,
+ 153,
+ 165,
+ 181,
+ 250,
+ 92,
+ 59,
+ 174,
+ 294,
+ 227,
+ 242,
+ 282
+ ],
+ [
+ 118,
+ 15,
+ 22,
+ 123,
+ 42,
+ 124,
+ 88,
+ 38,
+ 298,
+ 63,
+ 278,
+ 145,
+ 168,
+ 233,
+ 91,
+ 239,
+ 4,
+ 86,
+ 286,
+ 148
+ ],
+ [
+ 250,
+ 113,
+ 183,
+ 59,
+ 249,
+ 104,
+ 142,
+ 152,
+ 110,
+ 94,
+ 261,
+ 149,
+ 188,
+ 272,
+ 153,
+ 81,
+ 87,
+ 92,
+ 48,
+ 258
+ ],
+ [
+ 241,
+ 164,
+ 270,
+ 155,
+ 208,
+ 187,
+ 119,
+ 0,
+ 273,
+ 21,
+ 227,
+ 132,
+ 238,
+ 140,
+ 251,
+ 26,
+ 292,
+ 157,
+ 165,
+ 283
+ ],
+ [
+ 245,
+ 69,
+ 33,
+ 196,
+ 297,
+ 139,
+ 171,
+ 280,
+ 35,
+ 186,
+ 70,
+ 11,
+ 266,
+ 232,
+ 50,
+ 284,
+ 180,
+ 30,
+ 192,
+ 287
+ ],
+ [
+ 60,
+ 145,
+ 13,
+ 5,
+ 27,
+ 237,
+ 160,
+ 137,
+ 128,
+ 111,
+ 253,
+ 278,
+ 206,
+ 112,
+ 182,
+ 131,
+ 148,
+ 63,
+ 298,
+ 299
+ ],
+ [
+ 76,
+ 188,
+ 48,
+ 295,
+ 183,
+ 106,
+ 169,
+ 250,
+ 113,
+ 272,
+ 59,
+ 142,
+ 149,
+ 79,
+ 8,
+ 204,
+ 125,
+ 72,
+ 3,
+ 6
+ ],
+ [
+ 148,
+ 298,
+ 63,
+ 27,
+ 60,
+ 254,
+ 137,
+ 286,
+ 112,
+ 38,
+ 206,
+ 5,
+ 13,
+ 145,
+ 278,
+ 128,
+ 71,
+ 267,
+ 89,
+ 160
+ ],
+ [
+ 148,
+ 145,
+ 63,
+ 57,
+ 286,
+ 267,
+ 298,
+ 160,
+ 254,
+ 111,
+ 89,
+ 234,
+ 71,
+ 206,
+ 124,
+ 278,
+ 137,
+ 38,
+ 13,
+ 60
+ ],
+ [
+ 106,
+ 249,
+ 250,
+ 152,
+ 261,
+ 104,
+ 242,
+ 110,
+ 103,
+ 151,
+ 59,
+ 149,
+ 94,
+ 165,
+ 81,
+ 142,
+ 181,
+ 166,
+ 48,
+ 252
+ ],
+ [
+ 12,
+ 156,
+ 144,
+ 247,
+ 202,
+ 49,
+ 177,
+ 99,
+ 23,
+ 243,
+ 276,
+ 241,
+ 164,
+ 138,
+ 215,
+ 279,
+ 211,
+ 236,
+ 223,
+ 117
+ ],
+ [
+ 46,
+ 67,
+ 79,
+ 80,
+ 290,
+ 274,
+ 169,
+ 130,
+ 161,
+ 272,
+ 262,
+ 78,
+ 14,
+ 295,
+ 214,
+ 198,
+ 43,
+ 149,
+ 193,
+ 142
+ ],
+ [
+ 45,
+ 199,
+ 53,
+ 95,
+ 120,
+ 136,
+ 32,
+ 207,
+ 65,
+ 19,
+ 61,
+ 154,
+ 135,
+ 264,
+ 64,
+ 58,
+ 127,
+ 209,
+ 246,
+ 100
+ ],
+ [
+ 23,
+ 100,
+ 177,
+ 84,
+ 7,
+ 264,
+ 18,
+ 173,
+ 224,
+ 265,
+ 218,
+ 257,
+ 51,
+ 58,
+ 127,
+ 246,
+ 36,
+ 61,
+ 93,
+ 244
+ ],
+ [
+ 105,
+ 15,
+ 22,
+ 42,
+ 86,
+ 124,
+ 38,
+ 88,
+ 123,
+ 4,
+ 298,
+ 175,
+ 91,
+ 286,
+ 278,
+ 239,
+ 63,
+ 233,
+ 71,
+ 112
+ ],
+ [
+ 157,
+ 208,
+ 155,
+ 174,
+ 227,
+ 283,
+ 0,
+ 140,
+ 87,
+ 255,
+ 132,
+ 165,
+ 26,
+ 178,
+ 81,
+ 282,
+ 292,
+ 238,
+ 181,
+ 294
+ ],
+ [
+ 65,
+ 19,
+ 199,
+ 116,
+ 95,
+ 136,
+ 32,
+ 45,
+ 53,
+ 44,
+ 154,
+ 24,
+ 20,
+ 275,
+ 61,
+ 240,
+ 226,
+ 220,
+ 135,
+ 58
+ ],
+ [
+ 55,
+ 253,
+ 88,
+ 182,
+ 124,
+ 62,
+ 131,
+ 4,
+ 74,
+ 109,
+ 86,
+ 160,
+ 237,
+ 168,
+ 259,
+ 145,
+ 234,
+ 105,
+ 112,
+ 91
+ ],
+ [
+ 271,
+ 97,
+ 93,
+ 201,
+ 56,
+ 29,
+ 288,
+ 159,
+ 200,
+ 82,
+ 209,
+ 191,
+ 224,
+ 146,
+ 100,
+ 84,
+ 265,
+ 127,
+ 47,
+ 244
+ ],
+ [
+ 298,
+ 278,
+ 91,
+ 22,
+ 15,
+ 148,
+ 63,
+ 38,
+ 239,
+ 42,
+ 105,
+ 145,
+ 111,
+ 286,
+ 124,
+ 112,
+ 71,
+ 102,
+ 60,
+ 267
+ ],
+ [
+ 112,
+ 278,
+ 145,
+ 298,
+ 86,
+ 234,
+ 253,
+ 148,
+ 286,
+ 22,
+ 63,
+ 74,
+ 4,
+ 88,
+ 17,
+ 267,
+ 160,
+ 13,
+ 38,
+ 91
+ ],
+ [
+ 8,
+ 295,
+ 3,
+ 96,
+ 204,
+ 6,
+ 76,
+ 169,
+ 68,
+ 193,
+ 188,
+ 66,
+ 79,
+ 43,
+ 141,
+ 161,
+ 72,
+ 274,
+ 142,
+ 1
+ ],
+ [
+ 163,
+ 185,
+ 213,
+ 233,
+ 179,
+ 83,
+ 254,
+ 135,
+ 239,
+ 71,
+ 89,
+ 88,
+ 160,
+ 113,
+ 40,
+ 234,
+ 281,
+ 48,
+ 267,
+ 206
+ ],
+ [
+ 36,
+ 58,
+ 61,
+ 240,
+ 53,
+ 224,
+ 246,
+ 200,
+ 95,
+ 32,
+ 288,
+ 100,
+ 207,
+ 265,
+ 154,
+ 159,
+ 264,
+ 56,
+ 209,
+ 93
+ ],
+ [
+ 27,
+ 5,
+ 60,
+ 111,
+ 13,
+ 148,
+ 63,
+ 299,
+ 137,
+ 206,
+ 109,
+ 160,
+ 254,
+ 145,
+ 278,
+ 298,
+ 237,
+ 112,
+ 230,
+ 267
+ ],
+ [
+ 138,
+ 37,
+ 211,
+ 285,
+ 236,
+ 277,
+ 77,
+ 217,
+ 133,
+ 268,
+ 9,
+ 276,
+ 223,
+ 2,
+ 212,
+ 279,
+ 215,
+ 221,
+ 197,
+ 263
+ ],
+ [
+ 67,
+ 46,
+ 115,
+ 262,
+ 290,
+ 274,
+ 80,
+ 161,
+ 78,
+ 272,
+ 79,
+ 43,
+ 169,
+ 73,
+ 193,
+ 149,
+ 166,
+ 68,
+ 14,
+ 8
+ ],
+ [
+ 259,
+ 13,
+ 17,
+ 62,
+ 74,
+ 253,
+ 60,
+ 160,
+ 102,
+ 5,
+ 109,
+ 278,
+ 145,
+ 124,
+ 237,
+ 184,
+ 234,
+ 137,
+ 112,
+ 128
+ ],
+ [
+ 238,
+ 10,
+ 174,
+ 140,
+ 251,
+ 157,
+ 208,
+ 294,
+ 26,
+ 222,
+ 227,
+ 119,
+ 0,
+ 255,
+ 165,
+ 178,
+ 283,
+ 81,
+ 181,
+ 292
+ ],
+ [
+ 37,
+ 9,
+ 277,
+ 138,
+ 217,
+ 211,
+ 212,
+ 236,
+ 285,
+ 129,
+ 167,
+ 223,
+ 77,
+ 268,
+ 263,
+ 176,
+ 221,
+ 2,
+ 197,
+ 276
+ ],
+ [
+ 289,
+ 78,
+ 141,
+ 73,
+ 1,
+ 274,
+ 68,
+ 43,
+ 72,
+ 66,
+ 8,
+ 80,
+ 262,
+ 125,
+ 4,
+ 193,
+ 79,
+ 295,
+ 3,
+ 62
+ ],
+ [
+ 32,
+ 199,
+ 45,
+ 154,
+ 136,
+ 95,
+ 53,
+ 116,
+ 207,
+ 226,
+ 220,
+ 126,
+ 19,
+ 275,
+ 185,
+ 210,
+ 113,
+ 120,
+ 64,
+ 106
+ ],
+ [
+ 95,
+ 53,
+ 199,
+ 32,
+ 45,
+ 207,
+ 116,
+ 19,
+ 154,
+ 246,
+ 264,
+ 65,
+ 120,
+ 127,
+ 36,
+ 61,
+ 135,
+ 20,
+ 58,
+ 240
+ ],
+ [
+ 60,
+ 206,
+ 111,
+ 160,
+ 89,
+ 237,
+ 254,
+ 13,
+ 71,
+ 148,
+ 145,
+ 267,
+ 112,
+ 102,
+ 299,
+ 27,
+ 278,
+ 109,
+ 57,
+ 286
+ ],
+ [
+ 129,
+ 211,
+ 37,
+ 285,
+ 236,
+ 223,
+ 276,
+ 133,
+ 9,
+ 77,
+ 268,
+ 277,
+ 217,
+ 2,
+ 212,
+ 215,
+ 279,
+ 167,
+ 263,
+ 221
+ ],
+ [
+ 108,
+ 297,
+ 245,
+ 69,
+ 33,
+ 196,
+ 171,
+ 35,
+ 186,
+ 280,
+ 70,
+ 11,
+ 266,
+ 232,
+ 287,
+ 50,
+ 180,
+ 192,
+ 284,
+ 30
+ ],
+ [
+ 157,
+ 174,
+ 87,
+ 283,
+ 132,
+ 119,
+ 92,
+ 81,
+ 238,
+ 165,
+ 178,
+ 255,
+ 292,
+ 294,
+ 10,
+ 227,
+ 0,
+ 153,
+ 181,
+ 208
+ ],
+ [
+ 72,
+ 8,
+ 68,
+ 43,
+ 66,
+ 73,
+ 295,
+ 3,
+ 274,
+ 80,
+ 96,
+ 188,
+ 142,
+ 169,
+ 125,
+ 78,
+ 183,
+ 59,
+ 193,
+ 272
+ ],
+ [
+ 59,
+ 250,
+ 104,
+ 295,
+ 169,
+ 80,
+ 3,
+ 149,
+ 188,
+ 106,
+ 76,
+ 292,
+ 141,
+ 92,
+ 183,
+ 96,
+ 94,
+ 8,
+ 153,
+ 261
+ ],
+ [
+ 59,
+ 142,
+ 195,
+ 104,
+ 172,
+ 76,
+ 98,
+ 269,
+ 250,
+ 293,
+ 295,
+ 188,
+ 204,
+ 214,
+ 94,
+ 258,
+ 292,
+ 273,
+ 153,
+ 87
+ ],
+ [
+ 114,
+ 215,
+ 138,
+ 223,
+ 279,
+ 202,
+ 276,
+ 211,
+ 129,
+ 236,
+ 99,
+ 133,
+ 77,
+ 37,
+ 285,
+ 9,
+ 277,
+ 2,
+ 243,
+ 167
+ ],
+ [
+ 112,
+ 60,
+ 63,
+ 148,
+ 13,
+ 298,
+ 111,
+ 160,
+ 253,
+ 109,
+ 278,
+ 206,
+ 267,
+ 254,
+ 137,
+ 5,
+ 57,
+ 286,
+ 234,
+ 27
+ ],
+ [
+ 191,
+ 47,
+ 82,
+ 201,
+ 84,
+ 159,
+ 97,
+ 51,
+ 224,
+ 122,
+ 209,
+ 200,
+ 29,
+ 93,
+ 56,
+ 271,
+ 265,
+ 100,
+ 244,
+ 28
+ ],
+ [
+ 68,
+ 66,
+ 72,
+ 8,
+ 141,
+ 188,
+ 3,
+ 295,
+ 96,
+ 193,
+ 125,
+ 204,
+ 43,
+ 142,
+ 169,
+ 274,
+ 183,
+ 6,
+ 78,
+ 76
+ ],
+ [
+ 63,
+ 112,
+ 111,
+ 298,
+ 145,
+ 286,
+ 57,
+ 254,
+ 38,
+ 27,
+ 278,
+ 267,
+ 60,
+ 5,
+ 137,
+ 206,
+ 13,
+ 71,
+ 160,
+ 89
+ ],
+ [
+ 272,
+ 166,
+ 250,
+ 142,
+ 193,
+ 59,
+ 80,
+ 106,
+ 169,
+ 295,
+ 48,
+ 113,
+ 290,
+ 3,
+ 274,
+ 110,
+ 141,
+ 161,
+ 152,
+ 104
+ ],
+ [
+ 112,
+ 57,
+ 88,
+ 267,
+ 234,
+ 124,
+ 89,
+ 86,
+ 71,
+ 160,
+ 286,
+ 175,
+ 42,
+ 184,
+ 148,
+ 74,
+ 17,
+ 254,
+ 102,
+ 253
+ ],
+ [
+ 216,
+ 242,
+ 261,
+ 165,
+ 152,
+ 251,
+ 181,
+ 222,
+ 103,
+ 255,
+ 174,
+ 252,
+ 81,
+ 294,
+ 26,
+ 227,
+ 238,
+ 10,
+ 208,
+ 132
+ ],
+ [
+ 261,
+ 165,
+ 181,
+ 242,
+ 81,
+ 151,
+ 255,
+ 104,
+ 94,
+ 87,
+ 292,
+ 153,
+ 216,
+ 294,
+ 227,
+ 178,
+ 92,
+ 174,
+ 222,
+ 252
+ ],
+ [
+ 94,
+ 87,
+ 283,
+ 92,
+ 81,
+ 178,
+ 292,
+ 165,
+ 140,
+ 227,
+ 152,
+ 157,
+ 261,
+ 104,
+ 174,
+ 294,
+ 258,
+ 181,
+ 255,
+ 242
+ ],
+ [
+ 32,
+ 61,
+ 95,
+ 240,
+ 58,
+ 199,
+ 53,
+ 136,
+ 207,
+ 127,
+ 45,
+ 20,
+ 256,
+ 116,
+ 135,
+ 36,
+ 246,
+ 64,
+ 19,
+ 264
+ ],
+ [
+ 119,
+ 0,
+ 283,
+ 157,
+ 208,
+ 227,
+ 174,
+ 140,
+ 87,
+ 132,
+ 178,
+ 292,
+ 255,
+ 10,
+ 153,
+ 238,
+ 165,
+ 294,
+ 26,
+ 258
+ ],
+ [
+ 12,
+ 114,
+ 177,
+ 49,
+ 23,
+ 247,
+ 117,
+ 144,
+ 16,
+ 219,
+ 84,
+ 7,
+ 202,
+ 229,
+ 164,
+ 241,
+ 100,
+ 257,
+ 218,
+ 31
+ ],
+ [
+ 119,
+ 174,
+ 140,
+ 283,
+ 227,
+ 87,
+ 178,
+ 0,
+ 81,
+ 208,
+ 255,
+ 282,
+ 132,
+ 155,
+ 92,
+ 294,
+ 258,
+ 165,
+ 238,
+ 292
+ ],
+ [
+ 210,
+ 238,
+ 132,
+ 103,
+ 140,
+ 251,
+ 119,
+ 151,
+ 157,
+ 252,
+ 174,
+ 227,
+ 165,
+ 81,
+ 26,
+ 242,
+ 208,
+ 21,
+ 255,
+ 261
+ ],
+ [
+ 82,
+ 56,
+ 201,
+ 29,
+ 97,
+ 209,
+ 93,
+ 288,
+ 200,
+ 271,
+ 224,
+ 122,
+ 265,
+ 100,
+ 191,
+ 47,
+ 244,
+ 127,
+ 146,
+ 84
+ ],
+ [
+ 89,
+ 112,
+ 206,
+ 137,
+ 13,
+ 60,
+ 145,
+ 267,
+ 57,
+ 299,
+ 237,
+ 234,
+ 254,
+ 111,
+ 17,
+ 102,
+ 148,
+ 71,
+ 109,
+ 278
+ ],
+ [
+ 193,
+ 262,
+ 169,
+ 80,
+ 290,
+ 274,
+ 79,
+ 3,
+ 8,
+ 295,
+ 43,
+ 125,
+ 67,
+ 46,
+ 68,
+ 78,
+ 272,
+ 96,
+ 66,
+ 142
+ ],
+ [
+ 221,
+ 235,
+ 296,
+ 34,
+ 167,
+ 203,
+ 197,
+ 9,
+ 176,
+ 291,
+ 2,
+ 212,
+ 263,
+ 25,
+ 189,
+ 281,
+ 101,
+ 236,
+ 217,
+ 54
+ ],
+ [
+ 233,
+ 185,
+ 239,
+ 42,
+ 213,
+ 88,
+ 83,
+ 254,
+ 168,
+ 234,
+ 71,
+ 112,
+ 22,
+ 267,
+ 63,
+ 89,
+ 286,
+ 145,
+ 298,
+ 206
+ ],
+ [
+ 241,
+ 107,
+ 270,
+ 176,
+ 21,
+ 155,
+ 138,
+ 276,
+ 215,
+ 37,
+ 285,
+ 223,
+ 133,
+ 208,
+ 187,
+ 129,
+ 236,
+ 273,
+ 0,
+ 279
+ ],
+ [
+ 261,
+ 227,
+ 81,
+ 174,
+ 242,
+ 152,
+ 216,
+ 181,
+ 255,
+ 151,
+ 273,
+ 222,
+ 292,
+ 251,
+ 294,
+ 140,
+ 153,
+ 208,
+ 87,
+ 92
+ ],
+ [
+ 149,
+ 272,
+ 250,
+ 113,
+ 249,
+ 106,
+ 142,
+ 252,
+ 80,
+ 152,
+ 242,
+ 46,
+ 59,
+ 165,
+ 92,
+ 73,
+ 261,
+ 103,
+ 151,
+ 81
+ ],
+ [
+ 176,
+ 9,
+ 162,
+ 296,
+ 221,
+ 2,
+ 203,
+ 34,
+ 133,
+ 236,
+ 212,
+ 197,
+ 263,
+ 235,
+ 37,
+ 211,
+ 217,
+ 277,
+ 291,
+ 138
+ ],
+ [
+ 163,
+ 88,
+ 105,
+ 233,
+ 15,
+ 253,
+ 118,
+ 42,
+ 145,
+ 182,
+ 22,
+ 239,
+ 63,
+ 112,
+ 55,
+ 123,
+ 109,
+ 298,
+ 234,
+ 124
+ ],
+ [
+ 79,
+ 295,
+ 80,
+ 76,
+ 8,
+ 193,
+ 274,
+ 188,
+ 161,
+ 290,
+ 125,
+ 142,
+ 3,
+ 183,
+ 272,
+ 43,
+ 72,
+ 68,
+ 6,
+ 110
+ ],
+ [
+ 225,
+ 248,
+ 30,
+ 280,
+ 266,
+ 186,
+ 223,
+ 232,
+ 50,
+ 180,
+ 70,
+ 133,
+ 268,
+ 284,
+ 196,
+ 287,
+ 211,
+ 11,
+ 245,
+ 171
+ ],
+ [
+ 196,
+ 69,
+ 33,
+ 108,
+ 280,
+ 245,
+ 266,
+ 186,
+ 139,
+ 11,
+ 297,
+ 70,
+ 35,
+ 30,
+ 232,
+ 50,
+ 287,
+ 180,
+ 192,
+ 284
+ ],
+ [
+ 98,
+ 293,
+ 269,
+ 195,
+ 242,
+ 165,
+ 273,
+ 103,
+ 143,
+ 214,
+ 216,
+ 261,
+ 10,
+ 190,
+ 252,
+ 251,
+ 238,
+ 140,
+ 174,
+ 151
+ ],
+ [
+ 207,
+ 264,
+ 246,
+ 265,
+ 244,
+ 36,
+ 100,
+ 53,
+ 224,
+ 61,
+ 58,
+ 45,
+ 127,
+ 51,
+ 95,
+ 64,
+ 84,
+ 82,
+ 154,
+ 136
+ ],
+ [
+ 157,
+ 208,
+ 165,
+ 227,
+ 255,
+ 294,
+ 222,
+ 10,
+ 119,
+ 26,
+ 140,
+ 132,
+ 178,
+ 181,
+ 238,
+ 81,
+ 283,
+ 292,
+ 87,
+ 92
+ ],
+ [
+ 57,
+ 148,
+ 112,
+ 63,
+ 145,
+ 15,
+ 124,
+ 38,
+ 111,
+ 298,
+ 286,
+ 267,
+ 278,
+ 71,
+ 88,
+ 160,
+ 60,
+ 22,
+ 86,
+ 91
+ ],
+ [
+ 167,
+ 221,
+ 9,
+ 162,
+ 296,
+ 2,
+ 236,
+ 263,
+ 34,
+ 197,
+ 212,
+ 235,
+ 133,
+ 203,
+ 217,
+ 37,
+ 41,
+ 54,
+ 281,
+ 211
+ ],
+ [
+ 117,
+ 49,
+ 156,
+ 12,
+ 114,
+ 23,
+ 7,
+ 247,
+ 240,
+ 20,
+ 58,
+ 61,
+ 100,
+ 84,
+ 127,
+ 36,
+ 154,
+ 28,
+ 144,
+ 218
+ ],
+ [
+ 292,
+ 87,
+ 294,
+ 81,
+ 174,
+ 283,
+ 157,
+ 94,
+ 255,
+ 92,
+ 0,
+ 258,
+ 153,
+ 181,
+ 140,
+ 104,
+ 282,
+ 227,
+ 165,
+ 10
+ ],
+ [
+ 25,
+ 189,
+ 90,
+ 281,
+ 54,
+ 194,
+ 34,
+ 203,
+ 162,
+ 235,
+ 296,
+ 291,
+ 260,
+ 221,
+ 197,
+ 176,
+ 167,
+ 2,
+ 101,
+ 41
+ ],
+ [
+ 284,
+ 232,
+ 50,
+ 192,
+ 11,
+ 35,
+ 186,
+ 70,
+ 266,
+ 33,
+ 248,
+ 245,
+ 280,
+ 196,
+ 297,
+ 69,
+ 171,
+ 108,
+ 30,
+ 287
+ ],
+ [
+ 255,
+ 294,
+ 282,
+ 152,
+ 261,
+ 81,
+ 174,
+ 165,
+ 222,
+ 216,
+ 87,
+ 151,
+ 178,
+ 292,
+ 258,
+ 208,
+ 94,
+ 227,
+ 242,
+ 157
+ ],
+ [
+ 109,
+ 253,
+ 145,
+ 62,
+ 278,
+ 13,
+ 60,
+ 160,
+ 88,
+ 298,
+ 237,
+ 112,
+ 131,
+ 124,
+ 102,
+ 5,
+ 111,
+ 123,
+ 148,
+ 86
+ ],
+ [
+ 188,
+ 106,
+ 72,
+ 110,
+ 169,
+ 295,
+ 250,
+ 8,
+ 142,
+ 76,
+ 59,
+ 141,
+ 3,
+ 96,
+ 80,
+ 66,
+ 272,
+ 79,
+ 68,
+ 125
+ ],
+ [
+ 17,
+ 234,
+ 89,
+ 102,
+ 267,
+ 299,
+ 74,
+ 160,
+ 253,
+ 206,
+ 137,
+ 259,
+ 131,
+ 112,
+ 13,
+ 145,
+ 254,
+ 71,
+ 60,
+ 237
+ ],
+ [
+ 213,
+ 83,
+ 254,
+ 163,
+ 239,
+ 233,
+ 71,
+ 42,
+ 89,
+ 206,
+ 267,
+ 234,
+ 286,
+ 63,
+ 112,
+ 57,
+ 299,
+ 111,
+ 148,
+ 126
+ ],
+ [
+ 266,
+ 11,
+ 70,
+ 35,
+ 232,
+ 33,
+ 50,
+ 245,
+ 280,
+ 196,
+ 171,
+ 69,
+ 180,
+ 108,
+ 297,
+ 284,
+ 139,
+ 192,
+ 30,
+ 287
+ ],
+ [
+ 292,
+ 26,
+ 81,
+ 165,
+ 178,
+ 208,
+ 174,
+ 227,
+ 255,
+ 153,
+ 104,
+ 87,
+ 94,
+ 119,
+ 92,
+ 157,
+ 261,
+ 273,
+ 0,
+ 140
+ ],
+ [
+ 76,
+ 295,
+ 204,
+ 72,
+ 183,
+ 169,
+ 110,
+ 8,
+ 125,
+ 142,
+ 59,
+ 141,
+ 68,
+ 96,
+ 250,
+ 79,
+ 3,
+ 106,
+ 6,
+ 274
+ ],
+ [
+ 25,
+ 281,
+ 235,
+ 34,
+ 90,
+ 203,
+ 54,
+ 291,
+ 162,
+ 194,
+ 221,
+ 101,
+ 197,
+ 296,
+ 263,
+ 2,
+ 260,
+ 212,
+ 9,
+ 176
+ ],
+ [
+ 1,
+ 269,
+ 142,
+ 98,
+ 172,
+ 214,
+ 59,
+ 6,
+ 75,
+ 143,
+ 295,
+ 198,
+ 195,
+ 293,
+ 80,
+ 169,
+ 125,
+ 76,
+ 92,
+ 140
+ ],
+ [
+ 146,
+ 47,
+ 82,
+ 97,
+ 159,
+ 93,
+ 201,
+ 122,
+ 29,
+ 51,
+ 224,
+ 56,
+ 271,
+ 84,
+ 265,
+ 200,
+ 209,
+ 244,
+ 100,
+ 288
+ ],
+ [
+ 180,
+ 232,
+ 11,
+ 50,
+ 35,
+ 70,
+ 284,
+ 186,
+ 33,
+ 245,
+ 266,
+ 248,
+ 297,
+ 280,
+ 108,
+ 171,
+ 196,
+ 69,
+ 139,
+ 30
+ ],
+ [
+ 3,
+ 161,
+ 8,
+ 295,
+ 169,
+ 96,
+ 125,
+ 66,
+ 80,
+ 68,
+ 274,
+ 272,
+ 6,
+ 43,
+ 149,
+ 142,
+ 79,
+ 141,
+ 76,
+ 290
+ ],
+ [
+ 90,
+ 189,
+ 260,
+ 291,
+ 235,
+ 101,
+ 197,
+ 34,
+ 203,
+ 25,
+ 54,
+ 281,
+ 162,
+ 221,
+ 263,
+ 212,
+ 2,
+ 9,
+ 296,
+ 236
+ ],
+ [
+ 172,
+ 293,
+ 269,
+ 98,
+ 143,
+ 214,
+ 75,
+ 190,
+ 273,
+ 165,
+ 59,
+ 103,
+ 242,
+ 142,
+ 21,
+ 261,
+ 104,
+ 76,
+ 250,
+ 216
+ ],
+ [
+ 69,
+ 171,
+ 280,
+ 108,
+ 245,
+ 33,
+ 266,
+ 297,
+ 186,
+ 139,
+ 11,
+ 35,
+ 70,
+ 232,
+ 50,
+ 30,
+ 287,
+ 180,
+ 284,
+ 192
+ ],
+ [
+ 291,
+ 263,
+ 212,
+ 221,
+ 9,
+ 101,
+ 235,
+ 217,
+ 277,
+ 2,
+ 236,
+ 162,
+ 203,
+ 77,
+ 211,
+ 296,
+ 37,
+ 34,
+ 167,
+ 279
+ ],
+ [
+ 75,
+ 14,
+ 231,
+ 115,
+ 214,
+ 190,
+ 6,
+ 79,
+ 76,
+ 169,
+ 295,
+ 125,
+ 269,
+ 274,
+ 1,
+ 204,
+ 161,
+ 195,
+ 80,
+ 78
+ ],
+ [
+ 32,
+ 136,
+ 95,
+ 53,
+ 19,
+ 116,
+ 45,
+ 120,
+ 65,
+ 154,
+ 135,
+ 207,
+ 127,
+ 61,
+ 20,
+ 58,
+ 240,
+ 246,
+ 36,
+ 264
+ ],
+ [
+ 288,
+ 159,
+ 224,
+ 93,
+ 56,
+ 209,
+ 201,
+ 271,
+ 97,
+ 82,
+ 29,
+ 122,
+ 265,
+ 100,
+ 127,
+ 84,
+ 244,
+ 36,
+ 47,
+ 51
+ ],
+ [
+ 29,
+ 56,
+ 97,
+ 159,
+ 271,
+ 209,
+ 122,
+ 288,
+ 82,
+ 93,
+ 200,
+ 224,
+ 191,
+ 47,
+ 146,
+ 265,
+ 100,
+ 84,
+ 244,
+ 127
+ ],
+ [
+ 228,
+ 144,
+ 247,
+ 86,
+ 114,
+ 176,
+ 167,
+ 279,
+ 237,
+ 296,
+ 62,
+ 2,
+ 71,
+ 39,
+ 34,
+ 203,
+ 25,
+ 276,
+ 162,
+ 9
+ ],
+ [
+ 296,
+ 162,
+ 235,
+ 2,
+ 291,
+ 34,
+ 197,
+ 221,
+ 189,
+ 25,
+ 101,
+ 9,
+ 167,
+ 212,
+ 90,
+ 263,
+ 176,
+ 236,
+ 277,
+ 281
+ ],
+ [
+ 125,
+ 188,
+ 76,
+ 295,
+ 96,
+ 8,
+ 72,
+ 68,
+ 3,
+ 169,
+ 6,
+ 141,
+ 59,
+ 66,
+ 79,
+ 142,
+ 183,
+ 110,
+ 274,
+ 143
+ ],
+ [
+ 257,
+ 229,
+ 85,
+ 218,
+ 31,
+ 264,
+ 84,
+ 7,
+ 100,
+ 61,
+ 45,
+ 58,
+ 51,
+ 219,
+ 246,
+ 116,
+ 173,
+ 244,
+ 23,
+ 207
+ ],
+ [
+ 254,
+ 89,
+ 299,
+ 137,
+ 160,
+ 267,
+ 111,
+ 60,
+ 102,
+ 13,
+ 112,
+ 145,
+ 17,
+ 148,
+ 234,
+ 71,
+ 63,
+ 27,
+ 57,
+ 298
+ ],
+ [
+ 53,
+ 45,
+ 173,
+ 246,
+ 95,
+ 264,
+ 136,
+ 244,
+ 36,
+ 32,
+ 61,
+ 154,
+ 265,
+ 127,
+ 58,
+ 116,
+ 224,
+ 100,
+ 199,
+ 135
+ ],
+ [
+ 119,
+ 174,
+ 227,
+ 222,
+ 282,
+ 157,
+ 26,
+ 255,
+ 238,
+ 294,
+ 165,
+ 181,
+ 132,
+ 216,
+ 10,
+ 155,
+ 251,
+ 0,
+ 81,
+ 178
+ ],
+ [
+ 159,
+ 201,
+ 29,
+ 56,
+ 200,
+ 82,
+ 288,
+ 271,
+ 97,
+ 93,
+ 224,
+ 47,
+ 122,
+ 265,
+ 244,
+ 100,
+ 127,
+ 53,
+ 146,
+ 191
+ ],
+ [
+ 40,
+ 226,
+ 249,
+ 158,
+ 103,
+ 293,
+ 113,
+ 242,
+ 151,
+ 261,
+ 220,
+ 252,
+ 81,
+ 126,
+ 165,
+ 185,
+ 213,
+ 152,
+ 21,
+ 233
+ ],
+ [
+ 236,
+ 9,
+ 277,
+ 77,
+ 37,
+ 276,
+ 138,
+ 212,
+ 129,
+ 2,
+ 133,
+ 217,
+ 285,
+ 197,
+ 279,
+ 268,
+ 223,
+ 263,
+ 101,
+ 221
+ ],
+ [
+ 217,
+ 197,
+ 263,
+ 9,
+ 277,
+ 221,
+ 236,
+ 37,
+ 2,
+ 101,
+ 77,
+ 291,
+ 285,
+ 235,
+ 211,
+ 133,
+ 162,
+ 203,
+ 268,
+ 167
+ ],
+ [
+ 185,
+ 83,
+ 239,
+ 163,
+ 42,
+ 233,
+ 254,
+ 71,
+ 22,
+ 286,
+ 234,
+ 267,
+ 89,
+ 63,
+ 112,
+ 206,
+ 298,
+ 148,
+ 38,
+ 111
+ ],
+ [
+ 269,
+ 293,
+ 142,
+ 195,
+ 172,
+ 250,
+ 76,
+ 190,
+ 75,
+ 169,
+ 143,
+ 98,
+ 103,
+ 79,
+ 115,
+ 59,
+ 198,
+ 295,
+ 80,
+ 113
+ ],
+ [
+ 223,
+ 138,
+ 129,
+ 133,
+ 37,
+ 77,
+ 285,
+ 211,
+ 277,
+ 268,
+ 236,
+ 276,
+ 217,
+ 9,
+ 279,
+ 212,
+ 2,
+ 167,
+ 263,
+ 197
+ ],
+ [
+ 151,
+ 165,
+ 181,
+ 261,
+ 222,
+ 251,
+ 255,
+ 208,
+ 174,
+ 152,
+ 242,
+ 273,
+ 227,
+ 294,
+ 26,
+ 10,
+ 238,
+ 81,
+ 282,
+ 103
+ ],
+ [
+ 212,
+ 277,
+ 285,
+ 263,
+ 236,
+ 77,
+ 37,
+ 9,
+ 197,
+ 268,
+ 221,
+ 133,
+ 101,
+ 2,
+ 211,
+ 129,
+ 235,
+ 291,
+ 138,
+ 279
+ ],
+ [
+ 257,
+ 84,
+ 100,
+ 244,
+ 51,
+ 264,
+ 246,
+ 224,
+ 205,
+ 200,
+ 64,
+ 265,
+ 146,
+ 47,
+ 31,
+ 61,
+ 58,
+ 127,
+ 85,
+ 93
+ ],
+ [
+ 218,
+ 257,
+ 229,
+ 205,
+ 85,
+ 31,
+ 84,
+ 51,
+ 264,
+ 16,
+ 100,
+ 146,
+ 47,
+ 147,
+ 130,
+ 7,
+ 224,
+ 64,
+ 61,
+ 46
+ ],
+ [
+ 226,
+ 275,
+ 44,
+ 24,
+ 106,
+ 135,
+ 272,
+ 19,
+ 183,
+ 65,
+ 199,
+ 210,
+ 110,
+ 166,
+ 113,
+ 249,
+ 136,
+ 80,
+ 120,
+ 250
+ ],
+ [
+ 235,
+ 162,
+ 263,
+ 197,
+ 9,
+ 212,
+ 2,
+ 176,
+ 167,
+ 236,
+ 217,
+ 296,
+ 34,
+ 291,
+ 203,
+ 101,
+ 277,
+ 41,
+ 189,
+ 281
+ ],
+ [
+ 294,
+ 174,
+ 251,
+ 208,
+ 181,
+ 227,
+ 282,
+ 26,
+ 10,
+ 165,
+ 216,
+ 151,
+ 238,
+ 81,
+ 132,
+ 255,
+ 178,
+ 242,
+ 157,
+ 273
+ ],
+ [
+ 138,
+ 77,
+ 37,
+ 277,
+ 133,
+ 285,
+ 268,
+ 129,
+ 211,
+ 215,
+ 217,
+ 236,
+ 9,
+ 212,
+ 279,
+ 276,
+ 2,
+ 263,
+ 197,
+ 101
+ ],
+ [
+ 265,
+ 200,
+ 159,
+ 100,
+ 84,
+ 51,
+ 82,
+ 244,
+ 127,
+ 288,
+ 56,
+ 93,
+ 209,
+ 97,
+ 36,
+ 201,
+ 29,
+ 246,
+ 264,
+ 64
+ ],
+ [
+ 170,
+ 223,
+ 30,
+ 133,
+ 280,
+ 211,
+ 248,
+ 268,
+ 186,
+ 70,
+ 266,
+ 138,
+ 276,
+ 279,
+ 50,
+ 232,
+ 180,
+ 236,
+ 77,
+ 37
+ ],
+ [
+ 220,
+ 24,
+ 275,
+ 44,
+ 106,
+ 210,
+ 135,
+ 110,
+ 113,
+ 19,
+ 183,
+ 272,
+ 65,
+ 40,
+ 250,
+ 249,
+ 76,
+ 199,
+ 126,
+ 188
+ ],
+ [
+ 165,
+ 208,
+ 174,
+ 157,
+ 81,
+ 119,
+ 222,
+ 273,
+ 255,
+ 87,
+ 282,
+ 294,
+ 283,
+ 181,
+ 178,
+ 251,
+ 140,
+ 292,
+ 132,
+ 153
+ ],
+ [
+ 296,
+ 167,
+ 34,
+ 176,
+ 25,
+ 162,
+ 203,
+ 179,
+ 90,
+ 2,
+ 281,
+ 9,
+ 54,
+ 221,
+ 189,
+ 276,
+ 235,
+ 279,
+ 133,
+ 291
+ ],
+ [
+ 85,
+ 31,
+ 257,
+ 51,
+ 205,
+ 264,
+ 84,
+ 64,
+ 218,
+ 45,
+ 61,
+ 224,
+ 100,
+ 116,
+ 265,
+ 23,
+ 219,
+ 58,
+ 47,
+ 207
+ ],
+ [
+ 39,
+ 237,
+ 128,
+ 278,
+ 38,
+ 5,
+ 91,
+ 27,
+ 60,
+ 111,
+ 13,
+ 137,
+ 62,
+ 148,
+ 298,
+ 109,
+ 175,
+ 63,
+ 160,
+ 112
+ ],
+ [
+ 198,
+ 75,
+ 14,
+ 115,
+ 190,
+ 214,
+ 79,
+ 6,
+ 78,
+ 195,
+ 76,
+ 269,
+ 169,
+ 125,
+ 204,
+ 295,
+ 274,
+ 143,
+ 1,
+ 80
+ ],
+ [
+ 50,
+ 11,
+ 70,
+ 180,
+ 186,
+ 192,
+ 35,
+ 266,
+ 284,
+ 248,
+ 280,
+ 33,
+ 245,
+ 196,
+ 171,
+ 108,
+ 30,
+ 69,
+ 297,
+ 139
+ ],
+ [
+ 42,
+ 163,
+ 239,
+ 22,
+ 185,
+ 213,
+ 15,
+ 71,
+ 83,
+ 267,
+ 254,
+ 234,
+ 88,
+ 286,
+ 63,
+ 89,
+ 112,
+ 38,
+ 105,
+ 123
+ ],
+ [
+ 17,
+ 267,
+ 89,
+ 112,
+ 254,
+ 160,
+ 145,
+ 206,
+ 253,
+ 184,
+ 299,
+ 102,
+ 74,
+ 124,
+ 71,
+ 286,
+ 148,
+ 63,
+ 13,
+ 298
+ ],
+ [
+ 221,
+ 162,
+ 197,
+ 291,
+ 203,
+ 34,
+ 263,
+ 189,
+ 101,
+ 212,
+ 2,
+ 296,
+ 25,
+ 9,
+ 217,
+ 281,
+ 236,
+ 167,
+ 176,
+ 277
+ ],
+ [
+ 9,
+ 277,
+ 217,
+ 211,
+ 77,
+ 285,
+ 263,
+ 212,
+ 2,
+ 197,
+ 268,
+ 37,
+ 221,
+ 138,
+ 279,
+ 129,
+ 133,
+ 276,
+ 101,
+ 176
+ ],
+ [
+ 278,
+ 60,
+ 137,
+ 13,
+ 298,
+ 160,
+ 111,
+ 5,
+ 109,
+ 112,
+ 145,
+ 38,
+ 39,
+ 27,
+ 286,
+ 71,
+ 148,
+ 89,
+ 63,
+ 206
+ ],
+ [
+ 10,
+ 132,
+ 174,
+ 251,
+ 294,
+ 208,
+ 140,
+ 222,
+ 157,
+ 181,
+ 255,
+ 282,
+ 227,
+ 165,
+ 119,
+ 81,
+ 26,
+ 242,
+ 151,
+ 0
+ ],
+ [
+ 42,
+ 286,
+ 71,
+ 38,
+ 254,
+ 83,
+ 63,
+ 298,
+ 22,
+ 111,
+ 148,
+ 233,
+ 267,
+ 112,
+ 213,
+ 185,
+ 89,
+ 123,
+ 163,
+ 234
+ ],
+ [
+ 58,
+ 61,
+ 127,
+ 36,
+ 154,
+ 20,
+ 32,
+ 95,
+ 53,
+ 246,
+ 136,
+ 264,
+ 199,
+ 23,
+ 207,
+ 19,
+ 224,
+ 7,
+ 100,
+ 116
+ ],
+ [
+ 164,
+ 107,
+ 176,
+ 138,
+ 276,
+ 37,
+ 285,
+ 223,
+ 236,
+ 279,
+ 277,
+ 268,
+ 270,
+ 133,
+ 129,
+ 77,
+ 215,
+ 211,
+ 217,
+ 2
+ ],
+ [
+ 252,
+ 165,
+ 261,
+ 103,
+ 151,
+ 152,
+ 251,
+ 81,
+ 181,
+ 294,
+ 174,
+ 216,
+ 222,
+ 10,
+ 92,
+ 140,
+ 255,
+ 227,
+ 238,
+ 153
+ ],
+ [
+ 99,
+ 276,
+ 211,
+ 138,
+ 203,
+ 129,
+ 9,
+ 2,
+ 279,
+ 236,
+ 77,
+ 37,
+ 194,
+ 296,
+ 291,
+ 197,
+ 167,
+ 277,
+ 90,
+ 212
+ ],
+ [
+ 100,
+ 265,
+ 224,
+ 246,
+ 82,
+ 207,
+ 173,
+ 84,
+ 264,
+ 159,
+ 51,
+ 209,
+ 200,
+ 47,
+ 64,
+ 218,
+ 29,
+ 56,
+ 53,
+ 288
+ ],
+ [
+ 108,
+ 69,
+ 33,
+ 196,
+ 280,
+ 297,
+ 35,
+ 171,
+ 139,
+ 186,
+ 70,
+ 266,
+ 11,
+ 232,
+ 50,
+ 284,
+ 180,
+ 192,
+ 30,
+ 287
+ ],
+ [
+ 36,
+ 264,
+ 207,
+ 100,
+ 53,
+ 244,
+ 127,
+ 95,
+ 173,
+ 224,
+ 58,
+ 136,
+ 61,
+ 265,
+ 84,
+ 32,
+ 200,
+ 288,
+ 154,
+ 240
+ ],
+ [
+ 202,
+ 114,
+ 164,
+ 12,
+ 144,
+ 156,
+ 107,
+ 241,
+ 289,
+ 177,
+ 21,
+ 228,
+ 187,
+ 134,
+ 49,
+ 86,
+ 270,
+ 155,
+ 273,
+ 4
+ ],
+ [
+ 50,
+ 232,
+ 30,
+ 266,
+ 180,
+ 11,
+ 186,
+ 70,
+ 192,
+ 280,
+ 171,
+ 35,
+ 284,
+ 196,
+ 33,
+ 287,
+ 245,
+ 69,
+ 108,
+ 297
+ ],
+ [
+ 113,
+ 261,
+ 152,
+ 242,
+ 151,
+ 165,
+ 103,
+ 81,
+ 181,
+ 106,
+ 252,
+ 94,
+ 250,
+ 216,
+ 104,
+ 255,
+ 153,
+ 273,
+ 227,
+ 87
+ ],
+ [
+ 106,
+ 59,
+ 142,
+ 104,
+ 113,
+ 152,
+ 149,
+ 261,
+ 292,
+ 94,
+ 92,
+ 81,
+ 153,
+ 249,
+ 165,
+ 242,
+ 87,
+ 183,
+ 181,
+ 178
+ ],
+ [
+ 222,
+ 10,
+ 238,
+ 151,
+ 165,
+ 216,
+ 252,
+ 242,
+ 132,
+ 174,
+ 227,
+ 294,
+ 208,
+ 26,
+ 103,
+ 273,
+ 181,
+ 81,
+ 255,
+ 140
+ ],
+ [
+ 242,
+ 103,
+ 251,
+ 165,
+ 151,
+ 92,
+ 10,
+ 222,
+ 152,
+ 81,
+ 174,
+ 294,
+ 261,
+ 227,
+ 238,
+ 132,
+ 181,
+ 140,
+ 216,
+ 273
+ ],
+ [
+ 145,
+ 234,
+ 112,
+ 88,
+ 124,
+ 160,
+ 17,
+ 109,
+ 267,
+ 13,
+ 74,
+ 102,
+ 60,
+ 131,
+ 148,
+ 63,
+ 184,
+ 278,
+ 89,
+ 182
+ ],
+ [
+ 206,
+ 89,
+ 267,
+ 111,
+ 148,
+ 63,
+ 112,
+ 83,
+ 71,
+ 299,
+ 286,
+ 57,
+ 137,
+ 145,
+ 239,
+ 234,
+ 160,
+ 298,
+ 60,
+ 27
+ ],
+ [
+ 181,
+ 292,
+ 174,
+ 87,
+ 261,
+ 165,
+ 178,
+ 81,
+ 152,
+ 208,
+ 227,
+ 157,
+ 282,
+ 151,
+ 216,
+ 140,
+ 294,
+ 119,
+ 222,
+ 238
+ ],
+ [
+ 52,
+ 154,
+ 61,
+ 32,
+ 20,
+ 64,
+ 135,
+ 126,
+ 240,
+ 95,
+ 207,
+ 246,
+ 275,
+ 260,
+ 53,
+ 58,
+ 136,
+ 23,
+ 199,
+ 264
+ ],
+ [
+ 218,
+ 84,
+ 31,
+ 229,
+ 85,
+ 205,
+ 51,
+ 264,
+ 100,
+ 244,
+ 224,
+ 64,
+ 265,
+ 246,
+ 61,
+ 47,
+ 45,
+ 146,
+ 173,
+ 58
+ ],
+ [
+ 87,
+ 94,
+ 282,
+ 178,
+ 181,
+ 157,
+ 104,
+ 292,
+ 81,
+ 255,
+ 283,
+ 294,
+ 174,
+ 153,
+ 119,
+ 140,
+ 227,
+ 208,
+ 261,
+ 165
+ ],
+ [
+ 131,
+ 13,
+ 62,
+ 17,
+ 102,
+ 237,
+ 74,
+ 278,
+ 160,
+ 60,
+ 184,
+ 137,
+ 5,
+ 109,
+ 39,
+ 253,
+ 89,
+ 206,
+ 234,
+ 124
+ ],
+ [
+ 194,
+ 189,
+ 54,
+ 235,
+ 41,
+ 281,
+ 263,
+ 101,
+ 221,
+ 197,
+ 291,
+ 34,
+ 90,
+ 25,
+ 162,
+ 212,
+ 236,
+ 203,
+ 217,
+ 2
+ ],
+ [
+ 152,
+ 165,
+ 181,
+ 242,
+ 255,
+ 151,
+ 216,
+ 81,
+ 104,
+ 87,
+ 94,
+ 174,
+ 292,
+ 153,
+ 227,
+ 140,
+ 249,
+ 92,
+ 294,
+ 178
+ ],
+ [
+ 161,
+ 80,
+ 43,
+ 274,
+ 79,
+ 290,
+ 67,
+ 169,
+ 193,
+ 8,
+ 78,
+ 46,
+ 125,
+ 68,
+ 3,
+ 295,
+ 115,
+ 130,
+ 66,
+ 1
+ ],
+ [
+ 197,
+ 212,
+ 221,
+ 217,
+ 9,
+ 236,
+ 235,
+ 277,
+ 291,
+ 101,
+ 2,
+ 162,
+ 77,
+ 285,
+ 176,
+ 41,
+ 268,
+ 37,
+ 167,
+ 211
+ ],
+ [
+ 246,
+ 207,
+ 173,
+ 53,
+ 100,
+ 84,
+ 95,
+ 36,
+ 31,
+ 224,
+ 244,
+ 61,
+ 265,
+ 45,
+ 257,
+ 58,
+ 136,
+ 127,
+ 51,
+ 85
+ ],
+ [
+ 224,
+ 244,
+ 82,
+ 51,
+ 159,
+ 100,
+ 200,
+ 173,
+ 84,
+ 93,
+ 207,
+ 209,
+ 264,
+ 288,
+ 246,
+ 56,
+ 127,
+ 64,
+ 36,
+ 29
+ ],
+ [
+ 186,
+ 280,
+ 196,
+ 171,
+ 11,
+ 232,
+ 245,
+ 69,
+ 30,
+ 33,
+ 50,
+ 70,
+ 108,
+ 35,
+ 180,
+ 139,
+ 297,
+ 248,
+ 284,
+ 287
+ ],
+ [
+ 89,
+ 234,
+ 112,
+ 254,
+ 299,
+ 206,
+ 148,
+ 160,
+ 17,
+ 286,
+ 145,
+ 111,
+ 71,
+ 63,
+ 137,
+ 57,
+ 239,
+ 102,
+ 60,
+ 298
+ ],
+ [
+ 217,
+ 77,
+ 236,
+ 285,
+ 277,
+ 138,
+ 129,
+ 211,
+ 212,
+ 263,
+ 223,
+ 37,
+ 9,
+ 133,
+ 197,
+ 276,
+ 221,
+ 2,
+ 101,
+ 279
+ ],
+ [
+ 172,
+ 98,
+ 293,
+ 195,
+ 214,
+ 190,
+ 143,
+ 103,
+ 242,
+ 165,
+ 104,
+ 10,
+ 238,
+ 273,
+ 142,
+ 261,
+ 252,
+ 174,
+ 59,
+ 140
+ ],
+ [
+ 273,
+ 165,
+ 152,
+ 292,
+ 151,
+ 81,
+ 242,
+ 187,
+ 251,
+ 294,
+ 103,
+ 261,
+ 255,
+ 181,
+ 216,
+ 252,
+ 140,
+ 227,
+ 92,
+ 10
+ ],
+ [
+ 56,
+ 122,
+ 97,
+ 288,
+ 29,
+ 93,
+ 201,
+ 159,
+ 82,
+ 200,
+ 209,
+ 224,
+ 100,
+ 191,
+ 265,
+ 127,
+ 84,
+ 146,
+ 244,
+ 36
+ ],
+ [
+ 149,
+ 166,
+ 80,
+ 48,
+ 274,
+ 169,
+ 193,
+ 295,
+ 290,
+ 250,
+ 67,
+ 46,
+ 106,
+ 110,
+ 142,
+ 8,
+ 161,
+ 79,
+ 188,
+ 141
+ ],
+ [
+ 165,
+ 227,
+ 208,
+ 216,
+ 222,
+ 251,
+ 81,
+ 21,
+ 261,
+ 152,
+ 181,
+ 294,
+ 242,
+ 151,
+ 292,
+ 174,
+ 255,
+ 10,
+ 153,
+ 282
+ ],
+ [
+ 80,
+ 43,
+ 78,
+ 169,
+ 8,
+ 79,
+ 68,
+ 295,
+ 161,
+ 67,
+ 141,
+ 290,
+ 193,
+ 272,
+ 262,
+ 72,
+ 46,
+ 125,
+ 73,
+ 3
+ ],
+ [
+ 220,
+ 226,
+ 44,
+ 24,
+ 19,
+ 65,
+ 120,
+ 199,
+ 32,
+ 135,
+ 110,
+ 136,
+ 272,
+ 183,
+ 106,
+ 154,
+ 95,
+ 20,
+ 79,
+ 76
+ ],
+ [
+ 211,
+ 138,
+ 236,
+ 9,
+ 129,
+ 37,
+ 2,
+ 77,
+ 268,
+ 285,
+ 277,
+ 212,
+ 133,
+ 167,
+ 221,
+ 217,
+ 176,
+ 197,
+ 279,
+ 263
+ ],
+ [
+ 77,
+ 217,
+ 37,
+ 212,
+ 236,
+ 285,
+ 9,
+ 211,
+ 197,
+ 263,
+ 133,
+ 2,
+ 101,
+ 279,
+ 268,
+ 129,
+ 138,
+ 221,
+ 291,
+ 223
+ ],
+ [
+ 298,
+ 237,
+ 13,
+ 148,
+ 111,
+ 38,
+ 60,
+ 5,
+ 145,
+ 124,
+ 112,
+ 63,
+ 39,
+ 286,
+ 137,
+ 27,
+ 71,
+ 102,
+ 123,
+ 91
+ ],
+ [
+ 2,
+ 236,
+ 77,
+ 277,
+ 211,
+ 101,
+ 9,
+ 197,
+ 37,
+ 217,
+ 212,
+ 263,
+ 285,
+ 291,
+ 129,
+ 138,
+ 221,
+ 203,
+ 133,
+ 296
+ ],
+ [
+ 196,
+ 245,
+ 69,
+ 266,
+ 108,
+ 171,
+ 33,
+ 186,
+ 70,
+ 11,
+ 297,
+ 35,
+ 232,
+ 139,
+ 30,
+ 50,
+ 284,
+ 180,
+ 192,
+ 287
+ ],
+ [
+ 54,
+ 189,
+ 25,
+ 34,
+ 235,
+ 162,
+ 221,
+ 41,
+ 90,
+ 203,
+ 291,
+ 197,
+ 263,
+ 176,
+ 296,
+ 167,
+ 260,
+ 101,
+ 2,
+ 212
+ ],
+ [
+ 181,
+ 258,
+ 222,
+ 208,
+ 294,
+ 157,
+ 255,
+ 174,
+ 81,
+ 227,
+ 178,
+ 87,
+ 119,
+ 238,
+ 94,
+ 165,
+ 292,
+ 10,
+ 283,
+ 152
+ ],
+ [
+ 0,
+ 157,
+ 87,
+ 153,
+ 178,
+ 140,
+ 119,
+ 174,
+ 294,
+ 155,
+ 227,
+ 292,
+ 92,
+ 94,
+ 81,
+ 10,
+ 165,
+ 258,
+ 208,
+ 132
+ ],
+ [
+ 180,
+ 35,
+ 70,
+ 232,
+ 192,
+ 50,
+ 11,
+ 186,
+ 33,
+ 245,
+ 280,
+ 266,
+ 69,
+ 297,
+ 196,
+ 108,
+ 171,
+ 139,
+ 30,
+ 287
+ ],
+ [
+ 217,
+ 277,
+ 77,
+ 236,
+ 37,
+ 212,
+ 138,
+ 268,
+ 9,
+ 129,
+ 211,
+ 2,
+ 133,
+ 263,
+ 223,
+ 197,
+ 221,
+ 101,
+ 279,
+ 276
+ ],
+ [
+ 298,
+ 38,
+ 112,
+ 71,
+ 63,
+ 148,
+ 111,
+ 239,
+ 254,
+ 267,
+ 89,
+ 145,
+ 278,
+ 42,
+ 57,
+ 22,
+ 137,
+ 237,
+ 234,
+ 124
+ ],
+ [
+ 186,
+ 266,
+ 50,
+ 196,
+ 11,
+ 171,
+ 33,
+ 70,
+ 139,
+ 69,
+ 232,
+ 35,
+ 280,
+ 108,
+ 245,
+ 180,
+ 30,
+ 297,
+ 284,
+ 248
+ ],
+ [
+ 56,
+ 271,
+ 200,
+ 29,
+ 159,
+ 97,
+ 201,
+ 93,
+ 122,
+ 82,
+ 209,
+ 224,
+ 127,
+ 265,
+ 36,
+ 100,
+ 246,
+ 244,
+ 84,
+ 53
+ ],
+ [
+ 134,
+ 59,
+ 187,
+ 142,
+ 92,
+ 1,
+ 4,
+ 174,
+ 165,
+ 3,
+ 152,
+ 104,
+ 96,
+ 153,
+ 227,
+ 140,
+ 250,
+ 149,
+ 261,
+ 292
+ ],
+ [
+ 79,
+ 67,
+ 80,
+ 169,
+ 161,
+ 46,
+ 274,
+ 262,
+ 272,
+ 115,
+ 193,
+ 295,
+ 149,
+ 142,
+ 8,
+ 43,
+ 250,
+ 188,
+ 76,
+ 3
+ ],
+ [
+ 197,
+ 101,
+ 235,
+ 203,
+ 212,
+ 263,
+ 221,
+ 162,
+ 2,
+ 9,
+ 189,
+ 277,
+ 34,
+ 217,
+ 296,
+ 90,
+ 236,
+ 77,
+ 194,
+ 211
+ ],
+ [
+ 178,
+ 87,
+ 81,
+ 255,
+ 104,
+ 94,
+ 174,
+ 165,
+ 92,
+ 153,
+ 181,
+ 294,
+ 140,
+ 283,
+ 152,
+ 261,
+ 227,
+ 157,
+ 258,
+ 187
+ ],
+ [
+ 172,
+ 269,
+ 98,
+ 195,
+ 214,
+ 103,
+ 273,
+ 242,
+ 165,
+ 216,
+ 261,
+ 151,
+ 143,
+ 252,
+ 251,
+ 104,
+ 181,
+ 249,
+ 152,
+ 81
+ ],
+ [
+ 10,
+ 222,
+ 181,
+ 174,
+ 178,
+ 81,
+ 238,
+ 282,
+ 165,
+ 283,
+ 208,
+ 227,
+ 292,
+ 132,
+ 140,
+ 157,
+ 255,
+ 251,
+ 242,
+ 92
+ ],
+ [
+ 169,
+ 8,
+ 76,
+ 125,
+ 3,
+ 188,
+ 193,
+ 96,
+ 6,
+ 79,
+ 274,
+ 141,
+ 142,
+ 68,
+ 204,
+ 80,
+ 72,
+ 110,
+ 43,
+ 161
+ ],
+ [
+ 203,
+ 162,
+ 2,
+ 167,
+ 34,
+ 221,
+ 235,
+ 9,
+ 176,
+ 197,
+ 25,
+ 212,
+ 291,
+ 189,
+ 101,
+ 37,
+ 263,
+ 236,
+ 211,
+ 277
+ ],
+ [
+ 108,
+ 245,
+ 33,
+ 139,
+ 69,
+ 196,
+ 35,
+ 171,
+ 280,
+ 11,
+ 70,
+ 186,
+ 266,
+ 232,
+ 50,
+ 284,
+ 180,
+ 192,
+ 287,
+ 30
+ ],
+ [
+ 63,
+ 111,
+ 148,
+ 278,
+ 286,
+ 38,
+ 112,
+ 145,
+ 71,
+ 237,
+ 60,
+ 239,
+ 5,
+ 27,
+ 254,
+ 13,
+ 124,
+ 123,
+ 137,
+ 57
+ ],
+ [
+ 206,
+ 89,
+ 267,
+ 254,
+ 160,
+ 137,
+ 17,
+ 234,
+ 13,
+ 60,
+ 128,
+ 111,
+ 145,
+ 112,
+ 148,
+ 27,
+ 102,
+ 63,
+ 184,
+ 5
+ ]
+ ]
+ },
+ "matched": {
+ "modalityWeights": [
+ [
+ 0.6539129467956886,
+ 0.34608705320431143
+ ],
+ [
+ 0.5005528309450097,
+ 0.49944716905499037
+ ],
+ [
+ 0.5754464639047352,
+ 0.4245535360952648
+ ],
+ [
+ 0.5238097488945351,
+ 0.4761902511054649
+ ],
+ [
+ 0.5551602153654673,
+ 0.44483978463453255
+ ],
+ [
+ 0.6950728908326663,
+ 0.30492710916733373
+ ],
+ [
+ 0.5170909289701056,
+ 0.48290907102989444
+ ],
+ [
+ 0.5071023699651365,
+ 0.4928976300348636
+ ],
+ [
+ 0.5759762658977855,
+ 0.4240237341022145
+ ],
+ [
+ 0.5247072170267407,
+ 0.4752927829732592
+ ],
+ [
+ 0.45934724776205044,
+ 0.5406527522379496
+ ],
+ [
+ 0.4329898419101652,
+ 0.5670101580898348
+ ],
+ [
+ 0.5667427885527165,
+ 0.4332572114472835
+ ],
+ [
+ 0.4187031740808971,
+ 0.581296825919103
+ ],
+ [
+ 0.4738353748706149,
+ 0.5261646251293851
+ ],
+ [
+ 0.47337530310265474,
+ 0.5266246968973454
+ ],
+ [
+ 0.5235360118597621,
+ 0.4764639881402379
+ ],
+ [
+ 0.43639659064819025,
+ 0.5636034093518097
+ ],
+ [
+ 0.6683679330767514,
+ 0.33163206692324865
+ ],
+ [
+ 0.47557657521292945,
+ 0.5244234247870706
+ ],
+ [
+ 0.5178805829158881,
+ 0.48211941708411193
+ ],
+ [
+ 0.5214187644944117,
+ 0.4785812355055883
+ ],
+ [
+ 0.5323307997349953,
+ 0.46766920026500464
+ ],
+ [
+ 0.6870576722873746,
+ 0.31294232771262537
+ ],
+ [
+ 0.8259597653682544,
+ 0.17404023463174556
+ ],
+ [
+ 0.5069914827498374,
+ 0.4930085172501627
+ ],
+ [
+ 0.5473378205376608,
+ 0.45266217946233933
+ ],
+ [
+ 0.35956173084325194,
+ 0.6404382691567481
+ ],
+ [
+ 0.9784779677845821,
+ 0.021522032215417833
+ ],
+ [
+ 0.5561391128131472,
+ 0.44386088718685285
+ ],
+ [
+ 0.5221495607366702,
+ 0.4778504392633299
+ ],
+ [
+ 0.5092362639494996,
+ 0.49076373605050044
+ ],
+ [
+ 0.5170460533873412,
+ 0.4829539466126588
+ ],
+ [
+ 0.4784567904844509,
+ 0.5215432095155491
+ ],
+ [
+ 0.483863948120131,
+ 0.5161360518798689
+ ],
+ [
+ 0.4992023456702395,
+ 0.5007976543297605
+ ],
+ [
+ 0.583785383933228,
+ 0.41621461606677207
+ ],
+ [
+ 0.49540734251304397,
+ 0.504592657486956
+ ],
+ [
+ 0.22750323303484316,
+ 0.7724967669651568
+ ],
+ [
+ 0.44534629522465624,
+ 0.5546537047753437
+ ],
+ [
+ 1.0,
+ 6.159380605415085e-87
+ ],
+ [
+ 0.4070771594732166,
+ 0.5929228405267835
+ ],
+ [
+ 0.4763542312097961,
+ 0.5236457687902039
+ ],
+ [
+ 0.5009241018275572,
+ 0.49907589817244274
+ ],
+ [
+ 0.9688345219419929,
+ 0.03116547805800698
+ ],
+ [
+ 0.5227812541325075,
+ 0.47721874586749247
+ ],
+ [
+ 0.6091549376432754,
+ 0.39084506235672445
+ ],
+ [
+ 0.9992986751922646,
+ 0.0007013248077354533
+ ],
+ [
+ 0.717727731015722,
+ 0.28227226898427804
+ ],
+ [
+ 0.5228503083641532,
+ 0.47714969163584675
+ ],
+ [
+ 0.4968568588869422,
+ 0.5031431411130578
+ ],
+ [
+ 0.31414399094397794,
+ 0.6858560090560222
+ ],
+ [
+ 0.6744127127043199,
+ 0.32558728729568
+ ],
+ [
+ 0.41325032017766933,
+ 0.5867496798223306
+ ],
+ [
+ 0.5030237481423387,
+ 0.4969762518576613
+ ],
+ [
+ 0.5255294500892653,
+ 0.47447054991073473
+ ],
+ [
+ 0.3626925982317721,
+ 0.637307401768228
+ ],
+ [
+ 0.5942270839262097,
+ 0.40577291607379035
+ ],
+ [
+ 0.4861052295955077,
+ 0.5138947704044923
+ ],
+ [
+ 0.3910880208805629,
+ 0.6089119791194372
+ ],
+ [
+ 0.36039724230129494,
+ 0.6396027576987051
+ ],
+ [
+ 0.6597394922227338,
+ 0.3402605077772662
+ ],
+ [
+ 0.5866636728485289,
+ 0.41333632715147095
+ ],
+ [
+ 0.5581454650568256,
+ 0.4418545349431744
+ ],
+ [
+ 0.765759572846399,
+ 0.23424042715360102
+ ],
+ [
+ 0.4266349734123928,
+ 0.5733650265876072
+ ],
+ [
+ 0.4201132039507334,
+ 0.5798867960492666
+ ],
+ [
+ 0.48727012034410555,
+ 0.5127298796558946
+ ],
+ [
+ 0.5094910241802325,
+ 0.4905089758197675
+ ],
+ [
+ 0.4998877002466747,
+ 0.5001122997533253
+ ],
+ [
+ 0.5040858065849555,
+ 0.4959141934150445
+ ],
+ [
+ 0.5901653900662981,
+ 0.4098346099337019
+ ],
+ [
+ 0.45204814204624927,
+ 0.5479518579537508
+ ],
+ [
+ 0.5356667304500599,
+ 0.4643332695499401
+ ],
+ [
+ 0.5636200111170857,
+ 0.4363799888829143
+ ],
+ [
+ 0.4489282005898543,
+ 0.5510717994101458
+ ],
+ [
+ 0.5244647643326206,
+ 0.47553523566737954
+ ],
+ [
+ 0.4895644967554528,
+ 0.5104355032445471
+ ],
+ [
+ 0.46738631450167045,
+ 0.5326136854983295
+ ],
+ [
+ 0.6065089739278534,
+ 0.3934910260721465
+ ],
+ [
+ 0.381714537056911,
+ 0.6182854629430891
+ ],
+ [
+ 0.45811105350729264,
+ 0.5418889464927074
+ ],
+ [
+ 9.211395949881404e-05,
+ 0.9999078860405012
+ ],
+ [
+ 0.5778752941400925,
+ 0.42212470585990747
+ ],
+ [
+ 0.4822366035799107,
+ 0.5177633964200894
+ ],
+ [
+ 0.6826408478957554,
+ 0.31735915210424465
+ ],
+ [
+ 0.5223605453420473,
+ 0.4776394546579527
+ ],
+ [
+ 0.09493931515244465,
+ 0.9050606848475554
+ ],
+ [
+ 0.5892907931654076,
+ 0.41070920683459244
+ ],
+ [
+ 0.5158013427015847,
+ 0.4841986572984152
+ ],
+ [
+ 0.5981960821738809,
+ 0.4018039178261191
+ ],
+ [
+ 0.5748814230999185,
+ 0.42511857690008154
+ ],
+ [
+ 0.4449239019959095,
+ 0.5550760980040904
+ ],
+ [
+ 0.10298881233177998,
+ 0.8970111876682201
+ ],
+ [
+ 0.5841656004826093,
+ 0.41583439951739076
+ ],
+ [
+ 0.5108979656211354,
+ 0.48910203437886457
+ ],
+ [
+ 0.4798185479796011,
+ 0.520181452020399
+ ],
+ [
+ 0.591523526865333,
+ 0.40847647313466684
+ ],
+ [
+ 0.4424079567229262,
+ 0.5575920432770739
+ ],
+ [
+ 0.702054058010333,
+ 0.29794594198966695
+ ],
+ [
+ 1.282615438703733e-13,
+ 0.9999999999998719
+ ],
+ [
+ 0.4786374218803099,
+ 0.5213625781196902
+ ],
+ [
+ 0.4684735348958579,
+ 0.5315264651041421
+ ],
+ [
+ 0.515024047194819,
+ 0.484975952805181
+ ],
+ [
+ 0.5550285390092692,
+ 0.44497146099073087
+ ],
+ [
+ 0.4459074929864471,
+ 0.554092507013553
+ ],
+ [
+ 0.5772430456204218,
+ 0.4227569543795783
+ ],
+ [
+ 0.999999999989953,
+ 1.0046874056495182e-11
+ ],
+ [
+ 0.5001474686751688,
+ 0.4998525313248312
+ ],
+ [
+ 0.483911102449037,
+ 0.5160888975509629
+ ],
+ [
+ 0.4276246852687962,
+ 0.5723753147312038
+ ],
+ [
+ 0.43454872378826037,
+ 0.5654512762117396
+ ],
+ [
+ 0.7656055775084116,
+ 0.23439442249158843
+ ],
+ [
+ 0.5634965948020472,
+ 0.43650340519795283
+ ],
+ [
+ 0.4962045235156115,
+ 0.5037954764843885
+ ],
+ [
+ 0.5885499562930517,
+ 0.4114500437069483
+ ],
+ [
+ 0.6967649365333001,
+ 0.3032350634667
+ ],
+ [
+ 1.0,
+ 1.636908536993497e-39
+ ],
+ [
+ 0.41598736906553446,
+ 0.5840126309344655
+ ],
+ [
+ 0.39949565124430697,
+ 0.600504348755693
+ ],
+ [
+ 0.555834317812554,
+ 0.444165682187446
+ ],
+ [
+ 0.5304565941416386,
+ 0.4695434058583614
+ ],
+ [
+ 0.29451923775939254,
+ 0.7054807622406075
+ ],
+ [
+ 0.5998351258095482,
+ 0.4001648741904517
+ ],
+ [
+ 0.6370741145976254,
+ 0.36292588540237447
+ ],
+ [
+ 0.5383337613209332,
+ 0.4616662386790668
+ ],
+ [
+ 0.39850820524776825,
+ 0.6014917947522318
+ ],
+ [
+ 0.974494863577063,
+ 0.025505136422936968
+ ],
+ [
+ 0.31502256866274914,
+ 0.6849774313372508
+ ],
+ [
+ 0.5,
+ 0.5
+ ],
+ [
+ 0.7343781131474476,
+ 0.2656218868525524
+ ],
+ [
+ 0.5032966179678294,
+ 0.4967033820321705
+ ],
+ [
+ 0.4340542284183294,
+ 0.5659457715816707
+ ],
+ [
+ 0.4611471682520311,
+ 0.5388528317479689
+ ],
+ [
+ 1.0,
+ 9.61831250765711e-87
+ ],
+ [
+ 0.7218143744777258,
+ 0.27818562552227427
+ ],
+ [
+ 0.5318330453503561,
+ 0.4681669546496439
+ ],
+ [
+ 0.5145750918796326,
+ 0.4854249081203674
+ ],
+ [
+ 0.5673920115519827,
+ 0.43260798844801734
+ ],
+ [
+ 0.4922961828623184,
+ 0.5077038171376815
+ ],
+ [
+ 0.5148360286440321,
+ 0.4851639713559679
+ ],
+ [
+ 0.00948165933653279,
+ 0.9905183406634671
+ ],
+ [
+ 0.5954580138792054,
+ 0.40454198612079456
+ ],
+ [
+ 0.7334475708961621,
+ 0.266552429103838
+ ],
+ [
+ 0.5206679020387066,
+ 0.4793320979612933
+ ],
+ [
+ 0.5307509111695216,
+ 0.46924908883047844
+ ],
+ [
+ 0.9914446646035028,
+ 0.008555335396497181
+ ],
+ [
+ 0.7195202740445349,
+ 0.2804797259554651
+ ],
+ [
+ 0.5777707513242399,
+ 0.4222292486757601
+ ],
+ [
+ 0.43034160338052224,
+ 0.5696583966194778
+ ],
+ [
+ 0.49595930885442513,
+ 0.5040406911455749
+ ],
+ [
+ 0.4431855394511467,
+ 0.5568144605488533
+ ],
+ [
+ 0.4855826424764669,
+ 0.5144173575235332
+ ],
+ [
+ 0.604778368038688,
+ 0.395221631961312
+ ],
+ [
+ 0.5877113522284889,
+ 0.41228864777151103
+ ],
+ [
+ 0.4537569506754902,
+ 0.5462430493245098
+ ],
+ [
+ 0.9328534143438826,
+ 0.06714658565611743
+ ],
+ [
+ 0.3966873056190041,
+ 0.6033126943809959
+ ],
+ [
+ 0.5782141515071971,
+ 0.4217858484928028
+ ],
+ [
+ 0.27801254014498916,
+ 0.7219874598550108
+ ],
+ [
+ 0.5,
+ 0.5
+ ],
+ [
+ 0.4083562278562988,
+ 0.5916437721437012
+ ],
+ [
+ 0.5063671904993268,
+ 0.4936328095006731
+ ],
+ [
+ 0.6831439407472581,
+ 0.3168560592527418
+ ],
+ [
+ 0.3619760782663433,
+ 0.6380239217336567
+ ],
+ [
+ 0.8056306815575891,
+ 0.19436931844241082
+ ],
+ [
+ 0.44378098806152144,
+ 0.5562190119384786
+ ],
+ [
+ 0.5513013735963163,
+ 0.44869862640368363
+ ],
+ [
+ 0.4866636111844257,
+ 0.5133363888155743
+ ],
+ [
+ 0.4476430401585537,
+ 0.5523569598414463
+ ],
+ [
+ 0.3284850044674816,
+ 0.6715149955325185
+ ],
+ [
+ 0.49063437519201764,
+ 0.5093656248079823
+ ],
+ [
+ 0.42636713473862436,
+ 0.5736328652613756
+ ],
+ [
+ 0.46544811444333734,
+ 0.5345518855566627
+ ],
+ [
+ 0.18307260064772546,
+ 0.8169273993522745
+ ],
+ [
+ 0.8367791389810948,
+ 0.1632208610189053
+ ],
+ [
+ 0.6117964090101642,
+ 0.3882035909898358
+ ],
+ [
+ 1.0,
+ 3.3171870055164244e-35
+ ],
+ [
+ 0.5652248951142479,
+ 0.4347751048857522
+ ],
+ [
+ 0.4967307076325477,
+ 0.5032692923674523
+ ],
+ [
+ 0.5012839563375934,
+ 0.4987160436624066
+ ],
+ [
+ 0.42495630299744136,
+ 0.5750436970025586
+ ],
+ [
+ 0.5063533501936246,
+ 0.49364664980637535
+ ],
+ [
+ 0.7194826309880774,
+ 0.28051736901192265
+ ],
+ [
+ 0.517233741566509,
+ 0.4827662584334909
+ ],
+ [
+ 0.6430268821424182,
+ 0.35697311785758185
+ ],
+ [
+ 0.5213966465218048,
+ 0.47860335347819527
+ ],
+ [
+ 0.3787135037794901,
+ 0.6212864962205099
+ ],
+ [
+ 0.6772607144009676,
+ 0.3227392855990325
+ ],
+ [
+ 0.5433480022979812,
+ 0.45665199770201875
+ ],
+ [
+ 0.7818864845920206,
+ 0.21811351540797952
+ ],
+ [
+ 0.8856589190468186,
+ 0.11434108095318134
+ ],
+ [
+ 0.5000139087158877,
+ 0.49998609128411237
+ ],
+ [
+ 0.5518985847500698,
+ 0.44810141524993025
+ ],
+ [
+ 0.6593602001442394,
+ 0.34063979985576065
+ ],
+ [
+ 0.6096916521887126,
+ 0.3903083478112873
+ ],
+ [
+ 0.5063709508152925,
+ 0.49362904918470757
+ ],
+ [
+ 0.515729416712278,
+ 0.4842705832877221
+ ],
+ [
+ 0.5439872229653826,
+ 0.4560127770346174
+ ],
+ [
+ 0.5113668925551097,
+ 0.4886331074448902
+ ],
+ [
+ 1.7928447901445487e-14,
+ 0.999999999999982
+ ],
+ [
+ 0.37503634813683046,
+ 0.6249636518631695
+ ],
+ [
+ 0.2917380761578324,
+ 0.7082619238421676
+ ],
+ [
+ 0.5578670434477949,
+ 0.4421329565522052
+ ],
+ [
+ 0.5352031912747136,
+ 0.4647968087252864
+ ],
+ [
+ 0.4695829792230943,
+ 0.5304170207769058
+ ],
+ [
+ 0.5503724111353538,
+ 0.4496275888646461
+ ],
+ [
+ 0.32073483117256557,
+ 0.6792651688274345
+ ],
+ [
+ 0.3425329007365539,
+ 0.657467099263446
+ ],
+ [
+ 0.8704937422978836,
+ 0.12950625770211646
+ ],
+ [
+ 0.7528049697383559,
+ 0.24719503026164416
+ ],
+ [
+ 0.5429648741544187,
+ 0.4570351258455813
+ ],
+ [
+ 0.4635520839379096,
+ 0.5364479160620905
+ ],
+ [
+ 0.7375112361049967,
+ 0.26248876389500336
+ ],
+ [
+ 0.3279834098364834,
+ 0.6720165901635166
+ ],
+ [
+ 0.6423311998094785,
+ 0.35766880019052144
+ ],
+ [
+ 0.5593274117728365,
+ 0.4406725882271635
+ ],
+ [
+ 0.5058135102624539,
+ 0.49418648973754614
+ ],
+ [
+ 0.5425326848133906,
+ 0.4574673151866095
+ ],
+ [
+ 0.43678280590299756,
+ 0.5632171940970023
+ ],
+ [
+ 0.5830656581758247,
+ 0.4169343418241753
+ ],
+ [
+ 0.5040010447083654,
+ 0.4959989552916347
+ ],
+ [
+ 0.4487098367407678,
+ 0.5512901632592321
+ ],
+ [
+ 0.5299722963991693,
+ 0.4700277036008306
+ ],
+ [
+ 0.049173572827288015,
+ 0.950826427172712
+ ],
+ [
+ 0.49708623976758115,
+ 0.5029137602324188
+ ],
+ [
+ 0.9884352783831255,
+ 0.011564721616874456
+ ],
+ [
+ 0.4986043131823737,
+ 0.5013956868176264
+ ],
+ [
+ 1.7748510345357e-20,
+ 1.0
+ ],
+ [
+ 0.6381053505518863,
+ 0.3618946494481138
+ ],
+ [
+ 0.5817576292595322,
+ 0.4182423707404679
+ ],
+ [
+ 0.5155705288806158,
+ 0.4844294711193841
+ ],
+ [
+ 0.538595522112016,
+ 0.4614044778879839
+ ],
+ [
+ 0.5600294304130854,
+ 0.4399705695869146
+ ],
+ [
+ 0.5710075862051468,
+ 0.4289924137948532
+ ],
+ [
+ 0.5894036433814939,
+ 0.41059635661850613
+ ],
+ [
+ 0.5560805837381906,
+ 0.4439194162618095
+ ],
+ [
+ 0.4972693154037791,
+ 0.502730684596221
+ ],
+ [
+ 0.5266623154578403,
+ 0.4733376845421597
+ ],
+ [
+ 0.4768947489131259,
+ 0.523105251086874
+ ],
+ [
+ 0.47267017958204,
+ 0.52732982041796
+ ],
+ [
+ 0.4980925690317283,
+ 0.5019074309682717
+ ],
+ [
+ 0.5907022270726298,
+ 0.4092977729273703
+ ],
+ [
+ 0.447630340883095,
+ 0.552369659116905
+ ],
+ [
+ 0.3918126746828219,
+ 0.608187325317178
+ ],
+ [
+ 0.5016795890167863,
+ 0.4983204109832136
+ ],
+ [
+ 0.0056325617225152814,
+ 0.9943674382774846
+ ],
+ [
+ 1.0,
+ 1.902544448864155e-86
+ ],
+ [
+ 0.5318772178740377,
+ 0.4681227821259622
+ ],
+ [
+ 0.5895426376075852,
+ 0.41045736239241476
+ ],
+ [
+ 0.74432475118513,
+ 0.25567524881486997
+ ],
+ [
+ 0.39423041146908144,
+ 0.6057695885309186
+ ],
+ [
+ 0.5817142963763303,
+ 0.4182857036236697
+ ],
+ [
+ 0.4704483578971759,
+ 0.529551642102824
+ ],
+ [
+ 0.5078986194438246,
+ 0.4921013805561754
+ ],
+ [
+ 0.47665615988629134,
+ 0.5233438401137087
+ ],
+ [
+ 0.5635848547511222,
+ 0.4364151452488778
+ ],
+ [
+ 0.5507344256879345,
+ 0.44926557431206554
+ ],
+ [
+ 0.2177568790821683,
+ 0.7822431209178317
+ ],
+ [
+ 0.4751183375808172,
+ 0.5248816624191828
+ ],
+ [
+ 0.5372148074854338,
+ 0.4627851925145661
+ ],
+ [
+ 0.39707801839027673,
+ 0.6029219816097231
+ ],
+ [
+ 0.4415992457123444,
+ 0.5584007542876556
+ ],
+ [
+ 0.5020372031885291,
+ 0.4979627968114709
+ ],
+ [
+ 2.04125463897604e-29,
+ 1.0
+ ],
+ [
+ 0.06653853151418182,
+ 0.9334614684858182
+ ],
+ [
+ 0.5170408889158712,
+ 0.4829591110841288
+ ],
+ [
+ 0.527872933316187,
+ 0.47212706668381305
+ ],
+ [
+ 0.5779599210330736,
+ 0.4220400789669264
+ ],
+ [
+ 0.5983454181849006,
+ 0.40165458181509944
+ ],
+ [
+ 2.6616462166536603e-83,
+ 1.0
+ ],
+ [
+ 0.6197740020403802,
+ 0.38022599795961975
+ ],
+ [
+ 0.35230899744420985,
+ 0.6476910025557902
+ ],
+ [
+ 0.512018149986145,
+ 0.487981850013855
+ ],
+ [
+ 0.2838556963014053,
+ 0.7161443036985947
+ ],
+ [
+ 0.5553941853159597,
+ 0.44460581468404026
+ ],
+ [
+ 0.5416069728522092,
+ 0.45839302714779084
+ ],
+ [
+ 0.44779622570651806,
+ 0.5522037742934819
+ ],
+ [
+ 0.3727694354618617,
+ 0.6272305645381383
+ ],
+ [
+ 0.5113562893413754,
+ 0.4886437106586246
+ ],
+ [
+ 0.5287429168618254,
+ 0.4712570831381747
+ ],
+ [
+ 0.4550712746385342,
+ 0.5449287253614659
+ ],
+ [
+ 0.45010486273072786,
+ 0.5498951372692721
+ ],
+ [
+ 0.29270830788849717,
+ 0.7072916921115028
+ ],
+ [
+ 0.5412368079856221,
+ 0.458763192014378
+ ],
+ [
+ 0.5107053990422126,
+ 0.4892946009577875
+ ],
+ [
+ 0.4695028574968475,
+ 0.5304971425031525
+ ],
+ [
+ 0.5152940981984201,
+ 0.4847059018015798
+ ],
+ [
+ 0.03871643935583882,
+ 0.9612835606441611
+ ],
+ [
+ 1.0,
+ 5.816072819036227e-87
+ ],
+ [
+ 0.4142807514170913,
+ 0.5857192485829088
+ ],
+ [
+ 0.4581639720652422,
+ 0.5418360279347578
+ ],
+ [
+ 0.4285198839830853,
+ 0.5714801160169146
+ ],
+ [
+ 0.475141045885898,
+ 0.524858954114102
+ ],
+ [
+ 0.4297408762596822,
+ 0.5702591237403178
+ ],
+ [
+ 0.48770939283636167,
+ 0.5122906071636383
+ ],
+ [
+ 0.5694532429996854,
+ 0.4305467570003146
+ ],
+ [
+ 0.5076811044920134,
+ 0.4923188955079865
+ ],
+ [
+ 0.6693413162368838,
+ 0.33065868376311625
+ ],
+ [
+ 0.5158418705662594,
+ 0.48415812943374054
+ ]
+ ],
+ "neighborIndices": [
+ [
+ 181,
+ 273,
+ 157,
+ 227,
+ 119,
+ 81,
+ 292,
+ 255,
+ 87,
+ 208,
+ 178,
+ 132,
+ 174,
+ 261,
+ 282,
+ 151,
+ 216,
+ 283,
+ 10,
+ 153
+ ],
+ [
+ 3,
+ 193,
+ 142,
+ 6,
+ 295,
+ 190,
+ 76,
+ 125,
+ 204,
+ 169,
+ 161,
+ 96,
+ 43,
+ 59,
+ 8,
+ 80,
+ 106,
+ 68,
+ 143,
+ 78
+ ],
+ [
+ 211,
+ 203,
+ 37,
+ 189,
+ 77,
+ 235,
+ 129,
+ 138,
+ 9,
+ 34,
+ 212,
+ 133,
+ 221,
+ 197,
+ 223,
+ 236,
+ 101,
+ 285,
+ 277,
+ 296
+ ],
+ [
+ 96,
+ 204,
+ 295,
+ 193,
+ 8,
+ 125,
+ 1,
+ 76,
+ 106,
+ 66,
+ 43,
+ 169,
+ 68,
+ 161,
+ 143,
+ 6,
+ 188,
+ 110,
+ 142,
+ 72
+ ],
+ [
+ 86,
+ 278,
+ 38,
+ 234,
+ 39,
+ 233,
+ 91,
+ 105,
+ 102,
+ 88,
+ 71,
+ 13,
+ 74,
+ 131,
+ 128,
+ 254,
+ 118,
+ 124,
+ 137,
+ 259
+ ],
+ [
+ 298,
+ 22,
+ 42,
+ 239,
+ 148,
+ 39,
+ 131,
+ 15,
+ 118,
+ 18,
+ 63,
+ 62,
+ 74,
+ 105,
+ 123,
+ 254,
+ 259,
+ 60,
+ 109,
+ 91
+ ],
+ [
+ 125,
+ 78,
+ 8,
+ 76,
+ 1,
+ 161,
+ 110,
+ 3,
+ 169,
+ 43,
+ 295,
+ 262,
+ 79,
+ 204,
+ 106,
+ 193,
+ 143,
+ 96,
+ 188,
+ 68
+ ],
+ [
+ 246,
+ 95,
+ 136,
+ 240,
+ 199,
+ 53,
+ 20,
+ 36,
+ 58,
+ 154,
+ 135,
+ 127,
+ 23,
+ 183,
+ 61,
+ 100,
+ 264,
+ 141,
+ 275,
+ 65
+ ],
+ [
+ 262,
+ 169,
+ 68,
+ 161,
+ 110,
+ 295,
+ 43,
+ 72,
+ 3,
+ 125,
+ 66,
+ 78,
+ 106,
+ 6,
+ 79,
+ 96,
+ 272,
+ 188,
+ 274,
+ 204
+ ],
+ [
+ 133,
+ 138,
+ 212,
+ 37,
+ 197,
+ 263,
+ 221,
+ 291,
+ 296,
+ 236,
+ 211,
+ 162,
+ 268,
+ 277,
+ 2,
+ 235,
+ 77,
+ 203,
+ 217,
+ 179
+ ],
+ [
+ 294,
+ 208,
+ 251,
+ 92,
+ 132,
+ 238,
+ 174,
+ 261,
+ 165,
+ 151,
+ 242,
+ 119,
+ 181,
+ 227,
+ 81,
+ 222,
+ 152,
+ 178,
+ 255,
+ 26
+ ],
+ [
+ 35,
+ 70,
+ 50,
+ 232,
+ 33,
+ 192,
+ 266,
+ 245,
+ 280,
+ 69,
+ 108,
+ 186,
+ 297,
+ 180,
+ 171,
+ 248,
+ 139,
+ 30,
+ 284,
+ 196
+ ],
+ [
+ 156,
+ 270,
+ 279,
+ 84,
+ 229,
+ 114,
+ 116,
+ 218,
+ 95,
+ 23,
+ 31,
+ 130,
+ 275,
+ 58,
+ 135,
+ 49,
+ 65,
+ 281,
+ 164,
+ 44
+ ],
+ [
+ 206,
+ 160,
+ 60,
+ 128,
+ 111,
+ 278,
+ 131,
+ 5,
+ 145,
+ 17,
+ 112,
+ 102,
+ 109,
+ 234,
+ 55,
+ 267,
+ 137,
+ 27,
+ 237,
+ 299
+ ],
+ [
+ 198,
+ 115,
+ 75,
+ 46,
+ 231,
+ 214,
+ 67,
+ 262,
+ 79,
+ 204,
+ 1,
+ 295,
+ 161,
+ 3,
+ 169,
+ 290,
+ 48,
+ 6,
+ 80,
+ 130
+ ],
+ [
+ 42,
+ 123,
+ 118,
+ 239,
+ 22,
+ 105,
+ 91,
+ 185,
+ 233,
+ 298,
+ 5,
+ 148,
+ 62,
+ 39,
+ 18,
+ 124,
+ 63,
+ 131,
+ 259,
+ 175
+ ],
+ [
+ 188,
+ 68,
+ 73,
+ 43,
+ 147,
+ 66,
+ 169,
+ 3,
+ 72,
+ 141,
+ 8,
+ 1,
+ 204,
+ 262,
+ 193,
+ 78,
+ 96,
+ 274,
+ 125,
+ 161
+ ],
+ [
+ 234,
+ 13,
+ 145,
+ 131,
+ 160,
+ 74,
+ 253,
+ 278,
+ 55,
+ 89,
+ 102,
+ 206,
+ 259,
+ 112,
+ 267,
+ 111,
+ 128,
+ 254,
+ 60,
+ 184
+ ],
+ [
+ 131,
+ 91,
+ 62,
+ 267,
+ 185,
+ 239,
+ 123,
+ 259,
+ 298,
+ 74,
+ 5,
+ 39,
+ 278,
+ 118,
+ 124,
+ 182,
+ 15,
+ 234,
+ 137,
+ 60
+ ],
+ [
+ 44,
+ 199,
+ 65,
+ 120,
+ 32,
+ 135,
+ 61,
+ 95,
+ 20,
+ 226,
+ 240,
+ 154,
+ 127,
+ 275,
+ 45,
+ 136,
+ 36,
+ 207,
+ 23,
+ 116
+ ],
+ [
+ 240,
+ 199,
+ 32,
+ 154,
+ 7,
+ 136,
+ 61,
+ 135,
+ 19,
+ 23,
+ 58,
+ 95,
+ 275,
+ 36,
+ 127,
+ 49,
+ 65,
+ 53,
+ 246,
+ 226
+ ],
+ [
+ 251,
+ 165,
+ 294,
+ 208,
+ 227,
+ 10,
+ 92,
+ 273,
+ 178,
+ 238,
+ 119,
+ 103,
+ 293,
+ 81,
+ 152,
+ 26,
+ 222,
+ 282,
+ 195,
+ 151
+ ],
+ [
+ 42,
+ 15,
+ 105,
+ 239,
+ 5,
+ 118,
+ 298,
+ 123,
+ 148,
+ 63,
+ 124,
+ 39,
+ 74,
+ 91,
+ 233,
+ 38,
+ 286,
+ 234,
+ 131,
+ 86
+ ],
+ [
+ 61,
+ 58,
+ 49,
+ 127,
+ 154,
+ 53,
+ 135,
+ 207,
+ 36,
+ 226,
+ 275,
+ 45,
+ 240,
+ 32,
+ 199,
+ 20,
+ 44,
+ 136,
+ 95,
+ 100
+ ],
+ [
+ 120,
+ 116,
+ 65,
+ 220,
+ 244,
+ 218,
+ 31,
+ 226,
+ 32,
+ 19,
+ 135,
+ 44,
+ 265,
+ 275,
+ 207,
+ 199,
+ 84,
+ 146,
+ 95,
+ 270
+ ],
+ [
+ 236,
+ 203,
+ 129,
+ 37,
+ 277,
+ 162,
+ 296,
+ 189,
+ 279,
+ 34,
+ 2,
+ 138,
+ 276,
+ 9,
+ 291,
+ 263,
+ 221,
+ 54,
+ 197,
+ 235
+ ],
+ [
+ 119,
+ 157,
+ 174,
+ 92,
+ 132,
+ 208,
+ 222,
+ 178,
+ 294,
+ 282,
+ 255,
+ 165,
+ 227,
+ 10,
+ 151,
+ 261,
+ 190,
+ 152,
+ 238,
+ 113
+ ],
+ [
+ 5,
+ 237,
+ 128,
+ 60,
+ 111,
+ 254,
+ 13,
+ 148,
+ 63,
+ 145,
+ 278,
+ 109,
+ 112,
+ 206,
+ 38,
+ 137,
+ 57,
+ 160,
+ 298,
+ 89
+ ],
+ [
+ 47,
+ 229,
+ 51,
+ 223,
+ 279,
+ 85,
+ 235,
+ 281,
+ 217,
+ 84,
+ 129,
+ 162,
+ 34,
+ 64,
+ 218,
+ 191,
+ 107,
+ 241,
+ 146,
+ 31
+ ],
+ [
+ 201,
+ 97,
+ 93,
+ 271,
+ 56,
+ 209,
+ 288,
+ 122,
+ 159,
+ 82,
+ 200,
+ 258,
+ 224,
+ 94,
+ 152,
+ 100,
+ 283,
+ 190,
+ 40,
+ 26
+ ],
+ [
+ 284,
+ 186,
+ 196,
+ 248,
+ 266,
+ 280,
+ 232,
+ 50,
+ 35,
+ 33,
+ 11,
+ 69,
+ 171,
+ 180,
+ 287,
+ 139,
+ 245,
+ 108,
+ 70,
+ 192
+ ],
+ [
+ 229,
+ 84,
+ 257,
+ 85,
+ 51,
+ 218,
+ 116,
+ 205,
+ 191,
+ 146,
+ 47,
+ 64,
+ 224,
+ 244,
+ 265,
+ 95,
+ 207,
+ 45,
+ 264,
+ 135
+ ],
+ [
+ 135,
+ 58,
+ 95,
+ 199,
+ 127,
+ 61,
+ 240,
+ 154,
+ 36,
+ 20,
+ 45,
+ 23,
+ 207,
+ 226,
+ 65,
+ 53,
+ 136,
+ 19,
+ 275,
+ 44
+ ],
+ [
+ 171,
+ 108,
+ 69,
+ 280,
+ 245,
+ 297,
+ 266,
+ 11,
+ 70,
+ 35,
+ 50,
+ 196,
+ 139,
+ 192,
+ 186,
+ 180,
+ 232,
+ 30,
+ 248,
+ 284
+ ],
+ [
+ 197,
+ 9,
+ 37,
+ 296,
+ 162,
+ 221,
+ 235,
+ 2,
+ 211,
+ 203,
+ 25,
+ 129,
+ 212,
+ 138,
+ 263,
+ 223,
+ 236,
+ 277,
+ 291,
+ 189
+ ],
+ [
+ 232,
+ 50,
+ 11,
+ 192,
+ 180,
+ 186,
+ 139,
+ 70,
+ 33,
+ 248,
+ 284,
+ 196,
+ 280,
+ 108,
+ 266,
+ 30,
+ 297,
+ 245,
+ 69,
+ 171
+ ],
+ [
+ 207,
+ 127,
+ 58,
+ 135,
+ 61,
+ 240,
+ 53,
+ 154,
+ 23,
+ 95,
+ 32,
+ 136,
+ 199,
+ 246,
+ 45,
+ 44,
+ 226,
+ 20,
+ 65,
+ 275
+ ],
+ [
+ 212,
+ 133,
+ 197,
+ 9,
+ 129,
+ 203,
+ 138,
+ 221,
+ 277,
+ 2,
+ 211,
+ 77,
+ 223,
+ 217,
+ 34,
+ 263,
+ 236,
+ 291,
+ 189,
+ 285
+ ],
+ [
+ 239,
+ 286,
+ 278,
+ 298,
+ 148,
+ 71,
+ 111,
+ 63,
+ 237,
+ 112,
+ 86,
+ 254,
+ 145,
+ 42,
+ 60,
+ 234,
+ 267,
+ 128,
+ 123,
+ 124
+ ],
+ [
+ 278,
+ 239,
+ 298,
+ 102,
+ 38,
+ 259,
+ 182,
+ 86,
+ 118,
+ 131,
+ 254,
+ 111,
+ 233,
+ 148,
+ 5,
+ 286,
+ 91,
+ 60,
+ 237,
+ 13
+ ],
+ [
+ 255,
+ 132,
+ 273,
+ 157,
+ 119,
+ 87,
+ 0,
+ 149,
+ 26,
+ 227,
+ 113,
+ 152,
+ 292,
+ 166,
+ 92,
+ 282,
+ 261,
+ 174,
+ 181,
+ 208
+ ],
+ [
+ 263,
+ 221,
+ 268,
+ 212,
+ 217,
+ 133,
+ 129,
+ 54,
+ 236,
+ 281,
+ 37,
+ 197,
+ 9,
+ 162,
+ 138,
+ 77,
+ 285,
+ 291,
+ 277,
+ 235
+ ],
+ [
+ 22,
+ 15,
+ 118,
+ 239,
+ 105,
+ 5,
+ 123,
+ 298,
+ 148,
+ 63,
+ 38,
+ 233,
+ 39,
+ 86,
+ 286,
+ 74,
+ 213,
+ 234,
+ 4,
+ 131
+ ],
+ [
+ 125,
+ 8,
+ 262,
+ 78,
+ 169,
+ 68,
+ 274,
+ 80,
+ 3,
+ 141,
+ 161,
+ 106,
+ 79,
+ 66,
+ 193,
+ 6,
+ 72,
+ 295,
+ 143,
+ 96
+ ],
+ [
+ 154,
+ 61,
+ 199,
+ 226,
+ 127,
+ 45,
+ 275,
+ 135,
+ 36,
+ 23,
+ 207,
+ 19,
+ 32,
+ 58,
+ 240,
+ 49,
+ 95,
+ 20,
+ 183,
+ 65
+ ],
+ [
+ 135,
+ 127,
+ 199,
+ 207,
+ 95,
+ 61,
+ 58,
+ 154,
+ 116,
+ 23,
+ 32,
+ 36,
+ 226,
+ 44,
+ 53,
+ 240,
+ 275,
+ 136,
+ 19,
+ 20
+ ],
+ [
+ 115,
+ 14,
+ 67,
+ 198,
+ 130,
+ 48,
+ 75,
+ 3,
+ 290,
+ 204,
+ 295,
+ 262,
+ 16,
+ 161,
+ 231,
+ 68,
+ 1,
+ 272,
+ 193,
+ 80
+ ],
+ [
+ 229,
+ 51,
+ 191,
+ 28,
+ 85,
+ 217,
+ 218,
+ 279,
+ 84,
+ 223,
+ 162,
+ 235,
+ 211,
+ 257,
+ 31,
+ 64,
+ 281,
+ 146,
+ 34,
+ 116
+ ],
+ [
+ 3,
+ 204,
+ 46,
+ 8,
+ 262,
+ 68,
+ 161,
+ 169,
+ 6,
+ 115,
+ 1,
+ 14,
+ 110,
+ 295,
+ 79,
+ 16,
+ 78,
+ 272,
+ 43,
+ 106
+ ],
+ [
+ 23,
+ 154,
+ 61,
+ 20,
+ 240,
+ 58,
+ 127,
+ 199,
+ 32,
+ 136,
+ 275,
+ 53,
+ 36,
+ 177,
+ 135,
+ 7,
+ 226,
+ 183,
+ 95,
+ 207
+ ],
+ [
+ 232,
+ 35,
+ 180,
+ 248,
+ 192,
+ 11,
+ 186,
+ 284,
+ 70,
+ 266,
+ 33,
+ 30,
+ 139,
+ 280,
+ 196,
+ 108,
+ 69,
+ 287,
+ 171,
+ 297
+ ],
+ [
+ 84,
+ 265,
+ 224,
+ 257,
+ 229,
+ 64,
+ 47,
+ 191,
+ 85,
+ 31,
+ 146,
+ 244,
+ 218,
+ 100,
+ 264,
+ 82,
+ 200,
+ 28,
+ 159,
+ 116
+ ],
+ [
+ 256,
+ 99,
+ 194,
+ 167,
+ 215,
+ 90,
+ 260,
+ 265,
+ 243,
+ 268,
+ 120,
+ 263,
+ 114,
+ 236,
+ 291,
+ 296,
+ 220,
+ 25,
+ 24,
+ 41
+ ],
+ [
+ 58,
+ 36,
+ 136,
+ 127,
+ 95,
+ 7,
+ 207,
+ 23,
+ 135,
+ 32,
+ 61,
+ 199,
+ 45,
+ 240,
+ 154,
+ 246,
+ 20,
+ 264,
+ 49,
+ 100
+ ],
+ [
+ 77,
+ 133,
+ 236,
+ 129,
+ 221,
+ 263,
+ 138,
+ 268,
+ 9,
+ 197,
+ 212,
+ 176,
+ 41,
+ 37,
+ 34,
+ 281,
+ 291,
+ 203,
+ 189,
+ 101
+ ],
+ [
+ 13,
+ 121,
+ 234,
+ 253,
+ 145,
+ 86,
+ 88,
+ 182,
+ 17,
+ 278,
+ 102,
+ 128,
+ 160,
+ 112,
+ 89,
+ 74,
+ 38,
+ 233,
+ 267,
+ 131
+ ],
+ [
+ 271,
+ 97,
+ 29,
+ 122,
+ 288,
+ 201,
+ 209,
+ 93,
+ 159,
+ 82,
+ 200,
+ 224,
+ 173,
+ 100,
+ 265,
+ 191,
+ 244,
+ 127,
+ 36,
+ 146
+ ],
+ [
+ 60,
+ 160,
+ 267,
+ 128,
+ 112,
+ 124,
+ 278,
+ 253,
+ 111,
+ 237,
+ 145,
+ 137,
+ 109,
+ 38,
+ 254,
+ 13,
+ 102,
+ 206,
+ 286,
+ 148
+ ],
+ [
+ 135,
+ 127,
+ 61,
+ 36,
+ 207,
+ 154,
+ 23,
+ 32,
+ 53,
+ 240,
+ 95,
+ 45,
+ 199,
+ 136,
+ 65,
+ 49,
+ 275,
+ 226,
+ 246,
+ 44
+ ],
+ [
+ 142,
+ 250,
+ 261,
+ 113,
+ 87,
+ 157,
+ 106,
+ 104,
+ 96,
+ 92,
+ 80,
+ 227,
+ 181,
+ 282,
+ 149,
+ 0,
+ 152,
+ 178,
+ 3,
+ 255
+ ],
+ [
+ 109,
+ 254,
+ 137,
+ 13,
+ 278,
+ 145,
+ 267,
+ 5,
+ 128,
+ 206,
+ 111,
+ 27,
+ 131,
+ 160,
+ 102,
+ 237,
+ 182,
+ 185,
+ 253,
+ 89
+ ],
+ [
+ 127,
+ 154,
+ 23,
+ 58,
+ 226,
+ 135,
+ 207,
+ 36,
+ 240,
+ 44,
+ 32,
+ 199,
+ 45,
+ 275,
+ 20,
+ 49,
+ 136,
+ 53,
+ 95,
+ 19
+ ],
+ [
+ 185,
+ 131,
+ 91,
+ 239,
+ 18,
+ 124,
+ 259,
+ 123,
+ 298,
+ 137,
+ 286,
+ 160,
+ 278,
+ 109,
+ 60,
+ 102,
+ 254,
+ 74,
+ 182,
+ 39
+ ],
+ [
+ 298,
+ 148,
+ 5,
+ 74,
+ 286,
+ 60,
+ 91,
+ 145,
+ 38,
+ 42,
+ 22,
+ 254,
+ 278,
+ 185,
+ 239,
+ 105,
+ 62,
+ 15,
+ 131,
+ 118
+ ],
+ [
+ 85,
+ 114,
+ 229,
+ 146,
+ 28,
+ 51,
+ 218,
+ 279,
+ 162,
+ 47,
+ 84,
+ 189,
+ 217,
+ 25,
+ 281,
+ 235,
+ 34,
+ 156,
+ 116,
+ 270
+ ],
+ [
+ 120,
+ 135,
+ 32,
+ 19,
+ 58,
+ 199,
+ 20,
+ 136,
+ 207,
+ 36,
+ 23,
+ 226,
+ 240,
+ 44,
+ 275,
+ 116,
+ 61,
+ 95,
+ 53,
+ 24
+ ],
+ [
+ 141,
+ 68,
+ 183,
+ 8,
+ 188,
+ 96,
+ 72,
+ 169,
+ 3,
+ 43,
+ 295,
+ 125,
+ 16,
+ 193,
+ 147,
+ 73,
+ 246,
+ 204,
+ 78,
+ 161
+ ],
+ [
+ 290,
+ 46,
+ 80,
+ 142,
+ 115,
+ 130,
+ 272,
+ 295,
+ 252,
+ 161,
+ 158,
+ 274,
+ 193,
+ 214,
+ 262,
+ 72,
+ 3,
+ 68,
+ 204,
+ 166
+ ],
+ [
+ 66,
+ 188,
+ 72,
+ 169,
+ 8,
+ 16,
+ 141,
+ 3,
+ 43,
+ 96,
+ 295,
+ 73,
+ 204,
+ 272,
+ 78,
+ 125,
+ 106,
+ 193,
+ 1,
+ 274
+ ],
+ [
+ 108,
+ 245,
+ 171,
+ 297,
+ 33,
+ 280,
+ 266,
+ 11,
+ 70,
+ 196,
+ 139,
+ 35,
+ 50,
+ 192,
+ 180,
+ 30,
+ 186,
+ 232,
+ 284,
+ 287
+ ],
+ [
+ 11,
+ 35,
+ 108,
+ 245,
+ 33,
+ 280,
+ 232,
+ 266,
+ 297,
+ 50,
+ 69,
+ 171,
+ 192,
+ 180,
+ 248,
+ 186,
+ 139,
+ 30,
+ 284,
+ 196
+ ],
+ [
+ 89,
+ 86,
+ 278,
+ 112,
+ 39,
+ 145,
+ 88,
+ 102,
+ 38,
+ 253,
+ 237,
+ 134,
+ 234,
+ 254,
+ 160,
+ 286,
+ 137,
+ 259,
+ 267,
+ 13
+ ],
+ [
+ 68,
+ 272,
+ 8,
+ 141,
+ 66,
+ 188,
+ 169,
+ 43,
+ 106,
+ 295,
+ 80,
+ 274,
+ 204,
+ 78,
+ 125,
+ 96,
+ 110,
+ 3,
+ 73,
+ 142
+ ],
+ [
+ 188,
+ 16,
+ 141,
+ 68,
+ 43,
+ 72,
+ 66,
+ 3,
+ 78,
+ 8,
+ 272,
+ 204,
+ 193,
+ 115,
+ 1,
+ 274,
+ 262,
+ 169,
+ 80,
+ 76
+ ],
+ [
+ 234,
+ 91,
+ 86,
+ 131,
+ 102,
+ 17,
+ 278,
+ 62,
+ 253,
+ 105,
+ 39,
+ 148,
+ 145,
+ 259,
+ 182,
+ 298,
+ 118,
+ 18,
+ 185,
+ 55
+ ],
+ [
+ 198,
+ 14,
+ 231,
+ 115,
+ 214,
+ 46,
+ 6,
+ 204,
+ 76,
+ 295,
+ 190,
+ 1,
+ 3,
+ 143,
+ 195,
+ 67,
+ 169,
+ 269,
+ 125,
+ 8
+ ],
+ [
+ 204,
+ 3,
+ 295,
+ 125,
+ 193,
+ 1,
+ 110,
+ 96,
+ 188,
+ 6,
+ 43,
+ 143,
+ 161,
+ 8,
+ 104,
+ 169,
+ 106,
+ 290,
+ 79,
+ 142
+ ],
+ [
+ 101,
+ 197,
+ 203,
+ 129,
+ 212,
+ 277,
+ 37,
+ 223,
+ 221,
+ 236,
+ 133,
+ 211,
+ 138,
+ 2,
+ 54,
+ 291,
+ 285,
+ 263,
+ 217,
+ 9
+ ],
+ [
+ 274,
+ 79,
+ 8,
+ 43,
+ 161,
+ 6,
+ 262,
+ 125,
+ 272,
+ 110,
+ 68,
+ 106,
+ 169,
+ 295,
+ 80,
+ 72,
+ 250,
+ 193,
+ 3,
+ 1
+ ],
+ [
+ 262,
+ 161,
+ 78,
+ 8,
+ 110,
+ 125,
+ 169,
+ 43,
+ 104,
+ 6,
+ 106,
+ 290,
+ 250,
+ 274,
+ 80,
+ 204,
+ 76,
+ 3,
+ 295,
+ 190
+ ],
+ [
+ 142,
+ 262,
+ 161,
+ 274,
+ 290,
+ 43,
+ 272,
+ 169,
+ 149,
+ 79,
+ 67,
+ 59,
+ 261,
+ 193,
+ 295,
+ 8,
+ 3,
+ 72,
+ 141,
+ 113
+ ],
+ [
+ 0,
+ 178,
+ 292,
+ 92,
+ 294,
+ 151,
+ 283,
+ 174,
+ 157,
+ 227,
+ 10,
+ 208,
+ 181,
+ 165,
+ 103,
+ 140,
+ 119,
+ 153,
+ 255,
+ 251
+ ],
+ [
+ 159,
+ 93,
+ 56,
+ 97,
+ 29,
+ 122,
+ 271,
+ 209,
+ 201,
+ 288,
+ 265,
+ 200,
+ 224,
+ 191,
+ 244,
+ 100,
+ 146,
+ 47,
+ 84,
+ 51
+ ],
+ [
+ 230,
+ 63,
+ 298,
+ 175,
+ 286,
+ 5,
+ 148,
+ 60,
+ 254,
+ 109,
+ 239,
+ 137,
+ 124,
+ 111,
+ 267,
+ 123,
+ 57,
+ 185,
+ 131,
+ 62
+ ],
+ [
+ 51,
+ 218,
+ 31,
+ 257,
+ 229,
+ 100,
+ 224,
+ 191,
+ 244,
+ 146,
+ 47,
+ 116,
+ 85,
+ 265,
+ 205,
+ 64,
+ 264,
+ 28,
+ 246,
+ 200
+ ],
+ [
+ 229,
+ 51,
+ 257,
+ 31,
+ 84,
+ 47,
+ 64,
+ 191,
+ 28,
+ 218,
+ 146,
+ 116,
+ 205,
+ 279,
+ 217,
+ 281,
+ 12,
+ 162,
+ 130,
+ 270
+ ],
+ [
+ 234,
+ 71,
+ 91,
+ 278,
+ 4,
+ 39,
+ 102,
+ 145,
+ 74,
+ 38,
+ 89,
+ 88,
+ 124,
+ 118,
+ 62,
+ 55,
+ 182,
+ 253,
+ 233,
+ 112
+ ],
+ [
+ 178,
+ 283,
+ 94,
+ 292,
+ 157,
+ 258,
+ 153,
+ 261,
+ 255,
+ 152,
+ 81,
+ 140,
+ 181,
+ 104,
+ 119,
+ 174,
+ 227,
+ 165,
+ 92,
+ 294
+ ],
+ [
+ 253,
+ 278,
+ 121,
+ 102,
+ 182,
+ 89,
+ 145,
+ 71,
+ 267,
+ 233,
+ 112,
+ 160,
+ 234,
+ 163,
+ 86,
+ 55,
+ 124,
+ 128,
+ 185,
+ 13
+ ],
+ [
+ 71,
+ 267,
+ 234,
+ 160,
+ 145,
+ 259,
+ 112,
+ 131,
+ 102,
+ 278,
+ 254,
+ 13,
+ 253,
+ 137,
+ 88,
+ 60,
+ 86,
+ 17,
+ 206,
+ 39
+ ],
+ [
+ 194,
+ 291,
+ 215,
+ 296,
+ 243,
+ 236,
+ 263,
+ 9,
+ 260,
+ 268,
+ 277,
+ 197,
+ 34,
+ 203,
+ 37,
+ 54,
+ 133,
+ 212,
+ 276,
+ 189
+ ],
+ [
+ 185,
+ 278,
+ 123,
+ 86,
+ 239,
+ 74,
+ 62,
+ 39,
+ 102,
+ 4,
+ 267,
+ 124,
+ 286,
+ 71,
+ 38,
+ 298,
+ 234,
+ 145,
+ 253,
+ 131
+ ],
+ [
+ 178,
+ 294,
+ 81,
+ 157,
+ 119,
+ 152,
+ 174,
+ 227,
+ 26,
+ 10,
+ 261,
+ 208,
+ 87,
+ 165,
+ 283,
+ 132,
+ 140,
+ 94,
+ 255,
+ 292
+ ],
+ [
+ 209,
+ 159,
+ 82,
+ 271,
+ 97,
+ 122,
+ 56,
+ 288,
+ 200,
+ 29,
+ 201,
+ 224,
+ 265,
+ 100,
+ 191,
+ 244,
+ 84,
+ 127,
+ 146,
+ 47
+ ],
+ [
+ 178,
+ 258,
+ 283,
+ 152,
+ 92,
+ 87,
+ 26,
+ 119,
+ 190,
+ 282,
+ 113,
+ 157,
+ 238,
+ 294,
+ 165,
+ 153,
+ 104,
+ 250,
+ 227,
+ 208
+ ],
+ [
+ 32,
+ 135,
+ 45,
+ 7,
+ 127,
+ 58,
+ 199,
+ 36,
+ 136,
+ 154,
+ 53,
+ 240,
+ 61,
+ 207,
+ 23,
+ 116,
+ 19,
+ 44,
+ 275,
+ 246
+ ],
+ [
+ 3,
+ 125,
+ 106,
+ 169,
+ 66,
+ 295,
+ 68,
+ 76,
+ 43,
+ 188,
+ 1,
+ 8,
+ 193,
+ 110,
+ 204,
+ 59,
+ 143,
+ 142,
+ 72,
+ 161
+ ],
+ [
+ 271,
+ 201,
+ 29,
+ 93,
+ 209,
+ 288,
+ 56,
+ 159,
+ 122,
+ 82,
+ 200,
+ 258,
+ 283,
+ 190,
+ 282,
+ 224,
+ 152,
+ 94,
+ 113,
+ 100
+ ],
+ [
+ 293,
+ 172,
+ 208,
+ 251,
+ 195,
+ 10,
+ 92,
+ 294,
+ 269,
+ 227,
+ 190,
+ 261,
+ 178,
+ 103,
+ 119,
+ 155,
+ 87,
+ 238,
+ 165,
+ 282
+ ],
+ [
+ 194,
+ 167,
+ 90,
+ 215,
+ 243,
+ 276,
+ 268,
+ 236,
+ 296,
+ 260,
+ 263,
+ 25,
+ 277,
+ 9,
+ 129,
+ 211,
+ 279,
+ 138,
+ 212,
+ 133
+ ],
+ [
+ 84,
+ 244,
+ 224,
+ 246,
+ 265,
+ 218,
+ 200,
+ 264,
+ 93,
+ 7,
+ 51,
+ 159,
+ 36,
+ 23,
+ 97,
+ 82,
+ 127,
+ 271,
+ 288,
+ 209
+ ],
+ [
+ 77,
+ 235,
+ 197,
+ 221,
+ 277,
+ 189,
+ 37,
+ 211,
+ 212,
+ 129,
+ 223,
+ 291,
+ 217,
+ 133,
+ 2,
+ 203,
+ 179,
+ 285,
+ 279,
+ 9
+ ],
+ [
+ 278,
+ 259,
+ 234,
+ 131,
+ 182,
+ 206,
+ 13,
+ 254,
+ 267,
+ 39,
+ 253,
+ 89,
+ 160,
+ 74,
+ 137,
+ 145,
+ 91,
+ 233,
+ 17,
+ 60
+ ],
+ [
+ 151,
+ 10,
+ 251,
+ 252,
+ 81,
+ 255,
+ 242,
+ 294,
+ 216,
+ 208,
+ 293,
+ 132,
+ 181,
+ 238,
+ 92,
+ 119,
+ 165,
+ 261,
+ 227,
+ 152
+ ],
+ [
+ 106,
+ 87,
+ 161,
+ 94,
+ 110,
+ 79,
+ 178,
+ 250,
+ 190,
+ 261,
+ 174,
+ 290,
+ 292,
+ 92,
+ 258,
+ 187,
+ 26,
+ 59,
+ 142,
+ 152
+ ],
+ [
+ 118,
+ 42,
+ 22,
+ 233,
+ 15,
+ 74,
+ 239,
+ 38,
+ 298,
+ 39,
+ 63,
+ 5,
+ 86,
+ 123,
+ 148,
+ 278,
+ 168,
+ 91,
+ 182,
+ 4
+ ],
+ [
+ 161,
+ 125,
+ 104,
+ 262,
+ 8,
+ 43,
+ 250,
+ 96,
+ 110,
+ 272,
+ 3,
+ 113,
+ 78,
+ 72,
+ 59,
+ 142,
+ 79,
+ 149,
+ 80,
+ 6
+ ],
+ [
+ 241,
+ 164,
+ 129,
+ 211,
+ 221,
+ 34,
+ 197,
+ 268,
+ 279,
+ 37,
+ 2,
+ 223,
+ 156,
+ 235,
+ 162,
+ 203,
+ 189,
+ 217,
+ 243,
+ 25
+ ],
+ [
+ 69,
+ 245,
+ 297,
+ 171,
+ 33,
+ 280,
+ 266,
+ 70,
+ 139,
+ 11,
+ 196,
+ 35,
+ 192,
+ 50,
+ 180,
+ 232,
+ 186,
+ 30,
+ 248,
+ 284
+ ],
+ [
+ 60,
+ 137,
+ 267,
+ 182,
+ 128,
+ 278,
+ 131,
+ 254,
+ 145,
+ 13,
+ 102,
+ 124,
+ 62,
+ 160,
+ 5,
+ 253,
+ 39,
+ 185,
+ 206,
+ 111
+ ],
+ [
+ 8,
+ 161,
+ 295,
+ 76,
+ 125,
+ 6,
+ 250,
+ 169,
+ 3,
+ 79,
+ 204,
+ 106,
+ 78,
+ 274,
+ 80,
+ 272,
+ 59,
+ 72,
+ 142,
+ 188
+ ],
+ [
+ 13,
+ 128,
+ 254,
+ 60,
+ 278,
+ 206,
+ 27,
+ 160,
+ 298,
+ 148,
+ 5,
+ 63,
+ 38,
+ 112,
+ 137,
+ 145,
+ 237,
+ 286,
+ 109,
+ 17
+ ],
+ [
+ 160,
+ 71,
+ 145,
+ 89,
+ 13,
+ 88,
+ 253,
+ 55,
+ 278,
+ 102,
+ 206,
+ 184,
+ 128,
+ 234,
+ 17,
+ 121,
+ 267,
+ 86,
+ 150,
+ 182
+ ],
+ [
+ 152,
+ 149,
+ 132,
+ 190,
+ 26,
+ 258,
+ 119,
+ 92,
+ 157,
+ 87,
+ 178,
+ 261,
+ 294,
+ 250,
+ 59,
+ 94,
+ 282,
+ 174,
+ 227,
+ 208
+ ],
+ [
+ 156,
+ 12,
+ 64,
+ 25,
+ 144,
+ 279,
+ 236,
+ 276,
+ 189,
+ 99,
+ 215,
+ 167,
+ 243,
+ 162,
+ 146,
+ 164,
+ 211,
+ 138,
+ 270,
+ 218
+ ],
+ [
+ 46,
+ 14,
+ 75,
+ 67,
+ 198,
+ 204,
+ 130,
+ 231,
+ 3,
+ 262,
+ 80,
+ 1,
+ 161,
+ 295,
+ 169,
+ 290,
+ 48,
+ 68,
+ 79,
+ 272
+ ],
+ [
+ 31,
+ 84,
+ 218,
+ 24,
+ 95,
+ 45,
+ 135,
+ 229,
+ 51,
+ 32,
+ 120,
+ 65,
+ 58,
+ 257,
+ 207,
+ 191,
+ 205,
+ 199,
+ 270,
+ 146
+ ],
+ [
+ 145,
+ 17,
+ 86,
+ 71,
+ 254,
+ 89,
+ 134,
+ 74,
+ 118,
+ 177,
+ 168,
+ 4,
+ 18,
+ 247,
+ 39,
+ 105,
+ 137,
+ 148,
+ 289,
+ 42
+ ],
+ [
+ 15,
+ 42,
+ 105,
+ 239,
+ 39,
+ 22,
+ 233,
+ 86,
+ 91,
+ 123,
+ 148,
+ 38,
+ 185,
+ 182,
+ 298,
+ 168,
+ 88,
+ 278,
+ 5,
+ 124
+ ],
+ [
+ 157,
+ 155,
+ 208,
+ 26,
+ 178,
+ 255,
+ 227,
+ 282,
+ 92,
+ 0,
+ 132,
+ 174,
+ 294,
+ 87,
+ 261,
+ 283,
+ 165,
+ 10,
+ 152,
+ 81
+ ],
+ [
+ 65,
+ 24,
+ 19,
+ 199,
+ 116,
+ 220,
+ 32,
+ 135,
+ 240,
+ 20,
+ 95,
+ 45,
+ 44,
+ 226,
+ 136,
+ 61,
+ 36,
+ 58,
+ 275,
+ 207
+ ],
+ [
+ 102,
+ 88,
+ 182,
+ 55,
+ 253,
+ 278,
+ 234,
+ 145,
+ 86,
+ 233,
+ 160,
+ 267,
+ 91,
+ 89,
+ 39,
+ 185,
+ 259,
+ 60,
+ 71,
+ 118
+ ],
+ [
+ 271,
+ 56,
+ 201,
+ 209,
+ 97,
+ 29,
+ 159,
+ 93,
+ 288,
+ 82,
+ 200,
+ 224,
+ 191,
+ 100,
+ 265,
+ 146,
+ 84,
+ 244,
+ 47,
+ 173
+ ],
+ [
+ 239,
+ 15,
+ 91,
+ 259,
+ 185,
+ 131,
+ 62,
+ 298,
+ 39,
+ 278,
+ 18,
+ 118,
+ 42,
+ 22,
+ 254,
+ 124,
+ 148,
+ 5,
+ 102,
+ 233
+ ],
+ [
+ 286,
+ 62,
+ 267,
+ 278,
+ 137,
+ 109,
+ 185,
+ 91,
+ 60,
+ 131,
+ 234,
+ 253,
+ 18,
+ 145,
+ 102,
+ 182,
+ 112,
+ 128,
+ 298,
+ 239
+ ],
+ [
+ 43,
+ 76,
+ 6,
+ 96,
+ 106,
+ 3,
+ 8,
+ 295,
+ 193,
+ 78,
+ 204,
+ 79,
+ 110,
+ 169,
+ 188,
+ 161,
+ 1,
+ 143,
+ 262,
+ 68
+ ],
+ [
+ 163,
+ 213,
+ 247,
+ 88,
+ 168,
+ 4,
+ 160,
+ 118,
+ 175,
+ 185,
+ 39,
+ 281,
+ 48,
+ 259,
+ 5,
+ 298,
+ 131,
+ 71,
+ 42,
+ 105
+ ],
+ [
+ 226,
+ 61,
+ 45,
+ 23,
+ 154,
+ 135,
+ 58,
+ 44,
+ 36,
+ 207,
+ 275,
+ 199,
+ 32,
+ 49,
+ 95,
+ 240,
+ 53,
+ 20,
+ 19,
+ 65
+ ],
+ [
+ 13,
+ 27,
+ 5,
+ 111,
+ 60,
+ 254,
+ 278,
+ 109,
+ 267,
+ 160,
+ 206,
+ 137,
+ 102,
+ 145,
+ 182,
+ 299,
+ 230,
+ 148,
+ 63,
+ 38
+ ],
+ [
+ 37,
+ 221,
+ 203,
+ 197,
+ 138,
+ 212,
+ 77,
+ 211,
+ 223,
+ 2,
+ 236,
+ 133,
+ 235,
+ 9,
+ 277,
+ 162,
+ 217,
+ 263,
+ 291,
+ 215
+ ],
+ [
+ 270,
+ 218,
+ 84,
+ 46,
+ 116,
+ 205,
+ 31,
+ 229,
+ 219,
+ 67,
+ 51,
+ 12,
+ 115,
+ 16,
+ 95,
+ 68,
+ 66,
+ 169,
+ 141,
+ 48
+ ],
+ [
+ 259,
+ 62,
+ 234,
+ 102,
+ 18,
+ 17,
+ 239,
+ 137,
+ 13,
+ 39,
+ 278,
+ 60,
+ 109,
+ 74,
+ 253,
+ 123,
+ 182,
+ 5,
+ 160,
+ 145
+ ],
+ [
+ 157,
+ 174,
+ 238,
+ 119,
+ 10,
+ 227,
+ 255,
+ 26,
+ 261,
+ 294,
+ 208,
+ 0,
+ 165,
+ 222,
+ 251,
+ 282,
+ 92,
+ 178,
+ 87,
+ 152
+ ],
+ [
+ 37,
+ 212,
+ 9,
+ 221,
+ 197,
+ 263,
+ 236,
+ 129,
+ 211,
+ 138,
+ 268,
+ 223,
+ 291,
+ 277,
+ 2,
+ 203,
+ 77,
+ 176,
+ 296,
+ 54
+ ],
+ [
+ 71,
+ 89,
+ 86,
+ 253,
+ 88,
+ 145,
+ 247,
+ 121,
+ 102,
+ 55,
+ 278,
+ 150,
+ 233,
+ 163,
+ 289,
+ 259,
+ 39,
+ 168,
+ 118,
+ 254
+ ],
+ [
+ 207,
+ 36,
+ 58,
+ 127,
+ 61,
+ 45,
+ 23,
+ 154,
+ 32,
+ 65,
+ 199,
+ 95,
+ 226,
+ 44,
+ 240,
+ 275,
+ 20,
+ 53,
+ 19,
+ 136
+ ],
+ [
+ 240,
+ 199,
+ 7,
+ 36,
+ 53,
+ 20,
+ 95,
+ 61,
+ 58,
+ 23,
+ 135,
+ 154,
+ 65,
+ 32,
+ 207,
+ 127,
+ 49,
+ 183,
+ 275,
+ 45
+ ],
+ [
+ 60,
+ 109,
+ 278,
+ 254,
+ 267,
+ 131,
+ 62,
+ 102,
+ 128,
+ 286,
+ 89,
+ 145,
+ 124,
+ 206,
+ 13,
+ 160,
+ 185,
+ 39,
+ 111,
+ 259
+ ],
+ [
+ 9,
+ 37,
+ 203,
+ 77,
+ 129,
+ 212,
+ 133,
+ 236,
+ 197,
+ 2,
+ 211,
+ 221,
+ 34,
+ 54,
+ 291,
+ 223,
+ 263,
+ 179,
+ 285,
+ 276
+ ],
+ [
+ 35,
+ 232,
+ 108,
+ 196,
+ 171,
+ 297,
+ 33,
+ 186,
+ 50,
+ 11,
+ 69,
+ 192,
+ 245,
+ 280,
+ 70,
+ 30,
+ 266,
+ 248,
+ 287,
+ 180
+ ],
+ [
+ 165,
+ 283,
+ 81,
+ 157,
+ 0,
+ 92,
+ 178,
+ 119,
+ 208,
+ 227,
+ 195,
+ 174,
+ 294,
+ 10,
+ 87,
+ 238,
+ 132,
+ 255,
+ 251,
+ 21
+ ],
+ [
+ 72,
+ 66,
+ 68,
+ 43,
+ 8,
+ 73,
+ 295,
+ 78,
+ 3,
+ 188,
+ 274,
+ 183,
+ 96,
+ 80,
+ 142,
+ 125,
+ 169,
+ 193,
+ 59,
+ 272
+ ],
+ [
+ 59,
+ 80,
+ 261,
+ 290,
+ 250,
+ 252,
+ 149,
+ 87,
+ 104,
+ 106,
+ 166,
+ 174,
+ 295,
+ 3,
+ 181,
+ 92,
+ 0,
+ 272,
+ 96,
+ 227
+ ],
+ [
+ 3,
+ 204,
+ 43,
+ 125,
+ 104,
+ 106,
+ 6,
+ 76,
+ 1,
+ 8,
+ 96,
+ 262,
+ 161,
+ 142,
+ 79,
+ 188,
+ 295,
+ 110,
+ 59,
+ 78
+ ],
+ [
+ 215,
+ 90,
+ 167,
+ 114,
+ 99,
+ 296,
+ 279,
+ 236,
+ 194,
+ 9,
+ 276,
+ 138,
+ 291,
+ 2,
+ 277,
+ 25,
+ 211,
+ 162,
+ 133,
+ 37
+ ],
+ [
+ 253,
+ 278,
+ 112,
+ 254,
+ 60,
+ 234,
+ 13,
+ 160,
+ 89,
+ 109,
+ 63,
+ 86,
+ 102,
+ 17,
+ 88,
+ 148,
+ 71,
+ 267,
+ 185,
+ 55
+ ],
+ [
+ 229,
+ 218,
+ 191,
+ 84,
+ 31,
+ 85,
+ 51,
+ 64,
+ 279,
+ 217,
+ 47,
+ 116,
+ 24,
+ 114,
+ 257,
+ 162,
+ 265,
+ 28,
+ 164,
+ 241
+ ],
+ [
+ 66,
+ 183,
+ 188,
+ 8,
+ 68,
+ 43,
+ 169,
+ 16,
+ 262,
+ 141,
+ 100,
+ 73,
+ 49,
+ 6,
+ 136,
+ 72,
+ 106,
+ 246,
+ 7,
+ 78
+ ],
+ [
+ 63,
+ 5,
+ 254,
+ 39,
+ 168,
+ 298,
+ 38,
+ 239,
+ 118,
+ 74,
+ 278,
+ 42,
+ 145,
+ 131,
+ 22,
+ 112,
+ 60,
+ 102,
+ 86,
+ 286
+ ],
+ [
+ 113,
+ 272,
+ 166,
+ 80,
+ 132,
+ 142,
+ 274,
+ 261,
+ 250,
+ 87,
+ 59,
+ 193,
+ 161,
+ 92,
+ 290,
+ 106,
+ 190,
+ 157,
+ 152,
+ 119
+ ],
+ [
+ 86,
+ 112,
+ 88,
+ 237,
+ 71,
+ 57,
+ 124,
+ 234,
+ 267,
+ 89,
+ 253,
+ 38,
+ 160,
+ 102,
+ 27,
+ 74,
+ 145,
+ 278,
+ 175,
+ 121
+ ],
+ [
+ 216,
+ 103,
+ 251,
+ 255,
+ 10,
+ 222,
+ 81,
+ 208,
+ 261,
+ 181,
+ 132,
+ 174,
+ 294,
+ 26,
+ 165,
+ 242,
+ 152,
+ 227,
+ 119,
+ 238
+ ],
+ [
+ 178,
+ 92,
+ 255,
+ 294,
+ 261,
+ 119,
+ 87,
+ 227,
+ 165,
+ 113,
+ 238,
+ 132,
+ 94,
+ 26,
+ 174,
+ 282,
+ 292,
+ 208,
+ 258,
+ 181
+ ],
+ [
+ 181,
+ 292,
+ 0,
+ 81,
+ 242,
+ 249,
+ 87,
+ 216,
+ 283,
+ 261,
+ 94,
+ 92,
+ 252,
+ 227,
+ 273,
+ 166,
+ 103,
+ 10,
+ 187,
+ 152
+ ],
+ [
+ 61,
+ 240,
+ 127,
+ 58,
+ 199,
+ 23,
+ 135,
+ 44,
+ 36,
+ 45,
+ 32,
+ 20,
+ 275,
+ 49,
+ 207,
+ 95,
+ 7,
+ 136,
+ 53,
+ 226
+ ],
+ [
+ 119,
+ 178,
+ 227,
+ 208,
+ 157,
+ 282,
+ 255,
+ 165,
+ 0,
+ 92,
+ 283,
+ 87,
+ 294,
+ 174,
+ 258,
+ 152,
+ 140,
+ 26,
+ 132,
+ 10
+ ],
+ [
+ 203,
+ 25,
+ 189,
+ 2,
+ 129,
+ 162,
+ 211,
+ 34,
+ 268,
+ 276,
+ 279,
+ 235,
+ 223,
+ 243,
+ 107,
+ 281,
+ 217,
+ 164,
+ 12,
+ 241
+ ],
+ [
+ 119,
+ 174,
+ 0,
+ 132,
+ 282,
+ 26,
+ 255,
+ 227,
+ 92,
+ 155,
+ 87,
+ 178,
+ 283,
+ 261,
+ 208,
+ 81,
+ 140,
+ 294,
+ 258,
+ 152
+ ],
+ [
+ 210,
+ 252,
+ 67,
+ 181,
+ 219,
+ 0,
+ 153,
+ 132,
+ 292,
+ 142,
+ 157,
+ 174,
+ 273,
+ 242,
+ 187,
+ 103,
+ 119,
+ 227,
+ 261,
+ 214
+ ],
+ [
+ 209,
+ 93,
+ 97,
+ 56,
+ 201,
+ 122,
+ 29,
+ 82,
+ 271,
+ 288,
+ 200,
+ 224,
+ 265,
+ 191,
+ 100,
+ 244,
+ 84,
+ 47,
+ 146,
+ 127
+ ],
+ [
+ 13,
+ 112,
+ 206,
+ 89,
+ 145,
+ 128,
+ 278,
+ 267,
+ 253,
+ 60,
+ 102,
+ 17,
+ 109,
+ 111,
+ 88,
+ 137,
+ 237,
+ 57,
+ 254,
+ 234
+ ],
+ [
+ 262,
+ 79,
+ 80,
+ 169,
+ 290,
+ 193,
+ 8,
+ 110,
+ 272,
+ 274,
+ 106,
+ 78,
+ 3,
+ 295,
+ 43,
+ 6,
+ 125,
+ 149,
+ 1,
+ 142
+ ],
+ [
+ 221,
+ 212,
+ 37,
+ 9,
+ 129,
+ 34,
+ 197,
+ 203,
+ 217,
+ 296,
+ 235,
+ 211,
+ 25,
+ 41,
+ 189,
+ 236,
+ 291,
+ 263,
+ 223,
+ 279
+ ],
+ [
+ 71,
+ 86,
+ 168,
+ 88,
+ 233,
+ 247,
+ 134,
+ 39,
+ 89,
+ 259,
+ 118,
+ 234,
+ 121,
+ 150,
+ 102,
+ 4,
+ 42,
+ 145,
+ 148,
+ 239
+ ],
+ [
+ 241,
+ 107,
+ 268,
+ 34,
+ 211,
+ 197,
+ 2,
+ 296,
+ 133,
+ 260,
+ 221,
+ 162,
+ 215,
+ 37,
+ 176,
+ 129,
+ 236,
+ 279,
+ 189,
+ 223
+ ],
+ [
+ 200,
+ 92,
+ 178,
+ 152,
+ 140,
+ 208,
+ 119,
+ 155,
+ 283,
+ 294,
+ 10,
+ 227,
+ 26,
+ 255,
+ 238,
+ 258,
+ 195,
+ 94,
+ 190,
+ 174
+ ],
+ [
+ 149,
+ 181,
+ 216,
+ 273,
+ 252,
+ 0,
+ 59,
+ 261,
+ 142,
+ 292,
+ 272,
+ 80,
+ 151,
+ 174,
+ 92,
+ 103,
+ 113,
+ 26,
+ 222,
+ 208
+ ],
+ [
+ 215,
+ 296,
+ 162,
+ 25,
+ 9,
+ 133,
+ 194,
+ 34,
+ 176,
+ 221,
+ 263,
+ 138,
+ 90,
+ 236,
+ 37,
+ 99,
+ 276,
+ 277,
+ 291,
+ 129
+ ],
+ [
+ 163,
+ 148,
+ 118,
+ 38,
+ 182,
+ 233,
+ 105,
+ 88,
+ 121,
+ 86,
+ 253,
+ 39,
+ 234,
+ 239,
+ 15,
+ 102,
+ 55,
+ 60,
+ 145,
+ 131
+ ],
+ [
+ 8,
+ 68,
+ 161,
+ 79,
+ 66,
+ 295,
+ 43,
+ 262,
+ 96,
+ 125,
+ 188,
+ 110,
+ 3,
+ 6,
+ 80,
+ 193,
+ 72,
+ 274,
+ 1,
+ 76
+ ],
+ [
+ 225,
+ 50,
+ 30,
+ 180,
+ 284,
+ 232,
+ 35,
+ 248,
+ 280,
+ 192,
+ 139,
+ 266,
+ 287,
+ 196,
+ 186,
+ 70,
+ 108,
+ 33,
+ 11,
+ 245
+ ],
+ [
+ 69,
+ 33,
+ 245,
+ 108,
+ 266,
+ 280,
+ 297,
+ 70,
+ 11,
+ 196,
+ 139,
+ 35,
+ 180,
+ 50,
+ 192,
+ 30,
+ 186,
+ 232,
+ 284,
+ 287
+ ],
+ [
+ 98,
+ 293,
+ 195,
+ 208,
+ 165,
+ 92,
+ 269,
+ 251,
+ 227,
+ 10,
+ 190,
+ 81,
+ 103,
+ 174,
+ 261,
+ 119,
+ 238,
+ 140,
+ 59,
+ 152
+ ],
+ [
+ 224,
+ 56,
+ 207,
+ 264,
+ 244,
+ 265,
+ 122,
+ 100,
+ 271,
+ 159,
+ 97,
+ 205,
+ 201,
+ 82,
+ 209,
+ 288,
+ 36,
+ 29,
+ 246,
+ 84
+ ],
+ [
+ 157,
+ 165,
+ 261,
+ 181,
+ 132,
+ 10,
+ 208,
+ 255,
+ 227,
+ 81,
+ 26,
+ 119,
+ 294,
+ 222,
+ 178,
+ 283,
+ 238,
+ 140,
+ 92,
+ 152
+ ],
+ [
+ 230,
+ 298,
+ 15,
+ 123,
+ 62,
+ 239,
+ 5,
+ 185,
+ 83,
+ 42,
+ 22,
+ 18,
+ 63,
+ 91,
+ 131,
+ 118,
+ 124,
+ 286,
+ 254,
+ 137
+ ],
+ [
+ 281,
+ 221,
+ 77,
+ 101,
+ 235,
+ 203,
+ 133,
+ 129,
+ 54,
+ 223,
+ 138,
+ 34,
+ 197,
+ 37,
+ 189,
+ 211,
+ 212,
+ 162,
+ 2,
+ 285
+ ],
+ [
+ 160,
+ 13,
+ 206,
+ 121,
+ 111,
+ 86,
+ 17,
+ 233,
+ 71,
+ 89,
+ 237,
+ 124,
+ 150,
+ 117,
+ 134,
+ 168,
+ 39,
+ 118,
+ 247,
+ 289
+ ],
+ [
+ 155,
+ 92,
+ 119,
+ 282,
+ 152,
+ 255,
+ 294,
+ 94,
+ 87,
+ 81,
+ 258,
+ 157,
+ 292,
+ 0,
+ 208,
+ 227,
+ 283,
+ 26,
+ 165,
+ 174
+ ],
+ [
+ 101,
+ 138,
+ 211,
+ 129,
+ 203,
+ 34,
+ 221,
+ 37,
+ 197,
+ 296,
+ 162,
+ 133,
+ 189,
+ 212,
+ 41,
+ 2,
+ 235,
+ 277,
+ 291,
+ 217
+ ],
+ [
+ 50,
+ 192,
+ 35,
+ 266,
+ 284,
+ 232,
+ 11,
+ 248,
+ 70,
+ 280,
+ 33,
+ 108,
+ 297,
+ 69,
+ 171,
+ 30,
+ 186,
+ 245,
+ 139,
+ 196
+ ],
+ [
+ 0,
+ 294,
+ 255,
+ 292,
+ 174,
+ 261,
+ 208,
+ 216,
+ 282,
+ 178,
+ 81,
+ 87,
+ 151,
+ 152,
+ 157,
+ 273,
+ 227,
+ 222,
+ 153,
+ 132
+ ],
+ [
+ 109,
+ 102,
+ 253,
+ 278,
+ 145,
+ 88,
+ 60,
+ 39,
+ 62,
+ 160,
+ 121,
+ 131,
+ 234,
+ 13,
+ 128,
+ 267,
+ 185,
+ 112,
+ 233,
+ 86
+ ],
+ [
+ 141,
+ 66,
+ 275,
+ 7,
+ 49,
+ 154,
+ 136,
+ 240,
+ 20,
+ 199,
+ 23,
+ 44,
+ 246,
+ 58,
+ 188,
+ 147,
+ 36,
+ 53,
+ 68,
+ 135
+ ],
+ [
+ 299,
+ 17,
+ 160,
+ 234,
+ 206,
+ 112,
+ 13,
+ 89,
+ 102,
+ 259,
+ 267,
+ 131,
+ 253,
+ 111,
+ 74,
+ 55,
+ 237,
+ 145,
+ 137,
+ 278
+ ],
+ [
+ 91,
+ 62,
+ 278,
+ 254,
+ 123,
+ 60,
+ 145,
+ 239,
+ 131,
+ 102,
+ 267,
+ 182,
+ 124,
+ 253,
+ 286,
+ 137,
+ 18,
+ 259,
+ 234,
+ 88
+ ],
+ [
+ 232,
+ 248,
+ 196,
+ 50,
+ 35,
+ 30,
+ 11,
+ 266,
+ 287,
+ 284,
+ 33,
+ 70,
+ 139,
+ 280,
+ 180,
+ 245,
+ 171,
+ 108,
+ 192,
+ 69
+ ],
+ [
+ 292,
+ 208,
+ 87,
+ 261,
+ 26,
+ 104,
+ 81,
+ 153,
+ 227,
+ 165,
+ 174,
+ 0,
+ 178,
+ 92,
+ 119,
+ 255,
+ 157,
+ 94,
+ 10,
+ 80
+ ],
+ [
+ 16,
+ 68,
+ 66,
+ 73,
+ 204,
+ 169,
+ 3,
+ 125,
+ 76,
+ 96,
+ 72,
+ 43,
+ 141,
+ 8,
+ 183,
+ 246,
+ 295,
+ 1,
+ 193,
+ 143
+ ],
+ [
+ 101,
+ 212,
+ 133,
+ 203,
+ 37,
+ 2,
+ 211,
+ 197,
+ 236,
+ 235,
+ 221,
+ 9,
+ 129,
+ 138,
+ 277,
+ 162,
+ 268,
+ 176,
+ 34,
+ 54
+ ],
+ [
+ 258,
+ 26,
+ 92,
+ 119,
+ 113,
+ 178,
+ 282,
+ 157,
+ 132,
+ 294,
+ 208,
+ 94,
+ 152,
+ 149,
+ 227,
+ 250,
+ 174,
+ 10,
+ 98,
+ 87
+ ],
+ [
+ 47,
+ 51,
+ 229,
+ 146,
+ 257,
+ 84,
+ 218,
+ 31,
+ 85,
+ 116,
+ 28,
+ 217,
+ 279,
+ 64,
+ 24,
+ 244,
+ 205,
+ 219,
+ 241,
+ 130
+ ],
+ [
+ 35,
+ 180,
+ 50,
+ 11,
+ 232,
+ 70,
+ 280,
+ 33,
+ 284,
+ 266,
+ 248,
+ 108,
+ 297,
+ 139,
+ 69,
+ 245,
+ 186,
+ 171,
+ 30,
+ 196
+ ],
+ [
+ 3,
+ 1,
+ 161,
+ 125,
+ 76,
+ 204,
+ 295,
+ 272,
+ 290,
+ 43,
+ 169,
+ 106,
+ 96,
+ 8,
+ 80,
+ 72,
+ 6,
+ 142,
+ 68,
+ 149
+ ],
+ [
+ 90,
+ 260,
+ 215,
+ 291,
+ 167,
+ 99,
+ 263,
+ 268,
+ 277,
+ 236,
+ 296,
+ 189,
+ 197,
+ 9,
+ 243,
+ 276,
+ 101,
+ 129,
+ 212,
+ 54
+ ],
+ [
+ 98,
+ 227,
+ 293,
+ 269,
+ 140,
+ 172,
+ 178,
+ 92,
+ 251,
+ 208,
+ 283,
+ 21,
+ 165,
+ 190,
+ 152,
+ 238,
+ 282,
+ 119,
+ 87,
+ 103
+ ],
+ [
+ 186,
+ 30,
+ 287,
+ 232,
+ 139,
+ 248,
+ 171,
+ 69,
+ 35,
+ 284,
+ 108,
+ 280,
+ 33,
+ 266,
+ 245,
+ 50,
+ 11,
+ 297,
+ 70,
+ 180
+ ],
+ [
+ 211,
+ 212,
+ 37,
+ 223,
+ 133,
+ 291,
+ 235,
+ 277,
+ 77,
+ 263,
+ 9,
+ 34,
+ 221,
+ 129,
+ 101,
+ 203,
+ 217,
+ 268,
+ 236,
+ 2
+ ],
+ [
+ 75,
+ 14,
+ 231,
+ 115,
+ 46,
+ 214,
+ 204,
+ 295,
+ 76,
+ 67,
+ 6,
+ 48,
+ 1,
+ 3,
+ 169,
+ 68,
+ 79,
+ 188,
+ 130,
+ 190
+ ],
+ [
+ 240,
+ 32,
+ 135,
+ 20,
+ 154,
+ 95,
+ 45,
+ 136,
+ 61,
+ 44,
+ 127,
+ 7,
+ 19,
+ 36,
+ 58,
+ 275,
+ 23,
+ 53,
+ 65,
+ 120
+ ],
+ [
+ 93,
+ 288,
+ 122,
+ 209,
+ 159,
+ 97,
+ 271,
+ 224,
+ 56,
+ 201,
+ 82,
+ 29,
+ 265,
+ 84,
+ 100,
+ 127,
+ 244,
+ 36,
+ 51,
+ 47
+ ],
+ [
+ 29,
+ 97,
+ 209,
+ 271,
+ 56,
+ 122,
+ 93,
+ 159,
+ 288,
+ 82,
+ 200,
+ 224,
+ 191,
+ 100,
+ 146,
+ 47,
+ 265,
+ 84,
+ 244,
+ 258
+ ],
+ [
+ 247,
+ 228,
+ 144,
+ 114,
+ 86,
+ 230,
+ 71,
+ 62,
+ 237,
+ 39,
+ 286,
+ 176,
+ 137,
+ 167,
+ 279,
+ 298,
+ 296,
+ 124,
+ 175,
+ 2
+ ],
+ [
+ 129,
+ 77,
+ 212,
+ 2,
+ 37,
+ 211,
+ 197,
+ 223,
+ 138,
+ 133,
+ 189,
+ 235,
+ 34,
+ 25,
+ 9,
+ 162,
+ 221,
+ 291,
+ 296,
+ 277
+ ],
+ [
+ 3,
+ 76,
+ 188,
+ 125,
+ 295,
+ 143,
+ 110,
+ 1,
+ 8,
+ 96,
+ 68,
+ 6,
+ 72,
+ 193,
+ 169,
+ 43,
+ 79,
+ 161,
+ 142,
+ 48
+ ],
+ [
+ 257,
+ 31,
+ 84,
+ 218,
+ 229,
+ 85,
+ 116,
+ 219,
+ 45,
+ 173,
+ 264,
+ 51,
+ 224,
+ 7,
+ 265,
+ 100,
+ 244,
+ 207,
+ 130,
+ 58
+ ],
+ [
+ 160,
+ 13,
+ 299,
+ 111,
+ 112,
+ 128,
+ 60,
+ 254,
+ 17,
+ 137,
+ 145,
+ 102,
+ 89,
+ 267,
+ 278,
+ 184,
+ 57,
+ 253,
+ 27,
+ 55
+ ],
+ [
+ 36,
+ 58,
+ 127,
+ 53,
+ 61,
+ 135,
+ 173,
+ 246,
+ 45,
+ 264,
+ 95,
+ 23,
+ 265,
+ 32,
+ 224,
+ 244,
+ 100,
+ 154,
+ 136,
+ 240
+ ],
+ [
+ 119,
+ 227,
+ 174,
+ 155,
+ 282,
+ 222,
+ 10,
+ 294,
+ 0,
+ 255,
+ 165,
+ 181,
+ 157,
+ 26,
+ 132,
+ 178,
+ 238,
+ 81,
+ 216,
+ 251
+ ],
+ [
+ 271,
+ 97,
+ 288,
+ 56,
+ 29,
+ 201,
+ 93,
+ 82,
+ 159,
+ 122,
+ 40,
+ 258,
+ 200,
+ 255,
+ 282,
+ 119,
+ 149,
+ 283,
+ 190,
+ 26
+ ],
+ [
+ 214,
+ 87,
+ 187,
+ 272,
+ 149,
+ 72,
+ 227,
+ 40,
+ 274,
+ 181,
+ 252,
+ 80,
+ 290,
+ 142,
+ 110,
+ 293,
+ 261,
+ 82,
+ 158,
+ 152
+ ],
+ [
+ 197,
+ 223,
+ 9,
+ 2,
+ 212,
+ 189,
+ 37,
+ 129,
+ 235,
+ 77,
+ 203,
+ 138,
+ 221,
+ 133,
+ 236,
+ 277,
+ 34,
+ 101,
+ 162,
+ 276
+ ],
+ [
+ 37,
+ 133,
+ 197,
+ 9,
+ 217,
+ 263,
+ 291,
+ 77,
+ 277,
+ 221,
+ 211,
+ 235,
+ 203,
+ 236,
+ 129,
+ 162,
+ 268,
+ 138,
+ 285,
+ 2
+ ],
+ [
+ 22,
+ 42,
+ 126,
+ 5,
+ 239,
+ 118,
+ 148,
+ 15,
+ 298,
+ 39,
+ 123,
+ 175,
+ 105,
+ 163,
+ 63,
+ 131,
+ 259,
+ 38,
+ 74,
+ 233
+ ],
+ [
+ 190,
+ 269,
+ 198,
+ 142,
+ 75,
+ 195,
+ 80,
+ 293,
+ 172,
+ 272,
+ 98,
+ 115,
+ 161,
+ 274,
+ 290,
+ 59,
+ 295,
+ 187,
+ 169,
+ 110
+ ],
+ [
+ 296,
+ 167,
+ 263,
+ 291,
+ 268,
+ 90,
+ 260,
+ 9,
+ 133,
+ 37,
+ 194,
+ 236,
+ 277,
+ 129,
+ 212,
+ 197,
+ 2,
+ 138,
+ 41,
+ 279
+ ],
+ [
+ 151,
+ 181,
+ 273,
+ 103,
+ 0,
+ 10,
+ 261,
+ 255,
+ 208,
+ 292,
+ 166,
+ 81,
+ 251,
+ 119,
+ 222,
+ 294,
+ 26,
+ 174,
+ 153,
+ 157
+ ],
+ [
+ 212,
+ 277,
+ 263,
+ 197,
+ 37,
+ 221,
+ 235,
+ 129,
+ 223,
+ 41,
+ 77,
+ 162,
+ 285,
+ 101,
+ 268,
+ 279,
+ 133,
+ 9,
+ 189,
+ 211
+ ],
+ [
+ 84,
+ 257,
+ 31,
+ 229,
+ 51,
+ 116,
+ 146,
+ 191,
+ 205,
+ 47,
+ 244,
+ 64,
+ 100,
+ 219,
+ 85,
+ 265,
+ 224,
+ 270,
+ 24,
+ 28
+ ],
+ [
+ 218,
+ 205,
+ 257,
+ 229,
+ 84,
+ 31,
+ 85,
+ 130,
+ 16,
+ 67,
+ 158,
+ 173,
+ 224,
+ 290,
+ 51,
+ 146,
+ 47,
+ 264,
+ 214,
+ 46
+ ],
+ [
+ 120,
+ 20,
+ 24,
+ 226,
+ 275,
+ 199,
+ 65,
+ 44,
+ 240,
+ 19,
+ 32,
+ 7,
+ 136,
+ 135,
+ 183,
+ 188,
+ 246,
+ 36,
+ 265,
+ 244
+ ],
+ [
+ 133,
+ 235,
+ 197,
+ 129,
+ 162,
+ 101,
+ 212,
+ 37,
+ 211,
+ 9,
+ 77,
+ 223,
+ 263,
+ 281,
+ 217,
+ 34,
+ 236,
+ 2,
+ 41,
+ 176
+ ],
+ [
+ 294,
+ 208,
+ 174,
+ 26,
+ 151,
+ 119,
+ 132,
+ 10,
+ 178,
+ 238,
+ 251,
+ 227,
+ 282,
+ 255,
+ 157,
+ 181,
+ 92,
+ 165,
+ 216,
+ 261
+ ],
+ [
+ 197,
+ 211,
+ 235,
+ 212,
+ 129,
+ 77,
+ 37,
+ 221,
+ 285,
+ 203,
+ 277,
+ 133,
+ 217,
+ 263,
+ 2,
+ 101,
+ 9,
+ 138,
+ 34,
+ 236
+ ],
+ [
+ 265,
+ 84,
+ 200,
+ 159,
+ 100,
+ 51,
+ 82,
+ 93,
+ 244,
+ 97,
+ 122,
+ 288,
+ 127,
+ 56,
+ 209,
+ 36,
+ 201,
+ 29,
+ 246,
+ 264
+ ],
+ [
+ 170,
+ 280,
+ 266,
+ 180,
+ 171,
+ 33,
+ 70,
+ 108,
+ 54,
+ 50,
+ 30,
+ 11,
+ 281,
+ 211,
+ 232,
+ 77,
+ 223,
+ 133,
+ 268,
+ 248
+ ],
+ [
+ 61,
+ 127,
+ 45,
+ 44,
+ 23,
+ 207,
+ 32,
+ 135,
+ 36,
+ 275,
+ 58,
+ 199,
+ 154,
+ 49,
+ 20,
+ 240,
+ 95,
+ 19,
+ 65,
+ 53
+ ],
+ [
+ 208,
+ 119,
+ 87,
+ 157,
+ 0,
+ 174,
+ 92,
+ 165,
+ 132,
+ 178,
+ 255,
+ 283,
+ 282,
+ 273,
+ 261,
+ 152,
+ 81,
+ 10,
+ 294,
+ 251
+ ],
+ [
+ 167,
+ 296,
+ 179,
+ 176,
+ 34,
+ 25,
+ 162,
+ 203,
+ 133,
+ 2,
+ 276,
+ 90,
+ 9,
+ 279,
+ 202,
+ 221,
+ 281,
+ 54,
+ 189,
+ 235
+ ],
+ [
+ 85,
+ 51,
+ 31,
+ 84,
+ 257,
+ 47,
+ 218,
+ 191,
+ 116,
+ 146,
+ 64,
+ 28,
+ 205,
+ 12,
+ 219,
+ 279,
+ 217,
+ 130,
+ 45,
+ 270
+ ],
+ [
+ 83,
+ 175,
+ 38,
+ 298,
+ 39,
+ 128,
+ 137,
+ 286,
+ 5,
+ 278,
+ 62,
+ 60,
+ 111,
+ 267,
+ 237,
+ 63,
+ 91,
+ 148,
+ 124,
+ 109
+ ],
+ [
+ 198,
+ 75,
+ 14,
+ 115,
+ 46,
+ 214,
+ 48,
+ 204,
+ 262,
+ 6,
+ 79,
+ 295,
+ 8,
+ 78,
+ 67,
+ 76,
+ 143,
+ 269,
+ 3,
+ 68
+ ],
+ [
+ 50,
+ 35,
+ 186,
+ 248,
+ 284,
+ 192,
+ 180,
+ 11,
+ 196,
+ 139,
+ 70,
+ 30,
+ 266,
+ 287,
+ 280,
+ 33,
+ 108,
+ 245,
+ 69,
+ 297
+ ],
+ [
+ 102,
+ 182,
+ 88,
+ 39,
+ 118,
+ 105,
+ 234,
+ 86,
+ 278,
+ 15,
+ 239,
+ 267,
+ 259,
+ 121,
+ 42,
+ 38,
+ 71,
+ 123,
+ 89,
+ 253
+ ],
+ [
+ 74,
+ 86,
+ 102,
+ 131,
+ 253,
+ 145,
+ 278,
+ 17,
+ 89,
+ 39,
+ 55,
+ 267,
+ 13,
+ 259,
+ 182,
+ 38,
+ 124,
+ 121,
+ 91,
+ 254
+ ],
+ [
+ 197,
+ 221,
+ 211,
+ 101,
+ 223,
+ 37,
+ 212,
+ 129,
+ 2,
+ 285,
+ 189,
+ 34,
+ 203,
+ 277,
+ 217,
+ 9,
+ 281,
+ 162,
+ 138,
+ 176
+ ],
+ [
+ 133,
+ 212,
+ 9,
+ 189,
+ 268,
+ 138,
+ 77,
+ 263,
+ 37,
+ 129,
+ 221,
+ 54,
+ 197,
+ 291,
+ 277,
+ 2,
+ 211,
+ 41,
+ 285,
+ 217
+ ],
+ [
+ 27,
+ 278,
+ 60,
+ 39,
+ 13,
+ 160,
+ 71,
+ 38,
+ 137,
+ 112,
+ 111,
+ 89,
+ 150,
+ 109,
+ 145,
+ 298,
+ 206,
+ 286,
+ 102,
+ 57
+ ],
+ [
+ 132,
+ 251,
+ 152,
+ 10,
+ 178,
+ 294,
+ 92,
+ 208,
+ 174,
+ 119,
+ 222,
+ 227,
+ 26,
+ 165,
+ 155,
+ 140,
+ 282,
+ 157,
+ 255,
+ 94
+ ],
+ [
+ 123,
+ 39,
+ 298,
+ 15,
+ 118,
+ 38,
+ 91,
+ 42,
+ 131,
+ 259,
+ 148,
+ 185,
+ 22,
+ 62,
+ 5,
+ 286,
+ 254,
+ 63,
+ 18,
+ 182
+ ],
+ [
+ 154,
+ 61,
+ 20,
+ 32,
+ 58,
+ 199,
+ 127,
+ 36,
+ 135,
+ 136,
+ 7,
+ 95,
+ 23,
+ 53,
+ 207,
+ 49,
+ 246,
+ 19,
+ 275,
+ 65
+ ],
+ [
+ 164,
+ 107,
+ 34,
+ 268,
+ 211,
+ 2,
+ 221,
+ 133,
+ 176,
+ 37,
+ 236,
+ 129,
+ 279,
+ 235,
+ 189,
+ 162,
+ 223,
+ 215,
+ 277,
+ 217
+ ],
+ [
+ 249,
+ 252,
+ 10,
+ 153,
+ 251,
+ 151,
+ 103,
+ 81,
+ 181,
+ 294,
+ 261,
+ 292,
+ 216,
+ 165,
+ 140,
+ 92,
+ 174,
+ 152,
+ 227,
+ 0
+ ],
+ [
+ 90,
+ 179,
+ 99,
+ 9,
+ 296,
+ 268,
+ 236,
+ 291,
+ 133,
+ 260,
+ 276,
+ 194,
+ 212,
+ 211,
+ 2,
+ 263,
+ 54,
+ 37,
+ 189,
+ 34
+ ],
+ [
+ 224,
+ 265,
+ 84,
+ 100,
+ 207,
+ 218,
+ 173,
+ 36,
+ 51,
+ 191,
+ 82,
+ 246,
+ 159,
+ 31,
+ 135,
+ 58,
+ 24,
+ 200,
+ 56,
+ 264
+ ],
+ [
+ 69,
+ 108,
+ 171,
+ 280,
+ 297,
+ 33,
+ 266,
+ 70,
+ 11,
+ 35,
+ 196,
+ 139,
+ 192,
+ 50,
+ 232,
+ 180,
+ 186,
+ 30,
+ 284,
+ 287
+ ],
+ [
+ 36,
+ 7,
+ 207,
+ 264,
+ 100,
+ 53,
+ 127,
+ 84,
+ 58,
+ 244,
+ 95,
+ 224,
+ 173,
+ 136,
+ 61,
+ 265,
+ 240,
+ 200,
+ 288,
+ 32
+ ],
+ [
+ 86,
+ 39,
+ 121,
+ 259,
+ 118,
+ 74,
+ 233,
+ 134,
+ 168,
+ 71,
+ 289,
+ 239,
+ 148,
+ 89,
+ 163,
+ 123,
+ 254,
+ 105,
+ 42,
+ 298
+ ],
+ [
+ 232,
+ 284,
+ 186,
+ 50,
+ 30,
+ 180,
+ 35,
+ 196,
+ 70,
+ 11,
+ 287,
+ 192,
+ 266,
+ 139,
+ 280,
+ 33,
+ 108,
+ 245,
+ 171,
+ 69
+ ],
+ [
+ 242,
+ 153,
+ 261,
+ 292,
+ 252,
+ 181,
+ 81,
+ 216,
+ 151,
+ 152,
+ 0,
+ 10,
+ 251,
+ 87,
+ 227,
+ 165,
+ 187,
+ 255,
+ 103,
+ 273
+ ],
+ [
+ 110,
+ 190,
+ 113,
+ 94,
+ 92,
+ 178,
+ 104,
+ 282,
+ 258,
+ 294,
+ 152,
+ 161,
+ 59,
+ 106,
+ 79,
+ 78,
+ 208,
+ 26,
+ 157,
+ 21
+ ],
+ [
+ 10,
+ 151,
+ 238,
+ 222,
+ 208,
+ 132,
+ 294,
+ 242,
+ 227,
+ 165,
+ 103,
+ 216,
+ 252,
+ 174,
+ 81,
+ 152,
+ 26,
+ 261,
+ 98,
+ 119
+ ],
+ [
+ 181,
+ 242,
+ 142,
+ 261,
+ 292,
+ 0,
+ 103,
+ 273,
+ 166,
+ 10,
+ 227,
+ 132,
+ 174,
+ 59,
+ 251,
+ 151,
+ 153,
+ 87,
+ 216,
+ 92
+ ],
+ [
+ 145,
+ 234,
+ 88,
+ 278,
+ 102,
+ 131,
+ 182,
+ 112,
+ 267,
+ 160,
+ 55,
+ 121,
+ 89,
+ 60,
+ 124,
+ 17,
+ 13,
+ 74,
+ 109,
+ 71
+ ],
+ [
+ 278,
+ 60,
+ 145,
+ 185,
+ 137,
+ 102,
+ 182,
+ 39,
+ 131,
+ 109,
+ 267,
+ 148,
+ 239,
+ 234,
+ 259,
+ 89,
+ 62,
+ 111,
+ 206,
+ 112
+ ],
+ [
+ 119,
+ 178,
+ 157,
+ 181,
+ 261,
+ 152,
+ 282,
+ 174,
+ 151,
+ 132,
+ 292,
+ 0,
+ 208,
+ 87,
+ 227,
+ 294,
+ 26,
+ 165,
+ 92,
+ 81
+ ],
+ [
+ 52,
+ 265,
+ 224,
+ 99,
+ 154,
+ 240,
+ 20,
+ 61,
+ 146,
+ 244,
+ 64,
+ 84,
+ 218,
+ 120,
+ 32,
+ 114,
+ 58,
+ 173,
+ 23,
+ 31
+ ],
+ [
+ 31,
+ 229,
+ 51,
+ 84,
+ 205,
+ 85,
+ 218,
+ 191,
+ 47,
+ 116,
+ 146,
+ 244,
+ 64,
+ 264,
+ 224,
+ 219,
+ 265,
+ 100,
+ 45,
+ 173
+ ],
+ [
+ 87,
+ 178,
+ 94,
+ 282,
+ 283,
+ 157,
+ 181,
+ 119,
+ 294,
+ 255,
+ 104,
+ 292,
+ 174,
+ 81,
+ 208,
+ 152,
+ 92,
+ 140,
+ 227,
+ 261
+ ],
+ [
+ 131,
+ 102,
+ 89,
+ 39,
+ 62,
+ 234,
+ 123,
+ 239,
+ 17,
+ 278,
+ 13,
+ 254,
+ 60,
+ 137,
+ 160,
+ 91,
+ 182,
+ 74,
+ 253,
+ 71
+ ],
+ [
+ 263,
+ 268,
+ 291,
+ 194,
+ 9,
+ 277,
+ 197,
+ 296,
+ 236,
+ 41,
+ 54,
+ 212,
+ 133,
+ 217,
+ 34,
+ 129,
+ 101,
+ 221,
+ 189,
+ 37
+ ],
+ [
+ 174,
+ 152,
+ 87,
+ 165,
+ 255,
+ 157,
+ 132,
+ 10,
+ 208,
+ 92,
+ 181,
+ 119,
+ 227,
+ 178,
+ 294,
+ 292,
+ 59,
+ 151,
+ 216,
+ 26
+ ],
+ [
+ 161,
+ 79,
+ 8,
+ 43,
+ 80,
+ 290,
+ 169,
+ 78,
+ 274,
+ 106,
+ 125,
+ 6,
+ 272,
+ 67,
+ 3,
+ 193,
+ 68,
+ 295,
+ 110,
+ 46
+ ],
+ [
+ 268,
+ 291,
+ 197,
+ 212,
+ 277,
+ 217,
+ 9,
+ 133,
+ 236,
+ 37,
+ 221,
+ 129,
+ 77,
+ 41,
+ 260,
+ 296,
+ 235,
+ 34,
+ 54,
+ 138
+ ],
+ [
+ 207,
+ 246,
+ 84,
+ 257,
+ 53,
+ 31,
+ 100,
+ 36,
+ 173,
+ 95,
+ 224,
+ 45,
+ 58,
+ 265,
+ 61,
+ 244,
+ 51,
+ 136,
+ 127,
+ 85
+ ],
+ [
+ 224,
+ 244,
+ 51,
+ 82,
+ 84,
+ 159,
+ 100,
+ 93,
+ 200,
+ 207,
+ 173,
+ 288,
+ 191,
+ 56,
+ 36,
+ 264,
+ 209,
+ 127,
+ 64,
+ 246
+ ],
+ [
+ 280,
+ 171,
+ 245,
+ 69,
+ 33,
+ 108,
+ 11,
+ 70,
+ 297,
+ 30,
+ 180,
+ 50,
+ 186,
+ 35,
+ 196,
+ 232,
+ 192,
+ 248,
+ 139,
+ 284
+ ],
+ [
+ 278,
+ 60,
+ 128,
+ 102,
+ 109,
+ 137,
+ 89,
+ 182,
+ 88,
+ 124,
+ 253,
+ 160,
+ 131,
+ 234,
+ 254,
+ 13,
+ 145,
+ 91,
+ 185,
+ 112
+ ],
+ [
+ 263,
+ 133,
+ 212,
+ 236,
+ 197,
+ 9,
+ 291,
+ 37,
+ 277,
+ 221,
+ 41,
+ 260,
+ 217,
+ 129,
+ 296,
+ 54,
+ 34,
+ 77,
+ 211,
+ 189
+ ],
+ [
+ 195,
+ 190,
+ 98,
+ 283,
+ 293,
+ 172,
+ 94,
+ 238,
+ 250,
+ 165,
+ 140,
+ 178,
+ 21,
+ 92,
+ 214,
+ 227,
+ 152,
+ 251,
+ 208,
+ 258
+ ],
+ [
+ 273,
+ 152,
+ 292,
+ 165,
+ 187,
+ 151,
+ 81,
+ 294,
+ 251,
+ 242,
+ 10,
+ 103,
+ 261,
+ 140,
+ 255,
+ 181,
+ 216,
+ 92,
+ 252,
+ 227
+ ],
+ [
+ 288,
+ 56,
+ 97,
+ 209,
+ 29,
+ 201,
+ 93,
+ 122,
+ 82,
+ 159,
+ 200,
+ 40,
+ 258,
+ 157,
+ 282,
+ 255,
+ 132,
+ 113,
+ 119,
+ 190
+ ],
+ [
+ 149,
+ 274,
+ 161,
+ 193,
+ 80,
+ 72,
+ 8,
+ 290,
+ 166,
+ 295,
+ 169,
+ 67,
+ 106,
+ 142,
+ 110,
+ 46,
+ 48,
+ 141,
+ 79,
+ 250
+ ],
+ [
+ 227,
+ 0,
+ 181,
+ 165,
+ 208,
+ 216,
+ 292,
+ 255,
+ 10,
+ 119,
+ 261,
+ 151,
+ 132,
+ 251,
+ 294,
+ 222,
+ 81,
+ 174,
+ 157,
+ 87
+ ],
+ [
+ 78,
+ 80,
+ 43,
+ 272,
+ 161,
+ 262,
+ 169,
+ 68,
+ 8,
+ 79,
+ 149,
+ 295,
+ 290,
+ 67,
+ 141,
+ 193,
+ 72,
+ 110,
+ 125,
+ 190
+ ],
+ [
+ 44,
+ 226,
+ 154,
+ 199,
+ 23,
+ 61,
+ 20,
+ 49,
+ 183,
+ 127,
+ 240,
+ 58,
+ 32,
+ 135,
+ 19,
+ 65,
+ 220,
+ 95,
+ 36,
+ 136
+ ],
+ [
+ 138,
+ 129,
+ 217,
+ 236,
+ 279,
+ 41,
+ 291,
+ 211,
+ 162,
+ 212,
+ 37,
+ 277,
+ 263,
+ 9,
+ 203,
+ 34,
+ 25,
+ 268,
+ 133,
+ 221
+ ],
+ [
+ 197,
+ 263,
+ 77,
+ 101,
+ 217,
+ 291,
+ 212,
+ 37,
+ 129,
+ 279,
+ 9,
+ 223,
+ 268,
+ 133,
+ 221,
+ 236,
+ 211,
+ 41,
+ 203,
+ 34
+ ],
+ [
+ 298,
+ 39,
+ 145,
+ 102,
+ 38,
+ 13,
+ 71,
+ 60,
+ 267,
+ 182,
+ 91,
+ 253,
+ 286,
+ 237,
+ 148,
+ 111,
+ 112,
+ 234,
+ 137,
+ 128
+ ],
+ [
+ 277,
+ 129,
+ 77,
+ 2,
+ 223,
+ 101,
+ 197,
+ 189,
+ 236,
+ 34,
+ 203,
+ 217,
+ 37,
+ 211,
+ 291,
+ 221,
+ 263,
+ 162,
+ 212,
+ 25
+ ],
+ [
+ 266,
+ 245,
+ 33,
+ 108,
+ 69,
+ 171,
+ 297,
+ 70,
+ 11,
+ 192,
+ 35,
+ 196,
+ 30,
+ 50,
+ 180,
+ 232,
+ 139,
+ 186,
+ 284,
+ 287
+ ],
+ [
+ 176,
+ 221,
+ 235,
+ 223,
+ 138,
+ 212,
+ 197,
+ 54,
+ 217,
+ 37,
+ 263,
+ 129,
+ 211,
+ 189,
+ 162,
+ 101,
+ 203,
+ 2,
+ 41,
+ 34
+ ],
+ [
+ 157,
+ 178,
+ 119,
+ 208,
+ 255,
+ 258,
+ 294,
+ 181,
+ 227,
+ 132,
+ 174,
+ 222,
+ 26,
+ 92,
+ 87,
+ 10,
+ 238,
+ 283,
+ 81,
+ 152
+ ],
+ [
+ 94,
+ 87,
+ 0,
+ 165,
+ 178,
+ 227,
+ 157,
+ 140,
+ 81,
+ 152,
+ 153,
+ 119,
+ 294,
+ 92,
+ 174,
+ 155,
+ 258,
+ 292,
+ 10,
+ 282
+ ],
+ [
+ 248,
+ 30,
+ 232,
+ 180,
+ 50,
+ 186,
+ 196,
+ 35,
+ 192,
+ 287,
+ 11,
+ 70,
+ 266,
+ 139,
+ 280,
+ 33,
+ 108,
+ 69,
+ 245,
+ 171
+ ],
+ [
+ 223,
+ 235,
+ 77,
+ 37,
+ 197,
+ 212,
+ 129,
+ 217,
+ 138,
+ 211,
+ 203,
+ 101,
+ 2,
+ 277,
+ 34,
+ 236,
+ 221,
+ 9,
+ 263,
+ 133
+ ],
+ [
+ 298,
+ 124,
+ 278,
+ 38,
+ 62,
+ 137,
+ 63,
+ 91,
+ 185,
+ 239,
+ 145,
+ 60,
+ 148,
+ 112,
+ 254,
+ 267,
+ 111,
+ 71,
+ 109,
+ 74
+ ],
+ [
+ 186,
+ 196,
+ 50,
+ 248,
+ 30,
+ 232,
+ 266,
+ 35,
+ 139,
+ 284,
+ 11,
+ 33,
+ 180,
+ 171,
+ 70,
+ 69,
+ 280,
+ 108,
+ 192,
+ 245
+ ],
+ [
+ 56,
+ 271,
+ 97,
+ 122,
+ 93,
+ 29,
+ 200,
+ 159,
+ 209,
+ 201,
+ 82,
+ 224,
+ 127,
+ 265,
+ 36,
+ 100,
+ 84,
+ 244,
+ 53,
+ 246
+ ],
+ [
+ 247,
+ 86,
+ 134,
+ 278,
+ 74,
+ 71,
+ 150,
+ 17,
+ 121,
+ 89,
+ 88,
+ 168,
+ 39,
+ 259,
+ 105,
+ 148,
+ 254,
+ 137,
+ 118,
+ 4
+ ],
+ [
+ 80,
+ 161,
+ 142,
+ 67,
+ 262,
+ 79,
+ 272,
+ 274,
+ 149,
+ 193,
+ 169,
+ 46,
+ 59,
+ 72,
+ 106,
+ 104,
+ 295,
+ 43,
+ 110,
+ 76
+ ],
+ [
+ 197,
+ 263,
+ 212,
+ 9,
+ 277,
+ 37,
+ 77,
+ 133,
+ 129,
+ 296,
+ 101,
+ 268,
+ 236,
+ 203,
+ 211,
+ 189,
+ 34,
+ 221,
+ 217,
+ 260
+ ],
+ [
+ 181,
+ 81,
+ 0,
+ 178,
+ 87,
+ 255,
+ 261,
+ 153,
+ 152,
+ 273,
+ 174,
+ 104,
+ 92,
+ 227,
+ 294,
+ 216,
+ 165,
+ 157,
+ 283,
+ 94
+ ],
+ [
+ 98,
+ 172,
+ 10,
+ 208,
+ 251,
+ 103,
+ 195,
+ 269,
+ 81,
+ 294,
+ 227,
+ 119,
+ 151,
+ 92,
+ 181,
+ 261,
+ 273,
+ 216,
+ 165,
+ 178
+ ],
+ [
+ 10,
+ 92,
+ 181,
+ 178,
+ 208,
+ 222,
+ 152,
+ 174,
+ 81,
+ 119,
+ 132,
+ 282,
+ 165,
+ 238,
+ 157,
+ 251,
+ 261,
+ 283,
+ 227,
+ 26
+ ],
+ [
+ 76,
+ 3,
+ 8,
+ 204,
+ 110,
+ 125,
+ 169,
+ 96,
+ 161,
+ 142,
+ 68,
+ 1,
+ 193,
+ 80,
+ 6,
+ 272,
+ 67,
+ 72,
+ 188,
+ 290
+ ],
+ [
+ 9,
+ 291,
+ 215,
+ 263,
+ 34,
+ 133,
+ 268,
+ 37,
+ 260,
+ 162,
+ 25,
+ 212,
+ 138,
+ 197,
+ 203,
+ 2,
+ 129,
+ 221,
+ 277,
+ 236
+ ],
+ [
+ 108,
+ 69,
+ 245,
+ 171,
+ 33,
+ 280,
+ 70,
+ 266,
+ 11,
+ 139,
+ 35,
+ 196,
+ 192,
+ 50,
+ 180,
+ 232,
+ 186,
+ 30,
+ 284,
+ 287
+ ],
+ [
+ 5,
+ 239,
+ 63,
+ 62,
+ 286,
+ 39,
+ 278,
+ 22,
+ 18,
+ 148,
+ 131,
+ 123,
+ 42,
+ 74,
+ 15,
+ 185,
+ 38,
+ 124,
+ 118,
+ 105
+ ],
+ [
+ 206,
+ 13,
+ 184,
+ 160,
+ 89,
+ 128,
+ 112,
+ 267,
+ 17,
+ 60,
+ 111,
+ 254,
+ 137,
+ 145,
+ 237,
+ 27,
+ 234,
+ 57,
+ 278,
+ 102
+ ]
+ ],
+ "neighborAffinities": [
+ [
+ 0.8585054200504685,
+ 0.728740734787584,
+ 0.6448628469822687,
+ 0.5914735044997422,
+ 0.5893512474001957,
+ 0.589195706140552,
+ 0.5666235201756133,
+ 0.5504395316925504,
+ 0.5498082159822641,
+ 0.4926044408208189,
+ 0.48990341751961874,
+ 0.46524706217031386,
+ 0.4610108308369877,
+ 0.4464491166279154,
+ 0.4455376441974982,
+ 0.4316497274530715,
+ 0.4316191028713835,
+ 0.4114601498153632,
+ 0.4114468852887573,
+ 0.40712405700254245
+ ],
+ [
+ 0.8491142687365443,
+ 0.6527491499975082,
+ 0.6526546953311424,
+ 0.6440853190252295,
+ 0.6056771574149977,
+ 0.589805397017231,
+ 0.5841018262813032,
+ 0.5732986667359898,
+ 0.553727387660951,
+ 0.5484918253345066,
+ 0.5211318518425205,
+ 0.5191608019777764,
+ 0.49954240379030695,
+ 0.4633481546500255,
+ 0.4535410990252569,
+ 0.4524603535403843,
+ 0.42837691358410557,
+ 0.41847677709340547,
+ 0.4081659696017201,
+ 0.39988827862839604
+ ],
+ [
+ 0.8667822551785582,
+ 0.8295338975261666,
+ 0.6931689535861585,
+ 0.656649596139409,
+ 0.6548302746881149,
+ 0.6455824680431715,
+ 0.6401420209392469,
+ 0.6320512852155753,
+ 0.6290367259985894,
+ 0.6179983301239357,
+ 0.61760486403356,
+ 0.6145932318530178,
+ 0.6031939917765538,
+ 0.5976107591024248,
+ 0.5525484777181443,
+ 0.5215290607089367,
+ 0.5023438872902202,
+ 0.4710035121310877,
+ 0.45807560318606383,
+ 0.4432286769484357
+ ],
+ [
+ 0.7085755571039085,
+ 0.6989903523625024,
+ 0.6725052929628624,
+ 0.6538280898192302,
+ 0.6160956835576954,
+ 0.5989881106044067,
+ 0.5916847443230828,
+ 0.5883678559507433,
+ 0.49477060544582957,
+ 0.4921630010045731,
+ 0.48519576689625976,
+ 0.48453313009438137,
+ 0.48020956940465476,
+ 0.4765757152031307,
+ 0.4728804010484978,
+ 0.472789733890038,
+ 0.4627110545231746,
+ 0.4595157933924512,
+ 0.4402981499657511,
+ 0.378319305289259
+ ],
+ [
+ 0.8889420469640101,
+ 0.8414689797254136,
+ 0.7499468339410207,
+ 0.7329276349944779,
+ 0.662771320076108,
+ 0.6159352137704845,
+ 0.5978843793085702,
+ 0.5946680581852755,
+ 0.5897285094139166,
+ 0.5885555771999981,
+ 0.5609621224357264,
+ 0.538535864173161,
+ 0.5096533935450287,
+ 0.5083562373730391,
+ 0.5004344061856835,
+ 0.4902095762785219,
+ 0.489208421580773,
+ 0.476550501219828,
+ 0.4312490848986046,
+ 0.4303115934899937
+ ],
+ [
+ 0.8455676037723818,
+ 0.7546370424527122,
+ 0.7492359125266123,
+ 0.6528054475177861,
+ 0.6169159786037951,
+ 0.5786864114052936,
+ 0.5764238246512609,
+ 0.5383598219128884,
+ 0.5379515484422515,
+ 0.5338601730457064,
+ 0.5110677513883612,
+ 0.4851446601948936,
+ 0.4702032532797458,
+ 0.4531023666923971,
+ 0.4481477184684898,
+ 0.4397704394796945,
+ 0.43635756635040046,
+ 0.4333660519449985,
+ 0.4038153016173574,
+ 0.3938340133471287
+ ],
+ [
+ 0.8016161056444685,
+ 0.7169335615463166,
+ 0.7041804966786306,
+ 0.6731361930142389,
+ 0.6696886409124363,
+ 0.6508143643571419,
+ 0.6418488695520159,
+ 0.6359873962921452,
+ 0.6158436927624809,
+ 0.601741221624253,
+ 0.6013197676236454,
+ 0.5715921097563332,
+ 0.5538459351634055,
+ 0.5392547089417001,
+ 0.5119427677939992,
+ 0.4994377361067417,
+ 0.49574040713558964,
+ 0.4809467012338949,
+ 0.451055751603487,
+ 0.4069884813022618
+ ],
+ [
+ 0.8254552899826411,
+ 0.8053008525925457,
+ 0.7499390363415406,
+ 0.7266540718737655,
+ 0.7041102205276613,
+ 0.6726265811061886,
+ 0.6485379736626353,
+ 0.6127141286811237,
+ 0.6120126616683662,
+ 0.5770079490042874,
+ 0.565181562240625,
+ 0.5475554738972697,
+ 0.5461691240128556,
+ 0.5435564396489025,
+ 0.5346604514262806,
+ 0.4879618833527888,
+ 0.47030232144034434,
+ 0.4427886752242578,
+ 0.43684176959888227,
+ 0.4143958103064603
+ ],
+ [
+ 0.711380796699522,
+ 0.6507605034703356,
+ 0.6489616648008182,
+ 0.6197234450753419,
+ 0.6080452882743537,
+ 0.5981741337751629,
+ 0.5848434008371723,
+ 0.575743432699898,
+ 0.5696041621461931,
+ 0.5607902347631522,
+ 0.5558695900704071,
+ 0.5469455540461219,
+ 0.5294843620299655,
+ 0.5282580697200923,
+ 0.49874785558086665,
+ 0.45704504386773315,
+ 0.43557731385667114,
+ 0.42016968575466057,
+ 0.4138232803107065,
+ 0.39798330441903507
+ ],
+ [
+ 0.8809288388798084,
+ 0.8212112147947586,
+ 0.7917952829947388,
+ 0.7800951821467943,
+ 0.7559120514122752,
+ 0.6654333712864151,
+ 0.6574744806983404,
+ 0.6488126833554916,
+ 0.6376334354498582,
+ 0.6329236549885677,
+ 0.6239403969665591,
+ 0.5480129236686617,
+ 0.5317810453164983,
+ 0.5068781952882755,
+ 0.49442380173438055,
+ 0.46077265528123346,
+ 0.43651342599988485,
+ 0.43197578023506544,
+ 0.3793791539229231,
+ 0.3686905244153295
+ ],
+ [
+ 0.9956759140872848,
+ 0.7172090774005999,
+ 0.6390767746070529,
+ 0.5759577560878728,
+ 0.5701010874149988,
+ 0.5618251660314241,
+ 0.5515086128176165,
+ 0.548326940593417,
+ 0.5458247094958434,
+ 0.5281383231519674,
+ 0.5113516551804443,
+ 0.5065930621151575,
+ 0.5022960791982349,
+ 0.49937135969897917,
+ 0.4830130196827913,
+ 0.47415063523141415,
+ 0.46992062051567507,
+ 0.46119439705268694,
+ 0.4331493616867885,
+ 0.42962358094569164
+ ],
+ [
+ 0.9231218270035326,
+ 0.8506907166219038,
+ 0.7713992157772085,
+ 0.7290263894175852,
+ 0.7232128984854812,
+ 0.7040351771927225,
+ 0.6794217208271678,
+ 0.6693090713549078,
+ 0.652164409839999,
+ 0.643214603478959,
+ 0.634595664728375,
+ 0.6265875059409314,
+ 0.6251016979141487,
+ 0.6199176518223544,
+ 0.60397681951043,
+ 0.5612481114799764,
+ 0.5320708811952122,
+ 0.4978428962950371,
+ 0.4894906883919896,
+ 0.485917070550283
+ ],
+ [
+ 0.8525537636887908,
+ 0.6929719616542672,
+ 0.5836987276207894,
+ 0.5769593024410918,
+ 0.5745363915920502,
+ 0.5716556939912977,
+ 0.5664451453066616,
+ 0.5431012569587296,
+ 0.5272411336781077,
+ 0.5192560827878698,
+ 0.516358772725638,
+ 0.49297668423887264,
+ 0.46684568919516456,
+ 0.46356127786185963,
+ 0.4616379695119803,
+ 0.4578781223378733,
+ 0.4471960508437539,
+ 0.4469897600042505,
+ 0.43659550921324064,
+ 0.4349381381643864
+ ],
+ [
+ 0.740956799202794,
+ 0.7081527685540998,
+ 0.6984270117583435,
+ 0.6929894724268921,
+ 0.6411714633324535,
+ 0.611927137439461,
+ 0.6007801632035366,
+ 0.5515608368299496,
+ 0.542755888753642,
+ 0.5182918975451333,
+ 0.48345719350262994,
+ 0.4814628743319167,
+ 0.4501929989668121,
+ 0.44223835411393364,
+ 0.43749494555410406,
+ 0.4300967222371287,
+ 0.41079608778390353,
+ 0.38966440649982437,
+ 0.38937207044074346,
+ 0.381558079385845
+ ],
+ [
+ 0.9381683300258992,
+ 0.8642397316703949,
+ 0.8006432208096561,
+ 0.756150222480523,
+ 0.7015366533878296,
+ 0.5883224383429546,
+ 0.4873300088462587,
+ 0.43813522323693077,
+ 0.42205557924345205,
+ 0.4174780140933587,
+ 0.4068652337494755,
+ 0.3991185200082097,
+ 0.39519631676512756,
+ 0.39258748967418966,
+ 0.39166069863336794,
+ 0.38328530953111917,
+ 0.38195236223176243,
+ 0.3791798232016802,
+ 0.3784839260387426,
+ 0.372282577811697
+ ],
+ [
+ 0.8598919404336935,
+ 0.7666456002246385,
+ 0.7473901316712348,
+ 0.7311670192743001,
+ 0.6733589468513007,
+ 0.5440626705077932,
+ 0.5229379173670168,
+ 0.5083542469494761,
+ 0.499195252469899,
+ 0.4710599937168512,
+ 0.46236195981195033,
+ 0.4275007438027879,
+ 0.4245169156839175,
+ 0.42060962404482893,
+ 0.41340129386354246,
+ 0.4066884477088228,
+ 0.4034445292149391,
+ 0.39500793580324056,
+ 0.39465689086608546,
+ 0.39448943323963764
+ ],
+ [
+ 0.7562606329252815,
+ 0.7003418284757794,
+ 0.659666123196998,
+ 0.6214427493002341,
+ 0.6159485637027036,
+ 0.6001963212269713,
+ 0.5737338498653544,
+ 0.5394715236380229,
+ 0.5266866811043107,
+ 0.4995270765607342,
+ 0.4967038155405016,
+ 0.4884283218616937,
+ 0.4785791727515731,
+ 0.47719850271083114,
+ 0.47506623137794796,
+ 0.46041058145852026,
+ 0.42833313664779654,
+ 0.4230349454769366,
+ 0.4184858154079728,
+ 0.41792932454318643
+ ],
+ [
+ 0.7331738455541662,
+ 0.6974000181933457,
+ 0.6844464980731084,
+ 0.6797606442607859,
+ 0.6036741148269891,
+ 0.5877579664382697,
+ 0.5427408072287212,
+ 0.5258390335545464,
+ 0.5205203623089492,
+ 0.5009082893743843,
+ 0.4959124930682024,
+ 0.4950974039296976,
+ 0.4926167080450919,
+ 0.4802169773849881,
+ 0.47145905676984,
+ 0.45338304276528246,
+ 0.4391028889680123,
+ 0.4023849146271782,
+ 0.3894210245552766,
+ 0.3829848526722487
+ ],
+ [
+ 0.7926126860643213,
+ 0.68285911685964,
+ 0.6458177635840876,
+ 0.5448441894470599,
+ 0.5447450009579423,
+ 0.5389923073259101,
+ 0.5089674183856926,
+ 0.47312244360318434,
+ 0.44828308336828004,
+ 0.430040131021708,
+ 0.4204898223723354,
+ 0.41948395787831727,
+ 0.40524701963025755,
+ 0.40355691641060853,
+ 0.38720325748607154,
+ 0.3867055614562388,
+ 0.38300066257716936,
+ 0.37679672769839945,
+ 0.37486326298636297,
+ 0.3643831324685527
+ ],
+ [
+ 0.7715205654854052,
+ 0.7678111065619686,
+ 0.7116689577766617,
+ 0.7010329928960898,
+ 0.6348238119019285,
+ 0.6091913937631628,
+ 0.5964327511223997,
+ 0.5962750853958191,
+ 0.5719184740140046,
+ 0.5697663054096809,
+ 0.5621853378065749,
+ 0.559686049378993,
+ 0.5551685970359976,
+ 0.5366414446065204,
+ 0.5310051907720903,
+ 0.4766560316628796,
+ 0.46282637567359924,
+ 0.43708472080291705,
+ 0.432028720987949,
+ 0.4310939576713113
+ ],
+ [
+ 0.9591476286853233,
+ 0.7888681232100654,
+ 0.7319508887354197,
+ 0.6883360903924082,
+ 0.6371069421168749,
+ 0.6331277120489991,
+ 0.6260048565819962,
+ 0.5611131176915336,
+ 0.5544540981049548,
+ 0.5447929408416279,
+ 0.5377248315317578,
+ 0.5254376444527138,
+ 0.5059679056089679,
+ 0.4726005631982787,
+ 0.46727571019363223,
+ 0.46672506594691243,
+ 0.4647314169822506,
+ 0.41823629905893045,
+ 0.40949371275203383,
+ 0.3787834304283683
+ ],
+ [
+ 0.8267227724754045,
+ 0.7414136824720174,
+ 0.7154858761421973,
+ 0.7033964119593372,
+ 0.6891450140070233,
+ 0.6889318056093886,
+ 0.6753692496929408,
+ 0.6282241212346187,
+ 0.6088378838221868,
+ 0.5860544416069238,
+ 0.5824597492555346,
+ 0.5373360569365179,
+ 0.5288248864624007,
+ 0.4826605904572986,
+ 0.4744137990186741,
+ 0.47051524786936894,
+ 0.46859769925740735,
+ 0.44240921762299745,
+ 0.4419052402960068,
+ 0.441828830113844
+ ],
+ [
+ 1.0,
+ 0.6962848829502604,
+ 0.6318725984827936,
+ 0.6281890643698465,
+ 0.6255712132460914,
+ 0.5929121434109783,
+ 0.5928498360615153,
+ 0.5270956577539841,
+ 0.5262551879322906,
+ 0.5043240060428851,
+ 0.4393886447232441,
+ 0.42802253013084246,
+ 0.4280101803636289,
+ 0.42718184563098693,
+ 0.4208422839764263,
+ 0.40770861722797536,
+ 0.4070275577313342,
+ 0.39852763796977575,
+ 0.3956863461226422,
+ 0.38815495372365066
+ ],
+ [
+ 0.8583391916792918,
+ 0.8156184793861183,
+ 0.8048892587309152,
+ 0.7763030164128981,
+ 0.7227812201941798,
+ 0.6623471716785893,
+ 0.6284777238420184,
+ 0.6266362010781186,
+ 0.5945262309999032,
+ 0.5329870241158087,
+ 0.5312493287039521,
+ 0.5177359823378438,
+ 0.5096073326823197,
+ 0.5039742653346071,
+ 0.48419670187563,
+ 0.4817406739629653,
+ 0.44478469177707963,
+ 0.4119273444488839,
+ 0.40038294884684655,
+ 0.3372383746720845
+ ],
+ [
+ 0.9453499000932919,
+ 0.647707221528038,
+ 0.5682646367661963,
+ 0.5448424729774002,
+ 0.5200141277514134,
+ 0.5189640058990876,
+ 0.4965681625831576,
+ 0.4913549024508319,
+ 0.48649216256029243,
+ 0.47199119887658963,
+ 0.46116168602402524,
+ 0.4358081802017423,
+ 0.4172790511153247,
+ 0.41396455559165407,
+ 0.40757654571945884,
+ 0.406647254396842,
+ 0.4050116437260563,
+ 0.39669273882165523,
+ 0.3955137258620625,
+ 0.36827241823405155
+ ],
+ [
+ 0.8616970937841435,
+ 0.7924417911734175,
+ 0.772542505601395,
+ 0.7430075576442021,
+ 0.7204411344828954,
+ 0.7072852572183064,
+ 0.7019388976485266,
+ 0.696610851765211,
+ 0.6965698324221464,
+ 0.6849991157664491,
+ 0.6684177733948555,
+ 0.646813914480245,
+ 0.6440384182401149,
+ 0.6385808043546675,
+ 0.6347086929851398,
+ 0.6340996226589306,
+ 0.6213817240247328,
+ 0.6115186566186229,
+ 0.5920288433150138,
+ 0.5895329606689104
+ ],
+ [
+ 0.8057962367678833,
+ 0.7309721653169284,
+ 0.7307136403398768,
+ 0.6301649337963382,
+ 0.61441725067571,
+ 0.571580240156436,
+ 0.5546953203429078,
+ 0.5308499003103699,
+ 0.49599491596158307,
+ 0.46956540041190514,
+ 0.46402539432071194,
+ 0.44956150679181983,
+ 0.43883367088133074,
+ 0.43661206049953727,
+ 0.43212009078130903,
+ 0.4197734146965303,
+ 0.410338335753007,
+ 0.4010294080059167,
+ 0.38100250490865795,
+ 0.37856967401632485
+ ],
+ [
+ 0.7059423721365782,
+ 0.6385407493328512,
+ 0.6158119577088478,
+ 0.5791555893225255,
+ 0.5658002695974237,
+ 0.5435920673265638,
+ 0.5193379860272956,
+ 0.5012265821628354,
+ 0.46383221250297796,
+ 0.4557563025165801,
+ 0.4382059782705227,
+ 0.43523238151490085,
+ 0.4319474772978028,
+ 0.4291091552012554,
+ 0.42785500300317963,
+ 0.39990934961611085,
+ 0.3992231950468754,
+ 0.38758300599858214,
+ 0.3731096763970916,
+ 0.36023655702234636
+ ],
+ [
+ 0.989825020946027,
+ 0.8852175350495884,
+ 0.8296031836188289,
+ 0.7938986531512633,
+ 0.7688552800726509,
+ 0.7205924987193758,
+ 0.6813393192377667,
+ 0.6797421044183883,
+ 0.6785931637822842,
+ 0.6643308955631765,
+ 0.622555573672462,
+ 0.6038799390021252,
+ 0.6001791187471897,
+ 0.5868159821675706,
+ 0.538869151730047,
+ 0.5336945779268069,
+ 0.4757175750952588,
+ 0.37964041040913077,
+ 0.3781156388682088,
+ 0.36929056532012117
+ ],
+ [
+ 0.9845685503613684,
+ 0.9123758562911803,
+ 0.8940326569692179,
+ 0.8489632931917921,
+ 0.7911774698893105,
+ 0.7795803273593491,
+ 0.7065662430340162,
+ 0.699718748233486,
+ 0.6975805226059988,
+ 0.570744208069511,
+ 0.5295187738836125,
+ 0.3207937627776436,
+ 0.2873344699726347,
+ 0.2841328084443229,
+ 0.2654333724525344,
+ 0.2647114259356718,
+ 0.2592231786280294,
+ 0.2516801252777903,
+ 0.24416725199958556,
+ 0.24108244013032293
+ ],
+ [
+ 0.788368243597755,
+ 0.7866703651897912,
+ 0.7812156098903749,
+ 0.7727868180174619,
+ 0.7554199728170929,
+ 0.7051643291217742,
+ 0.69033191452253,
+ 0.6869670709320428,
+ 0.6513180506359241,
+ 0.6150565265701823,
+ 0.6104831026312942,
+ 0.5883406839792489,
+ 0.5817583722335432,
+ 0.5733012271772132,
+ 0.5647661673205845,
+ 0.5633433410437423,
+ 0.557338576536397,
+ 0.5468627678237176,
+ 0.5457323149887241,
+ 0.49033243889818867
+ ],
+ [
+ 0.9049698677988807,
+ 0.790405136743783,
+ 0.7899401107055966,
+ 0.7699426931217503,
+ 0.7213051380561587,
+ 0.6486806910959466,
+ 0.6457813012630325,
+ 0.6023318030652165,
+ 0.5011794011163242,
+ 0.47589820828897533,
+ 0.44573750894554376,
+ 0.41765348414045994,
+ 0.41380871039980194,
+ 0.40540240128753624,
+ 0.4036002656541552,
+ 0.40325489614020327,
+ 0.4016698498057649,
+ 0.40130017964956016,
+ 0.3974980153230934,
+ 0.393956930179282
+ ],
+ [
+ 0.8448241341605918,
+ 0.8148205190078738,
+ 0.7657137229569015,
+ 0.7574100638061415,
+ 0.7567774724691396,
+ 0.7522256165309272,
+ 0.7262557381015133,
+ 0.6508273548722864,
+ 0.6301423211736449,
+ 0.6204595116082813,
+ 0.5980759587950981,
+ 0.5867156623221238,
+ 0.584211517570213,
+ 0.5807013791045222,
+ 0.5449767110316017,
+ 0.4992276570576577,
+ 0.48562277550558197,
+ 0.4754708470097265,
+ 0.4726379257801066,
+ 0.4626727668451451
+ ],
+ [
+ 0.9656597534970136,
+ 0.926009577003239,
+ 0.9094929108696359,
+ 0.892687235711445,
+ 0.8788744011174925,
+ 0.8390819844146307,
+ 0.8010354705181579,
+ 0.7693459508141105,
+ 0.7465968626426001,
+ 0.7070063180968433,
+ 0.6447042165558952,
+ 0.6411122708208801,
+ 0.6203695773691296,
+ 0.6112146860553782,
+ 0.591309533649312,
+ 0.5761060479672635,
+ 0.5637245260398587,
+ 0.5563889659604193,
+ 0.4546550884017806,
+ 0.40661572043613037
+ ],
+ [
+ 0.7755722645421361,
+ 0.7596916847035917,
+ 0.7244552562767184,
+ 0.7199919700980174,
+ 0.6981445837618441,
+ 0.678273230599872,
+ 0.6710552904835408,
+ 0.6582064288068369,
+ 0.6497019087175888,
+ 0.6407818841535271,
+ 0.6300144372131404,
+ 0.6110528024791911,
+ 0.6068338008017429,
+ 0.6005656653000284,
+ 0.5691823742425809,
+ 0.563045841416174,
+ 0.5591067428730705,
+ 0.5586012351855809,
+ 0.5447328226499274,
+ 0.5289410278896709
+ ],
+ [
+ 0.9011469759835626,
+ 0.8869483187564253,
+ 0.8225585810353073,
+ 0.8186723389911013,
+ 0.7014389759914723,
+ 0.6939059431061374,
+ 0.6665936982022453,
+ 0.6602165364781758,
+ 0.6343319882819096,
+ 0.624287130743514,
+ 0.609367759678421,
+ 0.5931253292105527,
+ 0.5819195728204473,
+ 0.5809667774551794,
+ 0.5806606799139464,
+ 0.5604847188659234,
+ 0.554739752264555,
+ 0.5503976880074206,
+ 0.5380126811858177,
+ 0.5173231402108831
+ ],
+ [
+ 0.866845657569919,
+ 0.8470766564093926,
+ 0.8181409432832015,
+ 0.7661619746637791,
+ 0.6599575310302807,
+ 0.5481746587939818,
+ 0.542260225904818,
+ 0.5409211259894705,
+ 0.5106698564530641,
+ 0.49110865969381556,
+ 0.48504905754625915,
+ 0.48250480655274774,
+ 0.46028275345768466,
+ 0.42975722051764237,
+ 0.42941905252139323,
+ 0.40178924699429847,
+ 0.39062322634922253,
+ 0.35551153575194794,
+ 0.33207408531118254,
+ 0.29238235787184297
+ ],
+ [
+ 0.9647095158695926,
+ 0.9364437435693597,
+ 0.8181215392421982,
+ 0.7502190032220719,
+ 0.7362896348317233,
+ 0.6441954023884955,
+ 0.6417038764958758,
+ 0.6324004097575484,
+ 0.6277312071581205,
+ 0.6193553371940975,
+ 0.6191585177137777,
+ 0.6146397510728124,
+ 0.5996605167211899,
+ 0.5880546452901603,
+ 0.5765096635999016,
+ 0.5727323158051579,
+ 0.5695119862052926,
+ 0.5607424449838789,
+ 0.5439766096966301,
+ 0.5131615061898852
+ ],
+ [
+ 0.8200863579177116,
+ 0.8102976317828385,
+ 0.7820454938458614,
+ 0.7116128595266273,
+ 0.6181048937563937,
+ 0.5896749964771967,
+ 0.5682808285302418,
+ 0.5667692823580823,
+ 0.4882719618506132,
+ 0.4534432950091595,
+ 0.44416028314290323,
+ 0.442600653524337,
+ 0.4332207457580447,
+ 0.43137588668035576,
+ 0.4242666756634189,
+ 0.39824292794422844,
+ 0.39807733658953004,
+ 0.3874678326268799,
+ 0.3723258362482391,
+ 0.3684867212472222
+ ],
+ [
+ 0.7521601113476151,
+ 0.7046501575903708,
+ 0.6660325200510208,
+ 0.6572449367873366,
+ 0.6114866108153048,
+ 0.6054924491066042,
+ 0.6050558857466637,
+ 0.5555305890209423,
+ 0.5510599679537216,
+ 0.5372224326785904,
+ 0.5326473734382697,
+ 0.5298304657083265,
+ 0.5189808643895382,
+ 0.518801304202168,
+ 0.5086845374955158,
+ 0.5018323878071252,
+ 0.4999878230082325,
+ 0.44972661373932915,
+ 0.4475193555336903,
+ 0.4262293630098707
+ ],
+ [
+ 1.0,
+ 0.9904251822446815,
+ 0.8963746048641102,
+ 0.7963050220846345,
+ 0.7639368902297945,
+ 0.7521633888932792,
+ 0.6751948934487899,
+ 0.6426001570457202,
+ 0.6334815145041183,
+ 0.60854335672894,
+ 0.6081866718480906,
+ 0.48523620669762263,
+ 0.48211349284109917,
+ 0.47442550863047556,
+ 0.4729778584879733,
+ 0.4700524515704364,
+ 0.4613573457700625,
+ 0.4172506103493491,
+ 0.40505410170884826,
+ 0.36787944117144233
+ ],
+ [
+ 0.9097099422526898,
+ 0.8141045092130414,
+ 0.8090778715791768,
+ 0.7459569943386266,
+ 0.7387683987336233,
+ 0.728349772985645,
+ 0.7102035478033353,
+ 0.6860639536012267,
+ 0.6793220315313113,
+ 0.653076359874908,
+ 0.6514206124470392,
+ 0.5850536872107124,
+ 0.5808211054744572,
+ 0.5756412947135336,
+ 0.5462946845626988,
+ 0.539272521108139,
+ 0.5348375017588519,
+ 0.5341309055027971,
+ 0.5259235169586094,
+ 0.5198876761439508
+ ],
+ [
+ 0.9914406186038374,
+ 0.8716961151833262,
+ 0.7381888127299886,
+ 0.6987191977207066,
+ 0.6924203569596279,
+ 0.6084477343042259,
+ 0.5722341397183749,
+ 0.5719852999024905,
+ 0.5654769206333445,
+ 0.5419879805951198,
+ 0.523136690941057,
+ 0.5211049048549607,
+ 0.5045061974997817,
+ 0.4756747093542917,
+ 0.4510243093175787,
+ 0.43720796140577434,
+ 0.43699771861106773,
+ 0.4198419337428743,
+ 0.4189312930213621,
+ 0.4111607646848555
+ ],
+ [
+ 0.7802848031003088,
+ 0.7568653439024511,
+ 0.7087454875161054,
+ 0.6806762350000554,
+ 0.649197954639825,
+ 0.6152425595567628,
+ 0.6129859537237008,
+ 0.6101416391346713,
+ 0.593338893497739,
+ 0.5808298501631517,
+ 0.5657255229883612,
+ 0.5582898173216522,
+ 0.5561815197840985,
+ 0.518597721503312,
+ 0.5142323593755341,
+ 0.5110971850753365,
+ 0.49710763716381623,
+ 0.4925528173039006,
+ 0.4814569796286176,
+ 0.47799196217246187
+ ],
+ [
+ 0.9783044983215814,
+ 0.9555538905455778,
+ 0.8433313993123266,
+ 0.8353702536841487,
+ 0.8261713853842186,
+ 0.8078979086325264,
+ 0.8007525681516008,
+ 0.7026531678327351,
+ 0.6956641143723907,
+ 0.6766890012304124,
+ 0.6573001298366619,
+ 0.6372762411075359,
+ 0.5769765842228483,
+ 0.5637656826555218,
+ 0.5545090092531815,
+ 0.5086827713554241,
+ 0.4967327024737791,
+ 0.4863837677076879,
+ 0.40474759143763617,
+ 0.3853735272538427
+ ],
+ [
+ 0.7910525071404478,
+ 0.7640395514391561,
+ 0.6939360184030542,
+ 0.6834326759170836,
+ 0.6690337252951615,
+ 0.664628108994463,
+ 0.6588107469297264,
+ 0.6479283350739071,
+ 0.63779700195664,
+ 0.5917637745977166,
+ 0.5883345759866266,
+ 0.5708207765547634,
+ 0.5609322365011993,
+ 0.5365384611860594,
+ 0.5340218916047861,
+ 0.46280447952010484,
+ 0.4288260554851132,
+ 0.42237132111160886,
+ 0.41541525456366535,
+ 0.40986253204933765
+ ],
+ [
+ 0.820188832221044,
+ 0.794184780982263,
+ 0.7281778552747312,
+ 0.6331656896573479,
+ 0.6220807181920863,
+ 0.524198894163934,
+ 0.5183472320989028,
+ 0.48975370627116843,
+ 0.48402159324583727,
+ 0.47040493509285075,
+ 0.46905116305358285,
+ 0.45972756449852825,
+ 0.45767728700742627,
+ 0.45461122662424686,
+ 0.4504092073295217,
+ 0.4438204464060996,
+ 0.44360822549430523,
+ 0.44270527554322225,
+ 0.4288113750834287,
+ 0.4286714584063651
+ ],
+ [
+ 0.9994939553424885,
+ 0.9266247695746682,
+ 0.8439580553174099,
+ 0.7012204625662272,
+ 0.6660890853042024,
+ 0.5913322411577588,
+ 0.5252304759242761,
+ 0.5200976342225982,
+ 0.4969602239982234,
+ 0.4898392499472101,
+ 0.4297304908662429,
+ 0.42034412574594776,
+ 0.4180791171566931,
+ 0.41437650310357194,
+ 0.40267351221036407,
+ 0.40049806622954104,
+ 0.39010276716391085,
+ 0.3777211915482102,
+ 0.3708074580708833,
+ 0.3677717993934955
+ ],
+ [
+ 0.863784588910498,
+ 0.8488403513248525,
+ 0.7926336936033286,
+ 0.7777115208711418,
+ 0.7352259048165143,
+ 0.6999260497332764,
+ 0.6900297242465839,
+ 0.6870863458317699,
+ 0.6604905552816427,
+ 0.6572652038815273,
+ 0.6083205381651764,
+ 0.5838422122340977,
+ 0.5619577090566419,
+ 0.556366465659106,
+ 0.5126867140987288,
+ 0.5116311056259395,
+ 0.49399512465637174,
+ 0.49296019310618383,
+ 0.47372948945539695,
+ 0.4567795859118854
+ ],
+ [
+ 1.0,
+ 0.655781161371561,
+ 0.6278408255740009,
+ 0.6130972229441736,
+ 0.6044486071963677,
+ 0.5719740560710058,
+ 0.5432921617770673,
+ 0.539018299458266,
+ 0.520864234976183,
+ 0.5157917884252228,
+ 0.5147068579724757,
+ 0.4963900407085783,
+ 0.4729704165084534,
+ 0.4639025110338397,
+ 0.44473358685306474,
+ 0.4298971009872484,
+ 0.41435117438688207,
+ 0.4118853786917515,
+ 0.4043318564660724,
+ 0.4016354283208603
+ ],
+ [
+ 0.9366111589981891,
+ 0.8986288529127686,
+ 0.8511126349971793,
+ 0.7597734479029775,
+ 0.7508383909767472,
+ 0.7242830098779309,
+ 0.703550112902005,
+ 0.6813136337700293,
+ 0.598691746907198,
+ 0.5644811866352161,
+ 0.5366049575984224,
+ 0.5248037136815105,
+ 0.48471192090040016,
+ 0.4844469173240846,
+ 0.45166082514068895,
+ 0.4300487740223671,
+ 0.4081519993629169,
+ 0.40645479111002475,
+ 0.39963060335696166,
+ 0.3916055194728336
+ ],
+ [
+ 0.9025367758565842,
+ 0.7681996751149988,
+ 0.7506937346780141,
+ 0.7000110480568473,
+ 0.6921126565355731,
+ 0.689303313720648,
+ 0.654495538674735,
+ 0.632667889795783,
+ 0.6306934615546519,
+ 0.6176125012587508,
+ 0.5371611303515949,
+ 0.5331156688367931,
+ 0.5084374772416503,
+ 0.49955075082827693,
+ 0.395258502210907,
+ 0.3874457660372943,
+ 0.38536249903387787,
+ 0.38149486294579404,
+ 0.36689280809015307,
+ 0.3651941742006453
+ ],
+ [
+ 0.9999999999999999,
+ 0.8015438819005435,
+ 0.5851241924326395,
+ 0.5816012251090573,
+ 0.48238915446452696,
+ 0.4783635071240557,
+ 0.46620021954568225,
+ 0.44949004460369674,
+ 0.44207402126897277,
+ 0.43792066076472586,
+ 0.409155583371668,
+ 0.4037798382402314,
+ 0.39849179677628443,
+ 0.39482940905134806,
+ 0.3931568519168458,
+ 0.38746481063961874,
+ 0.3572550010938801,
+ 0.3562832659500845,
+ 0.35614961973393416,
+ 0.3550258878305793
+ ],
+ [
+ 0.7904082365619579,
+ 0.7543821192776627,
+ 0.7439208905606156,
+ 0.7402698331129101,
+ 0.7221040993822156,
+ 0.7220443027358673,
+ 0.700979833020017,
+ 0.6852222087299424,
+ 0.6406754650215908,
+ 0.5951083559219658,
+ 0.5910046918450378,
+ 0.5880480519063711,
+ 0.5690614509328424,
+ 0.5671133522345134,
+ 0.510867902854077,
+ 0.4857259563343521,
+ 0.42962868895535566,
+ 0.4295745350357466,
+ 0.41636813692980607,
+ 0.4160029694716054
+ ],
+ [
+ 0.8160685969370942,
+ 0.8112093376246111,
+ 0.79024368591912,
+ 0.7183216592493737,
+ 0.6965626173865112,
+ 0.6962842311691324,
+ 0.6777331478148189,
+ 0.6427708411794193,
+ 0.6422043387506999,
+ 0.6315538636430877,
+ 0.6249202845852228,
+ 0.6134625010712614,
+ 0.6044043907767718,
+ 0.6003876427998464,
+ 0.582158049712492,
+ 0.5751400751169586,
+ 0.5681794455926441,
+ 0.5536528950961374,
+ 0.5151024383263165,
+ 0.5138734209570142
+ ],
+ [
+ 0.7000774108427333,
+ 0.6747437832695702,
+ 0.6668890363328608,
+ 0.6009577388038483,
+ 0.5706582284621019,
+ 0.5488931043544235,
+ 0.49737887305104567,
+ 0.4966441599947995,
+ 0.49265935054809984,
+ 0.4913165062513445,
+ 0.48461511492474785,
+ 0.4690699601353485,
+ 0.46854773431264995,
+ 0.4234804778402376,
+ 0.4106614277622942,
+ 0.40657773430157,
+ 0.39168763007721186,
+ 0.37340137860264566,
+ 0.36058838184595343,
+ 0.35638966065114974
+ ],
+ [
+ 1.0,
+ 0.9229051196574047,
+ 0.8804722052778882,
+ 0.8718343316292774,
+ 0.8686989677197159,
+ 0.8411628919261658,
+ 0.8300311809740998,
+ 0.8228258953198457,
+ 0.7961725789239195,
+ 0.748407656758632,
+ 0.5841630210799319,
+ 0.3964419996310576,
+ 0.32756960843096444,
+ 0.3265059406654157,
+ 0.3181061844412731,
+ 0.3131847112895672,
+ 0.298001491678026,
+ 0.2830630558380446,
+ 0.2701915878121153,
+ 0.26169144664040783
+ ],
+ [
+ 0.7871742901679846,
+ 0.7595674758701796,
+ 0.6614748982018251,
+ 0.6208547347390642,
+ 0.6020152695181685,
+ 0.5697405066651718,
+ 0.569628746146295,
+ 0.5351643008167192,
+ 0.5159834341621998,
+ 0.49698993372150524,
+ 0.49072762286299004,
+ 0.48188807840624853,
+ 0.47446054227487355,
+ 0.4684364487014524,
+ 0.4555633515665736,
+ 0.4544725392750424,
+ 0.43265619163637037,
+ 0.42853022325121626,
+ 0.41450492624725144,
+ 0.40783873080739985
+ ],
+ [
+ 0.8810902376045922,
+ 0.8583756473266473,
+ 0.817020272451366,
+ 0.8130946385725049,
+ 0.7193516324360136,
+ 0.6640448596277668,
+ 0.6588939653041663,
+ 0.654767512228678,
+ 0.616223160376939,
+ 0.5933203556168654,
+ 0.5445035026485614,
+ 0.5278428648132791,
+ 0.46221056255585713,
+ 0.4172947138586078,
+ 0.3990847111799142,
+ 0.37440430304077826,
+ 0.3645409850466088,
+ 0.35137231967796667,
+ 0.343013893908993,
+ 0.31069853567554756
+ ],
+ [
+ 0.8022955108922241,
+ 0.6609042380923636,
+ 0.6410509005123098,
+ 0.5329837106548193,
+ 0.5215806063889996,
+ 0.5147772584300562,
+ 0.4798529081175831,
+ 0.4577085846105525,
+ 0.4338000274570271,
+ 0.4257920522606336,
+ 0.4064461975344612,
+ 0.3976972927104364,
+ 0.39223340952301766,
+ 0.3919631429342318,
+ 0.3893579487352864,
+ 0.3851171394791594,
+ 0.38459975912269,
+ 0.3840362479606722,
+ 0.37310016857423817,
+ 0.3658812541287919
+ ],
+ [
+ 0.8946303738220956,
+ 0.7132437533916238,
+ 0.6770450251781133,
+ 0.6550104565011992,
+ 0.6265380428063991,
+ 0.6167742756367665,
+ 0.595693947375208,
+ 0.5896752243954027,
+ 0.5300302166963337,
+ 0.518034632467915,
+ 0.509113433604037,
+ 0.4722441144959586,
+ 0.44081230222409007,
+ 0.43319282137547466,
+ 0.41743464054038837,
+ 0.41316031464403136,
+ 0.40096412685276506,
+ 0.3874723776216728,
+ 0.38590085696534937,
+ 0.37503004034277637
+ ],
+ [
+ 0.8318855878999165,
+ 0.8199857539724257,
+ 0.7409684639577219,
+ 0.7187949687852062,
+ 0.686067953683446,
+ 0.6846440390333118,
+ 0.6698213672846524,
+ 0.6673217642397229,
+ 0.6672590282962694,
+ 0.6567233080926795,
+ 0.5664448755764618,
+ 0.5528564318061813,
+ 0.5416251951535909,
+ 0.5314300379418458,
+ 0.49422574443637163,
+ 0.4874399521439706,
+ 0.4223966255871397,
+ 0.4122138024621427,
+ 0.3918444831709328,
+ 0.3733058980361928
+ ],
+ [
+ 0.7424995071051074,
+ 0.7047432592562353,
+ 0.6894670619680481,
+ 0.5662650250130966,
+ 0.5643916060990347,
+ 0.5471237340612456,
+ 0.5405920001342179,
+ 0.5391233462030114,
+ 0.5358111115474076,
+ 0.5253218178115366,
+ 0.49644160052430186,
+ 0.48598008620996636,
+ 0.4764281573951712,
+ 0.4525026873051813,
+ 0.4523702520292173,
+ 0.4437688860121015,
+ 0.4286375703268467,
+ 0.42291940410200846,
+ 0.41325138120642657,
+ 0.3957889276167583
+ ],
+ [
+ 0.8898478491156026,
+ 0.7968631213785546,
+ 0.7079601878963129,
+ 0.6537297709239678,
+ 0.6450644271800376,
+ 0.550214198622218,
+ 0.5176694752298711,
+ 0.5126354488174946,
+ 0.5084005500599087,
+ 0.503297323298409,
+ 0.4933113953479332,
+ 0.4929545389673196,
+ 0.4927171264797129,
+ 0.4891564316123277,
+ 0.4808450093408286,
+ 0.4714194305691388,
+ 0.4467427557942876,
+ 0.4240613831434613,
+ 0.40860650100217955,
+ 0.40791576694808995
+ ],
+ [
+ 0.8417540830511958,
+ 0.7832388922815634,
+ 0.7800539342280266,
+ 0.7320824209913183,
+ 0.7101576466465895,
+ 0.7010181367691684,
+ 0.6894948385499344,
+ 0.6746485125443202,
+ 0.6677941113965423,
+ 0.6535420036375093,
+ 0.6333651839892276,
+ 0.6260076369249005,
+ 0.5593892575140414,
+ 0.5490046662482294,
+ 0.5244853713432484,
+ 0.47558222948379736,
+ 0.47255511306281217,
+ 0.46183513310515767,
+ 0.350981736516697,
+ 0.3507857862882272
+ ],
+ [
+ 0.774496785937816,
+ 0.71288063868637,
+ 0.6344879853171514,
+ 0.6188473630320402,
+ 0.6175767698942465,
+ 0.583265103326201,
+ 0.5601298028691625,
+ 0.5533416821720747,
+ 0.5331537229679915,
+ 0.5320562451081736,
+ 0.5293222157033252,
+ 0.5153370569163709,
+ 0.5148539349434639,
+ 0.506664702303055,
+ 0.4879059777404138,
+ 0.4796944200431269,
+ 0.47723167385271403,
+ 0.46659669900575296,
+ 0.4599915533181541,
+ 0.44282978241562343
+ ],
+ [
+ 0.8955429528554126,
+ 0.883027757327165,
+ 0.7845101754463513,
+ 0.7643482015436319,
+ 0.6977290548801076,
+ 0.6666372429685017,
+ 0.6342936695737815,
+ 0.6258234184136691,
+ 0.6172608020271974,
+ 0.5482896912637402,
+ 0.47511961779229306,
+ 0.4607011112707008,
+ 0.45485938449512153,
+ 0.45234643740043895,
+ 0.4372162694353354,
+ 0.40672865257498625,
+ 0.4048488811314981,
+ 0.3705273777498211,
+ 0.36374202772728614,
+ 0.3510013721317749
+ ],
+ [
+ 0.7692190326400856,
+ 0.7476474743460773,
+ 0.6792580592317323,
+ 0.659781286871933,
+ 0.6239278185718852,
+ 0.5711903014531343,
+ 0.5556321919670502,
+ 0.5115591844837319,
+ 0.48553253225430804,
+ 0.4666452529268951,
+ 0.4656301122412587,
+ 0.46255108990286214,
+ 0.4624496011149653,
+ 0.45182383584636604,
+ 0.4494169703059187,
+ 0.43249225385738976,
+ 0.4055328698966335,
+ 0.4046027400624389,
+ 0.3912708103602457,
+ 0.38923284828856874
+ ],
+ [
+ 0.8764996177616473,
+ 0.8551211555238958,
+ 0.8330490682706689,
+ 0.7794535303468377,
+ 0.7563787996588845,
+ 0.6321702493862581,
+ 0.5762627881245588,
+ 0.5683239471843547,
+ 0.5534098652252994,
+ 0.5444199510912648,
+ 0.5428890927346712,
+ 0.4734891949789949,
+ 0.47163900120281776,
+ 0.4430305038548793,
+ 0.44219851533950266,
+ 0.4391927742751264,
+ 0.4048847779122736,
+ 0.40392199360310876,
+ 0.4025146549079792,
+ 0.3880858050663737
+ ],
+ [
+ 0.9254563928966286,
+ 0.9178364278430065,
+ 0.9027795567223011,
+ 0.8573780887267823,
+ 0.8319176868672157,
+ 0.8269099998931836,
+ 0.7757471398981666,
+ 0.6910591345802612,
+ 0.6804773429860936,
+ 0.655921490590449,
+ 0.6045058042714154,
+ 0.60307944027577,
+ 0.5580919187232306,
+ 0.5571629931884703,
+ 0.5521395253446499,
+ 0.5385622554176106,
+ 0.5104361242488658,
+ 0.5076543967914124,
+ 0.3977288070001568,
+ 0.37436452575070556
+ ],
+ [
+ 0.9193283206759364,
+ 0.7790933086381899,
+ 0.7532583712680763,
+ 0.7331826412344252,
+ 0.7331217971250255,
+ 0.7226874974506328,
+ 0.7117631009697907,
+ 0.7096642283378871,
+ 0.6986514095509262,
+ 0.6935594856364656,
+ 0.6604246031110143,
+ 0.6533358919799462,
+ 0.622763263122531,
+ 0.6069501618521667,
+ 0.6001171159859511,
+ 0.5525367253863509,
+ 0.4828014845526492,
+ 0.4323411399027953,
+ 0.43064101105835184,
+ 0.4046892341232612
+ ],
+ [
+ 0.853391220486433,
+ 0.7220933541424792,
+ 0.7073575383336244,
+ 0.6687721657453876,
+ 0.6097453579374825,
+ 0.5858439939309414,
+ 0.5606073474087958,
+ 0.5533872167644541,
+ 0.549505579408803,
+ 0.5239888174650869,
+ 0.4856105403823774,
+ 0.4848988447231335,
+ 0.4845580920116798,
+ 0.4784435542017009,
+ 0.47256704876806355,
+ 0.4496178196331335,
+ 0.44775587562302965,
+ 0.44076290306164356,
+ 0.4366744406001503,
+ 0.4280367862403823
+ ],
+ [
+ 0.7715321531996346,
+ 0.653958309068413,
+ 0.6498926141509747,
+ 0.5682950514939171,
+ 0.5622601982422182,
+ 0.5585150031806226,
+ 0.4783113720624187,
+ 0.4768488562604076,
+ 0.47460831820031774,
+ 0.45605673411462944,
+ 0.44928251474940306,
+ 0.4473526597503522,
+ 0.4465816256281743,
+ 0.44314909251706924,
+ 0.4368592445112658,
+ 0.43595379497337927,
+ 0.43520815913753863,
+ 0.42996157615138264,
+ 0.4254800912166924,
+ 0.41846820114547045
+ ],
+ [
+ 0.7169691892876855,
+ 0.6686623399299532,
+ 0.6515891399673011,
+ 0.6446043773152217,
+ 0.6083134380698356,
+ 0.6014109422404239,
+ 0.5294938687239024,
+ 0.4914896397364341,
+ 0.4906744657761566,
+ 0.4640151736561501,
+ 0.46022944495123874,
+ 0.43696911640802727,
+ 0.4327564733843604,
+ 0.42071738021065785,
+ 0.41543621796625074,
+ 0.41235244934184906,
+ 0.4053658843226491,
+ 0.40079978403335953,
+ 0.38431461917900256,
+ 0.378536461009006
+ ],
+ [
+ 1.0,
+ 0.6570577855928736,
+ 0.6095377987897921,
+ 0.597369811955438,
+ 0.5840715582133009,
+ 0.5565989834361439,
+ 0.49902376910231966,
+ 0.4956432809729486,
+ 0.48572067082182213,
+ 0.4794885073326614,
+ 0.46656269028401964,
+ 0.4608080287093301,
+ 0.46038046216276074,
+ 0.45879044839686434,
+ 0.42096255505320274,
+ 0.4031534587347191,
+ 0.40299023201840556,
+ 0.3943465811660766,
+ 0.3940846654859833,
+ 0.3860090369333833
+ ],
+ [
+ 1.0,
+ 0.7928580186398761,
+ 0.7284320343056914,
+ 0.6594787084396134,
+ 0.5541939632084033,
+ 0.48610732327086625,
+ 0.4762216490359108,
+ 0.4543428569831517,
+ 0.4508737724025192,
+ 0.4347199679260212,
+ 0.406799443546046,
+ 0.3963283339054794,
+ 0.38881350416049393,
+ 0.3862786843501004,
+ 0.3684374120947047,
+ 0.3674149140967956,
+ 0.36717415526875935,
+ 0.3636163201404675,
+ 0.35461928274929255,
+ 0.3512864183126708
+ ],
+ [
+ 0.8622469552584491,
+ 0.848945946762232,
+ 0.7943588363813976,
+ 0.7770216025493593,
+ 0.7033963145250792,
+ 0.6386461767442702,
+ 0.6278428757999598,
+ 0.6055201060658395,
+ 0.585394558300841,
+ 0.5807226331161884,
+ 0.5526057443863701,
+ 0.5473002778685532,
+ 0.5280810811876324,
+ 0.516856426657699,
+ 0.5157685808299134,
+ 0.5152316558469938,
+ 0.510465662427631,
+ 0.5055807506358158,
+ 0.48017805190670804,
+ 0.47400105478455684
+ ],
+ [
+ 0.8197338206004547,
+ 0.7619188076755096,
+ 0.7504076989167472,
+ 0.7165858305799898,
+ 0.6979184649695642,
+ 0.6826576020661426,
+ 0.6736560557139916,
+ 0.6402821966229069,
+ 0.6367229981853246,
+ 0.6120833312441697,
+ 0.5999868059805821,
+ 0.5998111105282378,
+ 0.5990360791090372,
+ 0.5846742821016255,
+ 0.5715368111462682,
+ 0.5700965794641193,
+ 0.5406117011474056,
+ 0.5397946717344742,
+ 0.5229198472717222,
+ 0.5185386667721694
+ ],
+ [
+ 0.7265713687438369,
+ 0.7244791008303488,
+ 0.6965882654978848,
+ 0.6872633592339841,
+ 0.6701124028652194,
+ 0.6633239394830582,
+ 0.6611619484958919,
+ 0.6047915351318507,
+ 0.5378733760285623,
+ 0.5369928722440375,
+ 0.524853980122717,
+ 0.51688819753151,
+ 0.498452495010126,
+ 0.491724202866648,
+ 0.4783603113146535,
+ 0.4586745879244334,
+ 0.43338297056858077,
+ 0.41137830477849585,
+ 0.39890472117831244,
+ 0.3872374776822146
+ ],
+ [
+ 0.8777675114115376,
+ 0.8626919244883431,
+ 0.8198369099648503,
+ 0.666750292851451,
+ 0.6640432479392919,
+ 0.6289761692418943,
+ 0.6167013727474283,
+ 0.5998174678932235,
+ 0.5993123387824214,
+ 0.5703411400693992,
+ 0.5700467041743406,
+ 0.5547273232354839,
+ 0.5405588157304455,
+ 0.5174928002298119,
+ 0.4799672511716134,
+ 0.4299896827154394,
+ 0.41754947050115454,
+ 0.4157067340867237,
+ 0.4140460982124919,
+ 0.4122679793464429
+ ],
+ [
+ 0.7143448369726162,
+ 0.6710894537984269,
+ 0.6698442300956339,
+ 0.6655977512505494,
+ 0.6631783546578796,
+ 0.5819855608617693,
+ 0.5413059028723729,
+ 0.5352725723210112,
+ 0.507469918862818,
+ 0.479558898934702,
+ 0.47726357026158894,
+ 0.4230994384844322,
+ 0.41699524988986125,
+ 0.40876813796547756,
+ 0.40737850952379584,
+ 0.39922851206167315,
+ 0.3519058623151061,
+ 0.34845469035634813,
+ 0.3304304130455956,
+ 0.32785151589612
+ ],
+ [
+ 0.7618509499751729,
+ 0.7063609332699929,
+ 0.6813897952711317,
+ 0.6429263823829631,
+ 0.6194458732827545,
+ 0.587513533799173,
+ 0.5522476367329479,
+ 0.5399287753356983,
+ 0.5343971290147025,
+ 0.5293805592453895,
+ 0.5283802152205268,
+ 0.5268586800084373,
+ 0.5206812194297394,
+ 0.5061995188432717,
+ 0.4883681986215979,
+ 0.4811138326913485,
+ 0.4797501024203221,
+ 0.47935544361045873,
+ 0.4656352290776529,
+ 0.44784493124835706
+ ],
+ [
+ 0.9999439317501033,
+ 0.9892670676627827,
+ 0.8234718996041946,
+ 0.8052126575962358,
+ 0.800643168909439,
+ 0.7791271072224443,
+ 0.7477953267718229,
+ 0.7361628513466896,
+ 0.7135190371780367,
+ 0.6517776829349713,
+ 0.634565893992089,
+ 0.6208923701105206,
+ 0.6044083807365312,
+ 0.5975748967583218,
+ 0.498122331801235,
+ 0.41790649551666537,
+ 0.4134249528058289,
+ 0.39735894135030353,
+ 0.38944509229907376,
+ 0.36784590699211683
+ ],
+ [
+ 0.8287415772902524,
+ 0.6707547564345313,
+ 0.6604992556701154,
+ 0.6478893128778587,
+ 0.6352711634268582,
+ 0.6158835055855758,
+ 0.5935014942764582,
+ 0.5913502851122432,
+ 0.5911672421798615,
+ 0.5741363203337684,
+ 0.5646686854855099,
+ 0.5615820797286664,
+ 0.5601285525028096,
+ 0.5600655762049659,
+ 0.5411132834624267,
+ 0.5327696206677904,
+ 0.49953632700438744,
+ 0.49758321892300594,
+ 0.4954449615760872,
+ 0.4899927203653833
+ ],
+ [
+ 0.7773497229519528,
+ 0.7499374381179567,
+ 0.6619229837133387,
+ 0.6580226090163216,
+ 0.6438715348033626,
+ 0.6123250796396476,
+ 0.6032100924516243,
+ 0.5293632484484028,
+ 0.5180742486303775,
+ 0.5175358022057196,
+ 0.5108256347414958,
+ 0.506590023516411,
+ 0.49585706764547655,
+ 0.4848867098636016,
+ 0.4497229954875863,
+ 0.44957849184923115,
+ 0.3847318396756871,
+ 0.3725574926600854,
+ 0.3580233831257763,
+ 0.33427433265094825
+ ],
+ [
+ 1.0,
+ 0.8338676816967674,
+ 0.6604605389482749,
+ 0.6567000079062637,
+ 0.6419020238600969,
+ 0.6057165408972692,
+ 0.5641107971655351,
+ 0.5126195254743955,
+ 0.5026105182716589,
+ 0.5001693804389244,
+ 0.49962912589506736,
+ 0.43582702773548043,
+ 0.4344473707492512,
+ 0.37924813602351587,
+ 0.34981015104437796,
+ 0.34491814771591106,
+ 0.33442335524540046,
+ 0.3192473511572308,
+ 0.3106790292298586,
+ 0.30510414511358763
+ ],
+ [
+ 0.7292346179569196,
+ 0.7264209813533478,
+ 0.5828138799602596,
+ 0.580599892098627,
+ 0.5743101127093286,
+ 0.5606855788945102,
+ 0.5323769314254279,
+ 0.515367348688039,
+ 0.49165092612566996,
+ 0.4914254223912051,
+ 0.46205045880119244,
+ 0.44932207880428787,
+ 0.42820601569208716,
+ 0.4260955407449038,
+ 0.42170045452020466,
+ 0.4142762625095291,
+ 0.4100069240240382,
+ 0.40799614544364615,
+ 0.3857618789308416,
+ 0.38543411059011556
+ ],
+ [
+ 0.9418673656393753,
+ 0.9105281117857825,
+ 0.909855759713626,
+ 0.771200869406351,
+ 0.6635589624672549,
+ 0.6629608970470249,
+ 0.6559360127849782,
+ 0.6460334033814435,
+ 0.6030863684545952,
+ 0.5646645429575329,
+ 0.5613623418713466,
+ 0.5484531553318809,
+ 0.5156593716973329,
+ 0.5051212994089849,
+ 0.5030169832802591,
+ 0.49286768620976834,
+ 0.48465637364121833,
+ 0.422429630062146,
+ 0.41683070359257224,
+ 0.37424291934859216
+ ],
+ [
+ 0.8901050389814955,
+ 0.7267263015236722,
+ 0.7202959489967864,
+ 0.674135169268308,
+ 0.6183396775532823,
+ 0.6053468615761981,
+ 0.6002465835369202,
+ 0.5967513061660623,
+ 0.5938837218378978,
+ 0.5694949469382646,
+ 0.5344626990512895,
+ 0.5300905263991966,
+ 0.49387635488234055,
+ 0.484096109118983,
+ 0.4743898738620008,
+ 0.4505487753435956,
+ 0.41103584237487184,
+ 0.40036208404685697,
+ 0.38603757381924864,
+ 0.3629969946732059
+ ],
+ [
+ 0.7604600825539876,
+ 0.6721661978975629,
+ 0.5768996396624784,
+ 0.5767767980168201,
+ 0.5747947821358064,
+ 0.5729508490483964,
+ 0.5634191979182275,
+ 0.5395708137960289,
+ 0.5346151152636236,
+ 0.5299180382815061,
+ 0.529165433959671,
+ 0.5122097284828062,
+ 0.5005204336343138,
+ 0.4945303337035578,
+ 0.48812500259767744,
+ 0.4646549251839114,
+ 0.4535350567739683,
+ 0.4496805966469415,
+ 0.44946526400116843,
+ 0.4207005862252218
+ ],
+ [
+ 0.8823059964056019,
+ 0.8140319098771,
+ 0.781942018529868,
+ 0.730573421119998,
+ 0.7173569652746109,
+ 0.6617007667342476,
+ 0.6349788771443332,
+ 0.6134907888647088,
+ 0.6032159567556115,
+ 0.6012042722162196,
+ 0.578553787178523,
+ 0.5653440816024529,
+ 0.5615812926053309,
+ 0.5465741657122605,
+ 0.5405065180505888,
+ 0.5385485935752834,
+ 0.5376290023815459,
+ 0.5315886785194085,
+ 0.5163841322560412,
+ 0.5067727420431731
+ ],
+ [
+ 0.7808566071038693,
+ 0.7186204160923568,
+ 0.6666567932921762,
+ 0.6055960550039086,
+ 0.5946698643756123,
+ 0.5648977321578265,
+ 0.5572720351511605,
+ 0.5512496821177475,
+ 0.5297280073166728,
+ 0.5022764142565969,
+ 0.4953157048764067,
+ 0.47558677387062165,
+ 0.4698245047891583,
+ 0.46688131393248916,
+ 0.46025470014903513,
+ 0.46016349568331616,
+ 0.45070973393735647,
+ 0.4119476944368108,
+ 0.40462816583482086,
+ 0.4001633673112985
+ ],
+ [
+ 0.798975841668791,
+ 0.6774353098019252,
+ 0.6540166675975082,
+ 0.6194762676765426,
+ 0.6047102609740618,
+ 0.6019651992277788,
+ 0.5899735395473127,
+ 0.5644849920276015,
+ 0.5516931673593239,
+ 0.5235347668335721,
+ 0.5048835093494479,
+ 0.5017642824643173,
+ 0.4905730951645907,
+ 0.48842786377422587,
+ 0.48644364964586106,
+ 0.455356348887372,
+ 0.4422975137913042,
+ 0.42573917445203546,
+ 0.41867778545978585,
+ 0.4085138712216935
+ ],
+ [
+ 0.9586996034516424,
+ 0.9528352724418125,
+ 0.8822075260359407,
+ 0.8782058386153132,
+ 0.8734008835756844,
+ 0.8710108733939953,
+ 0.8624225649144464,
+ 0.7903125459984826,
+ 0.7790168957000198,
+ 0.7698830072088205,
+ 0.7656533613949291,
+ 0.536151086461469,
+ 0.4887380154681442,
+ 0.42839592118810954,
+ 0.4279374501948589,
+ 0.3907666846495667,
+ 0.3730434986107306,
+ 0.36698481683123113,
+ 0.33399018711356626,
+ 0.33283708197357503
+ ],
+ [
+ 0.838475912432648,
+ 0.7977037894446148,
+ 0.7537695965184845,
+ 0.7214853953233181,
+ 0.6574627691220591,
+ 0.6529397369553158,
+ 0.6129844375945841,
+ 0.6015083019534118,
+ 0.6011359831771571,
+ 0.5435645610243983,
+ 0.5080587589487493,
+ 0.49360064161568173,
+ 0.49052326473778807,
+ 0.46272851347703714,
+ 0.4558094494484817,
+ 0.40407379025067336,
+ 0.40112769214612454,
+ 0.3849305126295398,
+ 0.383779043614193,
+ 0.37776913591431643
+ ],
+ [
+ 0.7946408414930011,
+ 0.774522591219611,
+ 0.7595036129897981,
+ 0.7351261531963936,
+ 0.7113695157199565,
+ 0.7113513034689805,
+ 0.7026648833607108,
+ 0.6634254648869293,
+ 0.6112375563152361,
+ 0.5877503413944272,
+ 0.582699247630392,
+ 0.5621753469846528,
+ 0.5554239789439617,
+ 0.5466744545559975,
+ 0.5074869602875396,
+ 0.5065959117286197,
+ 0.4823548974901646,
+ 0.4714023631598683,
+ 0.45680243580351115,
+ 0.4558085219020438
+ ],
+ [
+ 0.8644228215462099,
+ 0.8496404889824934,
+ 0.763204809940019,
+ 0.7542196839416067,
+ 0.6287038368373918,
+ 0.6071902723542881,
+ 0.6071712970109607,
+ 0.6054362698254738,
+ 0.5986104267825231,
+ 0.5828585885935363,
+ 0.580502662802972,
+ 0.5747835523032787,
+ 0.5208905358616935,
+ 0.5194990321165267,
+ 0.4947051933015004,
+ 0.4922828315398256,
+ 0.47886160605122996,
+ 0.47194821422113775,
+ 0.4684401623938613,
+ 0.45090174534906036
+ ],
+ [
+ 0.9392228443883711,
+ 0.9187101711761917,
+ 0.8907823195901643,
+ 0.8779672149049511,
+ 0.8446896454771357,
+ 0.8254189143716844,
+ 0.8222564850785903,
+ 0.630631023617868,
+ 0.6116011200125608,
+ 0.5532191475572128,
+ 0.5348374145375894,
+ 0.34094833518079515,
+ 0.2980313940591237,
+ 0.27073927208777454,
+ 0.2697110991039693,
+ 0.26118861107245417,
+ 0.25831889404484815,
+ 0.2559277775494788,
+ 0.2547019763763664,
+ 0.24592197700433535
+ ],
+ [
+ 0.9118056448569098,
+ 0.7502256352875427,
+ 0.6130544828532147,
+ 0.6122564605071698,
+ 0.578190811305736,
+ 0.5696337682834953,
+ 0.5508988855498967,
+ 0.528146986458226,
+ 0.49609584102838444,
+ 0.4862767476628566,
+ 0.4838467416160419,
+ 0.46551908166798994,
+ 0.4284496594822583,
+ 0.4279857548370676,
+ 0.4277594155857295,
+ 0.38942938514182845,
+ 0.3891350689134254,
+ 0.3797721967734712,
+ 0.37505085424231294,
+ 0.3744290534193208
+ ],
+ [
+ 0.8558169404809347,
+ 0.8163596559050856,
+ 0.7275843983527159,
+ 0.6789317216080696,
+ 0.6648865823077867,
+ 0.6104571439201116,
+ 0.5180188708225214,
+ 0.49979400597065926,
+ 0.4897634312098886,
+ 0.4769319111850216,
+ 0.4600191517840535,
+ 0.43613542866132815,
+ 0.4349827290616024,
+ 0.43392436838250614,
+ 0.42680154465665643,
+ 0.41590846620677957,
+ 0.4104654161447817,
+ 0.3989823539845744,
+ 0.3982580840416652,
+ 0.3923999116938627
+ ],
+ [
+ 0.9999999999998955,
+ 0.839708018957863,
+ 0.6966449862674232,
+ 0.5955501569042678,
+ 0.5850468963598039,
+ 0.5620401283546249,
+ 0.5391887305878127,
+ 0.4833387361633201,
+ 0.47940356647357113,
+ 0.4790719373355225,
+ 0.47324687961611456,
+ 0.44280110264867284,
+ 0.4320024737783819,
+ 0.42634774842860856,
+ 0.42387772376669947,
+ 0.4221871599446482,
+ 0.4146063286773183,
+ 0.4088039501657509,
+ 0.39189189831385757,
+ 0.37906755688914373
+ ],
+ [
+ 0.8985639447110239,
+ 0.7925447963873439,
+ 0.7672785974921786,
+ 0.7338833738724362,
+ 0.7207584408891201,
+ 0.7058832301683745,
+ 0.6081045850424109,
+ 0.5988758213575573,
+ 0.5948861553566518,
+ 0.5930671424551359,
+ 0.5690317738578231,
+ 0.5650574045881606,
+ 0.5397851413611288,
+ 0.5340815734656006,
+ 0.5307659522596838,
+ 0.5154407585322123,
+ 0.5014002634018516,
+ 0.4920873738185325,
+ 0.4909193736893352,
+ 0.4754543355709892
+ ],
+ [
+ 0.7080604034793088,
+ 0.654524949961067,
+ 0.642106808222205,
+ 0.6415368733990244,
+ 0.6066958512615616,
+ 0.5732146604828667,
+ 0.5457476566274264,
+ 0.527453524846107,
+ 0.5266542374689165,
+ 0.5015384369083457,
+ 0.495050690315864,
+ 0.49243589613012817,
+ 0.4810814327736103,
+ 0.4696281744074604,
+ 0.4684983454507311,
+ 0.458086376400266,
+ 0.4471347401080565,
+ 0.446629552675159,
+ 0.4404518589662699,
+ 0.43640887190618677
+ ],
+ [
+ 0.8618880112976653,
+ 0.6858278384016807,
+ 0.5776008051918429,
+ 0.5743717032647535,
+ 0.5733178192156719,
+ 0.561719960825551,
+ 0.5191350810296069,
+ 0.5085588523762505,
+ 0.5031628535404443,
+ 0.4785935759571558,
+ 0.45845431021965183,
+ 0.45652826858966833,
+ 0.443203943873269,
+ 0.4272860131985037,
+ 0.4240491478552192,
+ 0.4060564860180743,
+ 0.40208109841872114,
+ 0.3912055836007493,
+ 0.389772161835804,
+ 0.389314336763706
+ ],
+ [
+ 0.6456576700526714,
+ 0.6054019656603943,
+ 0.5792530738532428,
+ 0.549626350805681,
+ 0.5287840052664509,
+ 0.5218772935635799,
+ 0.5205273836269046,
+ 0.4934761278866295,
+ 0.4910501178918351,
+ 0.4794923050849714,
+ 0.47553404194852283,
+ 0.47251778439415354,
+ 0.4685797732788491,
+ 0.44557123508738244,
+ 0.4235358834521387,
+ 0.4136140066919952,
+ 0.41149206445586567,
+ 0.40173798251158477,
+ 0.3947349145245277,
+ 0.38946787822440626
+ ],
+ [
+ 0.8095193695090962,
+ 0.7648832258546955,
+ 0.7214713049187123,
+ 0.6612706011152273,
+ 0.6505483409585678,
+ 0.5849337929724454,
+ 0.539900980352552,
+ 0.536915234090606,
+ 0.512078033552861,
+ 0.4712833312215134,
+ 0.46695888483857384,
+ 0.46593387496102934,
+ 0.46478171482040065,
+ 0.45064226407950897,
+ 0.44615228051206124,
+ 0.44191887744217473,
+ 0.43939968466615653,
+ 0.42070549556174097,
+ 0.41349200017362875,
+ 0.4013911388076916
+ ],
+ [
+ 0.7994127949567874,
+ 0.6992476282271411,
+ 0.6602403206760148,
+ 0.6552973401095229,
+ 0.6527107682399425,
+ 0.6353024591011313,
+ 0.6247052088018401,
+ 0.6178007565889988,
+ 0.6022374531121559,
+ 0.5795443658954991,
+ 0.5717299406331049,
+ 0.5635546767947919,
+ 0.5467525796552605,
+ 0.5446752100839785,
+ 0.5367152960243118,
+ 0.5332402361213058,
+ 0.4800132192033344,
+ 0.47706661206232603,
+ 0.47685792167374075,
+ 0.46612729899184424
+ ],
+ [
+ 0.99999999999989,
+ 0.9214391594836,
+ 0.6464219327171667,
+ 0.6242750452496564,
+ 0.6204627312507919,
+ 0.6031479463341114,
+ 0.5981352009261454,
+ 0.5785835590303597,
+ 0.5568919667590557,
+ 0.5422809523850161,
+ 0.533922081354541,
+ 0.5286210091472912,
+ 0.516409575232445,
+ 0.5158675603775005,
+ 0.5120567232244899,
+ 0.5071032827074569,
+ 0.5015594117674158,
+ 0.4302051319233243,
+ 0.4001802674138495,
+ 0.3678794411710064
+ ],
+ [
+ 0.9684625161850068,
+ 0.945627503725011,
+ 0.9325727286601391,
+ 0.9064773181884571,
+ 0.8778632659854274,
+ 0.8596198449179877,
+ 0.7581611417833627,
+ 0.7577426534275453,
+ 0.7014243534006718,
+ 0.6918924379449842,
+ 0.6430711827254677,
+ 0.6417677595662762,
+ 0.5695552619442144,
+ 0.5680738005734918,
+ 0.5657182242776565,
+ 0.5222325067900419,
+ 0.5154027742383194,
+ 0.5071539087285006,
+ 0.43950068317664165,
+ 0.3852138803898083
+ ],
+ [
+ 0.8812764791150467,
+ 0.7370936074623569,
+ 0.6592309453616125,
+ 0.6586394928486098,
+ 0.6385310267298114,
+ 0.5857255724569044,
+ 0.533109981743193,
+ 0.5141512007443384,
+ 0.505902252428152,
+ 0.4815162574496893,
+ 0.46857461142474555,
+ 0.45660319374641717,
+ 0.44740189140470965,
+ 0.44387347634247765,
+ 0.411253366171643,
+ 0.3903062371880802,
+ 0.3852486970057152,
+ 0.37919544298242325,
+ 0.37132492741198175,
+ 0.3520218102850251
+ ],
+ [
+ 0.7435916075022754,
+ 0.7405281987193723,
+ 0.6906218956517529,
+ 0.6855769529141258,
+ 0.6541018085375402,
+ 0.6150195220539745,
+ 0.5945781068652642,
+ 0.5942506738384424,
+ 0.5870669106339207,
+ 0.5749072386773748,
+ 0.5676962660068122,
+ 0.5599565323150004,
+ 0.5571677360385745,
+ 0.5343190248533085,
+ 0.5233319673307708,
+ 0.5098890435517582,
+ 0.505315623093817,
+ 0.5005059034812518,
+ 0.4808954004199716,
+ 0.4802860906869542
+ ],
+ [
+ 0.8286093354434041,
+ 0.7554706662374864,
+ 0.6741768963456971,
+ 0.634326051455208,
+ 0.629090689743139,
+ 0.5881126686275829,
+ 0.5819064643278873,
+ 0.5632023857854948,
+ 0.5582100874590714,
+ 0.5487711493034687,
+ 0.5439366681924209,
+ 0.508481788070043,
+ 0.4882186957457335,
+ 0.4868708036991022,
+ 0.4830564930515266,
+ 0.4753493520432376,
+ 0.4513374900944217,
+ 0.4509443107150569,
+ 0.43920735865257177,
+ 0.4281388880628804
+ ],
+ [
+ 0.902544023655508,
+ 0.8455382142140495,
+ 0.731814592458373,
+ 0.6765549536393636,
+ 0.6604672024287176,
+ 0.6518943262079414,
+ 0.6257690943084275,
+ 0.5416137852517215,
+ 0.5308922485542468,
+ 0.4602298473470953,
+ 0.43627951095952205,
+ 0.43408909474590374,
+ 0.41555679252568467,
+ 0.41396236517259394,
+ 0.39068340487448366,
+ 0.380892378768504,
+ 0.365910584299056,
+ 0.3591098453503602,
+ 0.3590167904140181,
+ 0.34482966869962695
+ ],
+ [
+ 0.8127262968973767,
+ 0.804678564855317,
+ 0.7648840133033603,
+ 0.7538873825025691,
+ 0.7497612925366285,
+ 0.7404180837550525,
+ 0.7048629454248444,
+ 0.7047778821147062,
+ 0.6659418778831826,
+ 0.6215231624807012,
+ 0.609183700488119,
+ 0.5784405131405171,
+ 0.5766076469346554,
+ 0.572713847358904,
+ 0.5449563172281854,
+ 0.5192610920559504,
+ 0.4985710106362414,
+ 0.4741308999732195,
+ 0.454846801880356,
+ 0.4521181204211979
+ ],
+ [
+ 0.832406728751502,
+ 0.7756101805073266,
+ 0.6083175762685659,
+ 0.5881267567897349,
+ 0.5452666366802479,
+ 0.5296618553735872,
+ 0.5279552331617712,
+ 0.5140005805964938,
+ 0.5070156106900069,
+ 0.5034968558254317,
+ 0.5024608384621168,
+ 0.49047730378058174,
+ 0.48863110465338916,
+ 0.4873082290241458,
+ 0.47815301217097167,
+ 0.45680246408223246,
+ 0.4364859177348072,
+ 0.4344617759254143,
+ 0.42646030244103217,
+ 0.4250360785652608
+ ],
+ [
+ 0.9419238449980283,
+ 0.9338738726072154,
+ 0.7233938278941692,
+ 0.6976121979045302,
+ 0.6870010646675637,
+ 0.5365500092624559,
+ 0.5035884852890533,
+ 0.4967751352159304,
+ 0.49181697923374024,
+ 0.48612075499141383,
+ 0.4643833752739359,
+ 0.46217629508262864,
+ 0.4619514428478321,
+ 0.45660093188744333,
+ 0.45586496827804107,
+ 0.445284486825592,
+ 0.4420916172807731,
+ 0.44151304590697515,
+ 0.44052607560575346,
+ 0.4378818883127696
+ ],
+ [
+ 0.8203157234078902,
+ 0.7464126023777456,
+ 0.7459699970481699,
+ 0.6360258383215448,
+ 0.6117045295916724,
+ 0.6111448606163222,
+ 0.6026814152976908,
+ 0.591435193112199,
+ 0.5896897272404396,
+ 0.5683652244034605,
+ 0.5624858629622151,
+ 0.5210487627826449,
+ 0.4642234321012408,
+ 0.4640512274137905,
+ 0.4624209163920834,
+ 0.459830435329765,
+ 0.441764099264831,
+ 0.43473898106281955,
+ 0.41378599553086454,
+ 0.3974892898009466
+ ],
+ [
+ 1.0,
+ 0.8258921909552093,
+ 0.7700047828985588,
+ 0.7276150498558561,
+ 0.6950175199104274,
+ 0.6887229408499124,
+ 0.6456638449984113,
+ 0.6248364593687455,
+ 0.6062283353050256,
+ 0.5964736578332542,
+ 0.5618747368831248,
+ 0.5068590019968521,
+ 0.4990659485596123,
+ 0.4879507839746649,
+ 0.48546635037640135,
+ 0.4784849931414873,
+ 0.4663633843640719,
+ 0.4663154510127951,
+ 0.42061500005615016,
+ 0.4119825183284096
+ ],
+ [
+ 0.7868596124319651,
+ 0.7427342021467253,
+ 0.7246227103729789,
+ 0.7088812376695937,
+ 0.6187192386956786,
+ 0.5787149291241636,
+ 0.5656485704662945,
+ 0.5178497929674677,
+ 0.4957185665528687,
+ 0.4884423337428445,
+ 0.44337466321573266,
+ 0.4343792061514556,
+ 0.4247992365438563,
+ 0.4182393837840609,
+ 0.4093119237274229,
+ 0.40250398284222577,
+ 0.4014634954495995,
+ 0.39930783780346246,
+ 0.39188664576401294,
+ 0.3865861202123565
+ ],
+ [
+ 0.8421258323992462,
+ 0.810799246745854,
+ 0.7362943753184015,
+ 0.6642124968717561,
+ 0.643824249564869,
+ 0.6216067784247528,
+ 0.6214972412738131,
+ 0.5979922360143279,
+ 0.5766232534721716,
+ 0.5764508444325709,
+ 0.5523400684491331,
+ 0.54276167215462,
+ 0.4781070241443343,
+ 0.46070552283206034,
+ 0.4505680561187669,
+ 0.4406655437521901,
+ 0.4323980845514881,
+ 0.42247421139884245,
+ 0.4139917746161914,
+ 0.40232178781895295
+ ],
+ [
+ 0.7545148028002842,
+ 0.7319841125620289,
+ 0.6173025362746034,
+ 0.5564292255560804,
+ 0.5494644528565285,
+ 0.517823364353162,
+ 0.5080506205174655,
+ 0.4844623154571433,
+ 0.45223101713355657,
+ 0.44879339738318835,
+ 0.4473246478048808,
+ 0.42368202263289934,
+ 0.4210759164543261,
+ 0.4185941262926183,
+ 0.41353820632480554,
+ 0.3876256003946694,
+ 0.38070904956769747,
+ 0.37724067816869516,
+ 0.37507918169455734,
+ 0.37308616408947604
+ ],
+ [
+ 0.6975807469604285,
+ 0.6876914753861361,
+ 0.662470921415965,
+ 0.6304087943885386,
+ 0.6064139184955808,
+ 0.5868919515174604,
+ 0.5383908330251603,
+ 0.484603102534955,
+ 0.4608145878756904,
+ 0.4412989209269149,
+ 0.43355558037307373,
+ 0.4204928351866476,
+ 0.41252500664517977,
+ 0.41114742499878354,
+ 0.4058424251425825,
+ 0.38819743344384006,
+ 0.3858690587159728,
+ 0.3734478030199021,
+ 0.36714590316749207,
+ 0.3548124931293413
+ ],
+ [
+ 0.9484189330834716,
+ 0.946455312776599,
+ 0.9136124357672234,
+ 0.8913579882710394,
+ 0.8718195830121327,
+ 0.8662003650810631,
+ 0.8405896770650534,
+ 0.7960026420039239,
+ 0.7265781010078805,
+ 0.7054784127829181,
+ 0.6442773442006002,
+ 0.42512496842411523,
+ 0.3618591098179742,
+ 0.3523664648134845,
+ 0.3205960587316665,
+ 0.3155864157023702,
+ 0.31467337697346764,
+ 0.3063883828120487,
+ 0.29873349642966673,
+ 0.29269202147355805
+ ],
+ [
+ 0.9999999999999998,
+ 0.762826920120467,
+ 0.6969398342943696,
+ 0.6409063928887747,
+ 0.61054517438508,
+ 0.5922223294658442,
+ 0.579713288706887,
+ 0.5317935993038965,
+ 0.5214475301493304,
+ 0.5159666201730053,
+ 0.5022868636278565,
+ 0.4890731544721349,
+ 0.4665129298144539,
+ 0.44823539503332255,
+ 0.44781840679955465,
+ 0.43443936760531215,
+ 0.42513294084784126,
+ 0.415578547468982,
+ 0.40457949865202625,
+ 0.3942988036503793
+ ],
+ [
+ 0.8078969197887208,
+ 0.7785109098720842,
+ 0.7663878434583564,
+ 0.7101308101178221,
+ 0.6852444611063826,
+ 0.6755608473685129,
+ 0.5965957779364699,
+ 0.5690568395217097,
+ 0.5540970526760947,
+ 0.5348720223696966,
+ 0.4780000090217501,
+ 0.4724853291247586,
+ 0.46399432281305175,
+ 0.4504978814762205,
+ 0.43700581355872214,
+ 0.433525304991918,
+ 0.4250098998965417,
+ 0.4193073645112974,
+ 0.40132458992331166,
+ 0.4002749970205369
+ ],
+ [
+ 0.7351086087444344,
+ 0.6982452592135823,
+ 0.624566442946769,
+ 0.6177455154612328,
+ 0.6021384400155504,
+ 0.5902155143881438,
+ 0.5777021722098086,
+ 0.5642992012740208,
+ 0.5454698030028131,
+ 0.5305711142374665,
+ 0.518809089860391,
+ 0.5157099296771612,
+ 0.5098541359469542,
+ 0.4988982796399865,
+ 0.497263847837249,
+ 0.4815900851590748,
+ 0.46133806506364594,
+ 0.43615057625066755,
+ 0.4155506386741713,
+ 0.3966700978980093
+ ],
+ [
+ 0.7835943386069874,
+ 0.6995774208562009,
+ 0.6770299904486135,
+ 0.6615595969596144,
+ 0.6465464192344247,
+ 0.5848092173235651,
+ 0.5655267236982622,
+ 0.5645118763134763,
+ 0.5559786830495185,
+ 0.5341069511320611,
+ 0.5196838413508794,
+ 0.5160954062392668,
+ 0.5115400075010003,
+ 0.5025750226385073,
+ 0.5012292648403981,
+ 0.49966074918714243,
+ 0.49804018828923485,
+ 0.4964090602746949,
+ 0.49369071130529063,
+ 0.49197848154108736
+ ],
+ [
+ 0.9752768612734708,
+ 0.9410555821394742,
+ 0.8868795418004828,
+ 0.8627449139964971,
+ 0.8306644714564537,
+ 0.8298427534695959,
+ 0.8272134844618533,
+ 0.8251510218209868,
+ 0.7809453191124397,
+ 0.7610505398266475,
+ 0.6590767612709125,
+ 0.5722843821840451,
+ 0.5472848368690464,
+ 0.5418082729221693,
+ 0.48340335101086274,
+ 0.4336670914837478,
+ 0.4037598072537857,
+ 0.3999096387946056,
+ 0.3896731307462629,
+ 0.3619120062331303
+ ],
+ [
+ 0.7600182522614845,
+ 0.7134869973403256,
+ 0.6791778119428513,
+ 0.6608002478750077,
+ 0.6276919856723848,
+ 0.5793936791543733,
+ 0.5504901000340248,
+ 0.5464808015943164,
+ 0.5289609458082585,
+ 0.5219355303993459,
+ 0.4722562670637843,
+ 0.4520939781823797,
+ 0.43870994466235813,
+ 0.42534873557448427,
+ 0.4182624852347183,
+ 0.4074339204292346,
+ 0.39209117578812114,
+ 0.3909558892590063,
+ 0.3780776590477747,
+ 0.37501870674607335
+ ],
+ [
+ 0.8833634582035483,
+ 0.8293296420103041,
+ 0.7558351644934582,
+ 0.7512486672607599,
+ 0.7316865496412062,
+ 0.7182578942633208,
+ 0.7176526938847068,
+ 0.7151812322378905,
+ 0.6906855948709139,
+ 0.6839397205857212,
+ 0.6830592796391837,
+ 0.6731111560589201,
+ 0.6723548999181692,
+ 0.6198582132545725,
+ 0.6071405791813967,
+ 0.601841340862717,
+ 0.5961276495489665,
+ 0.5821229303059361,
+ 0.5580178160717624,
+ 0.5424636788821546
+ ],
+ [
+ 0.7903581434104157,
+ 0.66108007169751,
+ 0.6380703527741813,
+ 0.6256649586816931,
+ 0.614990368122508,
+ 0.5938400962466572,
+ 0.5844423851377125,
+ 0.5689793378158237,
+ 0.5373668669299351,
+ 0.5215890113845062,
+ 0.5195262386824306,
+ 0.5128863197757476,
+ 0.4728485764695741,
+ 0.46454237149295313,
+ 0.4578746591547064,
+ 0.45378017065322235,
+ 0.4395317164890931,
+ 0.41121309146466156,
+ 0.4101976730770608,
+ 0.40302755425701586
+ ],
+ [
+ 0.9061129166312827,
+ 0.6222157832901953,
+ 0.6123828634092203,
+ 0.572140819704771,
+ 0.5695604631064222,
+ 0.5557630491750012,
+ 0.5300851910060359,
+ 0.5247535484734658,
+ 0.5041547996944981,
+ 0.48951826897670725,
+ 0.48899686692627775,
+ 0.48771606760962477,
+ 0.47513842157129915,
+ 0.46228091365677315,
+ 0.4604864949502596,
+ 0.4536827037830442,
+ 0.4485124648707687,
+ 0.4480386092020027,
+ 0.4365631305772658,
+ 0.42987973448733696
+ ],
+ [
+ 0.7898380612298617,
+ 0.7883223651206381,
+ 0.6713646807751726,
+ 0.6298386805474498,
+ 0.6213483217430823,
+ 0.6093790323965882,
+ 0.5821071508209043,
+ 0.574577367959587,
+ 0.5659607125752222,
+ 0.5336463330860016,
+ 0.5271794206668743,
+ 0.4967441416062271,
+ 0.47375733004568477,
+ 0.4697902372845071,
+ 0.469497103226307,
+ 0.46642833262825034,
+ 0.4643427134409394,
+ 0.4639753831796001,
+ 0.45913582247995,
+ 0.43764893026842
+ ],
+ [
+ 0.974740547168546,
+ 0.9329468642665765,
+ 0.8684911510370259,
+ 0.8179355960481236,
+ 0.7755589838207302,
+ 0.6850920394405733,
+ 0.6435645339331842,
+ 0.6150935419851107,
+ 0.6007553654323496,
+ 0.5941861799194033,
+ 0.5861008220080184,
+ 0.5840913620799517,
+ 0.5639546385442449,
+ 0.5357242737565324,
+ 0.5338937151559542,
+ 0.5333796407624799,
+ 0.5289074871050419,
+ 0.526286004061074,
+ 0.5255956953418153,
+ 0.5186483201599474
+ ],
+ [
+ 1.0,
+ 0.8012691722165308,
+ 0.7042154892742656,
+ 0.6971332567579905,
+ 0.6878931499963008,
+ 0.622924950991324,
+ 0.6086346188587924,
+ 0.589709121886125,
+ 0.5584362326064047,
+ 0.5565807998442175,
+ 0.5453824252172321,
+ 0.5273724715365444,
+ 0.5231726737650837,
+ 0.5057102154394596,
+ 0.4865869092914297,
+ 0.4782338584492333,
+ 0.45741916356614404,
+ 0.3964037076146228,
+ 0.3867757714155238,
+ 0.36787944117144233
+ ],
+ [
+ 0.9566952755835761,
+ 0.9100882417976555,
+ 0.8979173012815178,
+ 0.8192981479822643,
+ 0.7593836205201655,
+ 0.7491774689292355,
+ 0.7346114132838777,
+ 0.700707601459176,
+ 0.6982846602973518,
+ 0.6640436044841665,
+ 0.6606580170535366,
+ 0.6520908678520567,
+ 0.6395012107737283,
+ 0.6379961913441932,
+ 0.6059118124432326,
+ 0.5724866843757096,
+ 0.5607244593070841,
+ 0.558528592405301,
+ 0.5190987414794876,
+ 0.5125941721224331
+ ],
+ [
+ 0.7584239849549699,
+ 0.7033189774834291,
+ 0.699303017136264,
+ 0.6920403863651393,
+ 0.683296129308574,
+ 0.6467602713250233,
+ 0.6340660969095415,
+ 0.59896629604036,
+ 0.5745780308451572,
+ 0.5718773525784953,
+ 0.566800259441226,
+ 0.5593968782253722,
+ 0.531172453005632,
+ 0.5256799853816194,
+ 0.5173682453057391,
+ 0.5052619425472435,
+ 0.5014685250086166,
+ 0.49509182735223595,
+ 0.4346969306601379,
+ 0.41684075645239305
+ ],
+ [
+ 0.8418589699391337,
+ 0.8166526590449583,
+ 0.6963026828814272,
+ 0.6860312406842264,
+ 0.6850546390974785,
+ 0.6525060487571397,
+ 0.5684791005802161,
+ 0.526820937885324,
+ 0.5148743292866598,
+ 0.5134982339104418,
+ 0.5132153106946316,
+ 0.5040109547206983,
+ 0.4990035303118079,
+ 0.4935317081111005,
+ 0.48596776948662895,
+ 0.46912852919384646,
+ 0.4503206254404818,
+ 0.4374028071854361,
+ 0.4285818676594335,
+ 0.4271770292918618
+ ],
+ [
+ 0.9225362934290908,
+ 0.8626589280550511,
+ 0.7663624490160801,
+ 0.7145626397449527,
+ 0.7028974785157233,
+ 0.690061742404886,
+ 0.672388065942124,
+ 0.6134417453151894,
+ 0.6107848134440318,
+ 0.5869320147195999,
+ 0.585530335146736,
+ 0.5839644697404336,
+ 0.5688168035803634,
+ 0.5347920430623634,
+ 0.5173246661953157,
+ 0.5036849174530578,
+ 0.49860201110767544,
+ 0.48332422824850024,
+ 0.4665915989636189,
+ 0.43264878921809563
+ ],
+ [
+ 0.7865043709236668,
+ 0.7739208969699642,
+ 0.7605123307119481,
+ 0.7354956745030635,
+ 0.6714154613166378,
+ 0.6661866462053102,
+ 0.665084399989697,
+ 0.6644876231253612,
+ 0.6636762175528831,
+ 0.6577450364060901,
+ 0.645571086870782,
+ 0.6191596243963801,
+ 0.6114580856122394,
+ 0.6094397438574854,
+ 0.5835715939077913,
+ 0.5834490923552607,
+ 0.5632933777420828,
+ 0.55742054942083,
+ 0.5089549980366386,
+ 0.49407801314612176
+ ],
+ [
+ 0.827084922377221,
+ 0.772117048192517,
+ 0.5969734261442865,
+ 0.5926756813325513,
+ 0.592567925895851,
+ 0.5589996065877607,
+ 0.5587744601988758,
+ 0.5221204045289372,
+ 0.5200507362091958,
+ 0.5169307492599491,
+ 0.5155349579260841,
+ 0.5062761555971775,
+ 0.49160173261520357,
+ 0.4851375080573822,
+ 0.48296526757366265,
+ 0.4814319688529956,
+ 0.47245894870427263,
+ 0.4524893160338001,
+ 0.43098246087386843,
+ 0.4142540779216021
+ ],
+ [
+ 0.9933694691588836,
+ 0.9072028894178819,
+ 0.868122430931761,
+ 0.8194361771255767,
+ 0.800997241605826,
+ 0.7036466367219301,
+ 0.6018268873258946,
+ 0.5473706348403166,
+ 0.545097857153806,
+ 0.532518475080959,
+ 0.5231991744098885,
+ 0.51287449105465,
+ 0.4994363777294438,
+ 0.4897427258604625,
+ 0.45905265408329016,
+ 0.4381686179862471,
+ 0.426167242223946,
+ 0.38026271858796534,
+ 0.37921359670835947,
+ 0.36656264646506675
+ ],
+ [
+ 0.862288421119904,
+ 0.8342872156723335,
+ 0.5933276309640256,
+ 0.5634105285802793,
+ 0.532075773838634,
+ 0.5306983585469464,
+ 0.49416905403199796,
+ 0.4791712228627348,
+ 0.4603974684050542,
+ 0.45472708088392855,
+ 0.44939023571746317,
+ 0.4340897311789694,
+ 0.41292596117320784,
+ 0.4096422417489215,
+ 0.39783681371167134,
+ 0.39375553962913196,
+ 0.3879185169120189,
+ 0.3862950374595411,
+ 0.38068541724752625,
+ 0.3755116786503202
+ ],
+ [
+ 0.862310515121196,
+ 0.7693586625060006,
+ 0.6758673672715009,
+ 0.6025362747294146,
+ 0.576133053421145,
+ 0.5592815983435201,
+ 0.5571428550491885,
+ 0.5467347861739993,
+ 0.5334755614284424,
+ 0.49146612232100084,
+ 0.48306891986001876,
+ 0.4541552264948663,
+ 0.4289653943538475,
+ 0.42356413520740344,
+ 0.4156524324000734,
+ 0.40923254026131345,
+ 0.40195510309237786,
+ 0.39956941238533455,
+ 0.39833767833979905,
+ 0.38626285263619686
+ ],
+ [
+ 0.8299719482051413,
+ 0.7748937577186394,
+ 0.7098578662971405,
+ 0.6771296107173321,
+ 0.6602511278774206,
+ 0.6413388441817164,
+ 0.6328139500435168,
+ 0.6032300345864037,
+ 0.5961752268226155,
+ 0.5512695479889402,
+ 0.5409858360578716,
+ 0.5319295884766535,
+ 0.5071367871031309,
+ 0.5013835753746325,
+ 0.4979503405681912,
+ 0.4900643837996129,
+ 0.48987716306990653,
+ 0.4790055085710315,
+ 0.44809991576268804,
+ 0.4398988269333227
+ ],
+ [
+ 0.7814943085039335,
+ 0.7603842142213166,
+ 0.6813609260020422,
+ 0.637685447249291,
+ 0.6094318035833506,
+ 0.5620848914884914,
+ 0.5092443285236636,
+ 0.4973083792040577,
+ 0.47572288068594293,
+ 0.44947119203205366,
+ 0.4493932328006225,
+ 0.44671009610197965,
+ 0.4459510381160572,
+ 0.435019196756335,
+ 0.408807758552252,
+ 0.40774067818416826,
+ 0.4042171282499369,
+ 0.39152236043704425,
+ 0.38242131643895017,
+ 0.37319321151440205
+ ],
+ [
+ 0.993378029959665,
+ 0.9739569484841434,
+ 0.912629911073723,
+ 0.8788614393084588,
+ 0.7788122478367999,
+ 0.7535946575345345,
+ 0.7164776591960944,
+ 0.6993394489851151,
+ 0.6900400217492929,
+ 0.5790021212644897,
+ 0.5706810967084274,
+ 0.5696941928504938,
+ 0.5310088183243236,
+ 0.5230662209800105,
+ 0.5130550572340287,
+ 0.506870411748922,
+ 0.4110572404498262,
+ 0.37926450471930073,
+ 0.37285754467426163,
+ 0.3652218789746598
+ ],
+ [
+ 0.9807495241698074,
+ 0.847054094721274,
+ 0.7077230599831231,
+ 0.6956645729130312,
+ 0.6747721214661004,
+ 0.6052492458997203,
+ 0.5880800466919209,
+ 0.5815841899777959,
+ 0.5543150630030069,
+ 0.5496421067985221,
+ 0.49100776837220256,
+ 0.42993404864389484,
+ 0.4232555241207192,
+ 0.42104985084220914,
+ 0.41936033362371394,
+ 0.40396617416963176,
+ 0.39930394645891754,
+ 0.39380599386336823,
+ 0.38630475213697707,
+ 0.36645660572789884
+ ],
+ [
+ 0.7117374177264624,
+ 0.681239010369117,
+ 0.6092819763786891,
+ 0.6019959463490636,
+ 0.5973654345488135,
+ 0.5957734561535197,
+ 0.5794379501798492,
+ 0.5732448218378668,
+ 0.528114929688599,
+ 0.523058517308663,
+ 0.5167053941215248,
+ 0.4871621414069569,
+ 0.479627086859393,
+ 0.4725244494847247,
+ 0.4710046907128478,
+ 0.45634651968616863,
+ 0.4526184008218135,
+ 0.4280515413599436,
+ 0.4177031933936022,
+ 0.41548382467802997
+ ],
+ [
+ 0.801715686150389,
+ 0.6914876768756323,
+ 0.65586998379842,
+ 0.6138797165771678,
+ 0.5509277011723097,
+ 0.5156653197428467,
+ 0.5127495181382,
+ 0.5112718868805687,
+ 0.5097268941409742,
+ 0.5085834700255741,
+ 0.5041341931100414,
+ 0.4656819676931019,
+ 0.4372540759227288,
+ 0.4279987849011657,
+ 0.4209562657671807,
+ 0.4184131369935631,
+ 0.4167814458097323,
+ 0.38825509943320285,
+ 0.3809579109345865,
+ 0.3779992276107201
+ ],
+ [
+ 0.7257256392230724,
+ 0.7106210670480693,
+ 0.7014228426300619,
+ 0.6635402461061534,
+ 0.6288713049971175,
+ 0.6240930564255961,
+ 0.6135127466283425,
+ 0.6024204733631647,
+ 0.594105868332844,
+ 0.5550297078548285,
+ 0.527683260183683,
+ 0.5257190909646607,
+ 0.51281488736378,
+ 0.48449722508875703,
+ 0.47677736775236296,
+ 0.440817953195762,
+ 0.42913369218649255,
+ 0.4211025471289708,
+ 0.4094144869394659,
+ 0.39107340547929165
+ ],
+ [
+ 0.8296294557963928,
+ 0.7273299026210593,
+ 0.6680438468099006,
+ 0.6542044438487781,
+ 0.6263961176183099,
+ 0.5688960496811577,
+ 0.5607003705031586,
+ 0.5288416454065983,
+ 0.5227767585028539,
+ 0.5227211260616361,
+ 0.5096411018209908,
+ 0.5065335654110177,
+ 0.49544320476279347,
+ 0.47520102928467994,
+ 0.4594602119530544,
+ 0.4480449438257139,
+ 0.4415296100881497,
+ 0.41567614935353575,
+ 0.4029458474822246,
+ 0.40083713688739453
+ ],
+ [
+ 0.8752546377269201,
+ 0.7702551714686725,
+ 0.682875035553284,
+ 0.680104936144058,
+ 0.6509606757003044,
+ 0.6457184259227562,
+ 0.6316253798355551,
+ 0.6002118747416945,
+ 0.5979654870847194,
+ 0.5935961829766471,
+ 0.5853112716526737,
+ 0.5360971445577641,
+ 0.522404783868407,
+ 0.48363331423844436,
+ 0.4384400713768215,
+ 0.4324848509285206,
+ 0.4232075674483325,
+ 0.4166254634446537,
+ 0.4060869425208969,
+ 0.4000534009846191
+ ],
+ [
+ 0.7633692959015552,
+ 0.7360555887186804,
+ 0.6492794974086216,
+ 0.6456629665451634,
+ 0.6424409481858502,
+ 0.5942297554639527,
+ 0.5635164189170243,
+ 0.509044330397175,
+ 0.5044599206722784,
+ 0.49060558427538326,
+ 0.47524493155004477,
+ 0.4670057873729111,
+ 0.44969168098695983,
+ 0.4478152919777333,
+ 0.44445829466124437,
+ 0.4299045536698733,
+ 0.42403957934829056,
+ 0.42222428100432385,
+ 0.41490467975520556,
+ 0.4113863335509159
+ ],
+ [
+ 0.9018763484429881,
+ 0.7565378557247644,
+ 0.7467508299853569,
+ 0.7075021846900197,
+ 0.6879074213866818,
+ 0.6696180534100753,
+ 0.6619038755086827,
+ 0.6395459050739769,
+ 0.614994936601958,
+ 0.5979679475778585,
+ 0.5741434386872857,
+ 0.5448779269675854,
+ 0.5322273866965277,
+ 0.49956863806461926,
+ 0.4989080857287142,
+ 0.48656460189106343,
+ 0.4424420025187151,
+ 0.44118483154057186,
+ 0.42141651679444636,
+ 0.41259040506502875
+ ],
+ [
+ 0.8373715697222114,
+ 0.7084060863536141,
+ 0.6299258397952021,
+ 0.6100257334390504,
+ 0.6001774661812114,
+ 0.5472499834892376,
+ 0.5300026331471395,
+ 0.5183888499765151,
+ 0.49717768183727873,
+ 0.49047548932581647,
+ 0.4831103998771997,
+ 0.4727625451212077,
+ 0.4698004113793495,
+ 0.4518720488647019,
+ 0.4458475477586908,
+ 0.438522190716778,
+ 0.43651763387126163,
+ 0.4358748284288181,
+ 0.4227315073102028,
+ 0.4117496627122398
+ ],
+ [
+ 0.9527420491109349,
+ 0.942250128458752,
+ 0.9371298492604104,
+ 0.8982670320624282,
+ 0.800328370100576,
+ 0.7610123354163234,
+ 0.7389456775482619,
+ 0.7057705617363323,
+ 0.7032385650013901,
+ 0.6796280941923615,
+ 0.6721159081058721,
+ 0.6550753209989881,
+ 0.6489284991560995,
+ 0.6316334078815282,
+ 0.6296876172617751,
+ 0.5728625151077662,
+ 0.5647601429550064,
+ 0.5387509748104718,
+ 0.4103241784198169,
+ 0.39552319897154686
+ ],
+ [
+ 0.9182982444432592,
+ 0.7337230969303155,
+ 0.7164637297546144,
+ 0.6810265632266299,
+ 0.639121737436337,
+ 0.5967728688820491,
+ 0.583341070258429,
+ 0.570284894424002,
+ 0.5375137159178719,
+ 0.5315102922966506,
+ 0.5288239214116703,
+ 0.5187153081688016,
+ 0.49373028097567206,
+ 0.4826126504614731,
+ 0.46091975205802577,
+ 0.44004096532093157,
+ 0.43155173648418593,
+ 0.3992258873614487,
+ 0.34507549680461225,
+ 0.33267627596535176
+ ],
+ [
+ 0.7174154739539715,
+ 0.6672540925097159,
+ 0.6573537330322945,
+ 0.5247871637393864,
+ 0.507835125793944,
+ 0.5052202057825852,
+ 0.48459472820019817,
+ 0.46579941483414533,
+ 0.45420531864533975,
+ 0.435189158043103,
+ 0.424593298007727,
+ 0.41495720453402724,
+ 0.4144559545553853,
+ 0.40589198893061074,
+ 0.40378044679253217,
+ 0.4010861814651421,
+ 0.40099486428639186,
+ 0.39923058983699683,
+ 0.3977580019595935,
+ 0.39562247104761716
+ ],
+ [
+ 0.9487577206257882,
+ 0.9386452263334824,
+ 0.9358264888373943,
+ 0.9207447916729871,
+ 0.9023175633622991,
+ 0.8941857694476824,
+ 0.8703639835568082,
+ 0.8456838727470517,
+ 0.8186061536289896,
+ 0.6903808884056752,
+ 0.6792131748305069,
+ 0.4910307524747209,
+ 0.40135552040059225,
+ 0.36067067921575846,
+ 0.3304301542343447,
+ 0.2993701764809624,
+ 0.29524326776285154,
+ 0.28783436529344253,
+ 0.2821290577048508,
+ 0.2684382649839516
+ ],
+ [
+ 0.9548312517143905,
+ 0.7782609235106037,
+ 0.7327872274550633,
+ 0.6798402042508105,
+ 0.6683235843183513,
+ 0.6541813098209149,
+ 0.6301079788958018,
+ 0.577335525551558,
+ 0.5677492040343844,
+ 0.5628634714053579,
+ 0.5417150270940951,
+ 0.5060896788664719,
+ 0.4966232503035345,
+ 0.4850559834708067,
+ 0.4782548296088247,
+ 0.4711560576419892,
+ 0.4638328510316692,
+ 0.46242335012857877,
+ 0.45134429280509686,
+ 0.43706454036137443
+ ],
+ [
+ 0.8248489920293104,
+ 0.6966266717972514,
+ 0.6796118876075179,
+ 0.6760640952030483,
+ 0.6716697992655429,
+ 0.6377597098683937,
+ 0.6036706637126042,
+ 0.5915273140148574,
+ 0.5569289332967384,
+ 0.5482540058390908,
+ 0.5429750030869115,
+ 0.5129496707923418,
+ 0.4705918213941299,
+ 0.4595117642758033,
+ 0.453615116461898,
+ 0.4337409992552858,
+ 0.43028655905053204,
+ 0.3803855012856452,
+ 0.36723869310967433,
+ 0.36448014677864005
+ ],
+ [
+ 0.8691535510375431,
+ 0.837301490136563,
+ 0.7999061310864362,
+ 0.7213431302448757,
+ 0.6870946413502296,
+ 0.6663444821020181,
+ 0.6535177402148633,
+ 0.6271620190673728,
+ 0.6115674959681983,
+ 0.6073150461865522,
+ 0.60199950740239,
+ 0.5958642177723246,
+ 0.5839978283596411,
+ 0.5662937040175569,
+ 0.5542288459292805,
+ 0.5426366729642084,
+ 0.5354520345168774,
+ 0.513466918427794,
+ 0.5057119318607184,
+ 0.5049873964860009
+ ],
+ [
+ 0.8374709641874538,
+ 0.7212641031799727,
+ 0.7136384123966255,
+ 0.6875906131435914,
+ 0.6804592557108112,
+ 0.6475362944927987,
+ 0.619257459108582,
+ 0.6191747926806217,
+ 0.6119589167978117,
+ 0.6052643796110669,
+ 0.5860839789386294,
+ 0.5827968984930578,
+ 0.5816522472305544,
+ 0.5069197770859324,
+ 0.5036415920439747,
+ 0.5003759795926399,
+ 0.4559455794931646,
+ 0.4557337460615434,
+ 0.4339503127867337,
+ 0.4275611177485463
+ ],
+ [
+ 1.0,
+ 0.9548487369998042,
+ 0.49608017408596305,
+ 0.47747939266510153,
+ 0.4701914155532795,
+ 0.45900956301132734,
+ 0.45848309223943173,
+ 0.4583700668204569,
+ 0.45688451337504005,
+ 0.4484458693604865,
+ 0.4453421443995793,
+ 0.4411526651681428,
+ 0.43668731900390934,
+ 0.43538839754262404,
+ 0.4287232985631688,
+ 0.42872050690675734,
+ 0.42792408179692754,
+ 0.42238544984355997,
+ 0.42057175174570766,
+ 0.41250768390297193
+ ],
+ [
+ 0.8056332794110386,
+ 0.7000633252971343,
+ 0.6624602194374435,
+ 0.6315063493985626,
+ 0.6267935979807869,
+ 0.5666993764968051,
+ 0.5541162724847324,
+ 0.5457421480095583,
+ 0.5434678060560098,
+ 0.526476523669126,
+ 0.5063230345137186,
+ 0.4970016599923584,
+ 0.4251954137988766,
+ 0.40131664505272413,
+ 0.39006211656139567,
+ 0.3887835549359966,
+ 0.3680180106435535,
+ 0.3505100817054076,
+ 0.3470997090418638,
+ 0.34631774787722586
+ ],
+ [
+ 0.7194769138290615,
+ 0.6060367683022254,
+ 0.5897744451631652,
+ 0.5785519827631588,
+ 0.5478688966264094,
+ 0.5223695904640392,
+ 0.5116803862938226,
+ 0.5115994444821284,
+ 0.5096968237477049,
+ 0.5018419839860989,
+ 0.4926976920261312,
+ 0.46859264104053266,
+ 0.45543918713000475,
+ 0.44679035844828785,
+ 0.4418885592689793,
+ 0.4319884409215059,
+ 0.4161141993103309,
+ 0.4105545177401948,
+ 0.39206553759079776,
+ 0.3801003671791736
+ ],
+ [
+ 0.9465253066341578,
+ 0.701675885710171,
+ 0.6860192639336214,
+ 0.6205686446923908,
+ 0.6187842833581427,
+ 0.6123395791427485,
+ 0.612027408142948,
+ 0.5874967983685669,
+ 0.5502155924981225,
+ 0.5502011262078068,
+ 0.538641947005644,
+ 0.528912202901273,
+ 0.5192488747907847,
+ 0.5176227468737427,
+ 0.5142276727584589,
+ 0.5001794079394922,
+ 0.4915057121686684,
+ 0.48164424446931464,
+ 0.4738763711921654,
+ 0.473692199554578
+ ],
+ [
+ 0.6738529346670605,
+ 0.6672936287911868,
+ 0.6324036208874686,
+ 0.6251301579007271,
+ 0.596810264412734,
+ 0.5760943647278916,
+ 0.5625191611440968,
+ 0.5232854927672788,
+ 0.5226279982122021,
+ 0.510137423338395,
+ 0.4750395035951881,
+ 0.47416911063187595,
+ 0.4658927095273274,
+ 0.4634139785275808,
+ 0.4598957411892929,
+ 0.4459476187366125,
+ 0.445684436455097,
+ 0.44514451993680015,
+ 0.44244076215669853,
+ 0.42895305286534957
+ ],
+ [
+ 0.8660623519823066,
+ 0.7766984982245774,
+ 0.7487945229491826,
+ 0.6974207200687612,
+ 0.6695564020752635,
+ 0.6617261613436743,
+ 0.634852816877493,
+ 0.6162750571752817,
+ 0.613076053795752,
+ 0.5798451442857988,
+ 0.5612564944056031,
+ 0.5439729147347325,
+ 0.5392092078913926,
+ 0.5324738817541771,
+ 0.507638627632725,
+ 0.4989272812803514,
+ 0.4866955160503296,
+ 0.45698962353330647,
+ 0.44778104607220715,
+ 0.4447236157041198
+ ],
+ [
+ 1.0,
+ 0.5094165699821328,
+ 0.5017717693104895,
+ 0.5007538211325042,
+ 0.4840480264335735,
+ 0.48324316183328175,
+ 0.48158269652199553,
+ 0.46264899833735185,
+ 0.4430889464562273,
+ 0.44192252026904166,
+ 0.4386352246500087,
+ 0.43580259085336864,
+ 0.42969251475656545,
+ 0.4293274725017221,
+ 0.41882570567326666,
+ 0.4052624248934069,
+ 0.39602075926997765,
+ 0.392766294960087,
+ 0.3799705253004826,
+ 0.3760051326053028
+ ],
+ [
+ 0.9553750830578325,
+ 0.9281077089728317,
+ 0.9249941931874379,
+ 0.9223065980228495,
+ 0.8685018586819999,
+ 0.8661745208661953,
+ 0.85019183713464,
+ 0.711503606214508,
+ 0.6827557729395077,
+ 0.6762522994198059,
+ 0.6400722285281819,
+ 0.6074895748018465,
+ 0.5732969573146206,
+ 0.5705294538913976,
+ 0.5555705678158942,
+ 0.5398236313326215,
+ 0.529529946953335,
+ 0.5097251906738924,
+ 0.39373303146375876,
+ 0.37299278017113063
+ ],
+ [
+ 0.9999999999999998,
+ 0.9066026213762788,
+ 0.7061706707729245,
+ 0.6409340092723788,
+ 0.5535764921584689,
+ 0.5503875432250164,
+ 0.5397598346856394,
+ 0.5153023025565628,
+ 0.5120171647049186,
+ 0.4659939384536227,
+ 0.46331504485107927,
+ 0.4547962493141417,
+ 0.4504597850737261,
+ 0.43406025827089684,
+ 0.4224033301206115,
+ 0.4201739127734054,
+ 0.4173798910310208,
+ 0.40554329722719584,
+ 0.40055213996402944,
+ 0.3989113680781279
+ ],
+ [
+ 0.7782150346333099,
+ 0.6486919597916576,
+ 0.6462763997396603,
+ 0.6429579229151463,
+ 0.6078974574279825,
+ 0.6026971812017257,
+ 0.5536768047040335,
+ 0.5505141328605854,
+ 0.5460199837088274,
+ 0.5405441590560214,
+ 0.5377356515819771,
+ 0.5100123710894265,
+ 0.5066155816033564,
+ 0.5023372815286853,
+ 0.4896074852867518,
+ 0.4858240374346285,
+ 0.4842704297167972,
+ 0.4553630866412389,
+ 0.45396217047796916,
+ 0.43418263532362844
+ ],
+ [
+ 0.9615992082207182,
+ 0.8683622214343706,
+ 0.7552808968658871,
+ 0.7313356618823753,
+ 0.7220036731803251,
+ 0.6945021076961002,
+ 0.6804277659550289,
+ 0.6781555551656779,
+ 0.6735768125545953,
+ 0.6404066100164236,
+ 0.6401652322637419,
+ 0.6317117114079873,
+ 0.611296464646325,
+ 0.6046983763014883,
+ 0.532107348478896,
+ 0.5184123590776648,
+ 0.5083723518062998,
+ 0.48611218513330434,
+ 0.45975524484356556,
+ 0.41384610274950595
+ ],
+ [
+ 0.9169634209104576,
+ 0.8641602513679785,
+ 0.7780837609928665,
+ 0.7210023872858643,
+ 0.7141061527606255,
+ 0.7060809983223683,
+ 0.6874335245102545,
+ 0.5848877833268638,
+ 0.5662244458557206,
+ 0.5456059027627095,
+ 0.5432791995941411,
+ 0.538844342493405,
+ 0.48116567377618324,
+ 0.48016769661519887,
+ 0.4719305265841503,
+ 0.4622547741078925,
+ 0.44730986855382016,
+ 0.4152043768381546,
+ 0.41081723048651614,
+ 0.38610109393766223
+ ],
+ [
+ 0.8634448891819289,
+ 0.7822492129315628,
+ 0.7382235410771576,
+ 0.7164115336312368,
+ 0.6767271684043588,
+ 0.6156300386919302,
+ 0.5985555703800804,
+ 0.5931903048418,
+ 0.57453762948491,
+ 0.5730828070746519,
+ 0.5537286786777612,
+ 0.5501678244527357,
+ 0.5251523838678366,
+ 0.5235892020647025,
+ 0.5204170172520759,
+ 0.5197393829513672,
+ 0.48769740852120375,
+ 0.4775845515001246,
+ 0.43976482512180615,
+ 0.42870313394552223
+ ],
+ [
+ 0.9999999999999994,
+ 0.8269139155107219,
+ 0.8074306796728251,
+ 0.7696614436095514,
+ 0.7407262296350405,
+ 0.7089678549998205,
+ 0.6642305995459159,
+ 0.6634503296715966,
+ 0.6484816224869094,
+ 0.6333260538032975,
+ 0.6279210365658082,
+ 0.6068069018388315,
+ 0.5845277945028124,
+ 0.5301409430287508,
+ 0.5291615701912425,
+ 0.4981577099732673,
+ 0.4921801675277304,
+ 0.482408796186272,
+ 0.45621145381315337,
+ 0.36787944117144233
+ ],
+ [
+ 0.699571422629406,
+ 0.6949994315584417,
+ 0.6409746996237464,
+ 0.6267312833628529,
+ 0.6161869683609364,
+ 0.5831855541357079,
+ 0.5686176420508162,
+ 0.5610466201030435,
+ 0.5584355017849542,
+ 0.5177875396431092,
+ 0.5116560880299867,
+ 0.5089710271892773,
+ 0.4971626569473029,
+ 0.4957647919909165,
+ 0.49121850777273135,
+ 0.4743580026530749,
+ 0.4425240128404943,
+ 0.4366034642409785,
+ 0.41814128932535405,
+ 0.41321641808760745
+ ],
+ [
+ 0.7984202146883599,
+ 0.7908134854915223,
+ 0.7721839492455886,
+ 0.7650379458081934,
+ 0.7451121176097019,
+ 0.7244031321606892,
+ 0.7079728314564142,
+ 0.7055863277398333,
+ 0.6996756902417345,
+ 0.6741592332392652,
+ 0.6586076687020413,
+ 0.651109097265901,
+ 0.6406122501272984,
+ 0.6302113055773628,
+ 0.6215182303384273,
+ 0.6116418205092007,
+ 0.5829322209458114,
+ 0.5818653258397931,
+ 0.5777118973904203,
+ 0.5631060525240135
+ ],
+ [
+ 0.9527660684984908,
+ 0.8767487661446748,
+ 0.7926325242903449,
+ 0.7733554317026676,
+ 0.7690857898357382,
+ 0.7569767615012531,
+ 0.6793954716203656,
+ 0.6710843272822287,
+ 0.6358171680863551,
+ 0.6181535984846545,
+ 0.6175291146421555,
+ 0.5966145474176331,
+ 0.5600056357739622,
+ 0.5594484495568951,
+ 0.5416130862857884,
+ 0.5217391624522583,
+ 0.519035140197466,
+ 0.5109402136491176,
+ 0.4435597256103626,
+ 0.39345870523146964
+ ],
+ [
+ 0.7679077961134044,
+ 0.6973041260960674,
+ 0.667537338057029,
+ 0.6281959728497817,
+ 0.5885561179972605,
+ 0.5473699113375589,
+ 0.5297946257398252,
+ 0.5294757981493168,
+ 0.5193840940583971,
+ 0.5174685421209096,
+ 0.5134421099557298,
+ 0.4922448313829966,
+ 0.4759179501782499,
+ 0.46560064150337177,
+ 0.46242696783349163,
+ 0.45221576527873764,
+ 0.44688747400033213,
+ 0.44472771798090405,
+ 0.4171815300115795,
+ 0.403236963358074
+ ],
+ [
+ 0.7637218690885313,
+ 0.7410543288718479,
+ 0.7090896126936987,
+ 0.6819983879927354,
+ 0.592878790503182,
+ 0.5452781349534184,
+ 0.5447299167297471,
+ 0.5392549855623576,
+ 0.524779290459965,
+ 0.5039533394856323,
+ 0.49301098838788304,
+ 0.482467321299506,
+ 0.4816976082174247,
+ 0.4779256538996872,
+ 0.46439488092920667,
+ 0.46390927335570276,
+ 0.4440244493468424,
+ 0.41301211997295084,
+ 0.4111672390107724,
+ 0.38492026998138745
+ ],
+ [
+ 0.7969314148497735,
+ 0.7709192325182794,
+ 0.7596953446194152,
+ 0.7381473935697499,
+ 0.6940014346906588,
+ 0.6715567764767224,
+ 0.6383981242104029,
+ 0.6309269469706712,
+ 0.616184242747668,
+ 0.5667512983883695,
+ 0.5449381050422497,
+ 0.4921443283570873,
+ 0.4822370251038434,
+ 0.47895776545409097,
+ 0.4619610786271908,
+ 0.46148332370102274,
+ 0.4118819408681207,
+ 0.39095706923763457,
+ 0.37253339024999155,
+ 0.36430810282494513
+ ],
+ [
+ 0.8131427602608388,
+ 0.6912808689385177,
+ 0.6832430858892429,
+ 0.6656742329510149,
+ 0.6528036380643298,
+ 0.6281840866454134,
+ 0.6244999198417187,
+ 0.6168978300506522,
+ 0.5786236591502485,
+ 0.5693779274728725,
+ 0.5381953097127499,
+ 0.5275343755487135,
+ 0.5253915574792063,
+ 0.48952664241856303,
+ 0.4833056863295249,
+ 0.45611489852314746,
+ 0.42123960151407513,
+ 0.4116221029758348,
+ 0.3902138882757761,
+ 0.3861550965695979
+ ],
+ [
+ 0.9712054103834353,
+ 0.7502518072621551,
+ 0.7071902554744682,
+ 0.6466294740727234,
+ 0.6348180671332123,
+ 0.6310101504331089,
+ 0.6258169628725714,
+ 0.6128045478404498,
+ 0.5836580075833722,
+ 0.5829721054791165,
+ 0.5823974748805478,
+ 0.5797764748386395,
+ 0.5741094819954957,
+ 0.5689282763121987,
+ 0.5681393308556435,
+ 0.5649393265819157,
+ 0.5596970285759462,
+ 0.5314127887284784,
+ 0.5208438620602367,
+ 0.5169202211316617
+ ],
+ [
+ 0.8819883281784117,
+ 0.8438244800778805,
+ 0.8393038169301315,
+ 0.8318775378065691,
+ 0.7971778277578829,
+ 0.7457867645559882,
+ 0.7018943434284588,
+ 0.7014488208708334,
+ 0.6659082600570932,
+ 0.6584022985793166,
+ 0.6053217150089529,
+ 0.601086091687113,
+ 0.5664609721411256,
+ 0.5527703901689404,
+ 0.5462002848035938,
+ 0.523128364970754,
+ 0.5202510745460638,
+ 0.4914662422153221,
+ 0.4913869085953316,
+ 0.4852254769592864
+ ],
+ [
+ 0.7890977154673149,
+ 0.6980524472858447,
+ 0.6351207029730104,
+ 0.6124202266609873,
+ 0.6043698194528468,
+ 0.5960197858121444,
+ 0.5899810208810384,
+ 0.5822900784114358,
+ 0.5624945454658938,
+ 0.5517426705380423,
+ 0.5432454329236013,
+ 0.530529340473346,
+ 0.5244506116588241,
+ 0.4878152205398219,
+ 0.47190367859766924,
+ 0.4607900140700923,
+ 0.4571236147352089,
+ 0.44223111642651114,
+ 0.4378807240390733,
+ 0.39980041360129315
+ ],
+ [
+ 0.7535908619805598,
+ 0.7389586621672367,
+ 0.6730429120225669,
+ 0.6409182220256531,
+ 0.6199063446815632,
+ 0.5657222562885237,
+ 0.5312619886396195,
+ 0.5308683485086415,
+ 0.5285780399858548,
+ 0.5279260743350874,
+ 0.5244815204816822,
+ 0.5202589639128637,
+ 0.5100709073246911,
+ 0.4994743399786117,
+ 0.4546722891723288,
+ 0.4490922787037405,
+ 0.43013264343140767,
+ 0.4140339017512105,
+ 0.39897146491311564,
+ 0.39066672300761085
+ ],
+ [
+ 0.9321336296522409,
+ 0.8816088439788642,
+ 0.8366719450454478,
+ 0.8298999941541647,
+ 0.8232360792002422,
+ 0.8029624652885275,
+ 0.7946788417004915,
+ 0.7940166717633141,
+ 0.7860037759067615,
+ 0.7652330469492953,
+ 0.745265772104412,
+ 0.7257823318527189,
+ 0.6952473951552132,
+ 0.6622779315426869,
+ 0.6565385571490197,
+ 0.6522398131580057,
+ 0.6007399225875798,
+ 0.5853538779944214,
+ 0.5809274193382257,
+ 0.5461148600774347
+ ],
+ [
+ 0.8464327228537041,
+ 0.8292828096442423,
+ 0.8128136843341714,
+ 0.7994197864431198,
+ 0.7659887228852488,
+ 0.7074098244218623,
+ 0.6544216707309979,
+ 0.5948275370926064,
+ 0.5725261023860385,
+ 0.5716817576835769,
+ 0.5632606267451999,
+ 0.5335556404946403,
+ 0.4887880705238492,
+ 0.4881507683295039,
+ 0.4469689684574363,
+ 0.43136097819755326,
+ 0.42026917002114406,
+ 0.42009403127883616,
+ 0.4147140515451765,
+ 0.381856597329072
+ ],
+ [
+ 0.9626650084266981,
+ 0.9317295264030481,
+ 0.9085640844606738,
+ 0.7402218708610886,
+ 0.7364728943201247,
+ 0.7355967213313964,
+ 0.7120202555713132,
+ 0.6786299812308365,
+ 0.6125461104828834,
+ 0.5691775534933102,
+ 0.5173691286518469,
+ 0.4499952939240927,
+ 0.4416120412561932,
+ 0.3843834335224276,
+ 0.38295814966438235,
+ 0.37689015748253535,
+ 0.37060450317364896,
+ 0.36482464732865033,
+ 0.3415915947964859,
+ 0.3292889509848067
+ ],
+ [
+ 0.941012360639409,
+ 0.9001275275830913,
+ 0.8879219567989832,
+ 0.8401208637024351,
+ 0.8360489505589883,
+ 0.6833398500441767,
+ 0.6566056912900738,
+ 0.6548337568765568,
+ 0.6373857542258343,
+ 0.5925461318561003,
+ 0.5790231087546551,
+ 0.5682627699355512,
+ 0.5619135468780139,
+ 0.5547258654365782,
+ 0.5457904931478487,
+ 0.5249372823520361,
+ 0.5198167963323668,
+ 0.49698491222184193,
+ 0.42827931232663546,
+ 0.38356204947271266
+ ],
+ [
+ 0.8977486485304143,
+ 0.7375573031788435,
+ 0.7021985858356108,
+ 0.6931155318696739,
+ 0.6712790674128556,
+ 0.5724681667349394,
+ 0.5723068152580888,
+ 0.5532308850798294,
+ 0.5487482080494408,
+ 0.5385291217857917,
+ 0.5141693283375656,
+ 0.5104472960164411,
+ 0.5098033144707058,
+ 0.4828554984461631,
+ 0.45634064879358327,
+ 0.43272108415916566,
+ 0.427312730309038,
+ 0.4191526233394532,
+ 0.41782074961444304,
+ 0.381507805809852
+ ],
+ [
+ 1.0,
+ 0.776996072380013,
+ 0.7292332970281283,
+ 0.7155561061975331,
+ 0.6870998196404438,
+ 0.6155892455424984,
+ 0.606864839511286,
+ 0.5902636874673826,
+ 0.5712217254772374,
+ 0.5610698416645239,
+ 0.5398687080403894,
+ 0.5211077061159112,
+ 0.5107674376029829,
+ 0.5046980876916762,
+ 0.4989762667716445,
+ 0.49548993423937027,
+ 0.4830605270620383,
+ 0.46042746288616254,
+ 0.45059081496124864,
+ 0.4335730641377399
+ ],
+ [
+ 0.9257727398353212,
+ 0.7511380782938324,
+ 0.751024009262826,
+ 0.7448315341972969,
+ 0.7134760551820755,
+ 0.6800124604907232,
+ 0.656522564561578,
+ 0.5764106792293393,
+ 0.558369779702014,
+ 0.5399886584540565,
+ 0.5383532878153108,
+ 0.5323484394435576,
+ 0.520424189594161,
+ 0.48588940224697197,
+ 0.44584306798975465,
+ 0.43234035076962074,
+ 0.4218178951754989,
+ 0.3806644623580111,
+ 0.3552111852291179,
+ 0.33183252979646116
+ ],
+ [
+ 0.8359886790081386,
+ 0.7386488141626866,
+ 0.7141827821552023,
+ 0.6938553023183691,
+ 0.6834004076861452,
+ 0.6832288183081515,
+ 0.6799125115960893,
+ 0.6722093107976179,
+ 0.6608454520883212,
+ 0.6535624464502042,
+ 0.6461395917562525,
+ 0.6373370242388374,
+ 0.6224880292737048,
+ 0.6074501625593222,
+ 0.6037939144818467,
+ 0.6036593972729628,
+ 0.5671527861256731,
+ 0.5523183144659303,
+ 0.5004106470271041,
+ 0.4778194007193426
+ ],
+ [
+ 0.8466523201418745,
+ 0.7766940125497964,
+ 0.7467301168638802,
+ 0.725321917347383,
+ 0.6954064291376327,
+ 0.6870504688524538,
+ 0.6749454853763226,
+ 0.6739968329035089,
+ 0.6724241678568699,
+ 0.652433782795826,
+ 0.6517281582656729,
+ 0.6390345576681893,
+ 0.6377835321069856,
+ 0.6145740219612865,
+ 0.5982568310058327,
+ 0.5372134494078324,
+ 0.5345772314298368,
+ 0.5248192024853426,
+ 0.47849728707951744,
+ 0.47578510552516806
+ ],
+ [
+ 1.0,
+ 0.9210740594171358,
+ 0.7718035014169536,
+ 0.7126540260067081,
+ 0.6467225398667023,
+ 0.5527911014655058,
+ 0.4509866810627477,
+ 0.4440509790095795,
+ 0.4285076481727811,
+ 0.41968994223618217,
+ 0.4191650009600877,
+ 0.4114808915086432,
+ 0.40853822757427716,
+ 0.4035876506862087,
+ 0.3973523931700733,
+ 0.38378382621884527,
+ 0.3795252405045695,
+ 0.37405557639983045,
+ 0.36376287999800183,
+ 0.36303826792939675
+ ],
+ [
+ 0.7899715358613317,
+ 0.7453581594279209,
+ 0.7427442087450989,
+ 0.7417909547542395,
+ 0.7374897994142381,
+ 0.6722253163093392,
+ 0.6670464477579718,
+ 0.6641705105619643,
+ 0.6195240794767956,
+ 0.6068143206544794,
+ 0.6043102420837536,
+ 0.5920407887215065,
+ 0.5876545190646074,
+ 0.5488315031071052,
+ 0.5359990429729917,
+ 0.5238805352058772,
+ 0.5131426453590435,
+ 0.4997976644215539,
+ 0.47687464215889636,
+ 0.448910287474294
+ ],
+ [
+ 0.9999999999999862,
+ 0.9820526293382901,
+ 0.9205839111334846,
+ 0.9185870484874404,
+ 0.8865529145125629,
+ 0.8210367213240263,
+ 0.8139605634051628,
+ 0.811656224781895,
+ 0.7701811551868828,
+ 0.7434915915892597,
+ 0.6903307288000805,
+ 0.6670267090196457,
+ 0.5656334351790098,
+ 0.5648177447143127,
+ 0.5578819303323391,
+ 0.5368668744625227,
+ 0.41807340054359515,
+ 0.4046736738531375,
+ 0.3821048548635032,
+ 0.36787944117143573
+ ],
+ [
+ 0.9833590955223324,
+ 0.9649681839811202,
+ 0.8440961155290643,
+ 0.8416027144518552,
+ 0.8182939358188509,
+ 0.8124692659376044,
+ 0.8087136566426625,
+ 0.7627494154161383,
+ 0.7218334118470782,
+ 0.57869483723239,
+ 0.5599403230696114,
+ 0.38301990067875963,
+ 0.34700339571114447,
+ 0.33599392258151195,
+ 0.309026696223245,
+ 0.3060952942097584,
+ 0.2956114705683183,
+ 0.28050780327964187,
+ 0.26625935615292173,
+ 0.24621135971812605
+ ],
+ [
+ 0.7993607130942629,
+ 0.7457592635315277,
+ 0.7230867735492563,
+ 0.6759170784624785,
+ 0.45137135109376564,
+ 0.43293680720967953,
+ 0.4133376773056932,
+ 0.40069912536539803,
+ 0.3797098351955729,
+ 0.3716261973776398,
+ 0.3598511725037456,
+ 0.3531102734492686,
+ 0.35029621314665776,
+ 0.349960890984185,
+ 0.3432054699220551,
+ 0.3375813591199982,
+ 0.33722112174780355,
+ 0.33317221760957144,
+ 0.33316957241336437,
+ 0.3331013107079759
+ ],
+ [
+ 0.8217401129872075,
+ 0.7641383205654013,
+ 0.7459296988747652,
+ 0.7217544485070253,
+ 0.7147138223112732,
+ 0.682156472120324,
+ 0.6387267980096086,
+ 0.6115837116866156,
+ 0.6043113748467226,
+ 0.5933291248330272,
+ 0.5546051836844134,
+ 0.5544552258061063,
+ 0.5245710194207756,
+ 0.521724626688612,
+ 0.510064194266079,
+ 0.4989573292232341,
+ 0.4985091177846723,
+ 0.4943148016449129,
+ 0.4693818538294476,
+ 0.45350333422409383
+ ],
+ [
+ 0.7851404508944432,
+ 0.7192024388266443,
+ 0.6639786935530604,
+ 0.5917548379086396,
+ 0.558738947122771,
+ 0.5143137695239839,
+ 0.5114879369556363,
+ 0.5086500487135898,
+ 0.5073570480708885,
+ 0.4906557099893468,
+ 0.4820780538386549,
+ 0.4722880108570219,
+ 0.4661286701082429,
+ 0.4604661734461678,
+ 0.4338618670679065,
+ 0.43157802299983544,
+ 0.4122776193416845,
+ 0.4086413084633581,
+ 0.3669818627859529,
+ 0.3642547324747353
+ ],
+ [
+ 0.9686241752349952,
+ 0.7888830310882966,
+ 0.7249727169179165,
+ 0.6925779310698557,
+ 0.657600133547672,
+ 0.6150386222141457,
+ 0.5889858819761955,
+ 0.5713263114581035,
+ 0.5172549920185939,
+ 0.5154444796902675,
+ 0.5118907383564846,
+ 0.5008545965486654,
+ 0.4956734891124077,
+ 0.4641243945328595,
+ 0.4558988660770064,
+ 0.4415815058099168,
+ 0.4113002978401381,
+ 0.40498175276134757,
+ 0.39372117355974773,
+ 0.3916143737229256
+ ],
+ [
+ 0.8601614428623433,
+ 0.8480601330044009,
+ 0.7908721497674391,
+ 0.6704123362594592,
+ 0.6156770491010458,
+ 0.6109017813580475,
+ 0.5898501501894912,
+ 0.5676143433211567,
+ 0.5604804405574586,
+ 0.5467138987849449,
+ 0.5390084025324926,
+ 0.5364672469871485,
+ 0.5221635226556487,
+ 0.5133086530800343,
+ 0.5128441082529621,
+ 0.5103186542411596,
+ 0.43680497555181,
+ 0.39180995863650436,
+ 0.37621968612493223,
+ 0.36039168725363535
+ ],
+ [
+ 0.959364216119315,
+ 0.8214369034217238,
+ 0.7210394031941625,
+ 0.7071515461048805,
+ 0.704837845780234,
+ 0.6904921079381694,
+ 0.6584421191885688,
+ 0.6516772882737439,
+ 0.6475039686231621,
+ 0.6147047851226212,
+ 0.5617430872743983,
+ 0.5051128719784064,
+ 0.5001978774275754,
+ 0.4967631796501588,
+ 0.49357667157093876,
+ 0.4901163828559106,
+ 0.48957251863835494,
+ 0.4791979033215533,
+ 0.47397770418578394,
+ 0.46915510349850537
+ ],
+ [
+ 0.9868213796971046,
+ 0.9175441745030266,
+ 0.7381886238195947,
+ 0.7065881391026713,
+ 0.6341424946734115,
+ 0.6269369669667478,
+ 0.6242031753995454,
+ 0.6236830083873777,
+ 0.6083339304427504,
+ 0.5745103068225179,
+ 0.5725114771426423,
+ 0.5637742858637692,
+ 0.5469124083204218,
+ 0.5234507047763846,
+ 0.5012840287441254,
+ 0.5004664212293239,
+ 0.46522541840493703,
+ 0.45653054678603167,
+ 0.45235419627690804,
+ 0.4495365846766649
+ ],
+ [
+ 0.9849256596996082,
+ 0.8757840584824338,
+ 0.7774419949137362,
+ 0.7623939323135577,
+ 0.758616709194243,
+ 0.7075351381548505,
+ 0.6743230986133231,
+ 0.6402511964331091,
+ 0.5619730407213905,
+ 0.45762234445961536,
+ 0.44277866225138957,
+ 0.42918094037353216,
+ 0.424800104648748,
+ 0.3989218482609353,
+ 0.3745850146482328,
+ 0.3652291075092638,
+ 0.34934223541742115,
+ 0.34217157996389364,
+ 0.3388513209751246,
+ 0.33249592217244694
+ ],
+ [
+ 0.8841057094015473,
+ 0.6240547300815966,
+ 0.6223160313539515,
+ 0.5882223253958823,
+ 0.5749686635967681,
+ 0.5639485951222447,
+ 0.5635758808588549,
+ 0.5550479872424698,
+ 0.5471019104381745,
+ 0.5445040187547182,
+ 0.5400871989716076,
+ 0.5120771788023474,
+ 0.5024085922758894,
+ 0.49069049540887,
+ 0.4783407626246866,
+ 0.4699556367395784,
+ 0.4692018178175108,
+ 0.46843121541092225,
+ 0.46786225544931637,
+ 0.4475688349931627
+ ],
+ [
+ 0.9392117360961412,
+ 0.7390608159103313,
+ 0.7334140930314635,
+ 0.7007169075114452,
+ 0.6782954835367436,
+ 0.6666000798293772,
+ 0.65827394323132,
+ 0.6466251772142219,
+ 0.6430679966135837,
+ 0.6245170885965567,
+ 0.6157411473017513,
+ 0.6103524826319877,
+ 0.5906930407046749,
+ 0.5693598691238847,
+ 0.5574722472078679,
+ 0.5248229221639845,
+ 0.5115628130139175,
+ 0.49599984995242696,
+ 0.45842454892464635,
+ 0.43075960203133923
+ ],
+ [
+ 0.8781967021280094,
+ 0.8298616494219739,
+ 0.8245222630988147,
+ 0.7072906460719657,
+ 0.6821517263129534,
+ 0.6730937979285133,
+ 0.6483214772639041,
+ 0.6481484250069239,
+ 0.6437349351552175,
+ 0.6251240306723383,
+ 0.6099731679561903,
+ 0.6025629795775579,
+ 0.5956655470135502,
+ 0.5933011111663977,
+ 0.5930097043649508,
+ 0.5830278761144095,
+ 0.5710945135299912,
+ 0.5254526092451324,
+ 0.5136982575885819,
+ 0.5035497198813597
+ ],
+ [
+ 0.9564133717698398,
+ 0.8951023468524486,
+ 0.7617650066229052,
+ 0.722810309449755,
+ 0.6807858682449422,
+ 0.6237130881708732,
+ 0.6202340320123033,
+ 0.6200428239464847,
+ 0.5934670567761973,
+ 0.564167075490442,
+ 0.5605783454690837,
+ 0.5345978677056153,
+ 0.5269092360110709,
+ 0.4954295369932696,
+ 0.44924651702612506,
+ 0.44146620540368364,
+ 0.4379027956837862,
+ 0.415636855380734,
+ 0.4144748022036904,
+ 0.4063090496057644
+ ],
+ [
+ 0.7976908845473137,
+ 0.7100489423365163,
+ 0.620670401050256,
+ 0.6044704838126891,
+ 0.5715831187787199,
+ 0.5303701382193905,
+ 0.5269944809519514,
+ 0.5259007281264537,
+ 0.4933708806076069,
+ 0.48970728373979766,
+ 0.47643853988073304,
+ 0.46738092619896626,
+ 0.4669509888466511,
+ 0.45994991308927724,
+ 0.4595555917606333,
+ 0.45562957862086245,
+ 0.45548722915679585,
+ 0.4482410539037005,
+ 0.44398331157791543,
+ 0.43657636902867475
+ ],
+ [
+ 0.989189701826092,
+ 0.8204095558321163,
+ 0.8011564118864359,
+ 0.763774315254722,
+ 0.7352769315596215,
+ 0.7100296259546679,
+ 0.6765548498088209,
+ 0.6526096218801598,
+ 0.6329022832504705,
+ 0.6251645223800877,
+ 0.6083287910581084,
+ 0.6015981689497002,
+ 0.5872433127466752,
+ 0.5571287457655116,
+ 0.520794024385552,
+ 0.46556149360334376,
+ 0.463095040593254,
+ 0.45212631320090074,
+ 0.4489846365750377,
+ 0.44462188263584307
+ ],
+ [
+ 0.9828516775155447,
+ 0.8139761269218837,
+ 0.6671327474847305,
+ 0.6584867602841193,
+ 0.6558696493811212,
+ 0.6289545709557693,
+ 0.6208086837247033,
+ 0.6207166800432155,
+ 0.5985610823627341,
+ 0.584112484050826,
+ 0.5813817406545156,
+ 0.5672689813886614,
+ 0.5142674913438339,
+ 0.5082957345178609,
+ 0.5031060867631734,
+ 0.5026062570785469,
+ 0.495195056545863,
+ 0.4899055239763481,
+ 0.46072941072604123,
+ 0.4453455874634902
+ ],
+ [
+ 0.8210188726434577,
+ 0.8002405432784963,
+ 0.7977958448977002,
+ 0.7462880529967313,
+ 0.7428672723972951,
+ 0.7102747129544625,
+ 0.699774132125671,
+ 0.6991868235497849,
+ 0.6859296732952114,
+ 0.6742769587023941,
+ 0.6591239048746795,
+ 0.6579596402426611,
+ 0.6224545224355527,
+ 0.6214561387412674,
+ 0.6090547010932781,
+ 0.6085307749278834,
+ 0.5923125792203292,
+ 0.5824376574064218,
+ 0.5688168032055814,
+ 0.5637059722545152
+ ],
+ [
+ 0.9872653526958806,
+ 0.7336196098212613,
+ 0.6761484434967469,
+ 0.6693826218640463,
+ 0.659237607376624,
+ 0.6343068852708872,
+ 0.5605836093802328,
+ 0.5522726201099342,
+ 0.5486386710029063,
+ 0.5291956111979129,
+ 0.5226763206268505,
+ 0.5045601383455446,
+ 0.46998857528136906,
+ 0.46470959315360527,
+ 0.45678142884146244,
+ 0.4082711548192862,
+ 0.39540998949890505,
+ 0.3725584852198288,
+ 0.3617916386713938,
+ 0.3594160833482854
+ ],
+ [
+ 0.9204061541163123,
+ 0.8385152027221716,
+ 0.7196959367056845,
+ 0.6741376588162787,
+ 0.6699033583741666,
+ 0.6516759171995534,
+ 0.5614833566962862,
+ 0.5245812365808131,
+ 0.5169561747159226,
+ 0.5028059763986468,
+ 0.4989009330458648,
+ 0.4743436529499614,
+ 0.46913901503972455,
+ 0.4608922341474941,
+ 0.4549903783748366,
+ 0.43501306074607793,
+ 0.4205695848725548,
+ 0.4197046474139653,
+ 0.40179664781644703,
+ 0.3788083290801033
+ ],
+ [
+ 0.8582346149318103,
+ 0.7656070737120828,
+ 0.728181585207486,
+ 0.6827900323927972,
+ 0.6442410756941008,
+ 0.6379235095003818,
+ 0.6230925050998276,
+ 0.613569177472802,
+ 0.5911013525161175,
+ 0.538441270734505,
+ 0.5202938163446624,
+ 0.48639934634308696,
+ 0.4774616024839404,
+ 0.4414144271109724,
+ 0.4352905613138405,
+ 0.40552555165059256,
+ 0.40479963572125544,
+ 0.4022065522645599,
+ 0.3810511560752579,
+ 0.36890209835028376
+ ],
+ [
+ 0.8473128341587903,
+ 0.84241465892585,
+ 0.8007151304735541,
+ 0.7880061367242762,
+ 0.7244905126721215,
+ 0.7129475820491857,
+ 0.7028432873675292,
+ 0.6777589399141501,
+ 0.6654666503775029,
+ 0.6593054223955563,
+ 0.645197482014016,
+ 0.6221676609303001,
+ 0.5974162814229814,
+ 0.5837226195332951,
+ 0.5813831494647695,
+ 0.5795945317393466,
+ 0.5731616014886376,
+ 0.5477217996115724,
+ 0.544205917850156,
+ 0.5374549053204201
+ ],
+ [
+ 0.8531757012596709,
+ 0.794917032430729,
+ 0.7942011411684733,
+ 0.6958872772459642,
+ 0.6810999555316783,
+ 0.6609078499638472,
+ 0.6452078979430733,
+ 0.6218297663027237,
+ 0.6029046091694914,
+ 0.5995447815256375,
+ 0.5630915168034096,
+ 0.5594825293054496,
+ 0.5321838443274781,
+ 0.5244629847226632,
+ 0.4874698621960831,
+ 0.4828853910453908,
+ 0.4680483685470791,
+ 0.4459760053860486,
+ 0.43843564219455733,
+ 0.4228665859869971
+ ],
+ [
+ 1.0,
+ 0.8890659149462619,
+ 0.8142848069251842,
+ 0.7924109827832605,
+ 0.7879052992516795,
+ 0.7555017878286995,
+ 0.7240380650638407,
+ 0.7037570633154415,
+ 0.6974537600247681,
+ 0.6968979138877464,
+ 0.6557693764122812,
+ 0.6490387025314082,
+ 0.6315311230775459,
+ 0.6158957529325002,
+ 0.5934007932274611,
+ 0.5925896160233773,
+ 0.5925761208755953,
+ 0.5827506375025415,
+ 0.5780784915435713,
+ 0.5325217274276776
+ ],
+ [
+ 1.0,
+ 0.7713569376704056,
+ 0.7194485518963165,
+ 0.66500481111418,
+ 0.6361766285766219,
+ 0.6300389900338007,
+ 0.5819822857212943,
+ 0.5794434741323364,
+ 0.5449653860205044,
+ 0.532962943558598,
+ 0.48825563461801963,
+ 0.46975763507306845,
+ 0.4624315830271425,
+ 0.45695700484228413,
+ 0.4489445463945036,
+ 0.43508522253544096,
+ 0.4333193175165605,
+ 0.4143533062017221,
+ 0.38694429531607816,
+ 0.3697851057115852
+ ],
+ [
+ 0.9999999999999999,
+ 0.5235559759266374,
+ 0.5235036171781217,
+ 0.49735096799388656,
+ 0.4836043883321518,
+ 0.4800377663959302,
+ 0.4800230411325287,
+ 0.4671131344857623,
+ 0.4592993734350256,
+ 0.4509222936671192,
+ 0.4320564388053985,
+ 0.4319202206078141,
+ 0.42138495489407657,
+ 0.41409318142936347,
+ 0.4121333095422486,
+ 0.4083538777588761,
+ 0.4075628747928454,
+ 0.40539106667513247,
+ 0.3968274386681495,
+ 0.3897982551320826
+ ],
+ [
+ 0.99366601688024,
+ 0.9610854098977012,
+ 0.8371209449283784,
+ 0.8202255514630707,
+ 0.7715000843390079,
+ 0.7458003803196406,
+ 0.7140137173082044,
+ 0.7069066940544795,
+ 0.685479843681016,
+ 0.6419439944495062,
+ 0.6392199655697759,
+ 0.6351138443355266,
+ 0.625492324275808,
+ 0.5646591203934282,
+ 0.5370331578875598,
+ 0.5096271344546807,
+ 0.4446945120161853,
+ 0.41646654487324575,
+ 0.3904176543304678,
+ 0.3692002799165097
+ ],
+ [
+ 0.9040439665711362,
+ 0.7949363144905712,
+ 0.732125695248272,
+ 0.703385380562288,
+ 0.6912777408827366,
+ 0.6426601788614458,
+ 0.6332616323526982,
+ 0.6329258739128004,
+ 0.6226752456265772,
+ 0.5690716784451715,
+ 0.5451589771413363,
+ 0.519291651804374,
+ 0.5086921661427186,
+ 0.5024958624628952,
+ 0.4945077646534175,
+ 0.4827845550118808,
+ 0.4738787191798284,
+ 0.467943639516641,
+ 0.4409397770387916,
+ 0.4259922491208675
+ ],
+ [
+ 1.0,
+ 0.9976207902750002,
+ 0.9884571585167498,
+ 0.8703344686128177,
+ 0.8279703703857557,
+ 0.8010773828375088,
+ 0.7355864110770655,
+ 0.696020638711723,
+ 0.6107975072603308,
+ 0.6028153622289619,
+ 0.5988198867062543,
+ 0.5880210629127026,
+ 0.5458784566263828,
+ 0.5259207559551063,
+ 0.45899782896433294,
+ 0.4529298086560604,
+ 0.45128354014319066,
+ 0.4171966267183494,
+ 0.3916165147095798,
+ 0.3782920354578563
+ ],
+ [
+ 0.8677792016207084,
+ 0.839428058157005,
+ 0.7262595151910729,
+ 0.7071359978796572,
+ 0.6253636719444862,
+ 0.6185952977562096,
+ 0.5931563839929431,
+ 0.5675160949008518,
+ 0.49644301821878256,
+ 0.47835500924539937,
+ 0.4761277258615819,
+ 0.46443945963918043,
+ 0.43861729316622644,
+ 0.35467610910355735,
+ 0.34570106607144224,
+ 0.3446646260427166,
+ 0.32596990959462596,
+ 0.32177519241371316,
+ 0.3006730873784835,
+ 0.30029773624928996
+ ],
+ [
+ 0.7168952125562771,
+ 0.6379633408873824,
+ 0.5800845780004963,
+ 0.5513334124238154,
+ 0.5402789032575984,
+ 0.5304846695126583,
+ 0.526270961602676,
+ 0.5147849230479208,
+ 0.5027277677546145,
+ 0.4944715092653595,
+ 0.490035525676819,
+ 0.4874684734913607,
+ 0.48132542149561625,
+ 0.4746699682356097,
+ 0.4656613122123885,
+ 0.46091316847389896,
+ 0.46019837323869095,
+ 0.45235058494660674,
+ 0.4431268394237138,
+ 0.42223033643376695
+ ],
+ [
+ 0.9369076262003082,
+ 0.9270293097005213,
+ 0.9049869818962879,
+ 0.6873021106284394,
+ 0.5594284117781323,
+ 0.5493795329707674,
+ 0.5129020268603117,
+ 0.4627282176125211,
+ 0.4425249611500462,
+ 0.41872617497818027,
+ 0.4115433025581156,
+ 0.40802101679216907,
+ 0.402265084593936,
+ 0.39331351853365404,
+ 0.38805148838198045,
+ 0.38656165433654877,
+ 0.38164080641741344,
+ 0.3755697373747583,
+ 0.3731709853503923,
+ 0.37141731813502465
+ ],
+ [
+ 0.9562426999772096,
+ 0.9480936500060776,
+ 0.865369498301259,
+ 0.8597694608352209,
+ 0.7632693855987647,
+ 0.7546966573826539,
+ 0.7292419443453978,
+ 0.7193840169246093,
+ 0.6804086598009546,
+ 0.6765092160145083,
+ 0.642971842750271,
+ 0.6305241777787557,
+ 0.5546064868122456,
+ 0.5508098560177818,
+ 0.5118371591149693,
+ 0.5116667101732275,
+ 0.45235303345319744,
+ 0.44978396945610366,
+ 0.42385522378206847,
+ 0.41477583185534495
+ ],
+ [
+ 0.7054225239148643,
+ 0.613561445708005,
+ 0.5934857604697545,
+ 0.5753181025100786,
+ 0.5589972418041853,
+ 0.5562092915486466,
+ 0.5158157981274436,
+ 0.5070249494944407,
+ 0.5067877605916293,
+ 0.5067049136122218,
+ 0.4913961299663636,
+ 0.49036382166140746,
+ 0.46087502128216407,
+ 0.46081897963409313,
+ 0.4567929950984727,
+ 0.450632150813832,
+ 0.4467734293434983,
+ 0.435105985710749,
+ 0.4291456625794856,
+ 0.42298159894644805
+ ],
+ [
+ 0.7221482032966917,
+ 0.7174020405749761,
+ 0.6958396852482389,
+ 0.6161890574309423,
+ 0.6140972713981054,
+ 0.5997203146809941,
+ 0.5872369166773539,
+ 0.5837766791083471,
+ 0.5175752222888701,
+ 0.5148372755075559,
+ 0.4955257292337817,
+ 0.49524094961954107,
+ 0.4872364982678197,
+ 0.4851485726635095,
+ 0.47960526571857914,
+ 0.4389344651314835,
+ 0.43865434841308304,
+ 0.4147710409909028,
+ 0.4087073291573754,
+ 0.40220704611643865
+ ],
+ [
+ 0.8577736796627262,
+ 0.8528662835204766,
+ 0.8158768621190735,
+ 0.7888333115834354,
+ 0.7763124840867126,
+ 0.7544284280166998,
+ 0.6966887577143193,
+ 0.6667934953178498,
+ 0.6522207326978646,
+ 0.6514675254703243,
+ 0.6511096391367986,
+ 0.641908660903763,
+ 0.6406491677025274,
+ 0.6377118427503342,
+ 0.6318991222037095,
+ 0.5888963924585656,
+ 0.5706728712907125,
+ 0.5562835987784357,
+ 0.5412700487674278,
+ 0.5307754353240817
+ ],
+ [
+ 0.7636779278168733,
+ 0.7425330101265698,
+ 0.7323355906009112,
+ 0.70533510356943,
+ 0.6888227532810772,
+ 0.6805087964841077,
+ 0.6798426729932346,
+ 0.6668925808841106,
+ 0.6644345877522191,
+ 0.6443921440859492,
+ 0.6171066491293755,
+ 0.5952303361693816,
+ 0.5649406713850357,
+ 0.5480729260274831,
+ 0.5363879810516046,
+ 0.49994908702422325,
+ 0.4998714990731036,
+ 0.4516191328226835,
+ 0.4425327633280274,
+ 0.42538361906186273
+ ],
+ [
+ 0.7109571866746105,
+ 0.7012944824370103,
+ 0.6205708635547251,
+ 0.5866819186494274,
+ 0.5670059734095064,
+ 0.5669990472156643,
+ 0.5353562788161693,
+ 0.5345793272099638,
+ 0.5341214219847883,
+ 0.5215458036556362,
+ 0.5045614127904917,
+ 0.4724797571540699,
+ 0.4612830256180373,
+ 0.44493962811237875,
+ 0.43860575535711177,
+ 0.4044530135912712,
+ 0.3884370162069453,
+ 0.38079766401005,
+ 0.3771186698403386,
+ 0.37359158645440677
+ ],
+ [
+ 0.6853805826771084,
+ 0.6687041117274984,
+ 0.6684991635482768,
+ 0.6464096034472937,
+ 0.6357717928395874,
+ 0.5986885326933639,
+ 0.59823255594593,
+ 0.556908735869317,
+ 0.5548305994763912,
+ 0.547776919005051,
+ 0.5028824779719551,
+ 0.48298623401131213,
+ 0.47804608398496695,
+ 0.4395748771030753,
+ 0.41850120741892954,
+ 0.4050045354064501,
+ 0.39891751007859744,
+ 0.3951310756723293,
+ 0.3925452444032307,
+ 0.3740197401845021
+ ],
+ [
+ 0.9512700649776967,
+ 0.7606927286972489,
+ 0.7020589797934949,
+ 0.6928951728230691,
+ 0.6621723313470932,
+ 0.6499399976761323,
+ 0.628924147933634,
+ 0.617149580556892,
+ 0.617080310258054,
+ 0.5846829504853714,
+ 0.5690187612744644,
+ 0.5664722455589932,
+ 0.5662583327055282,
+ 0.5660340968524957,
+ 0.5628643117355994,
+ 0.5460427897586921,
+ 0.5219443978770012,
+ 0.48388423752023,
+ 0.4675321118021427,
+ 0.4582601200515076
+ ],
+ [
+ 0.8317039430707823,
+ 0.8136209394763312,
+ 0.7767628307647955,
+ 0.737644500270534,
+ 0.7145795474807637,
+ 0.6934648176484323,
+ 0.6719033076258639,
+ 0.6476759974978309,
+ 0.6313062059697021,
+ 0.6072430417008134,
+ 0.6047639097326454,
+ 0.5657240636690404,
+ 0.5179963340735073,
+ 0.5047176591382162,
+ 0.45265675779740056,
+ 0.4325353651450415,
+ 0.42802341974300323,
+ 0.40725913210732323,
+ 0.3883023094676123,
+ 0.3793796134776233
+ ],
+ [
+ 1.0,
+ 0.9750002513161362,
+ 0.5508081609477589,
+ 0.5496373533779133,
+ 0.5351803074949033,
+ 0.5255127184828976,
+ 0.5251015529302001,
+ 0.5206597946446216,
+ 0.5133684478682093,
+ 0.5111320036658815,
+ 0.508863686561429,
+ 0.503991756720343,
+ 0.5035823454339383,
+ 0.5013162973265883,
+ 0.49732336804183463,
+ 0.49661198588386646,
+ 0.49648576479224205,
+ 0.49242834803756935,
+ 0.49177238022905756,
+ 0.4896244116067077
+ ],
+ [
+ 0.795379798161401,
+ 0.6604239851589153,
+ 0.6456759784562063,
+ 0.6217162111091382,
+ 0.6087755932032047,
+ 0.5456726457892247,
+ 0.5367367838096413,
+ 0.5344723031784744,
+ 0.48725032231042414,
+ 0.48317408415324437,
+ 0.4699214957823981,
+ 0.4649467067359732,
+ 0.4563758211494188,
+ 0.44521583833266665,
+ 0.4281358976230961,
+ 0.405733993795533,
+ 0.4032104311069263,
+ 0.39909051564713244,
+ 0.38898970271373423,
+ 0.3776806315978466
+ ],
+ [
+ 0.7300965603745259,
+ 0.7229879106199323,
+ 0.6154664597781415,
+ 0.6096715517644805,
+ 0.6051861681726922,
+ 0.5876816782447016,
+ 0.5422748718630117,
+ 0.5417412558613794,
+ 0.49445464232897157,
+ 0.4929418125877225,
+ 0.4811602031509101,
+ 0.47433755326112204,
+ 0.46764496480518614,
+ 0.4540079603223851,
+ 0.4495855712084017,
+ 0.44935397351621986,
+ 0.43482578979079844,
+ 0.4281300040130104,
+ 0.4276702090913632,
+ 0.4189559165715423
+ ],
+ [
+ 0.7880832582797124,
+ 0.7782291860198753,
+ 0.6925274907841223,
+ 0.6746390153034312,
+ 0.5950903435398265,
+ 0.5775191738270643,
+ 0.561513701556299,
+ 0.5245765521402012,
+ 0.5210140275770583,
+ 0.5149623293372054,
+ 0.5002112098594176,
+ 0.4726353003136877,
+ 0.42028536096620883,
+ 0.41650342676627095,
+ 0.4098464263326895,
+ 0.40841615603723747,
+ 0.4065038435648436,
+ 0.4056704228120892,
+ 0.39544848063949567,
+ 0.3830543407485796
+ ],
+ [
+ 0.9547815623597212,
+ 0.9528696184604452,
+ 0.9121661456972114,
+ 0.8916156370818972,
+ 0.8778265950853591,
+ 0.8520468849206309,
+ 0.8238715678104831,
+ 0.7518925118820337,
+ 0.7330281527081266,
+ 0.6567748270894076,
+ 0.6381969548962982,
+ 0.6127977361372825,
+ 0.5688735978408919,
+ 0.5682892217021214,
+ 0.5609636488003894,
+ 0.5606015020162167,
+ 0.5542960866056769,
+ 0.5426212410336964,
+ 0.41846640676601476,
+ 0.3546463796617722
+ ],
+ [
+ 0.996629057474804,
+ 0.8339992749742996,
+ 0.8046663853210406,
+ 0.7911638768102557,
+ 0.7558434292497652,
+ 0.6818824662403612,
+ 0.6665502900046355,
+ 0.650713609116856,
+ 0.6404777472403086,
+ 0.6309187441444943,
+ 0.6076622002854308,
+ 0.5939710595190151,
+ 0.5648767894524616,
+ 0.5261813207393812,
+ 0.5231601038716567,
+ 0.5055921672494751,
+ 0.4656191778519728,
+ 0.4636978025265016,
+ 0.45750372759044194,
+ 0.4383507689621662
+ ],
+ [
+ 1.0,
+ 0.9513390042873007,
+ 0.8929813529578636,
+ 0.8316762772069038,
+ 0.8046347799988465,
+ 0.7969216797964718,
+ 0.7556018356486544,
+ 0.696440921949799,
+ 0.6932355837309272,
+ 0.6490358188986497,
+ 0.6350118202130257,
+ 0.6203597625523921,
+ 0.6172977446759287,
+ 0.590297304978102,
+ 0.5799223280882563,
+ 0.5113949101530534,
+ 0.5107885679965382,
+ 0.5058349776119369,
+ 0.43958316217256616,
+ 0.36787944117144233
+ ],
+ [
+ 0.8983433890799913,
+ 0.8955519885186047,
+ 0.8546514564165433,
+ 0.8438252713699101,
+ 0.7761493998186316,
+ 0.7174707251771103,
+ 0.6999628432997105,
+ 0.6978618885756916,
+ 0.6124645293209249,
+ 0.6061391953714239,
+ 0.5979768958693938,
+ 0.5769421980676644,
+ 0.5615801763316337,
+ 0.5219555118214249,
+ 0.47159657568260527,
+ 0.4557719946854897,
+ 0.41559198573381034,
+ 0.4096781723421775,
+ 0.39369001755368044,
+ 0.3904796563877565
+ ],
+ [
+ 0.9663157336218238,
+ 0.7982810484114077,
+ 0.6273379784622313,
+ 0.622251740431959,
+ 0.6149719296834908,
+ 0.6006301902343676,
+ 0.5946444609722876,
+ 0.557344027431682,
+ 0.556854171515197,
+ 0.5303846647249153,
+ 0.5089208336468913,
+ 0.508461977664252,
+ 0.5040204969930981,
+ 0.49711070231145593,
+ 0.4809184127741197,
+ 0.4775602800015571,
+ 0.46946726454269994,
+ 0.4676657705753713,
+ 0.46658022149987927,
+ 0.4452857754668236
+ ],
+ [
+ 0.826888839243275,
+ 0.8058411578901717,
+ 0.7155470588662385,
+ 0.6991961141356413,
+ 0.6891168401652032,
+ 0.627114805418456,
+ 0.5540534640685292,
+ 0.5382726449438856,
+ 0.5338381252234025,
+ 0.5209960433961789,
+ 0.5147977111523621,
+ 0.513366283083846,
+ 0.5027094516697691,
+ 0.4865815056217814,
+ 0.4841832370881199,
+ 0.46193297192898963,
+ 0.4492186298343014,
+ 0.4360789414603446,
+ 0.4304049274062821,
+ 0.39956343989323767
+ ],
+ [
+ 0.8657092245250892,
+ 0.8044012716659066,
+ 0.731099214442671,
+ 0.6811818785789884,
+ 0.6566778467871355,
+ 0.6434542928205196,
+ 0.6411125079244435,
+ 0.6136435419136131,
+ 0.6092380925812666,
+ 0.5964058617140748,
+ 0.5799806947145766,
+ 0.5661017432686115,
+ 0.5115319704833439,
+ 0.5090447151822575,
+ 0.5033414630140005,
+ 0.4566175518459798,
+ 0.4548303564534285,
+ 0.42752453836461174,
+ 0.42143078011705765,
+ 0.4173845619117032
+ ],
+ [
+ 0.7506930011996183,
+ 0.5943506147323219,
+ 0.5760757788235691,
+ 0.573410089388888,
+ 0.5539893402251881,
+ 0.5506464674872817,
+ 0.5382404359144479,
+ 0.5264153443635199,
+ 0.509480123754785,
+ 0.4892436947534713,
+ 0.4872358997533423,
+ 0.4872114971395789,
+ 0.4820899184030239,
+ 0.469515394827984,
+ 0.4681835065895132,
+ 0.45455480007198934,
+ 0.4520119055566518,
+ 0.4466492802978359,
+ 0.4306387909837267,
+ 0.4282376372202654
+ ],
+ [
+ 0.946880852955026,
+ 0.7823729303037159,
+ 0.7626128632560166,
+ 0.6556723448172892,
+ 0.5975979267317285,
+ 0.5492043980200376,
+ 0.5339505701889784,
+ 0.5335886513221211,
+ 0.5281784252816799,
+ 0.5233275185913857,
+ 0.48818721256680697,
+ 0.4747912965000553,
+ 0.4668213504226617,
+ 0.45923103120462616,
+ 0.4322025029754225,
+ 0.41675504355931603,
+ 0.3983719800127928,
+ 0.39743086776457703,
+ 0.39694747492811666,
+ 0.389854255305567
+ ],
+ [
+ 0.7930922024020577,
+ 0.7837002075275782,
+ 0.7369287685625824,
+ 0.73125780844617,
+ 0.6381146731913103,
+ 0.6351491454881452,
+ 0.6040022265569125,
+ 0.6002050803967334,
+ 0.5943573083525817,
+ 0.5825258980205359,
+ 0.561942109887769,
+ 0.5545065019832504,
+ 0.5282553400778189,
+ 0.5107463096242508,
+ 0.5103830916369517,
+ 0.5091672838558848,
+ 0.47930388218925346,
+ 0.47920102227759676,
+ 0.4557074745961967,
+ 0.43175977392629206
+ ],
+ [
+ 0.745620894499174,
+ 0.7149559741472473,
+ 0.7012351744490282,
+ 0.6741808972637933,
+ 0.6175925949529362,
+ 0.6000622766580841,
+ 0.5990155673365983,
+ 0.5872253875678884,
+ 0.5786839641241659,
+ 0.5691776600397168,
+ 0.5654090916847345,
+ 0.555169779331831,
+ 0.5491759901932154,
+ 0.5400664354354431,
+ 0.53490263400428,
+ 0.44014627134558576,
+ 0.415832067009151,
+ 0.4156630357421761,
+ 0.41520081077462634,
+ 0.4044316876748043
+ ],
+ [
+ 0.986460585852835,
+ 0.7297996995059828,
+ 0.6232075861707946,
+ 0.5950627554349424,
+ 0.5844534896045653,
+ 0.5800111846054203,
+ 0.5632009054044465,
+ 0.5399330513232306,
+ 0.4949267245361343,
+ 0.4905332334231375,
+ 0.4871777085704705,
+ 0.47747789938105656,
+ 0.4722261035315207,
+ 0.4658199003867349,
+ 0.45111733893793504,
+ 0.45097420685149225,
+ 0.4462894932732877,
+ 0.4326004352378193,
+ 0.42988277819065607,
+ 0.42452923467626935
+ ],
+ [
+ 0.9699718529926737,
+ 0.9157997772430632,
+ 0.9093043738058236,
+ 0.9055968210696229,
+ 0.8693622542645629,
+ 0.8568990647320973,
+ 0.8093002481777811,
+ 0.6248560623863147,
+ 0.5457355008260504,
+ 0.5349253987384163,
+ 0.50733781496628,
+ 0.503559266029048,
+ 0.48694392892919175,
+ 0.4769793476745494,
+ 0.45226611636038194,
+ 0.4441742347876929,
+ 0.4330543524149938,
+ 0.4186699288818404,
+ 0.399193180127311,
+ 0.3792032022284386
+ ],
+ [
+ 0.8383589596439924,
+ 0.8236979635908208,
+ 0.7905927275515146,
+ 0.7528815082949195,
+ 0.6619749598082412,
+ 0.614743200312656,
+ 0.5644806400018398,
+ 0.5434510110328488,
+ 0.51092211912584,
+ 0.4968118017847394,
+ 0.4932264458853843,
+ 0.47804168570834726,
+ 0.4701861756907835,
+ 0.4427061763849891,
+ 0.44063081814834837,
+ 0.42386598008941345,
+ 0.4162183638953123,
+ 0.4048805673332743,
+ 0.3921189237488288,
+ 0.37649115816654255
+ ],
+ [
+ 0.9999999999999998,
+ 0.696336128136047,
+ 0.6090705485682205,
+ 0.58580079144771,
+ 0.5724882428360655,
+ 0.5652221012699417,
+ 0.5126188177893904,
+ 0.5077165220397382,
+ 0.5064219135681923,
+ 0.5014276248209256,
+ 0.48473489369603273,
+ 0.4597039205169111,
+ 0.45021899026204537,
+ 0.4417548476754588,
+ 0.4385985978631721,
+ 0.4385617764776845,
+ 0.4373404749069372,
+ 0.43403600710784684,
+ 0.4256944759676772,
+ 0.3822942633046687
+ ],
+ [
+ 0.9176850307042997,
+ 0.8729827143312758,
+ 0.7638387329630127,
+ 0.6345073297136123,
+ 0.6318555045583969,
+ 0.610931549523463,
+ 0.5857028172466376,
+ 0.5700602836434261,
+ 0.5693948847986748,
+ 0.5689738282314195,
+ 0.5508916294470921,
+ 0.5205081112797031,
+ 0.49650332625186755,
+ 0.4855807124968774,
+ 0.48368276200281196,
+ 0.48361084028970175,
+ 0.4822739618312113,
+ 0.47828337055705594,
+ 0.46922368893745475,
+ 0.45529782015923326
+ ],
+ [
+ 0.7329774890541345,
+ 0.7301465927434699,
+ 0.6700829055630453,
+ 0.649498605770579,
+ 0.6167630131923407,
+ 0.5979723610079575,
+ 0.5898401900328664,
+ 0.5497189755408417,
+ 0.524724907103921,
+ 0.5165823623853644,
+ 0.5157684612431128,
+ 0.5133682284422065,
+ 0.509865657844712,
+ 0.48500745117810595,
+ 0.4819975580301658,
+ 0.47955509117376954,
+ 0.44805717725939764,
+ 0.43498039907715125,
+ 0.40411020798843134,
+ 0.391465194614572
+ ],
+ [
+ 0.8709776197944792,
+ 0.8512558250415079,
+ 0.7298024515187126,
+ 0.6819449414700238,
+ 0.6736427683739193,
+ 0.6616693311040351,
+ 0.6482130116913101,
+ 0.6038967576732506,
+ 0.5685246699007643,
+ 0.5207659257038177,
+ 0.47187915741014413,
+ 0.47070942730231685,
+ 0.4464901186519996,
+ 0.4415898863005738,
+ 0.44123735976283296,
+ 0.4318665856214724,
+ 0.40994615696419867,
+ 0.4043340353993301,
+ 0.4005647480134473,
+ 0.3720907143493714
+ ],
+ [
+ 0.8096146722018863,
+ 0.7843471770075147,
+ 0.7372367374316555,
+ 0.7126629651715171,
+ 0.7080849758175688,
+ 0.6805946500125959,
+ 0.6777160775345352,
+ 0.6725949566293508,
+ 0.6086516418259532,
+ 0.6073135538283139,
+ 0.5989533383458995,
+ 0.5596719731939668,
+ 0.5594445556997247,
+ 0.55294171856163,
+ 0.5470683991836103,
+ 0.4970953074064648,
+ 0.49512938994391675,
+ 0.49010784624914583,
+ 0.48566515153855705,
+ 0.4779334816343344
+ ],
+ [
+ 1.0,
+ 0.9838896203273296,
+ 0.8949336552786296,
+ 0.6988628740342984,
+ 0.6757390181797787,
+ 0.6587155118249732,
+ 0.6293427668360644,
+ 0.5857907006899961,
+ 0.5842119030067254,
+ 0.5681763121274761,
+ 0.503712527985536,
+ 0.4644154401461472,
+ 0.4577300365859868,
+ 0.43551264066559775,
+ 0.4292766376537857,
+ 0.4172298350178547,
+ 0.40328605588754074,
+ 0.3976627951325517,
+ 0.3789453469504551,
+ 0.36787944117144233
+ ],
+ [
+ 1.0,
+ 0.6817925605900433,
+ 0.6779168423069792,
+ 0.6478752666277072,
+ 0.5720699629622318,
+ 0.5654777266482738,
+ 0.5652707242461105,
+ 0.556052622680798,
+ 0.5232552478155286,
+ 0.46243978598921626,
+ 0.4476944579992861,
+ 0.40394100885053075,
+ 0.396247135223757,
+ 0.39479829519815346,
+ 0.38606340342304557,
+ 0.38498522217411774,
+ 0.38460292114403116,
+ 0.37690849294773027,
+ 0.36797949370911204,
+ 0.365610683812408
+ ],
+ [
+ 0.9865167348486958,
+ 0.9222706121043858,
+ 0.8805797905748749,
+ 0.8528324662448963,
+ 0.8301879772883931,
+ 0.7923823725163794,
+ 0.7458026627255595,
+ 0.7372372576677085,
+ 0.7034369557644152,
+ 0.701739815182685,
+ 0.690222240897473,
+ 0.6712677556185674,
+ 0.6546605957951848,
+ 0.6347907414465941,
+ 0.6053190056748876,
+ 0.5970192224928623,
+ 0.5587092045524928,
+ 0.5441248848968983,
+ 0.4834497069873417,
+ 0.4245967891522051
+ ],
+ [
+ 0.7645402384023574,
+ 0.7514665269458032,
+ 0.717254404274433,
+ 0.7097257780503179,
+ 0.6923273233996217,
+ 0.6413157236709676,
+ 0.6119903057113425,
+ 0.5913742778896904,
+ 0.5879492469284628,
+ 0.5827270726221939,
+ 0.5786561070384073,
+ 0.5413040875628274,
+ 0.540106449306315,
+ 0.5264366221853302,
+ 0.5230797418988103,
+ 0.4944628547128221,
+ 0.48413136856989447,
+ 0.47375518802942224,
+ 0.4603417340614959,
+ 0.4536231053262685
+ ],
+ [
+ 1.0,
+ 0.7673386374456954,
+ 0.7418378663269063,
+ 0.7218021974564959,
+ 0.6969320222713193,
+ 0.6953576046855516,
+ 0.6500643894035136,
+ 0.606299567291757,
+ 0.5998312819394014,
+ 0.583164995167192,
+ 0.5831190191150832,
+ 0.5808667220834303,
+ 0.5647357138488754,
+ 0.5545459480982686,
+ 0.5460571975742559,
+ 0.5204257460373113,
+ 0.5204071571357564,
+ 0.519000632606651,
+ 0.5141296709991816,
+ 0.49308288610325435
+ ],
+ [
+ 0.8844807498442346,
+ 0.6403558672135559,
+ 0.6184505327919482,
+ 0.6143497868455834,
+ 0.6015737906998726,
+ 0.5565675485834294,
+ 0.5536172122637449,
+ 0.5168640543905731,
+ 0.501750879993955,
+ 0.459937035690574,
+ 0.4541483129912568,
+ 0.4521101444730822,
+ 0.4517231826493967,
+ 0.44238929413827877,
+ 0.43195778122465084,
+ 0.4041239976023119,
+ 0.40034198634413165,
+ 0.39661405724834464,
+ 0.39595429770885215,
+ 0.39587656984718955
+ ],
+ [
+ 1.0,
+ 0.7469650039189328,
+ 0.7449366048114396,
+ 0.7349090476651793,
+ 0.6969148912564462,
+ 0.6478106008461999,
+ 0.615631297910706,
+ 0.5902548370690756,
+ 0.5836012434668004,
+ 0.5667132389008469,
+ 0.5212951866746489,
+ 0.510733595702466,
+ 0.5052429161223031,
+ 0.44050525243217487,
+ 0.42787421573641043,
+ 0.4084961354251252,
+ 0.3977471789509264,
+ 0.38465189558423923,
+ 0.36998162628417336,
+ 0.36787944117144233
+ ],
+ [
+ 0.9530998094831998,
+ 0.864084465888946,
+ 0.8628921216678008,
+ 0.7993304636535394,
+ 0.7447108976579024,
+ 0.7384543327222118,
+ 0.7013769919436101,
+ 0.6314623050785002,
+ 0.5677043523057977,
+ 0.5478221495618558,
+ 0.4840343257829956,
+ 0.3082474634790088,
+ 0.3051085175209155,
+ 0.28931537714292255,
+ 0.28634904473685213,
+ 0.28481456566626306,
+ 0.2612899129210732,
+ 0.25925630777110276,
+ 0.25764274846132135,
+ 0.2505818298405793
+ ],
+ [
+ 0.8598342921668745,
+ 0.7918191333764408,
+ 0.7719310119922844,
+ 0.766901051065747,
+ 0.7458973530322257,
+ 0.6617130619985496,
+ 0.6070822334058448,
+ 0.5841164015709592,
+ 0.5646843273921978,
+ 0.5396126844312175,
+ 0.5219245829287746,
+ 0.5079237098287624,
+ 0.4913883475966283,
+ 0.4712207794990577,
+ 0.45843536935834617,
+ 0.43567820818096625,
+ 0.4311053307961846,
+ 0.41116290864044,
+ 0.3970614106539884,
+ 0.38626454626988893
+ ],
+ [
+ 0.758167583300769,
+ 0.7578661328603511,
+ 0.6138647075969379,
+ 0.601251157119558,
+ 0.5980668490648394,
+ 0.5623185173184182,
+ 0.5514165063557321,
+ 0.5350486641992737,
+ 0.5139597218618334,
+ 0.4958905157546011,
+ 0.47305368412543447,
+ 0.47248869547432026,
+ 0.46893254323120587,
+ 0.46352451789346993,
+ 0.4572191967627724,
+ 0.4524071645144771,
+ 0.4501164262910009,
+ 0.4468980907013842,
+ 0.44569445575640754,
+ 0.4232059236045266
+ ],
+ [
+ 0.7590187159131663,
+ 0.718101030972431,
+ 0.6376497651804132,
+ 0.5715710983616822,
+ 0.5666718700753096,
+ 0.5192071122464055,
+ 0.5115497783144356,
+ 0.5013590762575041,
+ 0.4984140743296604,
+ 0.49269695599548124,
+ 0.4880357302142146,
+ 0.4444063540410737,
+ 0.4305260999156332,
+ 0.38490291539092003,
+ 0.3682931656016695,
+ 0.3648763881480054,
+ 0.34753570969209163,
+ 0.3298852650784976,
+ 0.3154572638343515,
+ 0.29054774211314993
+ ],
+ [
+ 0.9091077193896865,
+ 0.8624721230814365,
+ 0.7355820606822608,
+ 0.7304802259788827,
+ 0.7071574183448305,
+ 0.7015556057290904,
+ 0.6886578461185973,
+ 0.5937934263056628,
+ 0.5926499357025784,
+ 0.5902375415777616,
+ 0.5862584093898642,
+ 0.5720729216338079,
+ 0.5714151086543402,
+ 0.5514462085649559,
+ 0.5285084206349404,
+ 0.5221243621036524,
+ 0.4879696477411864,
+ 0.4855559988783775,
+ 0.4557952361399239,
+ 0.45071962841961166
+ ],
+ [
+ 0.7905437801661309,
+ 0.7747686933564797,
+ 0.699272997025455,
+ 0.6770365520617427,
+ 0.6358915554935175,
+ 0.6282075548226255,
+ 0.6238529406792321,
+ 0.6110372428105817,
+ 0.6067084098068692,
+ 0.6066747429327572,
+ 0.5896917387284214,
+ 0.5833492490168631,
+ 0.578446588355144,
+ 0.5661802662430645,
+ 0.5567311537023375,
+ 0.5185882903408234,
+ 0.5072854830825286,
+ 0.4778492404529414,
+ 0.4704129189692171,
+ 0.4702726004236759
+ ],
+ [
+ 0.8454143444013094,
+ 0.8299794050064416,
+ 0.75652875850107,
+ 0.7133448046635743,
+ 0.6926056513937032,
+ 0.6777335662926751,
+ 0.6689801537904089,
+ 0.6317601366194467,
+ 0.613291182531161,
+ 0.6061031170801152,
+ 0.5559474460829967,
+ 0.5478191691838251,
+ 0.546670625652822,
+ 0.5235086165917824,
+ 0.5108268268538694,
+ 0.4891732531029739,
+ 0.45840524553236417,
+ 0.45210731767373624,
+ 0.4514992508075503,
+ 0.44765276418870426
+ ],
+ [
+ 0.6518021095609231,
+ 0.6374957736215378,
+ 0.6302982036874143,
+ 0.6109006276137325,
+ 0.5181172304263939,
+ 0.513725422442192,
+ 0.5115396379092528,
+ 0.5100302700293557,
+ 0.48113737320489414,
+ 0.4764075775522656,
+ 0.4698846679587283,
+ 0.4644119988734322,
+ 0.450691735801912,
+ 0.4496088948872758,
+ 0.44726418052906947,
+ 0.44638263198482164,
+ 0.4435332019071739,
+ 0.44273515287957116,
+ 0.42539189550733525,
+ 0.42393457664601325
+ ],
+ [
+ 0.8241383462264974,
+ 0.7483120003494715,
+ 0.7250771678077363,
+ 0.7207897745372984,
+ 0.7092382880137871,
+ 0.7081287851818922,
+ 0.7058548962268074,
+ 0.6836425002455113,
+ 0.6828410888195475,
+ 0.6627392627295547,
+ 0.6441097526926346,
+ 0.6424880867919112,
+ 0.6272157493382303,
+ 0.6192754649642771,
+ 0.6138535831006101,
+ 0.6028822419140492,
+ 0.5984059171811635,
+ 0.5963459062634433,
+ 0.5666611864876185,
+ 0.5551216651676174
+ ],
+ [
+ 0.9314701898451888,
+ 0.9310999636363757,
+ 0.9169202173752019,
+ 0.9135274446232649,
+ 0.9056972272387589,
+ 0.8904527712756005,
+ 0.8367198761089845,
+ 0.7636701343679253,
+ 0.7309342551225415,
+ 0.6527147872281984,
+ 0.6520709409109311,
+ 0.6345542912000144,
+ 0.6320489530984358,
+ 0.621798844416169,
+ 0.6083756519207839,
+ 0.5732363241281078,
+ 0.572961927507458,
+ 0.5484971499982132,
+ 0.43038489307720007,
+ 0.33994318853360106
+ ],
+ [
+ 0.9071476001839174,
+ 0.8813478912803353,
+ 0.8324899565372854,
+ 0.8162820457178555,
+ 0.7980840576940271,
+ 0.7821650626867912,
+ 0.7751948250335982,
+ 0.7643393718910569,
+ 0.7628836516225402,
+ 0.7579842274834465,
+ 0.7551977592750918,
+ 0.7446823801599494,
+ 0.7297837556386723,
+ 0.7262332675221932,
+ 0.7230959749928051,
+ 0.7151916699295986,
+ 0.7093122257580767,
+ 0.6943268814859965,
+ 0.6807306589674097,
+ 0.6792943683655888
+ ],
+ [
+ 0.8906906644495729,
+ 0.8760912341979784,
+ 0.8137561052749885,
+ 0.747526890773955,
+ 0.7139029112938842,
+ 0.6619121776731873,
+ 0.6377453085497351,
+ 0.6307294217152695,
+ 0.6276946764613796,
+ 0.5677863782128176,
+ 0.5600735269250471,
+ 0.5328781950094702,
+ 0.5257924083272635,
+ 0.5009753274535399,
+ 0.47569956636279265,
+ 0.459994698283293,
+ 0.4457410002226168,
+ 0.4404955998511841,
+ 0.41662773950323834,
+ 0.41581363162266716
+ ],
+ [
+ 0.8684278572655861,
+ 0.8436147681242716,
+ 0.7208962620070156,
+ 0.7176912720749922,
+ 0.6976369046326575,
+ 0.6800972903823973,
+ 0.6788935858946182,
+ 0.6719601248476509,
+ 0.6699692309680064,
+ 0.6499863944685188,
+ 0.6399445155682256,
+ 0.5941229675745408,
+ 0.583489381001496,
+ 0.5805284042491821,
+ 0.540523971264611,
+ 0.5281946186307784,
+ 0.5075722233789135,
+ 0.41539589042304215,
+ 0.40751154409957174,
+ 0.39728248723632725
+ ],
+ [
+ 0.8731173240503768,
+ 0.8247447709553013,
+ 0.7934345811756924,
+ 0.7836451108089619,
+ 0.7721217712058348,
+ 0.7324492952340598,
+ 0.6931607531503838,
+ 0.6923503463620143,
+ 0.6521940365555181,
+ 0.6037914791902697,
+ 0.584907683384388,
+ 0.5296133311267914,
+ 0.5236470157895627,
+ 0.5130520059365672,
+ 0.4804258963327972,
+ 0.4733293829501569,
+ 0.42333489754977016,
+ 0.41714500292886536,
+ 0.41613069335615116,
+ 0.4104636061019757
+ ],
+ [
+ 0.8848563855967215,
+ 0.8425411478337941,
+ 0.7664472713549255,
+ 0.7292255314146306,
+ 0.7213861967871797,
+ 0.6971337702734579,
+ 0.6774924598568098,
+ 0.6481075259108073,
+ 0.6434254148583638,
+ 0.6413530972950233,
+ 0.6214951237214152,
+ 0.6052299608833498,
+ 0.5996374554979368,
+ 0.59411655000922,
+ 0.5905366527065747,
+ 0.5548996777882085,
+ 0.5537903373736366,
+ 0.5418086015596685,
+ 0.5371932832067131,
+ 0.5348589270678971
+ ],
+ [
+ 0.7954849201061136,
+ 0.7710431220991019,
+ 0.6913632638227342,
+ 0.6556116673667995,
+ 0.628193489709682,
+ 0.6139911637027874,
+ 0.6088846541738331,
+ 0.5984451019714839,
+ 0.5536659071113338,
+ 0.5487404476251068,
+ 0.5242015117895882,
+ 0.5200300098579446,
+ 0.49498705057734405,
+ 0.4739944686976046,
+ 0.47045685569690554,
+ 0.470017116544597,
+ 0.458929604249381,
+ 0.4550223465997808,
+ 0.42409825914610844,
+ 0.3954776823038523
+ ],
+ [
+ 0.9683633496094662,
+ 0.8090798283418931,
+ 0.7570981161675205,
+ 0.7432132544455639,
+ 0.7096913439392056,
+ 0.6768883753673474,
+ 0.6252575747423712,
+ 0.6213945804195515,
+ 0.615243170457834,
+ 0.6067575986208946,
+ 0.5642122853694848,
+ 0.5330829297565619,
+ 0.5166139637439543,
+ 0.5116541386998836,
+ 0.511564254500704,
+ 0.48346936960298925,
+ 0.4692838364722046,
+ 0.4603131207452706,
+ 0.42590029065959284,
+ 0.4209258158292447
+ ],
+ [
+ 0.983256155097401,
+ 0.9627765017986655,
+ 0.8266835545378068,
+ 0.8243986250736537,
+ 0.8216475098599042,
+ 0.8133660888921939,
+ 0.8133525041423942,
+ 0.7748647308340684,
+ 0.760684617203571,
+ 0.7500474001878548,
+ 0.6529341098435704,
+ 0.5137846161025639,
+ 0.4514291328812497,
+ 0.4233033745280628,
+ 0.4198525376532687,
+ 0.40777122850534087,
+ 0.3628592373133003,
+ 0.36222471032687686,
+ 0.3615956173945232,
+ 0.354441898826373
+ ],
+ [
+ 1.0,
+ 0.8592803508805836,
+ 0.8582642765642685,
+ 0.7809925102658152,
+ 0.7643127821084993,
+ 0.6885877590298409,
+ 0.678652508257778,
+ 0.6772106962920881,
+ 0.6745022104103756,
+ 0.6684019041452945,
+ 0.6516040798317508,
+ 0.6345114012475923,
+ 0.6080884328387021,
+ 0.580319993793783,
+ 0.549465132029648,
+ 0.5438982258758669,
+ 0.5360915360446279,
+ 0.5118222613568101,
+ 0.4931031493306911,
+ 0.41237510573838315
+ ],
+ [
+ 0.8462186892606993,
+ 0.7476043565105441,
+ 0.672872046221132,
+ 0.6499608018791347,
+ 0.649234593794751,
+ 0.6155695593085244,
+ 0.5714721798038556,
+ 0.5482903630011363,
+ 0.5435136192359267,
+ 0.5007058918703509,
+ 0.4938078072911494,
+ 0.4539446285622832,
+ 0.43571895576303366,
+ 0.43404667509798917,
+ 0.39982756017860066,
+ 0.39375864866310006,
+ 0.3876898376362686,
+ 0.37216762762237265,
+ 0.37014058790440163,
+ 0.36787944117144233
+ ],
+ [
+ 0.7943631891406624,
+ 0.7933089728719205,
+ 0.7152949686501104,
+ 0.6680284797147753,
+ 0.6629756524503924,
+ 0.6127582209417133,
+ 0.6105795719658592,
+ 0.5999574159624446,
+ 0.556891057635871,
+ 0.5528645896437028,
+ 0.5513868743178407,
+ 0.5424704839364273,
+ 0.528299963603639,
+ 0.5247307662757437,
+ 0.5239021361773074,
+ 0.4884358431028368,
+ 0.48173436329005825,
+ 0.46412328774300626,
+ 0.4635019219939117,
+ 0.4624247229700438
+ ],
+ [
+ 0.7488978480503103,
+ 0.6972066912003604,
+ 0.6825881213651646,
+ 0.6799879597049332,
+ 0.6618661524044294,
+ 0.6398902006147444,
+ 0.5536062716079907,
+ 0.5454510411398401,
+ 0.5078873454298773,
+ 0.493198103481966,
+ 0.4411183199932568,
+ 0.44022239831699456,
+ 0.43076447018940617,
+ 0.4218753846304124,
+ 0.41416646588312306,
+ 0.40347447619426885,
+ 0.40228901246773224,
+ 0.4015898921636432,
+ 0.40109439126340174,
+ 0.3918731976427799
+ ],
+ [
+ 0.9422448170258992,
+ 0.7067062590556051,
+ 0.6392481455491086,
+ 0.6336446632708815,
+ 0.6306911600586252,
+ 0.6220553884180193,
+ 0.6091980674621732,
+ 0.5528745424784796,
+ 0.4977607699355271,
+ 0.4860726472095537,
+ 0.4702747000757933,
+ 0.46473370836855765,
+ 0.46394905986267365,
+ 0.4584480587420733,
+ 0.42188625766464094,
+ 0.4206986893675158,
+ 0.41645921680236253,
+ 0.40251948191070597,
+ 0.4011416448424227,
+ 0.3984278962399447
+ ],
+ [
+ 0.9360886098282506,
+ 0.6395274840241867,
+ 0.6197922517973068,
+ 0.6114783622333764,
+ 0.5958043299181577,
+ 0.5573349626403405,
+ 0.5425033707812391,
+ 0.5239077587968115,
+ 0.5049867062047504,
+ 0.4797838177166599,
+ 0.46639824398435126,
+ 0.452242786358805,
+ 0.44672327841688586,
+ 0.4356160443832435,
+ 0.4302914095853066,
+ 0.4273844328086751,
+ 0.4273738098910075,
+ 0.4229458626210032,
+ 0.4126512507077959,
+ 0.4111708677948354
+ ],
+ [
+ 0.902235339476755,
+ 0.8927026775182952,
+ 0.6741527762917963,
+ 0.6516773501090084,
+ 0.6339926121008617,
+ 0.6143258664351116,
+ 0.5888448754943169,
+ 0.5473488859875248,
+ 0.5355733454752292,
+ 0.5164690531891041,
+ 0.5153878941416781,
+ 0.5039133027018107,
+ 0.500542830440455,
+ 0.4632245105169527,
+ 0.43930078913506965,
+ 0.42430675630521575,
+ 0.4164071931955039,
+ 0.4144271930107544,
+ 0.37937981379700014,
+ 0.3694392648286271
+ ],
+ [
+ 0.8137945792324977,
+ 0.742989304647348,
+ 0.695979060441454,
+ 0.6705949844304122,
+ 0.6451086801905437,
+ 0.6161455703766523,
+ 0.6096953981015781,
+ 0.5966156452626025,
+ 0.5757727219708306,
+ 0.5433006298393654,
+ 0.5120030799056572,
+ 0.5089985912907313,
+ 0.5012896293892612,
+ 0.4926603668653614,
+ 0.4843187814796658,
+ 0.46699309728971644,
+ 0.46234864741936466,
+ 0.45435726654257735,
+ 0.4524526720232481,
+ 0.42513591510080284
+ ],
+ [
+ 0.9845839380864626,
+ 0.9255153597314225,
+ 0.9247209279343046,
+ 0.8748778410414207,
+ 0.8641350487002009,
+ 0.826325115425889,
+ 0.7517901118843952,
+ 0.7282891666749267,
+ 0.7278586455049296,
+ 0.6870404374246588,
+ 0.6779886212482745,
+ 0.6088331366339493,
+ 0.5884366809154945,
+ 0.5708244386277814,
+ 0.569766300490173,
+ 0.5292592841756737,
+ 0.5236564390851713,
+ 0.4749490187508282,
+ 0.3896672715013799,
+ 0.3521333599650295
+ ],
+ [
+ 0.8580035182241111,
+ 0.6903195671895037,
+ 0.6868912096305397,
+ 0.6580921650961482,
+ 0.6518109358867615,
+ 0.6347109084724922,
+ 0.5752426900482761,
+ 0.5751356496181426,
+ 0.5629500574876368,
+ 0.5616427004270714,
+ 0.5453740881552961,
+ 0.5207142696714979,
+ 0.5034667839950155,
+ 0.4922215551886081,
+ 0.4819794140536696,
+ 0.46418272261802085,
+ 0.4558879494074921,
+ 0.45507520693200554,
+ 0.4498093904786228,
+ 0.4490181093762455
+ ],
+ [
+ 0.9452416897699644,
+ 0.7472594403789202,
+ 0.7184117716843357,
+ 0.7179589985792247,
+ 0.6921128769384105,
+ 0.6140067804011875,
+ 0.600718864048077,
+ 0.6001987112971381,
+ 0.5752501561949719,
+ 0.5528126076555974,
+ 0.5494429133374201,
+ 0.5047637242989009,
+ 0.4919789252872878,
+ 0.4836186825842023,
+ 0.4445410279509916,
+ 0.43331371572972055,
+ 0.4291552804232768,
+ 0.42658784014035195,
+ 0.4137901679158279,
+ 0.4082975893778714
+ ]
+ ]
+ },
+ "seuratDefault": {
+ "modalityWeights": [
+ [
+ 0.5546128069847279,
+ 0.445387193015272
+ ],
+ [
+ 0.507301531350983,
+ 0.4926984686490171
+ ],
+ [
+ 0.5108437465785425,
+ 0.4891562534214575
+ ],
+ [
+ 0.5608836481274957,
+ 0.4391163518725043
+ ],
+ [
+ 0.47044900806550516,
+ 0.5295509919344948
+ ],
+ [
+ 0.5768070869077289,
+ 0.42319291309227103
+ ],
+ [
+ 0.49314062351543714,
+ 0.5068593764845628
+ ],
+ [
+ 0.4215148516419274,
+ 0.5784851483580726
+ ],
+ [
+ 0.5256730343188714,
+ 0.4743269656811287
+ ],
+ [
+ 0.51362230412425,
+ 0.48637769587574997
+ ],
+ [
+ 0.48862873847395705,
+ 0.5113712615260428
+ ],
+ [
+ 0.5109332501828671,
+ 0.48906674981713283
+ ],
+ [
+ 0.6567050036944636,
+ 0.3432949963055365
+ ],
+ [
+ 0.47089385644218046,
+ 0.5291061435578195
+ ],
+ [
+ 0.4924187462170206,
+ 0.5075812537829795
+ ],
+ [
+ 0.44834891504334284,
+ 0.551651084956657
+ ],
+ [
+ 0.6025367683603465,
+ 0.3974632316396534
+ ],
+ [
+ 0.4691268182706735,
+ 0.5308731817293266
+ ],
+ [
+ 0.4638701737609364,
+ 0.5361298262390636
+ ],
+ [
+ 0.49219746984933543,
+ 0.5078025301506646
+ ],
+ [
+ 0.4999773724249526,
+ 0.5000226275750475
+ ],
+ [
+ 0.5215839409631766,
+ 0.4784160590368233
+ ],
+ [
+ 0.472978066891568,
+ 0.527021933108432
+ ],
+ [
+ 0.4841932861281209,
+ 0.515806713871879
+ ],
+ [
+ 0.6803096113006248,
+ 0.3196903886993752
+ ],
+ [
+ 0.478760849640882,
+ 0.5212391503591179
+ ],
+ [
+ 0.5262026029223904,
+ 0.47379739707760954
+ ],
+ [
+ 0.45192632855266207,
+ 0.548073671447338
+ ],
+ [
+ 0.6228158417919051,
+ 0.37718415820809487
+ ],
+ [
+ 0.517627214412919,
+ 0.48237278558708097
+ ],
+ [
+ 0.5463461498818565,
+ 0.4536538501181436
+ ],
+ [
+ 0.5012165264656289,
+ 0.4987834735343711
+ ],
+ [
+ 0.5083207315221155,
+ 0.49167926847788457
+ ],
+ [
+ 0.4933899854348415,
+ 0.5066100145651585
+ ],
+ [
+ 0.4846168671038602,
+ 0.5153831328961399
+ ],
+ [
+ 0.5358314340819496,
+ 0.46416856591805056
+ ],
+ [
+ 0.44382200668935023,
+ 0.5561779933106497
+ ],
+ [
+ 0.49890194682233296,
+ 0.501098053177667
+ ],
+ [
+ 0.40557915953997736,
+ 0.5944208404600227
+ ],
+ [
+ 0.45977044571085535,
+ 0.5402295542891447
+ ],
+ [
+ 0.9985353873923644,
+ 0.0014646126076356815
+ ],
+ [
+ 0.464998802847911,
+ 0.535001197152089
+ ],
+ [
+ 0.45386928620218775,
+ 0.5461307137978122
+ ],
+ [
+ 0.48099871996422516,
+ 0.5190012800357747
+ ],
+ [
+ 0.6257872230951561,
+ 0.3742127769048439
+ ],
+ [
+ 0.5314373910979512,
+ 0.46856260890204887
+ ],
+ [
+ 0.5307855107922929,
+ 0.46921448920770714
+ ],
+ [
+ 0.9230827616100302,
+ 0.07691723838996982
+ ],
+ [
+ 0.5124331940552804,
+ 0.4875668059447196
+ ],
+ [
+ 0.45244573957030143,
+ 0.5475542604296986
+ ],
+ [
+ 0.5115802428559304,
+ 0.4884197571440696
+ ],
+ [
+ 0.4614965523139312,
+ 0.5385034476860687
+ ],
+ [
+ 0.5903157635095415,
+ 0.40968423649045854
+ ],
+ [
+ 0.4226534260238434,
+ 0.5773465739761566
+ ],
+ [
+ 0.4890247355950628,
+ 0.5109752644049371
+ ],
+ [
+ 0.5057457292516521,
+ 0.49425427074834805
+ ],
+ [
+ 0.45858927949729367,
+ 0.5414107205027063
+ ],
+ [
+ 0.5005759453475902,
+ 0.49942405465240985
+ ],
+ [
+ 0.4528470127272023,
+ 0.5471529872727977
+ ],
+ [
+ 0.4476354860998634,
+ 0.5523645139001366
+ ],
+ [
+ 0.46154628691165445,
+ 0.5384537130883456
+ ],
+ [
+ 0.5229446331657209,
+ 0.47705536683427907
+ ],
+ [
+ 0.5680125893298545,
+ 0.4319874106701455
+ ],
+ [
+ 0.5386927555632071,
+ 0.46130724443679294
+ ],
+ [
+ 0.02021253738747881,
+ 0.9797874626125211
+ ],
+ [
+ 0.45484728330630037,
+ 0.5451527166936996
+ ],
+ [
+ 0.4242883708629516,
+ 0.5757116291370483
+ ],
+ [
+ 0.5124220766468867,
+ 0.4875779233531133
+ ],
+ [
+ 0.487586115833259,
+ 0.5124138841667409
+ ],
+ [
+ 0.5156693098196049,
+ 0.48433069018039504
+ ],
+ [
+ 0.5168453148970179,
+ 0.48315468510298204
+ ],
+ [
+ 0.5171718156012357,
+ 0.4828281843987643
+ ],
+ [
+ 0.5015135144810888,
+ 0.4984864855189112
+ ],
+ [
+ 0.5903346475855243,
+ 0.40966535241447577
+ ],
+ [
+ 0.4913620067644125,
+ 0.5086379932355874
+ ],
+ [
+ 0.47780734002262953,
+ 0.5221926599773704
+ ],
+ [
+ 0.48566225081813974,
+ 0.5143377491818603
+ ],
+ [
+ 0.4722551660776324,
+ 0.5277448339223676
+ ],
+ [
+ 0.4974078290494594,
+ 0.5025921709505405
+ ],
+ [
+ 0.5152417194232877,
+ 0.48475828057671233
+ ],
+ [
+ 0.46591397883135793,
+ 0.5340860211686421
+ ],
+ [
+ 0.5106796589336006,
+ 0.48932034106639954
+ ],
+ [
+ 0.2149164897282123,
+ 0.7850835102717876
+ ],
+ [
+ 0.5266648114738349,
+ 0.4733351885261651
+ ],
+ [
+ 0.5420797535601005,
+ 0.4579202464398996
+ ],
+ [
+ 0.5175600864155697,
+ 0.4824399135844302
+ ],
+ [
+ 0.5319304198868555,
+ 0.4680695801131446
+ ],
+ [
+ 0.4869720186800087,
+ 0.5130279813199913
+ ],
+ [
+ 0.43767609982576244,
+ 0.5623239001742375
+ ],
+ [
+ 0.4847646952111005,
+ 0.5152353047888995
+ ],
+ [
+ 0.5404676885083876,
+ 0.45953231149161244
+ ],
+ [
+ 0.514316027983734,
+ 0.4856839720162661
+ ],
+ [
+ 0.47845764655083695,
+ 0.521542353449163
+ ],
+ [
+ 0.3781136547182048,
+ 0.6218863452817952
+ ],
+ [
+ 0.5307868920832981,
+ 0.46921310791670184
+ ],
+ [
+ 0.5098502199322215,
+ 0.4901497800677785
+ ],
+ [
+ 0.4773989970190616,
+ 0.5226010029809385
+ ],
+ [
+ 0.5171565854876448,
+ 0.4828434145123553
+ ],
+ [
+ 0.44185332201514455,
+ 0.5581466779848555
+ ],
+ [
+ 0.6345374761566258,
+ 0.3654625238433743
+ ],
+ [
+ 0.013674973083598324,
+ 0.9863250269164016
+ ],
+ [
+ 0.4792938236188808,
+ 0.5207061763811192
+ ],
+ [
+ 0.46540499413321085,
+ 0.5345950058667892
+ ],
+ [
+ 0.5051553581165549,
+ 0.4948446418834451
+ ],
+ [
+ 0.44876719000454385,
+ 0.5512328099954562
+ ],
+ [
+ 0.4359591970940289,
+ 0.564040802905971
+ ],
+ [
+ 0.41183173171085113,
+ 0.5881682682891488
+ ],
+ [
+ 0.9608267298226572,
+ 0.03917327017734285
+ ],
+ [
+ 0.4875889391452591,
+ 0.5124110608547409
+ ],
+ [
+ 0.47546682216874053,
+ 0.5245331778312595
+ ],
+ [
+ 0.5340506850848222,
+ 0.4659493149151779
+ ],
+ [
+ 0.4968546537684768,
+ 0.5031453462315232
+ ],
+ [
+ 0.583972863072473,
+ 0.416027136927527
+ ],
+ [
+ 0.41762872717679034,
+ 0.5823712728232097
+ ],
+ [
+ 0.6012459043303356,
+ 0.3987540956696643
+ ],
+ [
+ 0.5357506934567817,
+ 0.4642493065432182
+ ],
+ [
+ 0.5194422914841389,
+ 0.4805577085158611
+ ],
+ [
+ 0.9999999925468791,
+ 7.453120936055658e-09
+ ],
+ [
+ 0.3933282507302706,
+ 0.6066717492697294
+ ],
+ [
+ 0.47660383482705204,
+ 0.523396165172948
+ ],
+ [
+ 0.5438773045850781,
+ 0.4561226954149219
+ ],
+ [
+ 0.46199268820213224,
+ 0.5380073117978679
+ ],
+ [
+ 0.40658486951985695,
+ 0.5934151304801432
+ ],
+ [
+ 0.49938367881547513,
+ 0.5006163211845248
+ ],
+ [
+ 0.5264458096163304,
+ 0.47355419038366964
+ ],
+ [
+ 0.5267872658776778,
+ 0.4732127341223222
+ ],
+ [
+ 0.28950802246630536,
+ 0.7104919775336946
+ ],
+ [
+ 0.5666754203376866,
+ 0.4333245796623133
+ ],
+ [
+ 0.42263361529218674,
+ 0.5773663847078133
+ ],
+ [
+ 0.5,
+ 0.5
+ ],
+ [
+ 0.5676368392443424,
+ 0.4323631607556576
+ ],
+ [
+ 0.4961016171905128,
+ 0.5038983828094871
+ ],
+ [
+ 0.48616893473501815,
+ 0.5138310652649819
+ ],
+ [
+ 0.48808373176576036,
+ 0.5119162682342397
+ ],
+ [
+ 1.0,
+ 5.316160070399062e-18
+ ],
+ [
+ 0.47975595297226026,
+ 0.5202440470277396
+ ],
+ [
+ 0.5295093480608457,
+ 0.4704906519391544
+ ],
+ [
+ 0.507746805517961,
+ 0.49225319448203897
+ ],
+ [
+ 0.49636077280294405,
+ 0.503639227197056
+ ],
+ [
+ 0.4879452589607009,
+ 0.512054741039299
+ ],
+ [
+ 0.5425435649454495,
+ 0.4574564350545506
+ ],
+ [
+ 0.19806552443282027,
+ 0.8019344755671798
+ ],
+ [
+ 0.5105563418337544,
+ 0.4894436581662457
+ ],
+ [
+ 0.5022383956585774,
+ 0.49776160434142264
+ ],
+ [
+ 0.5915062231401588,
+ 0.4084937768598413
+ ],
+ [
+ 0.4941670696021286,
+ 0.5058329303978714
+ ],
+ [
+ 0.979426439546183,
+ 0.020573560453816985
+ ],
+ [
+ 0.6269056625244599,
+ 0.37309433747554
+ ],
+ [
+ 0.49737401619797394,
+ 0.502625983802026
+ ],
+ [
+ 0.4665672988623708,
+ 0.5334327011376292
+ ],
+ [
+ 0.47968870520935786,
+ 0.5203112947906422
+ ],
+ [
+ 0.4993279822753587,
+ 0.5006720177246413
+ ],
+ [
+ 0.5132841991097749,
+ 0.48671580089022504
+ ],
+ [
+ 0.5618319609921082,
+ 0.43816803900789175
+ ],
+ [
+ 0.5036686892365806,
+ 0.4963313107634195
+ ],
+ [
+ 0.5147861684362386,
+ 0.48521383156376147
+ ],
+ [
+ 0.804977824327006,
+ 0.19502217567299387
+ ],
+ [
+ 0.47713420939715917,
+ 0.5228657906028408
+ ],
+ [
+ 0.48982224374182187,
+ 0.510177756258178
+ ],
+ [
+ 0.37519554060939564,
+ 0.6248044593906044
+ ],
+ [
+ 0.4996478829295668,
+ 0.5003521170704331
+ ],
+ [
+ 0.45438545184313917,
+ 0.5456145481568607
+ ],
+ [
+ 0.4874360210420771,
+ 0.5125639789579229
+ ],
+ [
+ 0.496841911472493,
+ 0.503158088527507
+ ],
+ [
+ 0.4041726201107871,
+ 0.595827379889213
+ ],
+ [
+ 0.5529666646010554,
+ 0.4470333353989447
+ ],
+ [
+ 0.5207632642433511,
+ 0.47923673575664893
+ ],
+ [
+ 0.5082999297315223,
+ 0.4917000702684777
+ ],
+ [
+ 0.4313813633417036,
+ 0.5686186366582964
+ ],
+ [
+ 0.47613945404919383,
+ 0.5238605459508061
+ ],
+ [
+ 0.2742304235256735,
+ 0.7257695764743266
+ ],
+ [
+ 0.5021092528300547,
+ 0.4978907471699453
+ ],
+ [
+ 0.4480249004733207,
+ 0.5519750995266792
+ ],
+ [
+ 0.39683718758592307,
+ 0.6031628124140769
+ ],
+ [
+ 0.5002000853174445,
+ 0.49979991468255536
+ ],
+ [
+ 0.6120791363736625,
+ 0.3879208636263375
+ ],
+ [
+ 0.5231843122470424,
+ 0.4768156877529576
+ ],
+ [
+ 1.0,
+ 3.6581966264758953e-51
+ ],
+ [
+ 0.516880633901744,
+ 0.483119366098256
+ ],
+ [
+ 0.4587420595329121,
+ 0.5412579404670879
+ ],
+ [
+ 0.49036106584772254,
+ 0.5096389341522775
+ ],
+ [
+ 0.5241255712509462,
+ 0.4758744287490537
+ ],
+ [
+ 0.509280747995651,
+ 0.4907192520043489
+ ],
+ [
+ 0.6547460126385917,
+ 0.34525398736140844
+ ],
+ [
+ 0.5091499709560042,
+ 0.4908500290439959
+ ],
+ [
+ 0.4691154127018028,
+ 0.5308845872981972
+ ],
+ [
+ 0.5466580050027884,
+ 0.45334199499721156
+ ],
+ [
+ 0.47554617723425563,
+ 0.5244538227657445
+ ],
+ [
+ 0.5562390622390578,
+ 0.4437609377609422
+ ],
+ [
+ 0.4958053518818823,
+ 0.5041946481181178
+ ],
+ [
+ 0.529607243749097,
+ 0.4703927562509031
+ ],
+ [
+ 0.8017149243151389,
+ 0.19828507568486115
+ ],
+ [
+ 0.5014285937090849,
+ 0.4985714062909152
+ ],
+ [
+ 0.5415220008094376,
+ 0.45847799919056237
+ ],
+ [
+ 0.5346879460252986,
+ 0.46531205397470143
+ ],
+ [
+ 0.5339291959219409,
+ 0.466070804078059
+ ],
+ [
+ 0.4958932554721994,
+ 0.5041067445278007
+ ],
+ [
+ 0.48607862637564775,
+ 0.5139213736243522
+ ],
+ [
+ 0.545692331217431,
+ 0.454307668782569
+ ],
+ [
+ 0.49871559469056786,
+ 0.5012844053094322
+ ],
+ [
+ 0.03077983952796242,
+ 0.9692201604720376
+ ],
+ [
+ 0.48959559579731693,
+ 0.5104044042026831
+ ],
+ [
+ 0.30509296085631804,
+ 0.6949070391436819
+ ],
+ [
+ 0.4878231963416535,
+ 0.5121768036583465
+ ],
+ [
+ 0.5419629189328454,
+ 0.45803708106715446
+ ],
+ [
+ 0.4796739428805324,
+ 0.5203260571194677
+ ],
+ [
+ 0.512204266307415,
+ 0.48779573369258494
+ ],
+ [
+ 0.36350413542841087,
+ 0.6364958645715891
+ ],
+ [
+ 0.492076535289832,
+ 0.5079234647101681
+ ],
+ [
+ 0.5281712450790019,
+ 0.47182875492099813
+ ],
+ [
+ 0.4223602445244126,
+ 0.5776397554755873
+ ],
+ [
+ 0.5130220054547368,
+ 0.4869779945452633
+ ],
+ [
+ 0.4874788834906823,
+ 0.5125211165093178
+ ],
+ [
+ 0.5774060731445957,
+ 0.42259392685540426
+ ],
+ [
+ 0.3796963581689737,
+ 0.6203036418310264
+ ],
+ [
+ 0.5325717262975739,
+ 0.46742827370242607
+ ],
+ [
+ 0.5315901651733821,
+ 0.4684098348266179
+ ],
+ [
+ 0.4933651026035132,
+ 0.5066348973964868
+ ],
+ [
+ 0.49916869998881835,
+ 0.5008313000111817
+ ],
+ [
+ 0.5,
+ 0.5
+ ],
+ [
+ 0.4648510968296744,
+ 0.5351489031703256
+ ],
+ [
+ 0.5107321672603244,
+ 0.48926783273967567
+ ],
+ [
+ 0.49791713370133955,
+ 0.5020828662986605
+ ],
+ [
+ 0.5119653010396497,
+ 0.4880346989603504
+ ],
+ [
+ 0.41809773854165905,
+ 0.5819022614583409
+ ],
+ [
+ 0.33227953736751553,
+ 0.6677204626324845
+ ],
+ [
+ 0.7072207109474448,
+ 0.29277928905255524
+ ],
+ [
+ 0.4730476164863186,
+ 0.5269523835136813
+ ],
+ [
+ 0.0010529098730367255,
+ 0.9989470901269634
+ ],
+ [
+ 0.5828912849428305,
+ 0.4171087150571696
+ ],
+ [
+ 0.5781638694927834,
+ 0.4218361305072165
+ ],
+ [
+ 0.518397757346712,
+ 0.481602242653288
+ ],
+ [
+ 0.5920664401154362,
+ 0.40793355988456387
+ ],
+ [
+ 0.39792690283748694,
+ 0.6020730971625131
+ ],
+ [
+ 0.512656092331253,
+ 0.48734390766874697
+ ],
+ [
+ 0.5497549948112854,
+ 0.45024500518871463
+ ],
+ [
+ 0.5208408916018082,
+ 0.4791591083981917
+ ],
+ [
+ 0.5298086433701126,
+ 0.4701913566298874
+ ],
+ [
+ 0.5142794127355449,
+ 0.48572058726445505
+ ],
+ [
+ 0.4635236489589554,
+ 0.5364763510410445
+ ],
+ [
+ 0.47532904562910444,
+ 0.5246709543708955
+ ],
+ [
+ 0.492129187468385,
+ 0.5078708125316149
+ ],
+ [
+ 0.5426718723629981,
+ 0.457328127637002
+ ],
+ [
+ 0.4757313000830551,
+ 0.5242686999169449
+ ],
+ [
+ 0.37129203578157144,
+ 0.6287079642184285
+ ],
+ [
+ 0.5125438454627804,
+ 0.48745615453721963
+ ],
+ [
+ 0.2949653175208085,
+ 0.7050346824791914
+ ],
+ [
+ 0.9999994542288146,
+ 5.457711853948286e-07
+ ],
+ [
+ 0.5820864404646249,
+ 0.4179135595353752
+ ],
+ [
+ 0.5331632924227103,
+ 0.4668367075772898
+ ],
+ [
+ 0.5003461138256275,
+ 0.4996538861743724
+ ],
+ [
+ 0.49441690757453194,
+ 0.5055830924254681
+ ],
+ [
+ 0.4950452839843604,
+ 0.5049547160156396
+ ],
+ [
+ 0.48799135866832,
+ 0.5120086413316801
+ ],
+ [
+ 0.481398134188769,
+ 0.518601865811231
+ ],
+ [
+ 0.47013020719232107,
+ 0.5298697928076789
+ ],
+ [
+ 0.6219379073918603,
+ 0.37806209260813983
+ ],
+ [
+ 0.5611538882732809,
+ 0.43884611172671906
+ ],
+ [
+ 0.4781954929276147,
+ 0.5218045070723853
+ ],
+ [
+ 0.4221911885484403,
+ 0.5778088114515598
+ ],
+ [
+ 0.4969335264577101,
+ 0.50306647354229
+ ],
+ [
+ 0.501872817865574,
+ 0.498127182134426
+ ],
+ [
+ 0.48307238574021677,
+ 0.5169276142597832
+ ],
+ [
+ 0.5070350741393878,
+ 0.4929649258606122
+ ],
+ [
+ 0.2992710041542398,
+ 0.7007289958457601
+ ],
+ [
+ 0.46384113999282267,
+ 0.5361588600071774
+ ],
+ [
+ 0.5378515902959249,
+ 0.46214840970407506
+ ],
+ [
+ 0.4971607129201855,
+ 0.5028392870798146
+ ],
+ [
+ 0.5247672641858935,
+ 0.47523273581410647
+ ],
+ [
+ 0.4703423160724544,
+ 0.5296576839275455
+ ],
+ [
+ 0.01302803398499007,
+ 0.9869719660150099
+ ],
+ [
+ 0.5092476244775219,
+ 0.490752375522478
+ ],
+ [
+ 0.4317036157236103,
+ 0.5682963842763896
+ ],
+ [
+ 0.5205120321233392,
+ 0.4794879678766608
+ ],
+ [
+ 0.29161854453603697,
+ 0.708381455463963
+ ],
+ [
+ 0.6294991661979378,
+ 0.37050083380206217
+ ],
+ [
+ 0.5072808734074931,
+ 0.492719126592507
+ ],
+ [
+ 0.49183601999360915,
+ 0.508163980006391
+ ],
+ [
+ 0.4462257697634394,
+ 0.5537742302365606
+ ],
+ [
+ 0.49859417980198023,
+ 0.5014058201980197
+ ],
+ [
+ 0.5254741447785062,
+ 0.4745258552214939
+ ],
+ [
+ 0.5313371304958873,
+ 0.4686628695041128
+ ],
+ [
+ 0.4456941286201646,
+ 0.5543058713798353
+ ],
+ [
+ 0.4751241949477292,
+ 0.5248758050522709
+ ],
+ [
+ 0.5652152992542604,
+ 0.43478470074573966
+ ],
+ [
+ 0.499079999979725,
+ 0.5009200000202749
+ ],
+ [
+ 0.4410127363678391,
+ 0.5589872636321608
+ ],
+ [
+ 0.5434482334138009,
+ 0.4565517665861992
+ ],
+ [
+ 0.3536024742782771,
+ 0.6463975257217229
+ ],
+ [
+ 1.0,
+ 1.0140420883101653e-65
+ ],
+ [
+ 0.3921119160050666,
+ 0.6078880839949334
+ ],
+ [
+ 0.49217209582878396,
+ 0.507827904171216
+ ],
+ [
+ 0.5174003809805621,
+ 0.48259961901943793
+ ],
+ [
+ 0.48557043006677086,
+ 0.5144295699332291
+ ],
+ [
+ 0.48131477836826575,
+ 0.5186852216317344
+ ],
+ [
+ 0.5111931398872933,
+ 0.48880686011270663
+ ],
+ [
+ 0.5056066195270466,
+ 0.49439338047295345
+ ],
+ [
+ 0.5115068751813215,
+ 0.4884931248186784
+ ],
+ [
+ 0.5933290885376746,
+ 0.4066709114623254
+ ],
+ [
+ 0.5194988340051329,
+ 0.4805011659948672
+ ]
+ ],
+ "neighborIndices": [
+ [
+ 181,
+ 157,
+ 273,
+ 227,
+ 119,
+ 81,
+ 292,
+ 255,
+ 87,
+ 208,
+ 178,
+ 132,
+ 174,
+ 261,
+ 282,
+ 10,
+ 153,
+ 216,
+ 151,
+ 92
+ ],
+ [
+ 3,
+ 193,
+ 6,
+ 76,
+ 295,
+ 142,
+ 125,
+ 204,
+ 169,
+ 161,
+ 96,
+ 43,
+ 190,
+ 80,
+ 8,
+ 106,
+ 59,
+ 68,
+ 188,
+ 143
+ ],
+ [
+ 211,
+ 203,
+ 37,
+ 189,
+ 77,
+ 235,
+ 129,
+ 9,
+ 138,
+ 212,
+ 221,
+ 34,
+ 197,
+ 133,
+ 223,
+ 236,
+ 101,
+ 285,
+ 277,
+ 296
+ ],
+ [
+ 204,
+ 1,
+ 193,
+ 96,
+ 295,
+ 76,
+ 125,
+ 8,
+ 143,
+ 43,
+ 106,
+ 161,
+ 6,
+ 68,
+ 169,
+ 188,
+ 110,
+ 66,
+ 142,
+ 72
+ ],
+ [
+ 86,
+ 278,
+ 234,
+ 91,
+ 38,
+ 71,
+ 267,
+ 39,
+ 124,
+ 74,
+ 102,
+ 88,
+ 105,
+ 233,
+ 13,
+ 89,
+ 131,
+ 145,
+ 62,
+ 118
+ ],
+ [
+ 298,
+ 148,
+ 22,
+ 42,
+ 239,
+ 39,
+ 131,
+ 63,
+ 254,
+ 60,
+ 62,
+ 15,
+ 109,
+ 259,
+ 118,
+ 18,
+ 123,
+ 74,
+ 128,
+ 278
+ ],
+ [
+ 125,
+ 78,
+ 8,
+ 1,
+ 161,
+ 110,
+ 3,
+ 76,
+ 169,
+ 43,
+ 295,
+ 262,
+ 79,
+ 204,
+ 193,
+ 106,
+ 143,
+ 96,
+ 188,
+ 68
+ ],
+ [
+ 95,
+ 246,
+ 136,
+ 53,
+ 240,
+ 199,
+ 36,
+ 58,
+ 20,
+ 32,
+ 135,
+ 127,
+ 154,
+ 61,
+ 45,
+ 23,
+ 100,
+ 207,
+ 264,
+ 116
+ ],
+ [
+ 262,
+ 169,
+ 161,
+ 110,
+ 68,
+ 43,
+ 72,
+ 78,
+ 3,
+ 6,
+ 125,
+ 295,
+ 106,
+ 79,
+ 66,
+ 272,
+ 96,
+ 274,
+ 188,
+ 204
+ ],
+ [
+ 133,
+ 138,
+ 212,
+ 37,
+ 197,
+ 263,
+ 211,
+ 296,
+ 291,
+ 236,
+ 221,
+ 34,
+ 162,
+ 268,
+ 2,
+ 277,
+ 189,
+ 129,
+ 235,
+ 77
+ ],
+ [
+ 294,
+ 208,
+ 251,
+ 92,
+ 132,
+ 261,
+ 151,
+ 103,
+ 165,
+ 227,
+ 174,
+ 119,
+ 238,
+ 81,
+ 178,
+ 152,
+ 181,
+ 26,
+ 255,
+ 157
+ ],
+ [
+ 35,
+ 70,
+ 192,
+ 50,
+ 266,
+ 33,
+ 280,
+ 69,
+ 232,
+ 108,
+ 245,
+ 297,
+ 180,
+ 171,
+ 186,
+ 139,
+ 248,
+ 30,
+ 284,
+ 196
+ ],
+ [
+ 156,
+ 270,
+ 279,
+ 116,
+ 84,
+ 229,
+ 218,
+ 107,
+ 129,
+ 217,
+ 95,
+ 114,
+ 25,
+ 162,
+ 276,
+ 47,
+ 31,
+ 203,
+ 130,
+ 223
+ ],
+ [
+ 160,
+ 128,
+ 206,
+ 111,
+ 278,
+ 60,
+ 145,
+ 254,
+ 17,
+ 131,
+ 112,
+ 102,
+ 234,
+ 109,
+ 267,
+ 89,
+ 137,
+ 5,
+ 237,
+ 39
+ ],
+ [
+ 198,
+ 115,
+ 75,
+ 46,
+ 231,
+ 214,
+ 67,
+ 262,
+ 204,
+ 79,
+ 1,
+ 295,
+ 3,
+ 161,
+ 169,
+ 48,
+ 290,
+ 6,
+ 76,
+ 80
+ ],
+ [
+ 42,
+ 118,
+ 123,
+ 239,
+ 22,
+ 105,
+ 233,
+ 91,
+ 185,
+ 298,
+ 148,
+ 124,
+ 5,
+ 63,
+ 39,
+ 175,
+ 86,
+ 62,
+ 38,
+ 278
+ ],
+ [
+ 188,
+ 68,
+ 73,
+ 43,
+ 66,
+ 169,
+ 3,
+ 147,
+ 72,
+ 1,
+ 204,
+ 193,
+ 8,
+ 141,
+ 262,
+ 96,
+ 78,
+ 125,
+ 115,
+ 76
+ ],
+ [
+ 234,
+ 13,
+ 145,
+ 131,
+ 160,
+ 74,
+ 253,
+ 278,
+ 102,
+ 206,
+ 89,
+ 112,
+ 267,
+ 259,
+ 111,
+ 55,
+ 128,
+ 254,
+ 60,
+ 184
+ ],
+ [
+ 91,
+ 267,
+ 131,
+ 62,
+ 185,
+ 239,
+ 74,
+ 234,
+ 123,
+ 259,
+ 278,
+ 89,
+ 124,
+ 128,
+ 298,
+ 39,
+ 137,
+ 5,
+ 38,
+ 60
+ ],
+ [
+ 199,
+ 65,
+ 120,
+ 44,
+ 32,
+ 95,
+ 135,
+ 20,
+ 61,
+ 240,
+ 226,
+ 154,
+ 127,
+ 275,
+ 45,
+ 136,
+ 36,
+ 7,
+ 116,
+ 23
+ ],
+ [
+ 240,
+ 199,
+ 32,
+ 154,
+ 7,
+ 136,
+ 61,
+ 135,
+ 19,
+ 58,
+ 23,
+ 95,
+ 275,
+ 36,
+ 127,
+ 65,
+ 49,
+ 53,
+ 246,
+ 45
+ ],
+ [
+ 251,
+ 155,
+ 165,
+ 208,
+ 227,
+ 294,
+ 10,
+ 92,
+ 273,
+ 238,
+ 178,
+ 119,
+ 140,
+ 103,
+ 98,
+ 283,
+ 81,
+ 152,
+ 222,
+ 293
+ ],
+ [
+ 42,
+ 15,
+ 105,
+ 239,
+ 118,
+ 298,
+ 5,
+ 123,
+ 148,
+ 63,
+ 124,
+ 91,
+ 233,
+ 39,
+ 213,
+ 38,
+ 286,
+ 74,
+ 234,
+ 86
+ ],
+ [
+ 49,
+ 61,
+ 58,
+ 127,
+ 53,
+ 154,
+ 7,
+ 32,
+ 36,
+ 240,
+ 207,
+ 135,
+ 95,
+ 100,
+ 136,
+ 45,
+ 199,
+ 20,
+ 84,
+ 246
+ ],
+ [
+ 120,
+ 226,
+ 116,
+ 65,
+ 220,
+ 19,
+ 44,
+ 275,
+ 218,
+ 32,
+ 31,
+ 244,
+ 135,
+ 199,
+ 20,
+ 95,
+ 45,
+ 84,
+ 207,
+ 265
+ ],
+ [
+ 203,
+ 236,
+ 129,
+ 215,
+ 37,
+ 296,
+ 162,
+ 34,
+ 277,
+ 2,
+ 189,
+ 133,
+ 212,
+ 279,
+ 9,
+ 138,
+ 221,
+ 77,
+ 291,
+ 235
+ ],
+ [
+ 119,
+ 157,
+ 174,
+ 132,
+ 92,
+ 208,
+ 178,
+ 222,
+ 294,
+ 282,
+ 255,
+ 227,
+ 10,
+ 165,
+ 261,
+ 152,
+ 151,
+ 155,
+ 238,
+ 94
+ ],
+ [
+ 237,
+ 128,
+ 5,
+ 60,
+ 111,
+ 13,
+ 254,
+ 148,
+ 145,
+ 38,
+ 112,
+ 278,
+ 63,
+ 206,
+ 57,
+ 109,
+ 160,
+ 137,
+ 39,
+ 89
+ ],
+ [
+ 47,
+ 51,
+ 64,
+ 191,
+ 84,
+ 229,
+ 146,
+ 85,
+ 223,
+ 218,
+ 279,
+ 285,
+ 176,
+ 281,
+ 235,
+ 217,
+ 41,
+ 197,
+ 129,
+ 101
+ ],
+ [
+ 201,
+ 97,
+ 93,
+ 271,
+ 56,
+ 209,
+ 288,
+ 159,
+ 122,
+ 82,
+ 200,
+ 224,
+ 100,
+ 258,
+ 173,
+ 264,
+ 94,
+ 152,
+ 283,
+ 40
+ ],
+ [
+ 284,
+ 248,
+ 196,
+ 186,
+ 266,
+ 232,
+ 280,
+ 50,
+ 35,
+ 33,
+ 11,
+ 69,
+ 180,
+ 287,
+ 171,
+ 139,
+ 245,
+ 108,
+ 70,
+ 192
+ ],
+ [
+ 229,
+ 257,
+ 84,
+ 85,
+ 51,
+ 218,
+ 116,
+ 205,
+ 191,
+ 146,
+ 47,
+ 64,
+ 224,
+ 244,
+ 265,
+ 95,
+ 207,
+ 45,
+ 219,
+ 135
+ ],
+ [
+ 135,
+ 58,
+ 95,
+ 199,
+ 127,
+ 61,
+ 240,
+ 154,
+ 36,
+ 20,
+ 45,
+ 23,
+ 207,
+ 65,
+ 53,
+ 136,
+ 19,
+ 7,
+ 226,
+ 116
+ ],
+ [
+ 171,
+ 108,
+ 280,
+ 69,
+ 245,
+ 297,
+ 266,
+ 11,
+ 70,
+ 35,
+ 50,
+ 192,
+ 196,
+ 139,
+ 180,
+ 186,
+ 232,
+ 30,
+ 248,
+ 284
+ ],
+ [
+ 9,
+ 197,
+ 133,
+ 37,
+ 296,
+ 162,
+ 221,
+ 235,
+ 2,
+ 203,
+ 211,
+ 129,
+ 138,
+ 212,
+ 215,
+ 263,
+ 277,
+ 236,
+ 223,
+ 25
+ ],
+ [
+ 232,
+ 50,
+ 192,
+ 11,
+ 180,
+ 186,
+ 139,
+ 70,
+ 33,
+ 248,
+ 284,
+ 196,
+ 280,
+ 108,
+ 266,
+ 30,
+ 297,
+ 245,
+ 69,
+ 171
+ ],
+ [
+ 127,
+ 207,
+ 58,
+ 61,
+ 135,
+ 53,
+ 240,
+ 95,
+ 32,
+ 154,
+ 246,
+ 136,
+ 7,
+ 23,
+ 199,
+ 45,
+ 20,
+ 224,
+ 244,
+ 100
+ ],
+ [
+ 212,
+ 133,
+ 197,
+ 9,
+ 129,
+ 235,
+ 203,
+ 138,
+ 221,
+ 211,
+ 77,
+ 162,
+ 277,
+ 223,
+ 2,
+ 217,
+ 236,
+ 263,
+ 34,
+ 291
+ ],
+ [
+ 278,
+ 39,
+ 239,
+ 286,
+ 148,
+ 298,
+ 71,
+ 237,
+ 63,
+ 111,
+ 86,
+ 145,
+ 91,
+ 60,
+ 112,
+ 254,
+ 128,
+ 267,
+ 234,
+ 13
+ ],
+ [
+ 278,
+ 298,
+ 239,
+ 38,
+ 102,
+ 71,
+ 254,
+ 148,
+ 259,
+ 5,
+ 86,
+ 91,
+ 286,
+ 182,
+ 234,
+ 111,
+ 131,
+ 60,
+ 13,
+ 137
+ ],
+ [
+ 255,
+ 132,
+ 273,
+ 157,
+ 119,
+ 87,
+ 0,
+ 149,
+ 26,
+ 227,
+ 113,
+ 152,
+ 292,
+ 166,
+ 92,
+ 282,
+ 288,
+ 261,
+ 174,
+ 209
+ ],
+ [
+ 263,
+ 268,
+ 221,
+ 217,
+ 54,
+ 236,
+ 212,
+ 133,
+ 129,
+ 281,
+ 9,
+ 37,
+ 197,
+ 138,
+ 162,
+ 77,
+ 235,
+ 277,
+ 223,
+ 285
+ ],
+ [
+ 22,
+ 15,
+ 118,
+ 239,
+ 105,
+ 5,
+ 123,
+ 298,
+ 148,
+ 63,
+ 38,
+ 233,
+ 39,
+ 91,
+ 86,
+ 124,
+ 185,
+ 286,
+ 278,
+ 213
+ ],
+ [
+ 125,
+ 8,
+ 262,
+ 78,
+ 169,
+ 3,
+ 68,
+ 161,
+ 79,
+ 80,
+ 274,
+ 106,
+ 141,
+ 193,
+ 66,
+ 6,
+ 72,
+ 188,
+ 96,
+ 1
+ ],
+ [
+ 226,
+ 275,
+ 199,
+ 19,
+ 154,
+ 61,
+ 127,
+ 45,
+ 135,
+ 65,
+ 36,
+ 23,
+ 32,
+ 20,
+ 240,
+ 207,
+ 95,
+ 58,
+ 183,
+ 49
+ ],
+ [
+ 135,
+ 127,
+ 199,
+ 207,
+ 95,
+ 61,
+ 58,
+ 154,
+ 116,
+ 32,
+ 23,
+ 36,
+ 53,
+ 226,
+ 7,
+ 44,
+ 240,
+ 136,
+ 19,
+ 20
+ ],
+ [
+ 115,
+ 67,
+ 14,
+ 130,
+ 198,
+ 290,
+ 48,
+ 262,
+ 161,
+ 75,
+ 3,
+ 272,
+ 295,
+ 80,
+ 16,
+ 1,
+ 68,
+ 193,
+ 204,
+ 73
+ ],
+ [
+ 229,
+ 51,
+ 191,
+ 28,
+ 85,
+ 217,
+ 218,
+ 84,
+ 279,
+ 223,
+ 146,
+ 257,
+ 64,
+ 31,
+ 162,
+ 235,
+ 211,
+ 197,
+ 176,
+ 101
+ ],
+ [
+ 3,
+ 8,
+ 204,
+ 68,
+ 161,
+ 169,
+ 46,
+ 262,
+ 110,
+ 6,
+ 295,
+ 272,
+ 1,
+ 115,
+ 78,
+ 193,
+ 79,
+ 66,
+ 188,
+ 76
+ ],
+ [
+ 23,
+ 154,
+ 61,
+ 20,
+ 240,
+ 58,
+ 32,
+ 127,
+ 199,
+ 136,
+ 53,
+ 36,
+ 275,
+ 135,
+ 7,
+ 177,
+ 95,
+ 45,
+ 12,
+ 207
+ ],
+ [
+ 232,
+ 35,
+ 180,
+ 192,
+ 248,
+ 11,
+ 186,
+ 284,
+ 70,
+ 266,
+ 33,
+ 30,
+ 139,
+ 280,
+ 196,
+ 108,
+ 69,
+ 287,
+ 171,
+ 297
+ ],
+ [
+ 84,
+ 229,
+ 257,
+ 47,
+ 85,
+ 191,
+ 31,
+ 64,
+ 265,
+ 224,
+ 218,
+ 146,
+ 244,
+ 28,
+ 116,
+ 100,
+ 205,
+ 207,
+ 264,
+ 82
+ ],
+ [
+ 256,
+ 99,
+ 194,
+ 167,
+ 260,
+ 90,
+ 215,
+ 268,
+ 243,
+ 263,
+ 276,
+ 236,
+ 265,
+ 291,
+ 114,
+ 277,
+ 296,
+ 279,
+ 9,
+ 101
+ ],
+ [
+ 58,
+ 36,
+ 136,
+ 127,
+ 7,
+ 95,
+ 207,
+ 23,
+ 135,
+ 32,
+ 61,
+ 199,
+ 45,
+ 240,
+ 154,
+ 246,
+ 20,
+ 100,
+ 116,
+ 264
+ ],
+ [
+ 236,
+ 133,
+ 221,
+ 263,
+ 138,
+ 268,
+ 77,
+ 176,
+ 9,
+ 281,
+ 129,
+ 197,
+ 41,
+ 212,
+ 34,
+ 37,
+ 211,
+ 189,
+ 277,
+ 223
+ ],
+ [
+ 121,
+ 234,
+ 13,
+ 253,
+ 145,
+ 86,
+ 88,
+ 182,
+ 17,
+ 160,
+ 278,
+ 102,
+ 128,
+ 112,
+ 89,
+ 74,
+ 38,
+ 267,
+ 233,
+ 131
+ ],
+ [
+ 271,
+ 97,
+ 288,
+ 122,
+ 29,
+ 209,
+ 201,
+ 93,
+ 159,
+ 82,
+ 200,
+ 173,
+ 224,
+ 100,
+ 264,
+ 265,
+ 244,
+ 191,
+ 40,
+ 36
+ ],
+ [
+ 160,
+ 112,
+ 60,
+ 267,
+ 124,
+ 278,
+ 111,
+ 128,
+ 145,
+ 254,
+ 137,
+ 13,
+ 237,
+ 185,
+ 253,
+ 109,
+ 206,
+ 38,
+ 286,
+ 102
+ ],
+ [
+ 127,
+ 135,
+ 61,
+ 36,
+ 207,
+ 154,
+ 32,
+ 53,
+ 240,
+ 23,
+ 95,
+ 45,
+ 199,
+ 7,
+ 136,
+ 20,
+ 65,
+ 246,
+ 49,
+ 116
+ ],
+ [
+ 142,
+ 261,
+ 250,
+ 113,
+ 87,
+ 157,
+ 92,
+ 104,
+ 149,
+ 80,
+ 181,
+ 227,
+ 0,
+ 178,
+ 152,
+ 106,
+ 282,
+ 255,
+ 119,
+ 174
+ ],
+ [
+ 109,
+ 254,
+ 137,
+ 278,
+ 145,
+ 267,
+ 13,
+ 128,
+ 131,
+ 5,
+ 160,
+ 102,
+ 253,
+ 111,
+ 182,
+ 185,
+ 206,
+ 39,
+ 237,
+ 27
+ ],
+ [
+ 154,
+ 58,
+ 127,
+ 240,
+ 135,
+ 36,
+ 207,
+ 23,
+ 32,
+ 45,
+ 199,
+ 20,
+ 53,
+ 95,
+ 136,
+ 226,
+ 44,
+ 49,
+ 7,
+ 275
+ ],
+ [
+ 131,
+ 185,
+ 91,
+ 259,
+ 124,
+ 137,
+ 298,
+ 278,
+ 239,
+ 102,
+ 286,
+ 60,
+ 267,
+ 109,
+ 123,
+ 160,
+ 254,
+ 74,
+ 18,
+ 182
+ ],
+ [
+ 298,
+ 148,
+ 5,
+ 286,
+ 74,
+ 60,
+ 38,
+ 91,
+ 145,
+ 42,
+ 278,
+ 22,
+ 185,
+ 239,
+ 254,
+ 105,
+ 39,
+ 62,
+ 15,
+ 131
+ ],
+ [
+ 51,
+ 84,
+ 224,
+ 265,
+ 61,
+ 100,
+ 257,
+ 28,
+ 218,
+ 127,
+ 85,
+ 229,
+ 58,
+ 200,
+ 244,
+ 146,
+ 31,
+ 47,
+ 191,
+ 264
+ ],
+ [
+ 120,
+ 135,
+ 19,
+ 32,
+ 199,
+ 58,
+ 136,
+ 20,
+ 226,
+ 44,
+ 240,
+ 23,
+ 36,
+ 207,
+ 116,
+ 275,
+ 95,
+ 61,
+ 53,
+ 24
+ ],
+ [
+ 141,
+ 68,
+ 183,
+ 188,
+ 8,
+ 169,
+ 96,
+ 72,
+ 3,
+ 43,
+ 16,
+ 106,
+ 147,
+ 125,
+ 295,
+ 193,
+ 73,
+ 204,
+ 110,
+ 1
+ ],
+ [
+ 290,
+ 46,
+ 80,
+ 115,
+ 142,
+ 272,
+ 295,
+ 130,
+ 193,
+ 161,
+ 252,
+ 214,
+ 274,
+ 72,
+ 262,
+ 158,
+ 68,
+ 3,
+ 14,
+ 166
+ ],
+ [
+ 188,
+ 66,
+ 169,
+ 72,
+ 8,
+ 16,
+ 3,
+ 96,
+ 141,
+ 43,
+ 295,
+ 73,
+ 204,
+ 272,
+ 125,
+ 106,
+ 78,
+ 1,
+ 193,
+ 262
+ ],
+ [
+ 108,
+ 245,
+ 171,
+ 297,
+ 280,
+ 33,
+ 266,
+ 11,
+ 70,
+ 196,
+ 35,
+ 139,
+ 192,
+ 180,
+ 50,
+ 30,
+ 186,
+ 232,
+ 248,
+ 284
+ ],
+ [
+ 11,
+ 108,
+ 280,
+ 33,
+ 297,
+ 245,
+ 266,
+ 35,
+ 69,
+ 171,
+ 50,
+ 232,
+ 192,
+ 180,
+ 248,
+ 186,
+ 139,
+ 30,
+ 284,
+ 196
+ ],
+ [
+ 89,
+ 86,
+ 278,
+ 112,
+ 39,
+ 145,
+ 38,
+ 102,
+ 88,
+ 237,
+ 254,
+ 160,
+ 91,
+ 234,
+ 253,
+ 137,
+ 267,
+ 286,
+ 13,
+ 259
+ ],
+ [
+ 68,
+ 272,
+ 8,
+ 188,
+ 106,
+ 169,
+ 66,
+ 80,
+ 43,
+ 110,
+ 141,
+ 274,
+ 204,
+ 295,
+ 125,
+ 142,
+ 290,
+ 96,
+ 161,
+ 78
+ ],
+ [
+ 188,
+ 16,
+ 68,
+ 43,
+ 72,
+ 141,
+ 66,
+ 3,
+ 204,
+ 8,
+ 193,
+ 115,
+ 78,
+ 272,
+ 1,
+ 262,
+ 169,
+ 143,
+ 76,
+ 46
+ ],
+ [
+ 234,
+ 91,
+ 17,
+ 131,
+ 102,
+ 86,
+ 62,
+ 253,
+ 259,
+ 278,
+ 124,
+ 148,
+ 145,
+ 267,
+ 39,
+ 298,
+ 182,
+ 13,
+ 63,
+ 38
+ ],
+ [
+ 198,
+ 14,
+ 231,
+ 115,
+ 214,
+ 46,
+ 204,
+ 6,
+ 76,
+ 295,
+ 1,
+ 3,
+ 143,
+ 190,
+ 67,
+ 169,
+ 110,
+ 188,
+ 8,
+ 68
+ ],
+ [
+ 204,
+ 3,
+ 295,
+ 125,
+ 193,
+ 110,
+ 1,
+ 96,
+ 188,
+ 6,
+ 43,
+ 143,
+ 8,
+ 161,
+ 169,
+ 106,
+ 79,
+ 290,
+ 142,
+ 104
+ ],
+ [
+ 101,
+ 197,
+ 189,
+ 277,
+ 212,
+ 235,
+ 223,
+ 203,
+ 37,
+ 129,
+ 236,
+ 221,
+ 211,
+ 291,
+ 2,
+ 138,
+ 263,
+ 285,
+ 217,
+ 133
+ ],
+ [
+ 79,
+ 8,
+ 43,
+ 274,
+ 161,
+ 262,
+ 6,
+ 125,
+ 110,
+ 272,
+ 106,
+ 169,
+ 80,
+ 68,
+ 295,
+ 72,
+ 250,
+ 193,
+ 3,
+ 1
+ ],
+ [
+ 262,
+ 161,
+ 78,
+ 8,
+ 169,
+ 125,
+ 110,
+ 43,
+ 290,
+ 274,
+ 6,
+ 80,
+ 106,
+ 250,
+ 104,
+ 295,
+ 76,
+ 204,
+ 3,
+ 193
+ ],
+ [
+ 142,
+ 290,
+ 161,
+ 274,
+ 272,
+ 149,
+ 262,
+ 67,
+ 43,
+ 59,
+ 169,
+ 79,
+ 261,
+ 295,
+ 193,
+ 78,
+ 72,
+ 8,
+ 166,
+ 113
+ ],
+ [
+ 0,
+ 151,
+ 294,
+ 292,
+ 178,
+ 92,
+ 10,
+ 208,
+ 181,
+ 227,
+ 103,
+ 119,
+ 251,
+ 255,
+ 157,
+ 153,
+ 165,
+ 140,
+ 174,
+ 152
+ ],
+ [
+ 159,
+ 56,
+ 209,
+ 93,
+ 271,
+ 97,
+ 122,
+ 29,
+ 288,
+ 200,
+ 201,
+ 265,
+ 224,
+ 191,
+ 244,
+ 100,
+ 146,
+ 47,
+ 84,
+ 51
+ ],
+ [
+ 230,
+ 63,
+ 298,
+ 286,
+ 5,
+ 175,
+ 148,
+ 254,
+ 60,
+ 111,
+ 109,
+ 239,
+ 137,
+ 124,
+ 278,
+ 267,
+ 145,
+ 123,
+ 128,
+ 38
+ ],
+ [
+ 51,
+ 218,
+ 31,
+ 229,
+ 257,
+ 191,
+ 116,
+ 146,
+ 224,
+ 85,
+ 47,
+ 205,
+ 100,
+ 244,
+ 265,
+ 64,
+ 28,
+ 207,
+ 7,
+ 95
+ ],
+ [
+ 229,
+ 51,
+ 31,
+ 257,
+ 84,
+ 64,
+ 47,
+ 218,
+ 205,
+ 191,
+ 146,
+ 116,
+ 28,
+ 219,
+ 12,
+ 45,
+ 265,
+ 224,
+ 279,
+ 264
+ ],
+ [
+ 71,
+ 234,
+ 91,
+ 278,
+ 39,
+ 4,
+ 102,
+ 38,
+ 74,
+ 145,
+ 89,
+ 88,
+ 118,
+ 62,
+ 124,
+ 182,
+ 253,
+ 185,
+ 112,
+ 239
+ ],
+ [
+ 227,
+ 261,
+ 157,
+ 178,
+ 152,
+ 0,
+ 283,
+ 255,
+ 119,
+ 92,
+ 94,
+ 292,
+ 132,
+ 174,
+ 181,
+ 294,
+ 208,
+ 282,
+ 258,
+ 155
+ ],
+ [
+ 253,
+ 163,
+ 121,
+ 145,
+ 112,
+ 182,
+ 278,
+ 124,
+ 86,
+ 267,
+ 71,
+ 160,
+ 234,
+ 55,
+ 102,
+ 89,
+ 233,
+ 185,
+ 118,
+ 91
+ ],
+ [
+ 71,
+ 267,
+ 234,
+ 259,
+ 160,
+ 145,
+ 112,
+ 131,
+ 278,
+ 254,
+ 102,
+ 13,
+ 137,
+ 253,
+ 60,
+ 17,
+ 206,
+ 86,
+ 88,
+ 39
+ ],
+ [
+ 291,
+ 194,
+ 215,
+ 296,
+ 236,
+ 9,
+ 263,
+ 279,
+ 277,
+ 197,
+ 34,
+ 268,
+ 203,
+ 77,
+ 243,
+ 37,
+ 212,
+ 133,
+ 260,
+ 211
+ ],
+ [
+ 278,
+ 123,
+ 185,
+ 86,
+ 239,
+ 74,
+ 39,
+ 102,
+ 62,
+ 267,
+ 124,
+ 286,
+ 71,
+ 298,
+ 38,
+ 4,
+ 234,
+ 254,
+ 145,
+ 131
+ ],
+ [
+ 178,
+ 294,
+ 157,
+ 152,
+ 119,
+ 174,
+ 227,
+ 10,
+ 26,
+ 81,
+ 261,
+ 208,
+ 87,
+ 165,
+ 132,
+ 283,
+ 255,
+ 155,
+ 94,
+ 282
+ ],
+ [
+ 97,
+ 209,
+ 29,
+ 271,
+ 56,
+ 201,
+ 159,
+ 288,
+ 122,
+ 82,
+ 200,
+ 224,
+ 100,
+ 265,
+ 244,
+ 191,
+ 173,
+ 264,
+ 84,
+ 127
+ ],
+ [
+ 178,
+ 258,
+ 283,
+ 152,
+ 92,
+ 87,
+ 119,
+ 26,
+ 282,
+ 157,
+ 155,
+ 294,
+ 165,
+ 238,
+ 113,
+ 104,
+ 227,
+ 208,
+ 190,
+ 261
+ ],
+ [
+ 32,
+ 135,
+ 7,
+ 45,
+ 199,
+ 58,
+ 127,
+ 36,
+ 136,
+ 154,
+ 53,
+ 240,
+ 61,
+ 207,
+ 116,
+ 23,
+ 19,
+ 246,
+ 20,
+ 65
+ ],
+ [
+ 3,
+ 125,
+ 106,
+ 169,
+ 66,
+ 68,
+ 76,
+ 295,
+ 43,
+ 188,
+ 1,
+ 8,
+ 110,
+ 193,
+ 204,
+ 59,
+ 72,
+ 142,
+ 143,
+ 161
+ ],
+ [
+ 201,
+ 271,
+ 29,
+ 93,
+ 209,
+ 56,
+ 288,
+ 159,
+ 122,
+ 82,
+ 200,
+ 100,
+ 224,
+ 258,
+ 173,
+ 264,
+ 283,
+ 282,
+ 190,
+ 94
+ ],
+ [
+ 293,
+ 172,
+ 195,
+ 251,
+ 208,
+ 10,
+ 269,
+ 190,
+ 92,
+ 294,
+ 261,
+ 227,
+ 103,
+ 119,
+ 178,
+ 21,
+ 132,
+ 238,
+ 155,
+ 87
+ ],
+ [
+ 194,
+ 167,
+ 90,
+ 243,
+ 215,
+ 276,
+ 179,
+ 268,
+ 236,
+ 296,
+ 260,
+ 291,
+ 263,
+ 9,
+ 277,
+ 25,
+ 129,
+ 211,
+ 189,
+ 54
+ ],
+ [
+ 84,
+ 244,
+ 224,
+ 246,
+ 265,
+ 218,
+ 200,
+ 264,
+ 7,
+ 93,
+ 51,
+ 207,
+ 159,
+ 36,
+ 122,
+ 23,
+ 97,
+ 82,
+ 127,
+ 271
+ ],
+ [
+ 77,
+ 235,
+ 197,
+ 277,
+ 189,
+ 221,
+ 37,
+ 212,
+ 211,
+ 129,
+ 223,
+ 291,
+ 217,
+ 2,
+ 203,
+ 133,
+ 279,
+ 285,
+ 9,
+ 236
+ ],
+ [
+ 278,
+ 259,
+ 131,
+ 234,
+ 13,
+ 254,
+ 267,
+ 89,
+ 160,
+ 39,
+ 253,
+ 74,
+ 206,
+ 137,
+ 182,
+ 62,
+ 145,
+ 91,
+ 60,
+ 17
+ ],
+ [
+ 151,
+ 10,
+ 251,
+ 81,
+ 255,
+ 294,
+ 216,
+ 252,
+ 208,
+ 242,
+ 181,
+ 132,
+ 92,
+ 165,
+ 261,
+ 227,
+ 119,
+ 152,
+ 273,
+ 238
+ ],
+ [
+ 87,
+ 94,
+ 178,
+ 261,
+ 250,
+ 174,
+ 92,
+ 106,
+ 258,
+ 187,
+ 59,
+ 292,
+ 152,
+ 157,
+ 142,
+ 26,
+ 113,
+ 294,
+ 282,
+ 283
+ ],
+ [
+ 118,
+ 42,
+ 22,
+ 233,
+ 15,
+ 239,
+ 38,
+ 298,
+ 74,
+ 63,
+ 123,
+ 86,
+ 234,
+ 278,
+ 168,
+ 148,
+ 39,
+ 91,
+ 163,
+ 4
+ ],
+ [
+ 250,
+ 96,
+ 3,
+ 59,
+ 113,
+ 104,
+ 161,
+ 8,
+ 142,
+ 125,
+ 43,
+ 193,
+ 72,
+ 272,
+ 262,
+ 110,
+ 149,
+ 66,
+ 80,
+ 169
+ ],
+ [
+ 241,
+ 164,
+ 129,
+ 211,
+ 221,
+ 34,
+ 197,
+ 268,
+ 133,
+ 279,
+ 37,
+ 236,
+ 2,
+ 223,
+ 235,
+ 54,
+ 156,
+ 162,
+ 176,
+ 203
+ ],
+ [
+ 69,
+ 297,
+ 245,
+ 171,
+ 33,
+ 280,
+ 70,
+ 266,
+ 139,
+ 11,
+ 196,
+ 35,
+ 192,
+ 180,
+ 50,
+ 186,
+ 232,
+ 30,
+ 248,
+ 284
+ ],
+ [
+ 60,
+ 137,
+ 182,
+ 128,
+ 267,
+ 278,
+ 254,
+ 131,
+ 145,
+ 13,
+ 160,
+ 102,
+ 62,
+ 5,
+ 124,
+ 253,
+ 39,
+ 111,
+ 206,
+ 185
+ ],
+ [
+ 161,
+ 8,
+ 295,
+ 250,
+ 76,
+ 125,
+ 79,
+ 6,
+ 106,
+ 169,
+ 78,
+ 274,
+ 80,
+ 204,
+ 3,
+ 272,
+ 59,
+ 96,
+ 72,
+ 142
+ ],
+ [
+ 13,
+ 128,
+ 206,
+ 278,
+ 160,
+ 60,
+ 254,
+ 39,
+ 112,
+ 38,
+ 145,
+ 27,
+ 137,
+ 109,
+ 57,
+ 237,
+ 267,
+ 286,
+ 17,
+ 298
+ ],
+ [
+ 160,
+ 145,
+ 71,
+ 89,
+ 13,
+ 278,
+ 253,
+ 88,
+ 206,
+ 102,
+ 234,
+ 267,
+ 57,
+ 254,
+ 17,
+ 237,
+ 128,
+ 38,
+ 55,
+ 111
+ ],
+ [
+ 152,
+ 149,
+ 250,
+ 92,
+ 59,
+ 261,
+ 178,
+ 87,
+ 258,
+ 132,
+ 190,
+ 26,
+ 294,
+ 157,
+ 119,
+ 94,
+ 106,
+ 255,
+ 142,
+ 104
+ ],
+ [
+ 156,
+ 64,
+ 12,
+ 25,
+ 279,
+ 236,
+ 146,
+ 189,
+ 215,
+ 268,
+ 276,
+ 162,
+ 99,
+ 167,
+ 243,
+ 277,
+ 133,
+ 212,
+ 218,
+ 164
+ ],
+ [
+ 46,
+ 14,
+ 67,
+ 75,
+ 198,
+ 204,
+ 231,
+ 262,
+ 130,
+ 3,
+ 161,
+ 1,
+ 80,
+ 295,
+ 169,
+ 290,
+ 272,
+ 68,
+ 79,
+ 78
+ ],
+ [
+ 45,
+ 31,
+ 84,
+ 135,
+ 218,
+ 95,
+ 120,
+ 32,
+ 229,
+ 51,
+ 65,
+ 199,
+ 58,
+ 24,
+ 207,
+ 205,
+ 257,
+ 7,
+ 53,
+ 85
+ ],
+ [
+ 145,
+ 17,
+ 86,
+ 121,
+ 91,
+ 71,
+ 253,
+ 160,
+ 185,
+ 254,
+ 278,
+ 112,
+ 89,
+ 55,
+ 134,
+ 74,
+ 88,
+ 118,
+ 234,
+ 177
+ ],
+ [
+ 15,
+ 42,
+ 105,
+ 239,
+ 22,
+ 233,
+ 86,
+ 39,
+ 123,
+ 91,
+ 38,
+ 88,
+ 298,
+ 163,
+ 185,
+ 148,
+ 124,
+ 278,
+ 168,
+ 182
+ ],
+ [
+ 157,
+ 155,
+ 208,
+ 26,
+ 178,
+ 255,
+ 227,
+ 282,
+ 132,
+ 92,
+ 0,
+ 174,
+ 294,
+ 261,
+ 87,
+ 10,
+ 152,
+ 165,
+ 258,
+ 283
+ ],
+ [
+ 65,
+ 24,
+ 19,
+ 199,
+ 116,
+ 32,
+ 220,
+ 135,
+ 95,
+ 240,
+ 20,
+ 45,
+ 44,
+ 226,
+ 136,
+ 7,
+ 61,
+ 58,
+ 36,
+ 275
+ ],
+ [
+ 55,
+ 88,
+ 253,
+ 182,
+ 234,
+ 102,
+ 278,
+ 145,
+ 160,
+ 86,
+ 267,
+ 89,
+ 163,
+ 91,
+ 259,
+ 233,
+ 60,
+ 39,
+ 74,
+ 109
+ ],
+ [
+ 56,
+ 271,
+ 201,
+ 209,
+ 29,
+ 159,
+ 97,
+ 93,
+ 288,
+ 82,
+ 200,
+ 224,
+ 173,
+ 100,
+ 191,
+ 265,
+ 244,
+ 84,
+ 146,
+ 264
+ ],
+ [
+ 239,
+ 15,
+ 91,
+ 185,
+ 298,
+ 278,
+ 39,
+ 42,
+ 131,
+ 22,
+ 259,
+ 62,
+ 118,
+ 148,
+ 254,
+ 124,
+ 38,
+ 63,
+ 5,
+ 102
+ ],
+ [
+ 286,
+ 278,
+ 267,
+ 62,
+ 234,
+ 91,
+ 131,
+ 145,
+ 137,
+ 253,
+ 185,
+ 109,
+ 60,
+ 298,
+ 112,
+ 239,
+ 88,
+ 102,
+ 57,
+ 160
+ ],
+ [
+ 43,
+ 76,
+ 6,
+ 96,
+ 3,
+ 106,
+ 8,
+ 193,
+ 78,
+ 79,
+ 204,
+ 110,
+ 295,
+ 169,
+ 161,
+ 188,
+ 1,
+ 262,
+ 143,
+ 72
+ ],
+ [
+ 114,
+ 52,
+ 121,
+ 256,
+ 163,
+ 156,
+ 182,
+ 177,
+ 12,
+ 144,
+ 247,
+ 88,
+ 62,
+ 86,
+ 168,
+ 241,
+ 55,
+ 213,
+ 270,
+ 260
+ ],
+ [
+ 58,
+ 36,
+ 61,
+ 135,
+ 154,
+ 207,
+ 45,
+ 32,
+ 23,
+ 240,
+ 53,
+ 95,
+ 199,
+ 226,
+ 44,
+ 7,
+ 136,
+ 20,
+ 49,
+ 275
+ ],
+ [
+ 13,
+ 111,
+ 60,
+ 109,
+ 278,
+ 267,
+ 160,
+ 254,
+ 27,
+ 5,
+ 137,
+ 206,
+ 145,
+ 102,
+ 182,
+ 38,
+ 39,
+ 131,
+ 112,
+ 57
+ ],
+ [
+ 37,
+ 221,
+ 138,
+ 211,
+ 197,
+ 212,
+ 77,
+ 203,
+ 236,
+ 235,
+ 2,
+ 223,
+ 133,
+ 9,
+ 277,
+ 189,
+ 162,
+ 217,
+ 263,
+ 101
+ ],
+ [
+ 46,
+ 270,
+ 67,
+ 218,
+ 115,
+ 84,
+ 205,
+ 219,
+ 116,
+ 31,
+ 229,
+ 16,
+ 68,
+ 12,
+ 169,
+ 66,
+ 73,
+ 51,
+ 14,
+ 141
+ ],
+ [
+ 259,
+ 234,
+ 62,
+ 102,
+ 137,
+ 254,
+ 267,
+ 60,
+ 278,
+ 17,
+ 109,
+ 13,
+ 89,
+ 74,
+ 253,
+ 39,
+ 5,
+ 145,
+ 239,
+ 182
+ ],
+ [
+ 174,
+ 157,
+ 119,
+ 227,
+ 238,
+ 10,
+ 255,
+ 26,
+ 261,
+ 294,
+ 208,
+ 0,
+ 178,
+ 282,
+ 92,
+ 87,
+ 222,
+ 152,
+ 151,
+ 251
+ ],
+ [
+ 37,
+ 212,
+ 9,
+ 221,
+ 197,
+ 162,
+ 263,
+ 236,
+ 34,
+ 129,
+ 211,
+ 138,
+ 223,
+ 235,
+ 268,
+ 291,
+ 2,
+ 277,
+ 296,
+ 203
+ ],
+ [
+ 71,
+ 89,
+ 86,
+ 253,
+ 88,
+ 112,
+ 145,
+ 247,
+ 234,
+ 121,
+ 102,
+ 55,
+ 278,
+ 150,
+ 233,
+ 163,
+ 289,
+ 38,
+ 259,
+ 182
+ ],
+ [
+ 58,
+ 61,
+ 32,
+ 127,
+ 207,
+ 36,
+ 45,
+ 199,
+ 154,
+ 95,
+ 240,
+ 53,
+ 23,
+ 116,
+ 136,
+ 7,
+ 65,
+ 20,
+ 19,
+ 120
+ ],
+ [
+ 240,
+ 199,
+ 7,
+ 36,
+ 53,
+ 20,
+ 95,
+ 61,
+ 58,
+ 135,
+ 23,
+ 154,
+ 65,
+ 32,
+ 207,
+ 127,
+ 49,
+ 183,
+ 45,
+ 275
+ ],
+ [
+ 60,
+ 109,
+ 278,
+ 267,
+ 254,
+ 131,
+ 62,
+ 102,
+ 128,
+ 145,
+ 286,
+ 89,
+ 13,
+ 160,
+ 124,
+ 39,
+ 259,
+ 206,
+ 185,
+ 253
+ ],
+ [
+ 9,
+ 37,
+ 129,
+ 236,
+ 133,
+ 212,
+ 221,
+ 2,
+ 203,
+ 211,
+ 77,
+ 197,
+ 223,
+ 189,
+ 34,
+ 285,
+ 263,
+ 235,
+ 54,
+ 276
+ ],
+ [
+ 232,
+ 35,
+ 108,
+ 196,
+ 171,
+ 297,
+ 50,
+ 33,
+ 186,
+ 11,
+ 69,
+ 192,
+ 245,
+ 280,
+ 70,
+ 30,
+ 266,
+ 248,
+ 287,
+ 180
+ ],
+ [
+ 165,
+ 283,
+ 178,
+ 208,
+ 92,
+ 227,
+ 81,
+ 294,
+ 119,
+ 155,
+ 10,
+ 251,
+ 238,
+ 255,
+ 157,
+ 0,
+ 152,
+ 87,
+ 132,
+ 151
+ ],
+ [
+ 66,
+ 72,
+ 68,
+ 43,
+ 8,
+ 73,
+ 183,
+ 188,
+ 295,
+ 96,
+ 3,
+ 78,
+ 169,
+ 274,
+ 80,
+ 125,
+ 142,
+ 193,
+ 106,
+ 272
+ ],
+ [
+ 59,
+ 80,
+ 250,
+ 261,
+ 149,
+ 106,
+ 290,
+ 104,
+ 113,
+ 87,
+ 295,
+ 3,
+ 252,
+ 1,
+ 190,
+ 96,
+ 92,
+ 161,
+ 166,
+ 174
+ ],
+ [
+ 3,
+ 204,
+ 104,
+ 125,
+ 76,
+ 43,
+ 6,
+ 1,
+ 142,
+ 59,
+ 96,
+ 106,
+ 8,
+ 295,
+ 190,
+ 188,
+ 110,
+ 250,
+ 195,
+ 169
+ ],
+ [
+ 215,
+ 90,
+ 167,
+ 99,
+ 296,
+ 114,
+ 194,
+ 236,
+ 279,
+ 9,
+ 276,
+ 138,
+ 291,
+ 277,
+ 54,
+ 25,
+ 162,
+ 2,
+ 34,
+ 211
+ ],
+ [
+ 253,
+ 278,
+ 112,
+ 254,
+ 60,
+ 13,
+ 234,
+ 160,
+ 89,
+ 109,
+ 267,
+ 63,
+ 17,
+ 102,
+ 148,
+ 71,
+ 131,
+ 137,
+ 185,
+ 88
+ ],
+ [
+ 229,
+ 218,
+ 191,
+ 84,
+ 31,
+ 85,
+ 51,
+ 64,
+ 279,
+ 277,
+ 47,
+ 217,
+ 215,
+ 116,
+ 24,
+ 114,
+ 257,
+ 162,
+ 268,
+ 41
+ ],
+ [
+ 66,
+ 183,
+ 188,
+ 8,
+ 16,
+ 68,
+ 43,
+ 169,
+ 262,
+ 141,
+ 73,
+ 6,
+ 136,
+ 100,
+ 106,
+ 7,
+ 72,
+ 78,
+ 48,
+ 246
+ ],
+ [
+ 63,
+ 5,
+ 254,
+ 298,
+ 39,
+ 38,
+ 239,
+ 278,
+ 145,
+ 60,
+ 131,
+ 112,
+ 42,
+ 74,
+ 234,
+ 286,
+ 22,
+ 102,
+ 123,
+ 118
+ ],
+ [
+ 113,
+ 272,
+ 166,
+ 80,
+ 274,
+ 142,
+ 59,
+ 261,
+ 132,
+ 250,
+ 87,
+ 92,
+ 290,
+ 161,
+ 190,
+ 106,
+ 152,
+ 157,
+ 26,
+ 193
+ ],
+ [
+ 86,
+ 112,
+ 88,
+ 57,
+ 71,
+ 234,
+ 267,
+ 124,
+ 237,
+ 89,
+ 160,
+ 253,
+ 38,
+ 4,
+ 102,
+ 74,
+ 145,
+ 62,
+ 278,
+ 91
+ ],
+ [
+ 216,
+ 103,
+ 255,
+ 251,
+ 10,
+ 81,
+ 208,
+ 222,
+ 181,
+ 261,
+ 294,
+ 132,
+ 26,
+ 174,
+ 227,
+ 152,
+ 119,
+ 0,
+ 165,
+ 178
+ ],
+ [
+ 178,
+ 92,
+ 255,
+ 294,
+ 87,
+ 119,
+ 227,
+ 165,
+ 261,
+ 238,
+ 132,
+ 113,
+ 94,
+ 283,
+ 157,
+ 26,
+ 10,
+ 155,
+ 282,
+ 208
+ ],
+ [
+ 181,
+ 292,
+ 0,
+ 242,
+ 81,
+ 249,
+ 87,
+ 216,
+ 261,
+ 273,
+ 227,
+ 252,
+ 92,
+ 10,
+ 187,
+ 283,
+ 157,
+ 103,
+ 294,
+ 208
+ ],
+ [
+ 61,
+ 240,
+ 58,
+ 127,
+ 135,
+ 199,
+ 23,
+ 32,
+ 36,
+ 45,
+ 20,
+ 95,
+ 207,
+ 44,
+ 7,
+ 49,
+ 136,
+ 275,
+ 53,
+ 19
+ ],
+ [
+ 119,
+ 178,
+ 208,
+ 227,
+ 157,
+ 282,
+ 255,
+ 165,
+ 92,
+ 294,
+ 258,
+ 152,
+ 283,
+ 26,
+ 87,
+ 94,
+ 0,
+ 174,
+ 10,
+ 132
+ ],
+ [
+ 236,
+ 203,
+ 25,
+ 189,
+ 2,
+ 212,
+ 133,
+ 129,
+ 77,
+ 37,
+ 138,
+ 54,
+ 162,
+ 211,
+ 34,
+ 176,
+ 291,
+ 268,
+ 197,
+ 221
+ ],
+ [
+ 119,
+ 174,
+ 132,
+ 282,
+ 26,
+ 255,
+ 0,
+ 92,
+ 227,
+ 178,
+ 87,
+ 261,
+ 155,
+ 208,
+ 294,
+ 152,
+ 283,
+ 258,
+ 10,
+ 81
+ ],
+ [
+ 210,
+ 252,
+ 67,
+ 0,
+ 181,
+ 132,
+ 249,
+ 153,
+ 157,
+ 292,
+ 219,
+ 174,
+ 119,
+ 103,
+ 238,
+ 216,
+ 251,
+ 242,
+ 227,
+ 273
+ ],
+ [
+ 209,
+ 201,
+ 56,
+ 93,
+ 97,
+ 122,
+ 29,
+ 271,
+ 82,
+ 288,
+ 200,
+ 224,
+ 265,
+ 100,
+ 173,
+ 191,
+ 244,
+ 84,
+ 264,
+ 146
+ ],
+ [
+ 13,
+ 112,
+ 206,
+ 145,
+ 278,
+ 89,
+ 128,
+ 267,
+ 253,
+ 60,
+ 102,
+ 62,
+ 17,
+ 111,
+ 109,
+ 57,
+ 131,
+ 137,
+ 234,
+ 88
+ ],
+ [
+ 262,
+ 79,
+ 80,
+ 290,
+ 169,
+ 8,
+ 110,
+ 272,
+ 193,
+ 274,
+ 78,
+ 106,
+ 3,
+ 43,
+ 6,
+ 295,
+ 125,
+ 149,
+ 1,
+ 142
+ ],
+ [
+ 221,
+ 133,
+ 37,
+ 212,
+ 9,
+ 129,
+ 34,
+ 197,
+ 203,
+ 296,
+ 235,
+ 217,
+ 211,
+ 215,
+ 189,
+ 25,
+ 236,
+ 138,
+ 291,
+ 263
+ ],
+ [
+ 88,
+ 168,
+ 71,
+ 86,
+ 233,
+ 182,
+ 118,
+ 121,
+ 253,
+ 105,
+ 234,
+ 39,
+ 4,
+ 145,
+ 89,
+ 15,
+ 55,
+ 42,
+ 112,
+ 259
+ ],
+ [
+ 241,
+ 107,
+ 268,
+ 34,
+ 211,
+ 2,
+ 197,
+ 296,
+ 133,
+ 260,
+ 221,
+ 263,
+ 162,
+ 9,
+ 215,
+ 37,
+ 212,
+ 129,
+ 236,
+ 176
+ ],
+ [
+ 152,
+ 140,
+ 178,
+ 92,
+ 283,
+ 208,
+ 227,
+ 10,
+ 294,
+ 119,
+ 155,
+ 174,
+ 255,
+ 26,
+ 261,
+ 238,
+ 251,
+ 81,
+ 132,
+ 157
+ ],
+ [
+ 149,
+ 252,
+ 273,
+ 181,
+ 216,
+ 142,
+ 59,
+ 261,
+ 292,
+ 80,
+ 0,
+ 151,
+ 92,
+ 174,
+ 272,
+ 103,
+ 113,
+ 153,
+ 157,
+ 26
+ ],
+ [
+ 215,
+ 296,
+ 162,
+ 25,
+ 9,
+ 133,
+ 34,
+ 263,
+ 221,
+ 194,
+ 236,
+ 138,
+ 37,
+ 90,
+ 176,
+ 276,
+ 277,
+ 129,
+ 291,
+ 212
+ ],
+ [
+ 163,
+ 118,
+ 105,
+ 182,
+ 233,
+ 88,
+ 121,
+ 148,
+ 38,
+ 253,
+ 15,
+ 55,
+ 86,
+ 145,
+ 239,
+ 234,
+ 60,
+ 39,
+ 42,
+ 123
+ ],
+ [
+ 8,
+ 68,
+ 161,
+ 66,
+ 43,
+ 96,
+ 79,
+ 262,
+ 295,
+ 188,
+ 110,
+ 6,
+ 125,
+ 3,
+ 72,
+ 1,
+ 80,
+ 106,
+ 193,
+ 78
+ ],
+ [
+ 225,
+ 50,
+ 30,
+ 180,
+ 284,
+ 232,
+ 35,
+ 248,
+ 280,
+ 266,
+ 192,
+ 139,
+ 196,
+ 287,
+ 186,
+ 70,
+ 69,
+ 108,
+ 33,
+ 297
+ ],
+ [
+ 69,
+ 33,
+ 245,
+ 108,
+ 266,
+ 280,
+ 297,
+ 70,
+ 196,
+ 11,
+ 139,
+ 35,
+ 180,
+ 192,
+ 50,
+ 30,
+ 186,
+ 232,
+ 248,
+ 284
+ ],
+ [
+ 98,
+ 293,
+ 195,
+ 208,
+ 269,
+ 165,
+ 92,
+ 251,
+ 190,
+ 227,
+ 10,
+ 103,
+ 81,
+ 174,
+ 294,
+ 261,
+ 238,
+ 119,
+ 140,
+ 59
+ ],
+ [
+ 224,
+ 207,
+ 264,
+ 244,
+ 265,
+ 56,
+ 100,
+ 159,
+ 93,
+ 122,
+ 36,
+ 97,
+ 271,
+ 205,
+ 82,
+ 246,
+ 288,
+ 201,
+ 209,
+ 84
+ ],
+ [
+ 157,
+ 132,
+ 261,
+ 26,
+ 119,
+ 92,
+ 227,
+ 208,
+ 294,
+ 222,
+ 255,
+ 10,
+ 87,
+ 178,
+ 165,
+ 0,
+ 238,
+ 282,
+ 181,
+ 152
+ ],
+ [
+ 230,
+ 185,
+ 298,
+ 15,
+ 239,
+ 62,
+ 123,
+ 5,
+ 42,
+ 83,
+ 22,
+ 63,
+ 91,
+ 124,
+ 18,
+ 118,
+ 254,
+ 131,
+ 286,
+ 267
+ ],
+ [
+ 281,
+ 221,
+ 133,
+ 138,
+ 54,
+ 235,
+ 34,
+ 77,
+ 129,
+ 37,
+ 223,
+ 162,
+ 203,
+ 211,
+ 197,
+ 189,
+ 212,
+ 9,
+ 101,
+ 236
+ ],
+ [
+ 160,
+ 13,
+ 184,
+ 206,
+ 55,
+ 121,
+ 145,
+ 112,
+ 128,
+ 111,
+ 102,
+ 86,
+ 278,
+ 17,
+ 233,
+ 71,
+ 253,
+ 88,
+ 89,
+ 237
+ ],
+ [
+ 92,
+ 155,
+ 282,
+ 119,
+ 152,
+ 255,
+ 294,
+ 258,
+ 94,
+ 157,
+ 208,
+ 227,
+ 87,
+ 26,
+ 165,
+ 81,
+ 283,
+ 132,
+ 261,
+ 174
+ ],
+ [
+ 276,
+ 211,
+ 138,
+ 129,
+ 9,
+ 203,
+ 243,
+ 34,
+ 37,
+ 221,
+ 296,
+ 101,
+ 162,
+ 197,
+ 189,
+ 133,
+ 77,
+ 2,
+ 212,
+ 235
+ ],
+ [
+ 50,
+ 192,
+ 266,
+ 35,
+ 284,
+ 232,
+ 11,
+ 248,
+ 280,
+ 70,
+ 33,
+ 108,
+ 69,
+ 297,
+ 171,
+ 186,
+ 30,
+ 245,
+ 139,
+ 196
+ ],
+ [
+ 0,
+ 292,
+ 255,
+ 216,
+ 261,
+ 294,
+ 273,
+ 151,
+ 10,
+ 81,
+ 208,
+ 87,
+ 174,
+ 153,
+ 227,
+ 157,
+ 282,
+ 252,
+ 119,
+ 178
+ ],
+ [
+ 109,
+ 253,
+ 278,
+ 102,
+ 145,
+ 60,
+ 88,
+ 62,
+ 160,
+ 39,
+ 13,
+ 131,
+ 234,
+ 185,
+ 267,
+ 121,
+ 254,
+ 163,
+ 112,
+ 128
+ ],
+ [
+ 141,
+ 66,
+ 275,
+ 7,
+ 49,
+ 154,
+ 136,
+ 20,
+ 240,
+ 199,
+ 188,
+ 147,
+ 44,
+ 23,
+ 58,
+ 246,
+ 61,
+ 68,
+ 169,
+ 226
+ ],
+ [
+ 299,
+ 160,
+ 17,
+ 206,
+ 234,
+ 13,
+ 112,
+ 89,
+ 102,
+ 267,
+ 253,
+ 259,
+ 131,
+ 111,
+ 55,
+ 74,
+ 237,
+ 145,
+ 278,
+ 71
+ ],
+ [
+ 91,
+ 175,
+ 62,
+ 254,
+ 278,
+ 145,
+ 239,
+ 123,
+ 60,
+ 267,
+ 124,
+ 39,
+ 286,
+ 182,
+ 102,
+ 137,
+ 63,
+ 298,
+ 15,
+ 148
+ ],
+ [
+ 232,
+ 196,
+ 248,
+ 50,
+ 35,
+ 30,
+ 287,
+ 284,
+ 11,
+ 266,
+ 33,
+ 139,
+ 70,
+ 180,
+ 280,
+ 245,
+ 171,
+ 192,
+ 108,
+ 69
+ ],
+ [
+ 208,
+ 87,
+ 292,
+ 261,
+ 104,
+ 153,
+ 227,
+ 26,
+ 174,
+ 294,
+ 92,
+ 0,
+ 10,
+ 157,
+ 119,
+ 81,
+ 132,
+ 178,
+ 181,
+ 165
+ ],
+ [
+ 68,
+ 66,
+ 16,
+ 204,
+ 73,
+ 169,
+ 125,
+ 76,
+ 96,
+ 72,
+ 3,
+ 141,
+ 8,
+ 43,
+ 183,
+ 295,
+ 1,
+ 193,
+ 6,
+ 110
+ ],
+ [
+ 77,
+ 101,
+ 236,
+ 212,
+ 197,
+ 211,
+ 2,
+ 235,
+ 37,
+ 203,
+ 277,
+ 9,
+ 221,
+ 129,
+ 133,
+ 138,
+ 162,
+ 217,
+ 268,
+ 34
+ ],
+ [
+ 92,
+ 26,
+ 258,
+ 119,
+ 113,
+ 178,
+ 98,
+ 269,
+ 132,
+ 282,
+ 294,
+ 157,
+ 250,
+ 293,
+ 208,
+ 94,
+ 152,
+ 174,
+ 59,
+ 10
+ ],
+ [
+ 47,
+ 51,
+ 229,
+ 146,
+ 84,
+ 257,
+ 218,
+ 31,
+ 85,
+ 116,
+ 28,
+ 217,
+ 244,
+ 64,
+ 279,
+ 205,
+ 219,
+ 277,
+ 24,
+ 265
+ ],
+ [
+ 35,
+ 50,
+ 180,
+ 11,
+ 232,
+ 280,
+ 33,
+ 70,
+ 284,
+ 266,
+ 108,
+ 248,
+ 139,
+ 297,
+ 69,
+ 245,
+ 186,
+ 171,
+ 30,
+ 196
+ ],
+ [
+ 3,
+ 1,
+ 125,
+ 76,
+ 161,
+ 290,
+ 295,
+ 43,
+ 204,
+ 272,
+ 106,
+ 96,
+ 169,
+ 80,
+ 8,
+ 72,
+ 6,
+ 142,
+ 68,
+ 78
+ ],
+ [
+ 90,
+ 260,
+ 291,
+ 215,
+ 99,
+ 167,
+ 263,
+ 268,
+ 277,
+ 236,
+ 77,
+ 189,
+ 197,
+ 9,
+ 296,
+ 276,
+ 101,
+ 129,
+ 279,
+ 212
+ ],
+ [
+ 98,
+ 293,
+ 269,
+ 172,
+ 227,
+ 140,
+ 190,
+ 178,
+ 92,
+ 21,
+ 251,
+ 165,
+ 208,
+ 283,
+ 155,
+ 238,
+ 152,
+ 282,
+ 119,
+ 294
+ ],
+ [
+ 186,
+ 30,
+ 287,
+ 171,
+ 232,
+ 69,
+ 248,
+ 139,
+ 284,
+ 108,
+ 35,
+ 280,
+ 33,
+ 245,
+ 266,
+ 50,
+ 11,
+ 297,
+ 70,
+ 180
+ ],
+ [
+ 212,
+ 211,
+ 223,
+ 291,
+ 277,
+ 37,
+ 77,
+ 263,
+ 235,
+ 9,
+ 133,
+ 221,
+ 101,
+ 129,
+ 217,
+ 189,
+ 34,
+ 268,
+ 203,
+ 236
+ ],
+ [
+ 75,
+ 14,
+ 231,
+ 115,
+ 46,
+ 214,
+ 204,
+ 295,
+ 76,
+ 67,
+ 6,
+ 1,
+ 48,
+ 3,
+ 169,
+ 68,
+ 188,
+ 79,
+ 262,
+ 130
+ ],
+ [
+ 32,
+ 240,
+ 135,
+ 20,
+ 95,
+ 154,
+ 136,
+ 45,
+ 7,
+ 127,
+ 61,
+ 19,
+ 58,
+ 36,
+ 44,
+ 53,
+ 23,
+ 65,
+ 120,
+ 275
+ ],
+ [
+ 93,
+ 288,
+ 209,
+ 122,
+ 159,
+ 97,
+ 271,
+ 224,
+ 56,
+ 201,
+ 82,
+ 29,
+ 265,
+ 84,
+ 100,
+ 127,
+ 244,
+ 36,
+ 191,
+ 51
+ ],
+ [
+ 29,
+ 97,
+ 271,
+ 93,
+ 209,
+ 56,
+ 122,
+ 159,
+ 288,
+ 82,
+ 200,
+ 224,
+ 264,
+ 100,
+ 173,
+ 258,
+ 191,
+ 265,
+ 190,
+ 94
+ ],
+ [
+ 228,
+ 247,
+ 144,
+ 114,
+ 156,
+ 86,
+ 12,
+ 230,
+ 71,
+ 62,
+ 177,
+ 237,
+ 39,
+ 286,
+ 38,
+ 4,
+ 137,
+ 176,
+ 167,
+ 160
+ ],
+ [
+ 2,
+ 37,
+ 129,
+ 77,
+ 212,
+ 211,
+ 197,
+ 189,
+ 138,
+ 235,
+ 223,
+ 133,
+ 9,
+ 34,
+ 25,
+ 162,
+ 291,
+ 221,
+ 277,
+ 296
+ ],
+ [
+ 3,
+ 76,
+ 188,
+ 125,
+ 295,
+ 143,
+ 1,
+ 110,
+ 8,
+ 68,
+ 6,
+ 96,
+ 193,
+ 72,
+ 43,
+ 169,
+ 161,
+ 79,
+ 106,
+ 78
+ ],
+ [
+ 257,
+ 31,
+ 84,
+ 218,
+ 229,
+ 85,
+ 116,
+ 219,
+ 173,
+ 51,
+ 264,
+ 45,
+ 224,
+ 7,
+ 265,
+ 100,
+ 244,
+ 207,
+ 135,
+ 58
+ ],
+ [
+ 13,
+ 160,
+ 299,
+ 111,
+ 60,
+ 112,
+ 128,
+ 254,
+ 137,
+ 17,
+ 102,
+ 145,
+ 89,
+ 267,
+ 278,
+ 109,
+ 184,
+ 131,
+ 57,
+ 237
+ ],
+ [
+ 36,
+ 58,
+ 127,
+ 53,
+ 61,
+ 135,
+ 45,
+ 246,
+ 173,
+ 95,
+ 7,
+ 264,
+ 23,
+ 32,
+ 154,
+ 244,
+ 136,
+ 240,
+ 224,
+ 265
+ ],
+ [
+ 119,
+ 227,
+ 10,
+ 294,
+ 155,
+ 282,
+ 174,
+ 178,
+ 255,
+ 157,
+ 222,
+ 26,
+ 132,
+ 261,
+ 165,
+ 0,
+ 251,
+ 92,
+ 81,
+ 151
+ ],
+ [
+ 271,
+ 97,
+ 201,
+ 56,
+ 93,
+ 29,
+ 288,
+ 159,
+ 82,
+ 122,
+ 200,
+ 40,
+ 258,
+ 255,
+ 119,
+ 282,
+ 224,
+ 283,
+ 26,
+ 190
+ ],
+ [
+ 158,
+ 40,
+ 132,
+ 227,
+ 0,
+ 214,
+ 157,
+ 252,
+ 283,
+ 119,
+ 251,
+ 87,
+ 208,
+ 187,
+ 238,
+ 26,
+ 140,
+ 181,
+ 282,
+ 174
+ ],
+ [
+ 197,
+ 223,
+ 9,
+ 2,
+ 212,
+ 37,
+ 129,
+ 189,
+ 235,
+ 203,
+ 77,
+ 221,
+ 138,
+ 133,
+ 236,
+ 277,
+ 101,
+ 291,
+ 34,
+ 162
+ ],
+ [
+ 37,
+ 197,
+ 133,
+ 9,
+ 263,
+ 217,
+ 77,
+ 291,
+ 223,
+ 221,
+ 277,
+ 211,
+ 235,
+ 236,
+ 129,
+ 162,
+ 203,
+ 268,
+ 189,
+ 138
+ ],
+ [
+ 22,
+ 42,
+ 239,
+ 118,
+ 5,
+ 298,
+ 15,
+ 148,
+ 123,
+ 105,
+ 39,
+ 126,
+ 4,
+ 175,
+ 63,
+ 124,
+ 163,
+ 38,
+ 86,
+ 91
+ ],
+ [
+ 190,
+ 269,
+ 142,
+ 14,
+ 198,
+ 75,
+ 80,
+ 195,
+ 293,
+ 272,
+ 172,
+ 98,
+ 161,
+ 290,
+ 274,
+ 59,
+ 187,
+ 295,
+ 115,
+ 110
+ ],
+ [
+ 296,
+ 167,
+ 263,
+ 291,
+ 9,
+ 133,
+ 37,
+ 162,
+ 268,
+ 25,
+ 236,
+ 277,
+ 34,
+ 129,
+ 90,
+ 212,
+ 221,
+ 197,
+ 138,
+ 2
+ ],
+ [
+ 151,
+ 181,
+ 273,
+ 10,
+ 261,
+ 255,
+ 0,
+ 208,
+ 103,
+ 292,
+ 81,
+ 251,
+ 294,
+ 119,
+ 222,
+ 26,
+ 174,
+ 132,
+ 157,
+ 227
+ ],
+ [
+ 212,
+ 277,
+ 263,
+ 197,
+ 37,
+ 221,
+ 235,
+ 129,
+ 223,
+ 77,
+ 162,
+ 101,
+ 268,
+ 285,
+ 41,
+ 133,
+ 9,
+ 189,
+ 279,
+ 211
+ ],
+ [
+ 84,
+ 257,
+ 51,
+ 31,
+ 229,
+ 116,
+ 205,
+ 146,
+ 191,
+ 244,
+ 47,
+ 64,
+ 219,
+ 85,
+ 100,
+ 265,
+ 224,
+ 7,
+ 246,
+ 28
+ ],
+ [
+ 218,
+ 205,
+ 84,
+ 257,
+ 229,
+ 31,
+ 130,
+ 67,
+ 85,
+ 158,
+ 16,
+ 173,
+ 224,
+ 290,
+ 191,
+ 51,
+ 116,
+ 146,
+ 47,
+ 214
+ ],
+ [
+ 226,
+ 120,
+ 275,
+ 24,
+ 20,
+ 44,
+ 65,
+ 199,
+ 19,
+ 240,
+ 32,
+ 136,
+ 7,
+ 141,
+ 73,
+ 183,
+ 16,
+ 66,
+ 135,
+ 68
+ ],
+ [
+ 235,
+ 133,
+ 197,
+ 129,
+ 162,
+ 212,
+ 37,
+ 9,
+ 211,
+ 77,
+ 263,
+ 223,
+ 217,
+ 138,
+ 236,
+ 101,
+ 34,
+ 281,
+ 2,
+ 189
+ ],
+ [
+ 294,
+ 208,
+ 174,
+ 26,
+ 151,
+ 132,
+ 119,
+ 238,
+ 178,
+ 10,
+ 227,
+ 255,
+ 282,
+ 251,
+ 157,
+ 92,
+ 261,
+ 152,
+ 181,
+ 165
+ ],
+ [
+ 197,
+ 211,
+ 212,
+ 235,
+ 77,
+ 129,
+ 37,
+ 285,
+ 221,
+ 133,
+ 203,
+ 277,
+ 217,
+ 263,
+ 2,
+ 101,
+ 138,
+ 9,
+ 291,
+ 34
+ ],
+ [
+ 265,
+ 84,
+ 244,
+ 159,
+ 100,
+ 173,
+ 200,
+ 51,
+ 93,
+ 122,
+ 82,
+ 97,
+ 56,
+ 36,
+ 201,
+ 209,
+ 288,
+ 271,
+ 207,
+ 29
+ ],
+ [
+ 170,
+ 280,
+ 266,
+ 297,
+ 69,
+ 245,
+ 180,
+ 54,
+ 70,
+ 33,
+ 171,
+ 108,
+ 50,
+ 30,
+ 192,
+ 281,
+ 211,
+ 223,
+ 176,
+ 11
+ ],
+ [
+ 44,
+ 275,
+ 61,
+ 127,
+ 45,
+ 23,
+ 32,
+ 199,
+ 135,
+ 207,
+ 36,
+ 20,
+ 154,
+ 19,
+ 65,
+ 58,
+ 49,
+ 240,
+ 95,
+ 24
+ ],
+ [
+ 208,
+ 119,
+ 87,
+ 157,
+ 132,
+ 0,
+ 92,
+ 155,
+ 174,
+ 178,
+ 255,
+ 282,
+ 165,
+ 273,
+ 261,
+ 152,
+ 10,
+ 283,
+ 294,
+ 251
+ ],
+ [
+ 167,
+ 296,
+ 179,
+ 176,
+ 34,
+ 25,
+ 162,
+ 215,
+ 203,
+ 144,
+ 133,
+ 2,
+ 276,
+ 90,
+ 37,
+ 138,
+ 9,
+ 279,
+ 211,
+ 243
+ ],
+ [
+ 85,
+ 51,
+ 31,
+ 84,
+ 257,
+ 47,
+ 218,
+ 191,
+ 116,
+ 64,
+ 146,
+ 205,
+ 28,
+ 219,
+ 12,
+ 45,
+ 95,
+ 130,
+ 279,
+ 265
+ ],
+ [
+ 83,
+ 175,
+ 298,
+ 137,
+ 286,
+ 62,
+ 38,
+ 39,
+ 5,
+ 128,
+ 60,
+ 254,
+ 267,
+ 278,
+ 91,
+ 124,
+ 63,
+ 109,
+ 239,
+ 148
+ ],
+ [
+ 198,
+ 75,
+ 14,
+ 115,
+ 46,
+ 214,
+ 48,
+ 204,
+ 262,
+ 6,
+ 79,
+ 295,
+ 8,
+ 78,
+ 67,
+ 76,
+ 143,
+ 3,
+ 269,
+ 68
+ ],
+ [
+ 35,
+ 50,
+ 186,
+ 248,
+ 284,
+ 192,
+ 139,
+ 196,
+ 180,
+ 11,
+ 30,
+ 70,
+ 287,
+ 266,
+ 280,
+ 33,
+ 108,
+ 245,
+ 69,
+ 297
+ ],
+ [
+ 105,
+ 118,
+ 15,
+ 42,
+ 88,
+ 239,
+ 22,
+ 123,
+ 86,
+ 182,
+ 163,
+ 278,
+ 38,
+ 102,
+ 267,
+ 185,
+ 234,
+ 91,
+ 4,
+ 71
+ ],
+ [
+ 74,
+ 102,
+ 131,
+ 253,
+ 17,
+ 145,
+ 86,
+ 278,
+ 89,
+ 267,
+ 259,
+ 13,
+ 39,
+ 55,
+ 124,
+ 182,
+ 112,
+ 254,
+ 91,
+ 160
+ ],
+ [
+ 197,
+ 221,
+ 211,
+ 101,
+ 77,
+ 223,
+ 37,
+ 212,
+ 129,
+ 2,
+ 285,
+ 34,
+ 203,
+ 133,
+ 189,
+ 217,
+ 277,
+ 9,
+ 281,
+ 162
+ ],
+ [
+ 9,
+ 212,
+ 189,
+ 133,
+ 268,
+ 77,
+ 263,
+ 138,
+ 37,
+ 129,
+ 221,
+ 54,
+ 197,
+ 277,
+ 291,
+ 2,
+ 211,
+ 25,
+ 223,
+ 162
+ ],
+ [
+ 27,
+ 278,
+ 60,
+ 13,
+ 160,
+ 38,
+ 112,
+ 71,
+ 39,
+ 111,
+ 137,
+ 89,
+ 145,
+ 109,
+ 102,
+ 206,
+ 128,
+ 57,
+ 254,
+ 267
+ ],
+ [
+ 251,
+ 132,
+ 10,
+ 294,
+ 152,
+ 178,
+ 208,
+ 174,
+ 119,
+ 92,
+ 222,
+ 227,
+ 26,
+ 165,
+ 155,
+ 282,
+ 157,
+ 255,
+ 140,
+ 151
+ ],
+ [
+ 123,
+ 39,
+ 298,
+ 38,
+ 15,
+ 42,
+ 91,
+ 118,
+ 278,
+ 22,
+ 148,
+ 286,
+ 185,
+ 5,
+ 254,
+ 131,
+ 63,
+ 62,
+ 124,
+ 259
+ ],
+ [
+ 154,
+ 61,
+ 20,
+ 32,
+ 58,
+ 199,
+ 127,
+ 36,
+ 135,
+ 136,
+ 7,
+ 95,
+ 23,
+ 53,
+ 207,
+ 246,
+ 45,
+ 49,
+ 19,
+ 65
+ ],
+ [
+ 164,
+ 107,
+ 34,
+ 268,
+ 211,
+ 197,
+ 2,
+ 221,
+ 260,
+ 133,
+ 90,
+ 176,
+ 37,
+ 263,
+ 236,
+ 129,
+ 9,
+ 279,
+ 296,
+ 235
+ ],
+ [
+ 249,
+ 153,
+ 252,
+ 251,
+ 10,
+ 151,
+ 81,
+ 103,
+ 181,
+ 292,
+ 294,
+ 216,
+ 261,
+ 140,
+ 165,
+ 92,
+ 227,
+ 174,
+ 0,
+ 152
+ ],
+ [
+ 179,
+ 90,
+ 99,
+ 9,
+ 296,
+ 268,
+ 291,
+ 236,
+ 276,
+ 194,
+ 133,
+ 211,
+ 212,
+ 260,
+ 2,
+ 189,
+ 263,
+ 37,
+ 138,
+ 203
+ ],
+ [
+ 224,
+ 265,
+ 84,
+ 100,
+ 207,
+ 218,
+ 173,
+ 36,
+ 51,
+ 191,
+ 82,
+ 257,
+ 246,
+ 146,
+ 93,
+ 127,
+ 159,
+ 58,
+ 31,
+ 122
+ ],
+ [
+ 69,
+ 108,
+ 171,
+ 297,
+ 280,
+ 33,
+ 266,
+ 70,
+ 11,
+ 196,
+ 35,
+ 139,
+ 192,
+ 180,
+ 50,
+ 186,
+ 232,
+ 30,
+ 248,
+ 284
+ ],
+ [
+ 7,
+ 36,
+ 207,
+ 100,
+ 127,
+ 95,
+ 264,
+ 58,
+ 53,
+ 240,
+ 61,
+ 84,
+ 136,
+ 244,
+ 32,
+ 224,
+ 154,
+ 199,
+ 173,
+ 20
+ ],
+ [
+ 234,
+ 86,
+ 39,
+ 121,
+ 102,
+ 259,
+ 118,
+ 74,
+ 233,
+ 182,
+ 38,
+ 278,
+ 253,
+ 134,
+ 168,
+ 91,
+ 88,
+ 71,
+ 145,
+ 289
+ ],
+ [
+ 284,
+ 232,
+ 186,
+ 30,
+ 50,
+ 196,
+ 35,
+ 180,
+ 287,
+ 11,
+ 70,
+ 192,
+ 139,
+ 266,
+ 280,
+ 33,
+ 108,
+ 245,
+ 171,
+ 69
+ ],
+ [
+ 242,
+ 153,
+ 261,
+ 292,
+ 252,
+ 81,
+ 181,
+ 151,
+ 216,
+ 152,
+ 10,
+ 0,
+ 251,
+ 87,
+ 92,
+ 174,
+ 227,
+ 165,
+ 255,
+ 103
+ ],
+ [
+ 113,
+ 59,
+ 92,
+ 104,
+ 106,
+ 94,
+ 190,
+ 178,
+ 110,
+ 152,
+ 142,
+ 258,
+ 294,
+ 149,
+ 161,
+ 282,
+ 155,
+ 87,
+ 261,
+ 79
+ ],
+ [
+ 10,
+ 151,
+ 238,
+ 208,
+ 294,
+ 227,
+ 103,
+ 132,
+ 81,
+ 222,
+ 165,
+ 242,
+ 152,
+ 216,
+ 92,
+ 174,
+ 119,
+ 261,
+ 21,
+ 26
+ ],
+ [
+ 242,
+ 181,
+ 103,
+ 261,
+ 10,
+ 251,
+ 132,
+ 174,
+ 0,
+ 292,
+ 227,
+ 151,
+ 273,
+ 92,
+ 153,
+ 216,
+ 87,
+ 157,
+ 294,
+ 249
+ ],
+ [
+ 145,
+ 234,
+ 88,
+ 131,
+ 112,
+ 278,
+ 102,
+ 267,
+ 160,
+ 182,
+ 55,
+ 89,
+ 60,
+ 124,
+ 121,
+ 17,
+ 74,
+ 109,
+ 13,
+ 254
+ ],
+ [
+ 60,
+ 145,
+ 278,
+ 137,
+ 148,
+ 13,
+ 131,
+ 267,
+ 39,
+ 102,
+ 185,
+ 109,
+ 89,
+ 5,
+ 128,
+ 112,
+ 234,
+ 111,
+ 62,
+ 259
+ ],
+ [
+ 119,
+ 178,
+ 157,
+ 282,
+ 152,
+ 151,
+ 181,
+ 261,
+ 132,
+ 0,
+ 227,
+ 208,
+ 87,
+ 174,
+ 292,
+ 155,
+ 294,
+ 26,
+ 92,
+ 10
+ ],
+ [
+ 52,
+ 265,
+ 224,
+ 99,
+ 240,
+ 20,
+ 154,
+ 146,
+ 244,
+ 61,
+ 120,
+ 218,
+ 84,
+ 64,
+ 28,
+ 114,
+ 167,
+ 194,
+ 32,
+ 173
+ ],
+ [
+ 31,
+ 51,
+ 229,
+ 84,
+ 205,
+ 85,
+ 218,
+ 191,
+ 47,
+ 116,
+ 146,
+ 244,
+ 64,
+ 224,
+ 219,
+ 265,
+ 264,
+ 100,
+ 45,
+ 173
+ ],
+ [
+ 178,
+ 94,
+ 282,
+ 119,
+ 157,
+ 283,
+ 87,
+ 155,
+ 294,
+ 152,
+ 255,
+ 92,
+ 26,
+ 208,
+ 174,
+ 165,
+ 132,
+ 0,
+ 113,
+ 238
+ ],
+ [
+ 131,
+ 102,
+ 89,
+ 62,
+ 234,
+ 17,
+ 13,
+ 39,
+ 278,
+ 160,
+ 60,
+ 254,
+ 137,
+ 74,
+ 267,
+ 253,
+ 91,
+ 5,
+ 71,
+ 145
+ ],
+ [
+ 268,
+ 263,
+ 291,
+ 194,
+ 277,
+ 9,
+ 197,
+ 236,
+ 41,
+ 54,
+ 77,
+ 212,
+ 217,
+ 101,
+ 221,
+ 129,
+ 133,
+ 296,
+ 34,
+ 189
+ ],
+ [
+ 174,
+ 87,
+ 157,
+ 132,
+ 255,
+ 152,
+ 208,
+ 10,
+ 92,
+ 119,
+ 227,
+ 294,
+ 181,
+ 178,
+ 0,
+ 151,
+ 26,
+ 292,
+ 59,
+ 165
+ ],
+ [
+ 161,
+ 79,
+ 8,
+ 43,
+ 78,
+ 169,
+ 80,
+ 290,
+ 274,
+ 106,
+ 6,
+ 125,
+ 272,
+ 3,
+ 193,
+ 110,
+ 68,
+ 67,
+ 295,
+ 72
+ ],
+ [
+ 268,
+ 197,
+ 212,
+ 277,
+ 291,
+ 217,
+ 9,
+ 133,
+ 236,
+ 221,
+ 37,
+ 77,
+ 223,
+ 129,
+ 41,
+ 235,
+ 54,
+ 138,
+ 34,
+ 215
+ ],
+ [
+ 100,
+ 7,
+ 246,
+ 207,
+ 84,
+ 173,
+ 53,
+ 257,
+ 31,
+ 224,
+ 36,
+ 95,
+ 205,
+ 58,
+ 265,
+ 136,
+ 45,
+ 61,
+ 244,
+ 51
+ ],
+ [
+ 224,
+ 244,
+ 84,
+ 51,
+ 173,
+ 100,
+ 146,
+ 82,
+ 159,
+ 207,
+ 218,
+ 191,
+ 93,
+ 200,
+ 36,
+ 31,
+ 64,
+ 257,
+ 264,
+ 122
+ ],
+ [
+ 280,
+ 171,
+ 245,
+ 69,
+ 33,
+ 108,
+ 11,
+ 70,
+ 297,
+ 180,
+ 30,
+ 50,
+ 186,
+ 35,
+ 196,
+ 192,
+ 232,
+ 248,
+ 139,
+ 284
+ ],
+ [
+ 89,
+ 278,
+ 60,
+ 137,
+ 102,
+ 234,
+ 124,
+ 128,
+ 254,
+ 131,
+ 160,
+ 109,
+ 253,
+ 13,
+ 145,
+ 112,
+ 91,
+ 62,
+ 88,
+ 182
+ ],
+ [
+ 263,
+ 212,
+ 236,
+ 197,
+ 133,
+ 9,
+ 277,
+ 291,
+ 221,
+ 37,
+ 217,
+ 129,
+ 41,
+ 77,
+ 54,
+ 211,
+ 260,
+ 189,
+ 223,
+ 101
+ ],
+ [
+ 195,
+ 293,
+ 98,
+ 190,
+ 172,
+ 283,
+ 238,
+ 214,
+ 94,
+ 250,
+ 165,
+ 140,
+ 92,
+ 178,
+ 21,
+ 208,
+ 251,
+ 227,
+ 258,
+ 152
+ ],
+ [
+ 273,
+ 152,
+ 292,
+ 165,
+ 187,
+ 151,
+ 81,
+ 294,
+ 251,
+ 242,
+ 10,
+ 103,
+ 261,
+ 140,
+ 132,
+ 255,
+ 208,
+ 238,
+ 181,
+ 216
+ ],
+ [
+ 288,
+ 56,
+ 97,
+ 209,
+ 29,
+ 201,
+ 93,
+ 122,
+ 82,
+ 159,
+ 200,
+ 40,
+ 100,
+ 224,
+ 173,
+ 258,
+ 157,
+ 282,
+ 255,
+ 132
+ ],
+ [
+ 149,
+ 274,
+ 161,
+ 80,
+ 72,
+ 193,
+ 78,
+ 8,
+ 290,
+ 68,
+ 166,
+ 295,
+ 106,
+ 169,
+ 262,
+ 67,
+ 3,
+ 142,
+ 110,
+ 43
+ ],
+ [
+ 0,
+ 227,
+ 181,
+ 292,
+ 216,
+ 208,
+ 255,
+ 119,
+ 10,
+ 165,
+ 151,
+ 132,
+ 261,
+ 157,
+ 81,
+ 174,
+ 294,
+ 87,
+ 251,
+ 222
+ ],
+ [
+ 80,
+ 78,
+ 272,
+ 161,
+ 43,
+ 262,
+ 8,
+ 79,
+ 169,
+ 290,
+ 149,
+ 68,
+ 295,
+ 72,
+ 67,
+ 193,
+ 125,
+ 110,
+ 142,
+ 141
+ ],
+ [
+ 44,
+ 226,
+ 154,
+ 199,
+ 20,
+ 23,
+ 61,
+ 183,
+ 49,
+ 240,
+ 127,
+ 32,
+ 58,
+ 135,
+ 19,
+ 45,
+ 65,
+ 95,
+ 36,
+ 136
+ ],
+ [
+ 138,
+ 129,
+ 236,
+ 211,
+ 9,
+ 162,
+ 217,
+ 37,
+ 212,
+ 203,
+ 277,
+ 279,
+ 291,
+ 263,
+ 25,
+ 221,
+ 34,
+ 268,
+ 189,
+ 133
+ ],
+ [
+ 197,
+ 263,
+ 77,
+ 101,
+ 217,
+ 291,
+ 212,
+ 37,
+ 189,
+ 129,
+ 235,
+ 9,
+ 223,
+ 279,
+ 268,
+ 221,
+ 133,
+ 236,
+ 211,
+ 203
+ ],
+ [
+ 39,
+ 145,
+ 102,
+ 254,
+ 38,
+ 13,
+ 60,
+ 298,
+ 71,
+ 267,
+ 91,
+ 137,
+ 234,
+ 286,
+ 112,
+ 124,
+ 253,
+ 160,
+ 128,
+ 182
+ ],
+ [
+ 277,
+ 129,
+ 223,
+ 197,
+ 101,
+ 77,
+ 217,
+ 34,
+ 203,
+ 236,
+ 189,
+ 37,
+ 2,
+ 221,
+ 263,
+ 211,
+ 291,
+ 162,
+ 212,
+ 133
+ ],
+ [
+ 266,
+ 33,
+ 245,
+ 108,
+ 69,
+ 171,
+ 297,
+ 70,
+ 11,
+ 192,
+ 196,
+ 35,
+ 30,
+ 50,
+ 180,
+ 232,
+ 139,
+ 186,
+ 248,
+ 284
+ ],
+ [
+ 176,
+ 235,
+ 77,
+ 221,
+ 223,
+ 285,
+ 133,
+ 138,
+ 212,
+ 197,
+ 101,
+ 37,
+ 203,
+ 129,
+ 54,
+ 211,
+ 217,
+ 189,
+ 9,
+ 236
+ ],
+ [
+ 157,
+ 178,
+ 119,
+ 208,
+ 255,
+ 258,
+ 227,
+ 294,
+ 155,
+ 0,
+ 132,
+ 181,
+ 174,
+ 26,
+ 222,
+ 87,
+ 92,
+ 10,
+ 283,
+ 238
+ ],
+ [
+ 94,
+ 87,
+ 165,
+ 152,
+ 227,
+ 178,
+ 140,
+ 92,
+ 294,
+ 119,
+ 258,
+ 157,
+ 155,
+ 282,
+ 132,
+ 208,
+ 10,
+ 174,
+ 255,
+ 0
+ ],
+ [
+ 248,
+ 30,
+ 232,
+ 180,
+ 50,
+ 186,
+ 196,
+ 35,
+ 192,
+ 287,
+ 11,
+ 70,
+ 139,
+ 266,
+ 280,
+ 33,
+ 108,
+ 69,
+ 245,
+ 171
+ ],
+ [
+ 223,
+ 235,
+ 77,
+ 37,
+ 197,
+ 212,
+ 129,
+ 138,
+ 211,
+ 217,
+ 281,
+ 101,
+ 2,
+ 277,
+ 203,
+ 221,
+ 236,
+ 9,
+ 34,
+ 133
+ ],
+ [
+ 298,
+ 124,
+ 278,
+ 38,
+ 63,
+ 39,
+ 239,
+ 137,
+ 91,
+ 145,
+ 62,
+ 60,
+ 148,
+ 254,
+ 267,
+ 185,
+ 111,
+ 112,
+ 71,
+ 109
+ ],
+ [
+ 186,
+ 196,
+ 248,
+ 50,
+ 30,
+ 232,
+ 284,
+ 35,
+ 139,
+ 266,
+ 11,
+ 180,
+ 33,
+ 70,
+ 171,
+ 69,
+ 280,
+ 108,
+ 192,
+ 245
+ ],
+ [
+ 271,
+ 56,
+ 97,
+ 209,
+ 93,
+ 29,
+ 201,
+ 200,
+ 122,
+ 159,
+ 82,
+ 224,
+ 100,
+ 127,
+ 265,
+ 36,
+ 244,
+ 264,
+ 53,
+ 246
+ ],
+ [
+ 247,
+ 38,
+ 234,
+ 86,
+ 134,
+ 145,
+ 278,
+ 74,
+ 253,
+ 55,
+ 71,
+ 102,
+ 150,
+ 17,
+ 121,
+ 112,
+ 89,
+ 88,
+ 168,
+ 185
+ ],
+ [
+ 80,
+ 161,
+ 67,
+ 262,
+ 79,
+ 142,
+ 274,
+ 272,
+ 149,
+ 193,
+ 169,
+ 46,
+ 72,
+ 295,
+ 59,
+ 106,
+ 78,
+ 43,
+ 125,
+ 76
+ ],
+ [
+ 197,
+ 263,
+ 212,
+ 9,
+ 277,
+ 37,
+ 77,
+ 133,
+ 129,
+ 101,
+ 268,
+ 236,
+ 203,
+ 296,
+ 211,
+ 223,
+ 34,
+ 221,
+ 162,
+ 189
+ ],
+ [
+ 181,
+ 0,
+ 255,
+ 87,
+ 81,
+ 153,
+ 273,
+ 261,
+ 178,
+ 216,
+ 227,
+ 152,
+ 151,
+ 157,
+ 174,
+ 294,
+ 132,
+ 92,
+ 252,
+ 119
+ ],
+ [
+ 98,
+ 172,
+ 195,
+ 10,
+ 103,
+ 251,
+ 208,
+ 269,
+ 190,
+ 81,
+ 294,
+ 227,
+ 151,
+ 119,
+ 92,
+ 21,
+ 261,
+ 273,
+ 181,
+ 216
+ ],
+ [
+ 10,
+ 92,
+ 208,
+ 178,
+ 152,
+ 119,
+ 174,
+ 222,
+ 132,
+ 81,
+ 282,
+ 157,
+ 251,
+ 261,
+ 26,
+ 181,
+ 165,
+ 238,
+ 227,
+ 255
+ ],
+ [
+ 76,
+ 3,
+ 204,
+ 110,
+ 8,
+ 169,
+ 161,
+ 96,
+ 142,
+ 1,
+ 68,
+ 125,
+ 193,
+ 80,
+ 272,
+ 72,
+ 6,
+ 67,
+ 290,
+ 106
+ ],
+ [
+ 9,
+ 215,
+ 34,
+ 291,
+ 37,
+ 133,
+ 162,
+ 263,
+ 138,
+ 203,
+ 212,
+ 25,
+ 268,
+ 2,
+ 197,
+ 129,
+ 221,
+ 277,
+ 211,
+ 236
+ ],
+ [
+ 108,
+ 69,
+ 245,
+ 171,
+ 33,
+ 280,
+ 70,
+ 266,
+ 11,
+ 139,
+ 35,
+ 196,
+ 192,
+ 180,
+ 50,
+ 186,
+ 232,
+ 30,
+ 248,
+ 284
+ ],
+ [
+ 5,
+ 63,
+ 239,
+ 286,
+ 39,
+ 62,
+ 278,
+ 148,
+ 22,
+ 131,
+ 123,
+ 91,
+ 42,
+ 124,
+ 38,
+ 74,
+ 18,
+ 15,
+ 185,
+ 254
+ ],
+ [
+ 206,
+ 13,
+ 160,
+ 184,
+ 89,
+ 128,
+ 112,
+ 267,
+ 17,
+ 111,
+ 60,
+ 131,
+ 145,
+ 137,
+ 254,
+ 237,
+ 109,
+ 27,
+ 57,
+ 234
+ ]
+ ]
+ }
+}
diff --git a/tests/signature_contracts.py b/tests/signature_contracts.py
new file mode 100644
index 00000000..d58cbfda
--- /dev/null
+++ b/tests/signature_contracts.py
@@ -0,0 +1,90 @@
+import ast
+import hashlib
+import inspect
+import json
+import textwrap
+
+
+def _annotation_text(annotation: ast.expr | None) -> str | None:
+ if annotation is None:
+ return None
+ return ast.unparse(annotation)
+
+
+def _parameter(
+ argument: ast.arg,
+ kind: str,
+ default: ast.expr | None = None,
+ *,
+ required: bool = True,
+) -> dict[str, object]:
+ parameter: dict[str, object] = {
+ "annotation": _annotation_text(argument.annotation),
+ "kind": kind,
+ "name": argument.arg,
+ }
+ if not required:
+ if default is None:
+ raise ValueError("Optional signature parameter is missing its default")
+ parameter["default"] = ast.unparse(default)
+ return parameter
+
+
+def _source_signature(method: object) -> dict[str, object]:
+ source = textwrap.dedent(inspect.getsource(inspect.unwrap(method)))
+ module = ast.parse(source)
+ function = next(
+ node
+ for node in module.body
+ if isinstance(node, ast.FunctionDef | ast.AsyncFunctionDef)
+ )
+ arguments = function.args
+ positional = [*arguments.posonlyargs, *arguments.args]
+ default_offset = len(positional) - len(arguments.defaults)
+ parameters = []
+ for index, argument in enumerate(positional):
+ kind = (
+ "positional_only"
+ if index < len(arguments.posonlyargs)
+ else "positional_or_keyword"
+ )
+ has_default = index >= default_offset
+ default = arguments.defaults[index - default_offset] if has_default else None
+ parameters.append(
+ _parameter(
+ argument,
+ kind,
+ default,
+ required=not has_default,
+ )
+ )
+ if arguments.vararg is not None:
+ parameters.append(_parameter(arguments.vararg, "var_positional"))
+ for argument, default in zip(
+ arguments.kwonlyargs,
+ arguments.kw_defaults,
+ strict=True,
+ ):
+ parameters.append(
+ _parameter(
+ argument,
+ "keyword_only",
+ default,
+ required=default is None,
+ )
+ )
+ if arguments.kwarg is not None:
+ parameters.append(_parameter(arguments.kwarg, "var_keyword"))
+ return {
+ "async": isinstance(function, ast.AsyncFunctionDef),
+ "parameters": parameters,
+ "return": _annotation_text(function.returns),
+ }
+
+
+def signature_digest(methods: dict[str, object]) -> str:
+ payload = {
+ name: _source_signature(method) for name, method in sorted(methods.items())
+ }
+ encoded = json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
+ return hashlib.sha256(encoded).hexdigest()
diff --git a/tests/smoke_wheel.py b/tests/smoke_wheel.py
new file mode 100644
index 00000000..85e09acb
--- /dev/null
+++ b/tests/smoke_wheel.py
@@ -0,0 +1,309 @@
+import argparse
+import os
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+from zipfile import ZipFile
+
+
+_RETIRED_MODULES = {
+ "scarf/_types.py",
+ "scarf/ann.py",
+ "scarf/assay.py",
+ "scarf/bio_data.py",
+ "scarf/chunked.py",
+ "scarf/clustering/feature_graph.py",
+ "scarf/dendrogram.py",
+ "scarf/downloader.py",
+ "scarf/doublet_utils.py",
+ "scarf/feat_utils.py",
+ "scarf/features/lowess.py",
+ "scarf/harmony.py",
+ "scarf/harmony/__init__.py",
+ "scarf/harmony/api.py",
+ "scarf/harmony/models.py",
+ "scarf/harmony/optimizer.py",
+ "scarf/genomics/__init__.py",
+ "scarf/genomics/gff.py",
+ "scarf/genomics/intervals.py",
+ "scarf/genomics/melding.py",
+ "scarf/genomics/reference.py",
+ "scarf/graph/build.py",
+ "scarf/knn_utils.py",
+ "scarf/mapping/coral.py",
+ "scarf/mapping_reference.py",
+ "scarf/mapping_utils.py",
+ "scarf/markers.py",
+ "scarf/markers/__init__.py",
+ "scarf/markers/batching.py",
+ "scarf/markers/rank.py",
+ "scarf/markers/regression.py",
+ "scarf/markers/search.py",
+ "scarf/meld_assay.py",
+ "scarf/merge.py",
+ "scarf/metadata.py",
+ "scarf/metrics.py",
+ "scarf/neighbors/graph_store.py",
+ "scarf/neighbors/persistence.py",
+ "scarf/neighbors/query.py",
+ "scarf/parallel.py",
+ "scarf/plots.py",
+ "scarf/plots/__init__.py",
+ "scarf/plotting/_legacy.py",
+ "scarf/plotting/_legacy/__init__.py",
+ "scarf/plotting/unified.py",
+ "scarf/readers.py",
+ "scarf/readers/datasets.py",
+ "scarf/results.py",
+ "scarf/storage/zarr_store.py",
+ "scarf/trajectory/aggregation.py",
+ "scarf/symphony.py",
+ "scarf/umap.py",
+ "scarf/utils.py",
+ "scarf/utils/blocks.py",
+ "scarf/utils/memory.py",
+ "scarf/utils/storage.py",
+ "scarf/utils/system.py",
+ "scarf/utils/windows.py",
+ "scarf/writers.py",
+}
+_SOURCE_ROOT = Path(__file__).resolve().parents[1] / "scarf"
+_REQUIRED_MODULES = {
+ f"scarf/{path.relative_to(_SOURCE_ROOT).as_posix()}"
+ for path in _SOURCE_ROOT.rglob("*.py")
+}
+_SMOKE_CODE = """
+import importlib.util
+from pathlib import Path
+
+import scarf
+import scarf.plotting as plotting
+import scarf.cytebase
+import scarf.embeddings.harmony
+import scarf.features.genomic
+import scarf.features.markers
+import scarf.features.variability
+import scarf.matrix
+import scarf.merge
+import scarf.metadata
+import scarf.readers
+import scarf.writers
+from scarf.datastore.datastore import DataStore
+from scarf.datastore.graph_datastore import GraphDataStore
+from scarf.datastore.mapping_datastore import MappingDatastore
+from scarf.cytebase import Repository, connect, list_repositories
+from scarf.embeddings.harmony import Harmony, HarmonyResult, fit_harmony, run_harmony
+from scarf.features import (
+ GffReader,
+ coordinate_melding,
+ find_markers_by_rank,
+ fit_lowess,
+ select_highly_variable_features,
+)
+from scarf.matrix import Block, ChunkedArray
+from scarf.merge import AssayMerge, DatasetMerge, DummyAssay
+from scarf.metadata import MetaData, MetaDataRowBlock
+from scarf.readers import (
+ CSVReader,
+ CrDirReader,
+ CrH5Reader,
+ CrReader,
+ H5adReader,
+ LoomReader,
+ SeuratReader,
+ inspect_seurat,
+)
+from scarf.trajectory.feature_dynamics import knn_clustering
+from scarf.writers import (
+ CSVtoZarr,
+ CrToZarr,
+ H5adToZarr,
+ LoomToZarr,
+ SeuratImportResult,
+ SeuratToZarr,
+ SparseToZarr,
+ SubsetZarr,
+ create_zarr_count_assay,
+ create_zarr_dataset,
+ create_zarr_obj_array,
+ dask_to_zarr,
+ subset_assay_zarr,
+ to_h5ad,
+ to_mtx,
+ write_renorm_subset_to_zarr,
+)
+
+assert "site-packages" in Path(scarf.__file__).as_posix()
+assert issubclass(DataStore, MappingDatastore)
+assert issubclass(MappingDatastore, GraphDataStore)
+for harmony_object in (Harmony, HarmonyResult, fit_harmony, run_harmony):
+ assert harmony_object.__module__ == "scarf.embeddings.harmony"
+for matrix_class in (Block, ChunkedArray):
+ assert matrix_class.__module__ == "scarf.matrix"
+for metadata_class in (MetaData, MetaDataRowBlock):
+ assert metadata_class.__module__ == "scarf.metadata"
+assert scarf.AssayMerge is scarf.merge.AssayMerge is AssayMerge
+assert scarf.DatasetMerge is scarf.merge.DatasetMerge is DatasetMerge
+assert scarf.merge.DummyAssay is DummyAssay
+assert not hasattr(scarf, "ZarrMerge")
+assert not hasattr(scarf.merge, "ZarrMerge")
+for merge_class in (AssayMerge, DatasetMerge, DummyAssay):
+ assert merge_class.__module__ == "scarf.merge"
+assert scarf.CrH5Reader is scarf.readers.CrH5Reader
+assert scarf.CrToZarr is scarf.writers.CrToZarr
+assert scarf.cytebase.Repository is Repository
+assert scarf.cytebase.connect is connect
+assert scarf.cytebase.list_repositories is list_repositories
+assert scarf.GffReader is GffReader
+assert scarf.coordinate_melding is coordinate_melding
+for feature_function in (
+ coordinate_melding,
+ find_markers_by_rank,
+ fit_lowess,
+ select_highly_variable_features,
+):
+ assert callable(feature_function)
+assert callable(knn_clustering)
+for reader_class in (
+ CrH5Reader,
+ CrDirReader,
+ CrReader,
+ H5adReader,
+ LoomReader,
+ SeuratReader,
+ CSVReader,
+):
+ assert reader_class.__module__ == "scarf.readers"
+for writer_class in (
+ CrToZarr,
+ H5adToZarr,
+ LoomToZarr,
+ SeuratToZarr,
+ SparseToZarr,
+ SubsetZarr,
+ CSVtoZarr,
+):
+ assert writer_class.__module__ == "scarf.writers"
+assert SeuratImportResult.__module__ == "scarf.writers"
+assert inspect_seurat.__module__ == "scarf.readers"
+for writer_function in (
+ create_zarr_dataset,
+ create_zarr_obj_array,
+ create_zarr_count_assay,
+ subset_assay_zarr,
+ dask_to_zarr,
+ write_renorm_subset_to_zarr,
+ to_h5ad,
+ to_mtx,
+):
+ assert callable(writer_function)
+ assert writer_function.__module__ == "scarf.writers"
+for method in (
+ "run_mapping",
+ "run_marker_search",
+ "mark_hvgs",
+ "run_pseudotime_aggregation",
+ "run_pseudotime_marker_search",
+ "metric_lisi",
+):
+ assert callable(getattr(DataStore, method))
+for method in (
+ "_load_unified_layout_data",
+ "load_unified_graph",
+ "run_unified_tsne",
+ "run_unified_umap",
+):
+ assert not hasattr(DataStore, method)
+assert not hasattr(plotting, "unified_embedding")
+for name in (
+ "scarf._types",
+ "scarf.bio_data",
+ "scarf.chunked",
+ "scarf.downloader",
+ "scarf.doublet_utils",
+ "scarf.feat_utils",
+ "scarf.harmony",
+ "scarf.genomics",
+ "scarf.knn_utils",
+ "scarf.mapping.coral",
+ "scarf.mapping_reference",
+ "scarf.mapping_utils",
+ "scarf.markers",
+ "scarf.meld_assay",
+ "scarf.plotting.unified",
+ "scarf.symphony",
+):
+ assert importlib.util.find_spec(name) is None, name
+for name in (
+ "scarf.embeddings.harmony",
+ "scarf.features.genomic",
+ "scarf.features.markers",
+ "scarf.matrix",
+ "scarf.metadata",
+ "scarf.metrics",
+):
+ spec = importlib.util.find_spec(name)
+ assert spec is not None and spec.submodule_search_locations is not None, name
+"""
+
+
+def validate_wheel_contents(wheel: Path) -> None:
+ with ZipFile(wheel) as archive:
+ names = set(archive.namelist())
+ script_entries = [
+ info
+ for info in archive.infolist()
+ if info.filename.endswith(".data/scripts/sgtsne")
+ ]
+ retired = sorted(_RETIRED_MODULES.intersection(names))
+ missing = sorted(_REQUIRED_MODULES.difference(names))
+ if retired:
+ raise RuntimeError(f"Wheel contains retired modules: {retired}")
+ if missing:
+ raise RuntimeError(f"Wheel is missing required modules: {missing}")
+ if len(script_entries) != 1:
+ raise RuntimeError("Wheel must contain exactly one sgtsne script")
+ script_mode = (script_entries[0].external_attr >> 16) & 0o777
+ if script_mode != 0o755:
+ raise RuntimeError(
+ f"Wheel sgtsne script has mode {oct(script_mode)}, expected 0o755"
+ )
+
+
+def smoke_installed_wheel(wheel: Path) -> None:
+ python_version = f"{sys.version_info.major}.{sys.version_info.minor}"
+ env = os.environ.copy()
+ env["HNSWLIB_NO_NATIVE"] = "1"
+ with tempfile.TemporaryDirectory(prefix="scarf-wheel-smoke-") as temp_dir:
+ subprocess.run(
+ [
+ "uv",
+ "run",
+ "--isolated",
+ "--python",
+ python_version,
+ "--with",
+ str(wheel),
+ "python",
+ "-c",
+ _SMOKE_CODE,
+ ],
+ cwd=temp_dir,
+ env=env,
+ check=True,
+ )
+
+
+def main() -> None:
+ parser = argparse.ArgumentParser()
+ parser.add_argument("wheel", type=Path)
+ args = parser.parse_args()
+ wheel = args.wheel.resolve()
+ validate_wheel_contents(wheel)
+ smoke_installed_wheel(wheel)
+ print(f"Wheel smoke passed: {wheel.name}")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/tests/store_probes.py b/tests/store_probes.py
new file mode 100644
index 00000000..8d40d693
--- /dev/null
+++ b/tests/store_probes.py
@@ -0,0 +1,143 @@
+"""Zarr store wrappers that expose object operations to tests."""
+
+import asyncio
+import threading
+
+from zarr.storage import MemoryStore
+
+
+class StoreProbe:
+ """Shared operation log for a store and any clone Zarr makes of it.
+
+ Args:
+ delay: Seconds every operation awaits, making real overlap observable.
+ fail_on: Key whose write raises instead of storing bytes.
+ """
+
+ def __init__(self, *, delay: float = 0.0, fail_on: str | None = None):
+ self.delay = delay
+ self.fail_on = fail_on
+ self.ops: list[tuple[str, str]] = []
+ self.byte_ranges: list[tuple[str, str, object | None]] = []
+ self.max_in_flight = 0
+ self._in_flight = 0
+ self._in_flight_by_kind: dict[str, int] = {}
+ self._max_in_flight_by_kind: dict[str, int] = {}
+ self._lock = threading.Lock()
+
+ def reset(self) -> None:
+ with self._lock:
+ self.ops.clear()
+ self.byte_ranges.clear()
+ self.max_in_flight = 0
+ self._in_flight = 0
+ self._in_flight_by_kind.clear()
+ self._max_in_flight_by_kind.clear()
+
+ def chunk_ops(self, prefix: str) -> list[tuple[str, str]]:
+ return [(kind, key) for kind, key in self.ops if key.startswith(prefix)]
+
+ def enter(self, kind: str, key: str, byte_range: object | None = None) -> None:
+ with self._lock:
+ self.ops.append((kind, key))
+ self.byte_ranges.append((kind, key, byte_range))
+ self._in_flight += 1
+ self.max_in_flight = max(self.max_in_flight, self._in_flight)
+ self._in_flight_by_kind[kind] = self._in_flight_by_kind.get(kind, 0) + 1
+ self._max_in_flight_by_kind[kind] = max(
+ self._max_in_flight_by_kind.get(kind, 0),
+ self._in_flight_by_kind[kind],
+ )
+
+ def leave(self, kind: str) -> None:
+ with self._lock:
+ self._in_flight -= 1
+ self._in_flight_by_kind[kind] -= 1
+
+ def max_in_flight_for(self, kind: str) -> int:
+ with self._lock:
+ return self._max_in_flight_by_kind.get(kind, 0)
+
+
+class RecordingStore(MemoryStore):
+ """MemoryStore that records object operations and observed overlap."""
+
+ def __init__(
+ self,
+ store_dict=None,
+ *,
+ read_only: bool = False,
+ delay: float = 0.0,
+ fail_on: str | None = None,
+ probe: StoreProbe | None = None,
+ ):
+ super().__init__(store_dict, read_only=read_only)
+ self.probe = probe or StoreProbe(delay=delay, fail_on=fail_on)
+
+ # Zarr clones a store to change read-only state; the clone keeps reporting
+ # to the same probe so the test still observes every operation.
+ def with_read_only(self, read_only: bool = False) -> "RecordingStore":
+ return type(self)(
+ store_dict=self._store_dict,
+ read_only=read_only,
+ probe=self.probe,
+ )
+
+ @property
+ def ops(self) -> list[tuple[str, str]]:
+ return self.probe.ops
+
+ @property
+ def max_in_flight(self) -> int:
+ return self.probe.max_in_flight
+
+ def max_in_flight_for(self, kind: str) -> int:
+ return self.probe.max_in_flight_for(kind)
+
+ @property
+ def byte_ranges(self) -> list[tuple[str, str, object | None]]:
+ return self.probe.byte_ranges
+
+ def reset(self) -> None:
+ self.probe.reset()
+
+ def chunk_ops(self, prefix: str) -> list[tuple[str, str]]:
+ return self.probe.chunk_ops(prefix)
+
+ # The awaitable is built inside the tracker so a cancelled task never leaves
+ # an unawaited coroutine behind.
+ async def _tracked(self, kind, key, start, byte_range=None):
+ self.probe.enter(kind, key, byte_range)
+ try:
+ if self.probe.delay:
+ await asyncio.sleep(self.probe.delay)
+ return await start()
+ finally:
+ self.probe.leave(kind)
+
+ async def get(self, key, prototype, byte_range=None):
+ return await self._tracked(
+ "get",
+ key,
+ lambda: super(RecordingStore, self).get(key, prototype, byte_range),
+ byte_range,
+ )
+
+ async def set(self, key, value, byte_range=None):
+ if key == self.probe.fail_on:
+ self.probe.enter("set", key, byte_range)
+ self.probe.leave("set")
+ raise RuntimeError("injected write failure")
+ return await self._tracked(
+ "set",
+ key,
+ lambda: super(RecordingStore, self).set(key, value, byte_range),
+ byte_range,
+ )
+
+ async def delete(self, key):
+ return await self._tracked(
+ "delete",
+ key,
+ lambda: super(RecordingStore, self).delete(key),
+ )
diff --git a/tests/symphony_r_0_1_3_golden.json b/tests/symphony_r_0_1_3_golden.json
new file mode 100644
index 00000000..5203f212
--- /dev/null
+++ b/tests/symphony_r_0_1_3_golden.json
@@ -0,0 +1,80 @@
+{
+ "provenance": {
+ "package": "symphony",
+ "packageVersion": "0.1.3",
+ "sourceRepository": "https://github.com/immunogenomics/symphony",
+ "sourceCommit": "7c5905988734d9cfe6e1e97a658664717c4ba7b7",
+ "function": "mapQuery",
+ "doNormalize": false,
+ "doUmap": false,
+ "sigma": 0.1
+ },
+ "reference": {
+ "featureMeans": [0.0, 0.0],
+ "featureScales": [1.0, 1.0],
+ "loadings": [[1.0, 0.0], [0.0, 1.0]],
+ "centroids": [[-1.0, 0.0], [1.0, 0.0]],
+ "rawCentroids": [[-1.0, 0.0], [1.0, 0.0]],
+ "correctedCentroids": [[-1.0, 0.0], [1.0, 0.0]],
+ "clusterMass": [4.0, 4.0],
+ "sigma": [0.1, 0.1],
+ "correctionRidge": 1.0
+ },
+ "query": [
+ [-1.5, 0.2],
+ [-0.5, -0.2],
+ [0.5, 0.2],
+ [1.5, -0.2]
+ ],
+ "batchCodes": [0, 0, 0, 0],
+ "expectedProjected": [
+ [-1.5, 0.2],
+ [-0.5, -0.2],
+ [0.5, 0.2],
+ [1.5, -0.2]
+ ],
+ "expectedAssignments": [
+ [1.0, 6.0340367874094217e-18],
+ [1.0, 7.4251990982146727e-17],
+ [7.4251990982146727e-17, 1.0],
+ [6.0340367874094217e-18, 1.0]
+ ],
+ "expectedCorrected": [
+ [-1.5, 0.2],
+ [-0.5, -0.2],
+ [0.5, 0.2],
+ [1.5, -0.2]
+ ],
+ "nonzeroCorrection": {
+ "reference": {
+ "featureMeans": [0.0, 0.0],
+ "featureScales": [1.0, 1.0],
+ "loadings": [[1.0, 0.0], [0.0, 1.0]],
+ "centroids": [[1.0, 0.0]],
+ "rawCentroids": [[0.0, 0.0]],
+ "correctedCentroids": [[0.0, 0.0]],
+ "clusterMass": [4.0],
+ "sigma": [0.1],
+ "correctionRidge": 2.0
+ },
+ "query": [
+ [2.0, -2.0],
+ [4.0, -2.0],
+ [1.0, -1.0],
+ [5.0, -3.0]
+ ],
+ "batchCodes": [0, 0, 0, 0],
+ "expectedAssignments": [
+ [1.0],
+ [1.0],
+ [1.0],
+ [1.0]
+ ],
+ "expectedCorrected": [
+ [0.0, -0.6666666666666667],
+ [2.0, -0.6666666666666667],
+ [-1.0, 0.33333333333333326],
+ [3.0, -1.6666666666666667]
+ ]
+ }
+}
diff --git a/tests/test_ann.py b/tests/test_ann.py
new file mode 100644
index 00000000..bc773754
--- /dev/null
+++ b/tests/test_ann.py
@@ -0,0 +1,82 @@
+import numpy as np
+
+from scarf.neighbors.index import fix_knn_query
+
+
+def test_fix_knn_query_removes_leading_self_matches():
+ indices = np.array([[4, 1, 2], [7, 5, 6]])
+ distances = np.array([[0.0, 0.2, 0.4], [0.0, 0.3, 0.6]])
+
+ fixed_indices, fixed_distances, missed_self_hits = fix_knn_query(
+ indices,
+ distances,
+ ref_idx=np.array([4, 7]),
+ )
+
+ assert missed_self_hits == 0
+ assert np.array_equal(fixed_indices, [[1, 2], [5, 6]])
+ assert np.allclose(fixed_distances, [[0.2, 0.4], [0.3, 0.6]])
+
+
+def test_fix_knn_query_handles_each_self_neighbor_case():
+ indices = np.array(
+ [
+ [0, 1, 2, 3],
+ [2, 3, 1, 4],
+ [0, 2, 3, 4],
+ ]
+ )
+ distances = np.array(
+ [
+ [0.0, 0.1, 0.2, 0.3],
+ [0.1, 0.2, 0.3, 0.4],
+ [0.1, 0.2, 0.3, 0.4],
+ ]
+ )
+ original_indices = indices.copy()
+ original_distances = distances.copy()
+
+ fixed_indices, fixed_distances, missed_self_hits = fix_knn_query(
+ indices,
+ distances,
+ ref_idx=np.array([0, 1, 5]),
+ )
+
+ assert missed_self_hits == 1
+ assert np.array_equal(
+ fixed_indices,
+ np.array(
+ [
+ [1, 2, 3],
+ [2, 3, 4],
+ [0, 2, 3],
+ ]
+ ),
+ )
+ assert np.allclose(
+ fixed_distances,
+ np.array(
+ [
+ [0.1, 0.2, 0.3],
+ [0.1, 0.2, 0.4],
+ [0.1, 0.2, 0.3],
+ ]
+ ),
+ )
+ assert np.array_equal(indices, original_indices)
+ assert np.array_equal(distances, original_distances)
+
+
+def test_fix_knn_query_removes_only_the_first_repeated_self_label():
+ indices = np.array([[3, 8, 3, 9]])
+ distances = np.array([[0.0, 0.1, 0.2, 0.3]])
+
+ fixed_indices, fixed_distances, missed_self_hits = fix_knn_query(
+ indices,
+ distances,
+ ref_idx=np.array([3]),
+ )
+
+ assert missed_self_hits == 0
+ np.testing.assert_array_equal(fixed_indices, [[8, 3, 9]])
+ np.testing.assert_allclose(fixed_distances, [[0.1, 0.2, 0.3]])
diff --git a/tests/test_ann_mapping.py b/tests/test_ann_mapping.py
new file mode 100644
index 00000000..a0c824ac
--- /dev/null
+++ b/tests/test_ann_mapping.py
@@ -0,0 +1,55 @@
+import numpy as np
+import pytest
+
+from scarf.matrix import ChunkedArray
+from scarf.neighbors.stream import AnnStream
+
+
+def _ann_stream() -> AnnStream:
+ data = ChunkedArray.from_numpy(
+ np.array([[0.0, 0.0], [1.0, 1.0], [2.0, 2.0]], dtype=np.float64),
+ block_size=2,
+ )
+ return AnnStream(
+ data=data,
+ k=1,
+ n_cluster=2,
+ reduction_method="pca",
+ dims=2,
+ loadings=np.eye(2),
+ use_for_pca=np.ones(3, dtype=bool),
+ mu=np.array([0.5, 0.5]),
+ sigma=np.array([0.5, 0.5]),
+ ann_metric="l2",
+ ann_efc=10,
+ ann_ef=10,
+ ann_m=8,
+ nthreads=1,
+ ann_parallel=False,
+ rand_state=0,
+ do_kmeans_fit=False,
+ disable_scaling=False,
+ ann_idx=None,
+ lsi_skip_first=True,
+ lsi_params={},
+ harmonize=False,
+ )
+
+
+def test_transform_query_matches_existing_reference_reducer():
+ ann = _ann_stream()
+ query = np.array([[1.5, 0.0], [0.0, 1.5]], dtype=np.float64)
+
+ assert not hasattr(ann, "annPath")
+ assert not hasattr(ann, "_embedding_bytes")
+ assert not hasattr(ann, "embeddings")
+ np.testing.assert_array_equal(ann.transform_query(query), ann.reducer(query))
+
+
+def test_transform_query_validates_feature_shape_and_finite_results():
+ ann = _ann_stream()
+
+ with pytest.raises(ValueError, match="features"):
+ ann.transform_query(np.ones((2, 3)))
+ with pytest.raises(ValueError, match="non-finite"):
+ ann.transform_query(np.array([[np.nan, 0.0]]))
diff --git a/tests/test_anndata_export.py b/tests/test_anndata_export.py
new file mode 100644
index 00000000..0b334780
--- /dev/null
+++ b/tests/test_anndata_export.py
@@ -0,0 +1,130 @@
+import shutil
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+from scipy import sparse
+
+from scarf.datastore.datastore import DataStore
+
+
+@pytest.fixture
+def export_store(toy_crdir_writer, tmp_path):
+ destination = tmp_path / "toy.zarr"
+ shutil.copytree(toy_crdir_writer, destination)
+ return DataStore(
+ str(destination),
+ default_assay="RNA",
+ min_features_per_cell=0,
+ min_cells_per_feature=0,
+ nthreads=1,
+ )
+
+
+def test_to_anndata_exports_normed_csr_with_ordered_feature_indexes(export_store):
+ feature_indexes = np.array([3, 0])
+ cell_indexes = export_store.cells.active_index("I")
+
+ adata = export_store.to_anndata(
+ from_assay="RNA",
+ cell_key="I",
+ matrix="normed",
+ feature_indexes=feature_indexes,
+ )
+
+ expected = export_store.RNA.normed(
+ cell_idx=cell_indexes,
+ feat_idx=feature_indexes,
+ ).compute()
+ assert sparse.isspmatrix_csr(adata.X)
+ np.testing.assert_allclose(adata.X.toarray(), expected)
+ assert list(adata.var_names) == ["g4", "g1"]
+ assert list(adata.obs_names) == list(export_store.cells.fetch("ids", key="I"))
+
+
+def test_to_anndata_selects_names_and_aligns_raw_layers(export_store):
+ all_names = export_store.RNA.feats.fetch_all("names").astype(str)
+ requested = [all_names[2], all_names[0]]
+
+ adata = export_store.to_anndata(
+ from_assay="RNA",
+ feature_names=requested,
+ layers={"raw": "RNA"},
+ )
+
+ expected = export_store.RNA.to_raw_sparse("I")[:, [2, 0]]
+ assert sparse.isspmatrix_csr(adata.X)
+ np.testing.assert_array_equal(adata.X.toarray(), expected.toarray())
+ np.testing.assert_array_equal(adata.layers["raw"].toarray(), expected.toarray())
+ assert list(adata.var["names"].astype(str)) == requested
+
+
+def test_to_anndata_preserves_legacy_layer_behavior_without_subset(export_store):
+ adata = export_store.to_anndata(layers={"raw": "RNA"})
+
+ assert sparse.isspmatrix_csr(adata.X)
+ np.testing.assert_array_equal(
+ adata.layers["raw"].toarray(),
+ adata.X.toarray(),
+ )
+ assert adata.n_vars == export_store.RNA.feats.N
+
+
+def test_to_anndata_aligns_reordered_layer_ids_without_subset(
+ export_store,
+ monkeypatch,
+):
+ primary = export_store.RNA
+ primary_ids = primary.feats.fetch_all("ids").astype(str)
+ order = np.array([2, 0, 3, 1])
+ reordered_matrix = primary.to_raw_sparse("I")[:, order]
+ reordered_assay = SimpleNamespace(
+ feats=SimpleNamespace(
+ fetch_all=lambda column: primary_ids[order] if column == "ids" else None
+ ),
+ to_raw_sparse=lambda cell_key: reordered_matrix,
+ )
+ original_get_assay = export_store._get_assay
+
+ def get_assay(name):
+ if name == "reordered":
+ return reordered_assay
+ return original_get_assay(name)
+
+ monkeypatch.setattr(export_store, "_get_assay", get_assay)
+ adata = export_store.to_anndata(layers={"reordered": "reordered"})
+
+ np.testing.assert_array_equal(
+ adata.layers["reordered"].toarray(),
+ primary.to_raw_sparse("I").toarray(),
+ )
+
+
+def test_to_anndata_rejects_invalid_or_ambiguous_selectors(export_store):
+ names = export_store.RNA.feats.fetch_all("names").astype(str)
+
+ with pytest.raises(ValueError, match="mutually exclusive"):
+ export_store.to_anndata(feature_indexes=[0], feature_names=[names[0]])
+ with pytest.raises(ValueError, match="unique indexes"):
+ export_store.to_anndata(feature_indexes=[0, 0])
+ with pytest.raises(IndexError, match="out-of-range"):
+ export_store.to_anndata(feature_indexes=[export_store.RNA.feats.N])
+ with pytest.raises(ValueError, match="unique names"):
+ export_store.to_anndata(feature_names=[names[0], names[0]])
+ with pytest.raises(KeyError, match="not found"):
+ export_store.to_anndata(feature_names=["not-a-feature"])
+ with pytest.raises(ValueError, match="matrix must"):
+ export_store.to_anndata(matrix="scaled")
+
+ names[1] = names[0]
+ export_store.RNA.feats.insert("names", names, overwrite=True)
+ with pytest.raises(ValueError, match="not unique"):
+ export_store.to_anndata(feature_names=[names[0]])
+
+
+def test_to_anndata_rejects_unaligned_subset_layer(export_store):
+ with pytest.raises(ValueError, match="cannot align selected feature IDs"):
+ export_store.to_anndata(
+ feature_indexes=[0],
+ layers={"adt": "ADT"},
+ )
diff --git a/tests/test_artifact_graph_writer.py b/tests/test_artifact_graph_writer.py
new file mode 100644
index 00000000..fc139b32
--- /dev/null
+++ b/tests/test_artifact_graph_writer.py
@@ -0,0 +1,470 @@
+from dataclasses import fields
+
+import numpy as np
+import pytest
+
+from scarf import ArtifactRef
+from scarf.storage.artifacts import parse_artifact_path
+from scarf.graph.state import AssayState
+from tests.fixtures_datastore import build_neighbourhood_graph
+
+pytestmark = pytest.mark.slow
+
+
+def _graph_kwargs() -> dict:
+ return {
+ "from_assay": "RNA",
+ "cell_key": "I",
+ "feat_key": "artifact_hvgs",
+ "dims": 5,
+ "k": 3,
+ "n_centroids": 10,
+ "batch_size": 200,
+ "local_cache": False,
+ }
+
+
+def _prepare_features(datastore) -> None:
+ if "I__artifact_hvgs" not in datastore.get_assay("RNA").feats.columns:
+ datastore.mark_hvgs(
+ from_assay="RNA",
+ cell_key="I",
+ top_n=100,
+ hvg_key_name="artifact_hvgs",
+ show_plot=False,
+ )
+
+
+def _state_refs(state: AssayState) -> dict[str, str]:
+ return {
+ field.name: ref.artifact_id
+ for field in fields(state)
+ if (ref := getattr(state, field.name, None)) is not None
+ and hasattr(ref, "artifact_id")
+ }
+
+
+def test_graph_construction_chain_writes_only_artifacts_and_state(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_features(datastore)
+ assay_attrs_before = dict(datastore.get_assay("RNA").attrs)
+ build_neighbourhood_graph(datastore, **_graph_kwargs())
+ state = datastore.get_assay_state("RNA")
+
+ assert state is not None
+ assert state.cell_key == "I"
+ assert state.feat_key == "artifact_hvgs"
+ stored = datastore._lookup_stored_graph("RNA", "I", "artifact_hvgs")
+ assert stored.reduction_method == "pca"
+ assert stored.dims == 5
+ assert stored.pca_cell_key == "I"
+ assert stored.feat_scaling is True
+ assert stored.ann_metric is not None
+ assert stored.ann_efc is not None
+ assert stored.ann_ef is not None
+ assert stored.ann_m is not None
+ assert stored.rand_state is not None
+ assert stored.k == 3
+ assert stored.local_connectivity is not None
+ assert stored.bandwidth is not None
+ assert stored.n_centroids == 10
+ for field_name in (
+ "normalized",
+ "feature_scaling",
+ "reduction",
+ "ann_index",
+ "embedding_initialization",
+ "neighbors",
+ "connectivity_map",
+ ):
+ ref = getattr(state, field_name)
+ assert ref is not None
+ status = datastore.inspect_artifact(ref)
+ assert status.complete
+ assert status.provenance is not None
+ assert status.path.startswith(f"RNA/artifacts/{ref.kind}/")
+ attrs = datastore.zw[status.path].attrs
+ assert set(attrs) >= {
+ "artifact_id",
+ "kind",
+ "provenance",
+ "execution_options",
+ "complete",
+ }
+ assert state.reduction is not None
+ reduction_execution = datastore.inspect_artifact(state.reduction).execution_options
+ assert reduction_execution is not None
+ assert reduction_execution["local_cache"] is False
+
+ graph = datastore.load_graph()
+ assert graph.shape[0] == int(datastore.cells.fetch_all("I").sum())
+ assert graph.nnz > 0
+ assert datastore.get_assay("RNA").attrs.get(
+ "latest_cell_key"
+ ) == assay_attrs_before.get("latest_cell_key")
+ assert datastore.get_assay("RNA").attrs.get(
+ "latest_feat_key"
+ ) == assay_attrs_before.get("latest_feat_key")
+ assert "dataset_fingerprint" in datastore.get_assay("RNA").attrs
+ assert "RNA/normed__I__artifact_hvgs" not in datastore.zw
+ assert datastore.list_artifacts(
+ kind="cell_selection",
+ scope="datastore",
+ )
+ assert datastore.list_artifacts(
+ kind="feature_selection",
+ from_assay="RNA",
+ )
+ datastore.run_umap(n_epochs=10, label="artifact_umap")
+ datastore.run_leiden_clustering(label="artifact_leiden")
+ lisi = datastore.metric_lisi(
+ ["RNA_artifact_leiden"],
+ perplexity=1,
+ )
+ assert "RNA_artifact_umap1" in datastore.cells.columns
+ assert "RNA_artifact_leiden" in datastore.cells.columns
+ assert set(lisi) == {"RNA_artifact_leiden"}
+ scores = lisi["RNA_artifact_leiden"]
+ assert len(scores) == len(datastore.cells.fetch("RNA_artifact_leiden", key="I"))
+ assert np.isfinite(scores).all()
+ assert (scores >= 1).all()
+
+
+def test_graph_construction_chain_reuses_provenance(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_features(datastore)
+ kwargs = _graph_kwargs()
+ build_neighbourhood_graph(datastore, **kwargs)
+ first = datastore.get_assay_state("RNA")
+ assert first is not None
+ first_refs = _state_refs(first)
+
+ build_neighbourhood_graph(datastore, **kwargs)
+ reused = datastore.get_assay_state("RNA")
+ assert reused is not None
+ assert _state_refs(reused) == first_refs
+
+
+def test_graph_construction_chain_records_pca_selection(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_features(datastore)
+ pca_cells = np.asarray(datastore.cells.fetch_all("I"), dtype=bool)
+ pca_cells[np.flatnonzero(pca_cells)[::3]] = False
+ datastore.cells.insert("pca_cells", pca_cells, overwrite=True)
+ kwargs = _graph_kwargs() | {"pca_cell_key": "pca_cells"}
+
+ build_neighbourhood_graph(datastore, **kwargs)
+ first = datastore.get_assay_state("RNA")
+ assert first is not None
+
+ assert datastore.get_assay_state("RNA") == first
+ assert datastore._lookup_stored_graph().pca_cell_key == "pca_cells"
+
+
+def test_lsi_artifact_replay_keeps_requested_dimensions(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_features(datastore)
+ kwargs = _graph_kwargs() | {
+ "reduction_method": "lsi",
+ "dims": 3,
+ "feat_scaling": False,
+ "lsi_skip_first": True,
+ }
+ build_neighbourhood_graph(datastore, **kwargs)
+
+ state = datastore.get_assay_state("RNA")
+ assert state is not None and state.reduction is not None
+ reduction = datastore.load_artifact(state.reduction)
+ assert reduction["loadings"].shape[1] == 3
+ stored = datastore._lookup_stored_graph()
+ assert stored.reduction_method == "lsi"
+ assert stored.dims == 3
+ assert stored.pca_cell_key is None
+ assert stored.feat_scaling is False
+
+
+def test_graph_construction_chain_treats_structurally_invalid_cache_as_missing(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_features(datastore)
+ kwargs = _graph_kwargs()
+ build_neighbourhood_graph(datastore, **kwargs)
+ first = datastore.get_assay_state("RNA")
+ assert first is not None and first.normalized is not None
+ normalized_path = datastore.inspect_artifact(first.normalized).path
+ del datastore.zw[normalized_path]["data"]
+
+ build_neighbourhood_graph(datastore, **kwargs)
+ repaired = datastore.get_assay_state("RNA")
+
+ assert repaired is not None and repaired.normalized is not None
+ assert repaired.normalized != first.normalized
+ assert "data" in datastore.zw[datastore.inspect_artifact(repaired.normalized).path]
+ assert datastore.load_graph().nnz > 0
+
+
+def test_paris_writes_hierarchy_cut_and_dendrogram_artifacts(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_features(datastore)
+ build_neighbourhood_graph(datastore, **_graph_kwargs())
+
+ result = datastore.run_paris_clustering(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="artifact_hvgs",
+ n_clusters=5,
+ label="artifact_paris",
+ )
+ column = datastore.zw["cellData"]["RNA_artifact_paris"]
+ cut_ref = ArtifactRef.from_dict(column.attrs["source_artifact"])
+
+ assert result.n_clusters == 5
+ assert cut_ref.kind == "cluster_cut"
+ assert datastore.inspect_artifact(cut_ref).complete
+ assert datastore.list_artifacts(
+ kind="cluster_hierarchy",
+ from_assay="RNA",
+ )
+ assert datastore.list_artifacts(kind="dendrogram", from_assay="RNA")
+
+ adaptive = datastore.run_paris_clustering(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="artifact_hvgs",
+ n_clusters="auto",
+ min_cluster_size=10,
+ label="artifact_paris_auto",
+ )
+ assert adaptive.mode == "auto"
+ assert adaptive.n_clusters >= 1
+
+ tree = datastore._prepare_cluster_tree(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="artifact_hvgs",
+ cluster_key="RNA_artifact_paris",
+ )
+ assert tree["graph"].number_of_nodes() > 0
+ assert datastore.list_artifacts(kind="coalesced_tree", from_assay="RNA")
+
+
+def test_integrated_graphs_use_random_artifacts_and_label_index(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_features(datastore)
+ build_neighbourhood_graph(datastore, **_graph_kwargs())
+ build_neighbourhood_graph(
+ datastore,
+ from_assay="assay2",
+ cell_key="I",
+ feat_key="I",
+ dims=5,
+ k=3,
+ n_centroids=10,
+ batch_size=200,
+ local_cache=False,
+ )
+
+ integrated = datastore.integrate_assays(
+ ["RNA", "assay2"],
+ label="artifact_snn",
+ method="snn",
+ )
+ path = datastore._resolve_integrated_graph_path("artifact_snn")
+ ref = parse_artifact_path(path)
+ assert integrated == ref
+ assert ref.kind == "integrated_graph"
+ assert datastore.inspect_artifact(ref).complete
+ assert "integratedGraphs/artifact_snn" not in datastore.zw
+
+ umap_ref = datastore.run_umap(
+ integrated_graph="artifact_snn",
+ n_epochs=10,
+ label="UMAP",
+ )
+ assert "artifact_snn_UMAP1" in datastore.cells.columns
+ assert datastore.run_umap(integrated, n_epochs=10, label="UMAP") == umap_ref
+ with pytest.raises(ValueError, match="not both"):
+ datastore.run_umap(
+ integrated,
+ integrated_graph="artifact_snn",
+ n_epochs=10,
+ label="UMAP",
+ )
+ datastore.run_paris_clustering(
+ integrated_graph="artifact_snn",
+ n_clusters=3,
+ label="paris",
+ )
+ tree = datastore._prepare_cluster_tree(
+ integrated_graph="artifact_snn",
+ cluster_key="artifact_snn_paris",
+ )
+ assert tree["coalesced_location"].startswith(
+ "artifacts/coalesced_tree/",
+ )
+
+ alias = datastore.integrate_assays(
+ ["RNA", "assay2"],
+ label="artifact_snn_alias",
+ method="snn",
+ )
+ assert alias == integrated
+ with pytest.raises(ValueError, match="shared by labels"):
+ datastore.run_umap(integrated, n_epochs=10, label="UMAP")
+
+
+def test_wnn_uses_exact_nondefault_cell_selection(
+ datastore_ephemeral,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_features(datastore)
+ cells = np.asarray(datastore.cells.fetch_all("I"), dtype=bool)
+ datastore.cells.insert("wnn_cells", cells, overwrite=True)
+ datastore.cells.insert(
+ "wnn_batch",
+ np.where(np.arange(datastore.cells.N) % 2, "a", "b"),
+ overwrite=True,
+ )
+ datastore.RNA.feats.insert(
+ "wnn_cells__artifact_hvgs",
+ datastore.RNA.feats.fetch_all("I__artifact_hvgs"),
+ overwrite=True,
+ )
+ build_neighbourhood_graph(
+ datastore,
+ from_assay="RNA",
+ cell_key="wnn_cells",
+ feat_key="artifact_hvgs",
+ dims=3,
+ k=3,
+ n_centroids=10,
+ harmonize=True,
+ batch_columns=["wnn_batch"],
+ harmony_params={"nclust": 5},
+ local_cache=False,
+ )
+ build_neighbourhood_graph(
+ datastore,
+ from_assay="assay2",
+ cell_key="wnn_cells",
+ feat_key="I",
+ dims=3,
+ k=3,
+ n_centroids=10,
+ local_cache=False,
+ )
+
+ load_graph = datastore.load_graph
+
+ def reject_load_graph(*_args, **_kwargs):
+ raise AssertionError("WNN integration must read neighbors indices directly")
+
+ monkeypatch.setattr(datastore, "load_graph", reject_load_graph)
+ datastore.integrate_assays(
+ ["RNA", "assay2"],
+ label="artifact_wnn",
+ method="wnn",
+ )
+
+ path = datastore._resolve_integrated_graph_path("artifact_wnn")
+ assert path.startswith("artifacts/integrated_graph/")
+ ref = parse_artifact_path(path)
+ group = datastore.zw[path]
+ graph = load_graph(graph_loc=path)
+ assert graph.shape[0] == int(cells.sum())
+ assert group["edges"].dtype == np.dtype(np.uint32)
+ assert group["weights"].dtype == np.dtype(np.float32)
+ assert group["modality_weights"].dtype == np.dtype(np.float32)
+ assert group["modality_weights"].shape == (int(cells.sum()), 2)
+ assert group.attrs["assays"] == ["RNA", "assay2"]
+ np.testing.assert_allclose(
+ np.asarray(group["modality_weights"][:]).sum(axis=1),
+ 1,
+ rtol=1e-6,
+ )
+
+ status = datastore.inspect_artifact(ref)
+ assert status.parameters == {
+ "method": "wnn",
+ "assays": ["RNA", "assay2"],
+ "l2_normalize": True,
+ }
+ assert set(status.inputs["RNA"]) == {"neighbors", "coordinates"}
+ assert set(status.inputs["assay2"]) == {"neighbors", "coordinates"}
+
+ weight_columns = [
+ "artifact_wnn_RNA_weight",
+ "artifact_wnn_assay2_weight",
+ ]
+ for index, column in enumerate(weight_columns):
+ assert column in datastore.cells.columns
+ np.testing.assert_allclose(
+ datastore.cells.fetch(column, "wnn_cells"),
+ np.asarray(group["modality_weights"][:, index]),
+ )
+ attrs = datastore.zw["cellData"][column].attrs
+ assert attrs["source_artifact"] == ref.to_dict()
+ assert attrs["source_value"] == "modality_weights"
+ assert attrs["value_index"] == index
+
+ for column in weight_columns:
+ datastore.cells.drop(column)
+ datastore.integrate_assays(
+ ["RNA", "assay2"],
+ label="artifact_wnn",
+ method="wnn",
+ )
+ assert datastore._resolve_integrated_graph_path("artifact_wnn") == path
+ assert all(column in datastore.cells.columns for column in weight_columns)
+
+ datastore.integrate_assays(
+ ["RNA", "assay2"],
+ label="artifact_wnn_raw",
+ method="wnn",
+ l2_normalize=False,
+ )
+ raw_path = datastore._resolve_integrated_graph_path("artifact_wnn_raw")
+ assert raw_path != path
+ raw_ref = parse_artifact_path(raw_path)
+ assert datastore.inspect_artifact(raw_ref).parameters["l2_normalize"] is False
+ datastore.integrate_assays(
+ ["RNA", "assay2"],
+ label="artifact_wnn_raw",
+ method="wnn",
+ l2_normalize=False,
+ )
+ assert datastore._resolve_integrated_graph_path("artifact_wnn_raw") == raw_path
+
+ expected_edges = np.asarray(group["edges"][:])
+ expected_weights = np.asarray(group["weights"][:])
+ expected_modality_weights = np.asarray(group["modality_weights"][:])
+ datastore.integrate_assays(
+ ["RNA", "assay2"],
+ label="artifact_wnn_rechunked",
+ method="wnn",
+ chunk_size=17,
+ invalidate_cache=True,
+ )
+ rechunked_path = datastore._resolve_integrated_graph_path("artifact_wnn_rechunked")
+ assert rechunked_path != path
+ rechunked = datastore.zw[rechunked_path]
+ np.testing.assert_array_equal(rechunked["edges"][:], expected_edges)
+ np.testing.assert_array_equal(rechunked["weights"][:], expected_weights)
+ np.testing.assert_array_equal(
+ rechunked["modality_weights"][:],
+ expected_modality_weights,
+ )
diff --git a/tests/test_artifact_lineage.py b/tests/test_artifact_lineage.py
new file mode 100644
index 00000000..232aef4d
--- /dev/null
+++ b/tests/test_artifact_lineage.py
@@ -0,0 +1,557 @@
+import networkx as nx
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.datastore.base_datastore import BaseDataStore
+from scarf.lineage import ArtifactLineage
+from scarf.mapping.reference import MappingReference
+from scarf.storage.artifacts import (
+ ArtifactRef,
+ ArtifactScope,
+ ExternalArtifactRef,
+ artifact_path,
+ make_provenance,
+)
+
+
+def _ref(
+ kind: str,
+ token: str,
+ *,
+ scope: ArtifactScope = "assay",
+) -> ArtifactRef:
+ return ArtifactRef(
+ scope=scope,
+ assay="RNA" if scope == "assay" else None,
+ kind=kind,
+ artifact_id=token * 64,
+ )
+
+
+def _write_artifact(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ *,
+ operation: str,
+ inputs: dict,
+ parameters: dict | None = None,
+ execution_options: dict | None = None,
+ complete: bool = True,
+) -> None:
+ group = root.create_group(artifact_path(ref))
+ group.attrs.update(
+ {
+ "artifact_id": ref.artifact_id,
+ "kind": ref.kind,
+ "provenance": make_provenance(
+ operation=operation,
+ parameters=parameters or {},
+ inputs=inputs,
+ ),
+ "execution_options": execution_options or {},
+ "complete": complete,
+ }
+ )
+
+
+class _ReferenceDatastore:
+ def __init__(self, root: zarr.Group) -> None:
+ self.zw = root
+
+ def _get_assay(self, assay_name: str) -> zarr.Group:
+ return self.zw[assay_name]
+
+
+def _mapping_reference(
+ root: zarr.Group,
+ ref: ArtifactRef,
+ fingerprint: str,
+) -> MappingReference:
+ assert ref.assay is not None
+ if ref.assay not in root:
+ root.create_group(ref.assay)
+ root[ref.assay].attrs["dataset_fingerprint"] = fingerprint
+ reference = object.__new__(MappingReference)
+ object.__setattr__(reference, "datastore", _ReferenceDatastore(root))
+ object.__setattr__(reference, "ref", ref)
+ object.__setattr__(reference, "assay_name", ref.assay)
+ object.__setattr__(reference, "dataset_fingerprint", fingerprint)
+ return reference
+
+
+def test_lineage_merges_nested_inputs_and_named_outputs() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ selection = _ref("cell_selection", "1", scope="datastore")
+ normalized = _ref("normalized", "2")
+ reduction = _ref("reduction", "3")
+ embedding = _ref("embedding", "a")
+ _write_artifact(
+ root,
+ selection,
+ operation="manual_selection",
+ inputs={},
+ )
+ _write_artifact(
+ root,
+ normalized,
+ operation="run_normalization",
+ inputs={"selection": {"cells": selection}},
+ )
+ _write_artifact(
+ root,
+ reduction,
+ operation="run_pca",
+ inputs={
+ "normalized": normalized,
+ "copies": [normalized],
+ "source_fingerprint": {"value_fingerprint": "f" * 64},
+ },
+ parameters={"dims": 25},
+ execution_options={"local_cache": False},
+ )
+ _write_artifact(
+ root,
+ embedding,
+ operation="run_umap",
+ inputs={"coordinates": reduction},
+ )
+
+ lineage = ArtifactLineage.from_store(
+ root,
+ {
+ "pca": reduction,
+ "selected_reduction": reduction,
+ "umap": embedding,
+ },
+ )
+ reversed_outputs = ArtifactLineage.from_store(
+ root,
+ {
+ "umap": embedding,
+ "selected_reduction": reduction,
+ "pca": reduction,
+ },
+ )
+
+ assert nx.is_frozen(lineage.graph)
+ assert set(lineage.graph) == {selection, normalized, reduction, embedding}
+ assert lineage.graph.edges[selection, normalized]["inputs"] == ("selection.cells",)
+ assert lineage.graph.edges[normalized, reduction]["inputs"] == (
+ "copies[0]",
+ "normalized",
+ )
+ assert lineage.graph.nodes[reduction]["outputs"] == (
+ "pca",
+ "selected_reduction",
+ )
+ assert lineage.graph.nodes[embedding]["outputs"] == ("umap",)
+ assert lineage.to_mermaid() == reversed_outputs.to_mermaid()
+ assert lineage.to_mermaid() == "\n".join(
+ [
+ "flowchart LR",
+ (
+ ' artifact0["datastore / cell_selection | manual_selection | '
+ '111111111111"]'
+ ),
+ (' artifact1["RNA / normalized | run_normalization | 222222222222"]'),
+ (
+ ' artifact2["RNA / reduction | run_pca | 333333333333 | '
+ 'outputs: pca, selected_reduction"]'
+ ),
+ (
+ ' artifact3["RNA / embedding | run_umap | aaaaaaaaaaaa | '
+ 'outputs: umap"]'
+ ),
+ ' artifact0 -->|"selection.cells"| artifact1',
+ ' artifact1 -->|"copies[0], normalized"| artifact2',
+ ' artifact2 -->|"coordinates"| artifact3',
+ ]
+ )
+
+ markdown = lineage.to_markdown()
+ assert markdown.startswith("```mermaid\nflowchart LR\n")
+ assert "- Parameters: `dims=25`" in markdown
+ assert "- Execution options: `local_cache=false`" in markdown
+ assert "source_fingerprint=" in markdown
+ assert lineage._repr_markdown_() == markdown
+
+
+def test_datastore_lineage_builds_a_bounded_markdown_report() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ ref = _ref("quality_metric", "4", scope="datastore")
+ _write_artifact(
+ root,
+ ref,
+ operation="calculate_qc",
+ inputs={"fingerprint": "a" * 500},
+ parameters={"payload": "b" * 500},
+ )
+ datastore = BaseDataStore.__new__(BaseDataStore)
+ datastore.z = root
+ datastore.workspace = None
+
+ lineage = datastore.lineage(ref)
+ markdown = lineage.to_markdown()
+
+ assert isinstance(lineage, ArtifactLineage)
+ assert lineage.outputs == {"output": ref}
+ assert "a" * 200 not in markdown
+ assert "b" * 200 not in markdown
+ assert "..." in markdown
+
+
+def test_lineage_renders_missing_and_incomplete_artifacts() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ missing = _ref("normalized", "5")
+ consumer = _ref("reduction", "6")
+ unfinished = _ref("ann_index", "7")
+ _write_artifact(
+ root,
+ consumer,
+ operation="run_pca",
+ inputs={"normalized": missing},
+ )
+ _write_artifact(
+ root,
+ unfinished,
+ operation="build_ann_index",
+ inputs={"coordinates": consumer},
+ complete=False,
+ )
+
+ lineage = ArtifactLineage.from_store(root, unfinished)
+ mermaid = lineage.to_mermaid()
+
+ assert len(lineage.graph) == 3
+ assert "status: missing" in mermaid
+ assert "status: incomplete" in mermaid
+ assert lineage.graph.nodes[missing]["status"].exists is False
+ assert lineage.graph.nodes[unfinished]["status"].complete is False
+
+
+def test_lineage_rejects_cyclic_provenance() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ first = _ref("normalized", "8")
+ second = _ref("reduction", "9")
+ _write_artifact(
+ root,
+ first,
+ operation="run_normalization",
+ inputs={"reduction": second},
+ )
+ _write_artifact(
+ root,
+ second,
+ operation="run_pca",
+ inputs={"normalized": first},
+ )
+
+ with pytest.raises(ValueError, match="dependency cycle"):
+ ArtifactLineage.from_store(root, second)
+
+
+def test_lineage_renders_one_unresolved_external_node_without_opening_paths(
+ monkeypatch,
+) -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ mapping_ref = _ref("mapping_reference", "a")
+ external = ExternalArtifactRef("reference-dataset", mapping_ref)
+ projection = _ref("projection", "b")
+ _write_artifact(
+ root,
+ projection,
+ operation="map_query",
+ inputs={
+ "mapping_reference": external,
+ "query_fingerprint": "q" * 64,
+ },
+ )
+
+ def reject_open(*args, **kwargs):
+ raise AssertionError("Lineage must not discover or open an external path")
+
+ monkeypatch.setattr(zarr, "open_group", reject_open)
+ lineage = ArtifactLineage.from_store(root, projection)
+ markdown = lineage.to_markdown()
+
+ assert set(lineage.graph) == {projection, external}
+ assert lineage.graph.nodes[external]["status"] is None
+ assert lineage.graph.edges[external, projection]["inputs"] == ("mapping_reference",)
+ assert "unresolved external" in markdown
+ assert "reference-dataset" in markdown
+ assert "status: missing" not in markdown
+ assert "mapping_reference=" not in markdown
+ assert "query_fingerprint=" in markdown
+
+
+def test_lineage_rejects_malformed_external_artifact_inputs() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ projection = _ref("projection", "0")
+ _write_artifact(
+ root,
+ projection,
+ operation="map_query",
+ inputs={
+ "mapping_reference": {
+ "type": "external_artifact",
+ "dataset_fingerprint": "reference-dataset",
+ "path": "/tmp/reference.zarr",
+ }
+ },
+ )
+
+ with pytest.raises(
+ ValueError,
+ match="Invalid external artifact reference.*mapping_reference",
+ ):
+ ArtifactLineage.from_store(root, projection)
+
+
+def test_lineage_rejects_external_root_with_wrong_fingerprint() -> None:
+ query_root = zarr.open_group(store=MemoryStore(), mode="w")
+ external_root = zarr.open_group(store=MemoryStore(), mode="w")
+ external_root.create_group("RNA").attrs["dataset_fingerprint"] = "wrong-dataset"
+ external = ExternalArtifactRef(
+ "expected-dataset",
+ _ref("mapping_reference", "5"),
+ )
+ projection = _ref("projection", "6")
+ _write_artifact(
+ query_root,
+ projection,
+ operation="map_query",
+ inputs={"mapping_reference": external},
+ )
+
+ with pytest.raises(
+ ValueError,
+ match="Expected 'expected-dataset', received 'wrong-dataset'",
+ ):
+ ArtifactLineage.from_store(
+ query_root,
+ projection,
+ external_roots={"expected-dataset": external_root},
+ )
+
+
+def test_lineage_rejects_external_root_without_stored_fingerprint() -> None:
+ query_root = zarr.open_group(store=MemoryStore(), mode="w")
+ external_root = zarr.open_group(store=MemoryStore(), mode="w")
+ external_root.create_group("RNA")
+ external = ExternalArtifactRef(
+ "expected-dataset",
+ _ref("mapping_reference", "7"),
+ )
+ projection = _ref("projection", "8")
+ _write_artifact(
+ query_root,
+ projection,
+ operation="map_query",
+ inputs={"mapping_reference": external},
+ )
+
+ with pytest.raises(
+ ValueError,
+ match="assay 'RNA' has no stored dataset_fingerprint.*expected-dataset",
+ ):
+ ArtifactLineage.from_store(
+ query_root,
+ projection,
+ external_roots={"expected-dataset": external_root},
+ )
+
+
+def test_lineage_traverses_multiple_namespaced_external_roots() -> None:
+ query_root = zarr.open_group(store=MemoryStore(), mode="w")
+ first_root = zarr.open_group(store=MemoryStore(), mode="w")
+ second_root = zarr.open_group(store=MemoryStore(), mode="w")
+ shared_ref = _ref("mapping_reference", "c")
+ shared_input = _ref("reduction", "d")
+ for root, operation, fingerprint in (
+ (first_root, "first_reference", "first-dataset"),
+ (second_root, "second_reference", "second-dataset"),
+ ):
+ _write_artifact(
+ root,
+ shared_input,
+ operation="run_pca",
+ inputs={},
+ )
+ _write_artifact(
+ root,
+ shared_ref,
+ operation=operation,
+ inputs={"reduction": shared_input},
+ )
+ root["RNA"].attrs["dataset_fingerprint"] = fingerprint
+
+ first = ExternalArtifactRef("first-dataset", shared_ref)
+ second = ExternalArtifactRef("second-dataset", shared_ref)
+ projection = _ref("projection", "e")
+ _write_artifact(
+ query_root,
+ projection,
+ operation="map_query",
+ inputs={"references": [first, second]},
+ )
+
+ lineage = ArtifactLineage.from_store(
+ query_root,
+ projection,
+ external_roots={
+ "first-dataset": first_root,
+ "second-dataset": second_root,
+ },
+ )
+ first_input = ExternalArtifactRef("first-dataset", shared_input)
+ second_input = ExternalArtifactRef("second-dataset", shared_input)
+
+ assert set(lineage.graph) == {
+ projection,
+ first,
+ second,
+ first_input,
+ second_input,
+ }
+ assert first != second
+ assert first_input != second_input
+ assert lineage.graph.edges[first_input, first]["inputs"] == ("reduction",)
+ assert lineage.graph.edges[second_input, second]["inputs"] == ("reduction",)
+ assert lineage.graph.nodes[first]["status"].operation == "first_reference"
+ assert lineage.graph.nodes[second]["status"].operation == "second_reference"
+ assert "unresolved external" not in lineage.to_markdown()
+
+
+def test_lineage_traverses_external_datastore_scoped_dependencies() -> None:
+ query_root = zarr.open_group(store=MemoryStore(), mode="w")
+ reference_root = zarr.open_group(store=MemoryStore(), mode="w")
+ reference_root.create_group("RNA").attrs["dataset_fingerprint"] = (
+ "reference-dataset"
+ )
+ cell_selection = _ref("cell_selection", "3", scope="datastore")
+ mapping_ref = _ref("mapping_reference", "4")
+ _write_artifact(
+ reference_root,
+ cell_selection,
+ operation="manual_selection",
+ inputs={},
+ )
+ _write_artifact(
+ reference_root,
+ mapping_ref,
+ operation="build_mapping_reference",
+ inputs={"cell_selection": cell_selection},
+ )
+ external = ExternalArtifactRef("reference-dataset", mapping_ref)
+ projection = _ref("projection", "5")
+ _write_artifact(
+ query_root,
+ projection,
+ operation="map_query",
+ inputs={"mapping_reference": external},
+ )
+
+ lineage = ArtifactLineage.from_store(
+ query_root,
+ projection,
+ external_roots={"reference-dataset": reference_root},
+ )
+
+ operations = {
+ lineage.graph.nodes[node]["status"].operation for node in lineage.graph
+ }
+ assert operations == {
+ "map_query",
+ "build_mapping_reference",
+ "manual_selection",
+ }
+ assert "external reference-da / datastore" in lineage.to_markdown()
+
+
+def test_datastore_lineage_resolves_explicit_mapping_references() -> None:
+ query_root = zarr.open_group(store=MemoryStore(), mode="w")
+ first_root = zarr.open_group(store=MemoryStore(), mode="w")
+ second_root = zarr.open_group(store=MemoryStore(), mode="w")
+ first_ref = _ref("mapping_reference", "f")
+ second_ref = _ref("mapping_reference", "1")
+ _write_artifact(
+ first_root,
+ first_ref,
+ operation="build_mapping_reference",
+ inputs={},
+ )
+ _write_artifact(
+ second_root,
+ second_ref,
+ operation="build_mapping_reference",
+ inputs={},
+ )
+ first_reference = _mapping_reference(first_root, first_ref, "first-dataset")
+ second_reference = _mapping_reference(second_root, second_ref, "second-dataset")
+ projection = _ref("projection", "2")
+ first_external = first_reference.external_ref
+ second_external = second_reference.external_ref
+ _write_artifact(
+ query_root,
+ projection,
+ operation="map_query",
+ inputs={"references": [first_external, second_external]},
+ )
+ datastore = BaseDataStore.__new__(BaseDataStore)
+ datastore.z = query_root
+ datastore.workspace = None
+
+ partly_resolved = datastore.lineage(
+ projection,
+ references=first_reference,
+ )
+ resolved = datastore.lineage(
+ projection,
+ references=(first_reference, second_reference),
+ )
+
+ assert partly_resolved.graph.nodes[first_external]["status"].complete
+ assert partly_resolved.graph.nodes[second_external]["status"] is None
+ assert resolved.graph.nodes[first_external]["status"].complete
+ assert resolved.graph.nodes[second_external]["status"].complete
+
+
+def test_datastore_lineage_validates_reference_fingerprints_and_root_conflicts() -> (
+ None
+):
+ query_root = zarr.open_group(store=MemoryStore(), mode="w")
+ target = _ref("projection", "3")
+ _write_artifact(
+ query_root,
+ target,
+ operation="map_query",
+ inputs={},
+ )
+ datastore = BaseDataStore.__new__(BaseDataStore)
+ datastore.z = query_root
+ datastore.workspace = None
+
+ first_root = zarr.open_group(store=MemoryStore(), mode="w")
+ second_root = zarr.open_group(store=MemoryStore(), mode="w")
+ reference_ref = _ref("mapping_reference", "4")
+ first = _mapping_reference(first_root, reference_ref, "shared-dataset")
+ second = _mapping_reference(second_root, reference_ref, "shared-dataset")
+ same_root = _mapping_reference(first_root, reference_ref, "shared-dataset")
+ datastore.lineage(target, references=(first, same_root))
+ with pytest.raises(ValueError, match="conflicting roots"):
+ datastore.lineage(target, references=(first, second))
+
+ first_root["RNA"].attrs["dataset_fingerprint"] = "changed-dataset"
+ with pytest.raises(
+ ValueError,
+ match="Expected 'shared-dataset', received 'changed-dataset'",
+ ):
+ datastore.lineage(target, references=first)
+
+ del first_root["RNA"].attrs["dataset_fingerprint"]
+ with pytest.raises(
+ ValueError,
+ match="no stored dataset fingerprint.*build_mapping_reference",
+ ):
+ datastore.lineage(target, references=first)
diff --git a/tests/test_artifact_selection_errors.py b/tests/test_artifact_selection_errors.py
new file mode 100644
index 00000000..77106f3c
--- /dev/null
+++ b/tests/test_artifact_selection_errors.py
@@ -0,0 +1,185 @@
+import json
+import pickle
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+import scarf
+from scarf.graph.state import (
+ ArtifactSelectionError,
+ validate_cell_selection_artifact,
+)
+from scarf.storage.artifacts import artifact_path
+from scarf.storage.refs import ArtifactRef
+from scarf.storage.selections import resolve_selection_artifact
+
+
+def _root_with_selection() -> tuple[zarr.Group, MemoryStore, ArtifactRef]:
+ store = MemoryStore()
+ root = zarr.open_group(store=store, mode="w")
+ cells = root.create_group("cellData")
+ cell_ids = np.array(["a", "b", "c"])
+ selection = np.array([True, False, True])
+ cells.create_array("ids", data=cell_ids)
+ cells.create_array("I", data=selection)
+ ref = resolve_selection_artifact(
+ root,
+ scope="datastore",
+ kind="cell_selection",
+ values=selection,
+ row_ids=cell_ids,
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column="I",
+ )
+ return root, store, ref
+
+
+def _read_only_root(store: MemoryStore) -> zarr.Group:
+ return zarr.open_group(store=store.with_read_only(True), mode="r")
+
+
+def _assert_selection_error(
+ store: MemoryStore,
+ ref: ArtifactRef,
+ code: str,
+) -> ArtifactSelectionError:
+ with pytest.raises(ArtifactSelectionError) as caught:
+ validate_cell_selection_artifact(_read_only_root(store), ref, "I")
+
+ error = caught.value
+ assert isinstance(error, ValueError)
+ assert error.code == code
+ assert json.loads(json.dumps(error.context, allow_nan=False)) == error.context
+ assert "zarr_loc" not in error.context
+ return error
+
+
+def test_selection_validation_is_read_only() -> None:
+ _, store, ref = _root_with_selection()
+
+ validate_cell_selection_artifact(_read_only_root(store), ref, "I")
+
+
+def test_selection_error_for_reference_mismatch() -> None:
+ _, store, ref = _root_with_selection()
+ wrong_ref = ArtifactRef(
+ scope="assay",
+ assay="RNA",
+ kind="feature_selection",
+ artifact_id=ref.artifact_id,
+ )
+
+ error = _assert_selection_error(
+ store,
+ wrong_ref,
+ "artifact_reference_mismatch",
+ )
+
+ assert error.context["expected_scope"] == "datastore"
+ assert error.context["actual_scope"] == "assay"
+ assert error.context["actual_kind"] == "feature_selection"
+
+
+def test_selection_error_for_missing_artifact() -> None:
+ _, store, ref = _root_with_selection()
+ missing_ref = ArtifactRef(
+ scope=ref.scope,
+ kind=ref.kind,
+ artifact_id="f" * 64,
+ )
+
+ error = _assert_selection_error(store, missing_ref, "artifact_missing")
+
+ assert "does not exist" in str(error)
+ assert error.context["artifact_id"] == "f" * 64
+
+
+def test_selection_error_for_incomplete_artifact() -> None:
+ root, store, ref = _root_with_selection()
+ root[artifact_path(ref)].attrs["complete"] = False
+
+ error = _assert_selection_error(store, ref, "artifact_incomplete")
+
+ assert "incomplete" in str(error)
+
+
+def test_selection_error_for_missing_table() -> None:
+ root, store, ref = _root_with_selection()
+ del root["cellData"]
+
+ error = _assert_selection_error(store, ref, "selection_table_missing")
+
+ assert error.context["table"] == "cellData"
+
+
+def test_selection_error_for_missing_column() -> None:
+ root, store, ref = _root_with_selection()
+ del root["cellData/I"]
+
+ error = _assert_selection_error(store, ref, "selection_column_missing")
+
+ assert error.context["column"] == "I"
+
+
+def test_selection_error_for_missing_row_ids() -> None:
+ root, store, ref = _root_with_selection()
+ del root["cellData/ids"]
+
+ error = _assert_selection_error(store, ref, "selection_row_ids_missing")
+
+ assert error.context["table"] == "cellData"
+
+
+def test_selection_error_for_missing_artifact_values() -> None:
+ root, store, ref = _root_with_selection()
+ del root[f"{artifact_path(ref)}/values"]
+
+ error = _assert_selection_error(store, ref, "selection_values_missing")
+
+ assert error.context["kind"] == "cell_selection"
+
+
+def test_selection_error_for_changed_row_identity() -> None:
+ root, store, ref = _root_with_selection()
+ root["cellData/ids"][:] = np.array(["x", "y", "z"])
+
+ error = _assert_selection_error(store, ref, "row_identity_mismatch")
+
+ assert "row identity" in str(error)
+
+
+def test_selection_error_for_changed_values() -> None:
+ root, store, ref = _root_with_selection()
+ root["cellData/I"][:] = np.array([False, True, False])
+
+ error = _assert_selection_error(store, ref, "selection_values_changed")
+
+ assert "no longer matches" in str(error)
+
+
+def test_artifact_selection_error_has_public_facades() -> None:
+ assert scarf.ArtifactSelectionError is ArtifactSelectionError
+ assert scarf.graph.ArtifactSelectionError is ArtifactSelectionError
+
+
+def test_artifact_selection_error_round_trips_through_pickle() -> None:
+ error = ArtifactSelectionError(
+ "Stored selection is unavailable",
+ code="artifact_missing",
+ context={
+ "kind": "cell_selection",
+ "artifact_id": "f" * 64,
+ },
+ )
+
+ restored = pickle.loads(pickle.dumps(error))
+
+ assert type(restored) is ArtifactSelectionError
+ assert restored.args == error.args
+ assert str(restored) == str(error)
+ assert restored.code == error.code
+ assert restored.context == error.context
diff --git a/tests/test_artifact_writer.py b/tests/test_artifact_writer.py
new file mode 100644
index 00000000..82c18367
--- /dev/null
+++ b/tests/test_artifact_writer.py
@@ -0,0 +1,242 @@
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.storage.artifact_writer import (
+ ArrayRequirement,
+ AttributeRequirement,
+ finish_artifact,
+ plan_artifact,
+ reused_artifact_group,
+ start_artifact,
+)
+from scarf.storage.artifacts import artifact_path, inspect_artifact
+
+
+def test_artifact_writer_streams_to_random_path_then_reuses_provenance() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ arguments = {
+ "scope": "assay",
+ "assay": "RNA",
+ "kind": "normalized",
+ "operation": "run_normalization",
+ "parameters": {"log_transform": True},
+ "inputs": {"selection": {"artifact_id": "a" * 64}},
+ "execution_options": {"batch_size": 100},
+ }
+ planned = plan_artifact(root, **arguments)
+ assert not planned.reused
+ group = start_artifact(root, planned)
+ assert not inspect_artifact(root, planned.ref).complete
+ group.create_array("data", data=np.array([1.0, 2.0, 3.0]))
+ finish_artifact(group, planned)
+ assert inspect_artifact(root, planned.ref).complete
+
+ reused = plan_artifact(root, **arguments)
+ assert reused.reused
+ assert reused.ref == planned.ref
+ assert reused_artifact_group(root, reused).path == group.path
+
+ invalidated = plan_artifact(
+ root,
+ **arguments,
+ invalidate_cache=True,
+ )
+ assert not invalidated.reused
+ assert invalidated.ref != planned.ref
+ assert artifact_path(invalidated.ref) not in root
+ refreshed_group = start_artifact(root, invalidated)
+ refreshed_group.create_array("data", data=np.array([4.0, 5.0, 6.0]))
+ finish_artifact(refreshed_group, invalidated)
+ preferred = plan_artifact(root, **arguments)
+ assert preferred.reused
+ assert preferred.ref == invalidated.ref
+
+
+def test_incomplete_artifact_is_not_reused() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ arguments = {
+ "scope": "assay",
+ "assay": "RNA",
+ "kind": "ann_index",
+ "operation": "build_ann_index",
+ "parameters": {"ann_parallel": True},
+ "inputs": {"coordinates": {"artifact_id": "b" * 64}},
+ "execution_options": {},
+ }
+ first = plan_artifact(root, **arguments)
+ start_artifact(root, first)
+
+ second = plan_artifact(root, **arguments)
+ assert not second.reused
+ assert second.ref != first.ref
+
+
+def test_execution_options_use_artifact_value_serialization() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ planned = plan_artifact(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="normalized",
+ operation="run_normalization",
+ parameters={},
+ inputs={},
+ execution_options={
+ "nan": float("nan"),
+ "positive_infinity": float("inf"),
+ "negative_infinity": float("-inf"),
+ "bytes": b"\x00\xff",
+ "set": {2, 1},
+ "numpy_scalar": np.int64(3),
+ },
+ )
+
+ start_artifact(root, planned)
+ status = inspect_artifact(root, planned.ref)
+
+ assert status.execution_options == {
+ "nan": {"special_float": "nan"},
+ "positive_infinity": {"special_float": "inf"},
+ "negative_infinity": {"special_float": "-inf"},
+ "bytes": {"bytes_hex": "00ff"},
+ "set": [1, 2],
+ "numpy_scalar": 3,
+ }
+
+
+def test_missing_required_attribute_prevents_reuse() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ arguments = {
+ "scope": "assay",
+ "assay": "RNA",
+ "kind": "mapping_reference",
+ "operation": "build_mapping_reference",
+ "parameters": {"method": "symphony"},
+ "inputs": {"reduction": {"artifact_id": "c" * 64}},
+ "execution_options": {},
+ }
+ first = plan_artifact(root, **arguments)
+ group = start_artifact(root, first)
+ group.create_array("data", data=np.array([1.0]))
+ finish_artifact(group, first)
+
+ second = plan_artifact(
+ root,
+ **arguments,
+ required_arrays=("data",),
+ required_attributes=("reference_metadata",),
+ )
+
+ assert not second.reused
+ assert second.ref != first.ref
+
+
+def test_invalid_required_attribute_type_prevents_reuse() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ arguments = {
+ "scope": "assay",
+ "assay": "RNA",
+ "kind": "mapping_reference",
+ "operation": "build_mapping_reference",
+ "parameters": {"method": "symphony"},
+ "inputs": {"reduction": {"artifact_id": "d" * 64}},
+ "execution_options": {},
+ }
+ first = plan_artifact(root, **arguments)
+ group = start_artifact(root, first)
+ group.attrs["reference_metadata"] = "invalid"
+ finish_artifact(group, first)
+
+ second = plan_artifact(
+ root,
+ **arguments,
+ required_attributes=(
+ AttributeRequirement(
+ "reference_metadata",
+ expected_types=(dict,),
+ ),
+ ),
+ )
+
+ assert not second.reused
+ assert second.ref != first.ref
+
+
+def test_finish_rejects_payload_that_violates_declared_shape() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ planned = plan_artifact(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="normalized",
+ operation="run_normalization",
+ parameters={},
+ inputs={},
+ execution_options={},
+ required_arrays=(ArrayRequirement("data", shape=(3,), dtype_kind="f"),),
+ )
+ group = start_artifact(root, planned)
+ group.create_array("data", data=np.array([1.0, 2.0]))
+
+ with pytest.raises(ValueError, match="does not satisfy"):
+ finish_artifact(group, planned)
+
+ assert not inspect_artifact(root, planned.ref).complete
+
+
+def test_exact_dtype_requirement_prevents_reuse() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ arguments = {
+ "scope": "assay",
+ "assay": "RNA",
+ "kind": "reduction",
+ "operation": "run_pca",
+ "parameters": {},
+ "inputs": {},
+ "execution_options": {},
+ }
+ first = plan_artifact(root, **arguments)
+ group = start_artifact(root, first)
+ group.create_array("data", data=np.array([1.0], dtype=np.float64))
+ finish_artifact(group, first)
+
+ same_kind = plan_artifact(
+ root,
+ **arguments,
+ required_arrays=(ArrayRequirement("data", dtype_kind="f"),),
+ )
+ exact = plan_artifact(
+ root,
+ **arguments,
+ required_arrays=(ArrayRequirement("data", dtype=np.float32),),
+ )
+
+ assert same_kind.reused
+ assert not exact.reused
+
+
+def test_planned_artifact_invalidated_forces_new_ref() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ arguments = {
+ "scope": "assay",
+ "assay": "RNA",
+ "kind": "normalized",
+ "operation": "run_normalization",
+ "parameters": {"log_transform": True},
+ "inputs": {"selection": {"artifact_id": "e" * 64}},
+ "execution_options": {"batch_size": 50},
+ }
+ planned = plan_artifact(root, **arguments)
+ group = start_artifact(root, planned)
+ group.create_array("data", data=np.array([1.0, 2.0, 3.0]))
+ finish_artifact(group, planned)
+
+ reused = plan_artifact(root, **arguments)
+ assert reused.reused
+ invalidated = reused.invalidated(root)
+ assert not invalidated.reused
+ assert invalidated.ref != reused.ref
+ assert invalidated.provenance == reused.provenance
+ assert invalidated.execution_options == reused.execution_options
diff --git a/tests/test_artifacts.py b/tests/test_artifacts.py
new file mode 100644
index 00000000..389c8559
--- /dev/null
+++ b/tests/test_artifacts.py
@@ -0,0 +1,627 @@
+from pathlib import Path
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.assay import norm_lib_size, norm_tf_idf
+from scarf.datastore.datastore import DataStore
+from scarf.datastore.base_datastore import BaseDataStore
+from scarf.storage.artifact_writer import (
+ finish_artifact,
+ plan_artifact,
+ start_artifact,
+)
+from scarf.storage.artifacts import (
+ ARTIFACT_KINDS,
+ ArtifactRef,
+ ArtifactScope,
+ ExternalArtifactRef,
+ ValueFingerprintBuilder,
+ artifact_exists,
+ artifact_group,
+ artifact_path,
+ canonical_bytes,
+ callable_identity,
+ find_reusable_artifacts,
+ fingerprint_array,
+ fingerprint_stored_strings,
+ fingerprint_strings,
+ group_at,
+ inspect_artifact,
+ list_artifacts,
+ make_provenance,
+ new_artifact_id,
+ parse_artifact_path,
+ provenance_hash,
+ require_complete_artifact,
+ serialize_artifact_value,
+)
+
+
+def _ref(
+ kind: str = "normalized",
+ artifact_id: str = "a" * 64,
+) -> ArtifactRef:
+ return ArtifactRef(
+ scope="assay",
+ assay="RNA",
+ kind=kind,
+ artifact_id=artifact_id,
+ )
+
+
+def test_callable_identity_requires_dynamic_explicit_identity() -> None:
+ with pytest.raises(ValueError, match="artifact_identity"):
+ callable_identity(lambda values: values)
+
+ def configured(values):
+ return values
+
+ configured.artifact_identity = "configured" # type: ignore[attr-defined]
+ assert callable_identity(configured) == {"identity": "configured"}
+
+
+def test_tfidf_uses_semantic_artifact_identity_only_for_atac() -> None:
+ assert callable_identity(norm_tf_idf) == {
+ "identity": "scarf.assay.norm_tf_idf:selected-cell-df:total-count-tf"
+ }
+ assert callable_identity(norm_lib_size) == {
+ "module": "scarf.assay",
+ "qualname": "norm_lib_size",
+ }
+
+
+def test_datastore_load_artifact_resolves_workspace_path() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ workspace = root.create_group("analysis")
+ planned = plan_artifact(
+ workspace,
+ scope="assay",
+ assay="RNA",
+ kind="normalized",
+ operation="run_normalization",
+ parameters={},
+ inputs={},
+ execution_options={},
+ )
+ group = start_artifact(workspace, planned)
+ group.create_array("data", data=np.ones((2, 2)))
+ finish_artifact(group, planned)
+ datastore = BaseDataStore.__new__(BaseDataStore)
+ datastore.z = root
+ datastore.workspace = "analysis"
+
+ loaded = datastore.load_artifact(planned.ref)
+
+ assert loaded.path == f"analysis/{artifact_path(planned.ref)}"
+ np.testing.assert_array_equal(loaded["data"][:], np.ones((2, 2)))
+
+
+def _complete_artifact(
+ root: zarr.Group,
+ *,
+ kind: str,
+ operation: str,
+ parameters: dict,
+ inputs: dict,
+ scope: ArtifactScope = "assay",
+) -> ArtifactRef:
+ ref = ArtifactRef(
+ scope=scope,
+ assay="RNA" if scope == "assay" else None,
+ kind=kind,
+ artifact_id=new_artifact_id(),
+ )
+ group = root.create_group(artifact_path(ref))
+ group.attrs.update(
+ {
+ "artifact_id": ref.artifact_id,
+ "kind": ref.kind,
+ "provenance": make_provenance(
+ operation=operation,
+ parameters=parameters,
+ inputs=inputs,
+ ),
+ "execution_options": {},
+ "complete": True,
+ }
+ )
+ return ref
+
+
+def test_artifact_group_helpers_and_require_complete() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ ref = _complete_artifact(
+ root,
+ kind="normalized",
+ operation="run_normalization",
+ parameters={},
+ inputs={},
+ )
+ path = artifact_path(ref)
+ assert group_at(root, path).path == root[path].path
+ assert artifact_group(root, ref).path == root[path].path
+ status = require_complete_artifact(root, ref)
+ assert status.complete
+ assert status.path == path
+
+ missing = ArtifactRef(
+ scope="assay",
+ assay="RNA",
+ kind="normalized",
+ artifact_id="f" * 64,
+ )
+ with pytest.raises(KeyError, match="does not exist"):
+ require_complete_artifact(root, missing)
+
+ incomplete = ArtifactRef(
+ scope="assay",
+ assay="RNA",
+ kind="normalized",
+ artifact_id="c" * 64,
+ )
+ incomplete_path = artifact_path(incomplete)
+ group = root.create_group(incomplete_path)
+ group.attrs.update(
+ {
+ "artifact_id": incomplete.artifact_id,
+ "kind": incomplete.kind,
+ "provenance": make_provenance(
+ operation="run_normalization",
+ parameters={},
+ inputs={},
+ ),
+ "execution_options": {},
+ "complete": False,
+ }
+ )
+ with pytest.raises(RuntimeError, match="incomplete"):
+ require_complete_artifact(root, incomplete)
+
+
+def test_artifact_kinds_cover_the_reviewed_taxonomy() -> None:
+ assert {
+ "cell_selection",
+ "feature_selection",
+ "metadata_snapshot",
+ "normalized",
+ "feature_scaling",
+ "reduction",
+ "batch_correction",
+ "ann_index",
+ "neighbors",
+ "connectivity_map",
+ "embedding_initialization",
+ "embedding",
+ "cluster_labels",
+ "marker_table",
+ "enrichment_scores",
+ "diffusion_operator",
+ "mapping_reference",
+ "projection",
+ "integrated_graph",
+ "imported_coordinates",
+ "wnn_coordinates",
+ } <= ARTIFACT_KINDS
+
+
+def test_artifact_ids_are_random_storage_addresses() -> None:
+ first = new_artifact_id()
+ second = new_artifact_id()
+ assert first != second
+ assert len(first) == len(second) == 64
+
+ assay_ref = _ref(artifact_id=first)
+ datastore_ref = ArtifactRef(
+ scope="datastore",
+ kind="integrated_graph",
+ artifact_id=second,
+ )
+ assert artifact_path(assay_ref) == f"RNA/artifacts/normalized/{first}"
+ assert artifact_path(datastore_ref) == f"artifacts/integrated_graph/{second}"
+ assert ArtifactRef.from_dict(assay_ref.to_dict()) == assay_ref
+ assert parse_artifact_path(artifact_path(assay_ref)) == assay_ref
+ assert parse_artifact_path(artifact_path(datastore_ref)) == datastore_ref
+
+
+def test_artifact_ref_repr_truncates_the_artifact_id() -> None:
+ assay_ref = _ref(artifact_id="ab" * 32)
+ datastore_ref = ArtifactRef(
+ scope="datastore",
+ kind="integrated_graph",
+ artifact_id="cd" * 32,
+ )
+
+ assert repr(assay_ref) == (
+ "ArtifactRef(assay='RNA', kind='normalized', artifact_id='abababababab...')"
+ )
+ assert repr(datastore_ref) == (
+ "ArtifactRef(datastore, kind='integrated_graph', artifact_id='cdcdcdcdcdcd...')"
+ )
+ assert assay_ref.to_dict()["artifact_id"] == "ab" * 32
+
+
+def test_artifact_ref_rejects_malformed_values() -> None:
+ with pytest.raises(ValueError, match="require an assay"):
+ ArtifactRef(scope="assay", kind="normalized", artifact_id="a" * 64)
+ with pytest.raises(ValueError, match="cannot set assay"):
+ ArtifactRef(
+ scope="datastore",
+ assay="RNA",
+ kind="integrated_graph",
+ artifact_id="a" * 64,
+ )
+ with pytest.raises(ValueError, match="Unknown artifact kind"):
+ ArtifactRef(
+ scope="assay",
+ assay="RNA",
+ kind="normalized__log_true",
+ artifact_id="a" * 64,
+ )
+ with pytest.raises(ValueError, match="Not an artifact path"):
+ parse_artifact_path("RNA/normed__I__hvgs")
+ missing_type = _ref().to_dict()
+ del missing_type["type"]
+ with pytest.raises(ValueError, match="type must be"):
+ ArtifactRef.from_dict(missing_type)
+
+
+def test_external_artifact_ref_round_trips_strictly() -> None:
+ ref = _ref(kind="embedding", artifact_id="b" * 64)
+ external = ExternalArtifactRef(
+ dataset_fingerprint="reference-dataset",
+ ref=ref,
+ )
+ serialized = {
+ "type": "external_artifact",
+ "dataset_fingerprint": "reference-dataset",
+ "ref": ref.to_dict(),
+ }
+
+ assert external.to_dict() == serialized
+ assert ExternalArtifactRef.from_dict(serialized) == external
+ assert serialize_artifact_value(external) == serialized
+ assert canonical_bytes(external) == canonical_bytes(serialized)
+ assert not {
+ "path",
+ "uri",
+ "workspace",
+ "storage_options",
+ "credentials",
+ "schema",
+ "version",
+ } & set(serialized)
+
+
+def test_external_artifact_ref_rejects_malformed_values() -> None:
+ assay_ref = _ref()
+ datastore_ref = ArtifactRef(
+ scope="datastore",
+ kind="integrated_graph",
+ artifact_id="c" * 64,
+ )
+ with pytest.raises(ValueError, match="non-empty"):
+ ExternalArtifactRef(dataset_fingerprint="", ref=assay_ref)
+ with pytest.raises(ValueError, match="assay-scoped"):
+ ExternalArtifactRef(
+ dataset_fingerprint="reference-dataset",
+ ref=datastore_ref,
+ )
+
+ serialized = ExternalArtifactRef(
+ dataset_fingerprint="reference-dataset",
+ ref=assay_ref,
+ ).to_dict()
+ with pytest.raises(ValueError, match="contain exactly"):
+ ExternalArtifactRef.from_dict({**serialized, "path": "/tmp/reference"})
+ with pytest.raises(ValueError, match="contain exactly"):
+ ExternalArtifactRef.from_dict(
+ {
+ "type": "external_artifact",
+ "dataset_fingerprint": "reference-dataset",
+ }
+ )
+ with pytest.raises(ValueError, match="type must be"):
+ ExternalArtifactRef.from_dict({**serialized, "type": "artifact"})
+ with pytest.raises(ValueError, match="complete assay artifact"):
+ ExternalArtifactRef.from_dict(
+ {
+ **serialized,
+ "ref": {**assay_ref.to_dict(), "path": "reference.zarr"},
+ }
+ )
+ with pytest.raises(ValueError, match="assay-scoped"):
+ ExternalArtifactRef.from_dict(
+ {
+ **serialized,
+ "ref": {**datastore_ref.to_dict(), "assay": None},
+ }
+ )
+
+
+def test_external_artifact_ref_has_stable_provenance_identity() -> None:
+ ref = _ref(kind="mapping_reference", artifact_id="d" * 64)
+ external = ExternalArtifactRef("reference-dataset", ref)
+ round_tripped = ExternalArtifactRef.from_dict(external.to_dict())
+ same = make_provenance(
+ operation="map_query",
+ parameters={},
+ inputs={"mapping_reference": round_tripped},
+ )
+ original = make_provenance(
+ operation="map_query",
+ parameters={},
+ inputs={"mapping_reference": external},
+ )
+ other_dataset = make_provenance(
+ operation="map_query",
+ parameters={},
+ inputs={
+ "mapping_reference": ExternalArtifactRef(
+ "other-dataset",
+ ref,
+ )
+ },
+ )
+ other_artifact = make_provenance(
+ operation="map_query",
+ parameters={},
+ inputs={
+ "mapping_reference": ExternalArtifactRef(
+ "reference-dataset",
+ _ref(kind="mapping_reference", artifact_id="e" * 64),
+ )
+ },
+ )
+
+ assert original["inputs"]["mapping_reference"] == external.to_dict()
+ assert provenance_hash(original) == provenance_hash(same)
+ assert provenance_hash(original) != provenance_hash(other_dataset)
+ assert provenance_hash(original) != provenance_hash(other_artifact)
+
+
+def test_provenance_hash_is_typed_order_independent_and_not_persisted() -> None:
+ normalized = _ref()
+ first = make_provenance(
+ operation="run_reduction",
+ parameters={
+ "dims": np.int64(12),
+ "feat_scaling": True,
+ "method": "pca",
+ "zero": -0.0,
+ },
+ inputs={"normalized": normalized},
+ )
+ second = make_provenance(
+ operation="run_reduction",
+ parameters={
+ "zero": -0.0,
+ "method": "pca",
+ "feat_scaling": True,
+ "dims": 12,
+ },
+ inputs={"normalized": normalized},
+ )
+ integer = make_provenance(
+ operation="run_reduction",
+ parameters={"value": 1},
+ inputs={},
+ )
+ floating = make_provenance(
+ operation="run_reduction",
+ parameters={"value": 1.0},
+ inputs={},
+ )
+
+ assert provenance_hash(first) == provenance_hash(second)
+ assert provenance_hash(integer) != provenance_hash(floating)
+ assert canonical_bytes(Path("/tmp/cache")) == canonical_bytes("/tmp/cache")
+ assert "provenance_hash" not in first
+
+
+def test_provenance_replaces_array_values_with_fingerprints() -> None:
+ values = np.arange(12, dtype=np.float64).reshape(3, 4)
+ provenance = make_provenance(
+ operation="run_reduction",
+ parameters={},
+ inputs={"custom_loadings": values},
+ )
+
+ assert provenance["inputs"]["custom_loadings"] == {
+ "value_fingerprint": fingerprint_array(values)
+ }
+ with pytest.raises(ValueError, match="non-finite"):
+ canonical_bytes({"value": np.inf})
+ with pytest.raises(TypeError, match="value_fingerprint"):
+ canonical_bytes({"values": values})
+
+
+def test_streamed_value_fingerprint_is_block_size_independent() -> None:
+ values = np.arange(24, dtype=np.float64).reshape(6, 4)
+ whole = ValueFingerprintBuilder()
+ whole.update_array("values", values)
+ streamed = ValueFingerprintBuilder()
+ streamed.begin_array("values", values.shape, values.dtype)
+ streamed.update_array_block("values", (0, 0), values[:2])
+ streamed.update_array_block("values", (2, 0), values[2:5])
+ streamed.update_array_block("values", (5, 0), values[5:])
+ streamed.end_array("values")
+
+ assert streamed.hexdigest() == whole.hexdigest()
+
+
+def test_stored_string_fingerprint_supports_variable_length_strings() -> None:
+ values = np.array(["a", "longer", "中"], dtype=np.dtypes.StringDType())
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ array = root.create_array("ids", data=values, chunks=(1,))
+
+ expected = fingerprint_strings(values.astype("U6"))
+ assert fingerprint_stored_strings(array) == expected
+
+
+def test_value_fingerprint_normalizes_endian_alignment_and_padding() -> None:
+ little_dtype = np.dtype([("count", "i2"), ("score", ">f8")], align=True)
+ left = np.empty(6, dtype=little_dtype)
+ right = np.empty(6, dtype=little_dtype)
+ left.view(np.uint8)[:] = 0
+ right.view(np.uint8)[:] = 255
+ for values in (left, right):
+ values["count"] = np.arange(6)
+ values["score"] = np.linspace(0.0, 1.0, 6)
+ big_endian = np.empty(6, dtype=big_dtype)
+ big_endian["count"] = left["count"]
+ big_endian["score"] = left["score"]
+
+ assert fingerprint_array(left) == fingerprint_array(right)
+ assert fingerprint_array(left) == fingerprint_array(big_endian)
+
+
+@pytest.mark.parametrize("zarr_format", [2, 3])
+def test_artifact_inspection_listing_and_reuse_are_read_only(
+ zarr_format: int,
+) -> None:
+ root = zarr.open_group(
+ store=MemoryStore(),
+ mode="w",
+ zarr_format=zarr_format,
+ )
+ provenance = make_provenance(
+ operation="run_normalization",
+ parameters={"log_transform": True},
+ inputs={"cell_selection": _ref("cell_selection", "c" * 64)},
+ )
+ complete_ref = _complete_artifact(
+ root,
+ kind="normalized",
+ operation="run_normalization",
+ parameters=provenance["parameters"],
+ inputs=provenance["inputs"],
+ )
+ incomplete_ref = _ref(artifact_id="e" * 64)
+ incomplete = root.create_group(artifact_path(incomplete_ref))
+ incomplete.attrs.update(
+ {
+ "artifact_id": incomplete_ref.artifact_id,
+ "kind": incomplete_ref.kind,
+ "provenance": provenance,
+ "execution_options": {},
+ "complete": False,
+ }
+ )
+ datastore_ref = _complete_artifact(
+ root,
+ scope="datastore",
+ kind="integrated_graph",
+ operation="integrate_assays",
+ parameters={"method": "wnn"},
+ inputs={},
+ )
+ before = dict(root[artifact_path(complete_ref)].attrs)
+
+ status = inspect_artifact(root, complete_ref)
+ assert status.exists
+ assert status.complete
+ assert status.parameters == {"log_transform": True}
+ assert status.inputs == provenance["inputs"]
+ assert artifact_exists(root, complete_ref)
+ assert not artifact_exists(root, incomplete_ref)
+ assert list_artifacts(root, scope="assay", assay="RNA") == sorted(
+ [complete_ref, incomplete_ref],
+ key=lambda ref: ref.artifact_id,
+ )
+ assert list_artifacts(root, scope="datastore") == [datastore_ref]
+ assert find_reusable_artifacts(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="normalized",
+ provenance=provenance,
+ ) == [complete_ref]
+ assert (
+ find_reusable_artifacts(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="normalized",
+ provenance=provenance,
+ invalidate_cache=True,
+ )
+ == []
+ )
+ assert dict(root[artifact_path(complete_ref)].attrs) == before
+
+
+def test_nondeterministic_artifacts_use_normal_provenance_reuse() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ provenance = make_provenance(
+ operation="build_ann_index",
+ parameters={"ann_parallel": True, "ann_metric": "l2"},
+ inputs={"coordinates": _ref("reduction", "b" * 64)},
+ )
+ ref = _complete_artifact(
+ root,
+ kind="ann_index",
+ operation="build_ann_index",
+ parameters=provenance["parameters"],
+ inputs=provenance["inputs"],
+ )
+
+ assert find_reusable_artifacts(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="ann_index",
+ provenance=provenance,
+ ) == [ref]
+
+
+def test_datastore_exposes_artifact_inspection_without_writing() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ ref = _complete_artifact(
+ root,
+ kind="normalized",
+ operation="run_normalization",
+ parameters={},
+ inputs={},
+ )
+ datastore = DataStore.__new__(DataStore)
+ datastore.z = root
+ datastore.workspace = None
+ datastore._defaultAssay = "RNA"
+
+ assert datastore.inspect_artifact(ref).complete
+ assert datastore.list_artifacts() == [ref]
+
+
+def test_completed_artifact_requires_full_provenance_record() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ ref = _ref(artifact_id="4" * 64)
+ group = root.create_group(artifact_path(ref))
+ group.attrs.update(
+ {
+ "artifact_id": ref.artifact_id,
+ "kind": ref.kind,
+ "complete": True,
+ }
+ )
+ with pytest.raises(KeyError, match="missing attrs"):
+ inspect_artifact(root, ref)
+
+ group.attrs.update(
+ {
+ "provenance": {
+ "operation": "run_normalization",
+ "parameters": {},
+ "inputs": {},
+ },
+ "execution_options": {},
+ }
+ )
+ assert inspect_artifact(root, ref).complete
+
+ group.attrs["complete"] = "false"
+ with pytest.raises(TypeError, match="must be boolean"):
+ inspect_artifact(root, ref)
diff --git a/tests/test_assay_contracts.py b/tests/test_assay_contracts.py
new file mode 100644
index 00000000..75e8c2d4
--- /dev/null
+++ b/tests/test_assay_contracts.py
@@ -0,0 +1,304 @@
+import inspect
+from types import SimpleNamespace
+from typing import get_type_hints
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+import scarf.assay as assay_module
+from scarf.assay import (
+ ADTassay,
+ ATACassay,
+ Assay,
+ RNAassay,
+ lib_size_feature_stream_eligible,
+ norm_clr,
+ norm_dummy,
+ norm_lib_size,
+ norm_tf_idf,
+)
+from tests.signature_contracts import signature_digest
+
+
+_PUBLIC_CLASS_METHODS = {
+ Assay: (
+ "__init__",
+ "normed",
+ "to_raw_sparse",
+ "add_percent_feature",
+ "save_normalized_data",
+ "iter_normed_feature_wise",
+ "save_aggregated_ordering",
+ "mean_features",
+ "score_features",
+ "__repr__",
+ ),
+ RNAassay: (
+ "__init__",
+ "iter_normed_feature_wise",
+ "save_normalized_data",
+ "normed",
+ "iter_raw_column_blocks",
+ "iter_raw_feature_columns",
+ "set_feature_stats",
+ "get_feature_stats",
+ "set_summary_stats",
+ "set_hvgs",
+ "mark_hvgs",
+ ),
+ ATACassay: (
+ "__init__",
+ "normed",
+ "set_feature_stats",
+ "mark_prevalent_peaks",
+ ),
+ ADTassay: (
+ "__init__",
+ "normed",
+ ),
+}
+_PUBLIC_CLASS_SIGNATURE_DIGESTS = {
+ Assay: "98c61afcfba07dee60b2132c227882886ccb248832048d385c1f82d43f3617c7",
+ RNAassay: "26951c69d48c7cb169d8d59014850ff1cf47cdb7dc4cff63e3daf8851f74c604",
+ ATACassay: "491bb1c63ad83fa5d9634200c5b3778a3018e39abdd3ae87208eb3e85659633c",
+ ADTassay: "a1ff1bebdd8fcd3f30a1b64a42dbf2931f6b8031ddce75abcf362750eb4e9c34",
+}
+_MODULE_FUNCTIONS = (
+ "lib_size_feature_stream_eligible",
+ "norm_clr",
+ "norm_dummy",
+ "norm_lib_size",
+ "norm_lib_size_log",
+ "norm_tf_idf",
+)
+_MODULE_SIGNATURE_DIGEST = (
+ "3e073d20c4cb115fc4e8aed99fa62687509321692419598072535a3ec67f9223"
+)
+
+
+def test_assay_facade_surface_is_stable():
+ assert assay_module.__all__ == ["Assay", "RNAassay", "ATACassay", "ADTassay"]
+ expected = {
+ "ADTassay",
+ "ATACassay",
+ "Assay",
+ "NormMethod",
+ "PercentFeatures",
+ "RNAassay",
+ "_read_block",
+ "lib_size_feature_stream_eligible",
+ "norm_clr",
+ "norm_dummy",
+ "norm_lib_size",
+ "norm_lib_size_log",
+ "norm_tf_idf",
+ }
+ assert expected.issubset(vars(assay_module))
+
+
+def test_assay_class_and_method_signatures_are_stable():
+ for cls, names in _PUBLIC_CLASS_METHODS.items():
+ methods = {name: getattr(cls, name) for name in names}
+ assert signature_digest(methods) == _PUBLIC_CLASS_SIGNATURE_DIGESTS[cls]
+
+
+def test_assay_module_function_signatures_are_stable():
+ methods = {name: getattr(assay_module, name) for name in _MODULE_FUNCTIONS}
+ assert signature_digest(methods) == _MODULE_SIGNATURE_DIGEST
+
+
+def test_assay_normalization_type_hints_resolve_from_facade():
+ for name in (
+ "lib_size_feature_stream_eligible",
+ "norm_clr",
+ "norm_dummy",
+ "norm_lib_size",
+ "norm_lib_size_log",
+ "norm_tf_idf",
+ ):
+ assert get_type_hints(getattr(assay_module, name))
+
+
+def test_assay_public_metadata_remains_on_facade():
+ for cls, names in _PUBLIC_CLASS_METHODS.items():
+ assert cls.__module__ == "scarf.assay"
+ for name in names:
+ descriptor = inspect.getattr_static(cls, name)
+ if isinstance(descriptor, staticmethod):
+ method = descriptor.__func__
+ else:
+ method = descriptor
+ assert method.__module__ == "scarf.assay"
+ assert method.__qualname__.startswith(f"{cls.__name__}.")
+
+ for name in _MODULE_FUNCTIONS:
+ assert getattr(assay_module, name).__module__ == "scarf.assay"
+
+
+def test_assay_subclass_and_static_method_contracts_are_stable():
+ assert issubclass(RNAassay, Assay)
+ assert issubclass(ATACassay, Assay)
+ assert issubclass(ADTassay, Assay)
+ for name in (
+ "_create_subset_hash",
+ "_finalize_staged_mirror",
+ "_get_summary_stats_loc",
+ ):
+ assert isinstance(inspect.getattr_static(Assay, name), staticmethod)
+
+
+def test_default_normalizer_identity_is_stable(monkeypatch):
+ def initialize_base(self, *args, **kwargs):
+ self.attrs = {}
+
+ monkeypatch.setattr(Assay, "__init__", initialize_base)
+ rna = RNAassay(None, "RNA", None)
+ atac = ATACassay(None, "ATAC", None)
+ adt = ADTassay(None, "ADT", None)
+
+ assert rna.normMethod is norm_lib_size
+ assert lib_size_feature_stream_eligible(rna)
+ rna.normMethod = norm_dummy
+ assert not lib_size_feature_stream_eligible(rna)
+ assert atac.normMethod is norm_tf_idf
+ assert adt.normMethod is norm_clr
+
+
+def test_normalization_numerical_contracts_are_stable():
+ counts = np.array([[1.0, 3.0], [2.0, 4.0]])
+
+ clr_expected_scale = np.exp(np.log1p(counts).sum(axis=0) / len(counts))
+ np.testing.assert_allclose(
+ norm_clr(None, counts),
+ np.log1p(counts / clr_expected_scale),
+ )
+
+ atac = SimpleNamespace(
+ n_term_per_doc=np.array([4.0, 6.0]),
+ n_docs=2,
+ n_docs_per_term=np.array([2.0, 1.0]),
+ )
+ tf = counts / atac.n_term_per_doc.reshape(-1, 1)
+ idf = np.log2(1 + atac.n_docs / (atac.n_docs_per_term + 1))
+ np.testing.assert_allclose(norm_tf_idf(atac, counts), tf * idf)
+
+ rna = SimpleNamespace(
+ sf=1000.0,
+ scalar=np.array([4.0, 6.0]),
+ )
+ np.testing.assert_allclose(
+ norm_lib_size(rna, counts),
+ 1000.0 * counts / rna.scalar.reshape(-1, 1),
+ )
+
+
+def test_assay_read_block_facade_remains_patchable(monkeypatch):
+ from scarf.storage.budget import ResourceBudget
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ counts_t = root.create_array(
+ "countsT",
+ data=np.arange(12, dtype=np.uint32).reshape(4, 3),
+ )
+ rna = RNAassay.__new__(RNAassay)
+ rna.name = "RNA"
+ rna.z = root
+ rna.rawDataT = counts_t
+ rna.resources = ResourceBudget(memoryBytes=1024**2, workers=1)
+
+ original = assay_module._read_block
+ calls = []
+
+ def counted_read(array, rows, columns):
+ calls.append((rows.copy(), columns.copy()))
+ return original(array, rows, columns)
+
+ monkeypatch.setattr(assay_module, "_read_block", counted_read)
+ blocks = list(
+ rna.iter_raw_column_blocks(
+ cell_idx=np.array([0, 2]),
+ feat_idx=np.array([1, 3]),
+ batch_size=2,
+ )
+ )
+
+ assert len(calls) == 1
+ expected = np.asarray(counts_t[:])[[1, 3], :][:, [0, 2]].T
+ np.testing.assert_array_equal(blocks[0][1], expected)
+
+
+@pytest.mark.parametrize(
+ ("include_adaptive", "expected_column", "expected_variance"),
+ [
+ (False, "c_var__200__0.1", np.array([40.0, 20.0])),
+ (True, "c_var__adaptive__200__0.1", np.array([400.0, 200.0])),
+ ],
+)
+def test_get_feature_stats_reads_cached_columns_in_feature_key_order(
+ include_adaptive,
+ expected_column,
+ expected_variance,
+):
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ stats = root.create_group("summary_stats_I")
+ stats.create_array("nz_mean", data=np.array([1.0, 2.0, 3.0, 4.0]))
+ stats.create_array(
+ "c_var__200__0.1",
+ data=np.array([10.0, 20.0, 30.0, 40.0]),
+ )
+ if include_adaptive:
+ stats.create_array(
+ "c_var__adaptive__200__0.1",
+ data=np.array([100.0, 200.0, 300.0, 400.0]),
+ )
+ stats.create_array("normed_n", data=np.array([5.0, 6.0, 7.0, 8.0]))
+
+ rna = RNAassay.__new__(RNAassay)
+ rna.z = root
+ rna.feats = SimpleNamespace(
+ active_index=lambda key: np.array([3, 1]) if key == "selected" else None
+ )
+ rna._get_cell_feat_idx = lambda cell_key, feat_key: (
+ np.array([0, 2]),
+ np.arange(4),
+ )
+ validation_calls = []
+
+ def validate(stats_loc, cell_idx, feat_idx, delete_on_fail=True):
+ validation_calls.append((stats_loc, cell_idx, feat_idx, delete_on_fail))
+ return True
+
+ rna._validate_stats_loc = validate
+ values = rna.get_feature_stats("I", feat_key="selected")
+
+ assert list(values) == ["nz_mean", expected_column, "normed_n"]
+ np.testing.assert_array_equal(values["nz_mean"], np.array([4.0, 2.0]))
+ np.testing.assert_array_equal(values[expected_column], expected_variance)
+ np.testing.assert_array_equal(values["normed_n"], np.array([8.0, 6.0]))
+ assert validation_calls[0][3] is False
+
+
+def test_get_feature_stats_does_not_recompute_or_delete_invalid_cache():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ root.create_group("summary_stats_subset")
+
+ rna = RNAassay.__new__(RNAassay)
+ rna.z = root
+ rna.feats = SimpleNamespace(active_index=lambda key: np.arange(2))
+ rna._get_cell_feat_idx = lambda cell_key, feat_key: (
+ np.array([0]),
+ np.arange(2),
+ )
+ rna._validate_stats_loc = (
+ lambda stats_loc, cell_idx, feat_idx, delete_on_fail=True: False
+ )
+ rna.set_feature_stats = lambda cell_key: pytest.fail(
+ "get_feature_stats must not compute statistics"
+ )
+
+ with pytest.raises(KeyError, match="have not been calculated"):
+ rna.get_feature_stats("subset", columns=["nz_mean"])
+
+ assert "summary_stats_subset" in root
diff --git a/tests/test_assay_state.py b/tests/test_assay_state.py
new file mode 100644
index 00000000..24612982
--- /dev/null
+++ b/tests/test_assay_state.py
@@ -0,0 +1,972 @@
+import hashlib
+from dataclasses import replace
+from typing import Any
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.datastore.datastore import DataStore
+from scarf.datastore.graph_datastore import GraphDataStore
+from scarf.graph.encoded_paths import (
+ make_cell_graph_group_path,
+ make_kmeans_initialization_group_path,
+ make_nearest_neighbors_group_path,
+ make_neighbor_index_group_path,
+ make_normalized_group_path,
+ make_reduction_group_path,
+)
+from scarf.graph.paths import AssayGraphPaths, StoredAssayGraph, StoredIntegratedGraph
+from scarf.graph.state import (
+ ArtifactSelectionError,
+ AssayState,
+ _legacy_subset_hash,
+ read_assay_state,
+ stored_assay_graph_from_state,
+ validate_legacy_graph_selection,
+)
+from scarf.storage.artifacts import (
+ ArtifactRef,
+ ArtifactScope,
+ artifact_path,
+ fingerprint_strings,
+ inspect_artifact,
+ list_artifacts,
+ make_provenance,
+ new_artifact_id,
+)
+
+
+class _StateGraphStore(GraphDataStore):
+ @property
+ def assay_names(self) -> list[str]:
+ return ["RNA"]
+
+
+def _ref(kind: str, token: str) -> ArtifactRef:
+ return ArtifactRef(
+ scope="assay",
+ assay="RNA",
+ kind=kind,
+ artifact_id=token * 64,
+ )
+
+
+def _compose_assay_graph_paths(
+ *,
+ from_assay: str,
+ cell_key: str,
+ feat_key: str,
+ reduction_method: str,
+ dims: int,
+ pca_cell_key: str,
+ ann_metric: str,
+ ann_efc: int,
+ ann_ef: int,
+ ann_m: int,
+ rand_state: int,
+ k: int,
+ local_connectivity: float,
+ bandwidth: float,
+ n_centroids: int | None = None,
+ feat_scaling: bool = True,
+ harmony_contract_hash: str | None = None,
+) -> AssayGraphPaths:
+ normalized = make_normalized_group_path(from_assay, cell_key, feat_key)
+ reduction = make_reduction_group_path(
+ normalized, reduction_method, dims, pca_cell_key
+ )
+ neighbor_index = make_neighbor_index_group_path(
+ reduction,
+ ann_metric,
+ ann_efc,
+ ann_ef,
+ ann_m,
+ rand_state,
+ feat_scaling=feat_scaling,
+ harmony_contract_hash=harmony_contract_hash,
+ )
+ nearest_neighbors = make_nearest_neighbors_group_path(neighbor_index, k)
+ cell_graph = make_cell_graph_group_path(
+ nearest_neighbors, local_connectivity, bandwidth
+ )
+ kmeans = None
+ if n_centroids is not None:
+ kmeans = make_kmeans_initialization_group_path(
+ reduction, n_centroids, rand_state
+ )
+ return AssayGraphPaths(
+ normalized_group_path=normalized,
+ reduction_group_path=reduction,
+ neighbor_index_group_path=neighbor_index,
+ nearest_neighbors_group_path=nearest_neighbors,
+ cell_graph_group_path=cell_graph,
+ kmeans_initialization_group_path=kmeans,
+ )
+
+
+def test_legacy_graph_without_selection_provenance_fails_closed(
+ datastore_ephemeral,
+) -> None:
+ paths = _compose_assay_graph_paths(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I",
+ reduction_method="pca",
+ dims=5,
+ pca_cell_key="I",
+ ann_metric="l2",
+ ann_efc=50,
+ ann_ef=50,
+ ann_m=16,
+ rand_state=1,
+ k=3,
+ local_connectivity=1.0,
+ bandwidth=1.0,
+ )
+ datastore_ephemeral.zw.require_group(paths.normalized_group_path)
+
+ with pytest.raises(ValueError, match="selection provenance is missing"):
+ validate_legacy_graph_selection(
+ datastore_ephemeral,
+ paths.nearest_neighbors_group_path,
+ "RNA",
+ "I",
+ "I",
+ )
+
+
+@pytest.mark.parametrize("hash_format", ["current", "legacy"])
+def test_legacy_graph_accepts_supported_selection_hashes(
+ datastore_ephemeral,
+ hash_format: str,
+) -> None:
+ paths = _compose_assay_graph_paths(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I",
+ reduction_method="pca",
+ dims=5,
+ pca_cell_key="I",
+ ann_metric="l2",
+ ann_efc=50,
+ ann_ef=50,
+ ann_m=16,
+ rand_state=1,
+ k=3,
+ local_connectivity=1.0,
+ bandwidth=1.0,
+ )
+ normalized = datastore_ephemeral.zw.require_group(paths.normalized_group_path)
+ assay = datastore_ephemeral._get_assay("RNA")
+ cell_indices = datastore_ephemeral.cells.active_index("I")
+ feature_indices = assay.feats.active_index("I")
+ normalized.attrs["subset_hash"] = (
+ assay._create_subset_hash(cell_indices, feature_indices)
+ if hash_format == "current"
+ else _legacy_subset_hash(cell_indices, feature_indices)
+ )
+
+ validate_legacy_graph_selection(
+ datastore_ephemeral,
+ paths.nearest_neighbors_group_path,
+ "RNA",
+ "I",
+ "I",
+ )
+
+
+def test_legacy_graph_rejects_mismatched_or_invalid_selection_hash(
+ datastore_ephemeral,
+) -> None:
+ paths = _compose_assay_graph_paths(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I",
+ reduction_method="pca",
+ dims=5,
+ pca_cell_key="I",
+ ann_metric="l2",
+ ann_efc=50,
+ ann_ef=50,
+ ann_m=16,
+ rand_state=1,
+ k=3,
+ local_connectivity=1.0,
+ bandwidth=1.0,
+ )
+ normalized = datastore_ephemeral.zw.require_group(paths.normalized_group_path)
+ normalized.attrs["subset_hash"] = 0
+ with pytest.raises(ValueError, match="no longer matches"):
+ validate_legacy_graph_selection(
+ datastore_ephemeral,
+ paths.nearest_neighbors_group_path,
+ "RNA",
+ "I",
+ "I",
+ )
+
+ normalized.attrs["subset_hash"] = True
+ with pytest.raises(ValueError, match="invalid type"):
+ validate_legacy_graph_selection(
+ datastore_ephemeral,
+ paths.nearest_neighbors_group_path,
+ "RNA",
+ "I",
+ "I",
+ )
+
+
+def test_legacy_graph_qualifies_non_default_feature_key_exactly_once(
+ datastore_ephemeral,
+) -> None:
+ assay = datastore_ephemeral._get_assay("RNA")
+ qualified_key = "I__I__qualified"
+ assay.feats.insert(
+ qualified_key,
+ np.asarray(assay.feats.fetch_all("I"), dtype=bool),
+ overwrite=True,
+ )
+ paths = _compose_assay_graph_paths(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I__qualified",
+ reduction_method="pca",
+ dims=5,
+ pca_cell_key="I",
+ ann_metric="l2",
+ ann_efc=50,
+ ann_ef=50,
+ ann_m=16,
+ rand_state=1,
+ k=3,
+ local_connectivity=1.0,
+ bandwidth=1.0,
+ )
+ normalized = datastore_ephemeral.zw.require_group(paths.normalized_group_path)
+ normalized.attrs["subset_hash"] = _legacy_subset_hash(
+ datastore_ephemeral.cells.active_index("I"),
+ assay.feats.active_index(qualified_key),
+ )
+
+ validate_legacy_graph_selection(
+ datastore_ephemeral,
+ paths.nearest_neighbors_group_path,
+ "RNA",
+ "I",
+ "I__qualified",
+ )
+
+
+def _add_artifact(
+ root: zarr.Group,
+ *,
+ kind: str,
+ operation: str,
+ parameters: dict[str, Any] | None = None,
+ inputs: dict[str, Any] | None = None,
+ execution_options: dict[str, Any] | None = None,
+ scope: ArtifactScope = "assay",
+) -> tuple[ArtifactRef, zarr.Group]:
+ parameters = parameters or {}
+ inputs = inputs or {}
+ execution_options = execution_options or {}
+ ref = ArtifactRef(
+ scope=scope,
+ assay="RNA" if scope == "assay" else None,
+ kind=kind,
+ artifact_id=new_artifact_id(),
+ )
+ provenance = make_provenance(
+ operation=operation,
+ parameters=parameters,
+ inputs=inputs,
+ )
+ group = root.create_group(artifact_path(ref))
+ group.attrs.update(
+ {
+ "artifact_id": ref.artifact_id,
+ "kind": ref.kind,
+ "provenance": provenance,
+ "execution_options": execution_options,
+ "complete": True,
+ }
+ )
+ return ref, group
+
+
+def _state_store() -> tuple[_StateGraphStore, AssayState]:
+ datastore = _StateGraphStore.__new__(_StateGraphStore)
+ datastore.z = zarr.open_group(store=MemoryStore(), mode="w")
+ datastore.workspace = None
+ datastore.zarr_mode = "r+"
+ datastore._defaultAssay = "RNA"
+ datastore._integratedGraphsLoc = "integratedGraphs"
+ datastore._cachedMagicOperator = None
+ datastore._cachedMagicOperatorLoc = None
+ datastore.nthreads = 1
+ cell_ids = np.asarray(["c0", "c1", "c2"])
+ feature_ids = np.asarray(["f0", "f1", "f2", "f3"])
+ cell_data = datastore.z.create_group("cellData")
+ cell_data.create_array("ids", data=cell_ids)
+ cell_data.create_array("I", data=np.ones(3, dtype=bool))
+ feature_data = datastore.z.create_group("RNA/featureData")
+ feature_data.create_array("ids", data=feature_ids)
+ feature_data.create_array("I__hvgs", data=np.ones(4, dtype=bool))
+
+ cell_selection, cell_selection_group = _add_artifact(
+ datastore.z,
+ kind="cell_selection",
+ operation="manual_selection",
+ inputs={
+ "ordered_row_ids_fingerprint": fingerprint_strings(cell_ids),
+ "values_fingerprint": "cells",
+ },
+ execution_options={"source_column": "I"},
+ scope="datastore",
+ )
+ cell_selection_group.create_array(
+ "values",
+ data=np.ones(3, dtype=bool),
+ )
+ feature_selection, feature_selection_group = _add_artifact(
+ datastore.z,
+ kind="feature_selection",
+ operation="manual_selection",
+ inputs={
+ "ordered_row_ids_fingerprint": fingerprint_strings(feature_ids),
+ "values_fingerprint": "features",
+ },
+ execution_options={"source_column": "I__hvgs"},
+ )
+ feature_selection_group.create_array(
+ "values",
+ data=np.ones(4, dtype=bool),
+ )
+ normalized, normalized_group = _add_artifact(
+ datastore.z,
+ kind="normalized",
+ operation="run_normalization",
+ inputs={
+ "cell_selection": cell_selection,
+ "feature_selection": feature_selection,
+ },
+ execution_options={"cell_key": "I", "feat_key": "hvgs"},
+ )
+ normalized_group.create_array(
+ "data",
+ data=np.arange(12, dtype=np.float64).reshape(3, 4),
+ )
+ scaling, scaling_group = _add_artifact(
+ datastore.z,
+ kind="feature_scaling",
+ operation="calculate_feature_scaling",
+ inputs={"normalized": normalized},
+ )
+ scaling_group.create_array("mean", data=np.arange(4, dtype=np.float64))
+ scaling_group.create_array("scale", data=np.ones(4, dtype=np.float64))
+ reduction, reduction_group = _add_artifact(
+ datastore.z,
+ kind="reduction",
+ operation="run_reduction",
+ parameters={
+ "reduction_method": "pca",
+ "dims": 2,
+ "pca_cell_key": "I",
+ "feat_scaling": True,
+ "lsi_skip_first": True,
+ },
+ inputs={
+ "normalized": normalized,
+ "feature_scaling": scaling,
+ },
+ )
+ reduction_group.create_array(
+ "loadings",
+ data=np.arange(8, dtype=np.float64).reshape(4, 2),
+ )
+ correction, correction_group = _add_artifact(
+ datastore.z,
+ kind="batch_correction",
+ operation="run_harmony",
+ inputs={"reduction": reduction},
+ )
+ correction_group.create_array(
+ "data",
+ data=np.arange(6, dtype=np.float64).reshape(3, 2),
+ )
+ ann_index, _ann_index_group = _add_artifact(
+ datastore.z,
+ kind="ann_index",
+ operation="build_ann_index",
+ parameters={
+ "ann_metric": "l2",
+ "ann_efc": 50,
+ "ann_ef": 50,
+ "ann_m": 16,
+ "rand_state": 4466,
+ },
+ inputs={"coordinates": correction},
+ )
+ initialization, initialization_group = _add_artifact(
+ datastore.z,
+ kind="embedding_initialization",
+ operation="build_embedding_initialization",
+ parameters={"n_centroids": 3, "rand_state": 4466},
+ inputs={"reduction": reduction},
+ )
+ initialization_group.create_array(
+ "cluster_centers",
+ data=np.array([[0.0, 0.0], [1.0, 0.0], [0.0, 1.0]]),
+ )
+ initialization_group.create_array(
+ "cluster_labels",
+ data=np.array([0, 1, 2], dtype=np.int64),
+ )
+ neighbors, neighbors_group = _add_artifact(
+ datastore.z,
+ kind="neighbors",
+ operation="query_neighbors",
+ parameters={"k": 2},
+ inputs={
+ "ann_index": ann_index,
+ "coordinates": correction,
+ },
+ )
+ neighbors_group.create_array(
+ "indices",
+ data=np.array([[1, 2], [0, 2], [0, 1]], dtype=np.uint64),
+ )
+ neighbors_group.create_array(
+ "distances",
+ data=np.ones((3, 2), dtype=np.float64),
+ )
+ connectivity, connectivity_group = _add_artifact(
+ datastore.z,
+ kind="connectivity_map",
+ operation="build_connectivity_map",
+ parameters={"local_connectivity": 1.0, "bandwidth": 1.5},
+ inputs={"neighbors": neighbors},
+ )
+ connectivity_group.attrs["n_cells"] = 3
+ connectivity_group.attrs["n_neighbors"] = 2
+ connectivity_group.create_array(
+ "edges",
+ data=np.array(
+ [[0, 1], [0, 2], [1, 0], [1, 2], [2, 0], [2, 1]],
+ dtype=np.uint64,
+ ),
+ )
+ connectivity_group.create_array(
+ "weights",
+ data=np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6]),
+ )
+ embedding, _embedding_group = _add_artifact(
+ datastore.z,
+ kind="embedding",
+ operation="run_umap",
+ inputs={"connectivity_map": connectivity},
+ )
+
+ state = AssayState(
+ assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ normalized=normalized,
+ feature_scaling=scaling,
+ reduction=reduction,
+ batch_correction=correction,
+ ann_index=ann_index,
+ embedding_initialization=initialization,
+ neighbors=neighbors,
+ connectivity_map=connectivity,
+ named_results={"umap": embedding},
+ )
+ state_group = datastore.z.create_group("RNA/state")
+ state_group.attrs["state"] = state.to_dict()
+ legacy_normalized_path = "RNA/normed__I__legacy"
+ legacy_reduction_path = f"{legacy_normalized_path}/reduction__pca__5__I"
+ legacy_ann_path = f"{legacy_reduction_path}/ann__l2__50__50__16__4466"
+ legacy_neighbors_path = f"{legacy_ann_path}/knn__3"
+ legacy_graph_path = f"{legacy_neighbors_path}/graph__1.0__1.5"
+ legacy_normalized = datastore.z.create_group(legacy_normalized_path)
+ legacy_reduction = datastore.z.create_group(legacy_reduction_path)
+ legacy_ann = datastore.z.create_group(legacy_ann_path)
+ legacy_neighbors = datastore.z.create_group(legacy_neighbors_path)
+ legacy_graph = datastore.z.create_group(legacy_graph_path)
+ legacy_normalized.attrs["latest_reduction"] = legacy_reduction_path
+ legacy_reduction.attrs["latest_ann"] = legacy_ann_path
+ legacy_ann.attrs["latest_knn"] = legacy_neighbors_path
+ legacy_neighbors.attrs["latest_graph"] = legacy_graph_path
+ legacy_neighbors.create_array(
+ "indices",
+ data=np.array([[1, 2, 0], [0, 2, 1], [0, 1, 2]], dtype=np.uint64),
+ )
+ legacy_graph.create_array(
+ "edges",
+ data=np.array(
+ [[0, 1], [0, 2], [1, 0], [1, 2], [2, 0], [2, 1]],
+ dtype=np.uint64,
+ ),
+ )
+ legacy_graph.create_array(
+ "weights",
+ data=np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6]),
+ )
+ return datastore, state
+
+
+def _store_digest(root: zarr.Group) -> str:
+ digest = hashlib.blake2b(digest_size=32)
+
+ def visit(group: zarr.Group, prefix: str = "") -> None:
+ digest.update(prefix.encode())
+ digest.update(repr(dict(group.attrs)).encode())
+ for key in sorted(group.keys()):
+ node = group[key]
+ path = f"{prefix}/{key}" if prefix else key
+ digest.update(path.encode())
+ digest.update(repr(dict(node.attrs)).encode())
+ if isinstance(node, zarr.Array):
+ digest.update(str(node.dtype).encode())
+ digest.update(np.asarray(node[:]).tobytes())
+ else:
+ visit(node, path)
+
+ visit(root)
+ return digest.hexdigest()
+
+
+def test_assay_state_round_trip_and_validation() -> None:
+ datastore, state = _state_store()
+ assert AssayState.from_dict(state.to_dict()) == state
+ public_datastore = DataStore.__new__(DataStore)
+ public_datastore.z = datastore.z
+ public_datastore.workspace = None
+ public_datastore._defaultAssay = "RNA"
+ assert public_datastore.get_assay_state() == state
+
+ with pytest.raises(ValueError, match="requires kind"):
+ AssayState(
+ assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ normalized=_ref("neighbors", "a"),
+ )
+ with pytest.raises(ValueError, match="snake_case"):
+ AssayState(
+ assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ named_results={"UMAP 1": _ref("embedding", "a")},
+ )
+ malformed = state.to_dict()
+ malformed["normalized"] = "not-a-reference"
+ with pytest.raises(TypeError, match="normalized must be"):
+ AssayState.from_dict(malformed)
+ malformed = state.to_dict()
+ malformed["cell_key"] = None
+ with pytest.raises(TypeError, match="must be strings"):
+ AssayState.from_dict(malformed)
+
+
+def test_state_first_graph_lookup_uses_artifacts() -> None:
+ datastore, state = _state_store()
+
+ assert read_assay_state(datastore.zw, "RNA") == state
+ assert datastore.get_normalized_group_path("RNA", "I", "hvgs") == artifact_path(
+ state.normalized
+ )
+ assert datastore.get_latest_graph_loc("RNA", "I", "hvgs") == artifact_path(
+ state.connectivity_map
+ )
+ assert datastore.load_graph().shape == (3, 3)
+
+ stored = datastore._lookup_stored_graph("RNA", "I", "hvgs")
+ assert isinstance(stored, StoredAssayGraph)
+ assert stored.dims == 2
+ assert stored.k == 2
+ assert stored.ann_metric == "l2"
+ assert stored.local_connectivity == 1.0
+ assert stored.paths.kmeans_initialization_group_path == artifact_path(
+ state.embedding_initialization
+ )
+
+
+def test_state_graph_summary_rejects_missing_required_provenance() -> None:
+ datastore, state = _state_store()
+ assert state.ann_index is not None
+ ann_group = datastore.zw[artifact_path(state.ann_index)]
+ provenance = dict(ann_group.attrs["provenance"])
+ parameters = dict(provenance["parameters"])
+ del parameters["ann_efc"]
+ provenance["parameters"] = parameters
+ ann_group.attrs["provenance"] = provenance
+
+ with pytest.raises(ValueError, match="ann_index provenance is missing 'ann_efc'"):
+ datastore._lookup_stored_graph("RNA", "I", "hvgs")
+
+
+def test_state_graph_summary_rejects_invalid_optional_provenance_type() -> None:
+ datastore, state = _state_store()
+ assert state.reduction is not None
+ reduction_group = datastore.zw[artifact_path(state.reduction)]
+ provenance = dict(reduction_group.attrs["provenance"])
+ parameters = dict(provenance["parameters"])
+ parameters["feat_scaling"] = "false"
+ provenance["parameters"] = parameters
+ reduction_group.attrs["provenance"] = provenance
+
+ with pytest.raises(TypeError, match="must be boolean"):
+ datastore._lookup_stored_graph("RNA", "I", "hvgs")
+
+
+def test_explicit_artifact_graph_path_loads_without_latest_attrs() -> None:
+ datastore, state = _state_store()
+ graph_path = artifact_path(state.connectivity_map)
+
+ graph = datastore.load_graph(graph_loc=graph_path)
+ stored = datastore._lookup_stored_graph(graph_loc=graph_path)
+
+ assert graph.shape == (3, 3)
+ assert graph.nnz == 6
+ assert isinstance(stored, StoredAssayGraph)
+ assert stored.paths.cell_graph_group_path == graph_path
+
+ older_ref, older = _add_artifact(
+ datastore.z,
+ kind="connectivity_map",
+ operation="build_connectivity_map",
+ parameters={"local_connectivity": 0.5, "bandwidth": 1.0},
+ inputs={"neighbors": state.neighbors},
+ )
+ older.attrs["n_cells"] = 3
+ older.attrs["n_neighbors"] = 2
+ selected = datastore.z[graph_path]
+ older.create_array("edges", data=np.asarray(selected["edges"][:]))
+ older.create_array("weights", data=np.asarray(selected["weights"][:]) * 0.5)
+
+ explicit_older = datastore.load_graph(graph_loc=artifact_path(older_ref))
+ stored_older = datastore._lookup_stored_graph(graph_loc=artifact_path(older_ref))
+ assert explicit_older.shape == (3, 3)
+ assert explicit_older.nnz == 6
+ assert isinstance(stored_older, StoredAssayGraph)
+ assert stored_older.local_connectivity == 0.5
+
+
+def test_explicit_integrated_artifact_loads_without_legacy_slot() -> None:
+ datastore, state = _state_store()
+ assert state.normalized is not None
+ normalized_inputs = inspect_artifact(datastore.zw, state.normalized).inputs or {}
+ raw_selection = normalized_inputs.get("cell_selection")
+ assert isinstance(raw_selection, dict)
+ cell_selection = ArtifactRef.from_dict(raw_selection)
+ ref, group = _add_artifact(
+ datastore.z,
+ kind="integrated_graph",
+ operation="integrate_assays",
+ parameters={"method": "wnn"},
+ inputs={
+ "rna": state.connectivity_map,
+ "cell_selection": cell_selection,
+ },
+ scope="datastore",
+ )
+ group.attrs["n_cells"] = 3
+ group.attrs["n_neighbors"] = 2
+ group.create_array(
+ "edges",
+ data=np.array(
+ [[0, 1], [0, 2], [1, 0], [1, 2], [2, 0], [2, 1]],
+ dtype=np.uint64,
+ ),
+ )
+ group.create_array(
+ "weights",
+ data=np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6]),
+ )
+
+ graph = datastore.load_graph(graph_loc=artifact_path(ref))
+ stored = datastore._lookup_stored_graph(graph_loc=artifact_path(ref))
+
+ assert graph.shape == (3, 3)
+ assert graph.nnz == 6
+ assert isinstance(stored, StoredIntegratedGraph)
+ assert stored.cell_graph_group_path == artifact_path(ref)
+ assert stored.n_cells == 3
+ assert stored.n_neighbors == 2
+
+
+@pytest.mark.parametrize(
+ ("failure", "expected_code"),
+ [
+ ("missing", "artifact_missing"),
+ ("incomplete", "artifact_incomplete"),
+ ],
+)
+def test_integrated_graph_selection_failures_are_structured(
+ failure: str,
+ expected_code: str,
+) -> None:
+ datastore, state = _state_store()
+ assert state.normalized is not None
+ normalized_inputs = inspect_artifact(datastore.zw, state.normalized).inputs or {}
+ raw_selection = normalized_inputs.get("cell_selection")
+ assert isinstance(raw_selection, dict)
+ selection = ArtifactRef.from_dict(raw_selection)
+ graph_ref, _group = _add_artifact(
+ datastore.z,
+ kind="integrated_graph",
+ operation="integrate_assays",
+ inputs={"cell_selection": selection},
+ scope="datastore",
+ )
+ if failure == "missing":
+ del datastore.z[artifact_path(selection)]
+ else:
+ datastore.z[artifact_path(selection)].attrs["complete"] = False
+ before = _store_digest(datastore.z)
+
+ with pytest.raises(ArtifactSelectionError) as caught:
+ datastore.load_graph(graph_loc=artifact_path(graph_ref))
+
+ assert caught.value.code == expected_code
+ assert caught.value.context["artifact_id"] == selection.artifact_id
+ assert _store_digest(datastore.z) == before
+
+
+def test_artifact_embedding_initialization_is_state_resolved() -> None:
+ datastore, _state = _state_store()
+ embedding = datastore._get_ini_embed("RNA", "I", "hvgs", 2)
+
+ assert embedding.shape == (3, 2)
+ assert np.isfinite(embedding).all()
+
+
+def test_state_mismatch_keeps_legacy_normalized_path_behavior(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ datastore, _state = _state_store()
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.validate_legacy_graph_selection",
+ lambda *_args, **_kwargs: None,
+ )
+ assert datastore.get_normalized_group_path("RNA", "I", "legacy") == (
+ "RNA/normed__I__legacy"
+ )
+ stored = datastore._lookup_stored_graph("RNA", "I", "legacy")
+ assert isinstance(stored, StoredAssayGraph)
+ assert stored.paths.cell_graph_group_path.endswith("graph__1.0__1.5")
+ assert stored.dims == 5
+ assert stored.k == 3
+
+
+def test_state_reads_do_not_mutate_artifacts_or_state() -> None:
+ datastore, _state = _state_store()
+ before = _store_digest(datastore.z)
+
+ datastore.get_normalized_group_path("RNA", "I", "hvgs")
+ datastore.get_latest_graph_loc("RNA", "I", "hvgs")
+ datastore.load_graph(graph_loc=datastore.get_latest_graph_loc("RNA", "I", "hvgs"))
+ stored_assay_graph_from_state(datastore.zw, "RNA", "I", "hvgs")
+
+ assert _store_digest(datastore.z) == before
+
+
+def test_incomplete_selected_artifact_is_not_silently_used() -> None:
+ datastore, state = _state_store()
+ datastore.z[artifact_path(state.normalized)].attrs["complete"] = False
+
+ with pytest.raises(RuntimeError, match="incomplete"):
+ datastore.get_normalized_group_path("RNA", "I", "hvgs")
+
+ datastore.z[artifact_path(state.normalized)].attrs["complete"] = True
+ datastore.z[artifact_path(state.connectivity_map)].attrs["complete"] = False
+ with pytest.raises(RuntimeError, match="Graph artifact is incomplete"):
+ datastore.load_graph(graph_loc=artifact_path(state.connectivity_map))
+
+
+@pytest.mark.parametrize(
+ ("failure", "expected_code"),
+ [
+ ("missing", "artifact_missing"),
+ ("incomplete", "artifact_incomplete"),
+ ],
+)
+def test_selection_artifact_failures_are_structured_through_state_lookup(
+ failure: str,
+ expected_code: str,
+) -> None:
+ datastore, state = _state_store()
+ assert state.normalized is not None
+ inputs = inspect_artifact(datastore.zw, state.normalized).inputs or {}
+ raw_selection = inputs.get("cell_selection")
+ assert isinstance(raw_selection, dict)
+ selection = ArtifactRef.from_dict(raw_selection)
+ if failure == "missing":
+ del datastore.z[artifact_path(selection)]
+ else:
+ datastore.z[artifact_path(selection)].attrs["complete"] = False
+ before = _store_digest(datastore.z)
+
+ with pytest.raises(ArtifactSelectionError) as caught:
+ datastore.load_graph(from_assay="RNA", cell_key="I", feat_key="hvgs")
+
+ assert caught.value.code == expected_code
+ assert caught.value.context["artifact_id"] == selection.artifact_id
+ assert _store_digest(datastore.z) == before
+
+
+def test_matching_state_does_not_fall_back_to_stale_legacy_paths() -> None:
+ datastore, state = _state_store()
+ datastore.z.create_group("RNA/normed__I__hvgs")
+ state_without_normalized = replace(state, normalized=None)
+ datastore.z["RNA/state"].attrs["state"] = state_without_normalized.to_dict()
+
+ with pytest.raises(KeyError, match="no selected normalized"):
+ datastore.get_normalized_group_path("RNA", "I", "hvgs")
+
+
+def test_state_rejects_unrelated_complete_artifact_chains() -> None:
+ datastore, state = _state_store()
+ wrong_neighbors, _group = _add_artifact(
+ datastore.z,
+ kind="neighbors",
+ operation="query_neighbors",
+ parameters={"k": 3},
+ inputs={
+ "ann_index": state.ann_index,
+ "coordinates": state.batch_correction,
+ },
+ )
+ mismatched = replace(state, neighbors=wrong_neighbors)
+ datastore.z["RNA/state"].attrs["state"] = mismatched.to_dict()
+
+ with pytest.raises(ValueError, match="does not match AssayState"):
+ datastore.get_latest_graph_loc("RNA", "I", "hvgs")
+
+
+def test_state_rejects_incomplete_or_missing_graph_inputs() -> None:
+ datastore, state = _state_store()
+ datastore.z[artifact_path(state.feature_scaling)].attrs["complete"] = False
+ with pytest.raises(RuntimeError, match="Artifact is incomplete"):
+ datastore.get_latest_graph_loc("RNA", "I", "hvgs")
+
+ datastore.z[artifact_path(state.batch_correction)].attrs["complete"] = True
+ without_scaling = replace(state, feature_scaling=None)
+ datastore.z["RNA/state"].attrs["state"] = without_scaling.to_dict()
+ with pytest.raises(KeyError, match="feature_scaling"):
+ datastore.get_latest_graph_loc("RNA", "I", "hvgs")
+
+ datastore.z["RNA/state"].attrs["state"] = state.to_dict()
+ datastore.z[artifact_path(state.feature_scaling)].attrs["complete"] = True
+ datastore.z[artifact_path(state.batch_correction)].attrs["complete"] = False
+ with pytest.raises(RuntimeError, match="Artifact is incomplete"):
+ datastore.get_latest_graph_loc("RNA", "I", "hvgs")
+
+
+def test_mapping_reference_and_graph_do_not_clear_each_other(
+ analyzed_datastore_ephemeral,
+) -> None:
+ datastore = analyzed_datastore_ephemeral
+ start = datastore.get_assay_state("RNA")
+ assert start is not None
+ neighbors = start.neighbors
+ graph = start.connectivity_map
+ assert neighbors is not None and graph is not None
+
+ reference = datastore.build_mapping_reference(neighbors)
+ after_reference = datastore.get_assay_state("RNA")
+ assert after_reference is not None
+ assert after_reference.named_results["mapping_reference"] == reference.ref
+ # Publishing a mapping reference walks up to neighbors, so the graph is not
+ # in that lineage and used to be dropped.
+ assert after_reference.connectivity_map == graph
+ assert after_reference.embedding_initialization is not None
+
+ rebuilt = datastore.build_connectivity_map(neighbors, bandwidth=1.25)
+ assert rebuilt != graph
+ after_graph = datastore.get_assay_state("RNA")
+ assert after_graph is not None
+ assert after_graph.connectivity_map == rebuilt
+ # The named handle is not derived from the graph, so rebuilding the graph
+ # must leave it selectable.
+ assert after_graph.named_results["mapping_reference"] == reference.ref
+ assert datastore.get_mapping_reference().ref == reference.ref
+
+
+def test_publishing_recovers_from_an_incomplete_graph_artifact(
+ analyzed_datastore_ephemeral,
+) -> None:
+ datastore = analyzed_datastore_ephemeral
+ start = datastore.get_assay_state("RNA")
+ assert start is not None
+ assert start.neighbors is not None and start.connectivity_map is not None
+ datastore.z[artifact_path(start.connectivity_map)].attrs["complete"] = False
+
+ datastore.build_mapping_reference(start.neighbors)
+
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert "mapping_reference" in state.named_results
+ assert state.connectivity_map is None
+
+
+def test_named_result_is_dropped_when_the_chain_moves_underneath_it(
+ analyzed_datastore_ephemeral,
+) -> None:
+ datastore = analyzed_datastore_ephemeral
+ start = datastore.get_assay_state("RNA")
+ assert start is not None and start.neighbors is not None
+ datastore.build_mapping_reference(start.neighbors)
+
+ normalized = datastore.run_normalization(
+ cell_key="I",
+ feat_key="hvgs",
+ update_state=False,
+ )
+ reduction = datastore.run_pca(normalized, dims=5, update_state=False)
+ ann_index = datastore.build_ann_index(reduction, update_state=False)
+ moved = datastore.query_neighbors(ann_index, coordinates=reduction, k=3)
+
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.neighbors == moved
+ # Carrying the handle here would describe a chain that no longer exists, so
+ # it is dropped rather than preserved into an inconsistent state.
+ assert "mapping_reference" not in state.named_results
+ assert state.connectivity_map is None
+
+
+def test_derived_writers_do_not_mutate_graph_artifacts() -> None:
+ datastore, state = _state_store()
+ graph_path = artifact_path(state.connectivity_map)
+ before = _store_digest(datastore.z[graph_path])
+
+ datastore.get_diffusion_operator("RNA", "I", "hvgs", t=2)
+ diffusion_refs = list_artifacts(
+ datastore.z,
+ scope="assay",
+ assay="RNA",
+ kind="diffusion_operator",
+ )
+ assert len(diffusion_refs) == 1
+ # Artifact-backed Paris goes through _run_paris_from_artifacts. The stub
+ # store lacks MetaData, so label persistence may fail after hierarchy work;
+ # the connectivity artifact itself must stay unchanged either way.
+ try:
+ datastore._run_paris_from_artifacts(
+ graph_ref=state.connectivity_map,
+ graph_loc=artifact_path(state.connectivity_map),
+ from_assay="RNA",
+ label_assay="RNA",
+ cell_key="I",
+ fixed_cluster_count=2,
+ effective_min_cluster_size=None,
+ label="paris_cluster",
+ force_recalc=False,
+ )
+ except AttributeError:
+ pass
+ assert _store_digest(datastore.z[graph_path]) == before
diff --git a/tests/test_atac_tfidf.py b/tests/test_atac_tfidf.py
new file mode 100644
index 00000000..d1a41c8b
--- /dev/null
+++ b/tests/test_atac_tfidf.py
@@ -0,0 +1,681 @@
+from collections.abc import Callable
+
+import numpy as np
+import pytest
+from scipy.sparse import csr_matrix
+
+from scarf.assay import norm_tf_idf
+from scarf.datastore.datastore import DataStore
+from scarf.matrix import ChunkedArray
+from scarf.plotting.heatmaps import _prepare_marker_heatmap
+from scarf.utils import controlled_compute
+from scarf.writers import SparseToZarr
+
+
+def _build_store(store_path, counts):
+ writer = SparseToZarr(
+ csr_matrix(counts),
+ zarr_loc=str(store_path),
+ cell_ids=[f"cell_{i}" for i in range(counts.shape[0])],
+ feature_ids=[f"peak_{i}" for i in range(counts.shape[1])],
+ assay_name="ATAC",
+ nthreads=1,
+ )
+ writer.dump(batch_size=2)
+ return DataStore(
+ str(store_path),
+ default_assay="ATAC",
+ min_features_per_cell=0,
+ min_cells_per_feature=0,
+ nthreads=1,
+ )
+
+
+@pytest.fixture
+def atac_tfidf_store(tmp_path):
+ counts = np.array(
+ [
+ [2, 0, 1, 0],
+ [1, 3, 0, 0],
+ [0, 1, 4, 1],
+ [0, 2, 0, 5],
+ ],
+ dtype=np.uint32,
+ )
+ store = _build_store(tmp_path / "atac_tfidf.zarr", counts)
+ store.cells.insert(
+ "subset",
+ np.array([True, True, False, False]),
+ overwrite=True,
+ )
+ return store, counts
+
+
+def _reference_tfidf(
+ counts: np.ndarray,
+ cell_idx: np.ndarray,
+ feat_idx: np.ndarray,
+ *,
+ renormalize_subset: bool = False,
+) -> tuple[np.ndarray, np.ndarray]:
+ selected = counts[np.ix_(cell_idx, feat_idx)].astype(np.float64)
+ terms_per_document = (
+ selected.sum(axis=1, dtype=np.float64)
+ if renormalize_subset
+ else counts.sum(axis=1, dtype=np.float64)[cell_idx]
+ )
+ terms_per_document[terms_per_document == 0] = 1
+ document_frequency = np.count_nonzero(selected, axis=0)
+ term_frequency = selected / terms_per_document.reshape(-1, 1)
+ idf = np.log2(1 + len(cell_idx) / (document_frequency + 1))
+ return term_frequency * idf, document_frequency
+
+
+def test_subset_tfidf_and_fused_prevalence_match_reference(atac_tfidf_store):
+ store, counts = atac_tfidf_store
+ assay = store.ATAC
+ cell_idx = store.cells.active_index("subset")
+ feat_idx = assay.feats.active_index("I")
+ expected, expected_df = _reference_tfidf(counts, cell_idx, feat_idx)
+
+ assay.set_feature_stats("subset")
+
+ stats = assay.z["summary_stats_subset"]
+ np.testing.assert_array_equal(
+ np.asarray(stats["document_frequency"][:])[feat_idx],
+ expected_df,
+ )
+ np.testing.assert_allclose(
+ np.asarray(stats["prevalence"][:])[feat_idx],
+ expected.sum(axis=0),
+ rtol=1e-12,
+ atol=1e-12,
+ )
+ observed = controlled_compute(assay.normed(cell_idx, feat_idx), assay.nthreads)
+ np.testing.assert_allclose(observed, expected, rtol=1e-12, atol=1e-12)
+
+
+def test_subset_renormalization_uses_only_selected_peak_counts(atac_tfidf_store):
+ store, counts = atac_tfidf_store
+ assay = store.ATAC
+ cell_idx = np.array([0, 1, 2], dtype=np.int64)
+ feat_idx = np.array([0, 1], dtype=np.int64)
+ expected_full, _ = _reference_tfidf(counts, cell_idx, feat_idx)
+ expected_subset, _ = _reference_tfidf(
+ counts,
+ cell_idx,
+ feat_idx,
+ renormalize_subset=True,
+ )
+
+ observed_full = controlled_compute(
+ assay.normed(
+ cell_idx,
+ feat_idx,
+ renormalize_subset=False,
+ ),
+ assay.nthreads,
+ )
+ observed_subset = controlled_compute(
+ assay.normed(
+ cell_idx,
+ feat_idx,
+ renormalize_subset=True,
+ ),
+ assay.nthreads,
+ )
+
+ np.testing.assert_allclose(observed_full, expected_full, rtol=1e-12, atol=1e-12)
+ np.testing.assert_allclose(
+ observed_subset,
+ expected_subset,
+ rtol=1e-12,
+ atol=1e-12,
+ )
+ assert not np.allclose(observed_full, observed_subset)
+
+
+def test_atac_rejects_log_transform(atac_tfidf_store):
+ store, _ = atac_tfidf_store
+ assay = store.ATAC
+ cell_idx = store.cells.active_index("subset")
+ feat_idx = assay.feats.active_index("I")
+
+ with pytest.raises(
+ ValueError,
+ match="ATAC TF-IDF does not support log_transform",
+ ):
+ assay.normed(cell_idx, feat_idx, log_transform=True)
+ with pytest.raises(
+ ValueError,
+ match="ATAC TF-IDF does not support log_transform",
+ ):
+ store.run_normalization(
+ from_assay="ATAC",
+ cell_key="subset",
+ feat_key="I",
+ log_transform=True,
+ )
+
+
+def test_atac_marker_heatmap_defaults_to_unlogged_tfidf(
+ atac_tfidf_store,
+ monkeypatch,
+):
+ store, _ = atac_tfidf_store
+ assay = store.ATAC
+ store.cells.insert(
+ "atac_cluster",
+ np.array([0, 0, 1, 1]),
+ overwrite=True,
+ )
+ store.run_marker_search(
+ from_assay="ATAC",
+ group_key="atac_cluster",
+ cell_key="I",
+ feat_key="I",
+ )
+ observed_log_transform: list[bool] = []
+ original = assay.normed
+
+ def recording_normed(cell_idx=None, feat_idx=None, **kwargs):
+ observed_log_transform.append(kwargs["log_transform"])
+ return original(cell_idx, feat_idx, **kwargs)
+
+ monkeypatch.setattr(assay, "normed", recording_normed)
+ prepared = _prepare_marker_heatmap(
+ store,
+ from_assay="ATAC",
+ group_key="atac_cluster",
+ cell_key="I",
+ topn=1,
+ log_transform=None,
+ vmin=-1,
+ vmax=2,
+ )
+
+ assert observed_log_transform == [False]
+ assert not prepared["matrix"].empty
+ with pytest.raises(
+ ValueError,
+ match="ATAC TF-IDF does not support log_transform",
+ ):
+ _prepare_marker_heatmap(
+ store,
+ from_assay="ATAC",
+ group_key="atac_cluster",
+ cell_key="I",
+ topn=1,
+ log_transform=True,
+ vmin=-1,
+ vmax=2,
+ )
+
+
+def test_atac_mapping_requires_rna_query(
+ atac_tfidf_store,
+ analyzed_datastore_ephemeral,
+):
+ store, _ = atac_tfidf_store
+ state = analyzed_datastore_ephemeral.get_assay_state("RNA")
+ assert state is not None
+ assert state.neighbors is not None
+ reference = analyzed_datastore_ephemeral.build_mapping_reference(state.neighbors)
+
+ with pytest.raises(TypeError, match="RNA query assays"):
+ store.run_mapping(
+ reference,
+ "unsupported_atac_mapping",
+ query_assay="ATAC",
+ )
+
+
+def test_run_normalization_applies_and_records_atac_subset_renormalization(
+ atac_tfidf_store,
+):
+ store, counts = atac_tfidf_store
+ assay = store.ATAC
+ selected_features = np.array([True, True, False, False])
+ assay.feats.insert(
+ "subset__two_peaks",
+ selected_features,
+ overwrite=True,
+ )
+ cell_idx = store.cells.active_index("subset")
+ feat_idx = np.flatnonzero(selected_features)
+ expected, _ = _reference_tfidf(
+ counts,
+ cell_idx,
+ feat_idx,
+ renormalize_subset=True,
+ )
+
+ normalized = store.run_normalization(
+ from_assay="ATAC",
+ cell_key="subset",
+ feat_key="two_peaks",
+ renormalize_subset=True,
+ )
+
+ status = store.inspect_artifact(normalized)
+ observed = np.asarray(store.zw[status.path]["data"][:])
+ np.testing.assert_allclose(observed, expected, rtol=1e-6, atol=1e-7)
+ assert status.parameters["log_transform"] is False
+ assert status.parameters["renormalize_subset"] is True
+ assert (
+ store.run_normalization(
+ from_assay="ATAC",
+ cell_key="subset",
+ feat_key="two_peaks",
+ )
+ == normalized
+ )
+
+
+def test_document_frequency_is_feature_subset_invariant(atac_tfidf_store):
+ store, counts = atac_tfidf_store
+ assay = store.ATAC
+ cell_idx = np.array([0, 2], dtype=np.int64)
+ all_features = np.arange(counts.shape[1], dtype=np.int64)
+ selected_features = np.array([3, 1], dtype=np.int64)
+
+ assay.normed(cell_idx, all_features)
+ all_df = np.asarray(assay.n_docs_per_term).copy()
+ assay.normed(cell_idx, selected_features)
+
+ np.testing.assert_array_equal(
+ assay.n_docs_per_term,
+ all_df[selected_features],
+ )
+
+
+def _count_document_frequency_passes(monkeypatch) -> Callable[[], int]:
+ original = ChunkedArray.count_nonzero
+ calls = 0
+
+ def counting_count_nonzero(self, axis=None):
+ nonlocal calls
+ calls += 1
+ return original(self, axis)
+
+ monkeypatch.setattr(ChunkedArray, "count_nonzero", counting_count_nonzero)
+ return lambda: calls
+
+
+def test_normed_reuses_matching_df_and_recomputes_for_new_cells(
+ atac_tfidf_store,
+ monkeypatch,
+):
+ store, _ = atac_tfidf_store
+ assay = store.ATAC
+ cell_idx = store.cells.active_index("subset")
+ feat_idx = assay.feats.active_index("I")
+ assay.set_feature_stats("subset")
+
+ passes = _count_document_frequency_passes(monkeypatch)
+ assay.normed(cell_idx, feat_idx)
+ assert passes() == 0
+
+ assay.normed(np.array([0, 2], dtype=np.int64), feat_idx)
+ assert passes() == 1
+
+
+def test_normed_respects_arbitrary_cell_order(atac_tfidf_store):
+ store, counts = atac_tfidf_store
+ assay = store.ATAC
+ cell_idx = np.array([3, 0, 2], dtype=np.int64)
+ feat_idx = assay.feats.active_index("I")
+ expected, expected_df = _reference_tfidf(counts, cell_idx, feat_idx)
+
+ observed = controlled_compute(assay.normed(cell_idx, feat_idx), assay.nthreads)
+
+ np.testing.assert_allclose(observed, expected, rtol=1e-12, atol=1e-12)
+ np.testing.assert_array_equal(assay.n_docs_per_term, expected_df)
+
+
+def test_fused_stats_accumulate_across_stream_blocks(atac_tfidf_store):
+ store, counts = atac_tfidf_store
+ assay = store.ATAC
+ assay.rawData = ChunkedArray(
+ assay.rawData._backing,
+ nthreads=assay.nthreads,
+ resources=assay.resources,
+ block_size=1,
+ )
+ cell_idx = np.arange(counts.shape[0], dtype=np.int64)
+ feat_idx = assay.feats.active_index("I")
+ assert assay.rawData[:, feat_idx][cell_idx, :].numblocks[0] == len(cell_idx)
+ expected, expected_df = _reference_tfidf(counts, cell_idx, feat_idx)
+
+ document_frequency, prevalence = assay._streaming_tfidf_feature_stats(
+ cell_idx,
+ feat_idx,
+ )
+
+ np.testing.assert_array_equal(document_frequency, expected_df)
+ np.testing.assert_allclose(
+ prevalence,
+ expected.sum(axis=0),
+ rtol=1e-12,
+ atol=1e-12,
+ )
+
+
+def test_fused_stats_reserve_accumulators_and_float_block(
+ atac_tfidf_store,
+ monkeypatch,
+):
+ store, counts = atac_tfidf_store
+ assay = store.ATAC
+ assay.rawData = ChunkedArray(
+ assay.rawData._backing,
+ nthreads=assay.nthreads,
+ resources=assay.resources,
+ block_size=2,
+ )
+ captured_resident_bytes: list[int] = []
+ original = ChunkedArray._stream_blocks
+
+ def record_stream(
+ self,
+ *,
+ nthreads,
+ msg,
+ prefetch,
+ row_mask,
+ resident_bytes=0,
+ ):
+ captured_resident_bytes.append(resident_bytes)
+ return original(
+ self,
+ nthreads=nthreads,
+ msg=msg,
+ prefetch=prefetch,
+ row_mask=row_mask,
+ resident_bytes=resident_bytes,
+ )
+
+ monkeypatch.setattr(ChunkedArray, "_stream_blocks", record_stream)
+ cell_idx = np.arange(counts.shape[0], dtype=np.int64)
+ feat_idx = assay.feats.active_index("I")
+
+ assay._streaming_tfidf_feature_stats(cell_idx, feat_idx)
+
+ float_itemsize = np.dtype(np.float64).itemsize
+ expected = (
+ len(cell_idx) * float_itemsize
+ + 4 * len(feat_idx) * float_itemsize
+ + 2 * len(feat_idx) * float_itemsize
+ )
+ assert captured_resident_bytes == [expected]
+
+
+def test_normed_recomputes_df_when_stored_stats_predate_current_normalizer(
+ atac_tfidf_store,
+ monkeypatch,
+):
+ store, counts = atac_tfidf_store
+ assay = store.ATAC
+ cell_idx = store.cells.active_index("subset")
+ feat_idx = assay.feats.active_index("I")
+ assay.set_feature_stats("subset")
+ assay.z["summary_stats_subset"].attrs["normalization_identity"] = "older-tfidf"
+ expected, _ = _reference_tfidf(counts, cell_idx, feat_idx)
+
+ passes = _count_document_frequency_passes(monkeypatch)
+ observed = controlled_compute(assay.normed(cell_idx, feat_idx), assay.nthreads)
+
+ assert passes() == 1
+ np.testing.assert_allclose(observed, expected, rtol=1e-12, atol=1e-12)
+
+
+def test_normed_recomputes_df_for_features_absent_from_stored_stats(
+ atac_tfidf_store,
+ monkeypatch,
+):
+ store, counts = atac_tfidf_store
+ assay = store.ATAC
+ assay.feats.update_key(np.array([True, True, True, False]), "I")
+ assay.set_feature_stats("subset")
+ stored = np.asarray(assay.z["summary_stats_subset"]["document_frequency"][:])
+ assert np.isnan(stored[3])
+
+ assay.feats.reset_key("I")
+ cell_idx = store.cells.active_index("subset")
+ feat_idx = assay.feats.active_index("I")
+ expected, _ = _reference_tfidf(counts, cell_idx, feat_idx)
+
+ passes = _count_document_frequency_passes(monkeypatch)
+ observed = controlled_compute(assay.normed(cell_idx, feat_idx), assay.nthreads)
+
+ assert passes() == 1
+ np.testing.assert_allclose(observed, expected, rtol=1e-12, atol=1e-12)
+
+
+def test_filtered_prevalence_to_normalization_path_reuses_subset_df(
+ atac_tfidf_store,
+ monkeypatch,
+):
+ store, counts = atac_tfidf_store
+ store.mark_prevalent_peaks(
+ from_assay="ATAC",
+ cell_key="subset",
+ top_n=2,
+ )
+
+ def fail_count_nonzero(*_args, **_kwargs):
+ pytest.fail("Normalization should reuse DF from peak prevalence")
+
+ monkeypatch.setattr(ChunkedArray, "count_nonzero", fail_count_nonzero)
+ normalized = store.run_normalization(
+ from_assay="ATAC",
+ cell_key="subset",
+ feat_key="prevalent_peaks",
+ )
+
+ cell_idx = store.cells.active_index("subset")
+ feat_idx = store.ATAC.feats.active_index("subset__prevalent_peaks")
+ expected, _ = _reference_tfidf(counts, cell_idx, feat_idx)
+ status = store.inspect_artifact(normalized)
+ observed = np.asarray(store.zw[status.path]["data"][:])
+ np.testing.assert_allclose(observed, expected, rtol=1e-6, atol=1e-7)
+ assert status.parameters["log_transform"] is False
+ assert status.parameters["renormalize_subset"] is False
+ assert status.parameters["normalization_method"] == {
+ "external_hook": True,
+ "identity": "scarf.assay.norm_tf_idf:selected-cell-df:total-count-tf",
+ }
+
+
+@pytest.mark.parametrize(
+ "damage",
+ [
+ "missing_df",
+ "missing_prevalence",
+ "wrong_identity",
+ "wrong_cell_digest",
+ "malformed_df",
+ "nan_df",
+ "oversized_df",
+ ],
+)
+def test_tfidf_stats_reject_stale_or_malformed_cache(
+ atac_tfidf_store,
+ monkeypatch,
+ damage,
+):
+ store, _ = atac_tfidf_store
+ assay = store.ATAC
+ assay.set_feature_stats("subset")
+ stats = assay.z["summary_stats_subset"]
+
+ if damage == "missing_df":
+ del stats["document_frequency"]
+ elif damage == "missing_prevalence":
+ del stats["prevalence"]
+ elif damage == "wrong_identity":
+ stats.attrs["normalization_identity"] = "old-tfidf"
+ elif damage == "wrong_cell_digest":
+ stats.attrs["cell_index_digest"] = "digest-from-another-corpus"
+ elif damage == "oversized_df":
+ stats["document_frequency"][0] = store.cells.N + 1
+ elif damage == "malformed_df":
+ del stats["document_frequency"]
+ stats.create_array(
+ "document_frequency",
+ data=np.array([1.0]),
+ )
+ elif damage == "nan_df":
+ stats["document_frequency"][0] = np.nan
+
+ original: Callable = assay._streaming_tfidf_feature_stats
+ calls = 0
+
+ def counting_stats(cell_idx, feat_idx):
+ nonlocal calls
+ calls += 1
+ return original(cell_idx, feat_idx)
+
+ monkeypatch.setattr(assay, "_streaming_tfidf_feature_stats", counting_stats)
+ assay.set_feature_stats("subset")
+
+ assert calls == 1
+ repaired = assay.z["summary_stats_subset"]
+ assert "document_frequency" in repaired
+ assert "prevalence" in repaired
+ assert repaired.attrs["normalization_identity"] == getattr(
+ norm_tf_idf,
+ "artifact_identity",
+ )
+
+
+def test_custom_normalizer_keeps_generic_prevalence_path(
+ atac_tfidf_store,
+ monkeypatch,
+):
+ store, counts = atac_tfidf_store
+ assay = store.ATAC
+
+ def identity_normalizer(_assay, selected):
+ return selected
+
+ assay.normMethod = identity_normalizer
+
+ def fail_fused(*_args, **_kwargs):
+ pytest.fail("Custom normalizers must not use the TF-IDF fused path")
+
+ monkeypatch.setattr(assay, "_streaming_tfidf_feature_stats", fail_fused)
+ assay.set_feature_stats("subset")
+
+ feat_idx = assay.feats.active_index("I")
+ expected = counts[store.cells.active_index("subset")].sum(axis=0)
+ stats = assay.z["summary_stats_subset"]
+ np.testing.assert_array_equal(
+ np.asarray(stats["prevalence"][:])[feat_idx],
+ expected[feat_idx],
+ )
+ assert "document_frequency" not in stats
+
+
+def test_custom_normalizer_skips_document_frequency_pass(
+ atac_tfidf_store,
+ monkeypatch,
+):
+ store, _ = atac_tfidf_store
+ assay = store.ATAC
+ assay.normMethod = lambda _assay, selected: selected
+
+ def fail_count_nonzero(*_args, **_kwargs):
+ pytest.fail("Custom normalizers must not trigger a document frequency pass")
+
+ monkeypatch.setattr(ChunkedArray, "count_nonzero", fail_count_nonzero)
+ feat_idx = assay.feats.active_index("I")
+ assay.normed(store.cells.active_index("subset"), feat_idx)
+
+ np.testing.assert_array_equal(
+ assay.n_docs_per_term,
+ assay.feats.fetch_all("nCells")[feat_idx],
+ )
+
+
+def test_cells_without_accessible_peaks_keep_stats_finite_and_cacheable(
+ tmp_path,
+ monkeypatch,
+):
+ counts = np.array(
+ [
+ [1, 0, 1, 0],
+ [1, 1, 0, 0],
+ [0, 0, 0, 0],
+ [0, 1, 0, 1],
+ ],
+ dtype=np.uint32,
+ )
+ store = _build_store(tmp_path / "atac_empty_cell.zarr", counts)
+ assay = store.ATAC
+ store.cells.insert(
+ "all_cells", np.ones(counts.shape[0], dtype=bool), overwrite=True
+ )
+ cell_idx = store.cells.active_index("all_cells")
+ feat_idx = assay.feats.active_index("I")
+ expected, expected_df = _reference_tfidf(counts, cell_idx, feat_idx)
+
+ assay.set_feature_stats("all_cells")
+ stats = assay.z["summary_stats_all_cells"]
+ np.testing.assert_array_equal(
+ np.asarray(stats["document_frequency"][:])[feat_idx],
+ expected_df,
+ )
+ np.testing.assert_allclose(
+ np.asarray(stats["prevalence"][:])[feat_idx],
+ expected.sum(axis=0),
+ rtol=1e-12,
+ atol=1e-12,
+ )
+
+ original: Callable = assay._streaming_tfidf_feature_stats
+ calls = 0
+
+ def counting_stats(inner_cell_idx, inner_feat_idx):
+ nonlocal calls
+ calls += 1
+ return original(inner_cell_idx, inner_feat_idx)
+
+ monkeypatch.setattr(assay, "_streaming_tfidf_feature_stats", counting_stats)
+ assay.set_feature_stats("all_cells")
+ assert calls == 0
+
+ observed = controlled_compute(assay.normed(cell_idx, feat_idx), assay.nthreads)
+ np.testing.assert_allclose(observed, expected, rtol=1e-12, atol=1e-12)
+
+
+def test_empty_cell_selection_is_shape_safe(atac_tfidf_store):
+ store, _ = atac_tfidf_store
+ assay = store.ATAC
+ feat_idx = np.array([0, 1], dtype=np.int64)
+
+ values = controlled_compute(
+ assay.normed(np.array([], dtype=np.int64), feat_idx),
+ assay.nthreads,
+ )
+
+ assert values.shape == (0, len(feat_idx))
+ np.testing.assert_array_equal(
+ assay.n_docs_per_term,
+ np.zeros(len(feat_idx), dtype=np.int64),
+ )
+
+
+def test_peak_prevalence_rejects_empty_cell_corpus(atac_tfidf_store):
+ store, _ = atac_tfidf_store
+ store.cells.insert(
+ "empty",
+ np.zeros(store.cells.N, dtype=bool),
+ overwrite=True,
+ )
+
+ with pytest.raises(
+ ValueError,
+ match="Peak prevalence requires selected cells and features",
+ ):
+ store.ATAC.set_feature_stats("empty")
diff --git a/tests/test_balanced_cut.py b/tests/test_balanced_cut.py
new file mode 100644
index 00000000..88782e53
--- /dev/null
+++ b/tests/test_balanced_cut.py
@@ -0,0 +1,44 @@
+import numpy as np
+
+from scarf.clustering.balanced_cut import BalancedCut
+
+
+def _balanced_linkage() -> np.ndarray:
+ return np.asarray(
+ [
+ [0, 1, 1, 2],
+ [2, 3, 1, 2],
+ [4, 5, 1, 2],
+ [6, 7, 1, 2],
+ [8, 9, 2, 4],
+ [10, 11, 2, 4],
+ [12, 13, 10, 8],
+ ],
+ dtype=np.float64,
+ )
+
+
+def test_balanced_cut_respects_maximum_cluster_size() -> None:
+ cutter = BalancedCut(
+ _balanced_linkage(),
+ max_size=4,
+ min_size=1,
+ max_distance_fc=2.0,
+ )
+ labels = cutter.get_clusters()
+
+ assert np.array_equal(labels[:4], np.full(4, labels[0]))
+ assert np.array_equal(labels[4:], np.full(4, labels[4]))
+ assert labels[0] != labels[4]
+ assert np.bincount(labels)[1:].tolist() == [4, 4]
+
+
+def test_balanced_cut_accepts_root_sized_cluster() -> None:
+ cutter = BalancedCut(
+ _balanced_linkage(),
+ max_size=8,
+ min_size=1,
+ max_distance_fc=2.0,
+ )
+
+ assert cutter.get_clusters().tolist() == [1] * 8
diff --git a/tests/test_budget.py b/tests/test_budget.py
new file mode 100644
index 00000000..d7c8e3aa
--- /dev/null
+++ b/tests/test_budget.py
@@ -0,0 +1,251 @@
+import builtins
+
+import pytest
+
+import scarf.storage.budget as budget_module
+from scarf.storage.budget import (
+ ResourceBudget,
+ admitted_worker_count,
+ admitted_worker_split,
+ detect_total_memory_bytes,
+ detect_workers,
+ resolve_budget,
+)
+
+
+def test_resolve_budget_parses_suffix(monkeypatch):
+ monkeypatch.setenv("SCARF_MEM_BUDGET", "8G")
+ monkeypatch.setenv("SCARF_WORKERS", "3")
+ budget = resolve_budget()
+ assert budget.memoryBytes == 8 * 1024**3
+ assert budget.workers == 3
+
+
+def test_resolve_budget_raw_bytes_and_explicit_workers(monkeypatch):
+ monkeypatch.delenv("SCARF_MEM_BUDGET", raising=False)
+ budget = resolve_budget(memory=12345678, workers=2)
+ assert budget.memoryBytes == 12345678
+ assert budget.workers == 2
+
+
+def test_resolve_budget_fraction_of_total(monkeypatch):
+ monkeypatch.delenv("SCARF_MEM_BUDGET", raising=False)
+ total = detect_total_memory_bytes()
+ budget = resolve_budget(memory="0.5", workers=1)
+ assert abs(budget.memoryBytes - int(total * 0.5)) <= total * 0.01
+
+
+def test_resolve_budget_default_is_total_memory(monkeypatch):
+ monkeypatch.delenv("SCARF_MEM_BUDGET", raising=False)
+ monkeypatch.delenv("SCARF_WORKERS", raising=False)
+ budget = resolve_budget()
+ assert budget.memoryBytes == detect_total_memory_bytes()
+ assert budget.workers >= 1
+
+
+def test_detect_memory_fallback_when_meminfo_absent(monkeypatch):
+ real_open = builtins.open
+
+ def fake_open(path, *args, **kwargs):
+ if str(path) == "/proc/meminfo":
+ raise OSError("no meminfo")
+ return real_open(path, *args, **kwargs)
+
+ monkeypatch.setattr(builtins, "open", fake_open)
+ monkeypatch.setattr("os.sysconf", lambda name: -1)
+ assert detect_total_memory_bytes() == 8 * 1024 * 1024 * 1024
+
+
+@pytest.mark.parametrize(
+ "spec, expected",
+ [
+ (8 * 1024**3, 8 * 1024**3),
+ ("8G", 8 * 1024**3),
+ ("512M", 512 * 1024**2),
+ (12_345_678, 12_345_678),
+ ],
+)
+def test_memory_spec_valid(spec, expected, monkeypatch):
+ monkeypatch.delenv("SCARF_MEM_BUDGET", raising=False)
+ assert resolve_budget(memory=spec, workers=1).memoryBytes == expected
+
+
+@pytest.mark.parametrize("spec", ["1.0", "100", "abc", "-5", "0", "8Q", ""])
+def test_memory_spec_invalid_or_ambiguous_rejected(spec, monkeypatch):
+ monkeypatch.delenv("SCARF_MEM_BUDGET", raising=False)
+ with pytest.raises(ValueError):
+ resolve_budget(memory=spec, workers=1)
+
+
+def test_memory_spec_bool_rejected(monkeypatch):
+ monkeypatch.delenv("SCARF_MEM_BUDGET", raising=False)
+ with pytest.raises(ValueError, match="Invalid memory spec"):
+ resolve_budget(memory=True, workers=1)
+
+
+def test_detect_memory_uses_sysconf_when_meminfo_unavailable(monkeypatch):
+ real_open = builtins.open
+
+ def fake_open(path, *args, **kwargs):
+ if str(path) == "/proc/meminfo":
+ raise OSError("no meminfo")
+ return real_open(path, *args, **kwargs)
+
+ def fake_sysconf(name):
+ if name == "SC_PAGE_SIZE":
+ return 4096
+ if name == "SC_PHYS_PAGES":
+ return 1024
+ return -1
+
+ monkeypatch.setattr(builtins, "open", fake_open)
+ monkeypatch.setattr("os.sysconf", fake_sysconf)
+ assert detect_total_memory_bytes() == 4096 * 1024
+
+
+def test_detect_memory_uses_process_cgroup_path(monkeypatch):
+ nested_limit = 2 * 1024**3
+ monkeypatch.setattr(
+ budget_module,
+ "_process_cgroup_path",
+ lambda controller: "batch/job.scope" if controller == "" else None,
+ )
+ monkeypatch.setattr(
+ budget_module,
+ "_read_int",
+ lambda path: (
+ nested_limit if path == "/sys/fs/cgroup/batch/memory.max" else None
+ ),
+ )
+ monkeypatch.setattr(budget_module, "_physical_memory_bytes", lambda: 16 * 1024**3)
+
+ assert detect_total_memory_bytes() == nested_limit
+
+
+def test_process_cgroup_path_parses_unified_and_legacy_entries(monkeypatch):
+ content = "0::/batch/job.scope\n5:cpu,cpuacct:/legacy/job.scope\n"
+ monkeypatch.setattr("pathlib.Path.read_text", lambda path: content)
+
+ assert budget_module._process_cgroup_path("") == "batch/job.scope"
+ assert budget_module._process_cgroup_path("cpu") == "legacy/job.scope"
+ assert budget_module._process_cgroup_path("memory") is None
+
+
+def test_detect_workers_uses_process_cgroup_path(monkeypatch):
+ monkeypatch.setattr(
+ budget_module,
+ "_process_cgroup_path",
+ lambda controller: "batch/job.scope" if controller == "" else None,
+ )
+
+ def read_text(path):
+ if str(path) == "/sys/fs/cgroup/batch/job.scope/cpu.max":
+ return "250000 100000"
+ raise OSError("missing")
+
+ monkeypatch.setattr("pathlib.Path.read_text", read_text)
+ monkeypatch.setattr("os.cpu_count", lambda: 16)
+ monkeypatch.setattr("os.sched_getaffinity", lambda pid: set(range(16)))
+
+ assert detect_workers() == 2
+
+
+def test_detect_workers_uses_combined_legacy_controller_mount(monkeypatch):
+ monkeypatch.setattr(
+ budget_module,
+ "_process_cgroup_entry",
+ lambda controller: (
+ (["cpu", "cpuacct"], "batch/job.scope") if controller == "cpu" else None
+ ),
+ )
+
+ def read_text(path):
+ values = {
+ "/sys/fs/cgroup/cpu,cpuacct/batch/cpu.cfs_quota_us": "200000",
+ "/sys/fs/cgroup/cpu,cpuacct/batch/cpu.cfs_period_us": "100000",
+ }
+ try:
+ return values[str(path)]
+ except KeyError:
+ raise OSError("missing") from None
+
+ monkeypatch.setattr("pathlib.Path.read_text", read_text)
+ monkeypatch.setattr("os.cpu_count", lambda: 16)
+ monkeypatch.setattr("os.sched_getaffinity", lambda pid: set(range(16)))
+
+ assert detect_workers() == 2
+
+
+def test_invalid_workers_env_rejected(monkeypatch):
+ monkeypatch.delenv("SCARF_MEM_BUDGET", raising=False)
+ monkeypatch.setenv("SCARF_WORKERS", "not-a-number")
+ with pytest.raises(ValueError):
+ resolve_budget(memory="8G")
+
+
+def test_fraction_uses_total_memory(monkeypatch):
+ monkeypatch.delenv("SCARF_MEM_BUDGET", raising=False)
+ total = detect_total_memory_bytes()
+ got = resolve_budget(memory="0.25", workers=1).memoryBytes
+ assert abs(got - int(total * 0.25)) <= total * 0.01
+
+
+def test_admitted_worker_count_respects_cpu_memory_and_resident_bytes():
+ resources = ResourceBudget(memoryBytes=4 * 1024**3, workers=8)
+ assert admitted_worker_count(resources, taskBytes=1024**3) == 4
+ assert (
+ admitted_worker_count(
+ resources,
+ taskBytes=1024**3,
+ residentBytes=1024**3,
+ )
+ == 3
+ )
+ assert (
+ admitted_worker_count(
+ resources,
+ taskBytes=1024**3,
+ requested=2,
+ )
+ == 2
+ )
+
+
+def test_admitted_worker_count_rejects_oversized_task():
+ resources = ResourceBudget(memoryBytes=1024, workers=8)
+ with pytest.raises(MemoryError):
+ admitted_worker_count(resources, taskBytes=2048)
+
+
+def test_admitted_worker_split_bounds_outer_inner_and_resident_bytes():
+ resources = ResourceBudget(memoryBytes=500, workers=8)
+ outer, inner = admitted_worker_split(
+ resources,
+ nTasks=20,
+ taskBytes=lambda concurrency: 100 + 10 * concurrency,
+ residentBytes=100,
+ )
+ assert (outer, inner) == (3, 2)
+ assert outer * inner <= resources.workers
+ assert 100 + outer * (100 + 10 * inner) <= resources.memoryBytes
+
+
+def test_admitted_worker_split_rejects_resident_data_at_limit():
+ resources = ResourceBudget(memoryBytes=500, workers=8)
+ with pytest.raises(MemoryError, match="Resident data"):
+ admitted_worker_split(
+ resources,
+ nTasks=1,
+ taskBytes=lambda _: 1,
+ residentBytes=500,
+ )
+
+
+def test_admitted_worker_split_can_reduce_inner_concurrency_to_fit():
+ resources = ResourceBudget(memoryBytes=150, workers=8)
+ outer, inner = admitted_worker_split(
+ resources,
+ nTasks=1,
+ taskBytes=lambda concurrency: 100 + 20 * concurrency,
+ )
+ assert (outer, inner) == (1, 2)
diff --git a/tests/test_cell_cycle_genes.py b/tests/test_cell_cycle_genes.py
new file mode 100644
index 00000000..4d1d52e7
--- /dev/null
+++ b/tests/test_cell_cycle_genes.py
@@ -0,0 +1,9 @@
+from scarf.quality_control.cell_cycle_genes import g2m_phase_genes, s_phase_genes
+
+
+def test_cell_cycle_gene_lists_are_nonempty_and_unique():
+ assert len(s_phase_genes) > 20
+ assert len(g2m_phase_genes) > 20
+ assert len(s_phase_genes) == len(set(s_phase_genes))
+ assert len(g2m_phase_genes) == len(set(g2m_phase_genes))
+ assert set(s_phase_genes).isdisjoint(g2m_phase_genes)
diff --git a/tests/test_chunked.py b/tests/test_chunked.py
new file mode 100644
index 00000000..3c9d667c
--- /dev/null
+++ b/tests/test_chunked.py
@@ -0,0 +1,175 @@
+"""Tests for the ChunkedArray abstraction and the public rawData/normed API.
+
+These verify that ChunkedArray reproduces NumPy semantics for the operations
+Scarf relies on, and that the documented datastore access patterns from the
+vignettes keep working after Dask was removed.
+"""
+
+import numpy as np
+import pytest
+import zarr
+
+from scarf.matrix import ChunkedArray
+
+
+@pytest.fixture
+def backed_pair(tmp_path):
+ rng = np.random.default_rng(0)
+ n, m = 230, 70
+ dense = rng.integers(0, 6, size=(n, m)).astype(np.uint32)
+ root = zarr.open_group(str(tmp_path / "ca.zarr"), mode="w")
+ z = root.create_array("counts", shape=(n, m), chunks=(64, 32), dtype="uint32")
+ z[:, :] = dense
+ return ChunkedArray(root["counts"], nthreads=4), dense
+
+
+class TestChunkedArrayParity:
+ def test_shape_and_blocks(self, backed_pair):
+ ca, dense = backed_pair
+ from scarf.storage.layout import array_shard_rows
+
+ assert ca.shape == dense.shape
+ stream_rows = array_shard_rows(ca._backing)
+ assert ca.numblocks[0] == int(np.ceil(dense.shape[0] / stream_rows))
+ assert sum(b.shape[0] for b in ca.blocks) == dense.shape[0]
+ recon = np.vstack([b.compute() for b in ca.blocks])
+ assert np.array_equal(recon, dense)
+
+ def test_compute(self, backed_pair):
+ ca, dense = backed_pair
+ assert np.array_equal(ca.compute(), dense)
+
+ @pytest.mark.parametrize("axis", [0, 1])
+ def test_reductions(self, backed_pair, axis):
+ ca, dense = backed_pair
+ assert np.allclose(ca.sum(axis=axis).compute(), dense.sum(axis))
+ assert np.allclose(ca.mean(axis=axis).compute(), dense.mean(axis))
+ assert np.allclose(ca.var(axis=axis).compute(), dense.var(axis))
+ assert np.allclose(ca.std(axis=axis).compute(), dense.std(axis))
+
+ def test_count_nonzero_and_argmax(self, backed_pair):
+ ca, dense = backed_pair
+ assert np.array_equal(
+ np.asarray(ca.count_nonzero(axis=1)), np.count_nonzero(dense, axis=1)
+ )
+ assert np.array_equal(np.asarray(ca.argmax(axis=1)), dense.argmax(1))
+
+ def test_boolean_comparison_reduction(self, backed_pair):
+ ca, dense = backed_pair
+ assert (ca > 0).dtype == np.dtype(bool)
+ assert np.array_equal((ca > 0).sum(axis=0).compute(), (dense > 0).sum(0))
+
+ def test_scalar_reductions_weight_the_short_final_block(self, backed_pair):
+ ca, dense = backed_pair
+ assert np.allclose(ca.mean().compute(), dense.mean())
+ assert np.allclose(ca.var().compute(), dense.var())
+ assert np.allclose(ca.std().compute(), dense.std())
+ assert ca.count_nonzero().compute() == np.count_nonzero(dense)
+
+ def test_fancy_subset(self, backed_pair):
+ ca, dense = backed_pair
+ rng = np.random.default_rng(1)
+ fidx = np.sort(rng.choice(dense.shape[1], 25, replace=False))
+ cidx = np.sort(rng.choice(dense.shape[0], 90, replace=False))
+ sub = ca[:, fidx][cidx, :]
+ ref = dense[:, fidx][cidx, :]
+ assert sub.shape == ref.shape
+ assert np.array_equal(sub.compute(), ref)
+ assert np.allclose(sub.sum(axis=1).compute(), ref.sum(1))
+
+ def test_lib_size_normalization(self, backed_pair):
+ ca, dense = backed_pair
+ sub = ca[:, np.arange(dense.shape[1])]
+ scalar = dense.sum(1).astype(float)
+ scalar[scalar == 0] = 1
+ normed = 1e4 * sub / scalar.reshape(-1, 1)
+ ref = 1e4 * dense / scalar.reshape(-1, 1)
+ assert np.allclose(normed.compute(), ref)
+ assert np.allclose(np.log1p(normed).compute(), np.log1p(ref))
+
+ def test_clr_with_axis0_inside_expression(self, backed_pair):
+ ca, dense = backed_pair
+ rng = np.random.default_rng(2)
+ fidx = np.sort(rng.choice(dense.shape[1], 20, replace=False))
+ cidx = np.sort(rng.choice(dense.shape[0], 80, replace=False))
+ sub = ca[:, fidx][cidx, :]
+ ref = dense[:, fidx][cidx, :]
+ f = np.exp(np.log1p(sub).sum(axis=0) / len(sub))
+ clr = np.log1p(sub / f)
+ ref_f = np.exp(np.log1p(ref).sum(0) / ref.shape[0])
+ assert np.allclose(clr.compute(), np.log1p(ref / ref_f))
+
+ def test_column_subset_after_ops(self, backed_pair):
+ ca, dense = backed_pair
+ scalar = dense.sum(1).astype(float)
+ scalar[scalar == 0] = 1
+ normed = np.log1p(1e4 * ca / scalar.reshape(-1, 1))
+ ref = np.log1p(1e4 * dense / scalar.reshape(-1, 1))
+ cols = np.array([3, 10, 25, 40])
+ assert np.allclose(normed[:, cols].compute(), ref[:, cols])
+
+ def test_column_subset_after_two_dimensional_row_broadcast(self, backed_pair):
+ ca, dense = backed_pair
+ row_values = np.linspace(1.0, 2.0, dense.shape[1]).reshape(1, -1)
+ transformed = ca * row_values
+ cols = np.array([3, 10, 25, 40])
+
+ np.testing.assert_allclose(
+ transformed[:, cols].compute(),
+ (dense * row_values)[:, cols],
+ )
+
+ def test_dot(self, backed_pair):
+ ca, dense = backed_pair
+ rng = np.random.default_rng(3)
+ loadings = rng.standard_normal((dense.shape[1], 4))
+ assert np.allclose(ca.dot(loadings).compute(), dense @ loadings)
+
+ def test_from_numpy(self, backed_pair):
+ _, dense = backed_pair
+ ca = ChunkedArray.from_numpy(dense.astype(float), block_size=50, nthreads=2)
+ assert np.array_equal(ca.compute(), dense.astype(float))
+ assert np.allclose(ca.sum(axis=0).compute(), dense.sum(0))
+
+
+class TestPublicApiCompat:
+ """Mirror the rawData/normed usage documented in the vignettes."""
+
+ def test_rawdata_is_chunked(self, datastore):
+ raw = datastore.RNA.rawData
+ assert isinstance(raw, ChunkedArray)
+ assert len(raw.chunksize) == 2
+ assert raw.shape[0] == datastore.RNA.cells.N
+
+ def test_rawdata_mean_axis0_compute_reshape(self, datastore):
+ # Pattern from the MNIST vignette.
+ fidx = np.arange(20)
+ cidx = datastore.RNA.cells.active_index("I")[:50]
+ out = (
+ datastore.RNA.rawData[:, fidx][cidx, :]
+ .mean(axis=0)
+ .compute()
+ .reshape(1, -1)
+ )
+ assert out.shape == (1, 20)
+
+ def test_normed_mean_axis1_compute(self, datastore):
+ # Pattern from the pseudotime dynamics vignette.
+ vals = datastore.RNA.normed().mean(axis=1).compute()
+ assert vals.shape[0] == datastore.RNA.cells.active_index("I").shape[0]
+ assert np.all(np.isfinite(vals))
+
+ def test_custom_normmethod_numpy_semantics(self, datastore):
+ # User-overridable normMethod must accept NumPy-like array semantics.
+ def custom(_, counts):
+ lib = counts.sum(axis=1).reshape(-1, 1)
+ return np.log2(counts / lib * 1000 + 1)
+
+ assay = datastore.RNA
+ original = assay.normMethod
+ try:
+ assay.normMethod = custom
+ out = assay.normed().compute()
+ assert np.all(np.isfinite(out))
+ finally:
+ assay.normMethod = original
diff --git a/tests/test_cluster_separability.py b/tests/test_cluster_separability.py
new file mode 100644
index 00000000..af5e6912
--- /dev/null
+++ b/tests/test_cluster_separability.py
@@ -0,0 +1,360 @@
+import numpy as np
+import pytest
+
+from scarf.metrics import (
+ ClusterSeparabilityResult,
+ evaluate_cluster_separability,
+)
+
+
+def _score_by_name(result: ClusterSeparabilityResult, name: str):
+ rows = result.clustering_scores.set_index("clustering")
+ return rows.loc[name]
+
+
+def test_sampling_is_deterministic_stratified_and_shared():
+ rng = np.random.default_rng(12)
+ coordinates = rng.normal(size=(240, 4))
+ finest = np.repeat(np.arange(12), 20)
+ coarser = finest // 3
+ clusterings = {"finest": finest, "coarser": coarser}
+
+ first = evaluate_cluster_separability(
+ coordinates,
+ clusterings,
+ n_folds=3,
+ max_sample_cells=60,
+ max_silhouette_cells=40,
+ random_seed=91,
+ )
+ second = evaluate_cluster_separability(
+ coordinates,
+ clusterings,
+ n_folds=3,
+ max_sample_cells=60,
+ max_silhouette_cells=40,
+ random_seed=91,
+ )
+
+ np.testing.assert_array_equal(first.sample_indices, second.sample_indices)
+ np.testing.assert_array_equal(
+ np.bincount(finest[first.sample_indices], minlength=12),
+ np.full(12, 5),
+ )
+ assert set(first.clustering_scores["n_sampled_cells"]) == {60}
+ assert (
+ first.cluster_scores.groupby("clustering")["n_sampled_cells"].sum() == 60
+ ).all()
+
+
+def test_scores_are_held_out_and_cover_every_sampled_cell():
+ from sklearn.metrics import f1_score
+ from sklearn.pipeline import make_pipeline
+ from sklearn.preprocessing import StandardScaler
+ from sklearn.svm import LinearSVC
+
+ rng = np.random.default_rng(7)
+ labels = np.tile([0, 1, 2], 50)
+ coordinates = rng.normal(size=(len(labels), 120))
+
+ result = evaluate_cluster_separability(
+ coordinates,
+ {"noise": labels},
+ max_silhouette_cells=len(labels),
+ )
+ in_sample = make_pipeline(
+ StandardScaler(),
+ LinearSVC(
+ C=1.0,
+ class_weight="balanced",
+ dual="auto",
+ max_iter=10_000,
+ random_state=4444,
+ ),
+ ).fit(coordinates, labels)
+ in_sample_f1 = f1_score(
+ labels,
+ in_sample.predict(coordinates),
+ average="macro",
+ )
+ score = _score_by_name(result, "noise")
+
+ # These coordinates carry no cluster signal, so a model fitted and scored on
+ # the same rows reaches perfect separation while held-out folds cannot.
+ assert in_sample_f1 > 0.95
+ assert score["macro_f1_mean"] < 0.5
+ assert result.confusion["n_cells"].sum() == len(labels)
+ assert result.cluster_scores["n_sampled_cells"].sum() == len(labels)
+
+
+def test_separable_labels_score_higher_than_overlapping_labels():
+ rng = np.random.default_rng(34)
+ separable = np.repeat(np.arange(3), 80)
+ coordinates = np.column_stack(
+ (
+ separable * 5 + rng.normal(scale=0.35, size=len(separable)),
+ rng.normal(scale=0.5, size=len(separable)),
+ )
+ )
+ overlapping = np.tile(np.arange(3), 80)
+
+ result = evaluate_cluster_separability(
+ coordinates,
+ {"separable": separable, "overlapping": overlapping},
+ max_silhouette_cells=240,
+ )
+
+ separable_score = _score_by_name(result, "separable")
+ overlapping_score = _score_by_name(result, "overlapping")
+ assert separable_score["macro_f1_mean"] > 0.9
+ assert separable_score["macro_f1_mean"] > overlapping_score["macro_f1_mean"] + 0.4
+ assert separable_score["silhouette_score"] > overlapping_score["silhouette_score"]
+
+
+def test_macro_and_weighted_f1_reflect_cluster_imbalance():
+ rng = np.random.default_rng(52)
+ labels = np.concatenate((np.zeros(120), np.ones(30), np.full(10, 2))).astype(int)
+ coordinates = np.concatenate(
+ (
+ rng.normal(loc=(-3, 0), scale=0.5, size=(120, 2)),
+ rng.normal(loc=(3, 0), scale=0.5, size=(30, 2)),
+ rng.normal(loc=(-3, 0), scale=0.5, size=(10, 2)),
+ )
+ )
+
+ result = evaluate_cluster_separability(
+ coordinates,
+ {"imbalanced": labels},
+ max_silhouette_cells=len(labels),
+ )
+ score = _score_by_name(result, "imbalanced")
+
+ assert score["weighted_f1_mean"] > score["macro_f1_mean"]
+ cluster_scores = result.cluster_scores.set_index("cluster_label")
+ assert cluster_scores.loc[0, "f1_score"] > cluster_scores.loc[2, "f1_score"]
+
+
+def test_confusion_fractions_use_the_true_cluster_as_denominator():
+ rng = np.random.default_rng(11)
+ labels = np.asarray(["big"] * 200 + ["small"] * 25)
+ coordinates = np.concatenate(
+ (
+ rng.normal(loc=(0, 0), scale=1.0, size=(200, 2)),
+ rng.normal(loc=(2.2, 0), scale=1.0, size=(25, 2)),
+ )
+ )
+
+ result = evaluate_cluster_separability(
+ coordinates,
+ {"named": labels},
+ max_silhouette_cells=len(labels),
+ )
+ confusion = result.confusion
+ true_totals = confusion.groupby("true_cluster")["n_cells"].transform("sum")
+ predicted_totals = confusion.groupby("predicted_cluster")["n_cells"].transform(
+ "sum"
+ )
+
+ np.testing.assert_allclose(
+ confusion["fraction_of_true_cluster"].to_numpy(),
+ (confusion["n_cells"] / true_totals).to_numpy(),
+ )
+ np.testing.assert_allclose(
+ confusion.groupby("true_cluster")["fraction_of_true_cluster"].sum().to_numpy(),
+ 1,
+ )
+ assert set(result.cluster_scores["cluster_label"]) == {"big", "small"}
+ # The two clusters differ in size, so normalising by the predicted cluster
+ # instead would report different off-diagonal fractions.
+ off_diagonal = confusion["true_cluster"] != confusion["predicted_cluster"]
+ assert (confusion["n_cells"][off_diagonal] > 0).any()
+ assert not np.allclose(
+ confusion["fraction_of_true_cluster"][off_diagonal].to_numpy(),
+ (confusion["n_cells"] / predicted_totals)[off_diagonal].to_numpy(),
+ )
+
+
+def test_arbitrary_cluster_labels_round_trip():
+ coordinates = np.concatenate(
+ (
+ np.linspace(-3, -1, 18),
+ np.linspace(1, 3, 18),
+ )
+ )[:, None]
+ one_based = np.repeat([1, 4], 18)
+ strings = np.repeat(["alpha", "omega"], 18)
+
+ result = evaluate_cluster_separability(
+ coordinates,
+ {"one_based": one_based, "strings": strings},
+ n_folds=3,
+ max_silhouette_cells=len(coordinates),
+ )
+
+ assert set(
+ result.cluster_scores.query("clustering == 'one_based'")["cluster_label"]
+ ) == {1, 4}
+ assert set(result.confusion.query("clustering == 'strings'")["true_cluster"]) == {
+ "alpha",
+ "omega",
+ }
+
+
+def test_unscorable_clusterings_keep_silhouette_and_other_scores():
+ coordinates = np.concatenate(
+ (
+ np.linspace(-4, -2, 6),
+ np.linspace(2, 4, 4),
+ )
+ )[:, None]
+ too_small = np.asarray([1] * 6 + [2] * 4)
+ single = np.ones(10, dtype=int)
+ scorable = np.asarray([1] * 5 + [2] * 5)
+
+ result = evaluate_cluster_separability(
+ coordinates,
+ {"too_small": too_small, "single": single, "scorable": scorable},
+ n_folds=5,
+ max_silhouette_cells=len(coordinates),
+ )
+ too_small_score = _score_by_name(result, "too_small")
+ single_score = _score_by_name(result, "single")
+ scored = _score_by_name(result, "scorable")
+
+ assert too_small_score["status"] == "unscorable"
+ assert "fewer than 5" in too_small_score["status_reason"]
+ assert np.isnan(too_small_score["macro_f1_mean"])
+ assert np.isfinite(too_small_score["silhouette_score"])
+ assert single_score["status"] == "unscorable"
+ assert single_score["status_reason"] == "fewer than two clusters"
+ assert np.isnan(single_score["silhouette_score"])
+ assert set(
+ result.cluster_scores.query("clustering == 'single'")["n_sampled_cells"]
+ ) == {10}
+ assert scored["status"] == "scored"
+ assert np.isfinite(scored["macro_f1_mean"])
+ assert set(result.confusion["clustering"]) == {"scorable"}
+
+
+@pytest.mark.parametrize("max_silhouette_cells", [5, 10])
+def test_silhouette_is_skipped_when_the_cap_starves_clusters(max_silhouette_cells):
+ rng = np.random.default_rng(5)
+ labels = np.repeat(np.arange(10), 3)
+ coordinates = rng.normal(size=(len(labels), 2))
+
+ capped = evaluate_cluster_separability(
+ coordinates,
+ {"clusters": labels},
+ n_folds=3,
+ max_silhouette_cells=max_silhouette_cells,
+ )
+ generous = evaluate_cluster_separability(
+ coordinates,
+ {"clusters": labels},
+ n_folds=3,
+ max_silhouette_cells=len(labels),
+ )
+ capped_score = _score_by_name(capped, "clusters")
+
+ assert np.isnan(capped_score["silhouette_score"])
+ assert capped_score["status"] == "scored"
+ assert np.isfinite(capped_score["macro_f1_mean"])
+ assert np.isfinite(_score_by_name(generous, "clusters")["silhouette_score"])
+
+
+@pytest.mark.parametrize(
+ ("coordinates", "clusterings", "kwargs", "message"),
+ [
+ (np.ones(4), {"labels": np.arange(4)}, {}, "two-dimensional"),
+ (np.ones((4, 2)), {"labels": np.arange(3)}, {}, "coordinate rows"),
+ (
+ np.ones((4, 2)),
+ {"labels": np.asarray([0, 0, 1, np.nan])},
+ {},
+ "missing values",
+ ),
+ (
+ np.asarray([[0.0], [1.0], [np.inf], [2.0]]),
+ {"labels": np.asarray([0, 0, 1, 1])},
+ {},
+ "finite values",
+ ),
+ (
+ np.ones((4, 2)),
+ {"labels": np.arange(4)},
+ {"n_folds": 1},
+ "at least 2",
+ ),
+ ],
+)
+def test_invalid_inputs_are_rejected(coordinates, clusterings, kwargs, message):
+ with pytest.raises((TypeError, ValueError), match=message):
+ evaluate_cluster_separability(coordinates, clusterings, **kwargs)
+
+
+def test_datastore_wrapper_uses_explicit_pca_without_writes(
+ datastore,
+ graph_artifacts,
+ leiden_clustering,
+):
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.reduction is not None
+ columns_before = set(datastore.cells.columns)
+ artifacts_before = set(datastore.list_artifacts())
+
+ result = datastore.metric_cluster_separability(
+ state.reduction,
+ ["RNA_leiden_cluster"],
+ n_folds=3,
+ max_sample_cells=300,
+ max_silhouette_cells=100,
+ )
+
+ assert isinstance(result, ClusterSeparabilityResult)
+ assert list(result.clustering_scores["clustering"]) == ["RNA_leiden_cluster"]
+ assert int(result.clustering_scores["n_sampled_cells"].iloc[0]) == len(
+ result.sample_indices
+ )
+ assert len(result.sample_indices) == min(
+ 300,
+ len(datastore.cells.fetch("RNA_leiden_cluster", key="I")),
+ )
+ assert set(datastore.cells.columns) == columns_before
+ assert set(datastore.list_artifacts()) == artifacts_before
+ with pytest.raises(KeyError):
+ datastore.metric_cluster_separability(
+ state.reduction,
+ ["missing_cluster_column"],
+ )
+ assert state.normalized is not None
+ with pytest.raises(ValueError, match="reduction"):
+ datastore.metric_cluster_separability(
+ state.normalized,
+ ["RNA_leiden_cluster"],
+ )
+
+
+def test_datastore_wrapper_rejects_a_foreign_cell_selection(
+ datastore,
+ graph_artifacts,
+ leiden_clustering,
+):
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.reduction is not None
+ selected = np.asarray(datastore.cells.fetch_all("I"), dtype=bool)
+ subset = selected.copy()
+ subset[np.flatnonzero(selected)[::2]] = False
+ datastore.cells.insert(
+ column_name="separability_subset",
+ values=subset,
+ overwrite=True,
+ )
+
+ with pytest.raises(ValueError, match="does not match the cell selection"):
+ datastore.metric_cluster_separability(
+ state.reduction,
+ ["RNA_leiden_cluster"],
+ cell_key="separability_subset",
+ )
diff --git a/tests/test_cluster_tree.py b/tests/test_cluster_tree.py
new file mode 100644
index 00000000..407ede11
--- /dev/null
+++ b/tests/test_cluster_tree.py
@@ -0,0 +1,83 @@
+import numpy as np
+import pytest
+
+from scarf.clustering.cluster_tree import CoalesceTree, make_digraph
+
+
+def _balanced_linkage() -> np.ndarray:
+ return np.asarray(
+ [
+ [0, 1, 1, 2],
+ [2, 3, 1, 2],
+ [4, 5, 1, 2],
+ [6, 7, 1, 2],
+ [8, 9, 2, 4],
+ [10, 11, 2, 4],
+ [12, 13, 10, 8],
+ ],
+ dtype=np.float64,
+ )
+
+
+def test_make_digraph_preserves_linkage_topology_and_leaf_clusters() -> None:
+ dendrogram = _balanced_linkage()
+ clusters = np.asarray([0, 0, 1, 1, 2, 2, 3, 3])
+
+ graph = make_digraph(dendrogram, clust_info=clusters)
+
+ assert set(graph.nodes) == set(range(15))
+ assert set(graph.edges) == {
+ (8, 0),
+ (8, 1),
+ (9, 2),
+ (9, 3),
+ (10, 4),
+ (10, 5),
+ (11, 6),
+ (11, 7),
+ (12, 8),
+ (12, 9),
+ (13, 10),
+ (13, 11),
+ (14, 12),
+ (14, 13),
+ }
+ assert [graph.nodes[leaf]["cluster"] for leaf in range(8)] == clusters.tolist()
+ assert graph.nodes[14]["nleaves"] == 8
+ assert graph.nodes[14]["dist"] == 10
+
+
+def test_make_digraph_rejects_mismatched_cluster_info() -> None:
+ dendrogram = _balanced_linkage()
+ with pytest.raises(ValueError, match="cluster information"):
+ make_digraph(dendrogram, clust_info=np.zeros(3))
+
+
+def test_coalesce_tree_retains_cluster_holding_nodes_and_ancestors() -> None:
+ clusters = np.asarray([0, 0, 1, 1, 2, 2, 3, 3])
+ graph = make_digraph(_balanced_linkage(), clust_info=clusters)
+
+ coalesced = CoalesceTree(graph, clusters)
+
+ assert set(coalesced.nodes) == set(range(8, 15))
+ assert set(coalesced.edges) == {
+ (12, 8),
+ (12, 9),
+ (13, 10),
+ (13, 11),
+ (14, 12),
+ (14, 13),
+ }
+ assert {
+ int(node): int(attributes["partition_id"])
+ for node, attributes in coalesced.nodes(data=True)
+ if "partition_id" in attributes
+ } == {8: 0, 9: 1, 10: 2, 11: 3}
+
+
+def test_coalesce_tree_rejects_non_monophyletic_clusters() -> None:
+ clusters = np.asarray([0, 1, 0, 1, 2, 2, 3, 3])
+ graph = make_digraph(_balanced_linkage(), clust_info=clusters)
+
+ with pytest.raises(ValueError, match="not monophyletic"):
+ CoalesceTree(graph, clusters)
diff --git a/tests/test_compatibility_contracts.py b/tests/test_compatibility_contracts.py
new file mode 100644
index 00000000..19e53c5c
--- /dev/null
+++ b/tests/test_compatibility_contracts.py
@@ -0,0 +1,335 @@
+import inspect
+from dataclasses import FrozenInstanceError, fields
+from types import SimpleNamespace
+
+import numpy as np
+import pandas as pd
+import pytest
+import scarf
+import scarf.datastore.summary as datastore_summary
+import scarf.features as feature_algorithms
+import scarf.mapping as mapping
+import scarf.trajectory as trajectory
+
+
+_EXPECTED_TRAJECTORY_EXPORTS = (
+ "FateMappingResult",
+ "PseudotimeAggregationResult",
+ "PseudotimeMarkerResult",
+ "PseudotimeScoreResult",
+ "aggregate_feature_profiles",
+ "make_source_sink_vector",
+ "random_walk_laplacian_transpose",
+ "select_pseudotime_component",
+ "scatter_feature_clusters",
+ "truncated_pba_potential",
+ "validate_source_sink_labels",
+ "validate_source_sink_vector",
+ "validate_pseudotime_regressor",
+)
+
+_EXPECTED_RESULT_FIELDS = {
+ datastore_summary.ResourceSummary: (
+ "memory_bytes",
+ "workers",
+ "storage_profile",
+ ),
+ datastore_summary.ArtifactSummary: (
+ "ref",
+ "operation",
+ "complete",
+ ),
+ datastore_summary.AssaySummary: (
+ "name",
+ "assay_type",
+ "total_features",
+ "active_features",
+ "feature_columns",
+ "dataset_fingerprint",
+ "state",
+ "artifacts",
+ ),
+ datastore_summary.DataStoreSummary: (
+ "zarr_mode",
+ "workspace",
+ "default_assay",
+ "scarf_version",
+ "resources",
+ "total_cells",
+ "active_cells",
+ "cell_columns",
+ "assays",
+ "artifacts",
+ ),
+ feature_algorithms.EnrichmentResult: (
+ "data",
+ "source_names",
+ "source_sizes",
+ "cell_index",
+ "label",
+ "storage_path",
+ "assay",
+ "cell_key",
+ "feature_key",
+ "method",
+ ),
+ mapping.MappingResult: (
+ "ref",
+ "mapping_name",
+ "n_cells",
+ "correction_method",
+ "diagnostics",
+ "indices",
+ "distances",
+ "uninformative",
+ "reference",
+ ),
+ trajectory.FateMappingResult: (
+ "fate_keys",
+ "validity_key",
+ "sink_labels",
+ "assay",
+ "graph_cell_key",
+ "result_cell_key",
+ "feature_key",
+ "pseudotime_key",
+ "sink_key",
+ "values",
+ "valid",
+ ),
+ trajectory.PseudotimeScoreResult: (
+ "pseudotime_key",
+ "validity_key",
+ "assay",
+ "graph_cell_key",
+ "result_cell_key",
+ "feature_key",
+ "values",
+ "valid",
+ ),
+ trajectory.PseudotimeMarkerResult: (
+ "table",
+ "correlation_key",
+ "p_value_key",
+ "assay",
+ "cell_key",
+ "feature_key",
+ "pseudotime_key",
+ ),
+ trajectory.PseudotimeAggregationResult: (
+ "data",
+ "feature_indices",
+ "feature_clusters",
+ "cluster_key",
+ "storage_path",
+ "assay",
+ "cell_key",
+ "feature_key",
+ "pseudotime_key",
+ ),
+}
+
+
+def test_result_facades_and_constructor_fields_are_stable():
+ assert scarf.DataStoreSummary is datastore_summary.DataStoreSummary
+ assert scarf.EnrichmentResult is feature_algorithms.EnrichmentResult
+ assert scarf.MappingResult is mapping.MappingResult
+ assert tuple(trajectory.__all__) == _EXPECTED_TRAJECTORY_EXPORTS
+ for name in (
+ "FateMappingResult",
+ "PseudotimeAggregationResult",
+ "PseudotimeMarkerResult",
+ "PseudotimeScoreResult",
+ ):
+ assert getattr(scarf, name) is getattr(trajectory, name)
+
+ for result_class, expected_fields in _EXPECTED_RESULT_FIELDS.items():
+ assert tuple(field.name for field in fields(result_class)) == expected_fields
+ assert tuple(inspect.signature(result_class).parameters) == expected_fields
+ assert result_class.__dataclass_params__.frozen
+
+ mapping_parameters = inspect.signature(mapping.MappingResult).parameters
+ for name in (
+ "indices",
+ "distances",
+ "uninformative",
+ "reference",
+ ):
+ assert mapping_parameters[name].default is None
+
+
+def test_result_records_reject_attribute_assignment():
+ records = (
+ feature_algorithms.EnrichmentResult(
+ SimpleNamespace(shape=(2, 1)),
+ np.array(["set"]),
+ np.array([2]),
+ np.array([0, 1]),
+ "label",
+ "RNA/enrichment/label",
+ "RNA",
+ "I",
+ "I",
+ "waggr",
+ ),
+ mapping.MappingResult(
+ scarf.ArtifactRef(
+ scope="assay",
+ assay="RNA",
+ kind="projection",
+ artifact_id="a" * 64,
+ ),
+ "projection",
+ 2,
+ "none",
+ {},
+ ),
+ trajectory.FateMappingResult(
+ ("fate_A", "fate_B"),
+ "fate__valid",
+ ("A", "B"),
+ "RNA",
+ "I",
+ "I",
+ "hvgs",
+ "pseudotime",
+ "clusters",
+ np.array([[0.25, 0.75], [1.0, 0.0]]),
+ np.array([True, True]),
+ ),
+ trajectory.PseudotimeScoreResult(
+ "pseudotime",
+ "valid",
+ "RNA",
+ "I",
+ "I",
+ "hvgs",
+ np.array([0.0, 1.0]),
+ np.array([True, True]),
+ ),
+ trajectory.PseudotimeMarkerResult(
+ pd.DataFrame(
+ {
+ "feature_index": [0],
+ "feature_name": ["gene"],
+ "r_value": [1.0],
+ "p_value": [0.0],
+ }
+ ),
+ "correlation",
+ "p_value",
+ "RNA",
+ "I",
+ "hvgs",
+ "pseudotime",
+ ),
+ trajectory.PseudotimeAggregationResult(
+ SimpleNamespace(shape=(2, 3)),
+ np.array([0, 1]),
+ np.array([0, 1]),
+ "clusters",
+ "aggregated",
+ "RNA",
+ "I",
+ "hvgs",
+ "pseudotime",
+ ),
+ )
+
+ for record in records:
+ first_field = fields(record)[0].name
+ with pytest.raises(FrozenInstanceError):
+ setattr(record, first_field, None)
+
+
+def test_pseudotime_result_shape_validation_is_stable():
+ with pytest.raises(ValueError, match="rows must align"):
+ trajectory.FateMappingResult(
+ ("fate_A", "fate_B"),
+ "fate__valid",
+ ("A", "B"),
+ "RNA",
+ "I",
+ "I",
+ "hvgs",
+ "pseudotime",
+ "clusters",
+ np.array([[0.25, 0.75], [1.0, 0.0]]),
+ np.array([True]),
+ )
+
+ with pytest.raises(ValueError, match="same shape"):
+ trajectory.PseudotimeScoreResult(
+ "pseudotime",
+ "valid",
+ "RNA",
+ "I",
+ "I",
+ "hvgs",
+ np.array([0.0, 1.0]),
+ np.array([True]),
+ )
+
+ with pytest.raises(ValueError, match="missing columns"):
+ trajectory.PseudotimeMarkerResult(
+ pd.DataFrame({"feature_index": [0]}),
+ "correlation",
+ "p_value",
+ "RNA",
+ "I",
+ "hvgs",
+ "pseudotime",
+ )
+
+ with pytest.raises(ValueError, match="Feature clusters"):
+ trajectory.PseudotimeAggregationResult(
+ SimpleNamespace(shape=(2, 3)),
+ np.array([0, 1]),
+ np.array([0]),
+ "clusters",
+ "aggregated",
+ "RNA",
+ "I",
+ "hvgs",
+ "pseudotime",
+ )
+
+
+def test_mapping_execution_contract_is_query_owned():
+ signature = inspect.signature(scarf.DataStore.run_mapping)
+ assert str(signature) == (
+ "(self, reference: scarf.mapping.reference.MappingReference, "
+ "mapping_name: str, *, query_assay: str | None = None, "
+ "cell_key: str = 'I', save_k: int = 3, "
+ "missing_feature_policy: str = 'reference_mean', "
+ "query_batches: pandas.core.frame.DataFrame | None = None, "
+ "invalidate_cache: bool = False) -> scarf.mapping.models.MappingResult"
+ )
+ assert tuple(signature.parameters) == (
+ "self",
+ "reference",
+ "mapping_name",
+ "query_assay",
+ "cell_key",
+ "save_k",
+ "missing_feature_policy",
+ "query_batches",
+ "invalidate_cache",
+ )
+ with pytest.raises(TypeError, match="unexpected keyword argument 'target_assay'"):
+ scarf.DataStore.run_mapping(
+ object(),
+ object(),
+ "legacy",
+ target_assay=object(),
+ )
+
+
+def test_retired_unified_mapping_contract_is_absent():
+ for name in (
+ "_load_unified_layout_data",
+ "load_unified_graph",
+ "run_unified_tsne",
+ "run_unified_umap",
+ ):
+ assert not hasattr(scarf.DataStore, name)
diff --git a/tests/test_counts_t.py b/tests/test_counts_t.py
new file mode 100644
index 00000000..3d690554
--- /dev/null
+++ b/tests/test_counts_t.py
@@ -0,0 +1,943 @@
+import numpy as np
+import pandas as pd
+import pytest
+import zarr
+from scipy.sparse import csr_matrix
+from zarr.storage import MemoryStore
+
+from scarf.assay import Assay, RNAassay, lib_size_feature_stream_eligible, norm_dummy
+from scarf.features.markers.batching import feature_column_chunk
+from scarf.features.genomic.melding import coordinate_melding
+from scarf.features.markers import find_markers_by_rank, find_markers_by_regression
+from scarf.metadata import MetaData
+from scarf.quality_control.doublets import write_doublet_target_zarr
+from scarf.storage.budget import ResourceBudget
+from scarf.storage.sharding import write_counts_t
+from tests.store_probes import RecordingStore
+from scarf.writers import (
+ create_cell_data,
+ create_zarr_count_assay,
+)
+
+
+def _memory_root() -> zarr.Group:
+ return zarr.open_group(store=MemoryStore(), mode="w")
+
+
+def _write_small_assay(
+ root: zarr.Group,
+ *,
+ workspace: str | None,
+ values: np.ndarray,
+) -> zarr.Array:
+ n_cells, n_feats = values.shape
+ create_cell_data(
+ root,
+ workspace,
+ ids=np.array([f"c{i}" for i in range(n_cells)]),
+ names=np.array([f"c{i}" for i in range(n_cells)]),
+ )
+ create_zarr_count_assay(
+ z=root,
+ assay_name="RNA",
+ workspace=workspace,
+ n_cells=n_cells,
+ feat_ids=np.array([f"f{i}" for i in range(n_feats)]),
+ feat_names=np.array([f"g{i}" for i in range(n_feats)]),
+ dtype="uint32",
+ )
+ if workspace is None:
+ counts = root["RNA/counts"]
+ else:
+ counts = root["matrices/RNA/counts"]
+ counts[:] = values
+ group = root["RNA"] if workspace is None else root["matrices/RNA"]
+ write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(1024**3, 2),
+ )
+ return counts
+
+
+@pytest.mark.parametrize("workspace", [None, "ws"])
+def test_explicit_write_counts_t_builds_complete_counts_t(workspace):
+ root = _memory_root()
+ values = np.arange(20, dtype=np.uint32).reshape(5, 4)
+ counts = _write_small_assay(root, workspace=workspace, values=values)
+
+ if workspace is None:
+ group = root["RNA"]
+ else:
+ group = root["matrices/RNA"]
+ assert "countsT" in group
+ counts_t = group["countsT"]
+ assert counts_t.attrs["complete"] is True
+ assert counts_t.shape == (4, 5)
+ assert counts_t.dtype == counts.dtype
+ np.testing.assert_array_equal(counts_t[:], values.T)
+
+
+def test_write_counts_t_marks_incomplete_until_finished():
+ root = _memory_root()
+ group = root.create_group("RNA")
+ counts = group.create_array(
+ "counts",
+ shape=(4, 3),
+ chunks=(2, 2),
+ dtype=np.uint32,
+ fill_value=0,
+ )
+ counts[:] = np.arange(12, dtype=np.uint32).reshape(4, 3)
+ counts_t = write_counts_t(counts, group)
+ assert counts_t is not None
+ assert counts_t.attrs["complete"] is True
+ np.testing.assert_array_equal(counts_t[:], counts[:].T)
+
+
+def test_write_counts_t_feature_major_candidate_inherits_chunk_area():
+ root = _memory_root()
+ group = root.create_group("RNA")
+ values = np.arange(22 * 7, dtype=np.uint32).reshape(22, 7)
+ counts = group.create_array(
+ "counts",
+ data=values,
+ chunks=(6, 4),
+ )
+
+ counts_t = write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(1024**2, 2),
+ feature_major_layout=True,
+ )
+
+ assert counts_t is not None
+ assert counts_t.chunks == (1, 22)
+ np.testing.assert_array_equal(counts_t[:], values.T)
+
+
+def test_assay_loads_counts_t_and_falls_back_when_incomplete():
+ root = _memory_root()
+ values = np.arange(12, dtype=np.uint32).reshape(3, 4)
+ _write_small_assay(root, workspace=None, values=values)
+
+ cells = MetaData(root["cellData"])
+ assay = Assay(root, None, "RNA", cells, nthreads=1)
+ assert assay.rawDataT is not None
+ np.testing.assert_array_equal(assay.rawDataT[:], values.T)
+
+ root["RNA/countsT"].attrs["complete"] = False
+ broken = Assay(root, None, "RNA", cells, nthreads=1)
+ assert broken.rawDataT is None
+
+ root["RNA/countsT"].attrs["complete"] = "false"
+ string_flag = Assay(root, None, "RNA", cells, nthreads=1)
+ assert string_flag.rawDataT is None
+
+
+def test_assay_falls_back_on_wrong_shape_dtype_or_group_node():
+ root = _memory_root()
+ values = np.arange(12, dtype=np.uint32).reshape(3, 4)
+ _write_small_assay(root, workspace=None, values=values)
+ cells = MetaData(root["cellData"])
+
+ del root["RNA/countsT"]
+ root["RNA"].create_array(
+ "countsT",
+ shape=(2, 3),
+ chunks=(2, 2),
+ dtype=np.uint32,
+ fill_value=0,
+ )
+ root["RNA/countsT"].attrs["complete"] = True
+ assert Assay(root, None, "RNA", cells, nthreads=1).rawDataT is None
+
+ del root["RNA/countsT"]
+ root["RNA"].create_array(
+ "countsT",
+ shape=(4, 3),
+ chunks=(2, 2),
+ dtype=np.float32,
+ fill_value=0,
+ )
+ root["RNA/countsT"].attrs["complete"] = True
+ assert Assay(root, None, "RNA", cells, nthreads=1).rawDataT is None
+
+ del root["RNA/countsT"]
+ root["RNA"].create_group("countsT")
+ assert Assay(root, None, "RNA", cells, nthreads=1).rawDataT is None
+
+
+def test_rna_feature_reads_match_with_and_without_counts_t():
+ root = _memory_root()
+ values = np.array(
+ [
+ [4, 0, 1, 2],
+ [3, 0, 1, 0],
+ [0, 5, 1, 2],
+ [0, 6, 1, 2],
+ ],
+ dtype=np.uint32,
+ )
+ _write_small_assay(root, workspace=None, values=values)
+ del root["RNA/countsT"]
+ counts_t = root["RNA"].create_array(
+ "countsT",
+ data=values.T,
+ chunks=(1, 2),
+ )
+ counts_t.attrs["complete"] = True
+ cells = MetaData(root["cellData"])
+ cells.insert("RNA_nCounts", values.sum(axis=1).astype(np.float64), overwrite=True)
+
+ with_t = RNAassay(root, "RNA", cells, workspace=None, nthreads=1)
+ with_t.sf = 1000
+ assert with_t.rawDataT is not None
+
+ cell_idx = np.array([0, 2, 3])
+ feat_idx = np.array([1, 3, 0])
+ stats_with = with_t._streaming_feature_stats(cell_idx, feat_idx)
+ blocks_with = [
+ raw.copy()
+ for _, raw, _, _, _ in with_t.iter_raw_column_blocks(
+ cell_idx, feat_idx, batch_size=2
+ )
+ ]
+
+ root["RNA/countsT"].attrs["complete"] = False
+ without_t = RNAassay(root, "RNA", cells, workspace=None, nthreads=1)
+ without_t.sf = 1000
+ assert without_t.rawDataT is None
+ stats_without = without_t._streaming_feature_stats(cell_idx, feat_idx)
+ blocks_without = [
+ raw.copy()
+ for _, raw, _, _, _ in without_t.iter_raw_column_blocks(
+ cell_idx, feat_idx, batch_size=2
+ )
+ ]
+
+ for key in stats_with:
+ np.testing.assert_allclose(stats_with[key], stats_without[key], rtol=1e-10)
+ assert len(blocks_with) == len(blocks_without)
+ for left, right in zip(blocks_with, blocks_without, strict=True):
+ np.testing.assert_array_equal(left, right)
+
+ assert feature_column_chunk(with_t, n_features=4) == int(with_t.rawDataT.chunks[0])
+ assert feature_column_chunk(without_t, n_features=4) == int(
+ without_t.rawData._backing.chunks[1]
+ )
+
+
+def test_marker_results_match_with_and_without_counts_t():
+ root = _memory_root()
+ values = np.array(
+ [
+ [4, 0, 1, 2],
+ [3, 0, 1, 0],
+ [0, 5, 1, 2],
+ [0, 6, 1, 2],
+ ],
+ dtype=np.uint32,
+ )
+ _write_small_assay(root, workspace=None, values=values)
+ del root["RNA/countsT"]
+ counts_t = root["RNA"].create_array(
+ "countsT",
+ data=values.T,
+ chunks=(1, 2),
+ )
+ counts_t.attrs["complete"] = True
+ cells = MetaData(root["cellData"])
+ cells.insert("RNA_nCounts", values.sum(axis=1).astype(np.float64), overwrite=True)
+ cells.insert("cluster", np.array(["a", "a", "b", "b"]), overwrite=True)
+
+ with_t = RNAassay(
+ root, "RNA", cells, workspace=None, nthreads=1, min_cells_per_feature=1
+ )
+ with_t.sf = 1000.0
+ assert with_t.rawDataT is not None
+ results_with = find_markers_by_rank(
+ with_t,
+ group_key="cluster",
+ cell_key="I",
+ feat_key="I",
+ batch_size=2,
+ n_threads=1,
+ )
+
+ root["RNA/countsT"].attrs["complete"] = False
+ without_t = RNAassay(
+ root, "RNA", cells, workspace=None, nthreads=1, min_cells_per_feature=1
+ )
+ without_t.sf = 1000.0
+ assert without_t.rawDataT is None
+ results_without = find_markers_by_rank(
+ without_t,
+ group_key="cluster",
+ cell_key="I",
+ feat_key="I",
+ batch_size=2,
+ n_threads=1,
+ )
+
+ assert set(results_with) == set(results_without)
+ for group in results_with:
+ left = results_with[group].sort_index()
+ right = results_without[group].sort_index()
+ np.testing.assert_allclose(
+ left.to_numpy(dtype=np.float64),
+ right.to_numpy(dtype=np.float64),
+ rtol=1e-10,
+ equal_nan=True,
+ )
+
+
+def test_marker_stream_releases_consumed_block_before_reading_next(
+ monkeypatch: pytest.MonkeyPatch,
+):
+ import gc
+ import weakref
+
+ import scarf.assay.rna as rna_module
+
+ root = _memory_root()
+ values = np.arange(1, 37, dtype=np.uint32).reshape(6, 6)
+ _write_small_assay(root, workspace=None, values=values)
+ cells = MetaData(root["cellData"])
+ cells.insert("RNA_nCounts", values.sum(axis=1).astype(np.float64), overwrite=True)
+ cells.insert("cluster", np.array(["a", "a", "a", "b", "b", "b"]), overwrite=True)
+ assay = RNAassay(
+ root,
+ "RNA",
+ cells,
+ workspace=None,
+ nthreads=1,
+ min_cells_per_feature=1,
+ )
+ assay.sf = 1_000.0
+
+ original_read = rna_module._read_facade_block
+ previous: weakref.ReferenceType[np.ndarray] | None = None
+ read_count = 0
+
+ def tracked_read(
+ array: zarr.Array,
+ rows: np.ndarray,
+ columns: np.ndarray,
+ ) -> np.ndarray:
+ nonlocal previous, read_count
+ gc.collect()
+ if previous is not None:
+ assert previous() is None
+ raw = np.ascontiguousarray(original_read(array, rows, columns))
+ previous = weakref.ref(raw)
+ read_count += 1
+ return raw
+
+ monkeypatch.setattr(rna_module, "_read_facade_block", tracked_read)
+ find_markers_by_rank(
+ assay,
+ group_key="cluster",
+ cell_key="I",
+ feat_key="I",
+ batch_size=2,
+ n_threads=1,
+ )
+
+ gc.collect()
+ assert read_count == 3
+ assert previous is not None
+ assert previous() is None
+
+
+def _small_rna_with_ptime(values: np.ndarray) -> tuple[zarr.Group, MetaData]:
+ root = _memory_root()
+ _write_small_assay(root, workspace=None, values=values)
+ cells = MetaData(root["cellData"])
+ cells.insert("RNA_nCounts", values.sum(axis=1).astype(np.float64), overwrite=True)
+ cells.insert("ptime", np.linspace(0.0, 1.0, values.shape[0]), overwrite=True)
+ return root, cells
+
+
+def test_iter_normed_feature_wise_matches_with_and_without_counts_t():
+ values = np.array(
+ [
+ [4, 0, 1, 2],
+ [3, 0, 1, 0],
+ [0, 5, 1, 2],
+ [0, 6, 1, 2],
+ ],
+ dtype=np.uint32,
+ )
+ root, cells = _small_rna_with_ptime(values)
+
+ with_t = RNAassay(
+ root, "RNA", cells, workspace=None, nthreads=1, min_cells_per_feature=1
+ )
+ with_t.sf = 1000.0
+ assert with_t.rawDataT is not None
+ frames_with = list(
+ with_t.iter_normed_feature_wise("I", "I", 2, "normed feats", log_transform=True)
+ )
+
+ root["RNA/countsT"].attrs["complete"] = False
+ without_t = RNAassay(
+ root, "RNA", cells, workspace=None, nthreads=1, min_cells_per_feature=1
+ )
+ without_t.sf = 1000.0
+ assert without_t.rawDataT is None
+ frames_without = list(
+ without_t.iter_normed_feature_wise(
+ "I", "I", 2, "normed feats", log_transform=True
+ )
+ )
+
+ mat_with = pd.concat(frames_with, axis=1)
+ mat_without = pd.concat(frames_without, axis=1)
+ assert list(mat_with.columns) == list(mat_without.columns)
+ np.testing.assert_allclose(
+ mat_with.to_numpy(dtype=np.float64),
+ mat_without.to_numpy(dtype=np.float64),
+ rtol=1e-10,
+ )
+
+
+def test_slow_feature_iterator_never_exceeds_batch_size():
+ values = np.arange(1, 29, dtype=np.uint32).reshape(4, 7)
+ root, cells = _small_rna_with_ptime(values)
+ assay = RNAassay(
+ root,
+ "RNA",
+ cells,
+ workspace=None,
+ nthreads=1,
+ min_cells_per_feature=1,
+ )
+ assay.sf = 1000.0
+
+ frames = list(
+ assay.iter_normed_feature_wise(
+ "I",
+ "I",
+ 4,
+ None,
+ renormalize_subset=True,
+ )
+ )
+
+ assert [frame.shape[1] for frame in frames] == [4, 3]
+
+
+def test_regression_and_aggregation_match_with_and_without_counts_t():
+ values = np.array(
+ [
+ [4, 0, 1, 2],
+ [3, 0, 1, 0],
+ [0, 5, 1, 2],
+ [0, 6, 1, 2],
+ ],
+ dtype=np.uint32,
+ )
+ root, cells = _small_rna_with_ptime(values)
+ ptime = cells.fetch("ptime", key="I")
+
+ with_t = RNAassay(
+ root, "RNA", cells, workspace=None, nthreads=1, min_cells_per_feature=1
+ )
+ with_t.sf = 1000.0
+ assert with_t.rawDataT is not None
+ regression_with = find_markers_by_regression(
+ with_t,
+ cell_key="I",
+ feat_key="I",
+ regressor=ptime,
+ min_cells=1,
+ batch_size=2,
+ )
+ agg_with, feats_with = with_t.save_aggregated_ordering(
+ cell_key="I",
+ feat_key="I",
+ ordering_key="ptime",
+ min_exp=0.0,
+ smoothen=False,
+ z_scale=False,
+ window_size=2,
+ chunk_size=2,
+ batch_size=2,
+ )
+ agg_with_vals = np.asarray(agg_with.compute())
+ location = "aggregated_I_I_ptime"
+ del root["RNA"][location]
+
+ root["RNA/countsT"].attrs["complete"] = False
+ without_t = RNAassay(
+ root, "RNA", cells, workspace=None, nthreads=1, min_cells_per_feature=1
+ )
+ without_t.sf = 1000.0
+ assert without_t.rawDataT is None
+ regression_without = find_markers_by_regression(
+ without_t,
+ cell_key="I",
+ feat_key="I",
+ regressor=ptime,
+ min_cells=1,
+ batch_size=2,
+ )
+ agg_without, feats_without = without_t.save_aggregated_ordering(
+ cell_key="I",
+ feat_key="I",
+ ordering_key="ptime",
+ min_exp=0.0,
+ smoothen=False,
+ z_scale=False,
+ window_size=2,
+ chunk_size=2,
+ batch_size=2,
+ )
+
+ np.testing.assert_allclose(
+ regression_with.sort_index().to_numpy(dtype=np.float64),
+ regression_without.sort_index().to_numpy(dtype=np.float64),
+ rtol=1e-10,
+ equal_nan=True,
+ )
+ np.testing.assert_array_equal(feats_with, feats_without)
+ np.testing.assert_allclose(
+ agg_with_vals,
+ np.asarray(agg_without.compute()),
+ rtol=1e-10,
+ )
+
+
+def test_iter_normed_feature_wise_uses_slow_path_when_ineligible(monkeypatch):
+ values = np.array(
+ [
+ [4, 0, 1, 2],
+ [3, 0, 1, 0],
+ [0, 5, 1, 2],
+ [0, 6, 1, 2],
+ ],
+ dtype=np.uint32,
+ )
+ root, cells = _small_rna_with_ptime(values)
+ assay = RNAassay(
+ root, "RNA", cells, workspace=None, nthreads=1, min_cells_per_feature=1
+ )
+ assay.sf = 1000.0
+ assert lib_size_feature_stream_eligible(assay) is True
+ assert lib_size_feature_stream_eligible(assay, renormalize_subset=True) is False
+
+ calls = {"raw": 0, "base": 0}
+ orig_raw = RNAassay._iter_raw_feature_columns
+ orig_base = Assay.iter_normed_feature_wise
+
+ def spy_raw(self, *args, **kwargs):
+ calls["raw"] += 1
+ yield from orig_raw(self, *args, **kwargs)
+
+ def spy_base(self, *args, **kwargs):
+ calls["base"] += 1
+ yield from orig_base(self, *args, **kwargs)
+
+ monkeypatch.setattr(RNAassay, "_iter_raw_feature_columns", spy_raw)
+ monkeypatch.setattr(Assay, "iter_normed_feature_wise", spy_base)
+
+ list(assay.iter_normed_feature_wise("I", "I", 2, None))
+ assert calls["raw"] >= 1
+ assert calls["base"] == 0
+
+ calls["raw"] = 0
+ calls["base"] = 0
+ list(assay.iter_normed_feature_wise("I", "I", 2, None, renormalize_subset=True))
+ assert calls["raw"] == 0
+ assert calls["base"] >= 1
+
+ calls["raw"] = 0
+ calls["base"] = 0
+ assay.normMethod = norm_dummy
+ assert lib_size_feature_stream_eligible(assay) is False
+ list(assay.iter_normed_feature_wise("I", "I", 2, None))
+ assert calls["raw"] == 0
+ assert calls["base"] >= 1
+
+
+def test_coordinate_melding_leaves_counts_t_on_demand():
+ root = _memory_root()
+ values = np.array(
+ [
+ [1, 0, 2, 3],
+ [0, 4, 0, 0],
+ [0, 0, 5, 0],
+ ],
+ dtype=np.uint32,
+ )
+ n_cells, n_feats = values.shape
+ create_cell_data(
+ root,
+ None,
+ ids=np.array([f"c{i}" for i in range(n_cells)]),
+ names=np.array([f"c{i}" for i in range(n_cells)]),
+ )
+ peak_ids = np.array(
+ ["chr1:100-200", "chr1:150-250", "chr1:400-500", "chr2:100-200"]
+ )
+ create_zarr_count_assay(
+ z=root,
+ assay_name="ATAC",
+ workspace=None,
+ n_cells=n_cells,
+ feat_ids=peak_ids,
+ feat_names=peak_ids,
+ dtype="uint32",
+ )
+ root["ATAC/counts"][:] = values
+ cells = MetaData(root["cellData"])
+ n_features = (values > 0).sum(axis=1).astype(np.float64)
+ n_counts = values.sum(axis=1, dtype=np.float64)
+ n_cells_per_peak = (values > 0).sum(axis=0).astype(np.float64)
+ cells.insert("ATAC_nFeatures", n_features, overwrite=True)
+ cells.insert("ATAC_nCounts", n_counts, overwrite=True)
+ assay = Assay(root, None, "ATAC", cells, nthreads=1, min_cells_per_feature=1)
+ assay.feats.insert("nCells", n_cells_per_peak, overwrite=True)
+
+ import pandas as pd
+
+ feature_bed = pd.DataFrame(
+ {
+ 0: ["chr1", "chr1", "chr2"],
+ 1: [120, 300, 150],
+ 2: [160, 350, 180],
+ 3: ["a", "b", "c"],
+ 4: ["A", "B", "C"],
+ }
+ )
+ coordinate_melding(
+ assay,
+ workspace=None,
+ feature_bed=feature_bed,
+ new_assay_name="GENE",
+ peaks_col="ids",
+ renormalization=False,
+ )
+ assert "counts" in root["GENE"]
+ assert "countsT" not in root["GENE"]
+
+
+def test_count_assay_leaves_counts_t_on_demand():
+ root = _memory_root()
+ values = np.arange(12, dtype=np.float32).reshape(3, 4)
+ n_cells, n_feats = values.shape
+ create_cell_data(
+ root,
+ None,
+ ids=np.array([f"c{i}" for i in range(n_cells)]),
+ names=np.array([f"c{i}" for i in range(n_cells)]),
+ )
+ create_zarr_count_assay(
+ z=root,
+ assay_name="PTIME_MODULES",
+ workspace=None,
+ n_cells=n_cells,
+ feat_ids=np.array([f"group_{i}" for i in range(n_feats)]),
+ feat_names=np.array([f"group_{i}" for i in range(n_feats)]),
+ dtype="float",
+ )
+ root["PTIME_MODULES/counts"][:] = values
+ assert "countsT" not in root["PTIME_MODULES"]
+
+
+def test_write_doublet_target_zarr_skips_counts_t(tmp_path):
+ sim = csr_matrix(np.array([[1, 0, 2], [0, 3, 0]], dtype=np.uint32))
+ zarr_loc = str(tmp_path / "doublets.zarr")
+ root = write_doublet_target_zarr(
+ zarr_loc=zarr_loc,
+ assay_name="RNA",
+ sim_counts=sim,
+ feat_ids=np.array(["f0", "f1", "f2"]),
+ feat_names=np.array(["g0", "g1", "g2"]),
+ dtype="uint32",
+ )
+ assert "counts" in root["RNA"]
+ assert "countsT" not in root["RNA"]
+
+
+def test_write_doublet_target_rejects_summary_before_truncating_destination():
+ store = MemoryStore()
+ root = zarr.open_group(store=store, mode="w")
+ root.create_group("sentinel")
+
+ with pytest.raises(ValueError, match=r"reserved for DataStore\.summary"):
+ write_doublet_target_zarr(
+ zarr_loc=store,
+ assay_name="summary",
+ sim_counts=csr_matrix(np.ones((1, 1), dtype=np.uint32)),
+ feat_ids=np.array(["f0"]),
+ feat_names=np.array(["g0"]),
+ )
+
+ preserved = zarr.open_group(store=store, mode="r")
+ assert set(preserved.group_keys()) == {"sentinel"}
+
+
+def _counts_array(
+ values: np.ndarray,
+ *,
+ chunks: tuple[int, int] = (3, 2),
+ shards: tuple[int, int] | None = None,
+ store: MemoryStore | None = None,
+) -> tuple[zarr.Group, zarr.Array]:
+ root = zarr.open_group(
+ store=store if store is not None else MemoryStore(), mode="w"
+ )
+ group = root.create_group("RNA")
+ counts = group.create_array(
+ "counts",
+ shape=values.shape,
+ chunks=chunks,
+ shards=shards,
+ dtype=values.dtype,
+ fill_value=0,
+ )
+ counts[:] = values
+ return group, counts
+
+
+def _dense_values(n_cells: int, n_feats: int) -> np.ndarray:
+ return (np.arange(n_cells * n_feats, dtype=np.uint32) + 1).reshape(n_cells, n_feats)
+
+
+@pytest.mark.parametrize("workers", [1, 4])
+def test_write_counts_t_transposes_exactly(workers):
+ values = _dense_values(24, 8)
+ group, counts = _counts_array(values)
+ counts_t = write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, workers),
+ )
+ assert counts_t is not None
+ assert counts_t.attrs["complete"] is True
+ np.testing.assert_array_equal(counts_t[:], values.T)
+
+
+def test_write_counts_t_geometry_matches_serial_baseline():
+ values = _dense_values(22, 7)
+ metadata = []
+ for workers in (1, 4):
+ group, counts = _counts_array(values)
+ counts_t = write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, workers),
+ )
+ np.testing.assert_array_equal(counts_t[:], values.T)
+ metadata.append(counts_t.metadata.to_dict())
+
+ assert metadata[0] == metadata[1]
+ assert metadata[0]["shape"] == (7, 22)
+ assert metadata[0]["chunk_grid"]["configuration"]["chunk_shape"] == (2, 3)
+ assert metadata[0]["fill_value"] == 0
+
+
+def test_write_counts_t_covers_edge_tiles():
+ # 7 features over a 2-wide grid and 22 cells over a 6-wide grid leave a
+ # partial tile on both axes, including the shared corner tile.
+ values = _dense_values(22, 7)
+ group, counts = _counts_array(values)
+ counts_t = write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, 4),
+ )
+ np.testing.assert_array_equal(counts_t[0:2, 0:3], values[0:3, 0:2].T)
+ np.testing.assert_array_equal(counts_t[6:7, 0:3], values[0:3, 6:7].T)
+ np.testing.assert_array_equal(counts_t[0:2, 21:22], values[21:22, 0:2].T)
+ np.testing.assert_array_equal(counts_t[6:7, 21:22], values[21:22, 6:7].T)
+
+
+def test_write_counts_t_writes_each_chunk_once_without_destination_read():
+ store = RecordingStore()
+ values = _dense_values(22, 7)
+ group, counts = _counts_array(values, store=store)
+ store.reset()
+ counts_t = write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, 4),
+ )
+
+ chunk_ops = store.chunk_ops("RNA/countsT/c/")
+ assert {kind for kind, _ in chunk_ops} == {"set"}
+ keys = [key for _, key in chunk_ops]
+ assert len(keys) == len(set(keys))
+ n_tiles = len(range(0, 7, 2)) * len(range(0, 22, 3))
+ assert len(keys) == n_tiles
+ np.testing.assert_array_equal(counts_t[:], values.T)
+
+
+def test_write_counts_t_reads_each_source_inner_chunk_once():
+ store = RecordingStore()
+ values = _dense_values(24, 8)
+ group, counts = _counts_array(
+ values,
+ chunks=(3, 2),
+ shards=(6, 8),
+ store=store,
+ )
+ store.reset()
+
+ counts_t = write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, 4),
+ )
+
+ source_gets = [
+ (key, byte_range)
+ for kind, key, byte_range in store.byte_ranges
+ if kind == "get" and key.startswith("RNA/counts/c/")
+ ]
+ shard_rows, shard_columns = counts.metadata.shards
+ expected_shard_keys = {
+ f"RNA/counts/c/{row}/{column}"
+ for row in range((counts.shape[0] + shard_rows - 1) // shard_rows)
+ for column in range((counts.shape[1] + shard_columns - 1) // shard_columns)
+ }
+ assert {key for key, _ in source_gets} == expected_shard_keys
+ assert all(byte_range is not None for _, byte_range in source_gets)
+ index_reads = [
+ (key, byte_range.suffix)
+ for key, byte_range in source_gets
+ if hasattr(byte_range, "suffix")
+ ]
+ chunk_reads = [
+ (key, byte_range.start, byte_range.end)
+ for key, byte_range in source_gets
+ if hasattr(byte_range, "start")
+ ]
+ n_tasks = ((counts.shape[0] + shard_rows - 1) // shard_rows) * (
+ (counts.shape[1] + counts.chunks[1] - 1) // counts.chunks[1]
+ )
+ chunks_per_task = (shard_rows + counts.chunks[0] - 1) // counts.chunks[0]
+ assert len(index_reads) == n_tasks
+ assert len(chunk_reads) == len(set(chunk_reads))
+ assert len(chunk_reads) == n_tasks * chunks_per_task
+ np.testing.assert_array_equal(counts_t[:], values.T)
+
+
+def test_write_counts_t_overwrite_leaves_no_stale_chunks():
+ store = RecordingStore()
+ values = _dense_values(22, 7)
+ group, counts = _counts_array(values, store=store)
+ write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, 4),
+ )
+ stale = sorted(k for k in store._store_dict if k.startswith("RNA/countsT/c/"))
+
+ smaller = values[:6]
+ counts.resize(smaller.shape)
+ counts[:] = smaller
+ counts_t = write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, 4),
+ )
+
+ live = sorted(k for k in store._store_dict if k.startswith("RNA/countsT/c/"))
+ assert live == sorted(set(live))
+ assert len(live) < len(stale)
+ np.testing.assert_array_equal(counts_t[:], smaller.T)
+
+
+def test_write_counts_t_overlaps_tiles_within_worker_budget():
+ store = RecordingStore(delay=0.01)
+ values = _dense_values(24, 8)
+ group, counts = _counts_array(values, store=store)
+ store.reset()
+ write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, 4),
+ )
+ assert store.max_in_flight_for("set") > 1
+ assert store.max_in_flight_for("set") <= 4
+
+
+def test_write_counts_t_keeps_concurrency_within_worker_budget():
+ store = RecordingStore(delay=0.005)
+ values = _dense_values(24, 8)
+ group, counts = _counts_array(values, store=store)
+ store.reset()
+ write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, 2),
+ )
+ assert store.max_in_flight_for("set") <= 2
+
+
+def test_write_counts_t_restores_zarr_concurrency():
+ before = zarr.config.get("async.concurrency")
+ values = _dense_values(24, 8)
+
+ group, counts = _counts_array(values)
+ write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, 4),
+ )
+ assert zarr.config.get("async.concurrency") == before
+
+ store = RecordingStore(fail_on="RNA/countsT/c/1/1")
+ group, counts = _counts_array(values, store=store)
+ with pytest.raises(RuntimeError, match="injected write failure"):
+ write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, 4),
+ )
+ assert zarr.config.get("async.concurrency") == before
+
+
+def test_write_counts_t_stays_incomplete_when_a_tile_fails():
+ store = RecordingStore(fail_on="RNA/countsT/c/1/1")
+ values = _dense_values(24, 8)
+ group, counts = _counts_array(values, store=store)
+ with pytest.raises(RuntimeError, match="injected write failure"):
+ write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(8 * 1024**3, 4),
+ )
+ assert group["countsT"].attrs["complete"] is False
+
+
+def test_write_counts_t_rejects_oversized_tile():
+ values = _dense_values(24, 8)
+ group, counts = _counts_array(values)
+ with pytest.raises(MemoryError):
+ write_counts_t(
+ counts,
+ group,
+ resources=ResourceBudget(100, 8),
+ )
+
+
+def test_repack_preserves_counts_t_attrs(tmp_path):
+ from scarf.tools.repack_zarr import repack_store
+
+ src_path = tmp_path / "src.zarr"
+ dst_path = tmp_path / "dst.zarr"
+ root = zarr.open_group(str(src_path), mode="w")
+ values = np.arange(12, dtype=np.uint32).reshape(3, 4)
+ _write_small_assay(root, workspace=None, values=values)
+ assert root["RNA/countsT"].attrs["complete"] is True
+
+ repack_store(str(src_path), str(dst_path), profile="fast_local", shard_counts=False)
+ dst = zarr.open_group(str(dst_path), mode="r")
+ assert "countsT" in dst["RNA"]
+ assert dst["RNA/countsT"].attrs["complete"] is True
+ np.testing.assert_array_equal(dst["RNA/countsT"][:], values.T)
diff --git a/tests/test_cytebase.py b/tests/test_cytebase.py
new file mode 100644
index 00000000..08cfb008
--- /dev/null
+++ b/tests/test_cytebase.py
@@ -0,0 +1,428 @@
+import io
+import tarfile
+
+from huggingface_hub import BucketFile, BucketFolder
+import pytest
+
+
+def bucket_file(path: str, size: int) -> BucketFile:
+ return BucketFile(
+ type="file",
+ path=path,
+ size=size,
+ xetHash="test-hash",
+ mtime=None,
+ uploadedAt=None,
+ )
+
+
+def bucket_folder(path: str) -> BucketFolder:
+ return BucketFolder(
+ type="directory",
+ path=path,
+ uploadedAt=None,
+ )
+
+
+def test_list_repositories_is_sorted_and_anonymous(monkeypatch):
+ from scarf import cytebase
+
+ calls = []
+
+ def fake_list_bucket_tree(bucket_id, prefix=None, *, recursive=None, token=None):
+ calls.append((bucket_id, prefix, recursive, token))
+ return [
+ bucket_folder("scarf_docs"),
+ bucket_file("README.md", 3),
+ bucket_folder("cellxgene"),
+ ]
+
+ monkeypatch.setattr(cytebase, "list_bucket_tree", fake_list_bucket_tree)
+
+ assert cytebase.list_repositories() == ["cellxgene", "scarf_docs"]
+ assert calls == [("Nygen/cytebase", None, False, False)]
+
+
+def test_connect_returns_repository(monkeypatch):
+ from scarf import cytebase
+
+ monkeypatch.setattr(
+ cytebase,
+ "list_repositories",
+ lambda: ["cellxgene", "scarf_docs"],
+ )
+
+ repository = cytebase.connect("scarf_docs")
+
+ assert repository == cytebase.Repository("scarf_docs")
+
+
+def test_connect_rejects_unknown_repository(monkeypatch):
+ from scarf import cytebase
+
+ monkeypatch.setattr(cytebase, "list_repositories", lambda: ["scarf_docs"])
+
+ with pytest.raises(KeyError, match="Available repositories"):
+ cytebase.connect("missing")
+
+
+@pytest.mark.parametrize(
+ "name",
+ ["", ".", "..", "../outside", "/outside", r"..\outside"],
+)
+def test_connect_rejects_invalid_repository_name(name):
+ from scarf import cytebase
+
+ with pytest.raises(ValueError, match="Invalid repository name"):
+ cytebase.Repository(name)
+
+
+def test_repository_lists_datasets(monkeypatch):
+ from scarf import cytebase
+
+ calls = []
+
+ def fake_list_bucket_tree(bucket_id, prefix=None, *, recursive=None, token=None):
+ calls.append((bucket_id, prefix, recursive, token))
+ return [
+ bucket_folder("scarf_docs/zeta"),
+ bucket_file("scarf_docs/index.json", 2),
+ bucket_folder("scarf_docs/alpha"),
+ ]
+
+ monkeypatch.setattr(cytebase, "list_bucket_tree", fake_list_bucket_tree)
+
+ assert cytebase.Repository("scarf_docs").list_datasets() == ["alpha", "zeta"]
+ assert calls == [("Nygen/cytebase", "scarf_docs", False, False)]
+
+
+def test_repository_lists_relative_files(monkeypatch):
+ from scarf import cytebase
+
+ calls = []
+
+ def fake_list_bucket_tree(bucket_id, prefix=None, *, recursive=None, token=None):
+ calls.append((bucket_id, prefix, recursive, token))
+ return [
+ bucket_file("scarf_docs/alpha/matrix.mtx.gz", 4),
+ bucket_file("scarf_docs/alphabet/other.bin", 3),
+ bucket_file("scarf_docs/alpha/barcodes.tsv.gz", 2),
+ ]
+
+ monkeypatch.setattr(cytebase, "list_bucket_tree", fake_list_bucket_tree)
+
+ files = cytebase.Repository("scarf_docs").list_files(
+ "alpha",
+ recursive=False,
+ )
+
+ assert files == ["alpha/barcodes.tsv.gz", "alpha/matrix.mtx.gz"]
+ assert calls == [("Nygen/cytebase", "scarf_docs/alpha", False, False)]
+
+
+def test_repository_downloads_file_anonymously(monkeypatch, tmp_path):
+ from scarf import cytebase
+
+ file = bucket_file("scarf_docs/alpha/data.bin", 7)
+ monkeypatch.setattr(
+ cytebase,
+ "_bucket_files",
+ lambda repository, path, recursive: [file],
+ )
+ calls = []
+
+ def fake_download(bucket_id, files, *, raise_on_missing_files=False, token=None):
+ calls.append((bucket_id, files, raise_on_missing_files, token))
+ files[0][1].write_bytes(b"payload")
+
+ monkeypatch.setattr(cytebase, "download_bucket_files", fake_download)
+
+ downloaded = cytebase.Repository("scarf_docs").download(
+ "alpha/data.bin",
+ tmp_path,
+ )
+
+ destination = tmp_path / "alpha" / "data.bin"
+ assert downloaded == [destination]
+ assert destination.read_bytes() == b"payload"
+ assert calls[0][0] == "Nygen/cytebase"
+ assert calls[0][1][0][0] is file
+ assert calls[0][2:] == (False, False)
+ assert not any(
+ path.name.startswith(".cytebase-download-") for path in tmp_path.iterdir()
+ )
+
+
+def test_download_dataset_excludes_zarr_archive_by_default(monkeypatch, tmp_path):
+ from scarf import cytebase
+
+ source = bucket_file("scarf_docs/alpha/data.h5", 6)
+ archive = bucket_file("scarf_docs/alpha/data.zarr.tar.gz", 12)
+ monkeypatch.setattr(
+ cytebase,
+ "_bucket_files",
+ lambda repository, path, recursive: [archive, source],
+ )
+ requested = []
+
+ def fake_download(bucket_id, files, *, raise_on_missing_files=False, token=None):
+ requested.extend(file.path for file, _ in files)
+ for _, destination in files:
+ destination.write_bytes(b"source")
+
+ monkeypatch.setattr(cytebase, "download_bucket_files", fake_download)
+
+ dataset_path = cytebase.Repository("scarf_docs").download_dataset(
+ "alpha",
+ tmp_path,
+ )
+
+ assert dataset_path == tmp_path / "alpha"
+ assert requested == ["scarf_docs/alpha/data.h5"]
+ assert (dataset_path / "data.h5").read_bytes() == b"source"
+ assert not (dataset_path / "data.zarr.tar.gz").exists()
+
+
+def test_download_dataset_selects_and_extracts_zarr_archive(monkeypatch, tmp_path):
+ from scarf import cytebase
+
+ archive_buffer = io.BytesIO()
+ payload = b'{"zarr_format": 3, "node_type": "group"}'
+ with tarfile.open(fileobj=archive_buffer, mode="w:gz") as archive:
+ member = tarfile.TarInfo("data.zarr/zarr.json")
+ member.size = len(payload)
+ archive.addfile(member, io.BytesIO(payload))
+ archive_bytes = archive_buffer.getvalue()
+ source = bucket_file("scarf_docs/alpha/data.h5", 6)
+ zarr_archive = bucket_file(
+ "scarf_docs/alpha/data.zarr.tar.gz",
+ len(archive_bytes),
+ )
+ monkeypatch.setattr(
+ cytebase,
+ "_bucket_files",
+ lambda repository, path, recursive: [source, zarr_archive],
+ )
+
+ def fake_download(bucket_id, files, *, raise_on_missing_files=False, token=None):
+ assert [file.path for file, _ in files] == ["scarf_docs/alpha/data.zarr.tar.gz"]
+ files[0][1].write_bytes(archive_bytes)
+
+ monkeypatch.setattr(cytebase, "download_bucket_files", fake_download)
+
+ dataset_path = cytebase.Repository("scarf_docs").download_dataset(
+ "alpha",
+ tmp_path,
+ zarr=True,
+ )
+
+ assert (dataset_path / "data.zarr.tar.gz").read_bytes() == archive_bytes
+ assert (dataset_path / "data.zarr" / "zarr.json").read_bytes() == payload
+
+
+def test_download_dataset_reports_missing_zarr(monkeypatch, tmp_path):
+ from scarf import cytebase
+
+ monkeypatch.setattr(
+ cytebase,
+ "_bucket_files",
+ lambda repository, path, recursive: [
+ bucket_file("scarf_docs/alpha/data.h5", 6)
+ ],
+ )
+
+ with pytest.raises(FileNotFoundError, match="No Zarr archive"):
+ cytebase.Repository("scarf_docs").download_dataset(
+ "alpha",
+ tmp_path,
+ zarr=True,
+ )
+
+
+def test_download_dataset_reports_unknown_name(monkeypatch, tmp_path):
+ from scarf import cytebase
+
+ monkeypatch.setattr(cytebase, "_bucket_files", lambda *args, **kwargs: [])
+ monkeypatch.setattr(
+ cytebase.Repository,
+ "list_datasets",
+ lambda self: ["alpha", "beta"],
+ )
+
+ with pytest.raises(KeyError, match="Available datasets"):
+ cytebase.Repository("scarf_docs").download_dataset("missing", tmp_path)
+
+
+@pytest.mark.parametrize(
+ "path",
+ ["", ".", "..", "../outside", "/outside", r"..\outside", "a//b"],
+)
+def test_repository_rejects_invalid_remote_paths(path, tmp_path):
+ from scarf import cytebase
+
+ with pytest.raises(ValueError, match="Invalid download path"):
+ cytebase.Repository("scarf_docs").download(path, tmp_path)
+
+
+def test_download_preserves_existing_file_when_transfer_fails(monkeypatch, tmp_path):
+ from scarf import cytebase
+
+ file = bucket_file("scarf_docs/alpha/data.bin", 3)
+ monkeypatch.setattr(
+ cytebase,
+ "_bucket_files",
+ lambda repository, path, recursive: [file],
+ )
+ destination = tmp_path / "alpha" / "data.bin"
+ destination.parent.mkdir()
+ destination.write_bytes(b"existing")
+
+ def fail_download(bucket_id, files, *, raise_on_missing_files=False, token=None):
+ files[0][1].write_bytes(b"new")
+ raise RuntimeError("transfer failed")
+
+ monkeypatch.setattr(cytebase, "download_bucket_files", fail_download)
+
+ with pytest.raises(RuntimeError, match="transfer failed"):
+ cytebase.Repository("scarf_docs").download("alpha/data.bin", tmp_path)
+
+ assert destination.read_bytes() == b"existing"
+ assert sorted(path.name for path in tmp_path.iterdir()) == ["alpha"]
+
+
+def test_download_rejects_size_mismatch_and_preserves_existing(
+ monkeypatch,
+ tmp_path,
+):
+ from scarf import cytebase
+
+ file = bucket_file("scarf_docs/alpha/data.bin", 4)
+ monkeypatch.setattr(
+ cytebase,
+ "_bucket_files",
+ lambda repository, path, recursive: [file],
+ )
+ destination = tmp_path / "alpha" / "data.bin"
+ destination.parent.mkdir()
+ destination.write_bytes(b"existing")
+
+ def incomplete_download(
+ bucket_id,
+ files,
+ *,
+ raise_on_missing_files=False,
+ token=None,
+ ):
+ files[0][1].write_bytes(b"new")
+
+ monkeypatch.setattr(cytebase, "download_bucket_files", incomplete_download)
+
+ with pytest.raises(OSError, match="Downloaded 3 bytes"):
+ cytebase.Repository("scarf_docs").download("alpha/data.bin", tmp_path)
+
+ assert destination.read_bytes() == b"existing"
+
+
+def test_download_rejects_unsafe_tar_member(monkeypatch, tmp_path):
+ from scarf import cytebase
+
+ archive_buffer = io.BytesIO()
+ payload = b"unsafe"
+ with tarfile.open(fileobj=archive_buffer, mode="w:gz") as archive:
+ member = tarfile.TarInfo("../outside.txt")
+ member.size = len(payload)
+ archive.addfile(member, io.BytesIO(payload))
+ archive_bytes = archive_buffer.getvalue()
+ file = bucket_file(
+ "scarf_docs/alpha/data.tar.gz",
+ len(archive_bytes),
+ )
+ monkeypatch.setattr(
+ cytebase,
+ "_bucket_files",
+ lambda repository, path, recursive: [file],
+ )
+
+ def fake_download(bucket_id, files, *, raise_on_missing_files=False, token=None):
+ files[0][1].write_bytes(archive_bytes)
+
+ monkeypatch.setattr(cytebase, "download_bucket_files", fake_download)
+
+ with pytest.raises(ValueError, match="Unsafe tar archive member"):
+ cytebase.Repository("scarf_docs").download("alpha/data.tar.gz", tmp_path)
+
+ assert not (tmp_path / "outside.txt").exists()
+
+
+def test_open_zarr_uses_anonymous_read_only_store(monkeypatch):
+ from scarf import cytebase
+ import zarr
+ from zarr.storage import FsspecStore
+
+ calls = []
+ store = object()
+ group = object()
+
+ def fake_from_url(url, storage_options=None, read_only=False):
+ calls.append(("store", url, storage_options, read_only))
+ return store
+
+ def fake_open_group(*, store, mode):
+ calls.append(("group", store, mode))
+ return group
+
+ monkeypatch.setattr(FsspecStore, "from_url", fake_from_url)
+ monkeypatch.setattr(zarr, "open_group", fake_open_group)
+
+ result = cytebase.Repository("cellxgene").open_zarr("atlas/data.zarr")
+
+ assert result is group
+ assert calls == [
+ (
+ "store",
+ "hf://buckets/Nygen/cytebase/cellxgene/atlas/data.zarr",
+ {"token": False},
+ True,
+ ),
+ ("group", store, "r"),
+ ]
+
+
+@pytest.mark.integration
+def test_live_bucket_catalog_is_public():
+ from scarf import cytebase
+
+ assert "scarf_docs" in cytebase.list_repositories()
+ repository = cytebase.connect("scarf_docs")
+ datasets = set(repository.list_datasets())
+ assert {
+ "annotations",
+ "bastidas-ponce_4K_pancreas-d15_rnaseq",
+ "kang_14K_ifnb-pbmc_rnaseq",
+ "kang_15K_pbmc_rnaseq",
+ "tenx_10K_pbmc-v1_atacseq",
+ "tenx_5K_pbmc_rnaseq",
+ "tenx_8K_pbmc_citeseq",
+ } <= datasets
+ assert "tenx_5K_pbmc_rnaseq_legacy_master" in datasets
+ files = repository.list_files("tenx_5K_pbmc_rnaseq")
+ assert {
+ "tenx_5K_pbmc_rnaseq/data.h5",
+ "tenx_5K_pbmc_rnaseq/data.zarr.tar.gz",
+ "tenx_5K_pbmc_rnaseq/manifest.json",
+ } <= set(files)
+ assert any(name.startswith("tenx_5K_pbmc_rnaseq/data.zarr/") for name in files)
+
+
+@pytest.mark.integration
+def test_live_zarr_archive_download(tmp_path):
+ from scarf import cytebase
+
+ dataset_path = cytebase.connect("scarf_docs").download_dataset(
+ "tenx_5K_pbmc_rnaseq",
+ tmp_path,
+ zarr=True,
+ )
+
+ assert (dataset_path / "data.zarr.tar.gz").is_file()
+ assert (dataset_path / "data.zarr").is_dir()
diff --git a/tests/test_dataset_fingerprint.py b/tests/test_dataset_fingerprint.py
new file mode 100644
index 00000000..b235b144
--- /dev/null
+++ b/tests/test_dataset_fingerprint.py
@@ -0,0 +1,34 @@
+import pytest
+
+from scarf.storage.artifacts import ValueFingerprintBuilder
+
+
+def test_dataset_fingerprint_is_created_once_and_left_dormant(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ assay = datastore_ephemeral.get_assay("RNA")
+ if "dataset_fingerprint" in assay.attrs:
+ del assay.attrs["dataset_fingerprint"]
+
+ fingerprint = datastore_ephemeral._ensure_dataset_fingerprint("RNA")
+ assert assay.attrs["dataset_fingerprint"] == fingerprint
+
+ def fail_if_recomputed(*_args, **_kwargs):
+ raise AssertionError("dataset_fingerprint must remain dormant")
+
+ monkeypatch.setattr(ValueFingerprintBuilder, "update_array", fail_if_recomputed)
+ assert datastore_ephemeral._ensure_dataset_fingerprint("RNA") == fingerprint
+
+
+def test_read_only_store_does_not_create_dataset_fingerprint(
+ datastore_ephemeral,
+) -> None:
+ assay = datastore_ephemeral.get_assay("RNA")
+ if "dataset_fingerprint" in assay.attrs:
+ del assay.attrs["dataset_fingerprint"]
+ datastore_ephemeral.zarr_mode = "r"
+
+ with pytest.raises(PermissionError, match="cannot be stored read-only"):
+ datastore_ephemeral._ensure_dataset_fingerprint("RNA")
+ assert "dataset_fingerprint" not in assay.attrs
diff --git a/tests/test_datastore.py b/tests/test_datastore.py
new file mode 100644
index 00000000..7da7b686
--- /dev/null
+++ b/tests/test_datastore.py
@@ -0,0 +1,1133 @@
+from importlib import import_module, util
+
+import numpy as np
+import pandas as pd
+import pytest
+import zarr
+from scipy.spatial import procrustes
+from sklearn.metrics import adjusted_rand_score
+
+import scarf
+import scarf.plotting as splt
+from scarf.assay import Assay
+from scarf.datastore.datastore import DataStore
+from scarf.datastore.mapping_datastore import MappingDatastore
+from scarf.metadata import MetaData
+from scarf.storage.artifacts import ArtifactRef
+from scarf.trajectory.results import (
+ PseudotimeAggregationResult,
+ PseudotimeScoreResult,
+)
+from scarf.utils.arrays import array_digest
+from scarf.writers import create_cell_data, create_zarr_count_assay
+from tests.fixtures_datastore import build_neighbourhood_graph
+from tests.store_probes import RecordingStore
+
+from . import full_path
+
+
+_QC_VALUES = np.array(
+ [
+ [5, 0, 1, 0, 0, 2],
+ [0, 3, 0, 4, 0, 0],
+ [1, 2, 0, 0, 0, 0],
+ [0, 0, 0, 5, 0, 1],
+ [0, 0, 0, 0, 0, 0],
+ [2, 1, 3, 1, 0, 0],
+ ],
+ dtype=np.uint32,
+)
+_QC_FEATURE_NAMES = np.array(["MT-CO1", "RPS3", "GENE_A", "RPL5", "ZERO", "GENE_B"])
+
+
+def _qc_store() -> tuple[RecordingStore, int]:
+ store = RecordingStore()
+ root = zarr.open_group(store=store, mode="w")
+ n_cells, n_features = _QC_VALUES.shape
+ create_cell_data(
+ root,
+ None,
+ ids=np.array([f"c{i}" for i in range(n_cells)]),
+ names=np.array([f"c{i}" for i in range(n_cells)]),
+ profile="fast_local",
+ )
+ counts = create_zarr_count_assay(
+ root,
+ "RNA",
+ None,
+ n_cells,
+ feat_ids=np.array([f"f{i}" for i in range(n_features)]),
+ feat_names=_QC_FEATURE_NAMES,
+ dtype="uint32",
+ profile="fast_local",
+ targetChunkBytes=16,
+ targetShardBytes=48,
+ )
+ counts[:] = _QC_VALUES
+ assert counts.shards is not None
+ expected_reads = int(np.ceil(n_cells / counts.shards[0]))
+ store.reset()
+ return store, expected_reads
+
+
+def _open_qc_store(store: RecordingStore, **overrides) -> DataStore:
+ options = {
+ "default_assay": "RNA",
+ "min_features_per_cell": 0,
+ "min_cells_per_feature": 0,
+ "mito_pattern": "^MT-",
+ "ribo_pattern": "^(RPS|RPL)",
+ "nthreads": 1,
+ "zarrProfile": "fast_local",
+ }
+ options.update(overrides)
+ return DataStore(store, **options)
+
+
+def _count_chunk_gets(store: RecordingStore) -> list[str]:
+ return [
+ key for operation, key in store.chunk_ops("RNA/counts/c/") if operation == "get"
+ ]
+
+
+def _assert_one_counts_stream(store: RecordingStore, expected_reads: int) -> None:
+ gets = _count_chunk_gets(store)
+ assert len(gets) == expected_reads
+ assert len(set(gets)) == expected_reads
+
+
+def test_initialization_fuses_qc_stats_in_one_counts_stream():
+ store, expected_reads = _qc_store()
+ datastore = _open_qc_store(store)
+
+ expected_n_counts = _QC_VALUES.sum(axis=1).astype(np.float64)
+ expected_n_features = (_QC_VALUES > 0).sum(axis=1).astype(np.float64)
+ expected_n_cells = (_QC_VALUES > 0).sum(axis=0)
+ with np.errstate(divide="ignore", invalid="ignore"):
+ expected_mito = 100 * _QC_VALUES[:, 0] / expected_n_counts
+ expected_ribo = 100 * _QC_VALUES[:, [1, 3]].sum(axis=1) / expected_n_counts
+
+ np.testing.assert_array_equal(
+ datastore.cells.fetch_all("RNA_nCounts"),
+ expected_n_counts,
+ )
+ np.testing.assert_array_equal(
+ datastore.cells.fetch_all("RNA_nFeatures"),
+ expected_n_features,
+ )
+ np.testing.assert_allclose(
+ datastore.cells.fetch_all("RNA_percentMito"),
+ expected_mito,
+ equal_nan=True,
+ )
+ np.testing.assert_allclose(
+ datastore.cells.fetch_all("RNA_percentRibo"),
+ expected_ribo,
+ equal_nan=True,
+ )
+ np.testing.assert_array_equal(
+ datastore.RNA.feats.fetch_all("nCells"),
+ expected_n_cells,
+ )
+ np.testing.assert_array_equal(
+ datastore.RNA.feats.fetch_all("dropOuts"),
+ _QC_VALUES.shape[0] - expected_n_cells,
+ )
+ np.testing.assert_array_equal(
+ datastore.RNA.feats.fetch_all("I"),
+ expected_n_cells > 0,
+ )
+ _assert_one_counts_stream(store, expected_reads)
+
+ for column in (
+ "RNA_nCounts",
+ "RNA_nFeatures",
+ "RNA_percentMito",
+ "RNA_percentRibo",
+ ):
+ assert "source_artifact" not in datastore.zw["cellData"][column].attrs
+ for column in ("nCells", "dropOuts"):
+ assert "source_artifact" not in datastore.RNA.z["featureData"][column].attrs
+
+
+def test_cached_initialization_is_read_and_write_free():
+ store, _ = _qc_store()
+ _open_qc_store(store)
+ store.reset()
+
+ _open_qc_store(store, zarr_mode="r")
+
+ assert _count_chunk_gets(store) == []
+ assert [operation for operation, _ in store.ops if operation == "set"] == []
+
+
+def test_partial_initialization_preserves_hvg_cache():
+ store, expected_reads = _qc_store()
+ datastore = _open_qc_store(store)
+ datastore.RNA.set_feature_stats("I")
+ subset_hash = datastore.RNA.z["summary_stats_I"].attrs["subset_hash"]
+ feature_index = datastore.RNA.feats.fetch_all("I")
+ datastore.cells.drop("RNA_nFeatures")
+ store.reset()
+
+ reopened = _open_qc_store(store)
+
+ _assert_one_counts_stream(store, expected_reads)
+ assert reopened.RNA.z["summary_stats_I"].attrs["subset_hash"] == subset_hash
+ np.testing.assert_array_equal(
+ reopened.RNA.feats.fetch_all("I"),
+ feature_index,
+ )
+ reopened.RNA._streaming_feature_stats = lambda *_: pytest.fail(
+ "valid feature statistics should be reused"
+ )
+ reopened.RNA.set_feature_stats("I")
+
+
+def test_percent_cache_remains_attribute_only():
+ store, expected_reads = _qc_store()
+ datastore = _open_qc_store(store)
+ datastore.cells.drop("RNA_percentMito")
+ store.reset()
+
+ cached = _open_qc_store(store)
+
+ assert _count_chunk_gets(store) == []
+ assert "RNA_percentMito" not in cached.cells.columns
+ store.reset()
+
+ refreshed = _open_qc_store(store, mito_pattern="^MT-|^GENE_A$")
+
+ _assert_one_counts_stream(store, expected_reads)
+ assert "RNA_percentMito" in refreshed.cells.columns
+
+
+def test_partial_feature_props_are_recomputed_together():
+ store, expected_reads = _qc_store()
+ datastore = _open_qc_store(store)
+ datastore.RNA.feats.insert(
+ "nCells",
+ np.zeros(_QC_VALUES.shape[1], dtype=np.int64),
+ overwrite=True,
+ )
+ datastore.RNA.feats.drop("dropOuts")
+ feature_index = datastore.RNA.feats.fetch_all("I")
+ store.reset()
+
+ reopened = _open_qc_store(store)
+
+ _assert_one_counts_stream(store, expected_reads)
+ expected_n_cells = (_QC_VALUES > 0).sum(axis=0)
+ np.testing.assert_array_equal(
+ reopened.RNA.feats.fetch_all("nCells"),
+ expected_n_cells,
+ )
+ np.testing.assert_array_equal(
+ reopened.RNA.feats.fetch_all("I"),
+ feature_index,
+ )
+ expected_dropouts = _QC_VALUES.shape[0] - expected_n_cells[feature_index]
+ np.testing.assert_array_equal(
+ reopened.RNA.feats.fetch("dropOuts"),
+ expected_dropouts,
+ )
+
+
+def test_standalone_assay_keeps_eager_feature_initialization():
+ store, _ = _qc_store()
+ root = zarr.open_group(store=store, mode="r+")
+ assay = Assay(
+ root,
+ None,
+ "RNA",
+ MetaData(root["cellData"]),
+ nthreads=1,
+ min_cells_per_feature=0,
+ )
+
+ assert {"nCells", "dropOuts"}.issubset(assay.feats.columns)
+ assert assay._deferred_min_cells_per_feature is None
+
+
+class TestToyDataStore:
+ def test_toy_crdir_metadata(self, toy_crdir_ds):
+ assert np.all(
+ toy_crdir_ds.RNA.feats.fetch_all("ids") == ["g1", "g2", "g3", "g4"]
+ )
+ assert np.all(toy_crdir_ds.ADT.feats.fetch_all("ids") == ["a1", "a2", "a3"])
+ assert np.all(toy_crdir_ds.HTO.feats.fetch_all("ids") == ["h1"])
+ assert np.all(toy_crdir_ds.cells.fetch_all("ids") == ["b1", "b2", "b3"])
+
+ def test_toy_crdir_rawdata(self, toy_crdir_ds):
+ assert np.all(
+ toy_crdir_ds.RNA.rawData.compute()
+ == [[5, 0, 0, 2], [3, 3, 0, 7], [3, 3, 0, 7]]
+ )
+ assert np.all(
+ toy_crdir_ds.ADT.rawData.compute()
+ == [[30, 40, 30], [30, 50, 20], [0, 50, 20]]
+ )
+ assert np.all(toy_crdir_ds.HTO.rawData.compute() == [[200], [100], [100]])
+
+
+class TestDataStore:
+ def test_init_wrong_zarr_mode(self, tmp_path):
+ import pytest
+ import tarfile
+
+ from scarf.datastore.datastore import DataStore
+
+ fn = full_path("1K_pbmc_citeseq.zarr.tar.gz")
+ out_fn = tmp_path / "1K_pbmc_citeseq.zarr"
+ with tarfile.open(fn, "r:gz") as tar:
+ tar.extractall(out_fn, filter="data")
+ with pytest.raises(ValueError):
+ DataStore(str(out_fn), zarr_mode="wrong", default_assay="RNA")
+
+ def test_auto_filter_cells(self, datastore_ephemeral):
+ assert (
+ datastore_ephemeral.auto_filter_cells(
+ attrs=["nCounts", "nFeatures", "non_existing_column"],
+ show_qc_plots=False,
+ )
+ is None
+ )
+
+ def test_auto_filter_cells_uses_modern_distribution(
+ self, datastore_ephemeral, monkeypatch
+ ):
+ calls = []
+ active_before = int(datastore_ephemeral.cells.fetch_all("I").sum())
+
+ def record_distribution(store, *, keys, cell_key=None, **kwargs):
+ if cell_key is None:
+ n_cells = store.cells.N
+ else:
+ n_cells = len(store.cells.active_index(cell_key))
+ calls.append(
+ {
+ "keys": list(keys),
+ "cell_key": cell_key,
+ "n_cells": n_cells,
+ "active_i": int(store.cells.fetch_all("I").sum()),
+ **kwargs,
+ }
+ )
+
+ monkeypatch.setattr(splt, "distribution", record_distribution)
+ datastore_ephemeral.auto_filter_cells(
+ attrs=["RNA_nCounts", "RNA_nFeatures"],
+ show_qc_plots=True,
+ )
+
+ assert [call["title"] for call in calls] == [
+ "Pre-filtering distribution",
+ "Post-filtering distribution",
+ ]
+ assert [call["color"] for call in calls] == ["steelblue", "coral"]
+ assert [call["cell_key"] for call in calls] == [None, "I"]
+ assert all(call["keys"] == ["RNA_nCounts", "RNA_nFeatures"] for call in calls)
+ assert all(call["show"] is True for call in calls)
+ # Filtering happens before both plots; pre includes filtered-out cells.
+ assert calls[0]["active_i"] < active_before
+ assert calls[0]["n_cells"] == datastore_ephemeral.cells.N
+ assert calls[1]["n_cells"] == calls[1]["active_i"]
+ assert calls[0]["n_cells"] >= calls[1]["n_cells"]
+
+ def test_auto_filter_cells_skips_qc_when_no_attrs(
+ self, datastore_ephemeral, monkeypatch
+ ):
+ monkeypatch.setattr(
+ splt,
+ "distribution",
+ lambda *args, **kwargs: (_ for _ in ()).throw(
+ AssertionError("distribution should not be called")
+ ),
+ )
+ assert (
+ datastore_ephemeral.auto_filter_cells(
+ attrs=["missing_a", "missing_b"],
+ show_qc_plots=True,
+ )
+ is None
+ )
+
+ def test_assay_names_tolerates_repeated_group_listings(
+ self, datastore_ephemeral, monkeypatch
+ ):
+ # Object-store listings can yield the same group more than once.
+ root = datastore_ephemeral.z
+ keys = list(root.group_keys())
+ expected = sorted(
+ name
+ for name in dict.fromkeys(keys)
+ if "is_assay" in root[name].attrs.keys()
+ )
+ assert expected
+
+ class RepeatingGroup:
+ def group_keys(self):
+ return iter(keys + keys + keys)
+
+ def __getitem__(self, key):
+ return root[key]
+
+ monkeypatch.setattr(
+ type(datastore_ephemeral),
+ "zw",
+ property(lambda self: RepeatingGroup()),
+ )
+ assert datastore_ephemeral.assay_names == expected
+
+ def test_filter_cells(self, datastore_ephemeral):
+ assert (
+ datastore_ephemeral.filter_cells(
+ attrs=["nCounts", "nFeatures", "non_existing_column"],
+ lows=[None, None, None],
+ highs=[None, None, None],
+ reset_previous=True,
+ )
+ is None
+ )
+ # still doesn't access `if j is None:` cases for j and k
+
+ def test_graph_indices(self, graph_artifacts, datastore):
+ a = np.load(full_path("knn_indices.npy"))
+ b = datastore.z[graph_artifacts]["indices"][:]
+ assert np.array_equal(a, b)
+
+ def test_graph_distances(self, graph_artifacts, datastore):
+ a = np.sqrt(np.load(full_path("knn_distances.npy")))
+ b = datastore.z[graph_artifacts]["distances"][:]
+ np.testing.assert_allclose(a, b, rtol=0, atol=1e-3)
+
+ def test_graph_weights(self, graph_artifacts, datastore):
+ from umap.umap_ import compute_membership_strengths, smooth_knn_dist
+
+ indices = np.load(full_path("knn_indices.npy"))
+ distances = np.sqrt(np.load(full_path("knn_distances.npy"))).astype(np.float32)
+ sigmas, rhos = smooth_knn_dist(
+ distances,
+ k=indices.shape[1],
+ local_connectivity=1.0,
+ bandwidth=1.5,
+ )
+ _, _, expected, _ = compute_membership_strengths(
+ indices,
+ distances,
+ sigmas,
+ rhos,
+ )
+ a = np.asarray(expected, dtype=np.float32)
+ a = a[a > 0]
+ state = datastore.get_assay_state("RNA")
+ assert state is not None and state.connectivity_map is not None
+ graph_path = datastore.inspect_artifact(state.connectivity_map).path
+ b = datastore.z[graph_path]["weights"][:]
+ np.testing.assert_allclose(a, b, rtol=0, atol=1e-5)
+
+ def test_atac_graph_indices(self, make_atac_graph, atac_datastore):
+ expected = np.load(full_path("atac_knn_indices.npy"))
+ observed = np.asarray(atac_datastore.z[make_atac_graph]["indices"][:])
+ row_ids = np.arange(observed.shape[0])[:, None]
+ overlap = np.mean(
+ [
+ len(set(expected_row) & set(observed_row)) / len(expected_row)
+ for expected_row, observed_row in zip(
+ expected,
+ observed,
+ strict=True,
+ )
+ ]
+ )
+
+ assert expected.shape == observed.shape
+ assert observed.min() >= 0
+ assert observed.max() < observed.shape[0]
+ assert not np.any(observed == row_ids)
+ assert all(np.unique(row).size == row.size for row in observed)
+ assert overlap >= 0.15
+
+ def test_atac_graph_distances(self, make_atac_graph, atac_datastore):
+ from scipy.stats import spearmanr
+
+ expected = np.sqrt(np.load(full_path("atac_knn_distances.npy")))
+ observed = np.asarray(atac_datastore.z[make_atac_graph]["distances"][:])
+
+ assert expected.shape == observed.shape
+ assert np.isfinite(observed).all()
+ assert np.all(observed > 0)
+ assert np.all(np.diff(observed, axis=1) >= 0)
+ # Total-count TF changes cell-specific scale, so compare each local
+ # distance profile after removing that scale.
+ expected_profile = expected / expected.mean(axis=1, keepdims=True)
+ observed_profile = observed / observed.mean(axis=1, keepdims=True)
+ assert (
+ spearmanr(expected_profile.ravel(), observed_profile.ravel()).statistic
+ >= 0.8
+ )
+ relative_mae = np.mean(np.abs(expected_profile - observed_profile)) / np.mean(
+ expected_profile
+ )
+ assert relative_mae < 0.1
+
+ def test_leiden_values(self, leiden_clustering, cell_attrs):
+ assert len(set(leiden_clustering)) == 10
+ expected = cell_attrs["RNA_leiden_cluster"].values
+ agreement = adjusted_rand_score(expected, leiden_clustering)
+ assert agreement == pytest.approx(0.8850162225, abs=1e-6)
+
+ def test_paris_values(self, paris_clustering):
+ labels = np.asarray(paris_clustering, dtype=np.int32)
+ assert labels.ndim == 1
+ assert np.array_equal(np.unique(labels), np.arange(1, 11))
+ assert array_digest(labels) == ("306518457a86d4a4fe0d49a7cfeaeb39")
+
+ def test_paris_adaptive_values(self, paris_clustering_auto):
+ labels = np.asarray(paris_clustering_auto, dtype=np.int32)
+ unique = np.unique(labels)
+ assert labels.ndim == 1
+ assert np.array_equal(unique, np.arange(1, unique.size + 1))
+ assert np.bincount(labels)[1:].min() >= 10
+ assert array_digest(labels) == ("aeb885f85abcb41ec2b970f7fe1aaca2")
+
+ def test_run_cell_cycle_scoring(self, cell_cycle_scoring, cell_attrs):
+ assert np.array_equal(
+ cell_cycle_scoring, cell_attrs["RNA_cell_cycle_phase"].values
+ )
+
+ def test_umap_values(self, umap, cell_attrs):
+ precalc_umap = cell_attrs[["RNA_UMAP1", "RNA_UMAP2"]].values
+ assert umap.shape == precalc_umap.shape
+ _, _, disparity = procrustes(precalc_umap, umap)
+ assert disparity < 0.25
+
+ def test_get_markers(self, marker_search, paris_clustering, datastore):
+ markers = datastore.get_markers(group_key="RNA_cluster", group_id=1)
+
+ assert not markers.empty
+ assert set(markers.group_id) == {1}
+ assert markers.feature_name.is_unique
+ assert {"score", "fold_change", "p_value"}.issubset(markers.columns)
+ assert np.isfinite(markers.score).all()
+ assert np.isfinite(markers.fold_change).all()
+ assert markers.p_value.between(0, 1).all()
+
+ def test_get_markers_all_groups(self, marker_search, paris_clustering, datastore):
+ all_markers = datastore.get_markers(group_key="RNA_cluster", group_id=None)
+ assert "group_id" in all_markers.columns
+ groups = set(datastore.cells.fetch("RNA_cluster", key="I"))
+ assert set(all_markers["group_id"]).issubset(groups)
+ one = datastore.get_markers(group_key="RNA_cluster", group_id=1)
+ from_all = all_markers[all_markers["group_id"] == 1].reset_index(drop=True)
+ assert len(from_all) == len(one)
+ assert from_all["feature_name"].equals(one["feature_name"])
+
+ def test_export_markers_to_csv(
+ self, marker_search, paris_clustering, datastore, tmp_path
+ ):
+ out_file = str(tmp_path / "test_values_markers.csv")
+ datastore.export_markers_to_csv(group_key="RNA_cluster", csv_filename=out_file)
+ markers = pd.read_csv(out_file)
+ groups = sorted(np.unique(paris_clustering))
+ assert list(markers.columns) == [str(group) for group in groups]
+ for group in groups:
+ expected = datastore.get_markers(
+ group_key="RNA_cluster",
+ group_id=int(group),
+ ).feature_name.reset_index(drop=True)
+ actual = markers[str(group)].dropna().reset_index(drop=True)
+ assert actual.equals(expected)
+
+ def test_repr(self, datastore):
+ # TODO: Test if the expected values are printed
+ print(datastore)
+
+ def test_get_imputed(self, datastore):
+ # TODO: Test the output values
+ values = datastore.get_imputed(feature_name="CD4")
+ assert values.shape == datastore.cells.fetch("I").shape
+
+ def test_mean_features(self, datastore):
+ import pytest
+
+ names = list(datastore.RNA.feats.fetch("names", key="I")[:3])
+ values = datastore.RNA.mean_features(names)
+ active = datastore.cells.active_index("I")
+ feat_idx = datastore.RNA.feats.get_index_by(names, "names", None)
+ expected = (
+ datastore.RNA.normed(cell_idx=active, feat_idx=np.sort(feat_idx))
+ .mean(axis=1)
+ .compute()
+ )
+ assert values.shape == (len(active),)
+ np.testing.assert_allclose(values, expected)
+ with pytest.raises(ValueError, match="not found"):
+ datastore.RNA.mean_features(["__missing_feature__"])
+ skipped = datastore.RNA.mean_features(
+ [names[0], "__missing_feature__"],
+ missing="skip",
+ )
+ assert skipped.shape == (len(active),)
+
+ def test_run_doublet_detection(
+ self,
+ graph_artifacts,
+ paris_clustering,
+ datastore,
+ ):
+ score_col = datastore.run_doublet_detection(
+ cluster_key="RNA_cluster", simulation_ratio=0.5, random_seed=1
+ )
+ col = "RNA_doublet_score"
+ assert score_col == col
+ assert col in datastore.cells.columns
+ scores = datastore.cells.fetch(col)
+ n_active = datastore.cells.active_index("I").shape[0]
+ assert scores.shape[0] == n_active
+ assert not np.isnan(scores).any()
+ assert scores.min() >= 0.0 and scores.max() <= 1.0
+ raw_ref = datastore.zw["cellData"][col].attrs["source_artifact"]
+ assert ArtifactRef.from_dict(raw_ref).kind == "doublet_score"
+ # scratch column used for smoothing should be removed
+ assert "RNA_doublet_score__raw" not in datastore.cells.columns
+ # the temporary query owns and removes its projection
+ assert not datastore.list_artifacts(
+ kind="projection",
+ from_assay="RNA",
+ )
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.connectivity_map is not None
+ reference = datastore.get_mapping_reference()
+ assert reference.neighbors == state.neighbors
+ assert reference.symphony_state is None
+
+ def test_run_doublet_detection_bad_cluster_key(
+ self,
+ graph_artifacts,
+ datastore,
+ ):
+ import pytest
+
+ with pytest.raises(ValueError):
+ datastore.run_doublet_detection(cluster_key="not_a_column")
+
+ def test_run_pseudotime_scoring(self, pseudotime_scoring, cell_attrs):
+ diff = pseudotime_scoring - cell_attrs["RNA_pseudotime"].values
+ assert np.all(np.abs(diff) < 0.08)
+
+ def test_run_pseudotime_scoring_current_contract(
+ self,
+ leiden_clustering,
+ datastore_ephemeral,
+ monkeypatch,
+ ):
+ datastore_ephemeral.auto_filter_cells(show_qc_plots=False)
+ datastore_ephemeral.mark_hvgs(
+ from_assay="RNA",
+ cell_key="I",
+ top_n=100,
+ hvg_key_name="hvgs",
+ show_plot=False,
+ bin_strategy="fixed",
+ )
+ build_neighbourhood_graph(
+ datastore_ephemeral,
+ feat_key="hvgs",
+ local_cache=False,
+ )
+ datastore_ephemeral.cells.insert(
+ "RNA_leiden_cluster",
+ leiden_clustering,
+ key="I",
+ overwrite=True,
+ )
+ arguments = {
+ "source_sink_key": "RNA_leiden_cluster",
+ "sources": [6],
+ "sinks": [3],
+ "n_singular_vals": 10,
+ }
+ result = datastore_ephemeral.run_pseudotime_scoring(
+ **arguments,
+ label="reliability_test",
+ )
+
+ output_key = "RNA_reliability_test"
+ validity_key = f"{output_key}__valid"
+ assert isinstance(result, PseudotimeScoreResult)
+ assert result.pseudotime_key == output_key
+ assert result.validity_key == validity_key
+ assert result.values.shape == result.valid.shape
+ np.testing.assert_array_equal(result.valid, np.ones_like(result.valid))
+ assert validity_key in datastore_ephemeral.cells.columns
+ pseudotime_ref = ArtifactRef.from_dict(
+ datastore_ephemeral.zw["cellData"][output_key].attrs["source_artifact"]
+ )
+ assert pseudotime_ref.kind == "pseudotime"
+ assert (
+ ArtifactRef.from_dict(
+ datastore_ephemeral.zw["cellData"][validity_key].attrs[
+ "source_artifact"
+ ]
+ )
+ == pseudotime_ref
+ )
+ values = datastore_ephemeral.cells.fetch(output_key, key=validity_key)
+ assert np.isfinite(values).all()
+ assert values.min() >= 0.0
+ assert values.max() <= 1.0
+
+ from scarf.datastore._operations import trajectory as trajectory_operations
+
+ def fail_if_recomputed(*_args, **_kwargs):
+ raise AssertionError("pseudotime should have been reused")
+
+ monkeypatch.setattr(
+ trajectory_operations,
+ "_truncated_pba_potential_impl",
+ fail_if_recomputed,
+ )
+ cached = datastore_ephemeral.run_pseudotime_scoring(
+ **arguments,
+ label="reliability_cached",
+ )
+ cached_ref = ArtifactRef.from_dict(
+ datastore_ephemeral.zw["cellData"][cached.pseudotime_key].attrs[
+ "source_artifact"
+ ]
+ )
+ assert cached_ref == pseudotime_ref
+ np.testing.assert_allclose(cached.values, result.values)
+
+ def test_run_pseudotime_marker_search(
+ self,
+ pseudotime_markers,
+ datastore,
+ ):
+ precalc_markers = pd.read_csv(
+ full_path("pseudotime_markers_r_values.csv"), index_col=0
+ )
+ assert np.all(precalc_markers.index == pseudotime_markers.index)
+ assert np.all(precalc_markers.names.values == pseudotime_markers.names.values)
+ assert np.allclose(
+ precalc_markers.I__RNA_pseudotime__r.values,
+ pseudotime_markers.I__RNA_pseudotime__r.values,
+ rtol=0.15,
+ atol=0.15,
+ )
+ marker_ref = ArtifactRef.from_dict(
+ datastore.RNA.z["featureData/I__RNA_pseudotime__r"].attrs["source_artifact"]
+ )
+ assert marker_ref.kind == "pseudotime_markers"
+ assert (
+ ArtifactRef.from_dict(
+ datastore.RNA.z["featureData/I__RNA_pseudotime__p"].attrs[
+ "source_artifact"
+ ]
+ )
+ == marker_ref
+ )
+
+ def test_run_pseudotime_aggregation(
+ self,
+ pseudotime_aggregation,
+ datastore,
+ monkeypatch,
+ ):
+ precalc_values = np.load(full_path("aggregated_feat_idx.npy"))
+ agg_group = datastore.zw[pseudotime_aggregation.storage_path]
+ test_values = agg_group["feature_indices"][:]
+ assert isinstance(pseudotime_aggregation, PseudotimeAggregationResult)
+ assert (
+ ArtifactRef.from_dict(
+ datastore.RNA.z["featureData/pseudotime_clusters"].attrs[
+ "source_artifact"
+ ]
+ ).kind
+ == "pseudotime_aggregation"
+ )
+ assert np.array_equal(
+ precalc_values.astype(np.int64), test_values.astype(np.int64)
+ )
+
+ assert agg_group.attrs["complete"] is True
+ assert "input_fingerprints" in agg_group.attrs
+ assert "valid_features" in agg_group
+ assert np.isfinite(agg_group["data"][:]).all()
+ valid_features = np.asarray(agg_group["valid_features"][:], dtype=bool)
+ np.testing.assert_array_equal(
+ pseudotime_aggregation.feature_indices,
+ test_values[valid_features],
+ )
+ assert pseudotime_aggregation.data.shape[0] == valid_features.sum()
+ clusters = datastore.RNA.feats.fetch_all("pseudotime_clusters")
+ assigned = clusters[test_values[valid_features].astype(int)]
+ assert assigned.min() >= 1
+ assert assigned.max() <= 15
+ assert len(np.unique(assigned)) == 15
+ np.testing.assert_array_equal(
+ pseudotime_aggregation.feature_clusters,
+ assigned,
+ )
+ assert np.all(clusters[test_values[~valid_features].astype(int)] == -1)
+
+ def fail_if_recomputed(*_args, **_kwargs):
+ raise AssertionError("pseudotime aggregation should have been reused")
+
+ monkeypatch.setattr(
+ "scarf.trajectory.feature_dynamics.knn_clustering",
+ fail_if_recomputed,
+ )
+ cached = datastore.run_pseudotime_aggregation(
+ pseudotime_key="RNA_pseudotime",
+ cluster_label="pseudotime_clusters_cached",
+ n_clusters=15,
+ window_size=50,
+ chunk_size=10,
+ )
+ assert cached.storage_path == pseudotime_aggregation.storage_path
+ cached_ref = ArtifactRef.from_dict(
+ datastore.RNA.z["featureData/pseudotime_clusters_cached"].attrs[
+ "source_artifact"
+ ]
+ )
+ assert cached_ref == ArtifactRef.from_dict(
+ datastore.RNA.z["featureData/pseudotime_clusters"].attrs["source_artifact"]
+ )
+
+ def test_add_grouped_assay(self, grouped_assay, datastore):
+ test_values = datastore.get_cell_vals(
+ from_assay="PTIME_MODULES", cell_key="I", k="group_1"
+ )
+ groups = datastore.RNA.feats.fetch_all("pseudotime_clusters")
+ feature_index = np.where(groups == 1)[0]
+ expected = (
+ datastore.RNA.normed(
+ cell_idx=datastore.cells.active_index("I"),
+ feat_idx=feature_index,
+ )
+ .mean(axis=1)
+ .compute()
+ )
+ assert np.allclose(expected, test_values)
+
+ def test_make_bulk(self, leiden_clustering, datastore):
+ df = datastore.make_bulk(group_key="RNA_leiden_cluster")
+ assert df.shape == (18850, 10)
+ assert hash(tuple((df.values.flatten()))) == -3925915741848261436
+
+ def test_to_anndata(self, datastore):
+ # TODO: Check if all the attributes copied to anndata
+ datastore.to_anndata()
+
+ def test_run_topacedo_sampler(self, paris_clustering, topacedo_sampler):
+ assert topacedo_sampler.dtype == bool
+ assert topacedo_sampler.shape == paris_clustering.shape
+ cluster_sizes = np.bincount(paris_clustering)[1:]
+ sampled_sizes = np.bincount(
+ paris_clustering[topacedo_sampler],
+ minlength=int(paris_clustering.max()) + 1,
+ )[1:]
+ assert np.all(sampled_sizes >= 3)
+ assert np.all(sampled_sizes < cluster_sizes)
+
+ def test_plot_distributions(self, datastore):
+ result = splt.distribution(
+ datastore,
+ keys=["RNA_nCounts", "RNA_nFeatures"],
+ show=False,
+ )
+ assert isinstance(result, splt.PlotResult)
+ assert set(result.tables) == {"RNA_nCounts", "RNA_nFeatures"}
+ result.close()
+
+ def test_plot_embedding(self, umap, paris_clustering, datastore):
+ result = datastore.plots.embedding(
+ layout_key="RNA_UMAP",
+ color_by="RNA_cluster",
+ show=False,
+ )
+ assert isinstance(result, splt.PlotResult)
+ assert result.provenance.notes[:2] == ("embedding", "materialized")
+ result.close()
+
+ def test_plot_embedding_raster(self, umap, datastore):
+ result = splt.embedding_raster(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_nCounts",
+ pixels=64,
+ show=False,
+ )
+ assert isinstance(result, splt.PlotResult)
+ assert result.provenance.renderer == "matplotlib-raster"
+ result.close()
+
+ def test_plot_cluster_tree(self, paris_clustering, datastore):
+ result = splt.cluster_tree(
+ datastore,
+ cluster_key="RNA_cluster",
+ show=False,
+ )
+ assert isinstance(result, splt.PlotResult)
+ assert "cluster_summary" in result.tables
+ result.close()
+
+ def test_plot_marker_heatmap(self, marker_search, datastore):
+ result = splt.marker_heatmap(
+ datastore,
+ group_key="RNA_cluster",
+ show=False,
+ )
+ assert isinstance(result, splt.PlotResult)
+ assert "matrix" in result.tables
+ result.close()
+
+ def test_plot_pseudotime_heatmap(self, pseudotime_aggregation, datastore):
+ result = splt.pseudotime_heatmap(
+ datastore,
+ cell_key="I",
+ feat_key="I",
+ feature_cluster_key="pseudotime_clusters",
+ pseudotime_key="RNA_pseudotime",
+ show_features=["Wsb1", "Rest"],
+ show=False,
+ )
+ assert isinstance(result, splt.PlotResult)
+ assert "pseudotime_bins" in result.tables
+ result.close()
+
+ def test_legacy_plotting_apis_are_absent(self):
+ removed_datastore_methods = (
+ "plot_cells_dists",
+ "plot_layout",
+ "plot_marker_heatmap",
+ "plot_cluster_tree",
+ "plot_pseudotime_heatmap",
+ "plot_unified_layout",
+ )
+ for method_name in removed_datastore_methods:
+ assert not hasattr(DataStore, method_name)
+ assert not hasattr(MappingDatastore, "plot_unified_layout")
+
+ assert not hasattr(scarf, "plots")
+ assert not hasattr(splt, "_legacy")
+ for module_name in ("scarf.plots", "scarf.plotting._legacy"):
+ assert util.find_spec(module_name) is None
+ with pytest.raises(
+ ModuleNotFoundError, match=module_name.replace(".", r"\.")
+ ):
+ import_module(module_name)
+
+ def test_mark_hvgs_with_atac_assay(self, atac_datastore):
+ import pytest
+
+ with pytest.raises(TypeError):
+ atac_datastore.mark_hvgs()
+
+ def test_mark_hvgs_default_max_cells_excludes_ubiquitous(self, datastore_ephemeral):
+ from unittest.mock import patch
+
+ datastore = datastore_ephemeral
+ n_selected = int(np.asarray(datastore.cells.fetch_all("I"), dtype=bool).sum())
+ expected_max = n_selected - 20
+ assay = datastore.RNA
+
+ def _fake_mark_hvgs(*args, **kwargs):
+ assay.feats.insert(
+ "I__hvgs",
+ np.zeros(assay.feats.N, dtype=bool),
+ fill_value=False,
+ overwrite=True,
+ )
+
+ with patch.object(assay, "mark_hvgs", side_effect=_fake_mark_hvgs) as mock:
+ datastore.mark_hvgs(show_plot=False, top_n=10)
+ assert mock.call_args.kwargs["max_cells"] == expected_max
+ assert mock.call_args.kwargs["min_cells"] == 20
+ assert mock.call_args.kwargs["top_n"] == 10
+ assert mock.call_args.kwargs["bin_strategy"] == "adaptive"
+
+ with patch.object(assay, "mark_hvgs", side_effect=_fake_mark_hvgs) as mock:
+ datastore.mark_hvgs(
+ show_plot=False,
+ top_n=10,
+ max_cells=np.inf,
+ bin_strategy="adaptive",
+ )
+ assert mock.call_args.kwargs["max_cells"] == np.inf
+ assert mock.call_args.kwargs["bin_strategy"] == "adaptive"
+
+ def test_adaptive_hvg_stats_reuse_single_matrix_pass(
+ self,
+ auto_filter_cells,
+ datastore,
+ monkeypatch,
+ ):
+ import scarf.features.variability as variability
+
+ assay = datastore.RNA
+ identifier, fixed_column = assay.set_summary_stats(
+ "I",
+ bin_strategy="fixed",
+ )
+
+ def fail_matrix_pass(*args, **kwargs):
+ pytest.fail("adaptive correction must reuse cached feature statistics")
+
+ def fail_metadata_trend(*args, **kwargs):
+ pytest.fail("HVG correction must not route through metadata")
+
+ monkeypatch.setattr(assay, "_streaming_feature_stats", fail_matrix_pass)
+ monkeypatch.setattr(assay.feats, "remove_trend", fail_metadata_trend)
+ adaptive_identifier, adaptive_column = assay.set_summary_stats("I")
+
+ assert adaptive_identifier == identifier
+ assert fixed_column == "c_var__200__0.1"
+ assert adaptive_column == "c_var__adaptive__200__0.1"
+ assert f"{identifier}_{fixed_column}" in assay.feats.columns
+ assert f"{identifier}_{adaptive_column}" in assay.feats.columns
+
+ def fail_correction(*args, **kwargs):
+ pytest.fail("cached adaptive correction must be reused")
+
+ monkeypatch.setattr(variability, "fit_lowess", fail_correction)
+ assert assay.set_summary_stats("I") == (
+ identifier,
+ adaptive_column,
+ )
+
+ def test_mark_prevalent_peaks_with_rna_assay(self, datastore):
+ import pytest
+
+ with pytest.raises(TypeError):
+ datastore.mark_prevalent_peaks()
+
+ def test_mark_prevalent_peaks_links_selection_artifact(
+ self,
+ mark_prevalent_peaks,
+ atac_datastore,
+ monkeypatch,
+ ):
+ column = atac_datastore.ATAC.z["featureData/I__prevalent_peaks"]
+ ref = ArtifactRef.from_dict(column.attrs["source_artifact"])
+
+ assert ref.kind == "feature_selection"
+ status = atac_datastore.inspect_artifact(ref)
+ assert status.operation == "mark_prevalent_peaks"
+ assert status.inputs["feature_selection"]["kind"] == "feature_selection"
+ assert "values_fingerprint" not in status.inputs
+ atac_datastore.mark_prevalent_peaks(top_n=5000)
+ refreshed = ArtifactRef.from_dict(
+ atac_datastore.ATAC.z["featureData/I__prevalent_peaks"].attrs[
+ "source_artifact"
+ ]
+ )
+ assert refreshed == ref
+ before = np.asarray(atac_datastore.ATAC.feats.fetch_all("I__prevalent_peaks"))
+ monkeypatch.setattr(
+ "scarf.datastore._operations.quality_control."
+ "resolve_generated_selection_artifact",
+ lambda *args, **kwargs: (_ for _ in ()).throw(
+ RuntimeError("interrupted artifact write")
+ ),
+ )
+ with pytest.raises(RuntimeError, match="interrupted artifact write"):
+ atac_datastore.mark_prevalent_peaks(top_n=4000)
+ np.testing.assert_array_equal(
+ atac_datastore.ATAC.feats.fetch_all("I__prevalent_peaks"),
+ before,
+ )
+ assert (
+ ArtifactRef.from_dict(
+ atac_datastore.ATAC.z["featureData/I__prevalent_peaks"].attrs[
+ "source_artifact"
+ ]
+ )
+ == ref
+ )
+
+ def test_run_marker_search_with_no_groupkey(self, datastore):
+ import pytest
+
+ with pytest.raises(ValueError):
+ datastore.run_marker_search(group_key=None)
+
+ def test_run_marker_search_with_cellkey(self, datastore, paris_clustering):
+ datastore.run_marker_search(group_key="RNA_cluster", cell_key="I")
+
+ def test_streaming_feature_stats_matches_three_pass(self, datastore):
+ from scarf.utils import controlled_compute
+
+ assay = datastore.RNA
+ cell_idx, feat_idx = assay._get_cell_feat_idx("I", "I")
+
+ tiled = assay._streaming_feature_stats(cell_idx, feat_idx)
+
+ normed = assay.normed(cell_idx, feat_idx)
+ legacy_n = controlled_compute((normed > 0).sum(axis=0), assay.nthreads)
+ legacy_tot = controlled_compute(normed.sum(axis=0), assay.nthreads)
+ legacy_sigmas = controlled_compute(normed.var(axis=0), assay.nthreads)
+
+ np.testing.assert_allclose(tiled["normed_n"], legacy_n, rtol=0, atol=1e-6)
+ np.testing.assert_allclose(
+ tiled["normed_tot"], legacy_tot, rtol=1e-6, atol=1e-6
+ )
+ np.testing.assert_allclose(tiled["sigmas"], legacy_sigmas, rtol=1e-5, atol=1e-6)
+
+ def test_streaming_feature_stats_one_decode_per_physical_chunk(
+ self, datastore, monkeypatch
+ ):
+ import scarf.assay as assay_mod
+
+ assay = datastore.RNA
+ cell_idx, feat_idx = assay._get_cell_feat_idx("I", "I")
+ zarr_arr = assay.rawData._backing
+ row_chunk, col_chunk = zarr_arr.chunks[:2]
+ expected = len({int(i) // row_chunk for i in cell_idx}) * len(
+ {int(i) // col_chunk for i in feat_idx}
+ )
+ calls = {"n": 0}
+ original = assay_mod._read_block
+
+ def counted(zarr_arr_arg, rows, cols):
+ calls["n"] += 1
+ return original(zarr_arr_arg, rows, cols)
+
+ monkeypatch.setattr(assay_mod, "_read_block", counted)
+ assay._streaming_feature_stats(cell_idx, feat_idx)
+ assert calls["n"] == expected
+
+ def test_streaming_feature_stats_requires_sf(self, datastore):
+ import pytest
+
+ assay = datastore.RNA
+ cell_idx, feat_idx = assay._get_cell_feat_idx("I", "I")
+ original = assay.sf
+ try:
+ assay.sf = None
+ with pytest.raises(ValueError):
+ assay._streaming_feature_stats(cell_idx, feat_idx)
+ finally:
+ assay.sf = original
+
+ def test_read_block_preserves_order_and_selection(self, datastore):
+ from scarf.assay import _read_block
+
+ backing = datastore.RNA.rawData._backing
+
+ rows = np.array([3, 4, 5, 6])
+ cols = np.array([0, 1, 2])
+ contiguous = _read_block(backing, rows, cols)
+ reference = np.asarray(backing.get_orthogonal_selection((rows, cols)))
+ np.testing.assert_array_equal(contiguous, reference)
+
+ scattered_rows = np.array([7, 2, 9])
+ scattered_cols = np.array([4, 0, 2])
+ scattered = _read_block(backing, scattered_rows, scattered_cols)
+ ref2 = np.asarray(
+ backing.get_orthogonal_selection((scattered_rows, scattered_cols))
+ )
+ np.testing.assert_array_equal(scattered, ref2)
diff --git a/tests/test_datastore_contracts.py b/tests/test_datastore_contracts.py
new file mode 100644
index 00000000..650d0915
--- /dev/null
+++ b/tests/test_datastore_contracts.py
@@ -0,0 +1,387 @@
+import inspect
+
+from scarf.datastore.base_datastore import BaseDataStore
+from scarf.datastore.datastore import DataStore
+from scarf.datastore.graph_datastore import GraphDataStore
+from scarf.datastore.mapping_datastore import MappingDatastore
+from tests.signature_contracts import signature_digest
+
+
+_METHODS = {
+ BaseDataStore: (
+ "__init__",
+ "get_cell_vals",
+ "get_assay_state",
+ "inspect_artifact",
+ "lineage",
+ "list_artifacts",
+ "load_artifact",
+ "set_default_assay",
+ "summary",
+ ),
+ GraphDataStore: (
+ "__init__",
+ "build_ann_index",
+ "build_connectivity_map",
+ "build_embedding_initialization",
+ "build_mapping_reference",
+ "get_diffusion_operator",
+ "get_latest_graph_loc",
+ "get_normalized_group_path",
+ "get_imputed",
+ "get_mapping_reference",
+ "integrate_assays",
+ "load_graph",
+ "query_neighbors",
+ "run_fate_mapping",
+ "run_leiden_clustering",
+ "run_lsi",
+ "run_custom_reduction",
+ "run_harmony",
+ "run_normalization",
+ "run_paris_clustering",
+ "run_pca",
+ "run_pseudotime_scoring",
+ "run_topacedo_sampler",
+ "run_tsne",
+ "run_umap",
+ ),
+ MappingDatastore: (
+ "calibrate_label_transfer_threshold",
+ "get_mapping_result",
+ "get_mapping_score",
+ "get_target_classes",
+ "get_target_label_evidence",
+ "project_reference_embedding",
+ "run_mapping",
+ ),
+ DataStore: (
+ "__init__",
+ "add_grouped_assay",
+ "add_melded_assay",
+ "auto_filter_cells",
+ "calc_membership_strength",
+ "export_markers_to_csv",
+ "filter_cells",
+ "get_assay",
+ "get_enrichment",
+ "get_markers",
+ "make_bulk",
+ "mark_hto_identities",
+ "mark_hvgs",
+ "mark_prevalent_peaks",
+ "metric_clisi",
+ "metric_cluster_separability",
+ "metric_graph_connectivity",
+ "metric_graph_silhouette",
+ "metric_ilisi",
+ "metric_label_concordance",
+ "metric_lisi",
+ "metric_proportional_batch_mixing",
+ "run_aucell",
+ "run_cell_cycle_scoring",
+ "run_doublet_detection",
+ "run_marker_search",
+ "run_pseudotime_aggregation",
+ "run_pseudotime_marker_search",
+ "run_waggr",
+ "set_hvgs",
+ "show_zarr_tree",
+ "smart_label",
+ "to_anndata",
+ ),
+}
+
+_SIGNATURE_DIGESTS = {
+ BaseDataStore: "618821f381727cab2b5b19ec983871b486c5bdcab992435f3a0cb34033dfbcf4",
+ GraphDataStore: "eecfb41646f3b515e2979882a1afe56b0d1497775562110af406c8f178f7005a",
+ MappingDatastore: "c2d386b537ba464e9b08e7a67eb7342a72b4845b76053ef933a0b531b4800c67",
+ DataStore: "3da2164eafb9ac1ea17d9faa3ad8be10850a38b5d137f0773a7c9b306943574c",
+}
+
+
+def test_datastore_public_method_signatures_are_stable():
+ for cls, names in _METHODS.items():
+ methods = {name: getattr(cls, name) for name in names}
+ assert signature_digest(methods) == _SIGNATURE_DIGESTS[cls]
+
+
+def test_datastore_public_class_chain_is_stable():
+ public_classes = {BaseDataStore, GraphDataStore, MappingDatastore, DataStore}
+ assert [cls for cls in DataStore.mro() if cls in public_classes] == [
+ DataStore,
+ MappingDatastore,
+ GraphDataStore,
+ BaseDataStore,
+ ]
+ assert DataStore.__module__ == "scarf.datastore.datastore"
+ assert MappingDatastore.__module__ == "scarf.datastore.mapping_datastore"
+ assert GraphDataStore.__module__ == "scarf.datastore.graph_datastore"
+ assert BaseDataStore.__module__ == "scarf.datastore.base_datastore"
+
+
+def test_stored_graph_lookup_remains_internal():
+ assert not hasattr(GraphDataStore, "lookup_stored_graph")
+ assert hasattr(GraphDataStore, "_lookup_stored_graph")
+
+
+def test_datastore_property_contracts_are_stable():
+ for name in ("assay_names", "zw"):
+ descriptor = inspect.getattr_static(BaseDataStore, name)
+ assert isinstance(descriptor, property)
+ assert descriptor.fget is not None
+ assert list(inspect.signature(descriptor.fget).parameters) == ["self"]
+ assert inspect.getattr_static(DataStore, name) is descriptor
+
+
+def test_datastore_plot_namespace_contract_is_stable():
+ descriptor = inspect.getattr_static(DataStore, "plots")
+
+ assert isinstance(descriptor, property)
+ assert descriptor.fget is not None
+ assert list(inspect.signature(descriptor.fget).parameters) == ["self"]
+ assert "plots" in DataStore.__dict__
+ for cls in (BaseDataStore, GraphDataStore, MappingDatastore):
+ assert not hasattr(cls, "plots")
+
+
+def test_datastore_static_method_contracts_are_stable():
+ static_methods = {
+ GraphDataStore: ("_resolve_local_cache_plan",),
+ MappingDatastore: (
+ "_label_vote_decision",
+ "_projection_block_size",
+ "_query_batch_codes",
+ "calibrate_label_transfer_threshold",
+ ),
+ DataStore: ("_write_marker_slot",),
+ }
+ for cls, names in static_methods.items():
+ assert all(
+ isinstance(inspect.getattr_static(cls, name), staticmethod)
+ for name in names
+ )
+
+ for name in (
+ "_same_assay_store",
+ "_validate_projection_arrays",
+ "_projection_has_provenance",
+ "_PROJECTION_SCHEMA_VERSION",
+ "_LEGACY_PROJECTION_SCHEMA_VERSIONS",
+ "_LEGACY_PROJECTION_ATTRS",
+ "_LEGACY_PROJECTION_ARRAYS",
+ "_projection_attr",
+ "_projection_array_name",
+ ):
+ assert not hasattr(MappingDatastore, name)
+
+
+def test_graph_datastore_private_mixin_order_is_stable():
+ from scarf.datastore._operations.clustering import _ClusteringOperationsMixin
+ from scarf.datastore._operations.embeddings import _EmbeddingOperationsMixin
+ from scarf.datastore._operations.graph import _GraphOperationsMixin
+ from scarf.datastore._operations.mapping_reference import (
+ _MappingReferenceOperationsMixin,
+ )
+ from scarf.datastore._operations.trajectory import _TrajectoryOperationsMixin
+
+ assert GraphDataStore.__bases__ == (
+ _EmbeddingOperationsMixin,
+ _ClusteringOperationsMixin,
+ _TrajectoryOperationsMixin,
+ _MappingReferenceOperationsMixin,
+ _GraphOperationsMixin,
+ BaseDataStore,
+ )
+ assert not hasattr(GraphDataStore, "make_graph")
+
+
+def test_mapping_datastore_private_mixin_order_is_stable():
+ from scarf.datastore._operations.mapping import _MappingOperationsMixin
+
+ assert MappingDatastore.__bases__ == (
+ _MappingOperationsMixin,
+ GraphDataStore,
+ )
+ assert MappingDatastore.mro()[:3] == [
+ MappingDatastore,
+ _MappingOperationsMixin,
+ GraphDataStore,
+ ]
+
+
+def test_retired_unified_mapping_apis_are_absent():
+ for name in (
+ "_load_unified_layout_data",
+ "load_unified_graph",
+ "run_unified_tsne",
+ "run_unified_umap",
+ ):
+ assert not hasattr(MappingDatastore, name)
+
+
+def test_datastore_private_mixin_order_is_stable():
+ from scarf.datastore._operations.features import _FeatureOperationsMixin
+ from scarf.datastore._operations.integration_metrics import (
+ _IntegrationMetricsOperationsMixin,
+ )
+ from scarf.datastore._operations.presentation import _PresentationOperationsMixin
+ from scarf.datastore._operations.quality_control import (
+ _QualityControlOperationsMixin,
+ )
+ from scarf.datastore._operations.trajectory import (
+ _TrajectoryFeatureOperationsMixin,
+ )
+
+ assert DataStore.__bases__ == (
+ _QualityControlOperationsMixin,
+ _FeatureOperationsMixin,
+ _TrajectoryFeatureOperationsMixin,
+ _IntegrationMetricsOperationsMixin,
+ _PresentationOperationsMixin,
+ MappingDatastore,
+ )
+ assert DataStore.mro()[:7] == [
+ DataStore,
+ _QualityControlOperationsMixin,
+ _FeatureOperationsMixin,
+ _TrajectoryFeatureOperationsMixin,
+ _IntegrationMetricsOperationsMixin,
+ _PresentationOperationsMixin,
+ MappingDatastore,
+ ]
+
+
+def test_datastore_temporary_factory_uses_parent_budget_and_local_profile(tmp_path):
+ import numpy as np
+ from scipy.sparse import csr_matrix
+
+ from scarf.datastore._operations.quality_control import (
+ _QualityControlOperationsMixin,
+ )
+ from scarf.quality_control.doublets import write_doublet_target_zarr
+
+ descriptor = inspect.getattr_static(DataStore, "_create_temporary_datastore")
+ assert not isinstance(descriptor, staticmethod)
+ assert "_create_temporary_datastore" in DataStore.__dict__
+ assert "_create_temporary_datastore" not in _QualityControlOperationsMixin.__dict__
+
+ class DataStoreSubclass(DataStore):
+ pass
+
+ path = tmp_path / "temporary.zarr"
+ write_doublet_target_zarr(
+ zarr_loc=str(path),
+ assay_name="RNA",
+ sim_counts=csr_matrix(np.array([[1, 0], [0, 2]], dtype=np.uint16)),
+ feat_ids=np.array(["f1", "f2"]),
+ feat_names=np.array(["g1", "g2"]),
+ dtype="uint16",
+ mem_budget=64 * 1024 * 1024,
+ nthreads=1,
+ profile="fast_local",
+ )
+
+ source = object.__new__(DataStoreSubclass)
+ source.memoryBytes = 64 * 1024 * 1024
+ source.storageProfile = "cloud"
+ result = source._create_temporary_datastore(
+ str(path),
+ default_assay="RNA",
+ assay_types={"RNA": "RNA"},
+ nthreads=3,
+ )
+ assert result.memoryBytes == 64 * 1024 * 1024
+ assert result.nthreads == 3
+ assert result.storageProfile == "fast_local"
+
+
+def test_datastore_facades_only_own_composition_methods():
+ def defined_methods(cls: type) -> set[str]:
+ return {
+ name
+ for name, value in cls.__dict__.items()
+ if inspect.isfunction(value)
+ or isinstance(value, (classmethod, staticmethod))
+ }
+
+ assert defined_methods(GraphDataStore) == {"__init__"}
+ assert defined_methods(MappingDatastore) == set()
+ assert defined_methods(DataStore) == {
+ "__init__",
+ "_create_temporary_datastore",
+ "get_assay",
+ }
+
+
+def test_datastore_operation_mixins_have_unique_method_owners():
+ from scarf.datastore._operations.clustering import _ClusteringOperationsMixin
+ from scarf.datastore._operations.embeddings import _EmbeddingOperationsMixin
+ from scarf.datastore._operations.features import _FeatureOperationsMixin
+ from scarf.datastore._operations.graph import _GraphOperationsMixin
+ from scarf.datastore._operations.integration_metrics import (
+ _IntegrationMetricsOperationsMixin,
+ )
+ from scarf.datastore._operations.mapping import _MappingOperationsMixin
+ from scarf.datastore._operations.mapping_reference import (
+ _MappingReferenceOperationsMixin,
+ )
+ from scarf.datastore._operations.presentation import _PresentationOperationsMixin
+ from scarf.datastore._operations.quality_control import (
+ _QualityControlOperationsMixin,
+ )
+ from scarf.datastore._operations.trajectory import (
+ _TrajectoryFeatureOperationsMixin,
+ _TrajectoryOperationsMixin,
+ )
+
+ mixins = (
+ _EmbeddingOperationsMixin,
+ _ClusteringOperationsMixin,
+ _TrajectoryOperationsMixin,
+ _MappingReferenceOperationsMixin,
+ _GraphOperationsMixin,
+ _MappingOperationsMixin,
+ _QualityControlOperationsMixin,
+ _FeatureOperationsMixin,
+ _TrajectoryFeatureOperationsMixin,
+ _IntegrationMetricsOperationsMixin,
+ _PresentationOperationsMixin,
+ )
+ owners: dict[str, list[str]] = {}
+ for mixin in mixins:
+ for name, value in mixin.__dict__.items():
+ if inspect.isfunction(value) or isinstance(
+ value, (classmethod, staticmethod)
+ ):
+ owners.setdefault(name, []).append(mixin.__name__)
+ assert {name: owner for name, owner in owners.items() if len(owner) > 1} == {}
+
+
+def test_feature_selection_and_pseudotime_methods_have_domain_owners():
+ from scarf.datastore._operations.features import _FeatureOperationsMixin
+ from scarf.datastore._operations.quality_control import (
+ _QualityControlOperationsMixin,
+ )
+ from scarf.datastore._operations.trajectory import (
+ _TrajectoryFeatureOperationsMixin,
+ _TrajectoryOperationsMixin,
+ )
+
+ assert "mark_hvgs" in _FeatureOperationsMixin.__dict__
+ assert "get_enrichment" in _FeatureOperationsMixin.__dict__
+ assert "run_aucell" in _FeatureOperationsMixin.__dict__
+ assert "run_waggr" in _FeatureOperationsMixin.__dict__
+ assert "mark_hvgs" not in _QualityControlOperationsMixin.__dict__
+ assert "run_fate_mapping" in _TrajectoryOperationsMixin.__dict__
+ assert "run_fate_mapping" not in _TrajectoryFeatureOperationsMixin.__dict__
+ assert "run_pseudotime_marker_search" in (
+ _TrajectoryFeatureOperationsMixin.__dict__
+ )
+ assert "run_pseudotime_aggregation" in _TrajectoryFeatureOperationsMixin.__dict__
+ assert "run_pseudotime_marker_search" not in _FeatureOperationsMixin.__dict__
+ assert "run_pseudotime_aggregation" not in _FeatureOperationsMixin.__dict__
+ assert not hasattr(GraphDataStore, "run_pseudotime_marker_search")
+ assert not hasattr(GraphDataStore, "run_pseudotime_aggregation")
+ assert not hasattr(MappingDatastore, "run_pseudotime_marker_search")
+ assert not hasattr(MappingDatastore, "run_pseudotime_aggregation")
diff --git a/tests/test_datastore_ephemeral.py b/tests/test_datastore_ephemeral.py
new file mode 100644
index 00000000..efaaca1f
--- /dev/null
+++ b/tests/test_datastore_ephemeral.py
@@ -0,0 +1,247 @@
+import numpy as np
+import pytest
+
+from scarf.utils import configure_output
+from tests.fixtures_datastore import _has_graph, build_neighbourhood_graph
+
+pytestmark = pytest.mark.slow
+
+
+def _active_cell_count(datastore) -> int:
+ return len(datastore.cells.active_index("I"))
+
+
+def _ensure_graph(datastore):
+ assert _has_graph(datastore), "analyzed datastore fixture has no RNA/I/hvgs graph"
+
+
+def _build_symphony_reference(
+ datastore,
+ batch_column: str,
+ *,
+ k: int = 3,
+ nclust: int = 5,
+):
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.reduction is not None
+ correction = datastore.run_harmony(
+ [batch_column],
+ state.reduction,
+ harmony_params={"nclust": nclust},
+ update_state=False,
+ )
+ ann_index = datastore.build_ann_index(
+ correction,
+ update_state=False,
+ )
+ neighbors = datastore.query_neighbors(
+ ann_index,
+ coordinates=correction,
+ k=k,
+ update_state=False,
+ )
+ return datastore.build_mapping_reference(neighbors)
+
+
+def _clear_umap_columns(datastore):
+ for column in ("RNA_UMAP1", "RNA_UMAP2"):
+ if column in datastore.cells.columns:
+ datastore.cells.drop(column)
+
+
+def test_run_umap_recomputes_coordinates(analyzed_datastore_ephemeral):
+ ds = analyzed_datastore_ephemeral
+ _ensure_graph(ds)
+ _clear_umap_columns(ds)
+
+ ds.run_umap(n_epochs=10, parallel=False)
+
+ umap1 = ds.cells.fetch("RNA_UMAP1")
+ umap2 = ds.cells.fetch("RNA_UMAP2")
+ assert len(umap1) == _active_cell_count(ds)
+ assert len(umap2) == _active_cell_count(ds)
+
+
+def test_run_umap_uses_explicit_progress_for_verbose_output(
+ analyzed_datastore_ephemeral,
+ monkeypatch,
+):
+ import scarf.embeddings.umap as umap_module
+
+ ds = analyzed_datastore_ephemeral
+ _ensure_graph(ds)
+ observed: list[bool] = []
+
+ def fake_fit_transform(*, graph, verbose, **_kwargs):
+ observed.append(verbose)
+ return np.zeros((graph.shape[0], 2)), None, None
+
+ monkeypatch.setattr(umap_module, "fit_transform", fake_fit_transform)
+ try:
+ configure_output(progress=False)
+ ds.run_umap(
+ n_epochs=10,
+ label="progress_off_umap",
+ invalidate_cache=True,
+ )
+ configure_output(progress=True)
+ ds.run_umap(
+ n_epochs=10,
+ label="progress_on_umap",
+ invalidate_cache=True,
+ )
+ finally:
+ configure_output(progress=False)
+
+ assert observed == [False, True]
+
+
+def test_run_leiden_writes_cluster_labels(analyzed_datastore_ephemeral):
+ ds = analyzed_datastore_ephemeral
+ _ensure_graph(ds)
+
+ ds.run_leiden_clustering(label="ephemeral_leiden")
+
+ groups = ds.cells.fetch("RNA_ephemeral_leiden", key="I")
+ assert len(groups) == _active_cell_count(ds)
+ assert np.unique(groups).size >= 1
+
+
+def test_cached_harmony_can_rebuild_incomplete_mapping_reference(
+ analyzed_datastore_ephemeral,
+):
+ ds = analyzed_datastore_ephemeral
+ _ensure_graph(ds)
+ batches = np.where(np.arange(ds.cells.N) % 2 == 0, "a", "b")
+ ds.cells.insert("mapping_batch", batches, overwrite=True)
+ reference = _build_symphony_reference(ds, "mapping_batch")
+ state = ds.get_assay_state("RNA")
+ assert state is not None
+ first_harmony = state.batch_correction
+ first_ref = reference.ref
+ assert state.named_results["mapping_reference"] == first_ref
+ artifact = ds.zw[ds.inspect_artifact(first_ref).path]
+ for name in (
+ "reference_distance_quantiles",
+ "reference_distance_values",
+ ):
+ values = artifact[name][:]
+ del artifact[name]
+ with pytest.raises(ValueError, match="missing required arrays"):
+ ds.get_mapping_reference(first_ref)
+ artifact.create_array(name, data=values)
+ del artifact["sigma"]
+ with pytest.raises(ValueError, match="missing required arrays"):
+ ds.get_mapping_reference(first_ref)
+
+ assert state.neighbors is not None
+ rebuilt = ds.build_mapping_reference(state.neighbors)
+ loaded = ds.get_mapping_reference(rebuilt.ref)
+
+ assert rebuilt.method == "symphony"
+ assert loaded.ref == rebuilt.ref
+ assert ds.inspect_artifact(rebuilt.ref).complete
+ rebuilt_state = ds.get_assay_state("RNA")
+ assert rebuilt_state is not None
+ assert rebuilt_state.batch_correction == first_harmony
+ assert rebuilt_state.named_results["mapping_reference"] == rebuilt.ref
+ assert rebuilt.ref != first_ref
+
+
+def test_mapping_reference_rebuilds_after_missing_artifact(
+ analyzed_datastore_ephemeral,
+):
+ ds = analyzed_datastore_ephemeral
+ _ensure_graph(ds)
+ batches = np.where(np.arange(ds.cells.N) % 2 == 0, "a", "b")
+ ds.cells.insert("mapping_batch", batches, overwrite=True)
+ reference = _build_symphony_reference(ds, "mapping_batch")
+ first_state = ds.get_assay_state("RNA")
+ assert first_state is not None
+ first_ref = reference.ref
+ assert first_state.named_results["mapping_reference"] == first_ref
+ del ds.zw[ds.inspect_artifact(first_ref).path]
+
+ assert first_state.neighbors is not None
+ rebuilt = ds.build_mapping_reference(first_state.neighbors)
+ loaded = ds.get_mapping_reference(rebuilt.ref)
+
+ assert rebuilt.ref != first_ref
+ assert loaded.ref == rebuilt.ref
+ assert loaded.method == "symphony"
+ assert ds.inspect_artifact(rebuilt.ref).complete
+ rebuilt_state = ds.get_assay_state("RNA")
+ assert rebuilt_state is not None
+ assert rebuilt_state.named_results["mapping_reference"] == rebuilt.ref
+
+
+def test_mapping_reference_tracks_changed_batch_values(
+ analyzed_datastore_ephemeral,
+):
+ ds = analyzed_datastore_ephemeral
+ _ensure_graph(ds)
+ batches = np.where(np.arange(ds.cells.N) % 2 == 0, "a", "b")
+ ds.cells.insert("mapping_batch", batches, overwrite=True)
+ first_reference = _build_symphony_reference(ds, "mapping_batch")
+ first_state = ds.get_assay_state("RNA")
+ assert first_state is not None
+ assert first_state.named_results["mapping_reference"] == first_reference.ref
+
+ changed = np.where(np.arange(ds.cells.N) % 3 == 0, "a", "b")
+ ds.cells.insert("mapping_batch", changed, overwrite=True)
+ rebuilt = _build_symphony_reference(ds, "mapping_batch")
+ second_state = ds.get_assay_state("RNA")
+
+ assert second_state is not None
+ assert rebuilt.ref != first_reference.ref
+ assert rebuilt.batch_correction != first_reference.batch_correction
+ assert second_state.named_results["mapping_reference"] == rebuilt.ref
+ assert rebuilt.metadata["batch_columns"] == ["mapping_batch"]
+ assert ds.get_mapping_reference(rebuilt.ref).ref == rebuilt.ref
+
+
+def test_mapping_reference_rejects_unscaled_pca_chain(
+ analyzed_datastore_ephemeral,
+) -> None:
+ ds = analyzed_datastore_ephemeral
+ _ensure_graph(ds)
+ build_neighbourhood_graph(
+ ds,
+ feat_key="hvgs",
+ feat_scaling=False,
+ dims=5,
+ k=3,
+ n_centroids=10,
+ )
+ state = ds.get_assay_state("RNA")
+ assert state is not None
+ assert state.reduction is not None
+ assert state.neighbors is not None
+ with pytest.raises(ValueError, match="feature scaling"):
+ ds.build_mapping_reference(state.neighbors)
+ parameters = ds.inspect_artifact(state.reduction).parameters
+ assert parameters is not None
+ assert parameters["feat_scaling"] is False
+ assert ds.inspect_artifact(state.reduction).operation == "run_pca"
+
+
+def test_mapping_reference_identical_call_reuses_complete_chain(
+ analyzed_datastore_ephemeral,
+) -> None:
+ ds = analyzed_datastore_ephemeral
+ _ensure_graph(ds)
+ state = ds.get_assay_state("RNA")
+ assert state is not None
+ assert state.neighbors is not None
+
+ first_reference = ds.build_mapping_reference(state.neighbors)
+ first_state = ds.get_assay_state("RNA")
+ second_reference = ds.build_mapping_reference(state.neighbors)
+ second_state = ds.get_assay_state("RNA")
+ loaded = ds.get_mapping_reference(second_reference.ref)
+
+ assert first_reference.ref == second_reference.ref
+ assert loaded.ref == second_reference.ref
+ assert ds.inspect_artifact(second_reference.ref).complete
+ assert second_state == first_state
diff --git a/tests/test_datastore_plot_accessor.py b/tests/test_datastore_plot_accessor.py
new file mode 100644
index 00000000..efc0f945
--- /dev/null
+++ b/tests/test_datastore_plot_accessor.py
@@ -0,0 +1,280 @@
+import ast
+import inspect
+import textwrap
+from collections.abc import Callable
+from copy import copy
+from typing import Any, get_type_hints
+
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+import scarf.plotting as splt
+from scarf.datastore.datastore import DataStore
+from scarf.datastore.plot_accessor import DataStorePlotAccessor
+
+
+_STORE_PLOT_METHODS = (
+ "cluster_connectivity",
+ "cluster_tree",
+ "composition",
+ "distribution",
+ "dotplot",
+ "embedding",
+ "embedding_raster",
+ "marker_heatmap",
+ "mapping_calibration",
+ "mapping_confusion",
+ "mapping_evidence",
+ "mapping_projection",
+ "mapping_score",
+ "matrixplot",
+ "pseudotime_heatmap",
+ "run_recipe",
+)
+
+
+def _annotation_text(annotation: ast.expr | None) -> str | None:
+ if annotation is None:
+ return None
+ if isinstance(annotation, ast.Constant) and isinstance(annotation.value, str):
+ annotation = ast.parse(annotation.value, mode="eval").body
+ return ast.unparse(annotation)
+
+
+def _source_annotations(function: Callable[..., Any]) -> dict[str, str | None]:
+ source = textwrap.dedent(inspect.getsource(function))
+ node = next(
+ entry
+ for entry in ast.parse(source).body
+ if isinstance(entry, ast.FunctionDef | ast.AsyncFunctionDef)
+ )
+ arguments = node.args
+ annotations = {
+ argument.arg: _annotation_text(argument.annotation)
+ for argument in (
+ *arguments.posonlyargs,
+ *arguments.args,
+ *arguments.kwonlyargs,
+ )
+ }
+ if arguments.vararg is not None:
+ annotations[arguments.vararg.arg] = _annotation_text(
+ arguments.vararg.annotation
+ )
+ if arguments.kwarg is not None:
+ annotations[arguments.kwarg.arg] = _annotation_text(arguments.kwarg.annotation)
+ annotations["return"] = _annotation_text(node.returns)
+ return annotations
+
+
+def _parameter_contract(
+ function: Callable[..., Any],
+) -> list[tuple[str, Any, Any]]:
+ return [
+ (parameter.name, parameter.kind, parameter.default)
+ for parameter in list(inspect.signature(function).parameters.values())[1:]
+ ]
+
+
+def test_plot_accessor_surface_matches_store_first_plotting_exports():
+ accessor_methods = {
+ name
+ for name, value in vars(DataStorePlotAccessor).items()
+ if not name.startswith("_") and inspect.isfunction(value)
+ }
+ store_first_exports = {
+ name
+ for name in splt.__all__
+ if inspect.isfunction(value := getattr(splt, name))
+ and next(iter(inspect.signature(value).parameters), None) == "store"
+ }
+
+ assert accessor_methods == set(_STORE_PLOT_METHODS)
+ assert store_first_exports == set(_STORE_PLOT_METHODS)
+
+
+def test_unified_plot_accessor_is_absent():
+ assert "unified_embedding" not in splt.__all__
+ assert not hasattr(DataStorePlotAccessor, "unified_embedding")
+
+
+@pytest.mark.parametrize("name", _STORE_PLOT_METHODS)
+def test_plot_accessor_signatures_match_standalone_functions(name: str):
+ standalone = getattr(splt, name)
+ accessor_method = getattr(DataStorePlotAccessor, name)
+
+ assert _parameter_contract(accessor_method) == _parameter_contract(standalone)
+
+ standalone_annotations = _source_annotations(standalone)
+ accessor_annotations = _source_annotations(accessor_method)
+ standalone_annotations.pop("store")
+ accessor_annotations.pop("self")
+ assert accessor_annotations == standalone_annotations
+
+
+@pytest.mark.parametrize("name", _STORE_PLOT_METHODS)
+def test_plot_accessor_type_hints_match_standalone_functions(name: str):
+ standalone_hints = get_type_hints(getattr(splt, name))
+ accessor_hints = get_type_hints(getattr(DataStorePlotAccessor, name))
+
+ standalone_hints.pop("store")
+ assert accessor_hints == standalone_hints
+
+
+@pytest.mark.parametrize(
+ ("name", "args", "kwargs"),
+ [
+ (
+ "embedding",
+ (),
+ {"layout_key": "RNA_UMAP", "rasterize_threshold": 17},
+ ),
+ ("embedding_raster", (), {"layout_key": "RNA_UMAP", "pixels": 32}),
+ (
+ "dotplot",
+ (),
+ {
+ "features": ["GeneA"],
+ "group_by": "cluster",
+ "expression_cutoff": 1.5,
+ },
+ ),
+ (
+ "matrixplot",
+ (),
+ {"features": ["GeneA"], "group_by": "cluster", "value": "fraction"},
+ ),
+ ("composition", (), {"category_by": "cluster", "kind": "per_sample"}),
+ ("distribution", ("RNA_nCounts",), {"bins": 17}),
+ ("marker_heatmap", (), {"topn": 7, "linewidths": 0.25}),
+ (
+ "mapping_calibration",
+ ("atlas",),
+ {
+ "reference_class_group": "label",
+ "known_labels": ["a"],
+ },
+ ),
+ (
+ "mapping_confusion",
+ ("atlas",),
+ {
+ "reference_class_group": "label",
+ "known_labels": ["a"],
+ },
+ ),
+ (
+ "mapping_evidence",
+ ("atlas",),
+ {"reference_class_group": "label"},
+ ),
+ (
+ "mapping_projection",
+ ("atlas",),
+ {"reference_layout_key": "RNA_UMAP"},
+ ),
+ (
+ "mapping_score",
+ ("atlas",),
+ {
+ "kind": "histogram",
+ },
+ ),
+ (
+ "cluster_connectivity",
+ (),
+ {
+ "group_by": "cluster",
+ "layout_key": "RNA_UMAP",
+ "minimum_edge_weight": 0.1,
+ },
+ ),
+ ("run_recipe", ("recipe.toml",), {"show": False}),
+ ("cluster_tree", (), {"width": 2.5}),
+ ("pseudotime_heatmap", (), {"vmax": 3.0}),
+ ],
+)
+def test_plot_accessor_forwards_to_canonical_function(
+ monkeypatch: pytest.MonkeyPatch,
+ name: str,
+ args: tuple[Any, ...],
+ kwargs: dict[str, Any],
+):
+ store = object.__new__(DataStore)
+ accessor = DataStorePlotAccessor(store)
+ sentinel = object()
+ calls: list[tuple[tuple[Any, ...], dict[str, Any]]] = []
+
+ def canonical(*call_args: Any, **call_kwargs: Any) -> object:
+ calls.append((call_args, call_kwargs))
+ return sentinel
+
+ monkeypatch.setattr(splt, name, canonical)
+ method = getattr(accessor, name)
+ bound = inspect.signature(method).bind(*args, **kwargs)
+ bound.apply_defaults()
+ expected_kwargs = dict(bound.arguments)
+ expected_kwargs.update(expected_kwargs.pop("heatmap_kwargs", {}))
+ expected_args = [store]
+ if name == "distribution":
+ expected_args.append(expected_kwargs.pop("keys"))
+ elif name.startswith("mapping_"):
+ expected_args.append(expected_kwargs.pop("result"))
+ elif name == "run_recipe":
+ expected_args.append(expected_kwargs.pop("recipe"))
+
+ assert method(*args, **kwargs) is sentinel
+ assert calls == [(tuple(expected_args), expected_kwargs)]
+
+
+def test_datastore_plots_returns_a_fresh_store_bound_namespace():
+ store = object.__new__(DataStore)
+
+ first = store.plots
+ second = store.plots
+
+ assert first is not second
+ assert first._store is store
+ assert second._store is store
+ assert not hasattr(first, "__dict__")
+ assert store.__dict__ == {}
+
+
+def test_shallow_copy_gets_an_accessor_bound_to_the_copy():
+ store = object.__new__(DataStore)
+ original_accessor = store.plots
+
+ clone = copy(store)
+ clone_accessor = clone.plots
+
+ assert clone_accessor is not original_accessor
+ assert clone_accessor._store is clone
+
+
+@pytest.mark.parametrize("workspace", [None, "analysis"])
+@pytest.mark.parametrize("assay_name", ["plots", "summary"])
+def test_datastore_rejects_reserved_assay_before_store_mutation(
+ workspace: str | None,
+ assay_name: str,
+):
+ memory_store = MemoryStore()
+ root = zarr.open_group(store=memory_store, mode="w")
+ active = root if workspace is None else root.create_group(workspace)
+ assay = active.create_group(assay_name)
+ assay.attrs["is_assay"] = True
+
+ with pytest.raises(
+ ValueError,
+ match=rf"reserved for DataStore\.{assay_name}",
+ ):
+ DataStore(
+ memory_store,
+ default_assay=assay_name,
+ workspace=workspace,
+ min_features_per_cell=0,
+ min_cells_per_feature=0,
+ )
+
+ assert "defaultAssay" not in active.attrs
+ assert "assayTypes" not in active.attrs
diff --git a/tests/test_datastore_summary.py b/tests/test_datastore_summary.py
new file mode 100644
index 00000000..1bf2e10f
--- /dev/null
+++ b/tests/test_datastore_summary.py
@@ -0,0 +1,268 @@
+import json
+import shutil
+from dataclasses import FrozenInstanceError
+from pathlib import Path
+
+import numpy as np
+import pytest
+import zarr
+from scipy.sparse import csr_matrix
+
+import scarf
+from scarf.datastore.datastore import DataStore
+from scarf.metadata import MetaData
+from scarf.storage.artifact_writer import (
+ finish_artifact,
+ plan_artifact,
+ start_artifact,
+)
+from scarf.storage.refs import ArtifactRef, ArtifactScope
+from scarf.writers import SparseToZarr
+
+
+@pytest.fixture
+def summary_datastore(toy_crdir_writer: str, tmp_path: Path) -> DataStore:
+ location = tmp_path / "summary.zarr"
+ shutil.copytree(toy_crdir_writer, location)
+ return DataStore(
+ str(location),
+ default_assay="RNA",
+ nthreads=3,
+ mem_budget=64 * 1024 * 1024,
+ )
+
+
+def _create_minimal_datastore(location: Path) -> DataStore:
+ writer = SparseToZarr(
+ csr_matrix(np.array([[1, 0], [0, 2]], dtype=np.uint16)),
+ str(location),
+ cell_ids=["cell-1", "cell-2"],
+ feature_ids=["gene-1", "gene-2"],
+ mem_budget=64 * 1024 * 1024,
+ nthreads=1,
+ )
+ writer.dump()
+ return DataStore(
+ str(location),
+ default_assay="RNA",
+ min_features_per_cell=0,
+ min_cells_per_feature=0,
+ nthreads=1,
+ mem_budget=64 * 1024 * 1024,
+ )
+
+
+def _add_artifact(
+ datastore: DataStore,
+ *,
+ scope: ArtifactScope,
+ kind: str,
+ operation: str,
+ assay: str | None = None,
+ parameter: int,
+ complete: bool,
+) -> ArtifactRef:
+ planned = plan_artifact(
+ datastore.zw,
+ scope=scope,
+ assay=assay,
+ kind=kind,
+ operation=operation,
+ parameters={"branch": parameter},
+ inputs={},
+ execution_options={},
+ )
+ group = start_artifact(datastore.zw, planned)
+ if complete:
+ finish_artifact(group, planned)
+ return planned.ref
+
+
+def _file_snapshot(location: Path) -> dict[str, tuple[int, int]]:
+ return {
+ str(path.relative_to(location)): (path.stat().st_size, path.stat().st_mtime_ns)
+ for path in location.rglob("*")
+ if path.is_file()
+ }
+
+
+def test_summary_of_minimal_store_is_frozen_and_metadata_only(tmp_path: Path) -> None:
+ datastore = _create_minimal_datastore(tmp_path / "minimal.zarr")
+
+ summary = datastore.summary()
+
+ assert isinstance(summary, scarf.DataStoreSummary)
+ assert summary.zarr_mode == "r+"
+ assert summary.workspace is None
+ assert summary.default_assay == "RNA"
+ assert summary.scarf_version == scarf.__version__
+ assert summary.resources.memory_bytes == 64 * 1024 * 1024
+ assert summary.resources.workers == 1
+ assert summary.resources.storage_profile == "fast_local"
+ assert summary.total_cells == 2
+ assert summary.active_cells == 2
+ assert summary.cell_columns == tuple(sorted(summary.cell_columns))
+ assert len(summary.assays) == 1
+ assert summary.assays[0].name == "RNA"
+ assert summary.assays[0].assay_type == "RNA"
+ assert summary.assays[0].total_features == 2
+ assert summary.assays[0].active_features == 2
+ assert summary.assays[0].state is None
+ assert summary.artifacts == ()
+ assert not hasattr(summary, "__dict__")
+ with pytest.raises(FrozenInstanceError):
+ summary.total_cells = 3 # type: ignore[misc]
+
+
+def test_summary_counts_selections_blockwise(
+ summary_datastore: DataStore,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ expected_cells = sum(
+ block.active_global_indices.size
+ for block in summary_datastore.cells.iter_row_blocks(cell_key="I")
+ )
+ expected_features = {
+ assay_name: sum(
+ block.active_global_indices.size
+ for block in summary_datastore._get_assay(assay_name).feats.iter_row_blocks(
+ cell_key="I"
+ )
+ )
+ for assay_name in summary_datastore.assay_names
+ }
+
+ def fail_materialization(*args: object, **kwargs: object) -> None:
+ raise AssertionError("summary materialized a complete metadata column")
+
+ monkeypatch.setattr(MetaData, "active_index", fail_materialization)
+ monkeypatch.setattr(MetaData, "fetch_all", fail_materialization)
+
+ summary = summary_datastore.summary()
+
+ assert summary.active_cells == expected_cells
+ assert {
+ assay.name: assay.active_features for assay in summary.assays
+ } == expected_features
+
+
+def test_summary_groups_multiple_assays_and_artifact_branches(
+ summary_datastore: DataStore,
+) -> None:
+ complete_normalized = _add_artifact(
+ summary_datastore,
+ scope="assay",
+ assay="RNA",
+ kind="normalized",
+ operation="run_normalization",
+ parameter=2,
+ complete=True,
+ )
+ alternate_normalized = _add_artifact(
+ summary_datastore,
+ scope="assay",
+ assay="RNA",
+ kind="normalized",
+ operation="run_normalization",
+ parameter=1,
+ complete=True,
+ )
+ incomplete_reduction = _add_artifact(
+ summary_datastore,
+ scope="assay",
+ assay="RNA",
+ kind="reduction",
+ operation="run_pca",
+ parameter=1,
+ complete=False,
+ )
+ integrated = _add_artifact(
+ summary_datastore,
+ scope="datastore",
+ kind="integrated_graph",
+ operation="integrate_assays",
+ parameter=1,
+ complete=True,
+ )
+
+ summary = summary_datastore.summary()
+ rna = next(assay for assay in summary.assays if assay.name == "RNA")
+
+ assert tuple(assay.name for assay in summary.assays) == tuple(
+ sorted(summary_datastore.assay_names)
+ )
+ assert {assay.name for assay in summary.assays} == {"ADT", "HTO", "RNA"}
+ assert {artifact.ref for artifact in rna.artifacts} == {
+ complete_normalized,
+ alternate_normalized,
+ incomplete_reduction,
+ }
+ assert tuple(
+ (artifact.ref.kind, artifact.ref.artifact_id) for artifact in rna.artifacts
+ ) == tuple(
+ sorted(
+ (artifact.ref.kind, artifact.ref.artifact_id) for artifact in rna.artifacts
+ )
+ )
+ assert (
+ next(
+ artifact
+ for artifact in rna.artifacts
+ if artifact.ref == incomplete_reduction
+ ).complete
+ is False
+ )
+ assert all(
+ artifact.operation == "run_normalization"
+ for artifact in rna.artifacts
+ if artifact.ref.kind == "normalized"
+ )
+ assert tuple(artifact.ref for artifact in summary.artifacts) == (integrated,)
+ assert summary.artifacts[0].operation == "integrate_assays"
+ assert all(
+ artifact.ref.scope == "assay"
+ for assay in summary.assays
+ for artifact in assay.artifacts
+ )
+ assert all(artifact.ref.scope == "datastore" for artifact in summary.artifacts)
+
+
+def test_summary_to_dict_is_deterministic_json_and_omits_locations(
+ summary_datastore: DataStore,
+) -> None:
+ first = summary_datastore.summary().to_dict()
+ second = summary_datastore.summary().to_dict()
+
+ assert first == second
+ serialized = json.dumps(first, allow_nan=False, sort_keys=True)
+ assert str(summary_datastore.zarr_loc) not in serialized
+ assert "zarr_loc" not in serialized
+ assert "storage_options" not in serialized
+ assert "credentials" not in serialized
+ assert "private_bucket" not in serialized
+
+
+def test_summary_does_not_mutate_read_only_store(
+ tmp_path: Path,
+) -> None:
+ location = tmp_path / "read-only.zarr"
+ _create_minimal_datastore(location)
+ root = zarr.open_group(str(location), mode="r+")
+ root["RNA"].attrs.pop("dataset_fingerprint", None)
+ del root
+
+ datastore = DataStore(
+ str(location),
+ default_assay="RNA",
+ zarr_mode="r",
+ nthreads=1,
+ mem_budget=64 * 1024 * 1024,
+ )
+ before = _file_snapshot(location)
+
+ summary = datastore.summary()
+
+ assert _file_snapshot(location) == before
+ assert all(assay.dataset_fingerprint is None for assay in summary.assays)
+ reopened = zarr.open_group(str(location), mode="r")
+ assert "dataset_fingerprint" not in reopened["RNA"].attrs
diff --git a/tests/test_docs_cache.py b/tests/test_docs_cache.py
new file mode 100644
index 00000000..1100f7f1
--- /dev/null
+++ b/tests/test_docs_cache.py
@@ -0,0 +1,1213 @@
+import copy
+import io
+import shutil
+import tarfile
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+
+try:
+ import nbformat
+ from jupyter_cache import get_cache
+ from jupyter_cache.base import CacheBundleIn
+ from jupyter_cache.cache.main import NbArtifacts
+
+ import docs.execute_vignette as cache_tools
+ import docs.modal_cache as modal_cache
+ from docs.execute_all_vignettes import (
+ ExecutionBatchError,
+ _prepare_resume,
+ _record_result,
+ _valid_resume_uris,
+ execute_and_publish,
+ prune_and_publish,
+ )
+ from docs.execute_vignette import (
+ CacheBuildError,
+ CacheValidationError,
+ ProgressRenderError,
+ build_candidate,
+ close_cache,
+ discover_sources,
+ execution_fingerprint,
+ freeze_progress_outputs,
+ publish_candidate,
+ validate_cache,
+ validate_progress_outputs,
+ )
+ from docs.modal_cache import (
+ CacheTransportError,
+ SpawnedPageRunner,
+ await_page_cache,
+ pack_page_cache,
+ restore_page_cache,
+ )
+except ImportError:
+ pytest.skip("documentation dependencies are not installed", allow_module_level=True)
+
+
+def _write_source(
+ docs_root: Path,
+ name: str,
+ *,
+ code: str = "value = 1",
+ raises_exception: bool = False,
+) -> Path:
+ path = docs_root / "source" / f"{name}.md"
+ path.parent.mkdir(parents=True, exist_ok=True)
+ tags = ":tags: [raises-exception]\n\n" if raises_exception else ""
+ path.write_text(
+ "---\n"
+ "jupytext:\n"
+ " text_representation:\n"
+ " extension: .md\n"
+ " format_name: myst\n"
+ "kernelspec:\n"
+ " display_name: Python 3\n"
+ " language: python\n"
+ " name: python3\n"
+ "---\n\n"
+ f"# {name}\n\n"
+ "```{code-cell} ipython3\n"
+ f"{tags}"
+ f"{code}\n"
+ "```\n",
+ encoding="utf-8",
+ )
+ return path
+
+
+def _source(docs_root: Path, name: str):
+ return next(
+ source
+ for source in discover_sources(docs_root / "source", docs_root)
+ if source.path.stem == name
+ )
+
+
+def _executed_notebook(
+ source,
+ *,
+ text: str = "ok\n",
+ error: bool = False,
+) -> nbformat.NotebookNode:
+ notebook = copy.deepcopy(source.notebook)
+ execution_count = 1
+ for cell in notebook.cells:
+ if cell.cell_type != "code":
+ continue
+ cell.execution_count = execution_count
+ execution_count += 1
+ if error:
+ cell.outputs = [
+ nbformat.v4.new_output(
+ "error",
+ ename="ValueError",
+ evalue="expected",
+ traceback=["ValueError: expected"],
+ )
+ ]
+ else:
+ cell.outputs = [nbformat.v4.new_output("stream", name="stdout", text=text)]
+ return notebook
+
+
+def _first_code_cell(notebook: nbformat.NotebookNode) -> nbformat.NotebookNode:
+ return next(cell for cell in notebook.cells if cell.cell_type == "code")
+
+
+def _widget_progress_notebook(
+ source,
+ *,
+ value: float = 1.0,
+ maximum: float = 1.0,
+ root_model: str = "HBoxModel",
+ bar_style: str = "success",
+) -> nbformat.NotebookNode:
+ notebook = _executed_notebook(source)
+ root_id = "root"
+ label_id = "label"
+ progress_id = "progress"
+ detail_id = "detail"
+ digest = "a" * 64
+ label = f"Writing data to artifacts/normalized/{digest}/data: 100%| "
+ _first_code_cell(notebook).outputs = [
+ nbformat.v4.new_output(
+ "display_data",
+ data={
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": root_id,
+ "version_major": 2,
+ "version_minor": 0,
+ },
+ "text/plain": "widget progress",
+ },
+ )
+ ]
+ notebook.metadata["widgets"] = {
+ "application/vnd.jupyter.widget-state+json": {
+ "state": {
+ root_id: {
+ "model_name": root_model,
+ "state": {
+ "children": [
+ f"IPY_MODEL_{label_id}",
+ f"IPY_MODEL_{progress_id}",
+ f"IPY_MODEL_{detail_id}",
+ ]
+ },
+ },
+ label_id: {
+ "model_name": "HTMLModel",
+ "state": {"value": label},
+ },
+ progress_id: {
+ "model_name": "FloatProgressModel",
+ "state": {
+ "bar_style": bar_style,
+ "max": maximum,
+ "min": 0.0,
+ "value": value,
+ },
+ },
+ detail_id: {
+ "model_name": "HTMLModel",
+ "state": {"value": "1/1"},
+ },
+ },
+ "version_major": 2,
+ "version_minor": 0,
+ }
+ }
+ return notebook
+
+
+def _cache_output(
+ cache_path: Path,
+ source,
+ *,
+ text: str = "ok\n",
+ error: bool = False,
+ uri: str | None = None,
+ artifact_root: Path | None = None,
+ notebook: nbformat.NotebookNode | None = None,
+) -> None:
+ artifacts = None
+ if artifact_root is not None:
+ artifact = artifact_root / "nested" / "result.txt"
+ artifact.parent.mkdir(parents=True, exist_ok=True)
+ artifact.write_text("artifact", encoding="utf-8")
+ artifacts = NbArtifacts([artifact], in_folder=artifact_root)
+
+ cache = get_cache(cache_path)
+ try:
+ cache.cache_notebook_bundle(
+ CacheBundleIn(
+ notebook or _executed_notebook(source, text=text, error=error),
+ uri or source.uri,
+ artifacts=artifacts,
+ data={"execution_seconds": 0.01},
+ ),
+ check_validity=True,
+ overwrite=True,
+ )
+ finally:
+ close_cache(cache)
+
+
+def _output_text(cache_path: Path, source) -> str:
+ cache = get_cache(cache_path)
+ try:
+ record = cache.match_cache_notebook(source.notebook)
+ bundle = cache.get_cache_bundle(record.pk)
+ return bundle.nb.cells[0].outputs[0].get("text", "")
+ finally:
+ close_cache(cache)
+
+
+def _tree_bytes(path: Path) -> dict[str, bytes]:
+ return {
+ file.relative_to(path).as_posix(): file.read_bytes()
+ for file in sorted(path.rglob("*"))
+ if file.is_file()
+ }
+
+
+def _initialize_cache(path: Path) -> None:
+ cache = get_cache(path)
+ cache.db
+ close_cache(cache)
+
+
+def test_discovery_uses_myst_notebook_parser(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "executable")
+ prose = docs_root / "source" / "prose.md"
+ prose.write_text(
+ "# Prose\n\nThis mentions ``{code-cell}`` but is not a MyST notebook.\n",
+ encoding="utf-8",
+ )
+
+ sources = discover_sources(docs_root / "source", docs_root)
+
+ assert [source.uri for source in sources] == ["source/executable.md"]
+
+
+def test_empty_cache_does_not_match_current_source(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ cache_path = docs_root / ".jupyter_cache"
+ _initialize_cache(cache_path)
+
+ with pytest.raises(CacheValidationError, match="no matching cache output"):
+ validate_cache(
+ cache_path,
+ source_dir=docs_root / "source",
+ docs_root=docs_root,
+ )
+
+
+def test_relocated_cache_uses_relative_uri(tmp_path: Path) -> None:
+ original = tmp_path / "original" / "docs"
+ _write_source(original, "page")
+ source = _source(original, "page")
+ _cache_output(original / ".jupyter_cache", source)
+
+ relocated = tmp_path / "relocated" / "docs"
+ shutil.copytree(original, relocated)
+ report = validate_cache(
+ relocated / ".jupyter_cache",
+ source_dir=relocated / "source",
+ docs_root=relocated,
+ )
+ cache = get_cache(relocated / ".jupyter_cache")
+ try:
+ assert cache.list_cache_records()[0].uri == "source/page.md"
+ finally:
+ close_cache(cache)
+
+ assert report.source_count == 1
+
+
+def test_validation_rejects_absolute_uri_and_project_rows(tmp_path: Path) -> None:
+ absolute_root = tmp_path / "absolute" / "docs"
+ _write_source(absolute_root, "page")
+ absolute_source = _source(absolute_root, "page")
+ absolute_cache = absolute_root / ".jupyter_cache"
+ _cache_output(
+ absolute_cache,
+ absolute_source,
+ uri=str(absolute_source.path),
+ )
+ with pytest.raises(CacheValidationError, match="not canonical"):
+ validate_cache(
+ absolute_cache,
+ source_dir=absolute_root / "source",
+ docs_root=absolute_root,
+ )
+
+ project_root = tmp_path / "project" / "docs"
+ _write_source(project_root, "page")
+ project_source = _source(project_root, "page")
+ project_cache = project_root / ".jupyter_cache"
+ _cache_output(project_cache, project_source)
+ project_notebook = tmp_path / "project.ipynb"
+ nbformat.write(project_source.notebook, project_notebook)
+ cache = get_cache(project_cache)
+ try:
+ cache.add_nb_to_project(str(project_notebook))
+ finally:
+ close_cache(cache)
+ with pytest.raises(CacheValidationError, match="project row"):
+ validate_cache(
+ project_cache,
+ source_dir=project_root / "source",
+ docs_root=project_root,
+ )
+
+
+def test_subset_update_preserves_unrequested_output(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "first", code="first = 1")
+ _write_source(docs_root, "second", code="second = 2")
+ sources = discover_sources(docs_root / "source", docs_root)
+ first, second = sources
+ snapshot = tmp_path / "snapshot"
+ results = tmp_path / "results"
+ candidate = tmp_path / "candidate"
+ _cache_output(snapshot, first, text="old first\n")
+ _cache_output(snapshot, second, text="old second\n")
+ _cache_output(results, first, text="new first\n")
+
+ build_candidate(
+ sources,
+ {first.uri},
+ results,
+ snapshot,
+ candidate,
+ )
+ validate_cache(
+ candidate,
+ source_dir=docs_root / "source",
+ docs_root=docs_root,
+ )
+
+ assert _output_text(candidate, first) == "new first\n"
+ assert _output_text(candidate, second) == "old second\n"
+
+
+def test_shared_source_hash_needs_one_record(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "first")
+ _write_source(docs_root, "second")
+ sources = discover_sources(docs_root / "source", docs_root)
+ assert sources[0].hashkey == sources[1].hashkey
+ cache_path = docs_root / ".jupyter_cache"
+ _cache_output(cache_path, sources[0])
+
+ report = validate_cache(
+ cache_path,
+ source_dir=docs_root / "source",
+ docs_root=docs_root,
+ )
+
+ assert report.source_count == 2
+ assert report.record_count == 1
+
+
+def test_bundle_import_preserves_nested_artifacts(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ snapshot = tmp_path / "snapshot"
+ candidate = tmp_path / "candidate"
+ _cache_output(
+ snapshot,
+ source,
+ artifact_root=tmp_path / "artifact-source",
+ )
+
+ build_candidate(
+ [source],
+ set(),
+ tmp_path / "unused",
+ snapshot,
+ candidate,
+ )
+
+ artifact = (
+ candidate / "executed" / source.hashkey / "artifacts" / "nested" / "result.txt"
+ )
+ assert artifact.read_text(encoding="utf-8") == "artifact"
+
+
+def test_completed_widget_progress_freezes_to_static_html(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ notebook = _widget_progress_notebook(source, value=0.8)
+
+ assert freeze_progress_outputs(notebook, source.uri) == 1
+
+ output = _first_code_cell(notebook).outputs[0]
+ assert "widgets" not in notebook.metadata
+ assert "application/vnd.jupyter.widget-view+json" not in output.data
+ assert ' None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+
+ with pytest.raises(ProgressRenderError, match="incomplete"):
+ freeze_progress_outputs(
+ _widget_progress_notebook(
+ source,
+ value=9,
+ maximum=10,
+ bar_style="",
+ ),
+ source.uri,
+ )
+ with pytest.raises(ProgressRenderError, match="unsupported root model"):
+ freeze_progress_outputs(
+ _widget_progress_notebook(source, root_model="ButtonModel"),
+ source.uri,
+ )
+
+
+def test_fragmented_terminal_progress_preserves_regular_streams(
+ tmp_path: Path,
+) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ notebook = _executed_notebook(source)
+ _first_code_cell(notebook).outputs = [
+ nbformat.v4.new_output(
+ "stream",
+ name="stderr",
+ text="Working: 0%| 0/2 [00:00]\rWork",
+ ),
+ nbformat.v4.new_output("stream", name="stdout", text="keep this\n"),
+ nbformat.v4.new_output(
+ "stream",
+ name="stderr",
+ text="ing: 100%|##| 2/2 [00:01]\n",
+ ),
+ ]
+
+ assert freeze_progress_outputs(notebook, source.uri) == 1
+
+ outputs = _first_code_cell(notebook).outputs
+ assert outputs[0].output_type == "stream"
+ assert outputs[0].name == "stdout"
+ assert outputs[0].text == "keep this\n"
+ assert outputs[1].output_type == "display_data"
+ assert outputs[1].data["text/plain"] == "Working: 2 / 2 complete"
+
+
+def test_terminal_progress_applies_backspace_and_ansi_controls(
+ tmp_path: Path,
+) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ notebook = _executed_notebook(source)
+ _first_code_cell(notebook).outputs = [
+ nbformat.v4.new_output(
+ "stream",
+ name="stderr",
+ text="\x1b[32mTask: 10X\b0%|##| 2/2\x1b[0m\n",
+ )
+ ]
+
+ assert freeze_progress_outputs(notebook, source.uri) == 1
+ assert _first_code_cell(notebook).outputs[0].data["text/plain"] == (
+ "Task: 2 / 2 complete"
+ )
+
+
+def test_incomplete_terminal_progress_is_rejected(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ notebook = _executed_notebook(source)
+ _first_code_cell(notebook).outputs = [
+ nbformat.v4.new_output(
+ "stream",
+ name="stderr",
+ text="Working: 75%|###| 3/4\r",
+ )
+ ]
+
+ with pytest.raises(ProgressRenderError, match="incomplete"):
+ freeze_progress_outputs(notebook, source.uri)
+
+
+def test_progress_validation_rejects_raw_widget_state(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+
+ with pytest.raises(ProgressRenderError, match="widget state remains"):
+ validate_progress_outputs(_widget_progress_notebook(source), source.uri)
+
+
+def test_frozen_progress_survives_modal_cache_transport(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ notebook = _widget_progress_notebook(source)
+ freeze_progress_outputs(notebook, source.uri)
+ original = tmp_path / "original"
+ restored = docs_root / ".jupyter_cache"
+ _cache_output(original, source, notebook=notebook)
+
+ restore_page_cache(
+ (source.uri, source.hashkey, pack_page_cache(original)),
+ restored,
+ expected_uri=source.uri,
+ expected_hashkey=source.hashkey,
+ )
+
+ validate_cache(
+ restored,
+ source_dir=docs_root / "source",
+ docs_root=docs_root,
+ )
+
+
+def test_modal_page_cache_archive_round_trip(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ original = tmp_path / "original"
+ restored = docs_root / ".jupyter_cache"
+ _cache_output(
+ original,
+ source,
+ artifact_root=tmp_path / "artifact-source",
+ )
+
+ payload = (source.uri, source.hashkey, pack_page_cache(original))
+ restore_page_cache(
+ payload,
+ restored,
+ expected_uri=source.uri,
+ expected_hashkey=source.hashkey,
+ )
+
+ validate_cache(
+ restored,
+ source_dir=docs_root / "source",
+ docs_root=docs_root,
+ )
+ artifact = (
+ restored / "executed" / source.hashkey / "artifacts" / "nested" / "result.txt"
+ )
+ assert artifact.read_text(encoding="utf-8") == "artifact"
+
+
+def test_modal_page_cache_rejects_mismatched_identity(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ original = tmp_path / "original"
+ _cache_output(original, source)
+ archive = pack_page_cache(original)
+
+ with pytest.raises(CacheTransportError, match="does not match"):
+ restore_page_cache(
+ ("source/other.md", source.hashkey, archive),
+ tmp_path / "wrong-uri",
+ expected_uri=source.uri,
+ expected_hashkey=source.hashkey,
+ )
+ with pytest.raises(CacheTransportError, match="does not match"):
+ restore_page_cache(
+ (source.uri, "wrong-hash", archive),
+ tmp_path / "wrong-hash",
+ expected_uri=source.uri,
+ expected_hashkey=source.hashkey,
+ )
+
+ assert not (tmp_path / "wrong-uri").exists()
+ assert not (tmp_path / "wrong-hash").exists()
+
+
+def test_modal_page_cache_rejects_unsafe_archive_member(tmp_path: Path) -> None:
+ stream = io.BytesIO()
+ with tarfile.open(fileobj=stream, mode="w:gz") as archive:
+ member = tarfile.TarInfo("../outside")
+ member.size = 1
+ archive.addfile(member, io.BytesIO(b"x"))
+
+ with pytest.raises(CacheTransportError, match="unsafe path"):
+ restore_page_cache(
+ ("source/page.md", "hash", stream.getvalue()),
+ tmp_path / "restored",
+ expected_uri="source/page.md",
+ expected_hashkey="hash",
+ )
+
+ assert not (tmp_path / "outside").exists()
+ assert not (tmp_path / "restored").exists()
+
+
+@pytest.mark.parametrize(
+ "status",
+ ["FAILURE", "INIT_FAILURE", "TERMINATED", "TIMEOUT"],
+)
+def test_modal_wait_rejects_terminal_failure_status(status: str) -> None:
+ class FailedCall:
+ object_id = "fc-test"
+
+ def get(self, timeout: float):
+ raise TimeoutError
+
+ def get_call_graph(self):
+ return [
+ SimpleNamespace(
+ function_call_id=self.object_id,
+ status=SimpleNamespace(name=status),
+ children=[],
+ )
+ ]
+
+ with pytest.raises(RuntimeError, match=f"status={status}"):
+ await_page_cache(
+ FailedCall(),
+ poll_seconds=1,
+ deadline_seconds=10,
+ )
+
+
+def test_modal_wait_honors_polling_deadline(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ observed_timeouts: list[float] = []
+
+ class PendingCall:
+ def get(self, timeout: float):
+ observed_timeouts.append(timeout)
+ raise TimeoutError
+
+ times = iter([0.0, 0.0, 0.0, 2.0])
+ monkeypatch.setattr(modal_cache.time, "monotonic", lambda: next(times))
+
+ with pytest.raises(TimeoutError, match="within 1 seconds"):
+ await_page_cache(
+ PendingCall(),
+ poll_seconds=0.25,
+ deadline_seconds=1,
+ )
+
+ assert observed_timeouts == [0.25]
+
+
+@pytest.mark.parametrize("corruption", ["stale", "orphan", "journal"])
+def test_validation_detects_stale_and_orphan_state(
+ tmp_path: Path,
+ corruption: str,
+) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ cache_path = docs_root / ".jupyter_cache"
+ _cache_output(cache_path, source)
+
+ if corruption == "stale":
+ _write_source(docs_root, "stale", code="stale = True")
+ stale = _source(docs_root, "stale")
+ _cache_output(cache_path, stale)
+ (docs_root / "source" / "stale.md").unlink()
+ elif corruption == "orphan":
+ (cache_path / "executed" / ("f" * 32)).mkdir()
+ else:
+ (cache_path / "global.db-journal").write_bytes(b"journal")
+
+ with pytest.raises(CacheValidationError):
+ validate_cache(
+ cache_path,
+ source_dir=docs_root / "source",
+ docs_root=docs_root,
+ )
+
+
+def test_only_tagged_cells_may_contain_error_outputs(tmp_path: Path) -> None:
+ expected_root = tmp_path / "expected" / "docs"
+ _write_source(expected_root, "page", raises_exception=True)
+ expected = _source(expected_root, "page")
+ _cache_output(expected_root / ".jupyter_cache", expected, error=True)
+ validate_cache(
+ expected_root / ".jupyter_cache",
+ source_dir=expected_root / "source",
+ docs_root=expected_root,
+ )
+
+ unexpected_root = tmp_path / "unexpected" / "docs"
+ _write_source(unexpected_root, "page")
+ unexpected = _source(unexpected_root, "page")
+ _cache_output(unexpected_root / ".jupyter_cache", unexpected, error=True)
+ with pytest.raises(CacheValidationError, match="Unexpected error output"):
+ validate_cache(
+ unexpected_root / ".jupyter_cache",
+ source_dir=unexpected_root / "source",
+ docs_root=unexpected_root,
+ )
+
+
+def test_validation_rejects_incomplete_execution_counts(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ cache_path = docs_root / ".jupyter_cache"
+ _cache_output(cache_path, source)
+ notebook_path = cache_path / "executed" / source.hashkey / "base.ipynb"
+ notebook = nbformat.read(notebook_path, nbformat.NO_CONVERT)
+ notebook.cells[0].execution_count = None
+ nbformat.write(notebook, notebook_path)
+
+ with pytest.raises(CacheValidationError, match="execution counts are incomplete"):
+ validate_cache(
+ cache_path,
+ source_dir=docs_root / "source",
+ docs_root=docs_root,
+ )
+
+
+def test_execution_failure_preserves_target(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ target = docs_root / ".jupyter_cache"
+ _cache_output(target, source, text="old\n")
+ before = _tree_bytes(target)
+
+ def fail_runner(source, cache_path):
+ raise RuntimeError("execution failed")
+
+ with pytest.raises(ExecutionBatchError):
+ execute_and_publish(
+ ["page"],
+ docs_root=docs_root,
+ page_runner=fail_runner,
+ )
+
+ assert _tree_bytes(target) == before
+
+
+def test_import_failure_preserves_target(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ target = docs_root / ".jupyter_cache"
+ _cache_output(target, source, text="old\n")
+ before = _tree_bytes(target)
+
+ def empty_runner(source, cache_path):
+ _initialize_cache(cache_path)
+ return cache_path
+
+ with pytest.raises(ExecutionBatchError):
+ execute_and_publish(
+ ["page"],
+ docs_root=docs_root,
+ page_runner=empty_runner,
+ )
+
+ assert _tree_bytes(target) == before
+
+
+def test_candidate_validation_failure_preserves_target(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ target = docs_root / ".jupyter_cache"
+ _cache_output(target, source, text="old\n")
+ before = _tree_bytes(target)
+
+ def invalid_runner(source, cache_path):
+ _cache_output(cache_path, source, error=True)
+ return cache_path
+
+ with pytest.raises(CacheValidationError):
+ execute_and_publish(
+ ["page"],
+ docs_root=docs_root,
+ page_runner=invalid_runner,
+ )
+
+ assert _tree_bytes(target) == before
+
+
+def test_publication_failure_restores_target(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ target = docs_root / ".jupyter_cache"
+ candidate = docs_root / ".jupyter_cache.candidate-test"
+ _cache_output(target, source, text="old\n")
+ _cache_output(candidate, source, text="new\n")
+ before = _tree_bytes(target)
+ original_rename = cache_tools._rename_path
+
+ def fail_candidate_rename(source_path: Path, destination: Path) -> None:
+ if source_path == candidate and destination == target:
+ raise OSError("replacement failed")
+ original_rename(source_path, destination)
+
+ monkeypatch.setattr(cache_tools, "_rename_path", fail_candidate_rename)
+
+ with pytest.raises(OSError, match="replacement failed"):
+ publish_candidate(candidate, target)
+
+ assert _tree_bytes(target) == before
+ assert not cache_tools.backup_path(target).exists()
+
+
+def test_next_run_recovers_interrupted_swap(tmp_path: Path) -> None:
+ target = tmp_path / ".jupyter_cache"
+ target.mkdir()
+ (target / "sentinel").write_text("old", encoding="utf-8")
+ backup = cache_tools.backup_path(target)
+ cache_tools._rename_path(target, backup)
+
+ assert cache_tools.recover_interrupted_swap(target)
+ assert (target / "sentinel").read_text(encoding="utf-8") == "old"
+ assert not backup.exists()
+
+
+def test_resume_invalidates_source_and_execution_fingerprints(
+ tmp_path: Path,
+) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ resume_dir = docs_root / ".jupyter_cache.resume"
+ page_cache = tmp_path / "page-cache"
+ _cache_output(page_cache, source)
+ manifest = _prepare_resume(
+ resume_dir,
+ "fingerprint-one",
+ {source.uri},
+ use_resume=False,
+ runner_identity="local",
+ )
+ _record_result(source, page_cache, resume_dir, manifest)
+ assert _valid_resume_uris(
+ [source],
+ {source.uri},
+ resume_dir,
+ manifest,
+ ) == {source.uri}
+
+ _write_source(docs_root, "page", code="value = 2")
+ changed = _source(docs_root, "page")
+ assert not _valid_resume_uris(
+ [changed],
+ {changed.uri},
+ resume_dir,
+ manifest,
+ )
+
+ replaced = _prepare_resume(
+ resume_dir,
+ "fingerprint-two",
+ {changed.uri},
+ use_resume=True,
+ runner_identity="local",
+ )
+ assert replaced["entries"] == {}
+ assert not (resume_dir / "cache").exists()
+
+
+def test_resume_invalidates_runner_identity(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ resume_dir = docs_root / ".jupyter_cache.resume"
+ page_cache = tmp_path / "page-cache"
+ _cache_output(page_cache, source)
+ manifest = _prepare_resume(
+ resume_dir,
+ "shared-fingerprint",
+ {source.uri},
+ use_resume=False,
+ runner_identity="local",
+ )
+ _record_result(source, page_cache, resume_dir, manifest)
+
+ replaced = _prepare_resume(
+ resume_dir,
+ "shared-fingerprint",
+ {source.uri},
+ use_resume=True,
+ runner_identity="modal-python-3.14",
+ )
+
+ assert replaced["runnerIdentity"] == "modal-python-3.14"
+ assert replaced["entries"] == {}
+ assert not (resume_dir / "cache").exists()
+
+
+@pytest.mark.parametrize("filename", ["modal_cache.py", "modal_docs.py"])
+def test_modal_runner_files_participate_in_execution_fingerprint(
+ tmp_path: Path,
+ filename: str,
+) -> None:
+ repo_root = tmp_path / "repo"
+ docs_root = repo_root / "docs"
+ docs_root.mkdir(parents=True)
+ for runner_file in ("modal_cache.py", "modal_docs.py"):
+ (docs_root / runner_file).write_text("original\n", encoding="utf-8")
+ before = execution_fingerprint(repo_root, docs_root)
+
+ (docs_root / filename).write_text("changed\n", encoding="utf-8")
+
+ assert execution_fingerprint(repo_root, docs_root) != before
+
+
+def test_explicit_resume_reuses_matching_successes(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "first", code="first = 1")
+ _write_source(docs_root, "second", code="second = 2")
+ first, second = discover_sources(docs_root / "source", docs_root)
+ target = docs_root / ".jupyter_cache"
+ _cache_output(target, first, text="old first\n")
+ _cache_output(target, second, text="old second\n")
+
+ def interrupted_runner(source, cache_path):
+ if source.uri == second.uri:
+ raise RuntimeError("second failed")
+ _cache_output(cache_path, source, text="new first\n")
+ return cache_path
+
+ with pytest.raises(ExecutionBatchError):
+ execute_and_publish(
+ ["first", "second"],
+ docs_root=docs_root,
+ page_runner=interrupted_runner,
+ )
+
+ resumed_calls: list[str] = []
+
+ def resumed_runner(source, cache_path):
+ resumed_calls.append(source.uri)
+ _cache_output(cache_path, source, text="new second\n")
+ return cache_path
+
+ execute_and_publish(
+ ["first", "second"],
+ docs_root=docs_root,
+ resume=True,
+ page_runner=resumed_runner,
+ )
+
+ assert resumed_calls == [second.uri]
+ assert _output_text(target, first) == "new first\n"
+ assert _output_text(target, second) == "new second\n"
+
+
+def test_modal_fanout_spawns_before_wait_and_resumes_failures(
+ tmp_path: Path,
+) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "first", code="first = 1")
+ _write_source(docs_root, "second", code="second = 2")
+ first, second = discover_sources(docs_root / "source", docs_root)
+ target = docs_root / ".jupyter_cache"
+ _cache_output(target, first, text="old first\n")
+ _cache_output(target, second, text="old second\n")
+ before = _tree_bytes(target)
+
+ first_cache = tmp_path / "first-cache"
+ second_cache = tmp_path / "second-cache"
+ _cache_output(first_cache, first, text="new first\n")
+ _cache_output(second_cache, second, text="new second\n")
+ payloads = {
+ first.uri: (first.uri, first.hashkey, pack_page_cache(first_cache)),
+ second.uri: (second.uri, second.hashkey, pack_page_cache(second_cache)),
+ }
+
+ class FakeCall:
+ def __init__(
+ self,
+ outcome,
+ spawned: list[str],
+ observed_spawn_counts: list[int],
+ ) -> None:
+ self.outcome = outcome
+ self.spawned = spawned
+ self.observed_spawn_counts = observed_spawn_counts
+ self.cancelled = False
+
+ def get(self, timeout: float):
+ assert timeout > 0
+ self.observed_spawn_counts.append(len(self.spawned))
+ if isinstance(self.outcome, BaseException):
+ raise self.outcome
+ return self.outcome
+
+ def cancel(self) -> None:
+ self.cancelled = True
+
+ first_run_spawned: list[str] = []
+ observed_spawn_counts: list[int] = []
+
+ def first_run_spawn(source):
+ first_run_spawned.append(source.uri)
+ outcome = (
+ RuntimeError("second failed")
+ if source.uri == second.uri
+ else payloads[source.uri]
+ )
+ return FakeCall(outcome, first_run_spawned, observed_spawn_counts)
+
+ first_launcher = SpawnedPageRunner(first_run_spawn)
+ try:
+ with pytest.raises(ExecutionBatchError):
+ execute_and_publish(
+ ["first", "second"],
+ jobs=2,
+ docs_root=docs_root,
+ page_runner_factory=first_launcher.prepare,
+ warn_parallel_memory=False,
+ )
+ finally:
+ first_launcher.cancel_unclaimed()
+
+ assert first_run_spawned == [first.uri, second.uri]
+ assert observed_spawn_counts == [2, 2]
+ assert _tree_bytes(target) == before
+
+ resumed_spawned: list[str] = []
+ resumed_observed_counts: list[int] = []
+
+ def resumed_spawn(source):
+ resumed_spawned.append(source.uri)
+ return FakeCall(
+ payloads[source.uri],
+ resumed_spawned,
+ resumed_observed_counts,
+ )
+
+ resumed_launcher = SpawnedPageRunner(resumed_spawn)
+ try:
+ execute_and_publish(
+ ["first", "second"],
+ jobs=2,
+ resume=True,
+ docs_root=docs_root,
+ page_runner_factory=resumed_launcher.prepare,
+ warn_parallel_memory=False,
+ )
+ finally:
+ resumed_launcher.cancel_unclaimed()
+
+ assert resumed_spawned == [second.uri]
+ assert resumed_observed_counts == [1]
+ assert _output_text(target, first) == "new first\n"
+ assert _output_text(target, second) == "new second\n"
+
+
+def test_modal_submission_failure_cancels_submitted_calls(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "first", code="first = 1")
+ _write_source(docs_root, "second", code="second = 2")
+ first, second = discover_sources(docs_root / "source", docs_root)
+
+ class FakeCall:
+ def __init__(self) -> None:
+ self.cancelled = False
+
+ def cancel(self) -> None:
+ self.cancelled = True
+
+ first_call = FakeCall()
+
+ def spawn(source):
+ if source.uri == second.uri:
+ raise RuntimeError("submission failed")
+ return first_call
+
+ launcher = SpawnedPageRunner(spawn)
+ with pytest.raises(RuntimeError, match="submission failed"):
+ launcher.prepare([first, second])
+
+ assert first_call.cancelled
+
+
+def test_resume_without_a_staged_run_fails(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+
+ with pytest.raises(CacheBuildError, match="No resumable documentation run"):
+ execute_and_publish(
+ resume=True,
+ docs_root=docs_root,
+ page_runner=lambda source, cache_path: cache_path,
+ )
+
+
+def test_partial_run_fails_when_unrequested_source_has_no_match(
+ tmp_path: Path,
+) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "first", code="first = 1")
+ _write_source(docs_root, "second", code="second = 2")
+ first, second = discover_sources(docs_root / "source", docs_root)
+ target = docs_root / ".jupyter_cache"
+ _cache_output(target, first, text="old\n")
+ before = _tree_bytes(target)
+
+ def runner(source, cache_path):
+ _cache_output(cache_path, source, text="new\n")
+ return cache_path
+
+ with pytest.raises(CacheBuildError, match=second.uri):
+ execute_and_publish(
+ ["first"],
+ docs_root=docs_root,
+ page_runner=runner,
+ )
+
+ assert _tree_bytes(target) == before
+
+
+def test_prune_publishes_only_matching_current_sources(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "current", code="current = True")
+ _write_source(docs_root, "stale", code="stale = True")
+ current, stale = discover_sources(docs_root / "source", docs_root)
+ target = docs_root / ".jupyter_cache"
+ _cache_output(target, current)
+ _cache_output(target, stale)
+ (docs_root / "source" / "stale.md").unlink()
+ (target / "executed" / ("f" * 32)).mkdir()
+
+ report = prune_and_publish(docs_root)
+
+ assert report.source_count == 1
+ assert report.record_count == 1
+ validate_cache(
+ target,
+ source_dir=docs_root / "source",
+ docs_root=docs_root,
+ )
+
+
+def test_full_run_replaces_invalid_old_cache(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ target = docs_root / ".jupyter_cache"
+ _cache_output(target, source, text="old\n")
+ (target / "executed" / ("f" * 32)).mkdir()
+ calls: list[str] = []
+
+ def runner(source, cache_path):
+ calls.append(source.uri)
+ _cache_output(cache_path, source, text="new\n")
+ return cache_path
+
+ execute_and_publish(
+ full=True,
+ docs_root=docs_root,
+ page_runner=runner,
+ )
+
+ assert calls == [source.uri]
+ assert _output_text(target, source) == "new\n"
+ validate_cache(
+ target,
+ source_dir=docs_root / "source",
+ docs_root=docs_root,
+ )
+
+
+def test_validation_does_not_change_cache_bytes(tmp_path: Path) -> None:
+ docs_root = tmp_path / "docs"
+ _write_source(docs_root, "page")
+ source = _source(docs_root, "page")
+ cache_path = docs_root / ".jupyter_cache"
+ _cache_output(cache_path, source)
+ before = _tree_bytes(cache_path)
+
+ validate_cache(
+ cache_path,
+ source_dir=docs_root / "source",
+ docs_root=docs_root,
+ )
+
+ assert _tree_bytes(cache_path) == before
diff --git a/tests/test_docs_structure.py b/tests/test_docs_structure.py
new file mode 100644
index 00000000..8a9398a0
--- /dev/null
+++ b/tests/test_docs_structure.py
@@ -0,0 +1,90 @@
+import ast
+import re
+import tomllib
+from pathlib import Path
+
+
+_REPOSITORY_ROOT = Path(__file__).resolve().parents[1]
+_DOCS_SOURCE = _REPOSITORY_ROOT / "docs" / "source"
+_MARKDOWN_LINK = re.compile(r"\[[^\]]+\]\(([^)]+)\)")
+
+
+def test_llms_index_links_to_local_documentation() -> None:
+ contents = (_DOCS_SOURCE / "llms.txt").read_text()
+ targets = _MARKDOWN_LINK.findall(contents)
+
+ assert targets
+ for target in targets:
+ assert "://" not in target
+ page = target.split("#", maxsplit=1)[0]
+ assert page.endswith(".html")
+ source = _DOCS_SOURCE / f"{page.removesuffix('.html')}.md"
+ assert source.resolve().is_relative_to(_DOCS_SOURCE.resolve())
+ assert source.is_file(), target
+
+
+def test_agent_guide_and_llms_index_are_published() -> None:
+ toctree = (_DOCS_SOURCE / "toctree.yml").read_text()
+ conf = ast.parse((_DOCS_SOURCE / "conf.py").read_text())
+ html_extra_path = [
+ ast.literal_eval(node.value)
+ for node in conf.body
+ if isinstance(node, ast.Assign)
+ and any(
+ isinstance(target, ast.Name) and target.id == "html_extra_path"
+ for target in node.targets
+ )
+ ]
+
+ assert re.search(r"(?m)^\s*- file: analysis_with_agents\s*$", toctree)
+ assert html_extra_path == [["llms.txt"]]
+
+
+def test_agent_guide_is_linked_from_discovery_routes() -> None:
+ routes = {
+ _REPOSITORY_ROOT / "README.md": (
+ "https://scarf.readthedocs.io/en/latest/analysis_with_agents.html"
+ ),
+ _DOCS_SOURCE / "about.md": "{doc}`analysis_with_agents`",
+ _DOCS_SOURCE / "index.md": "{doc}`analysis_with_agents`",
+ _DOCS_SOURCE / "reference" / "faq.md": "{doc}`../analysis_with_agents`",
+ }
+
+ for source, expected_link in routes.items():
+ assert expected_link in source.read_text(), source
+
+
+def test_installation_uses_one_environment_for_install_and_runtime() -> None:
+ contents = (_DOCS_SOURCE / "installation.md").read_text()
+
+ assert "uv venv --python 3.12" in contents
+ assert 'python -c "import scarf; print(scarf.__version__)"' in contents
+ assert "uv pip install jupyterlab\njupyter lab" in contents
+ assert "uv run jupyter lab" not in contents
+ assert "pywin32" not in contents
+ assert contents.index("sudo apt install python3-dev python3-venv") < contents.index(
+ "python -m venv .venv"
+ )
+ assert "## Next steps" in contents
+
+
+def test_legacy_installation_url_and_package_links_are_preserved() -> None:
+ conf = ast.parse((_DOCS_SOURCE / "conf.py").read_text())
+ redirect_assignments = [
+ ast.literal_eval(node.value)
+ for node in conf.body
+ if isinstance(node, ast.Assign)
+ and any(
+ isinstance(target, ast.Name) and target.id == "redirects"
+ for target in node.targets
+ )
+ ]
+ metadata = tomllib.loads((_REPOSITORY_ROOT / "pyproject.toml").read_text())
+
+ assert len(redirect_assignments) == 1
+ assert redirect_assignments[0]["install"] == "installation.html"
+ assert metadata["project"]["urls"]["Installation"].endswith("/installation.html")
+ assert (
+ "https://scarf.readthedocs.io/en/latest/installation.html"
+ in (_REPOSITORY_ROOT / "README.md").read_text()
+ )
diff --git a/tests/test_embedding_reduction.py b/tests/test_embedding_reduction.py
new file mode 100644
index 00000000..1cd3ec83
--- /dev/null
+++ b/tests/test_embedding_reduction.py
@@ -0,0 +1,421 @@
+import numpy as np
+import pytest
+from sklearn.decomposition import IncrementalPCA, TruncatedSVD
+
+import scarf.embeddings.reduction as reduction_module
+from scarf.embeddings.reduction import (
+ _gram_pca_dispatch,
+ _mutable_fit_block,
+ _streaming_lsi_resident_bytes,
+ fit_incremental_pca,
+ fit_lsi,
+)
+from scarf.matrix import ChunkedArray
+from scarf.storage.budget import ResourceBudget
+
+
+class _ReadTrackingArray(ChunkedArray):
+ def __init__(self, values: np.ndarray, block_size: int) -> None:
+ super().__init__(values, block_size=block_size, nthreads=1, is_numpy=True)
+ self.read_ranges: list[tuple[int, int]] = []
+
+ def _materialize_range(self, start: int, end: int) -> np.ndarray:
+ self.read_ranges.append((start, end))
+ return super()._materialize_range(start, end)
+
+
+class _ResidentTrackingArray(ChunkedArray):
+ def __init__(
+ self,
+ values: np.ndarray,
+ *,
+ block_size: int,
+ resources: ResourceBudget,
+ ) -> None:
+ super().__init__(
+ values,
+ block_size=block_size,
+ nthreads=resources.workers,
+ resources=resources,
+ is_numpy=True,
+ )
+ self.resident_bytes: list[int] = []
+
+ def _stream_blocks(
+ self,
+ *,
+ nthreads,
+ msg,
+ prefetch,
+ row_mask,
+ resident_bytes=0,
+ ):
+ self.resident_bytes.append(resident_bytes)
+ yield from super()._stream_blocks(
+ nthreads=nthreads,
+ msg=msg,
+ prefetch=prefetch,
+ row_mask=row_mask,
+ resident_bytes=resident_bytes,
+ )
+
+
+def test_mutable_fit_block_reuses_owned_values_and_copies_shared_views():
+ owned = np.arange(24, dtype=np.float64).reshape(6, 4).copy()
+ assert _mutable_fit_block(owned) is owned
+
+ shared = owned[:, :2]
+ copied = _mutable_fit_block(shared)
+ assert copied is not shared
+ assert copied.flags.owndata
+ assert copied.flags.c_contiguous
+ np.testing.assert_array_equal(copied, shared)
+
+
+def test_incremental_pca_is_deterministic_across_chunked_input():
+ values = np.random.default_rng(42).normal(size=(24, 6))
+ original_values = values.copy()
+ data = ChunkedArray.from_numpy(values, block_size=8)
+ selected = np.ones(values.shape[0], dtype=bool)
+
+ first, first_model = fit_incremental_pca(
+ data,
+ dims=3,
+ batch_size=8,
+ use_for_pca=selected,
+ scale=None,
+ nthreads=1,
+ )
+ second, second_model = fit_incremental_pca(
+ data,
+ dims=3,
+ batch_size=8,
+ use_for_pca=selected,
+ scale=None,
+ nthreads=1,
+ )
+
+ assert first.shape == (6, 3)
+ np.testing.assert_allclose(first, second)
+ np.testing.assert_allclose(
+ first_model.explained_variance_ratio_,
+ second_model.explained_variance_ratio_,
+ )
+ np.testing.assert_array_equal(values, original_values)
+
+
+def test_gram_pca_matches_full_svd_and_preserves_model_contract():
+ values = np.random.default_rng(11).normal(size=(40, 7))
+ data = ChunkedArray.from_numpy(values, block_size=10)
+ selected = np.ones(values.shape[0], dtype=bool)
+
+ loadings, model = fit_incremental_pca(
+ data,
+ dims=3,
+ batch_size=10,
+ use_for_pca=selected,
+ scale=None,
+ nthreads=1,
+ )
+
+ centered = values - values.mean(axis=0)
+ _, singular_values, expected_components = np.linalg.svd(
+ centered,
+ full_matrices=False,
+ )
+ expected_components = expected_components[:4]
+ expected_variance = np.square(singular_values[:4]) / (len(values) - 1)
+ total_variance = values.var(axis=0, ddof=1).sum()
+
+ assert loadings.shape == (values.shape[1], 3)
+ assert model.components_.shape == (4, values.shape[1])
+ assert model.explained_variance_ratio_.shape == (4,)
+ np.testing.assert_allclose(loadings, model.components_[:-1].T)
+ np.testing.assert_allclose(
+ np.abs(model.components_ @ expected_components.T),
+ np.eye(4),
+ atol=1e-10,
+ )
+ np.testing.assert_allclose(model.explained_variance_, expected_variance)
+ np.testing.assert_allclose(
+ model.explained_variance_ratio_,
+ expected_variance / total_variance,
+ )
+ np.testing.assert_allclose(model.singular_values_, singular_values[:4])
+ np.testing.assert_allclose(model.mean_, values.mean(axis=0))
+ assert model.n_components_ == 4
+ assert model.n_features_in_ == values.shape[1]
+ assert model.n_samples_seen_ == values.shape[0]
+
+
+def test_gram_pca_matches_selected_scaled_rows():
+ values = np.random.default_rng(19).normal(size=(36, 5))
+ data = ChunkedArray.from_numpy(values, block_size=6)
+ selected = np.zeros(values.shape[0], dtype=bool)
+ selected[::2] = True
+ offset = np.linspace(-0.5, 0.5, values.shape[1])
+ divisor = np.linspace(1.0, 2.0, values.shape[1])
+
+ def scale(block: np.ndarray) -> np.ndarray:
+ return (block - offset) / divisor
+
+ loadings, model = fit_incremental_pca(
+ data,
+ dims=2,
+ batch_size=6,
+ use_for_pca=selected,
+ scale=scale,
+ nthreads=1,
+ )
+
+ expected_values = scale(values[selected])
+ centered = expected_values - expected_values.mean(axis=0)
+ _, _, expected_components = np.linalg.svd(centered, full_matrices=False)
+ expected_components = expected_components[:3]
+
+ np.testing.assert_allclose(
+ np.abs(model.components_ @ expected_components.T),
+ np.eye(3),
+ atol=1e-10,
+ )
+ np.testing.assert_allclose(model.mean_, expected_values.mean(axis=0))
+ np.testing.assert_allclose(loadings, model.components_[:-1].T)
+
+
+def test_gram_pca_dispatch_covers_both_fallback_reasons():
+ assert _gram_pca_dispatch(100, 100, 2) == (True, None)
+
+ use_gram, reason = _gram_pca_dispatch(101, 100, 2)
+ assert not use_gram
+ assert reason is not None and "exceed 100 rows per block" in reason
+
+ use_gram, reason = _gram_pca_dispatch(4097, 5000, 2)
+ assert not use_gram
+ assert reason is not None and "4096-feature limit" in reason
+
+ use_gram, reason = _gram_pca_dispatch(100, 100, 1)
+ assert not use_gram
+ assert reason == "the input has only one row block"
+
+
+def test_pca_logs_selected_solver(monkeypatch):
+ messages: list[str] = []
+
+ class RecordingLogger:
+ @staticmethod
+ def debug(message: str) -> None:
+ messages.append(message)
+
+ monkeypatch.setattr(reduction_module, "logger", RecordingLogger())
+ selected = np.ones(24, dtype=bool)
+ fit_incremental_pca(
+ ChunkedArray.from_numpy(
+ np.random.default_rng(21).normal(size=(24, 6)),
+ block_size=8,
+ ),
+ dims=2,
+ batch_size=8,
+ use_for_pca=selected,
+ scale=None,
+ nthreads=1,
+ )
+ assert "Fitting PCA with the Gram covariance solver" in messages.pop()
+
+ fit_incremental_pca(
+ ChunkedArray.from_numpy(
+ np.random.default_rng(22).normal(size=(24, 10)),
+ block_size=8,
+ ),
+ dims=2,
+ batch_size=8,
+ use_for_pca=selected,
+ scale=None,
+ nthreads=1,
+ )
+ captured = messages.pop()
+ assert "Fitting PCA with the IncrementalPCA solver" in captured
+ assert "10 features exceed 8 rows per block" in captured
+
+
+def test_incremental_pca_skips_blocks_without_selected_rows():
+ values = np.random.default_rng(3).normal(size=(24, 6))
+ data = _ReadTrackingArray(values, block_size=6)
+ selected = np.zeros(values.shape[0], dtype=bool)
+ selected[:6] = True
+ selected[-6:] = True
+
+ fit_incremental_pca(
+ data,
+ dims=2,
+ batch_size=6,
+ use_for_pca=selected,
+ scale=None,
+ nthreads=1,
+ )
+
+ assert data.read_ranges == [(0, 6), (18, 24)]
+
+
+def test_incremental_pca_propagates_final_fit_failure(monkeypatch):
+ values = np.random.default_rng(8).normal(size=(24, 10))
+ data = ChunkedArray.from_numpy(values, block_size=8)
+ selected = np.ones(values.shape[0], dtype=bool)
+ original_partial_fit = IncrementalPCA.partial_fit
+ calls = 0
+
+ def fail_final_fit(self, *args, **kwargs):
+ nonlocal calls
+ calls += 1
+ if calls == 3:
+ raise np.linalg.LinAlgError("final fit failed")
+ return original_partial_fit(self, *args, **kwargs)
+
+ monkeypatch.setattr(IncrementalPCA, "partial_fit", fail_final_fit)
+
+ with pytest.raises(np.linalg.LinAlgError, match="final fit failed"):
+ fit_incremental_pca(
+ data,
+ dims=2,
+ batch_size=8,
+ use_for_pca=selected,
+ scale=None,
+ nthreads=1,
+ )
+
+
+def test_lsi_returns_requested_components_and_mutates_reserved_params():
+ values = np.random.default_rng(7).uniform(size=(18, 5))
+ data = ChunkedArray.from_numpy(values, block_size=6)
+ params = {"n_iter": 4, "n_components": 99, "random_state": 99}
+
+ loadings = fit_lsi(
+ data,
+ dims=2,
+ skip_first=True,
+ params=params,
+ random_state=3,
+ nthreads=1,
+ )
+
+ assert loadings.shape == (5, 2)
+ assert params == {"n_iter": 4}
+ assert np.all(np.isfinite(loadings))
+
+
+def test_streaming_lsi_matches_exact_singular_subspace_and_is_deterministic():
+ rng = np.random.default_rng(19)
+ left, _ = np.linalg.qr(rng.normal(size=(60, 5)))
+ right, _ = np.linalg.qr(rng.normal(size=(20, 5)))
+ values = left @ np.diag([20.0, 12.0, 7.0, 3.0, 1.0]) @ right.T
+ values += 0.05 * rng.normal(size=values.shape)
+ data = ChunkedArray.from_numpy(values, block_size=7)
+ params = {
+ "solver": "streaming",
+ "n_iter": 3,
+ "n_oversamples": 4,
+ }
+
+ observed = fit_lsi(
+ data,
+ dims=4,
+ skip_first=False,
+ params=params,
+ random_state=11,
+ nthreads=1,
+ )
+ repeated = fit_lsi(
+ ChunkedArray.from_numpy(values, block_size=9),
+ dims=4,
+ skip_first=False,
+ params=params.copy(),
+ random_state=11,
+ nthreads=1,
+ )
+ _, _, right_vectors = np.linalg.svd(values, full_matrices=False)
+ expected = right_vectors[:4].T
+
+ np.testing.assert_allclose(
+ observed @ observed.T,
+ expected @ expected.T,
+ rtol=1e-6,
+ atol=1e-6,
+ )
+ np.testing.assert_allclose(observed, repeated, rtol=1e-10, atol=1e-10)
+ assert data.numblocks[0] > 1
+
+
+def test_streaming_lsi_never_stacks_all_input_blocks(monkeypatch):
+ values = np.random.default_rng(23).normal(size=(30, 12))
+
+ def fail_vstack(*_args, **_kwargs):
+ pytest.fail("Streaming LSI must not stack the full input matrix")
+
+ monkeypatch.setattr(np, "vstack", fail_vstack)
+ loadings = fit_lsi(
+ ChunkedArray.from_numpy(values, block_size=4),
+ dims=3,
+ skip_first=True,
+ params={"n_iter": 1},
+ random_state=3,
+ nthreads=1,
+ )
+
+ assert loadings.shape == (values.shape[1], 3)
+
+
+def test_streaming_lsi_reserves_accumulators_and_block_temporaries():
+ values = np.random.default_rng(31).normal(size=(30, 12))
+ resources = ResourceBudget(memoryBytes=10_000, workers=4)
+ data = _ResidentTrackingArray(
+ values,
+ block_size=4,
+ resources=resources,
+ )
+
+ fit_lsi(
+ data,
+ dims=3,
+ skip_first=True,
+ params={
+ "solver": "streaming",
+ "n_iter": 1,
+ "n_oversamples": 2,
+ },
+ random_state=5,
+ nthreads=4,
+ )
+
+ expected = _streaming_lsi_resident_bytes(data, width=6)
+ assert data.resident_bytes == [expected, expected, expected]
+ assert expected + data._block_owned_bytes() <= resources.memoryBytes
+
+
+def test_materialized_lsi_preserves_sklearn_compatibility():
+ values = np.random.default_rng(29).normal(size=(28, 9))
+ expected_model = TruncatedSVD(
+ n_components=4,
+ n_iter=3,
+ n_oversamples=2,
+ random_state=7,
+ )
+ expected_model.fit(values)
+
+ observed = fit_lsi(
+ ChunkedArray.from_numpy(values, block_size=5),
+ dims=3,
+ skip_first=True,
+ params={
+ "solver": "materialized",
+ "n_iter": 3,
+ "n_oversamples": 2,
+ },
+ random_state=7,
+ nthreads=1,
+ )
+
+ np.testing.assert_allclose(
+ observed,
+ expected_model.components_.T[:, 1:],
+ rtol=1e-12,
+ atol=1e-12,
+ )
diff --git a/tests/test_embedding_trajectory_contracts.py b/tests/test_embedding_trajectory_contracts.py
new file mode 100644
index 00000000..fc975f28
--- /dev/null
+++ b/tests/test_embedding_trajectory_contracts.py
@@ -0,0 +1,53 @@
+from dataclasses import fields
+from importlib.util import find_spec
+
+from scarf.embeddings.sgtsne import run_sgtsne
+from scarf.neighbors.stream import AnnStream
+from scarf.trajectory.feature_dynamics import validate_pseudotime_regressor
+from scarf.trajectory.results import PseudotimeScoreResult
+from tests.signature_contracts import signature_digest
+
+
+def test_embedding_and_trajectory_entry_point_signatures_are_stable():
+ methods = {
+ "AnnStream.__init__": AnnStream.__init__,
+ "run_sgtsne": run_sgtsne,
+ "validate_pseudotime_regressor": validate_pseudotime_regressor,
+ }
+ assert signature_digest(methods) == (
+ "b963e7139e72eea4843182060350a8c907e973014df48a84a13806ec6b435dfd"
+ )
+ assert [field.name for field in fields(PseudotimeScoreResult)] == [
+ "pseudotime_key",
+ "validity_key",
+ "assay",
+ "graph_cell_key",
+ "result_cell_key",
+ "feature_key",
+ "values",
+ "valid",
+ ]
+
+
+def test_moved_symbols_are_absent_from_old_hybrid_modules():
+ from scarf.features import markers
+ from scarf.datastore import datastore, graph_datastore
+
+ assert find_spec("scarf.knn_utils") is None
+ retired = {
+ markers: {"knn_clustering"},
+ datastore: {
+ "_scatter_feature_clusters",
+ "_validated_pseudotime_regressor",
+ },
+ graph_datastore: {
+ "_make_source_sink_vector",
+ "_random_walk_laplacian_transpose",
+ "_select_pseudotime_component",
+ "_truncated_pba_potential",
+ "_validate_source_sink_labels",
+ "_validate_source_sink_vector",
+ },
+ }
+ for module, names in retired.items():
+ assert names.isdisjoint(vars(module))
diff --git a/tests/test_enrichment_datastore.py b/tests/test_enrichment_datastore.py
new file mode 100644
index 00000000..fb562d37
--- /dev/null
+++ b/tests/test_enrichment_datastore.py
@@ -0,0 +1,648 @@
+import numba
+import numpy as np
+import pandas as pd
+import pytest
+import zarr
+
+from scarf.datastore.datastore import DataStore
+from scarf.assay import norm_lib_size_log
+from scarf.matrix import ChunkedArray
+from scarf.storage.artifacts import ArtifactRef
+from scarf.utils.arrays import array_digest
+from scarf.writers import (
+ create_cell_data,
+ create_zarr_count_assay,
+)
+
+pytestmark = pytest.mark.slow
+
+
+def _configure_enrichment_keys(datastore):
+ assay = datastore.RNA
+ active_cells = datastore.cells.active_index("I")[:8]
+ cell_mask = np.zeros(datastore.cells.N, dtype=bool)
+ cell_mask[active_cells] = True
+ datastore.cells.insert("enrichment_cells", cell_mask, overwrite=True)
+
+ all_names = np.asarray(assay.feats.fetch_all("names"))
+ selected_features = []
+ seen = set()
+ for index in assay.feats.active_index("I"):
+ normalized = str(all_names[index]).upper()
+ if normalized in seen:
+ continue
+ seen.add(normalized)
+ selected_features.append(int(index))
+ if len(selected_features) == 12:
+ break
+ selected_features = np.asarray(selected_features, dtype=np.int64)
+ feature_mask = np.zeros(assay.feats.N, dtype=bool)
+ feature_mask[selected_features] = True
+ assay.feats.insert("enrichment_features", feature_mask, overwrite=True)
+
+ target_index = selected_features[:6]
+ target_names = all_names[target_index].astype(str)
+ net = pd.DataFrame(
+ {
+ "source": ["Alpha"] * 3 + ["βeta"] * 3,
+ "target": target_names,
+ "weight": [1.0, -1.0, 2.0, 0.5, 1.5, -0.5],
+ }
+ )
+ return active_cells, selected_features, target_index, net
+
+
+def _enrichment_artifact(datastore, label):
+ index = datastore.RNA.z["enrichment"].attrs["artifact_results"]
+ ref = ArtifactRef.from_dict(index[label]["artifact"])
+ return ref, datastore.load_artifact(ref)
+
+
+def test_waggr_streaming_persistence_cache_and_lazy_subset(
+ datastore_ephemeral,
+ monkeypatch,
+):
+ cells, _, targets, net = _configure_enrichment_keys(datastore_ephemeral)
+
+ def fail_if_computed(*args, **kwargs):
+ raise AssertionError("WAGGR materialized the selected input matrix")
+
+ with monkeypatch.context() as compute_patch:
+ compute_patch.setattr(ChunkedArray, "compute", fail_if_computed)
+ result = datastore_ephemeral.run_waggr(
+ net,
+ "waggr_unicode",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+
+ raw = datastore_ephemeral.RNA.rawData[cells, :][:, targets].compute()
+ scalars = np.asarray(
+ datastore_ephemeral.cells.fetch_all("RNA_nCounts")[cells],
+ dtype=np.float64,
+ )
+ scalars[scalars == 0] = 1
+ normalized = float(datastore_ephemeral.RNA.sf) * raw / scalars[:, None]
+ expected = np.column_stack(
+ [
+ normalized[:, :3] @ np.array([1.0, -1.0, 2.0]) / 4.0,
+ normalized[:, 3:] @ np.array([0.5, 1.5, -0.5]) / 2.5,
+ ]
+ )
+
+ np.testing.assert_allclose(result.data.compute(), expected, rtol=1e-5)
+ np.testing.assert_array_equal(result.source_names, ["Alpha", "βeta"])
+ np.testing.assert_array_equal(result.source_sizes, [3, 3])
+ np.testing.assert_array_equal(result.cell_index, cells)
+ assert result.data.dtype == np.dtype(np.float32)
+ assert "/RNA/artifacts/enrichment_scores/" in f"/{result.storage_path}"
+ waggr_ref, slot = _enrichment_artifact(
+ datastore_ephemeral,
+ "waggr_unicode",
+ )
+ assert waggr_ref.kind == "enrichment_scores"
+ assert slot.attrs["complete"] is True
+ assert slot["scores"].dtype == np.dtype(np.float32)
+ assert slot.attrs["cell_digest"] == array_digest(cells.astype(np.int64))
+
+ def fail_if_streamed(*args, **kwargs):
+ raise AssertionError("cache hit streamed counts")
+
+ datastore_ephemeral.cells.insert(
+ "enrichment_cells_alias",
+ datastore_ephemeral.cells.fetch_all("enrichment_cells"),
+ overwrite=True,
+ )
+ datastore_ephemeral.RNA.feats.insert(
+ "enrichment_features_alias",
+ datastore_ephemeral.RNA.feats.fetch_all("enrichment_features"),
+ overwrite=True,
+ )
+ with monkeypatch.context() as cache_patch:
+ cache_patch.setattr(ChunkedArray, "stream_blocks", fail_if_streamed)
+ cached = datastore_ephemeral.run_waggr(
+ net,
+ "waggr_unicode",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+ alias = datastore_ephemeral.run_waggr(
+ net,
+ "waggr_alias",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+ key_alias = datastore_ephemeral.run_waggr(
+ net,
+ "waggr_key_alias",
+ cell_key="enrichment_cells_alias",
+ feat_key="enrichment_features_alias",
+ tmin=3,
+ )
+ assert cached.storage_path == result.storage_path
+ assert alias.storage_path == result.storage_path
+ assert key_alias.storage_path == result.storage_path
+ assert key_alias.cell_key == "enrichment_cells_alias"
+ assert key_alias.feature_key == "enrichment_features_alias"
+ enrichment_group = datastore_ephemeral.RNA.z["enrichment"]
+ artifact_results = dict(enrichment_group.attrs["artifact_results"])
+ alias_entry = dict(artifact_results["waggr_key_alias"])
+ alias_entry["cell_key"] = "missing_enrichment_cells"
+ artifact_results["waggr_key_alias"] = alias_entry
+ enrichment_group.attrs["artifact_results"] = artifact_results
+ recovered_alias = datastore_ephemeral.get_enrichment("waggr_key_alias")
+ assert recovered_alias.cell_key != "missing_enrichment_cells"
+ np.testing.assert_array_equal(
+ datastore_ephemeral.cells.fetch_all(recovered_alias.cell_key),
+ datastore_ephemeral.cells.fetch_all("enrichment_cells"),
+ )
+ alias_entry["cell_key"] = "enrichment_cells_alias"
+ artifact_results["waggr_key_alias"] = alias_entry
+ enrichment_group.attrs["artifact_results"] = artifact_results
+ assert (
+ _enrichment_artifact(datastore_ephemeral, "waggr_alias")[0]
+ == _enrichment_artifact(datastore_ephemeral, "waggr_unicode")[0]
+ )
+ original_ref = _enrichment_artifact(datastore_ephemeral, "waggr_unicode")[0]
+ invalidated = datastore_ephemeral.run_waggr(
+ net,
+ "waggr_unicode",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ invalidate_cache=True,
+ )
+ assert invalidated.storage_path != result.storage_path
+ assert _enrichment_artifact(datastore_ephemeral, "waggr_unicode")[0] != original_ref
+ preferred = datastore_ephemeral.run_waggr(
+ net,
+ "waggr_unicode",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+ assert preferred.storage_path == invalidated.storage_path
+
+ subset = datastore_ephemeral.get_enrichment(
+ "waggr_unicode",
+ sources=["βeta", "Alpha"],
+ )
+ np.testing.assert_array_equal(subset.source_names, ["βeta", "Alpha"])
+ np.testing.assert_allclose(subset.data.compute(), expected[:, [1, 0]], rtol=1e-5)
+ with pytest.raises(ValueError, match="non-empty"):
+ datastore_ephemeral.get_enrichment("waggr_unicode", sources=[])
+ with pytest.raises(ValueError, match="duplicate"):
+ datastore_ephemeral.get_enrichment(
+ "waggr_unicode",
+ sources=["Alpha", "Alpha"],
+ )
+ with pytest.raises(KeyError, match="not found"):
+ datastore_ephemeral.get_enrichment(
+ "waggr_unicode",
+ sources=["missing"],
+ )
+
+ logged = datastore_ephemeral.run_waggr(
+ net,
+ "waggr_logged",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ log_transform=True,
+ )
+ logged_values = np.log1p(normalized)
+ logged_expected = np.column_stack(
+ [
+ logged_values[:, :3] @ np.array([1.0, -1.0, 2.0]) / 4.0,
+ logged_values[:, 3:] @ np.array([0.5, 1.5, -0.5]) / 2.5,
+ ]
+ )
+ np.testing.assert_allclose(logged.data.compute(), logged_expected, rtol=1e-5)
+ logged_ref, logged_slot = _enrichment_artifact(
+ datastore_ephemeral,
+ "waggr_logged",
+ )
+ assert datastore_ephemeral.inspect_artifact(logged_ref).parameters["log_transform"]
+ assert "execution_digest" not in logged_slot.attrs
+ assert "execution_digest" not in slot.attrs
+
+ summed = datastore_ephemeral.run_waggr(
+ net,
+ "waggr_sum",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ mode="wsum",
+ tmin=3,
+ )
+ summed_expected = np.column_stack(
+ [
+ normalized[:, :3] @ np.array([1.0, -1.0, 2.0]),
+ normalized[:, 3:] @ np.array([0.5, 1.5, -0.5]),
+ ]
+ )
+ np.testing.assert_allclose(summed.data.compute(), summed_expected, rtol=1e-5)
+
+ original_norm = datastore_ephemeral.RNA.normMethod
+ datastore_ephemeral.RNA.normMethod = norm_lib_size_log
+ try:
+ with pytest.raises(ValueError, match="norm_lib_size"):
+ datastore_ephemeral.run_waggr(
+ net,
+ "wrong_normalization",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+ finally:
+ datastore_ephemeral.RNA.normMethod = original_norm
+ with pytest.raises(TypeError, match="RNAassay"):
+ datastore_ephemeral.run_waggr(
+ net,
+ "wrong_assay",
+ from_assay="assay2",
+ tmin=3,
+ )
+ for invalid_label in ("", ".", "..", "a/b", "a\\b", "bad\nlabel"):
+ with pytest.raises(ValueError):
+ datastore_ephemeral.run_waggr(net, invalid_label, tmin=3)
+
+ changed = net.copy()
+ changed.loc[0, "weight"] = 4.0
+ with pytest.raises(ValueError, match="different"):
+ datastore_ephemeral.run_waggr(
+ changed,
+ "waggr_unicode",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+
+ old_cell_index = result.cell_index.copy()
+ mutated_mask = np.zeros(datastore_ephemeral.cells.N, dtype=bool)
+ mutated_mask[cells[:4]] = True
+ datastore_ephemeral.cells.insert(
+ "enrichment_cells",
+ mutated_mask,
+ overwrite=True,
+ )
+ historical = datastore_ephemeral.get_enrichment("waggr_unicode")
+ np.testing.assert_array_equal(historical.cell_index, old_cell_index)
+ with pytest.raises(ValueError, match="different"):
+ datastore_ephemeral.run_waggr(
+ net,
+ "waggr_unicode",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+
+
+def test_aucell_ignores_weights_and_can_explicitly_replace_a_method(
+ datastore_ephemeral,
+ monkeypatch,
+):
+ _, features, _, net = _configure_enrichment_keys(datastore_ephemeral)
+ original_threads = numba.get_num_threads()
+ result = datastore_ephemeral.run_aucell(
+ net,
+ "activity",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ n_up=4,
+ tie_seed=7,
+ )
+ assert numba.get_num_threads() == original_threads
+
+ scores = result.data.compute()
+ assert scores.shape == (8, 2)
+ assert np.all((0 <= scores) & (scores <= 1))
+ _, slot = _enrichment_artifact(datastore_ephemeral, "activity")
+ assert slot.attrs["method"] == "aucell"
+ assert "rank_feature_index" in slot
+ assert sorted(np.asarray(slot["rank_feature_index"][:]).tolist()) == sorted(
+ features.tolist()
+ )
+ with pytest.raises(ValueError, match="different"):
+ datastore_ephemeral.run_aucell(
+ net,
+ "activity",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ n_up=4,
+ tie_seed=8,
+ )
+
+ weighted_differently = net.assign(weight=np.arange(1, 7, dtype=np.float64))
+
+ def fail_if_streamed(*args, **kwargs):
+ raise AssertionError("weight-only AUCell rerun streamed counts")
+
+ with monkeypatch.context() as cache_patch:
+ cache_patch.setattr(ChunkedArray, "stream_blocks", fail_if_streamed)
+ cached = datastore_ephemeral.run_aucell(
+ weighted_differently.sample(frac=1.0, random_state=3),
+ "activity",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ n_up=4,
+ tie_seed=7,
+ )
+ np.testing.assert_array_equal(cached.source_names, result.source_names)
+
+ with pytest.raises(ValueError, match="different"):
+ datastore_ephemeral.run_waggr(
+ net,
+ "activity",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+ replacement = datastore_ephemeral.run_waggr(
+ net,
+ "activity",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ overwrite=True,
+ )
+ assert replacement.method == "waggr"
+ _, active_slot = _enrichment_artifact(datastore_ephemeral, "activity")
+ assert "rank_feature_index" not in active_slot
+
+
+def test_incomplete_slots_and_read_only_access(
+ datastore_ephemeral,
+ monkeypatch,
+):
+ _, _, _, net = _configure_enrichment_keys(datastore_ephemeral)
+ original = datastore_ephemeral.run_waggr(
+ net,
+ "complete",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+ original_scores = original.data.compute()
+ original_ref, _original_slot = _enrichment_artifact(
+ datastore_ephemeral,
+ "complete",
+ )
+
+ read_only = DataStore(
+ datastore_ephemeral.zarr_loc,
+ default_assay="RNA",
+ zarr_mode="r",
+ )
+ loaded = read_only.get_enrichment("complete", sources=["Alpha"])
+ assert loaded.data.shape == (8, 1)
+ with pytest.raises(ValueError, match="zarr_mode='r\\+'"):
+ read_only.run_waggr(net, "read_only", tmin=3)
+
+ import scarf.datastore._operations.features as feature_operations
+
+ def interrupted_write(slot, **kwargs):
+ slot.attrs["method"] = "waggr"
+ raise RuntimeError("interrupted")
+
+ changed = net.copy()
+ changed.loc[0, "weight"] = 2.0
+ with monkeypatch.context() as fault:
+ fault.setattr(
+ feature_operations,
+ "_write_enrichment_slot",
+ interrupted_write,
+ )
+ with pytest.raises(RuntimeError, match="interrupted"):
+ datastore_ephemeral.run_waggr(
+ changed,
+ "complete",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ overwrite=True,
+ )
+ with pytest.raises(RuntimeError, match="interrupted"):
+ datastore_ephemeral.run_waggr(
+ changed,
+ "broken",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+
+ preserved = datastore_ephemeral.get_enrichment("complete")
+ np.testing.assert_array_equal(preserved.data.compute(), original_scores)
+ preserved_ref, preserved_slot = _enrichment_artifact(
+ datastore_ephemeral,
+ "complete",
+ )
+ assert preserved_ref == original_ref
+ assert "execution_digest" not in preserved_slot.attrs
+ with pytest.raises(KeyError, match="not found"):
+ datastore_ephemeral.get_enrichment("broken")
+
+ rebuilt = datastore_ephemeral.run_waggr(
+ changed,
+ "broken",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+ assert rebuilt.data.shape == (8, 2)
+
+ replacement = datastore_ephemeral.run_waggr(
+ changed,
+ "complete",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ overwrite=True,
+ )
+ assert replacement.data.shape == (8, 2)
+ replacement_ref, _ = _enrichment_artifact(
+ datastore_ephemeral,
+ "complete",
+ )
+ assert replacement_ref != original_ref
+ active_slot = datastore_ephemeral.zw[
+ datastore_ephemeral.inspect_artifact(replacement_ref).path
+ ]
+ active_slot["cell_index"][0] = (
+ int(active_slot["cell_index"][0]) + datastore_ephemeral.cells.N
+ )
+ with pytest.raises(ValueError, match="cell digest"):
+ datastore_ephemeral.get_enrichment("complete")
+
+
+def test_get_enrichment_reads_legacy_slots(datastore_ephemeral):
+ _, _, _, net = _configure_enrichment_keys(datastore_ephemeral)
+ expected = datastore_ephemeral.run_waggr(
+ net,
+ "artifact_source",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+ _, source = _enrichment_artifact(datastore_ephemeral, "artifact_source")
+ enrichment = datastore_ephemeral.RNA.z["enrichment"]
+ legacy = enrichment.create_group("legacy_waggr")
+ attr_names = (
+ "algorithm_version",
+ "cell_digest",
+ "cell_key",
+ "feature_digest",
+ "feat_key",
+ "layout",
+ "log_transform",
+ "method",
+ "network_digest",
+ "normalization",
+ "size_factor",
+ "tmin",
+ "waggr_mode",
+ )
+ attrs = {name: source.attrs[name] for name in attr_names}
+ attrs["execution_digest"] = "legacy-execution"
+
+ from scarf.datastore._operations.features import _write_enrichment_slot
+
+ _write_enrichment_slot(
+ legacy,
+ attrs=attrs,
+ score_batches=iter([np.asarray(source["scores"][:])]),
+ n_cells=source["scores"].shape[0],
+ source_names=np.asarray(source["source_names"][:]),
+ source_sizes=np.asarray(source["source_sizes"][:]),
+ cell_index=np.asarray(source["cell_index"][:]),
+ matched_feature_index=np.asarray(source["matched_feature_index"][:]),
+ rank_feature_index=None,
+ )
+
+ loaded = datastore_ephemeral.get_enrichment("legacy_waggr")
+ assert loaded.storage_path.endswith("RNA/enrichment/legacy_waggr")
+ np.testing.assert_array_equal(
+ loaded.data.compute(),
+ expected.data.compute(),
+ )
+
+
+def test_enrichment_rebuilds_wrong_score_dtype_and_validates_provenance(
+ datastore_ephemeral,
+):
+ _, _, _, net = _configure_enrichment_keys(datastore_ephemeral)
+ datastore_ephemeral.run_waggr(
+ net,
+ "validated",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+ original_ref, _ = _enrichment_artifact(datastore_ephemeral, "validated")
+ original_path = datastore_ephemeral.inspect_artifact(original_ref).path
+ original_group = datastore_ephemeral.zw[original_path]
+ scores = np.asarray(original_group["scores"][:], dtype=np.float64)
+ chunks = original_group["scores"].chunks
+ del original_group["scores"]
+ original_group.create_array("scores", data=scores, chunks=chunks)
+
+ datastore_ephemeral.run_waggr(
+ net,
+ "validated",
+ cell_key="enrichment_cells",
+ feat_key="enrichment_features",
+ tmin=3,
+ )
+ replacement_ref, _ = _enrichment_artifact(datastore_ephemeral, "validated")
+ assert replacement_ref != original_ref
+
+ replacement_path = datastore_ephemeral.inspect_artifact(replacement_ref).path
+ replacement_group = datastore_ephemeral.zw[replacement_path]
+ expected_network = replacement_group.attrs["network_digest"]
+ replacement_group.attrs["network_digest"] = "corrupt"
+ with pytest.raises(ValueError, match="network input"):
+ datastore_ephemeral.get_enrichment("validated")
+ replacement_group.attrs["network_digest"] = expected_network
+
+ provenance = dict(replacement_group.attrs["provenance"])
+ parameters = dict(provenance["parameters"])
+ normalization_method = dict(parameters["normalization_method"])
+ normalization_method["module"] = "corrupt.module"
+ parameters["normalization_method"] = normalization_method
+ provenance["parameters"] = parameters
+ replacement_group.attrs["provenance"] = provenance
+ with pytest.raises(ValueError, match="normalization provenance"):
+ datastore_ephemeral.get_enrichment("validated")
+
+
+def test_workspace_results_are_written_to_the_assay_shell(tmp_path):
+ path = tmp_path / "workspace.zarr"
+ root = zarr.open_group(str(path), mode="w")
+ create_cell_data(
+ root,
+ "ws",
+ ids=np.array([f"c{i}" for i in range(5)]),
+ names=np.array([f"c{i}" for i in range(5)]),
+ )
+ create_zarr_count_assay(
+ z=root,
+ assay_name="RNA",
+ workspace="ws",
+ n_cells=5,
+ feat_ids=np.array([f"f{i}" for i in range(6)]),
+ feat_names=np.array([f"g{i}" for i in range(6)]),
+ dtype="uint32",
+ )
+ root["matrices/RNA/counts"][:] = np.array(
+ [
+ [6, 5, 4, 3, 2, 1],
+ [1, 2, 3, 4, 5, 6],
+ [6, 1, 5, 2, 4, 3],
+ [3, 6, 1, 5, 2, 4],
+ [2, 5, 6, 1, 4, 3],
+ ],
+ dtype=np.uint32,
+ )
+ datastore = DataStore(
+ str(path),
+ default_assay="RNA",
+ workspace="ws",
+ min_features_per_cell=0,
+ min_cells_per_feature=0,
+ )
+ net = pd.DataFrame(
+ {
+ "source": ["A", "A", "A", "B", "B", "B"],
+ "target": [f"g{i}" for i in range(6)],
+ }
+ )
+
+ result = datastore.run_waggr(net, "workspace", tmin=3)
+ aucell = datastore.run_aucell(
+ net,
+ "workspace_aucell",
+ tmin=3,
+ n_up=4,
+ tie_seed=7,
+ )
+
+ assert result.storage_path.startswith("ws/RNA/artifacts/enrichment_scores/")
+ assert aucell.storage_path.startswith("ws/RNA/artifacts/enrichment_scores/")
+ np.testing.assert_allclose(
+ aucell.data.compute(),
+ [
+ [1.0, 0.0],
+ [0.0, 1.0],
+ [5.0 / 6.0, 1.0 / 6.0],
+ [0.5, 0.5],
+ [5.0 / 6.0, 1.0 / 6.0],
+ ],
+ )
+ assert "enrichment" in root["ws/RNA"]
+ assert "artifacts" in root["ws/RNA"]
+ assert "enrichment" not in root["matrices/RNA"]
diff --git a/tests/test_enrichment_methods.py b/tests/test_enrichment_methods.py
new file mode 100644
index 00000000..1c849d0e
--- /dev/null
+++ b/tests/test_enrichment_methods.py
@@ -0,0 +1,148 @@
+import numba
+import numpy as np
+import pandas as pd
+import pytest
+
+from scarf.features.enrichment.aucell import (
+ build_gene_set_index,
+ make_rank_permutation,
+ resolve_n_up,
+ score_aucell_block,
+)
+from scarf.features.enrichment.net import prepare_network
+from scarf.features.enrichment.waggr import build_waggr_model, score_waggr_block
+
+
+def _prepared_network(*, weighted: bool):
+ return prepare_network(
+ pd.DataFrame(
+ {
+ "source": ["Alpha", "Alpha", "Beta", "Beta"],
+ "target": ["g0", "g2", "g1", "g3"],
+ "weight": [-1.0, 2.0, 3.0, -1.0],
+ }
+ ),
+ active_feature_names=np.array(["g0", "g1", "g2", "g3"]),
+ active_feature_index=np.arange(4),
+ tmin=2,
+ weighted=weighted,
+ )
+
+
+def test_waggr_matches_frozen_decoupler_2_2_reference():
+ model = build_waggr_model(_prepared_network(weighted=True))
+ values = np.array([[1.0, 2.0, 3.0, 4.0], [4.0, 3.0, 2.0, 1.0]])
+
+ sums = score_waggr_block(values, model, mode="wsum")
+ means = score_waggr_block(values, model, mode="wmean")
+
+ np.testing.assert_allclose(sums, [[5.0, 2.0], [0.0, 8.0]])
+ np.testing.assert_allclose(means, [[5.0 / 3.0, 0.5], [0.0, 2.0]])
+
+
+def test_waggr_is_invariant_to_row_block_boundaries():
+ model = build_waggr_model(_prepared_network(weighted=True))
+ values = np.arange(28, dtype=np.float64).reshape(7, 4)
+
+ whole = score_waggr_block(values, model, mode="wmean")
+ split = np.vstack(
+ [
+ score_waggr_block(values[:2], model, mode="wmean"),
+ score_waggr_block(values[2:5], model, mode="wmean"),
+ score_waggr_block(values[5:], model, mode="wmean"),
+ ]
+ )
+
+ np.testing.assert_array_equal(whole, split)
+
+
+def test_waggr_rejects_invalid_values_and_modes():
+ model = build_waggr_model(_prepared_network(weighted=True))
+
+ with pytest.raises(ValueError, match="mode"):
+ score_waggr_block(np.ones((1, 4)), model, mode="median") # type: ignore
+ with pytest.raises(ValueError, match="finite"):
+ score_waggr_block(
+ np.array([[1.0, np.nan, 2.0, 3.0]]),
+ model,
+ mode="wmean",
+ )
+
+
+def test_aucell_matches_frozen_decoupler_2_2_reference():
+ network = _prepared_network(weighted=False)
+ permutation = make_rank_permutation(4, 0)
+ rank_feature_index = np.arange(4)[permutation]
+ sets = build_gene_set_index(network, rank_feature_index)
+ values = np.array(
+ [
+ [4, 3, 2, 1],
+ [1, 4, 3, 2],
+ ],
+ dtype=np.uint32,
+ )
+
+ scores = score_aucell_block(values, permutation, sets, n_up=3)
+
+ np.testing.assert_allclose(
+ scores,
+ [
+ [2.0 / 3.0, 1.0 / 3.0],
+ [1.0 / 3.0, 2.0 / 3.0],
+ ],
+ )
+
+
+def test_aucell_uses_one_global_seeded_tie_permutation():
+ network = _prepared_network(weighted=False)
+ permutation = make_rank_permutation(4, 0)
+ sets = build_gene_set_index(network, np.arange(4)[permutation])
+ tied = np.ones((3, 4), dtype=np.float64)
+
+ scores = score_aucell_block(tied, permutation, sets, n_up=2)
+
+ np.testing.assert_array_equal(permutation, [2, 0, 1, 3])
+ np.testing.assert_array_equal(scores, np.array([[1.0, 0.0]] * 3))
+ np.testing.assert_array_equal(
+ score_aucell_block(np.zeros((1, 4)), permutation, sets, n_up=2),
+ [[0.0, 0.0]],
+ )
+
+
+def test_aucell_is_block_and_thread_count_invariant():
+ network = _prepared_network(weighted=False)
+ permutation = make_rank_permutation(4, 11)
+ sets = build_gene_set_index(network, np.arange(4)[permutation])
+ values = np.array(
+ [[9, 3, 1, 0], [2, 4, 8, 1], [0, 1, 0, 2], [7, 7, 7, 7]],
+ dtype=np.float64,
+ )
+ original_threads = numba.get_num_threads()
+ try:
+ numba.set_num_threads(1)
+ single_thread = score_aucell_block(values, permutation, sets, n_up=3)
+ numba.set_num_threads(min(2, numba.config.NUMBA_NUM_THREADS))
+ two_threads = score_aucell_block(values, permutation, sets, n_up=3)
+ finally:
+ numba.set_num_threads(original_threads)
+ blocked = np.vstack(
+ [
+ score_aucell_block(values[:1], permutation, sets, n_up=3),
+ score_aucell_block(values[1:], permutation, sets, n_up=3),
+ ]
+ )
+
+ np.testing.assert_array_equal(single_thread, two_threads)
+ np.testing.assert_array_equal(single_thread, blocked)
+
+
+def test_resolve_n_up_validates_the_ranking_universe():
+ assert resolve_n_up(100, None) == 5
+ assert resolve_n_up(3, None) == 2
+
+ with pytest.raises(ValueError, match="at least two"):
+ resolve_n_up(1, None)
+ with pytest.raises(ValueError, match="greater than 1"):
+ resolve_n_up(4, 1)
+ with pytest.raises(ValueError, match="at most 4"):
+ resolve_n_up(4, 5)
diff --git a/tests/test_enrichment_net.py b/tests/test_enrichment_net.py
new file mode 100644
index 00000000..20b3451e
--- /dev/null
+++ b/tests/test_enrichment_net.py
@@ -0,0 +1,181 @@
+import numpy as np
+import pandas as pd
+import pytest
+
+from scarf.features.enrichment.net import prepare_network, read_gmt
+
+
+def test_read_gmt_parses_sets_and_ignores_descriptions(tmp_path):
+ path = tmp_path / "sets.gmt"
+ path.write_text(
+ "\nSet B\tdescription\tGene3\tGene4\nSet A\tna\tGene1\tGene2\n",
+ encoding="utf-8",
+ )
+
+ result = read_gmt(path)
+
+ pd.testing.assert_frame_equal(
+ result,
+ pd.DataFrame(
+ {
+ "source": ["Set B", "Set B", "Set A", "Set A"],
+ "target": ["Gene3", "Gene4", "Gene1", "Gene2"],
+ }
+ ),
+ )
+
+
+@pytest.mark.parametrize(
+ "contents, match",
+ [
+ ("set\tdescription\n", "at least 3"),
+ ("\tdescription\tgene\n", "source is empty"),
+ ("set\tdescription\t\n", "no targets"),
+ ("\n", "no gene sets"),
+ ],
+)
+def test_read_gmt_rejects_malformed_files(tmp_path, contents, match):
+ path = tmp_path / "bad.gmt"
+ path.write_text(contents, encoding="utf-8")
+
+ with pytest.raises(ValueError, match=match):
+ read_gmt(path)
+
+
+def test_prepare_network_matches_prunes_and_is_canonical():
+ original = pd.DataFrame(
+ {
+ "source": ["Beta", "Alpha", "Alpha", "Beta", "Dropped"],
+ "target": ["geneD", "geneA", "GENEC", "GENEB", "missing"],
+ "weight": [4.0, -1.0, 2.0, 3.0, 8.0],
+ }
+ )
+ original.index = ["edge-a", "edge-b", "edge-c", "edge-d", "edge-e"]
+ unchanged = original.copy(deep=True)
+ kwargs = {
+ "active_feature_names": np.array(["GeneA", "GeneB", "GeneC", "GeneD"]),
+ "active_feature_index": np.array([8, 2, 5, 9]),
+ "tmin": 2,
+ "weighted": True,
+ }
+
+ prepared = prepare_network(original, **kwargs)
+ shuffled = prepare_network(
+ original.sample(frac=1.0, random_state=7).reset_index(drop=True),
+ **kwargs,
+ )
+
+ pd.testing.assert_frame_equal(original, unchanged)
+ np.testing.assert_array_equal(prepared.source_names, ["Alpha", "Beta"])
+ np.testing.assert_array_equal(prepared.source_sizes, [2, 2])
+ np.testing.assert_array_equal(prepared.matched_feature_index, [2, 5, 8, 9])
+ for field in (
+ "source_names",
+ "source_sizes",
+ "matched_feature_index",
+ "edge_source_index",
+ "edge_feature_index",
+ "edge_weight",
+ ):
+ np.testing.assert_array_equal(
+ getattr(prepared, field),
+ getattr(shuffled, field),
+ )
+ assert prepared.network_digest == shuffled.network_digest
+
+
+def test_prepare_network_detects_duplicate_and_ambiguous_matches():
+ features = np.array(["GeneA", "GeneB"])
+ indices = np.array([0, 1])
+ duplicate = pd.DataFrame({"source": ["Set", "Set"], "target": ["GeneA", "GeneA"]})
+ collapsed_duplicate = pd.DataFrame(
+ {"source": ["Set", "Set"], "target": ["GeneA", "GENEA"]}
+ )
+
+ with pytest.raises(ValueError, match="duplicate source-target"):
+ prepare_network(
+ duplicate,
+ active_feature_names=features,
+ active_feature_index=indices,
+ tmin=1,
+ weighted=False,
+ )
+ with pytest.raises(ValueError, match="after case-insensitive"):
+ prepare_network(
+ collapsed_duplicate,
+ active_feature_names=features,
+ active_feature_index=indices,
+ tmin=1,
+ weighted=False,
+ )
+ with pytest.raises(ValueError, match="multiple active assay features"):
+ prepare_network(
+ pd.DataFrame({"source": ["Set"], "target": ["genea"]}),
+ active_feature_names=np.array(["GeneA", "GENEA"]),
+ active_feature_index=indices,
+ tmin=1,
+ weighted=False,
+ )
+
+
+def test_network_digest_only_uses_score_affecting_weights():
+ first = pd.DataFrame(
+ {
+ "source": ["Set", "Set"],
+ "target": ["GeneA", "GeneB"],
+ "weight": [1.0, 2.0],
+ }
+ )
+ second = first.assign(weight=[3.0, 4.0])
+ kwargs = {
+ "active_feature_names": np.array(["GeneA", "GeneB"]),
+ "active_feature_index": np.array([0, 1]),
+ "tmin": 2,
+ }
+
+ weighted_first = prepare_network(first, weighted=True, **kwargs)
+ weighted_second = prepare_network(second, weighted=True, **kwargs)
+ unweighted_first = prepare_network(first, weighted=False, **kwargs)
+ unweighted_second = prepare_network(second, weighted=False, **kwargs)
+
+ assert weighted_first.network_digest != weighted_second.network_digest
+ assert unweighted_first.network_digest == unweighted_second.network_digest
+ np.testing.assert_array_equal(unweighted_first.edge_weight, [1.0, 1.0])
+
+
+def test_weighted_network_drops_zero_edges_before_tmin():
+ net = pd.DataFrame(
+ {
+ "source": ["Set", "Set"],
+ "target": ["GeneA", "GeneB"],
+ "weight": [1.0, 0.0],
+ }
+ )
+
+ with pytest.raises(ValueError, match="tmin=2"):
+ prepare_network(
+ net,
+ active_feature_names=np.array(["GeneA", "GeneB"]),
+ active_feature_index=np.array([0, 1]),
+ tmin=2,
+ weighted=True,
+ )
+
+
+def test_network_rejects_boolean_weights():
+ net = pd.DataFrame(
+ {
+ "source": ["Set", "Set"],
+ "target": ["GeneA", "GeneB"],
+ "weight": [True, False],
+ }
+ )
+
+ with pytest.raises(ValueError, match="not boolean"):
+ prepare_network(
+ net,
+ active_feature_names=np.array(["GeneA", "GeneB"]),
+ active_feature_index=np.array([0, 1]),
+ tmin=1,
+ weighted=True,
+ )
diff --git a/tests/test_fate_mapping.py b/tests/test_fate_mapping.py
new file mode 100644
index 00000000..41b72bbc
--- /dev/null
+++ b/tests/test_fate_mapping.py
@@ -0,0 +1,644 @@
+from types import SimpleNamespace
+from typing import Any
+
+import numpy as np
+import pytest
+from scipy.sparse import csr_matrix, diags
+
+from scarf.datastore.graph_datastore import GraphDataStore
+from scarf.trajectory.fate import (
+ _make_transition,
+ _normalize_pseudotime,
+ compute_fate_probabilities,
+ make_sink_tokens,
+)
+from scarf.trajectory.results import FateMappingResult
+
+
+def _y_graph() -> tuple[csr_matrix, np.ndarray, np.ndarray]:
+ adjacency = np.zeros((5, 5), dtype=np.float64)
+ for first, second in ((0, 1), (1, 2), (0, 3), (3, 4)):
+ adjacency[first, second] = adjacency[second, first] = 1.0
+ pseudotime = np.array([0.0, 0.5, 1.0, 0.5, 1.0])
+ labels = np.array(["root", "a-mid", "A", "b-mid", "B"])
+ return csr_matrix(adjacency), pseudotime, labels
+
+
+def _dense_dirichlet_reference(
+ transition: csr_matrix,
+ labels: np.ndarray,
+ sinks: list[str],
+) -> np.ndarray:
+ matrix = np.eye(transition.shape[0]) - transition.toarray()
+ absorbing = np.isin(labels, sinks)
+ matrix[absorbing] = 0.0
+ matrix[np.flatnonzero(absorbing), np.flatnonzero(absorbing)] = 1.0
+ probabilities = np.empty((transition.shape[0], len(sinks)), dtype=np.float64)
+ for group, sink in enumerate(sinks):
+ boundary = np.asarray(labels == sink, dtype=np.float64)
+ probabilities[:, group] = np.linalg.solve(matrix, boundary)
+ return probabilities
+
+
+def test_soft_transition_preserves_support_and_normalizes_rows():
+ graph, pseudotime, _ = _y_graph()
+ graph.setdiag(2.0)
+ expected_support = graph.toarray() > 0
+ np.fill_diagonal(expected_support, False)
+
+ transition = _make_transition(
+ graph,
+ _normalize_pseudotime(pseudotime),
+ np.zeros(graph.shape[0], dtype=bool),
+ beta=10.0,
+ )
+
+ np.testing.assert_array_equal(transition.toarray() > 0, expected_support)
+ np.testing.assert_allclose(np.asarray(transition.sum(axis=1)).ravel(), 1.0)
+ assert transition.dtype == np.float64
+ assert transition[1, 0] < transition[1, 2]
+ assert transition[0, 1] == pytest.approx(transition[0, 3])
+
+
+def test_soft_transition_is_affine_invariant_and_beta_zero_is_unbiased():
+ graph, pseudotime, _ = _y_graph()
+ transformed = (pseudotime * 23.0) - 7.0
+
+ first = _make_transition(
+ graph.copy(),
+ _normalize_pseudotime(pseudotime),
+ np.zeros(graph.shape[0], dtype=bool),
+ beta=10.0,
+ )
+ second = _make_transition(
+ graph.copy(),
+ _normalize_pseudotime(transformed),
+ np.zeros(graph.shape[0], dtype=bool),
+ beta=10.0,
+ )
+ unbiased = _make_transition(
+ graph.copy(),
+ _normalize_pseudotime(pseudotime),
+ np.zeros(graph.shape[0], dtype=bool),
+ beta=0.0,
+ )
+ expected_unbiased = graph.toarray()
+ expected_unbiased /= expected_unbiased.sum(axis=1, keepdims=True)
+
+ np.testing.assert_allclose(first.toarray(), second.toarray())
+ np.testing.assert_allclose(unbiased.toarray(), expected_unbiased)
+
+
+def test_soft_transition_preserves_extreme_penalty_ratios():
+ adjacency = np.zeros((3, 3), dtype=np.float64)
+ adjacency[0, 1] = adjacency[1, 0] = 1.0
+ adjacency[0, 2] = adjacency[2, 0] = 1.0
+
+ transition = _make_transition(
+ csr_matrix(adjacency),
+ np.array([1.0, 0.2, 0.0]),
+ np.zeros(3, dtype=bool),
+ beta=1000.0,
+ )
+
+ assert transition[0, 1] > 1.0 - 1e-12
+ assert 0.0 < transition[0, 2] < 1e-80
+ np.testing.assert_allclose(np.asarray(transition.sum(axis=1)).ravel(), 1.0)
+
+
+def test_soft_transition_normalizes_maximum_finite_weights():
+ adjacency = np.full((3, 3), np.finfo(np.float64).max)
+ np.fill_diagonal(adjacency, 0.0)
+
+ transition = _make_transition(
+ csr_matrix(adjacency),
+ np.array([0.0, 0.5, 1.0]),
+ np.zeros(3, dtype=bool),
+ beta=0.0,
+ )
+
+ assert np.isfinite(transition.data).all()
+ np.testing.assert_allclose(transition.toarray().sum(axis=1), 1.0)
+ np.testing.assert_allclose(transition.data, 0.5)
+
+
+def test_pseudotime_normalization_handles_extreme_finite_values():
+ normalized = _normalize_pseudotime(np.array([-1e308, 0.0, 1e308], dtype=np.float64))
+
+ np.testing.assert_allclose(normalized, [0.0, 0.5, 1.0])
+
+
+def test_single_sink_assigns_probability_one_without_solver(
+ monkeypatch: pytest.MonkeyPatch,
+):
+ from scarf.trajectory import fate as fate_module
+
+ graph, pseudotime, labels = _y_graph()
+ monkeypatch.setattr(
+ fate_module,
+ "gmres",
+ lambda *_args, **_kwargs: pytest.fail("GMRES must not run for one sink"),
+ )
+
+ probabilities, valid, sink_labels = compute_fate_probabilities(
+ graph,
+ pseudotime,
+ labels,
+ ["A"],
+ )
+
+ assert sink_labels == ("A",)
+ np.testing.assert_array_equal(valid, np.ones(graph.shape[0], dtype=bool))
+ np.testing.assert_array_equal(probabilities, np.ones((graph.shape[0], 1)))
+
+
+def test_two_sink_branch_matches_dense_dirichlet_reference():
+ graph, pseudotime, labels = _y_graph()
+ reference_transition = _make_transition(
+ graph.copy(),
+ _normalize_pseudotime(pseudotime),
+ np.isin(labels, ["A", "B"]),
+ beta=10.0,
+ )
+ expected = _dense_dirichlet_reference(
+ reference_transition,
+ labels,
+ ["A", "B"],
+ )
+
+ probabilities, valid, _ = compute_fate_probabilities(
+ graph,
+ pseudotime,
+ labels,
+ ["A", "B"],
+ )
+
+ np.testing.assert_array_equal(valid, np.ones(graph.shape[0], dtype=bool))
+ assert probabilities.dtype == np.float32
+ np.testing.assert_allclose(probabilities, expected, rtol=1e-5, atol=1e-7)
+ np.testing.assert_array_equal(probabilities[2], [1.0, 0.0])
+ np.testing.assert_array_equal(probabilities[4], [0.0, 1.0])
+ np.testing.assert_allclose(probabilities.sum(axis=1), 1.0)
+
+
+def test_three_sink_branch_matches_dense_dirichlet_reference():
+ adjacency = np.zeros((7, 7), dtype=np.float64)
+ for first, second in ((0, 1), (1, 2), (0, 3), (3, 4), (0, 5), (5, 6)):
+ adjacency[first, second] = adjacency[second, first] = 1.0
+ graph = csr_matrix(adjacency)
+ pseudotime = np.array([0.0, 0.5, 1.0, 0.5, 1.0, 0.5, 1.0])
+ labels = np.array(["root", "a-mid", "A", "b-mid", "B", "c-mid", "C"])
+ sinks = ["A", "B", "C"]
+ reference_transition = _make_transition(
+ graph.copy(),
+ _normalize_pseudotime(pseudotime),
+ np.isin(labels, sinks),
+ beta=10.0,
+ )
+ expected = _dense_dirichlet_reference(reference_transition, labels, sinks)
+
+ probabilities, valid, _ = compute_fate_probabilities(
+ graph,
+ pseudotime,
+ labels,
+ sinks,
+ )
+
+ np.testing.assert_array_equal(valid, np.ones(graph.shape[0], dtype=bool))
+ np.testing.assert_allclose(probabilities, expected, rtol=1e-5, atol=1e-7)
+ np.testing.assert_allclose(probabilities.sum(axis=1), 1.0)
+
+
+def test_computation_does_not_mutate_input_graph():
+ graph, pseudotime, labels = _y_graph()
+ graph = graph.astype(np.float32)
+ original = graph.copy()
+
+ first, _, _ = compute_fate_probabilities(
+ graph,
+ pseudotime,
+ labels,
+ ["A", "B"],
+ )
+ second, _, _ = compute_fate_probabilities(
+ graph,
+ pseudotime,
+ labels,
+ ["A", "B"],
+ )
+
+ np.testing.assert_array_equal(graph.indptr, original.indptr)
+ np.testing.assert_array_equal(graph.indices, original.indices)
+ np.testing.assert_array_equal(graph.data, original.data)
+ np.testing.assert_array_equal(first, second)
+
+
+def test_sinkless_components_are_invalid_and_other_components_remain_valid():
+ adjacency = np.zeros((8, 8), dtype=np.float64)
+ for first, second in ((0, 1), (1, 2), (3, 4), (4, 5), (6, 7)):
+ adjacency[first, second] = adjacency[second, first] = 1.0
+ pseudotime = np.array([0.0, 0.5, 1.0, 0.0, 0.5, 1.0, 0.2, 0.8])
+ labels = np.array(["root", "mid", "A", "root", "mid", "B", "other", "other"])
+
+ probabilities, valid, _ = compute_fate_probabilities(
+ csr_matrix(adjacency),
+ pseudotime,
+ labels,
+ ["A", "B"],
+ )
+
+ np.testing.assert_array_equal(
+ valid,
+ np.array([True, True, True, True, True, True, False, False]),
+ )
+ np.testing.assert_allclose(probabilities[:3], [[1.0, 0.0]] * 3, atol=1e-7)
+ np.testing.assert_allclose(probabilities[3:6], [[0.0, 1.0]] * 3, atol=1e-7)
+ assert np.isnan(probabilities[6:]).all()
+
+
+def test_explicit_zero_weight_does_not_connect_components():
+ graph = csr_matrix(
+ (
+ np.array([1.0, 1.0, 0.0]),
+ np.array([1, 0, 2]),
+ np.array([0, 1, 3, 3]),
+ ),
+ shape=(3, 3),
+ )
+
+ probabilities, valid, _ = compute_fate_probabilities(
+ graph,
+ np.array([0.0, 1.0, 2.0]),
+ np.array(["root", "A", "other"]),
+ ["A"],
+ )
+
+ np.testing.assert_array_equal(valid, [True, True, False])
+ np.testing.assert_array_equal(probabilities[:2], [[1.0], [1.0]])
+ assert np.isnan(probabilities[2]).all()
+ assert graph.nnz == 3
+
+
+def test_nonfinite_pseudotime_in_sinkless_component_is_rejected():
+ adjacency = np.zeros((4, 4), dtype=np.float64)
+ adjacency[0, 1] = adjacency[1, 0] = 1.0
+ adjacency[2, 3] = adjacency[3, 2] = 1.0
+
+ with pytest.raises(ValueError, match="finite"):
+ compute_fate_probabilities(
+ csr_matrix(adjacency),
+ np.array([0.0, 1.0, np.nan, 1.0]),
+ np.array(["root", "A", "other", "other"]),
+ ["A"],
+ )
+
+
+def test_complex_graph_weights_are_rejected_clearly():
+ graph, pseudotime, labels = _y_graph()
+
+ with pytest.raises(TypeError, match="real numeric"):
+ compute_fate_probabilities(
+ graph.astype(np.complex128),
+ pseudotime,
+ labels,
+ ["A"],
+ )
+
+
+def test_asymmetric_graph_support_is_rejected():
+ graph = csr_matrix(
+ (
+ np.array([1.0]),
+ np.array([1]),
+ np.array([0, 1, 1]),
+ ),
+ shape=(2, 2),
+ )
+
+ with pytest.raises(ValueError, match="symmetric"):
+ compute_fate_probabilities(
+ graph,
+ np.array([0.0, 1.0]),
+ np.array(["root", "A"]),
+ ["A"],
+ )
+
+
+def test_malformed_csr_structure_is_rejected():
+ graph, pseudotime, labels = _y_graph()
+ graph.indices[0] = graph.shape[0]
+
+ with pytest.raises(ValueError, match="invalid CSR"):
+ compute_fate_probabilities(
+ graph,
+ pseudotime,
+ labels,
+ ["A", "B"],
+ )
+
+
+def test_weights_that_overflow_float64_are_rejected():
+ if np.finfo(np.longdouble).max <= np.finfo(np.float64).max:
+ pytest.skip("long double does not exceed float64 on this platform")
+ graph, pseudotime, labels = _y_graph()
+ graph = graph.astype(np.longdouble)
+ graph.data[0] = np.finfo(np.longdouble).max
+
+ with pytest.raises(ValueError, match="converted to float64"):
+ compute_fate_probabilities(
+ graph,
+ pseudotime,
+ labels,
+ ["A", "B"],
+ )
+
+
+def test_reordering_sinks_only_reorders_probability_columns():
+ graph, pseudotime, labels = _y_graph()
+ forward, _, _ = compute_fate_probabilities(
+ graph.copy(),
+ pseudotime,
+ labels,
+ ["A", "B"],
+ )
+ reverse, _, _ = compute_fate_probabilities(
+ graph.copy(),
+ pseudotime,
+ labels,
+ ["B", "A"],
+ )
+
+ np.testing.assert_allclose(forward, reverse[:, ::-1], rtol=1e-5, atol=1e-7)
+
+
+def test_sink_tokens_remain_unique_after_sanitization():
+ assert make_sink_tokens(("A", "A_2", "A!")) == ("A", "A_2", "A_3")
+
+
+@pytest.mark.parametrize(
+ ("sinks", "error", "message"),
+ [
+ ([], ValueError, "At least one"),
+ (["A", "A"], ValueError, "unique"),
+ (["missing"], ValueError, "not found"),
+ ([("A", "B")], TypeError, "scalar"),
+ (("A",), TypeError, "list"),
+ ],
+)
+def test_invalid_sink_definitions_fail_clearly(
+ sinks: Any,
+ error: type[Exception],
+ message: str,
+):
+ graph, pseudotime, labels = _y_graph()
+
+ with pytest.raises(error, match=message):
+ compute_fate_probabilities(
+ graph,
+ pseudotime,
+ labels,
+ sinks,
+ )
+
+
+def test_max_iterations_limits_gmres_inner_iterations():
+ n_cells = 80
+ graph = diags(
+ [np.ones(n_cells - 1), np.ones(n_cells - 1)],
+ [-1, 1],
+ shape=(n_cells, n_cells),
+ format="csr",
+ )
+ labels = np.full(n_cells, "other", dtype=object)
+ labels[0] = "A"
+ labels[-1] = "B"
+
+ with pytest.raises(RuntimeError, match="after 1 iteration"):
+ compute_fate_probabilities(
+ graph,
+ np.linspace(0.0, 1.0, n_cells),
+ labels,
+ ["A", "B"],
+ beta=0.0,
+ solver_tol=1e-12,
+ max_iterations=1,
+ )
+
+
+def test_loose_solver_tolerance_does_not_bypass_output_validation(
+ monkeypatch: pytest.MonkeyPatch,
+):
+ from scarf.trajectory import fate as fate_module
+
+ graph, pseudotime, labels = _y_graph()
+
+ def invalid_solution(operator: Any, *_args: Any, **_kwargs: Any):
+ return np.full(operator.shape[0], 2.0), 0
+
+ monkeypatch.setattr(fate_module, "gmres", invalid_solution)
+ with pytest.raises(RuntimeError, match="numerical bounds"):
+ compute_fate_probabilities(
+ graph,
+ pseudotime,
+ labels,
+ ["A", "B"],
+ solver_tol=0.9,
+ )
+
+
+def test_localized_solver_error_fails_residual_validation(
+ monkeypatch: pytest.MonkeyPatch,
+):
+ from scarf.trajectory import fate as fate_module
+
+ n_cells = 203
+ adjacency = np.zeros((n_cells, n_cells), dtype=np.float64)
+ adjacency[0, 1] = adjacency[1, 0] = 1.0
+ adjacency[1, 2] = adjacency[2, 1] = 1.0
+ labels = np.empty(n_cells, dtype=object)
+ labels[:3] = ["A", "middle", "B"]
+ labels[3:103] = "A"
+ labels[103:] = "B"
+ pseudotime = np.ones(n_cells, dtype=np.float64)
+ pseudotime[1] = 0.0
+ original_gmres = fate_module.gmres
+
+ def localized_error(*args: Any, **kwargs: Any):
+ solution, info = original_gmres(*args, **kwargs)
+ solution[1] += 0.01
+ return solution, info
+
+ monkeypatch.setattr(fate_module, "gmres", localized_error)
+ with pytest.raises(RuntimeError, match="residual"):
+ compute_fate_probabilities(
+ csr_matrix(adjacency),
+ pseudotime,
+ labels,
+ ["A", "B"],
+ beta=0.0,
+ solver_tol=1e-3,
+ )
+
+
+class _FateCells:
+ def __init__(self, pseudotime: np.ndarray, labels: np.ndarray):
+ self.values: dict[str, np.ndarray] = {
+ "I": np.ones(pseudotime.shape[0], dtype=bool),
+ "ptime": pseudotime,
+ "state": labels,
+ }
+ self.insertions: list[str] = []
+
+ @property
+ def columns(self) -> list[str]:
+ return list(self.values)
+
+ def fetch(self, column: str, key: str = "I") -> np.ndarray:
+ return np.asarray(self.values[column][self.values[key].astype(bool)])
+
+ def fetch_all(self, column: str) -> np.ndarray:
+ return np.asarray(self.values[column])
+
+ def active_index(self, key: str) -> np.ndarray:
+ return np.flatnonzero(self.values[key])
+
+ def insert(
+ self,
+ column_name: str,
+ values: np.ndarray,
+ fill_value: Any = np.nan,
+ key: str = "I",
+ overwrite: bool = False,
+ ) -> None:
+ assert overwrite
+ selected = self.values[key].astype(bool)
+ if np.asarray(values).dtype.kind == "b":
+ filled = np.full(selected.shape[0], bool(fill_value), dtype=bool)
+ else:
+ filled = np.full(selected.shape[0], fill_value, dtype=np.float32)
+ filled[selected] = values
+ self.values[column_name] = filled
+ self.insertions.append(column_name)
+
+
+class _FateStore:
+ def __init__(self):
+ self.graph, pseudotime, labels = _y_graph()
+ self.cells = _FateCells(pseudotime, labels)
+ self.assay = SimpleNamespace(name="RNA", cells=self.cells)
+ self.graph_loads = 0
+
+ @staticmethod
+ def _get_latest_keys(
+ _from_assay: str | None,
+ _cell_key: str | None,
+ _feat_key: str | None,
+ ) -> tuple[str, str, str]:
+ return "RNA", "I", "I"
+
+ def _get_assay(self, _from_assay: str) -> SimpleNamespace:
+ return self.assay
+
+ def load_graph(self, **_kwargs: Any) -> csr_matrix:
+ self.graph_loads += 1
+ return self.graph.copy()
+
+ @staticmethod
+ def _col_renamer(from_assay: str, cell_key: str, suffix: str) -> str:
+ if cell_key == "I":
+ return f"{from_assay}_{suffix}"
+ return f"{from_assay}_{cell_key}_{suffix}"
+
+
+def test_datastore_method_writes_aligned_fate_and_validity_columns():
+ store = _FateStore()
+
+ result = GraphDataStore.run_fate_mapping(
+ store,
+ pseudotime_key="ptime",
+ sink_key="state",
+ sinks=["A", "B"],
+ )
+
+ assert isinstance(result, FateMappingResult)
+ assert result.fate_keys == ("RNA_fate_A", "RNA_fate_B")
+ assert result.validity_key == "RNA_fate__valid"
+ assert result.sink_labels == ("A", "B")
+ assert store.cells.insertions == [
+ "RNA_fate_A",
+ "RNA_fate_B",
+ "RNA_fate__valid",
+ ]
+ np.testing.assert_allclose(
+ np.column_stack([store.cells.fetch_all(key) for key in result.fate_keys]),
+ result.values,
+ )
+ np.testing.assert_array_equal(
+ store.cells.fetch_all(result.validity_key),
+ result.valid,
+ )
+
+
+def test_datastore_method_aligns_subset_outputs_to_all_cells():
+ store = _FateStore()
+ store.cells.values["subset"] = np.array([True, True, True, False, False])
+ store.cells.values["ptime"][3:] = np.nan
+
+ result = GraphDataStore.run_fate_mapping(
+ store,
+ subset_cell_key="subset",
+ pseudotime_key="ptime",
+ sink_key="state",
+ sinks=["A"],
+ )
+
+ assert result.result_cell_key == "subset"
+ assert result.values.shape == (3, 1)
+ stored = store.cells.fetch_all(result.fate_keys[0])
+ np.testing.assert_array_equal(stored[:3], np.ones(3))
+ assert np.isnan(stored[3:]).all()
+ np.testing.assert_array_equal(
+ store.cells.fetch_all(result.validity_key),
+ [True, True, True, False, False],
+ )
+
+
+@pytest.mark.parametrize("sinks", [[], ("A",)])
+def test_datastore_rejects_invalid_sink_container_before_loading_graph(
+ sinks: Any,
+):
+ store = _FateStore()
+
+ with pytest.raises((TypeError, ValueError)):
+ GraphDataStore.run_fate_mapping(
+ store,
+ pseudotime_key="ptime",
+ sink_key="state",
+ sinks=sinks,
+ )
+
+ assert store.graph_loads == 0
+
+
+def test_failed_solver_writes_no_metadata(monkeypatch: pytest.MonkeyPatch):
+ from scarf.datastore._operations import trajectory as trajectory_operations
+
+ store = _FateStore()
+
+ def fail(*_args: Any, **_kwargs: Any):
+ raise RuntimeError("forced non-convergence")
+
+ monkeypatch.setattr(
+ trajectory_operations,
+ "_compute_fate_probabilities_impl",
+ fail,
+ )
+ with pytest.raises(RuntimeError, match="forced non-convergence"):
+ GraphDataStore.run_fate_mapping(
+ store,
+ pseudotime_key="ptime",
+ sink_key="state",
+ sinks=["A", "B"],
+ )
+
+ assert store.cells.insertions == []
diff --git a/tests/test_feat_utils.py b/tests/test_feat_utils.py
new file mode 100644
index 00000000..26537034
--- /dev/null
+++ b/tests/test_feat_utils.py
@@ -0,0 +1,645 @@
+import numpy as np
+import pandas as pd
+import pytest
+
+from scarf.features.scoring import binned_sampling
+from scarf.features.variability import fit_lowess, select_highly_variable_features
+from scarf.quality_control.hto import (
+ _background_clusters,
+ _classify_hto_identities,
+ _cluster_labels,
+ _clr_normalize,
+ _fit_negative_binomial_parameters,
+ _negative_binomial_cutoff,
+ _positive_hto_calls,
+ hto_demux,
+)
+
+
+def test_fit_lowess_returns_per_feature_corrections():
+ rng = np.random.default_rng(0)
+ n_genes = 80
+ mean_expr = rng.uniform(0.5, 20.0, n_genes)
+ variance = mean_expr**1.5 + rng.normal(0, 0.05, n_genes)
+ variance = np.clip(variance, 0.1, None)
+
+ corrected = fit_lowess(mean_expr, variance, n_bins=8, lowess_frac=0.6)
+ explicit_adaptive = fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=8,
+ lowess_frac=0.6,
+ bin_strategy="adaptive",
+ )
+ fixed = fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=8,
+ lowess_frac=0.6,
+ bin_strategy="fixed",
+ )
+
+ assert corrected.shape == (n_genes,)
+ assert np.all(np.isfinite(corrected))
+ assert np.all(corrected > 0)
+ np.testing.assert_array_equal(corrected, explicit_adaptive)
+ assert np.all(np.isfinite(fixed))
+ assert np.all(fixed > 0)
+
+
+def test_fit_lowess_fixed_regression():
+ mean_expr = np.array([0.5, 0.8, 1.2, 1.8, 2.7, 4.0, 6.0, 9.0, 13.0, 20.0])
+ variance = np.array([0.4, 0.9, 1.1, 3.2, 2.8, 8.5, 7.0, 25.0, 22.0, 70.0])
+ expected = np.array(
+ [1.0, 2.25, 2.75, 8 / 7, 1.0, 17 / 14, 1.0, 25 / 22, 1.0, 35 / 11]
+ )
+
+ corrected = fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=4,
+ lowess_frac=0.75,
+ bin_strategy="fixed",
+ )
+
+ np.testing.assert_allclose(corrected, expected, rtol=1e-12, atol=1e-12)
+
+
+def test_fit_lowess_adaptive_balances_bins_and_interpolates(monkeypatch):
+ rng = np.random.default_rng(3)
+ mean_expr = np.exp(rng.exponential(1.0, 101))
+ variance = mean_expr**1.2 * np.exp(rng.normal(0, 0.1, len(mean_expr)))
+ captured: dict[str, np.ndarray] = {}
+
+ def fake_lowess(
+ endog,
+ exog,
+ *,
+ return_sorted,
+ frac,
+ it,
+ ):
+ captured["endog"] = np.asarray(endog)
+ captured["exog"] = np.asarray(exog)
+ return np.asarray(exog)
+
+ monkeypatch.setattr(
+ "statsmodels.nonparametric.smoothers_lowess.lowess",
+ fake_lowess,
+ )
+ corrected = fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=4,
+ lowess_frac=0.4,
+ bin_strategy="adaptive",
+ )
+
+ order = np.argsort(np.log(mean_expr), kind="stable")
+ sorted_means = np.log(mean_expr)[order]
+ sorted_variances = np.log(variance)[order]
+ bins = (slice(0, 26), slice(26, 51), slice(51, 76), slice(76, 101))
+ expected_x = np.array([np.median(sorted_means[idx]) for idx in bins])
+ expected_y = np.array([np.quantile(sorted_variances[idx], 0.1) for idx in bins])
+
+ np.testing.assert_allclose(captured["exog"], expected_x)
+ np.testing.assert_allclose(captured["endog"], expected_y)
+ expected_correction = np.interp(np.log(mean_expr), expected_x, expected_x)
+ np.testing.assert_allclose(
+ corrected,
+ np.exp(np.log(variance) - expected_correction),
+ )
+
+
+def test_fit_lowess_adaptive_keeps_equal_means_in_one_bin(monkeypatch):
+ mean_expr = np.repeat([1.0, 2.0, 4.0], [30, 25, 25])
+ variance = np.linspace(1.0, 3.0, len(mean_expr))
+ captured: dict[str, np.ndarray] = {}
+
+ def fake_lowess(
+ endog,
+ exog,
+ *,
+ return_sorted,
+ frac,
+ it,
+ ):
+ captured["exog"] = np.asarray(exog)
+ return np.asarray(endog)
+
+ monkeypatch.setattr(
+ "statsmodels.nonparametric.smoothers_lowess.lowess",
+ fake_lowess,
+ )
+ corrected = fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=3,
+ lowess_frac=0.5,
+ bin_strategy="adaptive",
+ )
+
+ np.testing.assert_allclose(captured["exog"], np.log([1.0, 2.0, 4.0]))
+ assert np.all(np.isfinite(corrected))
+ assert np.all(corrected > 0)
+
+
+def test_fit_lowess_adaptive_reduces_bins_after_large_tie(monkeypatch):
+ mean_expr = np.concatenate([np.ones(30), np.geomspace(2.0, 100.0, 70)])
+ variance = mean_expr**1.2
+ captured: dict[str, np.ndarray] = {}
+
+ def fake_lowess(
+ endog,
+ exog,
+ *,
+ return_sorted,
+ frac,
+ it,
+ ):
+ captured["exog"] = np.asarray(exog)
+ return np.asarray(endog)
+
+ monkeypatch.setattr(
+ "statsmodels.nonparametric.smoothers_lowess.lowess",
+ fake_lowess,
+ )
+ corrected = fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=4,
+ lowess_frac=0.5,
+ bin_strategy="adaptive",
+ )
+
+ assert len(captured["exog"]) == 3
+ assert captured["exog"][0] == 0
+ assert np.all(np.isfinite(corrected))
+
+
+def test_fit_lowess_adaptive_resists_single_low_variance_outlier():
+ rng = np.random.default_rng(4)
+ mean_expr = np.geomspace(0.01, 100.0, 500)
+ variance = mean_expr**1.4 * np.exp(rng.normal(0, 0.08, len(mean_expr)))
+ baseline = fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=20,
+ lowess_frac=0.4,
+ bin_strategy="adaptive",
+ )
+
+ outlier_variance = variance.copy()
+ outlier_variance[12] *= 1e-12
+ with_outlier = fit_lowess(
+ mean_expr,
+ outlier_variance,
+ n_bins=20,
+ lowess_frac=0.4,
+ bin_strategy="adaptive",
+ )
+
+ unaffected = np.arange(len(mean_expr)) != 12
+ log_change = np.abs(np.log(baseline[unaffected] / with_outlier[unaffected]))
+ assert log_change.max() < 0.05
+
+
+def test_fit_lowess_adaptive_handles_small_and_invalid_inputs():
+ mean_expr = np.array([1.0, 2.0, 3.0, 4.0, np.nan, 0.0])
+ variance = np.array([1.0, 0.0, -1.0, 4.0, 2.0, 2.0])
+
+ corrected = fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=200,
+ lowess_frac=0.1,
+ bin_strategy="adaptive",
+ )
+
+ assert np.all(np.isfinite(corrected))
+ assert np.all(corrected[[0, 3]] > 0)
+ np.testing.assert_array_equal(corrected[[1, 2, 4, 5]], np.zeros(4))
+ np.testing.assert_array_equal(
+ fit_lowess(
+ np.array([0.0, np.nan]),
+ np.array([0.0, 1.0]),
+ n_bins=200,
+ lowess_frac=0.1,
+ bin_strategy="adaptive",
+ ),
+ np.zeros(2),
+ )
+
+ with pytest.raises(ValueError, match="n_bins"):
+ fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=0,
+ lowess_frac=0.1,
+ bin_strategy="adaptive",
+ )
+ with pytest.raises(TypeError, match="n_bins"):
+ fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=True,
+ lowess_frac=0.1,
+ bin_strategy="adaptive",
+ )
+ with pytest.raises(TypeError, match="n_bins"):
+ fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=2.5,
+ lowess_frac=0.1,
+ bin_strategy="adaptive",
+ )
+ with pytest.raises(ValueError, match="lowess_frac"):
+ fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=20,
+ lowess_frac=np.nan,
+ bin_strategy="adaptive",
+ )
+ with pytest.raises(ValueError, match="lowess_frac"):
+ fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=20,
+ lowess_frac=2.0,
+ bin_strategy="adaptive",
+ )
+ with pytest.raises(ValueError, match="bin_strategy"):
+ fit_lowess(
+ mean_expr,
+ variance,
+ n_bins=20,
+ lowess_frac=0.1,
+ bin_strategy="unknown",
+ )
+
+
+def test_highly_variable_feature_selection_applies_all_candidate_filters():
+ selected = select_highly_variable_features(
+ corrected_variance=np.array([10.0, 8.0, 6.0, 4.0, 2.0]),
+ normalized_cell_counts=np.full(5, 5),
+ mean_nonzero=np.full(5, 2.0),
+ active_features=np.array([True, True, True, True, False]),
+ feature_names=np.array(["A", "MT-X", "B", "C", "D"]),
+ min_cells=0,
+ max_cells=np.inf,
+ top_n=2,
+ min_var=-np.inf,
+ max_var=np.inf,
+ min_mean=-np.inf,
+ max_mean=np.inf,
+ blacklist="^MT-",
+ keep_bounds=False,
+ )
+
+ np.testing.assert_array_equal(
+ selected,
+ np.array([True, False, True, False, False]),
+ )
+
+
+def test_hvg_cell_count_bounds_include_minimum_and_exclude_maximum():
+ selected = select_highly_variable_features(
+ corrected_variance=np.array([100.0, 10.0, 8.0, 100.0]),
+ normalized_cell_counts=np.array([19, 20, 21, 80]),
+ mean_nonzero=np.ones(4),
+ active_features=np.ones(4, dtype=bool),
+ feature_names=np.array(["below", "minimum", "inside", "maximum"]),
+ min_cells=20,
+ max_cells=80,
+ top_n=1,
+ min_var=-np.inf,
+ max_var=np.inf,
+ min_mean=-np.inf,
+ max_mean=np.inf,
+ blacklist="",
+ keep_bounds=False,
+ )
+
+ np.testing.assert_array_equal(
+ selected,
+ np.array([False, True, False, False]),
+ )
+
+
+def test_binned_sampling_excludes_query_genes():
+ rng = np.random.default_rng(1)
+ gene_names = [f"gene_{i}" for i in range(120)]
+ values = pd.Series(rng.exponential(1.0, len(gene_names)), index=gene_names)
+ query_genes = gene_names[10:25]
+
+ controls = binned_sampling(
+ values,
+ feature_list=query_genes,
+ ctrl_size=8,
+ n_bins=6,
+ rand_seed=42,
+ )
+
+ assert len(controls) > 0
+ assert set(controls).isdisjoint(query_genes)
+ assert all(name in gene_names for name in controls)
+
+
+def test_hto_negative_binomial_cutoff_is_unshifted(monkeypatch):
+ assert _negative_binomial_cutoff(mu=1, alpha=1) == 6
+
+ counts = pd.DataFrame({"HTO_A": [1, 2, 6, 7]})
+ monkeypatch.setattr(
+ "scarf.quality_control.hto._fit_negative_binomial_parameters",
+ lambda values, hto_name: (1, 1),
+ )
+
+ positive = _positive_hto_calls(counts, np.asarray([0, 0, 1, 1]))
+
+ assert positive["HTO_A"].tolist() == [False, False, False, True]
+
+
+def test_hto_background_cluster_uses_raw_means():
+ counts = pd.DataFrame({"HTO_A": [0, 100, 40, 40]})
+ cluster_labels = np.asarray([0, 0, 1, 1])
+
+ background = _background_clusters(counts, cluster_labels)
+ normalized_means = _clr_normalize(counts).groupby(cluster_labels).mean()
+
+ assert background["HTO_A"] == 1
+ assert normalized_means["HTO_A"].idxmin() == 0
+
+
+def test_hto_classification_uses_clr_argmax_for_singlets():
+ index = ["negative", "singlet", "doublet", "tie"]
+ normalized = pd.DataFrame(
+ {
+ "HTO_A": [0.2, 0.2, 1.0, 0.5],
+ "HTO_B": [0.1, 1.5, 0.9, 0.5],
+ },
+ index=index,
+ )
+ positive = pd.DataFrame(
+ {
+ "HTO_A": [False, True, True, True],
+ "HTO_B": [False, False, True, False],
+ },
+ index=index,
+ )
+
+ identities = _classify_hto_identities(normalized, positive)
+
+ assert identities.to_dict() == {
+ "negative": "Negative",
+ "singlet": "HTO_B",
+ "doublet": "Doublet",
+ "tie": "HTO_A",
+ }
+
+
+def test_hto_demux_assigns_singlet_and_negative_labels():
+ rng = np.random.default_rng(2)
+ n_cells = 60
+ hto_names = ["cluster", "HTO_B", "HTO_C"]
+
+ background = rng.poisson(2, size=(n_cells, len(hto_names)))
+ counts = background.astype(float)
+ for i in range(n_cells):
+ dominant = i % len(hto_names)
+ counts[i, dominant] += rng.integers(30, 80)
+
+ hto_counts = pd.DataFrame(
+ counts,
+ columns=hto_names,
+ index=[f"cell_{index}" for index in range(n_cells)],
+ )
+ original = hto_counts.copy(deep=True)
+ assignments = hto_demux(hto_counts)
+ repeated = hto_demux(hto_counts)
+
+ assert len(assignments) == n_cells
+ pd.testing.assert_series_equal(assignments, repeated)
+ assert assignments.index.equals(hto_counts.index)
+ pd.testing.assert_frame_equal(hto_counts, original)
+ allowed = {"Negative", "Singlet", "Doublet", *hto_names}
+ assert set(assignments.unique()).issubset(allowed)
+ assert set(assignments.unique()) & set(hto_names)
+
+
+@pytest.mark.parametrize(
+ ("hto_counts", "error", "message"),
+ [
+ (
+ np.asarray([[1], [2]]),
+ TypeError,
+ "must be a pandas DataFrame",
+ ),
+ (
+ pd.DataFrame(index=range(2)),
+ ValueError,
+ "at least one HTO",
+ ),
+ (
+ pd.DataFrame(np.ones((3, 2)), columns=["HTO_A", "HTO_A"]),
+ ValueError,
+ "HTO IDs must be unique",
+ ),
+ (
+ pd.DataFrame({" ": [1, 2]}),
+ ValueError,
+ "non-empty strings",
+ ),
+ (
+ pd.DataFrame({"Negative": [1, 2]}),
+ ValueError,
+ "reserved identity labels",
+ ),
+ (
+ pd.DataFrame({"HTO_A": [1, 2]}, index=["cell", "cell"]),
+ ValueError,
+ "cell index must be unique",
+ ),
+ (
+ pd.DataFrame({"HTO_A": ["1", "2"]}),
+ TypeError,
+ "only numeric raw counts",
+ ),
+ (
+ pd.DataFrame({"HTO_A": [1, np.nan]}),
+ ValueError,
+ "only finite raw counts",
+ ),
+ (
+ pd.DataFrame({"HTO_A": [1, -1]}),
+ ValueError,
+ "only nonnegative raw counts",
+ ),
+ (
+ pd.DataFrame({"HTO_A": [1, 1.5]}),
+ ValueError,
+ "integer-valued raw counts",
+ ),
+ (
+ pd.DataFrame({"HTO_A": [0, 0]}),
+ ValueError,
+ "no positive counts",
+ ),
+ ],
+ ids=[
+ "not-dataframe",
+ "no-htos",
+ "duplicate-htos",
+ "empty-hto",
+ "reserved-hto",
+ "duplicate-cells",
+ "nonnumeric",
+ "nonfinite",
+ "negative",
+ "fractional",
+ "all-zero-hto",
+ ],
+)
+def test_hto_demux_rejects_invalid_input(hto_counts, error, message):
+ with pytest.raises(error, match=message):
+ hto_demux(hto_counts)
+
+
+def test_hto_demux_rejects_insufficiently_distinct_profiles():
+ hto_counts = pd.DataFrame(
+ {
+ "HTO_A": [1, 1, 1],
+ "HTO_B": [2, 2, 2],
+ }
+ )
+
+ with pytest.raises(ValueError, match="3 distinct normalized cell profiles"):
+ hto_demux(hto_counts)
+
+
+def test_hto_cluster_labels_rejects_collapsed_kmeans(monkeypatch):
+ class CollapsedKMeans:
+ def __init__(self, **kwargs):
+ pass
+
+ def fit_predict(self, values):
+ return np.asarray([0, 0, 1])
+
+ monkeypatch.setattr("sklearn.cluster.KMeans", CollapsedKMeans)
+ normalized = pd.DataFrame(
+ {
+ "HTO_A": [0.0, 1.0, 2.0],
+ "HTO_B": [2.0, 1.0, 0.0],
+ }
+ )
+
+ with pytest.raises(ValueError, match="2 occupied clusters; expected 3"):
+ _cluster_labels(normalized, random_seed=0)
+
+
+@pytest.mark.parametrize(
+ ("counts", "labels", "message"),
+ [
+ (
+ pd.DataFrame({"HTO_A": [1, 10, 20]}),
+ np.asarray([0, 1, 2]),
+ "at least two cells",
+ ),
+ (
+ pd.DataFrame({"HTO_A": [0, 0, 10, 20]}),
+ np.asarray([0, 0, 1, 1]),
+ "contains only zero counts",
+ ),
+ ],
+ ids=["single-cell", "all-zero"],
+)
+def test_hto_positive_calls_rejects_invalid_backgrounds(counts, labels, message):
+ with pytest.raises(ValueError, match=message):
+ _positive_hto_calls(counts, labels)
+
+
+def test_hto_negative_binomial_fit_rejects_nonconvergence(monkeypatch):
+ class FitResult:
+ mle_retvals = {"converged": False}
+ params = np.asarray([0.0, 1.0])
+
+ class NonconvergedModel:
+ def __init__(self, *args, **kwargs):
+ pass
+
+ def fit(self, **kwargs):
+ return FitResult()
+
+ monkeypatch.setattr(
+ "statsmodels.discrete.discrete_model.NegativeBinomial",
+ NonconvergedModel,
+ )
+
+ with pytest.raises(ValueError, match="did not converge for HTO 'HTO_A'"):
+ _fit_negative_binomial_parameters(np.asarray([1, 2]), "HTO_A")
+
+
+def test_hto_negative_binomial_fit_wraps_optimizer_errors(monkeypatch):
+ class FailingModel:
+ def __init__(self, *args, **kwargs):
+ raise RuntimeError("optimizer failed")
+
+ monkeypatch.setattr(
+ "statsmodels.discrete.discrete_model.NegativeBinomial",
+ FailingModel,
+ )
+
+ with pytest.raises(ValueError, match="fit failed for HTO 'HTO_A'"):
+ _fit_negative_binomial_parameters(np.asarray([1, 2]), "HTO_A")
+
+
+@pytest.mark.parametrize(
+ ("parameters", "message"),
+ [
+ (np.asarray([np.inf, 1.0]), "invalid mean"),
+ (np.asarray([0.0, 0.0]), "invalid dispersion"),
+ ],
+ ids=["mean", "dispersion"],
+)
+def test_hto_negative_binomial_fit_rejects_invalid_parameters(
+ monkeypatch,
+ parameters,
+ message,
+):
+ class FitResult:
+ mle_retvals = {"converged": True}
+ params = parameters
+
+ class InvalidModel:
+ def __init__(self, *args, **kwargs):
+ pass
+
+ def fit(self, **kwargs):
+ return FitResult()
+
+ monkeypatch.setattr(
+ "statsmodels.discrete.discrete_model.NegativeBinomial",
+ InvalidModel,
+ )
+
+ with pytest.raises(ValueError, match=message):
+ _fit_negative_binomial_parameters(np.asarray([1, 2]), "HTO_A")
+
+
+def test_hto_negative_binomial_cutoff_rejects_nonfinite_ppf(monkeypatch):
+ monkeypatch.setattr("scipy.stats.nbinom.ppf", lambda *args, **kwargs: np.inf)
+
+ with pytest.raises(ValueError, match="cutoff must be a finite integer"):
+ _negative_binomial_cutoff(mu=1, alpha=1)
+
+
+def test_hto_demux_rejects_too_few_cells():
+ hto_counts = pd.DataFrame(
+ {
+ "HTO_A": [1, 2],
+ "HTO_B": [2, 1],
+ }
+ )
+ with pytest.raises(ValueError, match="at least 3 selected cells"):
+ hto_demux(hto_counts)
diff --git a/tests/test_feature_mapping_metric_contracts.py b/tests/test_feature_mapping_metric_contracts.py
new file mode 100644
index 00000000..7de8bb40
--- /dev/null
+++ b/tests/test_feature_mapping_metric_contracts.py
@@ -0,0 +1,24 @@
+from scarf.features.genomic.gff import GffReader
+from scarf.features.genomic.melding import coordinate_melding
+from scarf.features.markers.search import (
+ find_markers_by_rank,
+ find_markers_by_regression,
+)
+from scarf.metrics.lisi import compute_lisi
+from scarf.metrics.silhouette import silhouette_scoring
+from tests.signature_contracts import signature_digest
+
+
+def test_feature_mapping_and_metric_entry_point_signatures_are_stable():
+ methods = {
+ "GffReader.__init__": GffReader.__init__,
+ "compute_lisi": compute_lisi,
+ "coordinate_melding": coordinate_melding,
+ "find_markers_by_rank": find_markers_by_rank,
+ "find_markers_by_regression": find_markers_by_regression,
+ "silhouette_scoring": silhouette_scoring,
+ }
+
+ assert signature_digest(methods) == (
+ "aeffb7fcecc0c76c9c83f5eb33b4cafbebd9af1963262c9a4b6c408d2bfa59e5"
+ )
diff --git a/tests/test_feature_stream.py b/tests/test_feature_stream.py
new file mode 100644
index 00000000..960c9cd0
--- /dev/null
+++ b/tests/test_feature_stream.py
@@ -0,0 +1,193 @@
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.storage.budget import ResourceBudget
+from scarf.storage.feature_stream import plan_feature_stream
+
+
+def _array(
+ *,
+ shape: tuple[int, int] = (12, 15),
+ chunks: tuple[int, int] = (5, 5),
+) -> zarr.Array:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ return root.create_array(
+ "counts",
+ shape=shape,
+ chunks=chunks,
+ dtype=np.uint32,
+ fill_value=0,
+ )
+
+
+def test_feature_stream_packs_only_adjacent_whole_bins() -> None:
+ array = _array(shape=(12, 20))
+ plan = plan_feature_stream(
+ array,
+ featureAxis=1,
+ cellAxis=0,
+ featureIndices=np.array([0, 1, 5, 6, 17]),
+ cellIndices=np.arange(12),
+ resources=ResourceBudget(10_000, 4),
+ blockBytes=lambda width: width * 100,
+ )
+
+ assert [block.bins for block in plan.blocks] == [(0, 1), (3,)]
+ np.testing.assert_array_equal(
+ plan.blocks[0].destinations,
+ np.array([0, 1, 2, 3]),
+ )
+
+
+def test_feature_stream_preserves_destination_order_for_unsorted_features() -> None:
+ array = _array()
+ plan = plan_feature_stream(
+ array,
+ featureAxis=1,
+ cellAxis=0,
+ featureIndices=np.array([12, 1, 6]),
+ cellIndices=np.arange(12),
+ resources=ResourceBudget(10_000, 2),
+ blockBytes=lambda width: width * 100,
+ )
+
+ destinations = np.concatenate([block.destinations for block in plan.blocks])
+ features = np.concatenate([block.indices for block in plan.blocks])
+ restored = np.empty(3, dtype=np.int64)
+ restored[destinations] = features
+ np.testing.assert_array_equal(restored, np.array([12, 1, 6]))
+
+
+def test_feature_stream_splits_one_bin_and_counts_repeated_tiles() -> None:
+ array = _array(shape=(12, 5), chunks=(5, 5))
+ plan = plan_feature_stream(
+ array,
+ featureAxis=1,
+ cellAxis=0,
+ featureIndices=np.arange(5),
+ cellIndices=np.array([0, 6, 11]),
+ resources=ResourceBudget(350, 4),
+ blockBytes=lambda width: width * 100,
+ )
+
+ assert [block.indices.size for block in plan.blocks] == [2, 2, 1]
+ assert plan.repeatedDecodeCount == 6
+
+
+def test_feature_stream_budgets_an_edge_chunk_at_its_nominal_size() -> None:
+ # A 5 x 3 chunk of uint32 decodes to 60 bytes even where it overhangs the
+ # 12 x 7 array and only 2 x 1 of its elements are addressable.
+ array = _array(shape=(12, 7), chunks=(5, 3))
+ kwargs = {
+ "featureAxis": 1,
+ "cellAxis": 0,
+ "featureIndices": np.array([6]),
+ "cellIndices": np.array([10, 11]),
+ "blockBytes": lambda width: width,
+ }
+
+ plan = plan_feature_stream(array, resources=ResourceBudget(61, 1), **kwargs)
+ assert plan.geometry.nominalChunkBytes() == 60
+ assert len(plan.blocks) == 1
+
+ with pytest.raises(MemoryError):
+ plan_feature_stream(array, resources=ResourceBudget(60, 1), **kwargs)
+
+
+def test_feature_stream_rejects_unaffordable_override() -> None:
+ array = _array(shape=(12, 10), chunks=(5, 5))
+ with pytest.raises(
+ MemoryError,
+ match="Requested feature batch width 5.*affordable width is 2",
+ ):
+ plan_feature_stream(
+ array,
+ featureAxis=1,
+ cellAxis=0,
+ featureIndices=np.arange(10),
+ cellIndices=np.arange(12),
+ resources=ResourceBudget(350, 2),
+ blockBytes=lambda width: width * 100,
+ requestedBatchSize=5,
+ )
+
+
+def test_feature_stream_accounts_for_resident_bytes() -> None:
+ array = _array(shape=(12, 10), chunks=(5, 5))
+ plan = plan_feature_stream(
+ array,
+ featureAxis=1,
+ cellAxis=0,
+ featureIndices=np.arange(10),
+ cellIndices=np.arange(12),
+ resources=ResourceBudget(1_000, 4),
+ residentBytes=600,
+ blockBytes=lambda width: width * 100,
+ )
+
+ widest = max(block.indices.size for block in plan.blocks)
+ decode = plan.geometry.nominalChunkBytes()
+ assert widest == 3
+ assert 600 + widest * 100 + decode <= 1_000
+
+
+def test_feature_stream_keeps_reads_and_their_decodes_inside_the_budget() -> None:
+ array = _array(shape=(20, 40), chunks=(5, 5))
+ plan = plan_feature_stream(
+ array,
+ featureAxis=1,
+ cellAxis=0,
+ featureIndices=np.array([0, 10, 20, 30]),
+ cellIndices=np.arange(20),
+ resources=ResourceBudget(650, 8),
+ blockBytes=lambda _width: 100,
+ )
+
+ decode = plan.geometry.nominalChunkBytes()
+ in_flight = plan.readWorkers * (100 + plan.ioConcurrency * decode)
+
+ assert len(plan.blocks) == 4
+ assert plan.readWorkers == 2
+ assert in_flight + 100 + decode <= 650
+ assert plan.readWorkers * plan.ioConcurrency <= 8
+
+
+def test_feature_stream_records_optional_shard_geometry() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ array = root.create_array(
+ "counts",
+ shape=(20, 20),
+ chunks=(5, 5),
+ shards=(10, 10),
+ dtype=np.uint32,
+ )
+ plan = plan_feature_stream(
+ array,
+ featureAxis=1,
+ cellAxis=0,
+ featureIndices=np.array([0]),
+ cellIndices=np.array([0]),
+ resources=ResourceBudget(1_000, 1),
+ blockBytes=lambda _width: 1,
+ )
+
+ assert plan.geometry.shards == (10, 10)
+
+
+def test_feature_stream_packs_many_single_feature_bins() -> None:
+ array = _array(shape=(10, 2_000), chunks=(5, 1))
+ plan = plan_feature_stream(
+ array,
+ featureAxis=1,
+ cellAxis=0,
+ featureIndices=np.arange(2_000),
+ cellIndices=np.arange(10),
+ resources=ResourceBudget(10_000, 2),
+ blockBytes=lambda width: width,
+ )
+
+ assert len(plan.blocks) == 1
+ assert plan.blocks[0].bins[0] == 0
+ assert plan.blocks[0].bins[-1] == 1_999
diff --git a/tests/test_frozen_master_compat.py b/tests/test_frozen_master_compat.py
new file mode 100644
index 00000000..a5a0bfd9
--- /dev/null
+++ b/tests/test_frozen_master_compat.py
@@ -0,0 +1,301 @@
+"""Master-format compatibility against a real frozen Scarf store.
+
+These tests download a genuine pre-1.0 (master-format, Zarr v2) store from the
+public cytebase bucket instead of hand-building synthetic hybrids. They pin the
+read paths that a released store must keep working: encoded graph-chain lookup,
+graph loading, initial embedding, and Paris recompute from a legacy dendrogram.
+They also assert that a read-only open does not mutate the store on disk.
+They are marked ``integration`` because they require network access, but the
+project's default test command includes integration tests.
+
+The corpus lives under a ``_legacy_master`` dataset. Cytebase publishes the
+current-format store under the plain dataset name, and the archive it replaced
+is preserved alongside it so this frozen layout stays reachable. See
+``scripts/publish_docs_datasets.py``.
+"""
+
+import hashlib
+import os
+import shutil
+
+import numpy as np
+import pytest
+
+_FROZEN_DATASET = "tenx_5K_pbmc_rnaseq_legacy_master"
+_ASSAY = "RNA"
+_CELL_KEY = "I"
+_FEAT_KEY = "hvgs"
+_EXPECTED_GRAPH_LOC = (
+ "RNA/normed__I__hvgs/reduction__pca__15__I/"
+ "ann__l2__50__50__48__4466/knn__11/graph__1.0__1.5"
+)
+_EXPECTED_KNN_LOC = _EXPECTED_GRAPH_LOC.rsplit("/", 1)[0]
+_EXPECTED_NORMED_LOC = "RNA/normed__I__hvgs"
+
+
+def _tree_digest(root: str) -> str:
+ """Order-independent digest of every file path, size, and content."""
+ digest = hashlib.blake2b(digest_size=32)
+ for dirpath, _dirnames, filenames in sorted(os.walk(root)):
+ for name in sorted(filenames):
+ path = os.path.join(dirpath, name)
+ rel = os.path.relpath(path, root)
+ with open(path, "rb") as handle:
+ payload = handle.read()
+ digest.update(rel.encode("utf-8"))
+ digest.update(len(payload).to_bytes(8, "little"))
+ digest.update(payload)
+ return digest.hexdigest()
+
+
+def _array_digest(array: np.ndarray) -> str:
+ values = np.asarray(array)
+ digest = hashlib.blake2b(digest_size=32)
+ digest.update(str(values.shape).encode())
+ digest.update(str(values.dtype).encode())
+ digest.update(values.tobytes(order="C"))
+ return digest.hexdigest()
+
+
+@pytest.fixture(scope="module")
+def frozen_master_store(tmp_path_factory) -> str:
+ from scarf import cytebase
+
+ dest = tmp_path_factory.mktemp("frozen_master")
+ repository = cytebase.connect("scarf_docs")
+ dataset_path = repository.download_dataset(_FROZEN_DATASET, dest, zarr=True)
+ store_path = os.path.join(str(dataset_path), "data.zarr")
+ assert os.path.isdir(store_path)
+ # A master-format store is Zarr v2 (uses .zgroup, not zarr.json).
+ assert os.path.isfile(os.path.join(store_path, ".zgroup"))
+ return store_path
+
+
+@pytest.mark.integration
+def test_frozen_master_store_is_v2_master_layout(frozen_master_store: str) -> None:
+ # The legacy graph chain and per-cluster markers are the master layout that
+ # later phases must not silently break.
+ graph_leaf = os.path.join(
+ frozen_master_store,
+ "RNA/normed__I__hvgs/reduction__pca__15__I/"
+ "ann__l2__50__50__48__4466/knn__11/graph__1.0__1.5",
+ )
+ assert os.path.isdir(graph_leaf)
+ dendrogram = os.path.join(graph_leaf, "dendrogram")
+ assert os.path.isdir(dendrogram)
+ markers = os.path.join(frozen_master_store, "RNA/markers/I__RNA_cluster/1")
+ assert os.path.isdir(markers)
+
+
+@pytest.mark.integration
+def test_frozen_master_graph_lookup_and_load(frozen_master_store: str) -> None:
+ from scarf.datastore.datastore import DataStore
+ from scarf.graph.state import validate_legacy_graph_selection
+
+ datastore = DataStore(frozen_master_store, default_assay=_ASSAY, zarr_mode="r")
+
+ assert (
+ datastore.get_latest_graph_loc(_ASSAY, _CELL_KEY, _FEAT_KEY)
+ == _EXPECTED_GRAPH_LOC
+ )
+ assert (
+ datastore.get_normalized_group_path(_ASSAY, _CELL_KEY, _FEAT_KEY)
+ == _EXPECTED_NORMED_LOC
+ )
+ stored = datastore._lookup_stored_graph(_ASSAY, _CELL_KEY, _FEAT_KEY)
+ assert stored.paths.cell_graph_group_path == _EXPECTED_GRAPH_LOC
+ assert stored.ann_metric == "l2"
+ assert stored.k == 11
+
+ latest_graph = datastore.load_graph(
+ from_assay=_ASSAY, cell_key=_CELL_KEY, feat_key=_FEAT_KEY
+ )
+ explicit_graph = datastore.load_graph(graph_loc=_EXPECTED_GRAPH_LOC)
+ assert latest_graph.shape == (3940, 3940)
+ assert latest_graph.nnz == 43340
+ assert (latest_graph != explicit_graph).nnz == 0
+ ann_stream = datastore._load_ann_stream(
+ _ASSAY,
+ _CELL_KEY,
+ _FEAT_KEY,
+ knn_loc=_EXPECTED_KNN_LOC,
+ )
+ assert ann_stream.k == 11
+ assert ann_stream.annIdx is not None
+ validate_legacy_graph_selection(
+ datastore,
+ _EXPECTED_KNN_LOC,
+ _ASSAY,
+ _CELL_KEY,
+ _FEAT_KEY,
+ )
+ assert datastore._keys_from_knn_path(_ASSAY, _EXPECTED_KNN_LOC) == (
+ _CELL_KEY,
+ _FEAT_KEY,
+ )
+ graph_group = datastore.zw[_EXPECTED_GRAPH_LOC]
+ assert (
+ _array_digest(graph_group["edges"][:])
+ == "33e3daf56516e5ea4869bf312e2033891b0cdd78232cfd8232addd07b863b36a"
+ )
+ assert (
+ _array_digest(graph_group["weights"][:])
+ == "a29ccc26aac9d7024327b824afb3f8ca0c030c3ecf5428b6ef3970ba6ff9f06a"
+ )
+ assert (
+ _array_digest(graph_group["dendrogram"][:])
+ == "f30c9ba746100c1217ee1b093c6cc71775e7f2eff0e3f0d06b73ab21257b2236"
+ )
+
+
+@pytest.mark.integration
+def test_frozen_master_initial_embedding_is_finite(
+ frozen_master_store: str,
+) -> None:
+ from scarf.datastore.datastore import DataStore
+
+ datastore = DataStore(frozen_master_store, default_assay=_ASSAY, zarr_mode="r")
+ graph = datastore.load_graph(
+ from_assay=_ASSAY, cell_key=_CELL_KEY, feat_key=_FEAT_KEY
+ )
+ ini_embed = datastore._get_ini_embed(_ASSAY, _CELL_KEY, _FEAT_KEY, 2)
+ assert ini_embed.shape == (graph.shape[0], 2)
+ assert np.isfinite(ini_embed).all()
+
+
+@pytest.mark.integration
+def test_frozen_master_read_only_open_does_not_mutate(
+ frozen_master_store: str,
+) -> None:
+ from scarf.datastore.datastore import DataStore
+
+ before = _tree_digest(frozen_master_store)
+ datastore = DataStore(frozen_master_store, default_assay=_ASSAY, zarr_mode="r")
+ datastore.get_normalized_group_path(_ASSAY, _CELL_KEY, _FEAT_KEY)
+ datastore.load_graph(graph_loc=_EXPECTED_GRAPH_LOC)
+ datastore._lookup_stored_graph(_ASSAY, _CELL_KEY, _FEAT_KEY)
+ datastore._load_ann_stream(
+ _ASSAY,
+ _CELL_KEY,
+ _FEAT_KEY,
+ knn_loc=_EXPECTED_KNN_LOC,
+ )
+ datastore._get_ini_embed(_ASSAY, _CELL_KEY, _FEAT_KEY, 2)
+ after = _tree_digest(frozen_master_store)
+ assert before == after
+
+
+@pytest.mark.integration
+def test_frozen_master_reads_and_recomputes_markers_without_mutation(
+ frozen_master_store: str,
+) -> None:
+ from scarf.datastore.datastore import DataStore
+
+ before = _tree_digest(frozen_master_store)
+ datastore = DataStore(frozen_master_store, default_assay=_ASSAY, zarr_mode="r")
+ stored = datastore.get_markers(
+ from_assay=_ASSAY,
+ cell_key=_CELL_KEY,
+ group_key="RNA_cluster",
+ group_id=1,
+ min_score=0,
+ min_frac_exp=0,
+ )
+ computed = datastore.run_marker_search(
+ from_assay=_ASSAY,
+ cell_key=_CELL_KEY,
+ feat_key="I__hvgs",
+ group_key="RNA_cluster",
+ gene_batch_size=100,
+ n_threads=1,
+ skip_save=True,
+ )
+
+ assert not stored.empty
+ assert computed
+ assert all(frame.shape[0] > 0 for frame in computed.values())
+ assert _tree_digest(frozen_master_store) == before
+
+
+@pytest.mark.integration
+def test_frozen_master_writable_copy_publishes_canonical_markers_additively(
+ frozen_master_store: str,
+ tmp_path,
+) -> None:
+ from scarf.datastore.datastore import DataStore
+ from scarf.features.markers.table import MARKER_STAT_COLUMNS
+
+ working_copy = str(tmp_path / "data.zarr")
+ shutil.copytree(frozen_master_store, working_copy)
+ legacy_subtree = os.path.join(
+ working_copy,
+ "RNA/markers/I__RNA_cluster",
+ )
+ legacy_before = _tree_digest(legacy_subtree)
+ datastore = DataStore(working_copy, default_assay=_ASSAY, zarr_mode="r+")
+
+ datastore.run_marker_search(
+ from_assay=_ASSAY,
+ cell_key=_CELL_KEY,
+ feat_key="I__hvgs",
+ group_key="RNA_cluster",
+ gene_batch_size=100,
+ n_threads=1,
+ )
+
+ assert _tree_digest(legacy_subtree) == legacy_before
+ published = datastore._resolve_marker_group(
+ _ASSAY,
+ _CELL_KEY,
+ "RNA_cluster",
+ )
+ assert "schema_version" not in published.attrs
+ assert list(published.attrs["stat_columns"]) == list(MARKER_STAT_COLUMNS)
+ assert published.attrs["method"] == "mannwhitneyu"
+ assert published.attrs["alternative"] == "two-sided"
+ assert published.attrs["adjustment_method"] == "fdr_bh"
+ assert published.attrs["adjustment_scope"] == ("within_group_all_tested_features")
+ assert all(
+ published[group].attrs["n_group"] >= 2
+ and published[group].attrs["n_reference"] >= 2
+ for group in published.group_keys()
+ )
+
+
+@pytest.mark.integration
+def test_frozen_master_legacy_dendrogram_triggers_hierarchy_rebuild(
+ frozen_master_store: str,
+ tmp_path,
+) -> None:
+ from scarf.datastore.datastore import DataStore
+ from scarf.storage.artifacts import list_artifacts
+
+ working_copy = str(tmp_path / "data.zarr")
+ shutil.copytree(frozen_master_store, working_copy)
+
+ # The legacy dendrogram signals that the current hierarchy must be refit
+ # from the stored graph. Its values are not reused as a current generation.
+ hierarchy_root = f"{_EXPECTED_GRAPH_LOC}/paris_hierarchy"
+ datastore = DataStore(working_copy, default_assay=_ASSAY, zarr_mode="r+")
+ assert hierarchy_root not in datastore.zw
+
+ result = datastore.run_paris_clustering(
+ from_assay=_ASSAY,
+ cell_key=_CELL_KEY,
+ feat_key=_FEAT_KEY,
+ n_clusters=8,
+ label="frozen_paris",
+ )
+
+ labels = datastore.cells.fetch("RNA_frozen_paris")
+ assert len(set(labels.tolist())) == 8
+ np.testing.assert_array_equal(labels, result.labels)
+ # The released graph remains untouched; the new hierarchy is independent.
+ assert hierarchy_root not in datastore.zw
+ hierarchies = list_artifacts(
+ datastore.zw,
+ scope="assay",
+ assay=_ASSAY,
+ kind="cluster_hierarchy",
+ )
+ assert result.hierarchy_generation_id in {ref.artifact_id for ref in hierarchies}
diff --git a/tests/test_graph_arguments.py b/tests/test_graph_arguments.py
new file mode 100644
index 00000000..5d252c44
--- /dev/null
+++ b/tests/test_graph_arguments.py
@@ -0,0 +1,336 @@
+from dataclasses import fields
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.graph.arguments import (
+ AnnIndexArguments,
+ ConnectivityMapArguments,
+ CustomReductionArguments,
+ EmbeddingInitializationArguments,
+ FeatureScalingArguments,
+ HarmonyArguments,
+ LsiArguments,
+ NeighborQueryArguments,
+ NormalizationArguments,
+ PcaArguments,
+ artifact_input,
+ execution,
+ parameter,
+)
+from scarf.storage.artifact_writer import finish_artifact, start_artifact
+from scarf.storage.artifacts import ArtifactRef
+
+
+def _ref(
+ kind: str,
+ token: str,
+ *,
+ scope: str = "assay",
+) -> ArtifactRef:
+ return ArtifactRef(
+ scope=scope, # type: ignore[arg-type]
+ assay="RNA" if scope == "assay" else None,
+ kind=kind,
+ artifact_id=token * 64,
+ )
+
+
+def test_normalization_arguments_partition_every_value() -> None:
+ arguments = NormalizationArguments(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ cell_selection=_ref("cell_selection", "2", scope="datastore"),
+ feature_selection=_ref("feature_selection", "3"),
+ normalization_method="norm_lib_size",
+ size_factor=1000.0,
+ log_transform=True,
+ renormalize_subset=False,
+ update_state=True,
+ invalidate_cache=False,
+ )
+ record = arguments.to_record()
+
+ assert set(record.inputs) == {"cell_selection", "feature_selection"}
+ assert record.parameters == {
+ "normalization_method": "norm_lib_size",
+ "size_factor": 1000.0,
+ "log_transform": True,
+ "renormalize_subset": False,
+ }
+ assert record.execution_options == {
+ "from_assay": "RNA",
+ "cell_key": "I",
+ "feat_key": "hvgs",
+ "update_state": True,
+ "invalidate_cache": False,
+ }
+
+
+def test_execution_options_do_not_change_provenance_hash() -> None:
+ common = {
+ "from_assay": "RNA",
+ "cell_key": "I",
+ "feat_key": "hvgs",
+ "cell_selection": _ref("cell_selection", "2", scope="datastore"),
+ "feature_selection": _ref("feature_selection", "3"),
+ "normalization_method": "norm_lib_size",
+ "size_factor": 1000.0,
+ "log_transform": True,
+ "renormalize_subset": False,
+ "update_state": True,
+ }
+ small_batch = NormalizationArguments(
+ invalidate_cache=False,
+ **common,
+ )
+ invalidated = NormalizationArguments(
+ invalidate_cache=True,
+ **common,
+ )
+ changed_parameter = NormalizationArguments(
+ invalidate_cache=False,
+ **(common | {"log_transform": False}),
+ )
+ changed_size_factor = NormalizationArguments(
+ invalidate_cache=False,
+ **(common | {"size_factor": 2000.0}),
+ )
+
+ assert small_batch.provenance_hash() == invalidated.provenance_hash()
+ assert small_batch.provenance_hash() != changed_parameter.provenance_hash()
+ assert small_batch.provenance_hash() != changed_size_factor.provenance_hash()
+
+
+def test_reduction_fingerprints_custom_loadings_as_input() -> None:
+ common = {
+ "normalized": _ref("normalized", "a"),
+ "feature_scaling": _ref("feature_scaling", "c"),
+ "update_state": False,
+ "invalidate_cache": False,
+ }
+ first = CustomReductionArguments(
+ loadings=np.arange(6, dtype=np.float64).reshape(3, 2),
+ **common,
+ )
+ copied = CustomReductionArguments(
+ loadings=np.arange(6, dtype=np.float64).reshape(3, 2),
+ **common,
+ )
+ changed = CustomReductionArguments(
+ loadings=np.ones((3, 2), dtype=np.float64),
+ **common,
+ )
+
+ assert first.provenance_hash() == copied.provenance_hash()
+ assert first.provenance_hash() != changed.provenance_hash()
+ assert "value_fingerprint" in first.to_record().inputs["loadings"]
+
+
+def test_stage_models_chain_logical_artifact_refs() -> None:
+ scaling = FeatureScalingArguments(
+ normalized=_ref("normalized", "0"),
+ enabled=True,
+ batch_size=100,
+ )
+ harmony = HarmonyArguments(
+ reduction=_ref("reduction", "1"),
+ batch_values=_ref("metadata_snapshot", "2", scope="datastore"),
+ batch_columns=("donor", "sample"),
+ harmony_parameters={"theta": 2.0},
+ algorithm_version="centroid_snapshot_v2",
+ batch_size=100,
+ )
+ neighbors = NeighborQueryArguments(
+ ann_index=_ref("ann_index", "3"),
+ coordinates=_ref("batch_correction", "4"),
+ k=15,
+ distance_metric="l2",
+ batch_size=100,
+ )
+ connectivity = ConnectivityMapArguments(
+ neighbors=_ref("neighbors", "5"),
+ local_connectivity=1.0,
+ bandwidth=1.5,
+ )
+
+ assert harmony.to_record().parameters["batch_columns"] == ["donor", "sample"]
+ assert scaling.to_record().parameters == {"enabled": True}
+ assert neighbors.to_record().inputs["ann_index"]["kind"] == "ann_index"
+ assert connectivity.to_record().parameters == {
+ "local_connectivity": 1.0,
+ "bandwidth": 1.5,
+ }
+
+
+def test_reduction_batch_size_does_not_change_reuse() -> None:
+ normalized = _ref("normalized", "0")
+ scaling_small = FeatureScalingArguments(
+ normalized=normalized,
+ enabled=False,
+ batch_size=100,
+ )
+ scaling_large = FeatureScalingArguments(
+ normalized=normalized,
+ enabled=False,
+ batch_size=500,
+ )
+ lsi_common = {
+ "normalized": normalized,
+ "feature_scaling": _ref("feature_scaling", "1"),
+ "dims": 5,
+ "skip_first": True,
+ "rand_state": 4466,
+ "solver": "streaming",
+ "n_iter": 5,
+ "n_oversamples": 10,
+ "update_state": False,
+ "invalidate_cache": False,
+ }
+ lsi_small = LsiArguments(batch_size=100, **lsi_common)
+ lsi_large = LsiArguments(batch_size=500, **lsi_common)
+ pca_common = {
+ "normalized": normalized,
+ "feature_scaling": _ref("feature_scaling", "1"),
+ "pca_cell_selection": _ref("cell_selection", "2", scope="datastore"),
+ "pca_cell_key": "I",
+ "dims": 5,
+ "feat_scaling": True,
+ "show_elbow_plot": False,
+ "update_state": False,
+ "invalidate_cache": False,
+ }
+ pca_small = PcaArguments(batch_size=100, **pca_common)
+ pca_large = PcaArguments(batch_size=500, **pca_common)
+
+ assert scaling_small.provenance_hash() == scaling_large.provenance_hash()
+ assert lsi_small.provenance_hash() == lsi_large.provenance_hash()
+ assert pca_small.provenance_hash() == pca_large.provenance_hash()
+
+
+def test_embedding_initialization_parameters_change_provenance() -> None:
+ initialization_common = {
+ "reduction": _ref("reduction", "3"),
+ "n_centroids": 20,
+ "rand_state": 4466,
+ "invalidate_cache": False,
+ }
+ initialization_small = EmbeddingInitializationArguments(
+ batch_size=100,
+ **initialization_common,
+ )
+ initialization_large = EmbeddingInitializationArguments(
+ batch_size=500,
+ **initialization_common,
+ )
+ initialization_sampled = EmbeddingInitializationArguments(
+ batch_size=100,
+ kmeans_sampling=0.2,
+ **initialization_common,
+ )
+ initialization_larger_minibatch = EmbeddingInitializationArguments(
+ batch_size=100,
+ kmeans_batch_size=20_000,
+ **initialization_common,
+ )
+ initialization_new_algorithm = EmbeddingInitializationArguments(
+ batch_size=100,
+ algorithm_version="minibatch_kmeans_v3",
+ **initialization_common,
+ )
+
+ assert (
+ initialization_small.provenance_hash() != initialization_large.provenance_hash()
+ )
+ assert (
+ initialization_small.provenance_hash()
+ != initialization_sampled.provenance_hash()
+ )
+ assert (
+ initialization_small.provenance_hash()
+ != initialization_larger_minibatch.provenance_hash()
+ )
+ assert (
+ initialization_small.provenance_hash()
+ != initialization_new_algorithm.provenance_hash()
+ )
+
+
+def test_ann_parallel_is_normal_provenance_not_cache_policy() -> None:
+ common = {
+ "coordinates": _ref("reduction", "1"),
+ "ann_metric": "l2",
+ "ann_efc": 50,
+ "ann_ef": 50,
+ "ann_m": 16,
+ "rand_state": 4466,
+ "batch_size": 100,
+ "invalidate_cache": False,
+ }
+ serial = AnnIndexArguments(
+ ann_parallel=False,
+ parallel_threads=None,
+ **common,
+ )
+ parallel = AnnIndexArguments(
+ ann_parallel=True,
+ parallel_threads=4,
+ **common,
+ )
+
+ assert serial.provenance_hash() != parallel.provenance_hash()
+ assert parallel.to_record().parameters["ann_parallel"] is True
+
+
+def test_dynamic_callable_requires_explicit_identity() -> None:
+ with pytest.raises(ValueError, match="artifact_identity"):
+ NormalizationArguments(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ cell_selection=_ref(
+ "cell_selection",
+ "2",
+ scope="datastore",
+ ),
+ feature_selection=_ref("feature_selection", "3"),
+ normalization_method=lambda values: values,
+ size_factor=1000.0,
+ log_transform=True,
+ renormalize_subset=False,
+ update_state=True,
+ ).to_record()
+
+
+def test_field_factories_set_argument_roles() -> None:
+ assert parameter().metadata["argument_role"] == "parameter"
+ assert execution(False).metadata["argument_role"] == "execution"
+ assert artifact_input().metadata["argument_role"] == "input"
+ roles = {
+ model_field.name: model_field.metadata["argument_role"]
+ for model_field in fields(NormalizationArguments)
+ }
+ assert roles["normalization_method"] == "parameter"
+ assert roles["cell_selection"] == "input"
+ assert roles["invalidate_cache"] == "execution"
+
+
+def test_operation_arguments_plan_reuses_complete_artifact() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ arguments = FeatureScalingArguments(
+ normalized=_ref("normalized", "1"),
+ enabled=True,
+ batch_size=100,
+ invalidate_cache=False,
+ )
+ first = arguments.plan(root, scope="assay", assay="RNA")
+ group = start_artifact(root, first)
+ group.create_array("mean", data=np.array([1.0, 2.0]))
+ finish_artifact(group, first)
+
+ reused = arguments.plan(root, scope="assay", assay="RNA")
+ assert reused.reused
+ assert reused.ref == first.ref
diff --git a/tests/test_graph_cache.py b/tests/test_graph_cache.py
new file mode 100644
index 00000000..79f44503
--- /dev/null
+++ b/tests/test_graph_cache.py
@@ -0,0 +1,59 @@
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.datastore.graph_datastore import GraphDataStore
+
+
+def _memory_group():
+ return zarr.open_group(store=MemoryStore(), mode="w")
+
+
+def test_get_latest_graph_loc_is_public(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.validate_legacy_graph_selection",
+ lambda *_args, **_kwargs: None,
+ )
+ root = _memory_group()
+ normed = root.create_group("RNA/normed__I__hvgs")
+ reduction = root.create_group(f"{normed.path}/reduction__pca__10__I")
+ ann = root.create_group(f"{reduction.path}/ann__l2__50__50__16__1")
+ knn = root.create_group(f"{ann.path}/knn__11")
+ graph = root.create_group(f"{knn.path}/graph__1.0__1.5")
+ normed.attrs["latest_reduction"] = reduction.path
+ reduction.attrs["latest_ann"] = ann.path
+ ann.attrs["latest_knn"] = knn.path
+ knn.attrs["latest_graph"] = graph.path
+
+ store = GraphDataStore.__new__(GraphDataStore)
+ store.z = root
+ store.workspace = None
+
+ assert store.get_normalized_group_path("RNA", "I", "hvgs") == "RNA/normed__I__hvgs"
+ assert store.get_latest_graph_loc("RNA", "I", "hvgs") == graph.path
+
+ del knn.attrs["latest_graph"]
+ with pytest.raises(KeyError):
+ store.get_latest_graph_loc("RNA", "I", "hvgs")
+
+
+def test_resolve_local_cache_plan(tmp_path):
+ local_root = _memory_group()
+ enabled, base, remove = GraphDataStore._resolve_local_cache_plan(
+ "/tmp/local.zarr", local_root, "auto"
+ )
+ assert enabled is False
+
+ enabled, base, remove = GraphDataStore._resolve_local_cache_plan(
+ "s3://bucket/path", local_root, False
+ )
+ assert enabled is False
+
+ enabled, base, remove = GraphDataStore._resolve_local_cache_plan(
+ "s3://bucket/path", local_root, str(tmp_path / "cache")
+ )
+ assert enabled is True
+ assert base == str(tmp_path / "cache")
+ assert remove is False
diff --git a/tests/test_graph_construction_api.py b/tests/test_graph_construction_api.py
new file mode 100644
index 00000000..5635299b
--- /dev/null
+++ b/tests/test_graph_construction_api.py
@@ -0,0 +1,1159 @@
+import numpy as np
+import pandas as pd
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.datastore._operations import graph as graph_operations
+from scarf.embeddings.harmony import fit_harmony
+from scarf.graph.state import stored_assay_graph_from_ref
+from scarf.storage.artifacts import ArtifactRef, artifact_path
+from scarf.storage.budget import ResourceBudget
+from scarf.utils import logger
+from tests import full_path
+
+pytestmark = pytest.mark.slow
+
+
+def test_streaming_lsi_block_rows_respect_memory_budget() -> None:
+ group = zarr.open_group(store=MemoryStore(), mode="w")
+ array = group.create_array(
+ "normalized",
+ shape=(100, 20),
+ chunks=(10, 20),
+ dtype=np.float32,
+ )
+
+ constrained = graph_operations._streaming_lsi_block_rows(
+ array,
+ ResourceBudget(memoryBytes=5_000, workers=1),
+ n_components=3,
+ n_oversamples=2,
+ )
+ roomy = graph_operations._streaming_lsi_block_rows(
+ array,
+ ResourceBudget(memoryBytes=1_000_000, workers=1),
+ n_components=3,
+ n_oversamples=2,
+ )
+
+ assert 1 <= constrained < roomy
+ assert roomy == array.shape[0]
+ with pytest.raises(MemoryError, match="Streaming LSI needs about"):
+ graph_operations._streaming_lsi_block_rows(
+ array,
+ ResourceBudget(memoryBytes=1_000, workers=1),
+ n_components=3,
+ n_oversamples=2,
+ )
+
+
+def _prepare_graph_features(datastore) -> None:
+ datastore.auto_filter_cells(show_qc_plots=False)
+ if "I__graph_hvgs" not in datastore.get_assay("RNA").feats.columns:
+ datastore.mark_hvgs(
+ from_assay="RNA",
+ cell_key="I",
+ top_n=100,
+ hvg_key_name="graph_hvgs",
+ show_plot=False,
+ )
+
+
+def test_graph_construction_methods_chain_refs_and_select_current_results(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="graph_hvgs",
+ )
+ pca = datastore.run_pca(normalized, dims=5, batch_size=100)
+ ann = datastore.build_ann_index(pca, batch_size=100)
+ neighbors = datastore.query_neighbors(ann, k=3, batch_size=100)
+ graph = datastore.build_connectivity_map(neighbors)
+
+ assert all(
+ isinstance(ref, ArtifactRef) for ref in (normalized, pca, ann, neighbors, graph)
+ )
+ normalized_group = datastore.zw[artifact_path(normalized)]
+ assert normalized_group["feature_sum"].dtype == np.dtype(np.float64)
+ assert normalized_group["feature_squared_sum"].dtype == np.dtype(np.float64)
+ normalized_values = normalized_group["data"][:]
+ np.testing.assert_allclose(
+ normalized_group["feature_sum"][:],
+ normalized_values.sum(axis=0, dtype=np.float64),
+ rtol=1e-6,
+ )
+ np.testing.assert_allclose(
+ normalized_group["feature_squared_sum"][:],
+ np.square(normalized_values, dtype=np.float64).sum(axis=0),
+ rtol=1e-6,
+ )
+ reduction_group = datastore.zw[artifact_path(pca)]
+ assert reduction_group["loadings"].dtype == np.dtype(np.float64)
+ assert reduction_group["data"].dtype == np.dtype(np.float32)
+ assert reduction_group["data"].shape == (
+ int(datastore.cells.fetch_all("I").sum()),
+ 5,
+ )
+ stored_scores = reduction_group["data"][:]
+ _, legacy_projection = datastore._load_reduction_stream(
+ pca,
+ batch_size=100,
+ )
+ legacy_scores = np.concatenate(
+ tuple(legacy_projection.iter_coordinate_blocks("")),
+ axis=0,
+ )
+ np.testing.assert_allclose(
+ stored_scores,
+ legacy_scores,
+ rtol=2e-5,
+ atol=2e-6,
+ )
+ reduction_inputs = datastore.inspect_artifact(pca).inputs
+ assert reduction_inputs is not None
+ scaling = ArtifactRef.from_dict(reduction_inputs["feature_scaling"])
+ scaling_group = datastore.load_artifact(scaling)
+ assert scaling_group["mean"].dtype == np.dtype(np.float64)
+ assert scaling_group["scale"].dtype == np.dtype(np.float64)
+ neighbors_group = datastore.zw[artifact_path(neighbors)]
+ assert neighbors_group["indices"].dtype == np.dtype(np.uint32)
+ assert neighbors_group["distances"].dtype == np.dtype(np.float32)
+ squared_distances = np.square(
+ stored_scores[:, np.newaxis, :] - stored_scores[np.newaxis, :, :],
+ dtype=np.float64,
+ ).sum(axis=2)
+ np.fill_diagonal(squared_distances, np.inf)
+ exact_neighbors = np.argpartition(squared_distances, kth=2, axis=1)[:, :3]
+ approximate_neighbors = neighbors_group["indices"][:]
+ expected_neighbor_distances = np.sqrt(
+ squared_distances[
+ np.arange(len(stored_scores))[:, np.newaxis],
+ approximate_neighbors,
+ ]
+ )
+ np.testing.assert_allclose(
+ neighbors_group["distances"][:],
+ expected_neighbor_distances,
+ rtol=2e-5,
+ atol=2e-6,
+ )
+ recall = np.mean(
+ [
+ len(set(exact) & set(approximate)) / 3
+ for exact, approximate in zip(
+ exact_neighbors,
+ approximate_neighbors,
+ strict=True,
+ )
+ ]
+ )
+ assert recall >= 0.95
+ graph_group = datastore.zw[artifact_path(graph)]
+ assert graph_group["edges"].dtype == np.dtype(np.uint32)
+ assert graph_group["weights"].dtype == np.dtype(np.float32)
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.normalized == normalized
+ assert state.reduction == pca
+ assert state.ann_index == ann
+ assert state.neighbors == neighbors
+ assert state.connectivity_map == graph
+ loaded = datastore.load_graph(graph_loc=artifact_path(graph))
+ assert loaded.shape[0] == int(datastore.cells.fetch_all("I").sum())
+ assert np.isfinite(loaded.data).all()
+
+
+def test_ann_index_logs_rebuild_and_reuse_accurately(datastore_ephemeral) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="graph_hvgs",
+ )
+ reduction = datastore.run_pca(normalized, dims=3)
+ messages: list[str] = []
+ sink = logger.add(
+ lambda message: messages.append(message.record["message"]),
+ level="INFO",
+ )
+ try:
+ first = datastore.build_ann_index(reduction)
+ second = datastore.build_ann_index(reduction)
+ finally:
+ logger.remove(sink)
+
+ assert first == second
+ assert any(message.startswith("Stored ANN index") for message in messages)
+ assert any(message.startswith("Reused ANN index") for message in messages)
+ assert all("Loaded existing ANN stream" not in message for message in messages)
+
+
+@pytest.mark.parametrize("dims", [0, -1, 1.5, True])
+def test_reduction_rejects_invalid_dimensions(datastore_ephemeral, dims) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+
+ with pytest.raises((TypeError, ValueError), match="dims"):
+ datastore.run_pca(normalized, dims=dims, update_state=False)
+
+
+@pytest.mark.parametrize("batch_size", [0, -1, 1.5, True])
+def test_reduction_rejects_invalid_batch_sizes(
+ datastore_ephemeral,
+ batch_size,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+
+ with pytest.raises((TypeError, ValueError), match="batch_size"):
+ datastore.run_pca(
+ normalized,
+ dims=3,
+ batch_size=batch_size,
+ update_state=False,
+ )
+
+
+def test_row_block_expands_to_aligned_minimum(monkeypatch) -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ data = root.create_array(
+ "data",
+ shape=(20, 3),
+ chunks=(2, 3),
+ dtype=np.float32,
+ )
+ warnings: list[str] = []
+ monkeypatch.setattr(graph_operations.logger, "warning", warnings.append)
+
+ assert graph_operations._row_block(data, None, minimum=5) == 6
+ assert graph_operations._row_block(data, 3, minimum=5) == 6
+ assert any("below the required minimum of 5" in message for message in warnings)
+
+
+def test_pca_rejects_empty_fit_selection(datastore_ephemeral) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+ datastore.cells.insert(
+ "no_pca_cells",
+ np.zeros(datastore.cells.N, dtype=bool),
+ overwrite=True,
+ )
+
+ with pytest.raises(ValueError, match="dims \\+ 1 selected cells"):
+ datastore.run_pca(
+ normalized,
+ dims=3,
+ pca_cell_key="no_pca_cells",
+ update_state=False,
+ )
+
+
+def test_custom_reduction_rejects_invalid_loadings(datastore_ephemeral) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+
+ with pytest.raises(ValueError, match="two-dimensional"):
+ datastore.run_custom_reduction(
+ np.ones(4),
+ normalized,
+ update_state=False,
+ )
+
+
+def test_graph_construction_operations_reuse_persistent_local_cache(
+ datastore_ephemeral,
+ monkeypatch,
+ tmp_path,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ cache_path = tmp_path / "normalized_cache"
+ monkeypatch.setattr(
+ graph_operations,
+ "is_remote_datastore",
+ lambda *_args: True,
+ )
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ update_state=False,
+ )
+ normalized_data = datastore.load_artifact(normalized)["data"]
+ assert normalized_data.chunks[0] == normalized_data.shape[0]
+ reduction = datastore.run_pca(
+ normalized,
+ dims=4,
+ batch_size=100,
+ local_cache=str(cache_path),
+ update_state=False,
+ )
+ ann = datastore.build_ann_index(
+ reduction,
+ batch_size=100,
+ update_state=False,
+ )
+ neighbors = datastore.query_neighbors(
+ ann,
+ k=3,
+ batch_size=100,
+ update_state=False,
+ )
+
+ staged = cache_path / normalized.artifact_id / "normed.zarr"
+ assert staged.is_dir()
+ assert datastore.inspect_artifact(normalized).execution_options == {
+ "from_assay": "RNA",
+ "cell_key": "I",
+ "feat_key": "graph_hvgs",
+ "update_state": False,
+ "invalidate_cache": False,
+ }
+ reduction_execution = datastore.inspect_artifact(reduction).execution_options or {}
+ assert reduction_execution["local_cache"] == str(cache_path)
+ for ref in (ann, neighbors):
+ execution = datastore.inspect_artifact(ref).execution_options or {}
+ assert "local_cache" not in execution
+
+
+@pytest.mark.parametrize("local_cache", [True, "auto"])
+def test_temporary_local_cache_is_removed_after_success(
+ datastore_ephemeral,
+ monkeypatch,
+ tmp_path,
+ local_cache,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ update_state=False,
+ )
+ cache_root = tmp_path / str(local_cache).lower()
+
+ def make_cache_dir(*_args, **_kwargs):
+ cache_root.mkdir()
+ return str(cache_root)
+
+ monkeypatch.setattr(
+ graph_operations,
+ "is_remote_datastore",
+ lambda *_args: True,
+ )
+ monkeypatch.setattr(graph_operations.tempfile, "mkdtemp", make_cache_dir)
+
+ with datastore._cache_normalized_artifact(
+ normalized,
+ local_cache,
+ 100,
+ ):
+ assert cache_root.is_dir()
+
+ assert not cache_root.exists()
+
+
+def test_temporary_local_cache_is_removed_after_failure(
+ datastore_ephemeral,
+ monkeypatch,
+ tmp_path,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ update_state=False,
+ )
+ cache_root = tmp_path / "failed"
+
+ def make_cache_dir(*_args, **_kwargs):
+ cache_root.mkdir()
+ return str(cache_root)
+
+ monkeypatch.setattr(
+ graph_operations,
+ "is_remote_datastore",
+ lambda *_args: True,
+ )
+ monkeypatch.setattr(graph_operations.tempfile, "mkdtemp", make_cache_dir)
+
+ with pytest.raises(RuntimeError, match="stop after staging"):
+ with datastore._cache_normalized_artifact(normalized, "auto", 100):
+ raise RuntimeError("stop after staging")
+
+ assert not cache_root.exists()
+
+
+def test_new_reduction_becomes_current_and_clears_downstream_refs(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+ first = datastore.run_pca(normalized, dims=4)
+ ann = datastore.build_ann_index(first)
+ neighbors = datastore.query_neighbors(ann, k=3)
+
+ assert datastore.run_pca(normalized, dims=4) == first
+ reused_state = datastore.get_assay_state("RNA")
+ assert reused_state is not None
+ assert reused_state.ann_index == ann
+ assert reused_state.neighbors == neighbors
+
+ second = datastore.run_pca(normalized, dims=6)
+ state = datastore.get_assay_state("RNA")
+
+ assert state is not None
+ assert state.reduction == second
+ assert state.ann_index is None
+ assert state.neighbors is None
+ assert state.connectivity_map is None
+
+
+def test_embedding_initialization_preserves_same_reduction_graph_state(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+ reduction = datastore.run_pca(normalized, dims=4)
+ ann = datastore.build_ann_index(reduction)
+ neighbors = datastore.query_neighbors(ann, k=3)
+ graph = datastore.build_connectivity_map(neighbors)
+ before = datastore.get_assay_state("RNA")
+ assert before is not None
+
+ initialization = datastore.build_embedding_initialization(
+ reduction,
+ n_centroids=5,
+ batch_size=2,
+ kmeans_batch_size=5,
+ )
+
+ after = datastore.get_assay_state("RNA")
+ assert after is not None
+ initialization_group = datastore.load_artifact(initialization)
+ assert initialization_group["cluster_centers"].shape == (5, 4)
+ assert initialization_group["cluster_labels"].dtype == np.uint32
+ assert after.embedding_initialization == initialization
+ assert after.batch_correction == before.batch_correction
+ assert after.ann_index == ann
+ assert after.neighbors == neighbors
+ assert after.connectivity_map == graph
+ assert after.named_results == before.named_results
+
+
+def test_connectivity_rejects_invalid_kernel_parameters(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+ reduction = datastore.run_pca(normalized, dims=4)
+ ann = datastore.build_ann_index(reduction)
+ neighbors = datastore.query_neighbors(ann, k=3)
+
+ for values in (
+ {"local_connectivity": -1.0},
+ {"local_connectivity": np.nan},
+ {"local_connectivity": True},
+ {"bandwidth": 0.0},
+ {"bandwidth": -1.0},
+ {"bandwidth": np.nan},
+ {"bandwidth": True},
+ ):
+ with pytest.raises((TypeError, ValueError)):
+ datastore.build_connectivity_map(
+ neighbors,
+ update_state=False,
+ **values,
+ )
+
+
+def test_ann_index_rejects_invalid_runtime_parameters(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+ reduction = datastore.run_pca(normalized, dims=4)
+
+ for values, error, match in (
+ ({"ann_metric": "ip"}, ValueError, "l2, cosine"),
+ ({"ann_efc": 1.5}, TypeError, "positive integer"),
+ ({"ann_ef": True}, TypeError, "positive integer"),
+ ({"ann_m": 1}, ValueError, "at least two"),
+ ({"rand_state": 0}, ValueError, "greater than zero"),
+ ({"batch_size": 0}, ValueError, "greater than zero"),
+ ):
+ with pytest.raises(error, match=match):
+ datastore.build_ann_index(
+ reduction,
+ update_state=False,
+ **values,
+ )
+
+
+def test_neighbor_count_changes_only_neighbor_and_connectivity_artifacts(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ update_state=False,
+ )
+ reduction = datastore.run_pca(
+ normalized,
+ dims=4,
+ update_state=False,
+ )
+ ann = datastore.build_ann_index(
+ reduction,
+ update_state=False,
+ )
+ neighbors_three = datastore.query_neighbors(
+ ann,
+ k=3,
+ update_state=False,
+ )
+ connectivity_three = datastore.build_connectivity_map(
+ neighbors_three,
+ update_state=False,
+ )
+ neighbors_four = datastore.query_neighbors(
+ ann,
+ k=4,
+ update_state=False,
+ )
+ connectivity_four = datastore.build_connectivity_map(
+ neighbors_four,
+ update_state=False,
+ )
+
+ assert (
+ datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ update_state=False,
+ )
+ == normalized
+ )
+ assert (
+ datastore.run_pca(
+ normalized,
+ dims=4,
+ update_state=False,
+ )
+ == reduction
+ )
+ assert (
+ datastore.build_ann_index(
+ reduction,
+ update_state=False,
+ )
+ == ann
+ )
+ assert neighbors_three != neighbors_four
+ assert connectivity_three != connectivity_four
+
+
+def test_cache_identity_distinguishes_parameters_from_execution_options(
+ analyzed_datastore_ephemeral,
+) -> None:
+ datastore = analyzed_datastore_ephemeral
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.normalized is not None
+ assert state.reduction is not None
+ assert state.ann_index is not None
+ assert state.neighbors is not None
+
+ reused_reduction = datastore.run_pca(
+ state.normalized,
+ dims=11,
+ local_cache="auto",
+ update_state=False,
+ )
+ reused_ann = datastore.build_ann_index(
+ state.reduction,
+ update_state=False,
+ )
+ reused_neighbors = datastore.query_neighbors(
+ state.ann_index,
+ k=11,
+ update_state=False,
+ )
+ changed_neighbors = datastore.query_neighbors(
+ state.ann_index,
+ k=3,
+ update_state=False,
+ )
+ invalidated_reduction = datastore.run_pca(
+ state.normalized,
+ dims=11,
+ local_cache=False,
+ update_state=False,
+ invalidate_cache=True,
+ )
+
+ assert reused_reduction == state.reduction
+ assert reused_ann == state.ann_index
+ assert reused_neighbors == state.neighbors
+ assert changed_neighbors != state.neighbors
+ assert invalidated_reduction != state.reduction
+ assert datastore.inspect_artifact(changed_neighbors).parameters == {
+ "k": 3,
+ "distance_metric": "l2",
+ }
+
+
+@pytest.mark.slow
+def test_seeded_graph_rebuild_is_deterministic(
+ analyzed_datastore_ephemeral,
+) -> None:
+ datastore = analyzed_datastore_ephemeral
+ state = datastore.get_assay_state("RNA")
+ assert state is not None and state.reduction is not None
+
+ def rebuild():
+ ann = datastore.build_ann_index(
+ state.reduction,
+ rand_state=4466,
+ update_state=False,
+ invalidate_cache=True,
+ )
+ neighbors = datastore.query_neighbors(
+ ann,
+ k=11,
+ update_state=False,
+ invalidate_cache=True,
+ )
+ connectivity = datastore.build_connectivity_map(
+ neighbors,
+ update_state=False,
+ invalidate_cache=True,
+ )
+ group = datastore.load_artifact(connectivity)
+ return connectivity, group["edges"][:], group["weights"][:]
+
+ first_ref, first_edges, first_weights = rebuild()
+ second_ref, second_edges, second_weights = rebuild()
+
+ assert first_ref != second_ref
+ np.testing.assert_array_equal(first_edges, second_edges)
+ np.testing.assert_allclose(first_weights, second_weights, rtol=0, atol=0)
+
+
+def test_update_state_false_keeps_current_selection(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+ current = datastore.run_pca(normalized, dims=4)
+ detached = datastore.run_pca(
+ normalized,
+ dims=6,
+ update_state=False,
+ )
+
+ assert detached != current
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.reduction == current
+
+ ann = datastore.build_ann_index(detached)
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.reduction == detached
+ assert state.ann_index == ann
+
+
+def test_reused_normalization_updates_requested_selection_aliases(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ source_mask = np.asarray(datastore.cells.fetch_all("I"), dtype=bool)
+ datastore.cells.insert(
+ "normalization_source",
+ source_mask,
+ overwrite=True,
+ )
+ datastore.cells.insert(
+ "normalization_alias",
+ source_mask,
+ overwrite=True,
+ )
+ feature_mask = np.asarray(
+ datastore.RNA.feats.fetch_all("I__graph_hvgs"),
+ dtype=bool,
+ )
+ datastore.RNA.feats.insert(
+ "normalization_source__graph_hvgs",
+ feature_mask,
+ overwrite=True,
+ )
+ datastore.RNA.feats.insert(
+ "normalization_alias__graph_hvgs",
+ feature_mask,
+ overwrite=True,
+ )
+ original = datastore.run_normalization(
+ from_assay="RNA",
+ cell_key="normalization_source",
+ feat_key="graph_hvgs",
+ update_state=False,
+ )
+ reused = datastore.run_normalization(
+ from_assay="RNA",
+ cell_key="normalization_alias",
+ feat_key="graph_hvgs",
+ )
+
+ assert reused == original
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.cell_key == "normalization_alias"
+ assert state.feat_key == "graph_hvgs"
+
+ reduction = datastore.run_pca(reused, dims=4)
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.reduction == reduction
+ assert state.cell_key == "normalization_alias"
+ assert state.feat_key == "graph_hvgs"
+
+
+def test_explicit_graph_preserves_prefixed_logical_feature_key(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ datastore.RNA.feats.insert(
+ "I__I__prefixed",
+ np.asarray(
+ datastore.RNA.feats.fetch_all("I__graph_hvgs"),
+ dtype=bool,
+ ),
+ overwrite=True,
+ )
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I__prefixed",
+ update_state=False,
+ )
+ reduction = datastore.run_pca(normalized, dims=3, update_state=False)
+ ann = datastore.build_ann_index(reduction, update_state=False)
+ neighbors = datastore.query_neighbors(ann, k=3, update_state=False)
+ connectivity = datastore.build_connectivity_map(
+ neighbors,
+ update_state=False,
+ )
+ stored = stored_assay_graph_from_ref(datastore.zw, connectivity)
+
+ assert stored.cell_key == "I"
+ assert stored.feat_key == "I__prefixed"
+
+
+def test_historical_neighbors_use_normalized_execution_keys(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ mask = np.asarray(datastore.cells.fetch_all("I"), dtype=bool)
+ datastore.cells.insert("selection_a", mask, overwrite=True)
+ datastore.cells.insert("selection_b", mask, overwrite=True)
+ datastore._ensure_cell_selection("selection_a")
+ feature_mask = np.asarray(
+ datastore.RNA.feats.fetch_all("I__graph_hvgs"),
+ dtype=bool,
+ )
+ datastore.RNA.feats.insert(
+ "selection_b__graph_hvgs",
+ feature_mask,
+ overwrite=True,
+ )
+
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ cell_key="selection_b",
+ feat_key="graph_hvgs",
+ )
+ reduction = datastore.run_pca(normalized, dims=4)
+ ann = datastore.build_ann_index(reduction)
+ neighbors = datastore.query_neighbors(ann, k=3)
+ datastore.run_normalization(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="graph_hvgs",
+ )
+
+ stale = mask.copy()
+ selected = np.flatnonzero(stale)
+ excluded = np.flatnonzero(~stale)
+ assert len(selected) > 0 and len(excluded) > 0
+ stale[selected[0]] = False
+ stale[excluded[0]] = True
+ datastore.cells.insert("selection_a", stale, overwrite=True, force=True)
+
+ assert datastore._keys_from_knn_path(
+ "RNA",
+ artifact_path(neighbors),
+ ) == ("selection_b", "graph_hvgs")
+
+
+def test_reduction_and_harmony_reject_changed_normalized_selection(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+ reduction = datastore.run_pca(normalized, dims=4)
+ datastore.cells.insert(
+ "graph_batch",
+ np.where(np.arange(datastore.cells.N) % 2, "a", "b"),
+ overwrite=True,
+ )
+ mask = np.asarray(datastore.cells.fetch_all("I"), dtype=bool)
+ selected = np.flatnonzero(mask)
+ excluded = np.flatnonzero(~mask)
+ assert len(selected) > 0 and len(excluded) > 0
+ mask[selected[0]] = False
+ mask[excluded[0]] = True
+ datastore.cells.insert("I", mask, overwrite=True, force=True)
+
+ with pytest.raises(ValueError, match="no longer matches"):
+ datastore.run_pca(
+ normalized,
+ dims=5,
+ invalidate_cache=True,
+ )
+ with pytest.raises(ValueError, match="no longer matches"):
+ datastore.run_harmony(
+ ["graph_batch"],
+ reduction,
+ harmony_params={"nclust": 5},
+ invalidate_cache=True,
+ )
+
+
+def test_datastore_inspects_and_loads_artifact_read_only(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ ref = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+
+ status = datastore.inspect_artifact(ref)
+ group = datastore.load_artifact(ref)
+
+ assert status.complete
+ assert status.operation == "run_normalization"
+ assert status.parameters is not None
+ assert "data" in group
+ with pytest.raises(ValueError):
+ group.attrs["invalid"] = True
+
+
+def test_graph_harmony_becomes_default_ann_coordinates(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ batches = np.where(np.arange(datastore.cells.N) % 2, "a", "b")
+ datastore.cells.insert("graph_batch", batches, overwrite=True)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+ pca = datastore.run_pca(normalized, dims=5)
+ pca_scores = datastore.load_artifact(pca)["data"][:]
+ active_batches = pd.DataFrame(
+ {
+ "graph_batch": datastore.cells.fetch(
+ "graph_batch",
+ key="I",
+ ).astype(object)
+ }
+ )
+ expected_correction = fit_harmony(
+ np.asarray(pca_scores.T, dtype=np.float64),
+ active_batches,
+ nclust=5,
+ )
+
+ def fail_legacy_projection(*_args, **_kwargs):
+ raise AssertionError("persisted coordinates should be used")
+
+ monkeypatch.setattr(
+ datastore,
+ "_load_reduction_stream",
+ fail_legacy_projection,
+ )
+
+ corrected = datastore.run_harmony(
+ ["graph_batch"],
+ pca,
+ harmony_params={"nclust": 5},
+ )
+ ann = datastore.build_ann_index(batch_size=100)
+ datastore.query_neighbors(ann, k=3, update_state=False)
+ datastore.build_embedding_initialization(
+ pca,
+ n_centroids=5,
+ update_state=False,
+ )
+
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.batch_correction == corrected
+ assert state.ann_index == ann
+ ann_inputs = datastore.inspect_artifact(ann).inputs
+ assert ann_inputs is not None
+ assert ann_inputs["coordinates"] == corrected.to_dict()
+ correction_group = datastore.load_artifact(corrected)
+ assert correction_group["data"].dtype == np.dtype(np.float32)
+ np.testing.assert_allclose(
+ correction_group["data"][:],
+ expected_correction.corrected.T,
+ rtol=2e-5,
+ atol=2e-6,
+ )
+ assert "assignments" not in correction_group
+ assert {
+ "cluster_mass",
+ "raw_centroids",
+ "corrected_centroids",
+ } <= set(correction_group.array_keys())
+
+
+def test_lsi_and_custom_reduction_have_distinct_public_methods(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+ lsi = datastore.run_lsi(
+ normalized,
+ dims=3,
+ n_iter=1,
+ n_oversamples=2,
+ )
+ materialized_lsi = datastore.run_lsi(
+ normalized,
+ dims=3,
+ solver="materialized",
+ n_iter=1,
+ n_oversamples=2,
+ update_state=False,
+ )
+ n_features = datastore.load_artifact(normalized)["data"].shape[1]
+ loadings = np.eye(n_features, 2, dtype=np.float64)
+ custom = datastore.run_custom_reduction(
+ loadings,
+ normalized,
+ update_state=False,
+ )
+
+ lsi_status = datastore.inspect_artifact(lsi)
+ assert lsi_status.operation == "run_lsi"
+ assert lsi_status.parameters["solver"] == "streaming"
+ assert lsi_status.parameters["n_iter"] == 1
+ assert lsi_status.parameters["n_oversamples"] == 2
+ assert datastore.inspect_artifact(materialized_lsi).parameters["solver"] == (
+ "materialized"
+ )
+ assert materialized_lsi != lsi
+ assert datastore.inspect_artifact(custom).operation == "run_custom_reduction"
+ ann = datastore.build_ann_index(custom, update_state=False)
+ neighbors = datastore.query_neighbors(ann, k=3, update_state=False)
+ connectivity = datastore.build_connectivity_map(
+ neighbors,
+ update_state=False,
+ )
+ stored = stored_assay_graph_from_ref(datastore.zw, connectivity)
+ assert stored.reduction_method == "custom"
+ assert stored.dims == 2
+ assert stored.pca_cell_key is None
+ assert stored.feat_scaling is False
+
+
+def test_graph_chain_matches_released_knn_golden(
+ analyzed_datastore_ephemeral,
+) -> None:
+ datastore = analyzed_datastore_ephemeral
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="hvgs",
+ update_state=False,
+ invalidate_cache=True,
+ )
+ reduction = datastore.run_pca(
+ normalized,
+ dims=11,
+ update_state=False,
+ invalidate_cache=True,
+ )
+ ann = datastore.build_ann_index(
+ reduction,
+ update_state=False,
+ invalidate_cache=True,
+ )
+ neighbors = datastore.query_neighbors(
+ ann,
+ coordinates=reduction,
+ k=11,
+ update_state=False,
+ invalidate_cache=True,
+ )
+ group = datastore.load_artifact(neighbors)
+
+ np.testing.assert_array_equal(
+ group["indices"][:],
+ np.load(full_path("knn_indices.npy")),
+ )
+ np.testing.assert_allclose(
+ group["distances"][:],
+ np.sqrt(np.load(full_path("knn_distances.npy"))),
+ rtol=0,
+ atol=1e-3,
+ )
+
+
+def test_connectivity_rebuild_requires_named_distance_metric(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+ reduction = datastore.run_pca(normalized, dims=4)
+ ann = datastore.build_ann_index(reduction)
+ neighbors = datastore.query_neighbors(ann, k=3)
+
+ neighbor_group = datastore.zw[artifact_path(neighbors)]
+ provenance = dict(neighbor_group.attrs["provenance"])
+ parameters = dict(provenance["parameters"])
+ assert parameters["distance_metric"] == "l2"
+
+ provenance["parameters"] = {"k": 3}
+ neighbor_group.attrs["provenance"] = provenance
+ with pytest.raises(ValueError, match="does not name the metric"):
+ datastore.build_connectivity_map(
+ neighbors,
+ update_state=False,
+ invalidate_cache=True,
+ )
+
+ provenance["parameters"] = {**parameters, "distance_metric": "cosine"}
+ neighbor_group.attrs["provenance"] = provenance
+ with pytest.raises(ValueError, match="does not match its ANN index input"):
+ datastore.build_connectivity_map(
+ neighbors,
+ update_state=False,
+ invalidate_cache=True,
+ )
+
+ provenance["parameters"] = parameters
+ neighbor_group.attrs["provenance"] = provenance
+ rebuilt = datastore.build_connectivity_map(
+ neighbors,
+ update_state=False,
+ invalidate_cache=True,
+ )
+ assert datastore.inspect_artifact(rebuilt).complete
+
+
+def test_corrupt_ann_bytes_are_not_reused(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _prepare_graph_features(datastore)
+ normalized = datastore.run_normalization(
+ from_assay="RNA",
+ feat_key="graph_hvgs",
+ )
+ reduction = datastore.run_pca(normalized, dims=3)
+ current = datastore.build_ann_index(reduction)
+ for attribute, invalid_value in (
+ ("metric", "cosine"),
+ ("dimensions", 2),
+ ("element_count", 1),
+ ):
+ ann_group = datastore.zw[artifact_path(current)]
+ ann_group["ann_idx_bytes"].attrs[attribute] = invalid_value
+ repaired = datastore.build_ann_index(reduction)
+ assert repaired != current
+ current = repaired
+
+ ann_group = datastore.zw[artifact_path(current)]
+ ann_group["ann_idx_bytes"][:] = 0
+ repaired = datastore.build_ann_index(reduction)
+ assert repaired != current
+ assert datastore.build_ann_index(reduction) == repaired
+ assert datastore.inspect_artifact(repaired).complete
+
+ legacy_group = datastore.zw[artifact_path(repaired)]["ann_idx_bytes"]
+ for attribute in (
+ "ann_index_format_version",
+ "metric",
+ "dimensions",
+ "element_count",
+ "payload_sha256",
+ ):
+ del legacy_group.attrs[attribute]
+ assert datastore.build_ann_index(reduction) == repaired
diff --git a/tests/test_graph_coverage.py b/tests/test_graph_coverage.py
new file mode 100644
index 00000000..6195ecc5
--- /dev/null
+++ b/tests/test_graph_coverage.py
@@ -0,0 +1,898 @@
+import shutil
+import sys
+from pathlib import Path
+from unittest.mock import Mock
+
+import numpy as np
+import pytest
+import zarr
+from scipy.sparse import coo_matrix, csr_matrix
+from zarr.storage import MemoryStore
+
+from scarf.datastore.datastore import DataStore
+from scarf.datastore.graph_datastore import GraphDataStore
+from scarf.storage.artifacts import ArtifactRef, artifact_path, list_artifacts
+
+
+class _MemoryGraphStore(GraphDataStore):
+ @property
+ def assay_names(self) -> list[str]:
+ return self._assay_names
+
+
+@pytest.fixture
+def isolated_toy_datastore(toy_crdir_writer: str, tmp_path: Path) -> DataStore:
+ zarr_path = tmp_path / "toy.zarr"
+ shutil.copytree(toy_crdir_writer, zarr_path)
+ return DataStore(
+ str(zarr_path),
+ default_assay="RNA",
+ min_features_per_cell=0,
+ min_cells_per_feature=0,
+ nthreads=1,
+ )
+
+
+def _memory_graph_store(
+ assay_names: list[str] | None = None,
+) -> _MemoryGraphStore:
+ store = _MemoryGraphStore.__new__(_MemoryGraphStore)
+ store.z = zarr.open_group(store=MemoryStore(), mode="w")
+ store.workspace = None
+ store.zarr_mode = "r+"
+ store._defaultAssay = "RNA"
+ store._assay_names = assay_names or []
+ store._integratedGraphsLoc = "integratedGraphs"
+ store._cachedMagicOperator = None
+ store._cachedMagicOperatorLoc = None
+ store.nthreads = 1
+ store.storageProfile = "fast_local"
+ return store
+
+
+def _add_test_graph(store: _MemoryGraphStore, label: str = "graph") -> str:
+ graph_loc = f"integratedGraphs/{label}"
+ graph_group = store.zw.create_group(graph_loc)
+ graph_group.attrs["n_cells"] = 3
+ graph_group.attrs["n_neighbors"] = 2
+ graph_group.create_array(
+ "edges",
+ data=np.array(
+ [
+ [0, 1],
+ [0, 2],
+ [1, 0],
+ [1, 2],
+ [2, 0],
+ [2, 1],
+ ],
+ dtype=np.uint64,
+ ),
+ )
+ graph_group.create_array(
+ "weights",
+ data=np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6]),
+ )
+ return graph_loc
+
+
+@pytest.mark.parametrize(
+ ("symmetric", "upper_only", "use_k", "expected"),
+ [
+ (
+ False,
+ False,
+ None,
+ np.array(
+ [
+ [0.0, 0.1, 0.2],
+ [0.3, 0.0, 0.4],
+ [0.5, 0.6, 0.0],
+ ]
+ ),
+ ),
+ (
+ False,
+ True,
+ 1,
+ np.array(
+ [
+ [0.0, 0.1, 0.0],
+ [0.3, 0.0, 0.0],
+ [0.5, 0.0, 0.0],
+ ]
+ ),
+ ),
+ (
+ False,
+ False,
+ 0,
+ np.array(
+ [
+ [0.0, 0.1, 0.0],
+ [0.3, 0.0, 0.0],
+ [0.5, 0.0, 0.0],
+ ]
+ ),
+ ),
+ (
+ False,
+ False,
+ 99,
+ np.array(
+ [
+ [0.0, 0.1, 0.2],
+ [0.3, 0.0, 0.4],
+ [0.5, 0.6, 0.0],
+ ]
+ ),
+ ),
+ (
+ True,
+ False,
+ None,
+ np.array(
+ [
+ [0.0, 0.37, 0.6],
+ [0.37, 0.0, 0.76],
+ [0.6, 0.76, 0.0],
+ ]
+ ),
+ ),
+ (
+ True,
+ True,
+ None,
+ np.array(
+ [
+ [0.0, 0.37, 0.6],
+ [0.0, 0.0, 0.76],
+ [0.0, 0.0, 0.0],
+ ]
+ ),
+ ),
+ ],
+)
+def test_load_graph_option_matrix(
+ symmetric: bool,
+ upper_only: bool,
+ use_k: int | None,
+ expected: np.ndarray,
+) -> None:
+ store = _memory_graph_store()
+ graph_loc = _add_test_graph(store)
+
+ graph = store.load_graph(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I",
+ symmetric=symmetric,
+ upper_only=upper_only,
+ use_k=use_k,
+ graph_loc=graph_loc,
+ )
+
+ np.testing.assert_allclose(graph.toarray(), expected)
+
+
+def test_load_graph_latest_location_formats_and_errors(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ store = _memory_graph_store()
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.validate_legacy_graph_selection",
+ lambda *_args, **_kwargs: None,
+ )
+ graph_loc = _add_test_graph(store)
+ normed_loc = "RNA/normed__I__I"
+ reduction_loc = f"{normed_loc}/reduction__pca__2__I"
+ ann_loc = f"{reduction_loc}/ann__l2__50__50__48__1"
+ knn_loc = f"{ann_loc}/knn__2"
+
+ normed_group = store.zw.create_group(normed_loc)
+ reduction_group = store.zw.create_group(reduction_loc)
+ ann_group = store.zw.create_group(ann_loc)
+ knn_group = store.zw.create_group(knn_loc)
+ normed_group.attrs["latest_reduction"] = reduction_loc
+ reduction_group.attrs["latest_ann"] = ann_loc
+ ann_group.attrs["latest_knn"] = knn_loc
+ knn_group.attrs["latest_graph"] = graph_loc
+
+ store._get_latest_cell_key = Mock(return_value="I")
+ store._get_latest_feat_key = Mock(return_value="I")
+ graph = store.load_graph()
+ np.testing.assert_allclose(
+ graph.toarray(),
+ np.array(
+ [
+ [0.0, 0.1, 0.2],
+ [0.3, 0.0, 0.4],
+ [0.5, 0.6, 0.0],
+ ]
+ ),
+ )
+ store._get_latest_cell_key.assert_called_once_with("RNA")
+ store._get_latest_feat_key.assert_called_once_with("RNA")
+
+ n_cells, graph_coo = store._store_to_sparse(graph_loc, sparse_format="coo", use_k=1)
+ assert n_cells == 3
+ assert graph_coo.getformat() == "coo"
+ assert graph_coo.nnz == 3
+
+ with pytest.raises(ValueError, match="not found in zarr location"):
+ store.load_graph(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I",
+ graph_loc="integratedGraphs/missing",
+ )
+
+ empty_store = _memory_graph_store()
+ with pytest.raises(KeyError):
+ empty_store.load_graph(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I",
+ )
+
+
+def test_latest_knn_loc_resolves_the_encoded_graph_chain() -> None:
+ store = _memory_graph_store(["RNA"])
+ assay_group = store.zw.create_group("RNA")
+ assay_group.attrs["latest_cell_key"] = "I"
+ assay_group.attrs["latest_feat_key"] = "I"
+ normed_loc = "RNA/normed__I__I"
+ reduction_loc = f"{normed_loc}/reduction__pca__2__I"
+ ann_loc = f"{reduction_loc}/ann__l2__50__50__48__1"
+ knn_loc = f"{ann_loc}/knn__2"
+ normed_group = store.zw.create_group(normed_loc)
+ reduction_group = store.zw.create_group(reduction_loc)
+ ann_group = store.zw.create_group(ann_loc)
+ store.zw.create_group(knn_loc)
+ normed_group.attrs["latest_reduction"] = reduction_loc
+ reduction_group.attrs["latest_ann"] = ann_loc
+ ann_group.attrs["latest_knn"] = knn_loc
+ reduction_group.create_array("reduction", data=np.eye(2))
+ store._load_default_assay = Mock(return_value="RNA")
+
+ assert store._get_latest_knn_loc() == knn_loc
+ store._load_default_assay.assert_called_once_with()
+
+ with pytest.raises(ValueError, match="Assay missing does not exist"):
+ store._get_latest_knn_loc("missing")
+
+ del reduction_group["reduction"]
+ with pytest.raises(ValueError, match="PCA Reduction not found"):
+ store._get_latest_knn_loc("RNA")
+
+
+def test_corrupt_zarr_ann_does_not_fall_back_to_legacy(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ store = _memory_graph_store()
+ ann_loc = "RNA/normed/reduction/ann__l2__50__50__48__1"
+ ann_group = store.zw.create_group(ann_loc)
+ ann_group.create_array("ann_idx_bytes", data=np.array([1, 2, 3], dtype=np.uint8))
+ legacy_path = tmp_path / "ann_idx"
+ legacy_path.write_bytes(b"legacy")
+ load_legacy = Mock()
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.load_ann_index",
+ Mock(side_effect=RuntimeError("corrupt Zarr ANN bytes")),
+ )
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.load_ann_index_from_path",
+ load_legacy,
+ )
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.legacy_ann_index_path",
+ lambda *_: str(legacy_path),
+ )
+
+ with pytest.raises(RuntimeError, match="corrupt Zarr ANN bytes"):
+ store._resolve_ann_index(ann_loc, "l2", 3)
+ load_legacy.assert_not_called()
+
+
+def test_legacy_ann_load_does_not_create_zarr_bytes(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ import hnswlib
+
+ store = _memory_graph_store()
+ ann_loc = "RNA/normed/reduction/ann__l2__50__50__48__1"
+ ann_group = store.zw.create_group(ann_loc)
+ before_attrs = dict(ann_group.attrs)
+ data = np.random.default_rng(9).random((20, 3), dtype=np.float32)
+ source = hnswlib.Index(space="l2", dim=3)
+ source.init_index(max_elements=len(data), ef_construction=50, M=16)
+ source.add_items(data)
+ legacy_path = tmp_path / "ann_idx"
+ source.save_index(str(legacy_path))
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.legacy_ann_index_path",
+ lambda *_: str(legacy_path),
+ )
+
+ loaded = store._resolve_ann_index(ann_loc, "l2", 3)
+ expected_indices, expected_distances = source.knn_query(data[:3], k=4)
+ actual_indices, actual_distances = loaded.knn_query(data[:3], k=4)
+ np.testing.assert_array_equal(actual_indices, expected_indices)
+ np.testing.assert_allclose(actual_distances, expected_distances)
+ assert "ann_idx_bytes" not in ann_group
+ assert dict(ann_group.attrs) == before_attrs
+
+
+def test_partial_normalization_statistics_cache_paths(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ store = _memory_graph_store()
+ data = Mock()
+ data.mean.return_value = np.array([2.0, 4.0])
+ data.std.return_value = np.array([1.5, 2.5])
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.show_dask_progress",
+ lambda values, *_: values,
+ )
+
+ missing_mu = store.zw.create_group("missingMu")
+ missing_mu.create_array("sigma", data=np.array([3.0, 5.0]))
+ mu, sigma = store._load_or_compute_norm_stats("missingMu", data, "pca")
+ np.testing.assert_allclose(mu, [2.0, 4.0])
+ np.testing.assert_allclose(sigma, [3.0, 5.0])
+ np.testing.assert_allclose(missing_mu["mu"][:], [2.0, 4.0])
+
+ missing_sigma = store.zw.create_group("missingSigma")
+ missing_sigma.create_array("mu", data=np.array([6.0, 8.0]))
+ mu, sigma = store._load_or_compute_norm_stats("missingSigma", data, "pca")
+ np.testing.assert_allclose(mu, [6.0, 8.0])
+ np.testing.assert_allclose(sigma, [1.5, 2.5])
+ np.testing.assert_allclose(missing_sigma["sigma"][:], [1.5, 2.5])
+
+ store.zarr_mode = "r"
+ read_only_mu = store.zw.create_group("readOnlyMu")
+ read_only_mu.create_array("sigma", data=np.array([3.0, 5.0]))
+ mu, sigma = store._load_or_compute_norm_stats("readOnlyMu", data, "pca")
+ np.testing.assert_allclose(mu, [2.0, 4.0])
+ np.testing.assert_allclose(sigma, [3.0, 5.0])
+ assert "mu" not in read_only_mu
+
+ read_only_sigma = store.zw.create_group("readOnlySigma")
+ read_only_sigma.create_array("mu", data=np.array([6.0, 8.0]))
+ mu, sigma = store._load_or_compute_norm_stats("readOnlySigma", data, "pca")
+ np.testing.assert_allclose(mu, [6.0, 8.0])
+ np.testing.assert_allclose(sigma, [1.5, 2.5])
+ assert "sigma" not in read_only_sigma
+
+
+def test_remote_cache_plan_auto_and_invalid(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ store = _memory_graph_store()
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.is_remote_datastore",
+ lambda *_: True,
+ )
+
+ enabled, cache_path, remove = store._resolve_local_cache_plan(
+ "s3://bucket/store", store.z, "auto"
+ )
+ assert enabled is True
+ assert cache_path is not None
+ assert Path(cache_path).is_dir()
+ assert remove is True
+ shutil.rmtree(cache_path)
+
+ with pytest.raises(TypeError, match="local_cache must be"):
+ store._resolve_local_cache_plan("s3://bucket/store", store.z, object())
+
+
+def test_get_imputed_creates_loads_and_reuses_operator() -> None:
+ store = _memory_graph_store()
+ graph_loc = _add_test_graph(store)
+ values = np.array([1.0, 2.0, 4.0])
+ graph = csr_matrix(
+ np.array(
+ [
+ [0.0, 1.0, 1.0],
+ [1.0, 0.0, 1.0],
+ [1.0, 1.0, 0.0],
+ ]
+ )
+ )
+ store._get_latest_keys = Mock(return_value=("RNA", "I", "I"))
+ store.get_cell_vals = Mock(return_value=values)
+ store.get_latest_graph_loc = Mock(return_value=graph_loc)
+ store.load_graph = Mock(return_value=graph)
+
+ with pytest.raises(ValueError, match="name for the feature"):
+ store.get_imputed(feature_name=None)
+ store.get_cell_vals.assert_not_called()
+
+ operator = store.get_diffusion_operator(t=1)
+ assert isinstance(operator, coo_matrix)
+ store.get_cell_vals.assert_not_called()
+
+ first = store.get_imputed(feature_name="gene", t=1)
+ np.testing.assert_allclose(first, np.array([3.0, 2.5, 1.5]))
+ diffusion_refs = list_artifacts(
+ store.zw,
+ scope="assay",
+ assay="RNA",
+ kind="diffusion_operator",
+ )
+ assert len(diffusion_refs) == 1
+ assert artifact_path(diffusion_refs[0]) in store.zw
+ assert store._cachedMagicOperatorLoc == diffusion_refs[0].artifact_id
+ assert store._cachedMagicOperator is not None
+ store.load_graph.assert_called_once_with(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I",
+ symmetric=True,
+ upper_only=False,
+ )
+
+ cached_operator = store._cachedMagicOperator
+ second = store.get_imputed(feature_name="gene", t=1)
+ np.testing.assert_allclose(second, first)
+ assert store._cachedMagicOperator is cached_operator
+ assert store.load_graph.call_count == 1
+
+ store._cachedMagicOperator = None
+ store._cachedMagicOperatorLoc = None
+ loaded = store.get_imputed(feature_name="gene", t=1, cache_operator=True)
+ np.testing.assert_allclose(loaded, first)
+ assert store._cachedMagicOperator is not None
+ assert store._cachedMagicOperatorLoc == diffusion_refs[0].artifact_id
+
+ store._cachedMagicOperator = None
+ store._cachedMagicOperatorLoc = None
+ uncached = store.get_imputed(feature_name="gene", t=1, cache_operator=False)
+ np.testing.assert_allclose(uncached, first)
+ assert store._cachedMagicOperator is None
+ assert store._cachedMagicOperatorLoc is None
+
+ squared = store.get_imputed(feature_name="gene", t=2, cache_operator=False)
+ np.testing.assert_allclose(squared, np.array([2.0, 2.25, 2.75]))
+ assert (
+ len(
+ list_artifacts(
+ store.zw,
+ scope="assay",
+ assay="RNA",
+ kind="diffusion_operator",
+ )
+ )
+ == 2
+ )
+ assert store.load_graph.call_count == 2
+ assert store._cachedMagicOperator is None
+ assert store._cachedMagicOperatorLoc is None
+
+
+def test_get_diffusion_operator_discards_stale_in_memory_cache() -> None:
+ store = _memory_graph_store()
+ graph_loc = _add_test_graph(store)
+ first_graph = csr_matrix(
+ np.array(
+ [
+ [0.0, 1.0, 1.0],
+ [1.0, 0.0, 1.0],
+ [1.0, 1.0, 0.0],
+ ]
+ )
+ )
+ replacement_graph = csr_matrix(np.eye(3))
+ store._get_latest_keys = Mock(return_value=("RNA", "I", "I"))
+ store.get_latest_graph_loc = Mock(return_value=graph_loc)
+ store.load_graph = Mock(side_effect=[first_graph, replacement_graph])
+
+ first = store.get_diffusion_operator(t=1)
+ first_ref = list_artifacts(
+ store.zw,
+ scope="assay",
+ assay="RNA",
+ kind="diffusion_operator",
+ )[0]
+ del store.zw[artifact_path(first_ref)]
+ replacement = store.get_diffusion_operator(t=1)
+
+ assert store.load_graph.call_count == 2
+ assert replacement is not first
+ np.testing.assert_allclose(replacement.toarray(), np.eye(3))
+
+
+def test_read_only_diffusion_operator_reuses_memory_cache() -> None:
+ store = _memory_graph_store()
+ graph_loc = _add_test_graph(store)
+ graph = csr_matrix(
+ np.array(
+ [
+ [0.0, 1.0, 1.0],
+ [1.0, 0.0, 1.0],
+ [1.0, 1.0, 0.0],
+ ]
+ )
+ )
+ store.zarr_mode = "r"
+ store._get_latest_keys = Mock(return_value=("RNA", "I", "I"))
+ store.get_latest_graph_loc = Mock(return_value=graph_loc)
+ store.load_graph = Mock(return_value=graph)
+
+ first = store.get_diffusion_operator(t=1, cache_operator=True)
+ second = store.get_diffusion_operator(t=1, cache_operator=True)
+
+ assert second is first
+ assert store.load_graph.call_count == 1
+ assert store._cachedMagicOperatorLoc is not None
+ assert store._cachedMagicOperatorLoc.startswith("read_only:")
+
+ invalidated = store.get_diffusion_operator(
+ t=1,
+ cache_operator=True,
+ invalidate_cache=True,
+ )
+ assert invalidated is not first
+ assert store.load_graph.call_count == 2
+
+ store.get_diffusion_operator(t=1, cache_operator=False)
+ assert store.load_graph.call_count == 3
+ assert store._cachedMagicOperator is None
+ assert store._cachedMagicOperatorLoc is None
+
+
+def test_filter_cells_open_bounds_reset_and_boundaries(
+ isolated_toy_datastore: DataStore,
+) -> None:
+ store = isolated_toy_datastore
+ attr = "RNA_nCounts"
+ values = store.cells.fetch_all(attr)
+ lower = float(values.min())
+ upper = float(values.max())
+
+ store.cells.reset_key("I")
+ store.filter_cells(
+ attrs=[attr],
+ lows=[lower],
+ highs=[None],
+ reset_previous=True,
+ )
+ expected = values > lower
+ np.testing.assert_array_equal(store.cells.fetch_all("I"), expected)
+
+ store.filter_cells(
+ attrs=[attr],
+ lows=[None],
+ highs=[upper],
+ )
+ expected &= values < upper
+ np.testing.assert_array_equal(store.cells.fetch_all("I"), expected)
+
+ store.filter_cells(
+ attrs=[attr, "missing"],
+ lows=[None, None],
+ highs=[None, None],
+ reset_previous=True,
+ )
+ assert store.cells.fetch_all("I").all()
+
+ store.filter_cells(
+ attrs=[attr],
+ lows=[lower],
+ highs=[upper],
+ reset_previous=True,
+ keep_bounds=True,
+ )
+ assert store.cells.fetch_all("I").all()
+
+
+def test_run_marker_search_skip_save_and_errors(
+ isolated_toy_datastore: DataStore,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ store = isolated_toy_datastore
+ monkeypatch.setattr(store, "_get_latest_feat_key", lambda _: "I")
+ markers = {"cluster": object()}
+ finder = Mock(return_value=markers)
+ monkeypatch.setattr("scarf.features.markers.find_markers_by_rank", finder)
+
+ with pytest.raises(ValueError, match="group_key"):
+ store.run_marker_search(group_key=None)
+
+ result = store.run_marker_search(
+ group_key="ids",
+ skip_save=True,
+ )
+ assert result is markers
+ first_call = finder.call_args.kwargs
+ assert first_call["assay"] is store.RNA
+ assert first_call["group_key"] == "ids"
+ assert first_call["cell_key"] == "I"
+ assert first_call["feat_key"] == "I"
+ assert first_call["batch_size"] is None
+ assert first_call["n_threads"] == store.nthreads
+ assert "markers" not in store.zw["RNA"]
+
+ finder.reset_mock()
+ result = store.run_marker_search(
+ group_key="ids",
+ cell_key="I",
+ feat_key="I",
+ gene_batch_size=2,
+ n_threads=3,
+ skip_save=True,
+ log_transform=False,
+ )
+ assert result is markers
+ second_call = finder.call_args.kwargs
+ assert second_call["batch_size"] == 2
+ assert second_call["n_threads"] == 3
+ assert second_call["log_transform"] is False
+
+ finder.side_effect = RuntimeError("marker failure")
+ with pytest.raises(RuntimeError, match="marker failure"):
+ store.run_marker_search(group_key="ids", skip_save=True)
+
+
+def test_run_tsne_orchestration_and_error_paths(
+ isolated_toy_datastore: DataStore,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ store = isolated_toy_datastore
+ graph = csr_matrix(
+ np.array(
+ [
+ [0.0, 1.0, 1.0],
+ [1.0, 0.0, 1.0],
+ [1.0, 1.0, 0.0],
+ ]
+ )
+ )
+ initial = np.array(
+ [
+ [0.0, 0.0],
+ [0.5, 0.5],
+ [1.0, 1.0],
+ ]
+ )
+ embedding = np.array(
+ [
+ [1.0, 2.0, 3.0],
+ [4.0, 5.0, 6.0],
+ ]
+ )
+ latest_keys = Mock(return_value=("RNA", "I", "I"))
+ load_graph = Mock(return_value=graph)
+ get_initial = Mock(return_value=initial)
+ runner = Mock(return_value=embedding)
+ graph_loc = "RNA/coverage_tsne_graph"
+ graph_group = store.zw.create_group(graph_loc)
+ graph_coo = graph.tocoo()
+ graph_group.create_array(
+ "edges",
+ data=np.column_stack((graph_coo.row, graph_coo.col)).astype(np.uint64),
+ )
+ graph_group.create_array("weights", data=graph_coo.data)
+ monkeypatch.setattr(store, "_get_latest_keys", latest_keys)
+ monkeypatch.setattr(store, "load_graph", load_graph)
+ monkeypatch.setattr(store, "get_latest_graph_loc", Mock(return_value=graph_loc))
+ monkeypatch.setattr(store, "_get_ini_embed", get_initial)
+ monkeypatch.setattr(
+ "scarf.datastore._operations.embeddings.validate_legacy_graph_selection",
+ Mock(),
+ )
+ monkeypatch.setattr("scarf.embeddings.sgtsne.run_sgtsne", runner)
+
+ tsne_ref = store.run_tsne(
+ symmetric_graph=True,
+ graph_upper_only=True,
+ parallel=True,
+ nthreads=None,
+ max_iter=20,
+ label="coverageTsne",
+ )
+ get_initial.assert_called_once_with("RNA", "I", "I", 2)
+ first_call = runner.call_args
+ assert first_call.args[0] is graph
+ np.testing.assert_array_equal(first_call.args[1], initial)
+ assert first_call.kwargs["parallel"] is True
+ assert first_call.kwargs["nthreads"] == store.nthreads
+ assert first_call.kwargs["max_iter"] == 20
+ np.testing.assert_allclose(store.cells.fetch("RNA_coverageTsne1"), embedding[0])
+ np.testing.assert_allclose(store.cells.fetch("RNA_coverageTsne2"), embedding[1])
+ first_ref = ArtifactRef.from_dict(
+ store.zw["cellData/RNA_coverageTsne1"].attrs["source_artifact"]
+ )
+ assert first_ref == tsne_ref
+ assert first_ref.kind == "embedding"
+ assert (
+ ArtifactRef.from_dict(
+ store.zw["cellData/RNA_coverageTsne2"].attrs["source_artifact"]
+ )
+ == first_ref
+ )
+ np.testing.assert_allclose(
+ store.load_artifact(first_ref)["values"][:],
+ embedding.T,
+ )
+
+ store.run_tsne(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I",
+ ini_embed=initial,
+ parallel=False,
+ label="serialTsne",
+ invalidate_cache=True,
+ )
+ assert runner.call_args.kwargs["nthreads"] == 1
+ assert runner.call_args.kwargs["parallel"] is False
+
+ with pytest.raises(ValueError, match="required shape"):
+ store.run_tsne(
+ ini_embed=np.zeros((2, 2)),
+ tsne_dims=2,
+ )
+
+ runner.side_effect = FileNotFoundError("sgtsne missing")
+ with pytest.raises(RuntimeError, match="SG-tSNE failed"):
+ store.run_tsne(
+ ini_embed=initial,
+ parallel=True,
+ nthreads=2,
+ label="missingTsne",
+ )
+ assert runner.call_args.kwargs["nthreads"] == 2
+
+ runner_calls = runner.call_count
+ monkeypatch.setattr(sys, "platform", "win32")
+ assert (
+ store.run_tsne(
+ symmetric_graph=True,
+ graph_upper_only=True,
+ parallel=True,
+ nthreads=None,
+ max_iter=20,
+ label="cachedTsne",
+ )
+ == first_ref
+ )
+ assert runner.call_count == runner_calls
+ np.testing.assert_allclose(
+ store.cells.fetch("RNA_cachedTsne1"),
+ embedding[0],
+ )
+
+
+def test_integrate_assays_snn_writes_and_overwrites_graph(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ store = _memory_graph_store(["RNA", "ADT"])
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.validate_cell_selection_artifact",
+ lambda *_args, **_kwargs: None,
+ )
+ selection_ref = ArtifactRef(
+ scope="datastore",
+ kind="cell_selection",
+ artifact_id="a" * 64,
+ )
+ store._ensure_cell_selection = Mock(return_value=selection_ref)
+ store._graph_cell_selection = Mock(return_value=selection_ref)
+ graphs = {
+ "RNA": csr_matrix(
+ np.array(
+ [
+ [0.0, 1.0, 2.0],
+ [3.0, 0.0, 4.0],
+ [5.0, 6.0, 0.0],
+ ]
+ )
+ ),
+ "ADT": csr_matrix(
+ np.array(
+ [
+ [0.0, 7.0, 8.0],
+ [9.0, 0.0, 10.0],
+ [11.0, 12.0, 0.0],
+ ]
+ )
+ ),
+ }
+ load_graph = Mock(side_effect=lambda **kwargs: graphs[kwargs["from_assay"]])
+ store.load_graph = load_graph
+ source_paths = {
+ assay: _add_test_graph(store, f"{assay}_source") for assay in graphs
+ }
+ store._get_latest_cell_key = Mock(return_value="I")
+ store._get_latest_feat_key = Mock(return_value="I")
+ store.get_latest_graph_loc = Mock(
+ side_effect=lambda assay, *_args: source_paths[assay]
+ )
+
+ first_ref = store.integrate_assays(
+ assays=["RNA", "ADT"],
+ label="joint",
+ method="snn",
+ chunk_size=2,
+ )
+ second_ref = store.integrate_assays(
+ assays=["RNA", "ADT"],
+ label="joint",
+ method="snn",
+ chunk_size=2,
+ )
+
+ assert first_ref.kind == "integrated_graph"
+ assert second_ref == first_ref
+ integrated_path = store._resolve_integrated_graph_path("joint")
+ assert artifact_path(first_ref) == integrated_path
+ integrated_group = store.zw[integrated_path]
+ assert integrated_group.attrs["n_cells"] == 3
+ assert integrated_group.attrs["n_neighbors"] == 2
+ assert integrated_group["edges"].shape == (6, 2)
+ assert integrated_group["weights"].shape == (6,)
+ assert load_graph.call_count == 2
+
+ integrated = GraphDataStore.load_graph(
+ store,
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I",
+ graph_loc=integrated_path,
+ )
+ assert integrated.shape == (3, 3)
+ np.testing.assert_array_equal(np.diff(integrated.indptr), [2, 2, 2])
+ np.testing.assert_allclose(
+ np.sort(integrated.data),
+ np.array([7.0, 8.0, 9.0, 10.0, 11.0, 12.0]),
+ )
+
+
+def test_integrate_assays_validation_errors() -> None:
+ store = _memory_graph_store(["RNA", "ADT"])
+ store._ensure_cell_selection = Mock(
+ return_value=ArtifactRef(
+ scope="datastore",
+ kind="cell_selection",
+ artifact_id="a" * 64,
+ )
+ )
+ graph = csr_matrix(
+ np.array(
+ [
+ [0.0, 1.0, 1.0],
+ [1.0, 0.0, 1.0],
+ [1.0, 1.0, 0.0],
+ ]
+ )
+ )
+ store.load_graph = Mock(return_value=graph)
+ source_path = _add_test_graph(store, "source")
+ store._get_latest_cell_key = Mock(return_value="I")
+ store._get_latest_feat_key = Mock(return_value="I")
+ store.get_latest_graph_loc = Mock(return_value=source_path)
+
+ with pytest.raises(ValueError, match="missing was not found"):
+ store.integrate_assays(
+ assays=["RNA", "missing"],
+ label="invalid",
+ method="snn",
+ )
+
+ with pytest.raises(ValueError, match="only two assays"):
+ store.integrate_assays(
+ assays=["RNA"],
+ label="invalid",
+ method="wnn",
+ )
+
+ with pytest.raises(ValueError, match="Method unknown not supported"):
+ store.integrate_assays(
+ assays=["RNA", "ADT"],
+ label="invalid",
+ method="unknown",
+ )
diff --git a/tests/test_graph_phase0_baseline.py b/tests/test_graph_phase0_baseline.py
new file mode 100644
index 00000000..7206a886
--- /dev/null
+++ b/tests/test_graph_phase0_baseline.py
@@ -0,0 +1,404 @@
+"""Graph-first Phase 0 baselines for master-format and v1_prep layouts.
+
+Pins path conventions, latest-pointer lookup, load_graph no-mutation, and
+workspace-relative resolution before later maintenance phases change metadata.
+"""
+
+import hashlib
+from typing import Any
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.datastore.graph_datastore import GraphDataStore
+
+
+class _MemoryGraphStore(GraphDataStore):
+ @property
+ def assay_names(self) -> list[str]:
+ return self._assay_names
+
+
+@pytest.fixture(autouse=True)
+def _isolate_path_parser_tests_from_selection_validation(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.validate_legacy_graph_selection",
+ lambda *_args, **_kwargs: None,
+ )
+
+
+def _memory_graph_store(
+ *,
+ workspace: str | None = None,
+ assay_names: list[str] | None = None,
+) -> _MemoryGraphStore:
+ store = _MemoryGraphStore.__new__(_MemoryGraphStore)
+ store.z = zarr.open_group(store=MemoryStore(), mode="w")
+ store.workspace = workspace
+ store.zarr_mode = "r+"
+ store._defaultAssay = "RNA"
+ store._assay_names = assay_names or ["RNA"]
+ store._integratedGraphsLoc = "integratedGraphs"
+ store._cachedMagicOperator = None
+ store._cachedMagicOperatorLoc = None
+ store.nthreads = 1
+ return store
+
+
+def _add_edges_weights(group: zarr.Group, n_cells: int = 3) -> None:
+ group.attrs["n_cells"] = n_cells
+ group.attrs["n_neighbors"] = 2
+ group.create_array(
+ "edges",
+ data=np.array(
+ [[0, 1], [0, 2], [1, 0], [1, 2], [2, 0], [2, 1]],
+ dtype=np.uint64,
+ ),
+ )
+ group.create_array(
+ "weights",
+ data=np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6]),
+ )
+
+
+def _add_knn_indices(knn_group: zarr.Group, n_cells: int = 3, k: int = 2) -> None:
+ knn_group.create_array(
+ "indices",
+ data=np.array([[1, 2], [0, 2], [0, 1]], dtype=np.uint64)[:n_cells, :k],
+ )
+
+
+def _install_master_format_assay_graph(
+ store: _MemoryGraphStore,
+ *,
+ assay: str = "RNA",
+ cell_key: str = "I",
+ feat_key: str = "hvgs",
+) -> dict[str, str]:
+ """Install a master-era parameter-encoded graph chain without v1_prep attrs."""
+ normed_loc = f"{assay}/normed__{cell_key}__{feat_key}"
+ reduction_loc = f"{normed_loc}/reduction__pca__10__{cell_key}"
+ ann_loc = f"{reduction_loc}/ann__l2__50__50__16__1"
+ knn_loc = f"{ann_loc}/knn__11"
+ graph_loc = f"{knn_loc}/graph__1.0__1.5"
+ kmeans_loc = f"{reduction_loc}/kmeans__100__1"
+
+ normed = store.zw.create_group(normed_loc)
+ reduction = store.zw.create_group(reduction_loc)
+ ann = store.zw.create_group(ann_loc)
+ knn = store.zw.create_group(knn_loc)
+ graph = store.zw.create_group(graph_loc)
+ store.zw.create_group(kmeans_loc)
+ _add_knn_indices(knn)
+ _add_edges_weights(graph)
+
+ # Master Harmony: unsuffixed ANN path + attrs/child, no contract-hash suffix.
+ ann.attrs["isHarmonized"] = True
+ reduction.create_array(
+ "harmonizedData",
+ data=np.zeros((3, 10), dtype=np.float32),
+ )
+
+ normed.attrs["latest_reduction"] = reduction_loc
+ reduction.attrs["latest_ann"] = ann_loc
+ reduction.attrs["latest_kmeans"] = kmeans_loc
+ ann.attrs["latest_knn"] = knn_loc
+ knn.attrs["latest_graph"] = graph_loc
+ return {
+ "normed": normed_loc,
+ "reduction": reduction_loc,
+ "ann": ann_loc,
+ "knn": knn_loc,
+ "graph": graph_loc,
+ "kmeans": kmeans_loc,
+ }
+
+
+def _group_attr_snapshot(group: zarr.Group) -> dict[str, Any]:
+ return {key: group.attrs[key] for key in group.attrs}
+
+
+def _collect_paths(root: zarr.Group, prefix: str = "") -> list[str]:
+ paths: list[str] = []
+ for key in root:
+ path = f"{prefix}/{key}" if prefix else key
+ paths.append(path)
+ child = root[key]
+ if hasattr(child, "keys"):
+ paths.extend(_collect_paths(child, path))
+ return paths
+
+
+def _store_keys_digest(root: zarr.Group) -> str:
+ digest = hashlib.sha256()
+ for path in sorted(_collect_paths(root)):
+ digest.update(path.encode())
+ node = root[path]
+ if hasattr(node, "attrs"):
+ for key in sorted(node.attrs):
+ digest.update(key.encode())
+ digest.update(repr(node.attrs[key]).encode())
+ if hasattr(node, "shape"):
+ digest.update(repr(tuple(node.shape)).encode())
+ digest.update(str(node.dtype).encode())
+ digest.update(np.asarray(node[:]).tobytes())
+ return digest.hexdigest()
+
+
+def test_normalized_group_path_convention_is_stable() -> None:
+ assay, cell_key, feat_key = "RNA", "I", "hvgs"
+ expected = f"{assay}/normed__{cell_key}__{feat_key}"
+ assert expected == "RNA/normed__I__hvgs"
+ store = _memory_graph_store()
+ assert store.get_normalized_group_path(assay, cell_key, feat_key) == expected
+ assert store.get_normalized_group_path(assay, cell_key, feat_key) == (
+ "RNA/normed__I__hvgs"
+ )
+
+
+def test_get_normalized_group_path_before_and_after_graph_exists() -> None:
+ store = _memory_graph_store()
+ before = store.get_normalized_group_path("RNA", "I", "hvgs")
+ assert before == "RNA/normed__I__hvgs"
+ assert before not in store.zw
+
+ locs = _install_master_format_assay_graph(store)
+ after = store.get_normalized_group_path("RNA", "I", "hvgs")
+ assert after == before
+ assert after == locs["normed"]
+ assert after in store.zw
+
+
+def test_get_latest_graph_loc_matches_master_chain() -> None:
+ store = _memory_graph_store()
+ locs = _install_master_format_assay_graph(store)
+
+ assert store.get_latest_graph_loc("RNA", "I", "hvgs") == locs["graph"]
+
+
+def test_lookup_and_load_graph_do_not_mutate_master_format_store() -> None:
+ store = _memory_graph_store()
+ locs = _install_master_format_assay_graph(store)
+ before = _store_keys_digest(store.z)
+ before_knn_attrs = _group_attr_snapshot(store.zw[locs["knn"]])
+
+ graph_loc = store.get_latest_graph_loc("RNA", "I", "hvgs")
+ graph = store.load_graph(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ graph_loc=graph_loc,
+ )
+ assert graph.shape == (3, 3)
+ assert graph.nnz == 6
+
+ after = _store_keys_digest(store.z)
+ after_knn_attrs = _group_attr_snapshot(store.zw[locs["knn"]])
+ assert after == before
+ assert after_knn_attrs == before_knn_attrs
+
+
+def test_master_format_chain_has_no_v1_prep_suffixes_or_schema_attrs() -> None:
+ store = _memory_graph_store()
+ locs = _install_master_format_assay_graph(store)
+
+ assert "__unscaled" not in locs["ann"]
+ assert "__harmony_" not in locs["ann"]
+ assert "ann_idx_bytes" not in store.zw[locs["ann"]]
+ assert "schemaVersion" not in store.zw[locs["ann"]].attrs
+ assert "schema_version" not in store.zw[locs["normed"]].attrs
+ assert store.zw[locs["ann"]].attrs["isHarmonized"] is True
+ assert "harmonizedData" in store.zw[locs["reduction"]]
+
+
+def test_v1_prep_suffix_conventions_are_distinct_from_master() -> None:
+ from scarf.graph.encoded_paths import parse_assay_graph_paths
+
+ store = _memory_graph_store()
+ assay = "RNA"
+ cell_key = "I"
+ feat_key = "hvgs"
+ normed_loc = f"{assay}/normed__{cell_key}__{feat_key}"
+ reduction_loc = f"{normed_loc}/reduction__pca__10__{cell_key}"
+ base_ann = f"{reduction_loc}/ann__l2__50__50__16__1"
+ unscaled_ann = f"{base_ann}__unscaled"
+ harmony_ann = f"{base_ann}__harmony_0123456789abcdef"
+ knn_loc = f"{harmony_ann}/knn__11"
+ graph_loc = f"{knn_loc}/graph__1.0__1.5"
+
+ store.zw.create_group(normed_loc).attrs["latest_reduction"] = reduction_loc
+ reduction = store.zw.create_group(reduction_loc)
+ reduction.attrs["latest_ann"] = harmony_ann
+ store.zw.create_group(unscaled_ann)
+ ann = store.zw.create_group(harmony_ann)
+ ann.attrs["latest_knn"] = knn_loc
+ ann.attrs["featureScaling"] = False
+ ann.attrs["isHarmonized"] = True
+ knn = store.zw.create_group(knn_loc)
+ knn.attrs["latest_graph"] = graph_loc
+ graph = store.zw.create_group(graph_loc)
+ _add_knn_indices(knn)
+ _add_edges_weights(graph)
+
+ assert store.get_latest_graph_loc(assay, cell_key, feat_key) == graph_loc
+ assert "__unscaled" in unscaled_ann
+ assert "__harmony_" in harmony_ann
+
+ parsed = parse_assay_graph_paths(graph_loc)
+ assert parsed.paths.cell_graph_group_path == graph_loc
+ assert parsed.harmony_contract_hash == "0123456789abcdef"
+ assert parsed.feat_scaling is True
+
+ unscaled_harmony = f"{unscaled_ann}__harmony_abcdef0123456789"
+ unscaled_parsed = parse_assay_graph_paths(
+ f"{unscaled_harmony}/knn__11/graph__1.0__1.5"
+ )
+ assert unscaled_parsed.feat_scaling is False
+ assert unscaled_parsed.harmony_contract_hash == "abcdef0123456789"
+
+
+def test_parse_master_and_v1_prep_assay_graph_paths() -> None:
+ from scarf.graph.encoded_paths import (
+ make_cell_graph_group_path,
+ make_nearest_neighbors_group_path,
+ make_neighbor_index_group_path,
+ make_normalized_group_path,
+ make_reduction_group_path,
+ parse_assay_graph_paths,
+ )
+ from scarf.graph.paths import AssayGraphPaths
+
+ master = (
+ "RNA/normed__I__hvgs/reduction__pca__10__I/"
+ "ann__l2__50__50__16__1/knn__11/graph__1.0__1.5"
+ )
+ stored = parse_assay_graph_paths(master)
+ assert stored.from_assay == "RNA"
+ assert stored.cell_key == "I"
+ assert stored.feat_key == "hvgs"
+ assert stored.reduction_method == "pca"
+ assert stored.dims == 10
+ assert stored.k == 11
+ assert stored.feat_scaling is True
+ assert stored.harmony_contract_hash is None
+ assert stored.local_connectivity == 1.0
+ assert stored.bandwidth == 1.5
+
+ normalized = make_normalized_group_path("RNA", "I", "hvgs")
+ reduction = make_reduction_group_path(normalized, "pca", 10, "I")
+ neighbor_index = make_neighbor_index_group_path(
+ reduction,
+ "l2",
+ 50,
+ 50,
+ 16,
+ 1,
+ feat_scaling=False,
+ harmony_contract_hash="deadbeefcafebabe",
+ )
+ nearest_neighbors = make_nearest_neighbors_group_path(neighbor_index, 11)
+ cell_graph = make_cell_graph_group_path(nearest_neighbors, 1.0, 1.5)
+ built = AssayGraphPaths(
+ normalized_group_path=normalized,
+ reduction_group_path=reduction,
+ neighbor_index_group_path=neighbor_index,
+ nearest_neighbors_group_path=nearest_neighbors,
+ cell_graph_group_path=cell_graph,
+ kmeans_initialization_group_path=None,
+ )
+ assert built.neighbor_index_group_path.endswith(
+ "ann__l2__50__50__16__1__unscaled__harmony_deadbeefcafebabe"
+ )
+ reparsed = parse_assay_graph_paths(built.cell_graph_group_path)
+ assert reparsed.feat_scaling is False
+ assert reparsed.harmony_contract_hash == "deadbeefcafebabe"
+
+
+def test_workspace_relative_latest_graph_lookup() -> None:
+ store = _memory_graph_store(workspace="analysis")
+ store.z.create_group("analysis")
+ locs = _install_master_format_assay_graph(store)
+
+ assert store.get_latest_graph_loc("RNA", "I", "hvgs") == locs["graph"]
+ graph = store.load_graph(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ graph_loc=locs["graph"],
+ )
+ assert graph.shape == (3, 3)
+
+
+def test_get_ini_embed_needs_only_reduction_and_kmeans() -> None:
+ # Initial embedding must not require an ANN index, KNN graph, or cell graph:
+ # only the reduction and its k-means initialization.
+ store = _memory_graph_store()
+ assay, cell_key, feat_key = "RNA", "I", "hvgs"
+ normed_loc = f"{assay}/normed__{cell_key}__{feat_key}"
+ reduction_loc = f"{normed_loc}/reduction__pca__10__{cell_key}"
+ kmeans_loc = f"{reduction_loc}/kmeans__4__1"
+ store.zw.create_group(normed_loc).attrs["latest_reduction"] = reduction_loc
+ store.zw.create_group(reduction_loc).attrs["latest_kmeans"] = kmeans_loc
+ kmeans = store.zw.create_group(kmeans_loc)
+ rng = np.random.default_rng(0)
+ kmeans.create_array("cluster_centers", data=rng.normal(size=(4, 10)))
+ kmeans.create_array(
+ "cluster_labels", data=np.array([0, 1, 2, 3, 0, 1], dtype=np.uint32)
+ )
+
+ # No latest_ann, latest_knn, or latest_graph pointers exist.
+ assert "latest_ann" not in store.zw[reduction_loc].attrs
+ assert f"{reduction_loc}/ann__l2__50__50__16__1" not in store.zw
+
+ ini = store._get_ini_embed(assay, cell_key, feat_key, 2)
+ assert ini.shape == (6, 2)
+
+
+def test_partial_knn_lookup_never_fabricates_a_graph_path() -> None:
+ from scarf.graph.encoded_paths import (
+ lookup_latest_assay_graph,
+ lookup_latest_nearest_neighbor_paths,
+ nearest_neighbor_paths_from_loc,
+ )
+ from scarf.graph.paths import AssayNearestNeighborPaths
+
+ store = _memory_graph_store()
+ normed_loc = "RNA/normed__I__hvgs"
+ reduction_loc = f"{normed_loc}/reduction__pca__10__I"
+ ann_loc = f"{reduction_loc}/ann__l2__50__50__16__1"
+ knn_loc = f"{ann_loc}/knn__11"
+ store.zw.create_group(normed_loc).attrs["latest_reduction"] = reduction_loc
+ store.zw.create_group(reduction_loc).attrs["latest_ann"] = ann_loc
+ store.zw.create_group(ann_loc).attrs["latest_knn"] = knn_loc
+ store.zw.create_group(knn_loc)
+
+ latest = lookup_latest_nearest_neighbor_paths(store.zw, "RNA", "I", "hvgs")
+ explicit = nearest_neighbor_paths_from_loc(knn_loc)
+ assert isinstance(latest, AssayNearestNeighborPaths)
+ assert latest == explicit
+ assert latest.nearest_neighbors_group_path == knn_loc
+ assert not hasattr(latest, "cell_graph_group_path")
+
+ with pytest.raises(KeyError, match="latest_graph"):
+ lookup_latest_assay_graph(store.zw, "RNA", "I", "hvgs")
+
+
+def test_integrated_graph_load_by_explicit_path_is_stable() -> None:
+ store = _memory_graph_store()
+ graph_loc = "integratedGraphs/joint"
+ group = store.zw.create_group(graph_loc)
+ _add_edges_weights(group)
+ before = _store_keys_digest(store.z)
+
+ graph = store.load_graph(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I",
+ graph_loc=graph_loc,
+ )
+ assert graph.shape == (3, 3)
+ assert _store_keys_digest(store.z) == before
diff --git a/tests/test_harmony_contracts.py b/tests/test_harmony_contracts.py
new file mode 100644
index 00000000..37845ea1
--- /dev/null
+++ b/tests/test_harmony_contracts.py
@@ -0,0 +1,170 @@
+import inspect
+from types import SimpleNamespace
+
+import numpy as np
+import pandas as pd
+
+import scarf.embeddings as embeddings
+import scarf.embeddings.harmony as harmony
+from scarf.embeddings.harmony.api import fit_harmony as implementation_fit_harmony
+from scarf.embeddings.harmony.api import run_harmony as implementation_run_harmony
+from scarf.embeddings.harmony.models import HarmonyResult as implementation_result
+from scarf.embeddings.harmony.optimizer import Harmony as implementation_optimizer
+from tests.signature_contracts import signature_digest
+
+
+def test_harmony_facade_exports_canonical_objects():
+ assert harmony.__all__ == [
+ "ClusterFn",
+ "Harmony",
+ "HarmonyResult",
+ "fit_harmony",
+ "moe_correct_ridge",
+ "run_harmony",
+ "safe_entropy",
+ ]
+ assert harmony.fit_harmony is implementation_fit_harmony
+ assert harmony.run_harmony is implementation_run_harmony
+ assert harmony.HarmonyResult is implementation_result
+ assert harmony.Harmony is implementation_optimizer
+ assert embeddings.Harmony is harmony.Harmony
+ assert embeddings.HarmonyResult is harmony.HarmonyResult
+ assert embeddings.fit_harmony is harmony.fit_harmony
+ assert embeddings.run_harmony is harmony.run_harmony
+
+
+def test_harmony_public_metadata_and_signatures_remain_stable():
+ public_objects = (
+ harmony.Harmony,
+ harmony.HarmonyResult,
+ harmony.fit_harmony,
+ harmony.moe_correct_ridge,
+ harmony.run_harmony,
+ harmony.safe_entropy,
+ )
+ assert {obj.__module__ for obj in public_objects} == {"scarf.embeddings.harmony"}
+ assert signature_digest(
+ {
+ "fit_harmony": harmony.fit_harmony,
+ "run_harmony": harmony.run_harmony,
+ }
+ ) == ("7b192e50655559a92f78d67ba963db2f4bb1df3195f5f7305effc37f913dbf9c")
+ assert (
+ inspect.signature(harmony.fit_harmony).parameters
+ == inspect.signature(harmony.run_harmony).parameters
+ )
+
+
+def test_run_harmony_resolves_fit_through_public_facade(monkeypatch):
+ corrected = np.array([[1.0, 2.0]])
+ calls = []
+
+ def fake_fit_harmony(*args, **kwargs):
+ calls.append((args, kwargs))
+ return SimpleNamespace(corrected=corrected)
+
+ monkeypatch.setattr(embeddings, "fit_harmony", fake_fit_harmony)
+ actual = embeddings.run_harmony(
+ np.zeros((1, 2)),
+ pd.DataFrame({"batch": ["a", "b"]}),
+ )
+
+ assert actual is corrected
+ assert len(calls) == 1
+
+
+def test_harmony_keeps_an_independent_original_coordinate_snapshot():
+ values = np.random.default_rng(4).normal(size=(3, 12))
+ metadata = pd.DataFrame({"batch": ["a", "b"] * 6})
+
+ result = harmony.fit_harmony(
+ values,
+ metadata,
+ nclust=2,
+ max_iter_harmony=1,
+ max_iter_kmeans=1,
+ )
+
+ assert result.original is not values
+ np.testing.assert_array_equal(result.original, values)
+ assert result.corrected.dtype == np.dtype(np.float64)
+
+
+def test_harmony_progress_completes_when_optimization_converges(monkeypatch):
+ closed: list[tuple[int, int]] = []
+
+ class Progress:
+ def __init__(self, total: int) -> None:
+ self.n = 0
+ self.total = total
+
+ def update(self) -> None:
+ self.n += 1
+
+ def refresh(self) -> None:
+ pass
+
+ def close(self) -> None:
+ closed.append((self.n, self.total))
+
+ monkeypatch.setattr(
+ "scarf.embeddings.harmony.optimizer.tqdmbar",
+ lambda *_, total, **__: Progress(total),
+ )
+ values = np.random.default_rng(5).normal(size=(3, 12))
+ metadata = pd.DataFrame({"batch": ["a", "b"] * 6})
+
+ harmony.fit_harmony(
+ values,
+ metadata,
+ nclust=2,
+ max_iter_harmony=50,
+ max_iter_kmeans=1,
+ epsilon_harmony=1e9,
+ )
+
+ assert len(closed) == 1
+ completed, total = closed[0]
+ assert completed == total
+ assert completed < 50
+
+
+def test_harmony_supports_numeric_batch_columns_without_global_rng_changes():
+ values = np.random.default_rng(9).normal(size=(3, 12))
+ metadata = pd.DataFrame({"batch": [0, 1] * 6})
+ np.random.seed(17)
+ expected_next = np.random.random()
+ np.random.seed(17)
+
+ result = harmony.fit_harmony(
+ values,
+ metadata,
+ nclust=2,
+ max_iter_harmony=1,
+ max_iter_kmeans=1,
+ )
+
+ assert result.parameters["clusterBackend"] == "sklearn.cluster.KMeans"
+ assert np.random.random() == expected_next
+
+
+def test_harmony_centroids_match_final_assignments_and_coordinates():
+ values = np.random.default_rng(11).normal(size=(4, 16))
+ metadata = pd.DataFrame({"batch": ["a", "b"] * 8})
+
+ result = harmony.fit_harmony(
+ values,
+ metadata,
+ nclust=3,
+ max_iter_harmony=2,
+ max_iter_kmeans=2,
+ )
+
+ normalized = result.corrected / np.linalg.norm(
+ result.corrected,
+ axis=0,
+ keepdims=True,
+ )
+ expected = normalized @ result.assignments.T
+ expected /= np.linalg.norm(expected, axis=0, keepdims=True)
+ np.testing.assert_allclose(result.centroids, expected)
diff --git a/tests/test_hto_seurat_golden.py b/tests/test_hto_seurat_golden.py
new file mode 100644
index 00000000..f1c00e80
--- /dev/null
+++ b/tests/test_hto_seurat_golden.py
@@ -0,0 +1,90 @@
+import hashlib
+import json
+from pathlib import Path
+from typing import Any, cast
+
+import numpy as np
+import pandas as pd
+
+from scarf.quality_control.hto import hto_demux
+
+_FIXTURE_PATH = Path(__file__).parent / "seurat_hto_5_5_1_golden.json"
+
+
+def _golden_fixture() -> dict[str, Any]:
+ return cast(dict[str, Any], json.loads(_FIXTURE_PATH.read_text()))
+
+
+def test_seurat_hto_golden_fixture_pins_its_provenance() -> None:
+ fixture = _golden_fixture()
+ provenance = fixture["provenance"]
+ inputs = fixture["inputs"]
+
+ assert provenance["package"] == "Seurat"
+ assert provenance["packageVersion"] == "5.5.1"
+ assert provenance["seuratObjectVersion"] == "5.4.0"
+ assert provenance["fitdistrplusVersion"] == "1.2.6"
+ assert provenance["dataset"] == "GSE245108"
+ assert provenance["sourceFile"] == (
+ "GSE245108_TNC-1-2-1_filtered_feature_bc_matrix.h5"
+ )
+ assert provenance["sourceSha256"] == (
+ "22d64482c0fb04a7685e641d762efdd95a6498156ac22e7b0264925b0907b589"
+ )
+ assert provenance["nCells"] == 1000
+ assert provenance["nHtos"] == 10
+ assert provenance["cellSampleSeed"] == 20260731
+ assert provenance["randomSeed"] == 0
+ assert provenance["normalization"] == {"method": "CLR", "margin": 1}
+ assert provenance["clustering"] == {
+ "method": "kmeans",
+ "clusterCount": 11,
+ "nStarts": 100,
+ }
+ assert provenance["cutoff"] == {
+ "distribution": "negative_binomial",
+ "positiveQuantile": 0.99,
+ "comparison": "strictly_greater",
+ }
+ assert provenance["referenceFunction"] == "Seurat::HTODemux"
+
+ fingerprint_values = {
+ "barcodes": inputs["barcodes"],
+ "htoNames": inputs["htoNames"],
+ "rawCounts": inputs["rawCounts"],
+ }
+ encoded = json.dumps(
+ fingerprint_values,
+ separators=(",", ":"),
+ sort_keys=True,
+ ).encode()
+ assert hashlib.sha256(encoded).hexdigest() == inputs["sha256"]
+
+
+def test_hto_demux_matches_seurat_5_5_1_golden_calls() -> None:
+ fixture = _golden_fixture()
+ provenance = fixture["provenance"]
+ inputs = fixture["inputs"]
+ expected = fixture["seurat"]
+ counts = pd.DataFrame(
+ inputs["rawCounts"],
+ index=inputs["barcodes"],
+ columns=inputs["htoNames"],
+ dtype=np.int64,
+ )
+
+ assignments = hto_demux(counts, random_seed=provenance["randomSeed"])
+
+ np.testing.assert_array_equal(
+ assignments.to_numpy(),
+ np.asarray(expected["hashId"]),
+ )
+ global_assignments = np.where(
+ assignments.isin(["Negative", "Doublet"]),
+ assignments,
+ "Singlet",
+ )
+ np.testing.assert_array_equal(
+ global_assignments,
+ np.asarray(expected["classificationGlobal"]),
+ )
diff --git a/tests/test_hvg_selection.py b/tests/test_hvg_selection.py
new file mode 100644
index 00000000..b1e6e6a7
--- /dev/null
+++ b/tests/test_hvg_selection.py
@@ -0,0 +1,168 @@
+from types import MethodType
+from typing import Literal
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.assay import ADTassay, RNAassay
+from scarf.datastore.datastore import DataStore
+from scarf.metadata import MetaData
+
+
+def _rna_with_feature_names(
+ names: list[str],
+) -> tuple[RNAassay, list[tuple[str, int, float, str]]]:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ feature_data = root.create_group("featureData")
+ feature_data.create_array("I", data=np.ones(len(names), dtype=bool))
+ feature_data.create_array(
+ "ids",
+ data=np.asarray([f"feature_{index}" for index in range(len(names))]),
+ )
+ feature_data.create_array("names", data=np.asarray(names))
+
+ assay = RNAassay.__new__(RNAassay)
+ assay.z = root
+ assay.feats = MetaData(feature_data)
+ calls: list[tuple[str, int, float, str]] = []
+
+ def set_summary_stats(
+ self: RNAassay,
+ cell_key: str,
+ n_bins: int,
+ lowess_frac: float,
+ *,
+ bin_strategy: Literal["fixed", "adaptive"] = "adaptive",
+ ) -> tuple[str, str]:
+ calls.append((cell_key, n_bins, lowess_frac, bin_strategy))
+ strategy = "" if bin_strategy == "fixed" else "adaptive__"
+ return f"stats_{cell_key}", f"c_var__{strategy}{n_bins}__{lowess_frac}"
+
+ assay.set_summary_stats = MethodType(set_summary_stats, assay)
+ return assay, calls
+
+
+def test_set_hvgs_installs_mask_and_applies_regex_exclusions():
+ assay, calls = _rna_with_feature_names(["MT-A", "GENE1", "RPL3", "KEEP"])
+
+ column = assay.set_hvgs(
+ "selected_cells",
+ mask=np.ones(4, dtype=bool),
+ blacklist="^MT-|^RPL",
+ blacklist_exclusions="^RPL",
+ )
+
+ assert column == "selected_cells__hvgs"
+ np.testing.assert_array_equal(
+ assay.feats.fetch_all(column),
+ np.array([False, True, True, True]),
+ )
+ assert calls == [("selected_cells", 200, 0.1, "adaptive")]
+
+
+def test_set_hvgs_accepts_global_indexes_and_prefers_blacklist_indexes():
+ assay, calls = _rna_with_feature_names(["MT-A", "GENE1", "GENE2", "KEEP"])
+
+ column = assay.set_hvgs(
+ "I",
+ feature_indexes=[0, 1, 3],
+ hvg_key_name="custom",
+ n_bins=50,
+ lowess_frac=0.2,
+ bin_strategy="adaptive",
+ blacklist="^KEEP",
+ blacklist_indexes=[1],
+ )
+
+ assert column == "I__custom"
+ np.testing.assert_array_equal(
+ assay.feats.fetch_all(column),
+ np.array([True, False, False, True]),
+ )
+ assert calls == [("I", 50, 0.2, "adaptive")]
+
+
+@pytest.mark.parametrize(
+ ("kwargs", "error"),
+ [
+ ({}, ValueError),
+ (
+ {
+ "mask": np.array([True, False]),
+ "feature_indexes": [0],
+ },
+ ValueError,
+ ),
+ ({"mask": np.array([1, 0, 0, 0])}, TypeError),
+ ({"mask": np.array([True, False])}, ValueError),
+ ({"feature_indexes": []}, ValueError),
+ ({"feature_indexes": [0, 0]}, ValueError),
+ ({"feature_indexes": [4]}, IndexError),
+ (
+ {
+ "feature_indexes": [0],
+ "blacklist_exclusions": "^MT-",
+ },
+ ValueError,
+ ),
+ ],
+)
+def test_set_hvgs_validates_selection_before_computing_stats(kwargs, error):
+ assay, calls = _rna_with_feature_names(["A", "B", "C", "D"])
+
+ with pytest.raises(error):
+ assay.set_hvgs("I", **kwargs)
+
+ assert calls == []
+
+
+@pytest.mark.parametrize(
+ ("kwargs", "error"),
+ [
+ ({"n_bins": True}, TypeError),
+ ({"lowess_frac": np.nan}, ValueError),
+ ({"bin_strategy": "unknown"}, ValueError),
+ ],
+)
+def test_set_summary_stats_validates_before_computing_feature_stats(kwargs, error):
+ assay = RNAassay.__new__(RNAassay)
+ assay.set_feature_stats = lambda cell_key: pytest.fail(
+ "invalid correction parameters must not trigger feature statistics"
+ )
+
+ with pytest.raises(error):
+ assay.set_summary_stats("I", **kwargs)
+
+
+def test_datastore_set_hvgs_delegates_and_rejects_non_rna_assays():
+ assay = RNAassay.__new__(RNAassay)
+ delegated = []
+
+ def set_hvgs(self: RNAassay, cell_key: str, **kwargs) -> str:
+ delegated.append((cell_key, kwargs))
+ return f"{cell_key}__hvgs"
+
+ assay.set_hvgs = MethodType(set_hvgs, assay)
+ store = DataStore.__new__(DataStore)
+ store._get_assay = lambda from_assay: assay
+
+ result = store.set_hvgs(
+ from_assay="RNA",
+ cell_key="selected",
+ feature_indexes=[2],
+ blacklist_indexes=[1],
+ bin_strategy="adaptive",
+ )
+
+ assert result == "selected__hvgs"
+ assert delegated[0][0] == "selected"
+ assert delegated[0][1]["feature_indexes"] == [2]
+ assert delegated[0][1]["blacklist_indexes"] == [1]
+ assert delegated[0][1]["bin_strategy"] == "adaptive"
+
+ adt = ADTassay.__new__(ADTassay)
+ store._get_assay = lambda from_assay: adt
+ with pytest.raises(TypeError, match="RNAassay"):
+ store.set_hvgs(cell_key="I", feature_indexes=[0])
diff --git a/tests/test_import_architecture.py b/tests/test_import_architecture.py
new file mode 100644
index 00000000..8897af52
--- /dev/null
+++ b/tests/test_import_architecture.py
@@ -0,0 +1,1123 @@
+import ast
+import subprocess
+import sys
+from importlib.util import find_spec
+from pathlib import Path
+
+
+_SCARF_ROOT = Path(__file__).resolve().parents[1] / "scarf"
+_MOVED_SYMBOLS = {
+ "datastore.datastore": {
+ "_MARKER_OUT_COLUMNS",
+ "_MARKER_STAT_COLUMNS",
+ "_feature_column_chunk",
+ "_group_assignment_digest",
+ "_load_marker_cluster_frame",
+ "_marker_stats_matrix",
+ "_scatter_feature_clusters",
+ "_shared_marker_feature_index",
+ "_validated_pseudotime_regressor",
+ "_validate_assay_pseudotime",
+ "_write_compact_marker_stats",
+ },
+ "datastore.graph_datastore": {
+ "EMBEDDING_CACHE_MAX_BYTES",
+ "_make_source_sink_vector",
+ "_random_walk_laplacian_transpose",
+ "_select_pseudotime_component",
+ "_truncated_pba_potential",
+ "_validate_source_sink_labels",
+ "_validate_source_sink_vector",
+ },
+ "knn_utils": {
+ "_is_umap_version_new",
+ "calc_snn",
+ "export_knn_to_mtx",
+ "merge_graphs",
+ "run_sgtsne",
+ "self_query_knn",
+ "smoothen_dists",
+ "weight_sort_indices",
+ "wnn_integration",
+ },
+}
+
+
+def _upward_imports(
+ package_name: str,
+ forbidden_packages: set[str],
+ allowed_modules: frozenset[str] = frozenset(),
+) -> set[tuple[str, str]]:
+ package_root = _SCARF_ROOT / package_name
+ violations: set[tuple[str, str]] = set()
+
+ for path in package_root.rglob("*.py"):
+ tree = ast.parse(path.read_text(), filename=str(path))
+ for node in ast.walk(tree):
+ targets: set[str] = set()
+ if isinstance(node, ast.Import):
+ for alias in node.names:
+ parts = alias.name.split(".")
+ if len(parts) > 1 and parts[0] == "scarf":
+ targets.add(".".join(parts[1:]))
+ elif isinstance(node, ast.ImportFrom):
+ if node.level >= 2:
+ if node.module:
+ targets.add(node.module)
+ else:
+ targets.update(alias.name for alias in node.names)
+ elif node.level == 0 and node.module:
+ parts = node.module.split(".")
+ if parts[0] == "scarf":
+ if len(parts) > 1:
+ targets.add(".".join(parts[1:]))
+ else:
+ targets.update(alias.name for alias in node.names)
+
+ for target in targets - allowed_modules:
+ root = target.split(".")[0]
+ if root in forbidden_packages:
+ violations.add((path.relative_to(package_root).as_posix(), root))
+
+ return violations
+
+
+def _root_imports(path: Path) -> set[str]:
+ parent_parts = list(path.relative_to(_SCARF_ROOT).parent.parts)
+ if parent_parts == ["."]:
+ parent_parts = []
+ imports: set[str] = set()
+ tree = ast.parse(path.read_text(), filename=str(path))
+
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Import):
+ for alias in node.names:
+ parts = alias.name.split(".")
+ if parts[0] == "scarf" and len(parts) > 1:
+ imports.add(".".join(parts[1:]))
+ continue
+ if not isinstance(node, ast.ImportFrom):
+ continue
+
+ if node.level == 0:
+ if not node.module:
+ continue
+ parts = node.module.split(".")
+ if parts[0] != "scarf":
+ continue
+ relative_parts = parts[1:]
+ else:
+ up = node.level - 1
+ relative_parts = parent_parts[: len(parent_parts) - up]
+ if node.module:
+ relative_parts += node.module.split(".")
+
+ if relative_parts:
+ imports.add(".".join(relative_parts))
+ else:
+ imports.update(alias.name for alias in node.names)
+
+ return imports
+
+
+def _facade_importers(facade_name: str) -> set[str]:
+ return {
+ path.relative_to(_SCARF_ROOT).as_posix()
+ for path in _SCARF_ROOT.rglob("*.py")
+ if path != _SCARF_ROOT / "__init__.py" and facade_name in _root_imports(path)
+ }
+
+
+def _resolved_module(path: Path, node: ast.ImportFrom) -> str | None:
+ if node.level == 0:
+ if node.module == "scarf":
+ return ""
+ if node.module and node.module.startswith("scarf."):
+ return node.module.removeprefix("scarf.")
+ return None
+
+ parent_parts = list(path.relative_to(_SCARF_ROOT).parent.parts)
+ up = node.level - 1
+ module_parts = parent_parts[: len(parent_parts) - up]
+ if node.module:
+ module_parts.extend(node.module.split("."))
+ return ".".join(module_parts)
+
+
+def _runtime_import_modules(
+ path: Path,
+ *,
+ include_function_local: bool = True,
+) -> set[str]:
+ class RuntimeImportVisitor(ast.NodeVisitor):
+ def __init__(self) -> None:
+ self.modules: set[str] = set()
+
+ def visit_FunctionDef(self, node: ast.FunctionDef) -> None:
+ if include_function_local:
+ self.generic_visit(node)
+
+ def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> None:
+ if include_function_local:
+ self.generic_visit(node)
+
+ def visit_If(self, node: ast.If) -> None:
+ if isinstance(node.test, ast.Name) and node.test.id == "TYPE_CHECKING":
+ for child in node.orelse:
+ self.visit(child)
+ return
+ self.generic_visit(node)
+
+ def visit_Import(self, node: ast.Import) -> None:
+ for alias in node.names:
+ if alias.name.startswith("scarf."):
+ self.modules.add(alias.name.removeprefix("scarf."))
+
+ def visit_ImportFrom(self, node: ast.ImportFrom) -> None:
+ module_name = _resolved_module(path, node)
+ if module_name is not None:
+ if node.level > 0 and node.module is None:
+ for alias in node.names:
+ resolved = (
+ f"{module_name}.{alias.name}" if module_name else alias.name
+ )
+ self.modules.add(resolved)
+ else:
+ self.modules.add(module_name)
+
+ visitor = RuntimeImportVisitor()
+ visitor.visit(ast.parse(path.read_text(), filename=str(path)))
+ return visitor.modules
+
+
+def _attribute_parts(node: ast.AST) -> list[str] | None:
+ if isinstance(node, ast.Name):
+ return [node.id]
+ if isinstance(node, ast.Attribute):
+ parts = _attribute_parts(node.value)
+ if parts is not None:
+ return [*parts, node.attr]
+ return None
+
+
+def _moved_symbol_imports() -> set[tuple[str, str, str]]:
+ violations: set[tuple[str, str, str]] = set()
+ for path in _SCARF_ROOT.rglob("*.py"):
+ tree = ast.parse(path.read_text(), filename=str(path))
+ aliases: dict[str, str] = {}
+
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Import):
+ for alias in node.names:
+ if not alias.name.startswith("scarf."):
+ continue
+ module_name = alias.name.removeprefix("scarf.")
+ if alias.asname:
+ aliases[alias.asname] = module_name
+ else:
+ aliases["scarf"] = ""
+ continue
+ if not isinstance(node, ast.ImportFrom):
+ continue
+
+ module_name = _resolved_module(path, node)
+ if module_name is None:
+ continue
+ moved_symbols = _MOVED_SYMBOLS.get(module_name, set())
+ for alias in node.names:
+ if alias.name in moved_symbols:
+ violations.add(
+ (
+ path.relative_to(_SCARF_ROOT).as_posix(),
+ module_name,
+ alias.name,
+ )
+ )
+ imported_module = ".".join(filter(None, [module_name, alias.name]))
+ if imported_module in _MOVED_SYMBOLS:
+ aliases[alias.asname or alias.name] = imported_module
+
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Attribute):
+ continue
+ parts = _attribute_parts(node)
+ if parts is None or parts[0] not in aliases:
+ continue
+ resolved = [*filter(None, aliases[parts[0]].split(".")), *parts[1:]]
+ if len(resolved) < 2:
+ continue
+ module_name = ".".join(resolved[:-1])
+ symbol_name = resolved[-1]
+ if symbol_name in _MOVED_SYMBOLS.get(module_name, set()):
+ violations.add(
+ (
+ path.relative_to(_SCARF_ROOT).as_posix(),
+ module_name,
+ symbol_name,
+ )
+ )
+
+ return violations
+
+
+def test_storage_has_no_upward_dependencies():
+ assert (
+ _upward_imports(
+ "storage",
+ {"assay", "datastore", "plotting", "writers"},
+ )
+ == set()
+ )
+
+
+def test_matrix_has_no_domain_or_orchestration_dependencies():
+ assert (
+ _upward_imports(
+ "matrix",
+ {
+ "assay",
+ "clustering",
+ "datastore",
+ "embeddings",
+ "features",
+ "mapping",
+ "merge",
+ "metadata",
+ "metrics",
+ "neighbors",
+ "plotting",
+ "quality_control",
+ "readers",
+ "trajectory",
+ "writers",
+ },
+ )
+ == set()
+ )
+
+
+def test_plotting_does_not_import_datastore():
+ assert _upward_imports("plotting", {"datastore"}) == set()
+
+
+def test_datastore_plot_namespace_defers_plotting_imports():
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ """
+import sys
+
+from scarf.datastore.datastore import DataStore
+
+optional = ("matplotlib", "seaborn")
+assert not any(
+ name == prefix or name.startswith(f"{prefix}.")
+ for name in sys.modules
+ for prefix in ("scarf.plotting", *optional)
+)
+
+store = object.__new__(DataStore)
+accessor = store.plots
+
+assert type(accessor).__module__ == "scarf.datastore.plot_accessor"
+concrete_modules = {
+ "scarf.plotting.composition",
+ "scarf.plotting.diagnostics",
+ "scarf.plotting.distribution",
+ "scarf.plotting.embedding",
+ "scarf.plotting.embedding_raster",
+ "scarf.plotting.heatmaps",
+ "scarf.plotting.summary",
+}
+assert concrete_modules.isdisjoint(sys.modules)
+assert not any(
+ name == prefix or name.startswith(f"{prefix}.")
+ for name in sys.modules
+ for prefix in optional
+)
+
+import scarf.plotting as plotting
+
+_ = plotting.embedding
+assert "scarf.plotting.embedding" in sys.modules
+assert not any(
+ name == prefix or name.startswith(f"{prefix}.")
+ for name in sys.modules
+ for prefix in optional
+)
+""",
+ ],
+ check=True,
+ )
+
+
+def test_algorithm_domains_do_not_import_orchestration_or_io():
+ # storage.refs holds the artifact reference value type and reads no store,
+ # so results that a caller persists may name it.
+ forbidden = {"datastore", "plotting", "readers", "storage", "writers"}
+ allowed = frozenset({"storage.refs"})
+ for package_name in ("clustering", "embeddings", "trajectory"):
+ package_root = _SCARF_ROOT / package_name
+ if package_root.is_dir():
+ assert _upward_imports(package_name, forbidden, allowed) == set()
+ assert {path for path, _target in _upward_imports("clustering", {"storage"})} == {
+ "paris_multiscale.py"
+ }
+
+
+def test_artifact_reference_module_has_no_storage_dependencies():
+ path = _SCARF_ROOT / "storage" / "refs.py"
+ tree = ast.parse(path.read_text(), filename=str(path))
+ imported = {
+ alias.name.split(".")[0]
+ for node in ast.walk(tree)
+ if isinstance(node, ast.Import)
+ for alias in node.names
+ } | {
+ node.module.split(".")[0]
+ for node in ast.walk(tree)
+ if isinstance(node, ast.ImportFrom) and node.module
+ }
+ assert imported <= {"re", "collections", "dataclasses", "typing"}
+
+
+def test_metrics_and_embedding_harmony_avoid_runtime_orchestration_and_io_imports():
+ forbidden_roots = {
+ "datastore",
+ "merge",
+ "plotting",
+ "readers",
+ "storage",
+ "writers",
+ }
+ paths = [
+ *_SCARF_ROOT.joinpath("metrics").glob("*.py"),
+ *_SCARF_ROOT.joinpath("embeddings", "harmony").glob("*.py"),
+ ]
+ for path in paths:
+ runtime_imports = _runtime_import_modules(path)
+ assert not {
+ module_name
+ for module_name in runtime_imports
+ if module_name.split(".", 1)[0] in forbidden_roots
+ }
+
+
+def test_pca_and_lsi_implementations_live_under_embeddings():
+ neighbor_source = (_SCARF_ROOT / "neighbors" / "stream.py").read_text()
+ reduction_source = (_SCARF_ROOT / "embeddings" / "reduction.py").read_text()
+
+ assert "sklearn.decomposition" not in neighbor_source
+ assert "IncrementalPCA" in reduction_source
+ assert "TruncatedSVD" in reduction_source
+
+
+def test_extracted_domains_have_only_narrow_storage_dependencies():
+ forbidden = {"datastore", "plotting", "readers", "writers"}
+ storage_exceptions = {
+ "features": {
+ "genomic/melding.py",
+ "markers/batching.py",
+ "markers/search.py",
+ },
+ "neighbors": set(),
+ "quality_control": {"doublets.py"},
+ }
+ for package_name, allowed_storage_importers in storage_exceptions.items():
+ assert _upward_imports(package_name, forbidden) == set()
+ storage_importers = {
+ path for path, target in _upward_imports(package_name, {"storage"})
+ }
+ assert storage_importers == allowed_storage_importers
+
+
+def test_read_paths_take_chunk_geometry_only_from_the_storage_geometry_module():
+ # scarf/storage/geometry.py owns the one read of an array's chunk grid.
+ # Write-path chunk specs in layout.py and sharding.py are a separate concern
+ # and stay out of this guard.
+ read_path_modules = (
+ "assay/rna.py",
+ "datastore/_operations/graph.py",
+ "datastore/_operations/mapping.py",
+ "mapping/confidence.py",
+ "mapping/hashing.py",
+ "matrix/chunked.py",
+ "metadata/rows.py",
+ "storage/artifacts.py",
+ "storage/copy.py",
+ "storage/feature_stream.py",
+ "storage/partition.py",
+ )
+ offenders = set()
+ for relative_path in read_path_modules:
+ path = _SCARF_ROOT / relative_path
+ for node in ast.walk(ast.parse(path.read_text(), filename=str(path))):
+ reads_subscript = (
+ isinstance(node, ast.Subscript)
+ and isinstance(node.value, ast.Attribute)
+ and node.value.attr in {"chunks", "shards"}
+ )
+ reads_getattr = (
+ isinstance(node, ast.Call)
+ and isinstance(node.func, ast.Name)
+ and node.func.id == "getattr"
+ and len(node.args) > 1
+ and isinstance(node.args[1], ast.Constant)
+ and node.args[1].value in {"chunks", "shards"}
+ )
+ if reads_subscript or reads_getattr:
+ offenders.add((relative_path, node.lineno))
+
+ assert offenders == set()
+
+
+def test_mapping_does_not_import_orchestration_or_general_io():
+ assert (
+ _upward_imports(
+ "mapping",
+ {"datastore", "plotting", "readers", "writers"},
+ )
+ == set()
+ )
+ storage_importers = {
+ path for path, target in _upward_imports("mapping", {"storage"})
+ }
+ assert storage_importers <= {
+ "artifact.py",
+ "confidence.py",
+ "features.py",
+ "hashing.py",
+ "models.py",
+ "projection.py",
+ "reference.py",
+ }
+
+
+def test_internal_modules_use_canonical_storage_and_utility_paths():
+ assert _facade_importers("ann") == set()
+ assert _facade_importers("dendrogram") == set()
+ assert _facade_importers("knn_utils") == set()
+ assert _facade_importers("results") == set()
+ assert _facade_importers("umap") == set()
+ assert _facade_importers("writers") == set()
+ assert _facade_importers("parallel") == set()
+ assert _facade_importers("storage.zarr_store") == set()
+ assert _facade_importers("utils") == set()
+ for retired_module in (
+ "bio_data",
+ "doublet_utils",
+ "feat_utils",
+ "meld_assay",
+ ):
+ assert _facade_importers(retired_module) == set()
+ for retired_module in ("blocks", "memory", "storage", "system", "windows"):
+ assert _facade_importers(f"utils.{retired_module}") == set()
+
+
+def test_internal_modules_do_not_use_moved_symbols_from_hybrid_facades():
+ assert _moved_symbol_imports() == set()
+
+
+def test_compatibility_only_modules_are_removed():
+ retired = {
+ _SCARF_ROOT / "_types.py",
+ _SCARF_ROOT / "ann.py",
+ _SCARF_ROOT / "bio_data.py",
+ _SCARF_ROOT / "chunked.py",
+ _SCARF_ROOT / "dendrogram.py",
+ _SCARF_ROOT / "downloader.py",
+ _SCARF_ROOT / "doublet_utils.py",
+ _SCARF_ROOT / "feat_utils.py",
+ _SCARF_ROOT / "harmony.py",
+ _SCARF_ROOT / "harmony" / "__init__.py",
+ _SCARF_ROOT / "harmony" / "api.py",
+ _SCARF_ROOT / "harmony" / "models.py",
+ _SCARF_ROOT / "harmony" / "optimizer.py",
+ _SCARF_ROOT / "genomics" / "__init__.py",
+ _SCARF_ROOT / "genomics" / "gff.py",
+ _SCARF_ROOT / "genomics" / "intervals.py",
+ _SCARF_ROOT / "genomics" / "melding.py",
+ _SCARF_ROOT / "genomics" / "reference.py",
+ _SCARF_ROOT / "knn_utils.py",
+ _SCARF_ROOT / "mapping" / "coral.py",
+ _SCARF_ROOT / "mapping_reference.py",
+ _SCARF_ROOT / "mapping_utils.py",
+ _SCARF_ROOT / "markers.py",
+ _SCARF_ROOT / "markers" / "__init__.py",
+ _SCARF_ROOT / "markers" / "batching.py",
+ _SCARF_ROOT / "markers" / "rank.py",
+ _SCARF_ROOT / "markers" / "regression.py",
+ _SCARF_ROOT / "markers" / "search.py",
+ _SCARF_ROOT / "meld_assay.py",
+ _SCARF_ROOT / "metadata.py",
+ _SCARF_ROOT / "metrics.py",
+ _SCARF_ROOT / "neighbors" / "persistence.py",
+ _SCARF_ROOT / "features" / "lowess.py",
+ _SCARF_ROOT / "clustering" / "_paris_mdl.py",
+ _SCARF_ROOT / "clustering" / "feature_graph.py",
+ _SCARF_ROOT / "clustering" / "hierarchy.py",
+ _SCARF_ROOT / "parallel.py",
+ _SCARF_ROOT / "plotting" / "unified.py",
+ _SCARF_ROOT / "results.py",
+ _SCARF_ROOT / "storage" / "zarr_store.py",
+ _SCARF_ROOT / "trajectory" / "aggregation.py",
+ _SCARF_ROOT / "symphony.py",
+ _SCARF_ROOT / "umap.py",
+ }
+ assert not {
+ path.relative_to(_SCARF_ROOT).as_posix() for path in retired if path.exists()
+ }
+
+
+def test_retired_root_import_paths_do_not_resolve():
+ for module_name in (
+ "scarf._types",
+ "scarf.chunked",
+ "scarf.downloader",
+ "scarf.harmony",
+ "scarf.genomics",
+ "scarf.knn_utils",
+ "scarf.mapping.coral",
+ "scarf.markers",
+ "scarf.plotting.unified",
+ "scarf.clustering._paris_mdl",
+ "scarf.clustering.feature_graph",
+ "scarf.clustering.hierarchy",
+ "scarf.features.lowess",
+ "scarf.trajectory.aggregation",
+ ):
+ assert find_spec(module_name) is None
+
+
+def test_utility_modules_use_domain_names():
+ retired_files = {
+ "blocks.py",
+ "memory.py",
+ "storage.py",
+ "system.py",
+ "windows.py",
+ }
+ assert not retired_files.intersection(
+ path.name for path in (_SCARF_ROOT / "utils").glob("*.py")
+ )
+
+
+def test_data_model_defers_domain_algorithms_until_method_calls():
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ """
+import sys
+
+import scarf.metadata
+
+assert not any(
+ name == "scarf.features" or name.startswith("scarf.features.")
+ for name in sys.modules
+)
+
+import scarf.assay
+
+assert not any(
+ name == "scarf.trajectory" or name.startswith("scarf.trajectory.")
+ for name in sys.modules
+)
+""",
+ ],
+ check=True,
+ )
+
+
+def test_features_facades_defer_nested_implementations():
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ """
+import sys
+
+import scarf
+import scarf.features as features
+
+assert "scarf.features.variability" not in sys.modules
+assert "scarf.features.genomic.gff" not in sys.modules
+assert "scarf.features.genomic.melding" not in sys.modules
+assert "scarf.features.markers.search" not in sys.modules
+assert "scarf.features.enrichment" not in sys.modules
+assert "scarf.features.enrichment.net" not in sys.modules
+assert "scarf.features.enrichment.results" not in sys.modules
+assert "scarf.features.enrichment.aucell" not in sys.modules
+assert "scarf.features.enrichment.waggr" not in sys.modules
+
+_ = scarf.read_gmt
+assert "scarf.features.enrichment.net" in sys.modules
+assert "scarf.features.enrichment.results" not in sys.modules
+assert "scarf.features.enrichment.aucell" not in sys.modules
+assert "scarf.features.enrichment.waggr" not in sys.modules
+assert features.read_gmt is scarf.read_gmt
+
+_ = features.EnrichmentResult
+assert "scarf.features.enrichment.results" in sys.modules
+assert "scarf.features.enrichment.aucell" not in sys.modules
+assert "scarf.features.enrichment.waggr" not in sys.modules
+
+_ = features.find_markers_by_rank
+assert "scarf.features.markers.search" in sys.modules
+assert "scarf.features.markers.batching" not in sys.modules
+
+_ = features.GffReader
+assert "scarf.features.genomic.gff" in sys.modules
+assert "scarf.features.genomic.melding" not in sys.modules
+""",
+ ],
+ check=True,
+ )
+
+
+def test_metrics_and_merge_do_not_import_datastore_at_runtime():
+ for module_name in ("scarf.metrics", "scarf.merge"):
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ (
+ f"import {module_name}; import sys; "
+ "assert 'scarf.datastore.datastore' not in sys.modules"
+ ),
+ ],
+ check=True,
+ )
+
+
+def test_merge_implementations_are_runtime_isolated():
+ merge_root = _SCARF_ROOT / "merge"
+ assert merge_root.is_dir()
+ assert not (_SCARF_ROOT / "merge.py").exists()
+ required_files = {"__init__.py", "assays.py", "datasets.py"}
+ assert {path.name for path in merge_root.glob("*.py")} == required_files
+ assert (
+ _runtime_import_modules(
+ merge_root / "__init__.py",
+ include_function_local=False,
+ )
+ == set()
+ )
+
+ forbidden_roots = {"datastore", "mapping", "plotting", "readers", "writers"}
+ expected_merge_edges = {
+ "assays.py": {"merge.controlled_compute"},
+ "datasets.py": {
+ "merge.AssayMerge",
+ "merge.DummyAssay",
+ "merge.assays",
+ },
+ }
+ for file_name, expected_edges in expected_merge_edges.items():
+ path = merge_root / file_name
+ runtime_imports = _runtime_import_modules(path)
+ assert not {
+ module_name
+ for module_name in runtime_imports
+ if module_name.split(".", 1)[0] in forbidden_roots
+ }
+ merge_edges = {
+ module_name
+ for module_name in runtime_imports
+ if module_name == "merge" or module_name.startswith("merge.")
+ }
+ assert merge_edges == expected_edges
+
+ module_scope_imports = _runtime_import_modules(
+ path,
+ include_function_local=False,
+ )
+ function_local_merge_edges = merge_edges - module_scope_imports
+ if file_name == "assays.py":
+ assert function_local_merge_edges == {"merge.controlled_compute"}
+ else:
+ assert function_local_merge_edges == {
+ "merge.AssayMerge",
+ "merge.DummyAssay",
+ }
+
+
+def test_mapping_does_not_import_datastore_at_runtime():
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ (
+ "import scarf.mapping; import sys; "
+ "assert 'scarf.datastore.datastore' not in sys.modules"
+ ),
+ ],
+ check=True,
+ )
+
+
+def test_reader_implementations_are_runtime_isolated():
+ readers_root = _SCARF_ROOT / "readers"
+ assert readers_root.is_dir()
+ assert not (_SCARF_ROOT / "readers.py").exists()
+ required_files = {
+ "__init__.py",
+ "_text.py",
+ "cellranger.py",
+ "csv.py",
+ "h5ad.py",
+ "loom.py",
+ "mtx.py",
+ "seurat.py",
+ }
+ assert required_files.issubset(path.name for path in readers_root.glob("*.py"))
+ assert _runtime_import_modules(
+ readers_root / "__init__.py",
+ include_function_local=False,
+ ) == {"readers._text"}
+
+ forbidden_roots = {"datastore", "merge", "plotting", "storage", "writers"}
+ format_modules = {
+ "readers.cellranger",
+ "readers.csv",
+ "readers.h5ad",
+ "readers.loom",
+ "readers.mtx",
+ "readers.seurat",
+ }
+ reader_edges = {
+ "readers.cellranger": {"readers.mtx"},
+ "readers.mtx": {"readers.cellranger"},
+ "readers.seurat": {"readers._rds", "readers._seurat"},
+ }
+ format_names = {name.rsplit(".", 1)[-1] for name in format_modules}
+ implementation_paths = [
+ readers_root / name for name in sorted(required_files - {"__init__.py"})
+ ]
+ for path in implementation_paths:
+ tree = ast.parse(path.read_text(), filename=str(path))
+ relative_sibling_imports = {
+ alias.name
+ for node in ast.walk(tree)
+ if isinstance(node, ast.ImportFrom)
+ and node.level == 1
+ and node.module is None
+ for alias in node.names
+ }
+ assert relative_sibling_imports.isdisjoint(format_names)
+
+ runtime_imports = _runtime_import_modules(path)
+ assert not {
+ module_name
+ for module_name in runtime_imports
+ if module_name.split(".", 1)[0] in forbidden_roots
+ }
+
+ current_module = f"readers.{path.stem}"
+ if current_module not in format_modules:
+ assert {
+ module_name
+ for module_name in runtime_imports
+ if module_name.startswith("readers")
+ } <= {"readers.get_file_handle"}
+ continue
+ allowed_edges = reader_edges.get(current_module, set())
+ sibling_modules = format_modules - {current_module} - allowed_edges
+ assert runtime_imports.isdisjoint(sibling_modules)
+
+ allowed_reader_imports: set[str] = set(allowed_edges)
+ if path.name == "cellranger.py":
+ allowed_reader_imports.update({"readers._assay_names", "readers.read_file"})
+ elif path.name == "h5ad.py":
+ allowed_reader_imports.update(
+ {"readers._assay_names", "readers._h5ad_inspect"}
+ )
+ assert not {
+ module_name
+ for module_name in runtime_imports
+ if module_name.startswith("readers")
+ and module_name not in allowed_reader_imports
+ }
+
+
+def test_writer_implementations_are_runtime_isolated():
+ writers_root = _SCARF_ROOT / "writers"
+ assert writers_root.is_dir()
+ assert not (_SCARF_ROOT / "writers.py").exists()
+ required_files = {
+ "__init__.py",
+ "_materialize.py",
+ "_store.py",
+ "cellranger.py",
+ "csv.py",
+ "export.py",
+ "h5ad.py",
+ "loom.py",
+ "sparse.py",
+ "subset.py",
+ "seurat.py",
+ }
+ assert {path.name for path in writers_root.glob("*.py")} == required_files
+ assert (
+ _runtime_import_modules(
+ writers_root / "__init__.py",
+ include_function_local=False,
+ )
+ == set()
+ )
+
+ forbidden_roots = {"assay", "datastore", "mapping", "merge", "plotting"}
+ format_modules = {
+ "writers.cellranger",
+ "writers.csv",
+ "writers.h5ad",
+ "writers.loom",
+ "writers.sparse",
+ "writers.subset",
+ "writers.seurat",
+ }
+ format_names = {name.rsplit(".", 1)[-1] for name in format_modules}
+ facade_edges = {
+ "writers.create_cell_data",
+ "writers.create_zarr_count_assay",
+ "writers.create_zarr_obj_array",
+ "writers.load_count_store",
+ "writers.load_zarr",
+ }
+ shared_edges = {"writers._materialize", "writers._store"}
+ matching_reader_exports = {
+ "cellranger.py": {"CrReader"},
+ "csv.py": {"CSVReader"},
+ "h5ad.py": {"H5adReader"},
+ "loom.py": {"LoomReader"},
+ "seurat.py": {"SeuratReader"},
+ }
+
+ for path in writers_root.glob("*.py"):
+ if path.name == "__init__.py":
+ continue
+ tree = ast.parse(path.read_text(), filename=str(path))
+ relative_sibling_imports = {
+ node.module
+ for node in ast.walk(tree)
+ if isinstance(node, ast.ImportFrom)
+ and node.level == 1
+ and node.module in format_names
+ }
+ assert relative_sibling_imports == set()
+
+ runtime_imports = _runtime_import_modules(path)
+ assert not {
+ module_name
+ for module_name in runtime_imports
+ if module_name.split(".", 1)[0] in forbidden_roots
+ }
+ writer_edges = {
+ module_name
+ for module_name in runtime_imports
+ if module_name == "writers" or module_name.startswith("writers.")
+ }
+ assert writer_edges <= shared_edges | facade_edges
+
+ reader_imports = {
+ node
+ for node in ast.walk(tree)
+ if isinstance(node, ast.ImportFrom)
+ and _resolved_module(path, node) == "readers"
+ }
+ imported_reader_exports = {
+ alias.name for node in reader_imports for alias in node.names
+ }
+ assert imported_reader_exports <= matching_reader_exports.get(path.name, set())
+ assert not {
+ module_name
+ for module_name in runtime_imports
+ if module_name.startswith("readers.")
+ and module_name != f"readers.{path.stem}"
+ }
+
+
+def test_assay_implementations_are_runtime_isolated():
+ assay_root = _SCARF_ROOT / "assay"
+ assert assay_root.is_dir()
+ assert not (_SCARF_ROOT / "assay.py").exists()
+
+ forbidden_roots = {"datastore", "merge", "plotting", "readers", "writers"}
+ modality_modules = {"assay.adt", "assay.atac", "assay.rna"}
+ modality_names = {name.rsplit(".", 1)[-1] for name in modality_modules}
+ for path in assay_root.glob("*.py"):
+ if path.name == "__init__.py":
+ continue
+ tree = ast.parse(path.read_text(), filename=str(path))
+ relative_sibling_imports = {
+ alias.name
+ for node in ast.walk(tree)
+ if isinstance(node, ast.ImportFrom)
+ and node.level == 1
+ and node.module is None
+ for alias in node.names
+ }
+ assert relative_sibling_imports.isdisjoint(modality_names)
+
+ module_scope_imports = _runtime_import_modules(
+ path,
+ include_function_local=False,
+ )
+ assert not {
+ module_name
+ for module_name in module_scope_imports
+ if module_name.split(".", 1)[0] in forbidden_roots
+ }
+ current_module = f"assay.{path.stem}"
+ sibling_modules = modality_modules - {current_module}
+ assert module_scope_imports.isdisjoint(sibling_modules)
+
+ allowed_function_local = {"plotting"}
+ if path.name == "base.py":
+ allowed_function_local.add("assay.rna")
+ function_local_imports = (
+ _runtime_import_modules(path)
+ - module_scope_imports
+ - allowed_function_local
+ )
+ assert not {
+ module_name
+ for module_name in function_local_imports
+ if module_name.split(".", 1)[0] in forbidden_roots
+ }
+ assert function_local_imports.isdisjoint(sibling_modules)
+
+
+def test_datastore_operation_mixins_are_runtime_isolated():
+ from scarf.datastore._operations.clustering import _ClusteringOperationsMixin
+ from scarf.datastore._operations.embeddings import _EmbeddingOperationsMixin
+ from scarf.datastore._operations.features import _FeatureOperationsMixin
+ from scarf.datastore._operations.graph import _GraphOperationsMixin
+ from scarf.datastore._operations.integration_metrics import (
+ _IntegrationMetricsOperationsMixin,
+ )
+ from scarf.datastore._operations.mapping import _MappingOperationsMixin
+ from scarf.datastore._operations.mapping_reference import (
+ _MappingReferenceOperationsMixin,
+ )
+ from scarf.datastore._operations.presentation import _PresentationOperationsMixin
+ from scarf.datastore._operations.quality_control import (
+ _QualityControlOperationsMixin,
+ )
+ from scarf.datastore._operations.trajectory import (
+ _TrajectoryFeatureOperationsMixin,
+ _TrajectoryOperationsMixin,
+ )
+
+ operations_root = _SCARF_ROOT / "datastore" / "_operations"
+ facade_modules = {
+ "datastore.base_datastore",
+ "datastore.datastore",
+ "datastore.graph_datastore",
+ "datastore.mapping_datastore",
+ }
+ allowed_operation_helpers = {
+ "datastore._operations.enrichment_store",
+ "datastore._operations.paris_persistence",
+ }
+ for path in operations_root.glob("*.py"):
+ runtime_imports = _runtime_import_modules(path)
+ assert runtime_imports.isdisjoint(facade_modules)
+ operation_imports = {
+ module_name
+ for module_name in runtime_imports
+ if module_name == "datastore._operations"
+ or module_name.startswith("datastore._operations.")
+ }
+ assert operation_imports.issubset(allowed_operation_helpers)
+
+ mixins = (
+ _EmbeddingOperationsMixin,
+ _ClusteringOperationsMixin,
+ _TrajectoryOperationsMixin,
+ _MappingReferenceOperationsMixin,
+ _GraphOperationsMixin,
+ _MappingOperationsMixin,
+ _QualityControlOperationsMixin,
+ _FeatureOperationsMixin,
+ _TrajectoryFeatureOperationsMixin,
+ _IntegrationMetricsOperationsMixin,
+ _PresentationOperationsMixin,
+ )
+ assert all("__init__" not in mixin.__dict__ for mixin in mixins)
+ assert all(mixin.__bases__ == (object,) for mixin in mixins)
+
+
+def test_assay_graph_path_grammar_is_centralized():
+ encoded_paths = _SCARF_ROOT / "graph" / "encoded_paths.py"
+ path_markers = ("normed__", "reduction__", "ann__", "knn__", "graph__")
+ pointer_names = {
+ "latest_reduction",
+ "latest_ann",
+ "latest_knn",
+ "latest_graph",
+ "latest_kmeans",
+ }
+ violations: list[tuple[str, int, str]] = []
+
+ for path in _SCARF_ROOT.rglob("*.py"):
+ if path == encoded_paths:
+ continue
+ tree = ast.parse(path.read_text(), filename=str(path))
+ docstring_constants = {
+ id(owner.body[0].value)
+ for owner in ast.walk(tree)
+ if isinstance(
+ owner,
+ ast.Module | ast.ClassDef | ast.FunctionDef | ast.AsyncFunctionDef,
+ )
+ and owner.body
+ and isinstance(owner.body[0], ast.Expr)
+ and isinstance(owner.body[0].value, ast.Constant)
+ and isinstance(owner.body[0].value.value, str)
+ }
+ for node in ast.walk(tree):
+ if (
+ isinstance(node, ast.Constant)
+ and isinstance(node.value, str)
+ and id(node) not in docstring_constants
+ and any(marker in node.value for marker in path_markers)
+ ):
+ violations.append(
+ (
+ path.relative_to(_SCARF_ROOT).as_posix(),
+ node.lineno,
+ "path construction",
+ )
+ )
+ elif (
+ isinstance(node, ast.Subscript)
+ and isinstance(node.ctx, ast.Load)
+ and isinstance(node.slice, ast.Constant)
+ and node.slice.value in pointer_names
+ ):
+ violations.append(
+ (
+ path.relative_to(_SCARF_ROOT).as_posix(),
+ node.lineno,
+ "latest pointer read",
+ )
+ )
+ elif (
+ isinstance(node, ast.Call)
+ and isinstance(node.func, ast.Attribute)
+ and node.func.attr == "get"
+ and node.args
+ and isinstance(node.args[0], ast.Constant)
+ and node.args[0].value in pointer_names
+ ):
+ violations.append(
+ (
+ path.relative_to(_SCARF_ROOT).as_posix(),
+ node.lineno,
+ "latest pointer read",
+ )
+ )
+
+ assert violations == []
diff --git a/tests/test_import_batching.py b/tests/test_import_batching.py
new file mode 100644
index 00000000..981b7c4b
--- /dev/null
+++ b/tests/test_import_batching.py
@@ -0,0 +1,526 @@
+from pathlib import Path
+
+import h5py
+import numpy as np
+import pandas as pd
+import pytest
+import zarr
+from scipy.sparse import coo_matrix, csc_matrix, csr_matrix
+from zarr.storage import MemoryStore
+
+from scarf.readers import CrH5Reader, H5adReader
+from scarf.storage.budget import ResourceBudget
+from scarf.storage.layout import array_shard_rows
+from scarf.storage.sharding import (
+ resolve_sparse_import_batch,
+ sparse_write_task_count,
+)
+from scarf.writers import CrToZarr, H5adToZarr, SparseToZarr
+
+
+def _planner_destinations() -> tuple[zarr.Array, zarr.Array]:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ first = root.create_array(
+ "first",
+ shape=(12, 4),
+ chunks=(2, 4),
+ shards=(4, 4),
+ dtype=np.uint32,
+ )
+ second = root.create_array(
+ "second",
+ shape=(12, 2),
+ chunks=(3, 2),
+ shards=(6, 2),
+ dtype=np.uint32,
+ )
+ return first, second
+
+
+def test_sparse_import_planner_prefers_geometry_and_shrinks_monotonically() -> None:
+ destinations = _planner_destinations()
+ large = resolve_sparse_import_batch(
+ destinations,
+ nRows=12,
+ resources=ResourceBudget(15_000, 2),
+ maxWindowNnz=lambda rows: min(rows, 12) * 4,
+ sourceDtype=np.uint32,
+ producerStagingBytes=lambda rows: rows * 1_000,
+ )
+ small = resolve_sparse_import_batch(
+ destinations,
+ nRows=12,
+ resources=ResourceBudget(10_000, 2),
+ maxWindowNnz=lambda rows: min(rows, 12) * 4,
+ sourceDtype=np.uint32,
+ producerStagingBytes=lambda rows: rows * 1_000,
+ )
+ wider = resolve_sparse_import_batch(
+ destinations,
+ nRows=12,
+ resources=ResourceBudget(15_000, 2),
+ maxWindowNnz=lambda rows: min(rows, 12) * 8,
+ sourceDtype=np.uint32,
+ producerStagingBytes=lambda rows: rows * 1_000,
+ )
+
+ assert large.batchRows == 4
+ assert large.producerReserveBytes == 8_272
+ assert large.writeTasks == 5
+ assert small.batchRows == 1
+ assert small.producerReserveBytes < large.producerReserveBytes
+ assert wider.batchRows < large.batchRows
+
+
+def test_sparse_import_planner_preserves_override_and_fails_one_band() -> None:
+ destinations = _planner_destinations()
+ explicit = resolve_sparse_import_batch(
+ destinations,
+ nRows=12,
+ resources=ResourceBudget(1_000_000, 2),
+ maxWindowNnz=lambda rows: min(rows, 12) * 4,
+ sourceDtype=np.uint32,
+ batchRows=7,
+ )
+ assert explicit.batchRows == 7
+ with pytest.raises(ValueError, match="batch_size must be positive"):
+ resolve_sparse_import_batch(
+ destinations,
+ nRows=12,
+ resources=ResourceBudget(1_000_000, 2),
+ maxWindowNnz=lambda rows: min(rows, 12) * 4,
+ sourceDtype=np.uint32,
+ batchRows=0,
+ )
+ with pytest.raises(MemoryError, match="one source row"):
+ resolve_sparse_import_batch(
+ destinations,
+ nRows=12,
+ resources=ResourceBudget(1_000, 1),
+ maxWindowNnz=lambda rows: min(rows, 12) * 4,
+ sourceDtype=np.uint32,
+ )
+
+
+class _PlanningReader:
+ def __init__(self, values: np.ndarray) -> None:
+ self.values = values
+ self.nCells, self.nFeatures = values.shape
+ self.matrix_dtype = values.dtype
+ self.assayFeats = pd.DataFrame(
+ {
+ "RNA": [
+ "Gene Expression",
+ 0,
+ self.nFeatures,
+ self.nFeatures,
+ ]
+ },
+ index=["type", "start", "end", "nFeatures"],
+ )
+ self.consumed: list[int] = []
+ self.windowRequests: list[int] = []
+
+ def cell_names(self) -> list[str]:
+ return [f"cell-{index}" for index in range(self.nCells)]
+
+ def feature_ids(self, assay: str) -> list[str]:
+ return [f"feature-{index}" for index in range(self.nFeatures)]
+
+ def feature_names(self, assay: str) -> list[str]:
+ return [f"gene-{index}" for index in range(self.nFeatures)]
+
+ def consume(self, batch_size: int, lines_in_mem: int):
+ self.consumed.append(batch_size)
+ for start in range(0, self.nCells, batch_size):
+ yield coo_matrix(self.values[start : start + batch_size])
+
+ def max_window_nnz(self, window_rows: int) -> int:
+ self.windowRequests.append(window_rows)
+ width = min(window_rows, self.nCells)
+ return max(
+ np.count_nonzero(self.values[start : start + width])
+ for start in range(self.nCells - width + 1)
+ )
+
+ def producer_staging_bytes(self, batch_size: int, lines_in_mem: int) -> int:
+ return 0
+
+
+def test_crtozarr_automatic_rows_and_preflight_before_consume(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ from scarf.storage import sharding
+
+ original_write = sharding.write_sparse_bands
+ observed: dict[str, int] = {}
+
+ def recording_write(writes, **kwargs):
+ count = 0
+
+ def recording_source():
+ nonlocal count
+ for write in writes:
+ count += 1
+ yield write
+
+ original_write(recording_source(), **kwargs)
+ observed["writes"] = count
+ observed["total"] = int(kwargs["total"])
+
+ monkeypatch.setattr(sharding, "write_sparse_bands", recording_write)
+ values = np.eye(8, 4, dtype=np.uint16)
+ reader = _PlanningReader(values)
+ writer = CrToZarr(
+ reader,
+ MemoryStore(),
+ dtype="uint16",
+ mem_budget="64M",
+ targetChunkBytes=16,
+ targetShardBytes=32,
+ )
+ writer.dump()
+
+ assert reader.consumed == [4]
+ assert writer._lastImportPlan.batchRows == 4
+ assert 4 in reader.windowRequests
+ assert observed == {"writes": 2, "total": 2}
+ assert writer._lastImportPlan.writeTasks == 2
+
+ rejected = _PlanningReader(values)
+ writer = CrToZarr(
+ rejected,
+ MemoryStore(),
+ dtype="uint16",
+ mem_budget=1,
+ targetChunkBytes=16,
+ targetShardBytes=32,
+ )
+ with pytest.raises(MemoryError, match="one source row"):
+ writer.dump()
+ assert rejected.consumed == []
+
+
+class _TrackingCsr(csr_matrix):
+ def __init__(self, values: np.ndarray) -> None:
+ super().__init__(values)
+ self.sliceCount = 0
+
+ def __getitem__(self, key):
+ self.sliceCount += 1
+ return super().__getitem__(key)
+
+
+def test_h5ad_and_sparse_preflight_before_source_iteration(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ values = np.eye(8, 4, dtype=np.uint16)
+ path = tmp_path / "preflight.h5ad"
+ _write_h5ad(path, values, "csr")
+ reader = H5adReader(str(path), feature_name_key="feature_name")
+ h5ad_writer = H5adToZarr(
+ reader,
+ MemoryStore(),
+ mem_budget="64M",
+ targetChunkBytes=16,
+ targetShardBytes=32,
+ )
+ consumed = False
+
+ def unexpected_consume(batch_size: int):
+ nonlocal consumed
+ consumed = True
+ yield coo_matrix(values[:batch_size])
+
+ monkeypatch.setattr(reader, "consume", unexpected_consume)
+ h5ad_writer.resources = ResourceBudget(1, 1)
+ try:
+ with pytest.raises(MemoryError, match="one source row"):
+ h5ad_writer.dump()
+ finally:
+ reader.h5.close()
+ assert not consumed
+
+ tracked = _TrackingCsr(values)
+ sparse_writer = SparseToZarr(
+ tracked,
+ MemoryStore(),
+ cell_ids=[f"cell-{index}" for index in range(values.shape[0])],
+ feature_ids=[f"feature-{index}" for index in range(values.shape[1])],
+ mem_budget="64M",
+ targetChunkBytes=16,
+ targetShardBytes=32,
+ )
+ sparse_writer.resources = ResourceBudget(1, 1)
+ with pytest.raises(MemoryError, match="one source row"):
+ sparse_writer.dump()
+ assert sparse_writer.mat.sliceCount == 0
+
+
+def _write_cr_h5(path: Path, values: np.ndarray) -> None:
+ matrix = csr_matrix(values)
+ with h5py.File(path, mode="w") as handle:
+ group = handle.create_group("matrix")
+ group.create_dataset("data", data=matrix.data)
+ group.create_dataset("indices", data=matrix.indices)
+ group.create_dataset("indptr", data=matrix.indptr)
+ group.create_dataset(
+ "barcodes",
+ data=np.array(
+ [f"cell-{index}".encode() for index in range(values.shape[0])]
+ ),
+ )
+ features = group.create_group("features")
+ features.create_dataset(
+ "id",
+ data=np.array(
+ [f"feature-{index}".encode() for index in range(values.shape[1])]
+ ),
+ )
+ features.create_dataset(
+ "name",
+ data=np.array(
+ [f"gene-{index}".encode() for index in range(values.shape[1])]
+ ),
+ )
+ features.create_dataset(
+ "feature_type",
+ data=np.array(
+ [
+ (
+ "Gene Expression" if index % 2 == 0 else "Antibody Capture"
+ ).encode()
+ for index in range(values.shape[1])
+ ]
+ ),
+ )
+
+
+def _write_h5ad(path: Path, values: np.ndarray, encoding: str) -> None:
+ matrix = csr_matrix(values) if encoding == "csr" else csc_matrix(values)
+ with h5py.File(path, mode="w") as handle:
+ group = handle.create_group("X")
+ group.attrs["encoding-type"] = f"{encoding}_matrix"
+ group.attrs["shape"] = values.shape
+ group.create_dataset("data", data=matrix.data)
+ group.create_dataset("indices", data=matrix.indices)
+ group.create_dataset("indptr", data=matrix.indptr)
+ obs = handle.create_group("obs")
+ obs.create_dataset(
+ "_index",
+ data=np.array(
+ [f"cell-{index}".encode() for index in range(values.shape[0])]
+ ),
+ )
+ var = handle.create_group("var")
+ var.create_dataset(
+ "_index",
+ data=np.array(
+ [f"feature-{index}".encode() for index in range(values.shape[1])]
+ ),
+ )
+ var.create_dataset(
+ "feature_name",
+ data=np.array(
+ [f"gene-{index}".encode() for index in range(values.shape[1])]
+ ),
+ )
+ var.create_dataset(
+ "feature_types",
+ data=np.array(
+ [
+ (
+ "Gene Expression" if index % 2 == 0 else "Antibody Capture"
+ ).encode()
+ for index in range(values.shape[1])
+ ]
+ ),
+ )
+ handle.create_group("obsm")
+
+
+def _assay_counts(store: MemoryStore) -> dict[str, np.ndarray]:
+ root = zarr.open_group(store=store, mode="r")
+ return {name: np.asarray(root[f"{name}/counts"][:]) for name in ("RNA", "ADT")}
+
+
+def _count_arrays(store: MemoryStore) -> tuple[zarr.Array, zarr.Array]:
+ root = zarr.open_group(store=store, mode="r")
+ return root["RNA/counts"], root["ADT/counts"]
+
+
+def test_cellranger_h5_automatic_matches_explicit_and_caches_planning(
+ tmp_path: Path,
+) -> None:
+ values = np.arange(48, dtype=np.uint16).reshape(8, 6) % 5
+ path = tmp_path / "counts.h5"
+ _write_cr_h5(path, values)
+ stores = [MemoryStore(), MemoryStore()]
+ readers = [CrH5Reader(str(path)), CrH5Reader(str(path))]
+ try:
+ automatic = CrToZarr(
+ readers[0],
+ stores[0],
+ dtype="uint16",
+ mem_budget="64M",
+ targetChunkBytes=24,
+ targetShardBytes=48,
+ )
+ explicit = CrToZarr(
+ readers[1],
+ stores[1],
+ dtype="uint16",
+ mem_budget="64M",
+ targetChunkBytes=24,
+ targetShardBytes=48,
+ )
+ automatic.dump()
+ explicit.dump(batch_size=3)
+ cached = readers[0]._indptrCache
+ assert cached is not None
+ readers[0].max_window_nnz(2)
+ assert readers[0]._indptrCache is cached
+ automatic_arrays = _count_arrays(stores[0])
+ explicit_arrays = _count_arrays(stores[1])
+ assert automatic._lastImportPlan.batchRows == min(
+ array_shard_rows(array) for array in automatic_arrays
+ )
+ assert automatic._lastImportPlan.writeTasks == sparse_write_task_count(
+ automatic_arrays,
+ values.shape[0],
+ )
+ without_projection = resolve_sparse_import_batch(
+ automatic_arrays,
+ nRows=values.shape[0],
+ resources=automatic.resources,
+ maxWindowNnz=readers[0].max_window_nnz,
+ sourceDtype=readers[0].matrix_dtype,
+ batchRows=automatic._lastImportPlan.batchRows,
+ producerStagingBytes=lambda rows: readers[0].producer_staging_bytes(
+ rows,
+ 100_000,
+ ),
+ )
+ assert (
+ automatic._lastImportPlan.producerReserveBytes
+ > without_projection.producerReserveBytes
+ )
+ assert explicit._lastImportPlan.batchRows == 3
+ for automatic_array, explicit_array in zip(
+ automatic_arrays,
+ explicit_arrays,
+ strict=True,
+ ):
+ assert automatic_array.dtype == explicit_array.dtype == np.dtype(np.uint16)
+ assert automatic_array.chunks == explicit_array.chunks
+ assert array_shard_rows(automatic_array) == array_shard_rows(explicit_array)
+ finally:
+ for reader in readers:
+ reader.close()
+
+ for name, expected in _assay_counts(stores[1]).items():
+ np.testing.assert_array_equal(_assay_counts(stores[0])[name], expected)
+
+
+@pytest.mark.parametrize("encoding", ["csr", "csc"])
+def test_h5ad_automatic_matches_explicit_for_split_assays(
+ tmp_path: Path,
+ encoding: str,
+) -> None:
+ values = np.arange(48, dtype=np.uint16).reshape(8, 6) % 7
+ path = tmp_path / f"{encoding}.h5ad"
+ _write_h5ad(path, values, encoding)
+ stores = [MemoryStore(), MemoryStore()]
+ readers = [
+ H5adReader(str(path), feature_name_key="feature_name"),
+ H5adReader(str(path), feature_name_key="feature_name"),
+ ]
+ try:
+ automatic = H5adToZarr(
+ readers[0],
+ stores[0],
+ assay_split_key="feature_types",
+ mem_budget="64M",
+ targetChunkBytes=24,
+ targetShardBytes=48,
+ )
+ explicit = H5adToZarr(
+ readers[1],
+ stores[1],
+ assay_split_key="feature_types",
+ mem_budget="64M",
+ targetChunkBytes=24,
+ targetShardBytes=48,
+ )
+ automatic.dump()
+ explicit.dump(batch_size=3)
+ automatic_arrays = _count_arrays(stores[0])
+ explicit_arrays = _count_arrays(stores[1])
+ assert automatic._lastImportPlan.batchRows == min(
+ array_shard_rows(array) for array in automatic_arrays
+ )
+ assert automatic._lastImportPlan.writeTasks == sparse_write_task_count(
+ automatic_arrays,
+ values.shape[0],
+ )
+ without_projection = resolve_sparse_import_batch(
+ automatic_arrays,
+ nRows=values.shape[0],
+ resources=automatic.resources,
+ maxWindowNnz=readers[0].max_batch_nnz,
+ sourceDtype=readers[0].sourceMatrixDtype,
+ batchRows=automatic._lastImportPlan.batchRows,
+ producerStagingBytes=readers[0].producer_batch_staging_bytes,
+ )
+ assert (
+ automatic._lastImportPlan.producerReserveBytes
+ > without_projection.producerReserveBytes
+ )
+ assert explicit._lastImportPlan.batchRows == 3
+ for automatic_array, explicit_array in zip(
+ automatic_arrays,
+ explicit_arrays,
+ strict=True,
+ ):
+ assert automatic_array.dtype == explicit_array.dtype == np.dtype(np.uint16)
+ assert automatic_array.chunks == explicit_array.chunks
+ assert array_shard_rows(automatic_array) == array_shard_rows(explicit_array)
+ finally:
+ for reader in readers:
+ reader.h5.close()
+
+ for name, expected in _assay_counts(stores[1]).items():
+ np.testing.assert_array_equal(_assay_counts(stores[0])[name], expected)
+
+
+def test_sparse_automatic_matches_explicit() -> None:
+ values = csr_matrix(np.arange(48, dtype=np.uint16).reshape(8, 6) % 5)
+ stores = [MemoryStore(), MemoryStore()]
+ writers = [
+ SparseToZarr(
+ values,
+ store,
+ cell_ids=[f"cell-{index}" for index in range(values.shape[0])],
+ feature_ids=[f"feature-{index}" for index in range(values.shape[1])],
+ mem_budget="64M",
+ targetChunkBytes=24,
+ targetShardBytes=48,
+ )
+ for store in stores
+ ]
+ writers[0].dump()
+ writers[1].dump(batch_size=3)
+
+ automatic_array = zarr.open_group(store=stores[0], mode="r")["RNA/counts"]
+ explicit_array = zarr.open_group(store=stores[1], mode="r")["RNA/counts"]
+ assert writers[0]._lastImportPlan.batchRows == array_shard_rows(automatic_array)
+ assert writers[0]._lastImportPlan.writeTasks == sparse_write_task_count(
+ (automatic_array,),
+ values.shape[0],
+ )
+ assert writers[1]._lastImportPlan.batchRows == 3
+ assert automatic_array.dtype == explicit_array.dtype == np.dtype(np.uint16)
+ assert automatic_array.chunks == explicit_array.chunks
+ assert array_shard_rows(automatic_array) == array_shard_rows(explicit_array)
+ np.testing.assert_array_equal(automatic_array[:], explicit_array[:])
diff --git a/tests/test_imported_coordinates.py b/tests/test_imported_coordinates.py
new file mode 100644
index 00000000..4f097280
--- /dev/null
+++ b/tests/test_imported_coordinates.py
@@ -0,0 +1,415 @@
+import hashlib
+from typing import Any
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.datastore.datastore import DataStore
+from scarf.embeddings.imported import (
+ validate_imported_embedding_artifact,
+ write_imported_coordinates,
+ write_imported_embedding,
+)
+from scarf.graph.state import (
+ ArtifactSelectionError,
+ read_assay_state,
+ validate_imported_coordinates_artifact,
+ validate_neighbors_artifact_selection,
+)
+from scarf.storage.artifacts import (
+ artifact_group,
+ artifact_path,
+ fingerprint_array,
+ fingerprint_strings,
+ list_artifacts,
+)
+from scarf.storage.budget import ResourceBudget
+from scarf.storage.refs import ArtifactRef
+from scarf.storage.selections import resolve_selection_artifact
+
+_SOURCE_DIGEST = hashlib.sha256(b"source-rds").digest()
+
+
+def _root_with_selection(
+ mask: np.ndarray | None = None,
+) -> tuple[zarr.Group, ArtifactRef, np.ndarray, np.ndarray]:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ root.create_group("RNA")
+ cell_ids = np.array([f"cell_{index}" for index in range(8)])
+ selection = (
+ np.ones(len(cell_ids), dtype=bool)
+ if mask is None
+ else np.asarray(mask, dtype=bool)
+ )
+ cell_data = root.create_group("cellData")
+ cell_data.create_array("ids", data=cell_ids)
+ cell_data.create_array("names", data=cell_ids)
+ cell_data.create_array("I", data=selection)
+ ref = resolve_selection_artifact(
+ root,
+ scope="datastore",
+ kind="cell_selection",
+ values=selection,
+ row_ids=cell_ids,
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column="I",
+ )
+ return root, ref, cell_ids, selection
+
+
+def _fingerprints(
+ *,
+ data: np.ndarray,
+ loadings: np.ndarray | None = None,
+ feature_ids: np.ndarray | None = None,
+ stdev: np.ndarray | None = None,
+) -> dict[str, str]:
+ values = {"data": fingerprint_array(data)}
+ if loadings is not None:
+ values["loadings"] = fingerprint_array(loadings)
+ if feature_ids is not None:
+ values["feature_ids"] = fingerprint_strings(feature_ids)
+ if stdev is not None:
+ values["stdev"] = fingerprint_array(stdev)
+ return values
+
+
+def _graph_store(root: zarr.Group) -> DataStore:
+ store = DataStore.__new__(DataStore)
+ store.z = root
+ store.workspace = None
+ store.zarr_mode = "r+"
+ store.storageProfile = "fast_local"
+ store.resources = ResourceBudget(memoryBytes=64 * 1024 * 1024, workers=1)
+ store.nthreads = 1
+ store._defaultAssay = "RNA"
+ return store
+
+
+def test_imported_coordinates_write_blockwise_with_honest_provenance() -> None:
+ root, selection, cell_ids, mask = _root_with_selection()
+ selected_ids = cell_ids[mask]
+ coordinates = np.arange(24, dtype=np.float32).reshape(8, 3)
+ loadings = np.arange(15, dtype=np.float64).reshape(5, 3)
+ feature_ids = np.array([f"gene_{index}" for index in range(5)])
+ stdev = np.array([3.0, 2.0, 1.0], dtype=np.float64)
+ pulls: list[int] = []
+
+ def coordinate_blocks():
+ for start in range(0, len(coordinates), 3):
+ pulls.append(start)
+ yield coordinates[start : start + 3]
+
+ ref = write_imported_coordinates(
+ root,
+ assay="RNA",
+ dimreduc_key="pca",
+ role="pca",
+ coordinates=coordinate_blocks,
+ coordinate_shape=coordinates.shape,
+ coordinate_dtype=coordinates.dtype,
+ source_digest=_SOURCE_DIGEST,
+ payload_fingerprints=_fingerprints(
+ data=coordinates,
+ loadings=loadings,
+ feature_ids=feature_ids,
+ stdev=stdev,
+ ),
+ source_cell_ids=selected_ids,
+ cell_selection=selection,
+ cell_key="I",
+ loadings=loadings,
+ feature_ids=feature_ids,
+ stdev=stdev,
+ named_result="seurat_pca",
+ feat_key="I",
+ block_rows=2,
+ )
+
+ assert pulls == [0, 3, 6]
+ assert ref.kind == "imported_coordinates"
+ status = root[artifact_path(ref)].attrs["provenance"]
+ assert status["operation"] == "import_dimreduc"
+ assert status["inputs"]["source_digest"] == {"bytes_hex": _SOURCE_DIGEST.hex()}
+ assert status["inputs"]["cell_selection"] == selection.to_dict()
+ assert status["inputs"]["ordered_cell_ids_fingerprint"] == fingerprint_strings(
+ selected_ids
+ )
+ group = artifact_group(root, ref)
+ np.testing.assert_array_equal(group["data"][:], coordinates)
+ np.testing.assert_array_equal(group["loadings"][:], loadings)
+ np.testing.assert_array_equal(group["feature_ids"][:], feature_ids)
+ np.testing.assert_array_equal(group["stdev"][:], stdev)
+ validate_imported_coordinates_artifact(root, ref, cell_key="I")
+ state = read_assay_state(root, "RNA")
+ assert state is not None
+ assert state.reduction is None
+ assert state.named_results["seurat_pca"] == ref
+
+
+def test_imported_coordinates_validate_alignment_before_artifact_creation() -> None:
+ root, selection, cell_ids, mask = _root_with_selection()
+ coordinates = np.arange(24, dtype=np.float32).reshape(8, 3)
+
+ with pytest.raises(ArtifactSelectionError) as caught:
+ write_imported_coordinates(
+ root,
+ assay="RNA",
+ dimreduc_key="pca",
+ role="pca",
+ coordinates=coordinates,
+ source_digest=_SOURCE_DIGEST,
+ payload_fingerprints={"data": fingerprint_array(coordinates)},
+ source_cell_ids=cell_ids[mask][::-1],
+ cell_selection=selection,
+ cell_key="I",
+ )
+
+ assert caught.value.code == "dimreduc_cell_identity_mismatch"
+ assert (
+ list_artifacts(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="imported_coordinates",
+ )
+ == []
+ )
+
+
+def test_imported_coordinate_alignment_reads_cell_ids_in_blocks() -> None:
+ root, selection, cell_ids, mask = _root_with_selection()
+ coordinates = np.arange(24, dtype=np.float32).reshape(8, 3)
+
+ class TrackedIds:
+ shape = cell_ids.shape
+
+ def __init__(self) -> None:
+ self.reads: list[tuple[int, int]] = []
+
+ def __getitem__(self, key: slice) -> np.ndarray:
+ assert isinstance(key, slice)
+ assert key.start is not None and key.stop is not None
+ self.reads.append((key.start, key.stop))
+ return cell_ids[key]
+
+ tracked: Any = TrackedIds()
+ write_imported_coordinates(
+ root,
+ assay="RNA",
+ dimreduc_key="pca",
+ role="pca",
+ coordinates=coordinates,
+ source_digest=_SOURCE_DIGEST,
+ payload_fingerprints={"data": fingerprint_array(coordinates)},
+ source_cell_ids=tracked,
+ cell_selection=selection,
+ cell_key="I",
+ block_rows=3,
+ )
+
+ assert tracked.reads == [(0, 3), (3, 6), (6, 8)]
+
+
+def test_imported_coordinate_loadings_stream_feature_ids_twice() -> None:
+ root, selection, cell_ids, mask = _root_with_selection()
+ coordinates = np.arange(24, dtype=np.float32).reshape(8, 3)
+ loadings = np.arange(18, dtype=np.float32).reshape(6, 3)
+ feature_ids = np.array([f"gene-{index}" for index in range(6)])
+
+ class TrackedFeatureIds:
+ def __init__(self) -> None:
+ self.reads: list[tuple[int, int]] = []
+
+ def __len__(self) -> int:
+ return len(feature_ids)
+
+ def __getitem__(self, key: slice) -> np.ndarray:
+ assert isinstance(key, slice)
+ assert key.start is not None and key.stop is not None
+ self.reads.append((key.start, key.stop))
+ return feature_ids[key]
+
+ tracked: Any = TrackedFeatureIds()
+ ref = write_imported_coordinates(
+ root,
+ assay="RNA",
+ dimreduc_key="pca",
+ role="pca",
+ coordinates=coordinates,
+ source_digest=_SOURCE_DIGEST,
+ payload_fingerprints={
+ "data": fingerprint_array(coordinates),
+ "loadings": fingerprint_array(loadings),
+ "feature_ids": fingerprint_strings(feature_ids),
+ },
+ source_cell_ids=cell_ids[mask],
+ cell_selection=selection,
+ cell_key="I",
+ loadings=loadings,
+ feature_ids=tracked,
+ block_rows=3,
+ )
+
+ assert artifact_group(root, ref)["feature_ids"][:].tolist() == feature_ids.tolist()
+ assert tracked.reads == [(0, 3), (3, 6), (0, 3), (3, 6)]
+
+
+def test_imported_coordinates_require_a_fixed_size_source_digest() -> None:
+ root, selection, cell_ids, mask = _root_with_selection()
+ coordinates = np.arange(24, dtype=np.float32).reshape(8, 3)
+
+ with pytest.raises(TypeError, match="exactly 32 bytes"):
+ write_imported_coordinates(
+ root,
+ assay="RNA",
+ dimreduc_key="pca",
+ role="pca",
+ coordinates=coordinates,
+ source_digest=b"short",
+ payload_fingerprints={"data": fingerprint_array(coordinates)},
+ source_cell_ids=cell_ids[mask],
+ cell_selection=selection,
+ cell_key="I",
+ )
+
+
+def test_imported_coordinates_reject_unstored_payload_fingerprint() -> None:
+ root, selection, cell_ids, mask = _root_with_selection()
+ coordinates = np.arange(24, dtype=np.float32).reshape(8, 3)
+
+ with pytest.raises(ValueError, match="Unexpected payload fingerprints"):
+ write_imported_coordinates(
+ root,
+ assay="RNA",
+ dimreduc_key="pca",
+ role="pca",
+ coordinates=coordinates,
+ source_digest=_SOURCE_DIGEST,
+ payload_fingerprints={
+ "data": fingerprint_array(coordinates),
+ "normalized": fingerprint_array(coordinates),
+ },
+ source_cell_ids=cell_ids[mask],
+ cell_selection=selection,
+ cell_key="I",
+ )
+
+ assert (
+ list_artifacts(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="imported_coordinates",
+ )
+ == []
+ )
+
+
+def test_imported_coordinate_validation_detects_payload_tampering() -> None:
+ root, selection, cell_ids, mask = _root_with_selection()
+ coordinates = np.arange(24, dtype=np.float32).reshape(8, 3)
+ ref = write_imported_coordinates(
+ root,
+ assay="RNA",
+ dimreduc_key="pca",
+ role="pca",
+ coordinates=coordinates,
+ source_digest=_SOURCE_DIGEST,
+ payload_fingerprints={"data": fingerprint_array(coordinates)},
+ source_cell_ids=cell_ids[mask],
+ cell_selection=selection,
+ cell_key="I",
+ )
+
+ artifact_group(root, ref)["data"][0, 0] = -1
+
+ with pytest.raises(ValueError, match="fingerprint.*does not match"):
+ validate_imported_coordinates_artifact(root, ref)
+
+
+def test_imported_embedding_writes_values_and_links_metadata_columns() -> None:
+ mask = np.array([True, False, True, False, True, True, False, True])
+ root, selection, cell_ids, mask = _root_with_selection(mask)
+ coordinates = np.arange(10, dtype=np.float32).reshape(5, 2)
+ ref = write_imported_embedding(
+ root,
+ assay="RNA",
+ dimreduc_key="umap",
+ role="umap",
+ coordinates=(coordinates[:2], coordinates[2:]),
+ coordinate_shape=coordinates.shape,
+ coordinate_dtype=coordinates.dtype,
+ source_digest=_SOURCE_DIGEST,
+ payload_fingerprints={"values": fingerprint_array(coordinates)},
+ source_cell_ids=cell_ids[mask],
+ cell_selection=selection,
+ cell_key="I",
+ block_rows=2,
+ )
+
+ assert ref.kind == "embedding"
+ assert (
+ root[artifact_path(ref)].attrs["provenance"]["operation"] == "import_dimreduc"
+ )
+ validate_imported_embedding_artifact(root, ref, cell_key="I")
+ for index, column_name in enumerate(("RNA_UMAP1", "RNA_UMAP2")):
+ column = root["cellData"][column_name]
+ np.testing.assert_array_equal(column[:][mask], coordinates[:, index])
+ assert np.isnan(column[:][~mask]).all()
+ assert column.attrs["source_artifact"] == ref.to_dict()
+ assert column.attrs["source_value"] == "values"
+ assert column.attrs["value_index"] == index
+
+
+def test_ann_and_neighbor_query_accept_detached_imported_coordinates() -> None:
+ root, selection, cell_ids, mask = _root_with_selection()
+ rng = np.random.default_rng(42)
+ coordinates = rng.normal(size=(8, 3)).astype(np.float32)
+ imported = write_imported_coordinates(
+ root,
+ assay="RNA",
+ dimreduc_key="pca",
+ role="pca",
+ coordinates=coordinates,
+ source_digest=_SOURCE_DIGEST,
+ payload_fingerprints={"data": fingerprint_array(coordinates)},
+ source_cell_ids=cell_ids[mask],
+ cell_selection=selection,
+ cell_key="I",
+ )
+ store = _graph_store(root)
+
+ with pytest.raises(ValueError, match="pass update_state=False"):
+ store.build_ann_index(imported, update_state=True)
+ assert read_assay_state(root, "RNA") is None
+
+ ann = store.build_ann_index(
+ imported,
+ ann_efc=10,
+ ann_ef=10,
+ ann_m=4,
+ update_state=False,
+ )
+ with pytest.raises(ValueError, match="pass update_state=False"):
+ store.query_neighbors(ann, k=3, update_state=True)
+ assert read_assay_state(root, "RNA") is None
+
+ neighbors = store.query_neighbors(ann, k=3, update_state=False)
+ validate_neighbors_artifact_selection(root, neighbors, "I", "unused")
+ group = artifact_group(root, neighbors)
+ assert group["indices"].shape == (8, 3)
+ assert group["distances"].shape == (8, 3)
+
+
+def test_imported_coordinates_kind_is_assay_scoped() -> None:
+ with pytest.raises(ValueError, match="must be assay-scoped"):
+ ArtifactRef(
+ scope="datastore",
+ kind="imported_coordinates",
+ artifact_id="a" * 64,
+ )
diff --git a/tests/test_knn_utils.py b/tests/test_knn_utils.py
new file mode 100644
index 00000000..7694c332
--- /dev/null
+++ b/tests/test_knn_utils.py
@@ -0,0 +1,997 @@
+import functools
+import json
+import warnings
+from pathlib import Path
+
+import numpy as np
+import pytest
+from scipy.sparse import coo_matrix, csr_matrix
+from sklearn.metrics import adjusted_rand_score
+
+from scarf.clustering.leiden import leiden_membership
+from scarf.neighbors.graph import (
+ build_connectivity_arrays,
+ calc_snn,
+ merge_graphs,
+ take_nearest_per_row,
+ weight_sort_indices,
+)
+from scarf.neighbors.diffusion import diffusion_operator
+from scarf.neighbors.integration import wnn_integration
+from scarf.utils import logger
+
+
+def _simple_knn_graph(n: int, k: int = 3) -> csr_matrix:
+ rows, cols, data = [], [], []
+ for i in range(n):
+ for j in range(1, k + 1):
+ neighbor = (i + j) % n
+ rows.append(i)
+ cols.append(neighbor)
+ data.append(float(j))
+ return csr_matrix((data, (rows, cols)), shape=(n, n))
+
+
+def _grouped_knn_graph(groups: list[list[int]]) -> csr_matrix:
+ n_cells = sum(len(group) for group in groups)
+ rows = []
+ cols = []
+ for group in groups:
+ for cell in group:
+ neighbors = [neighbor for neighbor in group if neighbor != cell]
+ rows.extend([cell] * len(neighbors))
+ cols.extend(neighbors)
+ return csr_matrix((np.ones(len(rows)), (rows, cols)), shape=(n_cells, n_cells))
+
+
+def _simple_knn_indices(n: int, k: int = 3) -> np.ndarray:
+ return np.asarray(
+ [[(cell + offset + 1) % n for offset in range(k)] for cell in range(n)],
+ dtype=np.int64,
+ )
+
+
+def _grouped_knn_indices(groups: list[list[int]]) -> np.ndarray:
+ graph = _grouped_knn_graph(groups)
+ degree = int(graph.getnnz(axis=1)[0])
+ return graph.indices.reshape(graph.shape[0], degree)
+
+
+def test_leiden_membership_preserves_disconnected_partitions():
+ graph = _grouped_knn_graph([[0, 1, 2, 3], [4, 5, 6, 7]])
+
+ actual = leiden_membership(graph, resolution=1.0, random_seed=4444)
+
+ assert adjusted_rand_score([1, 1, 1, 1, 2, 2, 2, 2], actual) == pytest.approx(1.0)
+
+
+def test_diffusion_operator_matches_powered_row_normalization():
+ graph = csr_matrix(
+ [
+ [0.0, 2.0, 0.0],
+ [1.0, 0.0, 1.0],
+ [0.0, 3.0, 0.0],
+ ]
+ )
+
+ actual = diffusion_operator(graph, power=2)
+
+ np.testing.assert_allclose(
+ actual.toarray(),
+ [
+ [0.5, 0.0, 0.5],
+ [0.0, 1.0, 0.0],
+ [0.5, 0.0, 0.5],
+ ],
+ rtol=0,
+ atol=1e-12,
+ )
+
+
+def _multimodal_wnn_inputs() -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
+ indices1 = _grouped_knn_indices([[0, 1, 2, 3], [4, 5, 6, 7]])
+ indices2 = _grouped_knn_indices([[0, 2, 4, 6], [1, 3, 5, 7]])
+ ld1 = np.array(
+ [
+ [0.0, 1.0, 0.2],
+ [0.1, 0.9, 0.3],
+ [-0.1, 1.1, 0.1],
+ [0.2, 0.8, 0.4],
+ [3.0, -1.0, 0.0],
+ [3.1, -0.9, 0.1],
+ [2.9, -1.1, -0.1],
+ [3.2, -0.8, 0.2],
+ ],
+ dtype=np.float64,
+ )
+ ld2 = np.array(
+ [
+ [1.0, 0.0],
+ [-1.0, 3.0],
+ [0.9, 0.1],
+ [-0.9, 3.1],
+ [1.1, -0.1],
+ [-1.1, 2.9],
+ [0.8, 0.2],
+ [-0.8, 3.2],
+ ],
+ dtype=np.float64,
+ )
+ return indices1, ld1, indices2, ld2
+
+
+def _reference_wnn(
+ indices1: np.ndarray,
+ ld1: np.ndarray,
+ indices2: np.ndarray,
+ ld2: np.ndarray,
+ *,
+ l2_normalize: bool,
+) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ def normalize(values: np.ndarray) -> np.ndarray:
+ output = np.asarray(values, dtype=np.float64).copy()
+ if not l2_normalize:
+ return output
+ norms = np.linalg.norm(output, axis=1)
+ np.divide(
+ output,
+ norms[:, np.newaxis],
+ out=output,
+ where=norms[:, np.newaxis] > 0,
+ )
+ return output
+
+ def kernel(
+ distances: np.ndarray,
+ nearest: float,
+ bandwidth: float,
+ ) -> np.ndarray:
+ adjusted = np.maximum(distances - nearest, 0)
+ tolerance = 8.0 * np.finfo(np.float64).eps * nearest
+ if bandwidth <= tolerance:
+ return (adjusted <= tolerance).astype(np.float64)
+ return np.exp(-(adjusted / bandwidth))
+
+ embedding1 = normalize(ld1)
+ embedding2 = normalize(ld2)
+ n_cells = len(indices1)
+ output_k = min(indices1.shape[1], indices2.shape[1])
+ selected_indices = np.empty((n_cells, output_k), dtype=np.int64)
+ selected_affinities = np.empty((n_cells, output_k), dtype=np.float64)
+ modality_weights = np.empty((n_cells, 2), dtype=np.float64)
+
+ for cell in range(n_cells):
+ neighbors1 = indices1[cell]
+ neighbors2 = indices2[cell]
+ candidates = np.union1d(neighbors1, neighbors2)
+ candidate1 = embedding1[candidates]
+ candidate2 = embedding2[candidates]
+ point1 = embedding1[cell]
+ point2 = embedding2[cell]
+ distances1 = np.linalg.norm(point1 - candidate1, axis=1)
+ distances2 = np.linalg.norm(point2 - candidate2, axis=1)
+ positions1 = np.searchsorted(candidates, neighbors1)
+ positions2 = np.searchsorted(candidates, neighbors2)
+ own1 = np.sort(distances1[positions1])
+ own2 = np.sort(distances2[positions2])
+ nearest1, nearest2 = float(own1[0]), float(own2[0])
+ bandwidth1 = float(own1[-1] - nearest1)
+ bandwidth2 = float(own2[-1] - nearest2)
+
+ within1 = kernel(
+ np.asarray([np.linalg.norm(point1 - candidate1[positions1].mean(axis=0))]),
+ nearest1,
+ bandwidth1,
+ )[0]
+ cross1 = kernel(
+ np.asarray([np.linalg.norm(point1 - candidate1[positions2].mean(axis=0))]),
+ nearest1,
+ bandwidth1,
+ )[0]
+ within2 = kernel(
+ np.asarray([np.linalg.norm(point2 - candidate2[positions2].mean(axis=0))]),
+ nearest2,
+ bandwidth2,
+ )[0]
+ cross2 = kernel(
+ np.asarray([np.linalg.norm(point2 - candidate2[positions1].mean(axis=0))]),
+ nearest2,
+ bandwidth2,
+ )[0]
+ score1 = np.clip(within1 / (cross1 + 1e-4), 0, 200)
+ score2 = np.clip(within2 / (cross2 + 1e-4), 0, 200)
+ score_max = max(score1, score2)
+ exp1, exp2 = np.exp(score1 - score_max), np.exp(score2 - score_max)
+ weight1 = exp1 / (exp1 + exp2)
+ weight2 = exp2 / (exp1 + exp2)
+ modality_weights[cell] = (weight1, weight2)
+
+ affinities = weight1 * kernel(
+ distances1,
+ nearest1,
+ bandwidth1,
+ ) + weight2 * kernel(
+ distances2,
+ nearest2,
+ bandwidth2,
+ )
+ selected = np.lexsort((candidates, -affinities))[:output_k]
+ selected_indices[cell] = candidates[selected]
+ selected_affinities[cell] = affinities[selected]
+
+ return selected_indices, selected_affinities, modality_weights
+
+
+def test_calc_snn_returns_normalized_overlap():
+ graph = _simple_knn_graph(6, k=3)
+ indices = graph.indices.reshape((6, 3))
+ snn = calc_snn(indices)
+ assert snn.shape == (6, 3)
+ assert np.all(snn >= 0)
+ assert np.all(snn <= 1)
+
+
+def test_weight_sort_indices_keeps_top_neighbors():
+ indices = np.array([4, 1, 2, 1, 3])
+ weights = np.array([0.2, 0.5, 0.4, 0.6, 0.1])
+ sort_weights = weights + np.array([0.0, 0.2, 0.1, 0.2, 0.0])
+ kept_idx, kept_w = weight_sort_indices(indices, weights, sort_weights, n=3)
+ assert len(kept_idx) == 3
+ assert len(kept_w) == 3
+ assert len(set(kept_idx)) == len(kept_idx)
+
+
+def test_merge_graphs_preserves_shape_and_edge_count():
+ g1 = _simple_knn_graph(8, k=3)
+ g2 = _simple_knn_graph(8, k=3)
+ merged = merge_graphs([g1, g2])
+ assert isinstance(merged, coo_matrix)
+ assert merged.shape == g1.shape
+ assert merged.nnz == g1.nnz
+
+
+def test_merge_graphs_rejects_mismatched_shapes():
+ g1 = _simple_knn_graph(6, k=3)
+ g2 = _simple_knn_graph(8, k=3)
+ with pytest.raises(ValueError, match="same shape"):
+ merge_graphs([g1, g2])
+
+
+def test_merge_graphs_rejects_one_neighbor_snn_input():
+ graph = _simple_knn_graph(6, k=1)
+
+ with pytest.raises(ValueError, match="at least two neighbors"):
+ merge_graphs([graph, graph.copy()])
+
+
+def test_build_connectivity_arrays_runs_in_memory():
+ n_cells, n_neighbors = 6, 5
+ idx = np.array(
+ [
+ [(row + offset + 1) % n_cells for offset in range(n_neighbors)]
+ for row in range(n_cells)
+ ]
+ )
+ dist = np.array(
+ [
+ [0.10, 0.30, 0.80, 1.50, 3.00],
+ [0.13, 0.35, 0.82, 1.60, 3.20],
+ [0.16, 0.40, 0.84, 1.70, 3.40],
+ [0.19, 0.45, 0.86, 1.80, 3.60],
+ [0.22, 0.50, 0.88, 1.90, 3.80],
+ [0.25, 0.55, 0.90, 2.00, 4.00],
+ ]
+ )
+ expected_weights = np.array(
+ [
+ 1.0,
+ 0.9779863,
+ 0.92504877,
+ 0.8557153,
+ 0.7241439,
+ 1.0,
+ 0.97687674,
+ 0.92925274,
+ 0.85528576,
+ 0.7214707,
+ 1.0,
+ 0.97586185,
+ 0.9331116,
+ 0.8548864,
+ 0.7190224,
+ 1.0,
+ 0.97493076,
+ 0.93666923,
+ 0.8545199,
+ 0.71678144,
+ 1.0,
+ 0.9740712,
+ 0.9399542,
+ 0.85416996,
+ 0.71470064,
+ 1.0,
+ 0.973276,
+ 0.9429993,
+ 0.8538406,
+ 0.7127714,
+ ],
+ dtype=np.float32,
+ )
+
+ edges, weights = build_connectivity_arrays(
+ idx,
+ dist,
+ local_connectivity=1.0,
+ bandwidth=1.5,
+ )
+
+ assert edges.shape == (n_cells * n_neighbors, 2)
+ assert weights.shape == (n_cells * n_neighbors,)
+ assert edges.dtype == np.uint32
+ assert weights.dtype == np.float32
+ assert np.all(weights > 0)
+ np.testing.assert_array_equal(
+ edges,
+ np.column_stack(
+ (
+ np.repeat(np.arange(n_cells), n_neighbors),
+ idx.reshape(-1),
+ )
+ ).astype(np.uint32),
+ )
+ np.testing.assert_allclose(weights, expected_weights, rtol=1e-6, atol=1e-7)
+
+
+def test_connectivity_omits_zero_membership_edges():
+ n_cells, n_neighbors = 10, 5
+ indices = np.array(
+ [
+ [(row + offset + 1) % n_cells for offset in range(n_neighbors)]
+ for row in range(n_cells)
+ ]
+ )
+ distances = np.tile(
+ np.array([0.0, 1.0, 1e20, 1e30, 1e35]),
+ (n_cells, 1),
+ )
+
+ edges, weights = build_connectivity_arrays(
+ indices,
+ distances,
+ local_connectivity=1.0,
+ bandwidth=1.5,
+ )
+
+ expected = np.tile(
+ np.array([1.0, 1.0, 1.0, 0.9512299], dtype=np.float32),
+ n_cells,
+ )
+ assert len(edges) == n_cells * 4
+ np.testing.assert_allclose(weights, expected, rtol=1e-6, atol=1e-7)
+
+
+def test_take_nearest_per_row_handles_rows_that_lost_zero_weight_edges():
+ # Row 1 kept two edges instead of three because a zero-weight edge was
+ # dropped. Assuming a fixed row width here selects the wrong neighbors.
+ edges = np.array(
+ [[0, 5], [0, 6], [0, 7], [1, 8], [1, 9], [2, 1], [2, 2], [2, 3]],
+ dtype=np.uint32,
+ )
+ weights = np.arange(1, 9, dtype=np.float32)
+
+ kept_weights, kept_edges = take_nearest_per_row(weights, edges, 3, 2)
+
+ np.testing.assert_array_equal(kept_edges[:, 1], [5, 6, 8, 9, 1, 2])
+ np.testing.assert_allclose(kept_weights, [1.0, 2.0, 4.0, 5.0, 6.0, 7.0])
+
+ full_weights, full_edges = take_nearest_per_row(weights, edges, 3, 3)
+ np.testing.assert_array_equal(full_edges, edges)
+ np.testing.assert_allclose(full_weights, weights)
+
+ with pytest.raises(ValueError, match="grouped by source cell"):
+ take_nearest_per_row(
+ np.ones(2, dtype=np.float32),
+ np.array([[1, 0], [0, 1]], dtype=np.uint32),
+ 2,
+ 1,
+ )
+
+
+def test_take_nearest_per_row_keeps_every_cell_when_a_row_is_empty():
+ edges = np.array([[0, 1], [0, 2], [2, 0]], dtype=np.uint32)
+ weights = np.array([0.5, 0.25, 0.75], dtype=np.float32)
+
+ kept_weights, kept_edges = take_nearest_per_row(weights, edges, 3, 1)
+
+ np.testing.assert_array_equal(kept_edges[:, 0], [0, 2])
+ np.testing.assert_allclose(kept_weights, [0.5, 0.75])
+
+
+def test_wnn_integration_handles_extreme_affinities_without_runtime_warnings():
+ indices1, ld1, indices2, ld2 = _multimodal_wnn_inputs()
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", RuntimeWarning)
+ merged, modality_weights = wnn_integration(
+ "RNA",
+ indices1,
+ ld1,
+ "ADT",
+ indices2,
+ ld2,
+ n_threads=1,
+ )
+
+ assert isinstance(merged, coo_matrix)
+ assert merged.shape == (len(indices1), len(indices1))
+ assert merged.nnz == indices1.size
+ np.testing.assert_array_equal(
+ np.bincount(merged.row, minlength=len(indices1)),
+ np.repeat(indices1.shape[1], len(indices1)),
+ )
+ assert np.all(np.isfinite(merged.data))
+ assert np.all(merged.data > 0)
+ assert np.all(merged.data <= 1)
+ assert modality_weights.dtype == np.float32
+ assert np.all(np.isfinite(modality_weights))
+ assert np.all(modality_weights >= 0)
+ np.testing.assert_allclose(modality_weights.sum(axis=1), 1, rtol=1e-6)
+
+
+def test_wnn_integration_is_invariant_to_cell_order():
+ indices1, _, indices2, _ = _multimodal_wnn_inputs()
+ rng = np.random.default_rng(42)
+ ld1 = rng.normal(size=(len(indices1), 3))
+ ld2 = rng.normal(size=(len(indices2), 4))
+ expected, expected_weights = wnn_integration(
+ "RNA",
+ indices1,
+ ld1,
+ "ADT",
+ indices2,
+ ld2,
+ n_threads=1,
+ )
+
+ permutation = np.array([5, 0, 7, 2, 6, 1, 4, 3])
+ old_to_new = np.argsort(permutation)
+ permuted, permuted_weights = wnn_integration(
+ "RNA",
+ old_to_new[indices1[permutation]],
+ ld1[permutation],
+ "ADT",
+ old_to_new[indices2[permutation]],
+ ld2[permutation],
+ n_threads=1,
+ )
+ inverse = np.argsort(permutation)
+ restored = permuted.tocsr()[inverse][:, inverse]
+
+ np.testing.assert_allclose(expected.toarray(), restored.toarray())
+ np.testing.assert_allclose(expected_weights, permuted_weights[inverse])
+
+
+def test_wnn_integration_rejects_mismatched_neighbor_rows():
+ indices1 = _simple_knn_indices(6, k=3)
+ indices2 = _simple_knn_indices(7, k=3)
+
+ with pytest.raises(ValueError, match="same number of cells"):
+ wnn_integration(
+ "RNA",
+ indices1,
+ np.zeros((6, 2)),
+ "ADT",
+ indices2,
+ np.zeros((7, 2)),
+ n_threads=1,
+ )
+
+
+@pytest.mark.parametrize(
+ ("indices", "error", "match"),
+ [
+ (np.arange(6), ValueError, "non-empty matrix"),
+ (np.zeros((6, 3), dtype=np.float64), TypeError, "integer indices"),
+ (
+ np.array(
+ [
+ [1, 1, 2],
+ [0, 2, 3],
+ [0, 1, 3],
+ [0, 1, 2],
+ [0, 1, 2],
+ [0, 1, 2],
+ ]
+ ),
+ ValueError,
+ "unique within each row",
+ ),
+ (
+ np.array(
+ [
+ [0, 1, 2],
+ [0, 2, 3],
+ [0, 1, 3],
+ [0, 1, 2],
+ [0, 1, 2],
+ [0, 1, 2],
+ ]
+ ),
+ ValueError,
+ "exclude self",
+ ),
+ (
+ np.array(
+ [
+ [1, 2, 6],
+ [0, 2, 3],
+ [0, 1, 3],
+ [0, 1, 2],
+ [0, 1, 2],
+ [0, 1, 2],
+ ]
+ ),
+ ValueError,
+ "outside cell range",
+ ),
+ ],
+)
+def test_wnn_integration_rejects_invalid_neighbor_matrices(indices, error, match):
+ valid = _simple_knn_indices(6, k=3)
+ embeddings = np.arange(12, dtype=np.float64).reshape(6, 2)
+
+ with pytest.raises(error, match=match):
+ wnn_integration(
+ "RNA",
+ indices,
+ embeddings,
+ "ADT",
+ valid,
+ embeddings,
+ n_threads=1,
+ )
+
+
+@pytest.mark.parametrize(
+ ("embedding", "match"),
+ [
+ (np.zeros((5, 2)), "one row per graph cell"),
+ (np.empty((6, 0)), "non-empty matrix"),
+ (
+ np.array(
+ [[0.0, 0.0]] * 5 + [[np.nan, 0.0]],
+ dtype=np.float64,
+ ),
+ "non-finite values",
+ ),
+ ],
+)
+def test_wnn_integration_rejects_invalid_embeddings(embedding, match):
+ indices = _simple_knn_indices(6, k=3)
+ valid_embedding = np.zeros((6, 2))
+
+ with pytest.raises(ValueError, match=match):
+ wnn_integration(
+ "RNA",
+ indices,
+ embedding,
+ "ADT",
+ indices,
+ valid_embedding,
+ n_threads=1,
+ )
+
+
+def test_wnn_integration_uses_minimum_neighbor_count_for_mismatched_graphs():
+ indices1 = _simple_knn_indices(8, k=3)
+ indices2 = _simple_knn_indices(8, k=2)
+ rng = np.random.default_rng(7)
+ ld1 = rng.normal(size=(8, 3))
+ ld2 = rng.normal(size=(8, 2))
+ messages = []
+ sink = logger.add(
+ lambda message: messages.append(message.record["message"]), level="WARNING"
+ )
+ try:
+ merged, modality_weights = wnn_integration(
+ "RNA",
+ indices1,
+ ld1,
+ "ADT",
+ indices2,
+ ld2,
+ n_threads=1,
+ )
+ swapped, swapped_weights = wnn_integration(
+ "ADT",
+ indices2,
+ ld2,
+ "RNA",
+ indices1,
+ ld1,
+ n_threads=1,
+ )
+ finally:
+ logger.remove(sink)
+
+ assert any("different neighbor counts" in message for message in messages)
+ assert merged.nnz == len(indices1) * 2
+ assert np.all(np.isfinite(merged.data))
+ np.testing.assert_allclose(merged.toarray(), swapped.toarray())
+ np.testing.assert_allclose(modality_weights, swapped_weights[:, ::-1])
+
+
+@pytest.mark.parametrize("l2_normalize", [True, False])
+@pytest.mark.parametrize(("modality", "scale"), [(1, 1e8), (2, 1e-7)])
+def test_wnn_integration_is_invariant_to_per_modality_scale(
+ l2_normalize,
+ modality,
+ scale,
+):
+ indices1, ld1, indices2, ld2 = _multimodal_wnn_inputs()
+ expected, expected_weights = wnn_integration(
+ "RNA",
+ indices1,
+ ld1,
+ "ADT",
+ indices2,
+ ld2,
+ n_threads=1,
+ l2_normalize=l2_normalize,
+ )
+ if modality == 1:
+ ld1 = ld1 * scale
+ else:
+ ld2 = ld2 * scale
+
+ actual, actual_weights = wnn_integration(
+ "RNA",
+ indices1,
+ ld1,
+ "ADT",
+ indices2,
+ ld2,
+ n_threads=1,
+ l2_normalize=l2_normalize,
+ )
+
+ np.testing.assert_array_equal(actual.row, expected.row)
+ np.testing.assert_array_equal(actual.col, expected.col)
+ np.testing.assert_allclose(actual.data, expected.data, rtol=2e-6, atol=1e-7)
+ np.testing.assert_allclose(
+ actual_weights,
+ expected_weights,
+ rtol=2e-6,
+ atol=1e-7,
+ )
+
+
+def test_wnn_integration_uses_kth_nonself_neighbor_for_bandwidth():
+ indices = _simple_knn_indices(5, k=2)
+ embedding = np.arange(5, dtype=np.float64).reshape(-1, 1)
+
+ graph, _ = wnn_integration(
+ "RNA",
+ indices,
+ embedding,
+ "ADT",
+ indices,
+ embedding,
+ n_threads=1,
+ l2_normalize=False,
+ )
+ row_zero = graph.data[graph.row == 0]
+
+ np.testing.assert_allclose(
+ row_zero,
+ np.array([1.0, np.exp(-1)], dtype=np.float32),
+ rtol=1e-6,
+ atol=1e-7,
+ )
+
+
+def test_wnn_integration_handles_degenerate_bandwidth_deterministically():
+ indices1, _, indices2, _ = _multimodal_wnn_inputs()
+ embedding1 = np.zeros((len(indices1), 3))
+ embedding2 = np.zeros((len(indices2), 2))
+
+ first, first_weights = wnn_integration(
+ "RNA",
+ indices1,
+ embedding1,
+ "ADT",
+ indices2,
+ embedding2,
+ n_threads=1,
+ )
+ second, second_weights = wnn_integration(
+ "RNA",
+ indices1,
+ embedding1,
+ "ADT",
+ indices2,
+ embedding2,
+ n_threads=1,
+ )
+
+ np.testing.assert_array_equal(first.row, second.row)
+ np.testing.assert_array_equal(first.col, second.col)
+ np.testing.assert_array_equal(first.data, np.ones(first.nnz, dtype=np.float32))
+ np.testing.assert_array_equal(first.data, second.data)
+ np.testing.assert_allclose(first_weights, 0.5)
+ np.testing.assert_array_equal(first_weights, second_weights)
+
+
+def test_wnn_integration_matches_scalar_affinity_reference():
+ indices1, ld1, indices2, ld2 = _multimodal_wnn_inputs()
+ expected_indices, expected_affinities, expected_weights = _reference_wnn(
+ indices1,
+ ld1,
+ indices2,
+ ld2,
+ l2_normalize=True,
+ )
+
+ actual, actual_weights = wnn_integration(
+ "RNA",
+ indices1,
+ ld1,
+ "ADT",
+ indices2,
+ ld2,
+ n_threads=1,
+ )
+
+ np.testing.assert_array_equal(
+ actual.col.reshape(expected_indices.shape),
+ expected_indices,
+ )
+ np.testing.assert_allclose(
+ actual.data.reshape(expected_affinities.shape),
+ expected_affinities,
+ rtol=1e-6,
+ atol=1e-7,
+ )
+ np.testing.assert_allclose(
+ actual_weights,
+ expected_weights,
+ rtol=1e-6,
+ atol=1e-7,
+ )
+
+
+def test_wnn_integration_follows_informative_modality_across_numeric_scales():
+ indices1 = _grouped_knn_indices([[0, 1, 2], [3, 4, 5]])
+ indices2 = _grouped_knn_indices([[0, 3, 4], [1, 2, 5]])
+ informative = np.array(
+ [
+ [1.0, 0.0],
+ [0.9, 0.1],
+ [1.1, -0.1],
+ [-1.0, 0.0],
+ [-0.9, 0.1],
+ [-1.1, -0.1],
+ ]
+ )
+ noisy = (
+ np.array(
+ [
+ [0.0, 0.0],
+ [4.0, 1.0],
+ [-3.0, 2.0],
+ [1.0, -4.0],
+ [-2.0, -3.0],
+ [3.0, 4.0],
+ ]
+ )
+ * 1e9
+ )
+
+ graph, modality_weights = wnn_integration(
+ "RNA",
+ indices1,
+ informative,
+ "ADT",
+ indices2,
+ noisy,
+ n_threads=1,
+ l2_normalize=False,
+ )
+ selected = graph.col.reshape(len(indices1), -1)
+ informative_overlap = np.mean(
+ [
+ len(set(row).intersection(neighbors)) / len(row)
+ for row, neighbors in zip(selected, indices1, strict=True)
+ ]
+ )
+ noisy_overlap = np.mean(
+ [
+ len(set(row).intersection(neighbors)) / len(row)
+ for row, neighbors in zip(selected, indices2, strict=True)
+ ]
+ )
+
+ assert np.mean(modality_weights[:, 0]) > 0.5
+ assert informative_overlap > noisy_overlap
+ assert all(
+ set(row).issubset(set(informative_neighbors))
+ for row, informative_neighbors in zip(selected, indices1, strict=True)
+ )
+
+
+def test_wnn_integration_is_scale_invariant_at_near_degenerate_bandwidth():
+ indices1 = _grouped_knn_indices([[0, 1, 2], [3, 4, 5]])
+ indices2 = _grouped_knn_indices([[0, 3, 4], [1, 2, 5]])
+ ld1 = np.array(
+ [
+ [1.0, 0.0],
+ [0.9, 0.1],
+ [1.1, -0.1],
+ [-1.0, 0.0],
+ [-0.9, 0.1],
+ [-1.1, -0.1],
+ ]
+ )
+ ld2 = (
+ np.array(
+ [
+ [0.0, 0.0],
+ [4.0, 1.0],
+ [-3.0, 2.0],
+ [1.0, -4.0],
+ [-2.0, -3.0],
+ [3.0, 4.0],
+ ]
+ )
+ * 1e9
+ )
+
+ results = [
+ wnn_integration(
+ "RNA",
+ indices1,
+ ld1 * scale,
+ "ADT",
+ indices2,
+ ld2,
+ n_threads=1,
+ l2_normalize=False,
+ )
+ for scale in (1.0, 1e6, 1e9)
+ ]
+ baseline, baseline_weights = results[0]
+
+ # Equidistant neighbours differ only by rounding noise in the distance
+ # reduction, so neither may be demoted below an unrelated candidate.
+ np.testing.assert_array_equal(
+ baseline.col.reshape(len(indices1), -1),
+ indices1,
+ )
+ for graph, weights in results[1:]:
+ np.testing.assert_array_equal(graph.col, baseline.col)
+ np.testing.assert_allclose(graph.data, baseline.data, rtol=1e-6, atol=1e-7)
+ np.testing.assert_allclose(weights, baseline_weights, rtol=1e-6, atol=1e-7)
+
+
+@functools.cache
+def _seurat_golden() -> dict:
+ path = Path(__file__).parent / "seurat_wnn_5_5_1_golden.json"
+ return json.loads(path.read_text())
+
+
+def _seurat_golden_wnn() -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ """Run Scarf on the fixture inputs and reshape the graph to one row per cell."""
+ fixture = _seurat_golden()
+ inputs = fixture["inputs"]
+ indices1 = np.asarray(inputs["rnaIndices"], dtype=np.uint32)
+ indices2 = np.asarray(inputs["adtIndices"], dtype=np.uint32)
+ graph, weights = wnn_integration(
+ "RNA",
+ indices1,
+ np.asarray(inputs["rnaEmbedding"], dtype=np.float64),
+ "ADT",
+ indices2,
+ np.asarray(inputs["adtEmbedding"], dtype=np.float64),
+ n_threads=1,
+ l2_normalize=fixture["provenance"]["l2Normalize"],
+ )
+ shape = indices1.shape
+ return (
+ graph.col.reshape(shape),
+ graph.data.reshape(shape).astype(np.float64),
+ weights.astype(np.float64),
+ )
+
+
+def test_seurat_wnn_golden_fixture_pins_its_provenance():
+ provenance = _seurat_golden()["provenance"]
+
+ assert provenance["package"] == "Seurat"
+ assert provenance["packageVersion"] == "5.5.1"
+ assert provenance["dataset"] == "tenx_8K_pbmc_citeseq"
+ assert provenance["l2Normalize"] is True
+ assert "Seurat:::PredictAssay" in provenance["matchedFunctions"]
+ assert provenance["defaultFunction"] == "Seurat::FindMultiModalNeighbors"
+
+
+def test_wnn_integration_matches_seurat_matched_equations():
+ """Seurat's own routines on Scarf's candidate pool and bandwidth.
+
+ Residual disagreement sits at the float32 resolution of the returned graph,
+ so the tolerance is set just above it. Any error in the affinity kernel,
+ the bandwidth index, the row normalization or the weight softmax moves the
+ result by several orders of magnitude more than this.
+ """
+ expected = _seurat_golden()["matched"]
+ selected, affinities, weights = _seurat_golden_wnn()
+
+ np.testing.assert_array_equal(
+ selected,
+ np.asarray(expected["neighborIndices"], dtype=selected.dtype),
+ )
+ np.testing.assert_allclose(
+ weights,
+ np.asarray(expected["modalityWeights"]),
+ rtol=0,
+ atol=1e-6,
+ )
+ np.testing.assert_allclose(
+ affinities,
+ np.asarray(expected["neighborAffinities"]),
+ rtol=0,
+ atol=1e-6,
+ )
+
+
+def test_wnn_integration_stays_close_to_seurat_defaults():
+ """Bound on how far Scarf's documented deviations move the result.
+
+ Seurat's defaults search a wider pool and use an SNN-far bandwidth, so exact
+ agreement is not expected. The floors sit below the values measured when the
+ fixture was written (0.89 neighbour overlap, 0.76 weight correlation) and
+ exist to catch the gap widening.
+ """
+ expected = _seurat_golden()["seuratDefault"]
+ selected, _affinities, weights = _seurat_golden_wnn()
+ k = selected.shape[1]
+ seurat_indices = np.asarray(expected["neighborIndices"])
+ seurat_weights = np.asarray(expected["modalityWeights"])
+
+ overlap = np.mean(
+ [
+ len(set(row.tolist()).intersection(reference.tolist())) / k
+ for row, reference in zip(selected, seurat_indices, strict=True)
+ ]
+ )
+ correlation = np.corrcoef(weights[:, 0], seurat_weights[:, 0])[0, 1]
+
+ assert overlap > 0.80
+ assert correlation > 0.65
+ assert abs(weights[:, 0].mean() - seurat_weights[:, 0].mean()) < 0.05
+
+
+def test_wnn_integration_output_contract():
+ indices1, ld1, indices2, ld2 = _multimodal_wnn_inputs()
+
+ graph, modality_weights = wnn_integration(
+ "RNA",
+ indices1,
+ ld1,
+ "ADT",
+ indices2,
+ ld2,
+ n_threads=1,
+ )
+
+ assert not np.any(graph.row == graph.col)
+ np.testing.assert_array_equal(
+ np.bincount(graph.row, minlength=len(indices1)),
+ np.full(len(indices1), min(indices1.shape[1], indices2.shape[1])),
+ )
+ assert graph.data.dtype == np.float32
+ assert np.all(np.isfinite(graph.data))
+ assert np.all((graph.data > 0) & (graph.data <= 1))
+ assert modality_weights.shape == (len(indices1), 2)
+ assert np.all(np.isfinite(modality_weights))
+ assert np.all(modality_weights >= 0)
+ np.testing.assert_allclose(modality_weights.sum(axis=1), 1, rtol=1e-6)
diff --git a/tests/test_mapping_alignment.py b/tests/test_mapping_alignment.py
new file mode 100644
index 00000000..d18bb2dd
--- /dev/null
+++ b/tests/test_mapping_alignment.py
@@ -0,0 +1,464 @@
+from typing import Any
+
+import numpy as np
+import pytest
+import zarr
+
+from scarf.assay import RNAassay, norm_lib_size
+from scarf.mapping.features import AlignedFeatureStream
+from scarf.matrix import ChunkedArray
+from scarf.storage.artifacts import callable_identity
+from scarf.storage.budget import ResourceBudget
+from tests.store_probes import RecordingStore
+
+
+class _MemoryMetadata:
+ def __init__(self, values: dict[str, np.ndarray]) -> None:
+ self._values = values
+
+ def fetch_all(self, name: str) -> np.ndarray:
+ return self._values[name]
+
+
+def _normalization(
+ *,
+ size_factor: float = 10.0,
+ log_transform: bool = False,
+ renormalize_subset: bool = False,
+) -> dict[str, Any]:
+ return {
+ "normalization_method": callable_identity(norm_lib_size),
+ "size_factor": size_factor,
+ "log_transform": log_transform,
+ "renormalize_subset": renormalize_subset,
+ }
+
+
+def _query_assay(
+ values: np.ndarray,
+ feature_ids: list[str],
+ *,
+ chunks: tuple[int, int] = (3, 2),
+ size_factor: int = 997,
+ read_only: bool = False,
+) -> tuple[RNAassay, RecordingStore, zarr.Array]:
+ store = RecordingStore()
+ writable_root = zarr.open_group(store=store, mode="w")
+ writable_counts = writable_root.create_array(
+ "counts",
+ shape=values.shape,
+ chunks=chunks,
+ dtype=values.dtype,
+ )
+ writable_counts[:] = values
+ root = (
+ zarr.open_group(store=store.with_read_only(True), mode="r")
+ if read_only
+ else writable_root
+ )
+ counts = root["counts"]
+ assert isinstance(counts, zarr.Array)
+
+ assay = object.__new__(RNAassay)
+ assay.name = "RNA"
+ assay.rawData = ChunkedArray(
+ counts,
+ nthreads=1,
+ resources=ResourceBudget(1_000_000, 1),
+ )
+ assay.feats = _MemoryMetadata({"ids": np.asarray(feature_ids)})
+ assay.cells = _MemoryMetadata({"RNA_nCounts": values.sum(axis=1, dtype=np.float64)})
+ assay.sf = size_factor
+ assay.scalar = np.array([123.0])
+ assay.z = root
+ return assay, store, writable_counts
+
+
+def _stream(
+ assay: RNAassay,
+ *,
+ cells: np.ndarray | None = None,
+ reference_ids: np.ndarray | None = None,
+ means: np.ndarray | None = None,
+ normalization: dict[str, Any] | None = None,
+ policy: str = "reference_mean",
+ resources: ResourceBudget | None = None,
+ reserved_resident_bytes: int = 0,
+ reserved_per_row_bytes: int = 0,
+) -> AlignedFeatureStream:
+ if cells is None:
+ cells = np.arange(assay.rawData.shape[0], dtype=np.int64)
+ if reference_ids is None:
+ reference_ids = np.array(["a", "missing", "b"])
+ if means is None:
+ means = np.array([1.5, 7.5, 2.5])
+ return AlignedFeatureStream(
+ query_assay=assay,
+ query_cell_indices=cells,
+ reference_feature_ids=reference_ids,
+ reference_normalized_means=means,
+ reference_normalization_parameters=normalization or _normalization(),
+ missing_feature_policy=policy,
+ resources=resources or ResourceBudget(1_000_000, 2),
+ reserved_resident_bytes=reserved_resident_bytes,
+ reserved_per_row_bytes=reserved_per_row_bytes,
+ )
+
+
+def _collect(stream: AlignedFeatureStream) -> np.ndarray:
+ blocks = list(stream)
+ assert [block.row_offset for block in blocks] == [
+ start for start, _ in stream.block_boundaries
+ ]
+ return np.concatenate([block.values for block in blocks], axis=0)
+
+
+def test_aligned_feature_stream_replays_in_reference_order() -> None:
+ counts = np.array(
+ [
+ [9, 2, 4, 1],
+ [3, 5, 7, 2],
+ [8, 1, 6, 5],
+ ],
+ dtype=np.uint32,
+ )
+ assay, _, _ = _query_assay(counts, ["extra", "b", "a", "c"])
+ cells = np.array([2, 0, 1])
+ reference_ids = np.array(["a", "missing", "b"])
+ means = np.array([1.5, 7.5, 2.5])
+ stream = _stream(
+ assay,
+ cells=cells,
+ reference_ids=reference_ids,
+ means=means,
+ )
+ reference_ids[0] = "changed"
+ means[:] = -1
+
+ expected_present = (
+ 10.0
+ * counts[np.ix_(cells, np.array([2, 1]))]
+ / counts.sum(axis=1)[cells, np.newaxis]
+ )
+ expected = np.column_stack(
+ (expected_present[:, 0], np.full(3, 7.5), expected_present[:, 1])
+ )
+ first = _collect(stream)
+ second = _collect(stream)
+
+ np.testing.assert_allclose(first, expected)
+ np.testing.assert_array_equal(second, first)
+ np.testing.assert_array_equal(stream.reference_feature_ids, ["a", "missing", "b"])
+ np.testing.assert_array_equal(stream.query_index_map, [2, 1])
+ np.testing.assert_array_equal(stream.reference_index_map, [0, 2])
+ np.testing.assert_array_equal(stream.reference_to_query_index_map, [2, -1, 1])
+ np.testing.assert_array_equal(stream.query_feature_indices, [2, 1])
+ np.testing.assert_array_equal(stream.reference_feature_indices, [0, 2])
+ np.testing.assert_array_equal(stream.query_cell_indices, cells)
+ assert stream.shape == (3, 3)
+ assert stream.dtype == np.dtype(np.float64)
+ assert stream.feature_coverage == pytest.approx(2 / 3)
+ assert len(stream.alignment_map_fingerprint) == 64
+ assert stream.alignment_map_hash == stream.alignment_map_fingerprint
+ assert not stream.reference_feature_ids.flags.writeable
+ with pytest.raises(ValueError, match="cannot set WRITEABLE flag"):
+ stream.reference_feature_ids.flags.writeable = True
+
+
+@pytest.mark.parametrize(
+ ("policy", "fill"),
+ [
+ ("reference_mean", 7.5),
+ ("zero", 0.0),
+ ],
+)
+def test_aligned_feature_stream_missing_feature_fills(
+ policy: str,
+ fill: float,
+) -> None:
+ counts = np.array([[2, 3], [5, 7]], dtype=np.uint32)
+ assay, _, _ = _query_assay(counts, ["a", "b"])
+ values = _collect(_stream(assay, policy=policy))
+
+ np.testing.assert_allclose(values[:, 1], fill)
+
+
+def test_aligned_feature_stream_error_policy_rejects_missing_features() -> None:
+ assay, _, _ = _query_assay(
+ np.array([[2, 3]], dtype=np.uint32),
+ ["a", "b"],
+ chunks=(1, 1),
+ )
+
+ with pytest.raises(ValueError, match="missing 1 required reference feature"):
+ _stream(assay, policy="error")
+
+
+def test_aligned_feature_stream_uses_reference_size_factor_and_log_semantics() -> None:
+ counts = np.array([[8, 2, 10], [0, 5, 5]], dtype=np.uint32)
+ assay, _, _ = _query_assay(counts, ["a", "b", "extra"], size_factor=999)
+ original_scalar = assay.scalar
+ stream = _stream(
+ assay,
+ reference_ids=np.array(["b", "a"]),
+ means=np.array([0.0, 0.0]),
+ normalization=_normalization(size_factor=20, log_transform=True),
+ policy="zero",
+ )
+
+ expected = np.log1p(
+ 20.0 * counts[:, [1, 0]] / counts.sum(axis=1, dtype=np.float64)[:, np.newaxis]
+ )
+ np.testing.assert_allclose(_collect(stream), expected)
+ assert assay.sf == 999
+ assert assay.scalar is original_scalar
+
+
+def test_aligned_feature_stream_renormalizes_over_matched_reference_features() -> None:
+ counts = np.array(
+ [
+ [2, 3, 100],
+ [0, 0, 7],
+ ],
+ dtype=np.uint32,
+ )
+ assay, _, _ = _query_assay(counts, ["a", "b", "extra"])
+ stream = _stream(
+ assay,
+ reference_ids=np.array(["b", "missing", "a"]),
+ means=np.array([0.0, 11.0, 0.0]),
+ normalization=_normalization(size_factor=10, renormalize_subset=True),
+ )
+
+ np.testing.assert_allclose(
+ _collect(stream),
+ np.array(
+ [
+ [6.0, 11.0, 4.0],
+ [0.0, 11.0, 0.0],
+ ]
+ ),
+ )
+
+
+def test_raw_expression_fingerprint_includes_unmatched_query_features() -> None:
+ counts = np.array([[2, 3, 100], [5, 7, 200]], dtype=np.uint32)
+ assay, _, writable_counts = _query_assay(counts, ["a", "b", "extra"])
+ options = {
+ "reference_ids": np.array(["a", "b"]),
+ "means": np.zeros(2),
+ "normalization": _normalization(renormalize_subset=True),
+ "policy": "zero",
+ }
+ before = _stream(assay, **options)
+ before_values = _collect(before)
+ before_fingerprint = before.raw_expression_fingerprint
+
+ writable_counts[0, 2] = 101
+ after = _stream(assay, **options)
+
+ np.testing.assert_array_equal(_collect(after), before_values)
+ assert after.alignment_map_fingerprint == before.alignment_map_fingerprint
+ assert after.raw_expression_fingerprint != before_fingerprint
+ assert before.raw_expression_fingerprint == before_fingerprint
+
+
+def test_aligned_feature_stream_bounds_rows_under_tiny_budget() -> None:
+ counts = np.arange(1, 29, dtype=np.uint32).reshape(7, 4)
+ assay, _, _ = _query_assay(
+ counts,
+ ["a", "b", "extra", "other"],
+ chunks=(5, 2),
+ )
+ roomy = _stream(assay)
+ two_row_budget = ResourceBudget(
+ roomy.resident_bytes + roomy.decoded_chunk_bytes + 2 * roomy.stream_row_bytes,
+ 8,
+ )
+ bounded = _stream(assay, resources=two_row_budget)
+ blocks = list(bounded)
+
+ assert bounded.row_geometry.block_rows == 2
+ assert max(len(block.values) for block in blocks) == 2
+ assert sum(len(block.values) for block in blocks) == len(counts)
+ assert bounded.block_boundaries == ((0, 2), (2, 4), (4, 6), (6, 7))
+
+
+def test_aligned_feature_stream_reserves_downstream_mapping_memory() -> None:
+ counts = np.arange(1, 29, dtype=np.uint32).reshape(7, 4)
+ assay, _, _ = _query_assay(
+ counts,
+ ["a", "b", "extra", "other"],
+ chunks=(5, 2),
+ )
+ baseline = _stream(assay)
+ budget = ResourceBudget(
+ baseline.resident_bytes
+ + baseline.decoded_chunk_bytes
+ + 4 * baseline.stream_row_bytes,
+ 2,
+ )
+ reserved = _stream(
+ assay,
+ resources=budget,
+ reserved_resident_bytes=baseline.stream_row_bytes,
+ reserved_per_row_bytes=baseline.stream_row_bytes,
+ )
+
+ assert (
+ reserved.resident_bytes == baseline.resident_bytes + baseline.stream_row_bytes
+ )
+ assert reserved.stream_row_bytes == 2 * baseline.stream_row_bytes
+ assert reserved.row_geometry.block_rows == 1
+ assert sum(len(block.values) for block in reserved) == len(counts)
+
+
+def test_raw_expression_fingerprint_uses_budgeted_full_width_blocks(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ counts = np.arange(1, 141, dtype=np.uint32).reshape(7, 20)
+ assay, _, _ = _query_assay(
+ counts,
+ ["a", "b", *(f"extra-{index}" for index in range(18))],
+ chunks=(5, 4),
+ )
+ options = {
+ "reference_ids": np.array(["a", "b"]),
+ "means": np.zeros(2),
+ "policy": "zero",
+ }
+ roomy = _stream(assay, **options)
+ all_column_bytes = counts.shape[1] * np.dtype(np.int64).itemsize
+ fingerprint_row_bytes = 2 * counts.shape[1] * counts.dtype.itemsize
+ budget = ResourceBudget(
+ roomy.resident_bytes
+ + all_column_bytes
+ + roomy.decoded_chunk_bytes
+ + 2 * fingerprint_row_bytes,
+ 4,
+ )
+ bounded = _stream(assay, resources=budget, **options)
+ observed: list[tuple[int, int, int]] = []
+ read_raw = bounded._read_raw
+
+ def observe(start: int, end: int, columns: np.ndarray) -> np.ndarray:
+ observed.append((start, end, len(columns)))
+ return read_raw(start, end, columns)
+
+ monkeypatch.setattr(bounded, "_read_raw", observe)
+ _ = bounded.raw_expression_fingerprint
+
+ assert max(end - start for start, end, _ in observed) == 2
+ assert all(width == counts.shape[1] for _, _, width in observed)
+
+
+def test_aligned_feature_stream_reads_read_only_counts_without_zarr_writes() -> None:
+ counts = np.array([[2, 3, 4], [5, 7, 11]], dtype=np.uint32)
+ assay, store, _ = _query_assay(
+ counts,
+ ["a", "b", "extra"],
+ chunks=(1, 2),
+ read_only=True,
+ )
+ store.reset()
+ stream = _stream(assay)
+
+ _collect(stream)
+ _ = stream.raw_expression_fingerprint
+
+ assert all(operation == "get" for operation, _ in store.ops)
+ assert all("normed__" not in key for _, key in store.ops)
+ assert list(assay.z.array_keys()) == ["counts"]
+
+
+@pytest.mark.parametrize("policy", ["intersection", "mean", "", "ERROR"])
+def test_aligned_feature_stream_rejects_unsupported_policies(policy: str) -> None:
+ assay, _, _ = _query_assay(
+ np.array([[1, 2]], dtype=np.uint32),
+ ["a", "b"],
+ chunks=(1, 1),
+ )
+
+ with pytest.raises(ValueError, match="missing_feature_policy"):
+ _stream(assay, policy=policy)
+
+
+def test_aligned_feature_stream_rejects_duplicate_and_disjoint_ids() -> None:
+ counts = np.array([[1, 2]], dtype=np.uint32)
+ assay, _, _ = _query_assay(counts, ["a", "b"], chunks=(1, 1))
+ with pytest.raises(
+ ValueError, match="Reference feature identifiers must be unique"
+ ):
+ _stream(
+ assay,
+ reference_ids=np.array(["a", "a"]),
+ means=np.zeros(2),
+ )
+
+ duplicate_query, _, _ = _query_assay(
+ counts,
+ ["a", "a"],
+ chunks=(1, 1),
+ )
+ with pytest.raises(ValueError, match="Query feature identifiers must be unique"):
+ _stream(duplicate_query)
+
+ with pytest.raises(ValueError, match="No reference features overlap"):
+ _stream(
+ assay,
+ reference_ids=np.array(["x", "y"]),
+ means=np.zeros(2),
+ )
+
+
+@pytest.mark.parametrize(
+ "means",
+ [
+ np.array([1.0, 2.0]),
+ np.array([1.0, np.nan, 3.0]),
+ np.array([1.0, np.inf, 3.0]),
+ np.array([True, False, True]),
+ np.array([1 + 2j, 2 + 0j, 3 + 0j]),
+ ],
+)
+def test_aligned_feature_stream_rejects_invalid_reference_means(
+ means: np.ndarray,
+) -> None:
+ assay, _, _ = _query_assay(
+ np.array([[1, 2]], dtype=np.uint32),
+ ["a", "b"],
+ chunks=(1, 1),
+ )
+
+ with pytest.raises(ValueError, match="Reference normalized means"):
+ _stream(assay, means=means)
+
+
+@pytest.mark.parametrize("size_factor", [0.0, -1.0, np.nan, np.inf])
+def test_aligned_feature_stream_rejects_invalid_reference_size_factor(
+ size_factor: float,
+) -> None:
+ assay, _, _ = _query_assay(
+ np.array([[1, 2]], dtype=np.uint32),
+ ["a", "b"],
+ chunks=(1, 1),
+ )
+
+ with pytest.raises(ValueError, match="size_factor must be finite and positive"):
+ _stream(assay, normalization=_normalization(size_factor=size_factor))
+
+
+def test_aligned_feature_stream_rejects_unknown_reference_normalizer() -> None:
+ assay, _, _ = _query_assay(
+ np.array([[1, 2]], dtype=np.uint32),
+ ["a", "b"],
+ chunks=(1, 1),
+ )
+ normalization = _normalization()
+ normalization["normalization_method"] = {
+ "module": "custom.normalization",
+ "qualname": "normalize",
+ }
+
+ with pytest.raises(ValueError, match="Unsupported reference normalization method"):
+ _stream(assay, normalization=normalization)
diff --git a/tests/test_mapping_label_transfer.py b/tests/test_mapping_label_transfer.py
new file mode 100644
index 00000000..719f4de6
--- /dev/null
+++ b/tests/test_mapping_label_transfer.py
@@ -0,0 +1,736 @@
+from dataclasses import replace
+from pathlib import Path
+import shutil
+
+import numpy as np
+import pandas as pd
+import pytest
+
+import scarf.datastore._operations.mapping as mapping_operations
+import scarf.mapping.projection as projection_storage
+from scarf.datastore.datastore import DataStore
+from scarf.mapping.models import MappingResult
+from scarf.mapping.projection import (
+ NO_QUERY_BATCH_FINGERPRINT,
+ ProjectionWriter,
+ load_projection,
+ plan_projection,
+)
+from scarf.metadata.artifacts import (
+ link_cell_data_column,
+ plan_cell_data_artifact,
+ write_cell_data_artifact,
+)
+from scarf.storage.artifact_writer import (
+ finish_artifact,
+ plan_artifact,
+ start_artifact,
+)
+from scarf.storage.artifacts import (
+ ArtifactRef,
+ ExternalArtifactRef,
+ artifact_group,
+ fingerprint_array,
+)
+from scarf.storage.selections import resolve_selection_artifact
+from scarf.storage.types import as_zarr_array as checked_zarr_array
+
+
+class _RecordingProjectionArray:
+ def __init__(self, array, name: str, reads: list[tuple[str, object]]) -> None:
+ self._array = array
+ self._name = name
+ self._reads = reads
+
+ def __getattr__(self, name: str):
+ return getattr(self._array, name)
+
+ def __getitem__(self, key):
+ if key == slice(None, None, None) or key is Ellipsis:
+ raise AssertionError(f"{self._name} was materialized with a full slice")
+ self._reads.append((self._name, key))
+ return self._array[key]
+
+ def __array__(self, dtype=None, copy=None):
+ raise AssertionError(f"{self._name} was materialized as a complete array")
+
+
+def _record_projection_reads(monkeypatch, reads: list[tuple[str, object]]) -> None:
+ def recording_array(node, *, name: str = ""):
+ array = checked_zarr_array(node, name=name)
+ if name in {"indices", "distances", "uninformative"}:
+ return _RecordingProjectionArray(array, name, reads)
+ return array
+
+ monkeypatch.setattr(mapping_operations, "as_zarr_array", recording_array)
+ monkeypatch.setattr(projection_storage, "as_zarr_array", recording_array)
+
+
+def _plain_reference(datastore):
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.neighbors is not None
+ return datastore.build_mapping_reference(state.neighbors)
+
+
+def _copied_query(datastore, path: Path, *, zarr_mode: str = "r+") -> DataStore:
+ shutil.copytree(datastore.zarr_loc, path)
+ return DataStore(
+ str(path),
+ default_assay="RNA",
+ zarr_mode=zarr_mode,
+ )
+
+
+def _snapshot_store(path: str) -> dict[str, bytes]:
+ root = Path(path)
+ return {
+ str(file.relative_to(root)): file.read_bytes()
+ for file in root.rglob("*")
+ if file.is_file()
+ }
+
+
+def _write_projection(
+ query,
+ reference,
+ *,
+ mapping_name: str,
+ indices: np.ndarray,
+ distances: np.ndarray,
+ uninformative: np.ndarray,
+ cell_key: str = "mapping_cells",
+ feature_coverage: float = 0.75,
+) -> MappingResult:
+ index_values = np.asarray(indices, dtype=np.uint64)
+ distance_values = np.asarray(distances, dtype=np.float64)
+ uninformative_values = np.asarray(uninformative, dtype=bool)
+ n_cells = len(index_values)
+ if cell_key == "I":
+ cell_mask = np.asarray(query.cells.fetch_all("I"), dtype=bool)
+ assert int(cell_mask.sum()) == n_cells
+ else:
+ cell_mask = np.zeros(query.cells.N, dtype=bool)
+ cell_mask[:n_cells] = True
+ query.cells.insert(cell_key, cell_mask, overwrite=True)
+ cell_selection = resolve_selection_artifact(
+ query.zw,
+ scope="datastore",
+ kind="cell_selection",
+ values=cell_mask,
+ row_ids=np.asarray(query.cells.fetch_all("ids")),
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column=cell_key,
+ )
+ feature_mask = np.asarray(query.RNA.feats.fetch_all("I"), dtype=bool)
+ feature_selection = resolve_selection_artifact(
+ query.zw,
+ scope="assay",
+ assay="RNA",
+ kind="feature_selection",
+ values=feature_mask,
+ row_ids=np.asarray(query.RNA.feats.fetch_all("ids")),
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column="I",
+ )
+ planned = plan_projection(
+ query.zw,
+ query_assay="RNA",
+ mapping_name=mapping_name,
+ n_cells=n_cells,
+ save_k=index_values.shape[1],
+ missing_feature_policy="reference_mean",
+ correction_method="none",
+ cell_selection=cell_selection,
+ feature_selection=feature_selection,
+ selected_expression_fingerprint="e" * 64,
+ query_batch_fingerprint=NO_QUERY_BATCH_FINGERPRINT,
+ mapping_reference=reference.external_ref,
+ reference_cell_count=reference.selected_cell_count,
+ )
+ writer = ProjectionWriter(
+ query.zw,
+ planned,
+ chunk_rows=max(1, min(n_cells, 2)),
+ )
+ writer.write_block(
+ 0,
+ index_values,
+ distance_values,
+ uninformative_values,
+ )
+ ref = writer.finish(
+ {
+ "featureCoverage": float(feature_coverage),
+ "queryBatchCount": 1,
+ "algorithmVariant": "scaled_pca",
+ "zeroNormCellCount": int(np.count_nonzero(uninformative_values)),
+ "queryScaledDispersion": 1.0,
+ }
+ )
+ return load_projection(query.zw, ref, reference=reference)
+
+
+def _write_reference_labels(reference, name: str = "reference_labels") -> np.ndarray:
+ labels = np.full(reference.selected_cell_count, "other", dtype=object)
+ labels[:2] = ["winner", "runner_up"]
+ reference.datastore.cells.insert(
+ name,
+ labels,
+ key=reference.cell_key,
+ overwrite=True,
+ )
+ return labels
+
+
+def _write_reference_layout(
+ reference,
+ *,
+ layout_key: str,
+ linked: bool,
+) -> tuple[np.ndarray, ArtifactRef | None]:
+ first = np.arange(reference.selected_cell_count, dtype=np.float64) * 10
+ layout = np.column_stack((first, first + 10))
+ source_ref = None
+ if linked:
+ planned = plan_cell_data_artifact(
+ reference.datastore.zw,
+ scope="assay",
+ assay=reference.assay_name,
+ kind="embedding",
+ operation="manual_reference_embedding",
+ parameters={"layout_key": layout_key},
+ inputs={},
+ execution_options={},
+ cell_selection=reference.cell_selection,
+ arrays={"values": (layout.shape, "f")},
+ )
+ write_cell_data_artifact(
+ reference.datastore.zw,
+ planned,
+ {"values": layout},
+ )
+ source_ref = planned.ref
+ for dimension in range(2):
+ column = f"{layout_key}{dimension + 1}"
+ reference.datastore.cells.insert(
+ column,
+ layout[:, dimension],
+ key=reference.cell_key,
+ overwrite=True,
+ )
+ if source_ref is not None:
+ link_cell_data_column(
+ reference.datastore.zw,
+ column,
+ source_ref,
+ value_name="values",
+ value_index=dimension,
+ )
+ return layout, source_ref
+
+
+@pytest.fixture
+def mapping_consumer_context(analyzed_datastore_ephemeral, tmp_path):
+ reference_store = analyzed_datastore_ephemeral
+ reference = _plain_reference(reference_store)
+ query = _copied_query(reference_store, tmp_path / "query.zarr")
+ return reference_store, reference, query
+
+
+def test_mapping_result_resolves_all_handles_and_rejects_reference_ambiguity(
+ mapping_consumer_context,
+):
+ _, reference, query = mapping_consumer_context
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name="atlas",
+ indices=np.array([[0, 1], [1, 0]]),
+ distances=np.array([[1.0, 9.0], [9.0, 1.0]]),
+ uninformative=np.array([False, False]),
+ )
+
+ by_session = query.get_mapping_result(result, load_arrays=True)
+ by_ref = query.get_mapping_result(result.ref, reference=reference)
+ by_name = query.get_mapping_result(
+ "atlas",
+ reference=reference,
+ query_assay="RNA",
+ )
+
+ assert by_session.ref == by_ref.ref == by_name.ref == result.ref
+ assert by_session.reference is reference
+ assert by_session.indices is not None
+ with pytest.raises(ValueError, match="MappingReference is required"):
+ query.get_mapping_result(result.ref)
+ with pytest.raises(ValueError, match="MappingReference is required"):
+ query.get_mapping_result("atlas")
+ with pytest.raises(ValueError, match="only valid when result is a string"):
+ query.get_mapping_result(
+ result.ref,
+ reference=reference,
+ query_assay="RNA",
+ )
+
+ mismatched = replace(
+ reference,
+ ref=ArtifactRef(
+ scope="assay",
+ assay=reference.assay_name,
+ kind="mapping_reference",
+ artifact_id="0" * 64,
+ ),
+ )
+ with pytest.raises(ValueError, match="does not match.*reference handle"):
+ query.get_mapping_result(result, reference=mismatched)
+
+
+def test_mapping_scores_exclude_uninformative_rows_and_preserve_groups(
+ mapping_consumer_context,
+):
+ _, reference, query = mapping_consumer_context
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name="scores",
+ indices=np.array([[0, 1], [0, 1], [1, 2], [2, 3]]),
+ distances=np.array([[1.0, 9.0], [1.0, 1.0], [1.0, 1.0], [1.0, 1.0]]),
+ uninformative=np.array([False, True, True, True]),
+ )
+ groups = np.array(["informative", "informative", "empty", "empty"])
+
+ weighted = list(
+ query.get_mapping_score(
+ result,
+ target_groups=groups,
+ log_transform=False,
+ multiplier=1.0,
+ )
+ )
+ unweighted = list(
+ query.get_mapping_score(
+ result.ref,
+ target_groups=groups,
+ reference=reference,
+ log_transform=False,
+ multiplier=1.0,
+ weighted=False,
+ fixed_weight=0.2,
+ )
+ )
+
+ assert [group for group, _ in weighted] == ["informative", "empty"]
+ assert [group for group, _ in unweighted] == ["informative", "empty"]
+ assert weighted[0][1].shape == (reference.selected_cell_count,)
+ # Published weight, 1 / (log(distance + 1) + 1), divided by one informative
+ # query cell times two neighbors.
+ expected = 1.0 / (np.log1p(np.array([1.0, 9.0])) + 1.0) / 2.0
+ np.testing.assert_allclose(weighted[0][1][:2], expected)
+ np.testing.assert_array_equal(weighted[1][1], 0.0)
+ np.testing.assert_allclose(unweighted[0][1][:2], [0.1, 0.1])
+ np.testing.assert_array_equal(unweighted[1][1], 0.0)
+
+ with pytest.raises(ValueError, match="one value per projected query cell"):
+ list(query.get_mapping_score(result, target_groups=np.array(["short"])))
+ with pytest.raises(ValueError, match="fixed_weight"):
+ list(query.get_mapping_score(result, fixed_weight=0.0))
+ with pytest.raises(TypeError, match="weighted"):
+ list(query.get_mapping_score(result, weighted=1))
+
+
+def test_labels_and_evidence_abstain_without_fabricating_metrics(
+ mapping_consumer_context,
+):
+ _, reference, query = mapping_consumer_context
+ _write_reference_labels(reference)
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name="labels",
+ indices=np.array([[0, 1], [0, 1], [1, 0]]),
+ distances=np.array([[1.0, 9.0], [1.0, 9.0], [1.0, 9.0]]),
+ uninformative=np.array([False, True, False]),
+ feature_coverage=0.625,
+ )
+
+ labels = query.get_target_classes(
+ result,
+ reference_class_group="reference_labels",
+ threshold_fraction=0.75,
+ )
+ evidence = query.get_target_label_evidence(
+ result.ref,
+ reference=reference,
+ reference_class_group="reference_labels",
+ threshold_fraction=0.75,
+ calibration_nonconformity=np.array([0.1, 0.2, 0.3]),
+ conformal_alpha=0.2,
+ )
+
+ assert isinstance(labels, pd.Series)
+ assert labels.tolist() == ["winner", "NA", "runner_up"]
+ assert evidence["label"].tolist() == ["winner", "NA", "runner_up"]
+ assert evidence["isUnknown"].tolist() == [False, True, False]
+ assert evidence["featureCoverage"].tolist() == [0.625] * 3
+ for column in (
+ "voteFraction",
+ "voteEntropy",
+ "topTwoMargin",
+ "referenceDistancePercentile",
+ ):
+ assert np.isnan(evidence.loc[1, column])
+ assert np.isfinite(evidence.loc[[0, 2], column]).all()
+ np.testing.assert_allclose(evidence.loc[[0, 2], "voteFraction"], [0.9, 0.9])
+ assert evidence.loc[1, "predictionSet"] == ()
+ assert "reference_labels" not in query.cells.columns
+
+
+def test_mapping_consumers_stream_projection_arrays(
+ mapping_consumer_context,
+ monkeypatch,
+):
+ _, reference, query = mapping_consumer_context
+ _write_reference_labels(reference)
+ _write_reference_layout(
+ reference,
+ layout_key="stream_layout",
+ linked=False,
+ )
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name="streaming",
+ indices=np.array([[0, 1], [0, 1], [1, 0], [1, 0]]),
+ distances=np.array([[1.0, 9.0], [2.0, 3.0], [1.0, 9.0], [4.0, 5.0]]),
+ uninformative=np.array([False, True, False, False]),
+ )
+ reads: list[tuple[str, object]] = []
+ _record_projection_reads(monkeypatch, reads)
+
+ consumers = (
+ lambda: list(query.get_mapping_score(result)),
+ lambda: query.get_target_classes(
+ result,
+ reference_class_group="reference_labels",
+ ),
+ lambda: query.get_target_label_evidence(
+ result,
+ reference_class_group="reference_labels",
+ ),
+ lambda: query.project_reference_embedding(
+ result,
+ reference_layout_key="stream_layout",
+ label="streamed_ref",
+ ),
+ )
+ for consume in consumers:
+ reads.clear()
+ consume()
+ assert {name for name, _ in reads} == {
+ "indices",
+ "distances",
+ "uninformative",
+ }
+ for _, key in reads:
+ assert isinstance(key, slice)
+ assert key.start is not None
+ assert key.stop is not None
+ assert 0 < key.stop - key.start <= 2
+
+
+def test_label_scores_are_allocated_only_for_conformal_evidence(
+ mapping_consumer_context,
+ monkeypatch,
+):
+ _, reference, query = mapping_consumer_context
+ labels = _write_reference_labels(reference)
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name="conformal_allocation",
+ indices=np.array([[0, 1], [1, 0]]),
+ distances=np.array([[1.0, 9.0], [1.0, 9.0]]),
+ uninformative=np.array([False, False]),
+ )
+ allocations: list[object] = []
+
+ class _NumpyProxy:
+ def __getattr__(self, name: str):
+ return getattr(np, name)
+
+ def zeros(self, shape, *args, **kwargs):
+ allocations.append(shape)
+ return np.zeros(shape, *args, **kwargs)
+
+ monkeypatch.setattr(mapping_operations, "np", _NumpyProxy())
+ score_shape = (result.n_cells, len(pd.unique(labels)))
+
+ query.get_target_label_evidence(
+ result,
+ reference_class_group="reference_labels",
+ )
+ assert score_shape not in allocations
+
+ allocations.clear()
+ query.get_target_label_evidence(
+ result,
+ reference_class_group="reference_labels",
+ calibration_nonconformity=np.array([0.1, 0.2]),
+ )
+ assert score_shape in allocations
+
+
+def test_reference_embedding_persists_reuses_and_links_selected_query_columns(
+ mapping_consumer_context,
+):
+ reference_store, reference, query = mapping_consumer_context
+ reference_layout, source_ref = _write_reference_layout(
+ reference,
+ layout_key="RNA_UMAP",
+ linked=True,
+ )
+ assert source_ref is not None
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name="embedding",
+ indices=np.array([[0, 1], [0, 1], [1, 0]]),
+ distances=np.array([[1.0, 9.0], [1.0, 1.0], [1.0, 9.0]]),
+ uninformative=np.array([False, False, True]),
+ )
+ reference_before = _snapshot_store(reference_store.zarr_loc)
+
+ embedding = query.project_reference_embedding(
+ result,
+ reference_layout_key="RNA_UMAP",
+ )
+
+ assert _snapshot_store(reference_store.zarr_loc) == reference_before
+ status = query.inspect_artifact(embedding)
+ assert status.operation == "project_reference_embedding"
+ assert ArtifactRef.from_dict((status.inputs or {})["projection"]) == result.ref
+ assert ExternalArtifactRef.from_dict(
+ (status.inputs or {})["reference_layout"]
+ ) == reference.external_ref.__class__(
+ dataset_fingerprint=reference.dataset_fingerprint,
+ ref=source_ref,
+ )
+ columns = [
+ "RNA_mapping_cells_ref_UMAP1",
+ "RNA_mapping_cells_ref_UMAP2",
+ ]
+ values = np.column_stack(
+ [query.cells.fetch(column, key="mapping_cells") for column in columns]
+ )
+ np.testing.assert_allclose(values[0], [1.0, 11.0])
+ np.testing.assert_allclose(values[1], [5.0, 15.0])
+ assert np.isnan(values[2]).all()
+ for dimension, column in enumerate(columns):
+ attrs = query.zw["cellData"][column].attrs
+ assert ArtifactRef.from_dict(attrs["source_artifact"]) == embedding
+ assert attrs["source_value"] == "values"
+ assert attrs["value_index"] == dimension
+
+ preserved_display = {
+ "kind": "continuous",
+ "colormap": "magma",
+ "minimum": -1.0,
+ "maximum": 20.0,
+ "scale": "linear",
+ }
+ query.zw["cellData"][columns[0]].attrs["display"] = preserved_display
+ reused = query.project_reference_embedding(
+ result.ref,
+ reference=reference,
+ reference_layout_key="RNA_UMAP",
+ )
+ assert reused == embedding
+ assert dict(query.zw["cellData"][columns[0]].attrs["display"]) == preserved_display
+ np.testing.assert_array_equal(
+ artifact_group(query.zw, reused)["values"][:],
+ values,
+ )
+ assert _snapshot_store(reference_store.zarr_loc) == reference_before
+ np.testing.assert_array_equal(reference.fetch_layout("RNA_UMAP"), reference_layout)
+
+
+def test_reference_embedding_fingerprints_unlinked_layout_and_uses_i_naming(
+ mapping_consumer_context,
+):
+ _, reference, query = mapping_consumer_context
+ reference_layout, source_ref = _write_reference_layout(
+ reference,
+ layout_key="manual_layout",
+ linked=False,
+ )
+ assert source_ref is None
+ n_cells = len(query.cells.active_index("I"))
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name="fallback_embedding",
+ indices=np.tile(np.array([[0, 1]], dtype=np.uint64), (n_cells, 1)),
+ distances=np.ones((n_cells, 2), dtype=np.float64),
+ uninformative=np.zeros(n_cells, dtype=bool),
+ cell_key="I",
+ )
+
+ embedding = query.project_reference_embedding(
+ result,
+ reference_layout_key="manual_layout",
+ label="manual_ref",
+ )
+
+ status = query.inspect_artifact(embedding)
+ assert (status.inputs or {})["reference_layout"] == {
+ "value_fingerprint": fingerprint_array(reference_layout)
+ }
+ assert "RNA_manual_ref1" in query.cells.columns
+ assert "RNA_manual_ref2" in query.cells.columns
+
+
+@pytest.mark.parametrize(
+ ("attribute", "malformed_value"),
+ (
+ ("value_index", 0),
+ ("source_value", "other_values"),
+ ),
+)
+def test_reference_embedding_fingerprints_malformed_layout_links(
+ mapping_consumer_context,
+ attribute,
+ malformed_value,
+):
+ _, reference, query = mapping_consumer_context
+ reference_layout, source_ref = _write_reference_layout(
+ reference,
+ layout_key="malformed_layout",
+ linked=True,
+ )
+ assert source_ref is not None
+ reference.datastore.zw["cellData"]["malformed_layout2"].attrs[attribute] = (
+ malformed_value
+ )
+ assert reference.layout_source("malformed_layout") is None
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name=f"malformed_{attribute}",
+ indices=np.array([[0, 1], [1, 0]]),
+ distances=np.ones((2, 2), dtype=np.float64),
+ uninformative=np.zeros(2, dtype=bool),
+ )
+
+ embedding = query.project_reference_embedding(
+ result,
+ reference_layout_key="malformed_layout",
+ label=f"malformed_{attribute}",
+ )
+
+ status = query.inspect_artifact(embedding)
+ assert (status.inputs or {})["reference_layout"] == {
+ "value_fingerprint": fingerprint_array(reference_layout)
+ }
+
+
+def test_reference_layout_source_requires_the_linked_value_array(
+ mapping_consumer_context,
+):
+ _, reference, _ = mapping_consumer_context
+ _write_reference_layout(
+ reference,
+ layout_key="missing_source_values",
+ linked=True,
+ )
+ for column in ("missing_source_values1", "missing_source_values2"):
+ reference.datastore.zw["cellData"][column].attrs["source_value"] = "missing"
+
+ assert reference.layout_source("missing_source_values") is None
+
+
+def test_reference_layout_reads_linked_immutable_artifact(
+ mapping_consumer_context,
+):
+ _, reference, _ = mapping_consumer_context
+ expected, _ = _write_reference_layout(
+ reference,
+ layout_key="linked_layout",
+ linked=True,
+ )
+ for column in ("linked_layout1", "linked_layout2"):
+ values = reference.datastore.zw["cellData"][column]
+ values[:] = np.full(values.shape, -999.0)
+
+ np.testing.assert_array_equal(reference.fetch_layout("linked_layout"), expected)
+
+
+def test_reference_embedding_requires_writable_query_store(
+ mapping_consumer_context,
+):
+ _, reference, query = mapping_consumer_context
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name="read_only",
+ indices=np.array([[0, 1]]),
+ distances=np.array([[1.0, 1.0]]),
+ uninformative=np.array([False]),
+ )
+ read_only = DataStore(
+ query.zarr_loc,
+ default_assay="RNA",
+ zarr_mode="r",
+ )
+
+ with pytest.raises(ValueError, match="read-write query store"):
+ read_only.project_reference_embedding(
+ result,
+ reference_layout_key="unused",
+ )
+
+
+def test_every_mapping_consumer_rejects_old_projection_artifacts(
+ mapping_consumer_context,
+):
+ _, reference, query = mapping_consumer_context
+ planned = plan_artifact(
+ query.zw,
+ scope="assay",
+ assay="RNA",
+ kind="projection",
+ operation="map_with_reference",
+ parameters={},
+ inputs={},
+ execution_options={},
+ )
+ group = start_artifact(query.zw, planned)
+ finish_artifact(group, planned)
+ old = planned.ref
+ consumers = (
+ lambda: query.get_mapping_result(old, reference=reference),
+ lambda: list(query.get_mapping_score(old, reference=reference)),
+ lambda: query.get_target_classes(
+ old,
+ reference_class_group="ids",
+ reference=reference,
+ ),
+ lambda: query.get_target_label_evidence(
+ old,
+ reference_class_group="ids",
+ reference=reference,
+ ),
+ lambda: query.project_reference_embedding(
+ old,
+ reference_layout_key="RNA_UMAP",
+ reference=reference,
+ ),
+ )
+
+ for consumer in consumers:
+ with pytest.raises(ValueError, match="Re-run run_mapping"):
+ consumer()
diff --git a/tests/test_mapping_orchestration.py b/tests/test_mapping_orchestration.py
new file mode 100644
index 00000000..4a073dd7
--- /dev/null
+++ b/tests/test_mapping_orchestration.py
@@ -0,0 +1,630 @@
+from pathlib import Path
+import shutil
+
+import numpy as np
+import pandas as pd
+import pytest
+
+import scarf.datastore._operations.mapping as mapping_operations
+from scarf.datastore.datastore import DataStore, mount_datastore
+from scarf.mapping.features import AlignedFeatureStream
+from scarf.mapping.projection import load_projection, resolve_projection
+from scarf.storage.artifacts import (
+ ArtifactRef,
+ ExternalArtifactRef,
+ artifact_group,
+ list_artifacts,
+)
+
+
+def _snapshot_store(path: str) -> dict[str, bytes]:
+ root = Path(path)
+ return {
+ str(file.relative_to(root)): file.read_bytes()
+ for file in root.rglob("*")
+ if file.is_file()
+ }
+
+
+def _plain_reference(datastore):
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.neighbors is not None
+ return datastore.build_mapping_reference(state.neighbors)
+
+
+def _symphony_reference(datastore):
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.reduction is not None
+ datastore.cells.insert(
+ "mapping_batch",
+ np.where(np.arange(datastore.cells.N) % 2, "a", "b"),
+ overwrite=True,
+ )
+ correction = datastore.run_harmony(
+ ["mapping_batch"],
+ state.reduction,
+ harmony_params={"nclust": 5},
+ update_state=False,
+ )
+ ann_index = datastore.build_ann_index(correction, update_state=False)
+ neighbors = datastore.query_neighbors(
+ ann_index,
+ coordinates=correction,
+ k=3,
+ update_state=False,
+ )
+ return datastore.build_mapping_reference(neighbors)
+
+
+def _copied_query(datastore, path: Path, *, zarr_mode: str = "r+") -> DataStore:
+ shutil.copytree(datastore.zarr_loc, path)
+ return DataStore(
+ str(path),
+ default_assay="RNA",
+ zarr_mode=zarr_mode,
+ )
+
+
+def _changed_files(
+ before: dict[str, bytes],
+ after: dict[str, bytes],
+) -> set[str]:
+ return {
+ path
+ for path in before.keys() | after.keys()
+ if before.get(path) != after.get(path)
+ }
+
+
+def test_plain_mapping_is_query_owned_and_reuses_exact_projection(
+ analyzed_datastore_ephemeral,
+ tmp_path,
+ monkeypatch,
+):
+ reference_store = analyzed_datastore_ephemeral
+ reference = _plain_reference(reference_store)
+ query = _copied_query(reference_store, tmp_path / "query.zarr")
+ reference_before = _snapshot_store(reference_store.zarr_loc)
+ query_before = _snapshot_store(query.zarr_loc)
+ query_sf = query.RNA.sf
+ query_scalar = query.RNA.scalar
+ query_attrs = dict(query.RNA.attrs)
+ query_ncounts = np.array(query.cells.fetch_all("RNA_nCounts"), copy=True)
+ query_feature_columns = set(query.RNA.feats.columns)
+ available_k = int(
+ artifact_group(reference_store.zw, reference.neighbors)["indices"].shape[1]
+ )
+ project_pca = mapping_operations.project_pca
+
+ def project_with_zero_row(values, model):
+ projected = project_pca(values, model)
+ projected[0] = 0
+ return projected
+
+ monkeypatch.setattr(mapping_operations, "project_pca", project_with_zero_row)
+ warning_messages = []
+ monkeypatch.setattr(
+ "scarf.datastore._operations.mapping.logger.warning",
+ warning_messages.append,
+ )
+
+ result = query.run_mapping(
+ reference,
+ "plain",
+ save_k=available_k + 10,
+ )
+
+ assert _snapshot_store(reference_store.zarr_loc) == reference_before
+ assert len(warning_messages) == 1
+ assert "save_k" in warning_messages[0]
+ assert result.reference is reference
+ assert result.mapping_name == "plain"
+ assert result.correction_method == "none"
+ assert result.indices is None
+ assert result.distances is None
+ assert result.uninformative is None
+ assert set(result.diagnostics) == {
+ "featureCoverage",
+ "queryBatchCount",
+ "algorithmVariant",
+ "zeroNormCellCount",
+ "queryScaledDispersion",
+ }
+ assert result.diagnostics["algorithmVariant"] == "scaled_pca"
+ assert result.diagnostics["queryBatchCount"] == 1
+ assert result.diagnostics["zeroNormCellCount"] > 0
+ assert query.RNA.sf == query_sf
+ assert query.RNA.scalar is query_scalar
+ assert dict(query.RNA.attrs) == query_attrs
+ np.testing.assert_array_equal(
+ query.cells.fetch_all("RNA_nCounts"),
+ query_ncounts,
+ )
+ assert set(query.RNA.feats.columns) == query_feature_columns
+
+ status = query.inspect_artifact(result.ref)
+ assert status.parameters == {
+ "mapping_name": "plain",
+ "save_k": available_k,
+ "missing_feature_policy": "reference_mean",
+ "correction_method": "none",
+ }
+ assert set(status.inputs or {}) == {
+ "cell_selection",
+ "feature_selection",
+ "selected_expression_fingerprint",
+ "query_batch_fingerprint",
+ "mapping_reference",
+ }
+ assert (
+ ExternalArtifactRef.from_dict((status.inputs or {})["mapping_reference"])
+ == reference.external_ref
+ )
+ lineage = query.lineage(result.ref, references=reference)
+ assert reference.external_ref in lineage.graph
+ assert all(
+ node["status"] is not None and node["status"].complete
+ for _, node in lineage.graph.nodes(data=True)
+ )
+ cell_selection = ArtifactRef.from_dict((status.inputs or {})["cell_selection"])
+ feature_selection = ArtifactRef.from_dict(
+ (status.inputs or {})["feature_selection"]
+ )
+ np.testing.assert_array_equal(
+ artifact_group(query.zw, cell_selection)["values"][:],
+ query.cells.fetch_all("I"),
+ )
+ query_feature_ids = np.asarray(query.RNA.feats.fetch_all("ids")).astype(str)
+ np.testing.assert_array_equal(
+ artifact_group(query.zw, feature_selection)["values"][:],
+ np.isin(query_feature_ids, reference.feature_ids),
+ )
+ feature_status = query.inspect_artifact(feature_selection)
+ assert isinstance(
+ (feature_status.inputs or {}).get("alignment_map_hash"),
+ str,
+ )
+ group = artifact_group(query.zw, result.ref)
+ assert set(group.array_keys()) == {"indices", "distances", "uninformative"}
+ assert set(group.group_keys()) == set()
+ assert set(group.attrs) == {
+ "artifact_id",
+ "kind",
+ "provenance",
+ "execution_options",
+ "created_at_ns",
+ "complete",
+ "diagnostics",
+ }
+ loaded = load_projection(
+ query.zw,
+ result.ref,
+ load_arrays=True,
+ reference=reference,
+ )
+ assert loaded.indices is not None
+ assert loaded.indices.shape == (result.n_cells, available_k)
+ assert loaded.distances is not None
+ assert loaded.distances.shape == loaded.indices.shape
+ assert loaded.uninformative is not None
+ assert (
+ int(np.count_nonzero(loaded.uninformative))
+ == result.diagnostics["zeroNormCellCount"]
+ )
+
+ changed = _changed_files(query_before, _snapshot_store(query.zarr_loc))
+ assert not any("aligned" in path or "normed" in path for path in changed)
+
+ reuse_before = _snapshot_store(query.zarr_loc)
+
+ def fail_ann_load(*args, **kwargs):
+ raise AssertionError("A reused mapping must not load or query the ANN")
+
+ monkeypatch.setattr(
+ "scarf.datastore._operations.mapping._load_reference_neighbor_query",
+ fail_ann_load,
+ )
+ reused = query.run_mapping(reference, "plain", save_k=available_k + 10)
+ assert reused.ref == result.ref
+ assert reused.reference is reference
+ assert reused.diagnostics == result.diagnostics
+ assert _snapshot_store(query.zarr_loc) == reuse_before
+
+
+def test_mapping_failure_leaves_projection_incomplete(
+ analyzed_datastore_ephemeral,
+ tmp_path,
+ monkeypatch,
+):
+ reference_store = analyzed_datastore_ephemeral
+ reference = _plain_reference(reference_store)
+ query = _copied_query(reference_store, tmp_path / "query.zarr")
+ before = set(
+ list_artifacts(
+ query.zw,
+ scope="assay",
+ assay="RNA",
+ kind="projection",
+ )
+ )
+
+ class FailingNeighborQuery:
+ @staticmethod
+ def query(_values):
+ raise RuntimeError("injected ANN failure")
+
+ monkeypatch.setattr(
+ mapping_operations,
+ "_load_reference_neighbor_query",
+ lambda *_args, **_kwargs: FailingNeighborQuery(),
+ )
+ with pytest.raises(RuntimeError, match="injected ANN failure"):
+ query.run_mapping(reference, "failure")
+
+ created = (
+ set(
+ list_artifacts(
+ query.zw,
+ scope="assay",
+ assay="RNA",
+ kind="projection",
+ )
+ )
+ - before
+ )
+ assert len(created) == 1
+ failed = query.inspect_artifact(created.pop())
+ assert failed.exists
+ assert not failed.complete
+
+
+def test_mapping_guards_precede_query_writes(
+ analyzed_datastore_ephemeral,
+ tmp_path,
+):
+ reference_store = analyzed_datastore_ephemeral
+ reference = _plain_reference(reference_store)
+ reference_before = _snapshot_store(reference_store.zarr_loc)
+
+ with pytest.raises(ValueError, match="same physical Zarr store"):
+ reference_store.run_mapping(reference, "same_store")
+ assert _snapshot_store(reference_store.zarr_loc) == reference_before
+
+ reopened = DataStore(reference_store.zarr_loc, default_assay="RNA")
+ with pytest.raises(ValueError, match="same physical Zarr store"):
+ reopened.run_mapping(reference, "reopened_same_store")
+ assert _snapshot_store(reference_store.zarr_loc) == reference_before
+
+ read_only = _copied_query(
+ reference_store,
+ tmp_path / "read-only.zarr",
+ zarr_mode="r",
+ )
+ read_only_before = _snapshot_store(read_only.zarr_loc)
+ with pytest.raises(ValueError, match="read-write query datastore"):
+ read_only.run_mapping(reference, "read_only")
+ assert _snapshot_store(read_only.zarr_loc) == read_only_before
+
+ writable = _copied_query(reference_store, tmp_path / "writable.zarr")
+ writable_before = _snapshot_store(writable.zarr_loc)
+ with pytest.raises(ValueError, match="only supported by Symphony"):
+ writable.run_mapping(
+ reference,
+ "plain_batches",
+ query_batches=pd.DataFrame({"batch": ["a"]}),
+ )
+ assert _snapshot_store(writable.zarr_loc) == writable_before
+
+ with pytest.raises(TypeError, match="RNA query assays"):
+ writable.run_mapping(reference, "non_rna", query_assay="assay2")
+ assert _snapshot_store(writable.zarr_loc) == writable_before
+
+ writable.cells.insert(
+ "empty_mapping_selection",
+ np.zeros(writable.cells.N, dtype=bool),
+ overwrite=True,
+ )
+ empty_before = _snapshot_store(writable.zarr_loc)
+ with pytest.raises(ValueError, match="at least one query cell"):
+ writable.run_mapping(
+ reference,
+ "empty",
+ cell_key="empty_mapping_selection",
+ )
+ assert _snapshot_store(writable.zarr_loc) == empty_before
+
+ original_fingerprint = reference_store.RNA.attrs["dataset_fingerprint"]
+ reference_store.RNA.attrs["dataset_fingerprint"] = "changed"
+ fingerprint_before = _snapshot_store(writable.zarr_loc)
+ try:
+ with pytest.raises(ValueError, match="dataset fingerprint mismatch"):
+ writable.run_mapping(reference, "stale_reference")
+ finally:
+ reference_store.RNA.attrs["dataset_fingerprint"] = original_fingerprint
+ assert _snapshot_store(writable.zarr_loc) == fingerprint_before
+
+
+def test_separately_mounted_query_can_map_its_source_reference(
+ analyzed_datastore_ephemeral,
+ tmp_path,
+):
+ reference_store = analyzed_datastore_ephemeral
+ reference = _plain_reference(reference_store)
+ reference_before = _snapshot_store(reference_store.zarr_loc)
+ read_only_reference_store = DataStore(
+ reference_store.zarr_loc,
+ default_assay="RNA",
+ zarr_mode="r",
+ )
+ read_only_reference = read_only_reference_store.get_mapping_reference(reference.ref)
+ query = mount_datastore(
+ reference_store.zarr_loc,
+ at=str(tmp_path / "mounted-query.zarr"),
+ default_assay="RNA",
+ )
+
+ result = query.run_mapping(read_only_reference, "mounted")
+
+ assert result.reference is read_only_reference
+ assert result.n_cells == len(query.cells.active_index("I"))
+ assert _snapshot_store(reference_store.zarr_loc) == reference_before
+
+
+def test_query_projection_reproduces_stored_reference_coordinates(
+ analyzed_datastore_ephemeral,
+):
+ reference_store = analyzed_datastore_ephemeral
+ reference = _plain_reference(reference_store)
+ stored = np.asarray(
+ artifact_group(reference_store.zw, reference.reduction)["data"][:],
+ dtype=np.float64,
+ )
+
+ stream = AlignedFeatureStream(
+ query_assay=reference_store.RNA,
+ query_cell_indices=reference_store.cells.active_index(reference.cell_key),
+ reference_feature_ids=reference.feature_ids,
+ reference_normalized_means=reference.model.feature_means,
+ reference_normalization_parameters=reference.normalization_parameters,
+ missing_feature_policy="error",
+ resources=reference_store.resources,
+ )
+ projected = np.vstack(
+ [
+ mapping_operations.project_pca(block.values, reference.model)
+ for block in stream
+ ]
+ )
+
+ assert stream.feature_coverage == 1.0
+ assert projected.shape == stored.shape
+ np.testing.assert_allclose(
+ projected,
+ stored,
+ rtol=0,
+ atol=1e-4 * float(np.abs(stored).max()),
+ )
+
+
+def test_self_mapping_recovers_the_reference_graph_and_labels(
+ analyzed_datastore_ephemeral,
+ tmp_path,
+):
+ reference_store = analyzed_datastore_ephemeral
+ reference_store.cells.insert(
+ "mapping_labels",
+ np.array([f"c{index % 7}" for index in range(reference_store.cells.N)]),
+ overwrite=True,
+ )
+ reference = _plain_reference(reference_store)
+ graph = artifact_group(reference_store.zw, reference.neighbors)
+ reference_indices = np.asarray(graph["indices"][:])
+ reference_distances = np.asarray(graph["distances"][:], dtype=np.float64)
+ available_k = int(reference_indices.shape[1])
+ assert not (reference_indices[:, 0] == np.arange(len(reference_indices))).all()
+
+ query = mount_datastore(
+ reference_store.zarr_loc,
+ at=str(tmp_path / "self-query.zarr"),
+ default_assay="RNA",
+ )
+ result = query.run_mapping(reference, "self", save_k=available_k)
+ loaded = query.get_mapping_result(result, reference=reference, load_arrays=True)
+ indices = loaded.indices
+ distances = loaded.distances
+ assert indices is not None and distances is not None
+
+ positions = np.arange(result.n_cells)
+ assert result.n_cells == reference.selected_cell_count
+ assert (indices[:, 0] == positions).mean() > 0.95
+ assert float(np.median(distances[:, 0])) < 1e-4
+ # The reference PCA is fitted on z-scored features, so a query that is the
+ # reference has to disperse exactly like it. This anchors the diagnostic:
+ # values well below 1 mean the query occupies a narrower region.
+ assert result.diagnostics["queryScaledDispersion"] == pytest.approx(1.0, abs=0.02)
+
+ agreement = indices[:, 1:] == reference_indices[:, : available_k - 1]
+ assert agreement.mean() > 0.9
+ np.testing.assert_allclose(
+ distances[:, 1:][agreement],
+ reference_distances[:, : available_k - 1][agreement],
+ rtol=1e-4,
+ atol=1e-4,
+ )
+ assert len(np.unique(indices)) > 0.9 * reference.selected_cell_count
+
+ transferred = query.get_target_classes(
+ loaded,
+ reference_class_group="mapping_labels",
+ reference=reference,
+ threshold_fraction=0.6,
+ ).to_numpy()
+ known = np.asarray(query.cells.fetch("mapping_labels"))
+ assert (transferred == known).mean() > 0.95
+
+ evidence = query.get_target_label_evidence(
+ loaded,
+ reference_class_group="mapping_labels",
+ reference=reference,
+ )
+ assert float(np.median(evidence["referenceDistancePercentile"])) == 0.0
+ assert float(np.median(evidence["voteFraction"])) > 0.99
+ assert not evidence["isUnknown"].mean() > 0.05
+
+
+def test_symphony_mapping_validates_batches_and_persists_diagnostics(
+ analyzed_datastore_ephemeral,
+ tmp_path,
+ monkeypatch,
+):
+ reference_store = analyzed_datastore_ephemeral
+ reference = _symphony_reference(reference_store)
+ query = _copied_query(reference_store, tmp_path / "query.zarr")
+ n_cells = len(query.cells.active_index("I"))
+
+ invalid_frames = (
+ pd.DataFrame(index=np.arange(n_cells)),
+ pd.DataFrame({"batch": ["a"] * (n_cells - 1)}),
+ pd.DataFrame(
+ np.column_stack((np.zeros(n_cells), np.ones(n_cells))),
+ columns=["batch", "batch"],
+ ),
+ pd.DataFrame({"batch": [None] + ["a"] * (n_cells - 1)}),
+ )
+ for batches in invalid_frames:
+ before = _snapshot_store(query.zarr_loc)
+ with pytest.raises(ValueError):
+ query.run_mapping(
+ reference,
+ "invalid_batches",
+ query_batches=batches,
+ )
+ assert _snapshot_store(query.zarr_loc) == before
+
+ batches = pd.DataFrame(
+ {
+ "donor": np.where(np.arange(n_cells) % 2, "a", "b"),
+ "library": np.arange(n_cells) % 3,
+ }
+ )
+ accumulated_rows = 0
+ original_accumulate = mapping_operations.accumulate_sufficient_statistics
+
+ def observe_accumulation(
+ counts,
+ sums,
+ coordinates,
+ assignments,
+ batch_codes,
+ ):
+ nonlocal accumulated_rows
+ assert not mapping_operations.zero_norm_rows(coordinates).any()
+ accumulated_rows += len(coordinates)
+ return original_accumulate(
+ counts,
+ sums,
+ coordinates,
+ assignments,
+ batch_codes,
+ )
+
+ monkeypatch.setattr(
+ mapping_operations,
+ "accumulate_sufficient_statistics",
+ observe_accumulation,
+ )
+ reference_before = _snapshot_store(reference_store.zarr_loc)
+ result = query.run_mapping(
+ reference,
+ "symphony",
+ query_batches=batches,
+ )
+
+ assert _snapshot_store(reference_store.zarr_loc) == reference_before
+ assert result.reference is reference
+ assert result.correction_method == "symphony"
+ assert result.diagnostics == {
+ "featureCoverage": 1.0,
+ "queryBatchCount": 6,
+ "algorithmVariant": "symphony",
+ "zeroNormCellCount": result.diagnostics["zeroNormCellCount"],
+ "queryScaledDispersion": result.diagnostics["queryScaledDispersion"],
+ }
+ assert accumulated_rows == n_cells - result.diagnostics["zeroNormCellCount"]
+ loaded = load_projection(query.zw, result.ref, reference=reference)
+ assert loaded.diagnostics == result.diagnostics
+ reused = query.run_mapping(
+ reference,
+ "symphony",
+ query_batches=batches.copy(),
+ )
+ assert reused.ref == result.ref
+ assert reused.diagnostics == result.diagnostics
+
+ omitted = query.run_mapping(reference, "symphony_omitted")
+ assert omitted.diagnostics["queryBatchCount"] == 1
+
+
+def test_projection_cache_tracks_counts_references_and_newest_pair(
+ analyzed_datastore_ephemeral,
+ tmp_path,
+):
+ reference_store = analyzed_datastore_ephemeral
+ first_reference = _plain_reference(reference_store)
+ state = reference_store.get_assay_state("RNA")
+ assert state is not None
+ assert state.neighbors is not None
+ second_reference = reference_store.build_mapping_reference(
+ state.neighbors,
+ invalidate_cache=True,
+ )
+ query = _copied_query(reference_store, tmp_path / "query.zarr")
+
+ first = query.run_mapping(first_reference, "shared")
+ second = query.run_mapping(second_reference, "shared")
+ assert first.ref != second.ref
+ assert (
+ resolve_projection(
+ query.zw,
+ query_assay="RNA",
+ mapping_name="shared",
+ mapping_reference=first_reference.external_ref,
+ )
+ == first.ref
+ )
+ assert (
+ resolve_projection(
+ query.zw,
+ query_assay="RNA",
+ mapping_name="shared",
+ mapping_reference=second_reference.external_ref,
+ )
+ == second.ref
+ )
+
+ newest = query.run_mapping(
+ first_reference,
+ "shared",
+ invalidate_cache=True,
+ )
+ assert newest.ref != first.ref
+ assert (
+ resolve_projection(
+ query.zw,
+ query_assay="RNA",
+ mapping_name="shared",
+ mapping_reference=first_reference.external_ref,
+ )
+ == newest.ref
+ )
+
+ backing = query.RNA.rawData._backing
+ selected_row = int(query.cells.active_index("I")[0])
+ original = int(backing[selected_row, 0])
+ backing[selected_row, 0] = original + 1
+ changed_counts = query.run_mapping(first_reference, "raw_edit")
+ backing[selected_row, 0] = original + 2
+ changed_again = query.run_mapping(first_reference, "raw_edit")
+ assert changed_again.ref != changed_counts.ref
diff --git a/tests/test_mapping_projection.py b/tests/test_mapping_projection.py
new file mode 100644
index 00000000..c270f0d3
--- /dev/null
+++ b/tests/test_mapping_projection.py
@@ -0,0 +1,667 @@
+from dataclasses import replace
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.mapping.models import ScaledPCAProjectionModel
+from scarf.mapping.projection import (
+ NO_QUERY_BATCH_FINGERPRINT,
+ ProjectionWriter,
+ load_projection,
+ plan_projection,
+ resolve_projection,
+)
+from scarf.mapping.reference import MappingReference
+from scarf.storage.artifact_writer import (
+ finish_artifact,
+ plan_artifact,
+ start_artifact,
+)
+from scarf.storage.artifacts import (
+ ArtifactRef,
+ ExternalArtifactRef,
+ artifact_group,
+ inspect_artifact,
+ list_artifacts,
+)
+
+
+def _selection(
+ root: zarr.Group,
+ *,
+ kind: str,
+ values: np.ndarray,
+ assay: str | None,
+) -> ArtifactRef:
+ scope = "datastore" if assay is None else "assay"
+ planned = plan_artifact(
+ root,
+ scope=scope,
+ assay=assay,
+ kind=kind,
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ execution_options={},
+ )
+ group = start_artifact(root, planned)
+ group.create_array(
+ "values",
+ data=np.asarray(values, dtype=bool),
+ chunks=(len(values),),
+ )
+ finish_artifact(group, planned)
+ return planned.ref
+
+
+def _query_inputs(
+ *,
+ n_cells: int = 4,
+) -> tuple[zarr.Group, ArtifactRef, ArtifactRef]:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ cell_selection = _selection(
+ root,
+ kind="cell_selection",
+ values=np.ones(n_cells, dtype=bool),
+ assay=None,
+ )
+ feature_selection = _selection(
+ root,
+ kind="feature_selection",
+ values=np.array([True, False, True]),
+ assay="RNA",
+ )
+ return root, cell_selection, feature_selection
+
+
+def _artifact_ref(
+ kind: str,
+ token: str,
+ *,
+ assay: str | None = "RNA",
+) -> ArtifactRef:
+ return ArtifactRef(
+ scope="datastore" if assay is None else "assay",
+ assay=assay,
+ kind=kind,
+ artifact_id=token * 64,
+ )
+
+
+class _ReferenceDatastore:
+ def __init__(self, root: zarr.Group) -> None:
+ self.zw = root
+
+ def _get_assay(self, name: str) -> SimpleNamespace:
+ return SimpleNamespace(attrs=self.zw[name].attrs)
+
+
+def _mapping_reference(
+ *,
+ fingerprint: str = "reference-dataset",
+ token: str = "a",
+) -> tuple[MappingReference, zarr.Group]:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ assay = root.create_group("RNA")
+ assay.attrs["dataset_fingerprint"] = fingerprint
+ reference = MappingReference(
+ datastore=_ReferenceDatastore(root),
+ ref=_artifact_ref("mapping_reference", token),
+ assay_name="RNA",
+ cell_key="I",
+ feature_key="I",
+ reduction=_artifact_ref("reduction", "1"),
+ ann_index=_artifact_ref("ann_index", "2"),
+ neighbors=_artifact_ref("neighbors", "3"),
+ cell_selection=_artifact_ref("cell_selection", "4", assay=None),
+ feature_selection=_artifact_ref("feature_selection", "5"),
+ batch_correction=None,
+ dataset_fingerprint=fingerprint,
+ selected_cell_count=3,
+ model=ScaledPCAProjectionModel(
+ feature_means=np.zeros(1),
+ feature_scales=np.ones(1),
+ loadings=np.ones((1, 1)),
+ ),
+ symphony_state=None,
+ feature_ids=np.array(["gene"]),
+ metadata={
+ "method": "pca",
+ "ann_metric": "l2",
+ "normalization_parameters": {"size_factor": 1.0},
+ },
+ reference_distance_quantiles=np.array([0.5]),
+ reference_distance_values=np.array([1.0]),
+ )
+ return reference, root
+
+
+def _plan(
+ root: zarr.Group,
+ cell_selection: ArtifactRef,
+ feature_selection: ArtifactRef,
+ external: ExternalArtifactRef,
+ *,
+ mapping_name: str = "atlas",
+ n_cells: int = 4,
+ save_k: int = 2,
+ invalidate_cache: bool = False,
+):
+ return plan_projection(
+ root,
+ query_assay="RNA",
+ mapping_name=mapping_name,
+ n_cells=n_cells,
+ save_k=save_k,
+ missing_feature_policy="reference_mean",
+ correction_method="none",
+ cell_selection=cell_selection,
+ feature_selection=feature_selection,
+ selected_expression_fingerprint="e" * 64,
+ query_batch_fingerprint=NO_QUERY_BATCH_FINGERPRINT,
+ mapping_reference=external,
+ reference_cell_count=3,
+ invalidate_cache=invalidate_cache,
+ )
+
+
+def _blocks() -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ return (
+ np.array([[0, 1], [1, 2], [2, 1], [0, 2]], dtype=np.uint32),
+ np.array(
+ [[0.1, 0.2], [0.0, 1.0], [2.0, 3.0], [0.5, 0.75]],
+ dtype=np.float32,
+ ),
+ np.array([False, True, False, True]),
+ )
+
+
+def _diagnostics(*, zero_norm_cell_count: int = 2) -> dict[str, object]:
+ return {
+ "featureCoverage": 0.75,
+ "queryBatchCount": 1,
+ "algorithmVariant": "scaled_pca",
+ "zeroNormCellCount": zero_norm_cell_count,
+ "queryScaledDispersion": 1.0,
+ }
+
+
+def _write(
+ root: zarr.Group,
+ plan,
+ *,
+ created_at_ns: int | None = None,
+) -> ArtifactRef:
+ indices, distances, uninformative = _blocks()
+ writer = ProjectionWriter(root, plan, chunk_rows=2)
+ writer.write_block(0, indices[:2], distances[:2], uninformative[:2])
+ writer.write_block(2, indices[2:], distances[2:], uninformative[2:])
+ ref = writer.finish(_diagnostics())
+ if created_at_ns is not None:
+ artifact_group(root, ref).attrs["created_at_ns"] = created_at_ns
+ return ref
+
+
+def test_projection_writer_persists_exact_contract_and_loads_copies() -> None:
+ root, cell_selection, feature_selection = _query_inputs()
+ reference, reference_root = _mapping_reference()
+ external = reference.external_ref
+ before_reference_attrs = dict(reference_root["RNA"].attrs)
+ plan = _plan(root, cell_selection, feature_selection, external)
+
+ assert not plan.reused
+ assert not inspect_artifact(root, plan.ref).exists
+ indices, distances, uninformative = _blocks()
+ writer = ProjectionWriter(root, plan, chunk_rows=2)
+ assert not inspect_artifact(root, plan.ref).complete
+ writer.write_block(0, indices[:1], distances[:1], uninformative[:1])
+ writer.write_block(1, indices[1:3], distances[1:3], uninformative[1:3])
+ writer.write_block(3, indices[3:], distances[3:], uninformative[3:])
+ assert writer.finish(_diagnostics()) == plan.ref
+ assert writer.finished
+
+ status = inspect_artifact(root, plan.ref)
+ assert status.complete
+ assert status.operation == "map_query"
+ assert status.parameters == {
+ "mapping_name": "atlas",
+ "save_k": 2,
+ "missing_feature_policy": "reference_mean",
+ "correction_method": "none",
+ }
+ assert set(status.inputs or {}) == {
+ "cell_selection",
+ "feature_selection",
+ "selected_expression_fingerprint",
+ "query_batch_fingerprint",
+ "mapping_reference",
+ }
+ assert (status.inputs or {})["mapping_reference"] == external.to_dict()
+ group = artifact_group(root, plan.ref)
+ assert set(group.array_keys()) == {"indices", "distances", "uninformative"}
+ assert set(group.group_keys()) == set()
+ assert group.attrs["diagnostics"] == _diagnostics()
+ assert "ann_metric" not in (status.parameters or {})
+ assert "reference_feature_indices" not in group
+
+ metadata = load_projection(root, plan.ref)
+ assert metadata.ref == plan.ref
+ assert metadata.mapping_name == "atlas"
+ assert metadata.n_cells == 4
+ assert metadata.correction_method == "none"
+ assert metadata.diagnostics == _diagnostics()
+ assert metadata.indices is None
+ assert metadata.distances is None
+ assert metadata.uninformative is None
+ assert metadata.reference is None
+
+ loaded = load_projection(
+ root,
+ plan.ref,
+ load_arrays=True,
+ reference=reference,
+ )
+ np.testing.assert_array_equal(loaded.indices, indices)
+ np.testing.assert_allclose(loaded.distances, distances)
+ np.testing.assert_array_equal(loaded.uninformative, uninformative)
+ assert loaded.reference is reference
+ assert replace(metadata, reference=reference) == metadata
+ assert "MappingReference" not in repr(loaded)
+ assert "[[" not in repr(loaded)
+ assert dict(reference_root["RNA"].attrs) == before_reference_attrs
+
+
+@pytest.mark.parametrize(
+ ("indices", "distances", "uninformative", "message"),
+ [
+ (
+ np.ones((2, 2), dtype=np.int64),
+ np.ones((2, 2), dtype=np.float64),
+ np.zeros(2, dtype=bool),
+ "unsigned",
+ ),
+ (
+ np.ones((2, 2), dtype=np.uint32),
+ np.array([[0.0, np.nan], [1.0, 2.0]]),
+ np.zeros(2, dtype=bool),
+ "finite",
+ ),
+ (
+ np.ones((2, 2), dtype=np.uint32),
+ np.array([[0.0, -1.0], [1.0, 2.0]]),
+ np.zeros(2, dtype=bool),
+ "non-negative",
+ ),
+ (
+ np.ones((2, 1), dtype=np.uint32),
+ np.ones((2, 1), dtype=np.float64),
+ np.zeros(2, dtype=bool),
+ "shape",
+ ),
+ (
+ np.array([[0, 3], [1, 2]], dtype=np.uint32),
+ np.ones((2, 2), dtype=np.float64),
+ np.zeros(2, dtype=bool),
+ "selected reference cells",
+ ),
+ ],
+)
+def test_projection_writer_aborts_after_invalid_block(
+ indices: np.ndarray,
+ distances: np.ndarray,
+ uninformative: np.ndarray,
+ message: str,
+) -> None:
+ root, cell_selection, feature_selection = _query_inputs()
+ reference, _ = _mapping_reference()
+ plan = _plan(root, cell_selection, feature_selection, reference.external_ref)
+ writer = ProjectionWriter(root, plan, chunk_rows=2)
+
+ with pytest.raises((TypeError, ValueError), match=message):
+ writer.write_block(0, indices, distances, uninformative)
+
+ assert writer.aborted
+ assert not inspect_artifact(root, plan.ref).complete
+ with pytest.raises(RuntimeError, match="aborted"):
+ writer.finish(_diagnostics())
+
+
+def test_projection_writer_requires_contiguous_complete_coverage_and_can_abort() -> (
+ None
+):
+ root, cell_selection, feature_selection = _query_inputs()
+ reference, _ = _mapping_reference()
+ plan = _plan(root, cell_selection, feature_selection, reference.external_ref)
+ indices, distances, uninformative = _blocks()
+ writer = ProjectionWriter(root, plan, chunk_rows=2)
+
+ writer.write_block(0, indices[:2], distances[:2], uninformative[:2])
+ with pytest.raises(ValueError, match="wrote 2 of 4"):
+ writer.finish(_diagnostics())
+ assert writer.aborted
+ assert not inspect_artifact(root, plan.ref).complete
+
+ second_plan = _plan(
+ root,
+ cell_selection,
+ feature_selection,
+ reference.external_ref,
+ invalidate_cache=True,
+ )
+ second = ProjectionWriter(root, second_plan, chunk_rows=2)
+ second.abort()
+ assert second.aborted
+ assert not inspect_artifact(root, second_plan.ref).complete
+ with pytest.raises(RuntimeError, match="aborted"):
+ second.write_block(0, indices, distances, uninformative)
+
+ third_plan = _plan(
+ root,
+ cell_selection,
+ feature_selection,
+ reference.external_ref,
+ invalidate_cache=True,
+ )
+ third = ProjectionWriter(root, third_plan, chunk_rows=2)
+ with pytest.raises(ValueError, match="expected 0, received 1"):
+ third.write_block(1, indices[:1], distances[:1], uninformative[:1])
+ assert third.aborted
+
+
+def test_projection_writer_reuses_only_a_valid_complete_artifact() -> None:
+ root, cell_selection, feature_selection = _query_inputs()
+ reference, _ = _mapping_reference()
+ first = _plan(root, cell_selection, feature_selection, reference.external_ref)
+ ref = _write(root, first)
+
+ reused = _plan(root, cell_selection, feature_selection, reference.external_ref)
+ assert reused.reused
+ assert reused.ref == ref
+ assert load_projection(root, ref).diagnostics["zeroNormCellCount"] == 2
+ with pytest.raises(ValueError, match="without a writer"):
+ ProjectionWriter(root, reused, chunk_rows=2)
+
+ artifact_group(root, ref).create_array(
+ "extra",
+ data=np.ones(1),
+ chunks=(1,),
+ )
+ replacement = _plan(
+ root,
+ cell_selection,
+ feature_selection,
+ reference.external_ref,
+ )
+ assert not replacement.reused
+ assert replacement.ref != ref
+
+
+def test_projection_plan_rejects_out_of_range_cached_neighbors() -> None:
+ root, cell_selection, feature_selection = _query_inputs()
+ reference, _ = _mapping_reference()
+ first = _plan(root, cell_selection, feature_selection, reference.external_ref)
+ ref = _write(root, first)
+ artifact_group(root, ref)["indices"][0, 0] = reference.selected_cell_count
+
+ replacement = _plan(
+ root,
+ cell_selection,
+ feature_selection,
+ reference.external_ref,
+ )
+
+ assert not replacement.reused
+ assert replacement.ref != ref
+
+
+def test_projection_finish_rejects_diagnostics_inconsistent_with_rows() -> None:
+ root, cell_selection, feature_selection = _query_inputs()
+ reference, _ = _mapping_reference()
+ plan = _plan(root, cell_selection, feature_selection, reference.external_ref)
+ indices, distances, uninformative = _blocks()
+ writer = ProjectionWriter(root, plan, chunk_rows=4)
+ writer.write_block(0, indices, distances, uninformative)
+
+ with pytest.raises(ValueError, match="number of uninformative"):
+ writer.finish(_diagnostics(zero_norm_cell_count=1))
+
+ assert not inspect_artifact(root, plan.ref).complete
+
+
+def _manual_projection(
+ root: zarr.Group,
+ cell_selection: ArtifactRef,
+ feature_selection: ArtifactRef,
+ mapping_reference: ArtifactRef | ExternalArtifactRef,
+ *,
+ operation: str = "map_query",
+) -> ArtifactRef:
+ planned = plan_artifact(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="projection",
+ operation=operation,
+ parameters={
+ "mapping_name": "atlas",
+ "save_k": 2,
+ "missing_feature_policy": "reference_mean",
+ "correction_method": "none",
+ },
+ inputs={
+ "cell_selection": cell_selection,
+ "feature_selection": feature_selection,
+ "selected_expression_fingerprint": "e" * 64,
+ "query_batch_fingerprint": NO_QUERY_BATCH_FINGERPRINT,
+ "mapping_reference": mapping_reference,
+ },
+ execution_options={},
+ )
+ group = start_artifact(root, planned)
+ indices, distances, uninformative = _blocks()
+ group.create_array("indices", data=indices, chunks=(2, 2))
+ group.create_array("distances", data=distances, chunks=(2, 2))
+ group.create_array("uninformative", data=uninformative, chunks=(2,))
+ group.attrs["diagnostics"] = _diagnostics()
+ finish_artifact(group, planned)
+ return planned.ref
+
+
+def test_projection_loader_rejects_old_and_local_reference_contracts() -> None:
+ root, cell_selection, feature_selection = _query_inputs()
+ reference, _ = _mapping_reference()
+ old = _manual_projection(
+ root,
+ cell_selection,
+ feature_selection,
+ reference.external_ref,
+ operation="map_with_reference",
+ )
+ local = _manual_projection(
+ root,
+ cell_selection,
+ feature_selection,
+ reference.ref,
+ )
+
+ for ref in (old, local):
+ with pytest.raises(ValueError, match="run_mapping"):
+ load_projection(root, ref)
+
+
+@pytest.mark.parametrize(
+ "malformation",
+ ["missing", "array", "group", "attribute", "diagnostics"],
+)
+def test_projection_loader_rejects_malformed_or_extra_payload(
+ malformation: str,
+) -> None:
+ root, cell_selection, feature_selection = _query_inputs()
+ reference, _ = _mapping_reference()
+ ref = _write(
+ root,
+ _plan(root, cell_selection, feature_selection, reference.external_ref),
+ )
+ group = artifact_group(root, ref)
+ if malformation == "missing":
+ del group["distances"]
+ elif malformation == "array":
+ group.create_array("extra", data=np.ones(1), chunks=(1,))
+ elif malformation == "group":
+ group.create_group("extra")
+ elif malformation == "attribute":
+ group.attrs["extra"] = "invalid"
+ else:
+ diagnostics = dict(group.attrs["diagnostics"])
+ diagnostics["zeroNormCellCount"] = 1
+ group.attrs["diagnostics"] = diagnostics
+
+ with pytest.raises(ValueError, match="run_mapping"):
+ load_projection(root, ref)
+
+
+def test_projection_written_without_the_dispersion_diagnostic_is_rejected() -> None:
+ root, cell_selection, feature_selection = _query_inputs()
+ reference, _ = _mapping_reference()
+ ref = _write(
+ root,
+ _plan(root, cell_selection, feature_selection, reference.external_ref),
+ )
+ group = artifact_group(root, ref)
+ diagnostics = dict(group.attrs["diagnostics"])
+ del diagnostics["queryScaledDispersion"]
+ group.attrs["diagnostics"] = diagnostics
+
+ # Projections predating the diagnostic are not silently migrated. They are
+ # cheap to recompute, so the loader names the gap instead of guessing a value.
+ with pytest.raises(ValueError, match="is missing queryScaledDispersion"):
+ load_projection(root, ref)
+
+
+def test_projection_loader_validates_and_matches_provided_reference() -> None:
+ root, cell_selection, feature_selection = _query_inputs()
+ reference, _ = _mapping_reference()
+ ref = _write(
+ root,
+ _plan(root, cell_selection, feature_selection, reference.external_ref),
+ )
+ other_artifact, _ = _mapping_reference(
+ fingerprint="reference-dataset",
+ token="b",
+ )
+ other_fingerprint, _ = _mapping_reference(
+ fingerprint="other-dataset",
+ token="a",
+ )
+
+ for mismatched in (other_artifact, other_fingerprint):
+ with pytest.raises(ValueError, match="run_mapping"):
+ load_projection(root, ref, reference=mismatched)
+
+ reference.datastore.zw["RNA"].attrs["dataset_fingerprint"] = "changed"
+ with pytest.raises(ValueError, match="run_mapping"):
+ load_projection(root, ref, reference=reference)
+
+
+def test_projection_loader_metadata_only_rejects_out_of_range_neighbor() -> None:
+ root, cell_selection, feature_selection = _query_inputs()
+ reference, _ = _mapping_reference()
+ ref = _write(
+ root,
+ _plan(root, cell_selection, feature_selection, reference.external_ref),
+ )
+ artifact_group(root, ref)["indices"][2, 1] = reference.selected_cell_count
+ mismatched, _ = _mapping_reference(
+ fingerprint="reference-dataset",
+ token="b",
+ )
+
+ with pytest.raises(
+ ValueError,
+ match="does not match the projection input.*run_mapping",
+ ):
+ load_projection(root, ref, reference=mismatched)
+
+ with pytest.raises(
+ ValueError,
+ match="outside the selected reference cell range.*run_mapping",
+ ):
+ load_projection(root, ref, reference=reference)
+
+
+def test_projection_resolver_scopes_names_by_reference_and_uses_newest() -> None:
+ root, cell_selection, feature_selection = _query_inputs()
+ first_reference, _ = _mapping_reference(token="a")
+ second_reference, _ = _mapping_reference(
+ fingerprint="second-dataset",
+ token="b",
+ )
+ first_old = _write(
+ root,
+ _plan(
+ root,
+ cell_selection,
+ feature_selection,
+ first_reference.external_ref,
+ mapping_name="shared",
+ ),
+ created_at_ns=10,
+ )
+ first_new = _write(
+ root,
+ _plan(
+ root,
+ cell_selection,
+ feature_selection,
+ first_reference.external_ref,
+ mapping_name="shared",
+ invalidate_cache=True,
+ ),
+ created_at_ns=30,
+ )
+ second = _write(
+ root,
+ _plan(
+ root,
+ cell_selection,
+ feature_selection,
+ second_reference.external_ref,
+ mapping_name="shared",
+ ),
+ created_at_ns=20,
+ )
+
+ assert first_old != first_new
+ assert (
+ resolve_projection(
+ root,
+ query_assay="RNA",
+ mapping_name="shared",
+ mapping_reference=first_reference.external_ref,
+ )
+ == first_new
+ )
+ assert (
+ resolve_projection(
+ root,
+ query_assay="RNA",
+ mapping_name="shared",
+ mapping_reference=second_reference.external_ref,
+ )
+ == second
+ )
+ assert (
+ len(
+ list_artifacts(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="projection",
+ complete_only=True,
+ )
+ )
+ == 3
+ )
diff --git a/tests/test_mapping_reference.py b/tests/test_mapping_reference.py
new file mode 100644
index 00000000..27f02b84
--- /dev/null
+++ b/tests/test_mapping_reference.py
@@ -0,0 +1,257 @@
+import numpy as np
+import pytest
+
+import scarf.mapping as mapping
+from scarf.datastore.datastore import DataStore
+from scarf.storage.artifacts import artifact_group
+
+
+_COMMON_ARRAYS = {
+ "feature_ids",
+ "feature_means",
+ "feature_scales",
+ "loadings",
+ "reference_distance_quantiles",
+ "reference_distance_values",
+}
+_SYMPHONY_ARRAYS = {
+ "centroids",
+ "raw_centroids",
+ "corrected_centroids",
+ "cluster_mass",
+ "sigma",
+}
+
+
+def test_embedded_mapping_reference_helpers_are_not_public():
+ for name in (
+ "LATEST_MAPPING_REFERENCE_ATTRIBUTE",
+ "MAPPING_REFERENCE_GROUP",
+ "MAPPING_REFERENCES_GROUP",
+ "load_mapping_reference",
+ "mapping_reference_hash",
+ "persist_mapping_reference",
+ "resolve_mapping_reference_group",
+ "validate_mapping_reference_artifact",
+ ):
+ assert name not in mapping.__all__
+ assert not hasattr(mapping, name)
+ assert not hasattr(mapping.MappingReference, "map_query")
+
+
+def _selected_neighbors(datastore):
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.neighbors is not None
+ return state.neighbors
+
+
+def _symphony_neighbors(datastore):
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.reduction is not None
+ datastore.cells.insert(
+ "mapping_batch",
+ np.where(np.arange(datastore.cells.N) % 2, "a", "b"),
+ overwrite=True,
+ )
+ correction = datastore.run_harmony(
+ ["mapping_batch"],
+ state.reduction,
+ harmony_params={"nclust": 5},
+ update_state=False,
+ )
+ ann_index = datastore.build_ann_index(
+ correction,
+ update_state=False,
+ )
+ return datastore.query_neighbors(
+ ann_index,
+ coordinates=correction,
+ k=3,
+ update_state=False,
+ )
+
+
+def test_plain_mapping_reference_packages_and_loads_existing_chain(
+ analyzed_datastore_ephemeral,
+):
+ datastore = analyzed_datastore_ephemeral
+ neighbors = _selected_neighbors(datastore)
+ before = set(datastore.list_artifacts(from_assay="RNA"))
+ reference = datastore.build_mapping_reference(neighbors)
+
+ assert reference.method == "pca"
+ assert reference.symphony_state is None
+ assert reference.neighbors == neighbors
+ assert reference.dataset_fingerprint == datastore.RNA.attrs["dataset_fingerprint"]
+ assert reference.selected_cell_count == len(
+ datastore.cells.fetch("ids", key=reference.cell_key)
+ )
+ assert reference.size_factor > 0
+ assert reference.ann_metric in {"l2", "cosine"}
+
+ group = artifact_group(datastore.zw, reference.ref)
+ assert set(group.array_keys()) == _COMMON_ARRAYS
+ status = datastore.inspect_artifact(reference.ref)
+ assert status.parameters == {"method": "pca"}
+ assert set((status.inputs or {})) == {
+ "reduction",
+ "ann_index",
+ "neighbors",
+ "cell_selection",
+ "feature_selection",
+ }
+ created = set(datastore.list_artifacts(from_assay="RNA")) - before
+ assert created == {reference.ref}
+
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.neighbors == neighbors
+ assert state.named_results["mapping_reference"] == reference.ref
+ assert datastore.get_mapping_reference().ref == reference.ref
+ assert datastore.get_mapping_reference(reference.ref).ref == reference.ref
+
+ datastore.cells.insert(
+ "reference_layout1",
+ np.arange(datastore.cells.N, dtype=np.float64),
+ overwrite=True,
+ )
+ datastore.cells.insert(
+ "reference_layout2",
+ -np.arange(datastore.cells.N, dtype=np.float64),
+ overwrite=True,
+ )
+ np.testing.assert_array_equal(
+ reference.fetch_cell_column("ids"),
+ datastore.cells.fetch("ids", key=reference.cell_key),
+ )
+ np.testing.assert_array_equal(
+ reference.fetch_layout("reference_layout"),
+ np.column_stack(
+ (
+ datastore.cells.fetch(
+ "reference_layout1",
+ key=reference.cell_key,
+ ),
+ datastore.cells.fetch(
+ "reference_layout2",
+ key=reference.cell_key,
+ ),
+ )
+ ),
+ )
+
+
+def test_symphony_mapping_reference_has_conditional_state_and_read_only_reload(
+ analyzed_datastore_ephemeral,
+):
+ datastore = analyzed_datastore_ephemeral
+ neighbors = _symphony_neighbors(datastore)
+ reference = datastore.build_mapping_reference(neighbors)
+
+ assert reference.method == "symphony"
+ assert reference.symphony_state is not None
+ assert reference.batch_correction is not None
+ assert reference.symphony_state.n_dims == reference.model.n_dims
+ assert reference.metadata["batch_columns"] == ["mapping_batch"]
+ assert reference.metadata["harmony_parameters"]["nclust"] == 5
+
+ group = artifact_group(datastore.zw, reference.ref)
+ assert set(group.array_keys()) == _COMMON_ARRAYS | _SYMPHONY_ARRAYS
+ assert set(group.attrs) == {
+ "artifact_id",
+ "kind",
+ "provenance",
+ "execution_options",
+ "created_at_ns",
+ "complete",
+ "reference_metadata",
+ }
+ status = datastore.inspect_artifact(reference.ref)
+ assert status.parameters == {"method": "symphony"}
+ assert set((status.inputs or {})) == {
+ "reduction",
+ "batch_correction",
+ "ann_index",
+ "neighbors",
+ "cell_selection",
+ "feature_selection",
+ }
+
+ read_only = DataStore(
+ datastore.zarr_loc,
+ default_assay="RNA",
+ zarr_mode="r",
+ )
+ loaded = read_only.get_mapping_reference(reference.ref)
+ assert loaded.ref == reference.ref
+ assert loaded.symphony_state is not None
+ np.testing.assert_array_equal(loaded.feature_ids, reference.feature_ids)
+ np.testing.assert_allclose(
+ loaded.symphony_state.corrected_centroids,
+ reference.symphony_state.corrected_centroids,
+ )
+
+
+def test_mapping_reference_rejects_invalid_chain_contract(
+ analyzed_datastore_ephemeral,
+):
+ datastore = analyzed_datastore_ephemeral
+ neighbors = _selected_neighbors(datastore)
+ neighbors_status = datastore.inspect_artifact(neighbors)
+ assert neighbors_status.inputs is not None
+ ann_index = neighbors_status.inputs["ann_index"]
+ ann_ref = type(neighbors).from_dict(ann_index)
+ ann_group = artifact_group(datastore.zw, ann_ref)
+ provenance = dict(ann_group.attrs["provenance"])
+ parameters = dict(provenance["parameters"])
+ parameters["ann_metric"] = "ip"
+ provenance["parameters"] = parameters
+ ann_group.attrs["provenance"] = provenance
+
+ with pytest.raises(ValueError, match="only l2 and cosine"):
+ datastore.build_mapping_reference(neighbors)
+
+
+def test_mapping_reference_rejects_old_embedded_and_incomplete_contracts(
+ analyzed_datastore_ephemeral,
+):
+ datastore = analyzed_datastore_ephemeral
+ neighbors = _selected_neighbors(datastore)
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.reduction is not None
+ artifact_group(datastore.zw, state.reduction).create_group("mappingReference")
+
+ with pytest.raises(ValueError, match="build_mapping_reference\\(neighbors\\)"):
+ datastore.get_mapping_reference()
+
+ reference = datastore.build_mapping_reference(neighbors)
+ group = artifact_group(datastore.zw, reference.ref)
+ del group["feature_scales"]
+ with pytest.raises(ValueError, match="build_mapping_reference\\(neighbors\\)"):
+ datastore.get_mapping_reference(reference.ref)
+
+
+def test_mapping_reference_validates_live_dataset_fingerprint(
+ analyzed_datastore_ephemeral,
+):
+ datastore = analyzed_datastore_ephemeral
+ reference = datastore.build_mapping_reference(_selected_neighbors(datastore))
+ datastore.RNA.attrs["dataset_fingerprint"] = "changed"
+
+ with pytest.raises(ValueError, match="dataset fingerprint"):
+ datastore.get_mapping_reference(reference.ref)
+
+
+def test_mapping_reference_rejects_nonmonotonic_distance_summary(
+ analyzed_datastore_ephemeral,
+):
+ datastore = analyzed_datastore_ephemeral
+ reference = datastore.build_mapping_reference(_selected_neighbors(datastore))
+ values = artifact_group(datastore.zw, reference.ref)["reference_distance_values"]
+ values[:] = np.linspace(1.0, 0.0, values.shape[0])
+
+ with pytest.raises(ValueError, match="distance summary"):
+ datastore.get_mapping_reference(reference.ref)
diff --git a/tests/test_mapping_utils.py b/tests/test_mapping_utils.py
new file mode 100644
index 00000000..bc1993b2
--- /dev/null
+++ b/tests/test_mapping_utils.py
@@ -0,0 +1,151 @@
+import numpy as np
+import pytest
+
+from scarf.mapping.confidence import (
+ _distance_quantile_summary,
+ conformal_prediction_sets,
+ distance_weights,
+ mapping_score_weights,
+)
+from scarf.mapping.features import _feature_ids
+from scarf.mapping.hashing import array_hash, array_store_hash
+
+
+def test_mapping_array_hashes_match_golden_values():
+ numeric = np.array([[1.5, -2.0], [0.0, 4.25]], dtype=np.float64)
+ identifiers = np.array(["gene_a", "gene_b"], dtype=" np.ndarray:
+ """Independent pandas and SciPy implementation used as the parity reference."""
+ ranked_vdf = vdf.rank(method="dense")
+ r = ranked_vdf.groupby(groups).mean().reindex(group_set)
+ r = r / r.sum()
+ g = np.array([pd.Series(groups).value_counts().reindex(group_set).values]).T
+ g_o = len(groups) - g
+ s = vdf.groupby(groups).sum().reindex(group_set)
+ m = s / g
+ m_o = (s.sum() - s) / g_o
+ s2 = (vdf > 0).groupby(groups).sum().reindex(group_set)
+ e = s2 / g
+ e_o = (s2.sum() - s2) / g_o
+ fc = (m / m_o).fillna(0)
+ pvals = pd.DataFrame(
+ np.vstack(
+ [
+ mannwhitneyu(
+ vdf.loc[groups == group].to_numpy(),
+ vdf.loc[groups != group].to_numpy(),
+ axis=0,
+ alternative="two-sided",
+ method="asymptotic",
+ use_continuity=True,
+ ).pvalue
+ for group in group_set
+ ]
+ ),
+ index=group_set,
+ columns=vdf.columns,
+ )
+ return np.array(
+ [r.values, m.values, m_o.values, e.values, e_o.values, fc.values, pvals.values]
+ ).T
+
+
+def test_batch_stats_matches_pandas_reference():
+ rng = np.random.default_rng(0)
+ n_cells, n_genes = 250, 16
+ # Zero-inflated counts exercise the tie correction path.
+ data = rng.poisson(0.6, size=(n_cells, n_genes)).astype(np.float64)
+ groups = rng.integers(1, 4, size=n_cells)
+ group_set = np.array(sorted(set(groups)))
+ idx_map = {v: i for i, v in enumerate(group_set)}
+ int_indices = np.array([idx_map[x] for x in groups])
+ group_counts = pd.Series(groups).value_counts().reindex(group_set).values
+
+ ref = _reference_calc(pd.DataFrame(data), groups, group_set)
+ got = _batch_stats(data, int_indices, group_counts, n_cells)
+
+ # score, mean, mean_rest, frac_exp, frac_exp_rest
+ assert np.allclose(got[:, :, :5], ref[:, :, :5], atol=1e-6)
+ # fold_change agrees where the reference is finite
+ finite = np.isfinite(ref[:, :, 5])
+ assert np.allclose(got[:, :, 5][finite], ref[:, :, 5][finite], atol=1e-6)
+ # two-sided p-values
+ assert np.allclose(got[:, :, 6], ref[:, :, 6], atol=1e-6)
+
+
+def test_batch_stats_preserves_float64_near_ties_against_scipy():
+ groups = np.array([0, 0, 0, 0, 1, 1, 1, 1])
+ values = 1.0 + 1e-9 * np.array([5, 6, 7, 8, 1, 2, 3, 4])
+ data = values[:, None]
+ assert np.unique(data.astype(np.float32)).size == 1
+
+ expected = mannwhitneyu(
+ values[groups == 0],
+ values[groups == 1],
+ alternative="two-sided",
+ method="asymptotic",
+ use_continuity=True,
+ )
+ got = _batch_stats(data, groups, np.bincount(groups), len(groups))
+
+ assert got[0, 0, 6] == pytest.approx(expected.pvalue, rel=1e-12, abs=1e-15)
+ assert got[0, 0, 7] == pytest.approx(
+ expected.statistic / 16.0,
+ rel=1e-12,
+ abs=1e-15,
+ )
+
+
+def test_rank_paths_match_scipy_continuity_correction():
+ data = np.array(
+ [
+ [8, 0, 0],
+ [7, 1, 1],
+ [6, 1, 2],
+ [5, 2, 3],
+ [4, 3, 0],
+ [3, 3, 1],
+ [2, 4, 2],
+ [1, 5, 3],
+ ],
+ dtype=np.uint32,
+ )
+ groups = np.array([0, 0, 0, 0, 1, 1, 1, 1])
+ group_set = np.array([0, 1])
+ group_counts = np.bincount(groups)
+ expected = np.stack(
+ [
+ mannwhitneyu(
+ data[groups == group],
+ data[groups != group],
+ axis=0,
+ alternative="two-sided",
+ method="asymptotic",
+ use_continuity=True,
+ ).pvalue
+ for group in group_set
+ ],
+ axis=1,
+ )
+
+ ranked = pd.DataFrame(data).rank(method="average")
+ from_ranks = mannwhitneyu_from_ranks(ranked, groups, group_set).to_numpy().T
+ cell_major = _batch_stats(data, groups, group_counts, len(groups))[:, :, 6]
+ gene_major = _batch_stats_gene_major(
+ data.T,
+ np.ones(len(groups), dtype=np.float32),
+ 1.0,
+ False,
+ groups,
+ group_counts,
+ len(groups),
+ )[:, :, 6]
+
+ np.testing.assert_allclose(from_ranks, expected, rtol=1e-12, atol=1e-12)
+ np.testing.assert_allclose(cell_major, expected, rtol=1e-12, atol=1e-12)
+ np.testing.assert_allclose(gene_major, expected, rtol=1e-12, atol=1e-12)
+
+
+def test_tie_correction_survives_more_than_two_million_tied_values():
+ # A cubed int64 tie group wraps negative beyond roughly 2.08 million ties, which
+ # inflates the variance instead of collapsing it.
+ n_zeros = 2_200_000
+ values = np.concatenate([np.zeros(n_zeros), np.array([10.0, 20.0, 30.0, 40.0])])
+ groups = np.concatenate(
+ [np.ones(n_zeros, dtype=np.int64), np.zeros(4, dtype=np.int64)]
+ )
+ group_set = np.array([0, 1])
+ group_counts = np.bincount(groups)
+ n_total = values.size
+
+ ranked = pd.DataFrame({"feature": values}).rank(method="average")
+ from_ranks = mannwhitneyu_from_ranks(ranked, groups, group_set)["feature"]
+ cell_major = _batch_stats(values[:, None], groups, group_counts, n_total)[0, :, 6]
+ gene_major = _batch_stats_gene_major(
+ values.astype(np.uint32)[None, :],
+ np.ones(n_total, dtype=np.float32),
+ 1.0,
+ False,
+ groups,
+ group_counts,
+ n_total,
+ )[0, :, 6]
+
+ # The feature is confined to one group's four cells, so separation is maximal.
+ assert (from_ranks.to_numpy() < 1e-100).all()
+ assert (cell_major < 1e-100).all()
+ assert (gene_major < 1e-100).all()
+
+
+def test_mannwhitneyu_from_ranks_returns_one_for_zero_variance():
+ ranked = pd.DataFrame({"constant": np.ones(4)}).rank(method="average")
+ groups = np.array([0, 0, 1, 1])
+
+ with np.errstate(divide="raise", invalid="raise"):
+ p_values = mannwhitneyu_from_ranks(ranked, groups, np.array([0, 1]))
+
+ np.testing.assert_array_equal(p_values["constant"], [1.0, 1.0])
+
+
+def test_batch_stats_distinguishes_zero_fold_change_from_zero_rest_sentinel():
+ data = np.array(
+ [
+ [0.0, 2.0, 1.0],
+ [0.0, 4.0, 3.0],
+ [0.0, 0.0, 2.0],
+ [0.0, 0.0, 2.0],
+ ]
+ )
+ stats = _batch_stats(
+ data,
+ int_indices=np.array([0, 0, 1, 1]),
+ group_counts=np.array([2, 2]),
+ n_total=4,
+ )
+
+ assert np.array_equal(stats[0, :, 5], [0.0, 0.0])
+ assert stats[1, 0, 5] == pytest.approx(100.1)
+ assert stats[1, 1, 5] == pytest.approx(0.0)
+ assert np.array_equal(stats[2, :, 5], [1.0, 1.0])
+ assert np.array_equal(stats[0, :, 6], [1.0, 1.0])
+
+
+def test_marker_stats_python_kernel_matches_compiled_kernel():
+ data = np.array(
+ [
+ [0.0, 5.0, 0.0, 1.0],
+ [0.0, 4.0, 1.0, 1.0],
+ [0.0, 0.0, 2.0, 2.0],
+ [0.0, 0.0, 3.0, 2.0],
+ [0.0, 0.0, 4.0, 3.0],
+ [0.0, 0.0, 5.0, 3.0],
+ ],
+ dtype=np.float32,
+ )
+ int_indices = np.array([0, 0, 1, 1, 2, 2])
+ group_counts = np.array([2, 2, 2], dtype=np.float32)
+ n_total = np.float32(len(data))
+
+ python_stats = _marker_stats_batch.py_func(
+ data,
+ int_indices,
+ group_counts,
+ n_total,
+ )
+ compiled_stats = _marker_stats_batch(
+ data,
+ int_indices,
+ group_counts,
+ n_total,
+ )
+
+ np.testing.assert_allclose(python_stats, compiled_stats)
+ assert python_stats[1, 0, 5] == pytest.approx(100.1)
+ assert np.array_equal(python_stats[0, :, 5], [0.0, 0.0, 0.0])
+
+
+def test_marker_stats_python_kernel_handles_single_cell_population():
+ stats = _marker_stats_batch.py_func(
+ np.array([[0.0, 2.0]], dtype=np.float32),
+ np.array([0]),
+ np.array([1], dtype=np.float32),
+ np.float32(1),
+ )
+
+ np.testing.assert_allclose(
+ stats[:, 0, :7],
+ np.array(
+ [
+ [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
+ [1.0, 2.0, 0.0, 1.0, 0.0, 100.1, 0.0],
+ ]
+ ),
+ )
+ assert np.isnan(stats[:, 0, 7]).all()
+
+
+@pytest.mark.parametrize(
+ "raw",
+ [
+ np.array(
+ [
+ [1, 2, 3],
+ [3, 2, 1],
+ [2, 1, 4],
+ [4, 3, 2],
+ ],
+ dtype=np.uint32,
+ ),
+ np.zeros((4, 3), dtype=np.uint32),
+ np.full((4, 3), 2, dtype=np.uint32),
+ np.array(
+ [
+ [0, 0, 0],
+ [0, 0, 5],
+ [0, 0, 0],
+ [0, 0, 0],
+ ],
+ dtype=np.uint32,
+ ),
+ np.array(
+ [
+ [0, 2, 1],
+ [0, 2, 1],
+ [3, 2, 0],
+ [3, 2, 0],
+ ],
+ dtype=np.uint32,
+ ),
+ np.array([[0, 2, 2]], dtype=np.uint32),
+ ],
+ ids=[
+ "no-zeros",
+ "all-zero",
+ "constant",
+ "single-nonzero",
+ "heavy-ties",
+ "single-cell",
+ ],
+)
+@pytest.mark.parametrize("log_transform", [False, True])
+def test_gene_major_zero_aware_kernel_is_bit_identical(
+ raw: np.ndarray,
+ log_transform: bool,
+) -> None:
+ n_cells = raw.shape[0]
+ groups = np.arange(n_cells, dtype=np.int64) % max(1, min(3, n_cells))
+ group_counts = np.bincount(
+ groups,
+ minlength=int(groups.max()) + 1,
+ )
+ scalar = raw.sum(axis=1).astype(np.float32)
+ scalar[scalar == 0] = 1
+ normalized = (float(1000.0) * raw.astype(np.float32)) / scalar[:, None]
+ if log_transform:
+ normalized = np.log1p(normalized)
+
+ expected = _batch_stats(
+ normalized,
+ groups,
+ group_counts,
+ n_cells,
+ )
+ observed = _batch_stats_gene_major(
+ raw.T,
+ scalar,
+ 1000.0,
+ log_transform,
+ groups,
+ group_counts,
+ n_cells,
+ )
+
+ np.testing.assert_array_equal(observed, expected)
+
+
+def test_gene_major_python_kernel_matches_compiled_kernel() -> None:
+ raw = np.array(
+ [
+ [0, 2, 0, 4],
+ [1, 2, 0, 0],
+ [1, 0, 3, 4],
+ [0, 0, 3, 0],
+ ],
+ dtype=np.uint32,
+ ).T
+ scalar = np.array([2, 4, 6, 8], dtype=np.float32)
+ groups = np.array([0, 0, 1, 1], dtype=np.int64)
+ group_counts = np.array([2, 2], dtype=np.float32)
+ destinations = np.arange(raw.shape[0], dtype=np.int64)
+ compiled = np.zeros((raw.shape[0], 2, 8), dtype=np.float64)
+ python = np.zeros_like(compiled)
+ args = (
+ raw,
+ scalar,
+ np.float32(1000),
+ False,
+ groups,
+ group_counts,
+ np.float32(4),
+ destinations,
+ )
+
+ _marker_stats_gene_major(*args, compiled)
+ _marker_stats_gene_major.py_func(*args, python)
+
+ np.testing.assert_allclose(compiled, python, rtol=1e-7, atol=1e-12)
+
+
+def test_sort_marker_results_adds_deterministic_tie_breakers():
+ named = pd.DataFrame(
+ {
+ "score": [0.8, 0.8, 0.8],
+ "p_value": [0.02, 0.01, 0.01],
+ "feature_name": ["zeta", "beta", "alpha"],
+ },
+ index=[7, 8, 9],
+ )
+
+ sorted_named = sort_marker_results(named)
+ unnamed = named.drop(columns="feature_name").iloc[[0, 2]].copy()
+ unnamed["p_value"] = 0.01
+ sorted_unnamed = sort_marker_results(unnamed)
+
+ assert "feature_index" not in named
+ assert sorted_named["feature_name"].tolist() == ["alpha", "beta", "zeta"]
+ assert sorted_named["feature_index"].tolist() == [9, 8, 7]
+ assert sorted_unnamed["feature_index"].tolist() == [7, 9]
+
+
+def test_find_markers_by_regression_handles_expression_threshold():
+ class Assay:
+ @staticmethod
+ def iter_normed_feature_wise(**_kwargs):
+ yield pd.DataFrame(
+ {
+ "correlated": [0.0, 1.0, 2.0, 3.0],
+ "at_threshold": [0.0, 1.0, 2.0, 0.0],
+ "too_sparse": [0.0, 0.0, 1.0, 0.0],
+ "constant": [1.0, 1.0, 1.0, 1.0],
+ }
+ )
+
+ result = find_markers_by_regression(
+ Assay(),
+ cell_key="I",
+ feat_key="I",
+ regressor=np.arange(4),
+ min_cells=2,
+ )
+
+ assert result.loc["correlated", "r_value"] == pytest.approx(1.0)
+ assert result.loc["correlated", "p_value"] < 1e-10
+ assert result.loc["at_threshold", "r_value"] != 0.0
+ assert result.loc["too_sparse", "r_value"] == 0.0
+ assert np.isnan(result.loc["too_sparse", "p_value"])
+ assert np.isnan(result.loc["too_sparse", "p_value_adjusted"])
+ assert result.loc["constant", "r_value"] == 0.0
+ assert np.isnan(result.loc["constant", "p_value"])
+ assert np.isnan(result.loc["constant", "p_value_adjusted"])
+
+
+def test_regression_r_batch_matches_py_func():
+ rng = np.random.default_rng(0)
+ data = rng.poisson(0.8, size=(40, 8)).astype(np.float64)
+ regressor = np.linspace(0.0, 1.0, 40)
+ x_centered = regressor - regressor.mean()
+ ssxm = float(np.dot(x_centered, x_centered) / regressor.size)
+ kwargs = (
+ np.ascontiguousarray(data),
+ np.ascontiguousarray(x_centered),
+ ssxm,
+ 2,
+ float(np.finfo(float).eps),
+ )
+ compiled = _regression_r_batch(*kwargs)
+ python = _regression_r_batch.py_func(*kwargs)
+ np.testing.assert_allclose(compiled[0], python[0])
+ np.testing.assert_array_equal(compiled[1], python[1])
+
+
+def test_regression_batch_matches_linregress():
+ rng = np.random.default_rng(1)
+ n_cells = 50
+ regressor = np.linspace(-1.0, 2.0, n_cells)
+ data = np.column_stack(
+ [
+ 2.0 * regressor + 0.1,
+ -1.5 * regressor,
+ rng.poisson(0.5, size=n_cells).astype(float),
+ np.zeros(n_cells),
+ np.where(np.arange(n_cells) < 3, 1.0, 0.0),
+ ]
+ )
+ x_centered = regressor - regressor.mean()
+ ssxm = float(np.dot(x_centered, x_centered) / n_cells)
+ labels = np.array(["pos", "neg", "sparseish", "constant", "too_sparse"])
+ r_vals, p_vals, status = _regression_batch_results(
+ np.ascontiguousarray(data),
+ np.ascontiguousarray(x_centered),
+ ssxm,
+ regressor,
+ min_cells=5,
+ feature_labels=labels,
+ )
+ for i, label in enumerate(labels):
+ v = data[:, i]
+ if (v > 0).sum() >= 5 and np.ptp(v) > np.finfo(float).eps:
+ ref = linregress(regressor, v)
+ assert r_vals[i] == pytest.approx(ref.rvalue, rel=1e-10, abs=1e-12)
+ assert p_vals[i] == pytest.approx(ref.pvalue, rel=1e-8, abs=1e-12)
+ assert status[i] == 0
+ else:
+ assert r_vals[i] == 0.0
+ assert np.isnan(p_vals[i])
+ assert status[i] == 1
+
+
+def test_two_cell_regression_preserves_r_but_marks_inference_untested():
+ regressor = np.array([0.0, 1.0])
+ data = np.array([[0.0, 1.0, 1.0], [1.0, 0.0, 1.0]])
+ x_centered = regressor - regressor.mean()
+ r_vals, p_vals, status = _regression_batch_results(
+ data,
+ x_centered,
+ float(np.dot(x_centered, x_centered) / regressor.size),
+ regressor,
+ min_cells=1,
+ feature_labels=np.array(["increasing", "decreasing", "constant"]),
+ )
+
+ np.testing.assert_array_equal(r_vals, [1.0, -1.0, 0.0])
+ assert np.isnan(p_vals).all()
+ np.testing.assert_array_equal(
+ status,
+ np.full(data.shape[1], _REG_SENTINEL, dtype=np.int8),
+ )
+
+
+def test_find_markers_by_regression_two_cell_batches_are_unadjusted():
+ class Assay:
+ @staticmethod
+ def iter_normed_feature_wise(**_kwargs):
+ yield pd.DataFrame({"a": [0.0, 1.0], "b": [1.0, 1.0]})
+ yield pd.DataFrame({"c": [1.0, 0.0]})
+
+ result = find_markers_by_regression(
+ Assay(),
+ cell_key="I",
+ feat_key="I",
+ regressor=np.array([0.0, 1.0]),
+ min_cells=1,
+ )
+ assert result.loc["a", "r_value"] == pytest.approx(1.0)
+ assert result.loc["c", "r_value"] == pytest.approx(-1.0)
+ assert "p_value_adjusted" in result.columns
+ assert result.loc["b", "r_value"] == 0.0
+ assert result["p_value"].isna().all()
+ assert result["p_value_adjusted"].isna().all()
+
+
+def test_find_markers_by_regression_rejects_non_dataframe_batches():
+ class Assay:
+ @staticmethod
+ def iter_normed_feature_wise(**_kwargs):
+ yield np.ones((3, 1))
+
+ with pytest.raises(TypeError, match="DataFrames"):
+ find_markers_by_regression(
+ Assay(),
+ cell_key="I",
+ feat_key="I",
+ regressor=np.arange(3),
+ min_cells=1,
+ )
+
+
+def test_find_markers_by_regression_identifies_nonfinite_feature():
+ class Assay:
+ @staticmethod
+ def iter_normed_feature_wise(**_kwargs):
+ yield pd.DataFrame({"bad_feature": [0.0, np.nan, 1.0]})
+
+ with pytest.raises(ValueError, match="bad_feature"):
+ find_markers_by_regression(
+ Assay(),
+ cell_key="I",
+ feat_key="I",
+ regressor=np.arange(3),
+ min_cells=1,
+ )
+
+
+def test_find_markers_by_rank_rejects_fast_path_for_non_rna_assay():
+ import numba
+
+ class Cells:
+ @staticmethod
+ def fetch(_group_key, _cell_key):
+ return np.array([0, 0, 1, 1])
+
+ class Assay:
+ def __init__(self):
+ self.cells = Cells()
+ self.normMethod = norm_lib_size
+ self.sf = 1.0
+
+ previous_threads = numba.get_num_threads()
+ with pytest.raises(TypeError, match="requires an RNAassay"):
+ find_markers_by_rank(
+ Assay(),
+ group_key="cluster",
+ cell_key="I",
+ feat_key="I",
+ batch_size=2,
+ n_threads=1,
+ )
+ assert numba.get_num_threads() == previous_threads
+
+
+def test_find_markers_by_rank_slow_path_returns_groupwise_statistics():
+ data = np.array(
+ [
+ [2.0, 0.0, 0.0, 1.0],
+ [4.0, 0.0, 0.0, 1.0],
+ [0.0, 0.0, 5.0, 1.0],
+ [0.0, 0.0, 5.0, 1.0],
+ ]
+ )
+
+ class Cells:
+ @staticmethod
+ def fetch(_group_key, _cell_key):
+ return np.array(["a", "a", "b", "b"])
+
+ class Feats:
+ @staticmethod
+ def active_index(_feat_key):
+ return np.array([10, 11, 12, 13])
+
+ class Assay:
+ cells = Cells()
+ feats = Feats()
+ normMethod = None
+ sf = None
+
+ @staticmethod
+ def iter_normed_feature_wise(**_kwargs):
+ yield pd.DataFrame(data[:, :2])
+ yield pd.DataFrame(data[:, 2:])
+
+ results = find_markers_by_rank(
+ Assay(),
+ group_key="cluster",
+ cell_key="I",
+ feat_key="I",
+ batch_size=2,
+ n_threads=1,
+ )
+ group_a = results["a"].set_index("feature_index")
+ group_b = results["b"].set_index("feature_index")
+
+ assert group_a.loc[10, "fold_change"] == pytest.approx(100.1)
+ assert group_a.loc[11, "fold_change"] == pytest.approx(0.0)
+ assert group_a.loc[13, "fold_change"] == pytest.approx(1.0)
+ assert group_b.loc[12, "fold_change"] == pytest.approx(100.1)
+ assert group_b.loc[10, "fold_change"] == pytest.approx(0.0)
+ assert np.isfinite(group_a["p_value"]).all()
+ assert np.isfinite(group_b["p_value"]).all()
+ assert "auc" in group_a.columns
+ assert "p_value_adjusted" in group_a.columns
+ assert np.isfinite(group_a["auc"]).all()
+ assert np.isfinite(group_a["p_value_adjusted"]).all()
+
+
+@pytest.mark.parametrize("batch_format", ["dataframe", "tuple"])
+@pytest.mark.parametrize(
+ "bad_value",
+ [np.nan, np.inf],
+ ids=["nan", "inf"],
+)
+def test_find_markers_by_rank_rejects_nonfinite_slow_batches(
+ batch_format: str,
+ bad_value: float,
+) -> None:
+ data = np.array(
+ [
+ [2.0, 0.0, 1.0, 3.0],
+ [4.0, 0.0, 2.0, bad_value],
+ [0.0, 1.0, 3.0, 5.0],
+ [0.0, 2.0, 4.0, 6.0],
+ ]
+ )
+
+ class Cells:
+ @staticmethod
+ def fetch(_group_key: str, _cell_key: str) -> np.ndarray:
+ return np.array(["a", "a", "b", "b"])
+
+ class Feats:
+ @staticmethod
+ def active_index(_feat_key: str) -> np.ndarray:
+ return np.array([10, 11, 12, 13])
+
+ class Assay:
+ cells = Cells()
+ feats = Feats()
+ normMethod = None
+ sf = None
+
+ @staticmethod
+ def iter_normed_feature_wise(**_kwargs):
+ for start in (0, 2):
+ values = data[:, start : start + 2]
+ labels = np.array([10 + start, 11 + start])
+ if batch_format == "dataframe":
+ yield pd.DataFrame(values, columns=labels)
+ else:
+ yield values.T, labels
+
+ with pytest.raises(
+ ValueError,
+ match=r"Feature .*13.* contains non-finite normalized values",
+ ):
+ find_markers_by_rank(
+ Assay(),
+ group_key="cluster",
+ cell_key="I",
+ feat_key="I",
+ batch_size=2,
+ n_threads=1,
+ )
+
+
+def test_find_markers_fast_raw_path_computes_groupwise_statistics(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ import zarr
+ from zarr.storage import MemoryStore
+
+ from scarf.storage.budget import ResourceBudget
+
+ data = np.array(
+ [
+ [4.0, 0.0, 1.0, 0.0],
+ [3.0, 0.0, 1.0, 0.0],
+ [0.0, 5.0, 1.0, 2.0],
+ [0.0, 6.0, 1.0, 2.0],
+ ]
+ )
+
+ class Cells:
+ @staticmethod
+ def fetch(_group_key, _cell_key):
+ return np.array(["a", "a", "b", "b"])
+
+ @staticmethod
+ def active_index(_cell_key):
+ return np.arange(4)
+
+ @staticmethod
+ def fetch_all(_key):
+ return data.sum(axis=1)
+
+ class Feats:
+ @staticmethod
+ def active_index(_feat_key):
+ return np.arange(4)
+
+ class FakeRNA:
+ def __init__(self):
+ self.cells = Cells()
+ self.feats = Feats()
+ self.normMethod = norm_lib_size
+ self.sf = 1_000.0
+ self.name = "RNA"
+ self.resources = ResourceBudget(1024**3, 2)
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ self.raw = root.create_array(
+ "counts",
+ data=data.astype(np.uint32),
+ chunks=(2, 2),
+ )
+
+ def _raw_feature_stream_source(self):
+ return self.raw, 1, 0
+
+ @staticmethod
+ def iter_raw_feature_major_blocks(cell_idx, plan, **_kwargs):
+ for block in plan.blocks:
+ yield (
+ block,
+ np.ascontiguousarray(
+ data[np.asarray(cell_idx)][:, block.indices].T
+ ),
+ 0.01,
+ "memory",
+ )
+
+ monkeypatch.setattr(marker_search_module, "RNAassay", FakeRNA)
+ results = find_markers_by_rank(
+ FakeRNA(),
+ group_key="cluster",
+ cell_key="I",
+ feat_key="I",
+ batch_size=2,
+ n_threads=1,
+ )
+
+ assert set(results) == {"a", "b"}
+ for frame in results.values():
+ assert len(frame) == data.shape[1]
+ assert np.isfinite(frame["p_value"]).all()
+
+
+def test_iter_raw_feature_columns_matches_normed(datastore):
+ assay = datastore.RNA
+ cell_idx = assay.cells.active_index("I")
+ feat_idx = assay.feats.active_index("I")
+ scalar = assay.cells.fetch_all(assay.name + "_nCounts")[cell_idx]
+
+ streamed = controlled_compute(
+ assay.normed(cell_idx=cell_idx, feat_idx=feat_idx), assay.nthreads
+ )
+
+ cols = []
+ mats = []
+ for mat, batch_cols in assay.iter_raw_feature_columns(
+ cell_idx=cell_idx,
+ feat_idx=feat_idx,
+ batch_size=37,
+ scalar=scalar,
+ sf=float(assay.sf),
+ ):
+ mats.append(mat)
+ cols.append(batch_cols)
+
+ fast = np.hstack(mats)
+ assert np.array_equal(np.concatenate(cols), feat_idx)
+ assert fast.shape == streamed.shape
+ assert np.allclose(fast, streamed, rtol=1e-4, atol=1e-4)
+
+
+def test_compact_marker_save_roundtrip():
+ import zarr
+ from zarr.storage import MemoryStore
+
+ from scarf.datastore._operations.features import _load_marker_cluster_frame
+ from scarf.datastore.datastore import DataStore
+ from scarf.features.markers.table import MARKER_STAT_COLUMNS
+
+ index = np.array([10, 5, 7], dtype=np.int32)
+ source = pd.DataFrame(
+ {
+ "score": [0.9, 0.4, 0.2],
+ "mean": [1.0, 2.0, 3.0],
+ "mean_rest": [0.5, 0.5, 0.5],
+ "frac_exp": [0.8, 0.2, 0.1],
+ "frac_exp_rest": [0.3, 0.3, 0.3],
+ "fold_change": [2.0, 4.0, 6.0],
+ "p_value": [0.01, 0.02, 0.03],
+ "auc": [0.9, 0.7, 0.4],
+ "p_value_adjusted": [0.03, 0.03, 0.03],
+ },
+ index=index,
+ )
+ source["feature_index"] = source.index
+ source = sort_marker_results(source)
+ markers = {1: source}
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ slot = root.create_group("slot")
+ DataStore._write_marker_slot(
+ slot,
+ markers,
+ group_cell_counts={1: (10, 40)},
+ )
+ feature_names = np.array([f"g{i}" for i in range(11)])
+ loaded = _load_marker_cluster_frame(
+ slot,
+ slot["1"],
+ feature_names,
+ group_id=1,
+ )
+ assert "schema_version" not in slot.attrs
+ assert list(slot.attrs["stat_columns"]) == list(MARKER_STAT_COLUMNS)
+ assert slot["1"].attrs["n_group"] == 10
+ assert slot["1"].attrs["n_reference"] == 40
+ assert "feature_index" in slot
+ assert "stats" in slot["1"]
+ assert len(loaded) == 3
+ assert loaded.iloc[0]["score"] == 0.9
+ assert loaded.iloc[0]["feature_name"] == "g10"
+ assert loaded.iloc[0]["auc"] == 0.9
+ assert loaded.iloc[0]["p_value_adjusted"] == 0.03
+
+
+def _canonical_marker_frame(feature_indices):
+ from scarf.features.markers.table import MARKER_STAT_COLUMNS
+
+ index = np.asarray(feature_indices)
+ frame = pd.DataFrame(
+ {column: np.linspace(0.1, 0.2, len(index)) for column in MARKER_STAT_COLUMNS},
+ index=index,
+ )
+ frame["feature_index"] = index
+ return frame
+
+
+@pytest.mark.parametrize(
+ ("markers", "message"),
+ [
+ (
+ {
+ 0: _canonical_marker_frame([0, 1]),
+ 1: _canonical_marker_frame([0, 2]),
+ },
+ "identical feature index sets",
+ ),
+ (
+ {
+ 0: _canonical_marker_frame([0, 0]),
+ 1: _canonical_marker_frame([0, 0]),
+ },
+ "unique",
+ ),
+ ],
+)
+def test_marker_writer_rejects_invalid_indices_before_publication(
+ markers,
+ message,
+):
+ import zarr
+ from zarr.storage import MemoryStore
+
+ from scarf.datastore.datastore import DataStore
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ slot = root.create_group("slot")
+
+ with pytest.raises(ValueError, match=message):
+ DataStore._write_marker_slot(
+ slot,
+ markers,
+ group_cell_counts={0: (2, 2), 1: (2, 2)},
+ )
+
+ assert dict(slot.attrs) == {}
+ assert list(slot.array_keys()) == []
+ assert list(slot.group_keys()) == []
+
+
+def test_marker_publication_preserves_legacy_marker_subtree():
+ import zarr
+ from zarr.storage import MemoryStore
+
+ from scarf.datastore.datastore import DataStore
+ from scarf.features.markers.table import load_marker_table
+ from scarf.storage.artifact_writer import (
+ ArrayRequirement,
+ AttributeRequirement,
+ finish_artifact,
+ plan_artifact,
+ start_artifact,
+ )
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ legacy_slot = root.create_group("RNA/markers/I__legacy")
+ legacy_cluster = legacy_slot.create_group("1")
+ legacy_cluster.create_array(
+ "feature_index",
+ data=np.array([1], dtype=np.int32),
+ )
+ legacy_cluster.create_array("score", data=np.array([0.5]))
+ legacy_values = {
+ name: np.asarray(legacy_cluster[name][:]).copy()
+ for name in legacy_cluster.array_keys()
+ }
+ legacy_attrs = dict(legacy_slot.attrs)
+
+ planned = plan_artifact(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="marker_table",
+ operation="run_marker_search",
+ parameters={},
+ inputs={},
+ execution_options={},
+ required_arrays=(ArrayRequirement("feature_index", dtype_kind="i"),),
+ required_attributes=(
+ AttributeRequirement(
+ "stat_columns",
+ expected_types=(list, tuple),
+ ),
+ ),
+ )
+ artifact = start_artifact(root, planned)
+ source = pd.DataFrame(
+ {
+ "score": [0.8, 0.4],
+ "mean": [1.0, 0.5],
+ "mean_rest": [0.5, 0.5],
+ "frac_exp": [0.9, 0.3],
+ "frac_exp_rest": [0.2, 0.2],
+ "fold_change": [2.0, 1.0],
+ "p_value": [0.01, 0.04],
+ "auc": [0.9, 0.6],
+ "p_value_adjusted": [0.02, 0.04],
+ },
+ index=np.array([1, 0], dtype=np.int32),
+ )
+ DataStore._write_marker_slot(
+ artifact,
+ {1: source},
+ group_cell_counts={1: (10, 20)},
+ )
+ finish_artifact(artifact, planned)
+ root["RNA/markers"].attrs["artifacts"] = {
+ "I__new": planned.ref.to_dict(),
+ }
+
+ assert dict(legacy_slot.attrs) == legacy_attrs
+ assert set(legacy_cluster.array_keys()) == set(legacy_values)
+ for name, expected in legacy_values.items():
+ np.testing.assert_array_equal(legacy_cluster[name][:], expected)
+ loaded = load_marker_table(
+ artifact,
+ artifact["1"],
+ np.array(["g0", "g1"]),
+ group_id=1,
+ )
+ assert loaded["feature_name"].tolist() == ["g1", "g0"]
+ assert "schema_version" not in artifact.attrs
+
+
+def test_legacy_marker_names_and_scores_are_readable():
+ import zarr
+ from zarr.storage import MemoryStore
+
+ from scarf.datastore._operations.features import _load_marker_cluster_frame
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ slot = root.create_group("slot")
+ cluster = slot.create_group("1")
+ cluster.create_array(
+ "names",
+ data=np.array(["id2", "id0"]),
+ )
+ cluster.create_array(
+ "scores",
+ data=np.array([0.9, 0.8]),
+ )
+
+ loaded = _load_marker_cluster_frame(
+ slot,
+ cluster,
+ np.array(["gene0", "gene1", "gene2"]),
+ group_id=1,
+ feature_ids=np.array(["id0", "id1", "id2"]),
+ )
+
+ assert loaded["feature_index"].tolist() == [2, 0]
+ assert loaded["feature_name"].tolist() == ["gene2", "gene0"]
+ assert loaded["score"].tolist() == [0.9, 0.8]
+ assert loaded["p_value"].isna().all()
+ assert loaded["p_value_adjusted"].isna().all()
+
+
+def test_get_markers_preserves_unresolved_legacy_names(datastore_ephemeral):
+ assay = datastore_ephemeral.RNA
+ group_key = "legacy_marker_groups"
+ datastore_ephemeral.cells.insert(
+ group_key,
+ np.zeros(datastore_ephemeral.cells.N, dtype=np.int32),
+ overwrite=True,
+ )
+ markers_group = (
+ assay.z["markers"] if "markers" in assay.z else assay.z.create_group("markers")
+ )
+ slot = markers_group.create_group(f"I__{group_key}")
+ cluster = slot.create_group("0")
+ known_id = str(assay.feats.fetch_all("ids")[0])
+ known_name = str(assay.feats.fetch_all("names")[0])
+ cluster.create_array(
+ "names",
+ data=np.array([known_id, "removed_feature"]),
+ )
+ cluster.create_array("scores", data=np.array([0.9, 0.8]))
+
+ loaded = datastore_ephemeral.get_markers(
+ from_assay="RNA",
+ cell_key="I",
+ group_key=group_key,
+ group_id=0,
+ min_score=0,
+ min_frac_exp=0,
+ )
+
+ assert loaded["feature_name"].tolist() == [known_name, "removed_feature"]
+ assert loaded["feature_index"].dtype == pd.Int64Dtype()
+ assert loaded.iloc[0]["feature_index"] == 0
+ assert pd.isna(loaded.iloc[1]["feature_index"])
+
+
+def test_load_marker_table_ignores_stale_schema_version_attribute():
+ from scarf.features.markers.table import load_marker_table
+
+ slot, cluster = _make_canonical_marker_slot()
+ slot.attrs["schema_version"] = 99
+ loaded = load_marker_table(
+ slot,
+ cluster,
+ np.array(["g0", "g1"]),
+ group_id=1,
+ )
+
+ assert loaded["feature_name"].tolist() == ["g1", "g0"]
+
+
+def test_legacy_compact_marker_uses_stored_stat_columns():
+ import zarr
+ from zarr.storage import MemoryStore
+
+ from scarf.features.markers.table import load_marker_table
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ slot = root.create_group("slot")
+ slot.attrs["stat_columns"] = [
+ "p_value",
+ "score",
+ "mean_rest",
+ "frac_exp_rest",
+ "fold_change",
+ "mean",
+ "frac_exp",
+ ]
+ slot.create_array("feature_index", data=np.array([1, 0], dtype=np.int32))
+ cluster = slot.create_group("1")
+ stats = np.array(
+ [
+ [0.01, 0.8, 0.5, 0.2, 2.0, 1.0, 0.9],
+ [0.04, 0.4, 0.5, 0.2, 1.0, 0.5, 0.3],
+ ],
+ dtype=np.float64,
+ )
+ cluster.create_array("stats", data=stats)
+ loaded = load_marker_table(
+ slot,
+ cluster,
+ np.array(["g0", "g1", "g2"]),
+ group_id=1,
+ )
+ assert loaded.iloc[0]["feature_name"] == "g1"
+ assert loaded.iloc[0]["score"] == 0.8
+ from statsmodels.stats.multitest import multipletests
+
+ _, expected, _, _ = multipletests([0.01, 0.04], method="fdr_bh")
+ assert loaded["p_value_adjusted"].tolist() == pytest.approx(list(expected))
+
+
+def test_unversioned_compact_marker_roundtrip_preserves_reordered_named_stats():
+ import zarr
+ from zarr.storage import MemoryStore
+
+ from scarf.features.markers.table import (
+ MARKER_STAT_COLUMNS,
+ load_marker_table,
+ )
+
+ source = pd.DataFrame(
+ {
+ "score": [0.4, 0.8],
+ "mean": [0.5, 1.0],
+ "mean_rest": [0.5, 0.5],
+ "frac_exp": [0.3, 0.9],
+ "frac_exp_rest": [0.2, 0.2],
+ "fold_change": [1.0, 2.0],
+ "p_value": [0.04, 0.01],
+ "auc": [0.6, 0.9],
+ "p_value_adjusted": [0.04, 0.02],
+ }
+ )
+ stored_columns = tuple(reversed(MARKER_STAT_COLUMNS))
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ slot = root.create_group("slot")
+ slot.attrs["stat_columns"] = list(stored_columns)
+ slot.create_array("feature_index", data=np.array([0, 1], dtype=np.int32))
+ cluster = slot.create_group("1")
+ cluster.create_array(
+ "stats",
+ data=np.column_stack([source[column].to_numpy() for column in stored_columns]),
+ )
+
+ loaded = load_marker_table(
+ slot,
+ cluster,
+ np.array(["g0", "g1"]),
+ group_id=1,
+ )
+
+ assert "schema_version" not in slot.attrs
+ assert loaded["feature_index"].tolist() == [1, 0]
+ expected = source.iloc[[1, 0]]
+ for column in MARKER_STAT_COLUMNS:
+ np.testing.assert_allclose(loaded[column], expected[column])
+
+
+def _make_canonical_marker_slot(columns=None):
+ import zarr
+ from zarr.storage import MemoryStore
+
+ from scarf.features.markers.table import MARKER_STAT_COLUMNS
+
+ if columns is None:
+ columns = MARKER_STAT_COLUMNS
+ values = {
+ "score": np.array([0.4, 0.8]),
+ "mean": np.array([0.5, 1.0]),
+ "mean_rest": np.array([0.5, 0.5]),
+ "frac_exp": np.array([0.3, 0.9]),
+ "frac_exp_rest": np.array([0.2, 0.2]),
+ "fold_change": np.array([1.0, 2.0]),
+ "p_value": np.array([0.04, 0.01]),
+ "auc": np.array([0.6, 0.9]),
+ "p_value_adjusted": np.array([0.04, 0.02]),
+ }
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ slot = root.create_group("slot")
+ slot.attrs.update(
+ {
+ "stat_columns": list(columns),
+ "method": "mannwhitneyu",
+ "alternative": "two-sided",
+ "tie_correction": True,
+ "continuity_correction": True,
+ "adjustment_method": "fdr_bh",
+ "adjustment_scope": "within_group_all_tested_features",
+ }
+ )
+ slot.create_array("feature_index", data=np.array([0, 1], dtype=np.int32))
+ cluster = slot.create_group("1")
+ cluster.attrs.update({"n_group": 10, "n_reference": 20})
+ cluster.create_array(
+ "stats",
+ data=np.column_stack([values[column] for column in columns]),
+ )
+ return slot, cluster
+
+
+def test_canonical_marker_reader_accepts_reordered_named_columns():
+ from scarf.features.markers.table import (
+ MARKER_STAT_COLUMNS,
+ load_marker_table,
+ )
+
+ columns = tuple(reversed(MARKER_STAT_COLUMNS))
+ slot, cluster = _make_canonical_marker_slot(columns)
+ loaded = load_marker_table(
+ slot,
+ cluster,
+ np.array(["g0", "g1"]),
+ group_id=1,
+ )
+
+ assert loaded["feature_name"].tolist() == ["g1", "g0"]
+ assert loaded["score"].tolist() == pytest.approx([0.8, 0.4])
+ assert loaded["auc"].tolist() == pytest.approx([0.9, 0.6])
+ assert loaded["p_value_adjusted"].tolist() == pytest.approx([0.02, 0.04])
+
+
+def test_canonical_marker_reader_rejects_missing_stats_array():
+ from scarf.features.markers.table import load_marker_table
+
+ slot, cluster = _make_canonical_marker_slot()
+ del cluster["stats"]
+
+ with pytest.raises(ValueError, match="require feature_index and stats"):
+ load_marker_table(
+ slot,
+ cluster,
+ np.array(["g0", "g1"]),
+ group_id=1,
+ )
+
+
+def test_canonical_marker_reader_rejects_non_finite_statistics():
+ from scarf.features.markers.table import (
+ MARKER_STAT_COLUMNS,
+ load_marker_table,
+ )
+
+ slot, cluster = _make_canonical_marker_slot()
+ stats = np.asarray(cluster["stats"][:])
+ stats[0, MARKER_STAT_COLUMNS.index("fold_change")] = np.inf
+ cluster["stats"][:] = stats
+
+ with pytest.raises(ValueError, match="statistics must all be finite"):
+ load_marker_table(
+ slot,
+ cluster,
+ np.array(["g0", "g1"]),
+ group_id=1,
+ )
+
+
+def test_canonical_marker_reader_rejects_all_nan_adjusted_values():
+ from scarf.features.markers.table import (
+ MARKER_STAT_COLUMNS,
+ load_marker_table,
+ )
+
+ slot, cluster = _make_canonical_marker_slot()
+ stats = np.asarray(cluster["stats"][:])
+ stats[:, MARKER_STAT_COLUMNS.index("p_value_adjusted")] = np.nan
+ cluster["stats"][:] = stats
+
+ with pytest.raises(ValueError, match="p_value_adjusted.*finite"):
+ load_marker_table(
+ slot,
+ cluster,
+ np.array(["g0", "g1"]),
+ group_id=1,
+ )
+
+
+@pytest.mark.parametrize(
+ "corruption",
+ [
+ "incomplete_provenance",
+ "feature_identity",
+ "slot_metadata",
+ "group_metadata",
+ "stats_shape",
+ "missing_stats",
+ "stat_values",
+ "adjusted_values",
+ ],
+)
+def test_marker_cache_reuse_revalidates_canonical_payload(
+ datastore_ephemeral,
+ monkeypatch,
+ corruption,
+):
+ import scarf.features.markers as marker_algorithms
+ from scarf.features.markers.table import MARKER_STAT_COLUMNS
+ from scarf.storage.artifacts import ArtifactRef, artifact_path
+
+ assay = datastore_ephemeral.RNA
+ group_key = f"cache_groups_{corruption}"
+ feat_key = f"cache_features_{corruption}"
+ datastore_ephemeral.cells.insert(
+ group_key,
+ np.arange(datastore_ephemeral.cells.N) % 2,
+ overwrite=True,
+ )
+ feature_mask = np.zeros(assay.feats.N, dtype=bool)
+ feature_mask[:8] = True
+ assay.feats.insert(feat_key, feature_mask, overwrite=True)
+ arguments = {
+ "from_assay": "RNA",
+ "group_key": group_key,
+ "cell_key": "I",
+ "feat_key": feat_key,
+ "gene_batch_size": 4,
+ "n_threads": 1,
+ }
+ datastore_ephemeral.run_marker_search(**arguments)
+ marker_index = assay.z["markers"].attrs["artifacts"]
+ old_ref = ArtifactRef.from_dict(marker_index[f"I__{group_key}"])
+ old_artifact = datastore_ephemeral.zw[artifact_path(old_ref)]
+ first_group_name = sorted(old_artifact.group_keys())[0]
+ first_group = old_artifact[first_group_name]
+ if corruption == "incomplete_provenance":
+ provenance = dict(old_artifact.attrs["provenance"])
+ parameters = dict(provenance["parameters"])
+ for field_name in (
+ "method",
+ "alternative",
+ "tie_correction",
+ "continuity_correction",
+ "adjustment_method",
+ "adjustment_scope",
+ ):
+ parameters.pop(field_name)
+ provenance["parameters"] = parameters
+ old_artifact.attrs["provenance"] = provenance
+ elif corruption == "feature_identity":
+ stored_indices = np.asarray(old_artifact["feature_index"][:])
+ stored_indices[0] = int(stored_indices.max()) + 1
+ old_artifact["feature_index"][:] = stored_indices
+ elif corruption == "slot_metadata":
+ old_artifact.attrs["method"] = "ttest"
+ elif corruption == "group_metadata":
+ first_group.attrs["n_group"] = 1
+ elif corruption == "stats_shape":
+ stats = np.asarray(first_group["stats"][:, :-1])
+ del first_group["stats"]
+ first_group.create_array("stats", data=stats)
+ elif corruption == "missing_stats":
+ del first_group["stats"]
+ elif corruption == "stat_values":
+ stats = np.asarray(first_group["stats"][:])
+ stats[0, MARKER_STAT_COLUMNS.index("fold_change")] = np.inf
+ first_group["stats"][:] = stats
+ else:
+ stats = np.asarray(first_group["stats"][:])
+ stats[:, MARKER_STAT_COLUMNS.index("p_value_adjusted")] = np.nan
+ first_group["stats"][:] = stats
+
+ original = marker_algorithms.find_markers_by_rank
+ calls = 0
+
+ def tracked_marker_search(*args, **kwargs):
+ nonlocal calls
+ calls += 1
+ return original(*args, **kwargs)
+
+ monkeypatch.setattr(
+ marker_algorithms,
+ "find_markers_by_rank",
+ tracked_marker_search,
+ )
+ datastore_ephemeral.run_marker_search(**arguments)
+
+ new_ref = ArtifactRef.from_dict(
+ assay.z["markers"].attrs["artifacts"][f"I__{group_key}"]
+ )
+ status = datastore_ephemeral.inspect_artifact(new_ref)
+ assert calls == 1
+ assert new_ref != old_ref
+ assert status.parameters["method"] == "mannwhitneyu"
+ assert status.parameters["alternative"] == "two-sided"
+ assert status.parameters["tie_correction"] is True
+ assert status.parameters["continuity_correction"] is True
+ assert status.parameters["adjustment_method"] == "fdr_bh"
+ assert status.parameters["adjustment_scope"] == "within_group_all_tested_features"
+ assert "schema_version" not in status.parameters
+ assert "algorithm_version" not in status.parameters
+ assert "correction_method" not in status.parameters
+
+
+@pytest.mark.parametrize(
+ ("owner", "metadata_name"),
+ [
+ ("slot", "method"),
+ ("slot", "alternative"),
+ ("slot", "tie_correction"),
+ ("slot", "continuity_correction"),
+ ("slot", "adjustment_method"),
+ ("slot", "adjustment_scope"),
+ ("slot", "stat_columns"),
+ ("cluster", "n_group"),
+ ("cluster", "n_reference"),
+ ],
+)
+def test_canonical_marker_reader_rejects_incomplete_metadata(
+ owner,
+ metadata_name,
+):
+ from scarf.features.markers.table import load_marker_table
+
+ slot, cluster = _make_canonical_marker_slot()
+ target = slot if owner == "slot" else cluster
+ del target.attrs[metadata_name]
+
+ with pytest.raises(ValueError, match=metadata_name):
+ load_marker_table(
+ slot,
+ cluster,
+ np.array(["g0", "g1"]),
+ group_id=1,
+ )
+
+
+@pytest.mark.parametrize(
+ ("owner", "metadata_name", "value"),
+ [
+ ("slot", "method", "ttest"),
+ ("slot", "alternative", "greater"),
+ ("slot", "tie_correction", False),
+ ("slot", "continuity_correction", False),
+ ("slot", "adjustment_method", "bonferroni"),
+ ("slot", "adjustment_scope", "all_groups"),
+ ("cluster", "n_group", 1),
+ ("cluster", "n_reference", 1),
+ ],
+)
+def test_canonical_marker_reader_rejects_invalid_metadata(
+ owner,
+ metadata_name,
+ value,
+):
+ from scarf.features.markers.table import load_marker_table
+
+ slot, cluster = _make_canonical_marker_slot()
+ target = slot if owner == "slot" else cluster
+ target.attrs[metadata_name] = value
+
+ with pytest.raises(ValueError, match=metadata_name):
+ load_marker_table(
+ slot,
+ cluster,
+ np.array(["g0", "g1"]),
+ group_id=1,
+ )
+
+
+def test_canonical_marker_reader_rejects_malformed_stat_columns():
+ from scarf.features.markers.table import (
+ MARKER_STAT_COLUMNS,
+ load_marker_table,
+ )
+
+ slot, cluster = _make_canonical_marker_slot()
+ slot.attrs["stat_columns"] = [
+ *MARKER_STAT_COLUMNS[:-1],
+ MARKER_STAT_COLUMNS[0],
+ ]
+
+ with pytest.raises(ValueError, match="duplicate"):
+ load_marker_table(
+ slot,
+ cluster,
+ np.array(["g0", "g1"]),
+ group_id=1,
+ )
+
+
+def test_legacy_marker_reader_preserves_unresolved_feature_identity():
+ import zarr
+ from zarr.storage import MemoryStore
+
+ from scarf.features.markers.table import load_marker_table
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ slot = root.create_group("slot")
+ cluster = slot.create_group("1")
+ cluster.create_array("names", data=np.array(["known", "missing"]))
+ cluster.create_array("scores", data=np.array([0.8, 0.7]))
+
+ loaded = load_marker_table(
+ slot,
+ cluster,
+ np.array(["known"]),
+ group_id=1,
+ )
+
+ assert loaded["feature_name"].tolist() == ["known", "missing"]
+ assert loaded["feature_index"].dtype == pd.Int64Dtype()
+ assert loaded.iloc[0]["feature_index"] == 0
+ assert pd.isna(loaded.iloc[1]["feature_index"])
+
+
+def test_legacy_marker_reader_rejects_negative_feature_index():
+ import zarr
+ from zarr.storage import MemoryStore
+
+ from scarf.features.markers.table import LEGACY_STAT_COLUMNS, load_marker_table
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ slot = root.create_group("slot")
+ slot.create_array("feature_index", data=np.array([-1], dtype=np.int32))
+ cluster = slot.create_group("1")
+ cluster.create_array(
+ "stats",
+ data=np.zeros((1, len(LEGACY_STAT_COLUMNS))),
+ )
+
+ with pytest.raises(ValueError, match="unresolved or out-of-range"):
+ load_marker_table(
+ slot,
+ cluster,
+ np.array(["known"]),
+ group_id=1,
+ )
+
+
+def test_bh_adjusted_pvalues_match_statsmodels_and_preserve_order():
+ from scarf.features.markers.correction import _bh_adjusted_pvalues
+ from statsmodels.stats.multitest import multipletests
+
+ p_values = np.array([0.04, 0.01, 0.2, np.nan, 0.03])
+ adjusted = _bh_adjusted_pvalues(p_values)
+ mask = np.isfinite(p_values)
+ _, expected, _, _ = multipletests(p_values[mask], method="fdr_bh")
+ assert adjusted[mask].tolist() == pytest.approx(list(expected))
+ assert np.isnan(adjusted[3])
+ reordered = p_values[[1, 0, 4, 3, 2]]
+ adjusted_reordered = _bh_adjusted_pvalues(reordered)
+ restore = np.empty_like(adjusted_reordered)
+ restore[[1, 0, 4, 3, 2]] = adjusted_reordered
+ np.testing.assert_allclose(restore, adjusted, equal_nan=True)
+
+
+def test_marker_auc_matches_scipy_mannwhitneyu():
+ data = np.array(
+ [
+ [8.0, 0.0],
+ [7.0, 1.0],
+ [6.0, 1.0],
+ [5.0, 2.0],
+ [0.0, 3.0],
+ [0.0, 4.0],
+ [1.0, 5.0],
+ [2.0, 6.0],
+ ]
+ )
+ groups = np.array([0, 0, 0, 0, 1, 1, 1, 1])
+ group_counts = np.bincount(groups)
+ got = _batch_stats(data, groups, group_counts, len(groups))
+ for gene in range(data.shape[1]):
+ for group in (0, 1):
+ sample = data[groups == group, gene]
+ rest = data[groups != group, gene]
+ u = mannwhitneyu(
+ sample,
+ rest,
+ alternative="two-sided",
+ method="asymptotic",
+ use_continuity=True,
+ ).statistic
+ expected_auc = u / (len(sample) * len(rest))
+ assert got[gene, group, 7] == pytest.approx(expected_auc, abs=1e-6)
+
+
+def test_find_markers_by_rank_rejects_invalid_group_sizes():
+ class Cells:
+ def __init__(self, groups):
+ self._groups = groups
+
+ def fetch(self, _group_key, _cell_key):
+ return self._groups
+
+ class Feats:
+ @staticmethod
+ def active_index(_feat_key):
+ return np.array([0])
+
+ class Assay:
+ def __init__(self, groups):
+ self.cells = Cells(groups)
+ self.feats = Feats()
+ self.normMethod = None
+ self.sf = None
+
+ @staticmethod
+ def iter_normed_feature_wise(**_kwargs):
+ yield pd.DataFrame([[1.0], [2.0], [3.0]])
+
+ with pytest.raises(ValueError, match="at least two populated groups"):
+ find_markers_by_rank(
+ Assay(np.array([0, 0, 0])),
+ group_key="cluster",
+ cell_key="I",
+ feat_key="I",
+ batch_size=1,
+ n_threads=1,
+ )
+ with pytest.raises(ValueError, match="at least two cells in every group"):
+ find_markers_by_rank(
+ Assay(np.array([0, 1, 1])),
+ group_key="cluster",
+ cell_key="I",
+ feat_key="I",
+ batch_size=1,
+ n_threads=1,
+ )
+
+
+def test_pseudotime_bh_excludes_untested_features():
+ from scarf.features.markers.correction import _bh_adjusted_pvalues
+
+ class Assay:
+ @staticmethod
+ def iter_normed_feature_wise(**_kwargs):
+ yield pd.DataFrame(
+ {
+ "tested": [0.0, 1.0, 2.0, 3.0],
+ "untested": [0.0, 0.0, 0.0, 0.0],
+ }
+ )
+
+ result = find_markers_by_regression(
+ Assay(),
+ cell_key="I",
+ feat_key="I",
+ regressor=np.array([0.0, 1.0, 2.0, 3.0]),
+ min_cells=2,
+ )
+ assert np.isfinite(result.loc["tested", "p_value"])
+ assert np.isnan(result.loc["untested", "p_value"])
+ assert result.loc["tested", "p_value_adjusted"] == pytest.approx(
+ float(_bh_adjusted_pvalues(np.array([result.loc["tested", "p_value"]]))[0])
+ )
+ assert np.isnan(result.loc["untested", "p_value_adjusted"])
+
+
+def test_explicit_marker_gene_batch_size_reaches_search(
+ datastore_ephemeral,
+ monkeypatch,
+):
+ groups = np.arange(datastore_ephemeral.cells.N) % 2
+ datastore_ephemeral.cells.insert("batch_contract_groups", groups, overwrite=True)
+ captured = {}
+ expected = {"group": pd.DataFrame()}
+
+ def capture_marker_search(**kwargs):
+ captured.update(kwargs)
+ return expected
+
+ monkeypatch.setattr(
+ "scarf.features.markers.find_markers_by_rank",
+ capture_marker_search,
+ )
+ monkeypatch.setattr(
+ datastore_ephemeral,
+ "_get_latest_keys",
+ lambda from_assay, cell_key, feat_key: ("RNA", "I", "I"),
+ )
+
+ result = datastore_ephemeral.run_marker_search(
+ group_key="batch_contract_groups",
+ gene_batch_size=100,
+ skip_save=True,
+ )
+
+ assert result is expected
+ assert captured["batch_size"] == 100
diff --git a/tests/test_master_citeseq_compat.py b/tests/test_master_citeseq_compat.py
new file mode 100644
index 00000000..fe276723
--- /dev/null
+++ b/tests/test_master_citeseq_compat.py
@@ -0,0 +1,434 @@
+import hashlib
+import json
+import os
+from pathlib import Path
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.datastore.datastore import DataStore
+from scarf.datastore.graph_datastore import GraphDataStore
+from scarf.graph.encoded_paths import parse_assay_graph_paths
+from scarf.graph.paths import StoredAssayGraph
+from scarf.graph.state import validate_legacy_graph_selection
+
+
+_DEFAULT_CORPUS = Path("/home/parashar/data/scarf_master_compat/1k_citeseq")
+_MASTER_COMMIT = "1ce016ed17710b7daebcf187c34c6f9b23aae0b4"
+_FINAL_TREE_DIGEST = "b27aee365d89d6653d0ac5912f0044d6fe78e5f3a5e20bd164824085c441fedd"
+
+_RNA_GRAPH_SMALL = (
+ "RNA/normed__I__compat_hvgs_100/reduction__pca__8__I/"
+ "ann__l2__50__50__48__4466/knn__7/graph__1.0__1.5"
+)
+_RNA_GRAPH_ALTERNATE = (
+ "RNA/normed__I__compat_hvgs_100/reduction__pca__12__I/"
+ "ann__l2__50__50__48__4466/knn__11/graph__1.2__1.3"
+)
+_RNA_GRAPH_HARMONY = (
+ "RNA/normed__I__compat_hvgs_200/reduction__pca__10__I/"
+ "ann__l2__50__50__48__4466/knn__9/graph__1.0__1.5"
+)
+_ADT_GRAPH = (
+ "assay2/normed__I__I/reduction__pca__8__I/"
+ "ann__l2__50__50__48__4466/knn__9/graph__1.0__1.5"
+)
+
+
+class _ObservedMasterGraphStore(GraphDataStore):
+ @property
+ def assay_names(self) -> list[str]:
+ return ["RNA"]
+
+
+def _observed_multibranch_store() -> _ObservedMasterGraphStore:
+ datastore = _ObservedMasterGraphStore.__new__(_ObservedMasterGraphStore)
+ datastore.z = zarr.open_group(store=MemoryStore(), mode="w")
+ datastore.workspace = None
+ datastore.zarr_mode = "r+"
+ datastore._defaultAssay = "RNA"
+ datastore._integratedGraphsLoc = "integratedGraphs"
+ datastore._cachedMagicOperator = None
+ datastore._cachedMagicOperatorLoc = None
+ datastore.nthreads = 1
+
+ normed_path = "RNA/normed__I__compat_hvgs_100"
+ normed = datastore.z.create_group(normed_path)
+ for graph_path in (_RNA_GRAPH_SMALL, _RNA_GRAPH_ALTERNATE):
+ knn_path = graph_path.rsplit("/", 1)[0]
+ ann_path = knn_path.rsplit("/", 1)[0]
+ reduction_path = ann_path.rsplit("/", 1)[0]
+ reduction = datastore.z.create_group(reduction_path)
+ ann = datastore.z.create_group(ann_path)
+ knn = datastore.z.create_group(knn_path)
+ graph = datastore.z.create_group(graph_path)
+ k = int(knn_path.rsplit("/", 1)[-1].removeprefix("knn__"))
+ reduction.attrs["latest_ann"] = ann_path
+ ann.attrs["latest_knn"] = knn_path
+ knn.attrs["latest_graph"] = graph_path
+ knn.create_array("indices", data=np.zeros((3, k), dtype=np.uint64))
+ graph.attrs["n_cells"] = 3
+ graph.attrs["n_neighbors"] = k
+ graph.create_array(
+ "edges",
+ data=np.array(
+ [[0, 1], [0, 2], [1, 0], [1, 2], [2, 0], [2, 1]],
+ dtype=np.uint64,
+ ),
+ )
+ graph.create_array("weights", data=np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6]))
+ alternate_knn = _RNA_GRAPH_ALTERNATE.rsplit("/", 1)[0]
+ alternate_ann = alternate_knn.rsplit("/", 1)[0]
+ normed.attrs["latest_reduction"] = alternate_ann.rsplit("/", 1)[0]
+ return datastore
+
+
+def _corpus_root() -> Path:
+ return Path(os.environ.get("SCARF_MASTER_CITESEQ_CORPUS", _DEFAULT_CORPUS))
+
+
+def _tree_digest(root: Path) -> str:
+ digest = hashlib.blake2b(digest_size=32)
+ files = sorted(path for path in root.rglob("*") if path.is_file())
+ for path in files:
+ relative_path = str(path.relative_to(root))
+ payload = path.read_bytes()
+ encoded_path = relative_path.encode()
+ digest.update(len(encoded_path).to_bytes(8, "little"))
+ digest.update(encoded_path)
+ digest.update(len(payload).to_bytes(8, "little"))
+ digest.update(payload)
+ return digest.hexdigest()
+
+
+@pytest.fixture(scope="module")
+def master_citeseq_corpus() -> tuple[Path, dict]:
+ root = _corpus_root()
+ store = root / "enriched"
+ manifest_path = root / "master_manifest.json"
+ if not store.is_dir() or not manifest_path.is_file():
+ pytest.skip(
+ "Master CITE-seq corpus is unavailable. Set "
+ "SCARF_MASTER_CITESEQ_CORPUS to its external directory."
+ )
+ return store, json.loads(manifest_path.read_text())
+
+
+@pytest.fixture(scope="module")
+def master_citeseq_datastore(
+ master_citeseq_corpus: tuple[Path, dict],
+) -> DataStore:
+ store, _manifest = master_citeseq_corpus
+ return DataStore(
+ str(store),
+ default_assay="RNA",
+ assay_types={"assay2": "ADT"},
+ min_cells_per_feature=0,
+ min_features_per_cell=0,
+ zarr_mode="r",
+ )
+
+
+def test_observed_master_graph_branches_parse_without_fixture() -> None:
+ small = parse_assay_graph_paths(_RNA_GRAPH_SMALL)
+ alternate = parse_assay_graph_paths(_RNA_GRAPH_ALTERNATE)
+ harmony = parse_assay_graph_paths(_RNA_GRAPH_HARMONY)
+ adt = parse_assay_graph_paths(_ADT_GRAPH)
+
+ assert (small.dims, small.k, small.local_connectivity, small.bandwidth) == (
+ 8,
+ 7,
+ 1.0,
+ 1.5,
+ )
+ assert (
+ alternate.dims,
+ alternate.k,
+ alternate.local_connectivity,
+ alternate.bandwidth,
+ ) == (12, 11, 1.2, 1.3)
+ assert harmony.harmony_contract_hash is None
+ assert harmony.k == 9
+ assert adt.from_assay == "assay2"
+ assert adt.feat_key == "I"
+
+
+def test_observed_master_siblings_keep_latest_and_explicit_reads_distinct(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ datastore = _observed_multibranch_store()
+ monkeypatch.setattr(
+ "scarf.datastore._operations.graph.validate_legacy_graph_selection",
+ lambda *_args, **_kwargs: None,
+ )
+
+ assert (
+ datastore.get_latest_graph_loc("RNA", "I", "compat_hvgs_100")
+ == _RNA_GRAPH_ALTERNATE
+ )
+ latest = datastore.load_graph(
+ from_assay="RNA", cell_key="I", feat_key="compat_hvgs_100"
+ )
+ explicit = datastore.load_graph(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="compat_hvgs_100",
+ graph_loc=_RNA_GRAPH_SMALL,
+ )
+ assert latest.shape == (3, 3)
+ assert explicit.shape == (3, 3)
+ assert (latest != explicit).nnz == 0
+
+
+@pytest.mark.integration
+def test_master_citeseq_manifest_covers_requested_artifacts(
+ master_citeseq_corpus: tuple[Path, dict],
+) -> None:
+ store, manifest = master_citeseq_corpus
+ successful = {
+ step["name"] for step in manifest["steps"] if step["status"] == "success"
+ }
+ steps = {step["name"]: step for step in manifest["steps"]}
+ required = {
+ "mark_hvgs_100",
+ "mark_hvgs_200",
+ "rna_graph_small",
+ "rna_graph_small_repeat",
+ "rna_graph_alternate",
+ "rna_graph_unscaled",
+ "rna_graph_harmony",
+ "adt_graph",
+ "integrate_assays_snn",
+ "integrate_assays_wnn",
+ "run_umap_a",
+ "run_umap_b",
+ "run_leiden_0_5",
+ "run_leiden_1_0",
+ "run_leiden_1_5",
+ "run_paris_fixed",
+ "run_paris_balanced",
+ "run_marker_search",
+ "get_imputed_t2",
+ "get_imputed_t3",
+ "run_pseudotime_scoring",
+ "run_pseudotime_marker_search",
+ "run_pseudotime_aggregation",
+ "run_mapping",
+ "run_mapping_coral",
+ "run_unified_umap",
+ "run_topacedo_sampler",
+ "metric_lisi",
+ }
+
+ assert manifest["master_commit"] == _MASTER_COMMIT
+ assert required <= successful
+ assert manifest["summary"] == {"failed": 2, "skipped": 1, "success": 39}
+ assert len(manifest["layout"]) == 308
+ assert manifest["final_tree_digest"] == _FINAL_TREE_DIGEST
+ assert _tree_digest(store) == _FINAL_TREE_DIGEST
+ assert (
+ steps["rna_graph_small"]["result"] == steps["rna_graph_small_repeat"]["result"]
+ )
+ assert steps["rna_graph_small_repeat"]["changed_file_count"] > 0
+ assert steps["rna_graph_unscaled"]["result"] == steps["rna_graph_harmony"]["result"]
+ assert any(
+ path.endswith("/harmonizedData")
+ for path in steps["rna_graph_harmony"]["changed_nodes"]
+ )
+ assert any(
+ path.startswith("RNA/normed__")
+ for path in steps["integrate_assays_wnn"]["changed_nodes"]
+ )
+
+
+@pytest.mark.integration
+def test_current_reader_loads_master_graph_branches(
+ master_citeseq_datastore: DataStore,
+) -> None:
+ datastore = master_citeseq_datastore
+ n_cells = int(datastore.cells.fetch_all("I").sum())
+
+ assert (
+ datastore.get_normalized_group_path("RNA", "I", "compat_hvgs_100")
+ == "RNA/normed__I__compat_hvgs_100"
+ )
+ assert (
+ datastore.get_latest_graph_loc("RNA", "I", "compat_hvgs_100")
+ == _RNA_GRAPH_ALTERNATE
+ )
+ assert (
+ datastore.get_latest_graph_loc("RNA", "I", "compat_hvgs_200")
+ == _RNA_GRAPH_HARMONY
+ )
+ assert datastore.get_latest_graph_loc(
+ "RNA", "I", "compat_hvgs_100"
+ ) == datastore.get_latest_graph_loc("RNA", "I", "compat_hvgs_100")
+
+ small = datastore.load_graph(graph_loc=_RNA_GRAPH_SMALL)
+ alternate = datastore.load_graph(graph_loc=_RNA_GRAPH_ALTERNATE)
+ harmony = datastore.load_graph(graph_loc=_RNA_GRAPH_HARMONY)
+ adt = datastore.load_graph(graph_loc=_ADT_GRAPH)
+ snn = datastore.load_graph(graph_loc="integratedGraphs/compat_snn")
+ wnn = datastore.load_graph(graph_loc="integratedGraphs/compat_wnn")
+
+ for graph in (small, alternate, harmony, adt, snn, wnn):
+ assert graph.shape == (n_cells, n_cells)
+ assert graph.nnz > 0
+
+ stored = datastore._lookup_stored_graph("RNA", "I", "compat_hvgs_100")
+ assert isinstance(stored, StoredAssayGraph)
+ assert stored.paths.cell_graph_group_path == _RNA_GRAPH_ALTERNATE
+ assert stored.dims == 12
+ assert stored.k == 11
+
+ harmony_knn = _RNA_GRAPH_HARMONY.rsplit("/", 1)[0]
+ harmony_ann = harmony_knn.rsplit("/", 1)[0]
+ harmony_reduction = harmony_ann.rsplit("/", 1)[0]
+ assert datastore.zw[harmony_ann].attrs["isHarmonized"] is False
+ assert "harmonizedData" in datastore.zw[harmony_reduction]
+
+ ann_stream = datastore._load_ann_stream(
+ "RNA",
+ "I",
+ "compat_hvgs_100",
+ knn_loc=_RNA_GRAPH_ALTERNATE.rsplit("/", 1)[0],
+ )
+ assert ann_stream.annIdx is not None
+ assert ann_stream.k == 11
+
+ initial_embedding = datastore._get_ini_embed("RNA", "I", "compat_hvgs_100", 2)
+ assert initial_embedding.shape == (n_cells, 2)
+ assert np.isfinite(initial_embedding).all()
+
+
+@pytest.mark.integration
+def test_current_reader_validates_released_selection_hash(
+ master_citeseq_datastore: DataStore,
+) -> None:
+ datastore = master_citeseq_datastore
+ knn_path = _RNA_GRAPH_ALTERNATE.rsplit("/", 1)[0]
+ stored_hash = datastore.zw["RNA/normed__I__compat_hvgs_100"].attrs["subset_hash"]
+
+ assert isinstance(stored_hash, int) and not isinstance(stored_hash, bool)
+ validate_legacy_graph_selection(
+ datastore,
+ knn_path,
+ "RNA",
+ "I",
+ "compat_hvgs_100",
+ )
+ assert datastore._keys_from_knn_path("RNA", knn_path) == (
+ "I",
+ "compat_hvgs_100",
+ )
+
+
+@pytest.mark.integration
+def test_current_reader_loads_master_derived_results(
+ master_citeseq_datastore: DataStore,
+) -> None:
+ datastore = master_citeseq_datastore
+ expected_columns = {
+ "RNA_compat_umap_a1",
+ "RNA_compat_umap_a2",
+ "RNA_compat_umap_b1",
+ "RNA_compat_umap_b2",
+ "RNA_compat_leiden_05",
+ "RNA_compat_leiden_1",
+ "RNA_compat_leiden_15",
+ "RNA_compat_paris_8",
+ "RNA_compat_paris_balanced",
+ "RNA_compat_pseudotime",
+ "RNA_compat_cell_cycle",
+ "RNA_I_cluster_membership_strength",
+ }
+ assert expected_columns <= set(datastore.cells.columns)
+
+ umap = np.column_stack(
+ (
+ datastore.cells.fetch_all("RNA_compat_umap_a1"),
+ datastore.cells.fetch_all("RNA_compat_umap_a2"),
+ )
+ )
+ assert np.isfinite(umap[datastore.cells.fetch_all("I")]).all()
+ assert len(set(datastore.cells.fetch("RNA_compat_leiden_1").tolist())) > 1
+
+ markers = datastore.get_markers(
+ from_assay="RNA",
+ cell_key="I",
+ group_key="RNA_compat_leiden_1",
+ group_id=1,
+ min_score=0.2,
+ min_frac_exp=0,
+ )
+ assert not markers.empty
+ assert {"feature_name", "score", "p_value"} <= set(markers.columns)
+
+ feature_name = str(datastore.get_assay("RNA").feats.fetch_all("names")[0])
+ imputed = datastore.get_imputed(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="compat_hvgs_100",
+ feature_name=feature_name,
+ t=2,
+ )
+ assert imputed.shape == datastore.cells.fetch_all("I").shape
+
+ with pytest.raises(
+ ValueError,
+ match=r"build_mapping_reference\(neighbors\)",
+ ):
+ datastore.get_mapping_reference(from_assay="RNA")
+
+ assert (
+ datastore.list_artifacts(
+ kind="projection",
+ from_assay="RNA",
+ )
+ == []
+ )
+ for legacy_name in ("compat_self", "compat_unified_umap"):
+ with pytest.raises(ValueError, match="MappingReference is required"):
+ datastore.get_mapping_result(legacy_name)
+
+ with pytest.raises(ValueError, match="MappingReference is required"):
+ datastore.get_target_classes(
+ "compat_self",
+ reference_class_group="RNA_compat_leiden_1",
+ )
+ with pytest.raises(ValueError, match="MappingReference is required"):
+ list(datastore.get_mapping_score("compat_self"))
+
+ with pytest.raises(AttributeError, match="_load_unified_layout_data"):
+ datastore._load_unified_layout_data("compat_unified_umap", "RNA")
+ with pytest.raises(AttributeError, match="unified_embedding"):
+ datastore.plots.unified_embedding()
+
+
+@pytest.mark.integration
+def test_current_reader_does_not_mutate_enriched_master_store(
+ master_citeseq_corpus: tuple[Path, dict],
+) -> None:
+ store, manifest = master_citeseq_corpus
+ before = _tree_digest(store)
+ datastore = DataStore(
+ str(store),
+ default_assay="RNA",
+ assay_types={"assay2": "ADT"},
+ min_cells_per_feature=0,
+ min_features_per_cell=0,
+ zarr_mode="r",
+ )
+
+ datastore.get_normalized_group_path("RNA", "I", "compat_hvgs_100")
+ datastore.get_latest_graph_loc("RNA", "I", "compat_hvgs_100")
+ datastore.load_graph(graph_loc=_RNA_GRAPH_SMALL)
+ datastore.load_graph(graph_loc="integratedGraphs/compat_wnn")
+ datastore._lookup_stored_graph("RNA", "I", "compat_hvgs_100")
+ datastore._get_ini_embed("RNA", "I", "compat_hvgs_100", 2)
+
+ after = _tree_digest(store)
+ assert before == manifest["final_tree_digest"]
+ assert after == before
diff --git a/tests/test_matrix_contracts.py b/tests/test_matrix_contracts.py
new file mode 100644
index 00000000..aae5688b
--- /dev/null
+++ b/tests/test_matrix_contracts.py
@@ -0,0 +1,43 @@
+import pickle
+from typing import get_type_hints
+
+import numpy as np
+
+import scarf.matrix as matrix
+from scarf.matrix.blocks import Block as implementation_block
+from scarf.matrix.chunked import ChunkedArray as implementation_chunked_array
+from tests.signature_contracts import signature_digest
+
+
+def test_matrix_facade_exports_canonical_classes():
+ assert matrix.__all__ == ["Block", "ChunkedArray"]
+ assert matrix.Block is implementation_block
+ assert matrix.ChunkedArray is implementation_chunked_array
+ assert matrix.Block.__module__ == "scarf.matrix"
+ assert matrix.ChunkedArray.__module__ == "scarf.matrix"
+
+
+def test_chunked_array_signatures_remain_stable():
+ methods = {
+ "ChunkedArray.__init__": matrix.ChunkedArray.__init__,
+ "ChunkedArray.from_numpy": matrix.ChunkedArray.from_numpy,
+ "ChunkedArray.stream_blocks": matrix.ChunkedArray.stream_blocks,
+ "ChunkedArray.dot": matrix.ChunkedArray.dot,
+ }
+
+ assert signature_digest(methods) == (
+ "0ac8b1f50482e55a8697f2fba39d0fa51c3b588c81923a0c70f4ef5a8ae2fee7"
+ )
+
+
+def test_matrix_type_hints_and_pickle_paths_resolve():
+ assert get_type_hints(matrix.Block.__init__)["parent"] is matrix.ChunkedArray
+ original = matrix.ChunkedArray.from_numpy(
+ np.arange(6, dtype=np.float64).reshape(3, 2),
+ block_size=2,
+ )
+
+ restored = pickle.loads(pickle.dumps(original))
+
+ assert type(restored) is matrix.ChunkedArray
+ np.testing.assert_array_equal(restored.compute(), original.compute())
diff --git a/tests/test_matrix_source.py b/tests/test_matrix_source.py
new file mode 100644
index 00000000..07fac069
--- /dev/null
+++ b/tests/test_matrix_source.py
@@ -0,0 +1,592 @@
+from pathlib import Path
+
+import numpy as np
+import pytest
+import zarr
+from obstore.store import MemoryStore as ObjectMemoryStore
+from zarr.abc.store import Store
+from zarr.codecs import BloscCodec, ZstdCodec
+from zarr.storage import ObjectStore
+
+from scarf.datastore.datastore import DataStore, mount_datastore
+from scarf.storage.budget import ResourceBudget
+from scarf.storage.sharding import write_counts_t
+from scarf.storage.stores import (
+ MATRIX_SOURCE_ATTR,
+ create_matrix_source,
+ is_remote_datastore,
+ resolve_matrix_source,
+)
+from tests.fixtures_datastore import build_neighbourhood_graph
+from scarf.writers import (
+ create_cell_data,
+ create_zarr_count_assay,
+ create_zarr_obj_array,
+)
+
+
+def _write_assay(
+ root: zarr.Group,
+ workspace: str | None,
+ assay_name: str,
+ values: np.ndarray,
+ *,
+ dataset_fingerprint: str | None = None,
+) -> None:
+ n_cells, n_feats = values.shape
+ create_zarr_count_assay(
+ z=root,
+ assay_name=assay_name,
+ workspace=workspace,
+ n_cells=n_cells,
+ feat_ids=np.array([f"{assay_name.lower()}-f{i}" for i in range(n_feats)]),
+ feat_names=np.array([f"{assay_name.lower()}-g{i}" for i in range(n_feats)]),
+ dtype="uint32",
+ )
+ if workspace is None:
+ counts = root[f"{assay_name}/counts"]
+ assay = root[assay_name]
+ else:
+ counts = root[f"matrices/{assay_name}/counts"]
+ assay = root[f"{workspace}/{assay_name}"]
+ counts[:] = values
+ matrix_group = (
+ root[assay_name] if workspace is None else root[f"matrices/{assay_name}"]
+ )
+ write_counts_t(
+ counts,
+ matrix_group,
+ resources=ResourceBudget(1024**2, 2),
+ )
+ if dataset_fingerprint is not None:
+ assay.attrs["dataset_fingerprint"] = dataset_fingerprint
+
+
+def _write_source_store(
+ path: str | Store,
+ *,
+ workspace: str | None,
+ values: np.ndarray | None = None,
+ dataset_fingerprint: str | None = None,
+) -> np.ndarray:
+ if values is None:
+ values = np.arange(40, dtype=np.uint32).reshape(10, 4)
+ n_cells, n_feats = values.shape
+ root = zarr.open_group(path, mode="w")
+ create_cell_data(
+ root,
+ workspace,
+ ids=np.array([f"c{i}" for i in range(n_cells)]),
+ names=np.array([f"c{i}" for i in range(n_cells)]),
+ )
+ _write_assay(
+ root,
+ workspace,
+ "RNA",
+ values,
+ dataset_fingerprint=dataset_fingerprint,
+ )
+ if workspace is None:
+ zw = root
+ else:
+ zw = root[workspace]
+ zw.attrs["defaultAssay"] = "RNA"
+ zw.attrs["assayTypes"] = {"RNA": "RNA"}
+ return values
+
+
+def _snapshot_store_files(path: str) -> dict[str, bytes]:
+ root = Path(path)
+ return {
+ str(file.relative_to(root)): file.read_bytes()
+ for file in root.rglob("*")
+ if file.is_file()
+ }
+
+
+@pytest.mark.parametrize("workspace", [None, "analysis"])
+def test_mount_datastore_creates_and_reopens(tmp_path, workspace):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ values = _write_source_store(source, workspace=workspace)
+
+ ds = mount_datastore(
+ source,
+ at=target,
+ workspace=workspace,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+ assert ds.workspace == workspace
+ assert "counts" not in ds.z["RNA" if workspace is None else f"{workspace}/RNA"]
+ np.testing.assert_array_equal(ds.RNA.rawData.compute(), values)
+
+ reopened = DataStore(
+ target,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+ assert reopened.workspace == workspace
+ np.testing.assert_array_equal(reopened.RNA.rawData.compute(), values)
+ assert MATRIX_SOURCE_ATTR in zarr.open_group(target, mode="r").attrs
+
+
+def test_mount_datastore_multiple_assays(tmp_path):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ rna_values = _write_source_store(source, workspace=None)
+ adt_values = np.arange(30, dtype=np.uint32).reshape(10, 3)
+ source_root = zarr.open_group(source, mode="r+")
+ _write_assay(source_root, None, "ADT", adt_values)
+ source_root.attrs["assayTypes"] = {"RNA": "RNA", "ADT": "ADT"}
+
+ ds = mount_datastore(
+ source,
+ at=target,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+ np.testing.assert_array_equal(ds.RNA.rawData.compute(), rna_values)
+ np.testing.assert_array_equal(ds.ADT.rawData.compute(), adt_values)
+ manifest = zarr.open_group(target, mode="r").attrs[MATRIX_SOURCE_ATTR]["assays"]
+ assert (
+ manifest["RNA"]["cellIdsFingerprint"] == manifest["ADT"]["cellIdsFingerprint"]
+ )
+ assert "counts" not in zarr.open_group(target, mode="r")["ADT"]
+
+
+def test_mounted_store_reopens_from_another_directory(monkeypatch, tmp_path):
+ monkeypatch.chdir(tmp_path)
+ values = _write_source_store("source.zarr", workspace=None)
+ mount_datastore(
+ "source.zarr",
+ at="target.zarr",
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+ target = str(tmp_path / "target.zarr")
+ location = zarr.open_group(target, mode="r").attrs[MATRIX_SOURCE_ATTR]["location"]
+ assert Path(location).resolve() == (tmp_path / "source.zarr").resolve()
+
+ monkeypatch.chdir(tmp_path.parent)
+ reopened = DataStore(
+ target,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+ np.testing.assert_array_equal(reopened.RNA.rawData.compute(), values)
+
+
+@pytest.mark.parametrize("target_kind", ["path", "store"])
+def test_failed_mount_discards_target_and_allows_retry(
+ monkeypatch,
+ tmp_path,
+ target_kind,
+):
+ source = str(tmp_path / "source.zarr")
+ _write_source_store(source, workspace=None)
+ target: str | Store = (
+ str(tmp_path / "target.zarr")
+ if target_kind == "path"
+ else ObjectStore(store=ObjectMemoryStore())
+ )
+
+ from scarf.storage import copy as copy_module
+
+ def fail_copy(*args, **kwargs):
+ raise OSError("injected metadata copy failure")
+
+ monkeypatch.setattr(copy_module, "copy_zarr_group_tree", fail_copy)
+ with pytest.raises(OSError, match="injected metadata copy failure"):
+ create_matrix_source(source, target, workspace=None)
+ if isinstance(target, str):
+ assert not Path(target).exists()
+ monkeypatch.undo()
+
+ retried = create_matrix_source(source, target, workspace=None)
+ assert MATRIX_SOURCE_ATTR in retried.attrs
+ assert "ids" in retried["cellData"]
+
+
+@pytest.mark.parametrize(
+ ("options", "error"),
+ [
+ ({"zarr_mode": "r"}, ValueError),
+ ({"zarr_loc": "elsewhere.zarr"}, TypeError),
+ ],
+)
+def test_mount_datastore_rejects_conflicting_options(tmp_path, options, error):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ _write_source_store(source, workspace=None)
+
+ with pytest.raises(error):
+ mount_datastore(
+ source,
+ at=target,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ **options,
+ )
+ assert not Path(target).exists()
+
+
+def test_mount_datastore_rejects_existing_target(tmp_path):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ _write_source_store(source, workspace=None)
+ mount_datastore(
+ source,
+ at=target,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+ with pytest.raises(FileExistsError):
+ mount_datastore(
+ source,
+ at=target,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+
+
+def test_mounted_store_writes_only_to_target(tmp_path):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ _write_source_store(source, workspace=None)
+ source_before = _snapshot_store_files(source)
+
+ ds = mount_datastore(
+ source,
+ at=target,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+ ds.cells.insert("mounted_flag", np.ones(ds.cells.N, dtype=bool), overwrite=True)
+ mask = np.zeros(ds.cells.N, dtype=bool)
+ mask[:3] = True
+ ds.cells.update_key(mask, key="I")
+
+ assert _snapshot_store_files(source) == source_before
+ assert "mounted_flag" in zarr.open_group(target, mode="r")["cellData"]
+ assert int(np.asarray(ds.cells.fetch_all("I")).sum()) == 3
+
+
+def test_mounted_store_computes_markers_without_writing_source(tmp_path):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ values = np.array(
+ [
+ [4, 0, 1, 0],
+ [3, 0, 1, 0],
+ [0, 5, 1, 2],
+ [0, 6, 1, 2],
+ ],
+ dtype=np.uint32,
+ )
+ _write_source_store(source, workspace=None, values=values)
+ source_before = _snapshot_store_files(source)
+ ds = mount_datastore(
+ source,
+ at=target,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+ ds.cells.insert(
+ "marker_groups",
+ np.array(["a", "a", "b", "b"]),
+ overwrite=True,
+ )
+
+ markers = ds.run_marker_search(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="I",
+ group_key="marker_groups",
+ gene_batch_size=None,
+ n_threads=1,
+ skip_save=True,
+ )
+
+ assert set(markers) == {"a", "b"}
+ assert _snapshot_store_files(source) == source_before
+ assert "markers" not in zarr.open_group(target, mode="r")["RNA"]
+
+
+@pytest.mark.parametrize(
+ ("group_path", "prefix"),
+ [
+ ("cellData", "cell"),
+ ("RNA/featureData", "feature"),
+ ],
+)
+def test_matrix_source_id_mismatch_fails_closed(tmp_path, group_path, prefix):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ _write_source_store(source, workspace=None)
+ create_matrix_source(source, target, workspace=None)
+
+ source_root = zarr.open_group(source, mode="r+")
+ group = source_root[group_path]
+ n_rows = group["ids"].shape[0]
+ create_zarr_obj_array(
+ group,
+ "ids",
+ np.array([f"{prefix}-{i}" for i in range(n_rows)]),
+ overwrite=True,
+ )
+ with pytest.raises(ValueError, match="cell or feature identifiers"):
+ DataStore(target, default_assay="RNA")
+
+
+@pytest.mark.parametrize(
+ ("shape", "dtype"),
+ [
+ ((9, 4), "uint32"),
+ ((10, 4), "uint16"),
+ ],
+)
+def test_matrix_source_count_identity_mismatch_fails_closed(
+ tmp_path,
+ shape,
+ dtype,
+):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ _write_source_store(source, workspace=None)
+ create_matrix_source(source, target, workspace=None)
+
+ source_root = zarr.open_group(source, mode="r+")
+ source_root["RNA"].create_array(
+ "counts",
+ data=np.zeros(shape, dtype=dtype),
+ chunks=(5, 2),
+ overwrite=True,
+ )
+ with pytest.raises(ValueError, match="count matrix identity"):
+ DataStore(target, default_assay="RNA")
+
+
+@pytest.mark.parametrize(
+ ("parent_path", "name"),
+ [
+ ("", "cellData"),
+ ("RNA", "featureData"),
+ ("RNA", "counts"),
+ ],
+)
+def test_invalid_source_does_not_create_target(tmp_path, parent_path, name):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ _write_source_store(source, workspace=None)
+ source_root = zarr.open_group(source, mode="r+")
+ parent = source_root if not parent_path else source_root[parent_path]
+ del parent[name]
+
+ with pytest.raises(KeyError):
+ create_matrix_source(source, target, workspace=None)
+ assert not Path(target).exists()
+
+
+def test_matrix_source_dataset_fingerprint_fast_path(tmp_path):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ _write_source_store(
+ source,
+ workspace=None,
+ dataset_fingerprint="abc123",
+ )
+ create_matrix_source(source, target, workspace=None)
+ manifest = zarr.open_group(target, mode="r").attrs[MATRIX_SOURCE_ATTR]
+ assert manifest["assays"]["RNA"]["datasetFingerprint"] == "abc123"
+ assert manifest["assays"]["RNA"]["cellIdsFingerprint"] is None
+
+ source_root = zarr.open_group(source, mode="r+")
+ source_root["RNA"].attrs["dataset_fingerprint"] = "changed"
+ with pytest.raises(ValueError, match="dataset fingerprint"):
+ resolve_matrix_source(zarr.open_group(target, mode="r"))
+
+
+def test_dataset_fingerprint_fast_path_reads_no_identifiers(monkeypatch, tmp_path):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ _write_source_store(source, workspace=None, dataset_fingerprint="abc123")
+ create_matrix_source(source, target, workspace=None)
+
+ read_array = zarr.Array.__getitem__
+
+ def reject_identifier_reads(array, selection):
+ if array.path.endswith(("cellData/ids", "featureData/ids")):
+ raise AssertionError(f"Fast path read identifiers at {array.path}")
+ return read_array(array, selection)
+
+ monkeypatch.setattr(zarr.Array, "__getitem__", reject_identifier_reads)
+ _, workspace = resolve_matrix_source(zarr.open_group(target, mode="r"))
+ assert workspace is None
+
+
+def test_source_open_does_not_change_target_profile(tmp_path):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ _write_source_store(source, workspace=None)
+
+ create_matrix_source(source, target, workspace=None)
+ target_ids = zarr.open_group(target, mode="r")["cellData/ids"]
+ assert isinstance(target_ids.compressors[0], BloscCodec)
+
+
+def test_mount_profile_applies_to_store_target(tmp_path):
+ source = str(tmp_path / "source.zarr")
+ _write_source_store(source, workspace=None)
+ target_store = ObjectStore(store=ObjectMemoryStore())
+
+ ds = mount_datastore(
+ source,
+ at=target_store,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ zarrProfile="cloud",
+ )
+
+ target_ids = ds.z["cellData/ids"]
+ assert isinstance(target_ids.compressors[0], ZstdCodec)
+
+
+_STATS_COLUMNS = ("normed_tot", "avg", "nz_mean", "sigmas", "normed_n")
+
+
+def test_mounted_datastore_reads_remote_counts_as_remote(monkeypatch, tmp_path):
+ values = np.arange(1, 41, dtype=np.uint32).reshape(10, 4)
+ reference_path = str(tmp_path / "reference.zarr")
+ _write_source_store(reference_path, workspace=None, values=values)
+ reference = DataStore(
+ reference_path,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+ reference.RNA.set_feature_stats("I")
+ expected_stats = reference.RNA.get_feature_stats("I", columns=_STATS_COLUMNS)
+
+ remote_store = ObjectStore(store=ObjectMemoryStore())
+ _write_source_store(remote_store, workspace=None, values=values)
+ location = "s3://atlas/pbmc.zarr"
+
+ from scarf.storage import stores as stores_module
+
+ real_make_store = stores_module.make_store
+
+ def fake_make_store(loc, *, storage_options=None, read_only=False):
+ if loc == location:
+ return remote_store
+ return real_make_store(
+ loc,
+ storage_options=storage_options,
+ read_only=read_only,
+ )
+
+ monkeypatch.setattr(stores_module, "make_store", fake_make_store)
+ ds = mount_datastore(
+ location,
+ at=str(tmp_path / "target.zarr"),
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+ assert is_remote_datastore(None, ds.RNA.rawData._backing) is True
+
+ cell_idx = ds.cells.active_index("I")
+ feat_idx = ds.RNA.feats.active_index("I")
+ blocks = list(
+ ds.RNA.iter_raw_column_blocks(
+ cell_idx,
+ feat_idx,
+ batch_size=1,
+ )
+ )
+ observed = np.concatenate([raw for _, raw, _, _, _ in blocks], axis=1)
+ np.testing.assert_array_equal(observed, values[np.ix_(cell_idx, feat_idx)])
+
+ ds.RNA.set_feature_stats("I")
+ mounted_stats = ds.RNA.get_feature_stats("I", columns=_STATS_COLUMNS)
+ for column in _STATS_COLUMNS:
+ np.testing.assert_allclose(mounted_stats[column], expected_stats[column])
+ assert "summary_stats_I" in zarr.open_group(str(tmp_path / "target.zarr"))["RNA"]
+
+
+def test_workspace_mismatch_raises(tmp_path):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ _write_source_store(source, workspace="analysis")
+ create_matrix_source(source, target, workspace="analysis")
+ with pytest.raises(ValueError, match="workspace does not match"):
+ DataStore(target, workspace="other", default_assay="RNA")
+
+
+def test_mounted_store_normalization_and_graph(tmp_path):
+ source = str(tmp_path / "source.zarr")
+ target = str(tmp_path / "target.zarr")
+ rng = np.random.default_rng(0)
+ values = rng.integers(1, 20, size=(50, 30), dtype=np.uint32)
+ _write_source_store(source, workspace=None, values=values)
+
+ ds = mount_datastore(
+ source,
+ at=target,
+ default_assay="RNA",
+ min_features_per_cell=1,
+ min_cells_per_feature=1,
+ )
+ # Guarantee enough selected features for IncrementalPCA(dims).
+ feature_mask = np.zeros(ds.RNA.feats.N, dtype=bool)
+ feature_mask[:12] = True
+ ds.RNA.feats.insert("I__hvgs", feature_mask, overwrite=True)
+ build_neighbourhood_graph(
+ ds,
+ feat_key="hvgs",
+ k=3,
+ dims=3,
+ batch_size=25,
+ local_cache=False,
+ )
+ state = ds.get_assay_state("RNA")
+ assert state is not None
+ assert state.connectivity_map is not None
+ assert "counts" not in zarr.open_group(target, mode="r")["RNA"]
+
+
+def test_mounted_store_build_mapping_reference(datastore_zarr_root, tmp_path):
+ target = str(tmp_path / "mounted.zarr")
+ source_before = _snapshot_store_files(datastore_zarr_root)
+ ds = mount_datastore(
+ datastore_zarr_root,
+ at=target,
+ default_assay="RNA",
+ )
+ ds.mark_hvgs(top_n=50, show_plot=False, bin_strategy="fixed")
+ build_neighbourhood_graph(
+ ds,
+ feat_key="hvgs",
+ k=3,
+ dims=5,
+ n_centroids=10,
+ )
+ state = ds.get_assay_state("RNA")
+ assert state is not None
+ assert state.neighbors is not None
+ reference = ds.build_mapping_reference(state.neighbors)
+ assert reference.method == "pca"
+ assert reference.dataset_fingerprint
+ assert "counts" not in zarr.open_group(target, mode="r")["RNA"]
+ assert _snapshot_store_files(datastore_zarr_root) == source_before
diff --git a/tests/test_meld_assay.py b/tests/test_meld_assay.py
new file mode 100644
index 00000000..d4d512aa
--- /dev/null
+++ b/tests/test_meld_assay.py
@@ -0,0 +1,640 @@
+import numpy as np
+import pandas as pd
+import pytest
+import zarr
+from scipy.sparse import csr_matrix
+from zarr.storage import MemoryStore
+
+from scarf.datastore.datastore import DataStore
+from scarf.matrix import ChunkedArray
+from scarf.features.genomic.gff import GffReader
+from scarf.features.genomic.intervals import (
+ binary_search,
+ create_bed_from_coord_ids,
+ get_feature_mappings,
+)
+from scarf.features.genomic.melding import create_counts_mat
+from scarf.storage.budget import ResourceBudget
+from scarf.storage.sharding import sparse_matrix_bytes
+from scarf.writers import SparseToZarr, create_zarr_dataset
+
+
+def _features_bed(rows):
+ return pd.DataFrame({i: [r[i] for r in rows] for i in range(6)}).sort_values(
+ by=[0, 1]
+ )
+
+
+class _FakeMeta:
+ def __init__(self, columns):
+ self._columns = columns
+
+ def fetch_all(self, name):
+ return np.asarray(self._columns[name])
+
+
+class _FakeRawData:
+ def __init__(self, blocks):
+ self.blocks = blocks
+ self.resident_bytes: list[int] = []
+ self.numblocks = (len(blocks),)
+ self.shape = (sum(b.shape[0] for b in blocks), blocks[0].shape[1])
+ self.dtype = np.asarray(blocks[0]).dtype
+ self.chunksize = (
+ max(block.shape[0] for block in blocks),
+ self.shape[1],
+ )
+
+ def stream_blocks(self, nthreads=None, msg=None, prefetch=None):
+ # Mirrors ChunkedArray.stream_blocks: yields materialized, in-order blocks.
+ yield from self._stream_blocks(
+ nthreads=nthreads,
+ msg=msg,
+ prefetch=prefetch,
+ row_mask=None,
+ resident_bytes=0,
+ )
+
+ def _stream_blocks(
+ self,
+ *,
+ nthreads=None,
+ msg=None,
+ prefetch=None,
+ row_mask=None,
+ resident_bytes=0,
+ ):
+ assert row_mask is None
+ self.resident_bytes.append(resident_bytes)
+ yield from (np.asarray(block) for block in self.blocks)
+
+ def _max_decode_bytes(self):
+ return 0
+
+ def _with_block_size(self, block_size):
+ values = np.vstack(self.blocks)
+ resized = _FakeRawData(
+ [
+ values[start : start + block_size]
+ for start in range(0, len(values), block_size)
+ ]
+ )
+ resized.resident_bytes = self.resident_bytes
+ return resized
+
+
+class _FakeAssay:
+ name = "ATAC"
+ nthreads = 1
+ resources = ResourceBudget(1024**3, 1)
+
+ def __init__(self, cells, feats, raw_data):
+ self.cells = cells
+ self.feats = feats
+ self.rawData = raw_data
+
+
+def _reference_melded(
+ raw,
+ n_counts_per_cell,
+ n_cells_per_peak,
+ mapping_dense,
+ scalar_coeff,
+ renorm,
+ idf_cell_idx=None,
+):
+ if idf_cell_idx is None:
+ n_docs = raw.shape[0]
+ document_frequency = n_cells_per_peak
+ else:
+ n_docs = len(idf_cell_idx)
+ document_frequency = np.count_nonzero(raw[idf_cell_idx], axis=0)
+ idf = np.log2(1 + (n_docs / (document_frequency + 1)))
+ tf = raw / n_counts_per_cell.reshape(-1, 1)
+ tfidf = tf * idf
+ melded = tfidf @ mapping_dense
+ if not renorm:
+ return melded
+ row_sums = melded.sum(axis=1)
+ out = np.zeros_like(melded)
+ nz = row_sums != 0
+ out[nz] = (scalar_coeff * melded[nz]) / row_sums[nz].reshape(-1, 1)
+ return out
+
+
+def _old_style_melded(
+ raw,
+ n_counts_per_cell,
+ n_cells_per_peak,
+ feature_to_peaks,
+ scalar_coeff,
+ renorm,
+):
+ n_docs = raw.shape[0]
+ idf = np.log2(1 + (n_docs / (n_cells_per_peak + 1)))
+ tf = raw / n_counts_per_cell.reshape(-1, 1)
+ tfidf = tf * idf
+
+ idx = np.array([i for i, peaks in enumerate(feature_to_peaks) if len(peaks)])
+ if idx.size == 0:
+ melded = np.zeros((raw.shape[0], len(feature_to_peaks)))
+ else:
+ feat_idx = np.repeat(idx, [len(feature_to_peaks[i]) for i in idx])
+ peak_idx = np.array(sum((feature_to_peaks[i] for i in idx), []))
+ df = pd.DataFrame(tfidf[:, peak_idx]).T
+ df["fidx"] = feat_idx
+ df = df.groupby("fidx").sum().T
+ melded = np.zeros((raw.shape[0], len(feature_to_peaks)))
+ melded[:, df.columns.to_numpy(dtype=int)] = df.values
+
+ if not renorm:
+ return melded
+ row_sums = melded.sum(axis=1)
+ out = np.zeros_like(melded)
+ nz = row_sums != 0
+ out[nz] = (scalar_coeff * melded[nz]) / row_sums[nz].reshape(-1, 1)
+ return out
+
+
+def _feature_to_peaks_from_intervals(peaks, features):
+ feature_to_peaks = []
+ for _, feat in features.iterrows():
+ peak_hits = []
+ for peak_pos, peak in peaks.iterrows():
+ if feat[0] == peak[0] and feat[1] < peak[2] and feat[2] > peak[1]:
+ peak_hits.append(int(peak_pos))
+ feature_to_peaks.append(peak_hits)
+ return feature_to_peaks
+
+
+def _build_melding_scenario():
+ peaks = create_bed_from_coord_ids(
+ ["chr1:100-200", "chr1:150-250", "chr1:400-500", "chr2:100-200"]
+ )
+ features = _features_bed(
+ [
+ ("chr1", 120, 160, "a", "A", "+"), # overlaps peaks 0, 1
+ ("chr1", 300, 350, "b", "B", "+"), # overlaps nothing
+ ("chr2", 150, 180, "c", "C", "+"), # overlaps peak 3
+ ]
+ )
+ _, _, mapping = get_feature_mappings(peaks, features)
+
+ # Rows are cells, columns are peaks (assay feature order).
+ raw = np.array(
+ [
+ [1.0, 0.0, 2.0, 3.0],
+ [0.0, 4.0, 0.0, 0.0],
+ [
+ 0.0,
+ 0.0,
+ 5.0,
+ 0.0,
+ ], # only peak 2, which maps to no feature -> melded row is all zero
+ ]
+ )
+ n_counts_per_cell = raw.sum(axis=1)
+ n_cells_per_peak = np.array([1.0, 1.0, 2.0, 1.0])
+
+ cells = _FakeMeta({"ATAC_nCounts": n_counts_per_cell})
+ feats = _FakeMeta({"nCells": n_cells_per_peak})
+ raw_data = _FakeRawData([raw[0:2], raw[2:3]])
+ assay = _FakeAssay(cells, feats, raw_data)
+ return assay, mapping, raw, n_counts_per_cell, n_cells_per_peak
+
+
+def _run_create_counts_mat(
+ assay,
+ mapping,
+ scalar_coeff,
+ renormalization,
+ idf_cell_idx=None,
+):
+ n_features = mapping.shape[1]
+ n_cells = assay.rawData.shape[0]
+ group = zarr.open_group(store=MemoryStore(), mode="w")
+ store = create_zarr_dataset(
+ group, "counts", (2, n_features), "float", (n_cells, n_features)
+ )
+ create_counts_mat(
+ assay=assay,
+ store=store,
+ mapping=mapping,
+ scalar_coeff=scalar_coeff,
+ renormalization=renormalization,
+ idf_cell_idx=idf_cell_idx,
+ )
+ return store[:]
+
+
+GFF_CONTENT = """##gff-version 3
+# test annotation
+chr1\t.\tgene\t1000\t2000\t.\t+\t.\tgene_id=gene_a;gene_name=GENE_A
+chr1\t.\tgene\t3000\t4500\t.\t-\t.\tgene_id=gene_b;gene_name=GENE_B
+chr2\t.\tgene\t500\t1500\t.\t+\t.\tgene_id=gene_c;gene_name=GENE_C
+"""
+
+
+def test_gff_reader_parses_header_and_streams(tmp_path):
+ gff_path = tmp_path / "test.gff"
+ gff_path.write_text(GFF_CONTENT)
+
+ reader = GffReader(str(gff_path), up_offset=500, down_offset=200, chunk_size=2)
+ assert len(reader.header) == 2
+ chunks = list(reader.stream())
+ assert len(chunks) == 2
+ assert chunks[0].shape[0] == 2
+ assert set(chunks[0][2]) == {"gene"}
+
+
+def test_gff_reader_promoter_and_body_coordinates(tmp_path):
+ gff_path = tmp_path / "coords.gff"
+ gff_path.write_text(GFF_CONTENT)
+ reader = GffReader(str(gff_path), up_offset=500, down_offset=200)
+ plus_row = pd.Series([None] * 9)
+ plus_row[3] = 1000
+ plus_row[4] = 2000
+ plus_row[6] = "+"
+
+ promoter_start, promoter_end = reader.get_promoter(plus_row)
+ assert promoter_start == 500
+ assert promoter_end == 1200
+
+ body_start, body_end = reader.get_body(plus_row)
+ assert body_start == 500
+ assert body_end == 2000
+
+ minus_row = plus_row.copy()
+ minus_row[3] = 3000
+ minus_row[4] = 4500
+ minus_row[6] = "-"
+ m_start, m_end = reader.get_promoter(minus_row)
+ assert m_start == 4499 - reader.down
+ assert m_end == 4500 + reader.up
+
+
+def test_gff_reader_get_ids_names():
+ row = pd.Series([None] * 9)
+ row[8] = "gene_id=abc;gene_name=XYZ"
+ gene_id, gene_name = GffReader.get_ids_names(row)
+ assert gene_id == "abc"
+ assert gene_name == "XYZ"
+
+
+def test_gff_reader_to_bed_promoter(tmp_path):
+ gff_path = tmp_path / "genes.gff"
+ gff_path.write_text(GFF_CONTENT)
+ bed_path = tmp_path / "out.bed"
+
+ reader = GffReader(str(gff_path))
+ reader.to_bed(str(bed_path), flavour="promoter")
+
+ bed = pd.read_csv(bed_path, sep="\t", header=None)
+ assert bed.shape[0] == 3
+ assert bed.shape[1] == 6
+
+
+def test_gff_reader_to_bed_rejects_unknown_flavour(tmp_path):
+ gff_path = tmp_path / "genes.gff"
+ gff_path.write_text(GFF_CONTENT)
+ reader = GffReader(str(gff_path))
+ with pytest.raises(ValueError, match="flavour"):
+ reader.to_bed(str(tmp_path / "out.bed"), flavour="exon")
+
+
+def test_create_bed_from_coord_ids_sorts_intervals():
+ bed = create_bed_from_coord_ids(["chr2:100-200", "chr1:50-150", "chr1:300-400"])
+ assert bed.iloc[0, 0] == "chr1"
+ assert bed.iloc[0, 1] == 50
+ assert bed.iloc[-1, 0] == "chr2"
+
+
+def test_binary_search_finds_overlapping_ranges():
+ ranges = np.array([[0, 10], [10, 20], [20, 30], [30, 40]], dtype=np.int64)
+ queries = np.array([[5, 8], [15, 18], [25, 28]], dtype=np.int64)
+ hits = binary_search(ranges, queries)
+ assert hits.shape == (3, 2)
+ assert hits[0, 0] <= hits[0, 1]
+
+
+def test_get_feature_mappings_builds_sparse_overlap_matrix():
+ peaks = create_bed_from_coord_ids(
+ ["chr1:100-200", "chr1:150-250", "chr1:400-500", "chr2:100-200"]
+ )
+ features = _features_bed(
+ [
+ ("chr1", 120, 160, "a", "A", "+"),
+ ("chr1", 300, 350, "b", "B", "+"),
+ ("chr2", 150, 180, "c", "C", "+"),
+ ]
+ )
+ feat_ids, feat_names, mapping = get_feature_mappings(peaks, features)
+
+ assert list(feat_ids) == ["a", "b", "c"]
+ assert list(feat_names) == ["A", "B", "C"]
+ assert mapping.shape == (4, 3)
+
+ expected = np.zeros((4, 3))
+ expected[[0, 1], 0] = 1 # feature A overlaps peaks at original positions 0 and 1
+ expected[3, 2] = 1 # feature C overlaps peak at original position 3
+ np.testing.assert_array_equal(mapping.toarray(), expected)
+
+
+def test_get_feature_mappings_keeps_features_without_peaks():
+ peaks = create_bed_from_coord_ids(["chr1:100-200"])
+ features = _features_bed(
+ [
+ ("chr1", 120, 160, "a", "A", "+"),
+ ("chr3", 100, 200, "d", "D", "+"),
+ ]
+ )
+ feat_ids, _, mapping = get_feature_mappings(peaks, features)
+
+ # Feature on peak-less chromosome must be retained (not silently dropped)
+ assert list(feat_ids) == ["a", "d"]
+ assert mapping.shape[1] == 2
+ assert mapping.getcol(1).nnz == 0
+
+
+def test_get_feature_mappings_rejects_zero_overlap():
+ peaks = create_bed_from_coord_ids(["chr1:100-200"])
+ features = _features_bed(
+ [
+ ("chr1", 300, 400, "a", "A", "+"),
+ ("chr2", 100, 200, "b", "B", "+"),
+ ]
+ )
+
+ with pytest.raises(ValueError, match="None of the provided features overlap"):
+ get_feature_mappings(peaks, features)
+
+
+def test_get_feature_mappings_uniquifies_duplicate_ids():
+ peaks = create_bed_from_coord_ids(["chr1:100-200"])
+ features = _features_bed(
+ [
+ ("chr1", 120, 160, "dup", "A", "+"),
+ ("chr1", 500, 600, "dup", "B", "+"),
+ ]
+ )
+ feat_ids, _, _ = get_feature_mappings(peaks, features)
+ assert list(feat_ids) == ["dup", "dup_2"]
+
+
+def test_get_feature_mappings_follows_feature_bed_order():
+ peaks = create_bed_from_coord_ids(["chr1:100-200", "chr2:100-200"])
+ features = _features_bed(
+ [
+ ("chr2", 120, 160, "b", "B", "+"),
+ ("chr3", 100, 200, "c", "C", "+"),
+ ("chr1", 120, 160, "a", "A", "+"),
+ ]
+ )
+ feat_ids, _, mapping = get_feature_mappings(peaks, features)
+
+ assert list(feat_ids) == ["a", "b", "c"]
+ assert mapping.getcol(2).nnz == 0
+
+
+def test_mapping_matmul_sums_overlapping_peaks():
+ peaks = create_bed_from_coord_ids(
+ ["chr1:100-200", "chr1:150-250", "chr1:400-500", "chr2:100-200"]
+ )
+ features = _features_bed(
+ [
+ ("chr1", 120, 160, "a", "A", "+"),
+ ("chr1", 300, 350, "b", "B", "+"),
+ ("chr2", 150, 180, "c", "C", "+"),
+ ]
+ )
+ _, _, mapping = get_feature_mappings(peaks, features)
+
+ tfidf = np.array(
+ [
+ [1.0, 2.0, 3.0, 4.0],
+ [0.5, 0.0, 0.0, 1.5],
+ ]
+ )
+ melded = tfidf @ mapping.toarray()
+
+ # Feature A melds peaks 0 and 1, feature B nothing, feature C peak 3
+ np.testing.assert_allclose(melded[:, 0], tfidf[:, 0] + tfidf[:, 1])
+ np.testing.assert_allclose(melded[:, 1], np.zeros(2))
+ np.testing.assert_allclose(melded[:, 2], tfidf[:, 3])
+
+
+def test_create_counts_mat_matches_old_groupby_oracle():
+ peaks = create_bed_from_coord_ids(
+ ["chr1:100-200", "chr1:150-250", "chr1:210-260", "chr2:50-120"]
+ )
+ features = _features_bed(
+ [
+ ("chr1", 120, 240, "a", "A", "+"),
+ ("chr1", 500, 600, "b", "B", "+"),
+ ("chr2", 70, 100, "c", "C", "+"),
+ ]
+ )
+ _, _, mapping = get_feature_mappings(peaks, features)
+ raw = np.array(
+ [
+ [3.0, 0.0, 1.0, 0.0],
+ [0.0, 2.0, 0.0, 5.0],
+ [4.0, 1.0, 0.0, 0.0],
+ [0.0, 0.0, 7.0, 3.0],
+ ]
+ )
+ n_counts = raw.sum(axis=1)
+ n_cells_peak = np.array([2.0, 2.0, 2.0, 2.0])
+ assay = _FakeAssay(
+ _FakeMeta({"ATAC_nCounts": n_counts}),
+ _FakeMeta({"nCells": n_cells_peak}),
+ _FakeRawData([raw[:2], raw[2:]]),
+ )
+ old_oracle = _old_style_melded(
+ raw,
+ n_counts,
+ n_cells_peak,
+ _feature_to_peaks_from_intervals(peaks, features),
+ scalar_coeff=1e4,
+ renorm=True,
+ )
+ written = _run_create_counts_mat(
+ assay, mapping, scalar_coeff=1e4, renormalization=True
+ )
+ np.testing.assert_allclose(written, old_oracle)
+
+
+def test_create_counts_mat_reads_real_chunked_array_stream_blocks():
+ assay, mapping, raw, n_counts, n_cells_peak = _build_melding_scenario()
+ group = zarr.open_group(store=MemoryStore(), mode="w")
+ counts = create_zarr_dataset(group, "raw", (2, raw.shape[1]), "float", raw.shape)
+ counts[:] = raw
+ assay.rawData = ChunkedArray(counts, nthreads=2)
+
+ written = _run_create_counts_mat(
+ assay, mapping, scalar_coeff=1e4, renormalization=True
+ )
+ expected = _reference_melded(
+ raw,
+ n_counts,
+ n_cells_peak,
+ mapping.toarray(),
+ scalar_coeff=1e4,
+ renorm=True,
+ )
+ np.testing.assert_allclose(written, expected)
+
+
+def test_create_counts_mat_without_renormalization():
+ assay, mapping, raw, n_counts, n_cells_peak = _build_melding_scenario()
+ written = _run_create_counts_mat(
+ assay, mapping, scalar_coeff=1e4, renormalization=False
+ )
+ expected = _reference_melded(
+ raw,
+ n_counts,
+ n_cells_peak,
+ mapping.toarray(),
+ scalar_coeff=1e4,
+ renorm=False,
+ )
+ np.testing.assert_allclose(written, expected)
+
+
+def test_create_counts_mat_learns_idf_from_selected_cells_and_scores_all_rows():
+ assay, mapping, raw, n_counts, n_cells_peak = _build_melding_scenario()
+ idf_cell_idx = np.array([1, 2], dtype=np.int64)
+
+ written = _run_create_counts_mat(
+ assay,
+ mapping,
+ scalar_coeff=1e4,
+ renormalization=False,
+ idf_cell_idx=idf_cell_idx,
+ )
+ expected = _reference_melded(
+ raw,
+ n_counts,
+ n_cells_peak,
+ mapping.toarray(),
+ scalar_coeff=1e4,
+ renorm=False,
+ idf_cell_idx=idf_cell_idx,
+ )
+ full_corpus = _reference_melded(
+ raw,
+ n_counts,
+ n_cells_peak,
+ mapping.toarray(),
+ scalar_coeff=1e4,
+ renorm=False,
+ )
+
+ np.testing.assert_allclose(written, expected)
+ assert not np.allclose(written, full_corpus)
+ assert written.shape[0] == raw.shape[0]
+ assert written[0].sum() > 0
+ minimum_df_resident = (
+ sparse_matrix_bytes(mapping)
+ + idf_cell_idx.nbytes
+ + raw.shape[0] * np.dtype(bool).itemsize
+ + n_counts.nbytes
+ + 2 * raw.shape[1] * np.dtype(np.int64).itemsize
+ + assay.rawData.chunksize[0] * raw.shape[1] * raw.dtype.itemsize
+ )
+ assert len(assay.rawData.resident_bytes) == 2
+ assert assay.rawData.resident_bytes[0] >= minimum_df_resident
+ assert assay.rawData.resident_bytes[1] > 0
+
+
+def test_add_melded_assay_uses_cell_key_for_idf_and_keeps_all_rows(tmp_path):
+ raw = np.array(
+ [
+ [2, 0, 1],
+ [0, 3, 0],
+ [1, 0, 4],
+ ],
+ dtype=np.uint32,
+ )
+ peak_ids = ["chr1:100-200", "chr1:250-350", "chr2:100-200"]
+ store_path = tmp_path / "atac_melding.zarr"
+ SparseToZarr(
+ csr_matrix(raw),
+ zarr_loc=str(store_path),
+ cell_ids=["cell_0", "cell_1", "cell_2"],
+ feature_ids=peak_ids,
+ assay_name="ATAC",
+ nthreads=1,
+ ).dump(batch_size=2)
+ store = DataStore(
+ str(store_path),
+ default_assay="ATAC",
+ min_features_per_cell=0,
+ min_cells_per_feature=0,
+ nthreads=1,
+ )
+ train_mask = np.array([False, True, True])
+ store.cells.insert("train", train_mask, overwrite=True)
+ feature_bed = _features_bed(
+ [
+ ("chr1", 120, 300, "gene_a", "GENE_A", "+"),
+ ("chr2", 120, 180, "gene_b", "GENE_B", "+"),
+ ]
+ )
+ bed_path = tmp_path / "genes.bed"
+ feature_bed.to_csv(bed_path, sep="\t", header=False, index=False)
+
+ store.add_melded_assay(
+ from_assay="ATAC",
+ external_bed_fn=str(bed_path),
+ assay_label="GeneScores",
+ renormalization=False,
+ cell_key="train",
+ )
+
+ _, _, mapping = get_feature_mappings(
+ create_bed_from_coord_ids(peak_ids),
+ feature_bed,
+ )
+ expected = _reference_melded(
+ raw,
+ raw.sum(axis=1, dtype=np.float64),
+ np.count_nonzero(raw, axis=0),
+ mapping.toarray(),
+ scalar_coeff=1e5,
+ renorm=False,
+ idf_cell_idx=np.flatnonzero(train_mask),
+ )
+ observed = np.vstack(
+ list(store.GeneScores.rawData.stream_blocks(nthreads=1, msg=None))
+ )
+
+ np.testing.assert_allclose(observed, expected, rtol=1e-6, atol=1e-7)
+ assert observed.shape[0] == raw.shape[0]
+ assert observed[0].sum() > 0
+ assert store.GeneScores.z.attrs["idfCellCount"] == int(train_mask.sum())
+ assert store.GeneScores.z.attrs["sourceAssay"] == "ATAC"
+ assert store.GeneScores.z.attrs["tfDenominator"] == "total_counts"
+
+
+def test_create_counts_mat_with_renormalization_and_zero_sum_cell():
+ assay, mapping, raw, n_counts, n_cells_peak = _build_melding_scenario()
+ written = _run_create_counts_mat(
+ assay, mapping, scalar_coeff=1e4, renormalization=True
+ )
+ expected = _reference_melded(
+ raw,
+ n_counts,
+ n_cells_peak,
+ mapping.toarray(),
+ scalar_coeff=1e4,
+ renorm=True,
+ )
+ np.testing.assert_allclose(written, expected)
+
+ # The cell whose only signal maps to no feature must be all zeros, not NaN.
+ assert np.all(np.isfinite(written))
+ np.testing.assert_array_equal(written[2], np.zeros(mapping.shape[1]))
+ # Non-empty cells are rescaled to sum to scalar_coeff.
+ np.testing.assert_allclose(written[0].sum(), 1e4)
+ np.testing.assert_allclose(written[1].sum(), 1e4)
diff --git a/tests/test_merge_contracts.py b/tests/test_merge_contracts.py
new file mode 100644
index 00000000..3e2b0799
--- /dev/null
+++ b/tests/test_merge_contracts.py
@@ -0,0 +1,231 @@
+import inspect
+import pickle
+import subprocess
+import sys
+from types import SimpleNamespace
+from typing import get_type_hints
+
+import numpy as np
+import scarf
+import scarf.merge as merge_module
+from scarf.merge import AssayMerge, DatasetMerge, DummyAssay
+from tests.signature_contracts import signature_digest
+
+
+_PUBLIC_CLASS_METHODS = {
+ AssayMerge: (
+ "__init__",
+ "perform_randomization_rows",
+ "check_feat_ids",
+ "get_feat_suffix",
+ "update_feat_ids",
+ "update_feat_ids_for_map",
+ "dump",
+ ),
+ DatasetMerge: (
+ "__init__",
+ "get_unique_assays",
+ "create_merge_generators",
+ "generate_dummy_assay",
+ "dump",
+ ),
+}
+_PUBLIC_CLASS_SIGNATURE_DIGESTS = {
+ AssayMerge: "28945aa6ef349e023f76969f30fd5b97fd95bb762b6b005106c3cd1ee0d7ac21",
+ DatasetMerge: "c556a12844cd893c19ae605443bce2f384fa1d9277dfe157a3b5e1ee456d7189",
+}
+_FACADE_METHODS = {DummyAssay: ("__init__",), **_PUBLIC_CLASS_METHODS}
+
+
+def test_merge_facade_surface_is_stable():
+ assert merge_module.__all__ == [
+ "DatasetMerge",
+ "AssayMerge",
+ ]
+ expected = {
+ "AssayMerge",
+ "DatasetMerge",
+ "DummyAssay",
+ "MergeAssay",
+ "_RowPlan",
+ "controlled_compute",
+ "load_zarr",
+ }
+ assert expected.issubset(dir(merge_module))
+ assert all(getattr(merge_module, name) is not None for name in expected)
+ assert not hasattr(merge_module, "ZarrMerge")
+
+
+def test_merge_class_and_method_signatures_are_stable():
+ for cls, names in _PUBLIC_CLASS_METHODS.items():
+ methods = {name: getattr(cls, name) for name in names}
+ assert signature_digest(methods) == _PUBLIC_CLASS_SIGNATURE_DIGESTS[cls]
+
+
+def test_merge_class_and_method_metadata_remains_on_facade():
+ for cls, names in _FACADE_METHODS.items():
+ assert cls.__module__ == "scarf.merge"
+ for name in names:
+ descriptor = inspect.getattr_static(cls, name)
+ if isinstance(descriptor, staticmethod):
+ method = descriptor.__func__
+ else:
+ method = descriptor
+ assert method.__module__ == "scarf.merge"
+ assert method.__qualname__.startswith(f"{cls.__name__}.")
+
+
+def test_merge_descriptor_and_hierarchy_contracts_are_stable():
+ assert isinstance(
+ inspect.getattr_static(AssayMerge, "_get_feat_ids"),
+ staticmethod,
+ )
+
+
+def test_merge_facade_classes_remain_pickle_resolvable():
+ for cls in _FACADE_METHODS:
+ assert pickle.loads(pickle.dumps(cls)) is cls
+
+
+def test_merge_datastore_type_hints_resolve_lazily():
+ assert get_type_hints(DummyAssay.__init__)["ds"] is scarf.DataStore
+ assert get_type_hints(DatasetMerge.__init__)["datasets"] == list[scarf.DataStore]
+ assert get_type_hints(DatasetMerge.generate_dummy_assay)["ds"] is scarf.DataStore
+
+
+def test_dataset_merge_resolves_assay_factory_from_facade(monkeypatch):
+ assay = SimpleNamespace(rawData=SimpleNamespace(chunksize=(2, 2)))
+ dataset = SimpleNamespace(
+ assay_names=["RNA"],
+ get_assay=lambda name: assay,
+ )
+ generator = SimpleNamespace(
+ permutations_rows={0: {0: np.array([0])}},
+ permutations_rows_offset={0: {0: np.array([0])}},
+ coordinates_permutations=np.array([[0, 0]]),
+ )
+ captured = {}
+
+ def factory(**kwargs):
+ captured.update(kwargs)
+ return generator
+
+ monkeypatch.setattr(merge_module, "AssayMerge", factory)
+ merge = object.__new__(DatasetMerge)
+ merge.datasets = [dataset]
+ merge.names = ["sample"]
+ merge.zarr_path = "merged.zarr"
+ merge.in_workspaces = None
+ merge.out_workspace = None
+ merge.dtype = None
+ merge.overwrite = False
+ merge.prepend_text = "orig"
+ merge.reset_cell_filter = True
+ merge.seed = 42
+ merge.storage_options = None
+ merge.source_column = None
+ merge.memBudget = 1024**3
+ merge.nthreads = 2
+ merge.profile = "fast_local"
+ merge.targetChunkBytes = None
+ merge.targetShardBytes = None
+ merge.unique_assays = ["RNA"]
+
+ assert merge.create_merge_generators() == [generator]
+ assert captured["assays"] == [assay]
+ assert captured["merge_assay_name"] == "RNA"
+
+
+def test_dataset_merge_resolves_dummy_factory_from_facade(monkeypatch):
+ reference_assay = SimpleNamespace(
+ rawData=SimpleNamespace(chunksize=(2, 2), dtype=np.dtype("uint16")),
+ feats=SimpleNamespace(N=2),
+ name="RNA",
+ )
+ dataset = SimpleNamespace(
+ assay_names=["RNA"],
+ cells=SimpleNamespace(N=3),
+ get_assay=lambda name: reference_assay,
+ nthreads=2,
+ resources=None,
+ )
+ sentinel = object()
+ captured = {}
+
+ def factory(*args):
+ captured["args"] = args
+ return sentinel
+
+ monkeypatch.setattr(merge_module, "DummyAssay", factory)
+ merge = object.__new__(DatasetMerge)
+ merge.datasets = [dataset]
+
+ assert merge.generate_dummy_assay(dataset, "RNA") is sentinel
+ assert captured["args"][0] is dataset
+ assert captured["args"][2] is reference_assay.feats
+ assert captured["args"][3] == "RNA"
+
+
+def test_merge_facade_loads_implementations_lazily_and_clears_on_reload():
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ """
+import importlib
+import sys
+
+import scarf.merge as merge
+
+assert "scarf.merge.assays" not in sys.modules
+assert "scarf.merge.datasets" not in sys.modules
+assert "scarf.datastore.datastore" not in sys.modules
+assert not any(
+ name == "scarf.assay" or name.startswith("scarf.assay.")
+ for name in sys.modules
+)
+assert not {"pandas", "scipy", "zarr"}.intersection(sys.modules)
+
+assay_merge = merge.AssayMerge
+assert "scarf.merge.assays" in sys.modules
+assert "scarf.merge.datasets" not in sys.modules
+assert "scarf.datastore.datastore" not in sys.modules
+assert {
+ "AssayMerge",
+ "DummyAssay",
+ "MergeAssay",
+ "_RowPlan",
+ "controlled_compute",
+ "load_zarr",
+} <= vars(merge).keys()
+assert "ZarrMerge" not in vars(merge)
+
+dataset_merge = merge.DatasetMerge
+assert "scarf.merge.datasets" in sys.modules
+assert "scarf.datastore.datastore" not in sys.modules
+for cls in (assay_merge, dataset_merge, merge.DummyAssay):
+ assert cls.__module__ == "scarf.merge"
+ for descriptor in cls.__dict__.values():
+ if isinstance(descriptor, (classmethod, staticmethod)):
+ method = descriptor.__func__
+ else:
+ method = descriptor
+ if callable(method) and hasattr(method, "__module__"):
+ assert method.__module__ == "scarf.merge"
+
+merge.AssayMerge = object()
+importlib.reload(merge)
+assert not {
+ "AssayMerge",
+ "DatasetMerge",
+ "DummyAssay",
+ "MergeAssay",
+ "_RowPlan",
+ "controlled_compute",
+ "load_zarr",
+}.intersection(vars(merge))
+assert merge.AssayMerge is assay_merge
+""",
+ ],
+ check=True,
+ )
diff --git a/tests/test_merger.py b/tests/test_merger.py
new file mode 100644
index 00000000..ba47f534
--- /dev/null
+++ b/tests/test_merger.py
@@ -0,0 +1,741 @@
+import numpy as np
+import pandas as pd
+import pytest
+import sys
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.matrix import ChunkedArray
+from scarf.merge import AssayMerge
+from scarf.storage.budget import ResourceBudget
+from scarf.storage.layout import count_array_spec
+
+
+class _MergeMeta:
+ def __init__(self, **columns):
+ self._columns = {key: np.asarray(value) for key, value in columns.items()}
+ self.columns = list(self._columns)
+ self.N = len(next(iter(self._columns.values())))
+
+ def to_pandas_dataframe(self, columns):
+ return pd.DataFrame({key: self._columns[key] for key in columns})
+
+ def fetch_all(self, key):
+ return self._columns[key]
+
+
+class _MergeAssay:
+ def __init__(
+ self,
+ name,
+ counts,
+ cell_ids,
+ feature_ids,
+ feature_names,
+ block_size,
+ ):
+ self.name = name
+ self.rawData = ChunkedArray.from_numpy(
+ np.asarray(counts),
+ block_size=block_size,
+ )
+ self.cells = _MergeMeta(
+ ids=cell_ids,
+ names=cell_ids,
+ I=np.ones(len(cell_ids), dtype=bool),
+ )
+ self.feats = _MergeMeta(ids=feature_ids, names=feature_names)
+
+
+class _MergeDataStore:
+ def __init__(self, assays):
+ self._assays = {assay.name: assay for assay in assays}
+ self.assay_names = list(self._assays)
+ self.cells = assays[0].cells
+ self.nthreads = 2
+ self.memoryBytes = 1024**3
+ self.resources = ResourceBudget(self.memoryBytes, self.nthreads)
+
+ def get_assay(self, name):
+ return self._assays[name]
+
+
+def test_assay_merge_rejects_summary_before_truncating_destination():
+ store = MemoryStore()
+ root = zarr.open_group(store=store, mode="w")
+ root.create_group("sentinel")
+
+ with pytest.raises(ValueError, match=r"reserved for DataStore\.summary"):
+ AssayMerge(
+ zarr_path=store,
+ assays=[],
+ names=[],
+ merge_assay_name="summary",
+ )
+
+ preserved = zarr.open_group(store=store, mode="r")
+ assert set(preserved.group_keys()) == {"sentinel"}
+
+
+def test_assay_merge(datastore, rna_raw_total, tmp_path):
+ fn = str(tmp_path / "merged.zarr")
+ writer = AssayMerge(
+ zarr_path=fn,
+ assays=[datastore.RNA, datastore.RNA],
+ names=["self1", "self2"],
+ merge_assay_name="RNA",
+ prepend_text="",
+ overwrite=True,
+ )
+ writer.dump()
+ tmp = zarr.open(fn + "/RNA/counts")
+ assert tmp.shape[0] == 2 * datastore.cells.N
+ assert int(tmp[...].sum()) == rna_raw_total * 2
+
+
+def test_assay_merge_maps_features_and_preserves_row_order(
+ tmp_path,
+):
+ cell_ids = ["c0", "c1", "c2"]
+ left = _MergeAssay(
+ "RNA",
+ [[1, 10], [2, 20], [0, 0]],
+ cell_ids,
+ ["id_a", "id_b"],
+ ["A", "B"],
+ block_size=1,
+ )
+ right = _MergeAssay(
+ "RNA",
+ [[30, 300], [40, 400], [50, 500]],
+ cell_ids,
+ ["id_b", "id_c"],
+ ["B", "C"],
+ block_size=3,
+ )
+ fn = str(tmp_path / "feature_order.zarr")
+ writer = AssayMerge(
+ zarr_path=fn,
+ assays=[left, right],
+ names=["left", "right"],
+ merge_assay_name="RNA",
+ prepend_text="",
+ seed=3,
+ )
+ writer.dump()
+
+ root = zarr.open_group(fn, mode="r")
+ counts_array = root["RNA/counts"]
+ counts = np.asarray(counts_array[:])
+ assert "countsT" not in root["RNA"]
+ merged_cell_ids = np.asarray(root["cellData/ids"][:]).astype(str)
+ merged_feature_ids = np.asarray(root["RNA/featureData/ids"][:]).astype(str)
+
+ np.testing.assert_array_equal(merged_feature_ids, ["id_a", "id_b", "id_c"])
+ expected = {
+ "left__c0": [1, 10, 0],
+ "left__c1": [2, 20, 0],
+ "left__c2": [0, 0, 0],
+ "right__c0": [0, 30, 300],
+ "right__c1": [0, 40, 400],
+ "right__c2": [0, 50, 500],
+ }
+ for cell_id, row in zip(merged_cell_ids, counts, strict=True):
+ np.testing.assert_array_equal(row, expected[cell_id])
+
+ spec = count_array_spec(
+ *counts_array.shape,
+ dtype=counts_array.dtype,
+ profile="fast_local",
+ )
+ assert spec.shards is not None
+ assert counts_array.chunks == spec.chunks
+ assert counts_array.metadata.shards == spec.shards
+ assert counts_array.fill_value == 0
+
+
+@pytest.mark.parametrize("dtype", [None, "uint16"])
+def test_assay_merge_widens_before_consolidating_features(tmp_path, dtype):
+ feature_ids = ["gene_0", "gene_1"]
+ left = _MergeAssay(
+ "RNA",
+ np.array([[200, 100]], dtype=np.uint8),
+ ["left"],
+ feature_ids,
+ feature_ids,
+ block_size=1,
+ )
+ right = _MergeAssay(
+ "RNA",
+ np.array([[150, 150]], dtype=np.uint8),
+ ["right"],
+ feature_ids,
+ feature_ids,
+ block_size=1,
+ )
+ path = str(tmp_path / "consolidated_features.zarr")
+
+ AssayMerge(
+ zarr_path=path,
+ assays=[left, right],
+ names=["left", "right"],
+ merge_assay_name="RNA",
+ prepend_text="",
+ dtype=dtype,
+ ).dump()
+
+ counts = zarr.open_group(path, mode="r")["RNA/counts"]
+ assert counts.dtype == np.dtype("uint16")
+ np.testing.assert_array_equal(counts[:], [[300], [300]])
+
+
+def test_assay_merge_rejects_metadata_over_memory_budget(tmp_path):
+ cell_ids = [f"cell-{index}-{'x' * 1000}" for index in range(100)]
+ left = _MergeAssay(
+ "RNA",
+ np.ones((100, 1), dtype=np.uint8),
+ cell_ids,
+ ["gene-id"],
+ ["gene"],
+ block_size=25,
+ )
+ right = _MergeAssay(
+ "RNA",
+ np.ones((100, 1), dtype=np.uint8),
+ cell_ids,
+ ["gene-id"],
+ ["gene"],
+ block_size=25,
+ )
+
+ with pytest.raises(MemoryError, match="operation limit"):
+ AssayMerge(
+ zarr_path=str(tmp_path / "metadata_budget.zarr"),
+ assays=[left, right],
+ names=["left", "right"],
+ merge_assay_name="RNA",
+ prepend_text="",
+ mem_budget="64K",
+ )
+
+
+def test_assay_merge_metadata_bytes_deduplicate_aliased_feature_maps():
+ merge = object.__new__(AssayMerge)
+ merge.mergedCells = pd.DataFrame()
+ merge.mergedFeats = pd.DataFrame()
+ merge.mergedFeats_map = pd.DataFrame()
+ mapping = {"gene-id": "gene"}
+ merge.featCollection = [mapping]
+ merge.featCollection_map = [mapping]
+
+ frame_bytes = sum(
+ int(frame.memory_usage(index=True, deep=True).sum())
+ for frame in (
+ merge.mergedCells,
+ merge.mergedFeats,
+ merge.mergedFeats_map,
+ )
+ )
+ expected = (
+ frame_bytes
+ + sys.getsizeof(merge.featCollection)
+ + sys.getsizeof(merge.featCollection_map)
+ + sys.getsizeof(mapping)
+ + sys.getsizeof("gene-id")
+ + sys.getsizeof("gene")
+ )
+ assert merge._metadata_resident_bytes() == expected
+
+
+def test_assay_merge_keeps_source_metadata_aligned_after_permutation(
+ tmp_path,
+):
+ left = _MergeAssay(
+ "RNA",
+ [[1], [2], [3]],
+ ["l0", "l1", "l2"],
+ ["gene"],
+ ["Gene"],
+ block_size=1,
+ )
+ right = _MergeAssay(
+ "RNA",
+ [[4], [5]],
+ ["r0", "r1"],
+ ["gene"],
+ ["Gene"],
+ block_size=1,
+ )
+ left.cells = _MergeMeta(
+ ids=["l0", "l1", "l2"],
+ names=["l0", "l1", "l2"],
+ I=[True, False, True],
+ cluster_labels=[10, 11, 12],
+ )
+ right.cells = _MergeMeta(
+ ids=["r0", "r1"],
+ names=["r0", "r1"],
+ I=[False, True],
+ cluster_labels=[20, 21],
+ )
+
+ fn = str(tmp_path / "aligned_metadata.zarr")
+ AssayMerge(
+ zarr_path=fn,
+ assays=[left, right],
+ names=["left", "right"],
+ merge_assay_name="RNA",
+ prepend_text="orig",
+ reset_cell_filter=False,
+ source_column="sample_id",
+ seed=7,
+ ).dump()
+
+ root = zarr.open_group(fn, mode="r")
+ ids = np.asarray(root["cellData/ids"][:]).astype(str)
+ sources = np.asarray(root["cellData/sample_id"][:]).astype(str)
+ clusters = np.asarray(root["cellData/orig_cluster_labels"][:])
+ valid = np.asarray(root["cellData/I"][:], dtype=bool)
+ observed = {
+ cell_id: (source, int(cluster), bool(is_valid))
+ for cell_id, source, cluster, is_valid in zip(
+ ids,
+ sources,
+ clusters,
+ valid,
+ strict=True,
+ )
+ }
+
+ assert observed == {
+ "left__l0": ("left", 10, True),
+ "left__l1": ("left", 11, False),
+ "left__l2": ("left", 12, True),
+ "right__r0": ("right", 20, False),
+ "right__r1": ("right", 21, True),
+ }
+
+
+def test_assay_merge_rejects_source_column_conflict(tmp_path):
+ assay = _MergeAssay(
+ "RNA",
+ [[1], [2]],
+ ["c0", "c1"],
+ ["gene"],
+ ["Gene"],
+ block_size=1,
+ )
+ assay.cells = _MergeMeta(
+ ids=["c0", "c1"],
+ names=["c0", "c1"],
+ I=[True, True],
+ sample_id=["existing", "existing"],
+ )
+
+ with pytest.raises(ValueError, match="conflicts with merged metadata"):
+ AssayMerge(
+ zarr_path=str(tmp_path / "source_conflict.zarr"),
+ assays=[assay],
+ names=["sample"],
+ merge_assay_name="RNA",
+ prepend_text="",
+ source_column="sample_id",
+ )
+
+
+def test_assay_merge_preserves_order_across_source_block_sizes(tmp_path):
+ cell_ids = ["c0", "c1", "c2", "c3"]
+ left = _MergeAssay(
+ "RNA",
+ [[1, 10], [2, 20], [3, 30], [4, 40]],
+ cell_ids,
+ ["id_a", "id_b"],
+ ["A", "B"],
+ block_size=1,
+ )
+ right = _MergeAssay(
+ "RNA",
+ [[50, 500], [60, 600], [70, 700], [80, 800]],
+ cell_ids,
+ ["id_b", "id_c"],
+ ["B", "C"],
+ block_size=4,
+ )
+ fn = str(tmp_path / "out_of_order_prefetch.zarr")
+ writer = AssayMerge(
+ zarr_path=fn,
+ assays=[left, right],
+ names=["left", "right"],
+ merge_assay_name="RNA",
+ prepend_text="",
+ seed=3,
+ )
+ writer.dump()
+
+ root = zarr.open_group(fn, mode="r")
+ counts = np.asarray(root["RNA/counts"][:])
+ merged_cell_ids = np.asarray(root["cellData/ids"][:]).astype(str)
+ expected = {
+ "left__c0": [1, 10, 0],
+ "left__c1": [2, 20, 0],
+ "left__c2": [3, 30, 0],
+ "left__c3": [4, 40, 0],
+ "right__c0": [0, 50, 500],
+ "right__c1": [0, 60, 600],
+ "right__c2": [0, 70, 700],
+ "right__c3": [0, 80, 800],
+ }
+ for cell_id, row in zip(merged_cell_ids, counts, strict=True):
+ np.testing.assert_array_equal(row, expected[cell_id])
+
+
+def test_dask_to_coo_sums_consolidated_features():
+ merge = object.__new__(AssayMerge)
+ merge.nFeats = 1
+
+ result = AssayMerge._dask_to_coo(
+ merge,
+ np.array([[2, 3], [5, 7]]),
+ np.array([0, 1]),
+ np.array([0, 0]),
+ 1,
+ )
+
+ assert result.nnz == 2
+ np.testing.assert_array_equal(result.toarray(), [[5], [12]])
+
+
+def test_dataset_merge_shares_row_order_across_assay_chunk_sizes(
+ tmp_path,
+):
+ from scarf.merge import DatasetMerge
+
+ cell_ids = ["c0", "c1", "c2", "c3", "c4"]
+ left = _MergeDataStore(
+ [
+ _MergeAssay(
+ "RNA",
+ [[1, 10], [2, 20], [3, 30], [4, 40], [5, 50]],
+ cell_ids,
+ ["rna_a", "rna_b"],
+ ["RNA A", "RNA B"],
+ block_size=3,
+ ),
+ _MergeAssay(
+ "ADT",
+ [[101, 110], [102, 120], [103, 130], [104, 140], [105, 150]],
+ cell_ids,
+ ["adt_a", "adt_b"],
+ ["ADT A", "ADT B"],
+ block_size=2,
+ ),
+ ]
+ )
+ right = _MergeDataStore(
+ [
+ _MergeAssay(
+ "RNA",
+ [[6, 60], [7, 70], [8, 80], [9, 90], [10, 100]],
+ cell_ids,
+ ["rna_a", "rna_b"],
+ ["RNA A", "RNA B"],
+ block_size=3,
+ ),
+ _MergeAssay(
+ "ADT",
+ [[106, 160], [107, 170], [108, 180], [109, 190], [110, 200]],
+ cell_ids,
+ ["adt_a", "adt_b"],
+ ["ADT A", "ADT B"],
+ block_size=2,
+ ),
+ ]
+ )
+ fn = str(tmp_path / "mixed_assay_chunks.zarr")
+
+ writer = DatasetMerge(
+ datasets=[left, right],
+ zarr_path=fn,
+ names=["left", "right"],
+ prepend_text="",
+ seed=3,
+ )
+ writer.dump()
+
+ assert writer.unique_assays == ["RNA", "ADT"]
+ for generator in writer.merge_generators:
+ assert (
+ max(
+ rows.size
+ for blocks in generator.permutations_rows.values()
+ for rows in blocks.values()
+ )
+ <= 2
+ )
+ root = zarr.open_group(fn, mode="r")
+ merged_cell_ids = np.asarray(root["cellData/ids"][:]).astype(str)
+ rna = np.asarray(root["RNA/counts"][:])
+ adt = np.asarray(root["ADT/counts"][:])
+ expected = {
+ "left__c0": ([1, 10], [101, 110]),
+ "left__c1": ([2, 20], [102, 120]),
+ "left__c2": ([3, 30], [103, 130]),
+ "left__c3": ([4, 40], [104, 140]),
+ "left__c4": ([5, 50], [105, 150]),
+ "right__c0": ([6, 60], [106, 160]),
+ "right__c1": ([7, 70], [107, 170]),
+ "right__c2": ([8, 80], [108, 180]),
+ "right__c3": ([9, 90], [109, 190]),
+ "right__c4": ([10, 100], [110, 200]),
+ }
+ for cell_id, rna_row, adt_row in zip(
+ merged_cell_ids,
+ rna,
+ adt,
+ strict=True,
+ ):
+ expected_rna, expected_adt = expected[cell_id]
+ np.testing.assert_array_equal(rna_row, expected_rna)
+ np.testing.assert_array_equal(adt_row, expected_adt)
+
+
+def test_dataset_merge_shared_row_plan_handles_missing_assays(
+ tmp_path,
+):
+ from scarf.merge import DatasetMerge
+
+ cell_ids = ["c0", "c1", "c2"]
+ left = _MergeDataStore(
+ [
+ _MergeAssay(
+ "RNA",
+ [[1, 10], [2, 20], [3, 30]],
+ cell_ids,
+ ["rna_a", "rna_b"],
+ ["RNA A", "RNA B"],
+ block_size=3,
+ )
+ ]
+ )
+ right = _MergeDataStore(
+ [
+ _MergeAssay(
+ "ADT",
+ [[101, 110], [102, 120], [103, 130]],
+ cell_ids,
+ ["adt_a", "adt_b"],
+ ["ADT A", "ADT B"],
+ block_size=2,
+ )
+ ]
+ )
+ fn = str(tmp_path / "missing_assays.zarr")
+
+ writer = DatasetMerge(
+ datasets=[left, right],
+ zarr_path=fn,
+ names=["left", "right"],
+ prepend_text="",
+ seed=3,
+ )
+ writer.dump()
+
+ root = zarr.open_group(fn, mode="r")
+ merged_cell_ids = np.asarray(root["cellData/ids"][:]).astype(str)
+ rna = np.asarray(root["RNA/counts"][:])
+ adt = np.asarray(root["ADT/counts"][:])
+ expected = {
+ "left__c0": ([1, 10], [0, 0]),
+ "left__c1": ([2, 20], [0, 0]),
+ "left__c2": ([3, 30], [0, 0]),
+ "right__c0": ([0, 0], [101, 110]),
+ "right__c1": ([0, 0], [102, 120]),
+ "right__c2": ([0, 0], [103, 130]),
+ }
+ for cell_id, rna_row, adt_row in zip(
+ merged_cell_ids,
+ rna,
+ adt,
+ strict=True,
+ ):
+ expected_rna, expected_adt = expected[cell_id]
+ np.testing.assert_array_equal(rna_row, expected_rna)
+ np.testing.assert_array_equal(adt_row, expected_adt)
+
+
+def test_dataset_merge_2(datastore, rna_raw_total, assay2_raw_total, tmp_path):
+ from scarf.merge import DatasetMerge
+
+ fn = str(tmp_path / "merged.zarr")
+ writer = DatasetMerge(
+ zarr_path=fn,
+ datasets=[datastore, datastore],
+ names=["self1", "self2"],
+ prepend_text="",
+ overwrite=True,
+ )
+ writer.dump()
+ rna_count = zarr.open(fn + "/RNA/counts")
+ assay2_count = zarr.open(fn + "/assay2/counts")
+ assert rna_count.shape[0] == 2 * datastore.cells.N
+ assert assay2_count.shape[0] == 2 * datastore.cells.N
+ assert int(rna_count[...].sum()) == rna_raw_total * 2
+ assert int(assay2_count[...].sum()) == assay2_raw_total * 2
+
+
+def test_dataset_merge_3(datastore, rna_raw_total, assay2_raw_total, tmp_path):
+ from scarf.merge import DatasetMerge
+
+ fn = str(tmp_path / "merged.zarr")
+ writer = DatasetMerge(
+ zarr_path=fn,
+ datasets=[datastore, datastore, datastore],
+ names=["self1", "self2", "self3"],
+ prepend_text="",
+ overwrite=True,
+ )
+ writer.dump()
+ rna_count = zarr.open(fn + "/RNA/counts")
+ assay2_count = zarr.open(fn + "/assay2/counts")
+ assert rna_count.shape[0] == 3 * datastore.cells.N
+ assert assay2_count.shape[0] == 3 * datastore.cells.N
+ assert int(rna_count[...].sum()) == rna_raw_total * 3
+ assert int(assay2_count[...].sum()) == assay2_raw_total * 3
+
+
+def test_dataset_merge_cells(datastore, tmp_path):
+ from scarf.datastore.datastore import DataStore
+ from scarf.merge import DatasetMerge
+
+ fn = str(tmp_path / "merged.zarr")
+ writer = DatasetMerge(
+ zarr_path=fn,
+ datasets=[datastore, datastore],
+ names=["self1", "self2"],
+ prepend_text="orig",
+ source_column="sample_id",
+ overwrite=True,
+ )
+ writer.dump()
+
+ ds = DataStore(
+ fn,
+ default_assay="RNA",
+ )
+
+ df = ds.cells.to_pandas_dataframe(ds.cells.columns)
+ df_diff = df[df["orig_RNA_nCounts"] != df["RNA_nCounts"]]
+ assert len(df_diff) == 0
+ assert df["sample_id"].value_counts().to_dict() == {
+ "self1": datastore.cells.N,
+ "self2": datastore.cells.N,
+ }
+
+
+def test_assay_merge_rejects_duplicate_sample_names(datastore, tmp_path):
+ fn = str(tmp_path / "merged_dup_names.zarr")
+ with pytest.raises(ValueError, match="unique name"):
+ AssayMerge(
+ zarr_path=fn,
+ assays=[datastore.RNA, datastore.RNA],
+ names=["dup", "dup"],
+ merge_assay_name="RNA",
+ prepend_text="",
+ overwrite=True,
+ )
+
+
+def test_assay_merge_rejects_existing_workspace_assay(
+ datastore,
+ rna_raw_total,
+ tmp_path,
+):
+ fn = str(tmp_path / "workspace_merged.zarr")
+ writer = AssayMerge(
+ zarr_path=fn,
+ assays=[datastore.RNA, datastore.RNA],
+ names=["self1", "self2"],
+ merge_assay_name="RNA",
+ out_workspace="merged",
+ prepend_text="",
+ )
+ writer.dump()
+
+ root = zarr.open_group(fn, mode="r")
+ counts = root["matrices/RNA/counts"]
+ assert counts.shape[0] == 2 * datastore.cells.N
+ assert int(counts[...].sum()) == 2 * rna_raw_total
+ assert "countsT" not in root["matrices/RNA"]
+
+ with pytest.raises(ValueError, match="already contains RNA assay"):
+ AssayMerge(
+ zarr_path=fn,
+ assays=[datastore.RNA, datastore.RNA],
+ names=["self1", "self2"],
+ merge_assay_name="RNA",
+ out_workspace="merged",
+ prepend_text="",
+ )
+
+
+def test_assay_merge_validation_failure_does_not_overwrite_store(monkeypatch):
+ calls = []
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ root.create_group("cellData")
+ root.create_group("RNA")
+
+ def fake_load_zarr(zarr_loc, mode, storage_options=None):
+ calls.append(mode)
+ if mode == "r":
+ return root
+ pytest.fail(f"Unexpected destructive open mode: {mode}")
+
+ monkeypatch.setattr("scarf.merge.assays.load_zarr", fake_load_zarr)
+ merge = object.__new__(AssayMerge)
+ merge.outWorkspace = None
+ merge.storage_options = None
+
+ with pytest.raises(ValueError, match="already contains RNA assay"):
+ AssayMerge._use_existing_zarr(merge, MemoryStore(), "RNA", False)
+
+ assert calls == ["r"]
+
+
+def test_assay_merge_store_skips_local_exists_guard(monkeypatch):
+ calls = []
+
+ def fake_load_zarr(zarr_loc, mode, storage_options=None):
+ calls.append((zarr_loc, mode, storage_options))
+ if mode == "r":
+ raise FileNotFoundError("missing")
+ return zarr.open_group(store=MemoryStore(), mode="w")
+
+ monkeypatch.setattr("scarf.merge.assays.load_zarr", fake_load_zarr)
+ merge = object.__new__(AssayMerge)
+ merge.outWorkspace = None
+ merge.storage_options = {"region": "us-east-1"}
+ root = AssayMerge._use_existing_zarr(merge, MemoryStore(), "RNA", False)
+ assert isinstance(root, zarr.Group)
+ assert calls[-1] == (calls[-1][0], "w", {"region": "us-east-1"})
+
+
+def test_dummy_assay_holds_zero_counts(datastore):
+ from scarf.merge import DummyAssay
+ from scarf.writers import create_zarr_dataset
+
+ mem = zarr.open_group(store=MemoryStore(), mode="w")
+ dummy_array = create_zarr_dataset(
+ mem,
+ "counts",
+ datastore.RNA.rawData.chunksize,
+ datastore.RNA.rawData.dtype,
+ (datastore.cells.N, datastore.RNA.feats.N),
+ )
+ dummy = DummyAssay(
+ datastore,
+ ChunkedArray(dummy_array, nthreads=1),
+ datastore.RNA.feats,
+ "RNA",
+ )
+ assert dummy.name == "RNA"
+ assert int(dummy.rawData.compute().sum()) == 0
diff --git a/tests/test_metadata.py b/tests/test_metadata.py
new file mode 100644
index 00000000..00d1f821
--- /dev/null
+++ b/tests/test_metadata.py
@@ -0,0 +1,53 @@
+import warnings
+
+import numpy as np
+import pytest
+
+
+@pytest.fixture
+def dummy_metadata(tmp_path):
+ import zarr
+
+ from scarf.metadata import MetaData
+
+ fn = str(tmp_path / "dummy_metadata.zarr")
+ g = zarr.open_group(fn, mode="w")
+ data = np.array([1, 1, 1, 1, 0, 0, 1, 1, 1]).astype(bool)
+ g.create_array(
+ "I",
+ data=data,
+ chunks=(100000,),
+ )
+ yield MetaData(g)
+
+
+def test_metadata_attrs(dummy_metadata):
+ assert dummy_metadata.N == 9
+ assert np.all(dummy_metadata.index == np.array(range(9)))
+
+
+def test_metadata_fetch(dummy_metadata):
+ assert len(dummy_metadata.fetch("I")) == 7
+ assert len(dummy_metadata.fetch_all("I")) == 9
+
+
+def test_metadata_verify_bool(dummy_metadata):
+ assert dummy_metadata._verify_bool("I") is True
+
+
+def test_metadata_active_index(dummy_metadata):
+ a = np.array([0, 1, 2, 3, 6, 7, 8])
+ assert np.all(dummy_metadata.active_index(key="I") == a)
+
+
+def test_metadata_partial_float_fill_does_not_cast_uninitialized_values(dummy_metadata):
+ values = np.arange(7, dtype=np.float32)
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error", RuntimeWarning)
+ filled = dummy_metadata._fill_to_index(values, np.nan, "I")
+
+ selected = dummy_metadata.fetch_all("I")
+ assert filled.dtype == values.dtype
+ np.testing.assert_array_equal(filled[selected], values)
+ assert np.isnan(filled[~selected]).all()
diff --git a/tests/test_metadata_artifacts.py b/tests/test_metadata_artifacts.py
new file mode 100644
index 00000000..d911a390
--- /dev/null
+++ b/tests/test_metadata_artifacts.py
@@ -0,0 +1,727 @@
+import numpy as np
+import pandas as pd
+import pytest
+
+import scarf.plotting as splt
+from scarf.datastore.datastore import DataStore
+from scarf.metadata.artifacts import (
+ categorical_display,
+ validate_display_metadata,
+)
+from scarf.plotting._contracts import CategoricalScale, ColorScale
+from scarf.plotting.embedding import _continuous_limits
+from scarf.storage.artifacts import ArtifactRef, artifact_path
+from scarf.storage.selections import resolve_selection_artifact
+from tests.fixtures_datastore import build_neighbourhood_graph
+
+
+def _ensure_graph(datastore) -> None:
+ datastore.auto_filter_cells(show_qc_plots=False)
+ if "I__metadata_hvgs" not in datastore.RNA.feats.columns:
+ datastore.mark_hvgs(
+ from_assay="RNA",
+ cell_key="I",
+ top_n=100,
+ hvg_key_name="metadata_hvgs",
+ show_plot=False,
+ min_cells=int(0.01 * datastore.cells.N),
+ max_cells=np.inf,
+ blacklist="^MT-|^RPS|^RPL|^MRPS|^MRPL|^CCN|^HLA-|^H2-|^HIST",
+ )
+ build_neighbourhood_graph(
+ datastore,
+ from_assay="RNA",
+ feat_key="metadata_hvgs",
+ dims=5,
+ k=3,
+ n_centroids=10,
+ local_cache=False,
+ )
+
+
+def _column_ref(datastore, column: str) -> ArtifactRef:
+ raw_ref = datastore.zw["cellData"][column].attrs["source_artifact"]
+ return ArtifactRef.from_dict(raw_ref)
+
+
+def test_umap_matrix_and_leiden_columns_link_authoritative_artifacts(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ hvg_display = dict(
+ datastore.RNA.z["featureData"]["I__metadata_hvgs"].attrs["display"]
+ )
+ assert hvg_display["kind"] == "categorical"
+ returned_umap = datastore.run_umap(n_epochs=10, label="metadata_umap")
+ returned_leiden = datastore.run_leiden_clustering(label="metadata_leiden")
+
+ umap1 = datastore.zw["cellData"]["RNA_metadata_umap1"]
+ umap2 = datastore.zw["cellData"]["RNA_metadata_umap2"]
+ umap_ref = ArtifactRef.from_dict(umap1.attrs["source_artifact"])
+ leiden_ref = _column_ref(datastore, "RNA_metadata_leiden")
+
+ assert returned_umap == umap_ref
+ assert returned_leiden == leiden_ref
+ assert ArtifactRef.from_dict(umap2.attrs["source_artifact"]) == umap_ref
+ assert umap1.attrs["value_index"] == 0
+ assert umap2.attrs["value_index"] == 1
+ first_input = datastore._resolve_cell_data_provenance_input(
+ "RNA_metadata_umap1",
+ cell_key="I",
+ )
+ second_input = datastore._resolve_cell_data_provenance_input(
+ "RNA_metadata_umap2",
+ cell_key="I",
+ )
+ assert first_input["artifact"] == second_input["artifact"]
+ assert first_input["value_index"] == 0
+ assert second_input["value_index"] == 1
+ assert first_input != second_input
+ display = dict(umap1.attrs["display"])
+ values = datastore.cells.fetch("RNA_metadata_umap1", key="I")
+ assert display["kind"] == "continuous"
+ assert display["minimum"] == float(values.min())
+ assert display["maximum"] == float(values.max())
+ assert datastore.load_artifact(umap_ref)["values"].shape[1] == 2
+ assert leiden_ref.kind == "cluster_labels"
+ leiden_display = dict(
+ datastore.zw["cellData"]["RNA_metadata_leiden"].attrs["display"]
+ )
+ assert leiden_display["kind"] == "categorical"
+ assert all(
+ set(category) == {"value", "label", "color"}
+ for category in leiden_display["categories"]
+ )
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_metadata_umap",
+ color_by="RNA_metadata_umap1",
+ show=False,
+ )
+ continuous = next(scale for scale in result.scales if isinstance(scale, ColorScale))
+ assert continuous.vmin == display["minimum"]
+ assert continuous.vmax == display["maximum"]
+ result.close()
+ multi = splt.embedding(
+ datastore,
+ layout_key="RNA_metadata_umap",
+ color_by=[
+ "RNA_metadata_umap1",
+ "RNA_metadata_leiden",
+ ],
+ show=False,
+ )
+ assert any(isinstance(scale, ColorScale) for scale in multi.scales)
+ assert any(isinstance(scale, CategoricalScale) for scale in multi.scales)
+ multi.close()
+ batches = np.asarray(
+ ["a" if index % 2 else "b" for index in range(datastore.cells.N)]
+ )
+ datastore.cells.insert("display_batch", batches, overwrite=True)
+ datastore.zw["cellData"]["display_batch"].attrs["display"] = categorical_display(
+ datastore.cells.fetch("display_batch", key="I")
+ )
+ leiden_values = datastore.cells.fetch(
+ "RNA_metadata_leiden",
+ key="I",
+ )
+ keep = list(np.unique(leiden_values)[:2])
+ grouped = splt.embedding(
+ datastore,
+ layout_key="RNA_metadata_umap",
+ color_by=["RNA_metadata_leiden", "display_batch"],
+ groups=keep,
+ show=False,
+ )
+ categorical_scales = [
+ scale for scale in grouped.scales if isinstance(scale, CategoricalScale)
+ ]
+ assert list(categorical_scales[0].order) == keep
+ assert set(categorical_scales[1].order) == {"a", "b"}
+ grouped.close()
+
+
+def test_membership_and_smart_labels_are_artifact_backed_and_lisi_is_read_only(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ datastore.run_leiden_clustering(label="independent_leiden")
+ cluster_key = "RNA_independent_leiden"
+
+ datastore.calc_membership_strength(
+ "RNA",
+ "I",
+ "metadata_hvgs",
+ cluster_key,
+ )
+ membership_key = "RNA_I_cluster_membership_strength"
+ membership_ref = _column_ref(datastore, membership_key)
+ assert membership_ref.kind == "membership_strength"
+ graph_loc = datastore.get_latest_graph_loc("RNA", "I", "metadata_hvgs")
+ n_cells, k = datastore._get_graph_ncells_k(graph_loc)
+ edges = np.asarray(datastore.zw[graph_loc]["edges"][:]).reshape(n_cells, k, 2)
+ clusters = np.asarray(datastore.cells.fetch(cluster_key, key="I"))
+ expected_membership = np.asarray(
+ [
+ pd.Series(row).value_counts(dropna=False).iloc[0] / k
+ for row in clusters[edges[:, :, 1]]
+ ]
+ ).round(3)
+ np.testing.assert_array_equal(
+ datastore.cells.fetch(membership_key, key="I"),
+ expected_membership,
+ )
+
+ columns_before = set(datastore.cells.columns)
+ artifacts_before = set(datastore.list_artifacts())
+ lisi = datastore.metric_lisi([cluster_key], perplexity=1)
+ repeated_lisi = datastore.metric_lisi([cluster_key], perplexity=1)
+ np.testing.assert_allclose(lisi[cluster_key], repeated_lisi[cluster_key])
+ assert set(datastore.cells.columns) == columns_before
+ assert set(datastore.list_artifacts()) == artifacts_before
+
+ read_only = DataStore(
+ datastore.zarr_loc,
+ default_assay="RNA",
+ zarr_mode="r",
+ )
+ read_only_lisi = read_only.metric_lisi(
+ [cluster_key],
+ perplexity=1,
+ )
+ np.testing.assert_allclose(read_only_lisi[cluster_key], lisi[cluster_key])
+
+ unsaved_smart = datastore.smart_label(
+ cluster_key,
+ cluster_key,
+ )
+ assert unsaved_smart == datastore.smart_label(
+ cluster_key,
+ cluster_key,
+ )
+ datastore.smart_label(
+ cluster_key,
+ cluster_key,
+ new_col_name="smart_clusters",
+ )
+ smart_ref = _column_ref(datastore, "smart_clusters")
+ assert smart_ref.kind == "smart_label"
+ with monkeypatch.context() as cached:
+ cached.setattr(
+ pd,
+ "crosstab",
+ lambda *args, **kwargs: (_ for _ in ()).throw(
+ AssertionError("cached smart labels were recomputed")
+ ),
+ )
+ datastore.smart_label(
+ cluster_key,
+ cluster_key,
+ new_col_name="smart_clusters",
+ )
+ assert _column_ref(datastore, "smart_clusters") == smart_ref
+
+
+def test_hto_identity_is_artifact_backed(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = datastore_ephemeral
+ n_active = len(datastore.cells.active_index("I"))
+ column_name = "sample_id"
+ expected = np.asarray(
+ ["negative" if index % 2 else "tag" for index in range(n_active)]
+ )
+ monkeypatch.setattr(
+ "scarf.datastore._operations.quality_control.hto_demux",
+ lambda counts, **kwargs: pd.Series(expected[: len(counts)]),
+ )
+
+ returned_label = datastore.mark_hto_identities(
+ from_assay="assay2",
+ cell_key="I",
+ label=column_name,
+ )
+
+ assert returned_label == column_name
+ ref = _column_ref(datastore, column_name)
+ assert ref.kind == "hto_identity"
+ parameters = datastore.inspect_artifact(ref).parameters
+ assert parameters is not None
+ assert parameters["method"] == {
+ "normalization": "clr_per_hto",
+ "clustering": {
+ "method": "kmeans",
+ "init": "random",
+ "n_starts": 100,
+ "cluster_count": "n_htos_plus_one",
+ },
+ "background": "raw_mean",
+ "cutoff": {
+ "distribution": "negative_binomial_nb2",
+ "quantile": 0.99,
+ "location": 0,
+ "comparison": "strictly_greater",
+ },
+ "singlet_assignment": "clr_argmax",
+ }
+ assert "algorithm_version" not in parameters
+ np.testing.assert_array_equal(
+ datastore.cells.fetch(column_name, key="I"),
+ expected,
+ )
+ monkeypatch.setattr(
+ "scarf.datastore._operations.quality_control.hto_demux",
+ lambda counts, **kwargs: (_ for _ in ()).throw(
+ AssertionError("cached HTO identities were recomputed")
+ ),
+ )
+ cached_label = datastore.mark_hto_identities(
+ from_assay="assay2",
+ cell_key="I",
+ label=column_name,
+ )
+ assert cached_label == column_name
+ assert _column_ref(datastore, column_name) == ref
+
+
+def test_user_display_metadata_survives_column_refresh(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ datastore.run_umap(n_epochs=10, label="display_umap")
+ datastore.run_leiden_clustering(label="display_leiden")
+ column = datastore.zw["cellData"]["RNA_display_leiden"]
+ custom = {
+ "kind": "categorical",
+ "categories": [
+ {
+ "value": int(category),
+ "label": f"Group {category}",
+ "color": "#123456",
+ }
+ for category in np.unique(
+ datastore.cells.fetch("RNA_display_leiden", key="I")
+ )
+ ],
+ }
+ column.attrs["display"] = custom
+
+ datastore.run_leiden_clustering(
+ label="display_leiden",
+ invalidate_cache=True,
+ )
+
+ assert (
+ dict(datastore.zw["cellData"]["RNA_display_leiden"].attrs["display"]) == custom
+ )
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_display_umap",
+ color_by="RNA_display_leiden",
+ show=False,
+ )
+ categorical = next(
+ scale for scale in result.scales if isinstance(scale, CategoricalScale)
+ )
+ assert categorical.palette is not None
+ assert set(categorical.palette.values()) == {"#123456"}
+ assert categorical.labels is not None
+ assert all(label.startswith("Group ") for label in categorical.labels.values())
+ result.close()
+
+
+def test_malformed_display_is_rejected_before_column_refresh(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ datastore.run_leiden_clustering(label="malformed_display")
+ column_name = "RNA_malformed_display"
+ column = datastore.zw["cellData"][column_name]
+ original_ref = dict(column.attrs["source_artifact"])
+ original_values = datastore.cells.fetch_all(column_name).copy()
+ column.attrs["display"] = "invalid"
+
+ with pytest.raises(TypeError, match="mapping"):
+ datastore.run_leiden_clustering(
+ label="malformed_display",
+ invalidate_cache=True,
+ )
+
+ np.testing.assert_array_equal(
+ datastore.cells.fetch_all(column_name),
+ original_values,
+ )
+ assert (
+ dict(datastore.zw["cellData"][column_name].attrs["source_artifact"])
+ == original_ref
+ )
+
+
+def test_multicolumn_display_validation_is_atomic(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ datastore.run_umap(n_epochs=10, label="atomic_display")
+ first = datastore.zw["cellData"]["RNA_atomic_display1"]
+ second = datastore.zw["cellData"]["RNA_atomic_display2"]
+ first_ref = dict(first.attrs["source_artifact"])
+ second_ref = dict(second.attrs["source_artifact"])
+ second.attrs["display"] = "invalid"
+
+ with pytest.raises(TypeError, match="mapping"):
+ datastore.run_umap(
+ n_epochs=10,
+ label="atomic_display",
+ invalidate_cache=True,
+ )
+
+ assert dict(first.attrs["source_artifact"]) == first_ref
+ assert dict(second.attrs["source_artifact"]) == second_ref
+
+
+def test_display_validation_rejects_nonfinite_and_duplicate_values() -> None:
+ with pytest.raises(TypeError, match="minimum"):
+ validate_display_metadata(
+ {
+ "kind": "continuous",
+ "colormap": "viridis",
+ "minimum": np.nan,
+ "maximum": 1.0,
+ "scale": "linear",
+ }
+ )
+ with pytest.raises(ValueError, match="unique"):
+ validate_display_metadata(
+ {
+ "kind": "categorical",
+ "categories": [
+ {"value": 1, "label": "A", "color": "#123456"},
+ {"value": 1, "label": "B", "color": "#654321"},
+ ],
+ }
+ )
+ with pytest.raises(ValueError, match="collide"):
+ validate_display_metadata(
+ {
+ "kind": "categorical",
+ "categories": [
+ {"value": True, "label": "Yes", "color": "#123456"},
+ {"value": 1, "label": "One", "color": "#654321"},
+ ],
+ }
+ )
+ with pytest.raises(ValueError, match="unknown fields"):
+ validate_display_metadata(
+ {
+ "kind": "continuous",
+ "colormap": "viridis",
+ "minimum": 0.0,
+ "maximum": 1.0,
+ "scale": "linear",
+ "extra": "invalid",
+ }
+ )
+
+
+def test_small_constant_log_bounds_remain_positive() -> None:
+ limits = _continuous_limits(
+ np.asarray([0.1, 0.1]),
+ ColorScale(
+ cmap="viridis",
+ vmin=0.1,
+ vmax=0.1,
+ scale="log",
+ ),
+ )
+
+ assert limits[0] > 0
+ assert limits[0] < 0.1 < limits[1]
+
+
+def test_composition_keeps_missing_category_distinct(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ n_cells = len(datastore.cells.active_index("I"))
+ values = np.asarray(
+ [
+ np.nan if index % 3 == 0 else 1.0 if index % 3 == 1 else 2.0
+ for index in range(n_cells)
+ ],
+ dtype=float,
+ )
+ datastore.cells.insert(
+ "composition_missing",
+ values,
+ key="I",
+ overwrite=True,
+ )
+ result = splt.composition(
+ datastore,
+ category_by="composition_missing",
+ categorical_scale=CategoricalScale(
+ order=(1.0, 2.0),
+ palette={1.0: "#123456", 2.0: "#654321"},
+ missing_color="#ff0000",
+ missing_label="Missing",
+ ),
+ show=False,
+ )
+
+ categories = result.tables["aggregate"]["category"].tolist()
+ assert 1.0 in categories
+ assert any(value is None for value in categories)
+ assert all(value != "\x00scarf_missing_category\x00" for value in categories)
+ returned = next(
+ scale for scale in result.scales if isinstance(scale, CategoricalScale)
+ )
+ assert returned.missing_color == "#ff0000"
+ result.close()
+
+
+def test_manual_column_edit_is_captured_as_new_artifact(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ datastore.run_leiden_clustering(label="editable_leiden")
+ column = "RNA_editable_leiden"
+ original = _column_ref(datastore, column)
+ edited = np.asarray(datastore.cells.fetch(column, key="I")).copy()
+ edited[0] = int(edited.max()) + 1
+ datastore.cells.insert(column, edited, key="I", overwrite=True)
+
+ captured = datastore._resolve_cell_data_input(column, cell_key="I")
+
+ assert captured != original
+ assert captured.kind == "metadata_snapshot"
+ np.testing.assert_array_equal(
+ datastore.load_artifact(captured)["values"][:],
+ edited,
+ )
+ assert _column_ref(datastore, column) == captured
+
+
+def test_cell_cycle_columns_share_one_artifact(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ datastore.auto_filter_cells(show_qc_plots=False)
+ datastore.run_cell_cycle_scoring()
+
+ refs = {
+ _column_ref(datastore, "RNA_S_score"),
+ _column_ref(datastore, "RNA_G2M_score"),
+ _column_ref(datastore, "RNA_cell_cycle_phase"),
+ }
+
+ assert len(refs) == 1
+ ref = refs.pop()
+ assert ref.kind == "cell_cycle"
+ assert set(datastore.load_artifact(ref).array_keys()) == {
+ "s_score",
+ "g2m_score",
+ "phase",
+ }
+
+
+def test_fate_columns_share_one_reusable_artifact(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ n_cells = len(datastore.cells.active_index("I"))
+ pseudotime = np.linspace(0.0, 1.0, n_cells)
+ labels = np.full(n_cells, "other", dtype=object)
+ labels[-2:] = ["A", "B"]
+ datastore.cells.insert(
+ "artifact_pseudotime",
+ pseudotime,
+ key="I",
+ overwrite=True,
+ )
+ datastore.cells.insert(
+ "artifact_sinks",
+ labels,
+ key="I",
+ overwrite=True,
+ )
+
+ result = datastore.run_fate_mapping(
+ pseudotime_key="artifact_pseudotime",
+ sink_key="artifact_sinks",
+ sinks=["A", "B"],
+ label="artifact_fate",
+ )
+ refs = {
+ _column_ref(datastore, column)
+ for column in (*result.fate_keys, result.validity_key)
+ }
+ assert len(refs) == 1
+ fate_ref = refs.pop()
+ assert fate_ref.kind == "fate_map"
+
+ from scarf.datastore._operations import trajectory as trajectory_operations
+
+ def fail_if_recomputed(*_args, **_kwargs):
+ raise AssertionError("fate mapping should have been reused")
+
+ monkeypatch.setattr(
+ trajectory_operations,
+ "_compute_fate_probabilities_impl",
+ fail_if_recomputed,
+ )
+ cached = datastore.run_fate_mapping(
+ pseudotime_key="artifact_pseudotime",
+ sink_key="artifact_sinks",
+ sinks=["A", "B"],
+ label="artifact_fate_cached",
+ )
+ assert _column_ref(datastore, cached.fate_keys[0]) == fate_ref
+ np.testing.assert_allclose(cached.values, result.values, equal_nan=True)
+
+
+def test_graph_outputs_reject_equal_size_different_cell_selection(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ mask = np.asarray(datastore.cells.fetch_all("I"), dtype=bool)
+ selected = np.flatnonzero(mask)
+ excluded = np.flatnonzero(~mask)
+ assert len(selected) > 0 and len(excluded) > 0
+ mask[selected[0]] = False
+ mask[excluded[0]] = True
+ datastore.cells.insert("I", mask, overwrite=True, force=True)
+
+ with pytest.raises(ValueError, match="no longer matches"):
+ datastore.run_leiden_clustering(label="misaligned")
+ with pytest.raises(ValueError, match="no longer matches"):
+ datastore.run_umap(n_epochs=10, label="misaligned_umap")
+ with pytest.raises(ValueError, match="no longer matches"):
+ datastore.get_diffusion_operator(invalidate_cache=True)
+ with pytest.raises(ValueError, match="no longer matches"):
+ datastore.metric_lisi(
+ ["names"],
+ perplexity=1,
+ )
+
+
+def test_graph_outputs_accept_content_equivalent_selection_artifacts(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ equivalent = resolve_selection_artifact(
+ datastore.zw,
+ scope="datastore",
+ kind="cell_selection",
+ values=np.asarray(datastore.cells.fetch_all("I"), dtype=bool),
+ row_ids=np.asarray(datastore.cells.fetch_all("ids")),
+ operation="equivalent_selection",
+ parameters={"source": "test"},
+ inputs={},
+ source_column="I",
+ )
+ datastore.zw["cellData/I"].attrs["source_artifact"] = equivalent.to_dict()
+ datastore.zw["cellData/I"].attrs["source_value"] = "values"
+
+ datastore.run_leiden_clustering(label="equivalent_selection")
+ assert "RNA_equivalent_selection" in datastore.cells.columns
+
+
+def test_graph_consumers_accept_an_explicit_connectivity_map(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ graph = state.connectivity_map
+ assert graph is not None
+
+ implicit = datastore.run_leiden_clustering(label="implicit_leiden")
+ explicit = datastore.run_leiden_clustering(graph, label="explicit_leiden")
+
+ assert explicit == implicit
+ np.testing.assert_array_equal(
+ datastore.cells.fetch("RNA_implicit_leiden", key="I"),
+ datastore.cells.fetch("RNA_explicit_leiden", key="I"),
+ )
+ umap_ref = datastore.run_umap(graph, n_epochs=10, label="explicit_umap")
+ assert umap_ref.kind == "embedding"
+ assert "RNA_explicit_umap1" in datastore.cells.columns
+
+ side_neighbors = datastore.query_neighbors(
+ state.ann_index,
+ k=5,
+ update_state=False,
+ )
+ side_graph = datastore.build_connectivity_map(
+ side_neighbors,
+ update_state=False,
+ )
+ side = datastore.run_leiden_clustering(side_graph, label="side_leiden")
+ assert side != implicit
+ assert "RNA_side_leiden" in datastore.cells.columns
+ assert datastore.get_assay_state("RNA").connectivity_map == graph
+
+ with pytest.raises(ValueError, match="does not match the graph"):
+ datastore.run_leiden_clustering(graph, feat_key="I", label="mismatched")
+ with pytest.raises(TypeError, match="artifact reference"):
+ datastore.run_umap("RNA/graph", n_epochs=10, label="not_a_ref")
+ with pytest.raises(ValueError, match="connectivity map"):
+ datastore.run_umap(state.reduction, n_epochs=10, label="wrong_kind")
+
+
+def test_lisi_rejects_incomplete_ann_dependency(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.ann_index is not None
+ assert state.neighbors is not None
+ ann_group = datastore.zw[artifact_path(state.ann_index)]
+ ann_group.attrs["complete"] = False
+
+ try:
+ with pytest.raises(ValueError, match="ann_index artifact"):
+ datastore.metric_lisi(
+ ["names"],
+ use_latest_knn=False,
+ knn_loc=artifact_path(state.neighbors),
+ perplexity=1,
+ )
+ finally:
+ ann_group.attrs["complete"] = True
+
+
+def test_unedited_paris_column_retains_cluster_cut_ref(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ _ensure_graph(datastore)
+ result = datastore.run_paris_clustering(
+ n_clusters=3,
+ label="metadata_paris",
+ )
+ assert result.label_key is not None
+ linked = _column_ref(datastore, result.label_key)
+
+ resolved = datastore._resolve_cell_data_input(
+ result.label_key,
+ cell_key="I",
+ )
+
+ assert linked.kind == "cluster_cut"
+ assert resolved == linked
diff --git a/tests/test_metadata_blocks.py b/tests/test_metadata_blocks.py
new file mode 100644
index 00000000..eb5abb5a
--- /dev/null
+++ b/tests/test_metadata_blocks.py
@@ -0,0 +1,157 @@
+"""Milestone C: MetaData.iter_row_blocks parity and make_bulk cell_key semantics."""
+
+import numpy as np
+
+from scarf.utils.logging import logger
+
+_PSEUDO_REP_WARNING = (
+ "make_bulk with pseudo_reps > 1 randomly splits cells within each "
+ "group into descriptive resamples"
+)
+
+
+def test_iter_row_blocks_matches_active_index_and_fetch(datastore):
+ cells = datastore.cells
+ col = "ids"
+ expected_idx = cells.active_index("I")
+ expected_vals = cells.fetch(col, key="I")
+
+ for block_rows in (None, 7, 1, cells.default_block_rows("I")):
+ blocks = list(
+ cells.iter_row_blocks(cell_key="I", columns=[col], block_rows=block_rows)
+ )
+ got_idx = (
+ np.concatenate([b.active_global_indices for b in blocks])
+ if blocks
+ else np.array([], dtype=np.int64)
+ )
+ got_vals = (
+ np.concatenate([b.values[col] for b in blocks]) if blocks else np.array([])
+ )
+ np.testing.assert_array_equal(got_idx, expected_idx)
+ np.testing.assert_array_equal(got_vals, expected_vals)
+ assert all(b.start < b.stop for b in blocks)
+ assert blocks[0].start == 0
+ assert blocks[-1].stop == cells.N
+
+
+def test_iter_row_blocks_chunk_edges_cover_full_range(datastore):
+ cells = datastore.cells
+ chunk = cells.default_block_rows("I")
+ blocks = list(cells.iter_row_blocks(cell_key="I", block_rows=chunk))
+ assert sum(b.stop - b.start for b in blocks) == cells.N
+ for b in blocks:
+ assert b.stop - b.start <= chunk
+
+
+def test_iter_row_blocks_respects_subset_cell_key(datastore):
+ cells = datastore.cells
+ keep = np.zeros(cells.N, dtype=bool)
+ keep[::3] = True
+ cells.insert("block_subset", keep, overwrite=True)
+ expected = cells.active_index("block_subset")
+ got = np.concatenate(
+ [
+ b.active_global_indices
+ for b in cells.iter_row_blocks(
+ cell_key="block_subset", columns=["ids"], block_rows=11
+ )
+ ]
+ )
+ np.testing.assert_array_equal(got, expected)
+
+
+def test_make_bulk_respects_non_default_cell_key(leiden_clustering, datastore):
+ """Inactive cells that share a group label must not enter the bulk profile."""
+ ds = datastore
+ active = ds.cells.fetch_all("I").copy()
+ assert active.dtype == bool
+ active_idx = np.flatnonzero(active)
+ drop = np.zeros(ds.cells.N, dtype=bool)
+ drop[active_idx[::2]] = True
+ subset = active & ~drop
+ ds.cells.insert("bulk_subset", subset, overwrite=True)
+
+ full = ds.make_bulk(
+ group_key="RNA_leiden_cluster",
+ cell_key="I",
+ aggr_type="sum",
+ remove_empty_features=False,
+ feature_label="index",
+ )
+ sub = ds.make_bulk(
+ group_key="RNA_leiden_cluster",
+ cell_key="bulk_subset",
+ aggr_type="sum",
+ remove_empty_features=False,
+ feature_label="index",
+ )
+ shared = [c for c in sub.columns if c in full.columns]
+ assert shared
+ for c in shared:
+ assert (sub[c].to_numpy() <= full[c].to_numpy() + 1e-6).all()
+
+ from scarf.utils import controlled_compute
+
+ clusters = ds.cells.fetch_all("RNA_leiden_cluster")
+ subset_idx = ds.cells.active_index("bulk_subset")
+ g_val = clusters[subset_idx[0]]
+ col = str(g_val)
+ assert col in sub.columns
+ idx_g = subset_idx[clusters[subset_idx] == g_val]
+ expected = controlled_compute(ds.RNA.rawData[idx_g].sum(axis=0), ds.nthreads)
+ np.testing.assert_allclose(sub[col].to_numpy(), expected, rtol=1e-5, atol=1e-6)
+
+
+def test_make_bulk_pseudo_reps_warns_without_changing_values(
+ leiden_clustering, datastore
+):
+ """Pseudo-replicate splits warn once; aggregation stays numerically identical."""
+ ds = datastore
+ kwargs = {
+ "group_key": "RNA_leiden_cluster",
+ "aggr_type": "sum",
+ "remove_empty_features": False,
+ "feature_label": "index",
+ "random_seed": 4466,
+ }
+
+ default_messages: list[str] = []
+ sink = logger.add(
+ lambda message: default_messages.append(message.record["message"]),
+ level="WARNING",
+ )
+ try:
+ default = ds.make_bulk(pseudo_reps=1, **kwargs)
+ finally:
+ logger.remove(sink)
+ assert not any(_PSEUDO_REP_WARNING in msg for msg in default_messages)
+
+ rep_messages: list[str] = []
+ sink = logger.add(
+ lambda message: rep_messages.append(message.record["message"]),
+ level="WARNING",
+ )
+ try:
+ with_reps = ds.make_bulk(pseudo_reps=2, **kwargs)
+ again = ds.make_bulk(pseudo_reps=2, **kwargs)
+ finally:
+ logger.remove(sink)
+
+ assert sum(_PSEUDO_REP_WARNING in msg for msg in rep_messages) == 2
+ assert any("not independent biological replicates" in msg for msg in rep_messages)
+ pd_values_equal = np.allclose(with_reps.to_numpy(), again.to_numpy())
+ assert pd_values_equal
+ assert with_reps.shape[1] == 2 * default.shape[1]
+ # Each group's two pseudo-reps partition the cells; their sums recover the
+ # unsplit group total (aggregation path unchanged by the warning).
+ for col in default.columns:
+ rep1 = f"{col}_Rep1"
+ rep2 = f"{col}_Rep2"
+ assert rep1 in with_reps.columns and rep2 in with_reps.columns
+ np.testing.assert_allclose(
+ with_reps[rep1].to_numpy() + with_reps[rep2].to_numpy(),
+ default[col].to_numpy(),
+ rtol=1e-5,
+ atol=1e-6,
+ )
diff --git a/tests/test_metadata_contracts.py b/tests/test_metadata_contracts.py
new file mode 100644
index 00000000..dba7296c
--- /dev/null
+++ b/tests/test_metadata_contracts.py
@@ -0,0 +1,144 @@
+import pickle
+
+import numpy as np
+import zarr
+from zarr.storage import MemoryStore
+
+import scarf.metadata as metadata
+from scarf.metadata.rows import MetaDataRowBlock as implementation_row_block
+from scarf.metadata.table import MetaData as implementation_metadata
+from tests.signature_contracts import signature_digest
+
+
+_METHODS = {
+ "__init__",
+ "__repr__",
+ "_col_renamer",
+ "_column_map",
+ "_fill_to_index",
+ "_get_array",
+ "_get_loc",
+ "_get_size",
+ "_save",
+ "_verify_bool",
+ "active_index",
+ "default_block_rows",
+ "drop",
+ "fetch",
+ "fetch_all",
+ "get_dtype",
+ "get_index_by",
+ "grep",
+ "head",
+ "index_to_bool",
+ "insert",
+ "iter_row_blocks",
+ "mount_location",
+ "multi_sift",
+ "remove_trend",
+ "reset_key",
+ "sift",
+ "to_pandas_dataframe",
+ "unmount_location",
+ "update_key",
+}
+
+
+def _metadata_fixture() -> metadata.MetaData:
+ group = zarr.open_group(store=MemoryStore(), mode="w")
+ group.create_array(
+ "I",
+ data=np.array([True, False, True, True]),
+ chunks=(2,),
+ )
+ group.create_array(
+ "ids",
+ data=np.array(["a", "b", "c", "d"]),
+ chunks=(2,),
+ )
+ group.create_array(
+ "names",
+ data=np.array(["Alpha", "Beta", "Alpine", "Delta"]),
+ chunks=(2,),
+ )
+ group.create_array(
+ "score",
+ data=np.array([0.5, 2.0, 3.5, 5.0]),
+ chunks=(2,),
+ )
+ return metadata.MetaData(group)
+
+
+def test_metadata_facade_exports_canonical_objects():
+ assert metadata.__all__ == ["MetaData", "MetaDataRowBlock"]
+ assert metadata.MetaData is implementation_metadata
+ assert metadata.MetaDataRowBlock is implementation_row_block
+ assert metadata.MetaData.__module__ == "scarf.metadata"
+ assert metadata.MetaDataRowBlock.__module__ == "scarf.metadata"
+
+
+def test_metadata_method_ownership_and_signatures_remain_stable():
+ assert _METHODS <= set(metadata.MetaData.__dict__)
+ methods = {name: getattr(metadata.MetaData, name) for name in _METHODS}
+
+ assert signature_digest(methods) == (
+ "f882e8fb00453f282ab117edfa27a4433ee2e46446b71631f8c4c450f0327d5d"
+ )
+
+
+def test_metadata_rows_and_queries_match_table_contract():
+ table = _metadata_fixture()
+
+ assert table.default_block_rows() == 2
+ blocks = list(table.iter_row_blocks(columns=["score"], block_rows=2))
+ np.testing.assert_array_equal(
+ np.concatenate([block.active_global_indices for block in blocks]),
+ [0, 2, 3],
+ )
+ np.testing.assert_allclose(
+ np.concatenate([block.values["score"] for block in blocks]),
+ [0.5, 3.5, 5.0],
+ )
+ np.testing.assert_array_equal(
+ table.sift("score", 1.0, 4.0), [False, True, True, False]
+ )
+ assert table.grep("^AL") == ["ALPHA", "ALPINE"]
+ assert table.head(2)["score"].tolist() == [0.5, 2.0]
+
+
+def test_metadata_remove_trend_preserves_fixed_strategy():
+ from scarf.features.variability import fit_lowess
+
+ rng = np.random.default_rng(5)
+ means = rng.uniform(0.5, 20.0, 80)
+ variances = np.clip(means**1.5 + rng.normal(0, 0.05, len(means)), 0.1, None)
+ group = zarr.open_group(store=MemoryStore(), mode="w")
+ group.create_array("I", data=np.ones(len(means), dtype=bool))
+ group.create_array("means", data=means)
+ group.create_array("variances", data=variances)
+ table = metadata.MetaData(group)
+
+ observed = table.remove_trend("means", "variances", n_bins=8, lowess_frac=0.6)
+ expected = fit_lowess(
+ means,
+ variances,
+ n_bins=8,
+ lowess_frac=0.6,
+ bin_strategy="fixed",
+ )
+
+ np.testing.assert_array_equal(observed, expected)
+
+
+def test_metadata_row_blocks_remain_pickle_resolvable():
+ block = metadata.MetaDataRowBlock(
+ start=0,
+ stop=2,
+ active_global_indices=np.array([0]),
+ values={"score": np.array([0.5])},
+ )
+
+ restored = pickle.loads(pickle.dumps(block))
+
+ assert type(restored) is metadata.MetaDataRowBlock
+ np.testing.assert_array_equal(restored.active_global_indices, [0])
diff --git a/tests/test_metrics.py b/tests/test_metrics.py
new file mode 100644
index 00000000..2814dd9c
--- /dev/null
+++ b/tests/test_metrics.py
@@ -0,0 +1,645 @@
+import numpy as np
+import pandas as pd
+import pytest
+import zarr
+from scipy.sparse import csr_matrix
+from scipy.sparse.csgraph import connected_components
+from zarr.storage import MemoryStore
+
+from scarf.metrics import (
+ calculate_knn_cluster_similarity,
+ calculate_top_k_neighbor_distances,
+ calculate_weighted_cluster_similarity,
+ clisi_knn,
+ compute_lisi,
+ compute_simpson,
+ graph_connectivity,
+ ilisi_knn,
+ knn_to_csr_matrix,
+ label_concordance_score,
+ lisi_batch_mixing_score,
+ silhouette_scoring,
+)
+from scarf.metrics.lisi import _effective_perplexity, _neighbor_probabilities
+
+
+def _uniform_self_free_knn() -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ indices = np.array(
+ [
+ [1, 2, 3],
+ [0, 2, 3],
+ [0, 1, 3],
+ [0, 1, 2],
+ ]
+ )
+ distances = np.ones_like(indices, dtype=np.float64)
+ labels = np.array([0, 0, 1, 1])
+ return distances, indices, labels
+
+
+def test_compute_lisi_uses_all_stored_neighbors():
+ metadata = pd.DataFrame({"batch": [1, 0, 0, 0]})
+ indices = np.tile(np.array([0, 1, 2]), (4, 1))
+ distances = np.ones_like(indices, dtype=np.float64)
+
+ scores = compute_lisi(
+ distances,
+ indices,
+ metadata,
+ label_colnames=["batch"],
+ perplexity=1,
+ )
+
+ assert np.allclose(scores[:, 0], 1.8)
+
+
+def test_compute_lisi_rejects_missing_labels():
+ metadata = pd.DataFrame({"batch": [0, 1, np.nan, 1]})
+ indices = np.tile(np.array([0, 1, 2]), (4, 1))
+ distances = np.ones_like(indices, dtype=np.float64)
+
+ with pytest.raises(ValueError, match="missing values"):
+ compute_lisi(distances, indices, metadata, ["batch"])
+
+
+def test_compute_lisi_returns_empty_matrix_when_no_labels_are_requested():
+ metadata = pd.DataFrame(index=np.arange(2))
+ distances = np.empty((2, 0))
+ indices = np.empty((2, 0), dtype=np.int64)
+
+ scores = compute_lisi(distances, indices, metadata, [])
+
+ assert scores.shape == (2, 0)
+ assert scores.dtype == np.float64
+
+
+def test_compute_lisi_handles_more_categories_than_neighbors():
+ metadata = pd.DataFrame({"batch": ["a", "b", "c", "d"]})
+ indices = np.array(
+ [
+ [0, 1, 2],
+ [1, 2, 3],
+ [2, 3, 0],
+ [3, 0, 1],
+ ]
+ )
+ distances = np.ones_like(indices, dtype=np.float64)
+
+ scores = compute_lisi(
+ distances,
+ indices,
+ metadata,
+ label_colnames=["batch"],
+ perplexity=1,
+ )
+
+ assert np.allclose(scores[:, 0], 3.0)
+
+
+def test_compute_lisi_caps_perplexity_to_neighbor_capacity():
+ metadata = pd.DataFrame({"batch": [0, 0, 1, 1]})
+ indices = np.tile(np.array([0, 1, 2, 3, 0, 1]), (4, 1))
+ distances = np.tile(np.arange(6, dtype=np.float64), (4, 1))
+
+ capped = compute_lisi(distances, indices, metadata, ["batch"], perplexity=2)
+ oversized = compute_lisi(distances, indices, metadata, ["batch"], perplexity=30)
+
+ assert np.allclose(oversized, capped)
+
+
+@pytest.mark.parametrize("perplexity", [0.5, np.inf, np.nan])
+def test_effective_perplexity_rejects_invalid_values(perplexity):
+ with pytest.raises(ValueError, match="finite value"):
+ _effective_perplexity(perplexity, n_neighbors=3)
+
+
+def test_effective_perplexity_requires_three_neighbors():
+ with pytest.raises(ValueError, match="at least three"):
+ _effective_perplexity(perplexity=1, n_neighbors=2)
+
+
+def test_neighbor_probabilities_calibrate_diffuse_and_concentrated_rows():
+ distances = np.array(
+ [
+ [0.0, 0.1, 0.2, 0.3],
+ [0.0, 10.0, 20.0, 30.0],
+ ]
+ )
+
+ probabilities = _neighbor_probabilities(
+ distances,
+ perplexity=2,
+ tol=1e-8,
+ max_iter=100,
+ )
+ log_probabilities = np.log(
+ probabilities,
+ out=np.zeros_like(probabilities),
+ where=probabilities > 0,
+ )
+ calibrated_perplexity = np.exp(-np.sum(probabilities * log_probabilities, axis=1))
+
+ assert np.allclose(probabilities.sum(axis=1), 1.0)
+ assert np.allclose(calibrated_perplexity, 2.0, rtol=1e-7)
+
+
+def test_compute_simpson_with_uniform_neighbor_weights():
+ distances = np.ones((3, 2), dtype=np.float64)
+ indices = np.array(
+ [
+ [0, 1],
+ [1, 2],
+ [2, 3],
+ ]
+ )
+ labels = pd.Categorical([0, 0, 1, 1])
+
+ simpson = compute_simpson(distances, indices, labels, perplexity=1)
+
+ assert np.allclose(simpson, 5 / 9)
+
+
+def test_compute_lisi_rejects_invalid_neighbor_distances():
+ metadata = pd.DataFrame({"batch": [0, 0, 1, 1]})
+ indices = np.tile(np.array([0, 1, 2]), (4, 1))
+ distances = np.ones_like(indices, dtype=np.float64)
+ distances[0, 0] = -1
+
+ with pytest.raises(ValueError, match="finite, non-negative"):
+ compute_lisi(distances, indices, metadata, ["batch"], perplexity=1)
+
+
+def test_ilisi_and_clisi_match_analytic_self_free_values():
+ distances, indices, labels = _uniform_self_free_knn()
+
+ assert ilisi_knn(distances, indices, labels) == pytest.approx(0.8)
+ assert clisi_knn(distances, indices, labels) == pytest.approx(0.2)
+ assert ilisi_knn(
+ distances,
+ indices,
+ labels,
+ perplexity=1,
+ scale=False,
+ ) == pytest.approx(1.8)
+
+
+def test_lisi_summary_default_perplexity_matches_floor_k_over_three():
+ distances, indices, labels = _uniform_self_free_knn()
+
+ assert ilisi_knn(distances, indices, labels) == pytest.approx(
+ ilisi_knn(distances, indices, labels, perplexity=1)
+ )
+ assert clisi_knn(distances, indices, labels) == pytest.approx(
+ clisi_knn(distances, indices, labels, perplexity=1)
+ )
+
+
+@pytest.mark.parametrize("metric", [ilisi_knn, clisi_knn])
+def test_lisi_summary_validates_categories_and_alignment(metric):
+ distances, indices, labels = _uniform_self_free_knn()
+
+ with pytest.raises(ValueError, match="at least two categories"):
+ metric(distances, indices, np.zeros(len(labels)), perplexity=1)
+ with pytest.raises(ValueError, match="missing values"):
+ metric(distances, indices, np.array([0, 0, 1, np.nan]), perplexity=1)
+ with pytest.raises(ValueError, match="number of cells"):
+ metric(distances, indices, labels[:-1], perplexity=1)
+ with pytest.raises(ValueError, match="at least two categories"):
+ metric(
+ np.empty((0, 3)),
+ np.empty((0, 3), dtype=np.int64),
+ np.array([]),
+ perplexity=1,
+ )
+
+
+def test_lisi_summaries_match_chunked_zarr_inputs():
+ distances, indices, labels = _uniform_self_free_knn()
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ z_distances = root.create_array(
+ "distances",
+ data=distances,
+ chunks=(2, 3),
+ )
+ z_indices = root.create_array(
+ "indices",
+ data=indices,
+ chunks=(2, 3),
+ )
+
+ assert ilisi_knn(z_distances, z_indices, labels) == pytest.approx(
+ ilisi_knn(distances, indices, labels)
+ )
+ assert clisi_knn(z_distances, z_indices, labels) == pytest.approx(
+ clisi_knn(distances, indices, labels)
+ )
+
+
+def test_proportional_batch_mixing_differs_from_ilisi_for_imbalanced_batches():
+ labels = np.array([0] * 8 + [1] * 2)
+ indices = np.array(
+ [[1, 8, 9]] + [[0, 8, 9] for _ in range(7)] + [[0, 1, 9], [0, 1, 8]]
+ )
+ distances = np.ones_like(indices, dtype=np.float64)
+ metadata = pd.DataFrame({"batch": labels})
+ per_cell = compute_lisi(
+ distances,
+ indices,
+ metadata,
+ ["batch"],
+ perplexity=1,
+ )[:, 0]
+
+ assert lisi_batch_mixing_score(per_cell, labels) == pytest.approx(1)
+ assert ilisi_knn(distances, indices, labels, perplexity=1) == pytest.approx(0.8)
+
+
+def _materialized_graph_connectivity(
+ edges: np.ndarray,
+ labels: np.ndarray,
+) -> float:
+ n_cells = len(labels)
+ graph = csr_matrix(
+ (
+ np.ones(len(edges)),
+ (edges[:, 0], edges[:, 1]),
+ ),
+ shape=(n_cells, n_cells),
+ )
+ symmetric = graph + graph.T - graph.multiply(graph.T)
+ scores = []
+ for label in np.unique(labels):
+ mask = labels == label
+ subgraph = symmetric[mask][:, mask]
+ n_components, component_labels = connected_components(
+ subgraph,
+ directed=False,
+ )
+ sizes = np.bincount(component_labels, minlength=n_components)
+ scores.append(sizes.max() / sizes.sum())
+ return float(np.mean(scores))
+
+
+def test_graph_connectivity_matches_materialized_symmetric_reference():
+ labels = np.array(["a", "a", "a", "b", "b", "c"])
+ edges = np.array([[0, 1], [1, 0], [3, 4]], dtype=np.int64)
+
+ expected = _materialized_graph_connectivity(edges, labels)
+
+ assert expected == pytest.approx((2 / 3 + 1 + 1) / 3)
+ assert graph_connectivity(edges, labels, batch_rows=1) == pytest.approx(expected)
+ assert graph_connectivity(
+ np.empty((0, 2), dtype=np.int64),
+ np.array(["isolated", "isolated", "isolated"]),
+ ) == pytest.approx(1 / 3)
+
+
+def test_graph_connectivity_matches_chunked_zarr_and_chunk_sizes():
+ labels = np.array([0, 0, 0, 1, 1])
+ edges = np.array(
+ [[0, 1], [1, 0], [1, 2], [3, 4], [4, 3]],
+ dtype=np.uint64,
+ )
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ z_edges = root.create_array("edges", data=edges, chunks=(2, 2))
+ expected = graph_connectivity(edges, labels, batch_rows=len(edges))
+
+ assert graph_connectivity(edges, labels, batch_rows=1) == pytest.approx(expected)
+ assert graph_connectivity(z_edges, labels, batch_rows=3) == pytest.approx(expected)
+
+
+def test_graph_connectivity_validates_inputs():
+ labels = np.array([0, 0])
+
+ with pytest.raises(TypeError, match="integers"):
+ graph_connectivity(np.array([[0.0, 1.0]]), labels)
+ with pytest.raises(IndexError, match="outside"):
+ graph_connectivity(np.array([[0, 2]]), labels)
+ with pytest.raises(ValueError, match="shape"):
+ graph_connectivity(np.array([0, 1]), labels)
+ with pytest.raises(ValueError, match="greater than zero"):
+ graph_connectivity(np.array([[0, 1]]), labels, batch_rows=0)
+ with pytest.raises(ValueError, match="at least one cell"):
+ graph_connectivity(np.empty((0, 2), dtype=np.int64), np.array([]))
+ with pytest.raises(ValueError, match="missing values"):
+ graph_connectivity(
+ np.array([[0, 1]]),
+ np.array([0, np.nan]),
+ )
+
+
+def test_metric_lisi_single_category_is_one(datastore, graph_artifacts):
+ labels = np.zeros(datastore.cells.N, dtype=np.int8)
+ datastore.cells.insert(
+ column_name="single_batch",
+ values=labels,
+ overwrite=True,
+ )
+ lisi = datastore.metric_lisi(
+ label_columns=["single_batch"],
+ )
+
+ assert np.allclose(lisi["single_batch"], 1)
+
+
+def test_knn_without_affinities_preserves_distances():
+ indices = np.array([[1, 2], [0, 2], [0, 1]])
+ distances = np.array([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
+
+ graph = knn_to_csr_matrix(indices, distances)
+
+ assert np.array_equal(
+ graph.toarray(),
+ np.array(
+ [
+ [0.0, 1.0, 2.0],
+ [3.0, 0.0, 4.0],
+ [5.0, 6.0, 0.0],
+ ]
+ ),
+ )
+
+
+def test_knn_affinities_decrease_with_distance():
+ indices = np.array([[1, 2], [0, 2], [3, 0], [2, 1]])
+ distances = np.array([[0.1, 10.0], [0.1, 3.0], [0.2, 4.0], [0.2, 5.0]])
+
+ graph = knn_to_csr_matrix(indices, distances, use_affinities=True)
+
+ assert graph[0, 1] > graph[0, 2]
+
+
+def test_cluster_similarity_is_symmetric_with_unit_diagonal():
+ graph = csr_matrix(
+ np.array(
+ [
+ [0.0, 1.0, 0.2, 0.0],
+ [1.0, 0.0, 0.3, 0.0],
+ [0.2, 0.3, 0.0, 1.0],
+ [0.0, 0.0, 1.0, 0.0],
+ ]
+ )
+ )
+ labels = np.array([0, 0, 1, 1])
+
+ similarities = calculate_weighted_cluster_similarity(graph, labels)
+
+ assert np.allclose(similarities, similarities.T)
+ assert np.allclose(np.diag(similarities), 1)
+ assert similarities[0, 1] > 0
+
+
+def test_streamed_knn_similarity_matches_csr_similarity():
+ indices = np.array([[1, 2], [0, 2], [3, 0], [2, 1]])
+ distances = np.array([[0.1, 10.0], [0.1, 3.0], [0.2, 4.0], [0.2, 5.0]])
+ labels = np.array([0, 0, 1, 1])
+ graph = knn_to_csr_matrix(indices, distances, use_affinities=True)
+
+ streamed = calculate_knn_cluster_similarity(
+ indices,
+ distances,
+ labels,
+ batch_rows=2,
+ )
+ materialized = calculate_weighted_cluster_similarity(graph, labels)
+
+ assert np.allclose(streamed, materialized)
+
+
+def test_top_k_distances_accept_all_candidates():
+ distances = calculate_top_k_neighbor_distances(
+ np.array([[0.0]]),
+ np.array([[1.0], [2.0]]),
+ k=2,
+ )
+
+ assert np.allclose(np.sort(distances[0]), [1.0, 2.0])
+
+
+def test_top_k_cosine_distances_handle_opposite_and_zero_vectors():
+ distances = calculate_top_k_neighbor_distances(
+ np.array([[1.0, 0.0], [0.0, 0.0]]),
+ np.array([[1.0, 0.0], [0.0, 1.0], [-1.0, 0.0]]),
+ k=10,
+ metric="cosine",
+ )
+
+ assert np.allclose(
+ np.sort(distances, axis=1),
+ np.array(
+ [
+ [0.0, 1.0, 2.0],
+ [1.0, 1.0, 1.0],
+ ]
+ ),
+ )
+
+
+def test_top_k_inner_product_distances_are_clipped_at_zero():
+ distances = calculate_top_k_neighbor_distances(
+ np.array([[2.0, 0.0], [0.5, 0.0]]),
+ np.array([[1.0, 0.0], [0.25, 0.0], [-1.0, 0.0]]),
+ k=3,
+ metric="ip",
+ )
+
+ assert np.allclose(
+ np.sort(distances, axis=1),
+ np.array(
+ [
+ [0.0, 0.5, 3.0],
+ [0.5, 0.875, 1.5],
+ ]
+ ),
+ )
+
+
+def test_top_k_distances_reject_unsupported_metric():
+ with pytest.raises(ValueError, match="Unsupported neighbor metric"):
+ calculate_top_k_neighbor_distances(
+ np.array([[0.0]]),
+ np.array([[1.0]]),
+ k=1,
+ metric="manhattan",
+ )
+
+
+def test_metric_silhouette(datastore, graph_artifacts, leiden_clustering):
+ scores = datastore.metric_graph_silhouette(random_seed=42)
+ scores_from_full_label = datastore.metric_graph_silhouette(
+ res_label="RNA_leiden_cluster",
+ random_seed=42,
+ )
+
+ assert scores is not None
+ assert scores_from_full_label is not None
+ assert np.array_equal(scores, scores_from_full_label, equal_nan=True)
+ assert np.isfinite(scores).any()
+ assert np.all(np.abs(scores[np.isfinite(scores)]) <= 1)
+
+
+def test_small_cluster_does_not_invalidate_other_silhouette_scores():
+ class Cells:
+ columns = ["RNA_subset_cluster"]
+
+ @staticmethod
+ def fetch(column, key="I"):
+ assert column == "RNA_subset_cluster"
+ assert key == "subset"
+ return np.array([1, 2, 2, 2, 2, 3, 3, 3, 3])
+
+ class Store:
+ cells = Cells()
+
+ class Ann:
+ annMetric = "l2"
+
+ @staticmethod
+ def reducer(values):
+ return values
+
+ data = np.array(
+ [
+ [20.0, 20.0],
+ [0.0, 0.0],
+ [0.0, 0.1],
+ [0.1, 0.0],
+ [0.1, 0.1],
+ [10.0, 10.0],
+ [10.0, 10.1],
+ [10.1, 10.0],
+ [10.1, 10.1],
+ ]
+ )
+ graph = csr_matrix(np.ones((len(data), len(data))) - np.eye(len(data)))
+
+ scores = silhouette_scoring(
+ Store(),
+ Ann(),
+ graph,
+ data,
+ "RNA",
+ "cluster",
+ cell_key="subset",
+ sample_size=2,
+ random_seed=42,
+ )
+
+ assert scores is not None
+ assert np.isnan(scores[0])
+ assert np.isfinite(scores[1:]).all()
+
+
+@pytest.mark.parametrize("metric", ["ari", "nmi"])
+def test_label_concordance_identical_partitions(metric):
+ labels = np.array([0, 0, 1, 1])
+
+ assert label_concordance_score([labels, labels], metric) == pytest.approx(1)
+
+
+@pytest.mark.parametrize(
+ ("metric", "expected"),
+ [
+ ("ari", 0.125),
+ ("nmi", 0.18872187554086706),
+ ],
+)
+def test_label_concordance_partial_agreement(metric, expected):
+ first = np.array([0, 0, 0, 0, 1, 1, 1, 1])
+ second = np.array([0, 0, 0, 1, 1, 1, 1, 0])
+
+ assert label_concordance_score([first, second], metric) == pytest.approx(expected)
+
+
+def test_label_concordance_requires_exactly_two_partitions():
+ with pytest.raises(ValueError, match="Exactly two"):
+ label_concordance_score([np.array([0, 1])])
+
+
+def test_lisi_batch_mixing_score():
+ labels = np.array([0, 0, 1, 1])
+
+ assert lisi_batch_mixing_score(np.ones(4), labels) == pytest.approx(0)
+ assert lisi_batch_mixing_score(np.full(4, 2.0), labels) == pytest.approx(1)
+
+
+def test_metric_label_concordance(datastore, graph_artifacts, leiden_clustering):
+ rng = np.random.default_rng(42)
+ labels1 = rng.integers(0, 2, datastore.cells.N)
+ labels2 = labels1.copy()
+ selected = np.zeros(datastore.cells.N, dtype=bool)
+ selected[: datastore.cells.N // 2] = True
+ labels2[~selected] = 1 - labels2[~selected]
+ datastore.cells.insert(
+ column_name="labels1",
+ values=labels1,
+ overwrite=True,
+ )
+ datastore.cells.insert(
+ column_name="labels2",
+ values=labels2,
+ overwrite=True,
+ )
+ datastore.cells.insert(
+ column_name="metric_subset",
+ values=selected,
+ overwrite=True,
+ )
+
+ assert datastore.metric_label_concordance(
+ ["labels1", "labels2"],
+ cell_key="metric_subset",
+ ) == pytest.approx(1)
+ assert (
+ datastore.metric_label_concordance(
+ ["labels1", "labels2"],
+ cell_key="I",
+ )
+ < 1
+ )
+ mixing_score = datastore.metric_proportional_batch_mixing("labels1")
+ assert 0 <= mixing_score <= 1
+
+
+def test_datastore_scib_metrics(datastore, graph_artifacts, leiden_clustering):
+ label_colname = "RNA_leiden_cluster"
+
+ ilisi = datastore.metric_ilisi(label_colname)
+ clisi = datastore.metric_clisi(label_colname)
+ latest_connectivity = datastore.metric_graph_connectivity(
+ label_colname,
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ )
+ graph_loc = datastore.get_latest_graph_loc(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ )
+ explicit_connectivity = datastore.metric_graph_connectivity(
+ label_colname,
+ graph_loc=graph_loc,
+ )
+
+ assert 0 <= ilisi <= 1
+ assert 0 <= clisi <= 1
+ assert 0 <= latest_connectivity <= 1
+ assert explicit_connectivity == pytest.approx(latest_connectivity)
+ with pytest.raises(ValueError, match="cell-key provenance"):
+ datastore.metric_graph_connectivity(
+ label_colname,
+ graph_loc=f"{datastore._integratedGraphsLoc}/test",
+ )
+
+
+def test_silhouette_scoring_missing_cluster_labels(datastore):
+ result = silhouette_scoring(
+ datastore,
+ ann_obj=None,
+ graph=None,
+ hvg_data=None,
+ assay_type="RNA",
+ res_label="missing_resolution_label",
+ )
+ assert result is None
diff --git a/tests/test_misc.py b/tests/test_misc.py
new file mode 100644
index 00000000..c416a090
--- /dev/null
+++ b/tests/test_misc.py
@@ -0,0 +1,31 @@
+import inspect
+
+import numpy as np
+from scipy.io import mmread
+from scipy.sparse import csr_matrix
+
+
+def test_export_knn_to_mtx(datastore, graph_artifacts, tmp_path):
+ from scarf.embeddings.sgtsne import export_knn_to_mtx
+
+ graph = datastore.load_graph(
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ symmetric=False,
+ upper_only=False,
+ )
+ fn = str(tmp_path / "test_export_mtx_from_graph.mtx")
+ ret_val = export_knn_to_mtx(fn, graph)
+ assert ret_val is None
+
+ read_options = (
+ {"spmatrix": True} if "spmatrix" in inspect.signature(mmread).parameters else {}
+ )
+ actual = csr_matrix(mmread(fn, **read_options))
+ expected = graph.tocsr(copy=True)
+ actual.sort_indices()
+ expected.sort_indices()
+ np.testing.assert_array_equal(actual.indptr, expected.indptr)
+ np.testing.assert_array_equal(actual.indices, expected.indices)
+ np.testing.assert_allclose(actual.data, expected.data, rtol=1e-7, atol=1e-8)
diff --git a/tests/test_mtx_import.py b/tests/test_mtx_import.py
new file mode 100644
index 00000000..064ff282
--- /dev/null
+++ b/tests/test_mtx_import.py
@@ -0,0 +1,658 @@
+import gzip
+import stat
+import warnings
+import zipfile
+from pathlib import Path
+
+import h5py
+import numpy as np
+import pytest
+import zarr
+from scipy.sparse import csr_matrix
+from zarr.storage import MemoryStore
+
+from scarf import DataStore
+from scarf.readers import CrH5Reader, MtxCandidate, MtxReader, inspect_mtx
+from scarf.writers import CrToZarr, MtxToZarr
+
+
+def _write_text(path: Path, text: str) -> None:
+ if path.name.endswith(".gz"):
+ with gzip.open(path, mode="wt") as handle:
+ handle.write(text)
+ else:
+ path.write_text(text)
+
+
+def _write_mex(
+ path: Path,
+ coordinates: list[tuple[int, int, int]],
+ *,
+ n_features: int = 4,
+ n_cells: int = 4,
+ matrix_name: str = "matrix.mtx",
+ prefix: str = "",
+ compressed_sidecars: bool = False,
+ feature_types: list[str] | None = None,
+) -> None:
+ matrix_path = path / matrix_name
+ rows = "\n".join(f"{row} {column} {value}" for row, column, value in coordinates)
+ _write_text(
+ matrix_path,
+ "%%MatrixMarket matrix coordinate integer general\n"
+ "% fixture\n"
+ f"{n_features} {n_cells} {len(coordinates)}\n"
+ f"{rows}\n",
+ )
+ suffix = ".gz" if compressed_sidecars else ""
+ types = feature_types or ["Gene Expression"] * n_features
+ _write_text(
+ path / f"{prefix}features.tsv{suffix}",
+ "".join(
+ f"feature-{index}\tgene-{index}\t{types[index]}\n"
+ for index in range(n_features)
+ ),
+ )
+ _write_text(
+ path / f"{prefix}barcodes.tsv{suffix}",
+ "".join(f"cell-{index}\n" for index in range(n_cells)),
+ )
+
+
+@pytest.mark.parametrize(
+ ("matrix_name", "compressed_sidecars"),
+ [("matrix.mtx", True), ("matrix.mtx.gz", False)],
+)
+def test_inspect_and_stream_canonical_mixed_compression(
+ tmp_path: Path,
+ matrix_name: str,
+ compressed_sidecars: bool,
+) -> None:
+ coordinates = [(1, 1, 2), (3, 1, 4), (2, 2, 3), (4, 4, 5)]
+ _write_mex(
+ tmp_path,
+ coordinates,
+ matrix_name=matrix_name,
+ compressed_sidecars=compressed_sidecars,
+ )
+
+ candidate = inspect_mtx(tmp_path)[0]
+ assert isinstance(candidate, MtxCandidate)
+ assert candidate.matrixOrientation == "featuresByCells"
+ reader = MtxReader(candidate)
+ explicit = MtxReader(
+ candidate.matrixPath,
+ candidate.featurePath,
+ candidate.cellPath,
+ )
+ try:
+ assert reader.coordinateOrder == "cellMajor"
+ observed = np.vstack(
+ [batch.toarray() for batch in reader.consume(2, lines_in_mem=2)]
+ )
+ expected = np.vstack(
+ [batch.toarray() for batch in explicit.consume(3, lines_in_mem=3)]
+ )
+ finally:
+ reader.close()
+ explicit.close()
+ np.testing.assert_array_equal(observed, expected)
+
+
+def test_filtered_cell_major_stream_does_not_prescan_matrix(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ _write_mex(
+ tmp_path,
+ [(1, 1, 1), (3, 1, 2), (2, 2, 3), (4, 3, 4)],
+ n_cells=3,
+ )
+
+ def unexpected_scan(*args, **kwargs):
+ raise AssertionError("filtered cell-major input must not be prescanned")
+
+ monkeypatch.setattr(
+ "scarf.readers.mtx._MtxEngine._scan_matrix",
+ unexpected_scan,
+ )
+ reader = MtxReader(inspect_mtx(tmp_path)[0])
+ try:
+ observed = np.vstack(
+ [batch.toarray() for batch in reader.consume(2, lines_in_mem=2)]
+ )
+ finally:
+ reader.close()
+ np.testing.assert_array_equal(
+ observed,
+ [[1, 0, 2, 0], [0, 3, 0, 0], [0, 0, 0, 4]],
+ )
+
+
+def test_matrix_market_count_dtype_overflow_requires_explicit_widening(
+ tmp_path: Path,
+) -> None:
+ value = int(np.iinfo(np.uint32).max) + 1
+ _write_mex(
+ tmp_path,
+ [(1, 1, value)],
+ n_features=1,
+ n_cells=1,
+ )
+ candidate = inspect_mtx(tmp_path)[0]
+
+ default_reader = MtxReader(candidate)
+ try:
+ with pytest.raises(
+ OverflowError,
+ match="Matrix Market count exceeds dtype uint32",
+ ):
+ list(default_reader.consume(1))
+ finally:
+ default_reader.close()
+
+ wide_reader = MtxReader(candidate, dtype=np.uint64)
+ try:
+ observed = list(wide_reader.consume(1))[0].toarray()
+ finally:
+ wide_reader.close()
+ np.testing.assert_array_equal(observed, np.array([[value]], dtype=np.uint64))
+
+
+def test_feature_major_disk_csr_parity_filtering_and_cleanup(tmp_path: Path) -> None:
+ feature_major = tmp_path / "feature-major"
+ cell_major = tmp_path / "cell-major"
+ feature_major.mkdir()
+ cell_major.mkdir()
+ feature_coordinates = [
+ (1, 1, 1),
+ (1, 3, 2),
+ (2, 2, 1),
+ (2, 2, 2),
+ (3, 1, 4),
+ (4, 4, 1),
+ ]
+ cell_coordinates = [
+ (1, 1, 1),
+ (3, 1, 4),
+ (2, 2, 3),
+ (1, 3, 2),
+ (4, 4, 1),
+ ]
+ _write_mex(feature_major, feature_coordinates)
+ _write_mex(cell_major, cell_coordinates)
+
+ feature_reader = MtxReader(
+ inspect_mtx(feature_major)[0],
+ is_filtered=False,
+ filtering_cutoff=1,
+ temp_dir=str(tmp_path),
+ )
+ cell_reader = MtxReader(
+ inspect_mtx(cell_major)[0],
+ is_filtered=False,
+ filtering_cutoff=1,
+ )
+ assert feature_reader.coordinateOrder == "featureMajor"
+ assert feature_reader.cell_names() == ["cell-0", "cell-1", "cell-2"]
+ feature_reader._prepare_sparse_import()
+ assert feature_reader.temporaryDiskBytes == 64
+ prepared = list(feature_reader.consume(2, lines_in_mem=2))
+ assert all(batch.tocsr().has_sorted_indices for batch in prepared)
+ widened = list(feature_reader.consume(2, lines_in_mem=2, dtype=np.uint64))
+ assert all(batch.dtype == np.dtype(np.uint64) for batch in widened)
+ np.testing.assert_array_equal(
+ np.vstack([batch.toarray() for batch in widened]),
+ np.vstack([batch.toarray() for batch in prepared]),
+ )
+
+ stores = [MemoryStore(), MemoryStore()]
+ MtxToZarr(
+ feature_reader,
+ stores[0],
+ mem_budget="64M",
+ targetChunkBytes=16,
+ targetShardBytes=32,
+ ).dump(lines_in_mem=2)
+ MtxToZarr(
+ cell_reader,
+ stores[1],
+ mem_budget="64M",
+ targetChunkBytes=16,
+ targetShardBytes=32,
+ ).dump(lines_in_mem=2)
+
+ first = zarr.open_group(store=stores[0], mode="r")["RNA/counts"][:]
+ second = zarr.open_group(store=stores[1], mode="r")["RNA/counts"][:]
+ np.testing.assert_array_equal(first, second)
+ assert not list(tmp_path.glob("scarf-mtx-csr-*"))
+
+
+def _write_parse(path: Path, *, both_aliases: bool = False) -> None:
+ _write_text(
+ path / "DGE.mtx.gz",
+ "%%MatrixMarket matrix coordinate integer general\n"
+ "3 2 5\n"
+ "1 1 1\n"
+ "1 2 2\n"
+ "2 1 3\n"
+ "3 2 1\n"
+ "3 2 4\n",
+ )
+ (path / "genes.csv").write_text(
+ "gene_id,gene_name\nfeature-0,Gene 0\nfeature-1,Gene 1\n"
+ )
+ header = (
+ "bc_wells,bc_index,sample,quality"
+ if both_aliases
+ else ("bc_wells,sample,quality")
+ )
+ rows = (
+ "cell-0,index-0,A,1\ncell-1,index-1,A,2\ncell-2,index-2,B,3\n"
+ if both_aliases
+ else "cell-0,A,1\ncell-1,A,2\ncell-2,B,3\n"
+ )
+ (path / "cell_metadata.csv").write_text(f"{header}\n{rows}")
+
+
+def test_parse_orientation_duplicate_sum_and_metadata(tmp_path: Path) -> None:
+ _write_parse(tmp_path)
+ candidate = inspect_mtx(tmp_path)[0]
+ assert candidate.matrixOrientation == "cellsByFeatures"
+ assert candidate.cellIdKeys == ("bc_wells",)
+ reader = MtxReader(candidate)
+ store = MemoryStore()
+ MtxToZarr(
+ reader,
+ store,
+ mem_budget="64M",
+ targetChunkBytes=8,
+ targetShardBytes=16,
+ ).dump(lines_in_mem=2)
+
+ root = zarr.open_group(store=store, mode="r")
+ np.testing.assert_array_equal(
+ root["RNA/counts"][:],
+ [[1, 2], [3, 0], [0, 5]],
+ )
+ np.testing.assert_array_equal(root["cellData/sample"][:], ["A", "A", "B"])
+ np.testing.assert_array_equal(root["cellData/quality"][:], [1, 2, 3])
+
+
+def test_parse_requires_explicit_ambiguous_cell_id_key(tmp_path: Path) -> None:
+ _write_parse(tmp_path, both_aliases=True)
+ candidate = inspect_mtx(tmp_path)[0]
+ assert candidate.cellIdKeys == ("bc_wells", "bc_index")
+ with pytest.raises(ValueError, match="ambiguous"):
+ MtxReader(candidate)
+ reader = MtxReader(candidate, cell_id_key="bc_index")
+ try:
+ assert reader.cell_names() == ["index-0", "index-1", "index-2"]
+ assert "bc_wells" in dict(reader.get_cell_columns())
+ finally:
+ reader.close()
+
+
+def test_prefixed_dragen_triplet_and_ambiguous_candidates(tmp_path: Path) -> None:
+ coordinates = [(1, 1, 1)]
+ _write_mex(
+ tmp_path,
+ coordinates,
+ n_features=1,
+ n_cells=1,
+ matrix_name="sample_matrix.mtx.gz",
+ prefix="sample_",
+ )
+ other = tmp_path / "other"
+ other.mkdir()
+ _write_mex(other, coordinates, n_features=1, n_cells=1)
+
+ candidate = inspect_mtx(tmp_path)[0]
+ assert Path(candidate.featurePath).name == "sample_features.tsv"
+ assert Path(candidate.cellPath).name == "sample_barcodes.tsv"
+
+ _write_mex(
+ tmp_path,
+ coordinates,
+ n_features=1,
+ n_cells=1,
+ matrix_name="second_matrix.mtx",
+ prefix="second_",
+ )
+ assert len(inspect_mtx(tmp_path)) == 2
+ with pytest.raises(ValueError, match="requires one"):
+ from scarf.readers import CrDirReader
+
+ CrDirReader(str(tmp_path))
+
+
+def _write_zip_mex(path: Path) -> None:
+ with zipfile.ZipFile(path, mode="w") as archive:
+ archive.writestr(
+ "bundle/sample_matrix.mtx",
+ "%%MatrixMarket matrix coordinate integer general\n2 2 2\n1 1 1\n2 2 2\n",
+ )
+ archive.writestr(
+ "bundle/sample_features.tsv",
+ "feature-0\tGene 0\nfeature-1\tGene 1\n",
+ )
+ archive.writestr(
+ "bundle/sample_barcodes.tsv",
+ "cell-0\ncell-1\n",
+ )
+ archive.writestr(
+ "bundle/protospacer_calls_per_cell.csv",
+ "cell_id,feature_call\ncell-0,feature-0\n",
+ )
+
+
+def test_direct_mex_zip_and_archive_safety(tmp_path: Path) -> None:
+ archive_path = tmp_path / "sample_MEX.zip"
+ _write_zip_mex(archive_path)
+ candidate = inspect_mtx(archive_path)[0]
+ assert candidate.relatedFiles == ("bundle/protospacer_calls_per_cell.csv",)
+ reader = MtxReader(candidate, temp_dir=str(tmp_path))
+ try:
+ assert candidate.relatedFiles[0] not in reader._engine._archivePaths
+ observed = np.vstack([batch.toarray() for batch in reader.consume(1)])
+ finally:
+ reader.close()
+ np.testing.assert_array_equal(observed, np.eye(2, dtype=np.uint32) * [1, 2])
+ assert not list(tmp_path.glob("scarf-mtx-archive-*"))
+
+ traversal = tmp_path / "traversal.zip"
+ with zipfile.ZipFile(traversal, mode="w") as archive:
+ archive.writestr("../matrix.mtx", "unsafe")
+ with pytest.raises(ValueError, match="Unsafe"):
+ inspect_mtx(traversal)
+
+ linked = tmp_path / "linked.zip"
+ info = zipfile.ZipInfo("matrix.mtx")
+ info.create_system = 3
+ info.external_attr = (stat.S_IFLNK | 0o777) << 16
+ with zipfile.ZipFile(linked, mode="w") as archive:
+ archive.writestr(info, "target")
+ with pytest.raises(ValueError, match="links"):
+ inspect_mtx(linked)
+
+ duplicate = tmp_path / "duplicate.zip"
+ with warnings.catch_warnings():
+ warnings.simplefilter("ignore")
+ with zipfile.ZipFile(duplicate, mode="w") as archive:
+ archive.writestr("matrix.mtx", "one")
+ archive.writestr("matrix.mtx", "two")
+ with pytest.raises(ValueError, match="Duplicate"):
+ inspect_mtx(duplicate)
+
+
+@pytest.mark.parametrize(
+ ("coordinates", "message"),
+ [
+ ([(1, 1, 1), (2, 2, 1), (1, 1, 1)], "neither cell-major"),
+ ([(0, 1, 1)], "outside the declared"),
+ ([(1, 1, -1)], "non-negative"),
+ ],
+)
+def test_invalid_matrix_market_coordinates(
+ tmp_path: Path,
+ coordinates: list[tuple[int, int, int]],
+ message: str,
+) -> None:
+ _write_mex(
+ tmp_path,
+ coordinates,
+ n_features=2,
+ n_cells=2,
+ )
+ with pytest.raises(ValueError, match=message):
+ MtxReader(inspect_mtx(tmp_path)[0])
+
+
+def test_duplicate_ids_and_dimension_mismatch_are_rejected(tmp_path: Path) -> None:
+ _write_mex(tmp_path, [(1, 1, 1)], n_features=1, n_cells=1)
+ (tmp_path / "barcodes.tsv").write_text("cell-0\ncell-0\n")
+ with pytest.raises(ValueError, match="do not match"):
+ inspect_mtx(tmp_path)
+
+ (tmp_path / "barcodes.tsv").write_text("cell-0\n")
+ (tmp_path / "features.tsv").write_text("feature-0\tGene 0\nfeature-0\tGene 1\n")
+ _write_text(
+ tmp_path / "matrix.mtx",
+ "%%MatrixMarket matrix coordinate integer general\n2 1 1\n1 1 1\n",
+ )
+ with pytest.raises(ValueError, match="unique"):
+ MtxReader(inspect_mtx(tmp_path)[0])
+
+
+def test_feature_major_cleanup_after_planning_and_write_failures(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ _write_mex(
+ tmp_path,
+ [(1, 1, 1), (2, 2, 2), (3, 1, 3)],
+ n_features=3,
+ n_cells=2,
+ )
+ candidate = inspect_mtx(tmp_path)[0]
+
+ planning_reader = MtxReader(candidate, temp_dir=str(tmp_path))
+ writer = MtxToZarr(
+ planning_reader,
+ MemoryStore(),
+ mem_budget=1,
+ targetChunkBytes=8,
+ targetShardBytes=8,
+ )
+ with pytest.raises(MemoryError, match="one source row"):
+ writer.dump(lines_in_mem=1)
+ assert not list(tmp_path.glob("scarf-mtx-csr-*"))
+
+ write_reader = MtxReader(candidate, temp_dir=str(tmp_path))
+
+ def fail_after_first(writes, **kwargs):
+ source = iter(writes)
+ next(source)
+ close = getattr(source, "close", None)
+ if callable(close):
+ close()
+ raise RuntimeError("injected cancellation")
+
+ monkeypatch.setattr(
+ "scarf.storage.sharding.write_sparse_bands",
+ fail_after_first,
+ )
+ writer = MtxToZarr(
+ write_reader,
+ MemoryStore(),
+ mem_budget="64M",
+ targetChunkBytes=8,
+ targetShardBytes=8,
+ )
+ with pytest.raises(RuntimeError, match="injected cancellation"):
+ writer.dump(lines_in_mem=1)
+ assert not list(tmp_path.glob("scarf-mtx-csr-*"))
+
+
+def test_feature_major_consume_uses_parse_budget_and_cleans_up_on_close(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ _write_mex(
+ tmp_path,
+ [(1, 1, 1), (1, 3, 2), (2, 2, 3), (3, 1, 4)],
+ n_features=3,
+ n_cells=3,
+ )
+ reader = MtxReader(inspect_mtx(tmp_path)[0], temp_dir=str(tmp_path))
+ observed_lines: list[int] = []
+ original = reader._engine._coalesced_chunks
+
+ def recording_chunks(lines_in_mem: int):
+ observed_lines.append(lines_in_mem)
+ yield from original(lines_in_mem)
+
+ monkeypatch.setattr(reader._engine, "_coalesced_chunks", recording_chunks)
+ source = reader.consume(1, lines_in_mem=2)
+ next(source)
+ assert observed_lines == [2]
+ assert list(tmp_path.glob("scarf-mtx-csr-*"))
+ source.close()
+ assert not list(tmp_path.glob("scarf-mtx-csr-*"))
+ completed = list(reader.consume(2, lines_in_mem=2))
+ assert not list(tmp_path.glob("scarf-mtx-csr-*"))
+ assert np.vstack([batch.toarray() for batch in completed]).sum() == 10
+
+
+def test_multimodal_mex_names_feature_reference_and_related_files(
+ tmp_path: Path,
+) -> None:
+ types = [
+ "mRNA",
+ "CRISPR Guide Capture",
+ "Multiplexing Capture",
+ "Antigen Capture",
+ "Custom",
+ "AbSeq",
+ ]
+ _write_mex(
+ tmp_path,
+ [(index, 1, index) for index in range(1, 7)],
+ n_features=6,
+ n_cells=1,
+ feature_types=types,
+ )
+ (tmp_path / "feature_reference.csv").write_text(
+ "id,sequence,pattern,read,target_gene_id\n"
+ + "".join(
+ f"feature-{index},SEQ{index},PAT{index},R2,target-{index}\n"
+ for index in range(6)
+ )
+ )
+ (tmp_path / "protospacer_calls_per_cell.csv").write_text(
+ "cell_id,feature_call\ncell-0,feature-1\n"
+ )
+ candidate = inspect_mtx(tmp_path)[0]
+ assert candidate.relatedFiles == (str(tmp_path / "protospacer_calls_per_cell.csv"),)
+ reader = MtxReader(candidate)
+ store = MemoryStore()
+ MtxToZarr(
+ reader,
+ store,
+ mem_budget="64M",
+ targetChunkBytes=8,
+ targetShardBytes=8,
+ ).dump(lines_in_mem=2)
+
+ root = zarr.open_group(store=store, mode="r")
+ assert set(root.group_keys()) == {
+ "RNA",
+ "CRISPR",
+ "HTO",
+ "ANTIGEN",
+ "CUSTOM",
+ "ADT",
+ "cellData",
+ }
+ np.testing.assert_array_equal(
+ root["CRISPR/featureData/sequence"][:],
+ ["SEQ1"],
+ )
+ assert "feature_call" not in root["cellData"]
+ datastore = DataStore(
+ store,
+ default_assay="RNA",
+ min_features_per_cell=0,
+ min_cells_per_feature=0,
+ mem_budget="64M",
+ nthreads=1,
+ )
+ summary = datastore.summary()
+ assert {assay.name: assay.assay_type for assay in summary.assays} == {
+ "ADT": "ADT",
+ "ANTIGEN": "ANTIGEN",
+ "CRISPR": "CRISPR",
+ "CUSTOM": "CUSTOM",
+ "HTO": "HTO",
+ "RNA": "RNA",
+ }
+
+
+def test_bd_guide_reclassification_is_explicit(tmp_path: Path) -> None:
+ _write_mex(
+ tmp_path,
+ [(1, 1, 1), (2, 1, 2)],
+ n_features=2,
+ n_cells=1,
+ feature_types=["mRNA", "mRNA"],
+ )
+ reader = MtxReader(inspect_mtx(tmp_path)[0])
+ reader.reclassify_features(
+ [1],
+ "CRISPR Guide Capture",
+ require_previous="mRNA",
+ )
+ assert tuple(reader.assayFeats.columns) == ("RNA", "CRISPR")
+
+
+def _write_tagged_h5(path: Path) -> None:
+ values = csr_matrix(np.array([[1, 0], [0, 2]], dtype=np.uint16))
+ with h5py.File(path, mode="w") as handle:
+ group = handle.create_group("matrix")
+ group.create_dataset("data", data=values.data)
+ group.create_dataset("indices", data=values.indices)
+ group.create_dataset("indptr", data=values.indptr)
+ group.create_dataset("barcodes", data=np.array([b"cell-0", b"cell-1"]))
+ features = group.create_group("features")
+ features.create_dataset("id", data=np.array([b"gene-0", b"guide-0"]))
+ features.create_dataset("name", data=np.array([b"Gene 0", b"Guide 0"]))
+ features.create_dataset(
+ "feature_type",
+ data=np.array([b"Gene Expression", b"CRISPR Guide Capture"]),
+ )
+ features.create_dataset(
+ "_all_tag_keys",
+ data=np.array([b"sequence", b"pattern", b"read", b"target_gene_id"]),
+ )
+ features.create_dataset("sequence", data=np.array([b"", b"ACGT"]))
+ features.create_dataset("pattern", data=np.array([b"", b"5P(BC)"]))
+ features.create_dataset("read", data=np.array([b"", b"R2"]))
+ features.create_dataset("target_gene_id", data=np.array([b"", b"gene-0"]))
+
+
+def test_cellranger_h5_preserves_guide_feature_tags(tmp_path: Path) -> None:
+ path = tmp_path / "tagged.h5"
+ _write_tagged_h5(path)
+ reader = CrH5Reader(str(path))
+ store = MemoryStore()
+ try:
+ CrToZarr(
+ reader,
+ store,
+ mem_budget="64M",
+ targetChunkBytes=8,
+ targetShardBytes=8,
+ ).dump()
+ finally:
+ reader.close()
+ feature_data = zarr.open_group(store=store, mode="r")["CRISPR/featureData"]
+ np.testing.assert_array_equal(feature_data["sequence"][:], ["ACGT"])
+ np.testing.assert_array_equal(feature_data["pattern"][:], ["5P(BC)"])
+ np.testing.assert_array_equal(feature_data["read"][:], ["R2"])
+ np.testing.assert_array_equal(feature_data["target_gene_id"][:], ["gene-0"])
+
+
+def test_archive_parse_failure_removes_temporary_extraction(tmp_path: Path) -> None:
+ archive_path = tmp_path / "invalid_MEX.zip"
+ with zipfile.ZipFile(archive_path, mode="w") as archive:
+ archive.writestr(
+ "matrix.mtx",
+ "%%MatrixMarket matrix coordinate integer general\n"
+ "2 2 3\n1 1 1\n2 2 1\n1 1 1\n",
+ )
+ archive.writestr("features.tsv", "f1\tg1\nf2\tg2\n")
+ archive.writestr("barcodes.tsv", "c1\nc2\n")
+ candidate = inspect_mtx(archive_path)[0]
+ with pytest.raises(ValueError, match="neither cell-major"):
+ MtxReader(candidate, temp_dir=str(tmp_path))
+ assert not list(tmp_path.glob("scarf-mtx-archive-*"))
diff --git a/tests/test_neighbor_stages.py b/tests/test_neighbor_stages.py
new file mode 100644
index 00000000..585683ad
--- /dev/null
+++ b/tests/test_neighbor_stages.py
@@ -0,0 +1,501 @@
+import numpy as np
+import pandas as pd
+import pytest
+
+from scarf.embeddings.harmony import HarmonyResult
+from scarf.matrix import ChunkedArray
+from scarf.neighbors.stages import (
+ AnnIndexStage,
+ BatchCorrectionStage,
+ KMeansInitializationStage,
+ LazyTransformStream,
+ NeighborQueryStage,
+ ReductionTransform,
+)
+from scarf.neighbors.stream import AnnStream
+
+
+class _CountingChunkedArray(ChunkedArray):
+ def __init__(self, values: np.ndarray, block_size: int) -> None:
+ super().__init__(
+ values,
+ block_size=block_size,
+ nthreads=1,
+ is_numpy=True,
+ )
+ self.read_count = 0
+
+ def _materialize_range(self, start: int, end: int) -> np.ndarray:
+ self.read_count += 1
+ return super()._materialize_range(start, end)
+
+
+def _custom_inputs() -> tuple[np.ndarray, np.ndarray]:
+ data = np.arange(32, dtype=np.float64).reshape(8, 4) / 10
+ loadings = np.array(
+ [
+ [1.0, 0.0],
+ [0.0, 1.0],
+ [0.5, -0.5],
+ [-0.25, 0.75],
+ ]
+ )
+ return data, loadings
+
+
+def _ann_stream() -> AnnStream:
+ values, loadings = _custom_inputs()
+ return AnnStream(
+ data=ChunkedArray.from_numpy(values, block_size=4, nthreads=1),
+ k=3,
+ n_cluster=3,
+ reduction_method="custom",
+ dims=2,
+ loadings=loadings,
+ use_for_pca=np.ones(values.shape[0], dtype=bool),
+ mu=np.zeros(values.shape[1]),
+ sigma=np.ones(values.shape[1]),
+ ann_metric="l2",
+ ann_efc=50,
+ ann_ef=50,
+ ann_m=16,
+ nthreads=1,
+ ann_parallel=False,
+ rand_state=4466,
+ do_kmeans_fit=True,
+ disable_scaling=True,
+ ann_idx=None,
+ lsi_skip_first=False,
+ lsi_params={},
+ harmonize=False,
+ )
+
+
+def test_reduction_transform_keeps_cell_coordinates_lazy() -> None:
+ values, loadings = _custom_inputs()
+ data = _CountingChunkedArray(values, block_size=3)
+ reduction = ReductionTransform(
+ data=data,
+ method="custom",
+ dims=2,
+ loadings=loadings,
+ use_for_pca=np.ones(values.shape[0], dtype=bool),
+ mu=np.zeros(values.shape[1]),
+ sigma=np.ones(values.shape[1]),
+ batch_size=3,
+ nthreads=1,
+ rand_state=4466,
+ disable_scaling=True,
+ lsi_skip_first=False,
+ lsi_params={},
+ )
+ stream = LazyTransformStream(
+ data=data,
+ transform=reduction.transform,
+ nthreads=1,
+ batch_size=3,
+ )
+
+ assert data.read_count == 0
+ first = next(stream.iter_transformed())
+ np.testing.assert_allclose(first, values[:3].dot(loadings))
+ assert 1 <= data.read_count <= 3
+
+
+def test_lsi_persisted_loadings_are_not_sliced_again() -> None:
+ values, loadings = _custom_inputs()
+ reduction = ReductionTransform(
+ data=ChunkedArray.from_numpy(values, block_size=4, nthreads=1),
+ method="lsi",
+ dims=2,
+ loadings=loadings,
+ use_for_pca=np.ones(values.shape[0], dtype=bool),
+ mu=np.array([], dtype=np.float64),
+ sigma=np.array([], dtype=np.float64),
+ batch_size=4,
+ nthreads=1,
+ rand_state=4466,
+ disable_scaling=True,
+ lsi_skip_first=True,
+ lsi_params={},
+ )
+
+ assert reduction.dims == 2
+ np.testing.assert_allclose(reduction.transform(values), values.dot(loadings))
+
+
+@pytest.mark.parametrize("skip_first", [False, True])
+def test_lsi_dims_are_final_output_dimensions(skip_first: bool) -> None:
+ from scarf.embeddings.reduction import fit_lsi
+
+ values, _loadings = _custom_inputs()
+ loadings = fit_lsi(
+ ChunkedArray.from_numpy(values, block_size=4, nthreads=1),
+ dims=2,
+ skip_first=skip_first,
+ params={},
+ random_state=4466,
+ nthreads=1,
+ )
+
+ assert loadings.shape == (values.shape[1], 2)
+
+
+def test_ann_stream_adapter_preserves_reduction_numerics() -> None:
+ values, loadings = _custom_inputs()
+ stream = _ann_stream()
+ expected = values.dot(loadings)
+
+ np.testing.assert_allclose(stream.transform_query(values), expected)
+ assert stream.clusterLabels.shape == (values.shape[0],)
+ indices, distances = stream.transform_ann(expected, k=3)
+ assert indices.shape == distances.shape == (values.shape[0], 3)
+
+
+def test_lazy_coordinate_stages_do_not_hide_a_cross_stage_cache() -> None:
+ values, loadings = _custom_inputs()
+ data = _CountingChunkedArray(values, block_size=3)
+ stream = LazyTransformStream(
+ data=data,
+ transform=lambda block: block.dot(loadings),
+ nthreads=1,
+ batch_size=3,
+ )
+
+ index = AnnIndexStage.fit(
+ coordinates=stream,
+ metric="l2",
+ dims=2,
+ n_cells=values.shape[0],
+ ef_construction=50,
+ ef=50,
+ m=16,
+ rand_state=4466,
+ ann_threads=1,
+ )
+ initialization = KMeansInitializationStage.fit(
+ stream=stream,
+ n_clusters=3,
+ rand_state=4466,
+ nthreads=1,
+ enabled=True,
+ )
+
+ assert data.read_count == 12
+ assert index.get_current_count() == values.shape[0]
+ query = NeighborQueryStage(index, k=3, metric="l2")
+ indices, distances = query.query(values.dot(loadings), k=3)
+ assert indices.shape == distances.shape == (values.shape[0], 3)
+ assert initialization.model is not None
+ assert initialization.labels.shape == (values.shape[0],)
+
+
+def test_neighbor_query_validates_and_converts_metric_distances() -> None:
+ stage = NeighborQueryStage(index=None, k=2, metric="l2")
+ distances = np.array([[4.0, -1e-7]], dtype=np.float32)
+ np.testing.assert_allclose(
+ stage._metric_distances(distances),
+ [[2.0, 0.0]],
+ )
+
+ cosine = NeighborQueryStage(index=None, k=2, metric="cosine")
+ with pytest.raises(ValueError, match="negative"):
+ cosine._metric_distances(np.array([[0.1, -0.01]], dtype=np.float32))
+ with pytest.raises(ValueError, match="non-finite"):
+ cosine._metric_distances(np.array([[0.1, np.nan]], dtype=np.float32))
+
+
+def test_kmeans_initialization_runs_on_demand_without_ann() -> None:
+ values, loadings = _custom_inputs()
+ data = _CountingChunkedArray(values, block_size=3)
+ stream = LazyTransformStream(
+ data=data,
+ transform=lambda block: block.dot(loadings),
+ nthreads=1,
+ batch_size=3,
+ )
+
+ disabled = KMeansInitializationStage.fit(
+ stream=stream,
+ n_clusters=3,
+ rand_state=4466,
+ nthreads=1,
+ enabled=False,
+ )
+ assert disabled.model is None
+ assert data.read_count == 0
+
+ enabled = KMeansInitializationStage.fit(
+ stream=stream,
+ n_clusters=3,
+ rand_state=4466,
+ nthreads=1,
+ enabled=True,
+ )
+ assert enabled.model is not None
+ assert enabled.labels.shape == (values.shape[0],)
+ assert data.read_count == 9
+
+
+def test_kmeans_initialization_uses_true_minibatches_for_one_full_block(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ values, loadings = _custom_inputs()
+ data = _CountingChunkedArray(values, block_size=values.shape[0])
+ stream = LazyTransformStream(
+ data=data,
+ transform=lambda block: block.dot(loadings),
+ nthreads=1,
+ batch_size=values.shape[0],
+ )
+ closed_progress: list[tuple[int, int]] = []
+
+ class Progress:
+ def __init__(self, total: int) -> None:
+ self.total = total
+ self.value = 0
+
+ def update(self) -> None:
+ self.value += 1
+
+ def close(self) -> None:
+ closed_progress.append((self.value, self.total))
+
+ monkeypatch.setattr(
+ "scarf.utils.progress.tqdmbar",
+ lambda *args, total, **kwargs: Progress(total),
+ )
+
+ result = KMeansInitializationStage.fit(
+ stream=stream,
+ n_clusters=3,
+ rand_state=4466,
+ nthreads=1,
+ enabled=True,
+ kmeans_sampling=0.5,
+ kmeans_batch_size=3,
+ )
+
+ assert result.model is not None
+ assert result.model.n_init == 1
+ assert result.model.init_size == 4
+ assert result.model.batch_size == 3
+ assert result.model.n_steps_ > 1
+ assert result.labels.shape == (values.shape[0],)
+ assert data.read_count == 1
+ assert closed_progress == [(1, 1)]
+
+
+def test_kmeans_streaming_samples_all_blocks_and_coalesces_updates(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ from sklearn.cluster import kmeans_plusplus as sklearn_kmeans_plusplus
+ from sklearn.utils.random import sample_without_replacement
+
+ values, loadings = _custom_inputs()
+ transformed = values.dot(loadings)
+ data = _CountingChunkedArray(values, block_size=2)
+ stream = LazyTransformStream(
+ data=data,
+ transform=lambda block: block.dot(loadings),
+ nthreads=1,
+ batch_size=2,
+ )
+ sampled: list[np.ndarray] = []
+
+ def capture_kmeans_plusplus(
+ values: np.ndarray,
+ *,
+ n_clusters: int,
+ random_state: int,
+ ) -> tuple[np.ndarray, np.ndarray]:
+ sampled.append(values.copy())
+ return sklearn_kmeans_plusplus(
+ values,
+ n_clusters=n_clusters,
+ random_state=random_state,
+ )
+
+ monkeypatch.setattr("sklearn.cluster.kmeans_plusplus", capture_kmeans_plusplus)
+ result = KMeansInitializationStage.fit(
+ stream=stream,
+ n_clusters=3,
+ rand_state=4466,
+ nthreads=1,
+ enabled=True,
+ kmeans_sampling=0.5,
+ kmeans_batch_size=5,
+ )
+
+ expected_indices = np.sort(
+ sample_without_replacement(
+ values.shape[0],
+ 4,
+ method="reservoir_sampling",
+ random_state=4466,
+ )
+ )
+ np.testing.assert_allclose(sampled[0], transformed[expected_indices])
+ assert result.model is not None
+ assert result.model.n_clusters == 3
+ assert result.model.batch_size == 5
+ assert result.model.n_steps_ == 2
+ assert result.labels.shape == (values.shape[0],)
+ assert data.read_count == 12
+
+ other_data = _CountingChunkedArray(values, block_size=3)
+ other_result = KMeansInitializationStage.fit(
+ stream=LazyTransformStream(
+ data=other_data,
+ transform=lambda block: block.dot(loadings),
+ nthreads=1,
+ batch_size=3,
+ ),
+ n_clusters=3,
+ rand_state=4466,
+ nthreads=1,
+ enabled=True,
+ kmeans_sampling=0.5,
+ kmeans_batch_size=5,
+ )
+ assert other_result.model is not None
+ np.testing.assert_allclose(
+ result.model.cluster_centers_,
+ other_result.model.cluster_centers_,
+ )
+ np.testing.assert_array_equal(result.labels, other_result.labels)
+ assert other_data.read_count == 9
+
+
+def test_kmeans_initialization_rejects_single_row_and_empty_inputs() -> None:
+ values, loadings = _custom_inputs()
+ single_data = ChunkedArray.from_numpy(values[:1], block_size=1, nthreads=1)
+ single_stream = LazyTransformStream(
+ data=single_data,
+ transform=lambda block: block.dot(loadings),
+ nthreads=1,
+ batch_size=1,
+ )
+ with pytest.raises(ValueError, match="at least two rows"):
+ KMeansInitializationStage.fit(
+ stream=single_stream,
+ n_clusters=5,
+ rand_state=4466,
+ nthreads=1,
+ enabled=True,
+ )
+
+ empty_stream = LazyTransformStream(
+ data=ChunkedArray.from_numpy(
+ np.empty((0, values.shape[1])),
+ block_size=1,
+ nthreads=1,
+ ),
+ transform=lambda block: block.dot(loadings),
+ nthreads=1,
+ batch_size=1,
+ )
+ with pytest.raises(ValueError, match="at least one row"):
+ KMeansInitializationStage.fit(
+ stream=empty_stream,
+ n_clusters=2,
+ rand_state=4466,
+ nthreads=1,
+ enabled=True,
+ )
+
+
+def test_harmony_stage_materializes_uncorrected_coordinates_once(
+ monkeypatch,
+) -> None:
+ values, loadings = _custom_inputs()
+ data = _CountingChunkedArray(values, block_size=3)
+ stream = LazyTransformStream(
+ data=data,
+ transform=lambda block: block.dot(loadings),
+ nthreads=1,
+ batch_size=3,
+ )
+ corrected_values = values.dot(loadings) + 1.0
+
+ def fake_harmony(
+ uncorrected: np.ndarray,
+ batches: pd.DataFrame,
+ **_parameters,
+ ) -> HarmonyResult:
+ np.testing.assert_allclose(uncorrected, values.dot(loadings).T)
+ assert list(batches.columns) == ["batch"]
+ return HarmonyResult(
+ original=uncorrected,
+ corrected=corrected_values.T,
+ assignments=np.ones((1, values.shape[0])),
+ centroids=np.zeros((1, 2)),
+ sigma=np.ones(1),
+ ridge=np.eye(1),
+ batch_columns=("batch",),
+ batch_levels=(("a", "b"),),
+ parameters={},
+ )
+
+ monkeypatch.setattr("scarf.neighbors.stages.fit_harmony", fake_harmony)
+ stage = BatchCorrectionStage(
+ stream=stream,
+ n_cells=values.shape[0],
+ dims=loadings.shape[1],
+ batch_size=3,
+ batches=pd.DataFrame({"batch": ["a", "b"] * 4}),
+ parameters={},
+ corrected_data=None,
+ nthreads=1,
+ )
+
+ first = stage.ensure_corrected()
+ second = stage.ensure_corrected()
+
+ np.testing.assert_allclose(first.compute(), corrected_values)
+ assert second is first
+ assert data.read_count == 3
+
+
+def test_invalid_ann_configuration_fails_before_harmony_reads(
+ monkeypatch,
+) -> None:
+ values, loadings = _custom_inputs()
+ data = _CountingChunkedArray(values, block_size=3)
+ harmony_called = False
+
+ def fail_if_called(*_args, **_kwargs):
+ nonlocal harmony_called
+ harmony_called = True
+ raise AssertionError("Harmony should not run")
+
+ monkeypatch.setattr("scarf.neighbors.stages.fit_harmony", fail_if_called)
+ with pytest.raises(RuntimeError):
+ AnnStream(
+ data=data,
+ k=3,
+ n_cluster=3,
+ reduction_method="custom",
+ dims=2,
+ loadings=loadings,
+ use_for_pca=np.ones(values.shape[0], dtype=bool),
+ mu=np.zeros(values.shape[1]),
+ sigma=np.ones(values.shape[1]),
+ ann_metric="not_a_metric",
+ ann_efc=50,
+ ann_ef=50,
+ ann_m=16,
+ nthreads=1,
+ ann_parallel=False,
+ rand_state=4466,
+ do_kmeans_fit=False,
+ disable_scaling=True,
+ ann_idx=None,
+ lsi_skip_first=False,
+ lsi_params={},
+ harmonize=True,
+ batches=pd.DataFrame({"batch": ["a", "b"] * 4}),
+ )
+
+ assert not harmony_called
+ assert data.read_count == 0
diff --git a/tests/test_operation_argument_contracts.py b/tests/test_operation_argument_contracts.py
new file mode 100644
index 00000000..7236c25d
--- /dev/null
+++ b/tests/test_operation_argument_contracts.py
@@ -0,0 +1,507 @@
+import ast
+import importlib
+import inspect
+import pkgutil
+import textwrap
+from collections.abc import Callable
+from dataclasses import dataclass, field, fields
+
+import pytest
+
+from scarf.datastore import _operations as operations_package
+from scarf.datastore._operations.clustering import _ClusteringOperationsMixin
+from scarf.datastore._operations.graph import _GraphOperationsMixin
+from scarf.datastore.datastore import DataStore
+from scarf.graph import arguments as graph_arguments
+from scarf.graph.arguments import OperationArguments
+from scarf.metadata import arguments as metadata_arguments
+
+
+_SIGNATURE_REASONS = {
+ "execution",
+ "parent_stage",
+ "publication",
+ "routing",
+ "transformed",
+}
+_MODEL_REASONS = {
+ "algorithm_version",
+ "derived",
+ "execution",
+ "output",
+ "resolved_input",
+}
+_OPERATION_MODULES = tuple(
+ importlib.import_module(f"{operations_package.__name__}.{module.name}")
+ for module in pkgutil.iter_modules(operations_package.__path__)
+)
+
+
+def _classified(reason: str, *names: str) -> dict[str, str]:
+ return dict.fromkeys(names, reason)
+
+
+@dataclass(frozen=True, slots=True)
+class OperationContract:
+ producer: Callable[..., object]
+ arguments: type[OperationArguments]
+ constructor: Callable[..., object] | None = None
+ aliases: dict[str, str] = field(default_factory=dict)
+ signature_only: dict[str, str] = field(default_factory=dict)
+ model_only: dict[str, str] = field(default_factory=dict)
+
+
+_CONTRACTS = (
+ OperationContract(
+ DataStore.build_ann_index,
+ graph_arguments.AnnIndexArguments,
+ signature_only={
+ **_classified("routing", "from_assay"),
+ **_classified("publication", "update_state"),
+ },
+ model_only={
+ **_classified("derived", "parallel_threads"),
+ },
+ ),
+ OperationContract(
+ DataStore.build_connectivity_map,
+ graph_arguments.ConnectivityMapArguments,
+ signature_only={
+ **_classified("routing", "from_assay"),
+ **_classified("publication", "update_state"),
+ },
+ ),
+ OperationContract(
+ DataStore.run_custom_reduction,
+ graph_arguments.CustomReductionArguments,
+ constructor=_GraphOperationsMixin._run_reduction_artifact_impl,
+ signature_only={
+ **_classified("execution", "batch_size"),
+ **_classified("execution", "local_cache"),
+ **_classified("routing", "from_assay"),
+ },
+ model_only=_classified("resolved_input", "feature_scaling"),
+ ),
+ OperationContract(
+ _GraphOperationsMixin._build_embedding_initialization,
+ graph_arguments.EmbeddingInitializationArguments,
+ ),
+ OperationContract(
+ _GraphOperationsMixin._run_reduction_artifact_impl,
+ graph_arguments.FeatureScalingArguments,
+ aliases={"feat_scaling": "enabled"},
+ signature_only=_classified(
+ "parent_stage",
+ "custom_loadings",
+ "dims",
+ "from_assay",
+ "lsi_n_iter",
+ "lsi_n_oversamples",
+ "lsi_solver",
+ "lsi_skip_first",
+ "method",
+ "pca_cell_key",
+ "rand_state",
+ "show_elbow_plot",
+ "update_state",
+ ),
+ ),
+ OperationContract(
+ DataStore.run_harmony,
+ graph_arguments.HarmonyArguments,
+ aliases={"harmony_params": "harmony_parameters"},
+ signature_only={
+ **_classified("routing", "from_assay"),
+ **_classified("publication", "update_state"),
+ },
+ model_only={
+ **_classified("resolved_input", "batch_values"),
+ **_classified("algorithm_version", "algorithm_version"),
+ },
+ ),
+ OperationContract(
+ DataStore.run_lsi,
+ graph_arguments.LsiArguments,
+ constructor=_GraphOperationsMixin._run_reduction_artifact_impl,
+ signature_only={
+ **_classified("execution", "local_cache"),
+ **_classified("routing", "from_assay"),
+ },
+ model_only=_classified("resolved_input", "feature_scaling"),
+ ),
+ OperationContract(
+ DataStore.query_neighbors,
+ graph_arguments.NeighborQueryArguments,
+ signature_only={
+ **_classified("routing", "from_assay"),
+ **_classified("publication", "update_state"),
+ },
+ model_only=_classified("derived", "distance_metric"),
+ ),
+ OperationContract(
+ DataStore.run_normalization,
+ graph_arguments.NormalizationArguments,
+ model_only={
+ **_classified(
+ "resolved_input",
+ "cell_selection",
+ "feature_selection",
+ ),
+ **_classified(
+ "derived",
+ "normalization_method",
+ "size_factor",
+ ),
+ },
+ ),
+ OperationContract(
+ DataStore.run_pca,
+ graph_arguments.PcaArguments,
+ constructor=_GraphOperationsMixin._run_reduction_artifact_impl,
+ signature_only={
+ **_classified("execution", "local_cache"),
+ **_classified("routing", "from_assay"),
+ },
+ model_only=_classified(
+ "resolved_input",
+ "feature_scaling",
+ "pca_cell_selection",
+ ),
+ ),
+ OperationContract(
+ DataStore.run_aucell,
+ metadata_arguments.AucellArguments,
+ aliases={"net": "network_digest"},
+ model_only={
+ **_classified("algorithm_version", "algorithm_version"),
+ **_classified(
+ "resolved_input",
+ "cell_selection",
+ "feature_selection",
+ ),
+ },
+ ),
+ OperationContract(
+ DataStore.run_cell_cycle_scoring,
+ metadata_arguments.CellCycleArguments,
+ aliases={
+ "g2m_genes": "g2m_gene_indices",
+ "s_genes": "s_gene_indices",
+ },
+ model_only=_classified(
+ "derived",
+ "control_size",
+ "normalization_method",
+ "size_factor",
+ ),
+ ),
+ OperationContract(
+ DataStore.run_doublet_detection,
+ metadata_arguments.DoubletScoreArguments,
+ aliases={"cluster_key": "clusters"},
+ model_only=_classified("resolved_input", "connectivity_map"),
+ ),
+ OperationContract(
+ DataStore.run_fate_mapping,
+ metadata_arguments.FateMappingArguments,
+ model_only=_classified(
+ "resolved_input",
+ "cell_selection",
+ "connectivity_map",
+ "pseudotime",
+ "sink_labels",
+ ),
+ ),
+ OperationContract(
+ DataStore.mark_hto_identities,
+ metadata_arguments.HtoIdentityArguments,
+ model_only={
+ **_classified("derived", "method"),
+ **_classified(
+ "resolved_input",
+ "cell_selection",
+ "feature_ids_fingerprint",
+ ),
+ },
+ ),
+ OperationContract(
+ DataStore.run_leiden_clustering,
+ metadata_arguments.LeidenArguments,
+ constructor=_ClusteringOperationsMixin._prepare_leiden_clustering,
+ ),
+ OperationContract(
+ DataStore.run_marker_search,
+ metadata_arguments.MarkerTableArguments,
+ aliases={"norm_params": "normalization"},
+ signature_only={
+ **_classified("routing", "from_assay"),
+ **_classified("publication", "skip_save"),
+ },
+ model_only=_classified(
+ "resolved_input",
+ "cell_selection",
+ "clusters",
+ "feature_selection",
+ "normalization_method",
+ "size_factor",
+ )
+ | _classified(
+ "derived",
+ "method",
+ "alternative",
+ "tie_correction",
+ "continuity_correction",
+ "adjustment_method",
+ "adjustment_scope",
+ ),
+ ),
+ OperationContract(
+ DataStore.calc_membership_strength,
+ metadata_arguments.MembershipStrengthArguments,
+ model_only={
+ **_classified("algorithm_version", "algorithm_version"),
+ **_classified(
+ "resolved_input",
+ "cell_selection",
+ "clusters",
+ "connectivity_map",
+ ),
+ **_classified("derived", "decimals"),
+ **_classified("output", "output_key"),
+ },
+ ),
+ OperationContract(
+ DataStore.mark_prevalent_peaks,
+ metadata_arguments.PrevalentPeakArguments,
+ model_only={
+ **_classified("algorithm_version", "algorithm_version"),
+ **_classified(
+ "resolved_input",
+ "cell_selection",
+ "feature_selection",
+ ),
+ **_classified("derived", "normalization_method"),
+ },
+ ),
+ OperationContract(
+ DataStore.run_pseudotime_aggregation,
+ metadata_arguments.PseudotimeAggregationArguments,
+ aliases={"norm_params": "normalization"},
+ model_only=_classified(
+ "resolved_input",
+ "cell_selection",
+ "feature_selection",
+ "normalization_method",
+ "pseudotime",
+ "size_factor",
+ )
+ | _classified("execution", "n_threads"),
+ ),
+ OperationContract(
+ DataStore.run_pseudotime_marker_search,
+ metadata_arguments.PseudotimeMarkerArguments,
+ aliases={"norm_params": "normalization"},
+ model_only=_classified(
+ "resolved_input",
+ "cell_selection",
+ "feature_selection",
+ "normalization_method",
+ "pseudotime",
+ "size_factor",
+ )
+ | _classified(
+ "derived",
+ "association_method",
+ "p_value_method",
+ "adjustment_method",
+ "adjustment_scope",
+ )
+ | _classified("execution", "n_threads"),
+ ),
+ OperationContract(
+ DataStore.run_pseudotime_scoring,
+ metadata_arguments.PseudotimeScoringArguments,
+ signature_only=_classified(
+ "transformed",
+ "source_sink_key",
+ "ss_vec",
+ ),
+ model_only=_classified(
+ "resolved_input",
+ "cell_selection",
+ "connectivity_map",
+ "source_sink",
+ ),
+ ),
+ OperationContract(
+ DataStore.smart_label,
+ metadata_arguments.SmartLabelArguments,
+ model_only={
+ **_classified("algorithm_version", "algorithm_version"),
+ **_classified(
+ "resolved_input",
+ "base_labels",
+ "cell_selection",
+ "values",
+ ),
+ **_classified("derived", "suffix_style"),
+ },
+ ),
+ OperationContract(
+ DataStore.run_topacedo_sampler,
+ metadata_arguments.TopacedoArguments,
+ model_only=_classified(
+ "resolved_input",
+ "cell_selection",
+ "clusters",
+ "dendrogram",
+ ),
+ ),
+ OperationContract(
+ DataStore.run_tsne,
+ metadata_arguments.TsneArguments,
+ aliases={
+ "ini_embed": "initialization",
+ "nthreads": "parallel_threads",
+ },
+ ),
+ OperationContract(
+ DataStore.run_umap,
+ metadata_arguments.UmapArguments,
+ aliases={
+ "ini_embed": "initialization",
+ "nthreads": "parallel_threads",
+ },
+ ),
+ OperationContract(
+ DataStore.run_waggr,
+ metadata_arguments.WaggrArguments,
+ aliases={"net": "network_digest"},
+ model_only={
+ **_classified("algorithm_version", "algorithm_version"),
+ **_classified(
+ "resolved_input",
+ "cell_selection",
+ "feature_selection",
+ ),
+ **_classified(
+ "derived",
+ "normalization_method",
+ "size_factor",
+ ),
+ },
+ ),
+)
+
+
+def _concrete_argument_classes() -> set[type[OperationArguments]]:
+ classes: set[type[OperationArguments]] = set()
+ for module in (graph_arguments, metadata_arguments):
+ for _name, candidate in inspect.getmembers(module, inspect.isclass):
+ if (
+ candidate is not OperationArguments
+ and issubclass(candidate, OperationArguments)
+ and candidate.__module__ == module.__name__
+ ):
+ classes.add(candidate)
+ return classes
+
+
+def _parameter_names(producer: Callable[..., object]) -> set[str]:
+ return set(inspect.signature(producer).parameters) - {"self"}
+
+
+def _model_field_names(arguments: type[OperationArguments]) -> set[str]:
+ return {model_field.name for model_field in fields(arguments)}
+
+
+def _call_name(call: ast.Call) -> str | None:
+ if isinstance(call.func, ast.Name):
+ return call.func.id
+ if isinstance(call.func, ast.Attribute):
+ return call.func.attr
+ return None
+
+
+def test_every_operation_arguments_class_has_a_contract() -> None:
+ registered = [contract.arguments for contract in _CONTRACTS]
+
+ assert len(registered) == len(set(registered))
+ assert set(registered) == _concrete_argument_classes()
+
+
+@pytest.mark.parametrize(
+ "contract",
+ _CONTRACTS,
+ ids=lambda contract: contract.arguments.operation,
+)
+def test_operation_signature_and_model_fields_are_classified(
+ contract: OperationContract,
+) -> None:
+ signature_names = _parameter_names(contract.producer)
+ model_names = _model_field_names(contract.arguments)
+ alias_sources = set(contract.aliases)
+ alias_targets = set(contract.aliases.values())
+
+ assert alias_sources <= signature_names - model_names
+ assert alias_targets <= model_names - signature_names
+ assert len(alias_targets) == len(contract.aliases)
+ assert set(contract.signature_only.values()) <= _SIGNATURE_REASONS
+ assert set(contract.model_only.values()) <= _MODEL_REASONS
+ assert signature_names - model_names == (
+ alias_sources | set(contract.signature_only)
+ )
+ assert model_names - signature_names == (alias_targets | set(contract.model_only))
+
+
+@pytest.mark.parametrize(
+ "contract",
+ _CONTRACTS,
+ ids=lambda contract: contract.arguments.operation,
+)
+def test_operation_constructor_wires_every_model_field(
+ contract: OperationContract,
+) -> None:
+ constructor = contract.constructor or contract.producer
+ source = textwrap.dedent(inspect.getsource(constructor))
+ tree = ast.parse(source)
+ calls = [
+ call
+ for call in ast.walk(tree)
+ if isinstance(call, ast.Call)
+ and _call_name(call) == contract.arguments.__name__
+ ]
+
+ assert calls, f"{contract.arguments.__name__} is not built by its producer"
+ expected_fields = _model_field_names(contract.arguments)
+ for call in calls:
+ keyword_names = {
+ keyword.arg for keyword in call.keywords if keyword.arg is not None
+ }
+ assert keyword_names == expected_fields
+
+
+@pytest.mark.parametrize(
+ "contract",
+ _CONTRACTS,
+ ids=lambda contract: contract.arguments.operation,
+)
+def test_operation_model_has_no_unregistered_producer(
+ contract: OperationContract,
+) -> None:
+ constructor = contract.constructor or contract.producer
+ expected_module = inspect.getmodule(constructor)
+ assert expected_module is not None
+ calls_by_module: dict[str, int] = {}
+ for module in _OPERATION_MODULES:
+ tree = ast.parse(textwrap.dedent(inspect.getsource(module)))
+ count = sum(
+ 1
+ for call in ast.walk(tree)
+ if isinstance(call, ast.Call)
+ and _call_name(call) == contract.arguments.__name__
+ )
+ if count:
+ calls_by_module[module.__name__] = count
+
+ assert calls_by_module == {expected_module.__name__: 1}
diff --git a/tests/test_output_configuration.py b/tests/test_output_configuration.py
new file mode 100644
index 00000000..14543eb3
--- /dev/null
+++ b/tests/test_output_configuration.py
@@ -0,0 +1,94 @@
+import json
+import re
+import subprocess
+import sys
+
+import pytest
+
+from scarf.utils import configure_output, get_log_level, logger, set_verbosity
+from scarf.utils.logging import progress_enabled
+
+
+@pytest.fixture(autouse=True)
+def _restore_output_configuration():
+ yield
+ set_verbosity("INFO")
+ configure_output(progress=False, timestamps=False)
+
+
+def test_import_defaults_are_notebook_first_in_isolated_process():
+ code = """
+import json
+import scarf
+from scarf.utils.logging import progress_enabled
+
+scarf.logger.info("default output")
+print(json.dumps({"level": scarf.get_log_level(), "progress": progress_enabled()}))
+"""
+ result = subprocess.run(
+ [sys.executable, "-c", code],
+ check=True,
+ capture_output=True,
+ text=True,
+ )
+ lines = result.stdout.splitlines()
+
+ assert lines[0] == "INFO: default output"
+ assert json.loads(lines[-1]) == {"level": 20, "progress": True}
+ assert "\x1b" not in result.stdout
+
+
+def test_configure_output_updates_settings_independently(capsys):
+ configure_output(level="WARNING", progress=True)
+ configure_output(timestamps=True)
+ logger.warning("timestamped")
+ output = capsys.readouterr().out
+
+ assert get_log_level() == 30
+ assert progress_enabled()
+ assert re.fullmatch(
+ r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}"
+ r"[+-]\d{2}:\d{2} \| WARNING \| timestamped\n",
+ output,
+ )
+
+
+def test_set_verbosity_file_output_has_no_ansi_codes(tmp_path):
+ path = tmp_path / "scarf.log"
+ set_verbosity("INFO", str(path))
+ logger.warning("file output")
+ output = path.read_text(encoding="utf-8")
+
+ assert output == "WARNING: file output\n"
+ assert "\x1b" not in output
+
+
+def test_reconfiguration_preserves_caller_sink_and_configured_level():
+ captured: list[str] = []
+ sink = logger.add(
+ lambda message: captured.append(message.record["message"]),
+ level="TRACE",
+ )
+ try:
+ configure_output(level="WARNING")
+ logger.debug("caller only")
+ logger.warning("visible warning")
+ finally:
+ logger.remove(sink)
+
+ assert captured == ["caller only", "visible warning"]
+ assert get_log_level() == 30
+
+
+def test_reconfiguration_tolerates_a_stale_scarf_handler(capsys):
+ logger.remove()
+ configure_output(level="DEBUG")
+ logger.debug("restored")
+
+ assert capsys.readouterr().out == "DEBUG: restored\n"
+
+
+@pytest.mark.parametrize("setting", ["progress", "timestamps"])
+def test_configure_output_requires_boolean_settings(setting):
+ with pytest.raises(TypeError, match=setting):
+ configure_output(**{setting: 1})
diff --git a/tests/test_parallel.py b/tests/test_parallel.py
new file mode 100644
index 00000000..ee347908
--- /dev/null
+++ b/tests/test_parallel.py
@@ -0,0 +1,247 @@
+import threading
+import time
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.matrix import ChunkedArray
+from scarf.storage.parallel import in_shard_context, map_shards, stream_shards
+
+
+def test_map_shards_preserves_order():
+ ranges = [(i * 10, i * 10 + 10) for i in range(6)]
+ out = map_shards(ranges, lambda idx, s, e: (idx, s, e), workers=8)
+ assert out == [(i, i * 10, i * 10 + 10) for i in range(6)]
+
+
+def test_map_shards_empty():
+ assert map_shards([], lambda i, s, e: i, workers=8) == []
+
+
+def test_map_shards_bounds_in_flight():
+ lock = threading.Lock()
+ in_flight = 0
+ max_seen = 0
+ ranges = [(i, i + 1) for i in range(16)]
+
+ def produce(idx, s, e):
+ nonlocal in_flight, max_seen
+ with lock:
+ in_flight += 1
+ max_seen = max(max_seen, in_flight)
+ time.sleep(0.01)
+ with lock:
+ in_flight -= 1
+ return idx
+
+ map_shards(ranges, produce, workers=8)
+ assert 1 < max_seen <= 8
+
+
+def test_map_shards_serial_backend_runs_inline():
+ seen_context = []
+
+ def produce(idx, s, e):
+ seen_context.append(in_shard_context())
+ return idx
+
+ out = map_shards([(0, 1), (1, 2)], produce, workers=8, backend="serial")
+ assert out == [0, 1]
+
+
+def test_nested_map_shards_runs_serial():
+ inner_context = []
+
+ def outer(idx, s, e):
+ assert in_shard_context() is True
+
+ def inner(j, a, b):
+ inner_context.append(in_shard_context())
+ return j
+
+ return map_shards([(0, 1), (1, 2), (2, 3)], inner, workers=8)
+
+ map_shards([(0, 1), (1, 2)], outer, workers=8)
+ assert inner_context and all(inner_context)
+
+
+def test_stream_shards_preserves_order():
+ out = list(stream_shards(range(5), lambda x: x * 2, workers=4))
+ assert out == [0, 2, 4, 6, 8]
+
+
+def test_stream_shards_bounds_and_restores_io_concurrency():
+ with zarr.config.set({"async.concurrency": 7}):
+ seen = []
+
+ def fn(x):
+ seen.append(zarr.config.get("async.concurrency"))
+ return x
+
+ out = list(stream_shards(range(4), fn, workers=2, io_concurrency=3))
+ assert out == [0, 1, 2, 3]
+ assert seen and all(s == 3 for s in seen)
+ assert zarr.config.get("async.concurrency") == 7
+
+
+def test_stream_shards_config_neutral_without_io():
+ with zarr.config.set({"async.concurrency": 5}):
+ seen = []
+
+ def fn(x):
+ seen.append(zarr.config.get("async.concurrency"))
+ return x
+
+ list(stream_shards(range(4), fn, workers=2))
+ assert seen and all(s == 5 for s in seen)
+
+
+def test_stream_shards_cancels_unconsumed_work():
+ started = []
+ lock = threading.Lock()
+
+ def work(value):
+ with lock:
+ started.append(value)
+ time.sleep(0.02)
+ return value
+
+ stream = stream_shards(range(100), work, workers=2)
+ assert next(stream) == 0
+ stream.close()
+ assert set(started).issubset({0, 1, 2})
+
+
+@pytest.mark.parametrize("workers", [1, 2])
+def test_stream_shards_closes_source_iterator(workers):
+ closed = False
+
+ def source():
+ nonlocal closed
+ try:
+ yield from range(100)
+ finally:
+ closed = True
+
+ stream = stream_shards(source(), lambda value: value, workers=workers)
+ assert next(stream) == 0
+ stream.close()
+ assert closed
+
+
+def test_stream_shards_cancels_pending_work_after_failure():
+ started = []
+ lock = threading.Lock()
+
+ def work(value):
+ with lock:
+ started.append(value)
+ if value == 0:
+ raise RuntimeError("injected worker failure")
+ time.sleep(0.05)
+ return value
+
+ with pytest.raises(RuntimeError, match="injected worker failure"):
+ list(stream_shards(range(100), work, workers=2))
+ assert set(started).issubset({0, 1})
+
+
+def test_io_concurrency_isolated_across_parallel_runtimes():
+ before = zarr.config.get("async.concurrency")
+ barrier = threading.Barrier(2)
+ seen = {3: [], 7: []}
+
+ def run(io_concurrency):
+ def inspect_config(value):
+ barrier.wait()
+ seen[io_concurrency].append(zarr.config.get("async.concurrency"))
+ return value
+
+ list(
+ stream_shards(
+ [0],
+ inspect_config,
+ workers=1,
+ io_concurrency=io_concurrency,
+ )
+ )
+
+ first = threading.Thread(target=run, args=(3,))
+ second = threading.Thread(target=run, args=(7,))
+ first.start()
+ second.start()
+ first.join()
+ second.join()
+ assert seen == {3: [3], 7: [7]}
+ assert zarr.config.get("async.concurrency") == before
+
+
+def test_map_shards_uses_worker_budget_once_across_tasks_and_io():
+ with zarr.config.set({"async.concurrency": 99}):
+ seen = []
+ lock = threading.Lock()
+ in_flight = 0
+ max_in_flight = 0
+
+ def produce(idx, s, e):
+ nonlocal in_flight, max_in_flight
+ with lock:
+ in_flight += 1
+ max_in_flight = max(max_in_flight, in_flight)
+ seen.append(zarr.config.get("async.concurrency"))
+ time.sleep(0.01)
+ with lock:
+ in_flight -= 1
+ return idx
+
+ map_shards([(i, i + 1) for i in range(16)], produce, workers=8)
+ assert seen and all(s == 1 for s in seen)
+ assert max_in_flight > 1
+ assert max_in_flight * seen[0] <= 8
+ assert zarr.config.get("async.concurrency") == 99
+
+
+def _toy_chunked(data, chunk_rows, nthreads):
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ arr = root.create_array(
+ "d", shape=data.shape, chunks=(chunk_rows, data.shape[1]), dtype=data.dtype
+ )
+ arr[:] = data
+ return ChunkedArray(arr, nthreads=nthreads)
+
+
+def test_stream_blocks_matches_serial_materialization():
+ rng = np.random.default_rng(0)
+ data = rng.standard_normal((35, 6)).astype(np.float32)
+ serial = np.vstack(list(_toy_chunked(data, 10, 1).stream_blocks(nthreads=1)))
+ parallel = np.vstack(list(_toy_chunked(data, 10, 4).stream_blocks(nthreads=4)))
+ assert np.array_equal(serial, data)
+ assert np.array_equal(serial, parallel)
+
+
+def test_reductions_bit_identical_across_threads():
+ rng = np.random.default_rng(1)
+ data = rng.standard_normal((37, 5)).astype(np.float32)
+ ca1 = _toy_chunked(data, 10, 1)
+ ca4 = _toy_chunked(data, 10, 4)
+ assert np.array_equal(
+ np.asarray(ca1.sum(axis=0).compute(1)),
+ np.asarray(ca4.sum(axis=0).compute(4)),
+ )
+ assert np.array_equal(
+ np.asarray(ca1.var(axis=0).compute(1)),
+ np.asarray(ca4.var(axis=0).compute(4)),
+ )
+ m1, s1 = ca1.mean_and_std(nthreads=1)
+ m4, s4 = ca4.mean_and_std(nthreads=4)
+ assert np.array_equal(m1, m4)
+ assert np.array_equal(s1, s4)
+
+
+def test_compute_matches_source_across_threads():
+ rng = np.random.default_rng(2)
+ data = rng.standard_normal((40, 4)).astype(np.float32)
+ assert np.array_equal(_toy_chunked(data, 8, 1).compute(1), data)
+ assert np.array_equal(_toy_chunked(data, 8, 5).compute(5), data)
diff --git a/tests/test_paris_core.py b/tests/test_paris_core.py
new file mode 100644
index 00000000..0bf74aa2
--- /dev/null
+++ b/tests/test_paris_core.py
@@ -0,0 +1,229 @@
+import numpy as np
+import pytest
+from scipy.sparse import csr_matrix
+from sknetwork.hierarchy import Paris
+
+from scarf.clustering._paris_core import _contract_graph, canonicalize_paris_graph
+from scarf.clustering.paris import (
+ fit_paris_hierarchy,
+ hierarchy_to_dendrogram,
+ paris_dendrogram,
+)
+
+
+def _leaf_sets(dendrogram: np.ndarray) -> dict[frozenset[int], float]:
+ n_leaves = dendrogram.shape[0] + 1
+ leaves = {node: frozenset({node}) for node in range(n_leaves)}
+ clades: dict[frozenset[int], float] = {}
+ for merge_index, row in enumerate(dendrogram):
+ members = leaves[int(row[0])] | leaves[int(row[1])]
+ leaves[n_leaves + merge_index] = members
+ clades[members] = float(row[2])
+ return clades
+
+
+def _tie_free_graph() -> csr_matrix:
+ rng = np.random.default_rng(742)
+ n_vertices = 40
+ rows = np.repeat(np.arange(n_vertices), 5)
+ columns = rng.integers(0, n_vertices, size=rows.size)
+ weights = rng.uniform(0.1, 2.0, size=rows.size)
+ return csr_matrix(
+ (weights, (rows, columns)),
+ shape=(n_vertices, n_vertices),
+ )
+
+
+def test_merge_distances_match_aggregated_graph_weights() -> None:
+ graph = canonicalize_paris_graph(_tie_free_graph())
+ hierarchy = fit_paris_hierarchy(graph, n_threads=2)
+ total_weight = float(graph.data.sum())
+ members = {
+ node: np.asarray([node], dtype=np.int64) for node in range(hierarchy.n_leaves)
+ }
+ volumes = np.asarray(graph.sum(axis=1)).ravel()
+
+ for merge_index, children in enumerate(hierarchy.children):
+ left, right = map(int, children)
+ left_members = members[left]
+ right_members = members[right]
+ between_weight = float(graph[left_members][:, right_members].sum())
+ left_volume = float(volumes[left_members].sum())
+ right_volume = float(volumes[right_members].sum())
+ expected = (left_volume / total_weight) * (right_volume / between_weight)
+ assert hierarchy.heights[merge_index] == pytest.approx(expected)
+ members[hierarchy.n_leaves + merge_index] = np.concatenate(
+ (left_members, right_members)
+ )
+
+
+def test_tie_free_topology_matches_scikit_network() -> None:
+ canonical = canonicalize_paris_graph(_tie_free_graph())
+ native = hierarchy_to_dendrogram(fit_paris_hierarchy(canonical, n_threads=4))
+ reference = np.asarray(Paris(reorder=False).fit_transform(canonical))
+ native_clades = _leaf_sets(native)
+ reference_clades = _leaf_sets(reference)
+
+ assert native_clades.keys() == reference_clades.keys()
+ for clade in native_clades:
+ assert native_clades[clade] == pytest.approx(
+ reference_clades[clade],
+ abs=1e-6,
+ rel=1e-6,
+ )
+
+
+def test_output_is_identical_across_thread_counts() -> None:
+ graph = _tie_free_graph()
+ reference = fit_paris_hierarchy(graph, n_threads=1)
+ for n_threads in (2, 4, 8):
+ result = fit_paris_hierarchy(graph, n_threads=n_threads)
+ assert np.array_equal(result.children, reference.children)
+ assert np.array_equal(result.heights, reference.heights)
+ assert np.array_equal(result.sizes, reference.sizes)
+ assert np.array_equal(result.component_roots, reference.component_roots)
+ assert np.array_equal(result.synthetic_joins, reference.synthetic_joins)
+
+
+def test_contraction_diagnostics_account_for_each_phase() -> None:
+ hierarchy = fit_paris_hierarchy(_tie_free_graph(), n_threads=2)
+ diagnostics = hierarchy.diagnostics
+ assert diagnostics is not None
+ assert diagnostics.rounds
+
+ for round_ in diagnostics.rounds:
+ phases = (
+ round_.contraction_remap_seconds,
+ round_.contraction_filter_seconds,
+ round_.contraction_build_seconds,
+ round_.contraction_cleanup_seconds,
+ )
+ assert all(seconds >= 0 for seconds in phases)
+ assert sum(phases) <= round_.contraction_seconds + 1e-9
+
+
+def test_parallel_contraction_matches_scipy_reference() -> None:
+ graph = canonicalize_paris_graph(_tie_free_graph())
+ group_ids = np.arange(graph.shape[0], dtype=graph.indices.dtype)
+ group_ids[20:30] = group_ids[:10]
+ logical_groups = np.unique(group_ids)
+ mapping = np.searchsorted(logical_groups, group_ids).astype(
+ graph.indices.dtype,
+ copy=False,
+ )
+ n_vertices = logical_groups.size
+
+ rows = np.repeat(
+ np.arange(graph.shape[0], dtype=graph.indices.dtype),
+ np.diff(graph.indptr),
+ )
+ mapped_rows = mapping[rows]
+ mapped_columns = mapping[graph.indices]
+ keep = mapped_rows != mapped_columns
+ expected = csr_matrix(
+ (
+ graph.data[keep],
+ (mapped_rows[keep], mapped_columns[keep]),
+ ),
+ shape=(n_vertices, n_vertices),
+ )
+ expected.sum_duplicates()
+ expected.eliminate_zeros()
+ expected.sort_indices()
+
+ values = np.full((2, graph.shape[0]), np.nan, dtype=np.float64)
+ contracted, *_timings = _contract_graph(
+ graph,
+ mapping,
+ n_vertices,
+ values,
+ )
+
+ assert np.array_equal(contracted.indptr, expected.indptr)
+ assert np.array_equal(contracted.indices, expected.indices)
+ assert np.array_equal(contracted.data, expected.data)
+ assert contracted.has_canonical_format
+ assert np.isnan(values).all()
+
+ repeated, *_timings = _contract_graph(
+ graph,
+ mapping,
+ n_vertices,
+ values,
+ )
+ assert np.array_equal(repeated.indptr, contracted.indptr)
+ assert np.array_equal(repeated.indices, contracted.indices)
+ assert np.array_equal(repeated.data, contracted.data)
+
+
+def test_ties_use_smallest_logical_tree_id() -> None:
+ graph = csr_matrix(
+ np.asarray(
+ [
+ [0, 1, 1, 1],
+ [1, 0, 0, 0],
+ [1, 0, 0, 0],
+ [1, 0, 0, 0],
+ ],
+ dtype=np.float64,
+ )
+ )
+ hierarchy = fit_paris_hierarchy(graph, n_threads=4)
+ assert hierarchy.children[0].tolist() == [0, 1]
+ assert hierarchy.children.tolist() == [[0, 1], [2, 4], [3, 5]]
+
+
+def test_canonicalization_sums_duplicates_sorts_and_removes_diagonal() -> None:
+ indptr = np.asarray([0, 4, 5, 6], dtype=np.int32)
+ indices = np.asarray([2, 1, 1, 0, 0, 1], dtype=np.int32)
+ data = np.asarray([2.0, 1.0, 3.0, 9.0, 4.0, 5.0])
+ graph = csr_matrix((data, indices, indptr), shape=(3, 3))
+ canonical = canonicalize_paris_graph(graph)
+
+ assert canonical.has_sorted_indices
+ assert canonical.diagonal().tolist() == [0.0, 0.0, 0.0]
+ assert np.array_equal(
+ canonical.toarray(),
+ np.asarray(
+ [
+ [0.0, 8.0, 2.0],
+ [8.0, 0.0, 5.0],
+ [2.0, 5.0, 0.0],
+ ]
+ ),
+ )
+
+
+@pytest.mark.parametrize("weight", [-1.0, np.nan, np.inf])
+def test_invalid_weights_are_rejected(weight: float) -> None:
+ graph = csr_matrix(np.asarray([[0.0, weight], [1.0, 0.0]]))
+ error = "non-negative" if weight < 0 else "finite"
+ with pytest.raises(ValueError, match=error):
+ fit_paris_hierarchy(graph)
+
+
+def test_components_isolates_and_synthetic_joins_are_explicit() -> None:
+ graph = csr_matrix(
+ np.asarray(
+ [
+ [0, 2, 0, 0, 0],
+ [2, 0, 0, 0, 0],
+ [0, 0, 0, 3, 0],
+ [0, 0, 3, 0, 0],
+ [0, 0, 0, 0, 0],
+ ],
+ dtype=np.float64,
+ )
+ )
+ hierarchy = fit_paris_hierarchy(graph)
+ raw = hierarchy_to_dendrogram(hierarchy)
+ compatibility = hierarchy_to_dendrogram(hierarchy, compatibility=True)
+
+ assert hierarchy.component_roots.tolist() == [5, 6, 4]
+ assert hierarchy.synthetic_joins.tolist() == [False, False, True, True]
+ assert np.isinf(raw[hierarchy.synthetic_joins, 2]).all()
+ assert np.all(compatibility[hierarchy.synthetic_joins, 2] == 0)
+ assert np.array_equal(paris_dendrogram(graph), compatibility)
+ assert hierarchy.children.dtype == np.int32
+ assert hierarchy.sizes.dtype == np.int32
+ assert hierarchy.heights.dtype == np.float64
diff --git a/tests/test_paris_modularity.py b/tests/test_paris_modularity.py
new file mode 100644
index 00000000..f7cdc07f
--- /dev/null
+++ b/tests/test_paris_modularity.py
@@ -0,0 +1,227 @@
+import numpy as np
+import pytest
+from scipy.sparse import csr_matrix
+
+from scarf.clustering._paris_core import ParisHierarchy
+from scarf.clustering._paris_modularity import (
+ aggregate_plateau_statistics,
+ collect_topology_statistics,
+ modularity_split_gains,
+)
+from scarf.clustering.paris_multiscale import (
+ adaptive_cut,
+ collapse_equal_height_plateaus,
+)
+from scarf.clustering.paris import fit_paris_hierarchy
+
+
+def _hierarchy(
+ children: list[tuple[int, int]],
+ heights: list[float],
+ *,
+ component_roots: list[int] | None = None,
+ synthetic_joins: list[bool] | None = None,
+) -> ParisHierarchy:
+ n_leaves = len(children) + 1
+ child_array = np.asarray(children, dtype=np.int32)
+ raw_sizes = np.ones(2 * n_leaves - 1, dtype=np.int32)
+ merge_sizes = np.empty(n_leaves - 1, dtype=np.int32)
+ for merge_index, (left, right) in enumerate(children):
+ size = int(raw_sizes[left]) + int(raw_sizes[right])
+ raw_sizes[n_leaves + merge_index] = size
+ merge_sizes[merge_index] = size
+ roots = [2 * n_leaves - 2] if component_roots is None else component_roots
+ synthetic = (
+ np.isinf(heights).tolist() if synthetic_joins is None else synthetic_joins
+ )
+ return ParisHierarchy(
+ children=child_array,
+ heights=np.asarray(heights, dtype=np.float64),
+ sizes=merge_sizes,
+ component_roots=np.asarray(roots, dtype=np.int32),
+ synthetic_joins=np.asarray(synthetic, dtype=bool),
+ n_leaves=n_leaves,
+ total_weight=1.0,
+ )
+
+
+def _canonical_graph(
+ n_vertices: int,
+ edges: list[tuple[int, int]],
+) -> csr_matrix:
+ rows: list[int] = []
+ columns: list[int] = []
+ for left, right in edges:
+ rows.extend((left, right))
+ columns.extend((right, left))
+ graph = csr_matrix(
+ (
+ np.ones(len(rows), dtype=np.float64),
+ (rows, columns),
+ ),
+ shape=(n_vertices, n_vertices),
+ )
+ graph.sum_duplicates()
+ graph.eliminate_zeros()
+ graph.sort_indices()
+ return graph
+
+
+def _erdos_renyi_null_graph(seed: int) -> csr_matrix:
+ n_vertices = 1_000
+ rows, columns = np.triu_indices(n_vertices, k=1)
+ keep = np.random.default_rng(seed).random(rows.size) < 0.01
+ upper = csr_matrix(
+ (
+ np.ones(int(keep.sum()), dtype=np.float64),
+ (rows[keep], columns[keep]),
+ ),
+ shape=(n_vertices, n_vertices),
+ )
+ return csr_matrix(upper + upper.T)
+
+
+def _balanced_four_hierarchy(*, plateau: bool = False) -> ParisHierarchy:
+ return _hierarchy(
+ [(0, 1), (2, 3), (4, 5)],
+ [1.0, 1.0, 1.0 if plateau else 2.0],
+ )
+
+
+def _balanced_eight_hierarchy() -> ParisHierarchy:
+ return _hierarchy(
+ [
+ (0, 1),
+ (2, 3),
+ (8, 9),
+ (4, 5),
+ (6, 7),
+ (11, 12),
+ (10, 13),
+ ],
+ [1.0, 1.0, 2.0, 1.0, 1.0, 2.0, 3.0],
+ )
+
+
+def test_offline_lca_and_event_statistics_match_scalars() -> None:
+ hierarchy = _balanced_four_hierarchy()
+ graph = _canonical_graph(4, [(0, 1), (2, 3), (0, 2), (1, 2)])
+ forest = collapse_equal_height_plateaus(hierarchy)
+
+ topology = collect_topology_statistics(graph, hierarchy)
+ repeated = collect_topology_statistics(graph, hierarchy)
+ assert topology.leaf_degrees.tolist() == [2, 2, 3, 1]
+ assert topology.lca_edge_counts.tolist() == [1, 1, 2]
+ assert topology.component_edge_counts.tolist() == [4]
+ assert topology.leaf_degrees.tobytes() == repeated.leaf_degrees.tobytes()
+ assert topology.lca_edge_counts.tobytes() == repeated.lca_edge_counts.tobytes()
+ assert not topology.lca_edge_counts.flags.writeable
+
+ events = aggregate_plateau_statistics(hierarchy, forest, topology)
+ assert events.cross_edges.tolist() == [1, 1, 2]
+ assert events.volumes.tolist() == [4, 4, 8]
+ assert not events.cross_edges.flags.writeable
+ assert not events.volumes.flags.writeable
+
+
+def test_plateau_statistics_ignore_binary_refinement() -> None:
+ first = _balanced_four_hierarchy(plateau=True)
+ second = _hierarchy(
+ [(0, 2), (1, 3), (4, 5)],
+ [1.0, 1.0, 1.0],
+ )
+ graph = _canonical_graph(4, [(0, 1), (2, 3), (0, 2), (1, 2)])
+ first_forest = collapse_equal_height_plateaus(first)
+ second_forest = collapse_equal_height_plateaus(second)
+ first_events = aggregate_plateau_statistics(
+ first,
+ first_forest,
+ collect_topology_statistics(graph, first),
+ )
+ second_events = aggregate_plateau_statistics(
+ second,
+ second_forest,
+ collect_topology_statistics(graph, second),
+ )
+
+ assert first_forest.representatives.size == 1
+ assert second_forest.representatives.size == 1
+ assert np.array_equal(first_events.cross_edges, second_events.cross_edges)
+ assert np.array_equal(first_events.volumes, second_events.volumes)
+
+
+def test_modularity_split_gain_matches_scalar_delta_q() -> None:
+ hierarchy = _balanced_four_hierarchy()
+ forest = collapse_equal_height_plateaus(hierarchy)
+ graph = _canonical_graph(4, [(0, 1), (2, 3), (0, 2), (1, 2)])
+ gains = modularity_split_gains(hierarchy, forest, graph)
+
+ two_m = 8.0
+ expected_root = (8.0**2 - (4.0**2 + 4.0**2)) / two_m**2 - 2.0 * 2.0 / two_m
+ root = int(forest.component_roots[0])
+ assert gains[root] == pytest.approx(expected_root)
+ expected_pair = (4.0**2 - (2.0**2 + 2.0**2)) / two_m**2 - 2.0 * 1.0 / two_m
+ assert gains[0] == pytest.approx(expected_pair)
+
+
+def test_split_gate_vetoes_exact_non_positive_gain_splits() -> None:
+ hierarchy = _balanced_eight_hierarchy()
+ graph = _canonical_graph(
+ 8,
+ [(0, 1), (2, 3), (4, 5), (6, 7), (1, 2), (5, 6), (3, 4), (0, 7)],
+ )
+ forest = collapse_equal_height_plateaus(hierarchy)
+ gains = modularity_split_gains(hierarchy, forest, graph)
+
+ unguarded = adaptive_cut(hierarchy, 2, plateau_forest=forest)
+ guarded = adaptive_cut(hierarchy, 2, plateau_forest=forest, split_gate=gains)
+ root_veto = np.ones(forest.representatives.size, dtype=np.float64)
+ root_veto[int(forest.component_roots[0])] = 0.0
+ coarsened = adaptive_cut(
+ hierarchy,
+ 2,
+ plateau_forest=forest,
+ split_gate=root_veto,
+ )
+ permissive = adaptive_cut(
+ hierarchy,
+ 2,
+ plateau_forest=forest,
+ split_gate=np.ones(forest.representatives.size),
+ )
+
+ assert unguarded.labels.tolist() == [1, 1, 2, 2, 3, 3, 4, 4]
+ assert guarded.labels.tolist() == [1, 1, 1, 1, 2, 2, 2, 2]
+ assert [item.selected_node for item in guarded.diagnostics] == [10, 13]
+ assert coarsened.labels.tolist() == [1] * 8
+ assert np.array_equal(permissive.labels, unguarded.labels)
+
+
+def test_guard_reduces_erdos_renyi_null_fragmentation() -> None:
+ unguarded_counts: list[int] = []
+ guarded_counts: list[int] = []
+ for seed in range(3):
+ graph = _erdos_renyi_null_graph(seed)
+ hierarchy = fit_paris_hierarchy(graph, n_threads=2)
+ forest = collapse_equal_height_plateaus(hierarchy)
+ assert hierarchy.component_roots.size == 1
+
+ unguarded = adaptive_cut(hierarchy, 11, plateau_forest=forest)
+ guarded = adaptive_cut(
+ hierarchy,
+ 11,
+ plateau_forest=forest,
+ split_gate=modularity_split_gains(hierarchy, forest, graph),
+ )
+ unguarded_counts.append(unguarded.n_clusters)
+ guarded_counts.append(guarded.n_clusters)
+
+ assert all(
+ guarded < unguarded
+ for guarded, unguarded in zip(
+ guarded_counts,
+ unguarded_counts,
+ strict=True,
+ )
+ )
+ assert 2 * sum(guarded_counts) <= sum(unguarded_counts)
diff --git a/tests/test_paris_multiscale.py b/tests/test_paris_multiscale.py
new file mode 100644
index 00000000..7f5fec45
--- /dev/null
+++ b/tests/test_paris_multiscale.py
@@ -0,0 +1,367 @@
+import itertools
+
+import numpy as np
+import pytest
+from scipy.sparse import csr_matrix
+
+from scarf.clustering._paris_core import ParisHierarchy
+from scarf.clustering.paris_multiscale import (
+ adaptive_cut,
+ collapse_equal_height_plateaus,
+)
+from scarf.clustering.paris import fit_paris_hierarchy
+
+
+def _hierarchy(
+ children: list[tuple[int, int]],
+ heights: list[float],
+ *,
+ component_roots: list[int] | None = None,
+ synthetic_joins: list[bool] | None = None,
+) -> ParisHierarchy:
+ n_leaves = len(children) + 1
+ child_array = np.asarray(children, dtype=np.int32)
+ sizes = np.ones(2 * n_leaves - 1, dtype=np.int32)
+ merge_sizes = np.empty(n_leaves - 1, dtype=np.int32)
+ for merge_index, (left, right) in enumerate(children):
+ size = int(sizes[left]) + int(sizes[right])
+ merge_sizes[merge_index] = size
+ sizes[n_leaves + merge_index] = size
+ synthetic = (
+ np.isinf(heights)
+ if synthetic_joins is None
+ else np.asarray(synthetic_joins, dtype=bool)
+ )
+ roots = (
+ np.asarray([2 * n_leaves - 2], dtype=np.int32)
+ if component_roots is None
+ else np.asarray(component_roots, dtype=np.int32)
+ )
+ return ParisHierarchy(
+ children=child_array,
+ heights=np.asarray(heights, dtype=np.float64),
+ sizes=merge_sizes,
+ component_roots=roots,
+ synthetic_joins=np.asarray(synthetic, dtype=bool),
+ n_leaves=n_leaves,
+ total_weight=1.0,
+ )
+
+
+def _nested_block_graph(seed: int = 0) -> csr_matrix:
+ rng = np.random.default_rng(seed)
+ block_size = 40
+ n_cells = 4 * block_size
+ rows: list[int] = []
+ columns: list[int] = []
+ weights: list[float] = []
+ for left in range(n_cells):
+ for right in range(left + 1, n_cells):
+ left_block = left // block_size
+ right_block = right // block_size
+ if left_block == right_block:
+ probability, weight = 0.28, 1.0
+ elif left_block // 2 == right_block // 2:
+ probability, weight = 0.12, 0.15
+ else:
+ probability, weight = 0.04, 0.002
+ if rng.random() >= probability:
+ continue
+ rows.extend((left, right))
+ columns.extend((right, left))
+ weights.extend((weight, weight))
+ return csr_matrix(
+ (weights, (rows, columns)),
+ shape=(n_cells, n_cells),
+ )
+
+
+def _plateau_variants() -> tuple[ParisHierarchy, ParisHierarchy]:
+ first = _hierarchy(
+ [(0, 1), (2, 3), (6, 7), (4, 5), (8, 9)],
+ [1, 1, 1, 2, 10],
+ )
+ second = _hierarchy(
+ [(0, 2), (1, 6), (3, 7), (4, 5), (8, 9)],
+ [1, 1, 1, 2, 10],
+ )
+ return first, second
+
+
+def _event_leaf_sets(
+ hierarchy: ParisHierarchy,
+ representatives: np.ndarray,
+) -> list[set[int]]:
+ n_leaves = hierarchy.n_leaves
+ members = {leaf: {leaf} for leaf in range(n_leaves)}
+ for merge_index, children in enumerate(hierarchy.children):
+ members[n_leaves + merge_index] = (
+ members[int(children[0])] | members[int(children[1])]
+ )
+ return [members[int(node)] for node in representatives]
+
+
+def test_equal_height_binary_refinements_have_the_same_cut() -> None:
+ first, second = _plateau_variants()
+ first_forest = collapse_equal_height_plateaus(first)
+ second_forest = collapse_equal_height_plateaus(second)
+ first_result = adaptive_cut(first, 2, plateau_forest=first_forest)
+ second_result = adaptive_cut(second, 2, plateau_forest=second_forest)
+
+ assert len(first_forest.representatives) == 3
+ assert len(second_forest.representatives) == 3
+ assert np.array_equal(first_result.labels, [1, 1, 1, 1, 2, 2])
+ assert np.array_equal(second_result.labels, first_result.labels)
+
+
+def test_adaptive_score_is_optimal_over_small_event_antichains() -> None:
+ hierarchy, _other = _plateau_variants()
+ forest = collapse_equal_height_plateaus(hierarchy)
+ result = adaptive_cut(hierarchy, 2, plateau_forest=forest)
+ leaf_sets = _event_leaf_sets(hierarchy, forest.representatives)
+ best_score = -np.inf
+
+ for mask in itertools.product((False, True), repeat=len(leaf_sets)):
+ selected = [index for index, keep in enumerate(mask) if keep]
+ if not selected:
+ continue
+ coverage = np.zeros(hierarchy.n_leaves, dtype=np.int8)
+ for event in selected:
+ coverage[list(leaf_sets[event])] += 1
+ if not np.all(coverage == 1):
+ continue
+ if any(int(forest.sizes[event]) < 2 for event in selected):
+ continue
+ score = 0.0
+ for event in selected:
+ parent = int(forest.parent_events[event])
+ if parent >= 0:
+ score += int(forest.sizes[event]) * (
+ 1.0 / (1.0 + float(forest.heights[event]))
+ - 1.0 / (1.0 + float(forest.heights[parent]))
+ )
+ best_score = max(best_score, score)
+
+ selected_score = sum(
+ 0.0 if item.persistence is None else item.persistence
+ for item in result.diagnostics
+ )
+ assert selected_score == pytest.approx(best_score)
+
+
+def test_extreme_coarse_intervals_do_not_hide_canonical_scale_substructure() -> None:
+ hierarchy = _hierarchy(
+ [
+ (0, 1),
+ (2, 3),
+ (4, 5),
+ (6, 7),
+ (8, 9),
+ (10, 11),
+ (12, 13),
+ ],
+ [1, 1, 1, 1, 10, 10, 10_000],
+ )
+ result = adaptive_cut(hierarchy, 2)
+
+ assert np.bincount(result.labels)[1:].tolist() == [2, 2, 2, 2]
+ assert [item.selected_node for item in result.diagnostics] == [8, 9, 10, 11]
+ assert all(item.persistence == pytest.approx(9 / 11) for item in result.diagnostics)
+
+
+def test_nested_graph_keeps_four_durable_subcommunities() -> None:
+ hierarchy = fit_paris_hierarchy(_nested_block_graph(), n_threads=2)
+ result = adaptive_cut(hierarchy, 10)
+
+ assert result.n_clusters == 4
+ block_labels = result.labels.reshape(4, 40)
+ assert all(np.unique(block).size == 1 for block in block_labels)
+ assert np.unique(block_labels[:, 0]).size == 4
+
+
+def test_unequal_depth_tree_keeps_durable_and_splits_transient_branches() -> None:
+ hierarchy = _hierarchy(
+ [
+ (0, 1),
+ (2, 3),
+ (8, 9),
+ (4, 5),
+ (6, 7),
+ (11, 12),
+ (10, 13),
+ ],
+ [1, 1, 2, 1, 1, 50, 100],
+ )
+ result = adaptive_cut(hierarchy, 2)
+
+ assert np.bincount(result.labels)[1:].tolist() == [4, 2, 2]
+ assert [item.selected_node for item in result.diagnostics] == [10, 11, 12]
+ assert all(not item.forced for item in result.diagnostics)
+
+
+def test_exact_score_ties_keep_the_parent_event() -> None:
+ hierarchy = _hierarchy(
+ [
+ (0, 1),
+ (2, 3),
+ (8, 9),
+ (4, 5),
+ (6, 7),
+ (11, 12),
+ (10, 13),
+ ],
+ [1 / 3, 1 / 3, 1, 1 / 3, 1 / 3, 1, 3],
+ )
+ result = adaptive_cut(hierarchy, 2)
+
+ assert np.bincount(result.labels)[1:].tolist() == [4, 4]
+ assert [item.selected_node for item in result.diagnostics] == [10, 13]
+ assert all(
+ item.decision_margin == pytest.approx(0.0) for item in result.diagnostics
+ )
+
+
+def test_resolution_bounds_and_zero_height_events_are_finite_safe() -> None:
+ hierarchy = _hierarchy(
+ [(0, 1), (2, 3), (4, 5)],
+ [0, 1, 10],
+ )
+ result = adaptive_cut(hierarchy, 2)
+ zero_event = result.diagnostics[0]
+
+ assert zero_event.resolution_lower == 0
+ assert zero_event.resolution_upper == 10
+ assert zero_event.persistence == pytest.approx(20 / 11)
+ assert np.isfinite(
+ [
+ item.persistence
+ for item in result.diagnostics
+ if item.persistence is not None
+ ]
+ ).all()
+
+
+@pytest.mark.parametrize("height", [np.nan, -1.0])
+def test_nan_and_negative_heights_are_rejected(height: float) -> None:
+ hierarchy = _hierarchy([(0, 1), (2, 3), (4, 5)], [1, 1, height])
+ with pytest.raises(ValueError, match="non-negative"):
+ adaptive_cut(hierarchy, 2)
+
+
+def test_strict_multiway_folding_selects_the_nearest_valid_ancestor() -> None:
+ hierarchy = _hierarchy(
+ [(0, 1), (2, 3), (6, 7), (4, 5), (8, 9)],
+ [0.5, 0.5, 1, 2, 10],
+ )
+ result = adaptive_cut(hierarchy, 3)
+
+ assert result.n_clusters == 1
+ assert np.array_equal(result.labels, np.ones(6, dtype=np.int32))
+ diagnostic = result.diagnostics[0]
+ assert diagnostic.forced
+ assert diagnostic.blocking_child_count == 1
+ assert diagnostic.folded_cell_count == 2
+ assert diagnostic.persistence is None
+ assert diagnostic.decision_margin is None
+
+
+def test_disconnected_components_and_isolates_are_forced_and_relabelled() -> None:
+ graph = csr_matrix(
+ np.asarray(
+ [
+ [0, 1, 0, 0, 0],
+ [1, 0, 0, 0, 0],
+ [0, 0, 0, 2, 0],
+ [0, 0, 2, 0, 0],
+ [0, 0, 0, 0, 0],
+ ],
+ dtype=np.float64,
+ )
+ )
+ result = adaptive_cut(fit_paris_hierarchy(graph), 2)
+
+ assert result.labels.tolist() == [1, 1, 2, 2, 3]
+ assert all(item.forced for item in result.diagnostics)
+ assert all(item.persistence is None for item in result.diagnostics)
+ assert [item.component for item in result.diagnostics] == [0, 1, 2]
+
+
+def test_labels_are_contiguous_read_only_and_cover_every_leaf() -> None:
+ hierarchy, _other = _plateau_variants()
+ result = adaptive_cut(hierarchy, 2)
+
+ assert result.labels.dtype == np.int32
+ assert set(result.labels) == set(range(1, result.n_clusters + 1))
+ assert not result.labels.flags.writeable
+ with pytest.raises(ValueError):
+ result.labels[0] = 99
+
+
+def test_split_gate_rejects_misaligned_or_non_finite_values() -> None:
+ hierarchy, _other = _plateau_variants()
+ forest = collapse_equal_height_plateaus(hierarchy)
+ n_events = forest.representatives.size
+
+ with pytest.raises(ValueError, match="one value per plateau event"):
+ adaptive_cut(
+ hierarchy,
+ 2,
+ plateau_forest=forest,
+ split_gate=np.ones(n_events + 1),
+ )
+ with pytest.raises(TypeError, match="real numbers"):
+ adaptive_cut(
+ hierarchy,
+ 2,
+ plateau_forest=forest,
+ split_gate=np.full(n_events, "positive"),
+ )
+ with pytest.raises(ValueError, match="finite"):
+ adaptive_cut(
+ hierarchy,
+ 2,
+ plateau_forest=forest,
+ split_gate=np.full(n_events, np.nan),
+ )
+
+
+def test_adaptive_cut_accepts_numpy_integer_minimum_size() -> None:
+ hierarchy, _other = _plateau_variants()
+
+ result = adaptive_cut(hierarchy, np.int64(2))
+
+ assert result.labels.tolist() == [1, 1, 1, 1, 2, 2]
+ with pytest.raises(TypeError, match="min_cluster_size"):
+ adaptive_cut(hierarchy, np.bool_(True))
+
+
+def test_plateau_storage_scales_linearly() -> None:
+ def forest_shape_and_bytes(n_leaves: int) -> tuple[int, int, int]:
+ children: list[tuple[int, int]] = [(0, 1)]
+ heights = [1.0]
+ for leaf in range(2, n_leaves):
+ children.append((n_leaves + leaf - 2, leaf))
+ heights.append(float(leaf))
+ forest = collapse_equal_height_plateaus(_hierarchy(children, heights))
+ stored_bytes = sum(
+ values.nbytes
+ for values in (
+ forest.representatives,
+ forest.heights,
+ forest.sizes,
+ forest.parent_events,
+ forest.child_offsets,
+ forest.child_refs,
+ forest.min_leaves,
+ forest.component_roots,
+ )
+ )
+ return forest.representatives.size, forest.child_refs.size, stored_bytes
+
+ small_events, small_children, small_bytes = forest_shape_and_bytes(1_000)
+ large_events, large_children, large_bytes = forest_shape_and_bytes(2_000)
+
+ assert (small_events, large_events) == (999, 1_999)
+ assert (small_children, large_children) == (1_998, 3_998)
+ assert 1.9 * small_bytes < large_bytes < 2.1 * small_bytes
+ assert large_bytes < 100 * 2_000
diff --git a/tests/test_paris_multiscale_datastore.py b/tests/test_paris_multiscale_datastore.py
new file mode 100644
index 00000000..febc6b18
--- /dev/null
+++ b/tests/test_paris_multiscale_datastore.py
@@ -0,0 +1,900 @@
+import sys
+import warnings
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+import zarr
+from scipy.sparse import csr_matrix
+from zarr.storage import MemoryStore
+
+from scarf.clustering._paris_core import ParisHierarchy
+from scarf.clustering.paris_multiscale import PlateauForest
+from scarf.datastore._operations.clustering import _ClusteringOperationsMixin
+from scarf.datastore._operations.presentation import _PresentationOperationsMixin
+from scarf.datastore._operations.paris_persistence import (
+ LATEST_PARIS_GENERATION,
+ estimate_paris_adaptive_cut_peak_bytes,
+ estimate_paris_peak_bytes,
+ estimate_hierarchy_group_peak_bytes,
+ generation_location,
+ load_hierarchy_group,
+)
+from scarf.storage.artifacts import (
+ ArtifactRef,
+ artifact_path,
+ inspect_artifact,
+ list_artifacts,
+)
+from scarf.storage.selections import resolve_selection_artifact
+from scarf.storage.budget import ResourceBudget
+
+
+class _Cells:
+ def __init__(self, active: np.ndarray, root: zarr.Group) -> None:
+ self.active = active
+ self.data: dict[str, np.ndarray] = {
+ "I": active.copy(),
+ "ids": np.asarray([f"cell_{i}" for i in range(len(active))]),
+ }
+ self.writes: list[str] = []
+ self.root = root
+ self.N = len(active)
+ cell_data = root.create_group("cellData")
+ cell_data.create_array("I", data=active.copy())
+ cell_data.create_array("ids", data=self.data["ids"])
+
+ @property
+ def columns(self) -> list[str]:
+ return list(self.data)
+
+ def fetch_all(self, name: str) -> np.ndarray:
+ return self.data[name]
+
+ def fetch(self, name: str, key: str = "I") -> np.ndarray:
+ return self.data[name][self.data[key].astype(bool)]
+
+ def insert(
+ self,
+ name: str,
+ values: np.ndarray,
+ *,
+ fill_value: object = np.nan,
+ key: str = "I",
+ overwrite: bool = False,
+ ) -> None:
+ del overwrite
+ active = self.data[key].astype(bool)
+ incoming = np.asarray(values)
+ if incoming.shape != (int(active.sum()),):
+ raise ValueError("metadata values do not match the active key")
+ filled = np.full(active.shape, fill_value, dtype=incoming.dtype)
+ filled[active] = incoming
+ self.data[name] = filled
+ self.writes.append(name)
+ cell_data = self.root["cellData"]
+ if name in cell_data:
+ del cell_data[name]
+ cell_data.create_array(name, data=filled)
+
+
+class _Store(_ClusteringOperationsMixin, _PresentationOperationsMixin):
+ _integratedGraphsLoc = "integratedGraphs"
+
+ def __init__(
+ self,
+ graph: csr_matrix,
+ *,
+ extra_cells: int = 0,
+ ) -> None:
+ self.zw = zarr.open_group(store=MemoryStore(), mode="w")
+ active = np.zeros(graph.shape[0] + extra_cells, dtype=bool)
+ active[: graph.shape[0]] = True
+ self.cells = _Cells(active, self.zw)
+ self.nthreads = 2
+ self.resources = ResourceBudget(8 * 1024**3, self.nthreads)
+ self.zarr_mode = "r+"
+ self.graphs: dict[str, csr_matrix] = {}
+ self.latest_graph_calls = 0
+ self.raise_on_latest_graph = False
+ self.load_graph_calls = 0
+ self._write_graph("RNA/graph", graph, k=3)
+
+ def _write_graph(self, location: str, graph: csr_matrix, *, k: int) -> None:
+ group = self.zw.create_group(location, overwrite=True)
+ coo = graph.tocoo()
+ group.create_array(
+ "edges",
+ data=np.column_stack((coo.row, coo.col)).astype(np.uint64),
+ )
+ group.create_array("weights", data=coo.data.astype(np.float64))
+ group.attrs.update(
+ {
+ "n_cells": graph.shape[0],
+ "n_neighbors": k,
+ }
+ )
+ self.graphs[location] = graph
+
+ def add_integrated_graph(
+ self, label: str, graph: csr_matrix, *, k: int = 3
+ ) -> None:
+ self._write_graph(f"{self._integratedGraphsLoc}/{label}", graph, k=k)
+
+ def _resolve_integrated_graph_path(self, label: str) -> str:
+ return f"{self._integratedGraphsLoc}/{label}"
+
+ def _get_latest_keys(
+ self,
+ from_assay: str | None,
+ cell_key: str | None,
+ feat_key: str | None,
+ ) -> tuple[str, str, str]:
+ return from_assay or "RNA", cell_key or "I", feat_key or "hvgs"
+
+ def get_latest_graph_loc(
+ self,
+ from_assay: str,
+ cell_key: str,
+ feat_key: str,
+ ) -> str:
+ del from_assay, cell_key, feat_key
+ self.latest_graph_calls += 1
+ if self.raise_on_latest_graph:
+ raise AssertionError("standard graph lookup was not expected")
+ return "RNA/graph"
+
+ def _lookup_stored_graph(
+ self,
+ from_assay: str | None = None,
+ cell_key: str | None = None,
+ feat_key: str | None = None,
+ graph_loc: str | None = None,
+ ):
+ from scarf.graph.paths import AssayGraphPaths, StoredAssayGraph
+
+ if graph_loc is not None:
+ path = graph_loc
+ else:
+ path = self.get_latest_graph_loc(
+ from_assay or "RNA",
+ cell_key or "I",
+ feat_key or "hvgs",
+ )
+ return StoredAssayGraph(
+ paths=AssayGraphPaths(
+ normalized_group_path="RNA/normed__I__hvgs",
+ reduction_group_path="RNA/normed__I__hvgs/reduction__pca__10__I",
+ neighbor_index_group_path=(
+ "RNA/normed__I__hvgs/reduction__pca__10__I/ann__l2__50__50__16__1"
+ ),
+ nearest_neighbors_group_path=(
+ "RNA/normed__I__hvgs/reduction__pca__10__I/"
+ "ann__l2__50__50__16__1/knn__11"
+ ),
+ cell_graph_group_path=path,
+ ),
+ from_assay=from_assay or "RNA",
+ cell_key=cell_key or "I",
+ feat_key=feat_key or "hvgs",
+ )
+
+ def _get_graph_ncells_k(self, graph_loc: str) -> tuple[int, int]:
+ attrs = self.zw[graph_loc].attrs
+ return int(attrs["n_cells"]), int(attrs["n_neighbors"])
+
+ def load_graph(self, *, graph_loc: str, **_kwargs: object) -> csr_matrix:
+ self.load_graph_calls += 1
+ return self.graphs[graph_loc]
+
+ @staticmethod
+ def _col_renamer(from_assay: str, cell_key: str, label: str) -> str:
+ if cell_key == "I":
+ return f"{from_assay}_{label}"
+ return f"{from_assay}_{cell_key}_{label}"
+
+ def get_cell_vals(
+ self,
+ *,
+ from_assay: str,
+ cell_key: str,
+ k: str,
+ ) -> np.ndarray:
+ del from_assay
+ return self.cells.fetch(k, key=cell_key)
+
+ def _ensure_cell_selection(self, column: str) -> ArtifactRef:
+ return resolve_selection_artifact(
+ self.zw,
+ scope="datastore",
+ kind="cell_selection",
+ values=self.cells.fetch_all(column),
+ row_ids=self.cells.fetch_all("ids"),
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column=column,
+ )
+
+
+def _block_graph() -> csr_matrix:
+ graph = np.zeros((14, 14), dtype=np.float64)
+ for start, stop in ((0, 6), (6, 14)):
+ rows, columns = np.triu_indices(stop - start, k=1)
+ graph[start + rows, start + columns] = 1
+ graph[5, 6] = 0.01
+ return csr_matrix(graph)
+
+
+def _disconnected_graph() -> csr_matrix:
+ edges = (
+ (0, 1, 9.0),
+ (1, 2, 4.0),
+ (2, 3, 1.0),
+ (4, 5, 8.0),
+ (5, 6, 3.0),
+ (6, 7, 0.5),
+ )
+ return csr_matrix(
+ (
+ [weight for _left, _right, weight in edges],
+ (
+ [left for left, _right, _weight in edges],
+ [right for _left, right, _weight in edges],
+ ),
+ ),
+ shape=(8, 8),
+ )
+
+
+def _load_artifact_hierarchy(
+ store: _Store,
+ artifact_id: str,
+) -> tuple[ParisHierarchy, PlateauForest]:
+ ref = ArtifactRef(
+ scope="assay",
+ assay="RNA",
+ kind="cluster_hierarchy",
+ artifact_id=artifact_id,
+ )
+ group = store.zw[artifact_path(ref)]
+ return load_hierarchy_group(group, artifact_id)
+
+
+def _column_artifact(store: _Store, column: str) -> ArtifactRef:
+ raw_ref = store.zw["cellData"][column].attrs["source_artifact"]
+ return ArtifactRef.from_dict(raw_ref)
+
+
+def test_auto_cut_persists_typed_hierarchy_and_reuses_diagnostics() -> None:
+ store = _Store(_block_graph(), extra_cells=2)
+ first = store.run_paris_clustering(min_cluster_size=2)
+ graph_group = store.zw["RNA/graph"]
+ generation_id = first.hierarchy_generation_id
+ assert generation_id is not None
+ hierarchy, forest = _load_artifact_hierarchy(store, generation_id)
+
+ assert hierarchy.n_leaves == 14
+ assert forest.n_leaves == 14
+ assert LATEST_PARIS_GENERATION not in graph_group.attrs
+ assert "latest_dendrogram" not in graph_group.attrs
+ assert first.n_clusters == 2
+ assert first.labels[:6].tolist() == [1] * 6
+ assert first.labels[6:].tolist() == [2] * 8
+ assert store.cells.data["RNA_paris_cluster"][-2:].tolist() == [-1, -1]
+ assert set(store.cells.data["RNA_paris_cluster"][:-2]) == {1, 2}
+ assert store.cells.writes == ["RNA_paris_cluster"]
+ assert first.ref is not None
+ assert first.ref == ArtifactRef.from_dict(
+ store.zw["cellData/RNA_paris_cluster"].attrs["source_artifact"]
+ )
+
+ second = store.run_paris_clustering(min_cluster_size=2)
+ assert second.ref == first.ref
+ assert second.hierarchy_generation_id == generation_id
+ assert np.array_equal(second.labels, first.labels)
+ assert second.diagnostics == first.diagnostics
+ assert store.load_graph_calls == 1
+ assert store.cells.writes == ["RNA_paris_cluster", "RNA_paris_cluster"]
+
+
+def test_fit_uses_additive_graph_and_fixed_cut_materializes_linkage_lazily() -> None:
+ graph = _block_graph()
+ store = _Store(graph)
+ result = store.run_paris_clustering(n_clusters=3)
+ generation_id = result.hierarchy_generation_id
+ assert generation_id is not None
+ hierarchy, _forest = _load_artifact_hierarchy(store, generation_id)
+ expected_children = np.asarray(
+ [
+ [0, 1],
+ [7, 8],
+ [2, 3],
+ [9, 10],
+ [4, 14],
+ [11, 12],
+ [13, 15],
+ [16, 18],
+ [5, 21],
+ [17, 19],
+ [20, 23],
+ [6, 24],
+ [22, 25],
+ ],
+ dtype=np.int64,
+ )
+ expected_heights = np.asarray(
+ [
+ 0.2906300860265055,
+ 0.5696349686119507,
+ 0.2906300860265055,
+ 0.5696349686119507,
+ 0.2906300860265055,
+ 0.5696349686119507,
+ 0.5696349686119507,
+ 0.2906300860265055,
+ 0.2912113461985585,
+ 0.5696349686119507,
+ 0.5696349686119507,
+ 0.5704487328528249,
+ 1954.034061846082,
+ ]
+ )
+
+ np.testing.assert_array_equal(hierarchy.children, expected_children)
+ np.testing.assert_allclose(hierarchy.heights, expected_heights, rtol=1e-12)
+ dendrograms = list_artifacts(
+ store.zw,
+ scope="assay",
+ assay="RNA",
+ kind="dendrogram",
+ )
+ assert len(dendrograms) == 1
+ assert artifact_path(dendrograms[0]) in store.zw
+ assert "latest_dendrogram" not in store.zw["RNA/graph"].attrs
+ assert result.mode == "fixed"
+ assert result.n_clusters == 3
+ assert set(result.labels) == {1, 2, 3}
+ assert result.diagnostics == ()
+
+
+def test_fixed_cut_uses_raw_hierarchy_for_disconnected_graph() -> None:
+ store = _Store(_disconnected_graph())
+
+ result = store.run_paris_clustering(n_clusters=3)
+
+ assert result.n_clusters == 3
+ assert np.unique(result.labels).size == 3
+ assert result.labels[0] == result.labels[1]
+ assert result.labels[2] == result.labels[3]
+ assert np.unique(result.labels[4:]).size == 1
+ assert len({result.labels[0], result.labels[2], result.labels[4]}) == 3
+ generation_id = result.hierarchy_generation_id
+ assert generation_id is not None
+ hierarchy, _forest = _load_artifact_hierarchy(store, generation_id)
+ assert hierarchy.synthetic_joins.sum() == 1
+ dendrogram_ref = list_artifacts(
+ store.zw,
+ scope="assay",
+ assay="RNA",
+ kind="dendrogram",
+ )[0]
+ compatibility = np.asarray(store.zw[artifact_path(dendrogram_ref)]["data"][:])
+ assert compatibility[hierarchy.synthetic_joins, 2].tolist() == [0.0]
+
+
+def test_auto_cut_applies_the_modularity_split_gate(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ import scarf.clustering._paris_modularity as modularity
+
+ store = _Store(_block_graph())
+ seen_graph_shapes: list[tuple[int, int]] = []
+
+ def veto_all_splits(
+ _hierarchy: ParisHierarchy,
+ forest: PlateauForest,
+ graph: csr_matrix,
+ ) -> np.ndarray:
+ seen_graph_shapes.append(graph.shape)
+ return np.zeros(forest.representatives.size, dtype=np.float64)
+
+ monkeypatch.setattr(modularity, "modularity_split_gains", veto_all_splits)
+ result = store.run_paris_clustering(min_cluster_size=2)
+
+ assert seen_graph_shapes == [(14, 14)]
+ assert result.n_clusters == 1
+ assert result.labels.tolist() == [1] * 14
+
+
+def test_auto_cut_defaults_minimum_cluster_size_to_graph_k_plus_one() -> None:
+ store = _Store(_block_graph())
+
+ result = store.run_paris_clustering()
+
+ assert result.min_cluster_size == 4
+
+
+def test_incomplete_adaptive_cache_is_recomputed() -> None:
+ store = _Store(_block_graph())
+ first = store.run_paris_clustering(min_cluster_size=2)
+ first_cut = _column_artifact(store, "RNA_paris_cluster")
+ del store.zw[artifact_path(first_cut)]["labels"]
+
+ second = store.run_paris_clustering(min_cluster_size=2)
+ second_cut = _column_artifact(store, "RNA_paris_cluster")
+
+ assert np.array_equal(second.labels, first.labels)
+ assert second_cut != first_cut
+ assert "labels" in store.zw[artifact_path(second_cut)]
+ assert second.hierarchy_generation_id == first.hierarchy_generation_id
+ assert store.load_graph_calls == 2
+
+
+def test_force_recalculation_retains_only_referenced_generations() -> None:
+ store = _Store(_block_graph())
+ first = store.run_paris_clustering(label="first", min_cluster_size=2)
+ second = store.run_paris_clustering(
+ label="second",
+ min_cluster_size=2,
+ force_recalc=True,
+ )
+ assert first.hierarchy_generation_id != second.hierarchy_generation_id
+ hierarchy_ids = {
+ ref.artifact_id
+ for ref in list_artifacts(
+ store.zw,
+ scope="assay",
+ assay="RNA",
+ kind="cluster_hierarchy",
+ )
+ }
+ assert hierarchy_ids == {
+ first.hierarchy_generation_id,
+ second.hierarchy_generation_id,
+ }
+
+ store.run_paris_clustering(n_clusters=2, label="first")
+ assert {
+ ref.artifact_id
+ for ref in list_artifacts(
+ store.zw,
+ scope="assay",
+ assay="RNA",
+ kind="cluster_hierarchy",
+ )
+ } == hierarchy_ids
+ assert "paris_hierarchy" not in store.zw["RNA/graph"]
+
+
+def test_stale_generation_pointer_recomputes_without_crashing() -> None:
+ store = _Store(_block_graph())
+ first = store.run_paris_clustering(n_clusters=2)
+ # A released-layout pointer must not influence new artifact reuse.
+ store.zw["RNA/graph"].attrs[LATEST_PARIS_GENERATION] = "missing-generation"
+ assert generation_location("RNA/graph", "missing-generation") not in store.zw
+
+ second = store.run_paris_clustering(n_clusters=2)
+
+ assert second.hierarchy_generation_id == first.hierarchy_generation_id
+ assert store.zw["RNA/graph"].attrs[LATEST_PARIS_GENERATION] == "missing-generation"
+ assert np.array_equal(second.labels, first.labels)
+
+
+def test_adaptive_cache_collection_prunes_only_unusable_configurations() -> None:
+ store = _Store(_block_graph())
+ first = store.run_paris_clustering(
+ min_cluster_size=2,
+ label="small",
+ )
+ second = store.run_paris_clustering(
+ min_cluster_size=3,
+ label="large",
+ )
+
+ cuts = list_artifacts(
+ store.zw,
+ scope="assay",
+ assay="RNA",
+ kind="cluster_cut",
+ )
+ assert len(cuts) == 2
+ assert first.hierarchy_generation_id == second.hierarchy_generation_id
+ assert "adaptive_clustering" not in store.zw["RNA/graph"]
+
+
+def test_stale_legacy_dendrogram_warns_once_and_is_not_reused() -> None:
+ store = _Store(_block_graph())
+ graph_group = store.zw["RNA/graph"]
+ graph_group.create_array("dendrogram", data=np.zeros((13, 4)))
+ graph_group.attrs["latest_dendrogram"] = "RNA/graph/dendrogram"
+
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ first = store.run_paris_clustering(n_clusters=2)
+ assert caught == []
+ second = store.run_paris_clustering(n_clusters=2)
+
+ assert first.hierarchy_generation_id == second.hierarchy_generation_id
+ assert (
+ str(store.zw["RNA/graph"].attrs["latest_dendrogram"]) == "RNA/graph/dendrogram"
+ )
+ assert (
+ len(
+ list_artifacts(
+ store.zw,
+ scope="assay",
+ assay="RNA",
+ kind="dendrogram",
+ )
+ )
+ == 1
+ )
+
+
+def test_integrated_graph_is_resolved_without_standard_graph_lookup() -> None:
+ graph = _block_graph()
+ store = _Store(graph)
+ store.add_integrated_graph("joint", graph)
+ store.raise_on_latest_graph = True
+
+ result = store.run_paris_clustering(
+ integrated_graph="joint",
+ min_cluster_size=2,
+ )
+
+ assert result.label_key == "joint_paris_cluster"
+ assert LATEST_PARIS_GENERATION not in store.zw["integratedGraphs/joint"].attrs
+ assert (
+ len(
+ list_artifacts(
+ store.zw,
+ scope="datastore",
+ kind="cluster_hierarchy",
+ )
+ )
+ == 1
+ )
+ assert store.latest_graph_calls == 0
+
+
+def test_memory_preflight_fails_before_loading_edges() -> None:
+ store = _Store(_block_graph())
+ store.resources = ResourceBudget(memoryBytes=1, workers=1)
+ with pytest.raises(MemoryError, match="resource budget"):
+ store.run_paris_clustering()
+
+ assert store.load_graph_calls == 0
+ assert LATEST_PARIS_GENERATION not in store.zw["RNA/graph"].attrs
+
+
+def test_cached_hierarchy_adaptive_preflight_fails_before_graph_load() -> None:
+ store = _Store(_block_graph())
+ fixed = store.run_paris_clustering(n_clusters=2)
+ assert store.load_graph_calls == 1
+ generation_id = fixed.hierarchy_generation_id
+ assert generation_id is not None
+ hierarchy_ref = ArtifactRef(
+ scope="assay",
+ assay="RNA",
+ kind="cluster_hierarchy",
+ artifact_id=generation_id,
+ )
+ cached_estimate = estimate_hierarchy_group_peak_bytes(
+ store.zw[artifact_path(hierarchy_ref)],
+ "adaptive",
+ )
+ graph_group = store.zw["RNA/graph"]
+ edges = graph_group["edges"]
+ weights = graph_group["weights"]
+ graph_estimate = estimate_paris_adaptive_cut_peak_bytes(
+ 14,
+ int(edges.shape[0]),
+ np.dtype(edges.dtype).itemsize,
+ np.dtype(weights.dtype).itemsize,
+ )
+ assert graph_estimate > cached_estimate
+
+ store.resources = ResourceBudget(
+ memoryBytes=(cached_estimate + graph_estimate) // 2,
+ workers=1,
+ )
+ with pytest.raises(MemoryError, match="^Paris adaptive cut"):
+ store.run_paris_clustering(
+ n_clusters="auto",
+ min_cluster_size=2,
+ label="guarded",
+ )
+
+ assert store.load_graph_calls == 1
+ assert "RNA/graph/adaptive_clustering/guarded" not in store.zw
+
+
+def test_cached_fixed_and_adaptive_cuts_preflight_hierarchy_loading() -> None:
+ store = _Store(_block_graph())
+ store.run_paris_clustering(n_clusters=2, label="fixed")
+ store.run_paris_clustering(
+ n_clusters="auto",
+ min_cluster_size=2,
+ label="adaptive",
+ )
+ writes_before = store.cells.writes.copy()
+ loads_before = store.load_graph_calls
+
+ store.resources = ResourceBudget(memoryBytes=1, workers=1)
+ with pytest.raises(MemoryError, match="Cached Paris fixed cut"):
+ store.run_paris_clustering(n_clusters=3, label="other_fixed")
+ with pytest.raises(MemoryError, match="Cached Paris adaptive cut"):
+ store.run_paris_clustering(
+ n_clusters="auto",
+ min_cluster_size=2,
+ label="adaptive",
+ )
+
+ assert store.cells.writes == writes_before
+ assert store.load_graph_calls == loads_before
+
+
+def test_memory_estimate_accounts_for_parallel_contraction_tables() -> None:
+ n_cells = 500_000
+ serial = estimate_paris_peak_bytes(
+ n_cells,
+ 7_500_000,
+ 4,
+ 4,
+ n_threads=1,
+ )
+ parallel = estimate_paris_peak_bytes(
+ n_cells,
+ 7_500_000,
+ 4,
+ 4,
+ n_threads=8,
+ )
+
+ assert parallel > serial
+ assert parallel - serial >= n_cells * (8 - 1) * 8
+ adaptive_cut_peak = estimate_paris_adaptive_cut_peak_bytes(
+ n_cells,
+ 7_500_000,
+ 4,
+ 4,
+ )
+ assert 0 < adaptive_cut_peak < serial
+
+
+def test_memory_estimate_switches_to_int64_before_doubled_counts_overflow() -> None:
+ int32_max = int(np.iinfo(np.int32).max)
+ edge_threshold = int32_max // 2
+ low_edge_estimate = estimate_paris_adaptive_cut_peak_bytes(
+ 1_000,
+ edge_threshold,
+ 4,
+ 4,
+ )
+ high_edge_estimate = estimate_paris_adaptive_cut_peak_bytes(
+ 1_000,
+ edge_threshold + 1,
+ 4,
+ 4,
+ )
+ leaf_threshold = (int32_max + 1) // 2
+ low_leaf_estimate = estimate_paris_adaptive_cut_peak_bytes(
+ leaf_threshold,
+ 1,
+ 4,
+ 4,
+ )
+ high_leaf_estimate = estimate_paris_adaptive_cut_peak_bytes(
+ leaf_threshold + 1,
+ 1,
+ 4,
+ 4,
+ )
+
+ assert high_edge_estimate > low_edge_estimate * 1.2
+ assert high_leaf_estimate > low_leaf_estimate * 1.2
+
+
+def test_interrupted_replacement_keeps_previous_generation_authoritative(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ import scarf.datastore._operations.paris_persistence as cache
+
+ store = _Store(_block_graph())
+ first = store.run_paris_clustering(min_cluster_size=2)
+ previous_generation = first.hierarchy_generation_id
+ previous_cut = _column_artifact(store, "RNA_paris_cluster")
+
+ def fail_write(*_args: object, **_kwargs: object) -> None:
+ raise OSError("simulated interrupted write")
+
+ monkeypatch.setattr(cache, "write_hierarchy_group", fail_write)
+ with pytest.raises(OSError, match="interrupted"):
+ store.run_paris_clustering(force_recalc=True)
+
+ assert _column_artifact(store, "RNA_paris_cluster") == previous_cut
+ hierarchy_ref = ArtifactRef(
+ scope="assay",
+ assay="RNA",
+ kind="cluster_hierarchy",
+ artifact_id=previous_generation,
+ )
+ assert inspect_artifact(store.zw, hierarchy_ref).complete
+ assert store.cells.writes == ["RNA_paris_cluster"]
+
+
+def test_graph_rebuild_clears_generation_pointer() -> None:
+ graph = _block_graph()
+ store = _Store(graph)
+ first = store.run_paris_clustering(min_cluster_size=2)
+ store._write_graph("RNA/graph", graph * 2, k=3)
+ assert LATEST_PARIS_GENERATION not in store.zw["RNA/graph"].attrs
+
+ second = store.run_paris_clustering(min_cluster_size=2)
+ assert second.hierarchy_generation_id != first.hierarchy_generation_id
+
+
+def test_cluster_tree_cache_tracks_generation_and_cluster_identity() -> None:
+ store = _Store(_block_graph())
+ store.run_paris_clustering(
+ min_cluster_size=2,
+ label="first",
+ )
+ prepared = store._prepare_cluster_tree(cluster_key="RNA_first")
+ cached = store._prepare_cluster_tree(cluster_key="RNA_first")
+ assert prepared["coalesced_location"] == cached["coalesced_location"]
+
+ values = store.cells.data["RNA_first"].copy()
+ values[values == 1] = -1
+ values[values == 2] = 1
+ values[values == -1] = 2
+ store.cells.data["RNA_first"] = values
+ changed = store._prepare_cluster_tree(cluster_key="RNA_first")
+ assert changed["coalesced_location"] == prepared["coalesced_location"]
+
+ store.run_paris_clustering(
+ n_clusters=3,
+ label="second",
+ )
+ second_tree = store._prepare_cluster_tree(cluster_key="RNA_second")
+ assert second_tree["coalesced_location"] != prepared["coalesced_location"]
+
+
+def test_topacedo_uses_the_generation_recorded_for_adaptive_labels(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ store = _Store(_block_graph())
+ result = store.run_paris_clustering(min_cluster_size=2)
+ captured: dict[str, np.ndarray] = {}
+
+ class Sampler:
+ def __init__(
+ self,
+ _graph: csr_matrix,
+ _clusters: np.ndarray,
+ dendrogram: np.ndarray,
+ *_args: object,
+ ) -> None:
+ captured["dendrogram"] = dendrogram
+ n_cells = len(_clusters)
+ self.densities = np.ones(n_cells)
+ self.meanSnn = np.ones(n_cells)
+ self.seeds = np.asarray([0], dtype=np.int64)
+ self.rand_state = int(_args[-1])
+
+ def run(self) -> tuple[np.ndarray, list[tuple[int, int]]]:
+ if self.rand_state == 99:
+ return np.asarray([0], dtype=np.int64), []
+ if self.rand_state == 100:
+ return np.asarray([0], dtype=np.int64), [(0, len(self.densities))]
+ return np.asarray([0, 1], dtype=np.int64), [(0, 1)]
+
+ monkeypatch.setitem(
+ sys.modules,
+ "topacedo",
+ SimpleNamespace(TopacedoSampler=Sampler),
+ )
+ monkeypatch.setattr(
+ "scarf.datastore._operations.clustering.validate_legacy_graph_selection",
+ lambda *_args, **_kwargs: None,
+ )
+ sampling_ref = store.run_topacedo_sampler(cluster_key=result.label_key)
+
+ output_columns = [
+ "RNA_sketched",
+ "RNA_cell_density",
+ "RNA_snn_value",
+ "RNA_sketch_seeds",
+ ]
+ output_refs = {
+ ArtifactRef.from_dict(store.zw[f"cellData/{column}"].attrs["source_artifact"])
+ for column in output_columns
+ }
+ assert output_refs == {sampling_ref}
+ assert sampling_ref.kind == "sampling"
+ sampling_group = store.zw[artifact_path(sampling_ref)]
+ assert set(sampling_group.array_keys()) == {
+ "sampled",
+ "density",
+ "mean_snn",
+ "seeds",
+ "edges",
+ }
+ np.testing.assert_array_equal(sampling_group["edges"][:], [[0, 1]])
+ monkeypatch.setitem(sys.modules, "topacedo", None)
+ assert store.run_topacedo_sampler(cluster_key=result.label_key) == sampling_ref
+ monkeypatch.setitem(
+ sys.modules,
+ "topacedo",
+ SimpleNamespace(TopacedoSampler=Sampler),
+ )
+ empty_ref = store.run_topacedo_sampler(
+ cluster_key=result.label_key,
+ rand_state=99,
+ )
+ empty_group = store.zw[artifact_path(empty_ref)]
+ assert empty_group["edges"].shape == (0, 2)
+ assert empty_group["edges"].chunks == (1, 2)
+ with pytest.raises(ValueError, match="edge endpoints"):
+ store.run_topacedo_sampler(
+ cluster_key=result.label_key,
+ rand_state=100,
+ )
+ assert captured["dendrogram"].shape == (13, 4)
+ dendrogram_ref = list_artifacts(
+ store.zw,
+ scope="assay",
+ assay="RNA",
+ kind="dendrogram",
+ )[0]
+ dendrogram_inputs = inspect_artifact(store.zw, dendrogram_ref).inputs
+ assert dendrogram_inputs is not None
+ assert (
+ ArtifactRef.from_dict(dendrogram_inputs["cluster_hierarchy"]).artifact_id
+ == result.hierarchy_generation_id
+ )
+
+
+@pytest.mark.parametrize(
+ ("arguments", "error_type", "message"),
+ [
+ ({"n_clusters": True}, TypeError, "integer or 'auto'"),
+ ({"n_clusters": np.bool_(True)}, TypeError, "integer or 'auto'"),
+ ({"n_clusters": "unknown"}, ValueError, "integer or 'auto'"),
+ ({"n_clusters": 0}, ValueError, "positive"),
+ ({"n_clusters": 15}, ValueError, "graph size"),
+ (
+ {"n_clusters": 2, "min_cluster_size": 3},
+ ValueError,
+ "only valid",
+ ),
+ (
+ {"n_clusters": "auto", "min_cluster_size": 1},
+ ValueError,
+ "at least 2",
+ ),
+ ],
+)
+def test_run_paris_clustering_validates_cut_arguments(
+ arguments: dict[str, object],
+ error_type: type[Exception],
+ message: str,
+) -> None:
+ store = _Store(_block_graph())
+ with pytest.raises(error_type, match=message):
+ store.run_paris_clustering(**arguments)
+
+
+def test_run_paris_clustering_accepts_numpy_integer_cut_parameters() -> None:
+ store = _Store(_block_graph())
+
+ fixed = store.run_paris_clustering(n_clusters=np.int64(2), label="fixed")
+ adaptive = store.run_paris_clustering(
+ n_clusters="auto",
+ min_cluster_size=np.int32(2),
+ label="adaptive",
+ )
+
+ assert fixed.n_clusters == 2
+ assert adaptive.labels.shape == (14,)
diff --git a/tests/test_paris_quality_gate.py b/tests/test_paris_quality_gate.py
new file mode 100644
index 00000000..2a4b7cc3
--- /dev/null
+++ b/tests/test_paris_quality_gate.py
@@ -0,0 +1,282 @@
+import numpy as np
+import pytest
+from scipy.sparse import csr_matrix
+
+from tests.paris_quality_gate import (
+ MAX_ADAPTIVE_CLUSTERS,
+ MIN_RARE_CLUSTER_F1,
+ ORACLE_ARI_TOLERANCE,
+ SUPPORTED_ALPHA_BETA_RATIOS,
+ _dense_event_pair_statistics,
+ _encoded_truth,
+ _event_pair_statistics,
+ _sparse_event_pair_statistics,
+ evaluate_hierarchy_headroom,
+ evaluate_quality_gate,
+ exact_pair_metrics,
+ exhaustive_pair_cuts,
+ maximum_pair_f1_cut,
+ oracle_gap_recovery,
+ supported_pair_cuts,
+)
+from scarf.clustering._paris_core import ParisHierarchy
+from scarf.clustering.paris_multiscale import (
+ collapse_equal_height_plateaus,
+ labels_from_selected_nodes,
+)
+
+
+def _hierarchy(
+ children: list[tuple[int, int]],
+ heights: list[float],
+ *,
+ component_roots: list[int] | None = None,
+ synthetic_joins: list[bool] | None = None,
+) -> ParisHierarchy:
+ n_leaves = len(children) + 1
+ child_array = np.asarray(children, dtype=np.int32)
+ sizes = np.ones(2 * n_leaves - 1, dtype=np.int32)
+ merge_sizes = np.empty(n_leaves - 1, dtype=np.int32)
+ for merge_index, (left, right) in enumerate(children):
+ size = int(sizes[left]) + int(sizes[right])
+ merge_sizes[merge_index] = size
+ sizes[n_leaves + merge_index] = size
+ roots = (
+ np.asarray([2 * n_leaves - 2], dtype=np.int32)
+ if component_roots is None
+ else np.asarray(component_roots, dtype=np.int32)
+ )
+ synthetic = (
+ np.isinf(heights)
+ if synthetic_joins is None
+ else np.asarray(synthetic_joins, dtype=bool)
+ )
+ return ParisHierarchy(
+ children=child_array,
+ heights=np.asarray(heights, dtype=np.float64),
+ sizes=merge_sizes,
+ component_roots=roots,
+ synthetic_joins=np.asarray(synthetic, dtype=bool),
+ n_leaves=n_leaves,
+ total_weight=1.0,
+ )
+
+
+def _tiny_oracle_cases() -> tuple[
+ tuple[ParisHierarchy, np.ndarray, int],
+ ...,
+]:
+ balanced_children: list[tuple[int, int]] = []
+ balanced_heights: list[float] = []
+ level = list(range(16))
+ height = 1.0
+ while len(level) > 1:
+ next_level: list[int] = []
+ for offset in range(0, len(level), 2):
+ balanced_children.append((level[offset], level[offset + 1]))
+ balanced_heights.append(height)
+ next_level.append(16 + len(balanced_children) - 1)
+ level = next_level
+ height *= 2
+ balanced = _hierarchy(balanced_children, balanced_heights)
+ equal_height_plateau = _hierarchy(
+ [(0, 1), (2, 3), (6, 7), (4, 5), (8, 9)],
+ [1, 1, 1, 2, 10],
+ )
+ minimum_size_folding = _hierarchy(
+ [(0, 1), (2, 3), (6, 7), (4, 5), (8, 9)],
+ [0.5, 0.5, 1, 2, 10],
+ )
+ disconnected_with_isolate = _hierarchy(
+ [(0, 1), (2, 3), (5, 6), (7, 4)],
+ [1, 1, 4, np.inf],
+ component_roots=[7, 4],
+ synthetic_joins=[False, False, False, True],
+ )
+ return (
+ (
+ balanced,
+ np.asarray([0, 0, 1, 1, 0, 2, 2, 2, 3, 3, 3, 4, 4, 5, 5, 5]),
+ 2,
+ ),
+ (
+ equal_height_plateau,
+ np.asarray([0, 0, 1, 1, 2, 2]),
+ 2,
+ ),
+ (
+ minimum_size_folding,
+ np.asarray([0, 0, 1, 1, 2, 2]),
+ 3,
+ ),
+ (
+ disconnected_with_isolate,
+ np.asarray([0, 0, 1, 1, 0]),
+ 2,
+ ),
+ )
+
+
+def test_exact_pair_metrics_uses_integer_pair_counts() -> None:
+ metrics = exact_pair_metrics(
+ np.asarray([1, 1, 1, 2]),
+ np.asarray(["a", "a", "b", "b"]),
+ )
+
+ assert metrics.true_positive_pairs == 1
+ assert metrics.false_positive_pairs == 2
+ assert metrics.false_negative_pairs == 1
+ assert metrics.true_negative_pairs == 2
+ assert metrics.predicted_positive_pairs == 3
+ assert metrics.truth_positive_pairs == 2
+ assert metrics.total_pairs == 6
+ assert metrics.precision == pytest.approx(1 / 3)
+ assert metrics.recall == pytest.approx(1 / 2)
+ assert metrics.f1 == pytest.approx(2 / 5)
+
+
+def test_pair_metrics_excludes_missing_and_unknown_truth() -> None:
+ metrics = exact_pair_metrics(
+ np.asarray([1, 1, 2, 2]),
+ np.asarray(["a", "unknown", None, "a"], dtype=object),
+ )
+
+ assert metrics.true_positive_pairs == 0
+ assert metrics.false_positive_pairs == 0
+ assert metrics.false_negative_pairs == 1
+ assert metrics.true_negative_pairs == 0
+
+
+def test_sparse_and_dense_event_histograms_are_identical() -> None:
+ hierarchy, truth, _min_cluster_size = _tiny_oracle_cases()[0]
+ truth = truth.astype(object)
+ truth[3] = "unknown"
+ forest = collapse_equal_height_plateaus(hierarchy)
+ codes, evaluated_items = _encoded_truth(truth)
+ n_classes = int(codes.max()) + 1
+
+ dense = _dense_event_pair_statistics(
+ forest,
+ codes,
+ evaluated_items,
+ n_classes,
+ )
+ sparse = _sparse_event_pair_statistics(
+ forest,
+ codes,
+ evaluated_items,
+ )
+
+ assert dense == sparse
+ assert dense == _event_pair_statistics(forest, truth)
+
+
+@pytest.mark.parametrize(
+ ("hierarchy", "truth", "min_cluster_size"),
+ _tiny_oracle_cases(),
+)
+def test_oracles_match_every_exhaustive_tiny_tree_objective(
+ hierarchy: ParisHierarchy,
+ truth: np.ndarray,
+ min_cluster_size: int,
+) -> None:
+ forest = collapse_equal_height_plateaus(hierarchy)
+ exhaustive = exhaustive_pair_cuts(forest, truth, min_cluster_size)
+ exhaustive_nodes = {cut.selected_nodes for cut in exhaustive}
+
+ for cut in exhaustive:
+ labels = labels_from_selected_nodes(
+ hierarchy,
+ np.asarray(cut.selected_nodes, dtype=np.int32),
+ )
+ assert exact_pair_metrics(labels, truth) == cut.metrics
+
+ maximum = maximum_pair_f1_cut(forest, truth, min_cluster_size)
+ best_f1 = max(cut.metrics.f1 for cut in exhaustive)
+ assert maximum.cut.metrics.f1 == pytest.approx(best_f1)
+ assert maximum.cut.selected_nodes in exhaustive_nodes
+
+ supported = supported_pair_cuts(forest, truth, min_cluster_size)
+ assert [cut.alpha_beta_ratio for cut in supported] == pytest.approx(
+ SUPPORTED_ALPHA_BETA_RATIOS
+ )
+ for supported_cut in supported:
+ exact_minimum = min(
+ supported_cut.alpha * cut.metrics.false_positive_pairs
+ + supported_cut.beta * cut.metrics.false_negative_pairs
+ for cut in exhaustive
+ )
+ assert supported_cut.weighted_error == exact_minimum
+ assert supported_cut.cut.selected_nodes in exhaustive_nodes
+
+
+def test_minimum_size_folding_forces_the_nearest_valid_ancestor() -> None:
+ hierarchy, truth, min_cluster_size = _tiny_oracle_cases()[2]
+ forest = collapse_equal_height_plateaus(hierarchy)
+
+ exhaustive = exhaustive_pair_cuts(forest, truth, min_cluster_size)
+ maximum = maximum_pair_f1_cut(forest, truth, min_cluster_size)
+
+ assert [cut.selected_nodes for cut in exhaustive] == [(10,)]
+ assert maximum.cut.selected_nodes == (10,)
+
+
+def test_headroom_evaluation_keeps_the_existing_cut_as_baseline() -> None:
+ hierarchy = _hierarchy(
+ [(0, 1), (2, 3), (4, 5)],
+ [1, 1, 10],
+ )
+ truth = np.asarray(["same"] * 4)
+ graph = csr_matrix(
+ (
+ np.ones(6, dtype=np.float64),
+ (
+ np.asarray([0, 1, 2, 3, 0, 2]),
+ np.asarray([1, 0, 3, 2, 2, 0]),
+ ),
+ ),
+ shape=(4, 4),
+ )
+
+ evaluation = evaluate_hierarchy_headroom(hierarchy, truth, graph=graph)
+
+ assert evaluation.baseline.f1 == pytest.approx(0.5)
+ assert evaluation.maximum_pair_f1.cut.selected_nodes == (6,)
+ assert evaluation.maximum_pair_f1.cut.metrics.f1 == pytest.approx(1.0)
+ assert evaluation.pair_f1_headroom == pytest.approx(0.5)
+ assert evaluation.has_measurable_headroom
+ assert len(evaluation.supported_frontier) == 1
+
+
+@pytest.mark.parametrize(
+ ("scorer", "baseline", "maximum", "expected"),
+ [
+ (0.75, 0.5, 1.0, 0.5),
+ (0.5, 0.5, 0.5, 0.0),
+ ],
+)
+def test_oracle_gap_recovery(
+ scorer: float,
+ baseline: float,
+ maximum: float,
+ expected: float,
+) -> None:
+ assert oracle_gap_recovery(scorer, baseline, maximum) == pytest.approx(expected)
+
+
+def test_unequal_depth_quality_gate_preserves_the_rare_branch() -> None:
+ report = evaluate_quality_gate(seed_count=5, n_threads=2)
+
+ assert report.passed
+ accepted = [result for result in report.seeds if result.accepted]
+ assert accepted, "no seed satisfied the discriminative acceptance criteria"
+ for result in accepted:
+ assert result.adaptive_rare_f1 >= MIN_RARE_CLUSTER_F1
+ assert result.adaptive_clusters <= MAX_ADAPTIVE_CLUSTERS
+ # The guarded cut must match the oracle horizontal cut that scans every
+ # cluster count, so a silent regression to a worse partition fails here.
+ assert result.adaptive_ari >= result.best_global_ari - ORACLE_ARI_TOLERANCE
+ # The modularity guard may only coarsen the unguarded cut and must not
+ # sacrifice recovered structure while doing so.
+ assert result.adaptive_clusters <= result.unguarded_clusters
+ assert result.adaptive_ari >= result.unguarded_ari - ORACLE_ARI_TOLERANCE
diff --git a/tests/test_pcst_fast.py b/tests/test_pcst_fast.py
new file mode 100644
index 00000000..78f58fe6
--- /dev/null
+++ b/tests/test_pcst_fast.py
@@ -0,0 +1,11 @@
+import pcst_fast
+
+
+def test_pcst_fast_sanity_on_numpy2() -> None:
+ """PyPI wheels for pcst-fast 1.0.10 need a source build with pybind11>=2.11."""
+ edges = [[0, 1], [1, 2], [2, 3]]
+ prizes = [1.0, 1.0, 1.0, 1.0]
+ costs = [0.8, 1.8, 2.8]
+ nodes, edge_ids = pcst_fast.pcst_fast(edges, prizes, costs, -1, 1, "strong", 0)
+ assert list(nodes) == [0, 1]
+ assert list(edge_ids) == [0]
diff --git a/tests/test_phase_f_formats.py b/tests/test_phase_f_formats.py
new file mode 100644
index 00000000..45d95f8f
--- /dev/null
+++ b/tests/test_phase_f_formats.py
@@ -0,0 +1,90 @@
+"""Phase F format locks: stable subset identity and ANN byte storage."""
+
+import numpy as np
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.assay.base import Assay
+from scarf.storage.ann_index import save_ann_index
+from scarf.storage.arrays import create_zarr_dataset
+
+
+def test_subset_hash_is_stable_string_digest() -> None:
+ cells = np.array([0, 2, 5], dtype=np.int64)
+ feats = np.array([1, 3, 4, 7], dtype=np.int64)
+ first = Assay._create_subset_hash(cells, feats)
+ second = Assay._create_subset_hash(cells.copy(), feats.copy())
+ assert isinstance(first, str)
+ assert first == second
+ assert first != Assay._create_subset_hash(cells[::-1], feats)
+
+
+def test_subset_hash_matches_pinned_golden_value() -> None:
+ # Pin the exact digest so the persisted subset identity cannot change
+ # silently. Changing this value breaks reuse of stored normalized data.
+ assert (
+ Assay._create_subset_hash(
+ np.array([0, 2, 5], dtype=np.int64),
+ np.array([1, 3, 4, 7], dtype=np.int64),
+ )
+ == "2e67c2c409b18b438d0451d9ab0f3e43"
+ )
+
+
+def test_subset_hash_encodes_cell_feature_boundary() -> None:
+ # Same concatenated indices but a different cell/feature split must not
+ # collide, otherwise a stale normalized cache could be silently reused.
+ left = Assay._create_subset_hash(
+ np.array([0, 1], dtype=np.int64), np.array([2, 3], dtype=np.int64)
+ )
+ right = Assay._create_subset_hash(
+ np.array([0, 1, 2], dtype=np.int64), np.array([3], dtype=np.int64)
+ )
+ assert left != right
+
+
+def test_save_ann_index_writes_exact_zarr_bytes() -> None:
+ class _FakeIndex:
+ def get_current_count(self) -> int:
+ return 3
+
+ def save_index(self, path: str) -> None:
+ with open(path, "wb") as handle:
+ handle.write(b"hnsw-bytes")
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ group = root.create_group("ann")
+ save_ann_index(
+ group,
+ _FakeIndex(),
+ profile="fast_local",
+ metric="l2",
+ dimensions=4,
+ element_count=3,
+ )
+
+ assert "ann_idx_bytes" in group
+ np.testing.assert_array_equal(
+ group["ann_idx_bytes"][:],
+ np.frombuffer(b"hnsw-bytes", dtype=np.uint8),
+ )
+ assert group["ann_idx_bytes"].attrs["byte_length"] == len(b"hnsw-bytes")
+ assert group["ann_idx_bytes"].attrs["metric"] == "l2"
+ assert group["ann_idx_bytes"].attrs["dimensions"] == 4
+ assert group["ann_idx_bytes"].attrs["element_count"] == 3
+ assert len(group["ann_idx_bytes"].attrs["payload_sha256"]) == 64
+
+
+def test_empty_array_uses_nonzero_chunk_dimensions() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+
+ output = create_zarr_dataset(
+ root,
+ "empty_edges",
+ (1, 2),
+ np.int64,
+ (0, 2),
+ )
+
+ assert output.shape == (0, 2)
+ assert output.chunks == (1, 2)
diff --git a/tests/test_pipeline.py b/tests/test_pipeline.py
new file mode 100644
index 00000000..99f256d0
--- /dev/null
+++ b/tests/test_pipeline.py
@@ -0,0 +1,721 @@
+import inspect
+from pathlib import Path
+from threading import Event, get_ident
+
+import pytest
+from scipy.sparse import tril
+
+import scarf.datastore.pipeline_accessor as pipeline_accessor_module
+from scarf.datastore.pipeline_accessor import PipelineEvent
+from scarf.storage.artifacts import ArtifactRef
+from scarf.utils import logger
+
+
+def test_basic_rna_pipeline_returns_only_named_artifact_refs(
+ datastore_ephemeral,
+) -> None:
+ artifacts = datastore_ephemeral.pipeline.run(
+ pipeline_id="basic_rna_analysis",
+ filtering={},
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "top_n": 100,
+ "hvg_key_name": "pipeline_hvgs",
+ },
+ pca={"dims": 5, "n_centroids": 10},
+ ann_index={"ann_m": 16},
+ neighbors={"k": 3},
+ connectivity={},
+ umap={"n_epochs": 10, "label": "pipeline_umap"},
+ leiden={
+ 0.5: {},
+ 1.0: {},
+ },
+ paris={"n_clusters": 3, "label": "pipeline_paris"},
+ doublet_scoring=False,
+ markers={
+ "clusters": {"leiden": 1.0},
+ "gene_batch_size": 100,
+ },
+ )
+
+ assert list(artifacts) == [
+ "cell_selection",
+ "highly_variable_features",
+ "normalized",
+ "pca",
+ "ann_index",
+ "neighbors",
+ "connectivity_map",
+ "embedding_initialization",
+ "umap",
+ "leiden_0.5",
+ "leiden_1.0",
+ "paris",
+ "selected_clusters",
+ "markers",
+ ]
+ assert all(isinstance(ref, ArtifactRef) for ref in artifacts.values())
+ assert artifacts["leiden_0.5"] != artifacts["leiden_1.0"]
+ assert all(
+ datastore_ephemeral.inspect_artifact(ref).complete for ref in artifacts.values()
+ )
+ cluster_column = "RNA_leiden_1.0"
+ edited = datastore_ephemeral.cells.fetch(
+ cluster_column,
+ key="I",
+ ).copy()
+ edited[0] = int(edited.max()) + 1
+ datastore_ephemeral.cells.insert(
+ cluster_column,
+ edited,
+ key="I",
+ overwrite=True,
+ )
+ with pytest.raises(ValueError, match="cluster labels are stale"):
+ datastore_ephemeral.get_markers(
+ from_assay="RNA",
+ cell_key="I",
+ group_key=cluster_column,
+ )
+
+
+def test_pipeline_emits_ordered_events_and_omits_skipped_stages(
+ datastore_ephemeral,
+) -> None:
+ events: list[PipelineEvent] = []
+ messages: list[str] = []
+ sink = logger.add(
+ lambda message: messages.append(message.record["message"]),
+ level="INFO",
+ )
+
+ try:
+ artifacts = datastore_ephemeral.pipeline.run(
+ filtering=False,
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "top_n": 50,
+ "hvg_key_name": "pipeline_callback_hvgs",
+ },
+ pca={"dims": 3, "n_centroids": 5},
+ neighbors={"k": 3},
+ umap=False,
+ leiden={},
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ callback=events.append,
+ )
+ finally:
+ logger.remove(sink)
+
+ expected_stages = [
+ "highly_variable_features",
+ "normalization",
+ "pca",
+ "ann_index",
+ "neighbors",
+ "connectivity",
+ "embedding_initialization",
+ ]
+ assert [(event.kind, event.stage) for event in events] == [
+ (kind, stage)
+ for stage in expected_stages
+ for kind in ("stage_started", "stage_completed")
+ ]
+ assert all(event.error is None for event in events)
+ lifecycle_messages = [
+ message
+ for message in messages
+ if message.startswith("Running pipeline stage:")
+ or message.startswith("Completed pipeline stage:")
+ or message.startswith("Pipeline completed")
+ ]
+ assert lifecycle_messages == [
+ *[
+ message
+ for stage in expected_stages
+ for message in (
+ f"Running pipeline stage: {stage.replace('_', ' ')}",
+ f"Completed pipeline stage: {stage.replace('_', ' ')}",
+ )
+ ],
+ f"Pipeline completed with {len(artifacts)} artifacts",
+ ]
+
+
+def test_pipeline_emits_failed_stage_and_reraises_original_error(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = datastore_ephemeral
+ events: list[PipelineEvent] = []
+ expected_error = RuntimeError("PCA callback test failure")
+
+ def fail_pca(self, *_args, **_kwargs):
+ raise expected_error
+
+ monkeypatch.setattr(type(datastore), "run_pca", fail_pca)
+
+ with pytest.raises(RuntimeError, match="PCA callback test failure") as raised:
+ datastore.pipeline.run(
+ filtering=False,
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "top_n": 50,
+ "hvg_key_name": "pipeline_callback_failure_hvgs",
+ },
+ pca={"dims": 3, "n_centroids": 5},
+ umap=False,
+ leiden={},
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ callback=events.append,
+ )
+
+ assert raised.value is expected_error
+ assert [(event.kind, event.stage) for event in events[-2:]] == [
+ ("stage_started", "pca"),
+ ("stage_failed", "pca"),
+ ]
+ assert events[-1].error is expected_error
+
+
+def test_pipeline_logs_callback_errors_and_continues(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ callback_logs: list[str] = []
+
+ class RecordingLogger:
+ def info(self, _message: str) -> None:
+ pass
+
+ def exception(self, message: str) -> None:
+ callback_logs.append(message)
+
+ def fail_callback(_event: PipelineEvent) -> None:
+ raise RuntimeError("Callback failed")
+
+ monkeypatch.setattr(
+ pipeline_accessor_module,
+ "logger",
+ RecordingLogger(),
+ )
+
+ artifacts = datastore_ephemeral.pipeline.run(
+ filtering=False,
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "top_n": 50,
+ "hvg_key_name": "pipeline_broken_callback_hvgs",
+ },
+ pca={"dims": 3, "n_centroids": 5},
+ neighbors={"k": 3},
+ umap=False,
+ leiden={},
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ callback=fail_callback,
+ )
+
+ assert artifacts["pca"].kind == "reduction"
+ assert callback_logs
+ assert all("Pipeline callback failed" in message for message in callback_logs)
+
+
+def test_pipeline_rejects_unknown_id(datastore_ephemeral) -> None:
+ with pytest.raises(ValueError, match="basic_rna_analysis"):
+ datastore_ephemeral.pipeline.run(pipeline_id="unknown")
+
+
+def test_pipeline_requires_highly_variable_features(datastore_ephemeral) -> None:
+ with pytest.raises(ValueError, match="highly_variable_features"):
+ datastore_ephemeral.pipeline.run(
+ highly_variable_features=False,
+ umap=False,
+ leiden={},
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ )
+
+
+def test_pipeline_selects_clusters_by_pca_silhouette(datastore_ephemeral) -> None:
+ events: list[PipelineEvent] = []
+ artifacts = datastore_ephemeral.pipeline.run(
+ filtering={},
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "top_n": 100,
+ "hvg_key_name": "pipeline_silhouette_hvgs",
+ },
+ pca={"dims": 5, "n_centroids": 10},
+ neighbors={"k": 3},
+ umap=False,
+ leiden={
+ 0.5: {},
+ 1.0: {},
+ },
+ paris={"n_clusters": 3, "label": "pipeline_silhouette_paris"},
+ clustering_concurrency=2,
+ doublet_scoring=False,
+ markers={
+ "gene_batch_size": 100,
+ },
+ callback=events.append,
+ )
+
+ assert "selected_clusters" in artifacts
+ selected = artifacts["selected_clusters"]
+ candidates = {
+ artifacts["leiden_0.5"]: "RNA_leiden_0.5",
+ artifacts["leiden_1.0"]: "RNA_leiden_1.0",
+ artifacts["paris"]: "RNA_pipeline_silhouette_paris",
+ }
+ assert selected in candidates
+ assert artifacts["markers"].kind == "marker_table"
+ for stage in ("cluster_selection", "markers"):
+ assert [event.kind for event in events if event.stage == stage] == [
+ "stage_started",
+ "stage_completed",
+ ]
+
+ store = datastore_ephemeral
+ labels = store.cells.fetch("RNA_clusters")
+ assert (labels == store.cells.fetch(candidates[selected])).all()
+ column = store.zw["cellData"]["RNA_clusters"]
+ assert ArtifactRef.from_dict(column.attrs["source_artifact"]) == selected
+ assert store.get_markers(group_key="RNA_clusters", group_id=labels[0]) is not None
+
+
+def test_pipeline_names_a_single_partition_without_silhouette(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ def fail(*args, **kwargs):
+ raise AssertionError("silhouette selection is unnecessary for one partition")
+
+ monkeypatch.setattr(
+ pipeline_accessor_module.PipelineAccessor,
+ "_select_clusters_by_pca_silhouette",
+ fail,
+ )
+ artifacts = datastore_ephemeral.pipeline.run(
+ filtering={},
+ cell_cycle_scoring=False,
+ highly_variable_features={"top_n": 100, "hvg_key_name": "pipeline_single_hvgs"},
+ pca={"dims": 5, "n_centroids": 10},
+ neighbors={"k": 3},
+ umap=False,
+ leiden={1.0: {}},
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ )
+
+ assert artifacts["selected_clusters"] == artifacts["leiden_1.0"]
+ store = datastore_ephemeral
+ assert (
+ store.cells.fetch("RNA_clusters") == store.cells.fetch("RNA_leiden_1.0")
+ ).all()
+
+
+def test_pipeline_rejects_downstream_steps_without_clustering(
+ datastore_ephemeral,
+) -> None:
+ with pytest.raises(ValueError, match="Marker needs a clustering result"):
+ datastore_ephemeral.pipeline.run(
+ filtering={},
+ cell_cycle_scoring=False,
+ highly_variable_features={"top_n": 100, "hvg_key_name": "pipeline_no_hvgs"},
+ pca={"dims": 5, "n_centroids": 10},
+ neighbors={"k": 3},
+ umap=False,
+ leiden={},
+ paris=False,
+ doublet_scoring=False,
+ markers={"gene_batch_size": 100},
+ )
+
+
+def test_pipeline_rejects_invalid_clustering_concurrency(datastore_ephemeral) -> None:
+ with pytest.raises(ValueError, match="clustering_concurrency"):
+ datastore_ephemeral.pipeline.run(
+ clustering_concurrency=0,
+ umap=False,
+ leiden={},
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ )
+
+
+def test_pipeline_reuses_leiden_without_recomputing(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = datastore_ephemeral
+ options = {
+ "filtering": False,
+ "cell_cycle_scoring": False,
+ "highly_variable_features": {
+ "top_n": 100,
+ "hvg_key_name": "pipeline_cached_leiden_hvgs",
+ },
+ "pca": {"dims": 3, "n_centroids": 5},
+ "neighbors": {"k": 3},
+ "umap": False,
+ "leiden": {0.5: {}},
+ "paris": False,
+ "doublet_scoring": False,
+ "markers": False,
+ }
+ first = datastore.pipeline.run(**options)
+
+ def fail_compute(*_args, **_kwargs):
+ pytest.fail("A reusable Leiden artifact must not be recomputed")
+
+ monkeypatch.setattr(
+ type(datastore),
+ "_compute_prepared_leiden",
+ staticmethod(fail_compute),
+ )
+ events: list[PipelineEvent] = []
+ second = datastore.pipeline.run(**options, callback=events.append)
+
+ assert second["leiden_0.5"] == first["leiden_0.5"]
+ assert [
+ (event.kind, event.stage) for event in events if event.stage == "leiden_0.5"
+ ] == [
+ ("stage_started", "leiden_0.5"),
+ ("stage_completed", "leiden_0.5"),
+ ]
+
+
+def test_pipeline_computes_leiden_with_requested_graph_options(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = datastore_ephemeral
+ original_compute = type(datastore)._compute_prepared_leiden
+ observed = False
+
+ def capture_graph(prepared, graph):
+ nonlocal observed
+ observed = True
+ assert prepared.symmetric_graph is True
+ assert prepared.graph_upper_only is True
+ assert tril(graph, k=-1).nnz == 0
+ return original_compute(prepared, graph)
+
+ monkeypatch.setattr(
+ type(datastore),
+ "_compute_prepared_leiden",
+ staticmethod(capture_graph),
+ )
+ datastore.pipeline.run(
+ filtering=False,
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "top_n": 100,
+ "hvg_key_name": "pipeline_graph_options_hvgs",
+ },
+ pca={"dims": 3, "n_centroids": 5},
+ neighbors={"k": 3},
+ umap=False,
+ leiden={
+ 0.5: {
+ "symmetric_graph": True,
+ "graph_upper_only": True,
+ }
+ },
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ )
+
+ assert observed is True
+
+
+def test_pipeline_reports_failed_and_aborted_clustering_jobs(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = datastore_ephemeral
+ original_compute = type(datastore)._compute_prepared_leiden
+ expected_error = RuntimeError("Leiden callback test failure")
+ events: list[PipelineEvent] = []
+
+ def fail_one_resolution(prepared, graph):
+ if prepared.resolution == 0.5:
+ raise expected_error
+ return original_compute(prepared, graph)
+
+ monkeypatch.setattr(
+ type(datastore),
+ "_compute_prepared_leiden",
+ staticmethod(fail_one_resolution),
+ )
+
+ with pytest.raises(RuntimeError, match="Leiden callback test failure") as raised:
+ datastore.pipeline.run(
+ filtering=False,
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "top_n": 100,
+ "hvg_key_name": "pipeline_failed_clustering_hvgs",
+ },
+ pca={"dims": 3, "n_centroids": 5},
+ neighbors={"k": 3},
+ umap=False,
+ leiden={0.5: {}, 1.0: {}},
+ paris=False,
+ clustering_concurrency=2,
+ doublet_scoring=False,
+ markers=False,
+ callback=events.append,
+ )
+
+ assert raised.value is expected_error
+ failed_events = {
+ event.stage: event
+ for event in events
+ if event.kind == "stage_failed" and event.stage in {"leiden_0.5", "leiden_1.0"}
+ }
+ assert failed_events["leiden_0.5"].error is expected_error
+ assert isinstance(failed_events["leiden_1.0"].error, RuntimeError)
+ assert "was not written" in str(failed_events["leiden_1.0"].error)
+ for stage in ("leiden_0.5", "leiden_1.0"):
+ assert [event.kind for event in events if event.stage == stage] == [
+ "stage_started",
+ "stage_failed",
+ ]
+
+
+def test_leiden_public_api_does_not_accept_precomputed_membership(
+ datastore_ephemeral,
+) -> None:
+ parameters = inspect.signature(datastore_ephemeral.run_leiden_clustering).parameters
+ assert "precomputed_membership" not in parameters
+
+
+def test_pipeline_overlaps_paris_compute_but_serializes_leiden_publish(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = datastore_ephemeral
+ compute_started = Event()
+ paris_started = Event()
+ paris_finished = Event()
+ original_compute = type(datastore)._compute_prepared_leiden
+ original_paris = type(datastore).run_paris_clustering
+ original_publish = type(datastore)._publish_prepared_leiden
+ callback_events: list[PipelineEvent] = []
+ callback_threads: list[int] = []
+ messages: list[str] = []
+ caller_thread = get_ident()
+
+ def record_event(event: PipelineEvent) -> None:
+ callback_events.append(event)
+ callback_threads.append(get_ident())
+
+ def observed_compute(prepared, graph):
+ compute_started.set()
+ assert paris_started.wait(timeout=5)
+ return original_compute(prepared, graph)
+
+ def observed_paris(self, *args, **kwargs):
+ paris_started.set()
+ assert compute_started.wait(timeout=5)
+ result = original_paris(self, *args, **kwargs)
+ paris_finished.set()
+ return result
+
+ def observed_publish(self, prepared, membership):
+ assert paris_finished.is_set()
+ return original_publish(self, prepared, membership)
+
+ monkeypatch.setattr(
+ type(datastore),
+ "_compute_prepared_leiden",
+ staticmethod(observed_compute),
+ )
+ monkeypatch.setattr(
+ type(datastore),
+ "run_paris_clustering",
+ observed_paris,
+ )
+ monkeypatch.setattr(
+ type(datastore),
+ "_publish_prepared_leiden",
+ observed_publish,
+ )
+
+ sink = logger.add(
+ lambda message: messages.append(message.record["message"]),
+ level="INFO",
+ )
+ try:
+ datastore.pipeline.run(
+ filtering=False,
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "top_n": 100,
+ "hvg_key_name": "pipeline_concurrency_hvgs",
+ },
+ pca={"dims": 3, "n_centroids": 5},
+ neighbors={"k": 3},
+ umap=False,
+ leiden={0.5: {}},
+ paris={"n_clusters": 3, "label": "pipeline_concurrency_paris"},
+ clustering_concurrency=2,
+ doublet_scoring=False,
+ markers=False,
+ callback=record_event,
+ )
+ finally:
+ logger.remove(sink)
+
+ assert paris_finished.is_set()
+ assert set(callback_threads) == {caller_thread}
+ for stage in ("leiden_0.5", "paris"):
+ assert [event.kind for event in callback_events if event.stage == stage] == [
+ "stage_started",
+ "stage_completed",
+ ]
+ label = stage.replace("_", " ")
+ expected_messages = [
+ f"Running pipeline stage: {label}",
+ f"Completed pipeline stage: {label}",
+ ]
+ assert [message for message in messages if message in expected_messages] == (
+ expected_messages
+ )
+
+
+def test_pipeline_rejects_unsaved_markers_before_writes(
+ datastore_ephemeral,
+) -> None:
+ before = set(datastore_ephemeral.cells.columns)
+
+ with pytest.raises(ValueError, match="skip_save"):
+ datastore_ephemeral.pipeline.run(
+ markers={"skip_save": True},
+ )
+
+ assert set(datastore_ephemeral.cells.columns) == before
+
+
+def test_basic_rna_pipeline_supports_optional_harmony(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ datastore.cells.insert(
+ "pipeline_batch",
+ ["a" if i % 2 else "b" for i in range(datastore.cells.N)],
+ overwrite=True,
+ )
+
+ artifacts = datastore.pipeline.run(
+ filtering={},
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "top_n": 50,
+ "hvg_key_name": "pipeline_harmony_hvgs",
+ },
+ pca={"dims": 3, "n_centroids": 5},
+ harmony={
+ "batch_columns": ["pipeline_batch"],
+ "harmony_params": {"nclust": 3},
+ },
+ neighbors={"k": 3},
+ umap=False,
+ leiden={},
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ )
+
+ state = datastore.get_assay_state("RNA")
+ assert "harmony" in artifacts
+ assert state is not None
+ assert state.batch_correction == artifacts["harmony"]
+
+
+def test_basic_rna_pipeline_runs_score_steps_after_clustering(
+ datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = datastore_ephemeral
+ events: list[PipelineEvent] = []
+ temporary_paths: list[Path] = []
+ original_run_mapping = type(datastore).run_mapping
+
+ def observe_mapping(query, reference, mapping_name, **kwargs):
+ assert query is not datastore
+ temporary_paths.append(Path(query.zarr_loc))
+ return original_run_mapping(
+ query,
+ reference,
+ mapping_name,
+ **kwargs,
+ )
+
+ monkeypatch.setattr(type(datastore), "run_mapping", observe_mapping)
+ artifacts = datastore.pipeline.run(
+ filtering={},
+ cell_cycle_scoring={},
+ highly_variable_features={
+ "top_n": 50,
+ "hvg_key_name": "pipeline_full_hvgs",
+ },
+ pca={"dims": 3, "n_centroids": 5},
+ neighbors={"k": 3},
+ umap={"n_epochs": 5, "label": "pipeline_full_umap"},
+ leiden={0.5: {}},
+ paris={"n_clusters": 3, "label": "pipeline_full_paris"},
+ doublet_scoring={
+ "clusters": "paris",
+ "cluster_sample_fraction": 0.01,
+ "max_cells_per_cluster": 2,
+ "simulation_ratio": 0.01,
+ "save_k": 3,
+ "smoothing_t": 1,
+ "random_seed": 9,
+ },
+ markers={
+ "clusters": {"leiden": 0.5},
+ "gene_batch_size": 100,
+ },
+ callback=events.append,
+ )
+
+ assert artifacts["cell_cycle"].kind == "cell_cycle"
+ assert artifacts["doublets"].kind == "doublet_score"
+ assert artifacts["markers"].kind == "marker_table"
+ assert list(artifacts).index("doublets") > list(artifacts).index("paris")
+ assert temporary_paths and all(not path.exists() for path in temporary_paths)
+ assert not datastore.list_artifacts(
+ kind="projection",
+ from_assay="RNA",
+ )
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.connectivity_map == artifacts["connectivity_map"]
+ reference = datastore.get_mapping_reference()
+ assert reference.neighbors == artifacts["neighbors"]
+ assert reference.symphony_state is None
+ for stage in (
+ "cell_cycle_scoring",
+ "umap",
+ "doublet_scoring",
+ "markers",
+ ):
+ assert [event.kind for event in events if event.stage == stage] == [
+ "stage_started",
+ "stage_completed",
+ ]
diff --git a/tests/test_plotting_cluster_connectivity.py b/tests/test_plotting_cluster_connectivity.py
new file mode 100644
index 00000000..7dac786e
--- /dev/null
+++ b/tests/test_plotting_cluster_connectivity.py
@@ -0,0 +1,330 @@
+import matplotlib
+
+matplotlib.use("Agg")
+
+import matplotlib.pyplot as plt
+import numpy as np
+import pytest
+from matplotlib.collections import LineCollection
+from scipy import sparse
+
+from scarf.plotting._contracts import CategoricalScale, SizeScale
+from scarf.plotting._figure import PlotResult
+from scarf.plotting.cluster_connectivity import cluster_connectivity
+
+
+class _FakeCells:
+ def __init__(self, values):
+ self.values = values
+ self.fetch_calls = []
+
+ def fetch(self, column, *, key):
+ self.fetch_calls.append((column, key))
+ return self.values[column]
+
+
+class _FakeStore:
+ _defaultAssay = "RNA"
+
+ def __init__(self, values, graph):
+ self.cells = _FakeCells(values)
+ self.graph = graph
+ self.graph_calls = []
+
+ def load_graph(self, **kwargs):
+ self.graph_calls.append(kwargs)
+ return self.graph
+
+
+def _symmetric_graph(n_cells, weighted_edges):
+ rows = []
+ columns = []
+ data = []
+ for source, target, weight in weighted_edges:
+ rows.extend((source, target))
+ columns.extend((target, source))
+ data.extend((weight, weight))
+ return sparse.csr_matrix((data, (rows, columns)), shape=(n_cells, n_cells))
+
+
+def _store():
+ values = {
+ "layout1": np.array([0.0, 0.0, 9.0, 2.0, 2.0, 8.0, 4.0, 4.0, 10.0]),
+ "layout2": np.array([0.0, 0.0, 3.0, 1.0, 1.0, 4.0, 0.0, 0.0, -3.0]),
+ "cluster": np.array(["A", "A", "A", "B", "B", "B", "C", "C", "C"]),
+ }
+ graph = _symmetric_graph(
+ 9,
+ [
+ (0, 1, 1.0),
+ (3, 4, 1.0),
+ (6, 7, 1.0),
+ (0, 3, 0.6),
+ (1, 4, 0.4),
+ (0, 6, 0.2),
+ (3, 6, 0.8),
+ ],
+ )
+ return _FakeStore(values, graph)
+
+
+def _plot(store, **kwargs):
+ return cluster_connectivity(
+ store,
+ group_by="cluster",
+ layout_key="layout",
+ show=False,
+ **kwargs,
+ )
+
+
+def test_cluster_connectivity_aggregates_reciprocals_once():
+ store = _store()
+ result = _plot(
+ store,
+ cell_key="selected",
+ from_assay="RNA",
+ feat_key="variable",
+ graph_loc="graphs/connectivity",
+ minimum_edge_weight=0.0,
+ )
+
+ assert isinstance(result, PlotResult)
+ assert result.owns_figure is True
+ assert list(result.axes) == ["cluster_connectivity"]
+ assert store.cells.fetch_calls == [
+ ("layout1", "selected"),
+ ("layout2", "selected"),
+ ("cluster", "selected"),
+ ]
+ assert store.graph_calls == [
+ {
+ "from_assay": "RNA",
+ "cell_key": "selected",
+ "feat_key": "variable",
+ "graph_loc": "graphs/connectivity",
+ "symmetric": True,
+ }
+ ]
+
+ nodes = result.tables["nodes"]
+ assert list(nodes.columns) == [
+ "category",
+ "x",
+ "y",
+ "nCells",
+ "proportion",
+ "size",
+ "displayLabel",
+ ]
+ assert nodes["category"].tolist() == ["A", "B", "C"]
+ np.testing.assert_allclose(nodes["x"], [0.0, 2.0, 4.0])
+ np.testing.assert_allclose(nodes["y"], [0.0, 1.0, 0.0])
+ assert nodes["nCells"].tolist() == [3, 3, 3]
+ np.testing.assert_allclose(nodes["proportion"], [1 / 3] * 3)
+
+ edges = result.tables["edges"]
+ assert list(edges.columns) == [
+ "source",
+ "target",
+ "rawWeight",
+ "normalizedWeight",
+ ]
+ assert list(zip(edges["source"], edges["target"], strict=True)) == [
+ ("A", "B"),
+ ("A", "C"),
+ ("B", "C"),
+ ]
+ np.testing.assert_allclose(edges["rawWeight"], [1.0, 0.2, 0.8])
+ np.testing.assert_allclose(
+ edges["normalizedWeight"],
+ [
+ 1.0 / np.sqrt(3.2 * 3.8),
+ 0.2 / np.sqrt(3.2 * 3.0),
+ 0.8 / np.sqrt(3.8 * 3.0),
+ ],
+ )
+ pairs = [frozenset(pair) for pair in zip(edges["source"], edges["target"])]
+ assert len(pairs) == len(set(pairs))
+ assert not (edges["source"] == edges["target"]).any()
+
+ assert isinstance(result.scales[0], CategoricalScale)
+ assert isinstance(result.scales[1], SizeScale)
+ assert result.provenance.n_cells == 9
+ assert result.provenance.extras["n_aggregated_edges"] == 3
+ assert result.provenance.extras["n_edges"] == 3
+ assert "sqrt" in result.provenance.extras["normalization"]
+ ax = result.axes["cluster_connectivity"]
+ assert isinstance(ax.collections[0], LineCollection)
+ assert ax.collections[0].get_zorder() < ax.collections[1].get_zorder()
+ assert len(ax.get_xticks()) == 0
+ assert len(ax.get_yticks()) == 0
+ assert ax.get_box_aspect() == pytest.approx(1.0)
+ result.close()
+
+
+def test_cluster_connectivity_threshold_and_degree_cap_are_deterministic():
+ thresholded = _plot(_store(), minimum_edge_weight=0.25)
+ assert thresholded.tables["edges"][["source", "target"]].values.tolist() == [
+ ["A", "B"]
+ ]
+ thresholded.close()
+
+ capped = _plot(
+ _store(),
+ minimum_edge_weight=0.0,
+ max_edges_per_node=1,
+ )
+ assert capped.tables["edges"][["source", "target"]].values.tolist() == [["A", "B"]]
+ capped.close()
+
+
+def test_cluster_connectivity_explicit_positions_scales_and_labels():
+ categorical_scale = CategoricalScale(
+ order=("C", "A", "B"),
+ palette={"A": "#aa0000", "B": "#00aa00", "C": "#0000aa"},
+ labels={"A": "Alpha", "C": "Gamma"},
+ )
+ size_scale = SizeScale(
+ vmin=0.0,
+ vmax=0.5,
+ size_min=20.0,
+ size_max=80.0,
+ )
+ positions = {
+ "A": (10.0, 11.0),
+ "B": (20.0, 21.0),
+ "C": (30.0, 31.0),
+ }
+
+ result = _plot(
+ _store(),
+ positions=positions,
+ categorical_scale=categorical_scale,
+ size_scale=size_scale,
+ minimum_edge_weight=0.0,
+ )
+
+ nodes = result.tables["nodes"]
+ assert nodes["category"].tolist() == ["C", "A", "B"]
+ np.testing.assert_allclose(nodes["x"], [30.0, 10.0, 20.0])
+ np.testing.assert_allclose(nodes["y"], [31.0, 11.0, 21.0])
+ assert nodes["displayLabel"].tolist() == ["Gamma", "Alpha", "B"]
+ np.testing.assert_allclose(nodes["size"], [60.0, 60.0, 60.0])
+ assert result.scales[0].order == ("C", "A", "B")
+ assert result.scales[0].labels == {
+ "C": "Gamma",
+ "A": "Alpha",
+ "B": "B",
+ }
+ assert result.scales[1] is size_scale
+ assert [text.get_text() for text in result.axes["cluster_connectivity"].texts] == [
+ "Gamma",
+ "Alpha",
+ "B",
+ ]
+ font_sizes = [
+ text.get_fontsize() for text in result.axes["cluster_connectivity"].texts
+ ]
+ assert font_sizes[0] < font_sizes[2]
+ result.close()
+
+
+def test_cluster_connectivity_mean_positions():
+ result = _plot(_store(), position="mean")
+ nodes = result.tables["nodes"]
+ np.testing.assert_allclose(nodes["x"], [3.0, 4.0, 6.0])
+ np.testing.assert_allclose(nodes["y"], [1.0, 2.0, -1.0])
+ result.close()
+
+
+def test_cluster_connectivity_uses_caller_owned_target_and_optional_cells():
+ figure, ax = plt.subplots()
+ result = _plot(
+ _store(),
+ target=ax,
+ show_cells=True,
+ labels=False,
+ )
+
+ assert result.figure is figure
+ assert result.axes == {"cluster_connectivity": ax}
+ assert result.owns_figure is False
+ assert [artist.get_zorder() for artist in ax.collections] == [0, 1, 2]
+ result.close()
+ assert plt.fignum_exists(figure.number)
+ plt.close(figure)
+
+
+@pytest.mark.parametrize(
+ ("column", "replacement", "message"),
+ [
+ ("layout1", np.array([0.0] * 8), "matching lengths"),
+ (
+ "layout2",
+ np.array([0.0, 0.0, np.nan, 1.0, 1.0, 4.0, 0.0, 0.0, -3.0]),
+ "non-finite coordinates",
+ ),
+ (
+ "cluster",
+ np.array(["A", "A", None, "B", "B", "B", "C", "C", "C"]),
+ "missing values",
+ ),
+ ],
+)
+def test_cluster_connectivity_validates_cell_arrays(column, replacement, message):
+ store = _store()
+ store.cells.values[column] = replacement
+ with pytest.raises(ValueError, match=message):
+ _plot(store)
+
+
+def test_cluster_connectivity_validates_graph_shape_and_weights():
+ wrong_shape = _store()
+ wrong_shape.graph = sparse.csr_matrix((8, 8))
+ with pytest.raises(ValueError, match="Graph shape"):
+ _plot(wrong_shape)
+
+ dense = _store()
+ dense.graph = dense.graph.toarray()
+ with pytest.raises(TypeError, match="sparse matrix"):
+ _plot(dense)
+
+ negative = _store()
+ negative.graph = negative.graph.copy()
+ negative.graph.data[0] = -1.0
+ with pytest.raises(ValueError, match="non-negative"):
+ _plot(negative)
+
+
+def test_cluster_connectivity_requires_exact_explicit_position_coverage():
+ with pytest.raises(ValueError, match="missing: C"):
+ _plot(
+ _store(),
+ positions={"A": (0.0, 0.0), "B": (1.0, 1.0)},
+ )
+ with pytest.raises(ValueError, match="unexpected: D"):
+ _plot(
+ _store(),
+ positions={
+ "A": (0.0, 0.0),
+ "B": (1.0, 1.0),
+ "C": (2.0, 2.0),
+ "D": (3.0, 3.0),
+ },
+ )
+
+
+@pytest.mark.parametrize(
+ ("kwargs", "message"),
+ [
+ ({"position": "mode"}, "position"),
+ ({"minimum_edge_weight": -0.1}, "minimum_edge_weight"),
+ ({"max_edges_per_node": -1}, "max_edges_per_node"),
+ ({"edge_alpha": 1.1}, "edge_alpha"),
+ ({"edge_width_range": (2.0, 1.0)}, "edge_width_range"),
+ ],
+)
+def test_cluster_connectivity_validates_plot_options(kwargs, message):
+ with pytest.raises(ValueError, match=message):
+ _plot(_store(), **kwargs)
diff --git a/tests/test_plotting_dependencies.py b/tests/test_plotting_dependencies.py
new file mode 100644
index 00000000..798a1495
--- /dev/null
+++ b/tests/test_plotting_dependencies.py
@@ -0,0 +1,37 @@
+"""Optional plotting dependency error tests."""
+
+import builtins
+
+import pytest
+
+from scarf.plotting._deps import (
+ require_kneed,
+ require_matplotlib,
+ require_seaborn,
+)
+
+
+@pytest.mark.parametrize(
+ ("loader", "blocked_package", "message"),
+ [
+ (require_matplotlib, "matplotlib", "matplotlib"),
+ (require_seaborn, "seaborn", "seaborn"),
+ (require_kneed, "kneed", "kneed"),
+ ],
+)
+def test_optional_dependency_errors_are_actionable(
+ monkeypatch,
+ loader,
+ blocked_package,
+ message,
+):
+ original_import = builtins.__import__
+
+ def blocked_import(name, *args, **kwargs):
+ if name.split(".", 1)[0] == blocked_package:
+ raise ModuleNotFoundError(name)
+ return original_import(name, *args, **kwargs)
+
+ monkeypatch.setattr(builtins, "__import__", blocked_import)
+ with pytest.raises(ImportError, match=message):
+ loader()
diff --git a/tests/test_plotting_diagnostics.py b/tests/test_plotting_diagnostics.py
new file mode 100644
index 00000000..c9fdefdb
--- /dev/null
+++ b/tests/test_plotting_diagnostics.py
@@ -0,0 +1,145 @@
+import matplotlib
+
+matplotlib.use("Agg")
+
+import numpy as np
+import pandas as pd
+import pytest
+from scipy import sparse
+
+import scarf.plotting as splt
+import scarf.plotting.diagnostics as diagnostics
+from scarf.plotting._figure import PlotResult
+
+
+class _KneeLocator:
+ def __init__(self, x, y, **kwargs):
+ self.elbow = 1
+
+
+@pytest.fixture
+def deterministic_kneed(monkeypatch):
+ monkeypatch.setattr(diagnostics, "require_kneed", lambda: _KneeLocator)
+
+
+def _qc_data() -> pd.DataFrame:
+ return pd.DataFrame(
+ {
+ "groups": ["a"] * 4 + ["b"] * 4,
+ "nCounts": [10.0, 12.0, 11.0, 13.0, 20.0, 22.0, 21.0, 23.0],
+ "nFeatures": [4.0, 5.0, 5.0, 6.0, 8.0, 9.0, 9.0, 10.0],
+ }
+ )
+
+
+def _graph():
+ return sparse.csr_matrix(
+ np.array(
+ [
+ [0.0, 0.5, 0.0],
+ [0.5, 0.0, 1.0],
+ [0.0, 1.0, 0.0],
+ ]
+ )
+ )
+
+
+def _hvg_arguments() -> tuple[np.ndarray, ...]:
+ return (
+ np.array([1.0, 2.0, 4.0, 8.0]),
+ np.array([2.0, 3.0, 8.0, 12.0]),
+ np.array([10, 20, 30, 40]),
+ np.array([False, True, False, True]),
+ )
+
+
+def test_diagnostics_show_default_and_suppression(
+ monkeypatch,
+ deterministic_kneed,
+):
+ shown = []
+
+ def track_show(result):
+ shown.append(result)
+
+ monkeypatch.setattr(PlotResult, "show", track_show)
+ defaults = [
+ splt.qc(_qc_data(), max_points=0),
+ splt.elbow([0.5, 0.3, 0.2]),
+ splt.graph_qc(_graph()),
+ splt.highly_variable_features(*_hvg_arguments()),
+ ]
+ suppressed = [
+ splt.qc(_qc_data(), max_points=0, show=False),
+ splt.elbow([0.5, 0.3, 0.2], show=False),
+ splt.graph_qc(_graph(), show=False),
+ splt.highly_variable_features(*_hvg_arguments(), show=False),
+ ]
+
+ assert shown == defaults
+ for result in defaults + suppressed:
+ result.close()
+
+
+def test_qc_result_contains_tables_metadata_and_artists():
+ data = _qc_data()
+ result = splt.qc(data, max_points=5, seed=13, show=False)
+
+ assert isinstance(result, PlotResult)
+ assert result.owns_figure is True
+ assert list(result.axes) == ["nCounts", "nFeatures"]
+ pd.testing.assert_frame_equal(result.tables["data"], data)
+ assert set(result.tables["summary"]["metric"]) == {"nCounts", "nFeatures"}
+ assert result.provenance.n_cells == len(data)
+ assert result.provenance.notes == ("qc",)
+ assert result.provenance.extras["displayed_points"] == {
+ "nCounts": 5,
+ "nFeatures": 5,
+ }
+ assert result.provenance.extras["seed"] == 13
+ assert result.legends[0].kind == "categorical"
+ assert all(ax.collections for ax in result.axes.values())
+ result.close()
+
+
+def test_elbow_result_records_detection_and_line_artists(deterministic_kneed):
+ values = np.array([0.6, 0.25, 0.1])
+ result = splt.elbow(values, show=False)
+
+ table = result.tables["variance_explained"]
+ np.testing.assert_array_equal(table["variance_explained"], values)
+ assert table.loc[table["is_elbow"], "component"].tolist() == [1]
+ assert result.provenance.extras["elbow"] == 1
+ assert result.legends[0].extras["component"] == 1
+ assert len(result.axes["elbow"].lines) == 2
+ result.close()
+
+
+def test_graph_qc_result_contains_graph_tables_and_histograms():
+ result = splt.graph_qc(_graph(), show=False)
+
+ assert result.provenance.n_cells == 3
+ assert result.provenance.extras["n_edges"] == 4
+ assert result.tables["node_degrees"]["degree"].tolist() == [1, 2, 1]
+ assert result.tables["degree_frequencies"].to_dict("list") == {
+ "degree": [1, 2],
+ "frequency": [2, 1],
+ }
+ assert len(result.tables["edge_weights"]) == 4
+ assert len(result.axes["node_degree"].patches) == 2
+ assert len(result.axes["edge_weight"].patches) == 30
+ result.close()
+
+
+def test_highly_variable_features_result_contains_feature_table_and_scatters():
+ args = _hvg_arguments()
+ result = splt.highly_variable_features(*args, show=False)
+
+ table = result.tables["features"]
+ assert table["selected"].tolist() == [False, True, False, True]
+ np.testing.assert_allclose(table["log2_mean_nonzero"], [0.0, 1.0, 2.0, 3.0])
+ assert result.provenance.extras["n_features"] == 4
+ assert result.provenance.extras["n_selected"] == 2
+ assert result.legends[0].kind == "categorical"
+ assert len(result.axes["highly_variable_features"].collections) == 2
+ result.close()
diff --git a/tests/test_plotting_export.py b/tests/test_plotting_export.py
new file mode 100644
index 00000000..d5ad8257
--- /dev/null
+++ b/tests/test_plotting_export.py
@@ -0,0 +1,120 @@
+"""Milestone D export and provenance tests."""
+
+import json
+import xml.etree.ElementTree as ET
+
+import matplotlib
+import numpy as np
+import pytest
+from PIL import Image
+
+matplotlib.use("Agg")
+
+import scarf.plotting as splt
+
+
+def test_tiff_export_exact_size_and_provenance_sidecar(
+ umap, leiden_clustering, datastore, tmp_path
+):
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ figsize=(3.0, 2.0),
+ show=False,
+ )
+ path = tmp_path / "figure.tiff"
+ with matplotlib.rc_context({"savefig.bbox": "tight"}):
+ result.save(path, dpi=120, provenance_sidecar=True)
+
+ with Image.open(path) as image:
+ assert image.format == "TIFF"
+ assert image.size == (360, 240)
+ assert image.tag_v2.get(259) == 5 # LZW compression
+
+ sidecar = tmp_path / "figure.tiff.json"
+ payload = json.loads(sidecar.read_text())
+ assert payload["provenance"]["renderer"] == "matplotlib"
+ assert payload["provenance"]["n_cells"] == len(datastore.cells.active_index("I"))
+ assert payload["figure"]["width_inches"] == pytest.approx(3.0)
+ assert payload["export"] == {
+ "dpi": 120.0,
+ "filename": "figure.tiff",
+ "format": "tiff",
+ }
+ assert "RNA_leiden_cluster" in payload["legends"][0]["label"]
+ result.close()
+
+
+def test_svg_export_preserves_physical_size_under_tight_global_setting(
+ umap, datastore, tmp_path
+):
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ figsize=(3.0, 2.0),
+ show=False,
+ )
+ path = tmp_path / "figure.svg"
+ with matplotlib.rc_context({"savefig.bbox": "tight"}):
+ result.save(path, exact_size=True)
+ root = ET.parse(path).getroot()
+ assert root.attrib["width"] == "216pt"
+ assert root.attrib["height"] == "144pt"
+ result.close()
+
+
+def test_save_provenance_handles_numpy_values(
+ umap, leiden_clustering, datastore, tmp_path
+):
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ show=False,
+ )
+ result.provenance.extras["array"] = np.array([1, 2], dtype=np.int64)
+ result.provenance.extras["missing"] = np.float64(np.nan)
+ path = result.save_provenance(tmp_path / "plot.json")
+ payload = json.loads(path.read_text())
+ assert payload["provenance"]["extras"]["array"] == [1, 2]
+ assert payload["provenance"]["extras"]["missing"] is None
+ result.close()
+
+
+def test_export_validates_extension_and_dpi(umap, datastore, tmp_path):
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ show=False,
+ )
+ with pytest.raises(ValueError, match="file extension"):
+ result.save(tmp_path / "no_extension")
+ with pytest.raises(ValueError, match="dpi must be positive"):
+ result.save(tmp_path / "plot.png", dpi=0)
+ with pytest.raises(ValueError, match="Unsupported export format"):
+ result.save(tmp_path / "plot.unsupported")
+ cropped = result.save(tmp_path / "plot.png", exact_size=False)
+ assert cropped.exists()
+ result.close()
+
+
+def test_embedding_rasterization_threshold(umap, datastore):
+ vector = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ rasterize_threshold=10**9,
+ show=False,
+ )
+ rasterized = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ rasterize_threshold=0,
+ show=False,
+ )
+ vector_collection = next(iter(vector.axes.values())).collections[0]
+ raster_collection = next(iter(rasterized.axes.values())).collections[0]
+ assert vector_collection.get_rasterized() is False
+ assert raster_collection.get_rasterized() is True
+ vector.close()
+ rasterized.close()
diff --git a/tests/test_plotting_figure_helpers.py b/tests/test_plotting_figure_helpers.py
new file mode 100644
index 00000000..004a7208
--- /dev/null
+++ b/tests/test_plotting_figure_helpers.py
@@ -0,0 +1,350 @@
+"""Figure ownership and composition helper tests."""
+
+import warnings
+
+import matplotlib
+import pytest
+
+matplotlib.use("Agg")
+
+import matplotlib.pyplot as plt
+from matplotlib.legend import Legend
+from matplotlib.lines import Line2D
+
+import scarf.plotting as splt
+from scarf.plotting._figure import (
+ PlotResult,
+ _remove_child_legend_artists,
+ compose_results,
+ normalize_axes_target,
+)
+
+
+def _categorical_child(figure, axis, name: str, values: tuple[str, ...]) -> PlotResult:
+ palette = {
+ value: plt.get_cmap("tab20")(index % 20) for index, value in enumerate(values)
+ }
+ return PlotResult(
+ figure=figure,
+ axes={name: axis},
+ tables={},
+ legends=(splt.LegendSpec(kind="categorical", label=name),),
+ scales=(splt.CategoricalScale(order=values, palette=palette),),
+ provenance=splt.PlotProvenance(scarf_version="test"),
+ owns_figure=False,
+ )
+
+
+def _plot_result(*, owns_figure: bool = True) -> PlotResult:
+ fig, ax = plt.subplots()
+ ax.plot([0, 1])
+ return PlotResult(
+ figure=fig,
+ axes={"main": ax},
+ tables={},
+ legends=(),
+ scales=(),
+ provenance=splt.PlotProvenance(scarf_version="test"),
+ owns_figure=owns_figure,
+ )
+
+
+def test_plot_result_show_uses_ipython_display_under_agg(monkeypatch):
+ result = _plot_result()
+ displayed = []
+ show_calls = []
+ assert result.figure.canvas.required_interactive_framework is None
+ monkeypatch.setattr("IPython.get_ipython", lambda: object())
+ monkeypatch.setattr("IPython.display.display", displayed.append)
+ monkeypatch.setattr(plt, "show", lambda: show_calls.append(True))
+
+ result.show()
+
+ assert displayed == [result.figure]
+ assert show_calls == []
+ assert not plt.fignum_exists(result.figure.number)
+ assert "rendered=True" in repr(result)
+
+
+def test_plot_result_show_delegates_for_interactive_gui_canvas(monkeypatch):
+ result = _plot_result()
+ displayed = []
+ show_calls = []
+ monkeypatch.setattr("IPython.get_ipython", lambda: None)
+ monkeypatch.setattr("IPython.display.display", displayed.append)
+ monkeypatch.setattr(plt, "show", lambda: show_calls.append(True))
+ monkeypatch.setattr(
+ result.figure.canvas,
+ "required_interactive_framework",
+ "qt",
+ raising=False,
+ )
+
+ result.show()
+
+ assert displayed == []
+ assert show_calls == [True]
+ assert not plt.fignum_exists(result.figure.number)
+ assert "rendered=True" in repr(result)
+
+
+def test_plot_result_show_is_warning_free_noop_for_headless_agg(monkeypatch):
+ result = _plot_result(owns_figure=False)
+ monkeypatch.setattr("IPython.get_ipython", lambda: None)
+ monkeypatch.setattr(
+ plt,
+ "show",
+ lambda: pytest.fail("headless Agg must not call plt.show()"),
+ )
+
+ with warnings.catch_warnings():
+ warnings.simplefilter("error")
+ result.show()
+
+ assert plt.fignum_exists(result.figure.number)
+ assert "rendered=False" in repr(result)
+ plt.close(result.figure)
+
+
+def test_plot_result_show_closes_owned_figure_and_retains_metadata(monkeypatch):
+ result = _plot_result()
+ figure = result.figure
+ axes = result.axes
+ tables = result.tables
+ legends = result.legends
+ scales = result.scales
+ provenance = result.provenance
+ monkeypatch.setattr("IPython.get_ipython", lambda: object())
+ monkeypatch.setattr("IPython.display.display", lambda _: None)
+
+ result.show()
+
+ assert not plt.fignum_exists(figure.number)
+ assert result.figure is figure
+ assert result.axes is axes
+ assert result.tables is tables
+ assert result.legends is legends
+ assert result.scales is scales
+ assert result.provenance is provenance
+
+ caller_owned = _plot_result(owns_figure=False)
+ caller_owned.close()
+ assert plt.fignum_exists(caller_owned.figure.number)
+ plt.close(caller_owned.figure)
+
+
+def test_plot_result_can_save_after_show(monkeypatch, tmp_path):
+ result = _plot_result()
+ monkeypatch.setattr("IPython.get_ipython", lambda: object())
+ monkeypatch.setattr("IPython.display.display", lambda _: None)
+
+ result.show()
+ output = result.save(tmp_path / "after-show.png", dpi=80)
+
+ assert output.exists()
+ assert output.stat().st_size > 0
+
+
+def test_dark_plot_exports_with_readable_opaque_background(tmp_path):
+ result = _plot_result()
+ result.theme = "dark"
+
+ output = result.save(tmp_path / "dark.png", dpi=60)
+ pixels = plt.imread(output)
+
+ assert float(pixels[0, 0, :3].mean()) < 0.2
+ assert result.figure.patch.get_facecolor()[:3] == pytest.approx((1, 1, 1))
+ result.close()
+
+
+def test_plot_result_repr_is_compact_and_never_renders(monkeypatch):
+ result = _plot_result()
+ displayed = []
+ show_calls = []
+ monkeypatch.setattr("IPython.get_ipython", lambda: object())
+ monkeypatch.setattr("IPython.display.display", displayed.append)
+ monkeypatch.setattr(plt, "show", lambda: show_calls.append(True))
+
+ unrendered = repr(result)
+
+ assert unrendered == (
+ "PlotResult(axes=1, tables=0, legends=0, scales=0, "
+ "owns_figure=True, rendered=False)"
+ )
+ assert displayed == []
+ assert show_calls == []
+ assert plt.fignum_exists(result.figure.number)
+
+ result.show()
+ rendered = repr(result)
+
+ assert rendered == (
+ "PlotResult(axes=1, tables=0, legends=0, scales=0, "
+ "owns_figure=True, rendered=True)"
+ )
+ assert "figure=<" not in rendered
+ assert displayed == [result.figure]
+ assert show_calls == []
+
+
+def test_rendered_plot_result_publishes_no_extra_notebook_output(monkeypatch):
+ from IPython.core.formatters import DisplayFormatter
+
+ result = _plot_result()
+ monkeypatch.setattr("IPython.get_ipython", lambda: object())
+ monkeypatch.setattr("IPython.display.display", lambda _: None)
+ result.show()
+
+ data, metadata = DisplayFormatter().format(result)
+
+ # An empty format bundle makes the IPython display hook publish nothing, so
+ # the already-displayed figure stays the only output of the cell.
+ assert data == {}
+ assert metadata == {}
+
+
+def test_unrendered_plot_result_still_displays_its_summary(monkeypatch):
+ published = []
+ monkeypatch.setattr("IPython.display.display", lambda *a, **k: published.append(a))
+ result = _plot_result()
+
+ result._ipython_display_()
+
+ assert published == [({"text/plain": repr(result)},)]
+ assert "rendered=False" in published[0][0]["text/plain"]
+ plt.close(result.figure)
+
+
+def test_normalize_axes_target_accepts_mappings_and_sequences():
+ fig, axes = plt.subplots(1, 2)
+ _, mapped, owns = normalize_axes_target(
+ {"left": axes[0], "right": axes[1]},
+ panel_keys=["left", "right"],
+ figsize=None,
+ )
+ assert owns is False
+ assert mapped == {"left": axes[0], "right": axes[1]}
+
+ _, sequenced, owns = normalize_axes_target(
+ axes,
+ panel_keys=["left", "right"],
+ figsize=None,
+ )
+ assert owns is False
+ assert list(sequenced.values()) == list(axes)
+ plt.close(fig)
+
+
+def test_normalize_axes_target_rejects_invalid_ownership():
+ fig_a, ax_a = plt.subplots()
+ fig_b, ax_b = plt.subplots()
+ with pytest.raises(KeyError, match="missing panel keys"):
+ normalize_axes_target(
+ {"left": ax_a},
+ panel_keys=["left", "right"],
+ figsize=None,
+ )
+ with pytest.raises(ValueError, match="same figure"):
+ normalize_axes_target(
+ [ax_a, ax_b],
+ panel_keys=["left", "right"],
+ figsize=None,
+ )
+ with pytest.raises(TypeError, match="single Axes"):
+ normalize_axes_target(
+ ax_a,
+ panel_keys=["left", "right"],
+ figsize=None,
+ )
+ plt.close(fig_a)
+ plt.close(fig_b)
+
+
+def test_shared_legends_stay_separated_beyond_three_scales():
+ figure, axes = plt.subplots(2, 2, figsize=(6, 5), layout="constrained")
+ flat = axes.ravel()
+ children = [
+ _categorical_child(figure, flat[index], f"scale{index}", ("a", "b", "c"))
+ for index in range(4)
+ ]
+
+ result = compose_results(figure, children, panel_labels=False)
+
+ assert len(figure.legends) == 3
+ figure.canvas.draw()
+ renderer = figure.canvas.get_renderer()
+ boxes = [legend.get_window_extent(renderer) for legend in figure.legends]
+ assert not any(
+ boxes[first].overlaps(boxes[second])
+ for first in range(len(boxes))
+ for second in range(first + 1, len(boxes))
+ )
+ merged = [text.get_text() for text in figure.legends[-1].get_texts()]
+ assert merged[:2] == ["scale2: a", "scale2: b"]
+ assert "scale3: c" in merged
+ result.close()
+ plt.close(figure)
+
+
+def test_tall_shared_legends_merge_instead_of_overlapping():
+ figure, axes = plt.subplots(2, 2, figsize=(6, 5), layout="constrained")
+ values = tuple(f"value{index}" for index in range(18))
+ children = [
+ _categorical_child(figure, axis, f"scale{index}", values)
+ for index, axis in enumerate(axes.ravel()[:3])
+ ]
+
+ result = compose_results(figure, children, panel_labels=False)
+
+ assert len(figure.legends) == 1
+ assert not any(isinstance(artist, Legend) for artist in figure.artists)
+ labels = [text.get_text() for text in figure.legends[0].get_texts()]
+ assert labels[0] == "scale0: value0"
+ assert labels[-1] == "scale2: value17"
+ result.close()
+ plt.close(figure)
+
+
+def test_child_legend_removal_clears_all_axis_and_figure_legends():
+ figure, ax = plt.subplots()
+ first = ax.legend(
+ handles=[Line2D([], [], label="first")],
+ loc="upper left",
+ )
+ ax.add_artist(first)
+ ax.legend(
+ handles=[Line2D([], [], label="second")],
+ loc="lower left",
+ )
+ figure.legend(handles=[Line2D([], [], label="figure")])
+
+ _remove_child_legend_artists(figure, [ax])
+
+ assert not any(isinstance(artist, Legend) for artist in ax.get_children())
+ assert figure.legends == []
+ plt.close(figure)
+
+
+def test_panel_labels_and_legend_collection(umap, leiden_clustering, datastore):
+ first = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ show=False,
+ )
+ second = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_nCounts",
+ show=False,
+ )
+ axes = list(first.axes.values())
+ splt.label_panels(first.axes)
+ assert axes[0].texts[-1].get_text() == "A"
+ with pytest.raises(ValueError, match="labels length"):
+ splt.label_panels(first.axes, labels=["A", "B"])
+
+ legends = splt.collect_legends(first.figure, [first, second])
+ assert legends == first.legends + second.legends
+ first.close()
+ second.close()
diff --git a/tests/test_plotting_foundation.py b/tests/test_plotting_foundation.py
new file mode 100644
index 00000000..efde3d0b
--- /dev/null
+++ b/tests/test_plotting_foundation.py
@@ -0,0 +1,1130 @@
+"""Foundation and integration tests for scarf.plotting."""
+
+import subprocess
+import sys
+from pathlib import Path
+
+import matplotlib
+
+matplotlib.use("Agg")
+
+import numpy as np
+import pandas as pd
+import pytest
+
+import scarf.plotting as splt
+
+
+def test_import_plotting_exports():
+ function_names = (
+ "cluster_connectivity",
+ "cluster_tree",
+ "collect_legends",
+ "compose_results",
+ "composition",
+ "distribution",
+ "dotplot",
+ "elbow",
+ "embedding",
+ "embedding_raster",
+ "graph_qc",
+ "highly_variable_features",
+ "label_panels",
+ "marker_heatmap",
+ "mapping_calibration",
+ "mapping_confusion",
+ "mapping_evidence",
+ "mapping_projection",
+ "mapping_score",
+ "matrixplot",
+ "pseudotime_heatmap",
+ "qc",
+ "register_theme",
+ "run_recipe",
+ "theme_context",
+ )
+ result_names = (
+ "CategoricalScale",
+ "CellField",
+ "ColorScale",
+ "DensityOverlay",
+ "FeatureRef",
+ "Highlight",
+ "LegendSpec",
+ "NormalizationSpec",
+ "PlotProvenance",
+ "PlotOutput",
+ "PlotOutputSettings",
+ "PlotPanelTarget",
+ "PlotRecipe",
+ "PlotRecipeResult",
+ "PlotResult",
+ "PlotStep",
+ "SizeScale",
+ "StudyDesign",
+ )
+
+ assert all(name in splt.__all__ for name in (*function_names, *result_names))
+ assert all(callable(getattr(splt, name)) for name in function_names)
+ assert all(getattr(splt, name) is not None for name in result_names)
+
+
+@pytest.mark.parametrize("name", ["mapping_correction", "unified_embedding"])
+def test_retired_mapping_plots_are_absent(name):
+ from scarf.plotting.recipes import ALLOWED_PLOTS
+
+ assert name not in splt.__all__
+ assert name not in ALLOWED_PLOTS
+ with pytest.raises(AttributeError):
+ getattr(splt, name)
+
+
+def test_plotting_modules_import_without_optional_dependencies():
+ script = """
+import builtins
+import pandas as pd
+
+original_import = builtins.__import__
+
+def block_plotting_dependencies(name, *args, **kwargs):
+ if name.split(".", 1)[0] in {"matplotlib", "seaborn", "kneed"}:
+ raise ModuleNotFoundError(name)
+ return original_import(name, *args, **kwargs)
+
+builtins.__import__ = block_plotting_dependencies
+import scarf.plotting as plotting
+try:
+ plotting.elbow([1.0, 0.5], show=False)
+except ImportError as exc:
+ assert "scarf[extra]" in str(exc)
+else:
+ raise AssertionError("plot use should require optional dependencies")
+try:
+ plotting.qc(
+ pd.DataFrame({"groups": ["a"], "value": [1.0]}),
+ show=False,
+ )
+except ImportError as exc:
+ assert "scarf[extra]" in str(exc)
+else:
+ raise AssertionError("plot use should require matplotlib")
+"""
+ completed = subprocess.run(
+ [sys.executable, "-c", script],
+ cwd=Path(__file__).resolve().parents[1],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+ assert completed.returncode == 0, completed.stderr
+
+
+def test_study_design_allows_pairing_rejects_tech_rep():
+ design = splt.StudyDesign(
+ sample_by="sample", subject_by="donor", condition_by="time"
+ )
+ assert design.subject_by == "donor"
+ with pytest.raises(NotImplementedError, match="technical_replicate_by"):
+ splt.StudyDesign(sample_by="sample", technical_replicate_by="lane")
+
+
+def test_size_scale_maps_fraction_to_area():
+ scale = splt.SizeScale(vmin=0, vmax=1, size_min=10, size_max=200)
+ areas = scale.areas(np.array([0.0, 0.5, 1.0]))
+ assert areas[0] == pytest.approx(10)
+ assert areas[1] == pytest.approx(105)
+ assert areas[2] == pytest.approx(200)
+
+
+def test_plotting_contracts_reject_invalid_values():
+ with pytest.raises(ValueError, match="source"):
+ splt.NormalizationSpec(source="scaled")
+ with pytest.raises(ValueError, match="quantiles"):
+ splt.ColorScale(quantiles=(0.9, 0.1))
+ with pytest.raises(ValueError, match="vmax"):
+ splt.ColorScale(vmin=2, vmax=1)
+ with pytest.raises(ValueError, match="size range"):
+ splt.SizeScale(size_min=20, size_max=10)
+ with pytest.raises(ValueError, match="kind"):
+ splt.CellField("group", kind="ordinal")
+
+
+def test_equal_weight_sample_aggregation_fixture():
+ """Two samples of very different size must weight equally."""
+ ps = pd.DataFrame(
+ {
+ "sample": ["A", "B"],
+ "group": ["g1", "g1"],
+ "feature": ["f1", "f1"],
+ "mean": [1.0, 10.0],
+ "fraction": [0.1, 0.9],
+ "n_cells": [10, 1000],
+ }
+ )
+ agg = (
+ ps.groupby(["group", "feature"], observed=False)
+ .agg(
+ mean=("mean", "mean"),
+ fraction=("fraction", "mean"),
+ n_cells=("n_cells", "sum"),
+ )
+ .reset_index()
+ )
+ assert len(agg) == 1
+ assert agg["mean"].iloc[0] == pytest.approx(5.5)
+ assert agg["fraction"].iloc[0] == pytest.approx(0.5)
+ # Cell-weighted would be ~9.91, not 5.5
+ cell_weighted = np.average(ps["mean"], weights=ps["n_cells"])
+ assert cell_weighted == pytest.approx(9.910891, rel=1e-5)
+ assert agg["mean"].iloc[0] != pytest.approx(cell_weighted)
+
+
+def test_embedding_keeps_square_panel_with_side_legend(
+ umap, leiden_clustering, datastore
+):
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ show=False,
+ )
+ ax = next(iter(result.axes.values()))
+ assert ax.get_box_aspect() == pytest.approx(1.0)
+ xlim = ax.get_xlim()
+ ylim = ax.get_ylim()
+ assert (xlim[1] - xlim[0]) == pytest.approx(ylim[1] - ylim[0])
+ result.figure.canvas.draw()
+ bbox = ax.get_window_extent()
+ assert bbox.width == pytest.approx(bbox.height, rel=1e-3)
+ result.close()
+
+
+def test_embedding_dotplot_matrixplot_on_fixture(umap, leiden_clustering, datastore):
+ ds = datastore
+ # Point sizes and sort order are part of the native embedding contract.
+ n = len(ds.cells.fetch("I", key="I"))
+ sizes = np.linspace(5, 40, n)
+ emb = splt.embedding(
+ ds,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ point_sizes=sizes,
+ sort_values=False,
+ show=False,
+ )
+ assert emb.owns_figure
+ assert len(emb.axes) == 1
+ assert emb.figure.legends or next(iter(emb.axes.values())).get_legend() is not None
+ emb.close()
+
+ # Gene coloring with sort_values (high expression on top)
+ names = ds.RNA.feats.fetch_all("names")
+ gene = str(names[0])
+ emb2 = splt.embedding(
+ ds,
+ layout_key="RNA_UMAP",
+ color_by=gene,
+ sort_values=True,
+ show=False,
+ )
+ assert emb2.provenance.extras.get("sort_values") is True
+ emb2.close()
+
+ dp = splt.dotplot(
+ ds,
+ features=[gene],
+ group_by="RNA_leiden_cluster",
+ show=False,
+ )
+ assert "aggregate" in dp.tables
+ assert "mean" in dp.tables["aggregate"].columns
+ assert "fraction" in dp.tables["aggregate"].columns
+ assert dp.provenance.n_cells == len(ds.cells.active_index("I"))
+ assert dp.figure.legends or next(iter(dp.axes.values())).get_legend() is not None
+ dp.close()
+
+ mp = splt.matrixplot(
+ ds,
+ features=[gene],
+ group_by="RNA_leiden_cluster",
+ show=False,
+ )
+ assert "matrix" in mp.tables
+ assert mp.provenance.n_cells == len(ds.cells.active_index("I"))
+ mp.close()
+
+
+def test_feature_ref_duplicate_raises(datastore):
+ # Looking up by a nonsense name
+ with pytest.raises(KeyError):
+ splt.FeatureRef # noqa: B018 - ensure import path
+ from scarf.plotting._data import resolve_feature
+
+ resolve_feature(datastore, "___not_a_real_feature___")
+
+
+def test_caller_owned_target(umap, datastore):
+ import matplotlib.pyplot as plt
+
+ fig, ax = plt.subplots()
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ target=ax,
+ show=False,
+ )
+ assert result.owns_figure is False
+ result.close() # must not close foreign figure
+ assert plt.fignum_exists(fig.number)
+ plt.close(fig)
+
+
+def test_summary_and_composition_accept_foreign_targets(
+ umap, leiden_clustering, datastore
+):
+ import matplotlib.pyplot as plt
+
+ ds = datastore
+ gene = str(ds.RNA.feats.fetch_all("names")[0])
+ fig, axes = plt.subplots(1, 3)
+ results = [
+ splt.dotplot(
+ ds,
+ features=[gene],
+ group_by="RNA_leiden_cluster",
+ target=axes[0],
+ show=False,
+ ),
+ splt.matrixplot(
+ ds,
+ features=[gene],
+ group_by="RNA_leiden_cluster",
+ target=axes[1],
+ show=False,
+ ),
+ splt.composition(
+ ds,
+ category_by="RNA_leiden_cluster",
+ kind="stacked",
+ target=axes[2],
+ show=False,
+ ),
+ ]
+ assert all(result.owns_figure is False for result in results)
+ for result in results:
+ result.close()
+ assert plt.fignum_exists(fig.number)
+ plt.close(fig)
+
+
+def test_sample_by_equal_weight_on_datastore(umap, leiden_clustering, datastore):
+ from scarf.plotting._data import summarize_features_by_group
+
+ ds = datastore
+ active_n = len(ds.cells.active_index("I"))
+ # Unbalanced samples among active cells: 5 vs rest
+ sample = np.array(["big"] * active_n, dtype=object)
+ sample[:5] = "small"
+ ds.cells.insert("plot_sample_id", sample, overwrite=True)
+
+ gene = str(ds.RNA.feats.fetch_all("names")[0])
+ agg, per = summarize_features_by_group(
+ ds,
+ features=[gene],
+ group_by="RNA_leiden_cluster",
+ sample_by="plot_sample_id",
+ )
+ assert per is not None
+ assert set(per["sample"].unique()) == {"big", "small"}
+
+ # For each group×feature present in both samples, aggregate mean ==
+ # unweighted mean of per-sample means (not cell-weighted).
+ both = per.groupby(["RNA_leiden_cluster", "feature"], observed=False)[
+ "sample"
+ ].nunique()
+ shared = both[both == 2].index
+ assert len(shared) > 0
+ for cluster, feature in shared:
+ rows = per[(per["RNA_leiden_cluster"] == cluster) & (per["feature"] == feature)]
+ expected = float(rows["mean"].mean())
+ cell_weighted = float(np.average(rows["mean"], weights=rows["n_cells"]))
+ got = float(
+ agg.loc[
+ (agg["RNA_leiden_cluster"] == cluster) & (agg["feature"] == feature),
+ "mean",
+ ].iloc[0]
+ )
+ assert got == pytest.approx(expected, rel=1e-6, abs=1e-8)
+ # When sample sizes and per-sample means differ, equal-weight != cell-weight
+ if (
+ rows["n_cells"].nunique() > 1
+ and rows["mean"].nunique() > 1
+ and not np.allclose(rows["mean"], 0)
+ ):
+ assert got != pytest.approx(cell_weighted, rel=1e-3)
+
+ dp = splt.dotplot(
+ ds,
+ features=[gene],
+ group_by="RNA_leiden_cluster",
+ sample_by="plot_sample_id",
+ show=False,
+ )
+ assert "per_sample" in dp.tables
+ assert "n_samples" in dp.tables["aggregate"].columns
+ assert dp.provenance.n_samples == 2
+ assert dp.provenance.extras["dropped_sample_cells"] == 0
+ dp.close()
+
+
+def test_facet_shared_color_limits(umap, datastore):
+ ds = datastore
+ active_n = len(ds.cells.active_index("I"))
+ condition = np.array(["low"] * active_n, dtype=object)
+ condition[active_n // 2 :] = "high"
+ score = np.zeros(active_n, dtype=np.float64)
+ score[condition == "low"] = np.linspace(0.0, 1.0, int((condition == "low").sum()))
+ score[condition == "high"] = np.linspace(
+ 10.0, 11.0, int((condition == "high").sum())
+ )
+ ds.cells.insert("plot_condition", condition, overwrite=True)
+ ds.cells.insert("plot_score", score, overwrite=True)
+
+ result = splt.embedding(
+ ds,
+ layout_key="RNA_UMAP",
+ color_by=splt.CellField("plot_score", kind="continuous"),
+ facet_by="plot_condition",
+ facet_order=["low", "high"],
+ show=False,
+ )
+ limits = result.provenance.extras["color_limits"]
+ assert "plot_score" in limits
+ vmin, vmax = limits["plot_score"]
+ assert vmin == pytest.approx(0.0, abs=1e-6)
+ assert vmax == pytest.approx(11.0, abs=1e-6)
+ # Both facet panels must exist and share coordinate limits
+ assert len(result.axes) == 2
+ xlims = {ax.get_xlim() for ax in result.axes.values()}
+ ylims = {ax.get_ylim() for ax in result.axes.values()}
+ assert len(xlims) == 1
+ assert len(ylims) == 1
+ result.close()
+
+ panel_result = splt.embedding(
+ ds,
+ layout_key="RNA_UMAP",
+ color_by=splt.CellField("plot_score", kind="continuous"),
+ facet_by="plot_condition",
+ facet_order=["low", "high"],
+ color_scale=splt.ColorScale(scope="panel"),
+ show=False,
+ )
+ panel_limits = list(panel_result.provenance.extras["color_limits"].values())
+ assert panel_limits[0] == pytest.approx((0.0, 1.0))
+ assert panel_limits[1] == pytest.approx((10.0, 11.0))
+ assert len(panel_result.figure.axes) == 4
+ panel_result.close()
+
+ ds.cells.insert("plot_score_scaled", score * 10, overwrite=True)
+ shared_result = splt.embedding(
+ ds,
+ layout_key="RNA_UMAP",
+ color_by=[
+ splt.CellField("plot_score", kind="continuous"),
+ splt.CellField("plot_score_scaled", kind="continuous"),
+ ],
+ color_scale=splt.ColorScale(scope="shared"),
+ show=False,
+ )
+ shared_limits = list(shared_result.provenance.extras["color_limits"].values())
+ assert shared_limits[0] == pytest.approx(shared_limits[1])
+ assert shared_limits[1][1] == pytest.approx(110.0)
+ shared_result.close()
+
+
+def test_composition_and_export(umap, leiden_clustering, datastore, tmp_path):
+ ds = datastore
+ active_n = len(ds.cells.active_index("I"))
+ sample = np.array([f"s{i % 3}" for i in range(active_n)], dtype=object)
+ ds.cells.insert("plot_comp_sample", sample, overwrite=True)
+
+ result = splt.composition(
+ ds,
+ category_by="RNA_leiden_cluster",
+ sample_by="plot_comp_sample",
+ kind="per_sample",
+ show=False,
+ )
+ assert "per_sample" in result.tables
+ out = result.save(tmp_path / "composition.png", dpi=100)
+ assert out.exists() and out.stat().st_size > 0
+ result.close()
+
+ stacked = splt.composition(
+ ds,
+ category_by="RNA_leiden_cluster",
+ sample_by="plot_comp_sample",
+ show=False,
+ )
+ pdf = stacked.save(tmp_path / "composition.pdf", exact_size=True)
+ assert pdf.exists()
+ stacked.close()
+
+
+def test_feature_plotting_uses_assay_normalization_adapter(
+ umap, datastore, monkeypatch
+):
+ ds = datastore
+ assay = ds.RNA
+ gene = str(assay.feats.fetch_all("names")[0])
+ native_normed = assay.normed
+ calls = []
+
+ def tracked_normed(*args, **kwargs):
+ calls.append(kwargs)
+ return native_normed(*args, **kwargs)
+
+ monkeypatch.setattr(assay, "normed", tracked_normed)
+ result = splt.embedding(
+ ds,
+ layout_key="RNA_UMAP",
+ color_by=gene,
+ normalization=splt.NormalizationSpec(transform="log1p"),
+ sort_values=True,
+ show=False,
+ )
+ result.close()
+ assert len(calls) == 1
+ assert set(calls[0]) == {"cell_idx", "feat_idx"}
+
+
+def _assert_plotting_fetch_matches_assay_normed(
+ datastore, assay_name, requested_indices, cell_idx
+):
+ from scarf.plotting._data import (
+ fetch_normalized_feature_matrix,
+ resolve_feature,
+ )
+ from scarf.utils import controlled_compute
+
+ assay = datastore._get_assay(assay_name)
+ resolved = [
+ resolve_feature(
+ datastore,
+ splt.FeatureRef(value=index, assay=assay_name, by="index"),
+ )
+ for index in requested_indices
+ ]
+ physical_indices = np.unique(np.asarray(requested_indices, dtype=np.int64))
+ expected = controlled_compute(
+ assay.normed(cell_idx=cell_idx, feat_idx=physical_indices),
+ datastore.nthreads,
+ ).astype(np.float64)
+ local_order = np.searchsorted(
+ physical_indices, np.asarray(requested_indices, dtype=np.int64)
+ )
+ expected = expected[:, local_order]
+ fetched = fetch_normalized_feature_matrix(
+ datastore,
+ resolved,
+ cell_idx,
+ normalization=splt.NormalizationSpec(source="assay"),
+ )
+ logged = fetch_normalized_feature_matrix(
+ datastore,
+ resolved,
+ cell_idx,
+ normalization=splt.NormalizationSpec(source="assay", transform="log1p"),
+ )
+ np.testing.assert_allclose(fetched, expected)
+ np.testing.assert_allclose(logged, np.log1p(expected))
+
+
+def test_plotting_fetch_matches_rna_normed(datastore):
+ cell_idx = datastore.cells.active_index("I")[:32]
+ _assert_plotting_fetch_matches_assay_normed(
+ datastore,
+ "RNA",
+ [1, 0],
+ cell_idx,
+ )
+
+
+def test_plotting_fetch_matches_adt_normed(toy_crdir_ds):
+ cell_idx = np.arange(toy_crdir_ds.cells.N, dtype=np.int64)
+ _assert_plotting_fetch_matches_assay_normed(
+ toy_crdir_ds,
+ "ADT",
+ [1, 0],
+ cell_idx,
+ )
+
+
+def test_plotting_fetch_matches_atac_normed(atac_datastore):
+ cell_idx = atac_datastore.cells.active_index("I")[:32]
+ _assert_plotting_fetch_matches_assay_normed(
+ atac_datastore,
+ "ATAC",
+ [1, 0],
+ cell_idx,
+ )
+
+
+def test_plotting_fetch_preserves_assay_groups_order_and_reduction(toy_crdir_ds):
+ from dataclasses import replace
+
+ from scarf.plotting._data import (
+ fetch_normalized_feature_matrix,
+ resolve_feature,
+ )
+ from scarf.utils import controlled_compute
+
+ cell_idx = np.arange(toy_crdir_ds.cells.N, dtype=np.int64)
+ rna = [
+ resolve_feature(
+ toy_crdir_ds,
+ splt.FeatureRef(value=index, assay="RNA", by="index"),
+ )
+ for index in (0, 1)
+ ]
+ adt = [
+ resolve_feature(
+ toy_crdir_ds,
+ splt.FeatureRef(value=index, assay="ADT", by="index"),
+ )
+ for index in (0, 1)
+ ]
+ rna_sum = replace(
+ rna[0],
+ indices=(0, 1),
+ ids=rna[0].ids + rna[1].ids,
+ names=rna[0].names + rna[1].names,
+ reduction="sum",
+ )
+ fetched = fetch_normalized_feature_matrix(
+ toy_crdir_ds,
+ [adt[1], rna_sum, adt[0]],
+ cell_idx,
+ )
+ rna_native = controlled_compute(
+ toy_crdir_ds.RNA.normed(
+ cell_idx=cell_idx,
+ feat_idx=np.asarray([0, 1], dtype=np.int64),
+ ),
+ toy_crdir_ds.nthreads,
+ )
+ adt_native = controlled_compute(
+ toy_crdir_ds.ADT.normed(
+ cell_idx=cell_idx,
+ feat_idx=np.asarray([0, 1], dtype=np.int64),
+ ),
+ toy_crdir_ds.nthreads,
+ )
+ expected = np.column_stack(
+ (adt_native[:, 1], rna_native.sum(axis=1), adt_native[:, 0])
+ )
+ np.testing.assert_allclose(fetched, expected)
+
+
+def test_normalization_spec_supports_raw_and_log1p(datastore):
+ from scarf.plotting._data import (
+ fetch_normalized_feature_matrix,
+ resolve_feature,
+ )
+ from scarf.utils import controlled_compute
+
+ assay = datastore.RNA
+ cell_idx = datastore.cells.active_index("I")
+ gene = str(assay.feats.fetch_all("names")[0])
+ resolved = [resolve_feature(datastore, gene)]
+ raw = fetch_normalized_feature_matrix(
+ datastore,
+ resolved,
+ cell_idx,
+ normalization=splt.NormalizationSpec(source="raw"),
+ )
+ expected_raw = controlled_compute(
+ assay.rawData[:, [resolved[0].indices[0]]][cell_idx, :],
+ datastore.nthreads,
+ )
+ normalized = fetch_normalized_feature_matrix(
+ datastore,
+ resolved,
+ cell_idx,
+ normalization=splt.NormalizationSpec(),
+ )
+ raw_logged = fetch_normalized_feature_matrix(
+ datastore,
+ resolved,
+ cell_idx,
+ normalization=splt.NormalizationSpec(source="raw", transform="log1p"),
+ )
+ logged = fetch_normalized_feature_matrix(
+ datastore,
+ resolved,
+ cell_idx,
+ normalization=splt.NormalizationSpec(transform="log1p"),
+ )
+ assert np.array_equal(raw, expected_raw)
+ assert np.allclose(raw_logged, np.log1p(raw))
+ assert np.allclose(logged, np.log1p(normalized))
+
+
+def test_figsize_rejected_with_owned_target(umap, datastore):
+ import matplotlib.pyplot as plt
+
+ fig, ax = plt.subplots()
+ with pytest.raises(ValueError, match="figsize"):
+ splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ target=ax,
+ figsize=(3, 3),
+ show=False,
+ )
+ plt.close(fig)
+
+
+def test_multi_gene_by_condition_embedding(umap, datastore):
+ ds = datastore
+ active_n = len(ds.cells.active_index("I"))
+ condition = np.array(["ctrl"] * active_n, dtype=object)
+ condition[active_n // 2 :] = "stim"
+ ds.cells.insert("plot_condition_mg", condition, overwrite=True)
+
+ names = [str(x) for x in ds.RNA.feats.fetch_all("names")[:2]]
+ result = splt.embedding(
+ ds,
+ layout_key="RNA_UMAP",
+ color_by=names,
+ facet_by="plot_condition_mg",
+ facet_order=["ctrl", "stim"],
+ sort_values=True,
+ show=False,
+ )
+ assert result.provenance.extras["n_colors"] == 2
+ assert result.provenance.extras["n_facets"] == 2
+ assert len(result.axes) == 4
+ limits = result.provenance.extras["color_limits"]
+ for gene in names:
+ assert gene in limits
+ vmin, vmax = limits[gene]
+ assert vmax >= vmin
+ # Panel keys are (gene, condition)
+ for gene in names:
+ for cond in ("ctrl", "stim"):
+ assert (gene, cond) in result.axes
+ result.close()
+
+
+def _add_secondary_embedding(datastore):
+ x = np.asarray(datastore.cells.fetch_all("RNA_UMAP1"), dtype=np.float64)
+ y = np.asarray(datastore.cells.fetch_all("RNA_UMAP2"), dtype=np.float64)
+ datastore.cells.insert(
+ "plot_secondary_embedding1",
+ -y,
+ overwrite=True,
+ )
+ datastore.cells.insert(
+ "plot_secondary_embedding2",
+ x,
+ overwrite=True,
+ )
+
+
+def _add_synthetic_embeddings(datastore, assay_name):
+ coordinates = np.linspace(-1.0, 1.0, datastore.cells.N)
+ layouts = [
+ f"plot_{assay_name.lower()}_native_a",
+ f"plot_{assay_name.lower()}_native_b",
+ ]
+ datastore.cells.insert(f"{layouts[0]}1", coordinates, overwrite=True)
+ datastore.cells.insert(f"{layouts[0]}2", coordinates**2, overwrite=True)
+ datastore.cells.insert(f"{layouts[1]}1", -coordinates, overwrite=True)
+ datastore.cells.insert(f"{layouts[1]}2", coordinates[::-1], overwrite=True)
+ return layouts
+
+
+def test_multi_layout_multi_color_embedding(umap, datastore):
+ _add_secondary_embedding(datastore)
+ layouts = ["RNA_UMAP", "plot_secondary_embedding"]
+ colors = ["RNA_nCounts", "RNA_nFeatures"]
+ expected_keys = [(layout, color) for layout in layouts for color in colors]
+
+ result = splt.embedding(
+ datastore,
+ layout_key=layouts,
+ color_by=colors,
+ show=False,
+ )
+
+ assert result.owns_figure is True
+ assert list(result.axes) == expected_keys
+ assert result.provenance.extras["layouts"] == layouts
+ assert result.provenance.extras["n_layouts"] == 2
+ assert set(result.provenance.extras["layout_provenance"]) == set(layouts)
+ assert "multi_layout" in result.provenance.notes
+ assert len(result.legends) == 2
+ assert len(result.scales) == 1
+ assert {legend.kind for legend in result.legends} == {"colorbar"}
+ assert {legend.label for legend in result.legends} == set(colors)
+ assert isinstance(result.scales[0], splt.ColorScale)
+ assert all(
+ np.isfinite([legend.extras["vmin"], legend.extras["vmax"]]).all()
+ for legend in result.legends
+ )
+ result.close()
+
+
+@pytest.mark.parametrize(
+ ("fixture_name", "assay_name"),
+ [
+ pytest.param("datastore", "RNA", id="rna"),
+ pytest.param("toy_crdir_ds", "ADT", id="adt"),
+ pytest.param("atac_datastore", "ATAC", id="atac"),
+ ],
+)
+def test_multi_layout_embedding_uses_native_feature_values(
+ request,
+ fixture_name,
+ assay_name,
+):
+ from scarf.utils import controlled_compute
+
+ datastore = request.getfixturevalue(fixture_name)
+ layouts = _add_synthetic_embeddings(datastore, assay_name)
+ label = f"{assay_name} native feature"
+ feature = splt.FeatureRef(
+ value=0,
+ assay=assay_name,
+ by="index",
+ label=label,
+ )
+ result = splt.embedding(
+ datastore,
+ layout_key=layouts,
+ color_by=feature,
+ show=False,
+ )
+
+ assay = datastore._get_assay(assay_name)
+ cell_index = datastore.cells.active_index("I")
+ native_values = controlled_compute(
+ assay.normed(
+ cell_idx=cell_index,
+ feat_idx=np.asarray([0], dtype=np.int64),
+ ),
+ datastore.nthreads,
+ ).reshape(-1)
+ expected_limits = (float(native_values.min()), float(native_values.max()))
+ if expected_limits[1] <= expected_limits[0]:
+ expected_limits = (expected_limits[0], expected_limits[0] + 1.0)
+
+ assert list(result.axes) == [(layout, label) for layout in layouts]
+ assert result.provenance.assay == assay_name
+ assert result.provenance.extras["assays"] == [assay_name]
+ for layout in layouts:
+ limits = result.provenance.extras["color_limits_by_layout"][layout]
+ assert limits[label] == pytest.approx(expected_limits)
+ result.close()
+
+
+def test_multi_layout_embedding_accepts_matching_target_axes(umap, datastore):
+ import matplotlib.pyplot as plt
+
+ _add_secondary_embedding(datastore)
+ layouts = ["RNA_UMAP", "plot_secondary_embedding"]
+ colors = ["RNA_nCounts", "RNA_nFeatures"]
+ panel_keys = [(layout, color) for layout in layouts for color in colors]
+ figure, target_axes = plt.subplots(2, 2)
+ target = dict(zip(panel_keys, target_axes.ravel(), strict=True))
+
+ result = splt.embedding(
+ datastore,
+ layout_key=layouts,
+ color_by=colors,
+ target=target,
+ show=False,
+ )
+
+ assert result.owns_figure is False
+ assert result.figure is figure
+ assert result.axes == target
+ result.close()
+ assert plt.fignum_exists(figure.number)
+ plt.close(figure)
+
+
+def test_embedding_show_default_suppression_and_later_show(
+ umap,
+ datastore,
+ monkeypatch,
+):
+ shown = []
+
+ def track_show(result):
+ shown.append(result)
+
+ monkeypatch.setattr(splt.PlotResult, "show", track_show)
+ default_result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_nCounts",
+ )
+ suppressed_result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_nCounts",
+ show=False,
+ )
+
+ assert shown == [default_result]
+ suppressed_result.show()
+ assert shown == [default_result, suppressed_result]
+ default_result.close()
+ suppressed_result.close()
+
+
+def test_resolve_feature_by_index(datastore):
+ from scarf.plotting._data import resolve_feature
+
+ resolved = resolve_feature(
+ datastore, splt.FeatureRef(value=0, by="index", assay="RNA")
+ )
+ assert resolved.indices == (0,)
+ assert resolved.assay == "RNA"
+ assert resolved.label
+
+
+def test_label_panels_and_collect_legends(umap, datastore):
+ import matplotlib.pyplot as plt
+
+ ds = datastore
+ fig, axes = plt.subplot_mosaic([["A", "B"]], figsize=(6, 3))
+ a = splt.embedding(
+ ds,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ target=axes["A"],
+ show=False,
+ )
+ gene = str(ds.RNA.feats.fetch_all("names")[0])
+ b = splt.embedding(
+ ds,
+ layout_key="RNA_UMAP",
+ color_by=gene,
+ target=axes["B"],
+ show=False,
+ )
+ splt.label_panels({"A": axes["A"], "B": axes["B"]}, labels=["A", "B"])
+ legends = splt.collect_legends(fig, [a, b])
+ assert len(legends) >= 1
+ a.close()
+ b.close()
+ plt.close(fig)
+
+
+def test_paired_composition_draws_subject_lines(umap, leiden_clustering, datastore):
+ ds = datastore
+ active_n = len(ds.cells.active_index("I"))
+ sample = np.array([f"s{i % 6}" for i in range(active_n)], dtype=object)
+ subject = np.array([f"d{i % 3}" for i in range(active_n)], dtype=object)
+ condition = np.array(
+ ["before" if i % 6 < 3 else "after" for i in range(active_n)],
+ dtype=object,
+ )
+ # Two samples per subject (s0,s3 -> d0; s1,s4 -> d1; s2,s5 -> d2)
+ ds.cells.insert("plot_pair_sample", sample, overwrite=True)
+ ds.cells.insert("plot_pair_subject", subject, overwrite=True)
+ ds.cells.insert("plot_pair_condition", condition, overwrite=True)
+
+ result = splt.composition(
+ ds,
+ category_by="RNA_leiden_cluster",
+ study_design=splt.StudyDesign(
+ sample_by="plot_pair_sample",
+ subject_by="plot_pair_subject",
+ condition_by="plot_pair_condition",
+ ),
+ kind="per_sample",
+ show=False,
+ )
+ assert "subject" in result.tables["per_sample"].columns
+ assert result.provenance.extras["n_pair_lines"] >= 1
+ assert any("paired_by=subject" in n for n in result.provenance.notes)
+ result.close()
+
+
+def test_paired_composition_requires_condition(leiden_clustering, datastore):
+ with pytest.raises(ValueError, match="requires condition_by"):
+ splt.composition(
+ datastore,
+ category_by="RNA_leiden_cluster",
+ sample_by="RNA_leiden_cluster",
+ subject_by="RNA_leiden_cluster",
+ kind="per_sample",
+ show=False,
+ )
+
+
+def test_embedding_clip_and_subset(umap, datastore):
+ ds = datastore
+ active_n = len(ds.cells.active_index("I"))
+ keep = np.zeros(active_n, dtype=bool)
+ keep[: max(10, active_n // 2)] = True
+ ds.cells.insert("plot_keep", keep, overwrite=True)
+ gene = str(ds.RNA.feats.fetch_all("names")[0])
+ result = splt.embedding(
+ ds,
+ layout_key="RNA_UMAP",
+ color_by=gene,
+ clip_fraction=0.01,
+ subset_by="plot_keep",
+ show=False,
+ )
+ assert result.provenance.extras["clip_fraction"] == 0.01
+ assert result.provenance.extras["subset_by"] == "plot_keep"
+ result.close()
+
+
+def test_embedding_groups_filters_categories(umap, leiden_clustering, datastore):
+ ds = datastore
+ labels = list(pd.unique(ds.cells.fetch("RNA_leiden_cluster")))
+ assert len(labels) >= 2
+ keep = labels[:2]
+ result = splt.embedding(
+ ds,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ groups=keep,
+ show=False,
+ )
+ assert result.provenance.extras["groups"] == list(keep)
+ cat_scale = next(s for s in result.scales if isinstance(s, splt.CategoricalScale))
+ assert list(cat_scale.order) == list(keep)
+ result.close()
+
+
+def test_distribution_violin(umap, leiden_clustering, datastore):
+ ds = datastore
+ result = splt.distribution(
+ ds,
+ keys=["RNA_nCounts", "RNA_nFeatures"],
+ group_by="RNA_leiden_cluster",
+ kind="violin",
+ max_points=200,
+ seed=1,
+ show=False,
+ )
+ assert len(result.axes) == 2
+ assert "RNA_nCounts" in result.tables
+ assert result.provenance.extras["approximate"] is True
+ assert "subsampled_display" in result.provenance.notes
+ ax = next(iter(result.axes.values()))
+ rotations = {tick.get_rotation() for tick in ax.get_xticklabels()}
+ assert 45 in rotations or any(abs(r - 45) < 1e-6 for r in rotations)
+ # Grouped violins should not be a single steelblue fill.
+ face_colors = {
+ tuple(np.round(c.get_facecolor()[0][:3], 3))
+ for c in ax.collections
+ if hasattr(c, "get_facecolor") and len(c.get_facecolor())
+ }
+ assert len(face_colors) >= 2
+ result.close()
+
+ gene = str(ds.RNA.feats.fetch_all("names")[0])
+ result2 = splt.distribution(ds, keys=gene, kind="box", max_points=100, show=False)
+ assert len(result2.axes) == 1
+ result2.close()
+
+
+def test_distribution_cell_key_none_includes_all_cells(datastore):
+ result = splt.distribution(
+ datastore,
+ keys="RNA_nCounts",
+ cell_key=None,
+ max_points=0,
+ show=False,
+ )
+ assert result.provenance.cell_key is None
+ assert result.provenance.n_cells == datastore.cells.N
+ assert len(result.tables["RNA_nCounts"]) == datastore.cells.N
+ result.close()
+
+
+def test_distribution_subset_and_groups(umap, leiden_clustering, datastore):
+ ds = datastore
+ active_n = len(ds.cells.active_index("I"))
+ keep = np.zeros(active_n, dtype=bool)
+ keep[: max(20, active_n // 2)] = True
+ ds.cells.insert("dist_keep", keep, overwrite=True)
+ labels = list(pd.unique(ds.cells.fetch("RNA_leiden_cluster")))
+ keep_groups = labels[:2]
+ result = splt.distribution(
+ ds,
+ keys="RNA_nCounts",
+ group_by="RNA_leiden_cluster",
+ groups=keep_groups,
+ subset_by="dist_keep",
+ kind="box",
+ max_points=0,
+ show=False,
+ )
+ assert result.provenance.extras["subset_by"] == "dist_keep"
+ assert result.provenance.extras["groups"] == list(keep_groups)
+ table_groups = set(result.tables["RNA_nCounts"]["group"].unique())
+ assert table_groups == set(keep_groups)
+ assert result.provenance.n_cells == len(result.tables["RNA_nCounts"])
+ assert result.provenance.n_cells < active_n
+ result.close()
+
+
+def test_distribution_hist_and_ecdf(umap, leiden_clustering, datastore):
+ ds = datastore
+ hist = splt.distribution(
+ ds,
+ keys="RNA_nCounts",
+ group_by="RNA_leiden_cluster",
+ kind="hist",
+ bins=20,
+ show=False,
+ )
+ assert hist.provenance.extras["bins"] == 20
+ assert hist.provenance.extras["approximate"] is False
+ ax = next(iter(hist.axes.values()))
+ n_groups = len(np.unique(ds.cells.fetch("RNA_leiden_cluster")))
+ assert len(ax.patches) == n_groups * 20
+ first_bins = [(patch.get_x(), patch.get_width()) for patch in ax.patches[:20]]
+ for group_index in range(1, n_groups):
+ offset = group_index * 20
+ group_bins = [
+ (patch.get_x(), patch.get_width())
+ for patch in ax.patches[offset : offset + 20]
+ ]
+ assert group_bins == pytest.approx(first_bins)
+ hist.close()
+
+ ecdf = splt.distribution(
+ ds,
+ keys="RNA_nFeatures",
+ kind="ecdf",
+ max_points=500,
+ seed=2,
+ show=False,
+ )
+ assert "ecdf" in ecdf.provenance.notes
+ assert ecdf.provenance.extras["approximate"] is True
+ ecdf.close()
+
+ duplicates = splt.distribution(
+ ds,
+ keys=["RNA_nCounts", "RNA_nCounts"],
+ kind="hist",
+ bins=5,
+ show=False,
+ )
+ assert set(duplicates.tables) == {"0:RNA_nCounts", "1:RNA_nCounts"}
+ duplicates.close()
diff --git a/tests/test_plotting_heatmaps.py b/tests/test_plotting_heatmaps.py
new file mode 100644
index 00000000..dcfdede2
--- /dev/null
+++ b/tests/test_plotting_heatmaps.py
@@ -0,0 +1,483 @@
+import matplotlib
+import networkx as nx
+import numpy as np
+import pandas as pd
+import pytest
+
+matplotlib.use("Agg")
+
+import scarf.plotting as splt
+from scarf.plotting.cluster_tree import _hierarchy_positions
+
+
+def test_hierarchy_positions_are_pure_and_complete():
+ graph = nx.DiGraph([(4, 2), (4, 3), (2, 0), (2, 1)])
+ before = graph.copy()
+
+ positions = _hierarchy_positions(graph, width=2.0)
+
+ assert set(positions) == set(graph)
+ assert nx.utils.graphs_equal(graph, before)
+ assert positions[4][1] == 0.0
+ assert positions[0][1] < positions[2][1]
+
+
+def test_marker_heatmap_returns_owned_result(
+ marker_search,
+ datastore,
+):
+ result = splt.marker_heatmap(
+ datastore,
+ group_key="RNA_cluster",
+ topn=3,
+ figsize=(4, 6),
+ show=False,
+ )
+
+ assert isinstance(result, splt.PlotResult)
+ assert result.owns_figure is True
+ assert tuple(result.figure.get_size_inches()) == (4, 6)
+ assert {"heatmap", "row_dendrogram", "column_dendrogram", "colorbar"} <= set(
+ result.axes
+ )
+ assert not result.tables["matrix"].empty
+ assert not result.tables["markers"].empty
+ assert result.legends
+ assert result.scales
+ assert result.provenance.notes[0] == "marker_heatmap"
+ result.close()
+
+
+def test_marker_heatmap_selects_features_by_named_score(
+ marker_search,
+ datastore,
+):
+ from scarf.features.markers.table import load_marker_table
+
+ assay = datastore.RNA
+ marker_slot = datastore._resolve_marker_group("RNA", "I", "RNA_cluster")
+ feature_names = np.asarray(assay.feats.fetch_all("names"))
+ feature_ids = np.asarray(assay.feats.fetch_all("ids"))
+ expected_by_group: dict[str, list[str]] = {}
+ for group_name in marker_slot.group_keys():
+ markers = load_marker_table(
+ marker_slot,
+ marker_slot[group_name],
+ feature_names,
+ group_id=group_name,
+ feature_ids=feature_ids,
+ )
+ ranked = markers.sort_values(
+ ["score", "feature_name"],
+ ascending=[False, True],
+ kind="mergesort",
+ ).head(2)
+ expected_by_group[group_name] = ranked["feature_name"].astype(str).tolist()
+
+ result = splt.marker_heatmap(
+ datastore,
+ group_key="RNA_cluster",
+ topn=2,
+ cluster_rows=False,
+ cluster_columns=False,
+ show=False,
+ )
+ selected = result.tables["markers"]
+ for group_name, expected_names in expected_by_group.items():
+ got = (
+ selected.loc[selected["group"].astype(str) == str(group_name)]
+ .sort_values("rank")["feature"]
+ .astype(str)
+ .tolist()
+ )
+ assert got == expected_names
+ result.close()
+
+
+def test_marker_heatmap_propagates_marker_metadata_errors(
+ marker_search,
+ datastore,
+):
+ marker_slot = datastore._resolve_marker_group("RNA", "I", "RNA_cluster")
+ original_method = marker_slot.attrs["method"]
+ marker_slot.attrs["method"] = "ttest"
+ try:
+ with pytest.raises(ValueError, match="Canonical marker metadata 'method'"):
+ splt.marker_heatmap(
+ datastore,
+ group_key="RNA_cluster",
+ topn=2,
+ show=False,
+ )
+ finally:
+ marker_slot.attrs["method"] = original_method
+
+
+def test_marker_heatmap_skips_unresolved_legacy_names_with_warning(
+ datastore_ephemeral,
+):
+ from scarf.utils import logger
+
+ assay = datastore_ephemeral.RNA
+ group_key = "legacy_heatmap_groups"
+ groups = np.arange(datastore_ephemeral.cells.N) % 2
+ datastore_ephemeral.cells.insert(group_key, groups, overwrite=True)
+ markers_group = (
+ assay.z["markers"] if "markers" in assay.z else assay.z.create_group("markers")
+ )
+ slot = markers_group.create_group(f"I__{group_key}")
+ feature_ids = np.asarray(assay.feats.fetch_all("ids")).astype(str)
+ for group_id, feature_index in enumerate((0, 1)):
+ cluster = slot.create_group(str(group_id))
+ cluster.create_array(
+ "names",
+ data=np.array(
+ [
+ f"removed_feature_{group_id}",
+ feature_ids[feature_index],
+ ]
+ ),
+ )
+ cluster.create_array("scores", data=np.array([1.0, 0.9]))
+
+ warnings: list[str] = []
+ sink = logger.add(
+ lambda message: warnings.append(message.record["message"]),
+ level="WARNING",
+ )
+ try:
+ result = splt.marker_heatmap(
+ datastore_ephemeral,
+ from_assay="RNA",
+ group_key=group_key,
+ cell_key="I",
+ topn=1,
+ cluster_rows=False,
+ cluster_columns=False,
+ show=False,
+ )
+ finally:
+ logger.remove(sink)
+
+ assert set(result.tables["markers"]["feature_index"]) == {0, 1}
+ assert all(
+ "removed_feature" not in value for value in result.tables["matrix"].index
+ )
+ assert any("unresolved legacy marker" in message for message in warnings)
+ result.close()
+
+
+def test_marker_heatmap_accepts_explicit_order_annotations_and_target(
+ marker_search,
+ datastore,
+):
+ import matplotlib.pyplot as plt
+
+ baseline = splt.marker_heatmap(
+ datastore,
+ group_key="RNA_cluster",
+ topn=2,
+ cluster_rows=False,
+ cluster_columns=False,
+ show=False,
+ )
+ row_order = list(reversed(baseline.tables["matrix"].index.tolist()))
+ column_order = list(reversed(baseline.tables["matrix"].columns.tolist()))
+ baseline.close()
+ row_annotation = {
+ feature: "first" if index < len(row_order) / 2 else "second"
+ for index, feature in enumerate(row_order)
+ }
+ figure, ax = plt.subplots(figsize=(4, 4))
+
+ result = splt.marker_heatmap(
+ datastore,
+ group_key="RNA_cluster",
+ topn=2,
+ row_order=row_order,
+ column_order=column_order,
+ cluster_rows=False,
+ cluster_columns=False,
+ row_annotations={"marker set": row_annotation},
+ target=ax,
+ show_legend=False,
+ show=False,
+ )
+
+ assert result.owns_figure is False
+ assert result.tables["matrix"].index.tolist() == row_order
+ assert result.tables["matrix"].columns.tolist() == column_order
+ assert "row_annotations" in result.tables
+ assert result.provenance.extras["cluster_rows"] is False
+ assert ax.patches
+ result.close()
+ assert plt.fignum_exists(figure.number)
+ plt.close(figure)
+
+
+def test_matrixplot_orders_clusters_and_annotates_axes(
+ umap,
+ leiden_clustering,
+ datastore,
+):
+ genes = [str(value) for value in datastore.RNA.feats.fetch_all("names")[:4]]
+ baseline = splt.matrixplot(
+ datastore,
+ features=genes,
+ group_by="RNA_leiden_cluster",
+ show=False,
+ )
+ groups = baseline.tables["matrix"].columns[1:].tolist()
+ baseline.close()
+ feature_order = list(reversed(genes))
+ group_order = list(reversed(groups))
+
+ ordered = splt.matrixplot(
+ datastore,
+ features=genes,
+ group_by="RNA_leiden_cluster",
+ feature_order=feature_order,
+ group_order=group_order,
+ row_annotations={
+ "panel": {
+ gene: "A" if index < 2 else "B"
+ for index, gene in enumerate(feature_order)
+ }
+ },
+ column_annotations={
+ "parity": {
+ group: "even" if int(group) % 2 == 0 else "odd" for group in group_order
+ }
+ },
+ show=False,
+ )
+
+ matrix = ordered.tables["matrix"]
+ assert matrix["feature"].tolist() == feature_order
+ assert matrix.columns[1:].tolist() == group_order
+ assert {"row_annotations", "column_annotations"} <= set(ordered.tables)
+ assert len(ordered.scales) == 3
+ assert ordered.axes["matrixplot"].patches
+ annotation_labels = {text.get_text() for text in ordered.axes["matrixplot"].texts}
+ assert {"panel", "parity"} <= annotation_labels
+ ordered.close()
+
+ requested = list(reversed(genes))
+ input_ordered = splt.matrixplot(
+ datastore,
+ features=requested,
+ group_by="RNA_leiden_cluster",
+ show=False,
+ )
+ assert input_ordered.tables["matrix"]["feature"].tolist() == requested
+ input_ordered.close()
+
+ clustered = splt.matrixplot(
+ datastore,
+ features=genes,
+ group_by="RNA_leiden_cluster",
+ cluster_features=True,
+ cluster_groups=True,
+ show=False,
+ )
+ assert clustered.provenance.extras["cluster_features"] is True
+ assert clustered.provenance.extras["cluster_groups"] is True
+ clustered.close()
+
+
+def test_annotation_strips_work_before_tick_labels_are_created():
+ import matplotlib.pyplot as plt
+
+ from scarf.plotting._heatmap_utils import draw_annotation_strips
+
+ figure, ax = plt.subplots()
+ row_colors = pd.DataFrame({"row group": ["#111111", "#222222"]})
+ column_colors = pd.DataFrame({"column group": ["#333333", "#444444"]})
+
+ xlim, ylim = draw_annotation_strips(
+ ax,
+ row_colors=row_colors,
+ column_colors=column_colors,
+ n_rows=2,
+ n_columns=2,
+ )
+
+ assert xlim[0] < -0.5
+ assert ylim[1] < -0.5
+ assert {text.get_text() for text in ax.texts} == {
+ "row group",
+ "column group",
+ }
+ plt.close(figure)
+
+
+def test_clustermap_annotation_legend_reserves_space_with_column_tree(
+ marker_search,
+ datastore,
+):
+ baseline = splt.marker_heatmap(
+ datastore,
+ group_key="RNA_cluster",
+ topn=2,
+ cluster_columns=False,
+ show=False,
+ )
+ groups = baseline.tables["matrix"].columns.tolist()
+ baseline.close()
+ levels = (
+ "relative cycling share: low",
+ "relative cycling share: medium",
+ "relative cycling share: high",
+ )
+ annotation = {
+ group: levels[index % len(levels)] for index, group in enumerate(groups)
+ }
+
+ result = splt.marker_heatmap(
+ datastore,
+ group_key="RNA_cluster",
+ topn=2,
+ cluster_columns=True,
+ column_annotations={"parity": annotation},
+ figsize=(6, 6),
+ show=False,
+ )
+
+ result.figure.canvas.draw()
+ renderer = result.figure.canvas.get_renderer()
+ legend_box = result.figure.legends[0].get_window_extent(renderer)
+ assert {text.get_text() for text in result.figure.legends[0].get_texts()} == {
+ f"parity: {level}" for level in levels
+ }
+ figure_box = result.figure.bbox
+ assert legend_box.x0 >= figure_box.x0
+ assert legend_box.x1 <= figure_box.x1
+ assert legend_box.y0 >= figure_box.y0
+ assert legend_box.y1 <= figure_box.y1
+ overlaps = [
+ name
+ for name, axis in result.axes.items()
+ if axis.get_visible() and legend_box.overlaps(axis.get_window_extent(renderer))
+ ]
+ assert overlaps == []
+ result.close()
+
+
+def test_cluster_tree_prepares_cache_and_returns_tables(
+ paris_clustering,
+ datastore,
+):
+ prepared = datastore._prepare_cluster_tree(cluster_key="RNA_cluster")
+ assert prepared["coalesced_location"] in datastore.zw
+ assert nx.is_tree(prepared["graph"])
+
+ cached = datastore._prepare_cluster_tree(cluster_key="RNA_cluster")
+ cached_labels = {
+ data["partition_id"]
+ for _, data in cached["graph"].nodes(data=True)
+ if "partition_id" in data
+ }
+ assert cached_labels == set(paris_clustering)
+
+ result = splt.cluster_tree(
+ datastore,
+ cluster_key="RNA_cluster",
+ figsize=(4, 4),
+ show=False,
+ )
+
+ assert isinstance(result, splt.PlotResult)
+ assert result.owns_figure is True
+ assert tuple(result.figure.get_size_inches()) == (4, 4)
+ assert {"nodes", "edges", "positions", "cluster_summary"} <= set(result.tables)
+ assert len(result.tables["cluster_summary"]) == len(set(paris_clustering))
+ assert result.provenance.extras["coalesced_location"] in datastore.zw
+ result.close()
+
+
+def test_pseudotime_heatmap_returns_aligned_tables(
+ pseudotime_aggregation,
+ datastore,
+):
+ result = splt.pseudotime_heatmap(
+ datastore,
+ cell_key="I",
+ feat_key="I",
+ feature_cluster_key="pseudotime_clusters",
+ pseudotime_key="RNA_pseudotime",
+ show_features=["Wsb1", "Rest"],
+ figsize=(4, 6),
+ show=False,
+ )
+
+ assert isinstance(result, splt.PlotResult)
+ assert result.owns_figure is True
+ assert tuple(result.figure.get_size_inches()) == (4, 6)
+ assert set(result.axes) == {
+ "heatmap",
+ "feature_clusters",
+ "colorbar",
+ "pseudotime",
+ }
+ assert result.tables["matrix"].shape[0] == len(result.tables["features"])
+ assert result.tables["matrix"].shape[1] == len(result.tables["pseudotime_bins"])
+ assert len(result.tables["pseudotime"]) == result.provenance.n_cells
+ assert result.legends
+ assert result.scales
+ result.close()
+
+
+def test_pseudotime_heatmap_accepts_composable_target(
+ pseudotime_aggregation,
+ datastore,
+):
+ import matplotlib.pyplot as plt
+
+ figure, axes = plt.subplot_mosaic(
+ [
+ ["heatmap", "feature_clusters", "colorbar"],
+ ["pseudotime", "pseudotime", "colorbar"],
+ ],
+ figsize=(6, 4),
+ )
+ result = splt.pseudotime_heatmap(
+ datastore,
+ cell_key="I",
+ feat_key="I",
+ feature_cluster_key="pseudotime_clusters",
+ pseudotime_key="RNA_pseudotime",
+ target=axes,
+ show=False,
+ )
+
+ assert result.owns_figure is False
+ assert result.figure is figure
+ assert np.isfinite(result.tables["matrix"].to_numpy()).all()
+ assert (
+ result.provenance.extras["feature_order"]
+ == result.tables["features"]["feature"].tolist()
+ )
+ result.close()
+ assert plt.fignum_exists(figure.number)
+ plt.close(figure)
+
+
+def test_pseudotime_heatmap_rejects_malformed_artifact_link(
+ pseudotime_aggregation,
+ datastore,
+):
+ column = datastore.RNA.z["featureData/pseudotime_clusters"]
+ original_ref = dict(column.attrs["source_artifact"])
+ column.attrs["source_artifact"] = "broken"
+ try:
+ with pytest.raises(ValueError, match="malformed source artifact"):
+ splt.pseudotime_heatmap(
+ datastore,
+ cell_key="I",
+ feat_key="I",
+ feature_cluster_key="pseudotime_clusters",
+ pseudotime_key="RNA_pseudotime",
+ show=False,
+ )
+ finally:
+ column.attrs["source_artifact"] = original_ref
diff --git a/tests/test_plotting_mapping.py b/tests/test_plotting_mapping.py
new file mode 100644
index 00000000..005f2e1e
--- /dev/null
+++ b/tests/test_plotting_mapping.py
@@ -0,0 +1,673 @@
+from pathlib import Path
+
+import matplotlib
+
+matplotlib.use("Agg")
+
+import numpy as np
+import pytest
+
+import scarf.mapping.confidence as mapping_confidence
+import scarf.plotting as splt
+from scarf.storage.artifacts import artifact_group
+from tests.test_mapping_label_transfer import (
+ _copied_query,
+ _plain_reference,
+ _write_projection,
+ _write_reference_layout,
+)
+
+
+@pytest.fixture
+def plotting_mapping_context(analyzed_datastore_ephemeral, tmp_path: Path):
+ reference_store = analyzed_datastore_ephemeral
+ reference = _plain_reference(reference_store)
+ query = _copied_query(reference_store, tmp_path / "plotting_query.zarr")
+ reference_layout, _ = _write_reference_layout(
+ reference,
+ layout_key="mapping_layout",
+ linked=True,
+ )
+ reference_labels = np.full(
+ reference.selected_cell_count,
+ "other",
+ dtype=object,
+ )
+ reference_labels[:4] = ["A", "A", "B", "B"]
+ reference.datastore.cells.insert(
+ "mapping_label",
+ reference_labels,
+ key=reference.cell_key,
+ overwrite=True,
+ )
+
+ query.cells.insert(
+ "mapping_layout1",
+ np.full(query.cells.N, -1000.0),
+ overwrite=True,
+ )
+ query.cells.insert(
+ "mapping_layout2",
+ np.full(query.cells.N, 1000.0),
+ overwrite=True,
+ )
+ query.cells.insert(
+ "mapping_label",
+ np.full(query.cells.N, "query_only", dtype=object),
+ overwrite=True,
+ )
+ indices = np.asarray(
+ [
+ [0, 1],
+ [0, 1],
+ [2, 3],
+ [2, 3],
+ [0, 2],
+ [2, 3],
+ ],
+ dtype=np.uint64,
+ )
+ distances = np.asarray(
+ [
+ [1.0, 9.0],
+ [2.0, 8.0],
+ [1.0, 9.0],
+ [2.0, 8.0],
+ [1.0, 1.0],
+ [1.0, 9.0],
+ ],
+ dtype=np.float64,
+ )
+ uninformative = np.asarray(
+ [False, False, False, False, True, False],
+ dtype=bool,
+ )
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name="plot_diagnostics",
+ indices=indices,
+ distances=distances,
+ uninformative=uninformative,
+ )
+ embedding = query.project_reference_embedding(
+ result,
+ reference_layout_key="mapping_layout",
+ )
+ query_coordinates = np.asarray(
+ artifact_group(query.zw, embedding)["values"][:],
+ dtype=np.float64,
+ )
+ return {
+ "reference": reference,
+ "reference_layout": reference_layout,
+ "reference_labels": reference_labels,
+ "query": query,
+ "result": result,
+ "embedding": embedding,
+ "query_coordinates": query_coordinates,
+ "query_groups": np.asarray(["q1", "q1", "q2", "q2", "q1", "q2"]),
+ "known_labels": np.asarray(["A", "A", "B", "B", "A", "B"]),
+ }
+
+
+def _sorted_rows(values: np.ndarray) -> np.ndarray:
+ return values[np.lexsort((values[:, 1], values[:, 0]))]
+
+
+def test_mapping_plot_families_use_query_result_and_reference_semantics(
+ plotting_mapping_context,
+):
+ context = plotting_mapping_context
+ query = context["query"]
+ result = context["result"]
+ reference = context["reference"]
+ query_groups = context["query_groups"]
+ known_labels = context["known_labels"]
+
+ score = splt.mapping_score(
+ query,
+ result,
+ reference=reference,
+ target_groups=query_groups,
+ layout_key="mapping_layout",
+ show=False,
+ )
+ evidence = splt.mapping_evidence(
+ query,
+ result,
+ reference=reference,
+ reference_class_group="mapping_label",
+ target_groups=query_groups,
+ metrics=("voteFraction", "topTwoMargin"),
+ show=False,
+ )
+ evidence_embedding = splt.mapping_evidence(
+ query,
+ result,
+ reference=reference,
+ reference_class_group="mapping_label",
+ metrics=("voteFraction", "referenceDistancePercentile"),
+ kind="embedding",
+ reference_layout_key="mapping_layout",
+ show=False,
+ )
+ confusion = splt.mapping_confusion(
+ query,
+ result,
+ reference=reference,
+ reference_class_group="mapping_label",
+ known_labels=known_labels,
+ show=False,
+ )
+ calibration = splt.mapping_calibration(
+ query,
+ result,
+ reference=reference,
+ reference_class_group="mapping_label",
+ known_labels=known_labels,
+ n_thresholds=3,
+ chosen_threshold=0.75,
+ show=False,
+ )
+ projection = splt.mapping_projection(
+ query,
+ result,
+ reference=reference,
+ reference_layout_key="mapping_layout",
+ reference_class_group="mapping_label",
+ show=False,
+ )
+
+ assert set(score.axes) == {"q1", "q2"}
+ n_reference = reference.selected_cell_count
+ assert len(score.tables["scores"]) == 2 * n_reference
+ score_coordinates = np.asarray(
+ next(iter(score.axes.values())).collections[0].get_offsets(),
+ dtype=np.float64,
+ )
+ np.testing.assert_allclose(
+ _sorted_rows(score_coordinates),
+ _sorted_rows(context["reference_layout"]),
+ )
+ assert score.provenance.extras["mapping_name"] == result.mapping_name
+
+ assert set(evidence.axes) == {"voteFraction", "topTwoMargin"}
+ assert len(evidence.tables["evidence"]) == len(query_groups)
+ assert set(evidence_embedding.axes) == {
+ "voteFraction",
+ "referenceDistancePercentile",
+ }
+ assert confusion.tables["counts"].set_index("known").loc["A", "A"] >= 2
+ assert {"precision", "recall", "support"} <= set(confusion.tables["perClass"])
+ assert {"coverage", "accuracy", "accuracyLower", "accuracyUpper"} <= set(
+ calibration.tables["calibration"]
+ )
+ assert calibration.tables["calibration"]["coverage"].between(0, 1).all()
+
+ projected_cells = projection.tables["cells"]
+ reference_rows = projected_cells.loc[projected_cells["source"] == "reference"]
+ query_rows = projected_cells.loc[projected_cells["source"] == "query"]
+ np.testing.assert_allclose(
+ reference_rows[["x", "y"]].to_numpy(dtype=np.float64),
+ context["reference_layout"],
+ )
+ np.testing.assert_allclose(
+ query_rows[["x", "y"]].to_numpy(dtype=np.float64),
+ context["query_coordinates"],
+ equal_nan=True,
+ )
+ assert "query_only" not in set(reference_rows["group"])
+ assert set(context["reference_labels"]) <= set(reference_rows["group"])
+ assert projection.provenance.extras["n_reference"] == n_reference
+ assert projection.provenance.extras["mapping_name"] == result.mapping_name
+
+ for plot in (
+ score,
+ evidence,
+ evidence_embedding,
+ confusion,
+ calibration,
+ projection,
+ ):
+ plot.close()
+
+
+def test_mapping_plots_resolve_string_result_with_query_assay(
+ plotting_mapping_context,
+):
+ context = plotting_mapping_context
+ plot = splt.mapping_score(
+ context["query"],
+ "plot_diagnostics",
+ reference=context["reference"],
+ query_assay="RNA",
+ kind="histogram",
+ show=False,
+ )
+
+ assert plot.provenance.extras["mapping_name"] == "plot_diagnostics"
+ plot.close()
+
+
+def test_mapping_evidence_box_kind_draws_one_box_per_query_group(
+ plotting_mapping_context,
+):
+ context = plotting_mapping_context
+ evidence = splt.mapping_evidence(
+ context["query"],
+ context["result"],
+ reference=context["reference"],
+ reference_class_group="mapping_label",
+ target_groups=context["query_groups"],
+ metrics=("voteFraction",),
+ kind="box",
+ show=False,
+ )
+
+ axis = evidence.axes["voteFraction"]
+ assert [text.get_text() for text in axis.get_xticklabels()] == ["q1", "q2"]
+ evidence.close()
+
+
+def test_mapping_score_uses_mapping_name_as_default_title(
+ plotting_mapping_context,
+):
+ context = plotting_mapping_context
+ plot = splt.mapping_score(
+ context["query"],
+ context["result"],
+ reference=context["reference"],
+ layout_key="mapping_layout",
+ show=False,
+ )
+
+ assert plot.tables["scores"]["group"].unique().tolist() == [0]
+ assert next(iter(plot.axes.values())).get_title() == "plot_diagnostics"
+ plot.close()
+
+
+def test_mapping_calibration_allows_genuine_zero_accuracy(
+ plotting_mapping_context,
+):
+ context = plotting_mapping_context
+ plot = splt.mapping_calibration(
+ context["query"],
+ context["result"],
+ reference=context["reference"],
+ reference_class_group="mapping_label",
+ known_labels=np.asarray(["B", "B", "A", "A", "B", "A"]),
+ n_thresholds=3,
+ show=False,
+ )
+
+ assert (plot.tables["calibration"]["accuracy"] == 0).all()
+ plot.close()
+
+
+def test_mapping_calibration_rejects_pairwise_label_type_mismatch(
+ plotting_mapping_context,
+):
+ context = plotting_mapping_context
+ reference = context["reference"]
+ numeric_labels = np.zeros(reference.selected_cell_count, dtype=np.int64)
+ numeric_labels[:4] = [1, 1, 2, 2]
+ reference.datastore.cells.insert(
+ "mapping_numeric_label",
+ numeric_labels,
+ key=reference.cell_key,
+ overwrite=True,
+ )
+
+ with pytest.raises(ValueError, match="after text conversion"):
+ splt.mapping_calibration(
+ context["query"],
+ context["result"],
+ reference=reference,
+ reference_class_group="mapping_numeric_label",
+ known_labels=np.asarray(["1", "1", "2", "2", "1", "2"]),
+ n_thresholds=3,
+ show=False,
+ )
+
+
+def test_mapping_calibration_warns_when_threshold_retains_nothing(
+ plotting_mapping_context,
+):
+ context = plotting_mapping_context
+ with pytest.warns(RuntimeWarning, match="retained no mapped cells"):
+ plot = splt.mapping_calibration(
+ context["query"],
+ context["result"],
+ reference=context["reference"],
+ reference_class_group="mapping_label",
+ known_labels=context["known_labels"],
+ chosen_threshold=2.0,
+ n_thresholds=3,
+ show=False,
+ )
+
+ assert not any(
+ "voteFraction =" in text.get_text()
+ for text in plot.axes["mapping_calibration"].texts
+ )
+ plot.close()
+
+
+def test_mapping_diagnostic_plots_accept_caller_owned_targets(
+ plotting_mapping_context,
+):
+ import matplotlib.pyplot as pyplot
+
+ context = plotting_mapping_context
+ figure, axes = pyplot.subplots(1, 2)
+ plot = splt.mapping_evidence(
+ context["query"],
+ context["result"],
+ reference=context["reference"],
+ reference_class_group="mapping_label",
+ metrics=("voteFraction", "topTwoMargin"),
+ target={
+ "voteFraction": axes[0],
+ "topTwoMargin": axes[1],
+ },
+ show=False,
+ )
+
+ assert plot.figure is figure
+ assert not plot.owns_figure
+ pyplot.close(figure)
+
+
+@pytest.mark.parametrize("plot_name", ("mapping_evidence", "mapping_projection"))
+def test_mapping_embedding_plots_require_persisted_reference_embedding(
+ analyzed_datastore_ephemeral,
+ tmp_path: Path,
+ plot_name: str,
+):
+ reference = _plain_reference(analyzed_datastore_ephemeral)
+ query = _copied_query(
+ analyzed_datastore_ephemeral,
+ tmp_path / f"missing_{plot_name}.zarr",
+ )
+ _write_reference_layout(
+ reference,
+ layout_key="missing_layout",
+ linked=False,
+ )
+ reference.datastore.cells.insert(
+ "missing_labels",
+ np.full(reference.selected_cell_count, "A", dtype=object),
+ key=reference.cell_key,
+ overwrite=True,
+ )
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name="missing_embedding",
+ indices=np.asarray([[0, 1], [1, 0]], dtype=np.uint64),
+ distances=np.ones((2, 2), dtype=np.float64),
+ uninformative=np.zeros(2, dtype=bool),
+ )
+ before = set(query.list_artifacts(kind="embedding", from_assay="RNA"))
+
+ kwargs = {
+ "reference": reference,
+ "reference_layout_key": "missing_layout",
+ "show": False,
+ }
+ if plot_name == "mapping_evidence":
+ kwargs.update(
+ {
+ "reference_class_group": "missing_labels",
+ "kind": "embedding",
+ "metrics": ("voteFraction",),
+ }
+ )
+ with pytest.raises(ValueError, match="project_reference_embedding"):
+ getattr(splt, plot_name)(query, result, **kwargs)
+
+ assert set(query.list_artifacts(kind="embedding", from_assay="RNA")) == before
+
+
+def test_mapping_plots_do_not_project_or_weight_coordinates(
+ plotting_mapping_context,
+ monkeypatch: pytest.MonkeyPatch,
+):
+ context = plotting_mapping_context
+ query = context["query"]
+ result = context["result"]
+ reference = context["reference"]
+ score_rows = list(
+ query.get_mapping_score(
+ result,
+ target_groups=context["query_groups"],
+ )
+ )
+ evidence = query.get_target_label_evidence(
+ result,
+ reference_class_group="mapping_label",
+ )
+
+ def cached_scores(*args, **kwargs):
+ yield from score_rows
+
+ def cached_evidence(*args, **kwargs):
+ return evidence.copy()
+
+ def unexpected(*args, **kwargs):
+ raise AssertionError("plotting attempted mapping computation")
+
+ monkeypatch.setattr(type(query), "get_mapping_score", cached_scores)
+ monkeypatch.setattr(type(query), "get_target_label_evidence", cached_evidence)
+ monkeypatch.setattr(type(query), "project_reference_embedding", unexpected)
+ monkeypatch.setattr(mapping_confidence, "distance_weights", unexpected)
+
+ plots = (
+ splt.mapping_score(
+ query,
+ result,
+ reference=reference,
+ target_groups=context["query_groups"],
+ layout_key="mapping_layout",
+ show=False,
+ ),
+ splt.mapping_evidence(
+ query,
+ result,
+ reference=reference,
+ reference_class_group="mapping_label",
+ kind="embedding",
+ metrics=("voteFraction",),
+ reference_layout_key="mapping_layout",
+ show=False,
+ ),
+ splt.mapping_confusion(
+ query,
+ result,
+ reference=reference,
+ reference_class_group="mapping_label",
+ known_labels=context["known_labels"],
+ show=False,
+ ),
+ splt.mapping_calibration(
+ query,
+ result,
+ reference=reference,
+ reference_class_group="mapping_label",
+ known_labels=context["known_labels"],
+ n_thresholds=3,
+ show=False,
+ ),
+ splt.mapping_projection(
+ query,
+ result,
+ reference=reference,
+ reference_layout_key="mapping_layout",
+ target_groups=context["query_groups"],
+ show=False,
+ ),
+ )
+
+ for plot in plots:
+ plot.close()
+
+
+def test_mapping_projection_background_mode_keeps_query_on_top(
+ plotting_mapping_context,
+):
+ context = plotting_mapping_context
+ plot = splt.mapping_projection(
+ context["query"],
+ context["result"],
+ reference=context["reference"],
+ reference_layout_key="mapping_layout",
+ target_groups=context["query_groups"],
+ reference_mode="background",
+ show=False,
+ )
+ ax = plot.axes["mapping_projection"]
+ assert len(ax.collections) == 2
+ query_offsets = np.asarray(ax.collections[1].get_offsets(), dtype=np.float64)
+ expected_visible = int(np.isfinite(context["query_coordinates"]).all(axis=1).sum())
+ assert len(query_offsets) == expected_visible
+ with pytest.raises(ValueError, match="reference_mode"):
+ splt.mapping_projection(
+ context["query"],
+ context["result"],
+ reference=context["reference"],
+ reference_layout_key="mapping_layout",
+ reference_mode="overlay",
+ show=False,
+ )
+ plot.close()
+
+
+def test_uninformative_query_rows_are_not_drawn_as_mapping_points(
+ plotting_mapping_context,
+):
+ context = plotting_mapping_context
+ expected_visible = int(np.isfinite(context["query_coordinates"]).all(axis=1).sum())
+ projection = splt.mapping_projection(
+ context["query"],
+ context["result"],
+ reference=context["reference"],
+ reference_layout_key="mapping_layout",
+ target_groups=context["query_groups"],
+ show=False,
+ )
+ evidence = splt.mapping_evidence(
+ context["query"],
+ context["result"],
+ reference=context["reference"],
+ reference_class_group="mapping_label",
+ metrics=("voteFraction",),
+ kind="embedding",
+ reference_layout_key="mapping_layout",
+ show_unknown=False,
+ show=False,
+ )
+
+ projection_offsets = (
+ projection.axes["mapping_projection"].collections[1].get_offsets()
+ )
+ evidence_offsets = evidence.axes["voteFraction"].collections[1].get_offsets()
+ assert len(projection_offsets) == expected_visible
+ assert len(evidence_offsets) == expected_visible
+ assert expected_visible == context["result"].n_cells - 1
+ query_rows = projection.tables["cells"].loc[
+ projection.tables["cells"]["source"] == "query"
+ ]
+ assert query_rows[["x", "y"]].isna().all(axis=1).sum() == 1
+ assert evidence.tables["evidence"].loc[4, "isUnknown"]
+ assert np.isnan(evidence.tables["evidence"].loc[4, "voteFraction"])
+ projection.close()
+ evidence.close()
+
+
+def test_reference_embedding_adapter_chooses_newest_exact_artifact(
+ plotting_mapping_context,
+):
+ context = plotting_mapping_context
+ query = context["query"]
+ first = context["embedding"]
+ second = query.project_reference_embedding(
+ context["result"],
+ reference=context["reference"],
+ reference_layout_key="mapping_layout",
+ invalidate_cache=True,
+ )
+ first_group = artifact_group(query.zw, first)
+ second_group = artifact_group(query.zw, second)
+ first_group.attrs["created_at_ns"] = 10
+ second_group.attrs["created_at_ns"] = 20
+ expected = np.asarray(second_group["values"][:], dtype=np.float64)
+ expected[np.isfinite(expected)] += 100.0
+ second_group["values"][:] = expected
+
+ loaded = query._load_mapping_reference_embedding(
+ context["result"],
+ "mapping_layout",
+ reference=context["reference"],
+ )
+
+ np.testing.assert_allclose(loaded, expected, equal_nan=True)
+
+
+def test_reference_embedding_adapter_matches_current_layout_fingerprint(
+ analyzed_datastore_ephemeral,
+ tmp_path: Path,
+):
+ reference = _plain_reference(analyzed_datastore_ephemeral)
+ query = _copied_query(
+ analyzed_datastore_ephemeral,
+ tmp_path / "layout_fingerprint_query.zarr",
+ )
+ layout, _ = _write_reference_layout(
+ reference,
+ layout_key="mutable_layout",
+ linked=False,
+ )
+ result = _write_projection(
+ query,
+ reference,
+ mapping_name="layout_fingerprint",
+ indices=np.asarray([[0, 1], [1, 0]], dtype=np.uint64),
+ distances=np.ones((2, 2), dtype=np.float64),
+ uninformative=np.zeros(2, dtype=bool),
+ )
+ query.project_reference_embedding(
+ result,
+ reference_layout_key="mutable_layout",
+ )
+ changed = layout[:, 0] + 1.0
+ reference.datastore.cells.insert(
+ "mutable_layout1",
+ changed,
+ key=reference.cell_key,
+ overwrite=True,
+ )
+
+ with pytest.raises(ValueError, match="project_reference_embedding"):
+ query._load_mapping_reference_embedding(
+ result,
+ "mutable_layout",
+ reference=reference,
+ )
+
+
+def test_reference_embedding_adapter_rejects_infinite_coordinates(
+ plotting_mapping_context,
+):
+ context = plotting_mapping_context
+ group = artifact_group(context["query"].zw, context["embedding"])
+ values = np.asarray(group["values"][:], dtype=np.float64)
+ values[0, 0] = np.inf
+ group["values"][:] = values
+
+ with pytest.raises(ValueError, match="infinite coordinates"):
+ context["query"]._load_mapping_reference_embedding(
+ context["result"],
+ "mapping_layout",
+ reference=context["reference"],
+ )
diff --git a/tests/test_plotting_modernization.py b/tests/test_plotting_modernization.py
new file mode 100644
index 00000000..6a364603
--- /dev/null
+++ b/tests/test_plotting_modernization.py
@@ -0,0 +1,694 @@
+"""Focused behavior tests for publication plotting features."""
+
+import matplotlib
+import numpy as np
+import pytest
+
+matplotlib.use("Agg")
+
+import matplotlib.pyplot as plt
+
+import scarf.plotting as splt
+from scarf.plotting._style import default_point_size, resolve_legend_loc
+
+
+def test_point_size_uses_population_and_panel_area():
+ assert default_point_size(500, panel_area=16) > default_point_size(
+ 5_000,
+ panel_area=16,
+ )
+ assert default_point_size(5_000, panel_area=16) > default_point_size(
+ 5_000,
+ panel_area=4,
+ )
+ assert resolve_legend_loc(80) == "right"
+
+
+def test_register_theme_extends_rcparams():
+ name = "test-publication-theme"
+ splt.register_theme(name, {"font.size": 7.5}, base="paper")
+ with pytest.raises(ValueError, match="already exists"):
+ splt.register_theme(name, {"font.size": 8})
+ with splt.theme_context(name):
+ assert matplotlib.rcParams["font.size"] == pytest.approx(7.5)
+
+
+def test_stored_display_metadata_does_not_hide_malformed_stores():
+ from scarf.plotting._display import stored_display_metadata
+
+ class MissingPlotStore:
+ pass
+
+ class MalformedPlotStore:
+ zw = {}
+
+ assert stored_display_metadata(MissingPlotStore(), "group") is None
+ with pytest.raises(KeyError, match="cellData"):
+ stored_display_metadata(MalformedPlotStore(), "group")
+
+
+def test_embedding_density_highlight_and_labeled_colorbar(umap, datastore):
+ n = len(datastore.cells.active_index("I"))
+ highlighted = np.zeros(n, dtype=bool)
+ highlighted[: max(3, n // 10)] = True
+ datastore.cells.insert("plot_highlight", highlighted, overwrite=True)
+
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_nCounts",
+ density_overlay=splt.DensityOverlay(pixels=32, levels=3, sigma=1),
+ highlight=splt.Highlight(by="plot_highlight"),
+ point_size_range=(2, 20),
+ show_titles=False,
+ show=False,
+ )
+
+ ax = next(iter(result.axes.values()))
+ assert len(ax.collections) >= 3
+ assert result.provenance.extras["highlight"]["n_highlighted"] == highlighted.sum()
+ assert all(
+ 2 <= value <= 20
+ for value in result.provenance.extras["point_size_by_panel"].values()
+ )
+ auxiliary_labels = [
+ axis.get_xlabel() for axis in result.figure.axes if axis is not ax
+ ]
+ assert "RNA_nCounts" in auxiliary_labels
+ assert ax.get_title() == ""
+ result.close()
+
+
+def test_embedding_mean_contours_require_and_use_continuous_values(
+ umap,
+ datastore,
+):
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_nCounts",
+ density_overlay=splt.DensityOverlay(
+ statistic="mean",
+ pixels=32,
+ sigma=1.5,
+ min_support=0.05,
+ levels=(0.6, 0.85),
+ max_hotspots=1,
+ ),
+ show=False,
+ )
+
+ ax = next(iter(result.axes.values()))
+ assert len(ax.collections) > 1
+ assert result.provenance.extras["density_overlay"]["statistic"] == "mean"
+ assert result.provenance.extras["density_overlay"]["max_hotspots"] == 1
+ result.close()
+
+ with pytest.raises(ValueError, match="continuous color_by"):
+ splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by=splt.CellField("I", kind="categorical"),
+ density_overlay=splt.DensityOverlay(statistic="mean"),
+ show=False,
+ )
+ with pytest.raises(ValueError, match="positive integer"):
+ splt.DensityOverlay(max_hotspots=0)
+
+
+def test_contour_hotspot_limit_keeps_the_strongest_region():
+ from scipy.ndimage import label
+
+ from scarf.plotting.embedding import _retain_strongest_hotspots
+
+ surface = np.zeros((12, 12), dtype=np.float64)
+ support = np.ones_like(surface)
+ surface[1:4, 1:4] = 2.0
+ surface[7:11, 7:11] = 3.0
+ surface[8:10, 8:10] = 0.0
+ support[7:11, 7:11] = 2.0
+
+ filtered = _retain_strongest_hotspots(
+ surface,
+ support,
+ level=1.0,
+ max_hotspots=1,
+ )
+ _, n_hotspots = label(filtered >= 1.0)
+
+ assert n_hotspots == 1
+ assert np.all(filtered[1:4, 1:4] < 1.0)
+ assert np.all(filtered[7:11, 7:11] >= 1.0)
+
+
+def test_embedding_point_size_uses_final_panel_area(umap, datastore):
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_nCounts",
+ point_size_range=(2, 20),
+ show=False,
+ )
+
+ result.figure.canvas.draw()
+ ax = next(iter(result.axes.values()))
+ bbox = ax.get_position()
+ width, height = result.figure.get_size_inches()
+ panel_area = float(bbox.width * width * bbox.height * height)
+ expected = default_point_size(
+ len(datastore.cells.active_index("I")),
+ panel_area=panel_area,
+ size_min=2,
+ size_max=20,
+ )
+ observed = next(iter(result.provenance.extras["point_size_by_panel"].values()))
+ assert observed == pytest.approx(expected)
+ result.close()
+
+
+def test_dotplot_feature_brackets_and_axis_swap(umap, leiden_clustering, datastore):
+ genes = [str(value) for value in datastore.RNA.feats.fetch_all("names")[:4]]
+ features = {"Lineage": genes[:2], "State": genes[2:]}
+ result = splt.dotplot(
+ datastore,
+ features=features,
+ group_by="RNA_leiden_cluster",
+ swap_axes=True,
+ marker_linewidth=0.6,
+ show=False,
+ )
+
+ ax = result.axes["dotplot"]
+ assert result.provenance.extras["feature_group_brackets"] == 2
+ assert sum(line.get_gid() == "feature-group-bracket" for line in ax.lines) == 2
+ assert ax.get_xlabel() == ""
+ result.close()
+
+
+def test_dotplot_marker_sizes_follow_physical_grid_cells(
+ umap,
+ leiden_clustering,
+ datastore,
+):
+ genes = [str(value) for value in datastore.RNA.feats.fetch_all("names")[:4]]
+ features = {"Lineage": genes[:2], "State": genes[2:]}
+ compact_figure, compact_ax = plt.subplots(figsize=(3, 2))
+ large_figure, large_ax = plt.subplots(figsize=(8, 6))
+
+ compact = splt.dotplot(
+ datastore,
+ features=features,
+ group_by="RNA_leiden_cluster",
+ target=compact_ax,
+ show_legend=False,
+ show=False,
+ )
+ large = splt.dotplot(
+ datastore,
+ features=features,
+ group_by="RNA_leiden_cluster",
+ target=large_ax,
+ show_legend=False,
+ show=False,
+ )
+
+ compact_sizes = compact_ax.collections[0].get_sizes()
+ large_sizes = large_ax.collections[0].get_sizes()
+ assert large_sizes.max() > compact_sizes.max()
+ assert compact.provenance.extras["size_scale_source"] == "panel"
+
+ compact_figure.canvas.draw()
+ renderer = compact_figure.canvas.get_renderer()
+ label_left = min(
+ label.get_window_extent(renderer).x0
+ for label in compact_ax.get_yticklabels()
+ if label.get_text()
+ )
+ bracket_right = max(
+ line.get_transform().transform(line.get_xydata())[:, 0].max()
+ for line in compact_ax.lines
+ if line.get_gid() == "feature-group-bracket"
+ )
+ assert bracket_right < label_left
+
+ compact.close()
+ large.close()
+ plt.close(compact_figure)
+ plt.close(large_figure)
+
+
+def test_dotplot_left_group_labels_clear_feature_tick_labels(
+ umap,
+ leiden_clustering,
+ datastore,
+):
+ genes = [str(value) for value in datastore.RNA.feats.fetch_all("names")[:4]]
+ result = splt.dotplot(
+ datastore,
+ features={"Myeloid lineage": genes[:2], "Cell state": genes[2:]},
+ group_by="RNA_leiden_cluster",
+ show_legend=False,
+ show=False,
+ )
+
+ figure = result.figure
+ figure.canvas.draw()
+ renderer = figure.canvas.get_renderer()
+ feature_label_left = min(
+ label.get_window_extent(renderer).x0
+ for label in result.axes["dotplot"].get_yticklabels()
+ if label.get_text()
+ )
+ group_label_right = max(
+ label.get_window_extent(renderer).x1
+ for label in result.axes["dotplot"].texts
+ if label.get_gid() == "feature-group-label"
+ )
+
+ assert feature_label_left - group_label_right >= figure.dpi * 6.0 / 72.0
+ result.close()
+
+
+def test_stacked_violin_standardizes_rows(umap, leiden_clustering, datastore):
+ genes = [str(value) for value in datastore.RNA.feats.fetch_all("names")[:2]]
+ result = splt.distribution(
+ datastore,
+ keys=genes,
+ group_by="RNA_leiden_cluster",
+ kind="stacked_violin",
+ row_standardize=True,
+ max_points=0,
+ show=False,
+ )
+
+ assert len(result.axes) == 2
+ for table in result.tables.values():
+ assert np.nanmean(table["display_value"]) == pytest.approx(0, abs=1e-7)
+ assert result.provenance.extras["row_standardize"] is True
+ result.close()
+
+
+def test_stacked_violin_can_share_value_scale(umap, leiden_clustering, datastore):
+ genes = [str(value) for value in datastore.RNA.feats.fetch_all("names")[:2]]
+ result = splt.distribution(
+ datastore,
+ keys=genes,
+ group_by="RNA_leiden_cluster",
+ kind="stacked_violin",
+ share_y=True,
+ max_points=0,
+ show=False,
+ )
+
+ limits = [axis.get_ylim() for axis in result.axes.values()]
+ assert all(limit == pytest.approx(limits[0]) for limit in limits[1:])
+ assert result.provenance.extras["share_y"] is True
+ result.close()
+
+
+def test_distribution_aggregates_biological_samples(
+ umap,
+ leiden_clustering,
+ datastore,
+):
+ n = len(datastore.cells.active_index("I"))
+ samples = np.asarray([f"sample_{index % 8}" for index in range(n)])
+ datastore.cells.insert("plot_distribution_sample", samples, overwrite=True)
+
+ result = splt.distribution(
+ datastore,
+ keys="RNA_nCounts",
+ group_by="RNA_leiden_cluster",
+ sample_by="plot_distribution_sample",
+ sample_stat="median",
+ kind="box",
+ max_points=0,
+ show=False,
+ )
+
+ table = result.tables["RNA_nCounts"]
+ assert {"sample", "group", "value", "display_value", "nCells"} <= set(table)
+ assert not table.duplicated(["sample", "group"]).any()
+ assert result.provenance.n_samples == 8
+ assert result.provenance.extras["sample_stat"] == "median"
+ result.close()
+
+
+def test_distribution_draws_sample_aware_split_violins(
+ umap,
+ leiden_clustering,
+ datastore,
+):
+ n = len(datastore.cells.active_index("I"))
+ sample_index = np.arange(n) % 8
+ samples = np.asarray([f"sample_{index}" for index in sample_index])
+ conditions = np.where(sample_index % 2 == 0, "control", "stimulated")
+ datastore.cells.insert("plot_split_sample", samples, overwrite=True)
+ datastore.cells.insert("plot_split_condition", conditions, overwrite=True)
+
+ result = splt.distribution(
+ datastore,
+ keys="RNA_nCounts",
+ group_by="RNA_leiden_cluster",
+ split_by="plot_split_condition",
+ study_design=splt.StudyDesign(sample_by="plot_split_sample"),
+ kind="violin",
+ max_points=30,
+ show=False,
+ )
+
+ table = result.tables["RNA_nCounts"]
+ assert set(table["split"]) == {"control", "stimulated"}
+ scale = result.scales[0]
+ assert isinstance(scale, splt.CategoricalScale)
+ assert scale.order == ("control", "stimulated")
+ assert result.legends[0].label == "plot_split_condition"
+ result.close()
+
+
+def test_distribution_seed_repeats_point_jitter(
+ umap,
+ leiden_clustering,
+ datastore,
+):
+ kwargs = {
+ "keys": "RNA_nCounts",
+ "group_by": "RNA_leiden_cluster",
+ "kind": "box",
+ "max_points": 80,
+ "seed": 17,
+ "show": False,
+ }
+ first = splt.distribution(datastore, **kwargs)
+ second = splt.distribution(datastore, **kwargs)
+
+ first_offsets = [
+ np.asarray(collection.get_offsets())
+ for collection in next(iter(first.axes.values())).collections
+ ]
+ second_offsets = [
+ np.asarray(collection.get_offsets())
+ for collection in next(iter(second.axes.values())).collections
+ ]
+ assert len(first_offsets) == len(second_offsets)
+ for observed, repeated in zip(first_offsets, second_offsets, strict=True):
+ np.testing.assert_array_equal(observed, repeated)
+ first.close()
+ second.close()
+
+
+def test_sample_aggregated_feature_axis_retains_requested_italics(
+ umap,
+ datastore,
+):
+ n = len(datastore.cells.active_index("I"))
+ samples = np.asarray([f"sample_{index % 8}" for index in range(n)])
+ datastore.cells.insert("plot_italic_sample", samples, overwrite=True)
+ gene = str(datastore.RNA.feats.fetch_all("names")[0])
+
+ result = splt.distribution(
+ datastore,
+ keys=gene,
+ sample_by="plot_italic_sample",
+ kind="box",
+ italicize_features=True,
+ show=False,
+ )
+
+ axis = next(iter(result.axes.values()))
+ assert axis.get_ylabel() == f"Sample mean {gene}"
+ assert axis.yaxis.label.get_fontstyle() == "italic"
+ result.close()
+
+
+def test_per_sample_composition_reports_uncertainty(
+ umap,
+ leiden_clustering,
+ datastore,
+):
+ n = len(datastore.cells.active_index("I"))
+ sample_index = np.arange(n) % 8
+ samples = np.asarray([f"sample_{index}" for index in sample_index])
+ conditions = np.where(sample_index < 4, "control", "stimulated")
+ subjects = np.asarray([f"subject_{index % 4}" for index in sample_index])
+ datastore.cells.insert("plot_ci_sample", samples, overwrite=True)
+ datastore.cells.insert("plot_ci_condition", conditions, overwrite=True)
+ datastore.cells.insert("plot_ci_subject", subjects, overwrite=True)
+
+ result = splt.composition(
+ datastore,
+ category_by="RNA_leiden_cluster",
+ study_design=splt.StudyDesign(
+ sample_by="plot_ci_sample",
+ condition_by="plot_ci_condition",
+ subject_by="plot_ci_subject",
+ ),
+ kind="per_sample",
+ uncertainty="ci95",
+ show=False,
+ )
+
+ summary = result.tables["summary"]
+ assert {"mean_proportion", "lower", "upper", "n_samples"} <= set(summary)
+ assert np.all(summary["lower"] <= summary["mean_proportion"])
+ assert np.all(summary["mean_proportion"] <= summary["upper"])
+ assert result.provenance.extras["uncertainty"] == "ci95"
+ assert result.provenance.extras["n_pair_lines"] > 0
+ assert any(legend.kind == "marker" for legend in result.legends)
+ axis = next(iter(result.axes.values()))
+ assert axis.get_ylim()[1] >= float(summary["upper"].max())
+ for legend in result.figure.legends:
+ if legend.get_title().get_text() == "Condition":
+ assert all(
+ not text.get_text().startswith("mean") for text in legend.get_texts()
+ )
+ rendered_labels = [
+ text.get_text()
+ for legend in result.figure.legends
+ for text in legend.get_texts()
+ ]
+ assert any(
+ label.startswith("mean") or label.startswith("Summary: mean")
+ for label in rendered_labels
+ )
+ result.close()
+
+
+def test_embedding_side_legend_stays_page_sized_for_many_categories(
+ umap,
+ datastore,
+):
+ n = len(datastore.cells.active_index("I"))
+ labels = np.full(n, "type_149", dtype=object)
+ labels[:149] = [f"type_{index:03d}" for index in range(149)]
+ datastore.cells.insert("plot_many_types", labels, overwrite=True)
+
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="plot_many_types",
+ legend_loc="right",
+ show=False,
+ )
+
+ width, _ = result.figure.get_size_inches()
+ assert width <= 12
+ legend = result.figure.legends[0]
+ assert len(legend.get_texts()) == 80
+ assert legend.get_title().get_text() == "plot_many_types (80 of 150)"
+ assert "type_149" in [text.get_text() for text in legend.get_texts()]
+ omitted = result.provenance.extras["omitted_legend_entries"]
+ panel_key = str(next(iter(result.axes)))
+ assert len(omitted[panel_key]) == 70
+ result.close()
+
+
+def test_cluster_connectivity_runs_on_real_datastore_graph(
+ umap,
+ leiden_clustering,
+ datastore,
+):
+ result = datastore.plots.cluster_connectivity(
+ group_by="RNA_leiden_cluster",
+ layout_key="RNA_UMAP",
+ feat_key="hvgs",
+ minimum_edge_weight=0,
+ max_edges_per_node=3,
+ show_cells=True,
+ show=False,
+ )
+
+ observed = len(np.unique(datastore.cells.fetch("RNA_leiden_cluster", key="I")))
+ assert len(result.tables["nodes"]) == observed
+ assert set(result.tables["edges"]) == {
+ "source",
+ "target",
+ "rawWeight",
+ "normalizedWeight",
+ }
+ assert result.provenance.extras["n_edges"] <= 3 * observed // 2
+ background = result.axes["cluster_connectivity"].collections[0]
+ assert background.get_alpha() == pytest.approx(0.3)
+ assert background.get_sizes()[0] >= 4
+ assert not hasattr(background, "_scarf_layout_point_size")
+ assert len(np.unique(background.get_facecolors(), axis=0)) > 1
+ assert result.provenance.extras["cell_size_source"] == "panel"
+ result.close()
+
+
+def test_composition_borders_labels_and_stored_palette(
+ umap,
+ leiden_clustering,
+ datastore,
+):
+ n = len(datastore.cells.active_index("I"))
+ samples = np.asarray([f"s{index % 3}" for index in range(n)], dtype=object)
+ datastore.cells.insert("plot_modern_samples", samples, overwrite=True)
+
+ embedding = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ show_legend=False,
+ show=False,
+ )
+ composition = splt.composition(
+ datastore,
+ category_by="RNA_leiden_cluster",
+ sample_by="plot_modern_samples",
+ segment_linewidth=0.8,
+ show_percent_labels=True,
+ label_min_fraction=0.01,
+ show=False,
+ )
+
+ bars = composition.axes["composition"].patches
+ assert bars
+ assert all(bar.get_linewidth() == pytest.approx(0.8) for bar in bars)
+ assert composition.axes["composition"].texts
+ embedding_scale = next(
+ scale for scale in embedding.scales if isinstance(scale, splt.CategoricalScale)
+ )
+ composition_scale = next(
+ scale
+ for scale in composition.scales
+ if isinstance(scale, splt.CategoricalScale)
+ )
+ assert embedding_scale.palette == composition_scale.palette
+ embedding.close()
+ composition.close()
+
+
+def test_compose_results_namespaces_tables_and_renders_shared_legend(
+ umap,
+ leiden_clustering,
+ datastore,
+):
+ n = len(datastore.cells.active_index("I"))
+ phases = np.asarray(["G1", "S", "G2M"] * (n // 3 + 1), dtype=object)[:n]
+ datastore.cells.insert("plot_composite_phase", phases, overwrite=True)
+ figure, axes = plt.subplot_mosaic(
+ [["embedding", "composition"]],
+ figsize=(7, 3),
+ layout="constrained",
+ )
+ first = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ target=axes["embedding"],
+ show_legend=True,
+ theme="paper",
+ show=False,
+ )
+ second = splt.composition(
+ datastore,
+ category_by="plot_composite_phase",
+ target=axes["composition"],
+ show_legend=True,
+ theme="paper",
+ show=False,
+ )
+
+ result = splt.compose_results(
+ figure,
+ {"embedding": first, "composition": second},
+ theme="paper",
+ )
+
+ assert result.owns_figure is False
+ assert result.provenance.notes == ("composite",)
+ assert "composition:aggregate" in result.tables
+ assert len(figure.legends) == 2
+ assert all(axis.get_legend() is None for axis in axes.values())
+ figure.canvas.draw()
+ renderer = figure.canvas.get_renderer()
+ legend_boxes = [legend.get_window_extent(renderer) for legend in figure.legends]
+ assert not legend_boxes[0].overlaps(legend_boxes[1])
+ panel_labels = [
+ text
+ for axis in axes.values()
+ for text in axis.texts
+ if text.get_text() in {"A", "B"}
+ ]
+ assert {text.get_fontsize() for text in panel_labels} == {
+ splt.THEMES["paper"]["axes.titlesize"]
+ }
+ assert {
+ text.get_fontsize() for legend in figure.legends for text in legend.get_texts()
+ } == {splt.THEMES["paper"]["legend.fontsize"]}
+ result.close()
+ assert plt.fignum_exists(figure.number)
+ plt.close(figure)
+
+
+def test_recipe_execution_is_headless_and_preserves_analysis_state(umap, datastore):
+ before = datastore.get_assay_state("RNA")
+ recipe = splt.PlotRecipe(
+ (
+ splt.PlotStep(
+ name="overview",
+ plot="embedding",
+ kwargs={
+ "layout_key": "RNA_UMAP",
+ "color_by": "RNA_nCounts",
+ },
+ ),
+ )
+ )
+
+ execution = datastore.plots.run_recipe(recipe)
+
+ assert not execution.written_paths
+ assert not execution.failures
+ assert len(execution.results) == 1
+ assert datastore.get_assay_state("RNA") == before
+ result = execution.results[0]
+ assert plt.fignum_exists(result.figure.number)
+ result.close()
+
+
+def test_recipe_batch_output_closes_owned_figure(umap, datastore, tmp_path):
+ recipe = splt.PlotRecipe(
+ (
+ splt.PlotStep(
+ name="overview",
+ plot="embedding",
+ kwargs={
+ "layout_key": "RNA_UMAP",
+ "color_by": "RNA_nCounts",
+ },
+ output=splt.PlotOutputSettings(
+ filename="overview.png",
+ dpi=90,
+ ),
+ ),
+ )
+ )
+
+ execution = splt.run_recipe(datastore, recipe, output_dir=tmp_path)
+ plot_result = execution.results[0]
+
+ assert execution.written_paths == (tmp_path / "overview.png",)
+ assert execution.written_paths[0].stat().st_size > 0
+ assert not plt.fignum_exists(plot_result.figure.number)
diff --git a/tests/test_plotting_raster.py b/tests/test_plotting_raster.py
new file mode 100644
index 00000000..15dc222b
--- /dev/null
+++ b/tests/test_plotting_raster.py
@@ -0,0 +1,366 @@
+"""Guarded-source and parity tests for Milestone C raster path."""
+
+import numpy as np
+import pytest
+
+import scarf.plotting as splt
+
+
+class _GuardedZarrArray:
+ """Minimal array stand-in that forbids full-column ``[:]`` reads."""
+
+ def __init__(self, data: np.ndarray, chunk: int = 8):
+ self._data = np.asarray(data)
+ self.chunks = (chunk,)
+ self.dtype = self._data.dtype
+ self.shape = self._data.shape
+
+ def __getitem__(self, key):
+ if key == slice(None) or key == ():
+ raise AssertionError(
+ "full-column read is forbidden in guarded raster tests"
+ )
+ if isinstance(key, slice):
+ if key.start is None and key.stop is None and key.step is None:
+ raise AssertionError("full-column read is forbidden")
+ return self._data[key]
+ return self._data[key]
+
+
+class _GuardedMeta:
+ """Duck-typed MetaData using guarded column arrays."""
+
+ def __init__(self, columns: dict[str, np.ndarray], chunk: int = 8):
+ self.N = len(next(iter(columns.values())))
+ self.columns = list(columns)
+ self._arrays = {
+ k: _GuardedZarrArray(v, chunk=chunk) for k, v in columns.items()
+ }
+ self.index = np.arange(self.N)
+
+ def _verify_bool(self, key: str) -> bool:
+ if self._arrays[key].dtype != bool:
+ raise TypeError("key must be bool")
+ return True
+
+ def default_block_rows(self, column: str = "I") -> int:
+ return int(self._arrays[column].chunks[0])
+
+ def active_index(self, key: str) -> np.ndarray:
+ return np.flatnonzero(self._arrays[key]._data)
+
+ def fetch(self, column: str, key: str = "I") -> np.ndarray:
+ idx = self.active_index(key)
+ return self._arrays[column]._data[idx]
+
+ def iter_row_blocks(self, *, cell_key="I", columns=None, block_rows=None):
+ from scarf.metadata import MetaDataRowBlock
+
+ self._verify_bool(cell_key)
+ if block_rows is None:
+ block_rows = self.default_block_rows(cell_key)
+ col_list = list(columns or [])
+ key_arr = self._arrays[cell_key]
+ for start in range(0, self.N, block_rows):
+ stop = min(start + block_rows, self.N)
+ key_slice = np.asarray(key_arr[start:stop], dtype=bool)
+ local = np.flatnonzero(key_slice)
+ active_global = (local + start).astype(np.int64, copy=False)
+ values = {
+ col: np.asarray(self._arrays[col][start:stop])[local]
+ for col in col_list
+ }
+ yield MetaDataRowBlock(
+ start=start,
+ stop=stop,
+ active_global_indices=active_global,
+ values=values,
+ )
+
+
+def test_raster_from_metadata_rejects_full_slice_and_matches_mean():
+ from scarf.plotting._raster import raster_from_metadata
+
+ rng = np.random.default_rng(0)
+ n = 40
+ x = rng.normal(size=n)
+ y = rng.normal(size=n)
+ c = rng.normal(size=n)
+ active = np.ones(n, dtype=bool)
+ active[::5] = False
+ cells = _GuardedMeta(
+ {"I": active, "UMAP1": x, "UMAP2": y, "score": c},
+ chunk=7,
+ )
+ canvas = raster_from_metadata(
+ cells,
+ x_key="UMAP1",
+ y_key="UMAP2",
+ color_key="score",
+ cell_key="I",
+ pixels=32,
+ block_rows=7,
+ quantiles=None,
+ seed=0,
+ )
+ assert canvas.n_cells == int(active.sum())
+ assert canvas.n_blocks >= 2
+ assert np.isfinite(canvas.vmin) and np.isfinite(canvas.vmax)
+ # Pixel means only where counts > 0
+ assert np.isnan(canvas.image[canvas.counts == 0]).all()
+ assert np.isfinite(canvas.image[canvas.counts > 0]).all()
+
+
+def test_embedding_raster_on_datastore(umap, leiden_clustering, datastore):
+ result = splt.embedding_raster(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_nCounts",
+ pixels=64,
+ block_rows=32,
+ show=False,
+ )
+ assert result.provenance.renderer == "matplotlib-raster"
+ assert "two_pass" in result.provenance.notes
+ assert result.provenance.n_cells > 0
+ result.close()
+
+
+def test_embedding_raster_rejects_categorical_metadata(
+ umap, leiden_clustering, datastore
+):
+ with pytest.raises(NotImplementedError, match="continuous color"):
+ splt.embedding_raster(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ pixels=32,
+ show=False,
+ )
+
+
+def test_embedding_raster_density_and_foreign_target(umap, datastore):
+ import matplotlib.pyplot as plt
+
+ fig, ax = plt.subplots()
+ result = splt.embedding_raster(
+ datastore,
+ layout_key="RNA_UMAP",
+ pixels=32,
+ target=ax,
+ show=False,
+ )
+ assert result.owns_figure is False
+ assert result.provenance.extras["color_mode"] == "density"
+ assert result.legends[0].label == "log1p cell count"
+ result.close()
+ assert plt.fignum_exists(fig.number)
+ plt.close(fig)
+
+
+def test_embedding_raster_image_fills_square_axes(umap, datastore):
+ result = splt.embedding_raster(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_nCounts",
+ pixels=48,
+ show=False,
+ )
+ ax = next(iter(result.axes.values()))
+ images = ax.get_images()
+ assert len(images) == 1
+ extent = images[0].get_extent()
+ xlim = ax.get_xlim()
+ ylim = ax.get_ylim()
+ assert extent[0] == pytest.approx(xlim[0])
+ assert extent[1] == pytest.approx(xlim[1])
+ assert extent[2] == pytest.approx(ylim[0])
+ assert extent[3] == pytest.approx(ylim[1])
+ assert (xlim[1] - xlim[0]) == pytest.approx(ylim[1] - ylim[0])
+ result.close()
+
+
+def test_raster_validates_quantiles():
+ from scarf.plotting._raster import raster_from_metadata
+
+ cells = _GuardedMeta(
+ {
+ "I": np.ones(4, dtype=bool),
+ "x": np.arange(4, dtype=float),
+ "y": np.arange(4, dtype=float),
+ "v": np.arange(4, dtype=float),
+ }
+ )
+ with pytest.raises(ValueError, match="quantiles"):
+ raster_from_metadata(
+ cells,
+ x_key="x",
+ y_key="y",
+ color_key="v",
+ quantiles=(0.9, 0.1),
+ )
+ with pytest.raises(ValueError, match="sample_capacity"):
+ raster_from_metadata(
+ cells,
+ x_key="x",
+ y_key="y",
+ color_key="v",
+ sample_capacity=0,
+ )
+
+
+def test_raster_without_color_encodes_log_density():
+ from scarf.plotting._raster import raster_from_metadata
+
+ cells = _GuardedMeta(
+ {
+ "I": np.ones(4, dtype=bool),
+ "x": np.zeros(4),
+ "y": np.zeros(4),
+ }
+ )
+ canvas = raster_from_metadata(
+ cells,
+ x_key="x",
+ y_key="y",
+ pixels=8,
+ )
+ assert canvas.counts.sum() == 4
+ assert np.nanmax(canvas.image) == pytest.approx(np.log1p(4))
+ assert canvas.vmax == pytest.approx(np.log1p(4))
+
+
+def test_density_canvas_from_points_uses_raster_contract():
+ from scarf.plotting._raster import density_canvas_from_points
+
+ canvas = density_canvas_from_points(
+ np.array([0.1, 0.1, 0.8]),
+ np.array([0.1, 0.1, 0.8]),
+ extent=(0, 1, 0, 1),
+ pixels=10,
+ )
+
+ assert canvas.counts.sum() == 3
+ assert canvas.counts.max() == 2
+ assert canvas.n_cells == 3
+ assert canvas.extent == (0, 1, 0, 1)
+ low_y_row = int(np.flatnonzero(canvas.counts[:, 1])[0])
+ high_y_row = int(np.flatnonzero(canvas.counts[:, 8])[0])
+ assert high_y_row < low_y_row
+
+
+def test_raster_is_block_size_invariant():
+ from scarf.plotting._raster import raster_from_metadata
+
+ rng = np.random.default_rng(4)
+ cells = _GuardedMeta(
+ {
+ "I": np.ones(60, dtype=bool),
+ "x": rng.normal(size=60),
+ "y": rng.normal(size=60),
+ "value": rng.normal(size=60),
+ }
+ )
+ kwargs = {
+ "x_key": "x",
+ "y_key": "y",
+ "color_key": "value",
+ "pixels": 24,
+ "sample_capacity": 15,
+ "seed": 9,
+ }
+ first = raster_from_metadata(cells, block_rows=7, **kwargs)
+ second = raster_from_metadata(cells, block_rows=13, **kwargs)
+ assert first.extent == pytest.approx(second.extent)
+ assert first.vmin == pytest.approx(second.vmin)
+ assert first.vmax == pytest.approx(second.vmax)
+ np.testing.assert_allclose(first.image, second.image, equal_nan=True)
+ np.testing.assert_array_equal(first.counts, second.counts)
+
+
+def test_raster_all_missing_color_has_finite_default_limits():
+ from scarf.plotting._raster import raster_from_metadata
+
+ cells = _GuardedMeta(
+ {
+ "I": np.ones(4, dtype=bool),
+ "x": np.arange(4, dtype=float),
+ "y": np.arange(4, dtype=float),
+ "value": np.full(4, np.nan),
+ }
+ )
+ canvas = raster_from_metadata(
+ cells,
+ x_key="x",
+ y_key="y",
+ color_key="value",
+ pixels=8,
+ )
+ assert (canvas.vmin, canvas.vmax) == (0.0, 1.0)
+ assert np.isnan(canvas.image).all()
+
+
+def test_raster_missing_pixels_default_white():
+ from scarf.plotting._raster import RasterCanvas, draw_raster_canvas
+
+ matplotlib = pytest.importorskip("matplotlib")
+ matplotlib.use("Agg")
+ import matplotlib.pyplot as plt
+
+ canvas = RasterCanvas(
+ image=np.full((8, 8), np.nan, dtype=np.float64),
+ counts=np.zeros((8, 8), dtype=np.int64),
+ extent=(0.0, 1.0, 0.0, 1.0),
+ vmin=0.0,
+ vmax=1.0,
+ n_cells=0,
+ n_blocks=0,
+ )
+ fig, ax = plt.subplots()
+ im = draw_raster_canvas(ax, canvas)
+ bad = np.asarray(im.cmap.get_bad())
+ # RGBA for the bad/missing color should be opaque white.
+ np.testing.assert_allclose(bad[:3], [1.0, 1.0, 1.0], atol=1e-5)
+ face = np.asarray(matplotlib.colors.to_rgba(ax.get_facecolor()))
+ np.testing.assert_allclose(face[:3], [1.0, 1.0, 1.0], atol=1e-5)
+ plt.close(fig)
+
+
+def test_raster_subset_by_reduces_cells():
+ from scarf.plotting._raster import raster_from_metadata
+
+ n = 20
+ keep = np.zeros(n, dtype=bool)
+ keep[:5] = True
+ cells = _GuardedMeta(
+ {
+ "I": np.ones(n, dtype=bool),
+ "x": np.linspace(0, 1, n),
+ "y": np.linspace(0, 1, n),
+ "value": np.arange(n, dtype=float),
+ "keep": keep,
+ }
+ )
+ full = raster_from_metadata(
+ cells, x_key="x", y_key="y", color_key="value", pixels=16
+ )
+ subset = raster_from_metadata(
+ cells,
+ x_key="x",
+ y_key="y",
+ color_key="value",
+ subset_by="keep",
+ pixels=16,
+ )
+ assert subset.n_cells == 5
+ assert subset.n_cells < full.n_cells
+
+
+def test_specialized_plotting_exports_are_callable():
+ assert callable(splt.qc)
+ assert callable(splt.elbow)
+ assert callable(splt.graph_qc)
+ assert callable(splt.highly_variable_features)
+ assert callable(splt.marker_heatmap)
+ assert callable(splt.cluster_tree)
+ assert callable(splt.pseudotime_heatmap)
diff --git a/tests/test_plotting_recipes.py b/tests/test_plotting_recipes.py
new file mode 100644
index 00000000..875c8a26
--- /dev/null
+++ b/tests/test_plotting_recipes.py
@@ -0,0 +1,544 @@
+import json
+import warnings
+from dataclasses import FrozenInstanceError
+from pathlib import Path
+
+import pytest
+
+import scarf.plotting as plotting
+from scarf.plotting.recipes import (
+ PlotOutputSettings,
+ PlotPanelTarget,
+ PlotRecipe,
+ PlotStep,
+ run_plot_recipe,
+)
+
+
+class FakePlotResult:
+ def __init__(self, name: str, *, owns_figure: bool = True) -> None:
+ self.name = name
+ self.owns_figure = owns_figure
+ self.saved: list[Path] = []
+ self.show_calls = 0
+ self.close_calls = 0
+ self.usable = True
+
+ def save(self, path: str | Path, **kwargs: object) -> Path:
+ output = Path(path)
+ output.parent.mkdir(parents=True, exist_ok=True)
+ output.write_text(self.name, encoding="utf-8")
+ self.saved.append(output)
+ return output
+
+ def show(self) -> None:
+ self.show_calls += 1
+
+ def close(self) -> None:
+ self.close_calls += 1
+ self.usable = False
+
+
+def test_runner_preserves_order_suppresses_plot_show_and_manages_outputs(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ store = object()
+ calls: list[tuple[str, object, dict[str, object]]] = []
+ plot_results: dict[str, FakePlotResult] = {}
+
+ def fake_plot(name: str):
+ def plot(store_arg: object, **kwargs: object) -> FakePlotResult:
+ calls.append((name, store_arg, kwargs))
+ result = FakePlotResult(name)
+ plot_results[name] = result
+ return result
+
+ return plot
+
+ monkeypatch.setattr(plotting, "embedding", fake_plot("embedding"))
+ monkeypatch.setattr(plotting, "dotplot", fake_plot("dotplot"))
+ recipe = PlotRecipe(
+ [
+ PlotStep(
+ name="overview",
+ plot="embedding",
+ kwargs={"layout_key": "umap", "show": True},
+ output_filename="figures/overview.png",
+ ),
+ PlotStep(
+ name="markers",
+ plot="dotplot",
+ kwargs={"features": ["A"], "group_by": "cluster"},
+ output_filename="figures/markers.svg",
+ ),
+ ]
+ )
+
+ result = run_plot_recipe(store, recipe, output_dir=tmp_path, show=True)
+
+ assert [name for name, _, _ in calls] == ["embedding", "dotplot"]
+ assert all(store_arg is store for _, store_arg, _ in calls)
+ assert all(kwargs["show"] is False for _, _, kwargs in calls)
+ assert calls[0][2]["layout_key"] == "umap"
+ expected_paths = (
+ tmp_path / "figures/overview.png",
+ tmp_path / "figures/markers.svg",
+ )
+ assert result.written_paths == expected_paths
+ assert [output.name for output in result.outputs] == ["overview", "markers"]
+ assert result.results == (
+ plot_results["embedding"],
+ plot_results["dotplot"],
+ )
+ assert all(path.exists() for path in result.written_paths)
+ for name, expected_path in zip(("embedding", "dotplot"), expected_paths):
+ plot_result = plot_results[name]
+ assert plot_result.saved == [expected_path]
+ assert plot_result.show_calls == 1
+ assert plot_result.close_calls == 1
+ assert result.failures == {}
+
+
+def test_no_output_mode_leaves_returned_owned_figure_usable(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ plot_result = FakePlotResult("embedding")
+
+ def embedding(store: object, **kwargs: object) -> FakePlotResult:
+ return plot_result
+
+ monkeypatch.setattr(plotting, "embedding", embedding)
+ recipe = PlotRecipe(
+ [PlotStep(name="overview", plot="embedding", kwargs={"layout_key": "umap"})]
+ )
+
+ result = run_plot_recipe(object(), recipe)
+
+ assert result.results == (plot_result,)
+ assert result.written_paths == ()
+ assert plot_result.saved == []
+ assert plot_result.show_calls == 0
+ assert plot_result.close_calls == 0
+ assert plot_result.usable
+
+
+def test_runner_does_not_close_caller_owned_results(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ plot_result = FakePlotResult("embedding", owns_figure=False)
+ monkeypatch.setattr(
+ plotting,
+ "embedding",
+ lambda store, **kwargs: plot_result,
+ )
+ recipe = PlotRecipe(
+ [
+ PlotStep(
+ name="overview",
+ plot="embedding",
+ output_filename="overview.png",
+ )
+ ]
+ )
+
+ run_plot_recipe(object(), recipe, output_dir=tmp_path, show=True)
+
+ assert plot_result.saved == [tmp_path / "overview.png"]
+ assert plot_result.show_calls == 1
+ assert plot_result.close_calls == 0
+ assert plot_result.usable
+
+
+def test_continue_on_error_captures_compact_failures_and_warnings(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ calls: list[str] = []
+ successful = FakePlotResult("dotplot")
+
+ def failing(store: object, **kwargs: object) -> FakePlotResult:
+ calls.append("bad")
+ warnings.warn("first warning", UserWarning)
+ raise RuntimeError("plot\nfailed")
+
+ def succeeding(store: object, **kwargs: object) -> FakePlotResult:
+ calls.append("good")
+ warnings.warn("second warning", RuntimeWarning)
+ return successful
+
+ monkeypatch.setattr(plotting, "embedding", failing)
+ monkeypatch.setattr(plotting, "dotplot", succeeding)
+ recipe = PlotRecipe(
+ [
+ PlotStep(name="bad", plot="embedding"),
+ PlotStep(name="good", plot="dotplot"),
+ ]
+ )
+
+ with pytest.raises(RuntimeError, match="plot"):
+ run_plot_recipe(object(), recipe)
+ assert calls == ["bad"]
+
+ calls.clear()
+ result = run_plot_recipe(object(), recipe, continue_on_error=True)
+
+ assert calls == ["bad", "good"]
+ assert result.failures == {"bad": "RuntimeError: plot failed"}
+ assert result.results == (successful,)
+ assert result.warnings == (
+ "bad: first warning",
+ "good: second warning",
+ )
+ assert successful.usable
+
+
+def test_json_and_toml_loaders_have_matching_camel_case_schema(
+ tmp_path: Path,
+) -> None:
+ config = {
+ "steps": [
+ {
+ "name": "overview",
+ "plot": "embedding",
+ "kwargs": {
+ "layoutKey": "umap",
+ "colorBy": "cluster",
+ },
+ "outputFilename": "figures/overview.png",
+ },
+ {
+ "name": "composition",
+ "plot": "composition",
+ "kwargs": {"categoryBy": "cluster"},
+ },
+ ]
+ }
+ json_path = tmp_path / "recipe.json"
+ json_path.write_text(json.dumps(config), encoding="utf-8")
+ toml_path = tmp_path / "recipe.toml"
+ toml_path.write_text(
+ """
+[[steps]]
+name = "overview"
+plot = "embedding"
+outputFilename = "figures/overview.png"
+
+[steps.kwargs]
+layoutKey = "umap"
+colorBy = "cluster"
+
+[[steps]]
+name = "composition"
+plot = "composition"
+
+[steps.kwargs]
+categoryBy = "cluster"
+""".strip(),
+ encoding="utf-8",
+ )
+
+ from_dict = PlotRecipe.from_dict(config)
+ from_json = PlotRecipe.from_json(json_path)
+ from_toml = PlotRecipe.from_toml(toml_path)
+
+ assert from_dict == from_json == from_toml
+ assert from_json.steps[0].output_filename == "figures/overview.png"
+ assert dict(from_toml.steps[1].kwargs) == {"category_by": "cluster"}
+
+
+def test_runner_resolves_json_config_path(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ seen: list[dict[str, object]] = []
+ plot_result = FakePlotResult("embedding")
+
+ def embedding(store: object, **kwargs: object) -> FakePlotResult:
+ seen.append(kwargs)
+ return plot_result
+
+ monkeypatch.setattr(plotting, "embedding", embedding)
+ path = tmp_path / "recipe.json"
+ path.write_text(
+ json.dumps(
+ {
+ "steps": [
+ {
+ "name": "overview",
+ "plot": "embedding",
+ "kwargs": {"layoutKey": "umap"},
+ }
+ ]
+ }
+ ),
+ encoding="utf-8",
+ )
+
+ result = run_plot_recipe(object(), path)
+
+ assert seen == [{"layout_key": "umap", "show": False}]
+ assert result.results == (plot_result,)
+
+
+@pytest.mark.parametrize(
+ "filename",
+ [
+ "/tmp/plot.png",
+ "../plot.png",
+ "nested/../../plot.png",
+ r"C:\plots\plot.png",
+ ],
+)
+def test_output_filename_rejects_absolute_and_traversal_paths(filename: str) -> None:
+ with pytest.raises(ValueError, match="relative|traversal"):
+ PlotStep(
+ name="overview",
+ plot="embedding",
+ output_filename=filename,
+ )
+
+
+def test_output_filename_requires_supported_format_and_output_directory() -> None:
+ with pytest.raises(ValueError, match="Unsupported output format"):
+ PlotStep(
+ name="overview",
+ plot="embedding",
+ output_filename="overview.txt",
+ )
+
+ recipe = PlotRecipe(
+ [
+ PlotStep(
+ name="overview",
+ plot="embedding",
+ output_filename="overview.png",
+ )
+ ]
+ )
+ with pytest.raises(ValueError, match="output_dir"):
+ run_plot_recipe(object(), recipe)
+
+
+@pytest.mark.parametrize(
+ ("config", "message"),
+ [
+ ({"steps": [], "extra": True}, "Unknown plot recipe"),
+ (
+ {
+ "steps": [
+ {
+ "name": "overview",
+ "plot": "embedding",
+ "output_filename": "overview.png",
+ }
+ ]
+ },
+ "Unknown plot step",
+ ),
+ (
+ {"steps": [{"name": "overview", "plot": "embedding", "kwargs": []}]},
+ "mapping",
+ ),
+ ({"steps": [{"name": "overview", "plot": "unknown"}]}, "Unknown plot"),
+ ],
+)
+def test_from_dict_rejects_unknown_keys_and_invalid_values(
+ config: dict[str, object],
+ message: str,
+) -> None:
+ with pytest.raises((TypeError, ValueError), match=message):
+ PlotRecipe.from_dict(config)
+
+
+def test_recipe_validates_non_empty_unique_names_and_is_frozen() -> None:
+ with pytest.raises(ValueError, match="at least one"):
+ PlotRecipe([])
+ with pytest.raises(ValueError, match="non-empty"):
+ PlotStep(name=" ", plot="embedding")
+
+ step = PlotStep(name="overview", plot="embedding")
+ with pytest.raises(ValueError, match="unique"):
+ PlotRecipe([step, step])
+ with pytest.raises(FrozenInstanceError):
+ step.name = "changed"
+ assert not hasattr(step, "__dict__")
+
+
+def test_external_kwargs_require_camel_case_and_reject_unknown_fields() -> None:
+ with pytest.raises(ValueError, match="camelCase"):
+ PlotRecipe.from_dict(
+ {
+ "steps": [
+ {
+ "name": "overview",
+ "plot": "embedding",
+ "kwargs": {"layout_key": "umap"},
+ }
+ ]
+ }
+ )
+ with pytest.raises(ValueError, match="Unknown kwargs"):
+ PlotRecipe.from_dict(
+ {
+ "steps": [
+ {
+ "name": "overview",
+ "plot": "embedding",
+ "kwargs": {
+ "layoutKey": "umap",
+ "notAPlotArgument": True,
+ },
+ }
+ ]
+ }
+ )
+
+
+def test_external_recipe_coerces_plot_contracts_and_feature_refs() -> None:
+ recipe = PlotRecipe.from_dict(
+ {
+ "steps": [
+ {
+ "name": "markers",
+ "plot": "dotplot",
+ "kwargs": {
+ "features": [
+ {
+ "value": "CD3D",
+ "assay": "RNA",
+ "by": "name",
+ }
+ ],
+ "groupBy": "cluster",
+ "colorScale": {
+ "cmap": "magma",
+ "quantiles": [0.01, 0.99],
+ "missingColor": "#cccccc",
+ },
+ "sizeScale": {
+ "sizeMin": 5,
+ "sizeMax": 80,
+ },
+ },
+ }
+ ]
+ }
+ )
+
+ kwargs = recipe.steps[0].kwargs
+ assert kwargs["features"] == [
+ plotting.FeatureRef(value="CD3D", assay="RNA", by="name")
+ ]
+ assert kwargs["color_scale"] == plotting.ColorScale(
+ cmap="magma",
+ quantiles=(0.01, 0.99),
+ missing_color="#cccccc",
+ )
+ assert kwargs["size_scale"] == plotting.SizeScale(size_min=5, size_max=80)
+
+
+def test_runner_injects_pipeline_artifacts_without_mutating_recipe_kwargs(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ seen: list[dict[str, object]] = []
+ plot_result = FakePlotResult("embedding")
+
+ def embedding(store: object, **kwargs: object) -> FakePlotResult:
+ seen.append(kwargs)
+ return plot_result
+
+ monkeypatch.setattr(plotting, "embedding", embedding)
+ recipe = PlotRecipe(
+ [
+ PlotStep(
+ name="overview",
+ plot="embedding",
+ kwargs={"layout_key": "umap"},
+ artifact_kwargs={"color_by": "clusterColumn"},
+ )
+ ]
+ )
+
+ result = run_plot_recipe(
+ object(),
+ recipe,
+ artifacts={"clusterColumn": "RNA_leiden_cluster"},
+ )
+
+ assert result.results == (plot_result,)
+ assert seen == [
+ {
+ "layout_key": "umap",
+ "color_by": "RNA_leiden_cluster",
+ "show": False,
+ }
+ ]
+ assert dict(recipe.steps[0].kwargs) == {"layout_key": "umap"}
+ with pytest.raises(KeyError, match="clusterColumn"):
+ run_plot_recipe(object(), recipe, artifacts={})
+
+
+def test_recipe_resolves_optional_panel_target_and_output_settings(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ seen: list[dict[str, object]] = []
+ result = FakePlotResult("embedding", owns_figure=False)
+
+ def embedding(store: object, **kwargs: object) -> FakePlotResult:
+ seen.append(kwargs)
+ return result
+
+ monkeypatch.setattr(plotting, "embedding", embedding)
+ recipe = PlotRecipe.from_dict(
+ {
+ "steps": [
+ {
+ "name": "overview",
+ "plot": "embedding",
+ "kwargs": {"layoutKey": "umap"},
+ "target": {"panel": "A"},
+ "output": {
+ "filename": "overview.png",
+ "dpi": 180,
+ "transparent": True,
+ "exactSize": False,
+ },
+ }
+ ]
+ }
+ )
+
+ assert recipe.steps[0].target == PlotPanelTarget("A")
+ assert recipe.steps[0].output == PlotOutputSettings(
+ "overview.png",
+ dpi=180,
+ transparent=True,
+ exact_size=False,
+ )
+ run_plot_recipe(
+ object(),
+ recipe,
+ targets={"A": "axes-a"},
+ output_dir=tmp_path,
+ )
+
+ assert seen == [
+ {
+ "layout_key": "umap",
+ "target": "axes-a",
+ "show": False,
+ }
+ ]
+ with pytest.raises(KeyError, match="requires panel"):
+ run_plot_recipe(object(), recipe, targets={}, output_dir=tmp_path)
+
+
+def test_json_loader_rejects_duplicate_keys_and_non_object_roots() -> None:
+ with pytest.raises(ValueError, match="Duplicate JSON key"):
+ PlotRecipe.from_json(
+ '{"steps": [{"name": "one", "name": "two", "plot": "embedding"}]}'
+ )
+ with pytest.raises(TypeError, match="mapping"):
+ PlotRecipe.from_json("[]")
diff --git a/tests/test_plotting_showcase.py b/tests/test_plotting_showcase.py
new file mode 100644
index 00000000..183a612b
--- /dev/null
+++ b/tests/test_plotting_showcase.py
@@ -0,0 +1,128 @@
+"""Showcase generator smoke tests."""
+
+import importlib.util
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+
+_EXPECTED_OUTPUTS = {
+ "categorical_embedding.png",
+ "cell_cycle_scores.png",
+ "cluster_connectivity.png",
+ "composition.png",
+ "continuous_embedding.png",
+ "dark_embedding.png",
+ "grouped_dotplot.png",
+ "highlighted_embedding.png",
+ "marker_heatmap.png",
+ "matrix_plot.png",
+ "publication_composite.png",
+ "publication_composite.svg",
+ "stacked_violin.png",
+}
+
+
+def _load_showcase_module():
+ script = (
+ Path(__file__).resolve().parents[1]
+ / "scripts"
+ / "generate_plotting_showcase.py"
+ )
+ spec = importlib.util.spec_from_file_location("scarf_plotting_showcase", script)
+ assert spec is not None and spec.loader is not None
+ module = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(module)
+ return module
+
+
+def _prepare_isolated_store(module, datastore_zarr_root, tmp_path):
+ work_directory = tmp_path / "showcase_store"
+ work_directory.mkdir()
+ return module._prepare_store(Path(datastore_zarr_root), work_directory)
+
+
+def test_showcase_generator_has_offline_fixture_cli():
+ script = (
+ Path(__file__).resolve().parents[1]
+ / "scripts"
+ / "generate_plotting_showcase.py"
+ )
+ completed = subprocess.run(
+ [sys.executable, str(script), "--help"],
+ cwd=script.parents[1],
+ capture_output=True,
+ text=True,
+ check=False,
+ )
+
+ assert completed.returncode == 0, completed.stderr
+ assert "--fixture" in completed.stdout
+ assert "--layout-fixture" in completed.stdout
+ assert "--output-dir" in completed.stdout
+
+
+@pytest.mark.slow
+def test_generate_requested_showcase_artifacts(
+ datastore_zarr_root,
+ tmp_path,
+):
+ requested = os.environ.get("SCARF_SHOWCASE_OUTPUT_DIR")
+ if requested is None:
+ pytest.skip("Set SCARF_SHOWCASE_OUTPUT_DIR to regenerate review figures")
+ module = _load_showcase_module()
+ store = _prepare_isolated_store(
+ module,
+ datastore_zarr_root,
+ tmp_path,
+ )
+ outputs = module.generate_showcase(store, Path(requested))
+
+ assert {path.name for path in outputs} == _EXPECTED_OUTPUTS
+ assert all(path.exists() and path.stat().st_size > 0 for path in outputs)
+
+
+@pytest.mark.slow
+@pytest.mark.visual
+def test_showcase_matches_visual_references(
+ datastore_zarr_root,
+ tmp_path,
+):
+ if os.environ.get("SCARF_RUN_VISUAL_REGRESSION") != "1":
+ pytest.skip("Set SCARF_RUN_VISUAL_REGRESSION=1 to compare showcase figures")
+ from matplotlib.testing.compare import compare_images
+
+ module = _load_showcase_module()
+ store = _prepare_isolated_store(
+ module,
+ datastore_zarr_root,
+ tmp_path,
+ )
+ output_directory = tmp_path / "showcase_outputs"
+ outputs = module.generate_showcase(store, output_directory)
+ assert {path.name for path in outputs} == _EXPECTED_OUTPUTS
+ reference_dir = Path(__file__).parent / "visual" / "showcase"
+ expected_pngs = {
+ reference_dir / name for name in _EXPECTED_OUTPUTS if name.endswith(".png")
+ }
+ assert set(reference_dir.glob("*.png")) == expected_pngs
+ tolerance = float(os.environ.get("SCARF_VISUAL_TOLERANCE", "1"))
+ failures = []
+ for actual in outputs:
+ if actual.suffix != ".png":
+ continue
+ expected = reference_dir / actual.name
+ if not expected.exists():
+ failures.append(f"missing reference: {expected}")
+ continue
+ comparison = compare_images(
+ str(expected),
+ str(actual),
+ tol=tolerance,
+ )
+ if comparison is not None:
+ failures.append(str(comparison))
+ assert not failures, "\n\n".join(failures)
diff --git a/tests/test_plotting_style.py b/tests/test_plotting_style.py
new file mode 100644
index 00000000..72061d9b
--- /dev/null
+++ b/tests/test_plotting_style.py
@@ -0,0 +1,153 @@
+"""Plot scale and theme behavior tests."""
+
+import matplotlib
+import pytest
+
+matplotlib.use("Agg")
+
+import scarf.plotting as splt
+from scarf.plotting._deps import require_matplotlib
+from scarf.plotting._style import (
+ categorical_color_map,
+ continuous_norm,
+ palette_for_n,
+ theme_context,
+)
+
+
+@pytest.mark.parametrize("size", [5, 15, 25, 50, 110])
+def test_palette_for_n_returns_requested_colors(size):
+ assert len(palette_for_n(size)) == size
+
+
+@pytest.mark.parametrize("palette_name", ["default", "colorblind"])
+@pytest.mark.parametrize("size", [8, 30, 110])
+def test_palette_for_n_never_recycles_colors(size, palette_name):
+ colors = palette_for_n(size, palette_name=palette_name)
+ assert len(set(colors)) == size
+
+
+def test_side_legend_columns_stay_page_sized():
+ from scarf.plotting._style import LEGEND_SIDE_MAX_COLUMNS, legend_side_columns
+
+ assert legend_side_columns(1) == 1
+ assert legend_side_columns(40) == 2
+ assert legend_side_columns(5_000) == LEGEND_SIDE_MAX_COLUMNS
+
+
+def test_categorical_color_map_validates_custom_palette():
+ with pytest.raises(KeyError, match="missing from palette"):
+ categorical_color_map(["a", "b"], palette={"a": "red"})
+ colors = categorical_color_map(
+ ["a"],
+ palette={"a": "red"},
+ missing_label="NA",
+ missing_color="gray",
+ )
+ assert colors == {"a": "red", "NA": "gray"}
+
+
+def test_continuous_norm_supports_center_and_validates_bounds():
+ _, mpl = require_matplotlib()
+ norm = continuous_norm(mpl, vmin=-2, vmax=3, vcenter=0)
+ assert norm.__class__.__name__ == "TwoSlopeNorm"
+ with pytest.raises(ValueError, match="vcenter"):
+ continuous_norm(mpl, vmin=0, vmax=3, vcenter=4)
+
+
+def test_square_axis_limits_and_dark_theme():
+ from scarf.plotting._style import (
+ apply_figure_chrome,
+ square_axis_limits,
+ scatter_edgecolor,
+ THEMES,
+ )
+
+ xlim, ylim = square_axis_limits((0.0, 2.0), (-1.0, 0.0))
+ assert xlim[1] - xlim[0] == pytest.approx(ylim[1] - ylim[0])
+ assert scatter_edgecolor("dark") == "#8f8f8f"
+ assert THEMES["notebook"]["figure.facecolor"] == "white"
+ assert THEMES["paper"]["savefig.transparent"] is False
+ assert THEMES["dark"]["text.color"] == "#e8e8e8"
+ with theme_context("dark"):
+ _, mpl = require_matplotlib()
+ assert mpl.rcParams["axes.edgecolor"] == "#e8e8e8"
+ plt, _ = require_matplotlib()
+ fig, ax = plt.subplots(1, 1)
+ apply_figure_chrome(fig, "notebook")
+ assert fig.patch.get_alpha() == 1.0
+ assert ax.patch.get_alpha() == 1.0
+ apply_figure_chrome(fig, "dark")
+ assert fig.patch.get_alpha() == 0
+ plt.close(fig)
+
+
+def test_density_and_legend_helpers():
+ from scarf.plotting._style import (
+ capped_figsize,
+ default_point_edgewidth,
+ default_point_size,
+ resolve_legend_loc,
+ sort_categories,
+ )
+
+ assert default_point_size(100) > default_point_size(20_000)
+ assert default_point_edgewidth(500) > 0
+ assert default_point_edgewidth(20_000) == 0.0
+ assert resolve_legend_loc(8) == "right"
+ assert resolve_legend_loc(20) == "on_data"
+ assert resolve_legend_loc(80) == "right"
+ assert resolve_legend_loc(20, "right") == "right"
+ assert capped_figsize(20.0, 4.0)[0] == pytest.approx(7.5)
+ assert sort_categories([1, 10, 2, "B", "A10", "A2"]) == [
+ 1,
+ 2,
+ 10,
+ "A2",
+ "A10",
+ "B",
+ ]
+ assert sort_categories(["10", "2", "1"]) == ["1", "2", "10"]
+
+
+def test_embedding_on_data_legend(umap, leiden_clustering, datastore):
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ legend_loc="on_data",
+ frame="none",
+ show=False,
+ )
+ ax = next(iter(result.axes.values()))
+ assert ax.get_xlabel() == ""
+ assert ax.get_ylabel() == ""
+ assert len(ax.texts) >= 1
+ assert not result.figure.legends
+ result.close()
+
+
+def test_embedding_panel_is_square(umap, leiden_clustering, datastore):
+ result = splt.embedding(
+ datastore,
+ layout_key="RNA_UMAP",
+ color_by="RNA_leiden_cluster",
+ show=False,
+ )
+ ax = next(iter(result.axes.values()))
+ result.figure.canvas.draw()
+ bbox = ax.get_window_extent()
+ assert ax.get_box_aspect() == pytest.approx(1.0)
+ assert bbox.width == pytest.approx(bbox.height, rel=1e-3)
+ xlim = ax.get_xlim()
+ ylim = ax.get_ylim()
+ assert (xlim[1] - xlim[0]) == pytest.approx(ylim[1] - ylim[0])
+ result.close()
+
+
+def test_theme_context_restores_matplotlib_state():
+ _, mpl = require_matplotlib()
+ original = mpl.rcParams["font.size"]
+ with theme_context("paper"):
+ assert mpl.rcParams["font.size"] == 8
+ assert mpl.rcParams["font.size"] == original
diff --git a/tests/test_prefetch.py b/tests/test_prefetch.py
new file mode 100644
index 00000000..c99b89eb
--- /dev/null
+++ b/tests/test_prefetch.py
@@ -0,0 +1,85 @@
+import threading
+import time
+
+import numpy as np
+import zarr
+
+from scarf.utils import iter_column_blocks
+
+
+def test_iter_column_blocks_preserves_order_and_bounds_reads():
+ lock = threading.Lock()
+ in_flight = 0
+ max_seen = 0
+
+ def read_block(block_idx: int) -> np.ndarray:
+ nonlocal in_flight, max_seen
+ with lock:
+ in_flight += 1
+ max_seen = max(max_seen, in_flight)
+ time.sleep(0.02)
+ with lock:
+ in_flight -= 1
+ return np.full((4, 2), block_idx, dtype=np.float64)
+
+ results = list(iter_column_blocks(8, read_block, workers=2))
+ seen = [int(arr[0, 0]) for _, arr, _, _ in results]
+ assert seen == list(range(8))
+ assert {source for _, _, _, source in results} == {"direct"}
+ assert max_seen <= 2
+
+
+def test_iter_column_blocks_respects_worker_limit():
+ lock = threading.Lock()
+ in_flight = 0
+ max_seen = 0
+
+ def read_block(block_idx: int) -> np.ndarray:
+ nonlocal in_flight, max_seen
+ with lock:
+ in_flight += 1
+ max_seen = max(max_seen, in_flight)
+ time.sleep(0.01)
+ with lock:
+ in_flight -= 1
+ return np.array([[block_idx]])
+
+ list(iter_column_blocks(4, read_block, workers=1))
+ assert max_seen == 1
+
+
+def test_iter_column_blocks_splits_workers_between_reads_and_object_io():
+ lock = threading.Lock()
+ in_flight = 0
+ max_seen = 0
+ io_concurrency = []
+
+ def read_block(block_idx: int) -> np.ndarray:
+ nonlocal in_flight, max_seen
+ with lock:
+ in_flight += 1
+ max_seen = max(max_seen, in_flight)
+ io_concurrency.append(zarr.config.get("async.concurrency"))
+ time.sleep(0.01)
+ with lock:
+ in_flight -= 1
+ return np.array([[block_idx]])
+
+ with zarr.config.set({"async.concurrency": 99}):
+ list(
+ iter_column_blocks(
+ 4,
+ read_block,
+ workers=2,
+ io_concurrency=4,
+ )
+ )
+ assert zarr.config.get("async.concurrency") == 99
+
+ assert max_seen > 1
+ assert io_concurrency and all(value == 4 for value in io_concurrency)
+ assert max_seen <= 2
+
+
+def test_iter_column_blocks_empty():
+ assert list(iter_column_blocks(0, lambda _: np.empty((0, 0)))) == []
diff --git a/tests/test_profiling_counts_t.py b/tests/test_profiling_counts_t.py
new file mode 100644
index 00000000..b11a72e1
--- /dev/null
+++ b/tests/test_profiling_counts_t.py
@@ -0,0 +1,115 @@
+import numpy as np
+import pytest
+import zarr
+
+from profiling import stages as profiling_stages
+from profiling.config import StageResources, WorkflowParameters
+from profiling.stages import run_stage
+from scarf.storage.sharding import write_counts_t
+
+
+def _resources() -> StageResources:
+ return StageResources(
+ modalMemoryRequestMb=4096,
+ modalMemoryLimitMb=4096,
+ modalCpuRequest=1.0,
+ modalCpuLimit=1.0,
+ scarfMemoryBudget=2 * 1024**3,
+ workers=1,
+ timeoutSeconds=600,
+ ephemeralDiskMb=524288,
+ )
+
+
+def test_write_counts_t_rewrites_incomplete_array(tmp_path):
+ root_path = tmp_path / "store.zarr"
+ root = zarr.open_group(str(root_path), mode="w")
+ group = root.create_group("RNA")
+ values = np.arange(24, dtype=np.uint32).reshape(6, 4)
+ counts = group.create_array("counts", shape=values.shape, dtype=values.dtype)
+ counts[:] = values
+ counts_t = write_counts_t(counts, group)
+ assert counts_t is not None
+ counts_t.attrs["complete"] = False
+ counts_t[0, 0] = 999
+
+ result = run_stage(
+ "writeCountsT",
+ nRows=6,
+ storeUri=str(root_path),
+ workflow=WorkflowParameters(),
+ resources=_resources(),
+ sampleIntervalSeconds=0.01,
+ )
+
+ assert result.status == "ok"
+ assert result.details is not None
+ assert result.details["complete"] is True
+ assert result.details["beforeComplete"] is False
+ reopened = zarr.open_group(str(root_path), mode="r")
+ fixed = reopened["RNA/countsT"]
+ assert fixed.attrs["complete"] is True
+ np.testing.assert_array_equal(fixed[:], values.T)
+
+
+def test_write_counts_t_runs_as_standard_profile_stage(tmp_path):
+ root_path = tmp_path / "store.zarr"
+ root = zarr.open_group(str(root_path), mode="w")
+ group = root.create_group("RNA")
+ values = np.arange(24, dtype=np.uint32).reshape(6, 4)
+ counts = group.create_array("counts", shape=values.shape, dtype=values.dtype)
+ counts[:] = values
+
+ result = run_stage(
+ "writeCountsT",
+ nRows=6,
+ storeUri=str(root_path),
+ workflow=WorkflowParameters(),
+ resources=_resources(),
+ sampleIntervalSeconds=0.01,
+ )
+
+ assert result.status == "ok"
+ assert result.inputSetupSeconds is not None
+ assert result.seconds is not None
+ assert result.validationPersistenceSeconds is not None
+ assert result.details is not None
+ assert result.details["complete"] is True
+ assert result.details["shape"] == [4, 6]
+ reopened = zarr.open_group(str(root_path), mode="r")
+ np.testing.assert_array_equal(reopened["RNA/countsT"][:], values.T)
+
+
+def test_write_counts_t_clears_complete_when_validation_fails(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path,
+):
+ root_path = tmp_path / "store.zarr"
+ root = zarr.open_group(str(root_path), mode="w")
+ group = root.create_group("RNA")
+ values = np.arange(24, dtype=np.uint32).reshape(6, 4)
+ counts = group.create_array("counts", shape=values.shape, dtype=values.dtype)
+ counts[:] = values
+ original_write = profiling_stages._write_counts_t
+
+ def corrupt_counts_t(*args, **kwargs):
+ counts_t = original_write(*args, **kwargs)
+ counts_t[0, 0] = 999
+ return counts_t
+
+ monkeypatch.setattr(profiling_stages, "_write_counts_t", corrupt_counts_t)
+
+ result = run_stage(
+ "writeCountsT",
+ nRows=6,
+ storeUri=str(root_path),
+ workflow=WorkflowParameters(),
+ resources=_resources(),
+ sampleIntervalSeconds=0.01,
+ )
+
+ assert result.status == "error"
+ assert result.error is not None
+ assert "tile mismatch" in result.error
+ reopened = zarr.open_group(str(root_path), mode="r")
+ assert reopened["RNA/countsT"].attrs["complete"] is False
diff --git a/tests/test_profiling_datasets_fixture.py b/tests/test_profiling_datasets_fixture.py
new file mode 100644
index 00000000..6ed90b48
--- /dev/null
+++ b/tests/test_profiling_datasets_fixture.py
@@ -0,0 +1,47 @@
+from pathlib import Path
+
+import h5py
+import numpy as np
+from scipy.sparse import csr_matrix
+
+from profiling.datasets import (
+ _load_string_column,
+ prepare_fixture_datasets,
+ write_fixture_h5ad,
+)
+
+
+def test_load_string_column_reads_categorical_feature_name(tmp_path: Path) -> None:
+ import anndata as ad
+ import pandas as pd
+
+ matrix = csr_matrix(np.array([[1.0, 0.0], [0.0, 2.0]], dtype=np.float32))
+ adata = ad.AnnData(matrix)
+ adata.obs_names = ["c0", "c1"]
+ adata.var_names = ["g0", "g1"]
+ adata.var["feature_name"] = pd.Categorical(["GeneA", "GeneB"])
+ path = tmp_path / "cat.h5ad"
+ adata.write_h5ad(path)
+
+ with h5py.File(path, "r") as h5:
+ assert isinstance(h5["var/feature_name"], h5py.Group)
+ names = _load_string_column(h5, "var/feature_name", expectedLength=2)
+ assert list(names) == ["GeneA", "GeneB"]
+
+
+def test_write_fixture_h5ad_is_readable_by_string_loader(tmp_path: Path) -> None:
+ path = tmp_path / "1000.h5ad"
+ artifact = write_fixture_h5ad(path, nRows=100, nColumns=50, seed=1)
+ assert artifact.targetRows == 100
+ assert path.is_file()
+ with h5py.File(path, "r") as h5:
+ names = _load_string_column(h5, "var/feature_name", expectedLength=50)
+ assert names.shape == (50,)
+ assert "MT-ND1" in set(names.tolist())
+
+
+def test_prepare_fixture_datasets_writes_requested_sizes(tmp_path: Path) -> None:
+ artifacts = prepare_fixture_datasets(tmp_path, targetRows=(10, 25), nColumns=20)
+ assert [item.targetRows for item in artifacts] == [10, 25]
+ assert (tmp_path / "10.h5ad").is_file()
+ assert (tmp_path / "25.h5ad").is_file()
diff --git a/tests/test_profiling_datasets_memory.py b/tests/test_profiling_datasets_memory.py
new file mode 100644
index 00000000..38b23db3
--- /dev/null
+++ b/tests/test_profiling_datasets_memory.py
@@ -0,0 +1,107 @@
+from pathlib import Path
+
+import h5py
+import numpy as np
+
+from profiling.config import load_profiling_config
+from profiling.datasets import (
+ SourceSpec,
+ load_csr_source_into_memory,
+ prepare_local_datasets,
+ select_nested_rows,
+ write_fixture_h5ad,
+ write_h5ad_sample,
+ write_h5ad_sample_from_memory,
+)
+
+
+def _fixture_spec(path: Path, *, nRows: int, nColumns: int, nnz: int) -> SourceSpec:
+ return SourceSpec(
+ datasetId="fixture",
+ versionId="fixture-v1",
+ url="file://fixture",
+ nRows=nRows,
+ nColumns=nColumns,
+ nnz=nnz,
+ sourceBytes=path.stat().st_size,
+ )
+
+
+def test_load_csr_downcasts_indices_to_int32(tmp_path: Path) -> None:
+ source = tmp_path / "source.h5ad"
+ artifact = write_fixture_h5ad(source, nRows=40, nColumns=25, seed=3)
+ spec = _fixture_spec(
+ source,
+ nRows=40,
+ nColumns=25,
+ nnz=artifact.nnz,
+ )
+ memory = load_csr_source_into_memory(source, spec=spec)
+ assert memory.indices.dtype == np.dtype(np.int32)
+ assert memory.indicesDtype == np.dtype(np.int64)
+ assert memory.data.dtype == np.dtype(np.float32)
+ assert int(memory.indptr[-1]) == artifact.nnz
+
+
+def test_write_from_memory_matches_disk_sample(tmp_path: Path) -> None:
+ source = tmp_path / "source.h5ad"
+ artifact = write_fixture_h5ad(source, nRows=60, nColumns=20, seed=4)
+ spec = _fixture_spec(
+ source,
+ nRows=60,
+ nColumns=20,
+ nnz=artifact.nnz,
+ )
+ rows = select_nested_rows(60, (15,), seed=1, sourceVersion=spec.versionId)[15]
+ memory = load_csr_source_into_memory(source, spec=spec)
+
+ disk_path = tmp_path / "disk.h5ad"
+ memory_path = tmp_path / "memory.h5ad"
+ disk = write_h5ad_sample(source, disk_path, rows, spec=spec)
+ mem = write_h5ad_sample_from_memory(memory, memory_path, rows)
+
+ assert disk.nnz == mem.nnz
+ assert disk.sourceRowsSha256 == mem.sourceRowsSha256
+ assert disk.finalSourceRow == mem.finalSourceRow
+
+ with h5py.File(disk_path, "r") as left, h5py.File(memory_path, "r") as right:
+ assert np.array_equal(left["X/data"][:], right["X/data"][:])
+ assert np.array_equal(left["X/indices"][:], right["X/indices"][:])
+ assert np.array_equal(left["X/indptr"][:], right["X/indptr"][:])
+
+
+def test_prepare_local_datasets_uses_in_memory_path(tmp_path: Path) -> None:
+ source = tmp_path / "source.h5ad"
+ artifact = write_fixture_h5ad(source, nRows=80, nColumns=30, seed=5)
+ spec = _fixture_spec(
+ source,
+ nRows=80,
+ nColumns=30,
+ nnz=artifact.nnz,
+ )
+ prepared = prepare_local_datasets(
+ source,
+ tmp_path / "subsets",
+ targetRows=(10, 25),
+ seed=0,
+ spec=spec,
+ )
+ assert [item.targetRows for item in prepared.artifacts] == [10, 25]
+ assert (tmp_path / "subsets" / "10.h5ad").is_file()
+ assert (tmp_path / "subsets" / "25.h5ad").is_file()
+
+ selections = select_nested_rows(
+ 80,
+ (10, 25),
+ seed=0,
+ sourceVersion=spec.versionId,
+ )
+ assert set(selections[10].tolist()).issubset(set(selections[25].tolist()))
+
+
+def test_example_config_loads_prepare_resources() -> None:
+ config = load_profiling_config(
+ Path(__file__).parents[1] / "profiling" / "config.example.toml"
+ )
+ assert config.prepareResources.modalMemoryRequestMb == 196_608
+ assert config.prepareResources.modalMemoryLimitMb == 212_992
diff --git a/tests/test_profiling_e2e.py b/tests/test_profiling_e2e.py
new file mode 100644
index 00000000..10f66655
--- /dev/null
+++ b/tests/test_profiling_e2e.py
@@ -0,0 +1,378 @@
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any
+
+import pytest
+
+from profiling import modal_app
+from profiling.config import (
+ CORE_STAGE_ORDER,
+ ProfilingConfig,
+ StageName,
+ load_profiling_config,
+)
+from profiling.metrics import ResourceMeasurement
+from profiling.stages import StageRunResult
+
+_EXAMPLE_CONFIG = Path(__file__).parents[1] / "profiling" / "config.example.toml"
+
+
+def _config(*, runTag: str = "e2e-test") -> ProfilingConfig:
+ return load_profiling_config(_EXAMPLE_CONFIG).model_copy(update={"runTag": runTag})
+
+
+class _Sampler:
+ def __init__(self, *_args: Any, **_kwargs: Any) -> None:
+ self.started = False
+
+ def start(self) -> "_Sampler":
+ self.started = True
+ return self
+
+ def stop(self) -> ResourceMeasurement:
+ assert self.started
+ return ResourceMeasurement(
+ sampleCount=2,
+ sampleIntervalSeconds=0.1,
+ operationBaselineBytes=100,
+ operationPeakBytes=400,
+ operationIncrementalPeakBytes=300,
+ operationPeakSource="cgroupMemoryCurrent",
+ processTreeRssBaselineBytes=90,
+ processTreeRssPeakBytes=350,
+ processTreeRssIncrementalPeakBytes=260,
+ processTreeRssAfterBytes=120,
+ cgroupMemoryCurrentBaselineBytes=100,
+ cgroupMemoryCurrentPeakBytes=400,
+ cgroupMemoryCurrentAfterBytes=130,
+ cgroupMemoryPeakScope="operation",
+ )
+
+
+def _stage_result(
+ stage: StageName,
+ *,
+ status: str = "ok",
+ error: str | None = None,
+) -> StageRunResult:
+ return StageRunResult(
+ stage=stage,
+ nRows=10_000,
+ status=status,
+ seconds=1.0,
+ peakRssBytes=350,
+ peakCgroupBytes=400,
+ modalMemoryMb=4096,
+ scarfMemoryBudget=2 * 1024**3,
+ storeUri="s3://bucket/store.zarr",
+ error=error,
+ )
+
+
+def _mock_e2e_dependencies(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> tuple[list[dict[str, Any]], list[StageRunResult]]:
+ funnel_payloads: list[dict[str, Any]] = []
+ stage_results: list[StageRunResult] = []
+ monkeypatch.setattr(modal_app, "_WORK", tmp_path)
+ monkeypatch.setattr(modal_app, "_e2e_conflicting_uris", lambda *_args: [])
+ monkeypatch.setattr(modal_app, "ResourceSampler", _Sampler)
+ monkeypatch.setattr(modal_app, "put_json_if_absent", lambda *_args: True)
+
+ def download(_uri: str, destination: Path) -> None:
+ destination.write_bytes(b"h5ad")
+
+ def write_stage(_config: ProfilingConfig, result: StageRunResult) -> str:
+ stage_results.append(result)
+ return _config.resultUri(result.nRows, result.stage)
+
+ def write_funnel(
+ _config: ProfilingConfig,
+ _n_rows: int,
+ payload: dict[str, Any],
+ ) -> str:
+ funnel_payloads.append(payload)
+ return _config.funnelResultUri(_n_rows)
+
+ monkeypatch.setattr(modal_app, "download_file", download)
+ monkeypatch.setattr(modal_app, "write_result", write_stage)
+ monkeypatch.setattr(modal_app, "write_funnel_result", write_funnel)
+ return funnel_payloads, stage_results
+
+
+def test_e2e_funnel_runs_graph_construction_core_once_on_r2(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ config = _config()
+ funnel_payloads, stage_results = _mock_e2e_dependencies(monkeypatch, tmp_path)
+ calls: list[tuple[StageName, dict[str, Any]]] = []
+
+ def run_stage(stage: StageName, **kwargs: Any) -> StageRunResult:
+ calls.append((stage, kwargs))
+ return _stage_result(stage)
+
+ monkeypatch.setattr(modal_app, "run_stage", run_stage)
+ monkeypatch.setattr(
+ modal_app,
+ "result_exists",
+ lambda *_args: (_ for _ in ()).throw(
+ AssertionError("e2e must not skip stage results")
+ ),
+ )
+
+ summary = modal_app.run_e2e_funnel_body(
+ config.model_dump(mode="python"),
+ 10_000,
+ )
+
+ assert [stage for stage, _kwargs in calls] == list(CORE_STAGE_ORDER)
+ assert all(
+ kwargs["storeUri"] == config.storeUri(10_000) for _stage, kwargs in calls
+ )
+ assert calls[0][1]["localH5adPath"].is_file()
+ assert all(kwargs["localH5adPath"] is None for _stage, kwargs in calls[1:])
+ assert all(kwargs["containerMemoryMb"] == 147_456 for _stage, kwargs in calls)
+ assert all(kwargs["containerCpuRequest"] == 16.0 for _stage, kwargs in calls)
+ assert all(kwargs["containerCpuLimit"] == 16.0 for _stage, kwargs in calls)
+ assert all(kwargs["resetCgroupPeak"] is False for _stage, kwargs in calls)
+ assert [result.stage for result in stage_results] == list(CORE_STAGE_ORDER)
+ assert summary["status"] == "ok"
+ assert summary["completedStages"] == list(CORE_STAGE_ORDER)
+ assert summary["peakRssBytes"] == 350
+ assert summary["peakCgroupBytes"] == 400
+ assert summary["rssAfterBytes"] == 120
+ assert summary["cgroupCurrentAfterBytes"] == 130
+ assert summary["modalResources"] == {
+ "modalMemoryRequestMb": 131_072,
+ "modalMemoryLimitMb": 147_456,
+ "modalCpuRequest": 16.0,
+ "modalCpuLimit": 16.0,
+ "ephemeralDiskMb": 524_288,
+ "timeoutSeconds": 86_400,
+ }
+ assert funnel_payloads == [summary]
+
+
+def test_e2e_funnel_stops_and_persists_failure(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ config = _config(runTag="e2e-failure")
+ funnel_payloads, stage_results = _mock_e2e_dependencies(monkeypatch, tmp_path)
+ calls: list[StageName] = []
+
+ def run_stage(stage: StageName, **_kwargs: Any) -> StageRunResult:
+ calls.append(stage)
+ if stage == "runPca":
+ return _stage_result(stage, status="error", error="RuntimeError: failed")
+ return _stage_result(stage)
+
+ monkeypatch.setattr(modal_app, "run_stage", run_stage)
+
+ summary = modal_app.run_e2e_funnel_body(
+ config.model_dump(mode="python"),
+ 10_000,
+ )
+
+ expected = list(CORE_STAGE_ORDER[: CORE_STAGE_ORDER.index("runPca") + 1])
+ assert calls == expected
+ assert [result.stage for result in stage_results] == expected
+ assert summary["status"] == "error"
+ assert summary["failedStage"] == "runPca"
+ assert summary["completedStages"] == expected[:-1]
+ assert funnel_payloads == [summary]
+
+
+def test_e2e_funnel_rejects_empty_or_reused_run_tag(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ with pytest.raises(ValueError, match="non-empty runTag"):
+ modal_app.run_e2e_funnel_body(
+ _config(runTag="").model_dump(mode="python"),
+ 10_000,
+ )
+
+ config = _config(runTag="used")
+ conflict = f"{config.storeUri(10_000)}/zarr.json"
+ monkeypatch.setattr(
+ modal_app,
+ "_e2e_conflicting_uris",
+ lambda *_args: [conflict],
+ )
+ with pytest.raises(FileExistsError, match="fresh runTag"):
+ modal_app.run_e2e_funnel_body(
+ config.model_dump(mode="python"),
+ 10_000,
+ )
+
+ monkeypatch.setattr(modal_app, "_e2e_conflicting_uris", lambda *_args: [])
+ monkeypatch.setattr(modal_app, "put_json_if_absent", lambda *_args: False)
+ with pytest.raises(FileExistsError, match="claimed concurrently"):
+ modal_app.run_e2e_funnel_body(
+ _config(runTag="racing").model_dump(mode="python"),
+ 10_000,
+ )
+
+
+def test_e2e_freshness_checks_store_and_all_results(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ config = _config()
+ checked: list[str] = []
+
+ def exists(uri: str) -> bool:
+ checked.append(uri)
+ return False
+
+ monkeypatch.setattr(modal_app, "object_exists", exists)
+
+ assert modal_app._e2e_conflicting_uris(config, 10_000) == []
+ assert f"{config.storeUri(10_000)}/zarr.json" in checked
+ assert f"{config.storeUri(10_000)}/.zgroup" in checked
+ assert config.e2eClaimUri() in checked
+ assert config.funnelResultUri(10_000) in checked
+ assert {config.resultUri(10_000, stage) for stage in CORE_STAGE_ORDER}.issubset(
+ checked
+ )
+
+
+def test_e2e_modal_options_use_max_resources_and_disable_retries(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ config = _config()
+ captured: dict[str, Any] = {}
+
+ def options(
+ passed_config: ProfilingConfig,
+ resources: Any,
+ *,
+ maxContainers: int,
+ retries: int,
+ ) -> dict[str, Any]:
+ captured.update(
+ config=passed_config,
+ resources=resources,
+ maxContainers=maxContainers,
+ retries=retries,
+ )
+ return {}
+
+ monkeypatch.setattr(modal_app, "modal_function_options", options)
+
+ result = modal_app._e2e_function_options(config)
+
+ assert captured["config"] is config
+ assert captured["maxContainers"] == 1
+ assert captured["retries"] == 0
+ assert result["memory"] == (131_072, 147_456)
+ assert result["cpu"] == (16.0, 16.0)
+ assert result["timeout"] == 86_400
+
+
+def test_e2e_rejects_unavailable_dynamic_ephemeral_disk() -> None:
+ config = _config()
+ resources = dict(config.stageResources)
+ resources["createStore"] = resources["createStore"].model_copy(
+ update={"ephemeralDiskMb": 524_289}
+ )
+ config = config.model_copy(update={"stageResources": resources})
+
+ with pytest.raises(ValueError, match="dynamic ephemeral_disk override"):
+ modal_app._e2e_resource_envelope(config)
+
+
+def test_run_e2e_cli_spawns_deployed_function_and_returns(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ config = _config()
+ captured: dict[str, Any] = {}
+
+ class _Target:
+ def with_options(self, **options: Any) -> "_Target":
+ captured["options"] = options
+ return self
+
+ def spawn(self, *args: Any) -> Any:
+ captured["spawnArgs"] = args
+ return SimpleNamespace(object_id="fc-e2e")
+
+ monkeypatch.setattr(modal_app, "_load_config", lambda _path: config)
+ monkeypatch.setattr(
+ modal_app,
+ "_e2e_function_options",
+ lambda _config: {"timeout": 86_400, "retries": 0},
+ )
+
+ def deployed(_config: ProfilingConfig, name: str) -> _Target:
+ captured["functionName"] = name
+ return _Target()
+
+ monkeypatch.setattr(modal_app, "_deployed_function", deployed)
+ monkeypatch.setattr(
+ modal_app,
+ "_print_spawned",
+ lambda label, call: captured.update(label=label, call=call),
+ )
+
+ modal_app.main(
+ "run-e2e",
+ "--config",
+ "unused.toml",
+ "--size",
+ "10000",
+ )
+
+ assert captured["functionName"] == "run_e2e_funnel_job"
+ assert captured["options"] == {"timeout": 86_400, "retries": 0}
+ payload, n_rows = captured["spawnArgs"]
+ assert payload["runTag"] == "e2e-test"
+ assert n_rows == 10_000
+ assert captured["label"] == "run_e2e_funnel_job 10000"
+
+
+def test_targeted_run_requires_force_to_overwrite_an_existing_result(
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ config = _config()
+ captured: dict[str, Any] = {}
+
+ class _Target:
+ def with_options(self, **options: Any) -> "_Target":
+ captured["options"] = options
+ return self
+
+ def spawn(self, *args: Any) -> Any:
+ captured["spawnArgs"] = args
+ return SimpleNamespace(object_id="fc-force")
+
+ monkeypatch.setattr(modal_app, "_load_config", lambda _path: config)
+ monkeypatch.setattr(modal_app, "result_exists", lambda *_args: True)
+ monkeypatch.setattr(
+ modal_app,
+ "modal_function_options",
+ lambda *_args, **_kwargs: {"retries": 0},
+ )
+ monkeypatch.setattr(modal_app, "run_stage_job", _Target())
+ monkeypatch.setattr(modal_app, "_print_spawned", lambda *_args: None)
+
+ base_args = (
+ "run",
+ "--config",
+ "unused.toml",
+ "--size",
+ "10000",
+ "--stage",
+ "findMarkers",
+ "--ephemeral",
+ )
+ modal_app.main(*base_args)
+ assert "spawnArgs" not in captured
+
+ modal_app.main(*base_args, "--force")
+ payload, n_rows, stage, force = captured["spawnArgs"]
+ assert payload["runTag"] == "e2e-test"
+ assert n_rows == 10_000
+ assert stage == "findMarkers"
+ assert force is True
diff --git a/tests/test_profiling_graph_stages.py b/tests/test_profiling_graph_stages.py
new file mode 100644
index 00000000..ad3e1146
--- /dev/null
+++ b/tests/test_profiling_graph_stages.py
@@ -0,0 +1,243 @@
+import time
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from profiling.config import (
+ GRAPH_CONSTRUCTION_STAGE_ORDER,
+ StageName,
+ StageResources,
+ WorkflowParameters,
+)
+from profiling.stages import _run_analysis, run_stage
+from scarf import DataStore
+
+
+def _resources() -> StageResources:
+ return StageResources(
+ modalMemoryRequestMb=4096,
+ modalMemoryLimitMb=4096,
+ modalCpuRequest=1.0,
+ modalCpuLimit=1.0,
+ scarfMemoryBudget=2 * 1024**3,
+ workers=3,
+ timeoutSeconds=600,
+ ephemeralDiskMb=524_288,
+ )
+
+
+class _RecordingStore:
+ def __init__(self) -> None:
+ self.calls: list[tuple[str, tuple[Any, ...], dict[str, Any]]] = []
+
+ def __getattr__(self, name: str) -> Any:
+ def record(*args: Any, **kwargs: Any) -> object:
+ self.calls.append((name, args, kwargs))
+ return object()
+
+ return record
+
+
+@pytest.mark.parametrize(
+ ("stage", "method", "expected"),
+ [
+ (
+ "runNormalization",
+ "run_normalization",
+ {
+ "from_assay": "RNA",
+ "cell_key": "I",
+ "feat_key": "hvgs",
+ "update_state": True,
+ "invalidate_cache": False,
+ },
+ ),
+ (
+ "runPca",
+ "run_pca",
+ {
+ "from_assay": "RNA",
+ "dims": 37,
+ "local_cache": False,
+ "show_elbow_plot": False,
+ "update_state": True,
+ "invalidate_cache": False,
+ },
+ ),
+ (
+ "buildEmbeddingInitialization",
+ "build_embedding_initialization",
+ {
+ "from_assay": "RNA",
+ "n_centroids": 321,
+ "rand_state": 99,
+ "kmeans_sampling": 0.2,
+ "kmeans_batch_size": 5_000,
+ "update_state": True,
+ "invalidate_cache": False,
+ },
+ ),
+ (
+ "buildAnnIndex",
+ "build_ann_index",
+ {
+ "from_assay": "RNA",
+ "ann_efc": 51,
+ "ann_ef": 51,
+ "ann_m": 55,
+ "ann_parallel": True,
+ "rand_state": 99,
+ "update_state": True,
+ "invalidate_cache": False,
+ },
+ ),
+ (
+ "queryNeighbors",
+ "query_neighbors",
+ {
+ "from_assay": "RNA",
+ "k": 17,
+ "update_state": True,
+ "invalidate_cache": False,
+ },
+ ),
+ (
+ "buildConnectivityMap",
+ "build_connectivity_map",
+ {
+ "from_assay": "RNA",
+ "update_state": True,
+ "invalidate_cache": False,
+ },
+ ),
+ ],
+)
+def test_graph_construction_profile_stage_selects_state_and_preserves_parameters(
+ stage: StageName,
+ method: str,
+ expected: dict[str, Any],
+) -> None:
+ store = _RecordingStore()
+ workflow = WorkflowParameters(
+ dims=37,
+ nCentroids=321,
+ graphSeed=99,
+ kmeansSampling=0.2,
+ kmeansBatchSize=5_000,
+ annParallel=True,
+ k=17,
+ graphLocalCache=False,
+ )
+
+ _run_analysis(
+ stage,
+ store,
+ workflow,
+ _resources(),
+ )
+
+ assert store.calls == [(method, (), expected)]
+
+
+def test_forced_profile_stages_invalidate_reusable_artifacts() -> None:
+ workflow = WorkflowParameters()
+ for stage in (*GRAPH_CONSTRUCTION_STAGE_ORDER, "findMarkers"):
+ store = _RecordingStore()
+ _run_analysis(
+ stage,
+ store,
+ workflow,
+ _resources(),
+ invalidateCache=True,
+ )
+ assert all(
+ kwargs.get("invalidate_cache") is True
+ for _method, _args, kwargs in store.calls
+ )
+
+
+def test_run_stage_reports_store_open_as_input_setup(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ store = object()
+ analysis_kwargs: dict[str, Any] = {}
+
+ def open_store(*_args: Any, **_kwargs: Any) -> object:
+ time.sleep(0.02)
+ return store
+
+ def run_analysis(*_args: Any, **kwargs: Any) -> None:
+ analysis_kwargs.update(kwargs)
+ time.sleep(0.02)
+
+ monkeypatch.setattr("profiling.stages._open_datastore", open_store)
+ monkeypatch.setattr("profiling.stages._run_analysis", run_analysis)
+
+ result = run_stage(
+ "runNormalization",
+ nRows=10_000,
+ storeUri=str(tmp_path / "store.zarr"),
+ workflow=WorkflowParameters(),
+ resources=_resources(),
+ sampleIntervalSeconds=0.005,
+ invalidateCache=True,
+ )
+
+ assert result.status == "ok"
+ assert result.inputSetupSeconds is not None
+ assert result.inputSetupSeconds >= 0.015
+ assert result.seconds is not None
+ assert result.seconds >= 0.015
+ assert result.wholeFunctionSeconds is not None
+ assert result.wholeFunctionSeconds >= (result.inputSetupSeconds + result.seconds)
+ assert result.modalMemoryMb == 4096
+ assert result.modalCpuRequest == 1.0
+ assert result.modalCpuLimit == 1.0
+ assert analysis_kwargs["invalidateCache"] is True
+
+
+@pytest.mark.slow
+def test_graph_construction_profile_stages_chain_through_persisted_assay_state(
+ datastore_ephemeral: DataStore,
+) -> None:
+ datastore_ephemeral.auto_filter_cells(show_qc_plots=False)
+ assay = datastore_ephemeral.get_assay("RNA")
+ if "I__profile_hvgs" not in assay.feats.columns:
+ datastore_ephemeral.mark_hvgs(
+ from_assay="RNA",
+ cell_key="I",
+ top_n=100,
+ hvg_key_name="profile_hvgs",
+ show_plot=False,
+ )
+ store_uri = str(datastore_ephemeral.zarr_loc)
+ workflow = WorkflowParameters(
+ hvgKey="profile_hvgs",
+ dims=5,
+ nCentroids=20,
+ k=3,
+ graphLocalCache=False,
+ )
+
+ for stage in GRAPH_CONSTRUCTION_STAGE_ORDER:
+ result = run_stage(
+ stage,
+ nRows=datastore_ephemeral.cells.N,
+ storeUri=store_uri,
+ workflow=workflow,
+ resources=_resources(),
+ sampleIntervalSeconds=0.01,
+ )
+ assert result.status == "ok", result.error
+
+ reopened = DataStore(store_uri)
+ state = reopened.get_assay_state("RNA")
+ assert state is not None
+ assert state.normalized is not None
+ assert state.reduction is not None
+ assert state.embedding_initialization is not None
+ assert state.ann_index is not None
+ assert state.neighbors is not None
+ assert state.connectivity_map is not None
diff --git a/tests/test_profiling_leiden.py b/tests/test_profiling_leiden.py
new file mode 100644
index 00000000..089e07cd
--- /dev/null
+++ b/tests/test_profiling_leiden.py
@@ -0,0 +1,240 @@
+import json
+import subprocess
+from pathlib import Path
+
+import pytest
+
+from profiling import leiden_worker
+from profiling.config import StageResources, WorkflowParameters
+from profiling.stages import (
+ _monitor_child_process,
+ _run_leiden_in_subprocess,
+ run_stage,
+)
+
+
+def _resources() -> StageResources:
+ return StageResources(
+ modalMemoryRequestMb=32_768,
+ modalMemoryLimitMb=32_768,
+ modalCpuRequest=2.0,
+ modalCpuLimit=2.0,
+ scarfMemoryBudget=24 * 1024**3,
+ workers=2,
+ timeoutSeconds=82_800,
+ ephemeralDiskMb=524_288,
+ )
+
+
+class _Store:
+ def __init__(self, *, error: Exception | None = None) -> None:
+ self.error = error
+ self.arguments: dict[str, object] | None = None
+
+ def run_leiden_clustering(self, **arguments: object) -> None:
+ self.arguments = arguments
+ if self.error is not None:
+ raise self.error
+
+
+def _request(
+ tmpPath: Path,
+ *,
+ workflow: WorkflowParameters | None = None,
+) -> tuple[Path, Path]:
+ status_path = tmpPath / "status.json"
+ request_path = tmpPath / "request.json"
+ request_path.write_text(
+ json.dumps(
+ {
+ "storeUri": "s3://bucket/store.zarr",
+ "workflow": (workflow or WorkflowParameters()).model_dump(mode="json"),
+ "resources": _resources().model_dump(mode="json"),
+ "statusPath": str(status_path),
+ }
+ ),
+ encoding="utf-8",
+ )
+ return request_path, status_path
+
+
+def test_worker_runs_leiden(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ store = _Store()
+ opened: dict[str, object] = {}
+
+ def fake_open(
+ storeUri: str,
+ workflow: WorkflowParameters,
+ resources: StageResources,
+ *,
+ initialize: bool,
+ ) -> _Store:
+ opened.update(
+ storeUri=storeUri,
+ workflow=workflow,
+ resources=resources,
+ initialize=initialize,
+ )
+ return store
+
+ monkeypatch.setattr(leiden_worker, "_open_datastore", fake_open)
+ request_path, status_path = _request(tmp_path)
+
+ leiden_worker.run_leiden_worker(request_path)
+
+ assert opened["storeUri"] == "s3://bucket/store.zarr"
+ assert opened["initialize"] is False
+ assert store.arguments == {
+ "from_assay": "RNA",
+ "cell_key": "I",
+ "feat_key": "hvgs",
+ "resolution": 1.0,
+ "label": "leiden_cluster",
+ "random_seed": 4444,
+ }
+ status = json.loads(status_path.read_text(encoding="utf-8"))
+ assert status["status"] == "ok"
+ assert status["error"] is None
+ assert status["inputSetupSeconds"] >= 0
+ assert status["operationSeconds"] >= 0
+ assert status["wholeWorkerSeconds"] >= (
+ status["inputSetupSeconds"] + status["operationSeconds"]
+ )
+
+
+def test_worker_records_error(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ store = _Store(error=ValueError("bad graph"))
+ monkeypatch.setattr(
+ leiden_worker,
+ "_open_datastore",
+ lambda *_args, **_kwargs: store,
+ )
+ request_path, status_path = _request(tmp_path)
+
+ with pytest.raises(ValueError, match="bad graph"):
+ leiden_worker.run_leiden_worker(request_path)
+
+ status = json.loads(status_path.read_text(encoding="utf-8"))
+ assert status["status"] == "error"
+ assert status["error"] == "ValueError: bad graph"
+ assert status["inputSetupSeconds"] >= 0
+ assert status["operationSeconds"] >= 0
+ assert status["wholeWorkerSeconds"] >= status["operationSeconds"]
+
+
+def test_monitor_warns_without_terminating(
+ monkeypatch: pytest.MonkeyPatch,
+ capsys: pytest.CaptureFixture[str],
+) -> None:
+ class Process:
+ pid = 123
+ waitCalls = 0
+
+ def wait(self, timeout: float | None = None) -> int:
+ self.waitCalls += 1
+ if self.waitCalls < 3:
+ if timeout is None:
+ raise AssertionError("monitor wait must use a timeout")
+ raise subprocess.TimeoutExpired("leiden", timeout)
+ return 0
+
+ clock = iter((0.0, 30.0, 1_801.0))
+ monkeypatch.setattr("profiling.stages.time.monotonic", lambda: next(clock))
+ process = Process()
+
+ assert (
+ _monitor_child_process(
+ process, # type: ignore[arg-type]
+ stageLabel="runLeiden",
+ warningSeconds=1_800.0,
+ pollSeconds=30.0,
+ )
+ == 0
+ )
+
+ output = capsys.readouterr().out
+ assert process.waitCalls == 3
+ assert output.count("WARNING") == 1
+ assert "continuing" in output
+
+
+def test_parent_starts_worker_module(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ commands: list[list[str]] = []
+
+ class Process:
+ pid = 456
+
+ def __init__(self, command: list[str]) -> None:
+ commands.append(command)
+ self.command = command
+
+ def wait(self, timeout: float | None = None) -> int:
+ request_path = Path(self.command[-1])
+ request = json.loads(request_path.read_text(encoding="utf-8"))
+ Path(request["statusPath"]).write_text(
+ json.dumps({"status": "ok", "error": None}),
+ encoding="utf-8",
+ )
+ return 0
+
+ monkeypatch.setattr("profiling.stages.subprocess.Popen", Process)
+
+ _run_leiden_in_subprocess(
+ storeUri="s3://bucket/store.zarr",
+ workflow=WorkflowParameters(),
+ resources=_resources(),
+ workDir=tmp_path,
+ )
+
+ assert commands[0][1:3] == ["-m", "profiling.leiden_worker"]
+ request = json.loads((tmp_path / "request.json").read_text(encoding="utf-8"))
+ assert request["storeUri"] == "s3://bucket/store.zarr"
+ assert "leidenBackend" not in request["workflow"]
+
+
+def test_run_stage_routes_leiden_to_child(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ called: dict[str, object] = {}
+
+ def fake_child(**arguments: object) -> dict[str, object]:
+ called.update(arguments)
+ return {
+ "inputSetupSeconds": 0.5,
+ "operationSeconds": 1.5,
+ "wholeWorkerSeconds": 2.25,
+ }
+
+ def unexpected_open(*_args: object, **_kwargs: object) -> None:
+ raise AssertionError("parent opened the datastore")
+
+ monkeypatch.setattr("profiling.stages._run_leiden_in_subprocess", fake_child)
+ monkeypatch.setattr("profiling.stages._open_datastore", unexpected_open)
+
+ result = run_stage(
+ "runLeiden",
+ nRows=5_000_000,
+ storeUri="s3://bucket/store.zarr",
+ workflow=WorkflowParameters(),
+ resources=_resources(),
+ workDir=tmp_path,
+ sampleIntervalSeconds=0.01,
+ )
+
+ assert result.status == "ok"
+ assert result.inputSetupSeconds == 0.5
+ assert result.seconds == 1.5
+ assert result.details is not None
+ assert result.details["workerWholeSeconds"] == 2.25
+ assert called["storeUri"] == "s3://bucket/store.zarr"
+ assert called["workDir"] == tmp_path
diff --git a/tests/test_profiling_metrics.py b/tests/test_profiling_metrics.py
new file mode 100644
index 00000000..d0792b8e
--- /dev/null
+++ b/tests/test_profiling_metrics.py
@@ -0,0 +1,347 @@
+import json
+import time
+from dataclasses import asdict, fields
+
+import pytest
+
+from profiling.metrics import (
+ DiskUsage,
+ ResourceMeasurement,
+ ResourceSampler,
+ StageTimer,
+ read_process_tree_rss_bytes,
+)
+
+
+def _write_status(proc_root, pid, parent_pid, rss_kib):
+ process = proc_root / str(pid)
+ process.mkdir(parents=True, exist_ok=True)
+ (process / "status").write_text(
+ f"Name:\tprocess-{pid}\nPPid:\t{parent_pid}\nVmRSS:\t{rss_kib} kB\n"
+ )
+
+
+def _write_cgroup(cgroup, *, current=100, peak=900):
+ cgroup.mkdir(parents=True, exist_ok=True)
+ (cgroup / "memory.current").write_text(str(current))
+ (cgroup / "memory.peak").write_text(str(peak))
+ (cgroup / "memory.max").write_text("4096")
+ (cgroup / "memory.swap.current").write_text("5")
+ (cgroup / "memory.swap.max").write_text("max")
+ (cgroup / "memory.events").write_text(
+ "low 0\nhigh 1\nmax 2\noom 0\noom_kill 0\noom_group_kill 0\n"
+ )
+ (cgroup / "cpu.max").write_text("200000 100000")
+
+
+def test_process_tree_rss_aggregates_descendants_by_parent_pid(tmp_path):
+ proc_root = tmp_path / "proc"
+ _write_status(proc_root, 100, 1, 10)
+ _write_status(proc_root, 101, 100, 20)
+ _write_status(proc_root, 102, 101, 30)
+ _write_status(proc_root, 200, 1, 400)
+ (proc_root / "103").mkdir()
+
+ assert read_process_tree_rss_bytes(100, procRoot=proc_root) == (10 + 20 + 30) * 1024
+
+
+def test_sampler_captures_events_limits_cpu_disk_and_operation_peak(tmp_path):
+ proc_root = tmp_path / "proc"
+ cgroup = tmp_path / "cgroup"
+ disk_path = tmp_path / "disk"
+ disk_path.mkdir()
+ _write_status(proc_root, 100, 1, 10)
+ _write_status(proc_root, 101, 100, 20)
+ _write_cgroup(cgroup)
+ disk_samples = iter(
+ (
+ DiskUsage(totalBytes=1000, freeBytes=900, usedBytes=100),
+ DiskUsage(totalBytes=1000, freeBytes=600, usedBytes=400),
+ DiskUsage(totalBytes=1000, freeBytes=750, usedBytes=250),
+ )
+ )
+
+ sampler = ResourceSampler(
+ sampleIntervalSeconds=60,
+ rootPid=100,
+ procRoot=proc_root,
+ cgroupPath=cgroup,
+ ephemeralDiskPath=disk_path,
+ affinityReader=lambda _pid: {3, 1},
+ diskUsageReader=lambda _path: next(disk_samples),
+ )
+ sampler.start()
+
+ assert (cgroup / "memory.peak").read_text() == "0"
+ _write_status(proc_root, 100, 1, 30)
+ _write_status(proc_root, 101, 100, 40)
+ (cgroup / "memory.current").write_text("350")
+ (cgroup / "memory.peak").write_text("420")
+ (cgroup / "memory.swap.current").write_text("20")
+ sampler.sample()
+
+ _write_status(proc_root, 100, 1, 15)
+ _write_status(proc_root, 101, 100, 10)
+ (cgroup / "memory.current").write_text("200")
+ (cgroup / "memory.swap.current").write_text("7")
+ (cgroup / "memory.events").write_text(
+ "low 0\nhigh 3\nmax 5\noom 1\noom_kill 1\noom_group_kill 1\n"
+ )
+ result = sampler.stop()
+
+ assert result.sampleCount == 3
+ assert result.processTreeRssBaselineBytes == 30 * 1024
+ assert result.processTreeRssPeakBytes == 70 * 1024
+ assert result.processTreeRssIncrementalPeakBytes == 40 * 1024
+ assert result.processTreeRssAfterBytes == 25 * 1024
+ assert result.cgroupMemoryCurrentBaselineBytes == 100
+ assert result.cgroupMemoryCurrentPeakBytes == 350
+ assert result.cgroupMemoryCurrentAfterBytes == 200
+ assert result.cgroupMemoryPeakBytes == 420
+ assert result.cgroupMemoryPeakScope == "operation"
+ assert result.operationBaselineBytes == 100
+ assert result.operationPeakBytes == 420
+ assert result.operationIncrementalPeakBytes == 320
+ assert result.operationPeakSource == "cgroupMemoryPeak"
+ assert result.memoryEventsDelta == {
+ "high": 2,
+ "low": 0,
+ "max": 3,
+ "oom": 1,
+ "oomGroupKill": 1,
+ "oomKill": 1,
+ }
+ assert result.memorySwapCurrentBeforeBytes == 5
+ assert result.memorySwapCurrentPeakBytes == 20
+ assert result.memorySwapCurrentAfterBytes == 7
+ assert result.memorySwapMaxBytes == "max"
+ assert result.memoryMaxBytes == 4096
+ assert result.cpuQuotaMicros == 200000
+ assert result.cpuPeriodMicros == 100000
+ assert result.cpuQuotaCores == 2.0
+ assert result.cpuAffinityCpus == (1, 3)
+ assert result.cpuAffinityCount == 2
+ assert result.ephemeralDiskBefore == DiskUsage(1000, 900, 100)
+ assert result.ephemeralDiskPeak == DiskUsage(1000, 600, 400)
+ assert result.ephemeralDiskAfter == DiskUsage(1000, 750, 250)
+ assert all("_" not in field.name for field in fields(ResourceMeasurement))
+ json.dumps(asdict(result))
+
+
+def test_unresettable_cgroup_peak_is_labeled_container_lifetime(tmp_path):
+ proc_root = tmp_path / "proc"
+ cgroup = tmp_path / "cgroup"
+ _write_status(proc_root, 100, 1, 10)
+ _write_cgroup(cgroup, current=100, peak=1000)
+
+ sampler = ResourceSampler(
+ sampleIntervalSeconds=60,
+ rootPid=100,
+ procRoot=proc_root,
+ cgroupPath=cgroup,
+ ephemeralDiskPath=None,
+ writeText=lambda _path, _value: None,
+ ).start()
+ (cgroup / "memory.current").write_text("300")
+ sampler.sample()
+ (cgroup / "memory.current").write_text("200")
+ result = sampler.stop()
+
+ assert result.cgroupMemoryPeakScope == "containerLifetime"
+ assert result.cgroupMemoryPeakBytes == 1000
+ assert result.operationPeakBytes == 300
+ assert result.operationPeakSource == "cgroupMemoryCurrent"
+ assert result.operationIncrementalPeakBytes == 200
+
+
+def test_sampler_can_preserve_an_outer_cgroup_peak(tmp_path):
+ proc_root = tmp_path / "proc"
+ cgroup = tmp_path / "cgroup"
+ _write_status(proc_root, 100, 1, 10)
+ _write_cgroup(cgroup, current=100, peak=1000)
+
+ sampler = ResourceSampler(
+ sampleIntervalSeconds=60,
+ rootPid=100,
+ procRoot=proc_root,
+ cgroupPath=cgroup,
+ ephemeralDiskPath=None,
+ resetCgroupPeak=False,
+ ).start()
+
+ assert (cgroup / "memory.peak").read_text() == "1000"
+ (cgroup / "memory.current").write_text("300")
+ sampler.sample()
+ result = sampler.stop()
+
+ assert result.cgroupMemoryPeakScope == "containerLifetime"
+ assert result.cgroupMemoryPeakBytes == 1000
+ assert result.operationPeakBytes == 300
+ assert result.operationPeakSource == "cgroupMemoryCurrent"
+
+
+def test_sampler_reads_cgroup_v1_memory_files(tmp_path):
+ proc_root = tmp_path / "proc"
+ cgroup_root = tmp_path / "cgroup"
+ memory = cgroup_root / "memory" / "job-1"
+ memory.mkdir(parents=True)
+ (memory / "memory.usage_in_bytes").write_text("150")
+ (memory / "memory.max_usage_in_bytes").write_text("900")
+ (memory / "memory.limit_in_bytes").write_text("4096")
+ (memory / "memory.memsw.usage_in_bytes").write_text("10")
+ (memory / "memory.memsw.limit_in_bytes").write_text("max")
+ proc = proc_root / "100"
+ proc.mkdir(parents=True)
+ (proc / "status").write_text("Name:\tpython\nPPid:\t1\nVmRSS:\t20 kB\n")
+ (proc / "cgroup").write_text("6:memory:/job-1\n")
+
+ sampler = ResourceSampler(
+ sampleIntervalSeconds=60,
+ rootPid=100,
+ procRoot=proc_root,
+ cgroupRoot=cgroup_root,
+ ephemeralDiskPath=None,
+ ).start()
+ (memory / "memory.usage_in_bytes").write_text("400")
+ sampler.sample()
+ result = sampler.stop()
+
+ assert result.cgroupMemoryCurrentBaselineBytes == 150
+ assert result.cgroupMemoryCurrentPeakBytes == 400
+ assert result.memoryMaxBytes == 4096
+ assert result.cgroupMemoryPeakScope == "containerLifetime"
+ assert result.cgroupMemoryPeakBytes == 900
+ assert result.operationPeakSource == "cgroupMemoryCurrent"
+ assert result.operationPeakBytes == 400
+
+
+def test_sampler_falls_back_to_flat_cgroup_v1_memory(tmp_path):
+ proc_root = tmp_path / "proc"
+ cgroup_root = tmp_path / "cgroup"
+ flat = cgroup_root / "memory"
+ flat.mkdir(parents=True)
+ (flat / "memory.usage_in_bytes").write_text("120")
+ (flat / "memory.max_usage_in_bytes").write_text("500")
+ (flat / "memory.limit_in_bytes").write_text("4096")
+ proc = proc_root / "100"
+ proc.mkdir(parents=True)
+ (proc / "status").write_text("Name:\tpython\nPPid:\t1\nVmRSS:\t20 kB\n")
+ (proc / "cgroup").write_text("6:memory:/ta-missing-nested\n")
+
+ sampler = ResourceSampler(
+ sampleIntervalSeconds=60,
+ rootPid=100,
+ procRoot=proc_root,
+ cgroupRoot=cgroup_root,
+ ephemeralDiskPath=None,
+ ).start()
+ (flat / "memory.usage_in_bytes").write_text("300")
+ sampler.sample()
+ result = sampler.stop()
+
+ assert result.cgroupPath == str(flat)
+ assert result.cgroupMemoryCurrentBaselineBytes == 120
+ assert result.cgroupMemoryCurrentPeakBytes == 300
+ assert result.operationPeakSource == "cgroupMemoryCurrent"
+ assert result.memoryEventsBefore is None
+
+
+def test_background_sampler_observes_process_and_cgroup_peaks(tmp_path):
+ proc_root = tmp_path / "proc"
+ cgroup = tmp_path / "cgroup"
+ _write_status(proc_root, 100, 1, 10)
+ _write_status(proc_root, 101, 100, 10)
+ cgroup.mkdir()
+ (cgroup / "memory.current").write_text("100")
+
+ sampler = ResourceSampler(
+ sampleIntervalSeconds=0.005,
+ rootPid=100,
+ procRoot=proc_root,
+ cgroupPath=cgroup,
+ ephemeralDiskPath=None,
+ ).start()
+ count_before_peak = sampler.sampleCount
+ _write_status(proc_root, 100, 1, 50)
+ _write_status(proc_root, 101, 100, 70)
+ (cgroup / "memory.current").write_text("800")
+
+ deadline = time.monotonic() + 1
+ while sampler.sampleCount < count_before_peak + 2:
+ assert time.monotonic() < deadline
+ time.sleep(0.002)
+
+ _write_status(proc_root, 100, 1, 5)
+ _write_status(proc_root, 101, 100, 5)
+ (cgroup / "memory.current").write_text("50")
+ result = sampler.stop()
+
+ assert result.sampleCount >= 4
+ assert result.processTreeRssPeakBytes == 120 * 1024
+ assert result.cgroupMemoryCurrentPeakBytes == 800
+ assert result.operationPeakBytes == 800
+ assert result.cgroupMemoryPeakScope == "unavailable"
+
+
+def test_unavailable_and_protected_metrics_do_not_escape_context(tmp_path):
+ def unavailable(*_args):
+ raise PermissionError("protected")
+
+ sampler = ResourceSampler(
+ sampleIntervalSeconds=0.002,
+ rootPid=100,
+ procRoot=tmp_path / "proc",
+ cgroupPath=tmp_path / "cgroup",
+ ephemeralDiskPath=tmp_path / "disk",
+ readText=unavailable,
+ writeText=unavailable,
+ listPids=unavailable,
+ affinityReader=unavailable,
+ diskUsageReader=unavailable,
+ )
+
+ with pytest.raises(RuntimeError, match="measured failure"):
+ with sampler:
+ time.sleep(0.005)
+ raise RuntimeError("measured failure")
+
+ result = sampler.result
+ assert result is not None
+ assert result.processTreeRssPeakBytes is None
+ assert result.cgroupMemoryCurrentPeakBytes is None
+ assert result.cgroupMemoryPeakBytes is None
+ assert result.cgroupMemoryPeakScope == "unavailable"
+ assert result.memoryEventsBefore is None
+ assert result.memoryEventsAfter is None
+ assert result.memoryEventsDelta is None
+ assert result.cpuQuotaMicros is None
+ assert result.cpuAffinityCpus is None
+ assert result.ephemeralDiskBefore is None
+ assert result.ephemeralDiskAfter is None
+ assert result.ephemeralDiskPeak is None
+ assert result.sampleCount >= 2
+ json.dumps(asdict(result))
+
+
+def test_stage_timer_records_four_independent_scopes():
+ ticks = iter((0.0, 1.0, 3.0, 4.0, 9.0, 10.0, 12.0, 15.0))
+ timer = StageTimer(clock=lambda: next(ticks))
+
+ with timer:
+ with timer.inputSetup():
+ pass
+ with timer.operation():
+ pass
+ with timer.validationPersistence():
+ pass
+
+ assert timer.result.inputSetupSeconds == 2.0
+ assert timer.result.measuredOperationSeconds == 5.0
+ assert timer.result.validationPersistenceSeconds == 2.0
+ assert timer.result.wholeFunctionSeconds == 15.0
+ assert asdict(timer.result) == {
+ "inputSetupSeconds": 2.0,
+ "measuredOperationSeconds": 5.0,
+ "validationPersistenceSeconds": 2.0,
+ "wholeFunctionSeconds": 15.0,
+ }
diff --git a/tests/test_profiling_paris.py b/tests/test_profiling_paris.py
new file mode 100644
index 00000000..db8cac9f
--- /dev/null
+++ b/tests/test_profiling_paris.py
@@ -0,0 +1,190 @@
+import json
+from pathlib import Path
+
+import pytest
+
+from profiling import paris_worker
+from profiling.config import StageResources, WorkflowParameters
+from profiling.stages import _run_paris_in_subprocess, run_stage
+
+
+def _resources() -> StageResources:
+ return StageResources(
+ modalMemoryRequestMb=16_384,
+ modalMemoryLimitMb=16_384,
+ modalCpuRequest=2.0,
+ modalCpuLimit=2.0,
+ scarfMemoryBudget=12 * 1024**3,
+ workers=2,
+ timeoutSeconds=82_800,
+ ephemeralDiskMb=524_288,
+ )
+
+
+class _Store:
+ def __init__(self, *, error: Exception | None = None) -> None:
+ self.error = error
+ self.arguments: dict[str, object] | None = None
+
+ def run_paris_clustering(self, **arguments: object) -> None:
+ self.arguments = arguments
+ if self.error is not None:
+ raise self.error
+
+
+def _request(
+ tmpPath: Path,
+ *,
+ workflow: WorkflowParameters | None = None,
+) -> tuple[Path, Path]:
+ status_path = tmpPath / "status.json"
+ request_path = tmpPath / "request.json"
+ request_path.write_text(
+ json.dumps(
+ {
+ "storeUri": "s3://bucket/store.zarr",
+ "workflow": (workflow or WorkflowParameters()).model_dump(mode="json"),
+ "resources": _resources().model_dump(mode="json"),
+ "statusPath": str(status_path),
+ }
+ ),
+ encoding="utf-8",
+ )
+ return request_path, status_path
+
+
+def test_worker_auto_cut_uses_minimum_cluster_size(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ store = _Store()
+
+ def fake_open(
+ storeUri: str,
+ workflow: WorkflowParameters,
+ resources: StageResources,
+ *,
+ initialize: bool,
+ ) -> _Store:
+ assert storeUri == "s3://bucket/store.zarr"
+ assert initialize is False
+ return store
+
+ monkeypatch.setattr(paris_worker, "_open_datastore", fake_open)
+ request_path, status_path = _request(
+ tmp_path,
+ workflow=WorkflowParameters(
+ parisNClusters="auto",
+ parisMinClusterSize=7,
+ ),
+ )
+
+ paris_worker.run_paris_worker(request_path)
+
+ assert store.arguments == {
+ "from_assay": "RNA",
+ "cell_key": "I",
+ "feat_key": "hvgs",
+ "label": "paris_cluster",
+ "n_clusters": "auto",
+ "min_cluster_size": 7,
+ }
+ status = json.loads(status_path.read_text(encoding="utf-8"))
+ assert status["status"] == "ok"
+ assert status["error"] is None
+ assert status["inputSetupSeconds"] >= 0
+ assert status["operationSeconds"] >= 0
+ assert status["wholeWorkerSeconds"] >= (
+ status["inputSetupSeconds"] + status["operationSeconds"]
+ )
+
+
+def test_worker_straight_cut_uses_n_clusters(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ store = _Store()
+ monkeypatch.setattr(
+ paris_worker,
+ "_open_datastore",
+ lambda *_args, **_kwargs: store,
+ )
+ request_path, status_path = _request(
+ tmp_path,
+ workflow=WorkflowParameters(parisNClusters=12),
+ )
+
+ paris_worker.run_paris_worker(request_path)
+
+ assert store.arguments == {
+ "from_assay": "RNA",
+ "cell_key": "I",
+ "feat_key": "hvgs",
+ "label": "paris_cluster",
+ "n_clusters": 12,
+ }
+ assert json.loads(status_path.read_text(encoding="utf-8"))["status"] == "ok"
+
+
+def test_parent_starts_paris_worker_module(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ commands: list[list[str]] = []
+
+ class Process:
+ pid = 789
+
+ def __init__(self, command: list[str]) -> None:
+ commands.append(command)
+ self.command = command
+
+ def wait(self, timeout: float | None = None) -> int:
+ request_path = Path(self.command[-1])
+ request = json.loads(request_path.read_text(encoding="utf-8"))
+ Path(request["statusPath"]).write_text(
+ json.dumps({"status": "ok", "error": None}),
+ encoding="utf-8",
+ )
+ return 0
+
+ monkeypatch.setattr("profiling.stages.subprocess.Popen", Process)
+
+ _run_paris_in_subprocess(
+ storeUri="s3://bucket/store.zarr",
+ workflow=WorkflowParameters(),
+ resources=_resources(),
+ workDir=tmp_path,
+ )
+
+ assert commands[0][1:3] == ["-m", "profiling.paris_worker"]
+
+
+def test_run_stage_routes_paris_to_child(
+ monkeypatch: pytest.MonkeyPatch,
+ tmp_path: Path,
+) -> None:
+ called: dict[str, object] = {}
+
+ def fake_child(**arguments: object) -> None:
+ called.update(arguments)
+
+ def unexpected_open(*_args: object, **_kwargs: object) -> None:
+ raise AssertionError("parent opened the datastore")
+
+ monkeypatch.setattr("profiling.stages._run_paris_in_subprocess", fake_child)
+ monkeypatch.setattr("profiling.stages._open_datastore", unexpected_open)
+
+ result = run_stage(
+ "runClustering",
+ nRows=5_000_000,
+ storeUri="s3://bucket/store.zarr",
+ workflow=WorkflowParameters(),
+ resources=_resources(),
+ workDir=tmp_path,
+ sampleIntervalSeconds=0.01,
+ )
+
+ assert result.status == "ok"
+ assert called["storeUri"] == "s3://bucket/store.zarr"
+ assert called["workDir"] == tmp_path
diff --git a/tests/test_profiling_r2.py b/tests/test_profiling_r2.py
new file mode 100644
index 00000000..2f5cb1cc
--- /dev/null
+++ b/tests/test_profiling_r2.py
@@ -0,0 +1,36 @@
+from profiling.r2 import join_uri, put_json, put_json_if_absent
+from obstore.store import MemoryStore
+
+
+def test_join_uri():
+ assert (
+ join_uri("s3://bucket/prefix", "10000.h5ad") == "s3://bucket/prefix/10000.h5ad"
+ )
+ assert join_uri("s3://bucket/prefix/", "/a/", "b") == "s3://bucket/prefix/a/b"
+
+
+def test_memory_store_put_get_roundtrip(monkeypatch):
+ store = MemoryStore()
+
+ def fake_open(_uri: str):
+ return store, "results/10000/createStore.json"
+
+ monkeypatch.setattr("profiling.r2.open_r2_object", fake_open)
+ put_json("s3://bucket/results/10000/createStore.json", {"status": "ok"})
+ body = bytes(store.get("results/10000/createStore.json").bytes())
+ assert b'"status":"ok"' in body
+
+
+def test_put_json_if_absent_claims_once(monkeypatch):
+ store = MemoryStore()
+
+ def fake_open(_uri: str):
+ return store, "results/e2e-claim.json"
+
+ monkeypatch.setattr("profiling.r2.open_r2_object", fake_open)
+ uri = "s3://bucket/results/e2e-claim.json"
+
+ assert put_json_if_absent(uri, {"runTag": "first"}) is True
+ assert put_json_if_absent(uri, {"runTag": "second"}) is False
+ body = bytes(store.get("results/e2e-claim.json").bytes())
+ assert b'"runTag":"first"' in body
diff --git a/tests/test_profiling_skip.py b/tests/test_profiling_skip.py
new file mode 100644
index 00000000..5097765b
--- /dev/null
+++ b/tests/test_profiling_skip.py
@@ -0,0 +1,113 @@
+from pathlib import Path
+
+from profiling.config import (
+ CORE_STAGE_ORDER,
+ StorageLayout,
+ WorkflowParameters,
+ _normalize_raw_config,
+ load_profiling_config,
+)
+from profiling.results import result_exists
+from profiling.stages import StageRunResult
+
+_EXAMPLE_CONFIG = Path(__file__).parents[1] / "profiling" / "config.example.toml"
+
+
+def test_example_config_loads():
+ config = load_profiling_config(_EXAMPLE_CONFIG)
+ assert config.modalEnvironmentName == "scarf_profiling"
+ assert set(config.stageResources) == set(CORE_STAGE_ORDER)
+ assert config.effectiveStages == CORE_STAGE_ORDER
+ assert "writeCountsT" in CORE_STAGE_ORDER
+ assert "runClustering" in CORE_STAGE_ORDER
+ assert config.datasetUri(10_000).endswith("/10000.h5ad")
+ assert config.resultUri(10_000, "createStore").endswith(
+ "/results/10000/createStore.json"
+ )
+ assert config.funnelResultUri(10_000).endswith("/results/10000/funnel.json")
+ assert config.e2eClaimUri().endswith("/results/e2e-claim.json")
+ leiden = config.resourcesFor("runLeiden")
+ assert leiden.modalMemoryLimitMb == 32_768
+ assert leiden.modalCpuLimit == 2.0
+
+
+def test_run_tag_isolates_store_and_result_uris():
+ # layouts/ is gitignored, so derive runTag settings from the committed example.
+ config = load_profiling_config(_EXAMPLE_CONFIG).model_copy(
+ update={
+ "runTag": "chunk256m",
+ "storageLayout": StorageLayout(targetChunkBytes=256 * 1024 * 1024),
+ }
+ )
+ assert config.runTag == "chunk256m"
+ assert config.storageLayout.targetChunkBytes == 256 * 1024 * 1024
+ assert config.storeUri(100_000).endswith("/stores/chunk256m/100000.zarr")
+ assert config.resultUri(100_000, "markHvgs").endswith(
+ "/results/chunk256m/100000/markHvgs.json"
+ )
+
+
+def test_fixed_resource_map_expands_the_current_funnel():
+ fixed = {"placeholder": "fixed"}
+ normalized = _normalize_raw_config(
+ {
+ "fixedResources": fixed,
+ }
+ )
+
+ assert normalized["stageResources"] == {stage: fixed for stage in CORE_STAGE_ORDER}
+
+
+def test_marker_group_key_matches_leiden_column():
+ workflow = WorkflowParameters()
+ assert workflow.resolvedMarkerGroupKey == "RNA_leiden_cluster"
+ assert workflow.resolvedHvgKey == "I__hvgs"
+
+
+def test_result_exists_skips_when_object_present(monkeypatch):
+ config = load_profiling_config(_EXAMPLE_CONFIG)
+ monkeypatch.setattr(
+ "profiling.results.object_exists",
+ lambda uri: uri.endswith("/results/10000/createStore.json"),
+ )
+ assert result_exists(config, 10_000, "createStore") is True
+ assert result_exists(config, 10_000, "filterCells") is False
+
+
+def test_stage_run_result_json_shape():
+ result = StageRunResult(
+ stage="reopenStore",
+ nRows=10_000,
+ status="ok",
+ seconds=1.25,
+ peakRssBytes=1024,
+ peakCgroupBytes=2048,
+ modalMemoryMb=20480,
+ modalCpuRequest=4.0,
+ modalCpuLimit=4.0,
+ scarfMemoryBudget=12884901888,
+ storeUri="s3://bucket/stores/10000.zarr",
+ rssBaselineBytes=512,
+ rssIncrementalPeakBytes=512,
+ rssAfterBytes=768,
+ cgroupCurrentBaselineBytes=1024,
+ cgroupCurrentPeakBytes=2048,
+ cgroupCurrentAfterBytes=1536,
+ operationBaselineBytes=1024,
+ operationIncrementalPeakBytes=1024,
+ operationPeakSource="cgroupMemoryCurrent",
+ cgroupPeakScope="operation",
+ )
+ payload = result.to_json()
+ assert payload["stage"] == "reopenStore"
+ assert payload["nRows"] == 10_000
+ assert payload["status"] == "ok"
+ assert payload["seconds"] == 1.25
+ assert payload["modalCpuRequest"] == 4.0
+ assert payload["modalCpuLimit"] == 4.0
+ assert payload["rssBaselineBytes"] == 512
+ assert payload["rssIncrementalPeakBytes"] == 512
+ assert payload["rssAfterBytes"] == 768
+ assert payload["cgroupCurrentAfterBytes"] == 1536
+ assert payload["operationPeakSource"] == "cgroupMemoryCurrent"
+ assert payload["cgroupPeakScope"] == "operation"
diff --git a/tests/test_pseudotime.py b/tests/test_pseudotime.py
new file mode 100644
index 00000000..47336ba2
--- /dev/null
+++ b/tests/test_pseudotime.py
@@ -0,0 +1,630 @@
+import numpy as np
+import pandas as pd
+import pytest
+import zarr
+from scipy.sparse import csr_matrix
+from zarr.storage import MemoryStore
+
+from scarf.assay import Assay
+from scarf.storage.budget import ResourceBudget
+from scarf.trajectory.feature_dynamics import knn_clustering
+from scarf.datastore.datastore import DataStore
+from scarf.trajectory.feature_dynamics import (
+ scatter_feature_clusters as _scatter_feature_clusters,
+ validate_pseudotime_regressor as _validated_pseudotime_regressor,
+)
+from scarf.trajectory.pseudotime import (
+ make_source_sink_vector as _make_source_sink_vector,
+ random_walk_laplacian_transpose as _random_walk_laplacian_transpose,
+ select_pseudotime_component as _select_pseudotime_component,
+ truncated_pba_potential as _truncated_pba_potential,
+ validate_source_sink_labels as _validate_source_sink_labels,
+ validate_source_sink_vector as _validate_source_sink_vector,
+)
+from scarf.trajectory.results import PseudotimeMarkerResult
+
+
+def test_source_sink_vector_supports_one_sided_supervision():
+ labels = pd.Series(["source", "source", "other", "other"])
+ vector = _make_source_sink_vector(labels, ["source"], [])
+
+ assert np.array_equal(vector, [-1.0, -1.0, 1.0, 1.0])
+ assert vector.sum() == pytest.approx(0.0)
+
+
+def test_source_sink_labels_reject_missing_and_overlap():
+ labels = pd.Series(["a", "b", "c"])
+
+ with pytest.raises(ValueError, match="overlap"):
+ _validate_source_sink_labels(labels, ["a"], ["a"], "test cells")
+ with pytest.raises(ValueError, match="Missing sources"):
+ _validate_source_sink_labels(labels, ["missing"], [], "test cells")
+
+
+def test_source_sink_vector_rejects_unbalanced_and_nonfinite_values():
+ for values in (
+ np.array([-2.0, 1.0]),
+ np.array([-1.0, 2.0]),
+ np.array([-1.0, np.nan]),
+ np.array([-1.0, np.inf]),
+ ):
+ with pytest.raises(ValueError):
+ _validate_source_sink_vector(values, 2, "ss_vec")
+
+ column = _validate_source_sink_vector(
+ np.array([[-1.0], [1.0]]),
+ 2,
+ "ss_vec",
+ )
+ assert column.shape == (2,)
+
+
+def test_source_sink_vector_rejects_wrong_shapes():
+ for values in (
+ np.zeros((2, 2)),
+ np.zeros((1, 2)),
+ np.zeros(3),
+ ):
+ with pytest.raises(ValueError):
+ _validate_source_sink_vector(values, 2, "ss_vec")
+
+
+def test_source_sink_vector_rejects_fully_labelled_cells():
+ labels = pd.Series(["source", "sink"])
+ with pytest.raises(ValueError, match="All selected cells"):
+ _make_source_sink_vector(labels, ["source"], ["sink"])
+
+
+def test_largest_component_selection_is_deterministic_on_ties():
+ adjacency = np.zeros((6, 6), dtype=float)
+ for start in (0, 3):
+ adjacency[start, start + 1] = adjacency[start + 1, start] = 1.0
+ adjacency[start + 1, start + 2] = adjacency[start + 2, start + 1] = 1.0
+ selected_indices = np.array([10, 11, 12, 1, 2, 3])
+
+ retained, sizes = _select_pseudotime_component(
+ csr_matrix(adjacency),
+ selected_indices,
+ "largest",
+ )
+
+ assert sizes == [3, 3]
+ assert np.array_equal(retained, [False, False, False, True, True, True])
+ with pytest.raises(ValueError, match="connected components"):
+ _select_pseudotime_component(
+ csr_matrix(adjacency),
+ selected_indices,
+ "error",
+ )
+
+
+def test_truncated_pba_operator_matches_dense_svd_reference():
+ adjacency = np.zeros((8, 8), dtype=float)
+ weights = [1.0, 2.0, 1.5, 3.0, 0.75, 2.5, 1.25]
+ for index, weight in enumerate(weights):
+ adjacency[index, index + 1] = weight
+ adjacency[index + 1, index] = weight
+ laplacian_transpose = _random_walk_laplacian_transpose(csr_matrix(adjacency))
+ source_sink = np.array([-1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])
+ k = 5
+
+ actual = _truncated_pba_potential(
+ laplacian_transpose,
+ k,
+ 17,
+ source_sink,
+ )
+
+ left, singular_values, right_t = np.linalg.svd(
+ laplacian_transpose.toarray(),
+ full_matrices=False,
+ )
+ retained_modes = np.argsort(singular_values)[:k]
+ retained_modes = retained_modes[np.argsort(singular_values[retained_modes])]
+ nonzero_modes = retained_modes[1:]
+ expected = left[:, nonzero_modes] @ (
+ (1.0 / singular_values[nonzero_modes])
+ * (right_t[nonzero_modes, :] @ source_sink)
+ )
+
+ assert np.allclose(actual, expected, rtol=1e-6, atol=1e-8)
+
+
+def test_truncated_pba_logs_svd_stage_start():
+ from loguru import logger
+
+ adjacency = np.zeros((6, 6), dtype=float)
+ for index in range(5):
+ adjacency[index, index + 1] = 1.0
+ adjacency[index + 1, index] = 1.0
+ laplacian_transpose = _random_walk_laplacian_transpose(csr_matrix(adjacency))
+ source_sink = np.array([-1.0, 0.0, 0.0, 0.0, 0.0, 1.0])
+ messages: list[str] = []
+ sink = logger.add(
+ lambda message: messages.append(message.record["message"]),
+ level="DEBUG",
+ )
+ try:
+ _truncated_pba_potential(laplacian_transpose, 3, 7, source_sink)
+ finally:
+ logger.remove(sink)
+
+ assert any(
+ "Pseudotime scoring: calculating SVD" in msg
+ and "k=3" in msg
+ and "shape=(6, 6)" in msg
+ for msg in messages
+ )
+
+
+def test_dense_pba_reference_orders_a_path_from_source_to_sink():
+ adjacency = np.zeros((7, 7), dtype=float)
+ for index in range(6):
+ adjacency[index, index + 1] = 1.0
+ adjacency[index + 1, index] = 1.0
+ laplacian_transpose = _random_walk_laplacian_transpose(csr_matrix(adjacency))
+ source_sink = np.array([-1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])
+
+ potential = np.linalg.pinv(laplacian_transpose.toarray().T) @ source_sink
+
+ assert potential[0] < potential[-1]
+ assert np.all(np.diff(potential) > 0)
+
+
+class _FakeCells:
+ def __init__(self, values: np.ndarray, columns: list[str] | None = None):
+ self.values = values
+ self.columns = ["ptime"] if columns is None else columns
+
+ def fetch(self, _column: str, key: str) -> np.ndarray:
+ assert key
+ return self.values
+
+ def active_index(self, _key: str) -> np.ndarray:
+ return np.arange(self.values.shape[0])
+
+
+class _FakeFeatures:
+ def __init__(self):
+ self.insertions: list[tuple[str, np.ndarray, str]] = []
+
+ @staticmethod
+ def active_index(key: str) -> np.ndarray:
+ assert key == "subset"
+ return np.array([1, 3])
+
+ def insert(
+ self,
+ column_name: str,
+ values: np.ndarray,
+ *,
+ key: str,
+ overwrite: bool,
+ ) -> None:
+ assert overwrite
+ self.insertions.append((column_name, values, key))
+
+ @staticmethod
+ def fetch_all(key: str) -> np.ndarray:
+ assert key == "names"
+ return np.array(["zero", "one", "two", "three"])
+
+
+class _MarkerAssay:
+ def __init__(self):
+ self.name = "RNA"
+ self.cells = _FakeCells(np.array([0.0, 1.0, 2.0, 3.0]))
+ self.feats = _FakeFeatures()
+ self.resources = ResourceBudget(1024**3, 1)
+
+ @staticmethod
+ def iter_normed_feature_wise(**_kwargs):
+ yield pd.DataFrame(
+ {
+ 1: [0.0, 1.0, 2.0, 3.0],
+ 3: [3.0, 2.0, 1.0, 0.0],
+ }
+ )
+
+
+class _MarkerStore:
+ def __init__(self):
+ self.assay = _MarkerAssay()
+ self.resources = ResourceBudget(1024**3, 1)
+
+ def _get_assay(self, _from_assay: str):
+ return self.assay
+
+
+def test_marker_search_writes_strict_feature_subset_with_subset_key():
+ store = _MarkerStore()
+
+ result = DataStore.run_pseudotime_marker_search(
+ store,
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="subset",
+ pseudotime_key="ptime",
+ min_cells=1,
+ )
+
+ assert [item[2] for item in store.assay.feats.insertions] == [
+ "subset",
+ "subset",
+ "subset",
+ ]
+ assert all(item[1].shape == (2,) for item in store.assay.feats.insertions)
+ assert isinstance(result, PseudotimeMarkerResult)
+ assert result.correlation_key == "I__ptime__r"
+ assert result.p_value_key == "I__ptime__p"
+ assert result.p_value_adjusted_key == "I__ptime__padj"
+ assert result.table["feature_index"].tolist() == [1, 3]
+ assert result.table["feature_name"].tolist() == ["one", "three"]
+ assert "p_value_adjusted" in result.table.columns
+
+
+def test_incomplete_current_pseudotime_marker_artifact_is_recomputed(
+ datastore_ephemeral,
+ monkeypatch,
+):
+ import scarf.features.markers as marker_algorithms
+ from scarf.storage.artifacts import ArtifactRef, artifact_path
+
+ assay = datastore_ephemeral.RNA
+ datastore_ephemeral.cells.insert(
+ "current_ptime",
+ np.linspace(0.0, 1.0, datastore_ephemeral.cells.N),
+ overwrite=True,
+ )
+ feature_mask = np.zeros(assay.feats.N, dtype=bool)
+ feature_mask[:16] = True
+ assay.feats.insert(
+ "current_ptime_features",
+ feature_mask,
+ overwrite=True,
+ )
+ arguments = {
+ "from_assay": "RNA",
+ "cell_key": "I",
+ "feat_key": "current_ptime_features",
+ "pseudotime_key": "current_ptime",
+ "min_cells": 1,
+ "gene_batch_size": 8,
+ }
+ first = datastore_ephemeral.run_pseudotime_marker_search(**arguments)
+ old_ref = ArtifactRef.from_dict(
+ assay.z["featureData"][first.correlation_key].attrs["source_artifact"]
+ )
+ old_artifact = datastore_ephemeral.zw[artifact_path(old_ref)]
+ del old_artifact["p_value_adjusted"]
+
+ original = marker_algorithms.find_markers_by_regression
+ calls = 0
+
+ def tracked_marker_search(*args, **kwargs):
+ nonlocal calls
+ calls += 1
+ return original(*args, **kwargs)
+
+ monkeypatch.setattr(
+ marker_algorithms,
+ "find_markers_by_regression",
+ tracked_marker_search,
+ )
+ second = datastore_ephemeral.run_pseudotime_marker_search(**arguments)
+
+ new_ref = ArtifactRef.from_dict(
+ assay.z["featureData"][second.correlation_key].attrs["source_artifact"]
+ )
+ assert calls == 1
+ assert new_ref != old_ref
+ assert "p_value_adjusted" not in old_artifact
+ assert "p_value_adjusted" in datastore_ephemeral.zw[artifact_path(new_ref)]
+
+
+def _prepare_legacy_pseudotime_marker_artifact(datastore_ephemeral):
+ from scarf.storage.artifacts import ArtifactRef, artifact_path
+
+ assay = datastore_ephemeral.RNA
+ n_cells = datastore_ephemeral.cells.N
+ datastore_ephemeral.cells.insert(
+ "legacy_ptime",
+ np.linspace(0.0, 1.0, n_cells),
+ overwrite=True,
+ )
+ feature_mask = np.zeros(assay.feats.N, dtype=bool)
+ feature_mask[:16] = True
+ assay.feats.insert(
+ "legacy_ptime_features",
+ feature_mask,
+ overwrite=True,
+ )
+ arguments = {
+ "from_assay": "RNA",
+ "cell_key": "I",
+ "feat_key": "legacy_ptime_features",
+ "pseudotime_key": "legacy_ptime",
+ "min_cells": 1,
+ "gene_batch_size": 8,
+ }
+ first = datastore_ephemeral.run_pseudotime_marker_search(**arguments)
+ ref = ArtifactRef.from_dict(
+ assay.z["featureData"][first.correlation_key].attrs["source_artifact"]
+ )
+ artifact = datastore_ephemeral.zw[artifact_path(ref)]
+ provenance = dict(artifact.attrs["provenance"])
+ parameters = dict(provenance["parameters"])
+ for field_name in (
+ "association_method",
+ "p_value_method",
+ "adjustment_method",
+ "adjustment_scope",
+ ):
+ parameters.pop(field_name)
+ provenance["parameters"] = parameters
+ artifact.attrs["provenance"] = provenance
+ raw_p_values = np.asarray(artifact["p_value"][:]).copy()
+ raw_r_values = np.asarray(artifact["r_value"][:]).copy()
+ assert np.isfinite(raw_p_values).any()
+ del artifact["p_value_adjusted"]
+ assay.feats.drop(first.p_value_adjusted_key)
+ return (
+ arguments,
+ first,
+ ref,
+ artifact,
+ raw_r_values,
+ raw_p_values,
+ )
+
+
+def test_legacy_pseudotime_marker_artifact_is_upgraded_additively(
+ datastore_ephemeral,
+ monkeypatch,
+):
+ import scarf.features.markers as marker_algorithms
+ from scarf.features.markers.correction import _bh_adjusted_pvalues
+ from scarf.storage.artifacts import ArtifactRef, artifact_path
+
+ (
+ arguments,
+ first,
+ ref,
+ artifact,
+ raw_r_values,
+ raw_p_values,
+ ) = _prepare_legacy_pseudotime_marker_artifact(datastore_ephemeral)
+ attrs_before = dict(artifact.attrs)
+ arrays_before = set(artifact.array_keys())
+
+ def fail_if_recomputed(*_args, **_kwargs):
+ raise AssertionError("legacy pseudotime markers should be reused")
+
+ monkeypatch.setattr(
+ marker_algorithms,
+ "find_markers_by_regression",
+ fail_if_recomputed,
+ )
+ reused = datastore_ephemeral.run_pseudotime_marker_search(**arguments)
+
+ reused_ref = ArtifactRef.from_dict(
+ datastore_ephemeral.RNA.z["featureData"][reused.correlation_key].attrs[
+ "source_artifact"
+ ]
+ )
+ published = datastore_ephemeral.zw[artifact_path(reused_ref)]
+ assert reused_ref != ref
+ assert set(artifact.array_keys()) == arrays_before
+ assert "p_value_adjusted" not in artifact
+ assert dict(artifact.attrs) == attrs_before
+ np.testing.assert_array_equal(artifact["r_value"][:], raw_r_values)
+ np.testing.assert_array_equal(artifact["p_value"][:], raw_p_values)
+ assert set(published.array_keys()) == {
+ "p_value",
+ "p_value_adjusted",
+ "r_value",
+ }
+ np.testing.assert_allclose(
+ reused.table["p_value_adjusted"].to_numpy(),
+ _bh_adjusted_pvalues(raw_p_values),
+ equal_nan=True,
+ )
+ adjusted_column = datastore_ephemeral.RNA.z["featureData"][
+ first.p_value_adjusted_key
+ ]
+ adjusted_ref = ArtifactRef.from_dict(adjusted_column.attrs["source_artifact"])
+ assert adjusted_ref == reused_ref
+ assert adjusted_column.attrs["source_value"] == "p_value_adjusted"
+ assert "p_value_adjusted" in published
+ status = datastore_ephemeral.inspect_artifact(reused_ref)
+ assert status.parameters["association_method"] == "pearson"
+ assert status.parameters["p_value_method"] == "student_t"
+ assert status.parameters["adjustment_method"] == "fdr_bh"
+ assert status.parameters["adjustment_scope"] == "tested_features"
+ assert "algorithm_version" not in status.parameters
+ assert "schema_version" not in status.parameters
+ assert "correction_method" not in status.parameters
+
+
+def test_read_only_legacy_pseudotime_marker_reuse_is_non_mutating(
+ datastore_ephemeral,
+ monkeypatch,
+):
+ import scarf.features.markers as marker_algorithms
+ from scarf.datastore.datastore import DataStore
+ from scarf.features.markers.correction import _bh_adjusted_pvalues
+
+ (
+ arguments,
+ first,
+ _ref,
+ artifact,
+ raw_r_values,
+ raw_p_values,
+ ) = _prepare_legacy_pseudotime_marker_artifact(datastore_ephemeral)
+ attrs_before = dict(artifact.attrs)
+ arrays_before = {
+ name: np.asarray(artifact[name][:]).copy() for name in artifact.array_keys()
+ }
+ feature_columns_before = set(datastore_ephemeral.RNA.z["featureData"].array_keys())
+
+ def fail_if_recomputed(*_args, **_kwargs):
+ raise AssertionError("legacy pseudotime markers should be reused")
+
+ monkeypatch.setattr(
+ marker_algorithms,
+ "find_markers_by_regression",
+ fail_if_recomputed,
+ )
+ read_only = DataStore(
+ datastore_ephemeral.zarr_loc,
+ default_assay="RNA",
+ zarr_mode="r",
+ )
+ reused = read_only.run_pseudotime_marker_search(**arguments)
+
+ np.testing.assert_allclose(
+ reused.table["p_value_adjusted"].to_numpy(),
+ _bh_adjusted_pvalues(raw_p_values),
+ equal_nan=True,
+ )
+ assert first.p_value_adjusted_key not in read_only.RNA.feats.columns
+ assert set(read_only.RNA.z["featureData"].array_keys()) == feature_columns_before
+ assert dict(artifact.attrs) == attrs_before
+ assert set(artifact.array_keys()) == set(arrays_before)
+ for name, expected in arrays_before.items():
+ np.testing.assert_array_equal(artifact[name][:], expected)
+ np.testing.assert_array_equal(artifact["r_value"][:], raw_r_values)
+
+
+def test_regressor_validation_points_to_component_validity_key():
+ with pytest.raises(ValueError, match="ptime__valid"):
+ _validated_pseudotime_regressor(
+ np.array([0.0, np.nan]),
+ 2,
+ "ptime",
+ "I",
+ has_validity_column=True,
+ )
+
+
+class _AggregationCells:
+ def __init__(self, ordering: np.ndarray):
+ self.ordering = ordering
+
+ def fetch(self, _ordering_key: str, key: str) -> np.ndarray:
+ assert key == "I"
+ return self.ordering
+
+
+class _AggregationAssay:
+ def __init__(self, expression: np.ndarray, ordering: np.ndarray):
+ self.expression = expression
+ self.cells = _AggregationCells(ordering)
+ self.z = zarr.open_group(store=MemoryStore(), mode="w")
+ self.nthreads = 1
+ self.resources = ResourceBudget(1024**3, 1)
+
+ def _get_cell_feat_idx(
+ self,
+ _cell_key: str,
+ _feat_key: str,
+ ) -> tuple[np.ndarray, np.ndarray]:
+ return np.arange(self.expression.shape[0]), np.arange(self.expression.shape[1])
+
+ def iter_normed_feature_wise(self, *_args, **_kwargs):
+ yield pd.DataFrame(
+ self.expression,
+ columns=np.arange(self.expression.shape[1]),
+ )
+
+
+def test_aggregation_orders_without_smoothing_and_filters_constant_profiles():
+ expression = np.array(
+ [
+ [10.0, 5.0],
+ [20.0, 5.0],
+ [30.0, 5.0],
+ [40.0, 5.0],
+ ]
+ )
+ assay = _AggregationAssay(expression, np.array([2.0, 0.0, 3.0, 1.0]))
+
+ aggregated, feature_indices = Assay.save_aggregated_ordering(
+ assay,
+ cell_key="I",
+ feat_key="I",
+ ordering_key="ptime",
+ min_exp=0.0,
+ smoothen=False,
+ z_scale=False,
+ window_size=20,
+ chunk_size=20,
+ batch_size=2,
+ )
+
+ assert np.array_equal(feature_indices, [0])
+ assert np.array_equal(
+ aggregated.compute(),
+ np.array([[20.0, 40.0, 10.0, 30.0]]),
+ )
+ group = assay.z["aggregated_I_I_ptime"]
+ assert np.array_equal(group["valid_features"][:], [True, False])
+ assert np.isfinite(group["data"][:]).all()
+ assert "schema_version" not in group.attrs
+ assert all(isinstance(value, str) for value in group.attrs["hashes"])
+
+
+def test_incomplete_aggregation_cache_is_rebuilt():
+ expression = np.array([[1.0, 2.0], [2.0, 3.0], [3.0, 4.0], [4.0, 5.0]])
+ assay = _AggregationAssay(expression, np.arange(4, dtype=float))
+ kwargs = {
+ "cell_key": "I",
+ "feat_key": "I",
+ "ordering_key": "ptime",
+ "smoothen": False,
+ "z_scale": False,
+ "window_size": 2,
+ "chunk_size": 2,
+ "batch_size": 2,
+ }
+
+ Assay.save_aggregated_ordering(assay, **kwargs)
+ group = assay.z["aggregated_I_I_ptime"]
+ del group["valid_features"]
+ group["data"][:] = 999.0
+
+ aggregated, _ = Assay.save_aggregated_ordering(assay, **kwargs)
+
+ assert not np.all(aggregated.compute() == 999.0)
+ assert "valid_features" in assay.z["aggregated_I_I_ptime"]
+ assert "schema_version" not in assay.z["aggregated_I_I_ptime"].attrs
+
+
+class _ShapeOnlyArray:
+ def __init__(self, shape: tuple[int, int]):
+ self.shape = shape
+
+
+def test_knn_clustering_rejects_infeasible_parameters():
+ with pytest.raises(ValueError, match="At least two"):
+ knn_clustering(_ShapeOnlyArray((1, 3)), 1, 1, 1)
+ with pytest.raises(ValueError, match="n_neighbours"):
+ knn_clustering(_ShapeOnlyArray((3, 3)), np.int64(3), np.int32(2), 1)
+ with pytest.raises(ValueError, match="n_clusters"):
+ knn_clustering(_ShapeOnlyArray((3, 3)), np.int32(1), np.int64(4), 1)
+ with pytest.raises(TypeError, match="n_clusters"):
+ knn_clustering(_ShapeOnlyArray((3, 3)), 1, np.bool_(True), 1)
+
+
+def test_feature_cluster_scatter_honors_custom_unassigned_value():
+ values = _scatter_feature_clusters(
+ 5,
+ np.array([1, 3]),
+ np.array([1, 2]),
+ 0,
+ )
+
+ assert np.array_equal(values, [0, 1, 0, 2, 0])
+ with pytest.raises(ValueError, match="conflicts"):
+ _scatter_feature_clusters(3, np.array([0]), np.array([1]), 1)
diff --git a/tests/test_public_api.py b/tests/test_public_api.py
new file mode 100644
index 00000000..9454dda3
--- /dev/null
+++ b/tests/test_public_api.py
@@ -0,0 +1,586 @@
+import json
+import subprocess
+import sys
+from importlib import import_module
+from typing import cast
+
+
+_EXPECTED_EXPORTS = {
+ "ArtifactLineage": "scarf.lineage",
+ "ArtifactRef": "scarf.storage.refs",
+ "ArtifactSelectionError": "scarf.graph.state",
+ "ArtifactStatus": "scarf.storage.artifacts",
+ "AssayMerge": "scarf.merge",
+ "AssayState": "scarf.graph.state",
+ "CSVReader": "scarf.readers",
+ "CSVtoZarr": "scarf.writers",
+ "CrDirReader": "scarf.readers",
+ "CrH5Reader": "scarf.readers",
+ "CrReader": "scarf.readers",
+ "CrToZarr": "scarf.writers",
+ "DataStore": "scarf.datastore.datastore",
+ "DataStoreSummary": "scarf.datastore.summary",
+ "DatasetMerge": "scarf.merge",
+ "EnrichmentResult": "scarf.features.enrichment.results",
+ "FateMappingResult": "scarf.trajectory.results",
+ "GffReader": "scarf.features.genomic.gff",
+ "H5adInspectResult": "scarf.readers",
+ "H5adReader": "scarf.readers",
+ "H5adToZarr": "scarf.writers",
+ "LoomReader": "scarf.readers",
+ "LoomToZarr": "scarf.writers",
+ "MtxReader": "scarf.readers",
+ "MtxToZarr": "scarf.writers",
+ "SeuratImportResult": "scarf.writers",
+ "SeuratInspectResult": "scarf.readers",
+ "SeuratReader": "scarf.readers",
+ "SeuratToZarr": "scarf.writers",
+ "MappingReference": "scarf.mapping.reference",
+ "MappingResult": "scarf.mapping.models",
+ "mount_datastore": "scarf.datastore.datastore",
+ "PseudotimeAggregationResult": "scarf.trajectory.results",
+ "PseudotimeMarkerResult": "scarf.trajectory.results",
+ "PseudotimeScoreResult": "scarf.trajectory.results",
+ "SparseToZarr": "scarf.writers",
+ "SubsetZarr": "scarf.writers",
+ "clean_array": "scarf.utils",
+ "configure_output": "scarf.utils",
+ "controlled_compute": "scarf.utils",
+ "coordinate_melding": "scarf.features.genomic.melding",
+ "create_zarr_count_assay": "scarf.writers",
+ "create_zarr_dataset": "scarf.writers",
+ "create_zarr_obj_array": "scarf.writers",
+ "dask_to_zarr": "scarf.writers",
+ "get_log_level": "scarf.utils",
+ "inspect_h5ad": "scarf.readers",
+ "inspect_mtx": "scarf.readers",
+ "inspect_seurat": "scarf.readers",
+ "load_zarr": "scarf.utils",
+ "logger": "scarf.utils",
+ "permute_into_chunks": "scarf.utils",
+ "read_gmt": "scarf.features.enrichment.net",
+ "rescale_array": "scarf.utils",
+ "rolling_window": "scarf.utils",
+ "set_verbosity": "scarf.utils",
+ "show_dask_progress": "scarf.utils",
+ "subset_assay_zarr": "scarf.writers",
+ "system_call": "scarf.utils",
+ "to_h5ad": "scarf.writers",
+ "to_mtx": "scarf.writers",
+ "tqdmbar": "scarf.utils",
+ "tqdm_params": "scarf.utils",
+ "write_renorm_subset_to_zarr": "scarf.writers",
+}
+
+_EXPECTED_MODULE_ATTRIBUTES = {
+ "assay": "scarf.assay",
+ "cytebase": "scarf.cytebase",
+ "datastore": "scarf.datastore",
+ "embeddings": "scarf.embeddings",
+ "features": "scarf.features",
+ "mapping": "scarf.mapping",
+ "matrix": "scarf.matrix",
+ "merge": "scarf.merge",
+ "metadata": "scarf.metadata",
+ "metrics": "scarf.metrics",
+ "quality_control": "scarf.quality_control",
+ "readers": "scarf.readers",
+ "storage": "scarf.storage",
+ "utils": "scarf.utils",
+ "writers": "scarf.writers",
+}
+
+_EXPECTED_UTILS_EXPORTS = [
+ "logger",
+ "tqdmbar",
+ "tqdm_params",
+ "configure_output",
+ "set_verbosity",
+ "get_log_level",
+ "system_call",
+ "rescale_array",
+ "clean_array",
+ "load_zarr",
+ "permute_into_chunks",
+ "show_dask_progress",
+ "controlled_compute",
+ "iter_column_blocks",
+ "process_rss_mb",
+ "rss_peak_tracker",
+ "array_digest",
+ "rolling_window",
+]
+
+_EXPECTED_PLOTTING_EXPORTS = (
+ "CategoricalScale",
+ "CellField",
+ "ColorScale",
+ "DensityOverlay",
+ "FeatureRef",
+ "FeatureSummary",
+ "Highlight",
+ "LegendSpec",
+ "NormalizationSpec",
+ "PlotProvenance",
+ "PlotOutput",
+ "PlotOutputSettings",
+ "PlotPanelTarget",
+ "PlotRecipe",
+ "PlotRecipeResult",
+ "PlotResult",
+ "PlotStep",
+ "SizeScale",
+ "StudyDesign",
+ "THEMES",
+ "cluster_tree",
+ "cluster_connectivity",
+ "collect_legends",
+ "compose_results",
+ "composition",
+ "distribution",
+ "dotplot",
+ "elbow",
+ "embedding",
+ "embedding_raster",
+ "graph_qc",
+ "highly_variable_features",
+ "label_panels",
+ "marker_heatmap",
+ "mapping_calibration",
+ "mapping_confusion",
+ "mapping_evidence",
+ "mapping_projection",
+ "mapping_score",
+ "matrixplot",
+ "pseudotime_heatmap",
+ "qc",
+ "register_theme",
+ "run_recipe",
+ "theme_context",
+)
+
+
+def _run_probe(source: str) -> dict[str, object]:
+ result = subprocess.run(
+ [sys.executable, "-c", source],
+ check=True,
+ capture_output=True,
+ text=True,
+ )
+ return cast(dict[str, object], json.loads(result.stdout))
+
+
+def test_bare_import_is_lazy():
+ lazy_names = [*_EXPECTED_EXPORTS, *_EXPECTED_MODULE_ATTRIBUTES]
+ result = _run_probe(
+ f"""
+import json
+import sys
+import scarf
+
+heavy_modules = {{
+ "dask",
+ "h5py",
+ "hnswlib",
+ "matplotlib",
+ "numba",
+ "numpy",
+ "pandas",
+ "scipy",
+ "sklearn",
+ "zarr",
+}}
+lazy_names = {lazy_names!r}
+print(json.dumps({{
+ "boundLazyNames": sorted(name for name in lazy_names if name in vars(scarf)),
+ "heavyModules": sorted(heavy_modules.intersection(sys.modules)),
+ "plotsInDir": "plots" in dir(scarf),
+ "plottingInDir": "plotting" in dir(scarf),
+ "scarfModules": sorted(
+ name for name in sys.modules
+ if name == "scarf" or name.startswith("scarf.")
+ ),
+ "versionIsSet": isinstance(scarf.__version__, str) and bool(scarf.__version__),
+}}))
+"""
+ )
+
+ assert result == {
+ "boundLazyNames": [],
+ "heavyModules": [],
+ "plotsInDir": False,
+ "plottingInDir": False,
+ "scarfModules": ["scarf"],
+ "versionIsSet": True,
+ }
+
+
+def test_utils_package_is_lazy():
+ result = _run_probe(
+ f"""
+import json
+import sys
+import scarf.utils as utils
+
+exports = {_EXPECTED_UTILS_EXPORTS!r}
+print(json.dumps({{
+ "advertised": sorted(name for name in exports if name in dir(utils)),
+ "bound": sorted(name for name in exports if name in vars(utils)),
+ "exports": utils.__all__,
+ "heavyModules": sorted({{"numba", "numpy", "scipy", "zarr"}}.intersection(sys.modules)),
+}}))
+"""
+ )
+
+ assert result == {
+ "advertised": sorted(_EXPECTED_UTILS_EXPORTS),
+ "bound": [],
+ "exports": _EXPECTED_UTILS_EXPORTS,
+ "heavyModules": [],
+ }
+
+
+def test_clustering_package_is_lazy():
+ exports = [
+ "BalancedCut",
+ "CoalesceTree",
+ "ParisClusterDiagnostic",
+ "ParisClusteringResult",
+ "adaptive_cut",
+ "balanced_cut",
+ "leiden_membership",
+ "make_digraph",
+ "paris_dendrogram",
+ "straight_cut",
+ ]
+ result = _run_probe(
+ f"""
+import json
+import sys
+import scarf.clustering as clustering
+
+exports = {exports!r}
+print(json.dumps({{
+ "advertised": sorted(name for name in exports if name in dir(clustering)),
+ "bound": sorted(name for name in exports if name in vars(clustering)),
+ "exports": clustering.__all__,
+ "heavyModules": sorted({{"numba", "numpy", "scipy"}}.intersection(sys.modules)),
+}}))
+"""
+ )
+
+ assert result == {
+ "advertised": sorted(exports),
+ "bound": [],
+ "exports": exports,
+ "heavyModules": [],
+ }
+
+
+def test_plotting_package_is_lazy():
+ result = _run_probe(
+ f"""
+import json
+import sys
+import scarf.plotting as plotting
+
+exports = {_EXPECTED_PLOTTING_EXPORTS!r}
+concrete = (
+ "scarf.plotting.cluster_connectivity",
+ "scarf.plotting.composition",
+ "scarf.plotting.diagnostics",
+ "scarf.plotting.distribution",
+ "scarf.plotting.embedding",
+ "scarf.plotting.embedding_raster",
+ "scarf.plotting.heatmaps",
+ "scarf.plotting.mapping",
+ "scarf.plotting.recipes",
+ "scarf.plotting.summary",
+)
+print(json.dumps({{
+ "advertised": sorted(name for name in exports if name in dir(plotting)),
+ "bound": sorted(name for name in exports if name in vars(plotting)),
+ "concreteModules": sorted(set(concrete).intersection(sys.modules)),
+ "exports": plotting.__all__,
+}}))
+"""
+ )
+
+ assert result == {
+ "advertised": sorted(_EXPECTED_PLOTTING_EXPORTS),
+ "bound": [],
+ "concreteModules": [],
+ "exports": list(_EXPECTED_PLOTTING_EXPORTS),
+ }
+
+
+def test_utils_surface_preserves_module_metadata():
+ utils = import_module("scarf.utils")
+
+ assert utils.__all__ == _EXPECTED_UTILS_EXPORTS
+ for name in _EXPECTED_UTILS_EXPORTS:
+ value = getattr(utils, name)
+ if callable(value):
+ assert value.__module__ == "scarf.utils"
+
+
+def test_public_exports_match_canonical_objects():
+ import scarf
+
+ assert scarf.__all__ == list(_EXPECTED_EXPORTS)
+ assert isinstance(scarf.__version__, str)
+ assert scarf.__version__
+ assert set(_EXPECTED_EXPORTS).issubset(dir(scarf))
+
+ for name, module_name in _EXPECTED_EXPORTS.items():
+ canonical = getattr(import_module(module_name), name)
+ exported = getattr(scarf, name)
+ assert exported is canonical
+ assert vars(scarf)[name] is canonical
+ if callable(exported):
+ assert exported.__module__ == module_name
+
+
+def test_marker_facade_does_not_export_layout_internals():
+ markers = import_module("scarf.features.markers")
+ internal_names = {
+ "LEGACY_STAT_COLUMNS",
+ "MARKER_STAT_COLUMNS",
+ "load_marker_table",
+ "read_legacy_marker_table",
+ }
+
+ assert internal_names.isdisjoint(markers.__all__)
+ assert internal_names.isdisjoint(dir(markers))
+
+
+def test_domain_packages_export_canonical_objects():
+ exports = {
+ (
+ "scarf.clustering",
+ "ParisClusterDiagnostic",
+ ): "scarf.clustering.paris_multiscale",
+ (
+ "scarf.clustering",
+ "ParisClusteringResult",
+ ): "scarf.clustering.paris_multiscale",
+ ("scarf.clustering", "adaptive_cut"): "scarf.clustering.paris_multiscale",
+ ("scarf.embeddings", "run_harmony"): "scarf.embeddings.harmony",
+ ("scarf.features", "binned_sampling"): "scarf.features.scoring",
+ ("scarf.features", "fit_lowess"): "scarf.features.variability",
+ (
+ "scarf.features",
+ "EnrichmentResult",
+ ): "scarf.features.enrichment.results",
+ ("scarf.features", "read_gmt"): "scarf.features.enrichment.net",
+ (
+ "scarf.features",
+ "select_highly_variable_features",
+ ): "scarf.features.variability",
+ ("scarf.features", "GffReader"): "scarf.features.genomic.gff",
+ ("scarf.features", "coordinate_melding"): "scarf.features.genomic.melding",
+ ("scarf.mapping", "MappingReference"): "scarf.mapping.reference",
+ ("scarf.mapping", "MappingResult"): "scarf.mapping.models",
+ (
+ "scarf.features",
+ "find_markers_by_rank",
+ ): "scarf.features.markers.search",
+ ("scarf.matrix", "ChunkedArray"): "scarf.matrix.chunked",
+ (
+ "scarf.metrics",
+ "ClusterSeparabilityResult",
+ ): "scarf.metrics.cluster_separability",
+ ("scarf.metrics", "clisi_knn"): "scarf.metrics.lisi",
+ ("scarf.metrics", "compute_lisi"): "scarf.metrics.lisi",
+ (
+ "scarf.metrics",
+ "evaluate_cluster_separability",
+ ): "scarf.metrics.cluster_separability",
+ ("scarf.metrics", "graph_connectivity"): "scarf.metrics.connectivity",
+ ("scarf.metrics", "ilisi_knn"): "scarf.metrics.lisi",
+ ("scarf.metrics", "silhouette_scoring"): "scarf.metrics.silhouette",
+ (
+ "scarf.quality_control",
+ "assign_cell_cycle_phase",
+ ): "scarf.quality_control.cell_cycle",
+ (
+ "scarf.quality_control",
+ "simulate_doublet_pairs",
+ ): "scarf.quality_control.doublets",
+ (
+ "scarf.quality_control",
+ "s_phase_genes",
+ ): "scarf.quality_control.cell_cycle_genes",
+ }
+
+ for (package_name, symbol), module_name in exports.items():
+ package = import_module(package_name)
+ canonical = import_module(module_name)
+ assert getattr(package, symbol) is getattr(canonical, symbol)
+
+
+def test_star_import_matches_all():
+ result = _run_probe(
+ """
+import json
+
+namespace = {}
+exec("from scarf import *", namespace)
+bound = sorted(name for name in namespace if not name.startswith("__"))
+print(json.dumps({"bound": bound}))
+"""
+ )
+
+ assert result["bound"] == sorted(_EXPECTED_EXPORTS)
+
+
+def test_de_facto_module_attributes_resolve_lazily():
+ result = _run_probe(
+ f"""
+import importlib
+import json
+import scarf
+
+module_names = {_EXPECTED_MODULE_ATTRIBUTES!r}
+before = sorted(name for name in module_names if name in vars(scarf))
+advertised = sorted(name for name in module_names if name in dir(scarf))
+identical = {{
+ name: getattr(scarf, name) is importlib.import_module(module_name)
+ for name, module_name in module_names.items()
+}}
+cached = sorted(name for name in module_names if name in vars(scarf))
+print(json.dumps({{
+ "advertised": advertised,
+ "before": before,
+ "cached": cached,
+ "identical": identical,
+}}))
+"""
+ )
+
+ expected_names = sorted(_EXPECTED_MODULE_ATTRIBUTES)
+ assert result["advertised"] == expected_names
+ assert result["before"] == []
+ assert result["cached"] == expected_names
+ assert result["identical"] == dict.fromkeys(_EXPECTED_MODULE_ATTRIBUTES, True)
+
+
+def test_legacy_dataset_download_api_is_absent():
+ from importlib.util import find_spec
+
+ import scarf
+
+ assert not hasattr(scarf, "fetch_dataset")
+ assert not hasattr(scarf, "show_available_datasets")
+ assert find_spec("scarf.readers.datasets") is None
+
+
+def test_lazy_facades_clear_cached_exports_on_reload():
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ """
+import importlib
+
+cases = (
+ ("scarf", "MappingResult"),
+ ("scarf.features", "fit_lowess"),
+ ("scarf.readers", "CSVReader"),
+ ("scarf.writers", "CSVtoZarr"),
+ ("scarf.merge", "DatasetMerge"),
+ ("scarf.utils", "clean_array"),
+ ("scarf.neighbors", "diffusion_operator"),
+ ("scarf.clustering", "balanced_cut"),
+ ("scarf.embeddings", "initial_embedding"),
+ ("scarf.trajectory", "PseudotimeScoreResult"),
+ ("scarf.plotting", "embedding"),
+)
+
+for module_name, export_name in cases:
+ module = importlib.import_module(module_name)
+ original = getattr(module, export_name)
+ setattr(module, export_name, object())
+ importlib.reload(module)
+ assert export_name not in vars(module), (module_name, export_name)
+ assert getattr(module, export_name) is original, (module_name, export_name)
+""",
+ ],
+ check=True,
+ )
+
+
+def test_zarr_warning_filter_does_not_make_import_eager():
+ result = _run_probe(
+ """
+import json
+import sys
+import warnings
+
+import scarf
+
+zarr_was_loaded = "zarr" in sys.modules
+
+import numpy as np
+import zarr
+from zarr.storage import MemoryStore
+
+with warnings.catch_warnings(record=True) as caught:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ root.create_array("text", data=np.asarray(["a", "bb"]))
+
+print(json.dumps({
+ "unstableWarnings": [
+ type(item.message).__name__
+ for item in caught
+ if type(item.message).__name__ == "UnstableSpecificationWarning"
+ ],
+ "zarrWasLoaded": zarr_was_loaded,
+}))
+"""
+ )
+
+ assert result == {"unstableWarnings": [], "zarrWasLoaded": False}
+
+
+def test_modern_plotting_surface_matches_baseline():
+ plotting = import_module("scarf.plotting")
+
+ assert tuple(plotting.__all__) == _EXPECTED_PLOTTING_EXPORTS
+ for name in _EXPECTED_PLOTTING_EXPORTS:
+ assert hasattr(plotting, name)
+
+
+def test_plotting_submodule_import_does_not_clobber_function_export():
+ plotting = import_module("scarf.plotting")
+ canonical_module = import_module("scarf.plotting.embedding")
+
+ assert plotting.embedding is canonical_module.embedding
+
+
+def test_legacy_plotting_surface_remains_absent():
+ from importlib.util import find_spec
+
+ from scarf import DataStore
+ from scarf.datastore.plot_accessor import DataStorePlotAccessor
+
+ plotting = import_module("scarf.plotting")
+
+ assert hasattr(DataStore, "plots")
+ assert DataStorePlotAccessor.__module__ == "scarf.datastore.plot_accessor"
+ assert not hasattr(import_module("scarf"), "DataStorePlotAccessor")
+ assert not [name for name in dir(DataStore) if name.startswith("plot_")]
+ assert "unified_embedding" not in plotting.__all__
+ assert not hasattr(plotting, "unified_embedding")
+ assert find_spec("scarf.plots") is None
+ assert find_spec("scarf.plotting._legacy") is None
+ assert find_spec("scarf.plotting.unified") is None
+
+
+def test_pipeline_accessor_has_a_public_import_path():
+ from scarf import DataStore
+ from scarf.datastore.pipeline_accessor import PipelineAccessor, StepOptions
+
+ assert hasattr(DataStore, "pipeline")
+ assert PipelineAccessor.__module__ == "scarf.datastore.pipeline_accessor"
+ assert StepOptions is not None
diff --git a/tests/test_quality_control.py b/tests/test_quality_control.py
new file mode 100644
index 00000000..8a4f926c
--- /dev/null
+++ b/tests/test_quality_control.py
@@ -0,0 +1,297 @@
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from scarf.quality_control.cell_cycle import assign_cell_cycle_phase
+from scarf.quality_control.doublets import simulate_doublet_pairs
+from scarf.quality_control.filtering import gaussian_quantile_bounds
+
+
+def test_simulate_doublet_pairs_is_seeded_and_heterotypic():
+ clusters = np.array([0, 0, 1, 1])
+ left, right = simulate_doublet_pairs(
+ clusters,
+ n_sim=12,
+ heterotypic_fraction=1.0,
+ rng=np.random.default_rng(11),
+ max_tries=100,
+ )
+
+ np.testing.assert_array_equal(left, [0, 0, 3, 1, 2, 2, 2, 0, 1, 0, 1, 3])
+ np.testing.assert_array_equal(right, [2, 3, 1, 3, 1, 1, 0, 2, 3, 2, 3, 0])
+ assert np.all(clusters[left] != clusters[right])
+
+
+def test_assign_cell_cycle_phase_preserves_rule_precedence():
+ phases = assign_cell_cycle_phase(
+ s_score=np.array([1.0, 0.1, -2.0, 0.0, -1.0]),
+ g2m_score=np.array([0.5, 0.2, -1.0, 0.0, 0.5]),
+ )
+
+ np.testing.assert_array_equal(phases, ["S", "G2M", "G1", "S", "G2M"])
+
+
+def test_gaussian_quantile_bounds_uses_median_and_population_deviation():
+ bounds = gaussian_quantile_bounds(
+ np.array([1.0, 2.0, 3.0, 4.0, 5.0]),
+ min_p=0.1,
+ max_p=0.9,
+ )
+
+ np.testing.assert_allclose(
+ bounds,
+ (1.1876123951263535, 4.8123876048736465),
+ rtol=0,
+ atol=1e-12,
+ )
+
+
+def _snapshot_store(path: str) -> dict[str, bytes]:
+ root = Path(path)
+ return {
+ str(file.relative_to(root)): file.read_bytes()
+ for file in root.rglob("*")
+ if file.is_file()
+ }
+
+
+def _insert_doublet_clusters(datastore) -> str:
+ column = "doublet_test_clusters"
+ datastore.cells.insert(
+ column,
+ np.arange(datastore.cells.N) % 4,
+ overwrite=True,
+ )
+ return column
+
+
+def test_doublet_mapping_is_query_owned_and_leaves_reference_unprojected(
+ analyzed_datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = analyzed_datastore_ephemeral
+ cluster_key = _insert_doublet_clusters(datastore)
+ selected_state = datastore.get_assay_state("RNA")
+ assert selected_state is not None
+ assert selected_state.connectivity_map is not None
+ selected_connectivity = selected_state.connectivity_map
+ reference_projections = set(
+ datastore.list_artifacts(
+ kind="projection",
+ from_assay="RNA",
+ )
+ )
+ temporary_paths: list[Path] = []
+ mapping_calls = 0
+ original_run_mapping = type(datastore).run_mapping
+
+ def observe_mapping(query, reference, mapping_name, **kwargs):
+ nonlocal mapping_calls
+ mapping_calls += 1
+ assert query is not datastore
+ assert mapping_name == "_doublet_sim_RNA"
+ assert kwargs == {"query_assay": "RNA", "save_k": 3}
+ temporary_path = Path(query.zarr_loc)
+ temporary_paths.append(temporary_path)
+ assert temporary_path.exists()
+ before = _snapshot_store(datastore.zarr_loc)
+ result = original_run_mapping(
+ query,
+ reference,
+ mapping_name,
+ **kwargs,
+ )
+ assert _snapshot_store(datastore.zarr_loc) == before
+ return result
+
+ monkeypatch.setattr(type(datastore), "run_mapping", observe_mapping)
+
+ score_column = datastore.run_doublet_detection(
+ cluster_key=cluster_key,
+ cluster_sample_fraction=0.01,
+ max_cells_per_cluster=2,
+ simulation_ratio=0.01,
+ save_k=3,
+ smoothing_t=1,
+ random_seed=19,
+ )
+
+ assert mapping_calls == 1
+ assert temporary_paths and all(not path.exists() for path in temporary_paths)
+ assert (
+ set(
+ datastore.list_artifacts(
+ kind="projection",
+ from_assay="RNA",
+ )
+ )
+ == reference_projections
+ )
+ scores = datastore.cells.fetch(score_column)
+ assert scores.shape == (len(datastore.cells.active_index("I")),)
+ assert np.all(np.isfinite(scores))
+ assert "RNA_doublet_score__raw" not in datastore.cells.columns
+
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.connectivity_map == selected_connectivity
+ reference = datastore.get_mapping_reference()
+ assert reference.neighbors == state.neighbors
+ assert reference.method == "pca"
+ assert reference.symphony_state is None
+
+ first_ref = datastore.zw["cellData"][score_column].attrs["source_artifact"]
+ assert (
+ datastore.run_doublet_detection(
+ cluster_key=cluster_key,
+ cluster_sample_fraction=0.01,
+ max_cells_per_cluster=2,
+ simulation_ratio=0.01,
+ save_k=3,
+ smoothing_t=1,
+ random_seed=19,
+ )
+ == score_column
+ )
+ assert mapping_calls == 1
+ assert datastore.zw["cellData"][score_column].attrs["source_artifact"] == first_ref
+
+
+def test_doublet_mapping_failure_removes_temporary_query_store(
+ analyzed_datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = analyzed_datastore_ephemeral
+ cluster_key = _insert_doublet_clusters(datastore)
+ temporary_paths: list[Path] = []
+ original_run_mapping = type(datastore).run_mapping
+
+ def capture_mapping(query, reference, mapping_name, **kwargs):
+ temporary_paths.append(Path(query.zarr_loc))
+ return original_run_mapping(
+ query,
+ reference,
+ mapping_name,
+ **kwargs,
+ )
+
+ def wrong_length_score(_query, _result, *_args, **_kwargs):
+ yield 0, np.zeros(len(datastore.cells.active_index("I")) - 1)
+
+ monkeypatch.setattr(type(datastore), "run_mapping", capture_mapping)
+ monkeypatch.setattr(type(datastore), "get_mapping_score", wrong_length_score)
+
+ with pytest.raises(RuntimeError, match="selected reference cells"):
+ datastore.run_doublet_detection(
+ cluster_key=cluster_key,
+ cluster_sample_fraction=0.01,
+ max_cells_per_cluster=2,
+ simulation_ratio=0.01,
+ save_k=3,
+ smoothing_t=1,
+ random_seed=23,
+ )
+
+ assert temporary_paths and all(not path.exists() for path in temporary_paths)
+ assert not datastore.list_artifacts(
+ kind="projection",
+ from_assay="RNA",
+ )
+ assert "RNA_doublet_score__raw" not in datastore.cells.columns
+
+
+def test_doublet_detection_rejects_legacy_graph_without_following_it(
+ analyzed_datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = analyzed_datastore_ephemeral
+ cluster_key = _insert_doublet_clusters(datastore)
+ del datastore.zw["RNA/state"].attrs["state"]
+
+ def fail_legacy_lookup(*_args, **_kwargs):
+ raise AssertionError("Legacy graph lookup must not run")
+
+ monkeypatch.setattr(datastore, "get_latest_graph_loc", fail_legacy_lookup)
+
+ with pytest.raises(
+ ValueError,
+ match="artifact-backed connectivity chain.*Rebuild",
+ ):
+ datastore.run_doublet_detection(
+ cluster_key=cluster_key,
+ from_assay="RNA",
+ cell_key="I",
+ feat_key="hvgs",
+ simulation_ratio=0.01,
+ )
+
+
+def test_doublet_detection_rejects_symphony_connectivity_chain(
+ analyzed_datastore_ephemeral,
+ monkeypatch,
+) -> None:
+ datastore = analyzed_datastore_ephemeral
+ cluster_key = _insert_doublet_clusters(datastore)
+ state = datastore.get_assay_state("RNA")
+ assert state is not None
+ assert state.reduction is not None
+ datastore.cells.insert(
+ "doublet_batch",
+ np.where(np.arange(datastore.cells.N) % 2, "a", "b"),
+ overwrite=True,
+ )
+ correction = datastore.run_harmony(
+ ["doublet_batch"],
+ state.reduction,
+ harmony_params={"nclust": 5},
+ update_state=False,
+ )
+ ann_index = datastore.build_ann_index(
+ correction,
+ update_state=False,
+ )
+ neighbors = datastore.query_neighbors(
+ ann_index,
+ coordinates=correction,
+ k=3,
+ update_state=False,
+ )
+ connectivity = datastore.build_connectivity_map(neighbors)
+ references_before = set(
+ datastore.list_artifacts(
+ kind="mapping_reference",
+ from_assay="RNA",
+ )
+ )
+
+ def fail_temporary_store(*_args, **_kwargs):
+ raise AssertionError("Temporary query store must not be created")
+
+ monkeypatch.setattr(
+ datastore,
+ "_create_temporary_datastore",
+ fail_temporary_store,
+ )
+
+ with pytest.raises(
+ ValueError,
+ match="plain scaled-PCA mapping reference.*Symphony",
+ ):
+ datastore.run_doublet_detection(
+ cluster_key=cluster_key,
+ simulation_ratio=0.01,
+ )
+
+ failed_state = datastore.get_assay_state("RNA")
+ assert failed_state is not None
+ assert failed_state.connectivity_map == connectivity
+ assert (
+ set(
+ datastore.list_artifacts(
+ kind="mapping_reference",
+ from_assay="RNA",
+ )
+ )
+ == references_before
+ )
diff --git a/tests/test_rds_wire.py b/tests/test_rds_wire.py
new file mode 100644
index 00000000..1c0d270c
--- /dev/null
+++ b/tests/test_rds_wire.py
@@ -0,0 +1,529 @@
+import bz2
+import gzip
+import hashlib
+import io
+import lzma
+import struct
+from collections.abc import Callable
+from pathlib import Path
+from types import SimpleNamespace
+
+import numpy as np
+import pytest
+import rdata
+import zstandard
+
+from scarf.readers._rds import (
+ AltRepValue,
+ BytecodeValue,
+ ClosureValue,
+ EnvironmentValue,
+ ExternalPointerValue,
+ LazyAtomicVector,
+ LazyStringVector,
+ PairValue,
+ PersistentValue,
+ PromiseValue,
+ RdsClosedError,
+ RdsCompression,
+ RdsFormatError,
+ RdsLimitError,
+ RdsLimits,
+ RType,
+ WeakReferenceValue,
+ open_rds,
+)
+
+
+class Wire:
+ def __init__(self, encoding: str = "xdr", version: int = 3) -> None:
+ self.encoding = encoding
+ self.version = version
+
+ def integer(self, value: int) -> bytes:
+ if self.encoding == "ascii":
+ return f"{value}\n".encode()
+ order = ">" if self.encoding == "xdr" else "<"
+ return struct.pack(f"{order}i", value)
+
+ def real(self, value: float) -> bytes:
+ if self.encoding == "ascii":
+ return f"{value:.16g}\n".encode()
+ order = ">" if self.encoding == "xdr" else "<"
+ return struct.pack(f"{order}d", value)
+
+ def string(self, value: bytes) -> bytes:
+ if self.encoding != "ascii":
+ return value
+ escaped = bytearray()
+ replacements = {
+ ord("\n"): b"\\n",
+ ord("\t"): b"\\t",
+ ord("\\"): b"\\\\",
+ ord("?"): b"\\?",
+ ord("'"): b"\\'",
+ ord('"'): b'\\"',
+ }
+ for byte in value:
+ replacement = replacements.get(byte)
+ if replacement is not None:
+ escaped.extend(replacement)
+ elif byte <= 32 or byte > 126:
+ escaped.extend(f"\\{byte:03o}".encode())
+ else:
+ escaped.append(byte)
+ escaped.append(ord("\n"))
+ return bytes(escaped)
+
+ def header(self) -> bytes:
+ marker = {"xdr": b"X\n", "native": b"B\n", "ascii": b"A\n"}[self.encoding]
+ result = (
+ marker
+ + self.integer(self.version)
+ + self.integer(4 * 65536 + 4 * 256)
+ + self.integer(3 * 65536 + 5 * 256)
+ )
+ if self.version == 3:
+ result += self.integer(5) + self.string(b"UTF-8")
+ return result
+
+ @staticmethod
+ def flags(
+ r_type: RType | int,
+ *,
+ attributes: bool = False,
+ tag: bool = False,
+ object_: bool = False,
+ gp: int = 0,
+ ) -> int:
+ return (
+ int(r_type)
+ | (int(object_) << 8)
+ | (int(attributes) << 9)
+ | (int(tag) << 10)
+ | (gp << 12)
+ )
+
+ def nil(self) -> bytes:
+ return self.integer(RType.NIL_VALUE)
+
+ def char(self, value: bytes | None, *, gp: int = 1 << 6) -> bytes:
+ result = self.integer(self.flags(RType.CHAR, gp=gp))
+ if value is None:
+ return result + self.integer(-1)
+ return result + self.integer(len(value)) + self.string(value)
+
+ def symbol(self, name: str) -> bytes:
+ return self.integer(RType.SYMBOL) + self.char(name.encode())
+
+ def integer_vector(
+ self,
+ values: list[int],
+ *,
+ attributes: bytes | None = None,
+ long_length: bool = False,
+ ) -> bytes:
+ result = self.integer(
+ self.flags(RType.INTEGER, attributes=attributes is not None)
+ )
+ if long_length:
+ result += self.integer(-1) + self.integer(0) + self.integer(len(values))
+ else:
+ result += self.integer(len(values))
+ result += b"".join(self.integer(value) for value in values)
+ return result + (attributes or b"")
+
+ def real_vector(self, values: list[float]) -> bytes:
+ return (
+ self.integer(RType.REAL)
+ + self.integer(len(values))
+ + b"".join(self.real(value) for value in values)
+ )
+
+ def raw_vector(self, values: bytes) -> bytes:
+ data = (
+ b"".join(f"{value:02x}\n".encode() for value in values)
+ if self.encoding == "ascii"
+ else values
+ )
+ return self.integer(RType.RAW) + self.integer(len(values)) + data
+
+ def string_vector(self, values: list[bytes | None]) -> bytes:
+ return (
+ self.integer(RType.STRING)
+ + self.integer(len(values))
+ + b"".join(self.char(value) for value in values)
+ )
+
+ def pair(
+ self,
+ car: bytes,
+ cdr: bytes,
+ *,
+ tag: bytes | None = None,
+ attributes: bytes | None = None,
+ r_type: RType = RType.PAIRLIST,
+ ) -> bytes:
+ return (
+ self.integer(
+ self.flags(
+ r_type,
+ attributes=attributes is not None,
+ tag=tag is not None,
+ )
+ )
+ + (attributes or b"")
+ + (tag or b"")
+ + car
+ + cdr
+ )
+
+ def vector(self, values: list[bytes], *, attributes: bytes | None = None) -> bytes:
+ return (
+ self.integer(self.flags(RType.VECTOR, attributes=attributes is not None))
+ + self.integer(len(values))
+ + b"".join(values)
+ + (attributes or b"")
+ )
+
+ def document(self, root: bytes) -> bytes:
+ return self.header() + root
+
+
+def test_xdr_numeric_vector_is_lazy_and_matches_rdata(tmp_path: Path) -> None:
+ wire = Wire()
+ payload = wire.document(wire.integer_vector([1, -2, 3, 4]))
+ path = tmp_path / "numbers.rds"
+ path.write_bytes(payload)
+
+ with open_rds(path) as document:
+ vector = document.root.value
+ assert isinstance(vector, LazyAtomicVector)
+ np.testing.assert_array_equal(vector.read_block(1, 3), [-2, 3])
+ assert document.metadata.format_version == 3
+ assert document.metadata.native_encoding == "UTF-8"
+ assert document.source.compression is RdsCompression.NONE
+ assert document.source.spooled is False
+ assert document.source.source_sha256 == hashlib.sha256(payload).hexdigest()
+ assert document.source.payload_sha256 == hashlib.sha256(payload).hexdigest()
+ assert document.temp_paths == ()
+
+ oracle = rdata.parser.parse_data(payload, extension=".rds")
+ np.testing.assert_array_equal(vector.materialize(), oracle.object.value)
+
+
+@pytest.mark.parametrize("encoding", ["xdr", "native", "ascii"])
+@pytest.mark.parametrize("version", [2, 3])
+def test_wire_encodings_and_versions(encoding: str, version: int) -> None:
+ wire = Wire(encoding, version)
+ payload = wire.document(wire.real_vector([1.25, -2.5]))
+ with open_rds(io.BytesIO(payload)) as document:
+ assert document.metadata.format_version == version
+ assert document.metadata.encoding.value == encoding
+ np.testing.assert_array_equal(document.root.value[:], [1.25, -2.5])
+
+
+def test_ascii_raw_and_long_vector_lengths() -> None:
+ ascii_wire = Wire("ascii")
+ with open_rds(
+ io.BytesIO(ascii_wire.document(ascii_wire.raw_vector(b"\x00\x7f\xff")))
+ ) as document:
+ np.testing.assert_array_equal(document.root.value[:], [0, 127, 255])
+
+ xdr_wire = Wire()
+ payload = xdr_wire.document(xdr_wire.integer_vector([7, 8, 9], long_length=True))
+ with open_rds(io.BytesIO(payload)) as document:
+ assert document.root.value.length == 3
+ np.testing.assert_array_equal(document.root.value[:], [7, 8, 9])
+
+
+@pytest.mark.parametrize(
+ ("compress", "expected"),
+ [
+ (gzip.compress, RdsCompression.GZIP),
+ (bz2.compress, RdsCompression.BZIP2),
+ (lzma.compress, RdsCompression.XZ),
+ (zstandard.ZstdCompressor().compress, RdsCompression.ZSTD),
+ ],
+)
+def test_streaming_compression_and_cleanup(
+ compress: Callable[[bytes], bytes],
+ expected: RdsCompression,
+ tmp_path: Path,
+) -> None:
+ wire = Wire()
+ payload = wire.document(wire.string_vector([b"alpha", None, b"beta"]))
+ document = open_rds(
+ io.BytesIO(compressed := compress(payload)),
+ temp_dir=tmp_path,
+ )
+ strings = document.root.value
+ assert isinstance(strings, LazyStringVector)
+ assert strings[:] == ["alpha", None, "beta"]
+ assert document.source.compression is expected
+ assert document.source.spooled is True
+ assert document.source.source_sha256 == hashlib.sha256(compressed).hexdigest()
+ assert document.source.payload_sha256 == hashlib.sha256(payload).hexdigest()
+ temp_paths = tuple(Path(path) for path in document.temp_paths)
+ assert temp_paths
+ assert all(path.exists() for path in temp_paths)
+
+ document.close()
+ document.close()
+ assert all(not path.exists() for path in temp_paths)
+ with pytest.raises(RdsClosedError, match="RDS document is closed"):
+ strings[0]
+
+
+def test_s4_slots_and_named_list_helpers() -> None:
+ wire = Wire()
+ slots = wire.pair(
+ wire.integer_vector([42]),
+ wire.nil(),
+ tag=wire.symbol("answer"),
+ )
+ s4 = wire.integer(wire.flags(RType.S4, attributes=True, object_=True)) + slots
+ with open_rds(io.BytesIO(wire.document(s4))) as document:
+ assert document.root.type is RType.S4
+ answer = document.root.slot("answer")
+ assert answer is not None
+ assert answer.value[0] == 42
+
+ names = wire.pair(
+ wire.string_vector([b"left", b"right"]),
+ wire.nil(),
+ tag=wire.symbol("names"),
+ )
+ named = wire.vector(
+ [wire.integer_vector([1]), wire.integer_vector([2])],
+ attributes=names,
+ )
+ with open_rds(io.BytesIO(wire.document(named))) as document:
+ right = document.root.named("right")
+ assert right is not None
+ assert right.value[0] == 2
+ assert [name for name, _value in document.root.named_items()] == [
+ "left",
+ "right",
+ ]
+
+
+def test_lazy_indexes_share_a_bounded_number_of_temp_files(tmp_path: Path) -> None:
+ wire = Wire()
+ payload = wire.document(
+ wire.vector(
+ [
+ wire.string_vector([b"one", b"two"]),
+ wire.string_vector([b"three"]),
+ ]
+ )
+ )
+
+ with open_rds(io.BytesIO(payload), temp_dir=tmp_path) as document:
+ left, right = document.root.value
+ assert left.value[:] == ["one", "two"]
+ assert right.value[:] == ["three"]
+ assert len(document.temp_paths) == 1
+ assert "string-index" in Path(document.temp_paths[0]).name
+
+
+def test_references_preserve_identity_and_environment_cycles() -> None:
+ wire = Wire()
+ repeated_symbol = wire.vector(
+ [
+ wire.symbol("shared"),
+ wire.integer((1 << 8) | RType.REFERENCE),
+ ]
+ )
+ with open_rds(io.BytesIO(wire.document(repeated_symbol))) as document:
+ first, second = document.root.value
+ assert first is second
+
+ frame = wire.pair(
+ wire.integer((1 << 8) | RType.REFERENCE),
+ wire.nil(),
+ tag=wire.symbol("self"),
+ )
+ environment = (
+ wire.integer(RType.ENVIRONMENT)
+ + wire.integer(0)
+ + wire.integer(RType.BASE_ENVIRONMENT)
+ + frame
+ + wire.nil()
+ + wire.nil()
+ )
+ with open_rds(io.BytesIO(wire.document(environment))) as document:
+ value = document.root.value
+ assert isinstance(value, EnvironmentValue)
+ assert isinstance(value.frame.value, PairValue)
+ assert value.frame.value.car is document.root
+
+
+def test_structural_executable_and_opaque_nodes_are_not_executed() -> None:
+ wire = Wire()
+ closure = (
+ wire.integer(wire.flags(RType.CLOSURE, tag=True))
+ + wire.integer(RType.BASE_ENVIRONMENT)
+ + wire.nil()
+ + wire.symbol("body")
+ )
+ with open_rds(io.BytesIO(wire.document(closure))) as document:
+ assert isinstance(document.root.value, ClosureValue)
+ assert document.root.value.body.type is RType.SYMBOL
+
+ promise = wire.pair(
+ wire.integer_vector([3]),
+ wire.symbol("expression"),
+ tag=wire.integer(RType.BASE_ENVIRONMENT),
+ r_type=RType.PROMISE,
+ )
+ with open_rds(io.BytesIO(wire.document(promise))) as document:
+ assert isinstance(document.root.value, PromiseValue)
+ assert document.root.value.expression.type is RType.SYMBOL
+
+ external_pointer = (
+ wire.integer(RType.EXTERNAL_POINTER)
+ + wire.integer((1 << 8) | RType.REFERENCE)
+ + wire.nil()
+ )
+ with open_rds(io.BytesIO(wire.document(external_pointer))) as document:
+ assert isinstance(document.root.value, ExternalPointerValue)
+ assert document.root.value.protected is document.root
+
+ weak_reference = wire.vector(
+ [
+ wire.integer(RType.WEAK_REFERENCE),
+ wire.integer((1 << 8) | RType.REFERENCE),
+ ]
+ )
+ with open_rds(io.BytesIO(wire.document(weak_reference))) as document:
+ first, second = document.root.value
+ assert isinstance(first.value, WeakReferenceValue)
+ assert first is second
+
+ package = (
+ wire.integer(RType.PACKAGE)
+ + wire.integer(0)
+ + wire.integer(1)
+ + wire.char(b"package:base")
+ )
+ with open_rds(io.BytesIO(wire.document(package))) as document:
+ assert isinstance(document.root.value, PersistentValue)
+ assert document.root.value.names[0] == "package:base"
+
+
+def test_bytecode_and_altrep_are_inert_structures() -> None:
+ wire = Wire()
+ bytecode = (
+ wire.integer(RType.BYTECODE)
+ + wire.integer(0)
+ + wire.integer_vector([12])
+ + wire.integer(1)
+ + wire.integer(RType.INTEGER)
+ + wire.integer_vector([99])
+ )
+ with open_rds(io.BytesIO(wire.document(bytecode))) as document:
+ value = document.root.value
+ assert isinstance(value, BytecodeValue)
+ assert value.constants[0].value[0] == 99
+
+ info = wire.pair(
+ wire.symbol("compact_intseq"),
+ wire.pair(wire.symbol("base"), wire.nil()),
+ )
+ altrep = (
+ wire.integer(RType.ALTREP)
+ + info
+ + wire.real_vector([3.0, 1.0, 1.0])
+ + wire.nil()
+ )
+ with open_rds(io.BytesIO(wire.document(altrep))) as document:
+ value = document.root.value
+ assert isinstance(value, AltRepValue)
+ assert value.class_name == "compact_intseq"
+ assert value.package_name == "base"
+ assert value.known is True
+
+
+def test_caps_report_deterministic_object_paths(tmp_path: Path) -> None:
+ wire = Wire()
+ vector_payload = wire.document(wire.integer_vector([1, 2, 3]))
+ with pytest.raises(
+ RdsLimitError,
+ match=r"max_objects exceeded: 1 > 0 at \$",
+ ):
+ open_rds(
+ io.BytesIO(vector_payload),
+ limits=RdsLimits(max_objects=0),
+ )
+
+ with pytest.raises(
+ RdsLimitError,
+ match=r"max_vector_length exceeded: 3 > 2 at \$\.length",
+ ):
+ open_rds(
+ io.BytesIO(vector_payload),
+ limits=RdsLimits(max_vector_length=2),
+ )
+
+ nested = wire.vector([wire.vector([wire.integer_vector([1])])])
+ with pytest.raises(
+ RdsLimitError,
+ match=r"max_depth exceeded: 2 > 1 at \$\[0\]\[0\]",
+ ):
+ open_rds(
+ io.BytesIO(wire.document(nested)),
+ limits=RdsLimits(max_depth=1),
+ )
+
+ version_two_wire = Wire(version=2)
+ strings = version_two_wire.document(version_two_wire.string_vector([b"too long"]))
+ with pytest.raises(
+ RdsLimitError,
+ match=r"max_string_bytes exceeded: 8 > 3 at \$\[0\]",
+ ):
+ open_rds(
+ io.BytesIO(strings),
+ limits=RdsLimits(max_string_bytes=3),
+ )
+
+ compressed = gzip.compress(vector_payload)
+ with pytest.raises(
+ RdsLimitError,
+ match=r"max_temp_bytes exceeded: .* at \$source",
+ ):
+ open_rds(
+ io.BytesIO(compressed),
+ limits=RdsLimits(max_temp_bytes=10),
+ temp_dir=tmp_path,
+ )
+ assert list(tmp_path.iterdir()) == []
+
+
+def test_scratch_free_space_is_checked_before_writing(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ wire = Wire()
+ payload = gzip.compress(wire.document(wire.integer_vector([1, 2, 3])))
+ monkeypatch.setattr(
+ "scarf.readers._rds._storage.shutil.disk_usage",
+ lambda _path: SimpleNamespace(free=0),
+ )
+
+ with pytest.raises(RdsLimitError, match=r"scratch_free_bytes exceeded.*\$source"):
+ open_rds(io.BytesIO(payload), temp_dir=tmp_path)
+ assert list(tmp_path.iterdir()) == []
+
+
+def test_rejects_rdata_headers_and_bad_references() -> None:
+ wire = Wire()
+ with pytest.raises(RdsFormatError, match="RData workspace header"):
+ open_rds(io.BytesIO(b"RDX3\n" + wire.document(wire.nil())))
+
+ bad_reference = wire.document(wire.integer((3 << 8) | RType.REFERENCE))
+ with pytest.raises(
+ RdsFormatError,
+ match=r"reference index 3 is out of range at \$",
+ ):
+ open_rds(io.BytesIO(bad_reference))
diff --git a/tests/test_readers.py b/tests/test_readers.py
new file mode 100644
index 00000000..c9141a0a
--- /dev/null
+++ b/tests/test_readers.py
@@ -0,0 +1,1069 @@
+import numpy as np
+import pytest
+
+
+def _write_sparse_group(parent, key, values, *, encoding_type="csr_matrix"):
+ from scipy.sparse import csc_matrix, csr_matrix
+
+ matrix_type = csc_matrix if encoding_type in {"csc", "csc_matrix"} else csr_matrix
+ matrix = matrix_type(values)
+ sparse = parent.create_group(key)
+ sparse.attrs["encoding-type"] = encoding_type
+ sparse.attrs["encoding-version"] = "0.1.0"
+ sparse.attrs["shape"] = matrix.shape
+ sparse.create_dataset("data", data=matrix.data)
+ sparse.create_dataset("indices", data=matrix.indices.astype(np.int64))
+ sparse.create_dataset("indptr", data=matrix.indptr.astype(np.int64))
+ return matrix
+
+
+def _write_sparse_h5ad(path, values, *, encoding_type="csr_matrix"):
+ import h5py
+
+ with h5py.File(path, mode="w") as h5:
+ matrix = _write_sparse_group(
+ h5,
+ "X",
+ values,
+ encoding_type=encoding_type,
+ )
+
+ obs = h5.create_group("obs")
+ obs.create_dataset(
+ "_index",
+ data=np.array(
+ [f"cell_{index}".encode() for index in range(matrix.shape[0])]
+ ),
+ )
+ var = h5.create_group("var")
+ var.create_dataset(
+ "_index",
+ data=np.array(
+ [f"feature_{index}".encode() for index in range(matrix.shape[1])]
+ ),
+ )
+ var.create_dataset(
+ "feature_name",
+ data=np.array(
+ [f"gene_{index}".encode() for index in range(matrix.shape[1])]
+ ),
+ )
+ h5.create_group("obsm")
+
+
+def _write_cr_h5(path, values, *, legacy=False):
+ import h5py
+ from scipy.sparse import csr_matrix
+
+ matrix = csr_matrix(values)
+ with h5py.File(path, mode="w") as h5:
+ group = h5.create_group("genome" if legacy else "matrix")
+ group.create_dataset("data", data=matrix.data)
+ group.create_dataset("indices", data=matrix.indices.astype(np.int64))
+ group.create_dataset("indptr", data=matrix.indptr.astype(np.int64))
+ group.create_dataset(
+ "barcodes",
+ data=np.array(
+ [f"cell_{index}".encode() for index in range(values.shape[0])]
+ ),
+ )
+ if legacy:
+ group.create_dataset(
+ "genes",
+ data=np.array(
+ [f"feature_{index}".encode() for index in range(values.shape[1])]
+ ),
+ )
+ group.create_dataset(
+ "gene_names",
+ data=np.array(
+ [f"gene_{index}".encode() for index in range(values.shape[1])]
+ ),
+ )
+ else:
+ features = group.create_group("features")
+ features.create_dataset(
+ "id",
+ data=np.array(
+ [f"feature_{index}".encode() for index in range(values.shape[1])]
+ ),
+ )
+ features.create_dataset(
+ "name",
+ data=np.array(
+ [f"gene_{index}".encode() for index in range(values.shape[1])]
+ ),
+ )
+ features.create_dataset(
+ "feature_type",
+ data=np.array(
+ ["Gene Expression".encode() for _ in range(values.shape[1])]
+ ),
+ )
+
+
+def test_toy_crdir_assay_feats_table(toy_crdir_reader):
+ assert np.all(
+ toy_crdir_reader.assayFeats.columns
+ == np.array(["RNA", "ADT", "RNA", "HTO", "RNA", "ADT"])
+ )
+ assert np.all(
+ toy_crdir_reader.assayFeats.values[1:]
+ == [
+ [0, 1, 3, 5, 6, 7],
+ [1, 3, 5, 6, 7, 8],
+ [1, 2, 2, 1, 1, 1],
+ ]
+ )
+ assert toy_crdir_reader.assayFeats.loc["nFeatures"].sum() == 8
+
+
+def test_toy_crdir_reader_cells_feats(toy_crdir_reader):
+ assert toy_crdir_reader.nCells == 3
+ assert toy_crdir_reader.nFeatures == 8
+ assert toy_crdir_reader.cell_names() == ["b1", "b2", "b3"]
+ assert toy_crdir_reader.feature_names() == [
+ "g1",
+ "a1",
+ "a2",
+ "g2",
+ "g3",
+ "h1",
+ "g4",
+ "a3",
+ ]
+ assert toy_crdir_reader.feature_ids() == [
+ "g1",
+ "a1",
+ "a2",
+ "g2",
+ "g3",
+ "h1",
+ "g4",
+ "a3",
+ ]
+
+
+def test_toy_crdir_reader_assay_subsets(toy_crdir_reader):
+ assert toy_crdir_reader.feature_names("RNA") == ["g1", "g2", "g3", "g4"]
+ assert toy_crdir_reader.feature_ids("ADT") == ["a1", "a2", "a3"]
+ assert toy_crdir_reader.feature_names("HTO") == ["h1"]
+
+ with pytest.raises(ValueError, match="Assay ID missing is not valid"):
+ toy_crdir_reader.feature_names("missing")
+
+
+def test_crdir_reader_filters_and_streams_selected_barcodes(tmp_path):
+ from scarf.readers import CrDirReader
+
+ (tmp_path / "features.tsv").write_text(
+ "\n".join(
+ [
+ "f1\tg1\tGene Expression",
+ "f2\tg2\tGene Expression",
+ "f3\tg3\tGene Expression",
+ ]
+ )
+ + "\n"
+ )
+ (tmp_path / "barcodes.tsv").write_text("b1\nb2\nb3\nb4\n")
+ (tmp_path / "matrix.mtx").write_text(
+ "\n".join(
+ [
+ "%%MatrixMarket matrix coordinate integer general",
+ "% tiny deterministic matrix",
+ "3 4 5",
+ "1 1 2",
+ "2 1 3",
+ "1 2 6",
+ "3 3 1",
+ "2 4 4",
+ ]
+ )
+ + "\n"
+ )
+
+ reader = CrDirReader(
+ str(tmp_path),
+ is_filtered=False,
+ filtering_cutoff=4,
+ )
+
+ np.testing.assert_array_equal(reader.validBarcodeIdx, np.array([1, 2]))
+ assert reader.nCells == 2
+ assert reader.cell_names() == ["b1", "b2"]
+ assert reader.read_header().iloc[0].to_dict() == {
+ "nFeatures": 3,
+ "nCells": 4,
+ "nCounts": 5,
+ }
+ np.testing.assert_array_equal(
+ reader._get_valid_barcodes(
+ filtering_cutoff=4,
+ batch_size=2,
+ lines_in_mem=2,
+ ),
+ np.array([1, 2]),
+ )
+
+ chunks = list(reader.consume(batch_size=1, lines_in_mem=2, dtype=np.uint16))
+ assert [chunk.shape for chunk in chunks] == [(1, 3), (1, 3)]
+ assert all(chunk.dtype == np.uint16 for chunk in chunks)
+ np.testing.assert_array_equal(chunks[0].toarray(), [[2, 3, 0]])
+ np.testing.assert_array_equal(chunks[1].toarray(), [[6, 0, 0]])
+
+
+def test_crdir_reader_splits_many_cells_from_one_input_chunk(tmp_path):
+ from scarf.readers import CrDirReader
+
+ (tmp_path / "features.tsv").write_text("f1\tg1\tGene Expression\n")
+ (tmp_path / "barcodes.tsv").write_text("".join(f"b{index}\n" for index in range(5)))
+ (tmp_path / "matrix.mtx").write_text(
+ "\n".join(
+ [
+ "%%MatrixMarket matrix coordinate integer general",
+ "1 5 5",
+ *(f"1 {index + 1} {index + 1}" for index in range(5)),
+ ]
+ )
+ + "\n"
+ )
+
+ reader = CrDirReader(str(tmp_path))
+ chunks = list(reader.consume(batch_size=1, lines_in_mem=100))
+
+ assert reader.producer_staging_bytes(1, 100) > (
+ 100 * 3 * np.dtype(np.int64).itemsize
+ )
+ assert [chunk.shape for chunk in chunks] == [(1, 1)] * 5
+ np.testing.assert_array_equal(
+ np.vstack([chunk.toarray() for chunk in chunks]),
+ np.arange(1, 6).reshape(-1, 1),
+ )
+
+
+def test_crdir_reader_coalesces_duplicates_across_input_chunks(tmp_path):
+ from scarf.readers import CrDirReader
+
+ n_entries = 1_000
+ (tmp_path / "features.tsv").write_text("f1\tg1\tGene Expression\n")
+ (tmp_path / "barcodes.tsv").write_text("b1\n")
+ (tmp_path / "matrix.mtx").write_text(
+ "\n".join(
+ [
+ "%%MatrixMarket matrix coordinate integer general",
+ f"1 1 {n_entries}",
+ *("1 1 1" for _ in range(n_entries)),
+ ]
+ )
+ + "\n"
+ )
+
+ reader = CrDirReader(str(tmp_path))
+ chunks = list(
+ reader.consume(
+ batch_size=1,
+ lines_in_mem=100,
+ dtype=np.uint32,
+ )
+ )
+
+ assert len(chunks) == 1
+ assert chunks[0].nnz == 1
+ np.testing.assert_array_equal(chunks[0].toarray(), [[n_entries]])
+
+
+def test_crdir_reader_supports_gzip_and_metadata_fallback(tmp_path):
+ import gzip
+
+ from scarf.readers import CrDirReader
+
+ files = {
+ "genes.tsv.gz": "f1\nf2\n",
+ "barcodes.tsv.gz": "b1\nb2\n",
+ "matrix.mtx.gz": "\n".join(
+ [
+ "%%MatrixMarket matrix coordinate integer general",
+ "2 2 2",
+ "1 1 7",
+ "2 2 9",
+ ]
+ )
+ + "\n",
+ }
+ for name, contents in files.items():
+ with gzip.open(tmp_path / name, mode="wt") as handle:
+ handle.write(contents)
+
+ reader = CrDirReader(str(tmp_path))
+
+ assert reader.feature_ids() == ["f1", "f2"]
+ assert reader.feature_names() == ["f1", "f2"]
+ assert reader.feature_types() == ["Gene Expression", "Gene Expression"]
+ assert reader.cell_names() == ["b1", "b2"]
+ with pytest.raises(ValueError, match="Dataset key must be provided"):
+ reader._read_dataset()
+
+ chunks = list(reader.consume(batch_size=2, lines_in_mem=1))
+ assert len(chunks) == 1
+ np.testing.assert_array_equal(chunks[0].toarray(), [[7, 0], [0, 9]])
+
+
+def test_toy_crdir_empty(toy_crdir_empty):
+ assert toy_crdir_empty.nCells == 0
+ assert toy_crdir_empty.nFeatures == 4
+ assert toy_crdir_empty.feature_names() == [
+ "g1",
+ "a1",
+ "a2",
+ "g2",
+ ]
+ assert toy_crdir_empty.feature_ids() == [
+ "g1",
+ "a1",
+ "a2",
+ "g2",
+ ]
+ # check for raise ValueError
+ try:
+ toy_crdir_empty.read_header()
+ except ValueError:
+ pass
+
+
+def test_crh5reader(crh5_reader):
+ assert crh5_reader.nCells == 892
+ assert crh5_reader.nFeatures == 36611
+ n_assay_feats = list(crh5_reader.assayFeats.T.nFeatures.values)
+ assert n_assay_feats == [36601, 10]
+
+
+def test_crh5reader_streams_counts(crh5_reader):
+ streamed_rows = 0
+ streamed_nnz = 0
+
+ indptr = crh5_reader.grp["indptr"]
+ assert crh5_reader.producer_staging_bytes(300, 1) > (
+ indptr.size * indptr.dtype.itemsize
+ )
+ for chunk in crh5_reader.consume(batch_size=300):
+ assert 0 < chunk.shape[0] <= 300
+ assert chunk.shape[1] == crh5_reader.nFeatures
+ streamed_rows += chunk.shape[0]
+ streamed_nnz += chunk.nnz
+
+ assert streamed_rows == crh5_reader.nCells
+ assert streamed_nnz == crh5_reader.grp["data"].shape[0]
+ assert len(crh5_reader.cell_names()) == crh5_reader.nCells
+
+
+def test_crh5reader_filters_background_barcodes(crh5_reader):
+ from scarf.readers import CrH5Reader
+
+ reader = CrH5Reader(
+ crh5_reader.h5obj.filename,
+ is_filtered=False,
+ filtering_cutoff=0,
+ )
+ try:
+ indptr = reader.grp["indptr"][:]
+ expected = np.flatnonzero(np.diff(indptr) > 0)
+ np.testing.assert_array_equal(reader.validBarcodeIdx, expected)
+ assert reader.cell_names() == list(
+ np.asarray(crh5_reader.cell_names())[expected]
+ )
+ finally:
+ reader.close()
+
+
+def test_crh5reader_preserves_filtered_values_dtype_and_batching(tmp_path):
+ from scarf.readers import CrH5Reader
+
+ values = np.array(
+ [
+ [1, 0, 2],
+ [0, 0, 0],
+ [3, 4, 0],
+ [0, 5, 6],
+ ],
+ dtype=np.uint16,
+ )
+ path = tmp_path / "modern.h5"
+ _write_cr_h5(path, values)
+ reader = CrH5Reader(str(path), is_filtered=False, filtering_cutoff=0)
+ try:
+ chunks = list(reader.consume(batch_size=2))
+ observed = np.concatenate([chunk.toarray() for chunk in chunks])
+ np.testing.assert_array_equal(reader.validBarcodeIdx, [0, 2, 3])
+ np.testing.assert_array_equal(observed, values[[0, 2, 3]])
+ assert [chunk.shape[0] for chunk in chunks] == [2, 1]
+ assert all(chunk.dtype == values.dtype for chunk in chunks)
+ finally:
+ reader.close()
+ assert not reader.h5obj.id.valid
+
+
+def test_crh5reader_preserves_legacy_layout_values(tmp_path):
+ from scarf.readers import CrH5Reader
+
+ values = np.array([[1, 0], [0, 2], [3, 4]], dtype=np.uint32)
+ path = tmp_path / "legacy.h5"
+ _write_cr_h5(path, values, legacy=True)
+ reader = CrH5Reader(str(path))
+ try:
+ observed = np.concatenate(
+ [chunk.toarray() for chunk in reader.consume(batch_size=2)]
+ )
+ np.testing.assert_array_equal(observed, values)
+ assert reader.feature_ids() == ["feature_0", "feature_1"]
+ assert reader.feature_names() == ["gene_0", "gene_1"]
+ assert reader.feature_types() == ["Gene Expression", "Gene Expression"]
+ finally:
+ reader.close()
+
+
+def test_crdir_reader(crdir_reader):
+ assert crdir_reader.nCells == 892
+ assert crdir_reader.nFeatures == 36601 # Does not contain 10 ADTs
+
+
+def test_h5ad_reader(h5ad_reader):
+ assert h5ad_reader.nCells == 3696 == len(h5ad_reader.cell_ids())
+ assert h5ad_reader.nFeatures == 27998 == len(h5ad_reader.feat_names())
+
+
+def test_inspect_h5ad_resolves_fixture_and_builds_reader(bastidas_ponce_data):
+ from scarf.readers import H5adReader, inspect_h5ad
+
+ inspection = inspect_h5ad(bastidas_ponce_data)
+
+ assert inspection.matrixKey == "X"
+ assert inspection.matrixEncoding == "csr"
+ assert inspection.matrixCandidates == ("X", "layers/spliced", "layers/unspliced")
+ assert inspection.featureAttrsKey == "var"
+ assert inspection.cellIdsKey == "index"
+ assert inspection.featureIdsKey == "index"
+ assert inspection.layers == ("spliced", "unspliced")
+ assert inspection.nCells == 3696
+ assert inspection.nFeatures == 27998
+
+ reader = H5adReader.from_inspect(inspection)
+ try:
+ assert reader.matrixKey == inspection.matrixKey
+ assert reader.cellIdsKey == inspection.cellIdsKey
+ assert reader.featIdsKey == inspection.featureIdsKey
+ assert reader.nCells == inspection.nCells
+ assert reader.nFeatures == inspection.nFeatures
+ finally:
+ reader.h5.close()
+
+
+def test_inspect_h5ad_prefers_dimension_matched_raw_counts(tmp_path):
+ import h5py
+
+ from scarf.readers import H5adReader, inspect_h5ad
+
+ file_name = tmp_path / "discovery.h5ad"
+ with h5py.File(file_name, mode="w") as h5:
+ _write_sparse_group(
+ h5,
+ "X",
+ np.array([[0.1, 0.0], [0.0, 1.5]], dtype=np.float32),
+ )
+ _write_sparse_group(
+ h5,
+ "raw/X",
+ np.array([[1, 0, 3], [0, 2, 0]], dtype=np.uint16),
+ )
+ layers = h5.create_group("layers")
+ layers.create_dataset(
+ "scaled",
+ data=np.array([[1.0, 2.0], [3.0, 4.0]], dtype=np.float32),
+ )
+
+ obs = h5.create_group("obs")
+ obs.create_dataset("barcode", data=np.array([b"cell-a", b"cell-b"]))
+ obs.create_dataset("batch", data=np.array([0, 1], dtype=np.int8))
+ categories = obs.create_group("categories")
+ categories.create_dataset("batch", data=np.array([b"A", b"B"]))
+
+ var = h5.create_group("var")
+ var.create_dataset("gene_ids", data=np.array([b"v1", b"v2"]))
+
+ raw_var = h5.create_group("raw/var")
+ raw_var.create_dataset(
+ "gene_ids",
+ data=np.array([b"ENSG00000000001", b"ENSG00000000002", b"AB-1"]),
+ )
+ raw_var.create_dataset(
+ "gene_symbol",
+ data=np.array([b"GENE1", b"GENE2", b"CD3"]),
+ )
+ raw_var.create_dataset(
+ "feature_types",
+ data=np.array(
+ [b"Gene Expression", b"Gene Expression", b"Antibody Capture"]
+ ),
+ )
+ uns = h5.create_group("uns")
+ uns.create_dataset("title", data=np.bytes_("Discovery fixture"))
+ uns.create_dataset("citation", data=np.bytes_("Synthetic citation"))
+
+ inspection = inspect_h5ad(str(file_name))
+
+ assert inspection.matrixKey == "raw/X"
+ assert inspection.matrixCandidates == ("raw/X", "layers/scaled", "X")
+ assert inspection.featureAttrsKey == "raw/var"
+ assert inspection.cellIdsKey == "barcode"
+ assert inspection.featureIdsKey == "gene_ids"
+ assert inspection.featureNameKey == "gene_symbol"
+ assert inspection.categoryNamesKey == "categories"
+ assert inspection.assaySplitKey == "feature_types"
+ assert inspection.suggestedAssays == {"RNA": 2, "ADT": 1}
+ assert inspection.layers == ("scaled",)
+ assert inspection.title == "Discovery fixture"
+ assert inspection.description == "Synthetic citation"
+ assert inspection.to_reader_kwargs()["feature_ids_key"] == "gene_ids"
+
+ reader = H5adReader.from_inspect(inspection)
+ try:
+ np.testing.assert_array_equal(
+ reader.feat_ids(),
+ [b"ENSG00000000001", b"ENSG00000000002", b"AB-1"],
+ )
+ np.testing.assert_array_equal(
+ reader.feat_names(),
+ [b"GENE1", b"GENE2", b"CD3"],
+ )
+ np.testing.assert_array_equal(
+ np.vstack([chunk.toarray() for chunk in reader.consume(1)]),
+ [[1, 0, 3], [0, 2, 0]],
+ )
+ np.testing.assert_array_equal(
+ dict(reader.get_cell_columns())["batch"],
+ [b"A", b"B"],
+ )
+ finally:
+ reader.h5.close()
+
+ overridden = H5adReader.from_inspect(
+ inspection,
+ feature_name_key="gene_ids",
+ )
+ try:
+ assert overridden.featNamesKey == "gene_ids"
+ finally:
+ overridden.h5.close()
+
+
+def test_inspect_h5ad_falls_back_from_mismatched_raw_var(tmp_path):
+ import h5py
+
+ from scarf.readers import inspect_h5ad
+
+ file_name = tmp_path / "mismatched_raw_var.h5ad"
+ with h5py.File(file_name, mode="w") as h5:
+ _write_sparse_group(
+ h5,
+ "X",
+ np.array([[1, 0], [0, 2]], dtype=np.uint16),
+ )
+ obs = h5.create_group("obs")
+ obs.create_dataset("barcode", data=np.array([b"c1", b"c2"]))
+ var = h5.create_group("var")
+ var.create_dataset(
+ "opaque_long",
+ data=np.array([b"ENSG00000000001", b"ENSG00000000002"]),
+ )
+ var.create_dataset("opaque_short", data=np.array([b"G1", b"G2"]))
+ raw_var = h5.create_group("raw/var")
+ raw_var.create_dataset(
+ "gene_ids",
+ data=np.array([b"raw1", b"raw2", b"raw3"]),
+ )
+
+ inspection = inspect_h5ad(str(file_name))
+
+ assert inspection.featureAttrsKey == "var"
+ assert inspection.featureIdsKey == "opaque_long"
+ assert inspection.featureNameKey == "opaque_short"
+
+
+def test_h5ad_reader_streams_sparse_matrix(h5ad_reader):
+ streamed_rows = 0
+ streamed_nnz = 0
+ streamed_sum = 0.0
+
+ for chunk in h5ad_reader.consume(batch_size=1000):
+ assert 0 < chunk.shape[0] <= 1000
+ assert chunk.shape[1] == h5ad_reader.nFeatures
+ assert chunk.dtype == h5ad_reader.matrixDtype
+ streamed_rows += chunk.shape[0]
+ streamed_nnz += chunk.nnz
+ streamed_sum += chunk.data.sum(dtype=np.float64)
+
+ matrix_data = h5ad_reader.h5[h5ad_reader.matrixKey]["data"]
+ assert streamed_rows == h5ad_reader.nCells
+ assert streamed_nnz == matrix_data.shape[0]
+ np.testing.assert_allclose(
+ streamed_sum,
+ matrix_data[:].sum(dtype=np.float64),
+ )
+
+
+@pytest.mark.parametrize(
+ ("values", "batch_size"),
+ [
+ (
+ np.array(
+ [
+ [1, 0, 2],
+ [0, 0, 0],
+ [0, 3, 0],
+ [0, 0, 0],
+ ],
+ dtype=np.uint32,
+ ),
+ 2,
+ ),
+ (
+ np.array(
+ [
+ [1, 0, 0],
+ [0, 2, 0],
+ [0, 0, 0],
+ [3, 0, 4],
+ [0, 5, 0],
+ ],
+ dtype=np.uint32,
+ ),
+ 2,
+ ),
+ ],
+)
+def test_h5ad_reader_preserves_sparse_batches(tmp_path, values, batch_size):
+ from scarf.readers import H5adReader
+
+ file_name = tmp_path / "sparse.h5ad"
+ _write_sparse_h5ad(file_name, values)
+ reader = H5adReader(str(file_name), feature_name_key="feature_name")
+ try:
+ expected_max_nnz = max(
+ np.count_nonzero(values[start : start + batch_size])
+ for start in range(
+ 0,
+ max(1, values.shape[0] - batch_size + 1),
+ )
+ )
+ assert reader.max_batch_nnz(batch_size) == expected_max_nnz
+ chunks = list(reader.consume(batch_size=batch_size))
+ assert sum(chunk.shape[0] for chunk in chunks) == values.shape[0]
+ assert sum(chunk.nnz for chunk in chunks) == np.count_nonzero(values)
+ np.testing.assert_array_equal(
+ np.vstack([chunk.toarray() for chunk in chunks]),
+ values,
+ )
+ finally:
+ reader.h5.close()
+
+
+def test_h5ad_reader_converts_csc_sparse_encoding(tmp_path):
+ import h5py
+ import zarr
+
+ from scarf.readers import H5adReader
+ from scarf.readers import inspect_h5ad
+ from scarf.writers import H5adToZarr
+
+ file_name = tmp_path / "csc.h5ad"
+ zarr_path = tmp_path / "csc.zarr"
+ values = np.array(
+ [
+ [1, 0, 2],
+ [0, 3, 0],
+ [4, 0, 5],
+ [0, 6, 0],
+ ],
+ dtype=np.uint16,
+ )
+ _write_sparse_h5ad(
+ file_name,
+ values,
+ encoding_type="csc_matrix",
+ )
+ with h5py.File(file_name, mode="r+") as h5:
+ shape = h5["X"].attrs["shape"]
+ del h5["X"].attrs["encoding-type"]
+ del h5["X"].attrs["shape"]
+ h5["X"].attrs["h5sparse_format"] = "csc"
+ h5["X"].attrs["h5sparse_shape"] = shape
+
+ inspection = inspect_h5ad(str(file_name))
+ assert inspection.matrixEncoding == "csc"
+
+ reader = H5adReader.from_inspect(inspection)
+ try:
+ chunks = list(reader.consume(batch_size=2))
+ np.testing.assert_array_equal(
+ np.vstack([chunk.toarray() for chunk in chunks]),
+ values,
+ )
+ writer = H5adToZarr(reader, zarr_loc=str(zarr_path))
+ writer.dump(batch_size=2)
+ finally:
+ reader.h5.close()
+
+ root = zarr.open_group(str(zarr_path), mode="r")
+ np.testing.assert_array_equal(root["RNA/counts"][:], values)
+ assert "countsT" not in root["RNA"]
+
+
+def test_h5ad_to_zarr_preserves_exact_sparse_batch(tmp_path):
+ import zarr
+
+ from scarf.readers import H5adReader
+ from scarf.writers import H5adToZarr
+
+ values = np.array(
+ [
+ [1, 0, 2],
+ [0, 0, 0],
+ [0, 3, 0],
+ [0, 0, 0],
+ ],
+ dtype=np.uint32,
+ )
+ file_name = tmp_path / "exact_batch.h5ad"
+ zarr_path = tmp_path / "exact_batch.zarr"
+ _write_sparse_h5ad(file_name, values)
+ reader = H5adReader(str(file_name), feature_name_key="feature_name")
+ try:
+ writer = H5adToZarr(reader, zarr_loc=str(zarr_path))
+ writer.dump(batch_size=2)
+ finally:
+ reader.h5.close()
+
+ root = zarr.open_group(str(zarr_path), mode="r")
+ np.testing.assert_array_equal(root["RNA/counts"][:], values)
+ assert "countsT" not in root["RNA"]
+
+
+def test_h5ad_reader_streams_cell_and_feature_metadata(h5ad_reader):
+ cell_columns = dict(h5ad_reader.get_cell_columns())
+
+ assert "index" not in cell_columns
+ assert {
+ "clusters_coarse",
+ "clusters",
+ "S_score",
+ "G2M_score",
+ "X_pca1",
+ "X_pca50",
+ "X_umap1",
+ "X_umap2",
+ } <= cell_columns.keys()
+ assert all(
+ values.shape == (h5ad_reader.nCells,) for values in cell_columns.values()
+ )
+ np.testing.assert_array_equal(
+ cell_columns["clusters_coarse"][:3],
+ np.array([b"Pre-endocrine", b"Ductal", b"Endocrine"]),
+ )
+ np.testing.assert_allclose(
+ cell_columns["X_umap1"][:3],
+ np.array([6.143066, -9.906417, 7.559791], dtype=np.float32),
+ )
+
+ feature_columns = dict(h5ad_reader.get_feat_columns())
+ assert feature_columns.keys() == {"highly_variable_genes"}
+ assert feature_columns["highly_variable_genes"].shape == (h5ad_reader.nFeatures,)
+ # Legacy codes here are {-1, 0, 1} against categories [False, True]; the
+ # -1 sentinel decodes to missing rather than wrapping to the last category.
+ np.testing.assert_array_equal(
+ feature_columns["highly_variable_genes"][:3],
+ np.array([b"False", None, None], dtype=object),
+ )
+ np.testing.assert_array_equal(
+ h5ad_reader._replace_category_values(
+ np.array([10_000]),
+ "clusters",
+ h5ad_reader.cellAttrsKey,
+ ),
+ np.array([10_000]),
+ )
+ assert h5ad_reader._check_exists("layers", "spliced")
+
+
+def test_h5ad_reader_dense_matrix_and_group_metadata(tmp_path):
+ import h5py
+ from loguru import logger
+
+ from scarf.readers import H5adReader, inspect_h5ad
+
+ file_name = tmp_path / "dense.h5ad"
+ with h5py.File(file_name, mode="w") as h5:
+ h5.create_dataset(
+ "X",
+ data=np.array([[1, 0], [0, 2], [3, 4]], dtype=np.float32),
+ )
+ obs = h5.create_group("obs")
+ obs.create_dataset("_index", data=np.array([b"c1", b"c2", b"c3"]))
+ obs.create_dataset("batch", data=np.array([0, 1, 0], dtype=np.int8))
+ obs_categories = obs.create_group("__categories")
+ obs_categories.create_dataset("batch", data=np.array([b"A", b"B"]))
+ state = obs.create_group("state")
+ state.create_dataset("codes", data=np.array([0, 1, 0], dtype=np.int8))
+ state.create_dataset("categories", data=np.array([b"cycling", b"resting"]))
+ n_genes = obs.create_group("nGenes")
+ n_genes.attrs["encoding-type"] = "nullable-integer"
+ n_genes.create_dataset("values", data=np.array([5, 7, 0], dtype=np.int64))
+ n_genes.create_dataset("mask", data=np.array([False, False, True]))
+
+ var = h5.create_group("var")
+ var.create_dataset("_index", data=np.array([b"f1", b"f2"]))
+ feature_names = var.create_group("gene_short_name")
+ feature_names.create_dataset("codes", data=np.array([1, 0]))
+ feature_names.create_dataset(
+ "categories",
+ data=np.array([b"Gene A", b"Gene B"]),
+ )
+ var.create_dataset("chromosome", data=np.array([b"1", b"2"]))
+ feature_type = var.create_group("feature_type")
+ feature_type.create_dataset("codes", data=np.array([0, 1], dtype=np.int8))
+ feature_type.create_dataset(
+ "categories",
+ data=np.array([b"Gene Expression", b"Antibody Capture"]),
+ )
+ highly_variable = var.create_group("highly_variable")
+ highly_variable.attrs["encoding-type"] = "nullable-boolean"
+ highly_variable.create_dataset("values", data=np.array([True, False]))
+ highly_variable.create_dataset("mask", data=np.array([False, False]))
+ reviewed = var.create_group("reviewed")
+ reviewed.attrs["encoding-type"] = "nullable-boolean"
+ reviewed.create_dataset("values", data=np.array([True, False]))
+ reviewed.create_dataset("mask", data=np.array([False, True]))
+ unreadable = var.create_group("per_cell_counts")
+ unreadable.attrs["encoding-type"] = "dataframe"
+ unreadable.create_dataset("column", data=np.array([1, 2]))
+
+ obsm = h5.create_group("obsm")
+ obsm.create_dataset(
+ "X_embed",
+ data=np.array([[1, 2], [3, 4], [5, 6]], dtype=np.float32),
+ )
+ obsm.create_dataset(
+ "bad_embed",
+ data=np.array([[1, 2], [3, 4]], dtype=np.float32),
+ )
+ _write_sparse_group(
+ obsm,
+ "sparse_embed",
+ np.array([[1, 0], [0, 2], [3, 0]], dtype=np.float32),
+ )
+
+ inspection = inspect_h5ad(str(file_name))
+ assert inspection.matrixKey == "X"
+ assert inspection.matrixEncoding == "dense"
+
+ reader = H5adReader(str(file_name))
+ messages: list[str] = []
+ sink = logger.add(
+ lambda message: messages.append(message.record["message"]), level="WARNING"
+ )
+ try:
+ assert reader.groupCodes == {"obs": 2, "var": 2, "obsm": 2, "X": 1}
+ np.testing.assert_array_equal(reader.cell_ids(), [b"c1", b"c2", b"c3"])
+ np.testing.assert_array_equal(reader.feat_ids(), [b"f1", b"f2"])
+ np.testing.assert_array_equal(reader.feat_names(), [b"Gene B", b"Gene A"])
+
+ cell_columns = dict(reader.get_cell_columns())
+ assert cell_columns.keys() == {
+ "batch",
+ "state",
+ "nGenes",
+ "X_embed1",
+ "X_embed2",
+ }
+ np.testing.assert_array_equal(cell_columns["batch"], [b"A", b"B", b"A"])
+ np.testing.assert_array_equal(
+ cell_columns["state"],
+ [b"cycling", b"resting", b"cycling"],
+ )
+ # Numeric nullable columns stay numeric by representing missing values
+ # as NaN.
+ assert cell_columns["nGenes"].dtype == np.dtype(np.float64)
+ np.testing.assert_allclose(
+ cell_columns["nGenes"],
+ np.array([5, 7, np.nan]),
+ equal_nan=True,
+ )
+ np.testing.assert_array_equal(cell_columns["X_embed2"], [2, 4, 6])
+
+ feature_columns = dict(reader.get_feat_columns())
+ assert feature_columns.keys() == {
+ "chromosome",
+ "feature_type",
+ "highly_variable",
+ "reviewed",
+ }
+ np.testing.assert_array_equal(feature_columns["chromosome"], [b"1", b"2"])
+ np.testing.assert_array_equal(
+ feature_columns["feature_type"],
+ [b"Gene Expression", b"Antibody Capture"],
+ )
+ # Nothing is masked here, so the source dtype is preserved.
+ assert feature_columns["highly_variable"].dtype == np.dtype(bool)
+ np.testing.assert_array_equal(feature_columns["highly_variable"], [True, False])
+ np.testing.assert_array_equal(
+ feature_columns["reviewed"],
+ np.array([True, None], dtype=object),
+ )
+ assert any(
+ "per_cell_counts" in message and "dataframe" in message
+ for message in messages
+ )
+ assert any("sparse_embed" in message for message in messages)
+ assert not any("__categories" in message for message in messages)
+ assert reader.feature_types("feature_type") == [
+ "Gene Expression",
+ "Antibody Capture",
+ ]
+
+ chunks = [chunk.toarray() for chunk in reader.consume(batch_size=2)]
+ assert len(chunks) == 2
+ np.testing.assert_array_equal(chunks[0], [[1, 0], [0, 2]])
+ np.testing.assert_array_equal(chunks[1], [[3, 4]])
+ finally:
+ logger.remove(sink)
+ reader.h5.close()
+
+
+def test_h5ad_reader_sizes_axes_from_nullable_columns(tmp_path):
+ import h5py
+
+ from scarf.readers import H5adReader
+
+ file_name = tmp_path / "nullable_only.h5ad"
+ with h5py.File(file_name, mode="w") as h5:
+ h5.create_dataset("X", data=np.ones((3, 2), dtype=np.float32))
+ obs = h5.create_group("obs")
+ n_genes = obs.create_group("nGenes")
+ n_genes.attrs["encoding-type"] = "nullable-integer"
+ n_genes.create_dataset("values", data=np.array([4, 5, 6], dtype=np.int64))
+ n_genes.create_dataset("mask", data=np.array([False, False, False]))
+ var = h5.create_group("var")
+ # A mask that does not line up with the values cannot be applied.
+ weight = var.create_group("weight")
+ weight.attrs["encoding-type"] = "nullable-integer"
+ weight.create_dataset("values", data=np.array([1, 2], dtype=np.int64))
+ weight.create_dataset("mask", data=np.array([False]))
+
+ reader = H5adReader(str(file_name))
+ try:
+ assert reader.nCells == 3
+ assert reader.nFeatures == 2
+ np.testing.assert_array_equal(reader.cell_ids(), ["cell_0", "cell_1", "cell_2"])
+ cell_columns = dict(reader.get_cell_columns())
+ assert cell_columns["nGenes"].dtype == np.dtype(np.int64)
+ np.testing.assert_array_equal(cell_columns["nGenes"], [4, 5, 6])
+ np.testing.assert_array_equal(
+ dict(reader.get_feat_columns())["weight"],
+ [1, 2],
+ )
+ finally:
+ reader.h5.close()
+
+
+def test_h5ad_reader_falls_back_without_metadata_groups(tmp_path):
+ import h5py
+
+ from scarf.readers import H5adReader
+
+ file_name = tmp_path / "sparse_without_metadata.h5ad"
+ with h5py.File(file_name, mode="w") as h5:
+ matrix = h5.create_group("X")
+ matrix.create_dataset("data", data=np.array([5, 8], dtype=np.int16))
+ matrix.create_dataset("indices", data=np.array([0, 1]))
+ matrix.create_dataset("indptr", data=np.array([0, 1, 2]))
+ matrix.create_dataset("shape", data=np.array([2, 2]))
+
+ reader = H5adReader(str(file_name))
+ try:
+ assert reader.nCells == reader.nFeatures == 2
+ np.testing.assert_array_equal(reader.cell_ids(), ["cell_0", "cell_1"])
+ np.testing.assert_array_equal(
+ reader.feat_ids(),
+ ["feature_0", "feature_1"],
+ )
+ np.testing.assert_array_equal(reader.feat_names(), reader.feat_ids())
+ assert list(reader.get_cell_columns()) == []
+ assert list(reader.get_feat_columns()) == []
+
+ chunks = list(reader.consume(batch_size=3))
+ assert len(chunks) == 1
+ np.testing.assert_array_equal(chunks[0].toarray(), [[5, 0], [0, 8]])
+ finally:
+ reader.h5.close()
+
+
+def test_loom_reader(loom_reader):
+ assert loom_reader.nCells == 298 == len(loom_reader.cell_ids())
+ assert loom_reader.nFeatures == 16892 == len(loom_reader.feature_names())
+
+
+def test_loom_reader_streams_metadata_and_counts(loom_reader):
+ cell_attributes = dict(loom_reader.get_cell_attrs())
+ assert cell_attributes.keys() == {"Area", "Cell_cluster"}
+ assert all(
+ values.shape == (loom_reader.nCells,) for values in cell_attributes.values()
+ )
+ assert dict(loom_reader.get_feature_attrs()) == {}
+
+ chunks = list(loom_reader.consume(batch_size=100))
+ assert [chunk.shape for chunk in chunks] == [
+ (100, loom_reader.nFeatures),
+ (100, loom_reader.nFeatures),
+ (98, loom_reader.nFeatures),
+ ]
+ assert sum(chunk.nnz for chunk in chunks) > 0
+
+
+def test_csv_reader_preserves_batches_skipped_columns_and_cell_metadata(tmp_path):
+ from scarf.readers import CSVReader
+
+ path = tmp_path / "counts.csv"
+ path.write_text("g1,g2,batch,drop\n1,2,a,10\n3,4,b,20\n5,6,c,30\n")
+ reader = CSVReader(
+ str(path),
+ skip_cols=["drop"],
+ cell_data_cols=["batch"],
+ batch_size=2,
+ )
+
+ assert reader.nCells == 3
+ assert reader.nFeatures == 2
+ np.testing.assert_array_equal(reader.cell_ids(), ["cell_0", "cell_1", "cell_2"])
+ np.testing.assert_array_equal(reader.feature_ids(), ["g1", "g2"])
+ batches = list(reader.consume())
+ np.testing.assert_array_equal(batches[0][0], [[1, 2], [3, 4]])
+ np.testing.assert_array_equal(batches[0][1], [["a"], ["b"]])
+ np.testing.assert_array_equal(batches[1][0], [[5, 6]])
+ np.testing.assert_array_equal(batches[1][1], [["c"]])
+
+
+def test_csv_reader_rejects_features_along_rows(tmp_path):
+ from scarf.readers import CSVReader
+
+ path = tmp_path / "counts.csv"
+ path.write_text("g1,g2\n1,2\n")
+ with pytest.raises(NotImplementedError, match="cells are along the rows"):
+ CSVReader(str(path), rows_are_cells=False)
+
+
+def test_csv_reader_rejects_non_mapping_pandas_kwargs(tmp_path):
+ from scarf.readers import CSVReader
+
+ path = tmp_path / "counts.csv"
+ path.write_text("g1,g2\n1,2\n", encoding="utf-8")
+ with pytest.raises(TypeError, match="pandas_kwargs must be a dictionary"):
+ CSVReader(str(path), pandas_kwargs=["header"])
diff --git a/tests/test_readers_contracts.py b/tests/test_readers_contracts.py
new file mode 100644
index 00000000..54f17c91
--- /dev/null
+++ b/tests/test_readers_contracts.py
@@ -0,0 +1,434 @@
+import gzip
+import io
+import inspect
+import pickle
+import subprocess
+import sys
+from typing import get_type_hints
+
+import h5py
+import numpy as np
+import pytest
+import zarr
+
+import scarf.readers as readers_module
+from scarf.readers import (
+ CSVReader,
+ CrDirReader,
+ CrH5Reader,
+ CrReader,
+ H5adInspectResult,
+ H5adReader,
+ LoomReader,
+ MtxCandidate,
+ MtxReader,
+ SeuratInspectResult,
+ SeuratReader,
+)
+from tests.signature_contracts import signature_digest
+
+
+_PUBLIC_CLASS_METHODS = {
+ CrReader: (
+ "__init__",
+ "consume",
+ "rename_assays",
+ "reclassify_features",
+ "feature_ids",
+ "feature_names",
+ "feature_types",
+ "cell_names",
+ ),
+ CrH5Reader: (
+ "__init__",
+ "cell_names",
+ "consume",
+ "close",
+ ),
+ CrDirReader: (
+ "__init__",
+ "read_header",
+ "process_batch",
+ "to_sparse",
+ "cell_names",
+ "rename_batches",
+ "consume",
+ ),
+ H5adReader: (
+ "__init__",
+ "from_inspect",
+ "cell_ids",
+ "feat_ids",
+ "feat_names",
+ "get_cell_columns",
+ "get_feat_columns",
+ "feature_types",
+ "assay_feature_slices",
+ "consume_dataset",
+ "consume_group",
+ "consume",
+ ),
+ LoomReader: (
+ "__init__",
+ "cell_names",
+ "cell_ids",
+ "get_cell_attrs",
+ "feature_names",
+ "feature_ids",
+ "get_feature_attrs",
+ "consume",
+ ),
+ CSVReader: (
+ "__init__",
+ "cell_ids",
+ "feature_ids",
+ "consume",
+ ),
+ MtxReader: (
+ "__init__",
+ "consume",
+ "close",
+ ),
+ SeuratReader: (
+ "__init__",
+ "close",
+ "get_assay",
+ "get_reduction",
+ "inspect",
+ ),
+}
+_PUBLIC_CLASS_SIGNATURE_DIGESTS = {
+ CrReader: "cfeac7ccf7bc316f1db1d9e177d6556b37a0169b3cbb2800a92e561b75f4fc4a",
+ CrH5Reader: "053373f2af2f2fc74a3e00cde9b067c5818aba92c09da3a9ac2e129566ca87b9",
+ CrDirReader: "51884c2390ad90fba1cdbd71808fc4dd98de548bd901e810c9caba6dbb9cf49a",
+ H5adReader: "aa4d19ec019beed92009edfd8b65a0a36fc6dbd66b19a887632862d770c4a3da",
+ LoomReader: "85c3ff965cb94a4fa201915b9d43890081e1f26e931327fcda6531bee4c3782a",
+ CSVReader: "8aa6c17c876afb62765584fc7ff64d2838c66ef53095da10d7198ca60ab83851",
+ MtxReader: "06376a32ff98ff0153ae1cc35f327509c88784ce027cb045bf9833b45dbccf2a",
+ SeuratReader: "f90ad28ab745692321245b3fd48e66273cbbe52795a9711f0cf17fb171820d3f",
+}
+_MODULE_SIGNATURE_DIGEST = (
+ "39b02f54928358aa8d3367a4f17f4a9b35c49bc8739e2dbb2f37038b6251f288"
+)
+
+
+def _write_minimal_cellranger_directory(path) -> None:
+ (path / "features.tsv").write_text(
+ "f1\tg1\tGene Expression\nf2\tg2\tGene Expression\n"
+ )
+ (path / "barcodes.tsv").write_text("c1\n")
+ (path / "matrix.mtx").write_text(
+ "%%MatrixMarket matrix coordinate integer general\n2 1 1\n1 1 1\n"
+ )
+
+
+def _write_mixed_cellranger_directory(path) -> None:
+ (path / "features.tsv").write_text(
+ "f1\tg1\tGene Expression\n"
+ "f2\th1\tAntibody Capture\n"
+ "f3\tg2\tGene Expression\n"
+ "f4\th2\tAntibody Capture\n"
+ "f5\ta1\tAntibody Capture\n"
+ )
+ (path / "barcodes.tsv").write_text("c1\n")
+ (path / "matrix.mtx").write_text(
+ "%%MatrixMarket matrix coordinate integer general\n5 1 3\n2 1 2\n4 1 4\n5 1 5\n"
+ )
+
+
+def test_readers_facade_surface_is_stable():
+ assert readers_module.__all__ == [
+ "CrH5Reader",
+ "CrDirReader",
+ "CrReader",
+ "H5adInspectResult",
+ "H5adReader",
+ "inspect_h5ad",
+ "MtxCandidate",
+ "MtxReader",
+ "inspect_mtx",
+ "SeuratInspectResult",
+ "SeuratReader",
+ "inspect_seurat",
+ "LoomReader",
+ "CSVReader",
+ ]
+ expected = {
+ "CSVReader",
+ "CrDirReader",
+ "CrH5Reader",
+ "CrReader",
+ "H5adInspectResult",
+ "H5adReader",
+ "LoomReader",
+ "MtxCandidate",
+ "MtxReader",
+ "SeuratInspectResult",
+ "SeuratReader",
+ "get_file_handle",
+ "inspect_h5ad",
+ "inspect_mtx",
+ "inspect_seurat",
+ "read_file",
+ }
+ assert expected.issubset(vars(readers_module))
+
+
+def test_readers_facade_loads_format_modules_lazily():
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ (
+ "import sys; import scarf.readers as readers; "
+ "assert 'scarf.readers.cellranger' not in sys.modules; "
+ "assert 'scarf.readers.csv' not in sys.modules; "
+ "assert 'scarf.readers.h5ad' not in sys.modules; "
+ "assert 'scarf.readers.loom' not in sys.modules; "
+ "assert 'scarf.readers.mtx' not in sys.modules; "
+ "assert 'scarf.readers.seurat' not in sys.modules; "
+ "assert 'h5py' not in sys.modules; "
+ "assert 'pandas' not in sys.modules; "
+ "assert 'scipy' not in sys.modules; "
+ "assert 'CSVReader' in dir(readers); "
+ "reader = readers.CSVReader; "
+ "assert reader.__module__ == 'scarf.readers'; "
+ "assert 'scarf.readers.csv' in sys.modules; "
+ "assert 'scarf.readers.cellranger' not in sys.modules; "
+ "assert 'scarf.readers.h5ad' not in sys.modules; "
+ "assert 'scarf.readers.loom' not in sys.modules; "
+ "assert 'scarf.readers.mtx' not in sys.modules; "
+ "assert 'scarf.readers.seurat' not in sys.modules; "
+ "assert 'pandas' in sys.modules; "
+ "assert 'h5py' not in sys.modules; "
+ "assert 'scipy' not in sys.modules"
+ ),
+ ],
+ check=True,
+ )
+
+
+def test_matrix_market_exports_load_together_lazily():
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ (
+ "import sys; import scarf.readers as readers; "
+ "assert 'scarf.readers.mtx' not in sys.modules; "
+ "reader = readers.MtxReader; "
+ "assert reader.__module__ == 'scarf.readers'; "
+ "assert readers.MtxCandidate.__module__ == 'scarf.readers'; "
+ "assert readers.inspect_mtx.__module__ == 'scarf.readers'; "
+ "assert 'scarf.readers.mtx' in sys.modules; "
+ "assert 'scarf.readers.h5ad' not in sys.modules; "
+ "assert 'scarf.readers.loom' not in sys.modules"
+ ),
+ ],
+ check=True,
+ )
+
+
+def test_seurat_exports_load_together_without_loading_the_writer():
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ (
+ "import sys; import scarf.readers as readers; "
+ "assert 'scarf.readers.seurat' not in sys.modules; "
+ "reader = readers.SeuratReader; "
+ "assert reader.__module__ == 'scarf.readers'; "
+ "assert readers.SeuratInspectResult.__module__ == 'scarf.readers'; "
+ "assert readers.inspect_seurat.__module__ == 'scarf.readers'; "
+ "assert 'scarf.readers.seurat' in sys.modules; "
+ "assert 'scarf.writers.seurat' not in sys.modules"
+ ),
+ ],
+ check=True,
+ )
+
+
+def test_seurat_reader_facade_objects_resolve_annotations_and_pickle():
+ for name in ("__init__", "get_assay", "get_reduction", "inspect"):
+ assert get_type_hints(getattr(SeuratReader, name))
+ for value in (
+ SeuratReader,
+ SeuratInspectResult,
+ readers_module.inspect_seurat,
+ ):
+ assert pickle.loads(pickle.dumps(value)) is value
+
+
+def test_reader_class_and_method_signatures_are_stable():
+ for cls, names in _PUBLIC_CLASS_METHODS.items():
+ methods = {name: getattr(cls, name) for name in names}
+ assert signature_digest(methods) == _PUBLIC_CLASS_SIGNATURE_DIGESTS[cls]
+
+
+def test_reader_module_function_signatures_are_stable():
+ methods = {
+ name: getattr(readers_module, name)
+ for name in (
+ "get_file_handle",
+ "inspect_h5ad",
+ "inspect_mtx",
+ "inspect_seurat",
+ "read_file",
+ )
+ }
+ assert signature_digest(methods) == _MODULE_SIGNATURE_DIGEST
+
+
+def test_reader_public_metadata_remains_on_facade():
+ for cls, names in _PUBLIC_CLASS_METHODS.items():
+ assert cls.__module__ == "scarf.readers"
+ for name in names:
+ descriptor = inspect.getattr_static(cls, name)
+ method = (
+ descriptor.__func__
+ if isinstance(descriptor, classmethod | staticmethod)
+ else descriptor
+ )
+ assert method.__module__ == "scarf.readers"
+ assert method.__qualname__.startswith(f"{cls.__name__}.")
+
+ assert H5adInspectResult.__module__ == "scarf.readers"
+ assert MtxCandidate.__module__ == "scarf.readers"
+ assert SeuratInspectResult.__module__ == "scarf.readers"
+ for name in (
+ "get_file_handle",
+ "inspect_h5ad",
+ "inspect_mtx",
+ "inspect_seurat",
+ "read_file",
+ ):
+ assert getattr(readers_module, name).__module__ == "scarf.readers"
+
+
+def test_cellranger_reader_hierarchy_and_abstract_contracts_are_stable():
+ assert inspect.isabstract(CrReader)
+ assert issubclass(CrH5Reader, CrReader)
+ assert issubclass(CrDirReader, CrReader)
+ for name in ("_handle_version", "_read_dataset", "consume"):
+ assert getattr(CrReader, name).__isabstractmethod__
+
+
+def test_reader_text_helpers_support_plain_gzip_and_missing_files(tmp_path):
+ plain = tmp_path / "plain.txt"
+ compressed = tmp_path / "compressed.txt.gz"
+ plain.write_text("one \n two\n")
+ with gzip.open(compressed, mode="wt") as handle:
+ handle.write("three\nfour \n")
+
+ assert list(readers_module.read_file(str(plain))) == ["one", " two"]
+ assert list(readers_module.read_file(str(compressed))) == ["three", "four"]
+
+ missing = tmp_path / "missing.txt"
+ try:
+ readers_module.get_file_handle(str(missing))
+ except FileNotFoundError as error:
+ assert str(error) == f"ERROR: FILE NOT FOUND: {missing}"
+ else:
+ raise AssertionError("Missing reader input did not raise FileNotFoundError")
+
+
+def test_get_file_handle_facade_remains_patchable_by_read_file(monkeypatch):
+ handle = io.StringIO("one\ntwo\n")
+ monkeypatch.setattr(
+ readers_module,
+ "get_file_handle",
+ lambda filename: handle,
+ )
+
+ assert list(readers_module.read_file("virtual.txt")) == ["one", "two"]
+ assert handle.closed
+
+
+def test_crreader_reclassifies_noncontiguous_features_atomically(tmp_path):
+ _write_mixed_cellranger_directory(tmp_path)
+ reader = CrDirReader(str(tmp_path))
+
+ reader.reclassify_features([1, 3], "HTO")
+
+ assert reader.feature_types() == [
+ "Gene Expression",
+ "HTO",
+ "Gene Expression",
+ "HTO",
+ "Antibody Capture",
+ ]
+ assert list(reader.assayFeats.columns) == ["RNA", "HTO", "RNA", "HTO", "ADT"]
+ assert reader.feature_names("HTO") == ["h1", "h2"]
+ assert reader.feature_names("ADT") == ["a1"]
+
+ before = reader.assayFeats.copy()
+ reader.reclassify_features([1, 3], "HTO")
+ assert reader.assayFeats.equals(before)
+
+ with pytest.raises(ValueError, match="conflicting"):
+ reader.reclassify_features([1], "RNA", require_previous=None)
+ assert reader.feature_types()[1] == "HTO"
+
+
+def test_crreader_reclassification_validates_before_mutation(tmp_path):
+ _write_mixed_cellranger_directory(tmp_path)
+ reader = CrDirReader(str(tmp_path))
+ original_types = reader.feature_types()
+ original_table = reader.assayFeats.copy()
+
+ with pytest.raises(ValueError, match="currently have type"):
+ reader.reclassify_features([1, 2], "HTO")
+ with pytest.raises(ValueError, match="unique"):
+ reader.reclassify_features([1, 1], "HTO")
+ with pytest.raises(IndexError, match="out-of-range"):
+ reader.reclassify_features([5], "HTO")
+
+ assert reader.feature_types() == original_types
+ assert reader.assayFeats.equals(original_table)
+
+
+def test_crreader_reclassification_locks_when_writer_captures_schema(tmp_path):
+ from scarf.writers import CrToZarr
+
+ source = tmp_path / "source"
+ source.mkdir()
+ _write_mixed_cellranger_directory(source)
+ reader = CrDirReader(str(source))
+ reader.reclassify_features([1, 3], "HTO")
+
+ destination = tmp_path / "out.zarr"
+ writer = CrToZarr(reader, str(destination))
+ writer.dump()
+ root = zarr.open_group(str(destination), mode="r")
+
+ assert set(root.group_keys()) >= {"RNA", "HTO", "ADT"}
+ np.testing.assert_array_equal(
+ np.asarray(root["HTO/featureData/ids"][:]).astype(str),
+ ["f2", "f4"],
+ )
+ np.testing.assert_array_equal(
+ np.asarray(root["ADT/featureData/ids"][:]).astype(str),
+ ["f5"],
+ )
+ np.testing.assert_array_equal(root["HTO/counts"][:], [[2, 4]])
+ np.testing.assert_array_equal(root["ADT/counts"][:], [[5]])
+ with pytest.raises(RuntimeError, match="captures the schema"):
+ reader.reclassify_features([4], "HTO")
+
+
+def test_loom_reader_preserves_cell_feature_orientation(tmp_path):
+ values = np.array([[1, 2, 3], [4, 5, 6]], dtype=np.uint16)
+ path = tmp_path / "orientation.loom"
+ with h5py.File(path, mode="w") as handle:
+ handle.create_dataset("matrix", data=values)
+ handle.create_group("col_attrs")
+ handle.create_group("row_attrs")
+
+ reader = LoomReader(str(path))
+ try:
+ chunks = [chunk.toarray() for chunk in reader.consume(batch_size=2)]
+ finally:
+ reader.h5.close()
+
+ np.testing.assert_array_equal(np.concatenate(chunks), values.T)
diff --git a/tests/test_renorm_subset.py b/tests/test_renorm_subset.py
new file mode 100644
index 00000000..f53ede78
--- /dev/null
+++ b/tests/test_renorm_subset.py
@@ -0,0 +1,210 @@
+import numpy as np
+
+from scarf.assay import RNAassay
+from scarf.writers import write_renorm_subset_to_zarr
+
+
+def _subset_indices(rna):
+ cell_idx = np.arange(rna.cells.N)
+ feat_idx = np.array([0, 1, 3])
+ return cell_idx, feat_idx
+
+
+def _reference_renorm_subset(rna, cell_idx, feat_idx, log_transform=False):
+ raw = rna.rawData[cell_idx, :][:, feat_idx].compute()
+ row_sum = raw.sum(axis=1)
+ row_sum[row_sum == 0] = 1
+ out = rna.sf * raw / row_sum[:, np.newaxis]
+ if log_transform:
+ out = np.log1p(out)
+ return out.astype(np.float32)
+
+
+def test_write_renorm_subset_matches_reference(toy_crdir_ds):
+ rna = toy_crdir_ds.RNA
+ cell_idx, feat_idx = _subset_indices(rna)
+ loc = "fused_normed"
+
+ rna.z.create_group(loc, overwrite=True)
+ write_renorm_subset_to_zarr(
+ rna, cell_idx, feat_idx, rna.z, f"{loc}/data", rna.nthreads
+ )
+ expected = _reference_renorm_subset(rna, cell_idx, feat_idx)
+ written = rna.z[f"{loc}/data"][:]
+ np.testing.assert_allclose(written, expected, rtol=1e-5)
+
+
+def test_write_renorm_subset_log_transform(toy_crdir_ds):
+ rna = toy_crdir_ds.RNA
+ cell_idx, feat_idx = _subset_indices(rna)
+ loc = "fused_normed_log"
+
+ rna.z.create_group(loc, overwrite=True)
+ write_renorm_subset_to_zarr(
+ rna,
+ cell_idx,
+ feat_idx,
+ rna.z,
+ f"{loc}/data",
+ rna.nthreads,
+ log_transform=True,
+ )
+ expected = _reference_renorm_subset(rna, cell_idx, feat_idx, log_transform=True)
+ written = rna.z[f"{loc}/data"][:]
+ np.testing.assert_allclose(written, expected, rtol=1e-5)
+
+
+def test_save_normalized_data_renorm_uses_fused_path(toy_crdir_ds, monkeypatch):
+ rna = toy_crdir_ds.RNA
+ called = {"normed": 0}
+ orig_normed = RNAassay.normed
+
+ def fake_normed(self, *args, **kwargs):
+ called["normed"] += 1
+ return orig_normed(self, *args, **kwargs)
+
+ monkeypatch.setattr(RNAassay, "normed", fake_normed)
+ monkeypatch.setattr(
+ rna,
+ "_get_cell_feat_idx",
+ lambda cell_key, feat_key: _subset_indices(rna),
+ )
+
+ rna.save_normalized_data(
+ cell_key="I",
+ feat_key="I",
+ location="normed_fused_test",
+ log_transform=False,
+ renormalize_subset=True,
+ update_keys=False,
+ )
+ assert called["normed"] == 0
+ cell_idx, feat_idx = _subset_indices(rna)
+ expected = _reference_renorm_subset(rna, cell_idx, feat_idx)
+ np.testing.assert_allclose(rna.z["normed_fused_test/data"][:], expected, rtol=1e-5)
+
+
+def test_save_normalized_data_without_renorm_still_uses_normed(
+ toy_crdir_ds, monkeypatch
+):
+ rna = toy_crdir_ds.RNA
+ called = {"normed": 0, "fused": 0}
+ orig_normed = RNAassay.normed
+
+ def fake_normed(self, *args, **kwargs):
+ called["normed"] += 1
+ return orig_normed(self, *args, **kwargs)
+
+ def fake_fused(*args, **kwargs):
+ called["fused"] += 1
+ return write_renorm_subset_to_zarr(*args, **kwargs)
+
+ monkeypatch.setattr(RNAassay, "normed", fake_normed)
+ monkeypatch.setattr(
+ "scarf.storage.materialize.write_renorm_subset_to_zarr",
+ fake_fused,
+ )
+ monkeypatch.setattr(
+ rna,
+ "_get_cell_feat_idx",
+ lambda cell_key, feat_key: _subset_indices(rna),
+ )
+
+ rna.save_normalized_data(
+ cell_key="I",
+ feat_key="I",
+ location="normed_standard_test",
+ log_transform=False,
+ renormalize_subset=False,
+ update_keys=False,
+ )
+ assert called["normed"] == 1
+ assert called["fused"] == 0
+ assert (
+ "normalization_identity"
+ not in rna.z["normed_standard_test"].attrs["subset_params"]
+ )
+
+
+def test_save_normalized_data_renorm_cache_hit(toy_crdir_ds, monkeypatch):
+ from scarf.storage.materialize import (
+ write_renorm_subset_to_zarr as materialize_renorm_subset,
+ )
+
+ rna = toy_crdir_ds.RNA
+ called = {"fused": 0}
+ orig_fused = materialize_renorm_subset
+
+ def counting_fused(*args, **kwargs):
+ called["fused"] += 1
+ return orig_fused(*args, **kwargs)
+
+ monkeypatch.setattr(
+ "scarf.storage.materialize.write_renorm_subset_to_zarr",
+ counting_fused,
+ )
+ monkeypatch.setattr(
+ rna,
+ "_get_cell_feat_idx",
+ lambda cell_key, feat_key: _subset_indices(rna),
+ )
+
+ kwargs = dict(
+ cell_key="I",
+ feat_key="I",
+ location="normed_cache_test",
+ log_transform=False,
+ renormalize_subset=True,
+ update_keys=False,
+ )
+ rna.save_normalized_data(**kwargs)
+ rna.save_normalized_data(**kwargs)
+ assert called["fused"] == 1
+
+
+def test_atac_legacy_cache_rejects_missing_normalizer_identity(
+ atac_datastore,
+ monkeypatch,
+):
+ assay = atac_datastore.ATAC
+ cell_idx = np.arange(8, dtype=np.int64)
+ feat_idx = assay.feats.active_index("I")[:4]
+ monkeypatch.setattr(
+ assay,
+ "_get_cell_feat_idx",
+ lambda cell_key, feat_key: (cell_idx, feat_idx),
+ )
+
+ original = assay.normed
+ calls = 0
+
+ def counting_normed(*args, **kwargs):
+ nonlocal calls
+ calls += 1
+ return original(*args, **kwargs)
+
+ monkeypatch.setattr(assay, "normed", counting_normed)
+ kwargs = dict(
+ cell_key="I",
+ feat_key="I",
+ location="atac_normalizer_identity_cache_test",
+ log_transform=False,
+ renormalize_subset=False,
+ update_keys=False,
+ )
+
+ assay.save_normalized_data(**kwargs)
+ group = assay.z["atac_normalizer_identity_cache_test"]
+ current_params = dict(group.attrs["subset_params"])
+ assert current_params["normalization_identity"] == getattr(
+ assay.normMethod,
+ "artifact_identity",
+ )
+
+ legacy_params = dict(current_params)
+ del legacy_params["normalization_identity"]
+ group.attrs["subset_params"] = legacy_params
+ assay.save_normalized_data(**kwargs)
+ assay.save_normalized_data(**kwargs)
+
+ assert calls == 2
diff --git a/tests/test_repack_zarr.py b/tests/test_repack_zarr.py
new file mode 100644
index 00000000..dd34a647
--- /dev/null
+++ b/tests/test_repack_zarr.py
@@ -0,0 +1,172 @@
+import numpy as np
+import pytest
+import zarr
+from zarr.codecs import ZstdCodec
+
+from scarf.tools.repack_zarr import repack_store
+
+
+def test_repack_store_round_trip(toy_crdir_writer, tmp_path):
+ output = tmp_path / "repacked.zarr"
+ repack_store(toy_crdir_writer, str(output), profile="fast_local", shard_counts=True)
+
+ src = zarr.open_group(toy_crdir_writer, mode="r")
+ dst = zarr.open_group(str(output), mode="r")
+
+ assert set(src.keys()) == set(dst.keys())
+ assay_names = [name for name in src.keys() if src[name].attrs.get("is_assay")]
+ for assay_name in assay_names:
+ src_assay = src[assay_name]
+ dst_assay = dst[assay_name]
+ assert src_assay.attrs.get("is_assay") is True
+ assert "counts" in dst_assay
+ assert src_assay["counts"].shape == dst_assay["counts"].shape
+ assert (src_assay["counts"][...] == dst_assay["counts"][...]).all()
+ assert dst_assay["countsT"].attrs["complete"] is True
+ np.testing.assert_array_equal(
+ dst_assay["countsT"][:],
+ np.asarray(dst_assay["counts"][:]).T,
+ )
+
+
+def test_repack_store_without_sharding(toy_crdir_writer, tmp_path):
+ output = tmp_path / "repacked_plain.zarr"
+ repack_store(
+ toy_crdir_writer,
+ str(output),
+ profile="fast_local",
+ shard_counts=False,
+ )
+ dst = zarr.open_group(str(output), mode="r")
+ assert "RNA" in dst
+ assert "counts" in dst["RNA"]
+
+
+def test_repack_v2_without_counts_t_builds_complete_transpose(tmp_path):
+ source = tmp_path / "source_v2.zarr"
+ output = tmp_path / "output_v3.zarr"
+ root = zarr.open_group(str(source), mode="w", zarr_format=2)
+ assay = root.create_group("RNA")
+ assay.attrs["is_assay"] = True
+ values = np.arange(12, dtype=np.uint32).reshape(3, 4)
+ assay.create_array("counts", data=values, chunks=(2, 2))
+
+ repack_store(str(source), str(output), shard_counts=True)
+
+ result = zarr.open_group(str(output), mode="r")
+ assert result.metadata.zarr_format == 3
+ assert result["RNA/countsT"].attrs["complete"] is True
+ np.testing.assert_array_equal(result["RNA/counts"][:], values)
+ np.testing.assert_array_equal(result["RNA/countsT"][:], values.T)
+
+
+def test_repack_rebuilds_incorrect_source_counts_t(tmp_path):
+ source = tmp_path / "source_wrong.zarr"
+ output = tmp_path / "output_fixed.zarr"
+ root = zarr.open_group(str(source), mode="w")
+ assay = root.create_group("RNA")
+ assay.attrs["is_assay"] = True
+ values = np.arange(15, dtype=np.uint32).reshape(5, 3)
+ assay.create_array("counts", data=values, chunks=(2, 3))
+ stale = assay.create_array(
+ "countsT",
+ data=np.zeros((3, 5), dtype=np.uint32),
+ )
+ stale.attrs["complete"] = True
+
+ repack_store(str(source), str(output), shard_counts=True)
+
+ result = zarr.open_group(str(output), mode="r")
+ np.testing.assert_array_equal(result["RNA/countsT"][:], values.T)
+ assert result["RNA/countsT"].attrs["complete"] is True
+
+
+def test_repack_workspace_counts_uses_requested_profile(
+ tmp_path,
+):
+ source = tmp_path / "source_workspace.zarr"
+ output = tmp_path / "output_workspace.zarr"
+ root = zarr.open_group(str(source), mode="w")
+ metadata = root.create_group("workspace/RNA")
+ metadata.attrs["is_assay"] = True
+ counts_group = root.create_group("matrices/RNA")
+ values = np.arange(8, dtype=np.uint32).reshape(4, 2)
+ counts_group.create_array("counts", data=values, chunks=(2, 2))
+
+ repack_store(
+ str(source),
+ str(output),
+ profile="cloud",
+ shard_counts=True,
+ )
+
+ result = zarr.open_group(str(output), mode="r")
+ assay = result["matrices/RNA"]
+ counts = assay["counts"]
+ counts_t = assay["countsT"]
+ np.testing.assert_array_equal(counts_t[:], values.T)
+ assert counts_t.attrs["complete"] is True
+ assert assay.attrs["scarf:zarr_spec"]["profile"] == "cloud"
+ assert isinstance(counts.compressors[0], ZstdCodec)
+ assert isinstance(counts_t.compressors[0], ZstdCodec)
+
+
+def test_repack_rejects_source_destination_alias_before_overwrite(tmp_path):
+ source = tmp_path / "source.zarr"
+ root = zarr.open_group(str(source), mode="w")
+ root.create_array("sentinel", data=np.array([1, 2, 3]))
+ equivalent_path = source / ".." / source.name
+
+ with pytest.raises(ValueError, match="different stores"):
+ repack_store(str(source), str(equivalent_path))
+
+ reopened = zarr.open_group(str(source), mode="r")
+ np.testing.assert_array_equal(reopened["sentinel"][:], [1, 2, 3])
+
+
+@pytest.mark.parametrize("destination_relation", ["child", "parent"])
+def test_repack_rejects_overlapping_local_paths_before_overwrite(
+ tmp_path,
+ destination_relation,
+):
+ if destination_relation == "child":
+ source = tmp_path / "source.zarr"
+ output = source / "nested.zarr"
+ else:
+ output = tmp_path / "container.zarr"
+ source = output / "source.zarr"
+
+ root = zarr.open_group(str(source), mode="w")
+ root.create_array("sentinel", data=np.array([1, 2, 3]))
+
+ with pytest.raises(ValueError, match="must not overlap"):
+ repack_store(str(source), str(output))
+
+ reopened = zarr.open_group(str(source), mode="r")
+ np.testing.assert_array_equal(reopened["sentinel"][:], [1, 2, 3])
+
+
+@pytest.mark.parametrize(
+ ("source", "output"),
+ [
+ ("s3://bucket/source.zarr", "s3://bucket/source.zarr/nested.zarr"),
+ ("s3://bucket/container.zarr/source.zarr", "s3://bucket/container.zarr"),
+ ],
+)
+def test_repack_rejects_overlapping_uri_paths_before_open(
+ source,
+ output,
+ monkeypatch,
+):
+ import scarf.tools.repack_zarr as repack_module
+
+ monkeypatch.setattr(
+ repack_module,
+ "open_store",
+ lambda *_args, **_kwargs: pytest.fail(
+ "overlap must be rejected before opening either store"
+ ),
+ )
+
+ with pytest.raises(ValueError, match="must not overlap"):
+ repack_store(source, output)
diff --git a/tests/test_sample_aware_mad_filtering.py b/tests/test_sample_aware_mad_filtering.py
new file mode 100644
index 00000000..96935507
--- /dev/null
+++ b/tests/test_sample_aware_mad_filtering.py
@@ -0,0 +1,498 @@
+import numpy as np
+import pytest
+from scarf.storage.artifacts import ArtifactRef
+from scarf.utils import logger
+
+from scarf.quality_control.filtering import (
+ _mad_bounds,
+ _metric_policy,
+ _sample_aware_mad_mask,
+ gaussian_quantile_bounds,
+)
+
+
+def test_mad_bounds_matches_hand_computed_example():
+ values = np.array([1.0, 2.0, 3.0, 4.0, 100.0])
+ median = 3.0
+ scaled_mad = 1.4826 * np.median(np.abs(values - median))
+ low, high, reported_mad = _mad_bounds(values, n_mads=3.0)
+
+ assert reported_mad == pytest.approx(scaled_mad)
+ assert low == pytest.approx(median - 3.0 * scaled_mad)
+ assert high == pytest.approx(median + 3.0 * scaled_mad)
+
+
+def test_metric_policy_defaults_and_custom_attrs():
+ assert _metric_policy("RNA_nCounts") == {
+ "transform": "log1p",
+ "bound_direction": "two_sided",
+ }
+ assert _metric_policy("RNA_percentMito") == {
+ "transform": "identity",
+ "bound_direction": "upper",
+ }
+ assert _metric_policy("custom_score") == {
+ "transform": "identity",
+ "bound_direction": "two_sided",
+ }
+
+
+@pytest.mark.parametrize("attr", ["RNA_nCounts", "RNA_nFeatures"])
+def test_sample_aware_mask_rejects_negative_count_values(attr):
+ with pytest.raises(ValueError, match="non-negative before log1p"):
+ _sample_aware_mad_mask(
+ values_by_attr={attr: np.array([1.0, 2.0, -2.0])},
+ sample_labels=np.array(["A", "A", "A"]),
+ active=np.ones(3, dtype=bool),
+ n_mads=3.0,
+ min_cells_per_sample=2,
+ attrs=[attr],
+ )
+
+
+def test_sample_aware_mask_rejects_mixed_label_types_without_collision():
+ with pytest.raises(ValueError, match="one consistent label type"):
+ _sample_aware_mad_mask(
+ values_by_attr={"score": np.arange(8, dtype=float)},
+ sample_labels=np.array([1] * 4 + ["1"] * 4, dtype=object),
+ active=np.ones(8, dtype=bool),
+ n_mads=3.0,
+ min_cells_per_sample=2,
+ attrs=["score"],
+ )
+
+
+def test_sample_aware_mask_rejects_blank_bytes_labels():
+ with pytest.raises(ValueError, match="missing labels"):
+ _sample_aware_mad_mask(
+ values_by_attr={"score": np.arange(4, dtype=float)},
+ sample_labels=np.array([b"A", b"A", b" ", b" "]),
+ active=np.ones(4, dtype=bool),
+ n_mads=3.0,
+ min_cells_per_sample=2,
+ attrs=["score"],
+ )
+
+
+def test_sample_aware_mask_isolates_outliers_per_sample():
+ # Sample A and B have different depths. A severe outlier in B must not
+ # change A's bounds, and must be removed only from B.
+ sample_a = np.linspace(9.0, 11.0, 20)
+ sample_b = np.concatenate([np.linspace(99.0, 101.0, 19), [1000.0]])
+ n_counts = np.concatenate([sample_a, sample_b])
+ labels = np.array(["A"] * 20 + ["B"] * 20)
+ active = np.ones(40, dtype=bool)
+
+ keep, provenance = _sample_aware_mad_mask(
+ values_by_attr={"RNA_nCounts": n_counts},
+ sample_labels=labels,
+ active=active,
+ n_mads=3.0,
+ min_cells_per_sample=5,
+ attrs=["RNA_nCounts"],
+ )
+
+ assert bool(keep[:20].all())
+ assert bool(keep[20:39].all())
+ assert not bool(keep[-1])
+ a_high = provenance["resolved_bounds"]["A"]["RNA_nCounts"]["high"]
+ b_high = provenance["resolved_bounds"]["B"]["RNA_nCounts"]["high"]
+ assert a_high < 50
+ assert b_high > 50
+ assert a_high < b_high
+
+
+def test_sample_aware_mask_uses_log_counts_and_upper_only_percent():
+ n_counts = np.expm1(
+ np.array([2.0, 2.1, 1.9, 2.05, 2.02, 2.01, 1.98, 2.03, 1.99, 2.04, 8.0])
+ )
+ percent_mito = np.array(
+ [1.0, 1.1, 0.9, 1.05, 1.02, 1.01, 0.98, 1.03, 0.99, 1.04, 40.0]
+ )
+ labels = np.array(["S"] * 11)
+ active = np.ones(11, dtype=bool)
+
+ keep, provenance = _sample_aware_mad_mask(
+ values_by_attr={
+ "RNA_nCounts": n_counts,
+ "RNA_percentMito": percent_mito,
+ },
+ sample_labels=labels,
+ active=active,
+ n_mads=3.0,
+ min_cells_per_sample=5,
+ attrs=["RNA_nCounts", "RNA_percentMito"],
+ )
+
+ count_bounds = provenance["resolved_bounds"]["S"]["RNA_nCounts"]
+ mito_bounds = provenance["resolved_bounds"]["S"]["RNA_percentMito"]
+ assert count_bounds["transform"] == "log1p"
+ assert count_bounds["bound_direction"] == "two_sided"
+ assert count_bounds["low"] is not None
+ assert mito_bounds["transform"] == "identity"
+ assert mito_bounds["bound_direction"] == "upper"
+ assert mito_bounds["low"] is None
+ assert not bool(keep[-1])
+ # An unusually low mito percentage must not be filtered by upper-only bounds.
+ low_mito = percent_mito.copy()
+ low_mito[0] = 0.0
+ keep_low, _ = _sample_aware_mad_mask(
+ values_by_attr={
+ "RNA_nCounts": n_counts,
+ "RNA_percentMito": low_mito,
+ },
+ sample_labels=labels,
+ active=active,
+ n_mads=3.0,
+ min_cells_per_sample=5,
+ attrs=["RNA_percentMito"],
+ )
+ assert bool(keep_low[0])
+
+
+def test_sample_aware_mask_skips_small_and_zero_mad_groups():
+ values = np.array([1.0, 2.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0])
+ labels = np.array(["tiny"] * 3 + ["flat"] * 5)
+ active = np.ones(8, dtype=bool)
+
+ keep, provenance = _sample_aware_mad_mask(
+ values_by_attr={"custom_score": values},
+ sample_labels=labels,
+ active=active,
+ n_mads=3.0,
+ min_cells_per_sample=5,
+ attrs=["custom_score"],
+ )
+
+ assert bool(keep.all())
+ assert provenance["skip_reasons"]["tiny"] == "insufficient_cells"
+ assert (
+ provenance["resolved_bounds"]["flat"]["custom_score"]["skip_reason"]
+ == "zero_mad"
+ )
+ assert any("tiny" in message for message in provenance["warnings"])
+ assert any("zero MAD" in message for message in provenance["warnings"])
+
+
+def test_auto_filter_cells_without_sample_column_matches_gaussian(
+ datastore_ephemeral,
+):
+ attrs = ["RNA_nCounts", "RNA_nFeatures"]
+ expected_bounds = {
+ attr: dict(
+ zip(
+ ("low", "high"),
+ gaussian_quantile_bounds(
+ np.asarray(datastore_ephemeral.cells.fetch_all(attr)),
+ 0.01,
+ 0.99,
+ ),
+ strict=True,
+ )
+ )
+ for attr in attrs
+ }
+
+ before = np.asarray(datastore_ephemeral.cells.fetch_all("I"), dtype=bool).copy()
+ datastore_ephemeral.auto_filter_cells(attrs=attrs, show_qc_plots=False)
+ after = np.asarray(datastore_ephemeral.cells.fetch_all("I"), dtype=bool)
+
+ cell_ref = ArtifactRef.from_dict(
+ datastore_ephemeral.zw["cellData"]["I"].attrs["source_artifact"]
+ )
+ status = datastore_ephemeral.inspect_artifact(cell_ref)
+ assert status.operation == "auto_filter_cells"
+ assert "sample_column" not in status.parameters
+ assert status.parameters["resolved_bounds"] == {
+ attr: {
+ "low": float(expected_bounds[attr]["low"]),
+ "high": float(expected_bounds[attr]["high"]),
+ }
+ for attr in attrs
+ }
+ assert after.sum() < before.sum()
+
+
+def test_auto_filter_cells_sample_column_raises_on_conflicts(
+ datastore_ephemeral,
+):
+ n = datastore_ephemeral.cells.N
+ datastore_ephemeral.cells.insert(
+ "sample_id",
+ np.array(["A"] * (n // 2) + ["B"] * (n - n // 2)),
+ overwrite=True,
+ )
+
+ with pytest.raises(ValueError, match="min_p and max_p"):
+ datastore_ephemeral.auto_filter_cells(
+ attrs=["RNA_nCounts"],
+ sample_column="sample_id",
+ min_p=0.05,
+ show_qc_plots=False,
+ )
+
+ with pytest.raises(ValueError, match="not found"):
+ datastore_ephemeral.auto_filter_cells(
+ attrs=["RNA_nCounts"],
+ sample_column="missing_sample",
+ show_qc_plots=False,
+ )
+
+
+@pytest.mark.parametrize("n_mads", [np.nan, np.inf, -np.inf])
+def test_auto_filter_cells_rejects_nonfinite_n_mads_without_mutating_selection(
+ datastore_ephemeral,
+ n_mads,
+):
+ n = datastore_ephemeral.cells.N
+ datastore_ephemeral.cells.insert(
+ "sample_id",
+ np.array(["A"] * (n // 2) + ["B"] * (n - n // 2)),
+ overwrite=True,
+ )
+ before = np.asarray(
+ datastore_ephemeral.cells.fetch_all("I"),
+ dtype=bool,
+ ).copy()
+
+ with pytest.raises(ValueError, match="finite"):
+ datastore_ephemeral.auto_filter_cells(
+ attrs=["RNA_nCounts"],
+ sample_column="sample_id",
+ n_mads=n_mads,
+ min_cells_per_sample=2,
+ show_qc_plots=False,
+ )
+
+ np.testing.assert_array_equal(
+ datastore_ephemeral.cells.fetch_all("I"),
+ before,
+ )
+
+
+def test_auto_filter_cells_rejects_overflowing_finite_n_mads_without_mutation(
+ datastore_ephemeral,
+):
+ n = datastore_ephemeral.cells.N
+ datastore_ephemeral.cells.insert(
+ "sample_id",
+ np.array(["A"] * n),
+ overwrite=True,
+ )
+ selection = datastore_ephemeral.zw["cellData"]["I"]
+ selection_before = np.asarray(selection[:], dtype=bool).copy()
+ provenance_before = dict(selection.attrs)
+
+ with pytest.raises(ValueError, match="non-finite"):
+ datastore_ephemeral.auto_filter_cells(
+ attrs=["RNA_nCounts"],
+ sample_column="sample_id",
+ n_mads=np.finfo(float).max,
+ min_cells_per_sample=2,
+ show_qc_plots=False,
+ )
+
+ np.testing.assert_array_equal(selection[:], selection_before)
+ assert dict(selection.attrs) == provenance_before
+
+
+def test_auto_filter_cells_validates_provenance_before_selection_mutation(
+ datastore_ephemeral,
+ monkeypatch,
+):
+ import scarf.datastore._operations.quality_control as qc_operations
+
+ n = datastore_ephemeral.cells.N
+ datastore_ephemeral.cells.insert(
+ "sample_id",
+ np.array(["A"] * n),
+ overwrite=True,
+ )
+ selection = datastore_ephemeral.zw["cellData"]["I"]
+ selection_before = np.asarray(selection[:], dtype=bool).copy()
+ provenance_before = dict(selection.attrs)
+
+ def malformed_provenance(**kwargs):
+ del kwargs
+ return np.ones(n, dtype=bool), {
+ "mad_scale": 1.4826,
+ "metric_policies": {"RNA_nCounts": {"transform": object()}},
+ "sample_sizes": {"A": n},
+ "skip_reasons": {},
+ "resolved_bounds": {},
+ "warnings": [],
+ }
+
+ monkeypatch.setattr(
+ qc_operations,
+ "_sample_aware_mad_mask",
+ malformed_provenance,
+ )
+ with pytest.raises(TypeError, match="Unsupported provenance value"):
+ datastore_ephemeral.auto_filter_cells(
+ attrs=["RNA_nCounts"],
+ sample_column="sample_id",
+ min_cells_per_sample=2,
+ show_qc_plots=False,
+ )
+
+ np.testing.assert_array_equal(selection[:], selection_before)
+ assert dict(selection.attrs) == provenance_before
+
+
+@pytest.mark.parametrize("attr", ["RNA_nCounts", "RNA_nFeatures"])
+def test_auto_filter_cells_rejects_negative_counts_without_selection_mutation(
+ datastore_ephemeral,
+ attr,
+):
+ n = datastore_ephemeral.cells.N
+ datastore_ephemeral.cells.insert(
+ "sample_id",
+ np.array(["A"] * n),
+ overwrite=True,
+ )
+ bad = np.asarray(datastore_ephemeral.cells.fetch_all(attr), dtype=float)
+ active = np.asarray(datastore_ephemeral.cells.fetch_all("I"), dtype=bool)
+ bad[int(np.flatnonzero(active)[0])] = -2.0
+ datastore_ephemeral.cells.insert(attr, bad, overwrite=True)
+ selection = datastore_ephemeral.zw["cellData"]["I"]
+ selection_before = active.copy()
+ provenance_before = dict(selection.attrs)
+
+ with pytest.raises(ValueError, match="non-negative before log1p"):
+ datastore_ephemeral.auto_filter_cells(
+ attrs=[attr],
+ sample_column="sample_id",
+ min_cells_per_sample=2,
+ show_qc_plots=False,
+ )
+
+ np.testing.assert_array_equal(
+ datastore_ephemeral.cells.fetch_all("I"),
+ selection_before,
+ )
+ assert dict(selection.attrs) == provenance_before
+
+
+def test_auto_filter_cells_sample_column_raises_on_missing_and_nonfinite(
+ datastore_ephemeral,
+):
+ n = datastore_ephemeral.cells.N
+ labels = np.array(["A"] * n, dtype=object)
+ labels[0] = ""
+ datastore_ephemeral.cells.insert("sample_id", labels, overwrite=True)
+ with pytest.raises(ValueError, match="missing labels"):
+ datastore_ephemeral.auto_filter_cells(
+ attrs=["RNA_nCounts"],
+ sample_column="sample_id",
+ show_qc_plots=False,
+ min_cells_per_sample=2,
+ )
+
+ datastore_ephemeral.cells.insert(
+ "sample_id",
+ np.array(["A"] * n),
+ overwrite=True,
+ )
+ bad = np.asarray(datastore_ephemeral.cells.fetch_all("RNA_nCounts"), dtype=float)
+ bad[0] = np.nan
+ datastore_ephemeral.cells.insert("RNA_nCounts", bad, overwrite=True)
+ with pytest.raises(ValueError, match="non-finite"):
+ datastore_ephemeral.auto_filter_cells(
+ attrs=["RNA_nCounts"],
+ sample_column="sample_id",
+ show_qc_plots=False,
+ min_cells_per_sample=2,
+ )
+
+
+def test_auto_filter_cells_sample_column_records_provenance(
+ datastore_ephemeral,
+):
+ from scarf.storage.artifacts import fingerprint_array, fingerprint_strings
+
+ n = datastore_ephemeral.cells.N
+ labels = np.array(["A"] * (n - 5) + ["tiny"] * 5)
+ datastore_ephemeral.cells.insert("sample_id", labels, overwrite=True)
+ active = np.asarray(datastore_ephemeral.cells.fetch_all("I"), dtype=bool)
+ prior_selection = datastore_ephemeral._ensure_cell_selection("I")
+ expected_sample_fingerprint = fingerprint_strings(labels[active])
+ expected_metric_fingerprints = {
+ attr: fingerprint_array(
+ np.asarray(datastore_ephemeral.cells.fetch_all(attr), dtype=float)[active]
+ )
+ for attr in ("RNA_nCounts", "RNA_percentMito")
+ }
+
+ captured: list[str] = []
+ sink = logger.add(
+ lambda message: captured.append(message.record["message"]),
+ level="WARNING",
+ )
+ try:
+ datastore_ephemeral.auto_filter_cells(
+ attrs=["RNA_nCounts", "RNA_percentMito"],
+ sample_column="sample_id",
+ n_mads=3.0,
+ min_cells_per_sample=20,
+ show_qc_plots=False,
+ )
+ finally:
+ logger.remove(sink)
+
+ cell_ref = ArtifactRef.from_dict(
+ datastore_ephemeral.zw["cellData"]["I"].attrs["source_artifact"]
+ )
+ status = datastore_ephemeral.inspect_artifact(cell_ref)
+ assert status.operation == "auto_filter_cells"
+ assert status.parameters["sample_column"] == "sample_id"
+ assert status.parameters["n_mads"] == 3.0
+ assert status.parameters["mad_scale"] == 1.4826
+ assert status.parameters["skip_reasons"]["tiny"] == "insufficient_cells"
+ assert status.parameters["metric_policies"]["RNA_nCounts"]["transform"] == "log1p"
+ assert (
+ status.parameters["metric_policies"]["RNA_percentMito"]["bound_direction"]
+ == "upper"
+ )
+ assert status.inputs["prior_cell_selection"] == prior_selection.to_dict()
+ assert (
+ status.inputs["sample_assignments_fingerprint"] == expected_sample_fingerprint
+ )
+ assert status.inputs["qc_metric_fingerprints"] == expected_metric_fingerprints
+ assert any("tiny" in message for message in captured)
+
+
+def test_pipeline_passes_sample_column_through(datastore_ephemeral):
+ n = datastore_ephemeral.cells.N
+ datastore_ephemeral.cells.insert(
+ "sample_id",
+ np.array(["A"] * (n // 2) + ["B"] * (n - n // 2)),
+ overwrite=True,
+ )
+ artifacts = datastore_ephemeral.pipeline.run(
+ pipeline_id="basic_rna_analysis",
+ filtering={
+ "sample_column": "sample_id",
+ "n_mads": 3.0,
+ "min_cells_per_sample": 20,
+ "attrs": ["RNA_nCounts"],
+ },
+ cell_cycle_scoring=False,
+ highly_variable_features={
+ "top_n": 50,
+ "hvg_key_name": "mad_pipeline_hvgs",
+ },
+ pca={"dims": 5, "n_centroids": 10},
+ ann_index={"ann_m": 16},
+ neighbors={"k": 3},
+ connectivity={},
+ umap=False,
+ leiden={},
+ paris=False,
+ doublet_scoring=False,
+ markers=False,
+ )
+ status = datastore_ephemeral.inspect_artifact(artifacts["cell_selection"])
+ assert status.operation == "auto_filter_cells"
+ assert status.parameters["sample_column"] == "sample_id"
+ assert status.parameters["n_mads"] == 3.0
diff --git a/tests/test_selections.py b/tests/test_selections.py
new file mode 100644
index 00000000..36523765
--- /dev/null
+++ b/tests/test_selections.py
@@ -0,0 +1,205 @@
+import numpy as np
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.storage.artifacts import ArtifactRef, artifact_path, inspect_artifact
+from scarf.storage.arrays import create_metadata_column
+from scarf.storage.selections import (
+ resolve_generated_selection_artifact,
+ resolve_selection_artifact,
+)
+
+
+def test_selection_artifacts_snapshot_values_and_reuse_by_provenance() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ values = np.array([True, False, True, False])
+ row_ids = np.array(["a", "b", "c", "d"])
+
+ first = resolve_selection_artifact(
+ root,
+ scope="datastore",
+ kind="cell_selection",
+ values=values,
+ row_ids=row_ids,
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column="I",
+ )
+ renamed = resolve_selection_artifact(
+ root,
+ scope="datastore",
+ kind="cell_selection",
+ values=values.copy(),
+ row_ids=row_ids.copy(),
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column="renamed_mask",
+ )
+
+ assert renamed == first
+ group = root[artifact_path(first)]
+ np.testing.assert_array_equal(group["values"][:], values)
+ status = inspect_artifact(root, first)
+ assert status.complete
+ assert status.execution_options == {"source_column": "I"}
+
+
+def test_changed_or_invalidated_selection_creates_another_random_artifact() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ values = np.array([True, False, True, False])
+ changed = np.array([True, True, False, False])
+ row_ids = np.arange(4).astype(str)
+
+ first = resolve_selection_artifact(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="feature_selection",
+ values=values,
+ row_ids=row_ids,
+ operation="select_hvgs",
+ parameters={"top_n": 2},
+ inputs={},
+ source_column="I__hvgs",
+ )
+ changed_ref = resolve_selection_artifact(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="feature_selection",
+ values=changed,
+ row_ids=row_ids,
+ operation="select_hvgs",
+ parameters={"top_n": 2},
+ inputs={},
+ source_column="I__hvgs",
+ )
+ invalidated = resolve_selection_artifact(
+ root,
+ scope="assay",
+ assay="RNA",
+ kind="feature_selection",
+ values=values,
+ row_ids=row_ids,
+ operation="select_hvgs",
+ parameters={"top_n": 2},
+ inputs={},
+ source_column="I__hvgs",
+ invalidate_cache=True,
+ )
+
+ assert (
+ len({first.artifact_id, changed_ref.artifact_id, invalidated.artifact_id}) == 3
+ )
+
+
+def test_integer_selection_payload_is_not_reused() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ values = np.array([True, False, True])
+ row_ids = np.array(["a", "b", "c"])
+ first = resolve_selection_artifact(
+ root,
+ scope="datastore",
+ kind="cell_selection",
+ values=values,
+ row_ids=row_ids,
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column="I",
+ )
+ group = root[artifact_path(first)]
+ del group["values"]
+ create_metadata_column(
+ group,
+ "values",
+ data=values.astype(np.int8),
+ dtype=np.int8,
+ )
+
+ replacement = resolve_selection_artifact(
+ root,
+ scope="datastore",
+ kind="cell_selection",
+ values=values,
+ row_ids=row_ids,
+ operation="manual_selection",
+ parameters={},
+ inputs={},
+ source_column="I",
+ )
+
+ assert replacement != first
+ assert root[artifact_path(replacement)]["values"].dtype == np.dtype(bool)
+
+
+def test_generated_selection_identity_excludes_output_values() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ values = np.array([True, False, True])
+ row_ids = np.array(["a", "b", "c"])
+ first, stored = resolve_generated_selection_artifact(
+ root,
+ scope="assay",
+ assay="ATAC",
+ kind="feature_selection",
+ values=values,
+ row_ids=row_ids,
+ operation="mark_prevalent_peaks",
+ parameters={"top_n": 2},
+ inputs={"feature_selection": {"artifact_id": "input"}},
+ source_column="I__prevalent_peaks",
+ )
+ reused, reused_values = resolve_generated_selection_artifact(
+ root,
+ scope="assay",
+ assay="ATAC",
+ kind="feature_selection",
+ values=~values,
+ row_ids=row_ids,
+ operation="mark_prevalent_peaks",
+ parameters={"top_n": 2},
+ inputs={"feature_selection": {"artifact_id": "input"}},
+ source_column="renamed",
+ )
+
+ assert reused == first
+ np.testing.assert_array_equal(stored, values)
+ np.testing.assert_array_equal(reused_values, values)
+ assert "values_fingerprint" not in inspect_artifact(root, first).inputs
+
+
+def test_filter_and_hvg_columns_link_to_provenance_artifacts(
+ datastore_ephemeral,
+) -> None:
+ datastore = datastore_ephemeral
+ datastore.filter_cells(
+ attrs=["RNA_nCounts"],
+ lows=[0],
+ highs=[None],
+ reset_previous=True,
+ )
+ cell_column = datastore.zw["cellData"]["I"]
+ cell_ref = ArtifactRef.from_dict(cell_column.attrs["source_artifact"])
+ cell_status = datastore.inspect_artifact(cell_ref)
+ assert cell_status.operation == "filter_cells"
+ assert cell_status.parameters["attrs"] == ["RNA_nCounts"]
+
+ datastore.mark_hvgs(
+ from_assay="RNA",
+ cell_key="I",
+ top_n=50,
+ hvg_key_name="selection_test_hvgs",
+ show_plot=False,
+ )
+ feature_column = datastore.get_assay("RNA").z["featureData"][
+ "I__selection_test_hvgs"
+ ]
+ feature_ref = ArtifactRef.from_dict(feature_column.attrs["source_artifact"])
+ feature_status = datastore.inspect_artifact(feature_ref)
+ assert feature_status.operation == "mark_hvgs"
+ assert feature_status.parameters["top_n"] == 50
+ assert (
+ feature_status.inputs["cell_selection"]["artifact_id"] == cell_ref.artifact_id
+ )
diff --git a/tests/test_seurat_import.py b/tests/test_seurat_import.py
new file mode 100644
index 00000000..6f76aa2e
--- /dev/null
+++ b/tests/test_seurat_import.py
@@ -0,0 +1,358 @@
+import dataclasses
+from pathlib import Path
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.datastore.datastore import DataStore
+from scarf.graph.state import read_assay_state
+from scarf.readers._rds import R_INT_NA, RdsClosedError
+from scarf.readers.seurat import SeuratReader
+from scarf.storage.artifacts import artifact_group
+from scarf.writers.seurat import SeuratImportResult, SeuratToZarr
+from tests.test_seurat_reader import (
+ _Wire,
+ _legacy_assay,
+ _reduction,
+ _write_fixture,
+)
+
+
+def _ordered_factor(
+ wire: _Wire,
+ values: list[int],
+ levels: list[str],
+) -> bytes:
+ return wire.integer_vector(
+ values,
+ attributes=[
+ ("levels", wire.string_vector(levels)),
+ ("class", wire.string_vector(["ordered", "factor"])),
+ ],
+ )
+
+
+def _partial_assay5(wire: _Wire) -> bytes:
+ layer_names = ["counts", "data"]
+ layers = wire.vector(
+ [
+ wire.matrix([1, 0, 2, 3], (2, 2)),
+ wire.matrix([0.0] * 4, (2, 2), real=True),
+ ],
+ names=layer_names,
+ )
+ metadata = wire.data_frame(
+ [("kind", wire.string_vector(["a", "b"]))],
+ 2,
+ )
+ return wire.s4(
+ [
+ ("layers", layers),
+ (
+ "cells",
+ wire.logmap([1, 1, 1, 1], ["c1", "c3"], layer_names),
+ ),
+ (
+ "features",
+ wire.logmap([1, 1, 1, 1], ["p1", "p2"], layer_names),
+ ),
+ ("meta.data", metadata),
+ ("class", wire.string_vector(["Assay5"])),
+ ]
+ )
+
+
+def _write_partial_fixture(path: Path, *, reduction_name: str = "pca") -> Path:
+ wire = _Wire()
+ metadata = wire.data_frame(
+ [
+ ("logical", wire.logical_vector([1, 0, R_INT_NA])),
+ ("integer", wire.integer_vector([1, R_INT_NA, 3])),
+ ("real", wire.real_vector([1.5, float("nan"), 3.5])),
+ ("character", wire.string_vector(["a", None, "c"])),
+ (
+ "group",
+ _ordered_factor(
+ wire,
+ [1, 2, R_INT_NA],
+ ["first", "second"],
+ ),
+ ),
+ ],
+ ["c1", "c2", "c3"],
+ )
+ root = wire.s4(
+ [
+ (
+ "assays",
+ wire.vector(
+ [_legacy_assay(wire), _partial_assay5(wire)],
+ names=["RNA", "ADT"],
+ ),
+ ),
+ ("meta.data", metadata),
+ ("active.assay", wire.string_vector(["RNA"])),
+ (
+ "active.ident",
+ wire.factor(
+ [2, 1, R_INT_NA],
+ ["zero", "one"],
+ names=["c3", "c1", "c2"],
+ ),
+ ),
+ (
+ "reductions",
+ wire.vector([_reduction(wire)], names=[reduction_name]),
+ ),
+ ("class", wire.string_vector(["Seurat"])),
+ ]
+ )
+ path.write_bytes(wire.document(root))
+ return path
+
+
+def _missing(group: zarr.Group, column_name: str) -> np.ndarray:
+ column = group[column_name]
+ return np.asarray(group[column.attrs["missing_mask"]][:], dtype=bool)
+
+
+def test_import_materializes_metadata_counts_membership_and_pca(
+ tmp_path: Path,
+) -> None:
+ source = _write_partial_fixture(tmp_path / "partial.rds")
+ destination = MemoryStore()
+
+ with SeuratReader(source) as reader:
+ result = SeuratToZarr(
+ reader,
+ destination,
+ mem_budget="64M",
+ nthreads=1,
+ targetChunkBytes=1024,
+ targetShardBytes=4096,
+ ).dump(batch_size=1)
+
+ root = zarr.open_group(store=destination, mode="r")
+ assert isinstance(result, SeuratImportResult)
+ assert result.assayNames == ("RNA", "ADT")
+ assert result.defaultAssay == "RNA"
+ assert root.attrs["complete"] is True
+ assert root.attrs["scarf:import_complete"] is True
+ assert root.attrs["scarf:import_source"] == "seurat"
+ assert len(root.attrs["scarf:import_source_sha256"]) == 64
+ assert len(root.attrs["scarf:import_payload_sha256"]) == 64
+ assert root.attrs["defaultAssay"] == "RNA"
+ assert root["cellData/ids"][:].tolist() == ["c1", "c2", "c3"]
+ assert root["cellData/names"][:].tolist() == ["c1", "c2", "c3"]
+
+ cells = root["cellData"]
+ assert cells["group"][:].tolist() == ["first", "second", ""]
+ assert cells["group"].attrs["levels"] == ["first", "second"]
+ assert cells["group"].attrs["ordered"] is True
+ np.testing.assert_array_equal(_missing(cells, "group"), [False, False, True])
+ assert cells["active.ident"][:].tolist() == ["zero", "", "one"]
+ assert cells["active.ident"].attrs["levels"] == ["zero", "one"]
+ np.testing.assert_array_equal(
+ _missing(cells, "active.ident"),
+ [False, True, False],
+ )
+ np.testing.assert_array_equal(
+ _missing(cells, "character"),
+ [False, True, False],
+ )
+ np.testing.assert_array_equal(cells["ADT_I"][:], [True, False, True])
+
+ np.testing.assert_array_equal(
+ root["RNA/counts"][:],
+ [[1, 0], [0, 2], [3, 0]],
+ )
+ np.testing.assert_array_equal(
+ root["ADT/counts"][:],
+ [[1, 0], [0, 0], [2, 3]],
+ )
+ assert root["RNA/featureData/ids"][:].tolist() == ["g1", "g2"]
+ assert root["RNA/featureData/names"][:].tolist() == ["g1", "g2"]
+ assert root["RNA/featureData/symbol"][:].tolist() == ["G1", "G2"]
+ assert root["ADT/featureData/kind"][:].tolist() == ["a", "b"]
+
+ selection = artifact_group(root, result.cellSelection)
+ np.testing.assert_array_equal(selection["values"][:], [True, True, True])
+ pca_ref = result.reductionArtifacts["pca"]
+ pca = artifact_group(root, pca_ref)
+ np.testing.assert_array_equal(
+ pca["data"][:],
+ [[1.0, 4.0], [2.0, 5.0], [3.0, 6.0]],
+ )
+ np.testing.assert_array_equal(pca["stdev"][:], [2.0, 1.0])
+ state = read_assay_state(root, "RNA")
+ assert state is not None
+ assert state.reduction is None
+ assert state.named_results["pca"] == pca_ref
+ assert any(
+ notice.code == "ignored_normalized_layer"
+ and notice.objectPath == "assays/ADT/layers/data"
+ for notice in result.notices
+ )
+ with pytest.raises(dataclasses.FrozenInstanceError):
+ result.defaultAssay = "ADT" # type: ignore[misc]
+
+
+def test_imported_store_is_readable_by_datastore(tmp_path: Path) -> None:
+ source = _write_fixture(tmp_path / "datastore.rds")
+ destination = MemoryStore()
+ with SeuratReader(source) as reader:
+ SeuratToZarr(
+ reader,
+ destination,
+ mem_budget="64M",
+ nthreads=1,
+ targetChunkBytes=1024,
+ targetShardBytes=4096,
+ ).dump(batch_size=2)
+
+ store = DataStore(
+ destination,
+ min_features_per_cell=0,
+ min_cells_per_feature=0,
+ nthreads=1,
+ mem_budget="64M",
+ )
+
+ assert store._defaultAssay == "RNA"
+ assert store.assay_names == ["ADT", "RNA"]
+ assert store.cells.fetch_all("ids").tolist() == ["c1", "c2", "c3"]
+ assert store.get_assay("RNA").rawData.shape == (3, 2)
+ assert store.get_assay_state("RNA") is not None
+ assert store.get_assay_state("RNA").reduction is None # type: ignore[union-attr]
+
+
+def test_import_normalizes_seurat_reduction_name_for_assay_state(
+ tmp_path: Path,
+) -> None:
+ source = _write_partial_fixture(
+ tmp_path / "named-reduction.rds",
+ reduction_name="UMAP.RNA",
+ )
+ destination = MemoryStore()
+
+ with SeuratReader(source) as reader:
+ result = SeuratToZarr(
+ reader,
+ destination,
+ mem_budget="64M",
+ nthreads=1,
+ targetChunkBytes=1024,
+ targetShardBytes=4096,
+ ).dump(batch_size=1)
+
+ root = zarr.open_group(store=destination, mode="r")
+ state = read_assay_state(root, "RNA")
+ assert state is not None
+ assert result.reductionArtifacts["UMAP.RNA"] == state.named_results["umap_rna"]
+
+
+def test_reader_must_remain_open_until_dump_finishes(tmp_path: Path) -> None:
+ source = _write_fixture(tmp_path / "closed.rds")
+ destination = MemoryStore()
+ reader = SeuratReader(source)
+ writer = SeuratToZarr(
+ reader,
+ destination,
+ mem_budget="64M",
+ nthreads=1,
+ targetChunkBytes=1024,
+ targetShardBytes=4096,
+ )
+ reader.close()
+
+ with pytest.raises(RdsClosedError, match="RDS document is closed"):
+ writer.dump(batch_size=1)
+
+ root = zarr.open_group(store=destination, mode="r")
+ assert root.attrs["complete"] is False
+ assert root.attrs["scarf:import_complete"] is False
+ with pytest.raises(RuntimeError, match="seurat import is incomplete"):
+ DataStore(
+ destination,
+ min_features_per_cell=0,
+ min_cells_per_feature=0,
+ nthreads=1,
+ mem_budget="64M",
+ )
+
+
+def test_matrix_metadata_and_reduction_reads_stay_bounded(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ source = _write_partial_fixture(tmp_path / "bounded.rds")
+ destination = MemoryStore()
+ with SeuratReader(source) as reader:
+ writer = SeuratToZarr(
+ reader,
+ destination,
+ mem_budget="64M",
+ nthreads=1,
+ targetChunkBytes=1024,
+ targetShardBytes=4096,
+ )
+ matrix_reads: dict[str, list[tuple[int, int]]] = {}
+ for assay_name in reader.assayNames:
+ matrix = reader.get_assay(assay_name).counts
+ original = matrix.read_cells
+ reads: list[tuple[int, int]] = []
+ matrix_reads[assay_name] = reads
+
+ def tracked_matrix_read(
+ start: int,
+ stop: int,
+ *,
+ read=original,
+ target=reads,
+ ):
+ target.append((start, stop))
+ return read(start, stop)
+
+ monkeypatch.setattr(matrix, "read_cells", tracked_matrix_read)
+
+ character = reader.cellMetadata.column("character")
+ original_metadata_read = character.read_block
+ metadata_reads: list[tuple[int, int]] = []
+
+ def tracked_metadata_read(start: int, stop: int):
+ metadata_reads.append((start, stop))
+ return original_metadata_read(start, stop)
+
+ monkeypatch.setattr(character, "read_block", tracked_metadata_read)
+ membership = reader.get_assay("ADT").cellMembership
+ original_membership_read = membership.read_block
+ membership_reads: list[tuple[int, int]] = []
+
+ def tracked_membership_read(start: int, stop: int):
+ membership_reads.append((start, stop))
+ return original_membership_read(start, stop)
+
+ monkeypatch.setattr(membership, "read_block", tracked_membership_read)
+ embeddings = reader.get_reduction("pca").cellEmbeddings
+ original_reduction_read = embeddings.read_rows
+ reduction_reads: list[tuple[int, int]] = []
+
+ def tracked_reduction_read(start: int, stop: int):
+ reduction_reads.append((start, stop))
+ return original_reduction_read(start, stop)
+
+ monkeypatch.setattr(embeddings, "read_rows", tracked_reduction_read)
+ writer.dump(batch_size=1)
+
+ assert all(
+ stop - start <= 1 for reads in matrix_reads.values() for start, stop in reads
+ )
+ assert all(stop - start <= 1 for start, stop in metadata_reads)
+ assert all(stop - start <= 1 for start, stop in membership_reads)
+ assert all(stop - start <= 1 for start, stop in reduction_reads)
+ assert {start for start, _ in matrix_reads["RNA"]} == {0, 1, 2}
+ assert {start for start, _ in matrix_reads["ADT"]} == {0, 1, 2}
+ assert {start for start, _ in membership_reads} == {0, 1, 2}
+ assert {start for start, _ in reduction_reads} == {0, 1, 2}
diff --git a/tests/test_seurat_matrix_sources.py b/tests/test_seurat_matrix_sources.py
new file mode 100644
index 00000000..6aaf00ca
--- /dev/null
+++ b/tests/test_seurat_matrix_sources.py
@@ -0,0 +1,2467 @@
+from collections.abc import Sequence
+from pathlib import Path
+from typing import Any
+
+import h5py
+import numpy as np
+import pytest
+from numpy.typing import NDArray
+from scipy.sparse import csc_matrix, csr_matrix
+
+from scarf.readers._seurat import (
+ BPCellsDirectoryMatrixSource,
+ BPCellsHDF5MatrixSource,
+ BinaryTransformMatrixSource,
+ CellBindMatrixSource,
+ CscMatrixSource,
+ DelayedSubassignmentMatrixSource,
+ DenseMatrixSource,
+ DtypeMatrixSource,
+ FeatureBindMatrixSource,
+ FRAGMENT_CAPABILITY_REGISTRY,
+ FragmentDerivedMatrixSource,
+ H5ADMatrixSource,
+ H5SparseMatrixSource,
+ HDF5ArrayMatrixSource,
+ LayerPlacement,
+ LayerStitchMatrixSource,
+ MappedMatrixSource,
+ MaskMatrixSource,
+ MatrixMultiplySource,
+ MatrixSource,
+ MatrixSourceError,
+ RankMatrixSource,
+ RenamedMatrixSource,
+ ResourceLimitError,
+ SidecarPathResolver,
+ SourceLimits,
+ Subassignment,
+ TENxMatrixSource,
+ TransposeMatrixSource,
+ UnaryTransformMatrixSource,
+ UnsafeSidecarError,
+ UnsupportedMatrixOperation,
+ build_matrix_operation,
+ decode_bp128,
+ decode_bp128_d1z,
+ decode_bp128_m1,
+ fragment_source_from_slots,
+ matrix_source_from_slots,
+ validate_hdf5_file,
+)
+
+
+_HEADERS = {
+ ("u", 4): b"UINT32v1",
+ ("u", 8): b"UINT64v1",
+ ("f", 4): b"FLOATSv1",
+ ("f", 8): b"DOUBLEv1",
+}
+
+
+class _TrackedVector:
+ def __init__(self, values: Sequence[int] | NDArray[Any]) -> None:
+ self.values = np.asarray(values)
+ self.shape = self.values.shape
+ self.dtype = self.values.dtype
+ self.reads: list[tuple[int | None, int | None]] = []
+
+ def __getitem__(self, key: slice) -> NDArray[Any]:
+ assert isinstance(key, slice)
+ self.reads.append((key.start, key.stop))
+ return self.values[key]
+
+
+def _pack_bp128_block(
+ values: Sequence[int] | NDArray[Any],
+) -> NDArray[np.uint32]:
+ values = np.asarray(values, dtype=np.uint32)
+ assert values.shape == (128,)
+ bits = max((int(value).bit_length() for value in values), default=0)
+ if bits == 0:
+ return np.empty(0, dtype=np.uint32)
+ packed: NDArray[np.uint64] = np.zeros((bits, 4), dtype=np.uint64)
+ mask = (1 << bits) - 1 if bits < 32 else 0xFFFFFFFF
+ for vector_index in range(32):
+ lanes = values[vector_index * 4 : vector_index * 4 + 4].astype(np.uint64)
+ lanes &= np.uint64(mask)
+ bit_position = vector_index * bits
+ word_index = bit_position // 32
+ shift = bit_position & 31
+ packed[word_index] |= (lanes << np.uint64(shift)) & np.uint64(0xFFFFFFFF)
+ if shift + bits > 32:
+ packed[word_index + 1] |= lanes >> np.uint64(32 - shift)
+ return packed.astype(np.uint32).reshape(-1)
+
+
+def _encode_bp128(
+ values: Sequence[int] | NDArray[Any],
+ transform: str,
+) -> tuple[
+ NDArray[np.uint32],
+ NDArray[np.uint32],
+ NDArray[np.uint64],
+ NDArray[np.uint32],
+]:
+ values = np.asarray(values, dtype=np.uint32)
+ data_parts: list[NDArray[np.uint32]] = []
+ indexes: list[int] = [0]
+ starts: list[int] = []
+ for block_start in range(0, values.size, 128):
+ block = values[block_start : block_start + 128]
+ if block.size < 128:
+ pad_value = int(block[-1]) if block.size else 0
+ block = np.pad(
+ block,
+ (0, 128 - block.size),
+ constant_values=pad_value,
+ )
+ if transform == "m1":
+ assert np.all(block > 0)
+ encoded = block - np.uint32(1)
+ elif transform == "d1z":
+ starts.append(int(block[0]))
+ signed = block.astype(np.int64)
+ differences = np.diff(signed, prepend=signed[0])
+ encoded = np.where(
+ differences >= 0,
+ differences * 2,
+ -differences * 2 - 1,
+ ).astype(np.uint32)
+ elif transform == "d1":
+ starts.append(int(block[0]))
+ previous = np.concatenate((block[:1], block[:-1]))
+ encoded = block - previous
+ elif transform == "plain":
+ encoded = block
+ else:
+ raise AssertionError(transform)
+ packed = _pack_bp128_block(encoded)
+ data_parts.append(packed)
+ indexes.append(indexes[-1] + packed.size)
+ data = np.concatenate(data_parts) if data_parts else np.empty(0, dtype=np.uint32)
+ return (
+ data,
+ np.asarray(indexes, dtype=np.uint32),
+ np.asarray([0, len(indexes)], dtype=np.uint64),
+ np.asarray(starts, dtype=np.uint32),
+ )
+
+
+def _write_numeric_file(directory: Path, name: str, values: Any) -> None:
+ values = np.asarray(values)
+ key = (values.dtype.kind, values.dtype.itemsize)
+ assert key in _HEADERS
+ little_endian = values.astype(values.dtype.newbyteorder("<"), copy=False)
+ (directory / name).write_bytes(_HEADERS[key] + little_endian.tobytes(order="C"))
+
+
+def _matrix_arrays(
+ values: NDArray[Any],
+ storage_order: str,
+) -> tuple[NDArray[Any], NDArray[np.uint32], NDArray[Any]]:
+ matrix = csc_matrix(values) if storage_order == "col" else csr_matrix(values)
+ return (
+ np.asarray(matrix.data),
+ matrix.indices.astype(np.uint32),
+ matrix.indptr,
+ )
+
+
+def _bpcells_payload(
+ values: NDArray[Any],
+ *,
+ packed: bool,
+ version: int,
+ storage_order: str,
+) -> dict[str, Any]:
+ data, indexes, pointers = _matrix_arrays(values, storage_order)
+ datatype = {
+ np.dtype(np.uint32): "uint",
+ np.dtype(np.float32): "float",
+ np.dtype(np.float64): "double",
+ }[values.dtype]
+ payload = {
+ "version": f"{'packed' if packed else 'unpacked'}-{datatype}-matrix-v{version}",
+ "shape": np.asarray(values.shape, dtype=np.uint32),
+ "idxptr": pointers.astype(np.uint32 if version == 1 else np.uint64),
+ "storage_order": storage_order,
+ "row_names": tuple(f"f{index}" for index in range(values.shape[0])),
+ "col_names": tuple(f"c{index}" for index in range(values.shape[1])),
+ }
+ if not packed:
+ payload["index"] = indexes
+ payload["val"] = data
+ return payload
+ (
+ payload["index_data"],
+ payload["index_idx"],
+ payload["index_idx_offsets"],
+ payload["index_starts"],
+ ) = _encode_bp128(indexes, "d1z")
+ if datatype == "uint":
+ (
+ payload["val_data"],
+ payload["val_idx"],
+ payload["val_idx_offsets"],
+ _,
+ ) = _encode_bp128(data, "m1")
+ else:
+ payload["val"] = data
+ return payload
+
+
+def _write_bpcells_directory(
+ path: Path,
+ payload: dict[str, Any],
+ *,
+ version: int,
+) -> None:
+ path.mkdir()
+ (path / "version").write_text(payload["version"] + "\n")
+ (path / "storage_order").write_text(payload["storage_order"] + "\n")
+ (path / "row_names").write_text("\n".join(payload["row_names"]) + "\n")
+ (path / "col_names").write_text("\n".join(payload["col_names"]) + "\n")
+ for name, values in payload.items():
+ if name in {
+ "version",
+ "storage_order",
+ "row_names",
+ "col_names",
+ }:
+ continue
+ if version == 1 and name.endswith("_idx_offsets"):
+ continue
+ _write_numeric_file(path, name, values)
+
+
+def _write_bpcells_hdf5(
+ path: Path,
+ payload: dict[str, Any],
+ *,
+ version: int,
+) -> None:
+ string_dtype = h5py.string_dtype("utf-8")
+ with h5py.File(path, mode="w") as handle:
+ group = handle.create_group("matrix")
+ group.attrs["version"] = payload["version"]
+ group.create_dataset(
+ "storage_order",
+ data=np.asarray(payload["storage_order"], dtype=string_dtype),
+ )
+ group.create_dataset(
+ "row_names",
+ data=np.asarray(payload["row_names"], dtype=string_dtype),
+ )
+ group.create_dataset(
+ "col_names",
+ data=np.asarray(payload["col_names"], dtype=string_dtype),
+ )
+ for name, values in payload.items():
+ if name in {
+ "version",
+ "storage_order",
+ "row_names",
+ "col_names",
+ }:
+ continue
+ if version == 1 and name.endswith("_idx_offsets"):
+ continue
+ group.create_dataset(name, data=values)
+
+
+def _memory_matrix_value(value: Any) -> Any:
+ if not isinstance(value, np.ndarray):
+ return value
+ if value.dtype == np.dtype(np.uint32):
+ return value.view(np.int32)
+ if value.dtype == np.dtype(np.uint64):
+ return value.astype(np.float64)
+ if value.dtype == np.dtype(np.float32):
+ return value.view(np.int32)
+ return value
+
+
+def _memory_matrix_spec(
+ payload: dict[str, Any],
+ *,
+ packed: bool,
+ datatype: str,
+ version: int,
+) -> dict[str, Any]:
+ class_datatype = "uint32_t" if datatype == "uint" else datatype
+ slots = {
+ name: _memory_matrix_value(value)
+ for name, value in payload.items()
+ if name
+ not in {
+ "shape",
+ "storage_order",
+ "row_names",
+ "col_names",
+ }
+ and not (version == 1 and name.endswith("_idx_offsets"))
+ }
+ slots.update(
+ {
+ "version": payload["version"],
+ "dim": payload["shape"].astype(np.float64),
+ "transpose": payload["storage_order"] == "row",
+ "dimnames": [payload["row_names"], payload["col_names"]],
+ }
+ )
+ return {
+ "class": [
+ f"{'Packed' if packed else 'Unpacked'}MatrixMem_{class_datatype}",
+ "IterableMatrix",
+ ],
+ "slots": slots,
+ }
+
+
+def _fragment_payload(*, packed: bool, version: int) -> dict[str, Any]:
+ cells = np.asarray([0, 1, 0, 2, 1, 2, 0], dtype=np.uint32)
+ starts = np.asarray([0, 5, 10, 12, 20, 1, 4], dtype=np.uint32)
+ ends = np.asarray([10, 15, 20, 18, 30, 9, 12], dtype=np.uint32)
+ payload: dict[str, Any] = {
+ "version": f"{'packed' if packed else 'unpacked'}-fragments-v{version}",
+ "end_max": np.asarray([30], dtype=np.uint32),
+ "chr_ptr": np.asarray(
+ [0, 5, 5, 7],
+ dtype=np.uint32 if version == 1 else np.uint64,
+ ),
+ "chr_names": ("chr1", "chr2"),
+ "cell_names": ("c1", "c2", "c3"),
+ }
+ if not packed:
+ payload.update({"cell": cells, "start": starts, "end": ends})
+ return payload
+ for name, values, transform in (
+ ("cell", cells, "plain"),
+ ("start", starts, "d1"),
+ ("end", ends - starts, "plain"),
+ ):
+ data, indexes, offsets, block_starts = _encode_bp128(values, transform)
+ payload[f"{name}_data"] = data
+ payload[f"{name}_idx"] = indexes
+ payload[f"{name}_idx_offsets"] = offsets
+ if name == "start":
+ payload["start_starts"] = block_starts
+ return payload
+
+
+def _write_fragment_directory(
+ path: Path,
+ payload: dict[str, Any],
+ *,
+ version: int,
+) -> None:
+ path.mkdir()
+ (path / "version").write_text(payload["version"] + "\n")
+ (path / "chr_names").write_text("\n".join(payload["chr_names"]) + "\n")
+ (path / "cell_names").write_text("\n".join(payload["cell_names"]) + "\n")
+ for name, values in payload.items():
+ if name in {"version", "chr_names", "cell_names"}:
+ continue
+ if version == 1 and name.endswith("_idx_offsets"):
+ continue
+ _write_numeric_file(path, name, values)
+
+
+def _write_fragment_hdf5(
+ path: Path,
+ payload: dict[str, Any],
+ *,
+ version: int,
+) -> None:
+ string_dtype = h5py.string_dtype("utf-8")
+ with h5py.File(path, mode="w") as handle:
+ group = handle.create_group("fragments")
+ group.attrs["version"] = payload["version"]
+ group.create_dataset(
+ "chr_names",
+ data=np.asarray(payload["chr_names"], dtype=string_dtype),
+ )
+ group.create_dataset(
+ "cell_names",
+ data=np.asarray(payload["cell_names"], dtype=string_dtype),
+ )
+ for name, values in payload.items():
+ if name in {"version", "chr_names", "cell_names"}:
+ continue
+ if version == 1 and name.endswith("_idx_offsets"):
+ continue
+ group.create_dataset(name, data=values)
+
+
+def _memory_fragment_value(value: Any) -> Any:
+ if not isinstance(value, np.ndarray):
+ return value
+ if value.dtype == np.dtype(np.uint32):
+ return value.view(np.int32)
+ if value.dtype == np.dtype(np.uint64):
+ return value.astype(np.float64)
+ return value
+
+
+def _fragment_leaf_spec(
+ tmp_path: Path,
+ *,
+ backend: str,
+ packed: bool,
+ version: int,
+) -> dict[str, Any]:
+ payload = _fragment_payload(packed=packed, version=version)
+ if backend == "memory":
+ slots = {name: _memory_fragment_value(value) for name, value in payload.items()}
+ slots["version"] = [payload["version"]]
+ return {
+ "class": [
+ "PackedMemFragments" if packed else "UnpackedMemFragments",
+ "IterableFragments",
+ ],
+ "slots": slots,
+ }
+ if backend == "directory":
+ path = tmp_path / f"fragments-{packed}-{version}"
+ _write_fragment_directory(path, payload, version=version)
+ return {
+ "class": ["FragmentsDir", "IterableFragments"],
+ "slots": {
+ "dir": str(path),
+ "compressed": packed,
+ "buffer_size": 128,
+ "chr_names": payload["chr_names"],
+ "cell_names": payload["cell_names"],
+ },
+ }
+ if backend == "hdf5":
+ path = tmp_path / f"fragments-{packed}-{version}.h5"
+ _write_fragment_hdf5(path, payload, version=version)
+ return {
+ "class": ["FragmentsHDF5", "IterableFragments"],
+ "slots": {
+ "path": str(path),
+ "group": "fragments",
+ "compressed": packed,
+ "buffer_size": 128,
+ "chr_names": payload["chr_names"],
+ "cell_names": payload["cell_names"],
+ },
+ }
+ raise AssertionError(backend)
+
+
+def _peak_matrix_spec(
+ fragments: dict[str, Any],
+ mode: str,
+ *,
+ transpose: bool = True,
+) -> dict[str, Any]:
+ feature_names = ("p0", "p_span", "p1", "p2", "p_chr2")
+ cell_names = ("c1", "c2", "c3")
+ return {
+ "class": ["PeakMatrix", "IterableMatrix"],
+ "slots": {
+ "fragments": fragments,
+ "chr_id": np.asarray([0, 0, 0, 0, 1], dtype=np.int32),
+ "start": np.asarray([0, 11, 10, 5, 0], dtype=np.int32),
+ "end": np.asarray([10, 14, 20, 25, 10], dtype=np.int32),
+ "chr_levels": ("chr1", "chr2"),
+ "mode": [mode],
+ "transpose": transpose,
+ "dim": (5, 3) if transpose else (3, 5),
+ "dimnames": (
+ (feature_names, cell_names)
+ if transpose
+ else (cell_names, feature_names)
+ ),
+ },
+ }
+
+
+def _tile_matrix_spec(
+ fragments: dict[str, Any],
+ mode: str,
+ *,
+ transpose: bool = True,
+) -> dict[str, Any]:
+ feature_names = ("t0", "t1", "t2", "t3", "t4")
+ cell_names = ("c1", "c2", "c3")
+ return {
+ "class": ["TileMatrix", "IterableMatrix"],
+ "slots": {
+ "fragments": fragments,
+ "chr_id": np.asarray([0, 1], dtype=np.int32),
+ "start": np.asarray([0, 0], dtype=np.int32),
+ "end": np.asarray([30, 15], dtype=np.int32),
+ "tile_width": np.asarray([10, 10], dtype=np.int32),
+ "chr_levels": ("chr1", "chr2"),
+ "mode": [mode],
+ "transpose": transpose,
+ "dim": (5, 3) if transpose else (3, 5),
+ "dimnames": (
+ (feature_names, cell_names)
+ if transpose
+ else (cell_names, feature_names)
+ ),
+ },
+ }
+
+
+def _write_h5_sparse_group(
+ parent: h5py.Group | h5py.File,
+ name: str,
+ values: NDArray[Any],
+ layout: str,
+) -> h5py.Group:
+ matrix = csr_matrix(values) if layout == "csr" else csc_matrix(values)
+ group = parent.create_group(name)
+ group.attrs["shape"] = matrix.shape
+ group.attrs["encoding-type"] = f"{layout}_matrix"
+ group.create_dataset("data", data=matrix.data)
+ group.create_dataset("indices", data=matrix.indices.astype(np.int64))
+ group.create_dataset("indptr", data=matrix.indptr.astype(np.int64))
+ return group
+
+
+def _write_axis_names(
+ handle: h5py.File,
+ features: Sequence[str],
+ cells: Sequence[str],
+) -> None:
+ obs = handle.create_group("obs")
+ obs.create_dataset("_index", data=np.asarray(cells, dtype="S"))
+ var = handle.create_group("var")
+ var.create_dataset("_index", data=np.asarray(features, dtype="S"))
+
+
+def test_dense_r_column_major_source_is_bounded_and_oriented() -> None:
+ feature_by_cell = np.asarray([[1, 2, 3], [4, 5, 6]], dtype=np.int16, order="F")
+ tracked = _TrackedVector(feature_by_cell.ravel(order="F"))
+ source = DenseMatrixSource(
+ tracked,
+ feature_by_cell.shape,
+ row_names=["g1", "g2"],
+ column_names=["c1", "c2", "c3"],
+ )
+
+ assert isinstance(source, MatrixSource)
+ assert source.shape == (2, 3)
+ assert source.dtype == np.dtype(np.int16)
+ assert source.row_names == ("g1", "g2")
+ assert source.column_names == ("c1", "c2", "c3")
+ np.testing.assert_array_equal(source.read_cells(1, 3), [[2, 5], [3, 6]])
+ assert tracked.reads == [(2, 6)]
+ assert source.estimate_read_memory(1, 3).peakBytes > 0
+
+
+def test_dense_source_enforces_block_memory_limit() -> None:
+ source = DenseMatrixSource(
+ np.arange(6, dtype=np.float64).reshape(3, 2),
+ limits=SourceLimits(maxBlockBytes=32),
+ )
+ with pytest.raises(ResourceLimitError, match="maxBlockBytes=32"):
+ source.read_cells(0, 2)
+
+
+def test_csc_source_preserves_matrix_classes_and_orientation() -> None:
+ feature_by_cell = np.asarray([[0, 2, 0], [1, 0, 3], [4, 5, 0]], dtype=np.float64)
+ matrix = csc_matrix(feature_by_cell)
+ source = CscMatrixSource(
+ matrix.data,
+ matrix.indices,
+ matrix.indptr,
+ matrix.shape,
+ class_name="dgCMatrix",
+ )
+
+ assert source.is_sparse
+ assert source.nnz == matrix.nnz
+ np.testing.assert_array_equal(
+ source.read_cells(1, 3).toarray(),
+ feature_by_cell[:, 1:3].T,
+ )
+
+ pattern = CscMatrixSource(
+ None,
+ matrix.indices,
+ matrix.indptr,
+ matrix.shape,
+ class_name="ngCMatrix",
+ )
+ np.testing.assert_array_equal(
+ pattern.read_cells(0, 3).toarray(),
+ (feature_by_cell != 0).T,
+ )
+
+
+def test_csc_source_validates_offsets_indexes_and_limits() -> None:
+ with pytest.raises(MatrixSourceError, match="nondecreasing"):
+ CscMatrixSource([1], [0], [0, 1, 0], (2, 2))
+ with pytest.raises(MatrixSourceError, match="out-of-range"):
+ CscMatrixSource([1], [2], [0, 1], (2, 1))
+ with pytest.raises(ResourceLimitError, match="maxNnz=1"):
+ CscMatrixSource(
+ [1, 2],
+ [0, 1],
+ [0, 2],
+ (2, 1),
+ limits=SourceLimits(maxNnz=1),
+ )
+
+
+def test_mapping_transpose_and_bind_sources() -> None:
+ first_values = np.asarray([[1, 2, 3], [4, 5, 6]])
+ first = DenseMatrixSource(
+ first_values,
+ row_names=["f1", "f2"],
+ column_names=["c1", "c2", "c3"],
+ )
+ mapped = MappedMatrixSource(
+ first,
+ feature_indices=[1, 0, 1],
+ cell_indices=[2, 0],
+ )
+ np.testing.assert_array_equal(
+ mapped.read_cells(0, 2),
+ [[6, 3, 6], [4, 1, 4]],
+ )
+
+ transposed = TransposeMatrixSource(first, tile_cells=1)
+ assert transposed.shape == (3, 2)
+ np.testing.assert_array_equal(
+ transposed.read_cells(0, 2),
+ first_values,
+ )
+
+ second_features = DenseMatrixSource(
+ np.asarray([[7, 8, 9]]),
+ row_names=["f3"],
+ column_names=["c1", "c2", "c3"],
+ )
+ feature_bound = FeatureBindMatrixSource([first, second_features])
+ np.testing.assert_array_equal(
+ feature_bound.read_cells(0, 3),
+ np.vstack([first_values, [[7, 8, 9]]]).T,
+ )
+
+ second_cells = DenseMatrixSource(
+ np.asarray([[7], [8]]),
+ row_names=["f1", "f2"],
+ column_names=["c4"],
+ )
+ cell_bound = CellBindMatrixSource([first, second_cells])
+ np.testing.assert_array_equal(
+ cell_bound.read_cells(2, 4),
+ [[3, 6], [7, 8]],
+ )
+
+
+def test_layer_stitching_maps_global_axes_and_rejects_conflicts() -> None:
+ layer_one = DenseMatrixSource(
+ np.asarray([[1, 2], [3, 4]]),
+ row_names=["f1", "f2"],
+ column_names=["c1", "c3"],
+ )
+ layer_two = DenseMatrixSource(
+ np.asarray([[5], [6]]),
+ row_names=["f2", "f3"],
+ column_names=["c2"],
+ )
+ stitched = LayerStitchMatrixSource(
+ [layer_one, layer_two],
+ row_names=["f1", "f2", "f3"],
+ column_names=["c1", "c2", "c3"],
+ )
+ np.testing.assert_array_equal(
+ stitched.read_cells(0, 3).toarray(),
+ [[1, 3, 0], [0, 5, 6], [2, 4, 0]],
+ )
+
+ conflicting = DenseMatrixSource(
+ np.asarray([[9]]),
+ row_names=["f2"],
+ column_names=["c1"],
+ )
+ with pytest.raises(MatrixSourceError, match="coordinate conflict"):
+ LayerStitchMatrixSource(
+ [
+ LayerPlacement(layer_one, name="counts.1"),
+ LayerPlacement(conflicting, name="counts.2"),
+ ],
+ row_names=["f1", "f2", "f3"],
+ column_names=["c1", "c2", "c3"],
+ )
+
+
+def test_local_transforms_mask_and_subassignment() -> None:
+ values = np.asarray([[0, 2], [3, 0]], dtype=np.float64)
+ sparse_matrix = csc_matrix(values)
+ source = CscMatrixSource(
+ sparse_matrix.data,
+ sparse_matrix.indices,
+ sparse_matrix.indptr,
+ sparse_matrix.shape,
+ )
+ log_source = UnaryTransformMatrixSource(source, "log1p")
+ assert log_source.is_sparse
+ np.testing.assert_allclose(
+ log_source.read_cells(0, 2).toarray(),
+ np.log1p(values.T),
+ )
+
+ exp_source = UnaryTransformMatrixSource(source, "exp")
+ assert not exp_source.is_sparse
+ np.testing.assert_allclose(
+ exp_source.read_cells(0, 2),
+ np.exp(values.T),
+ )
+
+ doubled = BinaryTransformMatrixSource(source, 2, "multiply")
+ np.testing.assert_array_equal(doubled.read_cells(0, 2).toarray(), values.T * 2)
+ added = BinaryTransformMatrixSource(source, 1, "add")
+ assert not added.is_sparse
+ np.testing.assert_array_equal(added.read_cells(0, 2), values.T + 1)
+
+ mask = DenseMatrixSource(np.asarray([[True, False], [False, True]]))
+ masked = MaskMatrixSource(source, mask)
+ np.testing.assert_array_equal(
+ masked.read_cells(0, 2).toarray(),
+ [[0, 0], [0, 0]],
+ )
+
+ assigned = DelayedSubassignmentMatrixSource(
+ source,
+ [
+ Subassignment([0], [1], 7),
+ Subassignment([1], [0], np.asarray([[8]])),
+ ],
+ )
+ np.testing.assert_array_equal(assigned.read_cells(0, 2), [[0, 8], [7, 0]])
+
+
+def test_operation_registry_builds_structural_nodes() -> None:
+ source = DenseMatrixSource(
+ np.asarray([[1, 2], [3, 4]]),
+ row_names=["f1", "f2"],
+ column_names=["c1", "c2"],
+ )
+ subset = build_matrix_operation(
+ {
+ "operation": "subset",
+ "className": "DelayedSubset",
+ "featureIndices": [1],
+ "cellIndices": [1, 0],
+ },
+ source=source,
+ object_path="assays/RNA/counts",
+ )
+ np.testing.assert_array_equal(subset.read_cells(0, 2), [[4], [3]])
+
+ renamed = build_matrix_operation(
+ {
+ "operation": "dimnames",
+ "className": "DelayedSetDimnames",
+ "rowNames": ["a", "b"],
+ "columnNames": ["x", "y"],
+ },
+ source=source,
+ )
+ assert isinstance(renamed, RenamedMatrixSource)
+ assert renamed.row_names == ("a", "b")
+
+ cast = build_matrix_operation(
+ {
+ "operation": "dtype",
+ "className": "DtypeMatrix",
+ "dtype": "float32",
+ },
+ source=source,
+ )
+ assert isinstance(cast, DtypeMatrixSource)
+ assert cast.dtype == np.dtype(np.float32)
+
+ bound = build_matrix_operation(
+ {
+ "operation": "rbind",
+ "class": ["DelayedAbind", "DelayedMatrix"],
+ "sources": [source, source],
+ }
+ )
+ np.testing.assert_array_equal(
+ bound.read_cells(0, 2),
+ np.hstack([source.read_cells(0, 2), source.read_cells(0, 2)]),
+ )
+
+ transposed = build_matrix_operation(
+ {
+ "operation": "aperm",
+ "className": "DelayedAperm",
+ "permutation": [2, 1],
+ },
+ source=source,
+ )
+ np.testing.assert_array_equal(
+ transposed.read_cells(0, 2),
+ np.asarray([[1, 2], [3, 4]]),
+ )
+
+ mask = DenseMatrixSource(np.asarray([[1, 0], [0, 1]]))
+ dropped = build_matrix_operation(
+ {
+ "operation": "mask",
+ "className": "MatrixMask",
+ "mask": mask,
+ "invert": False,
+ },
+ source=source,
+ )
+ np.testing.assert_array_equal(dropped.read_cells(0, 2), [[0, 3], [2, 0]])
+
+
+def test_plain_slot_mapping_factory_has_no_rds_parser_dependency(
+ tmp_path: Path,
+) -> None:
+ dense = matrix_source_from_slots(
+ {
+ "class": ["matrix", "array"],
+ "slots": {
+ ".Data": [1, 3, 2, 4],
+ "dim": [2, 2],
+ "dimnames": [["f1", "f2"], ["c1", "c2"]],
+ },
+ },
+ object_path="assays/RNA/counts",
+ )
+ np.testing.assert_array_equal(dense.read_cells(0, 2), [[1, 3], [2, 4]])
+
+ sparse = csc_matrix(np.asarray([[1, 0], [0, 2]]))
+ csc_source = matrix_source_from_slots(
+ {
+ "className": "dgCMatrix",
+ "x": sparse.data,
+ "i": sparse.indices,
+ "p": sparse.indptr,
+ "Dim": sparse.shape,
+ }
+ )
+ np.testing.assert_array_equal(
+ csc_source.read_cells(0, 2).toarray(), [[1, 0], [0, 2]]
+ )
+ wrapped = matrix_source_from_slots(
+ {
+ "class": ["Iterable_dgCMatrix_wrapper", "IterableMatrix"],
+ "slots": {
+ "mat": {
+ "className": "dgCMatrix",
+ "x": sparse.data,
+ "i": sparse.indices,
+ "p": sparse.indptr,
+ "Dim": sparse.shape,
+ },
+ "dim": sparse.shape,
+ "transpose": False,
+ },
+ }
+ )
+ np.testing.assert_array_equal(wrapped.read_cells(0, 2).toarray(), [[1, 0], [0, 2]])
+
+ rds = tmp_path / "object.rds"
+ rds.write_bytes(b"rds")
+ sidecar = tmp_path / "counts.h5"
+ with h5py.File(sidecar, mode="w") as handle:
+ handle.create_dataset("counts", data=np.asarray([[1, 2], [3, 4]]))
+ hdf5_source = matrix_source_from_slots(
+ {
+ "class": "HDF5ArraySeed",
+ "filepath": "counts.h5",
+ "name": "counts",
+ },
+ rds_path=rds,
+ )
+ np.testing.assert_array_equal(hdf5_source.read_cells(0, 2), [[1, 2], [3, 4]])
+
+
+def test_factory_executes_structural_delayedarray_nodes() -> None:
+ leaf = {
+ "class": ["matrix", "array"],
+ "slots": {
+ ".Data": [1, 2, 3, 4, 5, 6],
+ "dim": [3, 2],
+ },
+ }
+ delayed = {
+ "class": ["DelayedMatrix", "DelayedArray"],
+ "slots": {"seed": leaf},
+ }
+ source = matrix_source_from_slots(delayed, object_path="assays/RNA/counts")
+ np.testing.assert_array_equal(
+ source.read_cells(0, 2),
+ [[1, 2, 3], [4, 5, 6]],
+ )
+
+ subset = matrix_source_from_slots(
+ {
+ "class": ["DelayedSubset", "DelayedUnaryOp"],
+ "slots": {
+ "seed": delayed,
+ "index": [[3, 1], [2]],
+ },
+ },
+ object_path="assays/RNA/subset",
+ )
+ np.testing.assert_array_equal(subset.read_cells(0, 1), [[6, 4]])
+
+ transposed = matrix_source_from_slots(
+ {
+ "class": ["DelayedAperm", "DelayedUnaryOp"],
+ "slots": {
+ "seed": delayed,
+ "perm": [2, 1],
+ },
+ },
+ object_path="assays/RNA/aperm",
+ )
+ np.testing.assert_array_equal(
+ transposed.read_cells(0, 3),
+ [[1, 4], [2, 5], [3, 6]],
+ )
+
+ bound = matrix_source_from_slots(
+ {
+ "class": ["DelayedAbind", "DelayedNaryOp"],
+ "slots": {
+ "seeds": [delayed, delayed],
+ "along": [2],
+ },
+ },
+ object_path="assays/RNA/abind",
+ )
+ np.testing.assert_array_equal(
+ bound.read_cells(0, 4),
+ [[1, 2, 3], [4, 5, 6], [1, 2, 3], [4, 5, 6]],
+ )
+
+ assigned = matrix_source_from_slots(
+ {
+ "class": ["DelayedSubassign", "DelayedUnaryIsoOp"],
+ "slots": {
+ "seed": delayed,
+ "Lindex": [[2], [1, 2]],
+ "Rvalue": [9],
+ },
+ },
+ object_path="assays/RNA/subassign",
+ )
+ np.testing.assert_array_equal(
+ assigned.read_cells(0, 2),
+ [[1, 9, 3], [4, 9, 6]],
+ )
+
+
+def test_factory_executes_allowlisted_delayedarray_primitives() -> None:
+ source = DenseMatrixSource(np.asarray([[1, 2], [3, 4]], dtype=np.float64))
+ right_added = matrix_source_from_slots(
+ {
+ "class": ["DelayedUnaryIsoOpWithArgs", "DelayedUnaryIsoOp"],
+ "slots": {
+ "seed": source,
+ "OP": "+",
+ "Largs": [],
+ "Rargs": [2],
+ },
+ }
+ )
+ np.testing.assert_array_equal(right_added.read_cells(0, 2), [[3, 5], [4, 6]])
+
+ left_subtracted = matrix_source_from_slots(
+ {
+ "class": ["DelayedUnaryIsoOpWithArgs", "DelayedUnaryIsoOp"],
+ "slots": {
+ "seed": source,
+ "OP": "-",
+ "Largs": [10],
+ "Rargs": [],
+ },
+ }
+ )
+ np.testing.assert_array_equal(
+ left_subtracted.read_cells(0, 2),
+ [[9, 7], [8, 6]],
+ )
+
+ stacked = matrix_source_from_slots(
+ {
+ "class": ["DelayedUnaryIsoOpStack", "DelayedUnaryIsoOp"],
+ "slots": {
+ "seed": DenseMatrixSource(-np.square(np.asarray([[1, 2], [3, 4]]))),
+ "OPS": ["abs", "sqrt"],
+ },
+ }
+ )
+ np.testing.assert_array_equal(stacked.read_cells(0, 2), [[1, 3], [2, 4]])
+
+ nary = matrix_source_from_slots(
+ {
+ "class": ["DelayedNaryIsoOp", "DelayedNaryOp"],
+ "slots": {
+ "seeds": [source, source],
+ "OP": "+",
+ "Rargs": [1],
+ },
+ }
+ )
+ np.testing.assert_array_equal(nary.read_cells(0, 2), [[3, 7], [5, 9]])
+
+
+def test_factory_resolves_nested_bpcells_operations_and_fragments(
+ tmp_path: Path,
+) -> None:
+ nested = matrix_source_from_slots(
+ {
+ "class": ["MatrixSubset", "IterableMatrix"],
+ "slots": {
+ "matrix": {
+ "class": [
+ "TransformLog1p",
+ "TransformedMatrix",
+ "IterableMatrix",
+ ],
+ "slots": {
+ "matrix": {
+ "class": ["matrix", "array"],
+ "slots": {
+ ".Data": [1, 2, 3, 4, 5, 6],
+ "dim": [3, 2],
+ },
+ },
+ "dim": [3, 2],
+ },
+ },
+ "row_selection": [3, 1],
+ "col_selection": [2],
+ "zero_dims": [False, False],
+ "dim": [2, 1],
+ "dimnames": [["f3", "f1"], ["c2"]],
+ },
+ },
+ object_path="assays/RNA/data",
+ )
+ np.testing.assert_allclose(nested.read_cells(0, 1), np.log1p([[6, 4]]))
+ assert nested.row_names == ("f3", "f1")
+ assert nested.column_names == ("c2",)
+
+ fragment = matrix_source_from_slots(
+ _peak_matrix_spec(
+ _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ ),
+ "fragments",
+ ),
+ object_path="assays/ATAC/counts",
+ )
+ assert isinstance(fragment, RenamedMatrixSource)
+ assert isinstance(fragment.source, FragmentDerivedMatrixSource)
+ np.testing.assert_array_equal(
+ fragment.read_cells(1, 3).toarray(),
+ [
+ [1, 0, 1, 2, 0],
+ [0, 1, 1, 1, 1],
+ ],
+ )
+
+
+def test_factory_uses_dimensions_to_separate_storage_order_from_transpose(
+ tmp_path: Path,
+) -> None:
+ values = np.asarray([[1, 0, 2], [0, 3, 4]], dtype=np.uint32)
+ payload = _bpcells_payload(
+ values,
+ packed=True,
+ version=2,
+ storage_order="row",
+ )
+ matrix_dir = tmp_path / "row-matrix"
+ _write_bpcells_directory(matrix_dir, payload, version=2)
+
+ source = matrix_source_from_slots(
+ {
+ "class": ["MatrixDir", "IterableMatrix"],
+ "slots": {
+ "dir": matrix_dir,
+ "dim": list(values.shape),
+ "dimnames": [payload["row_names"], payload["col_names"]],
+ "transpose": True,
+ },
+ }
+ )
+ np.testing.assert_array_equal(source.read_cells(0, 3).toarray(), values.T)
+
+ transposed = matrix_source_from_slots(
+ {
+ "class": ["MatrixDir", "IterableMatrix"],
+ "slots": {
+ "dir": matrix_dir,
+ "dim": list(values.shape[::-1]),
+ "dimnames": [payload["col_names"], payload["row_names"]],
+ "transpose": False,
+ },
+ }
+ )
+ np.testing.assert_array_equal(transposed.read_cells(0, 2).toarray(), values)
+
+
+def test_unknown_registry_function_and_class_report_object_path() -> None:
+ source = DenseMatrixSource(np.eye(2))
+ with pytest.raises(UnsupportedMatrixOperation) as function_error:
+ build_matrix_operation(
+ {
+ "operation": "unary",
+ "className": "DelayedUnaryIsoOpStack",
+ "function": "custom_package_function",
+ },
+ source=source,
+ object_path="assays/RNA/layers/custom",
+ )
+ message = str(function_error.value)
+ assert "assays/RNA/layers/custom" in message
+ assert "custom_package_function" in message
+ assert "DelayedUnaryIsoOpStack" in message
+
+ with pytest.raises(UnsupportedMatrixOperation) as class_error:
+ build_matrix_operation(
+ {"operation": "subset", "className": "CustomSeed"},
+ source=source,
+ object_path="assays/RNA/counts",
+ )
+ message = str(class_error.value)
+ assert "assays/RNA/counts" in message
+ assert "CustomSeed" in message
+ assert "subset" in message
+
+
+def test_fragment_wrapper_reports_nested_custom_source_path() -> None:
+ with pytest.raises(UnsupportedMatrixOperation) as error:
+ matrix_source_from_slots(
+ {
+ "class": ["PeakMatrix", "IterableMatrix"],
+ "slots": {
+ "fragments": {
+ "class": ["ShiftFragments", "IterableFragments"],
+ "slots": {
+ "fragments": {
+ "class": ["CustomFragments", "IterableFragments"],
+ "slots": {},
+ },
+ "shift_start": [4],
+ "shift_end": [-4],
+ },
+ },
+ "chr_id": [0],
+ "start": [0],
+ "end": [10],
+ "chr_levels": ["chr1"],
+ "mode": ["insertions"],
+ "transpose": True,
+ "dim": [1, 1],
+ },
+ },
+ object_path="assays/ATAC/counts",
+ )
+ assert error.value.objectPath == "assays/ATAC/counts@fragments@fragments"
+ assert error.value.className == "CustomFragments"
+ assert error.value.reason == "unknown or custom fragment class"
+
+
+def test_custom_fragment_graph_rejection_has_exact_object_path() -> None:
+ with pytest.raises(UnsupportedMatrixOperation) as error:
+ matrix_source_from_slots(
+ {
+ "class": ["PeakMatrix", "IterableMatrix"],
+ "slots": {
+ "fragments": {
+ "class": ["CustomFragments", "IterableFragments"],
+ "slots": {"loader": "custom::open_fragments"},
+ },
+ "chr_id": [0],
+ "start": [0],
+ "end": [10],
+ "chr_levels": ["chr1"],
+ "mode": ["insertions"],
+ "transpose": True,
+ "dim": [1, 1],
+ },
+ },
+ object_path="assays/ATAC/counts",
+ )
+
+ assert error.value.objectPath == "assays/ATAC/counts@fragments"
+ assert error.value.className == "CustomFragments"
+ assert error.value.reason == "unknown or custom fragment class"
+
+
+def test_fragment_capability_registry_has_exact_supported_profile() -> None:
+ assert FRAGMENT_CAPABILITY_REGISTRY.acceptedClasses == (
+ "UnpackedMemFragments",
+ "PackedMemFragments",
+ "FragmentsDir",
+ "FragmentsHDF5",
+ "ShiftFragments",
+ "SelectLength",
+ "ChrSelectName",
+ "ChrSelectIndex",
+ "CellSelectName",
+ "CellSelectIndex",
+ "CellMerge",
+ "ChrRename",
+ "CellRename",
+ "CellPrefix",
+ "RegionSelect",
+ "MergeFragments",
+ )
+
+
+@pytest.mark.parametrize("backend", ["memory", "directory", "hdf5"])
+@pytest.mark.parametrize("packed", [False, True])
+@pytest.mark.parametrize("version", [1, 2])
+def test_fragment_leaves_and_all_formal_layouts_execute_through_factory(
+ tmp_path: Path,
+ backend: str,
+ packed: bool,
+ version: int,
+) -> None:
+ source = matrix_source_from_slots(
+ _peak_matrix_spec(
+ _fragment_leaf_spec(
+ tmp_path,
+ backend=backend,
+ packed=packed,
+ version=version,
+ ),
+ "overlaps",
+ ),
+ object_path="assays/ATAC/counts",
+ )
+
+ assert source.shape == (5, 3)
+ assert source.dtype == np.dtype(np.uint32)
+ assert source.row_names == ("p0", "p_span", "p1", "p2", "p_chr2")
+ assert source.column_names == ("c1", "c2", "c3")
+ np.testing.assert_array_equal(
+ source.read_cells(1, 3).toarray(),
+ [
+ [1, 1, 1, 2, 0],
+ [0, 1, 1, 1, 1],
+ ],
+ )
+
+
+def test_fragment_records_are_read_in_bounded_chromosome_blocks() -> None:
+ count = 130
+ cells = np.ones(count, dtype=np.int32)
+ cells[0] = 0
+ starts = np.arange(count, dtype=np.int32)
+ ends = starts + 1
+ tracked_cells = _TrackedVector(cells)
+ tracked_starts = _TrackedVector(starts)
+ tracked_ends = _TrackedVector(ends)
+ source = matrix_source_from_slots(
+ {
+ "class": ["PeakMatrix", "IterableMatrix"],
+ "slots": {
+ "fragments": {
+ "class": ["UnpackedMemFragments", "IterableFragments"],
+ "slots": {
+ "version": ["unpacked-fragments-v2"],
+ "cell": tracked_cells,
+ "start": tracked_starts,
+ "end": tracked_ends,
+ "end_max": np.asarray([128, 130], dtype=np.int32),
+ "chr_ptr": np.asarray([0, count], dtype=np.float64),
+ "chr_names": ("chr1",),
+ "cell_names": ("c1", "c2"),
+ },
+ },
+ "chr_id": np.asarray([0], dtype=np.int32),
+ "start": np.asarray([0], dtype=np.int32),
+ "end": np.asarray([1], dtype=np.int32),
+ "chr_levels": ("chr1",),
+ "mode": ["insertions"],
+ "transpose": True,
+ "dim": (1, 2),
+ "dimnames": (("p1",), ("c1", "c2")),
+ },
+ },
+ limits=SourceLimits(maxBlockBytes=16_384),
+ )
+
+ np.testing.assert_array_equal(source.read_cells(0, 1).toarray(), [[2]])
+ for tracked in (tracked_cells, tracked_starts, tracked_ends):
+ assert len(tracked.reads) > 1
+ assert all(
+ read_start is not None
+ and read_stop is not None
+ and read_stop - read_start <= 128
+ for read_start, read_stop in tracked.reads
+ )
+
+
+def test_fragment_end_max_matches_bpcells_at_aligned_chromosome_boundary() -> None:
+ first_starts = np.arange(128, dtype=np.uint32)
+ starts = np.concatenate((first_starts, np.asarray([0, 1], dtype=np.uint32)))
+ ends = np.concatenate(
+ (first_starts + np.uint32(1000), np.asarray([1, 2], dtype=np.uint32))
+ )
+ source = matrix_source_from_slots(
+ {
+ "class": ["PeakMatrix", "IterableMatrix"],
+ "slots": {
+ "fragments": {
+ "class": ["UnpackedMemFragments", "IterableFragments"],
+ "slots": {
+ "version": ["unpacked-fragments-v2"],
+ "cell": np.zeros(130, dtype=np.int32),
+ "start": starts.view(np.int32),
+ "end": ends.view(np.int32),
+ "end_max": np.asarray([1127, 1127], dtype=np.int32),
+ "chr_ptr": np.asarray(
+ [0, 128, 128, 130],
+ dtype=np.float64,
+ ),
+ "chr_names": ("chr1", "chr2"),
+ "cell_names": ("c1",),
+ },
+ },
+ "chr_id": np.asarray([1], dtype=np.int32),
+ "start": np.asarray([0], dtype=np.int32),
+ "end": np.asarray([2], dtype=np.int32),
+ "chr_levels": ("chr1", "chr2"),
+ "mode": ["insertions"],
+ "transpose": True,
+ "dim": (1, 1),
+ "dimnames": (("p1",), ("c1",)),
+ },
+ }
+ )
+
+ np.testing.assert_array_equal(source.read_cells(0, 1).toarray(), [[4]])
+
+
+@pytest.mark.parametrize(
+ "mode,expected",
+ [
+ (
+ "insertions",
+ [
+ [2, 0, 2, 3, 1],
+ [1, 0, 1, 3, 0],
+ [0, 1, 2, 2, 2],
+ ],
+ ),
+ (
+ "fragments",
+ [
+ [1, 0, 1, 2, 1],
+ [1, 0, 1, 2, 0],
+ [0, 1, 1, 1, 1],
+ ],
+ ),
+ (
+ "overlaps",
+ [
+ [1, 1, 1, 2, 1],
+ [1, 1, 1, 2, 0],
+ [0, 1, 1, 1, 1],
+ ],
+ ),
+ ],
+)
+def test_peak_matrix_modes_match_bpcells_endpoint_semantics(
+ tmp_path: Path,
+ mode: str,
+ expected: list[list[int]],
+) -> None:
+ source = matrix_source_from_slots(
+ _peak_matrix_spec(
+ _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=True,
+ version=2,
+ ),
+ mode,
+ )
+ )
+ np.testing.assert_array_equal(source.read_cells(0, 3).toarray(), expected)
+
+
+@pytest.mark.parametrize(
+ "mode,expected",
+ [
+ (
+ "insertions",
+ [
+ [2, 2, 0, 1, 1],
+ [1, 1, 2, 0, 0],
+ [0, 2, 0, 2, 0],
+ ],
+ ),
+ (
+ "fragments",
+ [
+ [1, 1, 0, 1, 0],
+ [1, 0, 1, 0, 0],
+ [0, 1, 0, 1, 0],
+ ],
+ ),
+ ],
+)
+def test_tile_matrix_modes_match_bpcells_duplicate_endpoint_handling(
+ tmp_path: Path,
+ mode: str,
+ expected: list[list[int]],
+) -> None:
+ source = matrix_source_from_slots(
+ _tile_matrix_spec(
+ _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=1,
+ ),
+ mode,
+ )
+ )
+ np.testing.assert_array_equal(source.read_cells(0, 3).toarray(), expected)
+
+
+def test_fragment_matrix_honors_native_storage_orientation(tmp_path: Path) -> None:
+ source = matrix_source_from_slots(
+ _peak_matrix_spec(
+ _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ ),
+ "fragments",
+ transpose=False,
+ )
+ )
+
+ assert source.shape == (3, 5)
+ assert source.row_names == ("c1", "c2", "c3")
+ assert source.column_names == ("p0", "p_span", "p1", "p2", "p_chr2")
+ np.testing.assert_array_equal(
+ source.read_cells(1, 4).toarray(),
+ [
+ [0, 0, 1],
+ [1, 1, 1],
+ [2, 2, 1],
+ ],
+ )
+
+
+def test_fragment_metadata_is_validated_before_reads(tmp_path: Path) -> None:
+ malformed = _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ )
+ malformed["slots"]["end_max"] = np.asarray([29], dtype=np.int32)
+ with pytest.raises(MatrixSourceError, match="end_max is inconsistent at block 0"):
+ matrix_source_from_slots(
+ _peak_matrix_spec(malformed, "insertions"),
+ object_path="assays/ATAC/counts",
+ )
+
+ unsorted = _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ )
+ unsorted["slots"]["start"] = np.asarray(
+ [0, 5, 10, 12, 20, 4, 1],
+ dtype=np.int32,
+ )
+ with pytest.raises(MatrixSourceError, match="not sorted on chromosome 1"):
+ matrix_source_from_slots(_peak_matrix_spec(unsorted, "insertions"))
+
+ mismatched_levels = _peak_matrix_spec(
+ _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ ),
+ "insertions",
+ )
+ mismatched_levels["slots"]["chr_levels"] = ("chr2", "chr1")
+ with pytest.raises(MatrixSourceError, match="do not match the fragment source"):
+ matrix_source_from_slots(mismatched_levels)
+
+ invalid_cell = _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ )
+ invalid_cell["slots"]["cell"] = np.asarray(
+ [3, 0, 1, 0, 2, 2, 1],
+ dtype=np.int32,
+ )
+ with pytest.raises(MatrixSourceError, match="cell ID is out of range at record 0"):
+ matrix_source_from_slots(_peak_matrix_spec(invalid_cell, "insertions"))
+
+ invalid_coordinates = _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ )
+ invalid_coordinates["slots"]["end"] = np.asarray(
+ [3, 4, 7, 10, 10, 4, 12],
+ dtype=np.int32,
+ )
+ with pytest.raises(MatrixSourceError, match="end precedes start at record 1"):
+ matrix_source_from_slots(_peak_matrix_spec(invalid_coordinates, "insertions"))
+
+ invalid_pointers = _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ )
+ invalid_pointers["slots"]["chr_ptr"] = np.asarray(
+ [0, 4, 5, 7],
+ dtype=np.float64,
+ )
+ with pytest.raises(MatrixSourceError, match="chromosome 1 starts at 5; expected 4"):
+ matrix_source_from_slots(_peak_matrix_spec(invalid_pointers, "insertions"))
+
+ invalid_chromosome = _peak_matrix_spec(
+ _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ ),
+ "insertions",
+ )
+ invalid_chromosome["slots"]["chr_id"] = np.asarray(
+ [0, 0, 0, 0, 2],
+ dtype=np.int32,
+ )
+ with pytest.raises(MatrixSourceError, match="out-of-range chromosome ID"):
+ matrix_source_from_slots(invalid_chromosome)
+
+ invalid_dimensions = _peak_matrix_spec(
+ _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ ),
+ "insertions",
+ )
+ invalid_dimensions["slots"]["dim"] = (4, 3)
+ with pytest.raises(MatrixSourceError, match="does not match derived shape"):
+ matrix_source_from_slots(invalid_dimensions)
+
+ invalid_compression = _fragment_leaf_spec(
+ tmp_path,
+ backend="directory",
+ packed=False,
+ version=2,
+ )
+ invalid_compression["slots"]["compressed"] = True
+ with pytest.raises(MatrixSourceError, match="compressed slot.*disagrees"):
+ matrix_source_from_slots(_peak_matrix_spec(invalid_compression, "insertions"))
+
+
+@pytest.mark.parametrize(
+ "limits,match",
+ [
+ (SourceLimits(maxFeatures=4), "maxFeatures=4"),
+ (SourceLimits(maxCells=2), "maxCells=2"),
+ (SourceLimits(maxNnz=6), "maxNnz=6"),
+ (SourceLimits(maxMetadataBytes=64), "maxMetadataBytes=64"),
+ (SourceLimits(maxBlockBytes=1024), "maxBlockBytes=1024"),
+ ],
+)
+def test_fragment_sources_enforce_resource_limits_at_inspection(
+ tmp_path: Path,
+ limits: SourceLimits,
+ match: str,
+) -> None:
+ with pytest.raises(ResourceLimitError, match=match):
+ matrix_source_from_slots(
+ _peak_matrix_spec(
+ _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=True,
+ version=2,
+ ),
+ "overlaps",
+ ),
+ limits=limits,
+ )
+
+
+def test_fragment_output_enforces_nnz_limit(tmp_path: Path) -> None:
+ source = matrix_source_from_slots(
+ _peak_matrix_spec(
+ _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ ),
+ "overlaps",
+ ),
+ limits=SourceLimits(maxNnz=7),
+ )
+ with pytest.raises(ResourceLimitError, match="maxNnz=7"):
+ source.read_cells(0, 3)
+
+
+def test_documented_fragment_wrappers_execute(tmp_path: Path) -> None:
+ base = _fragment_leaf_spec(
+ tmp_path,
+ backend="memory",
+ packed=False,
+ version=2,
+ )
+
+ def wrap(class_name: str, **slots: Any):
+ return fragment_source_from_slots(
+ {
+ "class": [class_name, "IterableFragments"],
+ "slots": {"fragments": base, **slots},
+ }
+ )
+
+ shifted = wrap("ShiftFragments", shift_start=1, shift_end=2)
+ first_original = next(fragment_source_from_slots(base).iter_chromosome(0))
+ first_shifted = next(shifted.iter_chromosome(0))
+ np.testing.assert_array_equal(first_shifted.starts, first_original.starts + 1)
+ np.testing.assert_array_equal(first_shifted.ends, first_original.ends + 2)
+
+ length_selected = wrap("SelectLength", min_len=10, max_len=12)
+ assert all(
+ np.all((block.ends - block.starts >= 10) & (block.ends - block.starts <= 12))
+ for chromosome in range(len(length_selected.chromosomeNames))
+ for block in length_selected.iter_chromosome(chromosome)
+ )
+
+ chromosome_by_name = wrap("ChrSelectName", chr_names=["chr2"])
+ chromosome_by_index = wrap("ChrSelectIndex", chr_index_selection=[2])
+ assert chromosome_by_name.chromosomeNames == ("chr2",)
+ assert chromosome_by_index.chromosomeNames == ("chr2",)
+
+ cells_by_name = wrap("CellSelectName", cell_names=["c3", "c1"])
+ cells_by_index = wrap("CellSelectIndex", cell_index_selection=[3, 1])
+ assert cells_by_name.cellNames == ("c3", "c1")
+ assert cells_by_index.cellNames == ("c3", "c1")
+
+ merged_cells = wrap(
+ "CellMerge",
+ group_ids=[0, 0, 1],
+ group_names=["ab", "c"],
+ )
+ assert merged_cells.cellNames == ("ab", "c")
+
+ renamed_chromosomes = wrap(
+ "ChrRename",
+ chr_names=["one", "two"],
+ )
+ renamed_cells = wrap(
+ "CellRename",
+ cell_names=["one", "two", "three"],
+ )
+ prefixed_cells = wrap("CellPrefix", prefix="sample_")
+ assert renamed_chromosomes.chromosomeNames == ("one", "two")
+ assert renamed_cells.cellNames == ("one", "two", "three")
+ assert prefixed_cells.cellNames == ("sample_c1", "sample_c2", "sample_c3")
+
+ selected_regions = wrap(
+ "RegionSelect",
+ chr_id=[0],
+ start=[0],
+ end=[10],
+ chr_levels=["chr1", "chr2"],
+ invert_selection=False,
+ )
+ assert all(
+ np.all((block.starts <= 10) & (block.ends >= 0))
+ for block in selected_regions.iter_chromosome(0)
+ )
+
+ merged = fragment_source_from_slots(
+ {
+ "class": ["MergeFragments", "IterableFragments"],
+ "slots": {"fragments_list": [base, base]},
+ }
+ )
+ assert merged.chromosomeNames == ("chr1", "chr2")
+ assert merged.cellNames == ("c1", "c2", "c3", "c1", "c2", "c3")
+ assert all(
+ int(block.cellIds.max(initial=0)) < len(merged.cellNames)
+ for chromosome in range(len(merged.chromosomeNames))
+ for block in merged.iter_chromosome(chromosome)
+ )
+
+
+@pytest.mark.parametrize("class_name", ["FragmentsTsv", "IterableFragments"])
+def test_unsupported_fragment_sources_are_typed_rejections(class_name: str) -> None:
+ with pytest.raises(UnsupportedMatrixOperation) as error:
+ fragment_source_from_slots(
+ {
+ "class": [class_name, "IterableFragments"],
+ "slots": {},
+ },
+ object_path="assays/ATAC/counts@fragments",
+ )
+ assert error.value.objectPath == "assays/ATAC/counts@fragments"
+ assert error.value.className == class_name
+ assert error.value.operation == "fragment-source"
+
+
+def test_rank_transform_matches_bpcells_zero_offset_and_ties() -> None:
+ dense = DenseMatrixSource(
+ np.array(
+ [
+ [0, -1, 2],
+ [2, 0, 1],
+ [2, 0, 0],
+ [5, 3, 0],
+ ],
+ dtype=np.float64,
+ )
+ )
+ ranked = RankMatrixSource(dense)
+ np.testing.assert_array_equal(
+ ranked.read_cells(0, 3),
+ np.array(
+ [
+ [0, 1.5, 1.5, 3],
+ [-1.5, 0, 0, 1.5],
+ [2.5, 1.5, 0, 0],
+ ],
+ dtype=np.float64,
+ ),
+ )
+
+ sparse = CscMatrixSource(
+ np.array([2.0, -1.0, 2.0, 3.0]),
+ np.array([1, 3, 2, 3]),
+ np.array([0, 2, 4]),
+ (4, 2),
+ )
+ sparse_ranked = RankMatrixSource(sparse)
+ assert sparse_ranked.is_sparse
+ np.testing.assert_array_equal(
+ sparse_ranked.read_cells(0, 2).toarray(),
+ np.array(
+ [
+ [0, 1.5, 0, -1.5],
+ [0, 0, 1.5, 2.5],
+ ],
+ dtype=np.float64,
+ ),
+ )
+
+
+def test_rank_transform_factory_and_nonlocal_axis_contract() -> None:
+ source = DenseMatrixSource(np.eye(3))
+ ranked = matrix_source_from_slots(
+ {
+ "class": "MatrixRankTransform",
+ "matrix": source,
+ "transpose": False,
+ "dim": (3, 3),
+ },
+ object_path="assays/RNA/counts",
+ )
+ assert isinstance(ranked, RankMatrixSource)
+ np.testing.assert_array_equal(ranked.read_cells(0, 3), np.eye(3) * 1.5)
+
+ row_ranked = matrix_source_from_slots(
+ {
+ "class": "MatrixRankTransform",
+ "matrix": source,
+ "transpose": True,
+ },
+ object_path="assays/RNA/counts",
+ limits=SourceLimits(tileCells=1),
+ )
+ np.testing.assert_array_equal(row_ranked.read_cells(1, 3), np.eye(3)[1:] * 1.5)
+
+
+def test_row_rank_streams_dense_and_sparse_sources() -> None:
+ dense = DenseMatrixSource(
+ np.array(
+ [
+ [0, -1, 2],
+ [2, 0, 1],
+ [2, 0, 0],
+ [5, 3, 0],
+ ],
+ dtype=np.float64,
+ )
+ )
+ dense_ranked = RankMatrixSource(
+ dense,
+ axis="row",
+ limits=SourceLimits(tileCells=1),
+ )
+ np.testing.assert_array_equal(
+ dense_ranked.read_cells(1, 3),
+ [[-1, 0, 0, 1], [1, 1, 0, 0]],
+ )
+
+ sparse = CscMatrixSource(
+ np.array([2.0, -1.0, 2.0, 3.0]),
+ np.array([1, 3, 2, 3]),
+ np.array([0, 2, 4]),
+ (4, 2),
+ )
+ sparse_ranked = RankMatrixSource(
+ sparse,
+ axis="row",
+ limits=SourceLimits(tileCells=1),
+ )
+ np.testing.assert_array_equal(
+ sparse_ranked.read_cells(0, 2).toarray(),
+ [[0, 1, 0, 1], [0, 0, 1, 2]],
+ )
+
+
+def test_bpcells_parameterized_numeric_transforms_execute_through_factory() -> None:
+ values = np.asarray([[1, 2, 3], [4, 5, 6]], dtype=np.float64)
+ source = DenseMatrixSource(values)
+
+ minimum = matrix_source_from_slots(
+ {
+ "class": ["TransformMinByRow", "TransformedMatrix"],
+ "slots": {
+ "matrix": source,
+ "row_params": [[2, 5]],
+ "dim": source.shape,
+ "transpose": False,
+ },
+ }
+ )
+ np.testing.assert_array_equal(
+ minimum.read_cells(0, 3),
+ [[1, 4], [2, 5], [2, 5]],
+ )
+ column_minimum = matrix_source_from_slots(
+ {
+ "class": ["TransformMinByCol", "TransformedMatrix"],
+ "slots": {
+ "matrix": source,
+ "col_params": [[2, 4, 10]],
+ "dim": source.shape,
+ "transpose": False,
+ },
+ }
+ )
+ np.testing.assert_array_equal(
+ column_minimum.read_cells(0, 3),
+ [[1, 2], [2, 4], [3, 6]],
+ )
+
+ active = np.ones((3, 2), dtype=bool)
+ row_parameters = np.asarray([[2, 3], [1, 2]], dtype=np.float64)
+ column_parameters = np.asarray(
+ [[1, 10, 100], [0, 5, 10]],
+ dtype=np.float64,
+ )
+ scaled = matrix_source_from_slots(
+ {
+ "class": ["TransformScaleShift", "TransformedMatrix"],
+ "slots": {
+ "matrix": source,
+ "row_params": row_parameters,
+ "col_params": column_parameters,
+ "global_params": [0.5, -1],
+ "active_transforms": active,
+ "dim": source.shape,
+ "transpose": False,
+ },
+ }
+ )
+ expected_scaled = values.T
+ expected_scaled = (
+ expected_scaled * row_parameters[0] * column_parameters[0, :, np.newaxis] * 0.5
+ + row_parameters[1]
+ + column_parameters[1, :, np.newaxis]
+ - 1
+ )
+ np.testing.assert_array_equal(scaled.read_cells(0, 3), expected_scaled)
+
+ theta_inverse = np.asarray([0.1, 0.2])
+ gene_beta = np.asarray([0.5, 1.0])
+ cell_reads = np.asarray([2.0, 3.0, 4.0])
+ mu = cell_reads[:, np.newaxis] * gene_beta
+ expected_pearson = (values.T - mu) / np.sqrt(mu + mu * mu * theta_inverse)
+ for class_name in ("SCTransformPearson", "SCTransformPearsonSlow"):
+ pearson = matrix_source_from_slots(
+ {
+ "class": [class_name, "TransformedMatrix"],
+ "slots": {
+ "matrix": source,
+ "row_params": np.vstack([theta_inverse, gene_beta]),
+ "col_params": cell_reads[np.newaxis, :],
+ "global_params": [np.inf, -10, 10],
+ "dim": source.shape,
+ "transpose": False,
+ },
+ }
+ )
+ np.testing.assert_allclose(pearson.read_cells(0, 3), expected_pearson)
+ for class_name in (
+ "SCTransformPearsonTranspose",
+ "SCTransformPearsonTransposeSlow",
+ ):
+ pearson = matrix_source_from_slots(
+ {
+ "class": [class_name, "TransformedMatrix"],
+ "slots": {
+ "matrix": source,
+ "row_params": cell_reads[np.newaxis, :],
+ "col_params": np.vstack([theta_inverse, gene_beta]),
+ "global_params": [np.inf, -10, 10],
+ "dim": source.shape,
+ "transpose": True,
+ },
+ }
+ )
+ np.testing.assert_allclose(pearson.read_cells(0, 3), expected_pearson)
+
+ feature_parameters = np.asarray([[1, 2], [3, 4]], dtype=np.float64)
+ cell_parameters = np.asarray(
+ [[1, 0, 2], [0, 1, 1]],
+ dtype=np.float64,
+ )
+ residual = matrix_source_from_slots(
+ {
+ "class": ["TransformLinearResidual", "TransformedMatrix"],
+ "slots": {
+ "matrix": source,
+ "row_params": feature_parameters,
+ "col_params": cell_parameters,
+ "dim": source.shape,
+ "transpose": False,
+ },
+ }
+ )
+ np.testing.assert_array_equal(
+ residual.read_cells(1, 3),
+ values.T[1:3] - cell_parameters[:, 1:3].T @ feature_parameters,
+ )
+
+ rounded = matrix_source_from_slots(
+ {
+ "class": ["TransformRound", "TransformedMatrix"],
+ "slots": {
+ "matrix": DenseMatrixSource(np.asarray([[1.24, 1.26], [2.55, 2.54]])),
+ "global_params": [1],
+ "dim": (2, 2),
+ },
+ }
+ )
+ np.testing.assert_array_equal(
+ rounded.read_cells(0, 2),
+ [[1.2, 2.6], [1.3, 2.5]],
+ )
+
+
+def test_matrix_multiply_streams_output_cells_and_inner_tiles() -> None:
+ left = DenseMatrixSource(
+ np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32),
+ row_names=("g1", "g2"),
+ column_names=("k1", "k2", "k3"),
+ )
+ right = DenseMatrixSource(
+ np.array([[7, 8], [9, 10], [11, 12]], dtype=np.float32),
+ row_names=("k1", "k2", "k3"),
+ column_names=("c1", "c2"),
+ )
+ source = MatrixMultiplySource(
+ left,
+ right=right,
+ limits=SourceLimits(tileCells=2),
+ )
+
+ assert source.shape == (2, 2)
+ assert source.row_names == ("g1", "g2")
+ assert source.column_names == ("c1", "c2")
+ np.testing.assert_array_equal(
+ source.read_cells(0, 2),
+ np.array([[58, 139], [64, 154]], dtype=np.float32),
+ )
+ np.testing.assert_array_equal(source.read_cells(1, 2), [[64, 154]])
+
+ built = build_matrix_operation(
+ {
+ "operation": "multiply",
+ "className": "MatrixMultiply",
+ "right": right,
+ },
+ source=left,
+ )
+ assert isinstance(built, MatrixMultiplySource)
+ np.testing.assert_array_equal(built.read_cells(0, 1), [[58, 139]])
+
+
+def test_matrix_multiply_validates_contract_and_memory_limit() -> None:
+ left = DenseMatrixSource(np.ones((3, 4), dtype=np.float64))
+ right = DenseMatrixSource(np.ones((5, 2), dtype=np.float64))
+ with pytest.raises(MatrixSourceError, match="inner dimensions"):
+ MatrixMultiplySource(left, right=right)
+ with pytest.raises(MatrixSourceError, match="requires a right operand"):
+ MatrixMultiplySource(left)
+
+ compatible = DenseMatrixSource(np.ones((4, 2), dtype=np.float64))
+ limited = MatrixMultiplySource(
+ left,
+ right=compatible,
+ limits=SourceLimits(maxBlockBytes=64, tileCells=1),
+ )
+ with pytest.raises(ResourceLimitError, match="maxBlockBytes"):
+ limited.read_cells(0, 2)
+
+
+def test_bp128_reference_decode_crosses_blocks_and_supports_windows() -> None:
+ values = np.asarray([(index * 17) % 257 for index in range(259)], dtype=np.uint32)
+ data, indexes, offsets, _ = _encode_bp128(values, "plain")
+ decoded = decode_bp128(
+ data,
+ indexes,
+ values.size,
+ index_offsets=offsets,
+ )
+ np.testing.assert_array_equal(decoded, values)
+ np.testing.assert_array_equal(
+ decode_bp128(
+ data,
+ indexes,
+ values.size,
+ index_offsets=offsets,
+ start=121,
+ stop=143,
+ ),
+ values[121:143],
+ )
+
+ positive = values + 1
+ m1_data, m1_indexes, m1_offsets, _ = _encode_bp128(positive, "m1")
+ np.testing.assert_array_equal(
+ decode_bp128_m1(
+ m1_data,
+ m1_indexes,
+ positive.size,
+ index_offsets=m1_offsets,
+ ),
+ positive,
+ )
+
+ d1_values = np.asarray(
+ [100 + ((index * 13) % 31) for index in range(259)],
+ dtype=np.uint32,
+ )
+ d1_data, d1_indexes, d1_offsets, starts = _encode_bp128(d1_values, "d1z")
+ np.testing.assert_array_equal(
+ decode_bp128_d1z(
+ d1_data,
+ d1_indexes,
+ starts,
+ d1_values.size,
+ index_offsets=d1_offsets,
+ ),
+ d1_values,
+ )
+
+
+@pytest.mark.parametrize("backend", ["directory", "hdf5"])
+@pytest.mark.parametrize("version", [1, 2])
+@pytest.mark.parametrize("packed", [False, True])
+@pytest.mark.parametrize("datatype", ["uint", "float", "double"])
+@pytest.mark.parametrize("storage_order", ["col", "row"])
+def test_bpcells_all_documented_matrix_leaves(
+ tmp_path: Path,
+ backend: str,
+ version: int,
+ packed: bool,
+ datatype: str,
+ storage_order: str,
+) -> None:
+ base = np.asarray(
+ [[1, 0, 2, 0], [3, 4, 0, 5], [0, 6, 7, 8]],
+ dtype={
+ "uint": np.uint32,
+ "float": np.float32,
+ "double": np.float64,
+ }[datatype],
+ )
+ if datatype != "uint":
+ base = base / 2
+ payload = _bpcells_payload(
+ base,
+ packed=packed,
+ version=version,
+ storage_order=storage_order,
+ )
+ source: MatrixSource
+ if backend == "directory":
+ path = tmp_path / "matrix"
+ _write_bpcells_directory(path, payload, version=version)
+ source = BPCellsDirectoryMatrixSource(path)
+ else:
+ path = tmp_path / "matrix.h5"
+ _write_bpcells_hdf5(path, payload, version=version)
+ source = BPCellsHDF5MatrixSource(path, group="matrix")
+
+ assert source.shape == base.shape
+ assert source.storageOrder == storage_order
+ assert source.row_names == ("f0", "f1", "f2")
+ assert source.column_names == ("c0", "c1", "c2", "c3")
+ np.testing.assert_array_equal(source.read_cells(0, 4).toarray(), base.T)
+ np.testing.assert_array_equal(source.read_cells(1, 3).toarray(), base[:, 1:3].T)
+
+
+@pytest.mark.parametrize("version", [1, 2])
+@pytest.mark.parametrize("packed", [False, True])
+@pytest.mark.parametrize("datatype", ["uint", "float", "double"])
+@pytest.mark.parametrize("storage_order", ["col", "row"])
+def test_bpcells_in_r_memory_leaves_execute_through_factory(
+ version: int,
+ packed: bool,
+ datatype: str,
+ storage_order: str,
+) -> None:
+ expected = np.asarray(
+ [[1, 0, 2, 0], [3, 4, 0, 5], [0, 6, 7, 8]],
+ dtype={
+ "uint": np.uint32,
+ "float": np.float32,
+ "double": np.float64,
+ }[datatype],
+ )
+ if datatype != "uint":
+ expected = expected / 2
+ payload = _bpcells_payload(
+ expected,
+ packed=packed,
+ version=version,
+ storage_order=storage_order,
+ )
+ source = matrix_source_from_slots(
+ _memory_matrix_spec(
+ payload,
+ packed=packed,
+ datatype=datatype,
+ version=version,
+ )
+ )
+ np.testing.assert_array_equal(source.read_cells(0, 4).toarray(), expected.T)
+
+
+def test_bpcells_directory_validates_numeric_header(tmp_path: Path) -> None:
+ directory = tmp_path / "broken"
+ directory.mkdir()
+ (directory / "version").write_text("unpacked-uint-matrix-v1\n")
+ (directory / "shape").write_bytes(b"NOTATYPE" + b"\x00" * 8)
+ with pytest.raises(MatrixSourceError, match="unknown 8-byte header"):
+ BPCellsDirectoryMatrixSource(directory)
+
+
+def test_sidecar_sources_enforce_shape_and_nnz_limits(tmp_path: Path) -> None:
+ values = np.asarray([[1, 2], [3, 4]], dtype=np.uint32)
+ payload = _bpcells_payload(
+ values,
+ packed=False,
+ version=2,
+ storage_order="col",
+ )
+ directory = tmp_path / "limited-matrix"
+ _write_bpcells_directory(directory, payload, version=2)
+ with pytest.raises(ResourceLimitError, match="maxNnz=2"):
+ BPCellsDirectoryMatrixSource(
+ directory,
+ limits=SourceLimits(maxNnz=2),
+ )
+
+ path = tmp_path / "limited-dense.h5"
+ with h5py.File(path, mode="w") as handle:
+ handle.create_dataset("counts", data=values.T)
+ with pytest.raises(ResourceLimitError, match="maxFeatures=1"):
+ HDF5ArrayMatrixSource(
+ path,
+ "counts",
+ limits=SourceLimits(maxFeatures=1),
+ )
+
+ metadata_path = tmp_path / "limited-metadata.h5"
+ with h5py.File(metadata_path, mode="w") as handle:
+ handle.attrs["large"] = "x" * 100
+ with pytest.raises(ResourceLimitError, match="maxMetadataBytes=32"):
+ validate_hdf5_file(
+ metadata_path,
+ limits=SourceLimits(maxMetadataBytes=32),
+ )
+
+
+def test_hdf5array_dense_orientation_and_metadata(tmp_path: Path) -> None:
+ feature_by_cell = np.asarray([[1, 2, 3], [4, 5, 6]], dtype=np.int16)
+ path = tmp_path / "dense.h5"
+ with h5py.File(path, mode="w") as handle:
+ handle.create_dataset("counts", data=feature_by_cell.T)
+ handle.create_dataset("features", data=np.asarray(["f1", "f2"], dtype="S"))
+ handle.create_dataset("cells", data=np.asarray(["c1", "c2", "c3"], dtype="S"))
+ source = HDF5ArrayMatrixSource(
+ path,
+ "counts",
+ row_names_path="/features",
+ column_names_path="/cells",
+ )
+ assert source.shape == (2, 3)
+ np.testing.assert_array_equal(source.read_cells(1, 3), [[2, 5], [3, 6]])
+
+
+def test_reshaped_hdf5array_and_facade_execute_through_factory(
+ tmp_path: Path,
+) -> None:
+ path = tmp_path / "reshaped.h5"
+ physical = np.arange(12, dtype=np.int16).reshape(2, 3, 2)
+ with h5py.File(path, mode="w") as handle:
+ handle.create_dataset("counts", data=physical)
+
+ seed = {
+ "class": "ReshapedHDF5ArraySeed",
+ "slots": {
+ "filepath": path,
+ "name": "counts",
+ "dim": [2, 3, 2],
+ "reshaped_dim": [3, 4],
+ },
+ }
+ source = matrix_source_from_slots(seed)
+ np.testing.assert_array_equal(
+ source.read_cells(1, 3),
+ physical.reshape(-1)[3:9].reshape(2, 3),
+ )
+
+ facade = matrix_source_from_slots(
+ {
+ "class": ["HDF5Matrix", "DelayedMatrix", "DelayedArray"],
+ "slots": {
+ "seed": seed,
+ "dim": [3, 4],
+ },
+ }
+ )
+ np.testing.assert_array_equal(
+ facade.read_cells(0, 4),
+ physical.reshape(-1).reshape(4, 3),
+ )
+
+
+@pytest.mark.parametrize("layout", ["csr", "csc"])
+def test_h5_sparse_matrix_physical_transpose(
+ layout: str,
+ tmp_path: Path,
+) -> None:
+ logical = np.asarray([[1, 0, 2], [0, 3, 4]], dtype=np.float32)
+ physical = logical.T
+ path = tmp_path / f"sparse-{layout}.h5"
+ with h5py.File(path, mode="w") as handle:
+ _write_h5_sparse_group(handle, "counts", physical, layout)
+ source = H5SparseMatrixSource(
+ path,
+ "counts",
+ row_names=["f1", "f2"],
+ column_names=["c1", "c2", "c3"],
+ )
+ np.testing.assert_array_equal(source.read_cells(0, 3).toarray(), logical.T)
+
+
+@pytest.mark.parametrize("layout", ["dense", "csr", "csc"])
+def test_h5ad_dense_and_sparse_sources(layout: str, tmp_path: Path) -> None:
+ logical = np.asarray([[1, 0, 2], [0, 3, 4]], dtype=np.float64)
+ physical = logical.T
+ path = tmp_path / f"matrix-{layout}.h5ad"
+ with h5py.File(path, mode="w") as handle:
+ if layout == "dense":
+ handle.create_dataset("X", data=physical)
+ else:
+ _write_h5_sparse_group(handle, "X", physical, layout)
+ _write_axis_names(handle, ["f1", "f2"], ["c1", "c2", "c3"])
+ source = H5ADMatrixSource(path)
+ assert source.row_names == ("f1", "f2")
+ assert source.column_names == ("c1", "c2", "c3")
+ np.testing.assert_array_equal(
+ source.read_cells(0, 3).toarray()
+ if source.is_sparse
+ else source.read_cells(0, 3),
+ logical.T,
+ )
+
+
+def test_tenx_source_orientation_and_names(tmp_path: Path) -> None:
+ logical = np.asarray([[1, 0, 2], [0, 3, 4]], dtype=np.uint32)
+ matrix = csc_matrix(logical)
+ path = tmp_path / "tenx.h5"
+ with h5py.File(path, mode="w") as handle:
+ group = handle.create_group("matrix")
+ group.create_dataset("shape", data=np.asarray(logical.shape, dtype=np.uint64))
+ group.create_dataset("data", data=matrix.data)
+ group.create_dataset("indices", data=matrix.indices.astype(np.int64))
+ group.create_dataset("indptr", data=matrix.indptr.astype(np.int64))
+ group.create_dataset("barcodes", data=np.asarray(["c1", "c2", "c3"], dtype="S"))
+ features = group.create_group("features")
+ features.create_dataset("name", data=np.asarray(["f1", "f2"], dtype="S"))
+ source = TENxMatrixSource(path)
+ np.testing.assert_array_equal(source.read_cells(0, 3).toarray(), logical.T)
+ assert source.row_names == ("f1", "f2")
+ assert source.column_names == ("c1", "c2", "c3")
+
+
+def test_sidecar_path_resolution_anchors_remaps_and_contains(
+ tmp_path: Path,
+) -> None:
+ rds = tmp_path / "object.rds"
+ rds.write_bytes(b"rds")
+ relative = tmp_path / "sidecar.h5"
+ relative.write_bytes(b"h5")
+ resolver = SidecarPathResolver(rds)
+ assert resolver.resolve("sidecar.h5", expect="file") == relative.resolve()
+
+ outside = tmp_path.parent / "outside-sidecar.h5"
+ outside.write_bytes(b"h5")
+ with pytest.raises(UnsafeSidecarError, match="escapes allowed root"):
+ resolver.resolve("../outside-sidecar.h5")
+ with pytest.raises(UnsafeSidecarError, match="escapes allowed root"):
+ resolver.resolve(r"..\outside-sidecar.h5")
+ with pytest.raises(UnsafeSidecarError, match="no explicit prefix remap"):
+ resolver.resolve(outside)
+
+ remapped_root = tmp_path / "remapped"
+ remapped_root.mkdir()
+ remapped = remapped_root / "counts.h5"
+ remapped.write_bytes(b"h5")
+ remapping = SidecarPathResolver(
+ rds,
+ absolute_prefix_remaps={"/original/data": remapped_root},
+ )
+ assert remapping.resolve("/original/data/counts.h5") == remapped.resolve()
+
+ windows_remapping = SidecarPathResolver(
+ rds,
+ absolute_prefix_remaps={r"C:\original\data": remapped_root},
+ )
+ assert (
+ windows_remapping.resolve(r"C:\original\data\counts.h5") == remapped.resolve()
+ )
+
+
+def test_path_resolution_rejects_symlink_escape(tmp_path: Path) -> None:
+ rds = tmp_path / "object.rds"
+ rds.write_bytes(b"rds")
+ outside = tmp_path.parent / "outside-target.h5"
+ outside.write_bytes(b"h5")
+ link = tmp_path / "linked.h5"
+ try:
+ link.symlink_to(outside)
+ except OSError:
+ pytest.skip("symlinks are not available")
+ with pytest.raises(UnsafeSidecarError, match="escapes allowed root"):
+ SidecarPathResolver(rds).resolve("linked.h5")
+
+
+def test_hdf5_rejects_external_links_and_live_handles(tmp_path: Path) -> None:
+ target = tmp_path / "target.h5"
+ with h5py.File(target, mode="w") as handle:
+ handle.create_dataset("data", data=np.arange(3))
+ linked = tmp_path / "linked.h5"
+ with h5py.File(linked, mode="w") as handle:
+ handle["external"] = h5py.ExternalLink(str(target), "/data")
+
+ with pytest.raises(UnsafeSidecarError, match="external link"):
+ validate_hdf5_file(linked)
+
+ with h5py.File(target, mode="r") as handle:
+ with pytest.raises(UnsafeSidecarError, match="live HDF5 handles"):
+ HDF5ArrayMatrixSource(handle, "data")
+
+
+def test_hdf5_rejects_external_storage_virtual_and_reference_data(
+ tmp_path: Path,
+) -> None:
+ raw = tmp_path / "external.raw"
+ external = tmp_path / "external-storage.h5"
+ with h5py.File(external, mode="w") as handle:
+ handle.create_dataset(
+ "data",
+ shape=(2,),
+ dtype=np.int32,
+ external=[(str(raw), 0, h5py.h5f.UNLIMITED)],
+ )
+ with pytest.raises(UnsafeSidecarError, match="external storage"):
+ validate_hdf5_file(external)
+
+ source_path = tmp_path / "vds-source.h5"
+ with h5py.File(source_path, mode="w") as handle:
+ handle.create_dataset("data", data=np.arange(3))
+ virtual_path = tmp_path / "virtual.h5"
+ layout = h5py.VirtualLayout(shape=(3,), dtype=np.int64)
+ layout[:] = h5py.VirtualSource(str(source_path), "data", shape=(3,))
+ with h5py.File(virtual_path, mode="w") as handle:
+ handle.create_virtual_dataset("data", layout)
+ with pytest.raises(UnsafeSidecarError, match="virtual dataset"):
+ validate_hdf5_file(virtual_path)
+
+ references = tmp_path / "references.h5"
+ with h5py.File(references, mode="w") as handle:
+ target = handle.create_dataset("target", data=np.arange(1))
+ dataset = handle.create_dataset("refs", shape=(1,), dtype=h5py.ref_dtype)
+ dataset[0] = target.ref
+ with pytest.raises(UnsafeSidecarError, match="reference dataset"):
+ validate_hdf5_file(references)
+
+
+def test_hdf5_rejects_unknown_required_filter(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ path = tmp_path / "unknown-filter.h5"
+ filter_id = h5py.h5z.FILTER_DEFLATE
+ with h5py.File(path, mode="w") as handle:
+ space = h5py.h5s.create_simple((1,))
+ properties = h5py.h5p.create(h5py.h5p.DATASET_CREATE)
+ properties.set_chunk((1,))
+ properties.set_filter(filter_id, 0, (1,))
+ h5py.h5d.create(
+ handle.id,
+ b"data",
+ h5py.h5t.NATIVE_INT32,
+ space,
+ dcpl=properties,
+ )
+ available = h5py.h5z.filter_avail
+ monkeypatch.setattr(
+ h5py.h5z,
+ "filter_avail",
+ lambda candidate: False if candidate == filter_id else available(candidate),
+ )
+ with pytest.raises(UnsafeSidecarError, match="requires unavailable filter"):
+ validate_hdf5_file(path)
diff --git a/tests/test_seurat_reader.py b/tests/test_seurat_reader.py
new file mode 100644
index 00000000..01a9e616
--- /dev/null
+++ b/tests/test_seurat_reader.py
@@ -0,0 +1,1234 @@
+import gzip
+import struct
+from pathlib import Path
+
+import h5py
+import numpy as np
+import pytest
+
+import scarf.readers.seurat as seurat_module
+from scarf.readers._rds import LazyAtomicVector, R_INT_NA, RdsClosedError, RType
+from scarf.readers._seurat import SourceLimits
+from scarf.readers.seurat import (
+ SeuratImportError,
+ SeuratReader,
+ SeuratStringVector,
+ inspect_seurat,
+)
+
+
+_FIXTURES = Path(__file__).resolve().parent / "datasets"
+_V4_FIXTURE = _FIXTURES / "seurat_v4_1_3_pbmc_mye.rds"
+_V5_FIXTURE = _FIXTURES / "seurat_assay5_synthetic.rds"
+
+
+class _Wire:
+ def integer(self, value: int) -> bytes:
+ return struct.pack(">i", value)
+
+ def real(self, value: float) -> bytes:
+ return struct.pack(">d", value)
+
+ @staticmethod
+ def flags(
+ r_type: RType,
+ *,
+ attributes: bool = False,
+ tag: bool = False,
+ object_: bool = False,
+ ) -> int:
+ return (
+ int(r_type)
+ | (int(object_) << 8)
+ | (int(attributes) << 9)
+ | (int(tag) << 10)
+ )
+
+ def header(self) -> bytes:
+ return (
+ b"X\n"
+ + self.integer(3)
+ + self.integer(4 * 65_536 + 4 * 256)
+ + self.integer(3 * 65_536 + 5 * 256)
+ + self.integer(5)
+ + b"UTF-8"
+ )
+
+ def nil(self) -> bytes:
+ return self.integer(RType.NIL_VALUE)
+
+ def char(self, value: str | None) -> bytes:
+ result = self.integer(self.flags(RType.CHAR))
+ if value is None:
+ return result + self.integer(-1)
+ encoded = value.encode()
+ return result + self.integer(len(encoded)) + encoded
+
+ def symbol(self, value: str) -> bytes:
+ return self.integer(RType.SYMBOL) + self.char(value)
+
+ def builtin(self, value: str) -> bytes:
+ encoded = value.encode()
+ return (
+ self.integer(self.flags(RType.BUILTIN))
+ + self.integer(len(encoded))
+ + encoded
+ )
+
+ def closure(self) -> bytes:
+ return self.integer(self.flags(RType.CLOSURE)) + self.nil() + self.nil()
+
+ def pair(self, car: bytes, cdr: bytes, *, tag: str) -> bytes:
+ return (
+ self.integer(self.flags(RType.PAIRLIST, tag=True))
+ + self.symbol(tag)
+ + car
+ + cdr
+ )
+
+ def attributes(self, values: list[tuple[str, bytes]]) -> bytes:
+ result = self.nil()
+ for name, value in reversed(values):
+ result = self.pair(value, result, tag=name)
+ return result
+
+ def string_vector(
+ self,
+ values: list[str | None],
+ *,
+ attributes: list[tuple[str, bytes]] | None = None,
+ ) -> bytes:
+ return (
+ self.integer(self.flags(RType.STRING, attributes=attributes is not None))
+ + self.integer(len(values))
+ + b"".join(self.char(value) for value in values)
+ + (self.attributes(attributes) if attributes is not None else b"")
+ )
+
+ def atomic_vector(
+ self,
+ r_type: RType,
+ values: list[int] | list[float],
+ *,
+ attributes: list[tuple[str, bytes]] | None = None,
+ ) -> bytes:
+ encode = self.real if r_type is RType.REAL else self.integer
+ return (
+ self.integer(self.flags(r_type, attributes=attributes is not None))
+ + self.integer(len(values))
+ + b"".join(encode(value) for value in values)
+ + (self.attributes(attributes) if attributes is not None else b"")
+ )
+
+ def integer_vector(
+ self,
+ values: list[int],
+ *,
+ attributes: list[tuple[str, bytes]] | None = None,
+ ) -> bytes:
+ return self.atomic_vector(RType.INTEGER, values, attributes=attributes)
+
+ def logical_vector(
+ self,
+ values: list[int],
+ *,
+ attributes: list[tuple[str, bytes]] | None = None,
+ ) -> bytes:
+ return self.atomic_vector(RType.LOGICAL, values, attributes=attributes)
+
+ def real_vector(
+ self,
+ values: list[float],
+ *,
+ attributes: list[tuple[str, bytes]] | None = None,
+ ) -> bytes:
+ return self.atomic_vector(RType.REAL, values, attributes=attributes)
+
+ def vector(
+ self,
+ values: list[bytes],
+ *,
+ names: list[str] | None = None,
+ attributes: list[tuple[str, bytes]] | None = None,
+ ) -> bytes:
+ attrs = list(attributes or [])
+ if names is not None:
+ attrs.insert(0, ("names", self.string_vector(names)))
+ return (
+ self.integer(self.flags(RType.VECTOR, attributes=bool(attrs)))
+ + self.integer(len(values))
+ + b"".join(values)
+ + (self.attributes(attrs) if attrs else b"")
+ )
+
+ def s4(self, slots: list[tuple[str, bytes]]) -> bytes:
+ return self.integer(
+ self.flags(RType.S4, attributes=True, object_=True)
+ ) + self.attributes(slots)
+
+ def dimnames(
+ self,
+ rows: list[str] | None,
+ columns: list[str] | None,
+ ) -> bytes:
+ return self.vector(
+ [
+ self.nil() if rows is None else self.string_vector(rows),
+ self.nil() if columns is None else self.string_vector(columns),
+ ]
+ )
+
+ def matrix(
+ self,
+ values: list[int] | list[float],
+ shape: tuple[int, int],
+ *,
+ rows: list[str] | None = None,
+ columns: list[str] | None = None,
+ real: bool = False,
+ ) -> bytes:
+ attributes = [
+ ("dim", self.integer_vector(list(shape))),
+ ("dimnames", self.dimnames(rows, columns)),
+ ("class", self.string_vector(["matrix", "array"])),
+ ]
+ if real:
+ return self.real_vector(
+ [float(value) for value in values],
+ attributes=attributes,
+ )
+ return self.integer_vector(
+ [int(value) for value in values],
+ attributes=attributes,
+ )
+
+ def factor(
+ self,
+ values: list[int],
+ levels: list[str],
+ *,
+ names: list[str] | None = None,
+ ) -> bytes:
+ attributes = [
+ ("levels", self.string_vector(levels)),
+ ("class", self.string_vector(["factor"])),
+ ]
+ if names is not None:
+ attributes.append(("names", self.string_vector(names)))
+ return self.integer_vector(values, attributes=attributes)
+
+ def data_frame(
+ self,
+ columns: list[tuple[str, bytes]],
+ row_names: list[str] | int,
+ *,
+ automatic_row_names: bool = True,
+ ) -> bytes:
+ encoded_rows = (
+ self.string_vector(row_names)
+ if isinstance(row_names, list)
+ else self.integer_vector(
+ [
+ R_INT_NA,
+ -row_names if automatic_row_names else row_names,
+ ]
+ )
+ )
+ return self.vector(
+ [value for _, value in columns],
+ attributes=[
+ ("names", self.string_vector([name for name, _ in columns])),
+ ("row.names", encoded_rows),
+ ("class", self.string_vector(["data.frame"])),
+ ],
+ )
+
+ def logmap(
+ self,
+ values: list[int],
+ rows: list[str],
+ layers: list[str],
+ ) -> bytes:
+ return self.logical_vector(
+ values,
+ attributes=[
+ ("dim", self.integer_vector([len(rows), len(layers)])),
+ ("class", self.string_vector(["LogMap"])),
+ ("dimnames", self.dimnames(rows, layers)),
+ ],
+ )
+
+ def document(self, root: bytes) -> bytes:
+ return self.header() + root
+
+
+def _legacy_assay(wire: _Wire) -> bytes:
+ counts = wire.matrix(
+ [1, 0, 0, 2, 3, 0],
+ (2, 3),
+ rows=["g1", "g2"],
+ columns=["c1", "c2", "c3"],
+ )
+ feature_metadata = wire.data_frame(
+ [("symbol", wire.string_vector(["G1", "G2"]))],
+ ["g1", "g2"],
+ )
+ return wire.s4(
+ [
+ ("counts", counts),
+ ("meta.features", feature_metadata),
+ ("class", wire.string_vector(["Assay"])),
+ ]
+ )
+
+
+def _assay5(
+ wire: _Wire,
+ *,
+ invalid_membership: bool,
+ overlap: bool,
+) -> bytes:
+ layer_names = ["counts.1", "counts.2", "data"]
+ cell_values = [
+ 1,
+ 1,
+ 0,
+ 0,
+ 1 if overlap else 0,
+ R_INT_NA if invalid_membership else 1,
+ 1,
+ 1,
+ 1,
+ ]
+ feature_values = [1, 1, 0, 0, 1, 1, 1, 1, 1]
+ layers = wire.vector(
+ [
+ wire.matrix([1, 3, 2, 4], (2, 2)),
+ wire.matrix(
+ [5, 6, 7, 8] if overlap else [5, 6],
+ (2, 2) if overlap else (2, 1),
+ ),
+ wire.matrix([0] * 9, (3, 3), real=True),
+ ],
+ names=layer_names,
+ )
+ metadata = wire.data_frame(
+ [("kind", wire.string_vector(["a", "b", "c"]))],
+ 3,
+ automatic_row_names=False,
+ )
+ return wire.s4(
+ [
+ ("layers", layers),
+ (
+ "cells",
+ wire.logmap(cell_values, ["c1", "c2", "c3"], layer_names),
+ ),
+ (
+ "features",
+ wire.logmap(feature_values, ["p1", "p2", "p3"], layer_names),
+ ),
+ ("meta.data", metadata),
+ ("class", wire.string_vector(["Assay5"])),
+ ]
+ )
+
+
+def _reduction(wire: _Wire) -> bytes:
+ embeddings = wire.matrix(
+ [1.0, 2.0, 3.0, 4.0, 5.0, 6.0],
+ (3, 2),
+ rows=["c1", "c2", "c3"],
+ columns=["PC_1", "PC_2"],
+ real=True,
+ )
+ empty_loadings = wire.matrix([], (0, 0), real=True)
+ return wire.s4(
+ [
+ ("cell.embeddings", embeddings),
+ ("feature.loadings", empty_loadings),
+ ("assay.used", wire.string_vector(["RNA"])),
+ ("global", wire.logical_vector([0])),
+ ("stdev", wire.real_vector([2.0, 1.0])),
+ ("key", wire.string_vector(["PC_"])),
+ ("class", wire.string_vector(["DimReduc"])),
+ ]
+ )
+
+
+def _seurat_payload(
+ *,
+ invalid_membership: bool = False,
+ overlap: bool = False,
+) -> bytes:
+ wire = _Wire()
+ metadata = wire.data_frame(
+ [
+ ("logical", wire.logical_vector([1, 0, R_INT_NA])),
+ ("integer", wire.integer_vector([1, R_INT_NA, 3])),
+ ("real", wire.real_vector([1.5, float("nan"), 3.5])),
+ ("character", wire.string_vector(["a", None, "c"])),
+ ("group", wire.factor([1, 2, R_INT_NA], ["first", "second"])),
+ ],
+ ["c1", "c2", "c3"],
+ )
+ root = wire.s4(
+ [
+ (
+ "assays",
+ wire.vector(
+ [
+ _legacy_assay(wire),
+ _assay5(
+ wire,
+ invalid_membership=invalid_membership,
+ overlap=overlap,
+ ),
+ ],
+ names=["RNA", "ADT"],
+ ),
+ ),
+ ("meta.data", metadata),
+ ("active.assay", wire.string_vector(["RNA"])),
+ (
+ "active.ident",
+ wire.factor(
+ [2, 1, R_INT_NA],
+ ["zero", "one"],
+ names=["c3", "c1", "c2"],
+ ),
+ ),
+ (
+ "reductions",
+ wire.vector([_reduction(wire)], names=["pca"]),
+ ),
+ ("graphs", wire.vector([], names=[])),
+ ("class", wire.string_vector(["Seurat"])),
+ ]
+ )
+ return wire.document(root)
+
+
+def _write_fixture(
+ path: Path,
+ *,
+ compressed: bool = False,
+ invalid_membership: bool = False,
+ overlap: bool = False,
+) -> Path:
+ payload = _seurat_payload(
+ invalid_membership=invalid_membership,
+ overlap=overlap,
+ )
+ path.write_bytes(gzip.compress(payload) if compressed else payload)
+ return path
+
+
+def _write_cached_sidecar_fixture(
+ path: Path,
+ *,
+ loader: str,
+ matrix: np.ndarray | None = None,
+ matrices: dict[str, np.ndarray] | None = None,
+ dataset: str = "counts",
+ package: str = "HDF5Array",
+ source_class: str = "DelayedMatrix",
+) -> Path:
+ values = (
+ np.asarray([[1, 0], [0, 2], [3, 0]], dtype=np.int32)
+ if matrix is None
+ else np.asarray(matrix)
+ )
+ sidecars = {"counts.h5": values} if matrices is None else matrices
+ for filename, sidecar_values in sidecars.items():
+ with h5py.File(path.with_name(filename), mode="w") as handle:
+ handle.create_dataset(dataset, data=np.asarray(sidecar_values))
+
+ wire = _Wire()
+ assay = wire.s4(
+ [
+ ("layers", wire.vector([], names=[])),
+ ("cells", wire.logmap([], ["c1", "c2", "c3"], [])),
+ ("features", wire.logmap([], ["g1", "g2"], [])),
+ (
+ "meta.data",
+ wire.data_frame(
+ [("symbol", wire.string_vector(["G1", "G2"]))],
+ 2,
+ ),
+ ),
+ ("class", wire.string_vector(["Assay5"])),
+ ]
+ )
+ cache = wire.data_frame(
+ [
+ ("layer", wire.string_vector(["counts"])),
+ ("path", wire.string_vector([",".join(sidecars)])),
+ ("class", wire.string_vector([source_class])),
+ ("pkg", wire.string_vector([package])),
+ ("fxn", wire.string_vector([loader])),
+ ("assay", wire.string_vector(["RNA"])),
+ ],
+ 1,
+ )
+ root = wire.s4(
+ [
+ ("assays", wire.vector([assay], names=["RNA"])),
+ (
+ "meta.data",
+ wire.data_frame(
+ [("group", wire.string_vector(["a", "b", "a"]))],
+ ["c1", "c2", "c3"],
+ ),
+ ),
+ ("active.assay", wire.string_vector(["RNA"])),
+ (
+ "active.ident",
+ wire.factor(
+ [1, 2, 1],
+ ["a", "b"],
+ names=["c1", "c2", "c3"],
+ ),
+ ),
+ ("reductions", wire.vector([], names=[])),
+ (
+ "tools",
+ wire.vector([cache], names=["SaveSeuratRds"]),
+ ),
+ ("class", wire.string_vector(["Seurat"])),
+ ]
+ )
+ path.write_bytes(wire.document(root))
+ return path
+
+
+def _write_delayed_hdf5array_fixture(
+ path: Path,
+ *,
+ transformed: bool = False,
+ delayed_primitive: bool = False,
+ executable_function: bool = False,
+) -> Path:
+ with h5py.File(path.with_name("delayed-counts.h5"), mode="w") as handle:
+ handle.create_dataset(
+ "counts",
+ data=np.asarray([[1, 0], [0, 2], [3, 0]], dtype=np.int32),
+ )
+ wire = _Wire()
+ seed = wire.s4(
+ [
+ ("filepath", wire.string_vector(["delayed-counts.h5"])),
+ ("name", wire.string_vector(["counts"])),
+ ("class", wire.string_vector(["HDF5ArraySeed"])),
+ ]
+ )
+ delayed = wire.s4(
+ [
+ ("seed", seed),
+ ("class", wire.string_vector(["DelayedMatrix", "DelayedArray"])),
+ ]
+ )
+ layer = delayed
+ if transformed:
+ layer = wire.s4(
+ [
+ ("matrix", delayed),
+ (
+ "row_params",
+ wire.matrix([2.0, 1.0], (1, 2), real=True),
+ ),
+ ("dim", wire.real_vector([2.0, 3.0])),
+ ("transpose", wire.logical_vector([0])),
+ (
+ "class",
+ wire.string_vector(["TransformMinByRow", "TransformedMatrix"]),
+ ),
+ ]
+ )
+ if delayed_primitive:
+ layer = wire.s4(
+ [
+ ("seed", delayed),
+ ("OP", wire.builtin("+")),
+ ("Largs", wire.vector([])),
+ ("Rargs", wire.vector([wire.real_vector([2.0])])),
+ (
+ "class",
+ wire.string_vector(
+ ["DelayedUnaryIsoOpWithArgs", "DelayedUnaryIsoOp"]
+ ),
+ ),
+ ]
+ )
+ if executable_function:
+ layer = wire.s4(
+ [
+ ("seed", delayed),
+ ("OP", wire.closure()),
+ ("Largs", wire.vector([])),
+ ("Rargs", wire.vector([])),
+ (
+ "class",
+ wire.string_vector(
+ ["DelayedUnaryIsoOpWithArgs", "DelayedUnaryIsoOp"]
+ ),
+ ),
+ ]
+ )
+ assay = wire.s4(
+ [
+ ("layers", wire.vector([layer], names=["counts"])),
+ ("cells", wire.logmap([1, 1, 1], ["c1", "c2", "c3"], ["counts"])),
+ ("features", wire.logmap([1, 1], ["g1", "g2"], ["counts"])),
+ (
+ "meta.data",
+ wire.data_frame(
+ [("symbol", wire.string_vector(["G1", "G2"]))],
+ 2,
+ ),
+ ),
+ ("class", wire.string_vector(["Assay5"])),
+ ]
+ )
+ root = wire.s4(
+ [
+ ("assays", wire.vector([assay], names=["RNA"])),
+ (
+ "meta.data",
+ wire.data_frame(
+ [("group", wire.string_vector(["a", "b", "a"]))],
+ ["c1", "c2", "c3"],
+ ),
+ ),
+ ("active.assay", wire.string_vector(["RNA"])),
+ (
+ "active.ident",
+ wire.factor(
+ [1, 2, 1],
+ ["a", "b"],
+ names=["c1", "c2", "c3"],
+ ),
+ ),
+ ("reductions", wire.vector([], names=[])),
+ ("class", wire.string_vector(["Seurat"])),
+ ]
+ )
+ path.write_bytes(wire.document(root))
+ return path
+
+
+def _write_bpcells_memory_fixture(path: Path) -> Path:
+ wire = _Wire()
+ layer = wire.s4(
+ [
+ ("idxptr", wire.real_vector([0.0, 1.0, 2.0, 4.0])),
+ ("index", wire.integer_vector([0, 1, 0, 1])),
+ ("val", wire.integer_vector([1, 2, 3, 4])),
+ ("version", wire.string_vector(["unpacked-uint-matrix-v2"])),
+ ("dim", wire.real_vector([2.0, 3.0])),
+ ("transpose", wire.logical_vector([0])),
+ ("dimnames", wire.dimnames(["g1", "g2"], ["c1", "c2", "c3"])),
+ (
+ "class",
+ wire.string_vector(["UnpackedMatrixMem_uint32_t", "IterableMatrix"]),
+ ),
+ ]
+ )
+ assay = wire.s4(
+ [
+ ("layers", wire.vector([layer], names=["counts"])),
+ ("cells", wire.logmap([1, 1, 1], ["c1", "c2", "c3"], ["counts"])),
+ ("features", wire.logmap([1, 1], ["g1", "g2"], ["counts"])),
+ (
+ "meta.data",
+ wire.data_frame(
+ [("symbol", wire.string_vector(["G1", "G2"]))],
+ 2,
+ ),
+ ),
+ ("class", wire.string_vector(["Assay5"])),
+ ]
+ )
+ root = wire.s4(
+ [
+ ("assays", wire.vector([assay], names=["RNA"])),
+ (
+ "meta.data",
+ wire.data_frame(
+ [("group", wire.string_vector(["a", "b", "a"]))],
+ ["c1", "c2", "c3"],
+ ),
+ ),
+ ("active.assay", wire.string_vector(["RNA"])),
+ (
+ "active.ident",
+ wire.factor(
+ [1, 2, 1],
+ ["a", "b"],
+ names=["c1", "c2", "c3"],
+ ),
+ ),
+ ("reductions", wire.vector([], names=[])),
+ ("class", wire.string_vector(["Seurat"])),
+ ]
+ )
+ path.write_bytes(wire.document(root))
+ return path
+
+
+def _write_fragment_matrix_fixture(path: Path) -> Path:
+ wire = _Wire()
+ fragments = wire.s4(
+ [
+ ("cell", wire.integer_vector([0, 1, 0, 2, 1, 2, 0])),
+ ("start", wire.integer_vector([0, 5, 10, 12, 20, 1, 4])),
+ ("end", wire.integer_vector([10, 15, 20, 18, 30, 9, 12])),
+ ("end_max", wire.integer_vector([30])),
+ ("chr_ptr", wire.real_vector([0.0, 5.0, 5.0, 7.0])),
+ ("chr_names", wire.string_vector(["chr1", "chr2"])),
+ ("cell_names", wire.string_vector(["c1", "c2", "c3"])),
+ ("version", wire.string_vector(["unpacked-fragments-v2"])),
+ (
+ "class",
+ wire.string_vector(["UnpackedMemFragments", "IterableFragments"]),
+ ),
+ ]
+ )
+ feature_ids = ["p0", "p_span", "p1", "p2", "p_chr2"]
+ layer = wire.s4(
+ [
+ ("fragments", fragments),
+ ("chr_id", wire.integer_vector([0, 0, 0, 0, 1])),
+ ("start", wire.integer_vector([0, 11, 10, 5, 0])),
+ ("end", wire.integer_vector([10, 14, 20, 25, 10])),
+ ("chr_levels", wire.string_vector(["chr1", "chr2"])),
+ ("mode", wire.string_vector(["insertions"])),
+ ("transpose", wire.logical_vector([1])),
+ ("dim", wire.real_vector([5.0, 3.0])),
+ ("dimnames", wire.dimnames(feature_ids, ["c1", "c2", "c3"])),
+ ("class", wire.string_vector(["PeakMatrix", "IterableMatrix"])),
+ ]
+ )
+ assay = wire.s4(
+ [
+ ("layers", wire.vector([layer], names=["counts"])),
+ ("cells", wire.logmap([1, 1, 1], ["c1", "c2", "c3"], ["counts"])),
+ (
+ "features",
+ wire.logmap([1] * len(feature_ids), feature_ids, ["counts"]),
+ ),
+ (
+ "meta.data",
+ wire.data_frame(
+ [("symbol", wire.string_vector(feature_ids))],
+ len(feature_ids),
+ ),
+ ),
+ ("class", wire.string_vector(["Assay5"])),
+ ]
+ )
+ root = wire.s4(
+ [
+ ("assays", wire.vector([assay], names=["ATAC"])),
+ (
+ "meta.data",
+ wire.data_frame(
+ [("group", wire.string_vector(["a", "b", "a"]))],
+ ["c1", "c2", "c3"],
+ ),
+ ),
+ ("active.assay", wire.string_vector(["ATAC"])),
+ (
+ "active.ident",
+ wire.factor(
+ [1, 2, 1],
+ ["a", "b"],
+ names=["c1", "c2", "c3"],
+ ),
+ ),
+ ("reductions", wire.vector([], names=[])),
+ ("class", wire.string_vector(["Seurat"])),
+ ]
+ )
+ path.write_bytes(wire.document(root))
+ return path
+
+
+def test_mixed_assay_dispatch_metadata_and_reduction(tmp_path: Path) -> None:
+ path = _write_fixture(tmp_path / "mixed.rds")
+
+ with SeuratReader(path) as reader:
+ assert reader.activeAssay == "RNA"
+ assert reader.cellIds == ("c1", "c2", "c3")
+ assert len(reader.inspection.sourceDigest) == 64
+ assert reader.inspection.payloadDigest == reader.inspection.sourceDigest
+ assert reader.inspection.compression == "none"
+ rna_inspection = reader.inspection.assay("RNA")
+ assert rna_inspection.sourceClass == "Assay"
+ assert rna_inspection.dtype == " None:
+ path = _write_fixture(tmp_path / "layer-overrides.rds")
+
+ with SeuratReader(
+ path,
+ assays=["ADT"],
+ assay_layers={"ADT": ["counts.1"]},
+ reductions=[],
+ ) as reader:
+ assay = reader.get_assay("ADT")
+ np.testing.assert_array_equal(
+ assay.counts.read_cells(0, 3).toarray(),
+ [[1, 3, 0], [2, 4, 0], [0, 0, 0]],
+ )
+ assert any(
+ notice.code == "ignored_unselected_count_layer"
+ and notice.objectPath == "assays/ADT/layers/counts.2"
+ for notice in assay.notices
+ )
+
+ with SeuratReader(
+ path,
+ assays=["ADT"],
+ assay_layers={"ADT": ["data"]},
+ reductions=[],
+ ) as reader:
+ diagnostic = reader.inspection.assay("ADT").blockingDiagnostic
+ assert diagnostic is not None
+ assert diagnostic.code == "invalid_layer_override"
+
+ invalid_ignored = _write_fixture(
+ tmp_path / "ignored-invalid-layer.rds",
+ invalid_membership=True,
+ )
+ with SeuratReader(
+ invalid_ignored,
+ assays=["ADT"],
+ assay_layers={"ADT": ["counts.1"]},
+ reductions=[],
+ ) as reader:
+ assert reader.inspection.assay("ADT").importable
+
+ with SeuratReader(
+ path,
+ assays=["RNA"],
+ assay_layers={"RNA": ["counts.1"]},
+ reductions=[],
+ ) as reader:
+ diagnostic = reader.inspection.assay("RNA").blockingDiagnostic
+ assert diagnostic is not None
+ assert diagnostic.code == "invalid_layer_override"
+
+
+def test_save_seurat_rds_cache_restores_safe_hdf5array_layer(
+ tmp_path: Path,
+) -> None:
+ source = _write_cached_sidecar_fixture(
+ tmp_path / "cached.rds",
+ loader=(
+ "function(x) HDF5Array::HDF5Array(filepath = x, "
+ "name = 'counts', as.sparse = FALSE)"
+ ),
+ )
+
+ with SeuratReader(source, reductions=[]) as reader:
+ inspection = reader.inspection.assay("RNA")
+ assert inspection.importable
+ assay = reader.get_assay("RNA")
+ np.testing.assert_array_equal(
+ assay.counts.read_cells(0, 3).toarray(),
+ [[1, 0], [0, 2], [3, 0]],
+ )
+ assert any(
+ notice.code == "restored_sidecar_cache_layer"
+ and notice.objectPath == "tools/SaveSeuratRds/0"
+ for notice in assay.notices
+ )
+ assert any(
+ notice.code == "used_save_seurat_rds_cache"
+ for notice in reader.inspection.notices
+ )
+
+
+def test_assay5_reads_delayedmatrix_hdf5array_seed(tmp_path: Path) -> None:
+ source = _write_delayed_hdf5array_fixture(tmp_path / "delayed.rds")
+
+ with SeuratReader(source, reductions=[]) as reader:
+ assay = reader.get_assay("RNA")
+ np.testing.assert_array_equal(
+ assay.counts.read_cells(0, 3).toarray(),
+ [[1, 0], [0, 2], [3, 0]],
+ )
+
+
+def test_assay5_reads_serialized_bpcells_parameter_matrix(tmp_path: Path) -> None:
+ source = _write_delayed_hdf5array_fixture(
+ tmp_path / "transformed.rds",
+ transformed=True,
+ )
+
+ with SeuratReader(source, reductions=[]) as reader:
+ assay = reader.get_assay("RNA")
+ np.testing.assert_array_equal(
+ assay.counts.read_cells(0, 3).toarray(),
+ [[1, 0], [0, 1], [2, 0]],
+ )
+
+
+def test_assay5_reads_allowlisted_delayedarray_primitive(tmp_path: Path) -> None:
+ source = _write_delayed_hdf5array_fixture(
+ tmp_path / "delayed-primitive.rds",
+ delayed_primitive=True,
+ )
+
+ with SeuratReader(source, reductions=[]) as reader:
+ assay = reader.get_assay("RNA")
+ np.testing.assert_array_equal(
+ assay.counts.read_cells(0, 3).toarray(),
+ [[3, 2], [2, 4], [5, 2]],
+ )
+
+
+def test_assay5_rejects_executable_delayedarray_function(tmp_path: Path) -> None:
+ source = _write_delayed_hdf5array_fixture(
+ tmp_path / "delayed-function.rds",
+ executable_function=True,
+ )
+
+ with SeuratReader(source, reductions=[]) as reader:
+ inspection = reader.inspection.assay("RNA")
+ assert not inspection.importable
+ assert inspection.blockingDiagnostic is not None
+ assert inspection.blockingDiagnostic.code == "unsupported_matrix_function"
+ assert inspection.blockingDiagnostic.objectPath.endswith("layers/counts/OP")
+ with pytest.raises(SeuratImportError, match="executable R semantics"):
+ reader.get_assay("RNA")
+
+
+def test_assay5_reads_serialized_bpcells_memory_leaf(tmp_path: Path) -> None:
+ source = _write_bpcells_memory_fixture(tmp_path / "memory.rds")
+
+ with SeuratReader(source, reductions=[]) as reader:
+ assay = reader.get_assay("RNA")
+ np.testing.assert_array_equal(
+ assay.counts.read_cells(0, 3).toarray(),
+ [[1, 0], [0, 2], [3, 4]],
+ )
+
+
+def test_assay5_reads_serialized_fragment_matrix_graph(tmp_path: Path) -> None:
+ source = _write_fragment_matrix_fixture(tmp_path / "fragments.rds")
+
+ with SeuratReader(source, reductions=[]) as reader:
+ assay = reader.get_assay("ATAC")
+ np.testing.assert_array_equal(
+ assay.counts.read_cells(0, 3).toarray(),
+ [
+ [2, 0, 2, 3, 1],
+ [1, 0, 1, 3, 0],
+ [0, 1, 2, 2, 2],
+ ],
+ )
+
+
+def test_save_seurat_rds_cache_restores_generated_bpcells_cbind_recipe(
+ tmp_path: Path,
+) -> None:
+ leaf = "function(x) BPCells::open_matrix_anndata_hdf5(path = x, group = 'X')"
+ loader = (
+ "function(x) { "
+ "paths <- unlist(x = strsplit(x = x, split = ',')); "
+ f'fxns <- list("{leaf}", "{leaf}"); '
+ "mats <- vector(mode = 'list', length = length(x = paths)); "
+ "for (i in seq_along(paths)) { "
+ "fn <- eval(str2lang(fxns[[i]])); "
+ "mats[[i]] <- fn(paths[i]); "
+ "}; return(Reduce(cbind, mats)); }"
+ )
+ source = _write_cached_sidecar_fixture(
+ tmp_path / "cached-bind.rds",
+ loader=loader,
+ matrices={
+ "part-one.h5": np.asarray([[1, 0]], dtype=np.int32),
+ "part-two.h5": np.asarray([[0, 2], [3, 0]], dtype=np.int32),
+ },
+ dataset="X",
+ package="BPCells",
+ source_class="IterableMatrix",
+ )
+
+ with SeuratReader(source, reductions=[]) as reader:
+ assay = reader.get_assay("RNA")
+ np.testing.assert_array_equal(
+ assay.counts.read_cells(0, 3).toarray(),
+ [[1, 0], [0, 2], [3, 0]],
+ )
+
+
+def test_save_seurat_rds_cache_rejects_code_and_irrecoverable_membership(
+ tmp_path: Path,
+) -> None:
+ unsafe = _write_cached_sidecar_fixture(
+ tmp_path / "unsafe.rds",
+ loader="function(x) system(x)",
+ )
+ with SeuratReader(unsafe, reductions=[]) as reader:
+ diagnostic = reader.inspection.assay("RNA").blockingDiagnostic
+ assert diagnostic is not None
+ assert diagnostic.code == "unsupported_sidecar_cache_recipe"
+ assert diagnostic.objectPath == "tools/SaveSeuratRds/0/fxn"
+
+ irrecoverable = _write_cached_sidecar_fixture(
+ tmp_path / "irrecoverable.rds",
+ loader=(
+ "function(x) HDF5Array::HDF5Array(filepath = x, "
+ "name = 'counts', as.sparse = FALSE)"
+ ),
+ matrix=np.asarray([[1, 0], [0, 2]], dtype=np.int32),
+ )
+ with SeuratReader(irrecoverable, reductions=[]) as reader:
+ diagnostic = reader.inspection.assay("RNA").blockingDiagnostic
+ assert diagnostic is not None
+ assert diagnostic.code == "irrecoverable_sidecar_cache"
+ assert diagnostic.objectPath == "assays/RNA/layers/counts/Dimnames/1"
+
+
+def test_single_open_lifecycle_and_bounded_atomic_windows(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ path = _write_fixture(tmp_path / "single-open.rds")
+ original_open = seurat_module.open_rds
+ open_count = 0
+
+ def tracked_open(*args, **kwargs):
+ nonlocal open_count
+ open_count += 1
+ return original_open(*args, **kwargs)
+
+ monkeypatch.setattr(seurat_module, "open_rds", tracked_open)
+ reader = SeuratReader(path, assays=["RNA"], reductions=["pca"])
+ assert open_count == 1
+
+ assay = reader.get_assay("RNA")
+ dense = assay.counts._source
+ target_counts = dense._values
+ reduction = reader.get_reduction("pca")
+ target_embeddings = reduction.cellEmbeddings._values
+ original_read = LazyAtomicVector.read_block
+ reads: dict[int, list[tuple[int, int]]] = {
+ id(target_counts): [],
+ id(target_embeddings): [],
+ }
+
+ def tracked_read(self, start, stop):
+ if id(self) in reads:
+ reads[id(self)].append((start, stop))
+ return original_read(self, start, stop)
+
+ monkeypatch.setattr(LazyAtomicVector, "read_block", tracked_read)
+ np.testing.assert_array_equal(assay.counts.read_cells(1, 2), [[0, 2]])
+ assert reads[id(target_counts)] == [(2, 4)]
+ reduction.cellEmbeddings.read_rows(1, 3)
+ assert reads[id(target_embeddings)] == [(1, 3), (4, 6)]
+
+ reader.close()
+ reader.close()
+ with pytest.raises(RdsClosedError, match="RDS document is closed"):
+ _ = reader.inspection
+ with pytest.raises(RdsClosedError, match="RDS document is closed"):
+ assay.counts.read_cells(0, 1)
+ with pytest.raises(RdsClosedError, match="RDS document is closed"):
+ reduction.cellEmbeddings.read_rows(0, 1)
+
+
+def test_identifier_axes_remain_block_readable_and_spill_indexes_cleanup(
+ tmp_path: Path,
+ monkeypatch: pytest.MonkeyPatch,
+) -> None:
+ path = _write_fixture(tmp_path / "streamed-identifiers.rds")
+ original = seurat_module._read_text_vector
+
+ def reject_materialized_ids(node, *, object_path, **kwargs):
+ if object_path.endswith("row.names") or "/Dimnames/" in object_path:
+ raise AssertionError(f"identifier axis was materialized at {object_path}")
+ return original(node, object_path=object_path, **kwargs)
+
+ monkeypatch.setattr(seurat_module, "_read_text_vector", reject_materialized_ids)
+ with SeuratReader(path, temp_dir=tmp_path) as reader:
+ assert isinstance(reader.cellIds, SeuratStringVector)
+ assert reader.cellIds.read_block(1, 3) == ("c2", "c3")
+ assert isinstance(reader.get_assay("RNA").featureIds, SeuratStringVector)
+ assert not tuple(tmp_path.glob("scarf-seurat-ids-*.sqlite3"))
+
+
+def test_identifier_index_budget_is_enforced(tmp_path: Path) -> None:
+ path = _write_fixture(tmp_path / "identifier-budget.rds")
+ with pytest.raises(SeuratImportError) as error:
+ SeuratReader(
+ path,
+ matrix_limits=SourceLimits(maxMetadataBytes=4_096),
+ )
+ assert error.value.code == "metadata_index_limit"
+ assert error.value.objectPath == "meta.data/row.names"
+
+
+def test_detached_inspection_cleans_compressed_scratch(tmp_path: Path) -> None:
+ source = _write_fixture(tmp_path / "compressed.rds", compressed=True)
+ scratch = tmp_path / "scratch"
+ scratch.mkdir()
+
+ result = inspect_seurat(
+ source,
+ temp_dir=scratch,
+ assays=["RNA"],
+ reductions=["pca"],
+ )
+
+ assert result.activeAssay == "RNA"
+ assert result.assay("RNA").dimensions == (2, 3)
+ assert result.reduction("pca").dimensions == (3, 2)
+ assert tuple(scratch.iterdir()) == ()
+
+
+def test_logmap_failure_is_item_local_and_strict_extraction_raises(
+ tmp_path: Path,
+) -> None:
+ path = _write_fixture(
+ tmp_path / "bad-logmap.rds",
+ invalid_membership=True,
+ )
+
+ with SeuratReader(path) as reader:
+ rna = reader.inspection.assay("RNA")
+ adt = reader.inspection.assay("ADT")
+ pca = reader.inspection.reduction("pca")
+ assert rna.importable
+ assert pca.importable
+ assert not adt.importable
+ assert adt.blockingDiagnostic is not None
+ assert adt.blockingDiagnostic.code == "invalid_logmap_membership"
+ assert adt.blockingDiagnostic.objectPath == "assays/ADT/cells/counts.2"
+
+ with pytest.raises(SeuratImportError) as error:
+ reader.get_assay("ADT")
+ assert error.value.code == "invalid_logmap_membership"
+ assert error.value.objectPath == "assays/ADT/cells/counts.2"
+ assert error.value.context == {"row": 2, "value": R_INT_NA}
+
+
+def test_logmap_coordinate_overlap_has_exact_diagnostic(tmp_path: Path) -> None:
+ path = _write_fixture(tmp_path / "overlap.rds", overlap=True)
+
+ with SeuratReader(path, assays=["ADT"], reductions=[]) as reader:
+ inspection = reader.inspection.assay("ADT")
+ assert not inspection.importable
+ assert inspection.blockingDiagnostic is not None
+ assert inspection.blockingDiagnostic.code == "layer_stitch_conflict"
+ assert inspection.blockingDiagnostic.objectPath == "assays/ADT/layers"
+ assert "counts.1" in inspection.blockingDiagnostic.message
+ assert "counts.2" in inspection.blockingDiagnostic.message
+
+
+def test_real_seurat_v4_fixture() -> None:
+ with SeuratReader(_V4_FIXTURE) as reader:
+ assert reader.activeAssay == "RNA"
+ assert reader.cellIds[:3] == (
+ "sample1_GAGTCATGTACCCGCA-1",
+ "sample1_TGGAGGAGTGTATACC-1",
+ "sample1_CCCGGAAGTTGGCTAT-1",
+ )
+ assay = reader.get_assay("RNA")
+ assert assay.sourceClass == "Assay"
+ assert assay.dimensions == (17_195, 1_000)
+ assert assay.featureIds[:3] == ("AL627309.1", "AL627309.5", "LINC01409")
+ assert assay.counts.read_cells(7, 9).shape == (2, 17_195)
+ assert reader.cellMetadata.columnNames[:3] == (
+ "orig.ident",
+ "nCount_RNA",
+ "nFeature_RNA",
+ )
+ assert reader.activeIdentity.levels == ("DC", "Mono CD14", "Mono FCGR3A")
+ assert reader.activeIdentity.read_decoded_block(0, 3) == (
+ "Mono CD14",
+ "Mono CD14",
+ "Mono CD14",
+ )
+ assert reader.get_reduction("pca").dimensions == (1_000, 50)
+ assert reader.get_reduction("pca").featureLoadings is not None
+ assert reader.get_reduction("umap").dimensions == (1_000, 2)
+ assert reader.get_reduction("umap").role == "displayEmbedding"
+
+
+def test_real_seurat_v5_fixture() -> None:
+ with SeuratReader(_V5_FIXTURE) as reader:
+ assert reader.activeAssay == "RNA"
+ assert reader.cellIds[:3] == ("Cell1", "Cell2", "Cell3")
+ assay = reader.get_assay("RNA")
+ assert assay.sourceClass == "Assay5"
+ assert assay.dimensions == (500, 300)
+ assert assay.featureIds[:3] == ("Gene1", "Gene2", "Gene3")
+ assert assay.counts.read_cells(11, 13).shape == (2, 500)
+ assert np.all(assay.cellMembership)
+ assert reader.activeIdentity.read_decoded_block(0, 3) == ("1", "0", "2")
+ pca = reader.get_reduction("pca")
+ assert pca.dimensions == (300, 20)
+ assert pca.featureLoadings is not None
+ assert pca.featureLoadings.shape == (200, 20)
+ assert reader.get_reduction("umap").dimensions == (300, 2)
+ assert reader.get_reduction("umap").role == "displayEmbedding"
diff --git a/tests/test_storage_streaming.py b/tests/test_storage_streaming.py
new file mode 100644
index 00000000..8c1ea7f0
--- /dev/null
+++ b/tests/test_storage_streaming.py
@@ -0,0 +1,148 @@
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.metadata import MetaData
+from scarf.storage.arrays import MetadataBlock, create_streamed_metadata_column
+from scarf.storage.artifacts import (
+ artifact_group,
+ fingerprint_string_blocks,
+ fingerprint_strings,
+)
+from scarf.storage.schema import (
+ create_empty_cell_data,
+ create_empty_zarr_count_assay,
+)
+from scarf.storage.selections import (
+ fingerprint_selected_stored_strings,
+ resolve_stored_selection_artifact,
+)
+
+
+def test_streamed_metadata_column_writes_contiguous_blocks_and_mask():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ group = root.create_group("cellData")
+ values = create_streamed_metadata_column(
+ group,
+ "cluster",
+ shape=4,
+ dtype="U2",
+ blocks=(
+ MetadataBlock(0, np.array(["a", ""], dtype="U2"), np.array([0, 1])),
+ MetadataBlock(2, np.array(["bb", "a"], dtype="U2"), np.array([0, 0])),
+ ),
+ hasMissing=True,
+ )
+
+ assert values[:].tolist() == ["a", "", "bb", "a"]
+ missing_name = values.attrs["missing_mask"]
+ assert group[missing_name][:].tolist() == [False, True, False, False]
+
+
+def test_streamed_metadata_column_rejects_gaps():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ with pytest.raises(ValueError, match="expected 0, received 1"):
+ create_streamed_metadata_column(
+ root,
+ "bad",
+ shape=1,
+ dtype=np.int32,
+ blocks=(MetadataBlock(1, np.array([], dtype=np.int32)),),
+ )
+
+
+def test_internal_missing_columns_are_not_public_metadata():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ group = create_empty_cell_data(root, None, 2, "U2", "U2")
+ group["ids"][:] = np.array(["c1", "c2"])
+ group["names"][:] = np.array(["c1", "c2"])
+ create_streamed_metadata_column(
+ group,
+ "label",
+ shape=2,
+ dtype="U1",
+ blocks=(MetadataBlock(0, np.array(["a", ""], dtype="U1"), np.array([0, 1])),),
+ hasMissing=True,
+ )
+
+ metadata = MetaData(group)
+ assert "label" in metadata.columns
+ assert "__scarf_missing__label" not in metadata.columns
+
+
+def test_empty_assay_schema_accepts_blockwise_feature_metadata():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ counts, feature_data = create_empty_zarr_count_assay(
+ root,
+ "RNA",
+ None,
+ 3,
+ 2,
+ "U2",
+ "U6",
+ dtype=np.uint32,
+ )
+ feature_data["ids"][:] = np.array(["g1", "g2"])
+ feature_data["names"][:] = np.array(["Gene 1", "Gene 2"])
+
+ assert counts.shape == (3, 2)
+ assert feature_data["ids"][:].tolist() == ["g1", "g2"]
+ assert feature_data["I"][:].tolist() == [True, True]
+
+
+def test_string_block_fingerprint_matches_materialized_values():
+ values = np.array(["c1", "cell-two", "c3"], dtype="U8")
+ digest = fingerprint_string_blocks(
+ ((0, values[:2]), (2, values[2:])),
+ length=3,
+ max_length=8,
+ )
+
+ assert digest == fingerprint_strings(values)
+
+
+def test_stored_selection_artifact_copies_values_blockwise():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ cells = create_empty_cell_data(root, None, 4, "U2", "U2")
+ cells["ids"][:] = np.array(["c1", "c2", "c3", "c4"])
+ cells["names"][:] = cells["ids"][:]
+ cells["I"][:] = np.array([True, False, True, True])
+
+ ref = resolve_stored_selection_artifact(
+ root,
+ table_path="cellData",
+ id_column="ids",
+ source_column="I",
+ scope="datastore",
+ kind="cell_selection",
+ operation="import_cell_selection",
+ parameters={"source": "seurat"},
+ inputs={},
+ )
+
+ assert artifact_group(root, ref)["values"][:].tolist() == [
+ True,
+ False,
+ True,
+ True,
+ ]
+
+
+def test_selected_stored_string_fingerprint_matches_materialized_values():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ ids = root.create_array(
+ "ids",
+ data=np.array(["cell-1", "cell-2", "cell-3", "cell-4"], dtype="U6"),
+ chunks=(2,),
+ )
+ selection = root.create_array(
+ "selection",
+ data=np.array([True, False, True, True]),
+ chunks=(2,),
+ )
+
+ digest, count = fingerprint_selected_stored_strings(ids, selection)
+
+ assert count == 3
+ assert digest == fingerprint_strings(ids[:][selection[:]])
diff --git a/tests/test_straight_cut.py b/tests/test_straight_cut.py
new file mode 100644
index 00000000..59bac6a0
--- /dev/null
+++ b/tests/test_straight_cut.py
@@ -0,0 +1,132 @@
+import numpy as np
+import pytest
+from scipy.cluster.hierarchy import linkage
+from scipy.spatial.distance import pdist
+from sknetwork.hierarchy import cut_straight as reference_straight_cut
+
+from scarf.clustering.paris import straight_cut
+
+
+@pytest.mark.parametrize("n_leaves", [3, 10, 40])
+def test_straight_cut_matches_scikit_network(n_leaves: int) -> None:
+ rng = np.random.default_rng(100 + n_leaves)
+ points = rng.normal(size=(n_leaves, 4))
+ dendrogram = linkage(pdist(points), method="average")
+ dendrogram[:, 2] = np.round(dendrogram[:, 2], 1)
+
+ for n_clusters in range(2, n_leaves + 1):
+ expected = reference_straight_cut(
+ dendrogram,
+ n_clusters=n_clusters,
+ )
+ assert np.array_equal(
+ straight_cut(dendrogram, n_clusters),
+ expected + 1,
+ )
+
+
+def test_straight_cut_matches_non_monotonic_dendrogram() -> None:
+ dendrogram = np.asarray(
+ [
+ [0, 1, 0.5, 2],
+ [2, 3, 0.5, 2],
+ [4, 5, 1.5, 2],
+ [6, 7, 2.0, 4],
+ [8, 9, 1.0, 6],
+ ],
+ dtype=np.float64,
+ )
+
+ for n_clusters in range(2, 7):
+ expected = reference_straight_cut(
+ dendrogram,
+ n_clusters=n_clusters,
+ )
+ assert np.array_equal(
+ straight_cut(dendrogram, n_clusters),
+ expected + 1,
+ )
+
+
+def test_equal_heights_can_return_more_clusters_than_requested() -> None:
+ dendrogram = np.asarray(
+ [
+ [0, 1, 1, 2],
+ [2, 3, 1, 2],
+ [4, 5, 2, 4],
+ ],
+ dtype=np.float64,
+ )
+
+ labels = straight_cut(dendrogram, n_clusters=3)
+
+ assert labels.tolist() == [1, 2, 3, 4]
+
+
+def test_one_cluster_and_single_leaf_are_supported() -> None:
+ dendrogram = np.asarray(
+ [
+ [0, 1, 1, 2],
+ [2, 3, 2, 3],
+ ],
+ dtype=np.float64,
+ )
+ assert straight_cut(dendrogram, n_clusters=1).tolist() == [1, 1, 1]
+ assert straight_cut(np.empty((0, 4)), n_clusters=1).tolist() == [1]
+
+
+@pytest.mark.parametrize("n_clusters", [0, 4])
+def test_straight_cut_rejects_out_of_range_cluster_counts(
+ n_clusters: int,
+) -> None:
+ dendrogram = np.asarray(
+ [
+ [0, 1, 1, 2],
+ [2, 3, 2, 3],
+ ],
+ dtype=np.float64,
+ )
+ with pytest.raises(ValueError, match="between 1"):
+ straight_cut(dendrogram, n_clusters)
+
+
+def test_straight_cut_validates_types_and_shape() -> None:
+ dendrogram = np.asarray(
+ [
+ [0, 1, 1, 2],
+ [2, 3, 2, 3],
+ ],
+ dtype=np.float64,
+ )
+ with pytest.raises(TypeError, match="integer"):
+ straight_cut(dendrogram, True)
+ with pytest.raises(ValueError, match="shape"):
+ straight_cut(np.ones((2, 3)), 2)
+
+
+@pytest.mark.parametrize(
+ ("children", "message"),
+ [
+ ([[np.nan, 1], [2, 3]], "finite integers"),
+ ([[np.inf, 1], [2, 3]], "finite integers"),
+ ([[0.5, 1], [2, 3]], "finite integers"),
+ ([[-1, 1], [2, 3]], "non-negative"),
+ ([[0, 0], [2, 3]], "two distinct children"),
+ ([[0, 3], [1, 2]], "only earlier nodes"),
+ ([[0, 1], [0, 3]], "consumed more than once"),
+ ],
+)
+def test_straight_cut_rejects_malformed_children_before_one_cluster_return(
+ children: list[list[float]],
+ message: str,
+) -> None:
+ dendrogram = np.asarray(
+ [
+ [*children[0], 1, 2],
+ [*children[1], 2, 3],
+ ],
+ dtype=np.float64,
+ )
+
+ with pytest.raises(ValueError, match=message):
+ straight_cut(dendrogram, n_clusters=1)
diff --git a/tests/test_stream_geometry.py b/tests/test_stream_geometry.py
new file mode 100644
index 00000000..31ac080b
--- /dev/null
+++ b/tests/test_stream_geometry.py
@@ -0,0 +1,489 @@
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.assay.persistence import _read_block
+from scarf.matrix.chunked import ChunkedArray
+from scarf.storage.budget import ResourceBudget, admit_stream
+from scarf.storage.feature_stream import plan_feature_stream
+from scarf.storage.geometry import ArrayGeometry, array_geometry
+from scarf.storage.partition import (
+ affordable_width,
+ checked_indices,
+ contiguous_ranges,
+ partition_indices,
+ row_band,
+)
+from scarf.utils.prefetch import iter_column_blocks
+
+from .store_probes import RecordingStore
+
+
+def _array(
+ *,
+ shape: tuple[int, int],
+ chunks: tuple[int, int],
+ shards: tuple[int, int] | None = None,
+ dtype: type[np.generic] = np.uint32,
+ compressors: None | str = "auto",
+) -> zarr.Array:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ return root.create_array(
+ "counts",
+ shape=shape,
+ chunks=chunks,
+ shards=shards,
+ dtype=dtype,
+ fill_value=0,
+ compressors=compressors,
+ )
+
+
+def _geometry(
+ *,
+ shape: tuple[int, int],
+ chunks: tuple[int, int],
+ shards: tuple[int, int] | None = None,
+ itemsize: int = 4,
+) -> ArrayGeometry:
+ return ArrayGeometry(
+ shape=shape,
+ chunks=chunks,
+ shards=shards,
+ itemsize=itemsize,
+ )
+
+
+def test_nominal_chunk_bytes_counts_the_whole_chunk_at_an_array_edge() -> None:
+ array = _array(shape=(12, 7), chunks=(5, 3))
+ geometry = array_geometry(array)
+
+ assert geometry is not None
+ # The trailing chunk covers two rows and one column of the array, but Zarr
+ # stores and decodes the full 5 x 3 chunk.
+ assert geometry.nominalChunkBytes() == 5 * 3 * 4
+
+
+def test_an_edge_chunk_decodes_to_the_nominal_size() -> None:
+ array = _array(shape=(12, 7), chunks=(5, 3), compressors=None)
+ array[10:12, 6:7] = 1
+ geometry = array_geometry(array)
+
+ uncompressed = [
+ len(array.store._store_dict[key].to_bytes())
+ for key in array.store._store_dict
+ if "/c/" in key
+ ]
+
+ assert geometry is not None
+ assert uncompressed == [geometry.nominalChunkBytes()]
+
+
+def test_array_geometry_is_absent_for_in_memory_arrays() -> None:
+ assert array_geometry(np.zeros((4, 4), dtype=np.uint32)) is None
+
+
+def test_array_geometry_records_shards_and_falls_back_to_chunks() -> None:
+ sharded = array_geometry(_array(shape=(20, 20), chunks=(5, 5), shards=(10, 10)))
+ plain = array_geometry(_array(shape=(20, 20), chunks=(5, 5)))
+
+ assert sharded is not None and plain is not None
+ assert sharded.shards == (10, 10)
+ assert sharded.axisShard(0) == 10
+ assert sharded.axisChunk(0) == 5
+ assert plain.shards is None
+ assert plain.axisShard(0) == plain.axisChunk(0) == 5
+
+
+def test_bin_of_maps_indices_to_their_chunk() -> None:
+ geometry = _geometry(shape=(12, 20), chunks=(5, 5))
+
+ np.testing.assert_array_equal(
+ geometry.binOf(1, np.array([0, 4, 5, 17])),
+ np.array([0, 0, 1, 3]),
+ )
+
+
+def test_row_band_follows_the_requested_stored_unit() -> None:
+ geometry = _geometry(shape=(20, 20), chunks=(5, 5), shards=(10, 10))
+
+ assert row_band(geometry, fallback=999) == 10
+ assert row_band(geometry, unit="chunk", fallback=999) == 5
+
+
+def test_row_band_uses_the_fallback_without_geometry() -> None:
+ assert row_band(None, fallback=10_000) == 10_000
+ assert row_band(None, fallback=0) == 1
+
+
+def test_contiguous_ranges_cover_the_axis_without_overlap() -> None:
+ assert contiguous_ranges(12, 5) == [(0, 5), (5, 10), (10, 12)]
+ assert contiguous_ranges(0, 5) == []
+
+
+def test_affordable_width_finds_the_largest_accepted_width() -> None:
+ assert affordable_width(lambda width: width <= 3, 10) == 3
+ assert affordable_width(lambda width: width <= 20, 10) == 10
+ assert affordable_width(lambda _width: False, 10) == 0
+
+
+def test_partition_indices_emits_one_block_per_chunk_by_default() -> None:
+ geometry = _geometry(shape=(12, 20), chunks=(5, 5))
+
+ blocks = partition_indices(geometry, 1, np.array([0, 1, 5, 6, 17]))
+
+ assert [block.bins for block in blocks] == [(0,), (1,), (3,)]
+ np.testing.assert_array_equal(blocks[0].indices, np.array([0, 1]))
+ np.testing.assert_array_equal(blocks[2].destinations, np.array([4]))
+
+
+def test_partition_indices_packs_only_adjacent_chunks() -> None:
+ geometry = _geometry(shape=(12, 20), chunks=(5, 5))
+
+ blocks = partition_indices(
+ geometry,
+ 1,
+ np.array([0, 1, 5, 6, 17]),
+ fits=lambda width: width <= 4,
+ )
+
+ assert [block.bins for block in blocks] == [(0, 1), (3,)]
+ np.testing.assert_array_equal(blocks[0].destinations, np.array([0, 1, 2, 3]))
+
+
+def test_partition_indices_preserves_destinations_for_unsorted_indices() -> None:
+ geometry = _geometry(shape=(12, 15), chunks=(5, 5))
+ requested = np.array([12, 1, 6])
+
+ blocks = partition_indices(
+ geometry,
+ 1,
+ requested,
+ fits=lambda width: width <= 3,
+ )
+
+ destinations = np.concatenate([block.destinations for block in blocks])
+ indices = np.concatenate([block.indices for block in blocks])
+ restored = np.empty(requested.size, dtype=np.int64)
+ restored[destinations] = indices
+
+ np.testing.assert_array_equal(restored, requested)
+
+
+def test_partition_indices_splits_a_chunk_that_is_too_wide_alone() -> None:
+ geometry = _geometry(shape=(12, 5), chunks=(5, 5))
+
+ blocks = partition_indices(
+ geometry,
+ 1,
+ np.arange(5),
+ fits=lambda width: width <= 2,
+ )
+
+ assert [block.indices.size for block in blocks] == [2, 2, 1]
+ assert [block.bins for block in blocks] == [(0,), (0,), (0,)]
+
+
+def test_partition_indices_cuts_fixed_width_blocks_in_request_order() -> None:
+ geometry = _geometry(shape=(12, 10), chunks=(5, 5))
+
+ blocks = partition_indices(geometry, 1, np.arange(10), maxWidth=4)
+
+ assert [block.indices.size for block in blocks] == [4, 4, 2]
+ np.testing.assert_array_equal(blocks[1].indices, np.arange(4, 8))
+ np.testing.assert_array_equal(blocks[1].destinations, np.arange(4, 8))
+ assert blocks[1].bins == (0, 1)
+
+
+def test_partition_indices_rejects_an_index_that_never_fits() -> None:
+ geometry = _geometry(shape=(12, 10), chunks=(5, 5))
+
+ with pytest.raises(MemoryError, match="does not fit the operation budget"):
+ partition_indices(geometry, 1, np.arange(10), fits=lambda _width: False)
+
+
+def test_partition_indices_rejects_both_width_and_predicate() -> None:
+ geometry = _geometry(shape=(12, 10), chunks=(5, 5))
+
+ with pytest.raises(ValueError, match="either maxWidth or fits"):
+ partition_indices(
+ geometry,
+ 1,
+ np.arange(4),
+ maxWidth=2,
+ fits=lambda _width: True,
+ )
+
+
+def test_partition_indices_returns_nothing_for_an_empty_selection() -> None:
+ geometry = _geometry(shape=(12, 10), chunks=(5, 5))
+
+ assert partition_indices(geometry, 1, np.array([], dtype=np.int64)) == []
+
+
+def test_checked_indices_rejects_malformed_selections() -> None:
+ with pytest.raises(ValueError, match="one-dimensional"):
+ checked_indices(np.zeros((2, 2), dtype=np.int64), limit=4, name="rows")
+ with pytest.raises(TypeError, match="must contain integers"):
+ checked_indices(np.array([0.5]), limit=4, name="rows")
+ with pytest.raises(IndexError, match="out-of-range"):
+ checked_indices(np.array([4]), limit=4, name="rows")
+ with pytest.raises(ValueError, match="duplicate"):
+ checked_indices(np.array([1, 1]), limit=4, name="rows")
+
+
+def test_admit_stream_charges_every_concurrent_chunk_decode() -> None:
+ resources = ResourceBudget(1_000, 8)
+
+ admission = admit_stream(
+ resources,
+ nBlocks=8,
+ blockBytes=400,
+ decodeBytes=100,
+ )
+
+ assert (admission.outerWorkers, admission.ioConcurrency) == (2, 1)
+ peak = admission.outerWorkers * (400 + admission.ioConcurrency * 100)
+ assert peak <= resources.memoryBytes
+
+
+def test_admit_stream_without_decode_bytes_matches_a_flat_task_cost() -> None:
+ resources = ResourceBudget(1_000, 8)
+
+ admission = admit_stream(resources, nBlocks=8, blockBytes=250)
+
+ assert admission.outerWorkers == 4
+ assert admission.outerWorkers * 250 <= resources.memoryBytes
+
+
+def test_admit_stream_honours_resident_bytes_and_a_requested_depth() -> None:
+ resources = ResourceBudget(1_000, 8)
+
+ admission = admit_stream(
+ resources,
+ nBlocks=8,
+ blockBytes=100,
+ residentBytes=600,
+ requested=2,
+ )
+
+ assert admission.outerWorkers == 2
+ assert 600 + admission.outerWorkers * 100 <= resources.memoryBytes
+
+
+def test_admit_stream_reports_when_one_block_cannot_fit() -> None:
+ with pytest.raises(MemoryError, match="One task needs"):
+ admit_stream(ResourceBudget(100, 4), nBlocks=4, blockBytes=1_000)
+
+
+def test_row_band_reproduces_every_expression_it_replaced() -> None:
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ sharded = root.create_array(
+ "sharded",
+ shape=(200, 8),
+ chunks=(10, 8),
+ shards=(50, 8),
+ dtype=np.uint32,
+ )
+ chunked = root.create_array("chunked", shape=(200, 8), chunks=(10, 8), dtype="u4")
+ scalar = root.create_array("scalar", shape=(), dtype="u4")
+ in_memory = np.zeros((200, 8), dtype=np.uint32)
+
+ for array in (sharded, chunked, scalar, in_memory):
+ geometry = array_geometry(array)
+ chunks = getattr(array, "chunks", None)
+ rows = int(array.shape[0]) if array.ndim else 0
+
+ # scarf.storage.layout.array_shard_rows
+ shards = getattr(getattr(array, "metadata", None), "shards", None)
+ if shards is not None and len(shards) > 0:
+ expected_shard = int(shards[0])
+ elif chunks is not None and len(chunks) > 0:
+ expected_shard = int(chunks[0])
+ else:
+ expected_shard = max(rows, 1)
+ assert row_band(geometry, fallback=max(rows, 1)) == expected_shard
+
+ # scarf.metadata.rows.default_block_rows
+ if chunks and len(chunks) > 0 and int(chunks[0]) > 0:
+ expected_metadata = int(chunks[0])
+ else:
+ expected_metadata = max(1, min(rows, 100_000))
+ assert (
+ row_band(geometry, unit="chunk", fallback=min(rows, 100_000))
+ == expected_metadata
+ )
+
+ # mapping._projection_block_size, confidence.py and hashing.py
+ if chunks is not None and len(chunks) > 0:
+ expected_projection = int(chunks[0])
+ else:
+ expected_projection = min(max(rows, 1), 10_000)
+ assert (
+ row_band(geometry, unit="chunk", fallback=min(max(rows, 1), 10_000))
+ == expected_projection
+ )
+
+ # scarf.storage.artifacts._stored_array_chunk_rows
+ if array.ndim > 0 and chunks is not None:
+ expected_artifact = max(int(chunks[0]), 1)
+ else:
+ expected_artifact = 1
+ assert row_band(geometry, unit="chunk", fallback=1) == expected_artifact
+
+
+def _plan_with(
+ *,
+ resources: ResourceBudget,
+ features: np.ndarray,
+ batch: int | None = 4,
+):
+ array = _array(shape=(25, 32), chunks=(25, 4))
+ cells = np.arange(25)
+ return plan_feature_stream(
+ array,
+ featureAxis=1,
+ cellAxis=0,
+ featureIndices=features,
+ cellIndices=cells,
+ resources=resources,
+ blockBytes=lambda width: width * cells.size * 4,
+ requestedBatchSize=batch,
+ )
+
+
+def test_a_stream_that_cannot_be_admitted_reads_one_chunk_at_a_time() -> None:
+ # Room for one block beside its decode, but not for a second read on top,
+ # so admission fails and the plan must not leave read-ahead concurrency on.
+ plan = _plan_with(
+ resources=ResourceBudget(1_000, 8),
+ features=np.arange(32),
+ )
+
+ assert len(plan.blocks) == 8
+ assert (plan.readWorkers, plan.ioConcurrency) == (1, 1)
+
+
+def test_a_stream_without_read_ahead_uses_current_decode_concurrency() -> None:
+ # A second block cannot fit, but the current block can afford two decoded
+ # chunks. Preserve that inner concurrency without admitting read-ahead.
+ plan = _plan_with(
+ resources=ResourceBudget(1_200, 8),
+ features=np.arange(32),
+ )
+
+ assert len(plan.blocks) == 8
+ assert (plan.readWorkers, plan.ioConcurrency) == (1, 2)
+
+
+def test_a_single_block_stream_budgets_its_own_decode_concurrency() -> None:
+ plan = _plan_with(
+ resources=ResourceBudget(1_600, 8),
+ features=np.arange(4),
+ )
+
+ assert len(plan.blocks) == 1
+ assert plan.readWorkers == 1
+ # One 400 byte block plus three 400 byte decodes exhausts the limit, so the
+ # plan admits three concurrent decodes rather than one per worker.
+ assert plan.ioConcurrency == 3
+ assert 400 + plan.ioConcurrency * 400 <= 1_600
+
+
+def test_an_empty_selection_plans_no_read_concurrency() -> None:
+ plan = _plan_with(
+ resources=ResourceBudget(1_000_000, 8),
+ features=np.array([], dtype=np.int64),
+ )
+
+ assert plan.blocks == ()
+ assert (plan.readWorkers, plan.ioConcurrency) == (1, 1)
+
+
+def test_streamed_row_blocks_decode_only_as_many_chunks_as_budgeted() -> None:
+ store = RecordingStore(delay=0.005)
+ root = zarr.open_group(store=store, mode="w")
+ array = root.create_array(
+ "counts",
+ shape=(64, 32),
+ chunks=(16, 8),
+ dtype=np.uint32,
+ compressors=None,
+ fill_value=0,
+ )
+ expected = np.arange(64 * 32, dtype=np.uint32).reshape(64, 32)
+ array[:] = expected
+
+ # A row block spans the four column chunks of one chunk row, so an
+ # unbounded read can hold all four decodes at once. The budget affords one
+ # 2048 byte block beside a single 512 byte chunk decode, and no more.
+ budgeted = ChunkedArray(array, resources=ResourceBudget(3_000, 4))
+ store.reset()
+ blocks = list(budgeted.stream_blocks())
+
+ np.testing.assert_array_equal(np.vstack(blocks), expected)
+ assert store.max_in_flight_for("get") == 1
+
+ unbudgeted = ChunkedArray(array)
+ store.reset()
+ list(unbudgeted.stream_blocks())
+
+ assert store.max_in_flight_for("get") > 1, (
+ "without a budget Zarr uses its own concurrency, so the probe should "
+ "observe the overlap this test bounds"
+ )
+
+
+def test_planned_reads_never_decode_more_chunks_than_budgeted() -> None:
+ # Budget chosen so admission has to trade read depth against per-read
+ # decodes: charging one decode per read admits four concurrent chunk
+ # decodes, which overruns the limit once they are actually in flight.
+ store = RecordingStore(delay=0.005)
+ root = zarr.open_group(store=store, mode="w")
+ array = root.create_array(
+ "counts",
+ shape=(64, 32),
+ chunks=(16, 4),
+ dtype=np.uint32,
+ compressors=None,
+ fill_value=0,
+ )
+ array[:] = np.arange(64 * 32, dtype=np.uint32).reshape(64, 32)
+
+ cells = np.arange(64)
+ resources = ResourceBudget(4_280, 8)
+ block_bytes = 4 * cells.size * 4
+ plan = plan_feature_stream(
+ array,
+ featureAxis=1,
+ cellAxis=0,
+ featureIndices=np.arange(32),
+ cellIndices=cells,
+ resources=resources,
+ blockBytes=lambda width: width * cells.size * 4,
+ requestedBatchSize=4,
+ )
+
+ assert len(plan.blocks) == 8
+ assert (plan.readWorkers, plan.ioConcurrency) == (2, 1)
+
+ store.reset()
+ blocks = list(
+ iter_column_blocks(
+ len(plan.blocks),
+ lambda index: _read_block(array, cells, plan.blocks[index].indices),
+ workers=plan.readWorkers,
+ io_concurrency=plan.ioConcurrency,
+ )
+ )
+
+ peak = store.max_in_flight_for("get")
+ decode = plan.geometry.nominalChunkBytes()
+ resident = block_bytes + decode
+ live = resident + plan.readWorkers * block_bytes + peak * decode
+
+ assert len(blocks) == 8
+ assert peak >= 2, "the probe never observed overlapping reads"
+ assert peak <= plan.readWorkers * plan.ioConcurrency
+ assert live <= resources.memoryBytes
diff --git a/tests/test_symphony_mapping.py b/tests/test_symphony_mapping.py
new file mode 100644
index 00000000..8c361213
--- /dev/null
+++ b/tests/test_symphony_mapping.py
@@ -0,0 +1,445 @@
+import json
+from dataclasses import replace
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+import pytest
+
+from scarf.embeddings.harmony import fit_harmony
+from scarf.mapping.models import (
+ ScaledPCAProjectionModel,
+ SymphonyCorrectionModel,
+)
+from scarf.mapping.symphony import (
+ accumulate_sufficient_statistics,
+ apply_query_correction,
+ initialize_sufficient_statistics,
+ project_pca,
+ scaled_dispersion_sum,
+ soft_cluster_assignments,
+ solve_query_correction,
+ weighted_centroids,
+ zero_norm_rows,
+)
+
+
+def _single_cluster_reference() -> tuple[
+ ScaledPCAProjectionModel,
+ SymphonyCorrectionModel,
+]:
+ return (
+ ScaledPCAProjectionModel(
+ feature_means=np.zeros(2),
+ feature_scales=np.ones(2),
+ loadings=np.eye(2),
+ ),
+ SymphonyCorrectionModel(
+ centroids=np.array([[1.0, 0.0]]),
+ raw_centroids=np.zeros((1, 2)),
+ corrected_centroids=np.zeros((1, 2)),
+ cluster_mass=np.array([4.0]),
+ sigma=np.array([0.1]),
+ ),
+ )
+
+
+def test_scaled_dispersion_reads_one_for_data_matching_the_reference():
+ rng = np.random.default_rng(0)
+ reference = rng.normal(size=(400, 6))
+ model = ScaledPCAProjectionModel(
+ feature_means=reference.mean(axis=0),
+ feature_scales=reference.std(axis=0),
+ loadings=np.eye(6),
+ )
+ n_features = reference.shape[1]
+
+ def dispersion(values: np.ndarray) -> float:
+ return scaled_dispersion_sum(values, model) / (len(values) * n_features)
+
+ # The reference is z-scored by construction, so it must read exactly 1.
+ assert dispersion(reference) == pytest.approx(1.0)
+ # A query holding a quarter of the reference spread reads a sixteenth, since
+ # the statistic is a variance rather than a standard deviation.
+ compressed = model.feature_means + 0.25 * (reference - model.feature_means)
+ assert dispersion(compressed) == pytest.approx(0.0625)
+ widened = model.feature_means + 2.0 * (reference - model.feature_means)
+ assert dispersion(widened) == pytest.approx(4.0)
+ # An offset query is not narrow, so a pure shift must not look compressed.
+ assert dispersion(reference + 3.0 * model.feature_scales) == pytest.approx(10.0)
+
+ with pytest.raises(ValueError, match="Expected query matrix"):
+ scaled_dispersion_sum(reference[:, :2], model)
+
+
+def _correct(
+ projection: ScaledPCAProjectionModel,
+ correction_model: SymphonyCorrectionModel,
+ query: np.ndarray,
+ batch_codes: np.ndarray,
+) -> np.ndarray:
+ projected = project_pca(query, projection)
+ assignments = soft_cluster_assignments(projected, correction_model)
+ counts, sums = initialize_sufficient_statistics(
+ batch_codes.max() + 1,
+ correction_model,
+ )
+ accumulate_sufficient_statistics(counts, sums, projected, assignments, batch_codes)
+ correction = solve_query_correction(counts, sums, correction_model)
+ return apply_query_correction(
+ projected,
+ assignments,
+ batch_codes,
+ correction_model,
+ correction,
+ )
+
+
+def test_symphony_closed_form_affine_shift_fixture():
+ projection, correction_model = _single_cluster_reference()
+ unshifted = np.array([[-1.0, 0.0], [1.0, 0.0]])
+ shifted = unshifted + np.array([3.0, -2.0])
+
+ corrected = _correct(
+ projection,
+ correction_model,
+ shifted,
+ np.zeros(2, dtype=np.int64),
+ )
+
+ np.testing.assert_allclose(
+ corrected,
+ np.array([[2 / 7, -6 / 7], [16 / 7, -6 / 7]]),
+ atol=1e-12,
+ )
+
+
+def test_symphony_no_shift_control_is_identity_for_centered_query():
+ projection, correction_model = _single_cluster_reference()
+ query = np.array([[-2.0, 1.0], [2.0, -1.0]])
+
+ corrected = _correct(
+ projection,
+ correction_model,
+ query,
+ np.zeros(2, dtype=np.int64),
+ )
+
+ np.testing.assert_allclose(corrected, query, atol=1e-12)
+
+
+def test_symphony_handles_empty_cluster_batch_statistics_without_nan():
+ _, correction_model = _single_cluster_reference()
+ counts, sums = initialize_sufficient_statistics(2, correction_model)
+ counts[0, 0] = 2.0
+ sums[0, 0] = np.array([4.0, -2.0])
+
+ correction = solve_query_correction(counts, sums, correction_model)
+
+ assert np.all(np.isfinite(correction.batch_offsets))
+ np.testing.assert_array_equal(correction.batch_offsets[1], np.zeros((1, 2)))
+
+
+def test_symphony_no_shift_composition_subset_stays_stable():
+ projection = ScaledPCAProjectionModel(
+ feature_means=np.zeros(2),
+ feature_scales=np.ones(2),
+ loadings=np.eye(2),
+ )
+ correction_model = SymphonyCorrectionModel(
+ centroids=np.array([[-2.0, 0.0], [2.0, 0.0]]),
+ raw_centroids=np.array([[-2.0, 0.0], [2.0, 0.0]]),
+ corrected_centroids=np.array([[-2.0, 0.0], [2.0, 0.0]]),
+ cluster_mass=np.array([100.0, 100.0]),
+ sigma=np.array([0.1, 0.1]),
+ )
+ cluster_zero_only = np.array([[-3.0, 0.0], [-1.0, 0.0]])
+
+ corrected = _correct(
+ projection,
+ correction_model,
+ cluster_zero_only,
+ np.zeros(2, dtype=np.int64),
+ )
+
+ np.testing.assert_allclose(corrected, cluster_zero_only, atol=1e-12)
+
+
+def test_symphony_composition_imbalance_preserves_distinct_populations():
+ projection = ScaledPCAProjectionModel(
+ feature_means=np.zeros(2),
+ feature_scales=np.ones(2),
+ loadings=np.eye(2),
+ )
+ correction_model = SymphonyCorrectionModel(
+ centroids=np.array([[-1.0, 0.0], [1.0, 0.0]]),
+ raw_centroids=np.array([[-2.0, 0.0], [2.0, 0.0]]),
+ corrected_centroids=np.array([[-2.0, 0.0], [2.0, 0.0]]),
+ cluster_mass=np.array([100.0, 100.0]),
+ sigma=np.array([0.01, 0.01]),
+ )
+ query = np.array(
+ [
+ [-3.0, 0.0],
+ [-2.5, 0.0],
+ [-1.5, 0.0],
+ [-1.0, 0.0],
+ [1.5, 0.0],
+ [2.5, 0.0],
+ ]
+ )
+ populations = np.array([0, 0, 0, 0, 1, 1])
+
+ corrected = _correct(
+ projection,
+ correction_model,
+ query,
+ np.zeros(len(query), dtype=np.int64),
+ )
+ separation = corrected[populations == 1].mean(axis=0) - corrected[
+ populations == 0
+ ].mean(axis=0)
+
+ np.testing.assert_allclose(corrected, query, atol=1e-12)
+ assert separation[0] == 4.0
+
+
+def test_symphony_joint_ridge_handles_independent_query_batches():
+ projection, correction_model = _single_cluster_reference()
+ centered = np.array([[-1.0, 0.0], [1.0, 0.0], [-2.0, 1.0], [2.0, -1.0]])
+ batches = np.array([0, 0, 1, 1], dtype=np.int64)
+ shifted = centered.copy()
+ shifted[batches == 0] += np.array([3.0, -2.0])
+ shifted[batches == 1] += np.array([-4.0, 5.0])
+
+ corrected = _correct(projection, correction_model, shifted, batches)
+
+ np.testing.assert_allclose(
+ corrected,
+ np.array(
+ [
+ [-1 / 12, -5 / 12],
+ [23 / 12, -5 / 12],
+ [-41 / 12, 35 / 12],
+ [7 / 12, 11 / 12],
+ ]
+ ),
+ atol=1e-12,
+ )
+
+
+def test_symphony_joint_ridge_shrinks_query_shift():
+ projection, correction_model = _single_cluster_reference()
+ reference = np.array([[-2.0, 0.0], [-1.0, 0.0], [1.0, 0.0], [2.0, 0.0]])
+ shifted = reference + np.array([8.0, -3.0])
+ corrected = _correct(
+ projection,
+ correction_model,
+ shifted,
+ np.zeros(4, dtype=np.int64),
+ )
+
+ assert np.linalg.norm(corrected.mean(axis=0)) < np.linalg.norm(shifted.mean(axis=0))
+
+
+def test_symphony_correction_is_row_order_invariant():
+ projection, correction_model = _single_cluster_reference()
+ query = np.array([[2.0, -1.0], [4.0, -1.0], [-3.0, 2.0], [-1.0, 2.0]])
+ batches = np.array([0, 0, 1, 1], dtype=np.int64)
+ order = np.array([2, 0, 3, 1])
+
+ expected = _correct(projection, correction_model, query, batches)
+ reordered = _correct(
+ projection,
+ correction_model,
+ query[order],
+ batches[order],
+ )
+ restored = np.empty_like(reordered)
+ restored[order] = reordered
+
+ np.testing.assert_allclose(restored, expected, atol=1e-12)
+
+
+def test_symphony_sufficient_statistics_are_chunk_invariant():
+ projection, correction_model = _single_cluster_reference()
+ query = np.array([[2.0, -1.0], [4.0, -1.0], [-3.0, 2.0], [-1.0, 2.0]])
+ batches = np.array([0, 0, 1, 1], dtype=np.int64)
+ coordinates = project_pca(query, projection)
+ assignments = soft_cluster_assignments(coordinates, correction_model)
+
+ expected_counts, expected_sums = initialize_sufficient_statistics(
+ 2,
+ correction_model,
+ )
+ accumulate_sufficient_statistics(
+ expected_counts,
+ expected_sums,
+ coordinates,
+ assignments,
+ batches,
+ )
+ chunked_counts, chunked_sums = initialize_sufficient_statistics(
+ 2,
+ correction_model,
+ )
+ for start, stop in ((0, 1), (1, 3), (3, 4)):
+ accumulate_sufficient_statistics(
+ chunked_counts,
+ chunked_sums,
+ coordinates[start:stop],
+ assignments[start:stop],
+ batches[start:stop],
+ )
+
+ np.testing.assert_allclose(chunked_counts, expected_counts, atol=1e-12)
+ np.testing.assert_allclose(chunked_sums, expected_sums, atol=1e-12)
+
+
+def test_symphony_query_correction_applies_the_upstream_unit_penalty():
+ _, correction_model = _single_cluster_reference()
+ counts = np.array([[1.0]])
+ sums = np.array([[[5.0, 0.0]]])
+
+ correction = solve_query_correction(counts, sums, correction_model)
+
+ # One cluster of mass 4 centred on the origin and one query batch holding a
+ # single cell at (5, 0). Upstream Symphony penalises each batch term by a
+ # fixed unit ridge, so the design cross-product is [[5, 1], [1, 2]] and the
+ # batch offset solves to 20 / 9. Any other penalty moves this number.
+ np.testing.assert_allclose(
+ correction.batch_offsets,
+ np.array([[[20.0 / 9.0, 0.0]]]),
+ rtol=0,
+ atol=1e-12,
+ )
+
+
+def test_zero_norm_rows_are_identified_without_nonfinite_assignments():
+ _, correction_model = _single_cluster_reference()
+ coordinates = np.array([[0.0, 0.0], [1.0, 0.0]])
+
+ mask = zero_norm_rows(coordinates)
+ assignments = soft_cluster_assignments(coordinates, correction_model)
+
+ assert mask.tolist() == [True, False]
+ assert np.all(np.isfinite(assignments))
+
+
+def test_harmony_accepts_numpy_scalar_sigma():
+ embedding = np.array([[-1.0, -0.8, 0.8, 1.0], [0.0, 0.2, -0.2, 0.0]])
+ metadata = pd.DataFrame({"batch": ["a", "a", "b", "b"]})
+
+ result = fit_harmony(
+ embedding,
+ metadata,
+ sigma=np.float64(0.1),
+ theta=np.float64(2.0),
+ lamb=np.int64(3),
+ nclust=2,
+ max_iter_harmony=2,
+ max_iter_kmeans=2,
+ )
+
+ assert result.sigma.shape == (2,)
+ assert result.parameters["sigma"] == [0.1, 0.1]
+ assert result.parameters["theta"] == [2.0, 2.0]
+ assert result.parameters["lambda"] == [3.0, 3.0]
+ assert result.parameters["clusterBackend"] == "sklearn.cluster.KMeans"
+
+
+def test_weighted_centroids_reject_empty_reference_clusters():
+ coordinates = np.array([[0.0, 0.0], [1.0, 1.0]])
+ assignments = np.array([[1.0, 1.0], [0.0, 0.0]])
+
+ with np.testing.assert_raises_regex(ValueError, "empty cluster"):
+ weighted_centroids(coordinates, assignments)
+ with np.testing.assert_raises_regex(ValueError, "masses must be positive"):
+ _, correction_model = _single_cluster_reference()
+ replace(correction_model, cluster_mass=np.array([0.0]))
+
+
+def test_symphony_r_0_1_3_static_golden_fixture():
+ fixture = json.loads(
+ (Path(__file__).parent / "symphony_r_0_1_3_golden.json").read_text()
+ )
+ reference = fixture["reference"]
+ projection = ScaledPCAProjectionModel(
+ feature_means=np.asarray(reference["featureMeans"]),
+ feature_scales=np.asarray(reference["featureScales"]),
+ loadings=np.asarray(reference["loadings"]),
+ )
+ correction_model = SymphonyCorrectionModel(
+ centroids=np.asarray(reference["centroids"]),
+ raw_centroids=np.asarray(reference["rawCentroids"]),
+ corrected_centroids=np.asarray(reference["correctedCentroids"]),
+ cluster_mass=np.asarray(reference["clusterMass"]),
+ sigma=np.asarray(reference["sigma"]),
+ )
+ query = np.asarray(fixture["query"])
+ batch_codes = np.asarray(fixture["batchCodes"], dtype=np.int64)
+ projected = project_pca(query, projection)
+ assignments = soft_cluster_assignments(projected, correction_model)
+ corrected = _correct(projection, correction_model, query, batch_codes)
+
+ assert fixture["provenance"]["packageVersion"] == "0.1.3"
+ np.testing.assert_allclose(
+ projected,
+ fixture["expectedProjected"],
+ rtol=0,
+ atol=1e-15,
+ )
+ np.testing.assert_allclose(
+ assignments,
+ fixture["expectedAssignments"],
+ rtol=1e-12,
+ atol=1e-18,
+ )
+ np.testing.assert_allclose(
+ corrected,
+ fixture["expectedCorrected"],
+ rtol=0,
+ atol=1e-12,
+ )
+
+ nonzero = fixture["nonzeroCorrection"]
+ nonzero_reference = nonzero["reference"]
+ nonzero_projection = ScaledPCAProjectionModel(
+ feature_means=np.asarray(nonzero_reference["featureMeans"]),
+ feature_scales=np.asarray(nonzero_reference["featureScales"]),
+ loadings=np.asarray(nonzero_reference["loadings"]),
+ )
+ nonzero_correction_model = SymphonyCorrectionModel(
+ centroids=np.asarray(nonzero_reference["centroids"]),
+ raw_centroids=np.asarray(nonzero_reference["rawCentroids"]),
+ corrected_centroids=np.asarray(nonzero_reference["correctedCentroids"]),
+ cluster_mass=np.asarray(nonzero_reference["clusterMass"]),
+ sigma=np.asarray(nonzero_reference["sigma"]),
+ )
+ nonzero_query = np.asarray(nonzero["query"])
+ nonzero_batches = np.asarray(nonzero["batchCodes"], dtype=np.int64)
+ nonzero_projected = project_pca(nonzero_query, nonzero_projection)
+ nonzero_assignments = soft_cluster_assignments(
+ nonzero_projected,
+ nonzero_correction_model,
+ )
+ nonzero_corrected = _correct(
+ nonzero_projection,
+ nonzero_correction_model,
+ nonzero_query,
+ nonzero_batches,
+ )
+
+ np.testing.assert_allclose(
+ nonzero_assignments,
+ nonzero["expectedAssignments"],
+ rtol=0,
+ atol=1e-15,
+ )
+ np.testing.assert_allclose(
+ nonzero_corrected,
+ nonzero["expectedCorrected"],
+ rtol=0,
+ atol=1e-12,
+ )
+ assert not np.allclose(nonzero_corrected, nonzero_query)
diff --git a/tests/test_types_and_utils.py b/tests/test_types_and_utils.py
new file mode 100644
index 00000000..f94a0303
--- /dev/null
+++ b/tests/test_types_and_utils.py
@@ -0,0 +1,216 @@
+import gc
+import weakref
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.storage.types import (
+ array_metadata_shards,
+ as_zarr_array,
+ as_zarr_group,
+)
+from scarf.utils import (
+ array_digest,
+ clean_array,
+ configure_output,
+ permute_into_chunks,
+ rescale_array,
+ rolling_window,
+ set_verbosity,
+ show_dask_progress,
+ tqdmbar,
+)
+from scarf.utils.progress import iter_progress
+
+
+def test_as_zarr_array_accepts_array():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ arr = root.create_array("data", shape=(3,), dtype="f8")
+ assert as_zarr_array(arr) is arr
+
+
+def test_as_zarr_array_rejects_group():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ group = root.create_group("nested")
+ with pytest.raises(TypeError, match="Expected Zarr array"):
+ as_zarr_array(group, name="nested")
+
+
+def test_as_zarr_group_accepts_group():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ group = root.create_group("nested")
+ assert as_zarr_group(group) is group
+
+
+def test_as_zarr_group_rejects_array():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ arr = root.create_array("data", shape=(2,), dtype="f8")
+ with pytest.raises(TypeError, match="Expected Zarr group"):
+ as_zarr_group(arr, name="data")
+
+
+def test_array_metadata_shards_returns_none_without_sharding():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ arr = root.create_array("data", shape=(4,), dtype="f8")
+ assert array_metadata_shards(arr) is None
+
+
+def test_clean_array_replaces_nan_inf_and_zero():
+ raw = np.array([1.0, np.nan, np.inf, -np.inf, 0.0])
+ cleaned = clean_array(raw, fill_val=-1.0)
+ assert cleaned[0] == 1.0
+ assert cleaned[-1] == -1.0
+ assert np.all(np.isfinite(cleaned))
+
+
+def test_rescale_array_trims_extreme_values():
+ values = np.concatenate([np.linspace(-2, 2, 499), np.array([100.0])])
+ trimmed = rescale_array(values, frac=0.9)
+ assert trimmed.max() < 100.0
+ assert trimmed.min() > -100.0
+
+
+def test_set_verbosity_rejects_invalid_level():
+ with pytest.raises(ValueError, match="Please provide a value for level"):
+ set_verbosity("NOT_A_REAL_LEVEL")
+
+
+def test_set_verbosity_accepts_valid_level():
+ set_verbosity("ERROR")
+ set_verbosity("INFO")
+
+
+def test_tqdmbar_uses_explicit_progress_independently_of_severity(monkeypatch):
+ import tqdm.auto as tqdm_auto
+
+ captured: list[bool] = []
+
+ class FakeTqdm:
+ def __init__(self, *args, **kwargs):
+ captured.append(bool(kwargs.get("disable")))
+
+ def __iter__(self):
+ return iter(())
+
+ monkeypatch.setattr(tqdm_auto, "tqdm", FakeTqdm)
+ try:
+ configure_output(level="ERROR", progress=True)
+ list(tqdmbar(range(1), desc="test"))
+ set_verbosity("WARNING")
+ list(tqdmbar(range(1), desc="test"))
+ configure_output(level="DEBUG", progress=False)
+ list(tqdmbar(range(1), desc="test"))
+ list(tqdmbar(range(1), desc="test", disable=False))
+ configure_output(progress=True)
+ list(tqdmbar(range(1), desc="test", disable=True))
+ finally:
+ configure_output(level="INFO", progress=False, timestamps=False)
+
+ assert captured == [False, False, True, True, True]
+
+
+def test_show_dask_progress_uses_explicit_progress_setting():
+ calls: list[tuple[int, str | None]] = []
+
+ class Deferred:
+ def compute(self, nthreads, msg):
+ calls.append((nthreads, msg))
+ return np.array([1])
+
+ try:
+ configure_output(progress=False)
+ show_dask_progress(Deferred(), "Computing", 2)
+ configure_output(progress=True)
+ show_dask_progress(Deferred(), "Computing", 3)
+ finally:
+ configure_output(progress=False)
+
+ assert calls == [(2, None), (3, "Computing")]
+
+
+def test_progress_iterator_releases_consumed_values():
+ references: list[weakref.ReferenceType[object]] = []
+
+ class Chunk:
+ pass
+
+ def source():
+ for _ in range(2):
+ chunk = Chunk()
+ references.append(weakref.ref(chunk))
+ yield chunk
+
+ stream = iter_progress(source(), total=2, disable=True)
+ first = next(stream)
+ second = next(stream)
+ del first
+ gc.collect()
+
+ assert references[0]() is None
+ assert references[1]() is second
+ stream.close()
+
+
+def test_progress_iterator_closes_source_on_early_exit():
+ closed = False
+
+ def source():
+ nonlocal closed
+ try:
+ yield object()
+ yield object()
+ finally:
+ closed = True
+
+ stream = iter_progress(source(), total=2, disable=True)
+ next(stream)
+ stream.close()
+
+ assert closed
+
+
+def test_rolling_window_smoothes_along_rows():
+ data = np.arange(10, dtype=float).reshape(5, 2)
+ smoothed = rolling_window(data, w=3)
+ expected = np.array(
+ [
+ [1.0, 2.0],
+ [2.0, 3.0],
+ [4.0, 5.0],
+ [6.0, 7.0],
+ [7.0, 8.0],
+ ]
+ )
+ assert np.array_equal(smoothed, expected)
+
+
+def test_rolling_window_even_and_oversized_windows():
+ data = np.arange(5, dtype=float).reshape(-1, 1)
+
+ assert np.array_equal(rolling_window(np.array([[7.0]]), w=1), [[7.0]])
+ assert np.array_equal(
+ rolling_window(data, w=2).ravel(),
+ [0.5, 1.5, 2.5, 3.5, 4.0],
+ )
+ assert np.array_equal(
+ rolling_window(data, w=20).ravel(),
+ [1.0, 1.5, 2.0, 2.5, 3.0],
+ )
+ for window_size in (0, -1):
+ with pytest.raises(ValueError, match="greater than zero"):
+ rolling_window(data, w=window_size)
+
+
+def test_array_digest_is_deterministic_and_shape_sensitive():
+ values = np.arange(6, dtype=np.int64)
+
+ assert array_digest(values) == array_digest(values.copy())
+ assert array_digest(values) != array_digest(values.reshape(2, 3))
+
+
+def test_permute_into_chunks_preserves_all_indices():
+ chunks = permute_into_chunks(10, 3, seed=7)
+ merged = np.concatenate(chunks)
+ assert np.array_equal(np.sort(merged), np.arange(10))
diff --git a/tests/test_umap_and_harmony.py b/tests/test_umap_and_harmony.py
new file mode 100644
index 00000000..07871bc7
--- /dev/null
+++ b/tests/test_umap_and_harmony.py
@@ -0,0 +1,207 @@
+import numpy as np
+import pandas as pd
+import pytest
+from scipy.sparse import coo_matrix
+
+from scarf.embeddings.initialization import initial_embedding
+from scarf.embeddings.umap import (
+ calc_dens_map_params,
+ fit_transform,
+ fuzzy_simplicial_set,
+ simplicial_set_embedding,
+)
+from scarf.embeddings.harmony import run_harmony
+
+
+def _ring_graph(n: int) -> coo_matrix:
+ rows, cols, data = [], [], []
+ for i in range(n):
+ for j in (i - 1, i + 1):
+ neighbor = j % n
+ if neighbor != i:
+ rows.append(i)
+ cols.append(neighbor)
+ data.append(1.0)
+ return coo_matrix((data, (rows, cols)), shape=(n, n))
+
+
+def test_calc_dens_map_params():
+ graph = _ring_graph(8)
+ dists = np.full((8, 8), 3.0, dtype=np.float32)
+ np.fill_diagonal(dists, 0.0)
+ for i in range(8):
+ for j in (i - 1, i + 1):
+ neighbor = j % 8
+ dists[i, neighbor] = (
+ 1.0 + ((min(i, neighbor) * 3 + max(i, neighbor)) % 5) / 5.0
+ )
+ mu_sum, r_term = calc_dens_map_params(graph, dists)
+ np.testing.assert_array_equal(mu_sum, np.full(8, 4.0, dtype=np.float32))
+ np.testing.assert_allclose(
+ r_term,
+ [
+ -0.1043465,
+ -1.270655,
+ 0.6717964,
+ 1.7731321,
+ 0.89659786,
+ -0.10434671,
+ -1.270655,
+ -0.59152323,
+ ],
+ rtol=1e-6,
+ atol=1e-6,
+ )
+
+
+def test_simplicial_embedding_restores_numba_threads_on_failure(monkeypatch):
+ import numba
+ import umap.layouts
+
+ def fail(**_kwargs):
+ raise RuntimeError("injected UMAP failure")
+
+ monkeypatch.setattr(umap.layouts, "optimize_layout_euclidean", fail)
+ graph = _ring_graph(4)
+ previous_threads = numba.get_num_threads()
+ with pytest.raises(RuntimeError, match="injected UMAP failure"):
+ simplicial_set_embedding(
+ graph,
+ np.zeros((4, 2), dtype=np.float32),
+ 2,
+ 1.0,
+ 1.0,
+ 1,
+ 1.0,
+ 1.0,
+ 5,
+ {},
+ False,
+ 1,
+ False,
+ )
+ assert numba.get_num_threads() == previous_threads
+
+
+def test_fuzzy_simplicial_set_produces_coo_graph():
+ graph = coo_matrix(([1.0, 0.6], ([0, 1], [1, 2])), shape=(4, 4))
+ merged = fuzzy_simplicial_set(graph, 1.0)
+ np.testing.assert_array_equal(
+ merged.toarray(),
+ [
+ [0.0, 1.0, 0.0, 0.0],
+ [1.0, 0.0, 0.6, 0.0],
+ [0.0, 0.6, 0.0, 0.0],
+ [0.0, 0.0, 0.0, 0.0],
+ ],
+ )
+
+
+def test_fit_transform_runs_short_embedding():
+ n_cells = 24
+ graph = _ring_graph(n_cells)
+ ini_embed = np.random.default_rng(0).normal(size=(n_cells, 2)).astype(np.float32)
+
+ def run_embedding() -> tuple[np.ndarray, float, float]:
+ return fit_transform(
+ graph,
+ ini_embed.copy(),
+ spread=1.0,
+ min_dist=0.5,
+ n_epochs=15,
+ random_seed=42,
+ repulsion_strength=1.0,
+ initial_alpha=1.0,
+ negative_sample_rate=5,
+ densmap_kwds={},
+ parallel=False,
+ nthreads=1,
+ verbose=False,
+ )
+
+ embedding, a, b = run_embedding()
+ repeated_embedding, repeated_a, repeated_b = run_embedding()
+
+ assert embedding.shape == (n_cells, 2)
+ assert np.all(np.isfinite(embedding))
+ assert np.all(np.ptp(embedding, axis=0) > 0)
+ assert not np.array_equal(embedding, ini_embed)
+ np.testing.assert_array_equal(embedding, repeated_embedding)
+ np.testing.assert_allclose(
+ [a, b],
+ [0.58303002, 1.33416699],
+ rtol=1e-6,
+ atol=1e-6,
+ )
+ np.testing.assert_allclose([a, b], [repeated_a, repeated_b], rtol=0, atol=0)
+
+
+def test_initial_embedding_matches_regression_values():
+ centers = np.array(
+ [
+ [0.0, 0.0, 0.0],
+ [2.0, 0.0, 1.0],
+ [0.0, 3.0, 2.0],
+ [2.0, 3.0, 4.0],
+ ]
+ )
+ labels = np.array([2, 0, 3, 1, 2])
+
+ actual = initial_embedding(centers, labels, 2)
+
+ np.testing.assert_allclose(
+ actual,
+ [
+ [1.0399364, -1.3556585],
+ [-2.435071, -0.58934784],
+ [2.4441376, 0.7173242],
+ [-1.3938057, 1.3570011],
+ [1.0399364, -1.3556585],
+ ],
+ rtol=1e-6,
+ atol=1e-6,
+ )
+
+
+def test_initial_embedding_accepts_integral_float_labels_and_rejects_invalid():
+ centers = np.eye(3)
+ integral = initial_embedding(centers, np.array([0.0, 1.0, 2.0]), 2)
+ assert integral.shape == (3, 2)
+
+ for labels in (
+ np.array([0.0, 1.5]),
+ np.array([0.0, -1.0]),
+ np.array([0.0, np.nan]),
+ np.array([0, 3]),
+ ):
+ with pytest.raises(ValueError):
+ initial_embedding(centers, labels, 2)
+
+
+def test_run_harmony_corrects_batch_structure():
+ rng = np.random.default_rng(0)
+ n_cells = 180
+ n_dims = 12
+ batch = rng.integers(0, 3, n_cells)
+ batch_effect = rng.normal(scale=2.0, size=(3, n_dims))
+ data = rng.normal(size=(n_dims, n_cells))
+ for cell_idx, batch_id in enumerate(batch):
+ data[:, cell_idx] += batch_effect[batch_id]
+
+ meta = pd.DataFrame({"batch": [f"batch_{x}" for x in batch]})
+ corrected = run_harmony(
+ data,
+ meta,
+ nclust=15,
+ max_iter_harmony=4,
+ max_iter_kmeans=5,
+ random_state=0,
+ )
+
+ assert corrected.shape == data.shape
+ assert np.all(np.isfinite(corrected))
+ batch_means_before = [data[:, batch == b].mean(axis=1) for b in range(3)]
+ batch_means_after = [corrected[:, batch == b].mean(axis=1) for b in range(3)]
+ spread_before = np.std([m.mean() for m in batch_means_before])
+ spread_after = np.std([m.mean() for m in batch_means_after])
+ assert spread_after <= spread_before
diff --git a/tests/test_writers.py b/tests/test_writers.py
new file mode 100644
index 00000000..22bc447c
--- /dev/null
+++ b/tests/test_writers.py
@@ -0,0 +1,1007 @@
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+from scarf.readers import CSVReader
+from scarf.writers import (
+ CSVtoZarr,
+ CrToZarr,
+ SubsetZarr,
+ bed_to_sparse_array,
+ subset_assay_zarr,
+)
+
+
+class _FakeCells:
+ def __init__(
+ self, n_cells: int, columns: dict[str, np.ndarray] | None = None
+ ) -> None:
+ self.N = n_cells
+ self._columns = columns or {}
+
+ def fetch_all(self, key: str) -> np.ndarray:
+ return self._columns[key]
+
+
+class _FakeAssay:
+ def __init__(
+ self,
+ name: str,
+ n_cells: int,
+ columns: dict[str, np.ndarray] | None = None,
+ ) -> None:
+ self.name = name
+ self.cells = _FakeCells(n_cells, columns)
+
+
+def test_crtozarr(crh5_reader, tmp_path):
+ from scarf.writers import CrToZarr
+
+ fn = str(tmp_path / "dummy_1K_pbmc_citeseq.zarr")
+ writer = CrToZarr(crh5_reader, zarr_loc=fn)
+ writer.dump()
+
+
+def test_crtozarr_fromdir(crdir_reader, tmp_path):
+ from scarf.writers import CrToZarr
+
+ fn = str(tmp_path / "1K_pbmc_citeseq_dir.zarr")
+ writer = CrToZarr(crdir_reader, zarr_loc=fn)
+ writer.dump()
+
+
+def test_crtozarr_preserves_exact_counts_metadata_and_transpose():
+ import pandas as pd
+ from scipy.sparse import coo_matrix
+
+ values = np.array(
+ [[1, 0, 2], [0, 3, 0], [4, 5, 6]],
+ dtype=np.uint16,
+ )
+
+ class ExactReader:
+ nCells = values.shape[0]
+ matrix_dtype = values.dtype
+ assayFeats = pd.DataFrame(
+ {"RNA": ["Gene Expression", 0, values.shape[1], values.shape[1]]},
+ index=["type", "start", "end", "nFeatures"],
+ )
+
+ def cell_names(self):
+ return ["c1", "c2", "c3"]
+
+ def feature_ids(self, assay_name):
+ return ["f1", "f2", "f3"]
+
+ def feature_names(self, assay_name):
+ return ["g1", "g2", "g3"]
+
+ def consume(self, batch_size, lines_in_mem):
+ for start in range(0, self.nCells, batch_size):
+ yield coo_matrix(values[start : start + batch_size])
+
+ def max_window_nnz(self, window_rows):
+ width = min(window_rows, self.nCells)
+ return max(
+ np.count_nonzero(values[start : start + width])
+ for start in range(self.nCells - width + 1)
+ )
+
+ def producer_staging_bytes(self, batch_size, lines_in_mem):
+ return 0
+
+ store = MemoryStore()
+ writer = CrToZarr(
+ ExactReader(),
+ zarr_loc=store,
+ dtype="uint16",
+ targetChunkBytes=12,
+ targetShardBytes=12,
+ )
+ writer.dump(batch_size=2)
+
+ root = zarr.open_group(store=store, mode="r")
+ np.testing.assert_array_equal(root["RNA/counts"][:], values)
+ assert "countsT" not in root["RNA"]
+ np.testing.assert_array_equal(root["cellData/ids"][:], ["c1", "c2", "c3"])
+ np.testing.assert_array_equal(root["RNA/featureData/ids"][:], ["f1", "f2", "f3"])
+
+
+def test_h5adtozarr(h5ad_reader, tmp_path):
+ from scarf.writers import H5adToZarr
+
+ fn = str(tmp_path / "bastidas.zarr")
+ writer = H5adToZarr(h5ad_reader, zarr_loc=fn)
+ writer.dump()
+
+
+def test_h5adtozarr_splits_noncontiguous_feature_types():
+ import tempfile
+ from pathlib import Path
+
+ import h5py
+ from scipy.sparse import csr_matrix
+
+ from scarf.readers import H5adReader
+ from scarf.writers import H5adToZarr
+
+ values = np.array(
+ [
+ [1, 2, 0, 3],
+ [4, 0, 5, 0],
+ [0, 6, 7, 8],
+ ],
+ dtype=np.uint16,
+ )
+ matrix = csr_matrix(values)
+
+ with tempfile.TemporaryDirectory() as directory:
+ path = Path(directory) / "multi.h5ad"
+ with h5py.File(path, mode="w") as h5:
+ sparse = h5.create_group("X")
+ sparse.attrs["encoding-type"] = "csr_matrix"
+ sparse.attrs["shape"] = matrix.shape
+ sparse.create_dataset("data", data=matrix.data)
+ sparse.create_dataset("indices", data=matrix.indices)
+ sparse.create_dataset("indptr", data=matrix.indptr)
+
+ obs = h5.create_group("obs")
+ obs.create_dataset("_index", data=np.array([b"c1", b"c2", b"c3"]))
+ obs.create_dataset("batch", data=np.array([b"A", b"A", b"B"]))
+
+ var = h5.create_group("var")
+ var.create_dataset(
+ "_index",
+ data=np.array([b"f1", b"a1", b"f2", b"a2"]),
+ )
+ var.create_dataset(
+ "feature_name",
+ data=np.array([b"g1", b"p1", b"g2", b"p2"]),
+ )
+ var.create_dataset(
+ "feature_types",
+ data=np.array(
+ [
+ b"Gene Expression",
+ b"Antibody Capture",
+ b"Gene Expression",
+ b"Antibody Capture",
+ ]
+ ),
+ )
+ var.create_dataset(
+ "chromosome",
+ data=np.array([b"1", b"na", b"2", b"na"]),
+ )
+
+ reader = H5adReader(str(path), feature_name_key="feature_name")
+ store = MemoryStore()
+ try:
+ assert tuple(
+ reader.assay_feature_slices(
+ "feature_types",
+ {"Antibody Capture": "HTO"},
+ )
+ ) == ("RNA", "HTO")
+ writer = H5adToZarr(
+ reader,
+ zarr_loc=store,
+ assay_name="ignored",
+ assay_split_key="feature_types",
+ targetChunkBytes=8,
+ targetShardBytes=8,
+ )
+ writer.dump(batch_size=2)
+ finally:
+ reader.h5.close()
+
+ root = zarr.open_group(store=store, mode="r")
+ assert set(root.group_keys()) == {"cellData", "RNA", "ADT"}
+ np.testing.assert_array_equal(root["RNA/counts"][:], values[:, [0, 2]])
+ np.testing.assert_array_equal(root["ADT/counts"][:], values[:, [1, 3]])
+ assert "countsT" not in root["RNA"]
+ assert "countsT" not in root["ADT"]
+ np.testing.assert_array_equal(root["cellData/batch"][:], ["A", "A", "B"])
+ np.testing.assert_array_equal(root["RNA/featureData/ids"][:], ["f1", "f2"])
+ np.testing.assert_array_equal(root["ADT/featureData/ids"][:], ["a1", "a2"])
+ np.testing.assert_array_equal(
+ root["RNA/featureData/chromosome"][:],
+ ["1", "2"],
+ )
+ np.testing.assert_array_equal(
+ root["ADT/featureData/feature_types"][:],
+ ["Antibody Capture", "Antibody Capture"],
+ )
+
+
+def _write_h5ad(
+ path,
+ values: np.ndarray,
+ *,
+ encoding: str = "csr",
+ feature_types: list[bytes] | None = None,
+):
+ """Write a minimal AnnData file with the requested matrix encoding."""
+ import h5py
+ from scipy.sparse import csc_matrix, csr_matrix
+
+ n_cells, n_feats = values.shape
+ with h5py.File(path, mode="w") as h5:
+ if encoding == "dense":
+ h5.create_dataset("X", data=values)
+ else:
+ matrix = csr_matrix(values) if encoding == "csr" else csc_matrix(values)
+ group = h5.create_group("X")
+ group.attrs["encoding-type"] = f"{encoding}_matrix"
+ group.attrs["shape"] = values.shape
+ group.create_dataset("data", data=matrix.data)
+ group.create_dataset("indices", data=matrix.indices)
+ group.create_dataset("indptr", data=matrix.indptr)
+
+ obs = h5.create_group("obs")
+ obs.create_dataset(
+ "_index",
+ data=np.array([f"c{i}".encode() for i in range(n_cells)]),
+ )
+ var = h5.create_group("var")
+ var.create_dataset(
+ "_index",
+ data=np.array([f"f{i}".encode() for i in range(n_feats)]),
+ )
+ var.create_dataset(
+ "feature_name",
+ data=np.array([f"g{i}".encode() for i in range(n_feats)]),
+ )
+ if feature_types is not None:
+ var.create_dataset("feature_types", data=np.array(feature_types))
+ return path
+
+
+# Sizes a 12-row uint32 assay into (4, n_feats) row shards.
+_SHARD_BAND_BUDGET = {
+ "mem_budget": 1024**2,
+ "nthreads": 4,
+ "targetChunkBytes": 48,
+ "targetShardBytes": 48,
+}
+
+
+def _band_counts(n_cells: int, n_feats: int) -> np.ndarray:
+ rng = np.random.default_rng(7)
+ values = rng.integers(0, 5, size=(n_cells, n_feats), dtype=np.uint32)
+ values[4:8] = 0
+ return values
+
+
+@pytest.mark.parametrize("encoding", ["csr", "csc", "dense"])
+def test_h5adtozarr_writes_shard_bands_for_every_encoding(tmp_path, encoding):
+ from scarf.readers import H5adReader
+ from scarf.storage.types import array_metadata_shards
+ from scarf.writers import H5adToZarr
+
+ values = _band_counts(12, 3)
+ path = _write_h5ad(tmp_path / f"{encoding}.h5ad", values, encoding=encoding)
+ reader = H5adReader(str(path), feature_name_key="feature_name")
+ store = MemoryStore()
+ try:
+ H5adToZarr(reader, zarr_loc=store, **_SHARD_BAND_BUDGET).dump(batch_size=5)
+ finally:
+ reader.h5.close()
+
+ root = zarr.open_group(store=store, mode="r")
+ counts = root["RNA/counts"]
+ assert array_metadata_shards(counts) == (4, 3)
+ np.testing.assert_array_equal(counts[:], values)
+ assert "countsT" not in root["RNA"]
+
+
+@pytest.mark.parametrize(
+ ("fractional", "expected_dtype"),
+ [(False, np.dtype("uint16")), (True, np.dtype("float32"))],
+)
+def test_h5adtozarr_uses_smallest_lossless_dtype_for_float_counts(
+ tmp_path,
+ fractional,
+ expected_dtype,
+):
+ from scarf.readers import H5adReader
+ from scarf.writers import H5adToZarr
+
+ values = _band_counts(12, 3).astype(np.float32)
+ values[0, 0] = 1.5 if fractional else 3308
+ path = _write_h5ad(tmp_path / "float_counts.h5ad", values, encoding="csr")
+ reader = H5adReader(str(path), feature_name_key="feature_name")
+ store = MemoryStore()
+ try:
+ H5adToZarr(
+ reader,
+ zarr_loc=store,
+ mem_budget=1024**2,
+ nthreads=2,
+ targetChunkBytes=48,
+ targetShardBytes=48,
+ ).dump(batch_size=4)
+ finally:
+ reader.h5.close()
+
+ root = zarr.open_group(store=store, mode="r")
+ assert np.dtype(root["RNA/counts"].dtype) == expected_dtype
+ np.testing.assert_array_equal(root["RNA/counts"][:], values)
+ assert "countsT" not in root["RNA"]
+
+
+@pytest.mark.parametrize("encoding", ["csr", "csc"])
+def test_h5adtozarr_preserves_duplicate_coordinate_sums(tmp_path, encoding):
+ import h5py
+
+ from scarf.readers import H5adReader
+ from scarf.writers import H5adToZarr
+
+ path = _write_h5ad(
+ tmp_path / f"duplicate_{encoding}.h5ad",
+ np.zeros((1, 1), dtype=np.float32),
+ encoding=encoding,
+ )
+ with h5py.File(path, mode="r+") as h5:
+ group = h5["X"]
+ for name in ("data", "indices", "indptr"):
+ del group[name]
+ group.create_dataset("data", data=np.array([200, 100], dtype=np.float32))
+ group.create_dataset("indices", data=np.array([0, 0], dtype=np.int32))
+ group.create_dataset("indptr", data=np.array([0, 2], dtype=np.int32))
+
+ reader = H5adReader(str(path), feature_name_key="feature_name")
+ store = MemoryStore()
+ try:
+ H5adToZarr(
+ reader,
+ zarr_loc=store,
+ mem_budget=1024**2,
+ nthreads=2,
+ targetChunkBytes=16,
+ targetShardBytes=16,
+ ).dump(batch_size=1)
+ finally:
+ reader.h5.close()
+
+ counts = zarr.open_group(store=store, mode="r")["RNA/counts"]
+ assert counts.dtype == np.dtype("float32")
+ assert counts[0, 0] == 300
+
+
+@pytest.mark.parametrize("encoding", ["csr", "csc"])
+def test_h5adtozarr_reduces_duplicates_before_explicit_dtype_cast(
+ tmp_path,
+ encoding,
+):
+ import h5py
+
+ from scarf.readers import H5adReader
+ from scarf.writers import H5adToZarr
+
+ path = _write_h5ad(
+ tmp_path / f"explicit_duplicate_{encoding}.h5ad",
+ np.zeros((1, 1), dtype=np.float32),
+ encoding=encoding,
+ )
+ with h5py.File(path, mode="r+") as h5:
+ group = h5["X"]
+ for name in ("data", "indices", "indptr"):
+ del group[name]
+ group.create_dataset(
+ "data",
+ data=np.array([100.5, 99.5], dtype=np.float32),
+ )
+ group.create_dataset("indices", data=np.array([0, 0], dtype=np.int32))
+ group.create_dataset("indptr", data=np.array([0, 2], dtype=np.int32))
+
+ reader = H5adReader(
+ str(path),
+ feature_name_key="feature_name",
+ dtype="uint8",
+ )
+ store = MemoryStore()
+ try:
+ H5adToZarr(
+ reader,
+ zarr_loc=store,
+ mem_budget=1024**2,
+ nthreads=1,
+ targetChunkBytes=16,
+ targetShardBytes=16,
+ ).dump(batch_size=1)
+ finally:
+ reader.h5.close()
+
+ counts = zarr.open_group(store=store, mode="r")["RNA/counts"]
+ assert counts.dtype == np.dtype("uint8")
+ assert counts[0, 0] == 200
+
+
+def test_h5adtozarr_reads_the_source_once_for_all_assays(tmp_path, monkeypatch):
+ from scarf.readers import H5adReader
+ from scarf.writers import H5adToZarr
+
+ values = _band_counts(12, 4)
+ # An empty leading band for the second assay must not shift its row offsets.
+ values[:5, [1, 3]] = 0
+ path = _write_h5ad(
+ tmp_path / "multi.h5ad",
+ values,
+ feature_types=[
+ b"Gene Expression",
+ b"Antibody Capture",
+ b"Gene Expression",
+ b"Antibody Capture",
+ ],
+ )
+
+ consumed: list[int] = []
+ original = H5adReader.consume
+
+ def spy(self, batch_size):
+ consumed.append(batch_size)
+ return original(self, batch_size)
+
+ monkeypatch.setattr(H5adReader, "consume", spy)
+
+ reader = H5adReader(str(path), feature_name_key="feature_name")
+ store = MemoryStore()
+ try:
+ H5adToZarr(
+ reader,
+ zarr_loc=store,
+ assay_split_key="feature_types",
+ **_SHARD_BAND_BUDGET,
+ ).dump(batch_size=5)
+ finally:
+ reader.h5.close()
+
+ assert consumed == [5]
+ root = zarr.open_group(store=store, mode="r")
+ assert set(root.group_keys()) == {"cellData", "RNA", "ADT"}
+ np.testing.assert_array_equal(root["RNA/counts"][:], values[:, [0, 2]])
+ np.testing.assert_array_equal(root["ADT/counts"][:], values[:, [1, 3]])
+ assert "countsT" not in root["RNA"]
+ assert "countsT" not in root["ADT"]
+ assert root["RNA/counts"].dtype == values.dtype
+
+
+def test_h5adtozarr_small_assay_does_not_serialize_row_band_writes(
+ tmp_path,
+):
+ from scarf.readers import H5adReader
+ from scarf.writers import H5adToZarr
+ from tests.store_probes import RecordingStore
+
+ values = _band_counts(12, 4)
+ path = _write_h5ad(
+ tmp_path / "uneven_multi.h5ad",
+ values,
+ feature_types=[
+ b"Gene Expression",
+ b"Gene Expression",
+ b"Gene Expression",
+ b"Antibody Capture",
+ ],
+ )
+ reader = H5adReader(str(path), feature_name_key="feature_name")
+ store = RecordingStore(delay=0.01)
+ try:
+ writer = H5adToZarr(
+ reader,
+ zarr_loc=store,
+ assay_split_key="feature_types",
+ **_SHARD_BAND_BUDGET,
+ )
+ store.reset()
+ writer.dump(batch_size=5)
+ finally:
+ reader.h5.close()
+
+ assert store.max_in_flight > 1
+ root = zarr.open_group(store=store, mode="r")
+ np.testing.assert_array_equal(root["RNA/counts"][:], values[:, :3])
+ np.testing.assert_array_equal(root["ADT/counts"][:], values[:, 3:])
+
+
+def test_h5adtozarr_propagates_band_write_failure(
+ tmp_path,
+):
+ from scarf.readers import H5adReader
+ from scarf.writers import H5adToZarr
+ from tests.store_probes import RecordingStore
+
+ values = _band_counts(12, 3)
+ path = _write_h5ad(tmp_path / "failing.h5ad", values)
+ reader = H5adReader(str(path), feature_name_key="feature_name")
+ store = RecordingStore(fail_on="RNA/counts/c/2/0")
+ try:
+ writer = H5adToZarr(reader, zarr_loc=store, **_SHARD_BAND_BUDGET)
+ with pytest.raises(RuntimeError, match="injected write failure"):
+ writer.dump(batch_size=5)
+ finally:
+ reader.h5.close()
+
+ assert ("set", "RNA/counts/c/2/0") in store.ops
+
+
+def test_h5adtozarr_counts_materialized_csr_as_resident_memory(tmp_path):
+ from scarf.readers import H5adReader
+ from scarf.writers import H5adToZarr
+
+ values = (
+ np.arange(400 * 400, dtype=np.uint32).reshape(400, 400) % 65_534 + 1
+ ).astype(np.uint16)
+ path = _write_h5ad(tmp_path / "resident_csc.h5ad", values, encoding="csc")
+ reader = H5adReader(str(path), feature_name_key="feature_name")
+ try:
+ reader.infer_storage_dtype()
+ conversion_peak = reader.csc_conversion_peak_bytes()
+ writer = H5adToZarr(
+ reader,
+ zarr_loc=MemoryStore(),
+ mem_budget=conversion_peak,
+ nthreads=4,
+ )
+ assert reader.materialized_csr_bytes() > 0
+ with pytest.raises(MemoryError, match="operation limit"):
+ writer.dump(batch_size=values.shape[0])
+ finally:
+ reader.h5.close()
+
+
+def test_loomtozarr(loom_reader, tmp_path):
+ from scarf.writers import LoomToZarr
+
+ fn = str(tmp_path / "sympathetic.zarr")
+ writer = LoomToZarr(loom_reader, zarr_loc=fn)
+ writer.dump()
+
+
+def test_loomtozarr_preserves_exact_counts_and_transpose(tmp_path):
+ import h5py
+
+ from scarf.readers import LoomReader
+ from scarf.writers import LoomToZarr
+
+ values = np.array([[1, 0], [0, 2], [3, 4]], dtype=np.uint16)
+ path = tmp_path / "exact.loom"
+ with h5py.File(path, mode="w") as handle:
+ handle.create_dataset("matrix", data=values.T)
+ cells = handle.create_group("col_attrs")
+ cells.create_dataset("obs_names", data=np.array([b"c1", b"c2", b"c3"]))
+ features = handle.create_group("row_attrs")
+ features.create_dataset("var_names", data=np.array([b"g1", b"g2"]))
+
+ reader = LoomReader(str(path))
+ store = MemoryStore()
+ try:
+ writer = LoomToZarr(
+ reader,
+ zarr_loc=store,
+ targetChunkBytes=8,
+ targetShardBytes=8,
+ )
+ writer.dump(batch_size=2)
+ finally:
+ reader.h5.close()
+
+ root = zarr.open_group(store=store, mode="r")
+ np.testing.assert_array_equal(root["RNA/counts"][:], values)
+ assert "countsT" not in root["RNA"]
+
+
+def test_sparsetozarr(tmp_path):
+ from scipy.sparse import csr_matrix
+
+ from scarf.writers import SparseToZarr
+
+ cols = [1, 3, 8, 2, 3, 1, 2, 8, 9]
+ rows = [0, 0, 0, 1, 1, 1, 2, 2, 2]
+ data = [1, 10, 15, 10, 20, 2, 3, 1, 5]
+ mat = (data, (rows, cols))
+ mat = csr_matrix(mat, shape=(3, 10))
+
+ fn = str(tmp_path / "dummy_sparse.zarr")
+
+ writer = SparseToZarr(
+ mat,
+ zarr_loc=fn,
+ cell_ids=[f"cell_{x}" for x in range(3)],
+ feature_ids=[f"feat_{x}" for x in range(10)],
+ )
+ writer.dump()
+ root = zarr.open_group(fn, mode="r")
+ np.testing.assert_array_equal(root["RNA/counts"][:], mat.toarray())
+ assert "countsT" not in root["RNA"]
+
+
+def test_sparsetozarr_sharded_layout(tmp_path):
+ import zarr
+ from scipy.sparse import csr_matrix
+
+ from scarf.writers import SparseToZarr
+
+ n_cells, n_feats = 5000, 200
+ rng = np.random.default_rng(0)
+ rows = rng.integers(0, n_cells, size=50_000)
+ cols = rng.integers(0, n_feats, size=50_000)
+ data = np.ones(50_000, dtype=np.uint32)
+ mat = csr_matrix((data, (rows, cols)), shape=(n_cells, n_feats))
+ fn = str(tmp_path / "dummy_sparse_sharded.zarr")
+ writer = SparseToZarr(
+ mat,
+ zarr_loc=fn,
+ cell_ids=[f"cell_{x}" for x in range(n_cells)],
+ feature_ids=[f"feat_{x}" for x in range(n_feats)],
+ )
+ writer.dump()
+ store = zarr.open_group(fn, mode="r")
+ counts = store["RNA/counts"]
+ assert counts.shape == (n_cells, n_feats)
+ assert counts.metadata.shards is not None
+ assert int(counts[...].sum()) > 0
+
+
+def test_csv_to_zarr_round_trip(tmp_path):
+ csv_path = tmp_path / "counts.csv"
+ csv_path.write_text(
+ "quality,geneA,geneB,geneC\n"
+ "10,1,0,2\n"
+ "20,0,3,0\n"
+ "30,4,5,6\n"
+ "40,7,0,8\n"
+ "50,9,10,0\n",
+ encoding="utf-8",
+ )
+ reader = CSVReader(
+ str(csv_path),
+ cell_data_cols=["quality"],
+ batch_size=2,
+ )
+ store = MemoryStore()
+ writer = CSVtoZarr(
+ reader,
+ zarr_loc=store,
+ assay_name="RNA",
+ dtype=np.dtype("uint16"),
+ )
+
+ writer.dump()
+
+ root = zarr.open_group(store=store, mode="r")
+ expected = np.array(
+ [
+ [1, 0, 2],
+ [0, 3, 0],
+ [4, 5, 6],
+ [7, 0, 8],
+ [9, 10, 0],
+ ],
+ dtype=np.uint16,
+ )
+ np.testing.assert_array_equal(root["RNA/counts"][:], expected)
+ assert "countsT" not in root["RNA"]
+ np.testing.assert_array_equal(
+ root["RNA/featureData/ids"][:],
+ np.array(["geneA", "geneB", "geneC"]),
+ )
+ np.testing.assert_array_equal(
+ root["cellData/ids"][:],
+ np.array(["cell_0", "cell_1", "cell_2", "cell_3", "cell_4"]),
+ )
+ np.testing.assert_array_equal(
+ root["cellData/quality"][:],
+ np.array([10, 20, 30, 40, 50]),
+ )
+
+
+def test_csv_to_zarr_writes_extra_cell_columns_into_workspace(tmp_path):
+ csv_path = tmp_path / "counts.csv"
+ csv_path.write_text(
+ "quality,geneA,geneB\n1,1,2\n2,3,4\n3,5,6\n",
+ encoding="utf-8",
+ )
+ reader = CSVReader(
+ str(csv_path),
+ cell_data_cols=["quality"],
+ batch_size=2,
+ )
+ store = MemoryStore()
+ writer = CSVtoZarr(
+ reader,
+ zarr_loc=store,
+ assay_name="RNA",
+ workspace="run1",
+ dtype=np.dtype("uint16"),
+ )
+
+ writer.dump()
+
+ root = zarr.open_group(store=store, mode="r")
+ assert "cellData" not in root
+ np.testing.assert_array_equal(
+ root["run1/cellData/quality"][:],
+ np.array([1, 2, 3]),
+ )
+ np.testing.assert_array_equal(
+ root["matrices/RNA/counts"][:],
+ np.array([[1, 2], [3, 4], [5, 6]], dtype=np.uint16),
+ )
+
+
+def test_subset_assay_zarr_selects_ordered_rows_and_columns():
+ store = MemoryStore()
+ root = zarr.open_group(store=store, mode="w")
+ source = root.create_array(
+ "source",
+ shape=(5, 4),
+ chunks=(2, 2),
+ dtype=np.uint16,
+ fill_value=0,
+ )
+ values = np.arange(20, dtype=np.uint16).reshape(5, 4)
+ source[:] = values
+ cells = np.array([4, 1, 3])
+ features = np.array([3, 0])
+
+ result = subset_assay_zarr(
+ store,
+ in_grp="source",
+ out_grp="selected",
+ cells_idx=cells,
+ feat_idx=features,
+ targetChunkBytes=8,
+ targetShardBytes=8,
+ )
+
+ selected = root["selected"]
+ assert result is None
+ assert selected.dtype == np.dtype(np.uint32)
+ np.testing.assert_array_equal(
+ selected[:],
+ values[np.ix_(cells, features)],
+ )
+
+
+def test_bed_to_sparse_array_bins_filters_and_drops_unknown_features(tmp_path):
+ bed_path = tmp_path / "fragments.bed"
+ bed_path.write_text(
+ "# test fragments\n"
+ "chr1\t0\t20\tcellB\t2\n"
+ "chr1\t100\t120\tcellA\t3\n"
+ "chr2\t0\t20\tcellB\t4\n"
+ "chr1\t0\t10\tcellC\t5\n"
+ "chr1\t10\t20\tcellA\t1\n"
+ "chr2\t200\t220\tcellD\t1\n",
+ encoding="utf-8",
+ )
+
+ matrix, cell_ids, feature_ids = bed_to_sparse_array(
+ str(bed_path),
+ bin_size=100,
+ chrom_sizes={"chr1": 199, "chr2": 99},
+ min_counts_per_cell=3,
+ read_chunk_size=2,
+ disable_tqdm=True,
+ )
+
+ assert cell_ids.tolist() == ["cellB", "cellA", "cellC"]
+ assert feature_ids.tolist() == ["chr1_0", "chr1_1", "chr2_0"]
+ np.testing.assert_array_equal(
+ matrix.toarray(),
+ np.array(
+ [
+ [2, 0, 4],
+ [1, 3, 0],
+ [5, 0, 0],
+ ]
+ ),
+ )
+
+
+def test_v2_fixture_read_only(datastore):
+ assert datastore.RNA.rawData.shape[0] > 0
+
+
+def test_to_h5ad(datastore, tmp_path):
+ from scarf.writers import to_h5ad
+
+ fn = str(tmp_path / "test_1K_pbmc_citeseq.h5ad")
+ to_h5ad(datastore.RNA, fn)
+
+
+def test_to_mtx(datastore, tmp_path):
+ from scarf.writers import to_mtx
+
+ fn = str(tmp_path / "test_1K_pbmc_citeseq_dir")
+ to_mtx(datastore.RNA, fn)
+
+
+def test_zarr_subset(datastore, tmp_path):
+ zarr_path = str(tmp_path / "subset.zarr")
+ writer = SubsetZarr(
+ zarr_loc=zarr_path, assays=[datastore.RNA], cell_idx=np.array([1, 10, 100, 500])
+ )
+ writer.dump()
+
+
+def test_subset_zarr_rejects_invalid_assay_inputs():
+ with pytest.raises(TypeError, match="should be a list"):
+ SubsetZarr._check_assays("RNA")
+ with pytest.raises(ValueError, match="actual assay objects"):
+ SubsetZarr._check_assays([object()])
+ with pytest.raises(ValueError, match="same numer of cells"):
+ SubsetZarr._check_assays(
+ [
+ _FakeAssay("RNA", 3),
+ _FakeAssay("ATAC", 4),
+ ]
+ )
+
+
+def test_subset_zarr_requires_cell_key_or_indices():
+ subset = object.__new__(SubsetZarr)
+ subset.assays = [_FakeAssay("RNA", 3)]
+
+ with pytest.raises(ValueError, match="cannot be None"):
+ subset._check_idx(None, None)
+
+
+@pytest.mark.parametrize(
+ ("cell_idx", "message"),
+ [
+ (np.array([0.5]), "integer type"),
+ (np.array([3]), "max value"),
+ ],
+)
+def test_subset_zarr_rejects_invalid_explicit_indices(cell_idx, message):
+ subset = object.__new__(SubsetZarr)
+ subset.assays = [_FakeAssay("RNA", 3)]
+
+ with pytest.raises(ValueError, match=message):
+ subset._check_idx(None, cell_idx)
+
+
+def test_subset_zarr_validates_cell_key():
+ subset = object.__new__(SubsetZarr)
+ subset.assays = [_FakeAssay("RNA", 3)]
+ with pytest.raises(ValueError, match="was not found"):
+ subset._check_idx("selected", None)
+
+ subset.assays = [
+ _FakeAssay("RNA", 3, {"selected": np.array([1, 0, 1])}),
+ ]
+ with pytest.raises(ValueError, match="not of boolean type"):
+ subset._check_idx("selected", None)
+
+
+def test_subset_zarr_resolves_consistent_cell_key():
+ selected = np.array([True, False, True, False])
+ subset = object.__new__(SubsetZarr)
+ subset.assays = [
+ _FakeAssay("RNA", 4, {"selected": selected}),
+ _FakeAssay("ATAC", 4, {"selected": selected.copy()}),
+ ]
+
+ np.testing.assert_array_equal(
+ subset._check_idx("selected", None),
+ np.array([0, 2]),
+ )
+
+
+def test_subset_zarr_local_path_guard(tmp_path):
+ existing = tmp_path / "out.zarr"
+ existing.mkdir()
+ subset = object.__new__(SubsetZarr)
+ subset.overFn = False
+ subset.storage_options = None
+ with pytest.raises(ValueError, match="already exists"):
+ SubsetZarr._check_files(subset, str(existing))
+
+
+def test_subset_zarr_store_skips_local_guard():
+ mem = MemoryStore()
+ subset = object.__new__(SubsetZarr)
+ subset.overFn = False
+ subset.storage_options = None
+ root = SubsetZarr._check_files(subset, mem)
+ assert isinstance(root, zarr.Group)
+
+
+def test_subset_zarr_remote_uri_skips_local_guard(monkeypatch):
+ calls = []
+
+ def fake_load_zarr(zarr_loc, mode, storage_options=None):
+ calls.append((zarr_loc, mode, storage_options))
+ return zarr.open_group(store=MemoryStore(), mode="w")
+
+ monkeypatch.setattr("scarf.writers.subset.load_zarr", fake_load_zarr)
+ subset = object.__new__(SubsetZarr)
+ subset.overFn = False
+ subset.storage_options = {"access_key_id": "key"}
+ SubsetZarr._check_files(subset, "s3://bucket/out.zarr")
+ assert calls == [("s3://bucket/out.zarr", "w", {"access_key_id": "key"})]
+
+
+def test_crtozarr_forwards_storage_options(monkeypatch):
+ captured = {}
+
+ def fake_load_zarr(zarr_loc, mode, storage_options=None):
+ captured["zarr_loc"] = zarr_loc
+ captured["mode"] = mode
+ captured["storage_options"] = storage_options
+ return zarr.open_group(store=MemoryStore(), mode="w")
+
+ monkeypatch.setattr("scarf.storage.stores.load_zarr", fake_load_zarr)
+ monkeypatch.setattr("scarf.storage.schema.create_cell_data", lambda **kwargs: None)
+ monkeypatch.setattr(
+ "scarf.storage.schema.create_zarr_count_assay",
+ lambda **kwargs: None,
+ )
+
+ class FakeCr:
+ def cell_names(self):
+ return ["c1"]
+
+ @property
+ def assayFeats(self):
+ import pandas as pd
+
+ return pd.DataFrame({"RNA": [0, 1]})
+
+ @property
+ def nCells(self):
+ return 1
+
+ def feature_ids(self, assay_name):
+ return ["f1"]
+
+ def feature_names(self, assay_name):
+ return ["f1"]
+
+ CrToZarr(
+ FakeCr(),
+ zarr_loc="s3://bucket/out.zarr",
+ storage_options={"access_key_id": "id"},
+ )
+ assert captured["storage_options"] == {"access_key_id": "id"}
+
+
+def test_h5adtozarr_applies_storage_resources_and_chunk_controls(tmp_path):
+ from scarf.readers import H5adReader
+ from scarf.storage.layout import count_array_spec
+ from scarf.writers import H5adToZarr
+
+ values = (np.arange(100 * 50, dtype=np.uint32).reshape(100, 50) % 1_000).astype(
+ np.uint16
+ )
+ path = _write_h5ad(tmp_path / "layout_controls.h5ad", values)
+ reader = H5adReader(str(path), feature_name_key="feature_name")
+ try:
+ writer = H5adToZarr(
+ reader,
+ zarr_loc=MemoryStore(),
+ mem_budget="2G",
+ nthreads=3,
+ targetChunkBytes=4_096,
+ targetShardBytes=20_480,
+ )
+ finally:
+ reader.h5.close()
+
+ expected = count_array_spec(
+ 100,
+ 50,
+ dtype=np.uint16,
+ profile="fast_local",
+ targetChunkBytes=4_096,
+ targetShardBytes=20_480,
+ )
+ counts = writer.z["RNA/counts"]
+ assert writer.resources.memoryBytes == 2 * 1024**3
+ assert writer.resources.workers == 3
+ assert counts.chunks == expected.chunks
+ assert counts.metadata.shards == expected.shards
diff --git a/tests/test_writers_contracts.py b/tests/test_writers_contracts.py
new file mode 100644
index 00000000..d2287f6d
--- /dev/null
+++ b/tests/test_writers_contracts.py
@@ -0,0 +1,360 @@
+import importlib
+import inspect
+import pickle
+import subprocess
+import sys
+from typing import get_type_hints
+
+import numpy as np
+import pytest
+import zarr
+from zarr.storage import MemoryStore
+
+import scarf.writers as writers_module
+from scarf.storage import arrays as storage_arrays
+from scarf.storage import materialize as storage_materialize
+from scarf.storage import schema as storage_schema
+from scarf.writers import (
+ CSVtoZarr,
+ CrToZarr,
+ H5adToZarr,
+ LoomToZarr,
+ MtxToZarr,
+ SeuratImportResult,
+ SeuratToZarr,
+ SparseToZarr,
+ SubsetZarr,
+)
+from tests.signature_contracts import signature_digest
+
+
+_PUBLIC_CLASS_METHODS = {
+ CrToZarr: (
+ "__init__",
+ "dump",
+ ),
+ H5adToZarr: (
+ "__init__",
+ "dump",
+ ),
+ LoomToZarr: (
+ "__init__",
+ "dump",
+ ),
+ SparseToZarr: (
+ "__init__",
+ "dump",
+ ),
+ CSVtoZarr: (
+ "__init__",
+ "dump",
+ ),
+ SubsetZarr: (
+ "__init__",
+ "dump",
+ ),
+ SeuratToZarr: (
+ "__init__",
+ "dump",
+ ),
+}
+_PUBLIC_CLASS_SIGNATURE_DIGESTS = {
+ CrToZarr: "6ddc400f3edcce221a338b60bc528a83f086409ee59b7d3e6baebfa0980f8f75",
+ H5adToZarr: "c1993f1caa8cdb973c1bee9969cf107a3f39dbf2a2269d3e0a86f6c5f0b1d94b",
+ LoomToZarr: "a840271f303b73a2187148efb1e4651a6671b4283af83c0f28251d449f5754dd",
+ SparseToZarr: "78dcbc0d0d7d48a3d971ba0654260be39fd69157a5d3cb87f4d7191f0dee450d",
+ CSVtoZarr: "30b241f964ead70d6a44e0f09be571ea88d227cd78335651b3f1a38cadf3268e",
+ SubsetZarr: "ae6eb9a73c50d32b3716ea33002993bcd005461c68e0e784aec87a66ac3061e5",
+ SeuratToZarr: "824c0169d8b921397643c54fd57b8d238720669da376e02c21d226be87adb86f",
+}
+_MODULE_FUNCTIONS = (
+ "create_zarr_count_assay",
+ "create_zarr_dataset",
+ "create_zarr_obj_array",
+ "dask_to_zarr",
+ "subset_assay_zarr",
+ "to_h5ad",
+ "to_mtx",
+ "write_renorm_subset_to_zarr",
+)
+_MODULE_SIGNATURE_DIGEST = (
+ "dbe97d2ed890118780c3618b3c3433ea56954dc3ed23b67c93b665cbae46debf"
+)
+
+
+def test_writers_facade_surface_is_stable():
+ assert writers_module.__all__ == [
+ "create_zarr_dataset",
+ "create_zarr_obj_array",
+ "create_zarr_count_assay",
+ "subset_assay_zarr",
+ "dask_to_zarr",
+ "write_renorm_subset_to_zarr",
+ "SubsetZarr",
+ "CrToZarr",
+ "MtxToZarr",
+ "H5adToZarr",
+ "LoomToZarr",
+ "SeuratImportResult",
+ "SeuratToZarr",
+ "SparseToZarr",
+ "to_h5ad",
+ "to_mtx",
+ "CSVtoZarr",
+ ]
+ expected = set(writers_module.__all__) | {
+ "bed_to_sparse_array",
+ "create_cell_data",
+ "load_count_store",
+ "load_zarr",
+ }
+ assert expected.issubset(dir(writers_module))
+ assert all(getattr(writers_module, name) is not None for name in expected)
+ assert not hasattr(writers_module, "sparse_writer")
+ assert MtxToZarr is CrToZarr
+
+
+def test_writer_class_and_method_signatures_are_stable():
+ for cls, names in _PUBLIC_CLASS_METHODS.items():
+ methods = {name: getattr(cls, name) for name in names}
+ assert signature_digest(methods) == _PUBLIC_CLASS_SIGNATURE_DIGESTS[cls]
+
+
+def test_writer_module_function_signatures_are_stable():
+ methods = {name: getattr(writers_module, name) for name in _MODULE_FUNCTIONS}
+ assert signature_digest(methods) == _MODULE_SIGNATURE_DIGEST
+
+
+def test_writer_public_metadata_remains_on_facade():
+ for cls, names in _PUBLIC_CLASS_METHODS.items():
+ assert cls.__module__ == "scarf.writers"
+ for name in names:
+ descriptor = inspect.getattr_static(cls, name)
+ if isinstance(descriptor, staticmethod):
+ method = descriptor.__func__
+ else:
+ method = descriptor
+ assert method.__module__ == "scarf.writers"
+ assert method.__qualname__.startswith(f"{cls.__name__}.")
+
+ for name in _MODULE_FUNCTIONS:
+ assert getattr(writers_module, name).__module__ == "scarf.writers"
+ assert SeuratImportResult.__module__ == "scarf.writers"
+
+
+def test_writer_static_method_contracts_are_stable():
+ for cls, names in {
+ CrToZarr: ("_prep_assay_input_ranges", "_prep_feat_index_offset"),
+ SubsetZarr: ("_check_assays",),
+ }.items():
+ for name in names:
+ assert isinstance(inspect.getattr_static(cls, name), staticmethod)
+
+
+def test_writer_storage_wrappers_remain_distinct_objects():
+ assert writers_module.create_zarr_dataset is not storage_arrays.create_zarr_dataset
+ assert (
+ writers_module.create_zarr_obj_array is not storage_arrays.create_zarr_obj_array
+ )
+ assert (
+ writers_module.create_zarr_count_assay
+ is not storage_schema.create_zarr_count_assay
+ )
+ assert writers_module.create_cell_data is not storage_schema.create_cell_data
+ assert writers_module.dask_to_zarr is not storage_materialize.dask_to_zarr
+ assert (
+ writers_module.write_renorm_subset_to_zarr
+ is not storage_materialize.write_renorm_subset_to_zarr
+ )
+
+
+def test_writer_rejects_reserved_assay_name_before_mutation():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+
+ with pytest.raises(ValueError, match="reserved for DataStore.plots"):
+ writers_module.create_zarr_count_assay(
+ root,
+ "plots",
+ None,
+ 2,
+ ["g1", "g2"],
+ ["Gene 1", "Gene 2"],
+ )
+
+ assert list(root.group_keys()) == []
+
+ with pytest.raises(ValueError, match=r"reserved for DataStore\.summary"):
+ writers_module.create_zarr_count_assay(
+ root,
+ "summary",
+ None,
+ 2,
+ ["g1", "g2"],
+ ["Gene 1", "Gene 2"],
+ )
+
+ assert list(root.group_keys()) == []
+
+ with pytest.raises(ValueError, match="artifact storage"):
+ writers_module.create_zarr_count_assay(
+ root,
+ "artifacts",
+ None,
+ 2,
+ ["g1", "g2"],
+ ["Gene 1", "Gene 2"],
+ )
+
+ assert list(root.group_keys()) == []
+
+
+def test_conversion_writers_reject_summary_before_truncating_destination():
+ import pandas as pd
+ from scipy.sparse import csr_matrix
+
+ class SummaryCellRangerReader:
+ assayFeats = pd.DataFrame({"summary": [0, 1]})
+
+ constructors = {
+ "cellranger": lambda store: CrToZarr(
+ SummaryCellRangerReader(),
+ zarr_loc=store,
+ ),
+ "csv": lambda store: CSVtoZarr(
+ object(),
+ zarr_loc=store,
+ assay_name="summary",
+ dtype=np.dtype("uint32"),
+ ),
+ "h5ad": lambda store: H5adToZarr(
+ object(),
+ zarr_loc=store,
+ assay_name="summary",
+ ),
+ "loom": lambda store: LoomToZarr(
+ object(),
+ zarr_loc=store,
+ assay_name="summary",
+ ),
+ "sparse": lambda store: SparseToZarr(
+ csr_matrix((1, 1), dtype=np.uint32),
+ zarr_loc=store,
+ cell_ids=["c1"],
+ feature_ids=["g1"],
+ assay_name="summary",
+ ),
+ }
+
+ for writer_name, construct in constructors.items():
+ store = MemoryStore()
+ root = zarr.open_group(store=store, mode="w")
+ root.create_group("sentinel")
+
+ with pytest.raises(
+ ValueError,
+ match=r"reserved for DataStore\.summary",
+ ):
+ construct(store)
+
+ preserved = zarr.open_group(store=store, mode="r")
+ assert set(preserved.group_keys()) == {"sentinel"}, writer_name
+
+
+def test_writer_type_hints_resolve_from_facade_objects():
+ for cls, names in _PUBLIC_CLASS_METHODS.items():
+ for name in names:
+ assert get_type_hints(getattr(cls, name))
+ for name in _MODULE_FUNCTIONS:
+ assert get_type_hints(getattr(writers_module, name))
+ assert get_type_hints(SeuratImportResult)
+
+
+def test_writer_facade_objects_remain_pickle_resolvable():
+ for cls in _PUBLIC_CLASS_METHODS:
+ assert pickle.loads(pickle.dumps(cls)) is cls
+ assert pickle.loads(pickle.dumps(SeuratImportResult)) is SeuratImportResult
+ for name in _MODULE_FUNCTIONS:
+ function = getattr(writers_module, name)
+ assert pickle.loads(pickle.dumps(function)) is function
+
+
+def test_writers_facade_loads_format_implementations_lazily():
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ """
+import inspect
+import sys
+
+import scarf.writers as writers
+
+writer_formats = {
+ "scarf.writers.cellranger",
+ "scarf.writers.csv",
+ "scarf.writers.h5ad",
+ "scarf.writers.loom",
+ "scarf.writers.sparse",
+ "scarf.writers.subset",
+ "scarf.writers.seurat",
+}
+reader_formats = {
+ "scarf.readers.cellranger",
+ "scarf.readers.csv",
+ "scarf.readers.h5ad",
+ "scarf.readers.loom",
+ "scarf.readers.seurat",
+}
+assert not {
+ name for name in sys.modules if name.startswith("scarf.writers.")
+}
+assert reader_formats.isdisjoint(sys.modules)
+
+writer_class = writers.CSVtoZarr
+assert writer_formats.intersection(sys.modules) == {"scarf.writers.csv"}
+assert reader_formats.intersection(sys.modules) == {"scarf.readers.csv"}
+assert "scarf.writers._store" not in sys.modules
+assert "h5py" not in sys.modules
+assert "scipy" not in sys.modules
+assert writer_class.__module__ == "scarf.writers"
+method = inspect.getattr_static(writer_class, "__init__")
+assert method.__module__ == "scarf.writers"
+assert method.__qualname__ == "CSVtoZarr.__init__"
+assert writers.CSVtoZarr is writer_class
+""",
+ ],
+ check=True,
+ )
+
+
+def test_seurat_writer_exports_load_together_lazily():
+ subprocess.run(
+ [
+ sys.executable,
+ "-c",
+ """
+import sys
+
+import scarf.writers as writers
+
+assert "scarf.writers.seurat" not in sys.modules
+writer = writers.SeuratToZarr
+assert writer.__module__ == "scarf.writers"
+assert writers.SeuratImportResult.__module__ == "scarf.writers"
+assert "scarf.writers.seurat" in sys.modules
+assert "scarf.readers.seurat" in sys.modules
+assert "scarf.datastore.datastore" not in sys.modules
+assert "scarf.writers.h5ad" not in sys.modules
+""",
+ ],
+ check=True,
+ )
+
+
+def test_writers_facade_reload_discards_cached_exports():
+ expected = writers_module.CSVtoZarr
+ writers_module.CSVtoZarr = object()
+
+ reloaded = importlib.reload(writers_module)
+
+ assert reloaded.CSVtoZarr is expected
diff --git a/tests/test_zarr_store.py b/tests/test_zarr_store.py
new file mode 100644
index 00000000..c7851226
--- /dev/null
+++ b/tests/test_zarr_store.py
@@ -0,0 +1,879 @@
+import types
+
+import numpy as np
+import pytest
+import zarr
+from scipy.sparse import csr_matrix
+from zarr.storage import MemoryStore
+
+from scarf.matrix.chunked import ChunkedArray
+from scarf.storage.arrays import create_numeric_array
+from scarf.storage.budget import ResourceBudget
+from scarf.storage.copy import (
+ copy_zarr_array,
+ copy_zarr_group_tree,
+ create_or_open_staged_normed_array,
+)
+from scarf.storage.layout import (
+ ZarrArraySpec,
+ _CODEC_MAX_BYTES,
+ DEFAULT_TARGET_CHUNK_BYTES,
+ DEFAULT_TARGET_SHARD_BYTES,
+ bounded_row_sharded_array_spec,
+ count_array_spec,
+ get_compressors,
+ normed_array_spec,
+ row_sharded_array_spec,
+)
+from scarf.storage.profiles import (
+ is_local_zarr_path,
+ is_remote_zarr_location,
+ resolve_storage_profile,
+)
+from scarf.storage.schema import create_zarr_count_assay
+from scarf.storage.sharding import (
+ accumulate_sparse_to_shards,
+ sparse_producer_peak_bytes,
+ write_dense_from_row_batches,
+ write_dense_in_shard_rows,
+ write_counts_t,
+)
+from scarf.storage.stores import (
+ is_remote_datastore,
+ make_store,
+ open_store,
+)
+from scarf.storage.types import array_metadata_shards
+from scarf.utils import load_zarr
+from tests.store_probes import RecordingStore
+
+
+def test_location_classification_is_pure():
+ memory = MemoryStore()
+ assert is_remote_zarr_location("s3://bucket/path")
+ assert is_remote_zarr_location("gs://bucket/path")
+ assert not is_remote_zarr_location("/tmp/data.zarr")
+ assert is_local_zarr_path("/tmp/data.zarr")
+ assert not is_local_zarr_path("s3://bucket/path")
+ assert not is_local_zarr_path(memory)
+ assert resolve_storage_profile("s3://bucket/path") == "cloud"
+ assert resolve_storage_profile("/tmp/data.zarr") == "fast_local"
+ assert resolve_storage_profile("s3://bucket/path", "fast_local") == "fast_local"
+
+
+def test_load_zarr_forwards_storage_options(monkeypatch):
+ captured = {}
+
+ def fake_make_store(location, storage_options=None, read_only=False):
+ captured.update(
+ location=location,
+ storageOptions=storage_options,
+ readOnly=read_only,
+ )
+ store = MemoryStore()
+ zarr.open_group(store=store, mode="w")
+ return store
+
+ monkeypatch.setattr("scarf.storage.stores.make_store", fake_make_store)
+ load_zarr(
+ "s3://bucket/path",
+ mode="r",
+ storage_options={"secret_access_key": "secret"},
+ )
+ assert captured == {
+ "location": "s3://bucket/path",
+ "storageOptions": {"secret_access_key": "secret"},
+ "readOnly": True,
+ }
+
+
+def test_store_opening_and_remote_detection(tmp_path):
+ path = str(tmp_path / "data.zarr")
+ assert make_store(path) == path
+ memory = MemoryStore()
+ assert make_store(memory) is memory
+
+ root = open_store(path, mode="w")
+ root.create_group("group")
+ assert "group" in open_store(path, mode="r")
+
+ memory_root = zarr.open_group(store=memory, mode="w")
+ values = memory_root.create_array("values", shape=(4,), dtype="i4")
+ assert not is_remote_datastore(None, memory_root)
+ assert not is_remote_datastore("", values)
+ assert is_remote_datastore("s3://bucket/path", memory_root)
+
+
+def test_remote_store_uses_obstore_without_mutating_profile(monkeypatch):
+ class FakeObstore:
+ pass
+
+ class FakeObjectStore:
+ def __init__(self, store, read_only=False):
+ self.store = store
+ self.read_only = read_only
+
+ fake_module = types.ModuleType("obstore.store")
+ fake_module.from_url = lambda url, **kwargs: FakeObstore()
+ monkeypatch.setitem(__import__("sys").modules, "obstore.store", fake_module)
+ monkeypatch.setattr("zarr.storage.ObjectStore", FakeObjectStore)
+
+ store = make_store("s3://bucket/path", read_only=True)
+ assert isinstance(store, FakeObjectStore)
+ assert store.read_only is True
+ assert resolve_storage_profile("/tmp/data.zarr") == "fast_local"
+
+
+def test_hugging_face_store_uses_fsspec(monkeypatch):
+ sentinel = object()
+ captured = {}
+
+ def from_url(url, *, storage_options=None, read_only=False):
+ captured.update(
+ url=url,
+ storageOptions=storage_options,
+ readOnly=read_only,
+ )
+ return sentinel
+
+ monkeypatch.setattr("zarr.storage.FsspecStore.from_url", from_url)
+
+ store = make_store(
+ "hf://buckets/Nygen/cytebase/demo/data.zarr",
+ storage_options={"token": False},
+ read_only=True,
+ )
+
+ assert store is sentinel
+ assert captured == {
+ "url": "hf://buckets/Nygen/cytebase/demo/data.zarr",
+ "storageOptions": {"token": False},
+ "readOnly": True,
+ }
+
+
+def test_count_plan_uses_aligned_five_chunk_shards():
+ spec = count_array_spec(
+ 250_000,
+ 45_525,
+ "uint16",
+ profile="cloud",
+ targetChunkBytes=128 * 1024**2,
+ targetShardBytes=5 * 128 * 1024**2,
+ )
+ assert spec.shards is not None
+ assert spec.shards[1] == 45_525
+ assert spec.shards[1] // spec.chunks[1] == 5
+ assert spec.chunks[0] * spec.chunks[1] * 2 <= 128 * 1024**2
+ assert spec.shards[0] * spec.shards[1] * 2 <= 5 * 128 * 1024**2
+
+
+@pytest.mark.parametrize(
+ ("n_features", "dtype"),
+ [
+ (101, "uint8"),
+ (997, "float64"),
+ (45_524, "uint16"),
+ ],
+)
+def test_count_plan_alignment_and_byte_limits_are_shape_independent(
+ n_features,
+ dtype,
+):
+ chunk_target = 1024**2
+ shard_target = 5 * chunk_target
+ spec = count_array_spec(
+ 10_000,
+ n_features,
+ dtype,
+ profile="cloud",
+ targetChunkBytes=chunk_target,
+ targetShardBytes=shard_target,
+ )
+ assert spec.shards is not None
+ assert spec.shards[1] == n_features
+ assert all(
+ shard % chunk == 0
+ for shard, chunk in zip(spec.shards, spec.chunks, strict=True)
+ )
+ itemsize = np.dtype(dtype).itemsize
+ assert np.prod(spec.chunks) * itemsize <= chunk_target
+ assert np.prod(spec.shards) * itemsize <= shard_target
+
+
+def test_count_plan_does_not_depend_on_process_resource_environment(monkeypatch):
+ kwargs = {
+ "profile": "cloud",
+ "targetChunkBytes": 1024**2,
+ "targetShardBytes": 5 * 1024**2,
+ }
+ monkeypatch.setenv("SCARF_MEM_BUDGET", "1G")
+ monkeypatch.setenv("SCARF_WORKERS", "1")
+ small_machine = count_array_spec(10_000, 997, "uint16", **kwargs)
+ monkeypatch.setenv("SCARF_MEM_BUDGET", "64G")
+ monkeypatch.setenv("SCARF_WORKERS", "32")
+ large_machine = count_array_spec(10_000, 997, "uint16", **kwargs)
+ assert small_machine == large_machine
+
+
+def test_count_plan_respects_codec_limit_and_small_dimensions():
+ spec = count_array_spec(
+ 3,
+ 7,
+ "float64",
+ profile="fast_local",
+ )
+ assert all(
+ chunk <= size for chunk, size in zip(spec.chunks, spec.shape, strict=True)
+ )
+ assert np.prod(spec.chunks) * 8 <= _CODEC_MAX_BYTES
+
+
+def test_numeric_array_adapts_codecs_to_zarr_format():
+ spec = count_array_spec(
+ 8,
+ 4,
+ "uint16",
+ profile="cloud",
+ targetChunkBytes=16,
+ targetShardBytes=32,
+ )
+ for zarr_format in (2, 3):
+ root = zarr.open_group(
+ store=MemoryStore(),
+ mode="w",
+ zarr_format=zarr_format,
+ )
+ array = create_numeric_array(root, "counts", spec)
+ values = np.arange(32, dtype=np.uint16).reshape(8, 4)
+ array[:] = values
+ np.testing.assert_array_equal(array[:], values)
+ if zarr_format == 2:
+ assert array_metadata_shards(array) is None
+ else:
+ assert array_metadata_shards(array) == spec.shards
+
+
+@pytest.mark.parametrize(
+ ("shape", "chunks", "shards"),
+ [
+ # A shard extent that is not a whole number of chunks.
+ ((10, 9), (4, 4), (10, 9)),
+ # A chunk clamped to a narrower shape leaves the shard misaligned.
+ ((10, 3), (4, 8), (8, 8)),
+ # A shard smaller than its own chunk.
+ ((10, 8), (4, 8), (2, 8)),
+ ],
+)
+def test_numeric_array_rejects_a_shard_that_is_not_whole_chunks(shape, chunks, shards):
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ spec = ZarrArraySpec(
+ shape=shape,
+ chunks=chunks,
+ shards=shards,
+ dtype="uint16",
+ compressors=get_compressors("fast_local"),
+ fillValue=0,
+ )
+ with pytest.raises(ValueError, match="whole chunks"):
+ create_numeric_array(root, "counts", spec)
+
+
+@pytest.mark.parametrize("zarr_format", [2, 3])
+def test_assay_records_the_metadata_the_stored_array_actually_has(zarr_format):
+ root = zarr.open_group(store=MemoryStore(), mode="w", zarr_format=zarr_format)
+ counts = create_zarr_count_assay(
+ root,
+ "RNA",
+ None,
+ 1_009,
+ [f"f{i}" for i in range(997)],
+ [f"g{i}" for i in range(997)],
+ profile="fast_local",
+ )
+ recorded = root["RNA"].attrs["scarf:zarr_spec"]
+ stored_shards = array_metadata_shards(counts)
+
+ assert recorded["chunks"] == list(counts.chunks)
+ assert recorded["shards"] == (
+ None if stored_shards is None else list(stored_shards)
+ )
+ assert recorded["zarr_format"] == zarr_format
+ if zarr_format == 2:
+ assert stored_shards is None
+
+
+def test_new_assay_in_zarr_v2_stays_chunk_only():
+ root = zarr.open_group(
+ store=MemoryStore(),
+ mode="w",
+ zarr_format=2,
+ )
+ counts = create_zarr_count_assay(
+ root,
+ "RNA",
+ None,
+ 8,
+ ["f0", "f1", "f2", "f3"],
+ ["g0", "g1", "g2", "g3"],
+ profile="fast_local",
+ )
+ values = np.arange(32, dtype=np.uint32).reshape(8, 4)
+ counts[:] = values
+
+ assert array_metadata_shards(counts) is None
+ assert root["RNA"].attrs["scarf:zarr_spec"]["zarr_format"] == 2
+ assert write_counts_t(counts, root["RNA"]) is None
+ np.testing.assert_array_equal(counts[:], values)
+
+
+def test_normed_plan_respects_codec_limit():
+ spec = normed_array_spec(
+ 10_000_000,
+ 2_000,
+ profile="cloud",
+ )
+ assert spec.shards is None
+ assert spec.chunks[0] * spec.chunks[1] * 4 <= _CODEC_MAX_BYTES
+
+
+def test_row_sharded_plan_uses_full_width_divisible_chunks():
+ spec = row_sharded_array_spec(
+ (10_000_000, 100),
+ np.float32,
+ profile="cloud",
+ band_rows=1_000_000,
+ )
+
+ assert spec.shards == (1_000_000, 100)
+ assert spec.chunks[1] == 100
+ assert spec.shards[0] % spec.chunks[0] == 0
+ assert np.prod(spec.chunks) * np.dtype(spec.dtype).itemsize <= 128 * 1024**2
+
+
+def test_bounded_row_sharded_plan_caps_wide_mapping_bands():
+ spec = bounded_row_sharded_array_spec(
+ (1_000_000, 4_000),
+ np.float64,
+ profile="cloud",
+ )
+
+ assert spec.shards is not None
+ assert spec.shards[0] < spec.shape[0]
+ assert spec.shards[0] % spec.chunks[0] == 0
+ itemsize = np.dtype(spec.dtype).itemsize
+ assert np.prod(spec.chunks) * itemsize <= DEFAULT_TARGET_CHUNK_BYTES
+ assert np.prod(spec.shards) * itemsize <= DEFAULT_TARGET_SHARD_BYTES
+
+
+def test_row_sharded_plan_uses_band_chunks_for_zarr_v2():
+ spec = row_sharded_array_spec(
+ (12, 3),
+ np.uint32,
+ profile="fast_local",
+ band_rows=5,
+ zarr_format=2,
+ )
+
+ assert spec.shards is None
+ assert spec.chunks == (5, 3)
+
+
+def test_row_sharded_plan_avoids_unit_chunks_for_irregular_rows():
+ spec = row_sharded_array_spec(
+ (500_009, 100),
+ np.float32,
+ profile="cloud",
+ band_rows=1_000_000,
+ )
+
+ assert spec.shards == (500_009, 100)
+ assert spec.chunks[0] > 1
+ assert spec.shards[0] % spec.chunks[0] == 0
+
+
+def test_row_sharded_plan_caps_zarr_v2_chunk_bytes():
+ spec = row_sharded_array_spec(
+ (300_000_000, 2),
+ np.uint32,
+ profile="cloud",
+ band_rows=300_000_000,
+ zarr_format=2,
+ )
+
+ assert spec.shards is None
+ assert np.prod(spec.chunks) * np.dtype(spec.dtype).itemsize <= _CODEC_MAX_BYTES
+
+
+def test_flat_connectivity_shards_align_to_one_million_cells():
+ k = 50
+ spec = row_sharded_array_spec(
+ (10_000_000 * k, 2),
+ np.uint32,
+ profile="cloud",
+ band_rows=1_000_000 * k,
+ )
+
+ assert spec.shards == (1_000_000 * k, 2)
+
+
+def test_dense_row_batches_flush_at_shard_boundaries():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ destination = root.create_array(
+ "counts",
+ shape=(7, 3),
+ chunks=(2, 3),
+ shards=(4, 3),
+ dtype=np.uint16,
+ fill_value=0,
+ )
+ expected = np.arange(21, dtype=np.int64).reshape(7, 3)
+ rows = write_dense_from_row_batches(
+ destination,
+ iter([expected[:1], expected[1:5], expected[5:]]),
+ dtype=np.uint16,
+ resources=ResourceBudget(1024**2, 4),
+ )
+ assert rows == 7
+ np.testing.assert_array_equal(destination[:], expected.astype(np.uint16))
+
+
+def test_dense_shard_summaries_are_merged_incrementally():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ destination = root.create_array(
+ "values",
+ shape=(100, 3),
+ chunks=(10, 3),
+ dtype=np.float32,
+ )
+ values = np.arange(300, dtype=np.float32).reshape(100, 3)
+ merge_count = 0
+
+ def summarize(block):
+ return (
+ block.sum(axis=0, dtype=np.float64),
+ np.square(block, dtype=np.float64).sum(axis=0),
+ )
+
+ def merge(accumulated, current):
+ nonlocal merge_count
+ merge_count += 1
+ accumulated[0][:] += current[0]
+ accumulated[1][:] += current[1]
+ return accumulated
+
+ summary = write_dense_in_shard_rows(
+ destination,
+ lambda start, end: values[start:end],
+ summarize=summarize,
+ merge_summary=merge,
+ )
+
+ assert merge_count == 9
+ np.testing.assert_allclose(summary[0], values.sum(axis=0, dtype=np.float64))
+ np.testing.assert_allclose(
+ summary[1],
+ np.square(values, dtype=np.float64).sum(axis=0),
+ )
+
+
+def test_sparse_batches_write_complete_shards():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ destination = root.create_array(
+ "counts",
+ shape=(12, 3),
+ chunks=(2, 3),
+ shards=(4, 3),
+ dtype=np.uint16,
+ fill_value=0,
+ )
+ expected = np.arange(36, dtype=np.uint16).reshape(12, 3)
+ expected[4:8] = 0
+ batches = (
+ csr_matrix(expected[start : start + 5]) for start in range(0, len(expected), 5)
+ )
+ rows = accumulate_sparse_to_shards(
+ destination,
+ batches,
+ resources=ResourceBudget(1024**2, 4),
+ producerReserveBytes=sparse_producer_peak_bytes(27, 15, 2),
+ )
+ assert rows == len(expected)
+ np.testing.assert_array_equal(destination[:], expected)
+
+
+@pytest.mark.parametrize(
+ ("dtype", "values"),
+ [
+ (np.uint8, np.array([200, 100], dtype=np.uint8)),
+ (bool, np.array([True, True], dtype=bool)),
+ ],
+)
+def test_sparse_duplicate_sum_rejects_destination_overflow(dtype, values):
+ from scipy.sparse import coo_matrix
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ destination = root.create_array(
+ "counts",
+ shape=(1, 1),
+ chunks=(1, 1),
+ shards=(1, 1),
+ dtype=dtype,
+ fill_value=0,
+ )
+ batch = coo_matrix(
+ (values, (np.array([0, 0]), np.array([0, 0]))),
+ shape=(1, 1),
+ )
+
+ with pytest.raises(OverflowError, match="destination dtype"):
+ accumulate_sparse_to_shards(
+ destination,
+ iter([batch]),
+ resources=ResourceBudget(1024**2, 1),
+ producerReserveBytes=sparse_producer_peak_bytes(
+ 2,
+ 2,
+ values.itemsize,
+ ),
+ )
+
+
+def test_empty_sparse_bands_clear_existing_values():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ destination = root.create_array(
+ "counts",
+ shape=(8, 3),
+ chunks=(2, 3),
+ shards=(4, 3),
+ dtype=np.uint16,
+ fill_value=0,
+ )
+ destination[:] = np.arange(1, 25, dtype=np.uint16).reshape(8, 3)
+
+ rows = accumulate_sparse_to_shards(
+ destination,
+ iter([csr_matrix((3, 3)), csr_matrix((5, 3))]),
+ resources=ResourceBudget(1024**2, 4),
+ producerReserveBytes=0,
+ )
+
+ assert rows == 8
+ np.testing.assert_array_equal(
+ destination[:],
+ np.zeros((8, 3), dtype=np.uint16),
+ )
+
+
+def test_complete_sparse_bands_put_each_shard_once_without_get():
+ store = RecordingStore()
+ root = zarr.open_group(store=store, mode="w")
+ destination = root.create_array(
+ "counts",
+ shape=(12, 3),
+ chunks=(2, 3),
+ shards=(4, 3),
+ dtype=np.uint16,
+ fill_value=0,
+ )
+ expected = np.arange(1, 37, dtype=np.uint16).reshape(12, 3)
+ store.reset()
+
+ accumulate_sparse_to_shards(
+ destination,
+ (csr_matrix(expected[start : start + 5]) for start in range(0, 12, 5)),
+ resources=ResourceBudget(1024**2, 4),
+ producerReserveBytes=sparse_producer_peak_bytes(27, 15, 2),
+ )
+
+ operations = store.chunk_ops("counts/c/")
+ assert not [key for kind, key in operations if kind == "get"]
+ written = [key for kind, key in operations if kind == "set"]
+ assert len(written) == len(set(written)) == 3
+
+
+def test_sparse_band_writes_respect_memory_admission():
+ store = RecordingStore(delay=0.01)
+ root = zarr.open_group(store=store, mode="w")
+ destination = root.create_array(
+ "counts",
+ shape=(12, 3),
+ chunks=(2, 3),
+ shards=(4, 3),
+ dtype=np.uint16,
+ fill_value=0,
+ )
+ expected = np.arange(1, 37, dtype=np.uint16).reshape(12, 3)
+ store.reset()
+
+ accumulate_sparse_to_shards(
+ destination,
+ (csr_matrix(expected[start : start + 5]) for start in range(0, 12, 5)),
+ resources=ResourceBudget(9_000, 4),
+ producerReserveBytes=sparse_producer_peak_bytes(27, 15, 2),
+ )
+
+ assert store.max_in_flight_for("set") == 1
+ np.testing.assert_array_equal(destination[:], expected)
+
+
+def test_sparse_writer_releases_completed_band_before_reading_more():
+ import gc
+ import weakref
+
+ from scarf.storage.sharding import (
+ SparseRowBand,
+ SparseWriteBand,
+ write_sparse_bands,
+ )
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ destination = root.create_array(
+ "counts",
+ shape=(8, 3),
+ chunks=(2, 3),
+ shards=(4, 3),
+ dtype=np.uint16,
+ fill_value=0,
+ )
+ expected = np.arange(1, 25, dtype=np.uint16).reshape(8, 3)
+
+ def writes():
+ for start in (0, 4):
+ row = np.repeat(np.arange(4, dtype=np.int64), 3)
+ row_ref = weakref.ref(row)
+ yield SparseWriteBand(
+ destination=destination,
+ band=SparseRowBand(
+ start=start,
+ end=start + 4,
+ nColumns=3,
+ row=row,
+ column=np.tile(np.arange(3, dtype=np.int64), 4),
+ data=expected[start : start + 4].ravel(),
+ dtype=np.uint16,
+ ),
+ )
+ del row
+ gc.collect()
+ assert row_ref() is None
+
+ write_sparse_bands(
+ writes(),
+ resources=ResourceBudget(7_000, 4),
+ )
+ np.testing.assert_array_equal(destination[:], expected)
+
+
+def test_sparse_writer_admits_row_chunks_and_retained_producer_bytes():
+ from scarf.storage.sharding import (
+ SparseRowBand,
+ SparseWriteBand,
+ write_sparse_bands,
+ )
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ row_chunked = root.create_array(
+ "row_chunked",
+ shape=(4, 3),
+ chunks=(2, 3),
+ shards=(4, 3),
+ dtype=np.uint16,
+ fill_value=0,
+ )
+
+ def empty_write(destination, producer_bytes=0):
+ return SparseWriteBand(
+ destination=destination,
+ band=SparseRowBand(
+ start=0,
+ end=4,
+ nColumns=3,
+ row=np.array([], dtype=np.int64),
+ column=np.array([], dtype=np.int64),
+ data=np.array([], dtype=np.uint16),
+ dtype=np.uint16,
+ ),
+ producerBytes=producer_bytes,
+ )
+
+ with pytest.raises(MemoryError):
+ write_sparse_bands(
+ iter([empty_write(row_chunked)]),
+ resources=ResourceBudget(5_000, 4),
+ )
+
+ destination = root.create_array(
+ "producer",
+ shape=(4, 3),
+ chunks=(4, 3),
+ shards=(4, 3),
+ dtype=np.uint16,
+ fill_value=0,
+ )
+ write = empty_write(destination, producer_bytes=5_000)
+
+ with pytest.raises(MemoryError):
+ write_sparse_bands(
+ iter([write]),
+ resources=ResourceBudget(7_000, 4),
+ )
+
+ pulled = False
+
+ def writes():
+ nonlocal pulled
+ pulled = True
+ yield write
+
+ with pytest.raises(MemoryError, match="Sparse producer"):
+ write_sparse_bands(
+ writes(),
+ resources=ResourceBudget(7_000, 4),
+ producerReserveBytes=7_001,
+ )
+ assert pulled is False
+
+
+def test_padded_shard_geometry_stays_readable_and_transposable():
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ counts = root.create_array(
+ "counts",
+ shape=(10, 7),
+ chunks=(3, 3),
+ shards=(6, 6),
+ dtype=np.uint16,
+ fill_value=0,
+ )
+ expected = np.arange(1, 71, dtype=np.uint16).reshape(10, 7)
+ resources = ResourceBudget(1024**2, 4)
+
+ rows = write_dense_from_row_batches(
+ counts,
+ iter([expected[:4], expected[4:]]),
+ resources=resources,
+ )
+ assert rows == 10
+ np.testing.assert_array_equal(counts[:], expected)
+ np.testing.assert_array_equal(
+ ChunkedArray(counts, resources=resources).compute(),
+ expected,
+ )
+
+ counts_t = write_counts_t(counts, root, resources=resources)
+ assert counts_t is not None
+ assert counts_t.attrs["complete"] is True
+ np.testing.assert_array_equal(counts_t[:], expected.T)
+
+
+def test_write_counts_t_works_inside_running_event_loop():
+ import asyncio
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ counts = root.create_array(
+ "counts",
+ shape=(4, 3),
+ chunks=(2, 3),
+ shards=(2, 3),
+ dtype=np.uint16,
+ fill_value=0,
+ )
+ expected = np.arange(12, dtype=np.uint16).reshape(4, 3)
+ counts[:] = expected
+
+ async def invoke():
+ return write_counts_t(
+ counts,
+ root,
+ resources=ResourceBudget(1024**2, 2),
+ )
+
+ counts_t = asyncio.run(invoke())
+ assert counts_t is not None
+ assert counts_t.attrs["complete"] is True
+ np.testing.assert_array_equal(counts_t[:], expected.T)
+
+
+def test_empty_dense_and_transpose_writes_need_no_task_memory():
+ from scipy.sparse import coo_matrix
+
+ root = zarr.open_group(store=MemoryStore(), mode="w")
+ counts = root.create_array(
+ "counts",
+ shape=(0, 3),
+ chunks=(1, 3),
+ shards=(1, 3),
+ dtype=np.uint8,
+ fill_value=0,
+ )
+ resources = ResourceBudget(1, 4)
+
+ assert write_dense_from_row_batches(counts, iter(()), resources=resources) == 0
+ counts_t = write_counts_t(counts, root, resources=resources)
+ assert counts_t is not None
+ assert counts_t.shape == (3, 0)
+ assert counts_t.attrs["complete"] is True
+
+ sparse_counts = root.create_array(
+ "sparse_counts",
+ shape=(0, 3),
+ chunks=(1, 3),
+ shards=(1, 3),
+ dtype=np.uint8,
+ fill_value=0,
+ )
+ rows = accumulate_sparse_to_shards(
+ sparse_counts,
+ iter([coo_matrix((0, 3), dtype=np.uint8)]),
+ resources=resources,
+ producerReserveBytes=1024,
+ )
+ assert rows == 0
+
+
+def test_copy_array_and_metadata_tree(tmp_path):
+ source_root = zarr.open_group(str(tmp_path / "source.zarr"), mode="w")
+ spec = normed_array_spec(64, 8, profile="fast_local")
+ source = create_numeric_array(source_root, "data", spec)
+ expected = np.random.default_rng(0).random((64, 8), dtype=np.float32)
+ source[:] = expected
+
+ target_root = zarr.open_group(str(tmp_path / "target.zarr"), mode="w")
+ target = create_numeric_array(target_root, "data", spec)
+ copy_zarr_array(
+ source,
+ target,
+ resources=ResourceBudget(1024**2, 2),
+ )
+ np.testing.assert_allclose(target[:], expected)
+
+ metadata = source_root.create_group("metadata")
+ score = metadata.create_array("score", data=np.array([1.0, 2.0, 3.0]))
+ score.attrs["display"] = {"label": "Score"}
+ copied_metadata = target_root.create_group("metadata")
+ copy_zarr_group_tree(metadata, copied_metadata)
+ np.testing.assert_array_equal(copied_metadata["score"][:], score[:])
+ assert copied_metadata["score"].attrs["display"] == {"label": "Score"}
+
+
+def test_staged_normed_array_reuses_matching_shape(tmp_path):
+ path = str(tmp_path / "cache" / "normalized.zarr")
+ first = create_or_open_staged_normed_array(path, (32, 4))
+ first[:] = 1
+ reopened = create_or_open_staged_normed_array(path, (32, 4))
+ np.testing.assert_array_equal(reopened[:], np.ones((32, 4), dtype=np.float32))
+
+
+def test_remote_store_requires_obstore(monkeypatch):
+ import builtins
+
+ real_import = builtins.__import__
+
+ def reject_obstore(name, *args, **kwargs):
+ if name in {"obstore", "obstore.store"}:
+ raise ImportError("missing obstore")
+ return real_import(name, *args, **kwargs)
+
+ monkeypatch.setattr(builtins, "__import__", reject_obstore)
+ with pytest.raises(ImportError, match="obstore"):
+ make_store("s3://bucket/path")
diff --git a/tests/visual/showcase/categorical_embedding.png b/tests/visual/showcase/categorical_embedding.png
new file mode 100644
index 00000000..00db4116
Binary files /dev/null and b/tests/visual/showcase/categorical_embedding.png differ
diff --git a/tests/visual/showcase/cell_cycle_scores.png b/tests/visual/showcase/cell_cycle_scores.png
new file mode 100644
index 00000000..2f44e80b
Binary files /dev/null and b/tests/visual/showcase/cell_cycle_scores.png differ
diff --git a/tests/visual/showcase/cluster_connectivity.png b/tests/visual/showcase/cluster_connectivity.png
new file mode 100644
index 00000000..df79040a
Binary files /dev/null and b/tests/visual/showcase/cluster_connectivity.png differ
diff --git a/tests/visual/showcase/composition.png b/tests/visual/showcase/composition.png
new file mode 100644
index 00000000..0dbc7e9a
Binary files /dev/null and b/tests/visual/showcase/composition.png differ
diff --git a/tests/visual/showcase/continuous_embedding.png b/tests/visual/showcase/continuous_embedding.png
new file mode 100644
index 00000000..1558d4f3
Binary files /dev/null and b/tests/visual/showcase/continuous_embedding.png differ
diff --git a/tests/visual/showcase/dark_embedding.png b/tests/visual/showcase/dark_embedding.png
new file mode 100644
index 00000000..df8ffea7
Binary files /dev/null and b/tests/visual/showcase/dark_embedding.png differ
diff --git a/tests/visual/showcase/grouped_dotplot.png b/tests/visual/showcase/grouped_dotplot.png
new file mode 100644
index 00000000..89aebf8c
Binary files /dev/null and b/tests/visual/showcase/grouped_dotplot.png differ
diff --git a/tests/visual/showcase/highlighted_embedding.png b/tests/visual/showcase/highlighted_embedding.png
new file mode 100644
index 00000000..758fc109
Binary files /dev/null and b/tests/visual/showcase/highlighted_embedding.png differ
diff --git a/tests/visual/showcase/marker_heatmap.png b/tests/visual/showcase/marker_heatmap.png
new file mode 100644
index 00000000..fe3401fe
Binary files /dev/null and b/tests/visual/showcase/marker_heatmap.png differ
diff --git a/tests/visual/showcase/matrix_plot.png b/tests/visual/showcase/matrix_plot.png
new file mode 100644
index 00000000..4b2614fc
Binary files /dev/null and b/tests/visual/showcase/matrix_plot.png differ
diff --git a/tests/visual/showcase/plotting_showcase_layout.npz b/tests/visual/showcase/plotting_showcase_layout.npz
new file mode 100644
index 00000000..58aa336e
Binary files /dev/null and b/tests/visual/showcase/plotting_showcase_layout.npz differ
diff --git a/tests/visual/showcase/publication_composite.png b/tests/visual/showcase/publication_composite.png
new file mode 100644
index 00000000..65ec46b1
Binary files /dev/null and b/tests/visual/showcase/publication_composite.png differ
diff --git a/tests/visual/showcase/publication_composite.svg b/tests/visual/showcase/publication_composite.svg
new file mode 100644
index 00000000..8a8c9d7e
--- /dev/null
+++ b/tests/visual/showcase/publication_composite.svg
@@ -0,0 +1,30057 @@
+
+
+
+
+
+
+ 2026-07-30T23:26:16.199522
+ image/svg+xml
+
+
+ Matplotlib v3.11.0, https://matplotlib.org/
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ A
+
+
+
+
+
+
+
+
+
+
+
+
+ B
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ C
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ CD3D
+
+
+
+
+
+
+
+
+ IL7R
+
+
+
+
+
+
+
+
+ LTB
+
+
+
+
+
+
+
+
+ MS4A1
+
+
+
+
+
+
+
+
+ CD79A
+
+
+
+
+
+
+
+
+ CD37
+
+
+
+
+
+
+
+
+ LST1
+
+
+
+
+
+
+
+
+ FCER1G
+
+
+
+
+
+
+
+
+ CTSS
+
+
+
+
+
+
+
+
+ NKG7
+
+
+
+
+
+
+
+
+ GNLY
+
+
+
+
+
+
+
+
+ GZMB
+
+
+
+
+
+
+
+
+
+
+
+
+ 1
+
+
+
+
+
+
+
+
+ 2
+
+
+
+
+
+
+
+
+ 3
+
+
+
+
+
+
+
+
+ 4
+
+
+
+
+
+
+
+
+ 5
+
+
+
+
+
+
+
+
+ 6
+
+
+
+
+
+
+
+
+ 7
+
+
+
+
+
+
+
+
+ 8
+
+
+
+
+
+
+
+
+ 9
+
+
+
+
+
+
+
+
+ 10
+
+
+
+ Leiden cluster
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ T cells
+
+
+ B cells
+
+
+ Myeloid
+
+
+ Cytotoxic
+
+
+ D
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 1
+
+
+
+
+
+
+
+
+ 2
+
+
+
+
+
+
+
+
+ 3
+
+
+
+
+
+
+
+
+ 4
+
+
+
+
+
+
+
+
+ 5
+
+
+
+
+
+
+
+
+ 6
+
+
+
+
+
+
+
+
+ 7
+
+
+
+
+
+
+
+
+ 8
+
+
+
+
+
+
+
+
+ 9
+
+
+
+
+
+
+
+
+ 10
+
+
+
+ Leiden cluster
+
+
+
+
+
+
+
+
+
+ 0.0
+
+
+
+
+
+
+
+
+ 0.2
+
+
+
+
+
+
+
+
+ 0.4
+
+
+
+
+
+
+
+
+ 0.6
+
+
+
+
+
+
+
+
+ 0.8
+
+
+
+
+
+
+
+
+ 1.0
+
+
+
+ proportion
+
+
+
+
+
+
+
+ E
+
+
+
+
+
+
+
+
+
+
+
+
+ 0
+
+
+
+
+
+
+
+
+ 1
+
+
+
+
+
+
+
+
+ 2
+
+
+
+ CD3D
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 0.0
+
+
+
+
+
+
+
+
+ 0.5
+
+
+
+
+
+
+
+
+ 1.0
+
+
+
+
+
+
+
+
+ 1.5
+
+
+
+
+
+
+
+
+ 2.0
+
+
+
+ Mean expression
+
+
+
+
+
+
+
+ Leiden clusters
+
+
+
+
+
+
+
+
+ 1
+
+
+
+
+
+
+
+
+ 2
+
+
+
+
+
+
+
+
+ 3
+
+
+
+
+
+
+
+
+ 4
+
+
+
+
+
+
+
+
+ 5
+
+
+
+
+
+
+
+
+ 6
+
+
+
+
+
+
+
+
+ 7
+
+
+
+
+
+
+
+
+ 8
+
+
+
+
+
+
+
+
+ 9
+
+
+
+
+
+
+
+
+ 10
+
+
+
+
+ Detected cells
+
+
+
+
+
+
+
+
+ 0%
+
+
+
+
+
+
+
+
+ 33%
+
+
+
+
+
+
+
+
+ 67%
+
+
+
+
+
+
+
+
+ 100%
+
+
+
+
+ Cell-cycle phase
+
+
+
+
+
+
+ G1
+
+
+
+
+
+
+ G2M
+
+
+
+
+
+
+ S
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
diff --git a/tests/visual/showcase/stacked_violin.png b/tests/visual/showcase/stacked_violin.png
new file mode 100644
index 00000000..75da1fb8
Binary files /dev/null and b/tests/visual/showcase/stacked_violin.png differ
diff --git a/uv.lock b/uv.lock
new file mode 100644
index 00000000..36ed2fa2
--- /dev/null
+++ b/uv.lock
@@ -0,0 +1,4200 @@
+version = 1
+revision = 3
+requires-python = ">=3.12"
+resolution-markers = [
+ "python_full_version >= '3.15' and sys_platform == 'win32'",
+ "python_full_version == '3.14.*' and sys_platform == 'win32'",
+ "python_full_version >= '3.15' and sys_platform == 'emscripten'",
+ "python_full_version == '3.14.*' and sys_platform == 'emscripten'",
+ "python_full_version >= '3.15' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version == '3.14.*' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+ "python_full_version < '3.14' and sys_platform == 'win32'",
+ "python_full_version < '3.14' and sys_platform == 'emscripten'",
+ "python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'",
+]
+
+[[package]]
+name = "accessible-pygments"
+version = "0.0.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pygments" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/bc/c1/bbac6a50d02774f91572938964c582fff4270eee73ab822a4aeea4d8b11b/accessible_pygments-0.0.5.tar.gz", hash = "sha256:40918d3e6a2b619ad424cb91e556bd3bd8865443d9f22f1dcdf79e33c8046872", size = 1377899, upload-time = "2024-05-10T11:23:10.216Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8d/3f/95338030883d8c8b91223b4e21744b04d11b161a3ef117295d8241f50ab4/accessible_pygments-0.0.5-py3-none-any.whl", hash = "sha256:88ae3211e68a1d0b011504b2ffc1691feafce124b845bd072ab6f9f66f34d4b7", size = 1395903, upload-time = "2024-05-10T11:23:08.421Z" },
+]
+
+[[package]]
+name = "aiohappyeyeballs"
+version = "2.7.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ce/f4/eec0465c2f67b2664688d0240b3212d5196fd89e741df67ddb81f8d35658/aiohappyeyeballs-2.7.1.tar.gz", hash = "sha256:065665c041c42a5938ed220bdcd7230f22527fbec085e1853d2402c8a3615d9d", size = 24757, upload-time = "2026-07-01T17:11:55.501Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/71/43/1947f06babed6b3f1d7f38b0c767f52df66bfb2bc10b468c4a7de9eceff2/aiohappyeyeballs-2.7.1-py3-none-any.whl", hash = "sha256:9243213661e29250eb41368e5daa826fc017156c3b8a11440826b2e3ed376472", size = 15038, upload-time = "2026-07-01T17:11:54.055Z" },
+]
+
+[[package]]
+name = "aiohttp"
+version = "3.14.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohappyeyeballs" },
+ { name = "aiosignal" },
+ { name = "attrs" },
+ { name = "frozenlist" },
+ { name = "multidict" },
+ { name = "propcache" },
+ { name = "typing-extensions", marker = "python_full_version < '3.13'" },
+ { name = "yarl" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/82/78/8ea7308cac6934de8c74a14f3d5f65d1c89287426688be79538d0e5c013d/aiohttp-3.14.1.tar.gz", hash = "sha256:307f2cff90a764d329e77040603fa032db89c5c24fdad50c4c15334cba744035", size = 7955794, upload-time = "2026-06-07T21:09:35.529Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1d/21/151624b51cd92553d95424daf4bf19f19ce9be9002d19253e7e7ce67197b/aiohttp-3.14.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:d35143e27778b4bb0fb189562d7f275bff79c62ab8e98459717c0ea617ff2480", size = 757402, upload-time = "2026-06-07T21:06:40.311Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/82/280619e0bd7bf2454987e19282616e84762255dd9c8468f62382e8c191f1/aiohttp-3.14.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:bcfb80a2cc36fba2534e5e5b5264dc7ae6fcd9bf15256da3e53d2f499e6fa29d", size = 512310, upload-time = "2026-06-07T21:06:42.207Z" },
+ { url = "https://files.pythonhosted.org/packages/55/b2/2aac325583aaa1353045f96dffa586d8a34e8322e14a7ba49cffeb103ab4/aiohttp-3.14.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:27fd7c91e51729b4f7e1577865fa6d34c9adccbc39aabe9000285b48af9f0ec2", size = 512448, upload-time = "2026-06-07T21:06:43.813Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/72/a60607cb849faa8af8a356c9329ea2eb6f395d49e82cc82ccba1fd8deb8f/aiohttp-3.14.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:64c567bf9eaf664280116a8688f63016e6b32db2505908e2bdaca1b6438142f2", size = 1766854, upload-time = "2026-06-07T21:06:45.391Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/d3/d9fe1c9ec7557ab4d0d82bebaa728c6418f0b93295ec2f4ab015f7710cc7/aiohttp-3.14.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:f5e6ff2bdbb8f4cd3fbe41f99e25bbcd58e3bf9f13d3dd31a11e7917251cc77a", size = 1740884, upload-time = "2026-06-07T21:06:47.413Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/dc/f2cecfaf9337ba3e63f181500814ff502aa3d00d9c7ec93a9d23d10a27b2/aiohttp-3.14.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2f73e01dc37122325caf079982621262f96d74823c179038a82fddfc50359264", size = 1810034, upload-time = "2026-06-07T21:06:50.165Z" },
+ { url = "https://files.pythonhosted.org/packages/66/d7/2ff65c5e65c0d7476daf7e15c032e0805e36811185b9623e3238ad6c763e/aiohttp-3.14.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:bb2c0c80d431c0d03f2c7dbf125150fedd4f0de17366a7ca33f7ccb822391842", size = 1904054, upload-time = "2026-06-07T21:06:52.035Z" },
+ { url = "https://files.pythonhosted.org/packages/20/9c/d445818389df371f56d141d881153ba23183c4735a03f7356ffb43f7757d/aiohttp-3.14.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3e6fc1a85fa7194a1a7d19f44e8609180f4a8eb5fa4c7ed8b4355f080fad235c", size = 1790278, upload-time = "2026-06-07T21:06:54.049Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/aa/bf04cb4d865fc6101c2229a294ad744973b72e513fdc5a6b791e6983d72a/aiohttp-3.14.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:686b6c0d3911ec387b444ddf5dc62fb7f7c0a7d5186a7861626496a5ab4aff95", size = 1591795, upload-time = "2026-06-07T21:06:55.911Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/b4/4dac0038960427ba832f6609dfb4ea5437d7fd80c72001b9e48f834f428b/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c6fa4dc7ad6f8109c70bb1499e589f76b0b792baf39f9b017eb92c8a81d0a199", size = 1728397, upload-time = "2026-06-07T21:06:57.777Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/f9/7cd4e8ad7aa3b75f17d56bb5498dd604a93d4e6eece822ba0568c413fff0/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:87a5eea1b2a5e21e1ebdbb33ad4165359189327e63fc4e4894693e7f821ac817", size = 1766504, upload-time = "2026-06-07T21:07:00.009Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/df/fc01d9fcad0f73fed3f3d361f1f94f975947b50dff82919f6dc2bf4316cc/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1c1421eb01d4fd608d88cc8290211d177a58532b55ad94076fb349c5bf467f0a", size = 1777806, upload-time = "2026-06-07T21:07:02.064Z" },
+ { url = "https://files.pythonhosted.org/packages/41/09/47e2d090bddcc8fb4ccb4c314aadc32d7c5d9bb55f50f6ad1c92fc15d501/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:34b257ec41345c1e8f2df68fa908a7952f5de932723871eb633ecbbff396c9a4", size = 1580707, upload-time = "2026-06-07T21:07:03.942Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/36/f1a4ce904ae0b6930cfe9afc96d0896f7ec1a620c400405d63783bb95a9c/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:de538791a80e5d862addbc183f70f0158ac9b9bb872bb147f1fd2a683691e087", size = 1798121, upload-time = "2026-06-07T21:07:05.987Z" },
+ { url = "https://files.pythonhosted.org/packages/70/0a/e0075ce9ca0279ee1d4f0c0b85f54fea02ebc83c3007651a72bece658fec/aiohttp-3.14.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6f71173be42d3241d428f760122febb748de0623f44308a6f120d0dd9ec572e3", size = 1767580, upload-time = "2026-06-07T21:07:07.873Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/61/a0c0a8f327a9c52095cdd8e312391b00d3ed64ab6c72bb5c33d8ec251cf7/aiohttp-3.14.1-cp312-cp312-win32.whl", hash = "sha256:ec8dc383ee57ea3e883477dcca3f11b65d58199f1080acaf4cd6ad9a99698be4", size = 452771, upload-time = "2026-06-07T21:07:09.669Z" },
+ { url = "https://files.pythonhosted.org/packages/df/d9/ea367c75f16ac9c6cdc8febb25e8318fa21a2b1bc8d6514d4b2d890bface/aiohttp-3.14.1-cp312-cp312-win_amd64.whl", hash = "sha256:2aa92c87868cd13674989f9ee83e5f9f7ea4237589b728048e1f0c8f6caa3271", size = 479873, upload-time = "2026-06-07T21:07:11.538Z" },
+ { url = "https://files.pythonhosted.org/packages/03/64/8d96784a7851156db8a4c6c3f6f91042fdf39fb15a4cc38c8b3c14833c45/aiohttp-3.14.1-cp312-cp312-win_arm64.whl", hash = "sha256:2c840c90759922cb5e6dda94596e079a30fb5a5ba548e7e0dc00574703940847", size = 448073, upload-time = "2026-06-07T21:07:13.637Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/97/bd137012dd97e1649162b099135a80e1fd59aaa807b2430fc448d1029aff/aiohttp-3.14.1-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:b3a03285a7f9c7b016324574a6d92a1c895da6b978cb8f1deee3ac72bc6da178", size = 506882, upload-time = "2026-06-07T21:07:15.501Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/79/e5cc690e9d922a66887ceeaca53a8ffd5a7b0be3816142b7abc433742d89/aiohttp-3.14.1-cp313-cp313-android_21_x86_64.whl", hash = "sha256:2a73f487ab8ef5abbb24b7aa9b73e98eaba9e9e031804ff2416f02eca315ccaf", size = 515270, upload-time = "2026-06-07T21:07:17.53Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/22/a73ccbf9dbd6e26dda0b24d5fd5db7da92ee3383a79f47677ffb834c5c5b/aiohttp-3.14.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:915fbb7b41b115192259f8c9ae58f3ddc444d2b5579917270211858e606a4afd", size = 485841, upload-time = "2026-06-07T21:07:19.555Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/b9/57ed8eaf596321c2ad747bd480fb1700dbd7177c60dfc9e4c187f629662e/aiohttp-3.14.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:7fb4bdf95b0561a79f259f9d28fbc109728c5ee7f27aff6391f0ca703a329abe", size = 492088, upload-time = "2026-06-07T21:07:21.581Z" },
+ { url = "https://files.pythonhosted.org/packages/78/c0/5ebe5270a7c140d7c6f79dcb018640225f14d406c149e4eec04a7d82fe71/aiohttp-3.14.1-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:1b9748363260121d2927704f5d4fc498150669ca3ae93625986ee89c8f80dcd4", size = 501564, upload-time = "2026-06-07T21:07:23.388Z" },
+ { url = "https://files.pythonhosted.org/packages/75/7f/8cdaa24fc7983865e0915153b96a9ac5bcdd3548d64c5a27d17cecccad2d/aiohttp-3.14.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:86a6dab78b0e43e2897a3bbe15745aa60dc5423ca437b7b0b164c069bf91b876", size = 751998, upload-time = "2026-06-07T21:07:25.046Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/f4/c4227aacfacc5cb0cc2d119b65301d177912a6842cd64e120c47af76064f/aiohttp-3.14.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:4dfd6e47d3c44c2279907607f73a4240b88c69eb8b90da7e2441a8045dfd21da", size = 510918, upload-time = "2026-06-07T21:07:27.28Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/01/a2d5f96cd4e74424864d30bc0a7e44d0a12dacdcfa91b5b2d1bd3dca6bf3/aiohttp-3.14.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:317acd9f8602858dc7d59679812c376c7f0b97bcbbf16e0d6237f54141d8a8a6", size = 508657, upload-time = "2026-06-07T21:07:29.252Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/ed/3c0fb5c500fdd8e7ebc10d1889c04384fffa1a9163eac1356088ca9da1b1/aiohttp-3.14.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bd869c427324e5cb15195793de951295710db28be7d818247f3097b4ab5d4b96", size = 1757907, upload-time = "2026-06-07T21:07:31.03Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/ab/d4c924d9bd5be3050c226612413ce68cb54c70d2c31b661bfc8d9a5b6a70/aiohttp-3.14.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:93b032b5ec3255473c143627d21a69ac74ae12f7f33974cb587c564d11b1066f", size = 1737565, upload-time = "2026-06-07T21:07:33.031Z" },
+ { url = "https://files.pythonhosted.org/packages/19/2a/37326821ff779084020cdc33224d20b19f42f4183a500ff92022a739eda7/aiohttp-3.14.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f234b4deb12f3ad59127e037bc57c40c21e45b45282df7d3a55a0f409f595296", size = 1799018, upload-time = "2026-06-07T21:07:35.003Z" },
+ { url = "https://files.pythonhosted.org/packages/b3/4f/6e947ba73e4ce09070761c05ed3a8ceb7c21f5e46798671d8b2aac0e4626/aiohttp-3.14.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:9af6779bfb46abf124068327abcdf9ce95c9ef8287a3e8da76ccf2d0f16c28fa", size = 1894416, upload-time = "2026-06-07T21:07:36.956Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/6e/dbf1d0625dc711fb2851f4f3c3055c39ed58bae92082d8c627dbe6013736/aiohttp-3.14.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:faccab372e66bc76d5731525e7f1143c922271725b9d38c9f97edcc66266b451", size = 1783881, upload-time = "2026-06-07T21:07:39.063Z" },
+ { url = "https://files.pythonhosted.org/packages/44/c2/5e25098a67268ed369483ae7d1a58bd0a13d03aab860d2a0e4a6eb25b046/aiohttp-3.14.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f380468b09d2a81633ee863b0ec5648d364bd17bb8ecfb8c2f387f7ac1faf42c", size = 1587572, upload-time = "2026-06-07T21:07:41.058Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/bd/cf9cee17e140f942a3de73e658a543aa8fbf35a5fc67a9d2538d52d77f0b/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:97e704dcd26271f5bda3fa07c3ce0fb76d6d3f8659f4baa1a24442cc9ba177ca", size = 1722137, upload-time = "2026-06-07T21:07:43.014Z" },
+ { url = "https://files.pythonhosted.org/packages/89/6d/5684f8c59045c96f81a18cefbc1fbbd79d25b88f1c622f2a5c5c08fcb632/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:269b76ac5394092b95bc4a098f4fc6c191c083c3bd12775d1e30e663132f6a09", size = 1755953, upload-time = "2026-06-07T21:07:45.933Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/40/35caf3170f8359760740a7d9aa0fff2e344bef98e1d1186f5a0f6dec17e6/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:5c0b3e614340c889d575451696374c9d17affd54cd607ca0babed8f8c37b9397", size = 1766479, upload-time = "2026-06-07T21:07:48.047Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/a1/b0c61e7a137f0d81de49a82023a6df73c3c16d6fefb0f8e4a93d21639002/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:5663ee9257cfa1add7253a7da3035a02f31b6600ec48261585e1800a81533080", size = 1580077, upload-time = "2026-06-07T21:07:50.069Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/41/194ea4623693009fcefebef7aef63c141754f153e9cd0d39d3b9e36c175c/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:603a2c834142172ffddc054067f5ec0ca65d57a0aa98a71bc81952573208e345", size = 1791688, upload-time = "2026-06-07T21:07:52.106Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/45/4de841f005cfe1fd63e2a2fe011262c515e2a62aa6994b15947e7d717ac9/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:cb21957bb8aca671c1765e32f58164cf0c50e6bf41c0bbbd16da20732ecaf588", size = 1761094, upload-time = "2026-06-07T21:07:54.113Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/ae/dbce10533d3896d544d5053939ed75b7dc31a1b0973d959b1b5ae21028d6/aiohttp-3.14.1-cp313-cp313-win32.whl", hash = "sha256:e509a55f681e6158c20f70f102f9cf61fb20fbc382272bc6d94b7343f2582780", size = 452662, upload-time = "2026-06-07T21:07:56.06Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/d9/0bf1a19362c32f06229da5e7ddfcec91f93474d6307f7a2d3135e9c674dc/aiohttp-3.14.1-cp313-cp313-win_amd64.whl", hash = "sha256:1ac8531b638959718e18c2207fbfe297819875da46a740b29dfa29beba64355a", size = 479748, upload-time = "2026-06-07T21:07:58.319Z" },
+ { url = "https://files.pythonhosted.org/packages/22/0a/62e7232dc9484fbec112ceb32efb6a624cc7994ec6e2b019286f17c4e8f2/aiohttp-3.14.1-cp313-cp313-win_arm64.whl", hash = "sha256:250d14af67f6b6a1a4a811049b1afa69d61d617fca6bf33149b3ab1a6dbcf7b8", size = 447723, upload-time = "2026-06-07T21:08:00.154Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/a1/5fafa04e1ca91ddb47608699d60649c1c6db3cf41c99e78fc4056f9513db/aiohttp-3.14.1-cp314-cp314-android_24_arm64_v8a.whl", hash = "sha256:7c106c26852ca1c2047c6b80384f17100b4e439af276f21ef3d4e2f450ae7e15", size = 508531, upload-time = "2026-06-07T21:08:02.093Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/2e/bfa02f699d87ffc86d5959270b28f1cb410add3ccaced8ed2e0b8a5238fc/aiohttp-3.14.1-cp314-cp314-android_24_x86_64.whl", hash = "sha256:20205f7f5ade7aaec9f4b500549bbc071b046453aed72f9c06dcab87896a83e8", size = 514718, upload-time = "2026-06-07T21:08:04.476Z" },
+ { url = "https://files.pythonhosted.org/packages/85/a5/9594ad6289eebbc97d167c44213d557807f90e59115caad24de21ad2c3b1/aiohttp-3.14.1-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:62a759436b29e677181a9e76bab8b8f689a29cb9c535f45f7c48c9c830d3f8c3", size = 487918, upload-time = "2026-06-07T21:08:06.377Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/61/16a32c36c3c49edec122a3dc811f2057df2f94d3b14aa107c8017d981618/aiohttp-3.14.1-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:2964cbf553df4d7a57348da44d961d871895fc1ee4e8c322b2a95612c7b17fba", size = 494014, upload-time = "2026-06-07T21:08:08.263Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/89/3ebcf96ed99c05bec9c434aaac6963fd3cbab4a786ae739908a144d9ce44/aiohttp-3.14.1-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:237651caadc3a59badd39319c54642b5299e9cc98a3a194310e55d5bb9f5e397", size = 502398, upload-time = "2026-06-07T21:08:10.244Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/3d/b74870a0c2d40c355928cd5b96c7a11fa821b8a40fc41365e64479b151fb/aiohttp-3.14.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:896e12dfdbbab9d8f7e16d2b28c6769a60126fa92095d1ebf9473d02593a2448", size = 758018, upload-time = "2026-06-07T21:08:12.447Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/66/f42f5c984d99e49c6cff5f26f590750f2e2f7ef1fcfb99966ab5be1b632e/aiohttp-3.14.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d03f281ed22579314ba00821ce20115a7c0ac430660b4cc05704a3f818b3e004", size = 512462, upload-time = "2026-06-07T21:08:14.624Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/a7/248e1aebe0c7810b0271e021a0f2a5eb6e78a051885b3c9df49f42a5802d/aiohttp-3.14.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:07eabb979d236335fed927e137a928c9adfb7df3b9ec7aa31726f133a62be983", size = 512824, upload-time = "2026-06-07T21:08:16.572Z" },
+ { url = "https://files.pythonhosted.org/packages/26/97/2aa0e5ba0727dc3bd5aaebb7ccbc510f7dfb7fb961ec87497cd496635ab1/aiohttp-3.14.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4fe1f1087cbadb280b5e1bb054a4f00d1423c74d6626c5e48400d871d34ecefe", size = 1749898, upload-time = "2026-06-07T21:08:18.635Z" },
+ { url = "https://files.pythonhosted.org/packages/00/8d/e97f6c96c891d457c8479d92a514ba194d0412f981d72c70341ee18488ed/aiohttp-3.14.1-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:367a9314fdc79dab0fac96e216cb41dd73c85bdca85306ce8999118ba7e0f333", size = 1710114, upload-time = "2026-06-07T21:08:20.892Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/e6/aa8d7e863048c8fceb5cd6ce74017311cec3ead07847387e12265fb4444e/aiohttp-3.14.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a24f677ebe83749039e7bdf862ff0bbb16818ae4193d4ef96505e269375bcce0", size = 1802541, upload-time = "2026-06-07T21:08:23.044Z" },
+ { url = "https://files.pythonhosted.org/packages/83/a8/72193137de57fda4ebfae4563182d082c8856e3b6e9871d0b46f028fb369/aiohttp-3.14.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c83afe0ba876be7e943d2e0ba645809ad441575d2840c895c21ee5de93b9377a", size = 1875776, upload-time = "2026-06-07T21:08:25.288Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/18/938441025db6769a3464596b2410af3afde0b21eb2f204c6f766f68af4bd/aiohttp-3.14.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:634e385930fb6d2d479cf3aa66515955863b77a5e3c2b5894ca259a25b308602", size = 1760329, upload-time = "2026-06-07T21:08:27.363Z" },
+ { url = "https://files.pythonhosted.org/packages/60/29/bf2496b4065e76e09fe48015aaffe5ce161d8f089b06ac6982070f653076/aiohttp-3.14.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:eeea07c4397bbc57719c4eed8f9c284874d4f175f9b6d57f7a1546b976d455ca", size = 1587293, upload-time = "2026-06-07T21:08:29.805Z" },
+ { url = "https://files.pythonhosted.org/packages/49/a2/2136674d52123b1354bd05dd5753c318db47dc0c927cc70b27bab3755456/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:335c0cc3e3545ce98dcb9cfcb836f40c3411f43fa03dab757597d80c89af8a35", size = 1714756, upload-time = "2026-06-07T21:08:32.094Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/b9/e5fd2e6f915503081c0f9b1e8540947037929c70c191da2e4d54b31a21a1/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:ae6be797afdef264e8a84864a85b196ca06045586481b3df8a967322fd2fa844", size = 1721052, upload-time = "2026-06-07T21:08:34.167Z" },
+ { url = "https://files.pythonhosted.org/packages/63/5a/2833e324a2263e104e31e2e91bc5bbee81bc499afd32203faee048a883f0/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:8560b4d712474335d08907db7973f71912d3a9a8f1dee992ec06b5d2fe359496", size = 1766888, upload-time = "2026-06-07T21:08:36.95Z" },
+ { url = "https://files.pythonhosted.org/packages/57/fa/dea6511870913162f3b2e8c42a7614eb203a4540b8c2da43e0bfb0548f3c/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:2b7edd08e0a5deb1e8564a2fcd8f4561014a3f05252334671bbf55ddd47db0e5", size = 1581679, upload-time = "2026-06-07T21:08:39.292Z" },
+ { url = "https://files.pythonhosted.org/packages/14/bd/3cf0d55e71784b33534e9710a67d382d900598b4787fbce6cc7317f8c42a/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:b6ff7fcee63287ae57b5df3e4f5957ce032122802509246dec1a5bcc55904c95", size = 1782021, upload-time = "2026-06-07T21:08:41.407Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/af/14bb5843eccbe234f4dfb78ab73e549d99727247e62ae5d62cbd22eaf5b0/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6ffbb2f4ec1ceaff7e07d43922954da26b223d188bf30658e561b98e23089444", size = 1742574, upload-time = "2026-06-07T21:08:43.795Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/1e/fbeb7af9210a67ac0f9c9bec0f8f4568497924e33137a3d5b48e1cf85f3f/aiohttp-3.14.1-cp314-cp314-win32.whl", hash = "sha256:a9875b46d910cff3ea2f5962f9d266b465459fe634e22556ab9bd6fc1192eea0", size = 457773, upload-time = "2026-06-07T21:08:46.168Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/2b/13e8d741a9ec5db7d900c060554cf8352ab85e44e2a4469ebb9d377bda17/aiohttp-3.14.1-cp314-cp314-win_amd64.whl", hash = "sha256:af8b4b81a960eeaf1234971ac3cd0ba5901f3cd42eae42a46b4d089a8b492719", size = 485001, upload-time = "2026-06-07T21:08:48.401Z" },
+ { url = "https://files.pythonhosted.org/packages/df/30/491acfa2c4d6c3ff59c49a14fc1b50be3241e25bbb0c84c09e2da4d11395/aiohttp-3.14.1-cp314-cp314-win_arm64.whl", hash = "sha256:cf4491381b1b57425c315a56a439251b1bdac07b2275f19a8c44bc57744532ec", size = 453809, upload-time = "2026-06-07T21:08:50.7Z" },
+ { url = "https://files.pythonhosted.org/packages/34/e3/19dbe1a1f4cc6230eb9e314de7fe68053b0992f9302b27d12141a0b5db53/aiohttp-3.14.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:819c054312f1af92947e6a55883d1b66feefab11531a7fc45e0fb9b63880b5c2", size = 793320, upload-time = "2026-06-07T21:08:52.775Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/20/1b7182219ba1b108430d6e4dc53d25ae02dcfcf5a045b33af4e8c5167527/aiohttp-3.14.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:10ee9c1753a8f706345b22496c79fbddb5be0599e0823f3738b1534058e25340", size = 529077, upload-time = "2026-06-07T21:08:55Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/c8/14ce60ec31a2e5f5274bb17d383a6f7a3aabca31ac04eee05585bbadab16/aiohttp-3.14.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1601cc37baf5750ccacae618ec2daf020769581695550e3b654a911f859c563d", size = 532476, upload-time = "2026-06-07T21:08:57.176Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/02/9ac85e081e53da2e061b02fa7758fe0a12d17b8ce2d1f5e6c7cb76730328/aiohttp-3.14.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4d6e0ac9da31c9c04c84e1c0182ad8d6df35965a85cae29cd71d089621b3ae94", size = 1922347, upload-time = "2026-06-07T21:08:59.563Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/3e/d3ba07a0ab38b5389e10bec4362d21e10a4f667cba2d79ba30837b3a5059/aiohttp-3.14.1-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:9e8f2d660c350b3d0e259c7a7e3d9b7fc8b41210cbcc3d4a7076ff0a5e5c2fdc", size = 1786465, upload-time = "2026-06-07T21:09:01.909Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/cb/e2ee978a00cfb2df829704a69528b18154eba5939f45bc1efa8f33aee4c5/aiohttp-3.14.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4691802dda97be727f79d86818acaad7eb8e9252626a1d6b519fedbb92d5e251", size = 1909423, upload-time = "2026-06-07T21:09:04.357Z" },
+ { url = "https://files.pythonhosted.org/packages/73/5d/1430334858b1022b58ae50399a918f0bd6fe8fa7fa183598d657ff61e040/aiohttp-3.14.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c389c482a7e9b9dc3ee2701ac46c4125297a3818875b9c305ddb603c04828fd1", size = 2001906, upload-time = "2026-06-07T21:09:06.722Z" },
+ { url = "https://files.pythonhosted.org/packages/66/4e/560c7472d3d198a23aa5c8b19a5115bf6a9b77b7d3e4bb363da320430ad2/aiohttp-3.14.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fc0cacab7ba4e56f0f81c82a98c09bed2f39c940107b03a34b168bdf7597edd3", size = 1877095, upload-time = "2026-06-07T21:09:09.011Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/f1/4745806578d447db4a784a8591e2dae3afdfc2bcb96f8f81271b13df6543/aiohttp-3.14.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:979ed4717f59b8bb12e3963378fa285d93d367e15bcd66c721311826d3c44a6c", size = 1676222, upload-time = "2026-06-07T21:09:11.461Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/c9/48255813cca749a229ef0ab476004ec623728ad79a9c0840616f6c076325/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:38e1e7daaea81df51c952e18483f323d878499a1e2bfe564790e0f9701d6f203", size = 1842922, upload-time = "2026-06-07T21:09:14.118Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/c0/bbd054e2bee909f529523a5af3891052606af5143c09f5f183ec3b234676/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:4132e72c608fe9fecb8f409113567605915b83e9bdd3ea56538d2f9cd35002f1", size = 1825035, upload-time = "2026-06-07T21:09:16.447Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/ae/90395d4376deceb74e09ec26b6adf7d2015a6f8802d6d84446af860fef04/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:eefd9cc9b6d4a2db5f00a26bc3e4f9acf71926a6ec557cd56c9c6f27c290b665", size = 1849512, upload-time = "2026-06-07T21:09:18.742Z" },
+ { url = "https://files.pythonhosted.org/packages/93/bd/fb25f3049957553d4ce0ba6ae480aa2f592a6985497fca590837d16c1be0/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:b165790117eea512d7f3fb22f1f6dad3d55a7189571993eb015591c1401276d1", size = 1668571, upload-time = "2026-06-07T21:09:21.458Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/22/7f73303d64dd567ff3addca90b556690ed1233a47b8f55d242fb90af3681/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:ed09c7eb1c391271c2ed0314a51903e72a3acb653d5ccfc264cdf3ef11f8269d", size = 1881159, upload-time = "2026-06-07T21:09:23.813Z" },
+ { url = "https://files.pythonhosted.org/packages/44/be/0474c5a8b5640e1e4aa1923430a91f4151be82e511373fe764189b89aef5/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:99abd37084b82f5830c635fddd0b4993b9742a66eb746dacf433c8590e8f9e3c", size = 1841409, upload-time = "2026-06-07T21:09:26.207Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/3c/bb4a7cba26956cb3da4553cc2056cf67be5b5ff6e6d8fa4fbdff73bfb7ae/aiohttp-3.14.1-cp314-cp314t-win32.whl", hash = "sha256:47ddf841cdecc810749921d25606dee45857d12d2ad5ddb7b5bd7eab12e4b365", size = 494166, upload-time = "2026-06-07T21:09:28.505Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/84/ec80c2c1f66a952555a9f86df6b33af65108a6febfa0471b69013a12f807/aiohttp-3.14.1-cp314-cp314t-win_amd64.whl", hash = "sha256:5e78b522b7a6e27e0b25d19b247b75039ac4c94f99823e3c9e53ae1603a9f7e9", size = 530255, upload-time = "2026-06-07T21:09:30.843Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/71/6e22be134a4061ada85a92951b842f2657f17d926b727f3f94c56ae963d6/aiohttp-3.14.1-cp314-cp314t-win_arm64.whl", hash = "sha256:90d53f1609c29ccc2193945ef732428382a28f78d0456ae4d3daf0d48b74f0f6", size = 469640, upload-time = "2026-06-07T21:09:33.028Z" },
+]
+
+[[package]]
+name = "aiosignal"
+version = "1.4.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "frozenlist" },
+ { name = "typing-extensions", marker = "python_full_version < '3.13'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/61/62/06741b579156360248d1ec624842ad0edf697050bbaf7c3e46394e106ad1/aiosignal-1.4.0.tar.gz", hash = "sha256:f47eecd9468083c2029cc99945502cb7708b082c232f9aca65da147157b251c7", size = 25007, upload-time = "2025-07-03T22:54:43.528Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fb/76/641ae371508676492379f16e2fa48f4e2c11741bd63c48be4b12a6b09cba/aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e", size = 7490, upload-time = "2025-07-03T22:54:42.156Z" },
+]
+
+[[package]]
+name = "alabaster"
+version = "1.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a6/f8/d9c74d0daf3f742840fd818d69cfae176fa332022fd44e3469487d5a9420/alabaster-1.0.0.tar.gz", hash = "sha256:c00dca57bca26fa62a6d7d0a9fcce65f3e026e9bfe33e9c538fd3fbb2144fd9e", size = 24210, upload-time = "2024-07-26T18:15:03.762Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7e/b3/6b4067be973ae96ba0d615946e314c5ae35f9f993eca561b356540bb0c2b/alabaster-1.0.0-py3-none-any.whl", hash = "sha256:fc6786402dc3fcb2de3cabd5fe455a2db534b371124f1f21de8731783dec828b", size = 13929, upload-time = "2024-07-26T18:15:02.05Z" },
+]
+
+[[package]]
+name = "anndata"
+version = "0.12.18"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "array-api-compat" },
+ { name = "h5py" },
+ { name = "legacy-api-wrap" },
+ { name = "natsort" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "pandas" },
+ { name = "scipy" },
+ { name = "scverse-misc" },
+ { name = "zarr" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/0f/02/e74c34c6668e04e3092f0224620f9eb4e3abfd8fca7bed2e790086a21e8e/anndata-0.12.18.tar.gz", hash = "sha256:8c14e1814a4119ae92cdfb5d2d1365e0044829e9eaed8f03522cb0eaddd0ccc5", size = 2257885, upload-time = "2026-06-23T09:26:36.702Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a2/d3/e0ce94d247ccc5240c6433f44155c3acdc54e812418f7d88c596bde845a3/anndata-0.12.18-py3-none-any.whl", hash = "sha256:23b97a2f449fa2b900f9bbbe1712d36d6f9d6d680feb21144e1c72802af1f014", size = 176113, upload-time = "2026-06-23T09:26:34.835Z" },
+]
+
+[[package]]
+name = "annotated-types"
+version = "0.7.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ee/67/531ea369ba64dcff5ec9c3402f9f51bf748cec26dde048a2f973a4eea7f5/annotated_types-0.7.0.tar.gz", hash = "sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89", size = 16081, upload-time = "2024-05-20T21:33:25.928Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl", hash = "sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", size = 13643, upload-time = "2024-05-20T21:33:24.1Z" },
+]
+
+[[package]]
+name = "anyio"
+version = "4.14.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "idna" },
+ { name = "typing-extensions", marker = "python_full_version < '3.13'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3b/72/5562aabb8dd7181e8e860622a38bea08d17842b99ecd4c91f84ac95251b0/anyio-4.14.1.tar.gz", hash = "sha256:8d648a3544c1a700e3ff78615cd679e4c5c3f149904287e73687b2596963629e", size = 254831, upload-time = "2026-06-24T20:56:06.017Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b0/7b/90df4a0a816d98d6ea26f559d87836d494a2cf1fcf063be67df50a7bcc30/anyio-4.14.1-py3-none-any.whl", hash = "sha256:4e5533c5b8ff0a24f5d7a176cbe6877129cd183893f66b537f8f227d10527d72", size = 124875, upload-time = "2026-06-24T20:56:04.413Z" },
+]
+
+[[package]]
+name = "appnope"
+version = "0.1.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/35/5d/752690df9ef5b76e169e68d6a129fa6d08a7100ca7f754c89495db3c6019/appnope-0.1.4.tar.gz", hash = "sha256:1de3860566df9caf38f01f86f65e0e13e379af54f9e4bee1e66b48f2efffd1ee", size = 4170, upload-time = "2024-02-06T09:43:11.258Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/81/29/5ecc3a15d5a33e31b26c11426c45c501e439cb865d0bff96315d86443b78/appnope-0.1.4-py2.py3-none-any.whl", hash = "sha256:502575ee11cd7a28c0205f379b525beefebab9d161b7c964670864014ed7213c", size = 4321, upload-time = "2024-02-06T09:43:09.663Z" },
+]
+
+[[package]]
+name = "array-api-compat"
+version = "1.15.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/86/12/65d297d3e425cb7fe5247b7ec14b4ed83539e9b5e89b65c15ea7ee274182/array_api_compat-1.15.0.tar.gz", hash = "sha256:53c5f922491bf15f62847afafc4e39eedfae57d218988fefb8cce39c2a9b3dea", size = 129305, upload-time = "2026-06-07T20:53:24.964Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/86/16/1a8fd2b19544b84575cf84ef7aa3ad4c173b756d5f087c91f85d1b295777/array_api_compat-1.15.0-py3-none-any.whl", hash = "sha256:7b1b9c53269061403fd5f45a8de349f16e7887653328bfa0c5f2d45299ff0a8e", size = 79113, upload-time = "2026-06-07T20:53:23.621Z" },
+]
+
+[[package]]
+name = "ast-serialize"
+version = "0.5.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/81/9d/09e27731bd5864a9ce04e3244074e674bb8936bf62b45e0357248717adac/ast_serialize-0.5.0.tar.gz", hash = "sha256:5880091bfe6f4f986f22866375c2e884843e7a0b6343ae41aeea659613d879b6", size = 61157, upload-time = "2026-05-17T17:48:29.429Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c0/9a/13dde51ba9e15f8b97957ab7cb0120d0e381524d651c6bd630b9c359227f/ast_serialize-0.5.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:8f5c14f169eb0972c0c21bada5358b23d6047c76583b005234f865b11f1fa00a", size = 1183520, upload-time = "2026-05-17T17:47:30.831Z" },
+ { url = "https://files.pythonhosted.org/packages/37/de/5a7f0a9fe68944f536632a5af84676739c7d2582be42deb082634bf3a754/ast_serialize-0.5.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7d1a2de9de5be04652f0ed60738356ef94f66db37924a9499fffe98dc491aa0b", size = 1175779, upload-time = "2026-05-17T17:47:32.551Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/81/0bb853e76e4f6e9a1855d569003c59e19ffac45f7079d91505d1bb212f92/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:be5173fb66f9b49026d9d5a2ff0fc7c7009077107c0eb285b2d60fdf1fe10bd1", size = 1233750, upload-time = "2026-05-17T17:47:34.731Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/d3/4cf705beeccc08754d0bbda99aefff26110e209b9a07ac8a6b60eec48531/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f8015cd071ac1339924ee2b8098c93e00e155f30a16f40ec9816fcf84f4753f6", size = 1235942, upload-time = "2026-05-17T17:47:36.287Z" },
+ { url = "https://files.pythonhosted.org/packages/26/c8/ee097e437ea27dd2b8b227865c875492b585650a5802a22d82b304c8201b/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5499e8797edff2a9186aa313ed382c6b422e798e9332d9953badcee6e69a88f2", size = 1442517, upload-time = "2026-05-17T17:47:38.17Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/bd/68063442838f1ba68ec72b5436430bc75b3bb17a1a3c3063f09b0c05ae2b/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:6848f2a093fb5548751a9a09bff8fcd229e2bbeb0e3331f391b6ae6d26cd9903", size = 1254081, upload-time = "2026-05-17T17:47:39.826Z" },
+ { url = "https://files.pythonhosted.org/packages/50/e2/1e520793bc6a4e4524a6ab022391e827825eaa0c3811828bfdc6852eca26/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:832d4c998e0b091fd60a6d6bceee535483c4d490de9ba85003af835225719261", size = 1259910, upload-time = "2026-05-17T17:47:41.369Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/e1/49b60f467979979cfe6913b43948ff25bca971ad0591d181812f163a988e/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:16db7c62ec0b8efe1d7afd283a388d8f74f2605d56032e5a37747d2de8dba027", size = 1250678, upload-time = "2026-05-17T17:47:43.702Z" },
+ { url = "https://files.pythonhosted.org/packages/74/ba/66ab9555de6275677566f6574e5ef6c29cb185ea866f643bc06f8280a8ee/ast_serialize-0.5.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:baf5eb061eb5bccade4128ad42da33787d72f6013809cd1b590376ece8b3c937", size = 1301603, upload-time = "2026-05-17T17:47:46.256Z" },
+ { url = "https://files.pythonhosted.org/packages/66/42/6aca9b9abc710014b2be9059689e5dd1679339e78f567ffb4d255a9e2050/ast_serialize-0.5.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:104e4a35bd7c124173c41760ef9aaea17ddb3f86c65cb643671d59afbe3ee94c", size = 1410332, upload-time = "2026-05-17T17:47:47.899Z" },
+ { url = "https://files.pythonhosted.org/packages/47/68/2f76594432a22581ecf878b5e75a9b8601c24b2241cf0bbeb1e21fcf370c/ast_serialize-0.5.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:36be371028fc1675acb38a331bde160dbab7ff907fdf00b67eb6911aa106951b", size = 1509979, upload-time = "2026-05-17T17:47:50.942Z" },
+ { url = "https://files.pythonhosted.org/packages/40/ac/a93c9b58292653f6c595752f677a08e608f903b710594909e9231a389b3b/ast_serialize-0.5.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:061ee58bdb52341c8201a6df41182a977736bae3b7ded87ca7176ca25a8a47ab", size = 1505002, upload-time = "2026-05-17T17:47:54.093Z" },
+ { url = "https://files.pythonhosted.org/packages/14/2e/b278f68c497ee2f1d1576cbbef8db5281cd4a5f2db040537592ac9c8862e/ast_serialize-0.5.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:b15219e9cdc9f53f6f4cb51c009203507228226148c05c5e8fe451c28b435eb3", size = 1456231, upload-time = "2026-05-17T17:47:56.311Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/43/419be1c566a4c504cd8fd60ce2f84e790f295495c0f327cfaeadf3d51012/ast_serialize-0.5.0-cp314-cp314t-win32.whl", hash = "sha256:842d1c004bb466c7df036f95fabef789570541922b10976b12f5592a69cf0b38", size = 1058668, upload-time = "2026-05-17T17:47:58.305Z" },
+ { url = "https://files.pythonhosted.org/packages/03/6f/c9d4d549295ed05111aeb8853232d1afd9d0a179fddb01eeffbb3a4a6842/ast_serialize-0.5.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b0c06d760909b095cc466356dfccd05a1c7233a6ca191c020dca2c6a6f16c24c", size = 1101075, upload-time = "2026-05-17T17:48:00.35Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/8e/d00c5ab30c58222e07d62956fca86c59d91b9ad32997e633c38b526623a3/ast_serialize-0.5.0-cp314-cp314t-win_arm64.whl", hash = "sha256:787baedb0262cc49e8ce37cc15c00ae818e46a165a3b36f5e21ed174998104cb", size = 1075347, upload-time = "2026-05-17T17:48:01.753Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/9e/dc2530acb3a60dc6e46d65abf27d1d9f86721694757906a148d90a6860de/ast_serialize-0.5.0-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:0668aa9459cfa8c9c49ddd2163ebcf43088ba045ef7492af6fe22e0098303101", size = 1191380, upload-time = "2026-05-17T17:48:03.738Z" },
+ { url = "https://files.pythonhosted.org/packages/26/0a/bd3d18a582f273d6c843d16bb9e22e9e16365ff7991e92f18f798e9f1224/ast_serialize-0.5.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:bf683d6363edf2b39eed6b6d4fe22d34b6203867a67e27134d9e2a2680c4bc4a", size = 1183879, upload-time = "2026-05-17T17:48:05.463Z" },
+ { url = "https://files.pythonhosted.org/packages/40/ae/1f919100f8620887af58fcc381c61a1f218cdf89c6e155f87b213e61010a/ast_serialize-0.5.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9cc22cf0c9be65e71cf88fda130af60d61eb4a79370ad4cfe7900d48a4aa2211", size = 1244529, upload-time = "2026-05-17T17:48:07.008Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/ca/6376559dcce707cdbc1d0d9a13c8d3baaaa501e949ce0ebdc4230cd881aa/ast_serialize-0.5.0-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f66173891548c9f2726bf27957b41cabce12fa679dc6da505ddbde4d4b3b31cf", size = 1240560, upload-time = "2026-05-17T17:48:08.46Z" },
+ { url = "https://files.pythonhosted.org/packages/35/b2/a620e206b5aeb7efbf2710336df57d457cffbb3991076bbcc1147ef9abd4/ast_serialize-0.5.0-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e42d729ef2be96a14efbad355093284739e3670ece3e534f82cc8832790911d9", size = 1451172, upload-time = "2026-05-17T17:48:09.922Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/e0/4ad5c04c24a40481b2935ce9a0ccdb6023dc8b667167d06ae530cc3512f2/ast_serialize-0.5.0-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b725026bafa801dbd7310eb13a75f0a2e370e7e51b2cb225f9d21fcfadf919ee", size = 1265072, upload-time = "2026-05-17T17:48:11.469Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/71/4d1d479aa56d0101c40e17720c3d6ac2af7269ea0487a80b18e7bfd1a5b7/ast_serialize-0.5.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b54f60c1d78767a53b67eaa663f0dfac3afe606aa07f1301572f588b73d64809", size = 1270488, upload-time = "2026-05-17T17:48:13.575Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/4f/0de1bbe06f6edef9fde4ed12ca8e7b3ec7e6e2bd4e672c5af487f7957665/ast_serialize-0.5.0-cp39-abi3-manylinux_2_31_riscv64.whl", hash = "sha256:27d51654fc240a1e87e742d353d98eb45b75f62f129086b3596ab53df2ac2a43", size = 1260702, upload-time = "2026-05-17T17:48:15.141Z" },
+ { url = "https://files.pythonhosted.org/packages/75/61/e00872439cfdddcc3c1b6cdaa6e5d904ba8e26a18807c67c4e14409d0ca8/ast_serialize-0.5.0-cp39-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:2782c36237c46dd1674542f2109740ea5ea485a169bf1431939ada0434e17934", size = 1311182, upload-time = "2026-05-17T17:48:16.779Z" },
+ { url = "https://files.pythonhosted.org/packages/76/8e/699a5b955f7926956c95e9e1d74132acad73c2fe7a426f94da89123c20aa/ast_serialize-0.5.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1943db345233cc7194a470f13afa9c59772c0b123dea0c9414c4d4ca54369759", size = 1421410, upload-time = "2026-05-17T17:48:18.527Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/ae/d5b7626874478997adc7a29ab28accf21e596fb590c944290401dfd0b29e/ast_serialize-0.5.0-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:df1c00022cbbcb064bfaa505aa9c9295362443ce5dacb459d1331d3da353f887", size = 1516587, upload-time = "2026-05-17T17:48:20.133Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/ce/b59e02a82d9c4244d64cde502e0b00e83e38816abe19155ceb5437402c7f/ast_serialize-0.5.0-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:cae65289fc456fde04af979a2be09302ef5d8ab92ef23e596d6746dc267ada27", size = 1515171, upload-time = "2026-05-17T17:48:21.921Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/38/d8d90042747d05aa08d4efcf1c99035a5f670a6bf4c214d31644392afbca/ast_serialize-0.5.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:239a4c354e8d676e9d94631d1d4a64edc6b266f86ff3a5a80aedd344f342c01d", size = 1464668, upload-time = "2026-05-17T17:48:23.544Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/51/5b840c4df7334104cecffa28f23904fe81ca89ca223d2450e288de39fd3c/ast_serialize-0.5.0-cp39-abi3-win32.whl", hash = "sha256:143a4ef63285a075871908fda3672dc21864b83a8ec3ee12304aa3e4c5387b9a", size = 1068311, upload-time = "2026-05-17T17:48:25.027Z" },
+ { url = "https://files.pythonhosted.org/packages/41/11/ca5672c7d491825bc4cd6702dea106a6b60d928707712ec257c7833ae476/ast_serialize-0.5.0-cp39-abi3-win_amd64.whl", hash = "sha256:cf25572c526add400f26a4750dc6ce0c3bb93fc1f75e7ae0cad4ce4f2cd5c590", size = 1108931, upload-time = "2026-05-17T17:48:26.591Z" },
+ { url = "https://files.pythonhosted.org/packages/45/19/cc8bd127d28a43da249aa955cfd164cf8fd534e79e42cea96c4854d72fd0/ast_serialize-0.5.0-cp39-abi3-win_arm64.whl", hash = "sha256:92a31c9c20d25a076edaeec76b128a3535d74a24f340b9a8a7e96c9b86dc9642", size = 1081181, upload-time = "2026-05-17T17:48:28.122Z" },
+]
+
+[[package]]
+name = "asttokens"
+version = "3.0.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/be/a5/8e3f9b6771b0b408517c82d97aed8f2036509bc247d46114925e32fe33f0/asttokens-3.0.1.tar.gz", hash = "sha256:71a4ee5de0bde6a31d64f6b13f2293ac190344478f081c3d1bccfcf5eacb0cb7", size = 62308, upload-time = "2025-11-15T16:43:48.578Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d2/39/e7eaf1799466a4aef85b6a4fe7bd175ad2b1c6345066aa33f1f58d4b18d0/asttokens-3.0.1-py3-none-any.whl", hash = "sha256:15a3ebc0f43c2d0a50eeafea25e19046c68398e487b9f1f5b517f7c0f40f976a", size = 27047, upload-time = "2025-11-15T16:43:16.109Z" },
+]
+
+[[package]]
+name = "attrs"
+version = "26.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/9a/8e/82a0fe20a541c03148528be8cac2408564a6c9a0cc7e9171802bc1d26985/attrs-26.1.0.tar.gz", hash = "sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32", size = 952055, upload-time = "2026-03-19T14:22:25.026Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" },
+]
+
+[[package]]
+name = "babel"
+version = "2.18.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/7d/b2/51899539b6ceeeb420d40ed3cd4b7a40519404f9baf3d4ac99dc413a834b/babel-2.18.0.tar.gz", hash = "sha256:b80b99a14bd085fcacfa15c9165f651fbb3406e66cc603abf11c5750937c992d", size = 9959554, upload-time = "2026-02-01T12:30:56.078Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/77/f5/21d2de20e8b8b0408f0681956ca2c69f1320a3848ac50e6e7f39c6159675/babel-2.18.0-py3-none-any.whl", hash = "sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35", size = 10196845, upload-time = "2026-02-01T12:30:53.445Z" },
+]
+
+[[package]]
+name = "beautifulsoup4"
+version = "4.15.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "soupsieve" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/43/65/318323f98dbee45d42dff61d8f047181bc6f2268a9068cfad035a46be5af/beautifulsoup4-4.15.0.tar.gz", hash = "sha256:288e3ca7d54b06f2ac191970bc275c1939cb46d450b255bf6718b04aa37ab4f7", size = 632571, upload-time = "2026-06-07T16:44:20.453Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/88/c6/92fcd42f1ba33e1184263f25bfabf3d27c383410470f169e4b8163bf9c17/beautifulsoup4-4.15.0-py3-none-any.whl", hash = "sha256:d6f88de62e1d4e38ecb1077eb9724cd0eff29d2a08ca16a401e9b9e93f117cf9", size = 109924, upload-time = "2026-06-07T16:44:21.566Z" },
+]
+
+[[package]]
+name = "cbor2"
+version = "6.1.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/3a/6f/07b4af8da8bd27f640362b1ac8271d80895407f2ede0c2bcc9433c06e1ca/cbor2-6.1.3.tar.gz", hash = "sha256:8d70680acb55c04ea5b5ad86da094f9612b53d5a8a65d0f5b3aafc3ce917ecbb", size = 89503, upload-time = "2026-07-04T10:36:48.793Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/31/16/cff14259c3d19a7f0ae88b6996fe4c85f6ff1764dad889ac8a39e843e39c/cbor2-6.1.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:3d939f55097c21e032f5a2d67592fcc57298986281f219356e2f519e4466f4ea", size = 412779, upload-time = "2026-07-04T10:36:04.975Z" },
+ { url = "https://files.pythonhosted.org/packages/50/6c/f3641d19b7b85a63cb2756c10164131489c2cb46b379ec51ae22283fefb9/cbor2-6.1.3-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:b025009478d644dab407164fd60e3ef4381af284f5af6966df94c663756d949e", size = 457781, upload-time = "2026-07-04T10:36:06.349Z" },
+ { url = "https://files.pythonhosted.org/packages/55/85/0c55a66f3037056bfb8e1c7184168085fdea67ae5830404498bcf466233b/cbor2-6.1.3-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:2226d32e102e375737656ad5d141ad8c6ae3e705e04e263f24756f0eb379c6c1", size = 468373, upload-time = "2026-07-04T10:36:07.769Z" },
+ { url = "https://files.pythonhosted.org/packages/46/74/40f7db3e0d880560193916a5c9b744fcf299558bed7113f77c28237c7c29/cbor2-6.1.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:e61d465244d66ffed36492eef3b44d43795d76a2bba0663a2f15c186af7f7513", size = 523844, upload-time = "2026-07-04T10:36:09.404Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/a1/b5e07d6a08441c3a552fe2ae48ccb7e9dfc5065b9f6a3bae9879b4f0fbc0/cbor2-6.1.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:87fe7be8fab6ec4796aa127c1a52e09e79dbafd2aa31caf809cf04b8080a5975", size = 536238, upload-time = "2026-07-04T10:36:10.914Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/99/e166be0fd74bf3a91f5a0d103e34883efbc438d970f72cc8200e274787e5/cbor2-6.1.3-cp312-cp312-win32.whl", hash = "sha256:da25d345f01e6a40b2e5c57ef96b4dcff7be69394fb62f0f70e07f437f2376a9", size = 279858, upload-time = "2026-07-04T10:36:12.247Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/1b/90b4a121e40aba189c55a5822dd3c698eaf487e1d4a780ab18c804a5ef1c/cbor2-6.1.3-cp312-cp312-win_amd64.whl", hash = "sha256:d5514f693db6fa6f433b4096e9b604e6a7bf151c9ef1d2db86d0858e4c5e768f", size = 300929, upload-time = "2026-07-04T10:36:13.564Z" },
+ { url = "https://files.pythonhosted.org/packages/19/db/52c58a8d33464927389dde8103997b3fa51b081ce29b347ac2cc4fd0dfbf/cbor2-6.1.3-cp312-cp312-win_arm64.whl", hash = "sha256:3d43183d7beb3d3cd198d69b31bd2ee487ed704a1150c75cb0a66d6ad63d8c1a", size = 290908, upload-time = "2026-07-04T10:36:14.857Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/8c/5024d623dcf3f2057ec8c991f584b939ba5f9025a5ce8c31f6fac067137a/cbor2-6.1.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:21c74b8ab67977c8b87b727247eeb730145b0068ad6d47f71e9f80f6b48c65f8", size = 412334, upload-time = "2026-07-04T10:36:16.299Z" },
+ { url = "https://files.pythonhosted.org/packages/61/f3/e50654203c3b746166a96bea680eb6463b20c2c160cc14dfbe43f215ef6c/cbor2-6.1.3-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:f291a0ae4c1ed96eadb0afa9752568c7424f7d6fa818676d5e33005fcd22ddd9", size = 457125, upload-time = "2026-07-04T10:36:17.684Z" },
+ { url = "https://files.pythonhosted.org/packages/44/86/6ef007f0d4f7afba90a80cb1657984de542e7474d2afaa7e920ac9860df3/cbor2-6.1.3-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:dc8e44c7bf172687195dcd428157885bc00ea06efc0ea30fb371163b92bef733", size = 467651, upload-time = "2026-07-04T10:36:19.007Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/f4/b5aa27813c02f37e03eb86bd908163562edd6fc7f99665bc7bbb25ef5e6c/cbor2-6.1.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:8ccf4d263983d830dd429d2b01f27be58ac02ba7c790c45d861f767eb63963e5", size = 523296, upload-time = "2026-07-04T10:36:20.504Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/46/e17b2bce2efdc26bfa045b4f6168f02923ac5f0e79732a1b3c42ce9ca9de/cbor2-6.1.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8719a7a2a2a82168844533389957b8f617a139f5f40e4d0ad7ed905fd3abebd1", size = 535537, upload-time = "2026-07-04T10:36:21.761Z" },
+ { url = "https://files.pythonhosted.org/packages/54/bc/add350acf37f367ae429f997f2e047b042f2d5ef9ca62461c923fbb0f3c3/cbor2-6.1.3-cp313-cp313-win32.whl", hash = "sha256:c73b54ce09dd8d522f3c1540426e36172ba0f34abf3d89eb93909a5e14590003", size = 279233, upload-time = "2026-07-04T10:36:23.071Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/6f/1bbfce3b3131e4e03e8a86966a38ff92ebb72215fcf36aeecea1547f3e4b/cbor2-6.1.3-cp313-cp313-win_amd64.whl", hash = "sha256:b77df56c462c10eb3444db8ef78d8c3e71d9ef8d021ea92e97c1f9e3aa918690", size = 300585, upload-time = "2026-07-04T10:36:24.563Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/65/3945702dd84b6e5b7800c9c7f1ada038d33d12d0042de10e38164cc03dfd/cbor2-6.1.3-cp313-cp313-win_arm64.whl", hash = "sha256:b144be2ab3e9584ee7b6359d2a92fee0a5bec1d00dbd34c215ccc2040ac0b2ab", size = 290357, upload-time = "2026-07-04T10:36:25.983Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/40/04ad7d34182b27487a1824422b361b32d2607727ef20e563056eba62d12a/cbor2-6.1.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:3f3167ca9920b90db4ff72652db109dfd93b56ee0d583aca12f3a5d9a7019477", size = 414615, upload-time = "2026-07-04T10:36:27.299Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/29/94238c61f90653a606535e7509a2af312089fe10dafcb4cf82d6905a7a1a/cbor2-6.1.3-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:48c677971e4b71e685491e1a267d9924dc205e7ddbe3f34fd2562f16c0f6bfca", size = 459084, upload-time = "2026-07-04T10:36:28.717Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/cb/6bd33461e8be8ded7ebb0fa38994a63752aefae2b4fcd1b2cc71ee3c06f1/cbor2-6.1.3-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:ad4f3c6dfc6b83331eb04c6975efb2839ab65a3aa81502bc2b3f7945d4c4aa44", size = 469310, upload-time = "2026-07-04T10:36:30.136Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/d1/94195bcd8fcc1030ecaf22a7a825faa08891b5bb2d3553e465e50fe115f5/cbor2-6.1.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:8bcd609eab4a39745123bfb28e73311abba3d14975a87f5906e0e8b910d918ac", size = 524287, upload-time = "2026-07-04T10:36:31.453Z" },
+ { url = "https://files.pythonhosted.org/packages/06/55/57178fbf2d1206af5299c688f9c917b83f30636694c8f932dc89c6652545/cbor2-6.1.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:672727ecb27d7fb3ca0bf8a58fc489d5374ab1fee680ed3d0348a11d9d3ca78f", size = 537031, upload-time = "2026-07-04T10:36:32.769Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/01/beefe26258d66ce39a9b057b679bc67dba81698d5a63e0ada5b4752a5c87/cbor2-6.1.3-cp314-cp314-win32.whl", hash = "sha256:1413cef2aa7f478a38298cd3492a055e8e8e45d17fb53bbe103e79ca15c33f3c", size = 286256, upload-time = "2026-07-04T10:36:34.078Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/2d/79eb513a2586a2053a6f18b33693beda65e2c766820d99676a306573fed5/cbor2-6.1.3-cp314-cp314-win_amd64.whl", hash = "sha256:59df264d4a508ba61daaa0bf3c2f92d63275509549a0875c1fa38176f651e4f8", size = 313744, upload-time = "2026-07-04T10:36:35.58Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/c7/ab9828e4efc26badf89f1397f866f3ef03ef65cdcd55518254b84bdaf86e/cbor2-6.1.3-cp314-cp314-win_arm64.whl", hash = "sha256:b62b5d80a0eb4305cd5f0217faa4d7747bd64fe0dff9b88415e7be3782f8249b", size = 304280, upload-time = "2026-07-04T10:36:36.865Z" },
+ { url = "https://files.pythonhosted.org/packages/33/cf/54a497ad1026833c1c92d482edbda0bdebb48314b51563d2fcea24ab89b4/cbor2-6.1.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:856ab525bfc599588b8d2a45babb7c3400c693ea0bb574d818467d997102fe24", size = 409638, upload-time = "2026-07-04T10:36:38.089Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/03/efdecd0848b9c9e43242537f6dd8ac5d441a077d362e5e6954c7775b866a/cbor2-6.1.3-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:a076f35abf1dd0a4de6e2f7f4d4932abafc951a26275b6aa4a3b370c2fd3bbf4", size = 452193, upload-time = "2026-07-04T10:36:39.56Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/bf/b5f43c75dc5f0ca3127919d3273d8671917932aa3e90767da63867e0f06f/cbor2-6.1.3-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:37ccab1d0bd3f57ff536a41e44165d0c99cb166ac6e5ffb8b93c42304b56e48d", size = 466614, upload-time = "2026-07-04T10:36:40.936Z" },
+ { url = "https://files.pythonhosted.org/packages/23/ee/e85b2ddd46b3b43e39a986704353b433efab0881234e1b4d824229fc2a75/cbor2-6.1.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:76d257ce797e651fa430b0269e8e8c43549c54ce8b0d12860569b3709bf1326f", size = 518503, upload-time = "2026-07-04T10:36:42.289Z" },
+ { url = "https://files.pythonhosted.org/packages/60/13/c740c0002f127dc3e9e87b61a5d1285dd3b71aa11d8e0f6a408fc1f36173/cbor2-6.1.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:4af35baadb66f7c9cbb3998eab469767c04641552416c1c69dd4d3d183797119", size = 534236, upload-time = "2026-07-04T10:36:43.641Z" },
+ { url = "https://files.pythonhosted.org/packages/73/cd/a57d97177f3777c96f3406efb0ad60794fdbf249d497ec24559bfd1d0328/cbor2-6.1.3-cp314-cp314t-win32.whl", hash = "sha256:61c92661665bccfed4ffa69d1fe10097f2c820d262f56a8cf909a5ebf9f6d8c6", size = 282446, upload-time = "2026-07-04T10:36:44.888Z" },
+ { url = "https://files.pythonhosted.org/packages/11/c8/dd54878589df22c863d526cb82e1bb20e953d40223436449a52481c49804/cbor2-6.1.3-cp314-cp314t-win_amd64.whl", hash = "sha256:dc6bcd030bf5043662b84b0ca0f0ca942491bf509105db30cedca6e2ce82d158", size = 310300, upload-time = "2026-07-04T10:36:46.212Z" },
+ { url = "https://files.pythonhosted.org/packages/05/d0/b0780a396d145e3356bfdc48578d021ade1818a6c7d7842a3d6bc6f16fbb/cbor2-6.1.3-cp314-cp314t-win_arm64.whl", hash = "sha256:da98a5e0ae9487bed497ac74e2c850b49975a3b7b5314b76c3843e2c83a6c8c4", size = 299391, upload-time = "2026-07-04T10:36:47.629Z" },
+]
+
+[[package]]
+name = "certifi"
+version = "2026.6.17"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c9/c7/424b75da314c1045981bd9777432fad05a9e0c69daa4ed7e308bbaffe405/certifi-2026.6.17.tar.gz", hash = "sha256:024c88eeec92ca068db80f02b8b07c9cef7b9fe261d1d535abfd5abd6f6af432", size = 134594, upload-time = "2026-06-17T10:31:07.894Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ef/2f/c5464532e965badff2f4c4c1a3a83f5697f0d7c407ed0cda44aaa99bb451/certifi-2026.6.17-py3-none-any.whl", hash = "sha256:2227dcbaafe0d2f59279d1762ddddc37783ed4354594f194ffc31d20f41fc3db", size = 133289, upload-time = "2026-06-17T10:31:06.348Z" },
+]
+
+[[package]]
+name = "cffi"
+version = "2.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pycparser", marker = "implementation_name != 'PyPy'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/eb/56/b1ba7935a17738ae8453301356628e8147c79dbb825bcbc73dc7401f9846/cffi-2.0.0.tar.gz", hash = "sha256:44d1b5909021139fe36001ae048dbdde8214afa20200eda0f64c068cac5d5529", size = 523588, upload-time = "2025-09-08T23:24:04.541Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ea/47/4f61023ea636104d4f16ab488e268b93008c3d0bb76893b1b31db1f96802/cffi-2.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6d02d6655b0e54f54c4ef0b94eb6be0607b70853c45ce98bd278dc7de718be5d", size = 185271, upload-time = "2025-09-08T23:22:44.795Z" },
+ { url = "https://files.pythonhosted.org/packages/df/a2/781b623f57358e360d62cdd7a8c681f074a71d445418a776eef0aadb4ab4/cffi-2.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8eca2a813c1cb7ad4fb74d368c2ffbbb4789d377ee5bb8df98373c2cc0dee76c", size = 181048, upload-time = "2025-09-08T23:22:45.938Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/df/a4f0fbd47331ceeba3d37c2e51e9dfc9722498becbeec2bd8bc856c9538a/cffi-2.0.0-cp312-cp312-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:21d1152871b019407d8ac3985f6775c079416c282e431a4da6afe7aefd2bccbe", size = 212529, upload-time = "2025-09-08T23:22:47.349Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/72/12b5f8d3865bf0f87cf1404d8c374e7487dcf097a1c91c436e72e6badd83/cffi-2.0.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b21e08af67b8a103c71a250401c78d5e0893beff75e28c53c98f4de42f774062", size = 220097, upload-time = "2025-09-08T23:22:48.677Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/95/7a135d52a50dfa7c882ab0ac17e8dc11cec9d55d2c18dda414c051c5e69e/cffi-2.0.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:1e3a615586f05fc4065a8b22b8152f0c1b00cdbc60596d187c2a74f9e3036e4e", size = 207983, upload-time = "2025-09-08T23:22:50.06Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/c8/15cb9ada8895957ea171c62dc78ff3e99159ee7adb13c0123c001a2546c1/cffi-2.0.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:81afed14892743bbe14dacb9e36d9e0e504cd204e0b165062c488942b9718037", size = 206519, upload-time = "2025-09-08T23:22:51.364Z" },
+ { url = "https://files.pythonhosted.org/packages/78/2d/7fa73dfa841b5ac06c7b8855cfc18622132e365f5b81d02230333ff26e9e/cffi-2.0.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3e17ed538242334bf70832644a32a7aae3d83b57567f9fd60a26257e992b79ba", size = 219572, upload-time = "2025-09-08T23:22:52.902Z" },
+ { url = "https://files.pythonhosted.org/packages/07/e0/267e57e387b4ca276b90f0434ff88b2c2241ad72b16d31836adddfd6031b/cffi-2.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3925dd22fa2b7699ed2617149842d2e6adde22b262fcbfada50e3d195e4b3a94", size = 222963, upload-time = "2025-09-08T23:22:54.518Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/75/1f2747525e06f53efbd878f4d03bac5b859cbc11c633d0fb81432d98a795/cffi-2.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2c8f814d84194c9ea681642fd164267891702542f028a15fc97d4674b6206187", size = 221361, upload-time = "2025-09-08T23:22:55.867Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/2b/2b6435f76bfeb6bbf055596976da087377ede68df465419d192acf00c437/cffi-2.0.0-cp312-cp312-win32.whl", hash = "sha256:da902562c3e9c550df360bfa53c035b2f241fed6d9aef119048073680ace4a18", size = 172932, upload-time = "2025-09-08T23:22:57.188Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/ed/13bd4418627013bec4ed6e54283b1959cf6db888048c7cf4b4c3b5b36002/cffi-2.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:da68248800ad6320861f129cd9c1bf96ca849a2771a59e0344e88681905916f5", size = 183557, upload-time = "2025-09-08T23:22:58.351Z" },
+ { url = "https://files.pythonhosted.org/packages/95/31/9f7f93ad2f8eff1dbc1c3656d7ca5bfd8fb52c9d786b4dcf19b2d02217fa/cffi-2.0.0-cp312-cp312-win_arm64.whl", hash = "sha256:4671d9dd5ec934cb9a73e7ee9676f9362aba54f7f34910956b84d727b0d73fb6", size = 177762, upload-time = "2025-09-08T23:22:59.668Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/8d/a0a47a0c9e413a658623d014e91e74a50cdd2c423f7ccfd44086ef767f90/cffi-2.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:00bdf7acc5f795150faa6957054fbbca2439db2f775ce831222b66f192f03beb", size = 185230, upload-time = "2025-09-08T23:23:00.879Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/d2/a6c0296814556c68ee32009d9c2ad4f85f2707cdecfd7727951ec228005d/cffi-2.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:45d5e886156860dc35862657e1494b9bae8dfa63bf56796f2fb56e1679fc0bca", size = 181043, upload-time = "2025-09-08T23:23:02.231Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/1e/d22cc63332bd59b06481ceaac49d6c507598642e2230f201649058a7e704/cffi-2.0.0-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:07b271772c100085dd28b74fa0cd81c8fb1a3ba18b21e03d7c27f3436a10606b", size = 212446, upload-time = "2025-09-08T23:23:03.472Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/f5/a2c23eb03b61a0b8747f211eb716446c826ad66818ddc7810cc2cc19b3f2/cffi-2.0.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d48a880098c96020b02d5a1f7d9251308510ce8858940e6fa99ece33f610838b", size = 220101, upload-time = "2025-09-08T23:23:04.792Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/7f/e6647792fc5850d634695bc0e6ab4111ae88e89981d35ac269956605feba/cffi-2.0.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:f93fd8e5c8c0a4aa1f424d6173f14a892044054871c771f8566e4008eaa359d2", size = 207948, upload-time = "2025-09-08T23:23:06.127Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/1e/a5a1bd6f1fb30f22573f76533de12a00bf274abcdc55c8edab639078abb6/cffi-2.0.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:dd4f05f54a52fb558f1ba9f528228066954fee3ebe629fc1660d874d040ae5a3", size = 206422, upload-time = "2025-09-08T23:23:07.753Z" },
+ { url = "https://files.pythonhosted.org/packages/98/df/0a1755e750013a2081e863e7cd37e0cdd02664372c754e5560099eb7aa44/cffi-2.0.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c8d3b5532fc71b7a77c09192b4a5a200ea992702734a2e9279a37f2478236f26", size = 219499, upload-time = "2025-09-08T23:23:09.648Z" },
+ { url = "https://files.pythonhosted.org/packages/50/e1/a969e687fcf9ea58e6e2a928ad5e2dd88cc12f6f0ab477e9971f2309b57c/cffi-2.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d9b29c1f0ae438d5ee9acb31cadee00a58c46cc9c0b2f9038c6b0b3470877a8c", size = 222928, upload-time = "2025-09-08T23:23:10.928Z" },
+ { url = "https://files.pythonhosted.org/packages/36/54/0362578dd2c9e557a28ac77698ed67323ed5b9775ca9d3fe73fe191bb5d8/cffi-2.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6d50360be4546678fc1b79ffe7a66265e28667840010348dd69a314145807a1b", size = 221302, upload-time = "2025-09-08T23:23:12.42Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/6d/bf9bda840d5f1dfdbf0feca87fbdb64a918a69bca42cfa0ba7b137c48cb8/cffi-2.0.0-cp313-cp313-win32.whl", hash = "sha256:74a03b9698e198d47562765773b4a8309919089150a0bb17d829ad7b44b60d27", size = 172909, upload-time = "2025-09-08T23:23:14.32Z" },
+ { url = "https://files.pythonhosted.org/packages/37/18/6519e1ee6f5a1e579e04b9ddb6f1676c17368a7aba48299c3759bbc3c8b3/cffi-2.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:19f705ada2530c1167abacb171925dd886168931e0a7b78f5bffcae5c6b5be75", size = 183402, upload-time = "2025-09-08T23:23:15.535Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/0e/02ceeec9a7d6ee63bb596121c2c8e9b3a9e150936f4fbef6ca1943e6137c/cffi-2.0.0-cp313-cp313-win_arm64.whl", hash = "sha256:256f80b80ca3853f90c21b23ee78cd008713787b1b1e93eae9f3d6a7134abd91", size = 177780, upload-time = "2025-09-08T23:23:16.761Z" },
+ { url = "https://files.pythonhosted.org/packages/92/c4/3ce07396253a83250ee98564f8d7e9789fab8e58858f35d07a9a2c78de9f/cffi-2.0.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:fc33c5141b55ed366cfaad382df24fe7dcbc686de5be719b207bb248e3053dc5", size = 185320, upload-time = "2025-09-08T23:23:18.087Z" },
+ { url = "https://files.pythonhosted.org/packages/59/dd/27e9fa567a23931c838c6b02d0764611c62290062a6d4e8ff7863daf9730/cffi-2.0.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c654de545946e0db659b3400168c9ad31b5d29593291482c43e3564effbcee13", size = 181487, upload-time = "2025-09-08T23:23:19.622Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/43/0e822876f87ea8a4ef95442c3d766a06a51fc5298823f884ef87aaad168c/cffi-2.0.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:24b6f81f1983e6df8db3adc38562c83f7d4a0c36162885ec7f7b77c7dcbec97b", size = 220049, upload-time = "2025-09-08T23:23:20.853Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/89/76799151d9c2d2d1ead63c2429da9ea9d7aac304603de0c6e8764e6e8e70/cffi-2.0.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:12873ca6cb9b0f0d3a0da705d6086fe911591737a59f28b7936bdfed27c0d47c", size = 207793, upload-time = "2025-09-08T23:23:22.08Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/dd/3465b14bb9e24ee24cb88c9e3730f6de63111fffe513492bf8c808a3547e/cffi-2.0.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:d9b97165e8aed9272a6bb17c01e3cc5871a594a446ebedc996e2397a1c1ea8ef", size = 206300, upload-time = "2025-09-08T23:23:23.314Z" },
+ { url = "https://files.pythonhosted.org/packages/47/d9/d83e293854571c877a92da46fdec39158f8d7e68da75bf73581225d28e90/cffi-2.0.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:afb8db5439b81cf9c9d0c80404b60c3cc9c3add93e114dcae767f1477cb53775", size = 219244, upload-time = "2025-09-08T23:23:24.541Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/0f/1f177e3683aead2bb00f7679a16451d302c436b5cbf2505f0ea8146ef59e/cffi-2.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:737fe7d37e1a1bffe70bd5754ea763a62a066dc5913ca57e957824b72a85e205", size = 222828, upload-time = "2025-09-08T23:23:26.143Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/0f/cafacebd4b040e3119dcb32fed8bdef8dfe94da653155f9d0b9dc660166e/cffi-2.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:38100abb9d1b1435bc4cc340bb4489635dc2f0da7456590877030c9b3d40b0c1", size = 220926, upload-time = "2025-09-08T23:23:27.873Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/aa/df335faa45b395396fcbc03de2dfcab242cd61a9900e914fe682a59170b1/cffi-2.0.0-cp314-cp314-win32.whl", hash = "sha256:087067fa8953339c723661eda6b54bc98c5625757ea62e95eb4898ad5e776e9f", size = 175328, upload-time = "2025-09-08T23:23:44.61Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/92/882c2d30831744296ce713f0feb4c1cd30f346ef747b530b5318715cc367/cffi-2.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:203a48d1fb583fc7d78a4c6655692963b860a417c0528492a6bc21f1aaefab25", size = 185650, upload-time = "2025-09-08T23:23:45.848Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/2c/98ece204b9d35a7366b5b2c6539c350313ca13932143e79dc133ba757104/cffi-2.0.0-cp314-cp314-win_arm64.whl", hash = "sha256:dbd5c7a25a7cb98f5ca55d258b103a2054f859a46ae11aaf23134f9cc0d356ad", size = 180687, upload-time = "2025-09-08T23:23:47.105Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/61/c768e4d548bfa607abcda77423448df8c471f25dbe64fb2ef6d555eae006/cffi-2.0.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:9a67fc9e8eb39039280526379fb3a70023d77caec1852002b4da7e8b270c4dd9", size = 188773, upload-time = "2025-09-08T23:23:29.347Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/ea/5f76bce7cf6fcd0ab1a1058b5af899bfbef198bea4d5686da88471ea0336/cffi-2.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7a66c7204d8869299919db4d5069a82f1561581af12b11b3c9f48c584eb8743d", size = 185013, upload-time = "2025-09-08T23:23:30.63Z" },
+ { url = "https://files.pythonhosted.org/packages/be/b4/c56878d0d1755cf9caa54ba71e5d049479c52f9e4afc230f06822162ab2f/cffi-2.0.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7cc09976e8b56f8cebd752f7113ad07752461f48a58cbba644139015ac24954c", size = 221593, upload-time = "2025-09-08T23:23:31.91Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/0d/eb704606dfe8033e7128df5e90fee946bbcb64a04fcdaa97321309004000/cffi-2.0.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:92b68146a71df78564e4ef48af17551a5ddd142e5190cdf2c5624d0c3ff5b2e8", size = 209354, upload-time = "2025-09-08T23:23:33.214Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/19/3c435d727b368ca475fb8742ab97c9cb13a0de600ce86f62eab7fa3eea60/cffi-2.0.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:b1e74d11748e7e98e2f426ab176d4ed720a64412b6a15054378afdb71e0f37dc", size = 208480, upload-time = "2025-09-08T23:23:34.495Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/44/681604464ed9541673e486521497406fadcc15b5217c3e326b061696899a/cffi-2.0.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:28a3a209b96630bca57cce802da70c266eb08c6e97e5afd61a75611ee6c64592", size = 221584, upload-time = "2025-09-08T23:23:36.096Z" },
+ { url = "https://files.pythonhosted.org/packages/25/8e/342a504ff018a2825d395d44d63a767dd8ebc927ebda557fecdaca3ac33a/cffi-2.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:7553fb2090d71822f02c629afe6042c299edf91ba1bf94951165613553984512", size = 224443, upload-time = "2025-09-08T23:23:37.328Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/5e/b666bacbbc60fbf415ba9988324a132c9a7a0448a9a8f125074671c0f2c3/cffi-2.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:6c6c373cfc5c83a975506110d17457138c8c63016b563cc9ed6e056a82f13ce4", size = 223437, upload-time = "2025-09-08T23:23:38.945Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/1d/ec1a60bd1a10daa292d3cd6bb0b359a81607154fb8165f3ec95fe003b85c/cffi-2.0.0-cp314-cp314t-win32.whl", hash = "sha256:1fc9ea04857caf665289b7a75923f2c6ed559b8298a1b8c49e59f7dd95c8481e", size = 180487, upload-time = "2025-09-08T23:23:40.423Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/41/4c1168c74fac325c0c8156f04b6749c8b6a8f405bbf91413ba088359f60d/cffi-2.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:d68b6cef7827e8641e8ef16f4494edda8b36104d79773a334beaa1e3521430f6", size = 191726, upload-time = "2025-09-08T23:23:41.742Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/3a/dbeec9d1ee0844c679f6bb5d6ad4e9f198b1224f4e7a32825f47f6192b0c/cffi-2.0.0-cp314-cp314t-win_arm64.whl", hash = "sha256:0a1527a803f0a659de1af2e1fd700213caba79377e27e4693648c2923da066f9", size = 184195, upload-time = "2025-09-08T23:23:43.004Z" },
+]
+
+[[package]]
+name = "charset-normalizer"
+version = "3.4.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e7/a1/67fe25fac3c7642725500a3f6cfe5821ad557c3abb11c9d20d12c7008d3e/charset_normalizer-3.4.7.tar.gz", hash = "sha256:ae89db9e5f98a11a4bf50407d4363e7b09b31e55bc117b4f7d80aab97ba009e5", size = 144271, upload-time = "2026-04-02T09:28:39.342Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0c/eb/4fc8d0a7110eb5fc9cc161723a34a8a6c200ce3b4fbf681bc86feee22308/charset_normalizer-3.4.7-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:eca9705049ad3c7345d574e3510665cb2cf844c2f2dcfe675332677f081cbd46", size = 311328, upload-time = "2026-04-02T09:26:24.331Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/e3/0fadc706008ac9d7b9b5be6dc767c05f9d3e5df51744ce4cc9605de7b9f4/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6178f72c5508bfc5fd446a5905e698c6212932f25bcdd4b47a757a50605a90e2", size = 208061, upload-time = "2026-04-02T09:26:25.568Z" },
+ { url = "https://files.pythonhosted.org/packages/42/f0/3dd1045c47f4a4604df85ec18ad093912ae1344ac706993aff91d38773a2/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e1421b502d83040e6d7fb2fb18dff63957f720da3d77b2fbd3187ceb63755d7b", size = 229031, upload-time = "2026-04-02T09:26:26.865Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/67/675a46eb016118a2fbde5a277a5d15f4f69d5f3f5f338e5ee2f8948fcf43/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:edac0f1ab77644605be2cbba52e6b7f630731fc42b34cb0f634be1a6eface56a", size = 225239, upload-time = "2026-04-02T09:26:28.044Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/f8/d0118a2f5f23b02cd166fa385c60f9b0d4f9194f574e2b31cef350ad7223/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5649fd1c7bade02f320a462fdefd0b4bd3ce036065836d4f42e0de958038e116", size = 216589, upload-time = "2026-04-02T09:26:29.239Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/f1/6d2b0b261b6c4ceef0fcb0d17a01cc5bc53586c2d4796fa04b5c540bc13d/charset_normalizer-3.4.7-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:203104ed3e428044fd943bc4bf45fa73c0730391f9621e37fe39ecf477b128cb", size = 202733, upload-time = "2026-04-02T09:26:30.5Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/c0/7b1f943f7e87cc3db9626ba17807d042c38645f0a1d4415c7a14afb5591f/charset_normalizer-3.4.7-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:298930cec56029e05497a76988377cbd7457ba864beeea92ad7e844fe74cd1f1", size = 212652, upload-time = "2026-04-02T09:26:31.709Z" },
+ { url = "https://files.pythonhosted.org/packages/38/dd/5a9ab159fe45c6e72079398f277b7d2b523e7f716acc489726115a910097/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:708838739abf24b2ceb208d0e22403dd018faeef86ddac04319a62ae884c4f15", size = 211229, upload-time = "2026-04-02T09:26:33.282Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/ff/531a1cad5ca855d1c1a8b69cb71abfd6d85c0291580146fda7c82857caa1/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:0f7eb884681e3938906ed0434f20c63046eacd0111c4ba96f27b76084cd679f5", size = 203552, upload-time = "2026-04-02T09:26:34.845Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/4c/a5fb52d528a8ca41f7598cb619409ece30a169fbdf9cdce592e53b46c3a6/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:4dc1e73c36828f982bfe79fadf5919923f8a6f4df2860804db9a98c48824ce8d", size = 230806, upload-time = "2026-04-02T09:26:36.152Z" },
+ { url = "https://files.pythonhosted.org/packages/59/7a/071feed8124111a32b316b33ae4de83d36923039ef8cf48120266844285b/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:aed52fea0513bac0ccde438c188c8a471c4e0f457c2dd20cdbf6ea7a450046c7", size = 212316, upload-time = "2026-04-02T09:26:37.672Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/35/f7dba3994312d7ba508e041eaac39a36b120f32d4c8662b8814dab876431/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:fea24543955a6a729c45a73fe90e08c743f0b3334bbf3201e6c4bc1b0c7fa464", size = 227274, upload-time = "2026-04-02T09:26:38.93Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/2d/a572df5c9204ab7688ec1edc895a73ebded3b023bb07364710b05dd1c9be/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:bb6d88045545b26da47aa879dd4a89a71d1dce0f0e549b1abcb31dfe4a8eac49", size = 218468, upload-time = "2026-04-02T09:26:40.17Z" },
+ { url = "https://files.pythonhosted.org/packages/86/eb/890922a8b03a568ca2f336c36585a4713c55d4d67bf0f0c78924be6315ca/charset_normalizer-3.4.7-cp312-cp312-win32.whl", hash = "sha256:2257141f39fe65a3fdf38aeccae4b953e5f3b3324f4ff0daf9f15b8518666a2c", size = 148460, upload-time = "2026-04-02T09:26:41.416Z" },
+ { url = "https://files.pythonhosted.org/packages/35/d9/0e7dffa06c5ab081f75b1b786f0aefc88365825dfcd0ac544bdb7b2b6853/charset_normalizer-3.4.7-cp312-cp312-win_amd64.whl", hash = "sha256:5ed6ab538499c8644b8a3e18debabcd7ce684f3fa91cf867521a7a0279cab2d6", size = 159330, upload-time = "2026-04-02T09:26:42.554Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/5d/481bcc2a7c88ea6b0878c299547843b2521ccbc40980cb406267088bc701/charset_normalizer-3.4.7-cp312-cp312-win_arm64.whl", hash = "sha256:56be790f86bfb2c98fb742ce566dfb4816e5a83384616ab59c49e0604d49c51d", size = 147828, upload-time = "2026-04-02T09:26:44.075Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/3b/66777e39d3ae1ddc77ee606be4ec6d8cbd4c801f65e5a1b6f2b11b8346dd/charset_normalizer-3.4.7-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f496c9c3cc02230093d8330875c4c3cdfc3b73612a5fd921c65d39cbcef08063", size = 309627, upload-time = "2026-04-02T09:26:45.198Z" },
+ { url = "https://files.pythonhosted.org/packages/2e/4e/b7f84e617b4854ade48a1b7915c8ccfadeba444d2a18c291f696e37f0d3b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0ea948db76d31190bf08bd371623927ee1339d5f2a0b4b1b4a4439a65298703c", size = 207008, upload-time = "2026-04-02T09:26:46.824Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/bb/ec73c0257c9e11b268f018f068f5d00aa0ef8c8b09f7753ebd5f2880e248/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a277ab8928b9f299723bc1a2dabb1265911b1a76341f90a510368ca44ad9ab66", size = 228303, upload-time = "2026-04-02T09:26:48.397Z" },
+ { url = "https://files.pythonhosted.org/packages/85/fb/32d1f5033484494619f701e719429c69b766bfc4dbc61aa9e9c8c166528b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3bec022aec2c514d9cf199522a802bd007cd588ab17ab2525f20f9c34d067c18", size = 224282, upload-time = "2026-04-02T09:26:49.684Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/07/330e3a0dda4c404d6da83b327270906e9654a24f6c546dc886a0eb0ffb23/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e044c39e41b92c845bc815e5ae4230804e8e7bc29e399b0437d64222d92809dd", size = 215595, upload-time = "2026-04-02T09:26:50.915Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/7c/fc890655786e423f02556e0216d4b8c6bcb6bdfa890160dc66bf52dee468/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:f495a1652cf3fbab2eb0639776dad966c2fb874d79d87ca07f9d5f059b8bd215", size = 201986, upload-time = "2026-04-02T09:26:52.197Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/97/bfb18b3db2aed3b90cf54dc292ad79fdd5ad65c4eae454099475cbeadd0d/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e712b419df8ba5e42b226c510472b37bd57b38e897d3eca5e8cfd410a29fa859", size = 211711, upload-time = "2026-04-02T09:26:53.49Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/a5/a581c13798546a7fd557c82614a5c65a13df2157e9ad6373166d2a3e645d/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:7804338df6fcc08105c7745f1502ba68d900f45fd770d5bdd5288ddccb8a42d8", size = 210036, upload-time = "2026-04-02T09:26:54.975Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/bf/b3ab5bcb478e4193d517644b0fb2bf5497fbceeaa7a1bc0f4d5b50953861/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:481551899c856c704d58119b5025793fa6730adda3571971af568f66d2424bb5", size = 202998, upload-time = "2026-04-02T09:26:56.303Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/4e/23efd79b65d314fa320ec6017b4b5834d5c12a58ba4610aa353af2e2f577/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:f59099f9b66f0d7145115e6f80dd8b1d847176df89b234a5a6b3f00437aa0832", size = 230056, upload-time = "2026-04-02T09:26:57.554Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/9f/1e1941bc3f0e01df116e68dc37a55c4d249df5e6fa77f008841aef68264f/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:f59ad4c0e8f6bba240a9bb85504faa1ab438237199d4cce5f622761507b8f6a6", size = 211537, upload-time = "2026-04-02T09:26:58.843Z" },
+ { url = "https://files.pythonhosted.org/packages/80/0f/088cbb3020d44428964a6c97fe1edfb1b9550396bf6d278330281e8b709c/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:3dedcc22d73ec993f42055eff4fcfed9318d1eeb9a6606c55892a26964964e48", size = 226176, upload-time = "2026-04-02T09:27:00.437Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/9f/130394f9bbe06f4f63e22641d32fc9b202b7e251c9aef4db044324dac493/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:64f02c6841d7d83f832cd97ccf8eb8a906d06eb95d5276069175c696b024b60a", size = 217723, upload-time = "2026-04-02T09:27:02.021Z" },
+ { url = "https://files.pythonhosted.org/packages/73/55/c469897448a06e49f8fa03f6caae97074fde823f432a98f979cc42b90e69/charset_normalizer-3.4.7-cp313-cp313-win32.whl", hash = "sha256:4042d5c8f957e15221d423ba781e85d553722fc4113f523f2feb7b188cc34c5e", size = 148085, upload-time = "2026-04-02T09:27:03.192Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/78/1b74c5bbb3f99b77a1715c91b3e0b5bdb6fe302d95ace4f5b1bec37b0167/charset_normalizer-3.4.7-cp313-cp313-win_amd64.whl", hash = "sha256:3946fa46a0cf3e4c8cb1cc52f56bb536310d34f25f01ca9b6c16afa767dab110", size = 158819, upload-time = "2026-04-02T09:27:04.454Z" },
+ { url = "https://files.pythonhosted.org/packages/68/86/46bd42279d323deb8687c4a5a811fd548cb7d1de10cf6535d099877a9a9f/charset_normalizer-3.4.7-cp313-cp313-win_arm64.whl", hash = "sha256:80d04837f55fc81da168b98de4f4b797ef007fc8a79ab71c6ec9bc4dd662b15b", size = 147915, upload-time = "2026-04-02T09:27:05.971Z" },
+ { url = "https://files.pythonhosted.org/packages/97/c8/c67cb8c70e19ef1960b97b22ed2a1567711de46c4ddf19799923adc836c2/charset_normalizer-3.4.7-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:c36c333c39be2dbca264d7803333c896ab8fa7d4d6f0ab7edb7dfd7aea6e98c0", size = 309234, upload-time = "2026-04-02T09:27:07.194Z" },
+ { url = "https://files.pythonhosted.org/packages/99/85/c091fdee33f20de70d6c8b522743b6f831a2f1cd3ff86de4c6a827c48a76/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1c2aed2e5e41f24ea8ef1590b8e848a79b56f3a5564a65ceec43c9d692dc7d8a", size = 208042, upload-time = "2026-04-02T09:27:08.749Z" },
+ { url = "https://files.pythonhosted.org/packages/87/1c/ab2ce611b984d2fd5d86a5a8a19c1ae26acac6bad967da4967562c75114d/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:54523e136b8948060c0fa0bc7b1b50c32c186f2fceee897a495406bb6e311d2b", size = 228706, upload-time = "2026-04-02T09:27:09.951Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/29/2b1d2cb00bf085f59d29eb773ce58ec2d325430f8c216804a0a5cd83cbca/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:715479b9a2802ecac752a3b0efa2b0b60285cf962ee38414211abdfccc233b41", size = 224727, upload-time = "2026-04-02T09:27:11.175Z" },
+ { url = "https://files.pythonhosted.org/packages/47/5c/032c2d5a07fe4d4855fea851209cca2b6f03ebeb6d4e3afdb3358386a684/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bd6c2a1c7573c64738d716488d2cdd3c00e340e4835707d8fdb8dc1a66ef164e", size = 215882, upload-time = "2026-04-02T09:27:12.446Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/c2/356065d5a8b78ed04499cae5f339f091946a6a74f91e03476c33f0ab7100/charset_normalizer-3.4.7-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:c45e9440fb78f8ddabcf714b68f936737a121355bf59f3907f4e17721b9d1aae", size = 200860, upload-time = "2026-04-02T09:27:13.721Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/cd/a32a84217ced5039f53b29f460962abb2d4420def55afabe45b1c3c7483d/charset_normalizer-3.4.7-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3534e7dcbdcf757da6b85a0bbf5b6868786d5982dd959b065e65481644817a18", size = 211564, upload-time = "2026-04-02T09:27:15.272Z" },
+ { url = "https://files.pythonhosted.org/packages/44/86/58e6f13ce26cc3b8f4a36b94a0f22ae2f00a72534520f4ae6857c4b81f89/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:e8ac484bf18ce6975760921bb6148041faa8fef0547200386ea0b52b5d27bf7b", size = 211276, upload-time = "2026-04-02T09:27:16.834Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/fe/d17c32dc72e17e155e06883efa84514ca375f8a528ba2546bee73fc4df81/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:a5fe03b42827c13cdccd08e6c0247b6a6d4b5e3cdc53fd1749f5896adcdc2356", size = 201238, upload-time = "2026-04-02T09:27:18.229Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/29/f33daa50b06525a237451cdb6c69da366c381a3dadcd833fa5676bc468b3/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:2d6eb928e13016cea4f1f21d1e10c1cebd5a421bc57ddf5b1142ae3f86824fab", size = 230189, upload-time = "2026-04-02T09:27:19.445Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/6e/52c84015394a6a0bdcd435210a7e944c5f94ea1055f5cc5d56c5fe368e7b/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e74327fb75de8986940def6e8dee4f127cc9752bee7355bb323cc5b2659b6d46", size = 211352, upload-time = "2026-04-02T09:27:20.79Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/d7/4353be581b373033fb9198bf1da3cf8f09c1082561e8e922aa7b39bf9fe8/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:d6038d37043bced98a66e68d3aa2b6a35505dc01328cd65217cefe82f25def44", size = 227024, upload-time = "2026-04-02T09:27:22.063Z" },
+ { url = "https://files.pythonhosted.org/packages/30/45/99d18aa925bd1740098ccd3060e238e21115fffbfdcb8f3ece837d0ace6c/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:7579e913a5339fb8fa133f6bbcfd8e6749696206cf05acdbdca71a1b436d8e72", size = 217869, upload-time = "2026-04-02T09:27:23.486Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/05/5ee478aa53f4bb7996482153d4bfe1b89e0f087f0ab6b294fcf92d595873/charset_normalizer-3.4.7-cp314-cp314-win32.whl", hash = "sha256:5b77459df20e08151cd6f8b9ef8ef1f961ef73d85c21a555c7eed5b79410ec10", size = 148541, upload-time = "2026-04-02T09:27:25.146Z" },
+ { url = "https://files.pythonhosted.org/packages/48/77/72dcb0921b2ce86420b2d79d454c7022bf5be40202a2a07906b9f2a35c97/charset_normalizer-3.4.7-cp314-cp314-win_amd64.whl", hash = "sha256:92a0a01ead5e668468e952e4238cccd7c537364eb7d851ab144ab6627dbbe12f", size = 159634, upload-time = "2026-04-02T09:27:26.642Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/a3/c2369911cd72f02386e4e340770f6e158c7980267da16af8f668217abaa0/charset_normalizer-3.4.7-cp314-cp314-win_arm64.whl", hash = "sha256:67f6279d125ca0046a7fd386d01b311c6363844deac3e5b069b514ba3e63c246", size = 148384, upload-time = "2026-04-02T09:27:28.271Z" },
+ { url = "https://files.pythonhosted.org/packages/94/09/7e8a7f73d24dba1f0035fbbf014d2c36828fc1bf9c88f84093e57d315935/charset_normalizer-3.4.7-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:effc3f449787117233702311a1b7d8f59cba9ced946ba727bdc329ec69028e24", size = 330133, upload-time = "2026-04-02T09:27:29.474Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/da/96975ddb11f8e977f706f45cddd8540fd8242f71ecdb5d18a80723dcf62c/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fbccdc05410c9ee21bbf16a35f4c1d16123dcdeb8a1d38f33654fa21d0234f79", size = 216257, upload-time = "2026-04-02T09:27:30.793Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/e8/1d63bf8ef2d388e95c64b2098f45f84758f6d102a087552da1485912637b/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:733784b6d6def852c814bce5f318d25da2ee65dd4839a0718641c696e09a2960", size = 234851, upload-time = "2026-04-02T09:27:32.44Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/40/e5ff04233e70da2681fa43969ad6f66ca5611d7e669be0246c4c7aaf6dc8/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a89c23ef8d2c6b27fd200a42aa4ac72786e7c60d40efdc76e6011260b6e949c4", size = 233393, upload-time = "2026-04-02T09:27:34.03Z" },
+ { url = "https://files.pythonhosted.org/packages/be/c1/06c6c49d5a5450f76899992f1ee40b41d076aee9279b49cf9974d2f313d5/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6c114670c45346afedc0d947faf3c7f701051d2518b943679c8ff88befe14f8e", size = 223251, upload-time = "2026-04-02T09:27:35.369Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/9f/f2ff16fb050946169e3e1f82134d107e5d4ae72647ec8a1b1446c148480f/charset_normalizer-3.4.7-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:a180c5e59792af262bf263b21a3c49353f25945d8d9f70628e73de370d55e1e1", size = 206609, upload-time = "2026-04-02T09:27:36.661Z" },
+ { url = "https://files.pythonhosted.org/packages/69/d5/a527c0cd8d64d2eab7459784fb4169a0ac76e5a6fc5237337982fd61347e/charset_normalizer-3.4.7-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3c9a494bc5ec77d43cea229c4f6db1e4d8fe7e1bbffa8b6f0f0032430ff8ab44", size = 220014, upload-time = "2026-04-02T09:27:38.019Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/80/8a7b8104a3e203074dc9aa2c613d4b726c0e136bad1cc734594b02867972/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8d828b6667a32a728a1ad1d93957cdf37489c57b97ae6c4de2860fa749b8fc1e", size = 218979, upload-time = "2026-04-02T09:27:39.37Z" },
+ { url = "https://files.pythonhosted.org/packages/02/9a/b759b503d507f375b2b5c153e4d2ee0a75aa215b7f2489cf314f4541f2c0/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:cf1493cd8607bec4d8a7b9b004e699fcf8f9103a9284cc94962cb73d20f9d4a3", size = 209238, upload-time = "2026-04-02T09:27:40.722Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/4e/0f3f5d47b86bdb79256e7290b26ac847a2832d9a4033f7eb2cd4bcf4bb5b/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:0c96c3b819b5c3e9e165495db84d41914d6894d55181d2d108cc1a69bfc9cce0", size = 236110, upload-time = "2026-04-02T09:27:42.33Z" },
+ { url = "https://files.pythonhosted.org/packages/96/23/bce28734eb3ed2c91dcf93abeb8a5cf393a7b2749725030bb630e554fdd8/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:752a45dc4a6934060b3b0dab47e04edc3326575f82be64bc4fc293914566503e", size = 219824, upload-time = "2026-04-02T09:27:43.924Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/6f/6e897c6984cc4d41af319b077f2f600fc8214eb2fe2d6bcb79141b882400/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:8778f0c7a52e56f75d12dae53ae320fae900a8b9b4164b981b9c5ce059cd1fcb", size = 233103, upload-time = "2026-04-02T09:27:45.348Z" },
+ { url = "https://files.pythonhosted.org/packages/76/22/ef7bd0fe480a0ae9b656189ec00744b60933f68b4f42a7bb06589f6f576a/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ce3412fbe1e31eb81ea42f4169ed94861c56e643189e1e75f0041f3fe7020abe", size = 225194, upload-time = "2026-04-02T09:27:46.706Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/a7/0e0ab3e0b5bc1219bd80a6a0d4d72ca74d9250cb2382b7c699c147e06017/charset_normalizer-3.4.7-cp314-cp314t-win32.whl", hash = "sha256:c03a41a8784091e67a39648f70c5f97b5b6a37f216896d44d2cdcb82615339a0", size = 159827, upload-time = "2026-04-02T09:27:48.053Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/1d/29d32e0fb40864b1f878c7f5a0b343ae676c6e2b271a2d55cc3a152391da/charset_normalizer-3.4.7-cp314-cp314t-win_amd64.whl", hash = "sha256:03853ed82eeebbce3c2abfdbc98c96dc205f32a79627688ac9a27370ea61a49c", size = 174168, upload-time = "2026-04-02T09:27:49.795Z" },
+ { url = "https://files.pythonhosted.org/packages/de/32/d92444ad05c7a6e41fb2036749777c163baf7a0301a040cb672d6b2b1ae9/charset_normalizer-3.4.7-cp314-cp314t-win_arm64.whl", hash = "sha256:c35abb8bfff0185efac5878da64c45dafd2b37fb0383add1be155a763c1f083d", size = 153018, upload-time = "2026-04-02T09:27:51.116Z" },
+ { url = "https://files.pythonhosted.org/packages/db/8f/61959034484a4a7c527811f4721e75d02d653a35afb0b6054474d8185d4c/charset_normalizer-3.4.7-py3-none-any.whl", hash = "sha256:3dce51d0f5e7951f8bb4900c257dad282f49190fdbebecd4ba99bcc41fef404d", size = 61958, upload-time = "2026-04-02T09:28:37.794Z" },
+]
+
+[[package]]
+name = "click"
+version = "8.4.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/76/d4/81420972a676e8ffea40450d8c8c92943e7218a78fe9b64359836cc9876b/click-8.4.2.tar.gz", hash = "sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6", size = 338000, upload-time = "2026-06-24T17:45:15.148Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fb/e2/79c688af8b210d232694e31e59da9f6ec747bae31c3f5946e4e9b98860d5/click-8.4.2-py3-none-any.whl", hash = "sha256:e6f9f66136c816745b9d65817da91d61d957fb16e02e4dcd0552553c5a197b76", size = 119243, upload-time = "2026-06-24T17:45:13.73Z" },
+]
+
+[[package]]
+name = "colorama"
+version = "0.4.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" },
+]
+
+[[package]]
+name = "comm"
+version = "0.2.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/4c/13/7d740c5849255756bc17888787313b61fd38a0a8304fc4f073dfc46122aa/comm-0.2.3.tar.gz", hash = "sha256:2dc8048c10962d55d7ad693be1e7045d891b7ce8d999c97963a5e3e99c055971", size = 6319, upload-time = "2025-07-25T14:02:04.452Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/60/97/891a0971e1e4a8c5d2b20bbe0e524dc04548d2307fee33cdeba148fd4fc7/comm-0.2.3-py3-none-any.whl", hash = "sha256:c615d91d75f7f04f095b30d1c1711babd43bdc6419c1be9886a85f2f4e489417", size = 7294, upload-time = "2025-07-25T14:02:02.896Z" },
+]
+
+[[package]]
+name = "contourpy"
+version = "1.3.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/58/01/1253e6698a07380cd31a736d248a3f2a50a7c88779a1813da27503cadc2a/contourpy-1.3.3.tar.gz", hash = "sha256:083e12155b210502d0bca491432bb04d56dc3432f95a979b429f2848c3dbe880", size = 13466174, upload-time = "2025-07-26T12:03:12.549Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/be/45/adfee365d9ea3d853550b2e735f9d66366701c65db7855cd07621732ccfc/contourpy-1.3.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b08a32ea2f8e42cf1d4be3169a98dd4be32bafe4f22b6c4cb4ba810fa9e5d2cb", size = 293419, upload-time = "2025-07-26T12:01:21.16Z" },
+ { url = "https://files.pythonhosted.org/packages/53/3e/405b59cfa13021a56bba395a6b3aca8cec012b45bf177b0eaf7a202cde2c/contourpy-1.3.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:556dba8fb6f5d8742f2923fe9457dbdd51e1049c4a43fd3986a0b14a1d815fc6", size = 273979, upload-time = "2025-07-26T12:01:22.448Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/1c/a12359b9b2ca3a845e8f7f9ac08bdf776114eb931392fcad91743e2ea17b/contourpy-1.3.3-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:92d9abc807cf7d0e047b95ca5d957cf4792fcd04e920ca70d48add15c1a90ea7", size = 332653, upload-time = "2025-07-26T12:01:24.155Z" },
+ { url = "https://files.pythonhosted.org/packages/63/12/897aeebfb475b7748ea67b61e045accdfcf0d971f8a588b67108ed7f5512/contourpy-1.3.3-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b2e8faa0ed68cb29af51edd8e24798bb661eac3bd9f65420c1887b6ca89987c8", size = 379536, upload-time = "2025-07-26T12:01:25.91Z" },
+ { url = "https://files.pythonhosted.org/packages/43/8a/a8c584b82deb248930ce069e71576fc09bd7174bbd35183b7943fb1064fd/contourpy-1.3.3-cp312-cp312-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:626d60935cf668e70a5ce6ff184fd713e9683fb458898e4249b63be9e28286ea", size = 384397, upload-time = "2025-07-26T12:01:27.152Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/8f/ec6289987824b29529d0dfda0d74a07cec60e54b9c92f3c9da4c0ac732de/contourpy-1.3.3-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4d00e655fcef08aba35ec9610536bfe90267d7ab5ba944f7032549c55a146da1", size = 362601, upload-time = "2025-07-26T12:01:28.808Z" },
+ { url = "https://files.pythonhosted.org/packages/05/0a/a3fe3be3ee2dceb3e615ebb4df97ae6f3828aa915d3e10549ce016302bd1/contourpy-1.3.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:451e71b5a7d597379ef572de31eeb909a87246974d960049a9848c3bc6c41bf7", size = 1331288, upload-time = "2025-07-26T12:01:31.198Z" },
+ { url = "https://files.pythonhosted.org/packages/33/1d/acad9bd4e97f13f3e2b18a3977fe1b4a37ecf3d38d815333980c6c72e963/contourpy-1.3.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:459c1f020cd59fcfe6650180678a9993932d80d44ccde1fa1868977438f0b411", size = 1403386, upload-time = "2025-07-26T12:01:33.947Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/8f/5847f44a7fddf859704217a99a23a4f6417b10e5ab1256a179264561540e/contourpy-1.3.3-cp312-cp312-win32.whl", hash = "sha256:023b44101dfe49d7d53932be418477dba359649246075c996866106da069af69", size = 185018, upload-time = "2025-07-26T12:01:35.64Z" },
+ { url = "https://files.pythonhosted.org/packages/19/e8/6026ed58a64563186a9ee3f29f41261fd1828f527dd93d33b60feca63352/contourpy-1.3.3-cp312-cp312-win_amd64.whl", hash = "sha256:8153b8bfc11e1e4d75bcb0bff1db232f9e10b274e0929de9d608027e0d34ff8b", size = 226567, upload-time = "2025-07-26T12:01:36.804Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/e2/f05240d2c39a1ed228d8328a78b6f44cd695f7ef47beb3e684cf93604f86/contourpy-1.3.3-cp312-cp312-win_arm64.whl", hash = "sha256:07ce5ed73ecdc4a03ffe3e1b3e3c1166db35ae7584be76f65dbbe28a7791b0cc", size = 193655, upload-time = "2025-07-26T12:01:37.999Z" },
+ { url = "https://files.pythonhosted.org/packages/68/35/0167aad910bbdb9599272bd96d01a9ec6852f36b9455cf2ca67bd4cc2d23/contourpy-1.3.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:177fb367556747a686509d6fef71d221a4b198a3905fe824430e5ea0fda54eb5", size = 293257, upload-time = "2025-07-26T12:01:39.367Z" },
+ { url = "https://files.pythonhosted.org/packages/96/e4/7adcd9c8362745b2210728f209bfbcf7d91ba868a2c5f40d8b58f54c509b/contourpy-1.3.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d002b6f00d73d69333dac9d0b8d5e84d9724ff9ef044fd63c5986e62b7c9e1b1", size = 274034, upload-time = "2025-07-26T12:01:40.645Z" },
+ { url = "https://files.pythonhosted.org/packages/73/23/90e31ceeed1de63058a02cb04b12f2de4b40e3bef5e082a7c18d9c8ae281/contourpy-1.3.3-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:348ac1f5d4f1d66d3322420f01d42e43122f43616e0f194fc1c9f5d830c5b286", size = 334672, upload-time = "2025-07-26T12:01:41.942Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/93/b43d8acbe67392e659e1d984700e79eb67e2acb2bd7f62012b583a7f1b55/contourpy-1.3.3-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:655456777ff65c2c548b7c454af9c6f33f16c8884f11083244b5819cc214f1b5", size = 381234, upload-time = "2025-07-26T12:01:43.499Z" },
+ { url = "https://files.pythonhosted.org/packages/46/3b/bec82a3ea06f66711520f75a40c8fc0b113b2a75edb36aa633eb11c4f50f/contourpy-1.3.3-cp313-cp313-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:644a6853d15b2512d67881586bd03f462c7ab755db95f16f14d7e238f2852c67", size = 385169, upload-time = "2025-07-26T12:01:45.219Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/32/e0f13a1c5b0f8572d0ec6ae2f6c677b7991fafd95da523159c19eff0696a/contourpy-1.3.3-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4debd64f124ca62069f313a9cb86656ff087786016d76927ae2cf37846b006c9", size = 362859, upload-time = "2025-07-26T12:01:46.519Z" },
+ { url = "https://files.pythonhosted.org/packages/33/71/e2a7945b7de4e58af42d708a219f3b2f4cff7386e6b6ab0a0fa0033c49a9/contourpy-1.3.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a15459b0f4615b00bbd1e91f1b9e19b7e63aea7483d03d804186f278c0af2659", size = 1332062, upload-time = "2025-07-26T12:01:48.964Z" },
+ { url = "https://files.pythonhosted.org/packages/12/fc/4e87ac754220ccc0e807284f88e943d6d43b43843614f0a8afa469801db0/contourpy-1.3.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ca0fdcd73925568ca027e0b17ab07aad764be4706d0a925b89227e447d9737b7", size = 1403932, upload-time = "2025-07-26T12:01:51.979Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/2e/adc197a37443f934594112222ac1aa7dc9a98faf9c3842884df9a9d8751d/contourpy-1.3.3-cp313-cp313-win32.whl", hash = "sha256:b20c7c9a3bf701366556e1b1984ed2d0cedf999903c51311417cf5f591d8c78d", size = 185024, upload-time = "2025-07-26T12:01:53.245Z" },
+ { url = "https://files.pythonhosted.org/packages/18/0b/0098c214843213759692cc638fce7de5c289200a830e5035d1791d7a2338/contourpy-1.3.3-cp313-cp313-win_amd64.whl", hash = "sha256:1cadd8b8969f060ba45ed7c1b714fe69185812ab43bd6b86a9123fe8f99c3263", size = 226578, upload-time = "2025-07-26T12:01:54.422Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/9a/2f6024a0c5995243cd63afdeb3651c984f0d2bc727fd98066d40e141ad73/contourpy-1.3.3-cp313-cp313-win_arm64.whl", hash = "sha256:fd914713266421b7536de2bfa8181aa8c699432b6763a0ea64195ebe28bff6a9", size = 193524, upload-time = "2025-07-26T12:01:55.73Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/b3/f8a1a86bd3298513f500e5b1f5fd92b69896449f6cab6a146a5d52715479/contourpy-1.3.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:88df9880d507169449d434c293467418b9f6cbe82edd19284aa0409e7fdb933d", size = 306730, upload-time = "2025-07-26T12:01:57.051Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/11/4780db94ae62fc0c2053909b65dc3246bd7cecfc4f8a20d957ad43aa4ad8/contourpy-1.3.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:d06bb1f751ba5d417047db62bca3c8fde202b8c11fb50742ab3ab962c81e8216", size = 287897, upload-time = "2025-07-26T12:01:58.663Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/15/e59f5f3ffdd6f3d4daa3e47114c53daabcb18574a26c21f03dc9e4e42ff0/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e4e6b05a45525357e382909a4c1600444e2a45b4795163d3b22669285591c1ae", size = 326751, upload-time = "2025-07-26T12:02:00.343Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/81/03b45cfad088e4770b1dcf72ea78d3802d04200009fb364d18a493857210/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ab3074b48c4e2cf1a960e6bbeb7f04566bf36b1861d5c9d4d8ac04b82e38ba20", size = 375486, upload-time = "2025-07-26T12:02:02.128Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/ba/49923366492ffbdd4486e970d421b289a670ae8cf539c1ea9a09822b371a/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:6c3d53c796f8647d6deb1abe867daeb66dcc8a97e8455efa729516b997b8ed99", size = 388106, upload-time = "2025-07-26T12:02:03.615Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/52/5b00ea89525f8f143651f9f03a0df371d3cbd2fccd21ca9b768c7a6500c2/contourpy-1.3.3-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:50ed930df7289ff2a8d7afeb9603f8289e5704755c7e5c3bbd929c90c817164b", size = 352548, upload-time = "2025-07-26T12:02:05.165Z" },
+ { url = "https://files.pythonhosted.org/packages/32/1d/a209ec1a3a3452d490f6b14dd92e72280c99ae3d1e73da74f8277d4ee08f/contourpy-1.3.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:4feffb6537d64b84877da813a5c30f1422ea5739566abf0bd18065ac040e120a", size = 1322297, upload-time = "2025-07-26T12:02:07.379Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/9e/46f0e8ebdd884ca0e8877e46a3f4e633f6c9c8c4f3f6e72be3fe075994aa/contourpy-1.3.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:2b7e9480ffe2b0cd2e787e4df64270e3a0440d9db8dc823312e2c940c167df7e", size = 1391023, upload-time = "2025-07-26T12:02:10.171Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/70/f308384a3ae9cd2209e0849f33c913f658d3326900d0ff5d378d6a1422d2/contourpy-1.3.3-cp313-cp313t-win32.whl", hash = "sha256:283edd842a01e3dcd435b1c5116798d661378d83d36d337b8dde1d16a5fc9ba3", size = 196157, upload-time = "2025-07-26T12:02:11.488Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/dd/880f890a6663b84d9e34a6f88cded89d78f0091e0045a284427cb6b18521/contourpy-1.3.3-cp313-cp313t-win_amd64.whl", hash = "sha256:87acf5963fc2b34825e5b6b048f40e3635dd547f590b04d2ab317c2619ef7ae8", size = 240570, upload-time = "2025-07-26T12:02:12.754Z" },
+ { url = "https://files.pythonhosted.org/packages/80/99/2adc7d8ffead633234817ef8e9a87115c8a11927a94478f6bb3d3f4d4f7d/contourpy-1.3.3-cp313-cp313t-win_arm64.whl", hash = "sha256:3c30273eb2a55024ff31ba7d052dde990d7d8e5450f4bbb6e913558b3d6c2301", size = 199713, upload-time = "2025-07-26T12:02:14.4Z" },
+ { url = "https://files.pythonhosted.org/packages/72/8b/4546f3ab60f78c514ffb7d01a0bd743f90de36f0019d1be84d0a708a580a/contourpy-1.3.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:fde6c716d51c04b1c25d0b90364d0be954624a0ee9d60e23e850e8d48353d07a", size = 292189, upload-time = "2025-07-26T12:02:16.095Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/e1/3542a9cb596cadd76fcef413f19c79216e002623158befe6daa03dbfa88c/contourpy-1.3.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:cbedb772ed74ff5be440fa8eee9bd49f64f6e3fc09436d9c7d8f1c287b121d77", size = 273251, upload-time = "2025-07-26T12:02:17.524Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/71/f93e1e9471d189f79d0ce2497007731c1e6bf9ef6d1d61b911430c3db4e5/contourpy-1.3.3-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:22e9b1bd7a9b1d652cd77388465dc358dafcd2e217d35552424aa4f996f524f5", size = 335810, upload-time = "2025-07-26T12:02:18.9Z" },
+ { url = "https://files.pythonhosted.org/packages/91/f9/e35f4c1c93f9275d4e38681a80506b5510e9327350c51f8d4a5a724d178c/contourpy-1.3.3-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a22738912262aa3e254e4f3cb079a95a67132fc5a063890e224393596902f5a4", size = 382871, upload-time = "2025-07-26T12:02:20.418Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/71/47b512f936f66a0a900d81c396a7e60d73419868fba959c61efed7a8ab46/contourpy-1.3.3-cp314-cp314-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:afe5a512f31ee6bd7d0dda52ec9864c984ca3d66664444f2d72e0dc4eb832e36", size = 386264, upload-time = "2025-07-26T12:02:21.916Z" },
+ { url = "https://files.pythonhosted.org/packages/04/5f/9ff93450ba96b09c7c2b3f81c94de31c89f92292f1380261bd7195bea4ea/contourpy-1.3.3-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f64836de09927cba6f79dcd00fdd7d5329f3fccc633468507079c829ca4db4e3", size = 363819, upload-time = "2025-07-26T12:02:23.759Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/a6/0b185d4cc480ee494945cde102cb0149ae830b5fa17bf855b95f2e70ad13/contourpy-1.3.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:1fd43c3be4c8e5fd6e4f2baeae35ae18176cf2e5cced681cca908addf1cdd53b", size = 1333650, upload-time = "2025-07-26T12:02:26.181Z" },
+ { url = "https://files.pythonhosted.org/packages/43/d7/afdc95580ca56f30fbcd3060250f66cedbde69b4547028863abd8aa3b47e/contourpy-1.3.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6afc576f7b33cf00996e5c1102dc2a8f7cc89e39c0b55df93a0b78c1bd992b36", size = 1404833, upload-time = "2025-07-26T12:02:28.782Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/e2/366af18a6d386f41132a48f033cbd2102e9b0cf6345d35ff0826cd984566/contourpy-1.3.3-cp314-cp314-win32.whl", hash = "sha256:66c8a43a4f7b8df8b71ee1840e4211a3c8d93b214b213f590e18a1beca458f7d", size = 189692, upload-time = "2025-07-26T12:02:30.128Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/c2/57f54b03d0f22d4044b8afb9ca0e184f8b1afd57b4f735c2fa70883dc601/contourpy-1.3.3-cp314-cp314-win_amd64.whl", hash = "sha256:cf9022ef053f2694e31d630feaacb21ea24224be1c3ad0520b13d844274614fd", size = 232424, upload-time = "2025-07-26T12:02:31.395Z" },
+ { url = "https://files.pythonhosted.org/packages/18/79/a9416650df9b525737ab521aa181ccc42d56016d2123ddcb7b58e926a42c/contourpy-1.3.3-cp314-cp314-win_arm64.whl", hash = "sha256:95b181891b4c71de4bb404c6621e7e2390745f887f2a026b2d99e92c17892339", size = 198300, upload-time = "2025-07-26T12:02:32.956Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/42/38c159a7d0f2b7b9c04c64ab317042bb6952b713ba875c1681529a2932fe/contourpy-1.3.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:33c82d0138c0a062380332c861387650c82e4cf1747aaa6938b9b6516762e772", size = 306769, upload-time = "2025-07-26T12:02:34.2Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/6c/26a8205f24bca10974e77460de68d3d7c63e282e23782f1239f226fcae6f/contourpy-1.3.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ea37e7b45949df430fe649e5de8351c423430046a2af20b1c1961cae3afcda77", size = 287892, upload-time = "2025-07-26T12:02:35.807Z" },
+ { url = "https://files.pythonhosted.org/packages/66/06/8a475c8ab718ebfd7925661747dbb3c3ee9c82ac834ccb3570be49d129f4/contourpy-1.3.3-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d304906ecc71672e9c89e87c4675dc5c2645e1f4269a5063b99b0bb29f232d13", size = 326748, upload-time = "2025-07-26T12:02:37.193Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/a3/c5ca9f010a44c223f098fccd8b158bb1cb287378a31ac141f04730dc49be/contourpy-1.3.3-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ca658cd1a680a5c9ea96dc61cdbae1e85c8f25849843aa799dfd3cb370ad4fbe", size = 375554, upload-time = "2025-07-26T12:02:38.894Z" },
+ { url = "https://files.pythonhosted.org/packages/80/5b/68bd33ae63fac658a4145088c1e894405e07584a316738710b636c6d0333/contourpy-1.3.3-cp314-cp314t-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ab2fd90904c503739a75b7c8c5c01160130ba67944a7b77bbf36ef8054576e7f", size = 388118, upload-time = "2025-07-26T12:02:40.642Z" },
+ { url = "https://files.pythonhosted.org/packages/40/52/4c285a6435940ae25d7410a6c36bda5145839bc3f0beb20c707cda18b9d2/contourpy-1.3.3-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b7301b89040075c30e5768810bc96a8e8d78085b47d8be6e4c3f5a0b4ed478a0", size = 352555, upload-time = "2025-07-26T12:02:42.25Z" },
+ { url = "https://files.pythonhosted.org/packages/24/ee/3e81e1dd174f5c7fefe50e85d0892de05ca4e26ef1c9a59c2a57e43b865a/contourpy-1.3.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:2a2a8b627d5cc6b7c41a4beff6c5ad5eb848c88255fda4a8745f7e901b32d8e4", size = 1322295, upload-time = "2025-07-26T12:02:44.668Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/b2/6d913d4d04e14379de429057cd169e5e00f6c2af3bb13e1710bcbdb5da12/contourpy-1.3.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:fd6ec6be509c787f1caf6b247f0b1ca598bef13f4ddeaa126b7658215529ba0f", size = 1391027, upload-time = "2025-07-26T12:02:47.09Z" },
+ { url = "https://files.pythonhosted.org/packages/93/8a/68a4ec5c55a2971213d29a9374913f7e9f18581945a7a31d1a39b5d2dfe5/contourpy-1.3.3-cp314-cp314t-win32.whl", hash = "sha256:e74a9a0f5e3fff48fb5a7f2fd2b9b70a3fe014a67522f79b7cca4c0c7e43c9ae", size = 202428, upload-time = "2025-07-26T12:02:48.691Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/96/fd9f641ffedc4fa3ace923af73b9d07e869496c9cc7a459103e6e978992f/contourpy-1.3.3-cp314-cp314t-win_amd64.whl", hash = "sha256:13b68d6a62db8eafaebb8039218921399baf6e47bf85006fd8529f2a08ef33fc", size = 250331, upload-time = "2025-07-26T12:02:50.137Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/8c/469afb6465b853afff216f9528ffda78a915ff880ed58813ba4faf4ba0b6/contourpy-1.3.3-cp314-cp314t-win_arm64.whl", hash = "sha256:b7448cb5a725bb1e35ce88771b86fba35ef418952474492cf7c764059933ff8b", size = 203831, upload-time = "2025-07-26T12:02:51.449Z" },
+]
+
+[[package]]
+name = "coverage"
+version = "7.14.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b4/91/0a7c28934e50d8ac9a7b117712d176f2953c3170bccced5eaacfa3e96175/coverage-7.14.3.tar.gz", hash = "sha256:1a7563a443f3d53fdeb040ec8c9f7466aed7ca3dc5891aa09d3ca3625fa4387f", size = 924398, upload-time = "2026-06-22T23:10:25.584Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/bd/b0/8a911f6ffe6974dac4df95b468ab9a2899d0e59f0f99a489afeec39f00bc/coverage-7.14.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3d74ff26299c4879ce3a4d826f9d3d4d556fd285fde7bbce3c0ef5a8ab1cec24", size = 220672, upload-time = "2026-06-22T23:08:26.621Z" },
+ { url = "https://files.pythonhosted.org/packages/36/16/0fc0cb52538783dbbae0934b834f5a58fd5354380ee6cad4a07b15dc845d/coverage-7.14.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:96150a9cf3468ea20f0bc5d0e21b3df8972c31480ef90fa7614b773cc6429665", size = 221035, upload-time = "2026-06-22T23:08:28.372Z" },
+ { url = "https://files.pythonhosted.org/packages/77/e2/421ccfbb48335ac49e93301478cf5d623b0c2bf1c0cadd8e2b2fc6c0c710/coverage-7.14.3-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:27d07a46500ba23515b838dbcf52512026af04090755cf6cc64166d88c9b9a1a", size = 252540, upload-time = "2026-06-22T23:08:30.226Z" },
+ { url = "https://files.pythonhosted.org/packages/06/c2/05b8c890097c61a7f4406b35396b997a635200ded0339eda83dfbe526c5f/coverage-7.14.3-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:621e13c6108234d7960aaf5762ab5c3c00f33c30c15af06dcbff0c73bf112727", size = 255274, upload-time = "2026-06-22T23:08:31.876Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/be/b6d9efe447f8ba3c3c854195f326bd64c54b907d936cd2fdebf8767ec72e/coverage-7.14.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4b60ca6d8af70473491a15a343cbabab2e8f9ea66a4376e81c7aa24876a6f977", size = 256389, upload-time = "2026-06-22T23:08:33.843Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/3c/f26e50acc429e608bc534ac06f0a3c169019c798178ec5e9de3dbc0df9c9/coverage-7.14.3-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c90a7cdd5e380e1ce02f19792e2ac2fbfbf177e35a27e69fd3e873b30d895c0c", size = 258648, upload-time = "2026-06-22T23:08:35.481Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/a2/01c1fabf816c8e1dae197e258edf878a3d3ddc86fbda34b76e5794277d8f/coverage-7.14.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5d788e5fd55347eef06ca0732c77d04a264de67e8ff24631270cdff3767a60cf", size = 252949, upload-time = "2026-06-22T23:08:37.562Z" },
+ { url = "https://files.pythonhosted.org/packages/89/c6/941166dd79c31fd44a13063780ae8d552eee0089a0a0930b9bdb7df554ed/coverage-7.14.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:62c7f79db2851c95ef020e5d28b97afde3daf9f7febcd35b53e05638f729063f", size = 254310, upload-time = "2026-06-22T23:08:39.174Z" },
+ { url = "https://files.pythonhosted.org/packages/10/31/80b1fd028201a961033ce95be3cd1e39e521b3762e6b4a1ac1616cb291e7/coverage-7.14.3-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:90f7608aeb5d9b60b523b9fb2a4ee1973867cc4865a3f26fe6c7577073b70205", size = 252453, upload-time = "2026-06-22T23:08:40.84Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/85/c3d9addd94c4b524f3f4af0232075f5fe7170ce99a1386edff803e5934db/coverage-7.14.3-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1e3b91f9c4740aeb571ecf82e5e8d8e4ab62d34fcb5a5d4e5baa38c6f7d2857c", size = 256522, upload-time = "2026-06-22T23:08:42.494Z" },
+ { url = "https://files.pythonhosted.org/packages/91/14/e5a0575f73795af3a7a9ae13dadf812e17d32422896839987dc3f86947e1/coverage-7.14.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:c946099774a7699de03cbd0ff0a64e21aed4525eed9d959adde4afe6d15758ef", size = 252023, upload-time = "2026-06-22T23:08:44.243Z" },
+ { url = "https://files.pythonhosted.org/packages/38/9b/9652ee531937ce3b8a63a8896885b2b4a2d56adc30e53c9540c666286d88/coverage-7.14.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:16b206e521feb8b7133a45754643dead0538489cf8b783b90cf5f4e3299625fd", size = 253893, upload-time = "2026-06-22T23:08:46.113Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/05/42678841c8c38e4b08bdfc48269f5a16dfbf5806000fe6a89b4cece3c691/coverage-7.14.3-cp312-cp312-win32.whl", hash = "sha256:ea3169c7116eb6cdf7608c6c7da9ecfcb3da40688e3a510fac2d1d2bafd6dc35", size = 222734, upload-time = "2026-06-22T23:08:47.858Z" },
+ { url = "https://files.pythonhosted.org/packages/df/87/07a4fcee55177a25f1b52331a8e92cf4f2c53b1a9c75ce2981fd59c684ad/coverage-7.14.3-cp312-cp312-win_amd64.whl", hash = "sha256:7ea52fc08f007bcc494d4bb3df3851e95843d881860ba38fe2c64dc100db5e7d", size = 223266, upload-time = "2026-06-22T23:08:49.494Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/34/2b8b66a989282ea7b370beb49f50bab29470dc30bb0b03935b6b802782f7/coverage-7.14.3-cp312-cp312-win_arm64.whl", hash = "sha256:8cec0ad652ec57790970d817490105bd917d783c2f7b38d6b58a0ca312e1a336", size = 222655, upload-time = "2026-06-22T23:08:51.766Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/83/7fefbf5df23ed2b7f489907564a7b34b9b07098128e12e0fdfa92626e456/coverage-7.14.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:47968988b367990ae4ab17523790c38cd125e02c6bfd379b6022be2d40bdc38c", size = 220699, upload-time = "2026-06-22T23:08:53.522Z" },
+ { url = "https://files.pythonhosted.org/packages/31/e6/38c3653ff6d56d704b29241362387ca824e38e15b76fdcb7096538195790/coverage-7.14.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0ee68f5c34812780f3a7063382c0a9fcbb99985b7ddcdcaa626e4f3fb2e0783a", size = 221068, upload-time = "2026-06-22T23:08:55.571Z" },
+ { url = "https://files.pythonhosted.org/packages/20/86/4f5c45d51c5cd10a128933f0fd235393c9146abbfd2ce2dfa68b3267ead3/coverage-7.14.3-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:fa9e5c6857a7e80fa22ace5cf3550ae392bbfc322f1d8dd2d2d5a8be38cec027", size = 252060, upload-time = "2026-06-22T23:08:57.464Z" },
+ { url = "https://files.pythonhosted.org/packages/82/50/dfce42eff2cecabcd5a9bbad5489449c87db3415f408d23ffee417ce01f6/coverage-7.14.3-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:98a0859b0e98e43e1178a9402e19c8127766b14f7109a374d976e5a62c0e5c73", size = 254657, upload-time = "2026-06-22T23:08:59.453Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/d2/639ceb1bc8038fd0d66768278d5dc22df3391918b8278c2a21aa2602a531/coverage-7.14.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:69918344541ed9c8368566c2adc03c0e33d4550d7faa87d1b35e49b6a3286ea9", size = 255892, upload-time = "2026-06-22T23:09:01.291Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/96/002094a10e113512500dc1e10430a449417e17b0f90f7d496bcb820208b7/coverage-7.14.3-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b7f300ac92cd4b570724c8ffbbd0c130fee298d2447f41d5a3abf58976fae1de", size = 258026, upload-time = "2026-06-22T23:09:03.017Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/ec/286a5d2fad9c4bee59bd724feeb7d5bf8303c6c9200b51d1dd945a9c72b0/coverage-7.14.3-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:11a7ec9f97ab950f4c5af62229befc7faf208fdbc0116d3902d7e306cf2c5abd", size = 252285, upload-time = "2026-06-22T23:09:04.773Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/7d/a17753a0b12dd48d0d50f5fab079ad99d3be1eac790494d89f3a417ca0b9/coverage-7.14.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a571bd889cd36c5922ce8e42e059f9d37d02301531d11374afa4c87a578625d5", size = 254023, upload-time = "2026-06-22T23:09:06.513Z" },
+ { url = "https://files.pythonhosted.org/packages/86/ef/a76c6ceba6a2c313f905310abf2701d534cada22d372db11731831e9e209/coverage-7.14.3-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:de76caefc8deabb0dd1678b6a980be97d14c8d87e213ac194dbf8b09e96d63fb", size = 251989, upload-time = "2026-06-22T23:09:08.382Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/39/353013a75fec0fb49f7553519f9d52b4441e902e5178c93f38eb6c07cedb/coverage-7.14.3-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:d20a15c622194234161535459affa8f7905830391c9ccfa060d495dbfe3a1c7f", size = 256144, upload-time = "2026-06-22T23:09:10.369Z" },
+ { url = "https://files.pythonhosted.org/packages/29/0e/613878555d734def11c5b20a2701a15cb3781b9e9ea749da27c5f436e928/coverage-7.14.3-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:b488bd4b23397db62e7a9459129d01ff06a846582a732efd24834b24a6ada498", size = 251808, upload-time = "2026-06-22T23:09:12.057Z" },
+ { url = "https://files.pythonhosted.org/packages/af/76/359c058c9cfdcf1e8b107663881225b03b364a320017eda24a2a66e55102/coverage-7.14.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6a3693b4153394d265f44fb855fdc80e72403024d4d6f91c4871b334d028e4e0", size = 253579, upload-time = "2026-06-22T23:09:13.858Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/d9/4ba2f060933a30ebe363cef9f67a365b0a317e580c0d5d9169d56a73ef1c/coverage-7.14.3-cp313-cp313-win32.whl", hash = "sha256:338b19131ab1a6b767b462bfcbaa692e7ae22f24463e39d49b02a83410ff6b37", size = 222741, upload-time = "2026-06-22T23:09:15.636Z" },
+ { url = "https://files.pythonhosted.org/packages/76/e8/196ebc25d8f34c06d43a6e9c8513c9266ef8dbf3b5672beb1a00cf5e29fa/coverage-7.14.3-cp313-cp313-win_amd64.whl", hash = "sha256:b3d77f7f196abdef7e01415de1bce09f216189e83e58159cfeef2b92d0464994", size = 223283, upload-time = "2026-06-22T23:09:17.478Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/af/51d2aac6417523a286f10fb25f09eb9518a84df9f1151e93ff6871f34849/coverage-7.14.3-cp313-cp313-win_arm64.whl", hash = "sha256:e6230e688c7c3e65cedd41a774eb4ec221adc6bfee13768231015b702d5e4150", size = 222678, upload-time = "2026-06-22T23:09:19.7Z" },
+ { url = "https://files.pythonhosted.org/packages/61/56/14e3b97facbfa1304dd19e676e26599ad359f04714bed32f7f1c5a88efdc/coverage-7.14.3-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:605ab2b566a22bd94834529d66d295c364aba84afd3e5498285c7a524017b1fc", size = 220741, upload-time = "2026-06-22T23:09:21.616Z" },
+ { url = "https://files.pythonhosted.org/packages/12/1d/db378b5cca433b90b893f26dab728b280ddd89f272a1fdfed4aeaa05c686/coverage-7.14.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:a3c2134809e80fac091bfed18a6991b5a5eb5df5ae32b17ac4f4f99864b73dd7", size = 221068, upload-time = "2026-06-22T23:09:23.452Z" },
+ { url = "https://files.pythonhosted.org/packages/47/f0/3f8421b20d9c4fcd39be9a8ca3c3fda8bc204b44efbd09fede153afd3e2f/coverage-7.14.3-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:c02efd507227bde9969cab0db8f48890eb3b5dcad6afac57a4792df4133543ce", size = 252117, upload-time = "2026-06-22T23:09:25.458Z" },
+ { url = "https://files.pythonhosted.org/packages/27/ca/59ea35fb99743549ec8b37eff141ece4431fea590c89e536ed8032ef45cf/coverage-7.14.3-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:1bb93c2aa61d2a5b38f1526546d95cf4132cb681e541a337bf8dfd092be816e5", size = 254622, upload-time = "2026-06-22T23:09:27.523Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/25/ec6de51ae7493b92a1cf74d1b763121c29636759167e2a593ba4db5881e4/coverage-7.14.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f502e948e03e866538048bba081c075caaa62e5bda6ea5b7432e45f587eb462a", size = 255968, upload-time = "2026-06-22T23:09:29.43Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/05/c8bfc77823f42b4664fb25842f13b567022f6f84a4c83c8ecbb16734b7cb/coverage-7.14.3-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9973ef2463f8e6cfb61a6324126bb3e17d67a85f22f58d856e583ea2e3ca6501", size = 258284, upload-time = "2026-06-22T23:09:31.397Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/15/1d1b242027124a32b26ef01f82018b8c4ef34ef174aa6aeba7b1eeef48e8/coverage-7.14.3-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9be4e7d4c5ca0427889f8f9d614bd630c2be741b1de7699bca3b2b6c0e41003e", size = 252143, upload-time = "2026-06-22T23:09:33.256Z" },
+ { url = "https://files.pythonhosted.org/packages/74/b6/d2a9842fd2a5d7d27f1ac851c043a734a494ad75402c5331db3da79ed691/coverage-7.14.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a574912f3bde4b0619f6e97d01aa590b70998859244793769eb3a6df78ee56d3", size = 253976, upload-time = "2026-06-22T23:09:35.351Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/30/e1600ddf7e226db5558bb5323d2186fff00f505c4b764643ec89ce5d8175/coverage-7.14.3-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:e343fb086c9cd780b38622fea7c369acd64c1a0724312149b5d769c387a2b1f5", size = 251942, upload-time = "2026-06-22T23:09:37.313Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/2c/9159de64f9dd648e324328d588a44cfab1e331eb5259ce1141afe2a92dfb/coverage-7.14.3-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:3c68df8e61f1e09633fefc7538297145623957a048534368c9d212782aa5e845", size = 256220, upload-time = "2026-06-22T23:09:39.165Z" },
+ { url = "https://files.pythonhosted.org/packages/91/67/b7f536cc2c124f48e91b22fbb741d2261f4e3d310faf6f76007f47566e5d/coverage-7.14.3-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:3e5b550a128419373c2f6cec28a244207013ef15f5cbcff6a5ca09d1dfaaf027", size = 251756, upload-time = "2026-06-22T23:09:41.056Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/ec/f3718038e2d4860c715a55428377ca7f6c75872caf98cabd982e1d76967d/coverage-7.14.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2bfc4dd0a912329eccc7484a7d0b2a38032b38c40663b1e1ac595f10c457954b", size = 253413, upload-time = "2026-06-22T23:09:43.306Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/a5/91f11efeef89b3cc9b30461128db15b0511ef813ab889a7b7ab636b3a497/coverage-7.14.3-cp314-cp314-win32.whl", hash = "sha256:0423d64c013057a06e70f070f073cec4b0cbc7d2b27f3c7007292f2ff1d52965", size = 222946, upload-time = "2026-06-22T23:09:45.261Z" },
+ { url = "https://files.pythonhosted.org/packages/58/fd/98ac9f524d9ec378de831c034dbdeb544ca7ef7d2d9c9996daf232a037fd/coverage-7.14.3-cp314-cp314-win_amd64.whl", hash = "sha256:92c22e19ce64ca3f2ad751f16f14df1468b4c231bd6af97185063a9c292a0cb3", size = 223436, upload-time = "2026-06-22T23:09:47.177Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/a0/7cd612d650a772a0ae80144443406bf61981c896c3d57c9e6e79fb2cdbd1/coverage-7.14.3-cp314-cp314-win_arm64.whl", hash = "sha256:41de778bd41780586e2b04912079c73089ab5d839624e28db3bdb26de638da92", size = 222861, upload-time = "2026-06-22T23:09:49.384Z" },
+ { url = "https://files.pythonhosted.org/packages/55/57/017353fab573779c0d00448e47d102edd36c792f7b6f233a4d89a7a08384/coverage-7.14.3-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:8427f370ca67db4c975d2a26acfc0e5783ca0b52444dbc50278ace0f35445949", size = 221474, upload-time = "2026-06-22T23:09:51.417Z" },
+ { url = "https://files.pythonhosted.org/packages/69/92/90cf1f1a5c468a9c1b7ba2716e0e205293ad9b02f5f573a6de4318b15ba1/coverage-7.14.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:d8e88f335544a47e22ae2e45b344772925ec65166555c958720d5ed971880891", size = 221738, upload-time = "2026-06-22T23:09:53.487Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/c0/4df964fa539f8399fd7679c09c472d73744de334686fd3f01e3a2465ce4e/coverage-7.14.3-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:beaab199b9e5ceaf5a225e16a9d4df136f2a1eae0a5c20de1e277c8a5225f388", size = 263101, upload-time = "2026-06-22T23:09:55.895Z" },
+ { url = "https://files.pythonhosted.org/packages/06/76/e5d33b2576ae3bf2be2058cd1cae57774b61e400f2c3c58f3783dc2ffb4a/coverage-7.14.3-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b3ff255799f5a1676c71c1c32ec01fd043aa09d57b3d95764b24992757184784", size = 265225, upload-time = "2026-06-22T23:09:57.904Z" },
+ { url = "https://files.pythonhosted.org/packages/61/d2/e52419afe391a39ba27fdefaf0737d8e34bf03faef6ab3b3006545bbd0d0/coverage-7.14.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:878832eaac515b62decfa76965aed558775f86bf1fc8cca76993c0c84ae31aed", size = 267643, upload-time = "2026-06-22T23:09:59.938Z" },
+ { url = "https://files.pythonhosted.org/packages/58/7a/f2625d8d5006b6b20fba5afaef00b24a763fe96476ea798a3076cbc1f84e/coverage-7.14.3-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:611e62cb9386096d81b63e0a05330750268617231e7bd598e1fe77482a2c58a5", size = 268762, upload-time = "2026-06-22T23:10:01.943Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/bf/908024006bba57127354d74e938954b9c3cd765cc2e0412dc9c37b415cda/coverage-7.14.3-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:02c41de2a88011b893050fc9830267d927a50a215f7ad5ec17349db7090ccf26", size = 262208, upload-time = "2026-06-22T23:10:03.954Z" },
+ { url = "https://files.pythonhosted.org/packages/34/a0/d4f9296441b909817442fdb26bd77a698f08272ec683a7394b00eb2e47a0/coverage-7.14.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:526ce9721116af23b1065089f0b75046fe521e7772ab94b641cd66b7a0421889", size = 265096, upload-time = "2026-06-22T23:10:05.936Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/da/4ae4f3f4e477b56a4ce1e5c48a35eff38a94b50130ce5bdc897024741cfc/coverage-7.14.3-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:e4ed44705ca4bead6fc977a8b741f2145608289b33c8a9b42a95d0f15aedbf4d", size = 262699, upload-time = "2026-06-22T23:10:07.973Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/7a/6927148073ff32856d78baa77b4ddc07a9be7e90020f9db0661c4ca523a1/coverage-7.14.3-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:2415902f385a23dcc4ccd26e0ba803249a169af6a930c003a4c715eeb9a5444e", size = 266433, upload-time = "2026-06-22T23:10:10.145Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/a7/774f658dbe9c4c3f5daa86a87e0459ac3832e4e3cc67affe078547f727b9/coverage-7.14.3-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:b75ee850fc2d7c831e883220c445b035f2224de2ba6103f1e56dbd237ab913f7", size = 261547, upload-time = "2026-06-22T23:10:12.191Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/14/a0c18c0376c43cbf973f43ef6ca20019c950597180e6396232f7b6a27102/coverage-7.14.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:dc9b4e35e7c3920e925ba7f14886fd5fbe481232754624e832ddba66c7535635", size = 263859, upload-time = "2026-06-22T23:10:14.492Z" },
+ { url = "https://files.pythonhosted.org/packages/10/ac/43a3d0f460af524b131a6191805bc5d18b806ab4e828fbf82e8c8c3af446/coverage-7.14.3-cp314-cp314t-win32.whl", hash = "sha256:7b27c822a8161afbe48e99f1adfb098d270ae7e0f7d7b0555ce110529bdb69cc", size = 223250, upload-time = "2026-06-22T23:10:16.758Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/5f/d5e5c56b0712e96ce8f69fe7dbf229ff938b437bc50862743c8a0d2cea84/coverage-7.14.3-cp314-cp314t-win_amd64.whl", hash = "sha256:39e1dbbb6ff2c338e0196a482558a792a1de3aa64261196f5cdb3da016ad9cda", size = 224082, upload-time = "2026-06-22T23:10:19.23Z" },
+ { url = "https://files.pythonhosted.org/packages/62/35/947cbd5be1d3bcbbdc43d6791de8a56c6501903311d42915ae06a82815f0/coverage-7.14.3-cp314-cp314t-win_arm64.whl", hash = "sha256:68520c90babfa2d560eca6d497921ed3a4f469623bd709733124491b2aa8ef3f", size = 223400, upload-time = "2026-06-22T23:10:21.24Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/e3/a0aa32bfa3a081951f60a23bc0e7b512891ef0eecda1153cf1d8ba36c6b1/coverage-7.14.3-py3-none-any.whl", hash = "sha256:fb7e18afb6e903c1a92401a2f0501ac277dca527bb9ca6fe1f691a8a0026a0e8", size = 212469, upload-time = "2026-06-22T23:10:23.405Z" },
+]
+
+[[package]]
+name = "cycler"
+version = "0.12.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a9/95/a3dbbb5028f35eafb79008e7522a75244477d2838f38cbb722248dabc2a8/cycler-0.12.1.tar.gz", hash = "sha256:88bb128f02ba341da8ef447245a9e138fae777f6a23943da4540077d3601eb1c", size = 7615, upload-time = "2023-10-07T05:32:18.335Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl", hash = "sha256:85cef7cff222d8644161529808465972e51340599459b8ac3ccbac5a854e0d30", size = 8321, upload-time = "2023-10-07T05:32:16.783Z" },
+]
+
+[[package]]
+name = "debugpy"
+version = "1.8.21"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f2/aa/12037145b7a56eaa5b29b41872f7a21b538e807e13f32c4d3c46e59be084/debugpy-1.8.21.tar.gz", hash = "sha256:a3c53278e84c94e11bd87c53970ec391d1a67396c8b22609fcac576520e611a6", size = 1697577, upload-time = "2026-06-01T19:30:35.156Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a2/df/bf625547431a9cadc9f4cbfeda38866e2b17f6aed147b625377e87834449/debugpy-1.8.21-cp312-cp312-macosx_15_0_universal2.whl", hash = "sha256:9f96713896f39c3dff0ee841f47320c3f2983d33c341e009361bb0ebc79adc4e", size = 2483609, upload-time = "2026-06-01T19:30:50.794Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/09/59324b903599031ff9faaec1758292409f6561a0ec2492fe4b703327705a/debugpy-1.8.21-cp312-cp312-manylinux_2_34_x86_64.whl", hash = "sha256:c193d474f0a211191f2b4449d2d06157c689013035bd952f3b617e0ef422b176", size = 3968900, upload-time = "2026-06-01T19:30:52.341Z" },
+ { url = "https://files.pythonhosted.org/packages/14/cd/27f65b805d7fe005c44e1a36b9183ecdfbcdbf9d3e721a5115d461ecc7ee/debugpy-1.8.21-cp312-cp312-win32.whl", hash = "sha256:4743373c1cac7f9e74a1b9915bf1dbe0e900eca657ffb170ae07ac8363205ae9", size = 5336340, upload-time = "2026-06-01T19:30:54.047Z" },
+ { url = "https://files.pythonhosted.org/packages/77/1d/c84e30c0c674184948b66f076ab271c01d940618a2824c23cd035a27bc20/debugpy-1.8.21-cp312-cp312-win_amd64.whl", hash = "sha256:bd7ba9dd3daa7c2f942c6ca8d4695a16bf9ac16b63615261c7982bc74f7ed20c", size = 5374751, upload-time = "2026-06-01T19:30:55.891Z" },
+ { url = "https://files.pythonhosted.org/packages/77/6b/d817e1f8cc77aa055d37fba092e0febfdff40fe652d8d53d4cd7a86ad98d/debugpy-1.8.21-cp313-cp313-macosx_15_0_universal2.whl", hash = "sha256:13678151fc401e2d68c9880b91e28714f797d40422994572b24560ef80910a88", size = 2477398, upload-time = "2026-06-01T19:30:57.644Z" },
+ { url = "https://files.pythonhosted.org/packages/48/57/412421516afc3055fa577516f00beec3d663f9b0ab330639547ae6c57720/debugpy-1.8.21-cp313-cp313-manylinux_2_34_x86_64.whl", hash = "sha256:ecbd158386c31ffe71d46f72d44d56e66331ab9b16cad649156d514368f23ab2", size = 3962096, upload-time = "2026-06-01T19:30:59.235Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/62/2c616337cf6ba7b07ebbc97f02c6c945a8e2f76b365e33ee809c32ee36d1/debugpy-1.8.21-cp313-cp313-win32.whl", hash = "sha256:2c2ae706dec41d99a9ca1f7ebc987a83e65578363be6f6b3ac9067504917fae1", size = 5336288, upload-time = "2026-06-01T19:31:00.79Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/99/9175103392f84c4b1bf7622888cdc68da07f0ff7d9e581266428f6776033/debugpy-1.8.21-cp313-cp313-win_amd64.whl", hash = "sha256:aa648733047443eb1d07682c4ef287d36a54507b643ffdf38b09a3ef002c72a0", size = 5376567, upload-time = "2026-06-01T19:31:02.56Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/3d/f4bbb323a548bfab2af3d6b4ffd9bf22636e55956a1285d317a1de643aad/debugpy-1.8.21-cp314-cp314-macosx_15_0_universal2.whl", hash = "sha256:9bb2a685287a2ac9b181cde89edcec64845cb51de7faaa75badb9a698bc24782", size = 2477209, upload-time = "2026-06-01T19:31:04.157Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/2d/6e7ec524984a1702777868de49a4c53202bddac2a432a76a093469587750/debugpy-1.8.21-cp314-cp314-manylinux_2_34_x86_64.whl", hash = "sha256:3d6922439bf33fd38a3e2c447869ebc7b97da5cd3d329ff1ef9bc06c4903437e", size = 3927115, upload-time = "2026-06-01T19:31:05.863Z" },
+ { url = "https://files.pythonhosted.org/packages/97/47/d1aa6d64005a98a9144647d99306b419396f9ad7bf1d73c119e17a81fb4d/debugpy-1.8.21-cp314-cp314-win32.whl", hash = "sha256:15d4963bd5ffa48f0da0947fd06757fa7621945048a14ad7705431566d3c0e7c", size = 5336724, upload-time = "2026-06-01T19:31:07.711Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/67/b905b90d163af11878c1af8abafa4a25206335e112e284e413454543a6da/debugpy-1.8.21-cp314-cp314-win_amd64.whl", hash = "sha256:fe0744a12353406de0ae8ccff0d0a4a666f00801a3db8fd04e7a5f761cd520e8", size = 5373803, upload-time = "2026-06-01T19:31:09.469Z" },
+ { url = "https://files.pythonhosted.org/packages/95/51/67e7cf11a53e40694f720457d5b3a1cdaaa3d5a9a633e482f225456b93ff/debugpy-1.8.21-py2.py3-none-any.whl", hash = "sha256:b1e37d333663c8851516a47364ef473da127f9caebe4417e6df6f5825a7e9a92", size = 5352888, upload-time = "2026-06-01T19:31:25.186Z" },
+]
+
+[[package]]
+name = "decorator"
+version = "5.3.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/60/8b/32f9823da46cde7df2087faa08cd98d01b908f8dcab982cdba9c84e85355/decorator-5.3.1.tar.gz", hash = "sha256:4cbcdd55a6efadb9dbea26b858f4fb3264567b52d69ca0d25b721b553f60ea82", size = 58084, upload-time = "2026-05-18T06:03:28.057Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/05/7f/798705f5296a58ca505d600456748d1be48078eac8a7050d8a98bc9edb89/decorator-5.3.1-py3-none-any.whl", hash = "sha256:f47fe6fdbd2edd623ecfe36875d37aba411624e2670dd395dddae1358689bb3c", size = 10365, upload-time = "2026-05-18T06:03:26.517Z" },
+]
+
+[[package]]
+name = "docutils"
+version = "0.22.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ae/b6/03bb70946330e88ffec97aefd3ea75ba575cb2e762061e0e62a213befee8/docutils-0.22.4.tar.gz", hash = "sha256:4db53b1fde9abecbb74d91230d32ab626d94f6badfc575d6db9194a49df29968", size = 2291750, upload-time = "2025-12-18T19:00:26.443Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/02/10/5da547df7a391dcde17f59520a231527b8571e6f46fc8efb02ccb370ab12/docutils-0.22.4-py3-none-any.whl", hash = "sha256:d0013f540772d1420576855455d050a2180186c91c15779301ac2ccb3eeb68de", size = 633196, upload-time = "2025-12-18T19:00:18.077Z" },
+]
+
+[[package]]
+name = "donfig"
+version = "0.8.1.post1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pyyaml" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/25/71/80cc718ff6d7abfbabacb1f57aaa42e9c1552bfdd01e64ddd704e4a03638/donfig-0.8.1.post1.tar.gz", hash = "sha256:3bef3413a4c1c601b585e8d297256d0c1470ea012afa6e8461dc28bfb7c23f52", size = 19506, upload-time = "2024-05-23T14:14:31.513Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0c/d5/c5db1ea3394c6e1732fb3286b3bd878b59507a8f77d32a2cebda7d7b7cd4/donfig-0.8.1.post1-py3-none-any.whl", hash = "sha256:2a3175ce74a06109ff9307d90a230f81215cbac9a751f4d1c6194644b8204f9d", size = 21592, upload-time = "2024-05-23T14:13:55.283Z" },
+]
+
+[[package]]
+name = "execnet"
+version = "2.1.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/bf/89/780e11f9588d9e7128a3f87788354c7946a9cbb1401ad38a48c4db9a4f07/execnet-2.1.2.tar.gz", hash = "sha256:63d83bfdd9a23e35b9c6a3261412324f964c2ec8dcd8d3c6916ee9373e0befcd", size = 166622, upload-time = "2025-11-12T09:56:37.75Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ab/84/02fc1827e8cdded4aa65baef11296a9bbe595c474f0d6d758af082d849fd/execnet-2.1.2-py3-none-any.whl", hash = "sha256:67fba928dd5a544b783f6056f449e5e3931a5c378b128bc18501f7ea79e296ec", size = 40708, upload-time = "2025-11-12T09:56:36.333Z" },
+]
+
+[[package]]
+name = "executing"
+version = "2.2.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cc/28/c14e053b6762b1044f34a13aab6859bbf40456d37d23aa286ac24cfd9a5d/executing-2.2.1.tar.gz", hash = "sha256:3632cc370565f6648cc328b32435bd120a1e4ebb20c77e3fdde9a13cd1e533c4", size = 1129488, upload-time = "2025-09-01T09:48:10.866Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c1/ea/53f2148663b321f21b5a606bd5f191517cf40b7072c0497d3c92c4a13b1e/executing-2.2.1-py2.py3-none-any.whl", hash = "sha256:760643d3452b4d777d295bb167ccc74c64a81df23fb5e08eff250c425a4b2017", size = 28317, upload-time = "2025-09-01T09:48:08.5Z" },
+]
+
+[[package]]
+name = "fastjsonschema"
+version = "2.21.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/20/b5/23b216d9d985a956623b6bd12d4086b60f0059b27799f23016af04a74ea1/fastjsonschema-2.21.2.tar.gz", hash = "sha256:b1eb43748041c880796cd077f1a07c3d94e93ae84bba5ed36800a33554ae05de", size = 374130, upload-time = "2025-08-14T18:49:36.666Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cb/a8/20d0723294217e47de6d9e2e40fd4a9d2f7c4b6ef974babd482a59743694/fastjsonschema-2.21.2-py3-none-any.whl", hash = "sha256:1c797122d0a86c5cace2e54bf4e819c36223b552017172f32c5c024a6b77e463", size = 24024, upload-time = "2025-08-14T18:49:34.776Z" },
+]
+
+[[package]]
+name = "filelock"
+version = "3.32.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c0/80/8232b582c4b318b817cf1274ba74976b07b34d35ef439b3eb948f98645a1/filelock-3.32.0.tar.gz", hash = "sha256:7be2ad23a14607ccc71808e68fe30848aeace7058ace17852f68e2a68e310402", size = 213757, upload-time = "2026-07-21T13:17:42.898Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/06/79/b4c714bef36bc4ec2beeae1e0c124f0223888cd8c6feb1cdc56038116920/filelock-3.32.0-py3-none-any.whl", hash = "sha256:d396bea984af47333ef05e50eae7eff88c84256de6112aea0ec48a233c064fe3", size = 97732, upload-time = "2026-07-21T13:17:41.55Z" },
+]
+
+[[package]]
+name = "fonttools"
+version = "4.63.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/84/69/c97f2c18e0db87d2c7b15da1974dace76ae938f1cfa22e2727a648b7ed43/fonttools-4.63.0.tar.gz", hash = "sha256:caeb583deeb5168e694b65cda8b4ee62abedfa66cf88488734466f2366b9c4e0", size = 3597189, upload-time = "2026-05-14T12:04:30.958Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/08/ef/b3c6b9b5be2f82416d73fe2ed2e96e2793cd80e7510bd6a17ca79cdd88ec/fonttools-4.63.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:37dd23e621e3b0aef1baa70a303b80aaf38449632cfc8fd2a55fb285bbccfc02", size = 2881131, upload-time = "2026-05-14T12:03:13.386Z" },
+ { url = "https://files.pythonhosted.org/packages/44/a0/c815bea63117fa63e4e1c01f8a1110d2112fa003f838e6467094ec2432ce/fonttools-4.63.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:a9faff9e0c1f76f9fd55899d2ce785832efebab37eb8ae13995853aef178bef0", size = 2426704, upload-time = "2026-05-14T12:03:15.801Z" },
+ { url = "https://files.pythonhosted.org/packages/44/04/0b91d8e916e92ad1fac9e4624760baf0fd5ff2ead614c2f68fb21373f03f/fonttools-4.63.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ef3048ef05dbb552b89817713d9cac912e00d0fde4a3105c00d29e52e10c89af", size = 5044298, upload-time = "2026-05-14T12:03:18.085Z" },
+ { url = "https://files.pythonhosted.org/packages/77/c7/2342da9830e3e9d4870305ca5d2091d2a83284f2953079b7bdd3b5e029d8/fonttools-4.63.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:58dc6bb86a78d782f00f9190ca02c119cf5bbe2807536e361e18d42019f877d8", size = 4999800, upload-time = "2026-05-14T12:03:20.161Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/6d/67fe16c48d7ce050979b33f47e0d28a318f02da030602e944c34f7a16ef3/fonttools-4.63.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ee08ebfa58f6e1aeff5697ab9582105bb620008c1caafb681e4c557e7483027b", size = 4982666, upload-time = "2026-05-14T12:03:22.87Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/00/3bbab338c07c71fa56269953845e92c951a61457bbbb0f1022551ea266d9/fonttools-4.63.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:27fdc65af8da6f88b9c6121c47a464cbe359fcfff7ff6fc2d37a1f395d755b78", size = 5133598, upload-time = "2026-05-14T12:03:25.168Z" },
+ { url = "https://files.pythonhosted.org/packages/62/f2/aa27c7f98db5b064883dadcc5283947e81e034de42e22a33675878d98b54/fonttools-4.63.0-cp312-cp312-win32.whl", hash = "sha256:af2fd1664d00a397d75f806985ddb36282091c2131a73a6485c23b4a34722263", size = 2292575, upload-time = "2026-05-14T12:03:27.496Z" },
+ { url = "https://files.pythonhosted.org/packages/87/36/cccb9bc2a6ab63d1b2980374f0dca72ce95ae267c9b4cfe77455bb70d0d4/fonttools-4.63.0-cp312-cp312-win_amd64.whl", hash = "sha256:59ac449f8cca9b4ffa08d2e7bbadad87ce710d69d1eda5c3c1ce579baa987272", size = 2343211, upload-time = "2026-05-14T12:03:30.057Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/8d/d8fec3dcde2963f8c908fb315e5ff2cd0ac34f82394bbbf73a2aa5145ce3/fonttools-4.63.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:cd7e9857e5e63738b9d9fd707bc1f59c8b09e5177726d23664db393c59bb08bd", size = 2876062, upload-time = "2026-05-14T12:03:32.554Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/71/d935dc54e4ff121bfdd11e08702db63a7e6f25af21d8a3d7b7212df53641/fonttools-4.63.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c2a2a42198b696a6f48fad91709afb55176e66a5e566131219dba372fb7f8c59", size = 2424594, upload-time = "2026-05-14T12:03:34.86Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/40/e76320afa1df918e146155ef239b1719ee266092e96f5423bfd075affba1/fonttools-4.63.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1e874792a8212b44583ea02189d9e693906b2f78b261f372f95d6c563210ac1d", size = 5024840, upload-time = "2026-05-14T12:03:36.745Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/36/0b805d8c485f872f65a509cbe3b58a5d0d17bee855333b54a150c79d3061/fonttools-4.63.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:22135da48a348785c5e2d5d2d9d6bec5ed44adacbaeb9db12d9493bf6c6bfa68", size = 4975801, upload-time = "2026-05-14T12:03:38.833Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/26/2cee03d0aa083ab022da5c07aff9ed3f689da1defb81ad6917c9627896da/fonttools-4.63.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ccf41f2efdf56994d22d73bef4ced1052161958169428d06ba9724ea9e9a64be", size = 4965009, upload-time = "2026-05-14T12:03:41.494Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/48/cc4b66d9058c0d0982c833fad10127c4b0e9324606aafa41382295ca4102/fonttools-4.63.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9ced0bd02ac751dd6319b0da88aaef24414e3b0dbc32bb4f24944821a3741a27", size = 5105892, upload-time = "2026-05-14T12:03:43.525Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/1f/a98a30a814b9ddef3a2e706025f90b9e0bc94890e6cb15254bc86547d11a/fonttools-4.63.0-cp313-cp313-win32.whl", hash = "sha256:85be818f5506e8a7753153def2c9550178f0ecae6a47b5e0e8dbb23f7cc90380", size = 2291313, upload-time = "2026-05-14T12:03:45.594Z" },
+ { url = "https://files.pythonhosted.org/packages/92/46/5177b01f3b4abfdd4409f31cca4ab279c9343a26efbe9ec78c97fc612e02/fonttools-4.63.0-cp313-cp313-win_amd64.whl", hash = "sha256:ba04cb5891d4c0c21b6da95eda8d7b090021508a294fff33464fc7d241e0856b", size = 2342299, upload-time = "2026-05-14T12:03:47.414Z" },
+ { url = "https://files.pythonhosted.org/packages/27/d2/23d25e3f247b328be58d04a4c9f894178a0d1eda7d42867cfb388adaf416/fonttools-4.63.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:fd1e3094f42d806d3d7c79162fc59e5910fcbe3a7360c385b8da969bc4493745", size = 2875338, upload-time = "2026-05-14T12:03:50.052Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/58/7dfa0c761cb3b2964e2a84c4dc986c926a87de0cb9fb60d5b28ded3f2914/fonttools-4.63.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:6e528da43bc3791085f8cb6141b1d13e459226790240340fcbb4625649238b03", size = 2422661, upload-time = "2026-05-14T12:03:52.154Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/87/64cfa18a7a1621d17b7f4502b2b0ed8a135a90c3db51ea590ee99043e76b/fonttools-4.63.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b2248c5decb223562f7902ff6325077a073f608ee8e33e88ad88db734eb9f49", size = 5010526, upload-time = "2026-05-14T12:03:54.647Z" },
+ { url = "https://files.pythonhosted.org/packages/36/e1/a8933a72c45a87177fbde2696e0d0755c8c9062f8c077a961c6215fa27b1/fonttools-4.63.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:308f957cdeaf8abe4e5f2f124902ef405448af92c90f80e302a3b771c2e6116b", size = 4923946, upload-time = "2026-05-14T12:03:56.984Z" },
+ { url = "https://files.pythonhosted.org/packages/27/60/872e6e233b8c5e8b41413796ff18b7fe479661bd40147e071b450dfad7a1/fonttools-4.63.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:bf00f21eb5fb721dbaf73d1e9da6d02a1af7768f2ebcf9798be98beab8ba90f6", size = 4962489, upload-time = "2026-05-14T12:03:59.443Z" },
+ { url = "https://files.pythonhosted.org/packages/30/c4/83c24f2ec38b90cfda84bf4b1a1f49df80e84a1db4e7ac6e0d41bf23bc39/fonttools-4.63.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c1aaa4b9c75798400ac043ce04d74e7830376c85095a5a6ed7cba2f17a266bf4", size = 5071870, upload-time = "2026-05-14T12:04:02.122Z" },
+ { url = "https://files.pythonhosted.org/packages/de/40/3ae22b60ff1d41ce0bd044b31238cdc72cef99f28b976f1e128ebd618c9b/fonttools-4.63.0-cp314-cp314-win32.whl", hash = "sha256:22693918177bd9ceabec4736d338045f357769416fc6b0b2508eefef75b08616", size = 2295026, upload-time = "2026-05-14T12:04:04.47Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/d4/98078064ccc76b45cb0f6c002452011e93c4bd26f6850344f0951cc1fe89/fonttools-4.63.0-cp314-cp314-win_amd64.whl", hash = "sha256:7d782fac32985914c351556f68ac0855391572bcd87de50e05970d3cd4c96fc5", size = 2347454, upload-time = "2026-05-14T12:04:06.752Z" },
+ { url = "https://files.pythonhosted.org/packages/49/4e/652d1580c5f4e39f7d103b0c793e4773129ad633dce4addd0cf4dfebde02/fonttools-4.63.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:6db5140a60a5d731d21ec076745b40a310607731b0a565b50776393188649001", size = 2958152, upload-time = "2026-05-14T12:04:08.706Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/55/ad864c9a9b219f552eb46b32cd7906c466e5a578ba0c3abfcc0fe7413eb6/fonttools-4.63.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7d76edbff9014094dbf03bd2d074709dfa6ec7aba13d838c937a2b33d2d6a86e", size = 2460809, upload-time = "2026-05-14T12:04:10.783Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/2b/0aa8db70f18cf52e49b4ed5ecec68547f981160bf5ded3b5aed6faa0a6f9/fonttools-4.63.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0eac00b9118c3c2f87d272e45341871c5b3066baa3c86897fa634a7c3fb59096", size = 5148649, upload-time = "2026-05-14T12:04:12.747Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/63/18e4369c25043096f1048e0c9915951adc4f842bd81c6b18155824d6fa99/fonttools-4.63.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:51394295f1a51de8b5f30bdb1e1b9a4231536c7064ef5c6e211eec19fa36036f", size = 4932147, upload-time = "2026-05-14T12:04:14.806Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/3f/67f3eac2ffd8a98446c5022f8ed3864eac878a5ff7af8df4c8286dba16cc/fonttools-4.63.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:9e12f105d2b6342c559c298afb674006bb2893afc7102dcf8a1b55b0486b4e40", size = 5027237, upload-time = "2026-05-14T12:04:17.675Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/ba/4e6214cb38a7b04779e97bb7636de9a5c7f20af7018d03dee0b64c08510a/fonttools-4.63.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:796f27556dbe094c4824f75ca85267e4df776c79036c8441469a4df37038c196", size = 5053933, upload-time = "2026-05-14T12:04:20.818Z" },
+ { url = "https://files.pythonhosted.org/packages/34/3b/214dcc19ee31d3d38fb5ad2755c11ef0514e5dc300bbaf41c0b69f393799/fonttools-4.63.0-cp314-cp314t-win32.whl", hash = "sha256:948428a275741f0b64b113c955425a953314f4b9ab9997f73a72c83e68e569c8", size = 2359326, upload-time = "2026-05-14T12:04:24.22Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/1e/3ff1a9b523058c2eeb6a9d50f5574e2a738200d0d94107d5bc4105e8da3f/fonttools-4.63.0-cp314-cp314t-win_amd64.whl", hash = "sha256:6d4741eb179121cab9eea4cb2393d24492373a260d7945006358c08cfbf45419", size = 2425829, upload-time = "2026-05-14T12:04:26.829Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/47/c99d5268f354002ce80f8d029cd9d7d872969da1de8b93d32de4dc56d6f4/fonttools-4.63.0-py3-none-any.whl", hash = "sha256:445af2eab030a16b9171ea8bdda7ebf7d96bda2df88ee182a464252f6e05e20d", size = 1164562, upload-time = "2026-05-14T12:04:29.092Z" },
+]
+
+[[package]]
+name = "frozenlist"
+version = "1.8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/2d/f5/c831fac6cc817d26fd54c7eaccd04ef7e0288806943f7cc5bbf69f3ac1f0/frozenlist-1.8.0.tar.gz", hash = "sha256:3ede829ed8d842f6cd48fc7081d7a41001a56f1f38603f9d49bf3020d59a31ad", size = 45875, upload-time = "2025-10-06T05:38:17.865Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/69/29/948b9aa87e75820a38650af445d2ef2b6b8a6fab1a23b6bb9e4ef0be2d59/frozenlist-1.8.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:78f7b9e5d6f2fdb88cdde9440dc147259b62b9d3b019924def9f6478be254ac1", size = 87782, upload-time = "2025-10-06T05:36:06.649Z" },
+ { url = "https://files.pythonhosted.org/packages/64/80/4f6e318ee2a7c0750ed724fa33a4bdf1eacdc5a39a7a24e818a773cd91af/frozenlist-1.8.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:229bf37d2e4acdaf808fd3f06e854a4a7a3661e871b10dc1f8f1896a3b05f18b", size = 50594, upload-time = "2025-10-06T05:36:07.69Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/94/5c8a2b50a496b11dd519f4a24cb5496cf125681dd99e94c604ccdea9419a/frozenlist-1.8.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f833670942247a14eafbb675458b4e61c82e002a148f49e68257b79296e865c4", size = 50448, upload-time = "2025-10-06T05:36:08.78Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/bd/d91c5e39f490a49df14320f4e8c80161cfcce09f1e2cde1edd16a551abb3/frozenlist-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:494a5952b1c597ba44e0e78113a7266e656b9794eec897b19ead706bd7074383", size = 242411, upload-time = "2025-10-06T05:36:09.801Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/83/f61505a05109ef3293dfb1ff594d13d64a2324ac3482be2cedc2be818256/frozenlist-1.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96f423a119f4777a4a056b66ce11527366a8bb92f54e541ade21f2374433f6d4", size = 243014, upload-time = "2025-10-06T05:36:11.394Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/cb/cb6c7b0f7d4023ddda30cf56b8b17494eb3a79e3fda666bf735f63118b35/frozenlist-1.8.0-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3462dd9475af2025c31cc61be6652dfa25cbfb56cbbf52f4ccfe029f38decaf8", size = 234909, upload-time = "2025-10-06T05:36:12.598Z" },
+ { url = "https://files.pythonhosted.org/packages/31/c5/cd7a1f3b8b34af009fb17d4123c5a778b44ae2804e3ad6b86204255f9ec5/frozenlist-1.8.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c4c800524c9cd9bac5166cd6f55285957fcfc907db323e193f2afcd4d9abd69b", size = 250049, upload-time = "2025-10-06T05:36:14.065Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/01/2f95d3b416c584a1e7f0e1d6d31998c4a795f7544069ee2e0962a4b60740/frozenlist-1.8.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d6a5df73acd3399d893dafc71663ad22534b5aa4f94e8a2fabfe856c3c1b6a52", size = 256485, upload-time = "2025-10-06T05:36:15.39Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/03/024bf7720b3abaebcff6d0793d73c154237b85bdf67b7ed55e5e9596dc9a/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:405e8fe955c2280ce66428b3ca55e12b3c4e9c336fb2103a4937e891c69a4a29", size = 237619, upload-time = "2025-10-06T05:36:16.558Z" },
+ { url = "https://files.pythonhosted.org/packages/69/fa/f8abdfe7d76b731f5d8bd217827cf6764d4f1d9763407e42717b4bed50a0/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:908bd3f6439f2fef9e85031b59fd4f1297af54415fb60e4254a95f75b3cab3f3", size = 250320, upload-time = "2025-10-06T05:36:17.821Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/3c/b051329f718b463b22613e269ad72138cc256c540f78a6de89452803a47d/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:294e487f9ec720bd8ffcebc99d575f7eff3568a08a253d1ee1a0378754b74143", size = 246820, upload-time = "2025-10-06T05:36:19.046Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/ae/58282e8f98e444b3f4dd42448ff36fa38bef29e40d40f330b22e7108f565/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:74c51543498289c0c43656701be6b077f4b265868fa7f8a8859c197006efb608", size = 250518, upload-time = "2025-10-06T05:36:20.763Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/96/007e5944694d66123183845a106547a15944fbbb7154788cbf7272789536/frozenlist-1.8.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:776f352e8329135506a1d6bf16ac3f87bc25b28e765949282dcc627af36123aa", size = 239096, upload-time = "2025-10-06T05:36:22.129Z" },
+ { url = "https://files.pythonhosted.org/packages/66/bb/852b9d6db2fa40be96f29c0d1205c306288f0684df8fd26ca1951d461a56/frozenlist-1.8.0-cp312-cp312-win32.whl", hash = "sha256:433403ae80709741ce34038da08511d4a77062aa924baf411ef73d1146e74faf", size = 39985, upload-time = "2025-10-06T05:36:23.661Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/af/38e51a553dd66eb064cdf193841f16f077585d4d28394c2fa6235cb41765/frozenlist-1.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:34187385b08f866104f0c0617404c8eb08165ab1272e884abc89c112e9c00746", size = 44591, upload-time = "2025-10-06T05:36:24.958Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/06/1dc65480ab147339fecc70797e9c2f69d9cea9cf38934ce08df070fdb9cb/frozenlist-1.8.0-cp312-cp312-win_arm64.whl", hash = "sha256:fe3c58d2f5db5fbd18c2987cba06d51b0529f52bc3a6cdc33d3f4eab725104bd", size = 40102, upload-time = "2025-10-06T05:36:26.333Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/40/0832c31a37d60f60ed79e9dfb5a92e1e2af4f40a16a29abcc7992af9edff/frozenlist-1.8.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:8d92f1a84bb12d9e56f818b3a746f3efba93c1b63c8387a73dde655e1e42282a", size = 85717, upload-time = "2025-10-06T05:36:27.341Z" },
+ { url = "https://files.pythonhosted.org/packages/30/ba/b0b3de23f40bc55a7057bd38434e25c34fa48e17f20ee273bbde5e0650f3/frozenlist-1.8.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:96153e77a591c8adc2ee805756c61f59fef4cf4073a9275ee86fe8cba41241f7", size = 49651, upload-time = "2025-10-06T05:36:28.855Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/ab/6e5080ee374f875296c4243c381bbdef97a9ac39c6e3ce1d5f7d42cb78d6/frozenlist-1.8.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f21f00a91358803399890ab167098c131ec2ddd5f8f5fd5fe9c9f2c6fcd91e40", size = 49417, upload-time = "2025-10-06T05:36:29.877Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/4e/e4691508f9477ce67da2015d8c00acd751e6287739123113a9fca6f1604e/frozenlist-1.8.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:fb30f9626572a76dfe4293c7194a09fb1fe93ba94c7d4f720dfae3b646b45027", size = 234391, upload-time = "2025-10-06T05:36:31.301Z" },
+ { url = "https://files.pythonhosted.org/packages/40/76/c202df58e3acdf12969a7895fd6f3bc016c642e6726aa63bd3025e0fc71c/frozenlist-1.8.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eaa352d7047a31d87dafcacbabe89df0aa506abb5b1b85a2fb91bc3faa02d822", size = 233048, upload-time = "2025-10-06T05:36:32.531Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/c0/8746afb90f17b73ca5979c7a3958116e105ff796e718575175319b5bb4ce/frozenlist-1.8.0-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:03ae967b4e297f58f8c774c7eabcce57fe3c2434817d4385c50661845a058121", size = 226549, upload-time = "2025-10-06T05:36:33.706Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/eb/4c7eefc718ff72f9b6c4893291abaae5fbc0c82226a32dcd8ef4f7a5dbef/frozenlist-1.8.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f6292f1de555ffcc675941d65fffffb0a5bcd992905015f85d0592201793e0e5", size = 239833, upload-time = "2025-10-06T05:36:34.947Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/4e/e5c02187cf704224f8b21bee886f3d713ca379535f16893233b9d672ea71/frozenlist-1.8.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:29548f9b5b5e3460ce7378144c3010363d8035cea44bc0bf02d57f5a685e084e", size = 245363, upload-time = "2025-10-06T05:36:36.534Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/96/cb85ec608464472e82ad37a17f844889c36100eed57bea094518bf270692/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ec3cc8c5d4084591b4237c0a272cc4f50a5b03396a47d9caaf76f5d7b38a4f11", size = 229314, upload-time = "2025-10-06T05:36:38.582Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/6f/4ae69c550e4cee66b57887daeebe006fe985917c01d0fff9caab9883f6d0/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:517279f58009d0b1f2e7c1b130b377a349405da3f7621ed6bfae50b10adf20c1", size = 243365, upload-time = "2025-10-06T05:36:40.152Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/58/afd56de246cf11780a40a2c28dc7cbabbf06337cc8ddb1c780a2d97e88d8/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:db1e72ede2d0d7ccb213f218df6a078a9c09a7de257c2fe8fcef16d5925230b1", size = 237763, upload-time = "2025-10-06T05:36:41.355Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/36/cdfaf6ed42e2644740d4a10452d8e97fa1c062e2a8006e4b09f1b5fd7d63/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:b4dec9482a65c54a5044486847b8a66bf10c9cb4926d42927ec4e8fd5db7fed8", size = 240110, upload-time = "2025-10-06T05:36:42.716Z" },
+ { url = "https://files.pythonhosted.org/packages/03/a8/9ea226fbefad669f11b52e864c55f0bd57d3c8d7eb07e9f2e9a0b39502e1/frozenlist-1.8.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:21900c48ae04d13d416f0e1e0c4d81f7931f73a9dfa0b7a8746fb2fe7dd970ed", size = 233717, upload-time = "2025-10-06T05:36:44.251Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/0b/1b5531611e83ba7d13ccc9988967ea1b51186af64c42b7a7af465dcc9568/frozenlist-1.8.0-cp313-cp313-win32.whl", hash = "sha256:8b7b94a067d1c504ee0b16def57ad5738701e4ba10cec90529f13fa03c833496", size = 39628, upload-time = "2025-10-06T05:36:45.423Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/cf/174c91dbc9cc49bc7b7aab74d8b734e974d1faa8f191c74af9b7e80848e6/frozenlist-1.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:878be833caa6a3821caf85eb39c5ba92d28e85df26d57afb06b35b2efd937231", size = 43882, upload-time = "2025-10-06T05:36:46.796Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/17/502cd212cbfa96eb1388614fe39a3fc9ab87dbbe042b66f97acb57474834/frozenlist-1.8.0-cp313-cp313-win_arm64.whl", hash = "sha256:44389d135b3ff43ba8cc89ff7f51f5a0bb6b63d829c8300f79a2fe4fe61bcc62", size = 39676, upload-time = "2025-10-06T05:36:47.8Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/5c/3bbfaa920dfab09e76946a5d2833a7cbdf7b9b4a91c714666ac4855b88b4/frozenlist-1.8.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:e25ac20a2ef37e91c1b39938b591457666a0fa835c7783c3a8f33ea42870db94", size = 89235, upload-time = "2025-10-06T05:36:48.78Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/d6/f03961ef72166cec1687e84e8925838442b615bd0b8854b54923ce5b7b8a/frozenlist-1.8.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:07cdca25a91a4386d2e76ad992916a85038a9b97561bf7a3fd12d5d9ce31870c", size = 50742, upload-time = "2025-10-06T05:36:49.837Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/bb/a6d12b7ba4c3337667d0e421f7181c82dda448ce4e7ad7ecd249a16fa806/frozenlist-1.8.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:4e0c11f2cc6717e0a741f84a527c52616140741cd812a50422f83dc31749fb52", size = 51725, upload-time = "2025-10-06T05:36:50.851Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/71/d1fed0ffe2c2ccd70b43714c6cab0f4188f09f8a67a7914a6b46ee30f274/frozenlist-1.8.0-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b3210649ee28062ea6099cfda39e147fa1bc039583c8ee4481cb7811e2448c51", size = 284533, upload-time = "2025-10-06T05:36:51.898Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/1f/fb1685a7b009d89f9bf78a42d94461bc06581f6e718c39344754a5d9bada/frozenlist-1.8.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:581ef5194c48035a7de2aefc72ac6539823bb71508189e5de01d60c9dcd5fa65", size = 292506, upload-time = "2025-10-06T05:36:53.101Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/3b/b991fe1612703f7e0d05c0cf734c1b77aaf7c7d321df4572e8d36e7048c8/frozenlist-1.8.0-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3ef2d026f16a2b1866e1d86fc4e1291e1ed8a387b2c333809419a2f8b3a77b82", size = 274161, upload-time = "2025-10-06T05:36:54.309Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/ec/c5c618767bcdf66e88945ec0157d7f6c4a1322f1473392319b7a2501ded7/frozenlist-1.8.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5500ef82073f599ac84d888e3a8c1f77ac831183244bfd7f11eaa0289fb30714", size = 294676, upload-time = "2025-10-06T05:36:55.566Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/ce/3934758637d8f8a88d11f0585d6495ef54b2044ed6ec84492a91fa3b27aa/frozenlist-1.8.0-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:50066c3997d0091c411a66e710f4e11752251e6d2d73d70d8d5d4c76442a199d", size = 300638, upload-time = "2025-10-06T05:36:56.758Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/4f/a7e4d0d467298f42de4b41cbc7ddaf19d3cfeabaf9ff97c20c6c7ee409f9/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:5c1c8e78426e59b3f8005e9b19f6ff46e5845895adbde20ece9218319eca6506", size = 283067, upload-time = "2025-10-06T05:36:57.965Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/48/c7b163063d55a83772b268e6d1affb960771b0e203b632cfe09522d67ea5/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:eefdba20de0d938cec6a89bd4d70f346a03108a19b9df4248d3cf0d88f1b0f51", size = 292101, upload-time = "2025-10-06T05:36:59.237Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/d0/2366d3c4ecdc2fd391e0afa6e11500bfba0ea772764d631bbf82f0136c9d/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:cf253e0e1c3ceb4aaff6df637ce033ff6535fb8c70a764a8f46aafd3d6ab798e", size = 289901, upload-time = "2025-10-06T05:37:00.811Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/94/daff920e82c1b70e3618a2ac39fbc01ae3e2ff6124e80739ce5d71c9b920/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:032efa2674356903cd0261c4317a561a6850f3ac864a63fc1583147fb05a79b0", size = 289395, upload-time = "2025-10-06T05:37:02.115Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/20/bba307ab4235a09fdcd3cc5508dbabd17c4634a1af4b96e0f69bfe551ebd/frozenlist-1.8.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:6da155091429aeba16851ecb10a9104a108bcd32f6c1642867eadaee401c1c41", size = 283659, upload-time = "2025-10-06T05:37:03.711Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/00/04ca1c3a7a124b6de4f8a9a17cc2fcad138b4608e7a3fc5877804b8715d7/frozenlist-1.8.0-cp313-cp313t-win32.whl", hash = "sha256:0f96534f8bfebc1a394209427d0f8a63d343c9779cda6fc25e8e121b5fd8555b", size = 43492, upload-time = "2025-10-06T05:37:04.915Z" },
+ { url = "https://files.pythonhosted.org/packages/59/5e/c69f733a86a94ab10f68e496dc6b7e8bc078ebb415281d5698313e3af3a1/frozenlist-1.8.0-cp313-cp313t-win_amd64.whl", hash = "sha256:5d63a068f978fc69421fb0e6eb91a9603187527c86b7cd3f534a5b77a592b888", size = 48034, upload-time = "2025-10-06T05:37:06.343Z" },
+ { url = "https://files.pythonhosted.org/packages/16/6c/be9d79775d8abe79b05fa6d23da99ad6e7763a1d080fbae7290b286093fd/frozenlist-1.8.0-cp313-cp313t-win_arm64.whl", hash = "sha256:bf0a7e10b077bf5fb9380ad3ae8ce20ef919a6ad93b4552896419ac7e1d8e042", size = 41749, upload-time = "2025-10-06T05:37:07.431Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/c8/85da824b7e7b9b6e7f7705b2ecaf9591ba6f79c1177f324c2735e41d36a2/frozenlist-1.8.0-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:cee686f1f4cadeb2136007ddedd0aaf928ab95216e7691c63e50a8ec066336d0", size = 86127, upload-time = "2025-10-06T05:37:08.438Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/e8/a1185e236ec66c20afd72399522f142c3724c785789255202d27ae992818/frozenlist-1.8.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:119fb2a1bd47307e899c2fac7f28e85b9a543864df47aa7ec9d3c1b4545f096f", size = 49698, upload-time = "2025-10-06T05:37:09.48Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/93/72b1736d68f03fda5fdf0f2180fb6caaae3894f1b854d006ac61ecc727ee/frozenlist-1.8.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:4970ece02dbc8c3a92fcc5228e36a3e933a01a999f7094ff7c23fbd2beeaa67c", size = 49749, upload-time = "2025-10-06T05:37:10.569Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/b2/fabede9fafd976b991e9f1b9c8c873ed86f202889b864756f240ce6dd855/frozenlist-1.8.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:cba69cb73723c3f329622e34bdbf5ce1f80c21c290ff04256cff1cd3c2036ed2", size = 231298, upload-time = "2025-10-06T05:37:11.993Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/3b/d9b1e0b0eed36e70477ffb8360c49c85c8ca8ef9700a4e6711f39a6e8b45/frozenlist-1.8.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:778a11b15673f6f1df23d9586f83c4846c471a8af693a22e066508b77d201ec8", size = 232015, upload-time = "2025-10-06T05:37:13.194Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/94/be719d2766c1138148564a3960fc2c06eb688da592bdc25adcf856101be7/frozenlist-1.8.0-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0325024fe97f94c41c08872db482cf8ac4800d80e79222c6b0b7b162d5b13686", size = 225038, upload-time = "2025-10-06T05:37:14.577Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/09/6712b6c5465f083f52f50cf74167b92d4ea2f50e46a9eea0523d658454ae/frozenlist-1.8.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:97260ff46b207a82a7567b581ab4190bd4dfa09f4db8a8b49d1a958f6aa4940e", size = 240130, upload-time = "2025-10-06T05:37:15.781Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/d4/cd065cdcf21550b54f3ce6a22e143ac9e4836ca42a0de1022da8498eac89/frozenlist-1.8.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:54b2077180eb7f83dd52c40b2750d0a9f175e06a42e3213ce047219de902717a", size = 242845, upload-time = "2025-10-06T05:37:17.037Z" },
+ { url = "https://files.pythonhosted.org/packages/62/c3/f57a5c8c70cd1ead3d5d5f776f89d33110b1addae0ab010ad774d9a44fb9/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:2f05983daecab868a31e1da44462873306d3cbfd76d1f0b5b69c473d21dbb128", size = 229131, upload-time = "2025-10-06T05:37:18.221Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/52/232476fe9cb64f0742f3fde2b7d26c1dac18b6d62071c74d4ded55e0ef94/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:33f48f51a446114bc5d251fb2954ab0164d5be02ad3382abcbfe07e2531d650f", size = 240542, upload-time = "2025-10-06T05:37:19.771Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/85/07bf3f5d0fb5414aee5f47d33c6f5c77bfe49aac680bfece33d4fdf6a246/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:154e55ec0655291b5dd1b8731c637ecdb50975a2ae70c606d100750a540082f7", size = 237308, upload-time = "2025-10-06T05:37:20.969Z" },
+ { url = "https://files.pythonhosted.org/packages/11/99/ae3a33d5befd41ac0ca2cc7fd3aa707c9c324de2e89db0e0f45db9a64c26/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:4314debad13beb564b708b4a496020e5306c7333fa9a3ab90374169a20ffab30", size = 238210, upload-time = "2025-10-06T05:37:22.252Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/60/b1d2da22f4970e7a155f0adde9b1435712ece01b3cd45ba63702aea33938/frozenlist-1.8.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:073f8bf8becba60aa931eb3bc420b217bb7d5b8f4750e6f8b3be7f3da85d38b7", size = 231972, upload-time = "2025-10-06T05:37:23.5Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/ab/945b2f32de889993b9c9133216c068b7fcf257d8595a0ac420ac8677cab0/frozenlist-1.8.0-cp314-cp314-win32.whl", hash = "sha256:bac9c42ba2ac65ddc115d930c78d24ab8d4f465fd3fc473cdedfccadb9429806", size = 40536, upload-time = "2025-10-06T05:37:25.581Z" },
+ { url = "https://files.pythonhosted.org/packages/59/ad/9caa9b9c836d9ad6f067157a531ac48b7d36499f5036d4141ce78c230b1b/frozenlist-1.8.0-cp314-cp314-win_amd64.whl", hash = "sha256:3e0761f4d1a44f1d1a47996511752cf3dcec5bbdd9cc2b4fe595caf97754b7a0", size = 44330, upload-time = "2025-10-06T05:37:26.928Z" },
+ { url = "https://files.pythonhosted.org/packages/82/13/e6950121764f2676f43534c555249f57030150260aee9dcf7d64efda11dd/frozenlist-1.8.0-cp314-cp314-win_arm64.whl", hash = "sha256:d1eaff1d00c7751b7c6662e9c5ba6eb2c17a2306ba5e2a37f24ddf3cc953402b", size = 40627, upload-time = "2025-10-06T05:37:28.075Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/c7/43200656ecc4e02d3f8bc248df68256cd9572b3f0017f0a0c4e93440ae23/frozenlist-1.8.0-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:d3bb933317c52d7ea5004a1c442eef86f426886fba134ef8cf4226ea6ee1821d", size = 89238, upload-time = "2025-10-06T05:37:29.373Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/29/55c5f0689b9c0fb765055629f472c0de484dcaf0acee2f7707266ae3583c/frozenlist-1.8.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:8009897cdef112072f93a0efdce29cd819e717fd2f649ee3016efd3cd885a7ed", size = 50738, upload-time = "2025-10-06T05:37:30.792Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/7d/b7282a445956506fa11da8c2db7d276adcbf2b17d8bb8407a47685263f90/frozenlist-1.8.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:2c5dcbbc55383e5883246d11fd179782a9d07a986c40f49abe89ddf865913930", size = 51739, upload-time = "2025-10-06T05:37:32.127Z" },
+ { url = "https://files.pythonhosted.org/packages/62/1c/3d8622e60d0b767a5510d1d3cf21065b9db874696a51ea6d7a43180a259c/frozenlist-1.8.0-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:39ecbc32f1390387d2aa4f5a995e465e9e2f79ba3adcac92d68e3e0afae6657c", size = 284186, upload-time = "2025-10-06T05:37:33.21Z" },
+ { url = "https://files.pythonhosted.org/packages/2d/14/aa36d5f85a89679a85a1d44cd7a6657e0b1c75f61e7cad987b203d2daca8/frozenlist-1.8.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:92db2bf818d5cc8d9c1f1fc56b897662e24ea5adb36ad1f1d82875bd64e03c24", size = 292196, upload-time = "2025-10-06T05:37:36.107Z" },
+ { url = "https://files.pythonhosted.org/packages/05/23/6bde59eb55abd407d34f77d39a5126fb7b4f109a3f611d3929f14b700c66/frozenlist-1.8.0-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:2dc43a022e555de94c3b68a4ef0b11c4f747d12c024a520c7101709a2144fb37", size = 273830, upload-time = "2025-10-06T05:37:37.663Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/3f/22cff331bfad7a8afa616289000ba793347fcd7bc275f3b28ecea2a27909/frozenlist-1.8.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:cb89a7f2de3602cfed448095bab3f178399646ab7c61454315089787df07733a", size = 294289, upload-time = "2025-10-06T05:37:39.261Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/89/5b057c799de4838b6c69aa82b79705f2027615e01be996d2486a69ca99c4/frozenlist-1.8.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:33139dc858c580ea50e7e60a1b0ea003efa1fd42e6ec7fdbad78fff65fad2fd2", size = 300318, upload-time = "2025-10-06T05:37:43.213Z" },
+ { url = "https://files.pythonhosted.org/packages/30/de/2c22ab3eb2a8af6d69dc799e48455813bab3690c760de58e1bf43b36da3e/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:168c0969a329b416119507ba30b9ea13688fafffac1b7822802537569a1cb0ef", size = 282814, upload-time = "2025-10-06T05:37:45.337Z" },
+ { url = "https://files.pythonhosted.org/packages/59/f7/970141a6a8dbd7f556d94977858cfb36fa9b66e0892c6dd780d2219d8cd8/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:28bd570e8e189d7f7b001966435f9dac6718324b5be2990ac496cf1ea9ddb7fe", size = 291762, upload-time = "2025-10-06T05:37:46.657Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/15/ca1adae83a719f82df9116d66f5bb28bb95557b3951903d39135620ef157/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:b2a095d45c5d46e5e79ba1e5b9cb787f541a8dee0433836cea4b96a2c439dcd8", size = 289470, upload-time = "2025-10-06T05:37:47.946Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/83/dca6dc53bf657d371fbc88ddeb21b79891e747189c5de990b9dfff2ccba1/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:eab8145831a0d56ec9c4139b6c3e594c7a83c2c8be25d5bcf2d86136a532287a", size = 289042, upload-time = "2025-10-06T05:37:49.499Z" },
+ { url = "https://files.pythonhosted.org/packages/96/52/abddd34ca99be142f354398700536c5bd315880ed0a213812bc491cff5e4/frozenlist-1.8.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:974b28cf63cc99dfb2188d8d222bc6843656188164848c4f679e63dae4b0708e", size = 283148, upload-time = "2025-10-06T05:37:50.745Z" },
+ { url = "https://files.pythonhosted.org/packages/af/d3/76bd4ed4317e7119c2b7f57c3f6934aba26d277acc6309f873341640e21f/frozenlist-1.8.0-cp314-cp314t-win32.whl", hash = "sha256:342c97bf697ac5480c0a7ec73cd700ecfa5a8a40ac923bd035484616efecc2df", size = 44676, upload-time = "2025-10-06T05:37:52.222Z" },
+ { url = "https://files.pythonhosted.org/packages/89/76/c615883b7b521ead2944bb3480398cbb07e12b7b4e4d073d3752eb721558/frozenlist-1.8.0-cp314-cp314t-win_amd64.whl", hash = "sha256:06be8f67f39c8b1dc671f5d83aaefd3358ae5cdcf8314552c57e7ed3e6475bdd", size = 49451, upload-time = "2025-10-06T05:37:53.425Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/a3/5982da14e113d07b325230f95060e2169f5311b1017ea8af2a29b374c289/frozenlist-1.8.0-cp314-cp314t-win_arm64.whl", hash = "sha256:102e6314ca4da683dca92e3b1355490fed5f313b768500084fbe6371fddfdb79", size = 42507, upload-time = "2025-10-06T05:37:54.513Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/9a/e35b4a917281c0b8419d4207f4334c8e8c5dbf4f3f5f9ada73958d937dcc/frozenlist-1.8.0-py3-none-any.whl", hash = "sha256:0c18a16eab41e82c295618a77502e17b195883241c563b00f0aa5106fc4eaa0d", size = 13409, upload-time = "2025-10-06T05:38:16.721Z" },
+]
+
+[[package]]
+name = "fsspec"
+version = "2026.6.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/10/a1/ae4e3e5003468d6391d2c77b6fa1cd73bd5d13511d81c642d7b28ac90ed4/fsspec-2026.6.0.tar.gz", hash = "sha256:f5bac145310fe30e16e1471bd6840b2d990d609e872251d7e674241822abf01a", size = 313646, upload-time = "2026-06-16T01:57:28.105Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e5/22/4222d7ddf3da30f363edaa98e329c2bce6c65497c9cb2810931c8b2c0fbc/fsspec-2026.6.0-py3-none-any.whl", hash = "sha256:02e0b71817df9b2169dc30a16832045764def1191b43dcff5bb85bdee212d2a1", size = 203949, upload-time = "2026-06-16T01:57:26.358Z" },
+]
+
+[[package]]
+name = "google-crc32c"
+version = "1.8.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/03/41/4b9c02f99e4c5fb477122cd5437403b552873f014616ac1d19ac8221a58d/google_crc32c-1.8.0.tar.gz", hash = "sha256:a428e25fb7691024de47fecfbff7ff957214da51eddded0da0ae0e0f03a2cf79", size = 14192, upload-time = "2025-12-16T00:35:25.142Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e9/5f/7307325b1198b59324c0fa9807cafb551afb65e831699f2ce211ad5c8240/google_crc32c-1.8.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:4b8286b659c1335172e39563ab0a768b8015e88e08329fa5321f774275fc3113", size = 31300, upload-time = "2025-12-16T00:21:56.723Z" },
+ { url = "https://files.pythonhosted.org/packages/21/8e/58c0d5d86e2220e6a37befe7e6a94dd2f6006044b1a33edf1ff6d9f7e319/google_crc32c-1.8.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:2a3dc3318507de089c5384cc74d54318401410f82aa65b2d9cdde9d297aca7cb", size = 30867, upload-time = "2025-12-16T00:38:31.302Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/a9/a780cc66f86335a6019f557a8aaca8fbb970728f0efd2430d15ff1beae0e/google_crc32c-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:14f87e04d613dfa218d6135e81b78272c3b904e2a7053b841481b38a7d901411", size = 33364, upload-time = "2025-12-16T00:40:22.96Z" },
+ { url = "https://files.pythonhosted.org/packages/21/3f/3457ea803db0198c9aaca2dd373750972ce28a26f00544b6b85088811939/google_crc32c-1.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb5c869c2923d56cb0c8e6bcdd73c009c36ae39b652dbe46a05eb4ef0ad01454", size = 33740, upload-time = "2025-12-16T00:40:23.96Z" },
+ { url = "https://files.pythonhosted.org/packages/df/c0/87c2073e0c72515bb8733d4eef7b21548e8d189f094b5dad20b0ecaf64f6/google_crc32c-1.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:3cc0c8912038065eafa603b238abf252e204accab2a704c63b9e14837a854962", size = 34437, upload-time = "2025-12-16T00:35:21.395Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/db/000f15b41724589b0e7bc24bc7a8967898d8d3bc8caf64c513d91ef1f6c0/google_crc32c-1.8.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:3ebb04528e83b2634857f43f9bb8ef5b2bbe7f10f140daeb01b58f972d04736b", size = 31297, upload-time = "2025-12-16T00:23:20.709Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/0d/8ebed0c39c53a7e838e2a486da8abb0e52de135f1b376ae2f0b160eb4c1a/google_crc32c-1.8.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:450dc98429d3e33ed2926fc99ee81001928d63460f8538f21a5d6060912a8e27", size = 30867, upload-time = "2025-12-16T00:43:14.628Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/42/b468aec74a0354b34c8cbf748db20d6e350a68a2b0912e128cabee49806c/google_crc32c-1.8.0-cp313-cp313-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:3b9776774b24ba76831609ffbabce8cdf6fa2bd5e9df37b594221c7e333a81fa", size = 33344, upload-time = "2025-12-16T00:40:24.742Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/e8/b33784d6fc77fb5062a8a7854e43e1e618b87d5ddf610a88025e4de6226e/google_crc32c-1.8.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:89c17d53d75562edfff86679244830599ee0a48efc216200691de8b02ab6b2b8", size = 33694, upload-time = "2025-12-16T00:40:25.505Z" },
+ { url = "https://files.pythonhosted.org/packages/92/b1/d3cbd4d988afb3d8e4db94ca953df429ed6db7282ed0e700d25e6c7bfc8d/google_crc32c-1.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:57a50a9035b75643996fbf224d6661e386c7162d1dfdab9bc4ca790947d1007f", size = 34435, upload-time = "2025-12-16T00:35:22.107Z" },
+ { url = "https://files.pythonhosted.org/packages/21/88/8ecf3c2b864a490b9e7010c84fd203ec8cf3b280651106a3a74dd1b0ca72/google_crc32c-1.8.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:e6584b12cb06796d285d09e33f63309a09368b9d806a551d8036a4207ea43697", size = 31301, upload-time = "2025-12-16T00:24:48.527Z" },
+ { url = "https://files.pythonhosted.org/packages/36/c6/f7ff6c11f5ca215d9f43d3629163727a272eabc356e5c9b2853df2bfe965/google_crc32c-1.8.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:f4b51844ef67d6cf2e9425983274da75f18b1597bb2c998e1c0a0e8d46f8f651", size = 30868, upload-time = "2025-12-16T00:48:12.163Z" },
+ { url = "https://files.pythonhosted.org/packages/56/15/c25671c7aad70f8179d858c55a6ae8404902abe0cdcf32a29d581792b491/google_crc32c-1.8.0-cp314-cp314-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b0d1a7afc6e8e4635564ba8aa5c0548e3173e41b6384d7711a9123165f582de2", size = 33381, upload-time = "2025-12-16T00:40:26.268Z" },
+ { url = "https://files.pythonhosted.org/packages/42/fa/f50f51260d7b0ef5d4898af122d8a7ec5a84e2984f676f746445f783705f/google_crc32c-1.8.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8b3f68782f3cbd1bce027e48768293072813469af6a61a86f6bb4977a4380f21", size = 33734, upload-time = "2025-12-16T00:40:27.028Z" },
+ { url = "https://files.pythonhosted.org/packages/08/a5/7b059810934a09fb3ccb657e0843813c1fee1183d3bc2c8041800374aa2c/google_crc32c-1.8.0-cp314-cp314-win_amd64.whl", hash = "sha256:d511b3153e7011a27ab6ee6bb3a5404a55b994dc1a7322c0b87b29606d9790e2", size = 34878, upload-time = "2025-12-16T00:35:23.142Z" },
+]
+
+[[package]]
+name = "greenlet"
+version = "3.5.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e2/f1/fbbfef6af0bad0548f09bc28948ea3c275b4edb19e17fc5ca9900a6a634d/greenlet-3.5.3.tar.gz", hash = "sha256:a61efc018fd3eb317eeca31aba90ee9e7f26f22884a79b6c6ec715bf71bb62f1", size = 200270, upload-time = "2026-06-26T19:28:24.832Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5d/6e/4c37d51a2b7f82d2ff11bb6b5f7d766d9a011726624af255e843727627a3/greenlet-3.5.3-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:719757059f5a53fd0dde23f78cffeafcdd97b21c850ddb7ca684a3c1a1f122e2", size = 288685, upload-time = "2026-06-26T18:22:08.977Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/73/815dd90131c1b71ebdf53dbc7c276cafec2a1173b97559f97aba72724a87/greenlet-3.5.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:efa9f765dd09f9d0cdac651ffdf631ee59ec5dc6ee7a73e0c012ba9c52fbdf5b", size = 604761, upload-time = "2026-06-26T19:07:10.114Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/57/079cfe76bcef36b153b25607ee91c6fcb58f17f8b23c86bbbeabe0c88d72/greenlet-3.5.3-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7faba15ac005376e02a0384504e0243be3370ce010296a44a820feb342b505ab", size = 617044, upload-time = "2026-06-26T19:10:07.25Z" },
+ { url = "https://files.pythonhosted.org/packages/37/87/b4d095775a3fb1bcafbb483fc206b27ebb785724c83051447737085dc54e/greenlet-3.5.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:87142215824be6ac05e2e8e2786eec307ccbc27c36723c3881959df654af6861", size = 614244, upload-time = "2026-06-26T18:32:17.594Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/70/7559b609683650fa2b95b8ab84b4ab0b26556a635d19675e12aa832d826d/greenlet-3.5.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:215275b1b49320987352e6c1b054acca0064f965a2c66992bed9a6f7d913f149", size = 1574210, upload-time = "2026-06-26T19:09:03.077Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/73/be55392074c60fc37655ca40fa6022457bfbf6718e9e342a7b0b41f96dd2/greenlet-3.5.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6b1b0eed82364b0e32c4ea0f221452d33e6bb17ae094d9f72aed9851812747ea", size = 1638627, upload-time = "2026-06-26T18:31:44.748Z" },
+ { url = "https://files.pythonhosted.org/packages/14/40/c57489acf8e37d74e2913d4eff63aa0dba17acccc4bdeef874dde2dbbec9/greenlet-3.5.3-cp312-cp312-win_amd64.whl", hash = "sha256:cde8adafa2365676f74a979744629589999093bc86e2484214f58e61df08902c", size = 239882, upload-time = "2026-06-26T18:23:27.518Z" },
+ { url = "https://files.pythonhosted.org/packages/71/fd/6fea0e3d6600f785069481ee637e09378dd4118acdfd38ad88ae2db31c98/greenlet-3.5.3-cp312-cp312-win_arm64.whl", hash = "sha256:c4e7b79d83805475f0102008843f6eb45fd3bb0b2e88c774adab5fbaab27117d", size = 238211, upload-time = "2026-06-26T18:22:37.671Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/ff/a620267401db30a50cc8450ee90730e2d4a85658c055c0e760d4ed47fb13/greenlet-3.5.3-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:c8d87c2134d871df96ecdea9cec7cbaab286dadab0f56476e57aaf9e8ac11550", size = 287609, upload-time = "2026-06-26T18:21:14.724Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/fa/5401ac78021c826a25b6dde0c705e0a8f29b617509f9185a31dac15fbe1b/greenlet-3.5.3-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a2d185dd1621757e70c3861cceffd5317ab4e7ed7eb09c82994828468527ade5", size = 607435, upload-time = "2026-06-26T19:07:11.412Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/76/1dc144a2e56e65d36405078ed774224375ea520a1870a6e46e08bb4ac7bf/greenlet-3.5.3-cp313-cp313-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1c514a468149bf8fbbab874188a3535cd8a48a3e353eb53a3d424296f8dbacd3", size = 619787, upload-time = "2026-06-26T19:10:08.396Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/87/c298cee62df1de4ad7fec32abda73526cff347fd143a6ed4ac369246668a/greenlet-3.5.3-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:915f887cf2682b66419b879423a2e072634aa7b7dce6f3ada4957cfced3f1e9a", size = 616786, upload-time = "2026-06-26T18:32:19.128Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/2e/e6f009885ed0705ccf33fe0583c117cfd03cde77e31a596dd5785a30762b/greenlet-3.5.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:766cfd421c13e450feb340cd472a3ed9957d438727b7b4593ad7c76c5d2b0deb", size = 1574316, upload-time = "2026-06-26T19:09:04.273Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/fe/43fd110b01e40da0adb7c90ac7ea744bef2d43dca00de5095fd2351c2a68/greenlet-3.5.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:2ecda9ec22edf38fa389369eaed8c3d37c05f3c54e69f69438dbb2cc1de1458b", size = 1638614, upload-time = "2026-06-26T18:31:46.297Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/7c/062447147a61f8b4337b156fe70d32a165fcf2f89d7ca6255e572806705c/greenlet-3.5.3-cp313-cp313-win_amd64.whl", hash = "sha256:c82304750f057167ff60d188df1d0cc1764ce9567eadf03e6a7443bcedd0b30b", size = 239850, upload-time = "2026-06-26T18:21:54.613Z" },
+ { url = "https://files.pythonhosted.org/packages/c7/7e/220a7f5824a64a60443fc03b39dfac4ea63a7fb6d481efa27eafa928e7f4/greenlet-3.5.3-cp313-cp313-win_arm64.whl", hash = "sha256:dc133a1569ee667b2a6ef56ce551084aeefd87a5acbc4736d336d1e2edc6cfc4", size = 238141, upload-time = "2026-06-26T18:22:48.507Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/93/43e116ee114b28737ba7e12952a0d4e2f55944d0f84e42bc91ba7192a3c9/greenlet-3.5.3-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:fd2e02fa07485778536a036222d616ab957b1d533f36b3ed98ce725d9c9d3117", size = 288202, upload-time = "2026-06-26T18:23:49.604Z" },
+ { url = "https://files.pythonhosted.org/packages/82/2f/146d218299046a43d1f029fd544b3d110d0f175a09c715c7e8da4a4a345d/greenlet-3.5.3-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:df0a0628d1597eb0897b62f55d1343f772405fd25f3b2a796c76874b0c2e22e8", size = 654096, upload-time = "2026-06-26T19:07:12.71Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/cc/04738cafb3f45fa991ea44f9de94c47dcec964f5a972300988a6751f49d9/greenlet-3.5.3-cp314-cp314-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ebd933a6adabc298bab47731a130fe6bfb888bd934eee37810f151159544540d", size = 666304, upload-time = "2026-06-26T19:10:09.503Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/aa/4e0dad5e605c270c784ab911c43da6adb136ccd4d81180f763ca429a723d/greenlet-3.5.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4b9d501b40e80b70e32323c799dd9b420a5577a9601469d362ae1ffb690f3a7c", size = 663635, upload-time = "2026-06-26T18:32:20.802Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/50/13efdbea246fe3d3b735e191fec08fb50809f53cd2383ebe123d0809e44b/greenlet-3.5.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a1fad1d11e7d6aab184107baa8e4ece11ccba3ec9599cd7efa5ff4d70d43256a", size = 1621252, upload-time = "2026-06-26T19:09:05.647Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/22/c0a336ae4a1410fd5f5121098e5bfbf1865f64c5ef80b4b5412886c4a332/greenlet-3.5.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:fad5aec764399f1b5cc347ad250a59660f20c8f8888ea6bae1f93b769cce1154", size = 1684824, upload-time = "2026-06-26T18:31:47.738Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/94/91aec0030bea75c4b3244251d0de60a1f3432d1ecb53ab6c437fb5c3ba61/greenlet-3.5.3-cp314-cp314-win_amd64.whl", hash = "sha256:7669aa24cf2a1041d6f7899575b494a3ab4cf68bfcc8609b1dc0be7272db835e", size = 240754, upload-time = "2026-06-26T18:22:15.669Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/06/68d0983e79e02138f64b4d303c500c27ddb48e5e77f3debb80888a921eae/greenlet-3.5.3-cp314-cp314-win_arm64.whl", hash = "sha256:5b4807c4082c9d1b6d9eed56fcd041863e37f2228106eef24c30ca096e238605", size = 239549, upload-time = "2026-06-26T18:22:42.996Z" },
+ { url = "https://files.pythonhosted.org/packages/91/95/3e161213d7f1d378d15aa9e792093e9bfe01844680d04b7fd6e0107c9098/greenlet-3.5.3-cp314-cp314t-macosx_11_0_universal2.whl", hash = "sha256:271a8ea7c1024e8a0d7dd2be66dd66dda8a07193f41a17b9e924f7600f5b62be", size = 296389, upload-time = "2026-06-26T18:22:20.657Z" },
+ { url = "https://files.pythonhosted.org/packages/00/92/715c44721abe2b4d1ae9abde4179411868a5bff312479f54e105d372f131/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:19131729ae0ddc3c2e1ef85e650169b5e37ee32e400f215f78b94d7b0d567310", size = 653382, upload-time = "2026-06-26T19:07:14.209Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/83/37a10372a1090a6624cca8e74c12df1a36c2dc36429ed0255b7fb1aeee23/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1540dd8e5fc2a5aec40fbb98ef8e149fa47c89a4b4a1cf2575a14d3d1869d7a8", size = 659401, upload-time = "2026-06-26T19:10:10.876Z" },
+ { url = "https://files.pythonhosted.org/packages/db/e2/d1509cad4207da559cc42986ecdd8fc67ad0d1bba2bf03023c467fd5e0f3/greenlet-3.5.3-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e81fa194a1d20967877bdf9c7794db2bc99063e5be36aee710c08f04c5bb087f", size = 656969, upload-time = "2026-06-26T18:32:22.272Z" },
+ { url = "https://files.pythonhosted.org/packages/86/7d/eaf70de20aadca3a5884aec58362861c64ce45e7b277f47ed026926a3b89/greenlet-3.5.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:55cf4d777485d43110e47133cbba6d74a8885a87ec1227ef0267f9ee80c5aa21", size = 1617822, upload-time = "2026-06-26T19:09:06.893Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/f9/414d38fc400ae4350d4185eaad1827676f7cf5287b9136e0ed1cbbe20a7f/greenlet-3.5.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:12a248ba75f6a9a236375f52296c498c89ff1d8badf32deb9eca7abd5853f7da", size = 1677983, upload-time = "2026-06-26T18:31:49.396Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/15/7edb977e08f9bff702fe42d6c902702786ff6b9694058b4e6a2a6ac90e57/greenlet-3.5.3-cp314-cp314t-win_amd64.whl", hash = "sha256:efc6bd60ea02e085862c74a3ef64b147ffc6f1a5ea7d9f26e7a939943f68c1e3", size = 243626, upload-time = "2026-06-26T18:24:41.485Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/8a/93928dce91e6b3598b5e779e8d1fd6576a504640c58e78627077f6a7a91a/greenlet-3.5.3-cp315-cp315-macosx_11_0_universal2.whl", hash = "sha256:ea03f2f04367845d6b58eeed276e1e56e51f0b97d8ad5a88a7d20a91dc9056cc", size = 288860, upload-time = "2026-06-26T18:22:48.07Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/ca/69db42d447a1378043e2c8f19c09cbbd1263371505053c496b49066d3d16/greenlet-3.5.3-cp315-cp315-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:78dbef602fda6d97d957eb7937f70c9ce9e9527330347f8f6b6f9e554a9e7a47", size = 659747, upload-time = "2026-06-26T19:07:15.565Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/0b/af7ac2ef8dd41e3da1a40dda6305c23b9a03e13ba975ec916357b50f8575/greenlet-3.5.3-cp315-cp315-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6f73857adb8fee13fa56c172bd11262f888c0c648f9fea113e777bb2c7904a81", size = 670419, upload-time = "2026-06-26T19:10:12.293Z" },
+ { url = "https://files.pythonhosted.org/packages/51/1e/1d51640cacbfc455dbe9f9a9f594c49e4e244f63b9971a2f4764e46cc53d/greenlet-3.5.3-cp315-cp315-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:232fec92e823addaf02d9472cf7381e24a1d046a6ced1103c5caa4c21b9dfc1d", size = 668787, upload-time = "2026-06-26T18:32:24.298Z" },
+ { url = "https://files.pythonhosted.org/packages/21/66/4030d5b0b5894500023f003bb054d9bb354dfbd1e186c3a296759172f5f5/greenlet-3.5.3-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:2421c3564da9429d5586d46ca31ebb26516b5498a802cf65c041a8e8a8980d34", size = 1626305, upload-time = "2026-06-26T19:09:08.281Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/50/5221371c7550108dfa3c378debc41d032aa9c78e89abb01d8011cfc93289/greenlet-3.5.3-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:e0f0d160f0b2e558e6c75f7930967183255dc9735e5f5b8cae58ee09c9576d8b", size = 1688631, upload-time = "2026-06-26T18:31:51.278Z" },
+ { url = "https://files.pythonhosted.org/packages/68/5d/00d469daae3c65d2bf620b10eee82eb022127d483c6bc8c69fae6f3fbf17/greenlet-3.5.3-cp315-cp315-win_amd64.whl", hash = "sha256:dd99329bbc15ca78dcc583dba05d0b1b0bae01ab6c2174989f5aaee3e41ac930", size = 241027, upload-time = "2026-06-26T18:22:38.203Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/e8/883785b44c5780ed71e83d3e4437e710470be17a2e181e8b601e2da0dc4a/greenlet-3.5.3-cp315-cp315-win_arm64.whl", hash = "sha256:499fef2acede88c1864a57bb586b4bf533c81e1b82df7ab93451cdb47dfec227", size = 240085, upload-time = "2026-06-26T18:23:54.217Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/da/4f4a8450962fad137c1c8981a3f1b8919d06c829993d4d476f9c525d5173/greenlet-3.5.3-cp315-cp315t-macosx_11_0_universal2.whl", hash = "sha256:176bc16a721fa5fc294d70b87b4dfa5fbdd251b3da5d5372735ecef9bd7d6d0c", size = 297221, upload-time = "2026-06-26T18:23:27.176Z" },
+ { url = "https://files.pythonhosted.org/packages/57/66/b3bfae3e220a9b63ea539a0eea681800c69ab1aada757eae8789f183e7ce/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:629b614d2b786e89c50440e246f33eea78f58a962d0bdbbcc809e6d13605903f", size = 657221, upload-time = "2026-06-26T19:07:16.973Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/81/b6d4d73a709684fc77e7fa034d7c2fe82cffa9fc920fadcaa659c2626213/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2b2e857ae16f5f72142edf75f9f176fe7526ba19a2841df1420516f83831c9f2", size = 663226, upload-time = "2026-06-26T19:10:13.723Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/07/e210b02b589f16e74ff48b730690e4a34ffe984219fce4f3c1a0e7ec8545/greenlet-3.5.3-cp315-cp315t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e515757e2e36bcbf1fad09a46e1557e8b1ae1797d4b44d09da7deed88ad28608", size = 660802, upload-time = "2026-06-26T18:32:26.081Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/2e/5303eb3fa06bca089060f479707182a93e360683bc252acf846c3090d34e/greenlet-3.5.3-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:b363d46ed1ea431825fdb01471bb024fc08399bad1572a616e853c7684415adb", size = 1622157, upload-time = "2026-06-26T19:09:09.527Z" },
+ { url = "https://files.pythonhosted.org/packages/54/70/50de47a488f14df260b50ae34fb5d56016e308b098eab02c878b5223c26a/greenlet-3.5.3-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:e44da2f5bbdaabaf7d80b73dbb430c7035771e9f244e3c8b769715c9d8fa0a16", size = 1681159, upload-time = "2026-06-26T18:31:52.986Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/13/1055e1dda7882073eda533e2b96c62e55bbd2db7fda6d5ece992febc7071/greenlet-3.5.3-cp315-cp315t-win_amd64.whl", hash = "sha256:8ff8bed3e3baa20a3ea261ce00526f1898ad4801d4886fd2220580ee0ad8fadf", size = 244007, upload-time = "2026-06-26T18:22:04.353Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/0d/ca7d15afbdc397e3401134c9e1800d51d12b829661786187a4ad08fe484f/greenlet-3.5.3-cp315-cp315t-win_arm64.whl", hash = "sha256:b7068bd09f761f3f5b4d214c2bed063186b2a86148c740b3873e3f56d79bac31", size = 242586, upload-time = "2026-06-26T18:23:37.93Z" },
+]
+
+[[package]]
+name = "grpclib"
+version = "0.4.9"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "h2" },
+ { name = "multidict" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/5b/28/5a2c299ec82a876a252c5919aa895a6f1d1d35c96417c5ce4a4660dc3a80/grpclib-0.4.9.tar.gz", hash = "sha256:cc589c330fa81004c6400a52a566407574498cb5b055fa927013361e21466c46", size = 84798, upload-time = "2025-12-14T22:23:14.349Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5c/90/b0cbbd9efcc82816c58f31a34963071aa19fb792a212a5d9caf8e0fc3097/grpclib-0.4.9-py3-none-any.whl", hash = "sha256:7762ec1c8ed94dfad597475152dd35cbd11aecaaca2f243e29702435ca24cf0e", size = 77063, upload-time = "2025-12-14T22:23:13.224Z" },
+]
+
+[[package]]
+name = "h11"
+version = "0.16.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" },
+]
+
+[[package]]
+name = "h2"
+version = "4.3.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "hpack" },
+ { name = "hyperframe" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/1d/17/afa56379f94ad0fe8defd37d6eb3f89a25404ffc71d4d848893d270325fc/h2-4.3.0.tar.gz", hash = "sha256:6c59efe4323fa18b47a632221a1888bd7fde6249819beda254aeca909f221bf1", size = 2152026, upload-time = "2025-08-23T18:12:19.778Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/69/b2/119f6e6dcbd96f9069ce9a2665e0146588dc9f88f29549711853645e736a/h2-4.3.0-py3-none-any.whl", hash = "sha256:c438f029a25f7945c69e0ccf0fb951dc3f73a5f6412981daee861431b70e2bdd", size = 61779, upload-time = "2025-08-23T18:12:17.779Z" },
+]
+
+[[package]]
+name = "h5py"
+version = "3.16.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/db/33/acd0ce6863b6c0d7735007df01815403f5589a21ff8c2e1ee2587a38f548/h5py-3.16.0.tar.gz", hash = "sha256:a0dbaad796840ccaa67a4c144a0d0c8080073c34c76d5a6941d6818678ef2738", size = 446526, upload-time = "2026-03-06T13:49:08.07Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c8/c0/5d4119dba94093bbafede500d3defd2f5eab7897732998c04b54021e530b/h5py-3.16.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:c5313566f4643121a78503a473f0fb1e6dcc541d5115c44f05e037609c565c4d", size = 3685604, upload-time = "2026-03-06T13:48:04.198Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/42/c84efcc1d4caebafb1ecd8be4643f39c85c47a80fe254d92b8b43b1eadaf/h5py-3.16.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:42b012933a83e1a558c673176676a10ce2fd3759976a0fedee1e672d1e04fc9d", size = 3061940, upload-time = "2026-03-06T13:48:05.783Z" },
+ { url = "https://files.pythonhosted.org/packages/89/84/06281c82d4d1686fde1ac6b0f307c50918f1c0151062445ab3b6fa5a921d/h5py-3.16.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:ff24039e2573297787c3063df64b60aab0591980ac898329a08b0320e0cf2527", size = 5198852, upload-time = "2026-03-06T13:48:07.482Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/e9/1a19e42cd43cc1365e127db6aae85e1c671da1d9a5d746f4d34a50edb577/h5py-3.16.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:dfc21898ff025f1e8e67e194965a95a8d4754f452f83454538f98f8a3fcb207e", size = 5405250, upload-time = "2026-03-06T13:48:09.628Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/8e/9790c1655eabeb85b92b1ecab7d7e62a2069e53baefd58c98f0909c7a948/h5py-3.16.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:698dd69291272642ffda44a0ecd6cd3bda5faf9621452d255f57ce91487b9794", size = 5190108, upload-time = "2026-03-06T13:48:11.26Z" },
+ { url = "https://files.pythonhosted.org/packages/51/d7/ab693274f1bd7e8c5f9fdd6c7003a88d59bedeaf8752716a55f532924fbb/h5py-3.16.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:2b2c02b0a160faed5fb33f1ba8a264a37ee240b22e049ecc827345d0d9043074", size = 5419216, upload-time = "2026-03-06T13:48:13.322Z" },
+ { url = "https://files.pythonhosted.org/packages/03/c1/0976b235cf29ead553e22f2fb6385a8252b533715e00d0ae52ed7b900582/h5py-3.16.0-cp312-cp312-win_amd64.whl", hash = "sha256:96b422019a1c8975c2d5dadcf61d4ba6f01c31f92bbde6e4649607885fe502d6", size = 3182868, upload-time = "2026-03-06T13:48:15.759Z" },
+ { url = "https://files.pythonhosted.org/packages/14/d9/866b7e570b39070f92d47b0ff1800f0f8239b6f9e45f02363d7112336c1f/h5py-3.16.0-cp312-cp312-win_arm64.whl", hash = "sha256:39c2838fb1e8d97bcf1755e60ad1f3dd76a7b2a475928dc321672752678b96db", size = 2653286, upload-time = "2026-03-06T13:48:17.279Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/9e/6142ebfda0cb6e9349c091eae73c2e01a770b7659255248d637bec54a88b/h5py-3.16.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:370a845f432c2c9619db8eed334d1e610c6015796122b0e57aa46312c22617d9", size = 3671808, upload-time = "2026-03-06T13:48:19.737Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/65/5e088a45d0f43cd814bc5bec521c051d42005a472e804b1a36c48dada09b/h5py-3.16.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:42108e93326c50c2810025aade9eac9d6827524cdccc7d4b75a546e5ab308edb", size = 3045837, upload-time = "2026-03-06T13:48:21.854Z" },
+ { url = "https://files.pythonhosted.org/packages/da/1e/6172269e18cc5a484e2913ced33339aad588e02ba407fafd00d369e22ef3/h5py-3.16.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:099f2525c9dcf28de366970a5fb34879aab20491589fa89ce2863a84218bb524", size = 5193860, upload-time = "2026-03-06T13:48:24.071Z" },
+ { url = "https://files.pythonhosted.org/packages/bd/98/ef2b6fe2903e377cbe870c3b2800d62552f1e3dbe81ce49e1923c53d1c5c/h5py-3.16.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:9300ad32dea9dfc5171f94d5f6948e159ed93e4701280b0f508773b3f582f402", size = 5400417, upload-time = "2026-03-06T13:48:25.728Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/81/5b62d760039eed64348c98129d17061fdfc7839fc9c04eaaad6dee1004e4/h5py-3.16.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:171038f23bccddfc23f344cadabdfc9917ff554db6a0d417180d2747fe4c75a7", size = 5185214, upload-time = "2026-03-06T13:48:27.436Z" },
+ { url = "https://files.pythonhosted.org/packages/28/c4/532123bcd9080e250696779c927f2cb906c8bf3447df98f5ceb8dcded539/h5py-3.16.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7e420b539fb6023a259a1b14d4c9f6df8cf50d7268f48e161169987a57b737ff", size = 5414598, upload-time = "2026-03-06T13:48:29.49Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/d9/a27997f84341fc0dfcdd1fe4179b6ba6c32a7aa880fdb8c514d4dad6fba3/h5py-3.16.0-cp313-cp313-win_amd64.whl", hash = "sha256:18f2bbcd545e6991412253b98727374c356d67caa920e68dc79eab36bf5fedad", size = 3175509, upload-time = "2026-03-06T13:48:31.131Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/23/bb8647521d4fd770c30a76cfc6cb6a2f5495868904054e92f2394c5a78ff/h5py-3.16.0-cp313-cp313-win_arm64.whl", hash = "sha256:656f00e4d903199a1d58df06b711cf3ca632b874b4207b7dbec86185b5c8c7d4", size = 2647362, upload-time = "2026-03-06T13:48:33.411Z" },
+ { url = "https://files.pythonhosted.org/packages/48/3c/7fcd9b4c9eed82e91fb15568992561019ae7a829d1f696b2c844355d95dd/h5py-3.16.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:9c9d307c0ef862d1cd5714f72ecfafe0a5d7529c44845afa8de9f46e5ba8bd65", size = 3678608, upload-time = "2026-03-06T13:48:35.183Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/b7/9366ed44ced9b7ef357ab48c94205280276db9d7f064aa3012a97227e966/h5py-3.16.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:8c1eff849cdd53cbc73c214c30ebdb6f1bb8b64790b4b4fc36acdb5e43570210", size = 3054773, upload-time = "2026-03-06T13:48:37.139Z" },
+ { url = "https://files.pythonhosted.org/packages/58/a5/4964bc0e91e86340c2bbda83420225b2f770dcf1eb8a39464871ad769436/h5py-3.16.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:e2c04d129f180019e216ee5f9c40b78a418634091c8782e1f723a6ca3658b965", size = 5198886, upload-time = "2026-03-06T13:48:38.879Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/16/d905e7f53e661ce2c24686c38048d8e2b750ffc4350009d41c4e6c6c9826/h5py-3.16.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:e4360f15875a532bc7b98196c7592ed4fc92672a57c0a621355961cafb17a6dd", size = 5404883, upload-time = "2026-03-06T13:48:41.324Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/f2/58f34cb74af46d39f4cd18ea20909a8514960c5a3e5b92fd06a28161e0a8/h5py-3.16.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:3fae9197390c325e62e0a1aa977f2f62d994aa87aab182abbea85479b791197c", size = 5192039, upload-time = "2026-03-06T13:48:43.117Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/ca/934a39c24ce2e2db017268c08da0537c20fa0be7e1549be3e977313fc8f5/h5py-3.16.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:43259303989ac8adacc9986695b31e35dba6fd1e297ff9c6a04b7da5542139cc", size = 5421526, upload-time = "2026-03-06T13:48:44.838Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/14/615a450205e1b56d16c6783f5ccd116cde05550faad70ae077c955654a75/h5py-3.16.0-cp314-cp314-win_amd64.whl", hash = "sha256:fa48993a0b799737ba7fd21e2350fa0a60701e58180fae9f2de834bc39a147ab", size = 3183263, upload-time = "2026-03-06T13:48:47.117Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/48/a6faef5ed632cae0c65ac6b214a6614a0b510c3183532c521bdb0055e117/h5py-3.16.0-cp314-cp314-win_arm64.whl", hash = "sha256:1897a771a7f40d05c262fc8f37376ec37873218544b70216872876c627640f63", size = 2663450, upload-time = "2026-03-06T13:48:48.707Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/32/0c8bb8aedb62c772cf7c1d427c7d1951477e8c2835f872bc0a13d1f85f86/h5py-3.16.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:15922e485844f77c0b9d275396d435db3baa58292a9c2176a386e072e0cf2491", size = 3760693, upload-time = "2026-03-06T13:48:50.453Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/1f/fcc5977d32d6387c5c9a694afee716a5e20658ac08b3ff24fdec79fb05f2/h5py-3.16.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:df02dd29bd247f98674634dfe41f89fd7c16ba3d7de8695ec958f58404a4e618", size = 3181305, upload-time = "2026-03-06T13:48:52.221Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/a1/af87f64b9f986889884243643621ebbd4ac72472ba8ec8cec891ac8e2ca1/h5py-3.16.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:0f456f556e4e2cebeebd9d66adf8dc321770a42593494a0b6f0af54a7567b242", size = 5074061, upload-time = "2026-03-06T13:48:54.089Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/d0/146f5eaff3dc246a9c7f6e5e4f42bd45cc613bce16693bcd4d1f7c958bf5/h5py-3.16.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:3e6cb3387c756de6a9492d601553dffea3fe11b5f22b443aac708c69f3f55e16", size = 5279216, upload-time = "2026-03-06T13:48:56.75Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/9d/12a13424f1e604fc7df9497b73c0356fb78c2fb206abd7465ce47226e8fd/h5py-3.16.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8389e13a1fd745ad2856873e8187fd10268b2d9677877bb667b41aebd771d8b7", size = 5070068, upload-time = "2026-03-06T13:48:59.169Z" },
+ { url = "https://files.pythonhosted.org/packages/41/8c/bbe98f813722b4873818a8db3e15aa3e625b59278566905ac439725e8070/h5py-3.16.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:346df559a0f7dcb31cf8e44805319e2ab24b8957c45e7708ce503b2ec79ba725", size = 5300253, upload-time = "2026-03-06T13:49:02.033Z" },
+ { url = "https://files.pythonhosted.org/packages/32/9e/87e6705b4d6890e7cecdf876e2a7d3e40654a2ae37482d79a6f1b87f7b92/h5py-3.16.0-cp314-cp314t-win_amd64.whl", hash = "sha256:4c6ab014ab704b4feaa719ae783b86522ed0bf1f82184704ed3c9e4e3228796e", size = 3381671, upload-time = "2026-03-06T13:49:04.351Z" },
+ { url = "https://files.pythonhosted.org/packages/96/91/9fad90cfc5f9b2489c7c26ad897157bce82f0e9534a986a221b99760b23b/h5py-3.16.0-cp314-cp314t-win_arm64.whl", hash = "sha256:faca8fb4e4319c09d83337adc80b2ca7d5c5a343c2d6f1b6388f32cfecca13c1", size = 2740706, upload-time = "2026-03-06T13:49:06.347Z" },
+]
+
+[[package]]
+name = "hf-xet"
+version = "1.5.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/63/39/67be8d71f900d9a55761b6022821d6679fb56c64f1b6063d5af2c2606727/hf_xet-1.5.2.tar.gz", hash = "sha256:73044bd31bae33c984af832d19c752a0dffb67518fee9ddbd91d616e1101cf47", size = 903674, upload-time = "2026-07-16T17:29:56.833Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/29/be/525eabac5d1736b679c39e342ecd4292534012546a2d18f0043c8e3b6021/hf_xet-1.5.2-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:4a5ecb9cda8512ba2aa8ee5d37c87a1422992165892d653098c7b90247481c3b", size = 4064284, upload-time = "2026-07-16T17:29:29.907Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/3f/699749dd78442480eda4e4fca494284b0e3542e4063cc37654d5fdc929e6/hf_xet-1.5.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:8764488197c1d7b1378c8438c18d2eea902e150dbca0b0f0d2d32603fb9b5576", size = 3828537, upload-time = "2026-07-16T17:29:31.549Z" },
+ { url = "https://files.pythonhosted.org/packages/22/d7/2658ac0a5b9f4664ca27ce31bd015044fe9dea50ed455fb5197aba819c11/hf_xet-1.5.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8d7446f72abbf7e01ca5ff131786bc2e74a56393462c17a6bf1e303fbab81db4", size = 4417133, upload-time = "2026-07-16T17:29:33.391Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/58/8343f3cb63c8fa058d576136df3871550f7d5214a8f048a7ea2eab6ac906/hf_xet-1.5.2-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:580e59e29bf37aece1f2b68537de1e3fb04f43a23d910dcf6f128280b5bfbba4", size = 4212613, upload-time = "2026-07-16T17:29:34.989Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/33/a968f4e4535037b36941ec00714625fb60e026302407e7e26ca9f3e65f4e/hf_xet-1.5.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:bee28c619622d36968056532fd49cf2b35ca75099b1d616c31a618a893491380", size = 4412710, upload-time = "2026-07-16T17:29:36.646Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/d9/9e33981173dbaf194ba0015202b02d467b624d44d4eba89e1bf06c0d2995/hf_xet-1.5.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:e396ab0faf6298199ad7a95305c3ca8498cb825978a6485be6d00587ee4ec577", size = 4628455, upload-time = "2026-07-16T17:29:38.352Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/4b/cc682832de4264a03880a2d1b5ec3e1fab3bf307f508817250baafdb9996/hf_xet-1.5.2-cp314-cp314t-win_amd64.whl", hash = "sha256:fd3add255549e8ef58fa35b2e42dc016961c050600444e7d77d030ba6b57120e", size = 3979044, upload-time = "2026-07-16T17:29:40.329Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/09/b2cdf2a0fb39a08af3222b96092a36bd3b40c54123eef07de4422e870971/hf_xet-1.5.2-cp314-cp314t-win_arm64.whl", hash = "sha256:d6f9c58549407b84b9a5383afd68db0acc42345326a3159990b36a5ca8a20e4e", size = 3808037, upload-time = "2026-07-16T17:29:42.357Z" },
+ { url = "https://files.pythonhosted.org/packages/de/ba/2b70603c7552db82baeb2623e2336898304a17328845151be4fe1f48d420/hf_xet-1.5.2-cp38-abi3-macosx_10_12_x86_64.whl", hash = "sha256:f922b8f5fb84f1dd3d7ab7a1316354a1bca9b1c73ecfc19c76e51a2a49d29799", size = 4033760, upload-time = "2026-07-16T17:29:43.884Z" },
+ { url = "https://files.pythonhosted.org/packages/60/ac/b097a86a1e4a6098f3a79382643ab09d5733d87ccc864877ad1e12b49b70/hf_xet-1.5.2-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:045f84440c55cdeb659cf1a1dd48c77bcd0d2e93632e2fea8f2c3bdee79f38ed", size = 3841438, upload-time = "2026-07-16T17:29:45.539Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/35/db860aa3a0780660324a506ad4b3d322ddc6ecbba4b9340aed0942cbf21c/hf_xet-1.5.2-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:db78c39c83d6279daddc98e2238f373ab8980685556d42472b4ec51abcf03e8c", size = 4428006, upload-time = "2026-07-16T17:29:46.996Z" },
+ { url = "https://files.pythonhosted.org/packages/af/6b/832dd980af4b0c3ae0660e309285f2ffcdff2faa38129390dbb47aa4a3f9/hf_xet-1.5.2-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:7db73c810500c54c6760be8c39d4b2e476974de85424c50063efc22fdda13025", size = 4221099, upload-time = "2026-07-16T17:29:48.525Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/05/ae50f0d34e3254e6c3e208beb2519f6b8673016fc4b3643badaf6450d186/hf_xet-1.5.2-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:6395cfe3c9cbead4f16b31808b0e67eac428b66c656f856e99636adaddea878f", size = 4420766, upload-time = "2026-07-16T17:29:50.092Z" },
+ { url = "https://files.pythonhosted.org/packages/07/a9/c050bc2743a2bcd68928bfee157b08681667a164a24ec95fbfcfcd717e08/hf_xet-1.5.2-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:cde8cd167126bb6109b2ceb19b844433a4988643e8f3e01dd9dd0e4a34535097", size = 4636716, upload-time = "2026-07-16T17:29:51.62Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/f8/68b01c5c2edb56ac9a67b3d076ffddcb90867abaee923923eb34e7a14e76/hf_xet-1.5.2-cp38-abi3-win_amd64.whl", hash = "sha256:ecf63d1cb69a9a7319910f8f83fcf9b46e7a32dfcf4b8f8eeddb55f647306e65", size = 3988373, upload-time = "2026-07-16T17:29:53.395Z" },
+ { url = "https://files.pythonhosted.org/packages/39/c6/988383e9dc17294d536fcbcd6fd16eed882e411ad16c954984a53e47b09c/hf_xet-1.5.2-cp38-abi3-win_arm64.whl", hash = "sha256:1da28519496eb7c8094c11e4d25509b4a468457a0302d58136099db2fd9a671d", size = 3816957, upload-time = "2026-07-16T17:29:54.991Z" },
+]
+
+[[package]]
+name = "hnswlib"
+version = "0.8.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/cf/7a/1a9b1405f2eb59515f06c3074750b03e0e96edf7fee0f6dd6df81d9c21d7/hnswlib-0.8.0.tar.gz", hash = "sha256:cb6d037eedebb34a7134e7dc78966441dfd04c9cf5ee93911be911ced951c44c", size = 36206, upload-time = "2023-12-03T04:16:17.55Z" }
+
+[[package]]
+name = "hpack"
+version = "4.2.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/26/5b/fcabf6028144a8723726318b07a32c2f3314acdff6265743cf08a344b18e/hpack-4.2.0.tar.gz", hash = "sha256:0895cfa3b5531fc65fe439c05eb65144f123bf7a394fcaa56aa423548d8e45c0", size = 51300, upload-time = "2026-06-23T18:34:46.667Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/71/b4/4a9fcfb2aef6ba44d9073ecd301443aa00b3dac95de5619f2a7de7ec8a91/hpack-4.2.0-py3-none-any.whl", hash = "sha256:858ac0b02280fa582b5080d68db0899c62a80375e0e5413a74970c5e518b6986", size = 34246, upload-time = "2026-06-23T18:34:45.472Z" },
+]
+
+[[package]]
+name = "httpcore"
+version = "1.0.9"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "certifi" },
+ { name = "h11" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" },
+]
+
+[[package]]
+name = "httpx"
+version = "0.28.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+ { name = "certifi" },
+ { name = "httpcore" },
+ { name = "idna" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" },
+]
+
+[[package]]
+name = "huggingface-hub"
+version = "1.24.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "click" },
+ { name = "filelock" },
+ { name = "fsspec" },
+ { name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64'" },
+ { name = "httpx" },
+ { name = "packaging" },
+ { name = "pyyaml" },
+ { name = "tqdm" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/df/9b/d3bb4e7d792835daf34dd7091bbc7d7b4e0437d9388f1ea7239cce49f478/huggingface_hub-1.24.0.tar.gz", hash = "sha256:18431ff4daae0749aa9ba102fc952e314c98e1d30ebdec5319d85ca0a83e1ae5", size = 921848, upload-time = "2026-07-17T09:54:01.022Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5f/c3/aeaaf3911d2529614be18d1c8b5496afc185560e76568063d517283318af/huggingface_hub-1.24.0-py3-none-any.whl", hash = "sha256:6ed4120a84a6beec900640aa7e346bd766a6b7341e41526fef5dc8bd81fb7d59", size = 771904, upload-time = "2026-07-17T09:53:59.106Z" },
+]
+
+[[package]]
+name = "hyperframe"
+version = "6.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/02/e7/94f8232d4a74cc99514c13a9f995811485a6903d48e5d952771ef6322e30/hyperframe-6.1.0.tar.gz", hash = "sha256:f630908a00854a7adeabd6382b43923a4c4cd4b821fcb527e6ab9e15382a3b08", size = 26566, upload-time = "2025-01-22T21:41:49.302Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/48/30/47d0bf6072f7252e6521f3447ccfa40b421b6824517f82854703d0f5a98b/hyperframe-6.1.0-py3-none-any.whl", hash = "sha256:b03380493a519fce58ea5af42e4a42317bf9bd425596f7a0835ffce80f1a42e5", size = 13007, upload-time = "2025-01-22T21:41:47.295Z" },
+]
+
+[[package]]
+name = "idna"
+version = "3.18"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cd/63/9496c57188a2ee585e0f1db071d75089a11e98aa86eb99d9d7618fc1edce/idna-3.18.tar.gz", hash = "sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848", size = 196711, upload-time = "2026-06-02T14:34:07.794Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1e/5e/d4e9f1a599fb8e573b7b87160658329fbf28d19eac2718f51fc3def3aa5a/idna-3.18-py3-none-any.whl", hash = "sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2", size = 65455, upload-time = "2026-06-02T14:34:06.319Z" },
+]
+
+[[package]]
+name = "igraph"
+version = "1.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "texttable" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/23/be/56bef1919005b4caf1f71522b300d359f7faeb7ae93a3b0baa9b4f146a87/igraph-1.0.0.tar.gz", hash = "sha256:2414d0be2e4d77ee5357807d100974b40f6082bb1bb71988ec46cfb6728651ee", size = 5077105, upload-time = "2025-10-23T12:22:50.127Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a5/03/3278ad0ceb3ea0e84d8ae3a85bdded4d0e57853aeb802a200feb43847b93/igraph-1.0.0-cp39-abi3-macosx_10_15_x86_64.whl", hash = "sha256:c2cbc415e02523e5a241eecee82319080bf928a70b1ba299f3b3e25bf029b6d4", size = 2257415, upload-time = "2025-10-23T12:22:27.246Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/bc/6281ec7f9baaf71ee57c3b1748da2d3148d15d253e1a03006f204aa68ca5/igraph-1.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a27753cd80680a8f676c2d5a467aaa4a95e510b30748398ec4e4aeb982130e8", size = 2048555, upload-time = "2025-10-23T12:22:29.49Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/38/3cd6428a4ed4c09a56df05998438e7774fd1d799ee4fb8fc481674f5f7fc/igraph-1.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:a55dc3a2a4e3fc3eba42479910c1511bfc3ecb33cdf5f0406891fd85f14b5aee", size = 5314141, upload-time = "2025-10-23T12:22:31.023Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/da/dd2867c25adbb41563720f14b5fc895c98bf88be682a3faff4f7b3118d2a/igraph-1.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:2d04c2c76f686fb1f554ee35dfd3085f5e73b7965ba6b4cf06d53e66b1955522", size = 5683134, upload-time = "2025-10-23T12:22:32.423Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/40/243c118d34ab80382d7009c4dcb99b887384c3d2ce84d29eeac19e2a007a/igraph-1.0.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:f2b52dc1757fff0fed29a9f7a276d971a11db4211569ed78b9eab36288dfcc9d", size = 6211583, upload-time = "2025-10-23T12:22:34.238Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/b7/88f433819c54b496cb0315fce28e658970cb20ff5dbd52a5a605ce2888de/igraph-1.0.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:05c79a2a8fca695b2f217a6fa7f2549f896f757d4db41be32a055400cb19cc30", size = 6594509, upload-time = "2025-10-23T12:22:35.831Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/5d/8f7f6f619d374e959aa3664ebc4b24c10abc90c2e8efbed97f2623fadaf5/igraph-1.0.0-cp39-abi3-win32.whl", hash = "sha256:c2bce3cd472fec3dd9c4d8a3ea5b6b9be65fb30edf760beb4850760dd4f2d479", size = 2725406, upload-time = "2025-10-23T12:22:37.588Z" },
+ { url = "https://files.pythonhosted.org/packages/af/77/a85b3745cf40a0572bae2de8cd9c2a2a8af78e5cf3e880fc0a249114e609/igraph-1.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:faeff8ede0cf15eb4ded44b0fcea6e1886740146e60504c24ad2da14e0939563", size = 3221663, upload-time = "2025-10-23T12:22:39.404Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/7e/5df541c37bdf6493035e89c22bd53f30d99b291bcda6c78e9a8afeecec2b/igraph-1.0.0-cp39-abi3-win_arm64.whl", hash = "sha256:b607cafc24b10a615e713ee96e58208ef27e0764af80140c7cc45d4724a3f2df", size = 2785701, upload-time = "2025-10-23T12:22:41.03Z" },
+]
+
+[[package]]
+name = "imagesize"
+version = "2.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/6c/e6/7bf14eeb8f8b7251141944835abd42eb20a658d89084b7e1f3e5fe394090/imagesize-2.0.0.tar.gz", hash = "sha256:8e8358c4a05c304f1fccf7ff96f036e7243a189e9e42e90851993c558cfe9ee3", size = 1773045, upload-time = "2026-03-03T14:18:29.941Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5f/53/fb7122b71361a0d121b669dcf3d31244ef75badbbb724af388948de543e2/imagesize-2.0.0-py2.py3-none-any.whl", hash = "sha256:5667c5bbb57ab3f1fa4bc366f4fbc971db3d5ed011fd2715fd8001f782718d96", size = 9441, upload-time = "2026-03-03T14:18:27.892Z" },
+]
+
+[[package]]
+name = "importlib-metadata"
+version = "9.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "zipp" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a9/01/15bb152d77b21318514a96f43af312635eb2500c96b55398d020c93d86ea/importlib_metadata-9.0.0.tar.gz", hash = "sha256:a4f57ab599e6a2e3016d7595cfd72eb4661a5106e787a95bcc90c7105b831efc", size = 56405, upload-time = "2026-03-20T06:42:56.999Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/38/3d/2d244233ac4f76e38533cfcb2991c9eb4c7bf688ae0a036d30725b8faafe/importlib_metadata-9.0.0-py3-none-any.whl", hash = "sha256:2d21d1cc5a017bd0559e36150c21c830ab1dc304dedd1b7ea85d20f45ef3edd7", size = 27789, upload-time = "2026-03-20T06:42:55.665Z" },
+]
+
+[[package]]
+name = "iniconfig"
+version = "2.3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" },
+]
+
+[[package]]
+name = "ipykernel"
+version = "7.3.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "appnope", marker = "sys_platform == 'darwin'" },
+ { name = "comm" },
+ { name = "debugpy" },
+ { name = "ipython" },
+ { name = "jupyter-client" },
+ { name = "jupyter-core" },
+ { name = "matplotlib-inline" },
+ { name = "nest-asyncio2" },
+ { name = "packaging" },
+ { name = "psutil" },
+ { name = "pyzmq" },
+ { name = "tornado" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3d/c4/e4a38f579de4225a561305666f7541cdabb30075def2aa1ac17bd73c1fb5/ipykernel-7.3.0.tar.gz", hash = "sha256:9acaaaf97d16355166e4085afe9d225bfbdf2b7ef520f9df3be8f2b248275e09", size = 184899, upload-time = "2026-06-10T08:41:25.481Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3d/02/77b271f5dc58bfbc0b577c877b2365d1ffea2afe66a80c13f2312820348c/ipykernel-7.3.0-py3-none-any.whl", hash = "sha256:897eb64da762549ef610698fca5e9675195ec6ac8ec7f19d81ce1ca20c876057", size = 120583, upload-time = "2026-06-10T08:41:23.648Z" },
+]
+
+[[package]]
+name = "ipython"
+version = "9.15.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+ { name = "decorator" },
+ { name = "ipython-pygments-lexers" },
+ { name = "jedi" },
+ { name = "matplotlib-inline" },
+ { name = "pexpect", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+ { name = "prompt-toolkit" },
+ { name = "psutil", marker = "sys_platform != 'cygwin' and sys_platform != 'emscripten'" },
+ { name = "pygments" },
+ { name = "stack-data" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/53/59/165d3b4d75cc34add3122c4417ecb229085140ac573103c223cd01dde96f/ipython-9.15.0.tar.gz", hash = "sha256:da2819ce2aa83135257df830660b1176d986c3d2876db24df01974fa955b2756", size = 4442580, upload-time = "2026-06-26T11:03:35.913Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/40/3a/948263ca3b9d65bb2b1b0c521b3a49fad5d59ada58724bd87d2bd5ff3f36/ipython-9.15.0-py3-none-any.whl", hash = "sha256:515ad9c3cdf0c932a5a9f6245419e8aba706b7bd03c3e1d3a1c83d9351d6aa6e", size = 630895, upload-time = "2026-06-26T11:03:33.809Z" },
+]
+
+[[package]]
+name = "ipython-autotime"
+version = "0.3.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "ipython" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c2/9d/f8b72a7cf940c704299ef712c75e862abd7fac3bbc4675a87f9851f20241/ipython-autotime-0.3.2.tar.gz", hash = "sha256:fc03ab920266072f0f6d7f1cc04247afe1c2f22f7bd052542af700939769d5f4", size = 6347, upload-time = "2023-11-01T11:43:24.725Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d1/6b/c668b9d6a3552eeb4d82c0e6f1e79debc9cfaecc2d59ddb5b2f38c24cdfc/ipython_autotime-0.3.2-py2.py3-none-any.whl", hash = "sha256:c51e1175b9d0d2c5da51ac17963f3a836036cf389b8ef20f8a7483573d2c3dce", size = 6971, upload-time = "2023-11-01T11:43:23.549Z" },
+]
+
+[[package]]
+name = "ipython-pygments-lexers"
+version = "1.1.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pygments" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ef/4c/5dd1d8af08107f88c7f741ead7a40854b8ac24ddf9ae850afbcf698aa552/ipython_pygments_lexers-1.1.1.tar.gz", hash = "sha256:09c0138009e56b6854f9535736f4171d855c8c08a563a0dcd8022f78355c7e81", size = 8393, upload-time = "2025-01-17T11:24:34.505Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d9/33/1f075bf72b0b747cb3288d011319aaf64083cf2efef8354174e3ed4540e2/ipython_pygments_lexers-1.1.1-py3-none-any.whl", hash = "sha256:a9462224a505ade19a605f71f8fa63c2048833ce50abc86768a0d81d876dc81c", size = 8074, upload-time = "2025-01-17T11:24:33.271Z" },
+]
+
+[[package]]
+name = "ipywidgets"
+version = "8.1.8"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "comm" },
+ { name = "ipython" },
+ { name = "jupyterlab-widgets" },
+ { name = "traitlets" },
+ { name = "widgetsnbextension" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/4c/ae/c5ce1edc1afe042eadb445e95b0671b03cee61895264357956e61c0d2ac0/ipywidgets-8.1.8.tar.gz", hash = "sha256:61f969306b95f85fba6b6986b7fe45d73124d1d9e3023a8068710d47a22ea668", size = 116739, upload-time = "2025-11-01T21:18:12.393Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/56/6d/0d9848617b9f753b87f214f1c682592f7ca42de085f564352f10f0843026/ipywidgets-8.1.8-py3-none-any.whl", hash = "sha256:ecaca67aed704a338f88f67b1181b58f821ab5dc89c1f0f5ef99db43c1c2921e", size = 139808, upload-time = "2025-11-01T21:18:10.956Z" },
+]
+
+[[package]]
+name = "jedi"
+version = "0.20.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "parso" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/46/b7/a3635f6a2d7cf5b5dd98064fc1d5fbbafcb25477bcea204a3a92145d158b/jedi-0.20.0.tar.gz", hash = "sha256:c3f4ccbd276696f4b19c54618d4fb18f9fc24b0aef02acf704b23f487daa1011", size = 3119416, upload-time = "2026-05-01T23:38:47.814Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9a/93/242e2eab5fe682ffcb8b0084bde703a41d51e17ee0f3a31ff0d9d813620a/jedi-0.20.0-py2.py3-none-any.whl", hash = "sha256:7bdd9c2634f56713299976f4cbd59cb3fa92165cc5e05ea811fb253480728b67", size = 4884812, upload-time = "2026-05-01T23:38:43.919Z" },
+]
+
+[[package]]
+name = "jinja2"
+version = "3.1.6"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "markupsafe" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d", size = 245115, upload-time = "2025-03-05T20:05:02.478Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" },
+]
+
+[[package]]
+name = "joblib"
+version = "1.5.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/41/f2/d34e8b3a08a9cc79a50b2208a93dce981fe615b64d5a4d4abee421d898df/joblib-1.5.3.tar.gz", hash = "sha256:8561a3269e6801106863fd0d6d84bb737be9e7631e33aaed3fb9ce5953688da3", size = 331603, upload-time = "2025-12-15T08:41:46.427Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl", hash = "sha256:5fc3c5039fc5ca8c0276333a188bbd59d6b7ab37fe6632daa76bc7f9ec18e713", size = 309071, upload-time = "2025-12-15T08:41:44.973Z" },
+]
+
+[[package]]
+name = "jsonschema"
+version = "4.26.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "attrs" },
+ { name = "jsonschema-specifications" },
+ { name = "referencing" },
+ { name = "rpds-py" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b3/fc/e067678238fa451312d4c62bf6e6cf5ec56375422aee02f9cb5f909b3047/jsonschema-4.26.0.tar.gz", hash = "sha256:0c26707e2efad8aa1bfc5b7ce170f3fccc2e4918ff85989ba9ffa9facb2be326", size = 366583, upload-time = "2026-01-07T13:41:07.246Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/69/90/f63fb5873511e014207a475e2bb4e8b2e570d655b00ac19a9a0ca0a385ee/jsonschema-4.26.0-py3-none-any.whl", hash = "sha256:d489f15263b8d200f8387e64b4c3a75f06629559fb73deb8fdfb525f2dab50ce", size = 90630, upload-time = "2026-01-07T13:41:05.306Z" },
+]
+
+[[package]]
+name = "jsonschema-specifications"
+version = "2025.9.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "referencing" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/19/74/a633ee74eb36c44aa6d1095e7cc5569bebf04342ee146178e2d36600708b/jsonschema_specifications-2025.9.1.tar.gz", hash = "sha256:b540987f239e745613c7a9176f3edb72b832a4ac465cf02712288397832b5e8d", size = 32855, upload-time = "2025-09-08T01:34:59.186Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe", size = 18437, upload-time = "2025-09-08T01:34:57.871Z" },
+]
+
+[[package]]
+name = "jupyter-cache"
+version = "1.0.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "attrs" },
+ { name = "click" },
+ { name = "importlib-metadata" },
+ { name = "nbclient" },
+ { name = "nbformat" },
+ { name = "pyyaml" },
+ { name = "sqlalchemy" },
+ { name = "tabulate" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/bb/f7/3627358075f183956e8c4974603232b03afd4ddc7baf72c2bc9fff522291/jupyter_cache-1.0.1.tar.gz", hash = "sha256:16e808eb19e3fb67a223db906e131ea6e01f03aa27f49a7214ce6a5fec186fb9", size = 32048, upload-time = "2024-11-15T16:03:55.322Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/64/6b/67b87da9d36bff9df7d0efbd1a325fa372a43be7158effaf43ed7b22341d/jupyter_cache-1.0.1-py3-none-any.whl", hash = "sha256:9c3cafd825ba7da8b5830485343091143dff903e4d8c69db9349b728b140abf6", size = 33907, upload-time = "2024-11-15T16:03:54.021Z" },
+]
+
+[[package]]
+name = "jupyter-client"
+version = "8.9.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jupyter-core" },
+ { name = "python-dateutil" },
+ { name = "pyzmq" },
+ { name = "tornado" },
+ { name = "traitlets" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/7d/dc/5512503b088997c2250b8bf18258fba9d9ce5ead641183700960d3c9d342/jupyter_client-8.9.1.tar.gz", hash = "sha256:a58f730dd9e728ba16ba1d62ebccf7ffe1ebbdbce4e95cfae941b7321ae1f4fa", size = 359256, upload-time = "2026-06-09T13:15:01.033Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3f/6f/56d39bf385c5c27988aebaf0c18a2a17e960575740100973511018bd904e/jupyter_client-8.9.1-py3-none-any.whl", hash = "sha256:0b7a295bc46e8751e9adae84781f726c851c1d911bd793edc4a3bde942e3da81", size = 109828, upload-time = "2026-06-09T13:14:58.835Z" },
+]
+
+[[package]]
+name = "jupyter-core"
+version = "5.9.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "platformdirs" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/02/49/9d1284d0dc65e2c757b74c6687b6d319b02f822ad039e5c512df9194d9dd/jupyter_core-5.9.1.tar.gz", hash = "sha256:4d09aaff303b9566c3ce657f580bd089ff5c91f5f89cf7d8846c3cdf465b5508", size = 89814, upload-time = "2025-10-16T19:19:18.444Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e7/e7/80988e32bf6f73919a113473a604f5a8f09094de312b9d52b79c2df7612b/jupyter_core-5.9.1-py3-none-any.whl", hash = "sha256:ebf87fdc6073d142e114c72c9e29a9d7ca03fad818c5d300ce2adc1fb0743407", size = 29032, upload-time = "2025-10-16T19:19:16.783Z" },
+]
+
+[[package]]
+name = "jupyterlab-widgets"
+version = "3.0.16"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/26/2d/ef58fed122b268c69c0aa099da20bc67657cdfb2e222688d5731bd5b971d/jupyterlab_widgets-3.0.16.tar.gz", hash = "sha256:423da05071d55cf27a9e602216d35a3a65a3e41cdf9c5d3b643b814ce38c19e0", size = 897423, upload-time = "2025-11-01T21:11:29.724Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ab/b5/36c712098e6191d1b4e349304ef73a8d06aed77e56ceaac8c0a306c7bda1/jupyterlab_widgets-3.0.16-py3-none-any.whl", hash = "sha256:45fa36d9c6422cf2559198e4db481aa243c7a32d9926b500781c830c80f7ecf8", size = 914926, upload-time = "2025-11-01T21:11:28.008Z" },
+]
+
+[[package]]
+name = "kiwisolver"
+version = "1.5.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d0/67/9c61eccb13f0bdca9307614e782fec49ffdde0f7a2314935d489fa93cd9c/kiwisolver-1.5.0.tar.gz", hash = "sha256:d4193f3d9dc3f6f79aaed0e5637f45d98850ebf01f7ca20e69457f3e8946b66a", size = 103482, upload-time = "2026-03-09T13:15:53.382Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4d/b2/818b74ebea34dabe6d0c51cb1c572e046730e64844da6ed646d5298c40ce/kiwisolver-1.5.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:4e9750bc21b886308024f8a54ccb9a2cc38ac9fa813bf4348434e3d54f337ff9", size = 123158, upload-time = "2026-03-09T13:13:23.127Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/d9/405320f8077e8e1c5c4bd6adc45e1e6edf6d727b6da7f2e2533cf58bff71/kiwisolver-1.5.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:72ec46b7eba5b395e0a7b63025490d3214c11013f4aacb4f5e8d6c3041829588", size = 66388, upload-time = "2026-03-09T13:13:24.765Z" },
+ { url = "https://files.pythonhosted.org/packages/99/9f/795fedf35634f746151ca8839d05681ceb6287fbed6cc1c9bf235f7887c2/kiwisolver-1.5.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ed3a984b31da7481b103f68776f7128a89ef26ed40f4dc41a2223cda7fb24819", size = 64068, upload-time = "2026-03-09T13:13:25.878Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/13/680c54afe3e65767bed7ec1a15571e1a2f1257128733851ade24abcefbcc/kiwisolver-1.5.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bb5136fb5352d3f422df33f0c879a1b0c204004324150cc3b5e3c4f310c9049f", size = 1477934, upload-time = "2026-03-09T13:13:27.166Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/2f/cebfcdb60fd6a9b0f6b47a9337198bcbad6fbe15e68189b7011fd914911f/kiwisolver-1.5.0-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b2af221f268f5af85e776a73d62b0845fc8baf8ef0abfae79d29c77d0e776aaf", size = 1278537, upload-time = "2026-03-09T13:13:28.707Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/0d/9b782923aada3fafb1d6b84e13121954515c669b18af0c26e7d21f579855/kiwisolver-1.5.0-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b0f172dc8ffaccb8522d7c5d899de00133f2f1ca7b0a49b7da98e901de87bf2d", size = 1296685, upload-time = "2026-03-09T13:13:30.528Z" },
+ { url = "https://files.pythonhosted.org/packages/27/70/83241b6634b04fe44e892688d5208332bde130f38e610c0418f9ede47ded/kiwisolver-1.5.0-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:6ab8ba9152203feec73758dad83af9a0bbe05001eb4639e547207c40cfb52083", size = 1346024, upload-time = "2026-03-09T13:13:32.818Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/db/30ed226fb271ae1a6431fc0fe0edffb2efe23cadb01e798caeb9f2ceae8f/kiwisolver-1.5.0-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:cdee07c4d7f6d72008d3f73b9bf027f4e11550224c7c50d8df1ae4a37c1402a6", size = 987241, upload-time = "2026-03-09T13:13:34.435Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/bd/c314595208e4c9587652d50959ead9e461995389664e490f4dce7ff0f782/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:7c60d3c9b06fb23bd9c6139281ccbdc384297579ae037f08ae90c69f6845c0b1", size = 2227742, upload-time = "2026-03-09T13:13:36.4Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/43/0499cec932d935229b5543d073c2b87c9c22846aab48881e9d8d6e742a2d/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:e315e5ec90d88e140f57696ff85b484ff68bb311e36f2c414aa4286293e6dee0", size = 2323966, upload-time = "2026-03-09T13:13:38.204Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/6f/79b0d760907965acfd9d61826a3d41f8f093c538f55cd2633d3f0db269f6/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:1465387ac63576c3e125e5337a6892b9e99e0627d52317f3ca79e6930d889d15", size = 1977417, upload-time = "2026-03-09T13:13:39.966Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/31/01d0537c41cb75a551a438c3c7a80d0c60d60b81f694dac83dd436aec0d0/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:530a3fd64c87cffa844d4b6b9768774763d9caa299e9b75d8eca6a4423b31314", size = 2491238, upload-time = "2026-03-09T13:13:41.698Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/34/8aefdd0be9cfd00a44509251ba864f5caf2991e36772e61c408007e7f417/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:1d9daea4ea6b9be74fe2f01f7fbade8d6ffab263e781274cffca0dba9be9eec9", size = 2294947, upload-time = "2026-03-09T13:13:43.343Z" },
+ { url = "https://files.pythonhosted.org/packages/ad/cf/0348374369ca588f8fe9c338fae49fa4e16eeb10ffb3d012f23a54578a9e/kiwisolver-1.5.0-cp312-cp312-win_amd64.whl", hash = "sha256:f18c2d9782259a6dc132fdc7a63c168cbc74b35284b6d75c673958982a378384", size = 73569, upload-time = "2026-03-09T13:13:45.792Z" },
+ { url = "https://files.pythonhosted.org/packages/28/26/192b26196e2316e2bd29deef67e37cdf9870d9af8e085e521afff0fed526/kiwisolver-1.5.0-cp312-cp312-win_arm64.whl", hash = "sha256:f7c7553b13f69c1b29a5bde08ddc6d9d0c8bfb84f9ed01c30db25944aeb852a7", size = 64997, upload-time = "2026-03-09T13:13:46.878Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/69/024d6711d5ba575aa65d5538042e99964104e97fa153a9f10bc369182bc2/kiwisolver-1.5.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:fd40bb9cd0891c4c3cb1ddf83f8bbfa15731a248fdc8162669405451e2724b09", size = 123166, upload-time = "2026-03-09T13:13:48.032Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/48/adbb40df306f587054a348831220812b9b1d787aff714cfbc8556e38fccd/kiwisolver-1.5.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c0e1403fd7c26d77c1f03e096dc58a5c726503fa0db0456678b8668f76f521e3", size = 66395, upload-time = "2026-03-09T13:13:49.365Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/3a/d0a972b34e1c63e2409413104216cd1caa02c5a37cb668d1687d466c1c45/kiwisolver-1.5.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:dda366d548e89a90d88a86c692377d18d8bd64b39c1fb2b92cb31370e2896bbd", size = 64065, upload-time = "2026-03-09T13:13:50.562Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/0a/7b98e1e119878a27ba8618ca1e18b14f992ff1eda40f47bccccf4de44121/kiwisolver-1.5.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:332b4f0145c30b5f5ad9374881133e5aa64320428a57c2c2b61e9d891a51c2f3", size = 1477903, upload-time = "2026-03-09T13:13:52.084Z" },
+ { url = "https://files.pythonhosted.org/packages/18/d8/55638d89ffd27799d5cc3d8aa28e12f4ce7a64d67b285114dbedc8ea4136/kiwisolver-1.5.0-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0c50b89ffd3e1a911c69a1dd3de7173c0cd10b130f56222e57898683841e4f96", size = 1278751, upload-time = "2026-03-09T13:13:54.673Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/97/b4c8d0d18421ecceba20ad8701358453b88e32414e6f6950b5a4bad54e65/kiwisolver-1.5.0-cp313-cp313-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4db576bb8c3ef9365f8b40fe0f671644de6736ae2c27a2c62d7d8a1b4329f099", size = 1296793, upload-time = "2026-03-09T13:13:56.287Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/10/f862f94b6389d8957448ec9df59450b81bec4abb318805375c401a1e6892/kiwisolver-1.5.0-cp313-cp313-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0b85aad90cea8ac6797a53b5d5f2e967334fa4d1149f031c4537569972596cb8", size = 1346041, upload-time = "2026-03-09T13:13:58.269Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/6a/f1650af35821eaf09de398ec0bc2aefc8f211f0cda50204c9f1673741ba9/kiwisolver-1.5.0-cp313-cp313-manylinux_2_39_riscv64.whl", hash = "sha256:d36ca54cb4c6c4686f7cbb7b817f66f5911c12ddb519450bbe86707155028f87", size = 987292, upload-time = "2026-03-09T13:13:59.871Z" },
+ { url = "https://files.pythonhosted.org/packages/de/19/d7fb82984b9238115fe629c915007be608ebd23dc8629703d917dbfaffd4/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:38f4a703656f493b0ad185211ccfca7f0386120f022066b018eb5296d8613e23", size = 2227865, upload-time = "2026-03-09T13:14:01.401Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/b9/46b7f386589fd222dac9e9de9c956ce5bcefe2ee73b4e79891381dda8654/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:3ac2360e93cb41be81121755c6462cff3beaa9967188c866e5fce5cf13170859", size = 2324369, upload-time = "2026-03-09T13:14:02.972Z" },
+ { url = "https://files.pythonhosted.org/packages/92/8b/95e237cf3d9c642960153c769ddcbe278f182c8affb20cecc1cc983e7cc5/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:c95cab08d1965db3d84a121f1c7ce7479bdd4072c9b3dafd8fecce48a2e6b902", size = 1977989, upload-time = "2026-03-09T13:14:04.503Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/95/980c9df53501892784997820136c01f62bc1865e31b82b9560f980c0e649/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:fc20894c3d21194d8041a28b65622d5b86db786da6e3cfe73f0c762951a61167", size = 2491645, upload-time = "2026-03-09T13:14:06.106Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/32/900647fd0840abebe1561792c6b31e6a7c0e278fc3973d30572a965ca14c/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7a32f72973f0f950c1920475d5c5ea3d971b81b6f0ec53b8d0a956cc965f22e0", size = 2295237, upload-time = "2026-03-09T13:14:08.891Z" },
+ { url = "https://files.pythonhosted.org/packages/be/8a/be60e3bbcf513cc5a50f4a3e88e1dcecebb79c1ad607a7222877becaa101/kiwisolver-1.5.0-cp313-cp313-win_amd64.whl", hash = "sha256:0bf3acf1419fa93064a4c2189ac0b58e3be7872bf6ee6177b0d4c63dc4cea276", size = 73573, upload-time = "2026-03-09T13:14:12.327Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/d2/64be2e429eb4fca7f7e1c52a91b12663aeaf25de3895e5cca0f47ef2a8d0/kiwisolver-1.5.0-cp313-cp313-win_arm64.whl", hash = "sha256:fa8eb9ecdb7efb0b226acec134e0d709e87a909fa4971a54c0c4f6e88635484c", size = 64998, upload-time = "2026-03-09T13:14:13.469Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/69/ce68dd0c85755ae2de490bf015b62f2cea5f6b14ff00a463f9d0774449ff/kiwisolver-1.5.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:db485b3847d182b908b483b2ed133c66d88d49cacf98fd278fadafe11b4478d1", size = 125700, upload-time = "2026-03-09T13:14:14.636Z" },
+ { url = "https://files.pythonhosted.org/packages/74/aa/937aac021cf9d4349990d47eb319309a51355ed1dbdc9c077cdc9224cb11/kiwisolver-1.5.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:be12f931839a3bdfe28b584db0e640a65a8bcbc24560ae3fdb025a449b3d754e", size = 67537, upload-time = "2026-03-09T13:14:15.808Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/20/3a87fbece2c40ad0f6f0aefa93542559159c5f99831d596050e8afae7a9f/kiwisolver-1.5.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:16b85d37c2cbb3253226d26e64663f755d88a03439a9c47df6246b35defbdfb7", size = 65514, upload-time = "2026-03-09T13:14:18.035Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/7f/f943879cda9007c45e1f7dba216d705c3a18d6b35830e488b6c6a4e7cdf0/kiwisolver-1.5.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4432b835675f0ea7414aab3d37d119f7226d24869b7a829caeab49ebda407b0c", size = 1584848, upload-time = "2026-03-09T13:14:19.745Z" },
+ { url = "https://files.pythonhosted.org/packages/37/f8/4d4f85cc1870c127c88d950913370dd76138482161cd07eabbc450deff01/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b0feb50971481a2cc44d94e88bdb02cdd497618252ae226b8eb1201b957e368", size = 1391542, upload-time = "2026-03-09T13:14:21.54Z" },
+ { url = "https://files.pythonhosted.org/packages/04/0b/65dd2916c84d252b244bd405303220f729e7c17c9d7d33dca6feeff9ffc4/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:56fa888f10d0f367155e76ce849fa1166fc9730d13bd2d65a2aa13b6f5424489", size = 1404447, upload-time = "2026-03-09T13:14:23.205Z" },
+ { url = "https://files.pythonhosted.org/packages/39/5c/2606a373247babce9b1d056c03a04b65f3cf5290a8eac5d7bdead0a17e21/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:940dda65d5e764406b9fb92761cbf462e4e63f712ab60ed98f70552e496f3bf1", size = 1455918, upload-time = "2026-03-09T13:14:24.74Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/d1/c6078b5756670658e9192a2ef11e939c92918833d2745f85cd14a6004bdf/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_39_riscv64.whl", hash = "sha256:89fc958c702ee9a745e4700378f5d23fddbc46ff89e8fdbf5395c24d5c1452a3", size = 1072856, upload-time = "2026-03-09T13:14:26.597Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/c8/7def6ddf16eb2b3741d8b172bdaa9af882b03c78e9b0772975408801fa63/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9027d773c4ff81487181a925945743413f6069634d0b122d0b37684ccf4f1e18", size = 2333580, upload-time = "2026-03-09T13:14:28.237Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/87/2ac1fce0eb1e616fcd3c35caa23e665e9b1948bb984f4764790924594128/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:5b233ea3e165e43e35dba1d2b8ecc21cf070b45b65ae17dd2747d2713d942021", size = 2423018, upload-time = "2026-03-09T13:14:30.018Z" },
+ { url = "https://files.pythonhosted.org/packages/67/13/c6700ccc6cc218716bfcda4935e4b2997039869b4ad8a94f364c5a3b8e63/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:ce9bf03dad3b46408c08649c6fbd6ca28a9fce0eb32fdfffa6775a13103b5310", size = 2062804, upload-time = "2026-03-09T13:14:32.888Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/bd/877056304626943ff0f1f44c08f584300c199b887cb3176cd7e34f1515f1/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:fc4d3f1fb9ca0ae9f97b095963bc6326f1dbfd3779d6679a1e016b9baaa153d3", size = 2597482, upload-time = "2026-03-09T13:14:34.971Z" },
+ { url = "https://files.pythonhosted.org/packages/75/19/c60626c47bf0f8ac5dcf72c6c98e266d714f2fbbfd50cf6dab5ede3aaa50/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:f443b4825c50a51ee68585522ab4a1d1257fac65896f282b4c6763337ac9f5d2", size = 2394328, upload-time = "2026-03-09T13:14:36.816Z" },
+ { url = "https://files.pythonhosted.org/packages/47/84/6a6d5e5bb8273756c27b7d810d47f7ef2f1f9b9fd23c9ee9a3f8c75c9cef/kiwisolver-1.5.0-cp313-cp313t-win_arm64.whl", hash = "sha256:893ff3a711d1b515ba9da14ee090519bad4610ed1962fbe298a434e8c5f8db53", size = 68410, upload-time = "2026-03-09T13:14:38.695Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/d7/060f45052f2a01ad5762c8fdecd6d7a752b43400dc29ff75cd47225a40fd/kiwisolver-1.5.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:8df31fe574b8b3993cc61764f40941111b25c2d9fea13d3ce24a49907cd2d615", size = 123231, upload-time = "2026-03-09T13:14:41.323Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/a7/78da680eadd06ff35edef6ef68a1ad273bad3e2a0936c9a885103230aece/kiwisolver-1.5.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:1d49a49ac4cbfb7c1375301cd1ec90169dfeae55ff84710d782260ce77a75a02", size = 66489, upload-time = "2026-03-09T13:14:42.534Z" },
+ { url = "https://files.pythonhosted.org/packages/49/b2/97980f3ad4fae37dd7fe31626e2bf75fbf8bdf5d303950ec1fab39a12da8/kiwisolver-1.5.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0cbe94b69b819209a62cb27bdfa5dc2a8977d8de2f89dfd97ba4f53ed3af754e", size = 64063, upload-time = "2026-03-09T13:14:44.759Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/f9/b06c934a6aa8bc91f566bd2a214fd04c30506c2d9e2b6b171953216a65b6/kiwisolver-1.5.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:80aa065ffd378ff784822a6d7c3212f2d5f5e9c3589614b5c228b311fd3063ac", size = 1475913, upload-time = "2026-03-09T13:14:46.247Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/f0/f768ae564a710135630672981231320bc403cf9152b5596ec5289de0f106/kiwisolver-1.5.0-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4e7f886f47ab881692f278ae901039a234e4025a68e6dfab514263a0b1c4ae05", size = 1282782, upload-time = "2026-03-09T13:14:48.458Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/9f/1de7aad00697325f05238a5f2eafbd487fb637cc27a558b5367a5f37fb7f/kiwisolver-1.5.0-cp314-cp314-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5060731cc3ed12ca3a8b57acd4aeca5bbc2f49216dd0bec1650a1acd89486bcd", size = 1300815, upload-time = "2026-03-09T13:14:50.721Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/c2/297f25141d2e468e0ce7f7a7b92e0cf8918143a0cbd3422c1ad627e85a06/kiwisolver-1.5.0-cp314-cp314-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:7a4aa69609f40fce3cbc3f87b2061f042eee32f94b8f11db707b66a26461591a", size = 1347925, upload-time = "2026-03-09T13:14:52.304Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/d3/f4c73a02eb41520c47610207b21afa8cdd18fdbf64ffd94674ae21c4812d/kiwisolver-1.5.0-cp314-cp314-manylinux_2_39_riscv64.whl", hash = "sha256:d168fda2dbff7b9b5f38e693182d792a938c31db4dac3a80a4888de603c99554", size = 991322, upload-time = "2026-03-09T13:14:54.637Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/46/d3f2efef7732fcda98d22bf4ad5d3d71d545167a852ca710a494f4c15343/kiwisolver-1.5.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:413b820229730d358efd838ecbab79902fe97094565fdc80ddb6b0a18c18a581", size = 2232857, upload-time = "2026-03-09T13:14:56.471Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/ec/2d9756bf2b6d26ae4349b8d3662fb3993f16d80c1f971c179ce862b9dbae/kiwisolver-1.5.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:5124d1ea754509b09e53738ec185584cc609aae4a3b510aaf4ed6aa047ef9303", size = 2329376, upload-time = "2026-03-09T13:14:58.072Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/9f/876a0a0f2260f1bde92e002b3019a5fabc35e0939c7d945e0fa66185eb20/kiwisolver-1.5.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e4415a8db000bf49a6dd1c478bf70062eaacff0f462b92b0ba68791a905861f9", size = 1982549, upload-time = "2026-03-09T13:14:59.668Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/4f/ba3624dfac23a64d54ac4179832860cb537c1b0af06024936e82ca4154a0/kiwisolver-1.5.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:d618fd27420381a4f6044faa71f46d8bfd911bd077c555f7138ed88729bfbe79", size = 2494680, upload-time = "2026-03-09T13:15:01.364Z" },
+ { url = "https://files.pythonhosted.org/packages/39/b7/97716b190ab98911b20d10bf92eca469121ec483b8ce0edd314f51bc85af/kiwisolver-1.5.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5092eb5b1172947f57d6ea7d89b2f29650414e4293c47707eb499ec07a0ac796", size = 2297905, upload-time = "2026-03-09T13:15:03.925Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/36/4e551e8aa55c9188bca9abb5096805edbf7431072b76e2298e34fd3a3008/kiwisolver-1.5.0-cp314-cp314-win_amd64.whl", hash = "sha256:d76e2d8c75051d58177e762164d2e9ab92886534e3a12e795f103524f221dd8e", size = 75086, upload-time = "2026-03-09T13:15:07.775Z" },
+ { url = "https://files.pythonhosted.org/packages/70/15/9b90f7df0e31a003c71649cf66ef61c3c1b862f48c81007fa2383c8bd8d7/kiwisolver-1.5.0-cp314-cp314-win_arm64.whl", hash = "sha256:fa6248cd194edff41d7ea9425ced8ca3a6f838bfb295f6f1d6e6bb694a8518df", size = 66577, upload-time = "2026-03-09T13:15:09.139Z" },
+ { url = "https://files.pythonhosted.org/packages/17/01/7dc8c5443ff42b38e72731643ed7cf1ed9bf01691ae5cdca98501999ed83/kiwisolver-1.5.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:d1ffeb80b5676463d7a7d56acbe8e37a20ce725570e09549fe738e02ca6b7e1e", size = 125794, upload-time = "2026-03-09T13:15:10.525Z" },
+ { url = "https://files.pythonhosted.org/packages/46/8a/b4ebe46ebaac6a303417fab10c2e165c557ddaff558f9699d302b256bc53/kiwisolver-1.5.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:bc4d8e252f532ab46a1de9349e2d27b91fce46736a9eedaa37beaca66f574ed4", size = 67646, upload-time = "2026-03-09T13:15:12.016Z" },
+ { url = "https://files.pythonhosted.org/packages/60/35/10a844afc5f19d6f567359bf4789e26661755a2f36200d5d1ed8ad0126e5/kiwisolver-1.5.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:6783e069732715ad0c3ce96dbf21dbc2235ab0593f2baf6338101f70371f4028", size = 65511, upload-time = "2026-03-09T13:15:13.311Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/8a/685b297052dd041dcebce8e8787b58923b6e78acc6115a0dc9189011c44b/kiwisolver-1.5.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e7c4c09a490dc4d4a7f8cbee56c606a320f9dc28cf92a7157a39d1ce7676a657", size = 1584858, upload-time = "2026-03-09T13:15:15.103Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/80/04865e3d4638ac5bddec28908916df4a3075b8c6cc101786a96803188b96/kiwisolver-1.5.0-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2a075bd7bd19c70cf67c8badfa36cf7c5d8de3c9ddb8420c51e10d9c50e94920", size = 1392539, upload-time = "2026-03-09T13:15:16.661Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/01/77a19cacc0893fa13fafa46d1bba06fb4dc2360b3292baf4b56d8e067b24/kiwisolver-1.5.0-cp314-cp314t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bdd3e53429ff02aa319ba59dfe4ceeec345bf46cf180ec2cf6fd5b942e7975e9", size = 1405310, upload-time = "2026-03-09T13:15:18.229Z" },
+ { url = "https://files.pythonhosted.org/packages/53/39/bcaf5d0cca50e604cfa9b4e3ae1d64b50ca1ae5b754122396084599ef903/kiwisolver-1.5.0-cp314-cp314t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3cdcb35dc9d807259c981a85531048ede628eabcffb3239adf3d17463518992d", size = 1456244, upload-time = "2026-03-09T13:15:20.444Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/7a/72c187abc6975f6978c3e39b7cf67aeb8b3c0a8f9790aa7fd412855e9e1f/kiwisolver-1.5.0-cp314-cp314t-manylinux_2_39_riscv64.whl", hash = "sha256:70d593af6a6ca332d1df73d519fddb5148edb15cd90d5f0155e3746a6d4fcc65", size = 1073154, upload-time = "2026-03-09T13:15:22.039Z" },
+ { url = "https://files.pythonhosted.org/packages/c7/ca/cf5b25783ebbd59143b4371ed0c8428a278abe68d6d0104b01865b1bbd0f/kiwisolver-1.5.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:377815a8616074cabbf3f53354e1d040c35815a134e01d7614b7692e4bf8acfa", size = 2334377, upload-time = "2026-03-09T13:15:23.741Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/e5/b1f492adc516796e88751282276745340e2a72dcd0d36cf7173e0daf3210/kiwisolver-1.5.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:0255a027391d52944eae1dbb5d4cc5903f57092f3674e8e544cdd2622826b3f0", size = 2425288, upload-time = "2026-03-09T13:15:25.789Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/e5/9b21fbe91a61b8f409d74a26498706e97a48008bfcd1864373d32a6ba31c/kiwisolver-1.5.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:012b1eb16e28718fa782b5e61dc6f2da1f0792ca73bd05d54de6cb9561665fc9", size = 2063158, upload-time = "2026-03-09T13:15:27.63Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/02/83f47986138310f95ea95531f851b2a62227c11cbc3e690ae1374fe49f0f/kiwisolver-1.5.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:0e3aafb33aed7479377e5e9a82e9d4bf87063741fc99fc7ae48b0f16e32bdd6f", size = 2597260, upload-time = "2026-03-09T13:15:29.421Z" },
+ { url = "https://files.pythonhosted.org/packages/07/18/43a5f24608d8c313dd189cf838c8e68d75b115567c6279de7796197cfb6a/kiwisolver-1.5.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:e7a116ae737f0000343218c4edf5bd45893bfeaff0993c0b215d7124c9f77646", size = 2394403, upload-time = "2026-03-09T13:15:31.517Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/b5/98222136d839b8afabcaa943b09bd05888c2d36355b7e448550211d1fca4/kiwisolver-1.5.0-cp314-cp314t-win_amd64.whl", hash = "sha256:1dd9b0b119a350976a6d781e7278ec7aca0b201e1a9e2d23d9804afecb6ca681", size = 79687, upload-time = "2026-03-09T13:15:33.204Z" },
+ { url = "https://files.pythonhosted.org/packages/99/a2/ca7dc962848040befed12732dff6acae7fb3c4f6fc4272b3f6c9a30b8713/kiwisolver-1.5.0-cp314-cp314t-win_arm64.whl", hash = "sha256:58f812017cd2985c21fbffb4864d59174d4903dd66fa23815e74bbc7a0e2dd57", size = 70032, upload-time = "2026-03-09T13:15:34.411Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/fa/2910df836372d8761bb6eff7d8bdcb1613b5c2e03f260efe7abe34d388a7/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-macosx_10_13_x86_64.whl", hash = "sha256:5ae8e62c147495b01a0f4765c878e9bfdf843412446a247e28df59936e99e797", size = 130262, upload-time = "2026-03-09T13:15:35.629Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/41/c5f71f9f00aabcc71fee8b7475e3f64747282580c2fe748961ba29b18385/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:f6764a4ccab3078db14a632420930f6186058750df066b8ea2a7106df91d3203", size = 138036, upload-time = "2026-03-09T13:15:36.894Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/06/7399a607f434119c6e1fdc8ec89a8d51ccccadf3341dee4ead6bd14caaf5/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c31c13da98624f957b0fb1b5bae5383b2333c2c3f6793d9825dd5ce79b525cb7", size = 194295, upload-time = "2026-03-09T13:15:38.22Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/91/53255615acd2a1eaca307ede3c90eb550bae9c94581f8c00081b6b1c8f44/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-win_amd64.whl", hash = "sha256:1f1489f769582498610e015a8ef2d36f28f505ab3096d0e16b4858a9ec214f57", size = 75987, upload-time = "2026-03-09T13:15:39.65Z" },
+]
+
+[[package]]
+name = "kneed"
+version = "0.8.6"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+ { name = "scipy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/72/64/4bb8f8a7a4627b585a66d5bec0c9b30ae5b39a4caea1775c8bfb3fb3f4cf/kneed-0.8.6.tar.gz", hash = "sha256:65b22727c623661701f15edf057f2e6c73e2b1ad4e68cd9ca4291675c318b5ef", size = 13161, upload-time = "2026-03-20T21:01:51.966Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4a/cd/23c89d53c36028bccb39f55aa5dd24c4bdaab76c4d556ad43dc8cf026918/kneed-0.8.6-py3-none-any.whl", hash = "sha256:3412e7b70bce07717386d24fab37f0f985968d1b85ea0c749a6b98caccaf65ec", size = 10797, upload-time = "2026-03-20T21:01:50.87Z" },
+]
+
+[[package]]
+name = "legacy-api-wrap"
+version = "1.5"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/58/49/f06f94048c8974205730d40beca879e43b6eee08efb0101cfb8623e60f41/legacy_api_wrap-1.5.tar.gz", hash = "sha256:b41ba6532f3ebfe3a897a35a7f97dec3be04b92a450f6c2bcf89f1b91c9cadf2", size = 11610, upload-time = "2025-11-03T13:21:12.437Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/41/5b/058db09c45ba58a7321bdf2294cae651b37d6fec68117265af90cde043b0/legacy_api_wrap-1.5-py3-none-any.whl", hash = "sha256:5a8ea50e3e3bcbcdec3447b77034fd0d32cb2cf4089db799238708e4d7e0098d", size = 10182, upload-time = "2025-11-03T13:21:11.102Z" },
+]
+
+[[package]]
+name = "leidenalg"
+version = "0.12.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "igraph" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/82/b7/45bae25ab59321a14902bec2f47e32b92e76f8b4b3887a13ee69a0b32c60/leidenalg-0.12.0.tar.gz", hash = "sha256:c81a45a2fb874fe71e903d43a869d0efeb7f907af70476c03f6945ef0e8f44c5", size = 453641, upload-time = "2026-05-24T10:17:53.556Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/53/7f/0900c05ccbb27744f9dd11321dc2978e9f5ffe678db54f23eb94cf4e5951/leidenalg-0.12.0-cp38-abi3-macosx_10_9_x86_64.whl", hash = "sha256:96eec407f540d9ddb0c103dadc632d5519cfcdb39689fb21ae0ceeaa97bb2ec3", size = 2257399, upload-time = "2026-05-24T10:17:32.257Z" },
+ { url = "https://files.pythonhosted.org/packages/72/f6/d15c889c7c65dbc41c84fe83243c203b083f70cef62ed0acc630098b2012/leidenalg-0.12.0-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:2e0e1c2321c2e06a7f4f8a73ca15fcd54e0139a62afb448933c42badcfab8adc", size = 1926700, upload-time = "2026-05-24T10:17:33.946Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/36/7480eef0473bd87f69826e0dfb6bb1c1ee0915fa6d31be445d0bfcbd3a1a/leidenalg-0.12.0-cp38-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f6a97f698f13a588e1ae7344908cddee9f87d33acc226fc5c5b99752ebd71f8", size = 2546580, upload-time = "2026-05-24T10:17:35.133Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/45/cf5eb36b4517bb85397760a6033628695e8b0723b45f10f487b07cec84df/leidenalg-0.12.0-cp38-abi3-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:78df2e463f3ffffffda590e9b0d790107e59f722f095731b78a58803d1ca92e5", size = 2845846, upload-time = "2026-05-24T10:17:37.115Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/be/dba31e662c2ee028e20dd0308c1bc6e398dd7a1786fdd0821722537b4124/leidenalg-0.12.0-cp38-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fcbd89655243cc739d28ba23fe894587d8d7235124a9248ba375819585e70090", size = 2739255, upload-time = "2026-05-24T10:17:38.376Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/02/22597cf6d7ee093523ad5885a31499d698812db5744d98086383c3da08b5/leidenalg-0.12.0-cp38-abi3-musllinux_1_2_i686.whl", hash = "sha256:c6a6231d3db490ee3cc4004f214b0bb947dc3fabde75bd61d878bc3fb9685d06", size = 4072277, upload-time = "2026-05-24T10:17:39.881Z" },
+ { url = "https://files.pythonhosted.org/packages/05/e5/2f3322376fddd6b85790eaddcd314c9402bf81e984b35870acb4cfcbd0e1/leidenalg-0.12.0-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:63b49f5716c7cb155f3a3e25505bcc33f3c8c62105734fc152bc07518e63e9ab", size = 3799963, upload-time = "2026-05-24T10:17:41.035Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/35/f4f07ddac6718a11c70d858fd4afcc849ca84f485b5d7092bcf09492ec06/leidenalg-0.12.0-cp38-abi3-win32.whl", hash = "sha256:283cd987623e2c5e7b11ed3f15b7805d597a934bf387a7e820d2f673001b635f", size = 1675833, upload-time = "2026-05-24T10:17:42.663Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/77/0c8fb67729e83bddd43971cd4851f497a96810fca63f8029e3f008e82b1b/leidenalg-0.12.0-cp38-abi3-win_amd64.whl", hash = "sha256:f5d9529b44d5f6add0847d68fa6ced99f581357f6ca02b54c049eb3a2f45ff04", size = 1990568, upload-time = "2026-05-24T10:17:44.071Z" },
+]
+
+[[package]]
+name = "librt"
+version = "0.11.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/40/08/9e7f6b5d2b5bed6ad055cdd5925f192bb403a51280f86b56554d9d0699a2/librt-0.11.0.tar.gz", hash = "sha256:075dc3ef4458a278e0195cbf6ac9d38808d9b906c5a6c7f7f79c3888276a3fb1", size = 200139, upload-time = "2026-05-10T18:17:25.138Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8b/d0/07c77e067f0838949b43bd89232c29d72efebb9d2801a9750184eb706b71/librt-0.11.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b87504f1690a23b9a2cca841191a04f83895d4fc2dd04df91d82b1a04ca2ad46", size = 144147, upload-time = "2026-05-10T18:15:53.227Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/24/8493538fa4f62f982686398a5b8f68008138a75086abdea19ade64bf4255/librt-0.11.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:40071fc5fe0ce8daa6de616702314a01e1250711682b0523d6ab8d4525910cb3", size = 143614, upload-time = "2026-05-10T18:15:54.657Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/1e/f8bad050810d9171f34a1648ed910e56814c2ba61639f2bd53c6377ae24b/librt-0.11.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:137e79445c896a0ea7b265f52d23954e05b64222ee1af69e2cb34219067cbb67", size = 485538, upload-time = "2026-05-10T18:15:56.117Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/fe/3594ebfbaf03084ba4b120c9ba5c3183fd938a48725e9bbe6ff0a5159ad8/librt-0.11.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:cca6644054e78746d8d4ef238681f9c34ff8b584fe6b988ecebb8db3b15e622a", size = 479623, upload-time = "2026-05-10T18:15:57.544Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/da/5d1876984b3746c85dbd219dbfcb73c85f54ee263fd32e5b2a632ec14571/librt-0.11.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d5b0eea49f5562861ee8d757a32ef7d559c1d35be2aaaa1ec28941d74c9ffc8a", size = 513082, upload-time = "2026-05-10T18:15:58.805Z" },
+ { url = "https://files.pythonhosted.org/packages/19/6e/55bdf5d5ca00c3e18430690bf2c953d8d3ffd3c337418173d33dec985dc9/librt-0.11.0-cp312-cp312-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0d1029d7e1ae1a7e647ed6fb5df8c4ce2dffefb7a9f5fd1376a4554d96dac09f", size = 508105, upload-time = "2026-05-10T18:16:00.2Z" },
+ { url = "https://files.pythonhosted.org/packages/07/10/f1f23a7c595ee90ece4d35c851e5d104b1311a887ed1b4ac4c35bbd13da8/librt-0.11.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:bc3ce6b33c5828d9e80592011a5c584cb2ce86edbc4088405f70da47dc1d1b3b", size = 522268, upload-time = "2026-05-10T18:16:01.708Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/02/5720f5697a7f54b78b3aefbe20df3a48cedcff1276618c4aa481177942ed/librt-0.11.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:936c5995f3514a42111f20099397d8177c79b4d7e70961e396c6f5a0a3566766", size = 527348, upload-time = "2026-05-10T18:16:03.496Z" },
+ { url = "https://files.pythonhosted.org/packages/50/db/b4a47c6f91db4ff76348a0b3dd0cc65e090a078b765a810a62ff9434c3d3/librt-0.11.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:9bc0ca6ad9381cbe8e4aa6e5726e4c80c78115a6e9723c599ed1d73e092bc49d", size = 516294, upload-time = "2026-05-10T18:16:05.173Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/58/9384b2f4eb1ed1d273d40948a7c5c4b2360213b402ef3be4641c06299f9c/librt-0.11.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:070aa8c26c0a74774317a72df8851facc7f0f012a5b406557ac56992d92e1ec8", size = 553608, upload-time = "2026-05-10T18:16:06.839Z" },
+ { url = "https://files.pythonhosted.org/packages/21/7b/5aa8848a7c6a9278c79375146da1812e695754ceec5f005e6043461a7315/librt-0.11.0-cp312-cp312-win32.whl", hash = "sha256:6bf14feb84b05ae945277395451998c89c54d0def4070eb5c08de544930b245a", size = 101879, upload-time = "2026-05-10T18:16:08.103Z" },
+ { url = "https://files.pythonhosted.org/packages/37/33/8a745436944947575b584231750a41417de1a38cf6a2e9251d1065651c09/librt-0.11.0-cp312-cp312-win_amd64.whl", hash = "sha256:75672f0bc524ede266287d532d7923dbce94c7514ad07627bac3d0c6d92cc4d9", size = 119831, upload-time = "2026-05-10T18:16:09.174Z" },
+ { url = "https://files.pythonhosted.org/packages/59/67/a6739ac96e28b7855808bdb0370e250606104a859750d209e5a0716fe7ab/librt-0.11.0-cp312-cp312-win_arm64.whl", hash = "sha256:2f10cf143e4a9bb0f4f5af568a00df94a2d69ef41c2579584454bb0fe5cc642c", size = 103470, upload-time = "2026-05-10T18:16:10.369Z" },
+ { url = "https://files.pythonhosted.org/packages/82/61/e59168d4d0bf2bf90f4f0caf7a001bfc60254c3af4586013b04dc3ef517b/librt-0.11.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:78dc31f7fdfe9c9d0eb0e8f42d139db230e826415bbcabd9f0e9faaaee909894", size = 144119, upload-time = "2026-05-10T18:16:11.771Z" },
+ { url = "https://files.pythonhosted.org/packages/61/fd/caa1d60b12f7dd79ccea23054e06eeaebe266a5f52c40a6b651069200ce5/librt-0.11.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:fa475675db22290c3158e1d42326d0f5a65f04f44a0e68c3630a25b53560fb9c", size = 143565, upload-time = "2026-05-10T18:16:13.334Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/a9/dc744f5c2b4978d48db970be29f22716d3413d28b14ad99740817315cf2c/librt-0.11.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:621db29691044bdeda22e789e482e1b0f3a985d90e3426c9c6d17606416205ea", size = 485395, upload-time = "2026-05-10T18:16:14.729Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/21/7f8e97a1e4dae952a5a95948f6f8507a173bc1e669f54340bba6ca1ca31b/librt-0.11.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:a9010e2ed5b3a9e158c5fd966b3ab7e834bb3d3aacc8f66c91dd4b57a3799230", size = 479383, upload-time = "2026-05-10T18:16:16.321Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/6d/d8ee9c114bebf2c50e29ec2aa940826fccb62a645c3e4c18760987d0e16d/librt-0.11.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7c39513d8b7477a2e1ed8c43fc21c524e8d5a0f8d4e8b7b074dbdbe7820a08e2", size = 513010, upload-time = "2026-05-10T18:16:17.647Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/43/0b5708af2bd30a46400e72ba6bdaa8f066f15fb9a688527e34220e8d6c06/librt-0.11.0-cp313-cp313-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:7aef3cf1d5af86e770ab04bfd993dfc4ae8b8c17f66fb77dd4a7d50de7bbb1a3", size = 508433, upload-time = "2026-05-10T18:16:19.309Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/50/356187247d09013490481033183b3532b58acf8028bcb34b2b56a375c9b2/librt-0.11.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:557183ddc36babe46b27dd60facbd5adb4492181a5be887587d57cda6e092f21", size = 522595, upload-time = "2026-05-10T18:16:20.642Z" },
+ { url = "https://files.pythonhosted.org/packages/40/e7/c6ac4240899c7f3248079d5a9900debe0dadb3fdeaf856684c987105ba47/librt-0.11.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:83d3e1f72bd42f6c5c0b7daec530c3f829bd02db42c70b8ddf0c2d90a2459930", size = 527255, upload-time = "2026-05-10T18:16:22.352Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/b5/a81322dbeedeeaf9c1ee6f001734d28a09d8383ac9e6779bc24bbd0743c6/librt-0.11.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:4ce1f21fbe589bc1afd7872dece84fb0e1144f794a288e58a10d2c54a55c43be", size = 516847, upload-time = "2026-05-10T18:16:23.627Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/66/6e6323787d592b55204a42595ff1102da5115601b53a7e9ddebc889a6da5/librt-0.11.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:970b09f7044ea2b64c9da42fd3d335666518cfd1c6e8a182c95da73d0214b41e", size = 553920, upload-time = "2026-05-10T18:16:25.025Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/21/623f8ca230857102066d9ca8c6c1734995908c4d0d1bee7bb2ef0021cb33/librt-0.11.0-cp313-cp313-win32.whl", hash = "sha256:78fddc31cd4d3caa897ad5d31f856b1faadc9474021ad6cb182b9018793e254e", size = 101898, upload-time = "2026-05-10T18:16:26.649Z" },
+ { url = "https://files.pythonhosted.org/packages/b3/1d/b4ebd44dd723f768469007515cb92251e0ae286c94c140f374801140fa74/librt-0.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:8ca8aa88751a775870b764e93bad5135385f563cb8dcee399abf034ea4d3cb47", size = 119812, upload-time = "2026-05-10T18:16:27.859Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/e4/b2f4ca7965ca373b491cdb4bc25cdb30c1649ca81a8782056a83850292a9/librt-0.11.0-cp313-cp313-win_arm64.whl", hash = "sha256:96f044bb325fd9cf1a723015638c219e9143f0dfbc0ca54c565df2b7fc748b44", size = 103448, upload-time = "2026-05-10T18:16:29.066Z" },
+ { url = "https://files.pythonhosted.org/packages/29/eb/dbce197da4e227779e56b5735f2decc3eb36e55a1cdbf1bd65d6639d76c1/librt-0.11.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:4a017a95e5837dc15a8c5661d60e05daa96b90908b1aa6b7acdf443cd25c8ebd", size = 143345, upload-time = "2026-05-10T18:16:30.674Z" },
+ { url = "https://files.pythonhosted.org/packages/76/a3/254bebd0c11c8ba684018efb8006ff22e466abce445215cca6c778e7d9de/librt-0.11.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:b1ecbd9819deccc39b7542bf4d2a740d8a620694d39989e58661d3763458f8d4", size = 143131, upload-time = "2026-05-10T18:16:32.037Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/3f/f77d6122d21ac7bf6ae8a7dfced1bd2a7ac545d3273ebdcaf8042f6d619f/librt-0.11.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7da327dacd7be8f8ec36547373550744a3cc0e536d54665cd83f8bcd961200e8", size = 477024, upload-time = "2026-05-10T18:16:33.493Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/0a/2c996dadebaa7d9bbbd43ef2d4f3e66b6da545f838a41694ef6172cebec8/librt-0.11.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:0dc56b1f8d06e60db362cc3fdae206681817f86ce4725d34511473487f12a34b", size = 474221, upload-time = "2026-05-10T18:16:34.864Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/7e/f5d92af8486b8272c23b3e686b46ff72d89c8169585eb61eef01a2ac7147/librt-0.11.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:05fb8fb2ab90e21c8d12ea240d744ad514da9baf381ebfa70d91d20d21713175", size = 505174, upload-time = "2026-05-10T18:16:36.705Z" },
+ { url = "https://files.pythonhosted.org/packages/af/1a/cb0734fe86398eb33193ab753b7326255c74cac5eb09e76b9b16536e7adb/librt-0.11.0-cp314-cp314-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:cae74872be221df4374d10fec61f93ed1513b9546ea84f2c0bf73ab3e9bd0b03", size = 497216, upload-time = "2026-05-10T18:16:38.418Z" },
+ { url = "https://files.pythonhosted.org/packages/18/06/094820f91558b66e29943c0ec41c9914f460f48dd51fc503c3101e10842d/librt-0.11.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:32bcc918c0148eb7e3d57385125bac7e5f9e4359d05f07448b09f6f778c2f31c", size = 513921, upload-time = "2026-05-10T18:16:39.848Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/c2/00de9018871a282f530cacb457d5ec0428f6ac7e6fedde9aff7468d9fb04/librt-0.11.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:f9743fc99135d5f78d2454435615f6dec0473ca507c26ce9d92b10b562a280d3", size = 520850, upload-time = "2026-05-10T18:16:41.471Z" },
+ { url = "https://files.pythonhosted.org/packages/51/9d/64631832348fd1834fb3a61b996434edddaaf25a31d03b0a76273159d2cf/librt-0.11.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:5ba067f4aadae8fda802d91d2124c90c42195ff32d9161d3549e6d05cfe26f96", size = 504237, upload-time = "2026-05-10T18:16:43.15Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/ec/ae5525eb16edc827a044e7bb8777a455ff95d4bca9379e7e6bddd7383647/librt-0.11.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:de3bf945454d032f9e390b85c4072e0a0570bf825421c8be0e71209fa65e1abe", size = 546261, upload-time = "2026-05-10T18:16:44.408Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/09/adce371f27ca039411da9659f7430fcc2ba6cd0c7b3e4467a0f091be7fa9/librt-0.11.0-cp314-cp314-win32.whl", hash = "sha256:d2277a05f6dcb9fd13db9566aac4fabd68c3ea1ea46ee5567d4eef8efa495a2f", size = 96965, upload-time = "2026-05-10T18:16:46.039Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/ee/8ac720d98548f173c7ce2e632a7ca94673f74cacd5c8162a84af5b35958a/librt-0.11.0-cp314-cp314-win_amd64.whl", hash = "sha256:ab73e8db5e3f564d812c1f5c3a175930a5f9bc96ccb5e3b22a34d7858b401cf7", size = 115151, upload-time = "2026-05-10T18:16:47.133Z" },
+ { url = "https://files.pythonhosted.org/packages/94/20/c900cf14efeb09b6bef2b2dff20779f73464b97fd58d1c6bccc379588ae3/librt-0.11.0-cp314-cp314-win_arm64.whl", hash = "sha256:aea3caa317752e3a466fa8af45d91ee0ea8c7fdd96e42b0a8dd9b76a7931eba1", size = 98850, upload-time = "2026-05-10T18:16:48.597Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/71/944bfe4b64e12abffcd3c15e1cce07f72f3d55655083786285f4dedeb532/librt-0.11.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:d1b36540d7aaf9b9101b3a6f376c8d8e9f7a9aec93ed05918f2c69d493ffef72", size = 151138, upload-time = "2026-05-10T18:16:49.839Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/10/99e64a5c86989357fda078c8143c533389585f6473b7439172dd8f3b3b2d/librt-0.11.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:efbb343ab2ce3540f4ecbe6315d677ed70f37cd9a72b1e58066c918ca83acbaa", size = 151976, upload-time = "2026-05-10T18:16:51.062Z" },
+ { url = "https://files.pythonhosted.org/packages/21/31/5072ad880946d83e5ea4147d6d018c78eefce85b77819b19bdd0ee229435/librt-0.11.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:aa0dd688aab3f7914d3e6e5e3554978e0383312fb8e771d84be008a35b9ee548", size = 557927, upload-time = "2026-05-10T18:16:52.632Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/8d/70b5fb7cfbab60edbe7381614ab985da58e144fbf465c86d44c95f43cdca/librt-0.11.0-cp314-cp314t-manylinux2014_i686.manylinux_2_17_i686.manylinux_2_28_i686.whl", hash = "sha256:f5fb36b8c6c63fdcbb1d526d94c0d1331610d43f4118cc1beb4efef4f3faacb2", size = 539698, upload-time = "2026-05-10T18:16:53.934Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/a3/ba3495a0b3edbd24a4cae0d1d3c64f39a9fc45d06e812101289b50c1a619/librt-0.11.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4a9a237d13addb93715b6fee74023d5ee3469b53fce527626c0e088aa585805f", size = 577162, upload-time = "2026-05-10T18:16:55.589Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/db/36e25fb81f99937ff1b96612a1dc9fd66f039cb9cc3aee12c01fac31aab9/librt-0.11.0-cp314-cp314t-manylinux_2_34_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5ddd17bd87b2c56ddd60e546a7984a2e64c4e8eab92fb4cf3830a48ad5469d51", size = 566494, upload-time = "2026-05-10T18:16:56.975Z" },
+ { url = "https://files.pythonhosted.org/packages/33/0d/3f622b47f0b013eeb9cf4cc07ae9bfe378d832a4eec998b2b209fe84244d/librt-0.11.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:bd43992b4473d42f12ff9e68326079f0696d9d4e6000e8f39a0238d482ba6ee2", size = 596858, upload-time = "2026-05-10T18:16:58.374Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/02/71b90bc93039c46a2000651f6ad60122b114c8f54c4ad306e0e96f5b75ad/librt-0.11.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:f8e3e8056dd674e279741485e2e512d6e9a751c7455809d0114e6ebf8d781085", size = 590318, upload-time = "2026-05-10T18:16:59.676Z" },
+ { url = "https://files.pythonhosted.org/packages/04/04/418cb3f75621e2b761fb1ab0f017f4d70a1a72a6e7c74ee4f7e8d198c2f3/librt-0.11.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:c1f708d8ae9c56cf38a903c44297243d2ec83fd82b396b977e0144a3e76217e3", size = 575115, upload-time = "2026-05-10T18:17:01.007Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/2c/5a2183ac58dd911f26b5d7e7d7d8f1d87fcecdddd99d6c12169a258ff62c/librt-0.11.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:0add982e0e7b9fc14cf4b33789d5f13f66581889b88c2f58099f6ce8f92617bd", size = 617918, upload-time = "2026-05-10T18:17:02.682Z" },
+ { url = "https://files.pythonhosted.org/packages/15/1f/dc6771a52592a4451be6effa200cbfc9cec61e4393d3033d81a9d307961d/librt-0.11.0-cp314-cp314t-win32.whl", hash = "sha256:2b481d846ac894c4e8403c5fd0e87c5d11d6499e404b474602508a224ff531c8", size = 103562, upload-time = "2026-05-10T18:17:03.99Z" },
+ { url = "https://files.pythonhosted.org/packages/62/4a/7d1415567027286a75ba1093ec4aca11f073e0f559c530cf3e0a757ad55c/librt-0.11.0-cp314-cp314t-win_amd64.whl", hash = "sha256:28edb433edde181112a908c78907af28f964eabc15f4dd16c9d66c834302677c", size = 124327, upload-time = "2026-05-10T18:17:05.465Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/62/b40b382fa0c66fee1478073eb8db352a4a6beda4a1adccf1df911d8c289c/librt-0.11.0-cp314-cp314t-win_arm64.whl", hash = "sha256:dee008f20b542e3cd162ba338a7f9ec0f6d23d395f66fe8aeeec3c9d067ea253", size = 102572, upload-time = "2026-05-10T18:17:06.809Z" },
+]
+
+[[package]]
+name = "llvmlite"
+version = "0.47.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/01/88/a8952b6d5c21e74cbf158515b779666f692846502623e9e3c39d8e8ba25f/llvmlite-0.47.0.tar.gz", hash = "sha256:62031ce968ec74e95092184d4b0e857e444f8fdff0b8f9213707699570c33ccc", size = 193614, upload-time = "2026-03-31T18:29:53.497Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fa/48/4b7fe0e34c169fa2f12532916133e0b219d2823b540733651b34fdac509a/llvmlite-0.47.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:306a265f408c259067257a732c8e159284334018b4083a9e35f67d19792b164f", size = 37232769, upload-time = "2026-03-31T18:28:43.735Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/4b/e3f2cd17822cf772a4a51a0a8080b0032e6d37b2dbe8cfb724eac4e31c52/llvmlite-0.47.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5853bf26160857c0c2573415ff4efe01c4c651e59e2c55c2a088740acfee51cd", size = 56275178, upload-time = "2026-03-31T18:28:48.342Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/55/a3b4a543185305a9bdf3d9759d53646ed96e55e7dfd43f53e7a421b8fbae/llvmlite-0.47.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:003bcf7fa579e14db59c1a1e113f93ab8a06b56a4be31c7f08264d1d4072d077", size = 55128632, upload-time = "2026-03-31T18:28:52.901Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/f5/d281ae0f79378a5a91f308ea9fdb9f9cc068fddd09629edc0725a5a8fde1/llvmlite-0.47.0-cp312-cp312-win_amd64.whl", hash = "sha256:f3079f25bdc24cd9d27c4b2b5e68f5f60c4fdb7e8ad5ee2b9b006007558f9df7", size = 38138692, upload-time = "2026-03-31T18:28:57.147Z" },
+ { url = "https://files.pythonhosted.org/packages/77/6f/4615353e016799f80fa52ccb270a843c413b22361fadda2589b2922fb9b0/llvmlite-0.47.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:a3c6a735d4e1041808434f9d440faa3d78d9b4af2ee64d05a66f351883b6ceec", size = 37232771, upload-time = "2026-03-31T18:29:01.324Z" },
+ { url = "https://files.pythonhosted.org/packages/31/b8/69f5565f1a280d032525878a86511eebed0645818492feeb169dfb20ae8e/llvmlite-0.47.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2699a74321189e812d476a43d6d7f652f51811e7b5aad9d9bba842a1c7927acb", size = 56275178, upload-time = "2026-03-31T18:29:05.748Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/da/b32cafcb926fb0ce2aa25553bf32cb8764af31438f40e2481df08884c947/llvmlite-0.47.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6c6951e2b29930227963e53ee152441f0e14be92e9d4231852102d986c761e40", size = 55128632, upload-time = "2026-03-31T18:29:11.235Z" },
+ { url = "https://files.pythonhosted.org/packages/46/9f/4898b44e4042c60fafcb1162dfb7014f6f15b1ec19bf29cfea6bf26df90d/llvmlite-0.47.0-cp313-cp313-win_amd64.whl", hash = "sha256:c2e9adf8698d813a9a5efb2d4370caf344dbc1e145019851fee6a6f319ba760e", size = 38138695, upload-time = "2026-03-31T18:29:15.43Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/d4/33c8af00f0bf6f552d74f3a054f648af2c5bc6bece97972f3bfadce4f5ec/llvmlite-0.47.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:de966c626c35c9dff5ae7bf12db25637738d0df83fc370cf793bc94d43d92d14", size = 37232773, upload-time = "2026-03-31T18:29:19.453Z" },
+ { url = "https://files.pythonhosted.org/packages/64/1d/a760e993e0c0ba6db38d46b9f48f6c7dceb8ac838824997fb9e25f97bc04/llvmlite-0.47.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ddbccff2aeaff8670368340a158abefc032fe9b3ccf7d9c496639263d00151aa", size = 56275176, upload-time = "2026-03-31T18:29:24.149Z" },
+ { url = "https://files.pythonhosted.org/packages/84/3b/e679bc3b29127182a7f4aa2d2e9e5bea42adb93fb840484147d59c236299/llvmlite-0.47.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d4a7b778a2e144fc64468fb9bf509ac1226c9813a00b4d7afea5d988c4e22fca", size = 55128631, upload-time = "2026-03-31T18:29:29.536Z" },
+ { url = "https://files.pythonhosted.org/packages/be/f7/19e2a09c62809c9e63bbd14ce71fb92c6ff7b7b3045741bb00c781efc3c9/llvmlite-0.47.0-cp314-cp314-win_amd64.whl", hash = "sha256:694e3c2cdc472ed2bd8bd4555ca002eec4310961dd58ef791d508f57b5cc4c94", size = 39153826, upload-time = "2026-03-31T18:29:33.681Z" },
+ { url = "https://files.pythonhosted.org/packages/40/a1/581a8c707b5e80efdbbe1dd94527404d33fe50bceb71f39d5a7e11bd57b7/llvmlite-0.47.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:92ec8a169a20b473c1c54d4695e371bde36489fc1efa3688e11e99beba0abf9c", size = 37232772, upload-time = "2026-03-31T18:29:37.952Z" },
+ { url = "https://files.pythonhosted.org/packages/11/03/16090dd6f74ba2b8b922276047f15962fbeea0a75d5601607edb301ba945/llvmlite-0.47.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fa1cbd800edd3b20bc141521f7fd45a6185a5b84109aa6855134e81397ffe72b", size = 56275178, upload-time = "2026-03-31T18:29:42.58Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/cb/0abf1dd4c5286a95ffe0c1d8c67aec06b515894a0dd2ac97f5e27b82ab0b/llvmlite-0.47.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f6725179b89f03b17dabe236ff3422cb8291b4c1bf40af152826dfd34e350ae8", size = 55128632, upload-time = "2026-03-31T18:29:46.939Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/79/d3bbab197e86e0ff4f9c07122895b66a3e0d024247fcff7f12c473cb36d9/llvmlite-0.47.0-cp314-cp314t-win_amd64.whl", hash = "sha256:6842cf6f707ec4be3d985a385ad03f72b2d724439e118fcbe99b2929964f0453", size = 39153839, upload-time = "2026-03-31T18:29:51.004Z" },
+]
+
+[[package]]
+name = "loguru"
+version = "0.7.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+ { name = "win32-setctime", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/3a/05/a1dae3dffd1116099471c643b8924f5aa6524411dc6c63fdae648c4f1aca/loguru-0.7.3.tar.gz", hash = "sha256:19480589e77d47b8d85b2c827ad95d49bf31b0dcde16593892eb51dd18706eb6", size = 63559, upload-time = "2024-12-06T11:20:56.608Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0c/29/0348de65b8cc732daa3e33e67806420b2ae89bdce2b04af740289c5c6c8c/loguru-0.7.3-py3-none-any.whl", hash = "sha256:31a33c10c8e1e10422bfd431aeb5d351c7cf7fa671e3c4df004162264b28220c", size = 61595, upload-time = "2024-12-06T11:20:54.538Z" },
+]
+
+[[package]]
+name = "markdown-it-py"
+version = "4.2.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "mdurl" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/06/ff/7841249c247aa650a76b9ee4bbaeae59370dc8bfd2f6c01f3630c35eb134/markdown_it_py-4.2.0.tar.gz", hash = "sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49", size = 82454, upload-time = "2026-05-07T12:08:28.36Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" },
+]
+
+[[package]]
+name = "markupsafe"
+version = "3.0.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/7e/99/7690b6d4034fffd95959cbe0c02de8deb3098cc577c67bb6a24fe5d7caa7/markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698", size = 80313, upload-time = "2025-09-27T18:37:40.426Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5a/72/147da192e38635ada20e0a2e1a51cf8823d2119ce8883f7053879c2199b5/markupsafe-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d53197da72cc091b024dd97249dfc7794d6a56530370992a5e1a08983ad9230e", size = 11615, upload-time = "2025-09-27T18:36:30.854Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/81/7e4e08678a1f98521201c3079f77db69fb552acd56067661f8c2f534a718/markupsafe-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1872df69a4de6aead3491198eaf13810b565bdbeec3ae2dc8780f14458ec73ce", size = 12020, upload-time = "2025-09-27T18:36:31.971Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/2c/799f4742efc39633a1b54a92eec4082e4f815314869865d876824c257c1e/markupsafe-3.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a7e8ae81ae39e62a41ec302f972ba6ae23a5c5396c8e60113e9066ef893da0d", size = 24332, upload-time = "2025-09-27T18:36:32.813Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/2e/8d0c2ab90a8c1d9a24f0399058ab8519a3279d1bd4289511d74e909f060e/markupsafe-3.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d6dd0be5b5b189d31db7cda48b91d7e0a9795f31430b7f271219ab30f1d3ac9d", size = 22947, upload-time = "2025-09-27T18:36:33.86Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/54/887f3092a85238093a0b2154bd629c89444f395618842e8b0c41783898ea/markupsafe-3.0.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:94c6f0bb423f739146aec64595853541634bde58b2135f27f61c1ffd1cd4d16a", size = 21962, upload-time = "2025-09-27T18:36:35.099Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/2f/336b8c7b6f4a4d95e91119dc8521402461b74a485558d8f238a68312f11c/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:be8813b57049a7dc738189df53d69395eba14fb99345e0a5994914a3864c8a4b", size = 23760, upload-time = "2025-09-27T18:36:36.001Z" },
+ { url = "https://files.pythonhosted.org/packages/32/43/67935f2b7e4982ffb50a4d169b724d74b62a3964bc1a9a527f5ac4f1ee2b/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:83891d0e9fb81a825d9a6d61e3f07550ca70a076484292a70fde82c4b807286f", size = 21529, upload-time = "2025-09-27T18:36:36.906Z" },
+ { url = "https://files.pythonhosted.org/packages/89/e0/4486f11e51bbba8b0c041098859e869e304d1c261e59244baa3d295d47b7/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:77f0643abe7495da77fb436f50f8dab76dbc6e5fd25d39589a0f1fe6548bfa2b", size = 23015, upload-time = "2025-09-27T18:36:37.868Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/e1/78ee7a023dac597a5825441ebd17170785a9dab23de95d2c7508ade94e0e/markupsafe-3.0.3-cp312-cp312-win32.whl", hash = "sha256:d88b440e37a16e651bda4c7c2b930eb586fd15ca7406cb39e211fcff3bf3017d", size = 14540, upload-time = "2025-09-27T18:36:38.761Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/5b/bec5aa9bbbb2c946ca2733ef9c4ca91c91b6a24580193e891b5f7dbe8e1e/markupsafe-3.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:26a5784ded40c9e318cfc2bdb30fe164bdb8665ded9cd64d500a34fb42067b1c", size = 15105, upload-time = "2025-09-27T18:36:39.701Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/f1/216fc1bbfd74011693a4fd837e7026152e89c4bcf3e77b6692fba9923123/markupsafe-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f", size = 13906, upload-time = "2025-09-27T18:36:40.689Z" },
+ { url = "https://files.pythonhosted.org/packages/38/2f/907b9c7bbba283e68f20259574b13d005c121a0fa4c175f9bed27c4597ff/markupsafe-3.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e1cf1972137e83c5d4c136c43ced9ac51d0e124706ee1c8aa8532c1287fa8795", size = 11622, upload-time = "2025-09-27T18:36:41.777Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/d9/5f7756922cdd676869eca1c4e3c0cd0df60ed30199ffd775e319089cb3ed/markupsafe-3.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:116bb52f642a37c115f517494ea5feb03889e04df47eeff5b130b1808ce7c219", size = 12029, upload-time = "2025-09-27T18:36:43.257Z" },
+ { url = "https://files.pythonhosted.org/packages/00/07/575a68c754943058c78f30db02ee03a64b3c638586fba6a6dd56830b30a3/markupsafe-3.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:133a43e73a802c5562be9bbcd03d090aa5a1fe899db609c29e8c8d815c5f6de6", size = 24374, upload-time = "2025-09-27T18:36:44.508Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/21/9b05698b46f218fc0e118e1f8168395c65c8a2c750ae2bab54fc4bd4e0e8/markupsafe-3.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ccfcd093f13f0f0b7fdd0f198b90053bf7b2f02a3927a30e63f3ccc9df56b676", size = 22980, upload-time = "2025-09-27T18:36:45.385Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/71/544260864f893f18b6827315b988c146b559391e6e7e8f7252839b1b846a/markupsafe-3.0.3-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:509fa21c6deb7a7a273d629cf5ec029bc209d1a51178615ddf718f5918992ab9", size = 21990, upload-time = "2025-09-27T18:36:46.916Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/28/b50fc2f74d1ad761af2f5dcce7492648b983d00a65b8c0e0cb457c82ebbe/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a4afe79fb3de0b7097d81da19090f4df4f8d3a2b3adaa8764138aac2e44f3af1", size = 23784, upload-time = "2025-09-27T18:36:47.884Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/76/104b2aa106a208da8b17a2fb72e033a5a9d7073c68f7e508b94916ed47a9/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:795e7751525cae078558e679d646ae45574b47ed6e7771863fcc079a6171a0fc", size = 21588, upload-time = "2025-09-27T18:36:48.82Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/99/16a5eb2d140087ebd97180d95249b00a03aa87e29cc224056274f2e45fd6/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8485f406a96febb5140bfeca44a73e3ce5116b2501ac54fe953e488fb1d03b12", size = 23041, upload-time = "2025-09-27T18:36:49.797Z" },
+ { url = "https://files.pythonhosted.org/packages/19/bc/e7140ed90c5d61d77cea142eed9f9c303f4c4806f60a1044c13e3f1471d0/markupsafe-3.0.3-cp313-cp313-win32.whl", hash = "sha256:bdd37121970bfd8be76c5fb069c7751683bdf373db1ed6c010162b2a130248ed", size = 14543, upload-time = "2025-09-27T18:36:51.584Z" },
+ { url = "https://files.pythonhosted.org/packages/05/73/c4abe620b841b6b791f2edc248f556900667a5a1cf023a6646967ae98335/markupsafe-3.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:9a1abfdc021a164803f4d485104931fb8f8c1efd55bc6b748d2f5774e78b62c5", size = 15113, upload-time = "2025-09-27T18:36:52.537Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/3a/fa34a0f7cfef23cf9500d68cb7c32dd64ffd58a12b09225fb03dd37d5b80/markupsafe-3.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:7e68f88e5b8799aa49c85cd116c932a1ac15caaa3f5db09087854d218359e485", size = 13911, upload-time = "2025-09-27T18:36:53.513Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/d7/e05cd7efe43a88a17a37b3ae96e79a19e846f3f456fe79c57ca61356ef01/markupsafe-3.0.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:218551f6df4868a8d527e3062d0fb968682fe92054e89978594c28e642c43a73", size = 11658, upload-time = "2025-09-27T18:36:54.819Z" },
+ { url = "https://files.pythonhosted.org/packages/99/9e/e412117548182ce2148bdeacdda3bb494260c0b0184360fe0d56389b523b/markupsafe-3.0.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:3524b778fe5cfb3452a09d31e7b5adefeea8c5be1d43c4f810ba09f2ceb29d37", size = 12066, upload-time = "2025-09-27T18:36:55.714Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/e6/fa0ffcda717ef64a5108eaa7b4f5ed28d56122c9a6d70ab8b72f9f715c80/markupsafe-3.0.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4e885a3d1efa2eadc93c894a21770e4bc67899e3543680313b09f139e149ab19", size = 25639, upload-time = "2025-09-27T18:36:56.908Z" },
+ { url = "https://files.pythonhosted.org/packages/96/ec/2102e881fe9d25fc16cb4b25d5f5cde50970967ffa5dddafdb771237062d/markupsafe-3.0.3-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8709b08f4a89aa7586de0aadc8da56180242ee0ada3999749b183aa23df95025", size = 23569, upload-time = "2025-09-27T18:36:57.913Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/30/6f2fce1f1f205fc9323255b216ca8a235b15860c34b6798f810f05828e32/markupsafe-3.0.3-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b8512a91625c9b3da6f127803b166b629725e68af71f8184ae7e7d54686a56d6", size = 23284, upload-time = "2025-09-27T18:36:58.833Z" },
+ { url = "https://files.pythonhosted.org/packages/58/47/4a0ccea4ab9f5dcb6f79c0236d954acb382202721e704223a8aafa38b5c8/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9b79b7a16f7fedff2495d684f2b59b0457c3b493778c9eed31111be64d58279f", size = 24801, upload-time = "2025-09-27T18:36:59.739Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/70/3780e9b72180b6fecb83a4814d84c3bf4b4ae4bf0b19c27196104149734c/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:12c63dfb4a98206f045aa9563db46507995f7ef6d83b2f68eda65c307c6829eb", size = 22769, upload-time = "2025-09-27T18:37:00.719Z" },
+ { url = "https://files.pythonhosted.org/packages/98/c5/c03c7f4125180fc215220c035beac6b9cb684bc7a067c84fc69414d315f5/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8f71bc33915be5186016f675cd83a1e08523649b0e33efdb898db577ef5bb009", size = 23642, upload-time = "2025-09-27T18:37:01.673Z" },
+ { url = "https://files.pythonhosted.org/packages/80/d6/2d1b89f6ca4bff1036499b1e29a1d02d282259f3681540e16563f27ebc23/markupsafe-3.0.3-cp313-cp313t-win32.whl", hash = "sha256:69c0b73548bc525c8cb9a251cddf1931d1db4d2258e9599c28c07ef3580ef354", size = 14612, upload-time = "2025-09-27T18:37:02.639Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/98/e48a4bfba0a0ffcf9925fe2d69240bfaa19c6f7507b8cd09c70684a53c1e/markupsafe-3.0.3-cp313-cp313t-win_amd64.whl", hash = "sha256:1b4b79e8ebf6b55351f0d91fe80f893b4743f104bff22e90697db1590e47a218", size = 15200, upload-time = "2025-09-27T18:37:03.582Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/72/e3cc540f351f316e9ed0f092757459afbc595824ca724cbc5a5d4263713f/markupsafe-3.0.3-cp313-cp313t-win_arm64.whl", hash = "sha256:ad2cf8aa28b8c020ab2fc8287b0f823d0a7d8630784c31e9ee5edea20f406287", size = 13973, upload-time = "2025-09-27T18:37:04.929Z" },
+ { url = "https://files.pythonhosted.org/packages/33/8a/8e42d4838cd89b7dde187011e97fe6c3af66d8c044997d2183fbd6d31352/markupsafe-3.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:eaa9599de571d72e2daf60164784109f19978b327a3910d3e9de8c97b5b70cfe", size = 11619, upload-time = "2025-09-27T18:37:06.342Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/64/7660f8a4a8e53c924d0fa05dc3a55c9cee10bbd82b11c5afb27d44b096ce/markupsafe-3.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c47a551199eb8eb2121d4f0f15ae0f923d31350ab9280078d1e5f12b249e0026", size = 12029, upload-time = "2025-09-27T18:37:07.213Z" },
+ { url = "https://files.pythonhosted.org/packages/da/ef/e648bfd021127bef5fa12e1720ffed0c6cbb8310c8d9bea7266337ff06de/markupsafe-3.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f34c41761022dd093b4b6896d4810782ffbabe30f2d443ff5f083e0cbbb8c737", size = 24408, upload-time = "2025-09-27T18:37:09.572Z" },
+ { url = "https://files.pythonhosted.org/packages/41/3c/a36c2450754618e62008bf7435ccb0f88053e07592e6028a34776213d877/markupsafe-3.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:457a69a9577064c05a97c41f4e65148652db078a3a509039e64d3467b9e7ef97", size = 23005, upload-time = "2025-09-27T18:37:10.58Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/20/b7fdf89a8456b099837cd1dc21974632a02a999ec9bf7ca3e490aacd98e7/markupsafe-3.0.3-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e8afc3f2ccfa24215f8cb28dcf43f0113ac3c37c2f0f0806d8c70e4228c5cf4d", size = 22048, upload-time = "2025-09-27T18:37:11.547Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/a7/591f592afdc734f47db08a75793a55d7fbcc6902a723ae4cfbab61010cc5/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:ec15a59cf5af7be74194f7ab02d0f59a62bdcf1a537677ce67a2537c9b87fcda", size = 23821, upload-time = "2025-09-27T18:37:12.48Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/33/45b24e4f44195b26521bc6f1a82197118f74df348556594bd2262bda1038/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:0eb9ff8191e8498cca014656ae6b8d61f39da5f95b488805da4bb029cccbfbaf", size = 21606, upload-time = "2025-09-27T18:37:13.485Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/0e/53dfaca23a69fbfbbf17a4b64072090e70717344c52eaaaa9c5ddff1e5f0/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2713baf880df847f2bece4230d4d094280f4e67b1e813eec43b4c0e144a34ffe", size = 23043, upload-time = "2025-09-27T18:37:14.408Z" },
+ { url = "https://files.pythonhosted.org/packages/46/11/f333a06fc16236d5238bfe74daccbca41459dcd8d1fa952e8fbd5dccfb70/markupsafe-3.0.3-cp314-cp314-win32.whl", hash = "sha256:729586769a26dbceff69f7a7dbbf59ab6572b99d94576a5592625d5b411576b9", size = 14747, upload-time = "2025-09-27T18:37:15.36Z" },
+ { url = "https://files.pythonhosted.org/packages/28/52/182836104b33b444e400b14f797212f720cbc9ed6ba34c800639d154e821/markupsafe-3.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581", size = 15341, upload-time = "2025-09-27T18:37:16.496Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/18/acf23e91bd94fd7b3031558b1f013adfa21a8e407a3fdb32745538730382/markupsafe-3.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:5a7d5dc5140555cf21a6fefbdbf8723f06fcd2f63ef108f2854de715e4422cb4", size = 14073, upload-time = "2025-09-27T18:37:17.476Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/f0/57689aa4076e1b43b15fdfa646b04653969d50cf30c32a102762be2485da/markupsafe-3.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:1353ef0c1b138e1907ae78e2f6c63ff67501122006b0f9abad68fda5f4ffc6ab", size = 11661, upload-time = "2025-09-27T18:37:18.453Z" },
+ { url = "https://files.pythonhosted.org/packages/89/c3/2e67a7ca217c6912985ec766c6393b636fb0c2344443ff9d91404dc4c79f/markupsafe-3.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1085e7fbddd3be5f89cc898938f42c0b3c711fdcb37d75221de2666af647c175", size = 12069, upload-time = "2025-09-27T18:37:19.332Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/00/be561dce4e6ca66b15276e184ce4b8aec61fe83662cce2f7d72bd3249d28/markupsafe-3.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b52b4fb9df4eb9ae465f8d0c228a00624de2334f216f178a995ccdcf82c4634", size = 25670, upload-time = "2025-09-27T18:37:20.245Z" },
+ { url = "https://files.pythonhosted.org/packages/50/09/c419f6f5a92e5fadde27efd190eca90f05e1261b10dbd8cbcb39cd8ea1dc/markupsafe-3.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50", size = 23598, upload-time = "2025-09-27T18:37:21.177Z" },
+ { url = "https://files.pythonhosted.org/packages/22/44/a0681611106e0b2921b3033fc19bc53323e0b50bc70cffdd19f7d679bb66/markupsafe-3.0.3-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f190daf01f13c72eac4efd5c430a8de82489d9cff23c364c3ea822545032993e", size = 23261, upload-time = "2025-09-27T18:37:22.167Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/57/1b0b3f100259dc9fffe780cfb60d4be71375510e435efec3d116b6436d43/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e56b7d45a839a697b5eb268c82a71bd8c7f6c94d6fd50c3d577fa39a9f1409f5", size = 24835, upload-time = "2025-09-27T18:37:23.296Z" },
+ { url = "https://files.pythonhosted.org/packages/26/6a/4bf6d0c97c4920f1597cc14dd720705eca0bf7c787aebc6bb4d1bead5388/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:f3e98bb3798ead92273dc0e5fd0f31ade220f59a266ffd8a4f6065e0a3ce0523", size = 22733, upload-time = "2025-09-27T18:37:24.237Z" },
+ { url = "https://files.pythonhosted.org/packages/14/c7/ca723101509b518797fedc2fdf79ba57f886b4aca8a7d31857ba3ee8281f/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5678211cb9333a6468fb8d8be0305520aa073f50d17f089b5b4b477ea6e67fdc", size = 23672, upload-time = "2025-09-27T18:37:25.271Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/df/5bd7a48c256faecd1d36edc13133e51397e41b73bb77e1a69deab746ebac/markupsafe-3.0.3-cp314-cp314t-win32.whl", hash = "sha256:915c04ba3851909ce68ccc2b8e2cd691618c4dc4c4232fb7982bca3f41fd8c3d", size = 14819, upload-time = "2025-09-27T18:37:26.285Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/8a/0402ba61a2f16038b48b39bccca271134be00c5c9f0f623208399333c448/markupsafe-3.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4faffd047e07c38848ce017e8725090413cd80cbc23d86e55c587bf979e579c9", size = 15426, upload-time = "2025-09-27T18:37:27.316Z" },
+ { url = "https://files.pythonhosted.org/packages/70/bc/6f1c2f612465f5fa89b95bead1f44dcb607670fd42891d8fdcd5d039f4f4/markupsafe-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa", size = 14146, upload-time = "2025-09-27T18:37:28.327Z" },
+]
+
+[[package]]
+name = "matplotlib"
+version = "3.11.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "contourpy" },
+ { name = "cycler" },
+ { name = "fonttools" },
+ { name = "kiwisolver" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "pillow" },
+ { name = "pyparsing" },
+ { name = "python-dateutil" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/1f/24/080c99d223d158d3a8902769269ab6da5b50f7a0e6e072513907e02b7a6c/matplotlib-3.11.0.tar.gz", hash = "sha256:68c0c7be01b30dcca3638934f7f591df73401235cbdbf0d1ab1c71e7db7f8b57", size = 33251176, upload-time = "2026-06-12T02:29:15.508Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/da/17/f5276b496c61477a6c4fc5e7401f4bfe1c2e5ef7c6cd67896f2ade3809cb/matplotlib-3.11.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:06b5872e9cf11adc8f589ded3ce11bc3e1061ad498259664fabc1f6615beb918", size = 9449976, upload-time = "2026-06-12T02:27:50.989Z" },
+ { url = "https://files.pythonhosted.org/packages/82/34/bdd77418adb2178a1d59f044bd67bfebb115896e91b840b8a197eb3f4f4e/matplotlib-3.11.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:0515d495124be3124340e59f164d901ed4484e2246a5b74cfa483cac3b80bd97", size = 9279307, upload-time = "2026-06-12T02:27:53.247Z" },
+ { url = "https://files.pythonhosted.org/packages/94/95/7f522393c88313336b20d70fc849555757b2e5febc22b83b3a3f0fd4bce9/matplotlib-3.11.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:be5f93a1d21981bfb802ded0d77a0caa92d4342a47d45754fac77e314a506344", size = 10031353, upload-time = "2026-06-12T02:27:55.215Z" },
+ { url = "https://files.pythonhosted.org/packages/87/ce/8f25a0e3186aefd61913e7467d1b999465bcd0d0c03ac695c1b26ca559b7/matplotlib-3.11.0-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:41635d7909d19e52e924a521dde6d8f670b0f53ab1d0e8c331fa831554f681d1", size = 10839232, upload-time = "2026-06-12T02:27:57.746Z" },
+ { url = "https://files.pythonhosted.org/packages/85/c2/db15da2bbdf9e3ca66df7db8e2c33a1dfed67be24a24d2c878efaaff01d6/matplotlib-3.11.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:94f5000f67ca9faa300863ea17f8bce9175cb67b88bec4bc7780502d53dd7c9e", size = 10923899, upload-time = "2026-06-12T02:28:00.223Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/2f/a58a4443a4d052a4ea77557478336aefc26c7981f6408d37adba763aa758/matplotlib-3.11.0-cp312-cp312-win_amd64.whl", hash = "sha256:ac6f1ef39f3d0f9e2463303013094992cdbe0f85f43bc54155bc472b2042768e", size = 9329528, upload-time = "2026-06-12T02:28:02.27Z" },
+ { url = "https://files.pythonhosted.org/packages/61/0f/4b669589d47733b97ab9df4b58d6fc1e68acb5ea42a928dc7cbdd6bf5871/matplotlib-3.11.0-cp312-cp312-win_arm64.whl", hash = "sha256:9dd11fb612ce7bc60b1de5b4fc87ff959d22317b5de42aabf392f66f97af22eb", size = 9003413, upload-time = "2026-06-12T02:28:04.49Z" },
+ { url = "https://files.pythonhosted.org/packages/55/41/aa47f156b061d14c98b906f76c428507397708ec63ff94f410ae1752b426/matplotlib-3.11.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:6ce3b839b34ae1f430b4616893a2945a2999debaa7e94e7e29a2a8bbf286f7b5", size = 9450532, upload-time = "2026-06-12T02:28:06.769Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/4f/5a9eb0375e81413953febf8af7b012a6b6357f53438a15c4f5ad86c6bbb5/matplotlib-3.11.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:373db8f91214e8ccaf35ac833cc1dd59dd961e148bbd55dd027141591dde1313", size = 9279760, upload-time = "2026-06-12T02:28:09.152Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/c0/1117d53077e3ac3152503a84e9cf7a5c239576805ee71276e80c2aaa7471/matplotlib-3.11.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:be152b7570324dc8d01574cc9474dd2d803237acf528bcbb5b211fa347461a09", size = 10031623, upload-time = "2026-06-12T02:28:11.26Z" },
+ { url = "https://files.pythonhosted.org/packages/92/7e/e937138daffad65b71bf831a377809dcbc830fb4f31a31e067dc1faa2575/matplotlib-3.11.0-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:126f256df600652d7e4b394cf3164ff75210a00038f287c95a012a6f58d0e83f", size = 10839372, upload-time = "2026-06-12T02:28:14.102Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/c2/438ecc197ffb8023b6b9922915542f2172f5fd45b76703b0b4fc47322243/matplotlib-3.11.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:03acfeddf87b0dddb11b081ef7740ad445a3ca8bcb6b8e3011b08f2cf802b75c", size = 10924099, upload-time = "2026-06-12T02:28:16.383Z" },
+ { url = "https://files.pythonhosted.org/packages/40/2e/395883da416f378b3ed2c9f3e843ac477eae1ce731b671b79adaa6f0bacd/matplotlib-3.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:ab3722f04f3ff34c23b5012c5873d2894174e06c3822fcdac3610965a5ac7d06", size = 9329727, upload-time = "2026-06-12T02:28:18.581Z" },
+ { url = "https://files.pythonhosted.org/packages/61/82/2c388956abf8bf392dfb5b8917c502f1082df6a941b781ab8c8e5ba2474b/matplotlib-3.11.0-cp313-cp313-win_arm64.whl", hash = "sha256:c945824670fb8915b4ac879e5e61f3c58e0913022f70a0de4c082b17372f8771", size = 9003506, upload-time = "2026-06-12T02:28:20.474Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/c1/34454baa44da7975ada82e9aea37105ec47059514dc967d3be14426ba8dc/matplotlib-3.11.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:3489c3dc487669b4a980bc3068f87856de7a1564248d3f6c629efb2a58b03f24", size = 9499838, upload-time = "2026-06-12T02:28:22.713Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/c3/98fe79a398cf232219f090163a7fa7e6766e9f2e0ad26df54d6f8934d8ee/matplotlib-3.11.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:6a98f5476ce784a50ce09998f4ae1e6a9f25043cef8a480c98949902eda74620", size = 9332298, upload-time = "2026-06-12T02:28:24.796Z" },
+ { url = "https://files.pythonhosted.org/packages/95/e4/b4b7c33151e74e5c802f3cde1ba807ebfc38401e329b44e215a5888dd76d/matplotlib-3.11.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:565af866fd63e4bd3f987d580afe27c44c2552a3b3305f4ecbb85133601ea6f3", size = 10045491, upload-time = "2026-06-12T02:28:27.141Z" },
+ { url = "https://files.pythonhosted.org/packages/71/28/394548efd68354110c1a1be11fe6b6e559e06d1a23da35908a0e316c55a9/matplotlib-3.11.0-cp313-cp313t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e6b3e64dea5062c570f04358e2711859f3531b459f29516274fbad889079e4f3", size = 10857059, upload-time = "2026-06-12T02:28:29.222Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/44/e7922e6e2a4d63bdfbc9dc4a53e3850ab438d46cf42e6779bb15ec92c948/matplotlib-3.11.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:942b37c5db1899610bd1543ce8e13e4ecff9a4633e7f63bb6aa9205d2644ebd1", size = 10939576, upload-time = "2026-06-12T02:28:31.66Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/be/b1ca96003a441d619b727fee21d671fdff7a5ce2f1bb797b2521aa2f679a/matplotlib-3.11.0-cp313-cp313t-win_amd64.whl", hash = "sha256:c08e649a6313e1291e713623b97a38e5bb4aa580b2a100a94a3309bc6b9c8eb3", size = 9379519, upload-time = "2026-06-12T02:28:33.888Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/72/4bf3b91821c34596dd6a7bdac5836d94f744144c8208939ef49d8ec43f7e/matplotlib-3.11.0-cp313-cp313t-win_arm64.whl", hash = "sha256:2746cd2c113742ff6ce37a864c5ac5fd7aa644568f445e66166e457ac78e40e0", size = 9055456, upload-time = "2026-06-12T02:28:35.878Z" },
+ { url = "https://files.pythonhosted.org/packages/57/52/a94102ac99eb78e2fe9b826674f9ef9ee23327110ea6ab4776c1b4eb6209/matplotlib-3.11.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:3338e3e3de128cf50d0d2fb92a122815daf9c755bd882a474343c05f8fd7ec79", size = 9452137, upload-time = "2026-06-12T02:28:37.93Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/03/b8cdb625a21f710dfa11bbca1f48fb4057d2c0286975f8b415bf80942c99/matplotlib-3.11.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:25c2e5455efd8d99f41fb79871a31feb7d301569642e332ec58d72cfe9282bc3", size = 9281514, upload-time = "2026-06-12T02:28:40.028Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/2d/4e1240ea82ee197dfb3851e71f71c87eeeb975f1753b56a0588e4e80739a/matplotlib-3.11.0-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d9695457a467ff86d23f35037a43deb6f1134dd6d3e2ac8ce1e2087cff09ffb9", size = 10843005, upload-time = "2026-06-12T02:28:42.39Z" },
+ { url = "https://files.pythonhosted.org/packages/29/dc/6377ecfaa5fef79430f74a1a16638b4e2aa30d4692bae2c19f9d76fe3b01/matplotlib-3.11.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:19c16c61dea63b3582918503e6b294193961261d9daa806d4ae2151f1ad05430", size = 11127459, upload-time = "2026-06-12T02:28:44.483Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/41/795c405aa7560443a3b01309424cde4a1113b85c90b8a63417444a749617/matplotlib-3.11.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2d72ea8b7924f3cb955e61518d21e43b3df1e6c8a793b480a0c1214f185d30ba", size = 10925160, upload-time = "2026-06-12T02:28:46.564Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/f7/3a9e6389a7cfaeff76c56e40c2dabcb13110e21e82f837228c834ebe748c/matplotlib-3.11.0-cp314-cp314-win_amd64.whl", hash = "sha256:1c02da0a629dfa9debf52725ea06866b74c1fb70a895bae05e4493d34074f9f2", size = 9485186, upload-time = "2026-06-12T02:28:49.344Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/c0/396478ee7cf2091d182db8b4a8695f6a37f1ddb978989cf9dbb84cd5c123/matplotlib-3.11.0-cp314-cp314-win_arm64.whl", hash = "sha256:aa55d73b3117d4b07f959cd9eb6f69b375d8df3414139c479388e551aa5d999d", size = 9160349, upload-time = "2026-06-12T02:28:51.382Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/6f/1c3bd51bb2b34eaacdcf3c3d859dbb357f952fc8020c617dc118ad7c9e38/matplotlib-3.11.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:a9d8c6e7cd2f0ddf11d8d92e520dd1d9d2abb0cf6ac8831e338666c81e905847", size = 9500921, upload-time = "2026-06-12T02:28:53.443Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/0d/4d861d0121840cb1a3fd4a10deb211efd6fccd481ed23e553f31f4f4da4a/matplotlib-3.11.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:be050fcf32f729eda99f7f75a80bf67612ce16ab9ac1c23a387dcaede95cb70e", size = 9332190, upload-time = "2026-06-12T02:28:55.623Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/cb/22f6bc35711a0b5639a784e74e653e77c86210bd4304449dd399a482f74e/matplotlib-3.11.0-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dfabef0230d0697aa0d717385194dd41162e00207a68bf4abf94c2bf4c27dca0", size = 10854181, upload-time = "2026-06-12T02:28:57.856Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/7e/9a9eaca731a2939589da520f0ebe8fd8753d0f51fca98c7d20af6dbe261a/matplotlib-3.11.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1644db30e759199443493ac5e5caec24fdb775a8f6123021f85ba47c4133c3cb", size = 11137715, upload-time = "2026-06-12T02:29:00.555Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/f9/9b030b6088354acb0296871bb624b25befc1c42509d3c6cd17420c83a5b8/matplotlib-3.11.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:15b0d160079cb10699a0e98b5989c70677b2df7cacdc62af67c30f2facec46d9", size = 10939427, upload-time = "2026-06-12T02:29:02.527Z" },
+ { url = "https://files.pythonhosted.org/packages/59/94/6b273eaee4ee250863567d100865da61a5c1527fa67f527b7ed22e0dd29c/matplotlib-3.11.0-cp314-cp314t-win_amd64.whl", hash = "sha256:446307e6b04b57b1f1239e228a1ec2af0d589a1008cebc3dfa3f5441d095cfb6", size = 9535809, upload-time = "2026-06-12T02:29:04.994Z" },
+ { url = "https://files.pythonhosted.org/packages/60/95/1d36bddf2b7e2692c1540e78a6e5bc88bc1496b137e3e35a611f91b65ac3/matplotlib-3.11.0-cp314-cp314t-win_arm64.whl", hash = "sha256:652fb5696271d4c50f196d22a5ff4f8e4444c74f847423570d7dc0aa2bbd0159", size = 9209226, upload-time = "2026-06-12T02:29:07.033Z" },
+]
+
+[[package]]
+name = "matplotlib-inline"
+version = "0.2.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/bd/c0/9f7c9a46090390368a4d7bcb76bb87a4a36c421e4c0792cdb53486ffac7a/matplotlib_inline-0.2.2.tar.gz", hash = "sha256:72f3fe8fce36b70d4a5b612f899090cd0401deddc4ea90e1572b9f4bfb058c79", size = 8150, upload-time = "2026-05-08T17:33:33.49Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/41/09/5b161152e2d90f7b87f781c2e1267494aef9c32498df793f73ad0a0a494a/matplotlib_inline-0.2.2-py3-none-any.whl", hash = "sha256:3c821cf1c209f59fb2d2d64abbf5b23b67bcb2210d663f9918dd851c6da1fcf6", size = 9534, upload-time = "2026-05-08T17:33:32.055Z" },
+]
+
+[[package]]
+name = "mdit-py-plugins"
+version = "0.6.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "markdown-it-py" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/59/fc/f8d0863f8862f25602c0404d75568e89fb6b4109804645e5cdfb1be5cf56/mdit_py_plugins-0.6.1.tar.gz", hash = "sha256:a2bca0f039f39dbd35fb74ae1b5f998608c437463371f0ff7f49a19a17a114d0", size = 56114, upload-time = "2026-05-13T09:03:38.91Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a5/69/6da5581c6a7fede7dc261bf4e67d6adca4196f176b43288b55b3db395b6e/mdit_py_plugins-0.6.1-py3-none-any.whl", hash = "sha256:214c82fb2ac524472ab6a5bcab1de80f73b50443e187f401bfd77efbc7c6481d", size = 66663, upload-time = "2026-05-13T09:03:37.76Z" },
+]
+
+[[package]]
+name = "mdurl"
+version = "0.1.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" },
+]
+
+[[package]]
+name = "modal"
+version = "1.5.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "aiohttp" },
+ { name = "cbor2" },
+ { name = "certifi" },
+ { name = "click" },
+ { name = "grpclib" },
+ { name = "protobuf" },
+ { name = "rich" },
+ { name = "synchronicity" },
+ { name = "toml" },
+ { name = "types-certifi" },
+ { name = "types-toml" },
+ { name = "typing-extensions" },
+ { name = "watchfiles" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b8/e6/44f609084472d16a3acf7748c184e23d3009944eed5860e855aaafc61065/modal-1.5.2.tar.gz", hash = "sha256:ac0d2983ecccacb80c54385906404d75f37a3751f1374c750974cd7853c0008a", size = 818040, upload-time = "2026-07-10T18:12:55.164Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e3/fe/7e284d889eb3ea2c307e48828eccdd9a984beb2e096bce5577ef796592e9/modal-1.5.2-py3-none-any.whl", hash = "sha256:7508a44f8742c6f26be4d28da71f46092ef3bcc07d0f27cbd0ea8c9d6d9fc873", size = 932258, upload-time = "2026-07-10T18:12:52.219Z" },
+]
+
+[[package]]
+name = "multidict"
+version = "6.7.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1a/c2/c2d94cbe6ac1753f3fc980da97b3d930efe1da3af3c9f5125354436c073d/multidict-6.7.1.tar.gz", hash = "sha256:ec6652a1bee61c53a3e5776b6049172c53b6aaba34f18c9ad04f82712bac623d", size = 102010, upload-time = "2026-01-26T02:46:45.979Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8d/9c/f20e0e2cf80e4b2e4b1c365bf5fe104ee633c751a724246262db8f1a0b13/multidict-6.7.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:a90f75c956e32891a4eda3639ce6dd86e87105271f43d43442a3aedf3cddf172", size = 76893, upload-time = "2026-01-26T02:43:52.754Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/cf/18ef143a81610136d3da8193da9d80bfe1cb548a1e2d1c775f26b23d024a/multidict-6.7.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3fccb473e87eaa1382689053e4a4618e7ba7b9b9b8d6adf2027ee474597128cd", size = 45456, upload-time = "2026-01-26T02:43:53.893Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/65/1caac9d4cd32e8433908683446eebc953e82d22b03d10d41a5f0fefe991b/multidict-6.7.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b0fa96985700739c4c7853a43c0b3e169360d6855780021bfc6d0f1ce7c123e7", size = 43872, upload-time = "2026-01-26T02:43:55.041Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/3b/d6bd75dc4f3ff7c73766e04e705b00ed6dbbaccf670d9e05a12b006f5a21/multidict-6.7.1-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:cb2a55f408c3043e42b40cc8eecd575afa27b7e0b956dfb190de0f8499a57a53", size = 251018, upload-time = "2026-01-26T02:43:56.198Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/80/c959c5933adedb9ac15152e4067c702a808ea183a8b64cf8f31af8ad3155/multidict-6.7.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eb0ce7b2a32d09892b3dd6cc44877a0d02a33241fafca5f25c8b6b62374f8b75", size = 258883, upload-time = "2026-01-26T02:43:57.499Z" },
+ { url = "https://files.pythonhosted.org/packages/86/85/7ed40adafea3d4f1c8b916e3b5cc3a8e07dfcdcb9cd72800f4ed3ca1b387/multidict-6.7.1-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c3a32d23520ee37bf327d1e1a656fec76a2edd5c038bf43eddfa0572ec49c60b", size = 242413, upload-time = "2026-01-26T02:43:58.755Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/57/b8565ff533e48595503c785f8361ff9a4fde4d67de25c207cd0ba3befd03/multidict-6.7.1-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9c90fed18bffc0189ba814749fdcc102b536e83a9f738a9003e569acd540a733", size = 268404, upload-time = "2026-01-26T02:44:00.216Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/50/9810c5c29350f7258180dfdcb2e52783a0632862eb334c4896ac717cebcb/multidict-6.7.1-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:da62917e6076f512daccfbbde27f46fed1c98fee202f0559adec8ee0de67f71a", size = 269456, upload-time = "2026-01-26T02:44:02.202Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/8d/5e5be3ced1d12966fefb5c4ea3b2a5b480afcea36406559442c6e31d4a48/multidict-6.7.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bfde23ef6ed9db7eaee6c37dcec08524cb43903c60b285b172b6c094711b3961", size = 256322, upload-time = "2026-01-26T02:44:03.56Z" },
+ { url = "https://files.pythonhosted.org/packages/31/6e/d8a26d81ac166a5592782d208dd90dfdc0a7a218adaa52b45a672b46c122/multidict-6.7.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3758692429e4e32f1ba0df23219cd0b4fc0a52f476726fff9337d1a57676a582", size = 253955, upload-time = "2026-01-26T02:44:04.845Z" },
+ { url = "https://files.pythonhosted.org/packages/59/4c/7c672c8aad41534ba619bcd4ade7a0dc87ed6b8b5c06149b85d3dd03f0cd/multidict-6.7.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:398c1478926eca669f2fd6a5856b6de9c0acf23a2cb59a14c0ba5844fa38077e", size = 251254, upload-time = "2026-01-26T02:44:06.133Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/bd/84c24de512cbafbdbc39439f74e967f19570ce7924e3007174a29c348916/multidict-6.7.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:c102791b1c4f3ab36ce4101154549105a53dc828f016356b3e3bcae2e3a039d3", size = 252059, upload-time = "2026-01-26T02:44:07.518Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/ba/f5449385510825b73d01c2d4087bf6d2fccc20a2d42ac34df93191d3dd03/multidict-6.7.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:a088b62bd733e2ad12c50dad01b7d0166c30287c166e137433d3b410add807a6", size = 263588, upload-time = "2026-01-26T02:44:09.382Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/11/afc7c677f68f75c84a69fe37184f0f82fce13ce4b92f49f3db280b7e92b3/multidict-6.7.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:3d51ff4785d58d3f6c91bdbffcb5e1f7ddfda557727043aa20d20ec4f65e324a", size = 259642, upload-time = "2026-01-26T02:44:10.73Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/17/ebb9644da78c4ab36403739e0e6e0e30ebb135b9caf3440825001a0bddcb/multidict-6.7.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fc5907494fccf3e7d3f94f95c91d6336b092b5fc83811720fae5e2765890dfba", size = 251377, upload-time = "2026-01-26T02:44:12.042Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/a4/840f5b97339e27846c46307f2530a2805d9d537d8b8bd416af031cad7fa0/multidict-6.7.1-cp312-cp312-win32.whl", hash = "sha256:28ca5ce2fd9716631133d0e9a9b9a745ad7f60bac2bccafb56aa380fc0b6c511", size = 41887, upload-time = "2026-01-26T02:44:14.245Z" },
+ { url = "https://files.pythonhosted.org/packages/80/31/0b2517913687895f5904325c2069d6a3b78f66cc641a86a2baf75a05dcbb/multidict-6.7.1-cp312-cp312-win_amd64.whl", hash = "sha256:fcee94dfbd638784645b066074b338bc9cc155d4b4bffa4adce1615c5a426c19", size = 46053, upload-time = "2026-01-26T02:44:15.371Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/5b/aba28e4ee4006ae4c7df8d327d31025d760ffa992ea23812a601d226e682/multidict-6.7.1-cp312-cp312-win_arm64.whl", hash = "sha256:ba0a9fb644d0c1a2194cf7ffb043bd852cea63a57f66fbd33959f7dae18517bf", size = 43307, upload-time = "2026-01-26T02:44:16.852Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/22/929c141d6c0dba87d3e1d38fbdf1ba8baba86b7776469f2bc2d3227a1e67/multidict-6.7.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:2b41f5fed0ed563624f1c17630cb9941cf2309d4df00e494b551b5f3e3d67a23", size = 76174, upload-time = "2026-01-26T02:44:18.509Z" },
+ { url = "https://files.pythonhosted.org/packages/c7/75/bc704ae15fee974f8fccd871305e254754167dce5f9e42d88a2def741a1d/multidict-6.7.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:84e61e3af5463c19b67ced91f6c634effb89ef8bfc5ca0267f954451ed4bb6a2", size = 45116, upload-time = "2026-01-26T02:44:19.745Z" },
+ { url = "https://files.pythonhosted.org/packages/79/76/55cd7186f498ed080a18440c9013011eb548f77ae1b297206d030eb1180a/multidict-6.7.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:935434b9853c7c112eee7ac891bc4cb86455aa631269ae35442cb316790c1445", size = 43524, upload-time = "2026-01-26T02:44:21.571Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/3c/414842ef8d5a1628d68edee29ba0e5bcf235dbfb3ccd3ea303a7fe8c72ff/multidict-6.7.1-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:432feb25a1cb67fe82a9680b4d65fb542e4635cb3166cd9c01560651ad60f177", size = 249368, upload-time = "2026-01-26T02:44:22.803Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/32/befed7f74c458b4a525e60519fe8d87eef72bb1e99924fa2b0f9d97a221e/multidict-6.7.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e82d14e3c948952a1a85503817e038cba5905a3352de76b9a465075d072fba23", size = 256952, upload-time = "2026-01-26T02:44:24.306Z" },
+ { url = "https://files.pythonhosted.org/packages/03/d6/c878a44ba877f366630c860fdf74bfb203c33778f12b6ac274936853c451/multidict-6.7.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:4cfb48c6ea66c83bcaaf7e4dfa7ec1b6bbcf751b7db85a328902796dfde4c060", size = 240317, upload-time = "2026-01-26T02:44:25.772Z" },
+ { url = "https://files.pythonhosted.org/packages/68/49/57421b4d7ad2e9e60e25922b08ceb37e077b90444bde6ead629095327a6f/multidict-6.7.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:1d540e51b7e8e170174555edecddbd5538105443754539193e3e1061864d444d", size = 267132, upload-time = "2026-01-26T02:44:27.648Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/fe/ec0edd52ddbcea2a2e89e174f0206444a61440b40f39704e64dc807a70bd/multidict-6.7.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:273d23f4b40f3dce4d6c8a821c741a86dec62cded82e1175ba3d99be128147ed", size = 268140, upload-time = "2026-01-26T02:44:29.588Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/73/6e1b01cbeb458807aa0831742232dbdd1fa92bfa33f52a3f176b4ff3dc11/multidict-6.7.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d624335fd4fa1c08a53f8b4be7676ebde19cd092b3895c421045ca87895b429", size = 254277, upload-time = "2026-01-26T02:44:30.902Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/b2/5fb8c124d7561a4974c342bc8c778b471ebbeb3cc17df696f034a7e9afe7/multidict-6.7.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:12fad252f8b267cc75b66e8fc51b3079604e8d43a75428ffe193cd9e2195dfd6", size = 252291, upload-time = "2026-01-26T02:44:32.31Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/96/51d4e4e06bcce92577fcd488e22600bd38e4fd59c20cb49434d054903bd2/multidict-6.7.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:03ede2a6ffbe8ef936b92cb4529f27f42be7f56afcdab5ab739cd5f27fb1cbf9", size = 250156, upload-time = "2026-01-26T02:44:33.734Z" },
+ { url = "https://files.pythonhosted.org/packages/db/6b/420e173eec5fba721a50e2a9f89eda89d9c98fded1124f8d5c675f7a0c0f/multidict-6.7.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:90efbcf47dbe33dcf643a1e400d67d59abeac5db07dc3f27d6bdeae497a2198c", size = 249742, upload-time = "2026-01-26T02:44:35.222Z" },
+ { url = "https://files.pythonhosted.org/packages/44/a3/ec5b5bd98f306bc2aa297b8c6f11a46714a56b1e6ef5ebda50a4f5d7c5fb/multidict-6.7.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:5c4b9bfc148f5a91be9244d6264c53035c8a0dcd2f51f1c3c6e30e30ebaa1c84", size = 262221, upload-time = "2026-01-26T02:44:36.604Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/f7/e8c0d0da0cd1e28d10e624604e1a36bcc3353aaebdfdc3a43c72bc683a12/multidict-6.7.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:401c5a650f3add2472d1d288c26deebc540f99e2fb83e9525007a74cd2116f1d", size = 258664, upload-time = "2026-01-26T02:44:38.008Z" },
+ { url = "https://files.pythonhosted.org/packages/52/da/151a44e8016dd33feed44f730bd856a66257c1ee7aed4f44b649fb7edeb3/multidict-6.7.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:97891f3b1b3ffbded884e2916cacf3c6fc87b66bb0dde46f7357404750559f33", size = 249490, upload-time = "2026-01-26T02:44:39.386Z" },
+ { url = "https://files.pythonhosted.org/packages/87/af/a3b86bf9630b732897f6fc3f4c4714b90aa4361983ccbdcd6c0339b21b0c/multidict-6.7.1-cp313-cp313-win32.whl", hash = "sha256:e1c5988359516095535c4301af38d8a8838534158f649c05dd1050222321bcb3", size = 41695, upload-time = "2026-01-26T02:44:41.318Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/35/e994121b0e90e46134673422dd564623f93304614f5d11886b1b3e06f503/multidict-6.7.1-cp313-cp313-win_amd64.whl", hash = "sha256:960c83bf01a95b12b08fd54324a4eb1d5b52c88932b5cba5d6e712bb3ed12eb5", size = 45884, upload-time = "2026-01-26T02:44:42.488Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/61/42d3e5dbf661242a69c97ea363f2d7b46c567da8eadef8890022be6e2ab0/multidict-6.7.1-cp313-cp313-win_arm64.whl", hash = "sha256:563fe25c678aaba333d5399408f5ec3c383ca5b663e7f774dd179a520b8144df", size = 43122, upload-time = "2026-01-26T02:44:43.664Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/b3/e6b21c6c4f314bb956016b0b3ef2162590a529b84cb831c257519e7fde44/multidict-6.7.1-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:c76c4bec1538375dad9d452d246ca5368ad6e1c9039dadcf007ae59c70619ea1", size = 83175, upload-time = "2026-01-26T02:44:44.894Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/76/23ecd2abfe0957b234f6c960f4ade497f55f2c16aeb684d4ecdbf1c95791/multidict-6.7.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:57b46b24b5d5ebcc978da4ec23a819a9402b4228b8a90d9c656422b4bdd8a963", size = 48460, upload-time = "2026-01-26T02:44:46.106Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/57/a0ed92b23f3a042c36bc4227b72b97eca803f5f1801c1ab77c8a212d455e/multidict-6.7.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e954b24433c768ce78ab7929e84ccf3422e46deb45a4dc9f93438f8217fa2d34", size = 46930, upload-time = "2026-01-26T02:44:47.278Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/66/02ec7ace29162e447f6382c495dc95826bf931d3818799bbef11e8f7df1a/multidict-6.7.1-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:3bd231490fa7217cc832528e1cd8752a96f0125ddd2b5749390f7c3ec8721b65", size = 242582, upload-time = "2026-01-26T02:44:48.604Z" },
+ { url = "https://files.pythonhosted.org/packages/58/18/64f5a795e7677670e872673aca234162514696274597b3708b2c0d276cce/multidict-6.7.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:253282d70d67885a15c8a7716f3a73edf2d635793ceda8173b9ecc21f2fb8292", size = 250031, upload-time = "2026-01-26T02:44:50.544Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/ed/e192291dbbe51a8290c5686f482084d31bcd9d09af24f63358c3d42fd284/multidict-6.7.1-cp313-cp313t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:0b4c48648d7649c9335cf1927a8b87fa692de3dcb15faa676c6a6f1f1aabda43", size = 228596, upload-time = "2026-01-26T02:44:51.951Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/7e/3562a15a60cf747397e7f2180b0a11dc0c38d9175a650e75fa1b4d325e15/multidict-6.7.1-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:98bc624954ec4d2c7cb074b8eefc2b5d0ce7d482e410df446414355d158fe4ca", size = 257492, upload-time = "2026-01-26T02:44:53.902Z" },
+ { url = "https://files.pythonhosted.org/packages/24/02/7d0f9eae92b5249bb50ac1595b295f10e263dd0078ebb55115c31e0eaccd/multidict-6.7.1-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:1b99af4d9eec0b49927b4402bcbb58dea89d3e0db8806a4086117019939ad3dd", size = 255899, upload-time = "2026-01-26T02:44:55.316Z" },
+ { url = "https://files.pythonhosted.org/packages/00/e3/9b60ed9e23e64c73a5cde95269ef1330678e9c6e34dd4eb6b431b85b5a10/multidict-6.7.1-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6aac4f16b472d5b7dc6f66a0d49dd57b0e0902090be16594dc9ebfd3d17c47e7", size = 247970, upload-time = "2026-01-26T02:44:56.783Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/06/538e58a63ed5cfb0bd4517e346b91da32fde409d839720f664e9a4ae4f9d/multidict-6.7.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:21f830fe223215dffd51f538e78c172ed7c7f60c9b96a2bf05c4848ad49921c3", size = 245060, upload-time = "2026-01-26T02:44:58.195Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/2f/d743a3045a97c895d401e9bd29aaa09b94f5cbdf1bd561609e5a6c431c70/multidict-6.7.1-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:f5dd81c45b05518b9aa4da4aa74e1c93d715efa234fd3e8a179df611cc85e5f4", size = 235888, upload-time = "2026-01-26T02:44:59.57Z" },
+ { url = "https://files.pythonhosted.org/packages/38/83/5a325cac191ab28b63c52f14f1131f3b0a55ba3b9aa65a6d0bf2a9b921a0/multidict-6.7.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:eb304767bca2bb92fb9c5bd33cedc95baee5bb5f6c88e63706533a1c06ad08c8", size = 243554, upload-time = "2026-01-26T02:45:01.054Z" },
+ { url = "https://files.pythonhosted.org/packages/20/1f/9d2327086bd15da2725ef6aae624208e2ef828ed99892b17f60c344e57ed/multidict-6.7.1-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:c9035dde0f916702850ef66460bc4239d89d08df4d02023a5926e7446724212c", size = 252341, upload-time = "2026-01-26T02:45:02.484Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/2c/2a1aa0280cf579d0f6eed8ee5211c4f1730bd7e06c636ba2ee6aafda302e/multidict-6.7.1-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:af959b9beeb66c822380f222f0e0a1889331597e81f1ded7f374f3ecb0fd6c52", size = 246391, upload-time = "2026-01-26T02:45:03.862Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/03/7ca022ffc36c5a3f6e03b179a5ceb829be9da5783e6fe395f347c0794680/multidict-6.7.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:41f2952231456154ee479651491e94118229844dd7226541788be783be2b5108", size = 243422, upload-time = "2026-01-26T02:45:05.296Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/1d/b31650eab6c5778aceed46ba735bd97f7c7d2f54b319fa916c0f96e7805b/multidict-6.7.1-cp313-cp313t-win32.whl", hash = "sha256:df9f19c28adcb40b6aae30bbaa1478c389efd50c28d541d76760199fc1037c32", size = 47770, upload-time = "2026-01-26T02:45:06.754Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/5b/2d2d1d522e51285bd61b1e20df8f47ae1a9d80839db0b24ea783b3832832/multidict-6.7.1-cp313-cp313t-win_amd64.whl", hash = "sha256:d54ecf9f301853f2c5e802da559604b3e95bb7a3b01a9c295c6ee591b9882de8", size = 53109, upload-time = "2026-01-26T02:45:08.044Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/a3/cc409ba012c83ca024a308516703cf339bdc4b696195644a7215a5164a24/multidict-6.7.1-cp313-cp313t-win_arm64.whl", hash = "sha256:5a37ca18e360377cfda1d62f5f382ff41f2b8c4ccb329ed974cc2e1643440118", size = 45573, upload-time = "2026-01-26T02:45:09.349Z" },
+ { url = "https://files.pythonhosted.org/packages/91/cc/db74228a8be41884a567e88a62fd589a913708fcf180d029898c17a9a371/multidict-6.7.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:8f333ec9c5eb1b7105e3b84b53141e66ca05a19a605368c55450b6ba208cb9ee", size = 75190, upload-time = "2026-01-26T02:45:10.651Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/22/492f2246bb5b534abd44804292e81eeaf835388901f0c574bac4eeec73c5/multidict-6.7.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:a407f13c188f804c759fc6a9f88286a565c242a76b27626594c133b82883b5c2", size = 44486, upload-time = "2026-01-26T02:45:11.938Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/4f/733c48f270565d78b4544f2baddc2fb2a245e5a8640254b12c36ac7ac68e/multidict-6.7.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0e161ddf326db5577c3a4cc2d8648f81456e8a20d40415541587a71620d7a7d1", size = 43219, upload-time = "2026-01-26T02:45:14.346Z" },
+ { url = "https://files.pythonhosted.org/packages/24/bb/2c0c2287963f4259c85e8bcbba9182ced8d7fca65c780c38e99e61629d11/multidict-6.7.1-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:1e3a8bb24342a8201d178c3b4984c26ba81a577c80d4d525727427460a50c22d", size = 245132, upload-time = "2026-01-26T02:45:15.712Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/f9/44d4b3064c65079d2467888794dea218d1601898ac50222ab8a9a8094460/multidict-6.7.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:97231140a50f5d447d3164f994b86a0bed7cd016e2682f8650d6a9158e14fd31", size = 252420, upload-time = "2026-01-26T02:45:17.293Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/13/78f7275e73fa17b24c9a51b0bd9d73ba64bb32d0ed51b02a746eb876abe7/multidict-6.7.1-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:6b10359683bd8806a200fd2909e7c8ca3a7b24ec1d8132e483d58e791d881048", size = 233510, upload-time = "2026-01-26T02:45:19.356Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/25/8167187f62ae3cbd52da7893f58cb036b47ea3fb67138787c76800158982/multidict-6.7.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:283ddac99f7ac25a4acadbf004cb5ae34480bbeb063520f70ce397b281859362", size = 264094, upload-time = "2026-01-26T02:45:20.834Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/e7/69a3a83b7b030cf283fb06ce074a05a02322359783424d7edf0f15fe5022/multidict-6.7.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:538cec1e18c067d0e6103aa9a74f9e832904c957adc260e61cd9d8cf0c3b3d37", size = 260786, upload-time = "2026-01-26T02:45:22.818Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/3b/8ec5074bcfc450fe84273713b4b0a0dd47c0249358f5d82eb8104ffe2520/multidict-6.7.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7eee46ccb30ff48a1e35bb818cc90846c6be2b68240e42a78599166722cea709", size = 248483, upload-time = "2026-01-26T02:45:24.368Z" },
+ { url = "https://files.pythonhosted.org/packages/48/5a/d5a99e3acbca0e29c5d9cba8f92ceb15dce78bab963b308ae692981e3a5d/multidict-6.7.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fa263a02f4f2dd2d11a7b1bb4362aa7cb1049f84a9235d31adf63f30143469a0", size = 248403, upload-time = "2026-01-26T02:45:25.982Z" },
+ { url = "https://files.pythonhosted.org/packages/35/48/e58cd31f6c7d5102f2a4bf89f96b9cf7e00b6c6f3d04ecc44417c00a5a3c/multidict-6.7.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:2e1425e2f99ec5bd36c15a01b690a1a2456209c5deed58f95469ffb46039ccbb", size = 240315, upload-time = "2026-01-26T02:45:27.487Z" },
+ { url = "https://files.pythonhosted.org/packages/94/33/1cd210229559cb90b6786c30676bb0c58249ff42f942765f88793b41fdce/multidict-6.7.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:497394b3239fc6f0e13a78a3e1b61296e72bf1c5f94b4c4eb80b265c37a131cd", size = 245528, upload-time = "2026-01-26T02:45:28.991Z" },
+ { url = "https://files.pythonhosted.org/packages/64/f2/6e1107d226278c876c783056b7db43d800bb64c6131cec9c8dfb6903698e/multidict-6.7.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:233b398c29d3f1b9676b4b6f75c518a06fcb2ea0b925119fb2c1bc35c05e1601", size = 258784, upload-time = "2026-01-26T02:45:30.503Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/c1/11f664f14d525e4a1b5327a82d4de61a1db604ab34c6603bb3c2cc63ad34/multidict-6.7.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:93b1818e4a6e0930454f0f2af7dfce69307ca03cdcfb3739bf4d91241967b6c1", size = 251980, upload-time = "2026-01-26T02:45:32.603Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/9f/75a9ac888121d0c5bbd4ecf4eead45668b1766f6baabfb3b7f66a410e231/multidict-6.7.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f33dc2a3abe9249ea5d8360f969ec7f4142e7ac45ee7014d8f8d5acddf178b7b", size = 243602, upload-time = "2026-01-26T02:45:34.043Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/e7/50bf7b004cc8525d80dbbbedfdc7aed3e4c323810890be4413e589074032/multidict-6.7.1-cp314-cp314-win32.whl", hash = "sha256:3ab8b9d8b75aef9df299595d5388b14530839f6422333357af1339443cff777d", size = 40930, upload-time = "2026-01-26T02:45:36.278Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/bf/52f25716bbe93745595800f36fb17b73711f14da59ed0bb2eba141bc9f0f/multidict-6.7.1-cp314-cp314-win_amd64.whl", hash = "sha256:5e01429a929600e7dab7b166062d9bb54a5eed752384c7384c968c2afab8f50f", size = 45074, upload-time = "2026-01-26T02:45:37.546Z" },
+ { url = "https://files.pythonhosted.org/packages/97/ab/22803b03285fa3a525f48217963da3a65ae40f6a1b6f6cf2768879e208f9/multidict-6.7.1-cp314-cp314-win_arm64.whl", hash = "sha256:4885cb0e817aef5d00a2e8451d4665c1808378dc27c2705f1bf4ef8505c0d2e5", size = 42471, upload-time = "2026-01-26T02:45:38.889Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/6d/f9293baa6146ba9507e360ea0292b6422b016907c393e2f63fc40ab7b7b5/multidict-6.7.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:0458c978acd8e6ea53c81eefaddbbee9c6c5e591f41b3f5e8e194780fe026581", size = 82401, upload-time = "2026-01-26T02:45:40.254Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/68/53b5494738d83558d87c3c71a486504d8373421c3e0dbb6d0db48ad42ee0/multidict-6.7.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:c0abd12629b0af3cf590982c0b413b1e7395cd4ec026f30986818ab95bfaa94a", size = 48143, upload-time = "2026-01-26T02:45:41.635Z" },
+ { url = "https://files.pythonhosted.org/packages/37/e8/5284c53310dcdc99ce5d66563f6e5773531a9b9fe9ec7a615e9bc306b05f/multidict-6.7.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:14525a5f61d7d0c94b368a42cff4c9a4e7ba2d52e2672a7b23d84dc86fb02b0c", size = 46507, upload-time = "2026-01-26T02:45:42.99Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/fc/6800d0e5b3875568b4083ecf5f310dcf91d86d52573160834fb4bfcf5e4f/multidict-6.7.1-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:17307b22c217b4cf05033dabefe68255a534d637c6c9b0cc8382718f87be4262", size = 239358, upload-time = "2026-01-26T02:45:44.376Z" },
+ { url = "https://files.pythonhosted.org/packages/41/75/4ad0973179361cdf3a113905e6e088173198349131be2b390f9fa4da5fc6/multidict-6.7.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7a7e590ff876a3eaf1c02a4dfe0724b6e69a9e9de6d8f556816f29c496046e59", size = 246884, upload-time = "2026-01-26T02:45:47.167Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/9c/095bb28b5da139bd41fb9a5d5caff412584f377914bd8787c2aa98717130/multidict-6.7.1-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:5fa6a95dfee63893d80a34758cd0e0c118a30b8dcb46372bf75106c591b77889", size = 225878, upload-time = "2026-01-26T02:45:48.698Z" },
+ { url = "https://files.pythonhosted.org/packages/07/d0/c0a72000243756e8f5a277b6b514fa005f2c73d481b7d9e47cd4568aa2e4/multidict-6.7.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a0543217a6a017692aa6ae5cc39adb75e587af0f3a82288b1492eb73dd6cc2a4", size = 253542, upload-time = "2026-01-26T02:45:50.164Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/6b/f69da15289e384ecf2a68837ec8b5ad8c33e973aa18b266f50fe55f24b8c/multidict-6.7.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f99fe611c312b3c1c0ace793f92464d8cd263cc3b26b5721950d977b006b6c4d", size = 252403, upload-time = "2026-01-26T02:45:51.779Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/76/b9669547afa5a1a25cd93eaca91c0da1c095b06b6d2d8ec25b713588d3a1/multidict-6.7.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9004d8386d133b7e6135679424c91b0b854d2d164af6ea3f289f8f2761064609", size = 244889, upload-time = "2026-01-26T02:45:53.27Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/a9/a50d2669e506dad33cfc45b5d574a205587b7b8a5f426f2fbb2e90882588/multidict-6.7.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e628ef0e6859ffd8273c69412a2465c4be4a9517d07261b33334b5ec6f3c7489", size = 241982, upload-time = "2026-01-26T02:45:54.919Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/bb/1609558ad8b456b4827d3c5a5b775c93b87878fd3117ed3db3423dfbce1b/multidict-6.7.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:841189848ba629c3552035a6a7f5bf3b02eb304e9fea7492ca220a8eda6b0e5c", size = 232415, upload-time = "2026-01-26T02:45:56.981Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/59/6f61039d2aa9261871e03ab9dc058a550d240f25859b05b67fd70f80d4b3/multidict-6.7.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:ce1bbd7d780bb5a0da032e095c951f7014d6b0a205f8318308140f1a6aba159e", size = 240337, upload-time = "2026-01-26T02:45:58.698Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/29/fdc6a43c203890dc2ae9249971ecd0c41deaedfe00d25cb6564b2edd99eb/multidict-6.7.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:b26684587228afed0d50cf804cc71062cc9c1cdf55051c4c6345d372947b268c", size = 248788, upload-time = "2026-01-26T02:46:00.862Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/14/a153a06101323e4cf086ecee3faadba52ff71633d471f9685c42e3736163/multidict-6.7.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:9f9af11306994335398293f9958071019e3ab95e9a707dc1383a35613f6abcb9", size = 242842, upload-time = "2026-01-26T02:46:02.824Z" },
+ { url = "https://files.pythonhosted.org/packages/41/5f/604ae839e64a4a6efc80db94465348d3b328ee955e37acb24badbcd24d83/multidict-6.7.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:b4938326284c4f1224178a560987b6cf8b4d38458b113d9b8c1db1a836e640a2", size = 240237, upload-time = "2026-01-26T02:46:05.898Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/60/c3a5187bf66f6fb546ff4ab8fb5a077cbdd832d7b1908d4365c7f74a1917/multidict-6.7.1-cp314-cp314t-win32.whl", hash = "sha256:98655c737850c064a65e006a3df7c997cd3b220be4ec8fe26215760b9697d4d7", size = 48008, upload-time = "2026-01-26T02:46:07.468Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/f7/addf1087b860ac60e6f382240f64fb99f8bfb532bb06f7c542b83c29ca61/multidict-6.7.1-cp314-cp314t-win_amd64.whl", hash = "sha256:497bde6223c212ba11d462853cfa4f0ae6ef97465033e7dc9940cdb3ab5b48e5", size = 53542, upload-time = "2026-01-26T02:46:08.809Z" },
+ { url = "https://files.pythonhosted.org/packages/4c/81/4629d0aa32302ef7b2ec65c75a728cc5ff4fa410c50096174c1632e70b3e/multidict-6.7.1-cp314-cp314t-win_arm64.whl", hash = "sha256:2bbd113e0d4af5db41d5ebfe9ccaff89de2120578164f86a5d17d5a576d1e5b2", size = 44719, upload-time = "2026-01-26T02:46:11.146Z" },
+ { url = "https://files.pythonhosted.org/packages/81/08/7036c080d7117f28a4af526d794aab6a84463126db031b007717c1a6676e/multidict-6.7.1-py3-none-any.whl", hash = "sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56", size = 12319, upload-time = "2026-01-26T02:46:44.004Z" },
+]
+
+[[package]]
+name = "mypy"
+version = "2.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "ast-serialize" },
+ { name = "librt", marker = "platform_python_implementation != 'PyPy'" },
+ { name = "mypy-extensions" },
+ { name = "pathspec" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/82/15/cca9d88503549ed6fedeaa1d448cdddd542ee8a490232d732e278036fbf2/mypy-2.1.0.tar.gz", hash = "sha256:81e76ad12c2d804512e9b13240d1588316531bfba07558286078bfbce9613633", size = 3898359, upload-time = "2026-05-11T18:37:36.237Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/95/b1/55861beb5c339b44f9a2ba92df9e2cb1eeb4ae1eee674cdf7772c797778b/mypy-2.1.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:244358bf1c0da7722230bce60683d52e8e9fd030554926f15b747a84efb5b3af", size = 14874381, upload-time = "2026-05-11T18:37:31.784Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/b3/b7f770114b7d0ac92d0f76e8d93c2780844a70488a90e91821927850da86/mypy-2.1.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4ec7c57657493c7a75534df2751c8ae2cda383c16ecc55d2106c54476b1b16f6", size = 13665501, upload-time = "2026-05-11T18:34:23.063Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/f3/8ae2037967e2126689a0c11d99e2b707134a565191e92c60ca2572aec60a/mypy-2.1.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d8161b6ff4392410023224f0969d17db93e1e154bc3e4ba62598e720723ae211", size = 14045750, upload-time = "2026-05-11T18:31:48.151Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/32/615eb5911859e43d054941b0d0a7d06cfa2870eba86529cf385b052b111c/mypy-2.1.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bf03e12003084a67395184d3eb8cbd6a489dc3655b5664b28c210a9e2403ab0b", size = 15061630, upload-time = "2026-05-11T18:37:06.898Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/03/4eafbfff8bfab1b87082741eae6e6a624028c984e6708b73bce2a8570c9d/mypy-2.1.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:20509760fd791c51579d573153407d226385ec1f8bcce55d730b354f3336bc22", size = 15288831, upload-time = "2026-05-11T18:31:18.07Z" },
+ { url = "https://files.pythonhosted.org/packages/99/ee/919661478e5891a3c96e549c036e467e64563ab85995b10c53c8358e16a3/mypy-2.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:6753d0c1fdd6b1a23b9e4f283ce80b2153b724adcb2653b20b85a8a28ac6436b", size = 11135228, upload-time = "2026-05-11T18:34:31.23Z" },
+ { url = "https://files.pythonhosted.org/packages/24/0a/6a12b9782ca0831a553192f351679f4548abc9d19a7cc93bb7feb02084c7/mypy-2.1.0-cp312-cp312-win_arm64.whl", hash = "sha256:98ebb6589bb3b6d0c6f0c459d53ca55b8091fbc13d277c4041c885392e8195e8", size = 10040684, upload-time = "2026-05-11T18:36:48.199Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/dd/c7191469c777f07689c032a8f7326e393ea34c92d6d76eb7ce5ba57ea66d/mypy-2.1.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:35aac3bb114e03888f535d5eb51b8bafbb3266586b599da1940f9b1be3ec5bd5", size = 14852174, upload-time = "2026-05-11T18:31:38.929Z" },
+ { url = "https://files.pythonhosted.org/packages/55/8c/aed55408879043d72bb9135f4d0d19a02b886dd569631e113e3d2706cb8d/mypy-2.1.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8de55a8c861f2a49331f807be98d90caeceeef520bde13d43a160207f8af613e", size = 13651542, upload-time = "2026-05-11T18:36:04.636Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/8e/f371a824b1f1fa8ea6e3dbb8703d232977d572be2329554a3bc4d960302f/mypy-2.1.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5fdf2941a07434af755837d9880f7d7d25f1dacb1af9dcd4b9b66f2220a3024e", size = 14033929, upload-time = "2026-05-11T18:35:55.742Z" },
+ { url = "https://files.pythonhosted.org/packages/94/21/f54be870d6dd53a82c674407e0f8eed7174b05ec78d42e5abd7b42e84fd5/mypy-2.1.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e195b817c13f02352a9c124301f9f30f078405444679b6753c1b96b6eed37285", size = 15039200, upload-time = "2026-05-11T18:33:10.281Z" },
+ { url = "https://files.pythonhosted.org/packages/17/99/bf21748626a40ce59fd29a39386ab46afec88b7bd2f0fa6c3a97c995523f/mypy-2.1.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:5431d42af987ebd92ba2f71d45c85ed41d8e6ca9f5fd209a69f68f707d2469e5", size = 15272690, upload-time = "2026-05-11T18:32:07.205Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/d7/9e90d2cf47100bea550ed2bc7b0d4de3a62181d84d5e37da0003e8462637/mypy-2.1.0-cp313-cp313-win_amd64.whl", hash = "sha256:767fe8c66dc3e01e19e1737d4c38ebefead16125e1b8e58ad421903b376f5c65", size = 11147435, upload-time = "2026-05-11T18:33:56.477Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/46/e5c449e858798e35ffc90946282a27c62a77be743fe17480e4977374eb91/mypy-2.1.0-cp313-cp313-win_arm64.whl", hash = "sha256:ecfe70d43775ab99562ab128ce49854a362044c9f894961f68f898c23cb7429d", size = 10035052, upload-time = "2026-05-11T18:32:30.049Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/ca/b279a672e874aedd5498ae25f722dacc8aa86bbffb939b3f97cbb1cf6686/mypy-2.1.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:7354c5a7f69d9345c3d6e69921d57088eea3ddeeb6b20d34c1b3855b02c36ec2", size = 14848422, upload-time = "2026-05-11T18:35:45.984Z" },
+ { url = "https://files.pythonhosted.org/packages/27/e6/3efe56c631d959b9b4454e208b0ac4b7f4f58b404c89f8bec7b49efdfc21/mypy-2.1.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:49890d4f76ac9e06ec117f9e09f3174da70a620a0c300953d8595c926e80947f", size = 13677374, upload-time = "2026-05-11T18:36:57.188Z" },
+ { url = "https://files.pythonhosted.org/packages/84/7f/8107ea87a44fd1f1b59882442f033c9c3488c127201b1d1d15f1cbd6022e/mypy-2.1.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:761be68e023ef5d94678772396a8af1220030f80837a3afd8d0aef3b419666f4", size = 14055743, upload-time = "2026-05-11T18:35:18.361Z" },
+ { url = "https://files.pythonhosted.org/packages/51/4d/b6d34db183133b83761b9199a82d31557cdbb70a380d8c3b3438e11882a3/mypy-2.1.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c90345fc182dc363b891350457ec69c35140858538f38b4540845afcc32b1aef", size = 15020937, upload-time = "2026-05-11T18:34:59.618Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/d7/f08360c691d758acb02f45022c34d98b92892f4ea756644e1000d4b9f3d8/mypy-2.1.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:b84802e7b5a6daf1f5e15bc9fcd7ddae77be13981ffab037f1c67bb84d67d135", size = 15253371, upload-time = "2026-05-11T18:36:41.081Z" },
+ { url = "https://files.pythonhosted.org/packages/67/1b/09460a13719530a19bce27bd3bc8449e83569dd2ba7faf51c9c3c30c0b61/mypy-2.1.0-cp314-cp314-win_amd64.whl", hash = "sha256:022c771234936ceac541ebaf836fe9e2abeb3f5e09aff21588fe543ff006fe21", size = 11326429, upload-time = "2026-05-11T18:34:13.526Z" },
+ { url = "https://files.pythonhosted.org/packages/40/62/75dbf0f82f7b6680340efc614af29dd0b3c17b8a4f1cd09b8bd2fd6bc814/mypy-2.1.0-cp314-cp314-win_arm64.whl", hash = "sha256:498207db725cec88829a6a5c2fc771205fd043719ef98bc49aba8fb9fc4e6d57", size = 10218799, upload-time = "2026-05-11T18:32:23.491Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/66/caca04ed7d972fb6eb6dd1ccd6df1de5c38fae8c5b3dc1c4e8e0d85ee6b9/mypy-2.1.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:7d5e5cad0efeba72b93cd17490cc0d69c5ac9ca132994fe3fb0314808aeeb83e", size = 15923458, upload-time = "2026-05-11T18:35:28.64Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/52/2d90cbe49d014b13ed7ff337930c30bad35893fe38a1e4641e756bb62191/mypy-2.1.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ff715050c127d724fd260a2e666e7747fdd83511c0c47d449d98238970aef780", size = 14757697, upload-time = "2026-05-11T18:36:14.208Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/37/d98f4a14e081b238992d0ed96b6d39c7cc0148c9699eb71eaa68629665ea/mypy-2.1.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:82208da9e09414d520e912d3e462d454854bed0810b71540bb016dcbca7308fd", size = 15405638, upload-time = "2026-05-11T18:33:48.249Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/c2/15c46613b24a84fad2aea1248bf9619b99c2767ae9071fe224c179a0b7d4/mypy-2.1.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e79ebc1b904b84f0310dff7469655a9c36c7a68bddb37bdd42b67a332df61d08", size = 16215852, upload-time = "2026-05-11T18:32:50.296Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/90/9c16a57f482c76d25f6379762b56bbf65c711d8158cf271fb2802cfb0640/mypy-2.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:e583edc957cfb0deb142079162ae826f58449b116c1d442f2d91c69d9fced081", size = 16452695, upload-time = "2026-05-11T18:33:38.182Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/4c/215a4eeb63cacc5f17f516691ea7285d11e249802b942476bff15922a314/mypy-2.1.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b33b6cd332695bba180d55e717a79d3038e479a2c49cc5eb3d53603409b9a5d7", size = 12866622, upload-time = "2026-05-11T18:34:39.945Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/50/1043e1db5f455ffe4c9ab22747cd8ca2bc492b1e4f4e21b130a44ee2b217/mypy-2.1.0-cp314-cp314t-win_arm64.whl", hash = "sha256:4f910fe825376a7b66ef7ca8c98e5a149e8cd64c19ae71d84047a74ee060d4e6", size = 10610798, upload-time = "2026-05-11T18:36:31.444Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/2a/13ca1f292f6db1b98ff495ef3467736b331621c5917cad984b7043e7348d/mypy-2.1.0-py3-none-any.whl", hash = "sha256:a663814603a5c563fb87a4f96fb473eeb30d1f5a4885afcf44f9db000a366289", size = 2693302, upload-time = "2026-05-11T18:31:29.246Z" },
+]
+
+[[package]]
+name = "mypy-extensions"
+version = "1.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/a2/6e/371856a3fb9d31ca8dac321cda606860fa4548858c0cc45d9d1d4ca2628b/mypy_extensions-1.1.0.tar.gz", hash = "sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558", size = 6343, upload-time = "2025-04-22T14:54:24.164Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" },
+]
+
+[[package]]
+name = "myst-nb"
+version = "1.4.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "importlib-metadata" },
+ { name = "ipykernel" },
+ { name = "ipython" },
+ { name = "jupyter-cache" },
+ { name = "myst-parser" },
+ { name = "nbclient" },
+ { name = "nbformat" },
+ { name = "pyyaml" },
+ { name = "sphinx" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/bd/b4/ff1abeea67e8cfe0a8c033389f6d1d8b0bfecfd611befb5cbdeab884fce6/myst_nb-1.4.0.tar.gz", hash = "sha256:c145598de62446a6fd009773dd071a40d3b76106ace780de1abdfc6961f614c2", size = 82285, upload-time = "2026-03-02T21:14:56.95Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/94/93/0a378b48488879a1d925b42a804edfc6e0cd0ef854220f2dce738a46e7e9/myst_nb-1.4.0-py3-none-any.whl", hash = "sha256:0e2c86e7d3b82c3aa51383f82d6268f7714f3b772c23a796ab09538a8e68b4e4", size = 82555, upload-time = "2026-03-02T21:14:55.652Z" },
+]
+
+[[package]]
+name = "myst-parser"
+version = "5.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "docutils" },
+ { name = "jinja2" },
+ { name = "markdown-it-py" },
+ { name = "mdit-py-plugins" },
+ { name = "pyyaml" },
+ { name = "sphinx" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/21/dc/603751677fff302f34396e206b610f556a59d7fe58b9a2145f54e96b48e8/myst_parser-5.1.0.tar.gz", hash = "sha256:ab69322dc6719dcc7f296479dbb70181b66df6ed315064f92dbc85c0e1bf2f02", size = 101182, upload-time = "2026-05-13T09:38:19.361Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/09/dc/f3dfb7488b770f3f67e6545085bf2abea5172e88f57b8ad25ef860ca704c/myst_parser-5.1.0-py3-none-any.whl", hash = "sha256:9c91c52b3cdb4d94a6506e4fab4e2f296c7623a0da0dcbe6de1565c3dad67a8a", size = 85817, upload-time = "2026-05-13T09:38:17.904Z" },
+]
+
+[[package]]
+name = "narwhals"
+version = "2.22.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/62/3c/c4ef2164a71c1a63d7f1ae411c4082c5fa872405106db60a4b7114989ad7/narwhals-2.22.1.tar.gz", hash = "sha256:d62920805a0a43b7ff8b54b0c0d3142d796f8a9301836ada37e573d6a33cbcd9", size = 647493, upload-time = "2026-06-05T12:34:34.051Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/48/ca/36339329c4604adbcc99c899b7eb1ce1a555c499b6a6860757dc9bfed36d/narwhals-2.22.1-py3-none-any.whl", hash = "sha256:60567d774edf77db53906f89d9fbd164e66e56d66d388e1e6990f17ac33cfb53", size = 454815, upload-time = "2026-06-05T12:34:32.289Z" },
+]
+
+[[package]]
+name = "natsort"
+version = "8.4.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e2/a9/a0c57aee75f77794adaf35322f8b6404cbd0f89ad45c87197a937764b7d0/natsort-8.4.0.tar.gz", hash = "sha256:45312c4a0e5507593da193dedd04abb1469253b601ecaf63445ad80f0a1ea581", size = 76575, upload-time = "2023-06-20T04:17:19.925Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ef/82/7a9d0550484a62c6da82858ee9419f3dd1ccc9aa1c26a1e43da3ecd20b0d/natsort-8.4.0-py3-none-any.whl", hash = "sha256:4732914fb471f56b5cce04d7bae6f164a592c7712e1c85f9ef585e197299521c", size = 38268, upload-time = "2023-06-20T04:17:17.522Z" },
+]
+
+[[package]]
+name = "nbclient"
+version = "0.11.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jupyter-client" },
+ { name = "jupyter-core" },
+ { name = "nbformat" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/28/a5/b3bae4b590c0cbcada2c63a34f7580024e834a8ba213e949a2f906705787/nbclient-0.11.0.tar.gz", hash = "sha256:04a134a5b087f2c5887f228aca155db50169b8cd9334dee6942c8e927e56081a", size = 62535, upload-time = "2026-06-05T07:52:41.746Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/36/c9/94d73e5a01c5b926c3fa2496e97d7a8dc28ed5a77c0b2ed712f1a62e6694/nbclient-0.11.0-py3-none-any.whl", hash = "sha256:ef7fa0d59d6e1d41103933d8a445a18d5de860ca6b613b87b8574accdb3c2895", size = 25288, upload-time = "2026-06-05T07:52:40.115Z" },
+]
+
+[[package]]
+name = "nbformat"
+version = "5.10.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "fastjsonschema" },
+ { name = "jsonschema" },
+ { name = "jupyter-core" },
+ { name = "traitlets" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/6d/fd/91545e604bc3dad7dca9ed03284086039b294c6b3d75c0d2fa45f9e9caf3/nbformat-5.10.4.tar.gz", hash = "sha256:322168b14f937a5d11362988ecac2a4952d3d8e3a2cbeb2319584631226d5b3a", size = 142749, upload-time = "2024-04-04T11:20:37.371Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a9/82/0340caa499416c78e5d8f5f05947ae4bc3cba53c9f038ab6e9ed964e22f1/nbformat-5.10.4-py3-none-any.whl", hash = "sha256:3b48d6c8fbca4b299bf3982ea7db1af21580e4fec269ad087b9e81588891200b", size = 78454, upload-time = "2024-04-04T11:20:34.895Z" },
+]
+
+[[package]]
+name = "nest-asyncio2"
+version = "1.7.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b4/73/731debf26e27e0a0323d7bda270dc2f634b398e38f040a09da1f4351d0aa/nest_asyncio2-1.7.2.tar.gz", hash = "sha256:1921d70b92cc4612c374928d081552efb59b83d91b2b789d935c665fa01729a8", size = 14743, upload-time = "2026-02-13T00:34:04.386Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c5/3c/3179b85b0e1c3659f0369940200cd6d0fa900e6cefcc7ea0bc6dd0e29ffb/nest_asyncio2-1.7.2-py3-none-any.whl", hash = "sha256:f5dfa702f3f81f6a03857e9a19e2ba578c0946a4ad417b4c50a24d7ba641fe01", size = 7843, upload-time = "2026-02-13T00:34:02.691Z" },
+]
+
+[[package]]
+name = "networkx"
+version = "3.6.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/6a/51/63fe664f3908c97be9d2e4f1158eb633317598cfa6e1fc14af5383f17512/networkx-3.6.1.tar.gz", hash = "sha256:26b7c357accc0c8cde558ad486283728b65b6a95d85ee1cd66bafab4c8168509", size = 2517025, upload-time = "2025-12-08T17:02:39.908Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" },
+]
+
+[[package]]
+name = "nextprod"
+version = "1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/e6/1a/d1fff6df6d7b24e4009d50428eb56855c5b7dcb50fe46d803ee6726095dc/nextprod-1.0.tar.gz", hash = "sha256:1c0eff048c819da613bff18fd3786763af776b75b36516b9a6e1843c2d598538", size = 1956, upload-time = "2019-09-11T01:31:33.412Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a6/df/24a79fbc8dd7aeeadfec1bcee3405810c9e6a3c1bb216a152bdab58fde8f/nextprod-1.0-py3-none-any.whl", hash = "sha256:a4e1199486ead04c75bc31a0030157c22d9c6870936f4e3e16a872cb55a21624", size = 2289, upload-time = "2019-09-11T01:31:30.791Z" },
+]
+
+[[package]]
+name = "numba"
+version = "0.65.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "llvmlite" },
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f6/c5/db2ac3685833d626c0dcae6bd2330cd68433e1fd248d15f70998160d3ad7/numba-0.65.1.tar.gz", hash = "sha256:19357146c32fe9ed25059ab915e8465fb13951cf6b0aace3826b76886373ab23", size = 2765600, upload-time = "2026-04-24T02:02:56.551Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/57/bc/76f8f8c5cf9adee47fdb7bbb03be8900f76f902d451d7477cf12b845e1de/numba-0.65.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:ac3f1e77c352dd0ea9712732c2d8f9ca507717435eec5b5013bf138ac33c4a08", size = 2681371, upload-time = "2026-04-24T02:02:26.105Z" },
+ { url = "https://files.pythonhosted.org/packages/69/47/a415af0283e4db0398104c6d1c11c9861a98dc67a7aa442a7769ed5d6196/numba-0.65.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:52bc6f3ceb8fcaff9b2ae26b4c6b1e9fee39db8d355534c0fe4f39a901246b84", size = 3802467, upload-time = "2026-04-24T02:02:27.712Z" },
+ { url = "https://files.pythonhosted.org/packages/46/36/246f73ec99cfeab2f2cb2ce7d4218766cc36a2da418901223f4f4da9c813/numba-0.65.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:90ca10b3463bae0bd70589726fe3c77d01d6b5fc86bee54bcdf9fb6b47c28977", size = 3502628, upload-time = "2026-04-24T02:02:29.763Z" },
+ { url = "https://files.pythonhosted.org/packages/db/9e/3c679b2ee078425b9e99a91e44f8d132a6830d8ccce5227bc5e9181aeed8/numba-0.65.1-cp312-cp312-win_amd64.whl", hash = "sha256:5971c632be2a2351500431f46213821dba8d02b18a9f7d02fd36bd2743e41a6a", size = 2750611, upload-time = "2026-04-24T02:02:31.477Z" },
+ { url = "https://files.pythonhosted.org/packages/79/37/14a4579049c1eb673afd0de0cb4842982acd55b9ce2643e763db858bcea0/numba-0.65.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:1735c15c1134a5108b4d6a5c77fc0947924ea066a738dc09a52008c13df9cad3", size = 2681344, upload-time = "2026-04-24T02:02:33.65Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/22/b8d873f6466b20aa563fc9b33acd48dec89a07803ddaa2f1c8ca1cd33126/numba-0.65.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c09f49117ef255e1f1c6dad0c7a1ed39868243862a73be5706793241a3755f1b", size = 3810619, upload-time = "2026-04-24T02:02:36.041Z" },
+ { url = "https://files.pythonhosted.org/packages/62/08/e16a8b5d9a018962ebb5c66be662317cde32b9f5dab08441f90bed5522fb/numba-0.65.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:594a8680b3fadac99e97e489b1fd89007177e5336713745c3b769528c635a464", size = 3509783, upload-time = "2026-04-24T02:02:38.245Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/a5/03c970d57f4c1741354837353ce39fb5206952ae1dba8922d29c86f64805/numba-0.65.1-cp313-cp313-win_amd64.whl", hash = "sha256:85be74c0d036842699a30058f82fb88fc5ffdc59f7615cab5792ea92914c9b62", size = 2750534, upload-time = "2026-04-24T02:02:39.903Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/2e/8aed9b726d9ba5f11ad287645fd479e88278db3060a25cb1225d730eb2b7/numba-0.65.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:33f5eb68eb1c843511615d14663ce60258525d6a4c65ab040e2c2b0c4cf17450", size = 2681554, upload-time = "2026-04-24T02:02:41.812Z" },
+ { url = "https://files.pythonhosted.org/packages/87/96/f3eb235fafa82a34e2ab5dd7dc9ffff998ebf5f0bbc23fa56a96aeb44da6/numba-0.65.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:71e73029bf53a62cc6afcf96be4bd942290d8b4c55f0a454fb536158115790f7", size = 3779602, upload-time = "2026-04-24T02:02:43.726Z" },
+ { url = "https://files.pythonhosted.org/packages/09/90/b0f09b48752d23640b8284f22aa597737e8adaddc7fbfacc4708b7f73a4c/numba-0.65.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a07635e0be926b9bdbffb09137c230fb13f6ec0e564914ba937cee12ce3eb35", size = 3479532, upload-time = "2026-04-24T02:02:45.427Z" },
+ { url = "https://files.pythonhosted.org/packages/56/46/3f7fc04fb853559e74b210e0b62c19974ec844cefec611f9e535f4da3761/numba-0.65.1-cp314-cp314-win_amd64.whl", hash = "sha256:2a20fcdabdefbdacf88d85caf70c3b18c4bcb7ebb8f82e6a19486383dd26ab63", size = 2752637, upload-time = "2026-04-24T02:02:47.664Z" },
+ { url = "https://files.pythonhosted.org/packages/81/7b/c1a341a9067367778f4152a5f01061cf281fb09582c92c510ec4918cabf6/numba-0.65.1-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:548dd4b3a4508d5062768d1514b2cd7b015f9a25ec7af651c50dee243965e652", size = 2684600, upload-time = "2026-04-24T02:02:49.653Z" },
+ { url = "https://files.pythonhosted.org/packages/03/36/98ddbcf3e4f04a6dd07e1c67249955920579ba4af6bb6868e3088f4ed282/numba-0.65.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:78abc28feff2c2ff8307fff3975b6438352759c9acb797ecd6b1fb6e7e39e31d", size = 3817198, upload-time = "2026-04-24T02:02:51.266Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/83/0dad21057ece5a835599f5d24099b091703995e23dbbf894f259e91c010b/numba-0.65.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee7676cb389555805f9b9a1840cbcd1ea6c8bd5376ab6918e3a29c5ea1dbda20", size = 3533862, upload-time = "2026-04-24T02:02:52.987Z" },
+ { url = "https://files.pythonhosted.org/packages/32/36/8be7118ffd4c8440881046eac3d0982cc5ab42909508cf5d67024d62a2e4/numba-0.65.1-cp314-cp314t-win_amd64.whl", hash = "sha256:20609346e3bd75204950dcbbfe383a8d7dbf4902f442aedbf00f97fef4aa8f38", size = 2758237, upload-time = "2026-04-24T02:02:54.612Z" },
+]
+
+[[package]]
+name = "numcodecs"
+version = "0.16.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/44/bd/8a391e7c356366224734efd24da929cc4796fff468bfb179fe1af6548535/numcodecs-0.16.5.tar.gz", hash = "sha256:0d0fb60852f84c0bd9543cc4d2ab9eefd37fc8efcc410acd4777e62a1d300318", size = 6276387, upload-time = "2025-11-21T02:49:48.986Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/75/cc/55420f3641a67f78392dc0bc5d02cb9eb0a9dcebf2848d1ac77253ca61fa/numcodecs-0.16.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:24e675dc8d1550cd976a99479b87d872cb142632c75cc402fea04c08c4898523", size = 1656287, upload-time = "2025-11-21T02:49:25.755Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/6c/86644987505dcb90ba6d627d6989c27bafb0699f9fd00187e06d05ea8594/numcodecs-0.16.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:94ddfa4341d1a3ab99989d13b01b5134abb687d3dab2ead54b450aefe4ad5bd6", size = 1148899, upload-time = "2025-11-21T02:49:26.87Z" },
+ { url = "https://files.pythonhosted.org/packages/97/1e/98aaddf272552d9fef1f0296a9939d1487914a239e98678f6b20f8b0a5c8/numcodecs-0.16.5-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b554ab9ecf69de7ca2b6b5e8bc696bd9747559cb4dd5127bd08d7a28bec59c3a", size = 8534814, upload-time = "2025-11-21T02:49:28.547Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/53/78c98ef5c8b2b784453487f3e4d6c017b20747c58b470393e230c78d18e8/numcodecs-0.16.5-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ad1a379a45bd3491deab8ae6548313946744f868c21d5340116977ea3be5b1d6", size = 9173471, upload-time = "2025-11-21T02:49:30.444Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/20/2fdec87fc7f8cec950d2b0bea603c12dc9f05b4966dc5924ba5a36a61bf6/numcodecs-0.16.5-cp312-cp312-win_amd64.whl", hash = "sha256:845a9857886ffe4a3172ba1c537ae5bcc01e65068c31cf1fce1a844bd1da050f", size = 801412, upload-time = "2025-11-21T02:49:32.123Z" },
+ { url = "https://files.pythonhosted.org/packages/38/38/071ced5a5fd1c85ba0e14ba721b66b053823e5176298c2f707e50bed11d9/numcodecs-0.16.5-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:25be3a516ab677dad890760d357cfe081a371d9c0a2e9a204562318ac5969de3", size = 1654359, upload-time = "2025-11-21T02:49:33.673Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/c0/5f84ba7525577c1b9909fc2d06ef11314825fc4ad4378f61d0e4c9883b4a/numcodecs-0.16.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:0107e839ef75b854e969cb577e140b1aadb9847893937636582d23a2a4c6ce50", size = 1144237, upload-time = "2025-11-21T02:49:35.294Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/00/787ea5f237b8ea7bc67140c99155f9c00b5baf11c49afc5f3bfefa298f95/numcodecs-0.16.5-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:015a7c859ecc2a06e2a548f64008c0ec3aaecabc26456c2c62f4278d8fc20597", size = 8483064, upload-time = "2025-11-21T02:49:36.454Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/e6/d359fdd37498e74d26a167f7a51e54542e642ea47181eb4e643a69a066c3/numcodecs-0.16.5-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:84230b4b9dad2392f2a84242bd6e3e659ac137b5a1ce3571d6965fca673e0903", size = 9126063, upload-time = "2025-11-21T02:49:38.018Z" },
+ { url = "https://files.pythonhosted.org/packages/27/72/6663cc0382ddbb866136c255c837bcb96cc7ce5e83562efec55e1b995941/numcodecs-0.16.5-cp313-cp313-win_amd64.whl", hash = "sha256:5088145502ad1ebf677ec47d00eb6f0fd600658217db3e0c070c321c85d6cf3d", size = 799275, upload-time = "2025-11-21T02:49:39.558Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/9e/38e7ca8184c958b51f45d56a4aeceb1134ecde2d8bd157efadc98502cc42/numcodecs-0.16.5-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:b05647b8b769e6bc8016e9fd4843c823ce5c9f2337c089fb5c9c4da05e5275de", size = 1654721, upload-time = "2025-11-21T02:49:40.602Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/37/260fa42e7b2b08e6e00ad632f8dd620961a60a459426c26cea390f8c68d0/numcodecs-0.16.5-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:3832bd1b5af8bb3e413076b7d93318c8e7d7b68935006b9fa36ca057d1725a8f", size = 1146887, upload-time = "2025-11-21T02:49:41.721Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/15/e2e1151b5a8b14a15dfd4bb4abccce7fff7580f39bc34092780088835f3a/numcodecs-0.16.5-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:49f7b7d24f103187f53135bed28bb9f0ed6b2e14c604664726487bb6d7c882e1", size = 8476987, upload-time = "2025-11-21T02:49:43.363Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/30/16a57fc4d9fb0ba06c600408bd6634f2f1753c54a7a351c99c5e09b51ee2/numcodecs-0.16.5-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:aec9736d81b70f337d89c4070ee3ffeff113f386fd789492fa152d26a15043e4", size = 9102377, upload-time = "2025-11-21T02:49:45.508Z" },
+ { url = "https://files.pythonhosted.org/packages/31/a5/a0425af36c20d55a3ea884db4b4efca25a43bea9214ba69ca7932dd997b4/numcodecs-0.16.5-cp314-cp314-win_amd64.whl", hash = "sha256:b16a14303800e9fb88abc39463ab4706c037647ac17e49e297faa5f7d7dbbf1d", size = 819022, upload-time = "2025-11-21T02:49:47.39Z" },
+]
+
+[[package]]
+name = "numpy"
+version = "2.4.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d0/ad/fed0499ce6a338d2a03ebae59cd15093910c8875328855781952abf6c2fe/numpy-2.4.6.tar.gz", hash = "sha256:f3a3570c4a2a16746ac2c31a7c7c7b0c186b95ce902e33db6f28094ed7387dda", size = 20735807, upload-time = "2026-05-18T23:37:14.07Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/95/2a/3d7b5ac8aac24feaf9ad7ed58f45b0bbc06d37e4338ae84c9f2298b570f9/numpy-2.4.6-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:001fbb8e08d942dd57599e781f2472269ee7f2755fae407b4f67b2f0b17da3f1", size = 16689119, upload-time = "2026-05-18T23:33:54.065Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/12/92c4c131527599e8288d6918e888d88726f84d805d784b771f32408aeaef/numpy-2.4.6-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ebfb099f8dcf083deef3ac1ca4c1503f387cf76296fcb3816b66f5ecb5f54fdb", size = 14699246, upload-time = "2026-05-18T23:33:57.621Z" },
+ { url = "https://files.pythonhosted.org/packages/ad/fe/c0a6b7b2ca128a8fb228575147073b660656734b8ebe4d76c8fd748dcc79/numpy-2.4.6-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:3213d622a0283a39a93d188f3cf72b26862df52fbb4ca3697f51705016523d41", size = 5204410, upload-time = "2026-05-18T23:34:00.302Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/d4/9770d14ba719432bb90a421bfd443872ed0f70f7264b64bec12ea363d5fd/numpy-2.4.6-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:357cc07a6d7b0b182ff02249616a03742827ebb1277546b5c7cd7f7620a45698", size = 6551240, upload-time = "2026-05-18T23:34:02.852Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/c6/50a46a6205feba2343f1d6d17438107c5dc491ed1c736e6ea68689fd906b/numpy-2.4.6-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f9fb9157b4ce2971008323afe46053787b526ef624fea915b261468a8421a0f", size = 15671012, upload-time = "2026-05-18T23:34:05.485Z" },
+ { url = "https://files.pythonhosted.org/packages/99/60/14115e6364fa676c5397c2ad3004e527e9aa487abf5d0706ec81bbd08529/numpy-2.4.6-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:90f9849678c75fe7afa2d348ac842c168b0a4d3d61919687216dfc547976d853", size = 16645538, upload-time = "2026-05-18T23:34:09.265Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/c5/693cbe59e57db94d2231fa519ca3978dc9e19da5a8f088588f5c6e947ff2/numpy-2.4.6-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c1a2af6c6ef86344a6b0db6b97834208bf598db514f2b155042439b62605601a", size = 17020706, upload-time = "2026-05-18T23:34:13.053Z" },
+ { url = "https://files.pythonhosted.org/packages/ef/fc/85b7c4eff9b4966ade25c2273cf7e7012e92366c032058653934b37de044/numpy-2.4.6-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e5805d5a22fd19c8ccff10a9561f9df94436b0545619ea579db2d3c35294bce2", size = 18368541, upload-time = "2026-05-18T23:34:17.024Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/81/e1b27545deedce7f4a0b348618c6b62d74e36a4dc9ccd42f3eb2f85eee32/numpy-2.4.6-cp312-cp312-win32.whl", hash = "sha256:e3eeb0aabd6bd5ce64faae67e9935203a6991b4bc2a485a767fbafb2c5125f45", size = 5962825, upload-time = "2026-05-18T23:34:20.3Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/ca/feab00bd44aa5fe1ad2c18f08b4d3bb92e26484b0b1d1443897809ed528c/numpy-2.4.6-cp312-cp312-win_amd64.whl", hash = "sha256:d8e8286dd7cea7895157318d1b91cdacac64c479f3cbc8dce548331728484751", size = 12321687, upload-time = "2026-05-18T23:34:23.095Z" },
+ { url = "https://files.pythonhosted.org/packages/63/cf/5a6d34850a39d1093558564f77ee8e8e0bee5061151b8f05a55711001ec7/numpy-2.4.6-cp312-cp312-win_arm64.whl", hash = "sha256:4081eb135ac24158bd51cdfbef16f1c64df7063b1143f24731387137c092bec8", size = 10221482, upload-time = "2026-05-18T23:34:25.876Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/82/bdab26d7438c6791ca31b7c024ca37c1eab8b726ba236129005cd4a06e45/numpy-2.4.6-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:511dbaf848decaaaf4b4ca48032619fb3138710c4bf7da7617765edad1ef96b0", size = 16684648, upload-time = "2026-05-18T23:34:29.41Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/30/a80189bcc7f5e4258b3fbc3968d909d1756f54d023299ecc39ad6fdb9ef8/numpy-2.4.6-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:bf162abab1c1a736333192707cef898e735a5ca00f38f27eeedf44b39d9e85eb", size = 14693902, upload-time = "2026-05-18T23:34:33.013Z" },
+ { url = "https://files.pythonhosted.org/packages/97/12/70b5d0d7c15e1ebb8a6a84a8caa1d19e181d84fb58bb6d70aca29099dec1/numpy-2.4.6-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:043191bfa8eab18c776647b62723ac9dddece59743b13f49b2016094129c2b3f", size = 5198992, upload-time = "2026-05-18T23:34:36.132Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/8c/ebd2a8f8a83541f8d38cc5667e8c2b69cecfd30da6e45693e8158857d44b/numpy-2.4.6-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:6180d8b35af935aed8ece3a85e0a43f87393ae0ac87c8d2c8bd2c993f7270ef3", size = 6546944, upload-time = "2026-05-18T23:34:38.484Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/c5/7b863a97a91671a0338f4253bd3b5a3d3852f0692dae91711c9f4a10e787/numpy-2.4.6-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:72fbe16c6fac95aedf5937fa873445cec2110be35d8a4e9433d7501fd98dae6b", size = 15669392, upload-time = "2026-05-18T23:34:41.257Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/9d/3584b9984ca4c047aea75214ce1a4c4c73d849bd71b604264b7f5653f8a8/numpy-2.4.6-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a7830bab239b79cda9c08c2da014761cafb48da6150e1da17ac06283f43b6089", size = 16633220, upload-time = "2026-05-18T23:34:45.075Z" },
+ { url = "https://files.pythonhosted.org/packages/05/ae/7c67fba23bd98caec7c99261f3a16072ade14813486b0282cb29846de832/numpy-2.4.6-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ef4aea96ce4d3b074422cb4f2f64e216bf9e213004bb58ecfdf50ea02ea8eb9a", size = 17020800, upload-time = "2026-05-18T23:34:49.065Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/5d/3b6725cb31d983c5e66916f5d36f6d7e5521129e4c4404d64f918292a5b6/numpy-2.4.6-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:dfa20cc6ca228e6b155b11da03825975ce66aea520985dbbddf0f2a5a495c605", size = 18357600, upload-time = "2026-05-18T23:34:52.709Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/da/2ccc6c2fe8898dee01d90c75c5f5f914a23daf99e3e0f59516a08760c8b5/numpy-2.4.6-cp313-cp313-win32.whl", hash = "sha256:56b39e5e0622a09a25bf5baf62f4bcf0cb8a41ae6e2819cf49bbc5a74c083f91", size = 5961134, upload-time = "2026-05-18T23:34:55.618Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/cd/9cc4dc876fb065d5c220aae4d5e14826b2715331bb7618ce1fb07a679d99/numpy-2.4.6-cp313-cp313-win_amd64.whl", hash = "sha256:c4fc99836233ea196540b17ab0983aff60ed07941751930f5f4d05bc3b3b7359", size = 12318598, upload-time = "2026-05-18T23:34:58.928Z" },
+ { url = "https://files.pythonhosted.org/packages/39/1e/c0bcba1f8694116485fe28fd1be698c278fcda4141c5b0e53a2aed8b12a8/numpy-2.4.6-cp313-cp313-win_arm64.whl", hash = "sha256:a7c711e21628b52034bb5ab8d1bce291f752fcc5e92accc615778acee1ff4778", size = 10222272, upload-time = "2026-05-18T23:35:02.167Z" },
+ { url = "https://files.pythonhosted.org/packages/63/6d/cc5619247c8f4204e507f5883528372e4ac4bb189e579fb859a12e480b1f/numpy-2.4.6-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:112b06a867b235ef466ed3508ddf0238050df9c727cafb5301ac385b899189a1", size = 14821197, upload-time = "2026-05-18T23:35:05.468Z" },
+ { url = "https://files.pythonhosted.org/packages/00/58/f1c39161c87d9e9bed660f1ed4bafc0e403d5ec9650b6dd77aead07d489b/numpy-2.4.6-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:eaf7fa2de5c0be8ae6ff8e9bea2ccd725e980541244521d8d4b5f3354a27babe", size = 5326287, upload-time = "2026-05-18T23:35:08.693Z" },
+ { url = "https://files.pythonhosted.org/packages/af/57/3917ab0fd97f271a8694513581b8a36c655f111c446852c302f04ccdb6fc/numpy-2.4.6-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:7265a2f3d436e54ef9f2b52b5c937e6be778781bd97a590319d7348f1c1ca997", size = 6646763, upload-time = "2026-05-18T23:35:11.459Z" },
+ { url = "https://files.pythonhosted.org/packages/eb/0f/037e64c494b67581ae18193d770adef354c41f3f2c8ebf865602d949bf8f/numpy-2.4.6-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f74a575920ab21fe304421a3fc28793d82e299cae9eccb37084e9fc7f3617c20", size = 15728070, upload-time = "2026-05-18T23:35:14.79Z" },
+ { url = "https://files.pythonhosted.org/packages/21/a6/5d2bae9c9542eb4df16dc9c46dc79c186e9bad53805dfa5399a6023c6db0/numpy-2.4.6-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ede83e07a75dd06bc501566c1eca2afc0d61677c1472ac9ad93fdee6e638a48d", size = 16681752, upload-time = "2026-05-18T23:35:18.836Z" },
+ { url = "https://files.pythonhosted.org/packages/92/14/23d1dfb410ae362cd59ce53e936b1513d545eb40db3949ced632e19a459e/numpy-2.4.6-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:68bb27509ac1b9a3443094260f6326150663b06abe40b73a2f81160623da5b67", size = 17086024, upload-time = "2026-05-18T23:35:22.52Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/6e/23595a2c642cdf3bc567877064bdd7f91c8b0038a4453cf2daf7248eafe9/numpy-2.4.6-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:a0df0043bdb289bde1f62da130d20df23d58b45429f752bc7a8fc5325a225ecd", size = 18403398, upload-time = "2026-05-18T23:35:26.398Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/90/0ac3bc947217e66dec77e7cbc6a1979d1af70b6461b82f620d3bccd5e4c8/numpy-2.4.6-cp313-cp313t-win32.whl", hash = "sha256:29a287e0cf63ff528da061de6b9f64a4618da591ca1046aafc54062e40ca7eab", size = 6084971, upload-time = "2026-05-18T23:35:29.387Z" },
+ { url = "https://files.pythonhosted.org/packages/77/71/5673e351671a1d2bd6063b91b44f70c0affea7d1516fa7a6572941ba4aa1/numpy-2.4.6-cp313-cp313t-win_amd64.whl", hash = "sha256:25c692919ac5a01f170a3bfcd62d745b24fd095c353d50812637d6fcab442e75", size = 12458532, upload-time = "2026-05-18T23:35:32.175Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/88/19d3503c5046e688f049274b27a3ef3d771152fa80d3ba3d01a3dff61abe/numpy-2.4.6-cp313-cp313t-win_arm64.whl", hash = "sha256:1e978ec1e8bd0e0e4de6bb75de9d30cbb74db6b6a2bb727618613703ca0167dd", size = 10291881, upload-time = "2026-05-18T23:35:35.465Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/91/3ab2044d05fd16d343c5ac2e69b127f1b2854040dd20b193257c78028bd3/numpy-2.4.6-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:06ca2f61ec4385a07a6977c55ba998a4466c123642b4a32694d3128fce18c079", size = 16683458, upload-time = "2026-05-18T23:35:38.353Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/62/764ce66fa4147ae6d73071a3abf804ffe606f174618697c571acdf26a7c9/numpy-2.4.6-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:38efbc8de75c7a0fc1ac190162d892787f3f47b57cc291231aafee36b80982b7", size = 14704559, upload-time = "2026-05-18T23:35:42.14Z" },
+ { url = "https://files.pythonhosted.org/packages/60/61/23f27c172f022e04025b7dc2367f4d63c1a398120607ec896228649a6f48/numpy-2.4.6-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:d581b735e177fdcdce6fed8e7e8880a3fb6ee4e3653a3ac6af01c6f4c03effc5", size = 5209716, upload-time = "2026-05-18T23:35:45.377Z" },
+ { url = "https://files.pythonhosted.org/packages/03/71/21cf70dc6ea3e3acb95fc53a265b2fc248b981f0194ceb5b475271b8809d/numpy-2.4.6-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:0a041d3d761dc3c35cc56ce0351506a02bcbc25f7b169f652435141a17db9096", size = 6543947, upload-time = "2026-05-18T23:35:47.926Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/91/64288395ee1799bd2e0b04a305dce9666da90c961e1f3fe982a05ee1c036/numpy-2.4.6-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:40fdc1ae7125e518ea98e53e69a4ebc27e1fd50510c47b7ea130cf21e5e1d42b", size = 15685197, upload-time = "2026-05-18T23:35:50.863Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/eb/ebffaa97dc55502df69584a8f0dcf07f69a3e0b3e2323670a2722db9aa39/numpy-2.4.6-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a2c306dea656c12c68f51f4cea133cbe78ca7435eb28c735eac1d3ebe73be6e8", size = 16638245, upload-time = "2026-05-18T23:35:54.752Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/0b/54f9da33128d7e350fab89c7455902eeae70349ee52bddb448dc4a576f45/numpy-2.4.6-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:33111801a01c12a8a1e3721f0a9232f8cfc8ae2c6b7098167e6f623c6073f402", size = 17036587, upload-time = "2026-05-18T23:35:58.355Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/f0/fdebc1052db1cc37c64beb22072d67cd6d1c71adca1299f53dec2b5e20d3/numpy-2.4.6-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ae506e6902902557576a26ff33eda8695e7ecb3cb36c3b573a0765dee114ebdb", size = 18363226, upload-time = "2026-05-18T23:36:02.845Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/b4/298628d98c72b57e57f7165ae6a481a1deaf6f3c28262a6e4c739c275930/numpy-2.4.6-cp314-cp314-win32.whl", hash = "sha256:aaf159caa35993cb1f56fb9b8e4610d35758e7ca005412eb1daa856a78c9c4b1", size = 6010196, upload-time = "2026-05-18T23:36:05.92Z" },
+ { url = "https://files.pythonhosted.org/packages/df/ac/46de6dda46478f7942f839e094970be2d4a861e005c4b3bf07c92e291a09/numpy-2.4.6-cp314-cp314-win_amd64.whl", hash = "sha256:b507f5c4c1d508876d1819b6bf9a49d365b96320b5d4993426b33a23ca4b8261", size = 12450334, upload-time = "2026-05-18T23:36:09.107Z" },
+ { url = "https://files.pythonhosted.org/packages/78/92/b8b798ac784102c0da830d2257d59358e3d3d90d1e2b3f2575dad976c5cf/numpy-2.4.6-cp314-cp314-win_arm64.whl", hash = "sha256:6f41ae150c4e32db4f3310cdaf64b1593a03dbabe29eec77fc9b50fe64061df6", size = 10495678, upload-time = "2026-05-18T23:36:12.766Z" },
+ { url = "https://files.pythonhosted.org/packages/30/34/ec28d1aa8115971537c01469ab2011ee96827930f0a124de1000cc2a7ed7/numpy-2.4.6-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ece3d2cfe132e7d51f44a832b303895e6f2d499c5e74dfbdb06ee246147a304a", size = 14823672, upload-time = "2026-05-18T23:36:16.473Z" },
+ { url = "https://files.pythonhosted.org/packages/16/bd/f6d1fede4e54e8042a7ff97bb495510f3c220f94bcd9e8b228e87c92cc0d/numpy-2.4.6-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:e3e5193ef5a3dc73bceee50f7fdc2c90dbb76c42df8d8fae3d1067a583df579e", size = 5328731, upload-time = "2026-05-18T23:36:19.767Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/f0/e105b9e2fd728a9910103884decd6951d9dd73896b914a98d9a231de02ee/numpy-2.4.6-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:17f9ade344e7d9b464a084d69bcf18fc691cb1db67c62ed80820bf4926d78f0e", size = 6649805, upload-time = "2026-05-18T23:36:22.266Z" },
+ { url = "https://files.pythonhosted.org/packages/82/dd/1206a7ca6ab15e3f02069707ca96222e202af681bb73756da7527f3cb837/numpy-2.4.6-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9cd5ffd25db4e7ba6a375693b3fc0fc1791ec636c17db3720da19bde7180ec43", size = 15730496, upload-time = "2026-05-18T23:36:25.713Z" },
+ { url = "https://files.pythonhosted.org/packages/51/e7/38d3ea825dcab85a591734decb2f6c67caa7c8367d374df1a1c3842f9b07/numpy-2.4.6-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7d92c3819208a60205a12a245c91ad70cb0a85336659b19b834205573ac8456e", size = 16679616, upload-time = "2026-05-18T23:36:29.652Z" },
+ { url = "https://files.pythonhosted.org/packages/93/b7/caabfdf53edf663e0b4eb74d7d405d83baef09eb5e83bcd32d601d72b93e/numpy-2.4.6-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e85b752a1e912b70eaad4fafbd4d1238007ab221de2009b9a2f5ae7461239895", size = 17085145, upload-time = "2026-05-18T23:36:33.449Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/45/68d7c33a6bcf3e5aa3bdbd57a367e6f615286dfd6482f97e8ffeb734306e/numpy-2.4.6-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:29cb7f67d10b479ff07c17d33e39f78c07f71c40ef30d63c153d340e96cd3fb4", size = 18403813, upload-time = "2026-05-18T23:36:37.369Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/50/0753655aa844c99cd9e018aacf76f130f1bd81d881bb74bc0aef5d73a8ba/numpy-2.4.6-cp314-cp314t-win32.whl", hash = "sha256:260a5d70215b61ab4fadf5c7baacd64821842975eea312125ed3c39a6391b063", size = 6156982, upload-time = "2026-05-18T23:36:40.817Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/d4/7c67becf668f973cb490cec3e98dfd799d866f9c989a54d355672cfa0db6/numpy-2.4.6-cp314-cp314t-win_amd64.whl", hash = "sha256:81a1cca95ed5bb92aa8b10dd2cdc9a0d3853a50fad926c28b5d7e8ea54389627", size = 12638908, upload-time = "2026-05-18T23:36:43.996Z" },
+ { url = "https://files.pythonhosted.org/packages/43/bb/e1c71a4295b1b1d1393d50dbb4f2a36283c6859d9d3892e84f00ec5a91d5/numpy-2.4.6-cp314-cp314t-win_arm64.whl", hash = "sha256:0c9136e14ed34a9e343a31c533d78a9813a69a3148332bce5e9821cb2f996e66", size = 10565867, upload-time = "2026-05-18T23:36:47.114Z" },
+]
+
+[[package]]
+name = "obstore"
+version = "0.11.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions", marker = "python_full_version < '3.13'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/2e/2f/f83afaab7945509d72245b2b00af0b4834ce78fdd2d9ae9f0ad1a3036a91/obstore-0.11.0.tar.gz", hash = "sha256:a2f55163bcd348b4a60d12e6893eac50eddc742bad8032a1705d49140b992204", size = 130565, upload-time = "2026-06-25T18:29:49.405Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fc/b2/00c213e7e5ca8065f97e37e55294adab836e3f6a88b23e4029069aaecf95/obstore-0.11.0-cp311-abi3-macosx_10_12_x86_64.whl", hash = "sha256:42f36546c7ac44dbab1173d2330a8a1b1a3f0e37950e553b8c904e3dd0744b25", size = 5491935, upload-time = "2026-06-25T18:28:32.029Z" },
+ { url = "https://files.pythonhosted.org/packages/ac/37/6a6b9a5e15a8a37c24d14317a87648097c4888593b588510c03c030d2e90/obstore-0.11.0-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:687bb9d3962d568b7c439c5d0c6fea19b2749862a8e5c8eebd0c058c4eccde9e", size = 4672619, upload-time = "2026-06-25T18:28:33.852Z" },
+ { url = "https://files.pythonhosted.org/packages/28/f9/6745ce8c4f7bfac19dc14a4438b48a2e93a689b92b0cecfc695e41a4e8b1/obstore-0.11.0-cp311-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:010b51578c7514a41719d795cdb7a1e6529be509dac3772e477187a59422bb97", size = 5072806, upload-time = "2026-06-25T18:28:36.127Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/18/991d3b3cdd851c0225e55f3dc45b47fd9e249827d188995011469f805132/obstore-0.11.0-cp311-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:cfaa8129a3f5d8518a3a75184d4b02348db0f6263177cd1f0951f6568243cc9e", size = 5303777, upload-time = "2026-06-25T18:28:37.89Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/e9/90e56015a45b5e56a84fc3188c4e5fb088b288d41992c73a629e10df6760/obstore-0.11.0-cp311-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:c790a5cb9ff2970d1f464a6a708d734dce9939e9f668cb6708c5dba5d61589b2", size = 5493871, upload-time = "2026-06-25T18:28:39.981Z" },
+ { url = "https://files.pythonhosted.org/packages/66/02/f1744091d59ce71c5523174eb860fbb298275c901e89b9ea6fbf3e654a33/obstore-0.11.0-cp311-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:827113e12fe8088e0281a9d57b90b2b8dbc8a6ffe3b15dadb9baa5feb3d266c1", size = 5361913, upload-time = "2026-06-25T18:28:42.089Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/59/3f47822683ee2b6db8685faa25829946d6343a561251ec2704548455d946/obstore-0.11.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a2ff6d3ed553298828fb760b4aef6347fbcc7b5c5e3ce3f8381ce805c370021a", size = 5638724, upload-time = "2026-06-25T18:28:43.897Z" },
+ { url = "https://files.pythonhosted.org/packages/23/50/1df335fdf9b527b3933f1e94ab6fc720ad314260fab8591cb0b6668ff192/obstore-0.11.0-cp311-abi3-manylinux_2_24_aarch64.whl", hash = "sha256:39d04b324fcf984e7050734ebda77b81764025b0c011750201a0d8954087f7aa", size = 5413508, upload-time = "2026-06-25T18:28:45.624Z" },
+ { url = "https://files.pythonhosted.org/packages/de/dc/a259aba149b841ca7c91fea177df9972a60a636b54077beed1a35b254994/obstore-0.11.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:37c0d15d775b1370ef5204ee3919a5ddf7e2592d11815213105f8db031f2ab8d", size = 5619995, upload-time = "2026-06-25T18:28:47.599Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/b4/ec25fdb4d6b060bc6eea647fc0e88f75fcc20fe8d16d67fb0dbe999d323b/obstore-0.11.0-cp311-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:7f468caf9b6e0f12ff151e5fe618de5fc9192befa9bd02734b06de4efd2e49f6", size = 5299512, upload-time = "2026-06-25T18:28:49.629Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/e5/29be060d06ec13e2af3d1b6cfb77b7c37f8be6c56b77295c945fefad73e4/obstore-0.11.0-cp311-abi3-musllinux_1_2_i686.whl", hash = "sha256:42d8e8fad85be8ee488c1a9a9b7c6a42128abb84e67175da40d3d1165c1846df", size = 5427026, upload-time = "2026-06-25T18:28:51.317Z" },
+ { url = "https://files.pythonhosted.org/packages/57/b7/577a965f440e9ea64243518663f9d16be7df8eafc7123818e8e841fa21ce/obstore-0.11.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:9c8fd2a544e2e0b926669c47fcfb8d2314e234abc240ea165dae04ee42e1d7ac", size = 5869187, upload-time = "2026-06-25T18:28:53.166Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/18/8fdbaee22bfd5b9c44e1fdff8ca0508e2fe60c42bf9fc85f0c9c27b4ecf2/obstore-0.11.0-cp311-abi3-win_amd64.whl", hash = "sha256:6fb3d4678c0f4242d3109362e9b1df5d7b27765f43d5aacb2e81af53a75cb9ef", size = 5329384, upload-time = "2026-06-25T18:28:55.305Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/8b/7555e48ec768728fcfc71a051c6b28d6ddaf1bececf492ce5ef995aab5f0/obstore-0.11.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:f3132393eff9f3f2b543ecbb3bcc12319a7c433fef06493b4350d6854d505a14", size = 5515763, upload-time = "2026-06-25T18:28:57.527Z" },
+ { url = "https://files.pythonhosted.org/packages/23/8f/94d83f3336421cbb5e436ab0ae5695eae72f7c82990d6b1ac090712c8052/obstore-0.11.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a3a8da19b47af4c14ecc694209b3c18ac6d89f96be5656ee3a19b77947c14155", size = 4649491, upload-time = "2026-06-25T18:28:59.386Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/86/11f4e1f51a8c6cf21a5915c018d2357201ad3c5799d418f0c6529fafaab2/obstore-0.11.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e91298b9b6c3a0408c28eece62bca6c5b6cda2f6350351d84e07b4dc8fb2631f", size = 5060659, upload-time = "2026-06-25T18:29:01.139Z" },
+ { url = "https://files.pythonhosted.org/packages/49/e7/fd3036b0923d10e878e2073020f1ef692a618ed1cc3980d3e4a468c93713/obstore-0.11.0-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3422c532486671dfb5e3e739bf15ec9ca2a8da544a3c23b74ae3857dcab1c6a8", size = 5277058, upload-time = "2026-06-25T18:29:02.96Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/a7/b016c3ac6857ac856326dc0a292e3871b8500d03f25f3b90e168b05de357/obstore-0.11.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3de027ce46cf0592c2b41654e2c27dbc52a4a726f6fbc511380acc0da3a9f658", size = 5475852, upload-time = "2026-06-25T18:29:05.032Z" },
+ { url = "https://files.pythonhosted.org/packages/85/9e/644ffe8db7757de7f71f94a036ab24222bccb4de290d3ca69f76547e812d/obstore-0.11.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:1a80a95548678210bc336b866e37139c293565c3b163eb3fef2433d5d6640a33", size = 5363082, upload-time = "2026-06-25T18:29:06.878Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/0b/26af6b5fa6ba96af84086f44f78b4e5b0af1729c31402d7b28c68989d174/obstore-0.11.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:acaa261dc15efb95bbeca06f8fe9b47ee23d7302a6aa1fa3a9654baab8b23d7c", size = 5629116, upload-time = "2026-06-25T18:29:08.771Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/ce/f30d502991c6719b2fbd7b8385ef3e39da07bfca099108bcc5eeed8b9c20/obstore-0.11.0-cp314-cp314t-manylinux_2_24_aarch64.whl", hash = "sha256:a3300cabbc3129670987b3723629c791d83c117ef1b6a0c670c2043648e000a1", size = 5404534, upload-time = "2026-06-25T18:29:11.294Z" },
+ { url = "https://files.pythonhosted.org/packages/52/20/d5bf5f816e868717ba647ed9a2109e800deb402d0265d410456b3fcb4376/obstore-0.11.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:f4e6a9480843645cd4ee122c41d5c5a46a56f1e9cdda85638826f2e0e439fe5c", size = 5613159, upload-time = "2026-06-25T18:29:13.414Z" },
+ { url = "https://files.pythonhosted.org/packages/db/b4/6d4c1c211e3b06cc8554189e0d4406e8fa1f98ed55f9213b8e398a11599f/obstore-0.11.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:9fb2b1814c4314b8903f4e2ebbe8c3365fea6543669615ee9a0288b0d3a2edeb", size = 5286279, upload-time = "2026-06-25T18:29:15.424Z" },
+ { url = "https://files.pythonhosted.org/packages/02/5e/d7b5589424a56171b16ab94cf92eb493490c300aaa044913bbdd94cace68/obstore-0.11.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:63fb9b072815eafe4705f617f567d896b1c858adcb390795fa1e269367791031", size = 5401780, upload-time = "2026-06-25T18:29:17.514Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/18/841baea8936e51a18b0e5d4c51f09c0a7798cb73b027e9794be2362a0f0b/obstore-0.11.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:086fafba314ff98cfab1c4bf7814699e862513e8889720cf6f7462296cb32787", size = 5853618, upload-time = "2026-06-25T18:29:19.354Z" },
+ { url = "https://files.pythonhosted.org/packages/83/9a/d6127f5422b78e0222b0a9eadcfd7a5aa8d873a9498da7d4a77d4ac8ce2e/obstore-0.11.0-cp314-cp314t-win_amd64.whl", hash = "sha256:676d1154f6f08721110f9b7d14ee3a3c0293abaf9da135bb90f54e276dca1cac", size = 5314113, upload-time = "2026-06-25T18:29:21.209Z" },
+]
+
+[[package]]
+name = "packaging"
+version = "26.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d7/f1/e7a6dd94a8d4a5626c03e4e99c87f241ba9e350cd9e6d75123f992427270/packaging-26.2.tar.gz", hash = "sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661", size = 228134, upload-time = "2026-04-24T20:15:23.917Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" },
+]
+
+[[package]]
+name = "pandas"
+version = "2.3.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+ { name = "python-dateutil" },
+ { name = "pytz" },
+ { name = "tzdata" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/33/01/d40b85317f86cf08d853a4f495195c73815fdf205eef3993821720274518/pandas-2.3.3.tar.gz", hash = "sha256:e05e1af93b977f7eafa636d043f9f94c7ee3ac81af99c13508215942e64c993b", size = 4495223, upload-time = "2025-09-29T23:34:51.853Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9c/fb/231d89e8637c808b997d172b18e9d4a4bc7bf31296196c260526055d1ea0/pandas-2.3.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6d21f6d74eb1725c2efaa71a2bfc661a0689579b58e9c0ca58a739ff0b002b53", size = 11597846, upload-time = "2025-09-29T23:19:48.856Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/bd/bf8064d9cfa214294356c2d6702b716d3cf3bb24be59287a6a21e24cae6b/pandas-2.3.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:3fd2f887589c7aa868e02632612ba39acb0b8948faf5cc58f0850e165bd46f35", size = 10729618, upload-time = "2025-09-29T23:39:08.659Z" },
+ { url = "https://files.pythonhosted.org/packages/57/56/cf2dbe1a3f5271370669475ead12ce77c61726ffd19a35546e31aa8edf4e/pandas-2.3.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ecaf1e12bdc03c86ad4a7ea848d66c685cb6851d807a26aa245ca3d2017a1908", size = 11737212, upload-time = "2025-09-29T23:19:59.765Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/63/cd7d615331b328e287d8233ba9fdf191a9c2d11b6af0c7a59cfcec23de68/pandas-2.3.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b3d11d2fda7eb164ef27ffc14b4fcab16a80e1ce67e9f57e19ec0afaf715ba89", size = 12362693, upload-time = "2025-09-29T23:20:14.098Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/de/8b1895b107277d52f2b42d3a6806e69cfef0d5cf1d0ba343470b9d8e0a04/pandas-2.3.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:a68e15f780eddf2b07d242e17a04aa187a7ee12b40b930bfdd78070556550e98", size = 12771002, upload-time = "2025-09-29T23:20:26.76Z" },
+ { url = "https://files.pythonhosted.org/packages/87/21/84072af3187a677c5893b170ba2c8fbe450a6ff911234916da889b698220/pandas-2.3.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:371a4ab48e950033bcf52b6527eccb564f52dc826c02afd9a1bc0ab731bba084", size = 13450971, upload-time = "2025-09-29T23:20:41.344Z" },
+ { url = "https://files.pythonhosted.org/packages/86/41/585a168330ff063014880a80d744219dbf1dd7a1c706e75ab3425a987384/pandas-2.3.3-cp312-cp312-win_amd64.whl", hash = "sha256:a16dcec078a01eeef8ee61bf64074b4e524a2a3f4b3be9326420cabe59c4778b", size = 10992722, upload-time = "2025-09-29T23:20:54.139Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/4b/18b035ee18f97c1040d94debd8f2e737000ad70ccc8f5513f4eefad75f4b/pandas-2.3.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:56851a737e3470de7fa88e6131f41281ed440d29a9268dcbf0002da5ac366713", size = 11544671, upload-time = "2025-09-29T23:21:05.024Z" },
+ { url = "https://files.pythonhosted.org/packages/31/94/72fac03573102779920099bcac1c3b05975c2cb5f01eac609faf34bed1ca/pandas-2.3.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:bdcd9d1167f4885211e401b3036c0c8d9e274eee67ea8d0758a256d60704cfe8", size = 10680807, upload-time = "2025-09-29T23:21:15.979Z" },
+ { url = "https://files.pythonhosted.org/packages/16/87/9472cf4a487d848476865321de18cc8c920b8cab98453ab79dbbc98db63a/pandas-2.3.3-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e32e7cc9af0f1cc15548288a51a3b681cc2a219faa838e995f7dc53dbab1062d", size = 11709872, upload-time = "2025-09-29T23:21:27.165Z" },
+ { url = "https://files.pythonhosted.org/packages/15/07/284f757f63f8a8d69ed4472bfd85122bd086e637bf4ed09de572d575a693/pandas-2.3.3-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:318d77e0e42a628c04dc56bcef4b40de67918f7041c2b061af1da41dcff670ac", size = 12306371, upload-time = "2025-09-29T23:21:40.532Z" },
+ { url = "https://files.pythonhosted.org/packages/33/81/a3afc88fca4aa925804a27d2676d22dcd2031c2ebe08aabd0ae55b9ff282/pandas-2.3.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4e0a175408804d566144e170d0476b15d78458795bb18f1304fb94160cabf40c", size = 12765333, upload-time = "2025-09-29T23:21:55.77Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/0f/b4d4ae743a83742f1153464cf1a8ecfafc3ac59722a0b5c8602310cb7158/pandas-2.3.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:93c2d9ab0fc11822b5eece72ec9587e172f63cff87c00b062f6e37448ced4493", size = 13418120, upload-time = "2025-09-29T23:22:10.109Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/c7/e54682c96a895d0c808453269e0b5928a07a127a15704fedb643e9b0a4c8/pandas-2.3.3-cp313-cp313-win_amd64.whl", hash = "sha256:f8bfc0e12dc78f777f323f55c58649591b2cd0c43534e8355c51d3fede5f4dee", size = 10993991, upload-time = "2025-09-29T23:25:04.889Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/ca/3f8d4f49740799189e1395812f3bf23b5e8fc7c190827d55a610da72ce55/pandas-2.3.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:75ea25f9529fdec2d2e93a42c523962261e567d250b0013b16210e1d40d7c2e5", size = 12048227, upload-time = "2025-09-29T23:22:24.343Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/5a/f43efec3e8c0cc92c4663ccad372dbdff72b60bdb56b2749f04aa1d07d7e/pandas-2.3.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:74ecdf1d301e812db96a465a525952f4dde225fdb6d8e5a521d47e1f42041e21", size = 11411056, upload-time = "2025-09-29T23:22:37.762Z" },
+ { url = "https://files.pythonhosted.org/packages/46/b1/85331edfc591208c9d1a63a06baa67b21d332e63b7a591a5ba42a10bb507/pandas-2.3.3-cp313-cp313t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6435cb949cb34ec11cc9860246ccb2fdc9ecd742c12d3304989017d53f039a78", size = 11645189, upload-time = "2025-09-29T23:22:51.688Z" },
+ { url = "https://files.pythonhosted.org/packages/44/23/78d645adc35d94d1ac4f2a3c4112ab6f5b8999f4898b8cdf01252f8df4a9/pandas-2.3.3-cp313-cp313t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:900f47d8f20860de523a1ac881c4c36d65efcb2eb850e6948140fa781736e110", size = 12121912, upload-time = "2025-09-29T23:23:05.042Z" },
+ { url = "https://files.pythonhosted.org/packages/53/da/d10013df5e6aaef6b425aa0c32e1fc1f3e431e4bcabd420517dceadce354/pandas-2.3.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:a45c765238e2ed7d7c608fc5bc4a6f88b642f2f01e70c0c23d2224dd21829d86", size = 12712160, upload-time = "2025-09-29T23:23:28.57Z" },
+ { url = "https://files.pythonhosted.org/packages/bd/17/e756653095a083d8a37cbd816cb87148debcfcd920129b25f99dd8d04271/pandas-2.3.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:c4fc4c21971a1a9f4bdb4c73978c7f7256caa3e62b323f70d6cb80db583350bc", size = 13199233, upload-time = "2025-09-29T23:24:24.876Z" },
+ { url = "https://files.pythonhosted.org/packages/04/fd/74903979833db8390b73b3a8a7d30d146d710bd32703724dd9083950386f/pandas-2.3.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:ee15f284898e7b246df8087fc82b87b01686f98ee67d85a17b7ab44143a3a9a0", size = 11540635, upload-time = "2025-09-29T23:25:52.486Z" },
+ { url = "https://files.pythonhosted.org/packages/21/00/266d6b357ad5e6d3ad55093a7e8efc7dd245f5a842b584db9f30b0f0a287/pandas-2.3.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1611aedd912e1ff81ff41c745822980c49ce4a7907537be8692c8dbc31924593", size = 10759079, upload-time = "2025-09-29T23:26:33.204Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/05/d01ef80a7a3a12b2f8bbf16daba1e17c98a2f039cbc8e2f77a2c5a63d382/pandas-2.3.3-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6d2cefc361461662ac48810cb14365a365ce864afe85ef1f447ff5a1e99ea81c", size = 11814049, upload-time = "2025-09-29T23:27:15.384Z" },
+ { url = "https://files.pythonhosted.org/packages/15/b2/0e62f78c0c5ba7e3d2c5945a82456f4fac76c480940f805e0b97fcbc2f65/pandas-2.3.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ee67acbbf05014ea6c763beb097e03cd629961c8a632075eeb34247120abcb4b", size = 12332638, upload-time = "2025-09-29T23:27:51.625Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/33/dd70400631b62b9b29c3c93d2feee1d0964dc2bae2e5ad7a6c73a7f25325/pandas-2.3.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c46467899aaa4da076d5abc11084634e2d197e9460643dd455ac3db5856b24d6", size = 12886834, upload-time = "2025-09-29T23:28:21.289Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/18/b5d48f55821228d0d2692b34fd5034bb185e854bdb592e9c640f6290e012/pandas-2.3.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6253c72c6a1d990a410bc7de641d34053364ef8bcd3126f7e7450125887dffe3", size = 13409925, upload-time = "2025-09-29T23:28:58.261Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/3d/124ac75fcd0ecc09b8fdccb0246ef65e35b012030defb0e0eba2cbbbe948/pandas-2.3.3-cp314-cp314-win_amd64.whl", hash = "sha256:1b07204a219b3b7350abaae088f451860223a52cfb8a6c53358e7948735158e5", size = 11109071, upload-time = "2025-09-29T23:32:27.484Z" },
+ { url = "https://files.pythonhosted.org/packages/89/9c/0e21c895c38a157e0faa1fb64587a9226d6dd46452cac4532d80c3c4a244/pandas-2.3.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:2462b1a365b6109d275250baaae7b760fd25c726aaca0054649286bcfbb3e8ec", size = 12048504, upload-time = "2025-09-29T23:29:31.47Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/82/b69a1c95df796858777b68fbe6a81d37443a33319761d7c652ce77797475/pandas-2.3.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:0242fe9a49aa8b4d78a4fa03acb397a58833ef6199e9aa40a95f027bb3a1b6e7", size = 11410702, upload-time = "2025-09-29T23:29:54.591Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/88/702bde3ba0a94b8c73a0181e05144b10f13f29ebfc2150c3a79062a8195d/pandas-2.3.3-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a21d830e78df0a515db2b3d2f5570610f5e6bd2e27749770e8bb7b524b89b450", size = 11634535, upload-time = "2025-09-29T23:30:21.003Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/1e/1bac1a839d12e6a82ec6cb40cda2edde64a2013a66963293696bbf31fbbb/pandas-2.3.3-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2e3ebdb170b5ef78f19bfb71b0dc5dc58775032361fa188e814959b74d726dd5", size = 12121582, upload-time = "2025-09-29T23:30:43.391Z" },
+ { url = "https://files.pythonhosted.org/packages/44/91/483de934193e12a3b1d6ae7c8645d083ff88dec75f46e827562f1e4b4da6/pandas-2.3.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:d051c0e065b94b7a3cea50eb1ec32e912cd96dba41647eb24104b6c6c14c5788", size = 12699963, upload-time = "2025-09-29T23:31:10.009Z" },
+ { url = "https://files.pythonhosted.org/packages/70/44/5191d2e4026f86a2a109053e194d3ba7a31a2d10a9c2348368c63ed4e85a/pandas-2.3.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:3869faf4bd07b3b66a9f462417d0ca3a9df29a9f6abd5d0d0dbab15dac7abe87", size = 13202175, upload-time = "2025-09-29T23:31:59.173Z" },
+]
+
+[[package]]
+name = "parso"
+version = "0.8.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/30/4b/90c937815137d43ce71ba043cd3566221e9df6b9c805f24b5d138c9d40a7/parso-0.8.7.tar.gz", hash = "sha256:eaaac4c9fdd5e9e8852dc778d2d7405897ec510f2a298071453e5e3a07914bb1", size = 401824, upload-time = "2026-05-01T23:13:02.138Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/99/5d/8268b644392ee874ee82a635cd0df1773de230bde356c38de28e298392cc/parso-0.8.7-py2.py3-none-any.whl", hash = "sha256:a8926eb2a1b915486941fdbd31e86a4baf88fe8c210f25f2f35ecec5b574ca1c", size = 107025, upload-time = "2026-05-01T23:12:58.867Z" },
+]
+
+[[package]]
+name = "pathspec"
+version = "1.1.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/5a/82/42f767fc1c1143d6fd36efb827202a2d997a375e160a71eb2888a925aac1/pathspec-1.1.1.tar.gz", hash = "sha256:17db5ecd524104a120e173814c90367a96a98d07c45b2e10c2f3919fff91bf5a", size = 135180, upload-time = "2026-04-27T01:46:08.907Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl", hash = "sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189", size = 57328, upload-time = "2026-04-27T01:46:07.06Z" },
+]
+
+[[package]]
+name = "patsy"
+version = "1.0.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/be/44/ed13eccdd0519eff265f44b670d46fbb0ec813e2274932dc1c0e48520f7d/patsy-1.0.2.tar.gz", hash = "sha256:cdc995455f6233e90e22de72c37fcadb344e7586fb83f06696f54d92f8ce74c0", size = 399942, upload-time = "2025-10-20T16:17:37.535Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f1/70/ba4b949bdc0490ab78d545459acd7702b211dfccf7eb89bbc1060f52818d/patsy-1.0.2-py2.py3-none-any.whl", hash = "sha256:37bfddbc58fcf0362febb5f54f10743f8b21dd2aa73dec7e7ef59d1b02ae668a", size = 233301, upload-time = "2025-10-20T16:17:36.563Z" },
+]
+
+[[package]]
+name = "pcst-fast"
+version = "1.0.10"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pybind11" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/d4/fd/64b51c867bad63e6622ff97cad6230b94b19b5a61e30424cd69c8353091c/pcst_fast-1.0.10.tar.gz", hash = "sha256:3b5694110ce2e004471f383267d5e4ab7fe1ba9828954e8c42560ac1e42b25e6", size = 1717415, upload-time = "2024-01-29T22:14:29.769Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/25/a7/d6d72f4dff7d46a7338e9dd638817938e0e208d131d9c508a85c4f9fabd3/pcst_fast-1.0.10-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:375990478236bde3b40925a3dcc279a9dc8bb061db01e79b23c47c5578a32df3", size = 96522, upload-time = "2024-01-29T22:13:51.022Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/d0/c8bd86957614dfe16dee5a843013bf7903ba369928b9684f2582fc18a75b/pcst_fast-1.0.10-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:6b3aa1cc8a544e9390584cd0453280ce5da3b93cedbc1b7623b439c40838ddb6", size = 1230119, upload-time = "2024-01-29T22:13:52.029Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/f7/f20c09bbea70c17c1f50273e82347ab8a5d7a2b3aed2288083bee078ea47/pcst_fast-1.0.10-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a051b58d66eeee139a4f4b274467fb737074447cabeb5d4360d31be3b7c6530a", size = 1260476, upload-time = "2024-01-29T22:13:53.481Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/75/68dffc0c863a3fd18e975db44072d0c2e19d6b867d8e32ade54ee0ba313e/pcst_fast-1.0.10-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:837be0ca99f3417b35c873046a0b22d94de309960f3106eb8fb1453b705ad85d", size = 1766196, upload-time = "2024-01-29T22:13:54.698Z" },
+ { url = "https://files.pythonhosted.org/packages/86/2b/eab1a1d752a94980dd0055f55bcee731ac893e169b527e84f0a6393a037e/pcst_fast-1.0.10-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:da194e710fa847aa7707ef9fed391b9a08fafa0b6d681a8846a9cc77d26bffca", size = 1746127, upload-time = "2024-01-29T22:13:56.534Z" },
+]
+
+[[package]]
+name = "pexpect"
+version = "4.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "ptyprocess", marker = "sys_platform != 'emscripten' and sys_platform != 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/42/92/cc564bf6381ff43ce1f4d06852fc19a2f11d180f23dc32d9588bee2f149d/pexpect-4.9.0.tar.gz", hash = "sha256:ee7d41123f3c9911050ea2c2dac107568dc43b2d3b0c7557a33212c398ead30f", size = 166450, upload-time = "2023-11-25T09:07:26.339Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9e/c3/059298687310d527a58bb01f3b1965787ee3b40dce76752eda8b44e9a2c5/pexpect-4.9.0-py2.py3-none-any.whl", hash = "sha256:7236d1e080e4936be2dc3e326cec0af72acf9212a7e1d060210e70a47e253523", size = 63772, upload-time = "2023-11-25T06:56:14.81Z" },
+]
+
+[[package]]
+name = "pillow"
+version = "12.2.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/8c/21/c2bcdd5906101a30244eaffc1b6e6ce71a31bd0742a01eb89e660ebfac2d/pillow-12.2.0.tar.gz", hash = "sha256:a830b1a40919539d07806aa58e1b114df53ddd43213d9c8b75847eee6c0182b5", size = 46987819, upload-time = "2026-04-01T14:46:17.687Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/58/be/7482c8a5ebebbc6470b3eb791812fff7d5e0216c2be3827b30b8bb6603ed/pillow-12.2.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2d192a155bbcec180f8564f693e6fd9bccff5a7af9b32e2e4bf8c9c69dbad6b5", size = 5308279, upload-time = "2026-04-01T14:43:13.246Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/95/0a351b9289c2b5cbde0bacd4a83ebc44023e835490a727b2a3bd60ddc0f4/pillow-12.2.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f3f40b3c5a968281fd507d519e444c35f0ff171237f4fdde090dd60699458421", size = 4695490, upload-time = "2026-04-01T14:43:15.584Z" },
+ { url = "https://files.pythonhosted.org/packages/de/af/4e8e6869cbed569d43c416fad3dc4ecb944cb5d9492defaed89ddd6fe871/pillow-12.2.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:03e7e372d5240cc23e9f07deca4d775c0817bffc641b01e9c3af208dbd300987", size = 6284462, upload-time = "2026-04-01T14:43:18.268Z" },
+ { url = "https://files.pythonhosted.org/packages/e9/9e/c05e19657fd57841e476be1ab46c4d501bffbadbafdc31a6d665f8b737b6/pillow-12.2.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b86024e52a1b269467a802258c25521e6d742349d760728092e1bc2d135b4d76", size = 8094744, upload-time = "2026-04-01T14:43:20.716Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/54/1789c455ed10176066b6e7e6da1b01e50e36f94ba584dc68d9eebfe9156d/pillow-12.2.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7371b48c4fa448d20d2714c9a1f775a81155050d383333e0a6c15b1123dda005", size = 6398371, upload-time = "2026-04-01T14:43:23.443Z" },
+ { url = "https://files.pythonhosted.org/packages/43/e3/fdc657359e919462369869f1c9f0e973f353f9a9ee295a39b1fea8ee1a77/pillow-12.2.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:62f5409336adb0663b7caa0da5c7d9e7bdbaae9ce761d34669420c2a801b2780", size = 7087215, upload-time = "2026-04-01T14:43:26.758Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/f8/2f6825e441d5b1959d2ca5adec984210f1ec086435b0ed5f52c19b3b8a6e/pillow-12.2.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:01afa7cf67f74f09523699b4e88c73fb55c13346d212a59a2db1f86b0a63e8c5", size = 6509783, upload-time = "2026-04-01T14:43:29.56Z" },
+ { url = "https://files.pythonhosted.org/packages/67/f9/029a27095ad20f854f9dba026b3ea6428548316e057e6fc3545409e86651/pillow-12.2.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fc3d34d4a8fbec3e88a79b92e5465e0f9b842b628675850d860b8bd300b159f5", size = 7212112, upload-time = "2026-04-01T14:43:32.091Z" },
+ { url = "https://files.pythonhosted.org/packages/be/42/025cfe05d1be22dbfdb4f264fe9de1ccda83f66e4fc3aac94748e784af04/pillow-12.2.0-cp312-cp312-win32.whl", hash = "sha256:58f62cc0f00fd29e64b29f4fd923ffdb3859c9f9e6105bfc37ba1d08994e8940", size = 6378489, upload-time = "2026-04-01T14:43:34.601Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/7b/25a221d2c761c6a8ae21bfa3874988ff2583e19cf8a27bf2fee358df7942/pillow-12.2.0-cp312-cp312-win_amd64.whl", hash = "sha256:7f84204dee22a783350679a0333981df803dac21a0190d706a50475e361c93f5", size = 7084129, upload-time = "2026-04-01T14:43:37.213Z" },
+ { url = "https://files.pythonhosted.org/packages/10/e1/542a474affab20fd4a0f1836cb234e8493519da6b76899e30bcc5d990b8b/pillow-12.2.0-cp312-cp312-win_arm64.whl", hash = "sha256:af73337013e0b3b46f175e79492d96845b16126ddf79c438d7ea7ff27783a414", size = 2463612, upload-time = "2026-04-01T14:43:39.421Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/01/53d10cf0dbad820a8db274d259a37ba50b88b24768ddccec07355382d5ad/pillow-12.2.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:8297651f5b5679c19968abefd6bb84d95fe30ef712eb1b2d9b2d31ca61267f4c", size = 4100837, upload-time = "2026-04-01T14:43:41.506Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/98/f3a6657ecb698c937f6c76ee564882945f29b79bad496abcba0e84659ec5/pillow-12.2.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:50d8520da2a6ce0af445fa6d648c4273c3eeefbc32d7ce049f22e8b5c3daecc2", size = 4176528, upload-time = "2026-04-01T14:43:43.773Z" },
+ { url = "https://files.pythonhosted.org/packages/69/bc/8986948f05e3ea490b8442ea1c1d4d990b24a7e43d8a51b2c7d8b1dced36/pillow-12.2.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:766cef22385fa1091258ad7e6216792b156dc16d8d3fa607e7545b2b72061f1c", size = 3640401, upload-time = "2026-04-01T14:43:45.87Z" },
+ { url = "https://files.pythonhosted.org/packages/34/46/6c717baadcd62bc8ed51d238d521ab651eaa74838291bda1f86fe1f864c9/pillow-12.2.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5d2fd0fa6b5d9d1de415060363433f28da8b1526c1c129020435e186794b3795", size = 5308094, upload-time = "2026-04-01T14:43:48.438Z" },
+ { url = "https://files.pythonhosted.org/packages/71/43/905a14a8b17fdb1ccb58d282454490662d2cb89a6bfec26af6d3520da5ec/pillow-12.2.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:56b25336f502b6ed02e889f4ece894a72612fe885889a6e8c4c80239ff6e5f5f", size = 4695402, upload-time = "2026-04-01T14:43:51.292Z" },
+ { url = "https://files.pythonhosted.org/packages/73/dd/42107efcb777b16fa0393317eac58f5b5cf30e8392e266e76e51cff28c3d/pillow-12.2.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f1c943e96e85df3d3478f7b691f229887e143f81fedab9b20205349ab04d73ed", size = 6280005, upload-time = "2026-04-01T14:43:54.242Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/68/b93e09e5e8549019e61acf49f65b1a8530765a7f812c77a7461bca7e4494/pillow-12.2.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:03f6fab9219220f041c74aeaa2939ff0062bd5c364ba9ce037197f4c6d498cd9", size = 8090669, upload-time = "2026-04-01T14:43:57.335Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/6e/3ccb54ce8ec4ddd1accd2d89004308b7b0b21c4ac3d20fa70af4760a4330/pillow-12.2.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5cdfebd752ec52bf5bb4e35d9c64b40826bc5b40a13df7c3cda20a2c03a0f5ed", size = 6395194, upload-time = "2026-04-01T14:43:59.864Z" },
+ { url = "https://files.pythonhosted.org/packages/67/ee/21d4e8536afd1a328f01b359b4d3997b291ffd35a237c877b331c1c3b71c/pillow-12.2.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:eedf4b74eda2b5a4b2b2fb4c006d6295df3bf29e459e198c90ea48e130dc75c3", size = 7082423, upload-time = "2026-04-01T14:44:02.74Z" },
+ { url = "https://files.pythonhosted.org/packages/78/5f/e9f86ab0146464e8c133fe85df987ed9e77e08b29d8d35f9f9f4d6f917ba/pillow-12.2.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:00a2865911330191c0b818c59103b58a5e697cae67042366970a6b6f1b20b7f9", size = 6505667, upload-time = "2026-04-01T14:44:05.381Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/1e/409007f56a2fdce61584fd3acbc2bbc259857d555196cedcadc68c015c82/pillow-12.2.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:1e1757442ed87f4912397c6d35a0db6a7b52592156014706f17658ff58bbf795", size = 7208580, upload-time = "2026-04-01T14:44:08.39Z" },
+ { url = "https://files.pythonhosted.org/packages/23/c4/7349421080b12fb35414607b8871e9534546c128a11965fd4a7002ccfbee/pillow-12.2.0-cp313-cp313-win32.whl", hash = "sha256:144748b3af2d1b358d41286056d0003f47cb339b8c43a9ea42f5fea4d8c66b6e", size = 6375896, upload-time = "2026-04-01T14:44:11.197Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/82/8a3739a5e470b3c6cbb1d21d315800d8e16bff503d1f16b03a4ec3212786/pillow-12.2.0-cp313-cp313-win_amd64.whl", hash = "sha256:390ede346628ccc626e5730107cde16c42d3836b89662a115a921f28440e6a3b", size = 7081266, upload-time = "2026-04-01T14:44:13.947Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/25/f968f618a062574294592f668218f8af564830ccebdd1fa6200f598e65c5/pillow-12.2.0-cp313-cp313-win_arm64.whl", hash = "sha256:8023abc91fba39036dbce14a7d6535632f99c0b857807cbbbf21ecc9f4717f06", size = 2463508, upload-time = "2026-04-01T14:44:16.312Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/a4/b342930964e3cb4dce5038ae34b0eab4653334995336cd486c5a8c25a00c/pillow-12.2.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:042db20a421b9bafecc4b84a8b6e444686bd9d836c7fd24542db3e7df7baad9b", size = 5309927, upload-time = "2026-04-01T14:44:18.89Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/de/23198e0a65a9cf06123f5435a5d95cea62a635697f8f03d134d3f3a96151/pillow-12.2.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:dd025009355c926a84a612fecf58bb315a3f6814b17ead51a8e48d3823d9087f", size = 4698624, upload-time = "2026-04-01T14:44:21.115Z" },
+ { url = "https://files.pythonhosted.org/packages/01/a6/1265e977f17d93ea37aa28aa81bad4fa597933879fac2520d24e021c8da3/pillow-12.2.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:88ddbc66737e277852913bd1e07c150cc7bb124539f94c4e2df5344494e0a612", size = 6321252, upload-time = "2026-04-01T14:44:23.663Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/83/5982eb4a285967baa70340320be9f88e57665a387e3a53a7f0db8231a0cd/pillow-12.2.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d362d1878f00c142b7e1a16e6e5e780f02be8195123f164edf7eddd911eefe7c", size = 8126550, upload-time = "2026-04-01T14:44:26.772Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/48/6ffc514adce69f6050d0753b1a18fd920fce8cac87620d5a31231b04bfc5/pillow-12.2.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2c727a6d53cb0018aadd8018c2b938376af27914a68a492f59dfcaca650d5eea", size = 6433114, upload-time = "2026-04-01T14:44:29.615Z" },
+ { url = "https://files.pythonhosted.org/packages/36/a3/f9a77144231fb8d40ee27107b4463e205fa4677e2ca2548e14da5cf18dce/pillow-12.2.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:efd8c21c98c5cc60653bcb311bef2ce0401642b7ce9d09e03a7da87c878289d4", size = 7115667, upload-time = "2026-04-01T14:44:32.773Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/fc/ac4ee3041e7d5a565e1c4fd72a113f03b6394cc72ab7089d27608f8aaccb/pillow-12.2.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9f08483a632889536b8139663db60f6724bfcb443c96f1b18855860d7d5c0fd4", size = 6538966, upload-time = "2026-04-01T14:44:35.252Z" },
+ { url = "https://files.pythonhosted.org/packages/c0/a8/27fb307055087f3668f6d0a8ccb636e7431d56ed0750e07a60547b1e083e/pillow-12.2.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:dac8d77255a37e81a2efcbd1fc05f1c15ee82200e6c240d7e127e25e365c39ea", size = 7238241, upload-time = "2026-04-01T14:44:37.875Z" },
+ { url = "https://files.pythonhosted.org/packages/ad/4b/926ab182c07fccae9fcb120043464e1ff1564775ec8864f21a0ebce6ac25/pillow-12.2.0-cp313-cp313t-win32.whl", hash = "sha256:ee3120ae9dff32f121610bb08e4313be87e03efeadfc6c0d18f89127e24d0c24", size = 6379592, upload-time = "2026-04-01T14:44:40.336Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/c4/f9e476451a098181b30050cc4c9a3556b64c02cf6497ea421ac047e89e4b/pillow-12.2.0-cp313-cp313t-win_amd64.whl", hash = "sha256:325ca0528c6788d2a6c3d40e3568639398137346c3d6e66bb61db96b96511c98", size = 7085542, upload-time = "2026-04-01T14:44:43.251Z" },
+ { url = "https://files.pythonhosted.org/packages/00/a4/285f12aeacbe2d6dc36c407dfbbe9e96d4a80b0fb710a337f6d2ad978c75/pillow-12.2.0-cp313-cp313t-win_arm64.whl", hash = "sha256:2e5a76d03a6c6dcef67edabda7a52494afa4035021a79c8558e14af25313d453", size = 2465765, upload-time = "2026-04-01T14:44:45.996Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/98/4595daa2365416a86cb0d495248a393dfc84e96d62ad080c8546256cb9c0/pillow-12.2.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:3adc9215e8be0448ed6e814966ecf3d9952f0ea40eb14e89a102b87f450660d8", size = 4100848, upload-time = "2026-04-01T14:44:48.48Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/79/40184d464cf89f6663e18dfcf7ca21aae2491fff1a16127681bf1fa9b8cf/pillow-12.2.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:6a9adfc6d24b10f89588096364cc726174118c62130c817c2837c60cf08a392b", size = 4176515, upload-time = "2026-04-01T14:44:51.353Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/63/703f86fd4c422a9cf722833670f4f71418fb116b2853ff7da722ea43f184/pillow-12.2.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:6a6e67ea2e6feda684ed370f9a1c52e7a243631c025ba42149a2cc5934dec295", size = 3640159, upload-time = "2026-04-01T14:44:53.588Z" },
+ { url = "https://files.pythonhosted.org/packages/71/e0/fb22f797187d0be2270f83500aab851536101b254bfa1eae10795709d283/pillow-12.2.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:2bb4a8d594eacdfc59d9e5ad972aa8afdd48d584ffd5f13a937a664c3e7db0ed", size = 5312185, upload-time = "2026-04-01T14:44:56.039Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/8c/1a9e46228571de18f8e28f16fabdfc20212a5d019f3e3303452b3f0a580d/pillow-12.2.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:80b2da48193b2f33ed0c32c38140f9d3186583ce7d516526d462645fd98660ae", size = 4695386, upload-time = "2026-04-01T14:44:58.663Z" },
+ { url = "https://files.pythonhosted.org/packages/70/62/98f6b7f0c88b9addd0e87c217ded307b36be024d4ff8869a812b241d1345/pillow-12.2.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:22db17c68434de69d8ecfc2fe821569195c0c373b25cccb9cbdacf2c6e53c601", size = 6280384, upload-time = "2026-04-01T14:45:01.5Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/03/688747d2e91cfbe0e64f316cd2e8005698f76ada3130d0194664174fa5de/pillow-12.2.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7b14cc0106cd9aecda615dd6903840a058b4700fcb817687d0ee4fc8b6e389be", size = 8091599, upload-time = "2026-04-01T14:45:04.5Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/35/577e22b936fcdd66537329b33af0b4ccfefaeabd8aec04b266528cddb33c/pillow-12.2.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8cbeb542b2ebc6fcdacabf8aca8c1a97c9b3ad3927d46b8723f9d4f033288a0f", size = 6396021, upload-time = "2026-04-01T14:45:07.117Z" },
+ { url = "https://files.pythonhosted.org/packages/11/8d/d2532ad2a603ca2b93ad9f5135732124e57811d0168155852f37fbce2458/pillow-12.2.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4bfd07bc812fbd20395212969e41931001fd59eb55a60658b0e5710872e95286", size = 7083360, upload-time = "2026-04-01T14:45:09.763Z" },
+ { url = "https://files.pythonhosted.org/packages/5e/26/d325f9f56c7e039034897e7380e9cc202b1e368bfd04d4cbe6a441f02885/pillow-12.2.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9aba9a17b623ef750a4d11b742cbafffeb48a869821252b30ee21b5e91392c50", size = 6507628, upload-time = "2026-04-01T14:45:12.378Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/f7/769d5632ffb0988f1c5e7660b3e731e30f7f8ec4318e94d0a5d674eb65a4/pillow-12.2.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:deede7c263feb25dba4e82ea23058a235dcc2fe1f6021025dc71f2b618e26104", size = 7209321, upload-time = "2026-04-01T14:45:15.122Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/7a/c253e3c645cd47f1aceea6a8bacdba9991bf45bb7dfe927f7c893e89c93c/pillow-12.2.0-cp314-cp314-win32.whl", hash = "sha256:632ff19b2778e43162304d50da0181ce24ac5bb8180122cbe1bf4673428328c7", size = 6479723, upload-time = "2026-04-01T14:45:17.797Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/8b/601e6566b957ca50e28725cb6c355c59c2c8609751efbecd980db44e0349/pillow-12.2.0-cp314-cp314-win_amd64.whl", hash = "sha256:4e6c62e9d237e9b65fac06857d511e90d8461a32adcc1b9065ea0c0fa3a28150", size = 7217400, upload-time = "2026-04-01T14:45:20.529Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/94/220e46c73065c3e2951bb91c11a1fb636c8c9ad427ac3ce7d7f3359b9b2f/pillow-12.2.0-cp314-cp314-win_arm64.whl", hash = "sha256:b1c1fbd8a5a1af3412a0810d060a78b5136ec0836c8a4ef9aa11807f2a22f4e1", size = 2554835, upload-time = "2026-04-01T14:45:23.162Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/ab/1b426a3974cb0e7da5c29ccff4807871d48110933a57207b5a676cccc155/pillow-12.2.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:57850958fe9c751670e49b2cecf6294acc99e562531f4bd317fa5ddee2068463", size = 5314225, upload-time = "2026-04-01T14:45:25.637Z" },
+ { url = "https://files.pythonhosted.org/packages/19/1e/dce46f371be2438eecfee2a1960ee2a243bbe5e961890146d2dee1ff0f12/pillow-12.2.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:d5d38f1411c0ed9f97bcb49b7bd59b6b7c314e0e27420e34d99d844b9ce3b6f3", size = 4698541, upload-time = "2026-04-01T14:45:28.355Z" },
+ { url = "https://files.pythonhosted.org/packages/55/c3/7fbecf70adb3a0c33b77a300dc52e424dc22ad8cdc06557a2e49523b703d/pillow-12.2.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:5c0a9f29ca8e79f09de89293f82fc9b0270bb4af1d58bc98f540cc4aedf03166", size = 6322251, upload-time = "2026-04-01T14:45:30.924Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/3c/7fbc17cfb7e4fe0ef1642e0abc17fc6c94c9f7a16be41498e12e2ba60408/pillow-12.2.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1610dd6c61621ae1cf811bef44d77e149ce3f7b95afe66a4512f8c59f25d9ebe", size = 8127807, upload-time = "2026-04-01T14:45:33.908Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/c3/a8ae14d6defd2e448493ff512fae903b1e9bd40b72efb6ec55ce0048c8ce/pillow-12.2.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0a34329707af4f73cf1782a36cd2289c0368880654a2c11f027bcee9052d35dd", size = 6433935, upload-time = "2026-04-01T14:45:36.623Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/32/2880fb3a074847ac159d8f902cb43278a61e85f681661e7419e6596803ed/pillow-12.2.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8e9c4f5b3c546fa3458a29ab22646c1c6c787ea8f5ef51300e5a60300736905e", size = 7116720, upload-time = "2026-04-01T14:45:39.258Z" },
+ { url = "https://files.pythonhosted.org/packages/46/87/495cc9c30e0129501643f24d320076f4cc54f718341df18cc70ec94c44e1/pillow-12.2.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:fb043ee2f06b41473269765c2feae53fc2e2fbf96e5e22ca94fb5ad677856f06", size = 6540498, upload-time = "2026-04-01T14:45:41.879Z" },
+ { url = "https://files.pythonhosted.org/packages/18/53/773f5edca692009d883a72211b60fdaf8871cbef075eaa9d577f0a2f989e/pillow-12.2.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f278f034eb75b4e8a13a54a876cc4a5ab39173d2cdd93a638e1b467fc545ac43", size = 7239413, upload-time = "2026-04-01T14:45:44.705Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/e4/4b64a97d71b2a83158134abbb2f5bd3f8a2ea691361282f010998f339ec7/pillow-12.2.0-cp314-cp314t-win32.whl", hash = "sha256:6bb77b2dcb06b20f9f4b4a8454caa581cd4dd0643a08bacf821216a16d9c8354", size = 6482084, upload-time = "2026-04-01T14:45:47.568Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/13/306d275efd3a3453f72114b7431c877d10b1154014c1ebbedd067770d629/pillow-12.2.0-cp314-cp314t-win_amd64.whl", hash = "sha256:6562ace0d3fb5f20ed7290f1f929cae41b25ae29528f2af1722966a0a02e2aa1", size = 7225152, upload-time = "2026-04-01T14:45:50.032Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/6e/cf826fae916b8658848d7b9f38d88da6396895c676e8086fc0988073aaf8/pillow-12.2.0-cp314-cp314t-win_arm64.whl", hash = "sha256:aa88ccfe4e32d362816319ed727a004423aab09c5cea43c01a4b435643fa34eb", size = 2556579, upload-time = "2026-04-01T14:45:52.529Z" },
+]
+
+[[package]]
+name = "platformdirs"
+version = "4.10.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/d7/47/e4501f49c178ae1d9f4a75073fda4204f52647993f075a9db4d14930e0c5/platformdirs-4.10.0.tar.gz", hash = "sha256:31e761a6a0ca04faf7353ea759bdba55652be214725111e5aac52dfa29d4bef7", size = 31224, upload-time = "2026-05-28T03:32:53.587Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/81/e6/cd9575ac904136b3cbf7aa7ee819ef86eedb7274e46f230e94ea4342e729/platformdirs-4.10.0-py3-none-any.whl", hash = "sha256:fb516cdb12eb0d857d0cd85a7c57cea4d060bee4578d6cf5a14dfdf8cbf8784a", size = 22743, upload-time = "2026-05-28T03:32:52.175Z" },
+]
+
+[[package]]
+name = "pluggy"
+version = "1.6.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" },
+]
+
+[[package]]
+name = "prompt-toolkit"
+version = "3.0.52"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "wcwidth" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a1/96/06e01a7b38dce6fe1db213e061a4602dd6032a8a97ef6c1a862537732421/prompt_toolkit-3.0.52.tar.gz", hash = "sha256:28cde192929c8e7321de85de1ddbe736f1375148b02f2e17edd840042b1be855", size = 434198, upload-time = "2025-08-27T15:24:02.057Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/84/03/0d3ce49e2505ae70cf43bc5bb3033955d2fc9f932163e84dc0779cc47f48/prompt_toolkit-3.0.52-py3-none-any.whl", hash = "sha256:9aac639a3bbd33284347de5ad8d68ecc044b91a762dc39b7c21095fcd6a19955", size = 391431, upload-time = "2025-08-27T15:23:59.498Z" },
+]
+
+[[package]]
+name = "propcache"
+version = "0.5.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ec/44/c87281c333769159c50594f22610f77398a47ccbfbbf23074e744e86f87c/propcache-0.5.2.tar.gz", hash = "sha256:01c4fc7480cd0598bb4b57022df55b9ca296da7fc5a8760bd8451a7e63a7d427", size = 50208, upload-time = "2026-05-08T21:02:12.199Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4a/cb/e27bc2b2737a0bb49962b275efa051e8f1c35a936df7d5139b6b658b7dc9/propcache-0.5.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:806719138ecd720339a12410fb9614ac9b2b2d3a5fdf8235d56981c36f4039ba", size = 95887, upload-time = "2026-05-08T21:00:11.277Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/13/b8ae04c59392f8d11c6cd9fb4011d1dc7c86b81225c770280300e259ffe1/propcache-0.5.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:db2b80ea58eab4f86b2beec3cc8b39e8ff9276ac20e96b7cce43c8ae84cd6b5a", size = 54654, upload-time = "2026-05-08T21:00:12.604Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/7d/49777a3e20b55863d4794384a38acd460c04157b0a00f8602b0d508b8431/propcache-0.5.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:e5cbfac9f61484f7e9f3597775500cd3ebe8274e9b050c38f9525c77c97520bf", size = 55190, upload-time = "2026-05-08T21:00:13.935Z" },
+ { url = "https://files.pythonhosted.org/packages/44/c7/085d0cd63062e84044e3f05797749c3f8e3938ff3aeb0eb2f69d43fafc91/propcache-0.5.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5dbc581d2814337da56222fab8dc5f161cd798a434e49bac27930aaef798e144", size = 59995, upload-time = "2026-05-08T21:00:15.526Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/42/32cf8e3009e92b2645cf1e944f701e8ea4e924dffde1ee26db860bcbf7e4/propcache-0.5.2-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:857187f381f88c8e2fa2fe56ab94879d011b883d5a2ee5a1b60a8cd2a06846d9", size = 63422, upload-time = "2026-05-08T21:00:16.824Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/1b/f112433f99fc979431b87a39ef169e3f8df070d99a72792c56d6937ac48b/propcache-0.5.2-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:178b4a2cdaac1818e2bf1c5a99b94383fa73ea5382e032a48dec07dc5668dc42", size = 64342, upload-time = "2026-05-08T21:00:18.362Z" },
+ { url = "https://files.pythonhosted.org/packages/14/15/5574111ae50dd6e879456888c0eadd4c5a869959775854e18e18a6b345f3/propcache-0.5.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6f328175a2cde1f0ff2c4ed8ce968b9dcfb55f3a7153f39e2957ed994da13476", size = 61639, upload-time = "2026-05-08T21:00:19.692Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/da/4d775080b1490c0ae604acda868bd71aabe3a89ed16f2aa4339eb8a283e7/propcache-0.5.2-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:5671d09a36b06d0fd4a3da0fccbcae360e9b1570924171a15e9e0997f0249fba", size = 61588, upload-time = "2026-05-08T21:00:21.155Z" },
+ { url = "https://files.pythonhosted.org/packages/04/ac/f076982cbe2195ee9cf32de5a1e46951d9fb399fc207f390562dd0fd8fb2/propcache-0.5.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:80168e2ebe4d3ec6599d10ad8f520304ae1cad9b6c5a95372aef1b66b7bfb53a", size = 60029, upload-time = "2026-05-08T21:00:22.713Z" },
+ { url = "https://files.pythonhosted.org/packages/70/60/189be62e0dd898dce3b331e1b8c7a543cd3a405ac0c81fe8ee8a9d5d77e1/propcache-0.5.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:45f11346f884bc47444f6e6647131055844134c3175b629f84952e2b5cd62b64", size = 56774, upload-time = "2026-05-08T21:00:24.001Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/9e/93377b9c7939c1ffae98f878dee955efadfd638078bc86dbc21f9d52f651/propcache-0.5.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:8e778ebd44ef4f66ed60a0416b06b489687db264a9c0b3620362f26489492913", size = 63532, upload-time = "2026-05-08T21:00:25.545Z" },
+ { url = "https://files.pythonhosted.org/packages/14/f9/590ef6cfb9b8028d516d287812ece32bb0bc5f11fbb9c8bf6b2e6313fec8/propcache-0.5.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:c0cb9ed24c8964e172768d455a38254c2dd8a552905729ce006cad3d3dda59b1", size = 61592, upload-time = "2026-05-08T21:00:27.186Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/5e/70958b3034c297a630bba2f17ca7abc2d5f39a803ad7e370ab79d1ecd022/propcache-0.5.2-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:1d1ad32d9d4355e2be65574fd0bfd3677e7066b009cd5b9b2dee8aa6a6393b33", size = 64788, upload-time = "2026-05-08T21:00:28.8Z" },
+ { url = "https://files.pythonhosted.org/packages/12/fd/77fe5936d8c3086ca9048f7f415f122ed82e53884a9ec193646b42deef06/propcache-0.5.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:c80f4ba3e8f00189165999a742ee526ebeccedf6c3f7beb0c7df821e9772435a", size = 62514, upload-time = "2026-05-08T21:00:30.098Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/74/66bd798b5b3be70aa1b391f5cc9d6a0a5532d7fd3b19ec0b213e72e6ad9d/propcache-0.5.2-cp312-cp312-win32.whl", hash = "sha256:8c7972d8f193740d9175f0998ab38717e6cd322d5935c5b0fef8c0d323fd9031", size = 39018, upload-time = "2026-05-08T21:00:31.622Z" },
+ { url = "https://files.pythonhosted.org/packages/61/7c/5c0d34aa3024694d6dcb9271cdbdd08c4e47c1c0ad95ec7e7bc74cdea145/propcache-0.5.2-cp312-cp312-win_amd64.whl", hash = "sha256:d9ee8826a7d47863a08ac44e1a5f611a462eefc3a194b492da242128bec75b42", size = 42322, upload-time = "2026-05-08T21:00:32.918Z" },
+ { url = "https://files.pythonhosted.org/packages/4d/91/875812f1a3feb20ceba818ef39fbe4d92f1081e04ac815c822496d0d038b/propcache-0.5.2-cp312-cp312-win_arm64.whl", hash = "sha256:2800a4a8ead6b28cccd1ec54b59346f0def7922ee1c7598e8499c733cfbb7c84", size = 38172, upload-time = "2026-05-08T21:00:35.124Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/09/f049e45385503fe67db75a6b6186a7b9f0c3930366dc960522c312a825b1/propcache-0.5.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:099aaf4b4d1a02265b92a977edf00b5c4f63b3b17ac6de39b0d637c9cac0188a", size = 94457, upload-time = "2026-05-08T21:00:36.355Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/65/83d1d05655baf63113731bd5a1008435e14f8d1e5a06cbe4ec5b23ad7a31/propcache-0.5.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:68ce1c44c7a813a7f71ea04315a8c7b330b63db99d059a797a4651bb6f69f117", size = 53835, upload-time = "2026-05-08T21:00:38.072Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/12/a6ba6482bb5ea3260c000c9b20881c95fa11c6b30173715668259f844ed7/propcache-0.5.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:fc299c129490f55f254cd90be0deca4764e36e9a7c08b4aa588479a3bbed3098", size = 54545, upload-time = "2026-05-08T21:00:39.319Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/19/7fa086f5764c59ec8a8e157cd93aa8497acc00aba9dcdec56bfffb32602d/propcache-0.5.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a6ae2198be502c10f09b2516e7b5d019816924bc3183a43ce792a7bd6625e6f4", size = 59886, upload-time = "2026-05-08T21:00:40.621Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/e4/5d7663dc8235956c8f5281698a3af1d351d8820341ddd890f59d9a9127f2/propcache-0.5.2-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6041d31504dc1779d700e1edcfb08eea334b357620b06681a4eabb57a74e574e", size = 63261, upload-time = "2026-05-08T21:00:41.775Z" },
+ { url = "https://files.pythonhosted.org/packages/4a/4a/15a03adee24d6350da4292caeac44c34c033d2afe5e87eb370f38854560f/propcache-0.5.2-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f7eabc04151c78a9f4d5bbb5f1faf571e4defeb4b585e0fe95b60ff2dbe4d3d7", size = 64184, upload-time = "2026-05-08T21:00:43.018Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/c6/979176efdaa3d239e36d503d5af63a0a773b36662ed8f52e5b6a6d9fd40e/propcache-0.5.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4db0ba63d693afd40d249bd93f842b5f144f8fcbb83de05660373bcf30517b1d", size = 61534, upload-time = "2026-05-08T21:00:44.507Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/22/63e8cd1bae4c2d2be6493b6b7d10566ddafad88137cfbc99964a1119853c/propcache-0.5.2-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:1dbcf7675229b35d31abb6547d8ebc8c27a830ac3f9a794edff6254873ec7c0a", size = 61500, upload-time = "2026-05-08T21:00:45.796Z" },
+ { url = "https://files.pythonhosted.org/packages/60/5a/28e5d9acbac1cc9ccb67045e8c1b943aa8d79fdf39c93bd73cacd68008ea/propcache-0.5.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d310c013aad2c72f1c3f2f8dd3279d460a858c551f97aeb8c63e4693cca7b4d2", size = 59994, upload-time = "2026-05-08T21:00:47.093Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/40/db650677f554a95b9c01a7c9d93d629e93a15562f5deb4573c9ee136fed2/propcache-0.5.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:06187263ddad280d05b4d8a8b3bb7d164cbebd469236544a42e6d9b28ac6a4fa", size = 56884, upload-time = "2026-05-08T21:00:48.376Z" },
+ { url = "https://files.pythonhosted.org/packages/80/45/70b39b89516ff8b96bf732fa6fded8cef20f293cb1508690101c3c07ec51/propcache-0.5.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:3115559b8effafd63b142ea5ed53d63a16ea6469cbc63dce4ee194b42db5d853", size = 63464, upload-time = "2026-05-08T21:00:49.954Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/e2/fa59d3a89eac5534293124af4f1d0d0ada091ce4a0ab4610ce03fd2bdd8d/propcache-0.5.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:c60462af8e6dc30c35407c7237ea908d777b22862bbee27bc4699c0d8bcdc45a", size = 61588, upload-time = "2026-05-08T21:00:51.281Z" },
+ { url = "https://files.pythonhosted.org/packages/0b/97/efb547a55c4bc7381cfb202d6a2239ac621045277bc1ea5dfd3a7f0516c0/propcache-0.5.2-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:40314bca9ac559716fe374094fc81c11dcc34b64fd6c585360f5775690505704", size = 64667, upload-time = "2026-05-08T21:00:52.602Z" },
+ { url = "https://files.pythonhosted.org/packages/92/56/f5c7d9b4b7595d5127da38974d791b2153f3d1eae6c674af3583ace92ad3/propcache-0.5.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:cfa21e036ce1e1db2be04ba3b85d2df1bb1702fa01932d984c5464c665228ff4", size = 62463, upload-time = "2026-05-08T21:00:54.303Z" },
+ { url = "https://files.pythonhosted.org/packages/bd/3b/484a3a65fc9f9f60c41dcd17b428bace5389544e2c680994534a20755066/propcache-0.5.2-cp313-cp313-win32.whl", hash = "sha256:f156a3529f38063b6dbaf356e15602a7f95f8055b1295a438433a6386f10463d", size = 38621, upload-time = "2026-05-08T21:00:55.808Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/fd/3f0f10dba4dabad3bf53102be007abf55481067952bde0fdddff439e7c61/propcache-0.5.2-cp313-cp313-win_amd64.whl", hash = "sha256:dfed59d0a5aeb01e242e66ff0300bc4a265a7c05f612d30016f0b60b1017d757", size = 41649, upload-time = "2026-05-08T21:00:57.061Z" },
+ { url = "https://files.pythonhosted.org/packages/90/ec/6ce619cc32bb500a482f811f9cd509368b4e58e638d13f2c68f370d6b475/propcache-0.5.2-cp313-cp313-win_arm64.whl", hash = "sha256:ba338430e87ceb9c8f0cf754de38a9860560261e56c00376debd628698a7364f", size = 37636, upload-time = "2026-05-08T21:00:58.646Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/82/c1d268bbbf2ef981c5bf0fbbe746db617c66e3bcefe431a1aa8943fbe23a/propcache-0.5.2-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:a592f5f3da71c8691c788c13cb6734b6d17663d2e1cb8caddf0673d01ef8847d", size = 98872, upload-time = "2026-05-08T21:00:59.889Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/d4/52c871e73e864e6b34c0e2d58ac1ec5ccd149497ddc7ad2137ae98323a35/propcache-0.5.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:6a997d0489e9668a384fcfd5061b857aa5361de73191cac204d04b889cfbbafa", size = 56257, upload-time = "2026-05-08T21:01:01.195Z" },
+ { url = "https://files.pythonhosted.org/packages/67/f0/9b90ca2a210b3d09bcfcd96ecd0f55545c091535abce2a45de2775cfd357/propcache-0.5.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:10734b5484ea113152ee25a91dccedf81631791805d2c9ccb054958e51842c94", size = 56696, upload-time = "2026-05-08T21:01:02.941Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/0e/6e9d4ba07c8e56e21ddec1e75f12148142b21ca83a51871babce095334f4/propcache-0.5.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cafca7e56c12bb02ae16d283742bef25a61122e9dab2b5b3f2ccbe589ce32164", size = 62378, upload-time = "2026-05-08T21:01:04.475Z" },
+ { url = "https://files.pythonhosted.org/packages/65/19/c10badaa463dde8a27ce884f8ee2ec37e6035b7c9f5ff0c8f74f06f08dac/propcache-0.5.2-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f064f8d2b59177878b7615df1735cd8fe3462ed6be8c7b217d17a276489c2b7f", size = 65283, upload-time = "2026-05-08T21:01:05.959Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/b6/93bea99ca80e19cef6512a8580e5b7857bbe09422d9daa7fd4ef5723306c/propcache-0.5.2-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:f78abfa8dfc32376fd1aacf597b2f2fbbe0ea751419aee718af5d4f82537ef8c", size = 66616, upload-time = "2026-05-08T21:01:07.228Z" },
+ { url = "https://files.pythonhosted.org/packages/83/e4/5c7462e50625f051f37fb38b8224f7639f667184bbd34424ec83819bb1b7/propcache-0.5.2-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f7467da8a9822bf1a55336f877340c5bcbd3c482afc43a99771169f74a26dedc", size = 63773, upload-time = "2026-05-08T21:01:08.514Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/b6/99238894047b13c823be25027e736626cd414a52a5e30d2c3347c2733529/propcache-0.5.2-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a6ddc6ac9e25de626c1f129c1b467d7ecd33ce2237d3fd0c4e429feef0a7ee1f", size = 63664, upload-time = "2026-05-08T21:01:09.874Z" },
+ { url = "https://files.pythonhosted.org/packages/85/1e/a3a1a63116a2b8edb415a8bb9a6f0c34bd03830b1e18e8ce2904e1dc1cf4/propcache-0.5.2-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:2f22cbbac9e26a8e864c0985ff1268d5d939d53d9d9411a9824279097e03a2cb", size = 62643, upload-time = "2026-05-08T21:01:11.132Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/03/893cf147de2fc6543c5eaa07ad833170e7e2a2385725bbebe8c0503723bb/propcache-0.5.2-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:fc76378c62a0f04d0cd82fbb1a2cd2d7e28fcb40d5873f28a6c44e388aaa2751", size = 59595, upload-time = "2026-05-08T21:01:12.387Z" },
+ { url = "https://files.pythonhosted.org/packages/86/3b/04c1a2e12c57766568ba75ba72b3bf2042818d4c1425fab6fc07155c7cff/propcache-0.5.2-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:acd2c8edba48e31e58a363b8cf4e5c7db3b04b3f9e371f601df30d9b0d244836", size = 65711, upload-time = "2026-05-08T21:01:13.676Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/34/80f8d0099f8d6bacc4de1624c85672681c8cd1149ca2da0e38fd120b817f/propcache-0.5.2-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:452b5065457eb9991ec5eb38ff41d6cd4c991c9ac7c531c4d5849ae473a9a13f", size = 64247, upload-time = "2026-05-08T21:01:14.936Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/1a/8b08f3a5f1037e9e370c55883ceeeee0f6dd0416fb2d2d67b8bfc91f2a79/propcache-0.5.2-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:3430bb2bfe1331885c427745a751e774ee679fd4344f80b97bf879815fe8fa55", size = 67102, upload-time = "2026-05-08T21:01:16.281Z" },
+ { url = "https://files.pythonhosted.org/packages/34/68/8bdb7bb7756d76e005490649d10e4a8369e610c74d619f71e1aedf889e9c/propcache-0.5.2-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:cef6cea3922890dd6c9654971001fa797b526c16ab5e1e46c05fd6f877be7568", size = 64964, upload-time = "2026-05-08T21:01:17.57Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/aa/50fb0b5d3968b61a510926ff8b8465f1d6e976b3ab74496d7a4b9fc42515/propcache-0.5.2-cp313-cp313t-win32.whl", hash = "sha256:72d61e16dd78228b58c5d47be830ff3da7e5f139abdf0aef9d86cde1c5cf2191", size = 42546, upload-time = "2026-05-08T21:01:18.946Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/4c/0ddbae64321bd4a95bcbfc19307238016b5b1fee645c84626c8d539e5b74/propcache-0.5.2-cp313-cp313t-win_amd64.whl", hash = "sha256:0958834041a0166d343b8d2cedcd8bcbaeb4fdbe0cf08320c5379f143c3be6e7", size = 46330, upload-time = "2026-05-08T21:01:20.162Z" },
+ { url = "https://files.pythonhosted.org/packages/00/d9/9cddc8efb78d8af264c5ec9f6d10b62f57c515feda8d321595f56010fb23/propcache-0.5.2-cp313-cp313t-win_arm64.whl", hash = "sha256:6de8bd93ddde9b992cf2b2e0d796d501a19026b5b9fd87356d7d0779531a8d96", size = 40521, upload-time = "2026-05-08T21:01:21.399Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/ea/23ee535d90ce8bcc465a3028eb3cc0ce3bd1005f4bb27710b30587de798d/propcache-0.5.2-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:46088abff4cba581dea21ae0467a480526cb25aa5f3c269e909f800328bc3999", size = 94662, upload-time = "2026-05-08T21:01:22.683Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/06/c5a52f419b5d8972f8d46a7577476090d8e3263ff589ce40b5ca4968d5be/propcache-0.5.2-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:fc88b26f08d634f7bc819a7852e5214f5802641ab8d9fd5326892292eee1993e", size = 53928, upload-time = "2026-05-08T21:01:23.986Z" },
+ { url = "https://files.pythonhosted.org/packages/63/b1/4260d67d6bd85e58a66b72d54ce15d5de789b6f3870cc6bedf8ff9667401/propcache-0.5.2-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:97797ebb098e670a2f92dd66f32897e30d7615b14e7f59711de23e30a9072539", size = 54650, upload-time = "2026-05-08T21:01:25.305Z" },
+ { url = "https://files.pythonhosted.org/packages/70/06/2f46c318e3307cd7a6a7481def374ce838c0fe20084b39dd54b0879d0e99/propcache-0.5.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ba57fffe4ac99c5d30076161b5866336d97600769bad35cc68f7774b15298a4e", size = 59912, upload-time = "2026-05-08T21:01:26.545Z" },
+ { url = "https://files.pythonhosted.org/packages/4c/29/fe1aebec2ce57ab985a9c382bded1124431f85078113aa222c5d278430d4/propcache-0.5.2-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:583c19759d9eec1e5b69e2fbef36a7d9c326041be9746cb822d335c8cedc2979", size = 63300, upload-time = "2026-05-08T21:01:27.937Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/18/2334b26768b6c82be8c69e83671b767d5ef426aa09b0cba6c2ea47816774/propcache-0.5.2-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:d0326e2e5e1f3163fa306c834e48e8d490e5fae607a097a40c0648109b47ba80", size = 64208, upload-time = "2026-05-08T21:01:29.484Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/76/7f1bfd6afff4c5e38e36a3c6d68eb5f4b7311ea80baf693db78d95b603c4/propcache-0.5.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e00820e192c8dbebcafb383ebbf99030895f09905e7a0eb2e0340a0bcc2bc825", size = 61633, upload-time = "2026-05-08T21:01:31.068Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/46/b3ff8aba2b4953a3e50de2cf72f1b5748b8eca93b15f3dc2c84339084c09/propcache-0.5.2-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c66afea89b1e43725731d2004732a046fe6fe955d51f952c3e95a7314a284a39", size = 61724, upload-time = "2026-05-08T21:01:32.374Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/01/814cfcafbcff954f94c01cf30e097ddc88a076b5440fbcf4570753437d40/propcache-0.5.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d4dc37dec6c6cdad0b57881a5658fd14fbf53e333b1a86cf86559f190e1d9ec4", size = 60069, upload-time = "2026-05-08T21:01:33.67Z" },
+ { url = "https://files.pythonhosted.org/packages/da/68/5c6f7622d510cc666a300687e06fd060c1a43361c0c9b20d284f06d8096a/propcache-0.5.2-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:5570dbcc97571c15f68068e529c92715a12f8d54030e272d264b377e22bd17a5", size = 57099, upload-time = "2026-05-08T21:01:34.915Z" },
+ { url = "https://files.pythonhosted.org/packages/55/27/9cb0b4c679124085327957d42521c99dba04c88c90c3e55a6f0b633ebccc/propcache-0.5.2-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:f814362777a9f841adddb200ecdf8f5cb1e5a3c4b7a86378edbd6ccb26edd702", size = 63391, upload-time = "2026-05-08T21:01:36.231Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/9d/7258aaa5bdf60fc6f27591eef6fe52768cb0beda7140be477c8b12c9794a/propcache-0.5.2-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:196913dea116aeb5a2ba95af4ddcb7ea85559ae07d8eee8751688310d09168c3", size = 61626, upload-time = "2026-05-08T21:01:37.545Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/0d/41c602003e8a9b16fe1e7eadf62c7bfba9d5474370b24200bf48b315f45f/propcache-0.5.2-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:6e7b8719005dd1175be4ab1cd25e9b98659a5e0347331506ec6760d2773a7fb5", size = 64781, upload-time = "2026-05-08T21:01:38.83Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/f3/38e66b1856e9bd079deea015bc4a55f7767c0e4db2f7dcf69e7e680ba4ce/propcache-0.5.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:51f96d685ab16e88cab128cd37a52c5da540809c8b879fa047731bfcb4ad35a4", size = 62570, upload-time = "2026-05-08T21:01:40.415Z" },
+ { url = "https://files.pythonhosted.org/packages/95/ca/bbfe9b910ce57dde8bb4876b4520fc02a4e89497c10de26be936758a3aaa/propcache-0.5.2-cp314-cp314-win32.whl", hash = "sha256:cc6fc3cc62e8501d3ed62894425040d2728ecddb1ed072737a5c70bd537aa9f0", size = 39436, upload-time = "2026-05-08T21:01:41.654Z" },
+ { url = "https://files.pythonhosted.org/packages/61/d2/45c9defbaa1ea297035d9d4cce9e8f80daafbf19319c6007f157c6256ea9/propcache-0.5.2-cp314-cp314-win_amd64.whl", hash = "sha256:81e3a30b0bb60caa22033dd0f8a3618d1d67356212514f62c57db75cb0ef410c", size = 42373, upload-time = "2026-05-08T21:01:43.041Z" },
+ { url = "https://files.pythonhosted.org/packages/44/68/9ea5103f41d5217d7d6ec24db90018e23aebec070c3f9a6e54d12b841fd8/propcache-0.5.2-cp314-cp314-win_arm64.whl", hash = "sha256:0d2c9bf8528f135dbb805ce027567e09164f7efa51a2be07458a2c0420f292d0", size = 38554, upload-time = "2026-05-08T21:01:44.336Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/81/fadf555f42d3b762eea8a53950b0489fdc0aa9da5f8ed9e10ce0a4e01b48/propcache-0.5.2-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:4bc8ff1feffc6a61c7002ffe84634c41b822e104990ae009f44a0834430070bb", size = 99395, upload-time = "2026-05-08T21:01:45.883Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/c9/c61e134a686949cf7971af3a390148b1156f7be81c73bc0cd12c873e2d48/propcache-0.5.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:79aa3ff0a9b566633b642fa9caf7e21ed1c13d6feca718187873f199e1514078", size = 56653, upload-time = "2026-05-08T21:01:47.307Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/73/daf935ea7048ddd7ec8eec5345b4a40b619d2d178b3c0a0900796bc3c794/propcache-0.5.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1b31822f4474c4036bae62de9402710051d431a606d6a0f907fec79935a071aa", size = 56914, upload-time = "2026-05-08T21:01:48.573Z" },
+ { url = "https://files.pythonhosted.org/packages/79/9f/aba959b435ea18617edd7cf0a7ad0b9c574b8fc7e3d2cd55fb59cb255d33/propcache-0.5.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:13fef48778b5a2a756523fdb781326b028ca75e32858b04f2cdd19f394564917", size = 62567, upload-time = "2026-05-08T21:01:49.903Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/a1/859942de9a791ff42f6141736f5b37749b8f53e65edfa49638c67dd67e6a/propcache-0.5.2-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8b73ab70f1a3351fbc71f663b3e645af6dd0329100c353081cf69c37433fc6fe", size = 65542, upload-time = "2026-05-08T21:01:51.204Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/61/315bc0fd6c0fc7f80a528b8afd209e5fc4a875ea79571b91b8f50f442907/propcache-0.5.2-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5538d2c13d93e4698af7e092b57bc7298fd35d1d58e656ae18f23ee0d0378e03", size = 66845, upload-time = "2026-05-08T21:01:52.539Z" },
+ { url = "https://files.pythonhosted.org/packages/47/f7/9f8122e3132e8e354ac41975ef8f1099be7d5a16bc7ae562734e993665c0/propcache-0.5.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:cd645f03898405cabe694fb8bc35241e3a9c332ec85627584fe3de201452b335", size = 63985, upload-time = "2026-05-08T21:01:53.847Z" },
+ { url = "https://files.pythonhosted.org/packages/c8/54/c317819ec157cbf6f35df9df9657a6f82daf34d5faf15948b2f639c2192e/propcache-0.5.2-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a473b3440261e0c60706e732b2ed2f517857344fc21bf48fdfe211e2d98eb285", size = 63999, upload-time = "2026-05-08T21:01:55.179Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/56/387e3f7dfce0a9233df41fb888aa1c30222cb4bbbf09537c02dd9bd85fe2/propcache-0.5.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:7afa37062e6650640e932e4cc9297d81f9f42d9944029cc386b8247dea4da837", size = 62779, upload-time = "2026-05-08T21:01:57.489Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/9c/596784cb5824ed61ee960d3f8655a3f0993e107c6e98ab6c818b7fb92ccb/propcache-0.5.2-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:8a90efd5777e996e42d568db9ac740b944d691e565cbfd31b2f7832f9184b2b8", size = 59796, upload-time = "2026-05-08T21:01:58.736Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/3d/1a6cfa1726a48542c1e8784a0761421476a5b68e09b7f36bf95eb954aaba/propcache-0.5.2-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:f19bb891234d72535764d703bfed1153cc34f4214d5bd7150aee1eec9e8f4366", size = 66023, upload-time = "2026-05-08T21:02:00.228Z" },
+ { url = "https://files.pythonhosted.org/packages/e4/0e/05fd6990369477076e4e280bcb970de760fddf0161a46e988bc95f7940ec/propcache-0.5.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:32775082acd2d807ee3db715c7770d38767b817870acfa08c29e057f3c4d5b56", size = 64448, upload-time = "2026-05-08T21:02:01.888Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/86/5f8da315a4309c62c10c0b2516b17492d5d3bbe1bb862b96604db67e2a37/propcache-0.5.2-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:9282fb1a3bccd038da9f768b927b24a0c753e466c086b7c4f3c6982851eefb2d", size = 67329, upload-time = "2026-05-08T21:02:03.484Z" },
+ { url = "https://files.pythonhosted.org/packages/da/d3/3368efe79ab21f0cdf86ef49895811c9cc933131d4cde1f28a624e22e712/propcache-0.5.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:cc49723e2f60d6b32a0f0b08a3fd6d13203c07f1cd9566cfce0f12a917c967a2", size = 65172, upload-time = "2026-05-08T21:02:04.745Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/07/127e8b0bacfb325396196f9d976a22453049b89b9b2b08477cc3145faa44/propcache-0.5.2-cp314-cp314t-win32.whl", hash = "sha256:2d7aa89ebca5acc98cba9d1472d976e394782f587bad6661003602a619fd1821", size = 43813, upload-time = "2026-05-08T21:02:06.025Z" },
+ { url = "https://files.pythonhosted.org/packages/88/fb/46dad6c0ae49ed230ab1b16c890c2b6314e2403e6c412976f4a72d64a527/propcache-0.5.2-cp314-cp314t-win_amd64.whl", hash = "sha256:d447bb0b3054be5818458fbb171208b1d9ff11eba14e18ca18b90cbb45767370", size = 47764, upload-time = "2026-05-08T21:02:07.353Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/c4/a47d0a63aa309d10d59ede6e9d4cff03a344a79d1f0f4cd0cd74997b53e0/propcache-0.5.2-cp314-cp314t-win_arm64.whl", hash = "sha256:fe67a3d11cd9b4efabfa45c3d00ffba2b26811442a73a581a94b67c2b5faccf6", size = 41140, upload-time = "2026-05-08T21:02:09.065Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/ed/1cdcab6ba3d6ab7feca11fc14f0eeea80755bb53ef4e892079f31b10a25f/propcache-0.5.2-py3-none-any.whl", hash = "sha256:be1ddfcbb376e3de5d2e2db1d58d6d67463e6b4f9f040c000de8e300295465fe", size = 14036, upload-time = "2026-05-08T21:02:10.673Z" },
+]
+
+[[package]]
+name = "protobuf"
+version = "6.33.6"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/66/70/e908e9c5e52ef7c3a6c7902c9dfbb34c7e29c25d2f81ade3856445fd5c94/protobuf-6.33.6.tar.gz", hash = "sha256:a6768d25248312c297558af96a9f9c929e8c4cee0659cb07e780731095f38135", size = 444531, upload-time = "2026-03-18T19:05:00.988Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fc/9f/2f509339e89cfa6f6a4c4ff50438db9ca488dec341f7e454adad60150b00/protobuf-6.33.6-cp310-abi3-win32.whl", hash = "sha256:7d29d9b65f8afef196f8334e80d6bc1d5d4adedb449971fefd3723824e6e77d3", size = 425739, upload-time = "2026-03-18T19:04:48.373Z" },
+ { url = "https://files.pythonhosted.org/packages/76/5d/683efcd4798e0030c1bab27374fd13a89f7c2515fb1f3123efdfaa5eab57/protobuf-6.33.6-cp310-abi3-win_amd64.whl", hash = "sha256:0cd27b587afca21b7cfa59a74dcbd48a50f0a6400cfb59391340ad729d91d326", size = 437089, upload-time = "2026-03-18T19:04:50.381Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/01/a3c3ed5cd186f39e7880f8303cc51385a198a81469d53d0fdecf1f64d929/protobuf-6.33.6-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:9720e6961b251bde64edfdab7d500725a2af5280f3f4c87e57c0208376aa8c3a", size = 427737, upload-time = "2026-03-18T19:04:51.866Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/90/b3c01fdec7d2f627b3a6884243ba328c1217ed2d978def5c12dc50d328a3/protobuf-6.33.6-cp39-abi3-manylinux2014_aarch64.whl", hash = "sha256:e2afbae9b8e1825e3529f88d514754e094278bb95eadc0e199751cdd9a2e82a2", size = 324610, upload-time = "2026-03-18T19:04:53.096Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/ca/25afc144934014700c52e05103c2421997482d561f3101ff352e1292fb81/protobuf-6.33.6-cp39-abi3-manylinux2014_s390x.whl", hash = "sha256:c96c37eec15086b79762ed265d59ab204dabc53056e3443e702d2681f4b39ce3", size = 339381, upload-time = "2026-03-18T19:04:54.616Z" },
+ { url = "https://files.pythonhosted.org/packages/16/92/d1e32e3e0d894fe00b15ce28ad4944ab692713f2e7f0a99787405e43533a/protobuf-6.33.6-cp39-abi3-manylinux2014_x86_64.whl", hash = "sha256:e9db7e292e0ab79dd108d7f1a94fe31601ce1ee3f7b79e0692043423020b0593", size = 323436, upload-time = "2026-03-18T19:04:55.768Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/72/02445137af02769918a93807b2b7890047c32bfb9f90371cbc12688819eb/protobuf-6.33.6-py3-none-any.whl", hash = "sha256:77179e006c476e69bf8e8ce866640091ec42e1beb80b213c3900006ecfba6901", size = 170656, upload-time = "2026-03-18T19:04:59.826Z" },
+]
+
+[[package]]
+name = "psutil"
+version = "7.2.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/aa/c6/d1ddf4abb55e93cebc4f2ed8b5d6dbad109ecb8d63748dd2b20ab5e57ebe/psutil-7.2.2.tar.gz", hash = "sha256:0746f5f8d406af344fd547f1c8daa5f5c33dbc293bb8d6a16d80b4bb88f59372", size = 493740, upload-time = "2026-01-28T18:14:54.428Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/51/08/510cbdb69c25a96f4ae523f733cdc963ae654904e8db864c07585ef99875/psutil-7.2.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:2edccc433cbfa046b980b0df0171cd25bcaeb3a68fe9022db0979e7aa74a826b", size = 130595, upload-time = "2026-01-28T18:14:57.293Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/f5/97baea3fe7a5a9af7436301f85490905379b1c6f2dd51fe3ecf24b4c5fbf/psutil-7.2.2-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:e78c8603dcd9a04c7364f1a3e670cea95d51ee865e4efb3556a3a63adef958ea", size = 131082, upload-time = "2026-01-28T18:14:59.732Z" },
+ { url = "https://files.pythonhosted.org/packages/37/d6/246513fbf9fa174af531f28412297dd05241d97a75911ac8febefa1a53c6/psutil-7.2.2-cp313-cp313t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1a571f2330c966c62aeda00dd24620425d4b0cc86881c89861fbc04549e5dc63", size = 181476, upload-time = "2026-01-28T18:15:01.884Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/b5/9182c9af3836cca61696dabe4fd1304e17bc56cb62f17439e1154f225dd3/psutil-7.2.2-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:917e891983ca3c1887b4ef36447b1e0873e70c933afc831c6b6da078ba474312", size = 184062, upload-time = "2026-01-28T18:15:04.436Z" },
+ { url = "https://files.pythonhosted.org/packages/16/ba/0756dca669f5a9300d0cbcbfae9a4c30e446dfc7440ffe43ded5724bfd93/psutil-7.2.2-cp313-cp313t-win_amd64.whl", hash = "sha256:ab486563df44c17f5173621c7b198955bd6b613fb87c71c161f827d3fb149a9b", size = 139893, upload-time = "2026-01-28T18:15:06.378Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/61/8fa0e26f33623b49949346de05ec1ddaad02ed8ba64af45f40a147dbfa97/psutil-7.2.2-cp313-cp313t-win_arm64.whl", hash = "sha256:ae0aefdd8796a7737eccea863f80f81e468a1e4cf14d926bd9b6f5f2d5f90ca9", size = 135589, upload-time = "2026-01-28T18:15:08.03Z" },
+ { url = "https://files.pythonhosted.org/packages/81/69/ef179ab5ca24f32acc1dac0c247fd6a13b501fd5534dbae0e05a1c48b66d/psutil-7.2.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:eed63d3b4d62449571547b60578c5b2c4bcccc5387148db46e0c2313dad0ee00", size = 130664, upload-time = "2026-01-28T18:15:09.469Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/64/665248b557a236d3fa9efc378d60d95ef56dd0a490c2cd37dafc7660d4a9/psutil-7.2.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7b6d09433a10592ce39b13d7be5a54fbac1d1228ed29abc880fb23df7cb694c9", size = 131087, upload-time = "2026-01-28T18:15:11.724Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/2e/e6782744700d6759ebce3043dcfa661fb61e2fb752b91cdeae9af12c2178/psutil-7.2.2-cp314-cp314t-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fa4ecf83bcdf6e6c8f4449aff98eefb5d0604bf88cb883d7da3d8d2d909546a", size = 182383, upload-time = "2026-01-28T18:15:13.445Z" },
+ { url = "https://files.pythonhosted.org/packages/57/49/0a41cefd10cb7505cdc04dab3eacf24c0c2cb158a998b8c7b1d27ee2c1f5/psutil-7.2.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e452c464a02e7dc7822a05d25db4cde564444a67e58539a00f929c51eddda0cf", size = 185210, upload-time = "2026-01-28T18:15:16.002Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/2c/ff9bfb544f283ba5f83ba725a3c5fec6d6b10b8f27ac1dc641c473dc390d/psutil-7.2.2-cp314-cp314t-win_amd64.whl", hash = "sha256:c7663d4e37f13e884d13994247449e9f8f574bc4655d509c3b95e9ec9e2b9dc1", size = 141228, upload-time = "2026-01-28T18:15:18.385Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/fc/f8d9c31db14fcec13748d373e668bc3bed94d9077dbc17fb0eebc073233c/psutil-7.2.2-cp314-cp314t-win_arm64.whl", hash = "sha256:11fe5a4f613759764e79c65cf11ebdf26e33d6dd34336f8a337aa2996d71c841", size = 136284, upload-time = "2026-01-28T18:15:19.912Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/36/5ee6e05c9bd427237b11b3937ad82bb8ad2752d72c6969314590dd0c2f6e/psutil-7.2.2-cp36-abi3-macosx_10_9_x86_64.whl", hash = "sha256:ed0cace939114f62738d808fdcecd4c869222507e266e574799e9c0faa17d486", size = 129090, upload-time = "2026-01-28T18:15:22.168Z" },
+ { url = "https://files.pythonhosted.org/packages/80/c4/f5af4c1ca8c1eeb2e92ccca14ce8effdeec651d5ab6053c589b074eda6e1/psutil-7.2.2-cp36-abi3-macosx_11_0_arm64.whl", hash = "sha256:1a7b04c10f32cc88ab39cbf606e117fd74721c831c98a27dc04578deb0c16979", size = 129859, upload-time = "2026-01-28T18:15:23.795Z" },
+ { url = "https://files.pythonhosted.org/packages/b5/70/5d8df3b09e25bce090399cf48e452d25c935ab72dad19406c77f4e828045/psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:076a2d2f923fd4821644f5ba89f059523da90dc9014e85f8e45a5774ca5bc6f9", size = 155560, upload-time = "2026-01-28T18:15:25.976Z" },
+ { url = "https://files.pythonhosted.org/packages/63/65/37648c0c158dc222aba51c089eb3bdfa238e621674dc42d48706e639204f/psutil-7.2.2-cp36-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b0726cecd84f9474419d67252add4ac0cd9811b04d61123054b9fb6f57df6e9e", size = 156997, upload-time = "2026-01-28T18:15:27.794Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/13/125093eadae863ce03c6ffdbae9929430d116a246ef69866dad94da3bfbc/psutil-7.2.2-cp36-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:fd04ef36b4a6d599bbdb225dd1d3f51e00105f6d48a28f006da7f9822f2606d8", size = 148972, upload-time = "2026-01-28T18:15:29.342Z" },
+ { url = "https://files.pythonhosted.org/packages/04/78/0acd37ca84ce3ddffaa92ef0f571e073faa6d8ff1f0559ab1272188ea2be/psutil-7.2.2-cp36-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b58fabe35e80b264a4e3bb23e6b96f9e45a3df7fb7eed419ac0e5947c61e47cc", size = 148266, upload-time = "2026-01-28T18:15:31.597Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/90/e2159492b5426be0c1fef7acba807a03511f97c5f86b3caeda6ad92351a7/psutil-7.2.2-cp37-abi3-win_amd64.whl", hash = "sha256:eb7e81434c8d223ec4a219b5fc1c47d0417b12be7ea866e24fb5ad6e84b3d988", size = 137737, upload-time = "2026-01-28T18:15:33.849Z" },
+ { url = "https://files.pythonhosted.org/packages/8c/c7/7bb2e321574b10df20cbde462a94e2b71d05f9bbda251ef27d104668306a/psutil-7.2.2-cp37-abi3-win_arm64.whl", hash = "sha256:8c233660f575a5a89e6d4cb65d9f938126312bca76d8fe087b947b3a1aaac9ee", size = 134617, upload-time = "2026-01-28T18:15:36.514Z" },
+]
+
+[[package]]
+name = "ptyprocess"
+version = "0.7.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/20/e5/16ff212c1e452235a90aeb09066144d0c5a6a8c0834397e03f5224495c4e/ptyprocess-0.7.0.tar.gz", hash = "sha256:5c5d0a3b48ceee0b48485e0c26037c0acd7d29765ca3fbb5cb3831d347423220", size = 70762, upload-time = "2020-12-28T15:15:30.155Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl", hash = "sha256:4b41f3967fce3af57cc7e94b888626c18bf37a083e3651ca8feeb66d492fef35", size = 13993, upload-time = "2020-12-28T15:15:28.35Z" },
+]
+
+[[package]]
+name = "pure-eval"
+version = "0.2.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cd/05/0a34433a064256a578f1783a10da6df098ceaa4a57bbeaa96a6c0352786b/pure_eval-0.2.3.tar.gz", hash = "sha256:5f4e983f40564c576c7c8635ae88db5956bb2229d7e9237d03b3c0b0190eaf42", size = 19752, upload-time = "2024-07-21T12:58:21.801Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl", hash = "sha256:1db8e35b67b3d218d818ae653e27f06c3aa420901fa7b081ca98cbedc874e0d0", size = 11842, upload-time = "2024-07-21T12:58:20.04Z" },
+]
+
+[[package]]
+name = "pybind11"
+version = "3.0.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/cc/f0/35145a3c3baffeef55d4b8324caa33abaa8fa56ab345ecd4b2211d09163e/pybind11-3.0.4.tar.gz", hash = "sha256:3286b59c8a774b9ee650169302dd5a4eedc30a8617905a0560dd8ee44775130c", size = 589533, upload-time = "2026-04-19T03:08:15.925Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b3/06/c3a23c9a0263b136c519f033a58d4641e73065fefc7754e9667ec206d992/pybind11-3.0.4-py3-none-any.whl", hash = "sha256:961720ee652da51d531b7b2451a6bd2bc042b0106e6d9baa48ecb7d58034ce63", size = 314166, upload-time = "2026-04-19T03:08:14.091Z" },
+]
+
+[[package]]
+name = "pycparser"
+version = "3.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1b/7d/92392ff7815c21062bea51aa7b87d45576f649f16458d78b7cf94b9ab2e6/pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29", size = 103492, upload-time = "2026-01-21T14:26:51.89Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0c/c3/44f3fbbfa403ea2a7c779186dc20772604442dde72947e7d01069cbe98e3/pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992", size = 48172, upload-time = "2026-01-21T14:26:50.693Z" },
+]
+
+[[package]]
+name = "pydantic"
+version = "2.13.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "annotated-types" },
+ { name = "pydantic-core" },
+ { name = "typing-extensions" },
+ { name = "typing-inspection" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/18/a5/b60d21ac674192f8ab0ba4e9fd860690f9b4a6e51ca5df118733b487d8d6/pydantic-2.13.4.tar.gz", hash = "sha256:c40756b57adaa8b1efeeced5c196f3f3b7c435f90e84ea7f443901bec8099ef6", size = 844775, upload-time = "2026-05-06T13:43:05.343Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl", hash = "sha256:45a282cde31d808236fd7ea9d919b128653c8b38b393d1c4ab335c62924d9aba", size = 472262, upload-time = "2026-05-06T13:43:02.641Z" },
+]
+
+[[package]]
+name = "pydantic-core"
+version = "2.46.4"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/9d/56/921726b776ace8d8f5db44c4ef961006580d91dc52b803c489fafd1aa249/pydantic_core-2.46.4.tar.gz", hash = "sha256:62f875393d7f270851f20523dd2e29f082bcc82292d66db2b64ea71f64b6e1c1", size = 471464, upload-time = "2026-05-06T13:37:06.98Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ce/8c/af022f0af448d7747c5154288d46b5f2bc5f17366eaa0e23e9aa04d59f3b/pydantic_core-2.46.4-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:3245406455a5d98187ec35530fd772b1d799b26667980872c8d4614991e2c4a2", size = 2106158, upload-time = "2026-05-06T13:38:57.215Z" },
+ { url = "https://files.pythonhosted.org/packages/19/95/6195171e385007300f0f5574592e467c568becce2d937a0b6804f218bc49/pydantic_core-2.46.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:962ccbab7b642487b1d8b7df90ef677e03134cf1fd8880bf698649b22a69371f", size = 1951724, upload-time = "2026-05-06T13:37:02.697Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/bc/f47d1ff9cbb1620e1b5b697eef06010035735f07820180e74178226b27b3/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8233f2947cf85404441fd7e0085f53b10c93e0ee78611099b5c7237e36aacbf7", size = 1975742, upload-time = "2026-05-06T13:37:09.448Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/11/9b9a5b0306345664a2da6410877af6e8082481b5884b3ddd78d47c6013ce/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3a233125ac121aa3ffba9a2b59edfc4a985a76092dc8279586ab4b71390875e7", size = 2052418, upload-time = "2026-05-06T13:37:38.234Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/b7/a65fec226f5d78fc39f4a13c4cc0c768c22b113438f60c14adc9d2865038/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5b712b53160b79a5850310b912a5ef8e57e56947c8ad690c227f5c9d7e561712", size = 2232274, upload-time = "2026-05-06T13:38:27.753Z" },
+ { url = "https://files.pythonhosted.org/packages/68/f0/92039db98b907ef49269a8271f67db9cb78ae2fc68062ef7e4e77adb5f61/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9401557acd873c3a7f3eb9383edef8ac4968f9510e340f4808d427e75667e7b4", size = 2309940, upload-time = "2026-05-06T13:38:05.353Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/97/2aab507d3d00ca626e8e57c1eac6a79e4e5fbcc63eb99733ff55d1717f65/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:926c9541b14b12b1681dca8a0b75feb510b06c6341b70a8e500c2fdcff837cce", size = 2094516, upload-time = "2026-05-06T13:39:10.577Z" },
+ { url = "https://files.pythonhosted.org/packages/22/37/a8aca44d40d737dde2bc05b3c6c07dff0de07ce6f82e9f3167aeaf4d5dea/pydantic_core-2.46.4-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:56cb4851bcaf3d117eddcef4fe66afd750a50274b0da8e22be256d10e5611987", size = 2136854, upload-time = "2026-05-06T13:40:22.59Z" },
+ { url = "https://files.pythonhosted.org/packages/24/99/fcef1b79238c06a8cbec70819ac722ba76e02bc8ada9b0fd66eba40da01b/pydantic_core-2.46.4-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c68fcd102d71ea85c5b2dfac3f4f8476eff42a9e078fd5faefff6d145063536b", size = 2180306, upload-time = "2026-05-06T13:40:10.666Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/6c/fc44000918855b42779d007ae63b0532794739027b2f417321cddbc44f6a/pydantic_core-2.46.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:b2f69dec1725e79a012d920df1707de5caf7ed5e08f3be4435e25803efc47458", size = 2190044, upload-time = "2026-05-06T13:40:43.231Z" },
+ { url = "https://files.pythonhosted.org/packages/6b/65/d9cadc9f1920d7a127ad2edba16c1db7916e59719285cd6c94600b0080ba/pydantic_core-2.46.4-cp312-cp312-musllinux_1_1_armv7l.whl", hash = "sha256:8d0820e8192167f80d88d64038e609c31452eeca865b4e1d9950a27a4609b00b", size = 2329133, upload-time = "2026-05-06T13:39:57.365Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/cf/c873d91679f3a30bcf5e7ac280ce5573483e72295307685120d0d5ad3416/pydantic_core-2.46.4-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:fbdb89b3e1c94a30cc5edfce477c6e6a5dc4d8f84665b455c27582f211a1c72c", size = 2374464, upload-time = "2026-05-06T13:38:06.976Z" },
+ { url = "https://files.pythonhosted.org/packages/47/bd/6f2fc8188f31bf10590f1e98e7b306336161fac930a8c514cd7bd828c7dc/pydantic_core-2.46.4-cp312-cp312-win32.whl", hash = "sha256:9aa768456404a8bf48a4406685ac2bec8e72b62c69313734fa3b73cf33b3a894", size = 1974823, upload-time = "2026-05-06T13:40:47.985Z" },
+ { url = "https://files.pythonhosted.org/packages/40/8c/985c1d41ea1107c2534abd9870e4ed5c8e7669b5c308297835c001e7a1c4/pydantic_core-2.46.4-cp312-cp312-win_amd64.whl", hash = "sha256:e9c26f834c65f5752f3f06cb08cb86a913ceb7274d0db6e267808a708b46bc89", size = 2072919, upload-time = "2026-05-06T13:39:21.153Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/ba/f463d006e0c47373ca7ec5e1a261c59dc01ef4d62b2657af925fb0deee3a/pydantic_core-2.46.4-cp312-cp312-win_arm64.whl", hash = "sha256:4fc73cb559bdb54b1134a706a2802a4cddd27a0633f5abb7e53056268751ac6a", size = 2027604, upload-time = "2026-05-06T13:39:03.753Z" },
+ { url = "https://files.pythonhosted.org/packages/51/a2/5d30b469c5267a17b39dec53208222f76a8d351dfac4af661888c5aee77d/pydantic_core-2.46.4-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:5d5902252db0d3cedf8d4a1bc68f70eeb430f7e4c7104c8c476753519b423008", size = 2106306, upload-time = "2026-05-06T13:37:48.029Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/81/4fa520eaffa8bd7d1525e644cd6d39e7d60b1592bc5b516693c7340b50f1/pydantic_core-2.46.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:c94f0688e7b8d0a67abf40e57a7eaaecd17cc9586706a31b76c031f63df052b4", size = 1951906, upload-time = "2026-05-06T13:37:17.012Z" },
+ { url = "https://files.pythonhosted.org/packages/03/d5/fd02da45b659668b05923b17ba3a0100a0a3d5541e3bd8fcc4ecb711309e/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f027324c56cd5406ca49c124b0db10e56c69064fec039acc571c29020cc87c76", size = 1976802, upload-time = "2026-05-06T13:37:35.113Z" },
+ { url = "https://files.pythonhosted.org/packages/21/f2/95727e1368be3d3ed485eaab7adbd7dda408f33f7a36e8b48e0144002b91/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e739fee756ba1010f8bcccb534252e85a35fe45ae92c295a06059ce58b74ccd3", size = 2052446, upload-time = "2026-05-06T13:37:12.313Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/86/5d99feea3f77c7234b8718075b23db11532773c1a0dbd9b9490215dc2eeb/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9d56801be94b86a9da183e5f3766e6310752b99ff647e38b09a9500d88e46e76", size = 2232757, upload-time = "2026-05-06T13:39:01.149Z" },
+ { url = "https://files.pythonhosted.org/packages/d2/3a/508ac615935ef7588cf6d9e9b91309fdc2da751af865e02a9098de88258c/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2412e734dcb48da14d4e4006b82b46b74f2518b8a26ee7e58c6844a6cd6d03c4", size = 2309275, upload-time = "2026-05-06T13:37:41.406Z" },
+ { url = "https://files.pythonhosted.org/packages/07/f8/41db9de19d7987d6b04715a02b3b40aea467000275d9d758ffaa31af7d50/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9551187363ffc0de2a00b2e47c25aeaeb1020b69b668762966df15fc5659dd5a", size = 2094467, upload-time = "2026-05-06T13:39:18.847Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/e2/f35033184cb11d0052daf4416e8e10a502ea2ac006fc4f459aee872727d1/pydantic_core-2.46.4-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:0186750b482eefa11d7f435892b09c5c606193ef3375bcf94aa00ae6bfb66262", size = 2134417, upload-time = "2026-05-06T13:40:17.944Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/7b/6ceeb1cc90e193862f444ebe373d8fdf613f0a82572dde03fb10734c6c71/pydantic_core-2.46.4-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5855698a4856556d86e8e6cd8434bc3ac0314ee8e12089ae0e143f64c6256e4e", size = 2179782, upload-time = "2026-05-06T13:40:32.618Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/f2/c8d7773ede6af08036423a00ae0ceffce266c3c52a096c435d68c896083f/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:cbaf13819775b7f769bf4a1f066cb6df7a28d4480081a589828ef190226881cd", size = 2188782, upload-time = "2026-05-06T13:36:51.018Z" },
+ { url = "https://files.pythonhosted.org/packages/59/31/0c864784e31f09f05cdd87606f08923b9c9e7f6e51dd27f20f62f975ce9f/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_armv7l.whl", hash = "sha256:633147d34cf4550417f12e2b1a0383973bdf5cdfde212cb09e9a581cf10820be", size = 2328334, upload-time = "2026-05-06T13:40:37.764Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/eb/4f6c8a41efa30baa755590f4141abf3a8c370fab610915733e74134a7270/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:82cf5301172168103724d49a1444d3378cb20cdee30b116a1bd6031236298a5d", size = 2372986, upload-time = "2026-05-06T13:39:34.152Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/24/b375a480d53113860c299764bfe9f349a3dc9108b3adc0d7f0d786492ebf/pydantic_core-2.46.4-cp313-cp313-win32.whl", hash = "sha256:9fa8ae11da9e2b3126c6426f147e0fba88d96d65921799bb30c6abd1cb2c97fb", size = 1973693, upload-time = "2026-05-06T13:37:55.072Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/e8/cff247591966f2d22ec8c003cd7587e27b7ba7b81ab2fb888e3ab75dc285/pydantic_core-2.46.4-cp313-cp313-win_amd64.whl", hash = "sha256:6b3ace8194b0e5204818c92802dcdca7fc6d88aabbb799d7c795540d9cd6d292", size = 2071819, upload-time = "2026-05-06T13:38:49.139Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/1a/f4aee670d5670e9e148e0c82c7db98d780be566c6e6a97ee8035528ca0b3/pydantic_core-2.46.4-cp313-cp313-win_arm64.whl", hash = "sha256:184c081504d17f1c1066e430e117142b2c77d9448a97f7b65c6ac9fd9aee238d", size = 2027411, upload-time = "2026-05-06T13:40:45.796Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/74/228a26ddad29c6672b805d9fd78e8d251cd04004fa7eed0e622096cd0250/pydantic_core-2.46.4-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:428e04521a40150c85216fc8b85e8d39fece235a9cf5e383761238c7fa9b96fb", size = 2102079, upload-time = "2026-05-06T13:38:41.019Z" },
+ { url = "https://files.pythonhosted.org/packages/ad/1f/8970b150a4b4365623ae00fc88603491f763c627311ae8031e3111356d6e/pydantic_core-2.46.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:23ace664830ee0bfe014a0c7bc248b1f7f25ed7ad103852c317624a1083af462", size = 1952179, upload-time = "2026-05-06T13:36:59.812Z" },
+ { url = "https://files.pythonhosted.org/packages/95/30/5211a831ae054928054b2f79731661087a2bc5c01e825c672b3a4a8f1b3e/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ce5c1d2a8b27468f433ca974829c44060b8097eedc39933e3c206a90ee49c4a9", size = 1978926, upload-time = "2026-05-06T13:37:39.933Z" },
+ { url = "https://files.pythonhosted.org/packages/57/e9/689668733b1eb67adeef047db3c2e8788fcf65a7fd9c9e2b46b7744fe245/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:7283d57845ecf5a163403eb0702dfc220cc4fbdd18919cb5ccea4f95ee1cdab4", size = 2046785, upload-time = "2026-05-06T13:38:01.995Z" },
+ { url = "https://files.pythonhosted.org/packages/60/d9/6715260422ff50a2109878fd24d948a6c3446bb2664f34ee78cd972b3acd/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8daafc69c93ee8a0204506a3b6b30f586ef54028f52aeeeb5c4cfc5184fd5914", size = 2228733, upload-time = "2026-05-06T13:40:50.371Z" },
+ { url = "https://files.pythonhosted.org/packages/18/ae/fdb2f64316afca925640f8e70bb1a564b0ec2721c1389e25b8eb4bf9a299/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cd2213145bcc2ba85884d0ac63d222fece9209678f77b9b4d76f054c561adb28", size = 2307534, upload-time = "2026-05-06T13:37:21.531Z" },
+ { url = "https://files.pythonhosted.org/packages/89/1d/8eff589b45bb8190a9d12c49cfad0f176a5cbd1534908a6b5125e2886239/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7a5f930472650a82629163023e630d160863fce524c616f4e5186e5de9d9a49b", size = 2099732, upload-time = "2026-05-06T13:39:31.942Z" },
+ { url = "https://files.pythonhosted.org/packages/06/d5/ee5a3366637fee41dee51a1fc91562dcf12ddbc68fda34e6b253da2324bb/pydantic_core-2.46.4-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:c1b3f518abeca3aa13c712fd202306e145abf59a18b094a6bafb2d2bbf59192c", size = 2129627, upload-time = "2026-05-06T13:37:25.033Z" },
+ { url = "https://files.pythonhosted.org/packages/94/33/2414be571d2c6a6c4d08be21f9292b6d3fdb08949a97b6dfe985017821db/pydantic_core-2.46.4-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1a7dd0b3ee80d90150e3495a3a13ac34dbcbfd4f012996a6a1d8900e91b5c0fb", size = 2179141, upload-time = "2026-05-06T13:37:14.046Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/79/7daa95be995be0eecc4cf75064cb33f9bbbfe3fe0158caf2f0d4a996a5c7/pydantic_core-2.46.4-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:3fb702cd90b0446a3a1c5e470bfa0dd23c0233b676a9099ddcc964fa6ca13898", size = 2184325, upload-time = "2026-05-06T13:36:53.615Z" },
+ { url = "https://files.pythonhosted.org/packages/9f/cb/d0a382f5c0de8a222dc61c65348e0ce831b1f68e0a018450d31c2cace3a5/pydantic_core-2.46.4-cp314-cp314-musllinux_1_1_armv7l.whl", hash = "sha256:b8458003118a712e66286df6a707db01c52c0f52f7db8e4a38f0da1d3b94fc4e", size = 2323990, upload-time = "2026-05-06T13:40:29.971Z" },
+ { url = "https://files.pythonhosted.org/packages/05/db/d9ba624cc4a5aced1598e88c04fdbd8310c8a69b9d38b9a3d39ce3a61ed7/pydantic_core-2.46.4-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:372429a130e469c9cd698925ce5fc50940b7a1336b0d82038e63d5bbc4edc519", size = 2369978, upload-time = "2026-05-06T13:37:23.027Z" },
+ { url = "https://files.pythonhosted.org/packages/f2/20/d15df15ba918c423461905802bfd2981c3af0bfa0e40d05e13edbfa48bc3/pydantic_core-2.46.4-cp314-cp314-win32.whl", hash = "sha256:85bb3611ff1802f3ee7fdd7dbff26b56f343fb432d57a4728fdd49b6ef35e2f4", size = 1966354, upload-time = "2026-05-06T13:38:03.499Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/b6/6b8de4c0a7d7ab3004c439c80c5c1e0a3e8d78bbae19379b01960383d9e5/pydantic_core-2.46.4-cp314-cp314-win_amd64.whl", hash = "sha256:811ff8e9c313ab425368bcbb36e5c4ebd7108c2bbf4e4089cfbb0b01eff63fac", size = 2072238, upload-time = "2026-05-06T13:39:40.807Z" },
+ { url = "https://files.pythonhosted.org/packages/32/36/51eb763beec1f4cf59b1db243a7dcc39cbb41230f050a09b9d69faaf0a48/pydantic_core-2.46.4-cp314-cp314-win_arm64.whl", hash = "sha256:bfec22eab3c8cc2ceec0248aec886624116dc079afa027ecc8ad4a7e62010f8a", size = 2018251, upload-time = "2026-05-06T13:37:26.72Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/91/855af51d625b23aa987116a19e231d2aaef9c4a415273ddc189b79a45fee/pydantic_core-2.46.4-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:af8244b2bef6aaad6d92cda81372de7f8c8d36c9f0c3ea36e827c60e7d9467a0", size = 2099593, upload-time = "2026-05-06T13:39:47.682Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/1b/8784a54c65edb5f49f0a14d6977cf1b209bba85a4c77445b255c2de58ab3/pydantic_core-2.46.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:5a4330cdbc57162e4b3aa303f588ba752257694c9c9be3e7ebb11b4aca659b5d", size = 1935226, upload-time = "2026-05-06T13:40:40.428Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/e7/1955d28d1afc56dd4b3ad7cc0cf39df1b9852964cf16e5d13912756d6d6b/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:29c61fc04a3d840155ff08e475a04809278972fe6aef51e2720554e96367e34b", size = 1974605, upload-time = "2026-05-06T13:37:32.029Z" },
+ { url = "https://files.pythonhosted.org/packages/93/e2/3fedbf0ba7a22850e6e9fd78117f1c0f10f950182344d8a6c535d468fdd8/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:c50f2528cf200c5eed56faf3f4e22fcd5f38c157a8b78576e6ba3168ec35f000", size = 2030777, upload-time = "2026-05-06T13:38:55.239Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/61/46be275fcaaba0b4f5b9669dd852267ce1ff616592dccf7a7845588df091/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0cbe8b01f948de4286c74cdd6c667aceb38f5c1e26f0693b3983d9d74887c65e", size = 2236641, upload-time = "2026-05-06T13:37:08.096Z" },
+ { url = "https://files.pythonhosted.org/packages/60/db/12e93e46a8bac9988be3c016860f83293daea8c716c029c9ace279036f2f/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:617d7e2ca7dcb8c5cf6bcb8c59b8832c94b36196bbf1cbd1bfb56ed341905edd", size = 2286404, upload-time = "2026-05-06T13:40:20.221Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/4a/4d8b19008f38d31c53b8219cfedc2e3d5de5fe99d90076b7e767de29274f/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7027560ee92211647d0d34e3f7cd6f50da56399d26a9c8ad0da286d3869a53f3", size = 2109219, upload-time = "2026-05-06T13:38:12.153Z" },
+ { url = "https://files.pythonhosted.org/packages/88/70/3cbc40978fefb7bb09c6708d40d4ad1a5d70fd7213c3d17f971de868ec1f/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:f99626688942fb746e545232e7726926f3be91b5975f8b55327665fafda991c7", size = 2110594, upload-time = "2026-05-06T13:40:02.971Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/20/b8d36736216e29491125531685b2f9e61aa5b4b2599893f8268551da3338/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:fc3e9034a63de20e15e8ade85358bc6efc614008cab72898b4b4952bea0509ff", size = 2159542, upload-time = "2026-05-06T13:39:27.506Z" },
+ { url = "https://files.pythonhosted.org/packages/1d/a2/367df868eb584dacf6bf82a389272406d7178e301c4ac82545ab98bc2dd9/pydantic_core-2.46.4-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:97e7cf2be5c77b7d1a9713a05605d49460d02c6078d38d8bef3cbe323c548424", size = 2168146, upload-time = "2026-05-06T13:38:31.93Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/b8/4460f77f7e201893f649a29ab355dddd3beee8a97bcb1a320db414f9a06e/pydantic_core-2.46.4-cp314-cp314t-musllinux_1_1_armv7l.whl", hash = "sha256:3bf92c5d0e00fefaab325a4d27828fe6b6e2a21848686b5b60d2d9eeb09d76c6", size = 2306309, upload-time = "2026-05-06T13:37:44.717Z" },
+ { url = "https://files.pythonhosted.org/packages/64/c4/be2639293acd87dc8ddbcec41a73cee9b2ebf996fe6d892a1a74e88ad3f7/pydantic_core-2.46.4-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:3ecbc122d18468d06ca279dc26a8c2e2d5acb10943bb35e36ae92096dc3b5565", size = 2369736, upload-time = "2026-05-06T13:37:05.645Z" },
+ { url = "https://files.pythonhosted.org/packages/30/a6/9f9f380dbb301f67023bf8f707aaa75daadf84f7152d95c410fd7e81d994/pydantic_core-2.46.4-cp314-cp314t-win32.whl", hash = "sha256:e846ae7835bf0703ae43f534ab79a867146dadd59dc9ca5c8b53d5c8f7c9ef02", size = 1955575, upload-time = "2026-05-06T13:38:51.116Z" },
+ { url = "https://files.pythonhosted.org/packages/40/1f/f1eb9eb350e795d1af8586289746f5c5677d16043040d63710e22abc43c9/pydantic_core-2.46.4-cp314-cp314t-win_amd64.whl", hash = "sha256:2108ba5c1c1eca18030634489dc544844144ee36357f2f9f780b93e7ddbb44b5", size = 2051624, upload-time = "2026-05-06T13:38:21.672Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/d2/42dd53d0a85c27606f316d3aa5d2869c4e8470a5ed6dec30e4a1abe19192/pydantic_core-2.46.4-cp314-cp314t-win_arm64.whl", hash = "sha256:4fcbe087dbc2068af7eda3aa87634eba216dbda64d1ae73c8684b621d33f6596", size = 2017325, upload-time = "2026-05-06T13:40:52.723Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/1d/8987ad40f65ae1432753072f214fb5c74fe47ffbd0698bb9cbbb585664f8/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:1d8ba486450b14f3b1d63bc521d410ec7565e52f887b9fb671791886436a42f7", size = 2095527, upload-time = "2026-05-06T13:39:52.283Z" },
+ { url = "https://files.pythonhosted.org/packages/64/d3/84c282a7eee1d3ac4c0377546ef5a1ea436ce26840d9ac3b7ed54a377507/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:3009f12e4e90b7f88b4f9adb1b0c4a3d58fe7820f3238c190047209d148026df", size = 1936024, upload-time = "2026-05-06T13:40:15.671Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/ca/eac61596cdeb4d7e174d3dc0bd8a6238f14f75f97a24e7b7db4c7e7340a0/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ad785e92e6dc634c21555edc8bd6b64957ab844541bcb96a1366c202951ae526", size = 1990696, upload-time = "2026-05-06T13:38:34.717Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/c3/7c8b240552251faf6b3a957db200fcfbbcec36763c050428b601e0c9b83b/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00c603d540afdd6b80eb39f078f33ebd46211f02f33e34a32d9f053bba711de0", size = 2147590, upload-time = "2026-05-06T13:39:29.883Z" },
+]
+
+[[package]]
+name = "pydata-sphinx-theme"
+version = "0.16.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "accessible-pygments" },
+ { name = "babel" },
+ { name = "beautifulsoup4" },
+ { name = "docutils" },
+ { name = "pygments" },
+ { name = "sphinx" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/00/20/bb50f9de3a6de69e6abd6b087b52fa2418a0418b19597601605f855ad044/pydata_sphinx_theme-0.16.1.tar.gz", hash = "sha256:a08b7f0b7f70387219dc659bff0893a7554d5eb39b59d3b8ef37b8401b7642d7", size = 2412693, upload-time = "2024-12-17T10:53:39.537Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e2/0d/8ba33fa83a7dcde13eb3c1c2a0c1cc29950a048bfed6d9b0d8b6bd710b4c/pydata_sphinx_theme-0.16.1-py3-none-any.whl", hash = "sha256:225331e8ac4b32682c18fcac5a57a6f717c4e632cea5dd0e247b55155faeccde", size = 6723264, upload-time = "2024-12-17T10:53:35.645Z" },
+]
+
+[[package]]
+name = "pygments"
+version = "2.20.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" },
+]
+
+[[package]]
+name = "pynndescent"
+version = "0.6.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "joblib" },
+ { name = "llvmlite" },
+ { name = "numba" },
+ { name = "scikit-learn" },
+ { name = "scipy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/4a/fb/7f58c397fb31666756457ee2ac4c0289ef2daad57f4ae4be8dec12f80b03/pynndescent-0.6.0.tar.gz", hash = "sha256:7ffde0fb5b400741e055a9f7d377e3702e02250616834231f6c209e39aac24f5", size = 2992987, upload-time = "2026-01-08T21:29:58.943Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b2/e6/94145d714402fd5ade00b5661f2d0ab981219e07f7db9bfa16786cdb9c04/pynndescent-0.6.0-py3-none-any.whl", hash = "sha256:dc8c74844e4c7f5cbd1e0cd6909da86fdc789e6ff4997336e344779c3d5538ef", size = 73511, upload-time = "2026-01-08T21:29:57.306Z" },
+]
+
+[[package]]
+name = "pyparsing"
+version = "3.3.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f3/91/9c6ee907786a473bf81c5f53cf703ba0957b23ab84c264080fb5a450416f/pyparsing-3.3.2.tar.gz", hash = "sha256:c777f4d763f140633dcb6d8a3eda953bf7a214dc4eff598413c070bcdc117cbc", size = 6851574, upload-time = "2026-01-21T03:57:59.36Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl", hash = "sha256:850ba148bd908d7e2411587e247a1e4f0327839c40e2e5e6d05a007ecc69911d", size = 122781, upload-time = "2026-01-21T03:57:55.912Z" },
+]
+
+[[package]]
+name = "pytest"
+version = "9.1.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+ { name = "iniconfig" },
+ { name = "packaging" },
+ { name = "pluggy" },
+ { name = "pygments" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e4/47/b9efed96c114afcfa3c9d3fe98a76a1d14c74a9e266d397cf6eb64be5e01/pytest-9.1.1.tar.gz", hash = "sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313", size = 1636369, upload-time = "2026-06-19T10:58:32.857Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/24/25/1de2678b631f5a49215c6c96fff41ba892b0a34df68d6d80292b1b48aa7f/pytest-9.1.1-py3-none-any.whl", hash = "sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c", size = 386536, upload-time = "2026-06-19T10:58:31.347Z" },
+]
+
+[[package]]
+name = "pytest-cov"
+version = "7.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "coverage" },
+ { name = "pluggy" },
+ { name = "pytest" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/b1/51/a849f96e117386044471c8ec2bd6cfebacda285da9525c9106aeb28da671/pytest_cov-7.1.0.tar.gz", hash = "sha256:30674f2b5f6351aa09702a9c8c364f6a01c27aae0c1366ae8016160d1efc56b2", size = 55592, upload-time = "2026-03-21T20:11:16.284Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9d/7a/d968e294073affff457b041c2be9868a40c1c71f4a35fcc1e45e5493067b/pytest_cov-7.1.0-py3-none-any.whl", hash = "sha256:a0461110b7865f9a271aa1b51e516c9a95de9d696734a2f71e3e78f46e1d4678", size = 22876, upload-time = "2026-03-21T20:11:14.438Z" },
+]
+
+[[package]]
+name = "pytest-xdist"
+version = "3.8.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "execnet" },
+ { name = "pytest" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/78/b4/439b179d1ff526791eb921115fca8e44e596a13efeda518b9d845a619450/pytest_xdist-3.8.0.tar.gz", hash = "sha256:7e578125ec9bc6050861aa93f2d59f1d8d085595d6551c2c90b6f4fad8d3a9f1", size = 88069, upload-time = "2025-07-01T13:30:59.346Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ca/31/d4e37e9e550c2b92a9cbc2e4d0b7420a27224968580b5a447f420847c975/pytest_xdist-3.8.0-py3-none-any.whl", hash = "sha256:202ca578cfeb7370784a8c33d6d05bc6e13b4f25b5053c30a152269fd10f0b88", size = 46396, upload-time = "2025-07-01T13:30:56.632Z" },
+]
+
+[[package]]
+name = "python-dateutil"
+version = "2.9.0.post0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "six" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" },
+]
+
+[[package]]
+name = "pytz"
+version = "2026.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ff/46/dd499ec9038423421951e4fad73051febaa13d2df82b4064f87af8b8c0c3/pytz-2026.2.tar.gz", hash = "sha256:0e60b47b29f21574376f218fe21abc009894a2321ea16c6754f3cad6eb7cdd6a", size = 320861, upload-time = "2026-05-04T01:35:29.667Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ec/dd/96da98f892250475bdf2328112d7468abdd4acc7b902b6af23f4ed958ea0/pytz-2026.2-py2.py3-none-any.whl", hash = "sha256:04156e608bee23d3792fd45c94ae47fae1036688e75032eea2e3bf0323d1f126", size = 510141, upload-time = "2026-05-04T01:35:27.408Z" },
+]
+
+[[package]]
+name = "pyyaml"
+version = "6.0.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063, upload-time = "2025-09-25T21:32:11.445Z" },
+ { url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973, upload-time = "2025-09-25T21:32:12.492Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116, upload-time = "2025-09-25T21:32:13.652Z" },
+ { url = "https://files.pythonhosted.org/packages/65/30/d7353c338e12baef4ecc1b09e877c1970bd3382789c159b4f89d6a70dc09/pyyaml-6.0.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c", size = 844011, upload-time = "2025-09-25T21:32:15.21Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc", size = 807870, upload-time = "2025-09-25T21:32:16.431Z" },
+ { url = "https://files.pythonhosted.org/packages/05/c0/b3be26a015601b822b97d9149ff8cb5ead58c66f981e04fedf4e762f4bd4/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e", size = 761089, upload-time = "2025-09-25T21:32:17.56Z" },
+ { url = "https://files.pythonhosted.org/packages/be/8e/98435a21d1d4b46590d5459a22d88128103f8da4c2d4cb8f14f2a96504e1/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea", size = 790181, upload-time = "2025-09-25T21:32:18.834Z" },
+ { url = "https://files.pythonhosted.org/packages/74/93/7baea19427dcfbe1e5a372d81473250b379f04b1bd3c4c5ff825e2327202/pyyaml-6.0.3-cp312-cp312-win32.whl", hash = "sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5", size = 137658, upload-time = "2025-09-25T21:32:20.209Z" },
+ { url = "https://files.pythonhosted.org/packages/86/bf/899e81e4cce32febab4fb42bb97dcdf66bc135272882d1987881a4b519e9/pyyaml-6.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b", size = 154003, upload-time = "2025-09-25T21:32:21.167Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344, upload-time = "2025-09-25T21:32:22.617Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" },
+ { url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" },
+ { url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" },
+ { url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" },
+ { url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" },
+ { url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" },
+ { url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/8c/f4bd7f6465179953d3ac9bc44ac1a8a3e6122cf8ada906b4f96c60172d43/pyyaml-6.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac", size = 181814, upload-time = "2025-09-25T21:32:35.712Z" },
+ { url = "https://files.pythonhosted.org/packages/bd/9c/4d95bb87eb2063d20db7b60faa3840c1b18025517ae857371c4dd55a6b3a/pyyaml-6.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310", size = 173809, upload-time = "2025-09-25T21:32:36.789Z" },
+ { url = "https://files.pythonhosted.org/packages/92/b5/47e807c2623074914e29dabd16cbbdd4bf5e9b2db9f8090fa64411fc5382/pyyaml-6.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7", size = 766454, upload-time = "2025-09-25T21:32:37.966Z" },
+ { url = "https://files.pythonhosted.org/packages/02/9e/e5e9b168be58564121efb3de6859c452fccde0ab093d8438905899a3a483/pyyaml-6.0.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788", size = 836355, upload-time = "2025-09-25T21:32:39.178Z" },
+ { url = "https://files.pythonhosted.org/packages/88/f9/16491d7ed2a919954993e48aa941b200f38040928474c9e85ea9e64222c3/pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5", size = 794175, upload-time = "2025-09-25T21:32:40.865Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/3f/5989debef34dc6397317802b527dbbafb2b4760878a53d4166579111411e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764", size = 755228, upload-time = "2025-09-25T21:32:42.084Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/ce/af88a49043cd2e265be63d083fc75b27b6ed062f5f9fd6cdc223ad62f03e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35", size = 789194, upload-time = "2025-09-25T21:32:43.362Z" },
+ { url = "https://files.pythonhosted.org/packages/23/20/bb6982b26a40bb43951265ba29d4c246ef0ff59c9fdcdf0ed04e0687de4d/pyyaml-6.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac", size = 156429, upload-time = "2025-09-25T21:32:57.844Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/f4/a4541072bb9422c8a883ab55255f918fa378ecf083f5b85e87fc2b4eda1b/pyyaml-6.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3", size = 143912, upload-time = "2025-09-25T21:32:59.247Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/f9/07dd09ae774e4616edf6cda684ee78f97777bdd15847253637a6f052a62f/pyyaml-6.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3", size = 189108, upload-time = "2025-09-25T21:32:44.377Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/78/8d08c9fb7ce09ad8c38ad533c1191cf27f7ae1effe5bb9400a46d9437fcf/pyyaml-6.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba", size = 183641, upload-time = "2025-09-25T21:32:45.407Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/5b/3babb19104a46945cf816d047db2788bcaf8c94527a805610b0289a01c6b/pyyaml-6.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c", size = 831901, upload-time = "2025-09-25T21:32:48.83Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/cc/dff0684d8dc44da4d22a13f35f073d558c268780ce3c6ba1b87055bb0b87/pyyaml-6.0.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702", size = 861132, upload-time = "2025-09-25T21:32:50.149Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/5e/f77dc6b9036943e285ba76b49e118d9ea929885becb0a29ba8a7c75e29fe/pyyaml-6.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c", size = 839261, upload-time = "2025-09-25T21:32:51.808Z" },
+ { url = "https://files.pythonhosted.org/packages/ce/88/a9db1376aa2a228197c58b37302f284b5617f56a5d959fd1763fb1675ce6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065", size = 805272, upload-time = "2025-09-25T21:32:52.941Z" },
+ { url = "https://files.pythonhosted.org/packages/da/92/1446574745d74df0c92e6aa4a7b0b3130706a4142b2d1a5869f2eaa423c6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65", size = 829923, upload-time = "2025-09-25T21:32:54.537Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/7a/1c7270340330e575b92f397352af856a8c06f230aa3e76f86b39d01b416a/pyyaml-6.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9", size = 174062, upload-time = "2025-09-25T21:32:55.767Z" },
+ { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" },
+]
+
+[[package]]
+name = "pyzmq"
+version = "27.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "cffi", marker = "implementation_name == 'pypy'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/04/0b/3c9baedbdf613ecaa7aa07027780b8867f57b6293b6ee50de316c9f3222b/pyzmq-27.1.0.tar.gz", hash = "sha256:ac0765e3d44455adb6ddbf4417dcce460fc40a05978c08efdf2948072f6db540", size = 281750, upload-time = "2025-09-08T23:10:18.157Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/92/e7/038aab64a946d535901103da16b953c8c9cc9c961dadcbf3609ed6428d23/pyzmq-27.1.0-cp312-abi3-macosx_10_15_universal2.whl", hash = "sha256:452631b640340c928fa343801b0d07eb0c3789a5ffa843f6e1a9cee0ba4eb4fc", size = 1306279, upload-time = "2025-09-08T23:08:03.807Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/5e/c3c49fdd0f535ef45eefcc16934648e9e59dace4a37ee88fc53f6cd8e641/pyzmq-27.1.0-cp312-abi3-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:1c179799b118e554b66da67d88ed66cd37a169f1f23b5d9f0a231b4e8d44a113", size = 895645, upload-time = "2025-09-08T23:08:05.301Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/e5/b0b2504cb4e903a74dcf1ebae157f9e20ebb6ea76095f6cfffea28c42ecd/pyzmq-27.1.0-cp312-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3837439b7f99e60312f0c926a6ad437b067356dc2bc2ec96eb395fd0fe804233", size = 652574, upload-time = "2025-09-08T23:08:06.828Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/9b/c108cdb55560eaf253f0cbdb61b29971e9fb34d9c3499b0e96e4e60ed8a5/pyzmq-27.1.0-cp312-abi3-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:43ad9a73e3da1fab5b0e7e13402f0b2fb934ae1c876c51d0afff0e7c052eca31", size = 840995, upload-time = "2025-09-08T23:08:08.396Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/bb/b79798ca177b9eb0825b4c9998c6af8cd2a7f15a6a1a4272c1d1a21d382f/pyzmq-27.1.0-cp312-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:0de3028d69d4cdc475bfe47a6128eb38d8bc0e8f4d69646adfbcd840facbac28", size = 1642070, upload-time = "2025-09-08T23:08:09.989Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/80/2df2e7977c4ede24c79ae39dcef3899bfc5f34d1ca7a5b24f182c9b7a9ca/pyzmq-27.1.0-cp312-abi3-musllinux_1_2_i686.whl", hash = "sha256:cf44a7763aea9298c0aa7dbf859f87ed7012de8bda0f3977b6fb1d96745df856", size = 2021121, upload-time = "2025-09-08T23:08:11.907Z" },
+ { url = "https://files.pythonhosted.org/packages/46/bd/2d45ad24f5f5ae7e8d01525eb76786fa7557136555cac7d929880519e33a/pyzmq-27.1.0-cp312-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:f30f395a9e6fbca195400ce833c731e7b64c3919aa481af4d88c3759e0cb7496", size = 1878550, upload-time = "2025-09-08T23:08:13.513Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/2f/104c0a3c778d7c2ab8190e9db4f62f0b6957b53c9d87db77c284b69f33ea/pyzmq-27.1.0-cp312-abi3-win32.whl", hash = "sha256:250e5436a4ba13885494412b3da5d518cd0d3a278a1ae640e113c073a5f88edd", size = 559184, upload-time = "2025-09-08T23:08:15.163Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/7f/a21b20d577e4100c6a41795842028235998a643b1ad406a6d4163ea8f53e/pyzmq-27.1.0-cp312-abi3-win_amd64.whl", hash = "sha256:9ce490cf1d2ca2ad84733aa1d69ce6855372cb5ce9223802450c9b2a7cba0ccf", size = 619480, upload-time = "2025-09-08T23:08:17.192Z" },
+ { url = "https://files.pythonhosted.org/packages/78/c2/c012beae5f76b72f007a9e91ee9401cb88c51d0f83c6257a03e785c81cc2/pyzmq-27.1.0-cp312-abi3-win_arm64.whl", hash = "sha256:75a2f36223f0d535a0c919e23615fc85a1e23b71f40c7eb43d7b1dedb4d8f15f", size = 552993, upload-time = "2025-09-08T23:08:18.926Z" },
+ { url = "https://files.pythonhosted.org/packages/60/cb/84a13459c51da6cec1b7b1dc1a47e6db6da50b77ad7fd9c145842750a011/pyzmq-27.1.0-cp313-cp313-android_24_arm64_v8a.whl", hash = "sha256:93ad4b0855a664229559e45c8d23797ceac03183c7b6f5b4428152a6b06684a5", size = 1122436, upload-time = "2025-09-08T23:08:20.801Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/b6/94414759a69a26c3dd674570a81813c46a078767d931a6c70ad29fc585cb/pyzmq-27.1.0-cp313-cp313-android_24_x86_64.whl", hash = "sha256:fbb4f2400bfda24f12f009cba62ad5734148569ff4949b1b6ec3b519444342e6", size = 1156301, upload-time = "2025-09-08T23:08:22.47Z" },
+ { url = "https://files.pythonhosted.org/packages/a5/ad/15906493fd40c316377fd8a8f6b1f93104f97a752667763c9b9c1b71d42d/pyzmq-27.1.0-cp313-cp313t-macosx_10_15_universal2.whl", hash = "sha256:e343d067f7b151cfe4eb3bb796a7752c9d369eed007b91231e817071d2c2fec7", size = 1341197, upload-time = "2025-09-08T23:08:24.286Z" },
+ { url = "https://files.pythonhosted.org/packages/14/1d/d343f3ce13db53a54cb8946594e567410b2125394dafcc0268d8dda027e0/pyzmq-27.1.0-cp313-cp313t-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:08363b2011dec81c354d694bdecaef4770e0ae96b9afea70b3f47b973655cc05", size = 897275, upload-time = "2025-09-08T23:08:26.063Z" },
+ { url = "https://files.pythonhosted.org/packages/69/2d/d83dd6d7ca929a2fc67d2c3005415cdf322af7751d773524809f9e585129/pyzmq-27.1.0-cp313-cp313t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d54530c8c8b5b8ddb3318f481297441af102517602b569146185fa10b63f4fa9", size = 660469, upload-time = "2025-09-08T23:08:27.623Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/cd/9822a7af117f4bc0f1952dbe9ef8358eb50a24928efd5edf54210b850259/pyzmq-27.1.0-cp313-cp313t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6f3afa12c392f0a44a2414056d730eebc33ec0926aae92b5ad5cf26ebb6cc128", size = 847961, upload-time = "2025-09-08T23:08:29.672Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/12/f003e824a19ed73be15542f172fd0ec4ad0b60cf37436652c93b9df7c585/pyzmq-27.1.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c65047adafe573ff023b3187bb93faa583151627bc9c51fc4fb2c561ed689d39", size = 1650282, upload-time = "2025-09-08T23:08:31.349Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/4a/e82d788ed58e9a23995cee70dbc20c9aded3d13a92d30d57ec2291f1e8a3/pyzmq-27.1.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:90e6e9441c946a8b0a667356f7078d96411391a3b8f80980315455574177ec97", size = 2024468, upload-time = "2025-09-08T23:08:33.543Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/94/2da0a60841f757481e402b34bf4c8bf57fa54a5466b965de791b1e6f747d/pyzmq-27.1.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:add071b2d25f84e8189aaf0882d39a285b42fa3853016ebab234a5e78c7a43db", size = 1885394, upload-time = "2025-09-08T23:08:35.51Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/6f/55c10e2e49ad52d080dc24e37adb215e5b0d64990b57598abc2e3f01725b/pyzmq-27.1.0-cp313-cp313t-win32.whl", hash = "sha256:7ccc0700cfdf7bd487bea8d850ec38f204478681ea02a582a8da8171b7f90a1c", size = 574964, upload-time = "2025-09-08T23:08:37.178Z" },
+ { url = "https://files.pythonhosted.org/packages/87/4d/2534970ba63dd7c522d8ca80fb92777f362c0f321900667c615e2067cb29/pyzmq-27.1.0-cp313-cp313t-win_amd64.whl", hash = "sha256:8085a9fba668216b9b4323be338ee5437a235fe275b9d1610e422ccc279733e2", size = 641029, upload-time = "2025-09-08T23:08:40.595Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/fa/f8aea7a28b0641f31d40dea42d7ef003fded31e184ef47db696bc74cd610/pyzmq-27.1.0-cp313-cp313t-win_arm64.whl", hash = "sha256:6bb54ca21bcfe361e445256c15eedf083f153811c37be87e0514934d6913061e", size = 561541, upload-time = "2025-09-08T23:08:42.668Z" },
+ { url = "https://files.pythonhosted.org/packages/87/45/19efbb3000956e82d0331bafca5d9ac19ea2857722fa2caacefb6042f39d/pyzmq-27.1.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:ce980af330231615756acd5154f29813d553ea555485ae712c491cd483df6b7a", size = 1341197, upload-time = "2025-09-08T23:08:44.973Z" },
+ { url = "https://files.pythonhosted.org/packages/48/43/d72ccdbf0d73d1343936296665826350cb1e825f92f2db9db3e61c2162a2/pyzmq-27.1.0-cp314-cp314t-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:1779be8c549e54a1c38f805e56d2a2e5c009d26de10921d7d51cfd1c8d4632ea", size = 897175, upload-time = "2025-09-08T23:08:46.601Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/2e/a483f73a10b65a9ef0161e817321d39a770b2acf8bcf3004a28d90d14a94/pyzmq-27.1.0-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7200bb0f03345515df50d99d3db206a0a6bee1955fbb8c453c76f5bf0e08fb96", size = 660427, upload-time = "2025-09-08T23:08:48.187Z" },
+ { url = "https://files.pythonhosted.org/packages/f5/d2/5f36552c2d3e5685abe60dfa56f91169f7a2d99bbaf67c5271022ab40863/pyzmq-27.1.0-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:01c0e07d558b06a60773744ea6251f769cd79a41a97d11b8bf4ab8f034b0424d", size = 847929, upload-time = "2025-09-08T23:08:49.76Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/2a/404b331f2b7bf3198e9945f75c4c521f0c6a3a23b51f7a4a401b94a13833/pyzmq-27.1.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:80d834abee71f65253c91540445d37c4c561e293ba6e741b992f20a105d69146", size = 1650193, upload-time = "2025-09-08T23:08:51.7Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/0b/f4107e33f62a5acf60e3ded67ed33d79b4ce18de432625ce2fc5093d6388/pyzmq-27.1.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:544b4e3b7198dde4a62b8ff6685e9802a9a1ebf47e77478a5eb88eca2a82f2fd", size = 2024388, upload-time = "2025-09-08T23:08:53.393Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/01/add31fe76512642fd6e40e3a3bd21f4b47e242c8ba33efb6809e37076d9b/pyzmq-27.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:cedc4c68178e59a4046f97eca31b148ddcf51e88677de1ef4e78cf06c5376c9a", size = 1885316, upload-time = "2025-09-08T23:08:55.702Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/59/a5f38970f9bf07cee96128de79590bb354917914a9be11272cfc7ff26af0/pyzmq-27.1.0-cp314-cp314t-win32.whl", hash = "sha256:1f0b2a577fd770aa6f053211a55d1c47901f4d537389a034c690291485e5fe92", size = 587472, upload-time = "2025-09-08T23:08:58.18Z" },
+ { url = "https://files.pythonhosted.org/packages/70/d8/78b1bad170f93fcf5e3536e70e8fadac55030002275c9a29e8f5719185de/pyzmq-27.1.0-cp314-cp314t-win_amd64.whl", hash = "sha256:19c9468ae0437f8074af379e986c5d3d7d7bfe033506af442e8c879732bedbe0", size = 661401, upload-time = "2025-09-08T23:08:59.802Z" },
+ { url = "https://files.pythonhosted.org/packages/81/d6/4bfbb40c9a0b42fc53c7cf442f6385db70b40f74a783130c5d0a5aa62228/pyzmq-27.1.0-cp314-cp314t-win_arm64.whl", hash = "sha256:dc5dbf68a7857b59473f7df42650c621d7e8923fb03fa74a526890f4d33cc4d7", size = 575170, upload-time = "2025-09-08T23:09:01.418Z" },
+]
+
+[[package]]
+name = "rdata"
+version = "1.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+ { name = "pandas" },
+ { name = "typing-extensions" },
+ { name = "xarray" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c3/6f/4bf8fc3981569b0fb32815db86aef880b233305353a48d2c1770a29fe9cb/rdata-1.1.0.tar.gz", hash = "sha256:37c8b19e8be82d7d7acb36c6623bfeb7ab1d95d9380ec00a6c063f88b1f50e3a", size = 106564, upload-time = "2026-06-11T18:45:40.187Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/29/ef/acc15553b522a3710837520a76556958d533753d27e5ead18a8600159398/rdata-1.1.0-py3-none-any.whl", hash = "sha256:12efb7597725d6db6cc78d84eb522a9634008f8fce1c2733b7fd42b9013bc41f", size = 287685, upload-time = "2026-06-11T18:45:38.837Z" },
+]
+
+[[package]]
+name = "referencing"
+version = "0.37.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "attrs" },
+ { name = "rpds-py" },
+ { name = "typing-extensions", marker = "python_full_version < '3.13'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/22/f5/df4e9027acead3ecc63e50fe1e36aca1523e1719559c499951bb4b53188f/referencing-0.37.0.tar.gz", hash = "sha256:44aefc3142c5b842538163acb373e24cce6632bd54bdb01b21ad5863489f50d8", size = 78036, upload-time = "2025-10-13T15:30:48.871Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231", size = 26766, upload-time = "2025-10-13T15:30:47.625Z" },
+]
+
+[[package]]
+name = "requests"
+version = "2.34.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "certifi" },
+ { name = "charset-normalizer" },
+ { name = "idna" },
+ { name = "urllib3" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ac/c3/e2a2b89f2d3e2179abd6d00ebd70bff6273f37fb3e0cc209f48b39d00cbf/requests-2.34.2.tar.gz", hash = "sha256:f288924cae4e29463698d6d60bc6a4da69c89185ad1e0bcc4104f584e960b9ed", size = 142856, upload-time = "2026-05-14T19:25:27.735Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0", size = 73075, upload-time = "2026-05-14T19:25:26.443Z" },
+]
+
+[[package]]
+name = "rich"
+version = "15.0.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "markdown-it-py" },
+ { name = "pygments" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/0722ca900cc807c13a6a0c696dacf35430f72e0ec571c4275d2371fca3e9/rich-15.0.0.tar.gz", hash = "sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36", size = 230680, upload-time = "2026-04-12T08:24:00.75Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" },
+]
+
+[[package]]
+name = "roman-numerals"
+version = "4.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ae/f9/41dc953bbeb056c17d5f7a519f50fdf010bd0553be2d630bc69d1e022703/roman_numerals-4.1.0.tar.gz", hash = "sha256:1af8b147eb1405d5839e78aeb93131690495fe9da5c91856cb33ad55a7f1e5b2", size = 9077, upload-time = "2025-12-17T18:25:34.381Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/04/54/6f679c435d28e0a568d8e8a7c0a93a09010818634c3c3907fc98d8983770/roman_numerals-4.1.0-py3-none-any.whl", hash = "sha256:647ba99caddc2cc1e55a51e4360689115551bf4476d90e8162cf8c345fe233c7", size = 7676, upload-time = "2025-12-17T18:25:33.098Z" },
+]
+
+[[package]]
+name = "rpds-py"
+version = "2026.5.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/2e/43/25a8dcd3feedd735039a8f0b5b7e3b118232b5eae288c4fd9ab200d41094/rpds_py-2026.5.1.tar.gz", hash = "sha256:07b24fea40541e28570e5b795a4a38fbdcd12550c06bd0748005ecc8116ca256", size = 64459, upload-time = "2026-05-28T12:02:13.232Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d4/e7/a78582dc57caa592dcc7d4fb69b61390561e908eb3d2f5df5928a8e354c0/rpds_py-2026.5.1-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:3abe24a66e57adcfa645d718063a5fa5103ecc71ddbf26d78af8f9368018ff1d", size = 353040, upload-time = "2026-05-28T11:59:12.531Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/43/35e3f136343aef451e545ce8c38d36c2f93c0ed88703db8b64ba2b205c68/rpds_py-2026.5.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:58b1d94308ddf0b1982f61f2eb54bf92997c9ece8a8093ef014250f4a517906c", size = 345775, upload-time = "2026-05-28T11:59:13.827Z" },
+ { url = "https://files.pythonhosted.org/packages/20/e1/0f2160c5982d3157734d5cb3ed63d8b2d583a73c9864f77b666449f32cf8/rpds_py-2026.5.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0fa92420128dadce7f54bd73ba1825a273e9268fe9e35dbf7e6362890efa4e08", size = 376329, upload-time = "2026-05-28T11:59:15.271Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/11/ee0ba42aff83bf4effdbc576673c6be64c5e173978c3f6d537e94482f77d/rpds_py-2026.5.1-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ca653c6546386227cd9800d1bef6a348099acf8db4250341da6d90f663d6dfcb", size = 383539, upload-time = "2026-05-28T11:59:16.665Z" },
+ { url = "https://files.pythonhosted.org/packages/11/df/d94aa6a499d4ac40afe2d7620f2c597fd3c0f182e854ad7cf3f596a81cb6/rpds_py-2026.5.1-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:66c93681c4729e4e3ecba31b8179fae083ff3118841672835140338b4b9867c1", size = 494674, upload-time = "2026-05-28T11:59:17.991Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/75/33d30f43bb2f458de11979486a591b1bf6e5651765ed1704c6197c2dc773/rpds_py-2026.5.1-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:40ff257542e04796880e011e15cd4dc21c2599975df2aaa8f2c8495ca574e1a5", size = 389268, upload-time = "2026-05-28T11:59:19.434Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/1e/2c9096fc19d5fd084b0184ca2b651e659aa0a37e6fdbecf6ece47f147fe1/rpds_py-2026.5.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b6825cc329b290e93c5f6a9be2393118a763f6ccf6abd83704e0c102ca583644", size = 376280, upload-time = "2026-05-28T11:59:21Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/e5/61ec9f8be8211ea7f48448195549e4aaf02004083475493b0e137702ecb2/rpds_py-2026.5.1-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:de42116e69cb53b911cc34aee5ab98f36c597b822545045d49e938818b99e5e4", size = 387233, upload-time = "2026-05-28T11:59:22.454Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/ca/bcec1005c4f4a234f92a29078631fee49206c7265ccae966f18fd332e80e/rpds_py-2026.5.1-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c0f920015df2a504bebaba6d4c31ccf3fcf942f92655c086da30b671aad19aa6", size = 405009, upload-time = "2026-05-28T11:59:23.845Z" },
+ { url = "https://files.pythonhosted.org/packages/72/e6/4d5718c5cf26c522dc7c9999e238da1e77380b81d0c5d1df11e271ddfeb1/rpds_py-2026.5.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:0408a24e44feb919423dc6d9da677cb5cddb894d2ca9e763967d156d9c60fab4", size = 553113, upload-time = "2026-05-28T11:59:25.184Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/25/2ee807bdb3e1f0b7eddf7782acd5665a8b5205a331a7d7244a52c4812fd9/rpds_py-2026.5.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:cea68bcd53467561ae2f96a6bdad1544299ba97b5b0ddcd5ac3d376e5c781c24", size = 618838, upload-time = "2026-05-28T11:59:26.749Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/c1/7d4c26f167f8c41501cc073d30ee22082b16ce358cf5b00ec97cbc7804ea/rpds_py-2026.5.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:4be8b1d2a705cc37d08256004e1d07de143fa0075c8e85a3df020b776f62b732", size = 582436, upload-time = "2026-05-28T11:59:28.11Z" },
+ { url = "https://files.pythonhosted.org/packages/04/1d/9d12b0a337bab46f4769f8857f4007e3b2d639e14f9a44a0efe157696e64/rpds_py-2026.5.1-cp312-cp312-win32.whl", hash = "sha256:6736718bd4fc49cbcb538ba30516fdbef161522acefb739657d48b97bd864fed", size = 212734, upload-time = "2026-05-28T11:59:29.689Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/93/e4116f2de7f56bc7406a76033dc501811ddeb22b7f056b92d632871ebb0c/rpds_py-2026.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:0a7d1eec967df0e9b22614a5e177622e0c89611d03727fa0cb48e45028907870", size = 229045, upload-time = "2026-05-28T11:59:31.033Z" },
+ { url = "https://files.pythonhosted.org/packages/cb/53/6c3419d85eb2ec5938a37627c585b42d76a63bb731d6e42ed4b079ebf486/rpds_py-2026.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:1841d067089e117142d79b98aa0df2f08b52f2ecc1819dd2700636c0db74a473", size = 223967, upload-time = "2026-05-28T11:59:32.318Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/32/14c961ad295f490eb0849ada8b79683e93a59b9de3afdd983eaf55fa6867/rpds_py-2026.5.1-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:efef4ac29c6ff495531eb17ee705b62841ecaa291b7c7077e848ea03e237164d", size = 352787, upload-time = "2026-05-28T11:59:33.655Z" },
+ { url = "https://files.pythonhosted.org/packages/ca/bb/d1b85117967c11191441a7274ae616c65d93901d082c588f89a50a8da5ae/rpds_py-2026.5.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:c39f5b67a8a2e67179ada2a954227d670fe65fa9098457f698f56ddf248709b3", size = 345179, upload-time = "2026-05-28T11:59:35Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/46/d84105f062e626a1b233f863907288a4708c2d833b8b4c6fb2764bc080c0/rpds_py-2026.5.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b5c30f3f04eef4fbd362226a6f31d7c8895ca4fbb6e0b790f6890a98d8da8559", size = 376173, upload-time = "2026-05-28T11:59:36.43Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/ae/469d7959ce5b1201e1de135dc735b86db3b35dd0d1734f6a44246d5f061c/rpds_py-2026.5.1-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:277f6c82f0580848796c7ecc8a7173aa3bfb928e4ff831261c2f60a81dc270db", size = 383162, upload-time = "2026-05-28T11:59:37.995Z" },
+ { url = "https://files.pythonhosted.org/packages/dc/a2/57853d31a1116a561aa072794602ad3f6341e18d70a8523f1bd5b9fc1e5a/rpds_py-2026.5.1-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:63c2c4c213f1a4e3f3de28ecab029dbdee976324e729c0d7a55211be72576b02", size = 495093, upload-time = "2026-05-28T11:59:39.453Z" },
+ { url = "https://files.pythonhosted.org/packages/99/63/3a8eabcad9314b7daf5c65f451d2c33d989235cd8a5762186cf2c3f5a4f8/rpds_py-2026.5.1-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3350ec808fb538fe71a1f94dfaa0e29c598dfad805ce49f0caec5ae3183c652b", size = 389829, upload-time = "2026-05-28T11:59:40.896Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/25/05678d97fc25e2622df14dc530fb82023174ecfff6733991ed0d78f167bd/rpds_py-2026.5.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b1b964e3ab599e718dc46c018d104b1ebc007cbc6567d827c94a687fca56d77e", size = 374786, upload-time = "2026-05-28T11:59:42.626Z" },
+ { url = "https://files.pythonhosted.org/packages/88/d1/8c90b6431e80a3b91b284a5c7c8c0c4f9c006444d90477a740d6e0f9c694/rpds_py-2026.5.1-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:19cb09fab7b7fc96b2a6e28f2e34b72a3705ff27b37edb77455316e5d3f3dc9b", size = 386920, upload-time = "2026-05-28T11:59:44.124Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/99/4638f672ab356682d633ee0da9255f5b67ce6efd0b85eb94ad3e255e65a5/rpds_py-2026.5.1-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:abe76bcdba31e576cb83eeb8797aa0d882b738fef6dc65d0601fc753806a5b46", size = 405059, upload-time = "2026-05-28T11:59:47.177Z" },
+ { url = "https://files.pythonhosted.org/packages/66/3f/3546524b6eb4cc2e1f363a3d638fa52f6c24faae3500c25fb488b02f1740/rpds_py-2026.5.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:8bff7073db3899158fff55ebf57b113a67030af26f80a18978f9f0aa60250ddf", size = 553030, upload-time = "2026-05-28T11:59:48.603Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/c3/7b3388c796fcf471bd17194242d4dc1a7608567c0fa422bcc1c5e79f9c1e/rpds_py-2026.5.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:8ba264fa49be666cd9cc56bf34ec7002fb3d27a4aee5bcb4d43d0d18feb1bb6f", size = 618975, upload-time = "2026-05-28T11:59:50.314Z" },
+ { url = "https://files.pythonhosted.org/packages/61/1e/a3cb07f2795075d1d88efddae2f541359fde5f08c81ee114c29c2949c90a/rpds_py-2026.5.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4860b603ddda0475a8885499b3729e90229d480105b42651962a5397d995fa89", size = 581178, upload-time = "2026-05-28T11:59:51.673Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/74/e758c03a5ef46f04c37f2651a2893db846d569ba8a7bca469d4b58939bcd/rpds_py-2026.5.1-cp313-cp313-win32.whl", hash = "sha256:7944270ae71383f6e2657dd7d5ce4eeb4ac2d0059a6738f0510583d462ab4842", size = 212481, upload-time = "2026-05-28T11:59:53.148Z" },
+ { url = "https://files.pythonhosted.org/packages/70/ec/a2aca432db9c7359b40fa393eeeaa0d166c2f70175be956e75fa24197c44/rpds_py-2026.5.1-cp313-cp313-win_amd64.whl", hash = "sha256:88647f43a73c4e01be19b04ceef0c8d3a1958153604d13c773becd8016f2a0cf", size = 228519, upload-time = "2026-05-28T11:59:54.505Z" },
+ { url = "https://files.pythonhosted.org/packages/29/60/a73bfdd45b096574556acf303bbd9fa9eed36ca8a818b514e2a5d5fe2b9d/rpds_py-2026.5.1-cp313-cp313-win_arm64.whl", hash = "sha256:453895624ecf7db7063b1004e44037522bbaef9ff6a945e59bc71662d7a03abd", size = 223446, upload-time = "2026-05-28T11:59:56.081Z" },
+ { url = "https://files.pythonhosted.org/packages/18/e2/408105fd611823f00882aea810f3989a30d26b1bab8b6beb20f98c724e0e/rpds_py-2026.5.1-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:b4e4bc98639ec915f512fde3aa7a95e0041d95d9c3cc86eea841fa63cb1e8600", size = 355287, upload-time = "2026-05-28T11:59:57.448Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/58/5c4a43436843c90d0f6d19f82c200c80e3843ca9fa07b237623327f6d384/rpds_py-2026.5.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:cacedb7a6e167680acba45ad5716e89067d225dc80da0d7040cae8c81d4572fa", size = 347033, upload-time = "2026-05-28T11:59:58.881Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/c2/1a71acdacaf4e259b10278fb87b039ded3cf80041bcd89dd8a3ea702ded6/rpds_py-2026.5.1-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:68700371c5d7ae1412862ddfa719090925c93ecf351c566d66f09d04b136ea00", size = 376891, upload-time = "2026-05-28T12:00:00.516Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/c8/535f3d9b65addd8e28aa87b83c6e526799c3717a88273db8ea795beeef7a/rpds_py-2026.5.1-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:296c799becfa849c779c8725494fe9ed94959ed886787df4364b058465bad7f0", size = 385646, upload-time = "2026-05-28T12:00:02.394Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/91/dc033f313345c354ade914dbe73cdb90b615a4409ea02430d5356794f3d8/rpds_py-2026.5.1-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d3858b908218ee108d0bbfb2095ccc237648053c9bf98affad7cb079acaf1d97", size = 498830, upload-time = "2026-05-28T12:00:04.189Z" },
+ { url = "https://files.pythonhosted.org/packages/27/fc/90fcbea459dbb8ddc18a2e0fd1de9412b48bc84ffff2db771cf714bacfd6/rpds_py-2026.5.1-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4fb8d2e7cb2f850b169806d61d1b991738acec96500a75c30f49caf064ce7cef", size = 392830, upload-time = "2026-05-28T12:00:05.797Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/1d/46cd11a228c9750684a798d98f878be6f614aa762438da7378f035e79e35/rpds_py-2026.5.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:27b74c10ed6a8f190f4287f53bcfea348b92a84a9c9f70d30183d1e6172d580d", size = 379613, upload-time = "2026-05-28T12:00:07.433Z" },
+ { url = "https://files.pythonhosted.org/packages/24/4a/d9b0c6af3a1de03eb93741bbe8be2bdce84d8fda8224f3005451d86df389/rpds_py-2026.5.1-cp313-cp313t-manylinux_2_31_riscv64.whl", hash = "sha256:b9a6528956191c48c52294a592dbd4a8386d7048bdb25c0efcb6b966466c6d83", size = 388183, upload-time = "2026-05-28T12:00:09.227Z" },
+ { url = "https://files.pythonhosted.org/packages/c5/b4/db7aaabdda6d020afc87d981bcc2f57a434c7dec60ecfc2ab3dd50b20351/rpds_py-2026.5.1-cp313-cp313t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:af03e34e860047bc7a352b842856fcf78798fbb81132cc98bd2f907ab4eb9cd2", size = 408578, upload-time = "2026-05-28T12:00:10.779Z" },
+ { url = "https://files.pythonhosted.org/packages/08/d6/070f6a41cbb343e2ac4171859bf3f3623e0ab002f72619d6d505313ec2de/rpds_py-2026.5.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:fea6e836d10abbe191d557d33bd58bd5987725fe63aa1eefe557d230209855bd", size = 553573, upload-time = "2026-05-28T12:00:12.443Z" },
+ { url = "https://files.pythonhosted.org/packages/75/ab/1a71ea3589c4345dac0a0518f0e6a031cb42689277851b683c46d27463a5/rpds_py-2026.5.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:fc0c0f878ea770a0a8a462456c5ad36fc9fe6358e6b76fdadc7f17575e0b8bf1", size = 620861, upload-time = "2026-05-28T12:00:14.09Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/22/9bf80a56069c0c443fcfefac639a86a744550a2898817a6dfd3e26654924/rpds_py-2026.5.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:e0b360f316d966b048b085857630b3cc51f3db2f07b06f440eac8f695374d1e3", size = 585633, upload-time = "2026-05-28T12:00:15.66Z" },
+ { url = "https://files.pythonhosted.org/packages/da/68/3b2c0a75c9e04125696f84ebdbbf304acf5a40b58ba4481cdb98a922c3ba/rpds_py-2026.5.1-cp313-cp313t-win32.whl", hash = "sha256:a2999883eedf72fdfb7520b92c7d4ec2572a71ff40239377aa604cc529eecafc", size = 210074, upload-time = "2026-05-28T12:00:17.291Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/8b/609157d5a25d37d4f29f92840ba531f416907c34ae5c5739dd21fc2bef98/rpds_py-2026.5.1-cp313-cp313t-win_amd64.whl", hash = "sha256:e07be2a9d7122bd6e82dea89814ef8dc893feb1aae97fec1630f3263bbb30e55", size = 228635, upload-time = "2026-05-28T12:00:18.73Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/6f/19c1918a4b590d8de87e712e4abe4b3875771eff60216fb6153cf6665c68/rpds_py-2026.5.1-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:1f2c391c3059798093b65df23aca2cac150460ae9c630d99dec83d703d9485b9", size = 349756, upload-time = "2026-05-28T12:00:20.217Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/60/a06fe7da34eca79dacbf958a2ba0c6eea85bc2b29de20080bf40f72f66fa/rpds_py-2026.5.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:413b424f7c4ee65ab5e5be91f5731be0f8b41a1ee2b12dfe810d716312e95a78", size = 343831, upload-time = "2026-05-28T12:00:21.711Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/ec/b2333b97b90e2a6ef6ca8ad386ee284968e74bcfe113b3f1a8d9036429a9/rpds_py-2026.5.1-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2c595a1d9255dce0599e13130d1440ab2506654f2b50294226ee06402f8fef63", size = 375127, upload-time = "2026-05-28T12:00:23.326Z" },
+ { url = "https://files.pythonhosted.org/packages/14/7f/e00aae54067f2b488c4637961d5f58204d470795fc791085fa3f15060d2e/rpds_py-2026.5.1-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:1c27c5f6102eac8c03e7595a00827a53b271ba40a53b59ff8709170e0855ea4a", size = 379034, upload-time = "2026-05-28T12:00:24.89Z" },
+ { url = "https://files.pythonhosted.org/packages/be/cc/423999bbb8ae8dc93c77fc1d5e984ade5eb89d237d3bb884ccfa72ae2890/rpds_py-2026.5.1-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:6c7fcf61d44cacecaf3aea542b0e053db77972a4573e7ceda16fb2b399161195", size = 490823, upload-time = "2026-05-28T12:00:26.676Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/aa/c671bf660f12e68d3c52ff86c7066ed1372df5a0f4f2ff584e419b8207e7/rpds_py-2026.5.1-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2c817a189d4ee14290420e5ff051e4dd6baa13f3edf84685071dee07a6d538ee", size = 388144, upload-time = "2026-05-28T12:00:28.577Z" },
+ { url = "https://files.pythonhosted.org/packages/19/c8/d63bb75b68afe77b229e3021c6031bcaf01da5db5b0e69d0d10f9ba679a7/rpds_py-2026.5.1-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:21846aac0ed2e0589f38c12dc44e77bb64e494b771eadbcf169cba00566ba7ba", size = 371959, upload-time = "2026-05-28T12:00:30.304Z" },
+ { url = "https://files.pythonhosted.org/packages/82/35/c51122014d8274ff37dc606d60049c3db7d83da02b5b282511e5a906a9a6/rpds_py-2026.5.1-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:b317c87a13f769a4e787819bd508aaa5d69aa09b0880de9af6d3a8a54571cdec", size = 383558, upload-time = "2026-05-28T12:00:31.764Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/f9/2790cb99c136a5363acdeacf5c27c56f3de0d4118a1f48fca83404c99c89/rpds_py-2026.5.1-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:ce87129d9f2c14fa6c4a8601fb80eb4488c80d38a20cd13758ef11123e14995d", size = 402789, upload-time = "2026-05-28T12:00:33.247Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/1b/e4fb584f8c75d35c38150ff6a332cda949e6f97acba1f4fd123b14ab56fe/rpds_py-2026.5.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9cdddb6c1207d284d94fd1530adf57fbd797fe7c4b8704ba85f49414f2557e7d", size = 551405, upload-time = "2026-05-28T12:00:34.819Z" },
+ { url = "https://files.pythonhosted.org/packages/d8/f7/a6731b4216cb3793ea1af5391da240f5683dacc0d13e034fe5fc3503f240/rpds_py-2026.5.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:4e237e139f94d3c036fd28eb9f564c99055476ff4ff05cd42be55ce349b5aa02", size = 616975, upload-time = "2026-05-28T12:00:36.268Z" },
+ { url = "https://files.pythonhosted.org/packages/2c/ea/2e051a81d95d8e63f4b35a1c463a87e8766bc3d083c067c5dfb6bf220747/rpds_py-2026.5.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ed0954b524873214369184a9c82b0eaa45a3fbb9a798cd95b17e0d98499e7ea0", size = 578701, upload-time = "2026-05-28T12:00:37.82Z" },
+ { url = "https://files.pythonhosted.org/packages/65/56/b5f6fdb2083e32bca8a8993d89e70db114b4756c9e2c38421328126689d2/rpds_py-2026.5.1-cp314-cp314-win32.whl", hash = "sha256:2d88621d6a7d4dfa633d21abe90f280bb205274e16b1d1e61c6ad4640b2453b7", size = 209806, upload-time = "2026-05-28T12:00:39.492Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/80/65a5aa96c155e611d1ed844e4e1f57f3e36b021f396d9f8585d756e6b90d/rpds_py-2026.5.1-cp314-cp314-win_amd64.whl", hash = "sha256:cef8ac28d26f4dda3533060c20fbf80a325458fa9fd23ea72a73cdfa8e978838", size = 225985, upload-time = "2026-05-28T12:00:40.94Z" },
+ { url = "https://files.pythonhosted.org/packages/27/7c/ad185212e87b05f196daef92bc5f3caf07298eb47c295b5585c3dd3093ac/rpds_py-2026.5.1-cp314-cp314-win_arm64.whl", hash = "sha256:eaaea962c68cdc68d4a533ba985ab8e9484277910bbfaa2ab3ef7732667bfed8", size = 221219, upload-time = "2026-05-28T12:00:43.15Z" },
+ { url = "https://files.pythonhosted.org/packages/23/58/e14ae18759020334646b031e708ab4158d653a938822bfb7b95ef2e93aa3/rpds_py-2026.5.1-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:21942f52dbbd5f8758bf021213d28bd45c39e873e65e2407faf5f1846f5761ad", size = 352148, upload-time = "2026-05-28T12:00:44.638Z" },
+ { url = "https://files.pythonhosted.org/packages/31/9b/5f4a1e2f960bca3ac5d052b139dd31eed97b259f9d909173821760d542e8/rpds_py-2026.5.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f414556f6e3958300ff941e40c9f97e3dc9774ddd1b3434c475d73dd354bbed3", size = 345196, upload-time = "2026-05-28T12:00:46.14Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/71/1d9574d6a2fa20ab60eaa55c7467f5aa20cbc770f341a05f09c0876f59e2/rpds_py-2026.5.1-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ef1013a8625c74043210190b246f5b1551e09757c1f356c6e4160ef96c5bc081", size = 374981, upload-time = "2026-05-28T12:00:47.531Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/9a/37e99f4915a80aa71670263c1267f7ae0af95f53a3f61e6c3bdc016d4515/rpds_py-2026.5.1-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:cc68e231a77a5f0d774ae278a1f8e55c0456501820847c1e4efb3829f3441df6", size = 379961, upload-time = "2026-05-28T12:00:49.216Z" },
+ { url = "https://files.pythonhosted.org/packages/a8/ff/6e73f74b89d2e0715e0fc86b7dde893f9a61ae2f9b256ff3bdfe41ac4e94/rpds_py-2026.5.1-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9baffb505aff33acc69b422a19f77806680f3c8632227d79f48de8a810d1c2c5", size = 495965, upload-time = "2026-05-28T12:00:51.111Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/e0/425faba25f59d74d4638b267f7c7a80e8649d2ef4db10a19b0c4a71e6e6f/rpds_py-2026.5.1-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b8d2f912928d426e8cfa396f7f3f8d29a59e6689c86dcca3c420730c1096322b", size = 389526, upload-time = "2026-05-28T12:00:52.77Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/76/7a41960e3fddae47fab43a28684d5da981401dffd88253de0944148654cb/rpds_py-2026.5.1-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:90f628283be835db980c941767d41c9a27b5239e54ba0a9c1335247e82406964", size = 376190, upload-time = "2026-05-28T12:00:54.215Z" },
+ { url = "https://files.pythonhosted.org/packages/27/60/5f38dc70824fc6951b51d35377e577a3a3a4c81a6769cc5a2de25ebe0ad1/rpds_py-2026.5.1-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:1ebb2f0ab7e16132995a72de805170e0203df0c3dd22e1ef1cd1fdd90bd7a131", size = 383921, upload-time = "2026-05-28T12:00:55.673Z" },
+ { url = "https://files.pythonhosted.org/packages/60/1a/d60a38caa1505f4b9483c3fbbde12c94e1079154f4f401a6da96f7e77621/rpds_py-2026.5.1-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:f3df3d16ded76f1f8c9cdebd0e1ea55fdf4c23b812de189814da7cf229c22a81", size = 404766, upload-time = "2026-05-28T12:00:57.518Z" },
+ { url = "https://files.pythonhosted.org/packages/87/ff/602fd3f174d6425f0bce05ad0dfbec0e96b38d0f7d08a79af5aa20083885/rpds_py-2026.5.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:9af8905b8f854990e40d5206aa5ac58d9b0fe0b7f351ff2bb086c20f6c8c6a47", size = 551343, upload-time = "2026-05-28T12:00:58.978Z" },
+ { url = "https://files.pythonhosted.org/packages/b8/c1/1be13327acdbead3eca1fde03b6a34dbb011f1e864e217f0d32cc1779a7f/rpds_py-2026.5.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:036a36a87fb1cd3b214d11c4b3c4f7d2ddad933625dca1c900b56a057c07740a", size = 618502, upload-time = "2026-05-28T12:01:00.656Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/d7/afb49b49d7f2be8b7ba1a9f0977fa5168003437b93086726f066544e8351/rpds_py-2026.5.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:62ae3853454fe9ef283a03c96c2d835d39e84b14643a9d62c82ef0fb87d702ca", size = 581916, upload-time = "2026-05-28T12:01:02.22Z" },
+ { url = "https://files.pythonhosted.org/packages/25/d1/dbef8c1f8a10f07beb62b5f054e20099fd9924b3ec001b8f0b6ac7813a85/rpds_py-2026.5.1-cp314-cp314t-win32.whl", hash = "sha256:6c3d771a46ec18b12af06ce36243a9a80b07a5d0515236332d90863ca8bb326a", size = 207855, upload-time = "2026-05-28T12:01:03.821Z" },
+ { url = "https://files.pythonhosted.org/packages/2a/72/bfa4e61ab8e7dc1c8adf397e05e6cbdd4239357bd72b248d3de662f23915/rpds_py-2026.5.1-cp314-cp314t-win_amd64.whl", hash = "sha256:c93c629be4636cf54337bd5f06c104d55e42ced54d681f6fe21ae510a65116f6", size = 225422, upload-time = "2026-05-28T12:01:05.194Z" },
+ { url = "https://files.pythonhosted.org/packages/27/3a/7b5da92b640f67b6717ccafc83cdd06bfa7ff2395c3685c68922bb54d703/rpds_py-2026.5.1-cp315-cp315-macosx_10_12_x86_64.whl", hash = "sha256:3574b55c604b8f75dacb007136508bbc0db406e626301778096a133327e7f2fb", size = 349576, upload-time = "2026-05-28T12:01:06.722Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/8a/2aafd7ad355a1bd48ca76e2262b74b15e6432b5a1efe150efd4d779cd55d/rpds_py-2026.5.1-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:94068eb3ae6d43f5a786b7db96a406a34e6d5c24489feef32fd6e8946ea7b291", size = 343640, upload-time = "2026-05-28T12:01:08.441Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/7d/6c9523c1abbe840a1b7fba3c516d48e1d3487cc80fea4366c4071cf56784/rpds_py-2026.5.1-cp315-cp315-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f3a5b10e8ce894825f380a8f1b6444cf73c294dfea62afbb2d13e3a9e630cec1", size = 375322, upload-time = "2026-05-28T12:01:09.934Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/5d/0b7b03fb1dc509321f01de3149784ab773e34c8573022029af8076afcb9c/rpds_py-2026.5.1-cp315-cp315-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:fc09f82e63d4bcd58149572f857a431bae851dc747e313c3b5bdf7abb907fda8", size = 379066, upload-time = "2026-05-28T12:01:11.48Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/e2/8ef6012999ebf1cb1c22f876d9ce5e63d960fd4631d2af3202d3f480aa25/rpds_py-2026.5.1-cp315-cp315-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e10464d17df3b582745c25cec695cb9558bca2cb6ddb631aee1787fc72c767b2", size = 494586, upload-time = "2026-05-28T12:01:13.051Z" },
+ { url = "https://files.pythonhosted.org/packages/80/af/1eeb029bec67582c226b7809172207cd005073af4ebd906e65ff494f4983/rpds_py-2026.5.1-cp315-cp315-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ba05adbf15d994c38ec0b7ab32e858e5110c21e9009a00a86545fd220f84e038", size = 388415, upload-time = "2026-05-28T12:01:14.631Z" },
+ { url = "https://files.pythonhosted.org/packages/18/23/ffbe10711c4d766c1cab0557d6906c074f795814863c67b351355d29354a/rpds_py-2026.5.1-cp315-cp315-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:77c004fdc7b891967106f78ddfd7b076bfe6813c6139c6fff6aed3bcaa960b26", size = 372427, upload-time = "2026-05-28T12:01:16.153Z" },
+ { url = "https://files.pythonhosted.org/packages/bd/3a/30ba4a6ad457e5b070c18d742a33fb77d8d922b565cc881f8a5313d63bfe/rpds_py-2026.5.1-cp315-cp315-manylinux_2_31_riscv64.whl", hash = "sha256:83bcf894486c9d78dd290d3c0124ff6dd8875d3025e2090a8ec49fcc37c55fdd", size = 383615, upload-time = "2026-05-28T12:01:17.809Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/69/62e242b53ce39c0814bd24e1a6e6eba6c92be716277745f317f9540a2e7b/rpds_py-2026.5.1-cp315-cp315-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c3df104083952a0e0c6f10de33e440eabe98fb6317d23e1a58c68f6df08d01b9", size = 402786, upload-time = "2026-05-28T12:01:19.419Z" },
+ { url = "https://files.pythonhosted.org/packages/38/c1/a770b9c186928a1ed0f7e6d7ae50e7f3950ed23e3f9e366dbc8e38cb55de/rpds_py-2026.5.1-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:980450826cf22e133c57e0835070bdd0dd3f73b9b708c3ce223def2cb9469e14", size = 551583, upload-time = "2026-05-28T12:01:21.013Z" },
+ { url = "https://files.pythonhosted.org/packages/21/7c/68e8579b95375b70d2a963103c42e705856cdb98569258bd807f4423891c/rpds_py-2026.5.1-cp315-cp315-musllinux_1_2_i686.whl", hash = "sha256:205dde846f24332ab0c1188699a043b8d165b79bb84529ce272c45048ff6be01", size = 616941, upload-time = "2026-05-28T12:01:22.548Z" },
+ { url = "https://files.pythonhosted.org/packages/70/a1/a6135aed5730ff03ab957182259987ac11e55fb392a28dc6f0592048a280/rpds_py-2026.5.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:3966b82dd563176396df030f3dd52a6e54cb69b718e95e78bd555ed3d1e0185d", size = 578349, upload-time = "2026-05-28T12:01:24.118Z" },
+ { url = "https://files.pythonhosted.org/packages/09/6e/f24201a76a84e6c49d0bdfdfcb735210e21701e9b21c5bfc0ba497dd62f6/rpds_py-2026.5.1-cp315-cp315-win32.whl", hash = "sha256:7818f8d0a415be74d2be3590b0a1c1f463a642f4d0217e7d10602dceef5b79aa", size = 209922, upload-time = "2026-05-28T12:01:25.522Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/e4/966bc240bb0485fc265278f6de44d05834bf0b3618886e0b22e33d54c49a/rpds_py-2026.5.1-cp315-cp315-win_amd64.whl", hash = "sha256:b3cc20c0d800af78fd0fac68086e28c1856cec51ea528bb81ea851aa40d39325", size = 226003, upload-time = "2026-05-28T12:01:27.062Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/5c/a15a59269cd5e74472734516c73795c15eccfc841b3d4b0228c3f53f19d0/rpds_py-2026.5.1-cp315-cp315-win_arm64.whl", hash = "sha256:3609e9939a8a76cd904cf98a3f1f13b5dc7e150adeaee89e0ea09652ea213e16", size = 221245, upload-time = "2026-05-28T12:01:28.51Z" },
+ { url = "https://files.pythonhosted.org/packages/e0/22/135ce03804e179a71ceb13be095deda4a279bc88f7a6b8fa161c5ad44e12/rpds_py-2026.5.1-cp315-cp315t-macosx_10_12_x86_64.whl", hash = "sha256:5d333a7127d4b307601ac37792bee01bb95c867cbfacf21b6375b804d6bbd723", size = 352015, upload-time = "2026-05-28T12:01:30.214Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/5f/f1f6d2652eb9d848f6eb369d8db83a2da6249bb49ad2c2a48f45d54538d3/rpds_py-2026.5.1-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:b5f077b44a4f7808520f66dae234988d867deb9aed9be5da057ce9ba831b2a41", size = 345016, upload-time = "2026-05-28T12:01:31.656Z" },
+ { url = "https://files.pythonhosted.org/packages/88/66/b74182775691ea2290c99e52ac8d5db844e56fbec90ce421f107658c8314/rpds_py-2026.5.1-cp315-cp315t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:55d8f9b7b78c9538fc9e04e82ec0e888ff0c3cffcfad152c77e57cd09351a98a", size = 374775, upload-time = "2026-05-28T12:01:33.136Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/8f/15e5a61d9f0a43902d36561d4f07cae6ae9f4716be825159fd72717f33af/rpds_py-2026.5.1-cp315-cp315t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e3a8ae58895ac107ed934a6bf51e5846f95c53b9b940c2c6d310838fd5846358", size = 380270, upload-time = "2026-05-28T12:01:34.574Z" },
+ { url = "https://files.pythonhosted.org/packages/02/c3/f859b12763a80540cdf2af0f15b19904cf756a71d7bdd3f82ff3e5b1bbf9/rpds_py-2026.5.1-cp315-cp315t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0957cf3c2b8632ec7aaebffebea8005b353cc2a237b6e2ae3c2cac0820704cfb", size = 495285, upload-time = "2026-05-28T12:01:36.127Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/c7/ff27c2ac8411d30b03b1829fd88cae8dad1a4d0da48dd25e57c4038042e6/rpds_py-2026.5.1-cp315-cp315t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c396c1304de421050b3681ea70f371874b54d41b0151e96109758144c231e30b", size = 389581, upload-time = "2026-05-28T12:01:37.635Z" },
+ { url = "https://files.pythonhosted.org/packages/6e/67/fe92ee32a6cc05c77228a2f8b1762e7124f386ec20ff83d0757b762d58d0/rpds_py-2026.5.1-cp315-cp315t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aad1bff7f666b9598e573815affd666aac6a13a585dde336f843e33350c7fadc", size = 376041, upload-time = "2026-05-28T12:01:39.307Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/91/b4d6685c27aba55bd82f25b278be8237038117d05f9659a6213ad3408130/rpds_py-2026.5.1-cp315-cp315t-manylinux_2_31_riscv64.whl", hash = "sha256:656a042550878f12d45752452d47094b7cfe5ad1e9d7b87b5a22ad3ae5ff8015", size = 383946, upload-time = "2026-05-28T12:01:41.043Z" },
+ { url = "https://files.pythonhosted.org/packages/bd/79/2c1d832a53c8e0f8e98fc970ec257b950fecd4f62be2ab7182b500a0cbc8/rpds_py-2026.5.1-cp315-cp315t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:73c4bd4f70294737b5206a3e8e30ccadbf8a60301831c8ea23eec5dbeea1ecfa", size = 405526, upload-time = "2026-05-28T12:01:43.032Z" },
+ { url = "https://files.pythonhosted.org/packages/78/c4/c98117b03c6a8581ab2c2dfccfe9a5ad82bd8128a3c28b46a6ad2d97c393/rpds_py-2026.5.1-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:43bca78665423cabae77146f2fe7ce55272b6c8d55d82cca83effd42c7e13972", size = 551165, upload-time = "2026-05-28T12:01:44.648Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/c1/bc479ca069200af730881b1bd525e3114b2b391a351509fcb1b772f28086/rpds_py-2026.5.1-cp315-cp315t-musllinux_1_2_i686.whl", hash = "sha256:42d0f20e85e549c870749d0e247f0c10d318a45b7e9676d575d2dcb04a1b2e66", size = 618778, upload-time = "2026-05-28T12:01:46.337Z" },
+ { url = "https://files.pythonhosted.org/packages/77/65/38ab2f90df44c2febfb63cc10ced40763d9b4bc94d173e734528663fe7f5/rpds_py-2026.5.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:b1be5c35683684d5331b93600c210e8367c254683d8a6df6bd21bd2da3a334fb", size = 581839, upload-time = "2026-05-28T12:01:48.109Z" },
+ { url = "https://files.pythonhosted.org/packages/15/2d/ce1f605fe036aadd460e5822e578c6c7ec3a860936cca37d6e0f299daa77/rpds_py-2026.5.1-cp315-cp315t-win32.whl", hash = "sha256:75808f6c38ce7749bb68cc2770161aae5045e6c6f6781a9782e74b93304399df", size = 207866, upload-time = "2026-05-28T12:01:49.648Z" },
+ { url = "https://files.pythonhosted.org/packages/79/cb/966040123eb102371559746908ef2c9471f4d43e17ec9a645a2258dab64b/rpds_py-2026.5.1-cp315-cp315t-win_amd64.whl", hash = "sha256:90bd6630002a1c7f09e7843dd79f0d24f3d2897cc25a753480917865d14f15b3", size = 225441, upload-time = "2026-05-28T12:01:51.408Z" },
+]
+
+[[package]]
+name = "ruff"
+version = "0.15.20"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/43/dc/35b341fc554ba02f217fc10da57d1a75168cfbcf75b0ef2202176d4c4f2d/ruff-0.15.20.tar.gz", hash = "sha256:1416eb04349192646b54de98f146c4f59afe37d0decfc02c3cbbf396f3a28566", size = 4755489, upload-time = "2026-06-25T17:20:37.578Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/94/d9/2d5014f0253ba541d2061d9fa7193f48e941c8b21bb88a7ff9bbe0bd0596/ruff-0.15.20-py3-none-linux_armv6l.whl", hash = "sha256:00e188c53e499c3c1637f73c91dcf2fb56d576cab76ce1be50a27c4e80e37078", size = 10839665, upload-time = "2026-06-25T17:19:44.702Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/d3/ac1798ba64f670698867fcfc591d50e7e421bef137db564858f619a30fcf/ruff-0.15.20-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:9ebd1fd9b9c95fc0bd7b2761aebec1f030013d2e193a2901b224af68fe47251b", size = 11208649, upload-time = "2026-06-25T17:19:48.787Z" },
+ { url = "https://files.pythonhosted.org/packages/47/47/d3ac899991202095dfcf3d5176be4272642be3cf981a2f1a30f72a2afb95/ruff-0.15.20-py3-none-macosx_11_0_arm64.whl", hash = "sha256:c5b16cdd67ca108185cd36dce98c576350c03b1660a751de725fb049193a0632", size = 10622638, upload-time = "2026-06-25T17:19:51.354Z" },
+ { url = "https://files.pythonhosted.org/packages/33/13/4e043fe30aa94d4ff5213a9881fc296d12960f5971b234a5263fdc225312/ruff-0.15.20-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3413bb3c3d2ca6a8208f1f4809cd2dca3c6de6d0b491c0e70847672bde6e6efd", size = 10984227, upload-time = "2026-06-25T17:19:54.044Z" },
+ { url = "https://files.pythonhosted.org/packages/76/e6/92e7bf40388bc5800073b96564f56264f7e48bfd1a498f5ced6ae6d5a769/ruff-0.15.20-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:bd7ec42b3bb3da066488db093308a69c4ac5ee6d2af333a86ba6e2eb2e7dd44b", size = 10622882, upload-time = "2026-06-25T17:19:57.037Z" },
+ { url = "https://files.pythonhosted.org/packages/13/7a/43460be3f24495a3aa46d4b16873e2c4941b3b5f0b00cf88c03b7b94b339/ruff-0.15.20-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:e1a36ad0eb77fba9aabfb69ede54de6f376d04ac18ebea022847046d340a8267", size = 11474808, upload-time = "2026-06-25T17:20:00.357Z" },
+ { url = "https://files.pythonhosted.org/packages/27/a0/f37077884873221c6b33b4ab49eb18f9f88e54a16a25a5bca59bef46dd66/ruff-0.15.20-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b6df3b1e4610432f0386dba04d853b5f08cbbc903410c6fcc02f620f05aff53c", size = 12293094, upload-time = "2026-06-25T17:20:03.446Z" },
+ { url = "https://files.pythonhosted.org/packages/a6/74/165545b60256a9704c21ac0ec4a0d07933b320812f9584836c9f4aca4292/ruff-0.15.20-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e89f198a1ea6ef0d727c1cf16088bc91a6cb0ab947dedc966715691647186eae", size = 11526176, upload-time = "2026-06-25T17:20:06.301Z" },
+ { url = "https://files.pythonhosted.org/packages/86/b1/a976a136d40ade83ce743578399865f57001003a409acadc0ecbb3051082/ruff-0.15.20-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:309809086c2acb67624950a3c8133e80f32d0d3e27106c0cd60ff26657c9f24b", size = 11520767, upload-time = "2026-06-25T17:20:09.191Z" },
+ { url = "https://files.pythonhosted.org/packages/19/0f/f032696cb01c9b54c0263fa393474d7758f1cdc021a01b04e3cbc2500999/ruff-0.15.20-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:2d2374caa2f2c2f9e2b7da0a50802cfb8b79f55a9b5e49379f564544fbf56487", size = 11500132, upload-time = "2026-06-25T17:20:13.602Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/f4/51b1a14bc69e8c224b15dab9cce8e99b425e0455d462caa2b3c9be2b6a8e/ruff-0.15.20-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:a1ed17b65293e0c2f22fc387bc13198a5de94bf4429589b0ff6946b0feaf21a3", size = 10943828, upload-time = "2026-06-25T17:20:16.635Z" },
+ { url = "https://files.pythonhosted.org/packages/71/4b/fe267640783cd02bf6c5cc290b1df1051be2ec294c678b5c15fe19e52343/ruff-0.15.20-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:f701305e66b38ea6c91882490eb73459796808e4c6362a1b765255e0cdcd4053", size = 10645418, upload-time = "2026-06-25T17:20:19.4Z" },
+ { url = "https://files.pythonhosted.org/packages/b0/c0/a65aa4ec2f5e87a1df32dc3ec1fede434fe3dfd5cbcf3b503cafc676ab54/ruff-0.15.20-py3-none-musllinux_1_2_i686.whl", hash = "sha256:5b9c0c367ad8e5d0d5b5b8537864c469a0a0e55417aadfbeca41fa61333be9f4", size = 11211770, upload-time = "2026-06-25T17:20:22.033Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/a4/0caa331d954ae2723d729d351c989cb4ca8b6077d5c6c2cb6de75e98c041/ruff-0.15.20-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:01cc00dd58f0df339d0e902219dd53990ea99996a0344e5d9cc8d45d5307e460", size = 11618698, upload-time = "2026-06-25T17:20:25.259Z" },
+ { url = "https://files.pythonhosted.org/packages/10/9b/5f14927848d2fd4aa891fd88d883788c5a7baba561c7874732364045708c/ruff-0.15.20-py3-none-win32.whl", hash = "sha256:ed65ef510e43a137207e0f01cfcf998aeddb1aeeda5c9d35023e910284d7cf21", size = 10857322, upload-time = "2026-06-25T17:20:28.612Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/f0/fe47c501f9dea92a26d788ff98bb5d92ed4cb4c88792c5c88af6b697dc8e/ruff-0.15.20-py3-none-win_amd64.whl", hash = "sha256:a525c81c70fb0380344dd1d8745d8cc1c890b7fc94a58d5a07bd8eb9557b8415", size = 11993274, upload-time = "2026-06-25T17:20:31.871Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/2b/9555445e1201d92b3195f45cdb153a0b68f24e0a4273f6e3d5ab46e212bb/ruff-0.15.20-py3-none-win_arm64.whl", hash = "sha256:2f5b2a6d614e8700388806a14996c40fab2c47b819ef57d790a34878858ed9ca", size = 11343498, upload-time = "2026-06-25T17:20:35.03Z" },
+]
+
+[[package]]
+name = "scarf"
+source = { editable = "." }
+dependencies = [
+ { name = "h5py" },
+ { name = "hnswlib" },
+ { name = "huggingface-hub" },
+ { name = "leidenalg" },
+ { name = "loguru" },
+ { name = "networkx" },
+ { name = "numba" },
+ { name = "numcodecs" },
+ { name = "numpy" },
+ { name = "obstore" },
+ { name = "packaging" },
+ { name = "pandas" },
+ { name = "scikit-learn" },
+ { name = "scipy" },
+ { name = "setuptools" },
+ { name = "sgtsnepi" },
+ { name = "statsmodels" },
+ { name = "threadpoolctl" },
+ { name = "tqdm" },
+ { name = "umap-learn" },
+ { name = "zarr" },
+ { name = "zstandard" },
+]
+
+[package.optional-dependencies]
+docs = [
+ { name = "jedi" },
+ { name = "jinja2" },
+ { name = "jupyter-cache" },
+ { name = "myst-nb" },
+ { name = "myst-parser" },
+ { name = "nbclient" },
+ { name = "sphinx" },
+ { name = "sphinx-autodoc-typehints" },
+ { name = "sphinx-book-theme" },
+ { name = "sphinx-copybutton" },
+ { name = "sphinx-external-toc" },
+ { name = "sphinx-reredirects" },
+ { name = "sphinx-tabs" },
+ { name = "sphinxcontrib-mermaid" },
+ { name = "topacedo" },
+]
+extra = [
+ { name = "anndata" },
+ { name = "ipython-autotime" },
+ { name = "ipywidgets" },
+ { name = "kneed" },
+ { name = "matplotlib" },
+ { name = "seaborn" },
+]
+test = [
+ { name = "anndata" },
+ { name = "pytest" },
+ { name = "pytest-cov" },
+ { name = "pytest-xdist" },
+ { name = "rdata" },
+ { name = "scikit-network" },
+ { name = "topacedo" },
+]
+
+[package.dev-dependencies]
+dev = [
+ { name = "mypy" },
+ { name = "ruff" },
+]
+docs-modal = [
+ { name = "modal" },
+]
+profiling = [
+ { name = "anndata" },
+ { name = "modal" },
+ { name = "pydantic" },
+]
+
+[package.metadata]
+requires-dist = [
+ { name = "anndata", marker = "extra == 'extra'", specifier = ">=0.12" },
+ { name = "anndata", marker = "extra == 'test'", specifier = ">=0.12" },
+ { name = "h5py", specifier = ">=3.11" },
+ { name = "hnswlib", specifier = ">=0.8" },
+ { name = "huggingface-hub", specifier = ">=1.24.0" },
+ { name = "ipython-autotime", marker = "extra == 'extra'", specifier = ">=0.3" },
+ { name = "ipywidgets", marker = "extra == 'extra'", specifier = ">=8" },
+ { name = "jedi", marker = "extra == 'docs'" },
+ { name = "jinja2", marker = "extra == 'docs'" },
+ { name = "jupyter-cache", marker = "extra == 'docs'", specifier = ">=1.0.1" },
+ { name = "kneed", marker = "extra == 'extra'", specifier = ">=0.8" },
+ { name = "leidenalg", specifier = ">=0.12" },
+ { name = "loguru", specifier = ">=0.7" },
+ { name = "matplotlib", marker = "extra == 'extra'", specifier = ">=3.9" },
+ { name = "myst-nb", marker = "extra == 'docs'" },
+ { name = "myst-parser", marker = "extra == 'docs'" },
+ { name = "nbclient", marker = "extra == 'docs'" },
+ { name = "networkx", specifier = ">=3" },
+ { name = "numba", specifier = ">=0.60" },
+ { name = "numcodecs", specifier = ">=0.16" },
+ { name = "numpy", specifier = ">=2" },
+ { name = "obstore", specifier = ">=0.5.1" },
+ { name = "packaging" },
+ { name = "pandas", specifier = ">=2.2.2" },
+ { name = "pytest", marker = "extra == 'test'", specifier = ">=7.0" },
+ { name = "pytest-cov", marker = "extra == 'test'", specifier = ">=4" },
+ { name = "pytest-xdist", marker = "extra == 'test'", specifier = ">=3" },
+ { name = "rdata", marker = "extra == 'test'", specifier = ">=1.1.0" },
+ { name = "scikit-learn", specifier = ">=1.5" },
+ { name = "scikit-network", marker = "extra == 'test'", specifier = ">=0.33.1" },
+ { name = "scipy", specifier = ">=1.13,!=1.17.0" },
+ { name = "seaborn", marker = "extra == 'extra'", specifier = ">=0.13" },
+ { name = "setuptools", specifier = ">=61" },
+ { name = "sgtsnepi" },
+ { name = "sphinx", marker = "extra == 'docs'" },
+ { name = "sphinx-autodoc-typehints", marker = "extra == 'docs'" },
+ { name = "sphinx-book-theme", marker = "extra == 'docs'" },
+ { name = "sphinx-copybutton", marker = "extra == 'docs'" },
+ { name = "sphinx-external-toc", marker = "extra == 'docs'" },
+ { name = "sphinx-reredirects", marker = "extra == 'docs'", specifier = ">=1.1.0" },
+ { name = "sphinx-tabs", marker = "extra == 'docs'" },
+ { name = "sphinxcontrib-mermaid", marker = "extra == 'docs'", specifier = ">=2.1.0" },
+ { name = "statsmodels", specifier = ">=0.14.2" },
+ { name = "threadpoolctl" },
+ { name = "topacedo", marker = "extra == 'docs'" },
+ { name = "topacedo", marker = "extra == 'test'", specifier = ">=0.2.10" },
+ { name = "tqdm", specifier = ">=4.60" },
+ { name = "umap-learn", specifier = ">=0.5.12" },
+ { name = "zarr", specifier = ">=3.2.0" },
+ { name = "zstandard", specifier = ">=0.25.0" },
+]
+provides-extras = ["extra", "test", "docs"]
+
+[package.metadata.requires-dev]
+dev = [
+ { name = "mypy", specifier = ">=2.1.0" },
+ { name = "ruff", specifier = ">=0.15.20" },
+]
+docs-modal = [{ name = "modal", specifier = ">=1.5.2" }]
+profiling = [
+ { name = "anndata", specifier = ">=0.12.18" },
+ { name = "modal", specifier = ">=1.5.2" },
+ { name = "pydantic", specifier = ">=2.13.4" },
+]
+
+[[package]]
+name = "scikit-learn"
+version = "1.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "joblib" },
+ { name = "narwhals" },
+ { name = "numpy" },
+ { name = "scipy" },
+ { name = "threadpoolctl" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/fa/6f/37092bdb25f712817231799fc5674d8e704066a8a70c1d2d40517e18b4ab/scikit_learn-1.9.0.tar.gz", hash = "sha256:8833266989d3a5110178a9fae30783675460724d0e1efb13b14901d2c660c557", size = 7750767, upload-time = "2026-06-02T11:54:32.706Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ac/20/75f915ff375d6249e6550ac740fdbbd66159a068fd3af1400ff62036b07a/scikit_learn-1.9.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2bd41b0d201bc81575531b96b713d3eb5e5f50fb0b82101ff0f92294fdc236ac", size = 8741122, upload-time = "2026-06-02T11:53:24.08Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/d5/2b5148f2279196775e1db2aeb85d14b70ac80e7e32b3b28e7ebeafb0901d/scikit_learn-1.9.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:5be45aa4a42a68a533913a6ed736cf309de2226411c79ef8d609a5456f1939b1", size = 8261512, upload-time = "2026-06-02T11:53:27.183Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/ee/5adbc77656b71f9456a2f5a7a9fdb4bcf9207a6b962889f1c2f9323afa4e/scikit_learn-1.9.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5e50ed4da51974e86e940690e9a3d82e729b62b5a49f7c9bac534d515d39d86f", size = 8837603, upload-time = "2026-06-02T11:53:30.328Z" },
+ { url = "https://files.pythonhosted.org/packages/6c/c2/63fdda36c56437eeb44aaf9493c8bcd62ce230ab1598924fc626ffbfa943/scikit_learn-1.9.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:056c92bb67ad4c28463c2f2653d9701449201e7e7a9e94e321be0f71c4fef2b8", size = 9132097, upload-time = "2026-06-02T11:53:33.456Z" },
+ { url = "https://files.pythonhosted.org/packages/83/a4/c8e67227c680e2259c8864ae72ff48b06e16a6f51253a22167aa02a8aa4e/scikit_learn-1.9.0-cp312-cp312-win_amd64.whl", hash = "sha256:4306775fad04cc4b472a1b15af1ae9cede1540fbfcc17fbce3767cd8dc7ae283", size = 8211173, upload-time = "2026-06-02T11:53:36.602Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/fd/3c0863792e98e67e9184aa4029288a175935eb65443afcd30d4f143450cf/scikit_learn-1.9.0-cp312-cp312-win_arm64.whl", hash = "sha256:26e22435f63bcdcf396b574273f29f13dd531f5ea035801f5be10ba1540a4e60", size = 7867451, upload-time = "2026-06-02T11:53:39.075Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/01/cf3310626b6d48d3e9be69a1223f9180360b5e6edb045f50fade723ce494/scikit_learn-1.9.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:80746d63bd4b6eaca54d36fe5feaf4d28bb38dc6f9470f81c7cad7c40155f119", size = 8705188, upload-time = "2026-06-02T11:53:41.964Z" },
+ { url = "https://files.pythonhosted.org/packages/3e/04/5acd7ae280c5f93b6ac5ef6cdec14eef4c8d1cd91d85b3292989c94d96b1/scikit_learn-1.9.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:5b934c45c252844a91d69fda3a34cff5e7307e1db10d77cb10a3980312c74713", size = 8228299, upload-time = "2026-06-02T11:53:44.817Z" },
+ { url = "https://files.pythonhosted.org/packages/0c/39/ffe829a5b8ecb40a518724a997794657fdc354ada5e8fe8e64d998c0bac9/scikit_learn-1.9.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:38c3dcb9a1ffb85505ec53d54c7b4aea0cff70050425a7760c2af661ac85df05", size = 8789690, upload-time = "2026-06-02T11:53:47.461Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/88/8dab5de10c638c083772a6be83a3d8106ced492f74a928c8693638e5bb50/scikit_learn-1.9.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:da76d09304a4706db7cc1e3ebaa3b6b98a67365cc11d2996c4f1e58ba47df714", size = 9087723, upload-time = "2026-06-02T11:53:50.702Z" },
+ { url = "https://files.pythonhosted.org/packages/20/3f/7917ca72464038f6240ec70c29f94862d08a34a74291ae4d4ec5eb8186a0/scikit_learn-1.9.0-cp313-cp313-win_amd64.whl", hash = "sha256:5808d98f15c6bf6d9d96d2348c1997392a5888ce7097e664105f930c4bca1277", size = 8184330, upload-time = "2026-06-02T11:53:53.396Z" },
+ { url = "https://files.pythonhosted.org/packages/78/c7/15739eb2f61fda3c54639e9942414e5a19ad8a8d1f5a3266afad7cb7df80/scikit_learn-1.9.0-cp313-cp313-win_arm64.whl", hash = "sha256:d77f54c017633791bc0225a43e2f8d03745fdcfe4880268fcc4df15f505dec2e", size = 7840653, upload-time = "2026-06-02T11:53:56.035Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/7d/c9a35cf59b20a86fec24d306f1547b78dec194b08d367ce2a3e4854169d9/scikit_learn-1.9.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:9656acd4e93f74e0b66c8a36c88830a99252dfa900044d36bc2212ae89a47162", size = 8713289, upload-time = "2026-06-02T11:53:58.788Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/a7/552a7821597c632b907f7bfe8f36f9f572777af8ef8a48353041cf8e091a/scikit_learn-1.9.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:24360002ae845e7866522b0a5bbf690802e7bc388cac8663502e78aa98598aa2", size = 8245141, upload-time = "2026-06-02T11:54:01.694Z" },
+ { url = "https://files.pythonhosted.org/packages/7d/79/f4a0c4fe9711154cddabf913471153af79056382ddc612cfe5ee0ff4b72e/scikit_learn-1.9.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5162ad10a418c8a282dde04c9aa06965de3e9a65f33c1440c0ae69bb1a09d913", size = 8847671, upload-time = "2026-06-02T11:54:04.448Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/af/4d72d9e475ac83719160c662619e4bf7b95c19507cd582e7d0167a3c3dae/scikit_learn-1.9.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1fea2cc5677ab49d6f5bade978c866da44957b712d92e9635e8b4f723013c3cb", size = 9118104, upload-time = "2026-06-02T11:54:07.205Z" },
+ { url = "https://files.pythonhosted.org/packages/a2/d5/6a58eea2cb9abbb9b3f2bb8b2cfb3243d1152d69f442d256c7af71304769/scikit_learn-1.9.0-cp314-cp314-win_amd64.whl", hash = "sha256:64fa347efc1c839c487433e40c5144d38c336e8a2b59c81aa8660373945c2673", size = 8290674, upload-time = "2026-06-02T11:54:10.087Z" },
+ { url = "https://files.pythonhosted.org/packages/65/5b/d4c879cf358f1187141cf90ced473f087183489090244f50c124a2ee478b/scikit_learn-1.9.0-cp314-cp314-win_arm64.whl", hash = "sha256:1b944b6db288f6b926e3650026ddafb988929de95d11fc2cc5fa117773c9ba42", size = 7978807, upload-time = "2026-06-02T11:54:12.769Z" },
+ { url = "https://files.pythonhosted.org/packages/8a/43/bfae3121ec67ae09150d453c442c7c1cc166e9aefe056e6ab3b7728a5cfc/scikit_learn-1.9.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:4ccacf04ca5f4b492158a5f28afe0ace43f81b2571e4b9a66d34848b46128949", size = 9031941, upload-time = "2026-06-02T11:54:15.436Z" },
+ { url = "https://files.pythonhosted.org/packages/75/b0/20a4546eb17f3b25d3c66df15810411c14ed5065bcfab50b53c96fb627b2/scikit_learn-1.9.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:ee1a8db2c18c08e34c7412d4b10be1cac214cd4ea7dc9715a6a327eb49a37c96", size = 8613528, upload-time = "2026-06-02T11:54:18.842Z" },
+ { url = "https://files.pythonhosted.org/packages/18/3c/e440e039bb82cd19004edaaad00acbde0fb9b461083c3ecf37941c557312/scikit_learn-1.9.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:147e9329ef0e39f75d4cffa02b2aa48d827832684926cd5210d9a2cb5c57246b", size = 8855050, upload-time = "2026-06-02T11:54:21.699Z" },
+ { url = "https://files.pythonhosted.org/packages/43/26/b341b8dab5998da6270a3a42c2152c578501354d36f944b5856757035ef8/scikit_learn-1.9.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5bad8f8b9950321b54c965fdcbac6c6c55e79e16646b49977bcf3668d3870a1a", size = 9097190, upload-time = "2026-06-02T11:54:24.454Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/de/b650b4d69b84468cfa2e28a3ff7b8103743029e6446ce1a97fe060ef688c/scikit_learn-1.9.0-cp314-cp314t-win_amd64.whl", hash = "sha256:78fc56eafd4edb9575d2d8950d1dd152061abb573341a1cb7e099fc40f6c6666", size = 8963204, upload-time = "2026-06-02T11:54:27.428Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/f3/ff83d76d7418112e5a61326443cdda87be3545dd8d6599c95b2481a4419e/scikit_learn-1.9.0-cp314-cp314t-win_arm64.whl", hash = "sha256:051075bda8b7aab87b1906ab3d4740a1e1224a19d7b3781a576736edc94e76aa", size = 8222661, upload-time = "2026-06-02T11:54:30.192Z" },
+]
+
+[[package]]
+name = "scikit-network"
+version = "0.33.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+ { name = "scipy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/46/6d/28b00fbef9ff7d8ba31861bf16705a1a74a1696fb65aab2a7c584f966bec/scikit_network-0.33.5.tar.gz", hash = "sha256:ae2149d9a280fdc4bbadd5f8a7b17c8af61c054bc3f834792bc61483e6783c12", size = 1784205, upload-time = "2025-11-19T09:45:14.402Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/72/32/f092fca9a3ae256e0608ec6a4d8830023ea4ae478c2e27e94cf5802824f0/scikit_network-0.33.5-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:ced625228be1632595d11adaf22d61d1d4c788909cd4d8c364e720b2814aac7f", size = 2874698, upload-time = "2025-11-19T09:44:52.765Z" },
+ { url = "https://files.pythonhosted.org/packages/4b/c7/5b2ce72f93b422af48a2b755fbcbab271bf980a6b46484f754d63978f1ff/scikit_network-0.33.5-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:808b625d28005c24b47cbdf65f780a3a355aa4080992e6b78f03434e873d06e6", size = 2854224, upload-time = "2025-11-19T09:44:55.574Z" },
+ { url = "https://files.pythonhosted.org/packages/86/02/974ae67f493ccf988108894e8a9dedfd00ca5113d2848e0b9fc2a4d18824/scikit_network-0.33.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f9f2d98059cd79bdb935ff6a638f1c3a12b0b1cb7ace9e1d3fb35476eeeaabaa", size = 7924650, upload-time = "2025-11-19T09:44:57.704Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/34/b67e48e111916a6f09fe29a971a9716c14f78525e0ab7c46e6a6538cf2f6/scikit_network-0.33.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aac2d3bc14214f02dac300624f5ec2650af9a98b52f304d300f0ec2813a0e544", size = 8012322, upload-time = "2025-11-19T09:45:00.079Z" },
+ { url = "https://files.pythonhosted.org/packages/c3/a2/49293b53b837b3248d19fffd41a9fd73dcb2eeabbab890cc4c8aa237b545/scikit_network-0.33.5-cp312-cp312-win_amd64.whl", hash = "sha256:2866b16aed9ef25ba42cb2f2e44ef2ad079337f336ce48d0604b55fa4af87688", size = 2746491, upload-time = "2025-11-19T09:45:01.997Z" },
+ { url = "https://files.pythonhosted.org/packages/d6/cd/0069244e970d27fa0ab0512394295a106605f00c271e85618182460d2c92/scikit_network-0.33.5-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:aa8b490a777e081dd6f69627a26b50cc4012f27fff683a1e4828819a88a5dcf2", size = 2862564, upload-time = "2025-11-19T09:45:03.93Z" },
+ { url = "https://files.pythonhosted.org/packages/51/37/85454864e50a65e528fe3b15eb3b41eb68b0c7f6d5c51c220b3198622ede/scikit_network-0.33.5-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:3615d073ba9ae1ae30dda2de747474cd23c86cededa82b317471ee9f9bebd1b2", size = 2842521, upload-time = "2025-11-19T09:45:06.31Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/b9/4023f35e430b51020f17b0f1d8933768cf1ed7cd1623cc089f5543048983/scikit_network-0.33.5-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2408d3f4c81256a3193d536aad4a6ffcfbb05d096abe6a9cc0b6b5e275df876d", size = 7871695, upload-time = "2025-11-19T09:45:08.101Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/ec/e50755f7459130ba745c42c37665c5ae9a7c7e357f43400b5b8b966f902e/scikit_network-0.33.5-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:526490a1e0e8e49ad0f4cca193f581d60082a2fa8c9a825eb0b6936050b0d02b", size = 7968762, upload-time = "2025-11-19T09:45:10.347Z" },
+ { url = "https://files.pythonhosted.org/packages/45/2a/616974a0adb9d04a791570e9371caaef14c54f8806b04dd59c80a7b60289/scikit_network-0.33.5-cp313-cp313-win_amd64.whl", hash = "sha256:722c15fcede5e07ac008354bbd6ef375e0f5bf1fd52bd40271775997be2fb715", size = 2742482, upload-time = "2025-11-19T09:45:12.843Z" },
+]
+
+[[package]]
+name = "scipy"
+version = "1.18.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/a7/25/c2700dfaf6442b4effaa91af24ebce5dc9d31bb4a69706313aae70d72cd0/scipy-1.18.0.tar.gz", hash = "sha256:67b2ad2ad54c72ca6d04975a9b2df8c3638c34ddd5b28738e94fc2b57929d378", size = 30774447, upload-time = "2026-06-19T15:01:43.456Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/6a/19/ca10ead60b0acc80b2b833c2c4a4f2ff753d0f58b811f70d911c7e94a25c/scipy-1.18.0-cp312-cp312-macosx_10_15_x86_64.whl", hash = "sha256:7bd21faaf5a1a3b2eff922d02db5f191b99a6518db9078a8fb23169f6d22259a", size = 31056519, upload-time = "2026-06-19T14:59:45.203Z" },
+ { url = "https://files.pythonhosted.org/packages/96/72/1e6442a00cd2924d361aa1b642ab6373ec35c6fabf311a760be9f76e0f13/scipy-1.18.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:265915e79107de9f946b855e50d7470d5893ec3f54b342e1aa6201cbdcd8bb6b", size = 28681889, upload-time = "2026-06-19T14:59:48.103Z" },
+ { url = "https://files.pythonhosted.org/packages/9b/2d/11dd93d21e147a73ba22bd75c0b9208d3a2e0ec76d53170ce7d9029b1015/scipy-1.18.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:9ab7b758be6940954a713ee466e2043e9f6e2ed965c1fce5c91039f4be3d90a9", size = 20423580, upload-time = "2026-06-19T14:59:50.665Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/01/93552f75e0d2a7dd115a45e59209c51e8d514daff02fc887d2623be06fe1/scipy-1.18.0-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:97b6cddaaee0a779ef6b5ca83c9604b27cc16b2b8fc22c142652df8793319fb8", size = 23054441, upload-time = "2026-06-19T14:59:53.564Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/23/21f5e703643d66f21faa6b4c73195bfcad70c55efcb4f1ab327cd7c4101a/scipy-1.18.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:52a96e21517c7292375c0e27dd796a811f03fcea5fd4d108fdfea8145dcf17ab", size = 33968720, upload-time = "2026-06-19T14:59:56.415Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/aa/1b939f6c67ed68635bb538e6752d3dacc02f66535182e939a89581a44e9c/scipy-1.18.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1f55797419e16e7f30cf88ffb3113ce0467f00cfe3f70d5c281730b21769bfc2", size = 35287115, upload-time = "2026-06-19T14:59:59.411Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/ff/eec46be7e9234208f801062b53e1983085eddebd693f6c9bfb03b459830d/scipy-1.18.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ad033410e2e0672ffdc1042110cef20e1c46f8fd0616cee1d44d8d58fad8fc11", size = 35577989, upload-time = "2026-06-19T15:00:02.235Z" },
+ { url = "https://files.pythonhosted.org/packages/84/ca/210d4759c7210bb7d269437421959b39a33434e2776b60c5cb8a763bb30a/scipy-1.18.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:4a55985d54c769c872e64b7f4c8a81cc30ef700cc04296abbbf3705439c126de", size = 37421717, upload-time = "2026-06-19T15:00:05.102Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/54/9a9edb45345bd6744da5ddfb6628e5d5185920494c6a67ec45b6381004cb/scipy-1.18.0-cp312-cp312-win_amd64.whl", hash = "sha256:71ccc8faa2dd16ac310233203474a8b5cb67f10dedd54a3116d34943f4b19132", size = 36597428, upload-time = "2026-06-19T15:00:08.112Z" },
+ { url = "https://files.pythonhosted.org/packages/99/0e/33f32a2a58987e26aec0f7df252cbbad1e90ae77bdbc76f40dd4ed0cf0ea/scipy-1.18.0-cp312-cp312-win_arm64.whl", hash = "sha256:d88363fd9d8fbd3511bd273f1a49efb2a540773ddf92a91d57498ce7dd7f3e76", size = 24351481, upload-time = "2026-06-19T15:00:11.103Z" },
+ { url = "https://files.pythonhosted.org/packages/05/52/9c0136c2de7ae0779b7b366447766cec6d9f0702c56bb8ffeb04c8fd3af4/scipy-1.18.0-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:09143f676d157d9f546d663504ef9c1becb819824f1afc018814176411942446", size = 31036107, upload-time = "2026-06-19T15:00:14.03Z" },
+ { url = "https://files.pythonhosted.org/packages/02/73/0291a64843270f4efb86cdcf2ee0f2048631b65ec6b405398b2b4dbf11bf/scipy-1.18.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:5efe260f69417b97ddae455bfb5a95e8359f7f66ad7fa9522a60feb66f169520", size = 28663303, upload-time = "2026-06-19T15:00:16.819Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/0f/10ffa0b697a572f4e0d48b92a88895d366422f019f723e7e14a84c050dac/scipy-1.18.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:68363b7eaacd8b5dd426df56d782cc156468ac79a127a1b87ca597d6e2e82197", size = 20404960, upload-time = "2026-06-19T15:00:19.635Z" },
+ { url = "https://files.pythonhosted.org/packages/7e/d2/e896cea21ba8edd6c81d4c55b1ffcc717e79698dcbebf9641b4cfb4c6622/scipy-1.18.0-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:c5557d8be5da8e41353fcd4d21491fdbab83b062fc579e94dc09a7c8ab4f669b", size = 23034074, upload-time = "2026-06-19T15:00:22.107Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/b2/e83ea34279a52c03374477c74006256ec78df65fc877baa4617d6de1d202/scipy-1.18.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0d13bca67c096d89fb95ced0d8921807300fce0275643aef9533cc63a0773468", size = 33942038, upload-time = "2026-06-19T15:00:24.964Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/af/e8fe5fb136f51e2b01678b92cb4106d10d8cd68ec147ead2e7cb0ac75398/scipy-1.18.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a46f9273dbd0eb1cefba61c9b8648b4dfe3cbc14a080176f9a73e44b8336dc7f", size = 35266390, upload-time = "2026-06-19T15:00:28.059Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/49/2c5cbb907b56695fc67517811d1db234dfd83381a84814ec220aded2794d/scipy-1.18.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5aba46108853ddfc77906b6557aac839d2b52e900c1d72a1180adaaab58d265f", size = 35551324, upload-time = "2026-06-19T15:00:31.014Z" },
+ { url = "https://files.pythonhosted.org/packages/bb/73/eda39f7a2d306ff0ffc574afd13c0bbb6d10a603d9a413998ee269487a80/scipy-1.18.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b6f758e35f12757b5d95c00bc6de2438e229c2664b7a92e96f205959d9f2dfa4", size = 37404785, upload-time = "2026-06-19T15:00:34.072Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/d2/ae881ee28d014f38e0ccbfd974a06a919ba9af34f1f74bf42b5301891d63/scipy-1.18.0-cp313-cp313-win_amd64.whl", hash = "sha256:1afac4a847207c7ff8efd321734a50b06d0280b3b2a2c0fc2f413101747ad7c7", size = 36554943, upload-time = "2026-06-19T15:00:36.903Z" },
+ { url = "https://files.pythonhosted.org/packages/70/3a/21154e2d54eb3639c6bf4dbae2e531c68356bfe95990daa30df33b30d556/scipy-1.18.0-cp313-cp313-win_arm64.whl", hash = "sha256:c5dbddf60e58c2312316d097271a8e73d40eaf2eabfa4d95ed7d3695bbf2ce7b", size = 24350911, upload-time = "2026-06-19T15:00:40.062Z" },
+ { url = "https://files.pythonhosted.org/packages/78/b5/915a19b3de2f7430062b509653563db1633ddbb6f021b06731521115d4e2/scipy-1.18.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:4c256ee70c0d1a8a2ace807e199ccd4e3f57037433842abb3fb36bc17eaa9578", size = 31036253, upload-time = "2026-06-19T15:00:43.216Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/88/b72def7262e150d16be13fca37a96481138d624e700340bc3362a7588929/scipy-1.18.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:2ef3abc54a4ffc53765374b0d5728532dfdd2585ed23f6b11c206a1f0b1b9af8", size = 28673758, upload-time = "2026-06-19T15:00:46.663Z" },
+ { url = "https://files.pythonhosted.org/packages/91/02/2e636a61a525632c373cf6a9c24442a3ffb79e364d38e98b32042964ac32/scipy-1.18.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:f2a6af57bd9e4a75d70e4117e78a1bbee84f79ae3fbb6d0111005d6ebcc4cb8d", size = 20415514, upload-time = "2026-06-19T15:00:49.399Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/b6/2135974442f6aba159d9d39d774a1c8cb19947016725d69fecc685df45bf/scipy-1.18.0-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:3f1ac564d3bf6c03d861d2cd87a1bea0da2887136f7fb1bf519c05a8971452d6", size = 23034398, upload-time = "2026-06-19T15:00:51.941Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/e6/ba89ec5abf6ee9257c0d1ec985573f3ae32742c24bc03e016388a40b1b15/scipy-1.18.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:40395a5fcd1abee49a5c7aaa98c29db393eedc835138560a588c47ec16156690", size = 33998032, upload-time = "2026-06-19T15:00:54.838Z" },
+ { url = "https://files.pythonhosted.org/packages/7f/c4/bc41eb19b0fd0db868f4132920879019318d80cc522ad8f2bca4611af808/scipy-1.18.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8ca01e8ae69f1b18e9a58d91afead31be3cef0dd905a10249dac559ee15460a0", size = 35283333, upload-time = "2026-06-19T15:00:58.152Z" },
+ { url = "https://files.pythonhosted.org/packages/53/a4/cbdeef6eb3830a8462a9d4ada814de5fc984345cc9ecf17cbec51a036f1e/scipy-1.18.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7a7f3b01647384dbc3a711e8c6778e0aabbe93959249fef5c7393396bcac0867", size = 35610216, upload-time = "2026-06-19T15:01:01.155Z" },
+ { url = "https://files.pythonhosted.org/packages/80/4d/b2b82502b65f661d1b789c1665dcdf315d5f12194e06fc0b37946294ebae/scipy-1.18.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6aa94e78ec192a30063a5e72e561c28af769dc311190b24fe91774eff1969709", size = 37418960, upload-time = "2026-06-19T15:01:04.155Z" },
+ { url = "https://files.pythonhosted.org/packages/93/3e/902d836831474b0ab5a37d16404f7bc5fafd9efba632890e271ba952635f/scipy-1.18.0-cp314-cp314-win_amd64.whl", hash = "sha256:2d8bbdc6c817f5b4006a54d799d4f5bab6f910193cbb9a1ff310833d4d270f61", size = 37288845, upload-time = "2026-06-19T15:01:07.822Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/43/8d73b337a3bdb14daa0314f0434210747c02d79d729ce1777574a817dcf6/scipy-1.18.0-cp314-cp314-win_arm64.whl", hash = "sha256:18e9575f1569b2c54174e6159d32942e03731177f63dce7975f0a0c88d102f5b", size = 24988971, upload-time = "2026-06-19T15:01:11.076Z" },
+ { url = "https://files.pythonhosted.org/packages/b4/b4/f11918b0508a2787031a0499a03fbe3546f3bb5ca05d01038c45b278c09a/scipy-1.18.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:f351e0dd702687d12a402b867a1b4146a256923e1c38317cbc472f6372b94707", size = 31399325, upload-time = "2026-06-19T15:01:13.723Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/d1/1f287b57c0ff0ee5185dff3946d92c8017d39b0e431f0ae79a3ff1859512/scipy-1.18.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:7c7a51b33ce387193c97f228320cf8e87361daa1bba750638677729598b3e677", size = 29092110, upload-time = "2026-06-19T15:01:16.908Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/1a/7b74eb6c392fdcb27d414c0e7558a6d0231eb3b6d73571f479bb81ea8794/scipy-1.18.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:84031d7b052a54fae2f8632e0ec802073d385476eb9a63079bce6e23ef9283d4", size = 20833811, upload-time = "2026-06-19T15:01:20.488Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/ad/f3941716320a7b9cb4d68734a903b45fe16eff5fb7da7e16f2e619304979/scipy-1.18.0-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:56abf29a7c067dde59be8b9a22d606a4ea1b2f2a4b756d9d903c62818f5dacce", size = 23396644, upload-time = "2026-06-19T15:01:23.364Z" },
+ { url = "https://files.pythonhosted.org/packages/22/22/1446b62ffe07f9719b7d9b1b6a4e05a772833ae8f441fe4c22c34c9b250f/scipy-1.18.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1ad44305cfa24b1ba5803cbbebf033590ccbac1aa5d612d727b785325ab408b0", size = 34079318, upload-time = "2026-06-19T15:01:26.002Z" },
+ { url = "https://files.pythonhosted.org/packages/56/3b/b87da667098bb470fa30c7011b0ba351ee976dd395c78798c66e941665a3/scipy-1.18.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:945c1761b93f38d7f99ae81ae80c63e621471608c7eeead563f6df025585cd58", size = 35324320, upload-time = "2026-06-19T15:01:28.881Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/a1/c7932f91909759b0267f75fdea34e91309f96b895757534b76a90b6b4344/scipy-1.18.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:1a4441f15d620578772a49e5ab48c0ee1f7a0220e387110283062729136b2553", size = 35699541, upload-time = "2026-06-19T15:01:31.968Z" },
+ { url = "https://files.pythonhosted.org/packages/f7/86/5185061a1fcc41d18c5dc2463969b3a3964b31d9ac67b2fb05d4c7ff7670/scipy-1.18.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9aac6192fac56bf2ca534389d24623f07b39ff83317d58287285e7fbd622ff76", size = 37472480, upload-time = "2026-06-19T15:01:35.136Z" },
+ { url = "https://files.pythonhosted.org/packages/31/8e/f04c68e39919a010d34f2ee1367fd705b0a25a02f609d755f0bfbc0a15fc/scipy-1.18.0-cp314-cp314t-win_amd64.whl", hash = "sha256:e40baea28ae7f5475c779741e2d90b1247c78531207b49c7030e698ff81cee3f", size = 37365390, upload-time = "2026-06-19T15:01:38.091Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/19/969dc072906c84dd0a3b05dcf57ea750936087d7873549e408b35cfc3f97/scipy-1.18.0-cp314-cp314t-win_arm64.whl", hash = "sha256:368e0a705903c466aa5f08eefb39e6b1b6b2d659e7352a31fd9e2438365be0f8", size = 25279661, upload-time = "2026-06-19T15:01:40.817Z" },
+]
+
+[[package]]
+name = "scverse-misc"
+version = "0.1.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "session-info2" },
+ { name = "typing-extensions", marker = "python_full_version < '3.13'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/e2/9b/e3d49620c7bacecad1de9c6474fb453969cc715202e5fc4f7929bfae4e0f/scverse_misc-0.1.1.tar.gz", hash = "sha256:5001572763a88d962f7e40309d1e47a62ae3c358aec93e377af998e4c7e6dd97", size = 42585, upload-time = "2026-06-22T13:01:27.68Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/cd/d7/324b9bdd6fa89fefb4765494dd20d76c06d0f4f0aa8e46233e122c9f9f21/scverse_misc-0.1.1-py3-none-any.whl", hash = "sha256:d402e470a6921c110ab44a63f2e606204d6f6ef25626a3cb2d7b567832148369", size = 22662, upload-time = "2026-06-22T13:01:26.43Z" },
+]
+
+[[package]]
+name = "seaborn"
+version = "0.13.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "matplotlib" },
+ { name = "numpy" },
+ { name = "pandas" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/86/59/a451d7420a77ab0b98f7affa3a1d78a313d2f7281a57afb1a34bae8ab412/seaborn-0.13.2.tar.gz", hash = "sha256:93e60a40988f4d65e9f4885df477e2fdaff6b73a9ded434c1ab356dd57eefff7", size = 1457696, upload-time = "2024-01-25T13:21:52.551Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/83/11/00d3c3dfc25ad54e731d91449895a79e4bf2384dc3ac01809010ba88f6d5/seaborn-0.13.2-py3-none-any.whl", hash = "sha256:636f8336facf092165e27924f223d3c62ca560b1f2bb5dff7ab7fad265361987", size = 294914, upload-time = "2024-01-25T13:21:49.598Z" },
+]
+
+[[package]]
+name = "session-info2"
+version = "0.4.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/50/17/c6a81d91781734bd10d7842c32de665f34395b4db234abdc80dac271632b/session_info2-0.4.1.tar.gz", hash = "sha256:3bb2bf7b73b2e13a1737e9aa91a6dae55e2c49e83bee973f24245f31ae264a1f", size = 25207, upload-time = "2026-04-08T11:30:55.959Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5e/d7/6893c9c2a52e4bcbeca2a2bf2aee970a686cc7bf555f97db13b00f35250e/session_info2-0.4.1-py3-none-any.whl", hash = "sha256:423b3f6bb7023433cfc3f791a6fdbb6a2cfbe226770ae6c127c3b2c4cf5a9d56", size = 17696, upload-time = "2026-04-08T11:30:54.707Z" },
+]
+
+[[package]]
+name = "setuptools"
+version = "82.0.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/4f/db/cfac1baf10650ab4d1c111714410d2fbb77ac5a616db26775db562c8fab2/setuptools-82.0.1.tar.gz", hash = "sha256:7d872682c5d01cfde07da7bccc7b65469d3dca203318515ada1de5eda35efbf9", size = 1152316, upload-time = "2026-03-09T12:47:17.221Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9d/76/f789f7a86709c6b087c5a2f52f911838cad707cc613162401badc665acfe/setuptools-82.0.1-py3-none-any.whl", hash = "sha256:a59e362652f08dcd477c78bb6e7bd9d80a7995bc73ce773050228a348ce2e5bb", size = 1006223, upload-time = "2026-03-09T12:47:15.026Z" },
+]
+
+[[package]]
+name = "sgtsnepi"
+version = "0.6"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "nextprod" },
+ { name = "numpy" },
+ { name = "scipy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/f7/6d/e144fee6fba769fed473224ab0f3a9500b79b39e193a4c4a3f4f90458ecd/sgtsnepi-0.6.tar.gz", hash = "sha256:adb6dd2992ea2b04553b40da822f30d7ebd1b7612eca392bb584c81ef36c298d", size = 17269500, upload-time = "2026-02-07T19:33:59.039Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f9/d8/ec1e97f38ebd00308e2d25acda255c174aeecadc7c5b95cea4982a021032/sgtsnepi-0.6-cp312-cp312-macosx_15_0_arm64.whl", hash = "sha256:59c26c492ba7da429bb2544b2fc0561b8f6be84570f090620d0564a20217f9b3", size = 863750, upload-time = "2026-02-07T19:33:41.741Z" },
+ { url = "https://files.pythonhosted.org/packages/e1/6d/8093e551cffbd4bc4ce8249c62b8ed3f8f20f954c2c4b7078867d3923d02/sgtsnepi-0.6-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:69b444d12ac2dfeb0665b7bcb92fce75429a5e319b7093e580fd5a35b701d323", size = 1529657, upload-time = "2026-02-07T19:33:43.794Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/cf/fd2f4995e2507164924f6ad3ef095db302fbdbfa620a60b7dcafebfc539e/sgtsnepi-0.6-cp313-cp313-macosx_15_0_arm64.whl", hash = "sha256:91e569f1543e8ada83bfe1184db6dfba455eac9050c324f743f81897eaab7e93", size = 863839, upload-time = "2026-02-07T19:33:46.111Z" },
+ { url = "https://files.pythonhosted.org/packages/cd/5a/d1d7d8edaf5547bbd9cf541784e29150fd38d99b74192d5199a8c4c51d68/sgtsnepi-0.6-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:38bf757c4d430151f48d73eeef3b39ff14a3b807eea00f1319c61e25485e69e4", size = 1529598, upload-time = "2026-02-07T19:33:47.826Z" },
+ { url = "https://files.pythonhosted.org/packages/bc/d6/c6151851e8b5af4a165eb7ab9b349d3121124cf4028eaad02af4ae0257a3/sgtsnepi-0.6-cp314-cp314-macosx_15_0_arm64.whl", hash = "sha256:bc3d46ff337f41934c1cb56082f4951ce78262bfa6e67678927d5287cdf7d1fe", size = 864069, upload-time = "2026-02-07T19:33:50.078Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/21/eccb2b171e2d8a8c19ea3bee1d1a2273c3cb917b3280c730b386666e1067/sgtsnepi-0.6-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:130a763021e6ee8e9e17bcf81efa814e5cb18398a5e476dc8b6e977a846336fd", size = 1529553, upload-time = "2026-02-07T19:33:51.834Z" },
+ { url = "https://files.pythonhosted.org/packages/be/b7/c1f786637c570089f328a0afeff4ad01a365882f825fcbfdb14555aae971/sgtsnepi-0.6-cp314-cp314t-macosx_15_0_arm64.whl", hash = "sha256:e2a9a8fadc206147bf1126ed84343a34a04708510f8d37f3dbc40a8bbfe1f495", size = 867814, upload-time = "2026-02-07T19:33:53.171Z" },
+ { url = "https://files.pythonhosted.org/packages/3b/4b/6fa22d2231b890ee03432f0428f183949740b5a5b04c77fd9030eabc3ceb/sgtsnepi-0.6-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4c31d432aae3399bbd39e29ee899b5ccf868a7a351e32eb09e0cb7f4ca6c817c", size = 1530918, upload-time = "2026-02-07T19:33:54.717Z" },
+]
+
+[[package]]
+name = "six"
+version = "1.17.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" },
+]
+
+[[package]]
+name = "snowballstemmer"
+version = "3.1.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/43/f8/0a71edf031f03c40db17503cb8ca78a69a171254e568e7db241b0ab57ea1/snowballstemmer-3.1.1.tar.gz", hash = "sha256:e07bbc54a0d798fe6010a12398422e62a8bfbba95c394fd0956ef58cb4d3e260", size = 123314, upload-time = "2026-06-03T00:56:40.194Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/4c/07/2ebca9b11fb9be7340a818d8d6f63feaebb146be2c4afbd6061701d6df6e/snowballstemmer-3.1.1-py3-none-any.whl", hash = "sha256:7e207fa178741da09cdee59d3ecec3827ad5f92b1fc5c9ff3755b639f71f5752", size = 104164, upload-time = "2026-06-03T00:56:38.614Z" },
+]
+
+[[package]]
+name = "soupsieve"
+version = "2.8.4"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/47/2c/0a5f6f8ee0d5589e48c7640213ed5175d52cf540a06725b628cc1a45d6ce/soupsieve-2.8.4.tar.gz", hash = "sha256:e121fd02e975c695e4e9e8774a5ee35d74714b59307868dcc5319ad2d9e3328e", size = 121110, upload-time = "2026-05-24T13:55:57.154Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5e/f5/0c41cb68dcae6b7de4fac4188a3a9589e21fb31df21ea3a2e888db95e6c9/soupsieve-2.8.4-py3-none-any.whl", hash = "sha256:e7e6b0769c8f51ed59acab6e994b00621096cfb1c640a7509295987388fbaf65", size = 37304, upload-time = "2026-05-24T13:55:55.406Z" },
+]
+
+[[package]]
+name = "sphinx"
+version = "9.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "alabaster" },
+ { name = "babel" },
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+ { name = "docutils" },
+ { name = "imagesize" },
+ { name = "jinja2" },
+ { name = "packaging" },
+ { name = "pygments" },
+ { name = "requests" },
+ { name = "roman-numerals" },
+ { name = "snowballstemmer" },
+ { name = "sphinxcontrib-applehelp" },
+ { name = "sphinxcontrib-devhelp" },
+ { name = "sphinxcontrib-htmlhelp" },
+ { name = "sphinxcontrib-jsmath" },
+ { name = "sphinxcontrib-qthelp" },
+ { name = "sphinxcontrib-serializinghtml" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/cd/bd/f08eb0f4eed5c83f1ba2a3bd18f7745a2b1525fad70660a1c00224ec468a/sphinx-9.1.0.tar.gz", hash = "sha256:7741722357dd75f8190766926071fed3bdc211c74dd2d7d4df5404da95930ddb", size = 8718324, upload-time = "2025-12-31T15:09:27.646Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/73/f7/b1884cb3188ab181fc81fa00c266699dab600f927a964df02ec3d5d1916a/sphinx-9.1.0-py3-none-any.whl", hash = "sha256:c84fdd4e782504495fe4f2c0b3413d6c2bf388589bb352d439b2a3bb99991978", size = 3921742, upload-time = "2025-12-31T15:09:25.561Z" },
+]
+
+[[package]]
+name = "sphinx-autodoc-typehints"
+version = "3.12.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "sphinx" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/99/f2/d99787d34d881b2352c04ce02bcdf0a61adf54b389c86d438b7a8ac3ae20/sphinx_autodoc_typehints-3.12.0.tar.gz", hash = "sha256:6571eb33c72cdc616a9945730fcb43c5bcf3685e79f1e8aea144735e43d5230d", size = 83874, upload-time = "2026-06-25T15:44:47.786Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3e/28/fa0f2ff73b8ca7987ebaa107d369c95459741ac73567e5079b4a881a981b/sphinx_autodoc_typehints-3.12.0-py3-none-any.whl", hash = "sha256:1a639b7cf71be13c4d8a4f0b552dcfcdf32ff39ac33ecb47398acb85863c2faf", size = 42548, upload-time = "2026-06-25T15:44:46.306Z" },
+]
+
+[[package]]
+name = "sphinx-book-theme"
+version = "1.2.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "pydata-sphinx-theme" },
+ { name = "sphinx" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/eb/f7/154786f3cfb7692cd7acc24b6dfe4dcd1146b66f376b17df9e47125555e9/sphinx_book_theme-1.2.0.tar.gz", hash = "sha256:4a7ebfc7da4395309ac942ddfc38fbec5c5254c3be22195e99ad12586fbda9e3", size = 443962, upload-time = "2026-03-09T23:20:30.442Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/02/bf/6f506a37c7f8ecc4576caf9486e303c7af249f6d70447bb51dde9d78cb99/sphinx_book_theme-1.2.0-py3-none-any.whl", hash = "sha256:709605d308e1991c5ef0cf19c481dbe9084b62852e317fafab74382a0ee7ccfa", size = 455936, upload-time = "2026-03-09T23:20:28.788Z" },
+]
+
+[[package]]
+name = "sphinx-copybutton"
+version = "0.5.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "sphinx" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/fc/2b/a964715e7f5295f77509e59309959f4125122d648f86b4fe7d70ca1d882c/sphinx-copybutton-0.5.2.tar.gz", hash = "sha256:4cf17c82fb9646d1bc9ca92ac280813a3b605d8c421225fd9913154103ee1fbd", size = 23039, upload-time = "2023-04-14T08:10:22.998Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9e/48/1ea60e74949eecb12cdd6ac43987f9fd331156388dcc2319b45e2ebb81bf/sphinx_copybutton-0.5.2-py3-none-any.whl", hash = "sha256:fb543fd386d917746c9a2c50360c7905b605726b9355cd26e9974857afeae06e", size = 13343, upload-time = "2023-04-14T08:10:20.844Z" },
+]
+
+[[package]]
+name = "sphinx-external-toc"
+version = "1.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "click" },
+ { name = "pyyaml" },
+ { name = "sphinx" },
+ { name = "sphinx-multitoc-numbering" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/c5/f8/85bcd2f1c142e580a1394c18920506d9399b8e8e97e4899bbee9c74a896e/sphinx_external_toc-1.1.0.tar.gz", hash = "sha256:f81833865006f6b4a9b2550a2474a6e3d7e7f2cb23ba23309260577ea65552f6", size = 37194, upload-time = "2026-01-16T13:15:59.03Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/9f/80/1704c9179012e289dee2178354e385277ea51f4fa827c4bf7e36c77b0f4b/sphinx_external_toc-1.1.0-py3-none-any.whl", hash = "sha256:26c390b8d85aa641366fed2d3674910ec6820f48b91027affef485a2655ad7d0", size = 30609, upload-time = "2026-01-16T13:15:57.926Z" },
+]
+
+[[package]]
+name = "sphinx-multitoc-numbering"
+version = "0.1.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "sphinx" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/37/1e/577bae038372885ebc34bd8c0f290295785a0250cac6528eb6d50e4b92d5/sphinx-multitoc-numbering-0.1.3.tar.gz", hash = "sha256:c9607671ac511236fa5d61a7491c1031e700e8d498c9d2418e6c61d1251209ae", size = 4542, upload-time = "2021-03-15T12:01:43.758Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ec/9f/902f2030674cd9473fdbe5a2c2dec2618c27ec853484c35f82cf8df40ece/sphinx_multitoc_numbering-0.1.3-py3-none-any.whl", hash = "sha256:33d2e707a9b2b8ad636b3d4302e658a008025106fe0474046c651144c26d8514", size = 4616, upload-time = "2021-03-15T12:01:42.419Z" },
+]
+
+[[package]]
+name = "sphinx-reredirects"
+version = "1.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "sphinx" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/1b/8d/0e39fe2740d7d71417edf9a6424aa80ca2c27c17fc21282cdc39f90d5a40/sphinx_reredirects-1.1.0.tar.gz", hash = "sha256:fb9b195335ab14b43f8273287d0c7eeb637ba6c56c66581c11b47202f6718b29", size = 614624, upload-time = "2025-12-22T08:28:02.792Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/51/81/b5dd07067f3daac6d23687ec737b2d593740671ebcd145830c8f92d381c5/sphinx_reredirects-1.1.0-py3-none-any.whl", hash = "sha256:4b5692273c72cd2d4d917f4c6f87d5919e4d6114a752d4be033f7f5f6310efd9", size = 6351, upload-time = "2025-12-22T08:27:59.724Z" },
+]
+
+[[package]]
+name = "sphinx-tabs"
+version = "3.5.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "docutils" },
+ { name = "pygments" },
+ { name = "sphinx" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ce/30/ca5b0de830f369968d8e3483dd45a8908fd10169c05cd9837f0bd075982e/sphinx_tabs-3.5.0.tar.gz", hash = "sha256:91dba1187e4c35fd37380a56ac228bbd54c6c649b2351829f3bf033718277537", size = 17006, upload-time = "2026-03-03T23:00:30.404Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/2e/45/6adc5efeb19fd5fed4027e520b5c668ce58236a2b271ade5533c4c116276/sphinx_tabs-3.5.0-py3-none-any.whl", hash = "sha256:154be49de4d5c8249ea08c5d9bf88ca8f9c31e00a178305a93cbc33e000339e5", size = 9871, upload-time = "2026-03-03T23:00:28.89Z" },
+]
+
+[[package]]
+name = "sphinxcontrib-applehelp"
+version = "2.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ba/6e/b837e84a1a704953c62ef8776d45c3e8d759876b4a84fe14eba2859106fe/sphinxcontrib_applehelp-2.0.0.tar.gz", hash = "sha256:2f29ef331735ce958efa4734873f084941970894c6090408b079c61b2e1c06d1", size = 20053, upload-time = "2024-07-29T01:09:00.465Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/5d/85/9ebeae2f76e9e77b952f4b274c27238156eae7979c5421fba91a28f4970d/sphinxcontrib_applehelp-2.0.0-py3-none-any.whl", hash = "sha256:4cd3f0ec4ac5dd9c17ec65e9ab272c9b867ea77425228e68ecf08d6b28ddbdb5", size = 119300, upload-time = "2024-07-29T01:08:58.99Z" },
+]
+
+[[package]]
+name = "sphinxcontrib-devhelp"
+version = "2.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/f6/d2/5beee64d3e4e747f316bae86b55943f51e82bb86ecd325883ef65741e7da/sphinxcontrib_devhelp-2.0.0.tar.gz", hash = "sha256:411f5d96d445d1d73bb5d52133377b4248ec79db5c793ce7dbe59e074b4dd1ad", size = 12967, upload-time = "2024-07-29T01:09:23.417Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/35/7a/987e583882f985fe4d7323774889ec58049171828b58c2217e7f79cdf44e/sphinxcontrib_devhelp-2.0.0-py3-none-any.whl", hash = "sha256:aefb8b83854e4b0998877524d1029fd3e6879210422ee3780459e28a1f03a8a2", size = 82530, upload-time = "2024-07-29T01:09:21.945Z" },
+]
+
+[[package]]
+name = "sphinxcontrib-htmlhelp"
+version = "2.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/43/93/983afd9aa001e5201eab16b5a444ed5b9b0a7a010541e0ddfbbfd0b2470c/sphinxcontrib_htmlhelp-2.1.0.tar.gz", hash = "sha256:c9e2916ace8aad64cc13a0d233ee22317f2b9025b9cf3295249fa985cc7082e9", size = 22617, upload-time = "2024-07-29T01:09:37.889Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/0a/7b/18a8c0bcec9182c05a0b3ec2a776bba4ead82750a55ff798e8d406dae604/sphinxcontrib_htmlhelp-2.1.0-py3-none-any.whl", hash = "sha256:166759820b47002d22914d64a075ce08f4c46818e17cfc9470a9786b759b19f8", size = 98705, upload-time = "2024-07-29T01:09:36.407Z" },
+]
+
+[[package]]
+name = "sphinxcontrib-jsmath"
+version = "1.0.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b2/e8/9ed3830aeed71f17c026a07a5097edcf44b692850ef215b161b8ad875729/sphinxcontrib-jsmath-1.0.1.tar.gz", hash = "sha256:a9925e4a4587247ed2191a22df5f6970656cb8ca2bd6284309578f2153e0c4b8", size = 5787, upload-time = "2019-01-21T16:10:16.347Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/c2/42/4c8646762ee83602e3fb3fbe774c2fac12f317deb0b5dbeeedd2d3ba4b77/sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl", hash = "sha256:2ec2eaebfb78f3f2078e73666b1415417a116cc848b72e5172e596c871103178", size = 5071, upload-time = "2019-01-21T16:10:14.333Z" },
+]
+
+[[package]]
+name = "sphinxcontrib-mermaid"
+version = "2.1.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "jinja2" },
+ { name = "pyyaml" },
+ { name = "sphinx" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/07/9d/bf3a48a657682c7e0445c71d916bca7ab8194454276cc22b16e7f974e502/sphinxcontrib_mermaid-2.1.0.tar.gz", hash = "sha256:13c5f9ac395cb6abf403eca34e228dc9fb3a30c9d960dbf3e40e9a8cef969549", size = 21695, upload-time = "2026-07-18T23:08:11.265Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f3/bd/d3348d62296e73a91420f0edf671450c9003ecd8704da40b1e3d9b868459/sphinxcontrib_mermaid-2.1.0-py3-none-any.whl", hash = "sha256:417cd144ec4b28852f46ba653f02ce8e538881c812111671a4c30344e87f2112", size = 16190, upload-time = "2026-07-18T23:08:10.098Z" },
+]
+
+[[package]]
+name = "sphinxcontrib-qthelp"
+version = "2.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/68/bc/9104308fc285eb3e0b31b67688235db556cd5b0ef31d96f30e45f2e51cae/sphinxcontrib_qthelp-2.0.0.tar.gz", hash = "sha256:4fe7d0ac8fc171045be623aba3e2a8f613f8682731f9153bb2e40ece16b9bbab", size = 17165, upload-time = "2024-07-29T01:09:56.435Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/27/83/859ecdd180cacc13b1f7e857abf8582a64552ea7a061057a6c716e790fce/sphinxcontrib_qthelp-2.0.0-py3-none-any.whl", hash = "sha256:b18a828cdba941ccd6ee8445dbe72ffa3ef8cbe7505d8cd1fa0d42d3f2d5f3eb", size = 88743, upload-time = "2024-07-29T01:09:54.885Z" },
+]
+
+[[package]]
+name = "sphinxcontrib-serializinghtml"
+version = "2.0.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/3b/44/6716b257b0aa6bfd51a1b31665d1c205fb12cb5ad56de752dfa15657de2f/sphinxcontrib_serializinghtml-2.0.0.tar.gz", hash = "sha256:e9d912827f872c029017a53f0ef2180b327c3f7fd23c87229f7a8e8b70031d4d", size = 16080, upload-time = "2024-07-29T01:10:09.332Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/52/a7/d2782e4e3f77c8450f727ba74a8f12756d5ba823d81b941f1b04da9d033a/sphinxcontrib_serializinghtml-2.0.0-py3-none-any.whl", hash = "sha256:6e2cb0eef194e10c27ec0023bfeb25badbbb5868244cf5bc5bdc04e4464bf331", size = 92072, upload-time = "2024-07-29T01:10:08.203Z" },
+]
+
+[[package]]
+name = "sqlalchemy"
+version = "2.0.51"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "greenlet", marker = "platform_machine == 'AMD64' or platform_machine == 'WIN32' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'ppc64le' or platform_machine == 'win32' or platform_machine == 'x86_64'" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/02/f1/a7a892f18d4d224e6b26f706531eafccc41e37594d37d304786969ee13cb/sqlalchemy-2.0.51.tar.gz", hash = "sha256:804dccd8a4a6242c4e30ad961e540e18a588f6527202f2d6791b01845d59fdc9", size = 9912201, upload-time = "2026-06-15T15:41:20.012Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d5/70/e868bc5412acd101a8280f25c95f10eeae0771c4eb806b02491142810ee8/sqlalchemy-2.0.51-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:7d78702b26ba1c18b2d0fb2ea940ba7f17a9581b42e8361ff93920ebbee1235a", size = 2160291, upload-time = "2026-06-15T16:08:48.918Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/1c/71ee0f8a6b9d7316a1ccd30430b4c62b6c2e36adc96017a4e3a72dce49d6/sqlalchemy-2.0.51-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:581921d849d6e6f994d560389192955e80e2950e18fcdfe2ccea863e01158e6e", size = 3343835, upload-time = "2026-06-15T16:19:42.613Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/7c/7ab9f9aadc5944fdd06612484ed7918fe376ad871a5f50404dc1536e0194/sqlalchemy-2.0.51-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1d21ce524ab86c23046e992a5b81cb54c21079c6df6e78b8fc77d77cac70a6b9", size = 3358470, upload-time = "2026-06-15T16:26:38.011Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/7d/ff77169fee6186de145a7f2b87006c39638391130abbab2b1f63ac6ea583/sqlalchemy-2.0.51-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c5d98a2709840027f5a347c3af0a7c3d5f6c1ff93af2ca1c54494e23cba8f389", size = 3289874, upload-time = "2026-06-15T16:19:45.212Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/3b/6c505903710d781b55bc3141ee34a062bf9745a6b5bc7333305b9ed63b33/sqlalchemy-2.0.51-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:1181256e0f16479691b5616d36375dc2620ad8332b25978763c3d206ad3f3f1d", size = 3321692, upload-time = "2026-06-15T16:26:39.747Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/b7/c5ffe50aa2f4d947c9250e1519d939260329a07fe6272edfccd784b3d007/sqlalchemy-2.0.51-cp312-cp312-win32.whl", hash = "sha256:9f380393be5abeb6815f68fd39271b95127173511b6706b0a630a9995d53f8f5", size = 2119674, upload-time = "2026-06-15T16:23:09.543Z" },
+ { url = "https://files.pythonhosted.org/packages/25/dc/46a65916af68a06ef6b972c6050ba4c8f97070fe3fb33097d34229d9bef6/sqlalchemy-2.0.51-cp312-cp312-win_amd64.whl", hash = "sha256:2cf39aabdf48e87c1c2c2ed6d20d33ffa0733b3071ce9c5f66357947dd009080", size = 2146670, upload-time = "2026-06-15T16:23:11.048Z" },
+ { url = "https://files.pythonhosted.org/packages/54/fe/a210d52fd1a90ecfae8a78e9d8b27e18d733d60818a8bf250ff690b75120/sqlalchemy-2.0.51-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7c2056838b6685b72fdb36c99996cf862753461a62f2e84f4196371d3b2d6a07", size = 2157184, upload-time = "2026-06-15T16:08:50.374Z" },
+ { url = "https://files.pythonhosted.org/packages/17/6b/2dce8369b199cb855110e056032f94a9f66dacc2237d3d39c115a86eac56/sqlalchemy-2.0.51-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:483b11bd46bf35fc14c52faf338b04300c9e6ce554bce9b11be85bfec3bc3195", size = 3284735, upload-time = "2026-06-15T16:19:46.934Z" },
+ { url = "https://files.pythonhosted.org/packages/53/ff/dbc495b8a14da840faffb353857a72d4190113cac33727906fb997047f0f/sqlalchemy-2.0.51-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1bed1ee8b01da6088210aa9412023326fb98a599ba502e6118308601dcbef77f", size = 3302756, upload-time = "2026-06-15T16:26:41.336Z" },
+ { url = "https://files.pythonhosted.org/packages/cf/d5/fde8f4dddcf518ee15ab35a7c6a28acc32c8ba548d1d2aa451f96e6dbb0b/sqlalchemy-2.0.51-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:72ca54c952107ba5cd58854b67a5a6268631289d21651a1235396f3b98b47400", size = 3232055, upload-time = "2026-06-15T16:19:49.286Z" },
+ { url = "https://files.pythonhosted.org/packages/67/d1/43d3a0ac955a58601c24fa23038b1c55ee3a1ec02c0f96ebb1eae2bcf614/sqlalchemy-2.0.51-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b3e693d15533a45cd5906f0589f9c35090bef6ef45bf1e8195c424aa0ae06a8d", size = 3269850, upload-time = "2026-06-15T16:26:43.017Z" },
+ { url = "https://files.pythonhosted.org/packages/94/df/de669c7054cd47c4439ac34b1b2ee8b804a794791fbb10720e997a2c87c7/sqlalchemy-2.0.51-cp313-cp313-win32.whl", hash = "sha256:b93ab07b5292dbe7e6b8da89475275e7042744283921344b56105f3eeb0f828b", size = 2117721, upload-time = "2026-06-15T16:23:12.36Z" },
+ { url = "https://files.pythonhosted.org/packages/d0/8a/403c51d064196bae20a0bc2476577f83a3f8dd299719a97417086b7f2ec5/sqlalchemy-2.0.51-cp313-cp313-win_amd64.whl", hash = "sha256:0f053118c30e53161857a953e4de667d90e274980dccbe5dd3829bbbeece72a5", size = 2143615, upload-time = "2026-06-15T16:23:13.906Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/49/a739be2e1d02a96a658eb71ab45d921c874249252358ad24a5bffdd02525/sqlalchemy-2.0.51-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:6ea306caaae6bd5afd0a46050003c88f6bf33227377a49298c498c3cb88ff491", size = 2158999, upload-time = "2026-06-15T16:08:51.759Z" },
+ { url = "https://files.pythonhosted.org/packages/23/6b/2e0e38cf75c8780eca78d9b2e78164f8bcfd70125e5caa588ff5cbb9c9f4/sqlalchemy-2.0.51-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c45a496d6bc05dec41dcd4c3a2b183723f47473255c159cd80b503c8f246424d", size = 3282539, upload-time = "2026-06-15T16:19:51.065Z" },
+ { url = "https://files.pythonhosted.org/packages/dd/a1/e77854cb5336fd37dc3c6ae3b71de242c98caac5725120be0b526b31cbd0/sqlalchemy-2.0.51-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4004ada0aafe8ae1991b2cd1d99c6d9146126e123bd6f883c260d974aa012e54", size = 3287545, upload-time = "2026-06-15T16:26:44.735Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/ab/9e17272fd4dac8df3b83c4fbe52b998a1c9d89a843c8c35ff29b74ff7364/sqlalchemy-2.0.51-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:0f6bcad487aee1c638d707235682fc96f741de00663619881ab235400d03289e", size = 3230929, upload-time = "2026-06-15T16:19:52.625Z" },
+ { url = "https://files.pythonhosted.org/packages/02/3c/52f408ea701781caee975606beccc48845f2aee8711ac29843d612c0306c/sqlalchemy-2.0.51-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:39a76529db6305693d8d4affa58ad5b5e2e18edd62daea628b29b97930b3513d", size = 3252888, upload-time = "2026-06-15T16:26:46.454Z" },
+ { url = "https://files.pythonhosted.org/packages/24/16/3efd2ee6bc4ca4693a30a1dd17a91b606cae15d517d2a4746611d9b73ce8/sqlalchemy-2.0.51-cp314-cp314-win32.whl", hash = "sha256:08a204d8b5638717c26a24df18fcf40af45a6b22e35b70b1d62f0113c2e278e8", size = 2120551, upload-time = "2026-06-15T16:23:15.629Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/78/55b12e70f45bccc40d9e483925c065027b3b98ea4cbbdf6f8c2546feaf6c/sqlalchemy-2.0.51-cp314-cp314-win_amd64.whl", hash = "sha256:96747bfbadb055466e5b46d572618170046b45ce5a4879167f50d70a5319a499", size = 2146318, upload-time = "2026-06-15T16:23:17.108Z" },
+ { url = "https://files.pythonhosted.org/packages/21/db/a9574ed40fed418924b1b1a3e54f47ee3963053b3d3d325a0d36b41f2c08/sqlalchemy-2.0.51-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:e5ea1a213be1fcd5e49d9904c3b9939211ded90bc2a64e93f4c01963474285de", size = 2178920, upload-time = "2026-06-15T15:59:56.285Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/90/a1bb5c7cbba76b7bc1fbd586d0a5479a7bc9c27b4a8298f22ec9423b2bb3/sqlalchemy-2.0.51-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7c6b36ed71f41942bdcd2ad2522be46bfce09d5705be5640ecf19bbc7660e4b7", size = 3566534, upload-time = "2026-06-15T15:58:35.024Z" },
+ { url = "https://files.pythonhosted.org/packages/15/4b/481f1fed30e0e9e8dd24aecbb49f29eb57fe7657ece5cf06ee9b84bb97d8/sqlalchemy-2.0.51-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0c2c62877097e1a0db401fba5cb4debee33265e5b2a55c4ccb489c02c53b4f72", size = 3535844, upload-time = "2026-06-15T16:02:43.973Z" },
+ { url = "https://files.pythonhosted.org/packages/02/71/0aa64aeda645510af0a43f7d9ee70932f0d1dc4263aed34c50ee891d9df3/sqlalchemy-2.0.51-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:0378d055e9e8cd6ce4d8dff683bdd3d7d413533c4ee51d67a2b1e0f9eacc0f23", size = 3475355, upload-time = "2026-06-15T15:58:36.592Z" },
+ { url = "https://files.pythonhosted.org/packages/05/db/6061db32316446135a3abae5f308d144ab988a34234726042da3e58b1c63/sqlalchemy-2.0.51-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:6e46fc36029eff666391e0531e5387b62ce6c4f1d8e50b3fb3099eaca1b42522", size = 3486591, upload-time = "2026-06-15T16:02:45.346Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/c9/f14fdf71bb8957e0c7e39db69bbdf12b5c80f4ef775fdfa127bf4e0d6760/sqlalchemy-2.0.51-cp314-cp314t-win32.whl", hash = "sha256:9161cfc9efce70d1715f47d6ff40f79c6778c00d53be4fbc09d70301e4b83ba7", size = 2151313, upload-time = "2026-06-15T16:03:39.127Z" },
+ { url = "https://files.pythonhosted.org/packages/6a/c6/673e618e6f4f297e126d9b56ea2f6478708f6c1af4e3223835c22e2c3697/sqlalchemy-2.0.51-cp314-cp314t-win_amd64.whl", hash = "sha256:159bb6ba32059f57ad7375a8f50d844dd2f19d14954ecf820cd33e20debd46b2", size = 2186280, upload-time = "2026-06-15T16:03:40.569Z" },
+ { url = "https://files.pythonhosted.org/packages/e2/22/dbf013a12ec759e54a34a119e9e217435b3f71b2dd5c61a7ade0a25dae87/sqlalchemy-2.0.51-py3-none-any.whl", hash = "sha256:bb024d8b621d0be75f4f44ecc7c950450026e76d66dc8f791bb5331d7fed59d5", size = 1944334, upload-time = "2026-06-15T16:09:22.418Z" },
+]
+
+[[package]]
+name = "stack-data"
+version = "0.6.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "asttokens" },
+ { name = "executing" },
+ { name = "pure-eval" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/28/e3/55dcc2cfbc3ca9c29519eb6884dd1415ecb53b0e934862d3559ddcb7e20b/stack_data-0.6.3.tar.gz", hash = "sha256:836a778de4fec4dcd1dcd89ed8abff8a221f58308462e1c4aa2a3cf30148f0b9", size = 44707, upload-time = "2023-09-30T13:58:05.479Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f1/7b/ce1eafaf1a76852e2ec9b22edecf1daa58175c090266e9f6c64afcd81d91/stack_data-0.6.3-py3-none-any.whl", hash = "sha256:d5558e0c25a4cb0853cddad3d77da9891a08cb85dd9f9f91b9f8cd66e511e695", size = 24521, upload-time = "2023-09-30T13:58:03.53Z" },
+]
+
+[[package]]
+name = "statsmodels"
+version = "0.14.6"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "pandas" },
+ { name = "patsy" },
+ { name = "scipy" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/0d/81/e8d74b34f85285f7335d30c5e3c2d7c0346997af9f3debf9a0a9a63de184/statsmodels-0.14.6.tar.gz", hash = "sha256:4d17873d3e607d398b85126cd4ed7aad89e4e9d89fc744cdab1af3189a996c2a", size = 20689085, upload-time = "2025-12-05T23:08:39.522Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/25/ce/308e5e5da57515dd7cab3ec37ea2d5b8ff50bef1fcc8e6d31456f9fae08e/statsmodels-0.14.6-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:fe76140ae7adc5ff0e60a3f0d56f4fffef484efa803c3efebf2fcd734d72ecb5", size = 10091932, upload-time = "2025-12-05T19:28:55.446Z" },
+ { url = "https://files.pythonhosted.org/packages/05/30/affbabf3c27fb501ec7b5808230c619d4d1a4525c07301074eb4bda92fa9/statsmodels-0.14.6-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:26d4f0ed3b31f3c86f83a92f5c1f5cbe63fc992cd8915daf28ca49be14463a1c", size = 9997345, upload-time = "2025-12-05T19:29:10.278Z" },
+ { url = "https://files.pythonhosted.org/packages/48/f5/3a73b51e6450c31652c53a8e12e24eac64e3824be816c0c2316e7dbdcb7d/statsmodels-0.14.6-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d8c00a42863e4f4733ac9d078bbfad816249c01451740e6f5053ecc7db6d6368", size = 10058649, upload-time = "2025-12-05T23:10:12.775Z" },
+ { url = "https://files.pythonhosted.org/packages/81/68/dddd76117df2ef14c943c6bbb6618be5c9401280046f4ddfc9fb4596a1b8/statsmodels-0.14.6-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:19b58cf7474aa9e7e3b0771a66537148b2df9b5884fbf156096c0e6c1ff0469d", size = 10339446, upload-time = "2025-12-05T23:10:28.503Z" },
+ { url = "https://files.pythonhosted.org/packages/56/4a/dce451c74c4050535fac1ec0c14b80706d8fc134c9da22db3c8a0ec62c33/statsmodels-0.14.6-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:81e7dcc5e9587f2567e52deaff5220b175bf2f648951549eae5fc9383b62bc37", size = 10368705, upload-time = "2025-12-05T23:10:44.339Z" },
+ { url = "https://files.pythonhosted.org/packages/60/15/3daba2df40be8b8a9a027d7f54c8dedf24f0d81b96e54b52293f5f7e3418/statsmodels-0.14.6-cp312-cp312-win_amd64.whl", hash = "sha256:b5eb07acd115aa6208b4058211138393a7e6c2cf12b6f213ede10f658f6a714f", size = 9543991, upload-time = "2025-12-05T23:10:58.536Z" },
+ { url = "https://files.pythonhosted.org/packages/81/59/a5aad5b0cc266f5be013db8cde563ac5d2a025e7efc0c328d83b50c72992/statsmodels-0.14.6-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:47ee7af083623d2091954fa71c7549b8443168f41b7c5dce66510274c50fd73e", size = 10072009, upload-time = "2025-12-05T23:11:14.021Z" },
+ { url = "https://files.pythonhosted.org/packages/53/dd/d8cfa7922fc6dc3c56fa6c59b348ea7de829a94cd73208c6f8202dd33f17/statsmodels-0.14.6-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:aa60d82e29fcd0a736e86feb63a11d2380322d77a9369a54be8b0965a3985f71", size = 9980018, upload-time = "2025-12-05T23:11:30.907Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/77/0ec96803eba444efd75dba32f2ef88765ae3e8f567d276805391ec2c98c6/statsmodels-0.14.6-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:89ee7d595f5939cc20bf946faedcb5137d975f03ae080f300ebb4398f16a5bd4", size = 10060269, upload-time = "2025-12-05T23:11:46.338Z" },
+ { url = "https://files.pythonhosted.org/packages/10/b9/fd41f1f6af13a1a1212a06bb377b17762feaa6d656947bf666f76300fc05/statsmodels-0.14.6-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:730f3297b26749b216a06e4327fe0be59b8d05f7d594fb6caff4287b69654589", size = 10324155, upload-time = "2025-12-05T23:12:01.805Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/0f/a6900e220abd2c69cd0a07e3ad26c71984be6061415a60e0f17b152ecf08/statsmodels-0.14.6-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:f1c08befa85e93acc992b72a390ddb7bd876190f1360e61d10cf43833463bc9c", size = 10349765, upload-time = "2025-12-05T23:12:18.018Z" },
+ { url = "https://files.pythonhosted.org/packages/98/08/b79f0c614f38e566eebbdcff90c0bcacf3c6ba7a5bbb12183c09c29ca400/statsmodels-0.14.6-cp313-cp313-win_amd64.whl", hash = "sha256:8021271a79f35b842c02a1794465a651a9d06ec2080f76ebc3b7adce77d08233", size = 9540043, upload-time = "2025-12-05T23:12:33.887Z" },
+ { url = "https://files.pythonhosted.org/packages/71/de/09540e870318e0c7b58316561d417be45eff731263b4234fdd2eee3511a8/statsmodels-0.14.6-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:00781869991f8f02ad3610da6627fd26ebe262210287beb59761982a8fa88cae", size = 10069403, upload-time = "2025-12-05T23:12:48.424Z" },
+ { url = "https://files.pythonhosted.org/packages/ab/f0/63c1bfda75dc53cee858006e1f46bd6d6f883853bea1b97949d0087766ca/statsmodels-0.14.6-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:73f305fbf31607b35ce919fae636ab8b80d175328ed38fdc6f354e813b86ee37", size = 9989253, upload-time = "2025-12-05T23:13:05.274Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/98/b0dfb4f542b2033a3341aa5f1bdd97024230a4ad3670c5b0839d54e3dcab/statsmodels-0.14.6-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e443e7077a6e2d3faeea72f5a92c9f12c63722686eb80bb40a0f04e4a7e267ad", size = 10090802, upload-time = "2025-12-05T23:13:20.653Z" },
+ { url = "https://files.pythonhosted.org/packages/34/0e/2408735aca9e764643196212f9069912100151414dd617d39ffc72d77eee/statsmodels-0.14.6-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3414e40c073d725007a6603a18247ab7af3467e1af4a5e5a24e4c27bc26673b4", size = 10337587, upload-time = "2025-12-05T23:13:37.597Z" },
+ { url = "https://files.pythonhosted.org/packages/0f/36/4d44f7035ab3c0b2b6a4c4ebb98dedf36246ccbc1b3e2f51ebcd7ac83abb/statsmodels-0.14.6-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:a518d3f9889ef920116f9fa56d0338069e110f823926356946dae83bc9e33e19", size = 10363350, upload-time = "2025-12-05T23:13:53.08Z" },
+ { url = "https://files.pythonhosted.org/packages/26/33/f1652d0c59fa51de18492ee2345b65372550501ad061daa38f950be390b6/statsmodels-0.14.6-cp314-cp314-win_amd64.whl", hash = "sha256:151b73e29f01fe619dbce7f66d61a356e9d1fe5e906529b78807df9189c37721", size = 9588010, upload-time = "2025-12-05T23:14:07.28Z" },
+]
+
+[[package]]
+name = "synchronicity"
+version = "0.12.5"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/5d/1c/f51dc54bbd302991026a53f9790735540e0e9e1184e9d5939f02446aa5bc/synchronicity-0.12.5.tar.gz", hash = "sha256:94d96b1d85698e3056b96a793b8c0949af6584e4a7d877fabdeb5385efe230aa", size = 60745, upload-time = "2026-06-18T21:06:23.545Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/06/74/ad9b99520f70c0bc3318e582e359d360cfc0f7afd7bf368a7f24013cece7/synchronicity-0.12.5-py3-none-any.whl", hash = "sha256:fdbbb10d437bc08a6b0f814fc66fddd1b58ffed314533d42f1ab555801e781af", size = 41107, upload-time = "2026-06-18T21:06:22.505Z" },
+]
+
+[[package]]
+name = "tabulate"
+version = "0.10.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/46/58/8c37dea7bbf769b20d58e7ace7e5edfe65b849442b00ffcdd56be88697c6/tabulate-0.10.0.tar.gz", hash = "sha256:e2cfde8f79420f6deeffdeda9aaec3b6bc5abce947655d17ac662b126e48a60d", size = 91754, upload-time = "2026-03-04T18:55:34.402Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl", hash = "sha256:f0b0622e567335c8fabaaa659f1b33bcb6ddfe2e496071b743aa113f8774f2d3", size = 39814, upload-time = "2026-03-04T18:55:31.284Z" },
+]
+
+[[package]]
+name = "texttable"
+version = "1.7.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/1c/dc/0aff23d6036a4d3bf4f1d8c8204c5c79c4437e25e0ae94ffe4bbb55ee3c2/texttable-1.7.0.tar.gz", hash = "sha256:2d2068fb55115807d3ac77a4ca68fa48803e84ebb0ee2340f858107a36522638", size = 12831, upload-time = "2023-10-03T09:48:12.272Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/24/99/4772b8e00a136f3e01236de33b0efda31ee7077203ba5967fcc76da94d65/texttable-1.7.0-py2.py3-none-any.whl", hash = "sha256:72227d592c82b3d7f672731ae73e4d1f88cd8e2ef5b075a7a7f01a23a3743917", size = 10768, upload-time = "2023-10-03T09:48:10.434Z" },
+]
+
+[[package]]
+name = "threadpoolctl"
+version = "3.6.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b7/4d/08c89e34946fce2aec4fbb45c9016efd5f4d7f24af8e5d93296e935631d8/threadpoolctl-3.6.0.tar.gz", hash = "sha256:8ab8b4aa3491d812b623328249fab5302a68d2d71745c8a4c719a2fcaba9f44e", size = 21274, upload-time = "2025-03-13T13:49:23.031Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/32/d5/f9a850d79b0851d1d4ef6456097579a9005b31fea68726a4ae5f2d82ddd9/threadpoolctl-3.6.0-py3-none-any.whl", hash = "sha256:43a0b8fd5a2928500110039e43a5eed8480b918967083ea48dc3ab9f13c4a7fb", size = 18638, upload-time = "2025-03-13T13:49:21.846Z" },
+]
+
+[[package]]
+name = "toml"
+version = "0.10.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/be/ba/1f744cdc819428fc6b5084ec34d9b30660f6f9daaf70eead706e3203ec3c/toml-0.10.2.tar.gz", hash = "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f", size = 22253, upload-time = "2020-11-01T01:40:22.204Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", hash = "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b", size = 16588, upload-time = "2020-11-01T01:40:20.672Z" },
+]
+
+[[package]]
+name = "topacedo"
+version = "0.2.10"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "networkx" },
+ { name = "numba" },
+ { name = "numpy" },
+ { name = "pandas" },
+ { name = "pcst-fast" },
+ { name = "scipy" },
+ { name = "tqdm" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/38/4b/3e5d36b7b96cc4e8dd61e9fdbe0ff748cdae4c61a9e5733a9b878de0044a/topacedo-0.2.10.tar.gz", hash = "sha256:13b3bd7753b2fdd6b7bdca13807f99f81c1decc50ebc9757d0b772bc4f6aab7f", size = 10789, upload-time = "2022-09-13T20:53:40.313Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/57/e0/cbe7b82090282888bdfa3c22c6b141e926c86a2c3310779754c59ed0f0af/topacedo-0.2.10-py3-none-any.whl", hash = "sha256:6a1131b13a105717715ec88791e8688a0a10f660ada622bcf6ced101b5701009", size = 11461, upload-time = "2022-09-13T20:53:38.652Z" },
+]
+
+[[package]]
+name = "tornado"
+version = "6.5.7"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/64/24/95ec527ad67b76d59299e5465b3935d05e4294b7e0290a3924b7487df30b/tornado-6.5.7.tar.gz", hash = "sha256:66c513a76cda70d53907bc27cf1447557699c2e95aa48ba27a442ff61c3ddfc2", size = 519252, upload-time = "2026-06-08T17:34:51.232Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/02/dc/c7043cab6fed8ae159fc1923ce829ada35c4dbd797d408a43858ffaf9639/tornado-6.5.7-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:148b2eb15c2c765a50796172c1e499649b35f30d2e3c3d3e15913cfa56bfb163", size = 448543, upload-time = "2026-06-08T17:34:38.052Z" },
+ { url = "https://files.pythonhosted.org/packages/92/4f/090b1431e5a43df696feceffc268c5383cc079ecb5f08ce58f917109aafe/tornado-6.5.7-cp39-abi3-macosx_10_9_x86_64.whl", hash = "sha256:9da38de27f1da3b78a966f0dae12b5a1ea9afe72ca805d84ff06508272ddf100", size = 446707, upload-time = "2026-06-08T17:34:39.594Z" },
+ { url = "https://files.pythonhosted.org/packages/37/d8/ef374952fd5da67d4463122c2b8e5a96536ec10b4b339254c6dcde81d01c/tornado-6.5.7-cp39-abi3-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:8d759e71906ee783f8867b93bf26a265743da4c1e2f4a018464c1ba019862972", size = 449774, upload-time = "2026-06-08T17:34:41.204Z" },
+ { url = "https://files.pythonhosted.org/packages/35/37/d434c73f4c6e014b745b9b37085f34f40c022f007efff3d7fe65991899f3/tornado-6.5.7-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8a46347a18f23fb92b396beebe0fb78f61dda0cc302445202c16203d8a18848b", size = 450745, upload-time = "2026-06-08T17:34:42.531Z" },
+ { url = "https://files.pythonhosted.org/packages/b6/2b/56b9aff361d7f1ab728a805ec7d7ea835f8807afa9f5cc690ea0e630efb9/tornado-6.5.7-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:7778b30bef919231265e91c69963ce0f49a1e9c07ac900bbe75b19ce2575ba92", size = 450578, upload-time = "2026-06-08T17:34:43.787Z" },
+ { url = "https://files.pythonhosted.org/packages/02/30/a7444fb23aa76860a14198fab96ac79f1866b0a6e19e26c4381b0938e50f/tornado-6.5.7-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e726f0c75da7726eec023aa62751ff8878bd2737e34fbdd33b1ae5897d2200f5", size = 449985, upload-time = "2026-06-08T17:34:45.326Z" },
+ { url = "https://files.pythonhosted.org/packages/5c/42/5f0e56c01e8d9d36f4e23f367b85ae6cae0c1ecddd5e6977d8388ad27488/tornado-6.5.7-cp39-abi3-win32.whl", hash = "sha256:f8de3bf12d3efdd0cbe7c8887868198f8a91415e3f29fcf258d9b8eb7b1d9ae4", size = 451047, upload-time = "2026-06-08T17:34:46.784Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/a4/b393076ffb21b469eec5b328a0534cf03a3b90bfc6b1f09507cdd075d938/tornado-6.5.7-cp39-abi3-win_amd64.whl", hash = "sha256:de942f843533a039ef9fa3d9c88c7cd8a7c94553fb5ad0154270989b3d99a2c4", size = 451485, upload-time = "2026-06-08T17:34:48.248Z" },
+ { url = "https://files.pythonhosted.org/packages/71/2e/7b1c769803121b809112cf9a00681c472eae1d80e32d7ec0e0bd61d0d0e1/tornado-6.5.7-cp39-abi3-win_arm64.whl", hash = "sha256:ff934fce95643af5f11efdae618eaa73d469dc588641e5c8d19295a0c65c4796", size = 450506, upload-time = "2026-06-08T17:34:49.702Z" },
+]
+
+[[package]]
+name = "tqdm"
+version = "4.68.3"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "colorama", marker = "sys_platform == 'win32'" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/87/d7/0535a28b1f5f24f6612fb3ff1e89fb1a8d160fee0f976e0aa6803862134b/tqdm-4.68.3.tar.gz", hash = "sha256:00dfa48452b6b6cfae3dd9885636c23d3422d1ec97c66d96818cbd5e0821d482", size = 170596, upload-time = "2026-06-17T07:36:52.105Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/d8/8e/bb97bb0c71802080bfc8952937d174e49cfc50de5c951dd47b2496f0dcdb/tqdm-4.68.3-py3-none-any.whl", hash = "sha256:39832cc2def2789a6f29df83f172db7416cea70052c0907a57801c5f2fdccb03", size = 78337, upload-time = "2026-06-17T07:36:50.132Z" },
+]
+
+[[package]]
+name = "traitlets"
+version = "5.15.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/57/a9/a2584b8313b89f94869ddb3c4074617a691de1812a614d2d50e32ca5a7a6/traitlets-5.15.1.tar.gz", hash = "sha256:7b1c07854fe25acb39e009bae49f11b79ff6cbb2f27999104e9110e7a6b53722", size = 163344, upload-time = "2026-06-03T12:26:06.181Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/96/8d/1080ee4c231f361b6ce4470d556c8c435b67c7e0753aaa641497ee92f88b/traitlets-5.15.1-py3-none-any.whl", hash = "sha256:770a53705f84b81ac107e83a1b3328ff2dae16094d8fc3cfc004e4b22dfd8e92", size = 85858, upload-time = "2026-06-03T12:26:04.395Z" },
+]
+
+[[package]]
+name = "types-certifi"
+version = "2021.10.8.3"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/52/68/943c3aeaf14624712a0357c4a67814dba5cea36d194f5c764dad7959a00c/types-certifi-2021.10.8.3.tar.gz", hash = "sha256:72cf7798d165bc0b76e1c10dd1ea3097c7063c42c21d664523b928e88b554a4f", size = 2095, upload-time = "2022-06-09T15:19:05.244Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b5/63/2463d89481e811f007b0e1cd0a91e52e141b47f9de724d20db7b861dcfec/types_certifi-2021.10.8.3-py3-none-any.whl", hash = "sha256:b2d1e325e69f71f7c78e5943d410e650b4707bb0ef32e4ddf3da37f54176e88a", size = 2136, upload-time = "2022-06-09T15:19:03.127Z" },
+]
+
+[[package]]
+name = "types-toml"
+version = "0.10.8.20260518"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/4b/11/6ece999e91f2ccb848ab4420f3f4816e78ac0541f739e6864affdaaa5737/types_toml-0.10.8.20260518.tar.gz", hash = "sha256:80e10facd24fdeda9d5c672187d72be3ac284843788d67f5aae59e3e016db6fe", size = 9419, upload-time = "2026-05-18T06:02:16.719Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/91/25/489751806bf5c95e4007f8e17409199c54d31e49ffbea07c5729b1286c8e/types_toml-0.10.8.20260518-py3-none-any.whl", hash = "sha256:0e564ab05f6fde62a315b3b5a9b6624fda569399795d30a37e64705a70459303", size = 9669, upload-time = "2026-05-18T06:02:15.86Z" },
+]
+
+[[package]]
+name = "typing-extensions"
+version = "4.15.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" },
+]
+
+[[package]]
+name = "typing-inspection"
+version = "0.4.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/55/e3/70399cb7dd41c10ac53367ae42139cf4b1ca5f36bb3dc6c9d33acdb43655/typing_inspection-0.4.2.tar.gz", hash = "sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464", size = 75949, upload-time = "2025-10-01T02:14:41.687Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611, upload-time = "2025-10-01T02:14:40.154Z" },
+]
+
+[[package]]
+name = "tzdata"
+version = "2026.2"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ba/19/1b9b0e29f30c6d35cb345486df41110984ea67ae69dddbc0e8a100999493/tzdata-2026.2.tar.gz", hash = "sha256:9173fde7d80d9018e02a662e168e5a2d04f87c41ea174b139fbef642eda62d10", size = 198254, upload-time = "2026-04-24T15:22:08.651Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/ce/e4/dccd7f47c4b64213ac01ef921a1337ee6e30e8c6466046018326977efd95/tzdata-2026.2-py2.py3-none-any.whl", hash = "sha256:bbe9af844f658da81a5f95019480da3a89415801f6cc966806612cc7169bffe7", size = 349321, upload-time = "2026-04-24T15:22:05.876Z" },
+]
+
+[[package]]
+name = "umap-learn"
+version = "0.5.12"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numba" },
+ { name = "numpy" },
+ { name = "pynndescent" },
+ { name = "scikit-learn" },
+ { name = "scipy" },
+ { name = "tqdm" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/02/ee/af4171241117f85c74b5ca6448ea1033cc28d599c13651d67289bacd4083/umap_learn-0.5.12.tar.gz", hash = "sha256:6aff02ecac5f2aad9f3c65ee518d7ae93e1a985ae38721fdcffceee4232c33c7", size = 96672, upload-time = "2026-04-08T20:03:54.012Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/1b/98/f63318ccbe75c810011fe9233884c5d348d94d90005de1b79e5f93bef9c0/umap_learn-0.5.12-py3-none-any.whl", hash = "sha256:f2a85d2a2adcb52b541bed9b27a23ca169b56bb1b23283abeebfb8dfb8a42fe5", size = 91849, upload-time = "2026-04-08T20:03:52.561Z" },
+]
+
+[[package]]
+name = "urllib3"
+version = "2.7.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" },
+]
+
+[[package]]
+name = "watchfiles"
+version = "1.2.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "anyio" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/cd/41/5e1a4bb12aac5f1493fa1bdc11154eca3b258ca4eba65d39c473fe19d8e9/watchfiles-1.2.0.tar.gz", hash = "sha256:c995fba777f1ea992f090f9236e9284cf7a5d1a0130dd5a3d82c598cacd76838", size = 108252, upload-time = "2026-05-18T04:32:04.251Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/b8/2f/e42c992d2afda3108ea1c02acecc991b9f31d05c14adc2a7cee9ee211fc4/watchfiles-1.2.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:bc13eb17538be00c874699dc0abe4ee2bc8d50bb1166a6b9e175ef3fd7eb8f26", size = 400115, upload-time = "2026-05-18T04:32:02.06Z" },
+ { url = "https://files.pythonhosted.org/packages/5f/8f/6af2ea19065c91d8b0ea3516fdfc8c0d349f407e8e9fbf4e5a17360de8ad/watchfiles-1.2.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:2d95ddc1eb6914154253d239089900813f6a767e174b8e6a50e7fdacb7e4236c", size = 393659, upload-time = "2026-05-18T04:30:50.951Z" },
+ { url = "https://files.pythonhosted.org/packages/13/01/b32a967c56fb3e3e5be3db52c3d3b87fa4513aa367d8ed1ad96d42952e5f/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8f70d8b291ef6e88d19b1f297a6905ddb978888d9272b0d05e6f53309856bcfc", size = 453207, upload-time = "2026-05-18T04:31:04.231Z" },
+ { url = "https://files.pythonhosted.org/packages/04/98/97557a812180338cb1abd32e1cffcc4588f59b5f23e0cb006b2ba95ba64a/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:56d8641cf834c2836922899105bd3ce3d0dfc69291d52edf0b4d0436829b34c0", size = 459273, upload-time = "2026-05-18T04:31:50.377Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/a8/b4b08dcb7653b8087c6586f7ce649505900e866bbcfe40dc9587af02e686/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:2581a94056e55d7d0a31a823ea92bf73749c489ca2285bfdc0fbe6b2bb49d50c", size = 489927, upload-time = "2026-05-18T04:31:42.485Z" },
+ { url = "https://files.pythonhosted.org/packages/50/94/3dceea03545d2e5ddfd839f0ddd5e1cecbf1697b5a428d5ba11cef6af95d/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:41bc1199f7523b3f82843c88cbb979180c949caef0342cf90968f178e5d49b01", size = 570476, upload-time = "2026-05-18T04:31:03.071Z" },
+ { url = "https://files.pythonhosted.org/packages/cc/f2/d39a5450c3532092b91f81d274360e613c2371bc874a89c7a1a3c5e8d138/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:7571e4464cb6e434958f867f7f730b8ab0b75e3f8e5eac0499168486ab3c33a8", size = 465650, upload-time = "2026-05-18T04:30:12.701Z" },
+ { url = "https://files.pythonhosted.org/packages/22/24/ed72f68cbc1333ca9b9f2200aa048bb6658ae41709bc1caad4310f4bdffd/watchfiles-1.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e53a384f76b631c3ae5334ce6a52f0baa3a911eb94a4eac7f160079868b716d5", size = 456398, upload-time = "2026-05-18T04:30:13.784Z" },
+ { url = "https://files.pythonhosted.org/packages/0d/64/982ef4a4e5bab5b6e5b6becc8cd5e732f6130a78b855f0abec6439a9a135/watchfiles-1.2.0-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:d20029a60a71a052a24c4db7673bc4de39ab89adbaccbfb5d67987c5d73f424d", size = 465140, upload-time = "2026-05-18T04:31:52.111Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/0c/95282abf4ed680b6096010bcfc30c5fa7a041fc5aa5a2ad17a2cc6c75bba/watchfiles-1.2.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:2cb93af48550faf1cea04c303107c8b75833de7013e57ce27d3b8d21d8d0f58c", size = 630259, upload-time = "2026-05-18T04:31:25.676Z" },
+ { url = "https://files.pythonhosted.org/packages/30/45/607c1de1530c4bdcf2cf1d1ecc2505ddba5d96bd43ba9f2b0e79876f850f/watchfiles-1.2.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:2995c176de7692b86a2e4c58d9ec718f753150a979cb4a754e2b4ffa38e70906", size = 659859, upload-time = "2026-05-18T04:30:24.333Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/08/d9e2e0f9e8e6791d33aefc694ad7eefa7f901f63caff84a81ded38692f9c/watchfiles-1.2.0-cp312-cp312-win32.whl", hash = "sha256:7a2cffd17d27d2ecbb310c2b1d8174f222a5495b1a721894afa88ec11e25b898", size = 275480, upload-time = "2026-05-18T04:30:31.307Z" },
+ { url = "https://files.pythonhosted.org/packages/1c/e6/9d42569c0102645cc8cea5d8c7d8a1e9d4ada2cb7f05f75e554b8aa2202a/watchfiles-1.2.0-cp312-cp312-win_amd64.whl", hash = "sha256:f155b3a1b2a5fc89cdc70d47ee5d54e3b75e88efa34982028a35daef9ba00379", size = 288718, upload-time = "2026-05-18T04:32:10.745Z" },
+ { url = "https://files.pythonhosted.org/packages/0a/26/88e0dc6ee3898169d7fa22bb6a69cabf2502d2ee25cb8c876d1262d204f8/watchfiles-1.2.0-cp312-cp312-win_arm64.whl", hash = "sha256:8fa585ede612ee9f9e91b18bebf9ba11b9ae29a4e3a0d0cf6fca3e382133f0d5", size = 281026, upload-time = "2026-05-18T04:30:22.23Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/4d/70a7feced9f87e2ff26dba42667290f41694fc64646c67261fbb8cab5d5c/watchfiles-1.2.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:01ea8d66f0693b9b60a6541c8d10263091ca9a9060d242f3c1f3143f9aad2c98", size = 399730, upload-time = "2026-05-18T04:31:38.162Z" },
+ { url = "https://files.pythonhosted.org/packages/31/3a/0da302f2307aee316922806ebd5726c542cbd787c938271cf14a074c7daf/watchfiles-1.2.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7ba0480b9a74af058f43b337e937a451e109295c420916d68ad24e3dc02f5e44", size = 392842, upload-time = "2026-05-18T04:30:27.051Z" },
+ { url = "https://files.pythonhosted.org/packages/db/ef/d5bdb705c224dbc256aa0c1ec47bf4e61ec52558f2afb44a71a1fe4d7015/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4f34e26a19f91f710c08e0183429f0d1d15df734e6bc78c31e77b9ea9c433658", size = 452989, upload-time = "2026-05-18T04:31:11.945Z" },
+ { url = "https://files.pythonhosted.org/packages/71/29/5495f2c1661949ef7a35e4d71111d129cfe7606414a26887a919d0a55406/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b4e77f6a55f858504069abd35d336a637555c09bca453dde1ee1e5ada8a6a1fb", size = 458978, upload-time = "2026-05-18T04:30:52.606Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/8c/7f9c07c433811c2fffd93e13fdfb7135de9aab5f2ae41be08960fa0047dc/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0cb4d80e212f116474a545c21c912b445f16bb0cef9e6a73a498164223e14e2f", size = 490248, upload-time = "2026-05-18T04:31:36.003Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/11/d93632febc52fbc21be90231bb7c17fd5387f46c9076fd40a5f9c2ae6910/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:b974946a10af379d425e2eef5b62f5c6ebeaccf91d45eaad6f5b27ecd4f91aa0", size = 571847, upload-time = "2026-05-18T04:31:10.862Z" },
+ { url = "https://files.pythonhosted.org/packages/55/b4/383173e73aabb07ad1d9c7aa859d95437ac46a6d6a1e11005facda0c9d19/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:86bc13c25a8d1fcd70b51d0ce7c9b65e90de5666fcbfd3e34957cc73ee19aeb5", size = 465974, upload-time = "2026-05-18T04:30:17.006Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/6c/89b1a230a78f57c52dd8893adb1f92f94411721b6ec12596c56d98c74356/watchfiles-1.2.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ca148d73dea36c9763aaa351e4d7a51780ec1584217c45276f4fe8239c768b71", size = 454782, upload-time = "2026-05-18T04:30:35.656Z" },
+ { url = "https://files.pythonhosted.org/packages/24/62/1732118367cfff0a9fce3bf62ff4bfded09ef5df21d9d446b858b3f70a96/watchfiles-1.2.0-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:c525543d91961c6955b2636b308569e84a1d1c5f5f2932041ab9ef46422f43e3", size = 465182, upload-time = "2026-05-18T04:30:20.846Z" },
+ { url = "https://files.pythonhosted.org/packages/28/96/716f7e5f51339bf22963f3345f9f27d7f3b30e2eadc597e257c881dd3c53/watchfiles-1.2.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:a204794696ffb8f9b10fba6f7cb5216d42f3b2b71860ccac6b6e42f5f10973b0", size = 629841, upload-time = "2026-05-18T04:31:05.397Z" },
+ { url = "https://files.pythonhosted.org/packages/4c/fe/c40783950fd771ccf66ab3ec2722d188a9af1c7f96c6e811f36e40c6e03f/watchfiles-1.2.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:10d86db20695afe7997ac9e1717637d6714a8d0220458c33f3d2061f54cec427", size = 658028, upload-time = "2026-05-18T04:31:48.22Z" },
+ { url = "https://files.pythonhosted.org/packages/71/72/4508db1856d1d87fcbb3b63f4839bab1b5682cb0e8d224d122263c09654a/watchfiles-1.2.0-cp313-cp313-win32.whl", hash = "sha256:eb283ee99e21ad6443c8cdb06ac5b34b1308c329cbdf03fa02b445363714c799", size = 275183, upload-time = "2026-05-18T04:30:59.57Z" },
+ { url = "https://files.pythonhosted.org/packages/f9/36/14b76ca57652e5cc5fd1c11f32a261292c08a0d19a00351013c2549cbfb2/watchfiles-1.2.0-cp313-cp313-win_amd64.whl", hash = "sha256:a0f27f01bee51861392bb6b7c4fdb290b27d1eb194e9e28788d68102a0e898d9", size = 288059, upload-time = "2026-05-18T04:32:07.937Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/8d/0a85e395398d8d20fadfe5c5d32c726eee17a519e78fb356f2cf7531bffe/watchfiles-1.2.0-cp313-cp313-win_arm64.whl", hash = "sha256:3651aa7058595e9cfb75d35dd5ada2bf9f48a5b8a0f3562821d3e210c507e077", size = 280186, upload-time = "2026-05-18T04:31:54.484Z" },
+ { url = "https://files.pythonhosted.org/packages/37/68/36db056f1fdcc5f07302f56e631774d6835bcd6fa3ace402304621d5f9e5/watchfiles-1.2.0-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:faea288b6f0ab1902ef08f4ca6de005dccf856c4e0c4f21b8c5fce02d90a1b08", size = 399031, upload-time = "2026-05-18T04:30:44.576Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/64/01a9d6f66a82a5c101ce939274106cc72759d62427e153f01edd2b9f87c2/watchfiles-1.2.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:01859b11fd9fbca670f4d5da00fbac282cfea9bd67a2125d8b2833a3b5617ea9", size = 391205, upload-time = "2026-05-18T04:30:25.413Z" },
+ { url = "https://files.pythonhosted.org/packages/84/2c/0a44fe058cb4bb7b8ede6b6670698bbb7c0400740e378d00022189b7b31d/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fff610d7bb2256a317bb1e96f0d7862c7aa8076733ee5df0fd41bbe76a24a4f4", size = 451892, upload-time = "2026-05-18T04:32:14.005Z" },
+ { url = "https://files.pythonhosted.org/packages/67/a1/351e0d56cd35e6488b5c8b4fb11a809a5bc923e8fe8fed9faf8920be0c89/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b141a4891c995a039cd89e9a49e62df1dc8a559a5d1a6e4c7106d16c12777a55", size = 458867, upload-time = "2026-05-18T04:31:22.279Z" },
+ { url = "https://files.pythonhosted.org/packages/d5/7d/9d09605187f1b838998624049fcf8bf47b73c1a3b76901fcac1782f62277/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f22943b7770483f6ea0721c6b11d022947a98eb0acae14694de034f4d0d38925", size = 490217, upload-time = "2026-05-18T04:31:43.657Z" },
+ { url = "https://files.pythonhosted.org/packages/60/5d/a17a16eccb182f04188cd308ec24b1a71a9b5c4e7098269cf35d9fa56d02/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1bc6195825b7dcd217968bb1f801a60fd4c16e8eeab5bedc7fe917d7d5995ab4", size = 571458, upload-time = "2026-05-18T04:32:11.875Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/3d/4dd457062083ab1938e5dfd45032eb425cee2ac817287ca8ff4356183e5d/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d4a4b147f5dca2a5d325a06a832fb43f345751adfbc63204aec30e0d9ca965a2", size = 464707, upload-time = "2026-05-18T04:30:43.492Z" },
+ { url = "https://files.pythonhosted.org/packages/c6/71/ea8c57b128f5383de74d0c7d2d9c57ad7c9a65a930c451bd25d524b295b7/watchfiles-1.2.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4543579a9bdb0c9560039b4ffddbdb39545707659fbc430ce4c10f3f68d557f9", size = 454663, upload-time = "2026-05-18T04:30:16.061Z" },
+ { url = "https://files.pythonhosted.org/packages/53/fd/2e812bf938406d7db351f0703ddd3fc6c061cf30d96153a77bc79a943a44/watchfiles-1.2.0-cp313-cp313t-manylinux_2_31_riscv64.whl", hash = "sha256:20aa0e708b920bde876a4aa82dc7dd6ebea228a63a67cda6632c2fc87b787efa", size = 463537, upload-time = "2026-05-18T04:31:44.9Z" },
+ { url = "https://files.pythonhosted.org/packages/86/56/d17a7f1dd1bc3035f1072694a551301272f1739c2d8e319c927cb9e29b38/watchfiles-1.2.0-cp313-cp313t-musllinux_1_1_aarch64.whl", hash = "sha256:d413349d565dab74297f2a63e84a097936be69bf8f3b3801f27f380e32040f44", size = 629194, upload-time = "2026-05-18T04:31:14.141Z" },
+ { url = "https://files.pythonhosted.org/packages/be/06/f1ff66bf5cae50aa4062779a0ecd0bbaf15e466195719074078947d9a17d/watchfiles-1.2.0-cp313-cp313t-musllinux_1_1_x86_64.whl", hash = "sha256:f28b2725eb8cce327b9b3ab02415c853011dc55c95832fe90de6bc56f5315f72", size = 656194, upload-time = "2026-05-18T04:31:47.14Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/54/a9c7ea9a82a4ac65e7004c0a03920b5cdd2f9c3b678757d9cd425aa51d53/watchfiles-1.2.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:b8c8358484d5fa12ef34f05b7f4168eaf1932f408725ff6d023c33ec17bd79d4", size = 400205, upload-time = "2026-05-18T04:32:05.153Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/5d/c9ab3534374a4a67450696905d6ef16a04405448b8dc52bd752ae50423d4/watchfiles-1.2.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:9f04b092229ad2c50126dd3c922c8822e51e605993764a33058d4a791ab42281", size = 392508, upload-time = "2026-05-18T04:30:54.849Z" },
+ { url = "https://files.pythonhosted.org/packages/26/ca/1ad30103535cf0cecd7b993e8d50edc5351b1820e38f2d22e3df58962feb/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7a7ce236284f002a156f70add88efe5c70879cccbb658be0822c54b1306fc09d", size = 452448, upload-time = "2026-05-18T04:30:53.727Z" },
+ { url = "https://files.pythonhosted.org/packages/37/a1/ceee2cdf2afbd715fa07758d39c9859513eae411b23196f7fd039e5feedd/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b9909cc2b48468b575eefa944919e1fe8a36c5849d5c7c168f80a8c1db69398e", size = 459605, upload-time = "2026-05-18T04:30:23.312Z" },
+ { url = "https://files.pythonhosted.org/packages/e8/f6/421e30fd1cb3907a84ed92ab3f1983e37ba2dca015e9a894a048418417a2/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0a37faaed405c67e28e6be45a1fa4f206ef5a2860f27c237db9fa30704c38242", size = 490757, upload-time = "2026-05-18T04:30:47.358Z" },
+ { url = "https://files.pythonhosted.org/packages/41/b0/55ed1b97ed08be7bba6f9a541cac15f2a858e1d74d2b07b6da70a82aab00/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9649193aa27bd9ff2e80ff29bfaa93085496c7a3a377592823cc58b77ee88add", size = 568672, upload-time = "2026-05-18T04:30:38.915Z" },
+ { url = "https://files.pythonhosted.org/packages/d1/cf/d8ae8a80dd7bafab395ea7681c10237311bbf34d37704a8c744e7cf31fc7/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4e4ff8e37f99cf1da89e255e07c9c4b37c214038c4283707bdec308cb1b0ea1f", size = 464197, upload-time = "2026-05-18T04:30:09.914Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/8a/3076c496ca8dafe0e8cd03fcebdfc47be4b1174b4e5b24ff6e396e6b3af2/watchfiles-1.2.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:054dc20fd2e3132b4c3883b4a00d72fd6e1f56fdaf89fccd12e8057d74cd74d7", size = 453181, upload-time = "2026-05-18T04:30:14.829Z" },
+ { url = "https://files.pythonhosted.org/packages/e5/10/9745e17c98e7b8a86454df0a3c7b5686bd650383f1e9f26e4ebcbd6cc0c0/watchfiles-1.2.0-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:e140ed30ebde76796b686e67c182cff10ea2fbab186fafd1560f74bb5a473a6e", size = 465109, upload-time = "2026-05-18T04:30:28.123Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/95/8ef4a95481d3e0cb52d62a06fa6e972e81424be2d9698b91a2fecca9904c/watchfiles-1.2.0-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:bb7e52ecf68ba46d22df23467b87cffeb2146908aa523ebfe803019618cfda06", size = 630653, upload-time = "2026-05-18T04:31:49.304Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/e4/3b3bf36b0f829b50c6ebcb8d031583863c59f923d6a6af3d485e470d0fac/watchfiles-1.2.0-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:23282a321c8baf9b3a3c4afff673f9fe65eb7fdc2338d765ccad9d3d1916a5ba", size = 657838, upload-time = "2026-05-18T04:31:06.497Z" },
+ { url = "https://files.pythonhosted.org/packages/21/b1/6cbbb50c1f3002ab568777d44aa21206dfb8807a840990c4037523b51812/watchfiles-1.2.0-cp314-cp314-win32.whl", hash = "sha256:c0db965c5f79aa49fe672d297cf1febc5ad149b658594944f49a54a2b96270a7", size = 275108, upload-time = "2026-05-18T04:30:06.891Z" },
+ { url = "https://files.pythonhosted.org/packages/92/45/190ce6db8dcb4536682cf75d3889ff1a27182a58cb519d343cb6d9ea63d8/watchfiles-1.2.0-cp314-cp314-win_amd64.whl", hash = "sha256:71283b39fd17e5408eb123bd37aeecfd9d54c81fc184421943208aadb879d103", size = 288441, upload-time = "2026-05-18T04:32:12.901Z" },
+ { url = "https://files.pythonhosted.org/packages/74/0d/3eae1c2313ab08378431d907c3f8095ecca00f3eda33111cf4f0f2591799/watchfiles-1.2.0-cp314-cp314-win_arm64.whl", hash = "sha256:c5c19526f4e54a00f2666a6c0e9e40d582c09e865055ea7378bf0009aab857b3", size = 280684, upload-time = "2026-05-18T04:31:26.902Z" },
+ { url = "https://files.pythonhosted.org/packages/b1/75/fb64e6c25d6b5ca636d03df34ffb1c6e9873303e76d27967e045f8df088f/watchfiles-1.2.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:d73a585accffa5ae39c17264c36ec3166d2fad7000c780f5ef83b2722afb9dd2", size = 398857, upload-time = "2026-05-18T04:32:17.108Z" },
+ { url = "https://files.pythonhosted.org/packages/73/4e/9f7adf01754cbf81843722ccfec169d8f26c69778281a302855cecd2ee08/watchfiles-1.2.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ae99b14c5f21e026e0e9d96f40e07d8570ebee6cafd9d8fc318354606daa7a28", size = 392413, upload-time = "2026-05-18T04:31:07.911Z" },
+ { url = "https://files.pythonhosted.org/packages/47/c8/bec626bcc2d69f44b9acb24ce7d60ed7b16b73628eea747fcbd169d8edda/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4429f3b105524a10b72c3a819b091c495d2811d419c1e1e8df773a5a5974f831", size = 452409, upload-time = "2026-05-18T04:31:20.142Z" },
+ { url = "https://files.pythonhosted.org/packages/00/b7/b6362068e81e7c556d155a34c35d40ac3ef42d747b06d7f6e5bf58e359c2/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:43d818978d06062d9b22c4fab2ebe44cf5213d42dc8e62bda8c2760cfa2eeb33", size = 458827, upload-time = "2026-05-18T04:32:06.219Z" },
+ { url = "https://files.pythonhosted.org/packages/67/f8/9a813fa42afb1e0b4625e75f0479826644d3ee8dc287e093799bc01f390c/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b9f732dc58b2dbe69e464ccf8fff7a03b0dd0be439da4c0720d3558527d3d6b4", size = 490104, upload-time = "2026-05-18T04:31:56.034Z" },
+ { url = "https://files.pythonhosted.org/packages/2f/bf/27dfb6094ca4c9aad21298b5525b6c53cb36121ee454331d05161e58d130/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8f200104103feb097de4cab8fe4f5dd18a2026934c7dea98c55a2f5fd6d5a33b", size = 571360, upload-time = "2026-05-18T04:31:57.133Z" },
+ { url = "https://files.pythonhosted.org/packages/fb/39/44a096d67270ea93df91d33877dbe91fbda3aa4f8ec2edf799d93eda8736/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:63ac26eefbf4af1741247d6fb68b11c49a25b2f7413fbd318a83a12aaa9cf666", size = 464644, upload-time = "2026-05-18T04:30:57.33Z" },
+ { url = "https://files.pythonhosted.org/packages/0e/80/c7472203bad6268e3ef1ad260739704847898938ad7ea8b63a5131f46b50/watchfiles-1.2.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0c4997d4e4a55f0d02b6cde327322daf3a0400e5df6c6b15948994bf72497925", size = 454771, upload-time = "2026-05-18T04:30:48.736Z" },
+ { url = "https://files.pythonhosted.org/packages/51/cf/3b10b268b4b7f0fc26e9debb5eef1998b515887840f444cd3ec80c688755/watchfiles-1.2.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:4c887eba18b7945ac73067a8b4a66f21cd46c2539b2bc68588f7be6c7eb6d26b", size = 463494, upload-time = "2026-05-18T04:31:33.826Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/3e/a4302545cd589262a0dc7d140e86f7688eba3f9c72776c27f7e23b8864c4/watchfiles-1.2.0-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:3416ff151bb6b5a8d8d11664974fbef4d9305b9b2957839ab5a270468fd8df30", size = 629383, upload-time = "2026-05-18T04:31:15.596Z" },
+ { url = "https://files.pythonhosted.org/packages/db/99/d5649df0a9a410d45b7c882304d0b790903ac9b6e8f2cfd12114e0c6b9f2/watchfiles-1.2.0-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:0e831a271c035d89789cffc386b6aa1375f39f1cd25eb7ca0997e4970d152fc5", size = 656093, upload-time = "2026-05-18T04:31:58.707Z" },
+ { url = "https://files.pythonhosted.org/packages/92/b9/362702539275019a54dd2e94511b31a9b89c5f9e6a21966de7eb692549fc/watchfiles-1.2.0-cp315-cp315-macosx_10_12_x86_64.whl", hash = "sha256:37a6721cdf3f65dbb13aa9503510ccb4451603ac837e44d265d7992a597e1374", size = 400109, upload-time = "2026-05-18T04:31:16.879Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/75/71d5ba62db781e5587bded1d944c675374bc4aa37ff33d5018d98e8b6538/watchfiles-1.2.0-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:2b37d10b5a63bd4d87e18472d80fa525bd670586fae62e5dd580452764879b65", size = 392167, upload-time = "2026-05-18T04:31:28.058Z" },
+ { url = "https://files.pythonhosted.org/packages/3c/01/c66dd95d0423fe30d31820e2d1d5bda773764131bbb6ac0cb1cf303ac328/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0a105bc2283f67e8fbec74253ec2d94925de92ed72c0393f1206bf326b7b7b69", size = 452372, upload-time = "2026-05-18T04:31:00.836Z" },
+ { url = "https://files.pythonhosted.org/packages/91/15/2fe99557e72f85627c6a8eed50d889e8d101623e060a22ad75b875cb932d/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5327989a465505f05cfe06f04fa9d0c2fd5432bb243e10e6f012b1bdca3c8579", size = 459596, upload-time = "2026-05-18T04:31:34.96Z" },
+ { url = "https://files.pythonhosted.org/packages/ed/23/d4acfa0023367428ed48351b3b9b267893037b6cadae55620c61c24bcfd4/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ecb47f183a8025b2aa18b546725c3657e542112ae9c0613a2af79b4fa8d04ad7", size = 490869, upload-time = "2026-05-18T04:31:59.923Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/5f/3164cbdce06c9fb95c4f7b9e2f9760b5e2797af43a9ecc317ef42a23a278/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8520a4ab0e37f770afc34459c4f8f7019e153f9124dc101c15538365875d1ab2", size = 571641, upload-time = "2026-05-18T04:32:00.948Z" },
+ { url = "https://files.pythonhosted.org/packages/41/e6/85d3731c55e65cd7690f3f803d24c139588aaf863e4bf2148fe7a7fa1a19/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:71cd71740ed2c15211ebb237ced4e39a1cdf6f80566e5fe95428da1626f4fde6", size = 464444, upload-time = "2026-05-18T04:30:34.298Z" },
+ { url = "https://files.pythonhosted.org/packages/f4/7d/562641012b8b09872742c3b8adf9629ec479fd78f8d68ae4a0c13da8add6/watchfiles-1.2.0-cp315-cp315-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f88af53d6ddaf72179ef613ddc905e6f4785f712b49b80b3bef9f3525e6194b4", size = 453593, upload-time = "2026-05-18T04:31:23.464Z" },
+ { url = "https://files.pythonhosted.org/packages/56/fe/cb8ef3d6f929d14158fdaaad9925985b7310abc9384dcd4d82dd0016fb59/watchfiles-1.2.0-cp315-cp315-manylinux_2_31_riscv64.whl", hash = "sha256:cee9d5efd929efdac5f7e58f72b3376f676b64050a91c5b99a7094c5b2317488", size = 465096, upload-time = "2026-05-18T04:31:30.384Z" },
+ { url = "https://files.pythonhosted.org/packages/25/91/80908e835e100527a9267147b08c0eee1fa6ab0ffec15edc04d1d44885f7/watchfiles-1.2.0-cp315-cp315-musllinux_1_1_aarch64.whl", hash = "sha256:b718bf356bbc15e559bd8ef41782b573b8ae0e3f177ab244b440568d7ea02cfb", size = 630638, upload-time = "2026-05-18T04:30:49.89Z" },
+ { url = "https://files.pythonhosted.org/packages/46/4b/95ab2f256bb4af3cb2eb23b9317bda984ee6e0f11733a5c004a6c95b06e3/watchfiles-1.2.0-cp315-cp315-musllinux_1_1_x86_64.whl", hash = "sha256:922c0e019fe68b3ae392965a766b02a71ba1168c932cebc3733cd52c5fe5b377", size = 657684, upload-time = "2026-05-18T04:31:32.027Z" },
+]
+
+[[package]]
+name = "wcwidth"
+version = "0.8.1"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/49/b4/51fe890511f0f242d07cb1ebe6a5b6db417262b9d2568b460347c57d95cc/wcwidth-0.8.1.tar.gz", hash = "sha256:faf5b4a5366a72dc49cad48cdf21f52bdf63bdda995178e483ba247ff79089b9", size = 1466072, upload-time = "2026-06-08T05:57:23.146Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/bd/6e/95b0e537de1f4d4301f76f944642c6da50d1511cc7b3d64dc418a66c7509/wcwidth-0.8.1-py3-none-any.whl", hash = "sha256:f453740b1e4a4f3291faa37944c555d71056c4da08d59809b307ef4feba695c8", size = 323092, upload-time = "2026-06-08T05:57:21.413Z" },
+]
+
+[[package]]
+name = "widgetsnbextension"
+version = "4.0.15"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/bd/f4/c67440c7fb409a71b7404b7aefcd7569a9c0d6bd071299bf4198ae7a5d95/widgetsnbextension-4.0.15.tar.gz", hash = "sha256:de8610639996f1567952d763a5a41af8af37f2575a41f9852a38f947eb82a3b9", size = 1097402, upload-time = "2025-11-01T21:15:55.178Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3f/0e/fa3b193432cfc60c93b42f3be03365f5f909d2b3ea410295cf36df739e31/widgetsnbextension-4.0.15-py3-none-any.whl", hash = "sha256:8156704e4346a571d9ce73b84bee86a29906c9abfd7223b7228a28899ccf3366", size = 2196503, upload-time = "2025-11-01T21:15:53.565Z" },
+]
+
+[[package]]
+name = "win32-setctime"
+version = "1.2.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b3/8f/705086c9d734d3b663af0e9bb3d4de6578d08f46b1b101c2442fd9aecaa2/win32_setctime-1.2.0.tar.gz", hash = "sha256:ae1fdf948f5640aae05c511ade119313fb6a30d7eabe25fef9764dca5873c4c0", size = 4867, upload-time = "2024-12-07T15:28:28.314Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/e1/07/c6fe3ad3e685340704d314d765b7912993bcb8dc198f0e7a89382d37974b/win32_setctime-1.2.0-py3-none-any.whl", hash = "sha256:95d644c4e708aba81dc3704a116d8cbc974d70b3bdb8be1d150e36be6e9d1390", size = 4083, upload-time = "2024-12-07T15:28:26.465Z" },
+]
+
+[[package]]
+name = "xarray"
+version = "2026.7.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "pandas" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/ea/96/3f7bdd00e505ec3698903415b30135024703e017b28c6b61f98da3193b0d/xarray-2026.7.0.tar.gz", hash = "sha256:361b495928fdbf5b58d0969bb6775339019da5e93ca74d61ddf4eb5edd6ce604", size = 3145348, upload-time = "2026-07-09T17:38:26.515Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/57/5b/28365212062939d213802e5e5fe855cdb231e1c2a93254ba1690066504e3/xarray-2026.7.0-py3-none-any.whl", hash = "sha256:bf9dd130b93806dc78e90c1b7ac24851b31557b888674c53622235691cf21824", size = 1426778, upload-time = "2026-07-09T17:38:24.224Z" },
+]
+
+[[package]]
+name = "yarl"
+version = "1.24.2"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "idna" },
+ { name = "multidict" },
+ { name = "propcache" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/79/12/1e8f37460ea0f7eb59c221fdaf0ed75e7ac43e97f8093b9c6f411df50a78/yarl-1.24.2.tar.gz", hash = "sha256:9ac374123c6fd7abf64d1fec93962b0bd4ee2c19751755a762a72dd96c0378f8", size = 210798, upload-time = "2026-05-19T21:31:05.599Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/f0/da/866bcb01076ba49d2b42b309867bed3826421f1c479655eb7a607b44f20b/yarl-1.24.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:b975866c184564c827e0877380f0dae57dcca7e52782128381b72feff6dfceb8", size = 129957, upload-time = "2026-05-19T21:28:51.695Z" },
+ { url = "https://files.pythonhosted.org/packages/bf/1d/fcefb70922ea2268a8971d8e5874d9a8218644200fb8465f1dcad55e6851/yarl-1.24.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3b075301a2836a0e297b1b658cb6d6135df535d62efefdd60366bd589c2c82f2", size = 92164, upload-time = "2026-05-19T21:28:53.242Z" },
+ { url = "https://files.pythonhosted.org/packages/29/b6/170e2b8d4e3bc30e6bfdcca53556537f5bf595e938632dfcb059311f3ff6/yarl-1.24.2-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:8ae44649b00947634ab0dab2a374a638f52923a6e67083f2c156cd5cbd1a881d", size = 91688, upload-time = "2026-05-19T21:28:54.865Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/a5/c9f655d5553ea0b99fdac9d6a99ad3f9b3e73b8e5758bb46f58c9831f74c/yarl-1.24.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:507cc19f0b45454e2d6dcd62ff7d062b9f77a2812404e62dbdaec05b50faa035", size = 102902, upload-time = "2026-05-19T21:28:56.963Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/bc/6b9664d815d79af4ee553337f9d606c56bbf269186ada9172de45f1b5f60/yarl-1.24.2-cp312-cp312-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c4c17bad5a530912d2111825d3f05e89bab2dd376aaa8cbc77e449e6db63e576", size = 97931, upload-time = "2026-05-19T21:28:58.56Z" },
+ { url = "https://files.pythonhosted.org/packages/98/ec/32ba48acae30fecd60928f5791188b80a9d6ee3840507ffda29fecd37b71/yarl-1.24.2-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f5f0cbb112838a4a293985b6ed73948a547dadcc1ba6d2089938e7abdedceef8", size = 111030, upload-time = "2026-05-19T21:29:00.148Z" },
+ { url = "https://files.pythonhosted.org/packages/82/5a/6f4cd081e5f4934d2ae3a8ef4abe3afacc010d26f0035ee91b35cd7d7c37/yarl-1.24.2-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5ec8356b8a6afcf81fc7aeeef13b1ff7a49dec00f313394bbb9e83830d32ccd7", size = 110392, upload-time = "2026-05-19T21:29:02.155Z" },
+ { url = "https://files.pythonhosted.org/packages/7a/da/323a01c349bd5fb01bb6652e314d9bb218cee630a736bdb810ad50e4013f/yarl-1.24.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7e7ebcdef69dec6c6451e616f32b622a6d4a2e92b445c992f7c8e5274a6bbc4c", size = 105612, upload-time = "2026-05-19T21:29:04.247Z" },
+ { url = "https://files.pythonhosted.org/packages/7c/80/264ab684f181e1a876389374519ff05d10248725535ae2ac4e8ac4e563d6/yarl-1.24.2-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:47a55d6cf6db2f401017a9e96e5288844e5051911fb4e0c8311a3980f5e59a7d", size = 104487, upload-time = "2026-05-19T21:29:06.491Z" },
+ { url = "https://files.pythonhosted.org/packages/41/07/efabe5df87e96d7ad5959760b888344be48cd6884db127b407c6b5503adc/yarl-1.24.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3065657c80a2321225e804048597ad55658a7e76b32d6f5ee4074d04c50401db", size = 102333, upload-time = "2026-05-19T21:29:08.267Z" },
+ { url = "https://files.pythonhosted.org/packages/44/0c/bcf7c42603e1009295f586d8890f2ba032c8b53310e815adf0a202c73d9f/yarl-1.24.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:cb84b80d88e19ede158619b80813968713d8d008b0e2497a576e6a0557d50712", size = 99025, upload-time = "2026-05-19T21:29:10.682Z" },
+ { url = "https://files.pythonhosted.org/packages/4f/82/84482ab1a57a0f21a08afe6a7004c61d741f8f2ecc3b05c321577c612164/yarl-1.24.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:990de4f680b1c217e77ff0d6aa0029f9eb79889c11fb3e9a3942c7eba29c1996", size = 110507, upload-time = "2026-05-19T21:29:12.954Z" },
+ { url = "https://files.pythonhosted.org/packages/c4/8d/a546ba1dfe1b0f290e05fef145cd07614c0f15df1a707195e512d1e39d1d/yarl-1.24.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:abb8ec0323b80161e3802da3150ef660b41d0e9be2048b76a363d93eee992c2b", size = 103719, upload-time = "2026-05-19T21:29:14.893Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/b6/267f2a09213138473adfce6b8a6e17791d7fee70bd4d9003218e4dec58b0/yarl-1.24.2-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:e7977781f83638a4c73e0f88425563d70173e0dfd90ac006a45c65036293ee3c", size = 110438, upload-time = "2026-05-19T21:29:16.485Z" },
+ { url = "https://files.pythonhosted.org/packages/48/2d/1c8d89c7c5f9cad9fb2902445d94e2ab1d7aa35de029afbb8ae95c42d00f/yarl-1.24.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e30dd55825dc554ec5b66a94953b8eda8745926514c5089dfcacecb9c99b5bd1", size = 105719, upload-time = "2026-05-19T21:29:18.367Z" },
+ { url = "https://files.pythonhosted.org/packages/a7/25/722e3b93bd687009afb2d59a35e13d30ddd8f80571445bb0c4e4ce26ec66/yarl-1.24.2-cp312-cp312-win_amd64.whl", hash = "sha256:7dafe10c12ddd4d120d528c4b5599c953bd7b12845347d507b95451195bb6cad", size = 92901, upload-time = "2026-05-19T21:29:20.014Z" },
+ { url = "https://files.pythonhosted.org/packages/39/47/4486ccfb674c04854a1ef8aa77868b6a6f765feaf69633409d7ca4f02cb8/yarl-1.24.2-cp312-cp312-win_arm64.whl", hash = "sha256:044a09d8401fcf8681977faef6d286b8ade1e2d2e9dceda175d1cfa5ca496f30", size = 87229, upload-time = "2026-05-19T21:29:22.1Z" },
+ { url = "https://files.pythonhosted.org/packages/82/62/fcf0ce677f17e5c471c06311dd25964be38a4c586993632910d2e75278bc/yarl-1.24.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:491ac9141decf49ee8030199e1ee251cdff0e131f25678817ff6aa5f837a3536", size = 128978, upload-time = "2026-05-19T21:29:23.83Z" },
+ { url = "https://files.pythonhosted.org/packages/d3/58/8e63299bb71ed61a834121d9d3fe6c9fcf2a6a5d09754ff4f20f2d20baf5/yarl-1.24.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e89418f65eda18f99030386305bd44d7d504e328a7945db1ead514fbe03a0607", size = 91733, upload-time = "2026-05-19T21:29:25.375Z" },
+ { url = "https://files.pythonhosted.org/packages/c1/24/16748d5dab6daec8b0ed81ccec639a1cded0f18dcc62a4f696b4fe366c37/yarl-1.24.2-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:cdfcce633b4a4bb8281913c57fcafd4b5933fbc19111a5e3930bbd299d6102f1", size = 91113, upload-time = "2026-05-19T21:29:26.928Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/66/b63fff7b71211e866624b21432d5943cbb633eb0c2872d9ee3070648f22c/yarl-1.24.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:863297ddede92ee49024e9a9b11ecb59f310ca85b60d8537f56bed9bbb5b1986", size = 103899, upload-time = "2026-05-19T21:29:28.842Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/ac/ba1974b8533909636f7733fe86cf677e3619527c3c2fa913e0ea89c48757/yarl-1.24.2-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:374423f70754a2c96942ede36a29d37dc6b0cb8f92f8d009ddf3ed78d3da5488", size = 97862, upload-time = "2026-05-19T21:29:31.086Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/a5/123ac993b5c2ba6f554a140305620cb8f150fa543711bbc49be3ec0a65a4/yarl-1.24.2-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:33a29b5d00ccbf3219bb3e351d7875739c19481e030779f48cc46a7a71681a9b", size = 111060, upload-time = "2026-05-19T21:29:32.657Z" },
+ { url = "https://files.pythonhosted.org/packages/23/37/c472d3af3509688392134a88a825276770a187f1daa4de3f6dc0a327a751/yarl-1.24.2-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a9532c57211730c515341af11fef6e9b61d157487272a096d0c04da445642592", size = 110613, upload-time = "2026-05-19T21:29:34.379Z" },
+ { url = "https://files.pythonhosted.org/packages/df/88/09c28dad91e662ccfaa1b78f1c57badde74fc9d0b23e74aef644750ecd73/yarl-1.24.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:91e72cf093fd833483a97ee648e0c053c7c629f51ff4a0e7edd84f806b0c5617", size = 107012, upload-time = "2026-05-19T21:29:36.216Z" },
+ { url = "https://files.pythonhosted.org/packages/07/ab/9d4f69d571a94f4d112fa7e2e007200f5a54d319f58c82ac7b7baa61f5c6/yarl-1.24.2-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b3177bc0a768ef3bacceb4f272632990b7bea352f1b2f1eee9d6d6ff16516f92", size = 105887, upload-time = "2026-05-19T21:29:38.746Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/9a/000b2b66c0d772a499fc531d21dab92dfeb73b640a12eed6ba89f49bb2d0/yarl-1.24.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:e196952aacaf3b232e265ff02980b64d483dc0972bd49bcb061171ff22ac203a", size = 103620, upload-time = "2026-05-19T21:29:40.368Z" },
+ { url = "https://files.pythonhosted.org/packages/41/7c/7c1050f73450fbdaa3f0c72017059f00ce5e13366692f3dba25275a1083d/yarl-1.24.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:204e7a61ce99919c0de1bf904ab5d7aa188a129ea8f690a8f76cfb6e2844dc44", size = 100599, upload-time = "2026-05-19T21:29:42.66Z" },
+ { url = "https://files.pythonhosted.org/packages/ec/b1/29e5756b3926705f5f6089bd5b9f50a56eaac550da6e260bf713ead44d04/yarl-1.24.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:4b156914620f0b9d78dc1adb3751141daee561cfec796088abb89ed49d220f1a", size = 110604, upload-time = "2026-05-19T21:29:44.632Z" },
+ { url = "https://files.pythonhosted.org/packages/a3/4b/8415bc96e9b150cde942fbac9a8182985e58f40ce5c54c34ed015407d3ee/yarl-1.24.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:8372a2b976cf70654b2be6619ab6068acabb35f724c0fda7b277fbf53d66a5cf", size = 105161, upload-time = "2026-05-19T21:29:46.755Z" },
+ { url = "https://files.pythonhosted.org/packages/8b/d4/cde059abfa229553b7298a2eadde2752e723d50aeedaef86ce59da2718ee/yarl-1.24.2-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:f9a1e9b622ca284143aab5d885848686dcd85453bb1ca9abcdb7503e64dc0056", size = 110619, upload-time = "2026-05-19T21:29:48.972Z" },
+ { url = "https://files.pythonhosted.org/packages/e7/2c/d6a6c9a61549f7b6c7e6dc6937d195bcf069582b47b7200dcd0e7b256acf/yarl-1.24.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:810e19b685c8c3c5862f6a38160a1f4e4c0916c9390024ec347b6157a45a0992", size = 107362, upload-time = "2026-05-19T21:29:51Z" },
+ { url = "https://files.pythonhosted.org/packages/92/dd/3ae5fe417e9d1c353a548553326eb9935e76b6b727161563b424cc296df3/yarl-1.24.2-cp313-cp313-win_amd64.whl", hash = "sha256:7d37fb7c38f2b6edab0f845c4f85148d4c44204f52bc127021bd2bc9fdbf1656", size = 92667, upload-time = "2026-05-19T21:29:52.743Z" },
+ { url = "https://files.pythonhosted.org/packages/10/cc/a7beb239f78f27fca1b053c8e8595e4179c02e62249b4687ec218c370c50/yarl-1.24.2-cp313-cp313-win_arm64.whl", hash = "sha256:1e831894be7c2954240e49791fa4b50c05a0dc881de2552cfe3ffd8631c7f461", size = 87069, upload-time = "2026-05-19T21:29:54.442Z" },
+ { url = "https://files.pythonhosted.org/packages/40/0e/e08087695fc12789263821c5dc0f8dc52b5b17efd0887cacf419f8a43ba3/yarl-1.24.2-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:f9312b3c02d9b3d23840f67952913c9c8721d7f1b7db305289faefa878f364c2", size = 129670, upload-time = "2026-05-19T21:29:56.631Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/98/ab4b5ed1b1b5cd973c8a3eb994c3a6aefb6ce6d399e21bb5f0316c33815c/yarl-1.24.2-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:a4f4d6cd615823bfc7fb7e9b5987c3f41666371d870d51058f77e2680fbe9630", size = 91916, upload-time = "2026-05-19T21:29:58.645Z" },
+ { url = "https://files.pythonhosted.org/packages/ba/b1/5297bb6a7df4782f7605bffc43b31f5044070935fbbcaa6c705a07e6ac65/yarl-1.24.2-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0c3063e5c0a8e8e62fae6c2596fa01da1561e4cd1da6fec5789f5cf99a8aefd8", size = 91625, upload-time = "2026-05-19T21:30:00.412Z" },
+ { url = "https://files.pythonhosted.org/packages/02/a7/45baabfff76829264e623b185cff0c340d7e11bf3e1cd9ea37e7d17934bd/yarl-1.24.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fecd17873a096036c1c87ab3486f1aef7f269ada7f23f7f856f93b1cc7744f14", size = 104574, upload-time = "2026-05-19T21:30:02.544Z" },
+ { url = "https://files.pythonhosted.org/packages/f3/40/3a5ab144d3d650ca37d4f4b57e56169be8af3ca34c448793e064b30baaed/yarl-1.24.2-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:a46d1ab4ba4d32e6dc80daf8a28ce0bd83d08df52fbc32f3e288663427734535", size = 97534, upload-time = "2026-05-19T21:30:04.319Z" },
+ { url = "https://files.pythonhosted.org/packages/9c/b5/5658fef3681fb5776b4513b052bec750009f47b3a592251c705d75375798/yarl-1.24.2-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:73e68edf6dfd5f73f9ca127d84e2a6f9213c65bdffb736bda19524c0564fcd14", size = 111481, upload-time = "2026-05-19T21:30:05.988Z" },
+ { url = "https://files.pythonhosted.org/packages/4c/06/fdcd7dde037f00866dce123ed4ba23dba94beb56fc4cf561668d27be37f2/yarl-1.24.2-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a296ca617f2d25fbceafb962b88750d627e5984e75732c712154d058ae8d79a3", size = 111529, upload-time = "2026-05-19T21:30:07.738Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/53/d81269aaafccea0d33396c03035de997b743f11e648e6e27a0df99c72980/yarl-1.24.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e51b2cf5ec89a8b8470177641ed62a3ba22d74e1e898e06ad53aa77972487208", size = 107338, upload-time = "2026-05-19T21:30:09.713Z" },
+ { url = "https://files.pythonhosted.org/packages/ae/04/23049463f729bd899df203a7960505a75333edd499cda8aa1d5a82b64df5/yarl-1.24.2-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:310fc687f7b2044ec54e372c8cbe923bb88f5c37bded0d3079e5791c2fc3cf50", size = 106147, upload-time = "2026-05-19T21:30:11.365Z" },
+ { url = "https://files.pythonhosted.org/packages/14/18/04a4b5830b43ed5e4c5015b40e9f6241ad91487d71611061b4e111d6ac80/yarl-1.24.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:297a2fe352ecf858b30a98f87948746ec16f001d279f84aebdbd3bd965e2f1bd", size = 104272, upload-time = "2026-05-19T21:30:12.978Z" },
+ { url = "https://files.pythonhosted.org/packages/5a/f7/8cffdf319aee7a7c1dbd07b61d91c3e3fda460c7a93b5f93e445f3806c4c/yarl-1.24.2-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:2a263e76b97bc42bdcd7c5f4953dec1f7cd62a1112fa7f869e57255229390d67", size = 99962, upload-time = "2026-05-19T21:30:15.001Z" },
+ { url = "https://files.pythonhosted.org/packages/d7/39/b3cce3b7dbef64ac700ad4cea156a207d01bede0f507587616c364b5468e/yarl-1.24.2-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:822519b64cf0b474f1a0aaef1dc621438ea46bb77c94df97a5b4d213a7d8a8b1", size = 111063, upload-time = "2026-05-19T21:30:16.683Z" },
+ { url = "https://files.pythonhosted.org/packages/a1/ea/100818505e7ebf165c7242ff17fdf7d9fee79e27234aeca871c1082920d7/yarl-1.24.2-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:b6067060d9dc594899ba83e6db6c48c68d1e494a6dab158156ed86977ca7bcb1", size = 105438, upload-time = "2026-05-19T21:30:18.769Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/d2/e075a0b32aa6625087de9e653087df0759fed5de4a435fef594181102a77/yarl-1.24.2-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:0063adad533e57171b79db3943b229d40dfafeeee579767f96541f106bac5f1b", size = 111458, upload-time = "2026-05-19T21:30:21.024Z" },
+ { url = "https://files.pythonhosted.org/packages/e6/5c/ceea7ba98b65c8eb8d947fdc52f9bedfcd43c6a57c9e3c90c17be8f324a3/yarl-1.24.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ee8e3fb34513e8dc082b586ef4910c98335d43a6fab688cd44d4851bacfce3e8", size = 107589, upload-time = "2026-05-19T21:30:23.412Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/d9/5582d57e2b2db9b85eb6663a22efdd78e08805f3f5389566e9fcad254d1b/yarl-1.24.2-cp314-cp314-win_amd64.whl", hash = "sha256:afb00d7fd8e0f285ca29a44cc50df2d622ff2f7a6d933fa641577b5f9d5f3db0", size = 94424, upload-time = "2026-05-19T21:30:25.425Z" },
+ { url = "https://files.pythonhosted.org/packages/92/10/7dc07a0e22806a9280f42a57361395506e800c64e22737cd7b0886feab42/yarl-1.24.2-cp314-cp314-win_arm64.whl", hash = "sha256:68cf6eacd6028ef1142bc4b48376b81566385ca6f9e7dde3b0fa91be08ffcb57", size = 88690, upload-time = "2026-05-19T21:30:27.623Z" },
+ { url = "https://files.pythonhosted.org/packages/9e/13/d5b8e2c8667db955bcb3de233f18798fefe7edf1d7429c2c9d4f9c401114/yarl-1.24.2-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:221ce1dd921ac4f603957f17d7c18c5cc0797fbb52f156941f92e04605d1d67b", size = 136248, upload-time = "2026-05-19T21:30:29.297Z" },
+ { url = "https://files.pythonhosted.org/packages/de/46/a4a97c05c9c9b8fd266bb2a0df12992c7fbd02391eb9640583411b6dab32/yarl-1.24.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:5f3224db28173a00d7afacdee07045cc4673dfab2b15492c7ae10deddbece761", size = 95084, upload-time = "2026-05-19T21:30:31.031Z" },
+ { url = "https://files.pythonhosted.org/packages/95/b2/845cf2074a015e6fe0d0808cf1a2d9e868386c4220d657ebd8302b199043/yarl-1.24.2-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c557165320d6244ebe3a02431b2a201a20080e02f41f0cfa0ccc47a183765da8", size = 95272, upload-time = "2026-05-19T21:30:33.062Z" },
+ { url = "https://files.pythonhosted.org/packages/fe/16/e69d4aa244aef45235ddfebc0e04036a6829842bc5a6a795aedc6c998d23/yarl-1.24.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:904065e6e85b1fa54d0d87438bd58c14c0bad97aad654ad1077fd9d87e8478ed", size = 101497, upload-time = "2026-05-19T21:30:34.842Z" },
+ { url = "https://files.pythonhosted.org/packages/15/94/c07107715d621076863ee88b3ddf183fa5e9d4aba5769623c9979828410a/yarl-1.24.2-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:8cec2a38d70edc10e0e856ceda886af5327a017ccbde8e1de1bd44d300357543", size = 94002, upload-time = "2026-05-19T21:30:37.724Z" },
+ { url = "https://files.pythonhosted.org/packages/a9/35/fc1bbdd895b5e4010b8fdd037f7ed3aa289d3863e08231b30231ca9a0815/yarl-1.24.2-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e7484b9361ed222ee1ca5b4337aa4cbdcc4618ce5aff57d9ef1582fd95893fc0", size = 106524, upload-time = "2026-05-19T21:30:40.196Z" },
+ { url = "https://files.pythonhosted.org/packages/1f/f2/32b66d0a4ba47c296cf86d03e2c67bff58399fe6d6d84d5205c04c66cc6d/yarl-1.24.2-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:84f9670b89f34db07f81e53aee83e0b938a3412329d51c8f922488be7fcc4024", size = 106165, upload-time = "2026-05-19T21:30:41.888Z" },
+ { url = "https://files.pythonhosted.org/packages/95/47/37cb5ff50c5e825d4d38e81bb04d1b7e96bf960f7ab89f9850b162f3f114/yarl-1.24.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:abb2759733d63a28b4956500a5dd57140f26486c92b2caedfb964ab7d9b79dbf", size = 103010, upload-time = "2026-05-19T21:30:43.985Z" },
+ { url = "https://files.pythonhosted.org/packages/6f/d2/4597912315096f7bb359e46e13bf8b60994fcbb2db29b804c0902ef4eff5/yarl-1.24.2-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:081c2bf54efe03774d0311172bc04fedf9ca01e644d4cd8c805688e527209bdc", size = 101128, upload-time = "2026-05-19T21:30:46.291Z" },
+ { url = "https://files.pythonhosted.org/packages/b9/d5/c8e86e120521e646013d02a8e3b8884392e28494be8f392366e50d208efc/yarl-1.24.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:86746bef442aa479107fe28132e1277237f9c24c2f00b0b0cf22b3ee0904f2bb", size = 101382, upload-time = "2026-05-19T21:30:48.085Z" },
+ { url = "https://files.pythonhosted.org/packages/fa/98/70b229236118f89dbeb739b76f10225bbf53b5497725502594c9a01d699a/yarl-1.24.2-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:2d07d21d0bc4b17558e8de0b02fbfdf1e347d3bb3699edd00bb92e7c57925420", size = 95964, upload-time = "2026-05-19T21:30:49.785Z" },
+ { url = "https://files.pythonhosted.org/packages/87/f8/56c386981e3c8648d279fdef2397ffec577e8320fd5649745e34d54faeb7/yarl-1.24.2-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:4fb1ac3fc5fecd8ae7453ea237e4d22b49befa70266dfe1629924245c21a0c7f", size = 106204, upload-time = "2026-05-19T21:30:51.862Z" },
+ { url = "https://files.pythonhosted.org/packages/1a/1e/765afe97811ca35933e2a7de70ac57b1997ea2e4ee895719ee7a231fb7e5/yarl-1.24.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:4da31a5512ed1729ca8d8aacde3f7faeb8843cde3165d6bcf7f88f74f17bb8aa", size = 101510, upload-time = "2026-05-19T21:30:53.62Z" },
+ { url = "https://files.pythonhosted.org/packages/ee/78/393913f4b9039e1edd09ae8a9bbb9d539be909a8abf6d8a2084585bed4b7/yarl-1.24.2-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:533ded4dceb5f1f3da7906244f4e82cf46cfd40d84c69a1faf5ac506aa65ecbe", size = 105584, upload-time = "2026-05-19T21:30:55.962Z" },
+ { url = "https://files.pythonhosted.org/packages/78/87/deb17b7049bbe74ea11a713b86f8f27800cc1c8648b0b797243ebb4830ba/yarl-1.24.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:7b3a85525f6e7eeabcfdd372862b21ee1915db1b498a04e8bf0e389b607ff0bd", size = 103410, upload-time = "2026-05-19T21:30:57.962Z" },
+ { url = "https://files.pythonhosted.org/packages/8f/be/f9f7594e23b5b93affff0318e4593c1920331bcaefda326cabcad94296a1/yarl-1.24.2-cp314-cp314t-win_amd64.whl", hash = "sha256:a7624b1ca46ca5d7b864ef0d2f8efe3091454085ee1855b4e992314529972215", size = 102980, upload-time = "2026-05-19T21:30:59.735Z" },
+ { url = "https://files.pythonhosted.org/packages/65/a4/ba80dccd3593ff1f01051a818694d07b58cb8232677ee9a22a5a1f93a9fc/yarl-1.24.2-cp314-cp314t-win_arm64.whl", hash = "sha256:e434a45ce2e7a947f951fc5a8944c8cc080b7e59f9c50ae80fd39107cf88126d", size = 91219, upload-time = "2026-05-19T21:31:01.934Z" },
+ { url = "https://files.pythonhosted.org/packages/fd/4d/4b880086bd0d3e034d25647be1d830afc3e3f610e98c4ab3490af6b1b6d5/yarl-1.24.2-py3-none-any.whl", hash = "sha256:2783d9226db8797636cd6896e4de81feed252d1db72265686c9558d97a4d94b9", size = 53576, upload-time = "2026-05-19T21:31:03.909Z" },
+]
+
+[[package]]
+name = "zarr"
+version = "3.2.1"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+ { name = "donfig" },
+ { name = "google-crc32c" },
+ { name = "numcodecs" },
+ { name = "numpy" },
+ { name = "packaging" },
+ { name = "typing-extensions" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/93/8d/aeb164004f87543b06ef54f885d02c342c31ceb274e2bbec470a98927621/zarr-3.2.1.tar.gz", hash = "sha256:71565b738a0e7e8ed226f0516eba8c6bb53440ad7669a8c48ebb3534a161d035", size = 675161, upload-time = "2026-05-05T12:37:22.383Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/88/0a/469e2bd01be1490336e6c8707386845655d59261543315778a3ccc7e8019/zarr-3.2.1-py3-none-any.whl", hash = "sha256:f78cdd3d9687ad0e9f9cba2c5683b64f0c52589c19f685eeabe872e93cc0d2c7", size = 319617, upload-time = "2026-05-05T12:37:20.66Z" },
+]
+
+[[package]]
+name = "zipp"
+version = "4.1.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/b9/d8/eab98a517c14134c0b2eb4e2387bc5f457334293ec5d2dd3857ec2966802/zipp-4.1.0.tar.gz", hash = "sha256:4cb57381f544315db7688e976e922a2b18cdb513d21cc194eb42232ba2a3e602", size = 26214, upload-time = "2026-05-18T20:08:57.967Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/3a/13/547360d81e6d88d58492968ffda9f9542854f11310ee556fef14260cc886/zipp-4.1.0-py3-none-any.whl", hash = "sha256:25ad4e16390cd314347dd8f1de67a2ac538ae658ed4ab9db16029c07c188e97f", size = 10238, upload-time = "2026-05-18T20:08:57.045Z" },
+]
+
+[[package]]
+name = "zstandard"
+version = "0.25.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/fd/aa/3e0508d5a5dd96529cdc5a97011299056e14c6505b678fd58938792794b1/zstandard-0.25.0.tar.gz", hash = "sha256:7713e1179d162cf5c7906da876ec2ccb9c3a9dcbdffef0cc7f70c3667a205f0b", size = 711513, upload-time = "2025-09-14T22:15:54.002Z" }
+wheels = [
+ { url = "https://files.pythonhosted.org/packages/82/fc/f26eb6ef91ae723a03e16eddb198abcfce2bc5a42e224d44cc8b6765e57e/zstandard-0.25.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7b3c3a3ab9daa3eed242d6ecceead93aebbb8f5f84318d82cee643e019c4b73b", size = 795738, upload-time = "2025-09-14T22:16:56.237Z" },
+ { url = "https://files.pythonhosted.org/packages/aa/1c/d920d64b22f8dd028a8b90e2d756e431a5d86194caa78e3819c7bf53b4b3/zstandard-0.25.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:913cbd31a400febff93b564a23e17c3ed2d56c064006f54efec210d586171c00", size = 640436, upload-time = "2025-09-14T22:16:57.774Z" },
+ { url = "https://files.pythonhosted.org/packages/53/6c/288c3f0bd9fcfe9ca41e2c2fbfd17b2097f6af57b62a81161941f09afa76/zstandard-0.25.0-cp312-cp312-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:011d388c76b11a0c165374ce660ce2c8efa8e5d87f34996aa80f9c0816698b64", size = 5343019, upload-time = "2025-09-14T22:16:59.302Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/15/efef5a2f204a64bdb5571e6161d49f7ef0fffdbca953a615efbec045f60f/zstandard-0.25.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6dffecc361d079bb48d7caef5d673c88c8988d3d33fb74ab95b7ee6da42652ea", size = 5063012, upload-time = "2025-09-14T22:17:01.156Z" },
+ { url = "https://files.pythonhosted.org/packages/b7/37/a6ce629ffdb43959e92e87ebdaeebb5ac81c944b6a75c9c47e300f85abdf/zstandard-0.25.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:7149623bba7fdf7e7f24312953bcf73cae103db8cae49f8154dd1eadc8a29ecb", size = 5394148, upload-time = "2025-09-14T22:17:03.091Z" },
+ { url = "https://files.pythonhosted.org/packages/e3/79/2bf870b3abeb5c070fe2d670a5a8d1057a8270f125ef7676d29ea900f496/zstandard-0.25.0-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:6a573a35693e03cf1d67799fd01b50ff578515a8aeadd4595d2a7fa9f3ec002a", size = 5451652, upload-time = "2025-09-14T22:17:04.979Z" },
+ { url = "https://files.pythonhosted.org/packages/53/60/7be26e610767316c028a2cbedb9a3beabdbe33e2182c373f71a1c0b88f36/zstandard-0.25.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5a56ba0db2d244117ed744dfa8f6f5b366e14148e00de44723413b2f3938a902", size = 5546993, upload-time = "2025-09-14T22:17:06.781Z" },
+ { url = "https://files.pythonhosted.org/packages/85/c7/3483ad9ff0662623f3648479b0380d2de5510abf00990468c286c6b04017/zstandard-0.25.0-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:10ef2a79ab8e2974e2075fb984e5b9806c64134810fac21576f0668e7ea19f8f", size = 5046806, upload-time = "2025-09-14T22:17:08.415Z" },
+ { url = "https://files.pythonhosted.org/packages/08/b3/206883dd25b8d1591a1caa44b54c2aad84badccf2f1de9e2d60a446f9a25/zstandard-0.25.0-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:aaf21ba8fb76d102b696781bddaa0954b782536446083ae3fdaa6f16b25a1c4b", size = 5576659, upload-time = "2025-09-14T22:17:10.164Z" },
+ { url = "https://files.pythonhosted.org/packages/9d/31/76c0779101453e6c117b0ff22565865c54f48f8bd807df2b00c2c404b8e0/zstandard-0.25.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1869da9571d5e94a85a5e8d57e4e8807b175c9e4a6294e3b66fa4efb074d90f6", size = 4953933, upload-time = "2025-09-14T22:17:11.857Z" },
+ { url = "https://files.pythonhosted.org/packages/18/e1/97680c664a1bf9a247a280a053d98e251424af51f1b196c6d52f117c9720/zstandard-0.25.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:809c5bcb2c67cd0ed81e9229d227d4ca28f82d0f778fc5fea624a9def3963f91", size = 5268008, upload-time = "2025-09-14T22:17:13.627Z" },
+ { url = "https://files.pythonhosted.org/packages/1e/73/316e4010de585ac798e154e88fd81bb16afc5c5cb1a72eeb16dd37e8024a/zstandard-0.25.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:f27662e4f7dbf9f9c12391cb37b4c4c3cb90ffbd3b1fb9284dadbbb8935fa708", size = 5433517, upload-time = "2025-09-14T22:17:16.103Z" },
+ { url = "https://files.pythonhosted.org/packages/5b/60/dd0f8cfa8129c5a0ce3ea6b7f70be5b33d2618013a161e1ff26c2b39787c/zstandard-0.25.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:99c0c846e6e61718715a3c9437ccc625de26593fea60189567f0118dc9db7512", size = 5814292, upload-time = "2025-09-14T22:17:17.827Z" },
+ { url = "https://files.pythonhosted.org/packages/fc/5f/75aafd4b9d11b5407b641b8e41a57864097663699f23e9ad4dbb91dc6bfe/zstandard-0.25.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:474d2596a2dbc241a556e965fb76002c1ce655445e4e3bf38e5477d413165ffa", size = 5360237, upload-time = "2025-09-14T22:17:19.954Z" },
+ { url = "https://files.pythonhosted.org/packages/ff/8d/0309daffea4fcac7981021dbf21cdb2e3427a9e76bafbcdbdf5392ff99a4/zstandard-0.25.0-cp312-cp312-win32.whl", hash = "sha256:23ebc8f17a03133b4426bcc04aabd68f8236eb78c3760f12783385171b0fd8bd", size = 436922, upload-time = "2025-09-14T22:17:24.398Z" },
+ { url = "https://files.pythonhosted.org/packages/79/3b/fa54d9015f945330510cb5d0b0501e8253c127cca7ebe8ba46a965df18c5/zstandard-0.25.0-cp312-cp312-win_amd64.whl", hash = "sha256:ffef5a74088f1e09947aecf91011136665152e0b4b359c42be3373897fb39b01", size = 506276, upload-time = "2025-09-14T22:17:21.429Z" },
+ { url = "https://files.pythonhosted.org/packages/ea/6b/8b51697e5319b1f9ac71087b0af9a40d8a6288ff8025c36486e0c12abcc4/zstandard-0.25.0-cp312-cp312-win_arm64.whl", hash = "sha256:181eb40e0b6a29b3cd2849f825e0fa34397f649170673d385f3598ae17cca2e9", size = 462679, upload-time = "2025-09-14T22:17:23.147Z" },
+ { url = "https://files.pythonhosted.org/packages/35/0b/8df9c4ad06af91d39e94fa96cc010a24ac4ef1378d3efab9223cc8593d40/zstandard-0.25.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ec996f12524f88e151c339688c3897194821d7f03081ab35d31d1e12ec975e94", size = 795735, upload-time = "2025-09-14T22:17:26.042Z" },
+ { url = "https://files.pythonhosted.org/packages/3f/06/9ae96a3e5dcfd119377ba33d4c42a7d89da1efabd5cb3e366b156c45ff4d/zstandard-0.25.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a1a4ae2dec3993a32247995bdfe367fc3266da832d82f8438c8570f989753de1", size = 640440, upload-time = "2025-09-14T22:17:27.366Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/14/933d27204c2bd404229c69f445862454dcc101cd69ef8c6068f15aaec12c/zstandard-0.25.0-cp313-cp313-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:e96594a5537722fdfb79951672a2a63aec5ebfb823e7560586f7484819f2a08f", size = 5343070, upload-time = "2025-09-14T22:17:28.896Z" },
+ { url = "https://files.pythonhosted.org/packages/6d/db/ddb11011826ed7db9d0e485d13df79b58586bfdec56e5c84a928a9a78c1c/zstandard-0.25.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:bfc4e20784722098822e3eee42b8e576b379ed72cca4a7cb856ae733e62192ea", size = 5063001, upload-time = "2025-09-14T22:17:31.044Z" },
+ { url = "https://files.pythonhosted.org/packages/db/00/87466ea3f99599d02a5238498b87bf84a6348290c19571051839ca943777/zstandard-0.25.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:457ed498fc58cdc12fc48f7950e02740d4f7ae9493dd4ab2168a47c93c31298e", size = 5394120, upload-time = "2025-09-14T22:17:32.711Z" },
+ { url = "https://files.pythonhosted.org/packages/2b/95/fc5531d9c618a679a20ff6c29e2b3ef1d1f4ad66c5e161ae6ff847d102a9/zstandard-0.25.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:fd7a5004eb1980d3cefe26b2685bcb0b17989901a70a1040d1ac86f1d898c551", size = 5451230, upload-time = "2025-09-14T22:17:34.41Z" },
+ { url = "https://files.pythonhosted.org/packages/63/4b/e3678b4e776db00f9f7b2fe58e547e8928ef32727d7a1ff01dea010f3f13/zstandard-0.25.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8e735494da3db08694d26480f1493ad2cf86e99bdd53e8e9771b2752a5c0246a", size = 5547173, upload-time = "2025-09-14T22:17:36.084Z" },
+ { url = "https://files.pythonhosted.org/packages/4e/d5/ba05ed95c6b8ec30bd468dfeab20589f2cf709b5c940483e31d991f2ca58/zstandard-0.25.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:3a39c94ad7866160a4a46d772e43311a743c316942037671beb264e395bdd611", size = 5046736, upload-time = "2025-09-14T22:17:37.891Z" },
+ { url = "https://files.pythonhosted.org/packages/50/d5/870aa06b3a76c73eced65c044b92286a3c4e00554005ff51962deef28e28/zstandard-0.25.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:172de1f06947577d3a3005416977cce6168f2261284c02080e7ad0185faeced3", size = 5576368, upload-time = "2025-09-14T22:17:40.206Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/35/398dc2ffc89d304d59bc12f0fdd931b4ce455bddf7038a0a67733a25f550/zstandard-0.25.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3c83b0188c852a47cd13ef3bf9209fb0a77fa5374958b8c53aaa699398c6bd7b", size = 4954022, upload-time = "2025-09-14T22:17:41.879Z" },
+ { url = "https://files.pythonhosted.org/packages/9a/5c/36ba1e5507d56d2213202ec2b05e8541734af5f2ce378c5d1ceaf4d88dc4/zstandard-0.25.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:1673b7199bbe763365b81a4f3252b8e80f44c9e323fc42940dc8843bfeaf9851", size = 5267889, upload-time = "2025-09-14T22:17:43.577Z" },
+ { url = "https://files.pythonhosted.org/packages/70/e8/2ec6b6fb7358b2ec0113ae202647ca7c0e9d15b61c005ae5225ad0995df5/zstandard-0.25.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:0be7622c37c183406f3dbf0cba104118eb16a4ea7359eeb5752f0794882fc250", size = 5433952, upload-time = "2025-09-14T22:17:45.271Z" },
+ { url = "https://files.pythonhosted.org/packages/7b/01/b5f4d4dbc59ef193e870495c6f1275f5b2928e01ff5a81fecb22a06e22fb/zstandard-0.25.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:5f5e4c2a23ca271c218ac025bd7d635597048b366d6f31f420aaeb715239fc98", size = 5814054, upload-time = "2025-09-14T22:17:47.08Z" },
+ { url = "https://files.pythonhosted.org/packages/b2/e5/fbd822d5c6f427cf158316d012c5a12f233473c2f9c5fe5ab1ae5d21f3d8/zstandard-0.25.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4f187a0bb61b35119d1926aee039524d1f93aaf38a9916b8c4b78ac8514a0aaf", size = 5360113, upload-time = "2025-09-14T22:17:48.893Z" },
+ { url = "https://files.pythonhosted.org/packages/8e/e0/69a553d2047f9a2c7347caa225bb3a63b6d7704ad74610cb7823baa08ed7/zstandard-0.25.0-cp313-cp313-win32.whl", hash = "sha256:7030defa83eef3e51ff26f0b7bfb229f0204b66fe18e04359ce3474ac33cbc09", size = 436936, upload-time = "2025-09-14T22:17:52.658Z" },
+ { url = "https://files.pythonhosted.org/packages/d9/82/b9c06c870f3bd8767c201f1edbdf9e8dc34be5b0fbc5682c4f80fe948475/zstandard-0.25.0-cp313-cp313-win_amd64.whl", hash = "sha256:1f830a0dac88719af0ae43b8b2d6aef487d437036468ef3c2ea59c51f9d55fd5", size = 506232, upload-time = "2025-09-14T22:17:50.402Z" },
+ { url = "https://files.pythonhosted.org/packages/d4/57/60c3c01243bb81d381c9916e2a6d9e149ab8627c0c7d7abb2d73384b3c0c/zstandard-0.25.0-cp313-cp313-win_arm64.whl", hash = "sha256:85304a43f4d513f5464ceb938aa02c1e78c2943b29f44a750b48b25ac999a049", size = 462671, upload-time = "2025-09-14T22:17:51.533Z" },
+ { url = "https://files.pythonhosted.org/packages/3d/5c/f8923b595b55fe49e30612987ad8bf053aef555c14f05bb659dd5dbe3e8a/zstandard-0.25.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:e29f0cf06974c899b2c188ef7f783607dbef36da4c242eb6c82dcd8b512855e3", size = 795887, upload-time = "2025-09-14T22:17:54.198Z" },
+ { url = "https://files.pythonhosted.org/packages/8d/09/d0a2a14fc3439c5f874042dca72a79c70a532090b7ba0003be73fee37ae2/zstandard-0.25.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:05df5136bc5a011f33cd25bc9f506e7426c0c9b3f9954f056831ce68f3b6689f", size = 640658, upload-time = "2025-09-14T22:17:55.423Z" },
+ { url = "https://files.pythonhosted.org/packages/5d/7c/8b6b71b1ddd517f68ffb55e10834388d4f793c49c6b83effaaa05785b0b4/zstandard-0.25.0-cp314-cp314-manylinux2010_i686.manylinux_2_12_i686.manylinux_2_28_i686.whl", hash = "sha256:f604efd28f239cc21b3adb53eb061e2a205dc164be408e553b41ba2ffe0ca15c", size = 5379849, upload-time = "2025-09-14T22:17:57.372Z" },
+ { url = "https://files.pythonhosted.org/packages/a4/86/a48e56320d0a17189ab7a42645387334fba2200e904ee47fc5a26c1fd8ca/zstandard-0.25.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:223415140608d0f0da010499eaa8ccdb9af210a543fac54bce15babbcfc78439", size = 5058095, upload-time = "2025-09-14T22:17:59.498Z" },
+ { url = "https://files.pythonhosted.org/packages/f8/ad/eb659984ee2c0a779f9d06dbfe45e2dc39d99ff40a319895df2d3d9a48e5/zstandard-0.25.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2e54296a283f3ab5a26fc9b8b5d4978ea0532f37b231644f367aa588930aa043", size = 5551751, upload-time = "2025-09-14T22:18:01.618Z" },
+ { url = "https://files.pythonhosted.org/packages/61/b3/b637faea43677eb7bd42ab204dfb7053bd5c4582bfe6b1baefa80ac0c47b/zstandard-0.25.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ca54090275939dc8ec5dea2d2afb400e0f83444b2fc24e07df7fdef677110859", size = 6364818, upload-time = "2025-09-14T22:18:03.769Z" },
+ { url = "https://files.pythonhosted.org/packages/31/dc/cc50210e11e465c975462439a492516a73300ab8caa8f5e0902544fd748b/zstandard-0.25.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e09bb6252b6476d8d56100e8147b803befa9a12cea144bbe629dd508800d1ad0", size = 5560402, upload-time = "2025-09-14T22:18:05.954Z" },
+ { url = "https://files.pythonhosted.org/packages/c9/ae/56523ae9c142f0c08efd5e868a6da613ae76614eca1305259c3bf6a0ed43/zstandard-0.25.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a9ec8c642d1ec73287ae3e726792dd86c96f5681eb8df274a757bf62b750eae7", size = 4955108, upload-time = "2025-09-14T22:18:07.68Z" },
+ { url = "https://files.pythonhosted.org/packages/98/cf/c899f2d6df0840d5e384cf4c4121458c72802e8bda19691f3b16619f51e9/zstandard-0.25.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a4089a10e598eae6393756b036e0f419e8c1d60f44a831520f9af41c14216cf2", size = 5269248, upload-time = "2025-09-14T22:18:09.753Z" },
+ { url = "https://files.pythonhosted.org/packages/1b/c0/59e912a531d91e1c192d3085fc0f6fb2852753c301a812d856d857ea03c6/zstandard-0.25.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:f67e8f1a324a900e75b5e28ffb152bcac9fbed1cc7b43f99cd90f395c4375344", size = 5430330, upload-time = "2025-09-14T22:18:11.966Z" },
+ { url = "https://files.pythonhosted.org/packages/a0/1d/7e31db1240de2df22a58e2ea9a93fc6e38cc29353e660c0272b6735d6669/zstandard-0.25.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:9654dbc012d8b06fc3d19cc825af3f7bf8ae242226df5f83936cb39f5fdc846c", size = 5811123, upload-time = "2025-09-14T22:18:13.907Z" },
+ { url = "https://files.pythonhosted.org/packages/f6/49/fac46df5ad353d50535e118d6983069df68ca5908d4d65b8c466150a4ff1/zstandard-0.25.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:4203ce3b31aec23012d3a4cf4a2ed64d12fea5269c49aed5e4c3611b938e4088", size = 5359591, upload-time = "2025-09-14T22:18:16.465Z" },
+ { url = "https://files.pythonhosted.org/packages/c2/38/f249a2050ad1eea0bb364046153942e34abba95dd5520af199aed86fbb49/zstandard-0.25.0-cp314-cp314-win32.whl", hash = "sha256:da469dc041701583e34de852d8634703550348d5822e66a0c827d39b05365b12", size = 444513, upload-time = "2025-09-14T22:18:20.61Z" },
+ { url = "https://files.pythonhosted.org/packages/3a/43/241f9615bcf8ba8903b3f0432da069e857fc4fd1783bd26183db53c4804b/zstandard-0.25.0-cp314-cp314-win_amd64.whl", hash = "sha256:c19bcdd826e95671065f8692b5a4aa95c52dc7a02a4c5a0cac46deb879a017a2", size = 516118, upload-time = "2025-09-14T22:18:17.849Z" },
+ { url = "https://files.pythonhosted.org/packages/f0/ef/da163ce2450ed4febf6467d77ccb4cd52c4c30ab45624bad26ca0a27260c/zstandard-0.25.0-cp314-cp314-win_arm64.whl", hash = "sha256:d7541afd73985c630bafcd6338d2518ae96060075f9463d7dc14cfb33514383d", size = 476940, upload-time = "2025-09-14T22:18:19.088Z" },
+]
+ 
+ 
+ 
+
+