Allow any unicode incl. `/` and `\` in `obsm` keys, `obs` columns, `uns` keys, …

[flying-sheep]

Follow-up to #2039

The idea would be that we add metadata to the parent of each group/array or so that contains the actual name of that group/array.

This of course breaks backwards compat for everything, thus #2057

[hanslovsky]

This is, indeed, very interesting! I have a dataset with / in layer names. Writing to h5ad on 0.12.x is not an issue, but re-reading the data fails. I am commenting here to share a workaround that works for me. Others might find this useful. This is a minimal working script that first demonstrates the failure case, then provides a workaround and proofs success:

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "anndata==0.12.14",
#   "h5py==3.16.0",
#   "numpy==2.4.5",
#   "scipy==1.17.1",
# ]
# ///

import tempfile
from pathlib import Path

import anndata as ad
import h5py
import numpy as np
from scipy import sparse

with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / "tmp.h5ad"
    ad.AnnData(np.empty((0, 0)), layers={"a/b": sparse.csr_matrix((0, 0))}).write_h5ad(path)
    try:
        ad.read_h5ad(path)
    except Exception as e:
        print(f"Failed with {e}")


    def read_h5ad_with_nested_layers(path: str | Path) -> ad.AnnData:

        def flatten(d, prefix=()):
            for k, v in d.items():
                key_path = (*prefix, k)
                if isinstance(v, dict):
                    yield from flatten(v, key_path)
                else:
                    yield "/".join(key_path), v

        with h5py.File(path, "r") as f:
            return ad.AnnData(
                var=ad.io.read_elem(f["var"]),
                obs=ad.io.read_elem(f["obs"]),
                layers=dict(flatten(ad.io.read_elem(f["layers"])))
            )


    adata = read_h5ad_with_nested_layers(path)
    print(f"Success with {adata.layers['a/b']}")

Expected output:

$ uv run --quiet mwe-nested-layers.py
/Users/hanslovp/Library/Caches/uv/environments-v2/mwe-nested-layers-cb29cfab8e9de83f/lib/python3.12/site-packages/anndata/_io/utils.py:272: FutureWarning: Forward slashes will be disallowed in h5 stores in the next minor release
  return func(*args, **kwargs)
Failed with 'dict' object has no attribute 'shape'
Success with <Compressed Sparse Row sparse matrix of dtype 'float64'
	with 0 stored elements and shape (0, 0)>

I figure this would similarly work for obsm, obsp etc, not just layers.

This is, of course, not a solution for the general case, and works only for the special case of /.

[flying-sheep]

Yeah, we recently changed the code so we don’t write unreadable anndata objects like yours anymore.

So you’d need a new workaround where you e.g. use anndata.experimental.write_dispatched and anndata.experimental.read_dispatched to replace the slashes when writing with some symbol you know you don’t use, and then replace it back when reading.

The reason we don’t do that is that we can’t be sure people aren’t using any given unicode symbol, so we better fix it properly.

[hanslovsky]

Thank you for the explanation of the workaround! Not a big deal for me at this time, but I will keep this in mind. I can also just replace / with :: (or other easily identifiable str) and just leave it like that downstream. That would be the most simple solution without the need for read_dispatched/write_dispatched.