`anndata` `0.2.0` on-disk format

[ilan-gold]

Please describe your wishes and possible alternatives to achieve the desired result.

A few things should probably go into a 0.2.0 version of the anndata spec:

• Writing to H5AD has changed in v0.12.1 #2052 brings up hard-specifying multi-dimensionalness of xxxm
• X should probably be disallowed after feat!: Unify X and layers #1707
• uns's restriction should be loosened to allow for JSON objects (or simply JSON objects should be considered "anndata-encoded") Allow storing heterogeneous lists in .uns (as JSON or so) #1979
• An escape character key for sub-keys of elements like obs/foo/bar - see (fix): disallow forward slash writes #1940
• A clear and defined file extension for zarr files: see Writing zarr as a zipped directory #1971 (comment) as well as anndata.io.read_zarr incompatible with a Zarr ZipStorage #2152, which would be made clean and easy by this
• Upstream categoricals if possible to github.com/zarr-conventions and in general, allow reading zarr-conventions as arrays if they can be interpreted properly e.g., genomics data
• Loosen the definition of an on-disk array to be anything that zarr officially understands/supports e.g., feat: optional codec and data type zarr-developers/zarr-extensions#33 or Add arrow-ipc Array -> Byes codec zarr-developers/zarr-extensions#41. See Handle new zarr dtypes (both custom and datetime) #2238
• Don’t require an index on-disk for dataframes
• Schemas for generating specialized anonymous indices (Cartesian coordinates, fixed length genomic regionals etc)

I also think this should be a clean zarr-only spec given the last point and the fact that it supports many more features (a bit of carrot in the carrot-and-stick of getting people to use zarr)

[flying-sheep]

Is this about changing the format in ways that aren’t covered in “encoding” changes?

OtherwiseApart from that maybe we should add #1940?

[ilan-gold]

I think you have that right if I understand you correctly, I'm talking about https://anndata.readthedocs.io/en/latest/fileformat-prose.html#anndata-specification-v0-1-0 specifically. So #1940 should probably count as we would want to specify the escape character somehow, although the "Element Specification" is not versioned and that would be the most reasonable option since making it a global setting seems wrong somehow. Perhaps it is something that could be specified as part of the anndata spec implicitly i.e., we state in this document that the escape character is XYZ and it does not need to be stored explicitly.