Skip to content

Pin the Zarr export to format v3 explicitly - #448

Merged
amarcozzi merged 1 commit into
mainfrom
447-zarr-export-format-v3
Jul 27, 2026
Merged

Pin the Zarr export to format v3 explicitly#448
amarcozzi merged 1 commit into
mainfrom
447-zarr-export-format-v3

Conversation

@amarcozzi

Copy link
Copy Markdown
Contributor

Refs #447. Paired with silvxlabs/fastfuels-sdk-python#189 (the client-side half).

The zarr grid export was written with a bare ds.to_zarr(zarr_dir), so its on-disk format version was whatever zarr-python defaulted to rather than something we chose. That artifact is zipped and handed to users via a signed download URL, which makes the format version part of the export contract — zarr-python 2.x cannot read format v3 at all.

Why now

v1:  zarr==2.18.2   -> exports were format v2
v2:  zarr 3.x       -> exports are format v3

A v1 migrant whose environment is still pinned to zarr 2.x downloads a v2 export and it fails to open on their machine, with a confusing error and nothing on our side indicating why.

We're staying on v3 rather than falling back to the more widely-readable v2 (zarr-python 3.x reads v2 fine, so v2 would technically be the more compatible choice for a static download). Reasoning: zarr 3 has been out ~18 months, the client base is small, we're in beta where a conscious break is acceptable, and one format across the codebase beats carrying a v2 compat path. The client-side exposure is closed by pinning zarr>=3 in the SDK.

Making the format explicit also guards against a future zarr-python default flip silently changing what users receive.

Changes

  • zarr_format=3 explicit in exporter/handlers/grid.py, with a comment on why the version is a contract and not a library default.
  • New tests/handlers/test_zarr.py: mocks the GCS upload, captures the zip, extracts, and asserts zarr_format == 3 on both the group and the array. Verified it has teeth rather than rubber-stamping the current default — flipping the kwarg to 2 fails it with assert 2 == 3.

Not included, deliberately

Internal grid stores (lib/zarr_utils.py:110) also write with the implicit default. Left alone: they're read only by our own zarr-python 3.x services and are already 100% v3 on disk (verified across 400 production stores, zero mixed-format legacy). No user tooling touches them.

Contract review

  1. Breaking change (end users): YES — deliberate. v1 users on pinned zarr 2.x envs cannot open v2 zarr exports. Already true on main; this PR makes the choice explicit rather than accidental. Their fix is pip install -U zarr. Acceptable under the beta policy, surfaced and decided rather than defaulted into.
  2. FastFuels-Web docs: YES — outstanding. The zarr>=3 requirement should be stated wherever the zarr export format is documented. Tracked as the remaining checkbox on Pin the user-facing Zarr export to format v3 explicitly, and pin zarr>=3 in the SDK #447.
  3. Python SDK: YES — handled. zarr>=3 pin in the paired SDK PR. No API surface change, so no SDK method work; the SDK never imports zarr (the dep exists purely to provision the user's env for opening exports).
  4. Webapp API client: NO. The webapp reads grids through the API, not via zarr exports.

Testing

89 passed — full exporter unit suite (tests/, excluding integration).

Context

Surfaced while assessing #194 (closed as not planned). Zarr v3's required-fill_value rule is what made that issue's premise unreachable.

The zarr grid export was written with a bare `ds.to_zarr(zarr_dir)`, so its
on-disk format version was whatever zarr-python defaulted to rather than
something we chose. That artifact is zipped and handed to users via a signed
download URL, which makes the format version part of the export contract:
zarr-python 2.x cannot read format v3 at all.

This matters most for v1 migrants. v1 pinned zarr==2.18.2 and therefore handed
out format v2; v2 hands out v3. A user whose environment is still pinned to
zarr 2.x downloads a v2 export and it fails to open on their machine, with a
confusing error and nothing on our side indicating why.

Staying on v3 rather than falling back to the more widely-readable v2: zarr 3
has been out ~18 months, the client base is small, we are in beta where a
conscious break is acceptable, and one format across the codebase beats
carrying a v2 compat path. The remaining exposure is closed from the client
side by pinning zarr>=3 in the SDK.

Making it explicit also guards against a future zarr-python default flip
silently changing what users receive.

Internal grid stores (lib/zarr_utils.py) keep the implicit default: they are
read only by our own zarr-python 3.x services and are already 100% v3 on disk
across 400 production stores, with no mixed-format legacy.

Refs #447
@amarcozzi amarcozzi added 🐛 cold fix There's a bug, but it's not critical to fix right away 📦 Export Service Exporter service labels Jul 15, 2026
@amarcozzi amarcozzi self-assigned this Jul 15, 2026
@amarcozzi
amarcozzi merged commit 94ef2de into main Jul 27, 2026
1 check passed
@amarcozzi
amarcozzi deleted the 447-zarr-export-format-v3 branch July 27, 2026 20:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🐛 cold fix There's a bug, but it's not critical to fix right away 📦 Export Service Exporter service

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pin the user-facing Zarr export to format v3 explicitly, and pin zarr>=3 in the SDK

1 participant