Pin the Zarr export to format v3 explicitly - #448
Merged
Conversation
The zarr grid export was written with a bare `ds.to_zarr(zarr_dir)`, so its on-disk format version was whatever zarr-python defaulted to rather than something we chose. That artifact is zipped and handed to users via a signed download URL, which makes the format version part of the export contract: zarr-python 2.x cannot read format v3 at all. This matters most for v1 migrants. v1 pinned zarr==2.18.2 and therefore handed out format v2; v2 hands out v3. A user whose environment is still pinned to zarr 2.x downloads a v2 export and it fails to open on their machine, with a confusing error and nothing on our side indicating why. Staying on v3 rather than falling back to the more widely-readable v2: zarr 3 has been out ~18 months, the client base is small, we are in beta where a conscious break is acceptable, and one format across the codebase beats carrying a v2 compat path. The remaining exposure is closed from the client side by pinning zarr>=3 in the SDK. Making it explicit also guards against a future zarr-python default flip silently changing what users receive. Internal grid stores (lib/zarr_utils.py) keep the implicit default: they are read only by our own zarr-python 3.x services and are already 100% v3 on disk across 400 production stores, with no mixed-format legacy. Refs #447
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #447. Paired with silvxlabs/fastfuels-sdk-python#189 (the client-side half).
The zarr grid export was written with a bare
ds.to_zarr(zarr_dir), so its on-disk format version was whatever zarr-python defaulted to rather than something we chose. That artifact is zipped and handed to users via a signed download URL, which makes the format version part of the export contract — zarr-python 2.x cannot read format v3 at all.Why now
A v1 migrant whose environment is still pinned to zarr 2.x downloads a v2 export and it fails to open on their machine, with a confusing error and nothing on our side indicating why.
We're staying on v3 rather than falling back to the more widely-readable v2 (zarr-python 3.x reads v2 fine, so v2 would technically be the more compatible choice for a static download). Reasoning: zarr 3 has been out ~18 months, the client base is small, we're in beta where a conscious break is acceptable, and one format across the codebase beats carrying a v2 compat path. The client-side exposure is closed by pinning
zarr>=3in the SDK.Making the format explicit also guards against a future zarr-python default flip silently changing what users receive.
Changes
zarr_format=3explicit inexporter/handlers/grid.py, with a comment on why the version is a contract and not a library default.tests/handlers/test_zarr.py: mocks the GCS upload, captures the zip, extracts, and assertszarr_format == 3on both the group and the array. Verified it has teeth rather than rubber-stamping the current default — flipping the kwarg to2fails it withassert 2 == 3.Not included, deliberately
Internal grid stores (
lib/zarr_utils.py:110) also write with the implicit default. Left alone: they're read only by our own zarr-python 3.x services and are already 100% v3 on disk (verified across 400 production stores, zero mixed-format legacy). No user tooling touches them.Contract review
main; this PR makes the choice explicit rather than accidental. Their fix ispip install -U zarr. Acceptable under the beta policy, surfaced and decided rather than defaulted into.zarr>=3requirement should be stated wherever the zarr export format is documented. Tracked as the remaining checkbox on Pin the user-facing Zarr export to format v3 explicitly, and pin zarr>=3 in the SDK #447.zarr>=3pin in the paired SDK PR. No API surface change, so no SDK method work; the SDK never imports zarr (the dep exists purely to provision the user's env for opening exports).Testing
89 passed— full exporter unit suite (tests/, excluding integration).Context
Surfaced while assessing #194 (closed as not planned). Zarr v3's required-
fill_valuerule is what made that issue's premise unreachable.