Repository navigation
Restore Blosc-LZ4 on CPU and GPU #166
Description
Reading a Zarr v3 array encoded with Blosc-LZ4, including bitshuffled data, currently fails with DAMACY_DECODE. Restore support on both the GPU backend and the CPU backend introduced in #160 so these arrays work without a local patch.
This was previously implemented: #4 added GPU Blosc-LZ4 using nvCOMP, with byte/bit shuffle support. #36 deliberately removed it during decompression simplification. The removal plan cites large worst-case LZ4 allocations and concerns about running multiple nvCOMP decoder instances simultaneously. Restoration should preserve the current memory budgets and avoid restoring those unconditional worst-case allocations.
The present behavior is explicit rejection, rather than a missing metadata name:
- Metadata parsing recognizes both
lz4andlz4hcasCODEC_BLOSC_LZ4. - The GPU planner rejects that codec; the GPU decoder, chunk-layout handling, and decode dispatch currently only support the Zstd Blosc path. Planner tests explicitly expect LZ4/LZ4HC rejection.
- The CPU executor also only accepts Blosc-Zstd. Its existing C-Blosc call can be extended to LZ4 with matching codec validation and submission checks.
Requested behavior and verification:
- Read Blosc
lz4andlz4hcarrays on CPU and GPU, including bitshuffle, byte shuffle, and no shuffle. Replace the intentional rejection tests with positive coverage. - On CPU, retain configured decode-worker counts and one internal C-Blosc thread per decode, including CPU-only builds without CUDA.
- On GPU, restore LZ4 block decoding and assembly within the current stream and memory-budget design. Handle valid split/unsplit blocks, partial final blocks, and Blosc's uncompressed (
MEMCPYED) form; retain validation of malformed or truncated input. - Verify exact outputs for sharded and unsharded arrays, translated crops, and mixed supported codecs. Include CPU and GPU end-to-end coverage and update the documented codec support and benchmark fixtures.
The Turin comparison for #160 already uses a small local CPU-only LZ4 enablement patch. It adds CODEC_BLOSC_LZ4 to decode_chunk and cpu_submit, checks BLOSC_LZ4_LIBNAME for compressed chunks, and expands the fixture writer and CPU codec tests. That patch is not part of #160 or #164 and does not enable GPU LZ4. It is saved at ~/tmp/2026-09-16-damacy-pr160-pareto/pr160-lz4.patch and can serve as a starting point for the CPU portion.
Activity