Skip to content

Restore Blosc-LZ4 on CPU and GPU #166

Open

Description

@nclack

Reading a Zarr v3 array encoded with Blosc-LZ4, including bitshuffled data, currently fails with DAMACY_DECODE. Restore support on both the GPU backend and the CPU backend introduced in #160 so these arrays work without a local patch.

This was previously implemented: #4 added GPU Blosc-LZ4 using nvCOMP, with byte/bit shuffle support. #36 deliberately removed it during decompression simplification. The removal plan cites large worst-case LZ4 allocations and concerns about running multiple nvCOMP decoder instances simultaneously. Restoration should preserve the current memory budgets and avoid restoring those unconditional worst-case allocations.

The present behavior is explicit rejection, rather than a missing metadata name:

  • Metadata parsing recognizes both lz4 and lz4hc as CODEC_BLOSC_LZ4.
  • The GPU planner rejects that codec; the GPU decoder, chunk-layout handling, and decode dispatch currently only support the Zstd Blosc path. Planner tests explicitly expect LZ4/LZ4HC rejection.
  • The CPU executor also only accepts Blosc-Zstd. Its existing C-Blosc call can be extended to LZ4 with matching codec validation and submission checks.

Requested behavior and verification:

  • Read Blosc lz4 and lz4hc arrays on CPU and GPU, including bitshuffle, byte shuffle, and no shuffle. Replace the intentional rejection tests with positive coverage.
  • On CPU, retain configured decode-worker counts and one internal C-Blosc thread per decode, including CPU-only builds without CUDA.
  • On GPU, restore LZ4 block decoding and assembly within the current stream and memory-budget design. Handle valid split/unsplit blocks, partial final blocks, and Blosc's uncompressed (MEMCPYED) form; retain validation of malformed or truncated input.
  • Verify exact outputs for sharded and unsharded arrays, translated crops, and mixed supported codecs. Include CPU and GPU end-to-end coverage and update the documented codec support and benchmark fixtures.

The Turin comparison for #160 already uses a small local CPU-only LZ4 enablement patch. It adds CODEC_BLOSC_LZ4 to decode_chunk and cpu_submit, checks BLOSC_LZ4_LIBNAME for compressed chunks, and expands the fixture writer and CPU codec tests. That patch is not part of #160 or #164 and does not enable GPU LZ4. It is saved at ~/tmp/2026-09-16-damacy-pr160-pareto/pr160-lz4.patch and can serve as a starting point for the CPU portion.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions