Add CPU decoding and pipeline components

[nclack]

Damacy currently requires CUDA even when the consumer needs a tensor in host memory. This change adds a CPU executor and explicit pipeline components so existing rectangular queries share metadata loading and chunk planning, then execute on either CPU or CUDA.

Benchmark uncertainty: median CUDA throughput is 5.8% below the baseline in three paired L40 runs (12.409 to 11.684 GB/s). Baseline runs themselves span about 6%, and shared NFS traffic was not controlled. This is an inconclusive throughput difference; these measurements do not establish a code regression or performance parity.

Behavior change: a DLPack view now keeps its output buffer until the view is destroyed, even after its Batch is released. Previously the buffer returned to the pool while views could still see it, so a kept tensor was silently overwritten by a later batch. Code that keeps views from two batches now raises PoolStarved; use .clone() (PyTorch) or .copy() (NumPy) to keep data.

Implementation

• Add C and Python factories for metadata readers/providers, chunk planners, bulk readers, executors, output specifications, and queue limits. Keep Pipeline(Config(...)) and damacy_create as CUDA construction adapters.
• Separate owned source/result plans from metadata-dependent construction. Executors consume plans without metadata-cache handles or CUDA details in shared planning; CUDA dispatch, staging, and streams belong to the CUDA executor.
• Decode raw, zstd, and Blosc-zstd chunks into RAM with bounded CPU workers and buffers, reuse distinct decoded chunks within each batch, and assemble f32 or bf16 results.
• Make result ownership independent of pipeline shutdown, including NumPy/PyTorch DLPack views. Cover component reuse, backpressure, FIFO ordering, and blocked-pop shutdown.
• Add the CPU CMake preset, dependency updates, a CPU CI workflow, and benchmark support. Fix the CUDA decoder's handling of uncompressed Blosc payloads exposed by independent-value tests.
• Launch the TSan test process with a fixed address layout and use string Blosc settings in the shared fixture writer for compatibility with current Zarr releases.

Index queries and transformed crops remain future work. The architecture document describes ordered per-dimension indices, NGFF level selection during query resolution, and sampler/interpolation/boundary settings carried by spatial queries.

Validation

• CPU: 23/23 CTest targets pass; Python reports 22 passed and 93 skipped CUDA or optional-PyTorch cases.
• CUDA on L40: 35/35 CTest targets pass, including all 115 Python tests with PyTorch.
• ASan/UBSan: all six selected metadata, planning, scheduling, and CPU pipeline targets pass.
• Installed CPU wheel: real crop decoding and a NumPy view retained after shutdown pass, with no CUDA, cudart, or nvCOMP dependencies or loaded libraries.
• Four parser fuzzers pass 100-run smoke checks. Changed Python passes Ruff; the public Python API and new component tests pass Pyright. Strict documentation build and diff/format checks pass.
• Shared fixture checks: all three Blosc shuffle modes and the affected chunk-layout, prefetcher, and CPU pipeline tests pass with Python 3.11/Zarr 3.1.6 and Python 3.12/Zarr 3.3.0.
• TSan CI: the instrumented prefetcher concurrency test passes.

Measurements and limits

The throughput workload produces 1 GiB of f32 crops per batch from sharded zstd arrays. With 16 decode workers, CPU useful-output throughput reaches 4.104 GB/s with 2.019 GiB peak RAM. CPU measurements use shared filesystem caches and one run per worker count; they are not cold-storage or training-loop measurements. The executor memory limit covers its buffers and codec workspace, not total process RSS.

The CUDA comparison alternates three baseline/refactor pairs on one L40 with matching compilers, flags, context setup, and input seeds. GDS and NUMA placement were unavailable on the test host and remain unvalidated here.

API examples and lifetime contracts · Architecture and future queries · Complete validation and measurements

[codecov]

Codecov Report

❌ Patch coverage is 68.16275% with 759 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.09%. Comparing base (dee3e66) to head (5126f09).

Files with missing lines	Patch %	Lines
python/damacy/_components.c	0.00%	145 Missing ⚠️
src/executor/cuda_executor.c	76.92%	47 Missing and 43 partials ⚠️
bench/main.c	0.00%	84 Missing ⚠️
src/executor/cpu_executor.c	81.75%	43 Missing and 30 partials ⚠️
src/planner/plan_builder.c	70.53%	38 Missing and 28 partials ⚠️
python/damacy/_api.c	0.00%	53 Missing ⚠️
src/pipeline/zarr_planner.c	74.22%	30 Missing and 20 partials ⚠️
src/pipeline/components.c	64.44%	25 Missing and 23 partials ⚠️
src/damacy_lifecycle.c	76.92%	19 Missing and 20 partials ⚠️
src/planner/planner.c	68.91%	13 Missing and 10 partials ⚠️
... and 12 more

❌ Your patch check has failed because the patch coverage (68.16%) is below the target coverage (70.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #160      +/-   ##
==========================================
+ Coverage   57.71%   59.09%   +1.37%     
==========================================
  Files          64       74      +10     
  Lines       10104    11386    +1282     
  Branches     1754     1970     +216     
==========================================
+ Hits         5832     6729     +897     
- Misses       3520     3828     +308     
- Partials      752      829      +77     

Flag	Coverage Δ
unittests	59.09% <68.16%> (+1.37%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/batch_pool/batch_pool.c	62.80% <ø> (-16.19%)	⬇️
src/damacy_plan.c	100.00% <100.00%> (+21.21%)	⬆️
src/executor/coalesce.c	90.29% <100.00%> (ø)
src/executor/group_chunks.c	86.79% <100.00%> (ø)
src/executor/read_op_sort.c	92.42% <ø> (ø)
src/numa/numa.c	12.28% <ø> (-4.80%)	⬇️
src/planner/plan.c	100.00% <100.00%> (ø)
src/threadpool/threadpool.c	58.64% <100.00%> (+58.64%)	⬆️
src/wave/wave_pool.c	79.90% <ø> (-0.82%)	⬇️
src/scheduler/scheduler.c	76.92% <94.73%> (+1.92%)	⬆️
... and 21 more

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.