Add indexed queries by nclack · Pull Request #161 · nclack/damacy

nclack · 2026-09-14T22:09:58Z

Add IndexQuery to CPU and CUDA pipelines, combining ordered index arrays with contiguous slices. Axes form a Cartesian product, preserving order, duplicates, and singleton dimensions. Rectangular and indexed requests can share a batch with the same output shape.

Stacked on #168; the base branch is fix/cpu-read-buffer-capacity. On this base the CPU executor keeps the merged reads from #164 and the input buffers from #168; index gathering happens during assembly, after decoding, so read planning is unchanged. #162 stacks on this branch.

query = damacy.IndexQuery(
    "image.zarr/0",
    selection=([7, 2, 7], slice(16, 80), [100, 3, 40, 3]),
)
assert query.shape == (3, 64, 4)

Preparation copies index vectors, enumerates only touched shards and chunks, and publishes owned gather plans. Index storage grows with vector lengths, without a planning record for every output voxel. CPU assembly and CUDA dispatch consume those same plans. CUDA keeps each chunk's selected range in a separate per-batch array that only indexed samples use, so rectangular chunk records keep their size. PlanLimits.max_plan_bytes includes owned index data.

CudaLimits.max_index_bytes sets device index storage within the total GPU budget. It must be zero or hold every index a batch can contain, 8 bytes × samples × the sum of the output shape, so an accepted query never runs out of room. A smaller nonzero value raises BudgetExceeded when the pipeline is created. With zero, push raises BudgetExceeded for an IndexQuery; the query is not consumed and the pipeline keeps running. Out-of-bounds coordinates are found when metadata arrives and raise InvalidArgument from pop, which stops the pipeline.

The C API uses damacy_sample.rank and tagged axes. Each active axis explicitly selects DAMACY_AXIS_INTERVAL with .interval or DAMACY_AXIS_INDICES with .indices. Zero and unknown tags, invalid intervals, and null or empty index arrays fail validation. Preparation derives the bounding AABB. damacy_tuning gains max_index_bytes; damacy_tuning_defaults sets it to 64 MiB, and a zeroed tuning rejects indexed samples. C callers must rebuild and migrate their query initializers and tuning setup. Negative indices are rejected at push and out-of-bounds coordinates at pop; interpolation and NGFF level selection remain separate future work.

CUDA decode-gap timing skips the first wave until a previous decode event has been recorded, avoiding an invalid event-handle call.

Validation:

CI at 61ce41c: the CPU workflow passes 25/25 CTest entries on Ubuntu and 26/26 on macOS; the macOS job passed on a rerun after the known test_scheduler timing failure. CUDA passes 37/37 with the coverage upload, TSan passes, and the build passes.
Ruff, Pyright, and the strict documentation build pass.
Before the review fixes, CPU-only: 24/24 CTest targets, including 70 Python tests.
Before the review fixes, AddressSanitizer/UndefinedBehaviorSanitizer: 5/5 targeted CTest targets covering selections, owned plans, lookahead, prefetch, and CPU execution.
Before the review fixes, CUDA on L40: all 36 CTest targets passed, including 197 Python tests.
Before the review fixes, CUDA compute-sanitizer: 35 query and malformed-input tests, zero memory or API errors.

closes #138

codecov · 2026-09-14T22:20:34Z

Codecov Report

❌ Patch coverage is 75.43860% with 140 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.40%. Comparing base (7ba9200) to head (b4bab59).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
python/damacy/_api.c	0.00%	60 Missing ⚠️
src/query/selection.c	86.95%	11 Missing and 13 partials ⚠️
src/render_job/render_job.c	63.63%	12 Missing and 4 partials ⚠️
src/executor/cpu_executor.c	77.27%	8 Missing and 2 partials ⚠️
src/planner/plan_builder.c	78.72%	4 Missing and 6 partials ⚠️
src/executor/dispatch.c	87.14%	4 Missing and 5 partials ⚠️
bench/main.c	0.00%	3 Missing ⚠️
python/damacy/__init__.py	94.54%	3 Missing ⚠️
src/wave/wave_pool.c	72.72%	3 Missing ⚠️
python/damacy/_components.c	0.00%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #161      +/-   ##
==========================================
+ Coverage   59.75%   60.40%   +0.64%     
==========================================
  Files          75       76       +1     
  Lines       11494    11963     +469     
  Branches     1987     2085      +98     
==========================================
+ Hits         6868     7226     +358     
- Misses       3812     3897      +85     
- Partials      814      840      +26

Flag	Coverage Δ
unittests	`60.40% <75.43%> (+0.64%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/damacy_config.c	`86.53% <100.00%> (ø)`
src/damacy_lifecycle.c	`75.67% <100.00%> (+0.13%)`	⬆️
src/damacy_push.c	`76.00% <100.00%> (+18.85%)`	⬆️
src/executor/cuda_executor.c	`77.61% <100.00%> (+0.68%)`	⬆️
src/executor/cuda_geometry.inc	`87.03% <100.00%> (-6.85%)`	⬇️
src/pipeline/zarr_planner.c	`79.31% <100.00%> (+3.19%)`	⬆️
src/planner/planner.c	`72.28% <100.00%> (ø)`
src/prefetch/prefetcher.c	`73.70% <100.00%> (-0.06%)`	⬇️
src/wave/wave_budget.c	`72.02% <100.00%> (-1.63%)`	⬇️
python/damacy/_components.c	`0.00% <0.00%> (ø)`
... and 10 more

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

CPU input buffers previously held one chunk per decode worker, limiting simultaneous file reads when each merged request contains several chunks. A new CPU executor setting, `chunks_per_input_buffer` (`CpuLimits.chunks_per_input_buffer` in Python, default 256), sets how many chunks each of the two input buffers holds. It must be at least `decode_workers`, so every worker can get a chunk, and at most 16384, the most chunks a batch can have. Values outside that range are rejected, with a message, when the executor is created. The worker pool and per-worker decoder workspaces stay the same. This restores simultaneous reads. Each merged read still holds at most `min(decode_workers, chunks_per_input_buffer)` chunks, the rule used in the measurements below, so a buffer holds several reads at once. With 32 workers, reads are at most 32 chunks, so a full 256-chunk buffer splits into at least 8 reads. With one decode worker, reads do not merge. A separate read-size setting is left for a later PR. This breaks existing budgets. The input reserve is `2 * chunks_per_input_buffer * max_encoded_chunk_bytes`, plus per-chunk bookkeeping. With the defaults, 256 chunks and a 4 MiB encoded bound, that is 2 GiB before decoder workspaces and output buffers, so a 1 GiB `CpuLimits` budget that worked before now fails with `BUDGET` when the pipeline starts. Set `max_encoded_chunk_bytes` to the largest encoded chunk expected, lower `chunks_per_input_buffer`, or raise `max_memory_bytes`. When the budget is too small, starting now logs the bytes it tried to reserve for input buffers, decoder workspaces, output buffers, and the rest; a read plan that cannot fit logs its size too. Documentation shows the cost and raises the example budget to 3 GiB. The native pipeline fixture now uses a 1 KiB encoded bound appropriate for its small test chunks, retaining its 32 MiB budget. The `throughput-cpu` bench scenario no longer fit its 4 GiB budget with the 4 MiB bound; it now sets `chunks_per_input_buffer` to 256 and a 5 GiB budget. The bench reads the setting from a scenario's `pipeline.chunks_per_input_buffer`, default 256. The motivating Turin experiment used 32 decode workers, 64 KiB chunks, 16 KiB Blosc blocks, and sixteen raw 1 GiB shards. In an isolated benchmark patch with persistent shard handles and default readahead, moving from 32 to 256 chunks improved cold translated 256×256 crop output by 47–55% and full-scan output by 71–97% across Blosc-LZ4 and Blosc-Zstd, adding 28.04 MiB of accounted memory at those 64 KiB chunk bounds, not the 2 GiB reserved with the default 4 MiB bound. These are benchmark-patch measurements; this PR does not include the persistent-handle or LZ4 additions and its final binary has not been benchmarked. Regression coverage checks 513 chunks through three input groups of exactly 256, 256, and 1 chunks at 1, 2, and 32 workers; merged reads of at most `decode_workers` chunks, with exact read ranges, shard order, and reads per buffer at 1, 2, and 32 workers; complete output; exact memory-budget admission; oversized-allocation rejection; and rejection of `chunks_per_input_buffer` below `decode_workers` or above 16384, in C and in Python. Private read-plan tests check complete coverage at 256, 257, and 1024 workers without starting hundreds of threads. Formatting and whitespace checks pass, and the strict documentation build passes locally. At `18cee45`, the [CPU workflow](https://github.com/nclack/damacy/actions/runs/36065751602) built without CUDA and passed 24/24 CTest entries on Ubuntu and 25/25 on macOS (the native test executables plus the Python pytest suite), followed by the native dependency audit; the macOS job passed on a rerun after the known `test_scheduler` timing failure. The [GPU-enabled regression workflow](https://github.com/nclack/damacy/actions/runs/36065753652) passed 36/36 CTest entries, the [TSan-labeled check](https://github.com/nclack/damacy/actions/runs/36065755729) passed 1/1, and the [build](https://github.com/nclack/damacy/actions/runs/36065749180) passed. Stacked on #164; the base is `fix/cpu-read-scheduling`. #161 stacks on this branch. --------- Co-authored-by: Nathan Clack <nclack@biohub.org>

Spatial queries can now resolve a fixed output grid against OME-Zarr 0.5 metadata before chunk planning. Resolution chooses a source level using the full affine sampling scale, including rotation and anisotropy. Aligned crops run on either CPU or CUDA; queries that still require resampling fail explicitly when submitted. This PR does not implement a resampler. Load an immutable `NgffImage` through an injected metadata reader, then call `image.resolve(query, shape=output.shape)`. Resolution performs no I/O and depends only on the requested sample shape. Batching, output allocation, and dtype conversion remain downstream. Each query carries its full-rank transform, sampler, boundary behavior, and automatic or explicit level choice. Results own their source URI and geometry independently of the image. Python returns one immutable, serializable value that can be pushed directly. C fills a caller-owned `damacy_spatial_resolution` with directly readable fields and a `clear()` operation; the existing sample conversion is the compatibility bridge to C submission. Public coordinates use reference-level voxel corners, with identity matching integer crops. NGFF center coordinates are converted at metadata loading. Loading also removes floating-point rounding error from each level's scale ratio and origin, so crops built from the reported level values are exact; the resolver itself never rounds a query into a crop. Validation checks consumed fields needed for array layout and coordinate interpretation, including ambiguous duplicate fields and finite transforms. Unused metadata, shared transforms that cancel in reference coordinates, and trailing content are left uninterpreted. Loading caps each `zarr.json` read at the part of `NgffLimits.max_metadata_bytes` still unused; a larger file fails with `DAMACY_BUDGET`. Whole-file metadata reads, capped or not, also fail that way above `UINT32_MAX` bytes. On macOS the metadata reader is the POSIX worker pool from #167, which had no capped read; this PR adds one there with the same behavior as Linux. See the [spatial query API and contracts](https://github.com/nclack/damacy/blob/ngff-resolution/docs/spatial.md) for C/Python examples, coordinate conversion, level selection, ownership, and the next executor operation. Stacked on #161 (`indexed-queries`). Validation: - CI at `382d921`: the [CPU workflow](https://github.com/nclack/damacy/actions/runs/36066539561) passes 26/26 CTest entries on Ubuntu and 27/27 on [macOS](https://github.com/nclack/damacy/actions/runs/36066539561/job/107857506786), where the metadata budget tests use the new capped read. [CUDA](https://github.com/nclack/damacy/actions/runs/36066541831) passes 38/38, including the native and Python spatial tests. [TSan](https://github.com/nclack/damacy/actions/runs/36066544271) passes 2/2, the prefetcher and spatial tests, and the [build](https://github.com/nclack/damacy/actions/runs/36066537440) passes. - Before the review fixes, CPU: 25/25 CTest targets passed; spatial Python tests: 62 passed, 7 CUDA cases skipped. - Before the review fixes, L40: native spatial tests and all 69 Python spatial tests passed; all 7 CUDA memory-check cases passed with zero sanitizer errors. - Before the review fixes, ASan/UBSan: JSON and spatial tests passed, including C result ownership and cleanup. - Ruff, Pyright for the package and spatial tests, strict documentation build, and C example syntax passed. - Regression coverage includes NGFF center/corner conversion, anisotropic level selection, interpolation bounds, permissive handling of unused metadata, invalid consumed fields, result serialization and ownership, fixed output shapes, 128 collapsed transforms, large differences in sampling scale, floating-point rounding in level scales and translations, unrelated keys that do not decode, and image comparison and copies. --------- Co-authored-by: Nathan Clack <nclack@biohub.org>

CPU input buffers previously held one chunk per decode worker, limiting simultaneous file reads when each merged request contains several chunks. A new CPU executor setting, `chunks_per_input_buffer` (`CpuLimits.chunks_per_input_buffer` in Python, default 256), sets how many chunks each of the two input buffers holds. It must be at least `decode_workers`, so every worker can get a chunk, and at most 16384, the most chunks a batch can have. Values outside that range are rejected, with a message, when the executor is created. The worker pool and per-worker decoder workspaces stay the same. This restores simultaneous reads. Each merged read still holds at most `min(decode_workers, chunks_per_input_buffer)` chunks, the rule used in the measurements below, so a buffer holds several reads at once. With 32 workers, reads are at most 32 chunks, so a full 256-chunk buffer splits into at least 8 reads. With one decode worker, reads do not merge. A separate read-size setting is left for a later PR. This breaks existing budgets. The input reserve is `2 * chunks_per_input_buffer * max_encoded_chunk_bytes`, plus per-chunk bookkeeping. With the defaults, 256 chunks and a 4 MiB encoded bound, that is 2 GiB before decoder workspaces and output buffers, so a 1 GiB `CpuLimits` budget that worked before now fails with `BUDGET` when the pipeline starts. Set `max_encoded_chunk_bytes` to the largest encoded chunk expected, lower `chunks_per_input_buffer`, or raise `max_memory_bytes`. When the budget is too small, starting now logs the bytes it tried to reserve for input buffers, decoder workspaces, output buffers, and the rest; a read plan that cannot fit logs its size too. Documentation shows the cost and raises the example budget to 3 GiB. The native pipeline fixture now uses a 1 KiB encoded bound appropriate for its small test chunks, retaining its 32 MiB budget. The `throughput-cpu` bench scenario no longer fit its 4 GiB budget with the 4 MiB bound; it now sets `chunks_per_input_buffer` to 256 and a 5 GiB budget. The bench reads the setting from a scenario's `pipeline.chunks_per_input_buffer`, default 256. The motivating Turin experiment used 32 decode workers, 64 KiB chunks, 16 KiB Blosc blocks, and sixteen raw 1 GiB shards. In an isolated benchmark patch with persistent shard handles and default readahead, moving from 32 to 256 chunks improved cold translated 256×256 crop output by 47–55% and full-scan output by 71–97% across Blosc-LZ4 and Blosc-Zstd, adding 28.04 MiB of accounted memory at those 64 KiB chunk bounds, not the 2 GiB reserved with the default 4 MiB bound. These are benchmark-patch measurements; this PR does not include the persistent-handle or LZ4 additions and its final binary has not been benchmarked. Regression coverage checks 513 chunks through three input groups of exactly 256, 256, and 1 chunks at 1, 2, and 32 workers; merged reads of at most `decode_workers` chunks, with exact read ranges, shard order, and reads per buffer at 1, 2, and 32 workers; complete output; exact memory-budget admission; oversized-allocation rejection; and rejection of `chunks_per_input_buffer` below `decode_workers` or above 16384, in C and in Python. Private read-plan tests check complete coverage at 256, 257, and 1024 workers without starting hundreds of threads. Formatting and whitespace checks pass, and the strict documentation build passes locally. At `18cee45`, the [CPU workflow](https://github.com/nclack/damacy/actions/runs/36065751602) built without CUDA and passed 24/24 CTest entries on Ubuntu and 25/25 on macOS (the native test executables plus the Python pytest suite), followed by the native dependency audit; the macOS job passed on a rerun after the known `test_scheduler` timing failure. The [GPU-enabled regression workflow](https://github.com/nclack/damacy/actions/runs/36065753652) passed 36/36 CTest entries, the [TSan-labeled check](https://github.com/nclack/damacy/actions/runs/36065755729) passed 1/1, and the [build](https://github.com/nclack/damacy/actions/runs/36065749180) passed. Stacked on #164; the base is `fix/cpu-read-scheduling`. #161 stacks on this branch. --------- Co-authored-by: Nathan Clack <nclack@biohub.org> 7ba9200

Spatial queries can now resolve a fixed output grid against OME-Zarr 0.5 metadata before chunk planning. Resolution chooses a source level using the full affine sampling scale, including rotation and anisotropy. Aligned crops run on either CPU or CUDA; queries that still require resampling fail explicitly when submitted. This PR does not implement a resampler. Load an immutable `NgffImage` through an injected metadata reader, then call `image.resolve(query, shape=output.shape)`. Resolution performs no I/O and depends only on the requested sample shape. Batching, output allocation, and dtype conversion remain downstream. Each query carries its full-rank transform, sampler, boundary behavior, and automatic or explicit level choice. Results own their source URI and geometry independently of the image. Python returns one immutable, serializable value that can be pushed directly. C fills a caller-owned `damacy_spatial_resolution` with directly readable fields and a `clear()` operation; the existing sample conversion is the compatibility bridge to C submission. Public coordinates use reference-level voxel corners, with identity matching integer crops. NGFF center coordinates are converted at metadata loading. Loading also removes floating-point rounding error from each level's scale ratio and origin, so crops built from the reported level values are exact; the resolver itself never rounds a query into a crop. Validation checks consumed fields needed for array layout and coordinate interpretation, including ambiguous duplicate fields and finite transforms. Unused metadata, shared transforms that cancel in reference coordinates, and trailing content are left uninterpreted. Loading caps each `zarr.json` read at the part of `NgffLimits.max_metadata_bytes` still unused; a larger file fails with `DAMACY_BUDGET`. Whole-file metadata reads, capped or not, also fail that way above `UINT32_MAX` bytes. On macOS the metadata reader is the POSIX worker pool from #167, which had no capped read; this PR adds one there with the same behavior as Linux. See the [spatial query API and contracts](https://github.com/nclack/damacy/blob/ngff-resolution/docs/spatial.md) for C/Python examples, coordinate conversion, level selection, ownership, and the next executor operation. Stacked on #161 (`indexed-queries`). Validation: - CI at `382d921`: the [CPU workflow](https://github.com/nclack/damacy/actions/runs/36066539561) passes 26/26 CTest entries on Ubuntu and 27/27 on [macOS](https://github.com/nclack/damacy/actions/runs/36066539561/job/107857506786), where the metadata budget tests use the new capped read. [CUDA](https://github.com/nclack/damacy/actions/runs/36066541831) passes 38/38, including the native and Python spatial tests. [TSan](https://github.com/nclack/damacy/actions/runs/36066544271) passes 2/2, the prefetcher and spatial tests, and the [build](https://github.com/nclack/damacy/actions/runs/36066537440) passes. - Before the review fixes, CPU: 25/25 CTest targets passed; spatial Python tests: 62 passed, 7 CUDA cases skipped. - Before the review fixes, L40: native spatial tests and all 69 Python spatial tests passed; all 7 CUDA memory-check cases passed with zero sanitizer errors. - Before the review fixes, ASan/UBSan: JSON and spatial tests passed, including C result ownership and cleanup. - Ruff, Pyright for the package and spatial tests, strict documentation build, and C example syntax passed. - Regression coverage includes NGFF center/corner conversion, anisotropic level selection, interpolation bounds, permissive handling of unused metadata, invalid consumed fields, result serialization and ownership, fixed output shapes, 128 collapsed transforms, large differences in sampling scale, floating-point rounding in level scales and translations, unrelated keys that do not decode, and image comparison and copies. --------- Co-authored-by: Nathan Clack <nclack@biohub.org> 6e87e2b

nclack marked this pull request as ready for review September 14, 2026 22:21

nclack mentioned this pull request Sep 15, 2026

Resolve spatial queries from NGFF metadata #162

Merged

nclack force-pushed the indexed-queries branch from 029677b to 61ce41c Compare September 24, 2026 22:38

nclack changed the base branch from cpu-pipeline to fix/cpu-read-buffer-capacity September 24, 2026 22:38

This was referenced Sep 24, 2026

Add CPU decoding and pipeline components #160

Merged

Make CPU input buffer size a setting, default 256 chunks #168

Merged

nclack added this pull request to stack #170 September 24, 2026 23:07

nclack force-pushed the indexed-queries branch from 61ce41c to f86232c Compare September 24, 2026 23:13

Base automatically changed from fix/cpu-read-buffer-capacity to main September 24, 2026 23:34

Nathan Clack added 9 commits September 24, 2026 16:34

Add indexed queries

73108db

Skip unrecorded decode timing

4b1584e

Use tagged query axes

ed6db97

Clarify selection iterator units

630db46

cpu: copy gathered rows as bytes

482efa9

cuda: keep gather spans out of chunks

43db8b0

cuda: check index capacity up front

e434722

clean up indexed query leftovers

896b862

docs: mention CPU index gathering

b4bab59

nclack force-pushed the indexed-queries branch from f86232c to b4bab59 Compare September 24, 2026 23:34

nclack merged commit 7908022 into main Sep 24, 2026
5 checks passed

nclack deleted the indexed-queries branch September 24, 2026 23:34

Navigation Menu

Add indexed queries - #161

Add indexed queries#161
nclack merged 9 commits into
mainfrom
indexed-queries

nclack commented Sep 14, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

nclack commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

nclack commented Sep 14, 2026 •

edited

Loading

codecov Bot commented Sep 14, 2026 •

edited

Loading