Repository navigation
Map packed input into edge chunks with scatter lookup tables - #295
Merged
Merged
Conversation
nclack
changed the title
nclack
marked this pull request as ready for review
October 4, 2026 18:14
Compile column spans and joined rows from the lookup tables once per array. Use those boundaries for bulk copies and tail clearing while retaining typed scatter for short spans. Reuse a single column boundary when padded rows cannot join, and include a conservative cache allowance in memory estimates. Cover partial appends, worker splits, row padding, ragged multidimensional spans, cache replacement, and multiarray switches with independent readback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Packed 9×13 frames were indexed as padded 16×16 frames, corrupting pixels and publishing one frame after three appends. Epoch boundaries, input limits, and final shapes now count logical elements; chunk edges and final LOD tails remain zero.
Single-level edge chunks use precomputed 64-bit row/column offsets through the same lookup scatter as LOD. The mapping handles padding without an extra frame buffer. CPU scatter caches contiguous column spans and joined rows once per array, then copies them in bulk; short runs retain typed scatter. CPU memory estimates include a conservative allowance for cached boundaries. Chunk-aligned input retains the existing transpose path.
Image replay submits packed frames; report readers still accept archived caller-padded results.
Validation: 57 CPU CTest checks including independent Zarr readback, six focused CPU/CUDA checks in the CUDA build, and three ASan/UBSan checks. Regressions cover split appends, worker boundaries, row padding, ragged multidimensional spans, storage permutations, rank-10 shapes, wide offsets, memory estimates, and multiarray switches. The two S3-service tests were excluded because no local S3 service was running.
Fixes #294.