bench: per-wave shard/read diagnostics - #159
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
❌ Your patch check has failed because the patch coverage (51.92%) is below the target coverage (70.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #159 +/- ##
==========================================
- Coverage 57.78% 57.68% -0.11%
==========================================
Files 64 64
Lines 10046 10104 +58
Branches 1748 1754 +6
==========================================
+ Hits 5805 5828 +23
- Misses 3494 3523 +29
- Partials 747 753 +6
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
github-actions Bot
added a commit
that referenced
this pull request
Jun 15, 2026
Per-wave dispatch instrumentation, built to investigate #154 (does read/decode parallelism track array/shard count or the io pool?). Generally useful for any read-width / wave-shape diagnosis, so proposing it for `main`. ## What it adds - **`damacy_stats` counters** (emitted in bench `results.json`): `wave_reads_sum`, `wave_distinct_shards_sum`, and `wave_stop_{drained,host,chunks,dev}`. Means are the sums over `waves_emitted` — `shards/wave` is the io width the coalescer's round-robin actually achieves; the `stop_*` split says whether waves cap on a budget (more work waiting) or drain (no more work). - **`wave_desc.stop_reason`** + `enum wave_stop_reason`, set at each break in `wave_dispatcher_reserve`. - **`DAMACY_TRACE_WAVES=<file>`** (opt-in): one line per reserved wave — `batch_id render_job_idx n_reads distinct_shards n_chunks input_bytes stop_reason`. Mirrors the existing `DAMACY_TRACE_READS` pattern. - **`bench/sweep.py`**: `reads/wave`, `shards/wave`, `stop d/h/c/v` columns. - **`bench/scenarios/array-starve.json`**: an `n_zarrs` sweep scenario (large shards) for `sweep.py --param dataset.n_zarrs`. Always-on cost is one O(n) pass plus a small distinct-shard scan per wave (n_reads is post-coalesce, typically 100–250); negligible vs io/decode. The trace is opt-in. ## What it found (full writeup on #154) Using these counters, io throughput scales cleanly with `shards/wave` — 6→1.09, 37→2.37, 236→4.32 GB/s — confirming a real read-width effect on few-shard sets. But it's mild: damacy stays io-bound and **beats tensorstore on every measured dataset, including a 1-array one**. The catastrophic losses that motivated #154 were decode amplification, not array count. So #154 wants re-scoping to "io-pool under-utilization on few-shard datasets" rather than a competitiveness bug. ## Not included `bench/tensorstore_bench.py` (used for the comparison) lives on its own branch `bench-tensorstore-comparison` (#153) and is intentionally not duplicated here. ctest 33/33 on L40. Co-authored-by: Nathan Clack <nclack@biohub.org> 7d860ac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per-wave dispatch instrumentation, built to investigate #154 (does read/decode
parallelism track array/shard count or the io pool?). Generally useful for any
read-width / wave-shape diagnosis, so proposing it for
main.What it adds
damacy_statscounters (emitted in benchresults.json):wave_reads_sum,wave_distinct_shards_sum, andwave_stop_{drained,host,chunks,dev}. Means arethe sums over
waves_emitted—shards/waveis the io width the coalescer'sround-robin actually achieves; the
stop_*split says whether waves cap on abudget (more work waiting) or drain (no more work).
wave_desc.stop_reason+enum wave_stop_reason, set at each break inwave_dispatcher_reserve.DAMACY_TRACE_WAVES=<file>(opt-in): one line per reserved wave —batch_id render_job_idx n_reads distinct_shards n_chunks input_bytes stop_reason.Mirrors the existing
DAMACY_TRACE_READSpattern.bench/sweep.py:reads/wave,shards/wave,stop d/h/c/vcolumns.bench/scenarios/array-starve.json: ann_zarrssweep scenario (largeshards) for
sweep.py --param dataset.n_zarrs.Always-on cost is one O(n) pass plus a small distinct-shard scan per wave
(n_reads is post-coalesce, typically 100–250); negligible vs io/decode. The trace
is opt-in.
What it found (full writeup on #154)
Using these counters, io throughput scales cleanly with
shards/wave— 6→1.09,37→2.37, 236→4.32 GB/s — confirming a real read-width effect on few-shard sets.
But it's mild: damacy stays io-bound and beats tensorstore on every measured
dataset, including a 1-array one. The catastrophic losses that motivated #154
were decode amplification, not array count. So #154 wants re-scoping to
"io-pool under-utilization on few-shard datasets" rather than a competitiveness bug.
Not included
bench/tensorstore_bench.py(used for the comparison) lives on its own branchbench-tensorstore-comparison(#153) and is intentionally not duplicated here.ctest 33/33 on L40.