Skip to content

dev: data reformatting plan (draft) - #158

Closed
nclack wants to merge 1 commit into
mainfrom
docs-reformatting-spec
Closed

nclack wants to merge 1 commit into
mainfrom
docs-reformatting-spec

Conversation

@nclack

@nclack nclack commented Jun 13, 2026

Copy link
Copy Markdown
Owner

Adds a draft data-reformatting plan under dev/ (design/plan docs, outside the
mkdocs site), the actionable follow-on to the optimization report (docs/optimization.md).

For the ~9 decode-bound training datasets, it gives a target chunk/shard shape per
archetype (static 2D / video / 3D z-range) and the predicted decode waste after
re-chunking, with before→after computed from the real array+patch shapes:

  • allencell-nuclear 131x -> 1.9x, allencell-static 128x -> 2.0x (target [1,1,1,256,256])
  • the video MIPs (dynacell/jacobo/nucmorph/emt) 22-33x -> 2-4x
  • raw 3D variants: their 76-163x is a 2D-crop-on-3D-data artifact; read with a
    z-range crop they're ~5-10x, and a small z-extent chunk takes them to ~2-5x

Principle: chunks small (64 KiB-1 MiB, smaller than the crop), with extent > 1 only
in dimensions that compress well together (x/y always, z in 3D, time sometimes,
channels split). Includes sharding guidance and verify-before-committing caveats.

Draft — predictions are from geometry; confirm one reformatted dataset with a real
bench run before doing the corpus.

@codecov

codecov Bot commented Jun 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.71%. Comparing base (cca3b09) to head (db1675f).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #158      +/-   ##
==========================================
- Coverage   57.78%   57.71%   -0.08%     
==========================================
  Files          64       64              
  Lines       10046    10055       +9     
  Branches     1748     1750       +2     
==========================================
- Hits         5805     5803       -2     
- Misses       3494     3501       +7     
- Partials      747      751       +4     
Flag Coverage Δ
unittests 57.71% <ø> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nclack
nclack force-pushed the docs-reformatting-spec branch from 27187f3 to db1675f Compare June 14, 2026 02:36
@nclack

nclack commented Jun 14, 2026

Copy link
Copy Markdown
Owner Author

Moved to docs/ in nclack/damacy-slides-2026-06; keeping these out of this repo.

@nclack nclack closed this Jun 14, 2026
@nclack
nclack deleted the docs-reformatting-spec branch June 14, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant