Skip to content

docs: optimization & data layout report - #157

Closed
nclack wants to merge 1 commit into
mainfrom
docs-optimization-report
Closed

nclack wants to merge 1 commit into
mainfrom
docs-optimization-report

Conversation

@nclack

@nclack nclack commented Jun 13, 2026

Copy link
Copy Markdown
Owner

Adds a docs page summarizing the data-loading optimization work: the pipeline
stages and how they work, the storage/read levers and the measurements behind
them, the corpus-wide finding that most datasets are limited by decompression
(not reading) due to chunk layout, the dataset-format categories, and a
damacy-vs-tensorstore comparison.

  • New page docs/optimization.md, linked from the mkdocs nav and the index page.
  • Cluster-specific absolute paths scrubbed; references point at the in-repo bench
    harness (bench/run.py, bench/report.py, bench/tensorstore_bench.py).
  • Numbers are from real benchmark runs (L40, fast networked storage).

@codecov

codecov Bot commented Jun 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.71%. Comparing base (cca3b09) to head (c307cfa).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #157      +/-   ##
==========================================
- Coverage   57.78%   57.71%   -0.08%     
==========================================
  Files          64       64              
  Lines       10046    10055       +9     
  Branches     1748     1750       +2     
==========================================
- Hits         5805     5803       -2     
- Misses       3494     3501       +7     
- Partials      747      751       +4     
Flag Coverage Δ
unittests 57.71% <ø> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@nclack
nclack force-pushed the docs-optimization-report branch from 4033cc8 to 41375d3 Compare June 13, 2026 15:53
@nclack
nclack force-pushed the docs-optimization-report branch from 41375d3 to c307cfa Compare June 13, 2026 20:57
@nclack

nclack commented Jun 14, 2026

Copy link
Copy Markdown
Owner Author

Moved to docs/ in nclack/damacy-slides-2026-06; keeping these out of this repo.

@nclack nclack closed this Jun 14, 2026
@nclack
nclack deleted the docs-optimization-report branch June 14, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant