Wit's storage design assumes source audio dedups near-perfectly across projects, but that has only been verified for exact whole-file duplicates.
Measured so far (docs/EXPERIMENTS.md §9): across 30 real Logic projects totalling 26 GB, 882 audio files / 21.95 GB, 5.38 GB (24.5%) are byte-for-byte duplicates.
That used size-bucketing followed by whole-file hashing. It cannot see partial reuse — a trimmed sample, a file re-exported at a different length, or the same recording with a different header.
The task
Extend the analysis to sub-file, content-defined chunking using experiments/cdc_dedup.py, and report:
- Whole-file dedup vs CDC dedup on the same library — how much extra does chunking find?
- How the answer varies with average chunk size (try 8 KB, 64 KB, 256 KB). The current 64 KB target is inherited from other systems and has never been tuned on music.
- Whether the win justifies the added complexity, or whether whole-file hashing captures nearly all of it.
Why this matters
It directly determines part of ADR-0002. If CDC finds little beyond whole-file dedup on real libraries, the audio path gets meaningfully simpler.
Known bounds — on a single 19.6 MB stem, CDC gave 99.6% reuse on a time-shift, 93.0% on a localized re-render, and 0.00% on a global re-render. The open question is what the mix looks like across a real library.
Note the harness is pure Python at ~1–2 MB/s; budget time, or sample the library. Report your material context (how many projects, total size, genre).
Wit's storage design assumes source audio dedups near-perfectly across projects, but that has only been verified for exact whole-file duplicates.
Measured so far (
docs/EXPERIMENTS.md§9): across 30 real Logic projects totalling 26 GB, 882 audio files / 21.95 GB, 5.38 GB (24.5%) are byte-for-byte duplicates.That used size-bucketing followed by whole-file hashing. It cannot see partial reuse — a trimmed sample, a file re-exported at a different length, or the same recording with a different header.
The task
Extend the analysis to sub-file, content-defined chunking using
experiments/cdc_dedup.py, and report:Why this matters
It directly determines part of ADR-0002. If CDC finds little beyond whole-file dedup on real libraries, the audio path gets meaningfully simpler.
Known bounds — on a single 19.6 MB stem, CDC gave 99.6% reuse on a time-shift, 93.0% on a localized re-render, and 0.00% on a global re-render. The open question is what the mix looks like across a real library.
Note the harness is pure Python at ~1–2 MB/s; budget time, or sample the library. Report your material context (how many projects, total size, genre).