BENCH V2 — Lossless codec benchmark

Metric = gzip(minimal restore bundle), verified round-trip

BENCH V2 is a small, strict benchmark to compare portable, lossless encoders (tokenizers / dictionary codecs) on enwik6/7/8.

Hard rules:

• Lossless only: every method must reconstruct the original bytes exactly.
• Fair size metric: measure gzip of the minimal restore bundle.
• Reproducible: commands, environment, and numbers are pinned.

If a method is larger than gzip(raw), it is not beating “just compress the data”.

---

Repo layout

.
├── data/
│   ├── enwik6
│   ├── enwik7
│   └── enwik8
├── out_v2/
│   ├── bench_v2.csv
│   ├── enwik6/
│   ├── enwik7/
│   └── enwik8/
├── big_bench.py
├── verify.py
├── repro.md
├── requirements.txt
├── uname.txt
├── python_version.txt
├── test_redumb_cycle.sh
├── sizes_gzip.sh
└── redumb_output/
├── dicts/
├── encodings/
├── sdicts/        # encode-helper only (NOT needed for restore)
└── reconstructed.txt

Notes:

• .venv/ is git-ignored.
• Artifacts under out_v2/** can be large.

---

Environment

Captured verbatim in:

• uname.txt (OS / container)
• python_version.txt (Python 3.13.9)
• requirements.txt

---

Methods (current)

1) tiktoken_remap (Python)

Artifacts (portable, join-bytes):

• model.bin (token bytes via decode_bytes)
• tokens.u32 (uint32 ids)

2) dumb_v1_raw_textfiles (Python)

Artifacts (native):

• dict.txt
• output.txt

3) redumb_chunked (Rust)

Minimal restore bundle:

• redumb_output/dicts/
• redumb_output/encodings/

(sdicts/ is encode-helper only and excluded from size.)

---

Verification (lossless)

python verify.py out_v2/enwik8/tiktoken_remap --orig data/enwik8
python verify.py out_v2/enwik8/dumb_v1_raw_textfiles --orig data/enwik8
./test_redumb_cycle.sh

All methods above reconstruct the original exactly.

---

Size metric (gzip bundle)

Rule:

• Measure gzip of the minimal restore artifacts only.
• For multi-file bundles, concatenation order must be deterministic.

Example (REDUMB, stable order):

LC_ALL=C (
  find ./redumb_output/dicts -type f -print0     | sort -z | xargs -0 cat
  find ./redumb_output/encodings -type f -print0 | sort -z | xargs -0 cat
) | gzip -c -9 | wc -c

---

Raw gzip baseline (input only)

This is the reference floor: compress the original dataset directly. No model. No tokens. Lossless by definition.

Dataset	Raw size (bytes)	gzip lvl9 (bytes)	gzip lvl6 (bytes)
enwik6	1,000,000	355,357	356,264
enwik7	10,000,000	3,685,296	3,693,807
enwik8	100,000,000	36,445,248	36,518,329

Any method larger than gzip(raw) is not beating “just compress the data”.

---

Results (enwik8)

Final size comparison (gzip lvl9, sorted by size):

Rank	Method	Impl	Bundle	Size (bytes)
1	redumb_chunked	Rust	dicts + encodings	35,568,418
2	dumb_v1_raw_textfiles	Python	dict.txt + output.txt	35,568,753
3	gzip_raw	tool	data/enwik8	36,445,248
4	tiktoken_remap	Python	model.bin + tokens.u32	36,696,040
5	dumb_v1_portable_bin_text	Python	model.bin (MB01, text semantics) + tokens.u32	36,873,618
6	tiktoken_remap_textbundle	Python	dict.txt + output.txt	37,151,990

---

Takeaways

• Best overall: redumb_chunked (Rust) — smallest bundle.
• Best Python: dumb_v1_raw_textfiles, essentially tied with redumb_chunked (Δ ≈ 335 bytes).
• Both DUMB/REDUMB beat gzip(raw) by ~876 KB.
• tiktoken_remap (portable binary) is +250,792 bytes over gzip(raw).
• Forcing tiktoken into a text container (tiktoken_remap_textbundle) makes it even worse: +706,742 bytes vs gzip(raw).
• dumb_v1_portable_bin_text shows the portable binary penalty for text-semantic codecs: +428,370 bytes vs gzip(raw).

Notes

• *_textbundle rows compare methods under the same gzip-friendly text container (dict.txt + output.txt).
• dumb_v1_portable_bin_text is lossless under DUMB text semantics (placeholder restore → UTF-8 text).
• Key result: even when normalized to the same container, tiktoken loses.

---

Reproducibility

Everything required to reproduce numbers exactly:

• repro.md (paths, commands, measurements)
• bench_v2.csv (machine-readable results)

---

Sentence piece addendum

— We benchmarked SentencePiece on enwik8 under a strict, byte-lossless setup (via explicit byte→Unicode mapping, encode/decode, and exact byte verification) to compare it fairly against gzip, DUMB, and tiktoken-style methods. In practice, SentencePiece does not compete as a compressor pre stage: Unigram is unstable on large byte-mapped corpora, while BPE is lossless but produces larger outputs than gzip(raw) (≈ +2.4 MB at vocab = 8192) and is far behind dumb_v1 / redumb_chunked, with much higher training cost. Conclusion: SentencePiece is a tokenizer, not a compression baseline, and is excluded from further compression-focused results.

Performance Notes (SP BPE @ 8192)

Metric	Value
Train time	~650 s
Encode time	~31.5 s
Decode time	~8.3 s
Vocab size	8192
Token count	~111M ids
Lossless	✅

| sp_bpe_bytesafe_8192 | SentencePiece | sp.model + output.ids | 38,899,227 | ✅ |

Next steps

• Norms layer (optional preprocessing toggles).
• Secondary coder lens (e.g. zstd bundle sizes).
• SentencePiece byte-safe baselines.
• Minimal CI (verify + sizes on enwik6).