OKCompressor / OKCv4 is a receipts-first research stack for reversible corpus structure, lossless transform experiments, and LM-era data preparation.
The project does not claim a universal magic compressor.
It publishes exact-restore proof modules, benchmark receipts, and public boundary documents.
Public line: no receipt, no claim.
Current modular architecture, source boundaries, and release surfaces:
The active modular surface is:
| Surface | Current role |
|---|---|
| DU3 / DUMBer | Reversible structural identity |
| Rare1 | Optional reversible long-tail filter |
| zRank | Optional reversible rank-model transform |
| Nereids | Exact structural interoperability into tokenizer/model vocabulary views |
| IDStream | Integer-stream interchange and verification |
| Entropy | Replaceable backend boundary |
Two related but distinct paths are kept explicit:
compression / representation:
STRUCTURE → [FILTER] → [MODEL] → [ENTROPY]
interoperability:
STRUCTURAL IDENTITY → NEREIDS → TOKENIZER / MODEL VIEWS
Current Nereids public surfaces:
- Source: https://github.com/OKCompressor/nereids
- 1 GB Phase 3 receipt: https://github.com/OKCompressor/nereids/blob/main/docs/benchmarks/enwik9-phase3-20260911.md
- Lab page: https://okcompressor.com/lab/nereids/
The Phase 3 receipt proves exact byte reconstruction through the tested
DU3-to-cl100k_base vocabulary projection. It does not claim canonical BPE
segmentation equality.
Current public modules:
| Module | Public role |
|---|---|
| Redumb | Reversible corpus transform proof |
| Rare1 | Reversible rare-token sideband proof |
| IDStream | Stable token-stream conversion and verification |
| Rengrams | Repeated symbolic-structure scanner over OKC integer streams |
| Sentinel Lite | Public macro replacement/restoration bridge |
| TCN RFC | Draft neural-codec notes showing why cleaner reversible streams matter |
The Zenodo-facing public report lives here:
docs/okcv4/zenodo_v0_1/OKCv4_PUBLIC_BRIDGE_REPORT_v0.1.mddocs/okcv4/zenodo_v0_1/OKCv4_PUBLIC_BRIDGE_REPORT_v0.1.pdf
Title:
OKCv4 Public Bridge Report v0.1: Receipts-first Reversible Corpus Structure
The current public Sentinel Lite bridge highlight is the enwik7 top1024 base100k lane:
| Metric | Value |
|---|---|
| Original gzip9 | 3,348,144 bytes |
| Main stream gzip9 | 3,337,017 bytes |
| Grammar gzip9 | 8,667 bytes |
| Bundle gzip9 | 3,345,684 bytes |
| gzip9 delta | -2,460 bytes |
| Restore | OK |
This is a small bridge win, not a final compression claim.
Interpretation:
- Rengrams discovers repeated symbolic structure.
- Sentinel Lite consumes a public macro grammar.
- The main stream is macro-replaced.
- Restore expands macros back to the original token sequence.
- Verification uses token-sequence SHA.
A short public chronology is maintained at:
Summary:
| Date | Public milestone |
|---|---|
| 2025-07-15 | dumb/pre-tokenizer public launch and early repository activity |
| 2025-07-16 | OKCompressor core public repository begins |
| 2025-07-22 | pipeline and n-gram/codebook direction visible in core |
| 2025-07-24 | DAG-ready direction visible in core |
| 2026-07-07 | OKCv4 public bridge report v0.1 added |
OKC is a reversible transform pipeline for LM-era data.
It is designed to evaluate exact-restore bundles, not isolated toy numbers. The practical thesis is that stable ID/token streams and reversible structure passes can expose compression-relevant patterns in structured or repetitive corpora.
Bring a corpus. OKC returns receipts.
Public:
- module proof docs
- exact-restore receipts
- benchmark summaries
- public bridge report
- safe RFC-level notes
Private until scrubbed:
- production packing formats
- final macro-selection strategy
- entropy-coding internals
- private Sentinel internals
- large unscrubbed archive receipts
- commercial tuning and operational know-how
This repository started as a personal R&D playground. Older module maps, dev notes, and experimental directions are preserved in:
Those notes are historical context, not the current public claim surface.