OKC Rare1

Rare1 is a reversible rare-token sideband transform for OKCompressor-style integer ID streams.

It is a small proof module, not the full OKCompressor pipeline.

What Rare1 does

Rare1 takes:

• an original dictionary file
• an ASCII integer ID stream

It emits:

• a compacted dictionary without rare token types
• an output ID stream where rare occurrences are replaced by a sentinel
• a rare.ones sideband
• a rares.dict rare-type list
• a rare1_stats.json receipt

Then the decoder reconstructs the original token ID sequence.

Current proof

The current public version is Rare1 v0b, a Python reference implementation.

It proves:

• apply succeeds
• decode succeeds
• token-sequence restore is exact
• threshold-1 rare types can be isolated into a reversible sideband
• gzip/xz bundle effects can be measured

Results snapshot

Threshold: 1

Dataset	Mode	Tokens	gzip delta	xz delta	Apply	Decode	Roundtrip
enwik7	ids	4,169,298	+25,063	-65,136	1.865s	1.189s	OK
enwik7	words	4,169,298	+151,758	-88,096	1.820s	1.260s	OK
enwik8	ids	40,770,722	+35,238	-372,784	17.129s	11.304s	OK
enwik8	words	40,770,722	+723,657	-514,176	16.659s	12.105s	OK

Interpretation:

• Rare1 threshold 1 does not improve gzip bundle size in these runs.
• Rare1 threshold 1 consistently improves xz bundle size.
• The main value is structural: Rare1 separates single-occurrence token types into a reversible sideband and reduces the active dictionary.

Modes

ids mode stores old token IDs in rare.ones.

words mode stores rare token strings in rare.ones.

ids mode is the canonical proof mode because it avoids duplicate-token ambiguity and supports exact token-ID restore.

Usage

Apply:

python3 okc_rare1_apply_v0b.py \
  --in-dict path/to/dict.txt \
  --in-ids path/to/ids.txt \
  --out-dir out/rare1_t1 \
  --thresh 1 \
  --rare-ones-mode ids

Decode:

python3 okc_rare1_decode_v0b.py \
  --rared-dict out/rare1_t1/dict.txt \
  --rared-ids out/rare1_t1/output.txt \
  --rare-ones out/rare1_t1/rare.ones \
  --rare-ones-mode ids \
  --orig-dict path/to/dict.txt \
  --out-ids out/rare1_t1/decoded_orig_ids.txt

Wrapper:

./scripts/run_rare1_lane.sh \
  path/to/dict.txt \
  path/to/ids.txt \
  run_name \
  ids \
  1

Restore standard

Rare1 verifies token-sequence equality.

Raw file SHA256 may differ because ASCII ID whitespace can be normalized during decode. The canonical receipt is token-sequence SHA256.

Boundary

This repository contains a public Python reference proof.

It is not:

• the private modulo_freqs_r1 pipeline
• Redumb3
• Rengrams
• graph routing
• final entropy coding
• a universal compression claim

Roadmap

• Rare1 R2 artifact accounting
• minimal ID-restore bundle
• full dictionary/word-restore bundle
• packed rare.ones
• dictionary delta format
• memory receipts with /usr/bin/time -v
• Rust port after semantics freeze
• integration with OKC idstream converters