Skip to content

README.old.dev.md

Latest commit

 

History

History
123 lines (79 loc) · 4.35 KB

README.old.dev.md

File metadata and controls

123 lines (79 loc) · 4.35 KB
Raw

OKCompressor Core

⚠️ dev/scene disclaimer:
This is a personal, in-progress R&D compression playground.
Code, modules, and docs are messy, fluid, and often “notes to self.”
Some submodules are private, drafts or WIP (experimental)—not all code is open right now.
Occasionally, Rust executables or binaries may be dropped in to speed up bottlenecks.
No production support, no guarantees—this repo is “backup-first,” not “release-ready.”
Fork, hack, or lurk at your own risk. PRs welcome but bring patience!

Modular, open corpus compression for the LLM era.

Directory Structure

Click to expand project tree 
/OKCompressor
  /core           # Orchestration, main scripts, API/readme, entrypoint

  /dumb_pre       # Baseline reversible word tokenization/dicts
  /redumb         # Rust-based, WIP, dumb_pre

  /ngram_pos      # N-gram positional tools
  /cc_nlp         # NLP & AI transforms, DAWG, codebook
	/ngram-dawg     # Modular DAWG & automata toolkit
	/rengrams       # Rust n-grams/faster routines

  /crux           # Custom compression transforms (BWT, MTF, etc.)

  /stego          # Steganography/watermarking layers (R&D)
  /mapper         # Symbol remapping, index management
  /cypher	  # pgp, aes per file/block for now. cc_PQC later. FHE.

	/mDAWG          # Multi-level DAWG
	/nGPE           # Next-gen prefix encoding (future/experimental)

  /ranking        # Token/symbol ranking modules

  /entrop         # Entropy coding: rANS/ANS/constriction hooks


* /pLM 		  # pseudo LM, statistical word models from the ngrams of OKC

API Table: Inputs, Outputs, Entrypoints

Module Entrypoint/Script Input(s) Output(s) Notes
dumb_pre dumb_pre.py / redumb raw.txt dict.txt, out.txt Baseline reversible
cc_nlp proc_post.py dicts/ .bwtmtf.txt, .meta.npz Mocked crux postproc
ngram_pos aggregate.py .bwtmtf.txt .npz, .tsv N-gram sweep/agg
cc_nlp ngram_analyzer.py .tsv codebook.json Semantic grouping
cc_nlp replace_ngrams.py ngram_db, input_dir output_dir N-gram replacement
ngram-dawg runner.py token files .edgelist, .order.npy DAWG build/export
crux crux.py token streams transformed streams Custom & transforms
core bench_logger.py output_dir .tsv/.json logs Benchmarking/summary
... ... ... ... ...

Minimal, Reversible Pipeline (v0.3-minimal-pipeline)

Core flow:

dumb_pre → cc_nlp (category tokenization) → ngram_pos (replacement) → crux2 BWT+MTF+RLE → final 7z archive.

    Archive should only include minimal files for lossless reverse.

Pipeline Stages

    Dumb Preprocessing:

        Input: data/enwik6

        Output: output/enwik6/00_dumb/output.txt, dict.txt

    CC-NLP Category Chunking:

        Output: sub_idxs_*.npy, cats_*.base4, cat{n}_commons.txt, cat{n}_uniqs.txt

    N-gram Aggregation (Replacement):

        Output: ngrams_cc_temp.db, ngrams_cc_dicts.npz

    N-gram Replacement (Codebook):

        Output: repl_subidx_*.npy, ngram_used_codebook.txt/.npz

    BWT → MTF → RLE Chain:

        Output: repl_subidx_*_bwtmtfrle.txt

    Minimal Archive Set for Decompression:

        repl_subidx_*_bwtmtfrle.txt, cats_*.base4, cat{n}_commons.txt, cat{n}_uniqs.txt, ngram_used_codebook.txt/.npz

Reverse:
Unarchive → RLE → MTF → BWT reverse → codebook restore → category dicts → join tokens → original file.

Quickstart

# Python 3.11+ recommended; PyPy optional for speed
pip install -r requirements.txt
# Or, for max speed:
pypy3 -m pip install -r requirements.txt

# If using Rust modules (experimental, for redumb/rengrams):
cargo build --release

cd core
python main.py config.yaml
# Or, for PyPy:
pypy3 main.py config.yaml

# Outputs: output/{your_corpus}/99_dicts_final/