OKC IDStream

OKC IDStream is a small converter and verifier for OKCompressor integer token streams.

It provides a stable bridge between:

• ASCII integer ID streams
• packed u32 little-endian streams
• packed u16 little-endian streams when possible
• NumPy .npy streams for Python and Rengrams experiments

CLI

./okcids canonicalize input.ids.txt output.ids.txt
./okcids verify a.ids.txt b.ids.txt
./okcids ascii-to-u32le input.ids.txt output.u32le
./okcids u32le-to-ascii input.u32le output.ids.txt
./okcids ascii-to-u16le input.ids.txt output.u16le
./okcids u16le-to-ascii input.u16le output.ids.txt
./okcids ascii-to-npy input.ids.txt output.npy --dtype uint32
./okcids npy-to-ascii input.npy output.ids.txt

Short helpers:

./okcid2npy input.ids.txt output.npy --dtype uint32
./okcnpy2id input.npy output.ids.txt

Restore standard

ASCII formatting may change during conversion.

The canonical equality check is token-sequence equality, reported by sequence SHA256.

Why this exists

Redumb and Rare1 currently use ASCII integer streams as proof/debug format.

Rengrams and ML experiments need NumPy arrays.

Future Rust paths may prefer packed u32/u16 streams.

OKC IDStream keeps those formats aligned.