Skip to content

MODULES.md

Latest commit

 

History

History
281 lines (185 loc) · 6.92 KB

MODULES.md

File metadata and controls

281 lines (185 loc) · 6.92 KB
Raw

OKC Module Map

Status: public architecture boundary
Updated: 2026-09-11

OKC is a set of reversible modules that can be composed without making any individual stage mandatory.

The canonical high-level pipeline is:

STRUCTURE → [FILTER] → [MODEL] → [ENTROPY]

The brackets are intentional.

Filters, models, and entropy backends are optional.

Composition is encouraged. Coupling is not.

Modules

Module Stage Role Source surface Binary / release surface
DUMBer / DU3 Structure Reversible structural representation Public DU lineage: https://github.com/OKCompressor/redumb https://github.com/OKCompressor/core/releases
Rare1 Filter Optional reversible long-tail transform Private R&D source https://github.com/OKCompressor/core/releases
zRank Model Optional reversible rank-model transform Private R&D source https://github.com/OKCompressor/core/releases
IDStream Interchange Canonical integer-stream conversion and verification https://github.com/OKCompressor/okc-idstream Repository releases / tools
Nereids Interoperability Exact projection from structural identity into tokenizer/model vocabulary views https://github.com/OKCompressor/nereids Repository receipts
Entropy Backend Final entropy coding Pluggable external or future OKC backends Backend-specific

Public module pages

DUMBer / DU3

DUMBer / DU3 owns the structural transform.

It is responsible for producing and restoring a reversible structural representation.

It does not require Rare1, zRank, zstd, or any specific entropy backend.

Current public lineage:

Native releases and benchmark receipts may advance independently of the browser or historical public implementations.

Rare1

Rare1 is an optional reversible long-tail transform over integer ID streams.

Standalone interface:

ID stream + dictionary
        |
      Rare1
        |
transformed ID stream + explicit sideband

Rare1 does not own upstream structure, downstream modeling, or entropy coding.

The current Rust R&D source repository is private.

Public binary distributions may be published independently as Binary-Only Releases under LNCL 1.2.

Public project page:

zRank

zRank is an optional reversible model transform over integer ID streams.

Standalone interface:

ID stream
   |
 zRank
   |
model + rank representation

The original ID stream is reconstructed from the explicit model and rank representation.

zRank does not own upstream structure, Rare1, or downstream entropy coding.

The current Rust R&D source repository is private.

Public binary distributions may be published independently as Binary-Only Releases under LNCL 1.2.

Public project page:

IDStream

IDStream defines interchange and verification conventions for integer token streams.

Public repository:

Current module implementations may contain compatible local readers and writers without requiring IDStream as a runtime dependency.

This keeps modules independently buildable while preserving a common stream boundary.

Nereids

Nereids is an interoperability layer over canonical structural identity.

It does not redefine the compression pipeline:

STRUCTURE → [FILTER] → [MODEL] → [ENTROPY]

Instead it exposes an orthogonal compatibility path:

structural identity
        ↓
      Nereids
        ↓
tokenizer / model vocabulary views

The current public DU3 adapter restores canonical DU lexeme bytes and projects them through a precomputed DU-dictionary-to-target-vocabulary map.

Current Phase 3 evidence demonstrates:

  • streamed translation of a 1,000,000,000-byte enwik9 DU3 stream;
  • projection into valid IDs from the full cl100k_base mergeable vocabulary;
  • zero unmappable DU dictionary entries in the tested corpus;
  • exact raw / DU-decoded / target-decoded byte reconstruction.

This does not claim canonical cl100k_base BPE segmentation equality, drop-in native-tokenizer equivalence, KV-cache portability, or learned-state exactness.

Public source:

Public project page:

Phase 3 receipt:

Entropy

Entropy coding is intentionally outside the structural, filtering, and model contracts.

zstd is the current reference backend in many receipts, but it is not part of DU3, Rare1, or zRank.

Future backends may include other LZ, RLE, ANS-family, domain-specific, or custom entropy stages.

A module result must not be presented as a final compression result unless the entropy/backend boundary is stated.

Composition

Valid compositions include, but are not limited to:

DU3
DU3 → entropy
DU3 → Rare1 → entropy
DU3 → zRank → entropy
DU3 → Rare1 → zRank → entropy

A future okc launcher may orchestrate these stages, but the launcher does not define the implementation boundary of any module.

Modules should remain usable through independent libraries, binaries, or plugin interfaces.

Release model

OKC releases may use either release mode defined by LNCL 1.2:

Source-Available
Binary-Only

Source availability is release-specific.

A Binary-Only Release does not create a source-disclosure obligation under LNCL 1.2.

Canonical license:

  • docs/LICENSE.md

Historical LNCL 1.1:

  • docs/LICENSE-v1.1.md

Public binary channel:

A binary release should carry at minimum:

binary
LICENSE.md
RELEASE_MODE.md
SHA256SUMS
validation / benchmark receipt

Validation policy

Public claims should identify the boundary actually proven.

Examples:

module transform exact
module inverse exact
combined module boundary exact
raw-file roundtrip exact
matched-backend comparison
equal-compute comparison

These are not interchangeable claims.

Structural reduction, model representation size, entropy-coded size, encode wall time, decode wall time, and peak RSS are separate dimensions.

No synthetic score is required to combine them.

R&D boundary

Private implementation work may explore:

  • bounded or chunked execution;
  • parallel scheduling;
  • lower-memory reconciliation;
  • model partitioning;
  • fused module execution;
  • alternative entropy backends;
  • domain-specific and federated models.

Those experiments do not change the public module boundaries until explicitly released.

Design rule

Global correctness does not require global execution.

Pipes, not soup.