Chi

Streaming, checkpointable building blocks for large text-corpus curation in Zig.

Chi is a pre-1.0 library for data infrastructure that cannot keep an entire corpus—or even an entire document—in memory. Its four modules cover bounded ingestion, text cleanup, document filtering, and exact/approximate deduplication. Each module exposes explicit ownership boundaries and a versioned checkpoint format for stopping or moving work between processes.

The current release targets Zig 0.16.0 and has no third-party runtime dependencies.

Modules

Module	Current scope
corpus_reader	Deterministic file and repository traversal plus pull-based stream adapters
text_cleaner	Streaming UTF-8/UTF-16/Latin-1/Windows-1252 decoding and conservative whitespace/control cleanup
document_filter	Bounded-memory length, UTF-8, visibility, replacement, cleaning-change, and repeated-run gates
deduplicator	Chunk-stable exact fingerprints, SimHash candidates, sampled duplicate spans, and a transactional index boundary

const chi = @import("chi");

const CorpusReader = chi.data.corpus_reader.CorpusReader;
const TextCleaner = chi.data.text_cleaner.TextCleaner;
const DocumentFilter = chi.data.document_filter.DocumentFilter;
const Deduplicator = chi.data.deduplicator.Deduplicator;

Each module has a focused README with API behavior, memory ownership, and checkpoint guarantees:

• data/corpus-reader
• data/text-cleaner
• data/document-filter
• data/deduplicator

Verify

Install Zig 0.16.0, clone the repository, and run:

zig build test

The suite exercises arbitrary chunk boundaries, checkpoint transfer, deterministic repository traversal, duplicate classification, and rollback behavior. Windows and Linux run the same test command in CI.

What Chi does not claim

Chi is a set of library components, not a turnkey distributed data platform. This release does not include:

• NFC or NFKC normalization;
• built-in language or safety classifiers;
• a database driver;
• a distributed deduplication backend;
• Unicode word segmentation or token-shingle MinHash; or
• published performance comparisons.

The provided approximate match signal uses byte-level ASCII word features. Downstream systems must evaluate whether that is appropriate for their data.

Status

Chi is early, intentionally narrow software. APIs and checkpoint formats may change before 1.0. The current release is useful as an inspectable foundation for streaming corpus tooling and as research infrastructure for YY's data systems; it is not offered with an SLA.

See SECURITY.md for private vulnerability reporting and CONTRIBUTING.md before proposing changes.

License and authorship

Copyright 2026 Cameron Emery. Licensed under Apache-2.0.

Chi was developed under Cameron Emery's direction with material AI assistance. Human review, release decisions, and responsibility remain with the maintainer.