bigtextpy

Comparison of serial and parallel implementations for processing large text corpora in Python.

Small, runnable examples covering:

• serial processing
• multiprocessing Pool
• thread-based parallelism
• streaming processing

Files

• src/processors.py — implementations
• src/utils.py — helpers
• src/bench.py — CLI benchmark harness
• tools/generate_corpus.py — generate sample large text files
• tests/test_processors.py — unit tests

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run tests:

pytest -q

Generate a sample corpus (10k lines):

python tools/generate_corpus.py --lines 10000 sample_corpus/large.txt

Run a benchmark:

python -m src.bench --file sample_corpus/large.txt --method multiprocessing --workers 4 --limit 10000

See src/bench.py --help for all options.