Production-grade PDF language detection and splitting — powered by fastText lid.176.ftz
Detects the language of every page in a PDF, groups consecutive pages into language segments, and optionally splits the PDF into per-language files — all in pure Python, with no external service dependencies.
Supports 176 languages via Meta's fastText lid.176.ftz model bundled directly in the package.
- Per-page language detection with confidence distribution
- Sequential segment grouping (contiguous pages of the same language)
- In-memory PDF splitting by language segment
- Batch processing with JSONL + CSV output
- Warmup API to absorb cold-start latency at service startup
- Structured logging via
loguru - Fully typed — passes
mypy --strict
PDF bytes
│
▼
┌─────────────────────────────────────┐
│ Text Extractor │
│ pypdf + opendataloader-pdf │
│ → {page_number: text} dict │
└──────────────────┬──────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Language Detector │
│ fastText lid.176.ftz (176 langs) │
│ → PageLanguageResult per page │
│ (primary_language, distribution) │
└──────────────────┬──────────────────┘
│
▼
┌─────────────────────────────────────┐
│ Segment Grouper │
│ Merges consecutive same-language │
│ pages into LanguageSegment runs │
└──────────────────┬──────────────────┘
│
▼
LanguageDetectionResult
(pages + segments + errors)
│
▼
┌─────────────────────────────────────┐
│ PDF Splitter │
│ split_pdf / split_pdf_by_groups │
│ → PDFSplit objects (in-memory) │
│ write_splits → files on disk │
└─────────────────────────────────────┘
| Layer | Technology |
|---|---|
| Language model | fastText lid.176.ftz (bundled) |
| PDF text extraction | pypdf + opendataloader-pdf |
| Logging | loguru |
| Package manager | uv |
| Linting / formatting | ruff |
| Type checking | mypy --strict |
| Testing | pytest |
git clone <repo-url>
cd pdflangdetect
uv sync --all-extrasmake download-modelThis downloads lid.176.ftz (~900KB) into src/pdflangdetect/_data/.
python example.py --pdf samples/59200_RGCOMBO2_EN_SP_FR_7_22.pdfWith output written to disk:
python example.py --pdf samples/59200_RGCOMBO2_EN_SP_FR_7_22.pdf --output-dir /tmp/splits --warmupCore entry point. Extracts text from every page, runs language detection, and groups pages into sequential segments.
from pathlib import Path
from pdflangdetect import detect_languages
pdf_bytes = Path("samples/59200_RGCOMBO2_EN_SP_FR_7_22.pdf").read_bytes()
result = detect_languages(pdf_bytes)
print(f"Total pages : {result.total_pages}")
print(f"Segments : {len(result.segments)}")
for seg in result.segments:
print(f" {seg.language_name} ({seg.language}) pages {seg.start_page}–{seg.end_page}")Example output:
Total pages : 12
Segments : 3
English (en) pages 1–4
Spanish (es) pages 5–8
French (fr) pages 9–12
Splits a PDF into per-language chunks in memory, one PDFSplit per language segment.
from pdflangdetect import detect_languages, split_pdf
result = detect_languages(pdf_bytes)
splits = split_pdf(pdf_bytes, result)
for sp in splits:
print(f"[{sp.language}] {sp.language_name} {len(sp.pdf_bytes):,} bytes pages={list(sp.pages)}")split_pdf_by_groups(pdf_bytes, page_groups, *, original_stem?, output_dir?) → dict[str, bytes | Path]
Groups arbitrary page lists by language key and merges non-contiguous pages into a single PDF per language. Returns bytes in memory or writes to disk when output_dir is set.
from pdflangdetect import detect_languages, split_pdf_by_groups
result = detect_languages(pdf_bytes)
# Build page_groups from segments
page_groups: dict[str, list[int]] = {}
for seg in result.segments:
page_groups.setdefault(seg.language, []).extend(seg.pages)
# In-memory
splits_bytes = split_pdf_by_groups(pdf_bytes, page_groups)
# Write to disk
paths = split_pdf_by_groups(
pdf_bytes,
page_groups,
original_stem="my_doc",
output_dir=Path("/tmp/splits"),
)Writes a list of PDFSplit objects to disk. Filenames follow the pattern {stem}_{language}.pdf.
from pdflangdetect import detect_languages, split_pdf, write_splits
result = detect_languages(pdf_bytes)
splits = split_pdf(pdf_bytes, result)
written = write_splits(splits, Path("/tmp/splits"), original_stem="my_doc")
for p in written:
print(p)
# /tmp/splits/my_doc_en.pdf
# /tmp/splits/my_doc_fr.pdfPre-loads the text extractor JVM and fastText model. Call once at service startup to absorb cold-start latency; subsequent detect_languages calls skip both initialisation steps.
from pdflangdetect import warmup
warmup()@dataclass(frozen=True, slots=True)
class PageLanguageResult:
page_number: int
primary_language: str # ISO 639-1 code, e.g. "en"
primary_language_name: str # Human-readable, e.g. "English"
distribution: dict[str, float] # All detected languages + confidence scores
@dataclass(frozen=True, slots=True)
class LanguageSegment:
language: str
language_name: str
start_page: int
end_page: int
pages: tuple[int, ...]
@dataclass(frozen=True, slots=True)
class LanguageDetectionResult:
total_pages: int
pages: tuple[PageLanguageResult, ...]
segments: tuple[LanguageSegment, ...]
errors: tuple[str, ...] # Non-fatal per-page errors
@dataclass(frozen=True, slots=True)
class PDFSplit:
language: str
language_name: str
pages: tuple[int, ...]
pdf_bytes: bytes
suggested_suffix: strProcess an entire directory of PDFs and emit JSONL + CSV results:
make batch # defaults: samples/ → logs/
make batch PDF_DIR=my_pdfs OUT_DIR=output
python batch.py --pdf-dir my_pdfs --output-dir output --no-warmupOutput files are timestamped: batch_20260429T224818Z.jsonl and batch_20260429T224818Z.csv.
CSV columns: file, status, total_pages, languages, language_names, read_time_s, extract_time_s, detect_time_s, group_time_s, total_time_s, error.
pdflangdetect/
├── src/pdflangdetect/
│ ├── __init__.py # Public API: detect_languages, split_pdf, warmup, …
│ ├── _detector.py # Per-page fastText inference
│ ├── _extractor.py # PDF text extraction (pypdf + opendataloader-pdf)
│ ├── _grouper.py # Sequential segment builder
│ ├── _splitter.py # PDF splitting (in-memory + disk)
│ ├── _model.py # fastText model loader (singleton)
│ ├── _types.py # Shared dataclasses
│ ├── _languages.py # ISO code → language name mapping
│ └── _data/
│ └── lid.176.ftz # Bundled fastText model (~900KB)
├── tests/
│ ├── conftest.py
│ ├── test_detector.py
│ ├── test_splitter.py
│ ├── test_grouper.py
│ ├── test_concurrency.py
│ ├── test_api.py
│ └── test_e2e.py
├── scripts/
│ └── download_model.py # One-time model download helper
├── samples/ # Sample multi-language PDFs for testing
├── logs/ # Batch run output (JSONL + CSV)
├── batch.py # Batch processing CLI
├── example.py # End-to-end walkthrough script
├── Makefile
└── pyproject.toml
Setup:
make install uv sync --all-extras
make download-model Download lid.176.ftz (run once after clone)
Development:
make format ruff format + ruff check --fix + mypy
make lint ruff check (no auto-fix)
make typecheck mypy on src/
Testing:
make test pytest tests/ -v --timeout=120
Batch:
make batch Run batch detection on samples/ → logs/
make batch PDF_DIR=x OUT_DIR=y Custom directories
Cleanup:
make clean Remove __pycache__, .mypy_cache, .ruff_cache, .pytest_cache
uv sync --all-extras
make download-model
make test
make lint
make typecheckPDF text extraction is powered by opendataloader-pdf — thank you to the opendataloader project for building and maintaining this library.
MIT