Compare PDF text extractors (pypdf, pdfplumber, PyMuPDF, Tesseract OCR, Mistral cloud OCR) on real documents, then pipe the output through an LLM for structured business summaries. Upload a doc, watch each stage, see which extractor wins.
This demos the classic document-AI flow: extract → clean → chunk → summarize → structured data: the same approach behind invoice processing, contract review, and report digestion pipelines.
pip install -r requirements.txt
# Generate sample PDFs
python samples/text_pdf.py
python samples/invoice_pdf.py
# Compare extractors on a PDF
python -m doc_extract.runner compare samples/business_report.pdf
# Include Tesseract and Mistral OCR in the comparison
python -m doc_extract.runner compare samples/invoice.pdf --ocr --mistral --output reports/invoice.json
# Score extractors against expected text (winner picked by F1, not char count)
python -m doc_extract.runner compare samples/invoice.pdf --ground-truth samples/invoice.txt
# Full pipeline: extract → summarize with Ollama (needs `ollama serve`)
python -m doc_extract.runner pipeline samples/invoice.pdf --output reports/invoice.json
# Extraction only, no LLM
python -m doc_extract.runner pipeline samples/invoice.pdf --no-llm
# Single extractor
python -m doc_extract.runner extract samples/invoice.pdf --backend pdfplumber --text
# Run tests
pytest tests/ -v
# Dashboard
streamlit run dashboard.pyNo PYTHONPATH needed: pytest.ini sets pythonpath = src.
Tesseract (local, free). pytesseract on pip is only a wrapper: you
also need the tesseract and poppler system binaries:
conda install -c conda-forge tesseract poppler(Homebrew dropped Intel Macs, so conda-forge is the reliable source there.) If the binaries aren't on your PATH after install, add conda's bin directory:
export PATH="$PATH:$(conda info --base)/bin"To make it permanent, add that line to your ~/.zshrc (or ~/.bashrc).
Mistral OCR (cloud, free tier). Get a key at console.mistral.ai, then:
export MISTRAL_API_KEY=your-key-hereNote: requirements.txt pins mistralai>=1.0,<2.0. v3 restructured the
entire SDK API. The free tier is rate-limited; if you hit 429s, wait a few
minutes and retry, or check your quota in the Mistral console.
1. Compare extractors. Runs pypdf, pdfplumber, and PyMuPDF on your PDF
and scores each on characters extracted, tables found, speed, and warnings.
Tesseract OCR runs automatically when all text extractors find <50 chars/page
(likely a scanned PDF), or when forced with --ocr. Mistral cloud OCR is
available too, opt-in via the dashboard checkbox or include_mistral=True
(needs MISTRAL_API_KEY; free tier at console.mistral.ai). The winner is picked
by evidence: most characters, then most tables, then speed.
Ground-truth scoring. Drop expected text in a <name>.txt next to the PDF
(or paste it in the dashboard, or pass --ground-truth file.txt), and each
extractor is scored on word-level precision / recall / F1. With ground truth,
the winner is picked by F1: accuracy over verbosity.
2. Run the pipeline. Auto-selects the best extractor, cleans the text (whitespace, de-hyphenation, page-number stripping), chunks it, then asks a local Ollama model for a business summary plus structured fields. Long documents (over 12,000 characters) are summarized map-reduce instead: each chunk is summarized first, then the chunk summaries are combined.
{
"summary": "Q3 revenue grew 22.8% year-over-year…",
"key_points": ["Enterprise division led growth", …],
"entities": {"dates": ["October 31, 2026"], "amounts": ["$12,400.00"], …},
"document_type": "invoice"
}Every run records provenance: which extractor won and why, which model was used, per-stage timings.
3. Visualize. The Streamlit dashboard gives you side-by-side extractor comparison, per-extractor text tabs, and the LLM summary with entities, plus one-click sample generation.
| Extractor | Speed | Tables | Best for |
|---|---|---|---|
| pypdf | Fast | No | Born-digital text PDFs |
| pdfplumber | Slower | Best | Invoices, reports with tables |
| PyMuPDF | Fastest | Basic | Large PDFs, layout fidelity |
| Tesseract OCR | Slow | No | Scanned / image-only PDFs, offline |
| Mistral OCR | API-bound | Parsed from markdown | Scanned PDFs, best accuracy (needs API key) |
See METHODOLOGY.md for when to use which and how the metrics work.
- OCR accuracy depends on scan quality. 200+ DPI, straight pages, good contrast. Bad scans → bad text, no matter the engine.
- Table extraction is heuristic. Merged cells, nested tables, and borderless layouts confuse every extractor. Verify against the source.
- Entity regex is approximate. The name finder will catch some false positives ("Contact Jane Smith"). It's a starting point, not NER.
- Document classifier is keyword-based. Ties and jargon-heavy docs return "unknown" rather than guessing wrong.
- LLM summaries need a local Ollama. Without
ollama serve, the pipeline returns extraction results with"summary": nulland a clear error. It never crashes, but there's no summary. - Mistral OCR needs quota. The free tier is rate-limited; 429s are reported as warnings, not crashes. Check your quota in the Mistral console.
No CI workflow is shipped yet: there is no .github directory, so
nothing runs automatically on pull requests. Run the suite locally with
pytest tests/ -v.
Nothing in the pipeline crashes on a missing dependency. One bad extractor
can't kill a comparison, a missing API key becomes a warning, and a dead
Ollama still returns extraction results. Every failure is recorded in the
report's warnings / llm_error fields: inspectable, not silent.
├── src/doc_extract/
│ ├── extractors.py # pypdf / pdfplumber / pymupdf / tesseract / mistral_ocr, one interface
│ ├── metrics.py # scoring + winner selection
│ ├── pipeline.py # extract → clean → chunk → summarize → JSON
│ ├── summarizer.py # Ollama client, entity regex, doc classifier
│ └── runner.py # CLI: compare | pipeline | extract
├── samples/
│ ├── text_pdf.py # generates 3-page business report
│ ├── invoice_pdf.py # generates 2-page invoice with tables
│ └── README.md # how to make a scanned PDF for OCR demo
├── tests/ # 70 tests, no network (everything mocked)
├── dashboard.py # Streamlit UI
├── METHODOLOGY.md # approaches, trade-offs, metrics explained