Document Vision RAG

Retrieve a document's words, then inspect exactly where they came from.

Document Vision RAG is a local OCR evidence toolkit. It turns Tesseract word boxes into searchable line records, ranks them with BM25, and returns quotes with page coordinates, word offsets, confidence values, and source digests. A standalone evidence atlas lets a reviewer inspect the selected words on the original page.

Chronology provenance: dated entries under benchmarks/chronology/ are synthetic test fixtures generated on September 12, 2026. Assigned dates are not evidence of when development occurred or when these projects were pushed.

flowchart LR
    A[Local raster image] --> B[Snapshot bytes and SHA-256]
    B --> C[Optional Tesseract subprocess]
    D[Recorded Tesseract TSV] --> E[Validate pages and word boxes]
    C --> E
    E --> F[Immutable OCR document]
    F --> G[Group words by OCR line]
    G --> H[Confidence filter for scoring]
    Q[Query and coverage requirement] --> I[BM25 lexical retrieval]
    H --> I
    I --> J{Matching evidence?}
    J -->|Yes| K[Complete OCR quote and exact word citations]
    J -->|No| L[Explicit insufficient evidence]
    K --> M[JSON report and page overlay HTML]
    L --> M

Loading

Where it helps

• Scanned handbook search: locate a retention rule and inspect its position on the page before using it.
• Document review interfaces: attach a page number and bounding box to every retrieval result instead of showing an unexplained snippet.
• OCR pipeline debugging: compare recognized text with source pixels, inspect confidence values, and see which words contributed to retrieval.
• RAG evidence preparation: provide downstream applications with traceable excerpts while retaining a clear distinction between evidence retrieval and answer generation.

This is a focused retrieval prototype. It does not infer table relationships, run a vision-language model, or generate a natural-language answer.

Quickstart without an OCR installation

Python 3.11 or newer; the core package has no third-party dependencies.

git clone https://github.com/ujwalpandey1/document-vision-rag.git
cd document-vision-rag
python -m venv .venv
source .venv/bin/activate
python -m pip install -e .
document-vision-rag demo
document-vision-rag demo --html evidence-demo.html

The default query, backup retention, returns one result from the deterministic synthetic layout:

{
  "status": "evidence_found",
  "citation_id": "northstar:1.1.1.2",
  "text": "Backup retention is 30 days.",
  "box": {"left": 60, "top": 145, "width": 448, "height": 32}
}

This is an abridged result. The actual output also contains the BM25 score, matched query terms, full word citations, half-open text offsets, confidence values, and provenance digests. demo --query "lunar migration" returns insufficient_evidence with an empty hit list.

The no-OCR atlas displays a geometry reconstruction, clearly labeled as distinct from a source image. The fixture's confidence values and boxes are assigned synthetic data.

Run actual OCR

Install Tesseract and language data separately. For example, brew install tesseract on macOS or sudo apt install tesseract-ocr on Ubuntu.

document-vision-rag ocr examples/synthetic-handbook.png handbook.json \
  --id handbook --language eng --psm 6
document-vision-rag search "backup retention" handbook.json \
  --html handbook-report.html --image examples/synthetic-handbook.png

The checked-in synthetic page was rendered specifically for this project. Tesseract 5.5.0 with English data and --psm 6 was actually run against it on September 12, 2026. Its recorded output contains the recognized line Backup retention is 30 days. at (89, 213, 450, 35). The example atlas embeds that same PNG. This single clean-page smoke test is not an OCR accuracy benchmark; results can vary with engine version, language data, font, and input quality.

To render a fresh synthetic input, install the optional image-generation dependency:

python -m pip install '.[demo-image]'
python examples/create_document_image.py fresh-handbook.png
document-vision-rag ocr fresh-handbook.png fresh-handbook.json --id fresh --psm 6

Pillow is only used by that example script. OCR runs through a local subprocess using an argument list and a timeout. No document is uploaded or fetched. Images are snapshotted before OCR so the source digest binds the exact bytes processed.

Import TSV and search multiple documents

document-vision-rag seed sample.tsv
document-vision-rag ingest-tsv sample.tsv sample.json --id sample
document-vision-rag search "retention" sample.json handbook.json --limit 5
document-vision-rag search "backup encryption" sample.json --min-coverage 0.5

By default, all distinct non-stopword query terms must occur in the same line. The final command explicitly permits partial lexical coverage. Coverage measures token overlap, not semantic support. --page and --document-id restrict results; --min-confidence excludes low-confidence words from scoring. Quotes still preserve the complete OCR line, including those excluded words, so filtering cannot silently remove a negation. Each word reports used_for_retrieval.

Outputs are written to new files only. Existing paths produce an error instead of being overwritten. Successful searches, including insufficient-evidence results, exit 0; invalid input or process failures exit 2. An application should inspect the JSON status field.

Python API

from pathlib import Path
from document_vision_rag import EvidenceIndex, parse_tsv, render_report

document = parse_tsv(
    Path("examples/synthetic-handbook.tsv").read_text(encoding="utf-8"),
    document_id="handbook",
    source="synthetic-handbook.tsv",
)
index = EvidenceIndex([document], min_confidence=60)
result = index.search("recovery owner", min_coverage=1.0)
report = index.report(result)
print(report["hits"][0]["citation"]["text"])
# The recovery owner is Priya Shah.
Path("owner-report.html").write_text(render_report(index, result), encoding="utf-8")

Use ocr_image(path, document_id="handbook") instead of parse_tsv to run the local engine. OCRDocument.to_dict() and OCRDocument.from_dict() support the versioned JSON format. Records and index state are immutable; reports are detached dictionaries. A search result belongs to the index that produced it, preventing accidental citation reassignment to a different source.

Citation contract

Field	Meaning
citation_id	Document ID plus Tesseract page/block/paragraph/line identifiers
text	Original OCR word strings joined with one space; original page whitespace is not reconstructed
words[].start/end	Half-open Python Unicode-character offsets into the returned quote
box	Pixel bounding rectangle around every word in the complete quoted line
words[].box	Original validated OCR word rectangle
confidence	Tesseract's score, with -1 meaning unknown; not a calibrated probability
tsv_sha256	SHA-256 of the supplied TSV text encoded as UTF-8
source_sha256	Digest of the snapshotted raster bytes, available when OCR ran through this package

An image overlay requires exactly one single-page document, a matching source digest, and a PNG or JPEG. Multi-page TSV/TIFF can be indexed, but the HTML viewer uses geometry reconstruction for those pages. Input images must already be rendered and oriented correctly; PDF conversion is outside this package.

Validation and limits

python -m unittest discover -s tests -v
# With Tesseract installed, include the actual OCR integration test:
DOCUMENT_VISION_OCR_TEST=1 python -m unittest discover -s tests -v

Tests cover malformed TSV, page boundaries, duplicate IDs, confidence values, immutable state, exact offsets, multi-page isolation, deterministic ranking, abstention, source binding, HTML escaping, safe subprocess behavior, timeouts, CLI round trips, and 50 seeded geometry scenarios. CI installs the package on Python 3.11 and 3.12; a separate job exercises real OCR.

BM25 matches words rather than visual or semantic similarity. OCR errors, handwriting, multi-column ordering, rotated scans, and tables can defeat useful retrieval. Lines follow Tesseract identifiers rather than geometric reading-order inference. The tool returns evidence, not verified facts or an assessment of whether an answer is correct. Source labels and supplied OCR JSON are caller-provided metadata; digests support integrity checks but do not authenticate the document's publisher. Very large inputs are held in memory; use trusted local files and external resource limits where needed. Retrieved text is escaped in reports and is never executed or treated as an instruction by this toolkit.

See architecture and invariants, input and ranking details, and contributing. The TSV integration follows the official Tesseract command-line documentation. Licensed under MIT.