RolmOCR Invoice Pipeline (local, Python)

This project uses RolmOCR (served locally via LM Studio OpenAI-compatible API) to:

1. OCR invoices (PDF or image)
2. Classify the invoice:
   • Sector
   • Use-case
3. Output the JSON Schema for the predicted sector × use-case
4. Detect invoice section bounding boxes and (optionally) write annotated images with boxes drawn:
   • vendor_block, buyer_block, invoice_meta, line_items, totals, payment, terms_notes

---

Prereqs (LM Studio)

1. Download a RolmOCR GGUF model in LM Studio (for example: mradermacher/rolmocr)
2. Make sure vision works in LM Studio for that model (mmproj if required)
3. Start Local Server and enable OpenAI-compatible API
   • Endpoint is typically: http://127.0.0.1:1234/v1
4. Confirm the server sees your model:

curl http://127.0.0.1:1234/v1/models

---

Install

cd rolmoocr_invoice_project
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Create a .env in the project root:

LMSTUDIO_BASE_URL=http://127.0.0.1:1234/v1
LMSTUDIO_MODEL=mradermacher/rolmocr

# keep outputs short (reduces context pressure)
MAX_TOKENS=512
TEMPERATURE=0

# render dpi (lower = fewer vision tokens)
DPI=150

---

Run

python3 -m invoice_rolmoocr.cli --input invoice22.pdf --out ./out

Optional flags:

# disable bbox detection
python3 -m invoice_rolmoocr.cli --input invoice22.pdf --out ./out --no-bboxes

# disable annotated images
python3 -m invoice_rolmoocr.cli --input invoice22.pdf --out ./out --no-annotate

# (experimental) extraction to JSON using the selected schema
python3 -m invoice_rolmoocr.cli --input invoice22.pdf --out ./out --extract

Outputs (in --out folder):

• classification.json
• schema.json
• ocr_text.txt
• bboxes.json (if enabled)
• page_0_annotated.png, page_1_annotated.png, ... (if enabled)

Example Annotation

---

How bbox detection works

A) Digital PDFs (text-heavy)

We do not rely on “anchor phrase bbox” (it only boxes labels).

Instead:

1. Extract PDF words with PyMuPDF (get_text("words"))
2. Build text blocks using PyMuPDF block_no grouping
3. Ask RolmOCR (text-only) to label each block into one of the standard sections
4. Union the labeled block rectangles into a single bbox per section
5. Small heuristic merges the right-side “values” column into invoice_meta when needed

B) Scanned PDFs / images

1. Render page to image
2. Use a CV layout detector to propose regions
3. OCR each region (RolmOCR)
4. Ask RolmOCR to map region IDs to sections
5. Union regions per section → bbox

---

Troubleshooting

LM Studio 400 errors

This project prints LM Studio’s full error body. Common causes:

• Context overflow → reduce DPI and MAX_TOKENS, or increase context length in LM Studio
• Model does not support vision → attach the required mmproj and restart the server