Supplement Orchestrator

Backend automation that watches a roofing CRM and an email inbox for new insurance supplement requests, extracts and normalizes the details, analyzes attached documents, and pushes an actionable summary to your team chat.

In the roofing/insurance world, a supplement is a request to add scope or cost to an existing claim. Each one arrives as a loose bundle of emails, PDFs, and photos, and someone has to read all of it, figure out what's missing, and decide the next move. This service does the first pass automatically so the human starts from a structured summary instead of a raw inbox.

What it does

1. Detects new supplements from two sources on a polling loop:
   • a CRM API (RoofLink) filtered by status and, optionally, a specific requester
   • an IMAP mailbox, filtered to supplement-related messages by keyword
2. Normalizes each one into a single canonical SupplementContext object, regardless of source.
3. Analyzes attachments — OCR on images (Tesseract), text/line-item extraction from PDFs (PyPDF2), with an optional OpenAI pass for structured field and line-item extraction.
4. Reasons about the package: builds a "what's missing" checklist, generates clarifying questions for the project manager, and ranks recommended next moves.
5. Persists everything to SQLite plus a JSON artifact per supplement.
6. Notifies a Slack/Discord/Teams webhook with a prioritized summary, including an urgency score.
7. Writes back a context snapshot to the CRM so the record stays in sync.

Features

• Dual ingestion (CRM API + IMAP email) running on independent daemon threads
• Source-agnostic canonical data model built with Python dataclasses and JSON (de)serialization
• Regex-based extraction of claim numbers, carriers, adjuster contact info, and phone/email
• Carrier detection against a known-carrier list, and estimate-intent detection ("provided" vs. "needs to be created")
• Document pipeline with a graceful fallback: uses built-in OCR/PDF parsing, and can load richer external "document model" plugins if present
• Automatic "what's missing" gap analysis across documents and required fields
• Urgency scoring and webhook notifications with recommended next actions
• SQLite persistence with a processing-step audit log
• Context merging that de-duplicates notes/attachments and tracks version deltas
• Secrets kept out of code — all credentials load from a git-ignored .env

Tech stack

• Language: Python 3.8+
• Concurrency: threading (one monitor loop per source)
• HTTP / CRM: requests with retry/backoff via urllib3
• Email: imaplib / email (IMAP over SSL)
• Documents: pytesseract (Tesseract OCR), PyPDF2, Pillow
• AI (optional): OpenAI API for image/PDF/contract analysis
• Storage: SQLite (sqlite3)
• Config: .env for secrets, optional init.yaml for non-secret settings

Architecture

orchestrator.py              # entry point: wires everything together, runs the loops
├── config/
│   ├── settings.py          # loads optional init.yaml + env defaults
│   └── env_loader.py        # loads/masks credentials from .env
├── api/
│   ├── rooflink_client.py   # CRM REST client (retry/backoff, read + write-back)
│   └── email_monitor.py     # IMAP polling, body + attachment extraction
├── core/
│   └── supplement_context.py# canonical model + extraction/normalization engine
├── processors/
│   └── document_analyzer.py # OCR, PDF parsing, optional AI analysis, gap analysis
├── database/
│   └── db_manager.py        # SQLite schema + persistence + processing log
└── notifiers/
    └── intelligent_notifier.py  # urgency scoring + webhook notifications

The processing pipeline (_process_supplement_context) runs the same seven stages no matter where a supplement came from: analyze → enrich → derive checklists → render → decide next moves → persist → notify → (optional) write-back.

Getting started

Prerequisites

• Python 3.8+
• Tesseract OCR binary (only needed for image OCR)
  • Windows: https://github.com/UB-Mannheim/tesseract/wiki
  • Linux: sudo apt-get install tesseract-ocr
  • macOS: brew install tesseract

Install

pip install -r requirements.txt

Configure

cp .env.example .env       # or run setup_credentials.bat on Windows
# edit .env with your CRM key, mailbox credentials, and webhook URL

Optionally copy init.yaml.example to init.yaml to override non-secret defaults such as storage paths. If init.yaml is absent, the app runs on built-in defaults.

Run

python orchestrator.py

This starts two monitor threads that each poll every 60 seconds. Use test_installation.py to verify your environment before the first run.

Configuration reference

Variable	Purpose
ROOFLINK_API_BASE / ROOFLINK_API_KEY	CRM API endpoint and bearer key
SUPPLEMENT_SMTP_HOST / _PORT / _USER / _PASS	IMAP mailbox to monitor
NOTIFY_WEBHOOK	Slack/Discord/Teams/custom webhook (optional)
OPENAI_API_KEY	Enables AI-assisted document analysis (optional)
PRIORITY_REQUESTER	If set, boosts urgency for one requester's supplements (optional)
DOCUMENT_MODELS_PATH	Directory of advanced document-model plugins (optional)

Data model

Every supplement becomes a SupplementContext:

{
  "id": "string",
  "source": "rooflink | email",
  "requester": "string",
  "job_id": "string | null",
  "subject": "string",
  "attachments": [{ "name": "...", "type": "image | pdf | other", "path": "..." }],
  "extracted": {
    "policy": { "carrier": "...", "claim_no": "...", "adjuster_name": "..." },
    "line_items": [{ "code": "...", "desc": "...", "qty": 0, "uom": "...", "rate": 0 }],
    "photos_required": ["..."],
    "estimate": { "provided": false, "needs_user_to_create": false }
  },
  "questions_for_pm": ["..."],
  "whats_missing": ["..."],
  "next_moves": ["..."],
  "audit": { "created_at": "...", "updated_at": "...", "version": 1 }
}

Project status

This is a working prototype of the ingestion → analysis → notification pipeline. The detection, extraction, document analysis, gap detection, persistence, notification, and CRM write-back paths are implemented. Two hooks are intentionally left as extension points and are stubbed out today:

• _render_report — rendering the summary into a standardized report template
• _enrich_context — grounding output in an external knowledge base

The AI analysis paths require an OpenAI key and degrade gracefully to OCR/regex extraction when one isn't configured.

License

MIT — see LICENSE.

Evaluation

Extraction quality is measured, not asserted. eval/ holds a golden set of 12 labeled supplement fixtures and a rubric-scored harness (eval/run_eval.py) that grades each field and gates CI on Tier‑1 extraction accuracy.

Tier-1 (extraction, gated)     97.0%  grade B   [66 checks]   GATE: PASS (threshold 85%)
Tier-2 (diagnostic reasoning)  81.1%  grade C   [13 checks]

The gap between tiers is the roadmap. Where the extractor is weak — measurements parsing, "what's missing" inference — is written down in docs/FAILURE_TAXONOMY.md, which catalogs the concrete ways a supplement gets mis-extracted (unit confusion SQ vs SQFT, carrier line-item aliasing, OCR digit errors, missing-vs-zero) with causes and mitigations.

Run it:

python eval/run_eval.py        # prints the scorecard; non-zero exit if below threshold