pdfdrill

Token-economical drill-down extraction + PDF→LaTeX OCR quality control.

🔗 Project page: https://pdfdrill.github.io

The badge opens drillspace — a codespace where drillui comes up in your browser on a drilled sample paper. Two clicks, nothing to install. To hack on pdfdrill itself, open this repo in a container instead (.devcontainer/).

pdfdrill is a flat CLI that returns prose, persists state in a sidecar next to each PDF, and wraps the heavy tools (poppler, pdfplumber) so an LLM can drill into a document with the cheapest sufficient command. It is paired with a unified document model (docmodel) and an operator pipeline (docops) that turn MathPix lines.json into a typed Document and emit QC artifacts — a LaTeX | KaTeX | MathPix-image comparison table, a full formula report, and TiddlyWiki tiddlers.

Interfaces (five ways to drive it)

#	Interface	Entry point	For
1	CLI	pdfdrill <cmd> <pdf> (or python -m pdfdrill)	scripts, agents, the SKILL
2	drillui terminal	bun tools/drillui_bridge.ts → browser at :8787 (spawns tools/drillui_chat.py, serves tools/drillui_term.html)	interactive ask-the-document chat in a browser terminal
3	MCP — stdio	tools/pdfdrill_mcp.py	Claude Desktop / Claude Code (local) — results as MCP resources
4	MCP — Streamable HTTP	tools/pdfdrill_mcp_http.py → https://<host>/mcp	claude.ai web custom connector
5	Batch	tools/drillbatch.py	drill a list of URLs/ids shallow→deep; --list-outputs

drillui details → tools/DRILLUI.md; MCP details → tools/MCP.md. All UI/server files have ONE canonical copy, in tools/.

Install

System prerequisites (not pip-installable): poppler-utils (core), ghostscript (the only page rasterizer), and the LaTeX DVI toolchain + dvisvgm (TikZ/table/chemistry SVG rendering). There is deliberately no OCR engine in that list — which reading a document gets is the lane contract's decision (docs/LANES.md), and its last lane is free and offline. bash bootstrap.sh installs whatever is missing via apt-get; pdfdrill doctor reports the state anytime.

apt-get install poppler-utils      # Debian/Ubuntu
dnf install poppler-utils          # Fedora
eopkg install poppler              # Solus
brew install poppler               # macOS

Then:

pip install -e .                   # installs the `pdfdrill` console script
# or, for the offline path only:
pip install -r requirements.txt

One step pip cannot do: the patched pdfminer. The glyph-identity features read LTChar.glyphname and LTChar.invisible, which stock pdfminer does not have — it parses the PostScript glyph name and discards it. The patch lives in this repository (vendor/pdfminer-glyph-identity.patch) and is applied by its own installer, so it is not a line in pyproject.toml: it is not a package on PyPI, and naming it as a requirement would break pip install for everyone.

bash vendor/install-pdfminer-fork.sh --user --rebuild   # or --venv in a venv
bash vendor/install-pdfminer-fork.sh --check            # is it in place?

bash bootstrap.sh ends with this step, so a bootstrap install already has it. A plain pip install -e . does not: everything still imports, the glyph-name features are silently off, and seven tests skip saying so.

--check is the way to ask. pdfdrill doctor does not answer this — it prints [OK ] pdfminer — core: born-digital text layer (model) whether the patch is there or not (measured on a base install, 2026-10-26), which is the same "looks clean, feature gone" shape the patch's own history is made of. python3 -c "from pdfreader import build_identity as b; print(b.summary())" names the loaded build, its five features and whether it is behind the patch.

First, unlock the build commands (once per machine, 24 h)

size, route, links, doctor and the other read-only commands answer immediately. Everything that BUILDS — model, report, inspect, md, drill, … — is blocked until you attest that you read the SKILL, because pdfdrill prose looks authoritative and is silently wrong when used without the usage rules:

pdfdrill preflight                            # the rules, and where the SKILL is
tail -3 .claude/skills/pdfdrill/SKILL.md      # the DRILL-… token, ABOVE the END marker
pdfdrill preflight --ack DRILL-xxxxxxxx       # the token you just read

The token is a checksum of the SKILL body and changes whenever the SKILL does, so read it from the file. Automation that has read the rules sets PDFDRILL_NO_PREFLIGHT=1. Full walk-through: QUICKSTART.md.

Quick start

pdfdrill size  paper.pdf           # one-sentence metadata (~40 ms)
pdfdrill links paper.pdf           # external URLs from the annotation layer,
                                   #   flags code/data hosts (~50 ms)
pdfdrill model paper.pdf           # build the unified docmodel from lines.json
pdfdrill compare paper.pdf         # LaTeX | KaTeX | MathPix-image QC table
pdfdrill report  paper.pdf         # full inline+display formula report
pdfdrill tiddlers paper.pdf        # TiddlyWiki tiddler array (--bibkey KEY sets the prefix)
pdfdrill translate paper.pdf --from DE --to EN-US   # DeepL-translate prose tiddlers
pdfdrill folder  ./papers          # batch-build every PDF that already has a
                                   #   sibling .lines.json — no network calls
pdfdrill markdown summary.md       # source-only model from LLM-summary Markdown
                                   #   (+ gold ```bibtex appendix)
pdfdrill tables  paper.pdf         # span-aware tables (keyless) + tables.html QA
pdfdrill semantic paper.pdf        # evidence-backed entity/relation graph
pdfdrill gaps    paper.pdf         # MISSING-information linter (acronyms/symbols/
                                   #   claims/citations)
pdfdrill rulebook paper.pdf        # claims/definitions -> kitems -> rulebook.md
                                   #   with [→k:hash] drill-down anchors

Every command's <pdf> argument is, interchangeably, a local file path, an https URL from a known host, or a bare arXiv id (pdfdrill model 2510.11170v2) — downloaded once, cached. pdfdrill does the acquisition and decides if/when to use the LaTeX source: for an arXiv input the abstract and the author LaTeX come from the free routes (injectlatex/model auto-download the e-print .tgz and ingest the gold equations, keyless, no MathPix). Pass the URL/id/path and let pdfdrill fetch it — never curl/wget/tar a PDF or e-print yourself, and never hand-process a .tex/.tgz.

Without the installed console script, run as a module:

PYTHONPATH=src python3 -m pdfdrill <command> <pdf> [args]

Which reading a document gets — one command, no choosing

pdfdrill drill <pdf>        # the whole way, free only, ends in an inspect.html
pdfdrill route <pdf>        # just the lane, and why

Five lanes in a fixed order; a lane is taken only because every cheaper one above it is unavailable, so the same document always gets the same answer. OCR is the last resort — the three OCR lanes are reached only when the document has neither a text layer of its own nor an author's source.

#	lane	when	cost
1	its own text layer	the PDF carries real text (not an OCR underlay)	free
2	the author's LaTeX	no text layer, but an arXiv e-print	free
3	gemmaocr	a scan + MAKORA_API_KEY	metered, ~0.0007 EUR/page
4	mathpix	a scan + MATHPIX_APP_ID/KEY only	paid
5	gemmaocr --geometry-only	a scan + no reading key	free (boxes and types, no words)

Nothing buys a reading unasked. A lane that costs money is named with the exact command and not run — by route, by drill, and by model, whose --mathpix is the explicit opt-in. PDFDRILL_SANDBOX=1 makes a keyed machine answer as a keyless install does, which is how a lane decision is checked before it ships. The contract, the measurements behind it and the two things its previous version got wrong are in docs/LANES.md.

Projections — one docmodel, many outputs

The unified docmodel is the canonical IR; every projection is codegen from it (build the model once with pdfdrill model, then project). Each is a single command that writes into the doc's .drill/ sidecar:

Command	Output	What it is
pdfdrill md <pdf>	<bibkey>.md	LLM-compact Markdown with materialized math transclusions (bi-layer source+translation after translate)
pdfdrill llmtext <pdf>	<bibkey>.llm.txt	Flat LLM dump — one unit per block, tiddler-titled, document order
pdfdrill context <pdf> "q" [--type …]	stdout / --out	Deterministic structural RAG: select typed objects by query/--type/--section/--concept, ranked, with a token budget
pdfdrill tiddlers <pdf>	<bibkey>.tiddlers.json	TiddlyWiki tiddler array (bibkey-prefixed titles; <$latex>/transclusions/SVG)
pdfdrill okf <pdf> [--semantic]	okf/<bibkey>/*.md	Open Knowledge Format bundle — one Markdown-with-frontmatter file per unit in per-type folders, relative links; --semantic projects the entity graph (Company/BankAccount/…) instead
pdfdrill distill <pdf>	<bibkey>.distill.html	Distill-structured single-file reading view (named-column grid, runtime TOC, late-bound ?? figure refs, hover citations, hard-dark, KaTeX)
pdfdrill report <pdf>	formula-report.html	Inline+display formula report (LaTeX | KaTeX | MathPix image)
pdfdrill compare <pdf>	compare.html	LaTeX | KaTeX | image comparison across competing provenances (MathPix/snip/vision/tex) + scores
pdfdrill inspect <pdf>	<bibkey>.inspect.html	DevTools-style docmodel inspector — every object a hover/click box, self-contained via the pdfminer route
pdfdrill latex <pdf> [--compile] [--dump-stages]	latex/<bibkey>.tex	PROJECT the docmodel to compilable LaTeX — the LaTeX analog of md. Reads the unified Document, so the SOURCE that built it (MathPix / arXiv / Gemma-4 / textscan) is irrelevant. Transclusion via a readarray formula array ({{id||FO}} → \Expr{i}), \cite, bibliography from References. Compile with xelatex (--compile); --dump-stages writes the inspectable generation stages
pdfdrill beamer <pdf> [--compile]	latex/<bibkey>.beamer.tex	PROJECT the docmodel to a beamer slide deck — one frame per Section (allowframebreaks), title + outline + References frames. Same source-agnostic projection as latex (transclusions, \cite, lists resolve), a deck instead of an article. Compile with xelatex
pdfdrill injectlatex <pdf> [--tex P]	model provenance	INJECT an author/MathPix LaTeX source (.tex/.tgz, arXiv e-print auto-downloaded) INTO the model as gold tex provenance (was latex; the INPUT direction)
pdfdrill scikgtex <pdf> [--compile]	<bibkey>.scikg.tex	SciKGTeX-annotated LaTeX → compiled PDF carries ORKG contribution metadata as XMP/RDF
pdfdrill stex <pdf> [--stex]	<bibkey>.glossaries.tex / .stex.tex	Enriched LaTeX / sTeX — acronyms, glossary, table-of-symbols, index from the named-concept layer
pdfdrill lean <pdf>	<bibkey>.lean	Lean 4 export of theorems (LLM-generated Lean stored per Theorem, then projected)

HTML projections accept --embed to base64-inline every CDN crop (fully self-contained); the .distill.html, .inspect.html, report.html and the okf/ bundle all open directly in drillui's Outputs panel.

What may carry a transclusion, and what renders in place

Every projection above resolves math through the same pivot: an inline formula is a DocObject with a title, and the TiddlyWiki <$latex .../> widget is the form every other projection converts FROM (okf derives $…$, the LaTeX lane derives an environment, the HTML views call katex.render).

Two rules govern which math takes that route:

• A transclusion is a claim about prose. Math on a table cell, a section heading, a title, a TOC entry, a figure label or inside a caption is not an inline formula and never becomes one. The object that owns the line already carries that math whole — a Table keeps its entire \begin{tabular}, a Section keeps its heading. src/docmodel/line_types.py states the set and the measurement; 29.0% of the corpus's non-equation inline-math occurrences sit on such a line.
• Refusing the transclusion never means losing the mathematics. src/docops/mathdelims.py converts MathPix's own \(…\) / \[…\], wherever it stays in a body text, into the same widget — in place, with no object and no title. So a heading with math is typeset, and the wiki no longer shows literal backslash-parenthesis.

Reports and evidence

Command	Output	What it is
pdfdrill evidence <pdf> --all-kinds --pdf	evidence-{equation,formula,table,image}.pdf	The published evidence surface: every object of a kind, one row each, with its page, confidence, LaTeX source, rendering and crop. An inline formula's page/confidence/crop are its HOST LINE's, and the file says so
pdfdrill residuals <pdf> --pdf	residuals.pdf	The action list — Corrected, Unresolved, Flagged, Low confidence, Doubted, worst first
pdfdrill marks <pdf> [--build]	marks.json	inkdrill's formula marks as a command (a subprocess, never an import). --build draws a rectangle on the formula evidence only where the ink is marked AND the row's line matches pdfdrill's own host line
pdfdrill publishready <pdf>	verdict	Artefacts, glyphs, ink freshness, timestamps, coverage, crop identity

Crop size is governed by a budget ladder (reports/budget.py): scale and quality step down a fixed ladder until the artefact fits, 1.00/q75 down to 0.42/q70 (105 dpi, the floor). docops/conserve.py is a ratchet gate — a projection that drops objects a previous one carried fails rather than ships.

The killer case

pdfdrill links reads the PDF annotation layer, so it finds hyperlinks that have no visible anchor text — e.g. a paper whose page-1 text says only "Our code is available here." where here is a link. The URL never appears in any rendered-text stream (plain extraction, Markdown, or a chatbot upload all drop it); links surfaces it in ~50 ms.

Translating a document (DeepL, in place)

pdfdrill translate <pdf> [--from RU] [--to EN-US] [--limit N] [--force] translates the document in place: the model prose is translated (original preserved under <field>_source), the bi-layer Markdown <bibkey>.md is re-projected (translation shown, original behind a show-source toggle), and the tiddler file is rewritten in place — text carries the translation, text_source the original. Math/code/image objects are untouched; re-runs are idempotent. Needs DEEPL_API_KEY (see API keys below).

Publish a document set to GitHub Pages

Turn a drilled set into a standalone TiddlyWiki served from github.io:

pdfdrill repoinit <repo> --username <you> --title "My Set"
pdfdrill tiddlers <pdf>          # per document (produces <bibkey>.tiddlers.json)
pdfdrill publish <repo> <pdf> …  # export tiddlers -> tiddlers/, PDFs -> files/
cd <repo> && npm i tiddlywiki && npx tiddlywiki . --output . --build index
git init && git add -A && git commit -m "my set"
gh repo create <repo> --public --source=. --push       # from your own machine
gh api -X POST repos/<you>/<repo>/pages -f 'source[branch]=main' -f 'source[path]=/'

Live at https://<you>.github.io/<repo>/. In the Claude.ai sandbox the docset-publish SKILL runs build → tar; you push from your own machine (no credential ever enters the sandbox). A one-time navigator on <you>.github.io auto-lists every doc-set repo.

Feature extraction (src/features/)

src/features/ is a small additive layer of source-agnostic extractors: each extract(text, page_id="") -> list[Feature] takes plain text and emits flat Feature objects, with flat Relation edges and a NetworkX build_graph. There is no standalone pdfdrill features command, but the extractors are now consumed by several commands — identifiers (ISBN/DOI/ids + author matching), entities (IBAN/BIC/German address/ids), spellqc and semantic (language detection).

pip install '.[features]'                      # dateparser, phonenumbers, price-parser, …
PYTHONPATH=src python3 -m features invoice.txt # → JSON of EMAIL/URL/DOI/DATE/PHONE/PRICE/… features

Always-on (regex / checksum, no deps): EMAIL, URL, DOI, IBAN, BIC, ISBN/ISSN, German admin ids, LANGUAGE. Library-backed (degrade to nothing when the dep is absent): DATE (dateparser), PHONE (phonenumbers), PRICE (price-parser), PERSON_NAME (probablepeople), ADDRESS (usaddress); match_entities (rapidfuzz) links OCR-typo/invoice/company duplicates as SAME_AS relations. Two read-only audits ship too: python -m features.audit_deps (module dependency graph) and python -m features.audit_nested (nested-container findings).

Layout

• src/pdfdrill/ — the CLI toolkit, sidecar state machine, and capture layer.
• src/docmodel/ — the unified document-object model (Stream / Anchor / Realization / Alignment / Region), built from MathPix lines.json.
• src/docops/ — Mutators and Projectors over a Document (plaintext, LLM-compact markdown, TiddlyWiki, comparison table, formula report).
• src/features/ — additive, source-agnostic feature extractors: flat Feature/Relation, a registry, and a NetworkX graph builder. No standalone command; consumed by identifiers/entities/spellqc/semantic.
• src/semantic/ — the evidence-backed semantic graph (entities/relations/ evidence, identity resolution, grounding layers G1–G4, concepts, kitems, the compiler, gap detection, bundle/rulebook/sTeX projections).
• tools/drillui_* — an "ask-the-document" terminal UI (see below).
• tools/pdfdrill_mcp.py + tools/drillbatch.py — a stdio MCP server (drill results as MCP resources) + a batch driver (see the MCP section below).

drillui — ask-the-document terminal

tools/drillui_* is a browser terminal for chatting with one drilled document. Three co-located pieces (full guide: tools/DRILLUI.md):

• drillui_chat.py — the brain (Python REPL over one doc): asks pdfdrill retrieve for grounded context, sends the enriched prompt to an LLM (claude -p), and stores the Q&A back via pdfdrill chatlog. Runs standalone in a terminal; self-locates pdfdrill from ../src.
• drillui_bridge.ts — the bridge (Bun): spawns one drillui_chat.py per WebSocket, pipes stdin/stdout, serves the UI + the files pdfdrill writes (/artifact) + a host-browser opener (/open). Pure plumbing.
• drillui_term.html — the UI (xterm.js terminal + retrieval rail + Outputs panel of clickable reports).

Run it (zero config from the repo root — the bridge finds its siblings):

bun tools/drillui_bridge.ts data/paper.pdf      # then open http://localhost:8787/
bun tools/test_drillui_bridge.ts data/paper.pdf # integration test

Command model in the terminal: open <url|file> opens a new window (handled locally, never an LLM call); a pdfdrill command name (status, report, model, latex, …) runs on the open doc; anything else is a grounded question. The document must already be drilled (pdfdrill model <doc>).

MCP server — drill results as clickable resources in a chat client

tools/pdfdrill_mcp.py is a pure-stdlib stdio MCP server (no mcp SDK, JSON-RPC 2.0 over stdin/stdout) exposing drill / md / tiddlers / report. Each tool runs pdfdrill on the resolved doc (local path, https URL, or arXiv id) and returns a text summary plus the produced files as MCP resources — resource_link items + embedded resource content, fetchable via resources/read. The client pulls files over the MCP connection, so there is no localhost port and no dead /artifact link (the failure mode when a drillui bridge URL is opened inside a hosted chat client).

Best fit: Claude Desktop / Claude Code (local stdio) — add to claude_desktop_config.json:

{ "mcpServers": { "pdfdrill": {
    "command": "python3",
    "args": ["/ABS/PATH/pdfdrill/tools/pdfdrill_mcp.py"] } } }

then ask: "drill arxiv.org/abs/2305.04710 standard" → the md/tiddlers come back as openable resources. The claude.ai web client uses tools/pdfdrill_mcp_http.py — the Streamable HTTP transport (one /mcp endpoint, SSE responses, session + optional bearer auth + CORS) behind your public HTTPS front, added as a custom connector. It imports the stdio server, so both transports expose the same tools. Batch helper: tools/drillbatch.py (shallow→standard→deep ladder; --list-outputs prints every produced file). Full guide: tools/MCP.md.

The layer tower

The whole toolchain is one stratified stack L0–L8 (container → raster → glyph → text lines → layout regions → typed objects → expression syntax → semantic graph → ontology). One canonical document per layer lives in docs/layers/; the inter-layer semantics (support/ abstraction maps, split recovery, level skipping, metrics) in docs/layers/TOWER.md.

API keys

Network features read their credentials from environment variables (or a git-ignored .env — copy .env.example); keys are never stored in the repo. The whole offline path (folder, model, compare, report, tiddlers, drill, geometry, embedimages, bibsource, latexbook, …) needs no keys at all. (ocr used to be in this list; it is RETIRED and reads nothing — the keyless measurement of a scan is gemmaocr --geometry-only, free and offline.)

Key	Used by	Get it
MATHPIX_APP_ID / MATHPIX_APP_KEY	mathpix, model (OCR → lines.json), snip	https://mathpix.com/ → Console → API Keys
OPENAI_API_KEY	vision (GPT-4o reads CDN crops MathPix left as images → math/TikZ/table)	https://platform.openai.com/api-keys
NOVITA_API_KEY	snip --gemma / the metered scan lane (gemmaocr; GEMMA_MODEL/NOVITA_BASE_URL override). The lane contract is docs/LANES.md — page count no longer picks a lane	https://novita.ai/ → Settings → Key Management
DEEPL_API_KEY	translate (DeepL-translate prose tiddlers)	https://www.deepl.com/your-account/keys (free keys end in :fx)
PERPLEXITY_API_KEY	bibfetch (online BibTeX enrichment)	https://www.perplexity.ai/ → Settings → API

Run pdfdrill doctor to see which system tools, Python deps, and keys are present and what each enables. A blocked sandbox host yields a clear "enable it in your network settings" message, never a stack trace.

Tests

for t in tests/test_*.py; do python3 "$t"; done

See CLAUDE.md for the full command list and architecture notes.