scoreocr

Transcribes scanned piano sheet music into MusicXML 4.0.

Give it page images (PNG/JPG) or a PDF; it detects the staff/system/measure geometry with OpenCV, asks a vision model to read each measure into a structured musical representation, then deterministically assembles, validates, and renders a MusicXML score you can open in MuseScore, Finale, Dorico, or any notation editor.

It is a personal, local, command-line tool: all state for a transcription lives in a single job directory on disk — there is no database and no server.

How it works

The transcription is a filesystem pipeline. Each stage reads the previous stage's files from the job directory and writes its own, advancing job.json's status. Because every input and output is a file, a run is inspectable at every step and a failed stage can be diagnosed (and, later, retried) from disk.

images/PDF
   │
   ▼
1. ingest ──► 2. geometry ──► 3. crop ──► 4. interpret ──► 5. assemble
                                                                │
                        8. self-check (optional) ◄─────────────┤
                                                                ▼
                                          7. render ◄──── 6. validate

1. ingest — Normalizes every input into one grayscale PNG per page under pages/<page>/source.png, preserving input order. PDFs are rasterized at 300 DPI (pypdfium2); image files are converted with Pillow. All pages of a single run form one job / one song.

2. geometry — Pure OpenCV, no ML. Staff lines are horizontal runs spanning ≥40% of the page width, grouped in fives and paired into grand-staff systems; barlines are vertical runs spanning ≥75% of a system's height, and delimit measures. Both tests look for one continuous run rather than for total ink in a row or column, so that collinear marks — a row of tuplet brackets between the staves, a treble stem above a bass stem — cannot add up into a phantom line. Measures are numbered absolutely and continuously across systems and pages. Writes pages/<page>/geometry.json. If the geometry is implausible (staff lines don't group into fives, no barlines found) it raises rather than feeding garbage downstream.

3. crop — Cuts a system image and a per-measure image for each detected measure into pages/<page>/crops/. The vertical span includes a margin above/below the system so ledger lines survive the crop.

4. interpret — The vision step (see Providers for how the model is chosen; default google/gemini-3.1-pro-preview via OpenRouter). One score-metadata call per score (key, time signature, title), then a two-tier read of the music, because output tokens dominate this pipeline's cost and most measures are easy:

   First pass — one call per grand-staff system, returning a SystemIR (a list of MeasureIR). Batching a system is both cheaper and more accurate than one call per measure: the system image is uploaded once instead of once per measure, one reasoning pass is amortised across every measure in it, and the model can see accidentals, ties, and beams that cross barlines — which an isolated measure crop cannot show it. Systems are read concurrently, at --effort medium by default.

   Escalation — only measures that come back unusable are re-read one at a time on the stronger tier (--model, always at high effort). A measure escalates when its note durations don't sum to the time signature, when the model reports confidence below 0.75, when it was skipped entirely, or when its system's call failed outright. Escalated reads receive the rejected attempt and the reason as feedback.

   Both tiers run the same bounded tool loop: the model may call zoom or grid_overlay up to 3 times to look closer, then is forced to return structured output (notes, chords, rests, rhythm, beams, ties, tuplets, dynamics/tempo/harmony directions). On a system read, grid_overlay also marks the barlines and labels each measure span with its number.

   A measure that fails both tiers is written as an mNNN.error.json sidecar and the job continues — one bad measure never kills the run.

5. assemble — Deterministically serializes the measure IR into MusicXML 4.0: one piano part, two staves (treble = voice 1 / staff 1, bass = voice 2 / staff 2, with <backup> between them), DIVISIONS = 24 per quarter note. Produces a consolidated output/score.musicxml for the whole song plus a standalone pages/<page>/output/page.musicxml per page (each re-states clef/key/time at its first measure so it renders alone).

6. validate — Structural and rhythmic checks on the assembled score (lxml): parseability, measure count and numbering continuity, per-voice duration sums, grace-note handling, final barline. Problems are reported, not fatal.

7. render — Verovio renders the MusicXML to SVG previews (output/preview/*.svg and per-page previews), honoring the encoded system/page breaks so the layout tracks the source.

8. self-check (optional, --self-check) — Rasterizes each rendered system back to an image (cairosvg) and asks the model to compare it against the source crop. Discrepant measures are bounced back through interpret → assemble → render, up to 2 rounds. Remaining discrepancies are reported.

Install

Requires Python ≥ 3.11.

pip install -e ".[dev]"

(Working from a virtualenv is recommended, e.g. python -m venv .venv && source .venv/bin/activate before installing.)

Configuration

The interpret and self-check stages call a vision model, so you need an API key. Put it in a .env file in the project root — the CLI loads it automatically at startup.

cp .env.example .env
# then edit .env and set your key:
# OPENROUTER_API_KEY=sk-or-v1-...

.env is git-ignored so your key is never committed. Real environment variables exported in your shell take precedence over the file, so CI/other environments can supply the key however they like.

Providers

Two providers are supported. OpenRouter is the default, because it fronts any vision model worth pointing at this pipeline.

Provider	Key	Default model
openrouter (default)	OPENROUTER_API_KEY	google/gemini-3.1-pro-preview
anthropic	ANTHROPIC_API_KEY	claude-opus-4-8

Both run and serve take --provider, --model, --fast-model, and --effort; the equivalent environment variables are SCOREOCR_PROVIDER, SCOREOCR_MODEL, SCOREOCR_FAST_MODEL, and SCOREOCR_EFFORT, and flags win over them. Point it at any OpenRouter model that supports vision, tool calling, and structured outputs:

score-transcribe run page.png --model anthropic/claude-opus-4.5
score-transcribe run page.png --provider anthropic          # Anthropic API directly

# Cheap first pass, strong model only for the measures that need it:
score-transcribe run page.png \
  --fast-model anthropic/claude-sonnet-4.5 \
  --model anthropic/claude-opus-4.5

In the web app the model is chosen per batch from a dropdown on the upload screen, which lists a curated set of models alongside their input and output prices per million tokens, read live from OpenRouter's public catalogue. The list lives in src/scoreocr/catalog.py; --model (or SCOREOCR_MODEL) sets the default a batch gets when it names no model of its own. Only --provider openrouter servers show the dropdown — the ids are OpenRouter-namespaced and mean nothing to the Anthropic API.

Prices come from the catalogue rather than from source, because hand-recorded prices drift and a stale number shown as a current one is worse than no number. When the catalogue is unreachable the dropdown still works and the prices show as —. Only the prompt and completion prices are shown; the catalogue's per-image, reasoning-token, and long-context override prices are not folded in.

On the CLI, model ids must match OpenRouter's catalogue exactly; a near-miss is a 404 at the first call, not a warning. Check one with curl -s https://openrouter.ai/api/v1/models | grep <id> before committing it to a config. The web app's dropdown mostly avoids this: when prices are available, an id absent from the catalogue is never offered. When the fetch failed, the full curated list goes out with null prices instead (see above), so a withdrawn id can still be selected there — either way, a model outside the curated list is rejected with a 400 before anything is uploaded.

Requests are sent with provider.require_parameters, so OpenRouter only routes to endpoints that actually honour the JSON schema and tool definitions — a model that silently ignored them would return prose that fails IR validation later in the pipeline.

If OPENROUTER_API_KEY is unset but ANTHROPIC_API_KEY is present, the CLI falls back to the Anthropic provider and says so on stderr. Passing --provider openrouter explicitly disables that fallback and fails instead.

How the OpenRouter adapter works

Interpreter speaks the Anthropic Messages shape natively. Rather than refactor it behind a provider protocol, scoreocr.openrouter.OpenRouterClient exposes the same client.messages.create(**kwargs) surface and scoreocr.openrouter.translate converts to and from OpenRouter's OpenAI-compatible /chat/completions — so interpreter.py, prompts.py, and tools.py are untouched and identical on both providers.

One asymmetry is worth knowing about: an OpenAI-shaped role: "tool" message carries a plain string and cannot hold an image, but both of our tools (zoom, grid_overlay) return images. The adapter therefore answers such a tool call with a short text acknowledgement and attaches the image to the user turn that immediately follows, so the tool loop behaves the same on both providers.

macOS + --self-check only: the self-check stage uses cairosvg, which loads native cairo. If it fails to find it, install cairo (brew install cairo) and export its location before running:

export DYLD_LIBRARY_PATH=/opt/homebrew/lib

This is not needed for a plain score-transcribe run.

Running it

score-transcribe run page1.png page2.png

Pass pages in reading order; a PDF can be given instead of images. Each run creates a fresh job directory under --jobs-root and prints the job path and output locations when it finishes.

Options:

Flag	Default	Meaning
pages	—	One or more input images or a PDF, in reading order.
--self-check	off	Render-vs-source comparison + auto-correction pass (see stage 8).
--jobs-root DIR	data/jobs	Where job directories are created.
--max-workers N	4	Concurrency for interpretation.
--model ID	provider default	Model used to re-read measures the first pass could not read cleanly.
--fast-model ID	--model	Cheaper model for the first pass over each system.
--effort LEVEL	medium	Reasoning effort for the first pass (low–max). Re-reads always run at high effort.

Exit code is 0 on success, 1 if validation reported issues or a stage failed. On failure the job is not left in a mystery state: job.json records status: "failed:<stage>" and an error message, and the CLI prints the failing stage to stderr.

Cost

Output tokens — the model's reasoning, not the images — are the bulk of the bill, so the levers that matter are the ones that reduce reasoning per unit of music. A page costs one metadata call plus one call per system (a handful, not a few dozen), plus one call per measure that fails validation.

Three dials, roughly in order of impact:

• --fast-model is the biggest one. Point the first pass at a cheaper model and let only the measures that fail validation reach --model. If most measures pass on the cheap tier, total cost drops several-fold; if the cheap model is too weak for your scores, you pay for both tiers, so check the escalation rate on one page before committing to it. job.json records input_tokens / output_tokens for the whole run.
• --effort trades first-pass accuracy for output tokens directly. medium is the default; low is worth trying on clean, engraved scores, and pushes more work onto the escalation tier on messy ones.
• --model sets the ceiling. Escalation only sees the hard measures, so a strong model here costs much less than it did when every measure went through it.

Because per-measure reads are now the exception rather than the rule, raising --max-workers mostly affects how many systems are read at once.

Output layout

For a job at data/jobs/<job-id>/:

Path	What it is
output/score.musicxml	The assembled full-song score — the main deliverable.
output/preview/score-NN.svg	Verovio SVG previews of the full score.
pages/<page>/output/page.musicxml	Standalone MusicXML for one page.
pages/<page>/output/preview-NN.svg	Per-page SVG previews.
pages/<page>/source.png	The normalized page image.
pages/<page>/geometry.json	Detected staves, systems, and barlines.
pages/<page>/crops/	System and per-measure crop images.
pages/<page>/transcription/mNNN.json	Per-measure interpreted IR.
job.json	Job status, token usage, and (on failure) the error.

Web app

A browser front end lets you drag-and-drop a batch of page photos, watch each page transcribe with per-photo progress, preview/play/download finished pages as they land, and merge the whole batch into one score at the end.

Build and run

npm --prefix frontend install   # first time only
npm --prefix frontend run build
score-transcribe serve --jobs-root data/jobs

Then open http://127.0.0.1:8000 — FastAPI serves the built SPA at / and the API under /api/... from the same process. serve also accepts --host and --port (defaults 127.0.0.1:8000), and --reload for development (see below).

Development workflow

Run the API and the Vite dev server side by side, so that backend and frontend edits both take effect without a manual restart or rebuild:

# terminal 1
score-transcribe serve --jobs-root data/jobs --reload

# terminal 2
npm --prefix frontend run dev

Open the Vite dev server URL (typically http://127.0.0.1:5173); its dev server proxies /api requests to 127.0.0.1:8000 (see frontend/vite.config.js), so the UI talks to the real backend.

--reload restarts the API whenever a file under src/scoreocr/ changes. Without it the server keeps serving the pipeline code it imported at startup, so editing a stage has no effect until you stop and restart — an easy way to spend a while re-diagnosing a bug you already fixed. Two things to know:

• It watches src/scoreocr/ only, not the working directory, so the files a running job writes under data/jobs/ never trigger a restart.
• A restart kills in-flight transcriptions. Batches run in daemon threads, so one that was mid-flight is left at status: "processing" in its manifest and never resumes; discard that job directory and start over.

API keys are unaffected either way. .env is read when a transcription starts rather than at server startup, so a new key applies to the next job without a restart — it is only code that needs one.

The static-SPA serving behavior — mounting frontend/dist at /, with a non-/api catch-all falling back to index.html — is covered by tests/test_web_api.py::test_root_serves_spa_when_built (it points FastAPI at a fake dist via the SCOREOCR_FRONTEND_DIST env var, so it runs without a real npm run build). When frontend/dist doesn't exist (e.g. it hasn't been built yet), the API-only routes still work; / and other non-/api paths simply 404.

Manual end-to-end smoke test

The automated tests above use a StubInterpreter and need no API key. To verify the real flow against a live model end to end (requires OPENROUTER_API_KEY, or ANTHROPIC_API_KEY with --provider anthropic):

# terminal 1 — build frontend then serve
cd frontend && npm run build && cd ..
score-transcribe serve --jobs-root data/jobs
# terminal 2 (or a browser): open http://127.0.0.1:8000

Verify: upload 2 page images → Convert → per-photo progress advances → finished cards show preview + play + download while a later photo is still processing → after completion, "Merge all" shows the combined score with playback and download.

Tests

pytest
cd frontend && npm test

The suite is fully offline — the interpreter is exercised with a fake Anthropic client, and the OpenRouter adapter with an httpx.MockTransport, so no test makes a network call or needs an API key. On stock macOS, the cairosvg-dependent self-check tests need DYLD_LIBRARY_PATH=/opt/homebrew/lib (see Configuration).