The LLM proposes. A deterministic verifier decides every number.
LedgerMind: verifiable financial question answering with a small language model that cites, a calculator that computes, and a verifier that decides.
A 4B open model, fine tuned with SFT and then GRPO against a verifier, answers FinQA questions statistically tied with gpt-5.5 on the same questions, at about 600 times lower cost per query if self hosted, and every answer comes with the figures it used and where each one is printed. It also ties Qwen3.8-27B through the same pipeline and beats gpt-oss-120b; the one model that does better is Qwen3.8-27B answering directly, by 2.7 points with about 8 times the output tokens and no verification.
Open the results dashboard and audit explorer
The problem. AI chatbots are unreliable at multistep arithmetic over financial documents, and a wrong number looks exactly like a right one. In finance, a figure that cannot be traced back to the filing it came from is not usable.
The approach. LedgerMind splits the job in three. A small open model (4 billion parameters) reads a page of a company's annual report and does only two things: it points to the figures it needs, cell by cell, and writes a short formula. A plain calculator runs the formula. A verifier then checks that every cited figure is really printed where the model says it is, and refuses the answer, with the reason, if any check fails. The model was trained on one GPU in two stages: first to learn this format, then with reinforcement learning in which the verifier itself is the reward.
The results. On the 1,139 test questions of FinQA, a benchmark written by finance professionals over S&P 500 filings, LedgerMind answers 74.9% correctly with every answer verified, against 20.4% for the same model before training. It is statistically tied with OpenAI's gpt-5.5 on the same questions at about 600 times lower cost per query, it cut made up figures from 46.6% to 1.7%, and its verifier caught every one of 7,212 deliberately corrupted answers. One much larger open model, Qwen3.8-27B, does better by 2.7 points when it answers directly, without verification.
Why it matters. A small, cheap model can match frontier accuracy on a specialised task while producing answers an auditor can check line by line. Every comparison is reported, including the one it loses.
Contents: How it works · Live demo · For finance readers · Results · Caveats · Quickstart · Conclusion · What it cost · Skills
The four steps below use a real FinQA question. Any reader can follow them; the diagram after them shows the system an engineer would build.
Screenshots of the trained model answering live on one NVIDIA H200 through vLLM (Slurm job 10969910, 10 October 2026), rendered by the demo page in serving/demo. Each answer took about half a second.
A question that is not in the dataset. Asked what share of AES's scheduled debt maturities falls due in 2003, the model cites the 2003 row (2,323) and the total (22,258), the executor computes 0.104, and the verifier accepts it.
A counterfactual filing. The amortization figure was edited from $12 million to $13 million. The model reads the edited document and answers 21, not the real filing's 20.
A refusal. This filing's table was flattened into one row in the dataset, and the model cites a cell that does not exist. The verifier rejects the answer as a bad pointer instead of returning it, even though the number happens to equal the true percentage change (FinQA's gold label for this question is itself wrong).
To run it yourself on a cluster: sbatch slurm/demo.sbatch, then tunnel to the port the job log prints.
FinQA is a public benchmark of about 8,000 questions written by finance professionals over real S&P 500 earnings reports: a table from a 10-K plus the surrounding text, and a question that needs one to several calculation steps. A typical answer from LedgerMind looks like this:
Question (AES, 2001 10-K): what percentage of scheduled maturities of total debt are due after 5 years?
Cited figures:
e1 = 12,806(debt maturities table, row "thereafter") ande2 = 22,258(same table, row "total"), both checked as printed at those cells. Plan:e1 / e2, run by a deterministic executor, not the model. Answer: 0.5753, that is 57.5%, accepted by the verifier and matching the gold answer.
If a cited figure is not printed where the model says it is, or the plan does not really use the figures, the answer is rejected with the reason instead of being returned. That is the property a reviewer, an auditor or a controller needs: every number is traceable to the source, and nothing is computed in the model's head. The audit explorer shows 30 such trails, including rejected and counterfactual cases.
- Post training: LoRA SFT, then GRPO with verifiable rewards on one H200, with prompt filtering by SFT pass rate so every GRPO step has signal.
- Reward and verifier design: one verifier used as the GRPO reward, the inference gate and the scorer, with reward hacking holes closed (unused evidence penalty, plan sensitivity check, literal policy).
- Evaluation rigor: frozen held out test set, bootstrap intervals, paired McNemar tests on identical questions, deterministic reruns, and published caveats where the comparison favors this project.
- Serving and cost: vLLM in bf16 and fp8 with a concurrency sweep, the measured cost of constrained decoding, and cost per query against API list prices.
- Engineering: typed
src/package, tests in CI, pinned environments, cluster jobs that resume across time limits, and every reported number traced to a committed artifact.
LLMs are unreliable at multistep financial arithmetic, and a wrong number looks exactly like a right one. LedgerMind never lets the model write the final number. The model cites figures with source pointers and writes a computation plan; a deterministic executor runs the plan; a verifier checks every cited figure against the document. The answer comes back with a full audit trail, or it is refused with a reason.
The verifier is also the reward: GRPO optimizes the model directly for answers that survive verification.
Question + Document
│
▼
Model (SFT + GRPO) ──► evidence with source pointers + computation plan
│
▼
Executor ──► the number (safe AST evaluation, no eval)
│
▼
Verifier ──► provenance · plan hygiene · invariants · document integrity
│
▼
Answer + audit trail, or REJECT with reason
All numbers are on FinQA test, scored by the same harness, and traced to files in docs/results (index of jobs, settings and code versions in docs/results/README.md). Verified accuracy means correct and accepted by the verifier, over all questions.
| System | Verified accuracy | Answers verified | Cost per 1k queries |
|---|---|---|---|
| gpt-5.5, direct answer | 80.3% [76.3, 84.5] | no | $12.90 |
| LedgerMind GRPO (4B, ours) | 78.5% [74.3, 82.5] | yes | $0.022 |
| gpt-5.5 through the LedgerMind pipeline | 75.0% | yes | $18.59 |
| gpt-5.4-mini, direct answer | 72.5% | no | $1.10 |
| gpt-5.4-mini through the LedgerMind pipeline | 68.8% | yes | $1.78 |
| Qwen3-4B base, zero shot | 21.3% | yes |
Paired on identical questions (McNemar): GRPO against gpt-5.5 direct is 1.75 points lower, p = 0.38, a tie; against gpt-5.5 through the pipeline it is 3.5 points higher, p = 0.08, also not significant; it beats gpt-5.4-mini both ways (p = 0.009 direct, p < 0.001 pipeline). The 400 questions are a seeded sample of the 1,139 grounded test questions, so intervals are about 4 points wide.
Self hosted with vLLM on one A100 80GB, thinking enabled, the reasoning stripped from each reply, and scored by the same harness.
| System | Verified accuracy | LedgerMind GRPO minus this | Output tokens per answer |
|---|---|---|---|
| Qwen3.8-27B, direct answer | 77.6% [75.2, 80.0] | -2.7 points [-5.1, -0.4], p = 0.029 | 1,367 |
| Qwen3.8-27B through the LedgerMind pipeline | 75.0% | -0.1 points, p = 1.0 (tie) | 1,268 |
| gpt-oss-120b, direct answer | 72.0% | +2.9 points [+0.4, +5.5], p = 0.034 | 385 |
| gpt-oss-120b through the LedgerMind pipeline | 62.7% | +12.2 points, p < 0.001 | 509 |
| LedgerMind GRPO (4B, ours) | 74.9% | 166 |
Qwen3.8-27B, about seven times larger and several releases newer than the Qwen3 base model LedgerMind was trained from, is the one system that beats it, and only when answering directly without verification. Through the verifiable pipeline the two are tied to the question (101 against 102 discordant answers). Spelling out the pipeline's conventions (prompt v2) changed neither open model's accuracy.
| Base, zero shot | + SFT | + GRPO | GRPO in fp8 | |
|---|---|---|---|---|
| Verified accuracy | 20.4% | 74.5% | 74.9% [72.3, 77.3] | 74.9% |
| Fabricated figures | 46.6% | 1.9% | 1.7% | 1.5% |
| Unconstrained JSON parse | 80.4% | 99.9% | 99.9% | 99.9% |
SFT does the heavy lifting (+54.1 points, p < 0.001). GRPO's gain under greedy decoding is not significant (+0.4 points, p = 0.40), but sampled at temperature 0.9 it is large: +7.1 points [+6.0, +8.2] over four seeds, and the questions solved on every seed rise from 552 to 784. GRPO sharpened the policy rather than changing its best answer, with a small cost in breadth (solved on at least one seed: 79.6% for SFT, 77.2% for GRPO).
On one H200 with vLLM, the GRPO model serves 44.7 requests per second at concurrency 64 in bf16 and 48.7 in fp8, with identical accuracy (74.89% both, 7 against 7 discordant answers). JSON schema constrained decoding halves peak throughput to about 21 requests per second, and the trained model does not need it: it produces valid JSON 99.9% of the time unconstrained.
| Test | Result |
|---|---|
| Mode A: corrupted model outputs caught (7,212 injected faults, 8 types) | 100% |
| Mode B: stale answers rejected after the document is tampered with | 99.7 to 100% |
| Mode B: tampering flagged when a printed total covers the cell | 65 to 85% |
| Mode C: LedgerMind answers recalling the real filing on counterfactual documents | 4 of 846 (0.5%), no significant accuracy drop |
| Mode C: gpt-oss-120b direct answers on the same documents | 9 of 846 recall the real filing; accuracy drops 3.3 points (p = 0.003) |
| Mode C: Qwen3.8-27B direct answers on the same documents | 9 of 846 recall the real filing; accuracy drops 1.7 points (p = 0.07) |
| Verifier false rejections on correct gold answers | 0.8% |
Mode B has a real limit: about two thirds of tampered cells sit outside any total and cannot be detected from the document alone.
- Specialist against generalist. LedgerMind was trained on FinQA's training split and learned its conventions; the API models see the task cold. The claim is that a small specialist matches a general flagship on this task.
- The pipeline prompt v1 favors the trained models. It names its functions without defining argument order and asks for millions to be converted to units, while FinQA answers stay in document units. A post hoc count puts the cost to gpt-5.5 through the pipeline at about 8 points. Prompt v2 fixes both; the API pipeline numbers above use v1.
- Direct answers are scored at the precision they state (1.64 counts for 1.63657), a rule added after reading the first outputs, where correctly rounded answers were marked wrong. Default scoring of the same answers gives 42.0% for gpt-5.5 and is reported too.
- Cost for the self hosted model assumes a rented H200 at $3.80 per hour (market range $2.00 to $10.60) at full utilization; the runs used a university cluster. API cost is list price times measured tokens. The frontier baselines actually cost $16.52 under a hard $20 cap.
- Determinism. Scored runs use vLLM's batch invariant kernels; two reruns produced identical outputs on all 1,139 questions, which the paired tests rely on.
- Open weight runs used A100 GPUs with a 12,288 token output budget, so their latency and cost are not comparable with the H200 numbers and they are left off the cost chart. Qwen3.8's long thinking made 271 of its requests hit the client's 300 second timeout; those questions were re-asked with a 30 minute timeout and merged in, and 37 answers that still ran out of tokens while thinking count as wrong. On the 400 question sample, Qwen3.8 direct and LedgerMind tie (78.5% each); the full test set has more power and shows the 2.7 point gap.
| Gate | Catches |
|---|---|
| Provenance | a figure not printed where it is cited: fabricated, wrong row, wrong period, wrong sign, wrong scale |
| Plan hygiene | a plan that does not really compute from the evidence, including disguised constants like e1 - e1 + 12 (caught by a sensitivity check) |
| Invariants | dimensional errors: percent combined with dollars, millions with billions, quarterly with annual |
| Integrity | tampered source tables, via line items that no longer sum to their totals |
src/ledgermind/ numbers, schema, document, dsl (executor), verifier, saboteur, data (FinQA),
rewards, training helpers, eval (harness, paired tests, cost), serving
training/ SFT and GRPO scripts, adapter merge, pass rate filter, configs
serving/ vLLM launch script, live demo page
slurm/ cluster jobs: data prep, SFT, GRPO, evaluation, benchmarks, demo
scripts/ dashboard data builder
docs/ DESIGN.md, the dashboard (GitHub Pages), every result file cited here
tests/ unit tests, run in CI on Python 3.11 and 3.12
uv sync
uv run pytest
uv run python -m ledgermind.data.prepare # download, verify and convert FinQA
uv run python -m ledgermind.eval.oracle # gold programs through the executor
# Recompute any paired comparison from the committed per question verdicts
uv run python -m ledgermind.eval.compare --a docs/results/eval/grpo_test_rows.jsonl \
--b docs/results/eval/sft_bi1_test_rows.jsonlTraining, serving and benchmark commands are in docs/DESIGN.md; the cluster jobs that produced every result are in slurm/.
Why I built it. Financial teams want to use language models, but they cannot use a number they cannot audit. I wanted to see whether a small model I could train myself could be both accurate and trustworthy on real financial filings, and to prove it with numbers a skeptical reviewer would accept.
What I built.
- The design. The model proposes, a deterministic engine computes, a verifier decides. The same verifier is the gate on every answer, the reward during reinforcement learning and the scorer in evaluation, so training, serving and measurement can never disagree about what correct means.
- The system, from scratch in Python: a FinQA converter that ties 99.3% of answers to source cells, a safe executor, a four gate verifier, a three mode adversarial tester (the Saboteur), an evaluation harness with paired significance tests, a live demo and a static results dashboard. About 5,100 lines of code and 299 tests, run in CI.
- The training: Qwen3-4B with LoRA, supervised fine tuning and then GRPO with verifiable rewards, on one NVIDIA H200 of Northeastern's research cluster, through resumable Slurm jobs and a pinned CUDA environment.
- The serving: vLLM in bf16 and fp8, with a concurrency sweep and the measured cost of constrained decoding.
- The comparison: five baseline models in nine configurations, including gpt-5.5 and two open models up to 120B parameters, with paid API calls held under a hard $20 budget ($16.52 spent).
What it showed.
- Verified accuracy rose from 20.4% to 74.9%, and fabricated figures fell from 46.6% to 1.7%.
- A 4B model tied gpt-5.5 on the same questions at about 600 times lower cost per query, with every answer auditable.
- The verifier caught 100% of 7,212 deliberately corrupted answers, and on counterfactual filings the model read the edited figures instead of recalling the real ones.
- GRPO turned out to sharpen the model rather than add capability: no gain in greedy accuracy, but 7.1 points more when sampling, with questions solved on every try rising from 552 to 784.
The judgment calls I am proudest of. Several findings would have flattered this project if I had left them alone, and I fixed each one: a scoring bug that undercounted the API models, a prompt that favoured my own model, and request timeouts that hid a stronger baseline. Fixing the last one is how I learned that Qwen3.8-27B beats LedgerMind by 2.7 points when it answers directly, and that result is reported above, not buried. Every number in this repository traces to a committed result file with its job and code version.
What I would do next: refresh the GRPO prompt set during training so later steps keep their signal, add abstention examples so the model refuses when a figure is missing, and extend the pipeline from single pages to retrieval over full filings.
| Resource | Amount |
|---|---|
| OpenAI API (gpt-5.5 and gpt-5.4-mini baselines, including a 20 question pilot) | $16.52, under a hard $20 cap enforced in code |
| NVIDIA H200 GPU time (SFT, GRPO, evaluations, benchmarks, live demo) | 7.8 GPU hours over 27 jobs, about $30 at the $3.80 per hour rental rate used above |
| NVIDIA A100 80GB GPU time (open weight baselines and their retries) | 7.3 GPU hours over 3 jobs |
| Development, tests and the dashboard | a laptop |
I paid for the frontier API calls myself because a comparison against the strongest model available was worth far more to me than the $16.52 it cost; a result that only beats small models proves little. The spending ledger is committed in docs/results/frontier/spend.json.
| Area | What this project uses |
|---|---|
| Post training | Supervised fine tuning (SFT), GRPO, reinforcement learning from verifiable rewards (RLVR, the RLHF family without a human preference model), LoRA and PEFT, reward design, reward hacking defences, pass rate prompt filtering, DAPO loss |
| LLM engineering | Qwen3, gpt-oss, OpenAI API, prompt design and versioning, structured JSON outputs, constrained decoding, reasoning models and reasoning parsers, tool style executors, hallucination and fabrication detection |
| Serving and inference | vLLM, fp8 and bf16 quantization, batch invariant deterministic inference, concurrency benchmarking, latency and throughput, cost per query modelling |
| Evaluation and statistics | Held out test sets, bootstrap confidence intervals, paired McNemar tests, paired bootstrap, adversarial testing (the Saboteur), counterfactual and contamination probes, selective prediction |
| ML infrastructure | PyTorch, Hugging Face Transformers, TRL, PEFT, CUDA 12.9, NVIDIA H200 and A100, Slurm, HPC clusters, resumable training jobs, pinned environments with uv |
| Software engineering | Python 3.12, typed packages, pytest (299 tests), GitHub Actions CI, ruff, Git, Bash and shell scripting, Linux, SSH tunnelling, reproducible pipelines |
| Data and visualisation | FinQA, financial statement tables (10-K), data cleaning and grounding, HTML, CSS and JavaScript, SVG, GitHub Pages dashboards, Playwright |
| Domain | Financial question answering, auditability, provenance and traceability of reported figures |
Apache 2.0. See LICENSE.