The claims gate could not see the preprint, and EUR reconciled as USD

[jeremiahsay]

Two fixes found while reviewing #30, plus the published figures they move.

1. The claims gate could not see the preprint

check_claims.py printed paper ok while checking none of the 53 percentages in paper/main.tex. strip_markup unescapes \% — but only after stripping TeX comments with a bare %.*, which treats the % in 45.7\% as a comment start and deletes the rest of the line. Every figure after the first percentage in a table row, and the tail of the abstract, vanished before claims_in ever saw them.

Two more gaps sat under the same docstring claim of checking "the published study pages, the preprint and the dataset card":

• README.md carries the same headline tables as FINDINGS.md and was not a surface at all.
• --offline silently drops the four live guide pages, then printed the unqualified "every published number". It now names what it skipped.

43% first-named and 93% of answers naming one of the top three had no figure behind them — they were passing only by rounding to the same integer as an unrelated figure. Both now come from aeo/results.json.

Coverage: 64 → 139 claims offline, 249 → 285 online.

2. Two different currencies must never reconcile

Every entry in units.CURRENCY carries a scale of 1.0, because the benchmark holds no exchange rates — an FX rate is a price on a date, not a unit conversion. That made EUR, GBP and USD interchangeable, so a model answering 85.00 EUR per tonne CO2e scored 85 against a USD/tCO2e truth and counted as an answer rather than an unscoreable one. sek: None was already refusing this way by having no scale at all; the rest were reconciling by accident.

parse_unit now records which currency a numerator is, and reconcile refuses a mismatched pair with a reason. Same currency, either spelling, is unaffected.

Three answers were being scored this way, all gemini-3.6-flash, all carbon_pricing.* — EU ETS 85.00, Spain 15.00, Netherlands 51.00 EUR/tCO2e against a USD truth. One of the three was being counted correct.

This is the same bug class as #23. The guard proposed in #30 only covers a currency written before the number, and all three live instances are written after it.

Published figures brought back into line

scoreable          315 -> 312     within_10pct   41.9% -> 42.0%
unscoreable         85 ->  88     within_50pct   78.1% -> 77.9%
correct            132 -> 131     conf. wrong    21.9% -> 22.1%
pct of all 467    28.3% -> 28.1%

Updated in FINDINGS.md, README.md, paper/main.tex (the abstract's range and the results table) and the page/ snapshots. No other model moves and paired.json is untouched.

results/absolute.json fed the pct_of_all column with no script behind it, so a scorer change moved it silently and --check-fresh could not see it. absolute.py now generates it — it reproduced the committed file exactly on the old scorer, model order and all — and check_fresh reruns it.

Also corrected while verifying: the by-category page said 867 pooled scored answers. The true figure was 886 before this change and is 883 after. CLAIM_RE only matches counts with a thousands separator, so the gate never saw it.

Verification

From a clean clone at the branch tip:

verify/test_claims_gate.py	PASS
verify/test_currency_mismatch.py	PASS
verify/figures.py --check-fresh	PASS
verify/dataset.py	PASS
verify/licences.py	PASS
verify/check_claims.py --offline	PASS
compare.py / paired.py / absolute.py rerun	results/ byte-identical

Both new tests go red without their fix — the gate test returns [] instead of three percentages on the old comment ordering.

check_claims.py online fails on guide:accuracy (41.9%, 28.3%), and that is the correct signal: the live pages still carry the old figures. Live post ids are 9316 (accuracy) and 9327 (by-category) — note 9327, not the page/9322-body.html snapshot, which is superseded. That deploy is not in this PR.

Not in scope

Opening separately: unenumerated dollar codes (A$, C$, R$, S$) collapsing to USD, and reconcile not comparing numerator dimensions (USD 5 per kWh against a kg CO2e/kWh truth). Neither has a live hit today.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NRueWxopDXHoWY2dPvsmLG