Split out of #18 so the two do not collide — this one is in score.py, #18 is in units.py.
candidates() captures the unit text that trails a number. Currency does not work that way: models write USD 5 /tCO2e, with the currency in front. So the extractor sees the unit as /tCO2e, which _clean() turns into per tco2e — and parse_unit() tests for " per " with a leading space, so a string that starts with "per" never splits. The whole thing is read as a numerator of tonnes CO2e, and the answer is dropped as unscoreable.
python3 -c "
import score, units
print(score.candidates('Best estimate: about USD 5 /tCO2e.'))
print(units.reconcile(5, '/tCO2e', 'USD/tCO2e'))
print(units.reconcile(5, 'USD /tCO2e','USD/tCO2e'))
"[(5.0, '/tCO2e.')]
(None, 'denominator mismatch (None vs mass)') <- as it stands
(5.0, None) <- with the currency reattached
The reconciliation is already correct. The currency simply never reaches it.
Scope
12 answers, all in carbon_pricing.*, where the truth unit is USD/tCO2e. Every one is a case where the model was in the right units all along.
What to change
candidates() in score.py — recognise a currency token immediately before a number and carry it into the unit string. USD, US$, $, EUR, €, GBP, £ at minimum; CURRENCY in units.py is the existing list.
An alternative fix is to make parse_unit() treat a leading per as a denominator with an implicit numerator. I tried that and it does not work — the numerator is genuinely absent from the trailing text, so there is nothing to scale by, and parse_unit correctly returns None. The information is in the prefix; it has to be picked up there.
What you should see
|
unscoreable |
within_10pct |
| as committed |
110 |
45.7 |
| with this fix |
98 |
~45.9 |
Report both. A fix that recovers twelve answers and barely moves the headline is more evidence the unscoreable pile is not hiding a bias, which is the robustness claim in FINDINGS.md.
Watch out for
A model quoting a different currency — NT$300 per tCO2e, NOK 1,000 per tCO2e, EUR 65-75 per tCO2e all appear in these same answers, usually alongside a USD figure in the same sentence. Do not invent an exchange rate to convert them. If the answer also states a USD figure, that is the one to score; if it states only a foreign currency, unscoreable is the honest outcome and it should stay that way.
Then check python3 verify/figures.py --check-fresh is still green.
Split out of #18 so the two do not collide — this one is in
score.py, #18 is inunits.py.candidates()captures the unit text that trails a number. Currency does not work that way: models writeUSD 5 /tCO2e, with the currency in front. So the extractor sees the unit as/tCO2e, which_clean()turns intoper tco2e— andparse_unit()tests for" per "with a leading space, so a string that starts with "per" never splits. The whole thing is read as a numerator of tonnes CO2e, and the answer is dropped as unscoreable.The reconciliation is already correct. The currency simply never reaches it.
Scope
12 answers, all in
carbon_pricing.*, where the truth unit isUSD/tCO2e. Every one is a case where the model was in the right units all along.What to change
candidates()inscore.py— recognise a currency token immediately before a number and carry it into the unit string.USD,US$,$,EUR,€,GBP,£at minimum;CURRENCYinunits.pyis the existing list.An alternative fix is to make
parse_unit()treat a leadingperas a denominator with an implicit numerator. I tried that and it does not work — the numerator is genuinely absent from the trailing text, so there is nothing to scale by, andparse_unitcorrectly returnsNone. The information is in the prefix; it has to be picked up there.What you should see
Report both. A fix that recovers twelve answers and barely moves the headline is more evidence the unscoreable pile is not hiding a bias, which is the robustness claim in
FINDINGS.md.Watch out for
A model quoting a different currency —
NT$300 per tCO2e,NOK 1,000 per tCO2e,EUR 65-75 per tCO2eall appear in these same answers, usually alongside a USD figure in the same sentence. Do not invent an exchange rate to convert them. If the answer also states a USD figure, that is the one to score; if it states only a foreign currency, unscoreable is the honest outcome and it should stay that way.Then check
python3 verify/figures.py --check-freshis still green.