[WesIngwersen]

Closes #1293. Part of #1270. Stacked on #1290, which is stacked on #1289.

This PR changes model results. Emission factors fall for industries that produce scrap. After merge, a new snapshot has to be cut on the merge commit and the snapshot key bumped, as in #1284. Until then, the snapshot tests on this branch fail.

Why

Scrap is not a commodity in the model. When an industry produces scrap alongside its main products, all of its purchases and emissions should be charged to those main products.

USEEIO does this in one step. It computes each industry's output including its scrap. Then it raises the industry's share of its main products by its scrap share. A mill with $10 of scrap in $100 of output has its shares raised by 1/0.9.

With s the scrap share of output including scrap and V_n = V q̂⁻¹ the market shares, USEEIO's purchases matrix is

A = U x̂_incl⁻¹ V_n (1 − ŝ)⁻¹ = U x̂_excl⁻¹ V_n

and B = (E x̂_incl⁻¹) V_n (1 − ŝ)⁻¹ = E x̂_excl⁻¹ V_n likewise. Each industry's purchases and emissions are divided by its output net of scrap.

Our model already removes scrap earlier. S00401 maps with weight 0 in the commodity correspondence, so scrap is dropped from the Make table, and the scrap that industries buy is dropped from the Use table, before industry output is calculated. So x is already x_excl. For example, iron and steel's x is $109.175bn, which is BEA's $109.358bn less its $0.183bn of scrap. U x̂⁻¹ V_n is therefore already USEEIO's A. derive_Vnorm_scrap_corrected then divided by 1 − s as well, with s = scrap / x_excl, so scrap was counted twice:

A_before = U x̂_excl⁻¹ V_n (1 − ŝ)⁻¹, which overstates each industry's column by s / (1 − s).

Example: a mill makes $90 of steel and $10 of scrap, and buys $20 of electricity. All $20 should fall on the steel, which is 0.222 of electricity per dollar of steel.

	Industry output	Scrap share	Electricity per $ of steel
USEEIO	$100	10%	20/100, raised by 1/0.9 = 0.222
This model before the fix	$90	11%	20/90, raised by 1/0.889 = 0.25

The extra applies to both the purchases matrix (A) and the direct emission factors (B).

What changes

• derive_Vnorm_scrap_corrected now returns the plain market shares, V_n = V q̂⁻¹, without the scrap raise. That gives the same A and B as USEEIO. Its docstring explains why. The function keeps its name for now, because nine diagnostics import it.
• Where B's x is BEA gross output, the correction stays (Ben's review). With use_ghg_year_x_in_B or the deflation branch, which the six 2025_usa_cornerstone_v0_3* configs use, x is BEA gross output, and that includes scrap. On that path derive_B_via_vnorm still divides the market shares by 1 − s, with s = scrap / output including scrap (scrap_share_of_output). Both paths are now USEEIO-exact: v0.3 B matches E / (x_GO (1 − ŝ)) V_n to 9e-16, and v1.0 B q = E.
• New integration test, test_b_times_q_conserves_emissions: B q must sum to E, the direct emission factors times commodity output adding up to total emissions. With the plain market shares, B q = E x̂⁻¹ V_n q = E x̂⁻¹ x = E. The existing check of life-cycle emissions against the inventory could not see this error, because it applies the same market shares to both sides of the comparison.
• The output check skips BEA's accounting sectors, as USEEIO 2 does. The check is q = L_dom y: domestic output equals the Leontief inverse times final demand. The skipped sectors are scrap, used and secondhand goods, noncomparable imports and the rest-of-world adjustment. They have no output of their own, so the check can't hold for them. Only used goods (S00402) is in the Cornerstone commodity list today.

An earlier version of this PR offset the double count inside the output check instead of fixing it. That version is reverted.

Results, 2024, compared with the current snapshot (7533da28)

Emissions now add up. B q sums to 4,808.5 Mt CO2e, equal to the inventory total E. Before the fix it summed to 4,812.5 Mt, 4.0 Mt more than the inventory.

Total emission factors fall for scrap-making industries:

Sector	Direct factor	Total factor
562213 remediation services	−11.1%	−11.1%
332119 metal crowns, closures and stampings	−5.8%	−5.9%
562HAZ hazardous waste management	−1.0%	−2.4%
332600 spring and wire products	−2.0%	−2.3%
562920 materials recovery facilities	−2.3%	−2.2%
332999 other fabricated metal products	−1.9%	−2.1%
562212 landfills	−1.5%	−1.5%

281 of the 406 sectors move by more than 0.1%. Across all sectors, the median change in the total factor is −0.17%. Weighted by commodity output, the change is −0.20%. Commodity output itself doesn't change.

Remediation moves most because the waste sectors are split from one BEA industry, 562000, and the correspondence gave each child an equal seventh of its scrap. Remediation's small output therefore carried an 11% scrap share. In v1.0, scrap shares no longer enter the factors at all.

On the gross-output path (the v0.3 configs), scrap shares still enter B. A split BEA industry now shares its scrap across its children by output (scrap_share_of_output), so all seven waste children get 562000's own share, 0.722%. Before that, remediation's share was 7.28%. Nothing outside 562* moves on that path.

q = L_dom y, together with #1290:

• 2017 to 2023 pass.
• 2024 fails only on computer terminals (334118), whose imports exceed its use. That is Make and Use disagree on commodity output, q = L_dom y fails for a small number of commodities #1270, to-do 4.
• Hazardous waste, which failed in 2022 and 2023, now passes.

Checks

Run locally: black, ruff, mypy, the diagnostics unit tests, the new emissions test and the life-cycle emissions test (2 passed, 2 expected failures).

🤖 Generated with Claude Code

[WesIngwersen]

Back to draft: the scrap correction here offsets an error in A, so it isn't right for our model.

USEEIO divides market shares by 1 − s, with industry output x including scrap. Then A q equals the Use row exactly. In our model, scrap is mapped out of the Make table before x is computed (S00401 has weight 0). For example, iron and steel's x is $109.175bn, which is BEA's $109.358bn less its $0.183bn of scrap. So U / x already charges all of an industry's inputs to its other products, and dividing by 1 − s charges them again.

A simple case: a mill makes $90 of steel and $10 of scrap and buys $20 of electricity.

• USEEIO: 20/100 × 1/0.9 = 0.222 per $ of steel, and A q = $20.
• Ours: 20/90 × 1/0.889 = 0.25, and A q = $22.50.

The extra $2.50 is 20 × s/(1 − s), which is what derive_scrap_correction subtracts. So this PR makes the check pass by cancelling an overstated A.

A second problem: a parent's scrap is copied in full onto each child. Remediation (562213) and hazardous waste (562HAZ) each carry all $0.339bn of 562000's scrap, which is why remediation's scrap share reads 18%. The electricity children get the same treatment (scrap_fraction.loc[code] = parent_scrap).

The right fix is in A: compute s against output including scrap, and split a parent's scrap across its children by output. That changes the factors for scrap-making industries and needs a new snapshot. The accounting-sector exemption in this PR is unaffected and can stay.

🤖 Generated with Claude Code

[WesIngwersen]

Correction to my last comment: the waste children's scrap is not copied. industry_corresp splits 562000's scrap equally across its seven children, 1/7 each, and total scrap after the mapping equals BEA's ($13.68bn in 2024). The equal split is crude, but it doesn't double count. The electricity branch that assigns the parent's scrap to each child is off in v1.0, and electricity has no scrap. The double count in A, and in B, is the real problem. It's filed separately.

🤖 Generated with Claude Code

[WesIngwersen]

The double count is #1293. Once it's fixed, derive_scrap_correction comes out of this PR and only the accounting-sector exemption stays.

[bl-young]

The double count is real. I reproduced the leak on v1.0: B q - E is +3.6 Mt (2017), +4.3 (2022), +6.7 (2023) and +4.0 (2024), and 0 in every year without the raise. With #1290, 562HAZ passes in 2022 and 2023, so the output check is down to S00402 (~$10M on q = 0) in 2017, 2022 and 2023. It also removes the duplicated parent scrap on the waste children, which is why 562213 moves 11%.

One change requested: the branch where B's x is BEA gross output. derive_B_via_vnorm uses derive_x_after_redefinition() when use_ghg_year_x_in_B is on (apply_io_year_adjustments: True). That is all six 2025_usa_cornerstone_v0_3* configs, which are live in release_v0_3_progression, the electricity ladders and test_sef_phi_wiring. BEA gross output includes scrap, so on that branch the 1/(1-s) raise was doing real work, and the docstring's "x is already output without scrap" doesn't hold there.

Against a USEEIO-exact B on v0.3, E / (x_GO (1 - s_incl)) V_n:

• main: +0.030% in total, mostly the waste children (562213 +7.7%)
• this PR: -0.045% in total, with fabricated metals short (332119 -5.8%, 332600 / 332999 -2.0%, 332114 -1.4%)

Suggest dividing by 1 - s_incl (scrap over BEA output including scrap) on the GO branch only. That makes both paths exact. Or, if v0.3 drift is acceptable, say so in Impact.

Smaller items:

• inflation_helpers_usa.py (~L427) says the scrap correction leaves V_norm column sums above 1; that's no longer true.
• derive_Vnorm_scrap_corrected no longer corrects for scrap; a rename follow-up would help.
• CI hasn't run: every job failed at setup (Python 3.11.13 not found for Ubuntu 26.04). It needs a rerun once the runner is fixed, then the snapshot cut and key bump.

[WesIngwersen]

Thanks @bl-young, you're right about the gross-output branch. Fixed in the latest commit on this branch:

• derive_B_via_vnorm divides the market shares by 1 − s on that path only. That covers use_ghg_year_x_in_B and the deflation branch, where x is BEA gross output. Here s is scrap over output including scrap (scrap_share_of_output). The electricity children now share a parent's scrap by output instead of copying it; electricity has no scrap today, so that only guards the future.
• Both paths are now USEEIO-exact:
  • 2025_usa_cornerstone_v0_3 and ..._electricity_disaggregation: B matches E / (x_GO (1 − s)) V_n to 9e-16. The largest s values are 562213 7.3%, 332119 6.1% and 332999 2.1%.
  • 2026_usa_cornerstone_v1_0: B q = E (4,808.491), unchanged.
• The inflation_helpers_usa comment now says the plain market shares' columns sum to 1, so the renormalisation is only a guard.
• The rename of derive_Vnorm_scrap_corrected stays a follow-up, since nine diagnostics import it.
• CI is green now that main pins Ubuntu 24.04 (ci: pin ubuntu-24.04 so setup-python can resolve 3.11.13 #1297). The snapshot cut and key bump come after merge.

🤖 Generated with Claude Code

[bl-young]

Thanks, the gross-output branch looks right, and keeping v1.0 on the plain shares is the correct split. Two things on 9ecd027:

1. CI isn't green yet. test fails on test_useeio_b_steps.py::test_useeio_b_adjust_divide_transform_steps with ValueError: matrices are not aligned (run 43325181336). The test runs the deflation branch on a toy I1/I2 model, so it now hits the new Vnorm.divide(1 - scrap_share_of_output()). scrap_share_of_output() isn't mocked, returns the real Cornerstone index, and the divide unions the two indexes. Monkeypatching it to a toy s, for example I1 0.1 and I2 0, and expecting 0.08 / 0.9 would also pin the new behavior.

2. Waste children still split scrap equally. industry_corresp gives each of 562000's seven children 1/7 of its $0.83bn of scrap, and s divides that by each child's own output. On v0.3:

   Sector	s (branch)	s (parent's share)
   562213 remediation	7.28%	0.72%
   562920 MRFs	1.92%	0.72%
   562212 landfills	1.25%	0.72%
   562111 collection	0.24%	0.72%

   That leaves v0.3 B high by 7.0% on 562213, 0.9% on 562920 and 0.5% on 562212, and +0.028% in total. Everything outside 562* is identical. The 9e-16 match is against B built from the same s, so it can't see this. Sharing the parent's scrap by output, as the electricity branch now does, gives every child the parent's 0.72%. That's what USEEIO's waste disaggregation does with the scrap column. It only affects the gross-output path; v1.0 is unaffected.

Minor: the Results section still says "With this fix, scrap shares no longer enter the factors at all", which holds for v1.0 but not v0.3.

[WesIngwersen]

Thanks @bl-young. Both fixed in 6a9083e8, and you're right that CI wasn't green; I was wrong to say it was.

1. test_useeio_b_adjust_divide_transform_steps now mocks scrap_share_of_output (I1 0.1, I2 0) and expects 0.08 / 0.9, as you suggested, so it pins the gross-output branch. It passes locally.
2. Waste children: scrap_share_of_output now shares a split BEA industry's scrap across its Cornerstone children by output, not by the correspondence's equal weights. On 2025_usa_cornerstone_v0_3, all seven 562* children are at 0.722%, which equals 562000's scrap over its own Make row in BEA's table, checked independently of the function. Nothing outside 562* changes. The electricity branch uses the same rule. v1.0 doesn't use the shares, so it is unaffected.

I also fixed the Results line: scrap shares no longer enter the factors in v1.0, but they still do on the v0.3 path.

🤖 Generated with Claude Code

[bl-young]

Checked 6a9083e8 on 2025_usa_cornerstone_v0_3: all seven 562* children are at 0.722%, every industry's s matches its BEA parent's scrap over its Make row (max difference 1e-7), mapped scrap totals BEA's $10.761bn, and every correspondence column of the new weights sums to 1. The B-steps test now pins the gross-output branch, and CI is green. v1.0 stays on the plain shares with B q = E. Approving; the snapshot cut and key bump follow merge.