[WesIngwersen]

Part of #1270 (to-dos 1 and 2). Stacked on #1289, which this needs: without it, derive_U_with_negatives returns the clipped table once derive_U_set has run, and the new column comes out empty.

What was wrong

BEA records some sales as negative intermediate purchases. The model sets negative intermediate cells to zero (derive_U_set), so these sales dropped out of the Use rows, and the rows no longer added up to commodity output. That was most of the remaining q = L_dom y failure in #1270:

• Used and secondhand goods (S00402): industries selling used goods book negative purchases of them: air transport (−$0.70bn in 2024), performing arts (−$0.58bn), water transport (−$0.51bn) and general government (S00500). It failed every year, by $0.3-2.0bn.
• Oilseed farming (1111A0): federal nondefense general government (S00600) books negative purchases, from selling farm commodities out of government stocks: −$0.23bn in 2017, −$1.27bn in 2022, −$0.51bn in 2024. It failed in six of eight years, by 1-3%.

The same cells are in BEA's published 2017 Use table after redefinitions, the basis our tables are on. For used goods, the air transport cell comes from BEA's redefinitions: it is +$0.01bn before them and −$0.30bn after. Our 2017 used-goods row runs about 1.31 times BEA's after-redefinitions row on every cell (−$0.93bn against −$0.71bn), which points to our 2017 rebuild scaling the whole row. The Import cells are zero (BEA has $9M on the 1111A0 × S00600 cell). 1111A0 imports overall are right: $997M against $963M published in 2017.

The change

• derive_negative_purchase_sales() returns these sales by commodity. They are the domestic cells whose Use value is itself negative. A cell that is negative only because imports exceed use, like the terminals cells (334118), is an error rather than a sale. It stays out, so it still shows in the test and in the Warn when a row's imports exceed its positive use, the case that makes domestic use negative (#1270) #1282 warning.
• derive_y_dom_from_use_final_demand adds them. That is the final demand the Make and Use disagree on commodity output, q = L_dom y fails for a small number of commodities #1270 test and the national balance diagnostic (Test the national balance on the Use table's final demand, not y back-computed from q (#1077) #1281) read.
• The published U (get_U) gets a new final-demand column, F09000 "Sales recorded as negative intermediate purchases (values are negative)", holding the same values, and it is listed in the final-demand metadata sheet. With it, each published U row still sums to output, and users who rebuild the model from the published matrices get q. The NIPA-based final-demand columns are unchanged. F09000 is not in FINAL_DEMANDS, because the BEA-sourced Y has no such column. F08 and F09 are unused in BEA's tables.

Effect

The model doesn't change. derive_U_set, A, y_nab and B are untouched, and ytot doesn't include the new column.

q = L_dom y, on v1.0.0_8365fa4, every year:

Year	Before: failing sectors, gap	After
2017	S00402, $0.93bn	S00402, $0.00bn (q = 0)
2018	S00402, 1111A0, $1.29bn	S00402, $0.00bn
2019	S00402, 1111A0, $1.37bn	S00402, $0.01bn
2020	S00402, $0.93bn	S00402, $0.00bn
2021	1111A0, S00402, $1.64bn	S00402, $0.00bn
2022	1111A0, S00402, 562HAZ, $2.71bn	562HAZ $0.21bn (1.0%), S00402
2023	S00402, 1111A0, 562HAZ, $2.92bn	562HAZ $0.35bn (1.6%), S00402
2024	S00402, 1111A0, 334118, $2.93bn	334118 $0.30bn (4.4%), S00402

• 1111A0 passes in every year.
• S00402 is left with $4-6M on a commodity with no output. Its relative error has no denominator, so it still counts as failing.
• 562HAZ (the scrap correction) and 334118 (imports above use) are separate to-dos on Make and Use disagree on commodity output, q = L_dom y fails for a small number of commodities #1270.

The test stays xfail, and its reason quotes the new 2024 numbers.

Checks

black, ruff, mypy, the publish tests and the national balance tests (9 passed), locally.

🤖 Generated with Claude Code

[WesIngwersen]

FYI @jvendries @MoLi7 This created a new final demand column with a specific residual which eliminates the non-sense of negatives in the intermediate transactions for used goods and but is needed in y to achieve the national accounting balance.

[bl-young]

Reproduced locally on v1.0.0_8365fa4 for 2017, 2022, 2023 and 2024. With F09000 in y, 1111A0 passes every year, and the remaining failures are exactly the ones the description lists (562HAZ 2022-23, 334118 2024, S00402 ~$0). F09000 totals -$1.19bn (2017) to -$2.59bn (2024), and none of it sits on 2211* rows.

A few things, none blocking:

1. 2017 S00402 vs BEA. The description says these cells are in BEA's published 2017 Use table. That holds for 1111A0 (-$0.225bn ours vs -$0.240bn BEA), but our S00402 row is -$0.93bn against -$0.38bn in BEA's Use_SUT (before redefinitions). The largest cell, air transport 481000 at -$0.39bn, isn't in BEA's table at all, and water and performing arts are 1.3-1.5x BEA. None of it is imports. Do you know whether that comes from the redefinitions or the 2017 rebuild?

2. Sign. F09000 holds negative values under the name "Sales recorded as negative intermediate purchases". Could the metadata name or description say the values are negative, so users summing U's final demand don't flip it?

3. Reaggregation branch of get_U. The sales are on the 407 index and are reindexed onto 405, so any sale on a G/T/D child row would trip the guard. That's fine today. Summing children into 221100, as derive_disagg_Ytot_reaggregated does, would be sturdier.

4. test_get_U_commodity_axis_is_parent_not_children mocks every get_U input except derive_negative_purchase_sales(), which now runs on real data inside that unit test. Worth monkeypatching too.

[WesIngwersen]

Thanks @bl-young. All four, in 146bdb06 (merged up into #1292 and #1295):

1. 2017 S00402 against BEA. Our tables are after redefinitions, so BEA's after-redefinitions Use table is the right comparison, and there the air transport cell is in BEA's own table:

   S00402 × buyer, 2017, $bn	BEA, before redef.	BEA, after redef.	ours	ours / BEA after
   481000 air transport	+0.012	−0.303	−0.394	1.30
   483000 water transport	−0.183	−0.183	−0.241	1.32
   711100 performing arts	−0.155	−0.155	−0.205	1.32
   S00500 general government	−0.049	−0.049	−0.067	1.37
   negative cells, total	−0.405	−0.708	−0.927	1.31

   So the air transport cell comes from BEA's redefinitions. The rest of the gap is a uniform factor of about 1.31 on the whole row. That looks like our 2017 rebuild scaling the used-goods row, not anything cell by cell, and it isn't imports. S00402 is an accounting sector the output check now skips (Stop counting scrap twice in the emission factors (#1293) #1292), so I've noted it in the description rather than chasing it here.

2. Sign: the F09000 description is now "Sales recorded as negative intermediate purchases (values are negative)", and the constant's comment says so.

3. Reaggregation: get_U now sums the sales on the electricity children into 221100 with reaggregate_electricity_children_y, as the final-demand block does, instead of reindexing them onto the 405 axis.

4. Test: test_get_U_commodity_axis_is_parent_not_children now mocks derive_negative_purchase_sales, on the 407 axis that derive_U_with_negatives uses when disaggregation is on. The reaggregation and publish tests pass (11).

🤖 Generated with Claude Code