Repository navigation
Turn on trade margin receivers by supply, and pin v1.0 to the CI rebuild v1.0.0_fd271dc (#1194, #1270) - #1302
Conversation
handle_negative_matrix_values and handle_negative_vector_values set negatives to zero in place. derive_U_set passes them the cached output of derive_U_with_negatives, so once derive_U_set ran, every later reader of derive_U_with_negatives saw the clipped frame. Both now return a copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…1270) BEA records some sales as negative intermediate purchases: industries selling used goods (S00402) and federal nondefense government selling farm commodities from its stocks (1111A0 bought by S00600). derive_U_set sets them to zero, so the Use row no longer added up to q. The Use-side final demand now carries them. Only cells whose Use value is negative move; a cell negative only because imports exceed use still shows as an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ses (#1270) derive_negative_purchase_sales returns, by commodity, the domestic Use cells that are negative in the Use table itself (used-goods sales, government sales from stocks). The Use-side final demand and a new published U column, F09000 "Sales recorded as negative intermediate purchases", carry them, so the NIPA-sourced final-demand columns are unchanged and each published U row still sums to output. F09000 is kept out of FINAL_DEMANDS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derive_Vnorm_scrap_corrected divides each industry's market shares by 1 - s, so A q counts its purchases at U / (1 - s) and the Use row no longer adds up to q. derive_scrap_correction returns the difference by commodity, -sum_j U[i, j] s_j / (1 - s_j), and derive_y_dom_from_use_final_demand includes it. The scrap share moves to _scrap_share, shared with derive_Vnorm_scrap_corrected (no change to A). On v1.0.0_8365fa4 it clears 562HAZ (1.0% and 1.6% in 2022-23) and the iron and steel gap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… does (#1270) ACCOUNTING_SECTORS (scrap, used goods, noncomparable imports, the rest-of-world adjustment) have no output of their own, so the check can't hold for them. compare_output_vs_leontief_x_demand still computes L @ y on every sector and compares the rest. The xfail reason now quotes the one 2024 failure, 334118; 2017-2023 pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It offset a double count in A and B instead of fixing it. #1293 fixes the scrap correction itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rap (#1293) USEEIO divides market shares by 1 - s, with s scrap's share of output including scrap. The Cornerstone Make table has no scrap (S00401 maps with weight 0), so x is already net of it, and U / x and E / x already land on the other products. Dividing again counted scrap twice in A and B. derive_Vnorm_scrap_corrected now returns the plain market shares, which gives exactly USEEIO's A and B. B @ q now equals E (4,808.491 in 2024; it was 4,812.5). N falls for scrap-making industries: remediation -11.1%, metal stampings -5.9%, hazardous waste -2.4%, landfills -1.5%; median -0.17%. New integration test: B @ q conserves E. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1270) Census domestic exports put the computers family (3341) at 1.28 times BEA's 2017 exports, and terminals (334118) export more than they produce, so its imports exceed its use. Each 3341 commodity's export cell is now its published 2017 F04000 cell moved by its own Census growth, the #763 construction for imports. Terminals' exports fall to 0.74-0.92 of output in 2017-2023 (were 1.01-1.13 in 2022-2024); 2024 sits at 1.00 because output falls while Census exports grow. Needs a rebuild to reach the tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l_demand Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review on #1292 (Ben): when use_ghg_year_x_in_B or the deflation branch puts BEA gross output in B's x, that output includes scrap, so the 1 / (1 - s) raise was doing real work there. derive_B_via_vnorm now divides the market shares by 1 - s on that path only, with s = scrap / output including scrap (scrap_share_of_output; electricity children share a parent's scrap by output). Both paths are now USEEIO-exact: v0.3 B matches E / (x_GO (1 - s)) V_n to 9e-16, and v1.0 B q = E (4,808.491). Also fixes the stale Vnorm column-sum comment in inflation_helpers_usa. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… steps test (#1293) Review on #1292 (Ben): industry_corresp gave each of 562000's seven waste children 1/7 of its scrap, so remediation's s was 7.28% against the parent's 0.72%, and v0.3 B ran 7.0% high on 562213. scrap_share_of_output now shares a split BEA industry's scrap across its Cornerstone children by output, so every child gets the parent's share (0.722% for all seven; nothing outside 562* moves). test_useeio_b_adjust_divide_transform_steps now mocks scrap_share_of_output (I1 0.1, I2 0) and expects 0.08 / 0.9, which pins the gross-output branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rowth (#1194, #1270) Review on #1295 (Ben): Mo's 325411 finding is the same Census-vs-BEA gap. Census puts preparations on 325411 ($7.3bn of 2017 exports against $0.08bn published) and 325413, so a row's own Census line is no mover: own-growth anchoring would put biologics (325414) at $52.7bn of 2024 exports against $48.0bn of output. FAMILY_GROWTH_EXPORT_FAMILIES = {'3254'} holds BEA's 2017 split of the family and moves it by the family's Census growth (1.81 by 2024). All four rows stay below output in 2017-2024 (325414 peaks at 0.87). anchored_exports applies both modes. Also from the review: rebuild_nowcast.yml pinned to ubuntu-24.04, as #1297 did for CI, and the rebuild script's anchor comment covers exports. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a test mock (#1270) Review on #1290 (Ben): - get_U's reaggregation branch sums the negative-purchase sales on the electricity children into 221100, as the final-demand block does, instead of reindexing them onto the 405 axis. - The F09000 description says its values are negative. - test_get_U_commodity_axis_is_parent_not_children mocks derive_negative_purchase_sales (on the 407 axis), so it no longer runs on real data. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
….0 configs (#1194) Step 4 spread each year's trade margin over the receiving goods by frozen 2017 shares, so goods whose supply grew slower than total trade got margins above their supply (motor homes 336213 at 1.17 of supply in 2024; Phi 0.19). #1260 added the flag; this turns it on in the nine v1.0 configs and adds it to the rebuild's flag check. Needs a rebuild. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run 43349311057 rebuilt 2017-2024 on main 83dfd6b (the #1289-#1295 stack) plus trade_margin_receivers_follow_supply. Uploaded to GCS, nothing overwritten. The nine v1.0 configs, the GHG FBS upstream pin, both 2024 pins (new sha256), the B time-series assessment and the config tests move from 8365fa4 to fd271dc. q = L_dom y passes in every year 2017-2024, so test_domestic_L_y_from_use_final_demand_equals_q drops its xfail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Sequence looks right: #1289-#1295 are on Pin plumbing checks out: nine configs + rebuild On the rebuild numbers: medicinal ingredients Phi back near the release (0.713 vs 0.723) looks like #1295 doing its job; motor homes 0.136 vs 0.009 on Dropping the xfail on uv run pytest -m eeio_integration bedrock/utils/validation/__tests__/test_eeio_diagnostics.py::test_domestic_L_y_from_use_final_demand_equals_q -v(That covers canonical 2024 only; the all-years claim stays on the rebuild notes.) After merge: cut the snapshot on the merge commit (canonical + year series), |
bl-young
left a comment
bl-young
left a comment
There was a problem hiding this comment.
Sequence looks right: #1289-#1295 are on main at 83dfd6ba, rebuild 43349311057 ran on fd271dc (= that tip + the flag), and the pin / nine configs / flag check / xfail drop match that vintage. Snapshot cut + key bump after merge (as in #1284) is the right next step; leave #1296 for after.
Pin plumbing checks out: nine configs + rebuild FLAGS + config tests all carry trade_margin_receivers_follow_supply and v1.0.0_fd271dc, 2024 SUT/GHG sha256 pins moved, upstream FBS pin and B assessment follow. Diff vs main is only that surface (17 files).
On the rebuild numbers: medicinal ingredients Phi back near the release (0.713 vs 0.723) looks like #1295 doing its job; motor homes 0.136 vs 0.009 on 8365fa4 is a real fix for #1194, with the 2023 step still the open seam. Margins above supply on apparel etc. being the same goods as 2017 is a fair read. Cars (#1296) remaining is fine to call out and defer.
Dropping the xfail on q = L_dom y matches the rebuild notes (every year 2017-2024). Default CI skips eeio_integration, so green CI won't re-run that test; fine to take the rebuild as the check. If you want a belt-and-suspenders pass later:
uv run pytest -m eeio_integration bedrock/utils/validation/__tests__/test_eeio_diagnostics.py::test_domestic_L_y_from_use_final_demand_equals_q -v(That covers canonical 2024 only; the all-years claim stays on the rebuild notes.)
After merge: cut the snapshot on the merge commit (canonical + year series), compare_snapshots 2024 vs unsuffixed, then the key-bump PR. Re-check #1194 motor homes on the 2023 member once the year series exists. Optional: B @ q == E / GHG total on fd271dc if you want a scrap+margins sanity pass before the bump.
Part of #1194 and #1270. After this merges, the snapshot is cut on the merge commit and the key bumped (as in #1284).
What this does
trade_margin_receivers_follow_supply(feat(nowcast): let trade margin receivers follow their own supply #1260) in the nine v1.0 configs and adds it to the rebuild's flag check. Step 4 spread each year's trade margin over the receiving goods by frozen 2017 shares. Goods whose supply grew more slowly than total trade then got margins above their supply: motor homes (336213) reached 1.17 of output plus imports in 2024. With the flag on, each good's 2017 share moves with its own supply.v1.0.0_fd271dc: run 43349311057, 16 jobs. It is main at83dfd6ba(the Clip negatives on a copy, not the cached input (#1270) #1289-Anchor the computers and pharmaceutical exports on BEA's 2017 cells (#1270, #1194) #1295 stack) plus the flag. The repinned files are the nine configs, the GHG flow-by-sector upstream pin, both 2024 pins with new sha256 values, the B time-series assessment and the config tests.test_domestic_L_y_from_use_final_demand_equals_q, because it now passes (Make and Use disagree on commodity output, q = L_dom y fails for a small number of commodities #1270).Uploaded to GCS, nothing overwritten (272 files): 16 balanced supply-use tables, 128 Make, Use, Import and Margins tables, 64 Energy and GHG flow-by-sector tables, the 16 Census trade flow-by-activity tables and the 32 trade flow-by-sector tables, each with its metadata.
Checks on the rebuild
q = L_dom ypasses in every year 2017-2024. On8365fa4, 2024 failed on terminals, and earlier years on oilseeds, hazardous waste and used goods.Phi (producers' over purchasers' value) and trade margins:
336213325411On
8365fa4, 2024 had motor homes at Phi 0.009 (margin 1.17 of output plus imports, now 0.83), medicinal ingredients at 0.047, and 30 goods with margins above supply. The release had 0.230 and 0.723. Medicinal ingredients is fixed by #1295's export anchoring. Motor homes is much better, but still drops at 2023, which looks like the AIES seam.A margin above output plus imports isn't wrong in itself: retail margins on apparel and textile furnishings can exceed producer value. The largest cases (farm machinery 2.44, apparel 2.12 in 2024) are the same goods as in 2017.
Imports above use:
334118) exports are 0.67 and 0.73 of output in 2022 and 2024 (Anchor the computers and pharmaceutical exports on BEA's 2017 cells (#1270, #1194) #1295), and imports no longer exceed use.336111) still export more than they produce (1.15 and 1.11). Imports exceed use by $4.3bn and $2.5bn, down from $7.9bn and $4.2bn. That is Make cars (336111) and light trucks (336112) mirror each other until they are merged #1296.GHG flow-by-sector total: 2024 is 4,954.43 Mt, unchanged, as expected: the fixes move emissions between sectors.
After merge
🤖 Generated with Claude Code