feat: parametrize assessment year to 2020, 2022, or 2024 by jeffseif · Pull Request #32 · cornerstone-data/luc

jeffseif · 2026-11-10T00:29:22Z

Background

This allows us to produce emissions factors which vary emissions, allocation, and production with assessment year. However, since MAPSPAM and CROPGRIDS have been snapshotted at 2020, the STATISTICAL and JURISDICTIONAL_DIRECT_PROXY allocation are fixed to 2020 as well.

Notes

Due to dataset coverage, we only allow 2020, 2022, and 2024
I broke the changes up so each commit changes only one script, with downstreams pinned to a static year
In order to cover the new assessment years, we updated USDA/NASS/CDL to ingest bands for 2020 through 2025 inclusive. That new version has been fully backfilled
Descals only covers through 2022, so technically it is stale for 2024
After 2020, the STATISTICAL uses 2020's production from MAPSPAM, and GPW livestock and CROPGRIDS stay at 2020
assessment_year: int has been added to the tabular outputs of attribute and trace as a new index
because emit, attribute, and trace now take assessment_year: int as new arguments, we reset their cache versions to 0

Manual testing

✅ tool.measure-drift for (HND, STATISTICAL, 2020)

results shows zero drift

uv run python tools/measure-drift.py --baseline d5f54a2 --iso HND 2>&1 | tee -a drift-2020.log

=== captured to /var/folders/n2/fvx_yb216ggbm2rg7fpq41880000gn/T/measure-drift/d5f54a2b0d2b ===
  target                             baseline rows  head rows  digest
  STATISTICAL-HND                              678        678  differs

=== STATISTICAL-HND: alignment ===

  STATISTICAL-HND: 678 baseline row(s), 678 head row(s), 678 shared

=== STATISTICAL-HND: per-value drift ===

  STATISTICAL-HND: 640 shared row(s) over 32 hardcoded crop(s)
    column                                 max|abs|    max|rel|   >1e-07   >0.001     >0.1  nan     up/dn  sign p
    commodity_hectares                            0   0.000e+00        0        0        0    0       0/0   1.000
    emissions_factor_kgco2e_per_kg                0   0.000e+00        0        0        0    0       0/0   1.000
    emissions_mt                                  0   0.000e+00        0        0        0    0       0/0   1.000
    forest_emissions_mt                           0   0.000e+00        0        0        0    0       0/0   1.000
    grassland_emissions_mt                        0   0.000e+00        0        0        0    0       0/0   1.000
    peatland_commodity_hectares                   0   0.000e+00        0        0        0    0       0/0   1.000
    peatland_conversion_emissions_mt              0   0.000e+00        0        0        0    0       0/0   1.000
    peatland_occupation_emissions_mt              0   0.000e+00        0        0        0    0       0/0   1.000
    production_kg                                 0   0.000e+00        0        0        0    0       0/0   1.000
    yield_kg_per_ha                               0   0.000e+00        0        0        0    0       0/0   1.000

    worst 5 row(s) by relative movement
      crop            admin    column                                   before        after        rel
      83              HND      yield_kg_per_ha                            1129         1129  0.000e+00
      56              HND      yield_kg_per_ha                          1718.3       1718.3  0.000e+00
      27              HND      yield_kg_per_ha                          4402.5       4402.5  0.000e+00
      15              HND      yield_kg_per_ha                          524.35       524.35  0.000e+00
      -201            HND      yield_kg_per_ha                            7000         7000  0.000e+00
    not checked: 38 row(s) over 2 crop(s) outside the hardcoded subset; nothing measurable, 0 NaN flip(s), 5 column(s) NaN throughout

=== STATISTICAL-HND: sum drift ===

  STATISTICAL-HND: 32 vs 32 national row(s) over 32 hardcoded crop(s)
    column                                   baseline           head         delta        rel
    commodity_hectares                    1.76313e+06    1.76313e+06            +0  +0.00e+00
    emissions_mt                          6.49357e+06    6.49357e+06            +0  +0.00e+00
    peatland_commodity_hectares               15477.8        15477.8            +0  +0.00e+00
    peatland_occupation_emissions_mt           577323         577323            +0  +0.00e+00
    production_kg                         9.47199e+09    9.47199e+09            +0  +0.00e+00
    forest_emissions_mt                   5.26072e+06    5.26072e+06            +0  +0.00e+00
    grassland_emissions_mt                     376821         376821            +0  +0.00e+00
    peatland_conversion_emissions_mt           278706         278706            +0  +0.00e+00
    implied_emissions_factor                 0.685555       0.685555            +0  +0.00e+00

=== STATISTICAL-HND: within-side invariants ===
  baseline  2 broken
  head      2 broken

  STATISTICAL-HND: 0 measure(s) moved beyond 1.19e-07

  -- STATISTICAL-HND breaks 'emissions equal the sum of their component columns' -- baseline
     worst 1.000e+00, head worst 1.000e+00. Not introduced by this change
  -- STATISTICAL-HND breaks 'additive columns are non-negative' -- baseline
     peatland_conversion_emissions_mt=1, head peatland_conversion_emissions_mt=1. Not introduced
     by this change

0 of 10 column(s) and 0 of 9 total(s) moved by more than 1.19e-07

✅ jdluc.trace for 2024, compared to 2020

drift from 2020 to 2024 is reasonable

First, generate the EF's for HND for both years under both methodologies:

for YYYY in 2020 2024 ; do
  for METH in STATISTICAL JURISDICTIONAL_DIRECT_PROXY ; do
    uv run python jdluc/trace.py HND --assessment-year=$YYYY --methodology-name=$METH ;
  done ;
done ;

Then, ask claude to compare them:


HND STATISTICAL: national factors, 2020 against 2024, largest moves first
                                              commodity_name  ef_2020  ef_2024  emissions_mt_2020  emissions_mt_2024  production_kg_2020  production_kg_2024  ef_ratio
commodity_id                                                                                                                                                          
289                                              Sesame seed    7.618    20.48          2.692e+04          2.476e+04           3.534e+06           1.209e+06     2.688
122                                           Sweet potatoes    2.355    1.188           2.83e+04          1.919e+04           1.202e+07           1.616e+07    0.5045
125                                           Cassava, fresh    1.088   0.5996          2.501e+04          1.476e+04           2.299e+07           2.462e+07     0.551
116                                                 Potatoes   0.1064  0.07037               2586               1781            2.43e+07            2.53e+07    0.6612
489                            Plantains and cooking bananas   0.4512   0.3092          3.964e+04          2.884e+04           8.785e+07           9.327e+07    0.6853
242                            Groundnuts, excluding shelled   0.8039    1.063              603.9              531.4           7.512e+05           5.001e+05     1.322
254                                           Oil palm fruit   0.4623   0.3521          8.316e+05           7.18e+05           1.799e+09           2.039e+09    0.7617
328                                    Seed cotton, unginned    0.757   0.5885               2155               1811           2.847e+06           3.077e+06    0.7775
656                                            Coffee, green    3.622    2.927          1.095e+06          1.024e+06           3.022e+08           3.498e+08    0.8082
27                                                      Rice   0.6053    0.492           2.82e+04          2.598e+04           4.659e+07            5.28e+07    0.8128
83                                                   Sorghum    1.611    1.317          6.498e+04          4.026e+04           4.034e+07           3.058e+07    0.8173
176                                               Beans, dry    5.515    4.581           5.39e+05          5.098e+05           9.773e+07           1.113e+08    0.8307
56                                              Maize (corn)    1.138   0.9501          6.555e+05          5.756e+05           5.759e+08           6.058e+08    0.8347
156                                               Sugar cane  0.02661  0.02279          1.513e+05           1.25e+05           5.687e+09           5.486e+09    0.8567
486                                                  Bananas  0.09753  0.08696          6.729e+04          5.438e+04           6.899e+08           6.253e+08    0.8916
15                                                     Wheat    37.74    34.73          4.229e+04          4.103e+04            1.12e+06           1.181e+06    0.9203
137                                                     Yams  0.04071  0.03748            0.04746            0.01524                1166               406.6    0.9206
249                                       Coconuts, in shell   0.8939   0.8265          1.291e+04          1.159e+04           1.445e+07           1.402e+07    0.9246
236                                               Soya beans      1.8     1.71               2737               2695           1.521e+06           1.576e+06    0.9501
867           Meat of cattle with the bone, fresh or chilled    45.74    43.53          2.732e+06          2.648e+06           5.974e+07           6.084e+07    0.9516

Totals over every national row, DROPPED included
                                      2020      2024  ratio
emissions_mt                     3.008e+07  3.28e+07   1.09
forest_emissions_mt              5.339e+06 4.503e+06 0.8435
grassland_emissions_mt           3.822e+05 3.742e+05 0.9791
peatland_conversion_emissions_mt 2.821e+05 4.988e+05  1.768
peatland_occupation_emissions_mt 5.896e+05 7.014e+05   1.19
production_kg                    9.472e+09 9.543e+09  1.007
commodity_hectares               1.763e+06 1.872e+06  1.062

Rows with a factor in only one year: 0

HND JURISDICTIONAL_DIRECT_PROXY: national factors, 2020 against 2024, largest moves first
                                                            commodity_name  ef_2020  ef_2024  emissions_mt_2020  emissions_mt_2024  production_kg_2020  production_kg_2024  ef_ratio
commodity_id                                                                                                                                                                        
242                                          Groundnuts, excluding shelled    57.57    19.11               4676               1562           8.122e+04           8.172e+04    0.3319
661                                                            Cocoa beans    7.743    3.699          1.084e+04               7398             1.4e+06               2e+06    0.4777
574                                                             Pineapples  0.09151  0.05095               8275               6660           9.043e+07           1.307e+08    0.5568
122                                                         Sweet potatoes   0.7659   0.4932          1.508e+04          1.309e+04           1.969e+07           2.654e+07     0.644
83                                                                 Sorghum    1.514    2.256          3.292e+04          3.384e+04           2.175e+07             1.5e+07     1.491
426                                                    Carrots and turnips    1.065   0.7172               4990               3396           4.687e+06           4.735e+06    0.6737
687                                               Pepper (Piper spp.), raw    7.607    5.145               4690               3866           6.165e+05           7.513e+05    0.6764
249                                                     Coconuts, in shell   0.4189    0.288               5631               3841           1.344e+07           1.333e+07    0.6876
176                                                             Beans, dry    2.515    1.756          3.218e+05          2.522e+05            1.28e+08           1.437e+08     0.698
401           Chillies and peppers, green (Capsicum spp. and Pimenta spp.)   0.2354   0.1694               7335               5322           3.116e+07           3.142e+07    0.7198
560                                                                 Grapes  0.09196  0.06657              16.85              12.52           1.833e+05           1.881e+05    0.7239
512                                             Other citrus fruit, n.e.c.    2.074    1.509               1803               1352           8.694e+05           8.961e+05    0.7276
125                                                         Cassava, fresh   0.4986    0.367          1.278e+04               9414           2.562e+07           2.565e+07    0.7361
56                                                            Maize (corn)    1.034   0.7709          6.836e+05          5.425e+05           6.612e+08           7.037e+08    0.7457
236                                                             Soya beans   0.3845   0.2916              844.2              639.4           2.196e+06           2.193e+06    0.7583
515                                                                 Apples   0.3229   0.2469              61.16              47.99           1.894e+05           1.943e+05    0.7648
289                                                            Sesame seed    3.629    2.806               4171               3311            1.15e+06            1.18e+06    0.7732
15                                                                   Wheat    3.161    2.447               3954               3134           1.251e+06           1.281e+06     0.774
397                                                 Cucumbers and gherkins  0.05618  0.04368               2265               1845           4.031e+07           4.225e+07    0.7775
406                                                           Green garlic   0.1573   0.1227               30.2              24.66            1.92e+05            2.01e+05    0.7801

Totals over every national row, DROPPED included
                                      2020      2024  ratio
emissions_mt                     3.015e+07 3.286e+07   1.09
forest_emissions_mt              5.361e+06 4.517e+06 0.8426
grassland_emissions_mt           3.862e+05 3.771e+05 0.9765
peatland_conversion_emissions_mt 2.977e+05  5.18e+05   1.74
peatland_occupation_emissions_mt  6.01e+05 7.122e+05  1.185
production_kg                    1.116e+10 1.069e+10 0.9585
commodity_hectares               2.018e+06 2.018e+06      1

Rows with a factor in only one year: 0

…rs of interest

…nt year Have JD proxy adopt this same window -- allowing us to remove the awkward coupling to FAOSTAT freshness another docs update

jeffseif requested a review from blallen November 10, 2026 00:29

jeffseif added this pull request to stack #33 November 10, 2026 00:33

danohu reviewed Nov 10, 2026

View reviewed changes

Comment thread jdluc/trace.py Resolved

danohu reviewed Nov 10, 2026

View reviewed changes

Comment thread jdluc/trace.py Resolved

jeffseif added 15 commits November 10, 2026 09:57

recover dataset version comments

1f43d08

Ingest 2020-2025 for USDA/NASS/CDL so it can cover all assessment yea…

7e7ccae

…rs of interest

Add --assessment-year to jdluc.emit

0c06fd9

Add assessment_year to jurisdictional_direct.workflow

f44e2dc

Add assessment_year to jurisdictional_direct_proxy.workflow

537d336

Add assessment_year to statistical.workflow

8524ecd

Add --assessment-year to jdluc.attribute

7d31cad

Add --assessment-year to jdluc.trace

b0d63c5

change JD production window to 3-year window centered on assessment_year

94a7c5d

Add assessment_year to tabular outputs as a new index column

0d345c0

Ensure tools and validation call with the correct arguments

5ff13ca

update docs

4e48b2d

document datasets which are stale in 2024

ee7ad8f

Inline some functions to shrink the statistical module

52ea440

Move livestock enums and functions from statistical to gpw_livestock

561fb10

jeffseif force-pushed the jeffseif/parametrize-assessment-year branch from 55ab004 to 561fb10 Compare November 10, 2026 18:01

Revert JD production year-window to be 4 years ending at the assessme…

d223e9c

…nt year Have JD proxy adopt this same window -- allowing us to remove the awkward coupling to FAOSTAT freshness another docs update

danohu approved these changes Nov 11, 2026

View reviewed changes

jeffseif merged commit 04033da into main Nov 11, 2026
2 checks passed

jeffseif deleted the jeffseif/parametrize-assessment-year branch November 11, 2026 15:49

jeffseif mentioned this pull request Nov 11, 2026

chore: some quick follow-ups to the assessment-year parametrization #34

Merged

Navigation Menu

Repository navigation

feat: parametrize assessment year to 2020, 2022, or 2024 - #32

feat: parametrize assessment year to 2020, 2022, or 2024#32
jeffseif merged 16 commits into
jeffseif/emit-sparse-formatfrom
jeffseif/parametrize-assessment-year

jeffseif commented Nov 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jeffseif commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Notes

Manual testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jeffseif commented Nov 10, 2026 •

edited

Loading