[jeffseif]

Background

This sets the stage for the assessment year to be parametrized. First, emit is refactored from a dense to a sparse representation of emissions over time (i.e., the undiscounted emission and its year are output and amortization is applied downstream. This changes results only within float epsilon. Second, the linear amortization weights vary annually instead of within each 5-year plateau. This changes results but was not directly measured.

Notes

1. emit and downstream caches technically should change, but since we are about to add assessment year we decided to keep them as-is and avoid two changes
2. emit now outputs 9 bands instead of 16. Runtime and data sizes don't reduce very much because the dense and sparse representations compress down the same 🤷

Manual testing

1. Before changing the weights, I ran the drift tool for HND and showed that changes were within floating epsilon

`tools.measure-drift` output

=== captured to /tmp/emit-sparse-format-hnd ===
  target                             baseline rows  head rows  digest
  STATISTICAL-HND                              678        678  differs

=== STATISTICAL-HND: alignment ===

  STATISTICAL-HND: 678 baseline row(s), 678 head row(s), 678 shared

=== STATISTICAL-HND: per-value drift ===

  STATISTICAL-HND: 640 shared row(s) over 32 hardcoded crop(s)
    column                                 max|abs|    max|rel|   >1e-07   >0.001     >0.1  nan     up/dn  sign p
    commodity_hectares                     0.036621   2.065e-07       19        0        0    0     65/40   0.019
    emissions_factor_kgco2e_per_kg       2.4213e-05   2.507e-07       60        0        0    0     88/96   0.606
    emissions_mt                          0.0063362   2.235e-07       35        0        0    0     73/54   0.110
    forest_emissions_mt                    0.043823   1.918e-07       32        0        0    0     74/63   0.393
    grassland_emissions_mt                0.0019531   1.846e-07       38        0        0    0     85/46   0.001
    peatland_commodity_hectares                   0   0.000e+00        0        0        0    0       0/0   1.000
    peatland_conversion_emissions_mt      0.0009768   1.538e-07       13        0        0    0     33/40   0.483
    peatland_occupation_emissions_mt              0   0.000e+00        0        0        0    0       0/0   1.000
    production_kg                            66.406   1.870e-07       24        0        0    0     67/44   0.036
    yield_kg_per_ha                        0.014311   2.065e-07       40        0        0    0     82/86   0.817

    worst 5 row(s) by relative movement
      crop            admin    column                                   before        after        rel
      289             HND017   emissions_factor_kgco2e_per_kg           96.599       96.599  2.507e-07
      116             HND001   emissions_factor_kgco2e_per_kg           2.9243       2.9243  2.235e-07
      116             HND001   emissions_mt                              17.07        17.07  2.235e-07
      486             HND018   emissions_factor_kgco2e_per_kg         0.057027     0.057027  2.214e-07
      254             HND016   emissions_factor_kgco2e_per_kg          0.13143      0.13143  2.192e-07
    not checked: 38 row(s) over 2 crop(s) outside the hardcoded subset; worst 1.333e+00 (peatland_conversion_emissions_mt -2/HND004), 0 NaN flip(s), 5 column(s) NaN throughout

=== STATISTICAL-HND: sum drift ===

  STATISTICAL-HND: 32 vs 32 national row(s) over 32 hardcoded crop(s)
    column                                   baseline           head         delta        rel
    commodity_hectares                    1.76313e+06    1.76313e+06      +0.05976  +3.39e-08
    emissions_mt                          6.56315e+06    6.56315e+06      +0.01255  +1.91e-09
    peatland_commodity_hectares               15477.8        15477.8            +0  +0.00e+00
    peatland_occupation_emissions_mt           577323         577323            +0  +0.00e+00
    production_kg                         9.47199e+09    9.47199e+09        +133.6  +1.41e-08
    forest_emissions_mt                   5.33641e+06    5.33641e+06     +0.006342  +1.19e-09
    grassland_emissions_mt                     373766         373766     +0.004646  +1.24e-08
    peatland_conversion_emissions_mt           275654         275654     -0.001272  -4.61e-09
    implied_emissions_factor                 0.692901       0.692901    -8.448e-09  -1.22e-08

=== STATISTICAL-HND: within-side invariants ===
  baseline  2 broken
  head      2 broken

  STATISTICAL-HND: 8 measure(s) moved beyond 1.19e-07

  -- STATISTICAL-HND commodity_hectares moved in one direction more than chance allows: 65 up
     against 40 down (p=0.019). Summation noise is symmetric, so this is a systematic shift even
     where the magnitudes are small
  -- STATISTICAL-HND grassland_emissions_mt moved in one direction more than chance allows: 85
     up against 46 down (p=0.001). Summation noise is symmetric, so this is a systematic shift
     even where the magnitudes are small
  -- STATISTICAL-HND production_kg moved in one direction more than chance allows: 67 up against
     44 down (p=0.036). Summation noise is symmetric, so this is a systematic shift even where
     the magnitudes are small
  -- 2 crop(s) outside the checked set moved where nothing checked them: worst 1.333e+00
     (peatland_conversion_emissions_mt -2/HND004), plus 0 NaN flip(s). Target.commodity_ids
     should cover every crop the capture holds, so this means they diverged
  -- STATISTICAL-HND breaks 'emissions equal the sum of their component columns' -- baseline
     worst 1.000e+00, head worst 1.000e+00. Not introduced by this change
  -- STATISTICAL-HND breaks 'additive columns are non-negative' -- baseline
     peatland_conversion_emissions_mt=2, head peatland_conversion_emissions_mt=1. Not introduced
     by this change

8 of 10 column(s) and 0 of 9 total(s) moved by more than 1.19e-07

2. I ran JDP on both commits as well and observed that outputs were bit-identical

[danohu]

Very cool!

One non-blocking comment. On the bit-identical check -- just confirming that this wouldn't have just been returning the stale cache data, since we didn't change the version/key. But that would show up soon anyway.