Add calibration package checkpointing, target config, and hyperparameter CLI#538

Draft

baogorek wants to merge 73 commits intomainfrom

calibration-pipeline-improvements

Commits on Mar 5, 2026

Add calibration package checkpointing, target config, and hyperparameter CLI

authored and
juaristi22
committed
Ignore all calibration run outputs in storage/calibration/

authored and
juaristi22
committed
Add --lambda-l0 to Modal runner, fix load_dataset dict handling

authored and
juaristi22
committed
Add --package-path support to Modal runner

authored and
juaristi22
committed
Add --log-freq for per-epoch calibration logging, fix output dir

authored and
juaristi22
committed
Create log directory before writing calibration log

authored and
juaristi22
committed
Add debug logging for CLI args and command in package path

authored and
juaristi22
committed
Fix chunked epoch display and rename Modal output files

authored and
juaristi22
committed
Replace per-clone Microsimulation with per-state precomputation

authored and
juaristi22
committed
Add Modal Volume support and fix CUDA OOM fragmentation

authored and
juaristi22
committed
Restrict targets to age demographics only for debugging

authored and
juaristi22
committed
Add include mode to target config, switch to age-only

authored and
juaristi22
committed
Switch target config to finest-grain include (~18K targets)

authored and
juaristi22
committed
Fix at-large district geoid mismatch (7 districts had 0 estimates)

baogorek
authored and
juaristi22
committed
Add CLI package validator, drop impossible roth_ira_contributions target

authored and
juaristi22
committed
Add population-based initial weights for L0 calibration

baogorek
authored and
juaristi22
committed
Drop inflated dollar targets, add ACA PTC, save full package

authored and
juaristi22
committed
Remove redundant --puf-dataset flag, add national targets

authored and
juaristi22
committed
fixing the stacked dataset builder

baogorek
authored and
juaristi22
committed
Derive cds_ordered from cd_geoid array instead of database query

baogorek
authored and
juaristi22
committed
Update notebook outputs from successful calibration pipeline run

authored and
juaristi22
committed
Fix takeup draw ordering mismatch between matrix builder and stacked builder

authored and
juaristi22
committed
checkpoint with aca_ptc randomness working

baogorek
authored and
juaristi22
committed
verify script

baogorek
authored and
juaristi22
committed
Prevent clone-to-CD collisions in geography assignment

authored and
juaristi22
committed
checkpoint

baogorek
authored and
juaristi22
committed
Fix cross-state cache pollution in matrix builder precomputation

authored and
juaristi22
committed
bens work on feb 25

baogorek
authored and
juaristi22
committed
Selective county-level precomputation via COUNTY_DEPENDENT_VARS

juaristi22
and
claude
committed
minor fixes
juaristi22
committed
small optimizations
juaristi22
committed
Parallelize clone loop in build_matrix() via ProcessPoolExecutor

juaristi22
and
claude
committed
Migrate from changelog_entry.yaml to towncrier fragments (#550)

authored and
juaristi22
committed
Add end-to-end test for calibration database build pipeline (#556)

authored and
juaristi22
committed
Parallelize clone loop in build_matrix() via ProcessPoolExecutor

juaristi22
and
claude
committed
add target config

baogorek
authored and
juaristi22
committed
Reorganize calibration modules from local_area_calibration to calibration/

authored and
juaristi22
committed
Fix modal run command to specify ::main entrypoint

authored and
juaristi22
committed
Fix worker stdout pollution breaking JSON result parsing

authored and
juaristi22
committed
Add volume-based verification after worker builds

authored and
juaristi22
committed
Fix at-large district GEOID round-trip conversion

authored and
juaristi22
committed
Always fresh-download calibration inputs, clear stale builds

authored and
juaristi22
committed
Normalize at-large district naming: 00 and 98 both map to 01

authored and
juaristi22
committed
Enable takeup re-randomization in stacked dataset H5 builds

authored and
juaristi22
committed
Streamline calibration pipeline: rename, upload, auto-trigger

authored and
juaristi22
committed
Add make pipeline: data → upload → calibrate → stage in one command

authored and
juaristi22
committed
documentation

baogorek
authored and
juaristi22
committed
flag

baogorek
authored and
juaristi22
committed
changes to remote calibration runner

baogorek
authored and
juaristi22
committed
Script cleanup, validation gating workflow, sanity checks, docs

authored and
juaristi22
committed
Use source-imputed dataset for H5 staging, upload it from calibration

authored and
juaristi22
committed
Fix sanity_checks H5 key lookup for group/period structure

authored and
juaristi22
committed
Fix stage-h5s: add ::main entrypoint to modal run

authored and
juaristi22
committed
Add validation job to publish workflow, promote target, fix OOM

authored and
juaristi22
committed
after county acknowledgement

baogorek
authored and
juaristi22
committed
Add provenance tracking, fix takeup rerandomization order, improve Modal runner

authored and
juaristi22
committed
Add national H5 pipeline, remove --prebuilt-matrices flag

authored and
juaristi22
committed
Fix JSON serialization crash: __version__ resolved to module

authored and
juaristi22
committed
Update upload_local_area_file docstring to list all subdirectories

authored and
juaristi22
committed
Age-only target config for national H5 experiment, fix national builder

authored and
juaristi22
committed
late night work

baogorek
authored and
juaristi22
committed
calibrated populaiton counts logging
juaristi22
committed
saving column sums
juaristi22
committed
removing debugging logs
juaristi22
committed
unify build_h5 with wrappers
juaristi22
committed
removing wrappers
juaristi22
committed
Full targets calibration + national staging support

authored and
juaristi22
committed
Add back save_geo_labels/load_geo_labels lost in rebase

authored and
juaristi22
committed
Add missing json import for save_geo_labels/load_geo_labels

authored and
juaristi22
committed
Add volume reload before checking national H5 exists

authored and
juaristi22
committed
Consolidate takeup draws to shared compute_block_takeup_for_entities

juaristi22
and
claude
committed
Consolidate calibration pipeline duplications

juaristi22
and
claude
committed
Regenerate uv.lock after rebase on main

juaristi22
and
claude
committed