Aggregate calibration: a single trained model under-predicts totals ~10x with ~0% interval coverage — does pooling fix it? #530

Measured 2026-09-30 on real data while the 300-lesson forecasting run was training. Not a delivery defect — it predates and outlives the FAO delivery work, and nothing here blocks that run.

The question that prompted it

Sum the served tower-map surface over a conflict-affected country and you get much less than is empirically observed. Conflict is rare so a low expected value is expected, and extremes should live in the tail rather than the point estimate — but it also just feels like bad calibration.

Half of that is arithmetic and half is real, and they need separating.

The arithmetic half — summing MAPs undershoots by construction

tower_point is a mode-like summary; its own docstring says it returns 0 "when the zero atom dominates, by density". For a zero-inflated heavy-tailed variable the mode is ~0 almost everywhere while the mean is not. Expectation is linear, modes are not:

E[sum] = sum E[x]        always
mode(sum) != sum mode(x) for skewed data

So summing MAP surfaces undershoots even for a perfectly calibrated model and says nothing about calibration. The right object is the distribution of the total, from summing the draws.

The real half — measured, and it is not reassuring

blue_stranger, validation partition, trained model, 468 origin-months, summing draws (not MAPs):

The observed total sits above essentially the entire predictive distribution in nearly every origin-month, and the predicted total is 6–15% of observed.

This is the timid prophet, quantified. This repository already records that MSLE rewards timid under-prediction, and that MCR (ŷ/ȳ) exists as the magnitude guardrail precisely because of it — the standing rule is never to rank on MSLE alone. MCR of 0.06–0.15 is that failure mode in the numbers rather than in principle.

The dependence finding, which compounds it

dependence_ratio = Var_d(total) / sum_cells Var_d(cell)

At ≈1.0, cells behave as independent marginals across draws rather than coherent joint scenarios. Summing ~13,110 (or ~64,818) independent cells washes out the tail: variances add instead of covariances accumulating, so every aggregate interval is too narrow regardless of how good the per-cell marginals are.

Independently confirmed on the forecasting side: violet_visitor's 300-lesson output gives dependence ratios of 0.992 / 1.107 / 0.997.

Note these are two different faults. A narrow interval is a variance problem; MCR 0.09 is a location problem, and a wider interval does not fix a 10× gap.

THE OPEN QUESTION — does pooling fix it?

This is the whole issue. Everything above is a single model with 16 draws. The delivered product is rusty_bucket: 8 models × 16 = 128 draws. Pooling adds between-model variation — switching model shifts many cells together, which is itself a source of dependence — so pooled coverage could be far better.

If coverage jumps toward 90% when pooled: pooling is doing its job and single models simply must not be read alone. Worth stating somewhere load-bearing.

If it stays near 0%: there is a genuine calibration finding that should be understood before these forecasts are used for anything, and it is independent of the delivery machinery.

What to run

rusty_bucket -r calibration -f -sa      # or validation — pooling only, no training
python -m tools.prereg.aggregate_calibration <pooled_dir> --actuals <actuals.parquet>

It needs a partition with observations. The 300-lesson run in flight is forecasting — months 561–596 have no observed values yet, so it can report the dependence ratio on 128 pooled draws but cannot give PIT, coverage or MCR. That was my own error in suggesting it; recorded so nobody repeats it.

Blocker today: there is no pooled calibration output on the laptop — ensembles/rusty_bucket/data/generated/ holds only log files. Either the pooled calibration run never happened on this roster or its output never came home. Per-model validation draws for at least one model are local, so a hand-pooled check is possible if all eight are present.

Tooling already in place

tools/prereg/aggregate_calibration.py (PR #529) computes all of the above with the falsifiers F1–F4 pre-registered in its docstring, and refuses to invent a comparison when actuals are absent. Eight tests against synthetic posteriors with known answers, including a control that a well-centred model trips nothing.

Caveats on the numbers above

• one model, 16 draws, not the ensemble
• lr_* values are log1p-scaled, so a "total" is a sum of log1p, not a fatality count. Predicted and observed are in the same space so PIT/coverage/MCR are valid, but 287 vs 3,112 is not "fatalities"
• validation partition, Africa+ME region (13,110 cells), from a 2026-09-07 run