Lower-side calibration follow-up: fix P80 coverage gap (0.85 vs 0.90 target)

[ducroq]

What this issue is

Tracker for the post-EXP-014 calibration follow-up explicitly queued in the M4 retrospective (`docs/articles/m4-metric-redesign-story.md` §9, ADR-006 `Consequences`, `docs/hypothesis-log.md`). LightGBM-Quantile's lower-side coverage sits at 0.854 vs the 0.90 target across the full eval log (1728 realised hours, 25 days). The promotion guardrail passed only because both models miss the target — ARF was 0.82, LGBM is 0.85, neither hits 0.90. Same gap inherited from ARF; not made worse by the swap, but unresolved.

Why it matters

• The dashboard claims an 80% predictive interval. If realised prices fall below p10 14% of the time instead of 10%, the interval is mislabelled.
• The EXP-014 promotion criterion deliberately used a one-sided guardrail (not >0.02 worse than ARF) rather than a bilateral target (hit 0.90). Promoting was correct given the constraint, but doesn't make the constraint go away.
• Coverage is bimodal day-to-day: trailing-14 mean 0.76 with 2 days below 0.50 (2026-06-02 hit 0.333). The mean masks the worst days.
• This is the one metric where the model is meaningfully wrong, after the dashboard rendering issues (Dashboard: EnergyZero timestamps shifted ~2h from Augur forecast (timezone mutation bug) #16, Dashboard: lower confidence band clamped at 0 hides negative-price predictions #17) are fixed.

Three candidate approaches

The M4 retrospective and the EXP-012 literature pass each independently surfaced these. They are not mutually exclusive; the natural sequence is to evaluate them one at a time as separate experiments (EXP-015, EXP-016, EXP-017).

A. Horizon-conditioned CQR

Current CQR uses one calibration set across all 72 horizons. The retrospective hypothesises that ARF's EWM band wins pinball-at-p10 at long horizons because its residual-std estimate is horizon-agnostic, while CQR widens uniformly based on whatever the trailing 7-day calibration set happens to contain (often dominated by one regime).

Test: split CQR calibration into the same three horizon groups already used by the LGBM forecaster (h+1..h+6 / h+7..h+24 / h+25..h+72). Per-group `q` widens bands differently per horizon.

Implementation: small change in `ml/shadow/update_shadow.py` + `ml/shadow/conformal.py`. Probably the cheapest experiment.

B. Adaptive Conformal Inference (ACI)

Gibbs & Candès 2021 / 2024 — `q` is adjusted online based on rolling coverage, targeting nominal level by a closed-loop control law. Designed exactly for the situation where exchangeability is mildly violated (which is what we have: regime shifts, daily seasonality, calibration set composition drift).

Implementation: replace fixed-target CQR with adaptive update. Reference implementations exist (`Predictive-Conformal-Intervals` libraries). Slightly more invasive than (A).

C. Train at more quantiles (9 or 19 instead of 3) + canonical twCRPS

The article §9 also flagged that the current 3-quantile training has a structural MQS asymmetry (LGBM's 3-quantile pinball average has a ~2× head start over a point forecast's MAE-as-CRPS-equivalent — see EXP-013 review). Training at 9 or 19 quantiles costs more retraining time but:

• removes the structural asymmetry from any future skill comparison
• enables a properly-implemented Gneiting-Ranjan canonical twCRPS as a descriptive tail metric
• gives more honest CDF estimates for diagnostics

This doesn't directly fix coverage but it removes a methodological caveat that bleeds into how we interpret any coverage fix.

Scope of this tracker

• EXP-015 (most likely first): horizon-conditioned CQR. Pre-commit a hypothesis in `docs/hypothesis-log.md` with target lower-side coverage ≥ 0.88 by horizon group, no group below 0.85, evaluated over a 14-day shadow.
• EXP-016 (if EXP-015 falls short or as parallel candidate): ACI.
• EXP-017 (if either of the above passes): retrain at 9 quantiles, recompute skill comparison + canonical twCRPS, update CLAUDE.md and the article.
• After successful experiment: update `docs/hypothesis-log.md`, append rows to `experiments/registry.jsonl`, update `docs/decisions/006-...` with the new calibration design, update CLAUDE.md known-weakness note.

Each EXP gets its own pre-committed criterion per ADR-007. This issue is the umbrella that ties them together so the work doesn't drift.

Out of scope

• Skill metric work (LGBM already beats ARF on aggregate MAE and weekday peak — that's not the open problem)
• Dashboard rendering (Dashboard: EnergyZero timestamps shifted ~2h from Augur forecast (timezone mutation bug) #16, Dashboard: lower confidence band clamped at 0 hides negative-price predictions #17 already fixed; Dashboard: clarify wholesale vs consumer comparators; verify EnergyZero endpoint really returns all-in pricing #18 separate)
• Cron freshness (Migrate daily orchestration to systemd, fix EDH timing so augur sees fresh data #12 — but worth fixing first, since stale exogenous data may itself bias the calibration set)
• Foundation-model baselines (Evaluate foundation models (Chronos, TabPFN-TS) as baselines / ensemble members #15 — orthogonal; neither Chronos nor TabPFN-TS directly fixes a CQR calibration set)

Dependencies

Soft dependency on #12: fixing exogenous freshness may itself shift coverage measurably (the LGBM training parquet currently sees 24h-stale forecasts). Worth resolving the freshness story before running the calibration experiments so we don't conflate two effects. Not a hard block — could be re-evaluated after.

References

• `docs/articles/m4-metric-redesign-story.md` §9 (the four open questions, of which three apply here)
• `docs/decisions/006-lightgbm-quantile-production-architecture.md` (`Consequences` section flags this)
• `docs/decisions/007-model-promotion-method.md` (the criterion framework for any EXP-015..017)
• `docs/hypothesis-log.md` (where pre-commits will land)
• `docs/metric-redesign-literature-review.md` (CQR / ACI references)
• `memory/MEMORY.md` → `arf-retired.md` (records the gap as 'queued as next experiment')
• Current numbers: `ml/shadow/eval_log.jsonl`, `ml/models/shadow/shadow_state.json` calibration_history

🤖 Generated with Claude Code

[ducroq]

Forward pointer: Phase-1 features (TTF gas + NL genmix) as a follow-on after EXP-015..017

Calendar revisit fired 2026-06-05 for the original ARF-era Phase-1 plan (memory: project_new_features_rewarmup.md, calendar reminder from before the 2026-05-29 deferral). The plan as written is stale — warmup_p1 / backtest_p1 / online_features.py kwargs are ARF tooling, and the ADR-005 ≥2 EUR/MWh MAE gate was set against ARF's error distribution. But the data precondition is now met, so it's worth banking the scoping note here so it isn't re-derived from scratch later.

Data precondition (HAN energydatahub on sadalsuud, verified 2026-06-05 after git pull):

• _generation_mix.json: 71 daily files, 2026-03-27 → 2026-06-04 (~70 days, ≥60 threshold)
• Memory's "late May 2026" estimate held

What ports from feat/new-features-ttf-genmix (parsers are model-agnostic):

• ml/data/consolidate.py:parse_market_history_file — daily TTF, next-day 00:00 UTC anchor
• ml/data/consolidate.py:parse_generation_mix_file — actuals, +24h shift required to avoid leakage (only _actual fields in the source; collector runs yesterday→today, so same-hour = leakage)

What needs new work for LGBM:

• Port the genmix feature consumption into ml/shadow/features_pandas.py (NOT online_features.py — that's ARF-side and not reused). Apply the same +24h shift on the LGBM side.
• New promotion criterion — the ADR-005 MAE gate doesn't apply. Likely a paired DM on MAE + a calibration-not-worse guardrail, analogous to EXP-014's design (ADR-007).

Why this sits behind EXP-015..017, not in parallel:
Running a feature ablation while the calibration set is mislabelled by ~4 percentage points would conflate two effects — improved features could be credited for what is actually a CQR fix, or vice-versa. Same reasoning as the soft #12 dependency already noted. So: resolve calibration first, then evaluate Phase-1-for-LGBM as a separate experiment (call it EXP-018 if it gets that far). Updating the deferred status in `project_new_features_rewarmup.md` to "data ready, queued behind augur#19" as part of `/curate`.

Out of scope for this issue (#19 is calibration only) — recording here so the trail doesn't get lost.

[ducroq]

EXP-015 designed, pre-committed, and run (offline Stage 1) — 2026-06-12. Verdict: IMPLEMENT = False, redirect to ACI (EXP-016).

Design redirect (baseline evidence, computed before the treatment)

The horizon-conditioned-CQR sketch in this issue assumed the deficit concentrates at certain horizons. Baseline from calibration_history (30 vintages, 2112 realised hours — not eval_log, whose rows mix 24/48/72h vintages and have permanent holes at 06-08/06-10, see #14):

	h1-6	h7-24	h25-72	pooled
lower-side	0.828	0.837	0.833	0.834
upper-side	0.878	0.876	0.892	0.886

Lower-side deficit is flat across horizons but asymmetric across sides — the single symmetric q from max(p10−y, y−p90) is the wrong tool. Also found while reading the implementation: compute_cqr_q conformalizes the already-widened stored bands rather than the raw model quantiles (a feedback loop; raws are stored since 2026-05-29 making the clean version possible).

So EXP-015 became: per-side split-conformal on raw sorted quantiles (q_lo and q_hi calibrated separately at 0.90/side, same trailing-7d window, no zero-floor). Pre-commit in docs/hypothesis-log.md (commit bcc3e78) before the replay ran, per ADR-007.

Stage-1 result (scripts/exp015_replay_cqr.py, 8 evaluable vintages / 528 rows, 06-02 → 06-11)

	lower	upper	winkler	med width
incumbent (stored bands)	0.778	0.903	146.4	84.4
treatment (per-side)	0.826	0.862	150.0	92.9

• Criterion 1 (≥ incumbent+0.03): PASS (+0.048)
• Criterion 2 (≥ 0.86): FAIL (0.826)
• Criterion 3 (upper ∈ [0.85, 0.97]): PASS — Criterion 4 (Winkler ≤ +5%): PASS (+2.5%)

Per-vintage lower-side tells the real story: treatment ≥ 0.847 (mostly ≥ 0.90) on every vintage from 06-04 on, but 0.375 / 0.708 on the regime-shift days 06-02/06-03. The pre-identified Alternative-1 failure mode fired in the offline data: static trailing-window calibration cannot adapt to regime shifts. That's exactly ACI's (Gibbs & Candès 2021) target.

The descriptive companion also closes the door on the original sketch: horizon-grouped calibration makes short horizons worse (h1-6 lower 0.646) — per-group calibration sets are too thin at 7 days.

Outcome + next

• EXP-015 logged as parked in experiments/registry.jsonl — the per-side score decomposition is right (+5 points at +2.5% Winkler) but insufficient alone; it carries into EXP-016's design.
• EXP-015: horizon-conditioned CQR → run as per-side CQR (redirected by baseline evidence), parked
• Next: EXP-016 = ACI with per-side nonconformity scores (closed-loop q adjustment targeting 0.90/side), fresh pre-commit per ADR-007. The 06-02-style days are the test it must pass.

Caveat for whoever designs EXP-016: the evaluable offline window is short (raws exist only since 05-29) and is the hard recent stretch — incumbent sits at 0.778 there vs 0.834 over 30 vintages. Every passing day adds 72 raw-bearing rows.

[ducroq]

EXP-016 (per-side ACI) designed, pre-committed, and run — 2026-06-12. Verdict: IMPLEMENT = False. The calibration-layer arc is now conclusive: the gap lives in the raw quantiles → EXP-017.

Same evaluable window and criteria as EXP-015 (8 vintages / 528 rows, pre-commit in 440b0b6 before the replay ran).

Stage-1 result (scripts/exp016_replay_aci.py, γ=0.10)

	lower	upper	winkler	med width
incumbent	0.778	0.903	146.4	84.4
EXP-015 static per-side	0.826	0.862	150.0	92.9
EXP-016 per-side ACI	0.852	0.860	163.9	109.4

• Criterion 1 (≥ incumbent+0.03): PASS (+0.074, best yet)
• Criterion 2 (≥ 0.86): FAIL at 0.852 — squarely in the pre-committed Alternative-1 near-miss zone [0.84, 0.86)
• Criterion 3: PASS — Criterion 4 (Winkler ≤ +5%): FAIL at +12% — Alternative 3

Why this is conclusive rather than just another miss

1. ACI works as advertised, where it can. Every vintage from 06-04 onward: lower ≥ 0.903 (mostly 1.000). The method fixed everything after the shift.
2. It cannot fix the first shift days. 06-02 = 0.375, 06-03 = 0.667. The α-trace shows α_lo had drifted narrower (0.10 → 0.126) during the calm warm-up — by design, ACI reacts only after misses arrive.
3. The ceiling is γ-independent: γ ∈ {0.05, 0.10, 0.20} all land at lower ≈ 0.85. The 69 hours missed on 06-02/06-03 alone cap any day-granularity calibration layer at ≈ 0.87 on this window — and approaching it pegged α at the 0.005 clip with q_lo ≈ 55–61 EUR/MWh (median width +30% → the Winkler trip).
4. No oscillation (lag-1 err autocorr +0.96), so the γ-retune path (Alternative 2) is ruled out.

Alternatives 1 and 3 — pre-committed to point at the same place — both fired: q10_raw is biased high entering regime shifts, and no conformal correction can widen a band for a shift it hasn't seen yet.

Status of the umbrella

• EXP-015: horizon-conditioned CQR → per-side CQR (redirected by baseline), parked — fixes side asymmetry (+0.048 at +2.5% Winkler)
• EXP-016: ACI, parked — fixes post-shift adaptation, γ-independent first-day ceiling
• EXP-017: 9-quantile training — now carries the arc. Better raw quantiles (denser tail information at τ=0.05/0.10/0.15…) so the base band entering a shift is honest to begin with. Needs a walk-forward backtest over a window that includes 06-01..06-03 (raw stored quantiles don't exist at 9 taus historically), fresh pre-commit per ADR-007. Both parked layers (per-side scores, adaptive α) are candidates to re-add on top if the raws improve.

Registry: EXP-015 + EXP-016 appended (experiments/registry.jsonl, 16 entries). Hypothesis-log entries resolved same-day with full traces.

[ducroq]

Update 2026-08-20: the per-side asymmetry has flipped to the upper side. Re-computing calibration_history coverage:

• Overall (2088 rows): lower 0.893 / upper 0.785 / band 0.679
• August: lower 0.916 / upper 0.755 / band 0.671
• vs the original framing (lower 0.834 / upper 0.886 through 2026-06-11)

So the weak side is now the upper tail, not the lower. Band coverage remains ~0.67 vs the 0.80 target. EXP-015/016 stayed parked; EXP-017 (9-quantile training) is still the queued next step, and a parallel feature-expansion track is now framed in docs/hypothesis-log.md (2026-08-20) — the model uses 24 features from 4 columns while 7+ collected series never reach consolidate.py. Deferred to next session.