EXP-018 candidate: HDD/CDD as long-horizon demand features (h+25..h+72)

[ducroq]

Context

energydatahub publishes `demand_weather_forecast.json` daily from `OpenMeteoWeatherCollector`: 11 NL/DE/BE population centers, 7-day horizon (extensible to 16), with pre-computed Heating Degree Days (HDD) and Cooling Degree Days (CDD) alongside raw temperature/humidity/wind/cloud. The file is collected today but augur does not consume it.

augur#20 framed the wire-up as a dashboard task. This issue is the model-feature framing, which is the higher-leverage angle and was missed in the original triage. Filed 2026-06-05 after a session-end observation about the horizon mismatch.

The horizon mismatch (why this is structural, not cosmetic)

Signal	Horizon coverage	Used by augur?
ENTSO-E NL load forecast (current demand-side feature)	~h+1..h+24 (occasionally h+48)	Yes
LightGBM-Quantile forecast horizon	h+1..h+72 (3 groups: 1-6, 7-24, 25-72)	n/a — this is the model's job
Open-Meteo demand_weather (HDD/CDD)	h+1..h+384 (16 days)	No (not in features_pandas.py)

For the h+25..h+72 range — half the model's predictive horizon — the load forecast is structurally unavailable. The feature builder forward-fills or NaN-fills the load column past h+24; either way it's a degraded input. HDD/CDD at population centers is the only demand-side signal that actually covers this horizon.

Why this likely connects to known weaknesses

augur#19 (lower-side calibration) — the coverage gap is worst at long horizons; trailing-14 day-to-day coverage averages 0.76 with worst day 0.33. Part of this may be demand-side blindness at h+25..h+72, not pure CQR mechanics. A clean ablation test (with vs without HDD/CDD) at the long-horizon group would isolate how much of #19 is calibration mechanics vs missing-feature signal.

augur#12 (cron→systemd freshness) — load forecasts are 24h stale today because of orchestration ordering. Open-Meteo forecasts don't have this problem because Open-Meteo is queried fresh every cron run. Adding HDD/CDD also marginally reduces dependence on the staleness-prone load feature.

Concrete experiment scope (EXP-018 candidate)

Hypothesis (pre-commit): adding HDD/CDD at NL population-weighted centers as LightGBM features improves MAE and/or coverage specifically at the h+25..h+72 horizon group, with no degradation at h+1..h+24.

Method (loose, to be tightened when pre-committed in `docs/hypothesis-log.md`):

1. Add `demand_weather_forecast.json` to `decrypt_data_cached.py:DATA_FILES` and the energydatahub→sadalsuud sync
2. Extend `ml/data/consolidate.py` to merge HDD/CDD into the training parquet (population-weighted across the 11 centers, or per-NL-center, design call)
3. Extend `ml/shadow/features_pandas.py` with `hdd` and `cdd` columns (apply same +24h leakage shift as genmix if forecast vintage matters here)
4. Retrain LightGBM and run a paired Diebold-Mariano on `|y - p50_w_HDD| - |y - p50_baseline|` at the h+25..h+72 horizon group (HAC bandwidth = max_h - 1 per ADR-007)
5. Calibration guardrail: coverage at h+25..h+72 not worse than baseline by more than 0.02

Criterion (pre-commit, ADR-007 style):

• Skill: paired DM at h+25..h+72, mean diff < 0 AND one-sided p < 0.10
• Guardrail: h+25..h+72 lower-side coverage not >0.02 worse than baseline
• Stability: same direction at h+1..h+24 (must not degrade short horizon)

Sequencing & dependencies

Hard dependency: augur#12 (freshness fix) — must land first so the load-staleness confound doesn't contaminate the experiment.

Soft dependency: augur#19 EXP-015..017 (calibration follow-on) — running EXP-018 against an uncalibrated baseline mixes two effects (added features improving things vs calibration improving things). Cleanest to land at least EXP-015 (horizon-conditioned CQR) first.

Therefore: EXP-018 is the natural candidate after #12 lands AND the first augur#19 calibration experiment lands, whichever order they happen in.

Out of scope

• Dashboard chart for demand-side weather — see augur#20 (parked there; this issue does NOT supersede Wire demand_weather_forecast.json into augur (consume + visualise) #20, they are independent angles on the same data source)
• Adding genmix/TTF features (`feat/new-features-ttf-genmix`) — sequenced separately under augur#19's umbrella per the 2026-06-05 comment thread
• Foundation-model baselines (Chronos / TabPFN-TS) — augur#15

References

• `docs/decisions/006-lightgbm-quantile-production-architecture.md` — current feature set (24 features, no HDD/CDD)
• `docs/decisions/007-model-promotion-method.md` — pre-committed criterion framework
• `docs/hypothesis-log.md` — where the EXP-018 pre-commit lands when scheduled
• augur#19, augur#12, augur#20 — sibling tracker issues
• `ml/shadow/features_pandas.py` — feature builder to extend

🤖 Generated with Claude Code