feat: backtest the forecast against every recorded session, with its sampling bias controlled

[zebraengine]

Problem

PRs #31–#33 were each validated by replaying one or two hand-picked moments from the production DB, and each found its counter-example within the hour. There was no way to score a model change against the whole history; contrib/backtest_derate_amp_control.py replays the controller against forecasts the server recorded, not whether those forecasts were right.

A first cut of this tool then produced a wrong conclusion of its own: scored on the runs that reached a plateau, the model read 3–4 °C optimistic, worst on hot days, and an ambient term fitted to that looked like the fix. The sample was the problem. The current and ambient every run happened at were chosen by the controller and the charger on the strength of this same model:

1. Regulated runs. The charger trims current as the handle nears 65 °C and holds it there — a setpoint, not an equilibrium. 20 of 52 long-enough runs.
2. Censoring. A run whose true plateau was above 65 °C tripped and ended before it could become truth. The optimistic errors that matter never reach the tables. 14 runs.
3. Confounding. The controller caps on hot days, so hot and low current arrive together; an ambient effect and a current-law error wear the same signature.

Code touched

contrib/backtest_forecast.py (new)

• Splits sessions into steady-current runs (same band rule as the live forecast). Truth = plateau of a run held ≥ 3 τ: --truth fit (whole-run exponential, τ free) or --truth last (handle's own last 3 min, model-free); the two bracket.
• Contamination controls: runs whose current sagged > 1.5 % (_current_sag_a / _free_plateau, the fitter's own gate) are never truth and are counted; runs with an alert 40 raised inside them are never truth and are listed with every method's prediction (a prediction under 65 °C is a miss the tables cannot show); every free-running run cut short after ≥ 1 τ — capped, tripped, ended — is scored against its own trajectory projection with the projection's SE (median 0.03 °C), as a proxy truth. The projection needs no ambient and no current law.
• Scores: in-run forecast tick by tick (trajectory vs model basis) by minutes-at-current and scenario; cross-current prediction at every change and session start from each ambient (sensor / previous run's implied / warmer of the two / idle handle) × law (I² / fitted n), scored session left out of the fit. Scenario tags: cold_start, warm_start, step_down, step_up, probe, hot. --verbose, --json.

tests/test_backtest_forecast.py: synthetic history with known truth (cold start then step-down, both ≥ 3 τ) — runs found and classified, plateaus recovered within 1 °C, trajectory converges, every cross-current method within 1.5 °C at the step, start scored from sensor + idle; run splitter on the session-80 ramp and a contactor drop.

docs/thermal-model.md: "Measuring the forecast" — usage, the three contamination channels and what is done about each, and the findings.

No model change. The ambient-term model built on the first (contaminated) pass is parked on wip/ambient-term with the evidence against it in this PR.

Risk

Read-only tool; nothing in the running system changes.

Verification

• uv run pytest: 154 passed (2 new).
• Production DB, 74 sessions, 312 runs. De-censored findings (fit truth / model-free truth):
  • Full-rate cold starts, hot days included: the deployed model is unbiased. Sensor: +0.3 °C median, |med| 0.7, 0 % optimistic by > 2 °C (7 runs); idle proxy: +0.7, |med| 1.0 (18 runs). e.g. s47 at 31 °C ambient: predicted 67.8, projected 67.8.
  • The optimism is a history effect. Step-downs after a hot full-rate run: −2.8 / −3.6 °C from the sensor; −0.7 / −1.4 from the previous run's implied ambient (carries the heat-soaked cable). The two worst clean truths (39.6 A "cold starts" on hot evenings) and the July-heat-wave probes (−7 °C, right after a trip) fit the same story.
  • The deployed configuration is on the right side of it: caps from the implied ambient (right for step-downs), restores from the sensor — at the 6 de-censored step-ups the sensor is unbiased-to-pessimistic (+2.6), and at session 80's own step-up predicted 64.9 vs projected 64.8. fix: work the sustainable current from whichever ambient leaves less headroom #33's "warmer" rule would have cost amps on restores.
  • Fitted exponent beats I² in every de-censored cell below 48 A (step-downs |med| 2.4 vs 3.9 from the implied ambient; probes 2.5 vs 3.7). Keep.
  • Trip table: of 6 real full-rate trips (all July, idle-proxy ambient), the deployed model predicted a trip for 4; the two misses were within 1.5 °C (63.5, 62.9).
• Next, if the history effect is worth chasing: a second, slow time constant for the cable — which needs designed data (probes at one current with a cold cable and a warm one), not more history.

🤖 Generated with Claude Code