Repository navigation
Description
This follows on from your two backtests on davidusb-geek/emhass#1341: the P10 bias one, and the spread one where 23 % of hours fell below P10 instead of 10 % and the member spread didn't help. I think the code explains a good part of both, so I've written up where the numbers come from. It's a suggestion more than a bug report, since it changes what the sensors publish. Line numbers are at 17f99ac.
What happens
learn()builds the ratios as daily actual / day-ahead ensemble (engine.py#L356-L362). They are daily totals, so the hourly errors inside a day mostly cancel out before they get there.quantiles_from_ratiostakes the 5th, 17th, 50th and 95th percentiles of those daily ratios for p10, p25, p50 and p90 (#L61-L68). The docstring already says the levels differ from the labels.run()publishesP10 = P50 * q10 / q50for every slot of every day until the next learn (#L523-L525). So one daily multiplier is used as an hourly band. Single hours scatter more than daily totals, so more hours fall below it than the daily level suggests. That would fit your 23 %, but I can't say how much of it: in-sample ratios (next point) and a few whole days like 4 November push the same way. The share of days whose total fell below P10 over the same period would tell them apart. It's also why the EMHASS quantile bias came out as a flat derate: blending P50 toward a fixed fraction of itself is just a smaller P50.- The ensemble
Ein those ratios is rebuilt with tonight's calibration (#L348-L350), which was just fitted on the same hours, so the ratios are in sample. Your comment on the level factor (#L367-L369) describes the same trap for k. I can't size this one: my synthetic test below can't show it.
P50 isn't affected: base divides q['p50'] back out (#L524), so P50 stays the ensemble mean times k.
Synthetic check (not a real roof)
I ran 17f99ac's learn() and run() offline, with HA and Open-Meteo faked as in tests/test_run.py, on the example config: one learn on 90 days, then a day-ahead run() for each of the next 120 days, five seeds pooled (600 held-out days). On daily totals, 6.3 % of days fell below P10 and 9.0 % above P90, against 10 % each. P90 only looked right on average: it was never passed on days forecast clear and passed on 14 % of the other days, which a fixed multiplier can't tell apart (that split is sharper in my synthetic world than it would be on a real roof). By hour, 10.8 % fell below P10, more than the 6.3 % of days, but that size comes straight from the hourly noise I picked, so your 23 % is the better evidence for the hourly side.
A possible path
Your plan to rerun with about three months of archive fits this. Each step can be checked against the archive before it changes anything:
- Out-of-sample ratios. Score the archived day-ahead
outrows against the actuals, asquality()does (#L416-L444), instead of the ensemble rebuilt with tonight's calibration. They need the curtailment mask today's ratios get froma.ok:quality()only takes out the outage hours, so throttled hours would land in the low tail. Fall back to today's method while there are too few days. - Hourly ratios for the hourly band. The same out-of-sample data taken hour by hour gives multipliers that match an hourly score. Your per-hour multiplier on 28 days sounds close to this, and the extra months should show whether it holds. Three catches: a clock hour moves against the sun over three months (and across the clock change), so the az/el bins you already use for shape may be a better key; 28 days is about three samples per hour in a 10 % tail, so neighbouring bins need pooling; and a band that is right per hour is too wide for the day's total, so it's worth deciding which of the two P10 should mean for EMHASS.
- True levels, 0.10 / 0.25 / 0.90, once steps 1 and 2 are in. Today's 5/95 may partly be making up for the in-sample ratios and for the hourly use. On today's ratios a true 10 % level would lift P10 and make your 23 % worse.
- A coverage attribute: the share of hours and of days below P10 and above P90 over the last 60 to 90 days. Then anyone can see whether the band is honest on their roof, before and after a change. This one changes nothing that's published, so it could go in first.
I've left out the member-spread idea, since your test showed it barely tracks the error on your roof.
What I didn't test
- Any real data. The numbers above are synthetic.
- Steps 1 to 4 as code.
- The effect on an EMHASS plan.
Activity
JustAnotherPU commentedon Nov 15, 2026
I tried steps 1 and 3 on 44 days of our own archive (1 Oct to 14 Nov): the published day-ahead band against the measured production.
The out-of-sample rows are scored on the same 30 days. 28- and 45-day windows give the same picture.
What I take from it:
So I've only done step 4 for now (#11):
band_coveragepublishes these numbers, so everyone can see how their own band behaves. I'm keeping this issue open for step 2.Only one roof and 44 days, autumn only. If someone can run it on a longer archive, the result could change.
JustAnotherPU commentedon Nov 15, 2026
A note on the numbers: the backtest table above (9.1 % of days below P10) and the figures in #11 (11.4 %) cover the same 44 days but do not pick the same hours. The backtest is my own script and keeps every hour with a P50 above 0.01 kWh, on days above 2 kWh.
band_coveragekeeps only the hours learning uses (no curtailment, no outage) and the engine's daylight threshold.LesIT1 commentedon Nov 15, 2026
Thanks for running it. Here's a second roof with a longer archive.
Setup. Perth, three faces (N and E at 4.84 kWp each, W at 1.3 kWp). I ran 4224c94 offline as a walk-forward:
learn()every morning, then the day-aheadrun()at 20:05 with the 8 global models from the example config's AU/NZ list, scored on the next day. After a 30-day warm-up that gives 105 scored days from 3 July to 22 October (winter into spring here). It predates #10 and #11. The quantile levels are the same, but I haven't rerun with #10's clip order.Two masks make it less clean than yours. Our export limit trims PV once the battery is full, so I dropped every hour where the battery was full and the grid was at the limit. That takes out a quarter of the daylight hours, mostly between 1 and 5 pm on sunny days, so the 901 hours left are thin on afternoons and what's left of them leans cloudy. I also dropped the days where the HA recorder was down and the PV statistic froze. Your backtest filter (P50 above 0.01 kWh, days above 2 kWh) keeps all 901 of these, so the hours differ from yours by the two masks and little else: a dozen twilight hours and the day after each outage are also out.
Results, day-ahead band against measured production:
Day totals are sums over the kept hours only. That's 4 days below P10 out of 105, so the daily band holds and the hourly band doesn't, as on your roof. On the days whose total stayed above P10, 20 % of hours still fell below it.
Where the hourly misses sit, by sun elevation:
Most of the excess is low sun. Part of it is a level miss on our roof rather than band width: below 10° in the morning, measured is 0.63 of P50 (66 % of those hours below P10, 2 % above), likely shading on the east side that the shape bins don't fully pick up. In the evening P50 is right (0.98) and the misses go both ways, 35 % each. Without the low morning hours it's 17 % below P10 overall. Morning and evening differ this much at the same elevation, which points to keying step 2 on the az/el bins rather than the clock hour or elevation alone.
By forecast day clearness (P50 over clear sky), the 11 days forecast under 0.5 had 29 % of hours below P10 and 33 % above P90. The 43 days forecast above 0.75 had 15 % and 8 %.
On what P10 should mean for EMHASS. When
weather_forecast_pv_quantile_biasis set (it's 0 by default), EMHASS uses P10 slot by slot with one blend factor for the whole horizon:bias * P10 + (1 - bias) * P50. That's the same for Solcast and for a passedpv_power_forecast_p10. What matters to a battery plan is mostly the energy over the next several hours, so I'd keep P10 as a cautious day total, which is what it is now.A band that's right per hour is too cautious once it's summed, but how much depends on the key. One flat multiplier that puts 10 % of hours below P10 would be 0.51 × P50 here, against 0.70 to 0.75 now, and no day's total fell below that. Keyed on elevation it's 0.29 below 10° up to 0.71 above 30°. That sums to about 0.67 × P50 per day, with 1 day in 105 below. The low hours carry so little energy (0-10° is 19 % of the hours and 2 % of the energy) that an elevation-keyed band would cost a battery plan little. Both are in-sample. If the flat multiplier stays, a per-hour bound would need its own attribute.
Same caveats as yours: one roof, no summer yet. The trim mask also means the afternoon numbers come from days the battery didn't fill, and I can't say which way that moves the hourly share.
JustAnotherPU commentedon Nov 16, 2026
Thanks, this settles what P10 means: it stays a cautious day total, and an hourly band would be its own attribute.
I tried step 2 on our 44 days, scored on the last 30, with each day's multipliers taken from earlier days only. Hourly out-of-sample ratios fix most of the hourly miss. Hours below P10 go from 27 % to 16 % at 10/90 and to between 9 and 11 % at 5/95, and the hourly P10 pinball drops from 0.185 to 0.150.
Keying didn't help here, though. One flat hourly multiplier scored as well as or better than elevation, morning/evening × elevation, or the 15×6 az/el bins. The keys only evened out the 0 to 10° bin. With 446 hours over 40 bins that's probably just too little data, and your roof has the shading a key would catch.
Summed per day, the hourly band is much more cautious (P10 at 0.37 to 0.52 × P50, against 0.62 now), so it shouldn't replace the daily one.
I'll redo this around mid-January with about 90 days, then decide on the attribute and its key. Leaving this open until then.