Skip to content

Add raw-weather Jev context benchmark - #127

Merged
janhelcl merged 6 commits into
feature/ml-experiment-platformfrom
feature/xc-jev-raw-site-context
Sep 22, 2026
Merged

janhelcl merged 6 commits into
feature/ml-experiment-platformfrom
feature/xc-jev-raw-site-context

Conversation

@janhelcl

@janhelcl janhelcl commented Sep 21, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • add a second, frozen Jev 1.13 context experiment
  • preserve all 77 canonical GFS values at 09:00, 12:00 and 15:00
  • add the production site name and every production takeoff with its stored wind direction
  • snapshot production metadata from public.sites and public.spots
  • fingerprint the 248-site / 535-takeoff snapshot into the resumable cache contract
  • retain the same pre-2023 priors, 11 Noul targets, held-out evaluator and model version
  • document the completed benchmark and close the Jev 1.13 experiment line

Result

Same 82,584-row xc-temporal-2024-jan-nov-v1 benchmark:

Model BCE ↓ Brier ↓ ROC-AUC ↑ Mono violation ↓
Conventional MLP (seed 42) 0.15686 0.04762 0.94206 0.00138
Jev semantic context 0.31116 0.08475 0.84939 0.17268
Jev raw weather + production sites 0.30780 0.08240 0.82614 0.110

The raw context modestly improved BCE, Brier and monotonicity over semantic context, but reduced ROC-AUC and remained decisively behind the conventional MLP. Approximate API cost: $32.

Decision: reject this variant, close the Jev 1.13 experiment line, and return to the planned shared per-pressure-level encoder with ordered aggregation.

Experimental boundary

This was one pre-specified context ablation. Because the first 2024 Jev result was already inspected, the repeated 2024 comparison is exploratory and is not fresh promotion evidence.

Validation

  • five targeted Jev tests passed locally
  • original semantic context and resumability remain covered
  • raw context test asserts all 77 features at all three forecast times
  • production snapshot validates 248 sites, 535 takeoffs and complete wind-direction metadata
  • repository CI passed on the implementation commit

@janhelcl
janhelcl marked this pull request as ready for review September 22, 2026 18:27
@janhelcl
janhelcl merged commit d4822a2 into feature/ml-experiment-platform Sep 22, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant