Skip to content

Benchmark Jev 1.13 on the XC temporal holdout - #126

Merged
janhelcl merged 2 commits into
feature/ml-experiment-platformfrom
feature/xc-jev-benchmark
Sep 21, 2026
Merged

janhelcl merged 2 commits into
feature/ml-experiment-platformfrom
feature/xc-jev-benchmark

Conversation

@janhelcl

Copy link
Copy Markdown
Owner

Summary

  • add a resumable hosted Jev 1.13 benchmark for the canonical XC contract
  • use a fixed semantic weather adapter and fit-only historical priors
  • evaluate all 82,584 rows in xc-temporal-2024-jan-nov-v1
  • document the completed result and reject Jev as the main XC model in ADR 0016

Result

Model BCE ↓ Brier ↓ ROC-AUC ↑ Mono rate ↓
Conventional MLP 0.15686 0.04762 0.94206 0.00138
Jev 1.13 0.31116 0.08475 0.84939 0.17268

Jev shows useful zero-shot semantic signal, especially at higher thresholds, but the predictive and monotonicity gaps are decisive. The experiment line is closed without tuning against the 2024 holdout.

Validation

  • full ML test suite: 98 passed
  • Jev unit coverage includes non-leaky state/questions and resumable partial-to-full execution
  • implementation tree previously verified against the locally tested commit

@janhelcl
janhelcl merged commit ee0e801 into feature/ml-experiment-platform Sep 21, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant