Skip to content

Generalize paired-seed confirmation and close Conv1D profile experiments - #124

Merged
janhelcl merged 13 commits into
feature/ml-experiment-platformfrom
feature/xc-weather-profile-agl-mask-confirmation
Sep 9, 2026
Merged

janhelcl merged 13 commits into
feature/ml-experiment-platformfrom
feature/xc-weather-profile-agl-mask-confirmation

Conversation

@janhelcl

@janhelcl janhelcl commented Sep 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • generalize glideator-ml confirm-seeds xc to accept multiple controls while training the candidate only once per seed
  • preserve the existing single-control command/report shape
  • add tests for multi-control pairing, candidate reuse and legacy behavior
  • record the paired 42–46 AGL/mask result
  • add ADR 0015 and close the current vertical Conv1D profile family

Confirmation result

The AGL/mask candidate does not clear the frozen conventional MLP promotion bar:

Metric MLP mean AGL/mask mean Delta AGL wins
BCE ↓ 0.156677 0.156987 +0.000310 2/5
Brier ↓ 0.047601 0.047635 +0.000034 3/5
ROC-AUC ↑ 0.941865 0.941867 +0.000002 3/5
Mono ↓ 0.002076 0.005656 +0.003580 2/5

Against the plain CNN, AGL/mask is also slightly worse on BCE/Brier; only ROC-AUC shows a small signal (+0.000191, 4/5 wins).

Decision

Reject AGL/mask as a promotion candidate and stop the current Conv1D family. Do not tune convolution width/depth/kernel size or add another coordinate channel to this architecture. The frozen conventional MLP remains the promotion reference.

Keep both CNN configs as reproducible negative experiments. Preserve AGL/masking as possible inputs for a different encoder family.

The next vertical-profile hypothesis should use a shared per-pressure-level encoder with simple ordered aggregation; level attention comes later as a separate experiment.

Harness improvement

A two-control five-seed confirmation now takes 15 trainings rather than 20: 5 candidate + 5 per control. The same candidate run for each seed is paired with both controls, preserving the statistical comparison while avoiding duplicate training.

@janhelcl janhelcl changed the title Reuse candidate runs for multi-control seed confirmation Generalize paired-seed confirmation and close Conv1D profile experiments Sep 9, 2026
@janhelcl
janhelcl changed the base branch from feature/xc-weather-profile-agl-mask to feature/ml-experiment-platform September 9, 2026 06:09
@janhelcl
janhelcl merged commit 9b41beb into feature/ml-experiment-platform Sep 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant