Skip to content

Test physics-aware AGL pressure-profile representation - #123

Merged
janhelcl merged 8 commits into
feature/ml-experiment-platformfrom
feature/xc-weather-profile-agl-mask
Sep 9, 2026
Merged

janhelcl merged 8 commits into
feature/ml-experiment-platformfrom
feature/xc-weather-profile-agl-mask

Conversation

@janhelcl

@janhelcl janhelcl commented Sep 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • add the site-relative AGL + above-ground mask representation on top of the frozen vertical CNN experiment
  • keep the CNN, raw bypass, shared [64,32] MLP, fusion tower and optimizer policy unchanged
  • derive z_AGL = geopotential_height - site_altitude from raw values
  • fit AGL scaling only on valid above-terrain training/noon levels
  • mask pressure surfaces below GFS terrain and append an explicit validity channel

Result

Seed 42 looked clearly positive against both the plain CNN and frozen conventional MLP, but paired seeds 42–46 did not confirm the improvement.

Against the conventional MLP, AGL/mask averaged:

  • BCE +0.000310 worse, wins 2/5
  • Brier +0.000034 worse, wins 3/5
  • ROC-AUC +0.000002, effectively tied, wins 3/5
  • monotonic violation +0.003580 worse, wins 2/5

Against the plain CNN, AGL/mask averaged:

  • BCE +0.000245 worse, wins 2/5
  • Brier +0.000024 worse, wins 2/5
  • ROC-AUC +0.000191 better, wins 4/5
  • monotonic violation -0.001227, wins 2/5

Decision

Do not promote AGL/mask. The small ROC-AUC signal versus the plain CNN does not improve the primary losses and does not clear the conventional promotion bar.

Stop the current Conv1D profile family rather than tuning width, depth, kernel size or adding another coordinate channel. Keep the implementation/config as a reproducible negative experiment; the next weather-specific architecture should change the inductive bias to a shared per-pressure-level encoder.

The multi-control confirmation workflow and durable decision are recorded in stacked PR #124 / ADR 0015.

@janhelcl janhelcl changed the title Add physics-aware AGL pressure-profile experiment Test physics-aware AGL pressure-profile representation Sep 9, 2026
@janhelcl
janhelcl changed the base branch from feature/xc-weather-profile-cnn to feature/ml-experiment-platform September 9, 2026 06:08
@janhelcl
janhelcl merged commit 8861769 into feature/ml-experiment-platform Sep 9, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant