Skip to content

Add shared pressure-level XC encoder experiment - #128

Merged
janhelcl merged 2 commits into
feature/ml-experiment-platformfrom
feature/xc-levelwise-profile-encoder
Sep 26, 2026
Merged

janhelcl merged 2 commits into
feature/ml-experiment-platformfrom
feature/xc-levelwise-profile-encoder

Conversation

@janhelcl

@janhelcl janhelcl commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • add a shared per-pressure-level MLP encoder for the canonical 5 × 13 XC weather profile
  • apply the same [16, 8] level encoder at every pressure surface and forecast time
  • preserve 1000 → 500 hPa order through flattening and a 32d profile projection
  • add an isolated seed-42 config against the frozen conventional MLP
  • record the completed result in ADR 0017 and close the experiment

Controlled experiment boundary

The candidate added only the profile branch:

standardized [u, v, T, RH, z] × 13 levels
  -> shared MLP 5 -> 16 -> 8 per level
  -> ordered flatten (13 × 8)
  -> Linear(104 -> 32)

The raw bypass, shared per-time MLP [64, 32], fusion [64, 32], 32d site embedding, independent sigmoid head, dropout 0.10, optimizer policy, and xc-temporal-2024-jan-nov-v1 benchmark remained unchanged.

Result

Metric Conventional MLP Shared level MLP Delta
Macro BCE ↓ 0.15686 0.15794 +0.00108
Macro Brier ↓ 0.04762 0.04785 +0.00023
Macro ROC-AUC ↑ 0.94206 0.94233 +0.00027
Mono violation ↓ 0.00140 0.00209 +0.00069
Best epoch 86 86 —
Parameters 48,523 58,259 +9,736

Decision

Reject at seed 42. The candidate regresses both primary calibration losses, worsens monotonicity, and adds about 20% more parameters. The isolated +0.00027 ROC-AUC gain does not earn paired-seed confirmation.

Do not run seeds 43–46, tune the level MLP, add AGL/masking, or proceed to level attention on this encoder. Preserve it as a reproducible negative experiment; the frozen conventional MLP remains the promotion baseline.

Validation

  • focused weather-profile tests: 10 passed
  • full ML suite: 103 passed
  • git diff --check: clean

@janhelcl
janhelcl merged commit 6faab94 into feature/ml-experiment-platform Sep 26, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant