Add perfect SCM calibration by ph-kev · Pull Request #4516 · CliMA/ClimaAtmos.jl

ph-kev · 2026-05-22T20:46:56Z

This PR adds a perfect SCM calibration using ClimaCalibrate v0.3 which should be similar to the calibration pipeline in ClimaCoupler and ClimaLand.

The entry point to the calibration is run_calibration.jl which will generate data from a simulation and run the calibration to fit against the data.

TODO

Add support for including average profiles, time series data, and aggregation/reductions
~~Ability to run multiple simulations from multiple configurations for a single calibration~~ (This feature will be added in another PR)
Add more reconstruct functions ClimaCalibrate.jl#319 which is needed for postprocessing iterations
Replace buildkite pipeline at calibration/test/pipeline.yml

Optional

Make a new release of ClimaCalibrate to use the new reconstruct functions

ph-kev · 2026-05-22T20:49:31Z

+    # This is where we would do preprocessing and minibatching before creating the
+    # EKP.Observation


I was not sure what preprocessing is desired here.

ph-kev · 2026-05-22T20:51:28Z

+    plot_constrained_params_and_errors(output_dir, ekp, prior)
+    return nothing


Is there any specific plotting that need to happen here?

nefrathenrici

This looks clean, we still need to decide on the priors and observation details.

ph-kev · 2026-06-09T20:17:08Z

I ran the calibration again on the latest commit and it seems to work now. Some of the parameters are not identifiable though.

nefrathenrici · 2026-06-09T20:22:36Z

I ran the calibration again on the latest commit and it seems to work now. Some of the parameters are not identifiable though.

The cloud ice terminal velocity parameter doesn't seem to be informed by the data, I think we can remove it from the calibration.

github-actions · 2026-06-10T02:55:08Z

📖 Docs preview for this PR: https://clima.github.io/ClimaAtmos.jl/previews/PR4516/

nefrathenrici

Thanks! I have a few comments to add finishing touches:

Delete the now-unused test files in calibration/test/. I think the entire folder can be removed.
Delete experiments/perfect_scm/run_calibration.sh if it is not run anywhere.
Update the base calibration/README.md to reference the nice README you already wrote.
Update the artifact path in the pipeline so that plots can be seen directly from buildkite.

odunbar

Hi @ph-kev This looks nice!

I think my points are essentially of one theme: over-wrapping. Currently we have really nicely contained a lot of the boilerplate - but at times I think this will come to the detriment to what a future user might want to use this for. (I understand by default it is fed into a test pipeline) but it should not be written as if it will not be run by a user.

With that in mind, I would urge exposing a couple of items to the run_calibration.jl top-level script:

The true parameter values should be accessible
The true processed data should be accessible (before creating the observations)
The observation recipe should be changeable.
Perhaps add at the end of a script a function that gets the iterations from the eki, and builds them with the outputVar
In this way we are giving users a similar degree of freedom as you do with the prior - which is easily modifiable at the top level!

Update: Created Issue CliMA/ClimaCalibrate.jl#328 for the user-defined covariances, that are beyond the scope of the PR

odunbar · 2026-06-11T18:31:42Z

+include(model_interface_filepath)
+
+# Set up model interface
+config = joinpath("config", "model_configs", "prognostic_edmfx_trmm_column.yml")


It is not clear what the true parameters are - I would obtain these for the user - extracted from the simulation, or at least direct them to their location (e.g. ClimaParams.jl defaults?), else explicitly overwrite them locally with fixed values so they do not change

I don't think there is an easy way of querying for the parameters of a simulation after it is created, so I ended up letting the user specify their TOML file that can be used to overwrite the parameters of the simulation.

odunbar · 2026-06-11T18:32:12Z

+
+# Set priors for parameters of interest
+means = [5.0, 1.0, 0.01, 0.01]
+mean_std_dev_lub(m) = (m + 0.05 * m, m * 0.2, 0.0, m + 4 * m * 0.2)


Just checking if the upper bound here is justified? (hard to tell as the user does not easily know the "true" params here

I typically would not place hard bounds unless they are needed (e.g. the 0.0 likely is)

I am going to put Inf as the upper bound, but @trontrytel would know.

odunbar · 2026-06-11T18:47:41Z

+    n_iterations,
+    prior,
+    interface.output_dir,
+)


It might be helpful for a user looking at this script, for them to see how to obtain the final u_ens and g_ens and reconstructed_g_ens ( via reconstruct_g_ens). Just to demonstrate the functionality at the top level?

I think it would be better to have a section for this in ClimaCalibrate (see CliMA/ClimaCalibrate.jl#330).

ph-kev force-pushed the kp/perfect-cal branch from 5ee35e7 to 8909b4d Compare May 22, 2026 20:50

ph-kev commented May 22, 2026

View reviewed changes

ph-kev force-pushed the kp/perfect-cal branch from 8909b4d to b13449a Compare May 22, 2026 20:52

ph-kev requested review from nefrathenrici and trontrytel May 22, 2026 20:53

ph-kev commented May 22, 2026

View reviewed changes

Comment thread calibration/experiments/perfect_scm/run_calibration.jl Outdated

ph-kev force-pushed the kp/perfect-cal branch from b13449a to bd0b951 Compare May 22, 2026 21:00

nefrathenrici reviewed May 22, 2026

View reviewed changes

Comment thread calibration/experiments/perfect_scm/Manifest.toml Outdated

Comment thread calibration/experiments/perfect_scm/run_calibration.sh Outdated

Comment thread calibration/experiments/perfect_scm/run_calibration.jl Outdated

ph-kev force-pushed the kp/perfect-cal branch 2 times, most recently from d185e69 to d22bf34 Compare May 22, 2026 21:42

This was referenced May 22, 2026

Use dims keyword argument for flatten CliMA/ClimaCalibrate.jl#316

Closed

Add more reconstruct functions CliMA/ClimaCalibrate.jl#319

Merged

ph-kev force-pushed the kp/perfect-cal branch 3 times, most recently from 28d7259 to e06f6df Compare June 9, 2026 18:57

ph-kev force-pushed the kp/perfect-cal branch 3 times, most recently from b78bb05 to 7c5d2ad Compare June 10, 2026 00:22

ph-kev force-pushed the kp/perfect-cal branch 5 times, most recently from e28c0f2 to f4eaf43 Compare June 10, 2026 20:11

ph-kev requested review from nefrathenrici and odunbar June 10, 2026 21:10

nefrathenrici approved these changes Jun 10, 2026

View reviewed changes

ph-kev force-pushed the kp/perfect-cal branch from f4eaf43 to 265da8c Compare June 10, 2026 21:23

ph-kev force-pushed the kp/perfect-cal branch 2 times, most recently from 9217047 to 23de47c Compare June 10, 2026 21:59

odunbar requested changes Jun 11, 2026

View reviewed changes

ph-kev mentioned this pull request Jun 12, 2026

Update FAQ for ObservationRecipe CliMA/ClimaCalibrate.jl#329

Open

ph-kev added 2 commits June 12, 2026 16:51

Add perfect SCM calibration

ff9eba1

Delete test calibration pipeline

affab7c

ph-kev force-pushed the kp/perfect-cal branch from 23de47c to affab7c Compare June 12, 2026 23:51

		# This is where we would do preprocessing and minibatching before creating the
		# EKP.Observation

		plot_constrained_params_and_errors(output_dir, ekp, prior)
		return nothing

Conversation

ph-kev commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nefrathenrici left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ph-kev commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nefrathenrici commented Jun 9, 2026

Uh oh!

github-actions Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nefrathenrici left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

odunbar left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ph-kev Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ph-kev commented May 22, 2026 •

edited

Loading

ph-kev commented Jun 9, 2026 •

edited

Loading

github-actions Bot commented Jun 10, 2026 •

edited

Loading

nefrathenrici left a comment •

edited

Loading

odunbar left a comment •

edited

Loading

ph-kev Jun 12, 2026 •

edited

Loading