Adds the thermo coupling

[alicebarthel]

This PR adds the thermo coupling - from forcing terms to thickness and tracer tendencies.
The energy of mass fluxes and phase changes are hard-coded into the tendency conversions, like it was in MPAS-O. Hopefully the documentation (inline or otherwise) clarifies the meaning of each term.

This PR is 3/3 to make #418 more digestible.

Checklist

• Documentation:
  • User's Guide has been updated
  • Developer's Guide has been updated
  • Documentation has been built locally and changes look as expected
• Linting
  • Pre-commit run locally
• Building
  • CMake build does not produce any new warnings from changes in this PR
• Testing

  • CTest unit tests: "All tests passed" on pm-cpu (gnu) and pm-gpu.

  • The Polaris omega_pr test suite
    has passed, using the Polaris e3sm_submodules/Omega baseline

  • New tests:

    • new CTest unit tests: thickness and tracer tendencies are tested by building a baseline then turning the tendencies on and comparing to baseline. This was done because vertical tendencies do not have a flag to be turned off.

[vanroekel]

I think 0 is a fine limit. Does this form of code preclude logging when we go below this threshold?

[alicebarthel]

@vanroekel note that this conditional is checking the validity of the tracer index (SaltIndex >= 0), not the value of salinity. For a check on salinity, it would be preferable to use Kokkos::clamp().
This was to make sure we are not using an invalid index, but we could do a clean fail rather than use 0 salinity...

[cbegeman]

I'm not sure that it's the job of this function to make sure that the tracer indices are initialized correctly. I'd be inclined to check for that elsewhere (and maybe not every timestep or every time forcing is updated) and remove the conditional. Thoughts?

[katsmith133]

Yea, I would agree. Are you thinking something like OceanInit, @cbegeman?

[cbegeman]

Yes, maybe Tracers::Init which is called during OceanInit?

[cbegeman]

@katsmith133 Should this be resolved or is this work still outstanding?

[sbrus89]

It seems like this is resolved looking at the code, but @katsmith133 can verify.

[vanroekel]

overall this looks quite good @alicebarthel I only had pretty minor comments

[alicebarthel]

One key test to do is to verify that omega_pr is BFB with develop baseline since we do not currently have a polaris test using the surface tracer forcing. @katsmith133

Here is my current test report:

Polaris omega_pr suite

• Baseline workdir: /global/homes/a/abarthel/pscratch/polaris-scratch/baseline-current-develop-cpu
• Baseline build: /global/homes/a/abarthel/pscratch/omega_scratch/develop-cpu
• PR build: /global/homes/a/abarthel/pscratch/omega_scratch/add-thermo-forcing-cpu-gnu
• PR workdir: /global/homes/a/abarthel/pscratch/polaris-scratch/add-thermo-cpu_vsbaseline
• Machine: pm-cpu
• Compiler: gnu
• Build type: Debug
• Log: not found
• Result:
  • Diffs (1 of 12):
    • ocean/planar/barotropic_gyre/munk/free-slip

[alicebarthel]

the log is at
/pscratch/sd/a/abarthel/polaris-scratch/add-thermo-cpu_vsbaseline/case_outputs/ocean_planar_barotropic_gyre_munk_free-slip.log
if that can be helpful

[cbegeman]

@alicebarthel Is this ready to test with standalone Omega (I'm thinking column cases) with the flux fields provided in the forcing stream?

[alicebarthel]

Yes @cbegeman additional testing with single-column polaris cases (and analytical equilibrated state) or more realistic cases could be performed at this stage. It may be good to resolve omega_pr fail above before then, though? (I will not be able to debug this myself, sorry).
If it can be helpful for the additional testing, I generated forcing files with realistic values (taken from an existing simulation) and the necessary variables for various meshes, stored on pm:
in /global/homes/a/abarthel/pscratch/omega_scratch/
forcing_nCells_7153.nc [forcing.nc]
forcing_nCells_2304.nc [forcingPlanar.nc]
forcing_nCells_2562.nc [forcingSphere.nc]
You may prefer to start with idealized forcing but thought I'd let you know.

[katsmith133]

Pulled this PR down onto PM this morning and verified it passes all CTests on PM-CPU and GPU. Also, the polaris omega_pr suite passed BFB on PM-CPU for me so perhaps @alicebarthel you had something wonky with your set up.

Polaris omega_pr suite

• Baseline workdir: /pscratch/sd/k/katsmith/polaris_PR461//baseline_omega_pr
• Baseline build: /pscratch/sd/k/katsmith/polaris-main/omega_build
• PR build: /pscratch/sd/k/katsmith/polaris-PR461/omega_build
• PR workdir: /pscratch/sd/k/katsmith/polaris_PR461/pr461_omega_pr
• Machine: pm-cpu
• Compiler: gnu
• Build type: Release
• Log: not found
• Result: All tests passed

@cbegeman, I think it should be safe to test with single-column now if you want

[katsmith133]

Pretty minor comment for the PR. Seems like overall its pretty good, but I'd also like to see the results of the single-column test. @cbegeman can I help in any way with this?

[katsmith133]

Was just checking the compile logs and saw this warning for PM-GPU:

TracerForcingVars.cpp(198): warning #20011-D: calling a __host__ function("OMEGA::Eos::calcPtFromCt(const double &, const double &) const") from \
a __host__ __device__ function("OMEGA::TracerForcingVars::computeSurfInsituTemp(const  ::Kokkos::View<double ***,  ::Kokkos::LayoutRight,  ::Kokkos::CudaSpace >  &, const  ::OMEGA::VertCoord *, const  ::OMEGA::Eos *) c\
onst::[lambda(int) (instance 1)]::operator () const") is not allowed

working on how to resolve this issue...

[katsmith133]

Was just checking the compile logs and saw this warning for PM-GPU:

TracerForcingVars.cpp(198): warning #20011-D: calling a __host__ function("OMEGA::Eos::calcPtFromCt(const double &, const double &) const") from \
a __host__ __device__ function("OMEGA::TracerForcingVars::computeSurfInsituTemp(const  ::Kokkos::View<double ***,  ::Kokkos::LayoutRight,  ::Kokkos::CudaSpace >  &, const  ::OMEGA::VertCoord *, const  ::OMEGA::Eos *) c\
onst::[lambda(int) (instance 1)]::operator () const") is not allowed

working on how to resolve this issue...

Ok, I figured out a solution to this that removes the compile warning, but I am now adding in Linear and Constant EOS support to this PR, which primarily is just adding non-TEOS10 options to the calcCtFreezing function. I believe I have that finished, but am working on testing. Note: Alice suggests (on L355-356 of Eos.cpp) that we use the UNESCO 1983 polynomial for any of the non-TEOS10 options, but I think this would be inconsistent? So for now I've implemented a linear option and a constant option. I would like feedback on that once I have it tested and pushed to this PR.

[katsmith133]

Retesting on Frontier after latest enthalpy calculation changes and rebasing to develop

CTest unit tests:

• Machine: Frontier
• Compiler: craygnu, craygnu-mphipcc
• Build type: Release
• Result: All tests passed

Polaris omega_pr suite

• Baseline workdir: /lustre/orion/cli115/scratch/kmsmith/polaris_testing_pr461/main_omega_pr
• Baseline build: /ccs/home/kmsmith/polaris-main/omega_build
• PR build: /ccs/home/kmsmith/polaris-pr461/omega_build
• PR workdir: /lustre/orion/cli115/scratch/kmsmith/polaris_testing_pr461/pr461_omega_pr
• Machine: frontier
• Partition: batch
• Compiler: craygnu-mphipcc
• Build type: Release
• Log: not found
• Result:
  • Failures (1 of 22):
    • ocean/column/ekman

ocean/column/ekman test failure is expected.

@cbegeman I think we can finish up with updating polaris and doing the final conservation testing.

[cbegeman]

@katsmith133 Thanks for testing! I also did testing last night. Results located here: E3SM-Project/polaris#663 (comment)

[cbegeman]

Testing

Polaris omega_pr suite

• Baseline workdir: /lus/flare/projects/E3SM_Dec/cbegeman/polaris-output/main-omega-submodule-20260818
• Baseline build: /lus/flare/projects/E3SM_Dec/cbegeman/polaris-output/main-omega-submodule-20260818/build
• PR build: /lus/flare/projects/E3SM_Dec/cbegeman/polaris-output/thermo-omega-thermo-20260818/build
• PR workdir: /lus/flare/projects/E3SM_Dec/cbegeman/polaris-output/main-omega-thermo-20260819
• Machine: aurora
• Compiler: oneapi-ifxgpu
• Build type: Release
• Log: not found
• Result:
  • Failures (1 of 22):
    • ocean/column/ekman

[cbegeman]

Testing

• All ctests pass on aurora, cpu
• All ctests but 11 featuring SCYCL pass on aurora, gpu