Skip to content

Upstream merge 260706 - #31

Merged
rfiorella merged 1298 commits into
developfrom
upstream-merge-260706
Jul 13, 2026
Merged

Upstream merge 260706#31
rfiorella merged 1298 commits into
developfrom
upstream-merge-260706

Conversation

@rfiorella

Copy link
Copy Markdown
Contributor

No description provided.

bartgol and others added 30 commits May 29, 2026 21:54
Off by default, turn on setting the machine
spec 'use_ninja' to True
Define frivinp_mesh for unstructured test
Make elm-lnd_docn_1way use driver-mct for now
Add or enable features in driver-moab that were missing. Also fix a few things revealed in testing.

Add rpointer robustness changes originally introduced in 5fe46fe
Enable writing of auxiliary history files.
Allow coupler to be on its own processor pool.
Introduce MOAB_LATERAL cpp to separate the lateral exchange infrastructure in ELM from driver-moab.
Close some memory leaks
Lower task count for some X-case tests
Replace redundant atm-to-ocn remap with saved tag values
Define river mesh input file for unstructured test the driver-moab needs

Fixes #8423
[BFB]
…346)

Update default MALI calving parameter defaults for Greenland meshes used in E3SM simulations

For MALI meshes that are used in prognostic, E3SM simulations with an
active Greenland ice sheet, set the default calving parameters to be
different than the standard default values used. This was tested via
builds with the following three meshes and confirmed working:
   mpas.gis4to40km, mpas.gis1to10kmR2, and mpas.gis1to10km.
For the first two, the desired default values show up in the mali_in
file after a build. For the last one, the desired default values show up
in the mali_in file after a build.

[NML] only for MALI configurations
[non-BFB] only for some configs with MALI
Fix for loop upper bound for rank-5 fields.

[BFB]
…s in the file

This allows, e.g., to avoid adding radiation's sw/lw bands variables
if no radiation-related vars are in the file.
EAMxx: support Ninja generator in test-all-eamxx

Off by default, turn on setting the machine spec 'use_ninja' to True.

[BFB]
It's already tracked as the Site on cdash
Add extra_debug testmod

Interacts with a new env XML setting 'EXTRA_DEBUG'.

Other components can extend this feature too.

Replaces the some of what env var "E3SM_ENABLE_KOKKOS_BOUNDS_CHECKING" was doing for us.

[BFB]
Add "splitform" theta advection type to homme. Also adds new test in homme
standalone which compares C++ and F90.
    
This does not change default in EAMxx, so is BFB.
    
[BFB]
…008)"

This reverts commit 6ae6364, reversing
changes made to 427c6ee.
Update CIME submodule

from 025f09aba0b72177df7e22705f0eadbe351b5add
to 1e2be8138c24aec7299654aca1b7ea6631af6c0a

Changes:
1) wait_for_tests: nows add sha and runtime to cdash build description
2) case.build: add --clean-cache flag
3) AI: Add copilot agents and fix skills
4) Add a comment in eri.py about a possible error this can catch
5) Allow extending baseline phase ** significant change
   Adds a new SystemTestsCommon method, additional_baseline_generation(), which the user can implement to perform extra operations during baseline generation. For example:
   - The user might have performed postprocessing during RUN that generated files to be included in the baseline.
   - The user might want to copy all files from a certain history tape, not just the last one.
6) add --driver option to jenkins_generic_job

Fixes:
1) Fix safe_copy(..., preserve_meta=False)
2) The variable DRV_RESTART_POINTER is only provided by cmeps and should not be required for the ERR test
3) Fix --mail-type args to use comma-separated values instead of repeated flags
4) Fix regular expression used in NLCOMP test

[BFB]
This revert changes from PR #8008. I misread the dashboard and merged
before realizing the whole host of DIFFs it actually caused.
    
Will create a new PR for splitform port.
In ice_comp_mct.F90, lnd_comp_mct.F90, and rof_comp_mct.F90, there is a Fortran WRITE statement with an extraneous comma.

write(logunit,*), '...'
This statement is not valid according to the Fortran standard. While gfortran accepts it without error,
other compilers, such as the Cray ftn compiler on Frontier, report it as a compilation error.
grnydawn and others added 17 commits June 29, 2026 14:45
This PR updates the Frontier machine configuration to align with the upcoming removal of ROCm 5 from the Frontier system.

* Updates software modules and compiler flags for the craycray and craycray-mphipcc compilers.
* Removes the crayamd and crayamd-mphipcc compilers based on recent discussions.
We never want errors to be lost, so use -e. There was no problem with the scripts-tests themselves.

Fixes #8510

This turned into a big PR due to me noticing that tests were very
broken with SCREAM_GENERATE_BASELINES_ONLY. I was getting baseline
compare problems even when GENERATE_ONLY and, worse, GENERATE_ONLY was
not generating anything in some cases. This PR refactors
CprncTest.cmake to check for SCREAM_GENERATE_BASELINES_ONLY and do a
copy if it's on.

Also, includes some more standardization of run_and_cmp tests.

[BFB]
Update the default values of ATM_NCPL for ne32/ne64/ne128

[BFB] (because we don't test with these grids)
Need ghci-snl-gnu_setup for scripts tests to work

[BFB]
…PR #8443)

Fix CICE CIME_cppdefs missing BLCK defines after buildnml -nodecomp configure

Fix missing BLCK defines in CIME_cppdefs: CICE configure script is called with
-nodecomp, so it writes CIME_cppdefs without the BLCK decomposition defines.
After buildnml appends those BLCK values to cppdefs, it now writes the complete
string back to CIME_cppdefs. Without this, a subsequent CMake reconfigure (e.g.,
after changing the pe-layout or rebuilding an existing case without cleaning) would
read an incomplete CIME_cppdefs and produce a broken build.

The fix is in the Python buildnml script and runs before any compilation. All machines
and compilers are affected equally.

This is specifically relevant when modifying and rebuilding an existing case without
cleaning: the first build succeeds because CICE_cppdefs.new triggers a full rebuild,
but a subsequent reconfigure reads the stale CIME_cppdefs and misses the BLCK defines.

[bfb]
Fixes memory leaks when accessing the string from MCT for data models

[BFB]
@rfiorella
rfiorella requested a review from matthewhoffman July 7, 2026 04:56
@matthewhoffman

Copy link
Copy Markdown
Collaborator

@rfiorella , I've identified 2 issues with what's coming in from E3SM/master, both are pretty unusual:

  • The ghci-oci machine definition that we are using for our CI tests was removed in April, because E3SM Project was having trouble maintaining it: remove ghci-oci machine and tests for now E3SM-Project/E3SM#8279 I've tested reverting the merge commit in this branch (locally) and this works with a little extra work:
    • some easy merge conflicts had to be resolved
    • there are a number of new env variables that need to be defined: NETCDF_C_ROOT, NETCDF_FORTRAN_ROOT, PARALLEL_NETCDF_ROOT. All can be set to /usr/local/packages and we can put this in a commit updating the machine definition. I suspect these are new env vars that have been added since the ghci machine definition was removed in April.
  • The default coupler was switched from MCT to MOAB a month ago: Switch default driver to moab E3SM-Project/E3SM#8431 They did extensive testing to make sure nearly all tests pass BFB with MOAB, so I hoped this would be seamless. However, I was very surprised to discover that MOAB is being included in E3SM as a pre-built library rather than compiled as part of the E3SM build. This seems like a decision that will make the build very fragile and complicate development, because I don't see a way that the version of MOAB is tracked/confirmed, and even if it is, any needed MOAB code changes are going to require a lot more steps to be made available to E3SM.
    • My editorial aside, the practical upshot is that MOAB needs to be pre-installed on every machine (and specified in config_machines.xml). If E3SM Project wants to ship a container with MOAB already installed, we could use that, but until they resurrect and update their ghci machine container, that's not an option.
    • The other option is that MCT can still be used (add "--driver mct" to create_test or _Vmct to a test definition). However, we would have to modify OLMT to do this. And we'd probably want to have OLMT do some sort of git history check or code check to determine if that's necessary, otherwise we would need everyone to switch over OLMT and E3SM versions at exactly the same time.

My thinking about these issues is:

  1. revert the removal of the ghci-oci machine def in this branch so that's already fixed as part of this merge. That will mean we cannot merge NGEE-Arctic/develop back to E3SM-Project/master, but I think that's probably already the case.
  2. Update our CI test in this branch to use _Vmct and avoid using MOAB for the near future.
  3. Update OLMT to use "--driver mct" conditional on E3SM code version and get that merged prior to merging this branch.

Neither of these issues has a perfect solution, and my suggestions have some drawbacks. I'm happy to entertain other ideas too.

One additional issue: After applying steps 1 and 2 I can get SMS_Vmct_D_Ln5_P4.ne4pg2_oQU480.F2010.ghci-oci_gnu (one of the failing CI tests) to compile successfully, but it is producing a segfault in HOMME at runtime (running in the E3SM CI container on my laptop), so there may be more going on than the above. I'm going to push commits adding steps 1 and 2 to this branch and see if GH CI fares any better. We can always remove them if we decide to go a different way.

The container does not have MOAB pre-installed, so it's easiest to just
use MCT for now.
@rfiorella

Copy link
Copy Markdown
Contributor Author

@matthewhoffman Oh yuck those are both pretty challenging. Thanks for the detailed explanation.

For 1, the description in E3SM-Project/E3SM#8279 sounds like it's just a matter of getting the inputdata into container (and perhaps setting those additional environment variables. My 2 cents here is since that seems pretty straightforward to do, and would allow reenabling on the E3SM project side (and hence, at least relieving any concerns about contributions back at least related to CI) - I'd like to take a shot at it!

2 and 3 both sound reasonable - I agree that for this project (and especially since we're not running fully coupled) I would prefer to invest a little effort into continuing to use MCT for now rather than possibly more into switching to MOAB.

@matthewhoffman

Copy link
Copy Markdown
Collaborator

For 1, I've already offered them a solution to the inputdata maintenance issue here: E3SM-Project/containers#33. I think there's a good chance they will merge it and regenerate the containers, but I don't imagine it will be a high priority for them, so it might take a few weeks to play out. It's possible they'll decide they don't want to bother anyway, because their primary nightly testing is now on a self-hosted machine. If that's the case, we can still implement these changes in our container setup and adjust our CI accordingly.

@rfiorella

rfiorella commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

I just saw your PR at E3SM/containers and was going to write a note that I saw you were already ahead of me 😄

One argument I might suggest to them about the utility of having this routinely run on GH is for ecosystem projects that don't have access to the same testing infrastructure. We can get setup to run the land tests, for example, and run our own local compile tests but it is nice to have something out there that is a universal test, on the same equipment, that E3SM and ES projects can run, that I can at least make sure passes before asking anyone to spend a second looking at a PR

@matthewhoffman

Copy link
Copy Markdown
Collaborator

That's a good point about ecosystem project testing. I'll mention that if there seems to be hesitancy.

Well, the good news is that the 3 CI tests that were passing before this PR are now passing with my additional commits pushed. There must have been something wonky locally with my attempt to run them. So I think we can proceed with my plan above - I've already pushed 1 and 2 in my list. I'll open an OLMT PR for #3, and once we merge that, we can proceed to merging this. As for the one test that is failing, I think we continue to let that fail until E3SM decides what they want to do with the inputdata-in-containers issue - no need to resolve that before merging this PR since it was already failing.

@rfiorella
rfiorella merged commit 2750ef0 into develop Jul 13, 2026
3 of 4 checks passed
@matthewhoffman
matthewhoffman deleted the upstream-merge-260706 branch July 13, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.