Update EKAT to require C++20

[tcclevenger]

Update EKAT to require C++20

Motivation

Needed to update to Kokkos 5.0.

Testing

CI already uses C++20.

[bartgol]

Asking the obv question: are we sure that E3SM will build everywhere with C++20 required? If not, this my hold back any ekat update in e3sm.

[rljacob]

It will not build with all of our current default compilers. In particular, the "intel" compiler on Chrysalis. Possibly others.

[bartgol]

Looks good. Only possible concern is whether e3sm is ready for c++20 across all machines.

[tcclevenger]

@bartgol @rljacob Good to know. I created this PR while I'm testing E3SM on the target machines, but since we already know there will be some that are not compatible, we will need to find a different way to handle this.

Kokkos 5 is going to require C++20, what would the process be to eventually update compilers on Chrysalis (and maybe others) so that we can upgrade? Or is this going to be a bigger roadblock? I guess the first step is for me to test exactly which compilers do not work.

[bartgol]

One solution would be to mis with a cmake2.0 way of specifying the std, via something like

if (NOT CMAKE_CXX_STANDARD)
  target_compile_features(ekat_core PUBLIC cxx_std_20)
endif()

Then, in the mach files for old compilers we keep lines like

set (CMAKE_CXX_STANDARD 17)

which will force ekat to honor the required standard. Of course, for machines where we do set 17 in the mach files, we won't be able to build anything that needs kokkos 5.0 (e.g. any F case, since EAM still use kokkos for the SL transport). This means that we cannot switch to kokkos 5 until we can abandon compilers that only support c++17.

[tcclevenger]

On Chrysalis:

• gnu: default compiler is compatible
• oneapi-ifx: version is technically compatible, but we get a build error about not being able to find "concepts" file. Output indicates oneAPI is not correctly configured.
• intel: Kokkos no longer supports INTEL's classic compilers (as of Kokkos 4.6, we currently use Kokkos 4.5). Maybe they are C++20 compliant, but we will need to change anyways to update Kokkos.

[bartgol]

• oneapi-ifx: version is technically compatible, but we get a build error about not being able to find "concepts" file. Output indicates oneAPI is not correctly configured.

Ugh, that's underwhelming. I wonder if this is on the chrys IT folks' radar...

[tcclevenger]

Ugh, that's underwhelming. I wonder if this is on the chrys IT folks' radar...

Yeah, I want to try and make a small reproducer outside of E3SM. Make sure it's not something we are doing.

[tcclevenger]

Update:

• PM-CPU/GPU and Aurora: Ran some tests from nightly, C++20 builds and runs BFB with master.
• Chrysalis_oneapi-ifx: Az update the oneapi-ifx compiler and I've tested against master baselines and C++20 builds and runs BFB (yay!)
• Chrysalis_intel: No longer supported in Kokkos 5, I think current one we are using isn't c++20 compatible.
• Chrysalis_gnu: technically compliant gnu version, but internal compiler error with C++20 (somewhere in ekat_pack)

@rljacob @bartgol How should we proceed? Can we require users of Chrysalis start switching to oneapi? Do we need to have a frozen version of Kokkos as a tpl for the other compilers? Could we require anyone using ekat (running with EAMxx) use oneapi?

[bartgol]

• PM-CPU/GPU and Aurora: Ran some tests from nightly, C++20 builds and runs BFB with master.

Did you run both gnu and intel on pm-cpu? I think @ndkeen is also using nvidia as compiler on pm-gpu, so maybe we should check that too.

@rljacob @bartgol How should we proceed? Can we require users of Chrysalis start switching to oneapi? Do we need to have a frozen version of Kokkos as a tpl for the other compilers? Could we require anyone using ekat (running with EAMxx) use oneapi?

I think we can keep both intel and oneapi. But we won't be able to run anything that uses kokkos with the intel compiler. I think this includes also any EAM case though, since EAM uses hommexx's SL transport by default, so this may impact a lot of ppl.

@jgfouca wild option: can CIME update a submodule conditionally on the compiler (or any other XML var)? E.g., can we do something like

if compiler is "intel":
   update_submodule(f'{SRCROOT}/externals/ekat', 'v1.1', recursive=True)  

to switch the submodule to an older tag?

[rljacob]

We will require Chrysalis users to use oneapi-ifx. It will be made the default. ifort will still be an option for anyone who needs it and they have to use an earlier hash.

We should make a plan on when this goes to master since its a breaking change for chrysalis.

[bartgol]

We will require Chrysalis users to use oneapi-ifx. It will be made the default. ifort will still be an option for anyone who needs it and they have to use an earlier hash.

We should make a plan on when this goes to master since its a breaking change for chrysalis.

We should prob make this change first, and then update ekat/kokkos (and the required cxx standard). Maybe mid-late March? I think 2 months may be a good enough window for ppl to switch to the new compiler.

[jgfouca]

@bartgol , CIME does not support that currently. We could, in theory, add support for configuration that allows for customizing submodules, but that seems kinda hacky.

[tcclevenger]

@rljacob @bartgol I'll do some runs with EAM cases to confirm that EAM will still need to compile Kokkos, then make a PR updating the default compiler.

[tcclevenger]

Testing update

• PM-CPU
  • intel: SMS_Lh4.ne4pg2_ne4pg2.F2010-SCREAMv1.pm-cpu_intel.eamxx-output-preset-1--eamxx-prod is BFB with master baseline
  • gnu: SMS_Lh4.ne4pg2_ne4pg2.F2010-SCREAMv1.eamxx-output-preset-1--eamxx-prod is BFB with master baseline
  • nvidia: SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.eamxx-output-preset-3 runs, but no baselines exist (I'm generating my own master baseline)
• PM-GPU:
  • gnugpu: SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.eamxx-output-preset-3 is BFB with master baseline
  • nvidiagpu: No baselines, CDASH only runs e3sm_gpuacc_next_nvidiagpu, which passes

I will next go to frontier, then I think we will have all the info we need to move forward.

[tcclevenger]

Re: intel compiler on chrysalis

Confirmed, the F2010 test case still requires ekat and does not work with C++20. I'll create a PR in E3SM switching the default compiler to oneapi on chrysalis. Not sure how we want to deal with trying to have an old version of Kokkos available vs. trying to require everyone switch compilers.

[bartgol]

Re: intel compiler on chrysalis

Confirmed, the F2010 test case still requires ekat and does not work with C++20. I'll create a PR in E3SM switching the default compiler to oneapi on chrysalis. Not sure how we want to deal with trying to have an old version of Kokkos available vs. trying to require everyone switch compilers.

I'm in favor of requiring ppl to switch. The master branch should always follow the latest development. If ppl need older compiler, they can use tags and swtich to older tags.

@rljacob may have other thoughts though...

[rljacob]

In your switch, will you also make a older pre-built version of Kokkos availble for ifort users? There are in-progress production runs with ifort and its not clear yet if they can switch. Also CCing @amametjanov

[bartgol]

In your switch, will you also make a older pre-built version of Kokkos availble for ifort users? There are in-progress production runs with ifort and its not clear yet if they can switch. Also CCing @amametjanov

Are the in-progress prod runs using version-of-the-day master? Or are they working on a particular tag? Imho, every production campaign should work on a tag, rather than use current master (which can also undergo non-bfb changes).

[tcclevenger]

In your switch, will you also make a older pre-built version of Kokkos availble for ifort users? There are in-progress production runs with ifort and its not clear yet if they can switch. Also CCing @amametjanov

I agree with Luca that any campaigns I think should be on a tag, and so my preference would be to not have a separate Kokkos, unless there was major pushback.

[tcclevenger]

As far as different compilers, intel won't work for F2010 or F2010-SCREAMv1, and right now gnu has an internal compiler error (ice) inside EKAT. Can we just put oneapi-ifx as the default and say "good luck" to anyone using gnu like we would with intel, or do we need to try and update gnu or solve the ice for gnu?

[rljacob]

the gnu compiler on Chrysalis is used for ice sheet model (MALI) development. Either they need to also switch to oneapi or yes the gnu issue has to be solved.

[bartgol]

the gnu compiler on Chrysalis is used for ice sheet model (MALI) development. Either they need to also switch to oneapi or yes the gnu issue has to be solved.

One big reason to bump to c++20 is b/c MALI would like to follow trilinos more closely. So I think they'd be ok with switching. Pinging @mperego.

Another option could be to bump gnu's version on chrys from 11 to something like 13 (which I think is what pm-cpu uses?).

[rljacob]

OMEGA also gets its Kokkos from EKAT so may need to chime in on this. @amametjanov should know what compilers/machines matter for OMEGA.

[bartgol]

@tcclevenger I think it needs a full rebase. I tried the "draft+undraft" trick, but the workflow files are evaluated on the PR branch itself, not the merge commit.

[Copilot]

Pull request overview

This PR updates EKAT’s build configuration to require C++20 to support upgrading to
Kokkos 5.0.

Changes:

• Set the project-wide C++ standard to C++20 in the top-level CMake.
• Remove the per-target C++ standard requirement from ekat_core.
• Update the spdlog FetchContent commit and remove redundant CI C++ standard
  configuration.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
CMakeLists.txt	Sets C++20 requirement at the project level.
src/core/CMakeLists.txt	Removes per-target C++ standard requirement from ekat_core.
cmake/tpls/EkatBuildSpdlog.cmake	Bumps the pinned spdlog commit SHA (noted as 1.17.0).
cacts.yaml	Drops redundant CMAKE_CXX_STANDARD CI configuration.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tcclevenger]

@copilot Now that C++20 is required at the top level, is this not redundant?

[Copilot]

Not quite. The top-level CMAKE_CXX_STANDARD makes EKAT itself build as C++20, but it does not advertise that requirement to downstream targets that link ekat::Core/ekat::AllLibs. The PUBLIC compile feature is what propagates the minimum standard through the exported interface, so I restored it in e1d6764.

[bartgol]

@tcclevenger I think we should keep THIS change, and rm the CMAKE_CXX_STANDARD/CMAKE_CXX_STANDARD_REQUIRED vars... They are pointless. All they do is ensuring ALL ekat targets use cxx 20. But virtually all of ekat links against ekat_core anyways...

[bartgol]

Given the latest change by copilot, I think we should rm the cmake vars. Setting targets properties is much more cmake-y than setting global vars.

[bartgol]

@tcclevenger I think we should keep THIS change, and rm the CMAKE_CXX_STANDARD/CMAKE_CXX_STANDARD_REQUIRED vars... They are pointless. All they do is ensuring ALL ekat targets use cxx 20. But virtually all of ekat links against ekat_core anyways...

[tcclevenger]

@bartgol With this version of the PR, compiling with

cmake \
  -DCMake_CXX_COMILER=mpicxx \
  -DEKAT_ENABLE_ALL_PACKAGES=ON \
  -DEKAT_TEST_DOUBLE_PRECISION=TRUE \
  -D EKAT_DEFAULT_BFB=ON 
  $EKAT_DIR

it sets the CXX standard to 17. So it is not picking it up from ekat core.

CMake output:

-- The C compiler identification is IntelLLVM 2025.3.1
-- The CXX compiler identification is IntelLLVM 2025.3.1
-- Cray Programming Environment 2.7.35 C
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/cray/pe/craype/2.7.35/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Cray Programming Environment 2.7.35 CXX
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/cray/pe/craype/2.7.35/bin/CC - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The Fortran compiler identification is IntelLLVM 2025.3.0
-- Cray Programming Environment 2.7.35 Fortran
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /opt/cray/pe/craype/2.7.35/bin/ftn - skipped
-- EKAT version: 2.1.0
-- Installation prefix: /usr/local
WARNING: Setting EKAT_DEFAULT_BFB in an optimized build may invalidate BFBness.
-- Found MPI_C: /opt/cray/pe/craype/2.7.35/bin/cc (found version "4.0")
-- Found MPI_CXX: /opt/cray/pe/craype/2.7.35/bin/CC (found version "4.0")
-- Found MPI_Fortran: /opt/cray/pe/craype/2.7.35/bin/ftn (found version "4.0")
-- Found MPI: TRUE (found version "4.0")
-- Looking for OMPI_MAJOR_VERSION
-- Looking for OMPI_MAJOR_VERSION - not found
-- Looking for MPICH_VERSION
-- Looking for MPICH_VERSION - found
-- EKAT_ENABLE_BOOST_STACKTRACE: OFF
    -> EKAT's assert macros will NOT provide a stacktrace.
-- Looking for feenableexcept
-- Looking for feenableexcept - found
-- Looking for a Kokkos installation ...
-- Looking for a Kokkos installation ... NOT FOUND
-- No Kokkos target already defined. Building locally from submodule
--   Kokkos_SOURCE_DIR: /global/homes/t/tccleve/E3SM-Project/E3SM_Main/E3SM/externals/ekat/cmake/tpls/../../extern/kokkos
--   Kokkos_BINARY_DIR: /pscratch/sd/t/tccleve/e3sm_scratch/pm-cpu/standalone/externals/kokkos
-- Setting default Kokkos CXX standard to 17
-- Kokkos version: 4.5.1
-- The project name is: Kokkos
-- Using internal gtest for testing
-- Configured git information in /pscratch/sd/t/tccleve/e3sm_scratch/pm-cpu/standalone/externals/kokkos/generated/Kokkos_Version_Info.cpp
-- SERIAL backend is being turned on to ensure there is at least one Host space. To change this, you must enable another host execution space and configure with -DKokkos_ENABLE_SERIAL=OFF or change CMakeCache.txt
-- Using -std=gnu++17 for C++17 extensions as feature
-- Built-in Execution Spaces:
--     Device Parallel: NoTypeDefined
--     Host Parallel: NoTypeDefined
--       Host Serial: SERIAL
-- 
-- Architectures:
-- Found TPLLIBDL: /usr/include
-- Using internal desul_atomics copy
-- Experimental mdspan support is enabled
-- Looking for C++ include experimental/mdspan
-- Looking for C++ include experimental/mdspan - not found
-- Looking for C++ include mdspan
-- Looking for C++ include mdspan - not found
-- Using internal mdspan directory /global/homes/t/tccleve/E3SM-Project/E3SM_Main/E3SM/externals/ekat/extern/kokkos/tpls/mdspan/include
-- Kokkos Backends: SERIAL
-- Looking for spdlog ...
-- Looking for spdlog ... NOT FOUND
--   Fetching spdlog via FetchContent
--   spdlog source dir already populated with correct version. Skipping populate...
-- Build spdlog: 1.17.0
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Build type: 
-- Looking for fwrite_unlocked
-- Looking for fwrite_unlocked - found
-- Generating install
-- Looking for yaml-cpp ...
-- Could NOT find yaml-cpp (missing: yaml-cpp_DIR)
-- Looking for yaml-cpp ... NOT FOUND
--   Fetching yaml-cpp via FetchContent
--   yaml-cpp source dir already populated with correct version. Skipping populate...
-- Looking for Catch2 ...
-- Looking for Catch2 ... NOT FOUND
--   Fetching Catch2 via FetchContent
--   Catch2 source dir already populated with correct version. Skipping populate...
-- Configuring done (21.8s)
-- Generating done (1.8s)
-- Build files have been written to: /pscratch/sd/t/tccleve/e3sm_scratch/pm-cpu/standalone

[bartgol]

Ah, Kokkos still defaults to 17. But I wonder if the ekat files are built with c++20...

[tcclevenger]

Ah, Kokkos still defaults to 17. But I wonder if the ekat files are built with c++20...

Yes, it looks like ekat_core was built with c++20

[bartgol]

Ah, Kokkos still defaults to 17. But I wonder if the ekat files are built with c++20...

Yes, it looks like ekat_core was built with c++20

Then I'm ok. We could run some tests forcing kokkos to use c++20, but I suppose it works fine. If ekat uses c++20, then conceptually we did enough. Upstream libs should default to whatever standard they want/need.

[tcclevenger]

@bartgol We have some warnings in yaml-cpp

/global/homes/t/tccleve/E3SM-Project/E3SM_Main/E3SM/externals/ekat/extern/yaml-cpp/include/yaml-cpp/node/detail/node_iterator.h:56:19: warning: 
      'iterator<std::forward_iterator_tag, YAML::detail::node_iterator_value<YAML::detail::node>, long, YAML::detail::node_iterator_value<YAML::detail::node> *,
      YAML::detail::node_iterator_value<YAML::detail::node>>' is deprecated [-Wdeprecated-declarations]
   56 |     : public std::iterator<std::forward_iterator_tag, node_iterator_value<V>,

Maybe these have gone away in newer versions? How interested are we in testing?

[bartgol]

Those warnings have been there for a while. IIRC, the cmake option EKAT_DISABLE_TPL_WARNINGS was introduced precisely to silence these (or maybe the spdlog ones). Unfortunately, I don't think this works as intended, since target_link_libraries does not have a "SYSTEM" option (like target_include_directories). Perhaps we could consider linking yaml-cpp "manually", by calling target_include_directories (with SYSTEM option) and then target_link_libraries. It is a bit underwhelming, but I blame CMake for not offering the SYSTEM option for linking targets...

Edit: actually, that option was to silence warnings when building TPLS, not when TPLs are linked. Do you know what file was being built when that happened?