Need to set `OMEGA_ARCH` when building as part of E3SM

[andrewdnolan]

In order to build with GPU support, we need to set the OMEGA_ARCH CMake variable when building as part of E3SM.

There's a TODO comment about this (Line 530) in OmegaBuild.cmake:

Omega/components/omega/OmegaBuild.cmake

Lines 523 to 536 in 2e5ecd7

	# set build-control-variables for e3sm build
	macro(setup_e3sm_build)
	set(OMEGA_BUILD_TYPE ${E3SM_DEFAULT_BUILD_TYPE})
	set(OMEGA_CXX_COMPILER ${CMAKE_CXX_COMPILER})
	#TODO: set OMEGA_ARCH according to E3SM variables
	set(OMEGA_ARCH "")
	set(OMEGA_BUILD_MODE "E3SM")
	message(STATUS "OMEGA_CXX_COMPILER = ${OMEGA_CXX_COMPILER}")
	endmacro()

When trying to build within e3sm using the craycray-mphipcc compiler on frontier, I would get build errors without adding:

diff --git a/components/omega/OmegaBuild.cmake b/components/omega/OmegaBuild.cmake
index 1474b3e898..db60b98384 100644
--- a/components/omega/OmegaBuild.cmake
+++ b/components/omega/OmegaBuild.cmake
@@ -527,8 +527,25 @@ macro(setup_e3sm_build)
 
   set(OMEGA_CXX_COMPILER ${CMAKE_CXX_COMPILER})
 
-  #TODO: set OMEGA_ARCH according to E3SM variables
-  set(OMEGA_ARCH "")
+  if(NOT DEFINED OMEGA_ARCH OR "${OMEGA_ARCH}" STREQUAL "")
+    if(USE_CUDA)
+      set(OMEGA_ARCH "CUDA")
+
+    elseif(USE_HIP)
+      set(OMEGA_ARCH "HIP")
+
+    elseif(USE_SYCL)
+      set(OMEGA_ARCH "SYCL")
+
+    elseif(compile_threaded)
+      set(OMEGA_ARCH "OPENMP")
+
+    else()
+      set(OMEGA_ARCH "SERIAL")
+
+    endif()
+  endif()
+
   set(OMEGA_BUILD_MODE "E3SM")
 
   message(STATUS "OMEGA_CXX_COMPILER = ${OMEGA_CXX_COMPILER}")
@@ -647,6 +664,10 @@ macro(update_variables)
 
   elseif("${OMEGA_ARCH}" STREQUAL "HIP")
     option(Kokkos_ENABLE_HIP "" ON)
+    # Frontier's MI250X GPUs use the gfx90a/VEGA90A architecture.
+    if("${AMDGPU_TARGETS}" MATCHES "gfx90a" OR "${MACH}" STREQUAL "frontier")
+      option(Kokkos_ARCH_VEGA90A "" ON)
+    endif()
     set(OMEGA_TARGET_DEVICE TRUE)
 
   elseif("${OMEGA_ARCH}" STREQUAL "SYCL")

From looking at eamxx (and EKAT) looks like they have machine specific files cmake files for exactly these variables.

[grnydawn]

Thanks, @andrewdnolan, for identifying this issue. Is there a coupled test case that I can use for validation once I fix it?

[andrewdnolan]

@grnydawn thanks for taking these on! Appreciate it.

Supported compsets are listed here:

Omega/components/omega/cime_config/config_compsets.xml

Lines 12 to 22 in 6ea86a8

	<compset>
	<alias>COMEGA-IAF</alias>
	<lname>2000_DATM%IAF_SLND_DICE%IAF_OMEGA%DATMFORCED_DROF%IAF_SGLC_SWAV</lname>
	<support_level>Experimental, under development</support_level>
	</compset>
	<compset>
	<alias>COMEGA-JRA1p4</alias>
	<lname>2000_DATM%JRA-1p4-2018_SLND_DICE%SSMI_OMEGA%DATMFORCED_DROF%JRA-1p4-2018_SGLC_SWAV</lname>
	<support_level>Experimental, under development</support_level>
	</compset>

Once #507 is merged there will be more, but for now it's just those two C-Cases. As it stands we only support the oQU240 and EC30to60E2r2 in coupled mode, once split-explicit timestepping is in we will add RRS18to6v3, but that too large for quick testing.

In practice, I was using:

./create_newcase \
    --case ... \
    --res TL319_EC30to60E2r2 \
    --compset COMEGA-JRA1p4 \
    --driver mct \
    --machine frontier \
    --compiler craycray-mphipcc

on Frontier to investigate the build issues.

As mentioned we can't build with craygnu-mphipcc on Frontier, because of datm issues as reported in E3SM-Project#8594.

That compset / grid combination, with the Ln5_Vmct test modifier can also be used to run any CIME test, but a plain old case might easier for debugging the builds. (Running anything more than a few steps with a RK4 timestep of 1 minute will take to long to be a usefull tests).

[grnydawn]

PR #526 has been created to resolve this issue.