Add the split-explicit time stepper

[hyungyukang]

This PR introduces a split-explicit time stepper for computationally efficient time integration in Omega.

• Added a framework for split-explicit time stepping.
  • Separated and modularized the baroclinic and barotropic time-stepping components.
• Added two new time steppers: SplitExplicitRK2 and UnsplitRK2.
  • SplitExplicitRK2: split time stepping using RK2 for the baroclinic time stepper
  • UnsplitRK2: unsplit RK2 time stepping.
• Added a barotropic time stepper: Predictor-Corrector
  • Predictor-Corrector: an explicitly subcycled predictor-corrector scheme.
• Added the related split time-stepping configuration options to config/Default.yml.

  TimeIntegration:
    CalendarType: No Leap
    TimeStepper: SplitExplicitRK2
    TimeStep: 0000_00:10:00
   ModeSplitShare:
      BtrTimeStepper: Predictor-Corrector
      BtrTimeStep: 0000_00:00:20
      NTimeStepIteration: 2
      NBclCoriolisIteration: 2
      ReinitSplitVelocity: false

• NormalBarotropicVelocity, NormalBaroclinicVelocity, BarotropicPressureAnomaly arrays were added to State.

Checklist

• Documentation:

  • User's Guide has been updated
  • Developer's Guide has been updated
  • Documentation has been built locally and changes look as expected

• Linting

  • Pre-commit run locally

• Building

  • CMake build does not produce any new warnings from changes in this PR

• Testing

  aurora, oneapi-ifx, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  chrysalis, oneapi-ifx, openmpi

  • CTests Pass
  • Polaris omega_pr Pass

  frontier, craygnu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  frontier, craygnu-mphipcc, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  pm-cpu, gnu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

  pm-gpu, gnugpu, mpich

  • CTests Pass
  • Polaris omega_pr Pass

• Provide relevant details in a comment to the PR titled Testing with the following:

  • Which machines CTest unit tests
    have been run on and indicate that are all passing.
  • The Polaris omega_pr test suite
    has passed, using the Polaris e3sm_submodules/Omega baseline
  • Document machine(s), compiler(s), and the build path(s) used for -p for both the baseline (Polaris e3sm_submodules/Omega) and the PR build
  • Indicate "All tests passed" or document failing tests
  • Document testing used to verify the changes including any tests that are added/modified/impacted.

• New tests:

  • CTest unit tests for new features have been added per the approved design.
  • Polaris tests for new features have been added per the approved design (and included in a test suite)

[hyungyukang]

I still need to do some additional work on omega_pr in Polaris for the split time stepper. I will post the results once that is complete.

So far, I have tested this branch with the baroclinic channel, overflow, and realistic global ocean test (for initial performance measurements), and compared the results against MPAS-Ocean.

• Baroclinic channel (1km)
  • Surface temperature at day 30 for different time step sizes + total KE time series

• Overflow
  • Temperature at 6 hours

• Temporal convergence of time steppers
  • Baroclinic channel test (4 km)
  • Reference solution: $\Delta t=1 s$ for each time stepper
  • Error (l2 norm of 3D normal velocity b/t test and reference) evaluated after 10 minutes of integration

[hyungyukang]

Initial computational performance was measured on Frontier using the realistic global ocean test on the EC30to60E2r2 mesh. The model was integrated for one day using a 30 min time step and a 1 min barotropic time step. The reported runtimes are averaged over three repeated runs.

• Strong scaling

Since the Omega-GPU performance saturates relatively quickly, I compared Omega-GPU and Omega-CPU performance for the three main stages of the split time stepper: Stage 1 (BclVel), Stage 2 (BtrVel), and Stage 3 (ThickTracer).

As expected, the explicity subcycling barotropic velocity update (Stage2: BtrVel, no-patterned red histogram) is a major bottleneck of Omega-GPU performance since it is a communication heavy process.

[hyungyukang]

• Stage-wise runtime comparison between Frontier-GPU and PM-GPU

[hyungyukang]

Additional timing measurements on Frontier GPU nodes for several Stage 2 components, including the computations (*Corr, *Pred, and accum) and halo exchanges, show that receiveUnpack is the dominant contributor to the Stage 2 runtime.

receiveUnpack in Halo.h:

Omega/components/omega/src/base/Halo.h

Lines 851 to 897 in 9da1a59

	Pacer::start("Halo:receiveUnpack", 4);
	while (!AllReceived) {
	for (int INghbr = 0; INghbr < NNghbr; ++INghbr) {
	if (RecvFlags[CurElem][INghbr]) {
	if (!Neighbors[INghbr].Received) {
	MPI_Test(&Neighbors[INghbr].RReq, &Neighbors[INghbr].Received,
	MPI_STATUS_IGNORE);
	if (Neighbors[INghbr].Received) {
	++NRcvd;
	}
	}
	if (Neighbors[INghbr].Received && !Neighbors[INghbr].Unpacked) {
	// If the device buffer will be used in unpackBuffer, but the
	// exchange was done with the host buffer, a deep copy from
	// host to device is needed.
	if (UseDevBuffer && !ExchOnDev) {
	expandBuffer(Neighbors[INghbr].RecvBuffer,
	Neighbors[INghbr].RecvBufferH.size());
	// The number of elements we need to copy is different
	// than the buffer allocation size
	const I4 BufferSize =
	TotSize * Neighbors[INghbr].RecvLists[CurElem].NTot;
	auto CopyRange = Kokkos::make_pair(0, BufferSize);
	deepCopy(Kokkos::subview(Neighbors[INghbr].RecvBuffer,
	CopyRange),
	Kokkos::subview(Neighbors[INghbr].RecvBufferH,
	CopyRange));
	}
	unpackBuffer(Array, INghbr);
	Neighbors[INghbr].Unpacked = true;
	}
	}
	}
	if (NRcvd == NMessages) {
	AllReceived = true;
	}
	++IPass;
	if (IPass == MaxIter) {
	LOG_ERROR("Halo: Maximum iterations reached during halo exchange");
	IErr = -1;
	break;
	}
	}
	Pacer::stop("Halo:receiveUnpack", 4);

[hyungyukang]

Strong scaling across GPU nodes and stage-wise GPU performance, including Frontier-GPU, PM-GPU, and Aurora-GPU:
Thanks a lot, @amametjanov, for running the performance tests on Aurora!