Higher Order Flux Corrected Horizontal Transport. by overfelt · Pull Request #447 · E3SM-Project/Omega

overfelt · 2026-06-26T18:24:13Z

Checklist

cbegeman · 2026-07-02T15:21:08Z

@overfelt I think in the meeting you mentioned success comparing the sphere_transport tests with MPAS-O? When this is ready for review, it would be great to see the convergence plots. Let me know if there are any hang-ups with the tests, as I imagine minimally namelist updates will be needed with this PR. I'm also thinking we would add versions of the tests that use FCT in addition to the current tests that do not, which would be a follow-on polaris PR with a submodule update to this PR after it is merged into Omega.

overfelt · 2026-07-24T15:58:06Z

Testing:
CTest unit tests run on chrysalis.lcrc.anl.gov with no new failures. The three new tests: TEND_PLANE_TEST, TEND_SPHERE_TEST and TEND_PLANE_SINGLE_PRECISION_TEST are calibrated to pass on Chrysalis.

Polaris omega_pr run against baselines on Chrysalis and passed.

The TEND_PLANE_TEST and TEND_SPHERE_TEST test all of the new functions added for higher order FCT.

There is no previous higher order FCT to compare against.

xylar · 2026-07-27T15:49:23Z

Some notes from Claude Opus from testing horizontal FCT on a global configuration

Feedback for the overfelt/FCTHigherOrderTendency PR, from trying the branch
at commit 4c23f7cb6a on a global EC30to60E2r2 case (Polaris
ocean/spherical/realistic_global/.../analysis_members_test, 5 days,
dt = 45 s, 384 MPI tasks on Chrysalis).

Motivation for trying it: Omega's unlimited horizontal tracer advection drives
the global minimum temperature from -1.95 C to -7.0 C over six days on this
configuration and is still falling, while MPAS-Ocean's monotonic limiter holds
its extrema flat over the same run. So this PR is very welcome.

The configuration that worked:

Omega:
  Advection:
    HorzTracerFluxOrder: 3
    Coef3rdOrder: 0.25
    HorzTracerFluxLimiterEnable: true
    HorzTracerFluxLimiterBudgetsEnable: false
    HorzTracerFluxLimiterMonotonicityCheckEnable: false

Two things I hit that might be worth addressing before merge.

1. `HorzTracerFluxOrder: 2` with the limiter enabled uses unallocated arrays

This is the one that would bite a user, because order 2 is what a lot of
existing configurations already have set, so "turn the limiter on" is a
one-line change that silently lands here.

Tendencies.cpp:218-221 sets ForceLowOrder when the order is 2:

if (Order == 2) {
   this->TracerHorzAdv.ForceLowOrder = true;
   this->TracerHorzAdv.Coef3rdOrder  = 0;
}

and then reads the limiter flag independently at Tendencies.cpp:230, with no
check that the two are compatible:

Err += AdvectConfig.get("HorzTracerFluxLimiterEnable", TracerHorzAdv.FCT);

But TracerHorzAdvOnCell::init() returns early on ForceLowOrder
(TendencyTerms.cpp:139-144):

if (ForceLowOrder) {
   // Return when the 2nd-order tracer horz adv
   deepCopy(NAdvCellsForEdge, 0);
   deepCopy(AdvMaskHighOrder, 0);
   return;
}

and the FCT allocation block (HProvInv, HNewInv, HProv, TracerCur,
TracerMax, TracerMin, HighOrderFlx, LowOrderFlx, WorkTend, FlxIn,
...) sits after that early return, at TendencyTerms.cpp:160. So with
HorzTracerFluxOrder: 2 and HorzTracerFluxLimiterEnable: true, FCT is
true, every FCT kernel runs, and all of the FCT arrays are default-constructed
zero-extent views.

I did not run this combination, so I can't say whether it segfaults or
silently produces garbage -- but neither is a good outcome, and there is no
test covering it: TendencyTermsTest.cpp:1209-1210 pairs FCT = true with
ForceLowOrder = false, and :1313-1314 and :1341-1342 pair
ForceLowOrder = true with FCT = false. The true/true corner is untested.

Either of these would fix it:

Reject the combination in Tendencies::init with an OMEGA_REQUIRE
alongside the existing order check -- something like "FCT requires
HorzTracerFluxOrder >= 3". Cheapest, and gives the user a clear message.
Or move the if (FCT) allocation block above the ForceLowOrder early
return, if order-2 FCT is meant to be supported (blending a 2nd-order
high-order flux against the upwind low-order flux is what MPAS-Ocean does
at config_horiz_tracer_adv_order = 2 with config_flux_limiter = 'monotonic', so users coming from MPAS-Ocean may well expect it to work).

A note in doc/userGuide/TendencyTerms.md next to
HorzTracerFluxLimiterEnable would help either way -- the table currently
documents the orders and the limiter independently, so there is nothing
telling the reader they interact.

2. `printf` in `FCTMonotonicityCheck` has three specifiers and two arguments

TendencyTerms.h:729-736:

if (TracerCur(ICell, K) < TracerMin(ICell, K) - Eps) {
   printf("Horizontal minimum out of bounds on tracer: %i %lg %lg\n",
          TracerMin(ICell, K), TracerCur(ICell, K));
}
if (TracerCur(ICell, K) > TracerMax(ICell, K) + Eps) {
   printf("Horizontal maximum out of bounds on tracer: %i %lg %lg\n",
          TracerMax(ICell, K), TracerCur(ICell, K));
}

Three conversion specifiers, two arguments, and the first argument is a Real
being consumed by %i. That is undefined behaviour and the printed numbers
will not be the intended ones. I assume the intent was ICell (or ICell, K)
followed by the two values.

Two smaller points on the same function while you are in there:

It prints once per offending cell per layer per time step. On this
configuration, with the overshoots that exist today, that would be a very
large amount of output -- I left the check disabled for that reason rather
than because of the format string. A per-step count, or a first-N-then-stop
guard, would make it usable on a global mesh.
A printf inside a Kokkos kernel is fine for a debug tool but will serialise
badly on GPU; worth a comment saying it is diagnostic-only if that is the
intent.

cbegeman · 2026-07-29T15:10:14Z

@overfelt Are you able to post relevant plots generated by the sphere_transport_with_viz test suite for both MPAS-O and Omega? Let me know if you need any help

cbegeman · 2026-08-03T20:25:46Z

@overfelt Can you rebase when you have a chance? Thanks!

overfelt · 2026-08-04T14:10:37Z

@cbegeman , I rebased the branch.

overfelt · 2026-08-04T14:12:24Z

These are the convergence plots from the ocean/spherical/icos/rotation_2d Polaris test. There are three tracers advected with mpas-ocean

Same with omega-ocean

xylar · 2026-08-04T14:14:08Z

Closed by accident, I assume?

overfelt · 2026-08-04T14:43:43Z

oops

cbegeman · 2026-08-07T14:40:43Z

Build issues on aurora, gpu:

Successful build with aurora, oneapi-ifxgpu, E3SM-Project/polaris@5546b58 + Omega submodule at c83fe0d

Unsuccessful build with aurora, oneapi-ifxgpu, E3SM-Project/polaris@8aa9ccb, Omega this branch 5e38631

Build log can be found here: /lus/flare/projects/E3SM_Dec/cbegeman/polaris-output/seamount-omega-fct-gpu-20260803/

Claude attributes the fail to https://github.com/overfelt/Omega/blob/d31b35395899bf9ae21671a7fa4abbf725f4dc7e/components/omega/src/ocn/TendencyTerms.h#L729-L744:

CUDA and HIP tolerate device-side printf; the Intel SYCL (DPC++) front end does not allow variadic calls in kernels at all, so this only shows up on the oneapi-ifxgpu / OMEGA_ARCH=SYCL Aurora build. That's why the same source builds fine on other machines/backends.

mwarusz · 2026-08-10T16:23:55Z

@overfelt

I ran CTests on pm-cpu with gnu and the following tests failed with a seg fault

19:IOSTREAM_TEST
27:STATE_TEST
42:FILL_VALUE_TEST
47:ANALYSIS_OP_TEST
48:ANALYSIS_SYS_TEST

The seg faults happen because the vertical advection module is not initialized in these tests. I opened #503 to fix this issue and to add checks for module dependencies.

CUDA and HIP tolerate device-side printf; the Intel SYCL (DPC++) front end does not allow variadic calls in kernels at all, so this only shows up on the oneapi-ifxgpu / OMEGA_ARCH=SYCL Aurora build. That's why the same source builds fine on other machines/backends.

You probably know this, but the fix is to use Kokkos::printf.

mwarusz

@overfelt

In addition to what we discussed offline, I have some minor suggestions and comments.

Give LinearEos and ConstantEos the same calcSpecVolDerivs entry point the TEOS-10 functor now has, so the higher-order pressure gradient can be run with any of the three equations of state rather than only with TEOS-10. Both are closed form. For the linear EOS the derivatives with respect to conservative temperature and absolute salinity are -DRhodT and -DRhodS times the square of the specific volume, and there is no pressure dependence at all; for the constant EOS all three vanish. The idealized Polaris cases that use these options therefore get an exactly known reference to test the pressure gradient against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Add the array-level entry point that the higher-order pressure gradient will call, dispatching on the configured EOS to the calcSpecVolDerivs kernel of the corresponding functor and filling the specific volume and its three first derivatives in one pass. Eos owns the SpecVolDCt, SpecVolDSa and SpecVolDP arrays just as it owns SpecVol, allocating them in the constructor and registering them as fields in the Eos group so they can be written to a stream. Their valid range spans the full range of Real rather than starting at zero, since the salinity derivative is negative everywhere and the temperature derivative is negative in cold, nearly fresh water. Since computeSpecVolAndDerivs fills SpecVol as well, it replaces a call to computeSpecVol rather than accompanying one. The two are kept separate because the derivatives roughly double the TEOS-10 arithmetic per cell and layer, and only the higher-order pressure gradient needs them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Check the new derivatives against the unmodified GSW-C library over a cross product of 216 states spanning the oceanographic range and its corners, including fresh water, sub-zero temperatures and the full pressure range. Add an array-level test over the mesh with a state that varies with depth, so the device kernel and the vertical chunking are exercised over a range of values rather than a single one, and closed-form tests for the linear and constant options. Also compare the thermal expansion and haline contraction coefficients used by the Brunt-Vaisala frequency against gsw_specvol_alpha_beta. Those two functions were previously exercised only through a single hardcoded BruntVaisalaFreqSq value at a tolerance too loose to catch a mistake in either; the check is added before the following commit rewrites them. Measured agreement with GSW-C: 2.4e-14 for the specific volume, 3.1e-14 for the temperature derivative, 8.5e-15 for the salinity derivative, and 2.4e-14 for alpha and beta. The pressure derivative agrees only to 2.3e-12 and is gated separately at 1e-10. The discrepancy is GSW-C's: its v_P comes from a table of coefficients pre-multiplied by their pressure exponents and rounded, so it departs from the exact derivative of the 75-term polynomial by about 2e-12 at 10000 dbar, growing with pressure. Evaluating the exact derivative in 60-digit arithmetic puts the Omega value within 1e-16 of it and GSW-C's at 1.8e-12, and rounding the coefficient table to 11 digits reproduces GSW-C's error pattern. The looser gate therefore bounds GSW-C's rounding rather than ours. Add a finite-difference check as well. It is redundant with the GSW-C comparison while that library is present and correct, which is the point: it pins the unit convention of the Omega interface, per degC, per (g/kg) and per Pa, without reference to GSW, and would catch a pressure derivative that silently became per dbar. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Rewrite Teos10BruntVaisalaFreqSq::calcAlpha and calcBeta over the TEOS-10 derivative helpers and delete the A and B coefficient tables, which were a second copy of the same 100 constants. Alpha is the temperature derivative of the specific volume divided by the specific volume and beta is minus the salinity derivative divided by it, so both follow directly. The coefficient assemblers are made static, since they carry no state, so the frequency functor can reach them without holding an equation of state instance. This changes answers at roundoff level: the same polynomial is evaluated in a different Horner arrangement, and the normalized salinity is now formed as sqrt((Sa + DeltaS) / SaNorm) as calcPCoeffs does it rather than from a separately rounded reciprocal. Measured against the GSW-C library over the 216 test states, the maximum relative difference moves from 2.4e-14 to 3.1e-14 for alpha and from 2.4e-14 to 3.2e-14 for beta. BruntVaisalaFreqSq is gated at 1e-10 against a hardcoded value and continues to pass unchanged. The guard test added in the previous commit is what makes this safe: alpha and beta are now pinned to gsw_specvol_alpha_beta over the full state range, so a mistake in the rewrite fails there rather than hiding inside the single frequency value. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Describe computeSpecVolAndDerivs in the developer guide: the signature, the unit convention, that the results land in Eos members registered as fields in the Eos group, that it replaces rather than accompanies computeSpecVol since it fills SpecVol too, and that the Brunt-Vaisala expansion and contraction coefficients are now derived from them. Record the licensing position, since it is the reason the implementation looks the way it does, and the measured agreement with GSW-C, including why the pressure derivative agrees less well than the other two. Add a shorter user guide section noting that the derivatives exist for all three EOS options, that they carry no configuration of their own, that they cost nothing unless a scheme that needs them is enabled, and that they are available for output like SpecVol. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Rename the single-state entry point to calcSpecVolAndDerivsAtPoint and the vertical-chunk entry point to calcSpecVolAndDerivsInChunk, so the two are told apart at the call site rather than by the presence of "And" in the name. Document what each is for.

Say explicitly that the higher-order pressure gradient will need the derivatives every time step and that computeSpecVolAndDerivs then replaces the computeSpecVol call, while configurations that do not use it keep the cheaper call. Note why there is no displaced counterpart and what adding one would take.

The Brunt-Vaisala frequency evaluates alpha and beta at the interface, from temperature, salinity and pressure averaged across the two adjacent layers, while SpecVolDCt and SpecVolDSa hold the derivatives at the layer centers. Record that in the calcAlpha and calcBeta comments so the duplication does not look accidental.

Both compare against GSW-C, but testEosTeos10Derivs covers the array-level machinery -- dispatch, chunking, layer masking, member arrays and field registration -- on one realistic profile, while checkValueGswcSpecVolDerivs covers the polynomial itself point by point at the corners of the oceanographic range. Say so in each test's comment.

AuxiliaryState::computeMomVertAux is the only caller of computeSpecVol; computeBruntVaisalaFreqSq consumes the SpecVol array rather than recomputing it, and computeSpecVolDisp is a separate evaluation. Describe the actual call graph, and give the two reasons the plain computeSpecVol is still wanted: the centered pressure gradient is the config default, and the VertMix refresh of SpecVol feeds nothing that reads the derivatives.

Vertical chunking is on its way out of Omega (PR E3SM-Project#473 removes it from the auxiliary variables and tendency terms), so this branch should not add more of it. The EOS is not touched by that PR, so remove chunking here across the board rather than leaving the new specific volume derivative code to be converted later. The functors now follow the same pattern as the de-chunked auxiliary variables: each takes a TeamMember and a cell index and loops over the active layers with parallelForInner(Team, Range{KMin, KMax}, INNER_LAMBDA(int K) { ... }); so the callers in Eos.cpp reduce to a single call inside parallelForOuter, with no vertRangeChunked and no inner loop over chunks. Dropping the chunk loop also removes VecLength from the TEOS-10 polynomial helpers. calcPCoeffs, calcPCoeffsDTt, calcPCoeffsDSs, calcDelta, calcDeltaDeriv and calcDeltaDP took arrays sized 6 * VecLength or 5 * VecLength together with a KVec index into them; they now take plain [6] and [5] arrays and no index. That storage was never shared between layers -- the coefficients are recomputed for every layer -- so nothing is lost, and the point-wise calcSpecVolAndDerivsAtPoint and the calcAlpha and calcBeta helpers no longer allocate VecLength times more stack than they use. With the chunk loop gone, the array-level derivative routine is exactly calcSpecVolAndDerivsAtPoint evaluated at each cell and layer, so it now calls it instead of repeating the polynomial evaluation. That leaves a single implementation of the TEOS-10 derivatives. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The InChunk suffix named an implementation detail that no longer exists, and it was flagged in review as unusual and potentially confusing. The auxiliary variables name these routines for the mesh element they act on rather than for the vertical loop structure -- computeVarsOnCell, computeVarsOnEdge, computeVarsOnVertex -- and keep those names through the removal of chunking, so follow that convention here. The point-wise entry point keeps the name calcSpecVolAndDerivsAtPoint; the pair now reads as scalars in and out versus arrays over the mesh. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

sbrus89 · 2026-08-18T14:22:32Z

@mwarusz - thanks for reviewing. Are you ready to approve, or do you have more suggestions?

Co-authored-by: Maciej Waruszewski <mwarusz@igf.fuw.edu.pl>

sbrus89 · 2026-08-20T13:51:30Z

@overfelt, it looks like this is getting close! It seems like the CI is picking up some linting issues.

overfelt · 2026-08-20T14:18:47Z

@sbrus89 It looks like the lint check has problems installing the environment. The "install dependencies" step fails when trying to install conda.

The field used in MPAS-Ocean is vertAleTransportTop which maps to TotalVerticalPseudoVelocity in Omega.

sbrus89

@overfelt, I'm seeing 3 CTest fails on Frontier (craygnu). Most of these are probably just tolerances that need to be recalibrated, but there are some inf values reported.

TEND_PLANE_TEST

[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTHProvInv LInf FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTHProvInv L2 FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTHNewInv LInf FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTHNewInv L2 FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTHighAndLowOrderFlux_High LInf FAIL, expected 6.938893903907228e-18, got 5.204170427930421e-18
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTHighAndLowOrderFlux_High L2 FAIL, expected 5.762546266224026e-18, got 3.645353341218854e-18
[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 LInf FAIL, expected 1.2480435640527313e-15, got 1.1987786865243336e-15
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 L2 FAIL, expected 6.665476716753009e-16, got 5.663839253635848e-16

TEND_PLANE_SINGLE_PRECISION_TEST

[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTHProv LInf FAIL, expected 0, got 1.5258789e-05
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTHProv L2 FAIL, expected 0, got 1.5258789e-05
[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTHProvInv LInf FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTHProvInv L2 FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTHNewInv LInf FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTHNewInv L2 FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTHighAndLowOrderFlux_Low LInf FAIL, expected 1.0095554e-15, got 0
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTHighAndLowOrderFlux_Low L2 FAIL, expected 1.0095554e-15, got 0
[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTHighAndLowOrderFlux_High LInf FAIL, expected 6.938894e-18, got 1.8626451e-09
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTHighAndLowOrderFlux_High L2 FAIL, expected 5.7625463e-18, got 1.643791e-09
[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 LInf FAIL, expected 1.2480435e-15, got 7.405691e-07
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 L2 FAIL, expected 6.665477e-16, got 3.0637568e-07

TEND_SPHERE_TEST

[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTHProvInv LInf FAIL, expected 3.0541724683419424e-05, got inf 
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTHProvInv L2 FAIL, expected 1.0779233406323438e-06, got inf 
[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTHNewInv LInf FAIL, expected 3.0541724683419424e-05, got inf 
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTHNewInv L2 FAIL, expected 1.0779233406323438e-06, got inf 
[error] [OceanTestCommon.h:676] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 LInf FAIL, expected 1.632639958580968e-15, got 1.521323597768628e-15
[error] [OceanTestCommon.h:681] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 L2 FAIL, expected 7.582979124216783e-16, got 7.094408556615681e-16

overfelt · 2026-08-26T18:17:39Z

@sbrus89 , I'll see if I can still access Frontier and try to replicate these failures. Some of the failing tests look like tolerances that can just be adjusted for Frontier but others are significant diffs.

sbrus89 · 2026-08-26T19:11:58Z

Sounds good @overfelt, I'm also seeing similar fails on Frontier with craygnu-mphicc in case it's helpful for adjusting tolerances:

TEND_PLANE_TEST

[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTHProvInv LInf FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTHProvInv L2 FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTHNewInv LInf FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTHNewInv L2 FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTHighAndLowOrderFlux_High LInf FAIL, expected 6.938893903907228e-18, got 1.0408340855860843e-17
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTHighAndLowOrderFlux_High L2 FAIL, expected 5.762546266224026e-18, got 1.2836244918860016e-17
[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 LInf FAIL, expected 1.2480435640527313e-15, got 9.524542988823467e-16
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 L2 FAIL, expected 6.665476716753009e-16, got 5.336197203674528e-16

TEND_PLANE_SINGLE_PRECISION_TEST

[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTHProv LInf FAIL, expected 0, got 1.5258789e-05
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTHProv L2 FAIL, expected 0, got 1.5258789e-05
[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTHProvInv LInf FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTHProvInv L2 FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTHNewInv LInf FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTHNewInv L2 FAIL, expected 0, got inf 
[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTHighAndLowOrderFlux_Low LInf FAIL, expected 1.0095554e-15, got 0
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTHighAndLowOrderFlux_Low L2 FAIL, expected 1.0095554e-15, got 0
[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTHighAndLowOrderFlux_High LInf FAIL, expected 6.938894e-18, got 1.8626451e-09
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTHighAndLowOrderFlux_High L2 FAIL, expected 5.7625463e-18, got 1.7900343e-09
[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 LInf FAIL, expected 1.2480435e-15, got 5.642431e-07
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 L2 FAIL, expected 6.665477e-16, got 2.4672764e-07

TEND_SPHERE_TEST

[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTHProvInv LInf FAIL, expected 3.0541724683419424e-05, got inf 
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTHProvInv L2 FAIL, expected 1.0779233406323438e-06, got inf 
[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTHNewInv LInf FAIL, expected 3.0541724683419424e-05, got inf 
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTHNewInv L2 FAIL, expected 1.0779233406323438e-06, got inf 
[error] [OceanTestCommon.h:677] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 LInf FAIL, expected 1.632639958580968e-15, got 1.5120472343676013e-15
[error] [OceanTestCommon.h:682] TendencyTermsTest: FCTAccumulateHighOrderFlux_0 L2 FAIL, expected 7.582979124216783e-16, got 6.215666716489436e-16

I'll run on pm-cpu/gpu and post any fails I see as well.

overfelt marked this pull request as draft June 26, 2026 18:24

overfelt force-pushed the overfelt/FCTHigherOrderTendency branch from 1403553 to 1b66468 Compare July 8, 2026 13:21

overfelt force-pushed the overfelt/FCTHigherOrderTendency branch 2 times, most recently from ab09d81 to 9e7262f Compare July 22, 2026 14:25

overfelt marked this pull request as ready for review July 24, 2026 15:59

overfelt force-pushed the overfelt/FCTHigherOrderTendency branch from 81dab92 to b9c2d57 Compare July 29, 2026 14:46

overfelt force-pushed the overfelt/FCTHigherOrderTendency branch from 4af06b6 to d0434c3 Compare August 3, 2026 12:43

mwarusz self-requested a review August 3, 2026 15:07

overfelt added 5 commits August 3, 2026 15:26

Convert some initial FCT algorithm from F90 to C++

5093edf

Update user guide

25c55dc

Add dumping of budget fields to output database.

4ef8465

Documentation updates.

0fed5e6

Adding unit tests for horizontal FCT transport.

1d6e3c1

overfelt force-pushed the overfelt/FCTHigherOrderTendency branch from d0434c3 to 45ee3e9 Compare August 3, 2026 21:28

overfelt closed this Aug 4, 2026

overfelt reopened this Aug 4, 2026

mwarusz reviewed Aug 11, 2026

View reviewed changes

overfelt and others added 2 commits August 17, 2026 06:37

Initialize Halo elements for unit tests

90b44bf

Initialize TimeStep member in AuxiliaryState

d702607

xylar and others added 12 commits August 17, 2026 06:37

overfelt force-pushed the overfelt/FCTHigherOrderTendency branch from eb73410 to 87d8ead Compare August 17, 2026 12:37

overfelt added 3 commits August 17, 2026 06:38

Merge branch 'E3SM-Project:develop' into overfelt/FCTHigherOrderTendency

0d86853

Remove Team.team_barrier() and Kokkos::fence(). Omega does not need them

12e1c2c

Adjust array sizes to be consistent

1ef3363

overfelt and others added 5 commits August 18, 2026 16:57

Modify TendencyTermsPlane test

e87adea

Update test results to pass on Chrysalis

78c7011

Merge branch 'E3SM-Project:develop' into overfelt/FCTHigherOrderTendency

f169174

Update components/omega/src/ocn/TendencyTerms.cpp

03015b8

Co-authored-by: Maciej Waruszewski <mwarusz@igf.fuw.edu.pl>

Changes from code review feedback

c92160c

overfelt added 3 commits August 20, 2026 16:23

Fix up Sphere tests for FCT

6f101ea

Change VerticalPseudoVelocity to TotalVerticalPseudoVelocity in FCT.

1321ec4

The field used in MPAS-Ocean is vertAleTransportTop which maps to TotalVerticalPseudoVelocity in Omega.

Clang format

096264a

sbrus89 reviewed Aug 26, 2026

View reviewed changes

Uh oh!

Conversation

overfelt commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cbegeman commented Jul 2, 2026

Uh oh!

overfelt commented Jul 24, 2026

Uh oh!

xylar commented Jul 27, 2026

Some notes from Claude Opus from testing horizontal FCT on a global configuration

1. HorzTracerFluxOrder: 2 with the limiter enabled uses unallocated arrays

2. printf in FCTMonotonicityCheck has three specifiers and two arguments

Uh oh!

cbegeman commented Jul 29, 2026

Uh oh!

cbegeman commented Aug 3, 2026

Uh oh!

overfelt commented Aug 4, 2026

Uh oh!

overfelt commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xylar commented Aug 4, 2026

Uh oh!

overfelt commented Aug 4, 2026 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cbegeman commented Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build issues on aurora, gpu:

Uh oh!

mwarusz commented Aug 10, 2026

Uh oh!

mwarusz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbrus89 commented Aug 18, 2026

Uh oh!

sbrus89 commented Aug 20, 2026

Uh oh!

overfelt commented Aug 20, 2026

Uh oh!

sbrus89 left a comment

Choose a reason for hiding this comment

Uh oh!

overfelt commented Aug 26, 2026

Uh oh!

sbrus89 commented Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

overfelt commented Jun 26, 2026 •

edited

Loading

1. `HorzTracerFluxOrder: 2` with the limiter enabled uses unallocated arrays

2. `printf` in `FCTMonotonicityCheck` has three specifiers and two arguments

overfelt commented Aug 4, 2026 •

edited

Loading

overfelt commented Aug 4, 2026 via email •

edited

Loading

cbegeman commented Aug 7, 2026 •

edited

Loading