History streams cannot write an initial-time (t=0) record

[xylar]

Summary

There is currently no way to get an Omega output stream to write a record for the initial state. IOStream::writeAll() is invoked from exactly one place — inside the time-stepping loop in ocnRun, after doStep() — so the earliest record any write stream can produce is at t = dt, never at t = 0.

Details

In src/ocn/OceanRun.cpp, the only call is at line 68, within the while loop:

while (Err == 0 && !(EndAlarm->isRinging())) {
   ++IStep;
   ...
   DefTimeStepper->doStep(DefOceanState, SimTime);
   // write restart file/output, anything needed post-timestep
   IOStream::writeAll(OmegaClock);
   ...
}

There is no writeAll() (or equivalent per-stream write()) call before the loop begins, so neither of the two mechanisms that might otherwise produce an initial record can fire:

1. Interval alarms. For standard time units, IOStream::create builds Alarm(AlarmName, AlarmInt, ClockStart) (src/infra/IOStream.cpp:494). The first ring is one interval after the clock start, so an interval-based History stream never emits a t = 0 frame regardless of Freq.

2. FreqUnits: OnStartup. The flag is parsed and stored (src/infra/IOStream.cpp:499), and writeStream honors it via the StartupShutdown condition (src/infra/IOStream.cpp:2480-2482), but because the first writeAll() happens after step 1 has already completed, an OnStartup write stream produces a record stamped t = dt rather than t = 0. OnStartup is therefore usable for read streams (HorzMeshIn, InitialState, InitialVertCoord) but not meaningfully for write streams.

Impact

MPAS-Ocean output streams do write a record at the reference time, so an MPAS-Ocean output.nc contains both the initial and final states while the corresponding Omega output.nc contains only the final state. This asymmetry breaks any post-processing that compares the first and last frames of a single output file.

Concretely, in Polaris this makes conservation checks vacuous for Omega. polaris/ocean/model/ocean_model_step.py implements property checks such as 'mass conservation' by evaluating a quantity at Time=0 and Time=-1 of the model output and comparing. For a convergence task where the output interval equals the run duration, the Omega file holds a single frame, so the check compares that frame against itself and trivially reports zero drift. The same check is meaningful for MPAS-Ocean. We hit this while porting the geostrophic (Williamson case 2) task to Omega in E3SM-Project/polaris#622 and chose to leave the conservation check disabled rather than include one that passes for the wrong reason.

Suggested fix

Call IOStream::writeAll(OmegaClock) once before entering the time loop in ocnRun (or equivalently at the end of ocnInit, after all fields have been initialized and attached). That would let OnStartup write streams emit a genuine initial-state record, and would let a History stream optionally capture t = 0 in addition to its interval-based frames.

The multi-frame append logic in writeStream already looks like it would handle the extra frame correctly: it reads the elapsed time of each existing frame and either overwrites a matching one or appends the next, so a t = 0 frame written first and a t = Freq frame written later should land as frames 0 and 1 of the same file.

Open questions

• Are all fields that a History stream might reference guaranteed to be computed and attached at the point where a pre-loop writeAll() would run, or would some diagnostics be written as fill values? The answer is almost certainly "no", and we would need the ability to compute diagnostic fields on startup to support this capability.
• For restart runs, should a startup write be suppressed to avoid duplicating the record at the restart time, or is the existing overwrite-on-matching-elapsed-time path in writeStream sufficient? My guess is that we would want to avoid writing on restart.

[xylar]

An alternative is that we make clear that writing at t = 0 is intentionally not supported in Omega, in which case Polaris and other tools need to use the fields available in the initial condition for conservation checks, not the first and last entry in output.

[philipwjones]

@xylar I ran into a similar (but opposite issue) while trying to fix #482 where resetting the clock after restart was causing an unexpected restart write at the time step after restart. I fixed some alarm issues, but the reset will still set alarms at the next ring interval and not at the current time, so this issue will still exist. And unfortunately, the OnStartup is used as a stream frequency so will conflict with writing periodic streams. The cleanest option would probably be to separate the OnStartup into its own stream flag that can coexist with other freq options and then add a call at the end of Init to write any startup. Another option would be to use the forceWrite feature to force a write at the end of init for those streams we would like to write (this would kinda hard-wire the startup writes for those streams). A third option would be to create duplicate OnStartup write streams for those streams we would like to dump on init. All options would require the addition of a writeAll at the end of Init though.