Summary
When a run is continued from a restart and its history stream writes multiple time frames to a single file with IfExists: Append, Omega overwrites the frames written by the previous segment instead of appending after them. The overwritten data is destroyed, no warning or error is emitted, and the run reports success. In the case below a 5-day run followed by a 10-day continuation produced a single output.nc containing only days 6 through 15; days 1 through 5 were lost.
The root cause is that the time axis written to file is measured from the current model clock's start time rather than from a fixed simulation origin, so each restart segment restarts its elapsed-time coordinate at zero. Frame selection on append then matches the new segment's first frame against the old segment's first frame and treats it as a re-run of the same time period.
Severity
Silent data loss. Nothing in the log, the exit status, or the file itself indicates that frames were destroyed. The corruption is only visible if you compare the record count or the SimulationTime<N> attributes against what you expected.
Version
Observed with components/omega on branch overfelt/FCTHigherOrderTendency at commit 4c23f7cb6a. The code paths involved are in the IO infrastructure and appear unrelated to that branch's tracer-advection changes, so this most likely affects develop as well; I have not confirmed that.
Reproducer
Run any configuration whose history stream is multi-frame and appends, for example:
TimeIntegration:
StartTime: 0001-01-01_00:00:00
RunDuration: 0005_00:00:00
IOStreams:
History:
UsePointerFile: false
Filename: output.nc
Mode: write
IfExists: append
Freq: 1
FreqUnits: days
FileFreq: 9999
FileFreqUnits: years
Contents:
- StateLet it finish, writing five daily frames into output.nc. Then continue it from the restart it wrote, changing only the time controls:
TimeIntegration:
StartTime: 0001-01-06_00:00:00
RunDuration: 0010_00:00:00(StartTime must be advanced to the restart time, because StopTime is computed as StartTime + RunDuration in TimeStepper.cpp before the restart read resets the clock. Leaving it at the original origin would make the continuation stop 10 days after 0001-01-01, which is only five days of new simulation.)
Expected: output.nc ends with 15 frames covering days 1 through 15. Observed: output.nc has 10 frames covering days 6 through 15, and is exactly twice the size of the original five-frame file. The time variable holds 86400 ... 864000 with units = "seconds since 0001-01-06 00:00:00", and the global attributes run SimulationTime0 = "0001-01-07_00:00:00" through SimulationTime9 = "0001-01-16_00:00:00". Frame 0, which held day 1, now holds day 6.
Mechanism
IfExists: Append is not the problem on its own. In IO.cpp (openFileWrite, the IfExists::Append case) an existing file is opened with PIOc_openfile and NewFile is returned false, so the file is genuinely opened rather than clobbered. The data loss happens in frame selection afterwards.
In IOStream.cpp:2501 the write path takes TimeInstant StartTime = ModelClock->getStartTime(), and at IOStream.cpp:2517 computes TimeInterval ElapsedTime = SimTime - StartTime. After a restart the model clock's start time is the config StartTime for that segment, so elapsed time is measured from the restart point and the value written into the time coordinate restarts at zero for every segment.
For a multi-frame file that is not newly created, IOStream.cpp then reads the existing time values and picks a frame by comparing them against the incoming elapsed time. At IOStream.cpp:2589 an exact match is interpreted as re-running the same period and the existing frame is overwritten:
if (std::abs(ElapsedTimeR8 - FrameTime) < 1.e-5) { // overwrite
Frame = IFrame;
break;
} else if (ElapsedTimeR8 > FrameTime) { // move to next frame
Frame = IFrame + 1;
} else { // time is earlier but doesn't match any frames
ABORT_ERROR("Existing multiframe file {} appears to be using "
"different time intervals or is from the wrong "
"time period", OutFileName);
}In the reproducer the continuation's first output is day 6, whose elapsed time from its own segment start is 86400. The existing file's frame 0 holds day 1, whose elapsed time from the original segment start is also 86400. They match to the tolerance, so the day-6 state is written over the day-1 state. The same collision occurs for the next four frames. From day 11 onward the incoming elapsed time exceeds every value in the file, the Frame = IFrame + 1 branch is taken, and frames 5 through 9 are appended normally. That is exactly the observed 10-frame file with the first five frames replaced.
Note that the ABORT_ERROR branch already exists for a closely related situation, and it does catch the case where the incoming time is earlier than existing frames without matching them. The failure here is silent only because the collision is exact.
A second, related problem with the same root cause
Even where frames do not collide, the file's time metadata is rewritten to describe the newest segment. Field.cpp:52 builds the coordinate's units string as "seconds since " + StartTimeStr from ModelClock->getStartTime(), and the write path re-enters define mode on every write (IOStream.cpp:2627) and rewrites field metadata. The result is that an appended file ends up with a single time:units attribute referring to the latest segment's origin, applied to frames that were written against a different origin. Any frames that did survive from an earlier segment would therefore be mislabeled in time. A reader has no way to detect this from the file alone.
Suggested directions
I have not implemented or tested any of these; they are offered only to frame the discussion.
The underlying fix is presumably to give the output time axis an origin that is stable across restarts rather than one tied to the current clock's start. That could be the calendar epoch, or a simulation start time recorded at initial startup and carried through the restart file, in which case both the frame matching and the units string become consistent between segments and the existing overwrite-on-match logic keeps its intended meaning of "we really are re-running this period."
A narrower alternative would be to make append-mode writes read the existing file's time:units, convert incoming times into that reference before matching frames, and refuse to rewrite the attribute. That would also turn the mislabeling described above into an explicit error when the references disagree.
Whatever the fix, it seems worth treating a units mismatch between an existing file and the current clock as a hard error rather than silently rewriting it, since that alone would have converted this data loss into an abort.
Workaround
Until this is fixed, avoid having a continuation write into a file produced by a previous segment. Giving the history stream a time-templated filename, or a FileFreq short enough that each segment starts a new file, should keep the segments in separate files and sidestep the frame collision entirely. I have not verified either variant. Copying output.nc aside before submitting a continuation is the reliable precaution.
Summary
When a run is continued from a restart and its history stream writes multiple time frames to a single file with
IfExists: Append, Omega overwrites the frames written by the previous segment instead of appending after them. The overwritten data is destroyed, no warning or error is emitted, and the run reports success. In the case below a 5-day run followed by a 10-day continuation produced a singleoutput.nccontaining only days 6 through 15; days 1 through 5 were lost.The root cause is that the time axis written to file is measured from the current model clock's start time rather than from a fixed simulation origin, so each restart segment restarts its elapsed-time coordinate at zero. Frame selection on append then matches the new segment's first frame against the old segment's first frame and treats it as a re-run of the same time period.
Severity
Silent data loss. Nothing in the log, the exit status, or the file itself indicates that frames were destroyed. The corruption is only visible if you compare the record count or the
SimulationTime<N>attributes against what you expected.Version
Observed with
components/omegaon branchoverfelt/FCTHigherOrderTendencyat commit4c23f7cb6a. The code paths involved are in the IO infrastructure and appear unrelated to that branch's tracer-advection changes, so this most likely affectsdevelopas well; I have not confirmed that.Reproducer
Run any configuration whose history stream is multi-frame and appends, for example:
Let it finish, writing five daily frames into
output.nc. Then continue it from the restart it wrote, changing only the time controls:(
StartTimemust be advanced to the restart time, becauseStopTimeis computed asStartTime + RunDurationinTimeStepper.cppbefore the restart read resets the clock. Leaving it at the original origin would make the continuation stop 10 days after0001-01-01, which is only five days of new simulation.)Expected:
output.ncends with 15 frames covering days 1 through 15. Observed:output.nchas 10 frames covering days 6 through 15, and is exactly twice the size of the original five-frame file. Thetimevariable holds86400 ... 864000withunits = "seconds since 0001-01-06 00:00:00", and the global attributes runSimulationTime0 = "0001-01-07_00:00:00"throughSimulationTime9 = "0001-01-16_00:00:00". Frame 0, which held day 1, now holds day 6.Mechanism
IfExists: Appendis not the problem on its own. InIO.cpp(openFileWrite, theIfExists::Appendcase) an existing file is opened withPIOc_openfileandNewFileis returned false, so the file is genuinely opened rather than clobbered. The data loss happens in frame selection afterwards.In
IOStream.cpp:2501the write path takesTimeInstant StartTime = ModelClock->getStartTime(), and atIOStream.cpp:2517computesTimeInterval ElapsedTime = SimTime - StartTime. After a restart the model clock's start time is the configStartTimefor that segment, so elapsed time is measured from the restart point and the value written into thetimecoordinate restarts at zero for every segment.For a multi-frame file that is not newly created,
IOStream.cppthen reads the existingtimevalues and picks a frame by comparing them against the incoming elapsed time. AtIOStream.cpp:2589an exact match is interpreted as re-running the same period and the existing frame is overwritten:In the reproducer the continuation's first output is day 6, whose elapsed time from its own segment start is
86400. The existing file's frame 0 holds day 1, whose elapsed time from the original segment start is also86400. They match to the tolerance, so the day-6 state is written over the day-1 state. The same collision occurs for the next four frames. From day 11 onward the incoming elapsed time exceeds every value in the file, theFrame = IFrame + 1branch is taken, and frames 5 through 9 are appended normally. That is exactly the observed 10-frame file with the first five frames replaced.Note that the
ABORT_ERRORbranch already exists for a closely related situation, and it does catch the case where the incoming time is earlier than existing frames without matching them. The failure here is silent only because the collision is exact.A second, related problem with the same root cause
Even where frames do not collide, the file's time metadata is rewritten to describe the newest segment.
Field.cpp:52builds the coordinate's units string as"seconds since " + StartTimeStrfromModelClock->getStartTime(), and the write path re-enters define mode on every write (IOStream.cpp:2627) and rewrites field metadata. The result is that an appended file ends up with a singletime:unitsattribute referring to the latest segment's origin, applied to frames that were written against a different origin. Any frames that did survive from an earlier segment would therefore be mislabeled in time. A reader has no way to detect this from the file alone.Suggested directions
I have not implemented or tested any of these; they are offered only to frame the discussion.
The underlying fix is presumably to give the output time axis an origin that is stable across restarts rather than one tied to the current clock's start. That could be the calendar epoch, or a simulation start time recorded at initial startup and carried through the restart file, in which case both the frame matching and the units string become consistent between segments and the existing overwrite-on-match logic keeps its intended meaning of "we really are re-running this period."
A narrower alternative would be to make append-mode writes read the existing file's
time:units, convert incoming times into that reference before matching frames, and refuse to rewrite the attribute. That would also turn the mislabeling described above into an explicit error when the references disagree.Whatever the fix, it seems worth treating a units mismatch between an existing file and the current clock as a hard error rather than silently rewriting it, since that alone would have converted this data loss into an abort.
Workaround
Until this is fixed, avoid having a continuation write into a file produced by a previous segment. Giving the history stream a time-templated filename, or a
FileFreqshort enough that each segment starts a new file, should keep the segments in separate files and sidestep the frame collision entirely. I have not verified either variant. Copyingoutput.ncaside before submitting a continuation is the reliable precaution.