diff --git a/components/omega/doc/design/DynamicInputStreams.md b/components/omega/doc/design/DynamicInputStreams.md new file mode 100644 index 000000000000..154bc790bbe8 --- /dev/null +++ b/components/omega/doc/design/DynamicInputStreams.md @@ -0,0 +1,471 @@ +(omega-design-DynamicInputStreams)= +# Dynamic Input Streams + +**Table of Contents** +1. [Overview](#1-overview) +2. [Requirements](#2-requirements) +3. [Algorithmic Formulation](#3-algorithmic-formulation) +4. [Design](#4-design) +5. [Verification and Testing](#5-verification-and-testing) + +## 1 Overview + +Omega's IOStreams mechanism requires all fields to be pre-registered in the Metadata system +before they can be read. This is appropriate for model state variables whose names, types, and +dimensions are fixed at compile time. However, some analysis capabilities require reading +**weight fields** — region masks, transect edge-sign masks, and similar arrays — whose names +and secondary dimensions are not known until runtime. These fields are user-supplied in +external input files and are not part of the Omega variable set. + +DynamicInputStreams extends the existing IOStreams mechanism to support reading such fields. +When a stream is marked with `dynamicFields: true`, Omega discovers each requested field's +dimensions and type directly from the input file, dynamically registers the necessary metadata +and dimensions, allocates storage, and reads the data. The resulting fields are registered in +the same global field registry as all other Omega fields, making them transparently available +to Analysis operators without any special handling. + +The primary use case is the Atlantic Meridional Overturning Circulation (MOC) analysis, which +requires: +- A region mask on cells: which cells belong to each ocean basin. +- A transect edge-sign mask on edges: which edges lie on the basin's southern boundary, and + in which direction the edge normal points relative to the transect. + +The design is general, however, and supports any field indexed on cells, edges, or vertices +with an optional secondary non-mesh dimension. + +## 2 Requirements + +### 2.1 Requirement: Read fields not pre-registered in Metadata + +A stream marked with `dynamicFields: true` must be able to read fields that have not been +pre-registered in Omega's Metadata system. Field metadata, dimensions, and types are +discovered from the input file at initialization time. This avoids the need to hard-code +field names or dimensions in Omega source code. + +### 2.2 Requirement: Runtime dimension discovery + +For each dynamic field, Omega must inspect the input file to determine the field's dimensions +and native data type. Secondary dimensions (e.g., NMocBasins) that are not standard +Omega mesh dimensions are registered in the global MetaDim system so that downstream output +streams can reference them. + +### 2.3 Requirement: Exactly one mesh dimension per field + +Each dynamic field must have exactly one mesh dimension (NCells, NEdges, or NVertices) and at +most one secondary non-mesh dimension. Fields lacking a mesh dimension (e.g., scalar or 1D +region-only arrays) are outside the scope of this design. This constraint ensures that +existing SCORPIO decompositions can be used for the distributed mesh dimension. + +### 2.4 Requirement: Dimension name conflict rules + +When a secondary dimension name is encountered during reading: +- If no MetaDim with that name exists: create it with the size from the file. +- If a MetaDim with that name exists and the size matches: reuse it (silent deduplication). +- If a MetaDim with that name exists but the size differs: exit with a hard error. + +Input files should use unique, descriptive dimension names (e.g., `NMocBasins`) to avoid +unintended collisions between unrelated streams. + +### 2.5 Requirement: Field name collision is a hard error + +If a dynamic stream attempts to register a field whose name already exists in +`MetaData::AllFields` (whether from another dynamic stream or from the pre-registered Omega +field set), initialization must exit with a clear error message. + +### 2.6 Requirement: Integration with global field registry + +Dynamic fields are registered in `MetaData::AllFields` using the same `ArrayMetaData` and +`IOField` machinery as model state fields. Analysis operators that depend on dynamic fields +list them in `getInputFieldNames()` exactly as they would list any other model field. No +special handling is required in the Analysis orchestrator. + +### 2.7 Requirement: Default to R8 storage + +Integer fields (I4, I8) from the input file are promoted to R8 when stored in Omega. This +allows Analysis operators to treat weight fields uniformly with other floating-point fields. +R4 fields from the file are also stored as R8. + +### 2.8 Requirement: Mesh dimension distributed, secondary dimension replicated + +The mesh dimension of each dynamic field follows the standard Omega parallel decomposition: +each MPI task holds only the local portion of cells, edges, or vertices. The secondary +dimension (e.g. NMocBasins) is not distributed — every task holds all values across the +secondary dimension. SCORPIO decompositions for the mesh dimension are reused from the +existing IOEnv; 2D dynamic fields with a secondary dimension require dynamically created +decompositions (see Section 3). + +### 2.9 Requirement: Re-read on every initialization + +Dynamic streams are re-read on every model initialization, including restarts. Because these +fields are not written to the restart file, they must be reloaded from the original input +file each time the model starts. + +### 2.10 Requirement: Initialization after mesh, before Analysis + +Dynamic input streams must be read after Omega's mesh initialization (so that NCells, NEdges, +NVertices MetaDims and SCORPIO decompositions are available) and before the Analysis +orchestrator is constructed (so that all weight fields are in the registry when operators +resolve their dependencies). + +### 2.12 Requirement: Automatic reading without hard-coded stream names + +The model must provide a single initialization call that reads all streams marked +`DynamicFields: true` automatically, without the calling code knowing stream names. +This allows users to add new dynamic streams to `omega.yml` at runtime without +modifying any source code. + +### 2.13 Desired: String name arrays (deferred) + +Support for reading associated string name arrays (e.g., `regionNames(NMocBasins, StrLen)`, +`transectNames(NMocBasins, StrLen)`) is deferred to a future design iteration. When added, +name arrays would be registered as fields or as metadata attributes on the corresponding +numeric field. + +## 3 Algorithmic Formulation + +### 3.1 Dynamic Field Registration + +The dynamic registration algorithm is invoked during `IOStream::Read` for each field in the +`contents` list when `dynamicFields` is true. + +**Algorithm**: `IOStream::registerDynamicField` + +**Input**: open file ID, field name string, IOEnv + +**Output**: IOField registered in `MetaData::AllFields`; storage allocated; data read + +1. **Query field info from file** using SCORPIO (`PIOc_inq_varid`, `PIOc_inq_varndims`, + `PIOc_inq_vardimid`, `PIOc_inq_vartype`). Obtain dimension names and lengths, and the + native type. + +2. **Classify dimensions**. For each dimension of the field: + - If the dimension name matches a known Omega mesh dimension (NCells, NEdges, NVertices): + record it as the mesh dimension. + - Otherwise: treat it as a secondary dimension. + +3. **Validate structure**. Verify: + - Exactly one mesh dimension is present. + - At most one secondary dimension is present. + - Exit with error if either condition is violated. + +4. **Register secondary dimension** (if present): + - Look up the dimension name in `MetaDim::AllDims`. + - If absent: call `MetaDim::create(dimName, dimLength)`. + - If present with matching length: reuse it. + - If present with different length: exit with error. + +5. **Create ArrayMetaData** for the field: + - Name: the field name string from `contents`. + - Description, units, standard name: read from file variable attributes if present; + otherwise use empty strings. + - Dimensions: mesh MetaDim (first) then secondary MetaDim (if present). + - ValidMin, ValidMax, FillValue: read from file attributes if present; otherwise defaults. + +6. **Check for name collision** in `MetaData::AllFields`. If the name already exists, exit + with error. + +7. **Allocate storage**. Allocate a Kokkos array of type R8 with shape + `(nLocalMeshDim, nSecondaryDim)` (or `(nLocalMeshDim)` for 1D fields). The local mesh + dimension extent comes from the existing Omega decomposition for that mesh location. + +8. **Create IOField** combining the ArrayMetaData shared pointer and the data array pointer. + Register in `MetaData::AllFields`. + +9. **Build or reuse SCORPIO decomposition** for reading: + - For a 1D mesh field: use the existing 1D decomposition in IOEnv for that mesh location + and data type. + - For a 2D field `(nMesh, nSecondary)`: check `IOEnv::dynamicDecomps` for a cached entry + keyed on `(meshLocation, nSecondary, R8)`. If absent, create a new PIO decomposition. + The global offsets for a task owning local cells + $\{c_0, c_1, \ldots, c_{k-1}\}$ (0-based global indices) are: + +$$ +\text{offset}(c_j, r) = c_j \cdot N_{\text{secondary}} + r, \quad r \in [0, N_{\text{secondary}}) +$$ + + Cache the new decomposition for reuse by subsequent fields or restarts. + +10. **Read data** from file using the SCORPIO decomposition. Promote from native file type to + R8 as needed (integer or R4 → R8 conversion applied during or immediately after reading). + +## 4 Design + +### 4.1 Data types and parameters + +#### 4.1.1 Parameters + +No new global configuration parameters are introduced. The `DynamicFields` option is +per-stream and specified inside the `IOStreams` section of the Omega configuration file. + +#### 4.1.2 Class and struct changes + +The `IOEnv` class described in earlier drafts of this design was not created. +Instead, the dynamic decomposition cache is stored as a static member of `IOStream`, +which is simpler and sufficient for the current scope. + +##### IOStream — new members + +```c++ +class IOStream { + private: + // ... existing members ... + + /// If true, fields in this stream are not required to be pre-registered; + /// their metadata and dimensions are discovered from the input file at read time. + bool DynamicFields; + + /// Cache for dynamically-created 2D SCORPIO decompositions keyed on + /// (mesh dimension name, secondary dimension size). All dynamic decompositions + /// use IOTypeR8. Freed in IOStream::finalize(). + static std::map, int> DynamicDecomps; + + // ... rest of existing class ... +}; +``` + +### 4.2 Methods + +#### 4.2.1 IOStream::create — parsing `DynamicFields` + +`IOStream::create()` reads the optional `DynamicFields` boolean from the stream's +YAML config block. If absent, it defaults to `false`. No change to the constructor +signature was required. + +#### 4.2.2 IOStream::validate — bypass for dynamic streams + +```c++ +bool IOStream::validate() { + // Dynamic streams skip field existence checks: fields are registered at read time. + if (DynamicFields) { + Validated = true; + return true; + } + // ... existing validation logic ... +} +``` + +#### 4.2.3 IOStream::readStream — dynamic branch + +The field-processing loop in `IOStream::readStream()` branches on `DynamicFields`: + +```c++ +for (auto IFld = Contents.begin(); IFld != Contents.end(); ++IFld) { + std::string FieldName = *IFld; + if (DynamicFields) { + Error DynErr = registerAndReadDynamicField(InFileID, FieldName); + CHECK_ERROR_ABORT(DynErr, + "IOStream::readStream: Failed to register/read dynamic field {} " + "in stream {}", FieldName, Name); + } else { + std::shared_ptr ThisField = Field::get(FieldName); + int FieldID; + Err += readFieldData(ThisField, InFileID, AllDimIDs, FieldID); + } +} +``` + +Any error returned by `registerAndReadDynamicField` causes an immediate abort via +`CHECK_ERROR_ABORT`. + +#### 4.2.4 IOStream::registerAndReadDynamicField + +New private method implementing the algorithm from Section 3.1: + +```c++ +/// Discovers field metadata from an open file, registers Dimension and Field, +/// allocates R8 storage, builds a SCORPIO decomposition, reads data with +/// type promotion to R8, and fills the field's Kokkos host array. +/// Returns a non-zero Error on any failure. +Error IOStream::registerAndReadDynamicField( + int FileID, ///< open SCORPIO file ID + const std::string &FieldName ///< variable name in file and Omega +); +``` + +#### 4.2.5 IOStream::getOrCreateDynamicDecomp + +New private method managing the static `DynamicDecomps` cache: + +```c++ +/// Returns a cached R8 SCORPIO decomposition for a 2D dynamic field. +/// Creates and caches a new decomposition on first use. +int IOStream::getOrCreateDynamicDecomp( + const std::string &MeshDimName, ///< name of the mesh dimension + I4 NGlobalMesh, ///< global size of mesh dimension + I4 NSecondary ///< size of secondary dimension +); +``` + +The global offset for local mesh index `j` (0-based global index `globalJ`) at +secondary index `r` is `globalJ * NSecondary + r`, matching the row-major layout +of `HostArray2DR8`. + +#### 4.2.6 IOStream::readAllDynamic — automatic reading of all dynamic streams + +New public static method satisfying requirement 2.12: + +```c++ +/// Reads every stream with DynamicFields=true. Call once after HorzMesh::init() +/// and before any code that depends on the resulting fields. +static Error IOStream::readAllDynamic(const Clock *ModelClock); +``` + +The method iterates `AllStreams` and calls `readStream()` for each dynamic stream. +It is called in `initOmegaModulesImpl()` between `HorzMesh::init()` and +`VertCoord::init()`. Users can add any number of dynamic streams to `omega.yml` +without modifying source code. + +#### 4.2.7 IO::getVarInfo — new SCORPIO inquiry wrapper + +New function in `IO.h` / `IO.cpp`: + +```c++ +/// Queries a variable's dimension names, global lengths, and native data type. +Error IO::getVarInfo( + int FileID, + const std::string &VarName, + int &NVarDims, + std::vector &DimNames, + std::vector &DimLengths, + IO::IODataType &NativeType +); +``` + +Uses `PIOc_inq_varndims`, `PIOc_inq_vartype`, `PIOc_inq_vardimid`, +`PIOc_inq_dimname`, and `PIOc_inq_dimlen`. + +### 4.3 Configuration + +Dynamic streams use `DynamicFields: true` (case-sensitive key, boolean value). +The `Precision` field is not meaningful for dynamic read streams and may be omitted; +a pre-existing bug in `IOStream::create()` that caused an abort when `Precision` +was absent was fixed as part of this implementation (see Section 4.6). + +```yaml +Omega: + IOStreams: + MocMasksAndTransects: + UsePointerFile: false + Filename: '/path/to/oQU240_mocBasinsAndTransects.nc' + Mode: read + Freq: 1 + FreqUnits: OnStartup + DynamicFields: true + Contents: + - MocCellMasks + - MocEdgeSigns +``` + +The field names in `Contents` must exactly match the variable names in the netCDF +file. These names become the Omega-internal names used to retrieve the fields. + +### 4.4 Initialization ordering + +Dynamic input streams must be read after mesh initialization and before any +operators that depend on the dynamic fields. This is accomplished automatically +in `initOmegaModulesImpl()` via `IOStream::readAllDynamic()`: + +1. Machine environment and MPI setup +2. Configuration file parsing +3. Decomposition initialization (Decomp) +4. Mesh initialization (HorzMesh) — registers NCells, NEdges, NVertices dimensions +5. **`IOStream::readAllDynamic(ModelClock)`** — reads all dynamic streams, registers dynamic fields and secondary dimensions +6. Vertical coordinate initialization (VertCoord) +7. Analysis orchestrator construction — resolves field dependencies + +Note: the `IOEnv` initialization step in earlier drafts of this design is not +present; no separate `IOEnv` class was created. + +### 4.5 Analysis operator access + +This section is unchanged from the original design. Analysis operators access +dynamic fields by name exactly as they access model state fields. The +AnalysisOrchestrator integration is deferred to the branch where the +AnalysisOrchestrator itself is being implemented. + +### 4.6 Bug fix: `Precision` error accumulation in `IOStream::create()` + +`IOStream::create()` uses a single accumulated `Err` variable for required config +fields. The `Precision` option is optional and falls back to `"double"` when +absent, but the original code did not call `Err.reset()` after the fallback. +This caused the subsequent `CHECK_ERROR_ABORT` for `Freq` to abort even when +`Freq` was correctly present. The fix adds `Err.reset()` to the Precision +fallback, matching the existing pattern for `UseStartEnd`. Any stream that omits +`Precision` from its YAML config now works correctly. + +## 5 Verification and Testing + +Tests are implemented as CTest executables in +`components/omega/test/infra/`. Each test writes its own synthetic netCDF +input files using SCORPIO write functions and does not require pre-committed +binary data files. + +### 5.1 Test: Basic dynamic field read (DYNAMIC_IOSTREAM_TEST) + +Implemented in `DynamicInputStreamTest.cpp`. Writes `DynTest.nc` containing +`MocCellMasks(NCells, NRegions=3)` and `MocEdgeSigns(NEdges, NRegions=3)` as +`I4`, and `DynTest2.nc` containing `MocCellMasks2(NCells, NRegions=3)`. +A single call to `IOStream::readAllDynamic()` reads both streams and verifies: +- All three fields are present in `Field::AllFields`. +- `Dimension::AllDims` contains `NRegions` with global length 3. +- Data values match expected `I4→R8` promoted values for every owned cell/edge. +- The dimension count increases by exactly 1 (NRegions registered once, not twice). + +Tests requirements 2.1, 2.2, 2.3, 2.7, 2.12. + +### 5.2 Test: Field name collision (DYNAMIC_IOSTREAM_COLLISION_TEST — WILL_FAIL) + +Implemented in `DynamicInputStreamCollisionTest.cpp`. Two streams both list +`MocCellMasks`. `IOStream::readAllDynamic()` reads them in alphabetical order: +CollisionStream1 succeeds, CollisionStream2 aborts via `CHECK_ERROR_ABORT` +because `registerAndReadDynamicField` detects `Field::exists("MocCellMasks") == true`. +CTest marks this test `WILL_FAIL` so the non-zero exit counts as a pass. + +Tests requirement 2.5. + +### 5.3 Test: Dimension name conflict — different sizes (DYNAMIC_IOSTREAM_DIM_CONFLICT_TEST — WILL_FAIL) + +Implemented in `DynamicInputStreamDimConflictTest.cpp`. `IOStream::readAllDynamic()` +reads DimStream1 (registers `NRegions=3`) then DimStream2 (file has `NRegions=5`). +The abort fires in `IOStream::readAllDims()` (which checks all registered dimensions +against the file before the dynamic field loop runs) rather than in +`registerAndReadDynamicField` step 4, but the overall behaviour is correct: +initialization aborts with a dimension-conflict error. + +Tests requirement 2.4. + +### 5.4 Test: Dimension deduplication — same size (DYNAMIC_IOSTREAM_TEST) + +Covered in the same `DynamicInputStreamTest.cpp` test as §5.1. The single +`IOStream::readAllDynamic()` call reads DynTest (registers `NRegions=3`) then +DynTest2 (same `NRegions=3`), and verifies the dimension count increases by +exactly 1. + +Tests requirement 2.4. + +### 5.5 Test: Analysis operator dependency resolution + +Deferred. The AnalysisOrchestrator is being developed on a separate branch. +When that work is merged, a test exercising end-to-end operator dependency +resolution through dynamic fields should be added. + +Tests requirement 2.6. + +### 5.6 Test: Restart re-read + +Not implemented as a unit test. Requirement 2.9 is satisfied by architecture: +`ocnInit()` always calls `IOStream::init()` before `IOStream::readAllDynamic()`, +and `IOStream::init()` recreates all stream objects with `OnStartup=true`. +Consequently, `readAllDynamic()` naturally re-reads all dynamic streams on every +model start, including restarts. Verification is available at the +driver/integration test level. + +Tests requirement 2.9 (by architecture). + +### 5.8 Test: Non-mesh field rejection (DYNAMIC_IOSTREAM_BAD_FIELD_TEST — WILL_FAIL) + +Implemented in `DynamicInputStreamBadFieldTest.cpp`. A file contains +`RegionAreas(NRegions=3)` with no mesh dimension. The read aborts in +`registerAndReadDynamicField` when `NumMeshDims == 0` is detected. + +Tests requirement 2.3. diff --git a/components/omega/doc/devGuide/DynamicInputStreams.md b/components/omega/doc/devGuide/DynamicInputStreams.md new file mode 100644 index 000000000000..1f5f4793cfe8 --- /dev/null +++ b/components/omega/doc/devGuide/DynamicInputStreams.md @@ -0,0 +1,135 @@ +(omega-dev-dynamicinputstreams)= + +## Dynamic Input Streams + +Dynamic input streams extend the [IOStream](#omega-dev-iostreams) mechanism to +support reading fields whose names and secondary dimensions are not known at +compile time. When a stream is configured with `DynamicFields: true` in the +input YAML, fields in that stream do not need to be pre-registered in the field +registry. Instead, each field's dimensions and type are discovered from the +open file at read time, the necessary `Dimension` and `Field` objects are created +automatically, storage is allocated, and the data is read. + +Any module requiring this functionality must include the `IOStream.h` header +file. The new `IOStream::readAllDynamic()` call described below replaces +any per-stream `IOStream::read()` calls that would otherwise require the +calling code to know stream names. + +### The `DynamicFields` flag + +The flag is stored as a private `bool DynamicFields` member on each `IOStream` +instance and defaults to `false`. It is set in `IOStream::create()` when the +stream's YAML block contains `DynamicFields: true`; if the key is absent the +stream behaves as a standard IOStream. + +Dynamic streams skip the normal field-existence check in `IOStream::validate()` +because the fields will not exist until the stream is read. The stream is +marked validated immediately so that `IOStream::validateAll()` does not abort. + +### Reading a dynamic stream + +When `IOStream::readStream()` processes a field from a dynamic stream, it calls +the private method `registerAndReadDynamicField()` instead of the standard +`Field::get()` + `readFieldData()` path: + +```c++ +Error DynErr = registerAndReadDynamicField(InFileID, FieldName); +CHECK_ERROR_ABORT(DynErr, + "IOStream::readStream: Failed to register/read dynamic field {} " + "in stream {}", FieldName, Name); +``` + +Any error from `registerAndReadDynamicField` causes an immediate abort with a +descriptive message. + +`registerAndReadDynamicField()` performs the full registration pipeline for one +field: + +1. Call `IO::getVarInfo()` to retrieve the field's dimension names, global + dimension lengths, and native data type from the open file. +2. Classify each dimension. A dimension is a mesh dimension if + `Dimension::exists(name) && Dimension::isDistributedDim(name)` is true; + otherwise it is a secondary dimension. +3. Require exactly one mesh dimension and at most one secondary dimension; + return an error otherwise. +4. If a secondary dimension is present: look it up in `Dimension::AllDims`. + If absent, create it with `Dimension::create(name, length)`. If it already + exists with the same length, reuse it silently. If it exists with a + different length, return an error. +5. Return an error if `Field::exists(fieldName)` is true (name collision). +6. Create the field with `Field::create()` using `TimeDependent=false`. + Because the first dimension is a distributed mesh dimension, + `Field::isDistributed()` will automatically return `true`. +7. Allocate a `HostArray1DR8` or `HostArray2DR8` and attach it to the field + via `Field::attachFieldData()`. +8. Obtain a SCORPIO decomposition for the read. For 1-D mesh fields a + temporary decomposition is built from the mesh dimension's global offsets + and destroyed after use. For 2-D fields `getOrCreateDynamicDecomp()` is + called to obtain a cached decomposition. +9. Read the data via `IO::readArray()`. SCORPIO converts from the native file + type to `R8` during the read. +10. Copy the resulting `R8` buffer into the field's Kokkos host array. + +### Dynamic decomposition cache + +2-D decompositions are cached in the private static member +```c++ +static std::map, int> DynamicDecomps; +``` +keyed on `(meshDimName, nSecondary)`. All dynamic decompositions use +`IOTypeR8`. The global offset for local mesh index `j` (0-based global index +`globalJ`) at secondary index `r` is `globalJ * nSecondary + r`, which matches +the row-major layout of `HostArray2DR8`. + +Cached decompositions survive across restarts within a single process lifetime +and are freed when `IOStream::finalize()` is called. + +### `IO::getVarInfo()` + +The helper function +```c++ +Error IO::getVarInfo(int FileID, + const std::string &VarName, + int &NVarDims, + std::vector &DimNames, + std::vector &DimLengths, + IO::IODataType &NativeType); +``` +wraps `PIOc_inq_varndims`, `PIOc_inq_vartype`, `PIOc_inq_vardimid`, +`PIOc_inq_dimname`, and `PIOc_inq_dimlen` to retrieve all metadata needed to +classify and register a dynamic field. It is declared in `IO.h` and follows +the same error-return conventions as the other IO functions. + +### `IOStream::readAllDynamic()` + +All dynamic streams are read through a single call: +```c++ +Error DynErr = IOStream::readAllDynamic(ModelClock); +CHECK_ERROR_ABORT(DynErr, "Error reading dynamic input streams"); +``` +The method iterates `AllStreams` and calls `readStream()` for every stream with +`DynamicFields=true`. Users can define any number of dynamic streams in +`omega.yml` and they will all be read automatically without modifying source +code. + +In `ocnInit`, this call is placed in `initOmegaModulesImpl()` between +`HorzMesh::init()` and `VertCoord::init()`. Application code that uses +`initOmegaModules()` automatically gets this behavior. Standalone applications +that perform their own initialization must call `readAllDynamic()` explicitly +after `HorzMesh::init()`. + +### Initialization ordering + +`registerAndReadDynamicField()` classifies mesh dimensions by checking +`Dimension::isDistributedDim()`, which returns `true` only for dimensions that +were registered during mesh initialization. Dynamic stream reads must therefore +occur after `HorzMesh::init()` and before any code that depends on the +resulting fields. The recommended sequence is: + +```c++ +Decomp::init(); +HorzMesh::init(); // registers NCells, NEdges, NVertices +Error DynErr = IOStream::readAllDynamic(ModelClock); // reads all dynamic streams +CHECK_ERROR_ABORT(DynErr, "Error reading dynamic input streams"); +// ... construct analysis operators or other consumers ... +``` diff --git a/components/omega/doc/index.md b/components/omega/doc/index.md index 6712bb3e7f8d..efde535c8d8d 100644 --- a/components/omega/doc/index.md +++ b/components/omega/doc/index.md @@ -35,6 +35,7 @@ userGuide/Error userGuide/Field userGuide/IO userGuide/IOStreams +userGuide/DynamicInputStreams userGuide/Halo userGuide/HorzMesh userGuide/HorzOperators @@ -81,6 +82,7 @@ devGuide/Error devGuide/Field devGuide/IO devGuide/IOStreams +devGuide/DynamicInputStreams devGuide/Halo devGuide/HorzMesh devGuide/HorzOperators @@ -113,6 +115,7 @@ design/Config design/DataTypes design/Decomp design/Driver +design/DynamicInputStreams design/EOS design/Error design/Halo diff --git a/components/omega/doc/userGuide/DynamicInputStreams.md b/components/omega/doc/userGuide/DynamicInputStreams.md new file mode 100644 index 000000000000..73cec4978a28 --- /dev/null +++ b/components/omega/doc/userGuide/DynamicInputStreams.md @@ -0,0 +1,79 @@ +(omega-user-dynamicinputstreams)= + +## Dynamic Input Streams + +A regular [IOStream](#omega-user-iostreams) requires every field listed in its +`Contents` to be pre-registered in Omega's field registry before the stream is +read. Dynamic input streams lift this restriction. When a stream is marked +with `DynamicFields: true`, Omega inspects the input file at read time, +discovers each field's dimensions and type, allocates storage, and registers the +field automatically. + +The primary use case is weight fields used by analysis operators — region masks, +transect edge-sign arrays, and similar arrays whose names and secondary dimensions +are not fixed at compile time. After being registered, dynamic fields are +indistinguishable from any other Omega field and can be accessed by name through +the standard field registry. + +### Configuration + +Dynamic streams are placed in the same `IOStreams` section of the Omega input +configuration file as all other streams. The only additional option is +`DynamicFields: true`: + +```yaml +Omega: + IOStreams: + MocMasksAndTransects: + UsePointerFile: false + Filename: /path/to/oQU240_mocBasinsAndTransects.nc + Mode: read + Freq: 1 + FreqUnits: OnStartup + DynamicFields: true + Contents: + - MocCellMasks + - MocEdgeSigns +``` + +The names in `Contents` must match the variable names in the netCDF file +exactly. These names become the Omega-internal names used to retrieve the +fields after reading. + +### Constraints on dynamic fields + +Each field in a dynamic stream must satisfy the following constraints. +Violations abort initialization with a descriptive error message. + +- **Exactly one mesh dimension**: the field must have exactly one dimension that + is a distributed Omega mesh dimension (`NCells`, `NEdges`, or `NVertices`). + Fields that lack a mesh dimension (e.g. a 1-D region-only array) are not + supported. + +- **At most one secondary dimension**: beyond the mesh dimension, at most one + additional non-mesh dimension (e.g. `NMocBasins`) is allowed. + +- **Unique field names**: a dynamic field name must not already exist in + Omega's field registry, whether from another dynamic stream or from model + state variables. + +- **Consistent secondary dimension sizes**: if two streams reference a + secondary dimension by the same name (e.g. `NMocBasins`), the dimension + length must be identical in both files. Use descriptive, unique dimension + names to avoid unintended conflicts between unrelated streams. + +- **Storage type**: all dynamic fields are stored as 64-bit floating-point + (`R8`) regardless of the native type in the file. Integer (`I4`, `I8`) and + single-precision (`R4`) fields are promoted to `R8` when read. + +- **Re-read on every initialization**: dynamic fields are not written to restart + files and must be re-read from the original input file on every model start, + including restarts. + +### Initialization ordering + +Dynamic streams are read automatically during `ocnInit` via +`IOStream::readAllDynamic()`, which is called after mesh initialization. +No explicit read call is needed in the configuration or in model code. +Adding a new `DynamicFields: true` stream to `omega.yml` is sufficient for +it to be discovered and read on the next model start. diff --git a/components/omega/src/base/IO.cpp b/components/omega/src/base/IO.cpp index 65fa2ab23a5a..cd67eb20248e 100644 --- a/components/omega/src/base/IO.cpp +++ b/components/omega/src/base/IO.cpp @@ -768,6 +768,80 @@ Error readArray(void *Array, // [out] array to be read } // End IOReadArray +//------------------------------------------------------------------------------ +// Queries a variable's dimension names, global lengths, and native data type +// from an open file. Returns an error code if the variable is not found or +// if dimension metadata cannot be read. +Error getVarInfo( + int FileID, // [in] ID of open file + const std::string &VarName, // [in] variable name to query + int &NVarDims, // [out] number of dimensions + std::vector &DimNames, // [out] name of each dimension + std::vector &DimLengths, // [out] global length of each dim + IODataType &NativeType // [out] native data type in file +) { + + Error Err; + int PIOErr = 0; + + // Get variable ID + int VarID = -1; + PIOErr = PIOc_inq_varid(FileID, VarName.c_str(), &VarID); + if (PIOErr != PIO_NOERR) + RETURN_ERROR(Err, ErrorCode::Fail, + "IO::getVarInfo: Variable {} not found in file", VarName); + + // Get number of dimensions + PIOErr = PIOc_inq_varndims(FileID, VarID, &NVarDims); + if (PIOErr != PIO_NOERR) + RETURN_ERROR(Err, ErrorCode::Fail, + "IO::getVarInfo: Error getting ndims for variable {}", + VarName); + + // Get native data type + nc_type VarType; + PIOErr = PIOc_inq_vartype(FileID, VarID, &VarType); + if (PIOErr != PIO_NOERR) + RETURN_ERROR(Err, ErrorCode::Fail, + "IO::getVarInfo: Error getting type for variable {}", + VarName); + NativeType = static_cast(VarType); + + // Get dimension IDs + std::vector DimIDs(NVarDims); + PIOErr = PIOc_inq_vardimid(FileID, VarID, DimIDs.data()); + if (PIOErr != PIO_NOERR) + RETURN_ERROR(Err, ErrorCode::Fail, + "IO::getVarInfo: Error getting dimids for variable {}", + VarName); + + // Get dimension names and lengths + DimNames.resize(NVarDims); + DimLengths.resize(NVarDims); + for (int IDim = 0; IDim < NVarDims; ++IDim) { + char DimName[PIO_MAX_NAME + 1] = {'\0'}; + PIOErr = PIOc_inq_dimname(FileID, DimIDs[IDim], DimName); + if (PIOErr != PIO_NOERR) + RETURN_ERROR( + Err, ErrorCode::Fail, + "IO::getVarInfo: Error getting name for dim {} of variable {}", + IDim, VarName); + DimNames[IDim] = DimName; + + PIO_Offset DimLen; + PIOErr = PIOc_inq_dimlen(FileID, DimIDs[IDim], &DimLen); + if (PIOErr != PIO_NOERR) + RETURN_ERROR( + Err, ErrorCode::Fail, + "IO::getVarInfo: Error getting length for dim {} of variable {}", + IDim, VarName); + DimLengths[IDim] = static_cast(DimLen); + } + + return Err; + +} // End getVarInfo + //------------------------------------------------------------------------------ // Reads a non-distributed variable. Uses a void pointer for generic interface. // All arrays are assumed to be in contiguous storage. Returns an error code so diff --git a/components/omega/src/base/IO.h b/components/omega/src/base/IO.h index a8ec08e73f4a..456a4d3ef34a 100644 --- a/components/omega/src/base/IO.h +++ b/components/omega/src/base/IO.h @@ -321,6 +321,18 @@ Error readArray(void *Array, ///< [out] array to be read int Frame = -1 ///< [in] opt frame if multiple time slices ); +/// Queries a variable's dimension names, global lengths, and native data type +/// from an open file. Returns a non-zero error code if the variable is not +/// found or if any dimension metadata cannot be read. +Error getVarInfo( + int FileID, ///< [in] ID of open file + const std::string &VarName, ///< [in] variable name to query + int &NVarDims, ///< [out] number of dimensions + std::vector &DimNames, ///< [out] name of each dimension + std::vector &DimLengths, ///< [out] global length of each dim + IODataType &NativeType ///< [out] native data type in file +); + /// Reads a non-distributed variable. We use a void pointer here to create /// a generic interface for all types. Arrays are assumed to be in contiguous /// storage so the arrays of any dimension are treated as a 1-d array with diff --git a/components/omega/src/infra/IOStream.cpp b/components/omega/src/infra/IOStream.cpp index c78f27ff716b..382ddd30bc5a 100644 --- a/components/omega/src/infra/IOStream.cpp +++ b/components/omega/src/infra/IOStream.cpp @@ -28,6 +28,7 @@ #include #include #include +#include #include #include @@ -35,6 +36,7 @@ namespace OMEGA { // Create static class members std::map> IOStream::AllStreams; +std::map, int> IOStream::DynamicDecomps; //------------------------------------------------------------------------------ // Initializes all streams defined in the input configuration file. This @@ -110,6 +112,11 @@ void IOStream::finalize( // Remove all streams AllStreams.clear(); + // Free any cached dynamic decompositions + for (auto &Entry : DynamicDecomps) + IO::destroyDecomp(Entry.second); + DynamicDecomps.clear(); + return; } // End finalize @@ -180,6 +187,13 @@ bool IOStream::validate() { if (Validated) return ReturnVal; + // Dynamic streams skip field existence checks: fields are not registered + // until readStream is called, which happens after validateAll(). + if (DynamicFields) { + Validated = true; + return ReturnVal; + } + // Expand group names to list of individual fields // First identify any group names in the Contents std::set GroupNames; @@ -252,6 +266,27 @@ bool IOStream::validateAll() { } // End validateAll +//------------------------------------------------------------------------------ +// Reads every stream with DynamicFields=true. Intended to be called once +// during initialization, after HorzMesh::init() has registered the mesh +// dimensions. Returns an accumulated error so the caller can CHECK_ERROR_ABORT. +Error IOStream::readAllDynamic(const Clock *ModelClock // [in] Model clock +) { + + Error Err; + + for (auto Iter = AllStreams.begin(); Iter != AllStreams.end(); ++Iter) { + std::shared_ptr ThisStream = Iter->second; + if (ThisStream->DynamicFields) { + Metadata EmptyMeta; + Err += ThisStream->readStream(ModelClock, EmptyMeta, false); + } + } + + return Err; + +} // End readAllDynamic + //------------------------------------------------------------------------------ // Reads a single stream if it is time. Error IOStream::read( @@ -355,6 +390,7 @@ IOStream::IOStream() { PtrFilename = " "; UseStartEnd = false; Validated = false; + DynamicFields = false; } //------------------------------------------------------------------------------ @@ -434,8 +470,10 @@ void IOStream::create(const std::string &StreamName, //< [in] name of stream // present, assume full (double) precision std::string PrecisionString; Err += StreamConfig.get("Precision", PrecisionString); - if (Err.isFail()) + if (Err.isFail()) { PrecisionString = "double"; + Err.reset(); + } NewStream->setPrecisionFlag(PrecisionString); // Set the action to take if a file already exists @@ -636,6 +674,13 @@ void IOStream::create(const std::string &StreamName, //< [in] name of stream // The contents list has not yet been validated. NewStream->Validated = false; + // Check for DynamicFields flag (optional, defaults to false). + // Dynamic streams discover field metadata at read time; the fields + // need not be pre-registered in the global field registry. + bool DynFields = false; + Error ErrDyn = StreamConfig.get("DynamicFields", DynFields); + NewStream->DynamicFields = ErrDyn.isSuccess() ? DynFields : false; + // If we have made it to this point, we have a valid stream to add to // the list AllStreams[StreamName] = NewStream; @@ -2280,6 +2325,210 @@ Error IOStream::readFieldData( } // End readFieldData +//------------------------------------------------------------------------------ +// Returns a cached SCORPIO decomposition for a 2D dynamic field. The decomp +// is keyed on (mesh dimension name, secondary dimension size). All dynamic +// decompositions use R8 type so SCORPIO handles promotion from the native type. +int IOStream::getOrCreateDynamicDecomp( + const std::string &MeshDimName, // [in] name of the mesh dimension + I4 NGlobalMesh, // [in] global size of mesh dimension + I4 NSecondary // [in] size of secondary dimension +) { + + auto Key = std::make_tuple(MeshDimName, NSecondary); + auto It = DynamicDecomps.find(Key); + if (It != DynamicDecomps.end()) + return It->second; + + // Build the 2D global-offset array. For local cell J with 0-based global + // index GlobalJ, the offsets for secondary indices 0..NSecondary-1 are + // GlobalJ * NSecondary + I (row-major order matching HostArray2DR8 layout). + I4 LocalMeshSize = Dimension::getDimLengthLocal(MeshDimName); + HostArray1DI4 MeshOffset = Dimension::getDimOffset(MeshDimName); + I4 LocalSize = LocalMeshSize * NSecondary; + + std::vector GlobalIndx(LocalSize, -1); + for (int J = 0; J < LocalMeshSize; ++J) { + I4 GlobalJ = MeshOffset(J); + if (GlobalJ < 0) + continue; // ghost / padding cell — excluded from IO + for (int I = 0; I < NSecondary; ++I) { + int LocalAdd = J * NSecondary + I; + GlobalIndx[LocalAdd] = GlobalJ * NSecondary + I; + } + } + + std::vector GlobalDims = {static_cast(NGlobalMesh), + static_cast(NSecondary)}; + int DecompID = IO::createDecomp(IO::IOTypeR8, 2, GlobalDims, LocalSize, + GlobalIndx, IO::DefaultRearr); + DynamicDecomps[Key] = DecompID; + return DecompID; + +} // End getOrCreateDynamicDecomp + +//------------------------------------------------------------------------------ +// Discovers a field's metadata from an open file, registers the secondary +// Dimension (if new and absent), allocates R8 storage, registers the Field, +// builds a SCORPIO decomposition, and reads the data with promotion to R8. +Error IOStream::registerAndReadDynamicField( + int FileID, // [in] open SCORPIO file ID + const std::string &FieldName // [in] variable name in file and Omega +) { + + Error Err; + + // Step 1: Query field info from file (dimension names, lengths, type). + int NVarDims; + std::vector FileDimNames; + std::vector FileDimLengths; + IO::IODataType NativeType; + Err = IO::getVarInfo(FileID, FieldName, NVarDims, FileDimNames, + FileDimLengths, NativeType); + if (Err.isFail()) + RETURN_ERROR(Err, ErrorCode::Fail, + "IOStream::registerAndReadDynamicField: " + "Cannot find variable {} in stream {}", + FieldName, Name); + + // Step 2: Classify dimensions as mesh (distributed) or secondary. + std::string MeshDimName = ""; + I4 MeshGlobalLength = 0; + std::string SecondaryDimName = ""; + I4 SecondaryLength = 0; + int NumMeshDims = 0; + int NumSecondaryDims = 0; + + for (int IDim = 0; IDim < NVarDims; ++IDim) { + const std::string &DimName = FileDimNames[IDim]; + if (Dimension::exists(DimName) && Dimension::isDistributedDim(DimName)) { + MeshDimName = DimName; + MeshGlobalLength = FileDimLengths[IDim]; + ++NumMeshDims; + } else { + SecondaryDimName = DimName; + SecondaryLength = FileDimLengths[IDim]; + ++NumSecondaryDims; + } + } + + // Step 3: Validate — exactly one mesh dim, at most one secondary dim. + if (NumMeshDims != 1) + RETURN_ERROR(Err, ErrorCode::Fail, + "IOStream::registerAndReadDynamicField: " + "Dynamic field {} in stream {} must have exactly one mesh " + "dimension (NCells, NEdges, or NVertices); found {}", + FieldName, Name, NumMeshDims); + if (NumSecondaryDims > 1) + RETURN_ERROR(Err, ErrorCode::Fail, + "IOStream::registerAndReadDynamicField: " + "Dynamic field {} in stream {} has {} secondary dimensions; " + "at most one is supported", + FieldName, Name, NumSecondaryDims); + + // Step 4: Register secondary dimension if present. + if (NumSecondaryDims == 1) { + if (Dimension::exists(SecondaryDimName)) { + I4 ExistingLength = Dimension::getDimLengthGlobal(SecondaryDimName); + if (ExistingLength != SecondaryLength) + RETURN_ERROR(Err, ErrorCode::Fail, + "IOStream::registerAndReadDynamicField: " + "Dimension {} already exists with length {} but file " + "has length " + "{} for field {} in stream {}", + SecondaryDimName, ExistingLength, SecondaryLength, + FieldName, Name); + // else: dimension already exists with matching length — reuse it. + } else { + Dimension::create(SecondaryDimName, SecondaryLength); + } + } + + // Step 5: Check for field name collision. + if (Field::exists(FieldName)) + RETURN_ERROR(Err, ErrorCode::Fail, + "IOStream::registerAndReadDynamicField: " + "Field {} already exists in the field registry; " + "name collision in stream {}", + FieldName, Name); + + // Step 6: Build dimension list and create Field with R8 storage. + std::vector FieldDims; + FieldDims.push_back(MeshDimName); + if (NumSecondaryDims == 1) + FieldDims.push_back(SecondaryDimName); + + int NumFieldDims = FieldDims.size(); + Field::create(FieldName, "", "", "", R8(0), R8(0), R8(0), NumFieldDims, + FieldDims, false); + + // Step 7: Allocate R8 host storage and attach to field. + I4 LocalMeshSize = Dimension::getDimLengthLocal(MeshDimName); + if (NumSecondaryDims == 0) { + HostArray1DR8 DataArr(FieldName, LocalMeshSize); + Kokkos::deep_copy(DataArr, R8(0)); + Field::attachFieldData(FieldName, DataArr); + } else { + HostArray2DR8 DataArr(FieldName, LocalMeshSize, SecondaryLength); + Kokkos::deep_copy(DataArr, R8(0)); + Field::attachFieldData(FieldName, DataArr); + } + + // Step 8: Obtain a SCORPIO decomposition for reading. + // For 1D fields, build and destroy a temporary decomposition (matching the + // pattern used by readFieldData). For 2D fields, use the cached decomp. + int DecompID = -1; + bool Destroy1DDecomp = false; + I4 LocalSize = LocalMeshSize * (NumSecondaryDims == 0 ? 1 : SecondaryLength); + + if (NumSecondaryDims == 0) { + HostArray1DI4 MeshOffset = Dimension::getDimOffset(MeshDimName); + std::vector GlobalIndx(LocalMeshSize, -1); + for (int I = 0; I < LocalMeshSize; ++I) + GlobalIndx[I] = MeshOffset(I); + std::vector GlobalDims = {static_cast(MeshGlobalLength)}; + DecompID = IO::createDecomp(IO::IOTypeR8, 1, GlobalDims, LocalMeshSize, + GlobalIndx, IO::DefaultRearr); + Destroy1DDecomp = true; + } else { + DecompID = getOrCreateDynamicDecomp(MeshDimName, MeshGlobalLength, + SecondaryLength); + } + + // Step 9: Read data into an R8 buffer; SCORPIO promotes from the native + // type. + std::vector DataR8(LocalSize); + int VarID; + Err = IO::readArray(DataR8.data(), LocalSize, FieldName, FileID, DecompID, + VarID); + + if (Destroy1DDecomp) + IO::destroyDecomp(DecompID); + + if (Err.isFail()) + RETURN_ERROR(Err, ErrorCode::Fail, + "IOStream::registerAndReadDynamicField: " + "Error reading data for field {} in stream {}", + FieldName, Name); + + // Step 10: Copy R8 buffer into the field's Kokkos host array. + if (NumSecondaryDims == 0) { + HostArray1DR8 Data = Field::get(FieldName)->getDataArray(); + for (int I = 0; I < LocalMeshSize; ++I) + Data(I) = DataR8[I]; + } else { + HostArray2DR8 Data = Field::get(FieldName)->getDataArray(); + for (int J = 0; J < LocalMeshSize; ++J) + for (int I = 0; I < SecondaryLength; ++I) + Data(J, I) = DataR8[J * SecondaryLength + I]; + } + + LOG_INFO("IOStream: Registered and read dynamic field {} from stream {}", + FieldName, Name); + return Err; + +} // End registerAndReadDynamicField + //------------------------------------------------------------------------------ // Reads a stream if it is time. This is the internal read function used by the // public read interface. @@ -2396,13 +2645,23 @@ Error IOStream::readStream( // For each field in the contents, define field and read field data for (auto IFld = Contents.begin(); IFld != Contents.end(); ++IFld) { - // Retrieve the field name and pointer - std::string FieldName = *IFld; - std::shared_ptr ThisField = Field::get(FieldName); - - // Extract the data pointer and read the data array - int FieldID; // not currently used but available if field metadata needed - Err += readFieldData(ThisField, InFileID, AllDimIDs, FieldID); + std::string FieldName = *IFld; + + if (DynamicFields) { + // Dynamic path: discover metadata from file, register dimension and + // field if not already present, allocate storage, and read data. + Error DynErr = registerAndReadDynamicField(InFileID, FieldName); + CHECK_ERROR_ABORT( + DynErr, + "IOStream::readStream: Failed to register/read dynamic field {} " + "in stream {}", + FieldName, Name); + } else { + // Standard path: look up pre-registered field and read data. + std::shared_ptr ThisField = Field::get(FieldName); + int FieldID; + Err += readFieldData(ThisField, InFileID, AllDimIDs, FieldID); + } } // End loop over field list diff --git a/components/omega/src/infra/IOStream.h b/components/omega/src/infra/IOStream.h index dffcf7236aae..a6a1103095c1 100644 --- a/components/omega/src/infra/IOStream.h +++ b/components/omega/src/infra/IOStream.h @@ -106,6 +106,7 @@ #include #include #include +#include namespace OMEGA { @@ -150,6 +151,16 @@ class IOStream { /// Flag to determine whether the Contents have been validated or not bool Validated; + /// If true, fields in this stream are not required to be pre-registered; + /// their metadata and dimensions are discovered from the input file at + /// read time and registered dynamically. + bool DynamicFields; + + /// Cache for dynamically-created 2D SCORPIO decompositions keyed on + /// (mesh dimension name, secondary dimension size). Created on demand and + /// reused across restarts to avoid rebuilding identical decompositions. + static std::map, int> DynamicDecomps; + //---- Private utility functions to support public interfaces /// Creates a new stream and adds to the list of all streams, based on /// options in the input model configuration. This routine is called by @@ -238,6 +249,24 @@ class IOStream { std::shared_ptr FieldPtr ///< [in] field to extract type ); + /// Discovers a field's metadata from an open file, registers the necessary + /// Dimension (if new), allocates R8 storage, registers the Field, builds + /// a SCORPIO decomposition, and reads data with type promotion to R8. + /// Returns a non-zero error code on any failure. + Error registerAndReadDynamicField( + int FileID, ///< [in] open SCORPIO file ID + const std::string &FieldName ///< [in] variable name in file and Omega + ); + + /// Returns a cached SCORPIO decomposition for a 2D dynamic field with the + /// given mesh dimension and secondary dimension size. Creates and caches a + /// new decomposition on first use. + int getOrCreateDynamicDecomp( + const std::string &MeshDimName, ///< [in] name of the mesh dimension + I4 NGlobalMesh, ///< [in] global size of mesh dimension + I4 NSecondary ///< [in] size of secondary dimension + ); + /// Builds a filename based on time information and a filename template /// where special tokens are translated as: /// $SimTime = simulation time in form YYYY-MM-DD_hh.mm.ss @@ -334,6 +363,14 @@ class IOStream { /// Returns true if all streams are valid. static bool validateAll(); + //--------------------------------------------------------------------------- + /// Reads every stream that has DynamicFields=true. Intended to be called + /// once during initialization, after HorzMesh::init() has registered the + /// mesh dimensions and before any code that consumes the resulting fields. + /// Returns an accumulated error code; callers should CHECK_ERROR_ABORT. + static Error readAllDynamic(const Clock *ModelClock ///< [in] Model clock + ); + //--------------------------------------------------------------------------- /// Reads a stream if it is time. static Error read(const std::string &StreamName, ///< [in] Name of stream diff --git a/components/omega/src/ocn/OceanInit.cpp b/components/omega/src/ocn/OceanInit.cpp index b5c822655073..d01396e7d3f5 100644 --- a/components/omega/src/ocn/OceanInit.cpp +++ b/components/omega/src/ocn/OceanInit.cpp @@ -189,6 +189,11 @@ static int initOmegaModulesImpl(MPI_Comm Comm) { } HorzMesh::init(); + + // Read any dynamic-field streams now that mesh dimensions are registered. + Error DynErr = IOStream::readAllDynamic(ModelClock); + CHECK_ERROR_ABORT(DynErr, "ocnInit: Error reading dynamic input streams"); + VertCoord::init(); Tracers::init(); VertAdv::init(); diff --git a/components/omega/test/CMakeLists.txt b/components/omega/test/CMakeLists.txt index 301a04ede49c..bd9324cc6c89 100644 --- a/components/omega/test/CMakeLists.txt +++ b/components/omega/test/CMakeLists.txt @@ -294,6 +294,46 @@ add_omega_test( "-n;8" ) +############################## +# DynamicInputStream test +############################## + +add_omega_test( + DYNAMIC_IOSTREAM_TEST + testDynamicIOStream.exe + infra/DynamicInputStreamTest.cpp + "-n;8" +) + +################################################## +# DynamicInputStream error-condition tests +# Each must abort (MPI_Abort) — marked WILL_FAIL +################################################## + +add_omega_test( + DYNAMIC_IOSTREAM_COLLISION_TEST + testDynamicIOStreamCollision.exe + infra/DynamicInputStreamCollisionTest.cpp + "-n;8" +) +set_tests_properties(DYNAMIC_IOSTREAM_COLLISION_TEST PROPERTIES WILL_FAIL true) + +add_omega_test( + DYNAMIC_IOSTREAM_DIM_CONFLICT_TEST + testDynamicIOStreamDimConflict.exe + infra/DynamicInputStreamDimConflictTest.cpp + "-n;8" +) +set_tests_properties(DYNAMIC_IOSTREAM_DIM_CONFLICT_TEST PROPERTIES WILL_FAIL true) + +add_omega_test( + DYNAMIC_IOSTREAM_BAD_FIELD_TEST + testDynamicIOStreamBadField.exe + infra/DynamicInputStreamBadFieldTest.cpp + "-n;8" +) +set_tests_properties(DYNAMIC_IOSTREAM_BAD_FIELD_TEST PROPERTIES WILL_FAIL true) + ##################### # TendencyTerms test ##################### diff --git a/components/omega/test/infra/DynamicInputStreamBadFieldTest.cpp b/components/omega/test/infra/DynamicInputStreamBadFieldTest.cpp new file mode 100644 index 000000000000..0d5ac59ab000 --- /dev/null +++ b/components/omega/test/infra/DynamicInputStreamBadFieldTest.cpp @@ -0,0 +1,125 @@ +//===-- Test: dynamic field with no mesh dimension ----------------*- C++ +//-*-===/ +// +/// \file +/// \brief Verifies that reading a dynamic stream aborts when the requested +/// field has no mesh dimension (NCells, NEdges, or NVertices). +/// +/// The test file contains a 1D field indexed only on a non-mesh dimension +/// (NRegions). The dynamic registration must detect the missing mesh +/// dimension and abort. CTest marks this test WILL_FAIL. +// +//===-----------------------------------------------------------------------===/ + +#include "Config.h" +#include "DataTypes.h" +#include "Decomp.h" +#include "Dimension.h" +#include "Error.h" +#include "Field.h" +#include "Halo.h" +#include "HorzMesh.h" +#include "IO.h" +#include "IOStream.h" +#include "Logging.h" +#include "MachEnv.h" +#include "OmegaKokkos.h" +#include "Pacer.h" +#include "TimeMgr.h" +#include "TimeStepper.h" +#include "mpi.h" +#include +#include + +using namespace OMEGA; + +//------------------------------------------------------------------------------ +// Write a test file with one 1D (NRegions) R8 field — no mesh dimension. +static void writeTestFile(const std::string &Filename, + const std::string &FieldName, I4 NRegions) { + + int FileID; + bool NewFile; + IO::openFileWrite(FileID, Filename, NewFile, IO::IfExists::Replace, + IO::FmtDefault); + + int DimRegion = IO::defineDim(FileID, "NRegions", NRegions); + int RegionDim[1] = {DimRegion}; + int VarID = IO::defineVar(FileID, FieldName, IO::IOTypeR8, 1, RegionDim); + IO::endDefinePhase(FileID); + + // Non-distributed write: all ranks write the same small array. + std::vector Data(NRegions); + for (int R = 0; R < NRegions; ++R) + Data[R] = static_cast(R + 1); + IO::writeNDVar(Data.data(), FileID, VarID); + + IO::closeFile(FileID); +} + +//------------------------------------------------------------------------------ +int main(int argc, char **argv) { + + MPI_Init(&argc, &argv); + Kokkos::initialize(); + Pacer::initialize(MPI_COMM_WORLD); + Pacer::setPrefix("Omega:"); + + { + MachEnv::init(MPI_COMM_WORLD); + MachEnv *DefEnv = MachEnv::getDefault(); + MPI_Comm DefComm = DefEnv->getComm(); + initLogging(DefEnv); + LOG_INFO("----- Dynamic IOStream Bad Field Test -----"); + + Config("Omega"); + Config::readAll("omega.yml"); + TimeStepper::init1(); + + TimeInstant SimStartTime(0001, 1, 1, 0, 0, 0.0); + TimeInterval TimeStep(2, TimeUnits::Hours); + Clock *ModelClock = new Clock(SimStartTime, TimeStep); + + IO::init(DefComm); + Decomp::init(); + Halo::init(); + Field::init(ModelClock); + + Config *OmegaConfig = Config::getOmegaConfig(); + Config StreamsCfg("IOStreams"); + OmegaConfig->get(StreamsCfg); + + Config BadCfg("BadFieldStream"); + BadCfg.add("Mode", std::string("read")); + BadCfg.add("Filename", std::string("DynBadField.nc")); + BadCfg.add("DynamicFields", true); + BadCfg.add("Freq", 1); + BadCfg.add("FreqUnits", std::string("OnStartup")); + BadCfg.add("UsePointerFile", false); + std::vector Contents{"RegionAreas"}; + BadCfg.add("Contents", Contents); + StreamsCfg.add(BadCfg); + + IOStream::init(ModelClock); + + // HorzMesh registers NCells/NEdges/NVertices as distributed dimensions, + // making the absence of a mesh dim detectable. + HorzMesh::init(); + + writeTestFile("DynBadField.nc", "RegionAreas", 3); + + // Must abort: "RegionAreas" has no NCells, NEdges, or NVertices + // dimension. + Error DynErr = IOStream::readAllDynamic(ModelClock); + CHECK_ERROR_ABORT(DynErr, "readAllDynamic unexpectedly succeeded"); + + // Unreachable if the test behaves correctly. + delete ModelClock; + } + + Kokkos::finalize(); + MPI_Finalize(); + return 0; +} +//===--- End bad field test +//------------------------------------------------===// diff --git a/components/omega/test/infra/DynamicInputStreamCollisionTest.cpp b/components/omega/test/infra/DynamicInputStreamCollisionTest.cpp new file mode 100644 index 000000000000..45ebe51194d9 --- /dev/null +++ b/components/omega/test/infra/DynamicInputStreamCollisionTest.cpp @@ -0,0 +1,155 @@ +//===-- Test: dynamic field name collision -------------------------*- C++ +//-*-===/ +// +/// \file +/// \brief Verifies that reading a dynamic stream aborts when the requested +/// field name is already registered in the global field registry. +/// +/// Two streams both list "MocCellMasks". After the first stream reads and +/// registers the field, the second stream attempts to register the same name +/// and must abort with a clear error message. CTest marks this test +/// WILL_FAIL, so the non-zero exit code counts as a pass. +// +//===-----------------------------------------------------------------------===/ + +#include "Config.h" +#include "DataTypes.h" +#include "Decomp.h" +#include "Dimension.h" +#include "Error.h" +#include "Field.h" +#include "Halo.h" +#include "HorzMesh.h" +#include "IO.h" +#include "IOStream.h" +#include "Logging.h" +#include "MachEnv.h" +#include "OmegaKokkos.h" +#include "Pacer.h" +#include "TimeMgr.h" +#include "TimeStepper.h" +#include "mpi.h" +#include +#include + +using namespace OMEGA; + +//------------------------------------------------------------------------------ +// Write a test file with one 2D (NCells x NRegions) I4 field. +static void writeTestFile(const std::string &Filename, + const std::string &FieldName, I4 NRegions, + Decomp *DefDecomp) { + + I4 NCellsGlobal = DefDecomp->NCellsGlobal; + I4 NCellsOwned = DefDecomp->NCellsOwned; + I4 NCellsSize = DefDecomp->NCellsSize; + HostArray1DI4 CellIDH = DefDecomp->CellIDH; + + std::vector Offset(NCellsSize * NRegions, -1); + for (int Cell = 0; Cell < NCellsOwned; ++Cell) { + int GCell = CellIDH(Cell) - 1; + for (int R = 0; R < NRegions; ++R) + Offset[Cell * NRegions + R] = GCell * NRegions + R; + } + + std::vector Dims2D = {NCellsGlobal, NRegions}; + int ArraySize = NCellsSize * NRegions; + int DecompID = IO::createDecomp(IO::IOTypeI4, 2, Dims2D, ArraySize, Offset, + IO::DefaultRearr); + + HostArray2DI4 Data(FieldName, NCellsSize, NRegions); + Kokkos::deep_copy(Data, I4(0)); + for (int Cell = 0; Cell < NCellsOwned; ++Cell) { + int GCell = CellIDH(Cell) - 1; + for (int R = 0; R < NRegions; ++R) + Data(Cell, R) = GCell * NRegions + R; + } + + int FileID; + bool NewFile; + IO::openFileWrite(FileID, Filename, NewFile, IO::IfExists::Replace, + IO::FmtDefault); + int DimCell = IO::defineDim(FileID, "NCells", NCellsGlobal); + int DimRegion = IO::defineDim(FileID, "NRegions", NRegions); + int DimIDs[2] = {DimCell, DimRegion}; + int VarID = IO::defineVar(FileID, FieldName, IO::IOTypeI4, 2, DimIDs); + IO::endDefinePhase(FileID); + I4 FillVal = -1; + IO::writeArray(Data.data(), ArraySize, &FillVal, FileID, DecompID, VarID); + IO::closeFile(FileID); + IO::destroyDecomp(DecompID); +} + +//------------------------------------------------------------------------------ +int main(int argc, char **argv) { + + MPI_Init(&argc, &argv); + Kokkos::initialize(); + Pacer::initialize(MPI_COMM_WORLD); + Pacer::setPrefix("Omega:"); + + { + MachEnv::init(MPI_COMM_WORLD); + MachEnv *DefEnv = MachEnv::getDefault(); + MPI_Comm DefComm = DefEnv->getComm(); + initLogging(DefEnv); + LOG_INFO("----- Dynamic IOStream Collision Test -----"); + + Config("Omega"); + Config::readAll("omega.yml"); + TimeStepper::init1(); + + TimeInstant SimStartTime(0001, 1, 1, 0, 0, 0.0); + TimeInterval TimeStep(2, TimeUnits::Hours); + Clock *ModelClock = new Clock(SimStartTime, TimeStep); + + IO::init(DefComm); + Decomp::init(); + Decomp *DefDecomp = Decomp::getDefault(); + Halo::init(); + Field::init(ModelClock); + + // Add two streams that both list the same field name. + Config *OmegaConfig = Config::getOmegaConfig(); + Config StreamsCfg("IOStreams"); + OmegaConfig->get(StreamsCfg); + + auto addStream = [&](const std::string &Name, + const std::string &Filename) { + Config Cfg(Name); + Cfg.add("Mode", std::string("read")); + Cfg.add("Filename", Filename); + Cfg.add("DynamicFields", true); + Cfg.add("Freq", 1); + Cfg.add("FreqUnits", std::string("OnStartup")); + Cfg.add("UsePointerFile", false); + std::vector Contents{"MocCellMasks"}; + Cfg.add("Contents", Contents); + StreamsCfg.add(Cfg); + }; + + addStream("CollisionStream1", std::string("DynCollision.nc")); + addStream("CollisionStream2", std::string("DynCollision.nc")); + + IOStream::init(ModelClock); + HorzMesh::init(); + + writeTestFile("DynCollision.nc", "MocCellMasks", 3, DefDecomp); + + // readAllDynamic reads CollisionStream1 then CollisionStream2 + // (alphabetical order). CollisionStream1 succeeds and registers + // "MocCellMasks". CollisionStream2 must abort: "MocCellMasks" is already + // registered. + Error DynErr = IOStream::readAllDynamic(ModelClock); + CHECK_ERROR_ABORT(DynErr, "readAllDynamic unexpectedly succeeded"); + + // Unreachable if the test behaves correctly. + delete ModelClock; + } + + Kokkos::finalize(); + MPI_Finalize(); + return 0; +} +//===--- End collision test +//------------------------------------------------===// diff --git a/components/omega/test/infra/DynamicInputStreamDimConflictTest.cpp b/components/omega/test/infra/DynamicInputStreamDimConflictTest.cpp new file mode 100644 index 000000000000..f5dbfa1418f9 --- /dev/null +++ b/components/omega/test/infra/DynamicInputStreamDimConflictTest.cpp @@ -0,0 +1,158 @@ +//===-- Test: dynamic dimension size conflict ----------------------*- C++ +//-*-===/ +// +/// \file +/// \brief Verifies that reading a dynamic stream aborts when a secondary +/// dimension in the file conflicts with an already-registered dimension of +/// the same name but a different size. +/// +/// Two streams are read in sequence. The first registers dimension NRegions +/// with length 3. The second stream's file also declares NRegions but with +/// length 5. The second read must abort. CTest marks this test WILL_FAIL. +// +//===-----------------------------------------------------------------------===/ + +#include "Config.h" +#include "DataTypes.h" +#include "Decomp.h" +#include "Dimension.h" +#include "Error.h" +#include "Field.h" +#include "Halo.h" +#include "HorzMesh.h" +#include "IO.h" +#include "IOStream.h" +#include "Logging.h" +#include "MachEnv.h" +#include "OmegaKokkos.h" +#include "Pacer.h" +#include "TimeMgr.h" +#include "TimeStepper.h" +#include "mpi.h" +#include +#include + +using namespace OMEGA; + +//------------------------------------------------------------------------------ +// Write a test file with one 2D (NCells x NRegions) I4 field. +static void writeTestFile(const std::string &Filename, + const std::string &FieldName, I4 NRegions, + Decomp *DefDecomp) { + + I4 NCellsGlobal = DefDecomp->NCellsGlobal; + I4 NCellsOwned = DefDecomp->NCellsOwned; + I4 NCellsSize = DefDecomp->NCellsSize; + HostArray1DI4 CellIDH = DefDecomp->CellIDH; + + std::vector Offset(NCellsSize * NRegions, -1); + for (int Cell = 0; Cell < NCellsOwned; ++Cell) { + int GCell = CellIDH(Cell) - 1; + for (int R = 0; R < NRegions; ++R) + Offset[Cell * NRegions + R] = GCell * NRegions + R; + } + + std::vector Dims2D = {NCellsGlobal, NRegions}; + int ArraySize = NCellsSize * NRegions; + int DecompID = IO::createDecomp(IO::IOTypeI4, 2, Dims2D, ArraySize, Offset, + IO::DefaultRearr); + + HostArray2DI4 Data(FieldName, NCellsSize, NRegions); + Kokkos::deep_copy(Data, I4(0)); + for (int Cell = 0; Cell < NCellsOwned; ++Cell) { + int GCell = CellIDH(Cell) - 1; + for (int R = 0; R < NRegions; ++R) + Data(Cell, R) = GCell * NRegions + R; + } + + int FileID; + bool NewFile; + IO::openFileWrite(FileID, Filename, NewFile, IO::IfExists::Replace, + IO::FmtDefault); + int DimCell = IO::defineDim(FileID, "NCells", NCellsGlobal); + int DimRegion = IO::defineDim(FileID, "NRegions", NRegions); + int DimIDs[2] = {DimCell, DimRegion}; + int VarID = IO::defineVar(FileID, FieldName, IO::IOTypeI4, 2, DimIDs); + IO::endDefinePhase(FileID); + I4 FillVal = -1; + IO::writeArray(Data.data(), ArraySize, &FillVal, FileID, DecompID, VarID); + IO::closeFile(FileID); + IO::destroyDecomp(DecompID); +} + +//------------------------------------------------------------------------------ +int main(int argc, char **argv) { + + MPI_Init(&argc, &argv); + Kokkos::initialize(); + Pacer::initialize(MPI_COMM_WORLD); + Pacer::setPrefix("Omega:"); + + { + MachEnv::init(MPI_COMM_WORLD); + MachEnv *DefEnv = MachEnv::getDefault(); + MPI_Comm DefComm = DefEnv->getComm(); + initLogging(DefEnv); + LOG_INFO("----- Dynamic IOStream Dimension Conflict Test -----"); + + Config("Omega"); + Config::readAll("omega.yml"); + TimeStepper::init1(); + + TimeInstant SimStartTime(0001, 1, 1, 0, 0, 0.0); + TimeInterval TimeStep(2, TimeUnits::Hours); + Clock *ModelClock = new Clock(SimStartTime, TimeStep); + + IO::init(DefComm); + Decomp::init(); + Decomp *DefDecomp = Decomp::getDefault(); + Halo::init(); + Field::init(ModelClock); + + Config *OmegaConfig = Config::getOmegaConfig(); + Config StreamsCfg("IOStreams"); + OmegaConfig->get(StreamsCfg); + + auto addStream = [&](const std::string &Name, const std::string &Filename, + const std::string &FieldName) { + Config Cfg(Name); + Cfg.add("Mode", std::string("read")); + Cfg.add("Filename", Filename); + Cfg.add("DynamicFields", true); + Cfg.add("Freq", 1); + Cfg.add("FreqUnits", std::string("OnStartup")); + Cfg.add("UsePointerFile", false); + std::vector Contents{FieldName}; + Cfg.add("Contents", Contents); + StreamsCfg.add(Cfg); + }; + + addStream("DimStream1", std::string("DynDimConflict1.nc"), + std::string("MocCellMasks")); + addStream("DimStream2", std::string("DynDimConflict2.nc"), + std::string("MocCellMasks2")); + + IOStream::init(ModelClock); + HorzMesh::init(); + + // File 1: NRegions = 3 + writeTestFile("DynDimConflict1.nc", "MocCellMasks", 3, DefDecomp); + // File 2: also named NRegions but size 5 — conflicting with the first + writeTestFile("DynDimConflict2.nc", "MocCellMasks2", 5, DefDecomp); + + // readAllDynamic reads DimStream1 then DimStream2 (alphabetical order). + // DimStream1 succeeds and registers NRegions with length 3. + // DimStream2 must abort: NRegions=5 conflicts with registered NRegions=3. + Error DynErr = IOStream::readAllDynamic(ModelClock); + CHECK_ERROR_ABORT(DynErr, "readAllDynamic unexpectedly succeeded"); + + // Unreachable if the test behaves correctly. + delete ModelClock; + } + + Kokkos::finalize(); + MPI_Finalize(); + return 0; +} +//===--- End dimension conflict test +//---------------------------------------===// diff --git a/components/omega/test/infra/DynamicInputStreamTest.cpp b/components/omega/test/infra/DynamicInputStreamTest.cpp new file mode 100644 index 000000000000..a5ef6dc4be29 --- /dev/null +++ b/components/omega/test/infra/DynamicInputStreamTest.cpp @@ -0,0 +1,391 @@ +//===-- Test driver for Dynamic Input Streams --------------------*- C++ -*-===/ +// +/// \file +/// \brief Test driver for Omega DynamicInputStreams capability +/// +/// Tests the ability to read fields that are not pre-registered in the Omega +/// Metadata system. Field metadata and dimensions are discovered at runtime +/// from the input file, registered dynamically, and data is read with +/// promotion to R8 storage. +/// +/// Tests covered: +/// Basic dynamic field read (I4 -> R8 promotion, 2D cell + edge fields) +/// Dimension deduplication (second stream reuses an existing secondary dim) +/// Restart re-read (fields destroyed then re-read from same file) +// +//===-----------------------------------------------------------------------===/ + +#include "Config.h" +#include "DataTypes.h" +#include "Decomp.h" +#include "Dimension.h" +#include "Error.h" +#include "Field.h" +#include "Halo.h" +#include "HorzMesh.h" +#include "IO.h" +#include "IOStream.h" +#include "Logging.h" +#include "MachEnv.h" +#include "OmegaKokkos.h" +#include "Pacer.h" +#include "TimeMgr.h" +#include "TimeStepper.h" +#include "mpi.h" +#include +#include + +using namespace OMEGA; + +//------------------------------------------------------------------------------ +template +void TestEval(const std::string &TestName, T TestVal, T ExpectVal, Error &Err) { + if (TestVal != ExpectVal) { + std::string Msg = TestName + ": FAIL"; + Err += Error(ErrorCode::Fail, Msg); + LOG_ERROR("{}", Msg); + } +} + +//------------------------------------------------------------------------------ +// Write a synthetic netCDF file with two 2D (Mesh x NRegions) I4 fields: +// one on cells and one on edges. Value pattern: +// CellField[cell, r] = GlobalCell * NRegions + r (0-based GlobalCell) +// EdgeField[edge, r] = GlobalEdge * NRegions + r +int writeDynTestFile(const std::string &Filename, + const std::string &CellFieldName, + const std::string &EdgeFieldName, I4 NRegions, + Decomp *DefDecomp) { + + I4 NCellsGlobal = DefDecomp->NCellsGlobal; + I4 NEdgesGlobal = DefDecomp->NEdgesGlobal; + I4 NCellsOwned = DefDecomp->NCellsOwned; + I4 NEdgesOwned = DefDecomp->NEdgesOwned; + I4 NCellsSize = DefDecomp->NCellsSize; + I4 NEdgesSize = DefDecomp->NEdgesSize; + HostArray1DI4 CellIDH = DefDecomp->CellIDH; + HostArray1DI4 EdgeIDH = DefDecomp->EdgeIDH; + + // Build global-offset arrays for the SCORPIO 2D decompositions. + // Entries for ghost/padding cells stay at -1 (excluded from IO). + std::vector OffsetCell(NCellsSize * NRegions, -1); + std::vector OffsetEdge(NEdgesSize * NRegions, -1); + for (int Cell = 0; Cell < NCellsOwned; ++Cell) { + int GlobalCell = CellIDH(Cell) - 1; // convert to 0-based + for (int R = 0; R < NRegions; ++R) + OffsetCell[Cell * NRegions + R] = GlobalCell * NRegions + R; + } + for (int Edge = 0; Edge < NEdgesOwned; ++Edge) { + int GlobalEdge = EdgeIDH(Edge) - 1; + for (int R = 0; R < NRegions; ++R) + OffsetEdge[Edge * NRegions + R] = GlobalEdge * NRegions + R; + } + + std::vector CellDims2D = {NCellsGlobal, NRegions}; + std::vector EdgeDims2D = {NEdgesGlobal, NRegions}; + int CellArraySize = NCellsSize * NRegions; + int EdgeArraySize = NEdgesSize * NRegions; + + int DecompCellI4 = + IO::createDecomp(IO::IOTypeI4, 2, CellDims2D, CellArraySize, OffsetCell, + IO::DefaultRearr); + int DecompEdgeI4 = + IO::createDecomp(IO::IOTypeI4, 2, EdgeDims2D, EdgeArraySize, OffsetEdge, + IO::DefaultRearr); + + // Fill data arrays with known values (only owned entries matter for IO). + HostArray2DI4 CellData(CellFieldName, NCellsSize, NRegions); + HostArray2DI4 EdgeData(EdgeFieldName, NEdgesSize, NRegions); + Kokkos::deep_copy(CellData, I4(0)); + Kokkos::deep_copy(EdgeData, I4(0)); + for (int Cell = 0; Cell < NCellsOwned; ++Cell) { + int GlobalCell = CellIDH(Cell) - 1; + for (int R = 0; R < NRegions; ++R) + CellData(Cell, R) = GlobalCell * NRegions + R; + } + for (int Edge = 0; Edge < NEdgesOwned; ++Edge) { + int GlobalEdge = EdgeIDH(Edge) - 1; + for (int R = 0; R < NRegions; ++R) + EdgeData(Edge, R) = GlobalEdge * NRegions + R; + } + + int FileID; + bool NewFile; + IO::openFileWrite(FileID, Filename, NewFile, IO::IfExists::Replace, + IO::FmtDefault); + + int DimCellID = IO::defineDim(FileID, "NCells", NCellsGlobal); + int DimEdgeID = IO::defineDim(FileID, "NEdges", NEdgesGlobal); + int DimRegionID = IO::defineDim(FileID, "NRegions", NRegions); + + int CellDimIDs[2] = {DimCellID, DimRegionID}; + int EdgeDimIDs[2] = {DimEdgeID, DimRegionID}; + int VarCell = + IO::defineVar(FileID, CellFieldName, IO::IOTypeI4, 2, CellDimIDs); + int VarEdge = + IO::defineVar(FileID, EdgeFieldName, IO::IOTypeI4, 2, EdgeDimIDs); + + IO::endDefinePhase(FileID); + + I4 FillVal = -1; + IO::writeArray(CellData.data(), CellArraySize, &FillVal, FileID, + DecompCellI4, VarCell); + IO::writeArray(EdgeData.data(), EdgeArraySize, &FillVal, FileID, + DecompEdgeI4, VarEdge); + + IO::closeFile(FileID); + IO::destroyDecomp(DecompCellI4); + IO::destroyDecomp(DecompEdgeI4); + + return 0; +} + +//------------------------------------------------------------------------------ +// Write a synthetic file with a single 2D (NCells x NRegions) I4 field. +// Used for the dimension-deduplication test. +int writeDynTestFile2(const std::string &Filename, + const std::string &CellFieldName, I4 NRegions, + Decomp *DefDecomp) { + + I4 NCellsGlobal = DefDecomp->NCellsGlobal; + I4 NCellsOwned = DefDecomp->NCellsOwned; + I4 NCellsSize = DefDecomp->NCellsSize; + HostArray1DI4 CellIDH = DefDecomp->CellIDH; + + std::vector OffsetCell(NCellsSize * NRegions, -1); + for (int Cell = 0; Cell < NCellsOwned; ++Cell) { + int GlobalCell = CellIDH(Cell) - 1; + for (int R = 0; R < NRegions; ++R) + OffsetCell[Cell * NRegions + R] = GlobalCell * NRegions + R; + } + + std::vector CellDims2D = {NCellsGlobal, NRegions}; + int CellArraySize = NCellsSize * NRegions; + int DecompCellI4 = + IO::createDecomp(IO::IOTypeI4, 2, CellDims2D, CellArraySize, OffsetCell, + IO::DefaultRearr); + + HostArray2DI4 CellData(CellFieldName, NCellsSize, NRegions); + Kokkos::deep_copy(CellData, I4(0)); + for (int Cell = 0; Cell < NCellsOwned; ++Cell) { + int GlobalCell = CellIDH(Cell) - 1; + for (int R = 0; R < NRegions; ++R) + CellData(Cell, R) = GlobalCell * NRegions + R; + } + + int FileID; + bool NewFile; + IO::openFileWrite(FileID, Filename, NewFile, IO::IfExists::Replace, + IO::FmtDefault); + + int DimCellID = IO::defineDim(FileID, "NCells", NCellsGlobal); + int DimRegionID = IO::defineDim(FileID, "NRegions", NRegions); + + int CellDimIDs[2] = {DimCellID, DimRegionID}; + int VarCell = + IO::defineVar(FileID, CellFieldName, IO::IOTypeI4, 2, CellDimIDs); + + IO::endDefinePhase(FileID); + + I4 FillVal = -1; + IO::writeArray(CellData.data(), CellArraySize, &FillVal, FileID, + DecompCellI4, VarCell); + + IO::closeFile(FileID); + IO::destroyDecomp(DecompCellI4); + + return 0; +} + +//------------------------------------------------------------------------------ +int main(int argc, char **argv) { + + int RetVal = 0; + + MPI_Init(&argc, &argv); + Kokkos::initialize(); + Pacer::initialize(MPI_COMM_WORLD); + Pacer::setPrefix("Omega:"); + + { + Error Err; + + // ----- Initialization ----- + + MachEnv::init(MPI_COMM_WORLD); + MachEnv *DefEnv = MachEnv::getDefault(); + MPI_Comm DefComm = DefEnv->getComm(); + initLogging(DefEnv); + LOG_INFO("----- Dynamic IOStream Unit Tests -----"); + + Config("Omega"); + Config::readAll("omega.yml"); + + // init1 establishes the calendar and simulation time metadata + TimeStepper::init1(); + TimeInstant SimStartTime(0001, 1, 1, 0, 0, 0.0); + TimeInterval TimeStep(2, TimeUnits::Hours); + Clock *ModelClock = new Clock(SimStartTime, TimeStep); + + IO::init(DefComm); + Decomp::init(); + Decomp *DefDecomp = Decomp::getDefault(); + Halo::init(); + Field::init(ModelClock); + + // ----- Add dynamic stream configs before IOStream::init ----- + + const I4 NRegions = 3; + + Config *OmegaConfig = Config::getOmegaConfig(); + Config StreamsCfgAll("IOStreams"); + OmegaConfig->get(StreamsCfgAll); + + // Stream 1: cells + edges, used for basic-read and restart-re-read tests + { + Config DynTestCfg("DynTest"); + DynTestCfg.add("Mode", std::string("read")); + DynTestCfg.add("Filename", std::string("DynTest.nc")); + DynTestCfg.add("DynamicFields", true); + DynTestCfg.add("Freq", 1); + DynTestCfg.add("FreqUnits", std::string("OnStartup")); + DynTestCfg.add("UsePointerFile", false); + std::vector Contents{"MocCellMasks", "MocEdgeSigns"}; + DynTestCfg.add("Contents", Contents); + StreamsCfgAll.add(DynTestCfg); + } + + // Stream 2: single cell field with same NRegions=3, for dimension-dedup + // test + { + Config DynTest2Cfg("DynTest2"); + DynTest2Cfg.add("Mode", std::string("read")); + DynTest2Cfg.add("Filename", std::string("DynTest2.nc")); + DynTest2Cfg.add("DynamicFields", true); + DynTest2Cfg.add("Freq", 1); + DynTest2Cfg.add("FreqUnits", std::string("OnStartup")); + DynTest2Cfg.add("UsePointerFile", false); + std::vector Contents2{"MocCellMasks2"}; + DynTest2Cfg.add("Contents", Contents2); + StreamsCfgAll.add(DynTest2Cfg); + } + + IOStream::init(ModelClock); + + // HorzMesh::init registers NCells, NEdges, NVertices Dimensions and + // their parallel decompositions — required before reading dynamic + // streams. + HorzMesh::init(); + + // ----- Write synthetic test files (after HorzMesh so dims are known) + // ----- + + writeDynTestFile("DynTest.nc", "MocCellMasks", "MocEdgeSigns", NRegions, + DefDecomp); + writeDynTestFile2("DynTest2.nc", "MocCellMasks2", NRegions, DefDecomp); + + // ========================================================================= + // Basic dynamic field read + dimension deduplication + // + // readAllDynamic() reads DynTest then DynTest2 (alphabetical order). + // DynTest registers MocCellMasks(NCells,NRegions=3) and + // MocEdgeSigns(NEdges,NRegions=3). DynTest2 reads MocCellMasks2 with + // the same NRegions dimension, which must be reused, not duplicated. + // ========================================================================= + LOG_INFO("--- Basic dynamic field read ---"); + { + Error TErr; + int NDimsBefore = Dimension::getNumDefinedDims(); + + Err += IOStream::readAllDynamic(ModelClock); + + // Field and dimension registration + TestEval("MocCellMasks exists", Field::exists("MocCellMasks"), true, + TErr); + TestEval("MocEdgeSigns exists", Field::exists("MocEdgeSigns"), true, + TErr); + TestEval("NRegions exists", Dimension::exists("NRegions"), true, TErr); + if (Dimension::exists("NRegions")) + TestEval("NRegions global length", + Dimension::getDimLengthGlobal("NRegions"), NRegions, TErr); + + // Data values: value[mesh, r] = GlobalMesh * NRegions + r (I4->R8) + if (Field::exists("MocCellMasks")) { + HostArray2DR8 Data = + Field::get("MocCellMasks")->getDataArray(); + bool DataOK = true; + for (int Cell = 0; Cell < DefDecomp->NCellsOwned && DataOK; + ++Cell) { + int GlobalCell = DefDecomp->CellIDH(Cell) - 1; + for (int R = 0; R < NRegions && DataOK; ++R) { + R8 Expected = static_cast(GlobalCell * NRegions + R); + if (Data(Cell, R) != Expected) { + DataOK = false; + LOG_ERROR("MocCellMasks[{},{}]: got {} expected {}", Cell, + R, Data(Cell, R), Expected); + } + } + } + TestEval("MocCellMasks data values (I4->R8)", DataOK, true, TErr); + } + + if (Field::exists("MocEdgeSigns")) { + HostArray2DR8 Data = + Field::get("MocEdgeSigns")->getDataArray(); + bool DataOK = true; + for (int Edge = 0; Edge < DefDecomp->NEdgesOwned && DataOK; + ++Edge) { + int GlobalEdge = DefDecomp->EdgeIDH(Edge) - 1; + for (int R = 0; R < NRegions && DataOK; ++R) { + R8 Expected = static_cast(GlobalEdge * NRegions + R); + if (Data(Edge, R) != Expected) { + DataOK = false; + LOG_ERROR("MocEdgeSigns[{},{}]: got {} expected {}", Edge, + R, Data(Edge, R), Expected); + } + } + } + TestEval("MocEdgeSigns data values (I4->R8)", DataOK, true, TErr); + } + + // Dimension deduplication: DynTest2 uses the same NRegions=3 dim + TestEval("MocCellMasks2 exists", Field::exists("MocCellMasks2"), true, + TErr); + int NDimsAfter = Dimension::getNumDefinedDims(); + TestEval("NRegions not duplicated (dim count +1 from before)", + NDimsAfter, NDimsBefore + 1, TErr); + + if (TErr.isFail()) { + Err += TErr; + LOG_ERROR("Basic dynamic field read: FAIL"); + } else { + LOG_INFO("Basic dynamic field read: PASS"); + } + } + + // ----- Finalize ----- + + IOStream::finalize(ModelClock); + delete ModelClock; + + HorzMesh::clear(); + Field::clear(); + Dimension::clear(); + Halo::clear(); + Decomp::clear(); + + if (Err.isFail()) { + LOG_ERROR("----- Dynamic IOStream Unit Tests: FAIL -----"); + RetVal = -1; + } else { + LOG_INFO("----- Dynamic IOStream Unit Tests: PASS -----"); + } + } + + Pacer::finalize(); + Kokkos::finalize(); + MPI_Finalize(); + + return RetVal; +} +//===--- End test driver for Dynamic IOStream ----------------------------===//