Add climatology maps to Omega analysis - #730
Draft
xylar wants to merge 53 commits into
Draft
Conversation
xylar
added
ocean
Collaborator
Author
TestingPolaris
|
xylar
force-pushed
the
add-omega-analysis-climatology-maps
branch
from
August 30, 2026 12:42
614174e to
741870f
Compare
This was referenced Aug 30, 2026
xylar
force-pushed
the
add-omega-analysis-climatology-maps
branch
from
August 30, 2026 13:03
741870f to
f771421
Compare
Collaborator
Author
Status update, 2026-08-30The whole analysis chain now runs end to end and publishes a browsable gallery. Three things changed today that affect every PR in the series. The branches were relinearized. They were a tree with independent arms; they are now a line in dependency order, and each PR should be reviewed against the one below it:
GitHub cannot base a cross-fork PR on another fork branch, so all seven are based on Two things moved as part of this. The vertical reduction was item 11 and is now item 2, at the bottom: it is a dependency-light leaf module that everything vertically reduced needs, and landing it first means no branch above it ships a limitation that a later branch retracts — the climatology maps now support every elevation from the start, and heat content produces every elevation range rather than the whole column only. Publication moved ahead of the products for the same reason, since every step that makes products must leave a manifest fragment for the Every product is now described in a manifest, so the gallery is populated rather than empty. Products are grouped by kind — All seven branches were force-pushed after the relinearization, so local copies of any of them are stale. What it producesA gallery generated by the whole chain against the QU240 one-year mock-up, which is the quickest way to see what the suite does: Three steps publish nothing, and all three are waiting on Omega rather than on Polaris: |
Collaborator
Author
TestingOn
The counts grow down the chain, since each branch's diff contains the branches below it: 460 at the design branch through 715 at the top. End to endThe whole chain was run as the The resulting gallery is published here: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/xasaydavis/omega_analysis_qu240_mockup_20260830/ Two checks on the numbers rather than on the plumbing, both from the mock-up: each numeric elevation agrees with its nearest layer index to the offset between them --- temperature at -100 m is 0.17 K colder in the mean than Inheritance was exercised separately, since one simulated year cannot: a two-year simulation was faked with symlinks, the 1--2 range inherited year 1 from the completed 1--1 step bit-identically and computed only year 2, and re-running with a different specific heat capacity correctly rejected both the sibling seed and the step's own stale cache and recomputed all 24 months. |
Collaborator
Author
|
In the mockup example, the plot titles are too big an are getting cropped. 
|
The analysis suite reads a simulation's output over a range of years, so every step that touches simulation output needs the same expansion of a file-name template into a list of files, and the same clear report when some of them are absent. Put it in a leaf module with no dependence on Step, so that it can be unit tested directly and reused by each step. A template may carry $Y and $M, $Y alone, or no date at all. The last case is not a degenerate one: Omega writes a run's analysis output to a single undated file, so it is the shape global statistics arrive in. Files come back as (path, year, month) rather than as bare paths. The year and month are what the missing-file report needs in order to say "year 0002: months 02, 03" instead of listing paths, and they are what the heat content accumulator will key its cache on later. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The user's normal input is the path to the simulation's Omega configuration file, from which the mesh, the vertical coordinate and the output streams and their file-name templates all follow, so that none of it has to be restated by hand. This is the pattern MPAS-Analysis uses to locate MPAS-Ocean output by reading its streams files. Reuse PolarisYaml rather than adding a second YAML loader; reading it with streams_section='IOStreams' already splits the file the way this reader wants. Note that this is the first time Polaris reads an omega.yml that a previous run wrote rather than one it is about to write, so its failure modes are ours to report. Reconstruct the names of analysis-group output the way Omega's analysis manager builds them. A group can write both time means and snapshots under different names, and today's simulations write only snapshots, so neither spelling can be assumed: against the QU240 mock-up this reproduces global_stats_1DayInstants exactly. Report rather than raise. This is the one place Polaris depends on the shape of Omega's configuration rather than on its output, and Omega is free to change that shape, so an absent stream, a stream that names no file, and an analysis group that is turned off are told apart and handed back to the caller, which knows which config option to name instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add analysis.cfg and the SimulationFiles resolver that turns the simulation's Omega configuration into the mesh, the vertical coordinate, and the lists of monthly-mean, global-statistics and MOC files covering a range of years. The design had the file-name templates as config options that override what the Omega configuration says. They cannot be: Polaris config files use ExtendedInterpolation, so a bare $Y in a value raises "invalid interpolation syntax" and takes the whole config combine down with it, not just that option. Escaping as $$Y would work but is a trap, since a template pasted out of omega.yml then fails confusingly. The templates are gone instead of escaped. An Omega run always writes an omega.yml, so it is required rather than a default to fall back from, and it is the analysis' only description of where the output lives. That removes the override layer, its precedence rules, and the one place a $ could reach a config value. What stays as config options are plain paths with no templating in them: simulation_path, mesh_filename and vert_coord_filename, for output or a mesh that has moved since the run. MPAS-Ocean is therefore unsupported for now, and says so rather than failing obscurely. Reading it needs a translator from its namelists and streams into the form this reads, which is separate work. Against the QU240 mock-up, one config option now resolves the mesh, the vertical coordinate, twelve monthly-mean files and the global statistics, and reports that the run predates Omega's MOC diagnostic. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One step class per product, each resolving the simulation files it reads in setup(), symlinking them into its work directory, and reporting them at run time. No diagnostic is computed yet; each product fills in its own run() as it lands. Finding the inputs is not a placeholder for the work. It is what establishes that the Omega configuration reader, the template expansion and the range keying hold together against a real simulation, and it is the part every later product would otherwise each get wrong in its own way. A shared AnalysisStep carries it, along with the year range that every analysis step has. The MOC step is written for the simulations that exist: Omega's MOC diagnostic is new, so a run that predates it reports that and produces nothing, rather than failing the suite. The climatology declares the twelve processes ncclimo will start rather than sizing itself to the machine, per the bounded-launching rule; the rest declare one. Memory is deliberately left undeclared until it is measured, since a guess that is too large wastes a node and one that is too small fails late. Maps are chunked by field group, so that adding a field costs that field and not the others. Add ssh to the default fields, since the design lists an ssh group but left it out of the default list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add the four analysis tasks and register them with the ocean component. Every step lives at a subdirectory named for the range it covers, so a setup with a new range creates steps in directories that have never run and therefore run, while a setup with the same range lands on directories that are complete and recomputes nothing. That is the whole of the re-running requirement, and it needs no framework change. It also makes it structurally impossible to get a plot labelled with one range whose contents came from another, since two ranges never share a directory. A task's subdirectory is fixed when the component is constructed, but its steps are not: polaris setup merges the user's config and then calls configure() before adding configs to steps, precisely so that steps created there are handled. The tasks discard and rebuild their step lists from the config options, the way the cosine bell tasks already do for resolutions. The climatology and time-series ranges are separate config sections and so move independently: changing the climatology moves the maps and the MOC and leaves the two time series where they were. Construction reads nothing but the packaged defaults. CI runs `polaris list`, which builds every task in every component, so reading the simulation's Omega configuration has to wait for step setup. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Named to match the existing omega_pr and omega_nightly suites. It takes no cfg file of its own: the analysis runs no model, so the default wall time and scheduler target are right, and there is nothing a suite-level option would have to say. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The step tree is a pure function of the config options, so almost all of the re-running behavior can be checked without running anything: construct the tasks, set a range, call configure(), and look at where the steps landed. What these pin down is the part that is easy to break silently. That the two ranges are independent, that a changed range leaves no step behind at the old subdirectory, and that one climatology is shared by every field group are all properties a later refactor could lose without any test failing on the products themselves. Name the file test_analysis_tasks.py rather than test_tasks.py: pytest imports test modules by basename when their directories are not packages, and tests/ocean/realistic_global/test_tasks.py already claims that name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The User's Guide page describing the suite and its config options comes with the products, once there are products to describe. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ten steps each printing the configuration file and the simulation path buried the lines that actually differ between them. Say the simulation once per step, then indent the files that step took from it, so that the useful part -- which stream or analysis group each path came from -- is what stands out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Write the half of the documentation that is stable now and that no later product will own: how to point the suite at a simulation, how to run it, where results go, and what re-analyzing a different range does. Each product adds its own section as it lands, to a page that already exists, rather than a page having to be invented once several products are in. The per-product descriptions are deliberately absent. A table lists the four tasks as not yet implemented, which is one edit per product PR and is the cheapest coupling available; describing plots that do not exist would be worse than saying nothing. The full config-option reference is also left out on purpose, since analysis.cfg already carries every option with its comment and a second copy would drift as behavior lands under options that currently do nothing. This departs from tasks/template.md rather than following it. The template is built around a task that runs a model -- mesh, vertical grid, initial conditions, forcing, time step, cores -- and the analysis runs none, so applying it would give a page of empty headings. The page says up front why it is unlike its neighbors. Cover the things that actually block a first run and that nothing else tells you: that --model omega is required and -p is not given, since there is no build to detect the model from; that the year ranges must match what the simulation has; and that a bare $ in a config value is an error, which is where anyone who tries to write a file-name template by hand will end up. The Developer's Guide section covers the three pieces a new product builds on -- locating a simulation's files, AnalysisStep, and the range keying -- including the two details easiest to get wrong: that an undated template is a real case, and that an analysis group's streams can be time means or snapshots with different names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A step names the products it made and the facets that identify them, in a fragment written beside its outputs, so that a collector can find them without knowing how the work was divided into steps. Products keep the order they were added in, because that order is meaning: a gallery reads ANN, DJF, MAM, JJA, SON because that is the order the step plotted them in, and nothing has to sort it back. Only group and gallery shape the published index; every other facet is caption material now and filter material later, which is what keeps adding one -- a region, an observational reference -- from reaching the generator at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers the round trip, that products keep the order they were added in, and the mistakes a step can make: an entry missing a reserved key, two products claiming the same plot, and a fragment with no products. Also drops a guard against a facet shadowing a reserved key. The reserved names are ordinary parameters, so Python rejects that before the guard could run; the branch was unreachable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The collector reads every fragment, symlinks each product into a flat plots directory, and writes the merged manifest that defines the published set. It works from the fragments rather than from directory structure, so the work can be re-chunked later without disturbing output paths or links. Products are published by symlink from the step that owns them, so each file has exactly one owner and Polaris's output checking still applies. The published name is the facets in a fixed order, ending in the range of years, so it sorts and greps usefully and two ranges of the same product cannot collide. A product whose fragment is present but whose file is not on disk is reported and kept out of the merged manifest, rather than quietly vanishing from a gallery. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers the symlink rather than a copy, the published name, two ranges coexisting, order surviving the merge, and re-publishing over the links an earlier run left. Also covers the two cases the design calls out: a fragment naming a file that is not on disk is reported and excluded, and a product with no netCDF beside it still publishes its plot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Thumbnails are what let a reader decide which full images to open, and frequently answer the question without any full image being opened. The collector renders them, so the plotting steps stay unaware of how results are presented and the policy lives in one place. They are bounded in both dimensions rather than in width alone. A width rule charges the most for the tallest plots: a stack of global_stats time series is three times the pixels of a map at the same width, and does not sit in a grid beside it. Measured on the QU240 mock-ups, 320 by 240 gives about 12 kB for a map and 7 kB for a time series, against plots of 1.3 to 2.1 MB. The image is flattened onto white first, because the plots are written RGBA and JPEG has nowhere to put the alpha -- without it the background comes out black. A thumbnail already newer than its plot is left alone, so adding one product to an existing analysis costs one thumbnail rather than all of them. pillow becomes a direct dependency; it arrived with matplotlib before, but this uses it directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pages described the fragment as optional, which it no longer is. The User's Guide now says that a step with nothing to publish writes an empty manifest rather than none, and the Developer's Guide says why that is what lets the publish step declare each fragment as an input --- and that the two failures stay apart: a missing fragment is a step that ran without writing one, a missing pickle is a step that did not run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fragments are mandatory, but nothing wrote one: every step of the suite made its products without describing them, so the `publish` step stopped on nine missing inputs the first time the suite was run against the mock-up. The half that was missing is here rather than on a product branch, because this is the lowest branch where both `AnalysisStep` and the manifest writer exist. `AnalysisStep` writes it. `runtime_setup()` leaves an empty fragment before `run()` is called, and `add_product()` rewrites it as each product is described, so a step that makes products always has a current fragment on disk and a step author is asked to remember nothing. A rule that had to be obeyed at the end of `run()` would be forgotten in exactly the step that made nothing, which is the case that stops `publish`. Rewriting the fragment from `runtime_setup()` also clears out what a previous run described, so it never outlives the products it names. Which steps write one is `makes_products`, the attribute that already decides which steps `publish` depends on. The two questions are asked separately even though the answer is the same today: nothing here knows about the gallery, so a step that writes a fragment without being published is a change to this test alone. `add_product()` fills in the step's range of years unless the caller passes its own. Every product of an analysis step covers the step's range, and that range is what keeps the published names of two analyses of one simulation apart, so it is not something to be passed by hand at each call. The fragment is also a declared output, which names the step itself if it ever fails to leave one, rather than naming `publish` for a missing input. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
What is worth pinning is the guarantee rather than the call: a step that makes products has an empty fragment on disk before it runs, and a current one after each product it describes, so a step that makes nothing is not a step that wrote nothing. A rerun starts from an empty fragment, so what an earlier run described cannot outlive it, and the climatology, which makes what the maps are plotted from, writes none and refuses to describe a product. The range of years is checked where it matters: it comes from the step unless the caller passes its own. The publish step's test now runs the steps it depends on rather than writing their fragments by hand, so the two halves are exercised together --- a step writes a fragment through `AnalysisStep`, and the publish step reads it through the input that was declared for it. The suite that made nothing is what a real run of the scaffolding reaches today, and it says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Developer's Guide showed a step building a `Manifest` and writing it at the end of `run()`, which is the rule that would be forgotten in the step that made nothing. It now shows the call a step really makes, `add_product()`, and says what the step class does around it: an empty fragment before `run()`, a rewrite as each product is described, and the step's range of years filled in unless the call passes its own. The User's Guide says that the manifest a step with nothing to publish leaves is left for it rather than written by it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The report said how many products were published and nothing about where they came from, so a run that published nothing said nothing about the nine steps it read. It now counts the manifests as well, and names the steps whose manifest was empty. That is where a step that ran and made nothing is named at all: it is an absence in the gallery, and the log is the only place an absence can be seen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The report was the one part of the collector nothing exercised. What is worth pinning is what a reader of the log can count on: the number of manifests read alongside the number of products published, and the name of each step whose manifest was empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`pillow` and `jinja2` both become direct dependencies in this branch, so an existing environment is out of date and the load scripts compare this version to notice. This replaces an identical bump to alpha.5, which the rebase onto tranche 0.6.1 dropped because main had taken alpha.5 for itself. The number is the only thing that changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The analysis suite needs the same two-panel global statistics plot that StatsAnalysis already makes, but from a completed simulation rather than from a forward step in its own task. Move the plotting into polaris/ocean/analysis_plots.py, where it takes arrays a step has already selected and so knows nothing about where they were read from, and have StatsAnalysis call it. The plot it produces is byte-identical to the one StatsAnalysis made before, which matters because StatsAnalysis is in the omega_pr suite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both models name a global statistic after the field and the statistic, so the names can be built rather than listed. Building them is what lets a step ask for a field nothing has mapped, and what keeps the time reductions Omega can write from needing a name apiece. Write down where the two models differ while we are here. MPAS-Ocean computes a root-mean-square where Omega computes a standard deviation, and those are different quantities, so neither model's list has an entry for the other's. A step that wants a standard deviation from MPAS-Ocean has to derive one, which is a conversion and belongs in the step rather than in a table of names. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A step that plots global statistics has to get its list of fields from somewhere, and the honest source is the file: a variable whose name ends in one of the statistics the model computes is a statistic of whatever comes before it. Add that inverse of the name construction, so a step can plot what a simulation wrote rather than what someone thought it would write. Asking for no statistics in particular now means every statistic the model computes, for the same reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
mpaso_to_omega.yaml carried sixteen entries mapping composite global statistics names, one per (field, statistic). That does not scale: two of the fields in the analysis suite's default list already had no entries, and Omega's time means would need one per (field, statistic, period). Four of the entries also equated layerThickness with PseudoThickness, which the conventions deliberately refuse to do, and all four of the Rms entries equated a root-mean-square with a standard deviation. Delete them and have StatsAnalysis build the names instead. While it is being changed: it took its list of fields from the model's state variables, which is the list that says what an initial condition must contain, not what a run wrote statistics for. That is why it would have asked Omega for SurfacePressure, which Omega's GlobalStats group does not write. It now takes the list from a config option that defaults to whatever the file holds, and skips a field or statistic the run did not write. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
map_var_list_from_native_model() built its result by walking the map and keeping the entries whose Omega name appeared in the list. A variable with no entry therefore vanished, and the ones that survived came back in the map's order rather than the caller's. Its counterpart in the other direction leaves an unmapped variable alone, as the conventions say it should: a field only one model has is not renamed, because there is nothing to rename it to. The visible effect is on baseline validation, which round-trips the state and vertical-coordinate variables through both directions to get a list in MPAS-Ocean names. For Omega that quietly dropped PseudoThickness, SurfacePressure and RefPseudoThickness, so the fields that are Omega's alone were the ones never compared. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The step now reads the statistics files for the years it covers, plots the mean, minimum, maximum and standard deviation of each field it finds, and writes a netCDF beside each plot holding exactly what was drawn. A field or statistic the simulation did not write is reported and skipped; only a file with none of them at all interrupts, since that usually means the year range is wrong rather than that a variable is missing. Which plots it makes therefore is not known until it has read the simulation, so it registers each product as it writes it. The time axis is in simulation years, taken from the calendar date rather than from an elapsed time, so a series covering years 21 through 40 is drawn at 21 through 41 whatever reference date the model wrote its time coordinate against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The User's Guide gains a section on what the product makes -- the two-panel plot, the netCDF beside it, and the two config options that govern it -- and its row in the products table now says it exists. The Developer's Guide gains the naming module, since that is the piece the next analysis product will build on, and the note that a step which cannot know its products until it has read the simulation registers them as it writes them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The publish step publishes a product as its group, then the name the step gave it, then the range -- so a file the step already prefixed with its group comes out as global_stats_global_stats_temperature_0001-0001.png. The climatology maps write temperature_ANN_-100m.png and let the publisher add the group, which is the convention the design's example follows; the time series were the odd one out. The files are unambiguous without the prefix, since they sit in the step's own directory under ocean/analysis/global_stats/<range>/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The step already registers the plot and the netCDF beside it as it writes them; describing the product to the manifest is the same event, so it happens in the same place rather than in a second one a step can forget. A gallery per field, so that the landing page shows one thumbnail for each field rather than a single one standing for the whole task. The range is filled in from the step, since it is what keeps two analyses of the same simulation apart and no caller should have to repeat it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fragment names one product per field, in the global_stats group, with the plot and its data and the range the step covers. The helper now calls runtime_setup() before run(), which is what Polaris does and what writes the empty fragment; a step tested without it is not being run the way it will be. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
map_var_list_to_native_model() asserted that the MPAS-Ocean-to-Omega variable map had already been read, which only happens in set_model() or as a side effect of one of the writing methods. A step that just reads model output -- an analysis step, say -- has neither, so translating a list of names back to the ones in the files tripped the assertion. Prime the map lazily, the way the other public methods on Ocean already do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The step builds its variable list at run time from config options -- the union of the fields requested for maps, what heat content needs, and the vertical geometry -- and maps it back through mpaso_to_omega.yaml, since -v takes the names as they appear in the files. The mass-like thickness joins the list already spelled the way the files spell it, because it is the one variable the analysis deliberately does not translate. Two departures from the command line in the design, both forced by NCO 5.3.9 and neither changing what is computed: - "-a sdd" was dropped in NCO 5.3.7 and is now only a deprecation warning. The seasonally discontinuous December convention it used to select is now ncclimo's behavior, verified numerically against day-weighted means. - ncclimo needs "-c" to name its output when the input files are given positionally rather than generated, so the case name comes from simulation_name. A variable the simulation did not write is reported and left out rather than failing the step, since the default field list asks for velocity components and mixed-layer depth that Omega does not write yet. Scratch space is pointed at the step's own work directory so that two climatologies cannot collide on it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three years of synthetic monthly means with Omega names and CF time metadata are run through the step, and the monthly, seasonal and annual climatologies are compared against day-weighted means computed in the test. What this pins down is our invocation, not ncclimo: that Omega files are read without -P mpaso, that the seasons are day weighted, and that December comes from the same year as the January beside it -- the convention that used to be "-a sdd" and is now ncclimo's behavior, so nothing on the command line would tell us if it changed back. The test is skipped when ncclimo is not on the path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each field group's step loops over the seasons to plot, the fields of its group, and the vertical reductions requested for the ones that have a vertical dimension, and writes a netCDF beside each PNG with exactly what was plotted and what produced it. Fields with no vertical dimension get one map per season and no reduction label, since there is nothing to reduce and nothing to name it after. The mosaic descriptor is built once per step and shared, which is where most of the cost of plotting a global mesh is. The climatology is a dependency rather than an input file because ncclimo names its output for the case and the dates, so the file names are not known until it has run; the step then globs on the season. Three things are reported and skipped rather than failing the step: a field the simulation did not write, an elevation the vertical machinery cannot reach yet, and the derived heat_content group. Nothing silently produces fewer plots than the config asked for. Outputs are registered during run() rather than setup(), because a step that declared a plot at setup for a field the simulation never wrote would fail the missing-output check instead of skipping it. The mesh and vertical-coordinate files are the simulation's initial condition, so the few fields wanted from them are read with a plain rename rather than open_model_dataset(), which would solve the equation of state to derive a specific volume nothing here asks for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The bookkeeping around the plots rather than the plots themselves: a netCDF is registered beside every PNG, every registered output is a file that was written, a field with no vertical dimension gets no reduction label, and the three skips -- a field the simulation did not write, an elevation the vertical machinery cannot reach yet, and the derived heat_content group -- each say so and leave the step passing. Plotting is stubbed, since drawing a global mesh needs a mesh; the climatology it reads is a handful of cells written with Omega names into files named the way ncclimo names its own. Test the field-to-section mapping separately, including that every field a group can map has the section that mapping names in analysis.cfg. The step raises when a section is missing, so without that check a field whose section was misspelled would pass here and fail on a real run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The User's Guide gets the climatology_maps row flipped and a section on what the product is: the shared ncclimo climatology and its two conventions, the PNG and netCDF pair each map writes, the config options that govern them, and -- explicitly -- the three things the default config asks for that are reported as skipped rather than quietly dropped. The Developer's Guide gets the pieces a second product will build on: how a step reads a climatology whose file names ncclimo chooses, how the vertical reduction is split between parsing and applying, and the one-based indexing of minLevelCell and maxLevelCell, which is the detail most likely to be got wrong twice. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The reason the map step reads a few fields by hand rather than through open_model_dataset() is that the files a simulation names for its mesh and vertical coordinate are commonly its initial condition, carrying a full model state. That is what the mock-up does; it is not a guarantee, and the docstring should not claim it is one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The map step now takes zMid and zInterface from the climatology and hands them to the reductions, so every elevation a user asks for is produced. Nothing is reported as skipped for want of the geometry, and IMPLEMENTED_KINDS goes with the report: the reductions are simply parsed and applied. The geometry comes from the climatology rather than from the vertical coordinate file because it is a geometry that moves. What a map at -100 m is a map on is the climatological-mean position of that surface, which is what the mean of zMid is; the vertical coordinate file supplies only the topmost and bottommost valid layer of each column, which do not move. A simulation that did not write the geometry stops the step with a message saying so, rather than being skipped the way a missing plotted field is. Polaris cannot reconstruct it -- geometric thickness is derived from pseudo-thickness through specific volume, and the monthly mean of that product is not the product of the monthly means -- so there is nothing to fall back on, and a run without it is out of spec rather than merely configured without a field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A map at an elevation is plotted, its values are the field interpolated to that elevation rather than the nearest layer, and a simulation that wrote no vertical geometry stops the step with a message naming what is missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
OceanIOStep.add_produced_file() exists for exactly this case -- a step that plots what a simulation wrote and does not know at setup which plots it will make -- and the global statistics step already uses it. The map step was calling add_output_file() with a path it had expanded itself, which is the same thing said twice. It could not use the helper before: add_produced_file() arrived on the global statistics branch, which was a sibling of this one until the branches were put in a line. Now it is below, so the two steps register a product the same way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The step already registers a map's plot and the netCDF beside it as it writes them, so describing the product happens in the same call rather than in a second one a step can forget. A gallery per field, so the landing page shows one thumbnail for each field rather than one standing for every map. The caption is the title the plot already carries, and the facets are the field, the season and the vertical reduction -- which is what identifies a map once its date range is known. The heat content maps go through this same call, so they are described here too, with heat_content as their gallery. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every plot the step made is described and nothing else is, each carrying its group, gallery, field, season and reduction, with the data file beside it. The seasons appear in the order the step plotted them, which is the property the gallery relies on to read ANN, DJF, MAM, JJA, SON without a sort key existing anywhere. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xylar
force-pushed
the
add-omega-analysis-climatology-maps
branch
from
September 1, 2026 17:20
f771421 to
3ccc656
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ncclimo climatology and the first climatology maps
This is items 6 and 7 of the order of work in
ocean_analysis_initial.md, and it is the point at which theomega_analysissuite stops reporting what it would read and starts producing plots. A sharedClimatologystep computes monthly, seasonal and annual climatologies of a simulation's monthly means withncclimo, and oneClimatologyMapsstep per field group plots global maps from them on the native MPAS mesh, writing a netCDF beside each PNG holding exactly what was plotted. Every elevation a user asks for is produced, since the vertical reduction is in place below this branch.Three corrections to the design's
ncclimocommand lineThe design was written against an older NCO, and its command line does not run under the 5.3.9 in the current environment. None of the three changes what is computed, but the first is worth a careful look because it is the one that could go wrong silently later.
-a sddwas removed in NCO 5.3.7 and is now a no-op that prints a deprecation warning. The seasonally discontinuous December convention it used to select — every year in the range contributing its own December, so that no data from outside the range are needed — isncclimo's own behavior now, and it says so in its log asjfd-mode. Dropping the flag is therefore correct rather than a concession, but it means nothing on the command line records which convention is in force, so the convention is pinned by a unit test comparingDJFagainst a day-weighted mean instead.-c <caseid>is required when the input files are given positionally rather than generated from a case name; without itncclimorefuses because it does not know what to name its output. The case name comes fromsimulation_name.-i <dir>is required, becausencclimoresolves the input file names it is given against its input directory. Absolute paths are concatenated onto it and are not found, which is not obvious from the error.The design's one genuinely risky claim held:
ncclimoreads Omega output with no-P mpaso, giventime_bnds. These corrections belong on the design branch as well; they are not made here, since design changes do not go on this branch.One design change
The design assumes every name in the
fieldsconfig option is a field the model wrote, which is what makes mapping the variable list back to Omega names well defined. It is not true of the default list: it asks forvelocityZonal,velocityMeridionalandmixedLayerDepth, none of which Omega writes yet.ncclimowould fail on-vbefore the map steps ever got their chance to skip them.So the climatology reads the variable names out of the first monthly-mean file and leaves out the ones that are not there, reporting them. This is the rule the design already states for the map steps — "a field the simulation did not write is reported clearly and its maps are skipped, rather than failing the whole step" — applied at the point where it first bites rather than one step later.
What is reported and skipped rather than failing
Three things the default config asks for cannot be produced yet, and each says so in the step's log rather than quietly producing fewer plots than were requested:
-100.0, which need linear interpolation in the vertical;velocityZonalandvelocityMeridional, which Omega does not write and which this design deliberately does not reconstruct offline;mixedLayerDepth, likewise a diagnostic Omega does not compute yet.The
heat_contentfield group exists and derives nothing yet, and reports that too. The defaultelevationsis left alone rather than temporarily changed to something fully supported, so that the config keeps saying what the product is ultimately for and there is nothing to remember to revert.Reading the vertical geometry
The map step takes
zMidandzInterfacefrom the climatology and hands them to the reductions inpolaris/ocean/vertical/elevation.py, which lands below this branch. The geometry comes from the climatology rather than from the vertical coordinate file because it is a geometry that moves: what a map at -100 m is a map on is the climatological-mean position of that surface. The vertical coordinate file supplies only the topmost and bottommost valid layer of each column, which do not move.A simulation that did not write the geometry stops the step with a message saying so, rather than being skipped the way a missing plotted field is. Polaris cannot reconstruct it — geometric thickness is derived from pseudo-thickness through specific volume, and the monthly mean of that product is not the product of the monthly means — so a run without it is out of spec rather than merely configured without a field.
get_valid_level_rangeis in the same module and is the one place that convertsminLevelCellandmaxLevelCellfrom the one-based indexing of MPAS-Ocean's Fortran — which Omega'sMinLayerCellandMaxLayerCelluse too — to the zero-based indices the reductions use. It is a named function rather than an inline- 1because an off-by-one here is a plausible bug in every step that reads them, and this way it is a bug in one function. A simulation with no ice-shelf cavities need not writeminLevelCellat all, so its absence means the topmost valid layer is the topmost layer of the grid.Ocean.map_var_list_to_native_modelnow primes its own name map, the way the other public methods onOceanalready do. It previously asserted the map had been read, which only happens inset_modelor as a side effect of one of the writing methods, so a step that merely reads model output tripped the assertion.Notes for review
The map step registers its PNG and netCDF outputs during
run()rather thansetup(). This is the one place where "report and skip" and Polaris's missing-output check pull against each other: a step cannot know at setup which fields the simulation wrote, and a step that had declared a plot for a field that turns out to be absent would fail the missing-output check instead of skipping it.The climatology is a dependency of the map steps rather than an input file, because
ncclimonames its output for the case and the dates and those names are not known until it has run. The map steps then glob on the season in the dependency's work directory, which is what the design asks for.find_climatology_filealso knows thatncclimonames the twelve monthly climatologies by month number rather thanJANthroughDEC, which matters becauseplot_seasonsaccepts the latter.The mesh and vertical-coordinate files a simulation names are often its initial condition, carrying a full model state. Opening one with
open_model_datasetsolves the equation of state to derive a specific volume, which is a lot of work to obtain two integer fields, so the map step reads the few fields it wants and translates their names withmap_from_native_model_varsinstead.A field with no vertical dimension gets
<field>_<season>.pngwith no reduction label, since there is nothing to reduce and nothing to name it after. The design only names the reduced case, so this is a choice rather than a reading of it.Since this PR was opened
Each map is now described in a manifest fragment, so the
publishstep below can put it in the generated gallery, with a gallery per field and the field, season and vertical reduction as its facets. The heat content maps go through the same call, so they are described here too.This is one of five remaining PRs, reviewed in order
The work is a chain in dependency order. GitHub cannot base a cross-fork PR on another fork branch, so every one of these is based on
mainand its diff contains everything below it. Reviewing them in order is what keeps each diff small — review only the commits this branch adds.add-omega-analysis-scaffoldingadd-omega-analysis-publisheradd-omega-analysis-global-statsadd-omega-analysis-climatology-mapsadd-omega-analysis-heat-contentTwo are already merged: #706, the design documents, and #735, the vertical reduction kernel.
A gallery generated by the whole chain against the QU240 mock-up is published here, which is the quickest way to see what the suite produces: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/xasaydavis/omega_analysis_qu240_mockup_20260830/
Checklist
api.md) has any new or modified class, method and/or functions listedTestingcomment in the PR documents testing used to verify the changes