Add the publisher and gallery to Omega analysis - #736
Draft
xylar wants to merge 112 commits into
Draft
Conversation
Adds docs/design_docs/ocean_analysis.md, the umbrella document for analysis of Omega and MPAS-Ocean simulations in Polaris. It records the long-term direction: why the capability is written from scratch with Polaris and Omega in mind rather than ported from MPAS-Analysis, where the line falls between diagnostics Omega computes in situ and those Polaris computes offline, and a three-phase roadmap. Most requirements are deliberately placeholders to be filled in as each piece is designed. The exceptions are the conventions that the rest of the family depends on: MPAS-Ocean variable and dimension names as the Polaris standard, observations remapped onto the MPAS mesh rather than the model onto a comparison grid, and per-year decomposition of expensive work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds docs/design_docs/ocean_analysis_initial.md, designing the set of analysis capabilities the E3SM Ocean Team has committed to delivering for Omega's initial coupled runs by September 15, 2026. The deliverable is an omega_analysis suite, pointed at a completed simulation through a user-supplied config file, that produces map-view climatologies at configurable elevations, ocean heat content over elevation ranges as both maps and a global time series, time series from Omega's GlobalStats output, and a latitude-elevation plot of the global MOC, each with the netCDF behind it. The document also covers the Omega-side dependencies it rests on (monthly means, the geometric vertical coordinate, and a mixed-layer depth diagnostic with an offline fallback), and how repeated analysis over different date ranges reuses per-year work through range-keyed and year-keyed shared steps. Out of scope and stated as such: zppy integration, comparison with observations, regional analysis, and a global MOC time series. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Addresses cbegeman's review comment asking whether analysis should use
PseudoThickness directly rather than converting it to a thickness.
Omega prognoses pseudo-height, z-tilde = -p / (rho0 g), so PseudoThickness
is a normalized pressure increment in meters, and
rho0 * h-tilde = rho * h = dp / g
is the layer's mass per unit area, exactly, by hydrostatic balance. The
geometric thickness is the derived quantity, h = rho0 * alpha * h-tilde,
which is why it needs the equation of state and cannot be recovered offline.
Record this in the conventions as a deliberate exception to the
MPAS-Ocean-names rule: PseudoThickness is not translated to layerThickness,
because the two mean the same thing for a mass-weighted integral and
different things for a geometric one. Analysis asks for geometry via zMid
and zInterface, and for mass via a get_layer_mass helper that reads each
model's own mass-like thickness variable. This matches what
mpaso_to_omega.yaml already does for the state variables.
Also correct the vertical-geometry section, which was missing the factor of
rho0 in Omega's accumulation of geometric elevation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Addresses cbegeman's suggestion to write the heat content integral with the
full density and with pseudo-thickness weights.
Heat content becomes a change of variable rather than an approximation:
Q = cp0 * int(rho * Theta dz) = rho0 * cp0 * int(Theta dz-tilde)
~ rho0 * cp0 * sum_k Theta_k * w-tilde_k
Because rho0 * h-tilde_k is the layer's mass per unit area exactly, a range
covering whole layers carries no reference-density error at all. This
removes the in-situ-versus-reference density error of the MPAS-Analysis
formulation, which weighted by geometric thickness. The only quadrature
error left is treating Theta as uniform within a layer.
The geometric coordinate now enters only through the partial layers at a
range boundary, as w-tilde_k = (w_k / h_k) * h-tilde_k. That fraction is
exact with respect to Omega's own discretization, since Omega uses a single
specific volume per layer, making z linear in z-tilde across the layer.
Ranges stay geometric, and the doc now says why: "0 to 700 m" means 700
geometric meters in MPAS-Analysis and in the observational products this is
compared against. Pseudo-depth ranges would be more natural for a
mass-conserving model, and the trade-off is stated so the choice can be
revisited.
rho0 is no longer a config option. Unlike cp0 it is not a modeling choice
we are making: it is the constant that defines pseudo-height, so it must be
the model's, read from the same PCD constant that pstar.py uses to build the
coordinate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Follows the algorithm change in the previous commit through to the implementation and testing sections. elevation_range_weights now takes the layer mass and returns mass per unit area within the range, computing the geometric overlap only to form the partial-layer fraction; heat_content drops its density argument. Passing interface differences as the layer mass recovers the purely geometric weights, which is what the elevation-slice utilities want. The one place that knows how each model spells its mass-like thickness stays get_layer_mass. The unit tests are rewritten in pseudo-height. Every synthetic column is built with h-tilde != h, so a formulation that confuses the two coordinates fails rather than coincidentally passing, and the whole-column result is checked to be invariant under perturbing the geometric interfaces while holding pseudo-thickness fixed. The previous tests leaned on a uniform-density column, in which the two coordinates agree and the distinction being tested disappears. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Addresses cbegeman's comment that zonal and meridional velocity should be the default Omega monthly output for memory reasons, with offline reconstruction available when they are absent. The requirement now asks for reconstructed velocities at cell centers rather than normalVelocity, and says why: two cell-centered fields are about two thirds the size of one edge field, and nothing else in this analysis reads normalVelocity. Offline reconstruction becomes the fallback for simulations that wrote the edge field rather than the intended steady state. The accuracy argument is unchanged and still stated -- reconstruction is linear, so both paths give the same answer -- so the preference rests on output volume alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Addresses cbegeman's comment asking that the design be explicit about using the shared steps infrastructure, and that it give a rough estimate of how many steps a typical analysis run should produce, so we can tell when a design choice is putting us over the edge. The algorithm design now names shared steps and get_or_create_shared_step where the per-year decomposition is introduced, rather than leaving it to the implementation section, and states that no caching layer of our own is involved. A new subsection works the count: about 130 steps for a 60-year record with a 20-year climatology, scaling linearly with record length. It names the thresholds worth noticing -- a few hundred is ordinary, a thousand is worth measuring setup time at -- and the two levers if we get there, which are merging the per-year heat content and mixed-layer depth steps, and coarsening the key from a year to a decade. Merging is preferred since it costs nothing in reuse. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Responds to cbegeman's suggestion to drop the claim that such fields get an entry in mpaso_to_omega.yaml, on the grounds that the mapping only holds variables that do have an MPAS-Ocean counterpart. The mechanism is right but the framing was wrong. The keys in that file are Polaris-standard names, which are MPAS-Ocean names wherever MPAS-Ocean has the field. A field new to Omega still needs an entry, because translation is a rename on read and a field without one arrives under its Omega name; what it does not need is a counterpart in MPAS-Ocean. Also note the one field this design deliberately does not translate, PseudoThickness, and that get_layer_mass is the sole intended branch on the model name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Analysis is a combinatorial workload, and how it is divided into steps and directories is the decision most likely to be regretted: cheap to get wrong early, expensive to change once results are archived and linked to. Record the principles that govern that division in the umbrella, so that individual design documents apply them rather than re-deriving them. The ten principles, in brief: 1. Two trees, two audiences. The work tree is machine-facing and should be shallow and uniform; the staging tree is human-facing. Neither is compromised for the other. 2. Products are described by a manifest, not by their path. A path has one dimension and the metadata has six. This is what lets the work be re-chunked later without breaking output paths, links, or the gallery. 3. Steps are for caching and selection, not for parallelism. A process pool inside a step is cheaper, costs no inodes, and works today. 4. Decompose along the axes a user edits between runs. Regions and observational references multiply, so they must not be step axes; one step per plot is never the answer. Fields are chunked by group, since things computed together belong together. 5. Directory levels must earn their keep and name things rather than mechanisms. If a level's best name describes how the work was chunked, the level is wrong. 6. Steps cache what the user asked for; files cache what the computation needed. Internal chunks of a divisible computation use a seeded accumulator instead of a directory apiece. 7. Discovery is scoped by construction and validated by content. The path establishes the search scope; a provenance stamp establishes admissibility. 8. A cache is an intermediate product, so its form follows its consumer. 9. Size steps for the production case, not for a regression test. 10. Split computation from plotting only where computation dominates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rewrites the two requirements that had the per-year decomposition baked into them, following the organizing principles added in the previous commit. Scalability and restartability now mandates a seeded accumulator per expensive product rather than a shared step per simulation year. It records why the change was made: a step per year paid a directory, a pickle, a config copy and a log file for a chunk no user ever asked for, and it made the completion marker the only cache-validity check there was, so a changed kernel or constant would have been inherited silently. It also notes the two properties that now come with the pattern rather than being added to it -- a partial cache is a valid starting point for a retry, and the remaining work is a pool inside one step. The concurrent-execution requirement had called the per-year decomposition "the real prize" for task parallelism Phase 2. That bullet inverts: the largest source of concurrency is now deliberately inside a step, which is a trade in favor of a capability that exists today. Analysis needs task parallelism less than this document previously claimed, and what it stands to gain is the coarse overlap of independent products. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fills in the presentation layer that the presentation-and-provenance requirement had left as "details to be added", following principles 1, 2 and 7. Every step writes a manifest fragment naming each product it made and the facets that identify it; a cheap collector gathers the fragments, publishes into the staging tree by symlink, and generates the index. Working from fragments rather than from directory structure is what allows the work to be re-chunked without disturbing the output. The staging tree is shallow with descriptive filenames plus a generated index, following MPAS-Analysis, where a gallery is generated rather than navigated. Provenance gains a second job: it is what makes an inherited cache safe to inherit. Phase 1 ships a minimal stamp -- simulation identity, the config options governing the product, and a hand-maintained kernel version -- and a record whose stamp does not match is recomputed rather than inherited. Two deferred items are added: a content-addressed stamp covering the full dependency graph, since a hand-bumped integer is correct only if it is remembered; and measuring Polaris's per-step overhead, so the step sizing targets rest on a number rather than on judgment. Code organization gains the manifest writer and the shared accumulator support, the latter because principle 7 is only as good as its least careful implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Applies principles 6 and 7 from the umbrella to the two products that were decomposed into one shared step per simulation year. Each becomes a single step keyed by the requested range, which discovers the cache files left by earlier runs of the same product, inherits the months they cover, computes only the rest over a process pool, and writes a complete cache for its own range. The shared machinery lives in accumulate.py so that a product supplies only its kernel and its cache layout. The design records why the per-year decomposition was wrong rather than just replacing it. It paid a directory, a pickle, a config copy and a log file for a chunk no user asked for, and it made the completion marker the only cache-validity check there was, so a changed kernel or constant would have been inherited in silence. Because this has software hunting for data on disk, four properties are stated explicitly: the search scope is only sibling directories of the same product; admissibility comes from a provenance stamp rather than from location, since task subdirectories do not encode which simulation was analyzed; only completed steps are candidates, which also disposes of half-written caches; and reuse is reported, with reuse_previous and reuse_search_path providing an explicit mode for anyone who wants determinism instead of discovery. Cache form follows the consumer, per principle 8. The heat content series is one file with an unlimited time dimension since a plotting step reads it whole; offline mixed-layer depth is monthly files because ncclimo reads monthly files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Responds to cbegeman's question about what the climatology_maps task does, and applies principle 4 from the umbrella. Reducing a field with a vertical dimension to a horizontal map is one operation with several cases -- slice at the surface, at an elevation, at a layer index, at the seafloor, or integrate over an elevation range. All of them turn nVertLevels into nothing. Keeping them behind a single apply_vertical_reduction entry point lets ocean heat content be a field group of the climatology maps rather than a product with its own step tree, code path, and config convention. A heat content map is a climatology map of a field that happens to be derived. Maps are chunked one step per field group. The field list is an axis a user edits between runs, so adding a field should cost that field alone; seasons and elevations are bounded and rarely change, so they are loops inside the step. The unit is a group rather than a variable because things computed together belong together: zonal and meridional velocity share a vector reconstruction, and heat content over several ranges shares one set of layer weights. Plots within a step are spread over a process pool rather than over steps, per principle 3, with the mosaic descriptor built once per step and shared. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Addresses cbegeman's comments that the years subdirectory is unhelpful, that climatology_maps was unclear, and that maps as a subdirectory earns nothing. Applies principle 5 from the umbrella: a level must earn its keep, and must name a thing rather than a mechanism. The layout now follows one rule, <product>/<period>/[<field group>], with the third level only for the one product that is chunked. Two levels are gone: - years/, which held one shared step per simulation year, because those steps are gone. Renaming it -- to offline_metrics or anything else -- would not have helped, since a directory whose best name describes how the work was chunked is a level that should not exist. - maps/, time_series/ and plot/, the single-step levels between a product and its period, because a level with one child and a name that repeats its parent earns nothing and costs depth on every path. Tasks no longer introduce directory levels of their own, and the ocean heat content task is gone: its maps are a field group of climatology_maps and its time series is heat_content_series. The step-count budget is rewritten against the new structure. The count drops from about 130 to 11 for a typical analysis, but the property that matters is that it no longer grows with the length of the record -- or, in later phases, with regions, seasons, or elevations, which are loops inside steps. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Applies principles 1, 2 and 10 from the umbrella, and answers cbegeman's question about whether plot-producing steps need to be identifiable by name or by symlinks in an html directory. Neither, as it turns out. Every step writes a manifest fragment naming each product and the facets that identify it -- field, season, vertical reduction, date range, and later region and observational reference. A publish step collects the fragments, symlinks products into a shallow staging tree with descriptive filenames, and generates an index over them. Encoding those facets in the path instead would mean a directory level per facet, and a path has one dimension where the metadata has six. Working from fragments rather than from directory structure is what allows the work to be re-chunked without disturbing output paths, links, or the gallery. Products are published by symlink so each file has one owner and Polaris's output checking still applies. Replotting answers the other half of cbegeman's comment on re-running an unchanged range. A replot config option removes the completion markers of the plot-producing steps and only those, so a colormap change costs the plotting and not the climatology or the accumulator caches. This is also why plotting does not need its own step: for a map, plotting is the expensive part, so the split would buy nothing the option does not. MPAS-Analysis replots unconditionally, which is rarely useful; the default here is not to. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Leftovers from the previous commits: parse_elevation_spec and extract_elevation_slice were still named alongside the vertical reduction that replaced them, and the map unit tests still referred to the old slicing function. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Addresses cbegeman's comment that we also need a suite for testing that includes a short Omega forward run. Everything else in this design consumes output from a simulation Polaris did not run, which is the point of the capability and also the reason nothing in it would otherwise be exercised by a suite we run ourselves. A regression in the climatology, the vertical reduction, or the accumulator would be found by a person analyzing a coupled run, which is the most expensive place to find it. The task runs a QU240 init chain and forward run with the Omega monthly-mean output fragment turned on, then runs the analysis products over it. One simulated year rather than a month or two: ncclimo needs whole years to form seasonal and annual climatologies, and the climatology is the step with the most external surface area, since it shells out to NCO and depends on Omega's time metadata being read by a tool we do not control. It is its own suite rather than an addition to omega_pr or omega_nightly, since it costs a forward run and is blocked on Omega capabilities the PR suite must not be blocked on. Adding it to omega_nightly once it is stable and its cost is known is the expected follow-up. The checks are the ones that do not need the run to be long enough to be scientifically meaningful: manifest completeness, plot and netCDF pairing, a finite positive heat content series, the annual climatology equaling the mean of the monthly ones, and a sea-surface map equaling the top valid layer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Principle 3 had gone too far. It said steps are for caching and selection and not for parallelism, and pointed at in-step process pools as the cheaper way to get concurrency. Taken to its limit that rebuilds exactly the ceiling task parallelism exists to remove: MPAS-Analysis parallelizes with multiprocessing.Process, which cannot span nodes, and that ceiling bites at the resolutions Omega targets. A step that absorbs the whole workload into an internal pool is not task-parallel at all, however parallel it is inside. The principle now distinguishes the two kinds of concurrency -- across steps, which scales past one node and is what the scheduler sees; and inside a step, which is free and available today but confined to one node -- and says to decompose into enough steps that the scheduler has something to balance, using an in-step pool only below that granularity. Where the load-balancing floor and the invalidation axes disagree, the floor wins: a cache coarser than ideal costs a recomputation, a step coarser than ideal costs a node. "Enough to balance" is a property of the machine rather than of the science, so a freely divisible workload should divide into a configurable number of steps rather than a number derived from the data, which also keeps the step budget from growing with the length of a simulation. Principle 6 gains the corollary that an accumulator may be more than one step, and usually should be. Sharding one is free precisely because inheritance is decided by content rather than by path, so how an earlier run was divided is invisible to a later one. That is what separates an accumulator from a path-keyed chunk: chunking the latter restricts reuse, chunking the former is purely a load-balancing decision and can change between runs. Principle 9 gains the matching target -- no step should be more than a fraction of the suite's total work -- and a note that this one is easily lost because nothing fails when it is missed; the suite just runs on one node. Principle 3 also now points at the task-parallel groundrules for the properties that make a step eligible for concurrent scheduling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reverses the position taken in "Clarify naming of fields with no MPAS-Ocean counterpart". Both reviewers read it the same way and they were right: for a field MPAS-Ocean does not have and is not expected to gain, a Polaris-standard synonym is a name no model ever writes, and the mapping entry recording it is a rename in appearance only. The rule is now stated once, and it is narrower than "everything is translated": the mapping reconciles two spellings of the same quantity and is not a naming authority. Three cases follow. Both models have the field, so it is mapped and analysis uses the MPAS-Ocean name. Only Omega has it, so there is nothing to reconcile and analysis uses the Omega name as written. Or both have a similar name for different quantities -- layerThickness and PseudoThickness -- and there is deliberately no entry. That third case was already a carve-out in the conventions; stating the rule this way makes it an instance rather than an exception. The cost is that Omega's spelling appears in analysis code and config section names for Omega-only fields, which is a small inconsistency of style and an accurate one. If such a field later gains a counterpart or is renamed upstream, adding an entry then is a one-line change, so there is nothing to be gained by adding it in advance. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The implementation section claimed the layer weights "depend only on the range and not on the season" and are therefore "computed once per season and reused across ranges", which contradicts itself and is wrong either way: the weights are exactly the range-dependent part. What is shared across ranges is the expensive part -- the climatology of conservative temperature, pseudo-thickness and the interface elevations is read once per season, and each range is then a masked weighted sum over levels. That is why elevation ranges are a loop inside the step rather than an axis of decomposition. Also states explicitly what was previously only implied: the heat content maps have no dependency on the accumulator. They come from the climatology like every other map, and the accumulator exists only for the time series, where the per-month values are the product rather than an intermediate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A single accumulator step would have been the largest piece of work in the suite and invisible to the scheduler: however many cores its internal pool used, it would run on one node. That is the ceiling principle 3 exists to avoid, and it is what collapsing the per-year steps into one accumulator had quietly reintroduced. The requested range is now split into a configurable number of shards, each an independent accumulator over its slice, followed by a cheap merge step that concatenates, plots and publishes. Three properties make the split a knob rather than a commitment. Sharding costs nothing in reuse, because a shard asks which months it needs and which some earlier run already produced, and neither question refers to how that run was divided -- so the count can differ between two analyses of the same record. The count comes from config rather than from the data, which would put the step budget back on a growth curve, or from the allocation, which is not knowable at setup since the step tree is fixed there while the job comes later. And shards split the requested range rather than the missing work: splitting the missing work would balance better on incremental runs, but setup would have to read the filesystem, so two setups with identical config could produce different step trees. Determinism is worth more than balance in the case that is cheap anyway. The step budget goes from 11 to 27 for a typical analysis, with the largest term now set by how much concurrency the machine can use rather than by how long the simulation ran. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ncclimo no longer consumes anything Polaris produced. The offline mixed-layer depth accumulator forms its own seasonal means in its merge step, and the map step reads those directly. Handing the monthly files to ncclimo was wrong on three counts. It made the climatology, and therefore every map step behind it, wait on a full-record pass, which is the worst shape available under principle 3 -- and it was the only serial dependency of that kind in the suite. It required a Polaris-computed field in ncclimo's -v list, which the step builds by mapping Polaris names back to Omega names, and a field Omega never wrote has no Omega name to map back to. And it was the only thing forcing that cache's format, when the format should follow from how the data are used. The cost is a second seasonal-averaging path. It is a small function, it reuses the same season definitions and length-of-month weighting ncclimo is configured with including the seasonally discontinuous December convention, and the regression test already checks an annual mean against the weighted mean of the twelve monthly means, which is what keeps the two from drifting apart. The ncclimo notes now say explicitly that every name in -v is a field the model wrote, which is what keeps the mapping back to Omega names well defined. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The design hedged at length about neglecting the covariance between conservative temperature and layer thickness when heat content maps are computed from a climatology, and the umbrella carried a deferred item to revisit it. Both overstated the concern. The term is already dropped at every timescale shorter than a month, the moment the analysis works from monthly means rather than model time steps. Dropping it again from month to season is the same approximation over a longer averaging period, not a new one, so being scrupulous about the second while accepting the first in silence was inconsistent. Its size settles it: for the 0 to -700 m range, a seasonal sea surface height of order 0.1 m against a near-surface seasonal temperature anomaly of a few kelvin gives order 1e6 J/m2 against a total near 2.9e10 J/m2 -- order 1e-4 of the signal. That is an order-of-magnitude sketch rather than a bound, but several orders below anything that would change a conclusion. The deferred item is dropped. Computing per-month integrated maps and averaging those would cost a full pass over the 3D record per season plotted, which is a large price for 1e-4, and keeping it on the list implied an intention we do not have. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The requirement says the default ranges are "the surface to -700 m" and "the whole ocean", but the config default said 0.0:-700.0 and 0.0:bottom. Those are not the same thing: 0.0 is the resting sea surface, so the range excluded the water between it and the free surface -- a different amount in every column and every season, and never what "0 to 700 m ocean heat content" means anywhere it is reported. The defaults are now top:-700.0, -700.0:-2000.0, -2000.0:bottom and top:bottom. A top boundary resolves per column to the free surface at zInterface[minLevelCell], expressed as z_top = +infinity in the weight expression, mirroring how bottom is already handled as z_bot = -infinity. This also makes the whole-column range cover every valid layer, so that every layer is whole and the geometric coordinate drops out of that answer entirely. The previous text claimed that property for 0:bottom, where it did not hold. Range labels in filenames change from a hyphen to _to_, since the elevations are themselves negative and -2000m--700m is unreadable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This reverts commit b982a88.
Redoes what the reverted commit got wrong. Having ncclimo consume the accumulator's output is fine; what it costs is the usual pattern of a single ncclimo call per component, once and for all, plus perhaps a reference climatology. A derived field cannot join that call -- it arrives in its own files, written by a different step, with a different variable list, and not until an expensive pass over the record has finished -- so it gets a second instance of the same Climatology step class. The gain from doing it this way rather than averaging the months ourselves is a single implementation of what a season is. The seasonally discontinuous December convention, the length-of-month weighting and the set of seasons are the same for a derived field as for a model field by construction rather than by test, with no second averaging path to keep in step. The scheduling consequence is stated rather than left to be discovered, since a serial chain behind an expensive step is what principle 3 warns against: it is confined to this product. The main climatology reads model output only and does not wait on the accumulator, so every other map step runs beside it. Only the mixed-layer depth maps sit behind shards, second climatology, maps -- and only in the fallback case, which exists because we would rather Omega computed the diagnostic in situ. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The implementation pseudocode opened bare relative filenames -- monthly means, climatology files, mesh.nc, output netCDF and PNG names -- and worked only because Polaris chdirs into each step's work directory before running it. That is exactly the pattern the task-parallel groundrules forbid under working-directory independence: a step that depends on the process working directory cannot run beside another step in one process and cannot be sent to a worker on another node at all. Declaring inputs and outputs by relative name is unchanged, since setup already resolves those. The change is in the body of run(), where every open, open_model_dataset, write_netcdf and output filename now goes through a step helper that resolves against the step's work directory. The pseudocode is written that way rather than cleaned up later on purpose: this is the habit that is nearly free while the code is being written and expensive to retrofit across dozens of steps, which is the argument the groundrules document makes for adopting the rules now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Under the groundrules' declared-resources requirement, a step has to say what it needs so a scheduler can decide how many may run at once. Cores are cpus_per_task with ntasks = 1, since nothing here is MPI, and the number is whatever internal parallelism the step actually starts. Memory is the quantity that distinguishes analysis from most existing Polaris steps: at high resolution a step can need a large fraction of a node, and a scheduler packing by cores alone will oversubscribe it. What sets each step's footprint is written down -- a shard holds one month of 3D input per pool worker, which makes reading a month at a time a resource decision rather than only a convenience; a map step holds one season of its field group's climatology plus the mosaic descriptor -- but the numbers are deliberately not guessed, since they need measurement at production resolution and that is being gathered separately. A guessed default that is too large wastes a node, and one that is too small fails late. Also records the temporary-file rule, which each step satisfies by writing into the work directory it already has. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The climatology step already requested cpus_per_task = 12 to match ncclimo's twelve background processes, which makes it the worked example of the groundrules' bounded-process-launching rule: the step declares what it will start, and -j comes from cpus_per_task rather than from the size of the machine, so a second step beside it does not find the node oversubscribed. Two things it was not doing. Its scratch space is now pointed at the step's own work directory rather than left to TMPDIR, since two climatologies can now run at once -- the model one and the mixed-layer depth one -- and would otherwise be free to collide on a temporary path. And it is launched through Polaris's subprocess helper with the step's logger, so its output lands in the step's log instead of interleaving on shared streams. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The map steps' plot pool is sized from cpus_per_task rather than from the machine, per bounded process launching. Two dependencies on the framework are recorded rather than worked around. polaris.viz assigns plt.rcParams['savefig.dpi'], which the groundrules name explicitly as process-global state; it is being made scope-safe in separate work, and these steps use whatever scoped form it takes. And plotting in a pool means not relying on pyplot's current-figure state, so each plot creates and closes its own figure explicitly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The collector reads every fragment, symlinks each product into a flat plots directory, and writes the merged manifest that defines the published set. It works from the fragments rather than from directory structure, so the work can be re-chunked later without disturbing output paths or links. Products are published by symlink from the step that owns them, so each file has exactly one owner and Polaris's output checking still applies. The published name is the facets in a fixed order, ending in the range of years, so it sorts and greps usefully and two ranges of the same product cannot collide. A product whose fragment is present but whose file is not on disk is reported and kept out of the merged manifest, rather than quietly vanishing from a gallery. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers the symlink rather than a copy, the published name, two ranges coexisting, order surviving the merge, and re-publishing over the links an earlier run left. Also covers the two cases the design calls out: a fragment naming a file that is not on disk is reported and excluded, and a product with no netCDF beside it still publishes its plot. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Thumbnails are what let a reader decide which full images to open, and frequently answer the question without any full image being opened. The collector renders them, so the plotting steps stay unaware of how results are presented and the policy lives in one place. They are bounded in both dimensions rather than in width alone. A width rule charges the most for the tallest plots: a stack of global_stats time series is three times the pixels of a map at the same width, and does not sit in a grid beside it. Measured on the QU240 mock-ups, 320 by 240 gives about 12 kB for a map and 7 kB for a time series, against plots of 1.3 to 2.1 MB. The image is flattened onto white first, because the plots are written RGBA and JPEG has nowhere to put the alpha -- without it the background comes out black. A thumbnail already newer than its plot is left alone, so adding one product to an existing analysis costs one thumbnail rather than all of them. pillow becomes a direct dependency; it arrived with matplotlib before, but this uses it directly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Covers the bounding box holding in both dimensions for a portrait plot as well as a landscape one, the flatten onto white, that an up-to-date thumbnail is not rendered again, and that webp is no larger than jpeg. The size test needed a better stand-in plot. A flat colour compresses to almost nothing as PNG, so no thumbnail could beat it by the order of magnitude the design claims, and the test failed on the fixture rather than on the code. The stand-in now carries smooth variation plus fine detail, like the contour maps this suite publishes, which puts the ratio at about 70x -- the same ballpark as the real plots. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`find_fragments()` walked the work directory for anything named `manifest.json`. That has to go: Polaris steps declare what they read and write at setup so that the framework can check it and a scheduler can order it, and a step that finds its own inputs by looking at the filesystem is invisible to both. Nothing was lost by removing it. A fragment's path is known at setup -- the filename is a constant and each step's work directory is fixed when the suite is built -- so the caller passes the list, which `publish()` already took. Publishing now tolerates a fragment that is not on disk, reporting it alongside the files a fragment named but that are missing. Fragments are optional: a step that makes no products writes none, and that is not an error. A step that never ran is a different failure, and is named by the `publish` step's dependency on it rather than by a missing file here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The third piece of the publisher: a landing page of gallery groups and one page of thumbnails per gallery, rendered from the merged manifest alone. Because the manifest is its only input, a facet added later, a richer presentation, or a different visual design costs nothing anywhere else -- no step, no fragment already written, and no published path changes. What the pages cost is the point of the design, so three things are built in rather than added later. The CSS is inlined, so a page is one request and its images. There is no JavaScript, so the site works the same from a local filesystem, from `python -m http.server`, and from a web portal, and there is nothing to break when a browser changes. Every image carries `loading="lazy"` so that thumbnails below the fold are not fetched at all, and its own `width` and `height` so that lazy loading does not make the page reflow as they arrive. The sizes come from measuring each thumbnail as it is published, which is why the merged manifest gained them. Two small pieces move to where both the collector and the generator can reach them: `range_key()`, which decides how a date range is written in a file name and in a heading, and `provenance.get_summary()`, which is the short provenance a page can carry rather than the exhaustive file `provenance.write()` writes. `jinja2` was already in the environment, used by `polaris/streams.py`, but it is a direct dependency now and is declared as one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tests worth having here are the ones that fail when a reader is inconvenienced: a link with the wrong prefix, an image that is not lazy or whose size is missing so the page reflows under it, a page that asks for a stylesheet or a script, a caption that interpolates rather than escapes. Every link and image source on every page is resolved against the page's own directory and checked to exist, which is the cheapest way to catch a prefix that is right on one page and wrong on the other. Also pinned: that the season order the step plotted in survives to the page, that the landing page shows each gallery's first product, that two ranges of one gallery are two groups and two pages, and that publishing nothing generates an empty gallery rather than failing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The suite gains one cheap step, last in `omega_analysis.txt`, that reads each step's manifest fragment, symlinks the products into the staging tree, renders the thumbnails, writes the merged manifest, and generates the gallery. It is the only step that knows how results are presented. It never looks for its inputs. Every step that makes products is declared as a dependency, so Polaris checks that each of them ran before this step is allowed to and names the ones that did not. A step that only computes intermediate results -- the climatology -- says so with `makes_products` and is left out, since it publishes nothing and would only be reported as having written no fragment. A fragment is optional. A step that ran and made no products writes none, which is reported rather than treated as a failure: it is what a developer iterating on a single plot has, and what the QU240 mock-up has today, where the simulation wrote no MOC output. That is now a message rather than the warning a file named by a manifest but absent from disk still gets. The wiring cannot rely on the order tasks are configured in. A dependency is a step object, Polaris checks that the object it was handed is one that was set up, and every analysis task discards its steps and builds new ones each time it is configured -- while the tasks sharing a config are held in a set and so are configured in an arbitrary order. Each task therefore has the publish task rebuild its step after rebuilding its own, and whichever task is configured last leaves the dependencies pointing at the steps that are really set up. Wiring it once at construction instead fails at setup with a stale range, which is how this was found. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tests that matter here are about the wiring, because that is what broke: that the dependencies are the step objects that are set up rather than ones that merely name the same directories, whichever order the tasks are configured in, and that a dependency is asked for one pickle however often the tasks rebuild. The step that makes no products is checked to be absent from the dependencies, since depending on it would report a fragment it was never going to write. The step itself is run against fragments written into the work directories of the steps it depends on, with all but one of them silent. That covers the config options by their names -- a typo in one is otherwise invisible until a suite is set up -- and that the thumbnails and the gallery are made from what was published. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The User's Guide page already said where the results go; it now says what is there and how to read it: the staging tree with the plots and their netCDF files published as symlinks, the gallery over them, the three thumbnail options and which one to reduce when a page is slow over a throttled link, and what a step that made no products does. The two cases that look alike are told apart there --- a step that ran and published nothing is a message, a step that has not run stops the publish step and is named. The Developer's Guide covers the three pieces of `polaris.analysis`, how a plotting step describes a product as it makes it, and the two things about the publish step that are easy to get wrong: that it declares its inputs rather than looking for them, and that its dependencies are step objects whose identity survives no task being reconfigured, which is why each task has it rebuilt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fragments are mandatory, per `5130c6b128` on `add-analysis-designs`, and that is what lets them be declared. The optional fragment was chosen here to let a developer publish after running only some products, and it does not do that: `add_dependency()` already makes each dependency's pickle an input, so a step that did not run stops the publish step before a missing fragment is ever reached. The only case optionality covered is a step that ran and made nothing, which an empty product list covers. So the step declares one input per step that makes products, linked into `fragments/` from the step that wrote it, and reads those declared inputs rather than composing paths from the base work directory. Polaris now checks every fragment before the step runs and names the ones that are not there. The dependency stays: it is what reaches anything knowable only after a step has run, and its error names a step that never ran. Reading the fragments through the links exposed a real bug. `publish()` resolved a product's files against the directory the fragment sits in, which is now the link's directory rather than the step's, so nothing would have been found. It resolves the link first. `publish()` keeps reporting a fragment that was never written, but that is now a library-level nicety for a caller whose list nothing has checked; the step does not rely on it, and both docstrings say so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The empty fragment first appears here, since the steps that will write one are on the product branches: every step the publish step depends on writes a fragment, and all but one of them write an empty product list. A suite that made nothing publishes an empty gallery rather than failing. The step is run the way `polaris setup` would have left it, with the fragments linked into `fragments/` in its work directory, so the test exercises the declared inputs rather than paths the test composed. That is what caught the collector resolving products against the link's directory. The declaration itself is checked too: one input per step that makes products, each pointing at that step's `manifest.json`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pages described the fragment as optional, which it no longer is. The User's Guide now says that a step with nothing to publish writes an empty manifest rather than none, and the Developer's Guide says why that is what lets the publish step declare each fragment as an input --- and that the two failures stay apart: a missing fragment is a step that ran without writing one, a missing pickle is a step that did not run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fragments are mandatory, but nothing wrote one: every step of the suite made its products without describing them, so the `publish` step stopped on nine missing inputs the first time the suite was run against the mock-up. The half that was missing is here rather than on a product branch, because this is the lowest branch where both `AnalysisStep` and the manifest writer exist. `AnalysisStep` writes it. `runtime_setup()` leaves an empty fragment before `run()` is called, and `add_product()` rewrites it as each product is described, so a step that makes products always has a current fragment on disk and a step author is asked to remember nothing. A rule that had to be obeyed at the end of `run()` would be forgotten in exactly the step that made nothing, which is the case that stops `publish`. Rewriting the fragment from `runtime_setup()` also clears out what a previous run described, so it never outlives the products it names. Which steps write one is `makes_products`, the attribute that already decides which steps `publish` depends on. The two questions are asked separately even though the answer is the same today: nothing here knows about the gallery, so a step that writes a fragment without being published is a change to this test alone. `add_product()` fills in the step's range of years unless the caller passes its own. Every product of an analysis step covers the step's range, and that range is what keeps the published names of two analyses of one simulation apart, so it is not something to be passed by hand at each call. The fragment is also a declared output, which names the step itself if it ever fails to leave one, rather than naming `publish` for a missing input. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
What is worth pinning is the guarantee rather than the call: a step that makes products has an empty fragment on disk before it runs, and a current one after each product it describes, so a step that makes nothing is not a step that wrote nothing. A rerun starts from an empty fragment, so what an earlier run described cannot outlive it, and the climatology, which makes what the maps are plotted from, writes none and refuses to describe a product. The range of years is checked where it matters: it comes from the step unless the caller passes its own. The publish step's test now runs the steps it depends on rather than writing their fragments by hand, so the two halves are exercised together --- a step writes a fragment through `AnalysisStep`, and the publish step reads it through the input that was declared for it. The suite that made nothing is what a real run of the scaffolding reaches today, and it says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Developer's Guide showed a step building a `Manifest` and writing it at the end of `run()`, which is the rule that would be forgotten in the step that made nothing. It now shows the call a step really makes, `add_product()`, and says what the step class does around it: an empty fragment before `run()`, a rewrite as each product is described, and the step's range of years filled in unless the call passes its own. The User's Guide says that the manifest a step with nothing to publish leaves is left for it rather than written by it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The report said how many products were published and nothing about where they came from, so a run that published nothing said nothing about the nine steps it read. It now counts the manifests as well, and names the steps whose manifest was empty. That is where a step that ran and made nothing is named at all: it is an absence in the gallery, and the log is the only place an absence can be seen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The report was the one part of the collector nothing exercised. What is worth pinning is what a reader of the log can count on: the number of manifests read alongside the number of products published, and the name of each step whose manifest was empty. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`pillow` and `jinja2` both become direct dependencies in this branch, so an existing environment is out of date and the load scripts compare this version to notice. This replaces an identical bump to alpha.5, which the rebase onto tranche 0.6.1 dropped because main had taken alpha.5 for itself. The number is the only thing that changed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xylar
added
framework
This was referenced Aug 30, 2026
Collaborator
Author
Status update, 2026-08-30The whole analysis chain now runs end to end and publishes a browsable gallery. Three things changed today that affect every PR in the series. The branches were relinearized. They were a tree with independent arms; they are now a line in dependency order, and each PR should be reviewed against the one below it:
GitHub cannot base a cross-fork PR on another fork branch, so all seven are based on Two things moved as part of this. The vertical reduction was item 11 and is now item 2, at the bottom: it is a dependency-light leaf module that everything vertically reduced needs, and landing it first means no branch above it ships a limitation that a later branch retracts — the climatology maps now support every elevation from the start, and heat content produces every elevation range rather than the whole column only. Publication moved ahead of the products for the same reason, since every step that makes products must leave a manifest fragment for the Every product is now described in a manifest, so the gallery is populated rather than empty. Products are grouped by kind — All seven branches were force-pushed after the relinearization, so local copies of any of them are stale. What it producesA gallery generated by the whole chain against the QU240 one-year mock-up, which is the quickest way to see what the suite does: Three steps publish nothing, and all three are waiting on Omega rather than on Polaris: |
Collaborator
Author
TestingOn
The counts grow down the chain, since each branch's diff contains the branches below it: 460 at the design branch through 715 at the top. End to endThe whole chain was run as the The resulting gallery is published here: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/xasaydavis/omega_analysis_qu240_mockup_20260830/ Two checks on the numbers rather than on the plumbing, both from the mock-up: each numeric elevation agrees with its nearest layer index to the offset between them --- temperature at -100 m is 0.17 K colder in the mean than Inheritance was exercised separately, since one simulated year cannot: a two-year simulation was faked with symlinks, the 1--2 range inherited year 1 from the completed 1--1 step bit-identically and computed only year 2, and re-running with a different specific heat capacity correctly rejected both the sibling seed and the step's own stale cache and recomputed all 24 months. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publication: manifests, the staging tree and the generated gallery
This is item 4 of the order of work: the layer that turns a work directory full of plots into something a reviewer can browse. It is three pieces, none of which knows anything about the ocean, so all three live in a component-neutral
polaris/analysis/package besidepolaris/viz: a manifest writer a step calls once per product, a collector that publishes each product into a staging tree and renders its thumbnail, and a site generator that renders the gallery from the merged manifest.It sits below every branch that makes products, because the manifest writer is a dependency of all of them and because every step that makes products must leave a fragment for the
publishstep to declare as an input.How a product reaches the gallery
Each step writes
manifest.jsonbeside its outputs, describing every product it made and the facets that identify it.AnalysisStepwrites it, not the step author:runtime_setup()writes the empty fragment before the step runs, and describing a product rewrites it, so there is no call at the end ofrun()to forget — and the step that would forget is the one that made nothing, which is exactly the case the empty fragment exists for.The
publishstep then reads those fragments, symlinks each product into the staging tree, renders a thumbnail, writes a merged manifest and generates the site. It never discovers its inputs: it declares one input per fragment, so Polaris checks them before the step runs and names the ones that are missing, and it declares each producing step as a dependency so a step that never ran is reported by name rather than producing an empty gallery.The gallery
A landing page of gallery groups, one page per gallery, rendered from
jinja2templates with the CSS inlined. No JavaScript, so it works identically from a local filesystem,python -m http.server, and the LCRC portal. Every image carriesloading="lazy"and explicitwidthandheight, so a page below the fold costs nothing until it is scrolled to and does not reflow as thumbnails arrive. Every page carries the simulation name, the date ranges and the Polaris provenance.pillowandjinja2become direct dependencies, so this branch ends with the alpha version bump.This is one of seven PRs, reviewed in order
The work is a chain in dependency order. GitHub cannot base a cross-fork PR on another fork branch, so every one of these is based on
mainand its diff contains everything below it. Reviewing them in order is what keeps each diff small — review only the commits this branch adds.add-analysis-designsadd-omega-analysis-elevationadd-omega-analysis-scaffoldingadd-omega-analysis-publisheradd-omega-analysis-global-statsadd-omega-analysis-climatology-mapsadd-omega-analysis-heat-contentA gallery generated by the whole chain against the QU240 mock-up is published here, which is the quickest way to see what the suite produces: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/xasaydavis/omega_analysis_qu240_mockup_20260830/
Checklist
api.md) has any new or modified class, method and/or functions listed