Skip to content

Stop derived fields from inheriting their parent's metadata - #694

Open
xylar wants to merge 17 commits into
E3SM-Project:mainfrom
xylar:fix-derived-field-attrs
Open

Stop derived fields from inheriting their parent's metadata#694
xylar wants to merge 17 commits into
E3SM-Project:mainfrom
xylar:fix-derived-field-attrs

Conversation

@xylar

@xylar xylar commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Problem

In every p-star product, minLevelCell, maxLevelCell and cellMask were labelled long_name = seafloor pressure, units = Pa and, simultaneously, unit = meters, while BottomPressure — the one field that really is a seafloor pressure — had lost every attribute it was given, along with RefPseudoThickness and vertCoordMovementWeights. Values are correct throughout; only metadata is wrong. Nothing in Polaris, MPAS-Ocean or Omega reads these attributes, so no results change — what changes is what a file staged to E3SM's inputdata server tells the next reader about itself.

The cause is that xarray propagates attrs through arithmetic, comparisons, where(), zeros_like() and NumPy ufuncs, so a level index derived from a pressure arrives claiming to be that pressure. Setting individual keys (da.attrs['units'] = ...) leaves everything else the parent carried in place. Note that xr.set_options(keep_attrs=False) is not a fix: it suppresses propagation through arithmetic and comparisons, but zeros_like() and where() keep attributes regardless, and both are on the failing path.

The damage was partial_cell_type-dependent and worst where nobody was looking: with partial_cell_type = none (which horiz_press_grad uses) every p-star field was mislabelled, not just the three index and mask fields.

What changed

  • New polaris/attrs.py with set_attrs(), which replaces a variable's attributes wholesale rather than merging, plus a developer-guide page on why that is necessary.
  • pstar.py now labels every field it creates, and preserves BottomPressure's caller-supplied attributes across the post-snap overwrite so horiz_press_grad keeps saying "seafloor gauge pressure".
  • pstar_init.py, the init-state helpers, ztilde.py, eos and the overflow p-star step all set attributes wholesale; a dead vertCoordMovementWeights branch in pstar_init.py is removed.
  • The whole topography chain is labelled: the combined fields (where base_elevation came out of the latitude-weighted blend calling itself Latitude of Grid Cell Centers, and ocean_mask claimed standard_name = latitude), and the masked and remapped fields downstream (where land_frac and ocean_frac — fractions in [0, 1] — claimed to be a bedrock elevation in metres). Dangling cell_measures, coordinates and grid_mapping attributes, all naming variables the remap step drops, are removed. ice_thickness keeps its CF standard_name; the other fields get none, since no verified CF name applies.
  • Tracer conversion no longer carries the source convention's metadata onto the converted tracer, where for example a standard_name of sea_water_conservative_temperature would survive conversion to potential temperature.
  • Two fixes found while verifying the suite output: RefPseudoThickness is no longer rebuilt into init.nc after write_initial_state_dataset drops it (that duplicate conversion also used the wrong surface pressure), and Density is consistently in-situ density, distinguishing it from a potential density.
  • One unrelated fix found along the way and bundled to save a second review: polaris cache computed its output json filename outside the per-component loop, so caching two components at once wrote both into one file named after whichever component came last, and read the wrong one back in.

Testing

pytest tests/ passes at 322 tests, up 30. The new tests seed their inputs with attributes that must not survive — a BottomPressure labelled as a pressure in Pascals, a geom_z_bot carrying the topography's unit and cell_measures — and assert nothing downstream keeps them; they fail on the pre-fix code and would fail again if a future xarray changed propagation in either direction. test_pstar.py is parametrized over all three partial_cell_type settings, which previously disagreed with one another.

Cached topography

The combine step is default_cached, so its labelling reaches no one until the combined topography is regenerated; the remap and cull steps are not cached and need nothing. All seven resolutions were rerun and re-uploaded, and cached_files.json points at them. Data is unchanged to remapping round-off (at most 6e-16 relative). Regenerating ne3000 also restored ocean_mask, which the previous cached file was missing entirely. One late change — restoring ice_thickness's CF standard_name — landed after that regeneration and so is not in the cached files; it will appear the next time the topography is rebuilt.

Checklist

  • User's Guide has been updated
  • Developer's Guide has been updated
  • API documentation in the Developer's Guide (api.md) has any new or modified class, method and/or functions listed
  • Documentation has been built locally and changes look as expected
  • Testing comment in the PR documents testing used to verify the changes
  • New tests have been added to a test suite

@xylar
xylar requested a review from katsmith133 August 7, 2026 12:17
@xylar xylar self-assigned this Aug 7, 2026
@xylar xylar added clean-up framework Changes relating to the polaris framework as opposed to individual tests or analysis ocean Related to the ocean component labels Aug 7, 2026
@xylar
xylar force-pushed the fix-derived-field-attrs branch 2 times, most recently from 8237a82 to 5c961d0 Compare August 8, 2026 09:53
@xylar
xylar marked this pull request as ready for review August 8, 2026 10:04
@xylar

xylar commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator Author

Testing

Polaris omega_pr suite

  • Baseline workdir: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/test_20260808/omega-pr-ers-fix
  • Baseline build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/test_20260808/omega-pr-ers-fix/build
  • PR build: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/test_20260808/omega-pr-ers-fix/build
  • PR workdir: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/test_20260808/omega-pr-fix-attrs
  • Machine: chrysalis
  • Partition: debug
  • Compiler: intel
  • Build type: Release
  • Log: /lcrc/group/e3sm/ac.xylar/polaris_1.1/chrysalis/test_20260808/omega-pr-fix-attrs/polaris_omega_pr.o1265884
  • Result:
    • Failures (1 of 22):
      • ocean/column/ekman

The primary purpose of this work is to fix attributes with an eye toward component_inputs to E3SM produced by the e3sm/init component. I will also test the full workflow but this branch need not wait for that testing to go in.

xylar and others added 17 commits August 8, 2026 06:41
xarray propagates attrs through arithmetic, comparisons, where() and
zeros_like(), so a variable computed from another arrives carrying its
parent's metadata.  Setting individual keys leaves the rest of the parent's
metadata behind.

set_attrs() replaces a variable's attributes wholesale, which is what it takes
to keep a derived field from inheriting a label that no longer describes it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
minLevelCell, maxLevelCell and cellMask were derived from BottomPressure via
zeros_like(), comparisons, where() and np.maximum, every one of which
propagates attrs, so all three claimed to be a seafloor pressure in Pascals.
RefPseudoThickness and vertCoordMovementWeights had no attributes at all.

Give each field it creates its own long_name and, where meaningful, units.
PseudoThickness, ZTildeMid and ZTildeInterface are labelled here too, so that
init_pstar_vertical_coord() produces a fully described coordinate on its own
rather than relying on PStarInitStep to finish the job.

BottomPressure is the caller's variable, so capture its attributes and put
them back after the post-snap overwrite instead of inventing new ones.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Setting long_name and units individually leaves any other attributes the
variable inherited from its parent in place, which is how BottomPressure ended
up saying both units = Pa and unit = meters.  Use set_attrs() so each written
field carries exactly the metadata it is given.

Drop the re-labelling of PseudoThickness, ZTildeMid and ZTildeInterface, now
done by init_pstar_vertical_coord(), and the vertCoordMovementWeights fallback
that could never run: init_pstar_vertical_coord() always puts that variable in
the dataset first, so the "not in ds" guard was never true.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
layerThickness, restingThickness and Density are all computed from p-star
fields, so they inherited whatever those carried -- including the unit and
cell_measures attributes that ride in from the topography.  Use set_attrs() so
they say only what they mean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
assign_attrs() merges into whatever the arithmetic already propagated, so
pressure_from_z_tilde() returned a pressure still carrying the pseudo-height's
metadata alongside its own.  compute_density() and compute_specvol() had the
same problem from setting units and long_name individually.

_as_converted_data_array() in teos10 is left alone: its merge is deliberate
and documented, not accidental inheritance.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
geom_z_bot, ct and sa are all derived arrays, so setting long_name and units
individually left whatever they inherited in place.  Nothing was mislabelled
here, but the pattern is what let the bug spread.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The combined topography labelled base_elevation, ice_draft and ice_thickness
with a non-standard singular unit = meters.  MPAS and CF both use units, and
everything downstream that derives a field from the topography inherits
whatever is here -- which is how a seafloor pressure ended up also claiming to
be meters.

Write units = m at all three sites, the only places in Polaris that wrote the
string "meters".  The loop over combined fields also overwrites the
units = meters that ice_thickness brings in from BedMachine/Bedmap3.

This does not on its own stop derived fields from inheriting the attribute --
land_frac and ocean_frac will now claim units = m where they used to claim
unit = meters.  Giving the remapped masks and fractions their own attributes
is a separate change.

The combine step is cached, so this reaches users only once the combined
topography files are regenerated and cached_files.json is updated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seed the inputs with attributes that must not survive -- a BottomPressure
labelled as a seafloor pressure in Pascals, a geom_z_bot carrying the
topography's unit and cell_measures -- and check that nothing downstream keeps
them.  Seeding is the point: the tests fail if attribute handling changes in
either direction.

test_pstar.py parametrizes over every partial_cell_type.  The three take
different paths through the pseudo-depth: partial and full rebuilt it from
attribute-free arrays and so lost the labels by accident, while none carried
them onto every field.  All three now agree.

Also assert that every variable the base class writes has a long_name, which
is what RefPseudoThickness and vertCoordMovementWeights lacked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add a framework page covering what xarray propagates, why setting individual
keys is not enough, and how to call set_attrs().  Note in the ocean framework
docs that init_pstar_vertical_coord() labels every field it creates.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_as_converted_data_array() merged the original tracer's attributes into the
converted one, updating only long_name and units.  Those attributes describe
the tracer before conversion: a standard_name of
sea_water_conservative_temperature on a potential temperature is wrong, and
WOA23 fields really do arrive carrying one, along with a cell_measures naming
an area variable that the remapping drops.

Give the converted tracer only the attributes of the convention it was
converted to.  No output changes today -- the tracers that reach this function
already carry just a long_name and units, both of which the merge overwrote.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RefPseudoThickness belongs to the vertical coordinate file.
write_initial_state_dataset() drops it with remove_vert_coord_vars(), and
write_model_dataset() then recreated it from restingThickness on the way out,
so it landed in init.nc anyway.

That copy of the conversion was also the wrong one: it used whatever surface
pressure the dataset happened to carry, where write_vert_coord_dataset()
passes zero, as resting thicknesses require.  And because it went through
pseudothickness_from_pressure(), init.nc labelled the field "pseudo-thickness"
while vert_coord.nc called the same variable a reference pseudo-layer
thickness.

Drop restingThickness from the geometric-to-pseudo conversion in
write_model_dataset(); write_vert_coord_dataset() keeps its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
compute_density() labelled its result "density" and horiz_press_grad set the
same bare label on its Density field, while add_density_from_specvol() said
"in-situ density".  Saying "in-situ" distinguishes it from a potential
density, which Polaris does not currently produce anywhere.

horiz_press_grad was computing 1/SpecVol by hand, which is exactly what
add_density_from_specvol() does, so use the shared helper there.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The combine step is cached, so "Say units = m in the topography combine
workflow" reached no one until the combined topography was regenerated.  All
seven resolutions were rerun on 2026-08-07 and uploaded under the 260807
stamp; the cubed-sphere entries also move under cubed_sphere/<res>/ to match
where the files now live.

Regenerating ne3000 also restores ocean_mask, which the 250523 file was
missing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every field in the combined topography is derived from something else --
base_elevation and ocean_mask by blending with a latitude-derived alpha, the
ice fields by renormalizing and remapping Bedmap3 -- and xarray propagates
attrs through all of it.

base_elevation came out of the blend calling itself "Latitude of Grid Cell
Centers" with latitude's axis, valid_min, valid_max and bounds.  ocean_mask, a
fraction, claimed standard_name = latitude alongside Bedmap3's surface-type
flag_values.  grounded_mask called itself an ice mask.

Give all six fields their own long_name and units, assigned wholesale so the
inherited keys are dropped.  This supersedes setting units = m on the three
height fields: that fixed the unit they claimed without clearing anything else
they had picked up.

Everything downstream of the combined topography inherits from here, so this
is the place to get it right.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
land_frac and ocean_frac are fractions in [0, 1] that claimed to be a bedrock
elevation in metres: the masks they come from are built by comparing
base_elevation against zero, and xarray carries its attributes through the
comparison.  Label the derived masks in MaskTopoStep instead of letting them
inherit.  grounded_frac and ice_frac, which had no long_name or units at all,
pick theirs up from the combined topography now that it is labelled.

Also drop cell_measures, coordinates and grid_mapping from the remapped
fields.  ncremap stamps the first two on everything it writes and this step
drops area, lat and lon; grid_mapping arrives from the projected Antarctic
source and names a variable that does not survive remapping either.  All three
were left pointing at variables that are not in the file.

Copy attributes as a dict rather than by reference, so a masked field and its
source no longer share one attrs dict.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
out_filename was computed once, outside the loop over components, from
whichever component the preceding loop happened to leave bound -- the last
step's.  Caching steps from two components in one invocation wrote both sets
of entries to a single file named after one of them, and the load at the top
of the loop read the wrong component's json into cached_files.  Move the
assignment inside the loop.

Also write the trailing newline json.dump leaves off, so a generated
cached_files.json no longer needs one added by hand.

Found while looking at what re-running polaris cache on the same day does; it
has nothing to do with attributes, but it is a small fix in a file this branch
already touches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Labelling the combined fields wholesale dropped every inherited attribute,
including the one that was right: BedMachine and Bedmap3 both supply
standard_name = land_ice_thickness, which really does describe what the field
holds.  Put it back.

The other five fields get no standard_name.  bedrock_altitude and
sea_area_fraction look like candidates, but neither has been checked against
the CF standard-name table, and an unverified standard name is worse than
none.

This lands after the topography was regenerated, so the cached files do not
carry it; it will appear the next time the topography is rebuilt.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@xylar
xylar force-pushed the fix-derived-field-attrs branch from 5c961d0 to cdc76e8 Compare August 8, 2026 11:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clean-up framework Changes relating to the polaris framework as opposed to individual tests or analysis ocean Related to the ocean component

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant