Stop derived fields from inheriting their parent's metadata - #694
Open
xylar wants to merge 17 commits into
Open
Conversation
xylar
force-pushed
the
fix-derived-field-attrs
branch
2 times, most recently
from
August 8, 2026 09:53
8237a82 to
5c961d0
Compare
xylar
marked this pull request as ready for review
August 8, 2026 10:04
Collaborator
Author
TestingPolaris
|
xarray propagates attrs through arithmetic, comparisons, where() and zeros_like(), so a variable computed from another arrives carrying its parent's metadata. Setting individual keys leaves the rest of the parent's metadata behind. set_attrs() replaces a variable's attributes wholesale, which is what it takes to keep a derived field from inheriting a label that no longer describes it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
minLevelCell, maxLevelCell and cellMask were derived from BottomPressure via zeros_like(), comparisons, where() and np.maximum, every one of which propagates attrs, so all three claimed to be a seafloor pressure in Pascals. RefPseudoThickness and vertCoordMovementWeights had no attributes at all. Give each field it creates its own long_name and, where meaningful, units. PseudoThickness, ZTildeMid and ZTildeInterface are labelled here too, so that init_pstar_vertical_coord() produces a fully described coordinate on its own rather than relying on PStarInitStep to finish the job. BottomPressure is the caller's variable, so capture its attributes and put them back after the post-snap overwrite instead of inventing new ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Setting long_name and units individually leaves any other attributes the variable inherited from its parent in place, which is how BottomPressure ended up saying both units = Pa and unit = meters. Use set_attrs() so each written field carries exactly the metadata it is given. Drop the re-labelling of PseudoThickness, ZTildeMid and ZTildeInterface, now done by init_pstar_vertical_coord(), and the vertCoordMovementWeights fallback that could never run: init_pstar_vertical_coord() always puts that variable in the dataset first, so the "not in ds" guard was never true. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
layerThickness, restingThickness and Density are all computed from p-star fields, so they inherited whatever those carried -- including the unit and cell_measures attributes that ride in from the topography. Use set_attrs() so they say only what they mean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
assign_attrs() merges into whatever the arithmetic already propagated, so pressure_from_z_tilde() returned a pressure still carrying the pseudo-height's metadata alongside its own. compute_density() and compute_specvol() had the same problem from setting units and long_name individually. _as_converted_data_array() in teos10 is left alone: its merge is deliberate and documented, not accidental inheritance. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
geom_z_bot, ct and sa are all derived arrays, so setting long_name and units individually left whatever they inherited in place. Nothing was mislabelled here, but the pattern is what let the bug spread. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The combined topography labelled base_elevation, ice_draft and ice_thickness with a non-standard singular unit = meters. MPAS and CF both use units, and everything downstream that derives a field from the topography inherits whatever is here -- which is how a seafloor pressure ended up also claiming to be meters. Write units = m at all three sites, the only places in Polaris that wrote the string "meters". The loop over combined fields also overwrites the units = meters that ice_thickness brings in from BedMachine/Bedmap3. This does not on its own stop derived fields from inheriting the attribute -- land_frac and ocean_frac will now claim units = m where they used to claim unit = meters. Giving the remapped masks and fractions their own attributes is a separate change. The combine step is cached, so this reaches users only once the combined topography files are regenerated and cached_files.json is updated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Seed the inputs with attributes that must not survive -- a BottomPressure labelled as a seafloor pressure in Pascals, a geom_z_bot carrying the topography's unit and cell_measures -- and check that nothing downstream keeps them. Seeding is the point: the tests fail if attribute handling changes in either direction. test_pstar.py parametrizes over every partial_cell_type. The three take different paths through the pseudo-depth: partial and full rebuilt it from attribute-free arrays and so lost the labels by accident, while none carried them onto every field. All three now agree. Also assert that every variable the base class writes has a long_name, which is what RefPseudoThickness and vertCoordMovementWeights lacked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add a framework page covering what xarray propagates, why setting individual keys is not enough, and how to call set_attrs(). Note in the ocean framework docs that init_pstar_vertical_coord() labels every field it creates. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_as_converted_data_array() merged the original tracer's attributes into the converted one, updating only long_name and units. Those attributes describe the tracer before conversion: a standard_name of sea_water_conservative_temperature on a potential temperature is wrong, and WOA23 fields really do arrive carrying one, along with a cell_measures naming an area variable that the remapping drops. Give the converted tracer only the attributes of the convention it was converted to. No output changes today -- the tracers that reach this function already carry just a long_name and units, both of which the merge overwrote. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RefPseudoThickness belongs to the vertical coordinate file. write_initial_state_dataset() drops it with remove_vert_coord_vars(), and write_model_dataset() then recreated it from restingThickness on the way out, so it landed in init.nc anyway. That copy of the conversion was also the wrong one: it used whatever surface pressure the dataset happened to carry, where write_vert_coord_dataset() passes zero, as resting thicknesses require. And because it went through pseudothickness_from_pressure(), init.nc labelled the field "pseudo-thickness" while vert_coord.nc called the same variable a reference pseudo-layer thickness. Drop restingThickness from the geometric-to-pseudo conversion in write_model_dataset(); write_vert_coord_dataset() keeps its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
compute_density() labelled its result "density" and horiz_press_grad set the same bare label on its Density field, while add_density_from_specvol() said "in-situ density". Saying "in-situ" distinguishes it from a potential density, which Polaris does not currently produce anywhere. horiz_press_grad was computing 1/SpecVol by hand, which is exactly what add_density_from_specvol() does, so use the shared helper there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The combine step is cached, so "Say units = m in the topography combine workflow" reached no one until the combined topography was regenerated. All seven resolutions were rerun on 2026-08-07 and uploaded under the 260807 stamp; the cubed-sphere entries also move under cubed_sphere/<res>/ to match where the files now live. Regenerating ne3000 also restores ocean_mask, which the 250523 file was missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every field in the combined topography is derived from something else -- base_elevation and ocean_mask by blending with a latitude-derived alpha, the ice fields by renormalizing and remapping Bedmap3 -- and xarray propagates attrs through all of it. base_elevation came out of the blend calling itself "Latitude of Grid Cell Centers" with latitude's axis, valid_min, valid_max and bounds. ocean_mask, a fraction, claimed standard_name = latitude alongside Bedmap3's surface-type flag_values. grounded_mask called itself an ice mask. Give all six fields their own long_name and units, assigned wholesale so the inherited keys are dropped. This supersedes setting units = m on the three height fields: that fixed the unit they claimed without clearing anything else they had picked up. Everything downstream of the combined topography inherits from here, so this is the place to get it right. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
land_frac and ocean_frac are fractions in [0, 1] that claimed to be a bedrock elevation in metres: the masks they come from are built by comparing base_elevation against zero, and xarray carries its attributes through the comparison. Label the derived masks in MaskTopoStep instead of letting them inherit. grounded_frac and ice_frac, which had no long_name or units at all, pick theirs up from the combined topography now that it is labelled. Also drop cell_measures, coordinates and grid_mapping from the remapped fields. ncremap stamps the first two on everything it writes and this step drops area, lat and lon; grid_mapping arrives from the projected Antarctic source and names a variable that does not survive remapping either. All three were left pointing at variables that are not in the file. Copy attributes as a dict rather than by reference, so a masked field and its source no longer share one attrs dict. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
out_filename was computed once, outside the loop over components, from whichever component the preceding loop happened to leave bound -- the last step's. Caching steps from two components in one invocation wrote both sets of entries to a single file named after one of them, and the load at the top of the loop read the wrong component's json into cached_files. Move the assignment inside the loop. Also write the trailing newline json.dump leaves off, so a generated cached_files.json no longer needs one added by hand. Found while looking at what re-running polaris cache on the same day does; it has nothing to do with attributes, but it is a small fix in a file this branch already touches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Labelling the combined fields wholesale dropped every inherited attribute, including the one that was right: BedMachine and Bedmap3 both supply standard_name = land_ice_thickness, which really does describe what the field holds. Put it back. The other five fields get no standard_name. bedrock_altitude and sea_area_fraction look like candidates, but neither has been checked against the CF standard-name table, and an unverified standard name is worse than none. This lands after the topography was regenerated, so the cached files do not carry it; it will appear the next time the topography is rebuilt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xylar
force-pushed
the
fix-derived-field-attrs
branch
from
August 8, 2026 11:41
5c961d0 to
cdc76e8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In every p-star product,
minLevelCell,maxLevelCellandcellMaskwere labelledlong_name = seafloor pressure,units = Paand, simultaneously,unit = meters, whileBottomPressure— the one field that really is a seafloor pressure — had lost every attribute it was given, along withRefPseudoThicknessandvertCoordMovementWeights. Values are correct throughout; only metadata is wrong. Nothing in Polaris, MPAS-Ocean or Omega reads these attributes, so no results change — what changes is what a file staged to E3SM's inputdata server tells the next reader about itself.The cause is that xarray propagates
attrsthrough arithmetic, comparisons,where(),zeros_like()and NumPy ufuncs, so a level index derived from a pressure arrives claiming to be that pressure. Setting individual keys (da.attrs['units'] = ...) leaves everything else the parent carried in place. Note thatxr.set_options(keep_attrs=False)is not a fix: it suppresses propagation through arithmetic and comparisons, butzeros_like()andwhere()keep attributes regardless, and both are on the failing path.The damage was
partial_cell_type-dependent and worst where nobody was looking: withpartial_cell_type = none(whichhoriz_press_graduses) every p-star field was mislabelled, not just the three index and mask fields.What changed
polaris/attrs.pywithset_attrs(), which replaces a variable's attributes wholesale rather than merging, plus a developer-guide page on why that is necessary.pstar.pynow labels every field it creates, and preservesBottomPressure's caller-supplied attributes across the post-snap overwrite sohoriz_press_gradkeeps saying "seafloor gauge pressure".pstar_init.py, the init-state helpers,ztilde.py,eosand the overflow p-star step all set attributes wholesale; a deadvertCoordMovementWeightsbranch inpstar_init.pyis removed.base_elevationcame out of the latitude-weighted blend calling itselfLatitude of Grid Cell Centers, andocean_maskclaimedstandard_name = latitude), and the masked and remapped fields downstream (whereland_fracandocean_frac— fractions in [0, 1] — claimed to be a bedrock elevation in metres). Danglingcell_measures,coordinatesandgrid_mappingattributes, all naming variables the remap step drops, are removed.ice_thicknesskeeps its CFstandard_name; the other fields get none, since no verified CF name applies.standard_nameofsea_water_conservative_temperaturewould survive conversion to potential temperature.RefPseudoThicknessis no longer rebuilt intoinit.ncafterwrite_initial_state_datasetdrops it (that duplicate conversion also used the wrong surface pressure), andDensityis consistentlyin-situ density, distinguishing it from a potential density.polaris cachecomputed its output json filename outside the per-component loop, so caching two components at once wrote both into one file named after whichever component came last, and read the wrong one back in.Testing
pytest tests/passes at 322 tests, up 30. The new tests seed their inputs with attributes that must not survive — aBottomPressurelabelled as a pressure in Pascals, ageom_z_botcarrying the topography'sunitandcell_measures— and assert nothing downstream keeps them; they fail on the pre-fix code and would fail again if a future xarray changed propagation in either direction.test_pstar.pyis parametrized over all threepartial_cell_typesettings, which previously disagreed with one another.Cached topography
The combine step is
default_cached, so its labelling reaches no one until the combined topography is regenerated; the remap and cull steps are not cached and need nothing. All seven resolutions were rerun and re-uploaded, andcached_files.jsonpoints at them. Data is unchanged to remapping round-off (at most 6e-16 relative). Regeneratingne3000also restoredocean_mask, which the previous cached file was missing entirely. One late change — restoringice_thickness's CFstandard_name— landed after that regeneration and so is not in the cached files; it will appear the next time the topography is rebuilt.Checklist
api.md) has any new or modified class, method and/or functions listedTestingcomment in the PR documents testing used to verify the changes