Skip to content

Doc/xr engine review cont. #951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sandorkertesz merged 1 commit into from
Apr 1, 2026
Merged

Doc/xr engine review cont. #951

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
257 changes: 196 additions & 61 deletions docs/source/how-tos/xr_engine/xarray_engine_chunks.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

325 changes: 221 additions & 104 deletions docs/source/how-tos/xr_engine/xarray_engine_chunks_on_dask_cluster.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

248 changes: 192 additions & 56 deletions docs/source/how-tos/xr_engine/xarray_engine_ensemble.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

67 changes: 25 additions & 42 deletions docs/source/how-tos/xr_engine/xarray_engine_field_dims.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,9 @@
"tags": []
},
"source": [
"When converting a GRIB fieldlist to Xarray with :py:meth:`~earthkit.data.indexing.xarray.XarrayMixIn.to_xarray` the last one or two dimensions representing the field values depend on the grid type and the ``flatten_values`` settings. \n",
"When converting a GRIB fieldlist to Xarray with :py:meth:`~earthkit.data.indexing.xarray.XarrayMixIn.to_xarray` the last one or two dimensions—representing the field values—depend on the grid type and the ``flatten_values`` setting. \n",
"\n",
"When ``flatten_values`` is False (the is the default, depending on the profile) the field shape is used to form the field dimensions, whose names vary with the grid type. Otherwise, a single dimension called \"values\" will represent a field."
"When ``flatten_values`` is ``False`` (the default, unless overriden by the selected ``profile``), the native field shape defines these dimensions, and their names vary with the grid type. Otherwise, a single dimension called \"values\" will represent a field."
]
},
{
Expand Down Expand Up @@ -73,6 +73,13 @@
"tags": []
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
" "
]
},
{
"data": {
"text/plain": [
Expand Down Expand Up @@ -157,33 +164,23 @@
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "cb5b551ef7ea43af902b2284e134f721",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"lambert_conformal.grib: 0%| | 0.00/55.5k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
"name": "stderr",
"output_type": "stream",
"text": [
" "
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Exception: SpecError: [BoundingBoxXY: cannot build, expecting bounding_box_xy: [min_x, min_y, max_x, max_y]] (/opt/actions-runner/work/_work/python-develop-bundle/python-develop-bundle/src/eckit/src/eckit/geo/area/BoundingBoxXY.cc:80 make_from_spec)\n",
"'Grid' object has no attribute 'to_distinct_latlon'\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"ECCODES ERROR : GridSpec: Exception thrown (SpecError: [BoundingBoxXY: cannot build, expecting bounding_box_xy: [min_x, min_y, max_x, max_y]])\n"
]
"text": []
},
{
"data": {
Expand Down Expand Up @@ -277,18 +274,11 @@
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3cb21ea3b4364be191681101db209e6d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"reduced_gg_O32.grib1: 0%| | 0.00/31.9k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
"name": "stderr",
"output_type": "stream",
"text": [
" "
]
},
{
"data": {
Expand Down Expand Up @@ -352,18 +342,11 @@
},
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ea4dc240212d4b75a7ec31492491c970",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"sh_t32.grib: 0%| | 0.00/13.1k [00:00<?, ?B/s]"
]
},
"metadata": {},
"output_type": "display_data"
"name": "stderr",
"output_type": "stream",
"text": [
" "
]
},
{
"data": {
Expand Down
134 changes: 107 additions & 27 deletions docs/source/how-tos/xr_engine/xarray_engine_holes.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

477 changes: 342 additions & 135 deletions docs/source/how-tos/xr_engine/xarray_engine_mono_variable.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

302 changes: 218 additions & 84 deletions docs/source/how-tos/xr_engine/xarray_engine_mono_variable_remapping.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

139 changes: 102 additions & 37 deletions docs/source/how-tos/xr_engine/xarray_engine_split.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

197 changes: 131 additions & 66 deletions docs/source/how-tos/xr_engine/xarray_engine_to_grib.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

530 changes: 400 additions & 130 deletions docs/source/how-tos/xr_engine/xarray_engine_variable_key.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

115 changes: 62 additions & 53 deletions src/earthkit/data/indexing/xarray.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,15 +67,16 @@ def to_xarray(self, engine="earthkit", xarray_open_dataset_kwargs=None, **kwargs
will update the corresponding profile value if it is a dict otherwise it will overwrite it. See:
:ref:`xr_profile` for more information.
* variable_key: str, None
Metadata key to specify the dataset variables. It cannot be
defined as a dimension. Default is "param" (in earthkit-data this is the same as "shortName").
The metadata key which will be used to name the Xarray Dataset variables.
Default is "parameter.variable" (which in the case of GRIB data is the same as
"metadata.shortName" and "metadata.param").
The same key cannot be used to define any dimension.
Only enabled when ``mono_variable`` is False or None.
* drop_variables: str, or iterable of str, None
A variable or list of variables to drop from the dataset. Default is None. Only used when
``variable_key`` is enabled.
* rename_variables: dict, None
Mapping to rename variables. Default is None. Only used when
``variable_key`` is enabled.
Mapping to rename variables. Default is None. Only used when ``variable_key`` is enabled.
* mono_variable: bool, str, None
If True or str, the dataset will contain a single variable called "data" (or the value
of the ``mono_variable`` kwarg when it is a str). If False, the dataset will contain
Expand All @@ -89,37 +90,39 @@ def to_xarray(self, engine="earthkit", xarray_open_dataset_kwargs=None, **kwargs

.. code-block:: python

# use key "expver" as a dimension
extra_dims = "expver"
# use keys "expver" and "steam" as a dimension
extra_dims = ["expver", "stream"]
# define dimensions "expver", mars_stream" and "mars_type" from
# metadata keys "expver", "stream" and "type"
# use GRIB key "expver" as a dimension
extra_dims = "metadata.expver"
# use keys "metadata.expver" and "metadata.steam" as a dimension
extra_dims = ["metadata.expver", "metadata.stream"]
# define dimensions "expver", "mars_stream" and "mars_type" from
# GRIB keys "expver", "stream" and "type"
extra_dims = [
"expver",
{"mars_stream": "stream"},
("mars_type", "type"),
"metadata.expver",
{"mars_stream": "metadata.stream"},
("mars_type", "metadata.type"),
]
extra_dims = [
{
"expver": "expver",
"mars_stream": "stream",
"mars_type": "type",
"expver": "metadata.expver",
"mars_stream": "metadata.stream",
"mars_type": "metadata.type",
}
]

* drop_dims: str, or iterable of str, None
* drop_dims: str, or iterable of str, None
Single or multiple dimensions to be ignored. Default is None.
Default is None.
* ensure_dims: str, or iterable of str, None
Every item may be one of the following:
- **A dimension name**

- **Dimension name**:
A dimension that must always be preserved in the output, even when
``squeeze=True`` and its size is 1, or when it appears in ``dims_as_attrs``.
- **A metadata key**
- **Metadata key**:
A key whose value defines an additional, non-squeezable dimension.
When a metadata key is listed here, it does *not* need to be repeated
in ``extra_dims``.

Default is None.
* fixed_dims: str, or iterable of str, None
Define all the dimensions to be generated. When used no other dimensions will be created.
Expand All @@ -129,19 +132,19 @@ def to_xarray(self, engine="earthkit", xarray_open_dataset_kwargs=None, **kwargs

.. code-block:: python

# use key "step" as a dimension
fixed_dims = "step"
# use keys "step" and "levelist" as a dimension
extra_dims = ["step", "levelist"]
# use key "time.step" as a dimension
fixed_dims = "time.step"
# use keys "time.step" and "vertical.level" as a dimension
extra_dims = ["time.step", "vertical.level"]
# define dimensions "step", level" and "level_type" from
# metadata keys "step", "levelist" and "levtype"
# metadata keys "metadata.step", "metadata.levelist" and "metadata.levtype"
extra_dims = [
"step",
{"level": "levelist"},
("level_type", "levtype"),
"metadata.step",
{"level": "metadata.levelist"},
("level_type", "metadata.levtype"),
]
extra_dims = [
{"step": "step", "level": "levelist", "level_type": "levtype"}
{"step": "metadata.step", "level": "metadata.levelist", "level_type": "metadata.levtype"}
]

* dim_roles: dict, None
Expand All @@ -165,7 +168,7 @@ def to_xarray(self, engine="earthkit", xarray_open_dataset_kwargs=None, **kwargs
- "valid_time": metadata key interpreted as valid time. Used when ``time_dim_mode`` is "valid_time" or
``add_valid_time_coord`` is True.
- "date": metadata key interpreted as base date. Used only when ``time_dim_mode`` is "raw".
- "time": metadata key interpreted as base time. Used only when ``time_dim_mode`` is "raw".
- "time": metadata key interpreted as base time. Used only when ``time_dim_mode`` is "raw".
- "level": metadata key interpreted as level
- "level_type": metadata key interpreted as level type

Expand All @@ -191,7 +194,7 @@ def to_xarray(self, engine="earthkit", xarray_open_dataset_kwargs=None, **kwargs
If True, the dimension names are formed from the role names. Otherwise, the
dimension names are formed from the metadata keys specified in ``dim_roles``.
Its default value (None) expands to True unless the ``profile`` overwrites it.
Only used when no ``fixed_dims`` are specified. *New in version 0.15.0*.
Only used when no ``fixed_dims`` are specified. **New in version 0.15.0**.
* rename_dims: dict, None
Mapping to rename dimensions. Default is None.
* dims_as_attrs: str, or iterable of str, None
Expand All @@ -207,28 +210,29 @@ def to_xarray(self, engine="earthkit", xarray_open_dataset_kwargs=None, **kwargs

- "forecast": adds two dimensions:

- "forecast_reference_time": built from the "date" and "time" roles
- "forecast_reference_time": built from the "forecast_reference_time" role
(see ``dim_roles``) as np.datetime64 values
- "step": built from the "step" role. When ``decode_times=True`` the values are
np.timedelta64
- "valid_time": adds a dimension called "valid_time" as described by the "valid_time"
- "valid_time": adds a dimension called "valid_time" built from the "valid_time"
role (see ``dim_roles``). Will contain np.datetime64 values.
- "raw": the "date", "time" and "step" roles are turned into 3 separate dimensions
* level_dim_mode: str, None
Controls how predefined vertical dimensions are constructed.
The default is ``"level"``.
Valid values are:
- ``"level"``

- ``"level"``:
Creates two separate dimensions, ``"level"`` and ``"level_type"``,
as defined by the corresponding roles in ``dim_roles``.
- ``"level_per_type"``
- ``"level_per_type"``:
Uses a template dimension ``"<level_per_type>"`` that is expanded
into one or more vertical dimensions.
The dimension name is taken from the metadata key with the role
``"level_type"`` (e.g. ``"isobaricInhPa"``), and the coordinate
``"level_type"`` (e.g. ``"pressure"``), and the coordinate
values come from the metadata key with the role ``"level"``
(e.g. ``[500, 700, 850, 1000]``).
- ``"level_and_type"``
- ``"level_and_type"``:
Produces a single combined dimension, ``"level_and_type"``,
in which the level value and the level type are merged.
* squeeze: bool, None
Expand All @@ -240,15 +244,16 @@ def to_xarray(self, engine="earthkit", xarray_open_dataset_kwargs=None, **kwargs
dataset. Only makes effect when ``time_dim_mode`` is not "valid_time". Its default
value (None) expands to False unless the ``profile`` overwrites it.
* decode_times: bool, None
If True, decode date and datetime coordinates into datetime64 values. If False, leave
coordinates representing date-like GRIB keys (e.g. "date", "validityDate") encoded as
native int values. The default value (None) expands to True unless the ``profile``
overwrites it.
If True, decode date and datetime coordinates into ``datetime64`` values.
If False, leave the coordinates in their native type
(e.g. ``int`` if the coordinates come from the GRIB key like "date" or "validityDate").
The default value (None) expands to True unless the ``profile`` overwrites it.
* decode_timedelta: bool, None
If True, decode coordinates representing time-like or duration-like GRIB keys
(e.g. "time", "validityTime", "step") into timedelta64 values. If False, leave time-like
coordinates encoded as native int values, while duration-like coordinates will be encoded
as int with the units attached to the coordinate as the "units" attribute.
If True, decode time-like or duration-like coordinates into ``timedelta64`` values.
If False, leave the coordinates in their native type (e.g. ``int`` if the coordinates come
from the GRIB key like "time", "validityTime", "step"); additionally, the duration-like
coordinates (e.g. derived from the GRIB key like "step", "endStep", etc.)
will have the attribute "units" appropriately set (to "minutes", "hours", etc.).
If None (default), assume the same value of ``decode_times`` unless the ``profile``
overwrites it.
* add_geo_coords: bool, None
Expand Down Expand Up @@ -304,7 +309,7 @@ def to_xarray(self, engine="earthkit", xarray_open_dataset_kwargs=None, **kwargs
* rename_attrs: dict, None
A dictionary of attribute to rename. Default is None.
* fill_metadata: dict, None
Define fill_metadata values to metadata keys. Default is None.
Define fill values to metadata keys. Default is None.
* remapping: dict, None
Define new metadata keys for indexing. Any key provided in ``remapping`` may be referenced
when specifying options such as ``variable_key``, ``extra_dims``, ``ensure_dims``, and others.
Expand Down Expand Up @@ -341,9 +346,9 @@ def to_xarray(self, engine="earthkit", xarray_open_dataset_kwargs=None, **kwargs
expanded to "numpy". **Deprecated since version 0.19.0**. Please use
``array_namespace`` instead. In versions before 0.19.0 an :obj:`ArrayBackend` was also
accepted here, which is no longer the case.
* array_namespace: str, array namespace,None
* array_namespace: str, array namespace, None
The array namespace to use for array operations. The default value (None) is
expanded to "numpy". ** New in version 0.19.0**.
expanded to "numpy". **New in version 0.19.0**.
* direct_backend: bool, None
If True, the backend is used directly bypassing :py:meth:`xarray.open_dataset`
and ignoring all non-backend related kwargs. If False, the data is read via
Expand Down Expand Up @@ -373,14 +378,18 @@ def to_xarray(self, engine="earthkit", xarray_open_dataset_kwargs=None, **kwargs
The default values of ``xarray_open_dataset_kwargs`` or ``**kwargs`` passed
to :py:func:`xarray.open_dataset` are as follows:

- when ``engine="earthkit"``::
- when ``engine="earthkit"``::

{"cache": True, "chunks": None, "engine": "earthkit"}
{"cache": True, "chunks": None, "engine": "earthkit"}

- when ``engine="cfgrib"``::
- when ``engine="cfgrib"``::

{"backend_kwargs": {"errors": "raise", "ignore_keys": [], "squeeze": False,},
"cache": True, "chunks": None, "engine": "cfgrib"}
{
"backend_kwargs": {"errors": "raise", "ignore_keys": [], "squeeze": False},
"cache": True,
"chunks": None,
"engine": "cfgrib"
}

Please note that settings ``errors="raise"`` and ``engine`` are always enforced
and cannot be changed.
Expand All @@ -389,7 +398,7 @@ def to_xarray(self, engine="earthkit", xarray_open_dataset_kwargs=None, **kwargs
Examples
--------
>>> import earthkit.data
>>> fs = earthkit.data.from_source("file", "test6.grib")
>>> fs = earthkit.data.from_source("sample", "pl.grib")
>>> ds = fs.to_xarray(time_dim_mode="forecast")
>>> # also possible to use the xarray_open_dataset_kwargs
>>> ds = fs.to_xarray(
Expand Down
2 changes: 1 addition & 1 deletion src/earthkit/data/utils/html.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ def tab(items, details=None, selected=None):
t = f"""<div class="eh-description">{details}</div>""" if details else ""
t += """
<div>
<section class="eh-section>
<section class="eh-section">
<div class="eh-tabs-container">
<div class="eh-tabs-block">
<div class="eh-tabs">
Expand Down
Loading
Loading