From 96e57c3a8b545673c8f243b3b69d64cbac7914e5 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 3 Jun 2026 18:30:07 +0000 Subject: [PATCH 01/27] Revamp docs: User Guide + Developer Guide structure - Rename 'post-processing toolchain' to 'workflow manager' - Create user_guide/ with tasks/ (parameter tables for all 11 tasks) and components/ (EAM/EAMxx, ELM, MOSART, MPAS-Ocean, MPAS-seaice) - Create dev_guide/tasks/ with coding/dependency descriptions - Expand dev_guide/testing.rst with full release testing directions - Rewrite dev_guide/release.rst with comprehensive RC + production release steps - Move contributing.rst into dev_guide/ - Add dev_guide/archive.rst for obsolete content - Docs build passes with only 1 pre-existing warning --- docs/source/dev_guide/archive.rst | 40 +++ docs/source/{ => dev_guide}/contributing.rst | 0 docs/source/dev_guide/index.rst | 9 +- docs/source/dev_guide/release.rst | 334 +++++++++++++----- docs/source/dev_guide/tasks/bundle.rst | 30 ++ docs/source/dev_guide/tasks/climo.rst | 59 ++++ docs/source/dev_guide/tasks/e3sm_diags.rst | 52 +++ docs/source/dev_guide/tasks/e3sm_to_cmip.rst | 28 ++ .../dev_guide/tasks/global_time_series.rst | 42 +++ docs/source/dev_guide/tasks/ilamb.rst | 33 ++ docs/source/dev_guide/tasks/index.rst | 43 +++ docs/source/dev_guide/tasks/livvkit.rst | 27 ++ docs/source/dev_guide/tasks/mpas_analysis.rst | 46 +++ docs/source/dev_guide/tasks/pcmdi_diags.rst | 28 ++ docs/source/dev_guide/tasks/tc_analysis.rst | 26 ++ docs/source/dev_guide/tasks/ts.rst | 45 +++ docs/source/dev_guide/testing.rst | 317 ++++++++++++++++- docs/source/index.rst | 61 ++-- docs/source/{ => user_guide}/campaigns.rst | 0 .../user_guide/components/atmosphere.rst | 140 ++++++++ docs/source/user_guide/components/index.rst | 20 ++ docs/source/user_guide/components/land.rst | 107 ++++++ docs/source/user_guide/components/ocean.rst | 90 +++++ docs/source/user_guide/components/rivers.rst | 69 ++++ docs/source/user_guide/components/seaice.rst | 54 +++ .../{ => user_guide}/getting_started.rst | 2 +- docs/source/user_guide/index.rst | 19 + docs/source/{ => user_guide}/parameters.rst | 4 +- docs/source/{ => user_guide}/schematics.rst | 4 +- docs/source/user_guide/tasks/bundle.rst | 59 ++++ docs/source/user_guide/tasks/climo.rst | 142 ++++++++ docs/source/user_guide/tasks/e3sm_diags.rst | 191 ++++++++++ docs/source/user_guide/tasks/e3sm_to_cmip.rst | 114 ++++++ .../user_guide/tasks/global_time_series.rst | 102 ++++++ docs/source/user_guide/tasks/ilamb.rst | 71 ++++ docs/source/user_guide/tasks/index.rst | 59 ++++ docs/source/user_guide/tasks/livvkit.rst | 59 ++++ .../source/user_guide/tasks/mpas_analysis.rst | 158 +++++++++ docs/source/user_guide/tasks/pcmdi_diags.rst | 218 ++++++++++++ docs/source/user_guide/tasks/tc_analysis.rst | 74 ++++ docs/source/user_guide/tasks/ts.rst | 158 +++++++++ docs/source/{ => user_guide}/tutorial.rst | 6 +- 42 files changed, 2983 insertions(+), 157 deletions(-) create mode 100644 docs/source/dev_guide/archive.rst rename docs/source/{ => dev_guide}/contributing.rst (100%) create mode 100644 docs/source/dev_guide/tasks/bundle.rst create mode 100644 docs/source/dev_guide/tasks/climo.rst create mode 100644 docs/source/dev_guide/tasks/e3sm_diags.rst create mode 100644 docs/source/dev_guide/tasks/e3sm_to_cmip.rst create mode 100644 docs/source/dev_guide/tasks/global_time_series.rst create mode 100644 docs/source/dev_guide/tasks/ilamb.rst create mode 100644 docs/source/dev_guide/tasks/index.rst create mode 100644 docs/source/dev_guide/tasks/livvkit.rst create mode 100644 docs/source/dev_guide/tasks/mpas_analysis.rst create mode 100644 docs/source/dev_guide/tasks/pcmdi_diags.rst create mode 100644 docs/source/dev_guide/tasks/tc_analysis.rst create mode 100644 docs/source/dev_guide/tasks/ts.rst rename docs/source/{ => user_guide}/campaigns.rst (100%) create mode 100644 docs/source/user_guide/components/atmosphere.rst create mode 100644 docs/source/user_guide/components/index.rst create mode 100644 docs/source/user_guide/components/land.rst create mode 100644 docs/source/user_guide/components/ocean.rst create mode 100644 docs/source/user_guide/components/rivers.rst create mode 100644 docs/source/user_guide/components/seaice.rst rename docs/source/{ => user_guide}/getting_started.rst (99%) create mode 100644 docs/source/user_guide/index.rst rename docs/source/{ => user_guide}/parameters.rst (98%) rename docs/source/{ => user_guide}/schematics.rst (97%) create mode 100644 docs/source/user_guide/tasks/bundle.rst create mode 100644 docs/source/user_guide/tasks/climo.rst create mode 100644 docs/source/user_guide/tasks/e3sm_diags.rst create mode 100644 docs/source/user_guide/tasks/e3sm_to_cmip.rst create mode 100644 docs/source/user_guide/tasks/global_time_series.rst create mode 100644 docs/source/user_guide/tasks/ilamb.rst create mode 100644 docs/source/user_guide/tasks/index.rst create mode 100644 docs/source/user_guide/tasks/livvkit.rst create mode 100644 docs/source/user_guide/tasks/mpas_analysis.rst create mode 100644 docs/source/user_guide/tasks/pcmdi_diags.rst create mode 100644 docs/source/user_guide/tasks/tc_analysis.rst create mode 100644 docs/source/user_guide/tasks/ts.rst rename docs/source/{ => user_guide}/tutorial.rst (96%) diff --git a/docs/source/dev_guide/archive.rst b/docs/source/dev_guide/archive.rst new file mode 100644 index 00000000..22a2486a --- /dev/null +++ b/docs/source/dev_guide/archive.rst @@ -0,0 +1,40 @@ +.. _archive: + +******* +Archive +******* + +This page collects documentation that is obsolete or refers to older +versions of ``zppy`` or the E3SM Unified environment. It is retained for +historical reference. + +Old testing tutorial (E3SM Unified 1.5.0) +========================================== + +The :doc:`tutorial_testing_e3sm_unified` page contains a tutorial written +for testing against E3SM Unified v1.5.0. For current testing directions, +see :ref:`release-testing`. + +Old documentation setup (initial Sphinx configuration) +======================================================= + +The section "Initial setup (obsolete/for reference only)" in +:doc:`contributing` describes the original one-time Sphinx configuration +steps. Those steps do not need to be repeated. + +Deprecated parameters +===================== + +The following ``zppy`` parameters no longer perform any function: + +.. code-block:: text + + e3sm_to_cmip_environment_commands + ts_fmt + scratch + atmosphere_only + plot_names + ncclimo_cmd (still accepted but ignored) + nrows (in global_time_series; specifying has no effect) + ncols (in global_time_series; specifying has no effect) + vars_exclude (deprecated; use vars = "" and filter in post-processing) diff --git a/docs/source/contributing.rst b/docs/source/dev_guide/contributing.rst similarity index 100% rename from docs/source/contributing.rst rename to docs/source/dev_guide/contributing.rst diff --git a/docs/source/dev_guide/index.rst b/docs/source/dev_guide/index.rst index af2b1a08..b05a461c 100644 --- a/docs/source/dev_guide/index.rst +++ b/docs/source/dev_guide/index.rst @@ -2,15 +2,20 @@ Developer Guide ############### +This guide covers everything needed to develop, test, and release ``zppy``. + .. toctree:: :maxdepth: 2 project-standards ci + contributing + tasks/index testing release_testing release - tutorial_testing_e3sm_unified + new_task new_diags_set new_glb_plot - new_task + tutorial_testing_e3sm_unified + archive diff --git a/docs/source/dev_guide/release.rst b/docs/source/dev_guide/release.rst index a6dd9ab5..409c8832 100644 --- a/docs/source/dev_guide/release.rst +++ b/docs/source/dev_guide/release.rst @@ -1,158 +1,298 @@ +.. _release-zppy: + How to Prepare a Release ======================== -In this guide, we'll cover: +This guide covers the full release process for both release candidates (RCs) +and production releases. It includes: + +- `Preparing the Documentation`_ +- `Bumping the Version`_ +- `Releasing on GitHub: production releases`_ +- `Releasing on conda-forge: production releases`_ +- `Creating a New Version of the Documentation`_ -* Preparing the Documentation -* Bumping the Version -* Releasing On GitHub -* Releasing The Software On Anaconda -* Creating a New Version of the Documentation +References: + +- Release notes are tracked in `GitHub Discussions + `_ +- Releases page: https://github.com/E3SM-Project/zppy/releases +- Tags page: https://github.com/E3SM-Project/zppy/tags Preparing the Documentation --------------------------- -This step is only required for production releases, not release candidates. +This step is **only required for production releases**, not release +candidates. -1. Checkout a branch +1. Checkout a branch: - :: + .. code-block:: bash + cd ~/ez/zppy + git status # confirm no uncommitted changes git fetch upstream main - git checkout -b versioned-docs upstream/main + git checkout -b prepare-docs-for- upstream/main -2. Edit ``docs/source/parameters.rst``: in ``https://github.com/E3SM-Project/zppy/blob/main/zppy/templates/default.ini``, replace ``main`` with the hash of the latest commit. -3. Create a pull request to the main repo and merge it. Now, when the next version of the documentation is created (see last part of this page), it will point to the relevant defaults. Mark yourself as the assignee, and mark "Documentation" as the label. +2. Update ``docs/source/user_guide/parameters.rst``: find the link to + ``this release's parameter defaults`` and replace the commit hash in + the URL with the hash of the latest commit on ``main``. You can find + this at https://github.com/E3SM-Project/zppy/commits/main. + + Before: + :: + + See `this release's parameter defaults /zppy/defaults/default.ini>`_ + + After: + :: + + See `this release's parameter defaults /zppy/defaults/default.ini>`_ + +3. Run pre-commit and commit: + + .. code-block:: bash + + conda activate + pre-commit run --all-files # should pass + git add -A + git commit -m "Update parameter documentation pointer for " + git push upstream prepare-docs-for- + +4. Create a PR, add the **Documentation** label, merge, and delete the + branch on GitHub. Bumping the Version ------------------- -1. Checkout a branch with the name of the version. +Determining the new version number +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Review commits since the last release at +https://github.com/E3SM-Project/zppy/commits/main to determine whether the +bump should be major, minor, or patch: + +- **Major** (X.0.0): breaking changes to the user-facing API +- **Minor** (x.Y.0): new features that are backward-compatible +- **Patch** (x.y.Z): bug fixes only + +Also update the E3SM Unified package table on `Confluence +`_: + +.. code-block:: text + + E3SM Unified : zppy v + +.. _release-candidates: + +Release candidates (zppy repo) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +1. Checkout a branch named after the new version: + + .. code-block:: bash + + cd ~/ez/zppy + git status # confirm no uncommitted changes + git fetch upstream main + git checkout -b vrc upstream/main + git log --oneline | head -n 3 + # Verify matches https://github.com/E3SM-Project/zppy/commits/main + +2. Bump the version using ``tbump``: + + .. code-block:: bash + + conda activate # must have tbump installed + tbump rc --no-tag + # "Error: Command `git push upstream main` failed" is expected + # (we are not on main) + +3. Push and create a PR: + + .. code-block:: bash - :: + git push upstream vrc - git fetch upstream main - # Prepend "v" to - # For release candidates, append "rc" to - git checkout -b v upstream/main + Create a PR, add the **Update version** label, merge, and delete the + branch on GitHub. -2. Bump version using tbump. Example, bumping to v1.1.0: +4. Create the tag on ``main``: - :: + .. code-block:: bash - # Exclude "v" and should match the above step. - # --no-tag is required since tagging is handled in "Releasing on GitHub" - $ tbump --no-tag + git checkout main + git fetch upstream + git reset --hard upstream/main + git tag -a vrc -m "vrc" + # Delete the local branch first (avoids push conflicts) + git branch -D vrc + git push upstream vrc - :: Bumping from 1.0.0 to 1.1.0 - => Would patch these files - - pyproject.toml: version = "1.0.0" - + pyproject.toml: version = "1.1.0" - - zppy/__init__.py:1 __version__ = "v1.0.0" - + zppy/__init__.py:1 __version__ = "v1.1.0" - - conda/meta.yaml:2 {% set version = "1.0.0" %} - + conda/meta.yaml:2 {% set version = "1.1.0" %} - - tbump.toml:5 current = "1.0.0" - + tbump.toml:5 current = "1.1.0" - => Would run these git commands - $ git add --update - $ git commit --message Bump to 1.1.0 - $ git push origin v1.1.0 - :: Looking good? (y/N) - > + The tag will appear on https://github.com/E3SM-Project/zppy/tags but + **not** on Releases. This is expected for RCs. -3. If you encounter ``Error: Command `git push upstream main` failed``, as in `issue 470 `_, you can run ``git push upstream `` yourself. +Production releases (zppy repo) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -4. Create a pull request to the main repo and merge it. Mark yourself as the assignee, and mark "Update version" as the label. +1. Complete `Preparing the Documentation`_ first. + +2. Checkout a branch: + + .. code-block:: bash + + cd ~/ez/zppy + git status # confirm no uncommitted changes + git fetch upstream main + git checkout -b v upstream/main + git log --oneline | head -n 3 + +3. Bump the version: + + .. code-block:: bash + + tbump --no-tag + # "Error: Command `git push upstream main` failed" is expected + git push upstream v + +4. Create a PR, compare the diff against the last RC's bump PR to verify + correctness, add the **Update version** label, wait for CI checks to + pass, merge, and delete the branch. .. _github-release: Releasing on GitHub: release candidates ---------------------------------------- +----------------------------------------- -1. Create a tag for the release candidate at https://github.com/E3SM-Project/zppy/tags. Example, bumping to v1.1.0rc1: +See the tagging step above in `Release candidates (zppy repo)`_. RC tags +appear on Tags but not on Releases. - :: +Releasing on GitHub: production releases +------------------------------------------ - $ git checkout main - $ git fetch upstream - $ git reset --hard upstream/main - $ git tag -a v1.1.0rc1 -m "v1.1.0rc1" - # Delete the branch from the tbump step. Otherwise, the push command won't work. - $ git branch -D v1.1.0rc1 - $ git push upstream v1.1.0rc1 +1. Go to https://github.com/E3SM-Project/zppy/releases and click **Draft a + new release**. +2. Set **Tag version** to ``v`` (include the ``v``). Set + **@ Target** to ``main``. +3. Set **Release title** to ``v`` (include the ``v``). +4. Write the release description: -Releasing on GitHub: production releases ----------------------------------------- + - **Summary of changes**: a high-level summary of what changed + - **Full list of changes**: categorized log of commits, organized by + type (new features, bug fixes, documentation, etc.) -1. Draft a new release `here `_. You can save this and come back to it later, if need be. -2. Set `Tag version` to ``v``, **including the "v"**. `@Target` should be ``main``. -3. Set `Release title` to ``v``, **including the "v"**. -4. Use `Describe this release` to summarize the changelog. + You can use https://github.com/E3SM-Project/zppy/commits/main to find + all commits since the last release. - * You can scroll through `zppy commits `_ for a list of changes. - * You can look at the last release to get an idea of how to format the description. +5. Ensure **Set as the latest release** is checked. +6. Click **Publish release**. -5. Click `Publish release`. -6. CI/CD release workflow is automatically triggered. + - The CI/CD release workflow is automatically triggered. It publishes + new docs and, for conda-forge, triggers a bot PR. + - Unlike RCs, the production version now appears on both + `Tags `_ and + `Releases `_. Releasing on conda-forge: release candidates --------------------------------------------- +--------------------------------------------- + +The conda-forge bot does **not** handle RCs automatically. Do this manually: + +1. If you don't have a local clone of the feedstock: + + .. code-block:: bash + + git clone git@github.com:conda-forge/zppy-feedstock.git + git remote add upstream git@github.com:conda-forge/zppy-feedstock.git -1. If you don't have a local version of the conda-forge repo, run: :: +2. If you don't have a fork, create one on + https://github.com/conda-forge/zppy-feedstock, then: - git clone git@github.com:conda-forge/zppy-feedstock.git - git remote add upstream git@github.com:conda-forge/zppy-feedstock.git + .. code-block:: bash -2. If you don't have a fork of the conda-forge repo, on `conda-forge `_, click the "Fork" button in the upper right hand corner. Then, on your fork, click the green "Code" button, and copy the SSH path. Run: :: + git remote add - git remote add +3. Get the sha256 hash of the RC tarball: -3. Get the sha256 of the tag you made in "Releasing on GitHub: release candidates": :: + .. code-block:: bash - curl -sL https://github.com/E3SM-Project/zppy/archive/v1.1.0rc1.tar.gz | openssl sha256 + curl -sL https://github.com/E3SM-Project/zppy/archive/vrc.tar.gz | openssl sha256 -4. Make changes on a local branch. Example, bumping to v1.1.0rc1: :: +4. Make changes on a branch: - $ git fetch upstream dev - $ git checkout -b v1.1.0rc1 upstream/dev # You can name the branch anything you want - # In `recipe/meta.yaml`, update the version and sha256 (and the build number if needed): - {% set version = "1.1.0rc1" %} # Set to your version - sha256: ... # The sha256 from the previous step - number: 0 # build > number should always be 0 - $ git add -A - $ git commit -m "v1.1.0rc1" - $ git push v1.1.0rc1 + .. code-block:: bash -5. Note that the conda-forge bot does not work for release candidates. So, make a PR manually from your fork of the feedstock to the ``dev`` branch of `conda-forge `_. Then, the package build on conda-forge will end up with the ``zppy_dev`` label. You can add the "automerge" label to have the PR automatically merge once CI checks pass. + git fetch upstream dev + git checkout -b vrc upstream/dev -6. After merging, CI runs again (in a slightly different way). Then, check the https://anaconda.org/conda-forge/zppy page to view the newly updated package. Release candidates are assigned the ``zppy_dev`` label. Note that it takes about 15 minutes for the files to propagate across conda-forge's mirroring services, which must happen before you can use the files. + In ``recipe/recipe.yaml`` (or ``recipe/meta.yaml`` if it still exists), + update: + + .. code-block:: yaml + + version: rc + sha256: + number: 0 # build number should always be 0 for a new version + + Also update any changed dependencies (compare + ``https://github.com/E3SM-Project/zppy/compare/v...v`` + for changes in ``dev.yml`` or ``pyproject.toml``). + + .. code-block:: bash + + git add -A + git commit -m "vrc" + git push vrc + +5. Open a PR from your fork to the ``dev`` branch of + https://github.com/conda-forge/zppy-feedstock. RC packages get the + ``zppy_dev`` label. + +6. After merging and CI completes, check + https://anaconda.org/conda-forge/zppy for the new package (allow ~15 + minutes for mirroring). Releasing on conda-forge: production releases --------------------------------------------- -1. Be sure to have already completed :ref:`Releasing On GitHub `. This triggers the CI/CD workflow that handles Anaconda releases. -2. Wait for a bot PR to come up automatically on conda-forge after the GitHub release. This can happen anywhere from 1 hour to 1 day later. -3. Re-render the PR (see `docs `_). -4. Merge the PR on conda-forge. -5. Check the https://anaconda.org/conda-forge/zppy page to view the newly updated package. Production releases are assigned the ``main`` label. -6. Notify the maintainers of the unified E3SM environment about the new release on the `E3SM Confluence site `_. +1. Complete :ref:`Releasing On GitHub ` first. This + triggers the CI/CD workflow that opens the bot PR on conda-forge. - * Be sure to only update the ``zppy`` version number in the correct version(s) of the E3SM Unified environment. - * This is almost certainly one of the E3SM Unified versions listed under “Next versions”. If you are uncertain of which to update, leave a comment on the page asking. +2. Wait for the bot PR at https://github.com/conda-forge/zppy-feedstock/pulls + (can take 1 hour to 1 day). Alternatively, post an issue with + ``@conda-forge-admin, please update version``. + +3. Complete any requirements to merge the PR (re-render if requested, etc.). + +4. Check https://anaconda.org/conda-forge/zppy/files to confirm the new + package has the ``main`` label. + +5. Notify maintainers of the E3SM Unified environment on the `Confluence + packages page + `_ + about the new release. Update the version number for the correct E3SM + Unified version(s) under "Next versions". Creating a New Version of the Documentation ------------------------------------------- -1. Be sure to have already completed :ref:`Releasing On GitHub `. This triggers the CI/CD workflow that handles publishing documentation versions. -2. Wait until the CI/CD build is successful. You can view all workflows at `All Workflows `_. -3. Changes will be available on the `zppy documentation page `_. You can check if it really is the latest version, by going to https://e3sm-project.github.io/zppy/_build/html/main/parameters.html, then checking if the "paramter defaults" link includes the proper hash. It should match the commit before "Update defaults in docs" on https://github.com/E3SM-Project/zppy/commits/main. +1. Complete :ref:`Releasing On GitHub `. This triggers + the CI/CD release workflow, which publishes new versioned docs. + +2. Wait for the CI/CD build to succeed at + https://github.com/E3SM-Project/zppy/actions. + +3. Verify the documentation is available at + https://docs.e3sm.org/zppy/_build/html/main/ + +4. Check that the "parameter defaults" link on the parameters page points + to the correct commit hash (should match the commit before "Update + parameter documentation pointer" on + https://github.com/E3SM-Project/zppy/commits/main). Extra Resources --------------- -Conda-forge: - -* https://conda-forge.org/docs/user/introduction.html#why-conda-forge -* https://conda-forge.org/docs/maintainer/infrastructure.html#admin-web-services -* https://acme-climate.atlassian.net/wiki/spaces/IPD/pages/3616735236/Releasing+E3SM+Software+on+Anaconda+conda-forge+channel +- Conda-forge docs: https://conda-forge.org/docs/user/introduction.html +- Admin web services: https://conda-forge.org/docs/maintainer/infrastructure.html#admin-web-services +- E3SM Anaconda release guide: https://acme-climate.atlassian.net/wiki/spaces/IPD/pages/3616735236/Releasing+E3SM+Software+on+Anaconda+conda-forge+channel diff --git a/docs/source/dev_guide/tasks/bundle.rst b/docs/source/dev_guide/tasks/bundle.rst new file mode 100644 index 00000000..3c37fdd2 --- /dev/null +++ b/docs/source/dev_guide/tasks/bundle.rst @@ -0,0 +1,30 @@ +.. _dev-task-bundle: + +bundle (Developer Reference) +============================== + +Implementation +-------------- + +- **Python module**: ``zppy/bundle.py`` +- **Jinja2 template**: ``zppy/templates/bundle.bash`` + +The ``Bundle`` class in ``bundle.py`` collects tasks that share the same +``bundle`` name. When ``handle_bundles()`` is called, it renders a single +SLURM script that runs all bundled tasks sequentially within one job. + +A bundle is named by its ``[bundle] [[name]]`` subsection, and tasks +opt in by setting ``bundle = name``. Bundle parameters (walltime, nodes, +etc.) are taken from the first task added or from the ``[bundle]`` section. + +Dependencies +------------ + +Bundles inherit the union of all their member tasks' dependencies. The +bundle job will not start until all upstream dependencies are satisfied +for all bundled tasks. + +**Downstream:** + +- Downstream tasks that depend on bundled tasks will list the bundle's + status file as a dependency. diff --git a/docs/source/dev_guide/tasks/climo.rst b/docs/source/dev_guide/tasks/climo.rst new file mode 100644 index 00000000..8989490b --- /dev/null +++ b/docs/source/dev_guide/tasks/climo.rst @@ -0,0 +1,59 @@ +.. _dev-task-climo: + +climo (Developer Reference) +============================ + +Implementation +-------------- + +- **Python module**: ``zppy/climo.py`` +- **Jinja2 template**: ``zppy/templates/climo.bash`` + +The ``climo`` function in ``climo.py`` iterates over all ``[climo]`` +tasks defined in the configuration file and generates a SLURM bash script +from the ``climo.bash`` Jinja2 template for each year-range/subsection +combination. + +Key steps in ``climo.py``: + +1. Load the ``climo.bash`` template via ``initialize_template()``. +2. Call ``get_tasks(config, "climo")`` to retrieve all ``[climo]`` + subsection configurations. +3. For each task: + + a. Call ``set_mapping_file()``, ``set_grid()``, + ``set_component_and_prc_typ()`` to resolve path and component + inference. + b. Construct the output filename prefix. For EAMxx, the subsection-level + ``case`` is used as the stream ID so ``ncclimo`` can locate input + files. + c. Render the template and write the bash script. + d. Submit the script via ``submit_script()``, registering it with + any bundle if requested. + +Dependencies +------------ + +**Upstream (what climo depends on):** + +- None. The ``climo`` task has no zppy dependencies. It reads raw + simulation output directly. + +**Downstream (what depends on climo):** + +- :doc:`e3sm_diags` — requires climatology files for most diagnostic sets +- :doc:`livvkit` — requires grid-native and regridded climatology files + +Template variables +------------------ + +The ``climo.bash`` template receives all configuration parameters as Jinja2 +variables. Key template variables: + +- ``{{ case }}``, ``{{ input }}``, ``{{ output }}`` — from ``[default]`` +- ``{{ mapping_file }}``, ``{{ grid }}`` — resolved by ``set_mapping_file()`` + and ``set_grid()`` +- ``{{ vars }}`` — variables to process (empty = all) +- ``{{ parallel }}`` — ncclimo parallel mode (default: ``mpi``) +- ``{{ nodes }}`` — number of nodes (default: 4) +- ``{{ environment_commands }}`` — environment setup diff --git a/docs/source/dev_guide/tasks/e3sm_diags.rst b/docs/source/dev_guide/tasks/e3sm_diags.rst new file mode 100644 index 00000000..f3920ed9 --- /dev/null +++ b/docs/source/dev_guide/tasks/e3sm_diags.rst @@ -0,0 +1,52 @@ +.. _dev-task-e3sm-diags: + +e3sm_diags (Developer Reference) +================================== + +Implementation +-------------- + +- **Python module**: ``zppy/e3sm_diags.py`` +- **Jinja2 template**: ``zppy/templates/e3sm_diags.bash`` + +``e3sm_diags.py`` is the most complex task module in ``zppy``. It handles +a wide variety of diagnostic sets, each with different parameter +requirements. + +Key functions in ``e3sm_diags.py``: + +- ``check_parameter_defined(c, param)``: raises ``ParameterNotProvidedError`` + if a required parameter is missing. +- ``check_set_specific_parameter(c, sets, param)``: if any requested + sets require this parameter, check it is present. +- ``check_parameters_for_bash(c)``: validates parameters needed for the + bash template. Runs early because it has few conditions. +- ``check_mvm_only_parameters_for_bash(c)``: validates model-vs-model + parameters. +- ``check_and_define_parameters(c)``: resolves all parameters (inference + + validation). Called later in the pipeline. + +Dependencies +------------ + +**Upstream (required — varies by diagnostic set):** + +- :doc:`climo` — required for most sets (``lat_lon``, ``zonal_mean_xy``, + etc.) +- :doc:`ts` — required for time-series sets (``enso_diags``, ``qbo``, + ``area_mean_time_series``, ``tropical_subseasonal``, ``streamflow``) +- Diurnal cycle ``[climo]`` subtask — required for ``diurnal_cycle`` set +- Daily ``[ts]`` subtask — required for ``tropical_subseasonal`` set + +The specific subtasks to depend on are set via: +``climo_subsection``, ``ts_subsection``, ``climo_diurnal_subsection``, +``ts_daily_subsection``. + +**Downstream:** + +- :doc:`global_time_series` — optionally + +Adding a new diagnostic set +----------------------------- + +See :doc:`../new_diags_set` for a step-by-step guide. diff --git a/docs/source/dev_guide/tasks/e3sm_to_cmip.rst b/docs/source/dev_guide/tasks/e3sm_to_cmip.rst new file mode 100644 index 00000000..c225fa88 --- /dev/null +++ b/docs/source/dev_guide/tasks/e3sm_to_cmip.rst @@ -0,0 +1,28 @@ +.. _dev-task-e3sm-to-cmip: + +e3sm_to_cmip (Developer Reference) +==================================== + +Implementation +-------------- + +- **Python module**: ``zppy/e3sm_to_cmip.py`` +- **Jinja2 template**: ``zppy/templates/e3sm_to_cmip.bash`` + +Dependencies +------------ + +**Upstream (required):** + +- :doc:`ts` — depends on one or more ``[ts]`` subtasks to provide + input time-series files + +The ``ts_subsection``, ``ts_atm_subsection``, and ``ts_land_subsection`` +parameters specify which ``[ts]`` subtasks to depend on. With +``infer_section_parameters = True``, these default to the subsection name, +``atm_monthly_180x360_aave``, and ``land_monthly`` respectively. + +**Downstream (what depends on e3sm_to_cmip):** + +- :doc:`ilamb` — requires CMIP-format land and atmosphere data +- :doc:`pcmdi_diags` — requires CMIP-format atmosphere data diff --git a/docs/source/dev_guide/tasks/global_time_series.rst b/docs/source/dev_guide/tasks/global_time_series.rst new file mode 100644 index 00000000..473add1c --- /dev/null +++ b/docs/source/dev_guide/tasks/global_time_series.rst @@ -0,0 +1,42 @@ +.. _dev-task-global-time-series: + +global_time_series (Developer Reference) +========================================== + +Implementation +-------------- + +- **Python module**: ``zppy/global_time_series.py`` +- **Template**: ``zppy/templates/global_time_series.bash`` and + ``zppy/templates/coupled_global.py`` + +Unlike ``e3sm_diags`` and ``mpas_analysis``, the Global Time Series +functionality is **built into** ``zppy`` rather than calling an external +package. The main plotting logic lives in ``coupled_global.py``. + +Adding new plots +~~~~~~~~~~~~~~~~ + +See :doc:`../new_glb_plot` for a step-by-step guide to adding new global +time series plots. + +The plot registry is in ``PLOT_DICT`` in ``coupled_global.py``. Each entry +maps a short name (used in ``plots_original``) to a function named +``plot_``. + +Dependencies +------------ + +**Upstream (required):** + +- :doc:`ts` — depends on ``[ts]`` global-mean subtasks for atmospheric, + land, and (optionally) ocean/sea-ice data + +**Upstream (optional):** + +- :doc:`mpas_analysis` — depends on MPAS-Analysis output for ocean and + sea-ice time series plots. Specified via ``mpas_analysis_subsections``. + +**Downstream:** + +- None. diff --git a/docs/source/dev_guide/tasks/ilamb.rst b/docs/source/dev_guide/tasks/ilamb.rst new file mode 100644 index 00000000..8ddeb90f --- /dev/null +++ b/docs/source/dev_guide/tasks/ilamb.rst @@ -0,0 +1,33 @@ +.. _dev-task-ilamb: + +ilamb (Developer Reference) +============================= + +Implementation +-------------- + +- **Python module**: ``zppy/ilamb.py`` +- **Jinja2 template**: ``zppy/templates/ilamb.bash`` + +Dependencies +------------ + +**Upstream (required):** + +- :doc:`e3sm_to_cmip` — depends on CMIP-format land and (for non-land-only + runs) atmosphere time-series files + + - Land subtask: ``e3sm_to_cmip_land_subsection`` (inferred as + ``land_monthly`` if not set) + - Atmosphere subtask: ``e3sm_to_cmip_atm_subsection`` (inferred as + ``atm_monthly_180x360_aave`` if not set, skipped for land-only runs) + +- :doc:`ts` — also depends on ``ts`` subtasks: + + - Land subtask: ``ts_land_subsection`` (inferred as ``land_monthly``) + - Atmosphere subtask: ``ts_atm_subsection`` (inferred as + ``atm_monthly_180x360_aave``, skipped for land-only runs) + +**Downstream:** + +- None. diff --git a/docs/source/dev_guide/tasks/index.rst b/docs/source/dev_guide/tasks/index.rst new file mode 100644 index 00000000..672ad51e --- /dev/null +++ b/docs/source/dev_guide/tasks/index.rst @@ -0,0 +1,43 @@ +.. _dev-tasks: + +Tasks (Developer Reference) +============================ + +This section describes how each ``zppy`` task is implemented, what +templates and Python modules it uses, and what dependencies it has on +other tasks. + +.. toctree:: + :maxdepth: 1 + + climo + ts + e3sm_to_cmip + e3sm_diags + mpas_analysis + global_time_series + tc_analysis + ilamb + livvkit + pcmdi_diags + bundle + +Dependency overview +------------------- + +The diagram below summarizes the typical dependency relationships between +tasks. Arrows indicate "depends on": + +.. code-block:: text + + climo ──────────────────────────────────────────► e3sm_diags + ts ─────────────────────────────────────────────► e3sm_diags + ts ─────────────────────────────────────────────► e3sm_to_cmip + ts ─────────────────────────────────────────────► global_time_series + ts ─────────────────────────────────────────────► pcmdi_diags (via e3sm_to_cmip) + mpas_analysis ──────────────────────────────────► global_time_series + e3sm_to_cmip ───────────────────────────────────► ilamb + e3sm_to_cmip ───────────────────────────────────► pcmdi_diags + climo ──────────────────────────────────────────► livvkit + +All tasks are independent unless they share the above relationships. diff --git a/docs/source/dev_guide/tasks/livvkit.rst b/docs/source/dev_guide/tasks/livvkit.rst new file mode 100644 index 00000000..d152a095 --- /dev/null +++ b/docs/source/dev_guide/tasks/livvkit.rst @@ -0,0 +1,27 @@ +.. _dev-task-livvkit: + +livvkit (Developer Reference) +============================== + +Implementation +-------------- + +- **Python module**: ``zppy/livvkit.py`` +- **Jinja2 template**: ``zppy/templates/livvkit.bash`` + +Dependencies +------------ + +**Upstream (required):** + +- :doc:`climo` — depends on multiple ``[climo]`` subtasks: + + - A grid-native climatology: assumed ``land_monthly_climo_native`` + - Reanalysis-grid climatologies: one per reanalysis comparison, named + ``land_monthly_climo_`` + + All required subtask names are listed in ``climo_subsections``. + +**Downstream:** + +- None. diff --git a/docs/source/dev_guide/tasks/mpas_analysis.rst b/docs/source/dev_guide/tasks/mpas_analysis.rst new file mode 100644 index 00000000..3c592312 --- /dev/null +++ b/docs/source/dev_guide/tasks/mpas_analysis.rst @@ -0,0 +1,46 @@ +.. _dev-task-mpas-analysis: + +mpas_analysis (Developer Reference) +===================================== + +Implementation +-------------- + +- **Python module**: ``zppy/mpas_analysis.py`` +- **Jinja2 template**: ``zppy/templates/mpas_analysis.bash`` + +The ``mpas_analysis`` module generates one SLURM job per combination of +year range (from ``ts_years``, ``climo_years``, ``enso_years``) and +subsection. + +Model-vs-model logic +~~~~~~~~~~~~~~~~~~~~ + +When ``reference_data_path`` is set, the run is treated as MVM. The module +locates MPAS-Analysis config files from prior runs: + +.. code-block:: text + + /post/analysis/mpas_analysis//cfg/ + mpas_analysis_.cfg + +``comparison_type`` is either ``mvo`` or ``mvm``. In ``auto`` mode: + +- If only one type is found, that type is used. +- If both are found, an error is raised and the user must set + ``reference_comparison_type`` or ``test_comparison_type`` explicitly. +- If ``reference_data_path`` points to a ``[[subsection]]``, zppy uses + that subsection's actual comparison type. + +Dependencies +------------ + +**Upstream:** + +- None required. MPAS-Analysis reads MPAS-Ocean and MPAS-seaice output + directly from the simulation archive. + +**Downstream (optional):** + +- :doc:`global_time_series` — can optionally depend on MPAS-Analysis output + for ocean/sea-ice time series plots diff --git a/docs/source/dev_guide/tasks/pcmdi_diags.rst b/docs/source/dev_guide/tasks/pcmdi_diags.rst new file mode 100644 index 00000000..385d1f8d --- /dev/null +++ b/docs/source/dev_guide/tasks/pcmdi_diags.rst @@ -0,0 +1,28 @@ +.. _dev-task-pcmdi-diags: + +pcmdi_diags (Developer Reference) +=================================== + +Implementation +-------------- + +- **Python module**: ``zppy/pcmdi_diags.py`` +- **Jinja2 template**: ``zppy/templates/pcmdi_diags.bash`` + +``pcmdi_diags.py`` handles multiple diagnostic sets (``mean_climate``, +``variability_modes_cpl``, ``variability_modes_atm``, ``enso``, +``synthetic_plots``). The ``current_set`` parameter (or the subsection +name) determines which set to run. + +Dependencies +------------ + +**Upstream (required):** + +- :doc:`e3sm_to_cmip` — depends on a ``[e3sm_to_cmip]`` atmosphere subtask + to provide CMIP-format input data. Specified via + ``e3sm_to_cmip_atm_subsection``. + +**Downstream:** + +- None. diff --git a/docs/source/dev_guide/tasks/tc_analysis.rst b/docs/source/dev_guide/tasks/tc_analysis.rst new file mode 100644 index 00000000..1440d5c0 --- /dev/null +++ b/docs/source/dev_guide/tasks/tc_analysis.rst @@ -0,0 +1,26 @@ +.. _dev-task-tc-analysis: + +tc_analysis (Developer Reference) +=================================== + +Implementation +-------------- + +- **Python module**: ``zppy/tc_analysis.py`` +- **Jinja2 template**: ``zppy/templates/tc_analysis.bash`` + +``tc_analysis`` processes high-frequency atmospheric output (``eam.h2``) +to detect and track tropical cyclones. + +Dependencies +------------ + +**Upstream:** + +- None. Reads high-frequency EAM output directly. + +**Downstream:** + +- None within ``zppy``. TC analysis results can be used as input to the + ``tc_analysis`` set in ``e3sm_diags`` (model-vs-model), but that + dependency is specified separately in the ``[e3sm_diags]`` configuration. diff --git a/docs/source/dev_guide/tasks/ts.rst b/docs/source/dev_guide/tasks/ts.rst new file mode 100644 index 00000000..f928fe71 --- /dev/null +++ b/docs/source/dev_guide/tasks/ts.rst @@ -0,0 +1,45 @@ +.. _dev-task-ts: + +ts (Developer Reference) +========================= + +Implementation +-------------- + +- **Python module**: ``zppy/ts.py`` +- **Jinja2 template**: ``zppy/templates/ts.bash`` + +The ``ts`` function iterates over all ``[ts]`` tasks and generates a SLURM +bash script from the ``ts.bash`` Jinja2 template for each +year-range/subsection combination. + +Key steps in ``ts.py``: + +1. Load the ``ts.bash`` template. +2. Call ``get_tasks(config, "ts")`` for all subtasks. +3. For each task: + + a. Resolve mapping file, grid, and component type. + b. Render the template and write the bash script. + c. Submit the script. + +Vertical remapping +~~~~~~~~~~~~~~~~~~ + +When ``vrt_remap_vars`` is non-empty, the template also calls ``ncremap`` +after ``ncclimo`` to remap specified variables from model levels to pressure +levels. Output is written to a sibling ``ts_vrt_remap/`` directory. + +Dependencies +------------ + +**Upstream (what ts depends on):** + +- None. The ``ts`` task reads raw simulation output directly. + +**Downstream (what depends on ts):** + +- :doc:`e3sm_diags` — many sets require time-series files +- :doc:`e3sm_to_cmip` — depends on ts for input time-series +- :doc:`global_time_series` — depends on ts global-mean subtasks +- :doc:`pcmdi_diags` — depends on ts via e3sm_to_cmip diff --git a/docs/source/dev_guide/testing.rst b/docs/source/dev_guide/testing.rst index fa9a6c46..b867b103 100644 --- a/docs/source/dev_guide/testing.rst +++ b/docs/source/dev_guide/testing.rst @@ -1,34 +1,319 @@ -************ +.. _testing-zppy: + +************* Testing zppy -************ +************* Unit tests ========== -Run all unit tests by doing the following: +Unit tests live in ``tests/test_*.py``. Run them with: - .. code:: +.. code-block:: bash - pip install . # Install your changes (`python -m pip install .` also works) - pytest tests/test_*.py # Run all unit tests + pip install . # or: python -m pip install . + pytest tests/test_*.py Integration tests ================= -Integration tests can be run on Chrysalis, Compy, or Perlmutter. -First review `machine-specific directions `_. +Integration tests can be run on Chrysalis, Compy, or Perlmutter. They +require an active E3SM Unified environment and access to simulation data. + +Machine-specific directions (generated setup commands, expected output files, +etc.) live in: +`tests/integration/generated/ `_ + +Run all integration tests: + +.. code-block:: bash + + pip install . + pytest tests/integration/test_*.py + +.. note:: + Image comparison tests require a compute node: + + .. code-block:: bash + + salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm + pytest tests/integration/test_images.py + cat test_images_summary.md + +Automated tests (CI) +==================== + +Unit tests run automatically via :ref:`GitHub Actions ` on every +pull request and push to ``main``. Integration tests must be run manually +on a supported machine. + +.. _release-testing: + +Full release testing directions +================================ + +This section describes the complete process for testing ``zppy`` before +making a release. See also :doc:`tutorial_testing_e3sm_unified` for a legacy +tutorial. + +Overview +-------- + +Three testing scenarios exist: + +a. **test dev** — run before making a new zppy RC (no new Unified RC yet) +b. **test new Unified RC** — run when a new E3SM Unified RC has been made +c. **test final Unified** — run after E3SM Unified is finalized + +Tests should be run on **three machines**: Chrysalis, Compy, and Perlmutter. + +Step 1: Set up zppy-interfaces (if needed) +------------------------------------------- + +Update ``zppy-interfaces`` to the latest main branch: + +.. code-block:: bash + + cd ~/ez/zppy-interfaces + git status # should be clean + git fetch upstream main + git checkout main + git reset --hard upstream/main + git log --oneline | head -n 1 + # Verify matches https://github.com/E3SM-Project/zppy-interfaces/commits/main + + bash # isolated subshell + source ~/.bashrc + source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified__chrysalis.sh + + # Verify zppy-interfaces tests + pytest tests/unit/global_time_series/test_*.py + pytest tests/unit/pcmdi_diags/test_*.py + +Step 2: Check out the zppy branch +----------------------------------- + +.. code-block:: bash + + cd ~/ez/zppy + git status # ensure clean workspace + + git fetch upstream main + git checkout -b test-zppy-- upstream/main + git log --oneline | head -n 3 + # Verify matches https://github.com/E3SM-Project/zppy/commits/main + +Step 3: Set up the environment +------------------------------- + +a. **test dev**: create a new dev environment + + .. code-block:: bash + + mamba clean --all + mamba env create -f conda/dev.yml -n zppy_dev_pre_rc + conda activate zppy_dev_pre_rc + pip install . + +b. **test new Unified RC**: activate the Unified RC + + - Chrysalis: ``source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh`` + - Compy: ``source /share/apps/E3SM/conda_envs/test_e3sm_unified_rc_compy.sh`` + - Perlmutter: ``source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_rc_pm-cpu.sh`` + +c. **test final Unified**: activate the latest released Unified + + - Chrysalis: ``source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh`` + - Compy: ``source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh`` + - Perlmutter: ``source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh`` + +Step 4: Configure ``tests/integration/utils.py`` +-------------------------------------------------- + +Edit ``tests/integration/utils.py`` to set the correct environment +and unique ID. Example ``TEST_SPECIFICS`` dictionary: + +.. code-block:: python + + TEST_SPECIFICS: Dict[str, Any] = { + # NCO path: "" uses production-version NCO commands + # Set to a specific path to use development-version NCO commands + "nco_path": "", + # Environment commands for specific tasks. + # Never set these to ""; that would override the task's environment + # with a blank string (no environment). + "e3sm_to_cmip_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", + "diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", + "mpas_analysis_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", + "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", + "livvkit_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", + "pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", + # General environment for other tasks. + # Leave as "" to use the latest Unified environment. + "environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", + # For a complete test, run the set of latest cfgs AND at least one set of legacy cfgs + "cfgs_to_run": [ + "weekly_bundles", + "weekly_comprehensive_v2", + "weekly_comprehensive_v3", + "weekly_legacy_3.1.0_bundles", + "weekly_legacy_3.1.0_comprehensive_v2", + "weekly_legacy_3.1.0_comprehensive_v3", + "weekly_legacy_3.0.0_bundles", + "weekly_legacy_3.0.0_comprehensive_v2", + "weekly_legacy_3.0.0_comprehensive_v3", + ], + "tasks_to_run": [ + "e3sm_diags", + "mpas_analysis", + "global_time_series", + "ilamb", + "livvkit", + "pcmdi_diags", + ], + "unique_id": "zppy__rc_", + } + +Step 5: Run unit tests +----------------------- + +.. code-block:: bash + + pytest tests/test_*.py + +All tests should pass. Failures in scenario (a) require code fixes. +Failures in scenario (c) may indicate a patch release of E3SM Unified is +needed. + +Step 6: Generate cfg files and check quota +------------------------------------------- + +.. code-block:: bash + + python tests/integration/utils.py + # Review output to confirm settings + + # Check quota before launching + mmlsquota -u --block-size T fs2 + lcrc-quota + + ls tests/integration/generated/test_weekly_*_chrysalis.cfg + +Step 7: Launch integration jobs +-------------------------------- + +To ensure no conflicts, wait for all jobs from one ``zppy`` run to finish +before launching another. Start with the most important: + +.. code-block:: bash + + # Launch all cfg files + zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg + + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg + + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg + + # Monitor job queue + sq | wc -l # subtract 1 for the header row + +Step 8: Run bundles a second time +---------------------------------- + +After all initial jobs complete, check for errors and run the bundles a +second time to complete remaining work: + +.. code-block:: bash + + # Check for errors first + cd /path/to/output/.../post/scripts + grep -v "OK" *status # Good if no output + + # Re-run bundles + zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg + +Step 9: Review all finished runs +---------------------------------- + +For each output directory, check for failures: + +.. code-block:: bash + + cd /path/to/zppy_weekly_comprehensive_v2_output/.../post/scripts + grep -v "OK" *status + + cd /path/to/zppy_weekly_comprehensive_v3_output/.../post/scripts + grep -v "OK" *status + + cd /path/to/zppy_weekly_bundles_output/.../post/scripts + grep -v "OK" *status + + # Repeat for all legacy output directories + +Step 10: Run Python integration tests +--------------------------------------- + +.. code-block:: bash + + pytest tests/integration/test_bash_generation.py + pytest tests/integration/test_campaign.py + pytest tests/integration/test_defaults.py + pytest tests/integration/test_bundles.py + pytest tests/integration/test_last_year.py + +For image comparison tests (requires a compute node): + +.. code-block:: bash + + salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm + bash + source ~/.bashrc + source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified__chrysalis.sh + pytest tests/integration/test_images.py + cat test_images_summary.md + +Step 11: Wrap up +----------------- + +After successful testing: + +a. **test dev**: proceed to :ref:`create the zppy RC `. + +b. **test new Unified RC**: proceed to create the zppy production release. + +c. **test final Unified**: run the "Commands to generate official expected + results for a release" from the machine-specific directions. Then clean + up branches and environments: -Run all integration tests by doing the following: + .. code-block:: bash - .. code:: + git branch + git branch -D - pip install . # Install your changes (`python -m pip install .` also works) + conda env list + conda remove -n --all - pytest tests/integration/test_*.py # Run all integration tests + Update expected test result files if there are known/expected differences: + use the "Commands to run to replace outdated expected files" from the + machine-specific directions, then re-run the tests. -Automated tests -=============== +Step 12: Record test results +------------------------------ -We have a :ref:`GitHub Actions ` Continuous Integration / Continuous Delivery (CI/CD) workflow. +Document test results in the `zppy testing discussion +`_ for the relevant +E3SM Unified release. Include: -The unit tests are run automatically as part of this. Integration tests must be run on a machine specified above. +- Machine name and date +- E3SM Unified version tested +- zppy version/branch tested +- Results summary (pass/fail per cfg) +- Any known issues or caveats diff --git a/docs/source/index.rst b/docs/source/index.rst index 980a9921..43727deb 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,7 +1,4 @@ -.. zppy documentation master file, created by - sphinx-quickstart on Fri Feb 19 18:07:14 2021. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. +.. zppy documentation master file .. _index-label: @@ -12,42 +9,36 @@ Zppy documentation What is zppy? ============= -``zppy`` (pronounced "zippee") is a post-processing toolchain for E3SM written in -Python. The goal of ``zppy`` is to speed-up the post-processing of E3SM simulations -by automating commonly performed tasks. +``zppy`` (pronounced "zippee") is a **workflow manager** for post-processing +E3SM simulations, written in Python. The goal of ``zppy`` is to speed up +post-processing by automating commonly performed tasks. ``zppy`` is built around a set of tasks that perform specific actions, such as generating (regridded) **time series** or **climatology** files using NCO's ``ncclimo``, or generating analysis figures with **E3SM Diags** -or **MPAS-Analysis**. ``zppy`` provides a framework to simplify the use of these -tools, especially for long simulations. - -.. - # TODO: redraw figure - The figure below illustrates schematically how zppy functions. - -``zppy`` is controlled entirely from a single user provided configuration file -(e.g., "mysimulation.cfg"). This configuration file specifies input and output -directories, as well as the list of tasks to run. Each task operates -for a specific model component and over -a specified number of years, for example generating atmosphere climatology -files every 20 and 50 years. - -``zppy`` parses the configuration file and generates batch -jobs that are submitted for execution by SLURM. Dependencies between tasks -are handled and passed to SLURM. Internally, each batch job is created by -instantiating `Jinja `_ template scripts -(usually written in bash, but other languages are supported as well). +or **MPAS-Analysis**. ``zppy`` provides a framework to simplify the use of +these tools, especially for long simulations. + +``zppy`` is controlled entirely from a single user-provided configuration file +(e.g., ``mysimulation.cfg``). This configuration file specifies input and +output directories, as well as the list of tasks to run. Each task operates +for a specific model component and over a specified number of years—for +example, generating atmosphere climatology files every 20 and 50 years. + +``zppy`` parses the configuration file and generates batch jobs that are +submitted for execution by SLURM. Dependencies between tasks are handled and +passed to SLURM. Internally, each batch job is created by instantiating +`Jinja `_ template scripts (usually +written in bash, but other languages are supported as well). .. toctree:: :maxdepth: 2 - :caption: Contents: - - self - schematics - getting_started - tutorial - parameters - campaigns + :caption: User Guide + + user_guide/index + +.. toctree:: + :maxdepth: 2 + :caption: Developer Guide + dev_guide/index - contributing diff --git a/docs/source/campaigns.rst b/docs/source/user_guide/campaigns.rst similarity index 100% rename from docs/source/campaigns.rst rename to docs/source/user_guide/campaigns.rst diff --git a/docs/source/user_guide/components/atmosphere.rst b/docs/source/user_guide/components/atmosphere.rst new file mode 100644 index 00000000..4b1c114a --- /dev/null +++ b/docs/source/user_guide/components/atmosphere.rst @@ -0,0 +1,140 @@ +.. _component-atmosphere: + +Atmosphere — EAM and EAMxx +=========================== + +E3SM has two atmosphere models: + +- **EAM** (E3SM Atmosphere Model) — the standard atmosphere model used in + most E3SM configurations. +- **EAMxx** (E3SM Atmosphere Model, next generation) — uses a different + vertical grid (L128) and a different file format. + +Relevant zppy tasks +------------------- + +The following ``zppy`` tasks are commonly used for atmosphere post-processing: + +- :doc:`../tasks/climo` — generate atmosphere climatology files +- :doc:`../tasks/ts` — generate atmosphere time-series files +- :doc:`../tasks/e3sm_to_cmip` — convert to CMIP6 format +- :doc:`../tasks/e3sm_diags` — atmospheric diagnostics (most sets) +- :doc:`../tasks/tc_analysis` — tropical cyclone analysis +- :doc:`../tasks/global_time_series` — global atmospheric time series plots +- :doc:`../tasks/pcmdi_diags` — atmospheric performance metrics + +EAM configuration +----------------- + +For EAM (standard resolution), typical history streams are: + +- ``eam.h0`` — monthly averages (most common for diagnostics) +- ``eam.h1`` — daily averages +- ``eam.h2`` — high-frequency output (used for TC analysis) + +Typical ``[climo]`` and ``[ts]`` subsection configuration for EAM: + +.. code-block:: cfg + + [climo] + active = True + + [[atm_monthly_180x360_aave]] + input_files = eam.h0 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + grid = 180x360_aave + years = 1:100:20 + + [ts] + active = True + + [[atm_monthly_180x360_aave]] + input_files = eam.h0 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + grid = 180x360_aave + years = 1:100:10 + + [[atm_daily_180x360_aave]] + input_files = eam.h1 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc + grid = 180x360_aave + frequency = daily + years = 1:100:10 + + [[atm_monthly_glb]] + input_files = eam.h0 + input_subdir = archive/atm/hist + years = 1:100:10 + +EAMxx configuration +------------------- + +EAMxx uses a 128-level vertical grid (L128) and stores output differently +from EAM. Key differences when configuring ``zppy`` for EAMxx: + +- Set ``input_component = eamxx`` in ``[climo]`` and ``[ts]`` sections. +- The ``vrt_in_file`` parameter may be needed for vertical remapping. + When ``input_component = eamxx``, ``zppy`` defaults to a + ``vert_L128.nc`` file under ``diagnostics_base_path``. +- Vertical remapping (``vrt_remap_vars``) is often needed to produce + pressure-level data for diagnostics. + +Example EAMxx configuration: + +.. code-block:: cfg + + [ts] + active = True + + [[eamxx_monthly_180x360_aave]] + input_component = eamxx + input_files = eamxx.h0 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.nc + grid = 180x360_aave + vrt_remap_vars = T,Q,U,V,Z3,OMEGA + years = 1:50:10 + +Mapping files +------------- + +Mapping files remap EAM/EAMxx output from the native spectral-element +grid (e.g., ``ne30pg2``) to a regular latitude-longitude grid +(e.g., ``180x360_aave``). These files are typically pre-computed and +available on supported machines under the ``diagnostics_base_path``. + +Machine-specific mapping file locations: + +- **Chrysalis/Anvil (LCRC)**: typically under + ``/home/ac.zender/data/maps/`` +- **Perlmutter (NERSC)**: typically under + ``/global/homes/z/zender/data/maps/`` +- **Compy**: typically under ``/compyfs/zender/maps/`` + +Diurnal cycle +------------- + +For diurnal cycle diagnostics in ``e3sm_diags``, high-frequency +(8× daily) climatologies are needed. Configure an additional ``[climo]`` +subsection: + +.. code-block:: cfg + + [climo] + [[atm_monthly_diurnal_8xdaily_180x360_aave]] + input_files = eam.h3 + input_subdir = archive/atm/hist + frequency = diurnal_8xdaily + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.nc + grid = 180x360_aave + years = 1:100:20 + + [e3sm_diags] + [[atm_monthly_180x360_aave]] + sets = diurnal_cycle,... + climo_diurnal_subsection = atm_monthly_diurnal_8xdaily_180x360_aave + climo_diurnal_frequency = diurnal_8xdaily + dc_obs_climo = /path/to/obs/climatology diff --git a/docs/source/user_guide/components/index.rst b/docs/source/user_guide/components/index.rst new file mode 100644 index 00000000..c390a954 --- /dev/null +++ b/docs/source/user_guide/components/index.rst @@ -0,0 +1,20 @@ +.. _user-components: + +Model Components +================ + +``zppy`` post-processes output from all E3SM model components. This section +provides component-specific guidance for developers and users working with +each component's output. + +.. toctree:: + :maxdepth: 1 + + atmosphere + land + rivers + ocean + seaice + +`E3SM component documentation `_ provides +further details on each component. diff --git a/docs/source/user_guide/components/land.rst b/docs/source/user_guide/components/land.rst new file mode 100644 index 00000000..71d091a9 --- /dev/null +++ b/docs/source/user_guide/components/land.rst @@ -0,0 +1,107 @@ +.. _component-land: + +Land — ELM +=========== + +**ELM** (E3SM Land Model) is E3SM's land component, based on CLM5. It +simulates vegetation, soil, hydrology, and carbon/nitrogen cycles. + +Relevant zppy tasks +------------------- + +- :doc:`../tasks/ts` — generate land time-series files +- :doc:`../tasks/e3sm_to_cmip` — convert land output to CMIP6 format +- :doc:`../tasks/e3sm_diags` — land diagnostics (e.g., ``lat_lon`` set with land variables) +- :doc:`../tasks/ilamb` — land benchmarking against observational datasets +- :doc:`../tasks/livvkit` — ice sheet validation (for cryosphere configurations) +- :doc:`../tasks/global_time_series` — land global time series (via ``plots_lnd``) + +ELM time-series configuration +------------------------------ + +ELM output is typically in monthly ``elm.h0`` files. A typical ``[ts]`` +configuration for land: + +.. code-block:: cfg + + [ts] + active = True + + [[land_monthly]] + input_files = elm.h0 + input_subdir = archive/lnd/hist + input_component = elm + mapping_file = /path/to/map_r05_to_cmip6_180x360_aave.nc + grid = 180x360_aave + years = 1:100:10 + +The ``input_component = elm`` setting tells ``zppy`` that this is ELM +output. This is required for correct processing by ``e3sm_to_cmip`` and +``ilamb``. + +CMIP conversion for ILAMB +-------------------------- + +ILAMB requires CMIP6-format data. The ``e3sm_to_cmip`` task must be run +first. Typical configuration: + +.. code-block:: cfg + + [e3sm_to_cmip] + active = True + + [[land_monthly]] + input_component = elm + ts_land_subsection = land_monthly + + [ilamb] + active = True + land_only = False + e3sm_to_cmip_land_subsection = land_monthly + +The ``ilamb`` task will depend on both the land and atmosphere +``e3sm_to_cmip`` subtasks (unless ``land_only = True``). With +``infer_section_parameters = True`` (default), subsection names for land +tasks default to ``land_monthly`` and atmosphere tasks default to +``atm_monthly_180x360_aave``. + +Ice sheet validation with LIVVkit +---------------------------------- + +For cryosphere-enabled simulations, the ``livvkit`` task validates the ice +sheet component against reanalysis datasets. LIVVkit requires grid-native +and regridded climatology files from the ``[climo]`` task: + +.. code-block:: cfg + + [climo] + [[land_monthly_climo_native]] + input_files = elm.h0 + input_subdir = archive/lnd/hist + input_component = elm + years = 1:100:20 + + [[land_monthly_climo_era5]] + input_files = elm.h0 + input_subdir = archive/lnd/hist + input_component = elm + mapping_file = /path/to/map_r05_to_era5.nc + grid = era5 + years = 1:100:20 + + [livvkit] + active = True + climo_subsections = land_monthly_climo_native, land_monthly_climo_era5 + icesheets = gis,ais + sets = cmb,smb,energy_era5 + +Notes for ELM developers +-------------------------- + +- The ``input_component = elm`` parameter is used internally by ``zppy`` + to set the correct processing type (``prc_typ``) for ``ncclimo``. +- ELM variables to process can be set via the ``vars`` parameter. Leaving + ``vars = ""`` processes all variables, which can be slow. Consider + specifying only the variables needed for your diagnostics. +- The land component mapping file maps from ELM's finite-volume grid (e.g., + ``r05``, ``ne30pg2``) to a regular lat-lon grid. diff --git a/docs/source/user_guide/components/ocean.rst b/docs/source/user_guide/components/ocean.rst new file mode 100644 index 00000000..e7f1af36 --- /dev/null +++ b/docs/source/user_guide/components/ocean.rst @@ -0,0 +1,90 @@ +.. _component-ocean: + +Ocean — MPAS-Ocean +=================== + +**MPAS-Ocean** is E3SM's ocean model, based on the Model for Prediction +Across Scales (MPAS) framework. It uses an unstructured mesh that allows +variable horizontal resolution. + +Relevant zppy tasks +------------------- + +- :doc:`../tasks/mpas_analysis` — ocean and sea-ice analysis (primary task) +- :doc:`../tasks/global_time_series` — ocean global time series plots + +MPAS-Analysis for ocean +------------------------ + +MPAS-Analysis is the primary tool for MPAS-Ocean diagnostics. It operates +on MPAS-Ocean's native unstructured mesh. The ``[mpas_analysis]`` task in +``zppy`` calls MPAS-Analysis automatically. + +Typical ``[mpas_analysis]`` configuration: + +.. code-block:: cfg + + [mpas_analysis] + active = True + walltime = 06:00:00 + shortTermArchive = True + ts_years = 1:100:50 + climo_years = 1:100:50 + enso_years = 1:100:50 + generate = all, no_landIceCavities, no_BGC, no_icebergs, no_min, no_max + +Key MPAS-Ocean-specific parameters: + +- ``mpaso_nml``: name of the MPAS-Ocean namelist file (default: ``mpaso_in``) +- ``stream_ocn``: name of the MPAS-Ocean streams file (default: ``streams.ocean``) +- ``PostMOC``: whether to post-process MOC (meridional overturning + circulation) data + +Model-vs-model ocean comparisons +--------------------------------- + +MPAS-Analysis supports comparing two model simulations. In ``zppy``, this +is configured via ``reference_data_path``: + +.. code-block:: cfg + + [mpas_analysis] + + [[main_run]] + ts_years = 1:100:50 + climo_years = 1:100:50 + + [[mvm_run]] + ts_years = 1:100:50 + climo_years = 1:100:50 + reference_data_path = [[main_run]] + # or: reference_data_path = /path/to/prior/zppy/output + +See the :doc:`../tasks/mpas_analysis` page and the :ref:`tutorial ` +for a complete model-vs-model example. + +Global time series ocean plots +------------------------------- + +The ``[global_time_series]`` task can produce ocean time series plots such +as: + +- ``change_ohc`` — ocean heat content change +- ``max_moc`` — maximum meridional overturning circulation +- ``change_sea_level`` — global mean sea level change + +These require access to MPAS-Ocean output. The ``input_subdir`` in +``[global_time_series]`` should be set to ``archive/ocn/hist`` (the default). + +Notes for MPAS-Ocean developers +--------------------------------- + +- The ``generate`` parameter in ``[mpas_analysis]`` controls which + MPAS-Analysis tasks to run. Use ``no_*`` filters to exclude irrelevant + analyses for your simulation (e.g., ``no_BGC`` if biogeochemistry is not + enabled, ``no_landIceCavities`` if land ice cavities are not used). +- The ``environment_commands`` for ``[mpas_analysis]`` must be consistent + across all year-range runs. Mixing environments can cause MPAS-Analysis to + fail. +- MPAS-Analysis caches intermediate results (``cache = True`` by default). + Set ``purge = True`` to clear previous output and start fresh. diff --git a/docs/source/user_guide/components/rivers.rst b/docs/source/user_guide/components/rivers.rst new file mode 100644 index 00000000..ea30ac0d --- /dev/null +++ b/docs/source/user_guide/components/rivers.rst @@ -0,0 +1,69 @@ +.. _component-rivers: + +Rivers — MOSART +================ + +**MOSART** (Model for Scale Adaptive River Transport) is E3SM's river +routing component. It routes runoff from ELM through river networks to +the ocean. + +Relevant zppy tasks +------------------- + +- :doc:`../tasks/ts` — generate river time-series files +- :doc:`../tasks/e3sm_diags` — streamflow diagnostics (``streamflow`` set) + +MOSART time-series configuration +--------------------------------- + +MOSART output is in monthly ``mosart.h0`` files. Configuration example: + +.. code-block:: cfg + + [ts] + active = True + + [[rof_monthly]] + input_files = mosart.h0 + input_subdir = archive/rof/hist + input_component = mosart + years = 1:100:10 + +E3SM Diags streamflow set +-------------------------- + +The ``streamflow`` set in ``e3sm_diags`` compares MOSART-simulated +streamflow against stream gauge observations. It requires: + +1. A river time-series subtask in ``[ts]`` +2. The ``streamflow_obs_ts`` parameter pointing to observational gauge data + +Example configuration: + +.. code-block:: cfg + + [ts] + [[rof_monthly]] + input_files = mosart.h0 + input_subdir = archive/rof/hist + input_component = mosart + years = 1:100:10 + + [e3sm_diags] + [[streamflow_diags]] + sets = streamflow + ts_subsection = rof_monthly + streamflow_obs_ts = /path/to/obs/streamflow + # For model-vs-model: + # run_type = model_vs_model + # reference_data_path_ts_rof = /path/to/reference/post/atm/... + # gauges_path = /path/to/gauges.nc + +Notes for MOSART developers +---------------------------- + +- The ``input_component = mosart`` parameter tells ``zppy`` how to + handle MOSART output files. +- River routing output can differ between E3SM configurations, so + the ``input_files`` stream name (``mosart.h0``) should be verified + for your specific simulation. diff --git a/docs/source/user_guide/components/seaice.rst b/docs/source/user_guide/components/seaice.rst new file mode 100644 index 00000000..f92d499b --- /dev/null +++ b/docs/source/user_guide/components/seaice.rst @@ -0,0 +1,54 @@ +.. _component-seaice: + +Sea Ice — MPAS-seaice +====================== + +**MPAS-seaice** is E3SM's sea-ice model, also based on the MPAS framework +and running on the same unstructured mesh as MPAS-Ocean. + +Relevant zppy tasks +------------------- + +- :doc:`../tasks/mpas_analysis` — primary task for sea-ice diagnostics +- :doc:`../tasks/global_time_series` — sea-ice global time series plots + +MPAS-Analysis for sea ice +-------------------------- + +MPAS-Analysis includes a suite of sea-ice diagnostics. These run +automatically alongside ocean diagnostics when ``[mpas_analysis]`` is +active. Sea-ice analyses can be controlled via the ``generate`` parameter: + +.. code-block:: cfg + + [mpas_analysis] + active = True + # To disable specific sea-ice analyses: + generate = all, no_icebergs, no_landIceCavities + +Key MPAS-seaice-specific parameters: + +- ``mpassi_nml``: name of the MPAS sea-ice namelist file (default: + ``mpassi_in``) +- ``stream_ice``: name of the MPAS sea-ice streams file (default: + ``streams.seaice``) + +Global time series sea-ice plots +--------------------------------- + +The ``[global_time_series]`` task can produce sea-ice time series plots when +``make_viewer = True``. Use the ``plots_ice`` parameter to specify which +sea-ice variables to include. + +Notes for MPAS-seaice developers +--------------------------------- + +- Sea-ice and ocean analyses share the same ``[mpas_analysis]`` section. + They are differentiated internally by MPAS-Analysis based on the + ``generate`` list. +- ``icebergs`` and ``landIceCavities`` are disabled by default in the + standard ``generate`` list because most E3SM configurations do not include + these features. Enable them if your simulation does. +- The ``anomalyRefYear`` parameter sets the reference year for anomaly + plots in MPAS-Analysis. For historical simulations, this is typically + set to the first year of the observational record. diff --git a/docs/source/getting_started.rst b/docs/source/user_guide/getting_started.rst similarity index 99% rename from docs/source/getting_started.rst rename to docs/source/user_guide/getting_started.rst index 7cbae75a..60e53d61 100644 --- a/docs/source/getting_started.rst +++ b/docs/source/user_guide/getting_started.rst @@ -264,7 +264,7 @@ Furthermore, the dev environment includes quality assurance (QA) tools such as c git commit -m "commit-message" - .. figure:: _static/pre-commit-passing.png + .. figure:: ../_static/pre-commit-passing.png :alt: pre-commit Output ``pre-commit`` Output diff --git a/docs/source/user_guide/index.rst b/docs/source/user_guide/index.rst new file mode 100644 index 00000000..db9d9c02 --- /dev/null +++ b/docs/source/user_guide/index.rst @@ -0,0 +1,19 @@ +.. _user-guide: + +********** +User Guide +********** + +This guide covers everything a researcher needs to use ``zppy`` to +post-process E3SM simulations. + +.. toctree:: + :maxdepth: 2 + + getting_started + tutorial + schematics + campaigns + parameters + tasks/index + components/index diff --git a/docs/source/parameters.rst b/docs/source/user_guide/parameters.rst similarity index 98% rename from docs/source/parameters.rst rename to docs/source/user_guide/parameters.rst index 4e4999ab..3110875d 100644 --- a/docs/source/parameters.rst +++ b/docs/source/user_guide/parameters.rst @@ -14,7 +14,9 @@ be overridden by parameters set in a ``[[subsection]``. See `this release's parameter defaults `_ on GitHub for a complete list of parameters and their default values. You can also view the most up-to-date, -`unreleased parameter defaults `_. +`unreleased parameter defaults `_. + +For per-task parameter tables, see the :ref:`Tasks ` section. Deprecated parameters ===================== diff --git a/docs/source/schematics.rst b/docs/source/user_guide/schematics.rst similarity index 97% rename from docs/source/schematics.rst rename to docs/source/user_guide/schematics.rst index 3a0a5301..560c4b5d 100644 --- a/docs/source/schematics.rst +++ b/docs/source/user_guide/schematics.rst @@ -38,7 +38,7 @@ The plots for years 51-100 have similar requirements – however the graph calls attention to the fact that MPAS-Analysis for years 51-100 itself depends on MPAS-Analysis for years 1-50. -.. image:: figures/zppy_jobs.png +.. image:: ../figures/zppy_jobs.png :scale: 80% Figure 1. Illustration of how zppy uses sections of the configuration file and @@ -47,7 +47,7 @@ climo stands for climatology and ts stands for time series. Note that the configuration file is abridged – see the :ref:`zppy tutorial ` for more complete examples. -.. image:: figures/zppy_dependencies.png +.. image:: ../figures/zppy_dependencies.png :scale: 80% Figure 2. Two possible job dependency graphs. diff --git a/docs/source/user_guide/tasks/bundle.rst b/docs/source/user_guide/tasks/bundle.rst new file mode 100644 index 00000000..bc419854 --- /dev/null +++ b/docs/source/user_guide/tasks/bundle.rst @@ -0,0 +1,59 @@ +.. _task-bundle: + +bundle — Job Bundling +====================== + +The ``bundle`` task groups multiple ``zppy`` tasks into a single SLURM job +script. This is useful for running many lightweight tasks together on a +single node, avoiding scheduler overhead and resource waste. + +When a task specifies ``bundle = ""`` in its configuration, it will +be included in the named bundle. The bundle itself is defined in a +``[bundle]`` section. + +.. note:: + Bundle parameters (e.g., ``walltime``, ``nodes``) are taken from the + *first task* added to the bundle, or from the ``[bundle]`` section + itself. The ``active`` parameter in ``[bundle]`` always overrides the + ``[default]`` value and defaults to ``True``. + +Configuration example +--------------------- + +.. code-block:: cfg + + [bundle] + [[long_jobs]] + walltime = 08:00:00 + nodes = 1 + + [climo] + [[atm_monthly_180x360_aave]] + bundle = long_jobs + years = 1:100:20 + + [ts] + [[atm_monthly_180x360_aave]] + bundle = long_jobs + years = 1:100:10 + +Parameters +---------- + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``True`` + - Whether the bundle is active. Overrides the ``[default]`` value. + * - ``bundle`` + - No + - ``""`` + - (Set on individual tasks) Name of the bundle to include this task + in. diff --git a/docs/source/user_guide/tasks/climo.rst b/docs/source/user_guide/tasks/climo.rst new file mode 100644 index 00000000..683c235e --- /dev/null +++ b/docs/source/user_guide/tasks/climo.rst @@ -0,0 +1,142 @@ +.. _task-climo: + +climo — Climatology Files +========================= + +The ``climo`` task generates climatology files from E3SM simulation output +using NCO's ``ncclimo``. It produces time-averaged files (monthly and/or +seasonal means) over a specified set of years, optionally regridding the +output to a target grid. + +Climatology files produced by ``climo`` are required inputs for the +:doc:`e3sm_diags` task. + +Configuration example +--------------------- + +.. code-block:: cfg + + [climo] + active = True + nodes = 4 + years = 1:100:20 + + [[atm_monthly_180x360_aave]] + input_files = eam.h0 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.nc + grid = 180x360_aave + vars = FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TS,TREFHT,CLDTOT + +Parameters +---------- + +The following parameters are specific to the ``climo`` task. In addition, +all :ref:`common parameters ` from the ``[default]`` section apply. + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``exclude`` + - No + - ``False`` + - Set to ``True`` to exclude this subtask from running (subsection-level). + * - ``parallel`` + - No + - ``"mpi"`` + - The ``parallel`` option passed to ``ncclimo``. + Overrides the ``[default]`` value (which is ``""``). + * - ``nodes`` + - No + - ``4`` + - Number of compute nodes. Overrides the ``[default]`` value (``1``). + * - ``climo_jobs`` + - No + - ``0`` + - Number of simultaneous ``ncclimo`` jobs. ``0`` uses ``ncclimo``'s + default. + * - ``vars`` + - No + - ``""`` + - Variables to process. An empty string processes *all* variables (no + ``-v`` flag is passed to ``ncclimo``). Overrides the ``[default]`` + value. + * - ``input_component`` + - No + - ``""`` + - Model component that generated the input files (e.g., ``eam``, + ``eamxx``, ``elm``, ``mosart``). Used to set processing type + internally. + +Inherited common parameters most relevant to ``climo`` +------------------------------------------------------ + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``case`` + - **Yes** + - *(none)* + - The case name of the simulation. + * - ``input`` + - **Yes** + - *(none)* + - The top-level directory of the simulation output to post-process. + * - ``output`` + - **Yes** + - *(none)* + - Where the post-processing results (``post/`` directory) should go. + * - ``years`` + - No + - ``[""]`` + - Year ranges to process. Format: ``"start:end:increment"`` (e.g., + ``"1:100:20"``). + * - ``input_files`` + - No + - ``"eam.h0"`` + - Pattern matching the input history files (e.g., ``eam.h0``, + ``eam.h1``). + * - ``input_subdir`` + - No + - ``"archive/atm/hist"`` + - Subdirectory under ``input``/``case`` containing the data files. + * - ``mapping_file`` + - No + - ``""`` + - Path to the mapping (regridding) file. Leave empty for no regridding. + * - ``grid`` + - No + - ``""`` + - Name of the output grid (e.g., ``180x360_aave``). Used for naming + output directories. + * - ``walltime`` + - No + - ``"02:00:00"`` + - Maximum wall time for the SLURM job. + * - ``partition`` + - No + - ``""`` + - SLURM partition to submit the job to. + * - ``account`` + - No + - ``""`` + - SLURM account to charge. + * - ``environment_commands`` + - No + - ``""`` + - Shell commands to set up the software environment before running the + task (e.g., ``source /path/to/e3sm_unified.sh``). diff --git a/docs/source/user_guide/tasks/e3sm_diags.rst b/docs/source/user_guide/tasks/e3sm_diags.rst new file mode 100644 index 00000000..8c022479 --- /dev/null +++ b/docs/source/user_guide/tasks/e3sm_diags.rst @@ -0,0 +1,191 @@ +.. _task-e3sm-diags: + +e3sm_diags — E3SM Diagnostics +============================== + +The ``e3sm_diags`` task runs `E3SM Diagnostics +`_ to produce a wide variety of +atmospheric, land, and river diagnostic plots. It depends on the :doc:`climo` +and/or :doc:`ts` tasks for its input files. + +For a complete list of available E3SM Diags parameters, see the +`E3SM Diags documentation +`_. + +For a table of which diagnostic sets require which parameters, see +`Confluence +`_. + +Diagnostic sets +--------------- + +The ``sets`` parameter controls which diagnostic sets are run. All 19 +available sets are: + +``aerosol_aeronet``, ``aerosol_budget``, ``annual_cycle_zonal_mean``, +``area_mean_time_series``, ``cosp_histogram``, ``diurnal_cycle``, +``enso_diags``, ``lat_lon``, ``meridional_mean_2d``, ``mp_partition``, +``polar``, ``precip_pdf``, ``qbo``, ``streamflow``, ``tc_analysis``, +``tropical_subseasonal``, ``zonal_mean_2d``, ``zonal_mean_2d_stratosphere``, +``zonal_mean_xy`` + +Parameters +---------- + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``sets`` + - No + - ``"lat_lon,zonal_mean_xy,..."`` + - List of diagnostic sets to run. + * - ``run_type`` + - No + - ``"model_vs_obs"`` + - Comparison type: ``"model_vs_obs"`` or ``"model_vs_model"``. + * - ``climo_subsection`` + - No + - ``""`` + - Name of the ``[climo]`` subtask to depend on. + * - ``backend`` + - No + - ``"mpl"`` + - Plotting backend. + * - ``cfg`` + - No + - ``""`` + - Path to an additional E3SM Diags configuration file. + * - ``multiprocessing`` + - No + - ``True`` + - Use multiprocessing. + * - ``num_workers`` + - No + - ``24`` + - Number of worker processes. + * - ``output_format`` + - No + - ``["png"]`` + - Output plot formats. + * - ``output_format_subplot`` + - No + - ``[]`` + - Output formats for subplots. + * - ``short_name`` + - No + - ``""`` + - Short name used as test name and label. + * - ``tag`` + - No + - ``"model_vs_obs"`` + - Label for the results directory. + * - ``swap_test_ref`` + - No + - ``False`` + - Swap test and reference in model-vs-model runs. + * - ``keep_mvm_case_name_in_fig`` + - No + - ``True`` + - Include case name in model-vs-model output paths. + * - ``obs_ts`` + - No + - ``""`` + - Path to observation time-series data for ``enso_diags``, ``qbo``, + ``area_mean_time_series`` sets. + * - ``reference_data_path`` + - No* + - ``""`` + - Path to reference climatology data. + **Required for** ``run_type="model_vs_model"``. + * - ``ref_name`` + - No* + - ``""`` + - Reference dataset name. + **Required for** ``run_type="model_vs_model"``. + * - ``short_ref_name`` + - No* + - ``""`` + - Short reference name. + **Required for** ``run_type="model_vs_model"``. + * - ``diff_title`` + - No + - ``"Model - Observations"`` + - Title for difference plots. + * - ``ref_years`` + - No + - ``[""]`` + - Year ranges for reference data in model-vs-model runs. + * - ``ref_start_yr`` + - No* + - ``""`` + - Start year of reference data. Required for ``enso_diags``, ``qbo``. + * - ``ref_final_yr`` + - No* + - ``""`` + - End year of reference data. Required for ``qbo`` and certain + model-vs-model sets. + * - ``ts_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` subtask to depend on. + * - ``ts_num_years_ref`` + - No + - ``5`` + - Year increment for reference time-series data. + * - ``ts_daily_subsection`` + - No + - ``""`` + - ``[ts]`` subtask for daily data (required for ``tropical_subseasonal``). + * - ``climo_diurnal_subsection`` + - No + - ``""`` + - ``[climo]`` subtask for diurnal cycle data. + * - ``climo_diurnal_frequency`` + - No + - ``""`` + - Frequency name for the diurnal cycle ``[climo]`` subtask. + * - ``dc_obs_climo`` + - No + - ``""`` + - Path to observation climatology for ``diurnal_cycle`` set. + * - ``reference_data_path_climo_diurnal`` + - No + - ``""`` + - Reference climatology path for ``diurnal_cycle`` MVM runs. + * - ``reference_data_path_tc`` + - No + - ``""`` + - Reference data path for ``tc_analysis`` MVM runs. + * - ``reference_data_path_ts`` + - No + - ``""`` + - Reference time-series path for ``enso_diags``/``qbo``/``area_mean_time_series``. + * - ``reference_data_path_ts_daily`` + - No + - ``""`` + - Reference daily time-series path for ``tropical_subseasonal``. + * - ``reference_data_path_ts_rof`` + - No + - ``""`` + - Reference river time-series path for ``streamflow`` MVM runs. + * - ``streamflow_obs_ts`` + - No + - ``""`` + - Path to observation data for ``streamflow`` set. + * - ``gauges_path`` + - No + - ``""`` + - Path to stream gauge data for ``streamflow`` MVM runs. + * - ``tc_obs`` + - No + - ``""`` + - Path to observation data for ``tc_analysis`` set. diff --git a/docs/source/user_guide/tasks/e3sm_to_cmip.rst b/docs/source/user_guide/tasks/e3sm_to_cmip.rst new file mode 100644 index 00000000..b7371c30 --- /dev/null +++ b/docs/source/user_guide/tasks/e3sm_to_cmip.rst @@ -0,0 +1,114 @@ +.. _task-e3sm-to-cmip: + +e3sm_to_cmip — CMIP6 Conversion +================================= + +The ``e3sm_to_cmip`` task converts E3SM output to CMIP6-compliant format +using the ``e3sm_to_cmip`` package. It depends on the :doc:`ts` task to +provide input time-series files. Its output is consumed by :doc:`ilamb` +and :doc:`pcmdi_diags`. + +Parameters +---------- + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``cmip_metadata`` + - No + - ``"inclusions/e3sm_to_cmip/default_metadata.json"`` + - Path to the CMIP metadata JSON file. + * - ``cmip_plevdata`` + - No + - ``""`` + - Path to pressure-level data NetCDF file. Defaults to a file under + ``diagnostics_base_path`` when empty. + * - ``cmip_vars`` + - No + - ``""`` + - Variables to convert to CMIP format. Leave empty to convert all + supported variables. + * - ``input_component`` + - No + - ``""`` + - Model component (e.g., ``eam``, ``eamxx``, ``elm``). + * - ``interp_vars`` + - No + - ``"U,V,T,Q,RELHUM,OMEGA,Z3"`` + - Variables to interpolate to pressure levels during conversion. + * - ``vrt_in_file`` + - No + - ``""`` + - Source vertical coordinate file for ``ncremap``. Required for + EAMxx (L128 grid). + * - ``ts_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` subtask to depend on. If empty and + ``infer_section_parameters`` is ``True``, inferred from the + subsection name. + * - ``ts_atm_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` atmosphere subtask (for multi-component runs). + * - ``ts_land_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` land subtask (for multi-component runs). + +Inherited common parameters most relevant to ``e3sm_to_cmip`` +------------------------------------------------------------- + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``case`` + - **Yes** + - *(none)* + - The case name of the simulation. + * - ``input`` + - **Yes** + - *(none)* + - The top-level directory of the simulation output. + * - ``output`` + - **Yes** + - *(none)* + - Where the post-processing results go. + * - ``years`` + - No + - ``[""]`` + - Year ranges to process. + * - ``ts_grid`` + - No + - ``"180x360_aave"`` + - Grid name of the ``[ts]`` subtask to depend on. + * - ``ts_atm_grid`` + - No + - ``"180x360_aave"`` + - Grid name of the atmosphere ``[ts]`` subtask. + * - ``ts_land_grid`` + - No + - ``"180x360_aave"`` + - Grid name of the land ``[ts]`` subtask. + * - ``ts_num_years`` + - No + - ``5`` + - Years increment from the dependent ``[ts]`` task. + * - ``environment_commands`` + - No + - ``""`` + - Shell commands to activate the software environment. diff --git a/docs/source/user_guide/tasks/global_time_series.rst b/docs/source/user_guide/tasks/global_time_series.rst new file mode 100644 index 00000000..4b294e67 --- /dev/null +++ b/docs/source/user_guide/tasks/global_time_series.rst @@ -0,0 +1,102 @@ +.. _task-global-time-series: + +global_time_series — Global Time Series Plots +============================================= + +The ``global_time_series`` task generates global time series plots of +key climate metrics. It can optionally produce a Viewer page. It depends on +the :doc:`ts` task (for atmospheric, land, and ocean global-mean time series) +and optionally on :doc:`mpas_analysis` (for ocean and sea-ice plots). + +Parameters +---------- + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``ts_years`` + - No + - ``[""]`` + - Year ranges for the time-series sub-runs to depend on. + * - ``climo_years`` + - No + - ``[""]`` + - Year ranges for climatology data (if needed). + * - ``color`` + - No + - ``"Blue"`` + - Color used for the plots. + * - ``experiment_name`` + - No + - ``""`` + - Label displayed on the plots. + * - ``figstr`` + - No + - ``""`` + - Prefix for output figure file names. + * - ``make_viewer`` + - No + - ``False`` + - Set to ``True`` to construct a Viewer page with extended plots. + * - ``regions`` + - No + - ``"glb,n,s"`` + - Regions to plot: ``glb`` (global), ``n`` (northern hemisphere), + ``s`` (southern hemisphere). + * - ``input_subdir`` + - No + - ``"archive/ocn/hist"`` + - Subdirectory with ocean data. Overrides the ``[default]`` value. + * - ``moc_file`` + - No + - ``""`` + - Path to the MOC file (for the ``max_moc`` plot). Only needed when + ``make_viewer = False``. + * - ``mpas_analysis_subsections`` + - No + - ``[""]`` + - Names of ``[mpas_analysis]`` subtasks to depend on. Leave empty if + no subsections are defined. + * - ``plots_original`` + - No + - ``"net_toa_flux_restom,..."`` + - Names of the standard plots to generate when ``make_viewer = False``. + Available plots and their variable requirements: + + - ``net_toa_flux_restom`` — requires ``RESTOM`` + - ``net_atm_energy_imbalance`` — requires ``RESTOM``, ``RESSURF`` + - ``global_surface_air_temperature`` — requires ``TREFHT`` + - ``toa_radiation`` — requires ``FSNTOA``, ``FLUT`` + - ``net_atm_water_imbalance`` — requires ``PRECC``, ``PRECL``, ``QFLX`` + - ``change_ohc`` — requires ocean data + - ``max_moc`` — requires ocean MOC data + - ``change_sea_level`` — requires ocean data + + Remove the three ocean plots if you don't have ocean data. + * - ``plots_atm`` + - No + - ``""`` + - Extra atmosphere plots (when ``make_viewer = True``). These should be + a subset of the variables from ``ts`` global subtasks. + * - ``plots_lnd`` + - No + - ``""`` + - Extra land plots (when ``make_viewer = True``). Set to ``"all"`` to + include every variable in the land CSV file. + * - ``plots_ocn`` + - No + - ``""`` + - Extra ocean plots (when ``make_viewer = True``). + * - ``plots_ice`` + - No + - ``""`` + - Extra sea-ice plots (when ``make_viewer = True``). diff --git a/docs/source/user_guide/tasks/ilamb.rst b/docs/source/user_guide/tasks/ilamb.rst new file mode 100644 index 00000000..24cc19c6 --- /dev/null +++ b/docs/source/user_guide/tasks/ilamb.rst @@ -0,0 +1,71 @@ +.. _task-ilamb: + +ilamb — Land Benchmarking +========================== + +The ``ilamb`` task runs `ILAMB `_ (International +Land Model Benchmarking) to evaluate E3SM's land component against +observational benchmarks. It depends on the :doc:`e3sm_to_cmip` task to +provide CMIP-format land and atmosphere time-series files. + +Parameters +---------- + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``cfg`` + - No + - ``"inclusions/ilamb/ilamb.cfg"`` + - Path to the ILAMB configuration file. + * - ``e3sm_to_cmip_atm_subsection`` + - No + - ``""`` + - Name of the ``[e3sm_to_cmip]`` atmosphere subtask to depend on. + Inferred as ``atm_monthly_180x360_aave`` if not specified. + * - ``e3sm_to_cmip_land_subsection`` + - No + - ``""`` + - Name of the ``[e3sm_to_cmip]`` land subtask to depend on. Inferred + as ``land_monthly`` if not specified. + * - ``ilamb_obs`` + - No + - ``""`` + - Path to ILAMB observational data. Defaults to + ``diagnostics_base_path/ilamb_data`` if empty. + * - ``land_only`` + - No + - ``False`` + - Set to ``True`` for land-only runs (skips atmosphere dependencies). + * - ``ts_atm_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` atmosphere subtask to depend on. Inferred as + ``atm_monthly_180x360_aave`` if not specified and not a land-only run. + * - ``ts_land_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` land subtask to depend on. Inferred as + ``land_monthly`` if not specified. + +Dependency inference +-------------------- + +When ``infer_section_parameters = True`` (the default), ``zppy`` will +automatically determine the following if not explicitly set: + +- ``ts_land_subsection`` defaults to ``land_monthly`` +- ``e3sm_to_cmip_land_subsection`` defaults to ``land_monthly`` +- ``ts_atm_subsection`` defaults to ``atm_monthly_180x360_aave`` (for + non-land-only runs) +- ``e3sm_to_cmip_atm_subsection`` defaults to ``atm_monthly_180x360_aave`` + (for non-land-only runs) diff --git a/docs/source/user_guide/tasks/index.rst b/docs/source/user_guide/tasks/index.rst new file mode 100644 index 00000000..c81210cf --- /dev/null +++ b/docs/source/user_guide/tasks/index.rst @@ -0,0 +1,59 @@ +.. _user-tasks: + +***** +Tasks +***** + +``zppy`` is organized around **tasks**. Each task corresponds to a specific +post-processing operation. Tasks are defined as sections (``[task_name]``) in +the configuration file and may contain subsections (``[[subsection_name]]``) +for multiple independent runs of the same task. + +**Available tasks:** + +.. list-table:: + :header-rows: 1 + :widths: 25 75 + + * - Task + - Description + * - :doc:`climo` + - Generate climatology files using NCO's ``ncclimo`` + * - :doc:`ts` + - Generate time-series files using NCO's ``ncclimo`` + * - :doc:`e3sm_to_cmip` + - Convert E3SM output to CMIP6 format using ``e3sm_to_cmip`` + * - :doc:`e3sm_diags` + - Run E3SM Diagnostics (atmosphere, land, river diagnostics, etc.) + * - :doc:`mpas_analysis` + - Run MPAS-Analysis (ocean and sea-ice diagnostics) + * - :doc:`global_time_series` + - Generate global time series plots + * - :doc:`tc_analysis` + - Tropical cyclone analysis + * - :doc:`ilamb` + - Run ILAMB land benchmarking + * - :doc:`livvkit` + - Run LIVVkit ice sheet validation + * - :doc:`pcmdi_diags` + - Run PCMDI metrics diagnostics + * - :doc:`bundle` + - Bundle multiple tasks into a single SLURM job + +**Common (default) parameters** shared by all tasks are documented on the +:doc:`../parameters` page. + +.. toctree:: + :hidden: + + climo + ts + e3sm_to_cmip + e3sm_diags + mpas_analysis + global_time_series + tc_analysis + ilamb + livvkit + pcmdi_diags + bundle diff --git a/docs/source/user_guide/tasks/livvkit.rst b/docs/source/user_guide/tasks/livvkit.rst new file mode 100644 index 00000000..4a1bcbd3 --- /dev/null +++ b/docs/source/user_guide/tasks/livvkit.rst @@ -0,0 +1,59 @@ +.. _task-livvkit: + +livvkit — Ice Sheet Validation +================================ + +The ``livvkit`` task runs `LIVVkit +`_ (Land Ice Verification and Validation +Toolkit) for ice sheet model evaluation. It compares climatology files +generated by the :doc:`climo` task against reanalysis datasets. + +Parameters +---------- + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``climo_subsections`` + - No + - ``[""]`` + - List of ``[climo]`` subtasks to depend on. + * - ``cfg`` + - No + - ``"inclusions/livvkit/livvkit_r05.jinja"`` + - Path to the LIVVkit configuration template. + * - ``icesheets`` + - No + - ``"gis,ais"`` + - Icesheets to analyze: ``gis`` (Greenland), ``ais`` (Antarctica). + * - ``sets`` + - No + - ``"cmb,smb,energy_racmo,energy_era5,energy_merra2,energy_ceres"`` + - Validation sets to perform. + +Climo dependency requirements +------------------------------ + +The ``livvkit`` task requires specific ``[climo]`` subtasks to be defined: + +- A grid-native climatology must be generated, assumed to be named + ``land_monthly_climo_native``. +- For each reanalysis comparison, a corresponding + ``land_monthly_climo_GRID`` subtask is needed, where ``GRID`` is the + reanalysis grid name. + +Available reanalysis comparisons and their grid names: + +- ``racmo_gis``, ``racmo_ais`` +- ``merra2`` +- ``era5`` +- ``ceres`` (uses the default CMIP6 grid) diff --git a/docs/source/user_guide/tasks/mpas_analysis.rst b/docs/source/user_guide/tasks/mpas_analysis.rst new file mode 100644 index 00000000..83211ed1 --- /dev/null +++ b/docs/source/user_guide/tasks/mpas_analysis.rst @@ -0,0 +1,158 @@ +.. _task-mpas-analysis: + +mpas_analysis — Ocean and Sea-Ice Analysis +========================================== + +The ``mpas_analysis`` task runs `MPAS-Analysis +`_ to produce ocean and sea-ice +diagnostics. It supports both model-vs-observations (``mvo``) and +model-vs-model (``mvm``) comparison modes. The comparison type is inferred +automatically: if ``reference_data_path`` is set, the run is treated as MVM; +otherwise it is treated as MVO. + +.. note:: + ``environment_commands`` should be the same for all related + ``[mpas_analysis]`` runs. Mixing environments across year ranges can cause + MPAS-Analysis to fail. + +Parameters +---------- + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``generate`` + - No + - ``"all,no_landIceCavities,..."`` + - List of MPAS-Analysis analyses to generate. + * - ``ts_years`` + - No + - ``[""]`` + - Year ranges for time-series sub-runs. + * - ``climo_years`` + - No + - ``[""]`` + - Year ranges for climatology sub-runs. + * - ``enso_years`` + - No + - ``[""]`` + - Year ranges for ENSO sub-runs. + * - ``reference_data_path`` + - No + - ``""`` + - For MVM runs: path to a prior zppy output directory (the one + containing ``post/``) to use as the reference run. Setting this + triggers MVM mode. + * - ``test_data_path`` + - No + - ``""`` + - For MVM runs: path to the test run's prior zppy output directory (if + different from the current run). + * - ``reference_comparison_type`` + - No + - ``"auto"`` + - Comparison type for the referenced prior run: ``"auto"``, ``"mvo"``, + or ``"mvm"``. + * - ``test_comparison_type`` + - No + - ``"auto"`` + - Comparison type for the test run's referenced prior run. + * - ``reference_case`` + - No* + - ``""`` + - Case name of the reference run. Required when + ``reference_data_path`` points to a non-subsection path. + * - ``ref_ts_years`` + - No + - ``[""]`` + - Year ranges for reference time-series sub-runs. Defaults to + ``ts_years`` if empty. + * - ``ref_climo_years`` + - No + - ``[""]`` + - Year ranges for reference climatology sub-runs. + * - ``ref_enso_years`` + - No + - ``[""]`` + - Year ranges for reference ENSO sub-runs. + * - ``anomalyRefYear`` + - No + - ``1`` + - Reference year for anomaly calculations. + * - ``cache`` + - No + - ``True`` + - Whether to cache MPAS-Analysis intermediate results. + * - ``mapMpiTasks`` + - No + - ``6`` + - Number of MPI tasks for mapping. + * - ``mpaso_nml`` + - No + - ``"mpaso_in"`` + - Name of the MPAS-Ocean namelist file. + * - ``mpassi_nml`` + - No + - ``"mpassi_in"`` + - Name of the MPAS sea-ice namelist file. + * - ``ncclimoThreads`` + - No + - ``12`` + - Number of threads for ``ncclimo`` in MPAS-Analysis. + * - ``ncclimoParallelMode`` + - No + - ``"bck"`` + - Parallel mode for ``ncclimo`` in MPAS-Analysis. + * - ``parallelTaskCount`` + - No + - ``12`` + - Number of parallel analysis tasks in MPAS-Analysis. + * - ``PostMOC`` + - No + - ``False`` + - Whether to post-process MOC data. + * - ``purge`` + - No + - ``False`` + - Whether to purge previous MPAS-Analysis output before running. + * - ``shortTermArchive`` + - No + - ``True`` + - Whether the input uses the short-term archive directory structure. + * - ``stream_ice`` + - No + - ``"streams.seaice"`` + - Name of the MPAS sea-ice streams file. + * - ``stream_ocn`` + - No + - ``"streams.ocean"`` + - Name of the MPAS ocean streams file. + * - ``walltime`` + - No + - ``"06:00:00"`` + - Maximum wall time. Overrides the ``[default]`` value (``02:00:00``). + +Year range behavior for MVM runs +--------------------------------- + +For model-vs-model comparisons, ``zppy`` supports separate year ranges for +the test and reference runs: + +- ``ts_years``, ``climo_years``, ``enso_years``: define the **test** run + year ranges. +- ``ref_ts_years``, ``ref_climo_years``, ``ref_enso_years``: optionally + override the **reference** run year ranges (defaults to test year ranges + if not set). +- If ``reference_data_path`` points to a prior ``[[subsection]]`` in the + same cfg, the reference year ranges default to that subsection's values. +- If a single reference year range is provided and multiple test ranges are + requested, the single reference range is used for each test range. diff --git a/docs/source/user_guide/tasks/pcmdi_diags.rst b/docs/source/user_guide/tasks/pcmdi_diags.rst new file mode 100644 index 00000000..7fd41916 --- /dev/null +++ b/docs/source/user_guide/tasks/pcmdi_diags.rst @@ -0,0 +1,218 @@ +.. _task-pcmdi-diags: + +pcmdi_diags — PCMDI Metrics +============================ + +The ``pcmdi_diags`` task runs `PCMDI Metrics Package (PMP) +`_ diagnostics to compute climate +model performance metrics. It depends on the :doc:`e3sm_to_cmip` task for +CMIP-format input data. + +Diagnostic sets +--------------- + +The ``current_set`` parameter selects which PMP diagnostic to run: + +- ``mean_climate`` — climatological mean metrics +- ``variability_modes_cpl`` — coupled variability modes (PDO, NPGO, AMO) +- ``variability_modes_atm`` — atmospheric variability modes (NAM, NAO, PNA, etc.) +- ``enso`` — ENSO metrics (not currently enabled by default) +- ``synthetic_plots`` — composite viewer page combining outputs from other sets + +Parameters +---------- + +Most useful at the task level (all sets) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``run_type`` + - No + - ``"model_vs_obs"`` + - ``"model_vs_obs"`` or ``"model_vs_model"``. Note: in PCMDI context, + model-vs-model compares two model simulations against observations. + * - ``figure_format`` + - No + - ``"png"`` + - Output figure format. + * - ``model_name`` + - No + - ``"e3sm.historical.v3-LR.0051"`` + - Model name (required for MVO runs). + * - ``model_tableID`` + - No + - ``"Amon"`` + - CMIP table ID (required for MVO runs). + * - ``model_name_ref`` + - No + - ``"ERA5"`` + - Reference model name (required for MVM runs). + * - ``model_tableID_ref`` + - No + - ``"Amon"`` + - Reference CMIP table ID (required for MVM runs). + * - ``e3sm_to_cmip_atm_subsection`` + - No + - ``""`` + - Name of the ``[e3sm_to_cmip]`` atm subtask to depend on. + * - ``generate_sftlf`` + - No + - ``True`` + - Flag to process the land/sea mask within PCMDI. + * - ``grid`` + - No + - ``"180x360_aave"`` + - Model data grid after remapping. + * - ``multiprocessing`` + - No + - ``True`` + - Use multiprocessing. + * - ``num_workers`` + - No + - ``24`` + - Number of worker processes. + * - ``obs_ts`` + - No + - ``""`` + - Path to observation time-series data. Required for + ``mean_climate``, ``variability_modes_*``, ``enso`` sets. + * - ``ts_num_years`` + - No + - ``5`` + - Year increment for test model data. + * - ``ts_years`` + - No + - ``[""]`` + - Year ranges for test model data. + +Per-subtask parameters +~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``current_set`` + - No + - ``""`` + - The diagnostic set to run. If not set, inferred from the subsection + name. + * - ``obs_sets`` + - No + - ``"default"`` + - Observational datasets to use (see ``reference_alias.json``). + * - ``ref_start_yr`` + - No + - ``""`` + - Start year for reference data. + * - ``ref_final_yr`` + - No + - ``""`` + - End year for reference data. + * - ``ref_years`` + - No + - ``[""]`` + - Year ranges for reference data. + +mean_climate parameters +~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``cmip_vars`` + - No + - ``"pr,prw,psl,..."`` + - Variables from the CMIP6 table to use. + * - ``clim_vars`` + - No + - ``"pr,prw,psl,..."`` + - Variables for mean climate metrics. + * - ``clim_regions`` + - No + - ``"global,ocean,land"`` + - Regions for mean climate metrics. + * - ``target_grid`` + - No + - ``"1x1"`` + - Target regridding grid. + * - ``target_grid_string`` + - No + - ``"1px1p"`` + - Description string for the target grid. + * - ``regrid_tool`` + - No + - ``"esmf"`` + - Regridding tool. + * - ``regrid_method`` + - No + - ``"regrid2"`` + - Regridding method. + +variability_modes parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``mova_vars`` + - No + - ``"psl"`` + - Variables for ``variability_modes_atm``. + * - ``mova_modes`` + - No + - ``"NAM,NAO,PNA,NPO,SAM,PSA1,PSA2"`` + - Modes for ``variability_modes_atm``. + * - ``movc_vars`` + - No + - ``"ts"`` + - Variables for ``variability_modes_cpl``. + * - ``movc_modes`` + - No + - ``"PDO,NPGO,AMO"`` + - Modes for ``variability_modes_cpl``. + * - ``CBF`` + - No + - ``True`` + - Use Common Base Function approach. + * - ``ConvEOF`` + - No + - ``True`` + - Compute conventional EOF. + * - ``EofScaling`` + - No + - ``False`` + - Apply EOF scaling. + * - ``eofn_mod_max`` + - No + - ``3`` + - Maximum number of EOF modes for model. + * - ``seasons`` + - No + - ``"monthly"`` + - Seasons to analyze. diff --git a/docs/source/user_guide/tasks/tc_analysis.rst b/docs/source/user_guide/tasks/tc_analysis.rst new file mode 100644 index 00000000..d1c793bc --- /dev/null +++ b/docs/source/user_guide/tasks/tc_analysis.rst @@ -0,0 +1,74 @@ +.. _task-tc-analysis: + +tc_analysis — Tropical Cyclone Analysis +======================================== + +The ``tc_analysis`` task performs tropical cyclone (TC) analysis on E3SM +output. It uses high-frequency atmospheric data (typically ``eam.h2``) to +detect and track TCs. + +Parameters +---------- + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``input_files`` + - No + - ``"eam.h2"`` + - Input history file pattern. Overrides the ``[default]`` value + (``eam.h0``). + * - ``res`` + - No + - ``""`` + - Resolution string for the ``--res`` option. If not set, ``zppy`` + attempts to infer it from the topography file. + +Inherited common parameters most relevant to ``tc_analysis`` +------------------------------------------------------------ + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``case`` + - **Yes** + - *(none)* + - The case name of the simulation. + * - ``input`` + - **Yes** + - *(none)* + - The top-level directory of the simulation output. + * - ``output`` + - **Yes** + - *(none)* + - Where the post-processing results go. + * - ``years`` + - No + - ``[""]`` + - Year ranges to process. + * - ``input_subdir`` + - No + - ``"archive/atm/hist"`` + - Subdirectory under ``input``/``case`` containing the data files. + * - ``environment_commands`` + - No + - ``""`` + - Shell commands to activate the software environment. + * - ``walltime`` + - No + - ``"02:00:00"`` + - Maximum wall time for the SLURM job. diff --git a/docs/source/user_guide/tasks/ts.rst b/docs/source/user_guide/tasks/ts.rst new file mode 100644 index 00000000..fd8c023b --- /dev/null +++ b/docs/source/user_guide/tasks/ts.rst @@ -0,0 +1,158 @@ +.. _task-ts: + +ts — Time Series Files +====================== + +The ``ts`` task generates per-variable time-series files from E3SM +simulation output using NCO's ``ncclimo``. It produces one file per variable +covering a specified year range. Time-series files produced by ``ts`` are +required inputs for the :doc:`e3sm_diags`, :doc:`global_time_series`, +:doc:`e3sm_to_cmip`, :doc:`ilamb`, and :doc:`pcmdi_diags` tasks. + +Configuration example +--------------------- + +.. code-block:: cfg + + [ts] + active = True + years = 1:100:10 + + [[atm_monthly_180x360_aave]] + input_files = eam.h0 + input_subdir = archive/atm/hist + mapping_file = /path/to/map_ne30pg2_to_cmip6_180x360_aave.nc + grid = 180x360_aave + vars = FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TS,TREFHT + + [[land_monthly]] + input_files = elm.h0 + input_subdir = archive/lnd/hist + input_component = elm + mapping_file = /path/to/map_r05_to_cmip6_180x360_aave.nc + grid = 180x360_aave + +Parameters +---------- + +The following parameters are specific to the ``ts`` task. + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``area_nm`` + - No + - ``"area"`` + - Name of the area variable in the input files. + * - ``dpf`` + - No + - ``30`` + - Days per file (for daily or sub-daily data). + * - ``extra_vars`` + - No + - ``""`` + - Additional variables to include alongside the primary ``vars`` list. + * - ``job_nbr`` + - No + - ``0`` + - Number of simultaneous ``ncclimo`` jobs. ``0`` uses ``ncclimo``'s + default. + * - ``tpd`` + - No + - ``1`` + - Time-steps per day (for sub-daily data, e.g., ``8`` for 3-hourly). + * - ``input_component`` + - No + - ``""`` + - Model component that generated the input files (e.g., ``eam``, + ``eamxx``, ``elm``, ``mosart``). Used to set processing type + internally. + * - ``vrt_remap_vars`` + - No + - ``""`` + - Variables to additionally regrid from model levels to pressure levels + after generating time-series files. Output is written to a sibling + ``ts_vrt_remap/`` directory. Empty string disables vertical remapping. + * - ``vrt_remap_file`` + - No + - ``""`` + - Path to the target vertical grid file (pressure levels) for + ``ncremap``. Defaults to a standard file under + ``diagnostics_base_path`` when empty. + * - ``vrt_in_file`` + - No + - ``""`` + - Path to the source vertical coordinate file for ``ncremap``. Required + for EAMxx (L128 grid). Defaults to a standard file under + ``diagnostics_base_path`` when ``input_component`` is ``eamxx`` and + this is left empty. + +Inherited common parameters most relevant to ``ts`` +--------------------------------------------------- + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``case`` + - **Yes** + - *(none)* + - The case name of the simulation. + * - ``input`` + - **Yes** + - *(none)* + - The top-level directory of the simulation output to post-process. + * - ``output`` + - **Yes** + - *(none)* + - Where the post-processing results (``post/`` directory) should go. + * - ``years`` + - No + - ``[""]`` + - Year ranges to process. Format: ``"start:end:increment"`` (e.g., + ``"1:100:10"``). + * - ``input_files`` + - No + - ``"eam.h0"`` + - Pattern matching the input history files. + * - ``input_subdir`` + - No + - ``"archive/atm/hist"`` + - Subdirectory under ``input``/``case`` containing the data files. + * - ``mapping_file`` + - No + - ``""`` + - Path to the mapping (regridding) file. Leave empty for no regridding. + * - ``grid`` + - No + - ``""`` + - Name of the output grid (e.g., ``180x360_aave``). + * - ``vars`` + - No + - ``"FSNTOA,FLUT,..."`` + - Variables to process. An empty string processes all variables. + * - ``walltime`` + - No + - ``"02:00:00"`` + - Maximum wall time for the SLURM job. + * - ``partition`` + - No + - ``""`` + - SLURM partition to submit the job to. + * - ``account`` + - No + - ``""`` + - SLURM account to charge. diff --git a/docs/source/tutorial.rst b/docs/source/user_guide/tutorial.rst similarity index 96% rename from docs/source/tutorial.rst rename to docs/source/user_guide/tutorial.rst index e4144f24..27ac107b 100644 --- a/docs/source/tutorial.rst +++ b/docs/source/user_guide/tutorial.rst @@ -18,7 +18,7 @@ Let's say we want to post-process 100 years of an existing simulation. Copy and paste the following into ``post.mysimulation.cfg`` -.. literalinclude:: post.mysimulation.cfg +.. literalinclude:: ../post.mysimulation.cfg :language: cfg :linenos: @@ -54,7 +54,7 @@ Example 2 This is another example of a configuration file, this time using a RRM simulation. -.. literalinclude:: post.rrm_simulation.cfg +.. literalinclude:: ../post.rrm_simulation.cfg :language: cfg :linenos: @@ -79,7 +79,7 @@ makes it an ``mvm`` run, otherwise it is an ``mvo`` run. If a referenced prior run could resolve to either ``mvo`` or ``mvm``, use ``reference_comparison_type`` or ``test_comparison_type`` to disambiguate. -.. literalinclude:: post.mpas_analysis_model_vs_model.cfg +.. literalinclude:: ../post.mpas_analysis_model_vs_model.cfg :language: cfg :linenos: From 61681bccf63138c11f0c1246b251a9b6478ad2b4 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 3 Jun 2026 18:31:29 +0000 Subject: [PATCH 02/27] Fix doc review issues: spelling, incomplete sentence, placeholder comments --- docs/source/dev_guide/testing.rst | 2 +- docs/source/user_guide/campaigns.rst | 2 +- docs/source/user_guide/parameters.rst | 6 +++--- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/source/dev_guide/testing.rst b/docs/source/dev_guide/testing.rst index b867b103..59ba1778 100644 --- a/docs/source/dev_guide/testing.rst +++ b/docs/source/dev_guide/testing.rst @@ -84,7 +84,7 @@ Update ``zppy-interfaces`` to the latest main branch: bash # isolated subshell source ~/.bashrc - source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified__chrysalis.sh + source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified__chrysalis.sh # e.g., 1.13.0rc10 # Verify zppy-interfaces tests pytest tests/unit/global_time_series/test_*.py diff --git a/docs/source/user_guide/campaigns.rst b/docs/source/user_guide/campaigns.rst index 779a223e..d4c70287 100644 --- a/docs/source/user_guide/campaigns.rst +++ b/docs/source/user_guide/campaigns.rst @@ -20,4 +20,4 @@ It will only work if you: - have access to the `bigmem nodes on NERSC Cori `_ - first load the necessary module with ``module load cmem``. -# TODO: supported on Perlmutter? +Support for Perlmutter has not been confirmed for this campaign type. diff --git a/docs/source/user_guide/parameters.rst b/docs/source/user_guide/parameters.rst index 3110875d..ed090f2f 100644 --- a/docs/source/user_guide/parameters.rst +++ b/docs/source/user_guide/parameters.rst @@ -35,7 +35,7 @@ Parameter checking & inferring -- for users There are two types of inferences, each with their own parameter in ``default.ini``: -* ``infer_path_parameters``: infer paths that are not explicitly provided in the configuraiton file. Default is ``True``. +* ``infer_path_parameters``: infer paths that are not explicitly provided in the configuration file. Default is ``True``. * ``infer_section_parameters``: infer subtask dependency names that are not explicitly provided in the configuration file. Default is ``True``. **Section inferences** @@ -137,8 +137,8 @@ There are many parameter-handling functions. In ``utils.py``: -* ``get_value_from_parameter``: check if parameter is in the configuration dictionary. If not, if inference is turned on (the default), then just use the value of ``second_choice_parameter``. If inferenceis turned off, raise a ``ParameterNotProvidedError``. Use this function if the backup option -* ``set_value_of_parameter_if_undefined``: check if parameter is in the configuration dictionary. If not, if inferenceis turned on (the default), then just set the parameter's value to the ``backup_option``. If inferenceis turned off, raise a ``ParameterNotProvidedError``. +* ``get_value_from_parameter``: check if parameter is in the configuration dictionary. If not, if inference is turned on (the default), then just use the value of ``second_choice_parameter``. If inference is turned off, raise a ``ParameterNotProvidedError``. Use this function if the backup option is another parameter (rather than a fixed value). +* ``set_value_of_parameter_if_undefined``: check if parameter is in the configuration dictionary. If not, if inference is turned on (the default), then just set the parameter's value to the ``backup_option``. If inference is turned off, raise a ``ParameterNotProvidedError``. In ``e3sm_diags.py``: From 5a7b3838c9498140957bb9a586aa10f2102f622d Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Wed, 3 Jun 2026 16:34:30 -0700 Subject: [PATCH 03/27] Revisions --- .../{archive.rst => archive/index.rst} | 5 + .../source/dev_guide/archive/initial_docs.rst | 72 +++++ docs/source/dev_guide/ci.rst | 24 -- docs/source/dev_guide/contributing.rst | 143 ++------- docs/source/dev_guide/index.rst | 7 +- docs/source/dev_guide/new_diags_set.rst | 2 +- docs/source/dev_guide/new_glb_plot.rst | 13 - docs/source/dev_guide/new_task.rst | 27 +- docs/source/dev_guide/parameters.rst | 22 ++ docs/source/dev_guide/project-standards.rst | 101 +----- docs/source/dev_guide/release.rst | 298 ------------------ docs/source/dev_guide/releases/index.rst | 16 + .../releases/production_releases.rst | 81 +++++ .../dev_guide/releases/release_candidates.rst | 106 +++++++ docs/source/dev_guide/tasks/e3sm_diags.rst | 2 +- .../dev_guide/tasks/global_time_series.rst | 16 +- docs/source/index.rst | 6 +- docs/source/user_guide/campaigns.rst | 16 +- docs/source/user_guide/getting_started.rst | 277 ++++------------ docs/source/user_guide/parameters.rst | 24 +- docs/source/user_guide/tasks/index.rst | 11 +- docs/source/user_guide/tutorial.rst | 36 ++- 22 files changed, 456 insertions(+), 849 deletions(-) rename docs/source/dev_guide/{archive.rst => archive/index.rst} (96%) create mode 100644 docs/source/dev_guide/archive/initial_docs.rst delete mode 100644 docs/source/dev_guide/ci.rst delete mode 100644 docs/source/dev_guide/new_glb_plot.rst create mode 100644 docs/source/dev_guide/parameters.rst delete mode 100644 docs/source/dev_guide/release.rst create mode 100644 docs/source/dev_guide/releases/index.rst create mode 100644 docs/source/dev_guide/releases/production_releases.rst create mode 100644 docs/source/dev_guide/releases/release_candidates.rst diff --git a/docs/source/dev_guide/archive.rst b/docs/source/dev_guide/archive/index.rst similarity index 96% rename from docs/source/dev_guide/archive.rst rename to docs/source/dev_guide/archive/index.rst index 22a2486a..342eb28d 100644 --- a/docs/source/dev_guide/archive.rst +++ b/docs/source/dev_guide/archive/index.rst @@ -8,6 +8,11 @@ This page collects documentation that is obsolete or refers to older versions of ``zppy`` or the E3SM Unified environment. It is retained for historical reference. +.. toctree:: + :maxdepth: 1 + + initial_docs + Old testing tutorial (E3SM Unified 1.5.0) ========================================== diff --git a/docs/source/dev_guide/archive/initial_docs.rst b/docs/source/dev_guide/archive/initial_docs.rst new file mode 100644 index 00000000..61b0dece --- /dev/null +++ b/docs/source/dev_guide/archive/initial_docs.rst @@ -0,0 +1,72 @@ +********************************** +How to set up new documentation +********************************** + +.. warning:: + The instructions below only apply for the initial configuration of the + Sphinx documentation on the Github repository. They are documented here + for reference only. Do not follow them unless you are setting up documentation + for a new repository. (Adapted from `Sphinx documentation on GitHub + `_.) + +Create Sphinx conda environment (see above). + +Create a new git branch (gh-pages): :: + + $ git branch gh-pages + $ git checkout gh-pages + +Clear out any­thing from the main branch and start fresh :: + + $ git symbolic-ref HEAD refs/heads/gh-pages + $ rm .git/index + $ git clean -fdx + +Create documentation :: + + $ sphinx-quickstart + +accept suggested default options, except :: + + Separate source and build directories (y/N) [n]: y + +Edit Makefile and change ``BUILDIR`` :: + + BUILDDIR = docs + +Remove old build directory :: + + $ rmdir build + +Change the Sphinx theme to 'ReadTheDocs'. Edit 'source/conf.py and change :: + + html_theme = 'alabaster' + +to :: + + import sphinx_rtd_theme + html_theme = "sphinx_rtd_theme" + html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] + +Try building documentation :: + + $ make html + +Create an empty .nojekyll file to indicate to Github.com that this +is not a Jekyll static website: :: + + $ touch .nojekyll + +Create a top-level re-direction file: :: + + $ vi index.html + +with the following: :: + + + +Commit and push back to Github: :: + + $ git add . + $ git commit + $ git push origin gh-pages \ No newline at end of file diff --git a/docs/source/dev_guide/ci.rst b/docs/source/dev_guide/ci.rst deleted file mode 100644 index 4d32a5c5..00000000 --- a/docs/source/dev_guide/ci.rst +++ /dev/null @@ -1,24 +0,0 @@ -Understanding CI -======================== - -In this guide, we'll cover: - -* Build Workflow -* Release Workflow -* GitHub Actions - -Build Workflow --------------- - -The build workflow is at https://github.com/E3SM-Project/zppy/blob/main/.github/workflows/build_workflow.yml. See comments in the file for in-depth explanations of each step. - -Release Workflow ----------------- - -The release workflow is at https://github.com/E3SM-Project/zppy/blob/main/.github/workflows/release_workflow.yml. See comments in the file for in-depth explanations of each step. - -GitHub Actions --------------- -Both of these workflows are run by GitHub Actions. See https://github.com/E3SM-Project/zppy/actions. - -When a pull request is made, the build workflow is run automatically on the pushed branch. When the pull request is merged, the build workflow is once again run, but this time on the ``main`` branch. diff --git a/docs/source/dev_guide/contributing.rst b/docs/source/dev_guide/contributing.rst index 23f67682..eae6785e 100644 --- a/docs/source/dev_guide/contributing.rst +++ b/docs/source/dev_guide/contributing.rst @@ -7,8 +7,7 @@ Contributing to This Documentation Getting Started =============== -This documentation is created using -`Sphinx `_. Sphinx is an open-source tool +This documentation is created using Sphinx, which is an open-source tool that makes it easy to create intelligent and beautiful documentation, written by Georg Brandl and licensed under the BSD license. @@ -27,138 +26,34 @@ using Sphinx, you can refer to * `First Steps with Sphinx `_ * `reStructuredText Primer `_ + .. code:: -1. Make sure that you have conda in your path. On NERSC machines, you can load it with: :: + # Sphinx is included in the dev environment, + # so we just need to set one up: + cd zppy + conda clean --all --y + conda env create -f conda/dev.yml -n env-name + python -m pip install . - $ module load python/3.9-anaconda-2021.11 + # Now, we can produce the documentation locally: + cd docs + make html -2. Clone the repository and checkout a branch from ``main``: :: + # And transfer it to a web server path so we can see it online: + cp -r _build/ web_server_path/zppy_docs + chmod -R 644 web_serber_path/zppy_docs - $ cd - $ git clone https://github.com//zppy.git - $ cd zppy - $ git checkout -b main + # Optional -- if you want to generate and view versioned docs: + sphinx-multiversion source _build/html -3. Create and activate the conda development environment :: - - $ cd /zppy - $ conda env create -f conda/dev.yml - $ conda activate zppy_dev - -4. To modify the documentation, simply edit the files under ``/docs/source``. - -5. To see the changes you made to the documentation, rebuild the web pages :: - - $ cd /zppy/docs - $ make html - -6. View them locally in a web browser at ``file:////zppy/docs/_build/html/index.html``. - - - Sometimes the browser caches Sphinx docs, so you might need to delete your cache to view changes. - -7. Once you are satisfied with your modifications, commit and push them back to the repository: :: - - $ cd /zppy - $ # `/docs/_build` is ignored by git since it does not need to be pushed - $ git add . - $ git commit - $ git push - -8. <``OPTIONAL``> If you want to generate and view versioned docs: :: - - $ # After commiting to your branch - $ cd /zppy/docs - $ sphinx-multiversion source _build/html - $ # Check the `_build/html` folder for all generated versioned docs - $ # Open `_build/html//index.html` to view in browser - -The docs version selector dropdown is in the bottom left-hand corner - -9. Create a pull request from ``/zppy/branch-name`` to ``E3SM-Project/zppy/main``. - -Once this pull request is merged and GitHub Actions finishes building the docs, changes will be available on the -`zppy documentation page `_. +Pull requests should bundle together associated code changes, test changes, and documentation changes. Once you've merged your pull request and GitHub Actions finishes building the docs, changes will be visible on the +`zppy documentation page `_. How Documentation is Versioned ------------------------------ The `sphinx-multiversion `_ package manages documentation versioning. ``sphinx-multiversion`` is configured to generate versioned docs for available tags and -branches on local, ``origin`` (the likely name for your ````) and -``upstream`` (the likely name for your ````).. - -Branches or tags that don’t contain both the sphinx ``source`` directory and the ``conf.py`` file will be skipped automatically. +branches. Ones that don’t contain both the sphinx ``source`` directory and the ``conf.py`` file will be skipped automatically. - Run ``sphinx-multiversion source _build/html --dump-metadata`` to see which tags/branches matched. - -Initial setup (obsolete/for reference only) -=========================================== - -The instructions below only apply for the initial configuration of the -Sphinx documentation on the Github repository. They are documented here -for reference only. Do not follow them unless you are setting up documentation -for a new repository. (Adapted from `Sphinx documentation on GitHub -`_.) - -Create Sphinx conda environment (see above). - -Create a new git branch (gh-pages): :: - - $ git branch gh-pages - $ git checkout gh-pages - -Clear out any­thing from the main branch and start fresh :: - - $ git symbolic-ref HEAD refs/heads/gh-pages - $ rm .git/index - $ git clean -fdx - -Create documentation :: - - $ sphinx-quickstart - -accept suggested default options, except :: - - Separate source and build directories (y/N) [n]: y - -Edit Makefile and change ``BUILDIR`` :: - - BUILDDIR = docs - -Remove old build directory :: - - $ rmdir build - -Change the Sphinx theme to 'ReadTheDocs'. Edit 'source/conf.py and change :: - - html_theme = 'alabaster' - -to :: - - import sphinx_rtd_theme - html_theme = "sphinx_rtd_theme" - html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] - -Try building documentation :: - - $ make html - -Create an empty .nojekyll file to indicate to Github.com that this -is not a Jekyll static website: :: - - $ touch .nojekyll - -Create a top-level re-direction file: :: - - $ vi index.html - -with the following: :: - - - -Commit and push back to Github: :: - - $ git add . - $ git commit - $ git push origin gh-pages - diff --git a/docs/source/dev_guide/index.rst b/docs/source/dev_guide/index.rst index b05a461c..b33e20cf 100644 --- a/docs/source/dev_guide/index.rst +++ b/docs/source/dev_guide/index.rst @@ -8,14 +8,13 @@ This guide covers everything needed to develop, test, and release ``zppy``. :maxdepth: 2 project-standards - ci contributing + parameters tasks/index testing release_testing - release + releases/index.rst new_task new_diags_set - new_glb_plot tutorial_testing_e3sm_unified - archive + archive/index diff --git a/docs/source/dev_guide/new_diags_set.rst b/docs/source/dev_guide/new_diags_set.rst index c4386bcc..67f157c9 100644 --- a/docs/source/dev_guide/new_diags_set.rst +++ b/docs/source/dev_guide/new_diags_set.rst @@ -1,5 +1,5 @@ ********************************************* -Adding a new set to the E3SM Diagnostics task +Examples: Adding a new set to the E3SM Diagnostics task ********************************************* This guide gives a general list of things to consider when adding a new diff --git a/docs/source/dev_guide/new_glb_plot.rst b/docs/source/dev_guide/new_glb_plot.rst deleted file mode 100644 index 4ab8fd45..00000000 --- a/docs/source/dev_guide/new_glb_plot.rst +++ /dev/null @@ -1,13 +0,0 @@ -*************************************** -Adding a new plot to Global Time Series -*************************************** - -This guide gives a general list of things to consider when adding a new -Global Time Series plot. The exact code changes required will differ amongst plots. - -Note that unlike E3SM Diags and MPAS-Analysis which are packages called by ``zppy``, Global Time Series is built into ``zppy``. - -- In `coupled_global.py `_: Add variables under "# Variables to extract". Add the code for your new plot under "Plotting functions"; the name of your function should start with ``plot_``. In ``PLOT_DICT``, add ``"function name without plot_" : function name``. -- If this plot should be added to the defaults, then complete this step: for ``plot_names`` under ``[global_time_series]`` in `default.ini `_, add the function name without plot to the string. -- Optionally, after making your edits, you can run ``pre-commit run --all-files`` to clean up your code and check for any obvious errors. -- Run the `integration tests `_ and examine the differences from the expected files. If they match what you expect, update the expected files following "Commands to run to replace outdated expected files" on the machine-specific directions. (The commands to run all the integration tests are ``pip install .`` followed by ``pytest tests/integration/test_*.py``). diff --git a/docs/source/dev_guide/new_task.rst b/docs/source/dev_guide/new_task.rst index dc160ce5..0d43a5a1 100644 --- a/docs/source/dev_guide/new_task.rst +++ b/docs/source/dev_guide/new_task.rst @@ -13,25 +13,16 @@ Some key parts, however, are displayed below: .. code:: #!/bin/bash - {% include 'slurm_header.sh' %} + {% include 'inclusions/slurm_header.sh' %} + {% include 'inclusions/boilerplate.bash' %} + set -e {{ environment_commands }} + set +e - # Turn on debug output if needed - debug={{ debug }} - if [[ "${debug,,}" == "true" ]]; then - set -x - fi - - # Script dir - cd {{ scriptDir }} - - # Get jobid - id=${SLURM_JOBID} - - # Update status file - STARTTIME=$(date +%s) - echo "RUNNING ${id}" > {{ prefix }}.status + set_pkg_manager + echo "${pkg_manager} list package-this-task-calls:" + ${pkg_manager} list package-this-task-calls || true # If we can't print this, just continue on. .. code:: @@ -159,8 +150,8 @@ Add the task to ``zppy/__main__.py``: .. code:: - # tasks - (config, scriptDir) + # task-name tasks + existing_bundles = task-name(config, script_dir, existing_bundles, job_ids_file) Update defaults =============== diff --git a/docs/source/dev_guide/parameters.rst b/docs/source/dev_guide/parameters.rst new file mode 100644 index 00000000..93935e78 --- /dev/null +++ b/docs/source/dev_guide/parameters.rst @@ -0,0 +1,22 @@ +.. _parameters_for_developers: + +************************************************ +Parameter checking & inferring -- for developers +************************************************ + +There are many parameter-handling functions. + +In ``utils.py``: + +* ``get_value_from_parameter``: check if parameter is in the configuration dictionary. If not, if inference is turned on (the default), then just use the value of ``second_choice_parameter``. If inference is turned off, raise a ``ParameterNotProvidedError``. Use this function if the backup option is another parameter (rather than a fixed value). +* ``set_value_of_parameter_if_undefined``: check if parameter is in the configuration dictionary. If not, if inference is turned on (the default), then just set the parameter's value to the ``backup_option``. If inference is turned off, raise a ``ParameterNotProvidedError``. + +In ``e3sm_diags.py``: + +* ``check_parameter_defined``: check if parameter is in the configuration dictionary, and if not raise a ``ParameterNotProvidedError``. +* ``check_set_specific_parameter``: if any requested ``e3sm_diags`` sets require this parameter, make sure it is present. If not, raise a ``ParameterNotProvidedError``. +* ``check_parameters_for_bash``: use ``check_set_specific_parameter`` to check the existence of parameters that aren't used until the bash script. +* ``check_mvm_only_parameters_for_bash``: similar, but these are specifically parameters used for model-vs-model runs. Uses ``check_parameter_defined`` in addition to ``check_set_specific_parameter``. +* ``check_and_define_parameters``: make sure all parameters are defined, using ``utils.py get_value_from_parameter``, ``utils.py set_value_of_parameter_if_undefined``, and ``check_mvm_only_parameters_for_bash``. + +``check_parameters_for_bash`` can be run immediately for each subtask because it has very few conditions. Other checks are included in ``check_and_define_parameters`` later on in the code. \ No newline at end of file diff --git a/docs/source/dev_guide/project-standards.rst b/docs/source/dev_guide/project-standards.rst index 9596d1fc..3738a403 100644 --- a/docs/source/dev_guide/project-standards.rst +++ b/docs/source/dev_guide/project-standards.rst @@ -21,15 +21,15 @@ Guidelines for VC 3. **Rebase** with ``main`` to avoid/resolve conflicts 4. Make sure ``pre-commit`` checks pass when committing (enforced in CI/CD build) 5. Open a pull-request (PR) early for discussion -6. Once the CI/CD build passes and PR is approved, **squash and rebase** your - commits -7. Merge PR into ``main`` and **delete the branch** +6. Once the CI/CD build passes and PR is approved, the PR is ready to merge. +7. Merge PR into ``main``. For small changes, use the "Squash and merge" option. For big changes with many distinct commits, or commits contributed by different developers, use the "Create a merge commit" option. +8. Delete the branch on GitHub and in your local workspace. Things to Avoid in VC ~~~~~~~~~~~~~~~~~~~~~ 1. Don't merge in broken or commented out code 2. Don't commit onto ``main`` directly -3. Don't merge with conflicts (handle conflicts upon rebasing) +3. Don't merge with conflicts (handle conflicts by rebasing: ``git rebase upstream/main``) Source: https://gist.github.com/jbenet/ee6c9ac48068889b0912 @@ -47,6 +47,16 @@ at the commit level before submitting code reviews. Helpful ``pre-commit`` Commands ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +``zppy`` uses the following pre-commit checks: + +* **trailing-whitespace** - Trims trailing whitespace from the end of each line. +* **end-of-file-fixer** - Ensures files end with a single newline character. +* **check-yaml** - Validates YAML files for correct syntax. +* **black** - Auto-formats Python code to a consistent style using the `black `__ formatter. +* **isort** - Sorts and organizes Python import statements alphabetically and by type using `isort `__. +* **flake8** - Lints Python code for style violations and common errors (PEP 8) using `flake8 `__. +* **mypy** - Runs static type checking on Python code using type annotations using `mypy `__. + Install into your cloned repo :: conda activate zppy_dev @@ -71,88 +81,13 @@ Run individual hook :: # Available hook ids: trailing-whitespace, end-of-file-fixer, check-yaml, black, isort, flake8, mypy pre-commit run -Squash and Rebase Commits -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Before you merge a support branch back into ``main``, the branch is typically -squashed down to a single* buildable commit, and then rebased on top of the main repo's ``main`` branch. - -\* *In some cases, it might be logical to have multiple squashed commits, as long as each commit passes the CI/CD build* - -Why squash and rebase commits? - -- Ensures build passes from the commit -- Cleans up Git history for easy navigation -- Makes collaboration and review process more efficient -- Makes handling conflicts from rebasing simple since there are fewer commits to fix conflicts on -- Makes ``git bisect`` easier and more effective to use. For example, it will show the exact commit that introduced a bug since the commit contains a relatively small changeset. On the otherhand, a merge commit bringing in many commits makes it much harder since it includes a large changeset. A changeset is a collection of commits that are logically grouped together. - - -How to squash and rebase commits -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Assuming that you followed :ref:`"(b) Development Environment" `: - -1. Sync ``main`` with the main repo's ``main`` :: - - git checkout main - git rebase /main - git push -f main - -2. Get the SHA of the commit OR number of commits to rebase to :: - - git log --graph --decorate --pretty=oneline --abbrev-commit - -3. Squash commits:: - - git rebase -i [SHA] - - # OR - - git rebase -i HEAD~[NUMBER OF COMMITS] - -4. Make sure your squashed commit messages are refined - -5. Rebase branch onto ``main`` :: - - git checkout - git rebase main - git push -f - -6. Force push to remote branch :: - - # You have to force push because the rebase rewrites the commit SHAs - git push -f - -Source: -https://blog.carbonfive.com/always-squash-and-rebase-your-git-commits/ - -Code Quality Assurance ----------------------- - -This project uses several tools for code formatting, linting, and type checking listed below. - -- Code Formatting: `black `__ -- Linting: `flake8 `__, `isort `__ -- Optional Type Checking: `mypy `__ - -You can run them as hooks manually/automatically when committing using ``pre-commit``, or manually through the terminal or IDE/text editor. - -Helpful Commands -~~~~~~~~~~~~~~~~ - -Run a tool - :: - - # Available tool names: black, flake8, isort, mypy - . .. _ci-cd: Continuous Integration / Continuous Delivery (CI/CD) ---------------------------------------------------- -This project uses `GitHub Actions `_ to run two CI/CD workflows. +This project uses `GitHub Actions `_ to run two CI/CD workflows. The workflows are defined `here `__. 1. CI/CD Build Workflow @@ -164,6 +99,8 @@ This project uses `GitHub Actions 2. Run test suite in a conda environment 3. Publish latest ``main`` docs (only on ``push``) + When a pull request is made, the build workflow is run automatically on the pushed branch. When the pull request is merged, the build workflow is once again run, but this time on the ``main`` branch. + 2. CI/CD Release Workflow This workflow is triggered by the Git ``publish`` event, which occurs when a new release is tagged. @@ -173,10 +110,6 @@ This project uses `GitHub Actions 1. Publish new release docs 2. Publish Anaconda package -Style Guide ------------ - -This project follows the Black code style. Please read about it more `here `__. API Documentation ----------------- diff --git a/docs/source/dev_guide/release.rst b/docs/source/dev_guide/release.rst deleted file mode 100644 index 409c8832..00000000 --- a/docs/source/dev_guide/release.rst +++ /dev/null @@ -1,298 +0,0 @@ -.. _release-zppy: - -How to Prepare a Release -======================== - -This guide covers the full release process for both release candidates (RCs) -and production releases. It includes: - -- `Preparing the Documentation`_ -- `Bumping the Version`_ -- `Releasing on GitHub: production releases`_ -- `Releasing on conda-forge: production releases`_ -- `Creating a New Version of the Documentation`_ - -References: - -- Release notes are tracked in `GitHub Discussions - `_ -- Releases page: https://github.com/E3SM-Project/zppy/releases -- Tags page: https://github.com/E3SM-Project/zppy/tags - -Preparing the Documentation ---------------------------- - -This step is **only required for production releases**, not release -candidates. - -1. Checkout a branch: - - .. code-block:: bash - - cd ~/ez/zppy - git status # confirm no uncommitted changes - git fetch upstream main - git checkout -b prepare-docs-for- upstream/main - -2. Update ``docs/source/user_guide/parameters.rst``: find the link to - ``this release's parameter defaults`` and replace the commit hash in - the URL with the hash of the latest commit on ``main``. You can find - this at https://github.com/E3SM-Project/zppy/commits/main. - - Before: - :: - - See `this release's parameter defaults /zppy/defaults/default.ini>`_ - - After: - :: - - See `this release's parameter defaults /zppy/defaults/default.ini>`_ - -3. Run pre-commit and commit: - - .. code-block:: bash - - conda activate - pre-commit run --all-files # should pass - git add -A - git commit -m "Update parameter documentation pointer for " - git push upstream prepare-docs-for- - -4. Create a PR, add the **Documentation** label, merge, and delete the - branch on GitHub. - -Bumping the Version -------------------- - -Determining the new version number -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Review commits since the last release at -https://github.com/E3SM-Project/zppy/commits/main to determine whether the -bump should be major, minor, or patch: - -- **Major** (X.0.0): breaking changes to the user-facing API -- **Minor** (x.Y.0): new features that are backward-compatible -- **Patch** (x.y.Z): bug fixes only - -Also update the E3SM Unified package table on `Confluence -`_: - -.. code-block:: text - - E3SM Unified : zppy v - -.. _release-candidates: - -Release candidates (zppy repo) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1. Checkout a branch named after the new version: - - .. code-block:: bash - - cd ~/ez/zppy - git status # confirm no uncommitted changes - git fetch upstream main - git checkout -b vrc upstream/main - git log --oneline | head -n 3 - # Verify matches https://github.com/E3SM-Project/zppy/commits/main - -2. Bump the version using ``tbump``: - - .. code-block:: bash - - conda activate # must have tbump installed - tbump rc --no-tag - # "Error: Command `git push upstream main` failed" is expected - # (we are not on main) - -3. Push and create a PR: - - .. code-block:: bash - - git push upstream vrc - - Create a PR, add the **Update version** label, merge, and delete the - branch on GitHub. - -4. Create the tag on ``main``: - - .. code-block:: bash - - git checkout main - git fetch upstream - git reset --hard upstream/main - git tag -a vrc -m "vrc" - # Delete the local branch first (avoids push conflicts) - git branch -D vrc - git push upstream vrc - - The tag will appear on https://github.com/E3SM-Project/zppy/tags but - **not** on Releases. This is expected for RCs. - -Production releases (zppy repo) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -1. Complete `Preparing the Documentation`_ first. - -2. Checkout a branch: - - .. code-block:: bash - - cd ~/ez/zppy - git status # confirm no uncommitted changes - git fetch upstream main - git checkout -b v upstream/main - git log --oneline | head -n 3 - -3. Bump the version: - - .. code-block:: bash - - tbump --no-tag - # "Error: Command `git push upstream main` failed" is expected - git push upstream v - -4. Create a PR, compare the diff against the last RC's bump PR to verify - correctness, add the **Update version** label, wait for CI checks to - pass, merge, and delete the branch. - -.. _github-release: - -Releasing on GitHub: release candidates ------------------------------------------ - -See the tagging step above in `Release candidates (zppy repo)`_. RC tags -appear on Tags but not on Releases. - -Releasing on GitHub: production releases ------------------------------------------- - -1. Go to https://github.com/E3SM-Project/zppy/releases and click **Draft a - new release**. -2. Set **Tag version** to ``v`` (include the ``v``). Set - **@ Target** to ``main``. -3. Set **Release title** to ``v`` (include the ``v``). -4. Write the release description: - - - **Summary of changes**: a high-level summary of what changed - - **Full list of changes**: categorized log of commits, organized by - type (new features, bug fixes, documentation, etc.) - - You can use https://github.com/E3SM-Project/zppy/commits/main to find - all commits since the last release. - -5. Ensure **Set as the latest release** is checked. -6. Click **Publish release**. - - - The CI/CD release workflow is automatically triggered. It publishes - new docs and, for conda-forge, triggers a bot PR. - - Unlike RCs, the production version now appears on both - `Tags `_ and - `Releases `_. - -Releasing on conda-forge: release candidates ---------------------------------------------- - -The conda-forge bot does **not** handle RCs automatically. Do this manually: - -1. If you don't have a local clone of the feedstock: - - .. code-block:: bash - - git clone git@github.com:conda-forge/zppy-feedstock.git - git remote add upstream git@github.com:conda-forge/zppy-feedstock.git - -2. If you don't have a fork, create one on - https://github.com/conda-forge/zppy-feedstock, then: - - .. code-block:: bash - - git remote add - -3. Get the sha256 hash of the RC tarball: - - .. code-block:: bash - - curl -sL https://github.com/E3SM-Project/zppy/archive/vrc.tar.gz | openssl sha256 - -4. Make changes on a branch: - - .. code-block:: bash - - git fetch upstream dev - git checkout -b vrc upstream/dev - - In ``recipe/recipe.yaml`` (or ``recipe/meta.yaml`` if it still exists), - update: - - .. code-block:: yaml - - version: rc - sha256: - number: 0 # build number should always be 0 for a new version - - Also update any changed dependencies (compare - ``https://github.com/E3SM-Project/zppy/compare/v...v`` - for changes in ``dev.yml`` or ``pyproject.toml``). - - .. code-block:: bash - - git add -A - git commit -m "vrc" - git push vrc - -5. Open a PR from your fork to the ``dev`` branch of - https://github.com/conda-forge/zppy-feedstock. RC packages get the - ``zppy_dev`` label. - -6. After merging and CI completes, check - https://anaconda.org/conda-forge/zppy for the new package (allow ~15 - minutes for mirroring). - -Releasing on conda-forge: production releases ---------------------------------------------- - -1. Complete :ref:`Releasing On GitHub ` first. This - triggers the CI/CD workflow that opens the bot PR on conda-forge. - -2. Wait for the bot PR at https://github.com/conda-forge/zppy-feedstock/pulls - (can take 1 hour to 1 day). Alternatively, post an issue with - ``@conda-forge-admin, please update version``. - -3. Complete any requirements to merge the PR (re-render if requested, etc.). - -4. Check https://anaconda.org/conda-forge/zppy/files to confirm the new - package has the ``main`` label. - -5. Notify maintainers of the E3SM Unified environment on the `Confluence - packages page - `_ - about the new release. Update the version number for the correct E3SM - Unified version(s) under "Next versions". - -Creating a New Version of the Documentation -------------------------------------------- - -1. Complete :ref:`Releasing On GitHub `. This triggers - the CI/CD release workflow, which publishes new versioned docs. - -2. Wait for the CI/CD build to succeed at - https://github.com/E3SM-Project/zppy/actions. - -3. Verify the documentation is available at - https://docs.e3sm.org/zppy/_build/html/main/ - -4. Check that the "parameter defaults" link on the parameters page points - to the correct commit hash (should match the commit before "Update - parameter documentation pointer" on - https://github.com/E3SM-Project/zppy/commits/main). - -Extra Resources ---------------- - -- Conda-forge docs: https://conda-forge.org/docs/user/introduction.html -- Admin web services: https://conda-forge.org/docs/maintainer/infrastructure.html#admin-web-services -- E3SM Anaconda release guide: https://acme-climate.atlassian.net/wiki/spaces/IPD/pages/3616735236/Releasing+E3SM+Software+on+Anaconda+conda-forge+channel diff --git a/docs/source/dev_guide/releases/index.rst b/docs/source/dev_guide/releases/index.rst new file mode 100644 index 00000000..149aa8aa --- /dev/null +++ b/docs/source/dev_guide/releases/index.rst @@ -0,0 +1,16 @@ +How to Prepare a Release +======================== + + +.. toctree:: + :maxdepth: 1 + + release_candidates + production_releases + +Extra Resources +--------------- + +- Conda-forge docs: https://conda-forge.org/docs/user/introduction.html +- Admin web services: https://conda-forge.org/docs/maintainer/infrastructure.html#admin-web-services +- E3SM Anaconda release guide: https://acme-climate.atlassian.net/wiki/spaces/IPD/pages/3616735236/Releasing+E3SM+Software+on+Anaconda+conda-forge+channel diff --git a/docs/source/dev_guide/releases/production_releases.rst b/docs/source/dev_guide/releases/production_releases.rst new file mode 100644 index 00000000..eb1d30a4 --- /dev/null +++ b/docs/source/dev_guide/releases/production_releases.rst @@ -0,0 +1,81 @@ +.. _production_releases: + +Preparing a production release +============================== + +Step 1: Testing +--------------- + +Be sure to run the entire integration test suite before making a production release. + +Step 2: Confluence +------------------ + +This step should already have been completed during the release-candidate phase, however it is good practice to double check that the `E3SM Unified version tracking page `__ has had the next E3SM Unified version updated with the new ``zppy`` version number. + +Step 3: Prepare documentation +------------------------------------------ + +Similar to the release-candidate directions, we'll use ``v1.2.3`` as an example verison number here. + + .. code-block:: bash + + cd zppy + git status # Confirm there's no uncommitted changes + git fetch upstream main # This assumes you've named your remote for the main repo as "upstream" + git checkout -b prepare-docs-for-1.2.3 upstream/main + + +Change the "this release's parameter defaults" link on the Parameters documentation page (``docs/source/user_guide/parameters.rst``) to refer to the latest commit's hash. See `zppy commits `_ to find the latest commit. + + .. code-block:: bash + + git add -A + conda activate env-name # Activate any zppy dev environment you have; we just need the pre-commit hooks + conda activate env-name + pre-commit run --all-files + git commit -m "Update parameter documentation pointer for v1.2.3" + git push upstream prepare-docs-for-1.2.3 + + +Create a PR, mark yourself as assignee, add the Documentation label, merge the PR and delete the GitHub branch. + +Step 4: tbump +------------- + + .. code-block:: bash + + # On Perlmutter + cd zppy + git status # Confirm there's no uncommitted changes + git fetch upstream main # This assumes you've named your remote for the main repo as "upstream" + git checkout -b v1.2.3 upstream/main + git log --oneline | head -n 5 + # Check that the latest commits match what's on https://github.com/E3SM-Project/zppy/commits/main/ + tbump 1.2.3 --no-tag + # This creates a commit, but won't push it (because the branch isn't named `main`) + git push upstream v1.2.3 + # Create, and "Update version" label" to, and merge the PR; delete the branch on GitHub + +Step 5: Make the release on the zppy repo +----------------------------------------- + +1. Draft a new release `here `_. Click "Draft a new release". +2. Set Tag version to ``v1.2.3``, including the “v”. ``@Target`` should be ``main``. Click "Tag", then "Create new tag" and enter "v.1.2.3" +3. Set Release title to ``v1.2.3``, including the “v”. +4. Use "Describe this release" to summarize the changelog. Write two sections: "Summary of changes" (the high-level summary) & "Full list of changes" (the categorized list of commits, from reviewing the `zppy commits `_) +5. Make sure "Set as the latest release" is checked. +6. Click "Publish release". Unlike the RCs, `v1.2.3` should now appear on _both_ `Tags `_ and `Releases `_. +7. CI/CD release workflow will be automatically triggered. The docs workflow is just for the docs. Clicking "Publish release" is responsible for triggering the bot PR on conda-forge. + +Step 6: zppy-feedstock repo +--------------------------- + +1. Wait for a bot PR to come up automatically on conda-forge after the GitHub release. This can happen anywhere from 1 hour to 1 day later. Check https://github.com/conda-forge/zppy-feedstock/pulls. (Alternative: open an issue with the bot command: ``@conda-forge-admin, please update version``` and the PR will opened). +2. Complete any requirements to merge the PR. +3. Check the https://anaconda.org/conda-forge/zppy/files/manage page to view the newly updated package. Check it has the `main` label. + +Step 7: Check the docs +---------------------- + +Wait for the docs workflow to complete successfully. Then go to https://docs.e3sm.org/zppy/_build/html/main/user_guide/parameters.html, and click on "this release’s parameter defaults". That should take you the code at the hash as of the latest commit. diff --git a/docs/source/dev_guide/releases/release_candidates.rst b/docs/source/dev_guide/releases/release_candidates.rst new file mode 100644 index 00000000..7be09faa --- /dev/null +++ b/docs/source/dev_guide/releases/release_candidates.rst @@ -0,0 +1,106 @@ +.. _release_candidates: + +Preparing a release candidate +============================= + +Step 1: Testing +--------------- + +Be sure to run the entire integration test suite before making a release candidate. + +Step 2: Determine what the new version should be +------------------------------------------------ + +This step only needs to be done for rc1. + +Review `zppy commits `__. Identify what changes since the last production release would require a patch, minor, or major version update (as in ``vMAJOR.MINOR.PATCH``). + +The highest-level version update will be used. For example, if you found 1 commit requiring a patch update, 1 commit requiring a minor update, and 0 commits requiring a major update, then you'd want to increment the minor version. + +Step 3: Confluence +------------------ + +This step only needs to be done for rc1. + +Update the `E3SM Unified version tracking page `__ with the new ``zppy`` version number. Be sure to update the section for the next E3SM Unified environment, not the current one. + +Step 4: tbump +------------- + +In this example, ``v1.2.3rc4`` is used -- so, the major version is 1, the minor version is 2, and the patch version is 3. The release candidate number is 4. + + .. code-block:: bash + + cd zppy + git status # Make sure you don't have any uncommitted changes + git fetch upstream main # This assumes you've named your remote for the main repo as "upstream" + git checkout -b v1.2.3rc4 upstream/main + conda activate env-name # Activate any zppy dev environment you have; we just need `tbump` + tbump 1.2.3rc4 --no-tag + # This creates a commit, but won't push it (because the branch isn't named `main`) + git diff HEAD^ HEAD | cat + # Review the change `tbump`` added + git push upstream v1.2.3rc4 + # Create, and "Update version" label" to, and merge the PR; delete the branch on GitHub + +Step 5: Tag the RC on the zppy repo +----------------------------------------- + + .. code-block:: bash + + git checkout main + git fetch upstream + git reset --hard upstream/main + git tag -a v1.2.3rc4 -m "v1.2.3rc4" # Add the tag for this RC + # Delete the branch from the tbump step. Otherwise, the push command won't work. + git branch -D v1.2.3rc4 + git push upstream v1.2.3rc4 # Push the new RC tag to GitHub + +``v1.2.3rc4`` should now appear on `Tags `__, but _not_ on `Releases `__. + +Step 6: zppy-feedstock repo +--------------------------- + +If you don't have a fork of zppy-feedstock, first go to the `zppy-feedstock_repo `__, and click the "Fork" button in the top right. + + .. code-block:: bash + + # If you don't have a clone of zppy-feedstock, first run: + # Clone using the SSH from the green "Code" button in the top right of the repo home page. + git clone git@github.com:conda-forge/zppy-feedstock.git + cd zppy-feedstock + git remote -v # See your remotes + # Optional; this lets you use upstream to refer to the main repo: + git remote add upstream git@github.com:conda-forge/zppy-feedstock.git + # Required; this allows you to push to your fork: + # Copy the SSH path from your fork's green "Code" button and run: + git remote add your-fork-name git@github.com:your-fork-name/zppy-feedstock.git + + # If you already have your clone set up, just run: + cd zppy-feedstock + + # In all cases: + curl -sL https://github.com/E3SM-Project/zppy/archive/v1.2.3rc4.tar.gz | openssl sha256 + # SHA2-256(stdin)= long hex string + git status # Check for uncommitted changes + git fetch upstream dev # Make sure you fetch the dev branch, not the main branch! + git checkout -b v1.2.3rc4 upstream/dev + emacs recipe/recipe.yaml + # Update the version and sha256 (and the build number if needed): + # version: 1.2.3.rc4 + # sha256: ... # The sha256 from a few commands earlier + # number: 0 # build >>> number should always be 0 + + # Check the diff since the last release/RC. Examples: + # - https://github.com/E3SM-Project/zppy/compare/v3.1.0...v3.2.0rc1 + # - https://github.com/E3SM-Project/zppy/compare/v1.2.3rc3...v1.2.3rc4 + # If there are changes in dependencies there (e.g., in `conda/dev.yml`, `pyproject.toml`), + # You'll want to include them in this feedstock PR too. + + git add -A + git commit -m "v1.2.3rc4" + git push your-fork-name v1.2.3rc4 + +Then, create a pull request to the ``dev`` branch. Do _not_ set the pull request to merge to ``main``, as that will create a production release! RC packages get the ``zppy_dev`` label. + +Follow any further directions given by the ``conda-forge/zppy-feedstock`` PR template. Once that PR is reviewed and merged. Once CI completes, check https://anaconda.org/conda-forge/zppy for the new package (allow ~15 minutes for mirroring). diff --git a/docs/source/dev_guide/tasks/e3sm_diags.rst b/docs/source/dev_guide/tasks/e3sm_diags.rst index f3920ed9..bf3493b6 100644 --- a/docs/source/dev_guide/tasks/e3sm_diags.rst +++ b/docs/source/dev_guide/tasks/e3sm_diags.rst @@ -44,7 +44,7 @@ The specific subtasks to depend on are set via: **Downstream:** -- :doc:`global_time_series` — optionally +- None. Adding a new diagnostic set ----------------------------- diff --git a/docs/source/dev_guide/tasks/global_time_series.rst b/docs/source/dev_guide/tasks/global_time_series.rst index 473add1c..d97303fc 100644 --- a/docs/source/dev_guide/tasks/global_time_series.rst +++ b/docs/source/dev_guide/tasks/global_time_series.rst @@ -7,22 +7,10 @@ Implementation -------------- - **Python module**: ``zppy/global_time_series.py`` -- **Template**: ``zppy/templates/global_time_series.bash`` and - ``zppy/templates/coupled_global.py`` +- **Template**: ``zppy/templates/global_time_series.bash`` -Unlike ``e3sm_diags`` and ``mpas_analysis``, the Global Time Series -functionality is **built into** ``zppy`` rather than calling an external -package. The main plotting logic lives in ``coupled_global.py``. -Adding new plots -~~~~~~~~~~~~~~~~ - -See :doc:`../new_glb_plot` for a step-by-step guide to adding new global -time series plots. - -The plot registry is in ``PLOT_DICT`` in ``coupled_global.py``. Each entry -maps a short name (used in ``plots_original``) to a function named -``plot_``. +``global_time_series`` is called through the ``zppy-interfaces`` entry-point. Dependencies ------------ diff --git a/docs/source/index.rst b/docs/source/index.rst index 43727deb..7f9a8e06 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -9,7 +9,7 @@ Zppy documentation What is zppy? ============= -``zppy`` (pronounced "zippee") is a **workflow manager** for post-processing +``zppy`` (pronounced "/ˈzɪpi/") is a **workflow manager** for post-processing E3SM simulations, written in Python. The goal of ``zppy`` is to speed up post-processing by automating commonly performed tasks. @@ -22,13 +22,13 @@ these tools, especially for long simulations. ``zppy`` is controlled entirely from a single user-provided configuration file (e.g., ``mysimulation.cfg``). This configuration file specifies input and output directories, as well as the list of tasks to run. Each task operates -for a specific model component and over a specified number of years—for +over a specified number of years—for example, generating atmosphere climatology files every 20 and 50 years. ``zppy`` parses the configuration file and generates batch jobs that are submitted for execution by SLURM. Dependencies between tasks are handled and passed to SLURM. Internally, each batch job is created by instantiating -`Jinja `_ template scripts (usually +Jinja2 template scripts (usually written in bash, but other languages are supported as well). .. toctree:: diff --git a/docs/source/user_guide/campaigns.rst b/docs/source/user_guide/campaigns.rst index d4c70287..a2cbcda6 100644 --- a/docs/source/user_guide/campaigns.rst +++ b/docs/source/user_guide/campaigns.rst @@ -7,17 +7,19 @@ Campaigns The ``campaign`` parameter allows users to choose a template of default values for their ``zppy`` run. Inheritance in configuration files still works with campaigns, so users may override values from a campaign's configuration file. -Campaign configuration files can be found in the `zppy/zppy/templates directory `_. ``zppy`` currently supports the following campaigns: +Campaign configuration files can be found in the `zppy/defaults directory `_. ``zppy`` currently supports the following campaigns: - cryosphere.cfg - high_res_v1.cfg - water_cycle.cfg -High-resolution analysis (``campaign = "high_res_v1"``) is only appropriate for some simulations -(e.g., those with the oRRS18to6 ocean and sea-ice mesh). -It will only work if you: -- have access to the `bigmem nodes on NERSC Cori `_ -- first load the necessary module with ``module load cmem``. +.. warning:: + High-resolution analysis (``campaign = "high_res_v1"``) is only appropriate for some simulations + (e.g., those with the oRRS18to6 ocean and sea-ice mesh). + It will only work if you: -Support for Perlmutter has not been confirmed for this campaign type. + - have access to the `bigmem nodes on NERSC Cori `_ + - first load the necessary module with ``module load cmem``. + + Perlmutter support has not yet been implemented. This is documented in https://github.com/E3SM-Project/zppy/issues/355 diff --git a/docs/source/user_guide/getting_started.rst b/docs/source/user_guide/getting_started.rst index 60e53d61..2ad80eb2 100644 --- a/docs/source/user_guide/getting_started.rst +++ b/docs/source/user_guide/getting_started.rst @@ -7,267 +7,105 @@ Getting started Activate e3sm_unified environment ================================= -If you have an account on one of the E3SM supported machines (NERSC, Compy, Acme1, -LCRC, Cooley, Rhea), you can access ``zppy`` by activating ``e3sm_unified``, which is +If you have an account on one of the E3SM supported machines, you can access ``zppy`` by activating ``e3sm_unified``, which is a conda environment that pulls together Python and other E3SM tools such as ``e3sm_diags`` and ``zstash``. -The paths to ``e3sm_unified`` activation scripts are machine dependent: +The paths to ``e3sm_unified`` activation scripts are machine dependent. As of E3SM Unified 1.13.0, the supported machines and their corresponding activation scripts are: -**Compy** +**Andes** :: - source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh - + source /ccs/proj/cli115/software/e3sm-unified/load_latest_e3sm_unified_andes.sh -**Perlmutter-CPU** +**Aurora** :: - source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh - -**Anvil** - :: - - source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_anvil.sh + source /lus/flare/projects/E3SMinput/soft/e3sm-unified/load_latest_e3sm_unified_aurora.sh **Chrysalis** :: source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh - -**Cooley** +**Compy** :: - source /lus/theta-fs0/projects/ccsm/acme/tools/e3sm-unified/load_latest_e3sm_unified_cooley.sh - + source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh -**acme1** +**Dane** :: - source /p/user_pub/e3sm_unified/envs/load_latest_e3sm_unified_acme1.sh - + source /usr/workspace/e3sm/apps/e3sm-unified/load_latest_e3sm_unified_dane.sh -**Andes** +**Frontier** :: - source /gpfs/alpine/proj-shared/cli115/e3sm-unified/load_latest_e3sm_unified_andes.sh - - -Change ``.sh`` to ``.csh`` for ``csh`` shells. - -Note that ``e3sm_unified``'s development cycle is not in phase with ``zppy``. -Therefore the version of ``zppy`` included may not be the latest. -To install the latest stable release, refer to the following: - -.. _conda_environment: - -Installation in a Conda Environment -=================================== - -If the E3SM Unified environment doesn't serve your needs, you can alternatively -install the latest version in your own custom conda environment. - -First, activate conda or install it if it's not available. Details vary amongst machines. - -.. _conda_environment_others: - -Others/Local ------------- - -If the system doesn't come with conda pre-installed, follow these instructions: - -1. Download Mambaforge - - Linux - :: - - wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh - - MacOS x86_64 (note that ``zppy`` is not supported on MacOS, but it may be useful to contribute to the documentation on MacOS) - :: - - wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-MacOSX-x86_64.sh - -2. Install Mambaforge - - Linux - :: - - bash ./Mambaforge-Linux-x86_64.sh - - - MacOS x86_64 - :: - - bash ./Mambaforge-MacOSX-x86_64.sh - - When you see: :: - - by running conda init? [yes|no] - [no] >>> yes - - respond with ``yes`` so ``conda`` and ``mamba`` commands are available on - initializing a new bash terminal. - -3. If you are working on a machine/network that intercepts SSL communications (such as -acme1), you will get an SSL error unless you disable the SSL verification: + source /ccs/proj/cli115/software/e3sm-unified/load_latest_e3sm_unified_frontier.sh +**Perlmutter (login or CPU nodes)** :: - conda config --set ssl_verify false - binstar config --set ssl_verify False - -4. Once conda and mamba are properly working, you can install the **(a) Latest Stable Release** or -create a **(b) Development Environment**. - -(a) Latest Stable Release -========================= - -Installation using mamba ------------------------- - -First, make sure that you're using ``bash``. :: - - bash - -You must have a conda base enviornment installed as well. -See :ref:`"Installation in a Conda Environment" ` section above for -installing conda. - -These steps should not be necessary if you installed Mambaforge as suggested -above but may be needed if you have previously installed Miniconda3 instead: :: - - conda install -y -n base mamba - conda config --add channels conda-forge - conda config --set channel_priority strict - -Create a new conda environment with ``zppy`` installed and activate it: :: - - mamba create -n zppy_env zppy - conda activate zppy_env - -Or (less recommended because of potential conflicts) you can install ``zppy`` -in an existing environment. :: + source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh - mamba install zppy +**ALCF Polaris** + :: -Updating --------- + source /lus/grand/projects/E3SMinput/soft/e3sm-unified/load_latest_e3sm_unified_polaris.sh -If you **installed into your own conda environment** (e.g., not through the -unified environment), you can update ``zppy`` by doing the following: :: - mamba update zppy +Change ``.sh`` to ``.csh`` for ``csh`` shells. -.. _dev-env: +E3SM Unified and zppy versions +============================== -(b) Development Environment -=========================== +``zppy`` development is largely synced with ``e3sm_unified``. The last several releases have been: -Unlike the latest stable release (i.e., the user environment), the development -environment does not include ``zppy``. -Instead, the developer will ``python -m pip install -e .`` to build ``zppy`` with changes -(see step 7 below). +* E3SM Unified 1.13.0: ``zppy 3.2.0``, ``zppy-interfaces 0.2.1`` +* E3SM Unified 1.12.0: ``zppy 3.1.0``, ``zppy-interfaces 0.2.0`` +* E3SM Unified 1.11.1: ``zppy 3.1.0``, ``zppy-interfaces 0.1.2`` +* E3SM Unified 1.11.0: ``zppy 3.0.0``, ``zppy-interfaces 0.1.1`` (new auxillary package) +* E3SM Unified 1.10.0: ``zppy 2.5.0`` -Furthermore, the dev environment includes quality assurance (QA) tools such as code formatters, linters, and ``pre-commit``. -**NOTE**: These QA tools are enforced using ``pre-commit`` checks in the continuous integration/continuous delivery (CI/CD) build, so you must use the dev environment for all contributions. +To use ``zppy`` features/fixes not yet in a production release, you can use a development environment. -1. Follow :ref:`"Others/Local" ` section for installing conda. +Note that ``e3sm_unified``'s development cycle is not in phase with ``zppy``. +Therefore the version of ``zppy`` included may not be the latest. +To install the latest stable release, refer to the following: -2. Clone your fork and keep it in sync with the main repo's ``main`` +Setting up a development environment +==================================== :: + # Get the code ######################################################################## + # Set up your fork: # Go to https://github.com/E3SM-Project/zppy - # Click "Fork" in the upper right hand corner. This will fork the main repo. # Click the green "Code" button - # Choose the HTTPS or SSH option. - # (To use the SSH option, you need to have a SSH connection to GitHub set up). - # Click the clipboard icon to copy the path. - # On your command line: - git clone - git remote -v - # You should see your fork listed as `origin` + # Choose the SSH option and paste it here: + git clone git@github.com:E3SM-Project/zppy.git + cd zppy_dev + git remote -v # You should see the main repo listed as `origin` + # A couple optional steps: + git remote add upstream git@github.com:E3SM-Project/zppy.git # Use the name "upstream" instead of origin + git remote add your-fork-name git@github.com:your-fork-name/zppy.git # Use your fork, if you have one - or if you already have a clone of your fork, rebase your fork on the main repo's ``main`` to keep it in sync: - - :: - - # Add the main repo as a remote. - # You can call it anything but "upstream" is recommended. - # We'll use `` here. - git remote add + # To use the latest code: + git checkout upstream main + # To use code from a specific branch: + git checkout upstream that-branch-name - # Fetch all the branches of that remote into remote-tracking branches - git fetch + # Set up the environment ############################################################## + # First, make sure you have conda activated. Then: + rm -rf build # Sometimes an existing `build` directory can cause problems. + conda clean --all --y # This makes sure conda will pick up the latest information. + conda env create -f conda/dev.yml -n env-name + conda activate env-name + pre-commit run --all-files # This is only necessary if you've made changes + python -m pip install . # Install the code into your development environment (env-name) - # Make sure that you're on your ``main`` branch: - git checkout main - - # Rewrite your `main` branch so that any of your commits that - # aren't already in /main are replayed on top of that branch: - git rebase /main - - # Push your main branch to your GitHub fork: - # Note that should be `origin` if you cloned your fork as above. - git push -f main - - - Checkout a new branch from ``main``: - - :: - - git checkout -b /main - -3. Remove any cached conda packages. This will ensure that you always get the latest packages. - - :: - - mamba clean --all - -4. Enter the fork's clone. - - :: - - cd zppy - -5. Use conda to create a new dev environment. -(``zppy`` **is not included in this environment**). - - - Tip: Add the flag ``-n `` to customize the name of the environment - - :: - - mamba env create -f conda/dev.yml - conda activate zppy_dev - -6. Install ``pre-commit``. - - :: - - pre-commit install - -7. install ``zppy`` into the conda environment in edit mode (so code changes - are reflected in the environment as you make them) with: - - :: - - python -m pip install -e . - -8. Make the desired changes to ``zppy``. - -9. Commit changes and make sure ``pre-commit`` checks pass: - - :: - - git commit -m "commit-message" - - .. figure:: ../_static/pre-commit-passing.png - :alt: pre-commit Output - - ``pre-commit`` Output +Note: if you'd like to contribute to ``zppy`` rather than just using the latest code, please refer to the Developer Guide instead. Configuration file @@ -293,6 +131,9 @@ To start the post-processing: :: zppy -c -``zppy`` will parse the configuration file and then generate and submit all batch jobs. -``zppy`` can be invoked safely multiple times -- it will simply check the status of previously +``zppy`` will parse the configuration file and then generate and submit all batch jobs. +For the most part, ``zppy`` can be invoked safely multiple times -- it will simply check the status of previously submitted tasks, only submitting new or previously failed tasks. + +.. warning:: + Some tasks cannot be safely rerun. Notably, ``mpas_analysis`` has historically needed to be rerun from scratch if a job failed. This is documented in https://github.com/E3SM-Project/zppy/issues/764. diff --git a/docs/source/user_guide/parameters.rst b/docs/source/user_guide/parameters.rst index ed090f2f..a999c9ef 100644 --- a/docs/source/user_guide/parameters.rst +++ b/docs/source/user_guide/parameters.rst @@ -30,8 +30,8 @@ The following parameters no longer perform any function: "atmosphere_only" "plot_names" -Parameter checking & inferring -- for users -=========================================== +Parameter checking & inferring +============================== There are two types of inferences, each with their own parameter in ``default.ini``: @@ -129,23 +129,3 @@ For the ``ilamb`` task: **Required parameters, by e3sm_diags set** See `Confluence `_ for tables of which sets require which parameters. - -Parameter checking & inferring -- for developers -================================================ - -There are many parameter-handling functions. - -In ``utils.py``: - -* ``get_value_from_parameter``: check if parameter is in the configuration dictionary. If not, if inference is turned on (the default), then just use the value of ``second_choice_parameter``. If inference is turned off, raise a ``ParameterNotProvidedError``. Use this function if the backup option is another parameter (rather than a fixed value). -* ``set_value_of_parameter_if_undefined``: check if parameter is in the configuration dictionary. If not, if inference is turned on (the default), then just set the parameter's value to the ``backup_option``. If inference is turned off, raise a ``ParameterNotProvidedError``. - -In ``e3sm_diags.py``: - -* ``check_parameter_defined``: check if parameter is in the configuration dictionary, and if not raise a ``ParameterNotProvidedError``. -* ``check_set_specific_parameter``: if any requested ``e3sm_diags`` sets require this parameter, make sure it is present. If not, raise a ``ParameterNotProvidedError``. -* ``check_parameters_for_bash``: use ``check_set_specific_parameter`` to check the existence of parameters that aren't used until the bash script. -* ``check_mvm_only_parameters_for_bash``: similar, but these are specifically parameters used for model-vs-model runs. Uses ``check_parameter_defined`` in addition to ``check_set_specific_parameter``. -* ``check_and_define_parameters``: make sure all parameters are defined, using ``utils.py get_value_from_parameter``, ``utils.py set_value_of_parameter_if_undefined``, and ``check_mvm_only_parameters_for_bash``. - -``check_parameters_for_bash`` can be run immediately for each subtask because it has very few conditions. Other checks are included in ``check_and_define_parameters`` later on in the code. diff --git a/docs/source/user_guide/tasks/index.rst b/docs/source/user_guide/tasks/index.rst index c81210cf..89b3f33f 100644 --- a/docs/source/user_guide/tasks/index.rst +++ b/docs/source/user_guide/tasks/index.rst @@ -11,34 +11,37 @@ for multiple independent runs of the same task. **Available tasks:** +Listed by the order they are evaluated in `__main__.py`: + .. list-table:: :header-rows: 1 :widths: 25 75 * - Task - Description + * - :doc:`bundle` + - Bundle multiple tasks into a single SLURM job * - :doc:`climo` - Generate climatology files using NCO's ``ncclimo`` * - :doc:`ts` - Generate time-series files using NCO's ``ncclimo`` * - :doc:`e3sm_to_cmip` - Convert E3SM output to CMIP6 format using ``e3sm_to_cmip`` + * - :doc:`tc_analysis` + - Tropical cyclone analysis * - :doc:`e3sm_diags` - Run E3SM Diagnostics (atmosphere, land, river diagnostics, etc.) * - :doc:`mpas_analysis` - Run MPAS-Analysis (ocean and sea-ice diagnostics) * - :doc:`global_time_series` - Generate global time series plots - * - :doc:`tc_analysis` - - Tropical cyclone analysis * - :doc:`ilamb` - Run ILAMB land benchmarking * - :doc:`livvkit` - Run LIVVkit ice sheet validation * - :doc:`pcmdi_diags` - Run PCMDI metrics diagnostics - * - :doc:`bundle` - - Bundle multiple tasks into a single SLURM job + **Common (default) parameters** shared by all tasks are documented on the :doc:`../parameters` page. diff --git a/docs/source/user_guide/tutorial.rst b/docs/source/user_guide/tutorial.rst index 27ac107b..17fc44c4 100644 --- a/docs/source/user_guide/tutorial.rst +++ b/docs/source/user_guide/tutorial.rst @@ -38,8 +38,7 @@ Once the ``[ts]`` subsection ``[[ atm_monthly_glb ]]`` and the ``[mpas_analysis] section finish, ``[global_time_series]`` will run. Post-processing results will be located in ``output`` and ``www``. Some machines have -a web server. ``www`` should be pointed to that so that E3SM Diags, MPAS-Analysis, and -the global time series plots will be visible online. +a web server. ``www`` should be pointed to that so that the plotting tasks (here: ``e3sm_diags`` ``mpas_analysis``, ``global_time_series``, ``livvkit``) will be visible online. Because we have specified ``campaign = "water_cycle"``, some parameters will be automatically set. ``zppy/templates/water_cycle.cfg`` specifies what @@ -86,15 +85,24 @@ or ``test_comparison_type`` to disambiguate. Debugging failures ================== - .. code:: - - $ cd /post/scripts - $ grep -v "OK" *status # See what failed - # Review `.o` files corresponding to failed `.status` files. - # If an error is obvious, make a fix in the bash file and rerun: - $ sbatch .bash - # If the error is not obvious, do the following: - $ emacs .bash - # In this file, set `debug = True`. This will provide more information. - # Note: another option is to set `debug = True` in your `cfg` and rerun `zppy`. - $ sbatch .bash + :: + + grep "output=" your_zppy_config.cfg # Easy way to remember your output directory + cd your_output_dir/post/scripts + grep -v "OK" *status # See what failed + + # Say failing_task.status is showing a non-OK status, then: + ls failing_task* # See everything associated with it + + # Say its job ID was 123456, then: + emacs failing_task.o123456 # Review the output + emacs failing_task.settings # Review how each of its parameters was set/configured. + + # If an error is obvious, make a fix in the bash file and rerun: + sbatch failing_task.bash + + # If the error is not obvious, do the following: + emacs failing_task.bash + # In this file, set `debug = True`. This will provide more information. + # Note: another option is to set `debug = True` in your `cfg` and rerun `zppy -c`. + sbatch .bash From 22ae4a6570e9c1763aadd46527a1a5ca11f6b41c Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Thu, 4 Jun 2026 20:02:50 -0700 Subject: [PATCH 04/27] Add dependencies graph --- docs/source/dev_guide/dependencies.dot | 104 +++++++++++++++++++++++++ docs/source/dev_guide/dependencies.rst | 7 ++ docs/source/dev_guide/index.rst | 1 + 3 files changed, 112 insertions(+) create mode 100644 docs/source/dev_guide/dependencies.dot create mode 100644 docs/source/dev_guide/dependencies.rst diff --git a/docs/source/dev_guide/dependencies.dot b/docs/source/dev_guide/dependencies.dot new file mode 100644 index 00000000..b2e70c41 --- /dev/null +++ b/docs/source/dev_guide/dependencies.dot @@ -0,0 +1,104 @@ +digraph TaskDependencies { + // Graph settings + rankdir=LR // Left-to-right layout (TB for top-to-bottom) + node [shape=box, style=rounded] + + // Reviewing the dependencies in each `.py` file + // and the test cfgs, we come up with the following: + + // VERTICES (tasks) + + climo_diurnal_atm [label="climo: diurnal, atm"] + climo_month_atm [label="climo: monthly, atm"] + climo_month_lnd [label="climo: monthly, lnd"] + + ts_daily_atm [label="ts: daily, atm"] + ts_month_atm [label="ts: monthly, atm"] + ts_month_atm_glb [label="ts: monthly, atm, glb"] + ts_month_lnd [label="ts: monthly, lnd"] + ts_month_lnd_glb [label="ts: monthly, lnd, glb"] + ts_month_rof [label="ts: monthly, rof"] + + e3sm_to_cmip_month_atm [label="e3sm_to_cmip: monthly, atm"] + e3sm_to_cmip_month_lnd [label="e3sm_to_cmip: monthly, lnd"] + + tc_analysis + + e3sm_diags_diurnal [label="e3sm_diags: diurnal_cycle"] + e3sm_diags_climo_month_atm [label="e3sm_diags: lat_lon,zonal_mean_xy,\nzonal_mean_2d,polar,cosp_histogram,\nmeridional_mean_2d,annual_cycle_zonal_mean,zonal_mean_2d_stratosphere,\naerosol_aeronet,aerosol_budget"] + e3sm_diags_climo_month_lnd [label="e3sm_diags: lat_lon_land"] + e3sm_diags_streamflow [label="e3sm_diags: streamflow"] + e3sm_diags_tc [label="e3sm_diags: tc_analysis"] + e3sm_diags_ts_daily_atm [label="e3sm_diags: tropical_subseasonal,precip_pdf"] + e3sm_diags_ts_month_atm [label="e3sm_diags: enso_diags,qbo,\narea_mean_time_series,mp_partition"] + + mpas_analysis + + global_time_series_classic_atm [label="global_time_series:\nThe 5 atm classic plots"] + global_time_series_classic_ocn [label="global_time_series:\nThe 3 ocn classic plots"] + global_time_series_plots_atm [label="global_time_series:\natm plots"] + global_time_series_plots_ice [label="global_time_series:\nice plots"] // Currently unused + global_time_series_plots_lnd [label="global_time_series:\nlnd plots"] + global_time_series_plots_ocn [label="global_time_series:\nocn plots"] + + ilamb_lnd_atm [label="ilamb: lnd & atm"] + ilamb_lnd_only [label="ilamb: lnd only"] + + livvkit + + pcmdi_diags_mean_climate [label="pcmdi_diags: mean_climate"] + pcmdi_diags_variability_modes_atm [label="pcmdi_diags: variability_modes_atm"] + pcmdi_diags_variability_modes_cpl [label="pcmdi_diags: variability_modes_cpl"] + pcmdi_diags_enso [label="pcmdi_diags: enso"] + pcmdi_diags_synthetic_plots [label="pcmdi_diags: synthetic_plots"] + + + // EDGES (dependencies) + + // Defined in climo.py: + // None + + // Defined in ts.py: + // None + + // Defined in e3sm_to_cmip.py: + ts_month_atm -> e3sm_to_cmip_month_atm + ts_month_lnd -> e3sm_to_cmip_month_lnd + + // Defined in tc_analysis.py: + // None + + // Defined in e3sm_diags.py: + climo_month_atm -> e3sm_diags_climo_month_atm + climo_month_lnd -> e3sm_diags_climo_month_lnd + climo_diurnal_atm -> e3sm_diags_diurnal + tc_analysis -> e3sm_diags_tc + ts_month_atm -> e3sm_diags_ts_month_atm + ts_month_rof -> e3sm_diags_streamflow + ts_daily_atm -> e3sm_diags_ts_daily_atm + + // Defined in mpas_analysis.py: + mpas_analysis -> mpas_analysis // Later year sets depend on earlier runs + + // Defined in global_time_series.py: + ts_month_atm_glb -> {global_time_series_classic_atm, global_time_series_plots_atm} + ts_month_lnd_glb -> global_time_series_plots_lnd + mpas_analysis -> {global_time_series_classic_ocn, global_time_series_plots_ocn} + + // Defined in ilamb.py: + ts_month_atm -> ilamb_lnd_atm + ts_month_lnd -> {ilamb_lnd_atm, ilamb_lnd_only} + e3sm_to_cmip_month_atm -> ilamb_lnd_atm + e3sm_to_cmip_month_lnd -> {ilamb_lnd_atm, ilamb_lnd_only} + + // Defined in livvkit.py: + // Data sources in livvkit.py: "cmb", "smb", "racmo", "merra2", "ceres", "era5" + // climo extensions in tests/integration/template_weekly_comprehensive_v3.cfg: traave, native, racmo_gis, racmo_ais, merra2, era5 + // ts extensions in tests/integration/template_weekly_comprehensive_v3.cfg: energy, smb + climo_month_lnd -> livvkit [label="Specify mapping files"] + ts_month_lnd -> livvkit [label="Specify mapping files"] + + // Defined in pcmdi_diags.py: + {ts_month_atm, e3sm_to_cmip_month_atm} -> {pcmdi_diags_mean_climate, pcmdi_diags_variability_modes_atm, pcmdi_diags_variability_modes_cpl, pcmdi_diags_enso} -> pcmdi_diags_synthetic_plots + +} diff --git a/docs/source/dev_guide/dependencies.rst b/docs/source/dev_guide/dependencies.rst new file mode 100644 index 00000000..24b2e062 --- /dev/null +++ b/docs/source/dev_guide/dependencies.rst @@ -0,0 +1,7 @@ +************ +Dependencies +************ + +.. literalinclude:: ./dependencies.dot + :language: dot + :linenos: diff --git a/docs/source/dev_guide/index.rst b/docs/source/dev_guide/index.rst index b33e20cf..b8a1727f 100644 --- a/docs/source/dev_guide/index.rst +++ b/docs/source/dev_guide/index.rst @@ -10,6 +10,7 @@ This guide covers everything needed to develop, test, and release ``zppy``. project-standards contributing parameters + dependencies tasks/index testing release_testing From 101544243cb490673f62be4c5df057b68b7c9136 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Fri, 5 Jun 2026 12:11:57 -0700 Subject: [PATCH 05/27] Update dependencies in task docs --- docs/source/dev_guide/tasks/climo.rst | 7 +++---- docs/source/dev_guide/tasks/e3sm_diags.rst | 17 +++++------------ docs/source/dev_guide/tasks/e3sm_to_cmip.rst | 14 ++++---------- .../dev_guide/tasks/global_time_series.rst | 13 ++++--------- docs/source/dev_guide/tasks/ilamb.rst | 19 ++++--------------- docs/source/dev_guide/tasks/livvkit.rst | 13 ++++--------- docs/source/dev_guide/tasks/mpas_analysis.rst | 10 ++++------ docs/source/dev_guide/tasks/pcmdi_diags.rst | 10 +++++----- docs/source/dev_guide/tasks/tc_analysis.rst | 10 ++++------ docs/source/dev_guide/tasks/ts.rst | 12 +++++++----- 10 files changed, 44 insertions(+), 81 deletions(-) diff --git a/docs/source/dev_guide/tasks/climo.rst b/docs/source/dev_guide/tasks/climo.rst index 8989490b..6cab3541 100644 --- a/docs/source/dev_guide/tasks/climo.rst +++ b/docs/source/dev_guide/tasks/climo.rst @@ -36,13 +36,12 @@ Dependencies **Upstream (what climo depends on):** -- None. The ``climo`` task has no zppy dependencies. It reads raw - simulation output directly. +- None **Downstream (what depends on climo):** -- :doc:`e3sm_diags` — requires climatology files for most diagnostic sets -- :doc:`livvkit` — requires grid-native and regridded climatology files +- :doc:`e3sm_diags` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly dirunal-atm climo: diurnal_cycle +- :doc:`livvkit` — Monthly-lnd climo Template variables ------------------ diff --git a/docs/source/dev_guide/tasks/e3sm_diags.rst b/docs/source/dev_guide/tasks/e3sm_diags.rst index bf3493b6..c62a03cb 100644 --- a/docs/source/dev_guide/tasks/e3sm_diags.rst +++ b/docs/source/dev_guide/tasks/e3sm_diags.rst @@ -29,20 +29,13 @@ Key functions in ``e3sm_diags.py``: Dependencies ------------ -**Upstream (required — varies by diagnostic set):** +**Upstream (what e3sm_diags depends on):** -- :doc:`climo` — required for most sets (``lat_lon``, ``zonal_mean_xy``, - etc.) -- :doc:`ts` — required for time-series sets (``enso_diags``, ``qbo``, - ``area_mean_time_series``, ``tropical_subseasonal``, ``streamflow``) -- Diurnal cycle ``[climo]`` subtask — required for ``diurnal_cycle`` set -- Daily ``[ts]`` subtask — required for ``tropical_subseasonal`` set +- :doc:`climo` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly dirunal-atm climo: diurnal_cycle +- :doc:`ts` — Monthly-atm ts: enso_diags, qbo, area_mean_time_series, mp_partition. Monthly-rof ts: streamflow. Daily-atm ts: tropical_subseasonal, precip_pdf. +- :doc:`tc_analysis` — Required for tc_analysis set. -The specific subtasks to depend on are set via: -``climo_subsection``, ``ts_subsection``, ``climo_diurnal_subsection``, -``ts_daily_subsection``. - -**Downstream:** +**Downstream (what depends on e3sm_diags):** - None. diff --git a/docs/source/dev_guide/tasks/e3sm_to_cmip.rst b/docs/source/dev_guide/tasks/e3sm_to_cmip.rst index c225fa88..2def64d0 100644 --- a/docs/source/dev_guide/tasks/e3sm_to_cmip.rst +++ b/docs/source/dev_guide/tasks/e3sm_to_cmip.rst @@ -12,17 +12,11 @@ Implementation Dependencies ------------ -**Upstream (required):** +**Upstream (what e3sm_to_cmip depends on):** -- :doc:`ts` — depends on one or more ``[ts]`` subtasks to provide - input time-series files - -The ``ts_subsection``, ``ts_atm_subsection``, and ``ts_land_subsection`` -parameters specify which ``[ts]`` subtasks to depend on. With -``infer_section_parameters = True``, these default to the subsection name, -``atm_monthly_180x360_aave``, and ``land_monthly`` respectively. +- :doc:`ts` — Monthly-atm ts: monthly-atm e3sm_to_cmip. Monthly-lnd ts: monthly-lnd e3sm_to_cmip. **Downstream (what depends on e3sm_to_cmip):** -- :doc:`ilamb` — requires CMIP-format land and atmosphere data -- :doc:`pcmdi_diags` — requires CMIP-format atmosphere data +- :doc:`ilamb` — Monthly-lnd e3sm_to_cmip: required. Monthly-atm e3sm_to_cmip: optional. +- :doc:`pcmdi_diags` — Monthly-atm e3sm_to_cmip: mean_climate, variability_modes_atm,variability_modes_cpl, enso diff --git a/docs/source/dev_guide/tasks/global_time_series.rst b/docs/source/dev_guide/tasks/global_time_series.rst index d97303fc..96bb307e 100644 --- a/docs/source/dev_guide/tasks/global_time_series.rst +++ b/docs/source/dev_guide/tasks/global_time_series.rst @@ -15,16 +15,11 @@ Implementation Dependencies ------------ -**Upstream (required):** +**Upstream (what global_time_series depends on):** -- :doc:`ts` — depends on ``[ts]`` global-mean subtasks for atmospheric, - land, and (optionally) ocean/sea-ice data +- :doc:`ts` — Monthly-atm-glb ts: the 5 classic atm plots, plots for specific atm variables. Monthly-lnd-glb ts: plots for specific lnd variables. +- :doc:`mpas_analysis` — Monthly-atm-glb ts: the 3 classic ocn plots, plots for specific ocn variables. -**Upstream (optional):** - -- :doc:`mpas_analysis` — depends on MPAS-Analysis output for ocean and - sea-ice time series plots. Specified via ``mpas_analysis_subsections``. - -**Downstream:** +**Downstream (what depends on global_time_series):** - None. diff --git a/docs/source/dev_guide/tasks/ilamb.rst b/docs/source/dev_guide/tasks/ilamb.rst index 8ddeb90f..708d8dd3 100644 --- a/docs/source/dev_guide/tasks/ilamb.rst +++ b/docs/source/dev_guide/tasks/ilamb.rst @@ -12,22 +12,11 @@ Implementation Dependencies ------------ -**Upstream (required):** +**Upstream (what ilamb depends on):** -- :doc:`e3sm_to_cmip` — depends on CMIP-format land and (for non-land-only - runs) atmosphere time-series files +- :doc:`ts` -- Monthly-lnd ts: required. Monthly-atm ts: optional. +- :doc:`e3sm_to_cmip` — Monthly-lnd e3sm_to_cmip: required. Monthly-atm e3sm_to_cmip: optional. - - Land subtask: ``e3sm_to_cmip_land_subsection`` (inferred as - ``land_monthly`` if not set) - - Atmosphere subtask: ``e3sm_to_cmip_atm_subsection`` (inferred as - ``atm_monthly_180x360_aave`` if not set, skipped for land-only runs) - -- :doc:`ts` — also depends on ``ts`` subtasks: - - - Land subtask: ``ts_land_subsection`` (inferred as ``land_monthly``) - - Atmosphere subtask: ``ts_atm_subsection`` (inferred as - ``atm_monthly_180x360_aave``, skipped for land-only runs) - -**Downstream:** +**Downstream (what depends on ilamb):** - None. diff --git a/docs/source/dev_guide/tasks/livvkit.rst b/docs/source/dev_guide/tasks/livvkit.rst index d152a095..55ea1324 100644 --- a/docs/source/dev_guide/tasks/livvkit.rst +++ b/docs/source/dev_guide/tasks/livvkit.rst @@ -12,16 +12,11 @@ Implementation Dependencies ------------ -**Upstream (required):** +**Upstream (what livvkit depends on):** -- :doc:`climo` — depends on multiple ``[climo]`` subtasks: +- :doc:`climo` — Monthly-lnd climo +- :doc:`ts` -- Monthly-lnd ts. - - A grid-native climatology: assumed ``land_monthly_climo_native`` - - Reanalysis-grid climatologies: one per reanalysis comparison, named - ``land_monthly_climo_`` - - All required subtask names are listed in ``climo_subsections``. - -**Downstream:** +**Downstream (what depends on livvkit):** - None. diff --git a/docs/source/dev_guide/tasks/mpas_analysis.rst b/docs/source/dev_guide/tasks/mpas_analysis.rst index 3c592312..3dffe4e1 100644 --- a/docs/source/dev_guide/tasks/mpas_analysis.rst +++ b/docs/source/dev_guide/tasks/mpas_analysis.rst @@ -35,12 +35,10 @@ locates MPAS-Analysis config files from prior runs: Dependencies ------------ -**Upstream:** +**Upstream (what mpas_analysis depends on):** -- None required. MPAS-Analysis reads MPAS-Ocean and MPAS-seaice output - directly from the simulation archive. +- None. Note that later year sets will depend on previously completed ones. -**Downstream (optional):** +**Downstream (what depends on mpas_analysis):** -- :doc:`global_time_series` — can optionally depend on MPAS-Analysis output - for ocean/sea-ice time series plots +- :doc:`global_time_series` — Monthly-atm-glb ts: the 3 classic ocn plots, plots for specific ocn variables. diff --git a/docs/source/dev_guide/tasks/pcmdi_diags.rst b/docs/source/dev_guide/tasks/pcmdi_diags.rst index 385d1f8d..4304bede 100644 --- a/docs/source/dev_guide/tasks/pcmdi_diags.rst +++ b/docs/source/dev_guide/tasks/pcmdi_diags.rst @@ -17,12 +17,12 @@ name) determines which set to run. Dependencies ------------ -**Upstream (required):** +**Upstream (what pcmdi_diags depends on):** -- :doc:`e3sm_to_cmip` — depends on a ``[e3sm_to_cmip]`` atmosphere subtask - to provide CMIP-format input data. Specified via - ``e3sm_to_cmip_atm_subsection``. +- :doc:`ts` — Monthly-atm ts: mean_climate, variability_modes_atm,variability_modes_cpl, enso +- :doc:`e3sm_to_cmip` — Monthly-atm e3sm_to_cmip: mean_climate, variability_modes_atm,variability_modes_cpl, enso +- Note that the synthetic_plots set depends on the other sets. -**Downstream:** +**Downstream (what depends on pcmdi_diags):** - None. diff --git a/docs/source/dev_guide/tasks/tc_analysis.rst b/docs/source/dev_guide/tasks/tc_analysis.rst index 1440d5c0..b2acdf25 100644 --- a/docs/source/dev_guide/tasks/tc_analysis.rst +++ b/docs/source/dev_guide/tasks/tc_analysis.rst @@ -15,12 +15,10 @@ to detect and track tropical cyclones. Dependencies ------------ -**Upstream:** +**Upstream (what tc_analysis depends on):** -- None. Reads high-frequency EAM output directly. +- None. -**Downstream:** +**Downstream (what depends on tc_analysis):** -- None within ``zppy``. TC analysis results can be used as input to the - ``tc_analysis`` set in ``e3sm_diags`` (model-vs-model), but that - dependency is specified separately in the ``[e3sm_diags]`` configuration. +- :doc:`e3sm_diags` — Required for tc_analysis set. diff --git a/docs/source/dev_guide/tasks/ts.rst b/docs/source/dev_guide/tasks/ts.rst index f928fe71..29ce7e47 100644 --- a/docs/source/dev_guide/tasks/ts.rst +++ b/docs/source/dev_guide/tasks/ts.rst @@ -35,11 +35,13 @@ Dependencies **Upstream (what ts depends on):** -- None. The ``ts`` task reads raw simulation output directly. +- None. **Downstream (what depends on ts):** -- :doc:`e3sm_diags` — many sets require time-series files -- :doc:`e3sm_to_cmip` — depends on ts for input time-series -- :doc:`global_time_series` — depends on ts global-mean subtasks -- :doc:`pcmdi_diags` — depends on ts via e3sm_to_cmip +- :doc:`e3sm_to_cmip` — Monthly-atm ts: monthly-atm e3sm_to_cmip. Monthly-lnd ts: monthly-lnd e3sm_to_cmip. +- :doc:`e3sm_diags` — Monthly-atm ts: enso_diags, qbo, area_mean_time_series, mp_partition. Monthly-rof ts: streamflow. Daily-atm ts: tropical_subseasonal, precip_pdf. +- :doc:`global_time_series` — Monthly-atm-glb ts: the 5 classic atm plots, plots for specific atm variables. Monthly-lnd-glb ts: plots for specific lnd variables. +- :doc:`ilamb` -- Monthly-lnd ts: required. Monthly-atm ts: optional. +- :doc:`livvkit` -- Monthly-lnd ts. +- :doc:`pcmdi_diags` — Monthly-atm ts: mean_climate, variability_modes_atm,variability_modes_cpl, enso From 925140af934afb70bf420bdac5d6fefda04e379e Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Fri, 5 Jun 2026 12:21:22 -0700 Subject: [PATCH 06/27] Move dependencies docs to user guide --- docs/source/{dev_guide => }/dependencies.dot | 0 docs/source/dev_guide/index.rst | 1 - docs/source/dev_guide/tasks/climo.rst | 12 ------------ docs/source/dev_guide/tasks/e3sm_diags.rst | 13 ------------- docs/source/dev_guide/tasks/e3sm_to_cmip.rst | 12 ------------ .../dev_guide/tasks/global_time_series.rst | 12 ------------ docs/source/dev_guide/tasks/ilamb.rst | 12 ------------ docs/source/dev_guide/tasks/livvkit.rst | 12 ------------ docs/source/dev_guide/tasks/mpas_analysis.rst | 11 ----------- docs/source/dev_guide/tasks/pcmdi_diags.rst | 13 ------------- docs/source/dev_guide/tasks/tc_analysis.rst | 11 ----------- docs/source/dev_guide/tasks/ts.rst | 16 ---------------- .../{dev_guide => user_guide}/dependencies.rst | 2 +- docs/source/user_guide/index.rst | 1 + docs/source/user_guide/tasks/climo.rst | 14 +++++++++++++- docs/source/user_guide/tasks/e3sm_diags.rst | 13 +++++++++++++ docs/source/user_guide/tasks/e3sm_to_cmip.rst | 14 +++++++++++++- .../user_guide/tasks/global_time_series.rst | 12 ++++++++++++ docs/source/user_guide/tasks/ilamb.rst | 14 +++++++++++++- docs/source/user_guide/tasks/livvkit.rst | 14 +++++++++++++- docs/source/user_guide/tasks/mpas_analysis.rst | 11 +++++++++++ docs/source/user_guide/tasks/pcmdi_diags.rst | 13 +++++++++++++ docs/source/user_guide/tasks/tc_analysis.rst | 13 ++++++++++++- docs/source/user_guide/tasks/ts.rst | 18 +++++++++++++++++- 24 files changed, 132 insertions(+), 132 deletions(-) rename docs/source/{dev_guide => }/dependencies.dot (100%) rename docs/source/{dev_guide => user_guide}/dependencies.rst (63%) diff --git a/docs/source/dev_guide/dependencies.dot b/docs/source/dependencies.dot similarity index 100% rename from docs/source/dev_guide/dependencies.dot rename to docs/source/dependencies.dot diff --git a/docs/source/dev_guide/index.rst b/docs/source/dev_guide/index.rst index b8a1727f..b33e20cf 100644 --- a/docs/source/dev_guide/index.rst +++ b/docs/source/dev_guide/index.rst @@ -10,7 +10,6 @@ This guide covers everything needed to develop, test, and release ``zppy``. project-standards contributing parameters - dependencies tasks/index testing release_testing diff --git a/docs/source/dev_guide/tasks/climo.rst b/docs/source/dev_guide/tasks/climo.rst index 6cab3541..6494ed01 100644 --- a/docs/source/dev_guide/tasks/climo.rst +++ b/docs/source/dev_guide/tasks/climo.rst @@ -31,18 +31,6 @@ Key steps in ``climo.py``: d. Submit the script via ``submit_script()``, registering it with any bundle if requested. -Dependencies ------------- - -**Upstream (what climo depends on):** - -- None - -**Downstream (what depends on climo):** - -- :doc:`e3sm_diags` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly dirunal-atm climo: diurnal_cycle -- :doc:`livvkit` — Monthly-lnd climo - Template variables ------------------ diff --git a/docs/source/dev_guide/tasks/e3sm_diags.rst b/docs/source/dev_guide/tasks/e3sm_diags.rst index c62a03cb..b6dd8a13 100644 --- a/docs/source/dev_guide/tasks/e3sm_diags.rst +++ b/docs/source/dev_guide/tasks/e3sm_diags.rst @@ -26,19 +26,6 @@ Key functions in ``e3sm_diags.py``: - ``check_and_define_parameters(c)``: resolves all parameters (inference + validation). Called later in the pipeline. -Dependencies ------------- - -**Upstream (what e3sm_diags depends on):** - -- :doc:`climo` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly dirunal-atm climo: diurnal_cycle -- :doc:`ts` — Monthly-atm ts: enso_diags, qbo, area_mean_time_series, mp_partition. Monthly-rof ts: streamflow. Daily-atm ts: tropical_subseasonal, precip_pdf. -- :doc:`tc_analysis` — Required for tc_analysis set. - -**Downstream (what depends on e3sm_diags):** - -- None. - Adding a new diagnostic set ----------------------------- diff --git a/docs/source/dev_guide/tasks/e3sm_to_cmip.rst b/docs/source/dev_guide/tasks/e3sm_to_cmip.rst index 2def64d0..32774239 100644 --- a/docs/source/dev_guide/tasks/e3sm_to_cmip.rst +++ b/docs/source/dev_guide/tasks/e3sm_to_cmip.rst @@ -8,15 +8,3 @@ Implementation - **Python module**: ``zppy/e3sm_to_cmip.py`` - **Jinja2 template**: ``zppy/templates/e3sm_to_cmip.bash`` - -Dependencies ------------- - -**Upstream (what e3sm_to_cmip depends on):** - -- :doc:`ts` — Monthly-atm ts: monthly-atm e3sm_to_cmip. Monthly-lnd ts: monthly-lnd e3sm_to_cmip. - -**Downstream (what depends on e3sm_to_cmip):** - -- :doc:`ilamb` — Monthly-lnd e3sm_to_cmip: required. Monthly-atm e3sm_to_cmip: optional. -- :doc:`pcmdi_diags` — Monthly-atm e3sm_to_cmip: mean_climate, variability_modes_atm,variability_modes_cpl, enso diff --git a/docs/source/dev_guide/tasks/global_time_series.rst b/docs/source/dev_guide/tasks/global_time_series.rst index 96bb307e..854b1fc0 100644 --- a/docs/source/dev_guide/tasks/global_time_series.rst +++ b/docs/source/dev_guide/tasks/global_time_series.rst @@ -11,15 +11,3 @@ Implementation ``global_time_series`` is called through the ``zppy-interfaces`` entry-point. - -Dependencies ------------- - -**Upstream (what global_time_series depends on):** - -- :doc:`ts` — Monthly-atm-glb ts: the 5 classic atm plots, plots for specific atm variables. Monthly-lnd-glb ts: plots for specific lnd variables. -- :doc:`mpas_analysis` — Monthly-atm-glb ts: the 3 classic ocn plots, plots for specific ocn variables. - -**Downstream (what depends on global_time_series):** - -- None. diff --git a/docs/source/dev_guide/tasks/ilamb.rst b/docs/source/dev_guide/tasks/ilamb.rst index 708d8dd3..ec68bb9d 100644 --- a/docs/source/dev_guide/tasks/ilamb.rst +++ b/docs/source/dev_guide/tasks/ilamb.rst @@ -8,15 +8,3 @@ Implementation - **Python module**: ``zppy/ilamb.py`` - **Jinja2 template**: ``zppy/templates/ilamb.bash`` - -Dependencies ------------- - -**Upstream (what ilamb depends on):** - -- :doc:`ts` -- Monthly-lnd ts: required. Monthly-atm ts: optional. -- :doc:`e3sm_to_cmip` — Monthly-lnd e3sm_to_cmip: required. Monthly-atm e3sm_to_cmip: optional. - -**Downstream (what depends on ilamb):** - -- None. diff --git a/docs/source/dev_guide/tasks/livvkit.rst b/docs/source/dev_guide/tasks/livvkit.rst index 55ea1324..394bba00 100644 --- a/docs/source/dev_guide/tasks/livvkit.rst +++ b/docs/source/dev_guide/tasks/livvkit.rst @@ -8,15 +8,3 @@ Implementation - **Python module**: ``zppy/livvkit.py`` - **Jinja2 template**: ``zppy/templates/livvkit.bash`` - -Dependencies ------------- - -**Upstream (what livvkit depends on):** - -- :doc:`climo` — Monthly-lnd climo -- :doc:`ts` -- Monthly-lnd ts. - -**Downstream (what depends on livvkit):** - -- None. diff --git a/docs/source/dev_guide/tasks/mpas_analysis.rst b/docs/source/dev_guide/tasks/mpas_analysis.rst index 3dffe4e1..be7dfa74 100644 --- a/docs/source/dev_guide/tasks/mpas_analysis.rst +++ b/docs/source/dev_guide/tasks/mpas_analysis.rst @@ -31,14 +31,3 @@ locates MPAS-Analysis config files from prior runs: ``reference_comparison_type`` or ``test_comparison_type`` explicitly. - If ``reference_data_path`` points to a ``[[subsection]]``, zppy uses that subsection's actual comparison type. - -Dependencies ------------- - -**Upstream (what mpas_analysis depends on):** - -- None. Note that later year sets will depend on previously completed ones. - -**Downstream (what depends on mpas_analysis):** - -- :doc:`global_time_series` — Monthly-atm-glb ts: the 3 classic ocn plots, plots for specific ocn variables. diff --git a/docs/source/dev_guide/tasks/pcmdi_diags.rst b/docs/source/dev_guide/tasks/pcmdi_diags.rst index 4304bede..a25a2b3e 100644 --- a/docs/source/dev_guide/tasks/pcmdi_diags.rst +++ b/docs/source/dev_guide/tasks/pcmdi_diags.rst @@ -13,16 +13,3 @@ Implementation ``variability_modes_cpl``, ``variability_modes_atm``, ``enso``, ``synthetic_plots``). The ``current_set`` parameter (or the subsection name) determines which set to run. - -Dependencies ------------- - -**Upstream (what pcmdi_diags depends on):** - -- :doc:`ts` — Monthly-atm ts: mean_climate, variability_modes_atm,variability_modes_cpl, enso -- :doc:`e3sm_to_cmip` — Monthly-atm e3sm_to_cmip: mean_climate, variability_modes_atm,variability_modes_cpl, enso -- Note that the synthetic_plots set depends on the other sets. - -**Downstream (what depends on pcmdi_diags):** - -- None. diff --git a/docs/source/dev_guide/tasks/tc_analysis.rst b/docs/source/dev_guide/tasks/tc_analysis.rst index b2acdf25..3d393b2c 100644 --- a/docs/source/dev_guide/tasks/tc_analysis.rst +++ b/docs/source/dev_guide/tasks/tc_analysis.rst @@ -11,14 +11,3 @@ Implementation ``tc_analysis`` processes high-frequency atmospheric output (``eam.h2``) to detect and track tropical cyclones. - -Dependencies ------------- - -**Upstream (what tc_analysis depends on):** - -- None. - -**Downstream (what depends on tc_analysis):** - -- :doc:`e3sm_diags` — Required for tc_analysis set. diff --git a/docs/source/dev_guide/tasks/ts.rst b/docs/source/dev_guide/tasks/ts.rst index 29ce7e47..a4a1ec62 100644 --- a/docs/source/dev_guide/tasks/ts.rst +++ b/docs/source/dev_guide/tasks/ts.rst @@ -29,19 +29,3 @@ Vertical remapping When ``vrt_remap_vars`` is non-empty, the template also calls ``ncremap`` after ``ncclimo`` to remap specified variables from model levels to pressure levels. Output is written to a sibling ``ts_vrt_remap/`` directory. - -Dependencies ------------- - -**Upstream (what ts depends on):** - -- None. - -**Downstream (what depends on ts):** - -- :doc:`e3sm_to_cmip` — Monthly-atm ts: monthly-atm e3sm_to_cmip. Monthly-lnd ts: monthly-lnd e3sm_to_cmip. -- :doc:`e3sm_diags` — Monthly-atm ts: enso_diags, qbo, area_mean_time_series, mp_partition. Monthly-rof ts: streamflow. Daily-atm ts: tropical_subseasonal, precip_pdf. -- :doc:`global_time_series` — Monthly-atm-glb ts: the 5 classic atm plots, plots for specific atm variables. Monthly-lnd-glb ts: plots for specific lnd variables. -- :doc:`ilamb` -- Monthly-lnd ts: required. Monthly-atm ts: optional. -- :doc:`livvkit` -- Monthly-lnd ts. -- :doc:`pcmdi_diags` — Monthly-atm ts: mean_climate, variability_modes_atm,variability_modes_cpl, enso diff --git a/docs/source/dev_guide/dependencies.rst b/docs/source/user_guide/dependencies.rst similarity index 63% rename from docs/source/dev_guide/dependencies.rst rename to docs/source/user_guide/dependencies.rst index 24b2e062..0fda4c25 100644 --- a/docs/source/dev_guide/dependencies.rst +++ b/docs/source/user_guide/dependencies.rst @@ -2,6 +2,6 @@ Dependencies ************ -.. literalinclude:: ./dependencies.dot +.. literalinclude:: ../dependencies.dot :language: dot :linenos: diff --git a/docs/source/user_guide/index.rst b/docs/source/user_guide/index.rst index db9d9c02..62e30b8f 100644 --- a/docs/source/user_guide/index.rst +++ b/docs/source/user_guide/index.rst @@ -15,5 +15,6 @@ post-process E3SM simulations. schematics campaigns parameters + dependencies tasks/index components/index diff --git a/docs/source/user_guide/tasks/climo.rst b/docs/source/user_guide/tasks/climo.rst index 683c235e..75287bb0 100644 --- a/docs/source/user_guide/tasks/climo.rst +++ b/docs/source/user_guide/tasks/climo.rst @@ -78,7 +78,7 @@ all :ref:`common parameters ` from the ``[default]`` section apply. internally. Inherited common parameters most relevant to ``climo`` ------------------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 @@ -140,3 +140,15 @@ Inherited common parameters most relevant to ``climo`` - ``""`` - Shell commands to set up the software environment before running the task (e.g., ``source /path/to/e3sm_unified.sh``). + +Dependencies +------------ + +**Upstream (what climo depends on):** + +- None + +**Downstream (what depends on climo):** + +- :doc:`e3sm_diags` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly dirunal-atm climo: diurnal_cycle +- :doc:`livvkit` — Monthly-lnd climo diff --git a/docs/source/user_guide/tasks/e3sm_diags.rst b/docs/source/user_guide/tasks/e3sm_diags.rst index 8c022479..c8c034db 100644 --- a/docs/source/user_guide/tasks/e3sm_diags.rst +++ b/docs/source/user_guide/tasks/e3sm_diags.rst @@ -189,3 +189,16 @@ Parameters - No - ``""`` - Path to observation data for ``tc_analysis`` set. + +Dependencies +------------ + +**Upstream (what e3sm_diags depends on):** + +- :doc:`climo` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly dirunal-atm climo: diurnal_cycle +- :doc:`ts` — Monthly-atm ts: enso_diags, qbo, area_mean_time_series, mp_partition. Monthly-rof ts: streamflow. Daily-atm ts: tropical_subseasonal, precip_pdf. +- :doc:`tc_analysis` — Required for tc_analysis set. + +**Downstream (what depends on e3sm_diags):** + +- None. diff --git a/docs/source/user_guide/tasks/e3sm_to_cmip.rst b/docs/source/user_guide/tasks/e3sm_to_cmip.rst index b7371c30..1d45816f 100644 --- a/docs/source/user_guide/tasks/e3sm_to_cmip.rst +++ b/docs/source/user_guide/tasks/e3sm_to_cmip.rst @@ -66,7 +66,7 @@ Parameters - Name of the ``[ts]`` land subtask (for multi-component runs). Inherited common parameters most relevant to ``e3sm_to_cmip`` -------------------------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 @@ -112,3 +112,15 @@ Inherited common parameters most relevant to ``e3sm_to_cmip`` - No - ``""`` - Shell commands to activate the software environment. + +Dependencies +------------ + +**Upstream (what e3sm_to_cmip depends on):** + +- :doc:`ts` — Monthly-atm ts: monthly-atm e3sm_to_cmip. Monthly-lnd ts: monthly-lnd e3sm_to_cmip. + +**Downstream (what depends on e3sm_to_cmip):** + +- :doc:`ilamb` — Monthly-lnd e3sm_to_cmip: required. Monthly-atm e3sm_to_cmip: optional. +- :doc:`pcmdi_diags` — Monthly-atm e3sm_to_cmip: mean_climate, variability_modes_atm,variability_modes_cpl, enso diff --git a/docs/source/user_guide/tasks/global_time_series.rst b/docs/source/user_guide/tasks/global_time_series.rst index 4b294e67..efd8f64c 100644 --- a/docs/source/user_guide/tasks/global_time_series.rst +++ b/docs/source/user_guide/tasks/global_time_series.rst @@ -100,3 +100,15 @@ Parameters - No - ``""`` - Extra sea-ice plots (when ``make_viewer = True``). + +Dependencies +------------ + +**Upstream (what global_time_series depends on):** + +- :doc:`ts` — Monthly-atm-glb ts: the 5 classic atm plots, plots for specific atm variables. Monthly-lnd-glb ts: plots for specific lnd variables. +- :doc:`mpas_analysis` — Monthly-atm-glb ts: the 3 classic ocn plots, plots for specific ocn variables. + +**Downstream (what depends on global_time_series):** + +- None. \ No newline at end of file diff --git a/docs/source/user_guide/tasks/ilamb.rst b/docs/source/user_guide/tasks/ilamb.rst index 24cc19c6..23cecd40 100644 --- a/docs/source/user_guide/tasks/ilamb.rst +++ b/docs/source/user_guide/tasks/ilamb.rst @@ -57,8 +57,20 @@ Parameters - Name of the ``[ts]`` land subtask to depend on. Inferred as ``land_monthly`` if not specified. +Dependencies +------------ + +**Upstream (what ilamb depends on):** + +- :doc:`ts` -- Monthly-lnd ts: required. Monthly-atm ts: optional. +- :doc:`e3sm_to_cmip` — Monthly-lnd e3sm_to_cmip: required. Monthly-atm e3sm_to_cmip: optional. + +**Downstream (what depends on ilamb):** + +- None. + Dependency inference --------------------- +~~~~~~~~~~~~~~~~~~~~ When ``infer_section_parameters = True`` (the default), ``zppy`` will automatically determine the following if not explicitly set: diff --git a/docs/source/user_guide/tasks/livvkit.rst b/docs/source/user_guide/tasks/livvkit.rst index 4a1bcbd3..acce900a 100644 --- a/docs/source/user_guide/tasks/livvkit.rst +++ b/docs/source/user_guide/tasks/livvkit.rst @@ -40,8 +40,20 @@ Parameters - ``"cmb,smb,energy_racmo,energy_era5,energy_merra2,energy_ceres"`` - Validation sets to perform. +Dependencies +------------ + +**Upstream (what livvkit depends on):** + +- :doc:`climo` — Monthly-lnd climo +- :doc:`ts` -- Monthly-lnd ts. + +**Downstream (what depends on livvkit):** + +- None. + Climo dependency requirements ------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``livvkit`` task requires specific ``[climo]`` subtasks to be defined: diff --git a/docs/source/user_guide/tasks/mpas_analysis.rst b/docs/source/user_guide/tasks/mpas_analysis.rst index 83211ed1..6ba5db7c 100644 --- a/docs/source/user_guide/tasks/mpas_analysis.rst +++ b/docs/source/user_guide/tasks/mpas_analysis.rst @@ -156,3 +156,14 @@ the test and reference runs: same cfg, the reference year ranges default to that subsection's values. - If a single reference year range is provided and multiple test ranges are requested, the single reference range is used for each test range. + +Dependencies +------------ + +**Upstream (what mpas_analysis depends on):** + +- None. Note that later year sets will depend on previously completed ones. + +**Downstream (what depends on mpas_analysis):** + +- :doc:`global_time_series` — Monthly-atm-glb ts: the 3 classic ocn plots, plots for specific ocn variables. diff --git a/docs/source/user_guide/tasks/pcmdi_diags.rst b/docs/source/user_guide/tasks/pcmdi_diags.rst index 7fd41916..c72d7b6e 100644 --- a/docs/source/user_guide/tasks/pcmdi_diags.rst +++ b/docs/source/user_guide/tasks/pcmdi_diags.rst @@ -216,3 +216,16 @@ variability_modes parameters - No - ``"monthly"`` - Seasons to analyze. + +Dependencies +------------ + +**Upstream (what pcmdi_diags depends on):** + +- :doc:`ts` — Monthly-atm ts: mean_climate, variability_modes_atm,variability_modes_cpl, enso +- :doc:`e3sm_to_cmip` — Monthly-atm e3sm_to_cmip: mean_climate, variability_modes_atm,variability_modes_cpl, enso +- Note that the synthetic_plots set depends on the other sets. + +**Downstream (what depends on pcmdi_diags):** + +- None. diff --git a/docs/source/user_guide/tasks/tc_analysis.rst b/docs/source/user_guide/tasks/tc_analysis.rst index d1c793bc..bc1b6d4d 100644 --- a/docs/source/user_guide/tasks/tc_analysis.rst +++ b/docs/source/user_guide/tasks/tc_analysis.rst @@ -34,7 +34,7 @@ Parameters attempts to infer it from the topography file. Inherited common parameters most relevant to ``tc_analysis`` ------------------------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 @@ -72,3 +72,14 @@ Inherited common parameters most relevant to ``tc_analysis`` - No - ``"02:00:00"`` - Maximum wall time for the SLURM job. + +Dependencies +------------ + +**Upstream (what tc_analysis depends on):** + +- None. + +**Downstream (what depends on tc_analysis):** + +- :doc:`e3sm_diags` — Required for tc_analysis set. diff --git a/docs/source/user_guide/tasks/ts.rst b/docs/source/user_guide/tasks/ts.rst index fd8c023b..5abbd493 100644 --- a/docs/source/user_guide/tasks/ts.rst +++ b/docs/source/user_guide/tasks/ts.rst @@ -97,7 +97,7 @@ The following parameters are specific to the ``ts`` task. this is left empty. Inherited common parameters most relevant to ``ts`` ---------------------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 @@ -156,3 +156,19 @@ Inherited common parameters most relevant to ``ts`` - No - ``""`` - SLURM account to charge. + +Dependencies +------------ + +**Upstream (what ts depends on):** + +- None. + +**Downstream (what depends on ts):** + +- :doc:`e3sm_to_cmip` — Monthly-atm ts: monthly-atm e3sm_to_cmip. Monthly-lnd ts: monthly-lnd e3sm_to_cmip. +- :doc:`e3sm_diags` — Monthly-atm ts: enso_diags, qbo, area_mean_time_series, mp_partition. Monthly-rof ts: streamflow. Daily-atm ts: tropical_subseasonal, precip_pdf. +- :doc:`global_time_series` — Monthly-atm-glb ts: the 5 classic atm plots, plots for specific atm variables. Monthly-lnd-glb ts: plots for specific lnd variables. +- :doc:`ilamb` -- Monthly-lnd ts: required. Monthly-atm ts: optional. +- :doc:`livvkit` -- Monthly-lnd ts. +- :doc:`pcmdi_diags` — Monthly-atm ts: mean_climate, variability_modes_atm,variability_modes_cpl, enso From aa1005822c35ac0d23f97c672935bf30df267cf6 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Fri, 5 Jun 2026 16:32:53 -0700 Subject: [PATCH 07/27] Add specific testing instructions --- .../dev_guide/test_expected_results.rst | 116 ++++ docs/source/dev_guide/tests.rst | 527 ++++++++++++++++++ 2 files changed, 643 insertions(+) create mode 100644 docs/source/dev_guide/test_expected_results.rst create mode 100644 docs/source/dev_guide/tests.rst diff --git a/docs/source/dev_guide/test_expected_results.rst b/docs/source/dev_guide/test_expected_results.rst new file mode 100644 index 00000000..a4ab6aeb --- /dev/null +++ b/docs/source/dev_guide/test_expected_results.rst @@ -0,0 +1,116 @@ +.. _updating-expected-results: + +*************************************** +Updating expected results for the tests +*************************************** + + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + machine_name=chrysalis + repo_parent_dir=~/ez/ # Or wherever you keep your repos + expected_results_dir=/lcrc/group/e3sm/public_html/zppy_test_resources + expected_results_records_dir=/lcrc/group/e3sm/public_html/zppy_test_resources_previous + + launch_compute_node() + { + salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm + } + +Compy: + +.. code-block:: bash + + machine_name=compy + repo_parent_dir=~/ez/ # Or wherever you keep your repos + expected_results_dir=/compyfs/www/zppy_test_resources + expected_results_records_dir=/compyfs/fors729/zppy_test_resources_previous + + launch_compute_node() + { + salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm + } + +Note that Compy doesn't give write access to ``/compyfs/www/``, so we can't add a new directory there. That's why ``zppy_test_resources_previous`` is in a separate path. + +Perlmutter: + +.. code-block:: bash + + machine_name=pm-cpu + repo_parent_dir=~/ez/ # Or wherever you keep your repos + expected_results_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources + expected_results_records_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources_previous + + launch_compute_node() + { + salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm + } + + +Process +~~~~~~~ + +.. code-block:: bash + + # First, let's copy over the old expected results ############################# + cp -r ${expected_results_dir} ${expected_results_records_dir}/expected_results_until_yyyymmdd + # Use today's date + # Chrysalis -- takes between 20 and 60 minutes + + # Second, update the expected results ######################################### + # Let's update the simpler tests' results first: + cd ${repo_parent_dir}/zppy + git status + # You might have changed branches since you ran the tests. + # Make sure you're now back on the correct branch: test-zppy-yyyymmdd + # Also confirm you're back in the correct env: zppy-yyyymmdd or the Unified env + + # Make sure the update script permissions are set up + chmod 755 tests/integration/generated/update_bash_generation_expected_files_${machine_name}.sh + chmod 755 tests/integration/generated/update_campaign_expected_files_${machine_name}.sh + chmod 755 tests/integration/generated/update_defaults_expected_files_${machine_name}.sh + chmod 755 tests/integration/generated/update_weekly_expected_files_${machine_name}.sh + + # These scripts update the expected results and re-run the tests: + ./tests/integration/generated/update_bash_generation_expected_files_${machine_name}.sh + ./tests/integration/generated/update_campaign_expected_files_${machine_name}.sh + ./tests/integration/generated/update_defaults_expected_files_${machine_name}.sh + + # Theis script only updates the expected results + ./tests/integration/generated/update_weekly_expected_files_${machine_name}.sh + # Chrysalis -- takes about 40 minutes + # Perlmutter -- takes about 25 minutes + ls ${expected_results_dir} + # Confirm there are expected results subdirs and image lists for each cfg + + cd ${repo_parent_dir}/zppy + pytest tests/integration/test_bundles.py + launch_compute_node + + start_bash_subshell + # EITHER: + # Activate EITHER a dev environment or the Unified env: + conda activate zppy-yyyymmdd + # OR: the command from `activate_unified_env` + + pytest tests/integration/test_images.py + exit # Exit bash shell + exit # Exit compute note + +Only do this part if you're updating the Official Results (i.e., the results we'd expect from a specific ``zppy`` version): + +.. code-block:: bash + + alias this_release=unified_1.13.0 # Or whatever the release is + + # Third, let's copy the expected results into a special official results dir ## + cp -r ${expected_results_dir} ${expected_results_records_dir}/expected_results_for_${this_release} + # Compy -- takes about 1h20m + ls ${expected_results_records_dir}/expected_results_for_${this_release} + # Confirm there are expected results subdirs and image lists for each cfg diff --git a/docs/source/dev_guide/tests.rst b/docs/source/dev_guide/tests.rst new file mode 100644 index 00000000..5d0a4678 --- /dev/null +++ b/docs/source/dev_guide/tests.rst @@ -0,0 +1,527 @@ +.. _testing-zppy: + +************* +Testing zppy +************* + +Follow the steps below to test ``zppy``. As you do so, please produce a Markdown report summarizing your results. + +Step 1: Determine what the current expected results are +======================================================= + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + expected_results_dir=/lcrc/group/e3sm/public_html/zppy_test_resources + expected_results_records_dir=/lcrc/group/e3sm/public_html/zppy_test_resources_previous + +Compy: + +.. code-block:: bash + + expected_results_dir=/compyfs/www/zppy_test_resources + expected_results_records_dir=/compyfs/fors729/zppy_test_resources_previous + +Note that Compy doesn't give write access to ``/compyfs/www/``, so we can't add a new directory there. That's why ``zppy_test_resources_previous`` is in a separate path. + +Perlmutter: + +.. code-block:: bash + + expected_results_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources + expected_results_records_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources_previous + +Process +~~~~~~~ + +.. code-block:: bash + + ls -lt ${expected_results_dir} + +In your Markdown report, note the date the expected results were last updated. + +Step 2: Determine what the current expected results are +======================================================= + +Now that we know the date the expected results are from, we can review what changes we'll be testing. + +Review each of the following commit logs and note commits made since the date the expected results were updated: + +* For the ``e3sm_to_cmip`` task: `e3sm_to_cmip `_ +* For the ``e3sm_diags`` task: `e3sm_diags `_ +* For the ``mpas_analysis`` task: `MPAS-Analysis `_ +* For the ``global_time_series`` and ``pcmdi_diags`` tasks: `zppy-interfaces `_ +* For ``zppy`` itself: `zppy `_ + +For the remaining tasks (``climo``, ``ts``, ``tc_analysis``, ``ilamb``, ``livvkit``), we typically just use the associated package's latest release rather than making dev environments. As such, their latest development will have no impact on our tests unless we have started using one of their newer releases. + +In your Markdown report, make a table like: + +.. code-block:: + + | Package | Changes since expected results were updated | + | --- | --- | + | [package name](link to package's commit log) | Links to all PRs merged since the expected results were updated | + ... + +Step 3: Set up environments for called packages +=============================================== + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + repo_parent_dir=~/ez/ # Or wherever you keep your repos + + start_bash_subshell() + { + bash + source ~/.bashrc # Or wherever you have your aliases, etc. defined + } + + activate_dev_env() + { + env_name=$1 + + lcrc_conda # Or however you activate conda + rm -rf build + conda clean --all --y + conda env create -f conda/dev.yml -n ${env_name} + conda activate ${env_name} + pre-commit run --all-files # Confirm this passes + python -m pip install . + } + + activate_unified_env() + { + source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh + } + +Compy: + +.. code-block:: bash + + repo_parent_dir=~/ez/ # Or wherever you keep your repos + + start_bash_subshell() + { + bash + source ~/.bash_profile # Or wherever you have your aliases, etc. defined + } + + activate_dev_env() + { + env_name=$1 + + compy_conda # Or however you activate conda + rm -rf build + conda clean --all --y + conda env create -f conda/dev.yml -n ${env_name} + conda activate ${env_name} + pre-commit run --all-files # Confirm this passes + python -m pip install . + } + + activate_unified_env() + { + source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh + } + +Perlmutter: + +.. code-block:: bash + + repo_parent_dir=~/ez/ # Or wherever you keep your repos + + start_bash_subshell() + { + bash + source ~/.bash_profile.ext # Or wherever you have your aliases, etc. defined + } + + activate_dev_env() + { + env_name=$1 + + nersc_conda # Or however you activate conda + rm -rf build + conda clean --all --y + conda env create -f conda/dev.yml -n ${env_name} + conda activate ${env_name} + pre-commit run --all-files # Confirm this passes + python -m pip install . + } + + activate_unified_env() + { + source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh + } + +Process +~~~~~~~ + +.. code-block:: bash + + set_up_repo() + { + repo_name=$1 + main_branch_name=$2 + + cd ${repo_parent_dir}/${repo_name} + git status # Check for uncommitted changes + + # If there are uncommitted changes, + # commit them so we can move cleanly to a new branch: + git add -A + git commit -m "Checkpoint" + + git fetch upstream ${main_branch_name} # This assumes you've named your remote for the main repo as "upstream" + git checkout ${main_branch_name} + git reset --hard upstream/${main_branch_name} + git log --oneline | head -n 1 + # Check that this matches the corresponding commit log: + # https://github.com/E3SM-Project/e3sm_to_cmip/commits/master + # https://github.com/E3SM-Project/e3sm_diags/commits/main + # https://github.com/MPAS-Dev/MPAS-Analysis/commits/develop/ + # https://github.com/E3SM-Project/zppy-interfaces/commits/main + + # Activate EITHER a dev environment or the Unified env: + # Dev environment -- test latest development + # Unified environment -- test latest Unified environment + activate_dev_env ${repo_name}-yyyymmdd # Use today's date + # OR: activate_unified_env ${repo_name}-yyyymmdd # Use today's date + } + + +.. code-block:: bash + + start_bash_subshell + set_up_repo e3sm_to_cmip master + exit # Exit bash subshell + + start_bash_subshell + set_up_repo e3sm_diags main + exit # Exit bash subshell + + start_bash_subshell + set_up_repo MPAS-Analysis develop + exit # Exit bash subshell + + start_bash_subshell + set_up_repo zppy-interfaces main + # Since zppy-interfaces is so integrated into `zppy`, + # run its unit tests as well: + pytest tests/unit/global_time_series/test_*.py + pytest tests/unit/pcmdi_diags/test_*.py + exit # Exit bash subshell + +Step 4: Set up zppy environment +=============================== + +.. code-block:: bash + + cd ${repo_parent_dir}/zppy + git status # Check for uncommitted changes + + # If there are uncommitted changes, + # commit them so we can move cleanly to a new branch: + git add -A + git commit -m "Checkpoint" + + git fetch upstream main # This assumes you've named your remote for the main repo as "upstream" + git checkout -b test-zppy-yyyymmdd upstream/main # Use today's date + git log --oneline | head -n 1 + # Check that this matches the corresponding commit log: + # https://github.com/E3SM-Project/zppy/commits/main + + start_bash_subshell + # Activate EITHER a dev environment or the Unified env: + # Dev environment -- test latest development + # Unified environment -- test latest Unified environment + activate_dev_env zppy-yyyymmdd # Use today's date + # OR: activate_unified_env zppy-yyyymmdd # Use today's date + + # Note the Python version being used + # If you activated dev env: + conda list python + # If you activated unified env: + pixi list python + + # Run zppy unit tests + pytest tests/test_*.py # 44 passed in 0.87s + +Step 5: Launch zppy jobs +======================== + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + conda_setup_cmd=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh + output_dir=/lcrc/group/e3sm/ac.forsyth2 + +Compy: + +.. code-block:: bash + + conda_setup_cmd=source /qfs/people/fors729/miniforge3/etc/profile.d/conda.sh + output_dir=/compyfs/fors729 + +Perlmutter: + +.. code-block:: bash + + conda_setup_cmd=source /global/homes/f/forsyth/miniforge3/etc/profile.d/conda.sh + output_dir=/global/cfs/cdirs/e3sm/forsyth + +Process +~~~~~~~ + +First, let's edit ``tests/integration/utils.py``. + +In place of ech ``{env_cmd}`` put either ``${conda_setup_cmd}; conda activate ${repo_name}-yyyymmdd`` or the command from ``activate_unified_env``. You may want to use dev environments for some tasks and the Unified environment for others. + +You can comment out ``cfg``s from ``cfgs_to_run`` to run fewer configuration files. Likewise, you can comment out tasks from ``tasks_torun`` to run fewer tasks. + +Be sure to set the ``unique_id``; this allows us to avoid path name collisions. + +.. code-block:: python + + TEST_SPECIFICS: Dict[str, Any] = { + # This is the NCO path. + # Keep as "" to use the production-version NCO commands. + # Set to a specific path to use development-version NCO commands. + "nco_path": "", + # These are custom environment_commands for specific tasks. + # Never set these to "", because they will print the line + # `environment_commands = ""` for the corresponding task, + # thus overriding the value set higher up in the cfg. + # That is, there will be no environment set. + # (`environment_commands = ""` only redirects to Unified + # if specified under the [default] task) + "e3sm_to_cmip_environment_commands": "{env_cmd}", + "diags_environment_commands": "{env_cmd}", + "mpas_analysis_environment_commands": "{env_cmd}", + "global_time_series_environment_commands": "{env_cmd}", + "livvkit_environment_commands": "{env_cmd}", + "pcmdi_diags_environment_commands": "{env_cmd}", + # This is the environment setup for other tasks. + # Leave as "" to use the latest Unified environment. + "environment_commands": "{env_cmd}", + # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs + "cfgs_to_run": [ + "weekly_bundles", # Typically, we run on Chrysalis, Compy + "weekly_comprehensive_v2", # Typically, we run on Chrysalis, Compy + "weekly_comprehensive_v3", # Typically, we run on all 3 machines + "weekly_legacy_3.1.0_bundles", # Typically, we run on Chrysalis + "weekly_legacy_3.1.0_comprehensive_v2", # Typically, we run on Chrysalis + "weekly_legacy_3.1.0_comprehensive_v3", # Typically, we run on Chrysalis + "weekly_legacy_3.0.0_bundles", # Typically, we run on Chrysalis + "weekly_legacy_3.0.0_comprehensive_v2", # Typically, we run on Chrysalis + "weekly_legacy_3.0.0_comprehensive_v3", # Typically, we run on Chrysalis + ], + "tasks_to_run": [ + "e3sm_diags", + "mpas_analysis", + "global_time_series", + "ilamb", + "livvkit", + "pcmdi_diags", + ], + "unique_id": "test_zppy_yyyymmdd", # Use today's date + } + +.. code-block:: bash + + git diff # Check that the diff looks as you expect + python tests/integration/utils.py + # This will generate the actual test cfgs based off the templates. + + # Set up an alias for checking jobs: + alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %.8Q %j" --sort=P,-t,-p' + alias sq='sqa -u $USER' + + sq + # Check that you have no jobs currently queued. + # It's ok if you do, but it makes counting remaining zppy jobs easier if you don't have any existing jobs. + + # Typically run on Chrysalis, Compy, Perlmutter: + zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg + + # Typically run on Chrysalis, Compy: + zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg + + # Typically run on Chrysalis: + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg + + + sq | wc -l # This includes the header row, so subtract 1 to get the number of jobs you have running + # WAIT until that returns 1 (i.e., 0 jobs running) + + +Step 6: Launch zppy jobs -- bundles part 2 +========================================== + +This section is only relevant only if you're running the ``_bundles_`` jobs. Only run the lines relevant to the jobs you launched in step 5. + +.. code-block:: bash + + # Check on bundles status + cd ${output_dir}/zppy_weekly_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.1.0_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.0.0_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + # Now, run bundles part 2 + cd ${repo_parent_dir}/zppy + git status + # You might have changed branches while you were waiting for jobs to finish. + # Make sure you're now back on the correct branch: test-zppy-yyyymmdd + # Also confirm you're back in the correct env: zppy-yyyymmdd or the Unified env + + + zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg + zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg + sq | wc -l + # WAIT until that returns 1 (i.e., 0 jobs running) + +Step 7: Review finished returns +=============================== + +Only run the lines relevant to the jobs you launched in steps 5/6. + +.. code-block:: bash + + ### v2 ### + cd ${output_dir}/zppy_weekly_comprehensive_v2_output/${unique_id}/v2.LR.historical_0201/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/${unique_id}/v2.LR.historical_0201/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/${unique_id}/v2.LR.historical_0201/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + ### v3 ### + cd ${output_dir}/zppy_weekly_comprehensive_v3_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + ### bundles ### + cd ${output_dir}/zppy_weekly_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.0.0_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + + cd ${output_dir}/zppy_weekly_legacy_3.1.0_bundles_output/${unique_id}/v3.LR.historical_0051/post/scripts + grep -v "OK" *status # Confirm no non-OK statuses appear + +In your Markdown report, any of the output subdirectories that had non-OK statuses. + +Step 8: Run Python tests +======================== + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm + } + +Compy: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm + } + +Perlmutter: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm + } + +Process +~~~~~~~ + +.. code-block:: bash + + cd ${repo_parent_dir}/zppy + git status + # You might have changed branches while you were waiting for jobs to finish. + # Make sure you're now back on the correct branch: test-zppy-yyyymmdd + # Also confirm you're back in the correct env: zppy-yyyymmdd or the Unified env + + # This test doesn't make use of an expected results directory. + pytest tests/integration/test_last_year.py + + # These tests do make use of an expected results directory. + # That is, the expected results may need to be updated if expected behavior has changed. + pytest tests/integration/test_bash_generation.py + pytest tests/integration/test_campaign.py + pytest tests/integration/test_defaults.py + + # These tests make use of an expected results directory + # AND rely on the jobs we just ran: + # 1. The bundles test: + pytest tests/integration/test_bundles.py + # 2. The image checker test, which we'll run from a compute node: + launch_compute_node + + start_bash_subshell + # EITHER: + # Activate EITHER a dev environment or the Unified env: + conda activate zppy-yyyymmdd + # OR: the command from `activate_unified_env` + + pytest tests/integration/test_images.py + # Typically takes between 10 and 20 minutes on Chrysalis and Perlmutter. + # Typically takes closer to 50 minutes on Compy. + cat test_images_summary.md + exit # Exit bash shell + exit # Exit compute note + +In your Markdown report: + +* From the ``pytest tests/integration/test_images.py `` command-line output, copy everything after ``Captured stdout call`` to a code block labeled "Output" +* Copy the results of ``cat test_images_summary.md`` to a section labeled "Complete summary table" +* Make a new section named "Summary table -- only failing image-check tests, sorted by task". For each task that has missing and/or mismatched images, copy the relevant rows from the summary table. Skip this section if there were no failing image-check tests. +* Note any test failures from the other Python tests. +* If there were no failures at all, print "All tests pass" From e90e878e40531c26f0262d955ff23b95c022b918 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Fri, 5 Jun 2026 16:35:33 -0700 Subject: [PATCH 08/27] Remove older test docs --- docs/source/dev_guide/release_testing.rst | 150 ------- docs/source/dev_guide/testing.rst | 319 --------------- .../source/dev_guide/testing_e3sm_unified.cfg | 93 ----- .../tutorial_testing_e3sm_unified.rst | 372 ------------------ 4 files changed, 934 deletions(-) delete mode 100644 docs/source/dev_guide/release_testing.rst delete mode 100644 docs/source/dev_guide/testing.rst delete mode 100644 docs/source/dev_guide/testing_e3sm_unified.cfg delete mode 100644 docs/source/dev_guide/tutorial_testing_e3sm_unified.rst diff --git a/docs/source/dev_guide/release_testing.rst b/docs/source/dev_guide/release_testing.rst deleted file mode 100644 index 7f0f5a84..00000000 --- a/docs/source/dev_guide/release_testing.rst +++ /dev/null @@ -1,150 +0,0 @@ -*************************************** -Testing directions for making a release -*************************************** - -1. Have three shells open: one on Chrysalis, one on Compy, and one on Perlmutter. Do the following steps on each machine. - -2. ``cd`` to the ``zppy`` directory. - -3. Check out a branch to test on. - - a. test dev (run before making a new zppy RC) :: - - git fetch upstream main - git checkout -b test_zppy_pre_#.#.#rc#_ upstream/main - git log # check the commits match https://github.com/E3SM-Project/zppy/commits/main - - b. test new Unified RC :: - - git fetch upstream main - git checkout -b test_unified_#.#.#rc#_ upstream/main - git log # check the commits match https://github.com/E3SM-Project/zppy/commits/main - - c. test final Unified :: - - git fetch upstream main - git checkout -b test_unified_#.#.#rc#_ upstream/main - git log # check the commits match https://github.com/E3SM-Project/zppy/commits/main - -4. Set up a dev environment for E3SM Diags. This is used for the ``diags_environment_commands`` parameter. Normally, this is only used for checking ``environment_commands`` works properly. However, by modifying the test templates, it's possible to run other E3SM Diags subtasks in a different environment. This can be useful if a bug fix in E3SM Diags hasn't been incorporated into the latest Unified RC yet. :: - - # `cd` to e3sm_diags directory - $ git checkout main - $ git fetch upstream - $ git reset --hard upstream/main - $ git log # Should match https://github.com/E3SM-Project/e3sm_diags/commits/main - $ mamba clean --all - $ mamba env create -f conda-env/dev.yml -n e3sm_diags_ - $ conda activate e3sm_diags_ - $ pip install . - $ `cd` back to zppy directory - -5. Make sure you're using the latest packages. In ``tests/integration/utils.py``, do the following: - - a. test dev (run before making a new zppy RC): - - * If it's the first zppy RC (i.e., a Unified RC hasn't been made yet): Change the last line ``generate_cfgs()`` to ``generate_cfgs(unified_testing=False)``. We want to use the latest officially released E3SM Unified environment. - * Otherwise: follow the (b: test new Unified RC) directions for this step. - - - b. test new Unified RC - - * Change the last line ``generate_cfgs()`` to ``generate_cfgs(unified_testing=True)``. We **don't** want to use the latest officially released E3SM Unified environment. - * Update ``environment_commands_test`` for the machine you're currently working on. If this is not done, then ``zppy`` will use the existing value, which is likely an old E3SM Unified RC launch script that doesn't even exist anymore. This parameter should be set to source the latest E3SM Unified RC launch script, since E3SM Unified is the only environment that all ``zppy`` tasks can run from. For instance, ``ncclimo`` isn't in the ``zppy`` dev environment. - - - c. test final Unified - - * Change the last line ``generate_cfgs()`` to ``generate_cfgs(unified_testing=False)``. We want to use the latest officially released E3SM Unified environment. - - For a, b, and c, always: - - * Update ``diags_environment_commands`` for the machine you're currently working on to use the environment created in the above step. If this is not done, the ``atm_monthly_180x360_aave_environment_commands`` subtask will use an older dev environment for E3SM Diags than we want. - * Update ``UNIQUE_ID`` to be a short description of what you're testing. - -6. Set up your environment. - - a. test dev (run before making a new zppy RC): Set up a new development environment. This ensures that testing will use the latest conda changes. Note that you will need to run ``conda remove -n zppy_dev_pre_zppy_#.#.#rc# --all`` first if you have previously done this step. :: - - mamba clean --all - mamba env create -f conda/dev.yml -n zppy_dev_pre_#.#.#rc# - conda activate zppy_dev_pre_#.#.#rc# - pip install . - - b. test new Unified RC: Launch the E3SM Unified environment for the machine you're on. Change out the version numbers below. - - * Chrysalis: ``source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_#.#.#rc#_chrysalis.sh`` - * Compy: ``source /share/apps/E3SM/conda_envs/test_e3sm_unified_#.#.#rc#_compy.sh`` - * Perlmutter: ``source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_#.#.#rc#_pm-cpu.sh`` - - c. test final Unified: Launch the latest E3SM Unified environment for the machine you're on. - - * Chrysalis: ``source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh`` - * Compy: ``source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh`` - * Perlmutter: ``source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh`` - -7. Run the unit tests with ``pytest tests/test_*.py``. - - a. test dev (run before making a new zppy RC): - - * If there are any failures, fix the code (or tests). If you make any conda changes, go back to step 6a. If you otherwise change zppy source code, run ``pip install .`` and then redo step 7. If you only make changes to tests, you can immediately redo step 7. - - b. test new Unified RC: - - * If there are any failures, fix the code and go back to step 1, following the (a: test dev (run before making a new zppy RC)) directions. - - c. test final Unified: - - * There should be no failures. If there are, a patch release of E3SM Unified may be required. - - For a, b, and c: - - * If there are no failures, proceed to the next step. - -8. Run the "Commands to run before running integration tests" for the current machine. To ensure you don't encounter issues from running ``zppy`` commands simultaneously, wait for all the automatically launched jobs from one run to finish before running ``zppy`` again. - - * `Chrysalis `_ - * `Compy `_ - * `Perlmutter `_ - -9. Run the integration tests with ``pytest tests/integration/test_*.py``. Note that ``test_complete_run.py`` takes approximately 75 minutes to run on Compy. - - a. test dev (run before making a new zppy RC): - - * If there are any unexpected failures, fix the code (or tests). If you make any conda changes, go back to step 6a. If you otherwise change zppy source code, run ``pip install .`` and then go back to step 7. If you only make changes to tests, you can immediately redo step 9. - - b. test new Unified RC: - - * If there are any unexpected failures, fix the code and go back to step 1, following the (a: test dev (run before making a new zppy RC)) directions. - - c. test final Unified: - - * There should be no unexpected failures. If there are, a patch release of E3SM Unified may be required. - - For a, b, and c: - - * If there are only expected failures, then update the expected files. Use the "Commands to run to replace outdated expected files" from the links on step 8. Then repeat step 9. - * If there are no failures at all, proceed to the next step. - -10. Run ``git diff``. All of your changes should be from editing ``tests/integration/utils.py`` in step 5, and running it in step 8. - - * If this is the case, you can delete this testing branch. - * If not, you have probably made code changes to get the tests to pass. Make a pull request to merge the changes. Add the "semver: bug" label. - -11. Wrap up release testing: - - a. test dev (run before making a new zppy RC): Create the next zppy RC by following the "release candidates" directions at https://e3sm-project.github.io/zppy/_build/html/main/dev_guide/release.html. - - b. test new Unified RC: Create the next zppy release following the "production releases" directions at https://e3sm-project.github.io/zppy/_build/html/main/dev_guide/release.html. - - c. test final Unified: Run the "Commands to generate official expected results for a release" from step 8. This will create a baseline of expected files for the release. You can now safely remove old branches and environments. At https://github.com/E3SM-Project/zppy/branches, delete any branches that are no longer needed. Also, run: :: - - # Branches - $ cd - $ git branch # Look at all branch names - $ git branch -D - - # Environments - $ conda env list - # For each environment you want to delete, run: - $ conda remove -n --all diff --git a/docs/source/dev_guide/testing.rst b/docs/source/dev_guide/testing.rst deleted file mode 100644 index 59ba1778..00000000 --- a/docs/source/dev_guide/testing.rst +++ /dev/null @@ -1,319 +0,0 @@ -.. _testing-zppy: - -************* -Testing zppy -************* - -Unit tests -========== - -Unit tests live in ``tests/test_*.py``. Run them with: - -.. code-block:: bash - - pip install . # or: python -m pip install . - pytest tests/test_*.py - -Integration tests -================= - -Integration tests can be run on Chrysalis, Compy, or Perlmutter. They -require an active E3SM Unified environment and access to simulation data. - -Machine-specific directions (generated setup commands, expected output files, -etc.) live in: -`tests/integration/generated/ `_ - -Run all integration tests: - -.. code-block:: bash - - pip install . - pytest tests/integration/test_*.py - -.. note:: - Image comparison tests require a compute node: - - .. code-block:: bash - - salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm - pytest tests/integration/test_images.py - cat test_images_summary.md - -Automated tests (CI) -==================== - -Unit tests run automatically via :ref:`GitHub Actions ` on every -pull request and push to ``main``. Integration tests must be run manually -on a supported machine. - -.. _release-testing: - -Full release testing directions -================================ - -This section describes the complete process for testing ``zppy`` before -making a release. See also :doc:`tutorial_testing_e3sm_unified` for a legacy -tutorial. - -Overview --------- - -Three testing scenarios exist: - -a. **test dev** — run before making a new zppy RC (no new Unified RC yet) -b. **test new Unified RC** — run when a new E3SM Unified RC has been made -c. **test final Unified** — run after E3SM Unified is finalized - -Tests should be run on **three machines**: Chrysalis, Compy, and Perlmutter. - -Step 1: Set up zppy-interfaces (if needed) -------------------------------------------- - -Update ``zppy-interfaces`` to the latest main branch: - -.. code-block:: bash - - cd ~/ez/zppy-interfaces - git status # should be clean - git fetch upstream main - git checkout main - git reset --hard upstream/main - git log --oneline | head -n 1 - # Verify matches https://github.com/E3SM-Project/zppy-interfaces/commits/main - - bash # isolated subshell - source ~/.bashrc - source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified__chrysalis.sh # e.g., 1.13.0rc10 - - # Verify zppy-interfaces tests - pytest tests/unit/global_time_series/test_*.py - pytest tests/unit/pcmdi_diags/test_*.py - -Step 2: Check out the zppy branch ------------------------------------ - -.. code-block:: bash - - cd ~/ez/zppy - git status # ensure clean workspace - - git fetch upstream main - git checkout -b test-zppy-- upstream/main - git log --oneline | head -n 3 - # Verify matches https://github.com/E3SM-Project/zppy/commits/main - -Step 3: Set up the environment -------------------------------- - -a. **test dev**: create a new dev environment - - .. code-block:: bash - - mamba clean --all - mamba env create -f conda/dev.yml -n zppy_dev_pre_rc - conda activate zppy_dev_pre_rc - pip install . - -b. **test new Unified RC**: activate the Unified RC - - - Chrysalis: ``source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh`` - - Compy: ``source /share/apps/E3SM/conda_envs/test_e3sm_unified_rc_compy.sh`` - - Perlmutter: ``source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_rc_pm-cpu.sh`` - -c. **test final Unified**: activate the latest released Unified - - - Chrysalis: ``source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh`` - - Compy: ``source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh`` - - Perlmutter: ``source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh`` - -Step 4: Configure ``tests/integration/utils.py`` --------------------------------------------------- - -Edit ``tests/integration/utils.py`` to set the correct environment -and unique ID. Example ``TEST_SPECIFICS`` dictionary: - -.. code-block:: python - - TEST_SPECIFICS: Dict[str, Any] = { - # NCO path: "" uses production-version NCO commands - # Set to a specific path to use development-version NCO commands - "nco_path": "", - # Environment commands for specific tasks. - # Never set these to ""; that would override the task's environment - # with a blank string (no environment). - "e3sm_to_cmip_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", - "diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", - "mpas_analysis_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", - "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", - "livvkit_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", - "pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", - # General environment for other tasks. - # Leave as "" to use the latest Unified environment. - "environment_commands": "source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_rc_chrysalis.sh", - # For a complete test, run the set of latest cfgs AND at least one set of legacy cfgs - "cfgs_to_run": [ - "weekly_bundles", - "weekly_comprehensive_v2", - "weekly_comprehensive_v3", - "weekly_legacy_3.1.0_bundles", - "weekly_legacy_3.1.0_comprehensive_v2", - "weekly_legacy_3.1.0_comprehensive_v3", - "weekly_legacy_3.0.0_bundles", - "weekly_legacy_3.0.0_comprehensive_v2", - "weekly_legacy_3.0.0_comprehensive_v3", - ], - "tasks_to_run": [ - "e3sm_diags", - "mpas_analysis", - "global_time_series", - "ilamb", - "livvkit", - "pcmdi_diags", - ], - "unique_id": "zppy__rc_", - } - -Step 5: Run unit tests ------------------------ - -.. code-block:: bash - - pytest tests/test_*.py - -All tests should pass. Failures in scenario (a) require code fixes. -Failures in scenario (c) may indicate a patch release of E3SM Unified is -needed. - -Step 6: Generate cfg files and check quota -------------------------------------------- - -.. code-block:: bash - - python tests/integration/utils.py - # Review output to confirm settings - - # Check quota before launching - mmlsquota -u --block-size T fs2 - lcrc-quota - - ls tests/integration/generated/test_weekly_*_chrysalis.cfg - -Step 7: Launch integration jobs --------------------------------- - -To ensure no conflicts, wait for all jobs from one ``zppy`` run to finish -before launching another. Start with the most important: - -.. code-block:: bash - - # Launch all cfg files - zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg - zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg - zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg - - zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg - zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg - zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg - - zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg - zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg - zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg - - # Monitor job queue - sq | wc -l # subtract 1 for the header row - -Step 8: Run bundles a second time ----------------------------------- - -After all initial jobs complete, check for errors and run the bundles a -second time to complete remaining work: - -.. code-block:: bash - - # Check for errors first - cd /path/to/output/.../post/scripts - grep -v "OK" *status # Good if no output - - # Re-run bundles - zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg - zppy -c tests/integration/generated/test_weekly_legacy_3.1.0_bundles_chrysalis.cfg - zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg - -Step 9: Review all finished runs ----------------------------------- - -For each output directory, check for failures: - -.. code-block:: bash - - cd /path/to/zppy_weekly_comprehensive_v2_output/.../post/scripts - grep -v "OK" *status - - cd /path/to/zppy_weekly_comprehensive_v3_output/.../post/scripts - grep -v "OK" *status - - cd /path/to/zppy_weekly_bundles_output/.../post/scripts - grep -v "OK" *status - - # Repeat for all legacy output directories - -Step 10: Run Python integration tests ---------------------------------------- - -.. code-block:: bash - - pytest tests/integration/test_bash_generation.py - pytest tests/integration/test_campaign.py - pytest tests/integration/test_defaults.py - pytest tests/integration/test_bundles.py - pytest tests/integration/test_last_year.py - -For image comparison tests (requires a compute node): - -.. code-block:: bash - - salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm - bash - source ~/.bashrc - source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified__chrysalis.sh - pytest tests/integration/test_images.py - cat test_images_summary.md - -Step 11: Wrap up ------------------ - -After successful testing: - -a. **test dev**: proceed to :ref:`create the zppy RC `. - -b. **test new Unified RC**: proceed to create the zppy production release. - -c. **test final Unified**: run the "Commands to generate official expected - results for a release" from the machine-specific directions. Then clean - up branches and environments: - - .. code-block:: bash - - git branch - git branch -D - - conda env list - conda remove -n --all - - Update expected test result files if there are known/expected differences: - use the "Commands to run to replace outdated expected files" from the - machine-specific directions, then re-run the tests. - -Step 12: Record test results ------------------------------- - -Document test results in the `zppy testing discussion -`_ for the relevant -E3SM Unified release. Include: - -- Machine name and date -- E3SM Unified version tested -- zppy version/branch tested -- Results summary (pass/fail per cfg) -- Any known issues or caveats diff --git a/docs/source/dev_guide/testing_e3sm_unified.cfg b/docs/source/dev_guide/testing_e3sm_unified.cfg deleted file mode 100644 index ef1a4d5a..00000000 --- a/docs/source/dev_guide/testing_e3sm_unified.cfg +++ /dev/null @@ -1,93 +0,0 @@ -[default] -input = -input_subdir = archive/atm/hist -# The directory that the `post` directory is placed in. -output = -case = 20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis -www = -partition = -environment_commands = "" - -[climo] -active = True -years = "51:70:20", -# Use default vars - - [[ atm_monthly_180x360_aave ]] - mapping_file = - frequency = "monthly" - - [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - input_subdir = "archive/atm/hist" - input_files = "eam.h4" - mapping_file = - vars = "PRECT" - frequency = "diurnal_8xdaily" - -[ts] -active = True -years = "51:70:10", - - [[ atm_monthly_180x360_aave ]] - input_subdir = "archive/atm/hist" - input_files = "eam.h0" - frequency = "monthly" - mapping_file = - # Use default vars - - [[ atm_daily_180x360_aave ]] - input_subdir = "archive/atm/hist" - input_files = "eam.h1" - frequency = "daily" - mapping_file = - vars = "PRECT" - - [[ atm_monthly_glb ]] - input_subdir = "archive/atm/hist" - input_files = "eam.h0" - frequency = "monthly" - mapping_file = "glb" - # Use default vars - - [[ land_monthly ]] - input_subdir = "archive/lnd/hist" - input_files = "elm.h0" - frequency = "monthly" - mapping_file = - vars = "FSH,RH2M" - -[e3sm_diags] -active = True -years = "51:70:20", -ts_num_years = 10 -ref_start_yr = 1979 -ref_final_yr = 2016 - - [[ atm_monthly_180x360_aave ]] - short_name = "20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis" - grid = "180x360_aave" - reference_data_path = "" - obs_ts = "" - dc_obs_climo = "" - climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave" - climo_monthly_frequency = "diurnal_8xdaily" - -[mpas_analysis] -active = True -walltime = "24:00:00" -parallelTaskCount = 6 -ts_years = "51-70", -enso_years = "51-70", -climo_years ="51-70", -mesh = "EC30to60E2r2" -anomalyRefYear = 51 - -[global_time_series] -active = True -years = "51-70", -ts_num_years = 10 -figstr=coupled_v2rc3e -moc_file=mocTimeSeries_0051-0070.nc -experiment_name=20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis -ts_years = "51-70", -climo_years ="51-70", diff --git a/docs/source/dev_guide/tutorial_testing_e3sm_unified.rst b/docs/source/dev_guide/tutorial_testing_e3sm_unified.rst deleted file mode 100644 index ab59386d..00000000 --- a/docs/source/dev_guide/tutorial_testing_e3sm_unified.rst +++ /dev/null @@ -1,372 +0,0 @@ -.. _testing_e3sm_unified: - -************************************* -Testing zppy for E3SM Unified release -************************************* - -This tutorial covers testing ``zppy`` for an E3SM Unified release. - -.. warning :: - - Examples are from testing for E3SM Unified v1.5.0. - They may be outdated because of more recent changes. - For example, ``/lcrc/group/e3sm/public_html/diagnostics/observations/Atm/`` - has replaced ``/lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/``. - -Parameters -========== - -Only a few parameters in the configuration file need to change for different machines. -Be sure to change the file paths to fit your own workspaces. - -.. warning:: - - Also consider the space quotas or purge policies for the file paths you use. - You'll be storing about 2 TB of data. - -Chrysalis: - -- : /lcrc/group/e3sm/ac.forsyth2/E3SM_simulations/e3sm_unified_test_simulation/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis -- : /lcrc/group/e3sm/ac.forsyth2/E3SM_simulations/e3sm_unified_test_simulation/ -- : /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/E3SM/e3sm_unified_test_simulation/ -- : compute -- : source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.5.0rc8_chrysalis.sh -- : /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc -- : /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology -- : /lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series -- : /lcrc/group/e3sm/public_html/e3sm_diags_test_data/unit_test_complete_run/obs/climatology - -Compy: - -- : /compyfs/fors729/e3sm_unified_test_zppy/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis -- : /compyfs/fors729/e3sm_unified_test_zppy/ -- : /compyfs/www/fors729/E3SM/e3sm_unified_test_simulation/ -- : slurm -- : source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.5.0rc8_compy.sh -- : /compyfs/zender/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc -- : /compyfs/e3sm_diags_data/obs_for_e3sm_diags/climatology -- : /compyfs/e3sm_diags_data/obs_for_e3sm_diags/time-series -- : /compyfs/e3sm_diags_data/obs_for_e3sm_diags/climatology - -Cori: - -- : /global/cscratch1/sd/forsyth/e3sm_unified_test_zppy/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis -- : /global/cscratch1/sd/forsyth/e3sm_unified_test_zppy/ -- : /global/cfs/cdirs/e3sm/www/forsyth/E3SM/e3sm_unified_test_simulation/ -- : haswell -- : source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.5.0rc8_cori-haswell.sh -- : /global/homes/z/zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc -- : /global/cfs/cdirs/e3sm/acme_diags/obs_for_e3sm_diags/climatology -- : /global/cfs/cdirs/e3sm/acme_diags/obs_for_e3sm_diags/time-series -- : /global/cfs/cdirs/e3sm/acme_diags/obs_for_e3sm_diags/climatology - -These values don't appear in the configuration file but are still useful to specify -here: - -Chrysalis: - -- : https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.forsyth2/E3SM/e3sm_unified_test_simulation/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/ -- : /lcrc/group/e3sm/ac.forsyth2/zstash_dir -- : /lcrc/group/e3sm/ac.forsyth2/e3sm_unified_test_zstash/unified_test_cache - -Compy: - -- : https://compy-dtn.pnl.gov/fors729/E3SM/e3sm_unified_test_simulation/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/ -- : /compyfs/fors729/e3sm_unified_test_zstash -- : /compyfs/fors729/e3sm_unified_test_zstash/unified_test_cache - -Cori: - -- : http://portal.nersc.gov/cfs/e3sm/forsyth/E3SM/e3sm_unified_test_simulation/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/ -- : /global/cscratch1/sd/forsyth/e3sm_unified_test_zstash -- : /global/cscratch1/sd/forsyth/e3sm_unified_test_zstash/unified_test_cache - -Testing zppy on a small simulation -================================== - -1. Find a simulation to test on. In this example, we'll use a simulation on Chrysalis: -``/lcrc/group/e3sm/ac.golaz/E3SM_simulations/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/``. - -2. The simulation might be too large to reasonably test on. We can copy over a -subsection to a testing directory with a script like the following. In this example, -the simulation data starts at year 51. We want to have at least 20 years, so we'll -copy over years 51-79 using the pattern ``*.00[5-7]*``. This subsection turns out to be -about 2 terabytes in size. - - :: - - #!/bin/bash - - input=/lcrc/group/e3sm/ac.golaz/E3SM_simulations/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/ - output=e3sm_unified_test_simulation/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis/ - - # For E3SM Diags: About 40 minutes to run - # Copy over years xx5x - xx7x - cp ${input}archive/atm/hist/*.00[5-7]* ${output}archive/atm/hist - cp ${input}archive/cpl/hist/*.00[5-7]* ${output}archive/cpl/hist - cp ${input}archive/ice/hist/*.00[5-7]* ${output}archive/ice/hist - cp ${input}archive/lnd/hist/*.00[5-7]* ${output}archive/lnd/hist - cp ${input}archive/ocn/hist/*.00[5-7]* ${output}archive/ocn/hist - cp ${input}archive/rof/hist/*.00[5-7]* ${output}archive/rof/hist - - # For MPAS-Analysis: Less than a minute to run - cp ${input}run/mpaso_in ${output}run/mpaso_in - cp ${input}run/mpassi_in ${output}run/mpassi_in - cp ${input}run/streams.ocean ${output}run/streams.ocean - cp ${input}run/streams.seaice ${output}run/streams.seaice - cp ${input}run/*rst* ${output}run/ - - -3. Now we need to load ``zppy``. We first want to test out ``zppy`` from -the new unified environment release candidate. -We can do this by running ````. -See 3a-c below if errors are encountered in later steps. - -3a. If ``zppy`` has problems in later steps, -we'll have to make changes in the ``zppy`` code and -create a new release candidate; until the new ``zppy`` release candidate is included in the -new unified environment release candidate, -we'll have to just use the ``zppy`` development environment. -This will require running ``pip install .`` in that environment from the top level of -the repo. - -3b. It may be the case that the configuration file had an error and thus neither -``zppy`` nor any package that ``zppy`` runs actually failed. In that case, fix the -configuration file. - -3c. Keep in mind if you run ``zppy -c`` multiple times, it will only rerun tasks that -have failed. This may or may not be what we want. For example, if we change the -environment we're using, we probably want to rerun everything using the new environment. - -4. Create a configuration file ``e3sm_unified_test_simulation.cfg``. - -.. literalinclude:: testing_e3sm_unified.cfg - :language: cfg - :linenos: - -5. We can now test out ``zppy`` by running ``zppy -c e3sm_unified_test_simulation.cfg``. -If that command fails, return to step 3. - -6. If the above command completes successfully, run: - :: - - # cd into output directory - cd /post/scripts - # Check status of files that either failed or are still running. - grep -v "OK" *status - # If there is an error, return to step 3. - # If you want to remove all output run the following two steps: - rm -rf /post - rm -rf /20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis - -7. Output can be viewed at the web link corresponding to ```` in the configuration -file (i.e., ```` defined in "Parameters" above). - -Archive simulation using zstash -=============================== - -1. Create a batch script ``zstash_create.bash`` to run ``zstash``. - :: - - #!/bin/bash - - #SBATCH --job-name=zstash_create - #SBATCH --nodes=1 - #SBATCH --output=/zstash_create.o%j - #SBATCH --exclusive - #SBATCH --time=04:00:00 - - - zstash create --hpss=none --cache= /20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis - -2. Run with ``sbatch zstash_create.bash``. This example takes about 2.5 hours to run. - -Transfer simulation to other machines -===================================== -We can follow an adapted version of the steps at -https://e3sm-project.github.io/zstash/_build/html/master/best_practices.html#transfer-to-nersc-hpss -to transfer the data. - -Login to Globus: https://www.globus.org/ (using NERSC credentials) - -Transfer to Cori ----------------- - -1. On the left sidebar, choose "ENDPOINTS". - -2. Search for "NERSC DTN". Click on Green power button to activate endpoint. - -3. On the left sidebar, choose "ENDPOINTS". - -4. Search for "lcrc#dtn_bebop". Click on Green power button to activate endpoint. -Log in using LCRC credentials. - -5. Paste the path to the ``zstash`` archive (```` for Chrysalis) -in the "Path" box. - -6. Click "Transfer or Sync to..." on the right side. The screen will now be split. - -7. On the left side, choose "Select all" - -8. On the right side, put "NERSC DTN" for collection. - -9. Paste the NERSC path you want the archive copied to -(```` for Cori). This path needs to already exist. - -10. Click "Transfer & Sync Options" in the center. - -11. Choose "sync - only transfer new or changed files" -(choose "modification time is newer" in the dropdown box), -"preserve source file modification times", and "verify file integrity after transfer". - -12. For "Label This Transfer", put something like "zstash archive LCRC to NERSC". - -13. On the left side, click "Start >". This will start the transfer from LCRC to NERSC. - -Transfer to Compy ------------------ - -1. On the left sidebar, choose "ENDPOINTS". - -2. Search for "pic#compy-dtn". Click on Green power button to activate endpoint. -Log in using Compy credentials. - -3. On the left sidebar, choose "File Manager." The screen will now be split. - -4. On the left side, put "lcrc#dtn_bebop" for "Collection". - -5. On the left side, paste the path to the ``zstash`` archive -(```` for Chrysalis) in the "Path" box. - -6. On the left side, choose "Select all". - -7. On the right side, put "pic#compy-dtn" for "Collection". - -8. Paste the Compy path you want the archive copied to -(```` for Compy). -This path needs to already exist. - -9. Click "Transfer & Sync Options" in the center. - -10. Choose "sync - only transfer new or changed files" -(choose "modification time is newer" in the dropdown box), -"preserve source file modification times", and "verify file integrity after transfer". - -11. For "Label This Transfer", put something like "zstash archive LCRC to Compy". - -12. On the left side, click "Start >". This will start the transfer from LCRC to Compy. - -Check transfers were successful -=============================== - -Cori ----- -1. Create a batch script ``zstash_check.bash``: :: - - #!/bin/bash - #SBATCH --job-name=zstash_check - #SBATCH --nodes=1 - #SBATCH --output=/zstash_check.o%j - #SBATCH --exclusive - #SBATCH --time=3:00:00 - #SBATCH -q regular - #SBATCH --constraint=haswell - - - zstash check --hpss=none --cache= - -2. Run ``sbatch zstash_check.bash``. This takes about an hour to run. - -Compy ------ - -1. Create a batch script ``zstash_check.bash``: :: - - #!/bin/bash - #SBATCH --job-name=zstash_check - #SBATCH --nodes=1 - #SBATCH --output=/zstash_check.o%j - #SBATCH --exclusive - #SBATCH --time=3:00:00 - - - zstash check --hpss=none --cache= - -2. Run ``sbatch zstash_check.bash``. This takes over 3 hours to run. - -Extract the data from the archives -================================== - -Use ``zstash extract``. - -Cori ----- - -1. ``mkdir unified_test_extraction``. - -2. Create a batch script ``zstash_extract.bash``: :: - - #!/bin/bash - #SBATCH --job-name=zstash_extract - #SBATCH --nodes=1 - #SBATCH --output=/zstash_extract.o%j - #SBATCH --exclusive - #SBATCH --time=4:00:00 - #SBATCH -q regular - #SBATCH --constraint=haswell - - - cd unified_test_extraction - zstash extract --hpss=none --cache= - -3. Run ``sbatch zstash_extract.bash``. - -Compy ------ - -1. ``mkdir unified_test_extraction``. - -2. Create a batch script ``zstash_extract.bash``: :: - - #!/bin/bash - #SBATCH --job-name=zstash_extract - #SBATCH --nodes=1 - #SBATCH --output=/zstash_extract.o%j - #SBATCH --exclusive - #SBATCH --time=6:00:00 - - - cd unified_test_extraction - zstash extract --hpss=none --cache= - -3. Run ``sbatch zstash_extract.bash``. - - -Testing zppy on other machines -============================== - -``mv e3sm_unified_test_zstash/unified_test_extraction e3sm_unified_test_zppy/20210528.v2rc3e.piControl.ne30pg2_EC30to60E2r2.chrysalis``. - -Go through steps 3-7 of "Testing zppy on a small simulation" but for -Cori (haswell) and Compy. - -Sample configuration files for Compy and Cori ---------------------------------------------- - -``/e3sm_unified_test_simulation.cfg``: -Use the configuration file from "Testing zppy on a small simulation" step 4, updating -the appropriate parameters for Compy or Cori. - -Run unit tests and integration tests -==================================== - -Run the unit tests and integration tests on Chrysalis: - -``cd`` into your clone of the ``zppy`` repo. Then: :: - - git checkout main # if not already on the `main` branch - git fetch upstream - git rebase upstream/main - # Load the appropriate environment - pytest tests/test_*.py # Run unit tests - pytest tests/integration/test_*.py # Run integration tests From 45e75a5641c2b650a4843ad2963035a818fbb96d Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Fri, 5 Jun 2026 16:38:48 -0700 Subject: [PATCH 09/27] Refine test doc references --- docs/source/dev_guide/index.rst | 5 ++--- docs/source/dev_guide/{tests.rst => test.rst} | 0 ...test_expected_results.rst => update_expected_results.rst} | 0 3 files changed, 2 insertions(+), 3 deletions(-) rename docs/source/dev_guide/{tests.rst => test.rst} (100%) rename docs/source/dev_guide/{test_expected_results.rst => update_expected_results.rst} (100%) diff --git a/docs/source/dev_guide/index.rst b/docs/source/dev_guide/index.rst index b33e20cf..ff7ad430 100644 --- a/docs/source/dev_guide/index.rst +++ b/docs/source/dev_guide/index.rst @@ -11,10 +11,9 @@ This guide covers everything needed to develop, test, and release ``zppy``. contributing parameters tasks/index - testing - release_testing + test + update_expected_results releases/index.rst new_task new_diags_set - tutorial_testing_e3sm_unified archive/index diff --git a/docs/source/dev_guide/tests.rst b/docs/source/dev_guide/test.rst similarity index 100% rename from docs/source/dev_guide/tests.rst rename to docs/source/dev_guide/test.rst diff --git a/docs/source/dev_guide/test_expected_results.rst b/docs/source/dev_guide/update_expected_results.rst similarity index 100% rename from docs/source/dev_guide/test_expected_results.rst rename to docs/source/dev_guide/update_expected_results.rst From 7872f74f5030e0d633b76bebd3a0af9459fed89a Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Fri, 5 Jun 2026 16:41:26 -0700 Subject: [PATCH 10/27] Update archives --- .../archive/deprecated_parameters.rst | 17 ++++++++++ docs/source/dev_guide/archive/index.rst | 32 +------------------ 2 files changed, 18 insertions(+), 31 deletions(-) create mode 100644 docs/source/dev_guide/archive/deprecated_parameters.rst diff --git a/docs/source/dev_guide/archive/deprecated_parameters.rst b/docs/source/dev_guide/archive/deprecated_parameters.rst new file mode 100644 index 00000000..8173db31 --- /dev/null +++ b/docs/source/dev_guide/archive/deprecated_parameters.rst @@ -0,0 +1,17 @@ +********************* +Deprecated parameters +********************* + +The following ``zppy`` parameters no longer perform any function: + +.. code-block:: text + + e3sm_to_cmip_environment_commands + ts_fmt + scratch + atmosphere_only + plot_names + ncclimo_cmd (still accepted but ignored) + nrows (in global_time_series; specifying has no effect) + ncols (in global_time_series; specifying has no effect) + vars_exclude (deprecated; use vars = "" and filter in post-processing) \ No newline at end of file diff --git a/docs/source/dev_guide/archive/index.rst b/docs/source/dev_guide/archive/index.rst index 342eb28d..5ffde3a5 100644 --- a/docs/source/dev_guide/archive/index.rst +++ b/docs/source/dev_guide/archive/index.rst @@ -11,35 +11,5 @@ historical reference. .. toctree:: :maxdepth: 1 + deprecated_parameters initial_docs - -Old testing tutorial (E3SM Unified 1.5.0) -========================================== - -The :doc:`tutorial_testing_e3sm_unified` page contains a tutorial written -for testing against E3SM Unified v1.5.0. For current testing directions, -see :ref:`release-testing`. - -Old documentation setup (initial Sphinx configuration) -======================================================= - -The section "Initial setup (obsolete/for reference only)" in -:doc:`contributing` describes the original one-time Sphinx configuration -steps. Those steps do not need to be repeated. - -Deprecated parameters -===================== - -The following ``zppy`` parameters no longer perform any function: - -.. code-block:: text - - e3sm_to_cmip_environment_commands - ts_fmt - scratch - atmosphere_only - plot_names - ncclimo_cmd (still accepted but ignored) - nrows (in global_time_series; specifying has no effect) - ncols (in global_time_series; specifying has no effect) - vars_exclude (deprecated; use vars = "" and filter in post-processing) From ad819184dd10045642f869f40bbdbcdc5ba5c5b9 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Mon, 8 Jun 2026 11:46:37 -0700 Subject: [PATCH 11/27] Address comments --- docs/source/dev_guide/archive/initial_docs.rst | 6 +++--- docs/source/dev_guide/new_diags_set.rst | 4 ++-- docs/source/dev_guide/new_task.rst | 6 +++--- docs/source/user_guide/tasks/climo.rst | 2 +- docs/source/user_guide/tasks/e3sm_diags.rst | 2 +- docs/source/user_guide/tasks/global_time_series.rst | 6 +++--- docs/source/user_guide/tasks/mpas_analysis.rst | 2 +- docs/source/user_guide/tutorial.rst | 10 +++++----- 8 files changed, 19 insertions(+), 19 deletions(-) diff --git a/docs/source/dev_guide/archive/initial_docs.rst b/docs/source/dev_guide/archive/initial_docs.rst index 61b0dece..5758883f 100644 --- a/docs/source/dev_guide/archive/initial_docs.rst +++ b/docs/source/dev_guide/archive/initial_docs.rst @@ -30,7 +30,7 @@ accept suggested default options, except :: Separate source and build directories (y/N) [n]: y -Edit Makefile and change ``BUILDIR`` :: +Edit Makefile and change ``BUILDDIR`` :: BUILDDIR = docs @@ -52,7 +52,7 @@ Try building documentation :: $ make html -Create an empty .nojekyll file to indicate to Github.com that this +Create an empty .nojekyll file to indicate to GitHub.com that this is not a Jekyll static website: :: $ touch .nojekyll @@ -65,7 +65,7 @@ with the following: :: -Commit and push back to Github: :: +Commit and push back to GitHub: :: $ git add . $ git commit diff --git a/docs/source/dev_guide/new_diags_set.rst b/docs/source/dev_guide/new_diags_set.rst index 67f157c9..64a72a09 100644 --- a/docs/source/dev_guide/new_diags_set.rst +++ b/docs/source/dev_guide/new_diags_set.rst @@ -1,6 +1,6 @@ -********************************************* +******************************************************* Examples: Adding a new set to the E3SM Diagnostics task -********************************************* +******************************************************* This guide gives a general list of things to consider when adding a new E3SM Diagnostics set. The exact code changes required will differ amongst sets. diff --git a/docs/source/dev_guide/new_task.rst b/docs/source/dev_guide/new_task.rst index 0d43a5a1..aa2b684c 100644 --- a/docs/source/dev_guide/new_task.rst +++ b/docs/source/dev_guide/new_task.rst @@ -146,12 +146,12 @@ Add the task to ``zppy/__main__.py``: .. code:: - from import + from task_name import task_name .. code:: - # task-name tasks - existing_bundles = task-name(config, script_dir, existing_bundles, job_ids_file) + # task_name tasks + existing_bundles = task_name(config, script_dir, existing_bundles, job_ids_file) Update defaults =============== diff --git a/docs/source/user_guide/tasks/climo.rst b/docs/source/user_guide/tasks/climo.rst index 75287bb0..527b7047 100644 --- a/docs/source/user_guide/tasks/climo.rst +++ b/docs/source/user_guide/tasks/climo.rst @@ -150,5 +150,5 @@ Dependencies **Downstream (what depends on climo):** -- :doc:`e3sm_diags` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly dirunal-atm climo: diurnal_cycle +- :doc:`e3sm_diags` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly diurnal-atm climo: diurnal_cycle - :doc:`livvkit` — Monthly-lnd climo diff --git a/docs/source/user_guide/tasks/e3sm_diags.rst b/docs/source/user_guide/tasks/e3sm_diags.rst index c8c034db..337ff74d 100644 --- a/docs/source/user_guide/tasks/e3sm_diags.rst +++ b/docs/source/user_guide/tasks/e3sm_diags.rst @@ -195,7 +195,7 @@ Dependencies **Upstream (what e3sm_diags depends on):** -- :doc:`climo` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly dirunal-atm climo: diurnal_cycle +- :doc:`climo` — Monthly-atm climo: lat_lon, zonal_mean_xy, zonal_mean_2d, polar, cosp_histogram, meridional_mean_2d, annual_cycle_zonal_mean, zonal_mean_2d_stratosphere, aerosol_aeronet, aerosol_budget. Monthly-lnd climo: lat_lon_land. Monthly diurnal-atm climo: diurnal_cycle - :doc:`ts` — Monthly-atm ts: enso_diags, qbo, area_mean_time_series, mp_partition. Monthly-rof ts: streamflow. Daily-atm ts: tropical_subseasonal, precip_pdf. - :doc:`tc_analysis` — Required for tc_analysis set. diff --git a/docs/source/user_guide/tasks/global_time_series.rst b/docs/source/user_guide/tasks/global_time_series.rst index efd8f64c..cc10995e 100644 --- a/docs/source/user_guide/tasks/global_time_series.rst +++ b/docs/source/user_guide/tasks/global_time_series.rst @@ -5,8 +5,8 @@ global_time_series — Global Time Series Plots The ``global_time_series`` task generates global time series plots of key climate metrics. It can optionally produce a Viewer page. It depends on -the :doc:`ts` task (for atmospheric, land, and ocean global-mean time series) -and optionally on :doc:`mpas_analysis` (for ocean and sea-ice plots). + the :doc:`ts` task (for atmospheric and land global-mean time series) + and on :doc:`mpas_analysis` when ocean plots are requested. Parameters ---------- @@ -107,7 +107,7 @@ Dependencies **Upstream (what global_time_series depends on):** - :doc:`ts` — Monthly-atm-glb ts: the 5 classic atm plots, plots for specific atm variables. Monthly-lnd-glb ts: plots for specific lnd variables. -- :doc:`mpas_analysis` — Monthly-atm-glb ts: the 3 classic ocn plots, plots for specific ocn variables. +- :doc:`mpas_analysis` — the 3 classic ocn plots, plots for specific ocn variables. **Downstream (what depends on global_time_series):** diff --git a/docs/source/user_guide/tasks/mpas_analysis.rst b/docs/source/user_guide/tasks/mpas_analysis.rst index 6ba5db7c..b0443687 100644 --- a/docs/source/user_guide/tasks/mpas_analysis.rst +++ b/docs/source/user_guide/tasks/mpas_analysis.rst @@ -166,4 +166,4 @@ Dependencies **Downstream (what depends on mpas_analysis):** -- :doc:`global_time_series` — Monthly-atm-glb ts: the 3 classic ocn plots, plots for specific ocn variables. +- :doc:`global_time_series` — the 3 classic ocn plots, plots for specific ocn variables. diff --git a/docs/source/user_guide/tutorial.rst b/docs/source/user_guide/tutorial.rst index 17fc44c4..6ae2e2e9 100644 --- a/docs/source/user_guide/tutorial.rst +++ b/docs/source/user_guide/tutorial.rst @@ -38,10 +38,10 @@ Once the ``[ts]`` subsection ``[[ atm_monthly_glb ]]`` and the ``[mpas_analysis] section finish, ``[global_time_series]`` will run. Post-processing results will be located in ``output`` and ``www``. Some machines have -a web server. ``www`` should be pointed to that so that the plotting tasks (here: ``e3sm_diags`` ``mpas_analysis``, ``global_time_series``, ``livvkit``) will be visible online. +a web server. ``www`` should be pointed to that so that the plotting tasks (here: ``e3sm_diags``, ``mpas_analysis``, ``global_time_series``, ``livvkit``) will be visible online. Because we have specified ``campaign = "water_cycle"``, some parameters will -be automatically set. ``zppy/templates/water_cycle.cfg`` specifies what +be automatically set. ``zppy/defaults/water_cycle.cfg`` specifies what ``[e3sm_diags] > sets``, and ``[mpas_analysis] > generate`` should be for the water cycle campaign. Users may specify their own values for any of these parameters, @@ -85,9 +85,9 @@ or ``test_comparison_type`` to disambiguate. Debugging failures ================== - :: +.. code-block:: bash - grep "output=" your_zppy_config.cfg # Easy way to remember your output directory + grep "output =" your_zppy_config.cfg # Easy way to remember your output directory cd your_output_dir/post/scripts grep -v "OK" *status # See what failed @@ -105,4 +105,4 @@ Debugging failures emacs failing_task.bash # In this file, set `debug = True`. This will provide more information. # Note: another option is to set `debug = True` in your `cfg` and rerun `zppy -c`. - sbatch .bash + sbatch failing_task.bash From e94f5ac3399e8be7723da427acadd775acfffebd Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Mon, 8 Jun 2026 11:57:53 -0700 Subject: [PATCH 12/27] Fix web server paths --- docs/source/dev_guide/contributing.rst | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/source/dev_guide/contributing.rst b/docs/source/dev_guide/contributing.rst index eae6785e..15005b1b 100644 --- a/docs/source/dev_guide/contributing.rst +++ b/docs/source/dev_guide/contributing.rst @@ -39,9 +39,21 @@ using Sphinx, you can refer to cd docs make html - # And transfer it to a web server path so we can see it online: - cp -r _build/ web_server_path/zppy_docs - chmod -R 644 web_serber_path/zppy_docs + # And transfer it to a web server path so we can see it online. + + # Example web server paths: + # Chrysalis: /lcrc/group/e3sm/public_html/diagnostic_output -> https://web.lcrc.anl.gov/public/e3sm/diagnostic_output + # Compy: /compyfs/www -> https://compy-dtn.pnl.gov + # Perlmutter: /global/cfs/cdirs/e3sm/www -> https://portal.nersc.gov/cfs/e3sm + + # For more machines, go to: + # https://github.com/E3SM-Project/mache/tree/main/mache/machines + # Select the cfg for the machine, and find the `[web_portal]`` section. + # That will show the `base_path` and `base_url`. + + cp -r _build/ base_url/$USER/zppy_docs + # Needs to be 755 to show up on https://portal.nersc.gov/cfs/e3sm + chmod -R 755 base_url/$USER/zppy_docs # Optional -- if you want to generate and view versioned docs: sphinx-multiversion source _build/html From 21c0fe751ee3a936569b62c860072ff00fa2f4d0 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Mon, 8 Jun 2026 12:33:57 -0700 Subject: [PATCH 13/27] Formatting fixes and other small changes --- docs/source/dev_guide/contributing.rst | 49 ++++++++++--------- docs/source/dev_guide/new_diags_set.rst | 4 +- docs/source/dev_guide/new_task.rst | 4 +- docs/source/dev_guide/project-standards.rst | 10 ++-- .../releases/production_releases.rst | 2 +- .../dev_guide/releases/release_candidates.rst | 6 +-- docs/source/dev_guide/tasks/index.rst | 20 -------- docs/source/index.rst | 2 +- docs/source/user_guide/dependencies.rst | 2 + docs/source/user_guide/getting_started.rst | 3 +- docs/source/user_guide/parameters.rst | 29 +++++------ docs/source/user_guide/tutorial.rst | 38 +++++++------- 12 files changed, 77 insertions(+), 92 deletions(-) diff --git a/docs/source/dev_guide/contributing.rst b/docs/source/dev_guide/contributing.rst index 15005b1b..4767831a 100644 --- a/docs/source/dev_guide/contributing.rst +++ b/docs/source/dev_guide/contributing.rst @@ -26,37 +26,38 @@ using Sphinx, you can refer to * `First Steps with Sphinx `_ * `reStructuredText Primer `_ - .. code:: - # Sphinx is included in the dev environment, - # so we just need to set one up: - cd zppy - conda clean --all --y - conda env create -f conda/dev.yml -n env-name - python -m pip install . +.. code-block:: bash - # Now, we can produce the documentation locally: - cd docs - make html + # Sphinx is included in the dev environment, + # so we just need to set one up: + cd zppy + conda clean --all --y + conda env create -f conda/dev.yml -n env-name + python -m pip install . - # And transfer it to a web server path so we can see it online. + # Now, we can produce the documentation locally: + cd docs + make html - # Example web server paths: - # Chrysalis: /lcrc/group/e3sm/public_html/diagnostic_output -> https://web.lcrc.anl.gov/public/e3sm/diagnostic_output - # Compy: /compyfs/www -> https://compy-dtn.pnl.gov - # Perlmutter: /global/cfs/cdirs/e3sm/www -> https://portal.nersc.gov/cfs/e3sm + # And transfer it to a web server path so we can see it online. - # For more machines, go to: - # https://github.com/E3SM-Project/mache/tree/main/mache/machines - # Select the cfg for the machine, and find the `[web_portal]`` section. - # That will show the `base_path` and `base_url`. + # Example web server paths: + # Chrysalis: /lcrc/group/e3sm/public_html/diagnostic_output -> https://web.lcrc.anl.gov/public/e3sm/diagnostic_output + # Compy: /compyfs/www -> https://compy-dtn.pnl.gov + # Perlmutter: /global/cfs/cdirs/e3sm/www -> https://portal.nersc.gov/cfs/e3sm - cp -r _build/ base_url/$USER/zppy_docs - # Needs to be 755 to show up on https://portal.nersc.gov/cfs/e3sm - chmod -R 755 base_url/$USER/zppy_docs + # For more machines, go to: + # https://github.com/E3SM-Project/mache/tree/main/mache/machines + # Select the cfg for the machine, and find the `[web_portal]`` section. + # That will show the `base_path` and `base_url`. - # Optional -- if you want to generate and view versioned docs: - sphinx-multiversion source _build/html + cp -r _build/ base_url/$USER/zppy_docs + # Needs to be 755 to show up on https://portal.nersc.gov/cfs/e3sm + chmod -R 755 base_url/$USER/zppy_docs + + # Optional -- if you want to generate and view versioned docs: + sphinx-multiversion source _build/html Pull requests should bundle together associated code changes, test changes, and documentation changes. Once you've merged your pull request and GitHub Actions finishes building the docs, changes will be visible on the `zppy documentation page `_. diff --git a/docs/source/dev_guide/new_diags_set.rst b/docs/source/dev_guide/new_diags_set.rst index 64a72a09..99a623a9 100644 --- a/docs/source/dev_guide/new_diags_set.rst +++ b/docs/source/dev_guide/new_diags_set.rst @@ -32,7 +32,7 @@ In ``zppy/e3sm_diags.py`` Diurnal Cycle: ``e3sm_diags`` had to be set up to depend on a new climatology subtask: - .. code:: + .. code-block:: python if "diurnal_cycle" in c['sets']: dependencies.append(os.path.join(scriptDir, 'climo_%s_%04d-%04d.status' % ( @@ -40,7 +40,7 @@ Diurnal Cycle: ``e3sm_diags`` had to be set up to depend on a new climatology su Streamflow: ``e3sm_diags`` had to be set up to depend on a new time series subtask: - .. code:: + .. code-block:: python if "streamflow" in c["sets"]: dependencies.append( diff --git a/docs/source/dev_guide/new_task.rst b/docs/source/dev_guide/new_task.rst index aa2b684c..5658f5de 100644 --- a/docs/source/dev_guide/new_task.rst +++ b/docs/source/dev_guide/new_task.rst @@ -144,11 +144,11 @@ The main file Add the task to ``zppy/__main__.py``: - .. code:: + .. code-block:: python from task_name import task_name - .. code:: + .. code-block:: python # task_name tasks existing_bundles = task_name(config, script_dir, existing_bundles, job_ids_file) diff --git a/docs/source/dev_guide/project-standards.rst b/docs/source/dev_guide/project-standards.rst index 3738a403..3132648a 100644 --- a/docs/source/dev_guide/project-standards.rst +++ b/docs/source/dev_guide/project-standards.rst @@ -52,10 +52,10 @@ Helpful ``pre-commit`` Commands * **trailing-whitespace** - Trims trailing whitespace from the end of each line. * **end-of-file-fixer** - Ensures files end with a single newline character. * **check-yaml** - Validates YAML files for correct syntax. -* **black** - Auto-formats Python code to a consistent style using the `black `__ formatter. -* **isort** - Sorts and organizes Python import statements alphabetically and by type using `isort `__. -* **flake8** - Lints Python code for style violations and common errors (PEP 8) using `flake8 `__. -* **mypy** - Runs static type checking on Python code using type annotations using `mypy `__. +* **black** - Auto-formats Python code to a consistent style using the `black `_ formatter. +* **isort** - Sorts and organizes Python import statements alphabetically and by type using `isort `_. +* **flake8** - Lints Python code for style violations and common errors (PEP 8) using `flake8 `_. +* **mypy** - Runs static type checking on Python code using type annotations using `mypy `_. Install into your cloned repo :: @@ -87,7 +87,7 @@ Run individual hook :: Continuous Integration / Continuous Delivery (CI/CD) ---------------------------------------------------- -This project uses `GitHub Actions `_ to run two CI/CD workflows. The workflows are defined `here `__. +This project uses `GitHub Actions `_ to run two CI/CD workflows. The workflows are defined `here `_. 1. CI/CD Build Workflow diff --git a/docs/source/dev_guide/releases/production_releases.rst b/docs/source/dev_guide/releases/production_releases.rst index eb1d30a4..e2266865 100644 --- a/docs/source/dev_guide/releases/production_releases.rst +++ b/docs/source/dev_guide/releases/production_releases.rst @@ -11,7 +11,7 @@ Be sure to run the entire integration test suite before making a production rele Step 2: Confluence ------------------ -This step should already have been completed during the release-candidate phase, however it is good practice to double check that the `E3SM Unified version tracking page `__ has had the next E3SM Unified version updated with the new ``zppy`` version number. +This step should already have been completed during the release-candidate phase, however it is good practice to double check that the `E3SM Unified version tracking page `_ has had the next E3SM Unified version updated with the new ``zppy`` version number. Step 3: Prepare documentation ------------------------------------------ diff --git a/docs/source/dev_guide/releases/release_candidates.rst b/docs/source/dev_guide/releases/release_candidates.rst index 7be09faa..0f793bd4 100644 --- a/docs/source/dev_guide/releases/release_candidates.rst +++ b/docs/source/dev_guide/releases/release_candidates.rst @@ -13,7 +13,7 @@ Step 2: Determine what the new version should be This step only needs to be done for rc1. -Review `zppy commits `__. Identify what changes since the last production release would require a patch, minor, or major version update (as in ``vMAJOR.MINOR.PATCH``). +Review `zppy commits `_. Identify what changes since the last production release would require a patch, minor, or major version update (as in ``vMAJOR.MINOR.PATCH``). The highest-level version update will be used. For example, if you found 1 commit requiring a patch update, 1 commit requiring a minor update, and 0 commits requiring a major update, then you'd want to increment the minor version. @@ -22,7 +22,7 @@ Step 3: Confluence This step only needs to be done for rc1. -Update the `E3SM Unified version tracking page `__ with the new ``zppy`` version number. Be sure to update the section for the next E3SM Unified environment, not the current one. +Update the `E3SM Unified version tracking page `_ with the new ``zppy`` version number. Be sure to update the section for the next E3SM Unified environment, not the current one. Step 4: tbump ------------- @@ -56,7 +56,7 @@ Step 5: Tag the RC on the zppy repo git branch -D v1.2.3rc4 git push upstream v1.2.3rc4 # Push the new RC tag to GitHub -``v1.2.3rc4`` should now appear on `Tags `__, but _not_ on `Releases `__. +``v1.2.3rc4`` should now appear on `Tags `__, but _not_ on `Releases `_. Step 6: zppy-feedstock repo --------------------------- diff --git a/docs/source/dev_guide/tasks/index.rst b/docs/source/dev_guide/tasks/index.rst index 672ad51e..a3e3d7b0 100644 --- a/docs/source/dev_guide/tasks/index.rst +++ b/docs/source/dev_guide/tasks/index.rst @@ -21,23 +21,3 @@ other tasks. livvkit pcmdi_diags bundle - -Dependency overview -------------------- - -The diagram below summarizes the typical dependency relationships between -tasks. Arrows indicate "depends on": - -.. code-block:: text - - climo ──────────────────────────────────────────► e3sm_diags - ts ─────────────────────────────────────────────► e3sm_diags - ts ─────────────────────────────────────────────► e3sm_to_cmip - ts ─────────────────────────────────────────────► global_time_series - ts ─────────────────────────────────────────────► pcmdi_diags (via e3sm_to_cmip) - mpas_analysis ──────────────────────────────────► global_time_series - e3sm_to_cmip ───────────────────────────────────► ilamb - e3sm_to_cmip ───────────────────────────────────► pcmdi_diags - climo ──────────────────────────────────────────► livvkit - -All tasks are independent unless they share the above relationships. diff --git a/docs/source/index.rst b/docs/source/index.rst index 7f9a8e06..252c453e 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -9,7 +9,7 @@ Zppy documentation What is zppy? ============= -``zppy`` (pronounced "/ˈzɪpi/") is a **workflow manager** for post-processing +``zppy`` (pronounced as "zippy", or in IPA: /ˈzɪpi/) is a **workflow manager** for post-processing E3SM simulations, written in Python. The goal of ``zppy`` is to speed up post-processing by automating commonly performed tasks. diff --git a/docs/source/user_guide/dependencies.rst b/docs/source/user_guide/dependencies.rst index 0fda4c25..2fe06684 100644 --- a/docs/source/user_guide/dependencies.rst +++ b/docs/source/user_guide/dependencies.rst @@ -2,6 +2,8 @@ Dependencies ************ +Below defines a dependency graph for ``zppy`` tasks. It can be visualized with graph-drawing tools like GraphViz or ``pydot``. + .. literalinclude:: ../dependencies.dot :language: dot :linenos: diff --git a/docs/source/user_guide/getting_started.rst b/docs/source/user_guide/getting_started.rst index 2ad80eb2..d3ce9086 100644 --- a/docs/source/user_guide/getting_started.rst +++ b/docs/source/user_guide/getting_started.rst @@ -76,7 +76,8 @@ To install the latest stable release, refer to the following: Setting up a development environment ==================================== - :: +.. code-block:: bash + # Get the code ######################################################################## # Set up your fork: diff --git a/docs/source/user_guide/parameters.rst b/docs/source/user_guide/parameters.rst index a999c9ef..bb72de38 100644 --- a/docs/source/user_guide/parameters.rst +++ b/docs/source/user_guide/parameters.rst @@ -84,25 +84,26 @@ For the ``mpas_analysis`` task: not need to set a comparison type for the current run. .. note:: - These parameter names are intentionally consistent with the terminology used by ``e3sm_diags`` for model-vs-model runs: in both cases, ``reference_data_path`` identifies the *reference simulation's zppy-generated outputs*. - The practical difference is what each downstream tool consumes: - ``e3sm_diags`` needs ``reference_data_path`` to be the specific directory containing the reference climatology files (typically under the reference run's ``post/.../clim`` tree), whereas ``mpas_analysis`` needs to find the reference MPAS-Analysis config file. - For MPAS-Analysis, ``zppy`` resolves the config file when ``reference_data_path`` points to the prior run's zppy output directory (the one containing ``post/``). + These parameter names are intentionally consistent with the terminology used by ``e3sm_diags`` for model-vs-model runs: in both cases, ``reference_data_path`` identifies the *reference simulation's zppy-generated outputs*. - ``reference_data_path`` is intended to point to the prior run's zppy output directory (the one containing ``post/``). ``zppy`` will then use: - ``/post/analysis/mpas_analysis//cfg/mpas_analysis_.cfg`` - where ```` is ``mvo`` or ``mvm``. + The practical difference is what each downstream tool consumes: + ``e3sm_diags`` needs ``reference_data_path`` to be the specific directory containing the reference climatology files (typically under the reference run's ``post/.../clim`` tree), whereas ``mpas_analysis`` needs to find the reference MPAS-Analysis config file. + For MPAS-Analysis, ``zppy`` resolves the config file when ``reference_data_path`` points to the prior run's zppy output directory (the one containing ``post/``). - For referenced prior runs, ``reference_comparison_type`` and - ``test_comparison_type`` can be set to ``"auto"``, ``"mvo"``, or ``"mvm"``. - The default is ``"auto"``. In auto mode: + ``reference_data_path`` is intended to point to the prior run's zppy output directory (the one containing ``post/``). ``zppy`` will then use: + ``/post/analysis/mpas_analysis//cfg/mpas_analysis_.cfg`` + where ```` is ``mvo`` or ``mvm``. - * if the path points to ``[[subsection]]``, zppy uses the referenced subsection's actual comparison type - * if the path points to an external zppy output directory, zppy looks for the matching cfg under ``mvo`` and ``mvm`` - * if both exist for the same identifier, zppy raises an error and the user should set ``reference_comparison_type`` or ``test_comparison_type`` explicitly + For referenced prior runs, ``reference_comparison_type`` and + ``test_comparison_type`` can be set to ``"auto"``, ``"mvo"``, or ``"mvm"``. + The default is ``"auto"``. In auto mode: - When ``reference_data_path`` is set to a non-subsection path, ``reference_case`` is required so the MVM output directory can include the reference case name. If ``reference_data_path`` is set to ``[[subsection]]``, ``reference_case`` is inferred to be the same as the current ``case``. + * if the path points to ``[[subsection]]``, zppy uses the referenced subsection's actual comparison type + * if the path points to an external zppy output directory, zppy looks for the matching cfg under ``mvo`` and ``mvm`` + * if both exist for the same identifier, zppy raises an error and the user should set ``reference_comparison_type`` or ``test_comparison_type`` explicitly + + When ``reference_data_path`` is set to a non-subsection path, ``reference_case`` is required so the MVM output directory can include the reference case name. If ``reference_data_path`` is set to ``[[subsection]]``, ``reference_case`` is inferred to be the same as the current ``case``. **MPAS-Analysis model-vs-model year ranges** diff --git a/docs/source/user_guide/tutorial.rst b/docs/source/user_guide/tutorial.rst index 6ae2e2e9..22fd7036 100644 --- a/docs/source/user_guide/tutorial.rst +++ b/docs/source/user_guide/tutorial.rst @@ -87,22 +87,22 @@ Debugging failures .. code-block:: bash - grep "output =" your_zppy_config.cfg # Easy way to remember your output directory - cd your_output_dir/post/scripts - grep -v "OK" *status # See what failed - - # Say failing_task.status is showing a non-OK status, then: - ls failing_task* # See everything associated with it - - # Say its job ID was 123456, then: - emacs failing_task.o123456 # Review the output - emacs failing_task.settings # Review how each of its parameters was set/configured. - - # If an error is obvious, make a fix in the bash file and rerun: - sbatch failing_task.bash - - # If the error is not obvious, do the following: - emacs failing_task.bash - # In this file, set `debug = True`. This will provide more information. - # Note: another option is to set `debug = True` in your `cfg` and rerun `zppy -c`. - sbatch failing_task.bash + grep "output =" your_zppy_config.cfg # Easy way to remember your output directory + cd your_output_dir/post/scripts + grep -v "OK" *status # See what failed + + # Say failing_task.status is showing a non-OK status, then: + ls failing_task* # See everything associated with it + + # Say its job ID was 123456, then: + emacs failing_task.o123456 # Review the output + emacs failing_task.settings # Review how each of its parameters was set/configured. + + # If an error is obvious, make a fix in the bash file and rerun: + sbatch failing_task.bash + + # If the error is not obvious, do the following: + emacs failing_task.bash + # In this file, set `debug = True`. This will provide more information. + # Note: another option is to set `debug = True` in your `cfg` and rerun `zppy -c`. + sbatch failing_task.bash From 14770bc2bd0b2f2ae9f2ee8f6f41bfc416e60802 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Mon, 8 Jun 2026 15:07:06 -0700 Subject: [PATCH 14/27] Update parameter docs for non-plot tasks --- .../archive/deprecated_parameters.rst | 17 - docs/source/dev_guide/archive/index.rst | 1 - docs/source/dev_guide/tasks/index.rst | 4 +- docs/source/user_guide/dependencies.rst | 2 + docs/source/user_guide/parameters.rst | 303 +++++++++++++++++- docs/source/user_guide/tasks/climo.rst | 90 ++---- docs/source/user_guide/tasks/e3sm_to_cmip.rst | 69 +--- docs/source/user_guide/tasks/index.rst | 4 +- docs/source/user_guide/tasks/tc_analysis.rst | 50 +-- docs/source/user_guide/tasks/ts.rst | 67 +--- 10 files changed, 338 insertions(+), 269 deletions(-) delete mode 100644 docs/source/dev_guide/archive/deprecated_parameters.rst diff --git a/docs/source/dev_guide/archive/deprecated_parameters.rst b/docs/source/dev_guide/archive/deprecated_parameters.rst deleted file mode 100644 index 8173db31..00000000 --- a/docs/source/dev_guide/archive/deprecated_parameters.rst +++ /dev/null @@ -1,17 +0,0 @@ -********************* -Deprecated parameters -********************* - -The following ``zppy`` parameters no longer perform any function: - -.. code-block:: text - - e3sm_to_cmip_environment_commands - ts_fmt - scratch - atmosphere_only - plot_names - ncclimo_cmd (still accepted but ignored) - nrows (in global_time_series; specifying has no effect) - ncols (in global_time_series; specifying has no effect) - vars_exclude (deprecated; use vars = "" and filter in post-processing) \ No newline at end of file diff --git a/docs/source/dev_guide/archive/index.rst b/docs/source/dev_guide/archive/index.rst index 5ffde3a5..1101f6db 100644 --- a/docs/source/dev_guide/archive/index.rst +++ b/docs/source/dev_guide/archive/index.rst @@ -11,5 +11,4 @@ historical reference. .. toctree:: :maxdepth: 1 - deprecated_parameters initial_docs diff --git a/docs/source/dev_guide/tasks/index.rst b/docs/source/dev_guide/tasks/index.rst index a3e3d7b0..285ed532 100644 --- a/docs/source/dev_guide/tasks/index.rst +++ b/docs/source/dev_guide/tasks/index.rst @@ -10,14 +10,14 @@ other tasks. .. toctree:: :maxdepth: 1 + bundle climo ts e3sm_to_cmip + tc_analysis e3sm_diags mpas_analysis global_time_series - tc_analysis ilamb livvkit pcmdi_diags - bundle diff --git a/docs/source/user_guide/dependencies.rst b/docs/source/user_guide/dependencies.rst index 2fe06684..4b665adc 100644 --- a/docs/source/user_guide/dependencies.rst +++ b/docs/source/user_guide/dependencies.rst @@ -1,3 +1,5 @@ +.. _dependency_graph: + ************ Dependencies ************ diff --git a/docs/source/user_guide/parameters.rst b/docs/source/user_guide/parameters.rst index bb72de38..189138a7 100644 --- a/docs/source/user_guide/parameters.rst +++ b/docs/source/user_guide/parameters.rst @@ -16,19 +16,277 @@ on GitHub for a complete list of parameters and their default values. You can also view the most up-to-date, `unreleased parameter defaults `_. -For per-task parameter tables, see the :ref:`Tasks ` section. - -Deprecated parameters -===================== +Parameters for individual tasks +=============================== -The following parameters no longer perform any function: - :: +For per-task parameter tables, see the :ref:`Tasks ` section. - "e3sm_to_cmip_environment_commands" - "ts_fmt" - "scratch" - "atmosphere_only" - "plot_names" +.. _parameters-top-level: + +Parameters at the top-level +=========================== + +These parameters appear in the top-level ``[default]`` section of ``zppy/defaults/default.ini``. They can be overridden in individual tasks and subtasks. + +There are 38 (non-deprecated) parameters that can be defined at the top level. + +**Input specifics** + +There are 2 input-specific parameters: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``case`` + - **Yes** + - *(none)* + - The case name of the simulation. + * - ``input`` + - **Yes** + - *(none)* + - The top-level directory of the simulation output to post-process. + +**Output specifics** + +There are 6 output-specific parameters: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``output`` + - **Yes** + - *(none)* + - Where the post-processing results (``post/`` directory) should go. + * - ``www`` + - **Yes** + - *(none)* + - Where the post-processing visuals should go (to be viewed online). + * - ``campaign`` + - No + - ``"none"`` + - Specify which campaign you are running. Campaigns can be found in ``zppy/defaults``. Possible campaigns include ``cryosphere`` and ``water_cycle``. + * - ``debug`` + - No + - ``False`` + - Set to True to have ``zppy`` produce more verbose output and retain temporary workdirs. This is helpful for debugging. + * - ``dry_run`` + - No + - ``False`` + - This should be set to True if you don't want the batch jobs to be submitted. I.e., you only want to see what *would* be submitted. + * - ``fail_on_dependency_skip`` + - No + - ``False`` + - If set to False, zppy will launch other jobs, if possible. + +**Machine specifics** + +There are 8 machine-specific parameters: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``account`` + - No + - ``""`` + - SLURM account to charge. + * - ``constraint`` + - No + - ``""`` + - The constraint of the machine to run on. + * - ``nodes`` + - No + - ``1`` + - Number of compute nodes. + * - ``parallel`` + - No + - ``"mpi"`` + - The ``parallel`` option passed to ``ncclimo``. + * - ``partition`` + - No + - ``""`` + - SLURM partition to submit the job to. + * - ``qos`` + - No + - ``"regular"`` + - Quality of service + * - ``reservation`` + - No + - ``""`` + - If you have access to a node reservation, specify it here. + * - ``walltime`` + - No + - ``"02:00:00"`` + - Maximum wall time for the SLURM job. + +**Environment specifics** + +There are 3 environment-specific parameters: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``environment_commands`` + - No + - ``""`` + - Shell commands to set up the software environment before running the + task (e.g., ``source /path/to/e3sm_unified.sh``). + * - ``environment_commands_secondary`` + - No + - ``""`` + - Shell commands to set up the *secondary* software environment before running the task (e.g., ``source /path/to/e3sm_unified.sh``). This is currently only used for the ``pcmdi_diags`` task. + * - ``nco_path`` + - No + - ``""`` + - Keep as the empty string to use production-version NCO commands. Set to the development-version path to use that instead. + +**Inference specifics** + +There are 2 inference-specific parameters. See "Parameter checking & inferring" below for more info. + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``infer_path_parameters`` + - No + - ``True`` + - If true, infer file paths not given. + * - ``infer_section_parameters`` + - No + - ``True`` + - If true, infer dependencies not given. + +**Dependency specifics** + +There are 7 dependency-specific parameters. See :ref:`Dependencies ` for more info on which tasks depend on which. + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``ts_atm_grid`` + - No + - ``"180x360_aave"`` + - Grid name of the atmosphere ``[ts]`` subtask. + * - ``ts_atm_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` atmosphere subtask (for multi-component runs). + * - ``ts_grid`` + - No + - ``"180x360_aave"`` + - Grid name of the ``[ts]`` subtask to depend on. + * - ``ts_land_grid`` + - No + - ``"180x360_aave"`` + - Grid name of the land ``[ts]`` subtask. + * - ``ts_land_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` land subtask (for multi-component runs). + * - ``ts_num_years`` + - No + - ``5`` + - Years increment from the dependent ``[ts]`` task. + * - ``ts_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` subtask to depend on. If empty and + ``infer_section_parameters`` is ``True``, inferred from the + subsection name. + +**Task specifics** + +There are 10 task-specific parameters: + +.. list-table:: + :header-rows: 1 + :widths: 22 10 18 50 + + * - Parameter + - Required + - Default + - Description + * - ``active`` + - No + - ``False`` + - Set to ``True`` to enable this task. + * - ``bundle`` + - No + - ``""`` + - (Set on individual tasks) Name of the bundle to include this task + in. + * - ``frequency`` + - No + - ``"monthly"`` + - The frequency of the data. Options include "monthly", "diurnal_8xdaily", "daily". + * - ``grid`` + - No + - ``""`` + - Name of the output grid (e.g., ``180x360_aave``). Used for naming + output directories. + * - ``input_files`` + - No + - ``"eam.h0"`` + - Pattern matching the input history files (e.g., ``eam.h0``, + ``eam.h1``). + * - ``input_subdir`` + - No + - ``"archive/atm/hist"`` + - Subdirectory under ``input``/``case`` containing the data files. + * - ``mapping_file`` + - No + - ``""`` + - Path to the mapping (regridding) file. Leave empty for no regridding. + * - ``plugins`` + - No + - ``""`` + - External zppy plugin modules + * - ``vars`` + - No + - ``"FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,PSL"`` + - Variables to process. An empty string processes all variables. + * - ``years`` + - No + - ``[""]`` + - Year ranges to process. Format: ``"start:end:increment"`` (e.g., + ``"1:100:20"``). + +**Deprecated Parameters** + +There is 1 deprecated parameter. Specifying it will have no effect. + +.. code-block:: text + + ncclimo_cmd Parameter checking & inferring ============================== @@ -130,3 +388,26 @@ For the ``ilamb`` task: **Required parameters, by e3sm_diags set** See `Confluence `_ for tables of which sets require which parameters. + +Deprecated parameters +===================== + +The following ``zppy`` parameters no longer perform any function. + +These are no longer defined in ``zppy/defaults/default.ini``: + +.. code-block:: text + + e3sm_to_cmip_environment_commands + ts_fmt + scratch + atmosphere_only + plot_names + vars_exclude + +These are still defined in ``zppy/defaults/default.ini``, but have no effect: + +.. code-block:: text + ncclimo_cmd + nrows + ncols diff --git a/docs/source/user_guide/tasks/climo.rst b/docs/source/user_guide/tasks/climo.rst index 527b7047..d52fa42c 100644 --- a/docs/source/user_guide/tasks/climo.rst +++ b/docs/source/user_guide/tasks/climo.rst @@ -31,8 +31,7 @@ Configuration example Parameters ---------- -The following parameters are specific to the ``climo`` task. In addition, -all :ref:`common parameters ` from the ``[default]`` section apply. +These 3 parameters are specific to the ``climo`` task: .. list-table:: :header-rows: 1 @@ -42,34 +41,15 @@ all :ref:`common parameters ` from the ``[default]`` section apply. - Required - Default - Description - * - ``active`` - - No - - ``False`` - - Set to ``True`` to enable this task. * - ``exclude`` - No - ``False`` - Set to ``True`` to exclude this subtask from running (subsection-level). - * - ``parallel`` - - No - - ``"mpi"`` - - The ``parallel`` option passed to ``ncclimo``. - Overrides the ``[default]`` value (which is ``""``). - * - ``nodes`` - - No - - ``4`` - - Number of compute nodes. Overrides the ``[default]`` value (``1``). * - ``climo_jobs`` - No - ``0`` - Number of simultaneous ``ncclimo`` jobs. ``0`` uses ``ncclimo``'s default. - * - ``vars`` - - No - - ``""`` - - Variables to process. An empty string processes *all* variables (no - ``-v`` flag is passed to ``ncclimo``). Overrides the ``[default]`` - value. * - ``input_component`` - No - ``""`` @@ -77,8 +57,11 @@ all :ref:`common parameters ` from the ``[default]`` section apply. ``eamxx``, ``elm``, ``mosart``). Used to set processing type internally. -Inherited common parameters most relevant to ``climo`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These 3 parameters have ``climo``-specific defaults, which means even if they are set at the top level (``[default]``) section, these default values will be used instead. Therefore, to specify a custom value, these parameters must be defined inside ``[climo]``: .. list-table:: :header-rows: 1 @@ -88,58 +71,23 @@ Inherited common parameters most relevant to ``climo`` - Required - Default - Description - * - ``case`` - - **Yes** - - *(none)* - - The case name of the simulation. - * - ``input`` - - **Yes** - - *(none)* - - The top-level directory of the simulation output to post-process. - * - ``output`` - - **Yes** - - *(none)* - - Where the post-processing results (``post/`` directory) should go. - * - ``years`` - - No - - ``[""]`` - - Year ranges to process. Format: ``"start:end:increment"`` (e.g., - ``"1:100:20"``). - * - ``input_files`` - - No - - ``"eam.h0"`` - - Pattern matching the input history files (e.g., ``eam.h0``, - ``eam.h1``). - * - ``input_subdir`` - - No - - ``"archive/atm/hist"`` - - Subdirectory under ``input``/``case`` containing the data files. - * - ``mapping_file`` - - No - - ``""`` - - Path to the mapping (regridding) file. Leave empty for no regridding. - * - ``grid`` - - No - - ``""`` - - Name of the output grid (e.g., ``180x360_aave``). Used for naming - output directories. - * - ``walltime`` - - No - - ``"02:00:00"`` - - Maximum wall time for the SLURM job. - * - ``partition`` + * - ``parallel`` - No - - ``""`` - - SLURM partition to submit the job to. - * - ``account`` + - ``"mpi"`` + - The ``parallel`` option passed to ``ncclimo``. + Overrides the ``[default]`` value (which is ``""``). + * - ``nodes`` - No - - ``""`` - - SLURM account to charge. - * - ``environment_commands`` + - ``4`` + - Number of compute nodes. Overrides the ``[default]`` value (``1``). + * - ``vars`` - No - ``""`` - - Shell commands to set up the software environment before running the - task (e.g., ``source /path/to/e3sm_unified.sh``). + - Variables to process. An empty string processes *all* variables (no + ``-v`` flag is passed to ``ncclimo``). Overrides the ``[default]`` + value. + +For other top-level parameters, see :ref:`top-level parameters `. Dependencies ------------ diff --git a/docs/source/user_guide/tasks/e3sm_to_cmip.rst b/docs/source/user_guide/tasks/e3sm_to_cmip.rst index 1d45816f..06b96d31 100644 --- a/docs/source/user_guide/tasks/e3sm_to_cmip.rst +++ b/docs/source/user_guide/tasks/e3sm_to_cmip.rst @@ -11,6 +11,8 @@ and :doc:`pcmdi_diags`. Parameters ---------- +These 6 parameters are specific to the ``e3sm_to_cmip`` task. + .. list-table:: :header-rows: 1 :widths: 22 10 18 50 @@ -19,10 +21,6 @@ Parameters - Required - Default - Description - * - ``active`` - - No - - ``False`` - - Set to ``True`` to enable this task. * - ``cmip_metadata`` - No - ``"inclusions/e3sm_to_cmip/default_metadata.json"`` @@ -50,68 +48,11 @@ Parameters - ``""`` - Source vertical coordinate file for ``ncremap``. Required for EAMxx (L128 grid). - * - ``ts_subsection`` - - No - - ``""`` - - Name of the ``[ts]`` subtask to depend on. If empty and - ``infer_section_parameters`` is ``True``, inferred from the - subsection name. - * - ``ts_atm_subsection`` - - No - - ``""`` - - Name of the ``[ts]`` atmosphere subtask (for multi-component runs). - * - ``ts_land_subsection`` - - No - - ``""`` - - Name of the ``[ts]`` land subtask (for multi-component runs). - -Inherited common parameters most relevant to ``e3sm_to_cmip`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. list-table:: - :header-rows: 1 - :widths: 22 10 18 50 +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ - * - Parameter - - Required - - Default - - Description - * - ``case`` - - **Yes** - - *(none)* - - The case name of the simulation. - * - ``input`` - - **Yes** - - *(none)* - - The top-level directory of the simulation output. - * - ``output`` - - **Yes** - - *(none)* - - Where the post-processing results go. - * - ``years`` - - No - - ``[""]`` - - Year ranges to process. - * - ``ts_grid`` - - No - - ``"180x360_aave"`` - - Grid name of the ``[ts]`` subtask to depend on. - * - ``ts_atm_grid`` - - No - - ``"180x360_aave"`` - - Grid name of the atmosphere ``[ts]`` subtask. - * - ``ts_land_grid`` - - No - - ``"180x360_aave"`` - - Grid name of the land ``[ts]`` subtask. - * - ``ts_num_years`` - - No - - ``5`` - - Years increment from the dependent ``[ts]`` task. - * - ``environment_commands`` - - No - - ``""`` - - Shell commands to activate the software environment. +For top-level parameters, see :ref:`top-level parameters `. Dependencies ------------ diff --git a/docs/source/user_guide/tasks/index.rst b/docs/source/user_guide/tasks/index.rst index 89b3f33f..b0b71dd3 100644 --- a/docs/source/user_guide/tasks/index.rst +++ b/docs/source/user_guide/tasks/index.rst @@ -49,14 +49,14 @@ Listed by the order they are evaluated in `__main__.py`: .. toctree:: :hidden: + bundle climo ts e3sm_to_cmip + tc_analysis e3sm_diags mpas_analysis global_time_series - tc_analysis ilamb livvkit pcmdi_diags - bundle diff --git a/docs/source/user_guide/tasks/tc_analysis.rst b/docs/source/user_guide/tasks/tc_analysis.rst index bc1b6d4d..fac8ce76 100644 --- a/docs/source/user_guide/tasks/tc_analysis.rst +++ b/docs/source/user_guide/tasks/tc_analysis.rst @@ -10,6 +10,8 @@ detect and track TCs. Parameters ---------- +This parameter is specific to the ``tc_analysis`` task: + .. list-table:: :header-rows: 1 :widths: 22 10 18 50 @@ -18,23 +20,16 @@ Parameters - Required - Default - Description - * - ``active`` - - No - - ``False`` - - Set to ``True`` to enable this task. - * - ``input_files`` - - No - - ``"eam.h2"`` - - Input history file pattern. Overrides the ``[default]`` value - (``eam.h0``). * - ``res`` - No - ``""`` - Resolution string for the ``--res`` option. If not set, ``zppy`` attempts to infer it from the topography file. -Inherited common parameters most relevant to ``tc_analysis`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This parameter has a ``tc_analysis``-specific default, which means even if it is set at the top level (``[default]``) section, the default value will be used instead. Therefore, to specify a custom value, this parameter must be defined inside ``[tc_analysis]``: .. list-table:: :header-rows: 1 @@ -44,34 +39,13 @@ Inherited common parameters most relevant to ``tc_analysis`` - Required - Default - Description - * - ``case`` - - **Yes** - - *(none)* - - The case name of the simulation. - * - ``input`` - - **Yes** - - *(none)* - - The top-level directory of the simulation output. - * - ``output`` - - **Yes** - - *(none)* - - Where the post-processing results go. - * - ``years`` - - No - - ``[""]`` - - Year ranges to process. - * - ``input_subdir`` - - No - - ``"archive/atm/hist"`` - - Subdirectory under ``input``/``case`` containing the data files. - * - ``environment_commands`` - - No - - ``""`` - - Shell commands to activate the software environment. - * - ``walltime`` + * - ``input_files`` - No - - ``"02:00:00"`` - - Maximum wall time for the SLURM job. + - ``"eam.h2"`` + - Input history file pattern. Overrides the ``[default]`` value + (``eam.h0``). + +For other top-level parameters, see :ref:`top-level parameters `. Dependencies ------------ diff --git a/docs/source/user_guide/tasks/ts.rst b/docs/source/user_guide/tasks/ts.rst index 5abbd493..5473dafd 100644 --- a/docs/source/user_guide/tasks/ts.rst +++ b/docs/source/user_guide/tasks/ts.rst @@ -35,7 +35,7 @@ Configuration example Parameters ---------- -The following parameters are specific to the ``ts`` task. +These 9 parameters are specific to the ``ts`` task. .. list-table:: :header-rows: 1 @@ -45,10 +45,6 @@ The following parameters are specific to the ``ts`` task. - Required - Default - Description - * - ``active`` - - No - - ``False`` - - Set to ``True`` to enable this task. * - ``area_nm`` - No - ``"area"`` @@ -96,66 +92,11 @@ The following parameters are specific to the ``ts`` task. ``diagnostics_base_path`` when ``input_component`` is ``eamxx`` and this is left empty. -Inherited common parameters most relevant to ``ts`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ -.. list-table:: - :header-rows: 1 - :widths: 22 10 18 50 +For top-level parameters, see :ref:`top-level parameters `. - * - Parameter - - Required - - Default - - Description - * - ``case`` - - **Yes** - - *(none)* - - The case name of the simulation. - * - ``input`` - - **Yes** - - *(none)* - - The top-level directory of the simulation output to post-process. - * - ``output`` - - **Yes** - - *(none)* - - Where the post-processing results (``post/`` directory) should go. - * - ``years`` - - No - - ``[""]`` - - Year ranges to process. Format: ``"start:end:increment"`` (e.g., - ``"1:100:10"``). - * - ``input_files`` - - No - - ``"eam.h0"`` - - Pattern matching the input history files. - * - ``input_subdir`` - - No - - ``"archive/atm/hist"`` - - Subdirectory under ``input``/``case`` containing the data files. - * - ``mapping_file`` - - No - - ``""`` - - Path to the mapping (regridding) file. Leave empty for no regridding. - * - ``grid`` - - No - - ``""`` - - Name of the output grid (e.g., ``180x360_aave``). - * - ``vars`` - - No - - ``"FSNTOA,FLUT,..."`` - - Variables to process. An empty string processes all variables. - * - ``walltime`` - - No - - ``"02:00:00"`` - - Maximum wall time for the SLURM job. - * - ``partition`` - - No - - ``""`` - - SLURM partition to submit the job to. - * - ``account`` - - No - - ``""`` - - SLURM account to charge. Dependencies ------------ From bc522bc3d351d526d76693e24f683ae4b59ba561 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 9 Jun 2026 00:05:49 +0000 Subject: [PATCH 15/27] Revise task parameter docs to match default.ini structure --- docs/source/user_guide/tasks/e3sm_diags.rst | 189 ++++++----- .../user_guide/tasks/global_time_series.rst | 65 +++- docs/source/user_guide/tasks/ilamb.rst | 21 +- docs/source/user_guide/tasks/livvkit.rst | 11 +- .../source/user_guide/tasks/mpas_analysis.rst | 44 ++- docs/source/user_guide/tasks/pcmdi_diags.rst | 312 ++++++++++++++---- 6 files changed, 459 insertions(+), 183 deletions(-) diff --git a/docs/source/user_guide/tasks/e3sm_diags.rst b/docs/source/user_guide/tasks/e3sm_diags.rst index 337ff74d..85e41b8f 100644 --- a/docs/source/user_guide/tasks/e3sm_diags.rst +++ b/docs/source/user_guide/tasks/e3sm_diags.rst @@ -32,6 +32,11 @@ available sets are: Parameters ---------- +These 35 parameters are specific to the ``e3sm_diags`` task. + +General parameters +~~~~~~~~~~~~~~~~~~ + .. list-table:: :header-rows: 1 :widths: 28 10 18 44 @@ -40,30 +45,42 @@ Parameters - Required - Default - Description - * - ``active`` + * - ``backend`` - No - - ``False`` - - Set to ``True`` to enable this task. - * - ``sets`` + - ``"mpl"`` + - Plotting backend. + * - ``cfg`` - No - - ``"lat_lon,zonal_mean_xy,..."`` - - List of diagnostic sets to run. - * - ``run_type`` + - ``""`` + - Path to an additional E3SM Diags configuration file. + * - ``climo_diurnal_frequency`` - No - - ``"model_vs_obs"`` - - Comparison type: ``"model_vs_obs"`` or ``"model_vs_model"``. + - ``""`` + - Frequency name for the diurnal cycle ``[climo]`` subtask. + * - ``climo_diurnal_subsection`` + - No + - ``""`` + - ``[climo]`` subtask for diurnal cycle data. * - ``climo_subsection`` - No - ``""`` - Name of the ``[climo]`` subtask to depend on. - * - ``backend`` + * - ``dc_obs_climo`` - No - - ``"mpl"`` - - Plotting backend. - * - ``cfg`` + - ``""`` + - Path to observation climatology for the ``diurnal_cycle`` set. + * - ``diff_title`` + - No + - ``"Model - Observations"`` + - Title for difference plots. + * - ``keep_mvm_case_name_in_fig`` + - No + - ``True`` + - Include case name in model-vs-model output paths. + * - ``gauges_path`` - No - ``""`` - - Path to an additional E3SM Diags configuration file. + - Path to stream gauge data for ``streamflow`` model-vs-model runs. * - ``multiprocessing`` - No - ``True`` @@ -72,6 +89,11 @@ Parameters - No - ``24`` - Number of worker processes. + * - ``obs_ts`` + - No + - ``""`` + - Path to observation time-series data for ``enso_diags``, ``qbo``, and + ``area_mean_time_series``. * - ``output_format`` - No - ``["png"]`` @@ -80,83 +102,41 @@ Parameters - No - ``[]`` - Output formats for subplots. - * - ``short_name`` - - No - - ``""`` - - Short name used as test name and label. - * - ``tag`` - - No - - ``"model_vs_obs"`` - - Label for the results directory. - * - ``swap_test_ref`` - - No - - ``False`` - - Swap test and reference in model-vs-model runs. - * - ``keep_mvm_case_name_in_fig`` - - No - - ``True`` - - Include case name in model-vs-model output paths. - * - ``obs_ts`` - - No - - ``""`` - - Path to observation time-series data for ``enso_diags``, ``qbo``, - ``area_mean_time_series`` sets. - * - ``reference_data_path`` + +Reference-data parameters +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``ref_final_yr`` - No* - ``""`` - - Path to reference climatology data. - **Required for** ``run_type="model_vs_model"``. + - End year of reference data. Required for ``qbo`` and certain + model-vs-model sets. * - ``ref_name`` - No* - ``""`` - Reference dataset name. **Required for** ``run_type="model_vs_model"``. - * - ``short_ref_name`` + * - ``ref_start_yr`` - No* - ``""`` - - Short reference name. - **Required for** ``run_type="model_vs_model"``. - * - ``diff_title`` - - No - - ``"Model - Observations"`` - - Title for difference plots. + - Start year of reference data. Required for ``enso_diags``, ``qbo``. * - ``ref_years`` - No - ``[""]`` - Year ranges for reference data in model-vs-model runs. - * - ``ref_start_yr`` - - No* - - ``""`` - - Start year of reference data. Required for ``enso_diags``, ``qbo``. - * - ``ref_final_yr`` + * - ``reference_data_path`` - No* - ``""`` - - End year of reference data. Required for ``qbo`` and certain - model-vs-model sets. - * - ``ts_subsection`` - - No - - ``""`` - - Name of the ``[ts]`` subtask to depend on. - * - ``ts_num_years_ref`` - - No - - ``5`` - - Year increment for reference time-series data. - * - ``ts_daily_subsection`` - - No - - ``""`` - - ``[ts]`` subtask for daily data (required for ``tropical_subseasonal``). - * - ``climo_diurnal_subsection`` - - No - - ``""`` - - ``[climo]`` subtask for diurnal cycle data. - * - ``climo_diurnal_frequency`` - - No - - ``""`` - - Frequency name for the diurnal cycle ``[climo]`` subtask. - * - ``dc_obs_climo`` - - No - - ``""`` - - Path to observation climatology for ``diurnal_cycle`` set. + - Path to reference climatology data. + **Required for** ``run_type="model_vs_model"``. * - ``reference_data_path_climo_diurnal`` - No - ``""`` @@ -177,18 +157,69 @@ Parameters - No - ``""`` - Reference river time-series path for ``streamflow`` MVM runs. + +Set-selection and dependency parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``run_type`` + - No + - ``"model_vs_obs"`` + - Comparison type: ``"model_vs_obs"`` or ``"model_vs_model"``. + * - ``sets`` + - No + - ``"lat_lon,zonal_mean_xy,..."`` + - List of diagnostic sets to run. + * - ``short_name`` + - No + - ``""`` + - Short name used as test name and label. + * - ``short_ref_name`` + - No* + - ``""`` + - Short reference name. + **Required for** ``run_type="model_vs_model"``. * - ``streamflow_obs_ts`` - No - ``""`` - Path to observation data for ``streamflow`` set. - * - ``gauges_path`` + * - ``swap_test_ref`` - No - - ``""`` - - Path to stream gauge data for ``streamflow`` MVM runs. + - ``False`` + - Swap test and reference in model-vs-model runs. + * - ``tag`` + - No + - ``"model_vs_obs"`` + - Label for the results directory. * - ``tc_obs`` - No - ``""`` - Path to observation data for ``tc_analysis`` set. + * - ``ts_num_years_ref`` + - No + - ``5`` + - Year increment for reference time-series data. + * - ``ts_subsection`` + - No + - ``""`` + - Name of the ``[ts]`` subtask to depend on. + * - ``ts_daily_subsection`` + - No + - ``""`` + - ``[ts]`` subtask for daily data (required for + ``tropical_subseasonal``). + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For top-level parameters, see :ref:`top-level parameters `. Dependencies ------------ diff --git a/docs/source/user_guide/tasks/global_time_series.rst b/docs/source/user_guide/tasks/global_time_series.rst index cc10995e..0284d5d8 100644 --- a/docs/source/user_guide/tasks/global_time_series.rst +++ b/docs/source/user_guide/tasks/global_time_series.rst @@ -11,6 +11,11 @@ key climate metrics. It can optionally produce a Viewer page. It depends on Parameters ---------- +These 14 parameters are specific to the ``global_time_series`` task. + +Core plotting options +~~~~~~~~~~~~~~~~~~~~~ + .. list-table:: :header-rows: 1 :widths: 28 10 18 44 @@ -19,14 +24,6 @@ Parameters - Required - Default - Description - * - ``active`` - - No - - ``False`` - - Set to ``True`` to enable this task. - * - ``ts_years`` - - No - - ``[""]`` - - Year ranges for the time-series sub-runs to depend on. * - ``climo_years`` - No - ``[""]`` @@ -52,10 +49,10 @@ Parameters - ``"glb,n,s"`` - Regions to plot: ``glb`` (global), ``n`` (northern hemisphere), ``s`` (southern hemisphere). - * - ``input_subdir`` + * - ``ts_years`` - No - - ``"archive/ocn/hist"`` - - Subdirectory with ocean data. Overrides the ``[default]`` value. + - ``[""]`` + - Year ranges for the time-series sub-runs to depend on. * - ``moc_file`` - No - ``""`` @@ -82,11 +79,27 @@ Parameters - ``change_sea_level`` — requires ocean data Remove the three ocean plots if you don't have ocean data. + +Viewer-specific plot options +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description * - ``plots_atm`` - No - ``""`` - Extra atmosphere plots (when ``make_viewer = True``). These should be a subset of the variables from ``ts`` global subtasks. + * - ``plots_ice`` + - No + - ``""`` + - Extra sea-ice plots (when ``make_viewer = True``). * - ``plots_lnd`` - No - ``""`` @@ -96,10 +109,34 @@ Parameters - No - ``""`` - Extra ocean plots (when ``make_viewer = True``). - * - ``plots_ice`` + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This parameter has a ``global_time_series``-specific default, which means even +if it is set at the top level (``[default]``) section, this default value will +be used instead. Therefore, to specify a custom value, this parameter must be +defined inside ``[global_time_series]``: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``input_subdir`` - No - - ``""`` - - Extra sea-ice plots (when ``make_viewer = True``). + - ``"archive/ocn/hist"`` + - Subdirectory with ocean data. Overrides the ``[default]`` value + (``archive/atm/hist``). + +Deprecated parameters +~~~~~~~~~~~~~~~~~~~~~ + +The following parameters are still in ``default.ini`` but are deprecated and +have no effect: ``ncols``, ``nrows``. Dependencies ------------ diff --git a/docs/source/user_guide/tasks/ilamb.rst b/docs/source/user_guide/tasks/ilamb.rst index 23cecd40..3751285d 100644 --- a/docs/source/user_guide/tasks/ilamb.rst +++ b/docs/source/user_guide/tasks/ilamb.rst @@ -11,6 +11,8 @@ provide CMIP-format land and atmosphere time-series files. Parameters ---------- +These 5 parameters are specific to the ``ilamb`` task. + .. list-table:: :header-rows: 1 :widths: 22 10 18 50 @@ -19,10 +21,6 @@ Parameters - Required - Default - Description - * - ``active`` - - No - - ``False`` - - Set to ``True`` to enable this task. * - ``cfg`` - No - ``"inclusions/ilamb/ilamb.cfg"`` @@ -46,16 +44,11 @@ Parameters - No - ``False`` - Set to ``True`` for land-only runs (skips atmosphere dependencies). - * - ``ts_atm_subsection`` - - No - - ``""`` - - Name of the ``[ts]`` atmosphere subtask to depend on. Inferred as - ``atm_monthly_180x360_aave`` if not specified and not a land-only run. - * - ``ts_land_subsection`` - - No - - ``""`` - - Name of the ``[ts]`` land subtask to depend on. Inferred as - ``land_monthly`` if not specified. + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For top-level parameters, see :ref:`top-level parameters `. Dependencies ------------ diff --git a/docs/source/user_guide/tasks/livvkit.rst b/docs/source/user_guide/tasks/livvkit.rst index acce900a..6ba8a0fe 100644 --- a/docs/source/user_guide/tasks/livvkit.rst +++ b/docs/source/user_guide/tasks/livvkit.rst @@ -11,6 +11,8 @@ generated by the :doc:`climo` task against reanalysis datasets. Parameters ---------- +These 4 parameters are specific to the ``livvkit`` task. + .. list-table:: :header-rows: 1 :widths: 22 10 18 50 @@ -19,10 +21,6 @@ Parameters - Required - Default - Description - * - ``active`` - - No - - ``False`` - - Set to ``True`` to enable this task. * - ``climo_subsections`` - No - ``[""]`` @@ -40,6 +38,11 @@ Parameters - ``"cmb,smb,energy_racmo,energy_era5,energy_merra2,energy_ceres"`` - Validation sets to perform. +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For top-level parameters, see :ref:`top-level parameters `. + Dependencies ------------ diff --git a/docs/source/user_guide/tasks/mpas_analysis.rst b/docs/source/user_guide/tasks/mpas_analysis.rst index b0443687..30274e8f 100644 --- a/docs/source/user_guide/tasks/mpas_analysis.rst +++ b/docs/source/user_guide/tasks/mpas_analysis.rst @@ -18,6 +18,11 @@ otherwise it is treated as MVO. Parameters ---------- +These 25 parameters are specific to the ``mpas_analysis`` task. + +MPAS-Analysis configuration parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + .. list-table:: :header-rows: 1 :widths: 28 10 18 44 @@ -26,14 +31,14 @@ Parameters - Required - Default - Description - * - ``active`` + * - ``anomalyRefYear`` - No - - ``False`` - - Set to ``True`` to enable this task. - * - ``generate`` + - ``1`` + - Reference year for anomaly calculations. + * - ``cache`` - No - - ``"all,no_landIceCavities,..."`` - - List of MPAS-Analysis analyses to generate. + - ``True`` + - Whether to cache MPAS-Analysis intermediate results. * - ``ts_years`` - No - ``[""]`` @@ -84,14 +89,10 @@ Parameters - No - ``[""]`` - Year ranges for reference ENSO sub-runs. - * - ``anomalyRefYear`` - - No - - ``1`` - - Reference year for anomaly calculations. - * - ``cache`` + * - ``generate`` - No - - ``True`` - - Whether to cache MPAS-Analysis intermediate results. + - ``"all,no_landIceCavities,..."`` + - List of MPAS-Analysis analyses to generate. * - ``mapMpiTasks`` - No - ``6`` @@ -136,6 +137,23 @@ Parameters - No - ``"streams.ocean"`` - Name of the MPAS ocean streams file. + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This parameter has an ``mpas_analysis``-specific default, which means even if +it is set at the top level (``[default]``) section, this default value will be +used instead. Therefore, to specify a custom value, this parameter must be +defined inside ``[mpas_analysis]``: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description * - ``walltime`` - No - ``"06:00:00"`` diff --git a/docs/source/user_guide/tasks/pcmdi_diags.rst b/docs/source/user_guide/tasks/pcmdi_diags.rst index c72d7b6e..31086718 100644 --- a/docs/source/user_guide/tasks/pcmdi_diags.rst +++ b/docs/source/user_guide/tasks/pcmdi_diags.rst @@ -22,8 +22,10 @@ The ``current_set`` parameter selects which PMP diagnostic to run: Parameters ---------- -Most useful at the task level (all sets) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +These 72 parameters are specific to the ``pcmdi_diags`` task. + +Task-level parameters (all sets) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 @@ -33,47 +35,54 @@ Most useful at the task level (all sets) - Required - Default - Description - * - ``active`` - - No - - ``False`` - - Set to ``True`` to enable this task. - * - ``run_type`` - - No - - ``"model_vs_obs"`` - - ``"model_vs_obs"`` or ``"model_vs_model"``. Note: in PCMDI context, - model-vs-model compares two model simulations against observations. * - ``figure_format`` - No - ``"png"`` - Output figure format. + * - ``run_type`` + - No + - ``"model_vs_obs"`` + - ``"model_vs_obs"`` or ``"model_vs_model"``. * - ``model_name`` - No - ``"e3sm.historical.v3-LR.0051"`` - - Model name (required for MVO runs). + - Model name (required for model-vs-observations runs). * - ``model_tableID`` - No - ``"Amon"`` - - CMIP table ID (required for MVO runs). + - CMIP table ID (required for model-vs-observations runs). * - ``model_name_ref`` - No - ``"ERA5"`` - - Reference model name (required for MVM runs). + - Reference model name (required for model-vs-model runs). * - ``model_tableID_ref`` - No - ``"Amon"`` - - Reference CMIP table ID (required for MVM runs). + - Reference CMIP table ID (required for model-vs-model runs). * - ``e3sm_to_cmip_atm_subsection`` - No - ``""`` - - Name of the ``[e3sm_to_cmip]`` atm subtask to depend on. + - Name of the ``[e3sm_to_cmip]`` atmosphere subtask to depend on. * - ``generate_sftlf`` - No - ``True`` - - Flag to process the land/sea mask within PCMDI. - * - ``grid`` + - Process the land/sea mask within PCMDI. + * - ``mov_plot_obs`` - No - - ``"180x360_aave"`` - - Model data grid after remapping. + - ``True`` + - Generate variability-modes observation plots. + * - ``mov_plot_model`` + - No + - ``True`` + - Generate variability-modes model plots. + * - ``mov_nc_out_obs`` + - No + - ``True`` + - Write variability-modes observation NetCDF output. + * - ``mov_nc_out_model`` + - No + - ``True`` + - Write variability-modes model NetCDF output. * - ``multiprocessing`` - No - ``True`` @@ -85,8 +94,23 @@ Most useful at the task level (all sets) * - ``obs_ts`` - No - ``""`` - - Path to observation time-series data. Required for - ``mean_climate``, ``variability_modes_*``, ``enso`` sets. + - Path to observation time-series data. + * - ``pcmdi_debug`` + - No + - ``False`` + - Enable debug mode in PCMDI diagnostics. + * - ``save_test_clims`` + - No + - ``True`` + - Save derived test climatology data. + * - ``reference_data_path`` + - No + - ``""`` + - Path to reference model data for model-vs-model runs. + * - ``reference_data_path_ts`` + - No + - ``""`` + - Path to reference model time-series data for model-vs-model runs. * - ``ts_num_years`` - No - ``5`` @@ -96,8 +120,8 @@ Most useful at the task level (all sets) - ``[""]`` - Year ranges for test model data. -Per-subtask parameters -~~~~~~~~~~~~~~~~~~~~~~~ +Per-subtask shared parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 @@ -110,27 +134,26 @@ Per-subtask parameters * - ``current_set`` - No - ``""`` - - The diagnostic set to run. If not set, inferred from the subsection - name. + - Diagnostic set to run. * - ``obs_sets`` - No - ``"default"`` - - Observational datasets to use (see ``reference_alias.json``). - * - ``ref_start_yr`` - - No - - ``""`` - - Start year for reference data. + - Observational dataset aliases to use. * - ``ref_final_yr`` - No - ``""`` - End year for reference data. + * - ``ref_start_yr`` + - No + - ``""`` + - Start year for reference data. * - ``ref_years`` - No - ``[""]`` - Year ranges for reference data. -mean_climate parameters -~~~~~~~~~~~~~~~~~~~~~~~~ +Mean-climate and regridding parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 @@ -143,15 +166,7 @@ mean_climate parameters * - ``cmip_vars`` - No - ``"pr,prw,psl,..."`` - - Variables from the CMIP6 table to use. - * - ``clim_vars`` - - No - - ``"pr,prw,psl,..."`` - - Variables for mean climate metrics. - * - ``clim_regions`` - - No - - ``"global,ocean,land"`` - - Regions for mean climate metrics. + - CMIP variables to include for mean-climate diagnostics. * - ``target_grid`` - No - ``"1x1"`` @@ -159,18 +174,34 @@ mean_climate parameters * - ``target_grid_string`` - No - ``"1px1p"`` - - Description string for the target grid. + - Descriptor string for the target grid. * - ``regrid_tool`` - No - ``"esmf"`` - - Regridding tool. + - Regridding tool for atmosphere fields. + * - ``regrid_tool_ocn`` + - No + - ``"esmf"`` + - Regridding tool for ocean fields. * - ``regrid_method`` - No - ``"regrid2"`` - - Regridding method. + - Regridding method for atmosphere fields. + * - ``regrid_method_ocn`` + - No + - ``"conservative"`` + - Regridding method for ocean fields. + * - ``clim_vars`` + - No + - ``"pr,prw,psl,..."`` + - Variables for mean-climate and synthetic-plot diagnostics. + * - ``clim_regions`` + - No + - ``"global,ocean,land"`` + - Regions used for mean-climate metrics. -variability_modes parameters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Variability-modes parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. list-table:: :header-rows: 1 @@ -180,6 +211,42 @@ variability_modes parameters - Required - Default - Description + * - ``CBF`` + - No + - ``True`` + - Use Common Base Function approach. + * - ``ConvEOF`` + - No + - ``True`` + - Compute conventional EOF. + * - ``eofn_mod_max`` + - No + - ``3`` + - Maximum number of EOF modes for model data. + * - ``EofScaling`` + - No + - ``False`` + - Apply EOF scaling. + * - ``landmask`` + - No + - ``False`` + - Apply land-mask handling in variability diagnostics. + * - ``ModUnitsAdjust`` + - No + - ``""`` + - Unit-adjustment keywords for model data. + * - ``ObsUnitsAdjust`` + - No + - ``""`` + - Unit-adjustment keywords for observation data. + * - ``RmDomainMean`` + - No + - ``True`` + - Remove domain mean before variability calculations. + * - ``seasons`` + - No + - ``"monthly"`` + - Seasons/frequency bins to analyze. * - ``mova_vars`` - No - ``"psl"`` @@ -196,26 +263,153 @@ variability_modes parameters - No - ``"PDO,NPGO,AMO"`` - Modes for ``variability_modes_cpl``. - * - ``CBF`` + +ENSO parameters +~~~~~~~~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``enso_groups`` + - No + - ``"ENSO_perf,ENSO_proc,ENSO_tel"`` + - ENSO metric groups to run. + * - ``enso_vars`` + - No + - ``"psl,pr,prsn,ts,tas,tauu,tauv,hflx,hfss,rlds,rsds,rlus,rlut,rsdt"`` + - Variables used by ENSO diagnostics. + +Synthetic-plots parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``clim_viewer`` - No - ``True`` - - Use Common Base Function approach. - * - ``ConvEOF`` + - Include mean-climate panels in the synthetic viewer. + * - ``clim_years`` + - No + - ``""`` + - Year range used for mean-climate synthetic panels. + * - ``mova_viewer`` - No - ``True`` - - Compute conventional EOF. - * - ``EofScaling`` + - Include atmospheric variability-mode panels in the synthetic viewer. + * - ``mova_years`` + - No + - ``""`` + - Year range used for atmospheric variability synthetic panels. + * - ``movc_viewer`` + - No + - ``True`` + - Include coupled variability-mode panels in the synthetic viewer. + * - ``movc_years`` + - No + - ``""`` + - Year range used for coupled variability synthetic panels. + * - ``enso_viewer`` - No - ``False`` - - Apply EOF scaling. - * - ``eofn_mod_max`` + - Include ENSO panels in the synthetic viewer. + * - ``enso_years`` - No - - ``3`` - - Maximum number of EOF modes for model. - * - ``seasons`` + - ``""`` + - Year range used for ENSO synthetic panels. + * - ``save_all_data`` - No - - ``"monthly"`` - - Seasons to analyze. + - ``True`` + - Save all intermediate synthetic-diagnostic data. + * - ``reference_alias`` + - No + - ``"inclusions/pcmdi_diags/reference_alias.json"`` + - Observation alias file. + * - ``regions_specs`` + - No + - ``"inclusions/pcmdi_diags/regions_specs.json"`` + - Region-specification file. + * - ``pcmdi_version`` + - No + - ``"v3.8.2"`` + - Version tag for the zppy-pcmdi workflow. + * - ``pcmdi_viewer_template`` + - No + - ``"pcmdi_data/viewer"`` + - Template directory for synthetic-viewer generation. + * - ``pcmdi_webtitle`` + - No + - ``"E3SM-PMP-Diagnostics"`` + - Title used in generated PCMDI web output. + * - ``synthetic_metrics_list`` + - No + - ``"inclusions/pcmdi_diags/synthetic_metrics_list.json"`` + - Metrics-list file used to assemble synthetic plots. + * - ``synthetic_sets`` + - No + - ``"portrait,parcoord"`` + - Synthetic plot types to generate. + * - ``cmip_clim_dir`` + - No + - ``""`` + - Directory containing CMIP mean-climate metrics. + * - ``cmip_enso_dir`` + - No + - ``""`` + - Directory containing CMIP ENSO metrics. + * - ``cmip_movs_dir`` + - No + - ``""`` + - Directory containing CMIP variability-mode metrics. + * - ``cmip_clim_set`` + - No + - ``"cmip6.historical.v20250707"`` + - CMIP mean-climate metrics set ID. + * - ``cmip_enso_set`` + - No + - ``"cmip6.historical.v20210620"`` + - CMIP ENSO metrics set ID. + * - ``cmip_movs_set`` + - No + - ``"cmip6.historical.v20220825"`` + - CMIP variability-mode metrics set ID. + +Parameters at the top-level +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +These 2 parameters have ``pcmdi_diags``-specific defaults, which means even if +they are set at the top level (``[default]``) section, these default values +will be used instead. Therefore, to specify custom values, these parameters +must be defined inside ``[pcmdi_diags]``: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``grid`` + - No + - ``"180x360_aave"`` + - Model data grid after remapping. Overrides the ``[default]`` value + (``""``). + * - ``frequency`` + - No + - ``"mo"`` + - Data frequency for variability-mode calculations. Overrides the + ``[default]`` value (``"monthly"``). Dependencies ------------ From 35e72f1794c4e3099b07eae10794e93088705a42 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Mon, 8 Jun 2026 19:17:43 -0700 Subject: [PATCH 16/27] Revise e3sm_diags parameters doc --- docs/source/user_guide/parameters.rst | 4 - docs/source/user_guide/tasks/e3sm_diags.rst | 399 ++++++++++++++++---- 2 files changed, 324 insertions(+), 79 deletions(-) diff --git a/docs/source/user_guide/parameters.rst b/docs/source/user_guide/parameters.rst index 189138a7..59e318bc 100644 --- a/docs/source/user_guide/parameters.rst +++ b/docs/source/user_guide/parameters.rst @@ -385,10 +385,6 @@ For the ``ilamb`` task: * If ``ilamb_obs`` (the path to observation data for ``ilamb``) is undefined, assume it is the ``diagnostics_base_path`` from Mache plus ``/ilamb_data``. -**Required parameters, by e3sm_diags set** - -See `Confluence `_ for tables of which sets require which parameters. - Deprecated parameters ===================== diff --git a/docs/source/user_guide/tasks/e3sm_diags.rst b/docs/source/user_guide/tasks/e3sm_diags.rst index 85e41b8f..cf5ba265 100644 --- a/docs/source/user_guide/tasks/e3sm_diags.rst +++ b/docs/source/user_guide/tasks/e3sm_diags.rst @@ -19,12 +19,12 @@ For a table of which diagnostic sets require which parameters, see Diagnostic sets --------------- -The ``sets`` parameter controls which diagnostic sets are run. All 19 +The ``sets`` parameter controls which diagnostic sets are run. All 20 available sets are: ``aerosol_aeronet``, ``aerosol_budget``, ``annual_cycle_zonal_mean``, ``area_mean_time_series``, ``cosp_histogram``, ``diurnal_cycle``, -``enso_diags``, ``lat_lon``, ``meridional_mean_2d``, ``mp_partition``, +``enso_diags``, ``lat_lon``, ``lat_lon_land``, ``meridional_mean_2d``, ``mp_partition``, ``polar``, ``precip_pdf``, ``qbo``, ``streamflow``, ``tc_analysis``, ``tropical_subseasonal``, ``zonal_mean_2d``, ``zonal_mean_2d_stratosphere``, ``zonal_mean_xy`` @@ -37,6 +37,8 @@ These 35 parameters are specific to the ``e3sm_diags`` task. General parameters ~~~~~~~~~~~~~~~~~~ +There are 12 general parameters: + .. list-table:: :header-rows: 1 :widths: 28 10 18 44 @@ -53,34 +55,10 @@ General parameters - No - ``""`` - Path to an additional E3SM Diags configuration file. - * - ``climo_diurnal_frequency`` - - No - - ``""`` - - Frequency name for the diurnal cycle ``[climo]`` subtask. - * - ``climo_diurnal_subsection`` - - No - - ``""`` - - ``[climo]`` subtask for diurnal cycle data. - * - ``climo_subsection`` - - No - - ``""`` - - Name of the ``[climo]`` subtask to depend on. - * - ``dc_obs_climo`` - - No - - ``""`` - - Path to observation climatology for the ``diurnal_cycle`` set. - * - ``diff_title`` - - No - - ``"Model - Observations"`` - - Title for difference plots. * - ``keep_mvm_case_name_in_fig`` - No - ``True`` - Include case name in model-vs-model output paths. - * - ``gauges_path`` - - No - - ``""`` - - Path to stream gauge data for ``streamflow`` model-vs-model runs. * - ``multiprocessing`` - No - ``True`` @@ -89,11 +67,6 @@ General parameters - No - ``24`` - Number of worker processes. - * - ``obs_ts`` - - No - - ``""`` - - Path to observation time-series data for ``enso_diags``, ``qbo``, and - ``area_mean_time_series``. * - ``output_format`` - No - ``["png"]`` @@ -102,9 +75,288 @@ General parameters - No - ``[]`` - Output formats for subplots. + * - ``run_type`` + - No + - ``"model_vs_obs"`` + - Comparison type: ``"model_vs_obs"`` or ``"model_vs_model"``. + * - ``sets`` + - No + - ``"lat_lon,zonal_mean_xy,..."`` + - List of diagnostic sets to run. + * - ``short_name`` + - No + - ``""`` + - Short name used as test name and label. + * - ``swap_test_ref`` + - No + - ``False`` + - Swap test and reference in model-vs-model runs. + * - ``tag`` + - No + - ``"model_vs_obs"`` + - Label for the results directory. -Reference-data parameters -~~~~~~~~~~~~~~~~~~~~~~~~~ +Set-specific parameters +~~~~~~~~~~~~~~~~~~~~~~~ + +There are 23 set-specific parameters. + +**Overview by set** + +We will review useful parameters by set. To do this, we will group them by dependency. For more info on dependencies, see :ref:`Dependencies `. + +.. list-table:: + :header-rows: 1 + :stub-columns: 1 + :widths: 20 80 + + * - Group + - Sets + * - ``climo_month_atm`` + - ``lat_lon``, ``zonal_mean_xy``, ``zonal_mean_2d``, ``polar``, + ``cosp_histogram``, ``meridional_mean_2d``, + ``annual_cycle_zonal_mean``, ``zonal_mean_2d_stratosphere``, + ``aerosol_aeronet``, ``aerosol_budget`` + * - ``climo_month_lnd`` + - ``lat_lon_land`` + * - ``diurnal`` + - ``diurnal_cycle`` + * - ``ts_month_atm`` + - ``enso_diags``, ``qbo``, ``area_mean_time_series``, ``mp_partition`` + * - ``ts_daily_atm`` + - ``tropical_subseasonal``, ``precip_pdf`` + * - ``streamflow`` + - ``streamflow`` + * - ``tc`` + - ``tc_analysis`` + +Cell values: **Always** = required for both run types; +**MvO** = ``model_vs_obs`` only; **MvM** = ``model_vs_model`` only; +**—** = not used. Values marked with \* differ across sets within the group +(see notes below the table). + +.. list-table:: + :header-rows: 1 + :stub-columns: 1 + :widths: 30 14 14 10 16 14 12 10 + + * - Parameter + - ``climo_month_atm`` + - ``climo_month_lnd`` + - ``diurnal`` + - ``ts_month_atm`` + - ``ts_daily_atm`` + - ``streamflow`` + - ``tc`` + * - ``climo_diurnal_frequency`` + - — + - — + - Always + - — + - — + - — + - — + * - ``climo_diurnal_subsection`` + - — + - — + - Always + - — + - — + - — + - — + * - ``climo_subsection`` [#subsection]_ + - Always + - Always + - — + - — + - — + - — + - — + * - ``dc_obs_climo`` [#dc_obs]_ + - — + - — + - MvO + - — + - — + - — + - — + * - ``diff_title`` + - — + - — + - — + - MvM + - MvM + - MvM + - MvM + * - ``gauges_path`` [#inferred]_ + - — + - — + - — + - — + - — + - MvM + - — + * - ``obs_ts`` [#inferred]_ + - — + - — + - — + - MvO + - MvO + - — + - — + * - ``ref_final_yr`` \* + - — + - — + - — + - Always \* + - MvM + - MvM + - MvM + * - ``ref_name`` + - — + - — + - — + - MvM + - MvM + - MvM + - MvM + * - ``ref_start_yr`` \* + - — + - — + - — + - Always \* + - MvM + - MvM + - MvM + * - ``reference_data_path`` [#refpath]_ + - Always + - Always + - Always + - — + - — + - — + - — + * - ``reference_data_path_climo_diurnal`` [#inferred]_ + - — + - — + - MvM + - — + - — + - — + - — + * - ``reference_data_path_tc`` [#inferred]_ + - — + - — + - — + - — + - — + - — + - MvM + * - ``reference_data_path_ts`` [#inferred]_ + - — + - — + - — + - MvM + - — + - — + - — + * - ``reference_data_path_ts_daily`` [#inferred]_ + - — + - — + - — + - — + - MvM + - — + - — + * - ``reference_data_path_ts_rof`` [#inferred]_ + - — + - — + - — + - — + - — + - MvM + - — + * - ``short_ref_name`` + - — + - — + - — + - MvM + - MvM + - MvM + - MvM + * - ``streamflow_obs_ts`` [#inferred]_ + - — + - — + - — + - — + - — + - MvO + - — + * - ``tc_obs`` [#inferred]_ + - — + - — + - — + - — + - — + - — + - MvO + * - ``ts_daily_subsection`` [#subsection]_ + - — + - — + - — + - — + - Always + - — + - — + * - ``ts_num_years_ref`` + - — + - — + - — + - MvM + - MvM + - MvM + - — + * - ``ts_subsection`` [#subsection]_ + - — + - — + - — + - Always + - — + - — + - — + +.. [#subsection] Falls back to ``sub`` → ``grid`` if not set. +.. [#refpath] Inferred from ``diagnostics_base_path`` if not set. In MvM, + also used as a base path from which ``reference_data_path_climo_diurnal`` + and ``reference_data_path_ts`` are derived. +.. [#dc_obs] Inferred from ``reference_data_path`` if not set (MvO only). +.. [#inferred] Inferred from ``diagnostics_base_path`` (or a sibling path) + if not set. + +Note on \* cells (``ts_month_atm``): within ``ts_month_atm`` the four sets do not behave uniformly for ``ref_start_yr`` and ``ref_final_yr``: + +.. list-table:: + :header-rows: 1 + :widths: 30 20 20 20 20 + + * - Parameter + - ``enso_diags`` + - ``qbo`` + - ``area_mean_time_series`` + - ``mp_partition`` + * - ``ref_final_yr`` + - MvM + - Always + - — + - MvM + * - ``ref_start_yr`` + - Always + - Always + - — + - MvM + + +**Parameter list** .. list-table:: :header-rows: 1 @@ -114,6 +366,35 @@ Reference-data parameters - Required - Default - Description + * - ``climo_diurnal_frequency`` + - No + - ``""`` + - Frequency name for the diurnal cycle ``[climo]`` subtask. + * - ``climo_diurnal_subsection`` + - No + - ``""`` + - ``[climo]`` subtask for diurnal cycle data. + * - ``climo_subsection`` + - No + - ``""`` + - Name of the ``[climo]`` subtask to depend on. + * - ``dc_obs_climo`` + - No + - ``""`` + - Path to observation climatology for the ``diurnal_cycle`` set. + * - ``diff_title`` + - No + - ``"Model - Observations"`` + - Title for difference plots. + * - ``gauges_path`` + - No + - ``""`` + - Path to stream gauge data for ``streamflow`` model-vs-model runs. + * - ``obs_ts`` + - No + - ``""`` + - Path to observation time-series data for ``enso_diags``, ``qbo``, and + ``area_mean_time_series``. * - ``ref_final_yr`` - No* - ``""`` @@ -128,10 +409,6 @@ Reference-data parameters - No* - ``""`` - Start year of reference data. Required for ``enso_diags``, ``qbo``. - * - ``ref_years`` - - No - - ``[""]`` - - Year ranges for reference data in model-vs-model runs. * - ``reference_data_path`` - No* - ``""`` @@ -141,6 +418,10 @@ Reference-data parameters - No - ``""`` - Reference climatology path for ``diurnal_cycle`` MVM runs. + * - ``ref_years`` + - No + - ``[""]`` + - Year ranges for reference data in model-vs-model runs. * - ``reference_data_path_tc`` - No - ``""`` @@ -148,7 +429,7 @@ Reference-data parameters * - ``reference_data_path_ts`` - No - ``""`` - - Reference time-series path for ``enso_diags``/``qbo``/``area_mean_time_series``. + - Reference time-series path for ``enso_diags``, ``qbo``, ``area_mean_time_series``. * - ``reference_data_path_ts_daily`` - No - ``""`` @@ -157,30 +438,6 @@ Reference-data parameters - No - ``""`` - Reference river time-series path for ``streamflow`` MVM runs. - -Set-selection and dependency parameters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -.. list-table:: - :header-rows: 1 - :widths: 28 10 18 44 - - * - Parameter - - Required - - Default - - Description - * - ``run_type`` - - No - - ``"model_vs_obs"`` - - Comparison type: ``"model_vs_obs"`` or ``"model_vs_model"``. - * - ``sets`` - - No - - ``"lat_lon,zonal_mean_xy,..."`` - - List of diagnostic sets to run. - * - ``short_name`` - - No - - ``""`` - - Short name used as test name and label. * - ``short_ref_name`` - No* - ``""`` @@ -190,18 +447,15 @@ Set-selection and dependency parameters - No - ``""`` - Path to observation data for ``streamflow`` set. - * - ``swap_test_ref`` - - No - - ``False`` - - Swap test and reference in model-vs-model runs. - * - ``tag`` - - No - - ``"model_vs_obs"`` - - Label for the results directory. * - ``tc_obs`` - No - ``""`` - Path to observation data for ``tc_analysis`` set. + * - ``ts_daily_subsection`` + - No + - ``""`` + - ``[ts]`` subtask for daily data (required for + ``tropical_subseasonal``). * - ``ts_num_years_ref`` - No - ``5`` @@ -210,11 +464,6 @@ Set-selection and dependency parameters - No - ``""`` - Name of the ``[ts]`` subtask to depend on. - * - ``ts_daily_subsection`` - - No - - ``""`` - - ``[ts]`` subtask for daily data (required for - ``tropical_subseasonal``). Parameters at the top-level ~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 3a5823fc9a652f2bdb4ea480ed53ade09aa2bafc Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Mon, 8 Jun 2026 19:34:05 -0700 Subject: [PATCH 17/27] Revise mpas_analysis parameters doc --- .../source/user_guide/tasks/mpas_analysis.rst | 152 +++++++++++------- 1 file changed, 98 insertions(+), 54 deletions(-) diff --git a/docs/source/user_guide/tasks/mpas_analysis.rst b/docs/source/user_guide/tasks/mpas_analysis.rst index 30274e8f..47c18de1 100644 --- a/docs/source/user_guide/tasks/mpas_analysis.rst +++ b/docs/source/user_guide/tasks/mpas_analysis.rst @@ -23,6 +23,10 @@ These 25 parameters are specific to the ``mpas_analysis`` task. MPAS-Analysis configuration parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +**General parameters** + +There are 8 general parameters: + .. list-table:: :header-rows: 1 :widths: 28 10 18 44 @@ -31,37 +35,10 @@ MPAS-Analysis configuration parameters - Required - Default - Description - * - ``anomalyRefYear`` - - No - - ``1`` - - Reference year for anomaly calculations. * - ``cache`` - No - ``True`` - Whether to cache MPAS-Analysis intermediate results. - * - ``ts_years`` - - No - - ``[""]`` - - Year ranges for time-series sub-runs. - * - ``climo_years`` - - No - - ``[""]`` - - Year ranges for climatology sub-runs. - * - ``enso_years`` - - No - - ``[""]`` - - Year ranges for ENSO sub-runs. - * - ``reference_data_path`` - - No - - ``""`` - - For MVM runs: path to a prior zppy output directory (the one - containing ``post/``) to use as the reference run. Setting this - triggers MVM mode. - * - ``test_data_path`` - - No - - ``""`` - - For MVM runs: path to the test run's prior zppy output directory (if - different from the current run). * - ``reference_comparison_type`` - No - ``"auto"`` @@ -74,8 +51,53 @@ MPAS-Analysis configuration parameters * - ``reference_case`` - No* - ``""`` - - Case name of the reference run. Required when + - Case name of the reference run. \*Required when ``reference_data_path`` points to a non-subsection path. + * - ``generate`` + - No + - ``['all', 'no_landIceCavities', 'no_BGC', 'no_icebergs', 'no_min', 'no_max', 'no_sose', 'no_waves', 'no_eke', 'no_climatologyMapAntarcticMelt', 'no_regionalTSDiagrams', 'no_timeSeriesAntarcticMelt', 'no_timeSeriesOceanRegions', 'no_climatologyMapSose', 'no_woceTransects', 'no_soseTransects', 'no_geojsonTransects', 'no_oceanRegionalProfiles', 'no_hovmollerOceanRegions', 'no_oceanConservation']`` + - List of MPAS-Analysis analyses to generate. + * - ``PostMOC`` + - No + - ``False`` + - Whether to post-process MOC data. + * - ``purge`` + - No + - ``False`` + - Whether to purge previous MPAS-Analysis output before running. + * - ``shortTermArchive`` + - No + - ``True`` + - Whether the input uses the short-term archive directory structure. + +**Year parameters** + +There are 7 year parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``anomalyRefYear`` + - No + - ``1`` + - Reference year for anomaly calculations. + * - ``ts_years`` + - No + - ``[""]`` + - Year ranges for time-series sub-runs. + * - ``climo_years`` + - No + - ``[""]`` + - Year ranges for climatology sub-runs. + * - ``enso_years`` + - No + - ``[""]`` + - Year ranges for ENSO sub-runs. * - ``ref_ts_years`` - No - ``[""]`` @@ -89,14 +111,30 @@ MPAS-Analysis configuration parameters - No - ``[""]`` - Year ranges for reference ENSO sub-runs. - * - ``generate`` + +**Data path parameters** + +There are 6 data path parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``reference_data_path`` - No - - ``"all,no_landIceCavities,..."`` - - List of MPAS-Analysis analyses to generate. - * - ``mapMpiTasks`` + - ``""`` + - For MVM runs: path to a prior zppy output directory (the one + containing ``post/``) to use as the reference run. Setting this + triggers MVM mode. + * - ``test_data_path`` - No - - ``6`` - - Number of MPI tasks for mapping. + - ``""`` + - For MVM runs: path to the test run's prior zppy output directory (if + different from the current run). * - ``mpaso_nml`` - No - ``"mpaso_in"`` @@ -105,6 +143,31 @@ MPAS-Analysis configuration parameters - No - ``"mpassi_in"`` - Name of the MPAS sea-ice namelist file. + * - ``stream_ice`` + - No + - ``"streams.seaice"`` + - Name of the MPAS sea-ice streams file. + * - ``stream_ocn`` + - No + - ``"streams.ocean"`` + - Name of the MPAS ocean streams file. + +**Computational parameters** + +There are 4 computational parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description + * - ``mapMpiTasks`` + - No + - ``6`` + - Number of MPI tasks for mapping. * - ``ncclimoThreads`` - No - ``12`` @@ -117,26 +180,7 @@ MPAS-Analysis configuration parameters - No - ``12`` - Number of parallel analysis tasks in MPAS-Analysis. - * - ``PostMOC`` - - No - - ``False`` - - Whether to post-process MOC data. - * - ``purge`` - - No - - ``False`` - - Whether to purge previous MPAS-Analysis output before running. - * - ``shortTermArchive`` - - No - - ``True`` - - Whether the input uses the short-term archive directory structure. - * - ``stream_ice`` - - No - - ``"streams.seaice"`` - - Name of the MPAS sea-ice streams file. - * - ``stream_ocn`` - - No - - ``"streams.ocean"`` - - Name of the MPAS ocean streams file. + Parameters at the top-level ~~~~~~~~~~~~~~~~~~~~~~~~~~~ From d2aecf59a845999d107f8942f24e6c344974e3a3 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Mon, 8 Jun 2026 19:55:57 -0700 Subject: [PATCH 18/27] Revise global_time_series parameters doc --- .../user_guide/tasks/global_time_series.rst | 118 +++++++++++------- 1 file changed, 73 insertions(+), 45 deletions(-) diff --git a/docs/source/user_guide/tasks/global_time_series.rst b/docs/source/user_guide/tasks/global_time_series.rst index 0284d5d8..d0758edd 100644 --- a/docs/source/user_guide/tasks/global_time_series.rst +++ b/docs/source/user_guide/tasks/global_time_series.rst @@ -5,16 +5,18 @@ global_time_series — Global Time Series Plots The ``global_time_series`` task generates global time series plots of key climate metrics. It can optionally produce a Viewer page. It depends on - the :doc:`ts` task (for atmospheric and land global-mean time series) - and on :doc:`mpas_analysis` when ocean plots are requested. +the :doc:`ts` task (for atmospheric and land global-mean time series) +and on :doc:`mpas_analysis` when ocean plots are requested. Parameters ---------- -These 14 parameters are specific to the ``global_time_series`` task. +These 14 (non-deprecated) parameters are specific to the ``global_time_series`` task. -Core plotting options -~~~~~~~~~~~~~~~~~~~~~ +How to plot +~~~~~~~~~~~ + +There are 4 parameters that define how to plot: .. list-table:: :header-rows: 1 @@ -24,10 +26,6 @@ Core plotting options - Required - Default - Description - * - ``climo_years`` - - No - - ``[""]`` - - Year ranges for climatology data (if needed). * - ``color`` - No - ``"Blue"`` @@ -43,30 +41,26 @@ Core plotting options * - ``make_viewer`` - No - ``False`` - - Set to ``True`` to construct a Viewer page with extended plots. - * - ``regions`` - - No - - ``"glb,n,s"`` - - Regions to plot: ``glb`` (global), ``n`` (northern hemisphere), - ``s`` (southern hemisphere). - * - ``ts_years`` - - No - - ``[""]`` - - Year ranges for the time-series sub-runs to depend on. - * - ``moc_file`` - - No - - ``""`` - - Path to the MOC file (for the ``max_moc`` plot). Only needed when - ``make_viewer = False``. - * - ``mpas_analysis_subsections`` - - No - - ``[""]`` - - Names of ``[mpas_analysis]`` subtasks to depend on. Leave empty if - no subsections are defined. + - Set to ``True`` to construct a Viewer page with extended plots. If ``False``, a classic PDF will be made and/or individual PDFs will be made for additional variables specified in the ``plot_`` parameters. + + +What to plot +~~~~~~~~~~~~ + +There are 6 parameters that define what to plot: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description * - ``plots_original`` - No - ``"net_toa_flux_restom,..."`` - - Names of the standard plots to generate when ``make_viewer = False``. + - Names of the standard ("classic") plots to generate. Available plots and their variable requirements: - ``net_toa_flux_restom`` — requires ``RESTOM`` @@ -79,9 +73,34 @@ Core plotting options - ``change_sea_level`` — requires ocean data Remove the three ocean plots if you don't have ocean data. + * - ``plots_atm`` + - No + - ``""`` + - Extra atmosphere plots. These should be + a subset of the variables from ``ts`` global subtasks. + * - ``plots_ice`` + - No + - ``""`` + - Extra sea-ice plots. + * - ``plots_lnd`` + - No + - ``""`` + - Extra land plots. Set to ``"all"`` to + include every variable in the built-in land CSV file. + * - ``plots_ocn`` + - No + - ``""`` + - Extra ocean plots. + * - ``regions`` + - No + - ``"glb,n,s"`` + - Regions to plot: ``glb`` (global), ``n`` (northern hemisphere), + ``s`` (southern hemisphere). + +Ocean plot requirements +~~~~~~~~~~~~~~~~~~~~~~~ -Viewer-specific plot options -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +There are 4 parameters that are used if ocn plots are included (``plots_ocn`` or any of the 3 ocn plots of ``plots_original``): .. list-table:: :header-rows: 1 @@ -91,24 +110,33 @@ Viewer-specific plot options - Required - Default - Description - * - ``plots_atm`` + * - ``climo_years`` - No - - ``""`` - - Extra atmosphere plots (when ``make_viewer = True``). These should be - a subset of the variables from ``ts`` global subtasks. - * - ``plots_ice`` + - ``[""]`` + - Year ranges for climatology data (if needed). + * - ``ts_years`` - No - - ``""`` - - Extra sea-ice plots (when ``make_viewer = True``). - * - ``plots_lnd`` + - ``[""]`` + - Year ranges for the time-series sub-runs to depend on. + * - ``moc_file`` - No - ``""`` - - Extra land plots (when ``make_viewer = True``). Set to ``"all"`` to - include every variable in the land CSV file. - * - ``plots_ocn`` + - Path to the MOC file (for the ``max_moc`` plot). + * - ``mpas_analysis_subsections`` - No - - ``""`` - - Extra ocean plots (when ``make_viewer = True``). + - ``[""]`` + - Names of ``[mpas_analysis]`` subtasks to depend on. Leave empty if + no subsections are defined. + +Deprecated Parameters +~~~~~~~~~~~~~~~~~~~~~ + +There are 2 deprecated parameters. Specifying either will have no effect. + +.. code-block:: text + + nrows + ncols Parameters at the top-level ~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 8d61c2def23eb8d4a354bcd9299cae4787ba5b9b Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Mon, 8 Jun 2026 20:21:04 -0700 Subject: [PATCH 19/27] Revise pcmdi_diags parameter doc --- docs/source/user_guide/tasks/pcmdi_diags.rst | 87 ++++++++++++++++++-- zppy/defaults/default.ini | 3 + 2 files changed, 81 insertions(+), 9 deletions(-) diff --git a/docs/source/user_guide/tasks/pcmdi_diags.rst b/docs/source/user_guide/tasks/pcmdi_diags.rst index 31086718..2461401c 100644 --- a/docs/source/user_guide/tasks/pcmdi_diags.rst +++ b/docs/source/user_guide/tasks/pcmdi_diags.rst @@ -22,11 +22,13 @@ The ``current_set`` parameter selects which PMP diagnostic to run: Parameters ---------- -These 72 parameters are specific to the ``pcmdi_diags`` task. +These 71 parameters are specific to the ``pcmdi_diags`` task. Task-level parameters (all sets) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +There are 6 task-level (all sets) parameters: + .. list-table:: :header-rows: 1 :widths: 28 10 18 44 @@ -59,6 +61,20 @@ Task-level parameters (all sets) - No - ``"Amon"`` - Reference CMIP table ID (required for model-vs-model runs). + +Task-level parameters (all sets except synthetic_plots) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are 14 task-level (all sets except synthetic_plots) parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description * - ``e3sm_to_cmip_atm_subsection`` - No - ``""`` @@ -111,10 +127,6 @@ Task-level parameters (all sets) - No - ``""`` - Path to reference model time-series data for model-vs-model runs. - * - ``ts_num_years`` - - No - - ``5`` - - Year increment for test model data. * - ``ts_years`` - No - ``[""]`` @@ -123,6 +135,8 @@ Task-level parameters (all sets) Per-subtask shared parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +There are 5 per-subtask shared parameters: + .. list-table:: :header-rows: 1 :widths: 28 10 18 44 @@ -152,8 +166,10 @@ Per-subtask shared parameters - ``[""]`` - Year ranges for reference data. -Mean-climate and regridding parameters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Mean_climate parameters +~~~~~~~~~~~~~~~~~~~~~~~ + +There are 7 mean_climate parameters: .. list-table:: :header-rows: 1 @@ -191,6 +207,20 @@ Mean-climate and regridding parameters - No - ``"conservative"`` - Regridding method for ocean fields. + +Mean_climate & synthetic_plots parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are 2 mean_climate & synthetic_plots parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description * - ``clim_vars`` - No - ``"pr,prw,psl,..."`` @@ -200,9 +230,11 @@ Mean-climate and regridding parameters - ``"global,ocean,land"`` - Regions used for mean-climate metrics. -Variability-modes parameters +Variability_modes parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +There are 9 variability_modes parameters: + .. list-table:: :header-rows: 1 :widths: 28 10 18 44 @@ -247,6 +279,20 @@ Variability-modes parameters - No - ``"monthly"`` - Seasons/frequency bins to analyze. + +Variability_modes & synthetic_plots parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are 4 variability_modes & synthetic_plots parameters: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description * - ``mova_vars`` - No - ``"psl"`` @@ -267,6 +313,8 @@ Variability-modes parameters ENSO parameters ~~~~~~~~~~~~~~~ +There is 1 ENSO parameter: + .. list-table:: :header-rows: 1 :widths: 28 10 18 44 @@ -279,14 +327,30 @@ ENSO parameters - No - ``"ENSO_perf,ENSO_proc,ENSO_tel"`` - ENSO metric groups to run. + +ENSO & synthetic_plots parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There is 1 ENSO & synthetic_plots parameter: + +.. list-table:: + :header-rows: 1 + :widths: 28 10 18 44 + + * - Parameter + - Required + - Default + - Description * - ``enso_vars`` - No - ``"psl,pr,prsn,ts,tas,tauu,tauv,hflx,hfss,rlds,rsds,rlus,rlut,rsdt"`` - Variables used by ENSO diagnostics. -Synthetic-plots parameters +Synthetic_plots parameters ~~~~~~~~~~~~~~~~~~~~~~~~~~ +There are 22 synthetic_plots parameters: + .. list-table:: :header-rows: 1 :widths: 28 10 18 44 @@ -405,6 +469,11 @@ must be defined inside ``[pcmdi_diags]``: - ``"180x360_aave"`` - Model data grid after remapping. Overrides the ``[default]`` value (``""``). + * - ``ts_num_years`` + - No + - ``5`` + - Year increment for test model data. Overrides the + ``[default]`` value (``"5"``), which in this case is actually the same value. * - ``frequency`` - No - ``"mo"`` diff --git a/zppy/defaults/default.ini b/zppy/defaults/default.ini index 948fadbb..05800b22 100755 --- a/zppy/defaults/default.ini +++ b/zppy/defaults/default.ini @@ -541,6 +541,7 @@ e3sm_to_cmip_atm_subsection = string(default="") # Flag to process the land/sea mask within pcmdi generate_sftlf = string(default=True) # model data grid after remapping +# NOTE: always overrides value in [default] grid = string(default="180x360_aave") mov_plot_obs = string(default=True) mov_plot_model = string(default=True) @@ -562,6 +563,7 @@ save_test_clims = string(default=True) reference_data_path = string(default="") reference_data_path_ts = string(default="") # The years increment for test model data +# NOTE: always overrides value in [default] ts_num_years = integer(default=5) # The years range for test model data ts_years = string_list(default=list("")) @@ -619,6 +621,7 @@ ConvEOF = string(default=True) eofn_mod_max = integer(default=3) EofScaling = string(default=False) # Frequency of the model data +# NOTE: always overrides value in [default] frequency = string(default="mo") landmask = string(default=False) # Keywords for unit conversion in pcmdi (model) From 4ef7e9191583598e38d172f4e859571479f58de3 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Mon, 8 Jun 2026 20:24:25 -0700 Subject: [PATCH 20/27] Add top-level parameter link --- docs/source/user_guide/tasks/global_time_series.rst | 2 ++ docs/source/user_guide/tasks/mpas_analysis.rst | 2 ++ docs/source/user_guide/tasks/pcmdi_diags.rst | 2 ++ 3 files changed, 6 insertions(+) diff --git a/docs/source/user_guide/tasks/global_time_series.rst b/docs/source/user_guide/tasks/global_time_series.rst index d0758edd..6652dd9b 100644 --- a/docs/source/user_guide/tasks/global_time_series.rst +++ b/docs/source/user_guide/tasks/global_time_series.rst @@ -160,6 +160,8 @@ defined inside ``[global_time_series]``: - Subdirectory with ocean data. Overrides the ``[default]`` value (``archive/atm/hist``). +For other top-level parameters, see :ref:`top-level parameters `. + Deprecated parameters ~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/source/user_guide/tasks/mpas_analysis.rst b/docs/source/user_guide/tasks/mpas_analysis.rst index 47c18de1..1dd1bdc5 100644 --- a/docs/source/user_guide/tasks/mpas_analysis.rst +++ b/docs/source/user_guide/tasks/mpas_analysis.rst @@ -203,6 +203,8 @@ defined inside ``[mpas_analysis]``: - ``"06:00:00"`` - Maximum wall time. Overrides the ``[default]`` value (``02:00:00``). +For other top-level parameters, see :ref:`top-level parameters `. + Year range behavior for MVM runs --------------------------------- diff --git a/docs/source/user_guide/tasks/pcmdi_diags.rst b/docs/source/user_guide/tasks/pcmdi_diags.rst index 2461401c..312c587e 100644 --- a/docs/source/user_guide/tasks/pcmdi_diags.rst +++ b/docs/source/user_guide/tasks/pcmdi_diags.rst @@ -480,6 +480,8 @@ must be defined inside ``[pcmdi_diags]``: - Data frequency for variability-mode calculations. Overrides the ``[default]`` value (``"monthly"``). +For other top-level parameters, see :ref:`top-level parameters `. + Dependencies ------------ From c65ca786cda14c0fa1b8185e9f77760c27130032 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Mon, 8 Jun 2026 20:43:39 -0700 Subject: [PATCH 21/27] Address comments --- docs/source/dev_guide/contributing.rst | 6 +-- docs/source/dev_guide/test.rst | 4 +- .../dev_guide/update_expected_results.rst | 2 +- docs/source/user_guide/getting_started.rst | 9 +++-- docs/source/user_guide/parameters.rst | 4 +- docs/source/user_guide/tasks/index.rst | 2 +- docs/source/user_guide/tasks/pcmdi_diags.rst | 4 +- docs/source/user_guide/tutorial.rst | 38 +++++++++---------- 8 files changed, 35 insertions(+), 34 deletions(-) diff --git a/docs/source/dev_guide/contributing.rst b/docs/source/dev_guide/contributing.rst index 4767831a..6a55900a 100644 --- a/docs/source/dev_guide/contributing.rst +++ b/docs/source/dev_guide/contributing.rst @@ -49,12 +49,12 @@ using Sphinx, you can refer to # For more machines, go to: # https://github.com/E3SM-Project/mache/tree/main/mache/machines - # Select the cfg for the machine, and find the `[web_portal]`` section. + # Select the cfg for the machine, and find the `[web_portal]` section. # That will show the `base_path` and `base_url`. - cp -r _build/ base_url/$USER/zppy_docs + cp -r _build/ base_path/$USER/zppy_docs # Needs to be 755 to show up on https://portal.nersc.gov/cfs/e3sm - chmod -R 755 base_url/$USER/zppy_docs + chmod -R 755 base_path/$USER/zppy_docs # Optional -- if you want to generate and view versioned docs: sphinx-multiversion source _build/html diff --git a/docs/source/dev_guide/test.rst b/docs/source/dev_guide/test.rst index 5d0a4678..d02a3fb2 100644 --- a/docs/source/dev_guide/test.rst +++ b/docs/source/dev_guide/test.rst @@ -44,8 +44,8 @@ Process In your Markdown report, note the date the expected results were last updated. -Step 2: Determine what the current expected results are -======================================================= +Step 2: Review changes since expected results were updated +========================================================== Now that we know the date the expected results are from, we can review what changes we'll be testing. diff --git a/docs/source/dev_guide/update_expected_results.rst b/docs/source/dev_guide/update_expected_results.rst index a4ab6aeb..64d8f8b4 100644 --- a/docs/source/dev_guide/update_expected_results.rst +++ b/docs/source/dev_guide/update_expected_results.rst @@ -82,7 +82,7 @@ Process ./tests/integration/generated/update_campaign_expected_files_${machine_name}.sh ./tests/integration/generated/update_defaults_expected_files_${machine_name}.sh - # Theis script only updates the expected results + # This script only updates the expected results ./tests/integration/generated/update_weekly_expected_files_${machine_name}.sh # Chrysalis -- takes about 40 minutes # Perlmutter -- takes about 25 minutes diff --git a/docs/source/user_guide/getting_started.rst b/docs/source/user_guide/getting_started.rst index d3ce9086..7ad49b0b 100644 --- a/docs/source/user_guide/getting_started.rst +++ b/docs/source/user_guide/getting_started.rst @@ -64,7 +64,7 @@ E3SM Unified and zppy versions * E3SM Unified 1.13.0: ``zppy 3.2.0``, ``zppy-interfaces 0.2.1`` * E3SM Unified 1.12.0: ``zppy 3.1.0``, ``zppy-interfaces 0.2.0`` * E3SM Unified 1.11.1: ``zppy 3.1.0``, ``zppy-interfaces 0.1.2`` -* E3SM Unified 1.11.0: ``zppy 3.0.0``, ``zppy-interfaces 0.1.1`` (new auxillary package) +* E3SM Unified 1.11.0: ``zppy 3.0.0``, ``zppy-interfaces 0.1.1`` (new auxiliary package) * E3SM Unified 1.10.0: ``zppy 2.5.0`` To use ``zppy`` features/fixes not yet in a production release, you can use a development environment. @@ -85,7 +85,7 @@ Setting up a development environment # Click the green "Code" button # Choose the SSH option and paste it here: git clone git@github.com:E3SM-Project/zppy.git - cd zppy_dev + cd zppy git remote -v # You should see the main repo listed as `origin` # A couple optional steps: @@ -93,9 +93,10 @@ Setting up a development environment git remote add your-fork-name git@github.com:your-fork-name/zppy.git # Use your fork, if you have one # To use the latest code: - git checkout upstream main + git fetch upstream + git checkout -b main upstream/main # To use code from a specific branch: - git checkout upstream that-branch-name + git checkout -b that-branch-name upstream/that-branch-name # Set up the environment ############################################################## # First, make sure you have conda activated. Then: diff --git a/docs/source/user_guide/parameters.rst b/docs/source/user_guide/parameters.rst index 59e318bc..ee44b65a 100644 --- a/docs/source/user_guide/parameters.rst +++ b/docs/source/user_guide/parameters.rst @@ -6,10 +6,10 @@ Parameters Parameters make use of inheritance. Parameters set in ``[default]`` can be overridden by parameters set in a ``[section]``, which can themselves -be overridden by parameters set in a ``[[subsection]``. +be overridden by parameters set in a ``[[subsection]]``. .. warning:: - Note that some parameters will be overriden by defaults if you define them too high up in the inheritance hierarchy. + Note that some parameters will be overridden by defaults if you define them too high up in the inheritance hierarchy. See `this release's parameter defaults `_ on GitHub for a complete list of parameters and their default values. diff --git a/docs/source/user_guide/tasks/index.rst b/docs/source/user_guide/tasks/index.rst index b0b71dd3..c0ed9fef 100644 --- a/docs/source/user_guide/tasks/index.rst +++ b/docs/source/user_guide/tasks/index.rst @@ -11,7 +11,7 @@ for multiple independent runs of the same task. **Available tasks:** -Listed by the order they are evaluated in `__main__.py`: +Listed by the order they are evaluated in ``__main__.py``: .. list-table:: :header-rows: 1 diff --git a/docs/source/user_guide/tasks/pcmdi_diags.rst b/docs/source/user_guide/tasks/pcmdi_diags.rst index 312c587e..d7d9b87a 100644 --- a/docs/source/user_guide/tasks/pcmdi_diags.rst +++ b/docs/source/user_guide/tasks/pcmdi_diags.rst @@ -451,7 +451,7 @@ There are 22 synthetic_plots parameters: Parameters at the top-level ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -These 2 parameters have ``pcmdi_diags``-specific defaults, which means even if +These 3 parameters have ``pcmdi_diags``-specific defaults, which means even if they are set at the top level (``[default]``) section, these default values will be used instead. Therefore, to specify custom values, these parameters must be defined inside ``[pcmdi_diags]``: @@ -473,7 +473,7 @@ must be defined inside ``[pcmdi_diags]``: - No - ``5`` - Year increment for test model data. Overrides the - ``[default]`` value (``"5"``), which in this case is actually the same value. + ``[default]`` value (``5``), which in this case is actually the same value. * - ``frequency`` - No - ``"mo"`` diff --git a/docs/source/user_guide/tutorial.rst b/docs/source/user_guide/tutorial.rst index 22fd7036..bf8a97c1 100644 --- a/docs/source/user_guide/tutorial.rst +++ b/docs/source/user_guide/tutorial.rst @@ -87,22 +87,22 @@ Debugging failures .. code-block:: bash - grep "output =" your_zppy_config.cfg # Easy way to remember your output directory - cd your_output_dir/post/scripts - grep -v "OK" *status # See what failed - - # Say failing_task.status is showing a non-OK status, then: - ls failing_task* # See everything associated with it - - # Say its job ID was 123456, then: - emacs failing_task.o123456 # Review the output - emacs failing_task.settings # Review how each of its parameters was set/configured. - - # If an error is obvious, make a fix in the bash file and rerun: - sbatch failing_task.bash - - # If the error is not obvious, do the following: - emacs failing_task.bash - # In this file, set `debug = True`. This will provide more information. - # Note: another option is to set `debug = True` in your `cfg` and rerun `zppy -c`. - sbatch failing_task.bash + grep "output =" your_zppy_config.cfg # Easy way to remember your output directory + cd your_output_dir/post/scripts + grep -v "OK" *status # See what failed + + # Say failing_task.status is showing a non-OK status, then: + ls failing_task* # See everything associated with it + + # Say its job ID was 123456, then: + emacs failing_task.o123456 # Review the output + emacs failing_task.settings # Review how each of its parameters was set/configured. + + # If an error is obvious, make a fix in the bash file and rerun: + sbatch failing_task.bash + + # If the error is not obvious, do the following: + emacs failing_task.bash + # In this file, set `debug = True`. This will provide more information. + # Note: another option is to set `debug = True` in your `cfg` and rerun `zppy -c`. + sbatch failing_task.bash From 1021c7aa8a109e12d12d0d0ff597aedefd15bcb9 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Wed, 17 Jun 2026 16:47:56 -0500 Subject: [PATCH 22/27] Explain task design philosophy --- docs/source/dev_guide/new_task.rst | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/source/dev_guide/new_task.rst b/docs/source/dev_guide/new_task.rst index 5658f5de..901751bf 100644 --- a/docs/source/dev_guide/new_task.rst +++ b/docs/source/dev_guide/new_task.rst @@ -2,6 +2,23 @@ Adding a new task ***************** +A task should be a wrapper around an external package. + +* The ``climo`` and ``ts`` tasks use NCO. ``tc_analysis`` and ``pcmdi_diags`` also make use of NCO tools. +* The ``global_time_series`` and ``pcmdi_diags`` tasks use ``zppy-interfaces``. +* The ``tc_analysis`` uses ``TempestRemap`` and ``TempestExtremes``. +* The ``e3sm_to_cmip``, ``e3sm_diags``, ``mpas_analysis``, ``ilamb``, and ``livvkit`` use their eponymous packages. + +In general, functionality should be added via calls to external packages. There are 2 reasons for this: + +* It creates a clearer separation of concerns: ``zppy`` is the workflow manager whereas the called packages do the actual post-processing. +* The called packages often have different environment needs than ``zppy``. A package can have its own dependency list and developer environment setup instructions. Putting all the functionality in a ``zppy`` bash template makes it more difficult to customize the environment for that specific task. + +The ``zppy-interfaces`` package is meant to be a collection of tools that are too complex to just write inline in a ``zppy`` bash file but not complex enough to be worth the effort of building and maintaining a new package just for it. Currently, ``zppy-interfaces`` is home to these two tools: + +* ``global_time_series``: this functionality was initially implemented in the ``zppy`` bash template itself, but grew complex enough to require separation. +* ``pcmdi_diags``: this is "glue code" to allow for seamless integration of the PCMDI Metrics Package with ``zppy``. + The task's bash file ===================== From 91160afe4932d3c4ac4e647d77cf35fe28e00cde Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Thu, 18 Jun 2026 12:31:21 -0500 Subject: [PATCH 23/27] Address Copilot review comments --- docs/source/dev_guide/contributing.rst | 3 ++- docs/source/dev_guide/new_task.rst | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/source/dev_guide/contributing.rst b/docs/source/dev_guide/contributing.rst index 6a55900a..0fd24064 100644 --- a/docs/source/dev_guide/contributing.rst +++ b/docs/source/dev_guide/contributing.rst @@ -34,7 +34,8 @@ using Sphinx, you can refer to cd zppy conda clean --all --y conda env create -f conda/dev.yml -n env-name - python -m pip install . + conda activate env-name + python -m pip install . # Now, we can produce the documentation locally: cd docs diff --git a/docs/source/dev_guide/new_task.rst b/docs/source/dev_guide/new_task.rst index 901751bf..65d5a9cd 100644 --- a/docs/source/dev_guide/new_task.rst +++ b/docs/source/dev_guide/new_task.rst @@ -30,7 +30,7 @@ Some key parts, however, are displayed below: .. code:: #!/bin/bash - {% include 'inclusions/slurm_header.sh' %} + {% include 'inclusions/slurm_header.bash' %} {% include 'inclusions/boilerplate.bash' %} set -e From 320736df9a5d7c96616c17cb6fe6701d37f7ebed Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Fri, 26 Jun 2026 12:30:35 -0700 Subject: [PATCH 24/27] Document recently merged code --- docs/source/dev_guide/index.rst | 1 + docs/source/dev_guide/provenance.rst | 88 ++++++++++++++++++++ docs/source/dev_guide/tasks/tc_analysis.rst | 24 +++++- docs/source/user_guide/tasks/tc_analysis.rst | 61 ++++++++++++-- 4 files changed, 165 insertions(+), 9 deletions(-) create mode 100644 docs/source/dev_guide/provenance.rst diff --git a/docs/source/dev_guide/index.rst b/docs/source/dev_guide/index.rst index ff7ad430..9ff5ade8 100644 --- a/docs/source/dev_guide/index.rst +++ b/docs/source/dev_guide/index.rst @@ -10,6 +10,7 @@ This guide covers everything needed to develop, test, and release ``zppy``. project-standards contributing parameters + provenance tasks/index test update_expected_results diff --git a/docs/source/dev_guide/provenance.rst b/docs/source/dev_guide/provenance.rst new file mode 100644 index 00000000..cd10f27a --- /dev/null +++ b/docs/source/dev_guide/provenance.rst @@ -0,0 +1,88 @@ +.. _dev-provenance: + +Provenance (Developer Reference) +================================== + +Overview +-------- + +Every ``zppy`` run writes two provenance files for each invocation: + +.. list-table:: + :header-rows: 1 + :widths: 35 65 + + * - File + - Contents + * - ``provenance..cfg`` + - A verbatim copy of the user-supplied ``.cfg`` configuration file. + * - ``provenance..settings`` + - Key-value metadata about the run: ``case_name``, ``machine``, + ``hpc_username``, and ``diagnostics_url``. Only written when at least + one field resolves to a non-empty value. + +Both files are written to the script directory +(``/post/scripts/``) and mirrored under ``//`` when +that directory is accessible. + +The ``.settings`` file format +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +One ``key = value`` pair per line, e.g.: + +.. code-block:: text + + case_name = v3.LR.historical_0051 + machine = pm-cpu + hpc_username = ac.zhang40 + diagnostics_url = https://portal.nersc.gov/cfs/e3sm/ac.zhang40/zppy_demo/v3.LR.historical_0051 + +Fields with empty or ``None`` values are omitted. + +Implementation +-------------- + +- **Python module**: ``zppy/provenance.py`` +- **Called from**: ``zppy/__main__.py`` + +``zppy/provenance.py`` exports four public helpers: + +``parse_env_case_xml(input_dir)`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Reads ``/case_scripts/env_case.xml`` and returns a dict with any +of ``case_name``, ``machine``, and ``hpc_username`` that were successfully +parsed. Degrades gracefully on a missing file, missing entries, or malformed +XML (logs a warning and returns a partial or empty dict). + +``build_diagnostics_url(www, case, machine_info)`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Constructs the public web-portal URL for the run's diagnostics output using +the machine's ``web_portal`` ``base_path`` and ``base_url`` from the +``mache`` ``MachineInfo`` config. Returns ``None`` when ``www`` is empty, +when ``www`` is not under the machine's ``base_path``, or when the machine +config lacks ``web_portal`` entries. + +``write_provenance_settings(settings_path, extras)`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Writes the ``provenance.settings`` file from a dict, skipping empty and +``None`` values. Does nothing if no usable entries remain. + +``build_provenance_extras(config_default, machine_info)`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Orchestrates the other three functions. Reads ``case_name``, ``machine``, +and ``hpc_username`` from ``env_case.xml`` under ``cfg["default"]["input"]``, +builds ``diagnostics_url`` from ``cfg["default"]["www"]`` and +``cfg["default"]["case"]``, and returns the assembled extras dict. Logs a +warning if the case name in the cfg disagrees with the value in +``env_case.xml`` (the XML value is treated as authoritative). + +Unit tests +---------- + +``tests/test_zppy_provenance.py`` covers all four public helpers, including +missing-file, partial-entry, malformed-XML, unsupported-machine, and +case-mismatch scenarios. diff --git a/docs/source/dev_guide/tasks/tc_analysis.rst b/docs/source/dev_guide/tasks/tc_analysis.rst index 3d393b2c..5782439f 100644 --- a/docs/source/dev_guide/tasks/tc_analysis.rst +++ b/docs/source/dev_guide/tasks/tc_analysis.rst @@ -9,5 +9,25 @@ Implementation - **Python module**: ``zppy/tc_analysis.py`` - **Jinja2 template**: ``zppy/templates/tc_analysis.bash`` -``tc_analysis`` processes high-frequency atmospheric output (``eam.h2``) -to detect and track tropical cyclones. +``tc_analysis`` processes high-frequency atmospheric output (``eam.h2`` by +default) to detect and track tropical cyclones and African Easterly Waves +using TempestExtremes. Both EAM and EAMxx are supported. + +Template variables +------------------ + +In addition to the standard top-level variables: + +- ``{{ input_grid }}`` — native model grid string (e.g. ``ne30pg2``) +- ``{{ res }}`` — legacy explicit resolution override +- ``{{ tc_vars }}`` — comma-separated 7-variable sequence for TempestExtremes +- ``{{ default_case }}`` — default case ID used to construct output filenames + +Dependencies +------------ + +**Upstream:** None. + +**Downstream:** + +- ``e3sm_diags`` — required for the ``tc_analysis`` diagnostic set. diff --git a/docs/source/user_guide/tasks/tc_analysis.rst b/docs/source/user_guide/tasks/tc_analysis.rst index fac8ce76..9cc0fe8b 100644 --- a/docs/source/user_guide/tasks/tc_analysis.rst +++ b/docs/source/user_guide/tasks/tc_analysis.rst @@ -3,14 +3,40 @@ tc_analysis — Tropical Cyclone Analysis ======================================== -The ``tc_analysis`` task performs tropical cyclone (TC) analysis on E3SM -output. It uses high-frequency atmospheric data (typically ``eam.h2``) to -detect and track TCs. +The ``tc_analysis`` task performs tropical cyclone (TC) and African Easterly +Wave (AEW) analysis on E3SM output. It uses high-frequency atmospheric data +(typically ``eam.h2``) to detect and track TCs and AEWs using TempestExtremes. + +Both EAM and EAMxx output are supported. EAMxx requires the ``input_grid`` +parameter to be set because its output sets ``topography_file="NONE"``, +preventing automatic grid inference. + +Configuration example +--------------------- + +EAM: + +.. code-block:: cfg + + [tc_analysis] + active = True + input_grid = ne30pg2 + years = 1:100:10 + +EAMxx: + +.. code-block:: cfg + + [tc_analysis] + active = True + input_grid = ne30pg2 + tc_vars = SeaLevelPressure,T_mid_at_200hPa,T_mid_at_500hPa,U_at_model_bot,V_at_model_bot,U_at_850hPa,V_at_850hPa + years = 1:100:10 Parameters ---------- -This parameter is specific to the ``tc_analysis`` task: +These 3 parameters are specific to the ``tc_analysis`` task: .. list-table:: :header-rows: 1 @@ -20,16 +46,37 @@ This parameter is specific to the ``tc_analysis`` task: - Required - Default - Description + * - ``input_grid`` + - No + - ``""`` + - Native model grid of the input files, e.g. ``ne30pg2``, ``ne120pg2``, + ``ne30np4``. When set, the resolution and pg2 flag are derived from + this value automatically. Required for EAMxx. Accepted formats: + ``neXpg2`` or ``neXnp4`` (e.g. ``ne30pg2``, ``ne120np4``). * - ``res`` - No - ``""`` - - Resolution string for the ``--res`` option. If not set, ``zppy`` - attempts to infer it from the topography file. + - Explicit ``--res`` value for TempestExtremes (EAM only). Used when + ``input_grid`` is empty. If both are empty, ``zppy`` attempts to infer + ``res`` from the ``topography_file`` global attribute of the input + files (EAM only). + * - ``tc_vars`` + - No + - ``"PSL,T200,T500,UBOT,VBOT,U850,V850"`` + - Ordered list of exactly 7 comma-separated variable names in the + sequence required by the TempestExtremes workflow: + SLP, T@200hPa, T@500hPa, U@model_bottom, V@model_bottom, U@850hPa, + V@850hPa. The default covers EAM output. EAMxx users should set this + to the corresponding EAMxx variable names, e.g. + ``SeaLevelPressure,T_mid_at_200hPa,T_mid_at_500hPa,U_at_model_bot,V_at_model_bot,U_at_850hPa,V_at_850hPa``. Parameters at the top-level ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -This parameter has a ``tc_analysis``-specific default, which means even if it is set at the top level (``[default]``) section, the default value will be used instead. Therefore, to specify a custom value, this parameter must be defined inside ``[tc_analysis]``: +This parameter has a ``tc_analysis``-specific default, which means even if it +is set at the top level (``[default]``) section, the default value will be +used instead. Therefore, to specify a custom value, this parameter must be +defined inside ``[tc_analysis]``: .. list-table:: :header-rows: 1 From 0a4d4f33bd5936e471e15a404c53bca36010023c Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Fri, 26 Jun 2026 12:40:56 -0700 Subject: [PATCH 25/27] Address Copilot review comments --- docs/source/dev_guide/test.rst | 4 ++-- docs/source/user_guide/tasks/index.rst | 2 +- docs/source/user_guide/tasks/ts.rst | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/source/dev_guide/test.rst b/docs/source/dev_guide/test.rst index d02a3fb2..ff57f678 100644 --- a/docs/source/dev_guide/test.rst +++ b/docs/source/dev_guide/test.rst @@ -289,9 +289,9 @@ Process First, let's edit ``tests/integration/utils.py``. -In place of ech ``{env_cmd}`` put either ``${conda_setup_cmd}; conda activate ${repo_name}-yyyymmdd`` or the command from ``activate_unified_env``. You may want to use dev environments for some tasks and the Unified environment for others. +In place of each ``{env_cmd}`` put either ``${conda_setup_cmd}; conda activate ${repo_name}-yyyymmdd`` or the command from ``activate_unified_env``. You may want to use dev environments for some tasks and the Unified environment for others. -You can comment out ``cfg``s from ``cfgs_to_run`` to run fewer configuration files. Likewise, you can comment out tasks from ``tasks_torun`` to run fewer tasks. +You can comment out ``cfg``s from ``cfgs_to_run`` to run fewer configuration files. Likewise, you can comment out tasks from ``tasks_to_run`` to run fewer tasks. Be sure to set the ``unique_id``; this allows us to avoid path name collisions. diff --git a/docs/source/user_guide/tasks/index.rst b/docs/source/user_guide/tasks/index.rst index c0ed9fef..9dc8dd63 100644 --- a/docs/source/user_guide/tasks/index.rst +++ b/docs/source/user_guide/tasks/index.rst @@ -11,7 +11,7 @@ for multiple independent runs of the same task. **Available tasks:** -Listed by the order they are evaluated in ``__main__.py``: +Listed for reference (bundle jobs are submitted after task jobs are generated in ``__main__.py``): .. list-table:: :header-rows: 1 diff --git a/docs/source/user_guide/tasks/ts.rst b/docs/source/user_guide/tasks/ts.rst index 5473dafd..ab44e478 100644 --- a/docs/source/user_guide/tasks/ts.rst +++ b/docs/source/user_guide/tasks/ts.rst @@ -7,7 +7,7 @@ The ``ts`` task generates per-variable time-series files from E3SM simulation output using NCO's ``ncclimo``. It produces one file per variable covering a specified year range. Time-series files produced by ``ts`` are required inputs for the :doc:`e3sm_diags`, :doc:`global_time_series`, -:doc:`e3sm_to_cmip`, :doc:`ilamb`, and :doc:`pcmdi_diags` tasks. +:doc:`e3sm_to_cmip`, :doc:`ilamb`, :doc:`livvkit`, and :doc:`pcmdi_diags` tasks. Configuration example --------------------- From ada447c3915673d6e9750b9efd7ee82f25a843c8 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Fri, 26 Jun 2026 12:47:35 -0700 Subject: [PATCH 26/27] Remove accidental mail-to links --- docs/source/user_guide/tasks/tc_analysis.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/user_guide/tasks/tc_analysis.rst b/docs/source/user_guide/tasks/tc_analysis.rst index 9cc0fe8b..9883c3e5 100644 --- a/docs/source/user_guide/tasks/tc_analysis.rst +++ b/docs/source/user_guide/tasks/tc_analysis.rst @@ -65,8 +65,8 @@ These 3 parameters are specific to the ``tc_analysis`` task: - ``"PSL,T200,T500,UBOT,VBOT,U850,V850"`` - Ordered list of exactly 7 comma-separated variable names in the sequence required by the TempestExtremes workflow: - SLP, T@200hPa, T@500hPa, U@model_bottom, V@model_bottom, U@850hPa, - V@850hPa. The default covers EAM output. EAMxx users should set this + ``SLP, T@200hPa, T@500hPa, U@model_bottom, V@model_bottom, U@850hPa, + V@850hPa``. The default covers EAM output. EAMxx users should set this to the corresponding EAMxx variable names, e.g. ``SeaLevelPressure,T_mid_at_200hPa,T_mid_at_500hPa,U_at_model_bot,V_at_model_bot,U_at_850hPa,V_at_850hPa``. From 93cc7f405e1c5492eedc0406e34e483b71c51687 Mon Sep 17 00:00:00 2001 From: Ryan Forsyth Date: Tue, 30 Jun 2026 09:38:32 -0700 Subject: [PATCH 27/27] Fix E3SM Unified wording --- docs/source/user_guide/getting_started.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/source/user_guide/getting_started.rst b/docs/source/user_guide/getting_started.rst index 7ad49b0b..8019606f 100644 --- a/docs/source/user_guide/getting_started.rst +++ b/docs/source/user_guide/getting_started.rst @@ -69,9 +69,7 @@ E3SM Unified and zppy versions To use ``zppy`` features/fixes not yet in a production release, you can use a development environment. -Note that ``e3sm_unified``'s development cycle is not in phase with ``zppy``. -Therefore the version of ``zppy`` included may not be the latest. -To install the latest stable release, refer to the following: +Nevertheless, it is possible that the version of ``zppy`` included in ``e3sm_unified`` may not be the latest. To install the latest development environment, refer to the following section. Setting up a development environment ====================================