diff --git a/docs/source/dev_guide/index.rst b/docs/source/dev_guide/index.rst index 9ff5ade8..42b780ec 100644 --- a/docs/source/dev_guide/index.rst +++ b/docs/source/dev_guide/index.rst @@ -12,9 +12,8 @@ This guide covers everything needed to develop, test, and release ``zppy``. parameters provenance tasks/index - test - update_expected_results - releases/index.rst + tests/index + releases/index new_task new_diags_set archive/index diff --git a/docs/source/dev_guide/tests/automated_test.rst b/docs/source/dev_guide/tests/automated_test.rst new file mode 100644 index 00000000..b25c57b3 --- /dev/null +++ b/docs/source/dev_guide/tests/automated_test.rst @@ -0,0 +1,346 @@ +.. _automated-testing-zppy: + +************************* +Automated testing of zppy +************************* + +Follow the steps below to test ``zppy``. As you do so, please produce a Markdown report summarizing your results. + +Step 1: Determine what the current expected results are +======================================================= + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + expected_results_dir=/lcrc/group/e3sm/public_html/zppy_test_resources + expected_results_records_dir=/lcrc/group/e3sm/public_html/zppy_test_resources_previous + +Compy: + +.. code-block:: bash + + expected_results_dir=/compyfs/www/zppy_test_resources + expected_results_records_dir=/compyfs/fors729/zppy_test_resources_previous + +Note that Compy doesn't give write access to ``/compyfs/www/``, so we can't add a new directory there. That's why ``zppy_test_resources_previous`` is in a separate path. + +Perlmutter: + +.. code-block:: bash + + expected_results_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources + expected_results_records_dir=/global/cfs/cdirs/e3sm/www/zppy_test_resources_previous + +Process +~~~~~~~ + +.. code-block:: bash + + ls -lt ${expected_results_dir} + +In your Markdown report, note the date the expected results were last updated. + +Step 2: Review changes since expected results were updated +========================================================== + +Now that we know the date the expected results are from, we can review what changes we'll be testing. + +Review each of the following commit logs and note commits made since the date the expected results were updated: + +* For the ``e3sm_to_cmip`` task: `e3sm_to_cmip `_ +* For the ``e3sm_diags`` task: `e3sm_diags `_ +* For the ``mpas_analysis`` task: `MPAS-Analysis `_ +* For the ``global_time_series`` and ``pcmdi_diags`` tasks: `zppy-interfaces `_ +* For ``zppy`` itself: `zppy `_ + +For the remaining tasks (``climo``, ``ts``, ``tc_analysis``, ``ilamb``, ``livvkit``), we typically just use the associated package's latest release rather than making dev environments. As such, their latest development will have no impact on our tests unless we have started using one of their newer releases. + +In your Markdown report, make a table like: + +.. code-block:: + + | Package | Changes since expected results were updated | + | --- | --- | + | [package name](link to package's commit log) | Links to all PRs merged since the expected results were updated | + ... + +The automated test script +========================= + +The automated test script handles the following steps from the manual testing process: + +* Step 3: Set up environments for called packages +* Step 4: Set up zppy environment +* Step 5: Launch zppy jobs +* Step 6: Launch zppy jobs – bundles part 2 +* Step 7: Review finished returns +* Step 8: Run Python tests (excluding the final ``pytest tests/integration/test_images.py`` call from a compute node) + +A. Set up the test script +~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + cd ${repo_parent_dir}/zppy + git status # Check for uncommitted changes + + # If there are uncommitted changes, + # commit them so we can move cleanly to a new branch: + git add -A + git commit -m "Checkpoint" + + git fetch upstream main # This assumes you've named your remote for the main repo as "upstream" + git checkout -b test-zppy-yyyymmdd upstream/main # Use today's date + git log --oneline | head -n 1 + # Check that this matches the corresponding commit log: + # https://github.com/E3SM-Project/zppy/commits/main + + # Now, copy the test script and cfg from the zppy repo into the directory + # that you'll be running the test script from. + mkdir -p ${test_script_dir}/test_yyyymmdd_runN + cd ${test_script_dir}/test_yyyymmdd_runN + cp ${repo_parent_dir}/zppy/tests/main_branch_testing/run_integration_test.bash . + cp ${repo_parent_dir}/zppy/tests/main_branch_testing/zppy_test.cfg . + + # Now, edit the test cfg as needed + emacs zppy_test.cfg + +B. Set up the test cfg +~~~~~~~~~~~~~~~~~~~~~~ + +Let's examine the parts of the test cfg. + +You'll likely just need to update the ``MACHINE`` name if you're not running on Chrysalis. + +.. code-block:: + + # For these, + + MACHINE=chrysalis # chrysalis | compy | perlmutter + START_PHASE=1 # 1 | 2 | 3 + AUTO_MODE=true # true = skip all interactive checkpoints + EXPLICIT_TAG="" # Leave empty to auto-generate; set to resume a prior run + +Update the ``RUN_NUMBER`` if you've already done a test run today. + +.. code-block:: + + RUN_NUMBER=1 + +Update the ``_BASE_BRANCH`` parameters if you plan to test new features or bug fixes that aren't yet included on whatever the repo calls its "official" branch. + +.. code-block:: + + DIAGS_BASE_BRANCH="main" + E3SM_TO_CMIP_BASE_BRANCH="master" + MPAS_BASE_BRANCH="develop" + ZI_BASE_BRANCH="main" + ZPPY_BASE_BRANCH="main" + +Update the ``_ENV_TYPE`` parameters if you want to use E3SM-Unified rather than a dev environment. If you plan to only run a subset of tasks, you can set the ones you aren't running to use E3SM-Unified, so that the script doesn't spend time building a dev environment that won't be used. + +.. code-block:: + + # "dev" = build a dedicated conda env from the repo's dev.yml + # "unified" = use the machine's e3sm-unified env (UNIFIED_ENV_CMD) + DIAGS_ENV_TYPE="dev" + E3SM_TO_CMIP_ENV_TYPE="dev" + MPAS_ENV_TYPE="dev" + ZI_ENV_TYPE="dev" + +Update the ``_EXISTING_ENV`` parameters if you already have an environment from a previous test run to use. + +.. code-block:: + + # Optional: reuse an existing named conda env instead of creating a new one. + # When non-empty AND the corresponding ENV_TYPE is "dev", the script skips + # conda env creation and activates this env directly. + # Leave empty to let the script auto-name and create the env as usual. + DIAGS_EXISTING_ENV="" + E3SM_TO_CMIP_EXISTING_ENV="" + MPAS_EXISTING_ENV="" + ZI_EXISTING_ENV="" + ZPPY_EXISTING_ENV="" + +Update these two parameters to configure which jobs run. + +.. code-block:: + + # Comma-separated list of zppy cfg names to generate and submit. + # These correspond to generated filenames: test_weekly__.cfg + # Any name containing "bundle" is treated as a bundle cfg and re-submitted in Phase 2. + CFGS_TO_RUN="weekly_bundles,weekly_comprehensive_v2,weekly_comprehensive_v3,weekly_legacy_3.1.0_bundles,weekly_legacy_3.1.0_comprehensive_v2,weekly_legacy_3.1.0_comprehensive_v3,weekly_legacy_3.0.0_bundles,weekly_legacy_3.0.0_comprehensive_v2,weekly_legacy_3.0.0_comprehensive_v3" + + # Comma-separated list of tasks to enable in utils.py. + TASKS_TO_RUN="e3sm_diags,mpas_analysis,global_time_series,ilamb,livvkit,pcmdi_diags" + + +These parameters are unlikely to change between runs. They just let the test script know where to find files in your particular workspace. It is recommended to clone a new copy of the repos and use that for each ``_DIR`` parameter listed below. The script will change branches, so using a distinct copy means you won't get your work overwritten. + +.. code-block:: + + HOME_DIR="$HOME" + EZ_DIR="$HOME_DIR/ez" + + E3SM_DIAGS_DIR="$EZ_DIR/e3sm_diags" + E3SM_TO_CMIP_DIR="$EZ_DIR/e3sm_to_cmip" + MPAS_ANALYSIS_DIR="$EZ_DIR/MPAS-Analysis" + ZPPY_INTERFACES_DIR="$EZ_DIR/zppy-interfaces" + ZPPY_DIR="$EZ_DIR/zppy" + + CONDA_PROFILE="$HOME_DIR/miniforge3/etc/profile.d/conda.sh" + TAG_CACHE_FILE="$HOME_DIR/.zppy_test_tag" + + +C. Run the test script +~~~~~~~~~~~~~~~~~~~~~~ + +Now that we have the test cfg set up, we can run it. + +.. code-block:: bash + + screen # Use `screen`` so that even if the terminal connection is interrupted, the script will keep running. + ulimit -s unlimited # This is necessary for MPAS-Analysis to work inside `screen` + cd ${test_script_dir}/test_yyyymmdd_runN + cat zppy_test.cfg # Make sure changes are there + time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test_runN.log + # Ctrl-A D to detach from screen + screen -ls # See what screen sessions you have + tail -f integration_test_runN.log + +Follow the ``tail`` output until you get to: + +.. code-block:: + + ✓ Phase 3 automated tests complete! + ✓ Remember to run test_images.py manually from a compute node. + ✓ Integration test automation complete! + +D. Review the output +~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: bash + + # CTRL C # Exit tail + screen -R # The script should have finished and there should be ``time`` output: real, user, sys + exit # Exit screen + cd ${test_script_dir}/test_yyyymmdd_runN + cat integration_test_runN.log + +Let's review the test script's output log. + +First, the unit tests. There are two blocks, starting with: + +.. code-block:: + + Running zppy-interfaces unit tests... + +and + +.. code-block:: + + Running zppy unit tests... + +Second, the output directories status. It should look like the following: + +.. code-block:: + + Checking all status files... + ... + ✓ All status files clean! + +If some status files were unsuccessful, you'll want to run the following to review the errors: + +.. code-block:: bash + + cd ${dir_with_failures} + grep -v "OK" * status # See what jobs failed + # Review errors: + tail ${job_that_failed}.o${id_of_job_that_failed} + grep -i error ${job_that_failed}.o${id_of_job_that_failed} + +Third, the integration tests. + +.. code-block:: + + test_last_year.py + test_bash_generation.py + test_campaign.py + test_defaults.py + test_bundles.py + +Errors here may actually be expected if the expected results haven't been updated yet to reflect a recently merged pull request. Another reason for errors on ``test_bundles.py`` in particular is if you didn't run all the jobs necessary (i.e., if you're running a partial test). + +If all 3 pieces look good, you can proceed with the final integration test, the image checker. + +Step 8: Run Python tests +======================== + +Machine-specific setup +~~~~~~~~~~~~~~~~~~~~~~ + +Chrysalis: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm + } + +Compy: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm + } + +Perlmutter: + +.. code-block:: bash + + launch_compute_node() + { + salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm + } + +Process +~~~~~~~ + +.. code-block:: bash + + cd ${repo_parent_dir}/zppy + git status + # You might have changed branches while you were waiting for jobs to finish. + # Make sure you're now back on the correct branch: test-zppy-yyyymmdd + # Also confirm you're back in the correct env: zppy-yyyymmdd or the Unified env + + # The image checker test, which we'll run from a compute node: + launch_compute_node + + start_bash_subshell + # EITHER: + # Activate EITHER a dev environment or the Unified env: + conda activate zppy-yyyymmdd + # OR: the command from `activate_unified_env` + + pytest tests/integration/test_images.py + # Typically takes between 10 and 20 minutes on Chrysalis and Perlmutter. + # Typically takes closer to 50 minutes on Compy. + cat test_images_summary.md + exit # Exit bash shell + exit # Exit compute note + +In your Markdown report: + +* From the ``pytest tests/integration/test_images.py `` command-line output, copy everything after ``Captured stdout call`` to a code block labeled "Output" +* Copy the results of ``cat test_images_summary.md`` to a section labeled "Complete summary table" +* Make a new section named "Summary table -- only failing image-check tests, sorted by task". For each task that has missing and/or mismatched images, copy the relevant rows from the summary table. Skip this section if there were no failing image-check tests. +* Note any test failures from the other Python tests. +* If there were no failures at all, print "All tests pass" diff --git a/docs/source/dev_guide/tests/index.rst b/docs/source/dev_guide/tests/index.rst new file mode 100644 index 00000000..f0119ea8 --- /dev/null +++ b/docs/source/dev_guide/tests/index.rst @@ -0,0 +1,14 @@ +.. _tests: + +******* +Tests +******* + +This page collects documentation on testing ``zppy``. + +.. toctree:: + :maxdepth: 1 + + manual_test + automated_test + update_expected_results diff --git a/docs/source/dev_guide/test.rst b/docs/source/dev_guide/tests/manual_test.rst similarity index 98% rename from docs/source/dev_guide/test.rst rename to docs/source/dev_guide/tests/manual_test.rst index ff57f678..c4d3ba02 100644 --- a/docs/source/dev_guide/test.rst +++ b/docs/source/dev_guide/tests/manual_test.rst @@ -1,8 +1,8 @@ -.. _testing-zppy: +.. _manually-testing-zppy: -************* -Testing zppy -************* +********************* +Manually testing zppy +********************* Follow the steps below to test ``zppy``. As you do so, please produce a Markdown report summarizing your results. @@ -93,6 +93,11 @@ Chrysalis: lcrc_conda # Or however you activate conda rm -rf build conda clean --all --y + # The dev.yml file may be in a different directory. + # e3sm_to_cmip, e3sm_diags: use conda-dev/ + # zppy-interfaces, zppy: use conda/ + # MPAS-Analysis: don't use this command, instead use: + # conda create --name ${env_name} --file dev-spec.txt --yes conda env create -f conda/dev.yml -n ${env_name} conda activate ${env_name} pre-commit run --all-files # Confirm this passes diff --git a/docs/source/dev_guide/update_expected_results.rst b/docs/source/dev_guide/tests/update_expected_results.rst similarity index 100% rename from docs/source/dev_guide/update_expected_results.rst rename to docs/source/dev_guide/tests/update_expected_results.rst diff --git a/tests/integration/template_weekly_bundles.cfg b/tests/integration/template_weekly_bundles.cfg index ae94c1d8..9f164ebb 100644 --- a/tests/integration/template_weekly_bundles.cfg +++ b/tests/integration/template_weekly_bundles.cfg @@ -51,10 +51,11 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" @@ -66,12 +67,13 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# bundle = "bundle2" # Override bundle1 frequency = "monthly" input_files = "eam.h0" @@ -80,6 +82,7 @@ years = "1985:1989:2", years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -88,7 +91,7 @@ years = "1985:1989:2", vars = "LAISHA,LAISUN" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# bundle = "bundle3" # Override bundle1, let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" extra_vars = 'areatotal2' frequency = "monthly" @@ -98,7 +101,7 @@ years = "1985:1989:2", vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True bundle = "bundle1" environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" @@ -106,14 +109,16 @@ ts_num_years = 2 years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # bundle = "bundle3" # Let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" # years = "1985:1989:2", diff --git a/tests/integration/template_weekly_comprehensive_v2.cfg b/tests/integration/template_weekly_comprehensive_v2.cfg index 7ba7aef8..9f3cec9b 100644 --- a/tests/integration/template_weekly_comprehensive_v2.cfg +++ b/tests/integration/template_weekly_comprehensive_v2.cfg @@ -19,20 +19,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_month_lnd# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -43,19 +44,20 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -65,7 +67,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -73,7 +75,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -82,6 +84,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -89,20 +92,22 @@ walltime = "00:30:00" vars = "LAISHA,LAISUN" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" [tc_analysis] -active = #expand active_e3sm_diags# +active = #expand active_tc_analysis# walltime = "00:30:00" [e3sm_diags] diff --git a/tests/integration/template_weekly_comprehensive_v3.cfg b/tests/integration/template_weekly_comprehensive_v3.cfg index 5c3d1f0b..75535312 100644 --- a/tests/integration/template_weekly_comprehensive_v3.cfg +++ b/tests/integration/template_weekly_comprehensive_v3.cfg @@ -22,26 +22,27 @@ frequency = "monthly" walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = True + active = #expand active_climo_month_lnd# input_files = "elm.h0" input_subdir = "archive/lnd/hist" mapping_file = "map_r05_to_cmip6_180x360_aave.20231110.nc" vars = "" [[ land_monthly_180x360_traave ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# input_files = "elm.h0" input_subdir = "archive/lnd/hist" mapping_file = "#expand livvkit_mapping_file_path#/map_r05_to_cmip6_180x360_traave.20231110.nc" @@ -49,7 +50,7 @@ walltime = "00:30:00" years = "1985:1994:10", [[ land_monthly_climo_native ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# climo_jobs = 12 input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -58,7 +59,7 @@ walltime = "00:30:00" years = "1985:1994:10" [[ land_monthly_climo_racmo_gis ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# climo_jobs = 12 climo_subsection = "racmo_gis" grid = "racmo_gis" @@ -69,7 +70,7 @@ walltime = "00:30:00" years = "1985:1994:10" [[ land_monthly_climo_racmo_ais ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# climo_jobs = 12 input_files = "elm.h0" climo_subsection = "racmo_ais" @@ -80,7 +81,7 @@ walltime = "00:30:00" years = "1985:1994:10" [[ land_monthly_climo_merra2 ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# climo_jobs = 12 input_files = "elm.h0" climo_subsection = "merra2" @@ -91,7 +92,7 @@ walltime = "00:30:00" years = "1985:1994:10" [[ land_monthly_climo_era5 ]] - active = #expand active_livvkit# + active = #expand active_climo_month_lnd_for_livvkit# climo_jobs = 12 input_files = "elm.h0" climo_subsection = "era5" @@ -106,6 +107,7 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -115,14 +117,14 @@ walltime = "00:30:00" vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U,PSL,LANDFRAC,CLD_CAL_TMPICE,CLD_CAL_TMPLIQ,CLDLIQ,T" [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -132,7 +134,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -140,7 +142,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -149,6 +151,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -157,7 +160,7 @@ walltime = "00:30:00" vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR" [[ land_monthly_energy ]] - active = #expand active_livvkit# + active = #expand active_ts_month_lnd_for_livvkit# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -166,8 +169,7 @@ walltime = "00:30:00" years = "1985:1994:10" [[ land_monthly_smb ]] - # The data necessary for this is not found in the data input path. - active = False + active = False # The data necessary for this is not found in the data input path. frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -176,13 +178,14 @@ walltime = "00:30:00" years = "1985:1994:10" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# cmip_plevdata = "#expand diagnostics_base_path#/e3sm_to_cmip_data/maps/vrt_remap_plev19.nc" cmip_vars = "ua, va, ta, wap, zg, hur, tas, ts, psl, ps, sfcWind, huss, pr, prc, prsn, evspsbl, tauu, tauv, hfls, clt, rlus, rsds, rsus, hfss, clivi, clwvi, rlut, rsdt, rsuscs, rsut, rtmt, abs550aer, od550aer, rsdscs, tasmax, tasmin" input_files = "eam.h0" @@ -191,12 +194,13 @@ walltime = "00:30:00" years = "1985:1995:2", # Need 10 years for pcmdi_diags task [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" ts_subsection = "land_monthly" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # walltime = "00:30:00" [e3sm_diags] diff --git a/tests/integration/template_weekly_legacy_3.0.0_bundles.cfg b/tests/integration/template_weekly_legacy_3.0.0_bundles.cfg index fac3483b..800a3c8b 100644 --- a/tests/integration/template_weekly_legacy_3.0.0_bundles.cfg +++ b/tests/integration/template_weekly_legacy_3.0.0_bundles.cfg @@ -53,10 +53,11 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" @@ -68,12 +69,13 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# bundle = "bundle2" # Override bundle1 frequency = "monthly" input_files = "eam.h0" @@ -82,6 +84,7 @@ years = "1985:1989:2", years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -90,7 +93,7 @@ years = "1985:1989:2", vars = "LAISHA,LAISUN" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# bundle = "bundle3" # Override bundle1, let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" extra_vars = 'areatotal2' frequency = "monthly" @@ -100,7 +103,7 @@ years = "1985:1989:2", vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True bundle = "bundle1" environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" @@ -108,14 +111,16 @@ ts_num_years = 2 years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # bundle = "bundle3" # Let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" # years = "1985:1989:2", diff --git a/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v2.cfg b/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v2.cfg index 2fd687c8..559e574f 100644 --- a/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v2.cfg +++ b/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v2.cfg @@ -21,20 +21,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_month_lnd# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -45,19 +46,20 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -67,7 +69,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -75,7 +77,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -84,6 +86,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -91,20 +94,22 @@ walltime = "00:30:00" vars = "LAISHA,LAISUN" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" [tc_analysis] -active = #expand active_e3sm_diags# +active = #expand active_tc_analysis# walltime = "00:30:00" [e3sm_diags] diff --git a/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v3.cfg b/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v3.cfg index 8b57e400..b87057c0 100644 --- a/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v3.cfg +++ b/tests/integration/template_weekly_legacy_3.0.0_comprehensive_v3.cfg @@ -28,20 +28,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_month_lnd# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -53,19 +54,20 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -75,7 +77,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -83,7 +85,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -92,6 +94,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -100,23 +103,25 @@ walltime = "00:30:00" vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" ts_subsection = "atm_monthly_180x360_aave" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" ts_subsection = "land_monthly" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # walltime = "00:30:00" [e3sm_diags] diff --git a/tests/integration/template_weekly_legacy_3.1.0_bundles.cfg b/tests/integration/template_weekly_legacy_3.1.0_bundles.cfg index 368303dc..60cdf6cb 100644 --- a/tests/integration/template_weekly_legacy_3.1.0_bundles.cfg +++ b/tests/integration/template_weekly_legacy_3.1.0_bundles.cfg @@ -53,10 +53,11 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" @@ -68,12 +69,13 @@ bundle = "bundle1" years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# bundle = "bundle2" # Override bundle1 frequency = "monthly" input_files = "eam.h0" @@ -82,6 +84,7 @@ years = "1985:1989:2", years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -90,7 +93,7 @@ years = "1985:1989:2", vars = "LAISHA,LAISUN" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# bundle = "bundle3" # Override bundle1, let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" extra_vars = 'areatotal2' frequency = "monthly" @@ -100,7 +103,7 @@ years = "1985:1989:2", vars = "RIVER_DISCHARGE_OVER_LAND_LIQ" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True bundle = "bundle1" environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" @@ -108,14 +111,16 @@ ts_num_years = 2 years = "1985:1989:2", [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # bundle = "bundle3" # Let bundle1 finish first because "e3sm_diags: atm_monthly_180x360_aave_mvm" requires "ts: atm_monthly_180x360_aave" # years = "1985:1989:2", diff --git a/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v2.cfg b/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v2.cfg index 6d570441..ea171c41 100644 --- a/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v2.cfg +++ b/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v2.cfg @@ -21,20 +21,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_month_lnd# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -45,19 +46,20 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -67,7 +69,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -75,7 +77,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -84,6 +86,7 @@ walltime = "00:30:00" years = "1980:1990:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -91,20 +94,22 @@ walltime = "00:30:00" vars = "LAISHA,LAISUN" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# input_files = "eam.h0" [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" [tc_analysis] -active = #expand active_e3sm_diags# +active = #expand active_tc_analysis# walltime = "00:30:00" [e3sm_diags] @@ -174,6 +179,7 @@ active = #expand active_mpas_analysis# anomalyRefYear = 1980 climo_years ="1980-1984", "1985-1990", enso_years = "1980-1984", "1985-1990", +environment_commands = "#expand mpas_analysis_environment_commands#" mesh = "EC30to60E2r2" parallelTaskCount = 6 partition = "#expand partition_long#" diff --git a/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v3.cfg b/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v3.cfg index ad39a309..2c1e65de 100644 --- a/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v3.cfg +++ b/tests/integration/template_weekly_legacy_3.1.0_comprehensive_v3.cfg @@ -23,20 +23,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_climo_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" vars = "" [[ atm_monthly_diurnal_8xdaily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_diurnal_atm# frequency = "diurnal_8xdaily" input_files = "eam.h3" input_subdir = "archive/atm/hist" vars = "PRECT" [[ land_monthly_climo ]] - active = #expand active_e3sm_diags# + active = #expand active_climo_month_lnd# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -48,20 +49,21 @@ active = True walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_ts_month_atm# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" years = "1985:1995:2", # Need 10 years for pcmdi_diags task [[ atm_daily_180x360_aave ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_daily_atm# frequency = "daily" input_files = "eam.h1" input_subdir = "archive/atm/hist" vars = "PRECT" [[ rof_monthly ]] - active = #expand active_e3sm_diags# + active = #expand active_ts_month_rof# extra_vars = 'areatotal2' frequency = "monthly" input_files = "mosart.h0" @@ -71,7 +73,7 @@ walltime = "00:30:00" [[ atm_monthly_glb ]] # Note global average won't work for 3D variables. - active = #expand active_global_time_series# + active = #expand active_ts_month_atm_glb# frequency = "monthly" input_files = "eam.h0" input_subdir = "archive/atm/hist" @@ -79,7 +81,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ lnd_monthly_glb ]] - active = #expand active_global_time_series# + active = #expand active_ts_month_lnd_glb# frequency = "monthly" input_files = "elm.h0" input_subdir = "archive/lnd/hist" @@ -88,6 +90,7 @@ walltime = "00:30:00" years = "1985:1995:5", [[ land_monthly ]] + active = #expand active_ts_month_lnd# extra_vars = "landfrac" frequency = "monthly" input_files = "elm.h0" @@ -96,13 +99,14 @@ walltime = "00:30:00" vars = "FSH,RH2M,LAISHA,LAISUN,QINTR,QOVER,QRUNOFF,QSOIL,QVEGE,QVEGT,SOILICE,SOILLIQ,SOILWATER_10CM,TSA,TSOI,H2OSNO,TOTLITC,CWDC,SOIL1C,SOIL2C,SOIL3C,SOIL4C,WOOD_HARVESTC,TOTVEGC,NBP,GPP,AR,HR" [e3sm_to_cmip] -active = #expand active_e3sm_to_cmip# +active = True environment_commands = "#expand e3sm_to_cmip_environment_commands#" frequency = "monthly" ts_num_years=2 walltime = "00:30:00" [[ atm_monthly_180x360_aave ]] + active = #expand active_e3sm_to_cmip_month_atm# cmip_plevdata = "#expand diagnostics_base_path#/e3sm_to_cmip_data/maps/vrt_remap_plev19.nc" cmip_vars = "ua, va, ta, wap, zg, hur, tas, ts, psl, ps, sfcWind, huss, pr, prc, prsn, evspsbl, tauu, tauv, hfls, clt, rlus, rsds, rsus, hfss, clivi, clwvi, rlut, rsdt, rsuscs, rsut, rtmt, abs550aer, od550aer, rsdscs, tasmax, tasmin" input_files = "eam.h0" @@ -111,12 +115,13 @@ walltime = "00:30:00" years = "1985:1995:2", # Need 10 years for pcmdi_diags task [[ land_monthly ]] + active = #expand active_e3sm_to_cmip_month_lnd# input_files = "elm.h0" ts_subsection = "land_monthly" # TODO: Add "tc_analysis" back in after empty dat is resolved. # [tc_analysis] -# active = True +# active = #expand active_tc_analysis# # walltime = "00:30:00" [e3sm_diags] @@ -207,6 +212,7 @@ active = #expand active_mpas_analysis# anomalyRefYear = 1985 climo_years = "1985-1989", "1990-1995", enso_years = "1985-1989", "1990-1995", +environment_commands = "#expand mpas_analysis_environment_commands#" mesh = "IcoswISC30E3r5" parallelTaskCount = 6 partition = "#expand partition_long#" diff --git a/tests/integration/utils.py b/tests/integration/utils.py index 17519e47..b82c9dec 100644 --- a/tests/integration/utils.py +++ b/tests/integration/utils.py @@ -180,29 +180,81 @@ def get_expansions(): expansions["environment_commands"] = TEST_SPECIFICS["environment_commands"] # Activate requested tests - expansions["active_e3sm_to_cmip"] = "False" + + # Dependencies + expansions["active_climo_month_atm"] = "False" # For e3sm_diags + expansions["active_climo_month_lnd"] = "False" # For e3sm_diags + expansions["active_climo_month_lnd_for_livvkit"] = "False" # For livvkit + expansions["active_climo_diurnal_atm"] = "False" # For e3sm_diags + + expansions["active_ts_daily_atm"] = "False" # For e3sm_diags + expansions["active_ts_month_atm"] = "False" # For e3sm_diags, ilamb, pcmdi_diags + expansions["active_ts_month_atm_glb"] = "False" # For global_time_series + expansions["active_ts_month_lnd"] = "False" # For ilamb + expansions["active_ts_month_lnd_for_livvkit"] = "False" # For livvkit + expansions["active_ts_month_lnd_glb"] = "False" # For global_time_series + expansions["active_ts_month_rof"] = "False" # For e3sm_diags + + expansions["active_e3sm_to_cmip_month_atm"] = "False" # For ilamb, pcmdi_diags + expansions["active_e3sm_to_cmip_month_lnd"] = "False" # For ilamb + + expansions["active_tc_analysis"] = "False" # For e3sm_diags + + # Plotting packages expansions["active_e3sm_diags"] = "False" - expansions["active_mpas_analysis"] = "False" + expansions["active_mpas_analysis"] = "False" # Also used for global_time_series expansions["active_global_time_series"] = "False" expansions["active_ilamb"] = "False" expansions["active_livvkit"] = "False" expansions["active_pcmdi_diags"] = "False" + if "e3sm_diags" in TEST_SPECIFICS["tasks_to_run"]: expansions["active_e3sm_diags"] = "True" + + expansions["active_climo_month_atm"] = "True" + expansions["active_climo_month_lnd"] = "True" + expansions["active_climo_diurnal_atm"] = "True" + + expansions["active_ts_month_atm"] = "True" + expansions["active_ts_month_rof"] = "True" + expansions["active_ts_daily_atm"] = "True" + + expansions["active_tc_analysis"] = "True" + if "mpas_analysis" in TEST_SPECIFICS["tasks_to_run"]: expansions["active_mpas_analysis"] = "True" + if "global_time_series" in TEST_SPECIFICS["tasks_to_run"]: expansions["active_global_time_series"] = "True" - expansions["active_mpas_analysis"] = "True" # For ocn plots - expansions["active_e3sm_to_cmip"] = "True" # For lnd plots - if "livvkit" in TEST_SPECIFICS["tasks_to_run"]: - expansions["active_livvkit"] = "True" + + expansions["active_ts_month_atm_glb"] = "True" + expansions["active_ts_month_lnd_glb"] = "True" + + expansions["active_mpas_analysis"] = "True" + if "ilamb" in TEST_SPECIFICS["tasks_to_run"]: expansions["active_ilamb"] = "True" - expansions["active_e3sm_to_cmip"] = "True" + + expansions["active_ts_month_atm"] = "True" + expansions["active_ts_month_lnd"] = "True" + + expansions["active_e3sm_to_cmip_month_atm"] = "True" + expansions["active_e3sm_to_cmip_month_lnd"] = "True" + + if "livvkit" in TEST_SPECIFICS["tasks_to_run"]: + expansions["active_livvkit"] = "True" + + expansions["active_climo_month_lnd_for_livvkit"] = "True" + + expansions["active_ts_month_lnd_for_livvkit"] = "True" + if "pcmdi_diags" in TEST_SPECIFICS["tasks_to_run"]: expansions["active_pcmdi_diags"] = "True" - expansions["active_e3sm_to_cmip"] = "True" + + expansions["active_ts_month_atm"] = "True" + + expansions["active_e3sm_to_cmip_month_atm"] = "True" + expansions["cfgs_to_run"] = TEST_SPECIFICS["cfgs_to_run"] expansions["tasks_to_run"] = TEST_SPECIFICS["tasks_to_run"] diff --git a/tests/main_branch_testing/README.md b/tests/main_branch_testing/README.md new file mode 100644 index 00000000..3331cbf2 --- /dev/null +++ b/tests/main_branch_testing/README.md @@ -0,0 +1,230 @@ +# zppy Integration Test Automation + +This automation system streamlines the zppy integration testing workflow, reducing manual steps and wait time while maintaining quality control checkpoints. + +## Quick Start + +### Basic Usage (Interactive Mode) +```bash +./run_integration_test.bash --config zppy_test.cfg +``` + +### Skip Straight to a Later Phase +Set `START_PHASE=2` or `START_PHASE=3` in your config file, then re-run: +```bash +./run_integration_test.bash --config zppy_test.cfg +``` + +### Non-Interactive (Auto) Mode +Set `AUTO_MODE=true` in your config file. All checkpoints are bypassed and the script runs end-to-end without waiting for user input. + +## Configuration + +Copy `zppy_test.cfg` and edit it before each test run. It has three sections: + +### Runtime settings (update as needed each run) + +| Variable | Description | +| --- | --- | +| `MACHINE` | `chrysalis`, `compy`, or `perlmutter` | +| `START_PHASE` | `1`, `2`, or `3` | +| `AUTO_MODE` | `true` to skip all interactive checkpoints | +| `EXPLICIT_TAG` | Leave empty to auto-generate; set to a prior TAG to resume | +| `RUN_NUMBER` | Increment if you run multiple tests on the same day | +| `DIAGS_BASE_BRANCH` | Branch to test for e3sm_diags (usually `main`) | +| `E3SM_TO_CMIP_BASE_BRANCH` | Branch to test for e3sm_to_cmip (usually `master`) | +| `MPAS_BASE_BRANCH` | Branch to test for MPAS-Analysis (usually `develop`) | +| `ZI_BASE_BRANCH` | Branch to test for zppy-interfaces (usually `main`) | +| `ZPPY_BASE_BRANCH` | Branch to test for zppy (usually `main`) | +| `DIAGS_ENV_TYPE` | `"dev"` to build a dedicated conda env; `"unified"` to use e3sm-unified | +| `E3SM_TO_CMIP_ENV_TYPE` | `"dev"` to build a dedicated conda env; `"unified"` to use e3sm-unified | +| `MPAS_ENV_TYPE` | `"dev"` to build a dedicated conda env; `"unified"` to use e3sm-unified | +| `ZI_ENV_TYPE` | `"dev"` to build a dedicated conda env; `"unified"` to use e3sm-unified | +| `DIAGS_EXISTING_ENV` | *(optional)* Name of an existing conda env to reuse for e3sm_diags; skips creation | +| `E3SM_TO_CMIP_EXISTING_ENV` | *(optional)* Name of an existing conda env to reuse for e3sm_to_cmip; skips creation | +| `MPAS_EXISTING_ENV` | *(optional)* Name of an existing conda env to reuse for MPAS-Analysis; skips creation | +| `ZI_EXISTING_ENV` | *(optional)* Name of an existing conda env to reuse for zppy-interfaces; skips creation | +| `ZPPY_EXISTING_ENV` | *(optional)* Name of an existing conda env to reuse for zppy; skips creation | +| `CFGS_TO_RUN` | Comma-separated list of zppy cfg names to generate and submit (see below) | +| `TASKS_TO_RUN` | Comma-separated list of tasks to enable (e.g. `e3sm_diags,mpas_analysis`) | + +The `*_EXISTING_ENV` variables only take effect when the corresponding `*_ENV_TYPE` is `"dev"`. When set, the script skips conda env creation entirely and activates the named env directly (still running `pip install .` to pick up any local changes). Leave them empty to let the script auto-name and create environments as usual. + +Example — reuse envs from a prior run on the same day: +``` +DIAGS_EXISTING_ENV="test-diags-main-20250601_run1" +E3SM_TO_CMIP_EXISTING_ENV="test-e3sm-to-cmip-master-20250601_run1" +MPAS_EXISTING_ENV="test-mpas-develop-20250601_run1" +ZI_EXISTING_ENV="test-zi-main-20250601_run1" +ZPPY_EXISTING_ENV="test-zppy-main-20250601_run1" +``` + +`CFGS_TO_RUN` values correspond to generated filenames `test__.cfg`. Any name containing `bundle` is automatically treated as a bundle cfg and re-submitted in Phase 2. Example to run only the v3 comprehensive and bundle cfgs: +``` +CFGS_TO_RUN="weekly_bundles,weekly_comprehensive_v3,weekly_legacy_3.1.0_bundles,weekly_legacy_3.1.0_comprehensive_v3,weekly_legacy_3.0.0_bundles,weekly_legacy_3.0.0_comprehensive_v3" +``` + +### One-time setup (paths that rarely change) + +| Variable | Description | +| --- | --- | +| `HOME_DIR` | Your home directory (default: `$HOME`) | +| `EZ_DIR` | Parent directory for all repos (default: `$HOME/ez`) | +| `E3SM_DIAGS_DIR` | Path to e3sm_diags repo | +| `E3SM_TO_CMIP_DIR` | Path to e3sm_to_cmip repo | +| `MPAS_ANALYSIS_DIR` | Path to MPAS-Analysis repo | +| `ZPPY_INTERFACES_DIR` | Path to zppy-interfaces repo | +| `ZPPY_DIR` | Path to zppy repo | +| `CONDA_PROFILE` | Path to your conda profile script | +| `TAG_CACHE_FILE` | Where the TAG is saved between phases (default: `~/.zppy_test_tag`) | + +Machine-specific settings (`OUTPUT_WORKSPACE`, conda activation command, unified environment path, `salloc` command) are derived automatically from `MACHINE` and do not appear in the config. + +## Workflow Phases + +### Phase 1: Setup +- Creates conda environments for each component where `ENV_TYPE="dev"` and no `*_EXISTING_ENV` is set; reuses the named env when `*_EXISTING_ENV` is set; skips env handling entirely for any component where `ENV_TYPE="unified"` +- Runs unit tests for zppy-interfaces and zppy +- Patches `tests/integration/utils.py` with test-specific environment commands, config list, and unique ID +- Generates config files via `python tests/integration/utils.py` +- Submits all 9 initial SLURM jobs (3 current + 3 legacy 3.1.0 + 3 legacy 3.0.0) +- Waits for jobs to complete (polls every 10 min, 4-hour max) + +### Phase 2: Bundles Part 2 +- Checks status files for all three bundles output directories; warns if any are non-OK before proceeding +- Submits bundles part 2 jobs (`weekly_bundles`, `legacy_3.1.0_bundles`, `legacy_3.0.0_bundles`) +- Waits for completion (polls every 10 min, 1-hour max) + +### Phase 3: Validation +- Checks status files for all 9 output directories +- Runs pytest integration tests: + - `test_last_year.py` + - `test_bash_generation.py` + - `test_campaign.py` + - `test_defaults.py` + - `test_bundles.py` +- Prints instructions for running `test_images.py` manually on a compute node + +## Output and Logs + +The script provides color-coded output: +- 🔵 **Blue**: Informational messages +- 🟢 **Green**: Success messages +- 🟡 **Yellow**: Warnings and checkpoints +- 🔴 **Red**: Errors + +### SLURM Job Monitoring +The script automatically monitors SLURM jobs and shows: +``` +Jobs remaining: 42 (elapsed: 1234s / max: 14400s) +``` +If all remaining jobs enter `DependencyNeverSatisfied`, they are cancelled automatically and the script exits with an error. + +### Status File Checking +Automatically checks for non-OK entries in all status directories: +``` +✓ v2: All status files OK +✓ Legacy 3.1.0 v2: All status files OK +✓ Legacy 3.0.0 v2: All status files OK +... +``` + +## Customization + +### Adjust Timeouts + +```bash +# In phase_1_setup(): +wait_for_slurm_jobs 600 14400 # Check every 10 min, max 4 hours + +# In phase_2_bundles_part2(): +wait_for_slurm_jobs 600 3600 # Check every 10 min, max 1 hour +``` + +## Troubleshooting + +### Script Exits Early +Check the error message. The script uses `set -e`, so it exits on any error. Common causes: +- A unit test failure during Phase 1 setup +- A SLURM timeout (increase the max-wait argument to `wait_for_slurm_jobs`) +- `DependencyNeverSatisfied` on all queued jobs (check your cfg files and SLURM account) + +Phase 2 checks bundle status files before resubmitting and warns if any are non-OK. Resolve any failures in the Phase 1 bundle runs before proceeding. You can restart from Phase 2 by setting `START_PHASE=2` in your config (the TAG from Phase 1 is saved in `~/.zppy_test_tag` and picked up automatically, or set `EXPLICIT_TAG` to be explicit): +```bash +./run_integration_test.bash --config zppy_test.cfg +``` + +### Environment Issues +```bash +# Remove and rebuild stale environments (only applies to components with ENV_TYPE="dev" +# and no *_EXISTING_ENV set) +conda remove --yes --all --name test-e3sm-to-cmip-master-YYYYMMDD_runN +conda remove --yes --all --name test-diags-main-YYYYMMDD_runN +conda remove --yes --all --name test-mpas-develop-YYYYMMDD_runN +conda remove --yes --all --name test-zi-main-YYYYMMDD_runN +conda remove --yes --all --name test-zppy-main-YYYYMMDD_runN + +# Then re-run from Phase 1 +./run_integration_test.bash --config zppy_test.cfg +``` + +## Files Created + +``` +~/ez/zppy/ +├── tests/integration/generated/ +│ ├── test_weekly_bundles_chrysalis.cfg +│ ├── test_weekly_comprehensive_v2_chrysalis.cfg +│ ├── test_weekly_comprehensive_v3_chrysalis.cfg +│ ├── test_weekly_legacy_3.1.0_bundles_chrysalis.cfg +│ ├── test_weekly_legacy_3.1.0_comprehensive_v2_chrysalis.cfg +│ ├── test_weekly_legacy_3.1.0_comprehensive_v3_chrysalis.cfg +│ ├── test_weekly_legacy_3.0.0_bundles_chrysalis.cfg +│ ├── test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg +│ └── test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg +└── test_images_summary.md + +/lcrc/group/e3sm// +├── zppy_weekly_bundles_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_comprehensive_v2_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_comprehensive_v3_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_legacy_3.1.0_bundles_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_legacy_3.1.0_comprehensive_v2_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_legacy_3.1.0_comprehensive_v3_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_legacy_3.0.0_bundles_output/zppy_main_branch_test_YYYYMMDD_runN/ +├── zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_main_branch_test_YYYYMMDD_runN/ +└── zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_main_branch_test_YYYYMMDD_runN/ +``` + +## Example Run + +```bash +# Confirm repos have no uncommitted changes +cd ~/ez/e3sm_to_cmip && git status +cd ~/ez/e3sm_diags && git status +cd ~/ez/MPAS-Analysis && git status +cd ~/ez/zppy-interfaces && git status +cd ~/ez/zppy && git status + +# Confirm no jobs are currently running +squeue -u $USER + +# Copy the script and config out of the repo (Phase 1 will change branches) +mkdir -p ~/ez/zppy_main_branch_tests/test_YYYYMMDD +cd ~/ez/zppy_main_branch_tests/test_YYYYMMDD +cp ~/ez/zppy/tests/main_branch_testing/run_integration_test.bash . +cp ~/ez/zppy/tests/main_branch_testing/zppy_test.cfg . + +# Edit configuration parameters +emacs zppy_test.cfg + +# Run inside a screen session so it will survive disconnections. +screen +cd ~/ez/zppy_main_branch_tests/test_YYYYMMDD +time ./run_integration_test.bash --config zppy_test.cfg 2>&1 | tee integration_test.log +# Ctrl-A D to detach from screen + +# Monitor progress from another terminal +screen -ls # Find the screen session +tail -f integration_test.log # Follow log output +``` diff --git a/tests/main_branch_testing/run_integration_test.bash b/tests/main_branch_testing/run_integration_test.bash new file mode 100755 index 00000000..a6609959 --- /dev/null +++ b/tests/main_branch_testing/run_integration_test.bash @@ -0,0 +1,844 @@ +#!/bin/bash +# zppy Integration Test Automation Script +# +# Usage: +# 1. Copy this file AND the sample config OUT of the zppy repo +# (this script will change branches). +# 2. Edit your config file (see zppy_test.cfg). +# 3. Run: ./run_integration_test.bash --config path/to/your.cfg +# +# Phases (set START_PHASE in your config): +# 1 - Full setup: build envs, run unit tests, generate configs, submit SLURM jobs +# 2 - Bundles Part 2 (run after Phase 1 jobs finish) +# 3 - Validation: status checks + pytest integration tests +# +# Notes: +# - test_images.py must be run manually from a compute node (see Phase 3 output). +# - To resume from Phase 2 or 3 on a later day, set EXPLICIT_TAG in your config +# to the TAG printed at the start of Phase 1 (or stored in ~/.zppy_test_tag), +# and set START_PHASE accordingly. + +set -e # Exit on error +set -u # Exit on undefined variable + +SCRIPT_RUN_DIR="$PWD" + +# ============================================================================ +# Parse arguments +# ============================================================================ + +CONFIG_FILE="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --config) CONFIG_FILE="$2"; shift 2 ;; + *) echo "Unknown argument: $1"; exit 1 ;; + esac +done + +if [[ -z "$CONFIG_FILE" ]]; then + echo "Error: --config is required." + echo "Usage: $0 --config path/to/your.cfg" + exit 1 +fi + +if [[ ! -f "$CONFIG_FILE" ]]; then + echo "Error: Config file not found: $CONFIG_FILE" + exit 1 +fi + +# Source the config. Variables defined there become the script's environment. +# shellcheck disable=SC1090 +source "$CONFIG_FILE" + +# Validate required config keys. +_required_vars=( + MACHINE START_PHASE AUTO_MODE EXPLICIT_TAG + RUN_NUMBER + DIAGS_BASE_BRANCH E3SM_TO_CMIP_BASE_BRANCH MPAS_BASE_BRANCH ZI_BASE_BRANCH ZPPY_BASE_BRANCH + DIAGS_ENV_TYPE E3SM_TO_CMIP_ENV_TYPE MPAS_ENV_TYPE ZI_ENV_TYPE + CFGS_TO_RUN TASKS_TO_RUN + HOME_DIR EZ_DIR + E3SM_DIAGS_DIR E3SM_TO_CMIP_DIR MPAS_ANALYSIS_DIR ZPPY_INTERFACES_DIR ZPPY_DIR + CONDA_PROFILE TAG_CACHE_FILE +) +_missing=() +for _var in "${_required_vars[@]}"; do + if [[ -z "${!_var+x}" ]]; then + _missing+=("$_var") + fi +done +if [[ ${#_missing[@]} -gt 0 ]]; then + echo "Error: The following required variables are missing from ${CONFIG_FILE}:" + printf ' %s\n' "${_missing[@]}" + exit 1 +fi + +# Apply defaults for optional *_EXISTING_ENV variables so the rest of the +# script can reference them unconditionally. +DIAGS_EXISTING_ENV="${DIAGS_EXISTING_ENV:-}" +E3SM_TO_CMIP_EXISTING_ENV="${E3SM_TO_CMIP_EXISTING_ENV:-}" +MPAS_EXISTING_ENV="${MPAS_EXISTING_ENV:-}" +ZI_EXISTING_ENV="${ZI_EXISTING_ENV:-}" +ZPPY_EXISTING_ENV="${ZPPY_EXISTING_ENV:-}" + +# Validate MACHINE value. +case "$MACHINE" in + chrysalis|compy|perlmutter) ;; + *) echo "Error: Unknown MACHINE '${MACHINE}'. Valid values: chrysalis | compy | perlmutter"; exit 1 ;; +esac + +# ============================================================================ +# Machine-specific settings +# ============================================================================ + +case "$MACHINE" in + chrysalis) + OUTPUT_WORKSPACE="/lcrc/group/e3sm/${USER}" + CONDA_ACTIVATION_CMD="lcrc_conda" + UNIFIED_ENV_CMD="source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh" + SALLOC_CMD="salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm" + ;; + compy) + OUTPUT_WORKSPACE="/compyfs/${USER}" + CONDA_ACTIVATION_CMD="compy_conda" + UNIFIED_ENV_CMD="source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh" + SALLOC_CMD="salloc --nodes=1 --partition=short --time=01:00:00 --account=e3sm" + ;; + perlmutter) + OUTPUT_WORKSPACE="/global/cfs/cdirs/e3sm/${USER}" + CONDA_ACTIVATION_CMD="nersc_conda" + UNIFIED_ENV_CMD="source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh" + SALLOC_CMD="salloc --nodes=1 --qos=interactive --time=01:00:00 --constraint=cpu --account=e3sm" + ;; +esac + +# Derive the filename suffix used by generated zppy cfg files. +case "$MACHINE" in + chrysalis) MACHINE_CFG_SUFFIX="chrysalis" ;; + compy) MACHINE_CFG_SUFFIX="compy" ;; + perlmutter) MACHINE_CFG_SUFFIX="pm-cpu" ;; +esac + +# Split comma-separated config lists into bash arrays. +# IFS = Internal Field Separator +IFS=',' read -ra CFGS_ARRAY <<< "$CFGS_TO_RUN" +IFS=',' read -ra TASKS_ARRAY <<< "$TASKS_TO_RUN" +# ============================================================================ +# +# Priority: +# 1. EXPLICIT_TAG from config (always wins when non-empty) +# 2. $TAG_CACHE_FILE written by a prior Phase 1 run (auto-resume) +# 3. Fresh timestamp (Phase 1 first run) +# +# Phase 1 always writes the resolved TAG to $TAG_CACHE_FILE so later phases +# can pick it up automatically without needing EXPLICIT_TAG. + +if [[ -n "$EXPLICIT_TAG" ]]; then + TAG="$EXPLICIT_TAG" + DATE_STAMP="${TAG%%_run*}" +elif [[ "$START_PHASE" -gt 1 && -f "$TAG_CACHE_FILE" ]]; then + TAG="$(cat "$TAG_CACHE_FILE")" + DATE_STAMP="${TAG%%_run*}" + echo "Loaded TAG from ${TAG_CACHE_FILE}: ${TAG}" + echo "(Set EXPLICIT_TAG in your config to override.)" +else + DATE_STAMP="$(date +%Y%m%d)" + TAG="${DATE_STAMP}_run${RUN_NUMBER}" +fi + +# ============================================================================ +# Derived (probably no edits needed) +# ============================================================================ + +UNIQUE_ID="zppy_main_branch_test_${TAG}" + +ZPPY_ENV="test-zppy-${ZPPY_BASE_BRANCH}-${TAG}" + +# Output directories (status file locations) +BUNDLES_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_bundles_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" +LEGACY_310_BUNDLES_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.1.0_bundles_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" +LEGACY_300_BUNDLES_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.0.0_bundles_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" +V2_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_comprehensive_v2_output/${UNIQUE_ID}/v2.LR.historical_0201/post/scripts" +LEGACY_310_V2_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.1.0_comprehensive_v2_output/${UNIQUE_ID}/v2.LR.historical_0201/post/scripts" +LEGACY_300_V2_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/${UNIQUE_ID}/v2.LR.historical_0201/post/scripts" +V3_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_comprehensive_v3_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" +LEGACY_310_V3_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.1.0_comprehensive_v3_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" +LEGACY_300_V3_OUTPUT="${OUTPUT_WORKSPACE}/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/${UNIQUE_ID}/v3.LR.historical_0051/post/scripts" + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# ============================================================================ +# Helper Functions +# ============================================================================ + +log() { + echo -e "${BLUE}[$(date +'%Y-%m-%d %H:%M:%S')]${NC} $*" +} + +log_success() { + echo -e "${GREEN}[$(date +'%Y-%m-%d %H:%M:%S')] ✓${NC} $*" +} + +log_warning() { + echo -e "${YELLOW}[$(date +'%Y-%m-%d %H:%M:%S')] ⚠${NC} $*" +} + +log_error() { + echo -e "${RED}[$(date +'%Y-%m-%d %H:%M:%S')] ✗${NC} $*" +} + +checkpoint() { + local message="$1" + if [ "$AUTO_MODE" = false ]; then + log_warning "CHECKPOINT: $message" + read -rp "Press Enter to continue or Ctrl+C to abort..." + else + log "AUTO MODE: Passing checkpoint -- $message" + fi +} + +# Activate conda and (optionally) a named environment. +activate_env() { + local env_name="${1:-}" + set +u + # shellcheck disable=SC1090 + source ~/.bashrc + $CONDA_ACTIVATION_CMD # Machine-specific conda init (lcrc_conda / compy_conda / nersc_conda) + + if [ -n "$env_name" ]; then + conda activate "$env_name" + log "Installing/updating package in '$env_name'..." + python -m pip install . + fi + set -u +} + +# Activate the machine-specific unified environment. +# UNIFIED_ENV_CMD is always "source /path/to/script.sh" (set in the +# machine-specific case block above), so we strip the leading "source " +# and source the path directly -- no eval required. +activate_unified_env() { + set +u + # shellcheck disable=SC1090 + source ~/.bashrc + $CONDA_ACTIVATION_CMD + # shellcheck disable=SC1090 + source "${UNIFIED_ENV_CMD#source }" + set -u +} + +# Create (if needed) and activate a conda environment. +setup_conda_env() { + local conda_dir="$1" # Directory containing dev.yml (e.g. "conda" or "conda-env") + local env_name="$2" + + activate_env # Ensure conda itself is available + + if conda env list | grep -q "^${env_name} "; then + log "Environment '$env_name' already exists, skipping creation" + else + log "Creating environment '$env_name' from ${conda_dir}/dev.yml..." + rm -rf build + conda clean --all --yes + if [[ "$conda_dir" == "none" ]]; then + conda create --name "$env_name" --file dev-spec.txt --yes + else + conda env create -f "${conda_dir}/dev.yml" -n "$env_name" + fi + fi + + activate_env "$env_name" + log_success "Environment '$env_name' ready" +} + + +# Checkout test branch, creating it from upstream/ if it doesn't exist. +# Stashes/commits any in-progress work first. +ensure_test_branch() { + local test_branch="$1" + local base_branch="$2" + local current_branch + current_branch=$(git rev-parse --abbrev-ref HEAD 2>/dev/null) + + if [ "$current_branch" = "$test_branch" ]; then + log "Already on branch '$test_branch'" + return 0 + fi + + log "Saving current work before switching branches..." + git add -A + git commit -m "Auto-save before test" --no-verify || true + + if git show-ref --verify --quiet "refs/heads/$test_branch"; then + log "Checking out existing branch '$test_branch'..." + git checkout "$test_branch" + else + log "Creating new branch '$test_branch' from upstream/${base_branch}..." + git fetch upstream "${base_branch}" + git checkout -b "$test_branch" "upstream/${base_branch}" + fi + log_success "On branch '$test_branch'" +} + +# Return the environment_commands string for a component. +# Usage: get_env_cmd "dev" "$ENV_NAME" +# get_env_cmd "unified" "" +get_env_cmd() { + local env_type="$1" + local env_name="$2" + if [[ "$env_type" == "dev" ]]; then + echo "source ${CONDA_PROFILE}; conda activate ${env_name}" + else + echo "$UNIFIED_ENV_CMD" + fi +} + +# Poll squeue until no user jobs remain (or timeout). +wait_for_slurm_jobs() { + local check_interval=${1:-600} # seconds between checks (default 10 min) + local max_wait=${2:-14400} # max total wait seconds (default 4 hours) + + log "Waiting for SLURM jobs to complete (checking every ${check_interval}s, max ${max_wait}s)..." + local elapsed=0 + local prev_failed_count=0 + + while true; do + local job_count + job_count=$(squeue -u "${USER}" | wc -l) + job_count=$((job_count - 1)) # subtract header + + # Detect DependencyNeverSatisfied + local failed_jobs + failed_jobs=$(squeue -u "${USER}" | grep "DependencyNeverSatisfied" || true) + local failed_count=0 + if [ -n "$failed_jobs" ]; then + failed_count=$(echo "$failed_jobs" | wc -l) + fi + + if [ "$failed_count" -gt "$prev_failed_count" ]; then + log_error "Jobs with DependencyNeverSatisfied:" + echo "$failed_jobs" + if [ "$job_count" -eq "$failed_count" ]; then + log_error "All remaining jobs have DependencyNeverSatisfied -- cancelling." + scancel -u "${USER}" + return 1 + fi + fi + prev_failed_count=$failed_count + + if [ "$job_count" -eq 0 ]; then + log_success "All SLURM jobs completed!" + return 0 + fi + + if [ "$elapsed" -ge "$max_wait" ]; then + log_error "Timeout after ${max_wait}s waiting for SLURM jobs" + log_error "This script is going to exit now. However, the jobs in the queue will NOT be terminated. Once they finish, you may re-invoke this script with START_PHASE=2 or START_PHASE=3 in your config to continue." + log_error " TAG for this run: ${TAG}" + log_error " Resume: set START_PHASE=2 and EXPLICIT_TAG=${TAG} in your config, then re-run." + log_error " (TAG is also saved in ${TAG_CACHE_FILE})" + return 1 + fi + + echo -ne "\r${YELLOW}Jobs remaining: $job_count${NC} (elapsed: ${elapsed}s / max: ${max_wait}s)" + sleep "$check_interval" + elapsed=$((elapsed + check_interval)) + done + echo "" +} + +# Grep status files in a directory for any non-OK lines. +# Returns 0 if all OK, 1 if any failures found. +check_status_files() { + local dir="$1" + local name="$2" + + if [ ! -d "$dir" ]; then + log_warning "$name: Directory not found: $dir" + return 1 + fi + + if ! compgen -G "${dir}/*status" > /dev/null; then + log_warning "$name: No status files found in ${dir}" + return 1 + fi + + local errors + errors=$(grep -v "OK" "${dir}"/*status 2>/dev/null || true) + + if [ -z "$errors" ]; then + log_success "$name: All status files OK in ${dir}" + return 0 + else + log_error "$name: Non-OK statuses found in ${dir}:" + echo "$errors" + return 1 + fi +} + +# ============================================================================ +# Phase 1: Environment Setup + Initial SLURM Jobs +# ============================================================================ + +phase_1_setup() { + # Save TAG immediately so later phases can find it even if the date changes. + echo "$TAG" > "$TAG_CACHE_FILE" + + log "=========================================" + log "Phase 1: Setup" + log "Config file: $CONFIG_FILE" + log "Date stamp: $DATE_STAMP" + log "TAG: $TAG (saved to ${TAG_CACHE_FILE})" + log "Unique ID: $UNIQUE_ID" + log "" + log "To resume from a later phase, set in your config:" + log " START_PHASE=2" + log " EXPLICIT_TAG=${TAG}" + log "=========================================" + + # ------------------------------------------------------------------ + # e3sm_to_cmip + # ------------------------------------------------------------------ + log "Setting up e3sm_to_cmip..." + + local E3SM_TO_CMIP_ENV="" + if [[ "$E3SM_TO_CMIP_ENV_TYPE" == "dev" ]]; then + if [[ -n "$E3SM_TO_CMIP_EXISTING_ENV" ]]; then + E3SM_TO_CMIP_ENV="$E3SM_TO_CMIP_EXISTING_ENV" + else + E3SM_TO_CMIP_ENV="test-e3sm-to-cmip-${E3SM_TO_CMIP_BASE_BRANCH}-${TAG}" + fi + fi + + ( + cd "$E3SM_TO_CMIP_DIR" + ensure_test_branch "test_e3sm_to_cmip_${TAG}" "$E3SM_TO_CMIP_BASE_BRANCH" + + log "Latest e3sm_to_cmip commit (should match https://github.com/E3SM-Project/e3sm_to_cmip/commits/${E3SM_TO_CMIP_BASE_BRANCH}):" + git log -1 --oneline + + if [[ "$E3SM_TO_CMIP_ENV_TYPE" == "dev" ]]; then + if [[ -n "$E3SM_TO_CMIP_EXISTING_ENV" ]]; then + log "Reusing existing 'e3sm_to_cmip' env: $E3SM_TO_CMIP_EXISTING_ENV (skipping creation)" + activate_env "$E3SM_TO_CMIP_EXISTING_ENV" + else + setup_conda_env "conda-env" "$E3SM_TO_CMIP_ENV" + fi + else + log "Using unified env for e3sm_to_cmip (skipping conda env creation)" + fi + ) + + # ------------------------------------------------------------------ + # e3sm_diags + # ------------------------------------------------------------------ + log "Setting up e3sm_diags..." + + local DIAGS_ENV="" + if [[ "$DIAGS_ENV_TYPE" == "dev" ]]; then + if [[ -n "$DIAGS_EXISTING_ENV" ]]; then + DIAGS_ENV="$DIAGS_EXISTING_ENV" + else + DIAGS_ENV="test-diags-${DIAGS_BASE_BRANCH}-${TAG}" + fi + fi + + ( + cd "$E3SM_DIAGS_DIR" + ensure_test_branch "test_e3sm_diags_${TAG}" "$DIAGS_BASE_BRANCH" + + log "Latest e3sm_diags commit (should match https://github.com/E3SM-Project/e3sm_diags/commits/${DIAGS_BASE_BRANCH}):" + git log -1 --oneline + + if [[ "$DIAGS_ENV_TYPE" == "dev" ]]; then + if [[ -n "$DIAGS_EXISTING_ENV" ]]; then + log "Reusing existing 'e3sm_diags' env: $DIAGS_EXISTING_ENV (skipping creation)" + activate_env "$DIAGS_EXISTING_ENV" + else + setup_conda_env "conda-env" "$DIAGS_ENV" + fi + else + log "Using unified env for e3sm_diags (skipping conda env creation)" + fi + ) + + # ------------------------------------------------------------------ + # MPAS-Analysis + # ------------------------------------------------------------------ + log "Setting up MPAS-Analysis..." + + local MPAS_ENV="" + if [[ "$MPAS_ENV_TYPE" == "dev" ]]; then + if [[ -n "$MPAS_EXISTING_ENV" ]]; then + MPAS_ENV="$MPAS_EXISTING_ENV" + else + MPAS_ENV="test-mpas-${MPAS_BASE_BRANCH}-${TAG}" + fi + fi + + ( + cd "$MPAS_ANALYSIS_DIR" + ensure_test_branch "test_mpas_${TAG}" "$MPAS_BASE_BRANCH" + + log "Latest MPAS-Analysis commit (should match https://github.com/MPAS-Dev/MPAS-Analysis/commits/${MPAS_BASE_BRANCH}):" + git log -1 --oneline + + if [[ "$MPAS_ENV_TYPE" == "dev" ]]; then + if [[ -n "$MPAS_EXISTING_ENV" ]]; then + log "Reusing existing 'MPAS-Analysis' env: $MPAS_EXISTING_ENV (skipping creation)" + activate_env "$MPAS_EXISTING_ENV" + else + setup_conda_env "none" "$MPAS_ENV" + fi + else + log "Using unified env for MPAS-Analysis (skipping conda env creation)" + fi + ) + + # ------------------------------------------------------------------ + # zppy-interfaces (includes unit tests) + # ------------------------------------------------------------------ + log "Setting up zppy-interfaces..." + + local ZI_ENV="" + if [[ "$ZI_ENV_TYPE" == "dev" ]]; then + if [[ -n "$ZI_EXISTING_ENV" ]]; then + ZI_ENV="$ZI_EXISTING_ENV" + else + ZI_ENV="test-zi-${ZI_BASE_BRANCH}-${TAG}" + fi + fi + + ( + cd "$ZPPY_INTERFACES_DIR" + ensure_test_branch "test_zi_${TAG}" "$ZI_BASE_BRANCH" + + log "Latest zppy-interfaces commit (should match https://github.com/E3SM-Project/zppy-interfaces/commits/${ZI_BASE_BRANCH}):" + git log -1 --oneline + + if [[ "$ZI_ENV_TYPE" == "dev" ]]; then + if [[ -n "$ZI_EXISTING_ENV" ]]; then + log "Reusing existing 'zppy-interfaces' env: $ZI_EXISTING_ENV (skipping creation)" + activate_env "$ZI_EXISTING_ENV" + else + setup_conda_env "conda" "$ZI_ENV" + fi + else + log "Using unified env for zppy-interfaces..." + activate_unified_env + fi + + log "Running zppy-interfaces unit tests..." + pytest tests/unit/global_time_series/test_*.py + pytest tests/unit/pcmdi_diags/test_*.py + log_success "zppy-interfaces unit tests passed" + ) + + # ------------------------------------------------------------------ + # zppy (includes unit tests + config generation) + # ------------------------------------------------------------------ + log "Setting up zppy..." + + # Resolve ZPPY_ENV name before the subshell so it's available for + # config generation and later phases. + if [[ -n "$ZPPY_EXISTING_ENV" ]]; then + ZPPY_ENV="$ZPPY_EXISTING_ENV" + fi + # (If ZPPY_EXISTING_ENV is empty, ZPPY_ENV retains the auto-generated + # name set at the top of the script.) + + ( + cd "$ZPPY_DIR" + ensure_test_branch "test_zppy_${TAG}" "$ZPPY_BASE_BRANCH" + + log "Latest zppy commit (should match https://github.com/E3SM-Project/zppy/commits/${ZPPY_BASE_BRANCH}):" + git log -1 --oneline + + if [[ -n "$ZPPY_EXISTING_ENV" ]]; then + log "Reusing existing 'zppy' env: $ZPPY_EXISTING_ENV (skipping creation)" + activate_env "$ZPPY_ENV" + else + setup_conda_env "conda" "$ZPPY_ENV" + fi + + log "Running zppy unit tests..." + pytest tests/test_*.py + log_success "zppy unit tests passed" + ) + + # ------------------------------------------------------------------ + # Generate config files (update utils.py TEST_SPECIFICS, then run it) + # ------------------------------------------------------------------ + log "Generating config files..." + + local E3SM_TO_CMIP_CMD + local DIAGS_CMD + local MPAS_CMD + local ZI_CMD + E3SM_TO_CMIP_CMD=$(get_env_cmd "$E3SM_TO_CMIP_ENV_TYPE" "$E3SM_TO_CMIP_ENV") + DIAGS_CMD=$(get_env_cmd "$DIAGS_ENV_TYPE" "$DIAGS_ENV") + MPAS_CMD=$(get_env_cmd "$MPAS_ENV_TYPE" "$MPAS_ENV") + ZI_CMD=$(get_env_cmd "$ZI_ENV_TYPE" "$ZI_ENV") + + # Config generation and job submission run in the parent shell so that + # the zppy command is available and cd/env state is consistent. + cd "$ZPPY_DIR" + activate_env "$ZPPY_ENV" + ensure_test_branch "test_zppy_${TAG}" "$ZPPY_BASE_BRANCH" + + UTILS_FILE="tests/integration/utils.py" + + # Build Python list literals from the bash arrays for injection into the heredoc. + local CFGS_PY_LIST TASKS_PY_LIST cfg task + CFGS_PY_LIST="" + for cfg in "${CFGS_ARRAY[@]}"; do + cfg="${cfg// /}" # strip any accidental whitespace + CFGS_PY_LIST+=" \"${cfg}\","$'\n' + done + TASKS_PY_LIST="" + for task in "${TASKS_ARRAY[@]}"; do + task="${task// /}" + TASKS_PY_LIST+="\"${task}\", " + done + TASKS_PY_LIST="${TASKS_PY_LIST%, }" # strip trailing comma+space + + python - < "${SCRIPT_RUN_DIR}/env_${TAG}.txt" + local cfg cfg_path + for cfg in "${CFGS_ARRAY[@]}"; do + cfg="${cfg// /}" + cfg_path="tests/integration/generated/test_${cfg}_${MACHINE_CFG_SUFFIX}.cfg" + if [[ ! -f "$cfg_path" ]]; then + log_error "Config file not found: $cfg_path" + log_error "Check that each CFGS_TO_RUN entry matches a cfg name generated by tests/integration/utils.py (e.g., weekly_comprehensive_v3)." + exit 1 + fi + log "Submitting: $cfg_path" + zppy -c "$cfg_path" + done + + local job_count + job_count=$(squeue -u "${USER}" | wc -l) + job_count=$((job_count - 1)) + log_success "Submitted jobs. Current queue depth: $job_count" + + checkpoint "Phase 1 jobs submitted. Waiting for them to finish..." + wait_for_slurm_jobs 600 14400 # Check every 10 min, max 4 hours + + log_success "Phase 1 complete!" +} + +# ============================================================================ +# Phase 2: Bundles Part 2 +# ============================================================================ + +phase_2_bundles_part2() { + log "=========================================" + log "Phase 2: Bundles Part 2" + log "TAG: $TAG" + log "=========================================" + + cd "$ZPPY_DIR" + activate_env "$ZPPY_ENV" + ensure_test_branch "test_zppy_${TAG}" "$ZPPY_BASE_BRANCH" + + # Verify bundle status files are clean before submitting part 2. + # These paths are fixed regardless of CFGS_TO_RUN; skip any that don't exist yet. + log "Checking bundle status files before submitting part 2..." + local all_ok=true + check_status_files "$BUNDLES_OUTPUT" "Bundles" || all_ok=false + check_status_files "$LEGACY_310_BUNDLES_OUTPUT" "Legacy 3.1.0 Bundles" || all_ok=false + check_status_files "$LEGACY_300_BUNDLES_OUTPUT" "Legacy 3.0.0 Bundles" || all_ok=false + + if [ "$all_ok" = false ]; then + log_error "One or more bundle status files have non-OK entries." + checkpoint "Errors found. Continue anyway?" + else + log_success "Bundle status files look clean -- safe to submit part 2." + fi + + log "Submitting bundles part 2..." + local cfg cfg_path + for cfg in "${CFGS_ARRAY[@]}"; do + cfg="${cfg// /}" + if [[ "$cfg" == *bundle* ]]; then + cfg_path="tests/integration/generated/test_${cfg}_${MACHINE_CFG_SUFFIX}.cfg" + if [[ ! -f "$cfg_path" ]]; then + log_error "Config file not found: $cfg_path" + log_error "Check that each CFGS_TO_RUN entry matches a cfg name generated by tests/integration/utils.py (e.g., weekly_bundles)." + exit 1 + fi + log "Submitting: $cfg_path" + zppy -c "$cfg_path" + fi + done + + local job_count + job_count=$(squeue -u "${USER}" | wc -l) + job_count=$((job_count - 1)) + log_success "Bundles part 2 submitted. Current queue depth: $job_count" + + wait_for_slurm_jobs 600 3600 # Check every 10 min, max 1 hour + + log_success "Phase 2 complete!" +} + +# ============================================================================ +# Phase 3: Validation (status checks + pytest integration tests) +# ============================================================================ + +phase_3_validation() { + log "=========================================" + log "Phase 3: Validation" + log "TAG: $TAG" + log "=========================================" + + cd "$ZPPY_DIR" + activate_env "$ZPPY_ENV" + ensure_test_branch "test_zppy_${TAG}" "$ZPPY_BASE_BRANCH" + + # ------------------------------------------------------------------ + # Status file checks + # ------------------------------------------------------------------ + log "Checking all status files..." + local all_good=true + + check_status_files "$V2_OUTPUT" "v2" || all_good=false + check_status_files "$LEGACY_310_V2_OUTPUT" "Legacy 3.1.0 v2" || all_good=false + check_status_files "$LEGACY_300_V2_OUTPUT" "Legacy 3.0.0 v2" || all_good=false + check_status_files "$V3_OUTPUT" "v3" || all_good=false + check_status_files "$LEGACY_310_V3_OUTPUT" "Legacy 3.1.0 v3" || all_good=false + check_status_files "$LEGACY_300_V3_OUTPUT" "Legacy 3.0.0 v3" || all_good=false + check_status_files "$BUNDLES_OUTPUT" "Bundles" || all_good=false + check_status_files "$LEGACY_310_BUNDLES_OUTPUT" "Legacy 3.1.0 Bundles" || all_good=false + check_status_files "$LEGACY_300_BUNDLES_OUTPUT" "Legacy 3.0.0 Bundles" || all_good=false + + if [ "$all_good" = false ]; then + log_error "Some status checks failed!" + checkpoint "Errors found in status files. Continue to pytest anyway?" + else + log_success "All status files clean!" + fi + + # ------------------------------------------------------------------ + # pytest integration tests + # ------------------------------------------------------------------ + log "Running integration tests..." + + log "Running test_last_year.py (no expected results dir)..." + pytest tests/integration/test_last_year.py \ + || log_warning "test_last_year.py had failures" + + log "Running test_bash_generation.py..." + pytest tests/integration/test_bash_generation.py \ + || log_warning "test_bash_generation.py had failures" + + log "Running test_campaign.py..." + pytest tests/integration/test_campaign.py \ + || log_warning "test_campaign.py had failures" + + log "Running test_defaults.py..." + pytest tests/integration/test_defaults.py \ + || log_warning "test_defaults.py had failures" + + log "Running test_bundles.py..." + pytest tests/integration/test_bundles.py \ + || log_warning "test_bundles.py had failures" + + # ------------------------------------------------------------------ + # test_images.py -- must run from a compute node + # ------------------------------------------------------------------ + log_warning "test_images.py requires a compute node and must be run manually." + log "To run it on ${MACHINE}:" + log " ${SALLOC_CMD}" + log " source ${CONDA_PROFILE}" + log " conda activate ${ZPPY_ENV}" + log " cd ${ZPPY_DIR}" + log " pytest tests/integration/test_images.py" + log " cat test_images_summary.md" + + log_success "Phase 3 automated tests complete!" + log_success "Remember to run test_images.py manually from a compute node." +} + +# ============================================================================ +# Main +# ============================================================================ + +main() { + log "Starting zppy integration test automation" + log "Config file: $CONFIG_FILE" + log "Machine: $MACHINE" + log "TAG: $TAG" + log "Auto mode: $AUTO_MODE" + log "Start phase: $START_PHASE" + + case "$START_PHASE" in + 1) + phase_1_setup + phase_2_bundles_part2 + phase_3_validation + ;; + 2) + phase_2_bundles_part2 + phase_3_validation + ;; + 3) + phase_3_validation + ;; + *) + log_error "Invalid START_PHASE: $START_PHASE (must be 1, 2, or 3)" + exit 1 + ;; + esac + + log_success "Integration test automation complete!" +} + +main diff --git a/tests/main_branch_testing/zppy_test.cfg b/tests/main_branch_testing/zppy_test.cfg new file mode 100644 index 00000000..983564f7 --- /dev/null +++ b/tests/main_branch_testing/zppy_test.cfg @@ -0,0 +1,66 @@ +# zppy integration test configuration +# Source: run_integration_test.bash --config path/to/this.cfg +# +# Sections are comments only -- bash sources this file as KEY=VALUE pairs. + +# ---------------------------------------------------------------------------- +# Runtime (formerly "Parse arguments") +# ---------------------------------------------------------------------------- + +MACHINE=chrysalis # chrysalis | compy | perlmutter +START_PHASE=1 # 1 | 2 | 3 +AUTO_MODE=true # true = skip all interactive checkpoints +EXPLICIT_TAG="" # Leave empty to auto-generate; set to resume a prior run + +# ---------------------------------------------------------------------------- +# Per-run settings +# ---------------------------------------------------------------------------- + +RUN_NUMBER=1 + +DIAGS_BASE_BRANCH="main" +E3SM_TO_CMIP_BASE_BRANCH="master" +MPAS_BASE_BRANCH="develop" +ZI_BASE_BRANCH="main" +ZPPY_BASE_BRANCH="main" + +# "dev" = build a dedicated conda env from the repo's dev.yml +# "unified" = use the machine's e3sm-unified env (UNIFIED_ENV_CMD) +DIAGS_ENV_TYPE="dev" +E3SM_TO_CMIP_ENV_TYPE="dev" +MPAS_ENV_TYPE="dev" +ZI_ENV_TYPE="dev" + +# Optional: reuse an existing named conda env instead of creating a new one. +# When non-empty AND the corresponding ENV_TYPE is "dev", the script skips +# conda env creation and activates this env directly. +# Leave empty to let the script auto-name and create the env as usual. +DIAGS_EXISTING_ENV="" +E3SM_TO_CMIP_EXISTING_ENV="" +MPAS_EXISTING_ENV="" +ZI_EXISTING_ENV="" +ZPPY_EXISTING_ENV="" + +# Comma-separated list of zppy cfg names to generate and submit. +# These correspond to generated filenames: test_weekly__.cfg +# Any name containing "bundle" is treated as a bundle cfg and re-submitted in Phase 2. +CFGS_TO_RUN="weekly_bundles,weekly_comprehensive_v2,weekly_comprehensive_v3,weekly_legacy_3.1.0_bundles,weekly_legacy_3.1.0_comprehensive_v2,weekly_legacy_3.1.0_comprehensive_v3,weekly_legacy_3.0.0_bundles,weekly_legacy_3.0.0_comprehensive_v2,weekly_legacy_3.0.0_comprehensive_v3" + +# Comma-separated list of tasks to enable in utils.py. +TASKS_TO_RUN="e3sm_diags,mpas_analysis,global_time_series,ilamb,livvkit,pcmdi_diags" + +# ---------------------------------------------------------------------------- +# One-time setup +# ---------------------------------------------------------------------------- + +HOME_DIR="$HOME" +EZ_DIR="$HOME_DIR/ez" + +E3SM_DIAGS_DIR="$EZ_DIR/e3sm_diags" +E3SM_TO_CMIP_DIR="$EZ_DIR/e3sm_to_cmip" +MPAS_ANALYSIS_DIR="$EZ_DIR/MPAS-Analysis" +ZPPY_INTERFACES_DIR="$EZ_DIR/zppy-interfaces" +ZPPY_DIR="$EZ_DIR/zppy" + +CONDA_PROFILE="$HOME_DIR/miniforge3/etc/profile.d/conda.sh" +TAG_CACHE_FILE="$HOME_DIR/.zppy_test_tag"