Enable partial updates of expected results for testing - #852
Open
forsyth2 wants to merge 3 commits into
Open
Conversation
Previously, update_weekly_expected_files_*.sh and
update_campaign_expected_files.sh always overwrote every diag
subdirectory (e.g. e3sm_diags, mpas_analysis, ilamb, ...) or every
campaign in one pass, making it impossible to accept new results for
one diagnostic while leaving another pinned to its old expected output.
- template_update_weekly_expected_files.sh: add a `diags` selector
array. Empty (default) preserves the old full-wipe-and-copy
behavior. Populated (e.g. diags=("e3sm_diags")), only those
subdirectories are refreshed; "bundle_files" is selectable the
same way for the bundle*.bash copy step. The image list is still
generated by walking the full directory, so it reflects both
updated and untouched diags.
- template_update_campaign_expected_files.sh: hardcoded campaign
list replaced with a `campaigns` array for the same kind of subset
updates.
- template_update_bash_generation_expected_files.sh and
template_update_defaults_expected_files.sh: unchanged -- each
produces a single expected-files dir with nothing to partially select.
- docs/source/dev_guide/tests/update_expected_results.rst: document
the new selectors and note the full backup step should still
always run first.
This diff was generated by Claude (Anthropic).
Add the `update_` files that `python tests/integration/utils.py` creates.
forsyth2
added
Documentation
Collaborator
Author
|
Updating expected results for the 8/12 test run Set up branchgit checkout partial-update-expectations
git fetch upstream main
git rebase upstream/main
git log --oneline | head -n 4
# b5d41f34 Add auto-generated update files
# ef72638c Change term from diags to tasks
# fffd2338 Support partial updates of weekly/campaign expected test results
# 4735ef4d Merge pull request #854 from E3SM-Project/update-pre-commit-deps
lcrc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pr852-20260814
conda activate zppy-pr852-20260814
pre-commit run --all-files
python -m pip install .
cd docs
make html
cp -r _build/ /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_docs_pr852
chmod -R 755 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_docs_pr852# First, let's copy over the old expected results #############################
cp -r /lcrc/group/e3sm/public_html/zppy_test_resources /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260814 # < 30 min
# Second, update the expected results #########################################
# Let's update the simpler tests' results first:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# You might have changed branches since you ran the tests.
# Good, on correct branch: test_zppy_20260812_run1
conda activate test-zppy-main-20260812_run1
# Good, now on correct env
# Make sure the update script permissions are set up
chmod 755 tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_defaults_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Specific to partial updates:
# Update the task list
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# This doesn't have the necessary lines yet because PR #852 didn't exist when this was run!
git add -A
git commit -m "Changes made during testing"
# Remove the old templates and replace with the files from PR #852
rm tests/integration/template_update_campaign_expected_files.sh
rm tests/integration/template_update_weekly_expected_files.sh
emacs tests/integration/template_update_campaign_expected_files.sh
emacs tests/integration/template_update_weekly_expected_files.sh
git diff # Looks ok
git add -A
git commit -m "Copy changes from PR 852"
# Now, we need to re-render the generated files
python tests/integration/utils.py
git diff # Looks ok
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Double check we have a record of the original expected results
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260814
# Update the task list now that it's available:
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# These are the 3 tasks we know we can update the expected results for:
# tasks=("e3sm_diags" "global_time_series" "pcmdi_diags")
git add -A
git commit -m "Update the update scripts"
# These scripts update the expected results and re-run the tests:
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 2.84s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 6 passed in 2.43s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.72s
# This script only updates the expected results
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # ~30 minSkipped because ls /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed there are expected results subdirs and image lists for each cfg
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed everything was updated today 8/14
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
pytest tests/integration/test_bundles.py
# 2 passed in 1.53s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
bash
source ~/.bashrc
lcrc_conda
conda activate test-zppy-main-20260812_run1
pytest tests/integration/test_images.py # ~40 min
exit # Exit bash shell
exit # Exit compute noteOutputOutput:
Failing image check tests:
Good, this table matches the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Objectives:
Select one: This pull request is...
Small Change