Test with frozen envs

[forsyth2]

Summary

Objectives:

• Allow the freezing of dev environments so that we can be sure diffs are introduced by packages we maintain rather than by dependencies. This involves two components:
  • If the current "expected results" were generated using dev environments, then we just need to freeze the versions used there.
  • If the current "expected results" were generated using the Unified release, then we need to freeze a dev environment that matches up with Unified as much as possible.

Issue resolution:

• Resolves Integration testing strategy for `pcmdi_diags` #848
• Follow-up to Add main branch testing automation #774

Select one: This pull request is...

• a bug fix: increment the patch version
• a small improvement: increment the minor version
• a new feature: increment the minor version
• an incompatible (non-backwards compatible) API change: increment the major version

Small Change

• To merge, I will use "Squash and merge". That is, this change should be a single commit.
• Logic: I have visually inspected the entire pull request myself.
• Pre-commit checks: All the pre-commits checks have passed.

[forsyth2]

Action items:

• The first 35 commits of this PR are actually from Add main branch testing automation #774, which adds the automated test script. Once that is merged, we can rebase this branch leaving only the relevant commits.
• Test the easier case: If the current "expected results" were generated using dev environments, then we just need to freeze the versions used there.
• Debug & test the harder case: If the current "expected results" were generated using the Unified release, then we need to freeze a dev environment that matches up with Unified as much as possible.
• Try to build frozen env setup into the main testing script rather than introducing a distinct script tests/main_branch_testing/frozen_env_setup/create_frozen_envs.sh.
• Documentation: Update both docs/source/dev_guide/tests/automated_test.rst and tests/main_branch_testing/README.md. (Alternatively, we may wish to remove the latter and move any relevant content to the former).

[forsyth2]

Currently, the relevant diff can be seen here

[xylar]

@forsyth2, I want to make super sure that I understand what the plan is here. If you are temporarily creating two matching dev environments so you can make a comparison, I'm good with that. If you are hoping to create a frozen dev environment that gets manually updated by maintainers, I am deeply opposed. This got us into a downward spiral in the past where zppy and friends tested with very old dependencies, creating nightmares at E3SM-Unified updates.

[forsyth2]

I want to make super sure that I understand what the plan is here.

@xylar See full discussion at #848. Summary: pcmdi_diags is showing some strange diffs despite no code changes, but this certainly isn't an uncommon situation. global_time_series for instance is currently showing diffs simply because of what appears to be a pixel shift. If we could just pin all dependency versions except for the packages we actually maintain, that would be great. We'd know any diffs we're seeing are actually because of changes we made, and not because say matplotlib version updates have shifted pixels.

If you are hoping to create a frozen dev environment that gets manually updated by maintainers
downward spiral in the past where zppy and friends tested with very old dependencies

Definitely not the idea! My thinking was some sort of cyclical workflow like:

1. E3SM Unified gets released.
2. I update the "expected results" of the integration tests using the Unified release for everything.
3. As packages add features, we can no longer use the Unified environment; we have to use a dev environment. Historically, I've done that by manually creating dev environments for relevant packages (e3sm_to_cmip, e3sm_diags, MPAS-Analysis, zppy-interfaces [that is, global_time_series & pcmdi_diags], zppy itself). The automated test script being introduced in Add main branch testing automation #774 takes the same approach. The problem is, dev environments don't always match previous builds from the same dev.yml and that can introduce things like the pixel shifts I've mentioned above. So, the idea of this PR is to freeze the dev env as much as possible. That is, the two components:

• If the current "expected results" were generated using dev environments, then we just need to freeze the versions used there (plus pip install the package in question)
• If the current "expected results" were generated using the Unified release, then we need to freeze a dev environment that matches up with Unified as much as possible. (plus pip install the package in question)

4. As new features get introduced in the packages, expected results will change, and so the official "expected results" will get updated, and we return to Step 3.
5. During the new Unified release testing period, we use the dependency versions as solved in the Unified RCs at that point.

That is, the dependencies should never be more out-of-date than the last E3SM Unified release. So, in practice, nothing should be more than ~6 months out of date.

[xylar]

I understand the tedium of chasing baselines that are a moving target, and I can understand that pinning dependencies is a tempting way to handle that.

Please try to see things from my perspective. Exactly this temptation is what causes E3SM-Unified integration to become a nightmare for me. I desperately need you to always be testing with the newest possible versions of all your dependencies, and only constraining them with upper bounds when you know they have broken something and you (or someone else) hasn't fixed it yet. E3SM-Project/e3sm_diags#1075 is an example of that kind of a temporary constraint.

So if you can test with a development environment that is as similar as possible to what you used to generate your baselines and test with another environment that has the latest and greatest of everything and be sure it isn't broken, I am good. But if you stop testing the latest dependencies, that's where you create a nightmare for me come October.

[forsyth2]

need you to always be testing with the newest possible versions of all your dependencies

if you can test with a development environment that is as similar as possible to what you used to generate your baselines and test with another environment that has the latest and greatest of everything and be sure it isn't broken, I am good

Thanks @xylar for further explaining the concern. I understand now. We can aim for two test runs per week:

1. Do test run 1. This will use frozen dev environments to see if our packages have caused diffs.
2. If the diffs are expected based on recent changes, we update the expected results. If they are not expected, we debug and return to step 1.
3. Do test run 2. This time, we use cutting-edge dev environments. Any diffs must be because of dependency drift since by this point, we've updated the expected results based on our changes. Here, we would basically always update the expected results regardless of the diffs, unless we see something truly out of the ordinary, because these diffs are outside our control.
4. Repeat again the following week, after more of our code changes have merged.

Am I understanding correctly that this workflow would be a good compromise for both of us? The hassle of running the test twice should be reduced by the automation being introduced in #774.

I should also note this is a bit of a moot point for the immediate future, since @chengzhuzhang noted this should be lower priority nice-to-have item on the discussion thread.

[xylar]

@forsyth2, if that is feasible, it would be amazing from my perspective!

[chengzhuzhang]

@forsyth2 Just to clarify, the current approach, the tests do use latest conda yaml files based on those from individual repos when testing main branches, correct?

[forsyth2]

the tests do use latest conda yaml files based on those from individual repos when testing main branches, correct?

@chengzhuzhang Correct, in my current approach (both manually as during Spring's Unified release AND automated as in #774), I create brand new dev environments using the latest fetched main/master/develop branch for that particular repo. That means the conda yaml files are also up-to-date.

[chengzhuzhang]

@forsyth2 Thanks for clarifying, this is good to confirm. I think we’re in a good position, as our testing is based on the latest dependencies that Conda resolves.