Automatically determine diags paths

[forsyth2]

The code block at https://github.com/E3SM-Project/zppy/blob/main/zppy/e3sm_diags.py#L68 already allows for reference paths to be "guessed" from the main reference path given. This could be extended further -- for example, dc_obs_climo and ts_obs could similarly be "guessed" from the observation paths.

[chengzhuzhang]

Bump up this issue, when finalizing the cfg file for v3, @golaz has suggested that we can make the .cfg files more portable by using mache. Mache has already used for mapping files, we can use it to auto provide machine-specific diags paths as well.

[forsyth2]

@chengzhuzhang This might already be resolved...

In the latest https://github.com/E3SM-Project/zppy/blob/main/zppy/e3sm_diags.py, we have the parameter inference feature to determine dc_obs_climo and tc_obs.

        if "diurnal_cycle" in c["sets"]:
            set_value_of_parameter_if_undefined(
                c,
                "dc_obs_climo",
                c["reference_data_path"],
                ParameterInferenceType.PATH_INFERENCE,
            )

    if "tc_analysis" in c["sets"]:
        set_value_of_parameter_if_undefined(
            c,
            "tc_obs",
            f"{c['diagnostics_base_path']}/observations/Atm/tc-analysis/",
            ParameterInferenceType.PATH_INFERENCE,
        )

The parameter inference feature was very extensive; I think it should cover many use cases.

[chengzhuzhang]

Oh, I didn't know. I think next step is to use it in .cfg to test if they work as intended.

[forsyth2]

@chengzhuzhang Just note that some of @golaz's suggested use cases are beyond zppy's inference capabilities -- e.g., #694 (comment), hence they require explicit parameter setting.

[chengzhuzhang]

I looked at the inference code. It looks like reference_data_path can be inferred:
https://github.com/E3SM-Project/zppy/blob/bfbba5f9c3794cd7e399e35f8153866b1c8f6910/zppy/e3sm_diags.py#L167C5-L172C6 did i miss understand?

[chengzhuzhang]

in my current attempt, i removed all the reference data path in the config, and to see if zppy can find create the correct paths.

[forsyth2]

It looks like reference_data_path can be inferred:

Yes, it can be inferred, but an inference isn't a guarantee. It works fine if the reference data is where zppy thinks it will be. So, the inference will work if the reference data actually is at f"{c['diagnostics_base_path']}/observations/Atm/climatology/". If it's organized in any other way, say we actually want data at f"{c['diagnostics_base_path']}/observations_new_data/Atm/climatology/", then it's going to be a bad inference.

[chengzhuzhang]

okay, got it. I think at this point we just need the standard paths, without needing to consider the edge cases mentioned. It looks like the bash file looks correct. I will update once the runs are finished. And in cases user needs to use a new obs path or they are not on a mache supported machines, they just need to manually add the paths parameters to point to the data.