pre_proc: real-data and static-field preparation pipelines for init_atmosphere#13
Open
daniloceano wants to merge 8 commits into
Open
pre_proc: real-data and static-field preparation pipelines for init_atmosphere#13daniloceano wants to merge 8 commits into
daniloceano wants to merge 8 commits into
Conversation
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- setup_environment.sh: stop inferring env existence from 'conda activate's exit code (which gives a misleading "Couldn't find the environment" when the env exists but a transient activate returns non-zero). Now: ensure the 'conda' shell function is loaded (source conda.sh if only the binary is on PATH), check the env in 'conda env list', then activate. Clearer not-found message. - install_conda_environment.sh: add '#!/usr/bin/env bash' shebang. It uses bash-only features (BASH_SOURCE, '[ == ]'); running it with sh/dash failed with "Bad substitution" / "unexpected operator" and left SCRIPT_DIR empty. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the pre-processing utilities for MPAS init_atmosphere under usp-utils/pre_proc/.
real_data/ — download + intermediate-file generation for real-data init:
- download_{era5,gfs,oisst}.py and {era5,gfs,gfs_sst,oisst,oisst_clim}_to_intermediate.py
- prepare_{era5,gfs,oisst}.sh end-to-end wrappers
- README plus per-scenario recipes (operational/hindcast x global/regional)
static_fields/ — static-field generation (init_atmosphere config_init_case = 7):
- download_static_data.sh (mesh + MPAS geog bundle, integrity-checked)
- STATIC_FIELDS_GUIDE.md walkthrough
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… downscaling docs Name and docs: - era5_to_intermediate.py defaults --prefix to 'ERA5' (was 'GFS'); the ERA5 intermediate is now ERA5:YYYY-MM-DD_HH, set config_met_prefix='ERA5'. GFS keeps 'GFS:'. prepare_era5.sh and the docs follow suit; GFS-source mentions unchanged. Regional downscaling docs: - hindcast_regional.md: explain --area as a download box (not the simulated region, which is the mesh) sized to mesh + margin; add a window-based recipe (START/END) covering a single month, a year, or multiple years, with the 6-hourly LBC loop + closing boundary and daily OISST SST. - README.md: clarify the --area bullet and link the recipe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
era5_to_intermediate.py silently emitted only 12 of 14 fields because two single-level lookups did not match the CDS GRIB: - mean sea level pressure (msl) is at typeOfLevel 'surface' here, not 'meanSea'; - sea-ice cover has GRIB shortName 'ci', which cfgrib loads as variable 'siconc', so the 'siconc'-keyed lookup found nothing. Make the lookups robust: _open_var accepts shortName aliases and returns the single filtered variable regardless of how cfgrib names it; sea ice tries 'ci'/'siconc'; msl tries 'surface' then 'meanSea'. The converter now writes all 14 fields (adds PMSL and SEAICE). README field-mapping notes the ci/siconc alias. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
prepare_era5_series.sh builds the ERA5 (ERA5:) intermediate series over a date window — the LBC series of a month / year / multiple years — by wrapping prepare_era5.sh in a window loop built for unattended runs: - Resumable: skips every time whose ERA5:<date>_<hh> intermediate already exists, so re-running only fetches the gaps. - Fault-tolerant: a failed CDS request logs FAIL and the series continues. - nohup-friendly: timestamped log lines; redirect and detach. - Optional parallelism (--jobs N via xargs -P), with the CDS throttle documented (keep N small). - Timestamps are parsed as UTC and iterated by epoch seconds, so the dates do not shift with the server's local timezone. hindcast_regional.md now recommends it (nohup example) for windows longer than a few days, keeping the equivalent manual loop for reference; README lists it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the MPAS
init_atmospherepre-processing utilities underusp-utils/pre_proc/, covering the two data-preparation stages required for a real-data run.None of this exists on
developtoday — this PR is purely additive.What this adds
real_data/— real-data initialization inputsIncludes utilities for preparing real-data initialization inputs:
Downloaders
download_era5.pydownload_gfs.pydownload_oisst.pyIntermediate-file generators
WPS intermediate format generators:
era5_to_intermediate.pygfs_to_intermediate.pygfs_sst_to_intermediate.pyoisst_to_intermediate.pyoisst_clim_to_intermediate.pyEnd-to-end wrappers
prepare_era5.shprepare_gfs.shprepare_oisst.shDocumentation and recipes
README.mdPer-scenario recipes under
recipes/, covering:static_fields/— static-field generationAdds support for static-field generation with
config_init_case = 7.download_static_data.shFetches the required mesh and MPAS-curated geography bundle, then:
WPS_GEOG/STATIC_FIELDS_GUIDE.mdProvides a full walkthrough covering:
namelistandstreamssetup