diff --git a/.gitignore b/.gitignore index 0d1c3f67..e86b337b 100644 --- a/.gitignore +++ b/.gitignore @@ -72,3 +72,22 @@ met_data # Python compiled bytecode folders __pycache__ + +# WPS geographical static data (large downloads — never commit) +WPS_GEOG/ +._WPS_GEOG +geog_*.tar.gz +geog_*.tar.bz2 + +# Stray per-user / HOME files created in the repo when a tool ran with HOME +# pointing here (conda cache+envs, shell history, etc.). Never commit; the real +# ones live in $HOME and these copies are safe to delete. +.conda/ +.cache/ +.local/ +.config/ +.claude/ +.condarc +.viminfo +.python_history +.bash_history diff --git a/usp-utils/install_conda_environment.sh b/usp-utils/install_conda_environment.sh index 3d3e4004..eeb9bca3 100755 --- a/usp-utils/install_conda_environment.sh +++ b/usp-utils/install_conda_environment.sh @@ -1,4 +1,9 @@ -#get directory where the sourced script is located +#!/usr/bin/env bash +# Run with bash (./install_conda_environment.sh or bash install_conda_environment.sh). +# It uses bash-only features (BASH_SOURCE, [ == ]); running it with `sh`/dash +# fails with "Bad substitution" / "unexpected operator". + +# get directory where this script is located SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd ) CYAN='\033[0;36m' diff --git a/usp-utils/pre_proc/real_data/README.md b/usp-utils/pre_proc/real_data/README.md new file mode 100644 index 00000000..9197a7d6 --- /dev/null +++ b/usp-utils/pre_proc/real_data/README.md @@ -0,0 +1,303 @@ +# `real_data/` — GFS / ERA5 → WPS intermediate met files for MPAS + +Scripts to fetch meteorological data and turn it into the **WPS intermediate-format** +file (`GFS:` or `ERA5:` `YYYY-MM-DD_HH`) that `init_atmosphere` reads to build the +real-data `init.nc`. MPAS has no built-in ungrib, so this is done in Python with +**pywinter** — no need to build WPS/ungrib. + +``` +download_gfs.py → GFS GRIB2 → gfs_to_intermediate.py → GFS:YYYY-MM-DD_HH +download_era5.py → ERA5 GRIB → era5_to_intermediate.py → ERA5:YYYY-MM-DD_HH +``` + +Each source writes its own prefix — GFS → `GFS:`, ERA5 → `ERA5:` — which is the +`config_met_prefix` you set in `namelist.init_atmosphere`. The field names and the +rest of the setup are identical either way; only the prefix differs. +This directory only produces that **met intermediate file** — running +`init_atmosphere` with it (namelist, streams, linking `*.static.nc`, etc.) is covered +by the model usage tutorial, not repeated here. Run these after the static fields +are ready (see `../static_fields/`). + +## Which source? (data availability) + +Two products feed init_atmosphere: the **atmosphere** (3D state → `GFS:`/`ERA5:` +intermediate, init case 7) and the **SST/sea-ice** lower boundary (→ `SST:` +intermediate, init case 8, +see "SST / sea-ice update"). + +**Atmosphere:** + +| Source | Coverage | Resolution | Notes | +|---|---|---|---| +| GFS `--source aws` (default) | **2021-03-23 → present** | 0.25° | S3 byte-range subset, no rate limit | +| GFS `--source nomads` | **last ~10 days** | 0.25/0.5/1° | grib_filter; can hit `Over Rate Limit` | +| GFS `--source rda` | **2015-01-15 → present** | 0.25° | NCAR RDA ds084.1, full files; may need a free RDA account | +| ERA5 (`download_era5.py`) | **1940 → present** | 0.25° | Copernicus CDS; needs a CDS account | + +**SST / sea-ice:** + +| Source | Coverage | Notes | +|---|---|---| +| GFS (`gfs_sst_to_intermediate.py`, or `--fields sst`) | same as GFS above | from the GFS GRIB; analysis **or** forecast | +| OISST observed (`download_oisst.py`) | **1981-09-01 → present** | NOAA OI SST v2.1 daily 0.25° (NCEI) | +| OISST climatology (`oisst_clim_to_intermediate.py`) | **any date** (LTM 1991-2020) | day-of-year climatology via OPeNDAP; for forecasts / typical-month runs | + +### GFS is a *forecast* model — analysis vs forecast +GFS runs four times a day, at **00, 06, 12 and 18 UTC**. Each of those runs is a +**cycle**: `--cycle 00` selects the 00 UTC run of `--date`. Within a cycle, `--fhour` +is the **forecast hour** (lead time, in hours after the cycle time). + +A cycle produces an **analysis** at forecast hour 0 (`--fhour 0`, the default) and +**forecasts** at later hours (`--fhour 3, 6, … up to 384`, i.e. 16 days). So: +- **Analysis** (`--fhour 0`) = best estimate of the atmosphere *at the cycle time*. Use it + for the initial conditions. Only exists for the past (and the current cycle). +- **Forecast** (`--fhour > 0`) = a prediction for times *after* the cycle, i.e. it can give + data for the **future** (times that have not happened yet) — that is what makes an + operational MPAS forecast possible. + +By contrast **ERA5 (reanalysis) and OISST observed only exist for the past.** For the +future (or any date with no observed SST) use the **OISST climatology**. + +There is **no GFS 0.25° before 2015-01-15** — for anything older, use ERA5 (atmosphere) +and OISST (SST). `download_gfs.py` fails fast with the exact limit for out-of-range dates. + +## Scripts + +| Script | What it does | +|--------|--------------| +| `download_gfs.py` | Downloads the GFS fields/levels MPAS needs, with a `*.provenance.json`. Sources `aws` / `nomads` / `rda` (see table above). `--fields sst` for a small surface-only (SST update) download. | +| `gfs_to_intermediate.py` | Converts one GFS GRIB2 file to a WPS intermediate file via pywinter. Flips latitude to S→N, writes pressure in **Pa**, prints the level count for `config_nfglevels`. | +| `download_era5.py` | Downloads ERA5 pressure- and single-level GRIB via the CDS API (cdsapi), with provenance. For dates GFS can't reach. | +| `era5_to_intermediate.py` | ERA5 counterpart of the GFS converter (same intermediate field names; handles geopotential→height and snow units). | +| `download_oisst.py` | Downloads NOAA OISST v2.1 daily 0.25° SST/sea-ice NetCDF (NCEI) over a date range. For SST update files (scientific pipeline). | +| `oisst_to_intermediate.py` | Converts OISST daily files to `SST:` intermediate files (SST→K, SEAICE, LANDSEA) — one per day. | +| `oisst_clim_to_intermediate.py` | Builds `SST:` files from the OISST daily **climatology** (LTM 1991-2020, via OPeNDAP) for target dates with no observed SST (forecasts / typical-month runs). | +| `gfs_sst_to_intermediate.py` | Extracts SST/SEAICE/LANDSEA from a GFS GRIB into an `SST:` intermediate file (operational SST update). | + +One-shot wrappers (download + convert in one call, inside the conda env): + +| Wrapper | Covers | +|---------|--------| +| `prepare_gfs.sh` | GFS atmosphere and/or SST — `--product atm` (default), `sst`, or `both` (one download, both products). | +| `prepare_era5.sh` | ERA5 atmosphere (download pl+sl → `ERA5:` intermediate). | +| `prepare_era5_series.sh` | ERA5 `ERA5:` series over a date window (the LBC series of a month/year/years). Resumable, fault-tolerant, `nohup`-friendly, optional `--jobs`. | +| `prepare_oisst.sh` | OISST SST over a date range — observed (default) or `--climatology` (→ daily `SST:` intermediates). | + +## Dependencies + +The `cgfd-usp-mpas` conda env: `cfgrib` + `eccodes` (conda-forge) and `pywinter` +(pip); plus `cdsapi` (pip) for the ERA5 path. Run with +`conda run -n cgfd-usp-mpas python