usp-utils: add MPAS install/build pipeline + guided setup documentation#12
Open
daniloceano wants to merge 16 commits into
Open
usp-utils: add MPAS install/build pipeline + guided setup documentation#12daniloceano wants to merge 16 commits into
daniloceano wants to merge 16 commits into
Conversation
Add a friendly, end-to-end install path under usp-utils/install/: - download_mpas_lib_sources.sh: fetch & verify the 6 library source tarballs (MPICH, zlib, HDF5, PnetCDF, NetCDF-C/Fortran) in the versions the build script expects; PnetCDF from the official Argonne mirror. - mpas_lib_install.sh: fix the build for gfortran >= 10 by adding -fallow-argument-mismatch to FFLAGS (MPICH's F77 check uses FFLAGS; without it MPICH never builds, cascading into "C compiler cannot create executables" in later libs that use the missing mpicc). - INSTALL_GUIDE.md: full recommended flow (conda -> julia -> source env -> build libs -> compile MPAS -> setup run dir), with the Julia LD_LIBRARY_PATH and Zeros caveats and a troubleshooting section. - .gitignore: keep machine-specific *.local.sh copies out of the repo. README.md: document the install/build flow and the copy-to-.local.sh pattern. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Use mpas_build_env.local.sh (git-ignored) consistently, sourced for both building and running. - Add mpirun launch commands for the cores and note that running atmosphere_model with no inputs aborts via MPI_Abort (expected). - Complete prerequisites: make, bzip2, tar/gzip. - Clarify that steps 4-6 (build/compile) don't require the conda/Julia envs. - Note the template is already patched for gfortran >= 10. - Document that PIO is optional (SMIOL fallback); NETCDF/PNETCDF always needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s optional Revert the -fallow-argument-mismatch change in the versioned template so it stays identical to the upstream NCAR script — users should try the original first. Document the fix instead, in a new "Known build issues (optional fixes)" section of INSTALL_GUIDE, to be applied in the user's .local.sh copy only if the MPICH build fails on gfortran >= 10. The note is explicit that the fix is compiler/OS-dependent and invites others to document alternative solutions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The install/build docs no longer hard-reference the workaround installer or UPSTREAM_ISSUE_zeros_compat.md (which live on the separate julia/zeros-compat-workaround branch). The Julia Step 2 now keeps the universal LD_LIBRARY_PATH guidance and points to that branch for the temporary Zeros resolve conflict, so this PR is self-contained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- setup_environment.sh: stop inferring env existence from 'conda activate's exit code (which gives a misleading "Couldn't find the environment" when the env exists but a transient activate returns non-zero). Now: ensure the 'conda' shell function is loaded (source conda.sh if only the binary is on PATH), check the env in 'conda env list', then activate. Clearer not-found message. - install_conda_environment.sh: add '#!/usr/bin/env bash' shebang. It uses bash-only features (BASH_SOURCE, '[ == ]'); running it with sh/dash failed with "Bad substitution" / "unexpected operator" and left SCRIPT_DIR empty. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a tool runs with HOME pointing at the repo, conda (cache+envs), shell history, and other per-user files get created here (.conda, .cache, .local, .config, .claude, .viminfo, ...). Ignore them so they don't pollute git status; the real ones live in $HOME and these copies are safe to delete. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- setup_environment.sh: stop inferring env existence from 'conda activate's exit code (which gives a misleading "Couldn't find the environment" when the env exists but a transient activate returns non-zero). Now: ensure the 'conda' shell function is loaded (source conda.sh if only the binary is on PATH), check the env in 'conda env list', then activate. Clearer not-found message. - install_conda_environment.sh: add '#!/usr/bin/env bash' shebang. It uses bash-only features (BASH_SOURCE, '[ == ]'); running it with sh/dash failed with "Bad substitution" / "unexpected operator" and left SCRIPT_DIR empty. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MPAS v8.4 miscompiles with gfortran 13 (builds, crashes at run time); gfortran 12 (the NCAR tutorial's version) produces a working model. Add mpas_lib_install_gcc12.sh — a gfortran-12 variant of the lib-install template (conda-forge gcc12 wrappers, separate libs-gcc12 prefix, generic $HOME paths) — and document the gcc12 toolchain in INSTALL_GUIDE (conda env, build steps, and the gcc12 build-environment block). Personal *.local.sh copies stay git-ignored. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rework the MPAS dependency install per F. Barbosa's review of the install PR: MPAS v8 uses its bundled SMIOL I/O layer whenever PIO is unset, so PnetCDF is the only required external I/O dependency. Drop PIO, NetCDF-C/Fortran, HDF5 and zlib entirely. - mpas_lib_install.sh: build PnetCDF only; MPICH is now optional (BUILD_MPICH=1), fetched from the official mpich.org source instead of the NCAR mirror. Prefer the system / HPC module MPI. - mpas_build_env.sh: new; sets PNETCDF and unsets PIO/NETCDF so the SMIOL path is selected. Replaces the old NETCDF/PIO/HDF5 build env. - README.md: rationale, quick start, config table, and a note that `make gnu` vs `make gfortran` differ only in two compile-time diagnostic flags (-std=f2008 -fimplicit-none) and produce numerically identical binaries. - Remove the obsolete heavy-stack scripts (download_mpas_lib_sources.sh, mpas_lib_install_gcc12.sh) and INSTALL_GUIDE.md. - gitignore *.local.sh at the repo root too. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…compile) The README only covered sourcing mpas_build_env.sh before 'make' (compile). It never stated the same env must be sourced to RUN init_atmosphere_model / atmosphere_model — without it mpiexec/mpirun aren't on PATH and the runtime libs aren't on LD_LIBRARY_PATH. Add a 'Step 4 — run the model' section with the mpiexec invocation, the 'command not found' symptom, and a branch caveat: this script lives only on setup/installation, so keep a branch-independent copy at ~/mpas-build/mpas_build_env.sh and source that regardless of checked-out branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # .gitignore
Bring the install + environment work to a single coherent state on top of the current develop (8.4.1): - Move install_conda_environment.sh and install_julia_environment.jl into install/ (conda yml path -> ../libs); setup_environment.sh stays at the usp-utils root as the per-session activation script. - install/README.md: keep the "run the model" step, add an environment_setup.md pointer at the top, and drop the now-obsolete branch caveat. - install/environment_setup.md: Python (conda) + Julia toolbox setup guide, with the env -u LD_LIBRARY_PATH note. The Julia env now installs straight from favba's upstream packages (the Zeros compat workaround is no longer needed). - usp-utils/README.md: general overview with install-first ordering and a folder map; drop the broken pre_proc/static_fields link (that belongs to the preproc PR). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a complete, documented installation and build pipeline for MPAS under
usp-utils/install/.developcurrently has no build toolchain and no builddocumentation; this PR introduces a lean, reproducible one plus step-by-step
guides.
What this PR adds
Build pipeline (new
usp-utils/install/):mpas_lib_install.sh— builds the library dependencies. It is lean bydesign: MPAS v8 uses its bundled SMIOL I/O layer when
PIOis unset, so theonly external I/O dependency is PnetCDF. MPI comes from the system / HPC
module stack; MPICH can optionally be built from the official mpich.org source
with
BUILD_MPICH=1.mpas_build_env.sh— the environment to source beforemakeand beforerunning the model (sets
PNETCDF, puts the MPI wrappers and runtime librarieson
PATH/LD_LIBRARY_PATH).Guided documentation:
install/README.md— step by step: detect whether you already have MPI →build the libraries → compile → run the model. Includes a
make gnuvsmake gfortrannote and a common-errors section.install/environment_setup.md— sets up the Python (conda) + Juliapre/post-processing toolbox and explains
setup_environment.sh.Reorganization:
install_conda_environment.shandinstall_julia_environment.jlfrom theusp-utils/root intoinstall/(all installers grouped together);setup_environment.shstays at the root as the per-session activation script.usp-utils/README.mdrewritten as a general overview: install-first ordering,folder map, and where runs/data go (
runs/,grids/,met_data/— git-ignored).*.local.shcopies.Design notes
The lean PnetCDF-only / system-MPI /
make gnuchoices follow @favba'srecommendations. The Julia environment installs directly from @favba's upstream
packages (no temporary
Zeroscompat workaround needed).Scope
Installation / environment setup only. Pre-processing (GFS/ERA5/OISST, static
fields) and post-processing (plotting) ship as separate PRs.
Testing
Built and ran
init_atmosphere_model/atmosphere_model(MPAS v8.4.1) on aplain workstation with
BUILD_MPICH=1+ system GCC 13, followinginstall/README.md.