Add remaining <compiler>-<backend>-env recipes currently used

[bartgol]

This completes the recipes for all the images currently used for ghci-snl testing by E3SM.

[Copilot]

Pull request overview

This PR completes the set of ghci-snl final environment image recipes (compiler/backend env combinations) used by E3SM’s ghci-snl GitHub Actions testing, and updates the CI workflow to build/publish them.

Changes:

• Add new final env Docker recipes for intel-cpu-env and gnu-cuda-env (including an NVIDIA repo file for CUDA drivers).
• Update the ghci-snl GitHub Actions workflow to build final env images from the corresponding compiler images and tag/publish them.
• Normalize the BASE_TAG default in the existing gnu-cpu-env recipe.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
ghci-snl/final/intel-cpu-env/Dockerfile	New Intel CPU final environment image (Spack libs + PyTorch).
ghci-snl/final/gnu-cuda-env/Dockerfile	New GNU CUDA final environment image (NVIDIA drivers + CUDA + MPI + Python pkgs).
ghci-snl/final/gnu-cuda-env/cuda-rhel9.repo	Adds NVIDIA CUDA repo configuration for RHEL9.
ghci-snl/final/gnu-cpu-env/Dockerfile	Adjusts the default BASE_TAG build arg value.
.github/workflows/ghci-snl.yaml	Extends CI to build/publish the new final env images and updates tag logic.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Suppressed comments (3)

ghci-snl/final/gnu-cpu-env/Dockerfile:70

• The leading . sources a file named git; it does not execute the Git command. With /usr/bin/git on PATH, Bash attempts to parse that binary as a shell script, so this image build fails before the CIME submodule is initialized. The profile was already sourced on line 68, so invoke git directly here.
  .github/workflows/ghci-snl.yaml:191
• This metadata is also applied to gnu-cuda-env, so its published OCI description incorrectly identifies the GPU/CUDA image as CPU-only. Use a generic description or derive the target from the matrix.

            org.opencontainers.image.description=${{ matrix.tag }} image for E3SM ghci-snl testing on CPU on GitHub actions

.github/workflows/ghci-snl.yaml:202

• On a ghci-snl-* tag push, the compiler job publishes a suffixed image (for example gnu-13.3.0-v1), but this expression always pulls the unsuffixed gnu-13.3.0 because it only distinguishes pull requests. Consequently, tagged final images are built from the previously published compiler image rather than the compiler image produced by the same tagged run. Tag handling must be propagated consistently through the base, compiler, and final build arguments.

            BASE_TAG=${{ matrix.compiler }}-${{ matrix.version }}${{ github.event_name == 'pull_request' && format('-pr-{0}', github.event.pull_request.number) || '' }}

[mahf708]

@copilot could you please review the logs in this PR and comment here on why they're failing and what could be the fix? Review the PR in light of the logs and code, not just code edits.

[Copilot]

@copilot could you please review the logs in this PR and comment here on why they're failing and what could be the fix? Review the PR in light of the logs and code, not just code...

Looked at the latest failing run (33224443413, commit 2f4375e). Two final-env jobs fail for unrelated reasons:

final-env (gnu, gnu-cpu-env) fails building MOAB:

RUN . /etc/profile.d/eamxx-env.sh && \
    git clone https://github.com/e3sm-project/e3sm /tmp/e3sm && \
    cd /tmp/e3sm && . git submodule update --init cime

/bin/bash: line 1: .: /usr/bin/git: cannot execute binary file (exit 126) — the . before git submodule is parsed as bash's source builtin, so it tries to source the git binary instead of running it. Should just be git submodule update --init cime (drop the leading .).

final-env (gnu, gnu-cuda-env) fails during pip install numpy cupy:

-------- Configuring Module: jitify --------
/usr/bin/ld: cannot find -lcuda
...
Exception: Your CUDA environment is invalid.

jitify is a required cupy module and its build probe needs to link -lcuda (the CUDA driver lib, not the toolkit runtime libs). Since Try to do away with cuda repo dropped the NVIDIA driver package install, there's no libcuda.so on the linker path anymore. Spack's cuda package does ship a stub libcuda.so for build-without-a-GPU (at <cuda prefix>/lib64/stubs), but cupy's setup only searches lib64/lib, not lib64/stubs. Fix is to point the linker at that stub dir for the pip install step, e.g.:

RUN source /etc/profile.d/modules.sh && \
    source /etc/profile.d/eamxx-env.sh && \
    LIBRARY_PATH="$CUDA_ROOT/lib64/stubs:$LIBRARY_PATH" \
    python3 -m pip install numpy cupy

(CUDA_ROOT is the env var Spack's lmod module sets for cuda, same convention as HDF5_ROOT/NETCDF_C_ROOT already used elsewhere in this recipe). No need to reinstall the NVIDIA repo/driver package. Just note the stubs dir is build-time only and shouldn't be added to LD_LIBRARY_PATH at runtime.

[mahf708]

@copilot, review the cuda-env error in the logs again please and suggest a solution.

[Copilot]

@copilot, review the cuda-env error in the logs again please and suggest a solution.

Looked at the latest run (33267785759, commit b4f2cfb). final-env (gnu, gnu-cuda-env) still fails at the same spot:

-------- Configuring Module: jitify --------
/usr/bin/ld: cannot find -lcuda
...
Exception: Your CUDA environment is invalid.

The log shows why the previous fix didn't help:

LIBRARY_PATH    : /lib64/stubs:

$CUDA_ROOT was empty — I was wrong that Spack sets that variable for cuda. Unlike hdf5/netcdf-c/boost/etc., there's no cuda: entry in modules.yaml to generate a CUDA_ROOT var. Spack's cuda package instead sets CUDA_HOME directly via its own setup_run_environment (pointing at the install prefix), so that's the variable that's actually available after module load cuda.

Fixed in a new commit by using $CUDA_HOME instead:

LIBRARY_PATH="$CUDA_HOME/lib64/stubs:$LIBRARY_PATH" \
    python3 -m pip install numpy cupy

This keeps the fix scoped to the gnu-cuda-env recipe only, rather than touching the base image's modules.yaml (which would invalidate build cache for every other final env image too).

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments....

Fixed the missing trailing separator: prefix=${{ matrix.tag }}- now matches the base/compiler jobs' format.

Re: "also appears on line 202" — that line is the BASE_TAG build-arg (${{ matrix.compiler }}-${{ matrix.version }}${{ ... }}), not a docker/metadata-action prefix. It already concatenates correctly: there's a literal - between compiler and version, and the PR ternary's format('-pr-{0}', ...) supplies its own leading -, so no separator is missing there. Left it unchanged.

[mahf708]

@bartgol I merged this because I was excited, but note a piece of minor feedback: we could utilize preexiting spack cache instead of rebuidling every little thing. Building cmake alone is a total monster of a job. :)

Example of how to do it, but there are others, and this could easily be an issue for claudio

containers/ghci/Dockerfile

Lines 80 to 96 in a54261d

	# TODO: do these intervene with each other?
	# https://cache.spack.io/tag/v0.23.0/
	RUN spack mirror add v0.23.0 https://binaries.spack.io/v0.23.0
	RUN spack buildcache keys --install --trust
	# https://cache.spack.io/tag/v0.23.0/?stack=e4s
	RUN spack mirror add v0.23.0-e4s https://binaries.spack.io/v0.23.0/e4s
	RUN spack buildcache keys --install --trust
	# https://cache.spack.io/tag/v0.23.0/?stack=root#
	RUN spack mirror add v0.23.0-root https://binaries.spack.io/v0.23.0/root
	RUN spack buildcache keys --install --trust
	# TODO: add e4s inventory when they support ubuntu 22.04+
	# # https://oaciss.uoregon.edu/e4s/inventory.html
	# RUN spack mirror add E4S-24.0 https://cache.e4s.io/24.0
	# RUN spack buildcache keys -it
	# RUN spack mirror add E4S https://cache.e4s.io
	# RUN spack buildcache keys -it