feat(docker): migrate from Spack to Pixi for container dependency management - #5025
Merged
Conversation
Contributor
|
You can preview documentation at https://esmci.github.io/cime/branch/jasonb5/docker/pixi-migration/html/index.html |
- Add strict error handling with 'set -Eeuo pipefail' - Validate CIME_MODEL must be 'e3sm' or 'cesm' (lowercase) - Verify Pixi environment directory exists before activation - Make /etc writes conditional for non-root/read-only container support - Prepend to PKG_CONFIG_PATH and LD_LIBRARY_PATH instead of overwriting - Initialize 'cores' variable to fix unbound variable error with set -u These changes ensure the entrypoint fails fast with clear error messages and supports running as non-root or in read-only mode.
- Pin Ubuntu 24.04 base image by digest (sha256:561618e2c...) - Pin uv to version 0.12.3 with SHA256 checksum verification - Fix tar extraction using --strip-components=1 to handle directory structure - Replace unpinned astral.sh installer with direct GitHub release download This ensures reproducible builds and validates dependencies against known checksums before installation.
- Accurately document that all components concurrently execute on same ranks - Update comment to reflect overlapping layout vs misleading 'one per component' - Remove trailing whitespace from XML and documentation The concurrent PE layout is valid for CIME and allows all components to share the available MPI tasks, preventing oversubscription in containers.
Docker README updates: - Clarify CIME_MODEL must be 'e3sm' or 'cesm' (lowercase) - Add 'Required flags' section explaining --hostname, --shm-size, and -e - Update all examples to include required --shm-size=1g flag - Add 'Running without a shell' section with pytest and create_test examples Contributing guide updates: - Fix OpenMPI → MPICH reference and add pixi/conda-forge mention - Add BuildKit requirement note - Update all examples with --shm-size=1g and concrete 'e3sm' values - Add --pesfile to create_test example - Add troubleshooting section for common issues: * CIME_MODEL validation errors * Out of memory errors * Container core count mismatches - Add reference to docker/README.md for complete documentation All examples are now complete, accurate, and copy-pasteable.
Query configuration examples:
- Update mpilibs from 'openmpi' to 'mpich'
- Update compilers from 'gnu,gnuX' to 'gnu'
- Update environment variables to show Pixi paths
- Remove obsolete OpenMPI-specific variables
- Update core counts to realistic values (4 instead of 8)
Machine configuration examples:
- Update example XML to use mpich mpilib
- Change mpirun arguments from --oversubscribe to -bind-to none
- Update paths from /storage to /root/storage
- Use $ENV{DOCKER_MAX_TASKS} for dynamic task limits
- Update environment variables to use $ENV{PIXI_ENV_ROOT} and $ENV{CIME_MODEL}
- Remove OpenMPI-specific environment variables
- Update support contact
These examples now accurately reflect the current Pixi/MPICH-based
container implementation.
The entrypoint was sourcing /root/.local/bin/env which no longer exists after the uv migration. The Dockerfile now installs uv directly to /usr/local/bin and creates a venv at ~/.venv, so only the venv activation is needed. This fixes a critical bug that breaks normal container startup.
The examples at lines 124 and 133 were missing required flags: - --hostname docker (required by NODENAME_REGEX validation) - -e CIME_MODEL=e3sm (required by entrypoint environment setup) - --shm-size=1g (required for MPI shared memory) Without these flags, the entrypoint validation fails and containers don't start properly.
Changed :ref: to backtick link syntax for the docker/README.md reference. The :ref: directive is for internal Sphinx cross-references, not external GitHub URLs. This fixes a Sphinx build error.
Excludes .git directory, untracked data/ files (~442 MB), build artifacts, and development files from the Docker build context. This significantly reduces build context size and speeds up image builds.
Pin pytest==8.4.2, pytest-cov==7.1.0, and evv4esm==0.6.2 to ensure reproducible test environments across the CI Python matrix (3.9-3.14). pytest 9.x requires Python 3.10+, so 8.4.2 is the latest version compatible with Python 3.9.
Remove the gnuX compiler option from Docker machine XML files. Only the gnu compiler is supported and tested in the container environment. Also prepend Pixi library paths to LD_LIBRARY_PATH and PKG_CONFIG_PATH instead of replacing them. This preserves any library paths set by the caller or entrypoint while ensuring Pixi dependencies take precedence.
Add explicit documentation that the container is Linux amd64 only and runs as root with storage under /root/storage. Update all build examples to include --platform linux/amd64 flag for clarity and consistency.
Major changes: - Build local Docker images for PRs, saved as compressed artifacts - Convert test jobs from GitHub container jobs to regular runners using docker run - Only publish to ghcr.io/esmci/cime:latest on pushes to master - Remove QEMU setup (amd64-only builds) - Expand path filters to include workflow config, dependencies, and .dockerignore This ensures fork PRs receive full container test coverage without requiring GHCR push permissions, and eliminates the stale-latest fallback behavior that could mask test failures.
Add data/ to .gitignore to prevent untracked test data from being committed to the repository. The .dockerignore already excludes this directory from Docker build contexts.
- Fix entrypoint.sh CIME_MODEL check to handle unset var with set -u - Make test jobs depend on successful build-pr-image result - Add fork-safe condition to Codecov uploads (same-repo only) - Fix tmate debug condition from !always() to failure() && !cancelled() - Make tar archive robust with --ignore-failed-read for missing dirs
jasonb5
force-pushed
the
jasonb5/docker/pixi-migration
branch
from
August 13, 2026 20:28
0926e9f to
333412c
Compare
GitHub Actions runners cannot reach portal.nersc.gov, causing Docker image builds to fail during the RUN step that downloads input data. Changes: - Add DOWNLOAD_SKIP_ON_ERROR flag to make download failures non-fatal during Docker build (when NERSC is unreachable from CI) - Add wget retry logic (--tries=5, --timeout=30, --waitretry=10) - Skip downloading files that already exist (idempotent) - Attempt to download missing files in background at container startup - Add .download_complete marker to track successful downloads - Clean up partial downloads on failure The image will now build successfully in CI even when NERSC is unreachable. Missing files will be retried at runtime when the container starts (e.g., when run locally where NERSC is accessible).
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates the project’s CI/development container from a Spack-based stack to Pixi/conda-forge, splitting dependencies into E3SM and CESM environments while updating container runtime behavior and GitHub Actions to build/test PR images locally and publish images from master.
Changes:
- Replaces Spack with Pixi (with separate
e3smandcesmenvironments) and updates container entrypoint/runtime configuration (CPU-aware PE sizing, MPICH, UCX settings). - Refactors Docker image build to use pinned
pixianduvbinaries with checksum verification; updates container docs and Sphinx docs accordingly. - Updates CI workflow to build a PR image, pass it to downstream jobs, and only publish
lateston pushes.
Reviewed changes
Copilot reviewed 15 out of 17 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
test-requirements.txt |
Pins Python test tooling versions for reproducible CI/containers. |
docker/spack.yaml |
Removes legacy Spack environment definition. |
docker/README.md |
Updates container documentation for Pixi-based environments and new runtime expectations. |
docker/pixi.toml |
Defines shared conda-forge dependency set plus E3SM/CESM feature envs. |
docker/pixi.lock |
Locks full conda-forge resolution for reproducible container builds. |
docker/entrypoint.sh |
Selects Pixi env by CIME_MODEL, computes cores for PE sizing, and manages runtime behaviors (inputdata, paths). |
docker/Dockerfile |
Builds Pixi envs in-image, builds cprnc, installs uv + Python test env, and stages input data. |
docker/.cime/docker/config_machines.xml |
Switches Docker machine config to MPICH and dynamic task sizing via DOCKER_MAX_TASKS. |
docker/.cime/docker.cmake |
Points include/lib flags at the model-selected Pixi environment. |
docker/.cime/config_pes.xml |
Adds a generic “cap-to-available-cores” PE layout for container runs. |
docker/.cime/config_machines.v2.xml |
Updates v2 machine config to MPICH and dynamic core/task sizing. |
doc/source/contributing-guide.rst |
Reworks contributing doc and updates container usage instructions to match new image behavior. |
doc/source/ccs/query-configuration.rst |
Updates documentation examples for the Docker machine config changes. |
doc/source/ccs/model-configuration/variables/machine.rst |
Updates example machine XML snippet to match MPICH + Pixi environment approach. |
.gitignore |
Ignores local data and Pixi state directories. |
.github/workflows/testing.yml |
Builds PR image as an artifact, runs tests by docker run, and publishes latest only on push. |
.dockerignore |
Excludes VCS, artifacts, and local dev state from Docker build context. |
Suppressed comments (3)
.github/workflows/testing.yml:221
- Same as above: this
docker loadreads a gzipped stream from stdin without decompression, which can break the job. Decompress before loading.
- name: Load Docker image
run: |
docker load < /tmp/cime-image.tar.gz
.github/workflows/testing.yml:93
- These Docker actions are still on older major versions. To avoid breakage when GitHub Actions updates Node runtimes, bump
setup-buildx-actionandlogin-actionto their current majors.
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to GHCR
uses: docker/login-action@v3
.github/workflows/testing.yml:100
docker/build-push-actionis still on v6 here. Consider bumping to the current major (v7) for Node runtime compatibility with GitHub Actions.
- name: Build and push
uses: docker/build-push-action@v6
with:
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Remove obsolete reference to legacy/spack.yaml in docker/README.md - Export PIXI_ENV_ROOT in entrypoint.sh for child process visibility - Fix silent background download to redirect both stdout and stderr - Remove trailing whitespace in contributing-guide.rst - Restore contributing-guide-running-tests reference target for Sphinx - Fix docker load commands to decompress gzipped tar before loading - Docker action versions already at latest (v3/v6), no update needed
Contributor
|
It seems like this PR does a few very different things. To help me in reviewing, could you please break it up into discrete PRs? |
jgfouca
approved these changes
Aug 21, 2026
jgfouca
left a comment
jgfouca
left a comment
Contributor
There was a problem hiding this comment.
LGTM but can you explain the reasoning behind this change a bit more in the description?
Contributor
|
Oh, this actually gets E3SM working again! Definitely would be nice to merge this ASAP. |
Collaborator
Author
|
@samsrabin That’s fair. The changes touch several areas, but they’re all part of the same Spack-to-Pixi migration and are fairly dependent on each other. Splitting them up would leave some of the intermediate PRs incomplete or non-functional. I can make that relationship clearer in the PR description and clean up the commits to make it easier to review. |
Contributor
|
Ah okay, sorry, I didn't understand that—makes sense now! Do you still need a review now that it's merged? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace Spack with Pixi for CIME Docker dependency management, with separate E3SM and CESM environments. Update the container runtime configuration and CI workflow to build and test PR images locally, while publishing images from
master.Notable changes
CIME_MODEL=e3smorCIME_MODEL=cesm.Testing