ci: sync source mirror statically, without concretizing - #6248
Merged
Conversation
Re-adds the nightly source mirror sync from #6069 (reverted in #6245). Nothing about mirroring sources requires concretization: URLs and sha256 checksums for version tarballs, resources, and patches are all known statically from `package.py` files. The `find-missing-mirror-artifacts.py` script here: 1. lists every sha256-addressed artifact in the builtin repo: - version tarballs - resources - URL patches (in packages and from dependencies) while skipping manual-download and non-redistributable packages; 2. compares digests against a listing of the mirror's content-addressed `_source-cache/archive/` prefix; and 3. emits one TSV line per missing artifact with its: - `sha256` - exact mirror path, computed with Spack's `default_mirror_layout()` - its candidate URLs in order of preference. A GitHub Actions workflow then downloads each artifact with curl, verifies the sha256, and uploads artifacts to S3 one at a time. disk usage is bounded and individual fetch failures only skip that artifact until the next nightly run. Co-authored-by: Alec Scott <alec@llnl.gov> Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
Merge the download/verify/upload loop into the finder script and fetch with spack's own fetch strategies via Stage.fetch()/check(). This honors per-package fetch_options, reuses spack's checksum and redirect handling, and drops the TSV/bash handoff. Uploads still go through `aws s3 cp`, one artifact at a time, so disk stays bounded. Without --upload-to, the script just lists missing artifacts, which makes it easy to run locally. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Todd Gamblin <tgamblin@llnl.gov>
tgamblin
requested review from
alalazo,
alecbcs,
haampie and
kwryankrattiger
as code owners
August 27, 2026 06:27
alecbcs
approved these changes
Aug 27, 2026
alecbcs
left a comment
alecbcs
left a comment
Member
There was a problem hiding this comment.
Looks good to me. It took me a while to figure out that not including an upload_to flag also disables downloading and staging the packages as more of a dry run and only prints what's missing.
Overall I like the structure and the ability to upload the files within the stage's lifecycle to save on disk space consumed.
We might need to be careful that this now uses more of the Spack python SDK without being tied into Spack's unit tests so it's possible that we'll modify Spack and accidentally break this script.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-adds the nightly source mirror sync from #6069 (reverted in #6245).
Nothing about mirroring sources requires concretization: URLs and sha256 checksums for version tarballs, resources, and patches are all known statically from
package.pyfiles. Thesync-src-mirror.pyscript here:finds every sha256-addressed artifact in the builtin repo that is missing from the mirror, by comparing digests against a listing of the content-addressed
_source-cache/archive/prefix:while skipping manual-download and non-redistributable packages;
fetches each missing artifact with Spack's own fetch strategies, which honor per-package
fetch_options, fallback URLs, and checksum verification; anduploads each verified artifact to S3 with
aws s3 cp, one at a time, so disk usage stays bounded.Mirror paths are computed with Spack's own
default_mirror_layout(), so uploaded files are found byspack fetch. Individual fetch failures are tolerated and only skip that artifact until the next nightly run.This behaves like
spack mirror create --all, in that all resources and patches are included regardless of theirwhen=conditions -- the mirror should hold artifacts for every possible configuration.Parts of this should probably be integrated into
spack mirror createeventually; it lives in spack-packages for now becausespack mirror create --allstill concretizes when run from an environment, and it's hard to write an environment that fetches only artifacts for specific package versions without reconcretizing. Once that's fixed, this script can be replaced.The mirror is currently missing 587 artifacts, which this job will backfill on its first runs -- modulo dead URLs and stale checksums (e.g.
abduco@0.4/@0.5, whose autogenerated GitHub tarballs no longer match their declared sha256s). #6247 marks the packages whose sources we can't legally mirror so we don't retry them every night.