ci: add nightly job to sync source mirror with pkgs in builtin - #6069
Merged
Conversation
Signed-off-by: Alec Scott <alec@llnl.gov>
alecbcs
requested review from
alalazo,
haampie and
kwryankrattiger
as code owners
August 12, 2026 22:22
scheibelp
requested changes
Aug 13, 2026
scheibelp
left a comment
scheibelp
left a comment
Member
There was a problem hiding this comment.
Other points (not requests for this PR):
- (request for later) If a patch/resource is dropped, this doesn't catch that: if we already placed the "main" version tarball in a mirror, and a resource was missed, then this job will also miss that (I see though that patches/resources generally are included, just not in this rare case). I think this can be addressed later.
- (not a request) What about wrong checksums (i.e. if someone specifies the wrong checksum for a version)? I think the behavior would be OK: this script will collect every wrong checksum, download the version (which should fail, but
spack mirrorgenerally proceeds on in this case). We'd want to clean those up though.
Signed-off-by: Alec Scott <alec@llnl.gov>
Signed-off-by: Alec Scott <alec@llnl.gov>
tgamblin
self-requested a review
August 26, 2026 16:54
Member
|
@scheibelp are your review comments addressed? |
scheibelp
approved these changes
Aug 26, 2026
tgamblin
approved these changes
Aug 26, 2026
tgamblin
added a commit
that referenced
this pull request
Aug 27, 2026
Re-adds the nightly source mirror sync from #6069 (reverted in #6245). Nothing about mirroring sources requires concretization: URLs and sha256 checksums for version tarballs, resources, and patches are all known statically from `package.py` files. The `find-missing-mirror-artifacts.py` script here: 1. lists every sha256-addressed artifact in the builtin repo: - version tarballs - resources - URL patches (in packages and from dependencies) while skipping manual-download and non-redistributable packages; 2. compares digests against a listing of the mirror's content-addressed `_source-cache/archive/` prefix; and 3. emits one TSV line per missing artifact with its: - `sha256` - exact mirror path, computed with Spack's `default_mirror_layout()` - its candidate URLs in order of preference. A GitHub Actions workflow then downloads each artifact with curl, verifies the sha256, and uploads artifacts to S3 one at a time. disk usage is bounded and individual fetch failures only skip that artifact until the next nightly run. Co-authored-by: Alec Scott <alec@llnl.gov> Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Todd Gamblin <tgamblin@llnl.gov> * sync-src-mirror: fetch with spack instead of curl Merge the download/verify/upload loop into the finder script and fetch with spack's own fetch strategies via Stage.fetch()/check(). This honors per-package fetch_options, reuses spack's checksum and redirect handling, and drops the TSV/bash handoff. Uploads still go through `aws s3 cp`, one artifact at a time, so disk stays bounded. Without --upload-to, the script just lists missing artifacts, which makes it easy to run locally. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Todd Gamblin <tgamblin@llnl.gov> --------- Signed-off-by: Todd Gamblin <tgamblin@llnl.gov> Co-authored-by: Alec Scott <alec@llnl.gov>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a minimal nightly GitHub Actions CI/CD job that compares the state of the package repository to the mirrored source cache and mirrors any artifacts that are missing from the cache. (As of the time of posting there are 439 missing package versions that would be uploaded by this workflow.)