Automate ingest and phylogenetic workflows - #22
Merged
Conversation
Currently just runs the ingest workflow and uploads the results to AWS S3. Subsequent commits will add automation for the phylogenetic workflow. Follows Zika PR #52 nextstrain/zika@d44f2ae
The phylogenetic workflow will run after the ingest workflow has completed successfully to use the latest available data. Subsequent commits will check if the ingest results included new data to only run the phylogenetic workflow when there's new data. Following Zika PR #52 nextstrain/zika@2c415e7
Uses GitHub Actions cache to store a file that contains the `Metadata.sh256sum` of the ingest files on S3 and use the `hashFiles` function to create a unique cache key. Then the existence of the cache key is an indicator that the ingest file contents have not been updated since a previous run on GH Actions. This does come with a big caveat that GH will remove any cache entries that have not been accessed in over 7 days.¹ If the workflow is not being automatically run within 7 days, then it will always run the phylogenetic job. If this works well, then we may want to consider moving this within the `pathogen-repo-build` reusable workflow to have the same functionality across pathogen automation workflows. ¹ https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy Follows Zika PR #52 nextstrain/zika@eb5e76d
Add individuals inputs per workflow to override the default Docker image used by `nextstrain build`. Having this input has been extremely helpful to continue running pathogen workflows when we run into new bugs that are not present in older nextstrain-base images. There are separate image inputs for the two workflows because they use different tools and may require different versions of images. Follows Zika PR #52 nextstrain/zika@65a8acc
Copied daily schedule of mpox ingest https://github.com/nextstrain/mpox/blob/e439235ff1c1d66e7285b774e9536e2896d9cd2f/.github/workflows/fetch-and-ingest.yaml#L4-L21 Daily runs seem fine since the ingest workflow currently takes less than 2 minutes to complete and it will not trigger the phylogenetic workflow if there's no new data. We can bring this down to once a week if it seems like overkill. Follows Zika PR #52 nextstrain/zika@77ca1d4
kimandrews
marked this pull request as ready for review
April 6, 2024 00:15
j23414
approved these changes
Apr 6, 2024
j23414
left a comment
j23414
left a comment
Contributor
There was a problem hiding this comment.
Looks reasonable to me, as far as I can tell
joverlee521
requested changes
Apr 8, 2024
Contributor
There was a problem hiding this comment.
Changes look good to me 👍
A couple things to follow up on before merging:
- drop the
pull_requesttrigger added in cbcb70e - include hash fixes added in nextstrain/zika#54
Follows Zika PR #52 nextstrain/zika@f615170 Uses the variable `AWS_DEFAULT_REGION` that was added to the Nextstrain GitHub organization variables.¹ ¹ https://github.com/organizations/nextstrain/settings/variables/actions
kimandrews
force-pushed
the
automate-workflows
branch
from
April 10, 2024 00:15
0b8a78e to
671df77
Compare
Contributor
|
The measles workflow is now showing up on the pathogen workflow status page
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of proposed changes
Follows zika PR: nextstrain/zika#52
Adds a single GH Action workflow to automate the ingest and phylogenetic workflows, set to run daily at the same time as the automated mpox ingest.
Uses GH Action caches to store hash of ingest results' Metadata.sha256sum values added to the S3 metadata within upload-to-s3. If the cache contains a match from previous runs of the GH Action workflow, then the workflow will skip the phylogenetic job.
Related issue(s)
Based on discussion in nextstrain/pathogen-repo-guide#25
nextstrain/zika#52
nextstrain/dengue#38
Checklist