Automate ingest and phylogenetic workflows - #38
Merged
Conversation
Currently just runs the ingest workflow and uploads the results to AWS S3. Subsequent commits will add automation for the phylogenetic workflow. Copied commit from Zika PR #52 nextstrain/zika@d44f2ae
The phylogenetic workflow will run after the ingest workflow has completed successfully to use the latest available data. Subsequent commits will check if the ingest results included new data to only run the phylogenetic workflow when there's new data. Copied commit from Zika PR #52 nextstrain/zika@2c415e7
Uses GitHub Actions cache to store a file that contains the `Metadata.sh256sum` of the ingest files on S3 and use the `hashFiles` function to create a unique cache key. Then the existence of the cache key is an indicator that the ingest file contents have not been updated since a previous run on GH Actions. This does come with a big caveat that GH will remove any cache entries that have not been accessed in over 7 days.¹ If the workflow is not being automatically run within 7 days, then it will always run the phylogenetic job. If this works well, then we may want to consider moving this within the `pathogen-repo-build` reusable workflow to have the same functionality across pathogen automation workflows. ¹ https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy Copied commit from Zika PR #52 nextstrain/zika@eb5e76d
Add individuals inputs per workflow to override the default Docker image used by `nextstrain build`. Having this input has been extremely helpful to continue running pathogen workflows when we run into new bugs that are not present in older nextstrain-base images. There are separate image inputs for the two workflows because they use different tools and may require different versions of images. Copied commit from Zika PR #52 nextstrain/zika@65a8acc
Copied daily schedule of mpox ingest https://github.com/nextstrain/mpox/blob/e439235ff1c1d66e7285b774e9536e2896d9cd2f/.github/workflows/fetch-and-ingest.yaml#L4-L21 Daily runs seem fine since the ingest workflow currently takes less than 2 minutes to complete and it will not trigger the phylogenetic workflow if there's no new data. We can bring this down to once a week if it seems like overkill. Copied commit from Zika PR #52 nextstrain/zika@77ca1d4
j23414
force-pushed
the
automate-workflows
branch
from
April 5, 2024 15:26
642a310 to
795546d
Compare
Contributor
|
Yay, the test run completed with plenty of time to spare for the phylogenetic workflow 🎉
I know this will be fixed in #18, but I would just update the phylo outputs with the hardcoded |
Contributor
|
FYI, after the initial trigger with |
2 tasks
joverlee521
requested changes
Apr 8, 2024
Contributor
There was a problem hiding this comment.
Changes look good to me 👍
A couple things to follow up on before merging:
- drop the
pull_requesttriggers added in 5daf8b7 - include hash fixes added in nextstrain/zika#54
Copied fix from zika, guided by @joverlee521 nextstrain/zika@cdd071e
Uses the variable `AWS_DEFAULT_REGION` that @joverlee521 added to the Nextstrain GitHub organization variables.¹ ¹ https://github.com/organizations/nextstrain/settings/variables/actions Copied fix from zika, guided by @joverlee521 nextstrain/zika@cdd071e
j23414
force-pushed
the
automate-workflows
branch
2 times, most recently
from
April 9, 2024 23:23
9839842 to
02504d6
Compare
Contributor
|
The dengue workflow is now showing up on the pathogen workflow status page
|
1 task
joverlee521
added a commit
that referenced
this pull request
Apr 11, 2024
I missed this in review of #38. The phylogenetic workflow was still pulling from old S3 URLs and not the ingest workflow output data. This commit corrects the S3 URL to the ingest output files and updates the `strain_id_field` config param to use the appropriate ID column from the ingest output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of proposed changes
Coordinated with @joverlee521 to copy commits from zika PR: nextstrain/zika#52
Adds a single GH Action workflow to automate the ingest and phylogenetic workflows, set to run daily at the same time as the automated mpox ingest.
Uses GH Action caches to store hash of ingest results'
Metadata.sha256sumvalues added to the S3 metadata within upload-to-s3. If the cache contains a match from previous runs of the GH Action workflow, then the workflow will skip the phylogenetic job.See commits for details.
Related issue(s)
Based on discussion in nextstrain/pathogen-repo-guide#25
Checklist
The manual run completed successfully although does not push to the live site since output files do not have "_genome" postfixs in the filenames: