Skip to content

Update zika instructions #147

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
j23414 merged 3 commits into from
Nov 29, 2023
Merged

Update zika instructions #147

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
34236c4
Update zika instructions
j23414 Nov 21, 2023
68bda7d
fixup: github markdown note
j23414 Nov 28, 2023
983706e
fixup: reflect reality
j23414 Nov 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 24 additions & 71 deletions builds/ZIKA.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,93 +1,46 @@
# ZIKA Pipeline Notes

## Setup
## Ingest data from NCBI GenBank

1. Make sure environment variables for connecting to fauna are set.

## Upload via ViPR and update citations

### [ViPR sequences](https://www.viprbrc.org/brc/vipr_genome_search.spg?method=ShowCleanSearch&decorator=flavi_zika)

1. Download sequences
* Select year >= 2013 and genome length >= 5000
* Download as Genome Fasta
* Set Custom Format Fields to 0: GenBank Accession, 1: Strain Name, 2: Segment, 3: Date, 4: Host, 5: Country, 6: Subtype, 7: Virus Species
* May also use the [ViPR API](https://www.viprbrc.org/brc/staticContent.spg?decorator=reo&type=ViprInfo&subtype=API)

```
curl "https://www.viprbrc.org/brc/api/sequence?datatype=genome&family=flavi&species=Zika%20virus&fromyear=2013&minlength=5000&metadata=genbank,strainname,segment,date,host,country,genotype,species&output=fasta" |\
tr '-' '_' |\
tr ' ' '_' |\
sed 's:N/A:NA:g' >\
GenomicFastaResults.fasta
```

The search-and-replace commands (`tr`, `sed`) are necessary because the API downloads fasta headers similar to:

`>KY241742|ZIKV_SG_072|N/A|2016-08-28|Human|Singapore|Asian|Zika virus`

but need to match the GUI downloaded headers similar to:

`>KY241742|ZIKV_SG_072|NA|2016_08_28|Human|Singapore|Asian|Zika_virus`


2. Move downloaded sequences to `fauna/data`
3. Extract `GenomicFastaResults.tar.gz` and rename the extracted file to `GenomicFastaResults.fasta`
4. Upload to vdb database
* `python3 vdb/zika_upload.py -db vdb -v zika --source genbank --locus genome --fname GenomicFastaResults.fasta`

### Update

* Update citation fields
* `python3 vdb/zika_update.py -db vdb -v zika --update_citations`
* updates `authors`, `title`, `url`, `journal` and `puburl` fields from genbank files
* If you get `ERROR: Couldn't connect with entrez, please run again` just run command again

## Download from Fauna, parse, compress and push to S3

### Download from Fauna
Navigate to the nextstrain/zika repository and [follow the instructions for ingest](https://github.com/nextstrain/zika/tree/persephone/ingest).

```
python3 vdb/download.py \
--database vdb \
--virus zika \
--fasta_fields strain virus accession collection_date region country division location source locus authors url title journal puburl \
--resolve_method choose_genbank \
--fstem zika
git clone https://github.com/nextstrain/zika.git
cd zika
git checkout persephone

@victorlin victorlin Nov 27, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm out of the loop. What's persephone and why use it instead of the default branch?

@j23414 j23414 Nov 28, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question! The reasoning was that the default branch doesn't include any ingest rules. Since we're still in the midst of designing the golden path but to meet the need to keep our zika build up-to-date, I created a persephone branch to "live long enough to update the build" but will be replaced by the official path in the future. The persephone branch also includes changes to the build parameters (e.g. indexing by "accession" instead of strain name) which were a necessary divergence from the default branch.

This PR is documenting the "reality" of updating the build, but—I agree—should point at the default branch when the default branch includes ingest and subsequent build parameter modifications.

@victorlin victorlin Nov 28, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, thanks for the explanation. Was the latest update to nextstrain.org/zika made using persephone or the default branch? If using persephone, I'd think to merge that into the default branch even if the golden path is not yet set in stone. Otherwise, what's seen at github.com/nextstrain/zika would be outdated.

@j23414 j23414 Nov 28, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest update was indeed using persephone, the old method (ViPR) doesn't work. The changes to ingest and phylogenetic are here:

Feel free to glance through, I’m certainly willing to submit a persephone PR but am certain it will encounter a block, given that some of the modifications overlap with an ongoing Dengue PR that is currently delayed and unmerged (nextstrain/dengue#13). Additionally, there are specific changes related to zika, such as the inclusion of fauna-zika-data processing steps and the merging of USVI data. And given that some people might be pointed to zika as a “template for new pathogens”, it might be more prudent to cherry-pick generally accepted changes from persephone as smaller PRs later. (Open to suggestions on how to split these out.)

The current github.com/nextstrain/zika is outdated but so is the current github.com/nextstrain/dengue .

You make good points! (And may yet be incorporated in subsequent PRs.) But I'd still scope this fauna PR as replacing the non-working ViPR-ingest documentation with some currently working zika pipeline documentation.

@joverlee521 joverlee521 Nov 28, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m certainly willing to submit a persephone PR but am certain it will encounter a block

I'm sorry if the review process has been a hindrance to your work! I did not realize you were running builds off of branches. I would make a PR and merge it so that the default branch is not outdated. We can do post-merge reviews and incremental updates later. For reproducibility and keeping everyone on the same page, I think it's prudent that the production builds are made using the default branch.

@j23414 j23414 Nov 29, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

given that some people might be pointed to zika as a “template for new pathogens”, it might be more prudent to cherry-pick generally accepted changes from persephone as smaller PRs later

I still think we can cherry-pick the phylogenetic modifications in smaller PRs later, the review process is fine as long as the live builds can be updated concurrently and those steps are documented somewhere.

Could I at least merge this fauna PR? This way we have appropriately documented the status quo?

I can submit a persephone zika PR separately but don’t want it to be a pre-requisite to this PR. And if persephone is eventually merged, I'm happy to make the subsequent change to builds/ZIKA.md by basically deleting a few persephones from link urls and removing a git checkout statement.

@joverlee521 joverlee521 Nov 29, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could I at least merge this fauna PR? This way we have appropriately documented the status quo?

Sure, I'm not against merging this. For future reference, I think it would be less likely to get out-of-sync and/or forgotten if documented within the zika repo itself.

cd ingest
nextstrain build .
```

This results in the file `data/zika.fasta` with FASTA header ordered as above.
This results in the files `results/metadata.tsv` and `results/sequences.fasta`

### Parse
## Compress

```
augur parse \
--sequences data/zika.fasta \
--output-sequences data/sequences.fasta \
--output-metadata data/metadata.tsv \
--fields strain virus accession date region country division city db segment authors url title journal paper_url \
--prettify-fields region country division city
zstd -T0 results/sequences.fasta
zstd -T0 results/metadata.tsv
```

This results in the files `data/sequences.fasta` and `data/metadata.tsv`.

### Compress
This results in the files `results/sequences.fasta.zst` and `results/metadata.tsv.zst`.

```
zstd -T0 data/sequences.fasta
zstd -T0 data/metadata.tsv
```
## Upload data to s3

This results in the files `data/sequences.fasta.zst` and `data/metadata.tsv.zst`.

### Push to S3
> [!NOTE]
> Make sure [authentication for the S3 remote](https://docs.nextstrain.org/projects/cli/en/stable/remotes/s3/#authentication) is configured.

```
nextstrain remote upload s3://nextstrain-data/files/zika/ data/sequences.fasta.zst data/metadata.tsv.zst
nextstrain remote upload s3://nextstrain-data/files/zika/ results/sequences.fasta.zst
nextstrain remote upload s3://nextstrain-data/files/zika/ results/metadata.tsv.zst
```

This pushes files to S3 to be made available at https://data.nextstrain.org/files/zika/sequences.fasta.zst and https://data.nextstrain.org/files/zika/metadata.tsv.zst.

## Run zika workflow

See instructions at https://github.com/nextstrain/zika.
See instructions at https://github.com/nextstrain/zika/tree/persephone/phylogenetic

```
cd ../phylogenetic
mv ingest/results data
nextstrain build .
```