[WIP] Sync datatables

[lldelisle]

This PR is opened just to open conversation and allow anyone to contribute.

[sveinugu]

Cool. Will add refgetstore creation/hash-calculation for the fasta files.

[B0r1sD]

We are interested! Ping @pauldg

[lldelisle]

Hi @sveinugu,
What is the difference between the python package refget and gtars.refget?
Is there a way to get all the digests of a fasta file but without creating a RefgetStore?
Thanks

[bernt-matthias]

This should trigger a warning as well IMO.

[lldelisle]

See above

[nsheff]

To answer some questions from above:

Hi @sveinugu, What is the difference between the python package refget and gtars.refget? Is there a way to get all the digests of a fasta file but without creating a RefgetStore? Thanks

Hi @lldelisle, to answer your question: gtars.refget is the Rust implementation. The refget Python package is a Python interface that uses the Rust gtars package under the hood, i.e. Python bindings into that Rust code, plus the higher-level pieces (seqcol models, clients, and the seqcolapi server). So they're the same engine with two entry points, not competing implementations.

And yes, you don't need a store at all to get the digests. gtars.refget.digest_fasta(path) returns the per-sequence GA4GH digests directly (name, length, sha512t24u). For the full sequence-collection digests, the refget Python package gives you fasta_to_seqcol_dict(path) to build the canonical collection and seqcol_digest(seqcol_dict) to compute the top-level (level-0) digest. No store, and no disk writes, involved.

share the Refgetstore as a separate CVMFS repo once it is built

Yes - 100%. Before that I would appreciate a discussion with the refgetstore devs (this would be @nsheff?) to make sure that we get most out of this project (ideally with added value for the refgetstore and the Galaxy project :))

Sure, happy to have that discussion. One thing worth putting on the table: I've been building S3-backed refgetstores for datasets like VGP. You can explore them here:

• Explorer home: https://refget.databio.org/explore
• VGP store: https://refget.databio.org/explore-store/overview?url=https%3A%2F%2Frefgenie.s3.us-east-1.amazonaws.com%2Frefget-store%2Fvgp%2F

This is a web interface built directly on top of a refgetstore living in S3, no backend server required. The benefit is a cloud-friendly storage layer: the store is just static files in a bucket, and the explorer (and the seqcol API) read from it over HTTP range requests. So instead of standing up and maintaining a server, you can drop the store in object storage and serve or explore it directly. Might be a useful distribution model to consider alongside CVMFS?

Also, your CVMFS tree already has a vgp.galaxyproject.org subtree (~679 genomes), so the VGP store linked above is drawn from the same source. That makes it a natural point of comparison for the sync effort.

[sveinugu]

@nsheff Thanks for the response!

The main thing I was unable to convey to the others was how refgenie v1 connects to this, as I was unable to locate the corresponding code base. The syncing efforts is linked to efforts reorganising how mapping indices etc are generated and populated.

Regarding S3: CVMFS is doing much of the same work as an S3 bucket, but with local caching and, importantly for us, relaying into our secure server setup at the University of Oslo which is cut off from the rest of the internet. Thus, I would very much like to investigate a frontend web interface running on top of local (e.g. CVMFS) storage. I was able to get this to work by setting up a simple FastAPI hosting of the local file system, but this was a bit hacky and would be nice if something like this would work out-of-the-box. There are potential security issues here, but since CVMFS in nature is a read-only file system I am sure it would be possible to secure if that is the source of the refgetstore.

[bernt-matthias]

Added a script to compute hashes (sha256) and manifest (hash, size, name, of all included files) of all data table entries starting from the yaml created by tool_data_table_conf_to_yaml.py (needs a bit of polishing .. but seems to run .. tested locally on my instance). It creates a yaml per non-empty data table.

Idea is that we can match the top-level hashes .. and use the manifest info only in case of conflict.

Edit: On my instance the script needed 2h and produced 500MB yaml (mostly due to the manifests .. maybe I should create one with and one without manifests). Guess that we should also include the index data tables.

[lldelisle]

Hi @bernt-matthias,
Why do you exclude from the all_tables_content_add_hashes.py some tables? It seems that there were a lot of manual rewritting so I would feel more confident if we also hash the fasta derived tables (maybe not the all_fasta as we will do the full digest on it).

[bernt-matthias]

Why do you exclude from the all_tables_content_add_hashes.py some tables

This was an oversight (caused by copy paste) I will change it.