Index downloading from dropbox

[hasindu2008]

@BonsonW with latest v0.3.0 binaries that I tried

This works:

wget "https://dropbox.com/scl/fi/m91tejs4ljfbfos9y591b/PGXX22394_reads_1000.blow5.idx?rlkey=fpvsdonzxkrqsz0rq3xruh84y&st=rncsmare&dl=1" -O index.blow5.idx
./slow5curl reads "https://dropbox.com/scl/fi/1uw17z0ufjq2p95as1gqk/PGXX22394_reads_1000.blow5?rlkey=dpscpz01usz40v65jx28k9tlv&st=ghgbo3wk&dl=1" --index index.blow5.idx

But this doesn't:

./slow5curl reads "https://dropbox.com/scl/fi/1uw17z0ufjq2p95as1gqk/PGXX22394_reads_1000.blow5?rlkey=dpscpz01usz40v65jx28k9tlv&st=ghgbo3wk&dl=1" --index "https://dropbox.com/scl/fi/m91tejs4ljfbfos9y591b/PGXX22394_reads_1000.blow5.idx?rlkey=fpvsdonzxkrqsz0rq3xruh84y&st=rncsmare&dl=1" 

which gives:

[reads_main] Loading remote BLOW5 file.
[reads_main] Loading index (this may take a while).
[s5curl_idx_init_from_url::ERROR] Reading idx failed: Unknown error -14. At src/index.c:106
[reads_main::ERROR] Error loading index file for https://dropbox.com/scl/fi/1uw17z0ufjq2p95as1gqk/PGXX22394_reads_1000.blow5?rlkey=dpscpz01usz40v65jx28k9tlv&st=ghgbo3wk&dl=1 from file path https://dropbox.com/scl/fi/m91tejs4ljfbfos9y591b/PGXX22394_reads_1000.blow5.idx?rlkey=fpvsdonzxkrqsz0rq3xruh84y&st=rncsmare&dl=1.

Any idea why?

[jelber2]

I have this issue too, but I for me, it is ok to have the index files locally. Super cool tool.

[BonsonW]

Hello! @jelber2 @hasindu2008

This was an unhandled case for custom remote indexes. Fixed now in dev branch:

$ ./slow5curl reads "https://dropbox.com/scl/fi/1uw17z0ufjq2p95as1gqk/PGXX22394_reads_1000.blow5?rlkey=dpscpz01usz40v65jx28k9tlv&st=ghgbo3wk&dl=1" --index "https://dropbox.com/scl/fi/m91tejs4ljfbfos9y591b/PGXX22394_reads_1000.blow5.idx?rlkey=fpvsdonzxkrqsz0rq3xruh84y&st=rncsmare&dl=1" | head
[reads_main] Loading remote BLOW5 file.
[reads_main] Loading index (this may take a while).
3c55eb84-c2f7-4746-b1a6-509d0fda52ee
e192f5df-0813-40ec-b2f4-c5e66932d362
187f520a-dddd-472d-88ee-3c7929db7a3c
f54c9a14-c6ec-47bb-b96e-467b47fa7a60
033cf1ad-6afa-4b18-a169-ebd53ba7df18
ed47c16d-9a47-4583-bd83-fff0669d5b09
55c39a14-7806-4ebd-ade3-6d6016c1bdfa
ef372097-8adc-4de3-a7a1-f33b12f4db93
8260c48a-38e0-498b-8342-f92f3feb01b4
87417bd4-a6a4-4389-b2ba-6177bb38874b

[main] cmd: ./slow5curl reads https://dropbox.com/scl/fi/1uw17z0ufjq2p95as1gqk/PGXX22394_reads_1000.blow5?rlkey=dpscpz01usz40v65jx28k9tlv&st=ghgbo3wk&dl=1 --index https://dropbox.com/scl/fi/m91tejs4ljfbfos9y591b/PGXX22394_reads_1000.blow5.idx?rlkey=fpvsdonzxkrqsz0rq3xruh84y&st=rncsmare&dl=1
[main] real time = 3.843 sec | CPU time = 0.125 sec | peak RAM = 0.014 GB

[hasindu2008]

@BonsonW Do you remember the exact problem we enocuntered a couple of years ago when testing the extraction of large number of reads from a large BLOW5 stored on the drop box? I think @jelber2 would be interested in knowing.

[jelber2]

So for my 250 Mbit connection in Austria, I am using the settings below. Not sure where the Dropbox datacenters are located for these files. These are ultra-long reads using the ULK kit (N50 ~ 90 Kbp)

./slow5curl get blow5file.blow5 --index blow5index.idx --list desired_reads.txt -o PAU99351_5ad51671_1aa2560c_0_2_not_herro.blow5 -c zstd -s svb-zd --skip --retry 10000 --wait 5 -t 10 -K 8

I think for Dropbox hosted files that are large (>1GB), it makes more sense to download the whole thing with wget or curl.