Bigtools hangs on inputs when they are not passed from stdin for certain inputs

[mrvollger]

Hi @jackh726,

I am running into an odd issue where if I pass a certain file via stdin bigtools quickly processes it, but if I provide the full file path bigtools gets stuck and just hangs (while using a decent amount of CPU). Here is an example script for which I have attached test data:

# does not hang
echo test1
cat temp-unreliable-coverage-regions.bed | bigtools bedtobigbed -s start -a bed3.as - mm10.fa.fai tmp.bb

# hangs
echo test2
bigtools bedtobigbed -s start -a bed3.as temp-unreliable-coverage-regions.bed mm10.fa.fai tmp.bb

I am using bigtools version 0.5.4

Thanks in advance!
Mitchell

bigtools-bug.zip

[mrvollger]

A further note, on other files the bigtools will hang in a seemingly identical way but the input will be coming from stdin. I would share an example of that as well but I haven't been able to make a small test case that is easy to share yet.

[mrvollger]

Ahh I didn't realize I wasn't on the latest version of bigtools, I updated to 0.5.5 and it appears it does not have this issue. Sorry! Though looking at the changelog I am not sure which fix actually fixed this issue.

[mrvollger]

Hmm I take it back, using v0.5.5 I am still able to find a situation where bigtools seems to hang, or at the very minimum take much much longer than ucsc's version of the tools. Example:

time \
  bedToBigBed -allow1bpOverlap -type=bed12+ \
  -as=bed12_filter.as \
  fibers.bed.gz mm10.fa.fai tmp.fibers.bb

time (cat fibers.bed.gz \
   | bgzip -cd -@ 8   \
   | bigtools bedtobigbed  \
     --inmemory -t 8 \
     -s start -a bed12_filter.as \
     - ./mm10.fa.fai tmp.fibers.bb )

UCSC result:

pass1 - making usageList (2 chroms): 192 millis
pass2 - checking and writing primary data (500000 records, 13 fields): 1102 millis

real    0m1.917s
user    0m2.182s
sys     0m0.159s

And bigtools is still running.

Data:

bigtools-bug-2.zip

[nvictus]

I encountered occasional deadlock writing via pybigtools, assuming it's the same underlying issue: https://gist.github.com/nvictus/dbb318b2941f006863af2ad63d7ab839

I haven't reproduced it recently, but i'll send a more minimal example once I do.