not enough data in stream

[bwlang]

from @kellywuthrich

I’m running the workflow on my HPC cluster using slurm. I got it to work before using a small subset of data, but now the alignReads process fails mid-run. It starts and immediately fails with this error:

Command output:
Input file size: 13.9476835839450359344482421875 GB
Memory allocated for this task: 64 GB

Command error:

sambamba 1.0.1
by Artem Tarasov and Pjotr Prins (C) 2012-2023
LDC 1.38.0 / DMD v2.108.1 / LLVM17.0.6 / bootstrap LDC - the LLVM D compiler (1.38.0)

[main_samview] fail to read the header from "/dev/stdin".
sambamba-sort: not enough data in stream_

It used to process a little bit of the file, I could see bwa-meth actively aligning but would then stall and fail. Now every time I get past the enoughReads it won't continue. Any advice would be greatly appreciated! I've been going in circles with this for days and can't find a solution.

[bwlang]

Seems like you're using the legacy em-seq workflow (based on the sambamba reference).

I've seen this sort of thing before when the input data is not valid - have a look at the intermediate outputs just before sambamba to get a clue (samblaster or one of the component bams).

[kellywuthrich]

Okay yes, thanks! I am now no longer using the legacy version and I am still getting errors during the alignment. It terminates with error exit status 1. It will align and output files are made, but fails at a broken pipe. I'm using bwameth=0.2.7 and mark-nonconverted-reads=1.2.

Command: bwameth.py -t 8 --read-group "@rg\tID:sample\tSM:sample" --reference ref.fasta -p read1.fq.gz read2.fq.gz | mark-nonconverted-reads.py --reference ref.fasta | samtools sort ...

Error: BrokenPipeError: [Errno 32] Broken pipe
File ".../bwameth.py", line 393, in as_bam
sys.stdout.write(str(aln) + '\n')

[bwlang]

It sounds like there may be something wrong with your read files... maybe try seqtk mergepe to see if it produces any errors?