Fastq file could not be opened

[jrpletch]

Hello, I am trying to extract COI genes from raw Illumina sequences of cestodes to try and ID the hosts. I am using the cestodes as a proof of concept but would eventually like to try it with whole genome sequences from fleas to see what other sequences we might incidentally capture such as hosts, other parasites, or contaminants. I managed to extract a vole mitogenome from this data using Novoplasty, but would like to find a way to ID other potential sequences and it looked like MitoGeneExtractor could be the way to go. I managed to get MitoGeneExtractor installed but got this error after trying to run it with my data. I'm just starting to learn all this so it is entirely possible I'm missing something obvious. I did have to reverse the direction of the slashes in the file paths, initially they were \ but they ended up all disappearing after I ran the command.

This is what I ran:

./MitoGeneExtractor-v1.9.5 -q D:/NovoplastyPractice_Josh/A013489/A013489_L000_R1_001.fastq/A013489_L000_R1_001.fastq -q D:/NovoplastyPractice_Josh/A013489/A013489_L000_R2_001.fastq/A013489_L000_R2_001.fastq -p mammalia_COI_genera_consensus -o A013489_mammal_alignment.fas -n 0 -c A013489_mammal_consensus.fas -t 0.5 -r 1 -C 2

Thanks,
Josh

[marievalerie]

Hello,
Are you sure your file path is correct?
It looks like that you've specified the file name twice i.e. your files are located in a directory that has the same name as your file?

[jrpletch]

Thanks, I think there was an issue with the file path being in Windows rather than Linux format. The files are indeed located in a directory with the same name. I've got that fixed and no longer get the fastq file could not be opened error, instead now it's just saying killed. I tried it with trimmed reads in case that might've been the problem but still received the same error. Any idea what might be the issue?

Edit: I realized there's a slight typo in the file path there, but ended up with the same result after fixing it.

[marievalerie]

assuming that MGE can call exonerate by its default value (which is 'exonerate'), then your command should in principle work... anyway, you would get another error if MGE cannot find exonerate, so this problem probably refers to something else. how long does it take until the process is killed? are you running this on a local machine or on a server and how large are your input data files?

[marievalerie]

@cmayer can you have a look?

[jrpletch]

Thanks for your assistance with this, it takes about 4 minutes before it is killed. I'm running on a local machine and the input files are about 24 gb each.

[marievalerie]

oh wow, 48 GB input is a lot ! I am almost certain that a local machine has not sufficient RAM to analyze this... I had sometimes issues with input data in that size range when I used a server with > 1 TB RAM..

[jrpletch]

Interesting, the computer I'm using has 80 gb of RAM and that seemed to be sufficient for Novoplasty to run these files, but maybe MGE needs more? Would you have any recommendations for making it more manageable?

[marievalerie]

yeah, the exonerate alignment requires a substantial amount of memory unfortunately. I don't think that this can be changed directly (although christoph might have an idea) but what you could try is to use only the forward OR the reverse file (it is paired-end data right?).
my idea would be that forward and reverse read originate from the same (mitochondrial) DNA fragment and in many cases, they "only" support each other, although this depends on the insert size and if forward and reverse read overlap. it is not ideal but you have so much data that one read pair might be more than sufficient....

[cmayer]

I am working on the memory issue.

The problem is the following:
Currently, MGE loads all fasta and fastq files (concatenated) into memory. This means that the size of all input files together is required as RAM, with some overhead. Now exonerate is run with all sequences as input and it also requires quite a lot of RAM.

I have to change the way the data is handled.
I need to load the input sequence files piece by piece, pass each piece to exonerate and add the results to the alignment. This should solves the issues some of you have with the RAM.
This is what I am working on at the moment.

The suboptimal design desicion is a result of the review process of the paper. Reviewers asked for more features and the quickest way to implement this was not well thought out in the sence of memory efficiency. Well, I should have seen this problem before I have to admit.

[cmayer]

I updated the main branch to reflect the most recent bug fixes. The previous version in the refactor-bugfix-v1.9.6 branch stopped when reading reads of unequal lengths from fastq files. Basically, the fastq to fasta file converter was not working properly. This has been fixed now.