OSError: [Errno 36] File name too long

[MrBleem]

Hello!
After get the expected result with the S2R data, I tried run TELR on other fly data.
Then I got the OSError:
OSError: [Errno 36] File name too long: '/home/boxu/temp/telr/result/intermediate_files/input_json/chr2L_1887214_1887214_21128_24591_ID=162;FBgn0000638_FB|ID=163;FBgn0000638_FB|ID=164;FBgn0000638_FB|ID=165;FBgn0000638_FB|ID=166;FBgn0000638_FB|ID=167;FBgn0000638_FB|ID=168;FBgn0000638_FB|ID=169;FBgn0000638_FB|ID=170;FBgn0000638_FB|ID=171;FBgn0000638_FB_input.json'

And I noticed that some insertion annotated with one type of TE but different ID in the result, just like:"ID=12;Tirant|ID=13;Tirant". Is it causing the problem？

Dose it means there is mulitple insertion seq from same TE? I'm confused for that.

Could you help me solve this?

Thank you very much!

[unavailable-2374]

Hello

I got the same problem , File name too long

Perform liftover...
[Errno 36] File name too long: '/public/home/caoshuo/project/pan_TE/MGX/intermediate_files/tmp/PN7_19261399_19261399_15469_21382_TE_00000005_TE_00000478_TE_00000717_TE_00000843_TE_00000888_TE_00001002_TE_00001026_TE_00001042_TE_00001832_TE_00001973_TE_00002053_TE_00002145_TE_00002148_TE_00002206_TE_00002384_TE_00007428_TE_00007866_TE_00008427_TE_00009810_TE_00010608_TE_00011078_TE_00011095_5p.fa'
liftover failed, exiting...

and also I find the log file shows

Error: Sorted input specified, but the file /public/home/caoshuo/project/pan_TE/MGX/intermediate_files/PN.fa.te.bed has the following out of order record

Should I clean similar sequence of rep.fa, or there is something should be changed in the script ?

Thank you!

[shunhuahan]

Hi @MrBleem, based on the error message, it appears that you ran TELR with a TE library that includes multiple identical or highly similar sequences under the same TE family. The current TELR assumes a TE library that contains one TE sequence for each family. I would suggest you to clean up the library and remove redundant sequences before running TELR. For D. melanogaster genome sequencing data, we suggest using v10.2 of the D. melanogaster canonical TE sequence library available at https://github.com/bergmanlab/drosophila-transposons/blob/master/releases/D_mel_transposon_sequence_set_v10.2.fa.

Hi @unavailable-2374, thanks for sharing the failed log. Yes, the issue appears to be caused by redundant TE sequences in the library described above. I agree with your intuition to clean redundant/similar sequences and try again.

Sorry for not making that clear in the README. Let me know if the issue still exists after the library clean up.

Best,
Shunhua

[unavailable-2374]

Oh, thank you! I will try it right now.

Best,
Shuocao

[MrBleem]

Ok, I will change the TE reference file.
Thank you very much !

Best
Bo

[unavailable-2374]

Hi @MrBleem, based on the error message, it appears that you ran TELR with a TE library that includes multiple identical or highly similar sequences under the same TE family. The current TELR assumes a TE library that contains one TE sequence for each family. I would suggest you to clean up the library and remove redundant sequences before running TELR. For D. melanogaster genome sequencing data, we suggest using v10.2 of the D. melanogaster canonical TE sequence library available at https://github.com/bergmanlab/drosophila-transposons/blob/master/releases/D_mel_transposon_sequence_set_v10.2.fa.

Hi @unavailable-2374, thanks for sharing the failed log. Yes, the issue appears to be caused by redundant TE sequences in the library described above. I agree with your intuition to clean redundant/similar sequences and try again.

Sorry for not making that clear in the README. Let me know if the issue still exists after the library clean up.

Best, Shunhua

Hello

Very unfortunately, I tried to merge the sequences with similarity 85+ in TE_lib with cd-hit and run TELR with the result as a library, but I still get "File name too long" errors.

Best, ShuoCao

[shunhuahan]

Thanks for reporting that the error persists @unavailable-2374. I will try to push a code update later this week to have this issue fixed.

[unavailable-2374]

Thanks for reporting that the error persists @unavailable-2374. I will try to push a code update later this week to have this issue fixed.

Thanks!

[shunhuahan]

Hi @unavailable-2374 and @MrBleem,

I have merged a fix into the master branch on the "file name too long" error (see #30). Please pull the latest commit from the master branch, reinstall TELR locally, and see if this new version can resolve the issue. Thanks!

cd TELR # this is the TELR git folder
git pull # necessary to use this patched update

conda activate TELR
pip install . # install this patched update

# run TELR
telr -h

Best,
Shunhua

[unavailable-2374]

Hi Shunhua

Thank, I will try it immediately！

Shuo

[unavailable-2374]

git pull 

Hi Shunhua

Unfortunately, what if I said that the error still exists?

[caoshuo@login04 pan_TE]$ tail -n 3 aug.out
Perform liftover...
[Errno 36] File name too long: '/public/home/caoshuo/project/pan_TE/MGX/intermediate_files/tmp/PN7_19261399_19261399_12323_18236_TE_00000005_TE_00000478_TE_00000717_TE_00000843_TE_00000888_TE_00001002_TE_00001026_TE_00001042_TE_00001832_TE_00001973_TE_00002053_TE_00002145_TE_00002148_TE_00002206_TE_00002384_TE_00007428_TE_00007866_TE_00008427_TE_00009810_TE_00010608_TE_00011078_TE_00011095_5p.fa'
liftover failed, exiting...
[caoshuo@login04 pan_TE]$ cd ~/tools/TELR/
[caoshuo@login04 TELR]$ git pull 
Already up-to-date.

Also reported in the additional .err file:

Error: Sorted input specified, but the file /public/home/caoshuo/project/pan_TE/MGX/intermediate_files/PN.fa.te.bed has the following out of order record

[shunhuahan]

Hi @unavailable-2374,

The error message indicates that the new TELR version was not being used in the new run. Could you double check that you have locally pip installed the new version of TELR after git pull?

cd TELR # this is the TELR git folder
conda activate TELR # active TELR environment
pip install . # install the latest update locally

# run TELR
telr -h

If you previously installed TELR through conda/mamba. You would notice that the telr version is the following from bioconda channel when listing out all the dependencies using conda list.

conda activate TELR
conda list
...
telr                      0.2                pyhdfd78af_2    bioconda
...

After you pip install TELR locally, you would see that the TELR version is now from pip when you check the environment dependencies.

conda activate TELR
conda list
...
telr                      1.0                      pypi_0    pypi
...

Hopefully this helps confirm the new pip version of TELR has been installed successfully.

Let me know if you still encounter issues. Sorry about the trouble.

Best,
Shunhua

[unavailable-2374]

Hi @shunhuahan

Traceback (most recent call last):
  File "/public/agis/zhouyongfeng_group/caoshuo/tools/miniconda3/envs/TELR/bin/telr", line 8, in <module>
    sys.exit(main())
  File "/public/agis/zhouyongfeng_group/caoshuo/tools/miniconda3/envs/TELR/lib/python3.6/site-packages/telr/telr.py", line 158, in main
    thread=args.thread,
  File "/public/agis/zhouyongfeng_group/caoshuo/tools/miniconda3/envs/TELR/lib/python3.6/site-packages/telr/TELR_te.py", line 924, in find_te
    telr_mode=True,
  File "/public/agis/zhouyongfeng_group/caoshuo/tools/miniconda3/envs/TELR/lib/python3.6/site-packages/telr/TELR_liftover.py", line 1042, in liftover
    with open(liftover_input, "w") as output:
OSError: [Errno 36] File name too long: '/public/agis/zhouyongfeng_group/caoshuo/data/TE/TELR/Vzc/intermediate_files/input_json/PN10_11553286_11553288_14059_15463_CACTA-nMITE:CACTA-nMITE1923|CACTA-nMITE:CACTA-nMITE1960|CACTA-nMITE:CACTA-nMITE1963|CACTA-nMITE:CACTA-nMITE3229|CACTA-nMITE:CACTA-nMITE846|CACTA-nMITE:CACTA-nMITE946|DIRS:DIRS3534|Gypsy:Gypsy301|Gypsy:Gypsy437|L1:L1225|MITE:MITE1319|MITE:MITE1890|MITE:MITE3676_input.json'

I have double checked this, sorry that the problem still here.

Best,
ShuoCao

[shunhuahan]

Hi @unavailable-2374,

I realized the previous update only partially fixes the issue. The error you posted in #29 (comment) and #29 (comment) should be gone already with the previous update at #30. The most recent error message is caused by the intermediately JSON file containing too many TE family names. I have made another pull request to fix this issue (see #32).

Please do git pull, conda activate TELR, pip install ., run TELR again and check if the most recent update works. Sorry for the back and forth on this. Kinda tricky to do the bug fix without having the input files that raise the error for testing.

Best,
Shunhua

[cbergman]

Hi @MrBleem and @unavailable-2374

We have updated the TELR recipe at bioconda with a new version (v1.1) that includes the fix to the filename problem you identified. You can install the latest version of TELR via bioconda as follows:

mamba create -n TELR --channel bioconda telr=1.1

Please let us know if this resolves your issue.

Best regards,
Casey