Hi!
I am pretraining transformers, however, I found every time I load the model from a checkpoint, it spent more than 16 hours skipping the pretraining steps (in other words, on pure I/O). I stored the data file in '/jmain02/home/J2ADxxx/[project]/[user]/XXX/XXX'. Furthermore, I wonder is there any other path on JADE2 that I could store data and accelerate I/O? If yes, is there a space quota?
Hi!
I am pretraining transformers, however, I found every time I load the model from a checkpoint, it spent more than 16 hours skipping the pretraining steps (in other words, on pure I/O). I stored the data file in '/jmain02/home/J2ADxxx/[project]/[user]/XXX/XXX'. Furthermore, I wonder is there any other path on JADE2 that I could store data and accelerate I/O? If yes, is there a space quota?