Kernel dies during embedding of sentences

[avinashsai]

Hi,
When I try to embed sentences for the 100M sentences using:
python src/sase.py --input $input --model data/sase.pth --spm_model data/sase.spm --batch_size 64 --cuda "True" --output $output kernel dies automatically.

What is the hardware requirements to embed sentences?

[jingfeidu]

Hi, what error message did you get when running embedding? We are using Volta-32GB GPU for inference. You can try to reduce batch_size if you came across issues on GPU memory.

[avinashsai]

I am not getting any error. But, after 580,00,000 samples inference stops and kernel dies. I am using V100 GPU with 32GB GPU RAM. I tried with a batch size of 16 but same problem arose.

[chetanambi]

@avinashsai Would you mind sharing sample code how to use?

[avinashsai]

@chetanambi I am sorry. Samsung doesn't allow publicizing code.

[chetanambi]

@avinashsai I understand that. No worries. I was thinking you might be trying this on your own. Let me try it out myself.