ret1=tokenizer([pairs[idx]], add_special_tokens=True,max_length=512,padding='max_length',truncation=True,return_tensors='pt') #tokenize them
Many functions have more than 512 tokens. Will max_length and truncation affect the similarity result? Is it ok to adjust a longer max_length?
In eval_save.py:142
Many functions have more than 512 tokens. Will max_length and truncation affect the similarity result? Is it ok to adjust a longer max_length?