中英文数据集处理问题

[wangguanhua]

您好，我看您数据集有中文的也有英文的。但中英文取token的方式不是不一样吗？英文是wordpiece，中文是直接切分，我没看到您的代码中有做相关的处理。或是我对您的代码理解有误？

[LittleSJL]

We use character-based tokenization for Chinese, and WordPiece tokenization for all other languages. Both models should work out-of-the-box without any code changes. We did update the implementation of BasicTokenizer in tokenization.py to support Chinese character tokenization, so please update if you forked it.
上面是谷歌BERT官方对于Tokenizer的解释，具体的你可以自己去看，意思就是对于中英文，BERT通过一个tokenizer就能无差别分词，不需要你根据不同的语言做不同的处理。

[wangguanhua]

多谢多谢

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人: "songyingxin/Bert-TextClassification" <notifications@github.com&gt;; 发送时间:&nbsp;2020年11月5日(星期四) 中午11:05 收件人:&nbsp;"songyingxin/Bert-TextClassification"<Bert-TextClassification@noreply.github.com&gt;; 抄送:&nbsp;"767477036"<767477036@qq.com&gt;;"Author"<author@noreply.github.com&gt;; 主题:&nbsp;Re: [songyingxin/Bert-TextClassification] 中英文数据集处理问题 (#20) We use character-based tokenization for Chinese, and WordPiece tokenization for all other languages. Both models should work out-of-the-box without any code changes. We did update the implementation of BasicTokenizer in tokenization.py to support Chinese character tokenization, so please update if you forked it. 上面是谷歌BERT官方对于Tokenizer的解释，具体的你可以自己去看，意思就是对于中英文，BERT通过一个tokenizer就能无差别分词，不需要你根据不同的语言做不同的处理。 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.