Skip to content

fix: count literal special tokens in TiktokenCounter - #13010

Closed
yusei21 wants to merge 1 commit into
deepset-ai:mainfrom
yusei21:fix/tiktoken-literal-special-tokens
Closed

yusei21 wants to merge 1 commit into
deepset-ai:mainfrom
yusei21:fix/tiktoken-literal-special-tokens