Skip to content

fix: count literal special tokens in TiktokenCounter - #13010

Closed
yusei21 wants to merge 1 commit into
deepset-ai:mainfrom
yusei21:fix/tiktoken-literal-special-tokens
Closed

fix: count literal special tokens in TiktokenCounter#13010
yusei21 wants to merge 1 commit into
deepset-ai:mainfrom
yusei21:fix/tiktoken-literal-special-tokens

Commits

  1. Commits on Sep 29, 2026