Skip to content

fix: TiktokenCounter counts literal special-token strings as ordinary text - #12888

Closed
Shubhampm18 wants to merge 1 commit into
deepset-ai:mainfrom
Shubhampm18:fix/tiktoken-counter-literal-special-tokens
Closed

fix: TiktokenCounter counts literal special-token strings as ordinary text#12888
Shubhampm18 wants to merge 1 commit into
deepset-ai:mainfrom
Shubhampm18:fix/tiktoken-counter-literal-special-tokens

Commits

  1. Commits on Sep 23, 2026