Skip to content

fix: count literal special-token strings as ordinary text in TiktokenCounter - #13009

Open
Naeemeh146 wants to merge 1 commit into
deepset-ai:mainfrom
Naeemeh146:fix/tiktoken-counter-12869
Open

fix: count literal special-token strings as ordinary text in TiktokenCounter#13009
Naeemeh146 wants to merge 1 commit into
deepset-ai:mainfrom
Naeemeh146:fix/tiktoken-counter-12869

Commits

  1. Commits on Sep 29, 2026