Skip to content

fix: count literal special-token strings as ordinary text in TiktokenCounter - #13009

Open
Naeemeh146 wants to merge 1 commit into
deepset-ai:mainfrom
Naeemeh146:fix/tiktoken-counter-12869
Open

fix: count literal special-token strings as ordinary text in TiktokenCounter#13009
Naeemeh146 wants to merge 1 commit into
deepset-ai:mainfrom
Naeemeh146:fix/tiktoken-counter-12869

fix: count literal special-token strings as ordinary text in Tiktoken…

965dc7b
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

1 warning
guard
succeeded Sep 29, 2026 in 8s