Skip to content

fix: count literal special-token strings as ordinary text in TiktokenCounter - #12876

Closed
vineoy wants to merge 1 commit into
deepset-ai:mainfrom
vineoy:fix/12869-tiktoken-ordinary
Closed

fix: count literal special-token strings as ordinary text in TiktokenCounter#12876
vineoy wants to merge 1 commit into
deepset-ai:mainfrom
vineoy:fix/12869-tiktoken-ordinary

fix: count literal special-token strings as ordinary text in Tiktoken…

8dbd35d
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

1 warning
guard
succeeded Sep 23, 2026 in 8s