Skip to content

Fix ValueError on literal special-token strings in TiktokenCounter (#… - #13016

Closed
ramashishmaurya wants to merge 1 commit into
deepset-ai:mainfrom
ramashishmaurya:fix-tiktoken-special-tokens
Closed

Fix ValueError on literal special-token strings in TiktokenCounter (#…#13016
ramashishmaurya wants to merge 1 commit into
deepset-ai:mainfrom
ramashishmaurya:fix-tiktoken-special-tokens

Commits

  1. Commits on Sep 29, 2026