Skip to content

fix: count literal special-token strings instead of crashing - #13014

Closed
milangeorge2000 wants to merge 2 commits into
deepset-ai:mainfrom
milangeorge2000:fix/12869-tiktoken-special-tokens
Closed

milangeorge2000 wants to merge 2 commits into
deepset-ai:mainfrom
milangeorge2000:fix/12869-tiktoken-special-tokens

Commits

  1. Commits on Sep 28, 2026

  2. Commits on Sep 29, 2026