Skip to content

fix: treat literal tiktoken special tokens as ordinary text in TiktokenCounter - #12871

Closed
HarshRajSinghania wants to merge 1 commit into
deepset-ai:mainfrom
HarshRajSinghania:fix/tiktoken-counter-ordinary-encode
Closed

fix: treat literal tiktoken special tokens as ordinary text in TiktokenCounter#12871
HarshRajSinghania wants to merge 1 commit into
deepset-ai:mainfrom
HarshRajSinghania:fix/tiktoken-counter-ordinary-encode

Commits

  1. Commits on Sep 23, 2026