Skip to content

fix: treat literal tiktoken special tokens as ordinary text in TiktokenCounter - #12871

Closed
HarshRajSinghania wants to merge 1 commit into
deepset-ai:mainfrom
HarshRajSinghania:fix/tiktoken-counter-ordinary-encode
Closed

fix: treat literal tiktoken special tokens as ordinary text in TiktokenCounter#12871
HarshRajSinghania wants to merge 1 commit into
deepset-ai:mainfrom
HarshRajSinghania:fix/tiktoken-counter-ordinary-encode

fix: count literal tiktoken special tokens as ordinary text

37f10ea
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

1 warning
sync
succeeded Sep 23, 2026 in 9s