Skip to content

fix(token-counter): count literal special-token markers as text - #12986

Closed
ducanhnguyen223 wants to merge 1 commit into
deepset-ai:mainfrom
ducanhnguyen223:contrib/tiktoken-literal-special-tokens
Closed

fix(token-counter): count literal special-token markers as text#12986
ducanhnguyen223 wants to merge 1 commit into
deepset-ai:mainfrom
ducanhnguyen223:contrib/tiktoken-literal-special-tokens

fix(token-counter): treat literal special tokens as text

d26c1b1
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

1 warning
triage
succeeded Sep 27, 2026 in 8s