Skip to content

fix: count literal special-token strings in TiktokenCounter as ordinary text - #12873

Closed
SEVEN-us wants to merge 1 commit into
deepset-ai:mainfrom
SEVEN-us:fix/tiktoken-counter-literal-special-tokens
Closed

fix: count literal special-token strings in TiktokenCounter as ordinary text#12873
SEVEN-us wants to merge 1 commit into
deepset-ai:mainfrom
SEVEN-us:fix/tiktoken-counter-literal-special-tokens

fix: count literal special-token strings in TiktokenCounter as ordina…

d814027
Select commit
Loading
Failed to load commit list.

Select a check to view from the sidebar