Skip to content

fix: TiktokenCounter counts literal special-token strings as ordinary text - #12888

Closed
Shubhampm18 wants to merge 1 commit into
deepset-ai:mainfrom
Shubhampm18:fix/tiktoken-counter-literal-special-tokens
Closed

fix: TiktokenCounter counts literal special-token strings as ordinary text#12888
Shubhampm18 wants to merge 1 commit into
deepset-ai:mainfrom
Shubhampm18:fix/tiktoken-counter-literal-special-tokens

fix: TiktokenCounter counts literal special-token strings as ordinary…

79692c5
Select commit
Loading
Failed to load commit list.

Select a check to view from the sidebar