Skip to content

fix: count literal special-token strings in TiktokenCounter - #12874

Closed
hahahahahayesyeseys wants to merge 1 commit into
deepset-ai:mainfrom
hahahahahayesyeseys:fix/issue-12869-tiktoken-special-token
Closed

fix: count literal special-token strings in TiktokenCounter#12874
hahahahahayesyeseys wants to merge 1 commit into
deepset-ai:mainfrom
hahahahahayesyeseys:fix/issue-12869-tiktoken-special-token

fix: count literal special-token strings in TiktokenCounter

44959f0
Select commit
Loading
Failed to load commit list.

Select a check to view from the sidebar