fix(token-counter): count literal special-token markers as text - #12986

fix(token-counter): count literal special-token markers as text#12986

ducanhnguyen223 wants to merge 1 commit into

deepset-ai:maindeepset-ai/haystack:mainfrom

ducanhnguyen223:contrib/tiktoken-literal-special-tokensducanhnguyen223/haystack:contrib/tiktoken-literal-special-tokensCopy head branch name to clipboard