Skip to content

fix(token-counter): count literal special-token markers as text - #12986

Open
ducanhnguyen223 wants to merge 1 commit into
deepset-ai:mainfrom
ducanhnguyen223:contrib/tiktoken-literal-special-tokens
Open

ducanhnguyen223 wants to merge 1 commit into
deepset-ai:mainfrom
ducanhnguyen223:contrib/tiktoken-literal-special-tokens

Conversation

@ducanhnguyen223

Copy link
Copy Markdown

Summary

  • use encode_ordinary() in TiktokenCounter so literal tiktoken markers in user text and tool schemas are counted as ordinary text
  • add regressions for both message and tool-schema paths
  • add a concise release note

Verification

  • upstream behavior reproduced: Encoding.encode("literal <|endoftext|> marker") raises ValueError; encode_ordinary() returns 9 tokens
  • targeted token-counter suite: 14 passed
  • Ruff and git diff --check pass

The repository Hatch command is not installed in this environment, so the targeted checks were run with uv; no full suite or type check is claimed.

@ducanhnguyen223
ducanhnguyen223 requested a review from a team as a code owner September 27, 2026 12:58
@ducanhnguyen223
ducanhnguyen223 requested review from julian-risch and removed request for a team September 27, 2026 12:58
@vercel

vercel Bot commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Someone is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Sep 27, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@ducanhnguyen223
ducanhnguyen223 force-pushed the contrib/tiktoken-literal-special-tokens branch from 45645b8 to d26c1b1 Compare September 27, 2026 13:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants