fix: count literal special-token strings in TiktokenCounter

[hahahahahayesyeseys]

Related Issues

• fixes TiktokenCounter raises ValueError on literal special-token strings like <|endoftext|> #12869

Proposed Changes:

TiktokenCounter passed rendered messages and tool schemas to tiktoken.Encoding.encode(). That method rejects literal special-token strings such as <|endoftext|> by default, even though Haystack treats the rendered input as ordinary user-controlled text. As a result, Agent compaction could fail while counting retrieved content or tool output that merely documents one of those markers.

This change uses encode_ordinary() for the rendered text, matching the behavior already adopted by Haystack's token-based document splitters. It also:

• updates the local encoder test double to exercise the ordinary-text path;
• adds real-encoder regressions for a marker in both a message body and a tool description;
• adds a release note describing the user-visible fix.

There is no public API or serialization change. Inputs that were already accepted keep the same ordinary-text tokenization; literal marker strings now return a count instead of raising ValueError.

How did you test it?

• Before the fix, both new regressions failed with ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
• The issue reproduction now returns a count of 17 with the real o200k_base encoder.
• hatch run test:all test/token_counters/test_tiktoken_counter.py -q --tb=short: 14 passed.
• hatch run test:all test/token_counters -q --tb=short: 46 passed, 1 skipped because OPENAI_API_KEY was not set.
• hatch run test:types haystack/token_counters/tiktoken_counter.py test/token_counters/test_tiktoken_counter.py: passed.
• hatch run fmt-check haystack/token_counters/tiktoken_counter.py test/token_counters/test_tiktoken_counter.py: passed.
• Pre-commit hooks on all three changed files: passed.

Notes for the reviewer

The behavioral change is limited to the tiktoken counter's handling of literal special-token text. encode_ordinary() does not enable special-token semantics; it counts the marker's characters as ordinary text. Non-text image and file charges are unchanged.

This contribution was generated with AI assistance. The final diff was reviewed before publication, and the listed tests and checks were executed locally.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes. The PR will link and close the issue; no standalone issue comment has been posted.
• I have added unit tests and updated the docstrings. Real-encoder regression tests were added; no public signature or documented contract changed, so no docstring update is needed.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code. The user-visible behavior change is covered by a release note.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@hahahahahayesyeseys is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Hi @hahahahahayesyeseys, thanks for your interest in contributing to Haystack! 🙏

⚠️ Issue #12869 is already being addressed by open pull request(s) #12871, #12873. Before opening a PR for an issue, please check whether a PR is already linked to it, and consider contributing to the existing PR instead. We may close duplicate PRs to keep the review queue manageable.

This is an automated message to help us keep the review queue healthy.