Fix ValueError on literal special-token strings in TiktokenCounter (#…

[ramashishmaurya]

Related Issues

• fixes TiktokenCounter raises ValueError on literal special-token strings like <|endoftext|> #12869

Proposed Changes:

TiktokenCounter.count() was previously raising a ValueError when encountering literal tiktoken special-token strings (like <|endoftext|>) inside the conversation or tool schemas. This was because it used self._encoder.encode() which strictly disallows special tokens by default.

This PR replaces encode() with encode_ordinary() in TiktokenCounter.count(). This ensures that special-token strings are treated as ordinary text and correctly counted, preventing crashes when an agent processes uncontrolled text (e.g., retrieving an OpenAI documentation page).

How did you test it?

• Added a new integration test test_counts_literal_special_tokens in test_tiktoken_counter.py that verifies TiktokenCounter successfully counts text containing <|endoftext|> without raising an exception.

Notes for the reviewer

This fix mirrors the approach taken in #12853 for document splitters.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@ramashishmaurya is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Hi @ramashishmaurya, thanks for your interest in contributing to Haystack! 🙏

⚠️ Issue #12869 is already being addressed by open pull request(s) #12871, #12876, #12879, #12888, #13009, #13014. Before opening a PR for an issue, please check whether a PR is already linked to it, and consider contributing to the existing PR instead. We may close duplicate PRs to keep the review queue manageable.

This is an automated message to help us keep the review queue healthy.