fix: count literal special-token strings as ordinary text in TiktokenCounter

[Naeemeh146]

Related Issues

• fixes TiktokenCounter raises ValueError on literal special-token strings like <|endoftext|> #12869

Proposed Changes:

TiktokenCounter.count() encoded the rendered conversation with a bare Encoding.encode(), which defaults to disallowed_special="all" and raises ValueError on any literal special-token string (e.g. <|endoftext|>) in message or tool content. Since allowed_special also defaults to empty, the strict check could never emit a real special token here — it only turned ordinary text into a crash.

This mirrors #12853, which fixed the same bug in DocumentSplitter/RecursiveDocumentSplitter: the counter now uses encode_ordinary(), so special-token-looking strings are counted as ordinary text. The class docstring, the tiktokencounter.mdx docs page, and a release note are updated accordingly.

How did you test it?

• Reproduced the issue's exact snippet before the fix (raised ValueError) and verified it returns a count afterwards, for both the message path and the tool-schema path.
• Added two regression tests in test/token_counters/test_tiktoken_counter.py.
• Ran the full test/token_counters/ suite: 46 passed, 1 skipped (requires OPENAI_API_KEY); ruff check and ruff format --check are clean on the changed files.

Notes for the reviewer

• The _FakeEncoder in the test module was renamed from encode to encode_ordinary to match the new call.
• This contribution was prepared with the help of an AI coding assistant; all changes were reviewed and tested locally before submission.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@Naeemeh146 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[github-actions]

Hi @Naeemeh146, thanks for your interest in contributing to Haystack! 🙏

⚠️ Issue #12869 is already being addressed by open pull request(s) #12871, #12876, #12879, #12888. Before opening a PR for an issue, please check whether a PR is already linked to it, and consider contributing to the existing PR instead. We may close duplicate PRs to keep the review queue manageable.

This is an automated message to help us keep the review queue healthy.

[Naeemeh146]

Quick note for triage: this PR is ready for review. The fix mirrors #12853 (encode_ordinary), with regression tests for both the message and tool-schema paths, a release note, and a docs update. Happy to address any feedback.