fix: count literal special-token strings in TiktokenCounter

[burakeyler]

Related Issues

Fixes #12869

Proposed Changes

TiktokenCounter.count() called Encoding.encode(), which defaults to disallowed_special="all" and therefore raises ValueError whenever a message or tool schema merely mentions a string like <|endoftext|>. Since allowed_special defaults to the empty set, no special tokens could ever be produced anyway, so the strict check only turned ordinary user text into a crash.

Switched to encode_ordinary(), the same fix applied to the document splitters in #12853, so special-token strings are counted as literal text.

How did you test it?

• Updated the existing _FakeEncoder in the unit tests to the encode_ordinary entry point.
• Added an integration test that counts a message containing <|endoftext|> with the real o200k_base encoding; it raises on main and passes here.
• pytest test/token_counters/test_tiktoken_counter.py → 13 passed.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title
• I documented my code
• I ran pre-commit hooks and fixed any issue

🤖 Generated with Claude Code

[vercel]

@burakeyler is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions]

Hi @burakeyler, thanks for your interest in contributing to Haystack! 🙏

⚠️ Issue #12869 is already being addressed by open pull request(s) #12871, #12873, #12876. Before opening a PR for an issue, please check whether a PR is already linked to it, and consider contributing to the existing PR instead. We may close duplicate PRs to keep the review queue manageable.

This is an automated message to help us keep the review queue healthy.

[HaystackBot]

Hi @burakeyler, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

[HaystackBot]

Hi @burakeyler, just a friendly reminder: this PR is still in draft because the Contributor License Agreement (CLA) hasn't been signed yet. We'd love to review your contribution! Please sign the CLA via the link in the license/cla check, and this PR will automatically be marked ready for review.

[julian-risch]

Thank you for your efforts! We're closing this PR because it duplicates #13009, which makes the same one-line fix to TiktokenCounter. This PR is also still a draft with the CLA unsigned, and its release note fails our backtick check.