fix: count literal special-token strings as ordinary text in TiktokenCounter

[vineoy]

Related Issues

• fixes TiktokenCounter raises ValueError on literal special-token strings like <|endoftext|> #12869

Proposed Changes:

TiktokenCounter.count() encoded the rendered conversation + tool schemas with Encoding.encode(), which defaults to disallowed_special="all" and raises ValueError on literal markers like <|endoftext|>. Since allowed_special defaults to set(), it could never emit a special token — only crash.

One-line fix mirroring #12853 (document splitters): use encode_ordinary() (= encode(text, disallowed_special=())), which counts markers as ordinary text and skips the special-token scan. No behavior change for inputs that worked before.

How did you test it?

• Reproduced first: counter.count([ChatMessage.from_user("The manual documents <|endoftext|> as a literal marker.")]) raised ValueError before, returns 17 after. Tools path (tool description with marker) returns 69 vs 3 plain.
• python -m pytest test/token_counters/ -q: 47 passed, 1 skipped.
• New tests: unit test_special_token_strings_are_counted_as_ordinary_text + integration test_special_token_string_in_message_is_counted and test_special_token_string_in_tool_description_is_counted — all pass.
• ruff check + ruff format --check on touched files: clean.

Notes for the reviewer

Follows the #12853 pattern exactly. _FakeEncoder in tests gains encode_ordinary to mirror the real tiktoken.Encoding API. Release note added.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.

[vercel]

@vineoy is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Hi @vineoy, thanks for your interest in contributing to Haystack! 🙏

⚠️ Issue #12869 is already being addressed by open pull request(s) #12871, #12873, #12874. Before opening a PR for an issue, please check whether a PR is already linked to it, and consider contributing to the existing PR instead. We may close duplicate PRs to keep the review queue manageable.

This is an automated message to help us keep the review queue healthy.

[CLAassistant]

All committers have signed the CLA.

[julian-risch]

Thank you for your efforts! We're closing this PR because it duplicates #13009, which makes the same one-line fix to TiktokenCounter.