fix: treat literal tiktoken special tokens as ordinary text in TiktokenCounter

[HarshRajSinghania]

Summary

TiktokenCounter.count() encoded rendered conversations with Encoding.encode(), which defaults to disallowed_special="all" and raises ValueError when the text contains a literal special-token marker such as <|endoftext|>.

This switches the call to encode_ordinary(), which counts those markers as ordinary text. That matches the approach already used by DocumentSplitter / RecursiveDocumentSplitter after #12853. TiktokenCounter was the remaining haystack/ caller that used a bare .encode() on user-supplied text.

Motivation

Fixes #12869.

TiktokenCounter is used by Agent conversation-compaction hooks. The counted text can include tool results and retrieved documents the developer does not control. A page that quotes <|endoftext|> currently crashes compaction instead of returning a count.

Implementation

• haystack/token_counters/tiktoken_counter.py: use self._encoder.encode_ordinary(...) instead of .encode(...).
• test/token_counters/test_tiktoken_counter.py: add encode_ordinary on the fake encoder so unit tests still record the rendered text; add two integration regression tests covering a message body and a tool description that contain the marker.

No behavior change for inputs that already worked, because encode() never emitted special tokens with the previous defaults.

Testing

Environment limits prevented installing Haystack's full test extras here (PyPI 502s on this runner). The change is the one-line API swap requested in #12869 plus tests that follow the existing file's patterns. Please rely on CI for pytest test/token_counters/test_tiktoken_counter.py.

[vercel]

Someone is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Harsh Raj Singhania seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[HaystackBot]

Hi @HarshRajSinghania, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

[HaystackBot]

Hi @HarshRajSinghania, just a friendly reminder: this PR is still in draft because the Contributor License Agreement (CLA) hasn't been signed yet. We'd love to review your contribution! Please sign the CLA via the link in the license/cla check, and this PR will automatically be marked ready for review.

[julian-risch]

Thank you for your efforts! We're closing this PR because it duplicates #13009, which makes the same one-line fix to TiktokenCounter. This PR is also still a draft with the CLA unsigned, and it has no release note.