fix: count literal special-token strings in TiktokenCounter as ordinary text

[SEVEN-us]

Related Issues

• fixes TiktokenCounter raises ValueError on literal special-token strings like <|endoftext|> #12869

Proposed Changes:

TiktokenCounter.count() encoded the rendered conversation and tool schemas with Encoding.encode(), which defaults to disallowed_special="all" and therefore raises ValueError on any source text containing a string such as <|endoftext|>. Because allowed_special also defaults to set(), the call can never emit a special token — the strict check can only turn ordinary text into a crash.

The counter now encodes with encode_ordinary(), so such strings are measured as ordinary text. TiktokenCounter was the last place in haystack/ still using the strict call; #12853 applied the same fix to DocumentSplitter and RecursiveDocumentSplitter.

• haystack/token_counters/tiktoken_counter.py: use encode_ordinary() and document the behaviour in the class docstring.
• test/token_counters/test_tiktoken_counter.py: _FakeEncoder now records encode_ordinary(), plus an integration test asserting a message containing <|endoftext|> is measured rather than rejected.
• docs-website/docs/token-counters/tiktokencounter.mdx: note that special-token strings are counted as ordinary text.
• Release note added.

How did you test it?

Locally with Hatch:

hatch run test:unit test/token_counters
# 41 passed, 6 deselected

hatch run test:integration test/token_counters/test_tiktoken_counter.py
# 5 passed, 9 deselected (includes the 2 new parametrized cases)

hatch run fmt-check haystack/token_counters test/token_counters
# All checks passed! / 12 files already formatted

hatch run test:types haystack/token_counters test/token_counters
# Success: no issues found in 12 source files

I also verified the new test catches the bug: with encode() restored, it fails with

ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.

Notes for the reviewer

Renaming _FakeEncoder.encode to encode_ordinary mirrors the mock change in #12853 and is what keeps the unit tests meaningful — if the counter went back to encode(), the stub no longer has that attribute.

This PR was fully generated with an AI assistant. I have reviewed the changes and run the relevant tests.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@SEVEN-us is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Hi @SEVEN-us, thanks for your interest in contributing to Haystack! 🙏

⚠️ Issue #12869 is already being addressed by open pull request(s) #12871. Before opening a PR for an issue, please check whether a PR is already linked to it, and consider contributing to the existing PR instead. We may close duplicate PRs to keep the review queue manageable.

This is an automated message to help us keep the review queue healthy.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[HaystackBot]

Hi @SEVEN-us, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

[SEVEN-us]

Closing this as a duplicate of #12871, which fixes #12869 with the same change and was opened earlier. I had checked for linked PRs before starting, but #12871 was opened while I was working and I did not re-check before submitting. Sorry for the queue noise.