feat: support split_by="token" in HierarchicalDocumentSplitter

[shivsin25]

Related Issues

• fixes Support split_by="token" in HierarchicalDocumentSplitter #12917

Proposed Changes:

• split_by now accepts "token", and a new keyword-only tokenizer_encoding parameter (default "o200k_base") picks the tiktoken encoding. Both are passed to the inner DocumentSplitter instances, so the token splitting from feat: DocumentSplitter — add split_by="token" mode using tiktoken #12529 is reused as is.
• New warm_up() that warms every inner splitter, so the encoding or the sentence tokenizer is loaded when the pipeline starts rather than on the first run(). It's idempotent because DocumentSplitter.warm_up() is, and tiktoken caches encodings by name, so all block sizes share one encoding object.
• tokenizer_encoding is serialized. Dicts saved before this change still load with the default.
• Updated the docstring, the docs page and added a release note.

How did you test it?

• Unit tests (no network): init passes both options to every inner splitter, serialization round trip, and warm_up() loads the encoding once per block size and is a no-op the second time. Updated the three existing tests that compare the exact serialized dict.
• Integration test with real tiktoken for o200k_base and cl100k_base: every block stays within its block size in tokens, parent/child links are correct, and the children of each block add up exactly to the block's text.
• The new and updated tests fail on main and pass with this change.
• hatch run test:unit test/components/preprocessors test/components/retrievers (includes AutoMergingRetriever): all pass. hatch run test:integration test/components/preprocessors: all pass.
• hatch run test:unit: 6494 passed. One unrelated failure on my Windows machine: test_byte_stream.py expects text/csv, but Windows maps .csv to Excel's MIME type.
• hatch run test:types and hatch run fmt-check are clean. The pre-commit hooks pass.

Notes for the reviewer

warm_up() also applies to split_by="sentence", which currently loads NLTK lazily on the first run(). Nothing changes for word, passage or page splits, where DocumentSplitter.warm_up() does nothing.

I used an AI assistant while working on this and reviewed and tested all changes myself.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@shivsin25 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[shivsin25]

The Integration / ubuntu-latest failure looks unrelated to this change: the only failing test is
test_link_content_fetcher_multiple_html_streams, which fetches live URLs (including google.com)
and failed after its automatic reruns. The new hierarchical splitter integration tests passed on
ubuntu, and the full integration job passed on windows and macos. Could someone re-run the failed
job when they get a chance? Thanks!

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/preprocessors
hierarchical_document_splitter.py
Project Total

This report was generated by python-coverage-comment-action

[shivsin25]

@davidsbatista is it ready to be merged or some chnages have to be made