fix: correct RecursiveDocumentSplitter split_idx_start for word overlap ending in a separator

[anishmehta24]

Related Issues

• fixes RecursiveDocumentSplitter split_idx_start is off by the trailing separator with split_unit="word" and split_overlap > 0 #12756

Proposed Changes:

With split_unit="word" and split_overlap > 0, RecursiveDocumentSplitter._run_one advanced the running position by len(chunk) - len(overlap_str). In word mode _get_overlap rebuilds the overlap as " ".join(last_words), which has no trailing whitespace, while in the chunk those words are followed by the separator (" " with the default separators). The subtraction therefore left the position one separator past where the next chunk starts, so every chunk after the first had split_idx_start shifted, and the _split_overlap ranges (derived from the same position) inherited the error. The repro from #11710 still showed this on main.

This PR adds _overlap_char_length, which for word mode measures the overlap from the first overlapping word (found with \S+ positions) to the end of the chunk, trailing whitespace included, and returns len(overlap) for the other units. current_position now lands on the first overlapping word, which is where the next chunk begins. Character and token splitting are unchanged.

Before / after for split_length=4, split_overlap=1, split_unit="word" on the #11710 sentence:

chunk	split_idx_start before	after	text.index(content)
"one. This is sentence"	18	17	17
"sentence two. This is"	31	30	30
"is sentence three. This"	50	49	49
"This is sentence four."	69	68	68

How did you test it?

Three new unit tests in test/components/preprocessors/test_recursive_splitter.py: the #11710 sentence with an explicit " " separator (asserting exact chunk contents and offsets), a plain-text run with the default separators, and a split_unit="char" control. A shared helper asserts text[split_idx_start:] starts with every chunk and that the _split_overlap ranges of consecutive chunks select the same text. The two word-mode tests fail on main and pass with this change; the whole module (63 tests) passes. ruff check, ruff format --check and mypy are clean on the changed files.

Notes for the reviewer

• Word mode still rebuilds chunk text with single spaces, so on input with newlines between words a chunk can differ from the source slice (RecursiveDocumentSplitter with overlap emits chunks not present in the source #12281 territory); this PR only fixes the offset arithmetic and does not change chunk contents.
• Touches the position-advance block near the code that fix: page_number for form feeds inside a RecursiveDocumentSplitter chunk #12590 changes for page_number; the hunks are adjacent but distinct.
• Release note added under releasenotes/notes/.
• This PR was written with an AI assistant (Claude Code). I found the bug by property-testing the splitters, reviewed the change, and ran the tests and checks locally.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@anishmehta24 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[HaystackBot]

Hi @anishmehta24, thanks a lot for your contribution! 🙏

We noticed that the Contributor License Agreement (CLA) check (license/cla) hasn't passed yet, so we've temporarily moved this PR to draft and paused the review assignment.

To get your PR reviewed, please sign the CLA via the link in the license/cla check below (or in the CLA bot comment). As soon as the check turns green, this PR will automatically be marked ready for review again and a reviewer will be re-assigned.

[HaystackBot]

Thanks for signing the CLA, @anishmehta24! 🎉 This PR is now ready for review again and the reviewer has been re-assigned.

[anishmehta24]

@anakin87 CLA is signed now, so this should be ready for review.

[anakin87]

Thank you!

This was superseded by #13006