fix: keep trailing whitespace and page breaks in RecursiveDocumentSplitter sentence splits

[Pothan0]

Related Issues

• No existing issue; found while testing page metadata in RecursiveDocumentSplitter.

Proposed Changes:

RecursiveDocumentSplitter drops whitespace at the end of a text piece when it splits by "sentence" with keep_white_spaces=True (the default). NLTK's span tokenizer strips trailing whitespace, so a trailing \f or \n\n disappears from the chunk. The chunks then no longer add up to the original text, page_number stops advancing, and split_idx_start drifts for every chunk after it.

This slices the original text between sentence start offsets instead, so no characters are lost. keep_white_spaces=False keeps the current behavior.

Repro:

text = "Intro para one. It has two sentences.\f\n\nSecond page starts. Also two sentences here.\f\n\nThird page. End."
splitter = RecursiveDocumentSplitter(split_length=6, split_overlap=0, split_unit="word")
splitter.warm_up()
docs = splitter.run([Document(content=text)])["documents"]

On main every chunk reports page_number 1 and "".join(d.content for d in docs) != text.

How did you test it?

• Added a regression test (fails on main, passes with this change).
• Existing test_recursive_splitter.py and test_sentence_tokenizer.py pass.
• ruff check and ruff format clean.

Notes for the reviewer

This doesn't change the page_number convention discussed in #12926; it only stops characters from being lost.

Prepared with an AI assistant; I've reviewed the change and the tests.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have added unit tests.
• I've used a conventional commit type for the PR title.
• I have added a release note file.

[vercel]

@Pothan0 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[julian-risch]

Thank you for your efforts! We're closing this PR because it's obsolete: #13006, merged on October 1, fixes the same issue, and this PR's own test passes on main.