Describe the bug
With split_unit="word" and split_overlap > 0, RecursiveDocumentSplitter sets split_idx_start one character (the length of the previous chunk's trailing separator) too far for every chunk after the first, and the _split_overlap ranges inherit the same shift. The repro from #11710 still fails on main (f1e61b5).
Cause, in _run_one: the position advances by len(chunk) - len(overlap_str), but in word mode _get_overlap rebuilds the overlap as " ".join(last_words), which has no trailing whitespace, while in the chunk those words are followed by the separator (" " with the default separators). So the subtraction leaves the position one separator past where the next chunk actually starts.
Error message
None, the metadata is silently wrong.
Expected behavior
document.content[split_idx_start : split_idx_start + len(chunk.content)] == chunk.content for every chunk (as the existing text.index(...) tests assert for the other modes), and the _split_overlap ranges of consecutive chunks point at the same text.
Additional context
from haystack import Document
from haystack.components.preprocessors import RecursiveDocumentSplitter
text = "This is sentence one. This is sentence two. This is sentence three. This is sentence four."
splitter = RecursiveDocumentSplitter(split_length=4, split_overlap=1, split_unit="word")
splitter.warm_up()
for doc in splitter.run(documents=[Document(content=text)])["documents"]:
start = doc.meta["split_idx_start"]
print(start, text.index(doc.content), repr(doc.content))
# 0 0 'This is sentence one. '
# 18 17 'one. This is sentence'
# 31 30 'sentence two. This is'
# 50 49 'is sentence three. This'
# 69 68 'This is sentence four.'
Same with separators=[" "] explicitly. split_unit="char" is correct.
To Reproduce
Run the snippet above.
FAQ Check
System:
- OS: Windows 11
- Haystack version (commit or version number): main at
f1e61b5
- Python: 3.12
I have a fix with regression tests and will open a PR. Found by property-testing the splitters; report and fix prepared with an AI assistant (Claude Code) and reviewed by me.
Describe the bug
With
split_unit="word"andsplit_overlap > 0,RecursiveDocumentSplittersetssplit_idx_startone character (the length of the previous chunk's trailing separator) too far for every chunk after the first, and the_split_overlapranges inherit the same shift. The repro from #11710 still fails onmain(f1e61b5).Cause, in
_run_one: the position advances bylen(chunk) - len(overlap_str), but in word mode_get_overlaprebuilds the overlap as" ".join(last_words), which has no trailing whitespace, while in the chunk those words are followed by the separator (" "with the default separators). So the subtraction leaves the position one separator past where the next chunk actually starts.Error message
None, the metadata is silently wrong.
Expected behavior
document.content[split_idx_start : split_idx_start + len(chunk.content)] == chunk.contentfor every chunk (as the existingtext.index(...)tests assert for the other modes), and the_split_overlapranges of consecutive chunks point at the same text.Additional context
Same with
separators=[" "]explicitly.split_unit="char"is correct.To Reproduce
Run the snippet above.
FAQ Check
System:
f1e61b5I have a fix with regression tests and will open a PR. Found by property-testing the splitters; report and fix prepared with an AI assistant (Claude Code) and reviewed by me.