Repository navigation
fix: count page breaks in overlapping regions once in RecursiveDocumentSplitter - #12909
Closed
TimurRakhmatullin86 wants to merge 1 commit into
Closed
TimurRakhmatullin86 wants to merge 1 commit into
TimurRakhmatullin86 wants to merge 1 commit into
Conversation
…ntSplitter With split_overlap > 0 the overlapping tail of a chunk is repeated at the start of the next chunk, so a page break character inside that region was counted once per chunk containing it. Every chunk after the break then carried a page_number shifted upwards by one per repetition, which can exceed the number of pages actually present in the document. Advance the page counter only over the characters the next chunk does not repeat, mirroring how current_position is already advanced. Behavior with split_overlap=0 is unchanged.
TimurRakhmatullin86
requested review from
bogdankostic
and removed request for
a team
September 24, 2026 04:19
Contributor
|
@TimurRakhmatullin86 is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
Contributor
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
This was referenced Sep 24, 2026
Contributor
|
Thanks for finding and reporting this, @TimurRakhmatullin86! The overlap drift is now fixed on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
No existing issue; found by code inspection.
Proposed Changes:
RecursiveDocumentSplittercounted every\fpage-break in each emitted chunk when advancing its running page counter. Withsplit_overlap > 0the overlapping tail of a chunk is repeated verbatim at the start of the next chunk, so a page break inside that region was counted once per chunk containing it. Every chunk after such a break then carried apage_numbershifted upward by one per repetition — sometimes exceeding the number of pages actually present in the document.The fix advances the page counter only over the characters the next chunk does not repeat (
chunk.count("\f", 0, len(chunk) - overlap_char_len)), mirroring howcurrent_positionis already advanced. Each chunk's ownpage_numberstill reflects the breaks within it, so the observable convention is unchanged. Withsplit_overlap = 0,overlap_char_lenis always 0 and behavior is bit-for-bit identical.How did you test it?
Added
test_run_count_page_breaks_once_with_overlap_split_unit_char: a two-page document ("This is page one.\fThis is page two, it is longer.") split withsplit_overlap=5, asserting the chunk contents and that the chunks following the overlapped break stay on page 2 instead of drifting to page 3+. The test fails without the fix (assert 3 == 2) and passes with it.ruff checkandruff format --checkare clean on the changed files.Notes for the reviewer
The change is orthogonal to the mid-chunk page-number convention discussion (e.g. the earlier #12590); it does not alter any existing test's expected page numbers, it only removes a double-count that could push page numbers past the real page count. This is the same bug class already fixed for
MarkdownHeaderSplitterin #12619.Checklist
fix:).