Skip to content

fix: count page breaks in overlapping regions once in RecursiveDocumentSplitter - #12909

Closed
TimurRakhmatullin86 wants to merge 1 commit into
deepset-ai:mainfrom
TimurRakhmatullin86:fix/recursive-splitter-overlap-page-count
Closed

TimurRakhmatullin86 wants to merge 1 commit into
deepset-ai:mainfrom
TimurRakhmatullin86:fix/recursive-splitter-overlap-page-count

Conversation

@TimurRakhmatullin86

Copy link
Copy Markdown
Contributor

Related Issues

No existing issue; found by code inspection.

Proposed Changes:

RecursiveDocumentSplitter counted every \f page-break in each emitted chunk when advancing its running page counter. With split_overlap > 0 the overlapping tail of a chunk is repeated verbatim at the start of the next chunk, so a page break inside that region was counted once per chunk containing it. Every chunk after such a break then carried a page_number shifted upward by one per repetition — sometimes exceeding the number of pages actually present in the document.

The fix advances the page counter only over the characters the next chunk does not repeat (chunk.count("\f", 0, len(chunk) - overlap_char_len)), mirroring how current_position is already advanced. Each chunk's own page_number still reflects the breaks within it, so the observable convention is unchanged. With split_overlap = 0, overlap_char_len is always 0 and behavior is bit-for-bit identical.

How did you test it?

Added test_run_count_page_breaks_once_with_overlap_split_unit_char: a two-page document ("This is page one.\fThis is page two, it is longer.") split with split_overlap=5, asserting the chunk contents and that the chunks following the overlapped break stay on page 2 instead of drifting to page 3+. The test fails without the fix (assert 3 == 2) and passes with it. ruff check and ruff format --check are clean on the changed files.

Notes for the reviewer

The change is orthogonal to the mid-chunk page-number convention discussion (e.g. the earlier #12590); it does not alter any existing test's expected page numbers, it only removes a double-count that could push page numbers past the real page count. This is the same bug class already fixed for MarkdownHeaderSplitter in #12619.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have added unit tests and updated the docstrings.
  • I've used a conventional commit type for my PR title (fix:).
  • I have added a release note file.

…ntSplitter

With split_overlap > 0 the overlapping tail of a chunk is repeated at the
start of the next chunk, so a page break character inside that region was
counted once per chunk containing it. Every chunk after the break then
carried a page_number shifted upwards by one per repetition, which can
exceed the number of pages actually present in the document.

Advance the page counter only over the characters the next chunk does not
repeat, mirroring how current_position is already advanced. Behavior with
split_overlap=0 is unchanged.
@TimurRakhmatullin86
TimurRakhmatullin86 requested a review from a team as a code owner September 24, 2026 04:19
@TimurRakhmatullin86
TimurRakhmatullin86 requested review from bogdankostic and removed request for a team September 24, 2026 04:19
@vercel

vercel Bot commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

@TimurRakhmatullin86 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added topic:tests type:documentation Improvements on the docs labels Sep 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/components/preprocessors
  recursive_splitter.py
Project Total  

This report was generated by python-coverage-comment-action

@bogdankostic

Copy link
Copy Markdown
Contributor

Thanks for finding and reporting this, @TimurRakhmatullin86! The overlap drift is now fixed on main by #13006, which removed RecursiveDocumentSplitter's running page counter and calculates page_number from the chunk's offset instead, so a page break in an overlapping region can no longer be counted twice. Your reproduction is covered by test_run_page_number_does_not_drift_with_overlap. Closing as superseded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:tests type:documentation Improvements on the docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants