Skip to content

page_number in RecursiveDocumentSplitter points to the page the chunk ends on #12926

Description

@bogdankostic

Summary

While reviewing #12909, I noticed that our three text splitters disagree on what page_number means.

Verified on main (8de125819), same 3-page source in each case:

splitter chunk spans reported
DocumentSplitter 'aa bb\fcc ' pages 1→2 1
MarkdownHeaderSplitter '# H1\nAAAA\fBBBB\n' pages 1→2 1
RecursiveDocumentSplitter 'AAAA\fBBBB' pages 1→2 2

So a chunk that is 95% on page 1 with a few characters spilling onto page 2 reports page 1 in two splitters and page 2 in the third. Since page_number is what users show as "this answer came from page N", the difference is user-visible.

Why now

#12909 fixes a bug in RecursiveDocumentSplitter: with split_overlap > 0 a \f inside an overlapping tail was counted once per chunk, so page numbers drifted upwards and could exceed the document's page count.

The PR's patch is on top of the end-of-chunk convention.

Prior art

@dylanpulver already proposed a fix for this in #12590. It was auto-closed after two weeks because the CLA was never signed.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High priority, add to the next sprint

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions