Summary
While reviewing #12909, I noticed that our three text splitters disagree on what page_number means.
Verified on main (8de125819), same 3-page source in each case:
| splitter |
chunk |
spans |
reported |
DocumentSplitter |
'aa bb\fcc ' |
pages 1→2 |
1 |
MarkdownHeaderSplitter |
'# H1\nAAAA\fBBBB\n' |
pages 1→2 |
1 |
RecursiveDocumentSplitter |
'AAAA\fBBBB' |
pages 1→2 |
2 |
So a chunk that is 95% on page 1 with a few characters spilling onto page 2 reports page 1 in two splitters and page 2 in the third. Since page_number is what users show as "this answer came from page N", the difference is user-visible.
Why now
#12909 fixes a bug in RecursiveDocumentSplitter: with split_overlap > 0 a \f inside an overlapping tail was counted once per chunk, so page numbers drifted upwards and could exceed the document's page count.
The PR's patch is on top of the end-of-chunk convention.
Prior art
@dylanpulver already proposed a fix for this in #12590. It was auto-closed after two weeks because the CLA was never signed.
Summary
While reviewing #12909, I noticed that our three text splitters disagree on what
page_numbermeans.DocumentSplitter→ the page the chunk starts onMarkdownHeaderSplitter→ the page the chunk starts on (since fix: MarkdownHeaderSplitter page_number drifts upwards when split_overlap is used #12619)RecursiveDocumentSplitter→ the page the chunk ends onVerified on
main(8de125819), same 3-page source in each case:DocumentSplitter'aa bb\fcc 'MarkdownHeaderSplitter'# H1\nAAAA\fBBBB\n'RecursiveDocumentSplitter'AAAA\fBBBB'So a chunk that is 95% on page 1 with a few characters spilling onto page 2 reports page 1 in two splitters and page 2 in the third. Since
page_numberis what users show as "this answer came from page N", the difference is user-visible.Why now
#12909 fixes a bug in
RecursiveDocumentSplitter: withsplit_overlap > 0a\finside an overlapping tail was counted once per chunk, so page numbers drifted upwards and could exceed the document's page count.The PR's patch is on top of the end-of-chunk convention.
Prior art
@dylanpulver already proposed a fix for this in #12590. It was auto-closed after two weeks because the CLA was never signed.