Repository navigation
Fix mmap reader yielding zero rows when the first row exceeds the head buffer - #338
Merged
Merged
Conversation
…d buffer MmapParser parsed its 500KB head buffer as the first window regardless of chunk_size. When that window held no complete row, trim_header() ran against an empty queue, marked the header as trimmed, and the reader silently produced no column names and no rows. - MmapParser::next() now falls through to a full read window after parsing the head, so the first read covers the head plus one window like StreamParser. - trim_header() tracks how many header rows it has popped and only marks the header trimmed once the header row itself is consumed. - initial_read() keeps reading until the header is consumed or EOF, throwing the existing "row larger than chunk size" error if a read makes no progress. This also fixes a header row that starts after the first read window ends, which affected both the mmap and stream paths. Fixes #337 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #338 +/- ##
==========================================
- Coverage 91.54% 91.51% -0.04%
==========================================
Files 55 55
Lines 3595 3605 +10
Branches 1068 1072 +4
==========================================
+ Hits 3291 3299 +8
- Misses 115 116 +1
- Partials 189 190 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The unconditional fallthrough after parsing the head made construction parse an extra full read window even when the head already produced rows, which penalized metadata queries and early-stop readers (500K-row file: ~60ms vs ~12ms to construct). Fall through only when the head made no progress; CSVReader::initial_read() keeps reading when the header row is still pending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #337
Root cause
MmapParser::next()parsed the pre-read head buffer (capped at 500KB regardless ofchunk_size) as the first window. When that window held no complete row,mmap_posrewound to 0 andread_csv()calledtrim_header()against an empty queue.trim_header()popped nothing but still setheader_trimmed = true, so the header row from the next window was never installed as column names.n_colsstayed at the 0 guessed from the head, and the defaultIGNORE_ROWpolicy dropped every row.Changes
MmapParser::next(): if the head produced no complete row (and the source isn't exhausted), fall through to a fullchunk_sizeread window from the rewound position. When the head already holds rows, it returns as before, so construction doesn't parse an extra window (500K-row file: ~12ms to construct, same asmaster). Head reuse and the rewind semantics the planned BOM refactor relies on are unchanged.CSVReader::trim_header(): tracks how many header rows have been popped (header_rows_trimmed_) and only marks the header trimmed once the header row itself is consumed.CSVReader::initial_read(): keeps reading until the header is consumed or EOF, and throws the existing "row larger than chunk size" error if a read makes no progress. Moved from the header intocsv_reader.cppwithCSV_INLINE.The
initial_read()change also fixes a related bug on both paths: withheader_row(N)and long preamble rows, the first read window could end before the header row, and the header row was then returned as data.Behavior note
An oversized header row now throws from the
CSVReaderconstructor instead of on first iteration. The constructor already rethrows first-window errors (e.g. UTF-16 BOM rejection), so this just moves one more first-window failure to the same place.Tests
New
[issue_337]cases intests/test_edge_cases_large_rows.cpp, each with mmap and streamSECTIONs:chunk_size(2MB)header_row(1)header_row(2)chunk_sizethrowsFull
csv_testsuite passes locally (MSVC, VS 2026, Release), and thesingle_include_testsmoke build compiles.BOM variant left for the BOM stripping refactor, as noted in the issue.
🤖 Generated with Claude Code