Skip to content

Fix mmap reader yielding zero rows when the first row exceeds the head buffer - #338

Merged
vincentlaucsb merged 2 commits into
masterfrom
claude/csv-parser-issue-337-864c98
Oct 10, 2026
Merged

vincentlaucsb merged 2 commits into
masterfrom
claude/csv-parser-issue-337-864c98

Conversation

@vincentlaucsb

@vincentlaucsb vincentlaucsb commented Oct 10, 2026 •

Copy link
Copy Markdown
Owner

Fixes #337

Root cause

MmapParser::next() parsed the pre-read head buffer (capped at 500KB regardless of chunk_size) as the first window. When that window held no complete row, mmap_pos rewound to 0 and read_csv() called trim_header() against an empty queue. trim_header() popped nothing but still set header_trimmed = true, so the header row from the next window was never installed as column names. n_cols stayed at the 0 guessed from the head, and the default IGNORE_ROW policy dropped every row.

Changes

  • MmapParser::next(): if the head produced no complete row (and the source isn't exhausted), fall through to a full chunk_size read window from the rewound position. When the head already holds rows, it returns as before, so construction doesn't parse an extra window (500K-row file: ~12ms to construct, same as master). Head reuse and the rewind semantics the planned BOM refactor relies on are unchanged.
  • CSVReader::trim_header(): tracks how many header rows have been popped (header_rows_trimmed_) and only marks the header trimmed once the header row itself is consumed.
  • CSVReader::initial_read(): keeps reading until the header is consumed or EOF, and throws the existing "row larger than chunk size" error if a read makes no progress. Moved from the header into csv_reader.cpp with CSV_INLINE.

The initial_read() change also fixes a related bug on both paths: with header_row(N) and long preamble rows, the first read window could end before the header row, and the header row was then returned as data.

Behavior note

An oversized header row now throws from the CSVReader constructor instead of on first iteration. The constructor already rethrows first-window errors (e.g. UTF-16 BOM rejection), so this just moves one more first-window failure to the same place.

Tests

New [issue_337] cases in tests/test_edge_cases_large_rows.cpp, each with mmap and stream SECTIONs:

Case Before (mmap / stream) After
600KB header, chunk_size(2MB) fail / pass pass
short preamble + 600KB header, header_row(1) fail / pass pass
header row starts after first read window, header_row(2) fail / fail pass
header row > chunk_size throws pass / pass pass

Full csv_test suite passes locally (MSVC, VS 2026, Release), and the single_include_test smoke build compiles.

BOM variant left for the BOM stripping refactor, as noted in the issue.

🤖 Generated with Claude Code

…d buffer

MmapParser parsed its 500KB head buffer as the first window regardless of
chunk_size. When that window held no complete row, trim_header() ran against
an empty queue, marked the header as trimmed, and the reader silently produced
no column names and no rows.

- MmapParser::next() now falls through to a full read window after parsing
  the head, so the first read covers the head plus one window like
  StreamParser.
- trim_header() tracks how many header rows it has popped and only marks the
  header trimmed once the header row itself is consumed.
- initial_read() keeps reading until the header is consumed or EOF, throwing
  the existing "row larger than chunk size" error if a read makes no progress.
  This also fixes a header row that starts after the first read window ends,
  which affected both the mmap and stream paths.

Fixes #337

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.51%. Comparing base (eee1732) to head (f890eca).

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #338      +/-   ##
==========================================
- Coverage   91.54%   91.51%   -0.04%     
==========================================
  Files          55       55              
  Lines        3595     3605      +10     
  Branches     1068     1072       +4     
==========================================
+ Hits         3291     3299       +8     
- Misses        115      116       +1     
- Partials      189      190       +1     
Flag Coverage Δ
cpp 91.36% <100.00%> (-0.04%) ⬇️
python 93.64% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The unconditional fallthrough after parsing the head made construction parse
an extra full read window even when the head already produced rows, which
penalized metadata queries and early-stop readers (500K-row file: ~60ms vs
~12ms to construct). Fall through only when the head made no progress;
CSVReader::initial_read() keeps reading when the header row is still pending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vincentlaucsb
vincentlaucsb merged commit 796611f into master Oct 10, 2026
35 checks passed
@vincentlaucsb
vincentlaucsb deleted the claude/csv-parser-issue-337-864c98 branch October 10, 2026 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

mmap path silently yields zero rows when the first row exceeds the 500KB head buffer

1 participant