Release v5.5.0: move BOM handling into the parse orchestrator

ARCHITECTURE.md

@@ -7,7 +7,6 @@ Primary architecture document:
 
 Subsystem deep-dive:
 - include/internal/THREADSAFE_DEQUE_DESIGN.md
-- include/internal/BOMStrippingRefactor.md
 
 Operational/testing guidance:
 - AGENTS.md
@@ -18,6 +17,7 @@ Operational/testing guidance:
 Notes:
 - Internal architecture content lives under include/internal to stay close to implementation.
 - Detailed file map, parser data flow, and component relationship diagrams are maintained in include/internal/ARCHITECTURE.md.
+- Unicode BOM ownership and source-offset invariants are maintained in include/internal/ARCHITECTURE.md.
 - Queue synchronization details are maintained only in THREADSAFE_DEQUE_DESIGN.md to avoid duplication.
 - Always update or remove incorrect comments.
 - Public API comments should remain user-facing and avoid references to internal helper/function details.

docs/source/internal_deep_dives.md

@@ -2,11 +2,10 @@
 
 # Internal / Deep Dives
 
-These notes document parser internals, concurrency invariants, and refactor
-plans. They are useful when debugging or changing csv-parser itself, but most
+These notes document parser internals and concurrency invariants.
+They are useful when debugging or changing csv-parser itself, but most
 users can stay with the README and public API reference.
 
 - @subpage internal_architecture
 - @subpage csv_field_journey
 - @subpage threadsafe_deque_design
-- @subpage bom_stripping_refactor

include/csv.hpp

@@ -1,5 +1,5 @@
 /*
-CSV for C++, version 5.4.0
+CSV for C++, version 5.5.0
 https://github.com/vincentlaucsb/csv-parser
 
 MIT License

include/internal/ARCHITECTURE.md

@@ -71,7 +71,8 @@ Two independent parser paths exist and must be kept behaviorally aligned:
 
 - CSVParserCore
   - Templated, non-virtual byte parser core in parser/core.hpp.
-  - Owns DFA state, BOM handling, field/row construction, and concrete row-sink emission.
+  - Owns DFA state, field/row construction, and concrete row-sink emission.
+  - Receives source windows after leading BOM handling by the orchestrator.
   - Source adapters feed byte windows into it; it does not own file, mmap, or stream source mechanics.
 
 - PermissiveParsePolicy
@@ -97,6 +98,8 @@ Two independent parser paths exist and must be kept behaviorally aligned:
 
 - parser/orchestrator.hpp
   - Chooses serial CSVParserCore parsing or speculative parallel parsing for a byte window.
+  - Owns Unicode BOM handling: scans the first source window once, rejects UTF-16/UTF-32,
+    and hands parsers a BOM-free view while reporting completed prefixes in source bytes.
 
 - MmapParser
   - Reads chunks from memory maps and handles chunk-transition remainder.
@@ -226,6 +229,39 @@ Fields spanning chunk boundaries must not be split/corrupted.
 
 Mmap and stream parsers must preserve the same externally observable behavior.
 
+### Unicode BOM handling and source offsets
+
+`CSVParseOrchestrator` checks the first source window once, before dispatching
+serial or speculative parsing. It rejects UTF-16/UTF-32 with a transcoder error
+and records a leading UTF-8 BOM for `CSVReader::utf8_bom()`. Only that first BOM
+is removed; another BOM remains part of the CSV data. Parser cores and
+speculative chunks carry no BOM state. Format guessing separately checks and
+strips the leading BOM once per `guess_format()` call, before scoring delimiter
+candidates.
+
+Both adapters' first windows include their pre-read head buffer, up to 500KB;
+the stream adapter may append more bytes before parsing. A first window too
+short to classify a BOM therefore occurs only at end of input.
+
+Source adapters advance in original source bytes while parsers receive
+`chunk.substr(skip)` at `base_offset + skip`:
+
+- `RawCSVData::source_start` identifies the source position of the parsed view.
+  A first row after a UTF-8 BOM starts at byte 3, and its `raw_str()` excludes
+  those three bytes. Field offsets remain relative to the backing view.
+- `CSVParseWindowResult::complete_prefix_length` includes the skipped bytes,
+  also reported as `skipped_prefix_length`. Mmap re-reads and stream leftovers
+  must begin after this prefix, even when the window completed no row.
+- Mmap head-window fallback uses `completed_row()` rather than position
+  arithmetic: skipping a BOM advances the source position without completing
+  a row. A BOM-only source emits no row and still consumes the BOM.
+
+Regression coverage lives in `tests/test_read_csv.cpp` (`[read_unicode_bom]`)
+for both adapters, guessed/explicit formats, encoding rejection, tiny inputs,
+row offsets, and a 600K-row file parsed serially and speculatively. The BOM
+variant of `[issue_337]` in `tests/test_edge_cases_large_rows.cpp` covers a
+header longer than the mmap head buffer.
+
 ### Field storage and conversion contract
 
 The parser realizes doubled-quote fields into `RawCSVData::quote_arena`; ordinary

include/internal/JOURNEY_OF_A_CSV_FIELD.md

@@ -12,7 +12,9 @@ have changed shape.
 A CSV field is usually not copied while parsing.
 
 1. A source adapter (`MmapParser` or `StreamParser`) provides a byte window.
-2. `CSVParseOrchestrator` chooses serial parsing or speculative parallel parsing.
+2. `CSVParseOrchestrator` strips a leading UTF-8 BOM from the first window
+   (rejecting UTF-16/UTF-32), then chooses serial parsing or speculative
+   parallel parsing.
 3. `CSVParserCore` walks bytes and records field boundaries in `RawCSVFieldList`.
 4. `CSVRow` stores a shared pointer to `RawCSVData`.
 5. `CSVRow` slices either the original bytes or parser-realized quoted-field
@@ -84,7 +86,6 @@ CSVReader
 - quote state
 - escaped quote pairs
 - CR, LF, and CRLF row endings
-- UTF-8 BOM skip
 - field boundary recording
 - row emission
 
@@ -143,15 +144,14 @@ does not know whether the row came from serial parsing or speculative parsing.
 Low-level flow:
 
 ```text
-source window bytes
+source window bytes (BOM already stripped by CSVParseOrchestrator)
   |
   v
 make_speculative_parse_chunks()
   |
   +--> chunk 0 bytes + owner + offset + sequence_number
   |       |
   |       +--> starts_at_record_boundary = true
-  |       +--> scan_bom = true
   |       +--> assumed_start_state = outside quotes
   |
   +--> chunk N bytes + owner + offset + sequence_number
@@ -173,7 +173,6 @@ make_speculative_parse_chunks()
           +--> sequence_number
           +--> assumed_start_state
           +--> starts_at_record_boundary
-          +--> scan_bom
 
 ParallelCSVParser::parse_chunks()
   |

include/internal/parser/core.hpp

@@ -95,9 +95,6 @@ namespace csv {
             return make_ws_flags(flags.data(), flags.size());
         }
 
-        /** Return the number of leading BOM bytes to skip, or throw for unsupported Unicode encodings. */
-        CSV_INLINE size_t get_bom_skip_or_throw(csv::string_view data, bool& utf8_bom);
-
         /** Explicit DFA state at a parse boundary.
          *
          *  This is intentionally about parser control flow, not field metadata.
@@ -132,15 +129,12 @@ namespace csv {
             ParserChunkOptions() noexcept = default;
             explicit ParserChunkOptions(
                 ParserDFAState initial_state,
-                bool scan_bom = true,
                 size_t source_start = 0
             ) noexcept
                 : initial_state(initial_state),
-                  scan_bom(scan_bom),
                   source_start(source_start) {}
 
             ParserDFAState initial_state;
-            bool scan_bom = true;
             size_t source_start = 0;
         };
 
@@ -161,7 +155,16 @@ namespace csv {
         };
 
         struct CSVParseWindowResult {
+            /** Source bytes fully consumed by this window, including skipped_prefix_length. */
             size_t complete_prefix_length = 0;
+
+            /** Leading source bytes consumed without being parsed (a UTF-8 BOM). */
+            size_t skipped_prefix_length = 0;
+
+            /** Whether this window produced at least one complete row. */
+            bool completed_row() const noexcept {
+                return this->complete_prefix_length > this->skipped_prefix_length;
+            }
         };
     }
 
@@ -417,9 +420,6 @@ namespace csv {
                 return quote_escape_flag(parse_flag(ch), this->quote_escape_);
             }
 
-            /** Whether or not this CSV has a UTF-8 byte order mark. */
-            CONSTEXPR bool utf8_bom() const { return this->utf8_bom_; }
-
             void set_output(RowSink& output) noexcept {
                 this->output_ = &output;
             }
@@ -485,7 +485,7 @@ namespace csv {
                     chunk,
                     std::move(owner),
                     output,
-                    ParserChunkOptions(this->initial_state_, true, source_start)
+                    ParserChunkOptions(this->initial_state_, source_start)
                 );
             }
 
@@ -550,9 +550,6 @@ namespace csv {
                 this->pending_linefeed_ = start_state.pending_linefeed;
                 this->data_pos_ = 0;
                 this->current_row_start() = 0;
-                if (this->scan_bom_for_current_chunk_) {
-                    this->strip_unicode_bom();
-                }
 
                 auto& in = this->data_ptr_->data;
 
@@ -670,15 +667,10 @@ namespace csv {
             bool field_has_double_quote_ = false;
             ParserDFAState initial_state_;
             ParserDFAState ending_state_;
-            bool scan_bom_for_current_chunk_ = true;
 
             /** Where we are in the current data block. */
             size_t data_pos_ = 0;
 
-            /** Whether or not an attempt to find Unicode BOM has been made. */
-            bool unicode_bom_scan_ = false;
-            bool utf8_bom_ = false;
-
             RowSink* output_ = nullptr;
             ParsePolicy policy_;
             FieldPolicy field_policy_;
@@ -793,16 +785,6 @@ namespace csv {
                 this->emit_row(std::move(current_row_));
             }
 
-            /** Handle possible Unicode byte order mark. */
-            void strip_unicode_bom() {
-                auto& data = this->data_ptr_->data;
-
-                if (!this->unicode_bom_scan_) {
-                    this->data_pos_ += get_bom_skip_or_throw(data, this->utf8_bom_);
-                    this->unicode_bom_scan_ = true;
-                }
-            }
-
             ParserChunkResult parse_prepared_chunk(
                 csv::string_view chunk,
                 std::shared_ptr<void> owner,
@@ -819,7 +801,6 @@ namespace csv {
                 this->data_ptr_->field_scalars.reserve_for_source_size(chunk.size());
                 this->data_ptr_->quote_arena.reserve_for_source_size(chunk.size());
                 this->initial_state_ = options.initial_state;
-                this->scan_bom_for_current_chunk_ = options.scan_bom;
                 this->current_row_ = this->row_policy_.make_initial_row(this->data_ptr_);
                 this->policy_.begin_chunk(this->data_ptr_);
                 this->policy_.begin_row(this->current_row_);
@@ -837,7 +818,6 @@ namespace csv {
                 this->initial_state_ = this->ending_state_.pending_linefeed
                     ? this->ending_state_
                     : ParserDFAState{};
-                this->scan_bom_for_current_chunk_ = true;
                 return remainder;
             }
         };

include/internal/parser/driver.hpp

@@ -29,6 +29,13 @@
 
 namespace csv {
     namespace internals {
+        /** Return the number of leading BOM bytes to skip, or throw for unsupported Unicode encodings.
+         *
+         *  BOM handling happens at the source-window level (CSVParseOrchestrator and
+         *  format guessing); CSVParserCore only ever sees BOM-free bytes.
+         */
+        CSV_INLINE size_t get_bom_skip_or_throw(csv::string_view data, bool& utf8_bom);
+
         namespace parser {
         struct GuessScore {
             size_t header;
@@ -175,7 +182,7 @@ namespace csv {
             virtual bool utf8_bom() const noexcept {
                 return this->parse_orchestrator_
                     ? this->parse_orchestrator_->utf8_bom()
-                    : CSVParserCore<>::utf8_bom();
+                    : false;
             }
 
         protected:

include/internal/parser/guessing.cpp

@@ -76,6 +76,12 @@ namespace csv {
              *  Header detection: If first row has >= columns than mode, use row 0.
              *  Otherwise use the first row with the mode length.
              */
+            // The parser core does not handle BOMs, so strip one here (once, not
+            // per candidate). This also rejects UTF-16/UTF-32 before guessing
+            // scores mis-decoded bytes.
+            bool utf8_bom = false;
+            head = head.substr(get_bom_skip_or_throw(head, utf8_bom));
+
             CSVFormat format;
             size_t max_score = 0;
             size_t header = 0;