Separate clean and dirty DataFrame handling without virtual dispatch

[vincentlaucsb]

Summary

The DataFrame code was becoming unmaintainable as evidenced by the issues discovered in #332 and #333.

• The DataFrame was forced to deal with both clean and edited data frame concerns and had grown to >1000 LOC.
• If a data frame was in an edited state, and a property or method logically depended on that state, forgetting to check if sparse edits existed was an easy way to cause logical errors

This architecture centralizes clean/dirty state handling, without imposing the overhead of virtual function dispatch.

Bonus: A DataFrameColumn refactor where we grab the raw string view from CSVRow instead of going through CSVField has sped up chunk_parallel_apply() column scanning operations by 25% in our preliminary benchmarks.

Full Details

DataFrame delegates storage behavior to internal CleanDataFrame and DirtyDataFrame classes while preserving its public API and C++11 compatibility. DirtyDataFrame owns edit overlays, visible column names, column mapping, and structural column insertion. A stable storage controller selects the concrete backend through an atomic tagged pointer and direct calls, with no virtual dispatch.

The first edit publishes a fully initialized dirty backend while retaining parsed storage, so existing row and cell proxies observe edits safely. Materialization returns to clean storage, selection snapshots edits independently, and stored keys—including string_view keys—retain their values and backing bytes across structural changes.

Read-only column views now extract decoded, trimmed field bytes directly through CSVRow's existing internal routine, avoiding CSVField construction and scalar metadata retrieval. Dirty overlays retain their existing lookup and locking. This adds no public API or object-layout change. Paired MSVC measurements showed about 25% less elapsed time for the full chunk_parallel_apply pipeline in both compiled-library and generated-header builds; a four-pair check of the integrated implementation reproduced that reduction.

The retained investigation includes methodology, raw data, reproduction instructions, and the rejected physical-column caching alternative. Standalone benchmark tools live outside the amalgamation input tree.

Fixes #333. Focused regressions in tests/test_data_frame_dirty.cpp exercise mmap and stream inputs, backend transitions, captured proxies, concurrent first edits, schema mapping, zero-column rows, selection, key preservation, and direct-view equivalence for quoted/trimmed/materialized fields.

Validation of the latest commit:

• MSVC C++20 DataFrame/ETL: 70 cases, 109072 assertions.
• Writer: 16 cases, 56 assertions.
• Generated single-header multi-TU, C++14, threads disabled: 82 cases, 109119 assertions.
• Dirty/backend regressions under MSVC AddressSanitizer: 14 cases, 559 assertions.
• Retained benchmark runner: warmups and four alternating baseline/integrated pairs, matching checksums.
• Astra reviewed the backend refactor and direct extraction, independently verified retained measurement summaries, and found no high-priority findings. Its minor documentation link finding was fixed and all report links resolve.

Cross-platform CI runs on the updated PR commit; the preceding backend-refactor commit passed the full platform matrix including Linux C++11. Performance percentages are measurements from the documented MSVC/processor setup.

[codecov]

Codecov Report

❌ Patch coverage is 91.03774% with 38 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.59%. Comparing base (9dbbd2e) to head (d7febfe).
⚠️ Report is 6 commits behind head on master.

Files with missing lines	Patch %	Lines
include/internal/data_frame/dirty_data_frame.hpp	82.65%	0 Missing and 17 partials ⚠️
include/internal/data_frame/data_frame.hpp	85.52%	0 Missing and 11 partials ⚠️
include/internal/data_frame/row_storage.hpp	92.15%	0 Missing and 4 partials ⚠️
include/internal/data_frame/data_frame_storage.hpp	96.38%	0 Missing and 3 partials ⚠️
include/internal/data_frame/key_storage.hpp	92.00%	1 Missing and 1 partial ⚠️
include/internal/data_frame/data_frame_cell.hpp	98.11%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #335      +/-   ##
==========================================
- Coverage   91.60%   91.59%   -0.02%     
==========================================
  Files          55       61       +6     
  Lines        3574     3770     +196     
  Branches     1058     1097      +39     
==========================================
+ Hits         3274     3453     +179     
+ Misses        115      110       -5     
- Partials      185      207      +22     

Flag	Coverage Δ
cpp	91.45% <91.03%> (-0.01%)	⬇️
python	93.64% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.