fix(CSVDocumentSplitter): report column positions when read_csv_kwargs sets a header

[sclfcz]

Fixes #12965

Supersedes #12968, which the first-time-contributor limit closed before it could be reviewed (that limit is now lifted, the MetaFieldRanker PR having been merged). Same branch, same two changes, rebased on current main.

What

CSVDocumentSplitter overrides header to None so columns are integer positions, and it reads both the label and the ordering straight off them:

"col_idx_start": int(split_df.columns[0]),
split_dfs.sort(key=lambda dataframe: (dataframe.index[0], dataframe.columns[0]))

A caller passing the documented read_csv_kwargs={"header": 0} (or "infer") gets the first row as string labels, so int("name") raised ValueError for every sub-table — the document could not be split at all — and, once past that, the sub-tables were ordered alphabetically by header instead of by column position.

Change

Map the labels back to their position in the original frame and use that for both the reported index and the sort key, so col_idx_start keeps meaning "starting column index of the sub-table in the original table" regardless of how the CSV was read.

Testing

case	before	after
default (header=None)	[(0, 0), (3, 0)]	unchanged
{"header": 0}	ValueError: invalid literal for int() with base 10: 'name'	[(0, 0), (2, 0)]
{"header": "infer"}	same ValueError	[(0, 0), (2, 0)]
z,_,a with {"header": 0} (column split)	[(0, 2, 0), (0, 0, 1)] — sub-tables swapped	[(0, 0, 0), (0, 2, 1)]

pytest test/components/preprocessors/test_csv_document_splitter_header_kwargs.py → 3 passed; reverting either half fails exactly one test and leaves the others passing.

AI assistance

Written with an AI coding assistant: it located the casts and the sort key, ran the before/after table above and wrote the tests. I reviewed the diff, the reproduction output and the reasoning, and I take responsibility for the change.

[vercel]

@sclfcz is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Hi @sclfcz, thanks for your interest in contributing to Haystack! 🙏

⚠️ Issue #12965 is already being addressed by open pull request(s) #12981. Before opening a PR for an issue, please check whether a PR is already linked to it, and consider contributing to the existing PR instead. We may close duplicate PRs to keep the review queue manageable.

This is an automated message to help us keep the review queue healthy.

[chrikrah]

@sclfcz the header fix itself is right and your three tests pin it: reverted to the pre-fix lines with your
tests kept, 2 of the 3 fail. I would hold this until one regression is closed, and GitHub refuses a
request-changes review from an account without write access here, so the verdict is in this sentence rather
than on the review.

blocking: split_mode="row-wise" with any header-setting read_csv_kwargs now raises KeyError: 0 at
csv_document_splitter.py:159. _split_by_row rebuilds each sub-frame as pd.DataFrame(row).T, so its
columns are a fresh RangeIndex and never the parent labels your column_positions is keyed by. Pre-fix
int(0) succeeded, so this worked before. Nothing catches it: test_split_by_row and
test_split_by_row_with_empty_rows both use default read_csv_kwargs, and your new module has no row-wise
case.

column_positions.get(label, label) at both use sites fixes it and keeps your column case, measured below.

blocking: no release note. CONTRIBUTING.md:25 requires a file under releasenotes/notes from
hatch run release-note <name>, and CONTRIBUTING.md:317 says a CI check enforces it.

non-blocking: the tests are in a new module,
test/components/preprocessors/test_csv_document_splitter_header_kwargs.py, while this component's suite is
test_csv_document_splitter.py. The next person editing the splitter runs one of them.

Evidence, Python 3.13.15, editable install plus pytest and pandas, at d8bbe16:

$ python -m pytest test/components/preprocessors/test_csv_document_splitter.py \
      test/components/preprocessors/test_csv_document_splitter_header_kwargs.py -q
32 passed, 2 warnings in 0.93s

# CSVDocumentSplitter(split_mode="row-wise", read_csv_kwargs={"header": 0}) on "name,score\nAda,9\n\nBob,8"
$ python probe_rowwise.py                      # at d8bbe16
KeyError: 0
   File ".../csv_document_splitter.py", line 159, in <lambda>

$ python probe_rowwise.py                      # same tree, the two lines reverted
OK n=3 [(0, 0, 'Ada,9'), (1, 0, ','), (2, 0, 'Bob,8')]

$ python probe_rowwise.py                      # same tree, column_positions.get(label, label)
OK n=3 [(0, 0, 'Ada,9'), (1, 0, ','), (2, 0, 'Bob,8')]
# and the column case still splits: PROBE order: [(0, 0, '1'), (0, 2, '2')]

Every claim above came from that run. I did not run the rest of the suite or the pipeline serialisation
tests, and duplicate header labels are not a risk here because pandas renames them to a, a.1.

Will you take the .get fallback plus a row-wise test with a header, or would you rather key the sort on
df.columns.get_indexer and leave _split_by_row alone?

[anakin87]

Thank you!

See #12965 (comment)

[chrikrah]

@sclfcz at c8fba1a the release note passes reno lint .. The row-wise KeyError: 0 still reproduces, now at csv_document_splitter.py:157. CI fails at format now and at mypy after that. With the regression and CI fixed I have nothing further.

blocking: split_mode="row-wise" still raises when read_csv_kwargs sets the header or string names, and with integer labels such as names=[1, 0, 2] it silently reports col_idx_start=1 on every row. Neither lookup I offered last time fixes this: .get(label, label) and get_indexer both map label 0 to position 1, and get_indexer returns -1 for string labels. If _split_by_row gives each row frame the labels of df, all three cases work:

split_df = pd.DataFrame([row], columns=df.columns, index=[idx])  # replaces pd.DataFrame(row).T and the index line

The suite cannot see this regression, so please add this beside test_split_by_row. It fails 3 of 3 on c8fba1a and passes with the line above:

    @pytest.mark.parametrize(
        "read_csv_kwargs, expected",
        [({"header": 0}, [0]), ({"names": ["n", "s", "t"]}, [0, 0]), ({"names": [1, 0, 2]}, [0, 0])],
    )
    def test_split_by_row_reports_column_positions(self, read_csv_kwargs: dict, expected: list[int]) -> None:
        splitter = CSVDocumentSplitter(split_mode="row-wise", read_csv_kwargs=read_csv_kwargs)
        result = splitter.run([Document(content="a,b,c\n1,2,3\n")])["documents"]
        assert [d.meta["col_idx_start"] for d in result] == expected

blocking: the format failure on cc40db2 came from the license-header step, since test_csv_document_splitter_header_kwargs.py has no SPDX header. Moving its three tests into test_csv_document_splitter.py, where test/AGENTS.md asks for them, clears that step. c8fba1a then moved from haystack import ... below haystack.lazy_imports. Now ruff check stops on I001 before the header step runs, and hatch run fmt restores the order. Once format and the unit tests pass, mypy flags d.content.strip(). Asserting [d.content for d in result["documents"]] == ["1\n3\n", "2\n4\n"] instead passes mypy and the test.

# each tree a git archive export on PYTHONPATH; Python 3.12.3, pandas 3.0.6
# probe: split_mode="row-wise" on "name,score,note\nAda,9,x\n\nBob,8,\n", col_idx_start per document
$ python probe_rowwise.py <tree>
11a0dd2         header=0: [0, 0, 0]   names=[str]: [0, 0, 0, 0]   names=[1, 0, 2]: [0, 0, 0, 0]
c8fba1a         header=0: KeyError 0 (line 157)   names=[str]: KeyError 0 (line 157)   names=[1, 0, 2]: [1, 1, 1, 1]
.get fallback   header=0: [0, 0, 0]   names=[str]: [0, 0, 0, 0]   names=[1, 0, 2]: [1, 1, 1, 1]
get_indexer     header=0: [-1, -1, -1]   names=[str]: [-1, -1, -1, -1]   names=[1, 0, 2]: [1, 1, 1, 1]
parent labels   header=0: [0, 0, 0]   names=[str]: [0, 0, 0, 0]   names=[1, 0, 2]: [0, 0, 0, 0]

$ python -m pytest test/components/preprocessors/test_csv_document_splitter*.py -q
29 passed in 0.67s              # 11a0dd2, the existing module only
32 passed in 0.44s              # c8fba1a
2 failed, 30 passed in 0.52s    # c8fba1a with csv_document_splitter.py from 11a0dd2
32 passed in 0.39s              # 11a0dd2 with the new test
3 failed, 32 passed in 0.47s    # c8fba1a with the new test
35 passed in 0.40s              # c8fba1a with the new test and the _split_by_row line
35 passed in 0.39s              # the same plus the line-28 assert
# same probe rows and counts on Python 3.10.21, the version CI's unit job sets up, with pandas 2.3.3

$ ruff check .                  # c8fba1a; ruff 0.16.9, and 0.16.0 from .pre-commit-config.yaml gives the same
I001 [*] Import block is un-sorted or un-formatted
  --> haystack/components/preprocessors/csv_document_splitter.py:5:1
Found 1 error.
$ ruff check .                  # cc40db2
All checks passed!

$ podman run --rm -v "$PWD:/github/workspace:Z" ghcr.io/korandoru/hawkeye:v6.5.1 check    # digest as pinned in tests.yml
ERROR hawkeye::subcommand: subcommand.rs:121 Found header missing in files: ["./test/components/preprocessors/test_csv_document_splitter_header_kwargs.py"]    # c8fba1a
INFO hawkeye::subcommand: subcommand.rs:130 No missing header file has been found.    # c8fba1a with its three tests moved into test_csv_document_splitter.py

$ mypy haystack test            # hatch run test:types without --install-types; mypy 2.3.1, the repo's [tool.mypy]
Success: no issues found in 550 source files        # 11a0dd2
test/components/preprocessors/test_csv_document_splitter_header_kwargs.py:28: error: Item "None" of "str | None" has no attribute "strip"  [union-attr]
Found 1 error in 1 file (checked 551 source files)  # c8fba1a
Success: no issues found in 551 source files        # c8fba1a with the new test, the _split_by_row line and the line-28 assert

$ reno lint . > /dev/null 2>&1; echo $?    # reno 4.1.0
0
# not run: the rest of the unit suite; no other test names the splitter

@sjrl you approved split_by_row in #9031 and merged the last two changes to this file. Can this pull request change _split_by_row too?