fix(CSVDocumentSplitter): use positional column index instead of column label - #12981
Tyagiquamar wants to merge 1 commit into
Conversation
…mn label read_csv_kwargs allows overriding how pandas reads the CSV, e.g. setting header=0 so the first row becomes the column labels. The splitter then crashed at int(split_df.columns[0]) because the label is a string, and sorted sub-tables by label instead of position. Resolve the first column of each sub-table to its positional index in the original DataFrame for both the sort and the col_idx_start metadata. Fixes deepset-ai#12965 Signed-off-by: Tyagiquamar <Tyagiquamar@users.noreply.github.com>
|
@Tyagiquamar is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
|
|
|
Hi @Tyagiquamar, thanks a lot for your contribution! 🙏 We noticed that the Contributor License Agreement (CLA) check ( To get your PR reviewed, please sign the CLA via the link in the |
Problem
CSVDocumentSplitteracceptsread_csv_kwargs, but settingheader=0makes a normal CSV crash during splitting. Withheader=0, pandas uses the first row as string column labels, and the splitter later executedint(split_df.columns[0])for thecol_idx_startmetadata, raising on the string label. The sub-table sort also ordered by label instead of position.Solution
Resolve the first column of each sub-table to its positional index in the original DataFrame with
df.columns.get_loc(...)for both the sort key and thecol_idx_startmetadata. Behaviour with the defaultheader=Noneis unchanged, where labels are already positional integers.Testing
python -m pytest test/components/preprocessors/test_csv_document_splitter.py -q— 30 passed (Docker, python:3.12-bookworm), including a new regression test running the exact scenario from the issue:read_csv_kwargs={"header": 0}with a blank-row split, assertingcol_idx_start == 0, correctrow_idx_startvalues and split contents.Fixes #12965
Note: this change was prepared with the help of an AI coding agent under human direction, per the AI-assistants contribution guidance.