Skip to content

fix: keep an XLSX cell from breaking the markdown table - #12972

Open
chrikrah wants to merge 1 commit into
deepset-ai:mainfrom
chrikrah:fix/xlsx-markdown-escape
Open

chrikrah wants to merge 1 commit into
deepset-ai:mainfrom
chrikrah:fix/xlsx-markdown-escape

Conversation

@chrikrah

Copy link
Copy Markdown

Related Issues

Proposed Changes:

XLSXToDocument writes cell values straight into a pipe table when table_format="markdown", with no escaping. A cell containing | gains a column and loses its tail to any Markdown reader. A cell containing an Alt+Enter line break is split across two table rows.

DOCXToDocument already solves exactly this. _escape_markdown_cell at haystack/components/converters/docx.py:314-328 handles the same two patterns and was added in #12779, merged 2026-09-24. This ports it to the XLSX converter.

Three-row sheet with B2 = "blue | red" and B3 = "line1\nline2":

$ python repro.py
markdown:                            csv (the control, same commit):
|  2 | widget  | blue | red |         2,widget,"blue | red"
|  3 | gadget  | line1      |         3,gadget,"line1
|    |         | line2      |         line2"

Rendered through markdown-it with commonmark plus tables, that Markdown reads as:

['2', 'widget', 'blue']      <- "| red" is gone
['3', 'gadget', 'line1']     <- phantom extra row
['',  '',       'line2']

The class docstring at xlsx.py:30 says the content "is the table which can be saved in CSV or Markdown format", so the two formats have to describe the same table. Here they do not.

There is an implementation note for the reviewer, below.

How did you test it?

$ .venv/bin/python -m pytest test -m "not integration" -q
6588 passed, 8 skipped, 338 deselected, 6 warnings in 45.19s

Baseline on 8a5406e is 6586 passed, 8 skipped, 338 deselected, so the delta is the two new parametrised cases. Reverting haystack/components/converters/xlsx.py and keeping the test gives 2 failed, 6586 passed, both of them test_run_markdown_escapes_cell_content, on the pipe and line-break cases. The test mirrors the merged DOCX one.

ruff check and ruff format --check are clean on both touched files, and mypy --python-version 3.12 is clean on xlsx.py.

What I did not run: hatch is not installed here, so ruff and mypy ran directly instead of through hatch run test:types and hatch run fmt, and mypy covered the touched file instead of the whole tree. The integration, e2e and slow suites did not run.

Notes for the reviewer

The .map() placement matters, and the obvious version breaks two existing tests. pandas re-infers StringDtype after .map(), so the later .where(notna(), None) can no longer store None. That undoes #12776 and fails test_run_markdown and test_run_markdown_missing_value. The patch inserts .astype(object) between the two, and both stay green.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
    No related issue exists, as noted above.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, and there are no breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

@sjrl you merged #12779, the DOCX half of this. Would you rather the escaping lived in one shared helper both converters import, instead of the copy this adds?

XLSXToDocument wrote cell values straight into a pipe table with no escaping, so
a cell containing a pipe gained a column and lost its tail to any Markdown
reader, and a cell containing an Alt+Enter line break was split across two table
rows.

DOCXToDocument already solves this. _escape_markdown_cell in
haystack/components/converters/docx.py was added for the same two patterns in
PR deepset-ai#12779, merged two days ago. This ports it to the XLSX converter.

The class docstring says the content is the table saved in CSV or Markdown
format, so the two formats have to describe the same table. With B2 set to
'blue | red' the CSV path emits it intact and the Markdown path loses '| red'.

The escape runs via DataFrame.map before the existing NaN substitution, with an
astype(object) in between: pandas re-infers StringDtype after map, and the later
where(notna(), None) can then no longer store None, which would undo deepset-ai#12776.
@chrikrah
chrikrah requested a review from a team as a code owner September 26, 2026 13:51
@chrikrah
chrikrah requested review from anakin87 and removed request for a team September 26, 2026 13:51
@vercel

vercel Bot commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

@chrikrah is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Sep 26, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants