Conversation
XLSXToDocument wrote cell values straight into a pipe table with no escaping, so a cell containing a pipe gained a column and lost its tail to any Markdown reader, and a cell containing an Alt+Enter line break was split across two table rows. DOCXToDocument already solves this. _escape_markdown_cell in haystack/components/converters/docx.py was added for the same two patterns in PR deepset-ai#12779, merged two days ago. This ports it to the XLSX converter. The class docstring says the content is the table saved in CSV or Markdown format, so the two formats have to describe the same table. With B2 set to 'blue | red' the CSV path emits it intact and the Markdown path loses '| red'. The escape runs via DataFrame.map before the existing NaN substitution, with an astype(object) in between: pandas re-infers StringDtype after map, and the later where(notna(), None) can then no longer store None, which would undo deepset-ai#12776.
Contributor
|
@chrikrah is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues
Proposed Changes:
XLSXToDocumentwrites cell values straight into a pipe table whentable_format="markdown", with no escaping. A cell containing|gains a column and loses its tail to any Markdown reader. A cell containing an Alt+Enter line break is split across two table rows.DOCXToDocumentalready solves exactly this._escape_markdown_cellathaystack/components/converters/docx.py:314-328handles the same two patterns and was added in #12779, merged 2026-09-24. This ports it to the XLSX converter.Three-row sheet with
B2 = "blue | red"andB3 = "line1\nline2":Rendered through
markdown-itwith commonmark plus tables, that Markdown reads as:The class docstring at
xlsx.py:30says the content "is the table which can be saved in CSV or Markdown format", so the two formats have to describe the same table. Here they do not.There is an implementation note for the reviewer, below.
How did you test it?
Baseline on
8a5406eis6586 passed, 8 skipped, 338 deselected, so the delta is the two new parametrised cases. Revertinghaystack/components/converters/xlsx.pyand keeping the test gives2 failed, 6586 passed, both of themtest_run_markdown_escapes_cell_content, on thepipeandline-breakcases. The test mirrors the merged DOCX one.ruff checkandruff format --checkare clean on both touched files, andmypy --python-version 3.12is clean onxlsx.py.What I did not run: hatch is not installed here, so
ruffandmypyran directly instead of throughhatch run test:typesandhatch run fmt, andmypycovered the touched file instead of the whole tree. The integration, e2e and slow suites did not run.Notes for the reviewer
The
.map()placement matters, and the obvious version breaks two existing tests. pandas re-infersStringDtypeafter.map(), so the later.where(notna(), None)can no longer storeNone. That undoes #12776 and failstest_run_markdownandtest_run_markdown_missing_value. The patch inserts.astype(object)between the two, and both stay green.Checklist
No related issue exists, as noted above.
fix:, and there are no breaking changes.@sjrl you merged #12779, the DOCX half of this. Would you rather the escaping lived in one shared helper both converters import, instead of the copy this adds?