Skip to content

fix: add missing KB parser routes for mdx/mkd/html/htm/csv - #9921

Draft
lxfight wants to merge 1 commit into
AstrBotDevs:masterfrom
lxfight:fix/kb-parser-chunker-whitelist
Draft

fix: add missing KB parser routes for mdx/mkd/html/htm/csv#9921
lxfight wants to merge 1 commit into
AstrBotDevs:masterfrom
lxfight:fix/kb-parser-chunker-whitelist

Conversation

@lxfight

@lxfight lxfight commented Sep 2, 2026

Copy link
Copy Markdown
Member

Fixes #9903

select_parser and the MarkdownChunker whitelist in kb_helper.py were inconsistent:

  • .mdx / .mkd are accepted by the chunker whitelist but had no parser route — uploads of these files always failed at the parsing stage with "暂时不支持的文件格式", making that chunker branch dead code.
  • .html / .htm / .csv had no route at all, although the bundled markitdown-no-magika version registers HtmlConverter and CsvConverter, so the capability was wasted.

Modifications / 改动点

  • astrbot/core/knowledge_base/parsers/util.py:
    • Route .mdx / .mkd to TextParser (plain markdown variants, no extra dependency; the chunker whitelist already expects them).
    • Route .html / .htm / .csv to MarkitdownParser.
  • tests/test_kb_parser_routing.py: cover the new routes, a real HTML/CSV parse smoke test, and that .rtf stays unsupported.

Note on .rtf: the issue suggested routing it to MarkitdownParser, but markitdown-no-magika 0.1.2 registers no RTF converter (verified: md.convert(...) raises UnsupportedFormatException for .rtf), so adding the route would only move the failure to a later stage with a worse error message. It intentionally keeps raising the explicit "unsupported format" error. Content sniffing for extension validation (optional in the issue) is left out to keep this change minimal.

  • This is NOT a breaking change. / 这不是一个破坏性变更。

Screenshots or Test Results / 运行截图或测试结果

$ uv run pytest tests/test_kb_parser_routing.py tests/test_epub_parser.py tests/test_kb_import.py -q
18 passed, 2 warnings in 2.88s
  • uv run ruff format / uv run ruff check pass.
  • Verification steps: upload an .mdx/.mkd/.html/.htm/.csv file to a knowledge base — previously .mdx/.mkd failed with "暂时不支持的文件格式" and html/csv were rejected outright; now all of them parse and index.

Checklist / 检查清单

  • 😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
    / 如果 PR 中有新加入的功能,已经通过 Issue / 邮件等方式和作者讨论过。

  • 👀 My changes have been well-tested, and "Verification Steps" and "Screenshots" have been provided above.
    / 我的更改经过了良好的测试,并已在上方提供了“验证步骤”和“运行截图”

  • 🤓 I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations in requirements.txt and pyproject.toml.
    / 我确保没有引入新依赖库,或者引入了新依赖库的同时将其添加到 requirements.txtpyproject.toml 文件相应位置。

  • 😮 My changes do not introduce malicious code.
    / 我的更改没有引入恶意代码。

Summary by Sourcery

Add missing knowledge-base parser routes for supported document extensions while preserving explicit handling for unsupported RTF files.

New Features:

  • Add parser support for MDX and MKD files.
  • Add HTML, HTM, and CSV parsing through the existing MarkItDown integration.

Bug Fixes:

  • Fix knowledge-base uploads failing for extensions already accepted by the Markdown chunker but lacking parser routes.

Tests:

  • Add parser-routing coverage, HTML and CSV parsing smoke tests, and regression coverage confirming RTF remains unsupported.

select_parser and the MarkdownChunker whitelist in kb_helper were
inconsistent: .mdx/.mkd were accepted by the chunker but had no parser
route, so uploads failed at the parsing stage with a generic unsupported
format error. .html/.htm/.csv were not routed at all although the bundled
markitdown-no-magika version ships HtmlConverter and CsvConverter.

Route .mdx/.mkd to TextParser (plain markdown variants, no extra
dependency) and .html/.htm/.csv to MarkitdownParser. .rtf intentionally
stays unsupported: markitdown-no-magika 0.1.2 registers no RTF converter
(verified: conversion raises UnsupportedFormatException).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] 知识库解析:select_parser 与 chunker 白名单不一致,mdx/mkd/html/csv/rtf 等格式无路由或路由矛盾

1 participant