fix: add missing KB parser routes for mdx/mkd/html/htm/csv - #9921
Draft
lxfight wants to merge 1 commit into
Draft
Conversation
select_parser and the MarkdownChunker whitelist in kb_helper were inconsistent: .mdx/.mkd were accepted by the chunker but had no parser route, so uploads failed at the parsing stage with a generic unsupported format error. .html/.htm/.csv were not routed at all although the bundled markitdown-no-magika version ships HtmlConverter and CsvConverter. Route .mdx/.mkd to TextParser (plain markdown variants, no extra dependency) and .html/.htm/.csv to MarkitdownParser. .rtf intentionally stays unsupported: markitdown-no-magika 0.1.2 registers no RTF converter (verified: conversion raises UnsupportedFormatException).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #9903
select_parserand theMarkdownChunkerwhitelist inkb_helper.pywere inconsistent:.mdx/.mkdare accepted by the chunker whitelist but had no parser route — uploads of these files always failed at the parsing stage with "暂时不支持的文件格式", making that chunker branch dead code..html/.htm/.csvhad no route at all, although the bundledmarkitdown-no-magikaversion registersHtmlConverterandCsvConverter, so the capability was wasted.Modifications / 改动点
astrbot/core/knowledge_base/parsers/util.py:.mdx/.mkdtoTextParser(plain markdown variants, no extra dependency; the chunker whitelist already expects them)..html/.htm/.csvtoMarkitdownParser.tests/test_kb_parser_routing.py: cover the new routes, a real HTML/CSV parse smoke test, and that.rtfstays unsupported.Note on
.rtf: the issue suggested routing it toMarkitdownParser, butmarkitdown-no-magika0.1.2 registers no RTF converter (verified:md.convert(...)raisesUnsupportedFormatExceptionfor.rtf), so adding the route would only move the failure to a later stage with a worse error message. It intentionally keeps raising the explicit "unsupported format" error. Content sniffing for extension validation (optional in the issue) is left out to keep this change minimal.Screenshots or Test Results / 运行截图或测试结果
uv run ruff format/uv run ruff checkpass..mdx/.mkd/.html/.htm/.csvfile to a knowledge base — previously.mdx/.mkdfailed with "暂时不支持的文件格式" and html/csv were rejected outright; now all of them parse and index.Checklist / 检查清单
😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
/ 如果 PR 中有新加入的功能,已经通过 Issue / 邮件等方式和作者讨论过。
👀 My changes have been well-tested, and "Verification Steps" and "Screenshots" have been provided above.
/ 我的更改经过了良好的测试,并已在上方提供了“验证步骤”和“运行截图”。
🤓 I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations in
requirements.txtandpyproject.toml./ 我确保没有引入新依赖库,或者引入了新依赖库的同时将其添加到
requirements.txt和pyproject.toml文件相应位置。😮 My changes do not introduce malicious code.
/ 我的更改没有引入恶意代码。
Summary by Sourcery
Add missing knowledge-base parser routes for supported document extensions while preserving explicit handling for unsupported RTF files.
New Features:
Bug Fixes:
Tests: