Skip to content

fix(kb): route plain text uploads to TextParser without markitdown - #9889

Open
xiaoyuyu6420 wants to merge 3 commits into
AstrBotDevs:masterfrom
xiaoyuyu6420:fix/9598-txt-parser-fallback
Open

fix(kb): route plain text uploads to TextParser without markitdown#9889
xiaoyuyu6420 wants to merge 3 commits into
AstrBotDevs:masterfrom
xiaoyuyu6420:fix/9598-txt-parser-fallback

Conversation

@xiaoyuyu6420

@xiaoyuyu6420 xiaoyuyu6420 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Fixes #9598

Problem

select_parser sent .txt / .md / .markdown to MarkitdownParser, which imports the optional markitdown-no-magika dependency at module level. When that dependency is not installed (e.g. manual deployments), uploading a plain text file raises ModuleNotFoundError, which kb_helper swallows into a generic "文档解析失败" message. The user gets no hint that the root cause is a missing optional dependency, and plain text uploads simply fail.

Fix

Plain text formats are decoded directly by the existing TextParser, which needs no extra dependency and auto-detects multiple encodings (utf-8 / gbk / gb2312 / gb18030):

  • .txt / .md / .markdownTextParser
  • .rst / .adoc / .xlsx / .docx / .xlsMarkitdownParser (unchanged; these rely on conversion)

Chunking in kb_helper already selects MarkdownChunker by file extension, so markdown heading hierarchy is preserved — the parser swap does not change downstream behavior for .md files.

Testing

  • uv run pytest tests/unit/test_parser_selection.py — 3 passed (new)
  • uv run pytest tests/test_epub_parser.py tests/unit/test_kb_upload_atomicity.py tests/unit/test_parser_selection.py — 26 passed
  • ruff format / ruff check clean

Summary by Sourcery

Bug Fixes:

  • Route plain-text knowledge-base uploads through TextParser so .txt, .md, and .markdown files work without the optional Markitdown dependency.

select_parser sent .txt/.md/.markdown to MarkitdownParser, which imports
the optional markitdown-no-magika dependency. When that dependency is not
installed, uploading a plain text file raised ModuleNotFoundError that was
swallowed into a generic "文档解析失败" message, with no hint about the
missing dependency (issue AstrBotDevs#9598).

Plain text formats are decoded directly by the existing TextParser, which
needs no extra dependency and supports multiple encodings (utf-8/gbk/...).
Route .txt/.md/.markdown to TextParser; keep .rst/.adoc/.xlsx/.docx/.xls
on MarkitdownParser since those rely on conversion. Chunking in
kb_helper still selects MarkdownChunker by file extension, so markdown
heading hierarchy is unchanged.

Fixes AstrBotDevs#9598

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="tests/unit/test_parser_selection.py" line_range="3" />
<code_context>
+import pytest
+
+from astrbot.core.knowledge_base.parsers.markitdown_parser import MarkitdownParser
+from astrbot.core.knowledge_base.parsers.text_parser import TextParser
+from astrbot.core.knowledge_base.parsers.util import select_parser
</code_context>
<issue_to_address>
**issue (testing):** The test module imports `MarkitdownParser` at collection time, so `ModuleNotFoundError` is raised before any test runs when the optional `markitdown-no-magika` dependency is absent—the exact deployment scenario this change is intended to support.

**Triggers:** When running the test suite in an installation without `markitdown-no-magika`.

**Suggested fix:** Import `MarkitdownParser` lazily inside the rich-format test and skip that test when the optional dependency is unavailable, or mock the parser module during selection testing.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 1 finding to address first, and if the routing is wrong, .txt or Markdown files could be parsed into incorrect knowledge-base content and embeddings, leaving bad indexed records after a revert. The records are bounded and can be cleared or regenerated, so reverting plus re-ingestion repairs the effect.

Blocking findings: tests/unit/test_parser_selection.py:3


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread tests/unit/test_parser_selection.py Outdated
xiaoyuyu6420 and others added 2 commits September 1, 2026 08:49
… optional dep

Sourcery flagged that importing MarkitdownParser at module scope raises
ModuleNotFoundError during test collection when markitdown-no-magika is
absent - exactly the deployment scenario this change supports. Import it
inside the rich-format test and skip that test when the optional
dependency is unavailable, so the plain-text selection tests still run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] 没有markitdown-no-magika依赖时知识库中上传txt文件会报错

3 participants