Skip to content

fix: sanitize KB retrieval query before dense embedding - #9923

Draft
lxfight wants to merge 2 commits into
AstrBotDevs:masterfrom
lxfight:fix/kb-dense-query-sanitize
Draft

fix: sanitize KB retrieval query before dense embedding#9923
lxfight wants to merge 2 commits into
AstrBotDevs:masterfrom
lxfight:fix/kb-dense-query-sanitize

Conversation

@lxfight

@lxfight lxfight commented Sep 2, 2026

Copy link
Copy Markdown
Member

Fixes #8135

Dense KB retrieval passed the raw query straight into the embedding provider. Some sticker/image-caption messages leave invisible control/format characters (zero-width chars, bidi controls, BOM, stray control bytes) in the query text, which certain embedding providers reject with HTTP 400 (e.g. SiliconFlow code 20015 "The parameter is invalid"), producing recurring "稠密检索失败" errors for those messages.

Modifications / 改动点

  • astrbot/core/knowledge_base/retrieval/manager.py:

    • Strip control (Cc, except tab/newline/CR) and format (Cf) characters at the RetrievalManager.retrieve() entry, and skip retrieval entirely when nothing remains, so invalid queries never reach the embedding provider (as suggested in the issue).
    • When dense retrieval fails, the error log now includes repr(query) and its length to make such cases diagnosable.
  • tests/test_retrieval_query_sanitize.py: cover the invisible-only query skip and the stripping of embedded invisible characters before _dense_retrieve is called.

  • This is NOT a breaking change. / 这不是一个破坏性变更。

Screenshots or Test Results / 运行截图或测试结果

$ uv run pytest tests/test_retrieval_query_sanitize.py -q
2 passed in 1.05s
$ uv run pytest tests/unit/test_kb_manager_resilience.py -q
4 passed in 1.25s
  • uv run ruff format / uv run ruff check pass.
  • Verification steps: send a sticker/image that previously logged 知识库 ... 稠密检索失败: ... 400 — the error no longer occurs (retrieval is skipped or the sanitized query embeds successfully); a failing embedding call now logs repr(query) and its length for diagnosis.

Checklist / 检查清单

  • 😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
    / 如果 PR 中有新加入的功能,已经通过 Issue / 邮件等方式和作者讨论过。

  • 👀 My changes have been well-tested, and "Verification Steps" and "Screenshots" have been provided above.
    / 我的更改经过了良好的测试,并已在上方提供了“验证步骤”和“运行截图”

  • 🤓 I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations in requirements.txt and pyproject.toml.
    / 我确保没有引入新依赖库,或者引入了新依赖库的同时将其添加到 requirements.txtpyproject.toml 文件相应位置。

  • 😮 My changes do not introduce malicious code.
    / 我的更改没有引入恶意代码。

Summary by Sourcery

Prevent invalid invisible characters from reaching dense embedding providers during knowledge-base retrieval.

Bug Fixes:

  • Sanitize knowledge-base retrieval queries to remove invisible control and format characters, skip empty sanitized queries, and prevent embedding-provider HTTP 400 failures.

Enhancements:

  • Include the sanitized query representation and length in dense retrieval failure logs to improve diagnosis.

Tests:

  • Add coverage for skipping invisible-only queries and sanitizing queries before dense retrieval.

Some sticker/image-caption pipelines leave invisible control/format
characters in the query, which certain embedding providers reject with
HTTP 400 (e.g. SiliconFlow code 20015), surfacing as recurring dense
retrieval failures.

Strip control (Cc) and format (Cf) characters at the RetrievalManager
entry and skip retrieval entirely when nothing remains. Also log
repr(query) and its length when dense retrieval fails to make such
cases diagnosable.
Comment thread astrbot/core/knowledge_base/retrieval/manager.py Fixed
CodeQL flagged the explicit control/format character ranges as an
overly permissive regular expression range. Classify characters via
unicodedata.category() instead, which is clearer and avoids the
suspicious-range pattern entirely.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] KB dense retrieval still fails for some sticker/image-caption messages

2 participants