fix: sanitize KB retrieval query before dense embedding - #9923
Draft
lxfight wants to merge 2 commits into
Draft
Conversation
Some sticker/image-caption pipelines leave invisible control/format characters in the query, which certain embedding providers reject with HTTP 400 (e.g. SiliconFlow code 20015), surfacing as recurring dense retrieval failures. Strip control (Cc) and format (Cf) characters at the RetrievalManager entry and skip retrieval entirely when nothing remains. Also log repr(query) and its length when dense retrieval fails to make such cases diagnosable.
CodeQL flagged the explicit control/format character ranges as an overly permissive regular expression range. Classify characters via unicodedata.category() instead, which is clearer and avoids the suspicious-range pattern entirely.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #8135
Dense KB retrieval passed the raw query straight into the embedding provider. Some sticker/image-caption messages leave invisible control/format characters (zero-width chars, bidi controls, BOM, stray control bytes) in the query text, which certain embedding providers reject with
HTTP 400(e.g. SiliconFlowcode 20015 "The parameter is invalid"), producing recurring "稠密检索失败" errors for those messages.Modifications / 改动点
astrbot/core/knowledge_base/retrieval/manager.py:Cc, except tab/newline/CR) and format (Cf) characters at theRetrievalManager.retrieve()entry, and skip retrieval entirely when nothing remains, so invalid queries never reach the embedding provider (as suggested in the issue).repr(query)and its length to make such cases diagnosable.tests/test_retrieval_query_sanitize.py: cover the invisible-only query skip and the stripping of embedded invisible characters before_dense_retrieveis called.This is NOT a breaking change. / 这不是一个破坏性变更。
Screenshots or Test Results / 运行截图或测试结果
uv run ruff format/uv run ruff checkpass.知识库 ... 稠密检索失败: ... 400— the error no longer occurs (retrieval is skipped or the sanitized query embeds successfully); a failing embedding call now logsrepr(query)and its length for diagnosis.Checklist / 检查清单
😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
/ 如果 PR 中有新加入的功能,已经通过 Issue / 邮件等方式和作者讨论过。
👀 My changes have been well-tested, and "Verification Steps" and "Screenshots" have been provided above.
/ 我的更改经过了良好的测试,并已在上方提供了“验证步骤”和“运行截图”。
🤓 I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations in
requirements.txtandpyproject.toml./ 我确保没有引入新依赖库,或者引入了新依赖库的同时将其添加到
requirements.txt和pyproject.toml文件相应位置。😮 My changes do not introduce malicious code.
/ 我的更改没有引入恶意代码。
Summary by Sourcery
Prevent invalid invisible characters from reaching dense embedding providers during knowledge-base retrieval.
Bug Fixes:
Enhancements:
Tests: