feat(kb): support pluggable knowledge base retrieval backends

[lxfight]

AstrBot plugins can already integrate external knowledge base systems through custom hooks, but there is no stable, shared contract for knowledge base discovery and retrieval.

This PR introduces a minimal public backend API so plugins can expose external knowledge bases through a consistent interface without depending on AstrBot's built-in storage implementation. The scope is intentionally limited to enabled knowledge base discovery and read-only retrieval.

Modifications / 改动点

• Added public plugin contracts for knowledge base backends:

  • BaseKnowledgeBaseBackend
  • KnowledgeBaseRef
  • KnowledgeBaseInfo
  • KnowledgeBaseQuery
  • KnowledgeBaseHit
  • KnowledgeBaseResponse
  • Standard backend error types

• Added backend registration and unregistration through the plugin Context.

• Added an adapter that exposes the built-in knowledge base through the same contract without replacing the existing built-in Agent flow.

• Added concurrent multi-backend retrieval with:

  • Per-backend timeouts
  • Failure isolation
  • Rank-based result merging
  • Global top_k truncation
  • Runtime validation of third-party responses

• Added first-class knowledge base identity to every retrieval hit.

• Integrated enabled external backends into the existing Agent knowledge base flow.

• Defined list_knowledge_bases() as returning only knowledge bases that are enabled and accessible for the current session.

• Added explicit plugin reload lifecycle behavior. Plugins unregister their backends in terminate() before releasing resources.

• Added Chinese and English plugin development documentation with a complete backend example.

• Added unit and adversarial tests for registration, discovery, retrieval, failure isolation, malformed responses, result identity, and reload behavior.

• This is NOT a breaking change. / 这不是一个破坏性变更.

Scope

This PR intentionally does not standardize:

• Knowledge base creation or deletion
• Document upload or management
• Chunk or index management
• Backup and restore
• Credential configuration
• Unified WebUI management

Plugins may provide those capabilities through their own configuration, commands, or Plugin Pages. Optional management contracts can be considered separately when concrete cross-backend requirements are available.

The existing AstrBot knowledge base database, upload flow, Dashboard APIs, and backup behavior are unchanged.

Screenshots or Test Results / 运行截图或测试结果

Verification performed:

uv run ruff format .
503 files left unchanged

uv run ruff check .
All checks passed

Knowledge base test suite:
51 passed

Unit test suite:
906 passed

Two existing Dashboard log-capture tests fail locally because warnings are emitted through the configured logger instead of caplog. The same failures were reproduced on upstream/master and are unrelated to this PR.

The Chinese and English VitePress documentation build also completed successfully.

---

Checklist / 检查清单

• 😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
  / 如果 PR 中有新加入的功能，已经通过 Issue / 邮件等方式和作者讨论过。

• 👀 My changes have been well-tested, and "Verification Steps" and "Screenshots" have been provided above.
  / 我的更改经过了良好的测试，并已在上方提供了“验证步骤”和“运行截图”。

• 🤓 I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations in requirements.txt and pyproject.toml.
  / 我确保没有引入新依赖库，或者引入了新依赖库的同时将其添加到 requirements.txt 和 pyproject.toml 文件相应位置。

• 😮 My changes do not introduce malicious code.
  / 我的更改没有引入恶意代码。

Summary by Sourcery

Enable plugins to expose accessible external knowledge bases through a shared discovery and retrieval interface.

New Features:

• Introduce a public plugin API for discovering and read-only retrieving knowledge bases through pluggable backends.
• Integrate enabled external backend results into the Agent knowledge base retrieval flow alongside the built-in knowledge base.
• Provide concurrent multi-backend retrieval with timeouts, failure isolation, validated responses, rank-based merging, and global result limits.

Enhancements:

• Expose the built-in knowledge base through the new backend contract while preserving existing built-in behavior.
• Add backend registration and lifecycle-safe unregistration through plugin Context, including reload support and in-flight operation synchronization.
• Include backend and knowledge base identity in standardized retrieval results.

Documentation:

• Add Chinese and English plugin development documentation with a complete external knowledge base backend example and API semantics.

Tests:

• Add coverage for backend contracts, registration and reload lifecycle, built-in adaptation, discovery and retrieval orchestration, malformed responses, failure isolation, result identity, and Agent integration.

[sourcery-ai]

Hey - I've found 2 issues

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="astrbot/core/tools/knowledge_base_tools.py" line_range="81-135" />
<code_context>
+    if kb_names:
</code_context>
<issue_to_address>
**issue (broader_impact):** When both built-in and external knowledge bases return `top_k` hits, the Agent appends the built-in results and the external results independently, so the complete retrieval request injects up to `2 * top_k` results instead of applying the documented global `top_k` limit.

**Triggers:** When built-in and external knowledge bases both have matching results.

**Suggested fix:** Merge or truncate the formatted results across built-in and external sources before returning the final context.
</issue_to_address>

### Comment 2
<location path="astrbot/core/knowledge_base/kb_mgr.py" line_range="110-111" />
<code_context>
+            raise ValueError(
+                "The built-in knowledge base backend cannot be unregistered."
+            )
+        if self.backends.pop(backend_id, None) is not None:
+            logger.info("Knowledge base backend unregistered: %s", backend_id)
+
+    async def list_registered_knowledge_bases(
</code_context>
<issue_to_address>
**issue (bug_risk):** Unregistering a backend only removes it from the registry; retrievals that already captured the backend in `selected_backends` continue running while the plugin proceeds to close the backend's resources in `terminate()`, causing in-flight requests to use closed clients or other released state during reload.

**Triggers:** When a plugin is reloaded or disabled while a backend listing or retrieval is in progress.

**Suggested fix:** Track in-flight backend operations and await or cancel them before allowing the plugin to release backend resources.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 2 findings to address first, and the change automatically injects content from plugin-provided external backends into the agent context, creating a new trust boundary where malicious or incorrect retrieved text could influence agent behavior and potentially trigger external actions. Reverting removes the integration, but any actions or disclosures caused while it was enabled cannot be undone.

Blocking findings: astrbot/core/tools/knowledge_base_tools.py:135, astrbot/core/knowledge_base/kb_mgr.py:111

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}