Skip to content

fix: prevent RecursiveCharacterChunker overlap from exceeding chunk_size - #9922

Draft
lxfight wants to merge 1 commit into
AstrBotDevs:masterfrom
lxfight:fix/kb-chunker-overlap-limit
Draft

fix: prevent RecursiveCharacterChunker overlap from exceeding chunk_size#9922
lxfight wants to merge 1 commit into
AstrBotDevs:masterfrom
lxfight:fix/kb-chunker-overlap-limit

Conversation

@lxfight

@lxfight lxfight commented Sep 2, 2026

Copy link
Copy Markdown
Member

Fixes #9901

In RecursiveCharacterChunker.chunk(), when a split triggered a flush, the next chunk was unconditionally built as overlap_text + split without re-checking the combined length against chunk_size. Every flushed chunk could therefore be up to chunk_size + overlap long (about 562 chars with the default 512/50 config), and a larger configured overlap made the repeated overlap portion dominate the chunks, distorting embedding counts and retrieval quality.

Meanwhile _split_by_character() already rejects overlap >= chunk_size, so the two code paths disagreed on the same invariant.

Modifications / 改动点

  • astrbot/core/knowledge_base/chunking/recursive.py:

    • After prepending the overlap to the next chunk, re-check the combined length; when overlap + split would exceed chunk_size, emit the overlap as a standalone chunk (bounded by overlap < chunk_size) and start the new chunk with the split only.
    • Add the same parameter validation as _split_by_character at chunk() entry (chunk_size > 0, 0 <= overlap < chunk_size) so both paths enforce the same invariant.
  • tests/test_recursive_chunker.py: regression test proving no output chunk exceeds chunk_size (fails on the old code with 110-char chunks), plus small-text, normal-text, and invalid-overlap cases.

  • This is NOT a breaking change. / 这不是一个破坏性变更。

Screenshots or Test Results / 运行截图或测试结果

$ uv run pytest tests/test_recursive_chunker.py -q
5 passed in 0.40s

# on the unfixed code the regression test fails as expected:
# FAILED test_overlap_never_pushes_chunks_over_chunk_size (110 > 100)
  • uv run ruff format / uv run ruff check pass.
  • Verification steps: upload a long document with the default chunker config and inspect the stored chunks — max chunk length is now within the configured chunk_size instead of chunk_size + overlap.

Checklist / 检查清单

  • 😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
    / 如果 PR 中有新加入的功能,已经通过 Issue / 邮件等方式和作者讨论过。

  • 👀 My changes have been well-tested, and "Verification Steps" and "Screenshots" have been provided above.
    / 我的更改经过了良好的测试,并已在上方提供了“验证步骤”和“运行截图”

  • 🤓 I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations in requirements.txt and pyproject.toml.
    / 我确保没有引入新依赖库,或者引入了新依赖库的同时将其添加到 requirements.txtpyproject.toml 文件相应位置。

  • 😮 My changes do not introduce malicious code.
    / 我的更改没有引入恶意代码。

Summary by Sourcery

Ensure recursive character chunking respects configured size limits while validating overlap parameters consistently.

Bug Fixes:

  • Prevent recursive chunking overlap from producing chunks larger than the configured chunk size.
  • Reject invalid chunk size and overlap configurations consistently across recursive chunking paths.

Tests:

  • Add regression and validation coverage for chunk size limits, small and normal inputs, and invalid overlap settings.

When a split triggered a flush, the chunker unconditionally built the
next chunk as overlap_text + split without re-checking the combined
length, so every flushed chunk could be up to chunk_size + overlap long
(e.g. 562 chars with the default 512/50 config). _split_by_character
already guarded against overlap >= chunk_size, but chunk() did not, so
the two paths disagreed.

Re-check the combined length after prepending the overlap and emit the
overlap as a standalone chunk when it would exceed the limit; also apply
the same parameter validation at chunk() entry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] 知识库分块:RecursiveCharacterChunker 的 overlap 重复计长导致输出块超限,embedding 数量失真

1 participant