Skip to content

fix: snap chunk start to a word boundary in CharacterBasedTextChunker - #2226

Merged
omri374 merged 3 commits into
data-privacy-stack:mainfrom
inesaranab:fix/chunker-word-boundary-start
Aug 9, 2026
Merged

fix: snap chunk start to a word boundary in CharacterBasedTextChunker#2226
omri374 merged 3 commits into
data-privacy-stack:mainfrom
inesaranab:fix/chunker-word-boundary-start

Conversation

@inesaranab

@inesaranab inesaranab commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Change Description

CharacterBasedTextChunker snaps end forward to the next boundary char so a chunk never ends mid-word. But start, computed as end - chunk_overlap, gets no equivalent treatment — the subtraction discards the alignment established two lines earlier.

With chunk_size=10, chunk_overlap=3:

text = "This is a test string for chunking purposes"

before: ['This is a test', 'st string for', 'or chunking', 'ing purposes']
after: ['This is a test', ' string for', ' chunking purposes']

Three of the four chunks begin mid-word. An NER recognizer receives 'st', 'or', 'ing' as standalone tokens and can confidently mislabel them as entities, producing a false positive that redacts the wrong span.

Fix: advance start to the next boundary char, mirroring the loop end already has. Three lines.

Present since #1805, which introduced this chunker.

Issue reference

No existing issue. Found while using the chunker via GLiNERRecognizer on long transcripts.

Checklist

  • I have reviewed the contribution guidelines
  • I agree to follow this project's Code of Conduct
  • I confirm that I have the right to submit this contribution and that it does not knowingly contain proprietary or confidential code.
  • My code includes unit tests
  • All unit tests and lint checks pass locally
  • My PR contains documentation updates / additions if required

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves CharacterBasedTextChunker to avoid starting overlapped chunks in the middle of words by snapping the computed start forward to a configured boundary character (mirroring the existing logic which snaps end forward), reducing the chance of NER false positives on partial tokens.

Changes:

  • Adjust CharacterBasedTextChunker.chunk() to advance start to the next boundary character after applying chunk_overlap.
  • Add a unit test asserting that every chunk after the first begins on (or immediately after) a boundary.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
presidio-analyzer/presidio_analyzer/chunkers/character_based_text_chunker.py Advances start to a boundary char after overlap to prevent mid-word chunk starts.
presidio-analyzer/tests/test_character_based_text_chunker.py Adds coverage to ensure chunk starts (after the first) land on a word boundary.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@omri374 omri374 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved! Thanks

@omri374
omri374 merged commit 760d6c8 into data-privacy-stack:main Aug 9, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants