fix: avoid splitting UTF-8 characters during charset detection

[afourney]

Charset detection samples the first 64 KiB of text input. If that boundary falls inside a multibyte UTF-8 character, the truncated sample can be mistaken for another encoding. Applying that guess to the full document can silently corrupt valid text—for example, turning  résumé  into  rĂ©sumĂ© —or cause conversion to fail.

This change completes a trailing UTF-8 character before passing the sample to  charset_normalizer :

• Use a strict incremental decoder to distinguish a split character from invalid UTF-8.
• Read at most three additional bytes, stopping as soon as the character is complete.
• Preserve the original sample if the prefix or continuation is invalid, or EOF prevents completion.

The existing charset detector still selects the encoding; the fix does not force UTF-8, discard bytes, or override an explicitly supplied charset. Stream-position restoration remains unchanged.

This is a narrow mitigation for sample-boundary splitting. It does not address other encoding-detection limitations, such as non-ASCII content appearing entirely beyond an ASCII-only sample.

[Copilot]

🟢 Approval recommended

The implementation is bounded, preserves existing fallback behavior, and has comprehensive focused tests.

Pull request overview

Prevents charset detection from misclassifying valid UTF-8 when the 64 KiB sample splits a multibyte character.

Changes:

• Completes trailing UTF-8 characters with bounded lookahead.
• Adds boundary, invalid-input, stream-position, and explicit-charset tests.

File summaries

File	Description
packages/markitdown/src/markitdown/_markitdown.py	Safely completes UTF-8 charset samples.
packages/markitdown/tests/test_charset_sample.py	Covers sampling behavior and regressions.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.