Fix ANSI Outlook MSG decoding for Japanese code pages and padded strings

[afourney]

Improve ANSI .msg conversion by removing trailing NUL padding and correctly decoding Japanese code pages 50220–50222.

Changes

• Strip NUL terminators before charset detection and decoding.
• Omit empty ANSI properties after terminator removal.
• Support extended Japanese and SO/SI Katakana sequences.
• Preserve charset detection fallback for invalid declared encodings.
• Add regression tests covering padding, empty values, Japanese headers and bodies, and fallback behavior.

[Copilot]

🟢 Approval recommended

The implementation is focused, fallback behavior is preserved, and the new edge cases are well tested.

Pull request overview

Improves legacy ANSI Outlook MSG conversion for padded strings and Japanese code pages.

Changes:

• Removes trailing NUL padding before decoding.
• Adds CP50221/50222 Katakana and SO/SI decoding.
• Adds regression coverage for padding, Japanese text, and fallbacks.

File summaries

File	Description
packages/markitdown/src/markitdown/converters/_outlook_msg_converter.py	Implements ANSI cleanup and Japanese decoding.
packages/markitdown/tests/test_outlook_msg_ansi.py	Adds comprehensive regression tests.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 0
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[schalkley94-debug]

Flourney I emailed you a few days ago from s.chalkley94@gmail.com
Please read DR.