Refactor Office OCR converters to reuse core conversion pipelines

packages/markitdown-ocr/README.md

@@ -14,6 +14,8 @@ Uses the same `llm_client` / `llm_model` pattern that MarkItDown already support
 
 ## Installation
 
+Requires `markitdown>=0.1.8b3`, which introduces the Office image-rendering hooks used by this plugin. Installing the plugin automatically resolves a compatible core version.
+
 ```bash
 pip install markitdown-ocr
 ```
@@ -97,9 +99,11 @@ When a file is converted:
 1. The OCR converter accepts the file
 2. It extracts embedded images from the document
 3. Each image is sent to the LLM with an extraction prompt
-4. The returned text is inserted inline, preserving document structure
+4. The returned text is placed alongside document content (XLSX images follow their sheet's table)
 5. If the LLM call fails, conversion continues without that image's text
 
+The DOCX, PPTX, and XLSX converters subclass their core counterparts and override the same semi-private `_image_to_html` method. Core handles native content, preprocessing, and placement; the plugin supplies escaped OCR HTML, which passes through the shared HTML-to-Markdown renderer. PDF uses its separate existing pipeline.
+
 ## Supported File Formats
 
 ### PDF
@@ -110,21 +114,23 @@ When a file is converted:
 
 ### DOCX
 
-- Images are extracted via document part relationships (`doc.part.rels`).
-- OCR is run before the DOCX→HTML→Markdown pipeline executes: placeholder tokens are injected into the HTML so that the markdown converter does not escape the OCR markers, and the final placeholders are replaced with the formatted `*[Image OCR]...[End OCR]*` blocks after conversion.
-- Document flow (headings, paragraphs, tables) is fully preserved around the OCR blocks.
+- Inherits core DOCX preprocessing, styles, math, and Mammoth conversion.
+- Mammoth provides each embedded image to `_image_to_html`. OCR fragments are inserted into the document's HTML before Markdown rendering, not substituted into finished Markdown.
+- Block fragments split enclosing paragraphs where necessary and remain inside their table cell or list item. Table-cell line breaks follow the shared HTML converter's existing limitations.
 
 ### PPTX
 
 - Picture shapes, placeholder shapes with images, and images inside groups are all supported.
-- Shapes are processed in top-to-left reading order per slide.
+- Inherits core shape ordering, native text, tables, charts, and speaker notes.
+- Slide content now uses the core converter's real line breaks rather than the old plugin's literal `\n` text, and inherits its empty-title and empty-notes handling.
 - If an `llm_client` is configured, the LLM is asked for a description first; OCR is used as the fallback when no description is returned.
 
 ### XLSX
 
-- Images embedded in worksheets (`sheet._images`) are extracted per sheet.
-- Cell position is calculated from the image anchor coordinates (column/row → Excel letter notation).
+- Inherits core workbook repair and table rendering; images are read from the same repaired workbook.
 - Images are listed under a `### Images in this sheet:` section after the sheet's data table — they are not interleaved into the table rows.
+- Sheet heading spacing follows the core converter; no new cell-position labels are added.
+- Legacy `.xls` files remain handled by the existing core converter, without image OCR.
 
 ### Output format
 
@@ -136,6 +142,10 @@ Every extracted OCR block is wrapped as:
 [End OCR]*
 ```
 
+For Office formats, recognized text is escaped as literal HTML text before Markdown rendering. Markdown escaping and line breaks therefore follow the shared HTML converter: for example, underscores may be backslash-escaped, and direct converter results use Markdown hard breaks. `MarkItDown` subsequently strips trailing whitespace from each output line. Empty recognition retains the native image representation (XLSX normally omits images).
+
+Repeated image bytes are recognized once per conversion, while the result is placed at every occurrence. The cache is not shared across documents or service overrides.
+
 ## Troubleshooting
 
 ### OCR text missing from output
@@ -165,6 +175,8 @@ markitdown --list-plugins   # should show: ocr
 
 The plugin propagates LLM API errors as warnings and continues conversion. Check your API key, quota, and that the chosen model supports vision inputs.
 
+For Office OCR, a service-reported error emits a warning and retains native image rendering. Exceptions raised by custom OCR services propagate from direct converter calls; `MarkItDown` can retry another applicable converter through its normal fallback behavior.
+
 ## Development
 
 ### Running Tests
@@ -178,8 +190,8 @@ pytest tests/ -v
 
 ```bash
 git clone https://github.com/microsoft/markitdown.git
-cd markitdown/packages/markitdown-ocr
-pip install -e .
+cd markitdown
+pip install -e 'packages/markitdown[docx,pptx,xlsx]' -e packages/markitdown-ocr
 ```
 
 ## Contributing

packages/markitdown-ocr/pyproject.toml

@@ -25,7 +25,7 @@ classifiers = [
 
 # Core dependencies — matches the file-format libraries markitdown already uses
 dependencies = [
-  "markitdown>=0.1.0",
+  "markitdown>=0.1.8b3",
   "pdfminer.six>=20251230",
   "pdfplumber>=0.11.9",
   "PyMuPDF>=1.24.0",

packages/markitdown-ocr/src/markitdown_ocr/_docx_converter_with_ocr.py

@@ -1,228 +1,71 @@
-"""
-Enhanced DOCX Converter with OCR support for embedded images.
-Extracts images from Word documents and performs OCR while maintaining context.
-"""
+"""DOCX image OCR using the core document conversion pipeline."""
 
-import io
-import re
-import sys
+import hashlib
+import html
 from typing import Any, BinaryIO, Optional
+from warnings import warn
 
-from markitdown.converters import HtmlConverter
-from markitdown.converter_utils.docx.pre_process import pre_process_docx
 from markitdown import DocumentConverterResult, StreamInfo
-from markitdown._exceptions import (
-    MissingDependencyException,
-    MISSING_DEPENDENCY_MESSAGE,
-)
-from ._ocr_service import LLMVisionOCRService
+from markitdown.converters import DocxConverter
 
-# Try loading dependencies
-_dependency_exc_info = None
-try:
-    import mammoth
-    from docx import Document
-except ImportError:
-    _dependency_exc_info = sys.exc_info()
+from ._ocr_service import LLMVisionOCRService, _extract_text_with_metadata
 
-# Placeholder injected into HTML so that mammoth never sees the OCR markers.
-# Must be a single token with no special markdown characters.
-_PLACEHOLDER = "MARKITDOWNOCRBLOCK{}"
 
-_UNDERLINE_STYLE_MAP = "u => u"
-
-
-def _read_embedded_style_map(file_stream: BinaryIO) -> Optional[str]:
-    """Read the style map embedded in a .docx, if it has one."""
-    position = file_stream.tell()
-    file_stream.seek(0)
-    try:
-        return mammoth.read_embedded_style_map(file_stream)
-    finally:
-        file_stream.seek(position)
-
-
-class DocxConverterWithOCR(HtmlConverter):
-    """
-    Enhanced DOCX Converter with OCR support for embedded images.
-    Maintains document flow while extracting text from images inline.
-    """
+class DocxConverterWithOCR(DocxConverter):
+    """Recognize embedded images while inheriting native DOCX conversion."""
 
     def __init__(self, ocr_service: Optional[LLMVisionOCRService] = None):
         super().__init__()
-        self._html_converter = HtmlConverter()
+        if not hasattr(DocxConverter, "_image_to_html"):
+            raise RuntimeError(
+                "DOCX OCR requires markitdown>=0.1.8b3 for the "
+                "DocxConverter._image_to_html hook. "
+                "Upgrade with: pip install --upgrade 'markitdown>=0.1.8b3'."
+            )
         self.ocr_service = ocr_service
 
-    def accepts(
+    def convert(
         self,
         file_stream: BinaryIO,
         stream_info: StreamInfo,
         **kwargs: Any,
-    ) -> bool:
-        mimetype = (stream_info.mimetype or "").lower()
-        extension = (stream_info.extension or "").lower()
-
-        if extension == ".docx":
-            return True
-
-        if mimetype.startswith(
-            "application/vnd.openxmlformats-officedocument.wordprocessingml"
-        ):
-            return True
-
-        return False
+    ) -> DocumentConverterResult:
+        # Keep repeated-image recognition local to this document, not the instance.
+        kwargs["_docx_ocr_cache"] = {}
+        return super().convert(file_stream, stream_info, **kwargs)
 
-    def convert(
+    def _image_to_html(
         self,
-        file_stream: BinaryIO,
+        image_stream: BinaryIO,
         stream_info: StreamInfo,
         **kwargs: Any,
-    ) -> DocumentConverterResult:
-        if _dependency_exc_info is not None:
-            raise MissingDependencyException(
-                MISSING_DEPENDENCY_MESSAGE.format(
-                    converter=type(self).__name__,
-                    extension=".docx",
-                    feature="docx",
-                )
-            ) from _dependency_exc_info[1].with_traceback(
-                _dependency_exc_info[2]
-            )  # type: ignore[union-attr]
-
-        # Get OCR service if available (from kwargs or instance)
-        ocr_service: Optional[LLMVisionOCRService] = (
-            kwargs.get("ocr_service") or self.ocr_service
-        )
-
-        # Pre-process once, up front, so that every subsequent read sees the
-        # repaired archive. pre_process_docx() also fixes .docx files whose ZIP
-        # local file headers disagree with the central directory; reading the
-        # original stream first would make those raise, or silently yield no
-        # images and drop the OCR output.
-        pre_process_stream = pre_process_docx(file_stream)
-
-        # Read the embedded style map and combine with any provided style map
-        caller_style_map = kwargs.get("style_map")
-        embedded_style_map = _read_embedded_style_map(pre_process_stream)
-
-        style_map = "\n".join(
-            part
-            for part in (
-                caller_style_map,
-                embedded_style_map,
-                _UNDERLINE_STYLE_MAP,
-            )
-            if part
-        )
-
-        if ocr_service:
-            # 1. Extract and OCR images — returns raw text per image
-            pre_process_stream.seek(0)
-            image_ocr_map = self._extract_and_ocr_images(
-                pre_process_stream, ocr_service
-            )
-
-            # 2. Convert DOCX → HTML via mammoth
-            pre_process_stream.seek(0)
-            html_result = mammoth.convert_to_html(
-                pre_process_stream,
-                style_map=style_map,
-                include_embedded_style_map=False,
-            ).value
-
-            # 3. Replace <img> tags with plain placeholder tokens so that
-            #    mammoth's HTML→markdown step never escapes our OCR markers.
-            html_with_placeholders, ocr_texts = self._inject_placeholders(
-                html_result, image_ocr_map
-            )
-
-            # 4. Convert HTML → markdown
-            md_result = self._html_converter.convert_string(
-                html_with_placeholders, **kwargs
-            )
-            md = md_result.markdown
-
-            # 5. Swap placeholders for the actual OCR blocks (post-conversion
-            #    so * and _ are never escaped by the markdown converter).
-            for i, raw_text in enumerate(ocr_texts):
-                placeholder = _PLACEHOLDER.format(i)
-                ocr_block = f"*[Image OCR]\n{raw_text}\n[End OCR]*"
-                md = md.replace(placeholder, ocr_block)
-
-            return DocumentConverterResult(markdown=md)
-        else:
-            # Standard conversion without OCR
-            pre_process_stream.seek(0)
-            return self._html_converter.convert_string(
-                mammoth.convert_to_html(
-                    pre_process_stream,
-                    style_map=style_map,
-                    include_embedded_style_map=False,
-                ).value,
-                **kwargs,
+    ) -> Optional[str]:
+        ocr_service = kwargs.get("ocr_service") or self.ocr_service
+        if ocr_service is None:
+            return None
+
+        cache: dict[bytes, Optional[str]] = kwargs.get("_docx_ocr_cache", {})
+        key = hashlib.sha256(image_stream.read()).digest()
+        image_stream.seek(0)
+        if key in cache:
+            return cache[key]
+
+        result = _extract_text_with_metadata(ocr_service, image_stream, stream_info)
+        if result.error:
+            warn(
+                f"DOCX image OCR failed: {result.error}. Keeping the native image.",
+                RuntimeWarning,
+                stacklevel=2,
             )
-
-    def _extract_and_ocr_images(
-        self, file_stream: BinaryIO, ocr_service: LLMVisionOCRService
-    ) -> dict[str, str]:
-        """
-        Extract images from DOCX and OCR them.
-
-        Returns:
-            Dict mapping image relationship IDs to raw OCR text (no markers).
-        """
-        ocr_map = {}
-
-        try:
-            file_stream.seek(0)
-            doc = Document(file_stream)
-
-            for rel in doc.part.rels.values():
-                if "image" in rel.target_ref.lower():
-                    try:
-                        image_bytes = rel.target_part.blob
-                        image_stream = io.BytesIO(image_bytes)
-                        ocr_result = ocr_service.extract_text(image_stream)
-
-                        if ocr_result.text.strip():
-                            # Store raw text only — markers added later
-                            ocr_map[rel.rId] = ocr_result.text.strip()
-
-                    except Exception:
-                        continue
-
-        except Exception:
-            pass
-
-        return ocr_map
-
-    def _inject_placeholders(
-        self, html: str, ocr_map: dict[str, str]
-    ) -> tuple[str, list[str]]:
-        """
-        Replace <img> tags with numbered placeholder tokens.
-
-        Returns:
-            (html_with_placeholders, ordered list of raw OCR texts)
-        """
-        if not ocr_map:
-            return html, []
-
-        ocr_texts = list(ocr_map.values())
-        used: list[int] = []
-
-        def replace_img(match: re.Match) -> str:  # type: ignore[type-arg]
-            for i in range(len(ocr_texts)):
-                if i not in used:
-                    used.append(i)
-                    return f"<p>{_PLACEHOLDER.format(i)}</p>"
-            return ""  # remove image if all OCR texts already used
-
-        result = re.sub(r"<img[^>]*>", replace_img, html)
-
-        # Any OCR texts that had no matching <img> tag go at the end
-        for i in range(len(ocr_texts)):
-            if i not in used:
-                result += f"<p>{_PLACEHOLDER.format(i)}</p>"
-
-        return result, ocr_texts
+            cache[key] = None
+            return None
+        text = result.text.strip()
+        if not text:
+            cache[key] = None
+            return None
+
+        text = text.replace("\r\n", "\n").replace("\r", "\n")
+        content = html.escape(text).replace("\n", "<br>")
+        fragment = f"<p><em>[Image OCR]<br>{content}<br>[End OCR]</em></p>"
+        cache[key] = fragment
+        return fragment

packages/markitdown-ocr/src/markitdown_ocr/_ocr_service.py

@@ -4,6 +4,7 @@
 """
 
 import base64
+import inspect
 from typing import Any, BinaryIO
 from dataclasses import dataclass
 
@@ -108,3 +109,19 @@ def extract_text(
             return OCRResult(text="", backend_used="llm_vision", error=str(e))
         finally:
             image_stream.seek(0)
+
+
+def _extract_text_with_metadata(
+    ocr_service: LLMVisionOCRService,
+    image_stream: BinaryIO,
+    stream_info: StreamInfo,
+) -> OCRResult:
+    """Forward metadata when supported, retaining the legacy stream-only call."""
+    extract_text = ocr_service.extract_text
+    try:
+        inspect.signature(extract_text).bind(image_stream, stream_info=stream_info)
+    except (TypeError, ValueError):
+        # Unsupported or uninspectable signatures keep the legacy invocation.
+        return extract_text(image_stream)
+    # Keep service errors outside the signature check; never retry an OCR call.
+    return extract_text(image_stream, stream_info=stream_info)