[opentelemetry-instrumentation-genai-anthropic] Capture Anthropic image and document inputs as GenAI `BlobPart`, `UriPart`, `FilePart`, and `GenericPart` message parts by rads-1996 · Pull Request #589 · open-telemetry/opentelemetry-python-genai

rads-1996 · 2026-09-03T16:55:49Z

Description

Ensures multimodal feature parity with - #296

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How has this been tested?

Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. List any relevant details for your test
configuration.

uv run tox -e py312-test-instrumentation-genai-anthropic -- -q
uv run tox -e py312-test-instrumentation-genai-anthropic-conformance -- -q
uv run --python 3.12 tox -e lint-instrumentation-genai-anthropic

Checklist

See CONTRIBUTING.md
for the style guide, changelog guidance, and more.

Followed the style guidelines of this project
Changelog updated if the change requires an entry
Unit tests added
Documentation updated

opentelemetry-pr-dashboard · 2026-09-03T17:05:33Z

Pull request dashboard status

Waiting on the author · refreshed 2026-09-11 22:33 UTC

Respond to 5 review items (e.g. link a commit, explain why not, ask a follow-up):

Inline threads: 1, 2, 3, 4, 5

Status above doesn't look right?

Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
Should this be with reviewers? Comment /dashboard route:reviewers to route it to them.
Anything wrong — including the routing? Report it with what you expected; it helps us improve the dashboard.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Adds multimodal (image + document) input extraction for the Anthropic instrumentation so spans represent inputs as GenAI BlobPart, UriPart, FilePart, and GenericPart.

Changes:

Extend Anthropic content conversion to support image/document sources (base64, url, file, nested content).
Add unit tests verifying extractor behavior and span-captured multimodal parts (sync/async/streaming) plus a new conformance scenario.
Add changelog entry and a new VCR cassette for the conformance scenario.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
instrumentation/opentelemetry-instrumentation-genai-anthropic/src/opentelemetry/instrumentation/genai/anthropic/utils.py	Implements multimodal source parsing into GenAI message parts.
instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/test_messages_extractors.py	Adds unit coverage for image/document source extraction and ordering.
instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/test_sync_messages.py	Adds sync span-capture tests for multimodal inputs and generator-preservation.
instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/test_async_messages.py	Adds async span-capture tests for multimodal inputs (create + streaming).
instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/test_conformance.py	Registers the new multimodal conformance scenario.
instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/conformance/multimodal.py	New conformance scenario asserting multimodal parts are emitted correctly.
instrumentation/opentelemetry-instrumentation-genai-anthropic/tests/cassettes/test_chat_anthropic_multimodal_image_llm_call.yaml	Adds VCR cassette for conformance replay.
instrumentation/opentelemetry-instrumentation-genai-anthropic/.changelog/589.added	Documents the new multimodal capture capability.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

🟡 Changes recommended

The multimodal streaming tests reference non-matching existing cassettes and the new document URL/text MIME type handling currently hard-codes values that can produce incorrect telemetry.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

instrumentation/opentelemetry-instrumentation-genai-anthropic/src/opentelemetry/instrumentation/genai/anthropic/utils.py:160

For document URL sources, the code hard-codes mime_type="application/pdf" even when the request doesn't provide a media type. This can record an incorrect MIME type for non-PDF documents; prefer using a provided "media_type" when present, otherwise leave mime_type as None.

            return [
                UriPart(
                    mime_type="application/pdf",
                    modality="document",
                    uri=url,

instrumentation/opentelemetry-instrumentation-genai-anthropic/src/opentelemetry/instrumentation/genai/anthropic/utils.py:173

For document text sources, the code hard-codes mime_type="text/plain" instead of using the request's "media_type" field. This can produce incorrect telemetry when a different media type is supplied.

                BlobPart(
                    mime_type="text/plain",
                    modality="document",
                    content=data.encode(),
                )
            ]

Files reviewed: 10/10 changed files
Comments generated: 5
Review effort level: Lite

Copilot

🟡 Changes recommended

The new document-content conversion can drop generator-backed nested document inputs entirely (losing evidence that a document was provided), and the PR description’s listed test commands don’t match the Anthropic changes.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

instrumentation/opentelemetry-instrumentation-genai-anthropic/.changelog/589.added:1

The PR description's "How has this been tested?" section lists only LangChain tox envs, but this change is in the Anthropic instrumentation package (new Anthropic unit + conformance coverage and cassettes). Please update the PR description to reflect the actual Anthropic test commands that were run so reviewers can reproduce accurately.

Capture Anthropic image and document inputs as GenAI ``BlobPart``, ``UriPart``, ``FilePart``, and ``GenericPart`` message parts

Files reviewed: 12/12 changed files
Comments generated: 1
Review effort level: Lite

…Part`, `FilePart`, and `GenericPart` message parts

lmolkova · 2026-09-11T22:24:12Z

+        if isinstance(data, PathLike) or callable(getattr(data, "read", None)):
+            return GenericPart(type=modality)


Suggested change

if isinstance(data, PathLike) or callable(getattr(data, "read", None)):

return GenericPart(type=modality)

if isinstance(data, PathLike) or callable(getattr(data, "read", None)):

return GenericPart(type="blob")

In GenAI semconv message parts, type is "blob" (with modality indicating "image" or "document"). "image" and "document" are not valid part types.

lmolkova · 2026-09-11T22:24:13Z

+            return [
+                UriPart(
+                    mime_type="application/pdf",
+                    modality="document",
+                    uri=url,
+                )
+            ]


Suggested change

return [

UriPart(

mime_type="application/pdf",

modality="document",

uri=url,

)

]

media_type = source_dict.get("media_type")

return [

UriPart(

mime_type=media_type if isinstance(media_type, str) else None,

modality="document",

uri=url,

)

]

Anthropic document URL sources can specify a media_type. Hardcoding "application/pdf" overrides the actual media type when non-PDF documents (or unknown types) are supplied.

lmolkova · 2026-09-11T22:24:13Z

+def _convert_document_block(block: Mapping[str, Any]) -> MessagePart | None:
+    parts = _extract_document_source(block.get("source"))
+    metadata = {
+        key: block[key]
+        for key in ("title", "context", "citations")
+        if block.get(key) is not None
+    }
+    source = block.get("source")
+    source_mapping = (
+        cast(Mapping[str, object], source)
+        if isinstance(source, Mapping)
+        else None
+    )
+    is_nested = (
+        source_mapping is not None and source_mapping.get("type") == "content"
+    )
+    if metadata or (is_nested and parts):
+        return GenericPart(type="document")
+    return parts[0] if parts else None


When document metadata like title, context, or citations is present, returning GenericPart(type="document") discards the extracted BlobPart or UriPart (and "document" is not a valid semconv part type). We should preserve the extracted content part regardless of metadata.

Also, if source.type == "content" has multiple nested parts (e.g. text + image), returning a single MessagePart forces dropping them. If convert_content_to_parts extends a list of parts per block, we can preserve all nested parts without collapsing them into GenericPart:

# Test showing the issue: content = [{ "type": "document", "title": "Report", "source": {"type": "base64", "media_type": "application/pdf", "data": "QUJD"}, }] # Expecting BlobPart(mime_type="application/pdf", modality="document", content=b"ABC") # Currently returns GenericPart(type="document")

lmolkova · 2026-09-11T22:24:13Z

    return parsed


+def _multimodal_input_message():


nit: _multimodal_input_message and _assert_multimodal_input are duplicated identically in tests/test_async_messages.py. Consider moving them to a shared helper or conftest to keep them in sync.

lmolkova · 2026-09-11T22:24:13Z

@@ -0,0 +1 @@
+Capture Anthropic image and document inputs as GenAI ``BlobPart``, ``UriPart``, ``FilePart``, and ``GenericPart`` message parts


The PR description mentions Fixes #348 and lists LangChain tox environments. Issue #348 was for OpenAI parity (already resolved in #540). Could you update the PR description to decouple from #348 and list the Anthropic test commands?

opentelemetry-pr-dashboard Bot mentioned this pull request Sep 3, 2026

Pull Request Dashboard #171

Open

rads-1996 force-pushed the anthropic-parity-for-multimodal branch 7 times, most recently from 1d081e3 to 2346929 Compare September 9, 2026 14:23

rads-1996 force-pushed the anthropic-parity-for-multimodal branch from 2346929 to ad6177c Compare September 9, 2026 21:18

rads-1996 marked this pull request as ready for review September 9, 2026 21:25

Copilot AI lite review requested due to automatic review settings September 9, 2026 21:25

rads-1996 requested a review from a team as a code owner September 9, 2026 21:25

Copilot AI reviewed Sep 9, 2026

View reviewed changes

rads-1996 marked this pull request as draft September 9, 2026 21:46

Copilot started reviewing on behalf of rads-1996 September 9, 2026 22:08 View session

rads-1996 force-pushed the anthropic-parity-for-multimodal branch 4 times, most recently from b0e67ad to 5ba834a Compare September 10, 2026 17:56

rads-1996 requested a lite review from Copilot September 10, 2026 19:32

Copilot started reviewing on behalf of rads-1996 September 10, 2026 19:44 View session

Copilot AI reviewed Sep 10, 2026

View reviewed changes

rads-1996 requested a lite review from Copilot September 10, 2026 21:05

Copilot started reviewing on behalf of rads-1996 September 10, 2026 21:05 View session

Copilot AI reviewed Sep 10, 2026

View reviewed changes

Comment thread ...y-instrumentation-genai-anthropic/src/opentelemetry/instrumentation/genai/anthropic/utils.py

rads-1996 force-pushed the anthropic-parity-for-multimodal branch from d3f947b to 777373f Compare September 10, 2026 22:41

Capture Anthropic image and document inputs as GenAI BlobPart, `Uri…

e6561a2

…Part`, `FilePart`, and `GenericPart` message parts

rads-1996 added 10 commits September 11, 2026 08:13

Add CHANGELOG

597e0e0

Add expected violations to multimodal scenario tests

10a9316

Fix formatting

ac2a3a9

Remove type references from GenericPart to match the updated utils

4f416db

Address feedback

aadcef4

Fix formatting

423842c

Fix incorrect import

5bb7093

Fix test

44c670f

Address feedback

3213035

Address feedback

d156507

rads-1996 force-pushed the anthropic-parity-for-multimodal branch from 08a265f to d156507 Compare September 11, 2026 15:14

rads-1996 marked this pull request as ready for review September 11, 2026 15:24

Merge branch 'main' into anthropic-parity-for-multimodal

8499a68

lmolkova reviewed Sep 11, 2026

View reviewed changes

github-project-automation Bot added this to GenAI Semantic Conventions and Instrumentation libraries Sep 11, 2026

		if isinstance(data, PathLike) or callable(getattr(data, "read", None)):
		return GenericPart(type=modality)

		@@ -0,0 +1 @@
		Capture Anthropic image and document inputs as GenAI ``BlobPart``, ``UriPart``, ``FilePart``, and ``GenericPart`` message parts

Uh oh!

Conversation

rads-1996 commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How has this been tested?

Checklist

Uh oh!

opentelemetry-pr-dashboard Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull request dashboard status

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (2)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (1)

Uh oh!

Uh oh!

lmolkova Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

lmolkova Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rads-1996 commented Sep 3, 2026 •

edited

Loading

opentelemetry-pr-dashboard Bot commented Sep 3, 2026 •

edited

Loading