Skip to content

fix(groq): keep multimodal input message content on spans - #3758

Open
Harsh23Kashyap wants to merge 5 commits into
Arize-ai:mainfrom
Harsh23Kashyap:fix/multimodal-input-message-content
Open

Harsh23Kashyap wants to merge 5 commits into
Arize-ai:mainfrom
Harsh23Kashyap:fix/multimodal-input-message-content

Conversation

@Harsh23Kashyap

@Harsh23Kashyap Harsh23Kashyap commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Part of #3757

The groq instrumentor writes a request message's content onto llm.input_messages.N.message.content as-is. For multimodal requests content is a list of typed parts (text, image_url), and the OpenTelemetry SDK drops the attribute because sequences may only hold scalars, so the user message silently disappears from the span.

Content parts are now flattened into message.contents.N.message_content.*, the same way the openai instrumentor does it:

  • text parts become type=text with message_content.text
  • image_url parts become type=image with message_content.image.image.url
  • parts keep their original index; a part type without an OpenInference mapping (such as the document part in newer groq releases) is skipped and leaves a gap instead of shifting later parts
  • content given as any sequence (list or tuple) is handled; content given as a generator is turned into a list before the call, so it is recorded on the span and the SDK still sends every part

Tests cover list, tuple and generator content with exhaustive attribute assertions, and a skipped part keeping later indices in place. The generator test also checks the request body the SDK sent.

Validation:

  • py310-ci-groq and py310-ci-groq-latest (ruff, mypy, 20 tests) pass
  • Checked live against Groq (qwen/qwen3.8-27b) with list and tuple content; both spans in Phoenix show the text and image parts under message.contents

@feiiiiii5

Copy link
Copy Markdown
Contributor

Ran one request through the three packages' own message-attribute functions at 9f15262 (checkout on PYTHONPATH, no network — just the extractor, so the span attributes are exactly what the instrumentor would set).

The message is a user turn with three content parts: text, then input_audio, then image_url. input_audio is the part none of the four helpers here produces attributes for, so it is the case that separates the two indexing conventions:

groq      ->  message.contents.0...  (text)
              message.contents.1...  (image)     # renumbered: the audio part is skipped
together  ->  message.contents.0...  (text)
              message.contents.2...  (image)     # gap at contents.1
portkey   ->  message.contents.0...  (text)
              message.contents.2...  (image)     # gap at contents.1
openai    ->  message.contents.0...  (text)
              message.contents.2...  (image)     # existing behaviour, for reference

groq/_utils.py:92-97 advances content_index only when a part yields attributes (if not part_attributes: continue), while together/_request_attributes_extractor.py:92 and portkey/_request_attributes_extractor.py:93 use enumerate(content), which keeps the position in the original list — as the pre-existing openai/_request_attributes_extractor.py:129 does.

So the same message comes out with different message.contents.<i> keys depending on which provider's instrumentor produced the span. Whichever rule is intended, it would help to have one of them across the three packages rather than two, since a consumer that walks message.contents.{i} until a key is missing stops at the gap and never sees the image on together/portkey, while one that keys off position cannot line a groq span up with an openai one. Not asking for a particular choice — enumerate matches openai, skipping matches the reasoning/text handling above it in the same groq function — just flagging that this PR introduces the split, and a test in each package that pins the index for a part that contributes nothing would keep the three from drifting again.

Two smaller notes from the same pass, both optional and both arguably out of scope here:

  • An input_audio (or file) part contributes nothing at all to the span under this PR, same as before it. That is the openai behaviour, so it is consistent, but it is the same class of silent drop this PR fixes for list content — worth its own issue rather than a change here.
  • The content-part branches use isinstance(content, list) (groq) / isinstance(content, List) (together, portkey), where openai uses is_iterable_of(content, dict). A tuple of parts still yields only message.role in all three, as it did before the PR; no regression, just a difference from the "mirrors the openai instrumentor" description in the summary.

Disclosure: I am not a maintainer and have no commit rights here. I work on the groq instrumentor in this repo (my open PR #3782 touches groq/_wrappers.py, which does not overlap with this diff), and this comment came from an AI-assisted local run of the snippets above that I read and reproduced myself; the numbers are from the commands, not from a model.

@Harsh23Kashyap

Copy link
Copy Markdown
Contributor Author

Thanks for the careful pass, and for the concrete repro. Fixed in ea5917b and eb11835: groq now enumerates content parts by their position in the original list, matching together, portkey, and the existing openai instrumentor, so a part with no attributes leaves a gap instead of renumbering the parts after it. Each of the three packages also has a new test with an input_audio part pinning that gap.

On the smaller notes: agreed that an input_audio or file part contributing nothing at all is the same class of silent drop, and that behaviour predates this PR in the openai instrumentor too, so a separate issue is the right scope for it. The isinstance(content, list) check is a fair flag; a tuple of parts behaves the same as before this PR in all three packages, so I left it untouched here.

@feiiiiii5

Copy link
Copy Markdown
Contributor

Re-ran your two commits myself at eb11835 and the fix does what you describe.

  • Groq, content = [text, input_audio (no attributes), image_url]: the ChatCompletion span gets ...contents.0.message_content.text and ...contents.2...image.url, with contents.1. absent — the gap is preserved and the image keeps its original index.
  • To check the test is not vacuous I mutated the installed _utils.py:92 from enumerate(content) to enumerate([p for p in content if list(_get_attributes_from_message_content(p))]), i.e. the skip-and-renumber shape: the same input then produced contents.1...image.url, so the index really does move back when the filter returns. Restored byte-identical afterwards (cmp clean) and the gap is back.
  • git diff --name-only 5da0a96..eb11835 -- python/instrumentation/openinference-instrumentation-openai is empty, and running the same three-part input through the openai instrumentor gives contents.0 + contents.2 with the input_audio part contributing nothing at all. So your point 2 holds and it is pre-existing there, not something this PR introduces.

Agreed on leaving the isinstance(content, list) gate alone here. Taking you up on the separate issue: I am filing one for the "part contributes nothing" class, and I will include the tuple case in its scope — at eb11835 a request whose content is a tuple of parts ends up with only llm.input_messages.0.message.role on the span, so tuple-valued content loses every part. Whatever base did, that is the same silent-drop family, and it belongs with input_audio/file rather than in this PR. No review debt left on my side for #3758.

@satyadevai

Copy link
Copy Markdown
Collaborator

@caroger This PR includes changes for Groq, Portkey, and TogetherAI. Please let me know if you'd prefer me to split them into three separate PRs.

@caroger

caroger commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

@satyadevai yes please separate

Harsh23Kashyap and others added 5 commits September 25, 2026 01:19
…spans

Content-part arrays (vision requests) were yielded as a raw list of dicts
on llm.input_messages.N.message.content. OpenTelemetry rejects attribute
values that are sequences of dicts, so the whole attribute was dropped and
the user message content silently vanished from the span.

Flatten typed content parts into message.contents.* instead, mirroring the
openai instrumentor, with a JSON fallback for non-mapping parts.

Fail-before/pass-after regression tests added for all three instrumentors.
The multimodal input flattening compacted the message.contents indices
whenever a content part contributed no attributes (for example
input_audio), shifting every later part one slot down. The together and
portkey instrumentors and the existing openai instrumentor keep the
original list positions, leaving a gap instead. Align groq with that
convention and add a test pinning the gap for an unsupported part.
@satyadevai
satyadevai force-pushed the fix/multimodal-input-message-content branch from 5eec3ff to df49e05 Compare September 24, 2026 19:54
@satyadevai satyadevai changed the title fix(groq,together,portkey): keep multimodal input message content on spans fix(groq): keep multimodal input message content on spans Sep 25, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

4 participants