Conversation
Publish video on message.contents next to images, complete the audio content-block shape, and document openai-agents input.audio.* as the voice-session convention rather than rewriting those spans onto llm.input_messages.
Land the published keys in Python, JS, Java, and Go so instrumentors can compose message_content.audio.audio.url and message_content.video.video.url from constants instead of string literals.
Export synthetic Gemini video, OpenAI chat audio, and openai-agents realtime spans so reviewers can compare today's capture with the proposed keys in Phoenix.
Document the _realtime.py event and finalize path, keep input.audio.url singular per span, and list the AudioAttributes constant swap for the instrumentor follow-up.
Infer video MIME from the URL instead of a published attribute. Over-limit audio and video externalize or redact. Prefer the OpenAI realtime .done transcript when a response is interrupted.
AudioAttributes is url and transcript only. Consumers infer MIME from the URL path or a data URI prefix, same as video.
This reverts commit 5d793c8.
Publish chat message_content.audio and message_content.video only. openai-agents realtime input.audio.* stays instrumentor-local until named SpanAttributes land.
Name the voice-session keys on SpanAttributes instead of a prefix rule. Point openai-agents realtime at those constants. USER and AUDIO kinds stay unpublished.
The span-root keys are named SpanAttributes now. Keep the event mapping tables in sync.
@arizeai/openinference-core
@arizeai/openinference-genai
@arizeai/openinference-instrumentation-anthropic
@arizeai/openinference-instrumentation-bedrock
@arizeai/openinference-instrumentation-bedrock-agent-runtime
@arizeai/openinference-instrumentation-beeai
@arizeai/openinference-instrumentation-claude-agent-sdk
@arizeai/openinference-instrumentation-langchain
@arizeai/openinference-instrumentation-langchain-v0
@arizeai/openinference-instrumentation-mcp
@arizeai/openinference-instrumentation-openai
@arizeai/openinference-instrumentation-openai-agents
@arizeai/openinference-semantic-conventions
@arizeai/openinference-tanstack-ai
@arizeai/openinference-vercel
commit: |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| export const INPUT_AUDIO_URL = | ||
| `${SemanticAttributePrefixes.input}.${SemanticAttributePrefixes.audio}.${AudioAttributesPostfixes.url}` as const; | ||
| export const INPUT_AUDIO_MIME_TYPE = | ||
| `${SemanticAttributePrefixes.input}.${SemanticAttributePrefixes.audio}.${AudioAttributesPostfixes.mime_type}` as const; | ||
| export const INPUT_AUDIO_TRANSCRIPT = | ||
| `${SemanticAttributePrefixes.input}.${SemanticAttributePrefixes.audio}.${AudioAttributesPostfixes.transcript}` as const; | ||
| /** | ||
| * The output of any span | ||
| */ | ||
| export const OUTPUT_VALUE = `${SemanticAttributePrefixes.output}.value` as const; | ||
| export const OUTPUT_MIME_TYPE = `${SemanticAttributePrefixes.output}.mime_type` as const; | ||
| export const OUTPUT_AUDIO_URL = | ||
| `${SemanticAttributePrefixes.output}.${SemanticAttributePrefixes.audio}.${AudioAttributesPostfixes.url}` as const; | ||
| export const OUTPUT_AUDIO_MIME_TYPE = | ||
| `${SemanticAttributePrefixes.output}.${SemanticAttributePrefixes.audio}.${AudioAttributesPostfixes.mime_type}` as const; | ||
| export const OUTPUT_AUDIO_TRANSCRIPT = | ||
| `${SemanticAttributePrefixes.output}.${SemanticAttributePrefixes.audio}.${AudioAttributesPostfixes.transcript}` as const; |
There was a problem hiding this comment.
The 6 new constants are not covered by any tests. Go and Python both received corresponding test updates in this PR (e.g., go/openinference-semantic-conventions/semconv_test.go and python/openinference-semantic-conventions/tests/openinference/semconv/test_attributes.py), but there are no tests for the JS semantic conventions package at all.
Per CLAUDE.md: "Include tests for new features and bug fixes"
Consider adding a test file (e.g., src/trace/SemanticConventions.test.ts) that asserts the string values of the new constants, matching the pattern used in Go and Python.
| public static final String INPUT_AUDIO_URL = | ||
| SemanticAttributePrefixes.INPUT | ||
| + "." | ||
| + SemanticAttributePrefixes.AUDIO | ||
| + "." | ||
| + AudioAttributesPostfixes.URL; |
There was a problem hiding this comment.
The 6 new Java constants have no corresponding test coverage. The Go and Python packages both received test updates for these same constants in this PR, but the Java openinference-semantic-conventions package has no test directory (src/test/ doesn't exist).
Per CLAUDE.md: "Include tests for new features and bug fixes"
Consider adding a src/test/java/com/arize/semconv/trace/SemanticConventionsTest.java that asserts the string values of the constants (following the pattern in go/openinference-semantic-conventions/semconv_test.go).
…ollow-up Chat-only PR 3700 should not ship the openai-agents event map or the USER-span demo.
…conv Keep openai_realtime.md and the realtime demo on this follow-up after 3700 dropped them.
| #### Span-root audio (voice sessions) | ||
|
|
||
| For realtime or live voice spans that are not chat message lists. These are named `SpanAttributes` (`INPUT_AUDIO_URL` is `input.audio.url`). Each span carries at most one audio payload (no `input.audio.<index>`). Span kinds `USER` and `AUDIO` stay unpublished. | ||
|
|
||
| - `input.audio.url` - User audio URL or WAV data URI | ||
| - `input.audio.mime_type` - User audio MIME type | ||
| - `input.audio.transcript` - User audio transcript | ||
| - `output.audio.url` - Assistant audio URL or WAV data URI | ||
| - `output.audio.mime_type` - Assistant audio MIME type | ||
| - `output.audio.transcript` - Assistant audio transcript | ||
|
|
There was a problem hiding this comment.
The six new span-root audio attributes are documented here as a subsection bullet list, but they're missing from the main Reserved Attributes table where all other published SpanAttributes constants are catalogued (see lines 136–143).
Every comparable published attribute has a row in that table — including the parent audio.url / audio.mime_type / audio.transcript keys and the peer input.value / output.value / input.mime_type / output.mime_type keys. The six new constants are the only newly-promoted SpanAttributes without a main-table entry, making the spec inconsistent and incomplete for consumers who reference that table to discover all reserved attributes.
Consider adding rows for each of the six attributes to the main table, e.g.:
| `input.audio.url` | String | `https://storage.com/audio.wav` | URL or WAV data URI of user audio for a voice session |
| `input.audio.mime_type` | String | `audio/wav` | MIME type of the user audio |
| `input.audio.transcript`| String | `"Hello, how are you?"` | Transcript of the user audio |
| `output.audio.url` | String | `https://storage.com/audio.wav` | URL or WAV data URI of assistant audio for a voice session |
| `output.audio.mime_type`| String | `audio/wav` | MIME type of the assistant audio |
| `output.audio.transcript`| String | `"I'm doing well, thanks."` | Transcript of the assistant audio |
…Config Shared mask() still only handles images. Do not list OPENINFERENCE_HIDE_*_AUDIO/VIDEO or BASE64_AUDIO/VIDEO_MAX_LENGTH as usable settings until TraceConfig implements them.
…conv Keep span-root SpanAttributes on this follow-up. Leave audio/video hide flags and size gates out of the published configuration table until TraceConfig implements them.
Code reviewNo issues found. Checked for bugs and CLAUDE.md compliance. |
This branch has not been deployed
1 participant
Summary
SpanAttributes(INPUT_AUDIO_URLisinput.audio.url, plus mime and transcript on input and output). This is not a “prefixinput.ontoAudioAttributes” rule.openai_realtime.mdandopenai_realtime_audio_demo.pyhere from feat(semconv): add audio and video semantic conventions #3700.USERandAUDIOspan kinds unpublished. Chat nesting (message_content.audio/message_content.video) stays on feat(semconv): add audio and video semantic conventions #3700.TraceConfig.mask()still only handles images. openai-agents keeps instrumentor-localOPENINFERENCE_HIDE_*_AUDIO/OPENINFERENCE_BASE64_AUDIO_MAX_LENGTHenv vars.Stacked on #3700 (
docs/audio-video-semconv). Merge that first.Test plan
test_attributes.pyTestSpanAttributesincludesinput.audio.*andoutput.audio.*GOWORK=off go test .ingo/openinference-semantic-conventionstest_audio_emitted_by_defaultandtest_attribute_keys_snapshot_on_full_turnSpanAttributes, no public span-root spec section, noopenai_realtime.md/openai_realtime_audio_demo.py, and no publishedOPENINFERENCE_HIDE_*_AUDIO/VIDEOsettings