Skip to content

google-genai: instrument_generate_content() unconditionally sets OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT=true, overriding config and user setting #619

Description

@alexmojaki

Package

opentelemetry-instrumentation-google-genai 1.0b0 (with opentelemetry-util-genai 1.0b0)

What happens

instrument_generate_content() unconditionally mutates a global process environment variable when instrumenting:

def instrument_generate_content(
    telemetry_handler: TelemetryHandler,
    generate_content_config_key_allowlist: AllowList,
) -> object:
    os.environ["OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT"] = "true"
    ...

This has a few problems:

  1. It overrides the documented default behavior. opentelemetry.util.genai.utils.should_emit_event() is documented to default based on OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT:

    • NO_CONTENT or SPAN_ONLY → events default off
    • EVENT_ONLY or SPAN_AND_EVENT → events default on

    Because instrumenting force-sets OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT=true (which should_emit_event() treats as highest priority), the gen_ai.client.inference.operation.details event is emitted for every request even in NO_CONTENT / SPAN_ONLY mode. In those modes the event carries no message content, so it just duplicates the span's attributes — redundant telemetry the user did not ask for.

  2. It overrides an explicit user setting. A user who sets OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT=false to disable the event has their choice silently overwritten by instrument().

  3. It mutates global process state as a side effect of instrumenting. Writing to os.environ affects the whole process (and any other GenAI instrumentation reading the same variable), is order-dependent, and is not thread-safe.

Minimal reproduction

import os

os.environ["OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT"] = "SPAN_ONLY"
os.environ["OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT"] = "false"  # explicitly disable the event

from opentelemetry.instrumentation.google_genai import GoogleGenAiSdkInstrumentor

GoogleGenAiSdkInstrumentor().instrument()

print(os.environ["OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT"])
# -> "true"   (the user's "false" was overwritten)

After this, a generate_content call emits a gen_ai.client.inference.operation.details event even though the user asked for SPAN_ONLY capture and explicitly set EMIT_EVENT=false.

Expected

instrument() should not write to os.environ. Whether the event is emitted should be decided by should_emit_event() from the (unmodified) configuration:

  • default off for NO_CONTENT / SPAN_ONLY,
  • default on for EVENT_ONLY / SPAN_AND_EVENT,
  • and an explicit OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT from the user always respected.

If there is a reason the event must be emitted regardless of capture mode, it would be clearer to pass that intent into the TelemetryHandler / invocation directly rather than mutating a global environment variable.

Context

Found while integrating this instrumentation into Pydantic Logfire. As a downstream workaround we snapshot OTEL_INSTRUMENTATION_GENAI_EMIT_EVENT before calling instrument() and restore it afterwards, but that only works because should_emit_event() reads the variable per request rather than at instrument time.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions