Skip to content

feat: add OpenTelemetry bridge for AI observability - #306

Merged
marandaneto merged 16 commits into
mainfrom
posthog-self-driving/featposthog-go-add-otel-bridge-for-ai-c11a1b
Sep 2, 2026
Merged

feat: add OpenTelemetry bridge for AI observability#306
marandaneto merged 16 commits into
mainfrom
posthog-self-driving/featposthog-go-add-otel-bridge-for-ai-c11a1b

Conversation

@posthog

@posthog posthog Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

💡 Motivation and Context

  • Go teams building agent backends had no supported path to PostHog AI observability. Over the last 30 days, not a single $ai_* event arrived with $lib = posthog-go, which matched the code: grepping the SDK for $ai_generation, $ai_span, or anything llm-shaped returned nothing. Those users got a flat "no" on a product they were otherwise ready to pay for.
  • The ingestion side already worked. posthog-python ships a generic OpenTelemetry bridge that keeps gen_ai./llm./ai./traceloop. spans and forwards them to /i/v0/ai/otel. Google's Agent Development Kit (ADK) for Go emits exactly those gen_ai.* spans — there was just nothing on the Go side pointing at them.

This PR ports that bridge to Go.

  • New github.com/posthog/posthog-go/otel module (posthogotel):
    • SpanProcessor — recommended integration; register it on a TracerProvider. Filters, batches, and exports AI spans.
    • Exporter — for frameworks that accept only a span exporter.
    • IsAISpan — the shared prefix filter (gen_ai., llm., ai., traceloop.), matched against the span name and every attribute key.
    • Both target {host}/i/v0/ai/otel over OTLP/HTTP with Authorization: Bearer <project api key>; WithHost switches cloud region.
  • example/ wires the processor to the tracer provider an ADK Go agent emits through.

Why a separate Go module

The core SDK today has a lean dependency graph. Folding the OTel SDK, otlptracehttp, and their protobuf/grpc transitive tree into the root go.mod would burden every posthog-go consumer for an optional feature. A nested module keeps those dependencies opt-in, so only users who import .../otel pull them.

💚 How did you test it?

  • Unit tests over IsAISpan: name-prefix match, attribute-key match, and a non-AI negative, using real ReadOnlySpans from a SpanRecorder.
  • End-to-end tests against an httptest OTLP server that decodes the protobuf payload and asserts:
    • only the AI span (gen_ai.chat) reaches the endpoint when an AI and a non-AI span are emitted together, for both SpanProcessor and Exporter;
    • the request path is /i/v0/ai/otel and the header is Authorization: Bearer <key>;
    • a batch of only non-AI spans produces no HTTP request.
  • Empty-API-key rejection and WithHost normalisation (blank falls back to default, trailing slash trimmed).
  • go build, go vet, and go test -race pass in the module. Added an otel-bridge CI job (Go 1.25 and 1.x), since the root ./... does not reach a nested module.

📝 Checklist

  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran pnpm changeset to generate a changeset file

The root SDK and nested otel module share one release version and cadence. The release workflow creates both vX.Y.Z and otel/vX.Y.Z tags at the same release commit, matching the synchronized nested-module approach used by Sentry Go.

🤖 Agent context

Autonomy: Fully autonomous

  • Authored by Claude (via PostHog Desktop) from an inbox report about a customer asking for ADK Go support in AI observability.
  • Read the posthog-python posthog/ai/otel/ bridge (spans.py, exporter.py, processor.py) as the reference and ported its behaviour.
  • Chose a separate Go module over inline .go files in the root package: the report suggested following the error_tracking*.go pattern, but that product added no third-party dependencies, whereas the OTel bridge pulls in grpc/protobuf. Isolating it keeps the core SDK lean.
  • Pinned OpenTelemetry to v1.43.0 and the bridge module to Go 1.25 to avoid reachable vulnerabilities in older OTel releases; the separate core module remains on Go 1.21.
  • Kept the example credential-free and runnable: real ADK wiring needs model credentials and a heavy dependency, so the example registers the processor on the tracer provider ADK Go emits through and emits one representative gen_ai.* span, with the ADK hookup documented.

Created with PostHog Desktop from this inbox report.

Adds a posthogotel Go module that forwards OpenTelemetry AI spans to PostHog
AI observability. It keeps only spans that follow a known AI semantic
convention (gen_ai., llm., ai., traceloop.) and sends them over OTLP/HTTP to
the /i/v0/ai/otel endpoint with the project API key as a bearer token.

The bridge is a separate Go module so the core posthog-go SDK stays free of
OpenTelemetry dependencies. It ports the posthog-python OTel bridge: a
SpanProcessor for TracerProvider setups and an Exporter for frameworks that
accept only a span exporter. An example wires it to a Google ADK Go agent
through the shared tracer provider.

Generated-By: PostHog Desktop
Task-Id: 5d65718b-f73a-48c4-9c4d-0fdd2d00fa17
@posthog

posthog Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

🦔 PostHog Review reviewed this pull request

Found 3 must fix, 8 should fix, 0 consider.

Published 11 findings (view the review).

Resolved comments: 8 fixed, 3 left for you

@github-actions

github-actions Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

posthog-go Compliance Report

Date: 2026-09-01 15:24:01 UTC
Duration: 169005ms

✅ All Tests Passed!

111/111 tests passed


Capture_V1 Tests

94/94 tests passed

View Details
Test Status Duration
Endpoint And Method.Targets V1 Endpoint 22ms
Endpoint And Method.Does Not Use Legacy Endpoints 6ms
Required Headers.Has Authorization Bearer Header 5ms
Required Headers.Has Content Type Json 16ms
Required Headers.Has Posthog Sdk Info Format 5ms
Required Headers.Has Posthog Attempt Header 6ms
Required Headers.Has Posthog Request Id 4ms
Required Headers.Has Posthog Request Timestamp 16ms
Required Headers.Has User Agent 6ms
Body Format.Body Has Created At And Batch 6ms
Body Format.No Api Key In Body 5ms
Body Format.No Sent At In Body 5ms
Event Format.Event Has Required Root Fields 5ms
Event Format.Event Uuid Is Valid 5ms
Event Format.Event Timestamp Is Rfc3339 6ms
Event Format.Distinct Id Is String 5ms
Event Format.Distinct Id At Root Not Properties 5ms
Event Format.Custom Properties Preserved 5ms
Event Format.Set Properties Preserved 5ms
Event Format.Set Once Properties Preserved 5ms
Event Format.Groups Properties Preserved 5ms
Event Format.Sdk Generates Uuid If Not Provided 5ms
Event Format.Event Has Required Root Fields Batch 6ms
Event Format.Event Uuid Is Valid Batch 8ms
Event Format.Event Timestamp Is Rfc3339 Batch 6ms
Event Format.Distinct Id Is String Batch 7ms
Event Format.Distinct Id At Root Not Properties Batch 7ms
Event Format.Custom Properties Preserved Batch 7ms
Event Format.Set Properties Preserved Batch 7ms
Event Format.Set Once Properties Preserved Batch 7ms
Event Format.Groups Properties Preserved Batch 7ms
Event Format.Sdk Generates Uuid If Not Provided Batch 6ms
Batch Behavior.Multiple Events In Single Batch 30ms
Batch Behavior.Batch Envelope Smoke 9ms
Batch Behavior.Flush With No Events Sends Nothing 2ms
Batch Behavior.Flush At Triggers Batch 1007ms
Batch Behavior.Created At Reflects Batch Creation Time 6ms
Deduplication.Generates Unique Uuids 13ms
Deduplication.Different Events Same Content Different Uuids 7ms
Deduplication.Preserves Uuid On Retry 5133ms
Deduplication.Preserves Timestamp On Retry 5132ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry 5136ms
Deduplication.No Duplicate Events In Batch 29ms
Header Behavior On Retry.Attempt Header Starts At One 5ms
Header Behavior On Retry.Attempt Header Increments On Retry 10139ms
Header Behavior On Retry.Request Id Preserved On Retry 5131ms
Header Behavior On Retry.Different Requests Have Different Request Ids 2022ms
Header Behavior On Retry.Request Timestamp Changes On Retry 5135ms
Response Format Validation.Success Response Has Uuid Keyed Results 7ms
Response Format Validation.Success Response Has Ok For Each Event 26ms
Response Format Validation.Success No Retry After When All Ok 27ms
Response Format Validation.Success Retry After Present When Retry Events 128ms
Response Format Validation.Success No Retry After When Drop Only 27ms
Response Format Validation.Response Echoes Request Id 5ms
Retry Behavior.Retries On 408 5133ms
Retry Behavior.Retries On 500 5133ms
Retry Behavior.Retries On 503 5135ms
Retry Behavior.Retries On 504 5135ms
Retry Behavior.Retryable Errors Have Retry After 2132ms
Retry Behavior.Respects Retry After On Retryable Error 8133ms
Retry Behavior.Does Not Retry On 400 2132ms
Retry Behavior.Does Not Retry On 401 2131ms
Retry Behavior.Does Not Retry On 402 2132ms
Retry Behavior.Does Not Retry On 413 2131ms
Retry Behavior.Does Not Retry On 415 2131ms
Retry Behavior.Non Retryable Errors Have No Retry After 2133ms
Retry Behavior.Implements Backoff 15131ms
Retry Behavior.Max Retries Respected 15145ms
Partial Batch Handling.Handles 200 Full Success 2009ms
Partial Batch Handling.Handles 200 With All Ok 3030ms
Partial Batch Handling.Does Not Retry Dropped Events 3028ms
Partial Batch Handling.Does Not Retry Limited Events 3031ms
Partial Batch Handling.Prunes Ok Events On Partial Retry 5134ms
Partial Batch Handling.Prunes Dropped Events On Partial Retry 5135ms
Partial Batch Handling.Retries Only Retry Events From Partial 5133ms
Partial Batch Handling.Partial Retry Preserves Uuids 5133ms
Partial Batch Handling.Partial Retry Attempt Header Increments 5136ms
Partial Batch Handling.Partial Retry Request Id Preserved 5135ms
Partial Batch Handling.Respects Retry After On Partial 5133ms
Partial Batch Handling.Unknown Result Treated As Terminal 3029ms
Partial Batch Handling.Mixed Ok Drop Limited No Retry 3032ms
Compression.Sends Gzip Content Encoding 15ms
Compression.No Content Encoding When Disabled 5ms
Compression.Compressed Body Is Decompressible 5ms
Error Handling.Does Not Retry On Unknown 4Xx 2131ms
Event Options.Cookieless Mode Override 6ms
Event Options.Disable Skew Correction Override 4ms
Event Options.Process Person Profile Override 5ms
Event Options.Product Tour Id Override 4ms
Event Options.Unset Options Omitted 5ms
Event Options.Options Override In Batch 7ms
Geoip And Historical Migration.Geoip Disable Injected Into Properties 4ms
Geoip And Historical Migration.Historical Migration Set In Body 5ms
Geoip And Historical Migration.Historical Migration Absent By Default 5ms

Feature_Flags Tests

17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id 14ms
Request Payload.Flags Request Uses V2 Query Param 15ms
Request Payload.Flags Request Hits Flags Path Not Decide 15ms
Request Payload.Flags Request Omits Authorization Header 14ms
Request Payload.Token In Flags Body Matches Init 14ms
Request Payload.Groups Round Trip 15ms
Request Payload.Groups Default To Empty Object 15ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False 15ms
Request Payload.Disable Geoip Omitted Defaults To False 16ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key 15ms
Request Lifecycle.No Flags Request On Init Alone 2ms
Request Lifecycle.No Flags Request On Normal Capture 5ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests 28ms
Request Lifecycle.Mock Response Value Is Returned To Caller 15ms
Retry Behavior.Retries Flags On 502 17ms
Retry Behavior.Retries Flags On 504 17ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event 16ms

@posthog
posthog Bot marked this pull request as ready for review August 28, 2026 23:50
@posthog
posthog Bot requested a review from a team as a code owner August 28, 2026 23:50
@posthog

posthog Bot commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

PostHog Review alpha 🦔 If you find any issues helpful - please reply "valid", "invalid", etc., for evaluation purposes 🙏

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review

Found 3 must fix, 8 should fix.

Comment thread otel/processor.go Outdated
Comment thread otel/example/main.go Outdated
Comment thread otel/README.md
Comment thread otel/processor.go
Comment thread otel/go.mod Outdated
Comment thread otel/config.go
Comment thread otel/config.go
Comment thread otel/README.md Outdated
Comment thread otel/README.md Outdated
Comment thread otel/go.mod Outdated
posthog Bot added 8 commits August 29, 2026 00:22
The PostHog AI observability OTLP endpoint rejects requests carrying more
than 100 spans with a non-retryable HTTP 400, which drops the whole batch.
NewSpanProcessor used the OTel SDK default batch size (512), so any window
producing more than 100 AI spans silently lost data.

Cap the batch span processor at a shared maxSpansPerRequest (100) constant
and add a regression test that emits more than 100 AI spans and asserts no
export request exceeds the limit.

Generated-By: PostHog Desktop
Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The example queued the span and immediately logged "sent AI span to
PostHog", then relied on the deferred Shutdown to flush. The batch span
processor's Shutdown returns only context errors, so a rejected export
(bad API key or host) never surfaced and the program printed success and
exited 0 regardless.

Call provider.ForceFlush before the success message so the export error is
surfaced and reported, and print the success line only once the export is
confirmed. On failure the program returns rather than exiting hard so the
deferred Shutdown still runs.

Generated-By: PostHog Desktop
Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The README's ADK Go section implied a complete integration ("no further
code"). The bridge forwards spans only, but ADK Go emits prompt and
response message content as OpenTelemetry log records, not span attributes,
so those fields arrive empty in PostHog. Document the gap so users do not
read the empty prompt/response as a broken integration.

Generated-By: PostHog Desktop
Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The PostHog AI Gateway captures its own $ai_generation on every routed
call. A service that both routes through the gateway and exports spans
through this bridge double-counts and double-bills every generation, with
no signal that it is happening.

The sibling posthog-python and posthog-js SDKs guard against this with a
warning; the Go port dropped it. Port the guard: SpanProcessor.OnEnd now
inspects the AI span's server.address / url.full attributes against the
known gateway hosts and logs a warning (matching the siblings' plain-warn
behaviour, never dropping the span). Host list and detection logic mirror
posthog-python's posthog/ai/gateway.py.

Generated-By: PostHog Desktop
Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
otlptracehttp.WithEndpointURL swallows a URL parse error: it logs and
keeps its localhost:4318 default, so a malformed WithHost value produces a
working-looking processor that never reaches PostHog while the request
still carries the Authorization header. A scheme-less host such as
"us.i.posthog.com" parses but leaves the endpoint empty. Both are silent
data loss reachable straight from the documented POSTHOG_ENDPOINT path.

Validate the resolved host in newConfig and return an error from
NewSpanProcessor / NewExporter, completing the input validation the module
already performs on the API key. The host must be an absolute http or https
URL with a hostname; query and fragment are left alone as the review
scoped.

Generated-By: PostHog Desktop
Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The PostHog AI observability endpoint rejects requests carrying more than
100 spans with a non-retryable HTTP 400 that discards the whole batch.
Capping the SpanProcessor's batch size covered only that entry point; the
caller-supplied Exporter path (wired with its own batch processor, default
512) could still hand an oversized slice to a single request, because
nothing below this module splits a batch.

Wrap the OTLP exporter in a chunkingExporter that splits each ExportSpans
call into slices of at most maxSpansPerRequest. This lives in
newOTLPExporter, so it protects both the SpanProcessor and the Exporter
regardless of the feeding processor's batch size. Adds a 101+-span
regression test for the Exporter path.

Generated-By: PostHog Desktop
Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The README's usage snippet reused the construction context for the
deferred provider.Shutdown and discarded its error. When that context is
already canceled at shutdown — the standard signal.NotifyContext graceful
shutdown idiom — the OTel SDK returns ctx.Err() before running the batch
processor's final export, so every buffered AI span is dropped with no
error surfaced. This is the module's primary copy-paste path and it
contradicted the fresh-timeout-context pattern the shipped example already
uses. Mirror the example: a fresh context.WithTimeout and a reported error.

Generated-By: PostHog Desktop
Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The usage snippet built a new sdktrace.TracerProvider and made it global.
For the flagship case — an ADK Go agent inside a service that already has
OpenTelemetry configured — this loses AI spans: the OTel global delegation
fires only once, so an already-installed provider's tracers (including
ADK's, cached at package init) stay bound to it and never reach the PostHog
processor, while the replacement provider also drops the user's resource,
sampler, and existing exporters. The README prose already said "register it
on your TracerProvider", which the code contradicted.

Show provider.RegisterSpanProcessor(processor) on the application's own
*sdktrace.TracerProvider, and shut down the processor (not the provider) so
PostHog's setup does not tear down the user's tracing pipeline. Keeps the
fresh-context, error-reported shutdown from the previous fix.

Generated-By: PostHog Desktop
Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
@marandaneto
marandaneto requested a review from a team August 31, 2026 07:05
@marandaneto marandaneto self-assigned this Aug 31, 2026
@marandaneto

marandaneto commented Aug 31, 2026

Copy link
Copy Markdown
Member

Live US Cloud validation succeeded for the expanded example: the span was ingested as $ai_generation, and trace/span IDs, provider, model, messages, token counts, latency, base URL, and calculated costs were all populated as expected.

cc @PostHog/team-ai-observability
One caveat for reviewers: gen_ai.response.finish_reasons and gen_ai.response.id remain raw OpenTelemetry properties rather than being normalized to $ai_stop_reason and $ai_generation_id. The Go bridge is exporting the standard GenAI attributes correctly; this is a generic PostHog OTel ingestion normalization gap, not a Go bridge issue.

@marandaneto

Copy link
Copy Markdown
Member

ingested event example

{
  "createdAt": "2026-08-31T09:44:37.958000+00:00",
  "event": "$ai_generation",
  "id": "01a05734-e115-7103-8a19-54b4da519853",
  "properties": {
    "$ai_ingestion_source": "otel",
    "$ai_span_id": "3d0d99fac071acbe",
    "$ai_trace_id": "b46ea40bb3555a8eb0be877729a2013b",
    "$geoip_disable": true,
    "example.run_id": "20260831T094437.958644000Z",
    "gen_ai.response.finish_reasons": [
      "stop"
    ],
    "gen_ai.response.id": "chatcmpl-posthog-go-example",
    "service.name": "posthog-go-otel-example",
    "$ip": "84.115.233.26",
    "$ai_input_tokens": 18,
    "$ai_output_tokens": 11,
    "$ai_model": "gpt-4o-mini-2024-07-18",
    "$ai_provider": "openai",
    "$ai_base_url": "api.openai.com",
    "$ai_span_name": "chat posthog-go OTel example",
    "$ai_latency": 0.051059583,
    "$ai_cache_reporting_exclusive": false,
    "$ai_input_cost_usd": 0.0000027,
    "$ai_output_cost_usd": 0.0000066,
    "$ai_request_cost_usd": 0,
    "$ai_web_search_cost_usd": 0,
    "$ai_total_cost_usd": 0.0000093,
    "$ai_model_cost_used": "openai/gpt-4o-mini-2024-07-18",
    "$ai_cost_model_source": "openrouter",
    "$ai_cost_model_provider": "openai",
    "$ai_input": [
      {
        "role": "system",
        "content": "Answer concisely."
      },
      {
        "role": "user",
        "content": "What is PostHog?"
      }
    ],
    "$ai_output_choices": [
      {
        "role": "assistant",
        "content": "PostHog is an open-source product analytics platform."
      }
    ]
  },
  "sentiment": null
}

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agent-led review, human-reviewed before posting.

Comment thread otel/exporter.go
Comment thread otel/go.mod
Comment thread otel/go.mod
@dustinbyrne
dustinbyrne requested a review from a team August 31, 2026 21:07
@marandaneto
marandaneto merged commit 897a0b4 into main Sep 2, 2026
28 checks passed
@marandaneto
marandaneto deleted the posthog-self-driving/featposthog-go-add-otel-bridge-for-ai-c11a1b branch September 2, 2026 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants