feat: add OpenTelemetry bridge for AI observability - #306
Conversation
Adds a posthogotel Go module that forwards OpenTelemetry AI spans to PostHog AI observability. It keeps only spans that follow a known AI semantic convention (gen_ai., llm., ai., traceloop.) and sends them over OTLP/HTTP to the /i/v0/ai/otel endpoint with the project API key as a bearer token. The bridge is a separate Go module so the core posthog-go SDK stays free of OpenTelemetry dependencies. It ports the posthog-python OTel bridge: a SpanProcessor for TracerProvider setups and an Exporter for frameworks that accept only a span exporter. An example wires it to a Google ADK Go agent through the shared tracer provider. Generated-By: PostHog Desktop Task-Id: 5d65718b-f73a-48c4-9c4d-0fdd2d00fa17
🦔 PostHog Review reviewed this pull requestFound 3 must fix, 8 should fix, 0 consider. Published 11 findings (view the review). Resolved comments: 8 fixed, 3 left for you |
posthog-go Compliance ReportDate: 2026-09-01 15:24:01 UTC ✅ All Tests Passed!111/111 tests passed Capture_V1 Tests✅ 94/94 tests passed View Details
Feature_Flags Tests✅ 17/17 tests passed View Details
|
|
PostHog Review alpha 🦔 If you find any issues helpful - please reply "valid", "invalid", etc., for evaluation purposes 🙏 |
The PostHog AI observability OTLP endpoint rejects requests carrying more than 100 spans with a non-retryable HTTP 400, which drops the whole batch. NewSpanProcessor used the OTel SDK default batch size (512), so any window producing more than 100 AI spans silently lost data. Cap the batch span processor at a shared maxSpansPerRequest (100) constant and add a regression test that emits more than 100 AI spans and asserts no export request exceeds the limit. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The example queued the span and immediately logged "sent AI span to PostHog", then relied on the deferred Shutdown to flush. The batch span processor's Shutdown returns only context errors, so a rejected export (bad API key or host) never surfaced and the program printed success and exited 0 regardless. Call provider.ForceFlush before the success message so the export error is surfaced and reported, and print the success line only once the export is confirmed. On failure the program returns rather than exiting hard so the deferred Shutdown still runs. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The README's ADK Go section implied a complete integration ("no further
code"). The bridge forwards spans only, but ADK Go emits prompt and
response message content as OpenTelemetry log records, not span attributes,
so those fields arrive empty in PostHog. Document the gap so users do not
read the empty prompt/response as a broken integration.
Generated-By: PostHog Desktop
Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The PostHog AI Gateway captures its own $ai_generation on every routed call. A service that both routes through the gateway and exports spans through this bridge double-counts and double-bills every generation, with no signal that it is happening. The sibling posthog-python and posthog-js SDKs guard against this with a warning; the Go port dropped it. Port the guard: SpanProcessor.OnEnd now inspects the AI span's server.address / url.full attributes against the known gateway hosts and logs a warning (matching the siblings' plain-warn behaviour, never dropping the span). Host list and detection logic mirror posthog-python's posthog/ai/gateway.py. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
otlptracehttp.WithEndpointURL swallows a URL parse error: it logs and keeps its localhost:4318 default, so a malformed WithHost value produces a working-looking processor that never reaches PostHog while the request still carries the Authorization header. A scheme-less host such as "us.i.posthog.com" parses but leaves the endpoint empty. Both are silent data loss reachable straight from the documented POSTHOG_ENDPOINT path. Validate the resolved host in newConfig and return an error from NewSpanProcessor / NewExporter, completing the input validation the module already performs on the API key. The host must be an absolute http or https URL with a hostname; query and fragment are left alone as the review scoped. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The PostHog AI observability endpoint rejects requests carrying more than 100 spans with a non-retryable HTTP 400 that discards the whole batch. Capping the SpanProcessor's batch size covered only that entry point; the caller-supplied Exporter path (wired with its own batch processor, default 512) could still hand an oversized slice to a single request, because nothing below this module splits a batch. Wrap the OTLP exporter in a chunkingExporter that splits each ExportSpans call into slices of at most maxSpansPerRequest. This lives in newOTLPExporter, so it protects both the SpanProcessor and the Exporter regardless of the feeding processor's batch size. Adds a 101+-span regression test for the Exporter path. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The README's usage snippet reused the construction context for the deferred provider.Shutdown and discarded its error. When that context is already canceled at shutdown — the standard signal.NotifyContext graceful shutdown idiom — the OTel SDK returns ctx.Err() before running the batch processor's final export, so every buffered AI span is dropped with no error surfaced. This is the module's primary copy-paste path and it contradicted the fresh-timeout-context pattern the shipped example already uses. Mirror the example: a fresh context.WithTimeout and a reported error. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
The usage snippet built a new sdktrace.TracerProvider and made it global. For the flagship case — an ADK Go agent inside a service that already has OpenTelemetry configured — this loses AI spans: the OTel global delegation fires only once, so an already-installed provider's tracers (including ADK's, cached at package init) stay bound to it and never reach the PostHog processor, while the replacement provider also drops the user's resource, sampler, and existing exporters. The README prose already said "register it on your TracerProvider", which the code contradicted. Show provider.RegisterSpanProcessor(processor) on the application's own *sdktrace.TracerProvider, and shut down the processor (not the provider) so PostHog's setup does not tear down the user's tracing pipeline. Keeps the fresh-context, error-reported shutdown from the previous fix. Generated-By: PostHog Desktop Task-Id: 01e1bd45-4478-4317-8196-8ddfcfe917bf
|
Live US Cloud validation succeeded for the expanded example: the span was ingested as cc @PostHog/team-ai-observability |
|
ingested event example {
"createdAt": "2026-08-31T09:44:37.958000+00:00",
"event": "$ai_generation",
"id": "01a05734-e115-7103-8a19-54b4da519853",
"properties": {
"$ai_ingestion_source": "otel",
"$ai_span_id": "3d0d99fac071acbe",
"$ai_trace_id": "b46ea40bb3555a8eb0be877729a2013b",
"$geoip_disable": true,
"example.run_id": "20260831T094437.958644000Z",
"gen_ai.response.finish_reasons": [
"stop"
],
"gen_ai.response.id": "chatcmpl-posthog-go-example",
"service.name": "posthog-go-otel-example",
"$ip": "84.115.233.26",
"$ai_input_tokens": 18,
"$ai_output_tokens": 11,
"$ai_model": "gpt-4o-mini-2024-07-18",
"$ai_provider": "openai",
"$ai_base_url": "api.openai.com",
"$ai_span_name": "chat posthog-go OTel example",
"$ai_latency": 0.051059583,
"$ai_cache_reporting_exclusive": false,
"$ai_input_cost_usd": 0.0000027,
"$ai_output_cost_usd": 0.0000066,
"$ai_request_cost_usd": 0,
"$ai_web_search_cost_usd": 0,
"$ai_total_cost_usd": 0.0000093,
"$ai_model_cost_used": "openai/gpt-4o-mini-2024-07-18",
"$ai_cost_model_source": "openrouter",
"$ai_cost_model_provider": "openai",
"$ai_input": [
{
"role": "system",
"content": "Answer concisely."
},
{
"role": "user",
"content": "What is PostHog?"
}
],
"$ai_output_choices": [
{
"role": "assistant",
"content": "PostHog is an open-source product analytics platform."
}
]
},
"sentiment": null
} |
dustinbyrne
left a comment
There was a problem hiding this comment.
Agent-led review, human-reviewed before posting.
💡 Motivation and Context
$ai_*event arrived with$lib = posthog-go, which matched the code: grepping the SDK for$ai_generation,$ai_span, or anythingllm-shaped returned nothing. Those users got a flat "no" on a product they were otherwise ready to pay for.posthog-pythonships a generic OpenTelemetry bridge that keepsgen_ai./llm./ai./traceloop.spans and forwards them to/i/v0/ai/otel. Google's Agent Development Kit (ADK) for Go emits exactly thosegen_ai.*spans — there was just nothing on the Go side pointing at them.This PR ports that bridge to Go.
github.com/posthog/posthog-go/otelmodule (posthogotel):SpanProcessor— recommended integration; register it on aTracerProvider. Filters, batches, and exports AI spans.Exporter— for frameworks that accept only a span exporter.IsAISpan— the shared prefix filter (gen_ai.,llm.,ai.,traceloop.), matched against the span name and every attribute key.{host}/i/v0/ai/otelover OTLP/HTTP withAuthorization: Bearer <project api key>;WithHostswitches cloud region.example/wires the processor to the tracer provider an ADK Go agent emits through.Why a separate Go module
The core SDK today has a lean dependency graph. Folding the OTel SDK,
otlptracehttp, and their protobuf/grpc transitive tree into the rootgo.modwould burden everyposthog-goconsumer for an optional feature. A nested module keeps those dependencies opt-in, so only users who import.../otelpull them.💚 How did you test it?
IsAISpan: name-prefix match, attribute-key match, and a non-AI negative, using realReadOnlySpans from aSpanRecorder.httptestOTLP server that decodes the protobuf payload and asserts:gen_ai.chat) reaches the endpoint when an AI and a non-AI span are emitted together, for bothSpanProcessorandExporter;/i/v0/ai/oteland the header isAuthorization: Bearer <key>;WithHostnormalisation (blank falls back to default, trailing slash trimmed).go build,go vet, andgo test -racepass in the module. Added anotel-bridgeCI job (Go 1.25 and 1.x), since the root./...does not reach a nested module.📝 Checklist
If releasing new changes
pnpm changesetto generate a changeset fileThe root SDK and nested
otelmodule share one release version and cadence. The release workflow creates bothvX.Y.Zandotel/vX.Y.Ztags at the same release commit, matching the synchronized nested-module approach used by Sentry Go.🤖 Agent context
Autonomy: Fully autonomous
posthog-pythonposthog/ai/otel/bridge (spans.py,exporter.py,processor.py) as the reference and ported its behaviour..gofiles in the root package: the report suggested following theerror_tracking*.gopattern, but that product added no third-party dependencies, whereas the OTel bridge pulls in grpc/protobuf. Isolating it keeps the core SDK lean.v1.43.0and the bridge module to Go 1.25 to avoid reachable vulnerabilities in older OTel releases; the separate core module remains on Go 1.21.gen_ai.*span, with the ADK hookup documented.Created with PostHog Desktop from this inbox report.