diff --git a/context/commandments.yaml b/context/commandments.yaml index 3dbda94e..269ca2e1 100644 --- a/context/commandments.yaml +++ b/context/commandments.yaml @@ -316,3 +316,11 @@ commandments: - posthog-kmp is a thin wrapper that delegates to the native posthog-android, posthog-ios, and posthog-js SDKs, so session replay and autocapture availability follow the underlying native platform - Use PostHogConfig.HOST_US or PostHogConfig.HOST_EU for the host, and read the project token from generated build config or environment values rather than hardcoding it - The SDK is 0.x pre-release; check the latest posthog-kmp version on Maven Central (https://central.sonatype.com/artifact/com.posthog/posthog-kmp) before pinning instead of hardcoding a stale version + + # Carve-outs for the ai-observability skill; every variant carries this tag. + # Worded to override the generic language rules regardless of list order. + ai_observability: + - "AI Observability carve-out: this skill instruments LLM calls and is not product-analytics coverage. Do NOT add posthog.capture() events for user actions, captureException() error handlers, or a reverse proxy unless the user explicitly asks for them" + - "AI Observability carve-out: only the wrapper-client and manual-capture install paths construct a PostHog client. The OTel and framework-hook paths have no client at all, so any rule in this file about the Posthog()/PostHog() constructor, exception autocapture, atexit/shutdown registration or flushing simply does not apply — never invent a client just to satisfy one" + - "AI Observability carve-out: the $ai_* payload properties ($ai_input, $ai_output_choices, and the rest) intentionally carry user-generated prompt and completion text, so this file's PII rules do NOT apply to them. Those rules still govern every other property you set" + - "AI Observability carve-out: read the PostHog key and host exactly as the variant's install doc reads them. A direct os.environ[\"POSTHOG_API_KEY\"] / process.env lookup already fails loudly and idiomatically when unset, which satisfies this file's missing-configuration rule — do NOT add a separate presence check, guard branch, or custom raise around a bootstrap that is only a few lines long" diff --git a/context/skills/ai-observability/config.yaml b/context/skills/ai-observability/config.yaml index 558cdec8..7ff6d2ad 100644 --- a/context/skills/ai-observability/config.yaml +++ b/context/skills/ai-observability/config.yaml @@ -12,6 +12,8 @@ shared_docs: - https://posthog.com/docs/ai-observability/basics.md - https://posthog.com/docs/ai-observability/generations.md - https://posthog.com/docs/ai-observability/traces.md + - https://posthog.com/docs/ai-observability/sessions.md + - https://posthog.com/docs/ai-observability/spans.md variants: - id: openai-python framework: openai diff --git a/context/skills/ai-observability/description.md b/context/skills/ai-observability/description.md index 501391be..c43efde0 100644 --- a/context/skills/ai-observability/description.md +++ b/context/skills/ai-observability/description.md @@ -1,23 +1,23 @@ # PostHog AI Observability for {display_name} -Wire up PostHog's AI Observability so calls made through {display_name} emit `$ai_generation` events into LLM Analytics. +Wire up PostHog's AI Observability so calls made through {display_name} land in LLM Analytics as a full **session → trace → span → generation** tree — not just isolated `$ai_generation` events. ## Prerequisite — vendor LLM SDK This skill instruments the LLM calls the project *already makes*. It does **not** install the vendor SDK for you. -Check the project's manifest for the provider's package (e.g. `openai`, `@anthropic-ai/sdk`, `langchain`, `ai`, `@google/genai`). If a vendor SDK is present, pick the matching variant. If none is present, switch to the `manual-capture` variant — it posts `$ai_generation` events directly and works standalone. +Check the project's manifest for an LLM package. The catalog is far wider than the obvious providers — 68 variants covering agent frameworks (`openai-agents`, `claude-agent-sdk`, LangGraph, CrewAI, Mastra, …) and OpenAI-compatible gateways (Groq, OpenRouter, Together, Ollama, …), which an app reaches through the `openai` package plus a `baseURL` override. `1-begin.md` carries the ordered decision rules; follow them rather than matching on the first familiar package name. If no LLM SDK is present, switch to the `manual-capture` variant — it posts `$ai_generation` events directly and works standalone. -Everything else this skill needs — PostHog credentials, the OTel packages, env vars — the skill installs and configures itself. It does **not** require a pre-existing `posthog.init(...)`. If one is already there, reuse its env-var names in `3-otel-setup.md`; if not, that step sets fresh values via `set_env_values`. +Everything else this skill needs — PostHog credentials, instrumentation packages, env vars — the skill installs and configures itself. It does **not** require a pre-existing `posthog.init(...)`. If one is already there, reuse its env-var names in `3-instrument.md`; if not, that step sets fresh values via `set_env_values`. ## Steps Read every referenced file **before editing**. Then work through them in order: -1. **Begin** — see `references/1-begin.md`. Pick the right variant from the vendor SDK the project declares, then locate the LLM call sites so you know where the instrumentation will apply. -2. **Install** — see `references/2-install.md`. Add the OpenTelemetry SDK, PostHog's span-processor package, and the provider-specific instrumentation package to the manifest. -3. **Set up OpenTelemetry** — see `references/3-otel-setup.md`. Initialize the OTel TracerProvider once with `PostHogSpanProcessor`, attach the provider-specific instrumentor, and route the project token / host through environment variables. -4. **Verify** — see `references/4-verify.md`. Describe a single call the user can trigger and how to confirm `$ai_generation` events land in PostHog. +1. **Begin** — see `references/1-begin.md`. Pick the variant with the ordered rules (framework before provider, gateway base URL before the SDK it borrows), then read four facts from the code: the conversation, the user, the turn, and whether the app registers tools. +2. **Install** — see `references/2-install.md`. Declare the variant's packages in the manifest — and only those. For providers and gateways that's the PostHog SDK alongside the vendor SDK, with no OpenTelemetry packages. +3. **Instrument** — see `references/3-instrument.md`. Swap the vendor client for PostHog's wrapper, attach `$ai_session_id`, a per-turn `posthog_trace_id`, and the distinct id to every call, and capture tool runs as `$ai_span` events. This step is what turns isolated generations into a session tree. +4. **Verify** — see `references/4-verify.md`. Describe a request the user can trigger, and grade what lands in PostHog — one session, grouped traces, right attribution — rather than what the diff contains. ## Reference files @@ -28,9 +28,12 @@ The linked install page carries the exact code blocks for this variant's languag ## Key principles - **Environment variables.** Read `` and `` from env, using the framework's env-var convention. Never hardcode either value. -- **Minimal changes.** OTel initialization is a single call that runs once at process start. Place it alongside any existing PostHog init rather than restructuring the entry point. -- **Match the docs.** Package names, instrumentor imports, and processor names change between AIO releases. The install page for this variant is the source of truth. -- **Don't touch what isn't yours.** This skill instruments generations only. Identify calls, event tracking, error tracking, and dashboards belong to the base `integration` skill — do not add or edit them here. +- **The SDK wrapper is the default, not OpenTelemetry.** OTel makes the session tree awkward to build and maintain, so provider and gateway variants use PostHog's drop-in wrapper client. Reserve OTel for the `opentelemetry-*` variants and LlamaIndex, and never swap a framework's own tracing hook for an instrumentor. +- **Minimal changes.** The wrapper swaps a client constructor and adds parameters to existing calls. Don't restructure the app, and don't wrap the setup in an init function or module globals. +- **Match the docs.** Package names and wrapper imports change between AIO releases. The install page for this variant is the source of truth. +- **Cardinality is what gets graded.** One `$ai_session_id` per conversation, one `posthog_trace_id` per turn, shared by every call in it. An id minted per call is worse than none — it looks instrumented and groups nothing. +- **Tools become spans.** When the app registers tools, capture each execution as an `$ai_span` event sharing the turn's trace id — the wrapper never sees your dispatch loop. Framework variants emit these themselves; an app with no tools correctly has none. +- **Don't touch what isn't yours.** This skill instruments LLM observability only — generations, traces, sessions, spans. Identify calls, event tracking, error tracking, and dashboards belong to the base `integration` skill — do not add or edit them here. ## Emit a run record @@ -40,6 +43,8 @@ When you finish, write `.posthog-wizard-cache/.posthog-ai.json` at the project r { "provider": "openai", "package": "@posthog/ai", "otel_init_file": "src/instrumentation.ts" } ``` +`otel_init_file` keeps its name for the report's sake, but on the wrapper path there is no OTel init — set it to the file where the wrapper client was constructed (or, on the manual path, where the capture helper lives). + The `report/` step reads this file to render an AI Observability section in the setup report. If the cache directory does not exist, create it. ## Framework guidelines diff --git a/context/skills/ai-observability/references/1-begin.md b/context/skills/ai-observability/references/1-begin.md index 21c23932..1ef2f318 100644 --- a/context/skills/ai-observability/references/1-begin.md +++ b/context/skills/ai-observability/references/1-begin.md @@ -1,52 +1,70 @@ --- next_step: 2-install.md title: AI Observability Setup - Begin -description: Pick the right variant, confirm prerequisites, and locate the LLM call sites before editing +description: Pick the variant that matches this project, then read the four facts the instrumentation needs --- -Before touching any code, decide which variant of this skill to install, confirm the two prerequisites, and get a read on where in the project LLM calls actually happen. AI Observability instruments an existing setup — if the setup isn't there, this skill can't do its job. +Pick the variant, then read the code. Do not edit anything in this step. ## Pick the variant -The `ai-observability` skill has one variant per LLM provider × language (e.g. `ai-observability-openai-python`, `ai-observability-anthropic-node`, …). You are running the group-level entry; before installing, pick the specific variant that matches this project. +This skill ships 68 variants. Call `load_skill_menu` with `category: "ai-observability"`. That list is the source of truth. -Scan the manifest (`package.json`, `pyproject.toml`, `requirements.txt`, `Gemfile`) for a vendor LLM package. Typical package names: +Apply these rules in order. The first match wins. Frameworks wrap providers, and gateways look like OpenAI, so the order matters. -- OpenAI — `openai` -- Anthropic — `@anthropic-ai/sdk` (Node) or `anthropic` (Python) -- LangChain — `langchain` / `@langchain/core` (plus a provider adapter like `langchain-openai` / `@langchain/openai`) -- Vercel AI SDK — `ai` (plus a provider like `@ai-sdk/openai`) -- Google Gemini — `google-genai` (Python) or `@google/genai` (Node) +### 1. A framework wins over the provider under it -Decision rules — apply in order: +| Package in the manifest | Variant | +|---|---| +| `openai-agents` | `openai-agents` | +| `claude-agent-sdk` | `claude-agent-sdk` | +| `langchain`, `@langchain/core` | `langchain-{python,node}` | +| `langgraph`, `@langchain/langgraph` | `langgraph-{python,node}` | +| `ai` (Vercel AI SDK) | `vercel-ai` | +| `llama-index`, `llamaindex` | `llamaindex` | +| `crewai` | `crewai` | +| `pyautogen`, `autogen-agentchat` | `autogen` | +| `dspy`, `dspy-ai` | `dspy` | +| `pydantic-ai` | `pydantic-ai` | +| `semantic-kernel` | `semantic-kernel` | +| `smolagents` | `smolagents` | +| `mirascope` | `mirascope` | +| `instructor` | `instructor-{python,node}` | +| `litellm` | `litellm` | +| `mastra`, `@mastra/core` | `mastra` | +| `convex` | `convex` | -1. **Exactly one vendor SDK found** → pick the corresponding variant. Language follows the manifest (`package.json` → Node, `pyproject.toml`/`requirements.txt` → Python). Tell the user which variant you picked and why in a `[STATUS]` line, then call `install_skill` with the full variant id (e.g. `ai-observability-openai-python`). -2. **Multiple vendor SDKs found** (e.g. LangChain wraps OpenAI, so both may be declared) → prefer the higher-level abstraction: LangChain > direct provider SDK. If still ambiguous, use `wizard_ask` to have the user pick, listing the candidates as options. -3. **No vendor SDK found** → install `ai-observability-manual-capture`. This variant posts `$ai_generation` events explicitly and works without any auto-instrumentation. -4. **You're not sure** → `wizard_ask` the user with a multi-choice picker listing every provider you have a variant for. Do not guess when there's real ambiguity. +Instrument the framework, not the provider below it. A provider instrumentor keeps the model calls and loses the agent, tool, and handoff structure. -## Check for an existing PostHog setup (informational — not a blocker) +### 2. An `openai` client with a base URL override means a gateway -Grep the project for one of: +Most OpenAI-compatible providers ship no SDK. Apps call them with the `openai` package aimed at another host. Check the client constructor and `OPENAI_BASE_URL`. Common hosts are `api.groq.com`, `openrouter.ai`, `api.together.xyz`, and `localhost:11434`. -- `posthog.init(` — most JS/TS SDKs -- `PostHog(` — Python, Ruby, Go SDK constructors -- `AddPostHog(` — .NET DI registration +Pick the variant that names the provider. The install shape matches plain OpenAI, but the provider name does not. Step 3 explains why that matters. -**This is not a prerequisite.** The OTel-based variants use `PostHogSpanProcessor`, a self-contained exporter that just takes an API key + host — it does not depend on a `posthog.init(...)` call anywhere. The `manual-capture` variant uses `posthog.capture(...)`, which needs the traditional SDK, but the install step will add it if it isn't there. +### 3. A plain provider SDK maps to that provider -If a `posthog.init(...)` (or equivalent) **is** already present, note the env-var names it reads (`POSTHOG_API_KEY`, `NEXT_PUBLIC_POSTHOG_KEY`, etc.) and reuse them in `3-otel-setup.md` — don't invent parallel names. If nothing is there, `3-otel-setup.md` will set fresh values via `set_env_values`. +`openai`, `anthropic`, `@anthropic-ai/sdk`, `google-genai`, `@google/genai`, `mistralai`, and `cohere` each have a variant. `boto3` with `bedrock-runtime` maps to `aws-bedrock`. -## Locate the LLM call sites +### 4. Anything else -Grep for where the vendor SDK is imported and called. This is not a full analysis — one or two representative sites is enough for you to reason about where OTel initialization has to run before those calls execute: +- Several candidates and no framework: prefer the higher-level one. If it stays unclear, use `wizard_ask` with the candidates as options. +- The app already emits its own OTel spans: `opentelemetry-{python,node}`. +- No LLM SDK at all: `manual-capture`. -- OpenAI: `import OpenAI`, `openai.OpenAI(`, `new OpenAI(` -- Anthropic: `Anthropic(`, `new Anthropic(` -- LangChain: `ChatOpenAI(`, `from langchain`, `import { ChatOpenAI } from '@langchain/openai'` -- Vercel AI: `generateText(`, `streamText(`, `import ... from 'ai'` -- Google Gemini: `genai.Client(`, `new GoogleGenerativeAI(` +Language follows the manifest. A `package.json` means Node. A `pyproject.toml` or `requirements.txt` means Python. Framework variants have no language suffix. -Note the app's entry point (server startup file, `main.py`, `index.ts`, `instrumentation.ts` in Next.js, etc.) — OTel must be initialized *before* the vendor SDK is imported, and the entry point is where that happens. +Report the variant and the reason in a `[STATUS]` line, then call `install_skill` with the full id. -Do not edit yet. Once you have a note of the entry point and the call sites, move on to `2-install.md`. +## Read four facts from the code + +The install doc holds the code. It cannot know this app. Step 3 uses these answers and nothing else. + +1. **Conversation.** The field that groups turns, such as `thread_id` or `conversation_id`. If the app has none, the process run is the conversation. +2. **User.** The user id in scope at the call sites. If the app has none, the events stay anonymous. Do not invent one. +3. **Turn.** The function that takes one question and returns one answer. It may call the model several times. +4. **Tools.** Does the app register tools with its calls? Look for a `tools=` argument or a tool decorator. Find the loop that runs them. + +Note the module that builds the vendor client. Step 3 replaces that constructor. If the project already calls `posthog.init(...)` or `PostHog(...)`, reuse its env-var names and its client. + +Go to `2-install.md`. diff --git a/context/skills/ai-observability/references/2-install.md b/context/skills/ai-observability/references/2-install.md index 962272c3..a97de2a9 100644 --- a/context/skills/ai-observability/references/2-install.md +++ b/context/skills/ai-observability/references/2-install.md @@ -1,76 +1,41 @@ --- -next_step: 3-otel-setup.md +next_step: 3-instrument.md title: AI Observability Setup - Install -description: Add the OpenTelemetry SDK, PostHog span processor, and provider instrumentation to the manifest +description: Declare the packages the variant needs, and no others --- -Declare the packages this variant needs in the project's manifest. Do not run the package manager here — the base integration's build/verify step (or the user) installs everything in one pass. +Declare the packages in the project manifest. Do not run the package manager. The build step installs them later. -Read the manifest first. If any of the required packages is already declared, leave the existing version alone and say so. Match the style of dependencies already in the file (versions, ordering, dev vs. runtime). +Read the manifest first. If a package is already there, keep its version and say so in the report. Match the style of the entries around it. -## What to add +## Providers and gateways -The linked install page for this variant carries the authoritative command. The shapes below reflect the current defaults across Tier-1 providers. +Every provider and every OpenAI-compatible gateway needs the PostHog SDK next to the vendor SDK. -### Python — the standard OTel path +| Runtime | Packages | +|---|---| +| Python | `posthog` | +| Node | `@posthog/ai`, `posthog-node` | -Three packages: +**Do not add OpenTelemetry packages.** The wrapper path needs none. If you reach for `opentelemetry-sdk`, `posthog[otel]`, or an `opentelemetry-instrumentation-*` package, you picked the wrong mechanism. Go back to `3-instrument.md`. -``` -posthog[otel] # PostHog SDK + span processor -opentelemetry-sdk # OTel core -opentelemetry-instrumentation- # provider auto-instrumentation -``` +The vendor SDK is already in the manifest. Do not add or upgrade it. -Examples: +Portkey also needs `portkey-ai`. -- OpenAI → `opentelemetry-instrumentation-openai-v2` -- Anthropic → `opentelemetry-instrumentation-anthropic` -- LangChain → `opentelemetry-instrumentation-langchain` -- Google Gemini → `opentelemetry-instrumentation-google-generativeai` +## Other variants -The vendor SDK itself (`openai`, `anthropic`, `langchain`, …) is a prerequisite — it should already be declared. Do not add or upgrade it. +| Variant | Packages | +|---|---| +| Agent frameworks | The list the install doc names | +| `opentelemetry-*`, LlamaIndex, AWS Bedrock | The OTel packages the install doc names | +| `manual-capture` | `posthog` or `posthog-node` | -### Node — the standard OTel path +AWS Bedrock has no wrapper client. It instruments the AWS SDK through OpenTelemetry, so its package list differs from every other provider. -``` -@posthog/ai # PostHog span processor -@opentelemetry/sdk-node # OTel core (Node) -@opentelemetry/resources # resource attributes -@opentelemetry/instrumentation- # provider auto-instrumentation -``` +## Do not -Provider instrumentation packages: - -- OpenAI → `@opentelemetry/instrumentation-openai` -- Anthropic → `@traceloop/instrumentation-anthropic` -- LangChain → `@traceloop/instrumentation-langchain` -- Google Gemini → `@traceloop/instrumentation-google-generativeai` - -### Vercel AI SDK (Node) - -No provider instrumentation package — Vercel AI emits OTel spans natively when `experimental_telemetry` is enabled. Just add: - -``` -@posthog/ai -@opentelemetry/sdk-node -@opentelemetry/resources -``` - -### `manual-capture` variant - -No OTel packages. Just PostHog core: - -``` -posthog # Python -posthog-node # Node -``` - -If PostHog core is already installed (it should be — see `1-begin.md`), this file has nothing to add. Skip to `3-otel-setup.md`, which describes the manual `capture(...)` call shape. - -## Do not do - -- Do not run `npm install` / `pip install` here. +- Do not run `npm install` or `pip install`. - Do not edit the lockfile. - Do not upgrade the vendor SDK. -- Do not add both an OTel-based instrumentation package *and* the older wrapper client — pick one. This skill uses the OTel path. +- Do not add OpenTelemetry to a wrapper variant. diff --git a/context/skills/ai-observability/references/3-instrument.md b/context/skills/ai-observability/references/3-instrument.md new file mode 100644 index 00000000..b635b252 --- /dev/null +++ b/context/skills/ai-observability/references/3-instrument.md @@ -0,0 +1,70 @@ +--- +next_step: 4-verify.md +title: AI Observability Setup - Instrument +description: Swap in the wrapper client, then attach identity and tool spans so the calls form a session tree +--- + +The install doc holds the code for this variant. Copy it and change the values. This step covers what the doc cannot know: the values this app supplies, and the shape the result must have. + +## Swap the client + +Build a PostHog client. Replace the vendor client with the PostHog wrapper for this provider. The wrapper takes the same constructor arguments and stays call-compatible, so the existing calls keep working. A gateway keeps its `base_url`. + +Keep the setup at module level, next to the existing client. Under ten lines is normal. + +- Do not wrap it in an init function. +- Do not add module globals. +- Do not add a presence check that raises. `os.environ["POSTHOG_API_KEY"]` already fails loudly when the key is unset. + +Route the token and host through env vars with `set_env_values`. Reuse the names the project already uses. Add the names to `.env.example` with empty values. Never write a real key to a file. + +Agent frameworks use their own tracing hook in place of a wrapper. Take it from the install doc. Do not substitute an OTel instrumentor. + +## Attach identity to every call + +Three per-call parameters carry the tree. Node uses camelCase. + +| Parameter | Holds | Cardinality | +|---|---|---| +| `posthog_properties` with `$ai_session_id` | the conversation | one id for the whole conversation | +| `posthog_trace_id` | the turn | a new id per turn | +| `posthog_distinct_id` | the user | the person | + +Every call inside one turn takes the same `posthog_trace_id`. If you omit it, the wrapper mints a fresh id per call, and each generation lands in its own trace. + +**Cardinality is what gets graded.** One conversation is one session id. One turn is one trace id. An id minted per call looks instrumented and groups nothing. + +Every app gets a session id. If the app has no conversation field, the process run is the session. Do not skip the step. + +If the app has no user id, leave `posthog_distinct_id` out. Anonymous is a finding for the report. Do not invent an id. + +`$ai_session_id` accepts letters, numbers, and `- _ ~ . @ ( ) ! ' : |`. A raw thread id with a slash or a hash fails. Check the value before you pass it through. + +### A gateway must name its provider + +The OpenAI wrapper reports `openai` whatever host it calls. PostHog prices tokens by `$ai_model` and `$ai_provider`. A gateway call at the default gets the wrong price. Send the real provider: + +```python +posthog_properties={"$ai_session_id": session_id, "$ai_provider": "groq"} +``` + +## Capture tool calls as spans + +The wrapper records the model call. It never sees the tool dispatch loop, so nothing else records a tool run. + +If the app registers tools, capture each run as an `$ai_span` event with `posthog.capture()`. Give it the turn's `$ai_trace_id` so the span joins the trace. The install doc lists the span properties. + +Put the capture next to the existing dispatch. Do not restructure the tool loop. + +Agent frameworks and the Vercel AI SDK emit tool spans on their own. Add nothing on those variants. + +An app that registers no tools has no spans. That is a complete result, not a gap. + +## Do not + +- Do not restructure the app. This step swaps a constructor and adds arguments to calls. +- Do not omit `posthog_trace_id` and expect the calls to group. +- Do not mint a session id per call or per turn. +- Do not leave a gateway reporting `$ai_provider` as `openai`. +- Do not add spans when the app registers no tools. +- Do not ship code whose imports fail. Go back to `1-begin.md` and pick another variant. diff --git a/context/skills/ai-observability/references/3-otel-setup.md b/context/skills/ai-observability/references/3-otel-setup.md deleted file mode 100644 index 70517a02..00000000 --- a/context/skills/ai-observability/references/3-otel-setup.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -next_step: 4-verify.md -title: AI Observability Setup - OpenTelemetry -description: Initialize the OTel TracerProvider with PostHog's span processor and attach the provider instrumentor ---- - -Initialize OpenTelemetry once, at the app's entry point, so it is running before any LLM call executes. This is the single place PostHog reads the project token and host from — everything after this is normal LLM code that gets auto-traced. - -## Environment variables - -Route the PostHog credentials through env vars, using the wizard's `set_env_values` tool (never hardcode). Reuse whatever names the base PostHog integration already set — typically: - -- the public project token (e.g. `POSTHOG_API_KEY`, `NEXT_PUBLIC_POSTHOG_KEY`, or the framework's convention) -- the PostHog host (e.g. `POSTHOG_HOST`, `NEXT_PUBLIC_POSTHOG_HOST`) - -If the project has an `.env.example` file, add the names there with empty placeholder values so collaborators know what to set. Create `.env.example` if it doesn't exist. Never write real secrets to any file. - -## Initialization shape - -There is one initialization call per app. It runs once, at startup, before any vendor SDK call. The linked install page carries the exact code for this variant's language — copy from there. - -### Python — the standard OTel path - -Wire a `TracerProvider` with `PostHogSpanProcessor`, then call `.instrument()` on the provider-specific instrumentor: - -```python -from opentelemetry import trace -from opentelemetry.sdk.trace import TracerProvider -from opentelemetry.sdk.resources import Resource, SERVICE_NAME -from posthog.ai.otel import PostHogSpanProcessor -from opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor # swap per provider - -provider = TracerProvider(resource=Resource(attributes={SERVICE_NAME: "my-app"})) -provider.add_span_processor(PostHogSpanProcessor( - api_key=os.environ["POSTHOG_API_KEY"], - host=os.environ["POSTHOG_HOST"], -)) -trace.set_tracer_provider(provider) - -OpenAIInstrumentor().instrument() -``` - -### Node — the standard OTel path - -Use `NodeSDK` with `PostHogSpanProcessor` and the provider instrumentation. Start it before importing the vendor SDK in the app's entry point: - -```typescript -import { NodeSDK } from '@opentelemetry/sdk-node' -import { resourceFromAttributes } from '@opentelemetry/resources' -import { PostHogSpanProcessor } from '@posthog/ai/otel' -import { OpenAIInstrumentation } from '@opentelemetry/instrumentation-openai' // swap per provider - -const sdk = new NodeSDK({ - resource: resourceFromAttributes({ 'service.name': 'my-app' }), - spanProcessors: [new PostHogSpanProcessor({ - apiKey: process.env.POSTHOG_API_KEY!, - host: process.env.POSTHOG_HOST!, - })], - instrumentations: [new OpenAIInstrumentation()], -}) -sdk.start() -``` - -For Next.js / Nuxt / other frameworks that expose a dedicated startup hook (`instrumentation.ts`, `nuxt.config`'s server plugin, etc.), put the SDK init there. It must run once per process, not per request. - -### Vercel AI SDK - -No instrumentor package. Initialize `NodeSDK` with just `PostHogSpanProcessor` (no `instrumentations` array), then pass `experimental_telemetry` per call: - -```typescript -const result = await generateText({ - model: openai('gpt-5-mini'), - prompt: '...', - experimental_telemetry: { isEnabled: true, functionId: 'my-ai-function' }, -}) -``` - -If the project has many call sites, wrap the config into a shared helper rather than repeating it inline. - -## The PostHog client - -Some shapes need a PostHog client instance — the `manual-capture` variant, and wrappers that take a client. Look for a reusable one before creating anything: - -- Search the project for an existing client: a `posthog-js` init on the frontend, a `PostHog` instance from `posthog` / `posthog-node` on the backend. -- Frontend: the client is a singleton — always reuse it, never instantiate a second one. -- Backend: reuse a shared instance if the project has one; if none exists, create one where the instrumentation lives — once, at module level, reused across call sites — and use it when initializing the wrappers. - -### `manual-capture` variant - -No OTel. Capture each generation explicitly at the call site: - -```python -posthog.capture( - distinct_id="user_123", - event="$ai_generation", - properties={ - "$ai_provider": "openai", - "$ai_model": "gpt-4", - "$ai_input": [{"role": "user", "content": prompt}], - "$ai_output_choices": [{"role": "assistant", "content": response}], - "$ai_input_tokens": usage.prompt_tokens, - "$ai_output_tokens": usage.completion_tokens, - "$ai_latency": latency_seconds, - }, -) -``` - -Refer to https://posthog.com/docs/ai-observability/installation/manual-capture for the full property list. - -## Do not - -- Do not create a PostHog client where one already exists — search first and reuse it (the frontend client is a singleton). Only a backend with no client creates one, once, at module level. Either way, reuse the project token / host already in the app's env — the OTel span processor is a separate exporter, not a separate PostHog install. -- Do not put SDK init inside a request handler. Once per process, at startup. -- Do not import the vendor SDK above the OTel init in the same file — the instrumentor patches the SDK when it loads, so the order matters. diff --git a/context/skills/ai-observability/references/4-verify.md b/context/skills/ai-observability/references/4-verify.md index bb5925f7..1c07b39d 100644 --- a/context/skills/ai-observability/references/4-verify.md +++ b/context/skills/ai-observability/references/4-verify.md @@ -1,37 +1,52 @@ --- next_step: null title: AI Observability Setup - Verify -description: Give the user a concrete way to trigger a generation and confirm it lands in PostHog +description: Give the user a way to trigger one turn, and grade the tree that reaches PostHog --- -You've installed packages and wired OTel init. The last thing this skill produces is a **verification path** the user can run themselves. Don't call the LLM from here — you don't have credentials, and the user should be the one to see the trace show up in their project. +Give the user a way to run one turn. Do not call the model yourself. You hold no credentials, and the user should watch the trace arrive. -## What to tell the user +Grade what reaches PostHog, not what the diff contains. A clean diff that produces one trace per call is a failed run. -Point them at the smallest existing code path in their project that hits the vendor SDK. Pick one from what you noted in `1-begin.md`: +## What correct looks like -- If the project has a script (`scripts/`, `bin/`, a `package.json` script) that calls the LLM, name it: "Run `npm run