From 8a69f193c70eba5a55e547fb5b637d3c8014c8ff Mon Sep 17 00:00:00 2001 From: Marco Gancitano Date: Thu, 23 Jul 2026 13:50:39 -0400 Subject: [PATCH 01/15] feat(ai-observability): make session/trace/span nesting a mandatory setup step MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The skill's success bar was 'one $ai_generation event appears', so runs that only did the OTel bootstrap passed while shipping flat, disconnected generations — no trace grouping, no session, no user attribution, and non-LLM steps invisible. Verified by running the skill twice against one unchanged fixture: one run built the tree, one didn't, both 'passed'. - 1-begin: map the app's logical structure up front (what is a session, a trace, a span-worthy step, the distinct-id source) - new 4-nesting: build the tree per mechanism — enclosing span (OTel), shared posthog_trace_id (wrapper SDKs), explicit $ai_parent_id (manual-capture); calls out the silent ai.* span-name drop rule and the Resource-vs-per-span session propagation caveat - 5-verify (was 4-verify): success is the tree — one trace holding the request's generations and spans, session grouping, person attribution; a lone generation is 'bootstrap verified, nesting unverified' - shared_docs: add sessions.md and spans.md Co-Authored-By: Claude Opus 4.8 --- context/skills/ai-observability/config.yaml | 2 + .../skills/ai-observability/description.md | 10 +- .../ai-observability/references/1-begin.md | 13 ++- .../references/3-otel-setup.md | 6 +- .../ai-observability/references/4-nesting.md | 96 +++++++++++++++++++ .../ai-observability/references/4-verify.md | 37 ------- .../ai-observability/references/5-verify.md | 47 +++++++++ 7 files changed, 166 insertions(+), 45 deletions(-) create mode 100644 context/skills/ai-observability/references/4-nesting.md delete mode 100644 context/skills/ai-observability/references/4-verify.md create mode 100644 context/skills/ai-observability/references/5-verify.md diff --git a/context/skills/ai-observability/config.yaml b/context/skills/ai-observability/config.yaml index 558cdec8..7ff6d2ad 100644 --- a/context/skills/ai-observability/config.yaml +++ b/context/skills/ai-observability/config.yaml @@ -12,6 +12,8 @@ shared_docs: - https://posthog.com/docs/ai-observability/basics.md - https://posthog.com/docs/ai-observability/generations.md - https://posthog.com/docs/ai-observability/traces.md + - https://posthog.com/docs/ai-observability/sessions.md + - https://posthog.com/docs/ai-observability/spans.md variants: - id: openai-python framework: openai diff --git a/context/skills/ai-observability/description.md b/context/skills/ai-observability/description.md index 501391be..b3037fe6 100644 --- a/context/skills/ai-observability/description.md +++ b/context/skills/ai-observability/description.md @@ -1,6 +1,6 @@ # PostHog AI Observability for {display_name} -Wire up PostHog's AI Observability so calls made through {display_name} emit `$ai_generation` events into LLM Analytics. +Wire up PostHog's AI Observability so calls made through {display_name} land in LLM Analytics as a full **session → trace → span → generation** tree — not just isolated `$ai_generation` events. ## Prerequisite — vendor LLM SDK @@ -14,10 +14,11 @@ Everything else this skill needs — PostHog credentials, the OTel packages, env Read every referenced file **before editing**. Then work through them in order: -1. **Begin** — see `references/1-begin.md`. Pick the right variant from the vendor SDK the project declares, then locate the LLM call sites so you know where the instrumentation will apply. +1. **Begin** — see `references/1-begin.md`. Pick the right variant from the vendor SDK the project declares, locate the LLM call sites, and map the app's logical structure (what a session, trace, span, and user are in this codebase). 2. **Install** — see `references/2-install.md`. Add the OpenTelemetry SDK, PostHog's span-processor package, and the provider-specific instrumentation package to the manifest. 3. **Set up OpenTelemetry** — see `references/3-otel-setup.md`. Initialize the OTel TracerProvider once with `PostHogSpanProcessor`, attach the provider-specific instrumentor, and route the project token / host through environment variables. -4. **Verify** — see `references/4-verify.md`. Describe a single call the user can trigger and how to confirm `$ai_generation` events land in PostHog. +4. **Build the nesting** — see `references/4-nesting.md`. Group each request's calls into one trace, attach session and distinct id, and wrap non-LLM steps as spans. Mandatory — the bootstrap alone produces flat, disconnected generations. +5. **Verify** — see `references/5-verify.md`. Describe a request the user can trigger and how to confirm the tree — grouped trace, session, attribution — lands in PostHog, not just a lone `$ai_generation`. ## Reference files @@ -30,7 +31,8 @@ The linked install page carries the exact code blocks for this variant's languag - **Environment variables.** Read `` and `` from env, using the framework's env-var convention. Never hardcode either value. - **Minimal changes.** OTel initialization is a single call that runs once at process start. Place it alongside any existing PostHog init rather than restructuring the entry point. - **Match the docs.** Package names, instrumentor imports, and processor names change between AIO releases. The install page for this variant is the source of truth. -- **Don't touch what isn't yours.** This skill instruments generations only. Identify calls, event tracking, error tracking, and dashboards belong to the base `integration` skill — do not add or edit them here. +- **The tree is the deliverable.** A run that only produces flat `$ai_generation` events is incomplete. Nesting (`4-nesting.md`) is part of setup, not a follow-up. +- **Don't touch what isn't yours.** This skill instruments LLM observability only — generations, traces, sessions, spans. Identify calls, event tracking, error tracking, and dashboards belong to the base `integration` skill — do not add or edit them here. ## Emit a run record diff --git a/context/skills/ai-observability/references/1-begin.md b/context/skills/ai-observability/references/1-begin.md index 21c23932..2cea513e 100644 --- a/context/skills/ai-observability/references/1-begin.md +++ b/context/skills/ai-observability/references/1-begin.md @@ -1,7 +1,7 @@ --- next_step: 2-install.md title: AI Observability Setup - Begin -description: Pick the right variant, confirm prerequisites, and locate the LLM call sites before editing +description: Pick the right variant, locate the LLM call sites, and map the app's session/trace/span structure before editing --- Before touching any code, decide which variant of this skill to install, confirm the two prerequisites, and get a read on where in the project LLM calls actually happen. AI Observability instruments an existing setup — if the setup isn't there, this skill can't do its job. @@ -49,4 +49,13 @@ Grep for where the vendor SDK is imported and called. This is not a full analysi Note the app's entry point (server startup file, `main.py`, `index.ts`, `instrumentation.ts` in Next.js, etc.) — OTel must be initialized *before* the vendor SDK is imported, and the entry point is where that happens. -Do not edit yet. Once you have a note of the entry point and the call sites, move on to `2-install.md`. +## Map the logical structure + +Instrumentation captures individual LLM calls; the *tree* that makes them useful — session → trace → span → generation — comes from application semantics only the code can tell you. Answer these four questions now, from the code, and write the answers down; `4-nesting.md` consumes them: + +- **What is one session here?** The unit of conversation — a conversation object, a thread id, a workflow run. Some apps have none; that's a valid answer. +- **What is one trace?** The unit of request — typically a request handler or the top-level function that may make several model calls to produce one result. +- **Which non-LLM steps deserve spans?** Retrieval, tool calls, validation — steps between model calls worth seeing in the trace. +- **What is the distinct-id source?** A `user_id` in scope at the call sites? None → events will be anonymous; note that too. + +Do not edit yet. Once you have the entry point, the call sites, and the structure map, move on to `2-install.md`. diff --git a/context/skills/ai-observability/references/3-otel-setup.md b/context/skills/ai-observability/references/3-otel-setup.md index 3c6f91be..e1c86f05 100644 --- a/context/skills/ai-observability/references/3-otel-setup.md +++ b/context/skills/ai-observability/references/3-otel-setup.md @@ -1,10 +1,12 @@ --- -next_step: 4-verify.md +next_step: 4-nesting.md title: AI Observability Setup - OpenTelemetry description: Initialize the OTel TracerProvider with PostHog's span processor and attach the provider instrumentor --- -Initialize OpenTelemetry once, at the app's entry point, so it is running before any LLM call executes. This is the single place PostHog reads the project token and host from — everything after this is normal LLM code that gets auto-traced. +Initialize OpenTelemetry once, at the app's entry point, so it is running before any LLM call executes. This is the single place PostHog reads the project token and host from. + +This bootstrap captures individual generations only. It does **not** build the session → trace → span tree — that is application-level work done in `4-nesting.md`, and it is a mandatory part of this skill, not optional polish. ## Environment variables diff --git a/context/skills/ai-observability/references/4-nesting.md b/context/skills/ai-observability/references/4-nesting.md new file mode 100644 index 00000000..8df3a439 --- /dev/null +++ b/context/skills/ai-observability/references/4-nesting.md @@ -0,0 +1,96 @@ +--- +next_step: 5-verify.md +title: AI Observability Setup - Nesting +description: Build the session → trace → span → generation tree on top of the bootstrap — this step is mandatory, not optional polish +--- + +The bootstrap from `3-otel-setup.md` captures each LLM call in isolation. It has no concept of what constitutes a *request* (a trace), a *conversation* (a session), or a *non-LLM step* (a span) — those are application semantics only the code can tell you. Without this step, a request that makes two model calls with a lookup in between lands as two disconnected single-generation traces, no session, no user attribution, and the lookup invisible. The product is built around the tree; flat generations degrade it to a per-call cost log. + +Use the structure map you wrote down in `1-begin.md`. This step turns it into code. + +## The nesting model + +One vocabulary, every variant: + +| Concept | Property | Notes | +|---|---|---| +| Session | `$ai_session_id` | optional; groups traces (a conversation, workflow, thread) | +| Trace | `$ai_trace_id` | **required**; groups one request's events | +| Span | `$ai_span` event + `$ai_span_id` / `$ai_span_name` | a non-LLM step (retrieval, tool call, validation) | +| Generation | `$ai_generation` event | one LLM call | +| Tree edge | `$ai_parent_id` | parent is a `trace_id` or another `span_id` | + +## Three ways to express it — branch by variant + +The model is constant; only the API changes. Pick the branch matching the mechanism you installed. + +### OTel auto-instrumentation (most variants) + +Wrap each logical operation in an **enclosing span**. Auto-instrumented generations nest under it automatically because they share the OTel trace context — spans in one OTel trace share `$ai_trace_id`, and a child's `parent_span_id` becomes `$ai_parent_id`. Nothing else is needed for trace grouping. + +```python +def ask(self, question: str) -> str: + with tracer.start_as_current_span( + "ai.support_request", # ai.* prefix — required, see below + attributes={ + "$ai_session_id": self.thread_id, # groups this thread's traces + "posthog.distinct_id": self.user_id, # attributes to the person + }, + ): + category = self._classify(question) # generation, nests automatically + order = self._lookup_order(category) # span, see below + return self._answer(question, category, order) + +def _lookup_order(self, category: str) -> dict | None: + if category != "order_status": + return None + with tracer.start_as_current_span("ai.lookup_order"): # non-LLM step + return lookup_order(self.user_id) +``` + +**The `ai.*` naming rule — silent-failure trap.** PostHog keeps a span only if its name *or an attribute key* begins with `gen_ai.`, `llm.`, `ai.`, or `traceloop.`. Everything else is dropped without error. An enclosing span named `handle_request` or a `lookup_order` span **vanishes** — taking its session and distinct-id attributes with it. Every manual span on the OTel path must be named `ai.*` (e.g. `ai.support_request`, `ai.lookup_order`). This is the single most likely way a well-intentioned run still produces a broken tree. + +**Session and distinct-id propagation.** OTel does not inherit attributes parent→child, and PostHog's processor adds no baggage propagation. A **Resource** attribute is process-global — fine only when one process equals one session/user. A **runtime-varying** session or user must be set as an attribute on every span you want grouped (in practice: the enclosing span of each request, as above). If per-request attribution matters and the variant has a wrapper SDK, prefer the wrapper — its per-call params sidestep this. + +### Wrapper SDK (`openai`, `anthropic`, `gemini`, `langchain`, `vercel`, `openai-agents` wrapper clients) + +Per-call parameters, no manual spans needed. Python kwargs: `posthog_trace_id`, `posthog_distinct_id`, `posthog_properties` (put `$ai_session_id` here), `posthog_groups`. Node equivalents are camelCase (`posthogTraceId`, …). + +**Trap:** `posthog_trace_id` auto-generates a *fresh UUID per call* when omitted. To group multiple calls into one trace you must pass a shared id explicitly: + +```python +from uuid import uuid4 + +def ask(self, question: str) -> str: + trace_id = str(uuid4()) # one per request; reuse for every call in it + ph = dict( + posthog_trace_id=trace_id, + posthog_distinct_id=self.user_id, + posthog_properties={"$ai_session_id": self.thread_id}, + ) + category = self._classify(question, ph) # client.messages.create(..., **ph) + order = lookup_order(self.user_id) if category == "order_status" else None + return self._answer(question, category, order, ph) +``` + +### `manual-capture` + +Emit `$ai_trace` / `$ai_span` / `$ai_generation` events explicitly and wire the tree yourself: every event in a request shares `$ai_trace_id`; child events set `$ai_parent_id` to the parent's `trace_id` or `span_id`; `$ai_session_id` goes in properties. The manual-capture install page carries the full property tables. + +## What to actually edit + +For each request-shaped code path you mapped in `1-begin.md`: + +1. Group its LLM calls into one trace (enclosing span, shared `posthog_trace_id`, or shared `$ai_trace_id` — per the branch above). +2. Set `$ai_session_id` from the conversation identifier, if the app has one. +3. Set the distinct id from the user identifier, if the app has one. +4. Wrap non-LLM steps worth seeing (retrieval, tool calls, validation) as spans — `ai.*`-named on the OTel path. + +Keep it minimal: instrument the representative paths you mapped, don't refactor the app. If the app genuinely has no conversation or user concept, say so in the report — omitting a session is a finding, not a failure. + +## Do not + +- Do not name a manual OTel span anything that doesn't start with `ai.` — it will be silently dropped. +- Do not rely on omitted `posthog_trace_id` to group wrapper-SDK calls — each call gets its own fresh UUID. +- Do not put a runtime-varying `$ai_session_id` or distinct id only on the Resource — the Resource is process-global. +- Do not skip this step because generations already appear in PostHog. Flat generations are the failure mode this step exists to prevent. diff --git a/context/skills/ai-observability/references/4-verify.md b/context/skills/ai-observability/references/4-verify.md deleted file mode 100644 index bb5925f7..00000000 --- a/context/skills/ai-observability/references/4-verify.md +++ /dev/null @@ -1,37 +0,0 @@ ---- -next_step: null -title: AI Observability Setup - Verify -description: Give the user a concrete way to trigger a generation and confirm it lands in PostHog ---- - -You've installed packages and wired OTel init. The last thing this skill produces is a **verification path** the user can run themselves. Don't call the LLM from here — you don't have credentials, and the user should be the one to see the trace show up in their project. - -## What to tell the user - -Point them at the smallest existing code path in their project that hits the vendor SDK. Pick one from what you noted in `1-begin.md`: - -- If the project has a script (`scripts/`, `bin/`, a `package.json` script) that calls the LLM, name it: "Run `npm run