feat: ai-observability test apps + evaluator rubric

apps/ai-observability/README.md

@@ -0,0 +1,66 @@
+# AI Observability test apps
+
+PostHog-less apps for testing `wizard ai-observability`. Each app calls an LLM
+in a shape with real structure — a conversation, a tool call, a multi-step
+answer — so the wizard has to produce the
+[session → trace → span → generation](https://posthog.com/docs/ai-observability/basics)
+tree, not just wrap a single call.
+
+Layout: `<docs-installation-slug>/<runtime>-<app-name>`. The top-level
+directory matches the app's page under
+[installation docs](https://posthog.com/docs/ai-observability/installation),
+so `ls` answers "which docs pages have coverage".
+
+## The apps
+
+Each app exists to test one thing the others don't:
+
+- `anthropic/python-weather` — the baseline: tool span, single-trace inferred session
+- `anthropic/node-weather` — multi-turn: one trace per turn, one session across turns
+- `openai/python-weather` — same flow, OpenAI shape; canary for stale SDK resolves
+- `openai/node-weather` — same flow, single-turn Node
+- `openai/python-docs-rag` — embeddings; hardest session (no field to read)
+- `groq/node-chat` — provider named by `baseURL`, not package
+- `openai-agents/python-travel-triage` — tracing processor; the SDK emits the tree
+- `vercel-ai/nextjs-support-chat` — per-request identity; framework bootstrap
+- `manual-capture/node-http-chat` — hand-built tree; must reuse the existing client
+
+The four weather apps implement the identical `get_weather` round trip from
+[Anthropic's tool-use docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview#how-tool-use-works),
+so a diff between any two isolates one variable: the SDK, the language, or the
+conversation structure.
+
+## Conventions
+
+- **Wrapper-first.** The expected mechanism is the PostHog SDK wrapper
+  (`posthog.ai.*` / `@posthog/ai`) with per-call `posthog_distinct_id`,
+  `posthog_trace_id`, and `posthog_properties`, as shown in the docs. OTel is
+  acceptable only where the same structure lands. Exceptions:
+  `openai-agents` (tracing processor), `vercel-ai` (`experimental_telemetry`),
+  `manual-capture` (no SDK to wrap).
+- **Every app gets a session** — single-trace apps included. The graded
+  property is **cardinality**: one id shared by the traces that belong
+  together. A fresh id per call groups nothing and is worse than none.
+- **Spans come from tool registration** (or explicit `$ai_span` capture on the
+  manual path). Never from hand-authored wrappers around helper functions.
+- Static fixtures in the style of [`../mcp-analytics`](../mcp-analytics): not
+  executed, no lockfiles, no keys. Node type-checks with `tsc --noEmit`.
+- Each README states the tree the app must produce and grades **emitted
+  events, not the diff** — every observed failure mode so far produced
+  plausible code and a broken tree.
+- `manual-capture/node-http-chat` alone ships PostHog product analytics, to
+  test the additive-only rule (reuse the client; never a second instance).
+
+## Running
+
+```bash
+pnpm wizard-ci --app ai-observability/<app> --evaluate --local
+```
+
+The command id (and so the `ai-observability` evaluator rubric) is inferred
+from the app path. After a run, confirm which skill version was installed
+before trusting results:
+
+```bash
+ls apps/ai-observability/<app>/.claude/skills/*/references/
+```

apps/ai-observability/anthropic/node-weather/README.md

@@ -0,0 +1,26 @@
+# wb-aio-anthropic-node-weather
+
+Weather assistant, Anthropic Node SDK, two-turn conversation. PostHog-less
+fixture for `wizard ai-observability`.
+
+```
+thread_abc                              ← $ai_session_id (Conversation.threadId)
+├─ ask("weather in San Francisco?")     ← trace
+│  ├─ messages.create (→ tool_use)      ← generation
+│  ├─ get_weather                       ← span
+│  └─ messages.create (→ answer)        ← generation
+└─ ask("How about Boston?")             ← trace (same shape)
+```
+
+## Expected outcome
+
+- one session (`thread_abc`), two traces of
+  `generation → span(get_weather) → generation`, all on `user_123`
+- `@posthog/ai` Anthropic wrapper: per-call `posthogDistinctId` /
+  `posthogProperties`, and a **shared `posthogTraceId` per `ask()`** — omitted,
+  each call mints its own and the turn splits
+- `$ai_session_id` spelled exactly; flushed before exit (CLI)
+- `weather.ts` and the tool loop untouched
+
+Fail: a hand-authored span around `getWeather`; a session id per turn; a trace
+per model call; no flush.

apps/ai-observability/anthropic/node-weather/package.json

@@ -0,0 +1,18 @@
+{
+  "name": "wb-aio-anthropic-node-weather",
+  "version": "0.0.0",
+  "private": true,
+  "type": "module",
+  "description": "PostHog-less weather assistant (Anthropic Node SDK, tool use) for testing `wizard ai-observability`.",
+  "scripts": {
+    "build": "tsc --noEmit",
+    "start": "tsx src/index.ts"
+  },
+  "dependencies": {
+    "@anthropic-ai/sdk": "^0.32.0"
+  },
+  "devDependencies": {
+    "tsx": "^4.19.2",
+    "typescript": "^5.6.3"
+  }
+}

apps/ai-observability/anthropic/node-weather/src/index.ts

@@ -0,0 +1,97 @@
+import Anthropic from '@anthropic-ai/sdk'
+
+import { getWeather } from './weather.js'
+
+const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY ?? '' })
+
+const MODEL = 'claude-opus-5'
+
+const tools: Anthropic.Tool[] = [
+    {
+        name: 'get_weather',
+        description: 'Get the current weather for a given location.',
+        input_schema: {
+            type: 'object',
+            properties: {
+                location: { type: 'string', description: 'City and state, e.g. San Francisco, CA' },
+            },
+            required: ['location'],
+        },
+    },
+]
+
+const toolChoice = { type: 'auto', disable_parallel_tool_use: true } as const
+
+function textOf(message: Anthropic.Message): string {
+    return message.content
+        .filter((block): block is Anthropic.TextBlock => block.type === 'text')
+        .map((block) => block.text)
+        .join('')
+}
+
+/** One chat thread. Every question asked below belongs to this thread. */
+class Conversation {
+    private messages: Anthropic.MessageParam[] = []
+
+    constructor(
+        readonly userId: string,
+        readonly threadId: string
+    ) {}
+
+    /** Answer one question, running the tool if the model asks for it. */
+    async ask(question: string): Promise<string> {
+        this.messages.push({ role: 'user', content: question })
+
+        const response = await client.messages.create({
+            model: MODEL,
+            max_tokens: 1024,
+            tools,
+            tool_choice: toolChoice,
+            messages: this.messages,
+        })
+
+        const toolUse = response.content.find(
+            (block): block is Anthropic.ToolUseBlock => block.type === 'tool_use'
+        )
+
+        if (!toolUse) {
+            const answer = textOf(response)
+            this.messages.push({ role: 'assistant', content: answer })
+            return answer
+        }
+
+        const { location } = toolUse.input as { location: string }
+        const result = getWeather(location)
+
+        this.messages.push(
+            { role: 'assistant', content: response.content },
+            {
+                role: 'user',
+                content: [{ type: 'tool_result', tool_use_id: toolUse.id, content: result }],
+            }
+        )
+
+        const followup = await client.messages.create({
+            model: MODEL,
+            max_tokens: 1024,
+            tools,
+            tool_choice: toolChoice,
+            messages: this.messages,
+        })
+
+        const answer = textOf(followup)
+        this.messages.push({ role: 'assistant', content: answer })
+        return answer
+    }
+}
+
+async function main(): Promise<void> {
+    const thread = new Conversation('user_123', 'thread_abc')
+    console.log(await thread.ask("What's the weather in San Francisco?"))
+    console.log(await thread.ask('How about Boston?'))
+}
+
+main().catch((err) => {
+    console.error(`fatal: ${String(err)}`)
+    process.exit(1)
+})

apps/ai-observability/anthropic/node-weather/src/weather.ts

@@ -0,0 +1,10 @@
+/** Backing implementation for the `get_weather` tool. No model call involved. */
+
+const FORECAST: Record<string, string> = {
+    'San Francisco, CA': '15 degrees Celsius, partly cloudy',
+    'Boston, MA': '4 degrees Celsius, snow showers',
+}
+
+export function getWeather(location: string): string {
+    return FORECAST[location] ?? `No forecast on file for ${location}.`
+}

apps/ai-observability/anthropic/node-weather/tsconfig.json

@@ -0,0 +1,12 @@
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "module": "ES2022",
+    "moduleResolution": "bundler",
+    "strict": true,
+    "esModuleInterop": true,
+    "skipLibCheck": true,
+    "noEmit": true
+  },
+  "include": ["src"]
+}

apps/ai-observability/anthropic/python-weather/README.md

@@ -0,0 +1,24 @@
+# wb-aio-anthropic-python-weather
+
+Weather assistant, Anthropic Python SDK, single question. PostHog-less fixture
+for `wizard ai-observability`.
+
+```
+the run                                 ← $ai_session_id (no field — infer it)
+└─ ask("weather in San Francisco?")     ← trace
+   ├─ messages.create (→ tool_use)      ← generation
+   ├─ get_weather                       ← span
+   └─ messages.create (→ answer)        ← generation
+```
+
+## Expected outcome
+
+- one session, one trace: `generation → span(get_weather) → generation`, all on
+  `user_123`
+- `posthog.ai.anthropic` wrapper: per-call `posthog_distinct_id`, shared
+  `posthog_trace_id`, `posthog_properties={"$ai_session_id": ...}`
+- `$ai_session_id` spelled exactly; flushed before exit (CLI)
+- `weather.py` and the tool loop untouched
+
+Fail: a hand-authored span around `get_weather`; a session id per call; no
+flush.

apps/ai-observability/anthropic/python-weather/main.py

@@ -0,0 +1,84 @@
+"""Weather assistant on the Anthropic Python SDK. The model may call the
+registered `get_weather` tool before answering, so one question is either one
+model call or two with a tool execution between them."""
+
+import os
+
+import anthropic
+
+from weather import get_weather
+
+client = anthropic.Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY", ""))
+
+MODEL = "claude-opus-5"
+
+USER_ID = "user_123"
+
+TOOLS = [
+    {
+        "name": "get_weather",
+        "description": "Get the current weather for a given location.",
+        "input_schema": {
+            "type": "object",
+            "properties": {
+                "location": {
+                    "type": "string",
+                    "description": "City and state, e.g. San Francisco, CA",
+                }
+            },
+            "required": ["location"],
+        },
+    }
+]
+
+TOOL_CHOICE = {"type": "auto", "disable_parallel_tool_use": True}
+
+
+def _text(response: anthropic.types.Message) -> str:
+    return "".join(block.text for block in response.content if block.type == "text")
+
+
+def ask(question: str) -> str:
+    """Answer one question, running the tool if the model asks for it."""
+    messages: list[dict] = [{"role": "user", "content": question}]
+
+    response = client.messages.create(
+        model=MODEL,
+        max_tokens=1024,
+        tools=TOOLS,
+        tool_choice=TOOL_CHOICE,
+        messages=messages,
+    )
+
+    tool_use = next((b for b in response.content if b.type == "tool_use"), None)
+    if tool_use is None:
+        return _text(response)
+
+    result = get_weather(**tool_use.input)
+
+    messages += [
+        {"role": "assistant", "content": response.content},
+        {
+            "role": "user",
+            "content": [
+                {"type": "tool_result", "tool_use_id": tool_use.id, "content": result}
+            ],
+        },
+    ]
+
+    followup = client.messages.create(
+        model=MODEL,
+        max_tokens=1024,
+        tools=TOOLS,
+        tool_choice=TOOL_CHOICE,
+        messages=messages,
+    )
+    return _text(followup)
+
+
+def main() -> None:
+    print(ask("What's the weather in San Francisco?"))
+
+
+if __name__ == "__main__":
+    main()

apps/ai-observability/anthropic/python-weather/requirements.txt

@@ -0,0 +1,2 @@
+# PostHog-less weather assistant. Anthropic SDK only.
+anthropic>=0.40.0

apps/ai-observability/anthropic/python-weather/weather.py

@@ -0,0 +1,10 @@
+"""Backing implementation for the `get_weather` tool. No model call involved."""
+
+_FORECAST = {
+    "San Francisco, CA": "15 degrees Celsius, partly cloudy",
+    "Boston, MA": "4 degrees Celsius, snow showers",
+}
+
+
+def get_weather(location: str) -> str:
+    return _FORECAST.get(location, f"No forecast on file for {location}.")

apps/ai-observability/groq/node-chat/README.md

@@ -0,0 +1,26 @@
+# wb-aio-groq-node-chat
+
+Two-turn chat served by Groq through the `openai` Node SDK — the provider is
+named only by `baseURL`. PostHog-less fixture for `wizard ai-observability`.
+
+```
+thread_abc                        ← $ai_session_id (Thread.threadId)
+├─ ask("What is a feature flag?") ← trace
+│  └─ chat.completions.create     ← generation
+└─ ask("How is that different?")  ← trace
+   ├─ condense                    ← generation
+   └─ reply                       ← generation
+```
+
+## Expected outcome
+
+- one session (`thread_abc`), two traces — one generation, then two — all on
+  `user_123`
+- **provider recorded as Groq, not OpenAI**; `baseURL` intact
+- `@posthog/ai` OpenAI wrapper (it reads `baseURL` for the provider): per-call
+  identity, shared `posthogTraceId` within turn two
+- `$ai_session_id` spelled exactly; flushed before exit (CLI)
+- `moderate()` untouched — a plain guard, not a tool
+
+Fail: provider attributed to OpenAI; a span around `moderate()`; turn two split
+into two traces; no flush.