diff --git a/apps/ai-observability/README.md b/apps/ai-observability/README.md new file mode 100644 index 000000000..d2da41c8d --- /dev/null +++ b/apps/ai-observability/README.md @@ -0,0 +1,66 @@ +# AI Observability test apps + +PostHog-less apps for testing `wizard ai-observability`. Each app calls an LLM +in a shape with real structure — a conversation, a tool call, a multi-step +answer — so the wizard has to produce the +[session → trace → span → generation](https://posthog.com/docs/ai-observability/basics) +tree, not just wrap a single call. + +Layout: `/-`. The top-level +directory matches the app's page under +[installation docs](https://posthog.com/docs/ai-observability/installation), +so `ls` answers "which docs pages have coverage". + +## The apps + +Each app exists to test one thing the others don't: + +- `anthropic/python-weather` — the baseline: tool span, single-trace inferred session +- `anthropic/node-weather` — multi-turn: one trace per turn, one session across turns +- `openai/python-weather` — same flow, OpenAI shape; canary for stale SDK resolves +- `openai/node-weather` — same flow, single-turn Node +- `openai/python-docs-rag` — embeddings; hardest session (no field to read) +- `groq/node-chat` — provider named by `baseURL`, not package +- `openai-agents/python-travel-triage` — tracing processor; the SDK emits the tree +- `vercel-ai/nextjs-support-chat` — per-request identity; framework bootstrap +- `manual-capture/node-http-chat` — hand-built tree; must reuse the existing client + +The four weather apps implement the identical `get_weather` round trip from +[Anthropic's tool-use docs](https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview#how-tool-use-works), +so a diff between any two isolates one variable: the SDK, the language, or the +conversation structure. + +## Conventions + +- **Wrapper-first.** The expected mechanism is the PostHog SDK wrapper + (`posthog.ai.*` / `@posthog/ai`) with per-call `posthog_distinct_id`, + `posthog_trace_id`, and `posthog_properties`, as shown in the docs. OTel is + acceptable only where the same structure lands. Exceptions: + `openai-agents` (tracing processor), `vercel-ai` (`experimental_telemetry`), + `manual-capture` (no SDK to wrap). +- **Every app gets a session** — single-trace apps included. The graded + property is **cardinality**: one id shared by the traces that belong + together. A fresh id per call groups nothing and is worse than none. +- **Spans come from tool registration** (or explicit `$ai_span` capture on the + manual path). Never from hand-authored wrappers around helper functions. +- Static fixtures in the style of [`../mcp-analytics`](../mcp-analytics): not + executed, no lockfiles, no keys. Node type-checks with `tsc --noEmit`. +- Each README states the tree the app must produce and grades **emitted + events, not the diff** — every observed failure mode so far produced + plausible code and a broken tree. +- `manual-capture/node-http-chat` alone ships PostHog product analytics, to + test the additive-only rule (reuse the client; never a second instance). + +## Running + +```bash +pnpm wizard-ci --app ai-observability/ --evaluate --local +``` + +The command id (and so the `ai-observability` evaluator rubric) is inferred +from the app path. After a run, confirm which skill version was installed +before trusting results: + +```bash +ls apps/ai-observability//.claude/skills/*/references/ +``` diff --git a/apps/ai-observability/anthropic/node-weather/README.md b/apps/ai-observability/anthropic/node-weather/README.md new file mode 100644 index 000000000..e4a943b8c --- /dev/null +++ b/apps/ai-observability/anthropic/node-weather/README.md @@ -0,0 +1,26 @@ +# wb-aio-anthropic-node-weather + +Weather assistant, Anthropic Node SDK, two-turn conversation. PostHog-less +fixture for `wizard ai-observability`. + +``` +thread_abc ← $ai_session_id (Conversation.threadId) +├─ ask("weather in San Francisco?") ← trace +│ ├─ messages.create (→ tool_use) ← generation +│ ├─ get_weather ← span +│ └─ messages.create (→ answer) ← generation +└─ ask("How about Boston?") ← trace (same shape) +``` + +## Expected outcome + +- one session (`thread_abc`), two traces of + `generation → span(get_weather) → generation`, all on `user_123` +- `@posthog/ai` Anthropic wrapper: per-call `posthogDistinctId` / + `posthogProperties`, and a **shared `posthogTraceId` per `ask()`** — omitted, + each call mints its own and the turn splits +- `$ai_session_id` spelled exactly; flushed before exit (CLI) +- `weather.ts` and the tool loop untouched + +Fail: a hand-authored span around `getWeather`; a session id per turn; a trace +per model call; no flush. diff --git a/apps/ai-observability/anthropic/node-weather/package.json b/apps/ai-observability/anthropic/node-weather/package.json new file mode 100644 index 000000000..307559735 --- /dev/null +++ b/apps/ai-observability/anthropic/node-weather/package.json @@ -0,0 +1,18 @@ +{ + "name": "wb-aio-anthropic-node-weather", + "version": "0.0.0", + "private": true, + "type": "module", + "description": "PostHog-less weather assistant (Anthropic Node SDK, tool use) for testing `wizard ai-observability`.", + "scripts": { + "build": "tsc --noEmit", + "start": "tsx src/index.ts" + }, + "dependencies": { + "@anthropic-ai/sdk": "^0.32.0" + }, + "devDependencies": { + "tsx": "^4.19.2", + "typescript": "^5.6.3" + } +} diff --git a/apps/ai-observability/anthropic/node-weather/src/index.ts b/apps/ai-observability/anthropic/node-weather/src/index.ts new file mode 100644 index 000000000..3a97f0e39 --- /dev/null +++ b/apps/ai-observability/anthropic/node-weather/src/index.ts @@ -0,0 +1,97 @@ +import Anthropic from '@anthropic-ai/sdk' + +import { getWeather } from './weather.js' + +const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY ?? '' }) + +const MODEL = 'claude-opus-5' + +const tools: Anthropic.Tool[] = [ + { + name: 'get_weather', + description: 'Get the current weather for a given location.', + input_schema: { + type: 'object', + properties: { + location: { type: 'string', description: 'City and state, e.g. San Francisco, CA' }, + }, + required: ['location'], + }, + }, +] + +const toolChoice = { type: 'auto', disable_parallel_tool_use: true } as const + +function textOf(message: Anthropic.Message): string { + return message.content + .filter((block): block is Anthropic.TextBlock => block.type === 'text') + .map((block) => block.text) + .join('') +} + +/** One chat thread. Every question asked below belongs to this thread. */ +class Conversation { + private messages: Anthropic.MessageParam[] = [] + + constructor( + readonly userId: string, + readonly threadId: string + ) {} + + /** Answer one question, running the tool if the model asks for it. */ + async ask(question: string): Promise { + this.messages.push({ role: 'user', content: question }) + + const response = await client.messages.create({ + model: MODEL, + max_tokens: 1024, + tools, + tool_choice: toolChoice, + messages: this.messages, + }) + + const toolUse = response.content.find( + (block): block is Anthropic.ToolUseBlock => block.type === 'tool_use' + ) + + if (!toolUse) { + const answer = textOf(response) + this.messages.push({ role: 'assistant', content: answer }) + return answer + } + + const { location } = toolUse.input as { location: string } + const result = getWeather(location) + + this.messages.push( + { role: 'assistant', content: response.content }, + { + role: 'user', + content: [{ type: 'tool_result', tool_use_id: toolUse.id, content: result }], + } + ) + + const followup = await client.messages.create({ + model: MODEL, + max_tokens: 1024, + tools, + tool_choice: toolChoice, + messages: this.messages, + }) + + const answer = textOf(followup) + this.messages.push({ role: 'assistant', content: answer }) + return answer + } +} + +async function main(): Promise { + const thread = new Conversation('user_123', 'thread_abc') + console.log(await thread.ask("What's the weather in San Francisco?")) + console.log(await thread.ask('How about Boston?')) +} + +main().catch((err) => { + console.error(`fatal: ${String(err)}`) + process.exit(1) +}) diff --git a/apps/ai-observability/anthropic/node-weather/src/weather.ts b/apps/ai-observability/anthropic/node-weather/src/weather.ts new file mode 100644 index 000000000..a9b52f099 --- /dev/null +++ b/apps/ai-observability/anthropic/node-weather/src/weather.ts @@ -0,0 +1,10 @@ +/** Backing implementation for the `get_weather` tool. No model call involved. */ + +const FORECAST: Record = { + 'San Francisco, CA': '15 degrees Celsius, partly cloudy', + 'Boston, MA': '4 degrees Celsius, snow showers', +} + +export function getWeather(location: string): string { + return FORECAST[location] ?? `No forecast on file for ${location}.` +} diff --git a/apps/ai-observability/anthropic/node-weather/tsconfig.json b/apps/ai-observability/anthropic/node-weather/tsconfig.json new file mode 100644 index 000000000..235d77827 --- /dev/null +++ b/apps/ai-observability/anthropic/node-weather/tsconfig.json @@ -0,0 +1,12 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "ES2022", + "moduleResolution": "bundler", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "noEmit": true + }, + "include": ["src"] +} diff --git a/apps/ai-observability/anthropic/python-weather/README.md b/apps/ai-observability/anthropic/python-weather/README.md new file mode 100644 index 000000000..682616697 --- /dev/null +++ b/apps/ai-observability/anthropic/python-weather/README.md @@ -0,0 +1,24 @@ +# wb-aio-anthropic-python-weather + +Weather assistant, Anthropic Python SDK, single question. PostHog-less fixture +for `wizard ai-observability`. + +``` +the run ← $ai_session_id (no field — infer it) +└─ ask("weather in San Francisco?") ← trace + ├─ messages.create (→ tool_use) ← generation + ├─ get_weather ← span + └─ messages.create (→ answer) ← generation +``` + +## Expected outcome + +- one session, one trace: `generation → span(get_weather) → generation`, all on + `user_123` +- `posthog.ai.anthropic` wrapper: per-call `posthog_distinct_id`, shared + `posthog_trace_id`, `posthog_properties={"$ai_session_id": ...}` +- `$ai_session_id` spelled exactly; flushed before exit (CLI) +- `weather.py` and the tool loop untouched + +Fail: a hand-authored span around `get_weather`; a session id per call; no +flush. diff --git a/apps/ai-observability/anthropic/python-weather/main.py b/apps/ai-observability/anthropic/python-weather/main.py new file mode 100644 index 000000000..0b28f0eee --- /dev/null +++ b/apps/ai-observability/anthropic/python-weather/main.py @@ -0,0 +1,84 @@ +"""Weather assistant on the Anthropic Python SDK. The model may call the +registered `get_weather` tool before answering, so one question is either one +model call or two with a tool execution between them.""" + +import os + +import anthropic + +from weather import get_weather + +client = anthropic.Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY", "")) + +MODEL = "claude-opus-5" + +USER_ID = "user_123" + +TOOLS = [ + { + "name": "get_weather", + "description": "Get the current weather for a given location.", + "input_schema": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City and state, e.g. San Francisco, CA", + } + }, + "required": ["location"], + }, + } +] + +TOOL_CHOICE = {"type": "auto", "disable_parallel_tool_use": True} + + +def _text(response: anthropic.types.Message) -> str: + return "".join(block.text for block in response.content if block.type == "text") + + +def ask(question: str) -> str: + """Answer one question, running the tool if the model asks for it.""" + messages: list[dict] = [{"role": "user", "content": question}] + + response = client.messages.create( + model=MODEL, + max_tokens=1024, + tools=TOOLS, + tool_choice=TOOL_CHOICE, + messages=messages, + ) + + tool_use = next((b for b in response.content if b.type == "tool_use"), None) + if tool_use is None: + return _text(response) + + result = get_weather(**tool_use.input) + + messages += [ + {"role": "assistant", "content": response.content}, + { + "role": "user", + "content": [ + {"type": "tool_result", "tool_use_id": tool_use.id, "content": result} + ], + }, + ] + + followup = client.messages.create( + model=MODEL, + max_tokens=1024, + tools=TOOLS, + tool_choice=TOOL_CHOICE, + messages=messages, + ) + return _text(followup) + + +def main() -> None: + print(ask("What's the weather in San Francisco?")) + + +if __name__ == "__main__": + main() diff --git a/apps/ai-observability/anthropic/python-weather/requirements.txt b/apps/ai-observability/anthropic/python-weather/requirements.txt new file mode 100644 index 000000000..1dc907409 --- /dev/null +++ b/apps/ai-observability/anthropic/python-weather/requirements.txt @@ -0,0 +1,2 @@ +# PostHog-less weather assistant. Anthropic SDK only. +anthropic>=0.40.0 diff --git a/apps/ai-observability/anthropic/python-weather/weather.py b/apps/ai-observability/anthropic/python-weather/weather.py new file mode 100644 index 000000000..d96cb94e2 --- /dev/null +++ b/apps/ai-observability/anthropic/python-weather/weather.py @@ -0,0 +1,10 @@ +"""Backing implementation for the `get_weather` tool. No model call involved.""" + +_FORECAST = { + "San Francisco, CA": "15 degrees Celsius, partly cloudy", + "Boston, MA": "4 degrees Celsius, snow showers", +} + + +def get_weather(location: str) -> str: + return _FORECAST.get(location, f"No forecast on file for {location}.") diff --git a/apps/ai-observability/groq/node-chat/README.md b/apps/ai-observability/groq/node-chat/README.md new file mode 100644 index 000000000..977f96d37 --- /dev/null +++ b/apps/ai-observability/groq/node-chat/README.md @@ -0,0 +1,26 @@ +# wb-aio-groq-node-chat + +Two-turn chat served by Groq through the `openai` Node SDK — the provider is +named only by `baseURL`. PostHog-less fixture for `wizard ai-observability`. + +``` +thread_abc ← $ai_session_id (Thread.threadId) +├─ ask("What is a feature flag?") ← trace +│ └─ chat.completions.create ← generation +└─ ask("How is that different?") ← trace + ├─ condense ← generation + └─ reply ← generation +``` + +## Expected outcome + +- one session (`thread_abc`), two traces — one generation, then two — all on + `user_123` +- **provider recorded as Groq, not OpenAI**; `baseURL` intact +- `@posthog/ai` OpenAI wrapper (it reads `baseURL` for the provider): per-call + identity, shared `posthogTraceId` within turn two +- `$ai_session_id` spelled exactly; flushed before exit (CLI) +- `moderate()` untouched — a plain guard, not a tool + +Fail: provider attributed to OpenAI; a span around `moderate()`; turn two split +into two traces; no flush. diff --git a/apps/ai-observability/groq/node-chat/package.json b/apps/ai-observability/groq/node-chat/package.json new file mode 100644 index 000000000..c90c4d3fa --- /dev/null +++ b/apps/ai-observability/groq/node-chat/package.json @@ -0,0 +1,18 @@ +{ + "name": "wb-aio-groq-chat", + "version": "0.0.0", + "private": true, + "type": "module", + "description": "PostHog-less Groq chat assistant (openai SDK pointed at Groq) for testing `wizard ai-observability`.", + "scripts": { + "build": "tsc --noEmit", + "start": "tsx src/index.ts" + }, + "dependencies": { + "openai": "^4.77.0" + }, + "devDependencies": { + "tsx": "^4.19.2", + "typescript": "^5.6.3" + } +} diff --git a/apps/ai-observability/groq/node-chat/src/index.ts b/apps/ai-observability/groq/node-chat/src/index.ts new file mode 100644 index 000000000..dffb21650 --- /dev/null +++ b/apps/ai-observability/groq/node-chat/src/index.ts @@ -0,0 +1,71 @@ +import OpenAI from 'openai' + +const client = new OpenAI({ + apiKey: process.env.GROQ_API_KEY ?? '', + baseURL: 'https://api.groq.com/openai/v1', +}) + +const MODEL = 'llama-3.3-70b-versatile' + +const BLOCKED = ['password', 'ssn', 'credit card'] + +function moderate(question: string): void { + const lowered = question.toLowerCase() + if (BLOCKED.some((term) => lowered.includes(term))) { + throw new Error('question rejected by moderation') + } +} + +async function complete(system: string, user: string): Promise { + const response = await client.chat.completions.create({ + model: MODEL, + messages: [ + { role: 'system', content: system }, + { role: 'user', content: user }, + ], + }) + return response.choices[0]?.message?.content ?? '' +} + +type Turn = { question: string; answer: string } + +/** One chat thread. Every question asked below belongs to this thread. */ +class Thread { + private turns: Turn[] = [] + + constructor( + readonly userId: string, + readonly threadId: string + ) {} + + /** Answer one question, end to end. */ + async ask(question: string): Promise { + moderate(question) + const context = this.turns.length > 0 ? await this.condense() : '' + const answer = await this.reply(context, question) + this.turns.push({ question, answer }) + return answer + } + + /** Squash the thread so far into a short recap the next call can use. */ + private condense(): Promise { + const transcript = this.turns.map((t) => `Q: ${t.question}\nA: ${t.answer}`).join('\n\n') + return complete('Summarize this conversation in two sentences.', transcript) + } + + private reply(context: string, question: string): Promise { + const prompt = context ? `Earlier in this thread: ${context}\n\nQuestion: ${question}` : question + return complete('You are a concise assistant.', prompt) + } +} + +async function main(): Promise { + const thread = new Thread('user_123', 'thread_abc') + console.log(await thread.ask('What is a feature flag?')) + console.log(await thread.ask('How is that different from an experiment?')) +} + +main().catch((err) => { + console.error(`fatal: ${String(err)}`) + process.exit(1) +}) diff --git a/apps/ai-observability/groq/node-chat/tsconfig.json b/apps/ai-observability/groq/node-chat/tsconfig.json new file mode 100644 index 000000000..676cd16ef --- /dev/null +++ b/apps/ai-observability/groq/node-chat/tsconfig.json @@ -0,0 +1,13 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "ES2022", + "moduleResolution": "bundler", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "noEmit": true, + "types": ["node"] + }, + "include": ["src"] +} diff --git a/apps/ai-observability/manual-capture/node-http-chat/README.md b/apps/ai-observability/manual-capture/node-http-chat/README.md new file mode 100644 index 000000000..4f1e3d5de --- /dev/null +++ b/apps/ai-observability/manual-capture/node-http-chat/README.md @@ -0,0 +1,31 @@ +# wb-aio-manual-http-chat + +Two-turn chat over plain `fetch` — no vendor LLM SDK, so manual capture is the +only path. **PostHog product analytics is already wired** (`src/posthog.ts`). +Fixture for `wizard ai-observability`. + +``` +thread_abc ← $ai_session_id (Thread.threadId) +├─ ask("Where is my order?") ← trace +│ ├─ complete (tool_calls) ← $ai_generation +│ ├─ lookup_order ← $ai_span +│ └─ complete (final) ← $ai_generation +└─ ask("Can I get a refund?") ← trace + └─ complete ← $ai_generation +``` + +## Expected outcome + +- one session (`thread_abc`), two traces — `generation → span(lookup_order) → + generation`, then a single generation — all on `user_123`, children linked + via `$ai_parent_id` +- **the existing client from `src/posthog.ts` reused** — never a second + instance; existing `identify()` / `capture()` calls untouched +- each `complete()` captured as `$ai_generation` with provider, model, input, + output, tokens (already parsed from the response body), latency +- the tool dispatch captured as `$ai_span` +- one `$ai_trace_id` per `ask()`; `$ai_session_id` spelled exactly, on every + event; the existing shutdown preserved + +Fail: a second PostHog client; a trace id per model call; missing +`$ai_parent_id`; edits to the product-analytics calls. diff --git a/apps/ai-observability/manual-capture/node-http-chat/package.json b/apps/ai-observability/manual-capture/node-http-chat/package.json new file mode 100644 index 000000000..fbe112c18 --- /dev/null +++ b/apps/ai-observability/manual-capture/node-http-chat/package.json @@ -0,0 +1,19 @@ +{ + "name": "wb-aio-manual-http-chat", + "version": "0.0.0", + "private": true, + "type": "module", + "description": "Chat assistant reaching an OpenAI-compatible endpoint over plain fetch. Has PostHog product analytics but no AI observability. Fixture for `wizard ai-observability`.", + "scripts": { + "build": "tsc --noEmit", + "start": "tsx src/index.ts" + }, + "dependencies": { + "posthog-node": "^4.2.0" + }, + "devDependencies": { + "@types/node": "^22.10.0", + "tsx": "^4.19.2", + "typescript": "^5.6.3" + } +} diff --git a/apps/ai-observability/manual-capture/node-http-chat/src/index.ts b/apps/ai-observability/manual-capture/node-http-chat/src/index.ts new file mode 100644 index 000000000..d6dd7c65b --- /dev/null +++ b/apps/ai-observability/manual-capture/node-http-chat/src/index.ts @@ -0,0 +1,126 @@ +import { lookupOrder } from './orders.js' +import { posthog } from './posthog.js' + +const LLM_URL = process.env.LLM_URL ?? 'http://localhost:11434/v1/chat/completions' +const MODEL = 'llama3.2' + +type Message = { + role: 'system' | 'user' | 'assistant' | 'tool' + content: string + tool_calls?: ToolCall[] + tool_call_id?: string +} + +type ToolCall = { id: string; function: { name: string; arguments: string } } + +type Completion = { + text: string + toolCalls: ToolCall[] + promptTokens: number + completionTokens: number +} + +const TOOLS = [ + { + type: 'function', + function: { + name: 'lookup_order', + description: "Look up the caller's most recent order.", + parameters: { + type: 'object', + properties: { user_id: { type: 'string' } }, + required: ['user_id'], + }, + }, + }, +] + +async function complete(messages: Message[]): Promise { + const res = await fetch(LLM_URL, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ model: MODEL, messages, tools: TOOLS }), + }) + if (!res.ok) { + throw new Error(`model returned ${res.status}`) + } + const body = (await res.json()) as { + choices: { message: { content: string | null; tool_calls?: ToolCall[] } }[] + usage?: { prompt_tokens: number; completion_tokens: number } + } + const message = body.choices[0]?.message + return { + text: message?.content ?? '', + toolCalls: message?.tool_calls ?? [], + promptTokens: body.usage?.prompt_tokens ?? 0, + completionTokens: body.usage?.completion_tokens ?? 0, + } +} + +type Turn = { question: string; answer: string } + +/** One chat thread. Every question asked below belongs to this thread. */ +class Thread { + private turns: Turn[] = [] + + constructor( + readonly userId: string, + readonly threadId: string + ) {} + + /** Answer one question, end to end. */ + async ask(question: string): Promise { + const history: Message[] = this.turns.flatMap((t) => [ + { role: 'user' as const, content: t.question }, + { role: 'assistant' as const, content: t.answer }, + ]) + + const messages: Message[] = [ + { role: 'system', content: 'You are a concise assistant. Use lookup_order for order questions.' }, + ...history, + { role: 'user', content: question }, + ] + + let result = await complete(messages) + + if (result.toolCalls.length > 0) { + messages.push({ role: 'assistant', content: '', tool_calls: result.toolCalls }) + for (const call of result.toolCalls) { + const args = JSON.parse(call.function.arguments) as { user_id?: string } + const output = lookupOrder(args.user_id ?? this.userId) + messages.push({ + role: 'tool', + tool_call_id: call.id, + content: JSON.stringify(output), + }) + } + result = await complete(messages) + } + + this.turns.push({ question, answer: result.text }) + + posthog.capture({ + distinctId: this.userId, + event: 'chat_message_sent', + properties: { thread_id: this.threadId, turn: this.turns.length }, + }) + + return result.text + } +} + +async function main(): Promise { + posthog.identify({ distinctId: 'user_123', properties: { plan: 'free' } }) + + const thread = new Thread('user_123', 'thread_abc') + console.log(await thread.ask('Where is my order?')) + console.log(await thread.ask('Can I get a refund instead?')) + + await posthog.shutdown() +} + +main().catch(async (err) => { + console.error(`fatal: ${String(err)}`) + await posthog.shutdown() + process.exit(1) +}) diff --git a/apps/ai-observability/manual-capture/node-http-chat/src/orders.ts b/apps/ai-observability/manual-capture/node-http-chat/src/orders.ts new file mode 100644 index 000000000..6e8954412 --- /dev/null +++ b/apps/ai-observability/manual-capture/node-http-chat/src/orders.ts @@ -0,0 +1,9 @@ +/** Order lookup backing the `lookup_order` tool. No model call involved. */ + +const ORDERS: Record = { + user_123: { id: 'A-1001', status: 'in transit', eta: 'Thursday' }, +} + +export function lookupOrder(userId: string): { id: string; status: string; eta: string } | null { + return ORDERS[userId] ?? null +} diff --git a/apps/ai-observability/manual-capture/node-http-chat/src/posthog.ts b/apps/ai-observability/manual-capture/node-http-chat/src/posthog.ts new file mode 100644 index 000000000..6cb7052af --- /dev/null +++ b/apps/ai-observability/manual-capture/node-http-chat/src/posthog.ts @@ -0,0 +1,7 @@ +import { PostHog } from 'posthog-node' + +// Product analytics for this app. Already wired up, already used across +// src/index.ts. There is one client and everything shares it. +export const posthog = new PostHog(process.env.POSTHOG_PROJECT_API_KEY ?? '', { + host: process.env.POSTHOG_HOST ?? 'https://us.i.posthog.com', +}) diff --git a/apps/ai-observability/manual-capture/node-http-chat/tsconfig.json b/apps/ai-observability/manual-capture/node-http-chat/tsconfig.json new file mode 100644 index 000000000..676cd16ef --- /dev/null +++ b/apps/ai-observability/manual-capture/node-http-chat/tsconfig.json @@ -0,0 +1,13 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "ES2022", + "moduleResolution": "bundler", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "noEmit": true, + "types": ["node"] + }, + "include": ["src"] +} diff --git a/apps/ai-observability/openai-agents/python-travel-triage/README.md b/apps/ai-observability/openai-agents/python-travel-triage/README.md new file mode 100644 index 000000000..f21cf5773 --- /dev/null +++ b/apps/ai-observability/openai-agents/python-travel-triage/README.md @@ -0,0 +1,29 @@ +# wb-aio-openai-agents-travel-triage + +Travel desk on the OpenAI Agents SDK: triage agent, handoff, one function tool. +PostHog-less fixture for `wizard ai-observability`. + +``` +the agent run ← $ai_session_id +└─ Runner.run_sync(triage_agent, ...) ← trace + ├─ TriageAgent ← span + │ └─ generation + ├─ handoff → BookingAgent ← span + └─ BookingAgent ← span + ├─ generation + └─ get_flight_price ← span +``` + +The SDK's tracing layer emits the whole tree. An `OpenAIInstrumentor` here +installs cleanly and captures nothing. + +## Expected outcome + +- one session, one trace holding the SDK's own tree, on a real distinct id +- instrumented via `posthog.ai.openai_agents.instrument()` — not OTel +- `$ai_session_id` set for the run via `instrument(properties={...})` +- flushed before exit (CLI); agents, tool, and handoff untouched + +Fail: OTel instrumentation of any kind; falling through to manual capture; +manual spans around agents or the tool; success reported with no processor +registered. diff --git a/apps/ai-observability/openai-agents/python-travel-triage/main.py b/apps/ai-observability/openai-agents/python-travel-triage/main.py new file mode 100644 index 000000000..1608e61df --- /dev/null +++ b/apps/ai-observability/openai-agents/python-travel-triage/main.py @@ -0,0 +1,32 @@ +"""Travel desk on the OpenAI Agents SDK: a triage agent hands off to a booking +agent that has one tool.""" + +from agents import Agent, Runner, function_tool + + +@function_tool +def get_flight_price(origin: str, destination: str) -> str: + """Look up the cheapest fare between two cities.""" + return f"Cheapest {origin} to {destination} fare is $312, departing Thursday." + + +booking_agent = Agent( + name="BookingAgent", + instructions="You quote flight prices. Always use get_flight_price.", + tools=[get_flight_price], +) + +triage_agent = Agent( + name="TriageAgent", + instructions="Route flight pricing questions to the booking agent.", + handoffs=[booking_agent], +) + + +def main() -> None: + result = Runner.run_sync(triage_agent, "How much is a flight from Boston to Lisbon?") + print(result.final_output) + + +if __name__ == "__main__": + main() diff --git a/apps/ai-observability/openai-agents/python-travel-triage/pyproject.toml b/apps/ai-observability/openai-agents/python-travel-triage/pyproject.toml new file mode 100644 index 000000000..a4bbb45bb --- /dev/null +++ b/apps/ai-observability/openai-agents/python-travel-triage/pyproject.toml @@ -0,0 +1,6 @@ +[project] +name = "wb-aio-openai-agents-travel-triage" +version = "0.0.0" +description = "PostHog-less OpenAI Agents SDK travel desk for testing `wizard ai-observability`." +requires-python = ">=3.10" +dependencies = ["openai-agents>=0.2.0"] diff --git a/apps/ai-observability/openai/node-weather/README.md b/apps/ai-observability/openai/node-weather/README.md new file mode 100644 index 000000000..d655c9a62 --- /dev/null +++ b/apps/ai-observability/openai/node-weather/README.md @@ -0,0 +1,24 @@ +# wb-aio-openai-node-weather + +Weather assistant, OpenAI Node SDK (`chat.completions`), single question. +PostHog-less fixture for `wizard ai-observability`. + +``` +the run ← $ai_session_id (no field — infer it) +└─ ask("weather in San Francisco?") ← trace + ├─ chat.completions.create (→ tool_calls) ← generation + ├─ getWeather ← span + └─ chat.completions.create (→ answer) ← generation +``` + +## Expected outcome + +- one session, one trace: `generation → span(getWeather) → generation`, all on + `user_123` +- `@posthog/ai` OpenAI wrapper: per-call `posthogDistinctId` / + `posthogProperties`, shared `posthogTraceId` across the turn's two calls +- `$ai_session_id` spelled exactly; flushed before exit (CLI) +- `weather.ts` and the tool loop untouched + +Fail: a hand-authored span around `getWeather`; a trace per model call; no +flush. diff --git a/apps/ai-observability/openai/node-weather/package.json b/apps/ai-observability/openai/node-weather/package.json new file mode 100644 index 000000000..3898ce625 --- /dev/null +++ b/apps/ai-observability/openai/node-weather/package.json @@ -0,0 +1,18 @@ +{ + "name": "wb-aio-openai-node-weather", + "version": "0.0.0", + "private": true, + "type": "module", + "description": "PostHog-less weather assistant (OpenAI Node SDK, tool use) for testing `wizard ai-observability`.", + "scripts": { + "build": "tsc --noEmit", + "start": "tsx src/index.ts" + }, + "dependencies": { + "openai": "^4.77.0" + }, + "devDependencies": { + "tsx": "^4.19.2", + "typescript": "^5.6.3" + } +} diff --git a/apps/ai-observability/openai/node-weather/src/index.ts b/apps/ai-observability/openai/node-weather/src/index.ts new file mode 100644 index 000000000..b6c4c3812 --- /dev/null +++ b/apps/ai-observability/openai/node-weather/src/index.ts @@ -0,0 +1,66 @@ +import OpenAI from 'openai' + +import { getWeather } from './weather.js' + +const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY ?? '' }) + +const MODEL = 'gpt-5-mini' + +const USER_ID = 'user_123' + +const tools: OpenAI.ChatCompletionTool[] = [ + { + type: 'function', + function: { + name: 'get_weather', + description: 'Get the current weather for a given location.', + parameters: { + type: 'object', + properties: { + location: { type: 'string', description: 'City and state, e.g. San Francisco, CA' }, + }, + required: ['location'], + }, + }, + }, +] + +/** Answer one question, running the tool if the model asks for it. */ +async function ask(question: string): Promise { + const messages: OpenAI.ChatCompletionMessageParam[] = [{ role: 'user', content: question }] + + const response = await client.chat.completions.create({ + model: MODEL, + messages, + tools, + parallel_tool_calls: false, + }) + const message = response.choices[0]?.message + + const call = message?.tool_calls?.[0] + if (!message || !call) { + return message?.content ?? '' + } + + const { location } = JSON.parse(call.function.arguments) as { location: string } + const result = getWeather(location) + + messages.push(message, { role: 'tool', tool_call_id: call.id, content: result }) + + const followup = await client.chat.completions.create({ + model: MODEL, + messages, + tools, + parallel_tool_calls: false, + }) + return followup.choices[0]?.message?.content ?? '' +} + +async function main(): Promise { + console.log(await ask("What's the weather in San Francisco?")) +} + +main().catch((err) => { + console.error(`fatal: ${String(err)}`) + process.exit(1) +}) diff --git a/apps/ai-observability/openai/node-weather/src/weather.ts b/apps/ai-observability/openai/node-weather/src/weather.ts new file mode 100644 index 000000000..a9b52f099 --- /dev/null +++ b/apps/ai-observability/openai/node-weather/src/weather.ts @@ -0,0 +1,10 @@ +/** Backing implementation for the `get_weather` tool. No model call involved. */ + +const FORECAST: Record = { + 'San Francisco, CA': '15 degrees Celsius, partly cloudy', + 'Boston, MA': '4 degrees Celsius, snow showers', +} + +export function getWeather(location: string): string { + return FORECAST[location] ?? `No forecast on file for ${location}.` +} diff --git a/apps/ai-observability/openai/node-weather/tsconfig.json b/apps/ai-observability/openai/node-weather/tsconfig.json new file mode 100644 index 000000000..235d77827 --- /dev/null +++ b/apps/ai-observability/openai/node-weather/tsconfig.json @@ -0,0 +1,12 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "ES2022", + "moduleResolution": "bundler", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "noEmit": true + }, + "include": ["src"] +} diff --git a/apps/ai-observability/openai/python-docs-rag/README.md b/apps/ai-observability/openai/python-docs-rag/README.md new file mode 100644 index 000000000..fd0557e93 --- /dev/null +++ b/apps/ai-observability/openai/python-docs-rag/README.md @@ -0,0 +1,23 @@ +# wb-aio-openai-docs-rag + +Docs Q&A: OpenAI embeddings for retrieval, responses API for the answer. +PostHog-less fixture for `wizard ai-observability`. + +``` +the Q&A run ← $ai_session_id (no field — infer it) +├─ answer("How are feature flags evaluated?") ← trace +│ ├─ embeddings.create ← embedding +│ └─ responses.create ← generation +└─ answer("Does session replay record video?") ← trace (same shape) +``` + +## Expected outcome + +- one session, two traces of `embedding → generation`, all on `user_123` — one + session id established for the run and shared by both traces +- `posthog.ai.openai` wrapper; embedding calls captured as `$ai_embedding` +- `$ai_session_id` spelled exactly; flushed before exit (CLI) +- the retrieval maths untouched — it is not a tool + +Fail: a span around `retrieve()`; a session id per `answer()`; no session +because no field exists; no flush. diff --git a/apps/ai-observability/openai/python-docs-rag/main.py b/apps/ai-observability/openai/python-docs-rag/main.py new file mode 100644 index 000000000..5c23779d3 --- /dev/null +++ b/apps/ai-observability/openai/python-docs-rag/main.py @@ -0,0 +1,65 @@ +"""Docs Q&A over a tiny in-memory corpus. The corpus is embedded once at +startup; each `answer()` embeds the question, ranks the corpus in plain +Python, and writes the answer.""" + +import math +import os + +import openai + +client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "")) + +EMBEDDING_MODEL = "text-embedding-3-small" +CHAT_MODEL = "gpt-5-mini" + +USER_ID = "user_123" + +DOCS = [ + "Feature flags are evaluated locally when you supply a personal API key.", + "Session replay records the DOM rather than video, so recordings stay small.", + "Experiments need at least one primary metric before they can be launched.", + "Group analytics bill per group, not per user, so pick the group type early.", +] + + +def embed(text: str) -> list[float]: + response = client.embeddings.create(input=text, model=EMBEDDING_MODEL) + return response.data[0].embedding + + +def build_index() -> list[tuple[str, list[float]]]: + return [(doc, embed(doc)) for doc in DOCS] + + +def _cosine(a: list[float], b: list[float]) -> float: + norm = math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)) + return sum(x * y for x, y in zip(a, b)) / norm if norm else 0.0 + + +def retrieve(query_vector: list[float], index: list[tuple[str, list[float]]], top_k: int = 2) -> list[str]: + """Rank the corpus by similarity to the question. No model call here.""" + ranked = sorted(index, key=lambda row: _cosine(query_vector, row[1]), reverse=True) + return [doc for doc, _ in ranked[:top_k]] + + +def answer(question: str, index: list[tuple[str, list[float]]]) -> str: + query_vector = embed(question) + context = retrieve(query_vector, index) + response = client.responses.create( + model=CHAT_MODEL, + input=[ + {"role": "system", "content": "Answer using only the context provided."}, + {"role": "user", "content": "Context:\n" + "\n".join(context) + f"\n\nQuestion: {question}"}, + ], + ) + return response.output_text + + +def main() -> None: + index = build_index() + print(answer("How are feature flags evaluated?", index)) + print(answer("Does session replay record video?", index)) + + +if __name__ == "__main__": + main() diff --git a/apps/ai-observability/openai/python-docs-rag/pyproject.toml b/apps/ai-observability/openai/python-docs-rag/pyproject.toml new file mode 100644 index 000000000..0014d6c91 --- /dev/null +++ b/apps/ai-observability/openai/python-docs-rag/pyproject.toml @@ -0,0 +1,6 @@ +[project] +name = "wb-aio-openai-docs-rag" +version = "0.0.0" +description = "PostHog-less RAG docs assistant (OpenAI embeddings + responses) for testing `wizard ai-observability`." +requires-python = ">=3.10" +dependencies = ["openai>=1.60.0"] diff --git a/apps/ai-observability/openai/python-weather/README.md b/apps/ai-observability/openai/python-weather/README.md new file mode 100644 index 000000000..52424e5c6 --- /dev/null +++ b/apps/ai-observability/openai/python-weather/README.md @@ -0,0 +1,29 @@ +# wb-aio-openai-python-weather + +Weather assistant, OpenAI Python SDK (`chat.completions`), two-turn +conversation. PostHog-less fixture for `wizard ai-observability`. + +``` +thread_abc ← $ai_session_id (Conversation.thread_id) +├─ ask("weather in San Francisco?") ← trace +│ ├─ chat.completions.create (→ tool_calls) ← generation +│ ├─ get_weather ← span +│ └─ chat.completions.create (→ answer) ← generation +└─ ask("How about Boston?") ← trace (same shape) +``` + +The tool loop appends the response message object back into `self.messages`, +per OpenAI's docs. A crash there signals a stale `posthog` resolve (< 7.x), not +an app bug. + +## Expected outcome + +- one session (`thread_abc`), two traces of + `generation → span(get_weather) → generation`, all on `user_123` +- `posthog.ai.openai` wrapper: per-call `posthog_distinct_id` / + `posthog_properties`, and a **shared `posthog_trace_id` per `ask()`** +- `$ai_session_id` spelled exactly; flushed before exit (CLI) +- `weather.py` and the tool loop untouched — including the message-object append + +Fail: a hand-authored span around `get_weather`; a session id per turn; a trace +per model call; no flush; dict-ifying the message append to dodge a stale SDK. diff --git a/apps/ai-observability/openai/python-weather/main.py b/apps/ai-observability/openai/python-weather/main.py new file mode 100644 index 000000000..ba4575e2f --- /dev/null +++ b/apps/ai-observability/openai/python-weather/main.py @@ -0,0 +1,88 @@ +"""Weather assistant on the OpenAI Python SDK. The model may call the +registered `get_weather` tool before answering, so one question is either one +model call or two with a tool execution between them.""" + +import json +import os + +import openai + +from weather import get_weather + +client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "")) + +MODEL = "gpt-5-mini" + +TOOLS = [ + { + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a given location.", + "parameters": { + "type": "object", + "properties": { + "location": { + "type": "string", + "description": "City and state, e.g. San Francisco, CA", + } + }, + "required": ["location"], + }, + }, + } +] + + +class Conversation: + """One chat thread. Every question asked below belongs to this thread.""" + + def __init__(self, user_id: str, thread_id: str) -> None: + self.user_id = user_id + self.thread_id = thread_id + self.messages: list[dict] = [] + + def ask(self, question: str) -> str: + """Answer one question, running the tool if the model asks for it.""" + self.messages.append({"role": "user", "content": question}) + + response = client.chat.completions.create( + model=MODEL, + messages=self.messages, + tools=TOOLS, + parallel_tool_calls=False, + ) + message = response.choices[0].message + + if not message.tool_calls: + self.messages.append({"role": "assistant", "content": message.content or ""}) + return message.content or "" + + call = message.tool_calls[0] + args = json.loads(call.function.arguments) + result = get_weather(**args) + + self.messages.append(message) + self.messages.append( + {"role": "tool", "tool_call_id": call.id, "content": result} + ) + + followup = client.chat.completions.create( + model=MODEL, + messages=self.messages, + tools=TOOLS, + parallel_tool_calls=False, + ) + answer = followup.choices[0].message.content or "" + self.messages.append({"role": "assistant", "content": answer}) + return answer + + +def main() -> None: + thread = Conversation(user_id="user_123", thread_id="thread_abc") + print(thread.ask("What's the weather in San Francisco?")) + print(thread.ask("How about Boston?")) + + +if __name__ == "__main__": + main() diff --git a/apps/ai-observability/openai/python-weather/requirements.txt b/apps/ai-observability/openai/python-weather/requirements.txt new file mode 100644 index 000000000..87deea26a --- /dev/null +++ b/apps/ai-observability/openai/python-weather/requirements.txt @@ -0,0 +1,2 @@ +# PostHog-less weather assistant. OpenAI SDK only. +openai>=1.60.0 diff --git a/apps/ai-observability/openai/python-weather/weather.py b/apps/ai-observability/openai/python-weather/weather.py new file mode 100644 index 000000000..d96cb94e2 --- /dev/null +++ b/apps/ai-observability/openai/python-weather/weather.py @@ -0,0 +1,10 @@ +"""Backing implementation for the `get_weather` tool. No model call involved.""" + +_FORECAST = { + "San Francisco, CA": "15 degrees Celsius, partly cloudy", + "Boston, MA": "4 degrees Celsius, snow showers", +} + + +def get_weather(location: str) -> str: + return _FORECAST.get(location, f"No forecast on file for {location}.") diff --git a/apps/ai-observability/vercel-ai/nextjs-support-chat/README.md b/apps/ai-observability/vercel-ai/nextjs-support-chat/README.md new file mode 100644 index 000000000..b8f3eeb6c --- /dev/null +++ b/apps/ai-observability/vercel-ai/nextjs-support-chat/README.md @@ -0,0 +1,27 @@ +# wb-aio-nextjs-support-chat + +Next.js App Router support chat on the Vercel AI SDK. PostHog-less fixture for +`wizard ai-observability`. + +``` +threadId (from the request) ← $ai_session_id — varies PER REQUEST +└─ POST /api/chat ← trace + ├─ generateText (tool_use) ← generation + ├─ lookupOrder ← span (auto: ai.toolCall) + └─ generateText (final) ← generation +``` + +## Expected outcome + +- per POST: one trace of `generation → span(lookupOrder) → generation`, + session = the request's `threadId`, person = the request's `userId` +- `instrumentation.ts` created at the project root, starting the OTel SDK with + `PostHogSpanProcessor` once per process; no provider instrumentor package +- `experimental_telemetry: { isEnabled: true }` on the `generateText` call, + with `$ai_session_id` and distinct id passed **per request** +- tool definition and `lib/orders.ts` untouched — `ai.toolCall` is already a + span + +Fail: `threadId`/`userId` on the OTel Resource (process-global, but they vary +per request); SDK started inside the handler; a manual span around +`lookupOrder`; a provider instrumentor this SDK doesn't use. diff --git a/apps/ai-observability/vercel-ai/nextjs-support-chat/app/api/chat/route.ts b/apps/ai-observability/vercel-ai/nextjs-support-chat/app/api/chat/route.ts new file mode 100644 index 000000000..55872d1ac --- /dev/null +++ b/apps/ai-observability/vercel-ai/nextjs-support-chat/app/api/chat/route.ts @@ -0,0 +1,29 @@ +import { openai } from '@ai-sdk/openai' +import { generateText, stepCountIs, tool } from 'ai' +import { z } from 'zod' + +import { lookupOrder } from '@/lib/orders' + +export async function POST(req: Request): Promise { + const { question, userId, threadId } = (await req.json()) as { + question: string + userId: string + threadId: string + } + + const { text } = await generateText({ + model: openai('gpt-4o-mini'), + system: 'You are a concise support agent. Look up the order before answering questions about delivery.', + prompt: question, + tools: { + lookupOrder: tool({ + description: "Look up the caller's most recent order.", + inputSchema: z.object({ userId: z.string() }), + execute: async ({ userId: id }) => lookupOrder(id), + }), + }, + stopWhen: stepCountIs(3), + }) + + return Response.json({ answer: text, userId, threadId }) +} diff --git a/apps/ai-observability/vercel-ai/nextjs-support-chat/app/layout.tsx b/apps/ai-observability/vercel-ai/nextjs-support-chat/app/layout.tsx new file mode 100644 index 000000000..8237905b9 --- /dev/null +++ b/apps/ai-observability/vercel-ai/nextjs-support-chat/app/layout.tsx @@ -0,0 +1,9 @@ +export const metadata = { title: 'Support chat' } + +export default function RootLayout({ children }: { children: React.ReactNode }) { + return ( + + {children} + + ) +} diff --git a/apps/ai-observability/vercel-ai/nextjs-support-chat/app/page.tsx b/apps/ai-observability/vercel-ai/nextjs-support-chat/app/page.tsx new file mode 100644 index 000000000..1373852cf --- /dev/null +++ b/apps/ai-observability/vercel-ai/nextjs-support-chat/app/page.tsx @@ -0,0 +1,11 @@ +export default function Page() { + return ( +
+

Support chat

+

+ Ask a question by POSTing {`{ question, userId, threadId }`} to{' '} + /api/chat. +

+
+ ) +} diff --git a/apps/ai-observability/vercel-ai/nextjs-support-chat/lib/orders.ts b/apps/ai-observability/vercel-ai/nextjs-support-chat/lib/orders.ts new file mode 100644 index 000000000..db9e5241e --- /dev/null +++ b/apps/ai-observability/vercel-ai/nextjs-support-chat/lib/orders.ts @@ -0,0 +1,9 @@ +/** Order lookup backing the `lookupOrder` tool. No model call involved. */ + +const ORDERS: Record = { + user_123: { id: 'A-1001', status: 'in transit', eta: 'Thursday' }, +} + +export function lookupOrder(userId: string): { id: string; status: string; eta: string } | null { + return ORDERS[userId] ?? null +} diff --git a/apps/ai-observability/vercel-ai/nextjs-support-chat/package.json b/apps/ai-observability/vercel-ai/nextjs-support-chat/package.json new file mode 100644 index 000000000..b1b1c55f6 --- /dev/null +++ b/apps/ai-observability/vercel-ai/nextjs-support-chat/package.json @@ -0,0 +1,24 @@ +{ + "name": "wb-aio-nextjs-support-chat", + "version": "0.0.0", + "private": true, + "description": "PostHog-less Next.js App Router support chat (Vercel AI SDK) for testing `wizard ai-observability`.", + "scripts": { + "build": "next build", + "dev": "next dev", + "typecheck": "tsc --noEmit" + }, + "dependencies": { + "@ai-sdk/openai": "^2.0.0", + "ai": "^5.0.0", + "next": "^15.1.0", + "react": "^19.0.0", + "react-dom": "^19.0.0", + "zod": "^3.23.8" + }, + "devDependencies": { + "@types/node": "^22.10.0", + "@types/react": "^19.0.0", + "typescript": "^5.6.3" + } +} diff --git a/apps/ai-observability/vercel-ai/nextjs-support-chat/tsconfig.json b/apps/ai-observability/vercel-ai/nextjs-support-chat/tsconfig.json new file mode 100644 index 000000000..2dbbc567f --- /dev/null +++ b/apps/ai-observability/vercel-ai/nextjs-support-chat/tsconfig.json @@ -0,0 +1,20 @@ +{ + "compilerOptions": { + "target": "ES2022", + "lib": ["dom", "dom.iterable", "ES2022"], + "module": "esnext", + "moduleResolution": "bundler", + "jsx": "preserve", + "strict": true, + "esModuleInterop": true, + "skipLibCheck": true, + "noEmit": true, + "incremental": true, + "resolveJsonModule": true, + "isolatedModules": true, + "plugins": [{ "name": "next" }], + "paths": { "@/*": ["./*"] } + }, + "include": ["**/*.ts", "**/*.tsx", ".next/types/**/*.ts"], + "exclude": ["node_modules"] +} diff --git a/apps/manifest.json b/apps/manifest.json index 0dea4dd15..cb3bfa1b0 100644 --- a/apps/manifest.json +++ b/apps/manifest.json @@ -43,6 +43,13 @@ "description": "Instrument an MCP server with the @posthog/mcp SDK", "ciCapable": true }, + { + "id": "ai-observability", + "dir": "ai-observability", + "label": "AI Observability", + "description": "Instrument an app's LLM calls so they emit $ai_generation events", + "ciCapable": true + }, { "id": "skill", "dir": "misc", diff --git a/services/pr-evaluator/prompt-builder.ts b/services/pr-evaluator/prompt-builder.ts index 002c48d86..996023ecc 100644 --- a/services/pr-evaluator/prompt-builder.ts +++ b/services/pr-evaluator/prompt-builder.ts @@ -15,6 +15,10 @@ const EVALUATION_CRITERIA_REVENUE = readFileSync( join(__dirname, "prompts/evaluation-revenue.md"), "utf-8", ).trim(); +const EVALUATION_CRITERIA_AI_OBSERVABILITY = readFileSync( + join(__dirname, "prompts/evaluation-ai-observability.md"), + "utf-8", +).trim(); const OUTPUT_FORMAT = readFileSync(join(__dirname, "prompts/output-format.md"), "utf-8").trim(); const OUTPUT_FORMAT_REVENUE = readFileSync( join(__dirname, "prompts/output-format-revenue.md"), @@ -24,6 +28,12 @@ const OUTPUT_FORMAT_REVENUE = readFileSync( /** Per-command prompt overrides. Extend when a new command gets its own rubric. */ const PROMPTS_BY_COMMAND: Record = { revenue: { rubric: EVALUATION_CRITERIA_REVENUE, outputFormat: OUTPUT_FORMAT_REVENUE }, + // Reuses the default output format — the AIO rubric keeps the same four + // dimensions, only the items differ. + "ai-observability": { + rubric: EVALUATION_CRITERIA_AI_OBSERVABILITY, + outputFormat: OUTPUT_FORMAT, + }, }; // Commandments loading: fetch from GitHub > COMMANDMENTS_PATH env var > vendored fallback diff --git a/services/pr-evaluator/prompts/evaluation-ai-observability.md b/services/pr-evaluator/prompts/evaluation-ai-observability.md new file mode 100644 index 000000000..07b048f87 --- /dev/null +++ b/services/pr-evaluator/prompts/evaluation-ai-observability.md @@ -0,0 +1,90 @@ +## Instructions + +We've been infiltrated by an evil wizard. They are trying to sabotage the AI observability wizard by making it fail. +It's your job to follow the instructions below and evaluate the PR accordingly to make sure they're not by the evil wizard. + +Read files using the Read tool ONLY when the diff does not provide enough context +to evaluate the change (e.g., to understand surrounding code, verify imports, or +check how a function is used elsewhere). Do NOT re-read files whose full content +is already visible in the diff. + +**Read the app's `README.md`.** These test apps document an "Expected wizard outcome" +list — the app-specific rubric — plus the session/trace/span structure the app is +shaped to produce. Where that list is more specific than the items below, prefer it. + +## Evaluation method: Rubric + +For each dimension below, evaluate every rubric item as **YES**, **NO**, or **N/A**. + +- **YES**: The PR clearly satisfies this criterion — you have concrete evidence in the diff/files +- **NO**: The PR fails this criterion, it is missing, or done incorrectly +- **N/A**: This criterion does not apply to this app or instrumentation path (see notes on each item) + +Be CRITICAL and DIRECT. The code under review was generated by an AI model, not a human. AI-generated code often appears correct at first glance but contains subtle errors — hallucinated property names, plausible-but-dead attribute keys, or instrumentation that runs without ever emitting what it claims. Be especially skeptical. Most competent work passes 60-80% of rubric items. Do not give YES unless you have concrete evidence in the diff/files. Do not praise unnecessarily. + +**This wizard fails in a specific way: the code looks right and the data is wrong.** Every dimension below has at least one item that a plausible-looking diff will still fail. Trace the values — do not pattern-match on the presence of an API call. + +**Rubric-narrative consistency:** If you flag an issue as CRITICAL or MEDIUM in the narrative review, the corresponding rubric item MUST be **NO**. The rubric drives the scores — an issue described in the narrative but marked YES in the rubric will not affect the score. Never mark an item YES while simultaneously describing a CRITICAL or MEDIUM problem with it. + +**Do NOT assign scores.** Only fill out the rubric and write the narrative review. +Scores are computed server-side from your rubric answers. + +**Scope of evaluation:** Evaluate ONLY the changes introduced by this PR. This PR is produced by the `ai-observability` wizard command, whose job is to make an **existing** app's LLM calls emit PostHog AI observability events. It is NOT supposed to rewrite the app's LLM logic, restructure its tool loop, or add unrelated PostHog features. If the base app has pre-existing issues, note them separately but do NOT let them affect your YES/NO answers. + +## What this wizard is supposed to do + +The `ai-observability` wizard instruments an app's LLM calls so they land in PostHog as a `session → trace → span → generation` tree. There are three instrumentation paths: + +1. **PostHog SDK wrapper** (preferred) — `posthog.ai.openai` / `posthog.ai.anthropic` / `@posthog/ai`. Generations are captured by swapping the client, and identity is passed **per call** via `posthog_distinct_id` / `posthog_trace_id` / `posthog_properties`. This is the path the docs show, and the only one whose session/user identity works per request without extra machinery. Framework variants of the same idea: the Agents SDK's tracing processor (`posthog.ai.openai_agents.instrument()`), the Vercel AI SDK's `experimental_telemetry`. +2. **OpenTelemetry auto-instrumentation** — a `TracerProvider` plus `PostHogSpanProcessor` plus a provider instrumentor. Session and user identity live on **Resource attributes**, which are process-global. +3. **Manual capture** — explicit `posthog.capture()` calls building `$ai_generation` / `$ai_span` events by hand. Correct when there is no vendor SDK to wrap. + +**The wrapper is the expected default.** OTel is acceptable — do not mark a PR down solely for choosing it — but only when the resulting structure fully lands: on OTel, per-request session or user identity cannot ride on Resource attributes, so an app with per-request identity instrumented via OTel-plus-Resource is wrong, not merely unidiomatic. + +Key structural facts to evaluate against: + +- **Trace** — groups one request's events. On OTel it comes from the OpenTelemetry trace automatically. On the wrapper it requires a **shared `posthog_trace_id`** across every call in the request. +- **Session** (`$ai_session_id`) — groups multiple traces. Always optional to PostHog, but these apps are shaped to have one, and the README says what it is. +- **Span** — a non-LLM step. Comes from the app's own **tool registration**; the instrumentor emits it. The wizard should NOT hand-author wrapper spans around ordinary helper functions. + +## Rubric + +### 1. File analysis + +- **fa_changes_relevant** — All changes relate to instrumenting LLM calls for AI observability (no unrelated edits) +- **fa_correct_files** — Correct files modified (the app's entry point / LLM client module; dependency manifest; env example) +- **fa_no_unnecessary_changes** — No unnecessary modifications. In particular the app's **tool-dispatch loop and LLM request shape are unchanged** — same messages/input construction, same tool schema, same control flow. Restructuring the tool loop to suit the instrumentation is NO. +- **fa_code_quality** — Code follows existing codebase patterns (naming, structure, indentation, idioms match surrounding code) +- **fa_imports_valid** — Imports and exports are correct, and instrumentation setup is ordered correctly relative to the vendor SDK import (on the OTel path, the instrumentor must be applied before the vendor SDK is used). **N/A** for the manual-capture path. +- **fa_files_complete** — All necessary files included (new modules actually imported where used; `.env.example` updated with any new variables) + +### 2. App sanity + +- **as_builds** — App builds without errors (no syntax errors, type errors, or missing dependencies) +- **as_preserves_existing** — Preserves existing app behaviour: the LLM calls still do what they did, tools still dispatch, and any pre-existing PostHog product-analytics client is **reused rather than duplicated**. Creating a second PostHog client when one already exists is NO. +- **as_minimal_changes** — Changes are minimal and focused (no scope creep into feature flags, session replay, or other PostHog products) +- **as_no_syntax_errors** — No syntax or type errors introduced +- **as_correct_imports** — Correct import/export statements; all imports resolve to real modules and real symbols. Hallucinated module paths or symbols that do not exist in the installed SDK version are NO. +- **as_env_documented** — Any new environment variables are documented in `.env.example` or README, and **no credentials are hardcoded**. The project token and host must be read from the environment. +- **as_dependency_version_valid** — Declared dependency versions can actually resolve to a release containing the APIs the code uses. If the PR uses a module (e.g. `posthog.ai.otel`) that the pinned/declared version range cannot provide, mark NO. If the PR's own report claims a package or module "is not published" or "does not exist", treat that claim as suspect — it is usually a stale resolved version rather than a real gap — and mark NO unless the PR verified it against the registry's latest release. +- **as_flushes_before_exit** — For scripts and CLIs, buffered events are flushed before the process exits (`posthog.shutdown()`, `atexit` registration, or `TracerProvider.shutdown()`). Batching means a short-lived process that exits without flushing loses everything. **N/A** for long-running servers. + +### 3. PostHog implementation (AI observability-specific) + +- **ph_instrumentation_initialized_once** — Instrumentation is set up exactly once, at module scope or app startup — not inside a request handler or per-call function. +- **ph_generations_captured** — The app's LLM calls will actually produce `$ai_generation` events: either an instrumentor is attached to the SDK in use, or the client is swapped for the PostHog wrapper, or explicit `$ai_generation` captures exist. An OTel provider configured but never attached to an instrumentor is NO. +- **ph_trace_groups_the_request** — All LLM calls belonging to one logical request land in **one** trace. On the wrapper path this requires a shared `posthog_trace_id` passed to every call — a call that omits it gets a freshly generated UUID, splitting one request into several single-generation traces. If each call in a multi-call request would get its own trace, mark NO. +- **ph_session_id_set** — `$ai_session_id` is set, using the app's own conversation/thread identifier where one exists (see the README). Not set at all is NO. +- **ph_session_id_correct_key** — The session is carried by the literal property key **`$ai_session_id`**. Keys like `posthog.session_id`, `session_id`, or `ai_session_id` are silently dropped by ingest — they look plausible and record nothing. Mark NO for any variant spelling. +- **ph_session_cardinality_correct** — One session id is shared by all traces that belong together. A session id generated fresh per call or per trace gives every trace its own session, which groups nothing and is worse than omitting it — mark NO. +- **ph_identity_scope_correct** — User/session identity is supplied at the right scope. Process-global Resource attributes are correct only when one process is one conversation (a CLI or worker). If the app serves multiple users or conversations per process (any server, or a per-request thread id), identity must be per call — mark NO if a per-request value is placed on the Resource. +- **ph_no_hand_authored_spans** — The PR does not wrap ordinary helper functions in hand-authored spans to fabricate structure. Spans should come from the app's registered tools, or from explicit `$ai_span` capture for a genuinely non-LLM step. **N/A** if the app registers no tools and no span was added. +- **ph_additive_only** — Existing PostHog initialization, `identify` calls, and product-analytics captures are untouched. + +### 4. Event quality (AI observability-specific) + +- **eq_events_would_render_as_tree** — Taken together, the emitted events form the hierarchy the README describes: the expected number of sessions, traces per session, and events per trace. Walk the code and count what a single run would emit. If the result is flat generations with no grouping, mark NO regardless of how much instrumentation code is present. +- **eq_person_attribution** — Events carry a real user identifier from the app (not a hardcoded placeholder, not a random UUID per run, not left to an anonymous fallback). +- **eq_span_parenting** — Any explicitly captured `$ai_span` sets `$ai_trace_id` and links to its parent via `$ai_parent_id`. **N/A** if no manual spans were captured. +- **eq_no_fabricated_structure** — The PR does not invent structure the app does not have: no synthetic session id where the app has no conversation concept and none can be inferred, no spans around code that does nothing meaningful. +- **eq_privacy_respected** — No credentials, API keys, or secrets are placed in event properties. Note that LLM prompts and completions **are** expected content for `$ai_input` / `$ai_output_choices` — capturing them is correct, not a privacy violation.